History

Merge tag 'mlx5-updates-2019-09-01-v2' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux

Saeed Mahameed says: ==================== mlx5-updates-2019-09-01 (Software steering support) Abstract: -------- Mellanox ConnetX devices supports packet matching, packet modification and redirection. These functionalities are also referred to as flow-steering. To configure a steering rule, the rule is written to the device owned memory, this memory is accessed and cached by the device when processing a packet. Steering rules are constructed from multiple steering entries (STE). Rules are configured using the Firmware command interface. The Firmware processes the given driver command and translates them to STEs, then writes them to the device memory in the current steering tables. This process is slow due to the architecture of the command interface and the processing complexity of each rule. The highlight of this patchset is to cut the middle man (The firmware) and do steering rules programming into device directly from the driver, with no firmware intervention whatsoever. Motivation: ----------- Software (driver managed) steering allows for high rule insertion rates compared to the FW steering described above, this is achieved by using internal RDMA writes to the device owned memory instead of the slow command interface to program steering rules. Software (driver managed) steering, doesn't depend on new FW for new steering functionality, new implementations can be done in the driver skipping the FW layer. Performance: ------------ The insertion rate on a single core using the new approach allows programming ~300K rules per sec. (Done via direct raw test to the new mlx5 sw steering layer, without any kernel layer involved). Test: TC L2 rules 33K/s with Software steering (this patchset). 5K/s with FW and current driver. This will improve OVS based solution performance. Architecture and implementation details: ---------------------------------------- Software steering will be dynamically selected via devlink device parameter. Example: $ devlink dev param show pci/0000:06:00.0 name flow_steering_mode pci/0000:06:00.0: name flow_steering_mode type driver-specific values: cmode runtime value smfs mlx5 software steering module a.k.a (DR - Direct Rule) is implemented and contained in mlx5/core/steering directory and controlled by MLX5_SW_STEERING kconfig flag. mlx5 core steering layer (fs_core) already provides a shim layer for implementing different steering mechanisms, software steering will leverage that as seen at the end of this series. When Software Steering for a specific steering domain (NIC/RDMA/Vport/ESwitch, etc ..) is supported, it will cause rules targeting this domain to be created using SW steering instead of FW. The implementation includes: Domain - The steering domain is the object that all other object resides in. It holds the memory allocator, send engine, locks and other shared data needed by lower objects such as table, matcher, rule, action. Each domain can contain multiple tables. Domain is equivalent to namespaces e.g (NIC/RDMA/Vport/ESwitch, etc ..) as implemented currently in mlx5_core fs_core (flow steering core). Table - Table objects are used for holding multiple matchers, each table has a level used to prevent processing loops. Packets are being directed to this table once it is set as the root table, this is done by fs_core using a FW command. A packet is being processed inside the table matcher by matcher until a successful hit, otherwise the packet will perform the default action. Matcher - Matchers objects are used to specify the fields mask for matching when processing a packet. A matcher belongs to a table, each matcher can hold multiple rules, each rule with different matching values corresponding to the matcher mask. Each matcher has a priority used for rule processing order inside the table. Action - Action objects are created to specify different steering actions such as count, reformat (encapsulate, decapsulate, ...), modify header, forward to table and many other actions. When creating a rule a sequence of actions can be provided to be executed on a successful match. Rule - Rule objects are used to specify a specific match on packets as well as the actions that should be executed. A rule belongs to a matcher. STE - This layer is used to hold the specific STE format for the device and to convert the requested rule to STEs. Each rule is constructed of an STE chain, Multiple rules construct a steering graph. Each node in the graph is a hash table containing multiple STEs. The index of each STE in the hash table is being calculated using a CRC32 hash function. Memory pool - Used for managing and caching device owned memory for rule insertion. The memory is being allocated using DM (device memory) API. Communication with device - layer for standard RDMA operation using RC QP to configure the device steering. Command utility - This module holds all of the FW commands that are required for SW steering to function. Patch planning and files: ------------------------- 1) First patch, adds the support to Add flow steering actions to fs_cmd shim layer. 2) Next 12 patch will add a file per each Software steering functionality/module as described above. (See patches with title: DR, *) 3) Add CONFIG_MLX5_SW_STEERING for software steering support and enable build with the new files 4) Next two patches will add the support for software steering in mlx5 steering shim layer net/mlx5: Add API to set the namespace steering mode net/mlx5: Add direct rule fs_cmd implementation 5) Last two patches will add the new devlink parameter to select mlx5 steering mode, will be valid only for switchdev mode for now. Two modes are supported: 1. DMFS - Device managed flow steering 2. SMFS - Software/Driver managed flow steering. In the DMFS mode, the HW steering entities are created through the FW. In the SMFS mode this entities are created though the driver directly. The driver will use the devlink steering mode only if the steering domain supports it, for now SMFS will manages only the switchdev eswitch steering domain. User command examples: - Set SMFS flow steering mode:: $ devlink dev param set pci/0000:06:00.0 name flow_steering_mode value "smfs" cmode runtime - Read device flow steering mode:: $ devlink dev param show pci/0000:06:00.0 name flow_steering_mode pci/0000:06:00.0: name flow_steering_mode type driver-specific values: cmode runtime value smfs ====================

David S. Miller committed 7 years ago

Commit 94810bd2 parents 67538eb e890acd44 files · +12464 −371Browse files

Unified · Split

44 files
  1. Documentation/networking/device_drivers/mellanox/mlx5.rst +33 −0
  2. drivers/infiniband/hw/mlx5/cmd.c +0 −130
  3. drivers/infiniband/hw/mlx5/cmd.h +0 −4
  4. drivers/infiniband/hw/mlx5/flow.c +12 −9
  5. drivers/infiniband/hw/mlx5/main.c +34 −75
  6. drivers/infiniband/hw/mlx5/mlx5_ib.h +4 −3
  7. drivers/net/ethernet/mellanox/mlx5/core/Kconfig +7 −0
  8. drivers/net/ethernet/mellanox/mlx5/core/Makefile +8 −1
  9. drivers/net/ethernet/mellanox/mlx5/core/devlink.c +111 −1
  10. drivers/net/ethernet/mellanox/mlx5/core/en/tc_tun.c +14 −13
  11. drivers/net/ethernet/mellanox/mlx5/core/en_rep.h +1 −1
  12. drivers/net/ethernet/mellanox/mlx5/core/en_tc.c +26 −25
  13. drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +4 −3
  14. drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +68 −19
  15. drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c +89 −27
  16. drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.h +25 −0
  17. drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +159 −1
  18. drivers/net/ethernet/mellanox/mlx5/core/fs_core.h +38 −1
  19. drivers/net/ethernet/mellanox/mlx5/core/lib/dm.c +223 −0
  20. drivers/net/ethernet/mellanox/mlx5/core/main.c +5 −0
  21. drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +3 −0
  22. drivers/net/ethernet/mellanox/mlx5/core/rdma.c +5 −3
  23. drivers/net/ethernet/mellanox/mlx5/core/steering/Makefile +2 −0
  24. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_action.c +1588 −0
  25. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_cmd.c +480 −0
  26. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_crc32.c +98 −0
  27. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_domain.c +395 −0
  28. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_fw.c +93 −0
  29. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_icm_pool.c +570 −0
  30. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_matcher.c +770 −0
  31. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_rule.c +1243 −0
  32. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c +976 −0
  33. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste.c +2308 −0
  34. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_table.c +294 −0
  35. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_types.h +1060 −0
  36. drivers/net/ethernet/mellanox/mlx5/core/steering/fs_dr.c +600 −0
  37. drivers/net/ethernet/mellanox/mlx5/core/steering/fs_dr.h +60 −0
  38. drivers/net/ethernet/mellanox/mlx5/core/steering/mlx5_ifc_dr.h +604 −0
  39. drivers/net/ethernet/mellanox/mlx5/core/steering/mlx5dr.h +212 −0
  40. include/linux/mlx5/device.h +7 −0
  41. include/linux/mlx5/driver.h +14 −0
  42. include/linux/mlx5/eswitch.h +7 −1
  43. include/linux/mlx5/fs.h +16 −17
  44. include/linux/mlx5/mlx5_ifc.h +198 −37
Documentation/networking/device_drivers/mellanox/mlx5.rst+33−0
@@ -11,6 +11,7 @@ Contents · 10 unmodified lines
11 11
12- `Enabling the driver and kconfig options`_12- `Enabling the driver and kconfig options`_
13- `Devlink info`_13- `Devlink info`_
14- `Devlink parameters`_
14- `Devlink health reporters`_15- `Devlink health reporters`_
15- `mlx5 tracepoints`_16- `mlx5 tracepoints`_
16 17
@@ -122,6 +123,38 @@ User command example:: · 105 unmodified lines
122 stored:123 stored:
123 fw.version 16.26.0100124 fw.version 16.26.0100
124 125
126Devlink parameters
127==================
128
129flow_steering_mode: Device flow steering mode
130---------------------------------------------
131The flow steering mode parameter controls the flow steering mode of the driver.
132Two modes are supported:
1331. 'dmfs' - Device managed flow steering.
1342. 'smfs - Software/Driver managed flow steering.
135
136In DMFS mode, the HW steering entities are created and managed through the
137Firmware.
138In SMFS mode, the HW steering entities are created and managed though by
139the driver directly into Hardware without firmware intervention.
140
141SMFS mode is faster and provides better rule inserstion rate compared to default DMFS mode.
142
143User command examples:
144
145- Set SMFS flow steering mode::
146
147 $ devlink dev param set pci/0000:06:00.0 name flow_steering_mode value "smfs" cmode runtime
148
149- Read device flow steering mode::
150
151 $ devlink dev param show pci/0000:06:00.0 name flow_steering_mode
152 pci/0000:06:00.0:
153 name flow_steering_mode type driver-specific
154 values:
155 cmode runtime value smfs
156
157
125Devlink health reporters158Devlink health reporters
126========================159========================
127 160
drivers/infiniband/hw/mlx5/cmd.c+0−130
@@ -186,136 +186,6 @@ int mlx5_cmd_dealloc_memic(struct mlx5_dm *dm, phys_addr_t addr, u64 length) · 185 unmodified lines
186 return err;186 return err;
187}187}
188 188
189int mlx5_cmd_alloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
190 u16 uid, phys_addr_t *addr, u32 *obj_id)
191{
192 struct mlx5_core_dev *dev = dm->dev;
193 u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {};
194 u32 in[MLX5_ST_SZ_DW(create_sw_icm_in)] = {};
195 unsigned long *block_map;
196 u64 icm_start_addr;
197 u32 log_icm_size;
198 u32 num_blocks;
199 u32 max_blocks;
200 u64 block_idx;
201 void *sw_icm;
202 int ret;
203
204 MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
205 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
206 MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, MLX5_OBJ_TYPE_SW_ICM);
207 MLX5_SET(general_obj_in_cmd_hdr, in, uid, uid);
208
209 switch (type) {
210 case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
211 icm_start_addr = MLX5_CAP64_DEV_MEM(dev,
212 steering_sw_icm_start_address);
213 log_icm_size = MLX5_CAP_DEV_MEM(dev, log_steering_sw_icm_size);
214 block_map = dm->steering_sw_icm_alloc_blocks;
215 break;
216 case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
217 icm_start_addr = MLX5_CAP64_DEV_MEM(dev,
218 header_modify_sw_icm_start_address);
219 log_icm_size = MLX5_CAP_DEV_MEM(dev,
220 log_header_modify_sw_icm_size);
221 block_map = dm->header_modify_sw_icm_alloc_blocks;
222 break;
223 default:
224 return -EINVAL;
225 }
226
227 num_blocks = (length + MLX5_SW_ICM_BLOCK_SIZE(dev) - 1) >>
228 MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
229 max_blocks = BIT(log_icm_size - MLX5_LOG_SW_ICM_BLOCK_SIZE(dev));
230 spin_lock(&dm->lock);
231 block_idx = bitmap_find_next_zero_area(block_map,
232 max_blocks,
233 0,
234 num_blocks, 0);
235
236 if (block_idx < max_blocks)
237 bitmap_set(block_map,
238 block_idx, num_blocks);
239
240 spin_unlock(&dm->lock);
241
242 if (block_idx >= max_blocks)
243 return -ENOMEM;
244
245 sw_icm = MLX5_ADDR_OF(create_sw_icm_in, in, sw_icm);
246 icm_start_addr += block_idx << MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
247 MLX5_SET64(sw_icm, sw_icm, sw_icm_start_addr,
248 icm_start_addr);
249 MLX5_SET(sw_icm, sw_icm, log_sw_icm_size, ilog2(length));
250
251 ret = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
252 if (ret) {
253 spin_lock(&dm->lock);
254 bitmap_clear(block_map,
255 block_idx, num_blocks);
256 spin_unlock(&dm->lock);
257
258 return ret;
259 }
260
261 *addr = icm_start_addr;
262 *obj_id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
263
264 return 0;
265}
266
267int mlx5_cmd_dealloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
268 u16 uid, phys_addr_t addr, u32 obj_id)
269{
270 struct mlx5_core_dev *dev = dm->dev;
271 u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {};
272 u32 in[MLX5_ST_SZ_DW(general_obj_in_cmd_hdr)] = {};
273 unsigned long *block_map;
274 u32 num_blocks;
275 u64 start_idx;
276 int err;
277
278 num_blocks = (length + MLX5_SW_ICM_BLOCK_SIZE(dev) - 1) >>
279 MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
280
281 switch (type) {
282 case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
283 start_idx =
284 (addr - MLX5_CAP64_DEV_MEM(
285 dev, steering_sw_icm_start_address)) >>
286 MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
287 block_map = dm->steering_sw_icm_alloc_blocks;
288 break;
289 case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
290 start_idx =
291 (addr -
292 MLX5_CAP64_DEV_MEM(
293 dev, header_modify_sw_icm_start_address)) >>
294 MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
295 block_map = dm->header_modify_sw_icm_alloc_blocks;
296 break;
297 default:
298 return -EINVAL;
299 }
300
301 MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
302 MLX5_CMD_OP_DESTROY_GENERAL_OBJECT);
303 MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, MLX5_OBJ_TYPE_SW_ICM);
304 MLX5_SET(general_obj_in_cmd_hdr, in, obj_id, obj_id);
305 MLX5_SET(general_obj_in_cmd_hdr, in, uid, uid);
306
307 err = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
308 if (err)
309 return err;
310
311 spin_lock(&dm->lock);
312 bitmap_clear(block_map,
313 start_idx, num_blocks);
314 spin_unlock(&dm->lock);
315
316 return 0;
317}
318
319int mlx5_cmd_query_ext_ppcnt_counters(struct mlx5_core_dev *dev, void *out)189int mlx5_cmd_query_ext_ppcnt_counters(struct mlx5_core_dev *dev, void *out)
320{190{
321 u32 in[MLX5_ST_SZ_DW(ppcnt_reg)] = {};191 u32 in[MLX5_ST_SZ_DW(ppcnt_reg)] = {};
drivers/infiniband/hw/mlx5/cmd.h+0−4
@@ -65,8 +65,4 @@ int mlx5_cmd_alloc_q_counter(struct mlx5_core_dev *dev, u16 *counter_id, · 64 unmodified lines
65 u16 uid);65 u16 uid);
66int mlx5_cmd_mad_ifc(struct mlx5_core_dev *dev, const void *inb, void *outb,66int mlx5_cmd_mad_ifc(struct mlx5_core_dev *dev, const void *inb, void *outb,
67 u16 opmod, u8 port);67 u16 opmod, u8 port);
68int mlx5_cmd_alloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
69 u16 uid, phys_addr_t *addr, u32 *obj_id);
70int mlx5_cmd_dealloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
71 u16 uid, phys_addr_t addr, u32 obj_id);
72#endif /* MLX5_IB_CMD_H */68#endif /* MLX5_IB_CMD_H */
drivers/infiniband/hw/mlx5/flow.c+12−9
@@ -322,11 +322,11 @@ void mlx5_ib_destroy_flow_action_raw(struct mlx5_ib_flow_action *maction) · 321 unmodified lines
322 switch (maction->flow_action_raw.sub_type) {322 switch (maction->flow_action_raw.sub_type) {
323 case MLX5_IB_FLOW_ACTION_MODIFY_HEADER:323 case MLX5_IB_FLOW_ACTION_MODIFY_HEADER:
324 mlx5_modify_header_dealloc(maction->flow_action_raw.dev->mdev,324 mlx5_modify_header_dealloc(maction->flow_action_raw.dev->mdev,
325 maction->flow_action_raw.action_id);325 maction->flow_action_raw.modify_hdr);
326 break;326 break;
327 case MLX5_IB_FLOW_ACTION_PACKET_REFORMAT:327 case MLX5_IB_FLOW_ACTION_PACKET_REFORMAT:
328 mlx5_packet_reformat_dealloc(maction->flow_action_raw.dev->mdev,328 mlx5_packet_reformat_dealloc(maction->flow_action_raw.dev->mdev,
329 maction->flow_action_raw.action_id);329 maction->flow_action_raw.pkt_reformat);
330 break;330 break;
331 case MLX5_IB_FLOW_ACTION_DECAP:331 case MLX5_IB_FLOW_ACTION_DECAP:
332 break;332 break;
@@ -352,10 +352,11 @@ mlx5_ib_create_modify_header(struct mlx5_ib_dev *dev, · 19 unmodified lines
352 if (!maction)352 if (!maction)
353 return ERR_PTR(-ENOMEM);353 return ERR_PTR(-ENOMEM);
354 354
355 ret = mlx5_modify_header_alloc(dev->mdev, namespace, num_actions, in,355 maction->flow_action_raw.modify_hdr =
356 &maction->flow_action_raw.action_id);356 mlx5_modify_header_alloc(dev->mdev, namespace, num_actions, in);
357 357
358 if (ret) {358 if (IS_ERR(maction->flow_action_raw.modify_hdr)) {
359 ret = PTR_ERR(maction->flow_action_raw.modify_hdr);
359 kfree(maction);360 kfree(maction);
360 return ERR_PTR(ret);361 return ERR_PTR(ret);
361 }362 }
@@ -479,11 +480,13 @@ static int mlx5_ib_flow_action_create_packet_reformat_ctx( · 117 unmodified lines
479 if (ret)480 if (ret)
480 return ret;481 return ret;
481 482
482 ret = mlx5_packet_reformat_alloc(dev->mdev, prm_prt, len,483 maction->flow_action_raw.pkt_reformat =
483 in, namespace,484 mlx5_packet_reformat_alloc(dev->mdev, prm_prt, len,
484 &maction->flow_action_raw.action_id);485 in, namespace);
485 if (ret)486 if (IS_ERR(maction->flow_action_raw.pkt_reformat)) {
487 ret = PTR_ERR(maction->flow_action_raw.pkt_reformat);
486 return ret;488 return ret;
489 }
487 490
488 maction->flow_action_raw.sub_type =491 maction->flow_action_raw.sub_type =
489 MLX5_IB_FLOW_ACTION_PACKET_REFORMAT;492 MLX5_IB_FLOW_ACTION_PACKET_REFORMAT;
drivers/infiniband/hw/mlx5/main.c+34−75
@@ -2280,6 +2280,7 @@ static inline int check_dm_type_support(struct mlx5_ib_dev *dev, · 2279 unmodified lines
2280 return -EOPNOTSUPP;2280 return -EOPNOTSUPP;
2281 break;2281 break;
2282 case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:2282 case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
2283 case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
2283 if (!capable(CAP_SYS_RAWIO) ||2284 if (!capable(CAP_SYS_RAWIO) ||
2284 !capable(CAP_NET_RAW))2285 !capable(CAP_NET_RAW))
2285 return -EPERM;2286 return -EPERM;
@@ -2344,20 +2345,20 @@ static int handle_alloc_dm_sw_icm(struct ib_ucontext *ctx, · 58 unmodified lines
2344 struct uverbs_attr_bundle *attrs,2345 struct uverbs_attr_bundle *attrs,
2345 int type)2346 int type)
2346{2347{
2347 struct mlx5_dm *dm_db = &to_mdev(ctx->device)->dm;2348 struct mlx5_core_dev *dev = to_mdev(ctx->device)->mdev;
2348 u64 act_size;2349 u64 act_size;
2349 int err;2350 int err;
2350 2351
2351 /* Allocation size must a multiple of the basic block size2352 /* Allocation size must a multiple of the basic block size
2352 * and a power of 2.2353 * and a power of 2.
2353 */2354 */
2354 act_size = round_up(attr->length, MLX5_SW_ICM_BLOCK_SIZE(dm_db->dev));2355 act_size = round_up(attr->length, MLX5_SW_ICM_BLOCK_SIZE(dev));
2355 act_size = roundup_pow_of_two(act_size);2356 act_size = roundup_pow_of_two(act_size);
2356 2357
2357 dm->size = act_size;2358 dm->size = act_size;
2358 err = mlx5_cmd_alloc_sw_icm(dm_db, type, act_size,2359 err = mlx5_dm_sw_icm_alloc(dev, type, act_size,
2359 to_mucontext(ctx)->devx_uid, &dm->dev_addr,2360 to_mucontext(ctx)->devx_uid, &dm->dev_addr,
2360 &dm->icm_dm.obj_id);2361 &dm->icm_dm.obj_id);
2361 if (err)2362 if (err)
2362 return err;2363 return err;
2363 2364
@@ -2365,9 +2366,9 @@ static int handle_alloc_dm_sw_icm(struct ib_ucontext *ctx, · 1 unmodified line
2365 MLX5_IB_ATTR_ALLOC_DM_RESP_START_OFFSET,2366 MLX5_IB_ATTR_ALLOC_DM_RESP_START_OFFSET,
2366 &dm->dev_addr, sizeof(dm->dev_addr));2367 &dm->dev_addr, sizeof(dm->dev_addr));
2367 if (err)2368 if (err)
2368 mlx5_cmd_dealloc_sw_icm(dm_db, type, dm->size,2369 mlx5_dm_sw_icm_dealloc(dev, type, dm->size,
2369 to_mucontext(ctx)->devx_uid,2370 to_mucontext(ctx)->devx_uid, dm->dev_addr,
2370 dm->dev_addr, dm->icm_dm.obj_id);2371 dm->icm_dm.obj_id);
2371 2372
2372 return err;2373 return err;
2373}2374}
@@ -2407,8 +2408,14 @@ struct ib_dm *mlx5_ib_alloc_dm(struct ib_device *ibdev, · 33 unmodified lines
2407 attrs);2408 attrs);
2408 break;2409 break;
2409 case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:2410 case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
2411 err = handle_alloc_dm_sw_icm(context, dm,
2412 attr, attrs,
2413 MLX5_SW_ICM_TYPE_STEERING);
2414 break;
2410 case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:2415 case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
2411 err = handle_alloc_dm_sw_icm(context, dm, attr, attrs, type);2416 err = handle_alloc_dm_sw_icm(context, dm,
2417 attr, attrs,
2418 MLX5_SW_ICM_TYPE_HEADER_MODIFY);
2412 break;2419 break;
2413 default:2420 default:
2414 err = -EOPNOTSUPP;2421 err = -EOPNOTSUPP;
@@ -2428,6 +2435,7 @@ int mlx5_ib_dealloc_dm(struct ib_dm *ibdm, struct uverbs_attr_bundle *attrs) · 13 unmodified lines
2428{2435{
2429 struct mlx5_ib_ucontext *ctx = rdma_udata_to_drv_context(2436 struct mlx5_ib_ucontext *ctx = rdma_udata_to_drv_context(
2430 &attrs->driver_udata, struct mlx5_ib_ucontext, ibucontext);2437 &attrs->driver_udata, struct mlx5_ib_ucontext, ibucontext);
2438 struct mlx5_core_dev *dev = to_mdev(ibdm->device)->mdev;
2431 struct mlx5_dm *dm_db = &to_mdev(ibdm->device)->dm;2439 struct mlx5_dm *dm_db = &to_mdev(ibdm->device)->dm;
2432 struct mlx5_ib_dm *dm = to_mdm(ibdm);2440 struct mlx5_ib_dm *dm = to_mdm(ibdm);
2433 u32 page_idx;2441 u32 page_idx;
@@ -2439,19 +2447,23 @@ int mlx5_ib_dealloc_dm(struct ib_dm *ibdm, struct uverbs_attr_bundle *attrs) · 5 unmodified lines
2439 if (ret)2447 if (ret)
2440 return ret;2448 return ret;
2441 2449
2442 page_idx = (dm->dev_addr -2450 page_idx = (dm->dev_addr - pci_resource_start(dev->pdev, 0) -
2443 pci_resource_start(dm_db->dev->pdev, 0) -2451 MLX5_CAP64_DEV_MEM(dev, memic_bar_start_addr)) >>
2444 MLX5_CAP64_DEV_MEM(dm_db->dev,2452 PAGE_SHIFT;
2445 memic_bar_start_addr)) >>
2446 PAGE_SHIFT;
2447 bitmap_clear(ctx->dm_pages, page_idx,2453 bitmap_clear(ctx->dm_pages, page_idx,
2448 DIV_ROUND_UP(dm->size, PAGE_SIZE));2454 DIV_ROUND_UP(dm->size, PAGE_SIZE));
2449 break;2455 break;
2450 case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:2456 case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
2457 ret = mlx5_dm_sw_icm_dealloc(dev, MLX5_SW_ICM_TYPE_STEERING,
2458 dm->size, ctx->devx_uid, dm->dev_addr,
2459 dm->icm_dm.obj_id);
2460 if (ret)
2461 return ret;
2462 break;
2451 case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:2463 case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
2452 ret = mlx5_cmd_dealloc_sw_icm(dm_db, dm->type, dm->size,2464 ret = mlx5_dm_sw_icm_dealloc(dev, MLX5_SW_ICM_TYPE_HEADER_MODIFY,
2453 ctx->devx_uid, dm->dev_addr,2465 dm->size, ctx->devx_uid, dm->dev_addr,
2454 dm->icm_dm.obj_id);2466 dm->icm_dm.obj_id);
2455 if (ret)2467 if (ret)
2456 return ret;2468 return ret;
2457 break;2469 break;
@@ -2646,7 +2658,8 @@ int parse_flow_flow_action(struct mlx5_ib_flow_action *maction, · 188 unmodified lines
2646 if (action->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)2658 if (action->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)
2647 return -EINVAL;2659 return -EINVAL;
2648 action->action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR;2660 action->action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR;
2649 action->modify_id = maction->flow_action_raw.action_id;2661 action->modify_hdr =
2662 maction->flow_action_raw.modify_hdr;
2650 return 0;2663 return 0;
2651 }2664 }
2652 if (maction->flow_action_raw.sub_type ==2665 if (maction->flow_action_raw.sub_type ==
@@ -2663,8 +2676,8 @@ int parse_flow_flow_action(struct mlx5_ib_flow_action *maction, · 10 unmodified lines
2663 return -EINVAL;2676 return -EINVAL;
2664 action->action |=2677 action->action |=
2665 MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;2678 MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;
2666 action->reformat_id =2679 action->pkt_reformat =
2667 maction->flow_action_raw.action_id;2680 maction->flow_action_raw.pkt_reformat;
2668 return 0;2681 return 0;
2669 }2682 }
2670 /* fall through */2683 /* fall through */
@@ -6096,8 +6109,6 @@ static struct ib_counters *mlx5_ib_create_counters(struct ib_device *device, · 3425 unmodified lines
6096 6109
6097static void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev)6110static void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev)
6098{6111{
6099 struct mlx5_core_dev *mdev = dev->mdev;
6100
6101 mlx5_ib_cleanup_multiport_master(dev);6112 mlx5_ib_cleanup_multiport_master(dev);
6102 if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {6113 if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
6103 srcu_barrier(&dev->mr_srcu);6114 srcu_barrier(&dev->mr_srcu);
@@ -6105,29 +6116,11 @@ static void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev) · 1 unmodified line
6105 }6116 }
6106 6117
6107 WARN_ON(!bitmap_empty(dev->dm.memic_alloc_pages, MLX5_MAX_MEMIC_PAGES));6118 WARN_ON(!bitmap_empty(dev->dm.memic_alloc_pages, MLX5_MAX_MEMIC_PAGES));
6108
6109 WARN_ON(dev->dm.steering_sw_icm_alloc_blocks &&
6110 !bitmap_empty(
6111 dev->dm.steering_sw_icm_alloc_blocks,
6112 BIT(MLX5_CAP_DEV_MEM(mdev, log_steering_sw_icm_size) -
6113 MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev))));
6114
6115 kfree(dev->dm.steering_sw_icm_alloc_blocks);
6116
6117 WARN_ON(dev->dm.header_modify_sw_icm_alloc_blocks &&
6118 !bitmap_empty(dev->dm.header_modify_sw_icm_alloc_blocks,
6119 BIT(MLX5_CAP_DEV_MEM(
6120 mdev, log_header_modify_sw_icm_size) -
6121 MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev))));
6122
6123 kfree(dev->dm.header_modify_sw_icm_alloc_blocks);
6124}6119}
6125 6120
6126static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)6121static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
6127{6122{
6128 struct mlx5_core_dev *mdev = dev->mdev;6123 struct mlx5_core_dev *mdev = dev->mdev;
6129 u64 header_modify_icm_blocks = 0;
6130 u64 steering_icm_blocks = 0;
6131 int err;6124 int err;
6132 int i;6125 int i;
6133 6126
@@ -6174,51 +6167,17 @@ static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev) · 40 unmodified lines
6174 INIT_LIST_HEAD(&dev->qp_list);6167 INIT_LIST_HEAD(&dev->qp_list);
6175 spin_lock_init(&dev->reset_flow_resource_lock);6168 spin_lock_init(&dev->reset_flow_resource_lock);
6176 6169
6177 if (MLX5_CAP_GEN_64(mdev, general_obj_types) &
6178 MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM) {
6179 if (MLX5_CAP64_DEV_MEM(mdev, steering_sw_icm_start_address)) {
6180 steering_icm_blocks =
6181 BIT(MLX5_CAP_DEV_MEM(mdev,
6182 log_steering_sw_icm_size) -
6183 MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev));
6184
6185 dev->dm.steering_sw_icm_alloc_blocks =
6186 kcalloc(BITS_TO_LONGS(steering_icm_blocks),
6187 sizeof(unsigned long), GFP_KERNEL);
6188 if (!dev->dm.steering_sw_icm_alloc_blocks)
6189 goto err_mp;
6190 }
6191
6192 if (MLX5_CAP64_DEV_MEM(mdev,
6193 header_modify_sw_icm_start_address)) {
6194 header_modify_icm_blocks = BIT(
6195 MLX5_CAP_DEV_MEM(
6196 mdev, log_header_modify_sw_icm_size) -
6197 MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev));
6198
6199 dev->dm.header_modify_sw_icm_alloc_blocks =
6200 kcalloc(BITS_TO_LONGS(header_modify_icm_blocks),
6201 sizeof(unsigned long), GFP_KERNEL);
6202 if (!dev->dm.header_modify_sw_icm_alloc_blocks)
6203 goto err_dm;
6204 }
6205 }
6206
6207 spin_lock_init(&dev->dm.lock);6170 spin_lock_init(&dev->dm.lock);
6208 dev->dm.dev = mdev;6171 dev->dm.dev = mdev;
6209 6172
6210 if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {6173 if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
6211 err = init_srcu_struct(&dev->mr_srcu);6174 err = init_srcu_struct(&dev->mr_srcu);
6212 if (err)6175 if (err)
6213 goto err_dm;6176 goto err_mp;
6214 }6177 }
6215 6178
6216 return 0;6179 return 0;
6217 6180
6218err_dm:
6219 kfree(dev->dm.steering_sw_icm_alloc_blocks);
6220 kfree(dev->dm.header_modify_sw_icm_alloc_blocks);
6221
6222err_mp:6181err_mp:
6223 mlx5_ib_cleanup_multiport_master(dev);6182 mlx5_ib_cleanup_multiport_master(dev);
6224 6183
drivers/infiniband/hw/mlx5/mlx5_ib.h+4−3
@@ -868,7 +868,10 @@ struct mlx5_ib_flow_action { · 867 unmodified lines
868 struct {868 struct {
869 struct mlx5_ib_dev *dev;869 struct mlx5_ib_dev *dev;
870 u32 sub_type;870 u32 sub_type;
871 u32 action_id;871 union {
872 struct mlx5_modify_hdr *modify_hdr;
873 struct mlx5_pkt_reformat *pkt_reformat;
874 };
872 } flow_action_raw;875 } flow_action_raw;
873 };876 };
874};877};
@@ -881,8 +884,6 @@ struct mlx5_dm { · 6 unmodified lines
881 */884 */
882 spinlock_t lock;885 spinlock_t lock;
883 DECLARE_BITMAP(memic_alloc_pages, MLX5_MAX_MEMIC_PAGES);886 DECLARE_BITMAP(memic_alloc_pages, MLX5_MAX_MEMIC_PAGES);
884 unsigned long *steering_sw_icm_alloc_blocks;
885 unsigned long *header_modify_sw_icm_alloc_blocks;
886};887};
887 888
888struct mlx5_read_counters_attr {889struct mlx5_read_counters_attr {
drivers/net/ethernet/mellanox/mlx5/core/Kconfig+7−0
@@ -154,3 +154,10 @@ config MLX5_EN_TLS · 153 unmodified lines
154 Build support for TLS cryptography-offload accelaration in the NIC.154 Build support for TLS cryptography-offload accelaration in the NIC.
155 Note: Support for hardware with this capability needs to be selected155 Note: Support for hardware with this capability needs to be selected
156 for this option to become available.156 for this option to become available.
157
158config MLX5_SW_STEERING
159 bool "Mellanox Technologies software-managed steering"
160 depends on MLX5_CORE_EN && MLX5_ESWITCH
161 default y
162 help
163 Build support for software-managed steering in the NIC.
drivers/net/ethernet/mellanox/mlx5/core/Makefile+8−1
@@ -15,7 +15,7 @@ mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \ · 14 unmodified lines
15 health.o mcg.o cq.o alloc.o qp.o port.o mr.o pd.o \15 health.o mcg.o cq.o alloc.o qp.o port.o mr.o pd.o \
16 transobj.o vport.o sriov.o fs_cmd.o fs_core.o pci_irq.o \16 transobj.o vport.o sriov.o fs_cmd.o fs_core.o pci_irq.o \
17 fs_counters.o rl.o lag.o dev.o events.o wq.o lib/gid.o \17 fs_counters.o rl.o lag.o dev.o events.o wq.o lib/gid.o \
18 lib/devcom.o lib/pci_vsc.o diag/fs_tracepoint.o \18 lib/devcom.o lib/pci_vsc.o lib/dm.o diag/fs_tracepoint.o \
19 diag/fw_tracer.o diag/crdump.o devlink.o19 diag/fw_tracer.o diag/crdump.o devlink.o
20 20
21#21#
@@ -67,3 +67,10 @@ mlx5_core-$(CONFIG_MLX5_EN_IPSEC) += en_accel/ipsec.o en_accel/ipsec_rxtx.o \ · 45 unmodified lines
67 67
68mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/tls.o en_accel/tls_rxtx.o en_accel/tls_stats.o \68mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/tls.o en_accel/tls_rxtx.o en_accel/tls_stats.o \
69 en_accel/ktls.o en_accel/ktls_tx.o69 en_accel/ktls.o en_accel/ktls_tx.o
70
71mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \
72 steering/dr_matcher.o steering/dr_rule.o \
73 steering/dr_icm_pool.o steering/dr_crc32.o \
74 steering/dr_ste.o steering/dr_send.o \
75 steering/dr_cmd.o steering/dr_fw.o \
76 steering/dr_action.o steering/fs_dr.o
drivers/net/ethernet/mellanox/mlx5/core/devlink.c+111−1
@@ -4,6 +4,7 @@ · 3 unmodified lines
4#include <devlink.h>4#include <devlink.h>
5 5
6#include "mlx5_core.h"6#include "mlx5_core.h"
7#include "fs_core.h"
7#include "eswitch.h"8#include "eswitch.h"
8 9
9static int mlx5_devlink_flash_update(struct devlink *devlink,10static int mlx5_devlink_flash_update(struct devlink *devlink,
@@ -107,12 +108,121 @@ void mlx5_devlink_free(struct devlink *devlink) · 97 unmodified lines
107 devlink_free(devlink);108 devlink_free(devlink);
108}109}
109 110
111static int mlx5_devlink_fs_mode_validate(struct devlink *devlink, u32 id,
112 union devlink_param_value val,
113 struct netlink_ext_ack *extack)
114{
115 struct mlx5_core_dev *dev = devlink_priv(devlink);
116 char *value = val.vstr;
117 int err = 0;
118
119 if (!strcmp(value, "dmfs")) {
120 return 0;
121 } else if (!strcmp(value, "smfs")) {
122 u8 eswitch_mode;
123 bool smfs_cap;
124
125 eswitch_mode = mlx5_eswitch_mode(dev->priv.eswitch);
126 smfs_cap = mlx5_fs_dr_is_supported(dev);
127
128 if (!smfs_cap) {
129 err = -EOPNOTSUPP;
130 NL_SET_ERR_MSG_MOD(extack,
131 "Software managed steering is not supported by current device");
132 }
133
134 else if (eswitch_mode == MLX5_ESWITCH_OFFLOADS) {
135 NL_SET_ERR_MSG_MOD(extack,
136 "Software managed steering is not supported when eswitch offlaods enabled.");
137 err = -EOPNOTSUPP;
138 }
139 } else {
140 NL_SET_ERR_MSG_MOD(extack,
141 "Bad parameter: supported values are [\"dmfs\", \"smfs\"]");
142 err = -EINVAL;
143 }
144
145 return err;
146}
147
148static int mlx5_devlink_fs_mode_set(struct devlink *devlink, u32 id,
149 struct devlink_param_gset_ctx *ctx)
150{
151 struct mlx5_core_dev *dev = devlink_priv(devlink);
152 enum mlx5_flow_steering_mode mode;
153
154 if (!strcmp(ctx->val.vstr, "smfs"))
155 mode = MLX5_FLOW_STEERING_MODE_SMFS;
156 else
157 mode = MLX5_FLOW_STEERING_MODE_DMFS;
158 dev->priv.steering->mode = mode;
159
160 return 0;
161}
162
163static int mlx5_devlink_fs_mode_get(struct devlink *devlink, u32 id,
164 struct devlink_param_gset_ctx *ctx)
165{
166 struct mlx5_core_dev *dev = devlink_priv(devlink);
167
168 if (dev->priv.steering->mode == MLX5_FLOW_STEERING_MODE_SMFS)
169 strcpy(ctx->val.vstr, "smfs");
170 else
171 strcpy(ctx->val.vstr, "dmfs");
172 return 0;
173}
174
175enum mlx5_devlink_param_id {
176 MLX5_DEVLINK_PARAM_ID_BASE = DEVLINK_PARAM_GENERIC_ID_MAX,
177 MLX5_DEVLINK_PARAM_FLOW_STEERING_MODE,
178};
179
180static const struct devlink_param mlx5_devlink_params[] = {
181 DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_FLOW_STEERING_MODE,
182 "flow_steering_mode", DEVLINK_PARAM_TYPE_STRING,
183 BIT(DEVLINK_PARAM_CMODE_RUNTIME),
184 mlx5_devlink_fs_mode_get, mlx5_devlink_fs_mode_set,
185 mlx5_devlink_fs_mode_validate),
186};
187
188static void mlx5_devlink_set_params_init_values(struct devlink *devlink)
189{
190 struct mlx5_core_dev *dev = devlink_priv(devlink);
191 union devlink_param_value value;
192
193 if (dev->priv.steering->mode == MLX5_FLOW_STEERING_MODE_DMFS)
194 strcpy(value.vstr, "dmfs");
195 else
196 strcpy(value.vstr, "smfs");
197 devlink_param_driverinit_value_set(devlink,
198 MLX5_DEVLINK_PARAM_FLOW_STEERING_MODE,
199 value);
200}
201
110int mlx5_devlink_register(struct devlink *devlink, struct device *dev)202int mlx5_devlink_register(struct devlink *devlink, struct device *dev)
111{203{
112 return devlink_register(devlink, dev);204 int err;
205
206 err = devlink_register(devlink, dev);
207 if (err)
208 return err;
209
210 err = devlink_params_register(devlink, mlx5_devlink_params,
211 ARRAY_SIZE(mlx5_devlink_params));
212 if (err)
213 goto params_reg_err;
214 mlx5_devlink_set_params_init_values(devlink);
215 devlink_params_publish(devlink);
216 return 0;
217
218params_reg_err:
219 devlink_unregister(devlink);
220 return err;
113}221}
114 222
115void mlx5_devlink_unregister(struct devlink *devlink)223void mlx5_devlink_unregister(struct devlink *devlink)
116{224{
225 devlink_params_unregister(devlink, mlx5_devlink_params,
226 ARRAY_SIZE(mlx5_devlink_params));
117 devlink_unregister(devlink);227 devlink_unregister(devlink);
118}228}
drivers/net/ethernet/mellanox/mlx5/core/en/tc_tun.c+14−13
@@ -291,14 +291,14 @@ int mlx5e_tc_tun_create_header_ipv4(struct mlx5e_priv *priv, · 290 unmodified lines
291 */291 */
292 goto out;292 goto out;
293 }293 }
294 294 e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev,
295 err = mlx5_packet_reformat_alloc(priv->mdev,295 e->reformat_type,
296 e->reformat_type,296 ipv4_encap_size, encap_header,
297 ipv4_encap_size, encap_header,297 MLX5_FLOW_NAMESPACE_FDB);
298 MLX5_FLOW_NAMESPACE_FDB,298 if (IS_ERR(e->pkt_reformat)) {
299 &e->encap_id);299 err = PTR_ERR(e->pkt_reformat);
300 if (err)
301 goto destroy_neigh_entry;300 goto destroy_neigh_entry;
301 }
302 302
303 e->flags |= MLX5_ENCAP_ENTRY_VALID;303 e->flags |= MLX5_ENCAP_ENTRY_VALID;
304 mlx5e_rep_queue_neigh_stats_work(netdev_priv(out_dev));304 mlx5e_rep_queue_neigh_stats_work(netdev_priv(out_dev));
@@ -407,13 +407,14 @@ int mlx5e_tc_tun_create_header_ipv6(struct mlx5e_priv *priv, · 102 unmodified lines
407 goto out;407 goto out;
408 }408 }
409 409
410 err = mlx5_packet_reformat_alloc(priv->mdev,410 e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev,
411 e->reformat_type,411 e->reformat_type,
412 ipv6_encap_size, encap_header,412 ipv6_encap_size, encap_header,
413 MLX5_FLOW_NAMESPACE_FDB,413 MLX5_FLOW_NAMESPACE_FDB);
414 &e->encap_id);414 if (IS_ERR(e->pkt_reformat)) {
415 if (err)415 err = PTR_ERR(e->pkt_reformat);
416 goto destroy_neigh_entry;416 goto destroy_neigh_entry;
417 }
417 418
418 e->flags |= MLX5_ENCAP_ENTRY_VALID;419 e->flags |= MLX5_ENCAP_ENTRY_VALID;
419 mlx5e_rep_queue_neigh_stats_work(netdev_priv(out_dev));420 mlx5e_rep_queue_neigh_stats_work(netdev_priv(out_dev));
drivers/net/ethernet/mellanox/mlx5/core/en_rep.h+1−1
@@ -161,7 +161,7 @@ struct mlx5e_encap_entry { · 160 unmodified lines
161 */161 */
162 struct hlist_node encap_hlist;162 struct hlist_node encap_hlist;
163 struct list_head flows;163 struct list_head flows;
164 u32 encap_id;164 struct mlx5_pkt_reformat *pkt_reformat;
165 const struct ip_tunnel_info *tun_info;165 const struct ip_tunnel_info *tun_info;
166 unsigned char h_dest[ETH_ALEN]; /* destination eth addr */166 unsigned char h_dest[ETH_ALEN]; /* destination eth addr */
167 167
drivers/net/ethernet/mellanox/mlx5/core/en_tc.c+26−25
@@ -61,7 +61,7 @@ · 60 unmodified lines
61struct mlx5_nic_flow_attr {61struct mlx5_nic_flow_attr {
62 u32 action;62 u32 action;
63 u32 flow_tag;63 u32 flow_tag;
64 u32 mod_hdr_id;64 struct mlx5_modify_hdr *modify_hdr;
65 u32 hairpin_tirn;65 u32 hairpin_tirn;
66 u8 match_level;66 u8 match_level;
67 struct mlx5_flow_table *hairpin_ft;67 struct mlx5_flow_table *hairpin_ft;
@@ -201,7 +201,7 @@ struct mlx5e_mod_hdr_entry { · 133 unmodified lines
201 201
202 struct mod_hdr_key key;202 struct mod_hdr_key key;
203 203
204 u32 mod_hdr_id;204 struct mlx5_modify_hdr *modify_hdr;
205 205
206 refcount_t refcnt;206 refcount_t refcnt;
207 struct completion res_ready;207 struct completion res_ready;
@@ -334,7 +334,7 @@ static void mlx5e_mod_hdr_put(struct mlx5e_priv *priv, · 126 unmodified lines
334 334
335 WARN_ON(!list_empty(&mh->flows));335 WARN_ON(!list_empty(&mh->flows));
336 if (mh->compl_result > 0)336 if (mh->compl_result > 0)
337 mlx5_modify_header_dealloc(priv->mdev, mh->mod_hdr_id);337 mlx5_modify_header_dealloc(priv->mdev, mh->modify_hdr);
338 338
339 kfree(mh);339 kfree(mh);
340}340}
@@ -395,11 +395,11 @@ static int mlx5e_attach_mod_hdr(struct mlx5e_priv *priv, · 54 unmodified lines
395 hash_add(tbl->hlist, &mh->mod_hdr_hlist, hash_key);395 hash_add(tbl->hlist, &mh->mod_hdr_hlist, hash_key);
396 mutex_unlock(&tbl->lock);396 mutex_unlock(&tbl->lock);
397 397
398 err = mlx5_modify_header_alloc(priv->mdev, namespace,398 mh->modify_hdr = mlx5_modify_header_alloc(priv->mdev, namespace,
399 mh->key.num_actions,399 mh->key.num_actions,
400 mh->key.actions,400 mh->key.actions);
401 &mh->mod_hdr_id);401 if (IS_ERR(mh->modify_hdr)) {
402 if (err) {402 err = PTR_ERR(mh->modify_hdr);
403 mh->compl_result = err;403 mh->compl_result = err;
404 goto alloc_header_err;404 goto alloc_header_err;
405 }405 }
@@ -412,9 +412,9 @@ attach_flow: · 6 unmodified lines
412 list_add(&flow->mod_hdr, &mh->flows);412 list_add(&flow->mod_hdr, &mh->flows);
413 spin_unlock(&mh->flows_lock);413 spin_unlock(&mh->flows_lock);
414 if (mlx5e_is_eswitch_flow(flow))414 if (mlx5e_is_eswitch_flow(flow))
415 flow->esw_attr->mod_hdr_id = mh->mod_hdr_id;415 flow->esw_attr->modify_hdr = mh->modify_hdr;
416 else416 else
417 flow->nic_attr->mod_hdr_id = mh->mod_hdr_id;417 flow->nic_attr->modify_hdr = mh->modify_hdr;
418 418
419 return 0;419 return 0;
420 420
@@ -906,7 +906,6 @@ mlx5e_tc_add_nic_flow(struct mlx5e_priv *priv, · 485 unmodified lines
906 struct mlx5_flow_destination dest[2] = {};906 struct mlx5_flow_destination dest[2] = {};
907 struct mlx5_flow_act flow_act = {907 struct mlx5_flow_act flow_act = {
908 .action = attr->action,908 .action = attr->action,
909 .reformat_id = 0,
910 .flags = FLOW_ACT_NO_APPEND,909 .flags = FLOW_ACT_NO_APPEND,
911 };910 };
912 struct mlx5_fc *counter = NULL;911 struct mlx5_fc *counter = NULL;
@@ -947,7 +946,7 @@ mlx5e_tc_add_nic_flow(struct mlx5e_priv *priv, · 34 unmodified lines
947 946
948 if (attr->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR) {947 if (attr->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR) {
949 err = mlx5e_attach_mod_hdr(priv, flow, parse_attr);948 err = mlx5e_attach_mod_hdr(priv, flow, parse_attr);
950 flow_act.modify_id = attr->mod_hdr_id;949 flow_act.modify_hdr = attr->modify_hdr;
951 kfree(parse_attr->mod_hdr_actions);950 kfree(parse_attr->mod_hdr_actions);
952 if (err)951 if (err)
953 return err;952 return err;
@@ -1304,14 +1303,13 @@ void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv, · 350 unmodified lines
1304 struct mlx5e_tc_flow *flow;1303 struct mlx5e_tc_flow *flow;
1305 int err;1304 int err;
1306 1305
1307 err = mlx5_packet_reformat_alloc(priv->mdev,1306 e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev,
1308 e->reformat_type,1307 e->reformat_type,
1309 e->encap_size, e->encap_header,1308 e->encap_size, e->encap_header,
1310 MLX5_FLOW_NAMESPACE_FDB,1309 MLX5_FLOW_NAMESPACE_FDB);
1311 &e->encap_id);1310 if (IS_ERR(e->pkt_reformat)) {
1312 if (err) {1311 mlx5_core_warn(priv->mdev, "Failed to offload cached encapsulation header, %lu\n",
1313 mlx5_core_warn(priv->mdev, "Failed to offload cached encapsulation header, %d\n",1312 PTR_ERR(e->pkt_reformat));
1314 err);
1315 return;1313 return;
1316 }1314 }
1317 e->flags |= MLX5_ENCAP_ENTRY_VALID;1315 e->flags |= MLX5_ENCAP_ENTRY_VALID;
@@ -1326,7 +1324,7 @@ void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv, · 8 unmodified lines
1326 esw_attr = flow->esw_attr;1324 esw_attr = flow->esw_attr;
1327 spec = &esw_attr->parse_attr->spec;1325 spec = &esw_attr->parse_attr->spec;
1328 1326
1329 esw_attr->dests[flow->tmp_efi_index].encap_id = e->encap_id;1327 esw_attr->dests[flow->tmp_efi_index].pkt_reformat = e->pkt_reformat;
1330 esw_attr->dests[flow->tmp_efi_index].flags |= MLX5_ESW_DEST_ENCAP_VALID;1328 esw_attr->dests[flow->tmp_efi_index].flags |= MLX5_ESW_DEST_ENCAP_VALID;
1331 /* Flow can be associated with multiple encap entries.1329 /* Flow can be associated with multiple encap entries.
1332 * Before offloading the flow verify that all of them have1330 * Before offloading the flow verify that all of them have
@@ -1395,7 +1393,7 @@ void mlx5e_tc_encap_flows_del(struct mlx5e_priv *priv, · 62 unmodified lines
1395 1393
1396 /* we know that the encap is valid */1394 /* we know that the encap is valid */
1397 e->flags &= ~MLX5_ENCAP_ENTRY_VALID;1395 e->flags &= ~MLX5_ENCAP_ENTRY_VALID;
1398 mlx5_packet_reformat_dealloc(priv->mdev, e->encap_id);1396 mlx5_packet_reformat_dealloc(priv->mdev, e->pkt_reformat);
1399}1397}
1400 1398
1401static struct mlx5_fc *mlx5e_tc_get_counter(struct mlx5e_tc_flow *flow)1399static struct mlx5_fc *mlx5e_tc_get_counter(struct mlx5e_tc_flow *flow)
@@ -1561,7 +1559,7 @@ static void mlx5e_encap_dealloc(struct mlx5e_priv *priv, struct mlx5e_encap_entr · 159 unmodified lines
1561 mlx5e_rep_encap_entry_detach(netdev_priv(e->out_dev), e);1559 mlx5e_rep_encap_entry_detach(netdev_priv(e->out_dev), e);
1562 1560
1563 if (e->flags & MLX5_ENCAP_ENTRY_VALID)1561 if (e->flags & MLX5_ENCAP_ENTRY_VALID)
1564 mlx5_packet_reformat_dealloc(priv->mdev, e->encap_id);1562 mlx5_packet_reformat_dealloc(priv->mdev, e->pkt_reformat);
1565 }1563 }
1566 1564
1567 kfree(e->encap_header);1565 kfree(e->encap_header);
@@ -1896,7 +1894,10 @@ static int __parse_cls_flower(struct mlx5e_priv *priv, · 328 unmodified lines
1896 *match_level = MLX5_MATCH_L2;1894 *match_level = MLX5_MATCH_L2;
1897 }1895 }
1898 } else if (*match_level != MLX5_MATCH_NONE) {1896 } else if (*match_level != MLX5_MATCH_NONE) {
1899 MLX5_SET(fte_match_set_lyr_2_4, headers_c, svlan_tag, 1);1897 /* cvlan_tag enabled in match criteria and
1898 * disabled in match value means both S & C tags
1899 * don't exist (untagged of both)
1900 */
1900 MLX5_SET(fte_match_set_lyr_2_4, headers_c, cvlan_tag, 1);1901 MLX5_SET(fte_match_set_lyr_2_4, headers_c, cvlan_tag, 1);
1901 *match_level = MLX5_MATCH_L2;1902 *match_level = MLX5_MATCH_L2;
1902 }1903 }
@@ -3045,7 +3046,7 @@ attach_flow: · 1142 unmodified lines
3045 flow->encaps[out_index].index = out_index;3046 flow->encaps[out_index].index = out_index;
3046 *encap_dev = e->out_dev;3047 *encap_dev = e->out_dev;
3047 if (e->flags & MLX5_ENCAP_ENTRY_VALID) {3048 if (e->flags & MLX5_ENCAP_ENTRY_VALID) {
3048 attr->dests[out_index].encap_id = e->encap_id;3049 attr->dests[out_index].pkt_reformat = e->pkt_reformat;
3049 attr->dests[out_index].flags |= MLX5_ESW_DEST_ENCAP_VALID;3050 attr->dests[out_index].flags |= MLX5_ESW_DEST_ENCAP_VALID;
3050 *encap_valid = true;3051 *encap_valid = true;
3051 } else {3052 } else {
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h+4−3
@@ -69,7 +69,7 @@ struct vport_ingress { · 68 unmodified lines
69 struct mlx5_flow_group *allow_spoofchk_only_grp;69 struct mlx5_flow_group *allow_spoofchk_only_grp;
70 struct mlx5_flow_group *allow_untagged_only_grp;70 struct mlx5_flow_group *allow_untagged_only_grp;
71 struct mlx5_flow_group *drop_grp;71 struct mlx5_flow_group *drop_grp;
72 int modify_metadata_id;72 struct mlx5_modify_hdr *modify_metadata;
73 struct mlx5_flow_handle *modify_metadata_rule;73 struct mlx5_flow_handle *modify_metadata_rule;
74 struct mlx5_flow_handle *allow_rule;74 struct mlx5_flow_handle *allow_rule;
75 struct mlx5_flow_handle *drop_rule;75 struct mlx5_flow_handle *drop_rule;
@@ -153,6 +153,7 @@ struct mlx5_eswitch_fdb { · 77 unmodified lines
153 } legacy;153 } legacy;
154 154
155 struct offloads_fdb {155 struct offloads_fdb {
156 struct mlx5_flow_namespace *ns;
156 struct mlx5_flow_table *slow_fdb;157 struct mlx5_flow_table *slow_fdb;
157 struct mlx5_flow_group *send_to_vport_grp;158 struct mlx5_flow_group *send_to_vport_grp;
158 struct mlx5_flow_group *peer_miss_grp;159 struct mlx5_flow_group *peer_miss_grp;
@@ -385,11 +386,11 @@ struct mlx5_esw_flow_attr { · 226 unmodified lines
385 struct {386 struct {
386 u32 flags;387 u32 flags;
387 struct mlx5_eswitch_rep *rep;388 struct mlx5_eswitch_rep *rep;
389 struct mlx5_pkt_reformat *pkt_reformat;
388 struct mlx5_core_dev *mdev;390 struct mlx5_core_dev *mdev;
389 u32 encap_id;
390 struct mlx5_termtbl_handle *termtbl;391 struct mlx5_termtbl_handle *termtbl;
391 } dests[MLX5_MAX_FLOW_FWD_VPORTS];392 } dests[MLX5_MAX_FLOW_FWD_VPORTS];
392 u32 mod_hdr_id;393 struct mlx5_modify_hdr *modify_hdr;
393 u8 inner_match_level;394 u8 inner_match_level;
394 u8 outer_match_level;395 u8 outer_match_level;
395 struct mlx5_fc *counter;396 struct mlx5_fc *counter;
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c+68−19
@@ -190,10 +190,10 @@ mlx5_eswitch_add_offloaded_rule(struct mlx5_eswitch *esw, · 189 unmodified lines
190 MLX5_FLOW_DEST_VPORT_VHCA_ID;190 MLX5_FLOW_DEST_VPORT_VHCA_ID;
191 if (attr->dests[j].flags & MLX5_ESW_DEST_ENCAP) {191 if (attr->dests[j].flags & MLX5_ESW_DEST_ENCAP) {
192 flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;192 flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;
193 flow_act.reformat_id = attr->dests[j].encap_id;193 flow_act.pkt_reformat = attr->dests[j].pkt_reformat;
194 dest[i].vport.flags |= MLX5_FLOW_DEST_VPORT_REFORMAT_ID;194 dest[i].vport.flags |= MLX5_FLOW_DEST_VPORT_REFORMAT_ID;
195 dest[i].vport.reformat_id =195 dest[i].vport.pkt_reformat =
196 attr->dests[j].encap_id;196 attr->dests[j].pkt_reformat;
197 }197 }
198 i++;198 i++;
199 }199 }
@@ -213,7 +213,7 @@ mlx5_eswitch_add_offloaded_rule(struct mlx5_eswitch *esw, · 13 unmodified lines
213 spec->match_criteria_enable |= MLX5_MATCH_INNER_HEADERS;213 spec->match_criteria_enable |= MLX5_MATCH_INNER_HEADERS;
214 214
215 if (flow_act.action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)215 if (flow_act.action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)
216 flow_act.modify_id = attr->mod_hdr_id;216 flow_act.modify_hdr = attr->modify_hdr;
217 217
218 fdb = esw_get_prio_table(esw, attr->chain, attr->prio, !!split);218 fdb = esw_get_prio_table(esw, attr->chain, attr->prio, !!split);
219 if (IS_ERR(fdb)) {219 if (IS_ERR(fdb)) {
@@ -276,7 +276,7 @@ mlx5_eswitch_add_fwd_rule(struct mlx5_eswitch *esw, · 56 unmodified lines
276 dest[i].vport.flags |= MLX5_FLOW_DEST_VPORT_VHCA_ID;276 dest[i].vport.flags |= MLX5_FLOW_DEST_VPORT_VHCA_ID;
277 if (attr->dests[i].flags & MLX5_ESW_DEST_ENCAP) {277 if (attr->dests[i].flags & MLX5_ESW_DEST_ENCAP) {
278 dest[i].vport.flags |= MLX5_FLOW_DEST_VPORT_REFORMAT_ID;278 dest[i].vport.flags |= MLX5_FLOW_DEST_VPORT_REFORMAT_ID;
279 dest[i].vport.reformat_id = attr->dests[i].encap_id;279 dest[i].vport.pkt_reformat = attr->dests[i].pkt_reformat;
280 }280 }
281 }281 }
282 dest[i].type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;282 dest[i].type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
@@ -1068,6 +1068,13 @@ static int esw_create_offloads_fdb_tables(struct mlx5_eswitch *esw, int nvports) · 785 unmodified lines
1068 err = -EOPNOTSUPP;1068 err = -EOPNOTSUPP;
1069 goto ns_err;1069 goto ns_err;
1070 }1070 }
1071 esw->fdb_table.offloads.ns = root_ns;
1072 err = mlx5_flow_namespace_set_mode(root_ns,
1073 esw->dev->priv.steering->mode);
1074 if (err) {
1075 esw_warn(dev, "Failed to set FDB namespace steering mode\n");
1076 goto ns_err;
1077 }
1071 1078
1072 max_flow_counter = (MLX5_CAP_GEN(dev, max_flow_counter_31_16) << 16) |1079 max_flow_counter = (MLX5_CAP_GEN(dev, max_flow_counter_31_16) << 16) |
1073 MLX5_CAP_GEN(dev, max_flow_counter_15_0);1080 MLX5_CAP_GEN(dev, max_flow_counter_15_0);
@@ -1207,6 +1214,8 @@ send_vport_err: · 133 unmodified lines
1207 esw_destroy_offloads_fast_fdb_tables(esw);1214 esw_destroy_offloads_fast_fdb_tables(esw);
1208 mlx5_destroy_flow_table(esw->fdb_table.offloads.slow_fdb);1215 mlx5_destroy_flow_table(esw->fdb_table.offloads.slow_fdb);
1209slow_fdb_err:1216slow_fdb_err:
1217 /* Holds true only as long as DMFS is the default */
1218 mlx5_flow_namespace_set_mode(root_ns, MLX5_FLOW_STEERING_MODE_DMFS);
1210ns_err:1219ns_err:
1211 kvfree(flow_group_in);1220 kvfree(flow_group_in);
1212 return err;1221 return err;
@@ -1226,6 +1235,9 @@ static void esw_destroy_offloads_fdb_tables(struct mlx5_eswitch *esw) · 13 unmodified lines
1226 1235
1227 mlx5_destroy_flow_table(esw->fdb_table.offloads.slow_fdb);1236 mlx5_destroy_flow_table(esw->fdb_table.offloads.slow_fdb);
1228 esw_destroy_offloads_fast_fdb_tables(esw);1237 esw_destroy_offloads_fast_fdb_tables(esw);
1238 /* Holds true only as long as DMFS is the default */
1239 mlx5_flow_namespace_set_mode(esw->fdb_table.offloads.ns,
1240 MLX5_FLOW_STEERING_MODE_DMFS);
1229}1241}
1230 1242
1231static int esw_create_offloads_table(struct mlx5_eswitch *esw, int nvports)1243static int esw_create_offloads_table(struct mlx5_eswitch *esw, int nvports)
@@ -1623,13 +1635,42 @@ static void mlx5_esw_offloads_unpair(struct mlx5_eswitch *esw) · 391 unmodified lines
1623 esw_del_fdb_peer_miss_rules(esw);1635 esw_del_fdb_peer_miss_rules(esw);
1624}1636}
1625 1637
1638static int mlx5_esw_offloads_set_ns_peer(struct mlx5_eswitch *esw,
1639 struct mlx5_eswitch *peer_esw,
1640 bool pair)
1641{
1642 struct mlx5_flow_root_namespace *peer_ns;
1643 struct mlx5_flow_root_namespace *ns;
1644 int err;
1645
1646 peer_ns = peer_esw->dev->priv.steering->fdb_root_ns;
1647 ns = esw->dev->priv.steering->fdb_root_ns;
1648
1649 if (pair) {
1650 err = mlx5_flow_namespace_set_peer(ns, peer_ns);
1651 if (err)
1652 return err;
1653
1654 mlx5_flow_namespace_set_peer(peer_ns, ns);
1655 if (err) {
1656 mlx5_flow_namespace_set_peer(ns, NULL);
1657 return err;
1658 }
1659 } else {
1660 mlx5_flow_namespace_set_peer(ns, NULL);
1661 mlx5_flow_namespace_set_peer(peer_ns, NULL);
1662 }
1663
1664 return 0;
1665}
1666
1626static int mlx5_esw_offloads_devcom_event(int event,1667static int mlx5_esw_offloads_devcom_event(int event,
1627 void *my_data,1668 void *my_data,
1628 void *event_data)1669 void *event_data)
1629{1670{
1630 struct mlx5_eswitch *esw = my_data;1671 struct mlx5_eswitch *esw = my_data;
1631 struct mlx5_eswitch *peer_esw = event_data;
1632 struct mlx5_devcom *devcom = esw->dev->priv.devcom;1672 struct mlx5_devcom *devcom = esw->dev->priv.devcom;
1673 struct mlx5_eswitch *peer_esw = event_data;
1633 int err;1674 int err;
1634 1675
1635 switch (event) {1676 switch (event) {
@@ -1638,9 +1679,12 @@ static int mlx5_esw_offloads_devcom_event(int event, · 2 unmodified lines
1638 mlx5_eswitch_vport_match_metadata_enabled(peer_esw))1679 mlx5_eswitch_vport_match_metadata_enabled(peer_esw))
1639 break;1680 break;
1640 1681
1641 err = mlx5_esw_offloads_pair(esw, peer_esw);1682 err = mlx5_esw_offloads_set_ns_peer(esw, peer_esw, true);
1642 if (err)1683 if (err)
1643 goto err_out;1684 goto err_out;
1685 err = mlx5_esw_offloads_pair(esw, peer_esw);
1686 if (err)
1687 goto err_peer;
1644 1688
1645 err = mlx5_esw_offloads_pair(peer_esw, esw);1689 err = mlx5_esw_offloads_pair(peer_esw, esw);
1646 if (err)1690 if (err)
@@ -1656,6 +1700,7 @@ static int mlx5_esw_offloads_devcom_event(int event, · 9 unmodified lines
1656 mlx5_devcom_set_paired(devcom, MLX5_DEVCOM_ESW_OFFLOADS, false);1700 mlx5_devcom_set_paired(devcom, MLX5_DEVCOM_ESW_OFFLOADS, false);
1657 mlx5_esw_offloads_unpair(peer_esw);1701 mlx5_esw_offloads_unpair(peer_esw);
1658 mlx5_esw_offloads_unpair(esw);1702 mlx5_esw_offloads_unpair(esw);
1703 mlx5_esw_offloads_set_ns_peer(esw, peer_esw, false);
1659 break;1704 break;
1660 }1705 }
1661 1706
@@ -1663,7 +1708,8 @@ static int mlx5_esw_offloads_devcom_event(int event, · 1 unmodified line
1663 1708
1664err_pair:1709err_pair:
1665 mlx5_esw_offloads_unpair(esw);1710 mlx5_esw_offloads_unpair(esw);
1666 1711err_peer:
1712 mlx5_esw_offloads_set_ns_peer(esw, peer_esw, false);
1667err_out:1713err_out:
1668 mlx5_core_err(esw->dev, "esw offloads devcom event failure, event %u err %d",1714 mlx5_core_err(esw->dev, "esw offloads devcom event failure, event %u err %d",
1669 event, err);1715 event, err);
@@ -1734,7 +1780,7 @@ static int esw_vport_ingress_prio_tag_config(struct mlx5_eswitch *esw, · 64 unmodified lines
1734 1780
1735 if (vport->ingress.modify_metadata_rule) {1781 if (vport->ingress.modify_metadata_rule) {
1736 flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR;1782 flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR;
1737 flow_act.modify_id = vport->ingress.modify_metadata_id;1783 flow_act.modify_hdr = vport->ingress.modify_metadata;
1738 }1784 }
1739 1785
1740 vport->ingress.allow_rule =1786 vport->ingress.allow_rule =
@@ -1770,9 +1816,11 @@ static int esw_vport_add_ingress_acl_modify_metadata(struct mlx5_eswitch *esw, · 29 unmodified lines
1770 MLX5_SET(set_action_in, action, data,1816 MLX5_SET(set_action_in, action, data,
1771 mlx5_eswitch_get_vport_metadata_for_match(esw, vport->vport));1817 mlx5_eswitch_get_vport_metadata_for_match(esw, vport->vport));
1772 1818
1773 err = mlx5_modify_header_alloc(esw->dev, MLX5_FLOW_NAMESPACE_ESW_INGRESS,1819 vport->ingress.modify_metadata =
1774 1, action, &vport->ingress.modify_metadata_id);1820 mlx5_modify_header_alloc(esw->dev, MLX5_FLOW_NAMESPACE_ESW_INGRESS,
1775 if (err) {1821 1, action);
1822 if (IS_ERR(vport->ingress.modify_metadata)) {
1823 err = PTR_ERR(vport->ingress.modify_metadata);
1776 esw_warn(esw->dev,1824 esw_warn(esw->dev,
1777 "failed to alloc modify header for vport %d ingress acl (%d)\n",1825 "failed to alloc modify header for vport %d ingress acl (%d)\n",
1778 vport->vport, err);1826 vport->vport, err);
@@ -1780,7 +1828,7 @@ static int esw_vport_add_ingress_acl_modify_metadata(struct mlx5_eswitch *esw, · 1 unmodified line
1780 }1828 }
1781 1829
1782 flow_act.action = MLX5_FLOW_CONTEXT_ACTION_MOD_HDR | MLX5_FLOW_CONTEXT_ACTION_ALLOW;1830 flow_act.action = MLX5_FLOW_CONTEXT_ACTION_MOD_HDR | MLX5_FLOW_CONTEXT_ACTION_ALLOW;
1783 flow_act.modify_id = vport->ingress.modify_metadata_id;1831 flow_act.modify_hdr = vport->ingress.modify_metadata;
1784 vport->ingress.modify_metadata_rule = mlx5_add_flow_rules(vport->ingress.acl,1832 vport->ingress.modify_metadata_rule = mlx5_add_flow_rules(vport->ingress.acl,
1785 &spec, &flow_act, NULL, 0);1833 &spec, &flow_act, NULL, 0);
1786 if (IS_ERR(vport->ingress.modify_metadata_rule)) {1834 if (IS_ERR(vport->ingress.modify_metadata_rule)) {
@@ -1794,7 +1842,7 @@ static int esw_vport_add_ingress_acl_modify_metadata(struct mlx5_eswitch *esw, · 7 unmodified lines
1794 1842
1795out:1843out:
1796 if (err)1844 if (err)
1797 mlx5_modify_header_dealloc(esw->dev, vport->ingress.modify_metadata_id);1845 mlx5_modify_header_dealloc(esw->dev, vport->ingress.modify_metadata);
1798 return err;1846 return err;
1799}1847}
1800 1848
@@ -1803,7 +1851,7 @@ void esw_vport_del_ingress_acl_modify_metadata(struct mlx5_eswitch *esw, · 2 unmodified lines
1803{1851{
1804 if (vport->ingress.modify_metadata_rule) {1852 if (vport->ingress.modify_metadata_rule) {
1805 mlx5_del_flow_rules(vport->ingress.modify_metadata_rule);1853 mlx5_del_flow_rules(vport->ingress.modify_metadata_rule);
1806 mlx5_modify_header_dealloc(esw->dev, vport->ingress.modify_metadata_id);1854 mlx5_modify_header_dealloc(esw->dev, vport->ingress.modify_metadata);
1807 1855
1808 vport->ingress.modify_metadata_rule = NULL;1856 vport->ingress.modify_metadata_rule = NULL;
1809 }1857 }
@@ -2113,9 +2161,10 @@ int esw_offloads_enable(struct mlx5_eswitch *esw) · 303 unmodified lines
2113 else2161 else
2114 esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_NONE;2162 esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_NONE;
2115 2163
2164 mlx5_rdma_enable_roce(esw->dev);
2116 err = esw_offloads_steering_init(esw);2165 err = esw_offloads_steering_init(esw);
2117 if (err)2166 if (err)
2118 return err;2167 goto err_steering_init;
2119 2168
2120 err = esw_set_passing_vport_metadata(esw, true);2169 err = esw_set_passing_vport_metadata(esw, true);
2121 if (err)2170 if (err)
@@ -2130,8 +2179,6 @@ int esw_offloads_enable(struct mlx5_eswitch *esw) · 8 unmodified lines
2130 esw_offloads_devcom_init(esw);2179 esw_offloads_devcom_init(esw);
2131 mutex_init(&esw->offloads.termtbl_mutex);2180 mutex_init(&esw->offloads.termtbl_mutex);
2132 2181
2133 mlx5_rdma_enable_roce(esw->dev);
2134
2135 return 0;2182 return 0;
2136 2183
2137err_reps:2184err_reps:
@@ -2139,6 +2186,8 @@ err_reps: · 1 unmodified line
2139 esw_set_passing_vport_metadata(esw, false);2186 esw_set_passing_vport_metadata(esw, false);
2140err_vport_metadata:2187err_vport_metadata:
2141 esw_offloads_steering_cleanup(esw);2188 esw_offloads_steering_cleanup(esw);
2189err_steering_init:
2190 mlx5_rdma_disable_roce(esw->dev);
2142 return err;2191 return err;
2143}2192}
2144 2193
@@ -2163,12 +2212,12 @@ static int esw_offloads_stop(struct mlx5_eswitch *esw, · 18 unmodified lines
2163 2212
2164void esw_offloads_disable(struct mlx5_eswitch *esw)2213void esw_offloads_disable(struct mlx5_eswitch *esw)
2165{2214{
2166 mlx5_rdma_disable_roce(esw->dev);
2167 esw_offloads_devcom_cleanup(esw);2215 esw_offloads_devcom_cleanup(esw);
2168 esw_offloads_unload_all_reps(esw);2216 esw_offloads_unload_all_reps(esw);
2169 mlx5_eswitch_disable_pf_vf_vports(esw);2217 mlx5_eswitch_disable_pf_vf_vports(esw);
2170 esw_set_passing_vport_metadata(esw, false);2218 esw_set_passing_vport_metadata(esw, false);
2171 esw_offloads_steering_cleanup(esw);2219 esw_offloads_steering_cleanup(esw);
2220 mlx5_rdma_disable_roce(esw->dev);
2172 esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_NONE;2221 esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_NONE;
2173}2222}
2174 2223
drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c+89−27
@@ -107,6 +107,50 @@ static int mlx5_cmd_stub_delete_fte(struct mlx5_flow_root_namespace *ns, · 106 unmodified lines
107 return 0;107 return 0;
108}108}
109 109
110static int mlx5_cmd_stub_packet_reformat_alloc(struct mlx5_flow_root_namespace *ns,
111 int reformat_type,
112 size_t size,
113 void *reformat_data,
114 enum mlx5_flow_namespace_type namespace,
115 struct mlx5_pkt_reformat *pkt_reformat)
116{
117 return 0;
118}
119
120static void mlx5_cmd_stub_packet_reformat_dealloc(struct mlx5_flow_root_namespace *ns,
121 struct mlx5_pkt_reformat *pkt_reformat)
122{
123}
124
125static int mlx5_cmd_stub_modify_header_alloc(struct mlx5_flow_root_namespace *ns,
126 u8 namespace, u8 num_actions,
127 void *modify_actions,
128 struct mlx5_modify_hdr *modify_hdr)
129{
130 return 0;
131}
132
133static void mlx5_cmd_stub_modify_header_dealloc(struct mlx5_flow_root_namespace *ns,
134 struct mlx5_modify_hdr *modify_hdr)
135{
136}
137
138static int mlx5_cmd_stub_set_peer(struct mlx5_flow_root_namespace *ns,
139 struct mlx5_flow_root_namespace *peer_ns)
140{
141 return 0;
142}
143
144static int mlx5_cmd_stub_create_ns(struct mlx5_flow_root_namespace *ns)
145{
146 return 0;
147}
148
149static int mlx5_cmd_stub_destroy_ns(struct mlx5_flow_root_namespace *ns)
150{
151 return 0;
152}
153
110static int mlx5_cmd_update_root_ft(struct mlx5_flow_root_namespace *ns,154static int mlx5_cmd_update_root_ft(struct mlx5_flow_root_namespace *ns,
111 struct mlx5_flow_table *ft, u32 underlay_qpn,155 struct mlx5_flow_table *ft, u32 underlay_qpn,
112 bool disconnect)156 bool disconnect)
@@ -412,11 +456,13 @@ static int mlx5_cmd_set_fte(struct mlx5_core_dev *dev, · 299 unmodified lines
412 } else {456 } else {
413 MLX5_SET(flow_context, in_flow_context, action,457 MLX5_SET(flow_context, in_flow_context, action,
414 fte->action.action);458 fte->action.action);
415 MLX5_SET(flow_context, in_flow_context, packet_reformat_id,459 if (fte->action.pkt_reformat)
416 fte->action.reformat_id);460 MLX5_SET(flow_context, in_flow_context, packet_reformat_id,
461 fte->action.pkt_reformat->id);
417 }462 }
418 MLX5_SET(flow_context, in_flow_context, modify_header_id,463 if (fte->action.modify_hdr)
419 fte->action.modify_id);464 MLX5_SET(flow_context, in_flow_context, modify_header_id,
465 fte->action.modify_hdr->id);
420 466
421 vlan = MLX5_ADDR_OF(flow_context, in_flow_context, push_vlan);467 vlan = MLX5_ADDR_OF(flow_context, in_flow_context, push_vlan);
422 468
@@ -468,7 +514,7 @@ static int mlx5_cmd_set_fte(struct mlx5_core_dev *dev, · 45 unmodified lines
468 MLX5_FLOW_DEST_VPORT_REFORMAT_ID));514 MLX5_FLOW_DEST_VPORT_REFORMAT_ID));
469 MLX5_SET(extended_dest_format, in_dests,515 MLX5_SET(extended_dest_format, in_dests,
470 packet_reformat_id,516 packet_reformat_id,
471 dst->dest_attr.vport.reformat_id);517 dst->dest_attr.vport.pkt_reformat->id);
472 }518 }
473 break;519 break;
474 default:520 default:
@@ -643,14 +689,15 @@ int mlx5_cmd_fc_bulk_query(struct mlx5_core_dev *dev, u32 base_id, int bulk_len, · 168 unmodified lines
643 return mlx5_cmd_exec(dev, in, sizeof(in), out, outlen);689 return mlx5_cmd_exec(dev, in, sizeof(in), out, outlen);
644}690}
645 691
646int mlx5_packet_reformat_alloc(struct mlx5_core_dev *dev,692static int mlx5_cmd_packet_reformat_alloc(struct mlx5_flow_root_namespace *ns,
647 int reformat_type,693 int reformat_type,
648 size_t size,694 size_t size,
649 void *reformat_data,695 void *reformat_data,
650 enum mlx5_flow_namespace_type namespace,696 enum mlx5_flow_namespace_type namespace,
651 u32 *packet_reformat_id)697 struct mlx5_pkt_reformat *pkt_reformat)
652{698{
653 u32 out[MLX5_ST_SZ_DW(alloc_packet_reformat_context_out)];699 u32 out[MLX5_ST_SZ_DW(alloc_packet_reformat_context_out)];
700 struct mlx5_core_dev *dev = ns->dev;
654 void *packet_reformat_context_in;701 void *packet_reformat_context_in;
655 int max_encap_size;702 int max_encap_size;
656 void *reformat;703 void *reformat;
@@ -693,35 +740,36 @@ int mlx5_packet_reformat_alloc(struct mlx5_core_dev *dev, · 36 unmodified lines
693 memset(out, 0, sizeof(out));740 memset(out, 0, sizeof(out));
694 err = mlx5_cmd_exec(dev, in, inlen, out, sizeof(out));741 err = mlx5_cmd_exec(dev, in, inlen, out, sizeof(out));
695 742
696 *packet_reformat_id = MLX5_GET(alloc_packet_reformat_context_out,743 pkt_reformat->id = MLX5_GET(alloc_packet_reformat_context_out,
697 out, packet_reformat_id);744 out, packet_reformat_id);
698 kfree(in);745 kfree(in);
699 return err;746 return err;
700}747}
701EXPORT_SYMBOL(mlx5_packet_reformat_alloc);
702 748
703void mlx5_packet_reformat_dealloc(struct mlx5_core_dev *dev,749static void mlx5_cmd_packet_reformat_dealloc(struct mlx5_flow_root_namespace *ns,
704 u32 packet_reformat_id)750 struct mlx5_pkt_reformat *pkt_reformat)
705{751{
706 u32 in[MLX5_ST_SZ_DW(dealloc_packet_reformat_context_in)];752 u32 in[MLX5_ST_SZ_DW(dealloc_packet_reformat_context_in)];
707 u32 out[MLX5_ST_SZ_DW(dealloc_packet_reformat_context_out)];753 u32 out[MLX5_ST_SZ_DW(dealloc_packet_reformat_context_out)];
754 struct mlx5_core_dev *dev = ns->dev;
708 755
709 memset(in, 0, sizeof(in));756 memset(in, 0, sizeof(in));
710 MLX5_SET(dealloc_packet_reformat_context_in, in, opcode,757 MLX5_SET(dealloc_packet_reformat_context_in, in, opcode,
711 MLX5_CMD_OP_DEALLOC_PACKET_REFORMAT_CONTEXT);758 MLX5_CMD_OP_DEALLOC_PACKET_REFORMAT_CONTEXT);
712 MLX5_SET(dealloc_packet_reformat_context_in, in, packet_reformat_id,759 MLX5_SET(dealloc_packet_reformat_context_in, in, packet_reformat_id,
713 packet_reformat_id);760 pkt_reformat->id);
714 761
715 mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));762 mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
716}763}
717EXPORT_SYMBOL(mlx5_packet_reformat_dealloc);
718 764
719int mlx5_modify_header_alloc(struct mlx5_core_dev *dev,765static int mlx5_cmd_modify_header_alloc(struct mlx5_flow_root_namespace *ns,
720 u8 namespace, u8 num_actions,766 u8 namespace, u8 num_actions,
721 void *modify_actions, u32 *modify_header_id)767 void *modify_actions,
768 struct mlx5_modify_hdr *modify_hdr)
722{769{
723 u32 out[MLX5_ST_SZ_DW(alloc_modify_header_context_out)];770 u32 out[MLX5_ST_SZ_DW(alloc_modify_header_context_out)];
724 int max_actions, actions_size, inlen, err;771 int max_actions, actions_size, inlen, err;
772 struct mlx5_core_dev *dev = ns->dev;
725 void *actions_in;773 void *actions_in;
726 u8 table_type;774 u8 table_type;
727 u32 *in;775 u32 *in;
@@ -772,26 +820,26 @@ int mlx5_modify_header_alloc(struct mlx5_core_dev *dev, · 44 unmodified lines
772 memset(out, 0, sizeof(out));820 memset(out, 0, sizeof(out));
773 err = mlx5_cmd_exec(dev, in, inlen, out, sizeof(out));821 err = mlx5_cmd_exec(dev, in, inlen, out, sizeof(out));
774 822
775 *modify_header_id = MLX5_GET(alloc_modify_header_context_out, out, modify_header_id);823 modify_hdr->id = MLX5_GET(alloc_modify_header_context_out, out, modify_header_id);
776 kfree(in);824 kfree(in);
777 return err;825 return err;
778}826}
779EXPORT_SYMBOL(mlx5_modify_header_alloc);
780 827
781void mlx5_modify_header_dealloc(struct mlx5_core_dev *dev, u32 modify_header_id)828static void mlx5_cmd_modify_header_dealloc(struct mlx5_flow_root_namespace *ns,
829 struct mlx5_modify_hdr *modify_hdr)
782{830{
783 u32 in[MLX5_ST_SZ_DW(dealloc_modify_header_context_in)];831 u32 in[MLX5_ST_SZ_DW(dealloc_modify_header_context_in)];
784 u32 out[MLX5_ST_SZ_DW(dealloc_modify_header_context_out)];832 u32 out[MLX5_ST_SZ_DW(dealloc_modify_header_context_out)];
833 struct mlx5_core_dev *dev = ns->dev;
785 834
786 memset(in, 0, sizeof(in));835 memset(in, 0, sizeof(in));
787 MLX5_SET(dealloc_modify_header_context_in, in, opcode,836 MLX5_SET(dealloc_modify_header_context_in, in, opcode,
788 MLX5_CMD_OP_DEALLOC_MODIFY_HEADER_CONTEXT);837 MLX5_CMD_OP_DEALLOC_MODIFY_HEADER_CONTEXT);
789 MLX5_SET(dealloc_modify_header_context_in, in, modify_header_id,838 MLX5_SET(dealloc_modify_header_context_in, in, modify_header_id,
790 modify_header_id);839 modify_hdr->id);
791 840
792 mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));841 mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
793}842}
794EXPORT_SYMBOL(mlx5_modify_header_dealloc);
795 843
796static const struct mlx5_flow_cmds mlx5_flow_cmds = {844static const struct mlx5_flow_cmds mlx5_flow_cmds = {
797 .create_flow_table = mlx5_cmd_create_flow_table,845 .create_flow_table = mlx5_cmd_create_flow_table,
@@ -803,6 +851,13 @@ static const struct mlx5_flow_cmds mlx5_flow_cmds = { · 5 unmodified lines
803 .update_fte = mlx5_cmd_update_fte,851 .update_fte = mlx5_cmd_update_fte,
804 .delete_fte = mlx5_cmd_delete_fte,852 .delete_fte = mlx5_cmd_delete_fte,
805 .update_root_ft = mlx5_cmd_update_root_ft,853 .update_root_ft = mlx5_cmd_update_root_ft,
854 .packet_reformat_alloc = mlx5_cmd_packet_reformat_alloc,
855 .packet_reformat_dealloc = mlx5_cmd_packet_reformat_dealloc,
856 .modify_header_alloc = mlx5_cmd_modify_header_alloc,
857 .modify_header_dealloc = mlx5_cmd_modify_header_dealloc,
858 .set_peer = mlx5_cmd_stub_set_peer,
859 .create_ns = mlx5_cmd_stub_create_ns,
860 .destroy_ns = mlx5_cmd_stub_destroy_ns,
806};861};
807 862
808static const struct mlx5_flow_cmds mlx5_flow_cmd_stubs = {863static const struct mlx5_flow_cmds mlx5_flow_cmd_stubs = {
@@ -815,9 +870,16 @@ static const struct mlx5_flow_cmds mlx5_flow_cmd_stubs = { · 6 unmodified lines
815 .update_fte = mlx5_cmd_stub_update_fte,870 .update_fte = mlx5_cmd_stub_update_fte,
816 .delete_fte = mlx5_cmd_stub_delete_fte,871 .delete_fte = mlx5_cmd_stub_delete_fte,
817 .update_root_ft = mlx5_cmd_stub_update_root_ft,872 .update_root_ft = mlx5_cmd_stub_update_root_ft,
873 .packet_reformat_alloc = mlx5_cmd_stub_packet_reformat_alloc,
874 .packet_reformat_dealloc = mlx5_cmd_stub_packet_reformat_dealloc,
875 .modify_header_alloc = mlx5_cmd_stub_modify_header_alloc,
876 .modify_header_dealloc = mlx5_cmd_stub_modify_header_dealloc,
877 .set_peer = mlx5_cmd_stub_set_peer,
878 .create_ns = mlx5_cmd_stub_create_ns,
879 .destroy_ns = mlx5_cmd_stub_destroy_ns,
818};880};
819 881
820static const struct mlx5_flow_cmds *mlx5_fs_cmd_get_fw_cmds(void)882const struct mlx5_flow_cmds *mlx5_fs_cmd_get_fw_cmds(void)
821{883{
822 return &mlx5_flow_cmds;884 return &mlx5_flow_cmds;
823}885}
drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.h+25−0
@@ -75,6 +75,30 @@ struct mlx5_flow_cmds { · 74 unmodified lines
75 struct mlx5_flow_table *ft,75 struct mlx5_flow_table *ft,
76 u32 underlay_qpn,76 u32 underlay_qpn,
77 bool disconnect);77 bool disconnect);
78
79 int (*packet_reformat_alloc)(struct mlx5_flow_root_namespace *ns,
80 int reformat_type,
81 size_t size,
82 void *reformat_data,
83 enum mlx5_flow_namespace_type namespace,
84 struct mlx5_pkt_reformat *pkt_reformat);
85
86 void (*packet_reformat_dealloc)(struct mlx5_flow_root_namespace *ns,
87 struct mlx5_pkt_reformat *pkt_reformat);
88
89 int (*modify_header_alloc)(struct mlx5_flow_root_namespace *ns,
90 u8 namespace, u8 num_actions,
91 void *modify_actions,
92 struct mlx5_modify_hdr *modify_hdr);
93
94 void (*modify_header_dealloc)(struct mlx5_flow_root_namespace *ns,
95 struct mlx5_modify_hdr *modify_hdr);
96
97 int (*set_peer)(struct mlx5_flow_root_namespace *ns,
98 struct mlx5_flow_root_namespace *peer_ns);
99
100 int (*create_ns)(struct mlx5_flow_root_namespace *ns);
101 int (*destroy_ns)(struct mlx5_flow_root_namespace *ns);
78};102};
79 103
80int mlx5_cmd_fc_alloc(struct mlx5_core_dev *dev, u32 *id);104int mlx5_cmd_fc_alloc(struct mlx5_core_dev *dev, u32 *id);
@@ -90,5 +114,6 @@ int mlx5_cmd_fc_bulk_query(struct mlx5_core_dev *dev, u32 base_id, int bulk_len, · 9 unmodified lines
90 u32 *out);114 u32 *out);
91 115
92const struct mlx5_flow_cmds *mlx5_fs_cmd_get_default(enum fs_flow_table_type type);116const struct mlx5_flow_cmds *mlx5_fs_cmd_get_default(enum fs_flow_table_type type);
117const struct mlx5_flow_cmds *mlx5_fs_cmd_get_fw_cmds(void);
93 118
94#endif119#endif
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c+159−1
@@ -1415,7 +1415,8 @@ static bool mlx5_flow_dests_cmp(struct mlx5_flow_destination *d1, · 1414 unmodified lines
1415 ((d1->vport.flags & MLX5_FLOW_DEST_VPORT_VHCA_ID) ?1415 ((d1->vport.flags & MLX5_FLOW_DEST_VPORT_VHCA_ID) ?
1416 (d1->vport.vhca_id == d2->vport.vhca_id) : true) &&1416 (d1->vport.vhca_id == d2->vport.vhca_id) : true) &&
1417 ((d1->vport.flags & MLX5_FLOW_DEST_VPORT_REFORMAT_ID) ?1417 ((d1->vport.flags & MLX5_FLOW_DEST_VPORT_REFORMAT_ID) ?
1418 (d1->vport.reformat_id == d2->vport.reformat_id) : true)) ||1418 (d1->vport.pkt_reformat->id ==
1419 d2->vport.pkt_reformat->id) : true)) ||
1419 (d1->type == MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE &&1420 (d1->type == MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE &&
1420 d1->ft == d2->ft) ||1421 d1->ft == d2->ft) ||
1421 (d1->type == MLX5_FLOW_DESTINATION_TYPE_TIR &&1422 (d1->type == MLX5_FLOW_DESTINATION_TYPE_TIR &&
@@ -2888,3 +2889,160 @@ out: · 1466 unmodified lines
2888 return err;2889 return err;
2889}2890}
2890EXPORT_SYMBOL(mlx5_fs_remove_rx_underlay_qpn);2891EXPORT_SYMBOL(mlx5_fs_remove_rx_underlay_qpn);
2892
2893static struct mlx5_flow_root_namespace
2894*get_root_namespace(struct mlx5_core_dev *dev, enum mlx5_flow_namespace_type ns_type)
2895{
2896 struct mlx5_flow_namespace *ns;
2897
2898 if (ns_type == MLX5_FLOW_NAMESPACE_ESW_EGRESS ||
2899 ns_type == MLX5_FLOW_NAMESPACE_ESW_INGRESS)
2900 ns = mlx5_get_flow_vport_acl_namespace(dev, ns_type, 0);
2901 else
2902 ns = mlx5_get_flow_namespace(dev, ns_type);
2903 if (!ns)
2904 return NULL;
2905
2906 return find_root(&ns->node);
2907}
2908
2909struct mlx5_modify_hdr *mlx5_modify_header_alloc(struct mlx5_core_dev *dev,
2910 u8 ns_type, u8 num_actions,
2911 void *modify_actions)
2912{
2913 struct mlx5_flow_root_namespace *root;
2914 struct mlx5_modify_hdr *modify_hdr;
2915 int err;
2916
2917 root = get_root_namespace(dev, ns_type);
2918 if (!root)
2919 return ERR_PTR(-EOPNOTSUPP);
2920
2921 modify_hdr = kzalloc(sizeof(*modify_hdr), GFP_KERNEL);
2922 if (!modify_hdr)
2923 return ERR_PTR(-ENOMEM);
2924
2925 modify_hdr->ns_type = ns_type;
2926 err = root->cmds->modify_header_alloc(root, ns_type, num_actions,
2927 modify_actions, modify_hdr);
2928 if (err) {
2929 kfree(modify_hdr);
2930 return ERR_PTR(err);
2931 }
2932
2933 return modify_hdr;
2934}
2935EXPORT_SYMBOL(mlx5_modify_header_alloc);
2936
2937void mlx5_modify_header_dealloc(struct mlx5_core_dev *dev,
2938 struct mlx5_modify_hdr *modify_hdr)
2939{
2940 struct mlx5_flow_root_namespace *root;
2941
2942 root = get_root_namespace(dev, modify_hdr->ns_type);
2943 if (WARN_ON(!root))
2944 return;
2945 root->cmds->modify_header_dealloc(root, modify_hdr);
2946 kfree(modify_hdr);
2947}
2948EXPORT_SYMBOL(mlx5_modify_header_dealloc);
2949
2950struct mlx5_pkt_reformat *mlx5_packet_reformat_alloc(struct mlx5_core_dev *dev,
2951 int reformat_type,
2952 size_t size,
2953 void *reformat_data,
2954 enum mlx5_flow_namespace_type ns_type)
2955{
2956 struct mlx5_pkt_reformat *pkt_reformat;
2957 struct mlx5_flow_root_namespace *root;
2958 int err;
2959
2960 root = get_root_namespace(dev, ns_type);
2961 if (!root)
2962 return ERR_PTR(-EOPNOTSUPP);
2963
2964 pkt_reformat = kzalloc(sizeof(*pkt_reformat), GFP_KERNEL);
2965 if (!pkt_reformat)
2966 return ERR_PTR(-ENOMEM);
2967
2968 pkt_reformat->ns_type = ns_type;
2969 pkt_reformat->reformat_type = reformat_type;
2970 err = root->cmds->packet_reformat_alloc(root, reformat_type, size,
2971 reformat_data, ns_type,
2972 pkt_reformat);
2973 if (err) {
2974 kfree(pkt_reformat);
2975 return ERR_PTR(err);
2976 }
2977
2978 return pkt_reformat;
2979}
2980EXPORT_SYMBOL(mlx5_packet_reformat_alloc);
2981
2982void mlx5_packet_reformat_dealloc(struct mlx5_core_dev *dev,
2983 struct mlx5_pkt_reformat *pkt_reformat)
2984{
2985 struct mlx5_flow_root_namespace *root;
2986
2987 root = get_root_namespace(dev, pkt_reformat->ns_type);
2988 if (WARN_ON(!root))
2989 return;
2990 root->cmds->packet_reformat_dealloc(root, pkt_reformat);
2991 kfree(pkt_reformat);
2992}
2993EXPORT_SYMBOL(mlx5_packet_reformat_dealloc);
2994
2995int mlx5_flow_namespace_set_peer(struct mlx5_flow_root_namespace *ns,
2996 struct mlx5_flow_root_namespace *peer_ns)
2997{
2998 if (peer_ns && ns->mode != peer_ns->mode) {
2999 mlx5_core_err(ns->dev,
3000 "Can't peer namespace of different steering mode\n");
3001 return -EINVAL;
3002 }
3003
3004 return ns->cmds->set_peer(ns, peer_ns);
3005}
3006
3007/* This function should be called only at init stage of the namespace.
3008 * It is not safe to call this function while steering operations
3009 * are executed in the namespace.
3010 */
3011int mlx5_flow_namespace_set_mode(struct mlx5_flow_namespace *ns,
3012 enum mlx5_flow_steering_mode mode)
3013{
3014 struct mlx5_flow_root_namespace *root;
3015 const struct mlx5_flow_cmds *cmds;
3016 int err;
3017
3018 root = find_root(&ns->node);
3019 if (&root->ns != ns)
3020 /* Can't set cmds to non root namespace */
3021 return -EINVAL;
3022
3023 if (root->table_type != FS_FT_FDB)
3024 return -EOPNOTSUPP;
3025
3026 if (root->mode == mode)
3027 return 0;
3028
3029 if (mode == MLX5_FLOW_STEERING_MODE_SMFS)
3030 cmds = mlx5_fs_cmd_get_dr_cmds();
3031 else
3032 cmds = mlx5_fs_cmd_get_fw_cmds();
3033 if (!cmds)
3034 return -EOPNOTSUPP;
3035
3036 err = cmds->create_ns(root);
3037 if (err) {
3038 mlx5_core_err(root->dev, "Failed to create flow namespace (%d)\n",
3039 err);
3040 return err;
3041 }
3042
3043 root->cmds->destroy_ns(root);
3044 root->cmds = cmds;
3045 root->mode = mode;
3046
3047 return 0;
3048}
drivers/net/ethernet/mellanox/mlx5/core/fs_core.h+38−1
@@ -37,6 +37,24 @@ · 36 unmodified lines
37#include <linux/mlx5/fs.h>37#include <linux/mlx5/fs.h>
38#include <linux/rhashtable.h>38#include <linux/rhashtable.h>
39#include <linux/llist.h>39#include <linux/llist.h>
40#include <steering/fs_dr.h>
41
42struct mlx5_modify_hdr {
43 enum mlx5_flow_namespace_type ns_type;
44 union {
45 struct mlx5_fs_dr_action action;
46 u32 id;
47 };
48};
49
50struct mlx5_pkt_reformat {
51 enum mlx5_flow_namespace_type ns_type;
52 int reformat_type; /* from mlx5_ifc */
53 union {
54 struct mlx5_fs_dr_action action;
55 u32 id;
56 };
57};
40 58
41/* FS_TYPE_PRIO_CHAINS is a PRIO that will have namespaces only,59/* FS_TYPE_PRIO_CHAINS is a PRIO that will have namespaces only,
42 * and those are in parallel to one another when going over them to connect60 * and those are in parallel to one another when going over them to connect
@@ -80,9 +98,15 @@ enum fs_fte_status { · 37 unmodified lines
80 FS_FTE_STATUS_EXISTING = 1UL << 0,98 FS_FTE_STATUS_EXISTING = 1UL << 0,
81};99};
82 100
101enum mlx5_flow_steering_mode {
102 MLX5_FLOW_STEERING_MODE_DMFS,
103 MLX5_FLOW_STEERING_MODE_SMFS
104};
105
83struct mlx5_flow_steering {106struct mlx5_flow_steering {
84 struct mlx5_core_dev *dev;107 struct mlx5_core_dev *dev;
85 struct kmem_cache *fgs_cache;108 enum mlx5_flow_steering_mode mode;
109 struct kmem_cache *fgs_cache;
86 struct kmem_cache *ftes_cache;110 struct kmem_cache *ftes_cache;
87 struct mlx5_flow_root_namespace *root_ns;111 struct mlx5_flow_root_namespace *root_ns;
88 struct mlx5_flow_root_namespace *fdb_root_ns;112 struct mlx5_flow_root_namespace *fdb_root_ns;
@@ -128,6 +152,7 @@ struct mlx5_flow_handle { · 39 unmodified lines
128/* Type of children is mlx5_flow_group */152/* Type of children is mlx5_flow_group */
129struct mlx5_flow_table {153struct mlx5_flow_table {
130 struct fs_node node;154 struct fs_node node;
155 struct mlx5_fs_dr_table fs_dr_table;
131 u32 id;156 u32 id;
132 u16 vport;157 u16 vport;
133 unsigned int max_fte;158 unsigned int max_fte;
@@ -168,6 +193,7 @@ struct mlx5_ft_underlay_qp { · 34 unmodified lines
168/* Type of children is mlx5_flow_rule */193/* Type of children is mlx5_flow_rule */
169struct fs_fte {194struct fs_fte {
170 struct fs_node node;195 struct fs_node node;
196 struct mlx5_fs_dr_rule fs_dr_rule;
171 u32 val[MLX5_ST_SZ_DW_MATCH_PARAM];197 u32 val[MLX5_ST_SZ_DW_MATCH_PARAM];
172 u32 dests_size;198 u32 dests_size;
173 u32 index;199 u32 index;
@@ -203,6 +229,7 @@ struct mlx5_flow_group_mask { · 29 unmodified lines
203/* Type of children is fs_fte */229/* Type of children is fs_fte */
204struct mlx5_flow_group {230struct mlx5_flow_group {
205 struct fs_node node;231 struct fs_node node;
232 struct mlx5_fs_dr_matcher fs_dr_matcher;
206 struct mlx5_flow_group_mask mask;233 struct mlx5_flow_group_mask mask;
207 u32 start_index;234 u32 start_index;
208 u32 max_ftes;235 u32 max_ftes;
@@ -214,6 +241,8 @@ struct mlx5_flow_group { · 5 unmodified lines
214 241
215struct mlx5_flow_root_namespace {242struct mlx5_flow_root_namespace {
216 struct mlx5_flow_namespace ns;243 struct mlx5_flow_namespace ns;
244 enum mlx5_flow_steering_mode mode;
245 struct mlx5_fs_dr_domain fs_dr_domain;
217 enum fs_flow_table_type table_type;246 enum fs_flow_table_type table_type;
218 struct mlx5_core_dev *dev;247 struct mlx5_core_dev *dev;
219 struct mlx5_flow_table *root_ft;248 struct mlx5_flow_table *root_ft;
@@ -231,6 +260,14 @@ void mlx5_fc_queue_stats_work(struct mlx5_core_dev *dev, · 11 unmodified lines
231void mlx5_fc_update_sampling_interval(struct mlx5_core_dev *dev,260void mlx5_fc_update_sampling_interval(struct mlx5_core_dev *dev,
232 unsigned long interval);261 unsigned long interval);
233 262
263const struct mlx5_flow_cmds *mlx5_fs_cmd_get_fw_cmds(void);
264
265int mlx5_flow_namespace_set_peer(struct mlx5_flow_root_namespace *ns,
266 struct mlx5_flow_root_namespace *peer_ns);
267
268int mlx5_flow_namespace_set_mode(struct mlx5_flow_namespace *ns,
269 enum mlx5_flow_steering_mode mode);
270
234int mlx5_init_fs(struct mlx5_core_dev *dev);271int mlx5_init_fs(struct mlx5_core_dev *dev);
235void mlx5_cleanup_fs(struct mlx5_core_dev *dev);272void mlx5_cleanup_fs(struct mlx5_core_dev *dev);
236 273
drivers/net/ethernet/mellanox/mlx5/core/lib/dm.c+223−0
@@ -0,0 +1,223 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2// Copyright (c) 2019 Mellanox Technologies
3
4#include <linux/mlx5/driver.h>
5#include <linux/mlx5/device.h>
6
7#include "mlx5_core.h"
8#include "lib/mlx5.h"
9
10struct mlx5_dm {
11 /* protect access to icm bitmask */
12 spinlock_t lock;
13 unsigned long *steering_sw_icm_alloc_blocks;
14 unsigned long *header_modify_sw_icm_alloc_blocks;
15};
16
17struct mlx5_dm *mlx5_dm_create(struct mlx5_core_dev *dev)
18{
19 u64 header_modify_icm_blocks = 0;
20 u64 steering_icm_blocks = 0;
21 struct mlx5_dm *dm;
22
23 if (!(MLX5_CAP_GEN_64(dev, general_obj_types) & MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM))
24 return 0;
25
26 dm = kzalloc(sizeof(*dm), GFP_KERNEL);
27 if (!dm)
28 return ERR_PTR(-ENOMEM);
29
30 spin_lock_init(&dm->lock);
31
32 if (MLX5_CAP64_DEV_MEM(dev, steering_sw_icm_start_address)) {
33 steering_icm_blocks =
34 BIT(MLX5_CAP_DEV_MEM(dev, log_steering_sw_icm_size) -
35 MLX5_LOG_SW_ICM_BLOCK_SIZE(dev));
36
37 dm->steering_sw_icm_alloc_blocks =
38 kcalloc(BITS_TO_LONGS(steering_icm_blocks),
39 sizeof(unsigned long), GFP_KERNEL);
40 if (!dm->steering_sw_icm_alloc_blocks)
41 goto err_steering;
42 }
43
44 if (MLX5_CAP64_DEV_MEM(dev, header_modify_sw_icm_start_address)) {
45 header_modify_icm_blocks =
46 BIT(MLX5_CAP_DEV_MEM(dev, log_header_modify_sw_icm_size) -
47 MLX5_LOG_SW_ICM_BLOCK_SIZE(dev));
48
49 dm->header_modify_sw_icm_alloc_blocks =
50 kcalloc(BITS_TO_LONGS(header_modify_icm_blocks),
51 sizeof(unsigned long), GFP_KERNEL);
52 if (!dm->header_modify_sw_icm_alloc_blocks)
53 goto err_modify_hdr;
54 }
55
56 return dm;
57
58err_modify_hdr:
59 kfree(dm->steering_sw_icm_alloc_blocks);
60
61err_steering:
62 kfree(dm);
63
64 return ERR_PTR(-ENOMEM);
65}
66
67void mlx5_dm_cleanup(struct mlx5_core_dev *dev)
68{
69 struct mlx5_dm *dm = dev->dm;
70
71 if (!dev->dm)
72 return;
73
74 if (dm->steering_sw_icm_alloc_blocks) {
75 WARN_ON(!bitmap_empty(dm->steering_sw_icm_alloc_blocks,
76 BIT(MLX5_CAP_DEV_MEM(dev, log_steering_sw_icm_size) -
77 MLX5_LOG_SW_ICM_BLOCK_SIZE(dev))));
78 kfree(dm->steering_sw_icm_alloc_blocks);
79 }
80
81 if (dm->header_modify_sw_icm_alloc_blocks) {
82 WARN_ON(!bitmap_empty(dm->header_modify_sw_icm_alloc_blocks,
83 BIT(MLX5_CAP_DEV_MEM(dev,
84 log_header_modify_sw_icm_size) -
85 MLX5_LOG_SW_ICM_BLOCK_SIZE(dev))));
86 kfree(dm->header_modify_sw_icm_alloc_blocks);
87 }
88
89 kfree(dm);
90}
91
92int mlx5_dm_sw_icm_alloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type,
93 u64 length, u16 uid, phys_addr_t *addr, u32 *obj_id)
94{
95 u32 num_blocks = DIV_ROUND_UP_ULL(length, MLX5_SW_ICM_BLOCK_SIZE(dev));
96 u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {};
97 u32 in[MLX5_ST_SZ_DW(create_sw_icm_in)] = {};
98 struct mlx5_dm *dm = dev->dm;
99 unsigned long *block_map;
100 u64 icm_start_addr;
101 u32 log_icm_size;
102 u32 max_blocks;
103 u64 block_idx;
104 void *sw_icm;
105 int ret;
106
107 if (!dev->dm)
108 return -EOPNOTSUPP;
109
110 if (!length || (length & (length - 1)) ||
111 length & (MLX5_SW_ICM_BLOCK_SIZE(dev) - 1))
112 return -EINVAL;
113
114 MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
115 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
116 MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, MLX5_OBJ_TYPE_SW_ICM);
117 MLX5_SET(general_obj_in_cmd_hdr, in, uid, uid);
118
119 switch (type) {
120 case MLX5_SW_ICM_TYPE_STEERING:
121 icm_start_addr = MLX5_CAP64_DEV_MEM(dev, steering_sw_icm_start_address);
122 log_icm_size = MLX5_CAP_DEV_MEM(dev, log_steering_sw_icm_size);
123 block_map = dm->steering_sw_icm_alloc_blocks;
124 break;
125 case MLX5_SW_ICM_TYPE_HEADER_MODIFY:
126 icm_start_addr = MLX5_CAP64_DEV_MEM(dev, header_modify_sw_icm_start_address);
127 log_icm_size = MLX5_CAP_DEV_MEM(dev,
128 log_header_modify_sw_icm_size);
129 block_map = dm->header_modify_sw_icm_alloc_blocks;
130 break;
131 default:
132 return -EINVAL;
133 }
134
135 if (!block_map)
136 return -EOPNOTSUPP;
137
138 max_blocks = BIT(log_icm_size - MLX5_LOG_SW_ICM_BLOCK_SIZE(dev));
139 spin_lock(&dm->lock);
140 block_idx = bitmap_find_next_zero_area(block_map,
141 max_blocks,
142 0,
143 num_blocks, 0);
144
145 if (block_idx < max_blocks)
146 bitmap_set(block_map,
147 block_idx, num_blocks);
148
149 spin_unlock(&dm->lock);
150
151 if (block_idx >= max_blocks)
152 return -ENOMEM;
153
154 sw_icm = MLX5_ADDR_OF(create_sw_icm_in, in, sw_icm);
155 icm_start_addr += block_idx << MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
156 MLX5_SET64(sw_icm, sw_icm, sw_icm_start_addr,
157 icm_start_addr);
158 MLX5_SET(sw_icm, sw_icm, log_sw_icm_size, ilog2(length));
159
160 ret = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
161 if (ret) {
162 spin_lock(&dm->lock);
163 bitmap_clear(block_map,
164 block_idx, num_blocks);
165 spin_unlock(&dm->lock);
166
167 return ret;
168 }
169
170 *addr = icm_start_addr;
171 *obj_id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
172
173 return 0;
174}
175EXPORT_SYMBOL_GPL(mlx5_dm_sw_icm_alloc);
176
177int mlx5_dm_sw_icm_dealloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type,
178 u64 length, u16 uid, phys_addr_t addr, u32 obj_id)
179{
180 u32 num_blocks = DIV_ROUND_UP_ULL(length, MLX5_SW_ICM_BLOCK_SIZE(dev));
181 u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {};
182 u32 in[MLX5_ST_SZ_DW(general_obj_in_cmd_hdr)] = {};
183 struct mlx5_dm *dm = dev->dm;
184 unsigned long *block_map;
185 u64 icm_start_addr;
186 u64 start_idx;
187 int err;
188
189 if (!dev->dm)
190 return -EOPNOTSUPP;
191
192 switch (type) {
193 case MLX5_SW_ICM_TYPE_STEERING:
194 icm_start_addr = MLX5_CAP64_DEV_MEM(dev, steering_sw_icm_start_address);
195 block_map = dm->steering_sw_icm_alloc_blocks;
196 break;
197 case MLX5_SW_ICM_TYPE_HEADER_MODIFY:
198 icm_start_addr = MLX5_CAP64_DEV_MEM(dev, header_modify_sw_icm_start_address);
199 block_map = dm->header_modify_sw_icm_alloc_blocks;
200 break;
201 default:
202 return -EINVAL;
203 }
204
205 MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
206 MLX5_CMD_OP_DESTROY_GENERAL_OBJECT);
207 MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, MLX5_OBJ_TYPE_SW_ICM);
208 MLX5_SET(general_obj_in_cmd_hdr, in, obj_id, obj_id);
209 MLX5_SET(general_obj_in_cmd_hdr, in, uid, uid);
210
211 err = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
212 if (err)
213 return err;
214
215 start_idx = (addr - icm_start_addr) >> MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
216 spin_lock(&dm->lock);
217 bitmap_clear(block_map,
218 start_idx, num_blocks);
219 spin_unlock(&dm->lock);
220
221 return 0;
222}
223EXPORT_SYMBOL_GPL(mlx5_dm_sw_icm_dealloc);
drivers/net/ethernet/mellanox/mlx5/core/main.c+5−0
@@ -876,6 +876,10 @@ static int mlx5_init_once(struct mlx5_core_dev *dev) · 875 unmodified lines
876 goto err_eswitch_cleanup;876 goto err_eswitch_cleanup;
877 }877 }
878 878
879 dev->dm = mlx5_dm_create(dev);
880 if (IS_ERR(dev->dm))
881 mlx5_core_warn(dev, "Failed to init device memory%d\n", err);
882
879 dev->tracer = mlx5_fw_tracer_create(dev);883 dev->tracer = mlx5_fw_tracer_create(dev);
880 dev->hv_vhca = mlx5_hv_vhca_create(dev);884 dev->hv_vhca = mlx5_hv_vhca_create(dev);
881 885
@@ -910,6 +914,7 @@ static void mlx5_cleanup_once(struct mlx5_core_dev *dev) · 28 unmodified lines
910{914{
911 mlx5_hv_vhca_destroy(dev->hv_vhca);915 mlx5_hv_vhca_destroy(dev->hv_vhca);
912 mlx5_fw_tracer_destroy(dev->tracer);916 mlx5_fw_tracer_destroy(dev->tracer);
917 mlx5_dm_cleanup(dev);
913 mlx5_fpga_cleanup(dev);918 mlx5_fpga_cleanup(dev);
914 mlx5_eswitch_cleanup(dev->priv.eswitch);919 mlx5_eswitch_cleanup(dev->priv.eswitch);
915 mlx5_sriov_cleanup(dev);920 mlx5_sriov_cleanup(dev);
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h+3−0
@@ -198,6 +198,9 @@ int mlx5_set_mtpps(struct mlx5_core_dev *mdev, u32 *mtpps, u32 mtpps_size); · 197 unmodified lines
198int mlx5_query_mtppse(struct mlx5_core_dev *mdev, u8 pin, u8 *arm, u8 *mode);198int mlx5_query_mtppse(struct mlx5_core_dev *mdev, u8 pin, u8 *arm, u8 *mode);
199int mlx5_set_mtppse(struct mlx5_core_dev *mdev, u8 pin, u8 arm, u8 mode);199int mlx5_set_mtppse(struct mlx5_core_dev *mdev, u8 pin, u8 arm, u8 mode);
200 200
201struct mlx5_dm *mlx5_dm_create(struct mlx5_core_dev *dev);
202void mlx5_dm_cleanup(struct mlx5_core_dev *dev);
203
201#define MLX5_PPS_CAP(mdev) (MLX5_CAP_GEN((mdev), pps) && \204#define MLX5_PPS_CAP(mdev) (MLX5_CAP_GEN((mdev), pps) && \
202 MLX5_CAP_GEN((mdev), pps_modify) && \205 MLX5_CAP_GEN((mdev), pps_modify) && \
203 MLX5_CAP_MCAM_FEATURE((mdev), mtpps_fs) && \206 MLX5_CAP_MCAM_FEATURE((mdev), mtpps_fs) && \
drivers/net/ethernet/mellanox/mlx5/core/rdma.c+5−3
@@ -14,9 +14,6 @@ static void mlx5_rdma_disable_roce_steering(struct mlx5_core_dev *dev) · 13 unmodified lines
14{14{
15 struct mlx5_core_roce *roce = &dev->priv.roce;15 struct mlx5_core_roce *roce = &dev->priv.roce;
16 16
17 if (!roce->ft)
18 return;
19
20 mlx5_del_flow_rules(roce->allow_rule);17 mlx5_del_flow_rules(roce->allow_rule);
21 mlx5_destroy_flow_group(roce->fg);18 mlx5_destroy_flow_group(roce->fg);
22 mlx5_destroy_flow_table(roce->ft);19 mlx5_destroy_flow_table(roce->ft);
@@ -145,6 +142,11 @@ static int mlx5_rdma_add_roce_addr(struct mlx5_core_dev *dev) · 122 unmodified lines
145 142
146void mlx5_rdma_disable_roce(struct mlx5_core_dev *dev)143void mlx5_rdma_disable_roce(struct mlx5_core_dev *dev)
147{144{
145 struct mlx5_core_roce *roce = &dev->priv.roce;
146
147 if (!roce->ft)
148 return;
149
148 mlx5_rdma_disable_roce_steering(dev);150 mlx5_rdma_disable_roce_steering(dev);
149 mlx5_rdma_del_roce_addr(dev);151 mlx5_rdma_del_roce_addr(dev);
150 mlx5_nic_vport_disable_roce(dev);152 mlx5_nic_vport_disable_roce(dev);
drivers/net/ethernet/mellanox/mlx5/core/steering/Makefile+2−0
@@ -0,0 +1,2 @@
1# SPDX-License-Identifier: GPL-2.0-only
2subdir-ccflags-y += -I$(src)/..
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_action.c+1588−0
@@ -0,0 +1,1588 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2/* Copyright (c) 2019 Mellanox Technologies. */
3
4#include "dr_types.h"
5
6enum dr_action_domain {
7 DR_ACTION_DOMAIN_NIC_INGRESS,
8 DR_ACTION_DOMAIN_NIC_EGRESS,
9 DR_ACTION_DOMAIN_FDB_INGRESS,
10 DR_ACTION_DOMAIN_FDB_EGRESS,
11 DR_ACTION_DOMAIN_MAX,
12};
13
14enum dr_action_valid_state {
15 DR_ACTION_STATE_ERR,
16 DR_ACTION_STATE_NO_ACTION,
17 DR_ACTION_STATE_REFORMAT,
18 DR_ACTION_STATE_MODIFY_HDR,
19 DR_ACTION_STATE_MODIFY_VLAN,
20 DR_ACTION_STATE_NON_TERM,
21 DR_ACTION_STATE_TERM,
22 DR_ACTION_STATE_MAX,
23};
24
25static const enum dr_action_valid_state
26next_action_state[DR_ACTION_DOMAIN_MAX][DR_ACTION_STATE_MAX][DR_ACTION_TYP_MAX] = {
27 [DR_ACTION_DOMAIN_NIC_INGRESS] = {
28 [DR_ACTION_STATE_NO_ACTION] = {
29 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
30 [DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
31 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
32 [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_NON_TERM,
33 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
34 [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_REFORMAT,
35 [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_REFORMAT,
36 [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
37 [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
38 },
39 [DR_ACTION_STATE_REFORMAT] = {
40 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
41 [DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
42 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
43 [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_REFORMAT,
44 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_REFORMAT,
45 [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
46 [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
47 },
48 [DR_ACTION_STATE_MODIFY_HDR] = {
49 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
50 [DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
51 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
52 [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_MODIFY_HDR,
53 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_HDR,
54 },
55 [DR_ACTION_STATE_MODIFY_VLAN] = {
56 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
57 [DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
58 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
59 [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_MODIFY_VLAN,
60 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_VLAN,
61 [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
62 [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
63 },
64 [DR_ACTION_STATE_NON_TERM] = {
65 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
66 [DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
67 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
68 [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_NON_TERM,
69 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
70 [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_REFORMAT,
71 [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_REFORMAT,
72 [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
73 [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
74 },
75 [DR_ACTION_STATE_TERM] = {
76 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_TERM,
77 },
78 },
79 [DR_ACTION_DOMAIN_NIC_EGRESS] = {
80 [DR_ACTION_STATE_NO_ACTION] = {
81 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
82 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
83 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
84 [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
85 [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
86 [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
87 [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
88 },
89 [DR_ACTION_STATE_REFORMAT] = {
90 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
91 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
92 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_REFORMAT,
93 },
94 [DR_ACTION_STATE_MODIFY_HDR] = {
95 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
96 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
97 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_HDR,
98 [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
99 [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
100 [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
101 },
102 [DR_ACTION_STATE_MODIFY_VLAN] = {
103 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
104 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
105 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_VLAN,
106 [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
107 [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
108 [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
109 },
110 [DR_ACTION_STATE_NON_TERM] = {
111 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
112 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
113 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
114 [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
115 [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
116 [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
117 [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
118 },
119 [DR_ACTION_STATE_TERM] = {
120 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_TERM,
121 },
122 },
123 [DR_ACTION_DOMAIN_FDB_INGRESS] = {
124 [DR_ACTION_STATE_NO_ACTION] = {
125 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
126 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
127 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
128 [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_REFORMAT,
129 [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_REFORMAT,
130 [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
131 [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
132 [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
133 },
134 [DR_ACTION_STATE_REFORMAT] = {
135 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
136 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
137 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_REFORMAT,
138 [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
139 [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
140 [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
141 },
142 [DR_ACTION_STATE_MODIFY_HDR] = {
143 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
144 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
145 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_HDR,
146 [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
147 },
148 [DR_ACTION_STATE_MODIFY_VLAN] = {
149 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
150 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
151 [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
152 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_VLAN,
153 [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
154 [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
155 },
156 [DR_ACTION_STATE_NON_TERM] = {
157 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
158 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
159 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
160 [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_REFORMAT,
161 [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_REFORMAT,
162 [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
163 [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
164 [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
165 },
166 [DR_ACTION_STATE_TERM] = {
167 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_TERM,
168 },
169 },
170 [DR_ACTION_DOMAIN_FDB_EGRESS] = {
171 [DR_ACTION_STATE_NO_ACTION] = {
172 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
173 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
174 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
175 [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
176 [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
177 [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
178 [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
179 [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
180 },
181 [DR_ACTION_STATE_REFORMAT] = {
182 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
183 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
184 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_REFORMAT,
185 [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
186 },
187 [DR_ACTION_STATE_MODIFY_HDR] = {
188 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
189 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
190 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_HDR,
191 [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
192 [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
193 [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
194 [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
195 },
196 [DR_ACTION_STATE_MODIFY_VLAN] = {
197 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
198 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
199 [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
200 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_VLAN,
201 [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
202 [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
203 [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
204 },
205 [DR_ACTION_STATE_NON_TERM] = {
206 [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
207 [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
208 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
209 [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
210 [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
211 [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
212 [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
213 [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
214 },
215 [DR_ACTION_STATE_TERM] = {
216 [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_TERM,
217 },
218 },
219};
220
221struct dr_action_modify_field_conv {
222 u16 hw_field;
223 u8 start;
224 u8 end;
225 u8 l3_type;
226 u8 l4_type;
227};
228
229static const struct dr_action_modify_field_conv dr_action_conv_arr[] = {
230 [MLX5_ACTION_IN_FIELD_OUT_SMAC_47_16] = {
231 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L2_1, .start = 16, .end = 47,
232 },
233 [MLX5_ACTION_IN_FIELD_OUT_SMAC_15_0] = {
234 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L2_1, .start = 0, .end = 15,
235 },
236 [MLX5_ACTION_IN_FIELD_OUT_ETHERTYPE] = {
237 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L2_2, .start = 32, .end = 47,
238 },
239 [MLX5_ACTION_IN_FIELD_OUT_DMAC_47_16] = {
240 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L2_0, .start = 16, .end = 47,
241 },
242 [MLX5_ACTION_IN_FIELD_OUT_DMAC_15_0] = {
243 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L2_0, .start = 0, .end = 15,
244 },
245 [MLX5_ACTION_IN_FIELD_OUT_IP_DSCP] = {
246 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_1, .start = 0, .end = 5,
247 },
248 [MLX5_ACTION_IN_FIELD_OUT_TCP_FLAGS] = {
249 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_0, .start = 48, .end = 56,
250 .l4_type = MLX5DR_ACTION_MDFY_HW_HDR_L4_TCP,
251 },
252 [MLX5_ACTION_IN_FIELD_OUT_TCP_SPORT] = {
253 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_0, .start = 0, .end = 15,
254 .l4_type = MLX5DR_ACTION_MDFY_HW_HDR_L4_TCP,
255 },
256 [MLX5_ACTION_IN_FIELD_OUT_TCP_DPORT] = {
257 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_0, .start = 16, .end = 31,
258 .l4_type = MLX5DR_ACTION_MDFY_HW_HDR_L4_TCP,
259 },
260 [MLX5_ACTION_IN_FIELD_OUT_IP_TTL] = {
261 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_1, .start = 8, .end = 15,
262 .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV4,
263 },
264 [MLX5_ACTION_IN_FIELD_OUT_IPV6_HOPLIMIT] = {
265 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_1, .start = 8, .end = 15,
266 .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
267 },
268 [MLX5_ACTION_IN_FIELD_OUT_UDP_SPORT] = {
269 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_0, .start = 0, .end = 15,
270 .l4_type = MLX5DR_ACTION_MDFY_HW_HDR_L4_UDP,
271 },
272 [MLX5_ACTION_IN_FIELD_OUT_UDP_DPORT] = {
273 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_0, .start = 16, .end = 31,
274 .l4_type = MLX5DR_ACTION_MDFY_HW_HDR_L4_UDP,
275 },
276 [MLX5_ACTION_IN_FIELD_OUT_SIPV6_127_96] = {
277 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_3, .start = 32, .end = 63,
278 .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
279 },
280 [MLX5_ACTION_IN_FIELD_OUT_SIPV6_95_64] = {
281 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_3, .start = 0, .end = 31,
282 .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
283 },
284 [MLX5_ACTION_IN_FIELD_OUT_SIPV6_63_32] = {
285 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_4, .start = 32, .end = 63,
286 .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
287 },
288 [MLX5_ACTION_IN_FIELD_OUT_SIPV6_31_0] = {
289 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_4, .start = 0, .end = 31,
290 .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
291 },
292 [MLX5_ACTION_IN_FIELD_OUT_DIPV6_127_96] = {
293 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_0, .start = 32, .end = 63,
294 .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
295 },
296 [MLX5_ACTION_IN_FIELD_OUT_DIPV6_95_64] = {
297 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_0, .start = 0, .end = 31,
298 .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
299 },
300 [MLX5_ACTION_IN_FIELD_OUT_DIPV6_63_32] = {
301 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_2, .start = 32, .end = 63,
302 .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
303 },
304 [MLX5_ACTION_IN_FIELD_OUT_DIPV6_31_0] = {
305 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_2, .start = 0, .end = 31,
306 .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
307 },
308 [MLX5_ACTION_IN_FIELD_OUT_SIPV4] = {
309 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_0, .start = 0, .end = 31,
310 .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV4,
311 },
312 [MLX5_ACTION_IN_FIELD_OUT_DIPV4] = {
313 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_0, .start = 32, .end = 63,
314 .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV4,
315 },
316 [MLX5_ACTION_IN_FIELD_METADATA_REG_A] = {
317 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_METADATA, .start = 0, .end = 31,
318 },
319 [MLX5_ACTION_IN_FIELD_METADATA_REG_B] = {
320 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_METADATA, .start = 32, .end = 63,
321 },
322 [MLX5_ACTION_IN_FIELD_METADATA_REG_C_0] = {
323 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_REG_0, .start = 32, .end = 63,
324 },
325 [MLX5_ACTION_IN_FIELD_METADATA_REG_C_1] = {
326 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_REG_0, .start = 0, .end = 31,
327 },
328 [MLX5_ACTION_IN_FIELD_METADATA_REG_C_2] = {
329 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_REG_1, .start = 32, .end = 63,
330 },
331 [MLX5_ACTION_IN_FIELD_METADATA_REG_C_3] = {
332 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_REG_1, .start = 0, .end = 31,
333 },
334 [MLX5_ACTION_IN_FIELD_METADATA_REG_C_4] = {
335 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_REG_2, .start = 32, .end = 63,
336 },
337 [MLX5_ACTION_IN_FIELD_METADATA_REG_C_5] = {
338 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_REG_2, .start = 0, .end = 31,
339 },
340 [MLX5_ACTION_IN_FIELD_OUT_TCP_SEQ_NUM] = {
341 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_1, .start = 32, .end = 63,
342 },
343 [MLX5_ACTION_IN_FIELD_OUT_TCP_ACK_NUM] = {
344 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_1, .start = 0, .end = 31,
345 },
346 [MLX5_ACTION_IN_FIELD_OUT_FIRST_VID] = {
347 .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L2_2, .start = 0, .end = 15,
348 },
349};
350
351#define MAX_VLANS 2
352struct dr_action_vlan_info {
353 int count;
354 u32 headers[MAX_VLANS];
355};
356
357struct dr_action_apply_attr {
358 u32 modify_index;
359 u16 modify_actions;
360 u32 decap_index;
361 u16 decap_actions;
362 u8 decap_with_vlan:1;
363 u64 final_icm_addr;
364 u32 flow_tag;
365 u32 ctr_id;
366 u16 gvmi;
367 u16 hit_gvmi;
368 u32 reformat_id;
369 u32 reformat_size;
370 struct dr_action_vlan_info vlans;
371};
372
373static int
374dr_action_reformat_to_action_type(enum mlx5dr_action_reformat_type reformat_type,
375 enum mlx5dr_action_type *action_type)
376{
377 switch (reformat_type) {
378 case DR_ACTION_REFORMAT_TYP_TNL_L2_TO_L2:
379 *action_type = DR_ACTION_TYP_TNL_L2_TO_L2;
380 break;
381 case DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L2:
382 *action_type = DR_ACTION_TYP_L2_TO_TNL_L2;
383 break;
384 case DR_ACTION_REFORMAT_TYP_TNL_L3_TO_L2:
385 *action_type = DR_ACTION_TYP_TNL_L3_TO_L2;
386 break;
387 case DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L3:
388 *action_type = DR_ACTION_TYP_L2_TO_TNL_L3;
389 break;
390 default:
391 return -EINVAL;
392 }
393
394 return 0;
395}
396
397static void dr_actions_init_next_ste(u8 **last_ste,
398 u32 *added_stes,
399 enum mlx5dr_ste_entry_type entry_type,
400 u16 gvmi)
401{
402 (*added_stes)++;
403 *last_ste += DR_STE_SIZE;
404 mlx5dr_ste_init(*last_ste, MLX5DR_STE_LU_TYPE_DONT_CARE, entry_type, gvmi);
405}
406
407static void dr_actions_apply_tx(struct mlx5dr_domain *dmn,
408 u8 *action_type_set,
409 u8 *last_ste,
410 struct dr_action_apply_attr *attr,
411 u32 *added_stes)
412{
413 bool encap = action_type_set[DR_ACTION_TYP_L2_TO_TNL_L2] ||
414 action_type_set[DR_ACTION_TYP_L2_TO_TNL_L3];
415
416 /* We want to make sure the modify header comes before L2
417 * encapsulation. The reason for that is that we support
418 * modify headers for outer headers only
419 */
420 if (action_type_set[DR_ACTION_TYP_MODIFY_HDR]) {
421 mlx5dr_ste_set_entry_type(last_ste, MLX5DR_STE_TYPE_MODIFY_PKT);
422 mlx5dr_ste_set_rewrite_actions(last_ste,
423 attr->modify_actions,
424 attr->modify_index);
425 }
426
427 if (action_type_set[DR_ACTION_TYP_PUSH_VLAN]) {
428 int i;
429
430 for (i = 0; i < attr->vlans.count; i++) {
431 if (i || action_type_set[DR_ACTION_TYP_MODIFY_HDR])
432 dr_actions_init_next_ste(&last_ste,
433 added_stes,
434 MLX5DR_STE_TYPE_TX,
435 attr->gvmi);
436
437 mlx5dr_ste_set_tx_push_vlan(last_ste,
438 attr->vlans.headers[i],
439 encap);
440 }
441 }
442
443 if (encap) {
444 /* Modify header and encapsulation require a different STEs.
445 * Since modify header STE format doesn't support encapsulation
446 * tunneling_action.
447 */
448 if (action_type_set[DR_ACTION_TYP_MODIFY_HDR] ||
449 action_type_set[DR_ACTION_TYP_PUSH_VLAN])
450 dr_actions_init_next_ste(&last_ste,
451 added_stes,
452 MLX5DR_STE_TYPE_TX,
453 attr->gvmi);
454
455 mlx5dr_ste_set_tx_encap(last_ste,
456 attr->reformat_id,
457 attr->reformat_size,
458 action_type_set[DR_ACTION_TYP_L2_TO_TNL_L3]);
459 /* Whenever prio_tag_required enabled, we can be sure that the
460 * previous table (ACL) already push vlan to our packet,
461 * And due to HW limitation we need to set this bit, otherwise
462 * push vlan + reformat will not work.
463 */
464 if (MLX5_CAP_GEN(dmn->mdev, prio_tag_required))
465 mlx5dr_ste_set_go_back_bit(last_ste);
466 }
467
468 if (action_type_set[DR_ACTION_TYP_CTR])
469 mlx5dr_ste_set_counter_id(last_ste, attr->ctr_id);
470}
471
472static void dr_actions_apply_rx(u8 *action_type_set,
473 u8 *last_ste,
474 struct dr_action_apply_attr *attr,
475 u32 *added_stes)
476{
477 if (action_type_set[DR_ACTION_TYP_CTR])
478 mlx5dr_ste_set_counter_id(last_ste, attr->ctr_id);
479
480 if (action_type_set[DR_ACTION_TYP_TNL_L3_TO_L2]) {
481 mlx5dr_ste_set_entry_type(last_ste, MLX5DR_STE_TYPE_MODIFY_PKT);
482 mlx5dr_ste_set_rx_decap_l3(last_ste, attr->decap_with_vlan);
483 mlx5dr_ste_set_rewrite_actions(last_ste,
484 attr->decap_actions,
485 attr->decap_index);
486 }
487
488 if (action_type_set[DR_ACTION_TYP_TNL_L2_TO_L2])
489 mlx5dr_ste_set_rx_decap(last_ste);
490
491 if (action_type_set[DR_ACTION_TYP_POP_VLAN]) {
492 int i;
493
494 for (i = 0; i < attr->vlans.count; i++) {
495 if (i ||
496 action_type_set[DR_ACTION_TYP_TNL_L2_TO_L2] ||
497 action_type_set[DR_ACTION_TYP_TNL_L3_TO_L2])
498 dr_actions_init_next_ste(&last_ste,
499 added_stes,
500 MLX5DR_STE_TYPE_RX,
501 attr->gvmi);
502
503 mlx5dr_ste_set_rx_pop_vlan(last_ste);
504 }
505 }
506
507 if (action_type_set[DR_ACTION_TYP_MODIFY_HDR]) {
508 if (mlx5dr_ste_get_entry_type(last_ste) == MLX5DR_STE_TYPE_MODIFY_PKT)
509 dr_actions_init_next_ste(&last_ste,
510 added_stes,
511 MLX5DR_STE_TYPE_MODIFY_PKT,
512 attr->gvmi);
513 else
514 mlx5dr_ste_set_entry_type(last_ste, MLX5DR_STE_TYPE_MODIFY_PKT);
515
516 mlx5dr_ste_set_rewrite_actions(last_ste,
517 attr->modify_actions,
518 attr->modify_index);
519 }
520
521 if (action_type_set[DR_ACTION_TYP_TAG]) {
522 if (mlx5dr_ste_get_entry_type(last_ste) == MLX5DR_STE_TYPE_MODIFY_PKT)
523 dr_actions_init_next_ste(&last_ste,
524 added_stes,
525 MLX5DR_STE_TYPE_RX,
526 attr->gvmi);
527
528 mlx5dr_ste_rx_set_flow_tag(last_ste, attr->flow_tag);
529 }
530}
531
532/* Apply the actions on the rule STE array starting from the last_ste.
533 * Actions might require more than one STE, new_num_stes will return
534 * the new size of the STEs array, rule with actions.
535 */
536static void dr_actions_apply(struct mlx5dr_domain *dmn,
537 enum mlx5dr_ste_entry_type ste_type,
538 u8 *action_type_set,
539 u8 *last_ste,
540 struct dr_action_apply_attr *attr,
541 u32 *new_num_stes)
542{
543 u32 added_stes = 0;
544
545 if (ste_type == MLX5DR_STE_TYPE_RX)
546 dr_actions_apply_rx(action_type_set, last_ste, attr, &added_stes);
547 else
548 dr_actions_apply_tx(dmn, action_type_set, last_ste, attr, &added_stes);
549
550 last_ste += added_stes * DR_STE_SIZE;
551 *new_num_stes += added_stes;
552
553 mlx5dr_ste_set_hit_gvmi(last_ste, attr->hit_gvmi);
554 mlx5dr_ste_set_hit_addr(last_ste, attr->final_icm_addr, 1);
555}
556
557static enum dr_action_domain
558dr_action_get_action_domain(enum mlx5dr_domain_type domain,
559 enum mlx5dr_ste_entry_type ste_type)
560{
561 switch (domain) {
562 case MLX5DR_DOMAIN_TYPE_NIC_RX:
563 return DR_ACTION_DOMAIN_NIC_INGRESS;
564 case MLX5DR_DOMAIN_TYPE_NIC_TX:
565 return DR_ACTION_DOMAIN_NIC_EGRESS;
566 case MLX5DR_DOMAIN_TYPE_FDB:
567 if (ste_type == MLX5DR_STE_TYPE_RX)
568 return DR_ACTION_DOMAIN_FDB_INGRESS;
569 return DR_ACTION_DOMAIN_FDB_EGRESS;
570 default:
571 WARN_ON(true);
572 return DR_ACTION_DOMAIN_MAX;
573 }
574}
575
576static
577int dr_action_validate_and_get_next_state(enum dr_action_domain action_domain,
578 u32 action_type,
579 u32 *state)
580{
581 u32 cur_state = *state;
582
583 /* Check action state machine is valid */
584 *state = next_action_state[action_domain][cur_state][action_type];
585
586 if (*state == DR_ACTION_STATE_ERR)
587 return -EOPNOTSUPP;
588
589 return 0;
590}
591
592static int dr_action_handle_cs_recalc(struct mlx5dr_domain *dmn,
593 struct mlx5dr_action *dest_action,
594 u64 *final_icm_addr)
595{
596 int ret;
597
598 switch (dest_action->action_type) {
599 case DR_ACTION_TYP_FT:
600 /* Allow destination flow table only if table is a terminating
601 * table, since there is an *assumption* that in such case FW
602 * will recalculate the CS.
603 */
604 if (dest_action->dest_tbl.is_fw_tbl) {
605 *final_icm_addr = dest_action->dest_tbl.fw_tbl.rx_icm_addr;
606 } else {
607 mlx5dr_dbg(dmn,
608 "Destination FT should be terminating when modify TTL is used\n");
609 return -EINVAL;
610 }
611 break;
612
613 case DR_ACTION_TYP_VPORT:
614 /* If destination is vport we will get the FW flow table
615 * that recalculates the CS and forwards to the vport.
616 */
617 ret = mlx5dr_domain_cache_get_recalc_cs_ft_addr(dest_action->vport.dmn,
618 dest_action->vport.num,
619 final_icm_addr);
620 if (ret) {
621 mlx5dr_err(dmn, "Failed to get FW cs recalc flow table\n");
622 return ret;
623 }
624 break;
625
626 default:
627 break;
628 }
629
630 return 0;
631}
632
633#define WITH_VLAN_NUM_HW_ACTIONS 6
634
635int mlx5dr_actions_build_ste_arr(struct mlx5dr_matcher *matcher,
636 struct mlx5dr_matcher_rx_tx *nic_matcher,
637 struct mlx5dr_action *actions[],
638 u32 num_actions,
639 u8 *ste_arr,
640 u32 *new_hw_ste_arr_sz)
641{
642 struct mlx5dr_domain_rx_tx *nic_dmn = nic_matcher->nic_tbl->nic_dmn;
643 bool rx_rule = nic_dmn->ste_type == MLX5DR_STE_TYPE_RX;
644 struct mlx5dr_domain *dmn = matcher->tbl->dmn;
645 u8 action_type_set[DR_ACTION_TYP_MAX] = {};
646 struct mlx5dr_action *dest_action = NULL;
647 u32 state = DR_ACTION_STATE_NO_ACTION;
648 struct dr_action_apply_attr attr = {};
649 enum dr_action_domain action_domain;
650 bool recalc_cs_required = false;
651 u8 *last_ste;
652 int i, ret;
653
654 attr.gvmi = dmn->info.caps.gvmi;
655 attr.hit_gvmi = dmn->info.caps.gvmi;
656 attr.final_icm_addr = nic_dmn->default_icm_addr;
657 action_domain = dr_action_get_action_domain(dmn->type, nic_dmn->ste_type);
658
659 for (i = 0; i < num_actions; i++) {
660 struct mlx5dr_action *action;
661 int max_actions_type = 1;
662 u32 action_type;
663
664 action = actions[i];
665 action_type = action->action_type;
666
667 switch (action_type) {
668 case DR_ACTION_TYP_DROP:
669 attr.final_icm_addr = nic_dmn->drop_icm_addr;
670 break;
671 case DR_ACTION_TYP_FT:
672 dest_action = action;
673 if (!action->dest_tbl.is_fw_tbl) {
674 if (action->dest_tbl.tbl->dmn != dmn) {
675 mlx5dr_dbg(dmn,
676 "Destination table belongs to a different domain\n");
677 goto out_invalid_arg;
678 }
679 if (action->dest_tbl.tbl->level <= matcher->tbl->level) {
680 mlx5dr_dbg(dmn,
681 "Destination table level should be higher than source table\n");
682 goto out_invalid_arg;
683 }
684 attr.final_icm_addr = rx_rule ?
685 action->dest_tbl.tbl->rx.s_anchor->chunk->icm_addr :
686 action->dest_tbl.tbl->tx.s_anchor->chunk->icm_addr;
687 } else {
688 struct mlx5dr_cmd_query_flow_table_details output;
689 int ret;
690
691 /* get the relevant addresses */
692 if (!action->dest_tbl.fw_tbl.rx_icm_addr) {
693 ret = mlx5dr_cmd_query_flow_table(action->dest_tbl.fw_tbl.mdev,
694 action->dest_tbl.fw_tbl.ft->type,
695 action->dest_tbl.fw_tbl.ft->id,
696 &output);
697 if (!ret) {
698 action->dest_tbl.fw_tbl.tx_icm_addr =
699 output.sw_owner_icm_root_1;
700 action->dest_tbl.fw_tbl.rx_icm_addr =
701 output.sw_owner_icm_root_0;
702 } else {
703 mlx5dr_dbg(dmn,
704 "Failed mlx5_cmd_query_flow_table ret: %d\n",
705 ret);
706 return ret;
707 }
708 }
709 attr.final_icm_addr = rx_rule ?
710 action->dest_tbl.fw_tbl.rx_icm_addr :
711 action->dest_tbl.fw_tbl.tx_icm_addr;
712 }
713 break;
714 case DR_ACTION_TYP_QP:
715 mlx5dr_info(dmn, "Domain doesn't support QP\n");
716 goto out_invalid_arg;
717 case DR_ACTION_TYP_CTR:
718 attr.ctr_id = action->ctr.ctr_id +
719 action->ctr.offeset;
720 break;
721 case DR_ACTION_TYP_TAG:
722 attr.flow_tag = action->flow_tag;
723 break;
724 case DR_ACTION_TYP_TNL_L2_TO_L2:
725 break;
726 case DR_ACTION_TYP_TNL_L3_TO_L2:
727 attr.decap_index = action->rewrite.index;
728 attr.decap_actions = action->rewrite.num_of_actions;
729 attr.decap_with_vlan =
730 attr.decap_actions == WITH_VLAN_NUM_HW_ACTIONS;
731 break;
732 case DR_ACTION_TYP_MODIFY_HDR:
733 attr.modify_index = action->rewrite.index;
734 attr.modify_actions = action->rewrite.num_of_actions;
735 recalc_cs_required = action->rewrite.modify_ttl;
736 break;
737 case DR_ACTION_TYP_L2_TO_TNL_L2:
738 case DR_ACTION_TYP_L2_TO_TNL_L3:
739 attr.reformat_size = action->reformat.reformat_size;
740 attr.reformat_id = action->reformat.reformat_id;
741 break;
742 case DR_ACTION_TYP_VPORT:
743 attr.hit_gvmi = action->vport.caps->vhca_gvmi;
744 dest_action = action;
745 if (rx_rule) {
746 /* Loopback on WIRE vport is not supported */
747 if (action->vport.num == WIRE_PORT)
748 goto out_invalid_arg;
749
750 attr.final_icm_addr = action->vport.caps->icm_address_rx;
751 } else {
752 attr.final_icm_addr = action->vport.caps->icm_address_tx;
753 }
754 break;
755 case DR_ACTION_TYP_POP_VLAN:
756 max_actions_type = MAX_VLANS;
757 attr.vlans.count++;
758 break;
759 case DR_ACTION_TYP_PUSH_VLAN:
760 max_actions_type = MAX_VLANS;
761 if (attr.vlans.count == MAX_VLANS)
762 return -EINVAL;
763
764 attr.vlans.headers[attr.vlans.count++] = action->push_vlan.vlan_hdr;
765 break;
766 default:
767 goto out_invalid_arg;
768 }
769
770 /* Check action duplication */
771 if (++action_type_set[action_type] > max_actions_type) {
772 mlx5dr_dbg(dmn, "Action type %d supports only max %d time(s)\n",
773 action_type, max_actions_type);
774 goto out_invalid_arg;
775 }
776
777 /* Check action state machine is valid */
778 if (dr_action_validate_and_get_next_state(action_domain,
779 action_type,
780 &state)) {
781 mlx5dr_dbg(dmn, "Invalid action sequence provided\n");
782 return -EOPNOTSUPP;
783 }
784 }
785
786 *new_hw_ste_arr_sz = nic_matcher->num_of_builders;
787 last_ste = ste_arr + DR_STE_SIZE * (nic_matcher->num_of_builders - 1);
788
789 /* Due to a HW bug, modifying TTL on RX flows will cause an incorrect
790 * checksum calculation. In this case we will use a FW table to
791 * recalculate.
792 */
793 if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB &&
794 rx_rule && recalc_cs_required && dest_action) {
795 ret = dr_action_handle_cs_recalc(dmn, dest_action, &attr.final_icm_addr);
796 if (ret) {
797 mlx5dr_dbg(dmn,
798 "Failed to handle checksum recalculation err %d\n",
799 ret);
800 return ret;
801 }
802 }
803
804 dr_actions_apply(dmn,
805 nic_dmn->ste_type,
806 action_type_set,
807 last_ste,
808 &attr,
809 new_hw_ste_arr_sz);
810
811 return 0;
812
813out_invalid_arg:
814 return -EINVAL;
815}
816
817#define CVLAN_ETHERTYPE 0x8100
818#define SVLAN_ETHERTYPE 0x88a8
819#define HDR_LEN_L2_ONLY 14
820#define HDR_LEN_L2_VLAN 18
821#define REWRITE_HW_ACTION_NUM 6
822
823static int dr_actions_l2_rewrite(struct mlx5dr_domain *dmn,
824 struct mlx5dr_action *action,
825 void *data, size_t data_sz)
826{
827 struct mlx5_ifc_l2_hdr_bits *l2_hdr = data;
828 u64 ops[REWRITE_HW_ACTION_NUM] = {};
829 u32 hdr_fld_4b;
830 u16 hdr_fld_2b;
831 u16 vlan_type;
832 bool vlan;
833 int i = 0;
834 int ret;
835
836 vlan = (data_sz != HDR_LEN_L2_ONLY);
837
838 /* dmac_47_16 */
839 MLX5_SET(dr_action_hw_set, ops + i,
840 opcode, MLX5DR_ACTION_MDFY_HW_OP_SET);
841 MLX5_SET(dr_action_hw_set, ops + i,
842 destination_length, 0);
843 MLX5_SET(dr_action_hw_set, ops + i,
844 destination_field_code, MLX5DR_ACTION_MDFY_HW_FLD_L2_0);
845 MLX5_SET(dr_action_hw_set, ops + i,
846 destination_left_shifter, 16);
847 hdr_fld_4b = MLX5_GET(l2_hdr, l2_hdr, dmac_47_16);
848 MLX5_SET(dr_action_hw_set, ops + i,
849 inline_data, hdr_fld_4b);
850 i++;
851
852 /* smac_47_16 */
853 MLX5_SET(dr_action_hw_set, ops + i,
854 opcode, MLX5DR_ACTION_MDFY_HW_OP_SET);
855 MLX5_SET(dr_action_hw_set, ops + i,
856 destination_length, 0);
857 MLX5_SET(dr_action_hw_set, ops + i,
858 destination_field_code, MLX5DR_ACTION_MDFY_HW_FLD_L2_1);
859 MLX5_SET(dr_action_hw_set, ops + i,
860 destination_left_shifter, 16);
861 hdr_fld_4b = (MLX5_GET(l2_hdr, l2_hdr, smac_31_0) >> 16 |
862 MLX5_GET(l2_hdr, l2_hdr, smac_47_32) << 16);
863 MLX5_SET(dr_action_hw_set, ops + i,
864 inline_data, hdr_fld_4b);
865 i++;
866
867 /* dmac_15_0 */
868 MLX5_SET(dr_action_hw_set, ops + i,
869 opcode, MLX5DR_ACTION_MDFY_HW_OP_SET);
870 MLX5_SET(dr_action_hw_set, ops + i,
871 destination_length, 16);
872 MLX5_SET(dr_action_hw_set, ops + i,
873 destination_field_code, MLX5DR_ACTION_MDFY_HW_FLD_L2_0);
874 MLX5_SET(dr_action_hw_set, ops + i,
875 destination_left_shifter, 0);
876 hdr_fld_2b = MLX5_GET(l2_hdr, l2_hdr, dmac_15_0);
877 MLX5_SET(dr_action_hw_set, ops + i,
878 inline_data, hdr_fld_2b);
879 i++;
880
881 /* ethertype + (optional) vlan */
882 MLX5_SET(dr_action_hw_set, ops + i,
883 opcode, MLX5DR_ACTION_MDFY_HW_OP_SET);
884 MLX5_SET(dr_action_hw_set, ops + i,
885 destination_field_code, MLX5DR_ACTION_MDFY_HW_FLD_L2_2);
886 MLX5_SET(dr_action_hw_set, ops + i,
887 destination_left_shifter, 32);
888 if (!vlan) {
889 hdr_fld_2b = MLX5_GET(l2_hdr, l2_hdr, ethertype);
890 MLX5_SET(dr_action_hw_set, ops + i, inline_data, hdr_fld_2b);
891 MLX5_SET(dr_action_hw_set, ops + i, destination_length, 16);
892 } else {
893 hdr_fld_2b = MLX5_GET(l2_hdr, l2_hdr, ethertype);
894 vlan_type = hdr_fld_2b == SVLAN_ETHERTYPE ? DR_STE_SVLAN : DR_STE_CVLAN;
895 hdr_fld_2b = MLX5_GET(l2_hdr, l2_hdr, vlan);
896 hdr_fld_4b = (vlan_type << 16) | hdr_fld_2b;
897 MLX5_SET(dr_action_hw_set, ops + i, inline_data, hdr_fld_4b);
898 MLX5_SET(dr_action_hw_set, ops + i, destination_length, 18);
899 }
900 i++;
901
902 /* smac_15_0 */
903 MLX5_SET(dr_action_hw_set, ops + i,
904 opcode, MLX5DR_ACTION_MDFY_HW_OP_SET);
905 MLX5_SET(dr_action_hw_set, ops + i,
906 destination_length, 16);
907 MLX5_SET(dr_action_hw_set, ops + i,
908 destination_field_code, MLX5DR_ACTION_MDFY_HW_FLD_L2_1);
909 MLX5_SET(dr_action_hw_set, ops + i,
910 destination_left_shifter, 0);
911 hdr_fld_2b = MLX5_GET(l2_hdr, l2_hdr, smac_31_0);
912 MLX5_SET(dr_action_hw_set, ops + i,
913 inline_data, hdr_fld_2b);
914 i++;
915
916 if (vlan) {
917 MLX5_SET(dr_action_hw_set, ops + i,
918 opcode, MLX5DR_ACTION_MDFY_HW_OP_SET);
919 hdr_fld_2b = MLX5_GET(l2_hdr, l2_hdr, vlan_type);
920 MLX5_SET(dr_action_hw_set, ops + i,
921 inline_data, hdr_fld_2b);
922 MLX5_SET(dr_action_hw_set, ops + i,
923 destination_length, 16);
924 MLX5_SET(dr_action_hw_set, ops + i,
925 destination_field_code, MLX5DR_ACTION_MDFY_HW_FLD_L2_2);
926 MLX5_SET(dr_action_hw_set, ops + i,
927 destination_left_shifter, 0);
928 i++;
929 }
930
931 action->rewrite.data = (void *)ops;
932 action->rewrite.num_of_actions = i;
933 action->rewrite.chunk->byte_size = i * sizeof(*ops);
934
935 ret = mlx5dr_send_postsend_action(dmn, action);
936 if (ret) {
937 mlx5dr_dbg(dmn, "Writing encapsulation action to ICM failed\n");
938 return ret;
939 }
940
941 return 0;
942}
943
944static struct mlx5dr_action *
945dr_action_create_generic(enum mlx5dr_action_type action_type)
946{
947 struct mlx5dr_action *action;
948
949 action = kzalloc(sizeof(*action), GFP_KERNEL);
950 if (!action)
951 return NULL;
952
953 action->action_type = action_type;
954 refcount_set(&action->refcount, 1);
955
956 return action;
957}
958
959struct mlx5dr_action *mlx5dr_action_create_drop(void)
960{
961 return dr_action_create_generic(DR_ACTION_TYP_DROP);
962}
963
964struct mlx5dr_action *
965mlx5dr_action_create_dest_table(struct mlx5dr_table *tbl)
966{
967 struct mlx5dr_action *action;
968
969 refcount_inc(&tbl->refcount);
970
971 action = dr_action_create_generic(DR_ACTION_TYP_FT);
972 if (!action)
973 goto dec_ref;
974
975 action->dest_tbl.tbl = tbl;
976
977 return action;
978
979dec_ref:
980 refcount_dec(&tbl->refcount);
981 return NULL;
982}
983
984struct mlx5dr_action *
985mlx5dr_create_action_dest_flow_fw_table(struct mlx5_flow_table *ft,
986 struct mlx5_core_dev *mdev)
987{
988 struct mlx5dr_action *action;
989
990 action = dr_action_create_generic(DR_ACTION_TYP_FT);
991 if (!action)
992 return NULL;
993
994 action->dest_tbl.is_fw_tbl = 1;
995 action->dest_tbl.fw_tbl.ft = ft;
996 action->dest_tbl.fw_tbl.mdev = mdev;
997
998 return action;
999}
1000
1001struct mlx5dr_action *
1002mlx5dr_action_create_flow_counter(u32 counter_id)
1003{
1004 struct mlx5dr_action *action;
1005
1006 action = dr_action_create_generic(DR_ACTION_TYP_CTR);
1007 if (!action)
1008 return NULL;
1009
1010 action->ctr.ctr_id = counter_id;
1011
1012 return action;
1013}
1014
1015struct mlx5dr_action *mlx5dr_action_create_tag(u32 tag_value)
1016{
1017 struct mlx5dr_action *action;
1018
1019 action = dr_action_create_generic(DR_ACTION_TYP_TAG);
1020 if (!action)
1021 return NULL;
1022
1023 action->flow_tag = tag_value & 0xffffff;
1024
1025 return action;
1026}
1027
1028static int
1029dr_action_verify_reformat_params(enum mlx5dr_action_type reformat_type,
1030 struct mlx5dr_domain *dmn,
1031 size_t data_sz,
1032 void *data)
1033{
1034 if ((!data && data_sz) || (data && !data_sz) || reformat_type >
1035 DR_ACTION_TYP_L2_TO_TNL_L3) {
1036 mlx5dr_dbg(dmn, "Invalid reformat parameter!\n");
1037 goto out_err;
1038 }
1039
1040 if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB)
1041 return 0;
1042
1043 if (dmn->type == MLX5DR_DOMAIN_TYPE_NIC_RX) {
1044 if (reformat_type != DR_ACTION_TYP_TNL_L2_TO_L2 &&
1045 reformat_type != DR_ACTION_TYP_TNL_L3_TO_L2) {
1046 mlx5dr_dbg(dmn, "Action reformat type not support on RX domain\n");
1047 goto out_err;
1048 }
1049 } else if (dmn->type == MLX5DR_DOMAIN_TYPE_NIC_TX) {
1050 if (reformat_type != DR_ACTION_TYP_L2_TO_TNL_L2 &&
1051 reformat_type != DR_ACTION_TYP_L2_TO_TNL_L3) {
1052 mlx5dr_dbg(dmn, "Action reformat type not support on TX domain\n");
1053 goto out_err;
1054 }
1055 }
1056
1057 return 0;
1058
1059out_err:
1060 return -EINVAL;
1061}
1062
1063#define ACTION_CACHE_LINE_SIZE 64
1064
1065static int
1066dr_action_create_reformat_action(struct mlx5dr_domain *dmn,
1067 size_t data_sz, void *data,
1068 struct mlx5dr_action *action)
1069{
1070 u32 reformat_id;
1071 int ret;
1072
1073 switch (action->action_type) {
1074 case DR_ACTION_TYP_L2_TO_TNL_L2:
1075 case DR_ACTION_TYP_L2_TO_TNL_L3:
1076 {
1077 enum mlx5dr_action_type rt;
1078
1079 if (action->action_type == DR_ACTION_TYP_L2_TO_TNL_L2)
1080 rt = MLX5_REFORMAT_TYPE_L2_TO_L2_TUNNEL;
1081 else
1082 rt = MLX5_REFORMAT_TYPE_L2_TO_L3_TUNNEL;
1083
1084 ret = mlx5dr_cmd_create_reformat_ctx(dmn->mdev, rt, data_sz, data,
1085 &reformat_id);
1086 if (ret)
1087 return ret;
1088
1089 action->reformat.reformat_id = reformat_id;
1090 action->reformat.reformat_size = data_sz;
1091 return 0;
1092 }
1093 case DR_ACTION_TYP_TNL_L2_TO_L2:
1094 {
1095 return 0;
1096 }
1097 case DR_ACTION_TYP_TNL_L3_TO_L2:
1098 {
1099 /* Only Ethernet frame is supported, with VLAN (18) or without (14) */
1100 if (data_sz != HDR_LEN_L2_ONLY && data_sz != HDR_LEN_L2_VLAN)
1101 return -EINVAL;
1102
1103 action->rewrite.chunk = mlx5dr_icm_alloc_chunk(dmn->action_icm_pool,
1104 DR_CHUNK_SIZE_8);
1105 if (!action->rewrite.chunk)
1106 return -ENOMEM;
1107
1108 action->rewrite.index = (action->rewrite.chunk->icm_addr -
1109 dmn->info.caps.hdr_modify_icm_addr) /
1110 ACTION_CACHE_LINE_SIZE;
1111
1112 ret = dr_actions_l2_rewrite(dmn, action, data, data_sz);
1113 if (ret) {
1114 mlx5dr_icm_free_chunk(action->rewrite.chunk);
1115 return ret;
1116 }
1117 return 0;
1118 }
1119 default:
1120 mlx5dr_info(dmn, "Reformat type is not supported %d\n", action->action_type);
1121 return -EINVAL;
1122 }
1123}
1124
1125struct mlx5dr_action *mlx5dr_action_create_pop_vlan(void)
1126{
1127 return dr_action_create_generic(DR_ACTION_TYP_POP_VLAN);
1128}
1129
1130struct mlx5dr_action *mlx5dr_action_create_push_vlan(struct mlx5dr_domain *dmn,
1131 __be32 vlan_hdr)
1132{
1133 u32 vlan_hdr_h = ntohl(vlan_hdr);
1134 u16 ethertype = vlan_hdr_h >> 16;
1135 struct mlx5dr_action *action;
1136
1137 if (ethertype != SVLAN_ETHERTYPE && ethertype != CVLAN_ETHERTYPE) {
1138 mlx5dr_dbg(dmn, "Invalid vlan ethertype\n");
1139 return NULL;
1140 }
1141
1142 action = dr_action_create_generic(DR_ACTION_TYP_PUSH_VLAN);
1143 if (!action)
1144 return NULL;
1145
1146 action->push_vlan.vlan_hdr = vlan_hdr_h;
1147 return action;
1148}
1149
1150struct mlx5dr_action *
1151mlx5dr_action_create_packet_reformat(struct mlx5dr_domain *dmn,
1152 enum mlx5dr_action_reformat_type reformat_type,
1153 size_t data_sz,
1154 void *data)
1155{
1156 enum mlx5dr_action_type action_type;
1157 struct mlx5dr_action *action;
1158 int ret;
1159
1160 refcount_inc(&dmn->refcount);
1161
1162 /* General checks */
1163 ret = dr_action_reformat_to_action_type(reformat_type, &action_type);
1164 if (ret) {
1165 mlx5dr_dbg(dmn, "Invalid reformat_type provided\n");
1166 goto dec_ref;
1167 }
1168
1169 ret = dr_action_verify_reformat_params(action_type, dmn, data_sz, data);
1170 if (ret)
1171 goto dec_ref;
1172
1173 action = dr_action_create_generic(action_type);
1174 if (!action)
1175 goto dec_ref;
1176
1177 action->reformat.dmn = dmn;
1178
1179 ret = dr_action_create_reformat_action(dmn,
1180 data_sz,
1181 data,
1182 action);
1183 if (ret) {
1184 mlx5dr_dbg(dmn, "Failed creating reformat action %d\n", ret);
1185 goto free_action;
1186 }
1187
1188 return action;
1189
1190free_action:
1191 kfree(action);
1192dec_ref:
1193 refcount_dec(&dmn->refcount);
1194 return NULL;
1195}
1196
1197static const struct dr_action_modify_field_conv *
1198dr_action_modify_get_hw_info(u16 sw_field)
1199{
1200 const struct dr_action_modify_field_conv *hw_action_info;
1201
1202 if (sw_field >= ARRAY_SIZE(dr_action_conv_arr))
1203 goto not_found;
1204
1205 hw_action_info = &dr_action_conv_arr[sw_field];
1206 if (!hw_action_info->end && !hw_action_info->start)
1207 goto not_found;
1208
1209 return hw_action_info;
1210
1211not_found:
1212 return NULL;
1213}
1214
1215static int
1216dr_action_modify_sw_to_hw(struct mlx5dr_domain *dmn,
1217 __be64 *sw_action,
1218 __be64 *hw_action,
1219 const struct dr_action_modify_field_conv **ret_hw_info)
1220{
1221 const struct dr_action_modify_field_conv *hw_action_info;
1222 u8 offset, length, max_length, action;
1223 u16 sw_field;
1224 u8 hw_opcode;
1225 u32 data;
1226
1227 /* Get SW modify action data */
1228 action = MLX5_GET(set_action_in, sw_action, action_type);
1229 length = MLX5_GET(set_action_in, sw_action, length);
1230 offset = MLX5_GET(set_action_in, sw_action, offset);
1231 sw_field = MLX5_GET(set_action_in, sw_action, field);
1232 data = MLX5_GET(set_action_in, sw_action, data);
1233
1234 /* Convert SW data to HW modify action format */
1235 hw_action_info = dr_action_modify_get_hw_info(sw_field);
1236 if (!hw_action_info) {
1237 mlx5dr_dbg(dmn, "Modify action invalid field given\n");
1238 return -EINVAL;
1239 }
1240
1241 max_length = hw_action_info->end - hw_action_info->start + 1;
1242
1243 switch (action) {
1244 case MLX5_ACTION_TYPE_SET:
1245 hw_opcode = MLX5DR_ACTION_MDFY_HW_OP_SET;
1246 /* PRM defines that length zero specific length of 32bits */
1247 if (!length)
1248 length = 32;
1249
1250 if (length + offset > max_length) {
1251 mlx5dr_dbg(dmn, "Modify action length + offset exceeds limit\n");
1252 return -EINVAL;
1253 }
1254 break;
1255
1256 case MLX5_ACTION_TYPE_ADD:
1257 hw_opcode = MLX5DR_ACTION_MDFY_HW_OP_ADD;
1258 offset = 0;
1259 length = max_length;
1260 break;
1261
1262 default:
1263 mlx5dr_info(dmn, "Unsupported action_type for modify action\n");
1264 return -EOPNOTSUPP;
1265 }
1266
1267 MLX5_SET(dr_action_hw_set, hw_action, opcode, hw_opcode);
1268
1269 MLX5_SET(dr_action_hw_set, hw_action, destination_field_code,
1270 hw_action_info->hw_field);
1271
1272 MLX5_SET(dr_action_hw_set, hw_action, destination_left_shifter,
1273 hw_action_info->start + offset);
1274
1275 MLX5_SET(dr_action_hw_set, hw_action, destination_length,
1276 length == 32 ? 0 : length);
1277
1278 MLX5_SET(dr_action_hw_set, hw_action, inline_data, data);
1279
1280 *ret_hw_info = hw_action_info;
1281
1282 return 0;
1283}
1284
1285static int
1286dr_action_modify_check_field_limitation(struct mlx5dr_domain *dmn,
1287 const __be64 *sw_action)
1288{
1289 u16 sw_field;
1290 u8 action;
1291
1292 sw_field = MLX5_GET(set_action_in, sw_action, field);
1293 action = MLX5_GET(set_action_in, sw_action, action_type);
1294
1295 /* Check if SW field is supported in current domain (RX/TX) */
1296 if (action == MLX5_ACTION_TYPE_SET) {
1297 if (sw_field == MLX5_ACTION_IN_FIELD_METADATA_REG_A) {
1298 if (dmn->type != MLX5DR_DOMAIN_TYPE_NIC_TX) {
1299 mlx5dr_dbg(dmn, "Unsupported field %d for RX/FDB set action\n",
1300 sw_field);
1301 return -EINVAL;
1302 }
1303 }
1304
1305 if (sw_field == MLX5_ACTION_IN_FIELD_METADATA_REG_B) {
1306 if (dmn->type != MLX5DR_DOMAIN_TYPE_NIC_RX) {
1307 mlx5dr_dbg(dmn, "Unsupported field %d for TX/FDB set action\n",
1308 sw_field);
1309 return -EINVAL;
1310 }
1311 }
1312 } else if (action == MLX5_ACTION_TYPE_ADD) {
1313 if (sw_field != MLX5_ACTION_IN_FIELD_OUT_IP_TTL &&
1314 sw_field != MLX5_ACTION_IN_FIELD_OUT_IPV6_HOPLIMIT &&
1315 sw_field != MLX5_ACTION_IN_FIELD_OUT_TCP_SEQ_NUM &&
1316 sw_field != MLX5_ACTION_IN_FIELD_OUT_TCP_ACK_NUM) {
1317 mlx5dr_dbg(dmn, "Unsupported field %d for add action\n", sw_field);
1318 return -EINVAL;
1319 }
1320 } else {
1321 mlx5dr_info(dmn, "Unsupported action %d modify action\n", action);
1322 return -EOPNOTSUPP;
1323 }
1324
1325 return 0;
1326}
1327
1328static bool
1329dr_action_modify_check_is_ttl_modify(const u64 *sw_action)
1330{
1331 u16 sw_field = MLX5_GET(set_action_in, sw_action, field);
1332
1333 return sw_field == MLX5_ACTION_IN_FIELD_OUT_IP_TTL;
1334}
1335
1336static int dr_actions_convert_modify_header(struct mlx5dr_domain *dmn,
1337 u32 max_hw_actions,
1338 u32 num_sw_actions,
1339 __be64 sw_actions[],
1340 __be64 hw_actions[],
1341 u32 *num_hw_actions,
1342 bool *modify_ttl)
1343{
1344 const struct dr_action_modify_field_conv *hw_action_info;
1345 u16 hw_field = MLX5DR_ACTION_MDFY_HW_FLD_RESERVED;
1346 u32 l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_NONE;
1347 u32 l4_type = MLX5DR_ACTION_MDFY_HW_HDR_L4_NONE;
1348 int ret, i, hw_idx = 0;
1349 __be64 *sw_action;
1350 __be64 hw_action;
1351
1352 *modify_ttl = false;
1353
1354 for (i = 0; i < num_sw_actions; i++) {
1355 sw_action = &sw_actions[i];
1356
1357 ret = dr_action_modify_check_field_limitation(dmn, sw_action);
1358 if (ret)
1359 return ret;
1360
1361 if (!(*modify_ttl))
1362 *modify_ttl = dr_action_modify_check_is_ttl_modify(sw_action);
1363
1364 /* Convert SW action to HW action */
1365 ret = dr_action_modify_sw_to_hw(dmn,
1366 sw_action,
1367 &hw_action,
1368 &hw_action_info);
1369 if (ret)
1370 return ret;
1371
1372 /* Due to a HW limitation we cannot modify 2 different L3 types */
1373 if (l3_type && hw_action_info->l3_type &&
1374 hw_action_info->l3_type != l3_type) {
1375 mlx5dr_dbg(dmn, "Action list can't support two different L3 types\n");
1376 return -EINVAL;
1377 }
1378 if (hw_action_info->l3_type)
1379 l3_type = hw_action_info->l3_type;
1380
1381 /* Due to a HW limitation we cannot modify two different L4 types */
1382 if (l4_type && hw_action_info->l4_type &&
1383 hw_action_info->l4_type != l4_type) {
1384 mlx5dr_dbg(dmn, "Action list can't support two different L4 types\n");
1385 return -EINVAL;
1386 }
1387 if (hw_action_info->l4_type)
1388 l4_type = hw_action_info->l4_type;
1389
1390 /* HW reads and executes two actions at once this means we
1391 * need to create a gap if two actions access the same field
1392 */
1393 if ((hw_idx % 2) && hw_field == hw_action_info->hw_field) {
1394 /* Check if after gap insertion the total number of HW
1395 * modify actions doesn't exceeds the limit
1396 */
1397 hw_idx++;
1398 if ((num_sw_actions + hw_idx - i) >= max_hw_actions) {
1399 mlx5dr_dbg(dmn, "Modify header action number exceeds HW limit\n");
1400 return -EINVAL;
1401 }
1402 }
1403 hw_field = hw_action_info->hw_field;
1404
1405 hw_actions[hw_idx] = hw_action;
1406 hw_idx++;
1407 }
1408
1409 *num_hw_actions = hw_idx;
1410
1411 return 0;
1412}
1413
1414static int dr_action_create_modify_action(struct mlx5dr_domain *dmn,
1415 size_t actions_sz,
1416 __be64 actions[],
1417 struct mlx5dr_action *action)
1418{
1419 struct mlx5dr_icm_chunk *chunk;
1420 u32 max_hw_actions;
1421 u32 num_hw_actions;
1422 u32 num_sw_actions;
1423 __be64 *hw_actions;
1424 bool modify_ttl;
1425 int ret;
1426
1427 num_sw_actions = actions_sz / DR_MODIFY_ACTION_SIZE;
1428 max_hw_actions = mlx5dr_icm_pool_chunk_size_to_entries(DR_CHUNK_SIZE_16);
1429
1430 if (num_sw_actions > max_hw_actions) {
1431 mlx5dr_dbg(dmn, "Max number of actions %d exceeds limit %d\n",
1432 num_sw_actions, max_hw_actions);
1433 return -EINVAL;
1434 }
1435
1436 chunk = mlx5dr_icm_alloc_chunk(dmn->action_icm_pool, DR_CHUNK_SIZE_16);
1437 if (!chunk)
1438 return -ENOMEM;
1439
1440 hw_actions = kcalloc(1, max_hw_actions * DR_MODIFY_ACTION_SIZE, GFP_KERNEL);
1441 if (!hw_actions) {
1442 ret = -ENOMEM;
1443 goto free_chunk;
1444 }
1445
1446 ret = dr_actions_convert_modify_header(dmn,
1447 max_hw_actions,
1448 num_sw_actions,
1449 actions,
1450 hw_actions,
1451 &num_hw_actions,
1452 &modify_ttl);
1453 if (ret)
1454 goto free_hw_actions;
1455
1456 action->rewrite.chunk = chunk;
1457 action->rewrite.modify_ttl = modify_ttl;
1458 action->rewrite.data = (u8 *)hw_actions;
1459 action->rewrite.num_of_actions = num_hw_actions;
1460 action->rewrite.index = (chunk->icm_addr -
1461 dmn->info.caps.hdr_modify_icm_addr) /
1462 ACTION_CACHE_LINE_SIZE;
1463
1464 ret = mlx5dr_send_postsend_action(dmn, action);
1465 if (ret)
1466 goto free_hw_actions;
1467
1468 return 0;
1469
1470free_hw_actions:
1471 kfree(hw_actions);
1472free_chunk:
1473 mlx5dr_icm_free_chunk(chunk);
1474 return ret;
1475}
1476
1477struct mlx5dr_action *
1478mlx5dr_action_create_modify_header(struct mlx5dr_domain *dmn,
1479 u32 flags,
1480 size_t actions_sz,
1481 __be64 actions[])
1482{
1483 struct mlx5dr_action *action;
1484 int ret = 0;
1485
1486 refcount_inc(&dmn->refcount);
1487
1488 if (actions_sz % DR_MODIFY_ACTION_SIZE) {
1489 mlx5dr_dbg(dmn, "Invalid modify actions size provided\n");
1490 goto dec_ref;
1491 }
1492
1493 action = dr_action_create_generic(DR_ACTION_TYP_MODIFY_HDR);
1494 if (!action)
1495 goto dec_ref;
1496
1497 action->rewrite.dmn = dmn;
1498
1499 ret = dr_action_create_modify_action(dmn,
1500 actions_sz,
1501 actions,
1502 action);
1503 if (ret) {
1504 mlx5dr_dbg(dmn, "Failed creating modify header action %d\n", ret);
1505 goto free_action;
1506 }
1507
1508 return action;
1509
1510free_action:
1511 kfree(action);
1512dec_ref:
1513 refcount_dec(&dmn->refcount);
1514 return NULL;
1515}
1516
1517struct mlx5dr_action *
1518mlx5dr_action_create_dest_vport(struct mlx5dr_domain *dmn,
1519 u32 vport, u8 vhca_id_valid,
1520 u16 vhca_id)
1521{
1522 struct mlx5dr_cmd_vport_cap *vport_cap;
1523 struct mlx5dr_domain *vport_dmn;
1524 struct mlx5dr_action *action;
1525 u8 peer_vport;
1526
1527 peer_vport = vhca_id_valid && (vhca_id != dmn->info.caps.gvmi);
1528 vport_dmn = peer_vport ? dmn->peer_dmn : dmn;
1529 if (!vport_dmn) {
1530 mlx5dr_dbg(dmn, "No peer vport domain for given vhca_id\n");
1531 return NULL;
1532 }
1533
1534 if (vport_dmn->type != MLX5DR_DOMAIN_TYPE_FDB) {
1535 mlx5dr_dbg(dmn, "Domain doesn't support vport actions\n");
1536 return NULL;
1537 }
1538
1539 vport_cap = mlx5dr_get_vport_cap(&vport_dmn->info.caps, vport);
1540 if (!vport_cap) {
1541 mlx5dr_dbg(dmn, "Failed to get vport %d caps\n", vport);
1542 return NULL;
1543 }
1544
1545 action = dr_action_create_generic(DR_ACTION_TYP_VPORT);
1546 if (!action)
1547 return NULL;
1548
1549 action->vport.dmn = vport_dmn;
1550 action->vport.caps = vport_cap;
1551
1552 return action;
1553}
1554
1555int mlx5dr_action_destroy(struct mlx5dr_action *action)
1556{
1557 if (refcount_read(&action->refcount) > 1)
1558 return -EBUSY;
1559
1560 switch (action->action_type) {
1561 case DR_ACTION_TYP_FT:
1562 if (!action->dest_tbl.is_fw_tbl)
1563 refcount_dec(&action->dest_tbl.tbl->refcount);
1564 break;
1565 case DR_ACTION_TYP_TNL_L2_TO_L2:
1566 refcount_dec(&action->reformat.dmn->refcount);
1567 break;
1568 case DR_ACTION_TYP_TNL_L3_TO_L2:
1569 mlx5dr_icm_free_chunk(action->rewrite.chunk);
1570 refcount_dec(&action->reformat.dmn->refcount);
1571 break;
1572 case DR_ACTION_TYP_L2_TO_TNL_L2:
1573 case DR_ACTION_TYP_L2_TO_TNL_L3:
1574 mlx5dr_cmd_destroy_reformat_ctx((action->reformat.dmn)->mdev,
1575 action->reformat.reformat_id);
1576 refcount_dec(&action->reformat.dmn->refcount);
1577 break;
1578 case DR_ACTION_TYP_MODIFY_HDR:
1579 mlx5dr_icm_free_chunk(action->rewrite.chunk);
1580 refcount_dec(&action->rewrite.dmn->refcount);
1581 break;
1582 default:
1583 break;
1584 }
1585
1586 kfree(action);
1587 return 0;
1588}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_cmd.c+480−0
@@ -0,0 +1,480 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2/* Copyright (c) 2019 Mellanox Technologies. */
3
4#include "dr_types.h"
5
6int mlx5dr_cmd_query_esw_vport_context(struct mlx5_core_dev *mdev,
7 bool other_vport,
8 u16 vport_number,
9 u64 *icm_address_rx,
10 u64 *icm_address_tx)
11{
12 u32 out[MLX5_ST_SZ_DW(query_esw_vport_context_out)] = {};
13 u32 in[MLX5_ST_SZ_DW(query_esw_vport_context_in)] = {};
14 int err;
15
16 MLX5_SET(query_esw_vport_context_in, in, opcode,
17 MLX5_CMD_OP_QUERY_ESW_VPORT_CONTEXT);
18 MLX5_SET(query_esw_vport_context_in, in, other_vport, other_vport);
19 MLX5_SET(query_esw_vport_context_in, in, vport_number, vport_number);
20
21 err = mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
22 if (err)
23 return err;
24
25 *icm_address_rx =
26 MLX5_GET64(query_esw_vport_context_out, out,
27 esw_vport_context.sw_steering_vport_icm_address_rx);
28 *icm_address_tx =
29 MLX5_GET64(query_esw_vport_context_out, out,
30 esw_vport_context.sw_steering_vport_icm_address_tx);
31 return 0;
32}
33
34int mlx5dr_cmd_query_gvmi(struct mlx5_core_dev *mdev, bool other_vport,
35 u16 vport_number, u16 *gvmi)
36{
37 u32 in[MLX5_ST_SZ_DW(query_hca_cap_in)] = {};
38 int out_size;
39 void *out;
40 int err;
41
42 out_size = MLX5_ST_SZ_BYTES(query_hca_cap_out);
43 out = kzalloc(out_size, GFP_KERNEL);
44 if (!out)
45 return -ENOMEM;
46
47 MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);
48 MLX5_SET(query_hca_cap_in, in, other_function, other_vport);
49 MLX5_SET(query_hca_cap_in, in, function_id, vport_number);
50 MLX5_SET(query_hca_cap_in, in, op_mod,
51 MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1 |
52 HCA_CAP_OPMOD_GET_CUR);
53
54 err = mlx5_cmd_exec(mdev, in, sizeof(in), out, out_size);
55 if (err) {
56 kfree(out);
57 return err;
58 }
59
60 *gvmi = MLX5_GET(query_hca_cap_out, out, capability.cmd_hca_cap.vhca_id);
61
62 kfree(out);
63 return 0;
64}
65
66int mlx5dr_cmd_query_esw_caps(struct mlx5_core_dev *mdev,
67 struct mlx5dr_esw_caps *caps)
68{
69 caps->drop_icm_address_rx =
70 MLX5_CAP64_ESW_FLOWTABLE(mdev,
71 sw_steering_fdb_action_drop_icm_address_rx);
72 caps->drop_icm_address_tx =
73 MLX5_CAP64_ESW_FLOWTABLE(mdev,
74 sw_steering_fdb_action_drop_icm_address_tx);
75 caps->uplink_icm_address_rx =
76 MLX5_CAP64_ESW_FLOWTABLE(mdev,
77 sw_steering_uplink_icm_address_rx);
78 caps->uplink_icm_address_tx =
79 MLX5_CAP64_ESW_FLOWTABLE(mdev,
80 sw_steering_uplink_icm_address_tx);
81 caps->sw_owner =
82 MLX5_CAP_ESW_FLOWTABLE_FDB(mdev,
83 sw_owner);
84
85 return 0;
86}
87
88int mlx5dr_cmd_query_device(struct mlx5_core_dev *mdev,
89 struct mlx5dr_cmd_caps *caps)
90{
91 caps->prio_tag_required = MLX5_CAP_GEN(mdev, prio_tag_required);
92 caps->eswitch_manager = MLX5_CAP_GEN(mdev, eswitch_manager);
93 caps->gvmi = MLX5_CAP_GEN(mdev, vhca_id);
94 caps->flex_protocols = MLX5_CAP_GEN(mdev, flex_parser_protocols);
95
96 if (mlx5dr_matcher_supp_flex_parser_icmp_v4(caps)) {
97 caps->flex_parser_id_icmp_dw0 = MLX5_CAP_GEN(mdev, flex_parser_id_icmp_dw0);
98 caps->flex_parser_id_icmp_dw1 = MLX5_CAP_GEN(mdev, flex_parser_id_icmp_dw1);
99 }
100
101 if (mlx5dr_matcher_supp_flex_parser_icmp_v6(caps)) {
102 caps->flex_parser_id_icmpv6_dw0 =
103 MLX5_CAP_GEN(mdev, flex_parser_id_icmpv6_dw0);
104 caps->flex_parser_id_icmpv6_dw1 =
105 MLX5_CAP_GEN(mdev, flex_parser_id_icmpv6_dw1);
106 }
107
108 caps->nic_rx_drop_address =
109 MLX5_CAP64_FLOWTABLE(mdev, sw_steering_nic_rx_action_drop_icm_address);
110 caps->nic_tx_drop_address =
111 MLX5_CAP64_FLOWTABLE(mdev, sw_steering_nic_tx_action_drop_icm_address);
112 caps->nic_tx_allow_address =
113 MLX5_CAP64_FLOWTABLE(mdev, sw_steering_nic_tx_action_allow_icm_address);
114
115 caps->rx_sw_owner = MLX5_CAP_FLOWTABLE_NIC_RX(mdev, sw_owner);
116 caps->max_ft_level = MLX5_CAP_FLOWTABLE_NIC_RX(mdev, max_ft_level);
117
118 caps->tx_sw_owner = MLX5_CAP_FLOWTABLE_NIC_TX(mdev, sw_owner);
119
120 caps->log_icm_size = MLX5_CAP_DEV_MEM(mdev, log_steering_sw_icm_size);
121 caps->hdr_modify_icm_addr =
122 MLX5_CAP64_DEV_MEM(mdev, header_modify_sw_icm_start_address);
123
124 caps->roce_min_src_udp = MLX5_CAP_ROCE(mdev, r_roce_min_src_udp_port);
125
126 return 0;
127}
128
129int mlx5dr_cmd_query_flow_table(struct mlx5_core_dev *dev,
130 enum fs_flow_table_type type,
131 u32 table_id,
132 struct mlx5dr_cmd_query_flow_table_details *output)
133{
134 u32 out[MLX5_ST_SZ_DW(query_flow_table_out)] = {};
135 u32 in[MLX5_ST_SZ_DW(query_flow_table_in)] = {};
136 int err;
137
138 MLX5_SET(query_flow_table_in, in, opcode,
139 MLX5_CMD_OP_QUERY_FLOW_TABLE);
140
141 MLX5_SET(query_flow_table_in, in, table_type, type);
142 MLX5_SET(query_flow_table_in, in, table_id, table_id);
143
144 err = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
145 if (err)
146 return err;
147
148 output->status = MLX5_GET(query_flow_table_out, out, status);
149 output->level = MLX5_GET(query_flow_table_out, out, flow_table_context.level);
150
151 output->sw_owner_icm_root_1 = MLX5_GET64(query_flow_table_out, out,
152 flow_table_context.sw_owner_icm_root_1);
153 output->sw_owner_icm_root_0 = MLX5_GET64(query_flow_table_out, out,
154 flow_table_context.sw_owner_icm_root_0);
155
156 return 0;
157}
158
159int mlx5dr_cmd_sync_steering(struct mlx5_core_dev *mdev)
160{
161 u32 out[MLX5_ST_SZ_DW(sync_steering_out)] = {};
162 u32 in[MLX5_ST_SZ_DW(sync_steering_in)] = {};
163
164 MLX5_SET(sync_steering_in, in, opcode, MLX5_CMD_OP_SYNC_STEERING);
165
166 return mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
167}
168
169int mlx5dr_cmd_set_fte_modify_and_vport(struct mlx5_core_dev *mdev,
170 u32 table_type,
171 u32 table_id,
172 u32 group_id,
173 u32 modify_header_id,
174 u32 vport_id)
175{
176 u32 out[MLX5_ST_SZ_DW(set_fte_out)] = {};
177 void *in_flow_context;
178 unsigned int inlen;
179 void *in_dests;
180 u32 *in;
181 int err;
182
183 inlen = MLX5_ST_SZ_BYTES(set_fte_in) +
184 1 * MLX5_ST_SZ_BYTES(dest_format_struct); /* One destination only */
185
186 in = kvzalloc(inlen, GFP_KERNEL);
187 if (!in)
188 return -ENOMEM;
189
190 MLX5_SET(set_fte_in, in, opcode, MLX5_CMD_OP_SET_FLOW_TABLE_ENTRY);
191 MLX5_SET(set_fte_in, in, table_type, table_type);
192 MLX5_SET(set_fte_in, in, table_id, table_id);
193
194 in_flow_context = MLX5_ADDR_OF(set_fte_in, in, flow_context);
195 MLX5_SET(flow_context, in_flow_context, group_id, group_id);
196 MLX5_SET(flow_context, in_flow_context, modify_header_id, modify_header_id);
197 MLX5_SET(flow_context, in_flow_context, destination_list_size, 1);
198 MLX5_SET(flow_context, in_flow_context, action,
199 MLX5_FLOW_CONTEXT_ACTION_FWD_DEST |
200 MLX5_FLOW_CONTEXT_ACTION_MOD_HDR);
201
202 in_dests = MLX5_ADDR_OF(flow_context, in_flow_context, destination);
203 MLX5_SET(dest_format_struct, in_dests, destination_type,
204 MLX5_FLOW_DESTINATION_TYPE_VPORT);
205 MLX5_SET(dest_format_struct, in_dests, destination_id, vport_id);
206
207 err = mlx5_cmd_exec(mdev, in, inlen, out, sizeof(out));
208 kvfree(in);
209
210 return err;
211}
212
213int mlx5dr_cmd_del_flow_table_entry(struct mlx5_core_dev *mdev,
214 u32 table_type,
215 u32 table_id)
216{
217 u32 out[MLX5_ST_SZ_DW(delete_fte_out)] = {};
218 u32 in[MLX5_ST_SZ_DW(delete_fte_in)] = {};
219
220 MLX5_SET(delete_fte_in, in, opcode, MLX5_CMD_OP_DELETE_FLOW_TABLE_ENTRY);
221 MLX5_SET(delete_fte_in, in, table_type, table_type);
222 MLX5_SET(delete_fte_in, in, table_id, table_id);
223
224 return mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
225}
226
227int mlx5dr_cmd_alloc_modify_header(struct mlx5_core_dev *mdev,
228 u32 table_type,
229 u8 num_of_actions,
230 u64 *actions,
231 u32 *modify_header_id)
232{
233 u32 out[MLX5_ST_SZ_DW(alloc_modify_header_context_out)] = {};
234 void *p_actions;
235 u32 inlen;
236 u32 *in;
237 int err;
238
239 inlen = MLX5_ST_SZ_BYTES(alloc_modify_header_context_in) +
240 num_of_actions * sizeof(u64);
241 in = kvzalloc(inlen, GFP_KERNEL);
242 if (!in)
243 return -ENOMEM;
244
245 MLX5_SET(alloc_modify_header_context_in, in, opcode,
246 MLX5_CMD_OP_ALLOC_MODIFY_HEADER_CONTEXT);
247 MLX5_SET(alloc_modify_header_context_in, in, table_type, table_type);
248 MLX5_SET(alloc_modify_header_context_in, in, num_of_actions, num_of_actions);
249 p_actions = MLX5_ADDR_OF(alloc_modify_header_context_in, in, actions);
250 memcpy(p_actions, actions, num_of_actions * sizeof(u64));
251
252 err = mlx5_cmd_exec(mdev, in, inlen, out, sizeof(out));
253 if (err)
254 goto out;
255
256 *modify_header_id = MLX5_GET(alloc_modify_header_context_out, out,
257 modify_header_id);
258out:
259 kvfree(in);
260 return err;
261}
262
263int mlx5dr_cmd_dealloc_modify_header(struct mlx5_core_dev *mdev,
264 u32 modify_header_id)
265{
266 u32 out[MLX5_ST_SZ_DW(dealloc_modify_header_context_out)] = {};
267 u32 in[MLX5_ST_SZ_DW(dealloc_modify_header_context_in)] = {};
268
269 MLX5_SET(dealloc_modify_header_context_in, in, opcode,
270 MLX5_CMD_OP_DEALLOC_MODIFY_HEADER_CONTEXT);
271 MLX5_SET(dealloc_modify_header_context_in, in, modify_header_id,
272 modify_header_id);
273
274 return mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
275}
276
277int mlx5dr_cmd_create_empty_flow_group(struct mlx5_core_dev *mdev,
278 u32 table_type,
279 u32 table_id,
280 u32 *group_id)
281{
282 u32 out[MLX5_ST_SZ_DW(create_flow_group_out)] = {};
283 int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
284 u32 *in;
285 int err;
286
287 in = kzalloc(inlen, GFP_KERNEL);
288 if (!in)
289 return -ENOMEM;
290
291 MLX5_SET(create_flow_group_in, in, opcode, MLX5_CMD_OP_CREATE_FLOW_GROUP);
292 MLX5_SET(create_flow_group_in, in, table_type, table_type);
293 MLX5_SET(create_flow_group_in, in, table_id, table_id);
294
295 err = mlx5_cmd_exec(mdev, in, inlen, out, sizeof(out));
296 if (err)
297 goto out;
298
299 *group_id = MLX5_GET(create_flow_group_out, out, group_id);
300
301out:
302 kfree(in);
303 return err;
304}
305
306int mlx5dr_cmd_destroy_flow_group(struct mlx5_core_dev *mdev,
307 u32 table_type,
308 u32 table_id,
309 u32 group_id)
310{
311 u32 in[MLX5_ST_SZ_DW(destroy_flow_group_in)] = {};
312 u32 out[MLX5_ST_SZ_DW(destroy_flow_group_out)] = {};
313
314 MLX5_SET(create_flow_group_in, in, opcode, MLX5_CMD_OP_DESTROY_FLOW_GROUP);
315 MLX5_SET(destroy_flow_group_in, in, table_type, table_type);
316 MLX5_SET(destroy_flow_group_in, in, table_id, table_id);
317 MLX5_SET(destroy_flow_group_in, in, group_id, group_id);
318
319 return mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
320}
321
322int mlx5dr_cmd_create_flow_table(struct mlx5_core_dev *mdev,
323 u32 table_type,
324 u64 icm_addr_rx,
325 u64 icm_addr_tx,
326 u8 level,
327 bool sw_owner,
328 bool term_tbl,
329 u64 *fdb_rx_icm_addr,
330 u32 *table_id)
331{
332 u32 out[MLX5_ST_SZ_DW(create_flow_table_out)] = {};
333 u32 in[MLX5_ST_SZ_DW(create_flow_table_in)] = {};
334 void *ft_mdev;
335 int err;
336
337 MLX5_SET(create_flow_table_in, in, opcode, MLX5_CMD_OP_CREATE_FLOW_TABLE);
338 MLX5_SET(create_flow_table_in, in, table_type, table_type);
339
340 ft_mdev = MLX5_ADDR_OF(create_flow_table_in, in, flow_table_context);
341 MLX5_SET(flow_table_context, ft_mdev, termination_table, term_tbl);
342 MLX5_SET(flow_table_context, ft_mdev, sw_owner, sw_owner);
343 MLX5_SET(flow_table_context, ft_mdev, level, level);
344
345 if (sw_owner) {
346 /* icm_addr_0 used for FDB RX / NIC TX / NIC_RX
347 * icm_addr_1 used for FDB TX
348 */
349 if (table_type == MLX5_FLOW_TABLE_TYPE_NIC_RX) {
350 MLX5_SET64(flow_table_context, ft_mdev,
351 sw_owner_icm_root_0, icm_addr_rx);
352 } else if (table_type == MLX5_FLOW_TABLE_TYPE_NIC_TX) {
353 MLX5_SET64(flow_table_context, ft_mdev,
354 sw_owner_icm_root_0, icm_addr_tx);
355 } else if (table_type == MLX5_FLOW_TABLE_TYPE_FDB) {
356 MLX5_SET64(flow_table_context, ft_mdev,
357 sw_owner_icm_root_0, icm_addr_rx);
358 MLX5_SET64(flow_table_context, ft_mdev,
359 sw_owner_icm_root_1, icm_addr_tx);
360 }
361 }
362
363 err = mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
364 if (err)
365 return err;
366
367 *table_id = MLX5_GET(create_flow_table_out, out, table_id);
368 if (!sw_owner && table_type == MLX5_FLOW_TABLE_TYPE_FDB)
369 *fdb_rx_icm_addr =
370 (u64)MLX5_GET(create_flow_table_out, out, icm_address_31_0) |
371 (u64)MLX5_GET(create_flow_table_out, out, icm_address_39_32) << 32 |
372 (u64)MLX5_GET(create_flow_table_out, out, icm_address_63_40) << 40;
373
374 return 0;
375}
376
377int mlx5dr_cmd_destroy_flow_table(struct mlx5_core_dev *mdev,
378 u32 table_id,
379 u32 table_type)
380{
381 u32 out[MLX5_ST_SZ_DW(destroy_flow_table_out)] = {};
382 u32 in[MLX5_ST_SZ_DW(destroy_flow_table_in)] = {};
383
384 MLX5_SET(destroy_flow_table_in, in, opcode,
385 MLX5_CMD_OP_DESTROY_FLOW_TABLE);
386 MLX5_SET(destroy_flow_table_in, in, table_type, table_type);
387 MLX5_SET(destroy_flow_table_in, in, table_id, table_id);
388
389 return mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
390}
391
392int mlx5dr_cmd_create_reformat_ctx(struct mlx5_core_dev *mdev,
393 enum mlx5_reformat_ctx_type rt,
394 size_t reformat_size,
395 void *reformat_data,
396 u32 *reformat_id)
397{
398 u32 out[MLX5_ST_SZ_DW(alloc_packet_reformat_context_out)] = {};
399 size_t inlen, cmd_data_sz, cmd_total_sz;
400 void *prctx;
401 void *pdata;
402 void *in;
403 int err;
404
405 cmd_total_sz = MLX5_ST_SZ_BYTES(alloc_packet_reformat_context_in);
406 cmd_data_sz = MLX5_FLD_SZ_BYTES(alloc_packet_reformat_context_in,
407 packet_reformat_context.reformat_data);
408 inlen = ALIGN(cmd_total_sz + reformat_size - cmd_data_sz, 4);
409 in = kvzalloc(inlen, GFP_KERNEL);
410 if (!in)
411 return -ENOMEM;
412
413 MLX5_SET(alloc_packet_reformat_context_in, in, opcode,
414 MLX5_CMD_OP_ALLOC_PACKET_REFORMAT_CONTEXT);
415
416 prctx = MLX5_ADDR_OF(alloc_packet_reformat_context_in, in, packet_reformat_context);
417 pdata = MLX5_ADDR_OF(packet_reformat_context_in, prctx, reformat_data);
418
419 MLX5_SET(packet_reformat_context_in, prctx, reformat_type, rt);
420 MLX5_SET(packet_reformat_context_in, prctx, reformat_data_size, reformat_size);
421 memcpy(pdata, reformat_data, reformat_size);
422
423 err = mlx5_cmd_exec(mdev, in, inlen, out, sizeof(out));
424 if (err)
425 return err;
426
427 *reformat_id = MLX5_GET(alloc_packet_reformat_context_out, out, packet_reformat_id);
428 kvfree(in);
429
430 return err;
431}
432
433void mlx5dr_cmd_destroy_reformat_ctx(struct mlx5_core_dev *mdev,
434 u32 reformat_id)
435{
436 u32 out[MLX5_ST_SZ_DW(dealloc_packet_reformat_context_out)] = {};
437 u32 in[MLX5_ST_SZ_DW(dealloc_packet_reformat_context_in)] = {};
438
439 MLX5_SET(dealloc_packet_reformat_context_in, in, opcode,
440 MLX5_CMD_OP_DEALLOC_PACKET_REFORMAT_CONTEXT);
441 MLX5_SET(dealloc_packet_reformat_context_in, in, packet_reformat_id,
442 reformat_id);
443
444 mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
445}
446
447int mlx5dr_cmd_query_gid(struct mlx5_core_dev *mdev, u8 vhca_port_num,
448 u16 index, struct mlx5dr_cmd_gid_attr *attr)
449{
450 u32 out[MLX5_ST_SZ_DW(query_roce_address_out)] = {};
451 u32 in[MLX5_ST_SZ_DW(query_roce_address_in)] = {};
452 int err;
453
454 MLX5_SET(query_roce_address_in, in, opcode,
455 MLX5_CMD_OP_QUERY_ROCE_ADDRESS);
456
457 MLX5_SET(query_roce_address_in, in, roce_address_index, index);
458 MLX5_SET(query_roce_address_in, in, vhca_port_num, vhca_port_num);
459
460 err = mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
461 if (err)
462 return err;
463
464 memcpy(&attr->gid,
465 MLX5_ADDR_OF(query_roce_address_out,
466 out, roce_address.source_l3_address),
467 sizeof(attr->gid));
468 memcpy(attr->mac,
469 MLX5_ADDR_OF(query_roce_address_out, out,
470 roce_address.source_mac_47_32),
471 sizeof(attr->mac));
472
473 if (MLX5_GET(query_roce_address_out, out,
474 roce_address.roce_version) == MLX5_ROCE_VERSION_2)
475 attr->roce_ver = MLX5_ROCE_VERSION_2;
476 else
477 attr->roce_ver = MLX5_ROCE_VERSION_1;
478
479 return 0;
480}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_crc32.c+98−0
@@ -0,0 +1,98 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2/* Copyright (c) 2019 Mellanox Technologies. */
3
4/* Copyright (c) 2011-2015 Stephan Brumme. All rights reserved.
5 * Slicing-by-16 contributed by Bulat Ziganshin
6 *
7 * This software is provided 'as-is', without any express or implied warranty.
8 * In no event will the author be held liable for any damages arising from the
9 * of this software.
10 *
11 * Permission is granted to anyone to use this software for any purpose,
12 * including commercial applications, and to alter it and redistribute it
13 * freely, subject to the following restrictions:
14 *
15 * 1. The origin of this software must not be misrepresented; you must not
16 * claim that you wrote the original software.
17 * 2. If you use this software in a product, an acknowledgment in the product
18 * documentation would be appreciated but is not required.
19 * 3. Altered source versions must be plainly marked as such, and must not be
20 * misrepresented as being the original software.
21 *
22 * Taken from http://create.stephan-brumme.com/crc32/ and adapted.
23 */
24
25#include "dr_types.h"
26
27#define DR_STE_CRC_POLY 0xEDB88320L
28
29static u32 dr_ste_crc_tab32[8][256];
30
31static void dr_crc32_calc_lookup_entry(u32 (*tbl)[256], u8 i, u8 j)
32{
33 tbl[i][j] = (tbl[i - 1][j] >> 8) ^ tbl[0][tbl[i - 1][j] & 0xff];
34}
35
36void mlx5dr_crc32_init_table(void)
37{
38 u32 crc, i, j;
39
40 for (i = 0; i < 256; i++) {
41 crc = i;
42 for (j = 0; j < 8; j++) {
43 if (crc & 0x00000001L)
44 crc = (crc >> 1) ^ DR_STE_CRC_POLY;
45 else
46 crc = crc >> 1;
47 }
48 dr_ste_crc_tab32[0][i] = crc;
49 }
50
51 /* Init CRC lookup tables according to crc_slice_8 algorithm */
52 for (i = 0; i < 256; i++) {
53 dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 1, i);
54 dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 2, i);
55 dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 3, i);
56 dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 4, i);
57 dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 5, i);
58 dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 6, i);
59 dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 7, i);
60 }
61}
62
63/* Compute CRC32 (Slicing-by-8 algorithm) */
64u32 mlx5dr_crc32_slice8_calc(const void *input_data, size_t length)
65{
66 const u32 *curr = (const u32 *)input_data;
67 const u8 *curr_char;
68 u32 crc = 0, one, two;
69
70 if (!input_data)
71 return 0;
72
73 /* Process eight bytes at once (Slicing-by-8) */
74 while (length >= 8) {
75 one = *curr++ ^ crc;
76 two = *curr++;
77
78 crc = dr_ste_crc_tab32[0][(two >> 24) & 0xff]
79 ^ dr_ste_crc_tab32[1][(two >> 16) & 0xff]
80 ^ dr_ste_crc_tab32[2][(two >> 8) & 0xff]
81 ^ dr_ste_crc_tab32[3][two & 0xff]
82 ^ dr_ste_crc_tab32[4][(one >> 24) & 0xff]
83 ^ dr_ste_crc_tab32[5][(one >> 16) & 0xff]
84 ^ dr_ste_crc_tab32[6][(one >> 8) & 0xff]
85 ^ dr_ste_crc_tab32[7][one & 0xff];
86
87 length -= 8;
88 }
89
90 curr_char = (const u8 *)curr;
91 /* Remaining 1 to 7 bytes (standard algorithm) */
92 while (length-- != 0)
93 crc = (crc >> 8) ^ dr_ste_crc_tab32[0][(crc & 0xff)
94 ^ *curr_char++];
95
96 return ((crc >> 24) & 0xff) | ((crc << 8) & 0xff0000) |
97 ((crc >> 8) & 0xff00) | ((crc << 24) & 0xff000000);
98}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_domain.c+395−0
@@ -0,0 +1,395 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2/* Copyright (c) 2019 Mellanox Technologies. */
3
4#include <linux/mlx5/eswitch.h>
5#include "dr_types.h"
6
7static int dr_domain_init_cache(struct mlx5dr_domain *dmn)
8{
9 /* Per vport cached FW FT for checksum recalculation, this
10 * recalculation is needed due to a HW bug.
11 */
12 dmn->cache.recalc_cs_ft = kcalloc(dmn->info.caps.num_vports,
13 sizeof(dmn->cache.recalc_cs_ft[0]),
14 GFP_KERNEL);
15 if (!dmn->cache.recalc_cs_ft)
16 return -ENOMEM;
17
18 return 0;
19}
20
21static void dr_domain_uninit_cache(struct mlx5dr_domain *dmn)
22{
23 int i;
24
25 for (i = 0; i < dmn->info.caps.num_vports; i++) {
26 if (!dmn->cache.recalc_cs_ft[i])
27 continue;
28
29 mlx5dr_fw_destroy_recalc_cs_ft(dmn, dmn->cache.recalc_cs_ft[i]);
30 }
31
32 kfree(dmn->cache.recalc_cs_ft);
33}
34
35int mlx5dr_domain_cache_get_recalc_cs_ft_addr(struct mlx5dr_domain *dmn,
36 u32 vport_num,
37 u64 *rx_icm_addr)
38{
39 struct mlx5dr_fw_recalc_cs_ft *recalc_cs_ft;
40
41 recalc_cs_ft = dmn->cache.recalc_cs_ft[vport_num];
42 if (!recalc_cs_ft) {
43 /* Table not in cache, need to allocate a new one */
44 recalc_cs_ft = mlx5dr_fw_create_recalc_cs_ft(dmn, vport_num);
45 if (!recalc_cs_ft)
46 return -EINVAL;
47
48 dmn->cache.recalc_cs_ft[vport_num] = recalc_cs_ft;
49 }
50
51 *rx_icm_addr = recalc_cs_ft->rx_icm_addr;
52
53 return 0;
54}
55
56static int dr_domain_init_resources(struct mlx5dr_domain *dmn)
57{
58 int ret;
59
60 ret = mlx5_core_alloc_pd(dmn->mdev, &dmn->pdn);
61 if (ret) {
62 mlx5dr_dbg(dmn, "Couldn't allocate PD\n");
63 return ret;
64 }
65
66 dmn->uar = mlx5_get_uars_page(dmn->mdev);
67 if (!dmn->uar) {
68 mlx5dr_err(dmn, "Couldn't allocate UAR\n");
69 goto clean_pd;
70 }
71
72 dmn->ste_icm_pool = mlx5dr_icm_pool_create(dmn, DR_ICM_TYPE_STE);
73 if (!dmn->ste_icm_pool) {
74 mlx5dr_err(dmn, "Couldn't get icm memory for %s\n",
75 dev_name(dmn->mdev->device));
76 goto clean_uar;
77 }
78
79 dmn->action_icm_pool = mlx5dr_icm_pool_create(dmn, DR_ICM_TYPE_MODIFY_ACTION);
80 if (!dmn->action_icm_pool) {
81 mlx5dr_err(dmn, "Couldn't get action icm memory for %s\n",
82 dev_name(dmn->mdev->device));
83 goto free_ste_icm_pool;
84 }
85
86 ret = mlx5dr_send_ring_alloc(dmn);
87 if (ret) {
88 mlx5dr_err(dmn, "Couldn't create send-ring for %s\n",
89 dev_name(dmn->mdev->device));
90 goto free_action_icm_pool;
91 }
92
93 return 0;
94
95free_action_icm_pool:
96 mlx5dr_icm_pool_destroy(dmn->action_icm_pool);
97free_ste_icm_pool:
98 mlx5dr_icm_pool_destroy(dmn->ste_icm_pool);
99clean_uar:
100 mlx5_put_uars_page(dmn->mdev, dmn->uar);
101clean_pd:
102 mlx5_core_dealloc_pd(dmn->mdev, dmn->pdn);
103
104 return ret;
105}
106
107static void dr_domain_uninit_resources(struct mlx5dr_domain *dmn)
108{
109 mlx5dr_send_ring_free(dmn, dmn->send_ring);
110 mlx5dr_icm_pool_destroy(dmn->action_icm_pool);
111 mlx5dr_icm_pool_destroy(dmn->ste_icm_pool);
112 mlx5_put_uars_page(dmn->mdev, dmn->uar);
113 mlx5_core_dealloc_pd(dmn->mdev, dmn->pdn);
114}
115
116static int dr_domain_query_vport(struct mlx5dr_domain *dmn,
117 bool other_vport,
118 u16 vport_number)
119{
120 struct mlx5dr_cmd_vport_cap *vport_caps;
121 int ret;
122
123 vport_caps = &dmn->info.caps.vports_caps[vport_number];
124
125 ret = mlx5dr_cmd_query_esw_vport_context(dmn->mdev,
126 other_vport,
127 vport_number,
128 &vport_caps->icm_address_rx,
129 &vport_caps->icm_address_tx);
130 if (ret)
131 return ret;
132
133 ret = mlx5dr_cmd_query_gvmi(dmn->mdev,
134 other_vport,
135 vport_number,
136 &vport_caps->vport_gvmi);
137 if (ret)
138 return ret;
139
140 vport_caps->num = vport_number;
141 vport_caps->vhca_gvmi = dmn->info.caps.gvmi;
142
143 return 0;
144}
145
146static int dr_domain_query_vports(struct mlx5dr_domain *dmn)
147{
148 struct mlx5dr_esw_caps *esw_caps = &dmn->info.caps.esw_caps;
149 struct mlx5dr_cmd_vport_cap *wire_vport;
150 int vport;
151 int ret;
152
153 /* Query vports (except wire vport) */
154 for (vport = 0; vport < dmn->info.caps.num_esw_ports - 1; vport++) {
155 ret = dr_domain_query_vport(dmn, !!vport, vport);
156 if (ret)
157 return ret;
158 }
159
160 /* Last vport is the wire port */
161 wire_vport = &dmn->info.caps.vports_caps[vport];
162 wire_vport->num = WIRE_PORT;
163 wire_vport->icm_address_rx = esw_caps->uplink_icm_address_rx;
164 wire_vport->icm_address_tx = esw_caps->uplink_icm_address_tx;
165 wire_vport->vport_gvmi = 0;
166 wire_vport->vhca_gvmi = dmn->info.caps.gvmi;
167
168 return 0;
169}
170
171static int dr_domain_query_fdb_caps(struct mlx5_core_dev *mdev,
172 struct mlx5dr_domain *dmn)
173{
174 int ret;
175
176 if (!dmn->info.caps.eswitch_manager)
177 return -EOPNOTSUPP;
178
179 ret = mlx5dr_cmd_query_esw_caps(mdev, &dmn->info.caps.esw_caps);
180 if (ret)
181 return ret;
182
183 dmn->info.caps.fdb_sw_owner = dmn->info.caps.esw_caps.sw_owner;
184 dmn->info.caps.esw_rx_drop_address = dmn->info.caps.esw_caps.drop_icm_address_rx;
185 dmn->info.caps.esw_tx_drop_address = dmn->info.caps.esw_caps.drop_icm_address_tx;
186
187 dmn->info.caps.vports_caps = kcalloc(dmn->info.caps.num_esw_ports,
188 sizeof(dmn->info.caps.vports_caps[0]),
189 GFP_KERNEL);
190 if (!dmn->info.caps.vports_caps)
191 return -ENOMEM;
192
193 ret = dr_domain_query_vports(dmn);
194 if (ret) {
195 mlx5dr_dbg(dmn, "Failed to query vports caps\n");
196 goto free_vports_caps;
197 }
198
199 dmn->info.caps.num_vports = dmn->info.caps.num_esw_ports - 1;
200
201 return 0;
202
203free_vports_caps:
204 kfree(dmn->info.caps.vports_caps);
205 dmn->info.caps.vports_caps = NULL;
206 return ret;
207}
208
209static int dr_domain_caps_init(struct mlx5_core_dev *mdev,
210 struct mlx5dr_domain *dmn)
211{
212 struct mlx5dr_cmd_vport_cap *vport_cap;
213 int ret;
214
215 if (MLX5_CAP_GEN(mdev, port_type) != MLX5_CAP_PORT_TYPE_ETH) {
216 mlx5dr_dbg(dmn, "Failed to allocate domain, bad link type\n");
217 return -EOPNOTSUPP;
218 }
219
220 dmn->info.caps.num_esw_ports = mlx5_eswitch_get_total_vports(mdev);
221
222 ret = mlx5dr_cmd_query_device(mdev, &dmn->info.caps);
223 if (ret)
224 return ret;
225
226 ret = dr_domain_query_fdb_caps(mdev, dmn);
227 if (ret)
228 return ret;
229
230 switch (dmn->type) {
231 case MLX5DR_DOMAIN_TYPE_NIC_RX:
232 if (!dmn->info.caps.rx_sw_owner)
233 return -ENOTSUPP;
234
235 dmn->info.supp_sw_steering = true;
236 dmn->info.rx.ste_type = MLX5DR_STE_TYPE_RX;
237 dmn->info.rx.default_icm_addr = dmn->info.caps.nic_rx_drop_address;
238 dmn->info.rx.drop_icm_addr = dmn->info.caps.nic_rx_drop_address;
239 break;
240 case MLX5DR_DOMAIN_TYPE_NIC_TX:
241 if (!dmn->info.caps.tx_sw_owner)
242 return -ENOTSUPP;
243
244 dmn->info.supp_sw_steering = true;
245 dmn->info.tx.ste_type = MLX5DR_STE_TYPE_TX;
246 dmn->info.tx.default_icm_addr = dmn->info.caps.nic_tx_allow_address;
247 dmn->info.tx.drop_icm_addr = dmn->info.caps.nic_tx_drop_address;
248 break;
249 case MLX5DR_DOMAIN_TYPE_FDB:
250 if (!dmn->info.caps.eswitch_manager)
251 return -ENOTSUPP;
252
253 if (!dmn->info.caps.fdb_sw_owner)
254 return -ENOTSUPP;
255
256 dmn->info.rx.ste_type = MLX5DR_STE_TYPE_RX;
257 dmn->info.tx.ste_type = MLX5DR_STE_TYPE_TX;
258 vport_cap = mlx5dr_get_vport_cap(&dmn->info.caps, 0);
259 if (!vport_cap) {
260 mlx5dr_dbg(dmn, "Failed to get esw manager vport\n");
261 return -ENOENT;
262 }
263
264 dmn->info.supp_sw_steering = true;
265 dmn->info.tx.default_icm_addr = vport_cap->icm_address_tx;
266 dmn->info.rx.default_icm_addr = vport_cap->icm_address_rx;
267 dmn->info.rx.drop_icm_addr = dmn->info.caps.esw_rx_drop_address;
268 dmn->info.tx.drop_icm_addr = dmn->info.caps.esw_tx_drop_address;
269 break;
270 default:
271 mlx5dr_dbg(dmn, "Invalid domain\n");
272 ret = -EINVAL;
273 break;
274 }
275
276 return ret;
277}
278
279static void dr_domain_caps_uninit(struct mlx5dr_domain *dmn)
280{
281 kfree(dmn->info.caps.vports_caps);
282}
283
284struct mlx5dr_domain *
285mlx5dr_domain_create(struct mlx5_core_dev *mdev, enum mlx5dr_domain_type type)
286{
287 struct mlx5dr_domain *dmn;
288 int ret;
289
290 if (type > MLX5DR_DOMAIN_TYPE_FDB)
291 return NULL;
292
293 dmn = kzalloc(sizeof(*dmn), GFP_KERNEL);
294 if (!dmn)
295 return NULL;
296
297 dmn->mdev = mdev;
298 dmn->type = type;
299 refcount_set(&dmn->refcount, 1);
300 mutex_init(&dmn->mutex);
301
302 if (dr_domain_caps_init(mdev, dmn)) {
303 mlx5dr_dbg(dmn, "Failed init domain, no caps\n");
304 goto free_domain;
305 }
306
307 dmn->info.max_log_action_icm_sz = DR_CHUNK_SIZE_4K;
308 dmn->info.max_log_sw_icm_sz = min_t(u32, DR_CHUNK_SIZE_1024K,
309 dmn->info.caps.log_icm_size);
310
311 if (!dmn->info.supp_sw_steering) {
312 mlx5dr_err(dmn, "SW steering not supported for %s\n",
313 dev_name(mdev->device));
314 goto uninit_caps;
315 }
316
317 /* Allocate resources */
318 ret = dr_domain_init_resources(dmn);
319 if (ret) {
320 mlx5dr_err(dmn, "Failed init domain resources for %s\n",
321 dev_name(mdev->device));
322 goto uninit_caps;
323 }
324
325 ret = dr_domain_init_cache(dmn);
326 if (ret) {
327 mlx5dr_err(dmn, "Failed initialize domain cache\n");
328 goto uninit_resourses;
329 }
330
331 /* Init CRC table for htbl CRC calculation */
332 mlx5dr_crc32_init_table();
333
334 return dmn;
335
336uninit_resourses:
337 dr_domain_uninit_resources(dmn);
338uninit_caps:
339 dr_domain_caps_uninit(dmn);
340free_domain:
341 kfree(dmn);
342 return NULL;
343}
344
345/* Assure synchronization of the device steering tables with updates made by SW
346 * insertion.
347 */
348int mlx5dr_domain_sync(struct mlx5dr_domain *dmn, u32 flags)
349{
350 int ret = 0;
351
352 if (flags & MLX5DR_DOMAIN_SYNC_FLAGS_SW) {
353 mutex_lock(&dmn->mutex);
354 ret = mlx5dr_send_ring_force_drain(dmn);
355 mutex_unlock(&dmn->mutex);
356 if (ret)
357 return ret;
358 }
359
360 if (flags & MLX5DR_DOMAIN_SYNC_FLAGS_HW)
361 ret = mlx5dr_cmd_sync_steering(dmn->mdev);
362
363 return ret;
364}
365
366int mlx5dr_domain_destroy(struct mlx5dr_domain *dmn)
367{
368 if (refcount_read(&dmn->refcount) > 1)
369 return -EBUSY;
370
371 /* make sure resources are not used by the hardware */
372 mlx5dr_cmd_sync_steering(dmn->mdev);
373 dr_domain_uninit_cache(dmn);
374 dr_domain_uninit_resources(dmn);
375 dr_domain_caps_uninit(dmn);
376 mutex_destroy(&dmn->mutex);
377 kfree(dmn);
378 return 0;
379}
380
381void mlx5dr_domain_set_peer(struct mlx5dr_domain *dmn,
382 struct mlx5dr_domain *peer_dmn)
383{
384 mutex_lock(&dmn->mutex);
385
386 if (dmn->peer_dmn)
387 refcount_dec(&dmn->peer_dmn->refcount);
388
389 dmn->peer_dmn = peer_dmn;
390
391 if (dmn->peer_dmn)
392 refcount_inc(&dmn->peer_dmn->refcount);
393
394 mutex_unlock(&dmn->mutex);
395}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_fw.c+93−0
@@ -0,0 +1,93 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2/* Copyright (c) 2019 Mellanox Technologies. */
3
4#include <linux/types.h>
5#include "dr_types.h"
6
7struct mlx5dr_fw_recalc_cs_ft *
8mlx5dr_fw_create_recalc_cs_ft(struct mlx5dr_domain *dmn, u32 vport_num)
9{
10 struct mlx5dr_fw_recalc_cs_ft *recalc_cs_ft;
11 u32 table_id, group_id, modify_hdr_id;
12 u64 rx_icm_addr, modify_ttl_action;
13 int ret;
14
15 recalc_cs_ft = kzalloc(sizeof(*recalc_cs_ft), GFP_KERNEL);
16 if (!recalc_cs_ft)
17 return NULL;
18
19 ret = mlx5dr_cmd_create_flow_table(dmn->mdev, MLX5_FLOW_TABLE_TYPE_FDB,
20 0, 0, dmn->info.caps.max_ft_level - 1,
21 false, true, &rx_icm_addr, &table_id);
22 if (ret) {
23 mlx5dr_err(dmn, "Failed creating TTL W/A FW flow table %d\n", ret);
24 goto free_ttl_tbl;
25 }
26
27 ret = mlx5dr_cmd_create_empty_flow_group(dmn->mdev,
28 MLX5_FLOW_TABLE_TYPE_FDB,
29 table_id, &group_id);
30 if (ret) {
31 mlx5dr_err(dmn, "Failed creating TTL W/A FW flow group %d\n", ret);
32 goto destroy_flow_table;
33 }
34
35 /* Modify TTL action by adding zero to trigger CS recalculation */
36 modify_ttl_action = 0;
37 MLX5_SET(set_action_in, &modify_ttl_action, action_type, MLX5_ACTION_TYPE_ADD);
38 MLX5_SET(set_action_in, &modify_ttl_action, field, MLX5_ACTION_IN_FIELD_OUT_IP_TTL);
39
40 ret = mlx5dr_cmd_alloc_modify_header(dmn->mdev, MLX5_FLOW_TABLE_TYPE_FDB, 1,
41 &modify_ttl_action,
42 &modify_hdr_id);
43 if (ret) {
44 mlx5dr_err(dmn, "Failed modify header TTL %d\n", ret);
45 goto destroy_flow_group;
46 }
47
48 ret = mlx5dr_cmd_set_fte_modify_and_vport(dmn->mdev,
49 MLX5_FLOW_TABLE_TYPE_FDB,
50 table_id, group_id, modify_hdr_id,
51 vport_num);
52 if (ret) {
53 mlx5dr_err(dmn, "Failed setting TTL W/A flow table entry %d\n", ret);
54 goto dealloc_modify_header;
55 }
56
57 recalc_cs_ft->modify_hdr_id = modify_hdr_id;
58 recalc_cs_ft->rx_icm_addr = rx_icm_addr;
59 recalc_cs_ft->table_id = table_id;
60 recalc_cs_ft->group_id = group_id;
61
62 return recalc_cs_ft;
63
64dealloc_modify_header:
65 mlx5dr_cmd_dealloc_modify_header(dmn->mdev, modify_hdr_id);
66destroy_flow_group:
67 mlx5dr_cmd_destroy_flow_group(dmn->mdev,
68 MLX5_FLOW_TABLE_TYPE_FDB,
69 table_id, group_id);
70destroy_flow_table:
71 mlx5dr_cmd_destroy_flow_table(dmn->mdev, table_id, MLX5_FLOW_TABLE_TYPE_FDB);
72free_ttl_tbl:
73 kfree(recalc_cs_ft);
74 return NULL;
75}
76
77void mlx5dr_fw_destroy_recalc_cs_ft(struct mlx5dr_domain *dmn,
78 struct mlx5dr_fw_recalc_cs_ft *recalc_cs_ft)
79{
80 mlx5dr_cmd_del_flow_table_entry(dmn->mdev,
81 MLX5_FLOW_TABLE_TYPE_FDB,
82 recalc_cs_ft->table_id);
83 mlx5dr_cmd_dealloc_modify_header(dmn->mdev, recalc_cs_ft->modify_hdr_id);
84 mlx5dr_cmd_destroy_flow_group(dmn->mdev,
85 MLX5_FLOW_TABLE_TYPE_FDB,
86 recalc_cs_ft->table_id,
87 recalc_cs_ft->group_id);
88 mlx5dr_cmd_destroy_flow_table(dmn->mdev,
89 recalc_cs_ft->table_id,
90 MLX5_FLOW_TABLE_TYPE_FDB);
91
92 kfree(recalc_cs_ft);
93}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_icm_pool.c+570−0
@@ -0,0 +1,570 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2/* Copyright (c) 2019 Mellanox Technologies. */
3
4#include "dr_types.h"
5
6#define DR_ICM_MODIFY_HDR_ALIGN_BASE 64
7#define DR_ICM_SYNC_THRESHOLD (64 * 1024 * 1024)
8
9struct mlx5dr_icm_pool;
10
11struct mlx5dr_icm_bucket {
12 struct mlx5dr_icm_pool *pool;
13
14 /* Chunks that aren't visible to HW not directly and not in cache */
15 struct list_head free_list;
16 unsigned int free_list_count;
17
18 /* Used chunks, HW may be accessing this memory */
19 struct list_head used_list;
20 unsigned int used_list_count;
21
22 /* HW may be accessing this memory but at some future,
23 * undetermined time, it might cease to do so. Before deciding to call
24 * sync_ste, this list is moved to sync_list
25 */
26 struct list_head hot_list;
27 unsigned int hot_list_count;
28
29 /* Pending sync list, entries from the hot list are moved to this list.
30 * sync_ste is executed and then sync_list is concatenated to the free list
31 */
32 struct list_head sync_list;
33 unsigned int sync_list_count;
34
35 u32 total_chunks;
36 u32 num_of_entries;
37 u32 entry_size;
38 /* protect the ICM bucket */
39 struct mutex mutex;
40};
41
42struct mlx5dr_icm_pool {
43 struct mlx5dr_icm_bucket *buckets;
44 enum mlx5dr_icm_type icm_type;
45 enum mlx5dr_icm_chunk_size max_log_chunk_sz;
46 enum mlx5dr_icm_chunk_size num_of_buckets;
47 struct list_head icm_mr_list;
48 /* protect the ICM MR list */
49 struct mutex mr_mutex;
50 struct mlx5dr_domain *dmn;
51};
52
53struct mlx5dr_icm_dm {
54 u32 obj_id;
55 enum mlx5_sw_icm_type type;
56 u64 addr;
57 size_t length;
58};
59
60struct mlx5dr_icm_mr {
61 struct mlx5dr_icm_pool *pool;
62 struct mlx5_core_mkey mkey;
63 struct mlx5dr_icm_dm dm;
64 size_t used_length;
65 size_t length;
66 u64 icm_start_addr;
67 struct list_head mr_list;
68};
69
70static int dr_icm_create_dm_mkey(struct mlx5_core_dev *mdev,
71 u32 pd, u64 length, u64 start_addr, int mode,
72 struct mlx5_core_mkey *mkey)
73{
74 u32 inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
75 u32 in[MLX5_ST_SZ_DW(create_mkey_in)] = {};
76 void *mkc;
77
78 mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry);
79
80 MLX5_SET(mkc, mkc, access_mode_1_0, mode);
81 MLX5_SET(mkc, mkc, access_mode_4_2, (mode >> 2) & 0x7);
82 MLX5_SET(mkc, mkc, lw, 1);
83 MLX5_SET(mkc, mkc, lr, 1);
84 if (mode == MLX5_MKC_ACCESS_MODE_SW_ICM) {
85 MLX5_SET(mkc, mkc, rw, 1);
86 MLX5_SET(mkc, mkc, rr, 1);
87 }
88
89 MLX5_SET64(mkc, mkc, len, length);
90 MLX5_SET(mkc, mkc, pd, pd);
91 MLX5_SET(mkc, mkc, qpn, 0xffffff);
92 MLX5_SET64(mkc, mkc, start_addr, start_addr);
93
94 return mlx5_core_create_mkey(mdev, mkey, in, inlen);
95}
96
97static struct mlx5dr_icm_mr *
98dr_icm_pool_mr_create(struct mlx5dr_icm_pool *pool,
99 enum mlx5_sw_icm_type type,
100 size_t align_base)
101{
102 struct mlx5_core_dev *mdev = pool->dmn->mdev;
103 struct mlx5dr_icm_mr *icm_mr;
104 size_t align_diff;
105 int err;
106
107 icm_mr = kvzalloc(sizeof(*icm_mr), GFP_KERNEL);
108 if (!icm_mr)
109 return NULL;
110
111 icm_mr->pool = pool;
112 INIT_LIST_HEAD(&icm_mr->mr_list);
113
114 icm_mr->dm.type = type;
115
116 /* 2^log_biggest_table * entry-size * double-for-alignment */
117 icm_mr->dm.length = mlx5dr_icm_pool_chunk_size_to_byte(pool->max_log_chunk_sz,
118 pool->icm_type) * 2;
119
120 err = mlx5_dm_sw_icm_alloc(mdev, icm_mr->dm.type, icm_mr->dm.length, 0,
121 &icm_mr->dm.addr, &icm_mr->dm.obj_id);
122 if (err) {
123 mlx5dr_err(pool->dmn, "Failed to allocate SW ICM memory, err (%d)\n", err);
124 goto free_icm_mr;
125 }
126
127 /* Register device memory */
128 err = dr_icm_create_dm_mkey(mdev, pool->dmn->pdn,
129 icm_mr->dm.length,
130 icm_mr->dm.addr,
131 MLX5_MKC_ACCESS_MODE_SW_ICM,
132 &icm_mr->mkey);
133 if (err) {
134 mlx5dr_err(pool->dmn, "Failed to create SW ICM MKEY, err (%d)\n", err);
135 goto free_dm;
136 }
137
138 icm_mr->icm_start_addr = icm_mr->dm.addr;
139
140 align_diff = icm_mr->icm_start_addr % align_base;
141 if (align_diff)
142 icm_mr->used_length = align_base - align_diff;
143
144 list_add_tail(&icm_mr->mr_list, &pool->icm_mr_list);
145
146 return icm_mr;
147
148free_dm:
149 mlx5_dm_sw_icm_dealloc(mdev, icm_mr->dm.type, icm_mr->dm.length, 0,
150 icm_mr->dm.addr, icm_mr->dm.obj_id);
151free_icm_mr:
152 kvfree(icm_mr);
153 return NULL;
154}
155
156static void dr_icm_pool_mr_destroy(struct mlx5dr_icm_mr *icm_mr)
157{
158 struct mlx5_core_dev *mdev = icm_mr->pool->dmn->mdev;
159 struct mlx5dr_icm_dm *dm = &icm_mr->dm;
160
161 list_del(&icm_mr->mr_list);
162 mlx5_core_destroy_mkey(mdev, &icm_mr->mkey);
163 mlx5_dm_sw_icm_dealloc(mdev, dm->type, dm->length, 0,
164 dm->addr, dm->obj_id);
165 kvfree(icm_mr);
166}
167
168static int dr_icm_chunk_ste_init(struct mlx5dr_icm_chunk *chunk)
169{
170 struct mlx5dr_icm_bucket *bucket = chunk->bucket;
171
172 chunk->ste_arr = kvzalloc(bucket->num_of_entries *
173 sizeof(chunk->ste_arr[0]), GFP_KERNEL);
174 if (!chunk->ste_arr)
175 return -ENOMEM;
176
177 chunk->hw_ste_arr = kvzalloc(bucket->num_of_entries *
178 DR_STE_SIZE_REDUCED, GFP_KERNEL);
179 if (!chunk->hw_ste_arr)
180 goto out_free_ste_arr;
181
182 chunk->miss_list = kvmalloc(bucket->num_of_entries *
183 sizeof(chunk->miss_list[0]), GFP_KERNEL);
184 if (!chunk->miss_list)
185 goto out_free_hw_ste_arr;
186
187 return 0;
188
189out_free_hw_ste_arr:
190 kvfree(chunk->hw_ste_arr);
191out_free_ste_arr:
192 kvfree(chunk->ste_arr);
193 return -ENOMEM;
194}
195
196static int dr_icm_chunks_create(struct mlx5dr_icm_bucket *bucket)
197{
198 size_t mr_free_size, mr_req_size, mr_row_size;
199 struct mlx5dr_icm_pool *pool = bucket->pool;
200 struct mlx5dr_icm_mr *icm_mr = NULL;
201 struct mlx5dr_icm_chunk *chunk;
202 enum mlx5_sw_icm_type dm_type;
203 size_t align_base;
204 int i, err = 0;
205
206 mr_req_size = bucket->num_of_entries * bucket->entry_size;
207 mr_row_size = mlx5dr_icm_pool_chunk_size_to_byte(pool->max_log_chunk_sz,
208 pool->icm_type);
209
210 if (pool->icm_type == DR_ICM_TYPE_STE) {
211 dm_type = MLX5_SW_ICM_TYPE_STEERING;
212 /* Align base is the biggest chunk size / row size */
213 align_base = mr_row_size;
214 } else {
215 dm_type = MLX5_SW_ICM_TYPE_HEADER_MODIFY;
216 /* Align base is 64B */
217 align_base = DR_ICM_MODIFY_HDR_ALIGN_BASE;
218 }
219
220 mutex_lock(&pool->mr_mutex);
221 if (!list_empty(&pool->icm_mr_list)) {
222 icm_mr = list_last_entry(&pool->icm_mr_list,
223 struct mlx5dr_icm_mr, mr_list);
224
225 if (icm_mr)
226 mr_free_size = icm_mr->dm.length - icm_mr->used_length;
227 }
228
229 if (!icm_mr || mr_free_size < mr_row_size) {
230 icm_mr = dr_icm_pool_mr_create(pool, dm_type, align_base);
231 if (!icm_mr) {
232 err = -ENOMEM;
233 goto out_err;
234 }
235 }
236
237 /* Create memory aligned chunks */
238 for (i = 0; i < mr_row_size / mr_req_size; i++) {
239 chunk = kvzalloc(sizeof(*chunk), GFP_KERNEL);
240 if (!chunk) {
241 err = -ENOMEM;
242 goto out_err;
243 }
244
245 chunk->bucket = bucket;
246 chunk->rkey = icm_mr->mkey.key;
247 /* mr start addr is zero based */
248 chunk->mr_addr = icm_mr->used_length;
249 chunk->icm_addr = (uintptr_t)icm_mr->icm_start_addr + icm_mr->used_length;
250 icm_mr->used_length += mr_req_size;
251 chunk->num_of_entries = bucket->num_of_entries;
252 chunk->byte_size = chunk->num_of_entries * bucket->entry_size;
253
254 if (pool->icm_type == DR_ICM_TYPE_STE) {
255 err = dr_icm_chunk_ste_init(chunk);
256 if (err)
257 goto out_free_chunk;
258 }
259
260 INIT_LIST_HEAD(&chunk->chunk_list);
261 list_add(&chunk->chunk_list, &bucket->free_list);
262 bucket->free_list_count++;
263 bucket->total_chunks++;
264 }
265 mutex_unlock(&pool->mr_mutex);
266 return 0;
267
268out_free_chunk:
269 kvfree(chunk);
270out_err:
271 mutex_unlock(&pool->mr_mutex);
272 return err;
273}
274
275static void dr_icm_chunk_ste_cleanup(struct mlx5dr_icm_chunk *chunk)
276{
277 kvfree(chunk->miss_list);
278 kvfree(chunk->hw_ste_arr);
279 kvfree(chunk->ste_arr);
280}
281
282static void dr_icm_chunk_destroy(struct mlx5dr_icm_chunk *chunk)
283{
284 struct mlx5dr_icm_bucket *bucket = chunk->bucket;
285
286 list_del(&chunk->chunk_list);
287 bucket->total_chunks--;
288
289 if (bucket->pool->icm_type == DR_ICM_TYPE_STE)
290 dr_icm_chunk_ste_cleanup(chunk);
291
292 kvfree(chunk);
293}
294
295static void dr_icm_bucket_init(struct mlx5dr_icm_pool *pool,
296 struct mlx5dr_icm_bucket *bucket,
297 enum mlx5dr_icm_chunk_size chunk_size)
298{
299 if (pool->icm_type == DR_ICM_TYPE_STE)
300 bucket->entry_size = DR_STE_SIZE;
301 else
302 bucket->entry_size = DR_MODIFY_ACTION_SIZE;
303
304 bucket->num_of_entries = mlx5dr_icm_pool_chunk_size_to_entries(chunk_size);
305 bucket->pool = pool;
306 mutex_init(&bucket->mutex);
307 INIT_LIST_HEAD(&bucket->free_list);
308 INIT_LIST_HEAD(&bucket->used_list);
309 INIT_LIST_HEAD(&bucket->hot_list);
310 INIT_LIST_HEAD(&bucket->sync_list);
311}
312
313static void dr_icm_bucket_cleanup(struct mlx5dr_icm_bucket *bucket)
314{
315 struct mlx5dr_icm_chunk *chunk, *next;
316
317 mutex_destroy(&bucket->mutex);
318 list_splice_tail_init(&bucket->sync_list, &bucket->free_list);
319 list_splice_tail_init(&bucket->hot_list, &bucket->free_list);
320
321 list_for_each_entry_safe(chunk, next, &bucket->free_list, chunk_list)
322 dr_icm_chunk_destroy(chunk);
323
324 WARN_ON(bucket->total_chunks != 0);
325
326 /* Cleanup of unreturned chunks */
327 list_for_each_entry_safe(chunk, next, &bucket->used_list, chunk_list)
328 dr_icm_chunk_destroy(chunk);
329}
330
331static u64 dr_icm_hot_mem_size(struct mlx5dr_icm_pool *pool)
332{
333 u64 hot_size = 0;
334 int chunk_order;
335
336 for (chunk_order = 0; chunk_order < pool->num_of_buckets; chunk_order++)
337 hot_size += pool->buckets[chunk_order].hot_list_count *
338 mlx5dr_icm_pool_chunk_size_to_byte(chunk_order, pool->icm_type);
339
340 return hot_size;
341}
342
343static bool dr_icm_reuse_hot_entries(struct mlx5dr_icm_pool *pool,
344 struct mlx5dr_icm_bucket *bucket)
345{
346 u64 bytes_for_sync;
347
348 bytes_for_sync = dr_icm_hot_mem_size(pool);
349 if (bytes_for_sync < DR_ICM_SYNC_THRESHOLD || !bucket->hot_list_count)
350 return false;
351
352 return true;
353}
354
355static void dr_icm_chill_bucket_start(struct mlx5dr_icm_bucket *bucket)
356{
357 list_splice_tail_init(&bucket->hot_list, &bucket->sync_list);
358 bucket->sync_list_count += bucket->hot_list_count;
359 bucket->hot_list_count = 0;
360}
361
362static void dr_icm_chill_bucket_end(struct mlx5dr_icm_bucket *bucket)
363{
364 list_splice_tail_init(&bucket->sync_list, &bucket->free_list);
365 bucket->free_list_count += bucket->sync_list_count;
366 bucket->sync_list_count = 0;
367}
368
369static void dr_icm_chill_bucket_abort(struct mlx5dr_icm_bucket *bucket)
370{
371 list_splice_tail_init(&bucket->sync_list, &bucket->hot_list);
372 bucket->hot_list_count += bucket->sync_list_count;
373 bucket->sync_list_count = 0;
374}
375
376static void dr_icm_chill_buckets_start(struct mlx5dr_icm_pool *pool,
377 struct mlx5dr_icm_bucket *cb,
378 bool buckets[DR_CHUNK_SIZE_MAX])
379{
380 struct mlx5dr_icm_bucket *bucket;
381 int i;
382
383 for (i = 0; i < pool->num_of_buckets; i++) {
384 bucket = &pool->buckets[i];
385 if (bucket == cb) {
386 dr_icm_chill_bucket_start(bucket);
387 continue;
388 }
389
390 /* Freeing the mutex is done at the end of that process, after
391 * sync_ste was executed at dr_icm_chill_buckets_end func.
392 */
393 if (mutex_trylock(&bucket->mutex)) {
394 dr_icm_chill_bucket_start(bucket);
395 buckets[i] = true;
396 }
397 }
398}
399
400static void dr_icm_chill_buckets_end(struct mlx5dr_icm_pool *pool,
401 struct mlx5dr_icm_bucket *cb,
402 bool buckets[DR_CHUNK_SIZE_MAX])
403{
404 struct mlx5dr_icm_bucket *bucket;
405 int i;
406
407 for (i = 0; i < pool->num_of_buckets; i++) {
408 bucket = &pool->buckets[i];
409 if (bucket == cb) {
410 dr_icm_chill_bucket_end(bucket);
411 continue;
412 }
413
414 if (!buckets[i])
415 continue;
416
417 dr_icm_chill_bucket_end(bucket);
418 mutex_unlock(&bucket->mutex);
419 }
420}
421
422static void dr_icm_chill_buckets_abort(struct mlx5dr_icm_pool *pool,
423 struct mlx5dr_icm_bucket *cb,
424 bool buckets[DR_CHUNK_SIZE_MAX])
425{
426 struct mlx5dr_icm_bucket *bucket;
427 int i;
428
429 for (i = 0; i < pool->num_of_buckets; i++) {
430 bucket = &pool->buckets[i];
431 if (bucket == cb) {
432 dr_icm_chill_bucket_abort(bucket);
433 continue;
434 }
435
436 if (!buckets[i])
437 continue;
438
439 dr_icm_chill_bucket_abort(bucket);
440 mutex_unlock(&bucket->mutex);
441 }
442}
443
444/* Allocate an ICM chunk, each chunk holds a piece of ICM memory and
445 * also memory used for HW STE management for optimizations.
446 */
447struct mlx5dr_icm_chunk *
448mlx5dr_icm_alloc_chunk(struct mlx5dr_icm_pool *pool,
449 enum mlx5dr_icm_chunk_size chunk_size)
450{
451 struct mlx5dr_icm_chunk *chunk = NULL; /* Fix compilation warning */
452 bool buckets[DR_CHUNK_SIZE_MAX] = {};
453 struct mlx5dr_icm_bucket *bucket;
454 int err;
455
456 if (chunk_size > pool->max_log_chunk_sz)
457 return NULL;
458
459 bucket = &pool->buckets[chunk_size];
460
461 mutex_lock(&bucket->mutex);
462
463 /* Take chunk from pool if available, otherwise allocate new chunks */
464 if (list_empty(&bucket->free_list)) {
465 if (dr_icm_reuse_hot_entries(pool, bucket)) {
466 dr_icm_chill_buckets_start(pool, bucket, buckets);
467 err = mlx5dr_cmd_sync_steering(pool->dmn->mdev);
468 if (err) {
469 dr_icm_chill_buckets_abort(pool, bucket, buckets);
470 mlx5dr_dbg(pool->dmn, "Sync_steering failed\n");
471 chunk = NULL;
472 goto out;
473 }
474 dr_icm_chill_buckets_end(pool, bucket, buckets);
475 } else {
476 dr_icm_chunks_create(bucket);
477 }
478 }
479
480 if (!list_empty(&bucket->free_list)) {
481 chunk = list_last_entry(&bucket->free_list,
482 struct mlx5dr_icm_chunk,
483 chunk_list);
484 if (chunk) {
485 list_del_init(&chunk->chunk_list);
486 list_add_tail(&chunk->chunk_list, &bucket->used_list);
487 bucket->free_list_count--;
488 bucket->used_list_count++;
489 }
490 }
491out:
492 mutex_unlock(&bucket->mutex);
493 return chunk;
494}
495
496void mlx5dr_icm_free_chunk(struct mlx5dr_icm_chunk *chunk)
497{
498 struct mlx5dr_icm_bucket *bucket = chunk->bucket;
499
500 if (bucket->pool->icm_type == DR_ICM_TYPE_STE) {
501 memset(chunk->ste_arr, 0,
502 bucket->num_of_entries * sizeof(chunk->ste_arr[0]));
503 memset(chunk->hw_ste_arr, 0,
504 bucket->num_of_entries * DR_STE_SIZE_REDUCED);
505 }
506
507 mutex_lock(&bucket->mutex);
508 list_del_init(&chunk->chunk_list);
509 list_add_tail(&chunk->chunk_list, &bucket->hot_list);
510 bucket->hot_list_count++;
511 bucket->used_list_count--;
512 mutex_unlock(&bucket->mutex);
513}
514
515struct mlx5dr_icm_pool *mlx5dr_icm_pool_create(struct mlx5dr_domain *dmn,
516 enum mlx5dr_icm_type icm_type)
517{
518 enum mlx5dr_icm_chunk_size max_log_chunk_sz;
519 struct mlx5dr_icm_pool *pool;
520 int i;
521
522 if (icm_type == DR_ICM_TYPE_STE)
523 max_log_chunk_sz = dmn->info.max_log_sw_icm_sz;
524 else
525 max_log_chunk_sz = dmn->info.max_log_action_icm_sz;
526
527 pool = kvzalloc(sizeof(*pool), GFP_KERNEL);
528 if (!pool)
529 return NULL;
530
531 pool->buckets = kcalloc(max_log_chunk_sz + 1,
532 sizeof(pool->buckets[0]),
533 GFP_KERNEL);
534 if (!pool->buckets)
535 goto free_pool;
536
537 pool->dmn = dmn;
538 pool->icm_type = icm_type;
539 pool->max_log_chunk_sz = max_log_chunk_sz;
540 pool->num_of_buckets = max_log_chunk_sz + 1;
541 INIT_LIST_HEAD(&pool->icm_mr_list);
542
543 for (i = 0; i < pool->num_of_buckets; i++)
544 dr_icm_bucket_init(pool, &pool->buckets[i], i);
545
546 mutex_init(&pool->mr_mutex);
547
548 return pool;
549
550free_pool:
551 kvfree(pool);
552 return NULL;
553}
554
555void mlx5dr_icm_pool_destroy(struct mlx5dr_icm_pool *pool)
556{
557 struct mlx5dr_icm_mr *icm_mr, *next;
558 int i;
559
560 mutex_destroy(&pool->mr_mutex);
561
562 list_for_each_entry_safe(icm_mr, next, &pool->icm_mr_list, mr_list)
563 dr_icm_pool_mr_destroy(icm_mr);
564
565 for (i = 0; i < pool->num_of_buckets; i++)
566 dr_icm_bucket_cleanup(&pool->buckets[i]);
567
568 kfree(pool->buckets);
569 kvfree(pool);
570}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_matcher.c+770−0
@@ -0,0 +1,770 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2/* Copyright (c) 2019 Mellanox Technologies. */
3
4#include "dr_types.h"
5
6static bool dr_mask_is_smac_set(struct mlx5dr_match_spec *spec)
7{
8 return (spec->smac_47_16 || spec->smac_15_0);
9}
10
11static bool dr_mask_is_dmac_set(struct mlx5dr_match_spec *spec)
12{
13 return (spec->dmac_47_16 || spec->dmac_15_0);
14}
15
16static bool dr_mask_is_src_addr_set(struct mlx5dr_match_spec *spec)
17{
18 return (spec->src_ip_127_96 || spec->src_ip_95_64 ||
19 spec->src_ip_63_32 || spec->src_ip_31_0);
20}
21
22static bool dr_mask_is_dst_addr_set(struct mlx5dr_match_spec *spec)
23{
24 return (spec->dst_ip_127_96 || spec->dst_ip_95_64 ||
25 spec->dst_ip_63_32 || spec->dst_ip_31_0);
26}
27
28static bool dr_mask_is_l3_base_set(struct mlx5dr_match_spec *spec)
29{
30 return (spec->ip_protocol || spec->frag || spec->tcp_flags ||
31 spec->ip_ecn || spec->ip_dscp);
32}
33
34static bool dr_mask_is_tcp_udp_base_set(struct mlx5dr_match_spec *spec)
35{
36 return (spec->tcp_sport || spec->tcp_dport ||
37 spec->udp_sport || spec->udp_dport);
38}
39
40static bool dr_mask_is_ipv4_set(struct mlx5dr_match_spec *spec)
41{
42 return (spec->dst_ip_31_0 || spec->src_ip_31_0);
43}
44
45static bool dr_mask_is_ipv4_5_tuple_set(struct mlx5dr_match_spec *spec)
46{
47 return (dr_mask_is_l3_base_set(spec) ||
48 dr_mask_is_tcp_udp_base_set(spec) ||
49 dr_mask_is_ipv4_set(spec));
50}
51
52static bool dr_mask_is_eth_l2_tnl_set(struct mlx5dr_match_misc *misc)
53{
54 return misc->vxlan_vni;
55}
56
57static bool dr_mask_is_ttl_set(struct mlx5dr_match_spec *spec)
58{
59 return spec->ttl_hoplimit;
60}
61
62#define DR_MASK_IS_L2_DST(_spec, _misc, _inner_outer) (_spec.first_vid || \
63 (_spec).first_cfi || (_spec).first_prio || (_spec).cvlan_tag || \
64 (_spec).svlan_tag || (_spec).dmac_47_16 || (_spec).dmac_15_0 || \
65 (_spec).ethertype || (_spec).ip_version || \
66 (_misc)._inner_outer##_second_vid || \
67 (_misc)._inner_outer##_second_cfi || \
68 (_misc)._inner_outer##_second_prio || \
69 (_misc)._inner_outer##_second_cvlan_tag || \
70 (_misc)._inner_outer##_second_svlan_tag)
71
72#define DR_MASK_IS_ETH_L4_SET(_spec, _misc, _inner_outer) ( \
73 dr_mask_is_l3_base_set(&(_spec)) || \
74 dr_mask_is_tcp_udp_base_set(&(_spec)) || \
75 dr_mask_is_ttl_set(&(_spec)) || \
76 (_misc)._inner_outer##_ipv6_flow_label)
77
78#define DR_MASK_IS_ETH_L4_MISC_SET(_misc3, _inner_outer) ( \
79 (_misc3)._inner_outer##_tcp_seq_num || \
80 (_misc3)._inner_outer##_tcp_ack_num)
81
82#define DR_MASK_IS_FIRST_MPLS_SET(_misc2, _inner_outer) ( \
83 (_misc2)._inner_outer##_first_mpls_label || \
84 (_misc2)._inner_outer##_first_mpls_exp || \
85 (_misc2)._inner_outer##_first_mpls_s_bos || \
86 (_misc2)._inner_outer##_first_mpls_ttl)
87
88static bool dr_mask_is_gre_set(struct mlx5dr_match_misc *misc)
89{
90 return (misc->gre_key_h || misc->gre_key_l ||
91 misc->gre_protocol || misc->gre_c_present ||
92 misc->gre_k_present || misc->gre_s_present);
93}
94
95#define DR_MASK_IS_OUTER_MPLS_OVER_GRE_UDP_SET(_misc2, gre_udp) ( \
96 (_misc2).outer_first_mpls_over_##gre_udp##_label || \
97 (_misc2).outer_first_mpls_over_##gre_udp##_exp || \
98 (_misc2).outer_first_mpls_over_##gre_udp##_s_bos || \
99 (_misc2).outer_first_mpls_over_##gre_udp##_ttl)
100
101#define DR_MASK_IS_FLEX_PARSER_0_SET(_misc2) ( \
102 DR_MASK_IS_OUTER_MPLS_OVER_GRE_UDP_SET((_misc2), gre) || \
103 DR_MASK_IS_OUTER_MPLS_OVER_GRE_UDP_SET((_misc2), udp))
104
105static bool dr_mask_is_flex_parser_tnl_set(struct mlx5dr_match_misc3 *misc3)
106{
107 return (misc3->outer_vxlan_gpe_vni ||
108 misc3->outer_vxlan_gpe_next_protocol ||
109 misc3->outer_vxlan_gpe_flags);
110}
111
112static bool dr_mask_is_flex_parser_icmpv6_set(struct mlx5dr_match_misc3 *misc3)
113{
114 return (misc3->icmpv6_type || misc3->icmpv6_code ||
115 misc3->icmpv6_header_data);
116}
117
118static bool dr_mask_is_wqe_metadata_set(struct mlx5dr_match_misc2 *misc2)
119{
120 return misc2->metadata_reg_a;
121}
122
123static bool dr_mask_is_reg_c_0_3_set(struct mlx5dr_match_misc2 *misc2)
124{
125 return (misc2->metadata_reg_c_0 || misc2->metadata_reg_c_1 ||
126 misc2->metadata_reg_c_2 || misc2->metadata_reg_c_3);
127}
128
129static bool dr_mask_is_reg_c_4_7_set(struct mlx5dr_match_misc2 *misc2)
130{
131 return (misc2->metadata_reg_c_4 || misc2->metadata_reg_c_5 ||
132 misc2->metadata_reg_c_6 || misc2->metadata_reg_c_7);
133}
134
135static bool dr_mask_is_gvmi_or_qpn_set(struct mlx5dr_match_misc *misc)
136{
137 return (misc->source_sqn || misc->source_port);
138}
139
140static bool
141dr_matcher_supp_flex_parser_vxlan_gpe(struct mlx5dr_domain *dmn)
142{
143 return dmn->info.caps.flex_protocols &
144 MLX5_FLEX_PARSER_VXLAN_GPE_ENABLED;
145}
146
147int mlx5dr_matcher_select_builders(struct mlx5dr_matcher *matcher,
148 struct mlx5dr_matcher_rx_tx *nic_matcher,
149 bool ipv6)
150{
151 if (ipv6) {
152 nic_matcher->ste_builder = nic_matcher->ste_builder6;
153 nic_matcher->num_of_builders = nic_matcher->num_of_builders6;
154 } else {
155 nic_matcher->ste_builder = nic_matcher->ste_builder4;
156 nic_matcher->num_of_builders = nic_matcher->num_of_builders4;
157 }
158
159 if (!nic_matcher->num_of_builders) {
160 mlx5dr_dbg(matcher->tbl->dmn,
161 "Rule not supported on this matcher due to IP related fields\n");
162 return -EINVAL;
163 }
164
165 return 0;
166}
167
168static int dr_matcher_set_ste_builders(struct mlx5dr_matcher *matcher,
169 struct mlx5dr_matcher_rx_tx *nic_matcher,
170 bool ipv6)
171{
172 struct mlx5dr_domain_rx_tx *nic_dmn = nic_matcher->nic_tbl->nic_dmn;
173 struct mlx5dr_domain *dmn = matcher->tbl->dmn;
174 struct mlx5dr_match_param mask = {};
175 struct mlx5dr_match_misc3 *misc3;
176 struct mlx5dr_ste_build *sb;
177 u8 *num_of_builders;
178 bool inner, rx;
179 int idx = 0;
180 int ret, i;
181
182 if (ipv6) {
183 sb = nic_matcher->ste_builder6;
184 num_of_builders = &nic_matcher->num_of_builders6;
185 } else {
186 sb = nic_matcher->ste_builder4;
187 num_of_builders = &nic_matcher->num_of_builders4;
188 }
189
190 rx = nic_dmn->ste_type == MLX5DR_STE_TYPE_RX;
191
192 /* Create a temporary mask to track and clear used mask fields */
193 if (matcher->match_criteria & DR_MATCHER_CRITERIA_OUTER)
194 mask.outer = matcher->mask.outer;
195
196 if (matcher->match_criteria & DR_MATCHER_CRITERIA_MISC)
197 mask.misc = matcher->mask.misc;
198
199 if (matcher->match_criteria & DR_MATCHER_CRITERIA_INNER)
200 mask.inner = matcher->mask.inner;
201
202 if (matcher->match_criteria & DR_MATCHER_CRITERIA_MISC2)
203 mask.misc2 = matcher->mask.misc2;
204
205 if (matcher->match_criteria & DR_MATCHER_CRITERIA_MISC3)
206 mask.misc3 = matcher->mask.misc3;
207
208 ret = mlx5dr_ste_build_pre_check(dmn, matcher->match_criteria,
209 &matcher->mask, NULL);
210 if (ret)
211 return ret;
212
213 /* Outer */
214 if (matcher->match_criteria & (DR_MATCHER_CRITERIA_OUTER |
215 DR_MATCHER_CRITERIA_MISC |
216 DR_MATCHER_CRITERIA_MISC2 |
217 DR_MATCHER_CRITERIA_MISC3)) {
218 inner = false;
219
220 if (dr_mask_is_wqe_metadata_set(&mask.misc2))
221 mlx5dr_ste_build_general_purpose(&sb[idx++], &mask, inner, rx);
222
223 if (dr_mask_is_reg_c_0_3_set(&mask.misc2))
224 mlx5dr_ste_build_register_0(&sb[idx++], &mask, inner, rx);
225
226 if (dr_mask_is_reg_c_4_7_set(&mask.misc2))
227 mlx5dr_ste_build_register_1(&sb[idx++], &mask, inner, rx);
228
229 if (dr_mask_is_gvmi_or_qpn_set(&mask.misc) &&
230 (dmn->type == MLX5DR_DOMAIN_TYPE_FDB ||
231 dmn->type == MLX5DR_DOMAIN_TYPE_NIC_RX)) {
232 ret = mlx5dr_ste_build_src_gvmi_qpn(&sb[idx++], &mask,
233 &dmn->info.caps,
234 inner, rx);
235 if (ret)
236 return ret;
237 }
238
239 if (dr_mask_is_smac_set(&mask.outer) &&
240 dr_mask_is_dmac_set(&mask.outer)) {
241 ret = mlx5dr_ste_build_eth_l2_src_des(&sb[idx++], &mask,
242 inner, rx);
243 if (ret)
244 return ret;
245 }
246
247 if (dr_mask_is_smac_set(&mask.outer))
248 mlx5dr_ste_build_eth_l2_src(&sb[idx++], &mask, inner, rx);
249
250 if (DR_MASK_IS_L2_DST(mask.outer, mask.misc, outer))
251 mlx5dr_ste_build_eth_l2_dst(&sb[idx++], &mask, inner, rx);
252
253 if (ipv6) {
254 if (dr_mask_is_dst_addr_set(&mask.outer))
255 mlx5dr_ste_build_eth_l3_ipv6_dst(&sb[idx++], &mask,
256 inner, rx);
257
258 if (dr_mask_is_src_addr_set(&mask.outer))
259 mlx5dr_ste_build_eth_l3_ipv6_src(&sb[idx++], &mask,
260 inner, rx);
261
262 if (DR_MASK_IS_ETH_L4_SET(mask.outer, mask.misc, outer))
263 mlx5dr_ste_build_ipv6_l3_l4(&sb[idx++], &mask,
264 inner, rx);
265 } else {
266 if (dr_mask_is_ipv4_5_tuple_set(&mask.outer))
267 mlx5dr_ste_build_eth_l3_ipv4_5_tuple(&sb[idx++], &mask,
268 inner, rx);
269
270 if (dr_mask_is_ttl_set(&mask.outer))
271 mlx5dr_ste_build_eth_l3_ipv4_misc(&sb[idx++], &mask,
272 inner, rx);
273 }
274
275 if (dr_mask_is_flex_parser_tnl_set(&mask.misc3) &&
276 dr_matcher_supp_flex_parser_vxlan_gpe(dmn))
277 mlx5dr_ste_build_flex_parser_tnl(&sb[idx++], &mask,
278 inner, rx);
279
280 if (DR_MASK_IS_ETH_L4_MISC_SET(mask.misc3, outer))
281 mlx5dr_ste_build_eth_l4_misc(&sb[idx++], &mask, inner, rx);
282
283 if (DR_MASK_IS_FIRST_MPLS_SET(mask.misc2, outer))
284 mlx5dr_ste_build_mpls(&sb[idx++], &mask, inner, rx);
285
286 if (DR_MASK_IS_FLEX_PARSER_0_SET(mask.misc2))
287 mlx5dr_ste_build_flex_parser_0(&sb[idx++], &mask,
288 inner, rx);
289
290 misc3 = &mask.misc3;
291 if ((DR_MASK_IS_FLEX_PARSER_ICMPV4_SET(misc3) &&
292 mlx5dr_matcher_supp_flex_parser_icmp_v4(&dmn->info.caps)) ||
293 (dr_mask_is_flex_parser_icmpv6_set(&mask.misc3) &&
294 mlx5dr_matcher_supp_flex_parser_icmp_v6(&dmn->info.caps))) {
295 ret = mlx5dr_ste_build_flex_parser_1(&sb[idx++],
296 &mask, &dmn->info.caps,
297 inner, rx);
298 if (ret)
299 return ret;
300 }
301 if (dr_mask_is_gre_set(&mask.misc))
302 mlx5dr_ste_build_gre(&sb[idx++], &mask, inner, rx);
303 }
304
305 /* Inner */
306 if (matcher->match_criteria & (DR_MATCHER_CRITERIA_INNER |
307 DR_MATCHER_CRITERIA_MISC |
308 DR_MATCHER_CRITERIA_MISC2 |
309 DR_MATCHER_CRITERIA_MISC3)) {
310 inner = true;
311
312 if (dr_mask_is_eth_l2_tnl_set(&mask.misc))
313 mlx5dr_ste_build_eth_l2_tnl(&sb[idx++], &mask, inner, rx);
314
315 if (dr_mask_is_smac_set(&mask.inner) &&
316 dr_mask_is_dmac_set(&mask.inner)) {
317 ret = mlx5dr_ste_build_eth_l2_src_des(&sb[idx++],
318 &mask, inner, rx);
319 if (ret)
320 return ret;
321 }
322
323 if (dr_mask_is_smac_set(&mask.inner))
324 mlx5dr_ste_build_eth_l2_src(&sb[idx++], &mask, inner, rx);
325
326 if (DR_MASK_IS_L2_DST(mask.inner, mask.misc, inner))
327 mlx5dr_ste_build_eth_l2_dst(&sb[idx++], &mask, inner, rx);
328
329 if (ipv6) {
330 if (dr_mask_is_dst_addr_set(&mask.inner))
331 mlx5dr_ste_build_eth_l3_ipv6_dst(&sb[idx++], &mask,
332 inner, rx);
333
334 if (dr_mask_is_src_addr_set(&mask.inner))
335 mlx5dr_ste_build_eth_l3_ipv6_src(&sb[idx++], &mask,
336 inner, rx);
337
338 if (DR_MASK_IS_ETH_L4_SET(mask.inner, mask.misc, inner))
339 mlx5dr_ste_build_ipv6_l3_l4(&sb[idx++], &mask,
340 inner, rx);
341 } else {
342 if (dr_mask_is_ipv4_5_tuple_set(&mask.inner))
343 mlx5dr_ste_build_eth_l3_ipv4_5_tuple(&sb[idx++], &mask,
344 inner, rx);
345
346 if (dr_mask_is_ttl_set(&mask.inner))
347 mlx5dr_ste_build_eth_l3_ipv4_misc(&sb[idx++], &mask,
348 inner, rx);
349 }
350
351 if (DR_MASK_IS_ETH_L4_MISC_SET(mask.misc3, inner))
352 mlx5dr_ste_build_eth_l4_misc(&sb[idx++], &mask, inner, rx);
353
354 if (DR_MASK_IS_FIRST_MPLS_SET(mask.misc2, inner))
355 mlx5dr_ste_build_mpls(&sb[idx++], &mask, inner, rx);
356
357 if (DR_MASK_IS_FLEX_PARSER_0_SET(mask.misc2))
358 mlx5dr_ste_build_flex_parser_0(&sb[idx++], &mask, inner, rx);
359 }
360 /* Empty matcher, takes all */
361 if (matcher->match_criteria == DR_MATCHER_CRITERIA_EMPTY)
362 mlx5dr_ste_build_empty_always_hit(&sb[idx++], rx);
363
364 if (idx == 0) {
365 mlx5dr_dbg(dmn, "Cannot generate any valid rules from mask\n");
366 return -EINVAL;
367 }
368
369 /* Check that all mask fields were consumed */
370 for (i = 0; i < sizeof(struct mlx5dr_match_param); i++) {
371 if (((u8 *)&mask)[i] != 0) {
372 mlx5dr_info(dmn, "Mask contains unsupported parameters\n");
373 return -EOPNOTSUPP;
374 }
375 }
376
377 *num_of_builders = idx;
378
379 return 0;
380}
381
382static int dr_matcher_connect(struct mlx5dr_domain *dmn,
383 struct mlx5dr_matcher_rx_tx *curr_nic_matcher,
384 struct mlx5dr_matcher_rx_tx *next_nic_matcher,
385 struct mlx5dr_matcher_rx_tx *prev_nic_matcher)
386{
387 struct mlx5dr_table_rx_tx *nic_tbl = curr_nic_matcher->nic_tbl;
388 struct mlx5dr_domain_rx_tx *nic_dmn = nic_tbl->nic_dmn;
389 struct mlx5dr_htbl_connect_info info;
390 struct mlx5dr_ste_htbl *prev_htbl;
391 int ret;
392
393 /* Connect end anchor hash table to next_htbl or to the default address */
394 if (next_nic_matcher) {
395 info.type = CONNECT_HIT;
396 info.hit_next_htbl = next_nic_matcher->s_htbl;
397 } else {
398 info.type = CONNECT_MISS;
399 info.miss_icm_addr = nic_tbl->default_icm_addr;
400 }
401 ret = mlx5dr_ste_htbl_init_and_postsend(dmn, nic_dmn,
402 curr_nic_matcher->e_anchor,
403 &info, info.type == CONNECT_HIT);
404 if (ret)
405 return ret;
406
407 /* Connect start hash table to end anchor */
408 info.type = CONNECT_MISS;
409 info.miss_icm_addr = curr_nic_matcher->e_anchor->chunk->icm_addr;
410 ret = mlx5dr_ste_htbl_init_and_postsend(dmn, nic_dmn,
411 curr_nic_matcher->s_htbl,
412 &info, false);
413 if (ret)
414 return ret;
415
416 /* Connect previous hash table to matcher start hash table */
417 if (prev_nic_matcher)
418 prev_htbl = prev_nic_matcher->e_anchor;
419 else
420 prev_htbl = nic_tbl->s_anchor;
421
422 info.type = CONNECT_HIT;
423 info.hit_next_htbl = curr_nic_matcher->s_htbl;
424 ret = mlx5dr_ste_htbl_init_and_postsend(dmn, nic_dmn, prev_htbl,
425 &info, true);
426 if (ret)
427 return ret;
428
429 /* Update the pointing ste and next hash table */
430 curr_nic_matcher->s_htbl->pointing_ste = prev_htbl->ste_arr;
431 prev_htbl->ste_arr[0].next_htbl = curr_nic_matcher->s_htbl;
432
433 if (next_nic_matcher) {
434 next_nic_matcher->s_htbl->pointing_ste = curr_nic_matcher->e_anchor->ste_arr;
435 curr_nic_matcher->e_anchor->ste_arr[0].next_htbl = next_nic_matcher->s_htbl;
436 }
437
438 return 0;
439}
440
441static int dr_matcher_add_to_tbl(struct mlx5dr_matcher *matcher)
442{
443 struct mlx5dr_matcher *next_matcher, *prev_matcher, *tmp_matcher;
444 struct mlx5dr_table *tbl = matcher->tbl;
445 struct mlx5dr_domain *dmn = tbl->dmn;
446 bool first = true;
447 int ret;
448
449 next_matcher = NULL;
450 if (!list_empty(&tbl->matcher_list))
451 list_for_each_entry(tmp_matcher, &tbl->matcher_list, matcher_list) {
452 if (tmp_matcher->prio >= matcher->prio) {
453 next_matcher = tmp_matcher;
454 break;
455 }
456 first = false;
457 }
458
459 prev_matcher = NULL;
460 if (next_matcher && !first)
461 prev_matcher = list_entry(next_matcher->matcher_list.prev,
462 struct mlx5dr_matcher,
463 matcher_list);
464 else if (!first)
465 prev_matcher = list_entry(tbl->matcher_list.prev,
466 struct mlx5dr_matcher,
467 matcher_list);
468
469 if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB ||
470 dmn->type == MLX5DR_DOMAIN_TYPE_NIC_RX) {
471 ret = dr_matcher_connect(dmn, &matcher->rx,
472 next_matcher ? &next_matcher->rx : NULL,
473 prev_matcher ? &prev_matcher->rx : NULL);
474 if (ret)
475 return ret;
476 }
477
478 if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB ||
479 dmn->type == MLX5DR_DOMAIN_TYPE_NIC_TX) {
480 ret = dr_matcher_connect(dmn, &matcher->tx,
481 next_matcher ? &next_matcher->tx : NULL,
482 prev_matcher ? &prev_matcher->tx : NULL);
483 if (ret)
484 return ret;
485 }
486
487 if (prev_matcher)
488 list_add(&matcher->matcher_list, &prev_matcher->matcher_list);
489 else if (next_matcher)
490 list_add_tail(&matcher->matcher_list,
491 &next_matcher->matcher_list);
492 else
493 list_add(&matcher->matcher_list, &tbl->matcher_list);
494
495 return 0;
496}
497
498static void dr_matcher_uninit_nic(struct mlx5dr_matcher_rx_tx *nic_matcher)
499{
500 mlx5dr_htbl_put(nic_matcher->s_htbl);
501 mlx5dr_htbl_put(nic_matcher->e_anchor);
502}
503
504static void dr_matcher_uninit_fdb(struct mlx5dr_matcher *matcher)
505{
506 dr_matcher_uninit_nic(&matcher->rx);
507 dr_matcher_uninit_nic(&matcher->tx);
508}
509
510static void dr_matcher_uninit(struct mlx5dr_matcher *matcher)
511{
512 struct mlx5dr_domain *dmn = matcher->tbl->dmn;
513
514 switch (dmn->type) {
515 case MLX5DR_DOMAIN_TYPE_NIC_RX:
516 dr_matcher_uninit_nic(&matcher->rx);
517 break;
518 case MLX5DR_DOMAIN_TYPE_NIC_TX:
519 dr_matcher_uninit_nic(&matcher->tx);
520 break;
521 case MLX5DR_DOMAIN_TYPE_FDB:
522 dr_matcher_uninit_fdb(matcher);
523 break;
524 default:
525 WARN_ON(true);
526 break;
527 }
528}
529
530static int dr_matcher_init_nic(struct mlx5dr_matcher *matcher,
531 struct mlx5dr_matcher_rx_tx *nic_matcher)
532{
533 struct mlx5dr_domain *dmn = matcher->tbl->dmn;
534 int ret, ret_v4, ret_v6;
535
536 ret_v4 = dr_matcher_set_ste_builders(matcher, nic_matcher, false);
537 ret_v6 = dr_matcher_set_ste_builders(matcher, nic_matcher, true);
538
539 if (ret_v4 && ret_v6) {
540 mlx5dr_dbg(dmn, "Cannot generate IPv4 or IPv6 rules with given mask\n");
541 return -EINVAL;
542 }
543
544 if (!ret_v4)
545 nic_matcher->ste_builder = nic_matcher->ste_builder4;
546 else
547 nic_matcher->ste_builder = nic_matcher->ste_builder6;
548
549 nic_matcher->e_anchor = mlx5dr_ste_htbl_alloc(dmn->ste_icm_pool,
550 DR_CHUNK_SIZE_1,
551 MLX5DR_STE_LU_TYPE_DONT_CARE,
552 0);
553 if (!nic_matcher->e_anchor)
554 return -ENOMEM;
555
556 nic_matcher->s_htbl = mlx5dr_ste_htbl_alloc(dmn->ste_icm_pool,
557 DR_CHUNK_SIZE_1,
558 nic_matcher->ste_builder[0].lu_type,
559 nic_matcher->ste_builder[0].byte_mask);
560 if (!nic_matcher->s_htbl) {
561 ret = -ENOMEM;
562 goto free_e_htbl;
563 }
564
565 /* make sure the tables exist while empty */
566 mlx5dr_htbl_get(nic_matcher->s_htbl);
567 mlx5dr_htbl_get(nic_matcher->e_anchor);
568
569 return 0;
570
571free_e_htbl:
572 mlx5dr_ste_htbl_free(nic_matcher->e_anchor);
573 return ret;
574}
575
576static int dr_matcher_init_fdb(struct mlx5dr_matcher *matcher)
577{
578 int ret;
579
580 ret = dr_matcher_init_nic(matcher, &matcher->rx);
581 if (ret)
582 return ret;
583
584 ret = dr_matcher_init_nic(matcher, &matcher->tx);
585 if (ret)
586 goto uninit_nic_rx;
587
588 return 0;
589
590uninit_nic_rx:
591 dr_matcher_uninit_nic(&matcher->rx);
592 return ret;
593}
594
595static int dr_matcher_init(struct mlx5dr_matcher *matcher,
596 struct mlx5dr_match_parameters *mask)
597{
598 struct mlx5dr_table *tbl = matcher->tbl;
599 struct mlx5dr_domain *dmn = tbl->dmn;
600 int ret;
601
602 if (matcher->match_criteria >= DR_MATCHER_CRITERIA_MAX) {
603 mlx5dr_info(dmn, "Invalid match criteria attribute\n");
604 return -EINVAL;
605 }
606
607 if (mask) {
608 if (mask->match_sz > sizeof(struct mlx5dr_match_param)) {
609 mlx5dr_info(dmn, "Invalid match size attribute\n");
610 return -EINVAL;
611 }
612 mlx5dr_ste_copy_param(matcher->match_criteria,
613 &matcher->mask, mask);
614 }
615
616 switch (dmn->type) {
617 case MLX5DR_DOMAIN_TYPE_NIC_RX:
618 matcher->rx.nic_tbl = &tbl->rx;
619 ret = dr_matcher_init_nic(matcher, &matcher->rx);
620 break;
621 case MLX5DR_DOMAIN_TYPE_NIC_TX:
622 matcher->tx.nic_tbl = &tbl->tx;
623 ret = dr_matcher_init_nic(matcher, &matcher->tx);
624 break;
625 case MLX5DR_DOMAIN_TYPE_FDB:
626 matcher->rx.nic_tbl = &tbl->rx;
627 matcher->tx.nic_tbl = &tbl->tx;
628 ret = dr_matcher_init_fdb(matcher);
629 break;
630 default:
631 WARN_ON(true);
632 return -EINVAL;
633 }
634
635 return ret;
636}
637
638struct mlx5dr_matcher *
639mlx5dr_matcher_create(struct mlx5dr_table *tbl,
640 u16 priority,
641 u8 match_criteria_enable,
642 struct mlx5dr_match_parameters *mask)
643{
644 struct mlx5dr_matcher *matcher;
645 int ret;
646
647 refcount_inc(&tbl->refcount);
648
649 matcher = kzalloc(sizeof(*matcher), GFP_KERNEL);
650 if (!matcher)
651 goto dec_ref;
652
653 matcher->tbl = tbl;
654 matcher->prio = priority;
655 matcher->match_criteria = match_criteria_enable;
656 refcount_set(&matcher->refcount, 1);
657 INIT_LIST_HEAD(&matcher->matcher_list);
658
659 mutex_lock(&tbl->dmn->mutex);
660
661 ret = dr_matcher_init(matcher, mask);
662 if (ret)
663 goto free_matcher;
664
665 ret = dr_matcher_add_to_tbl(matcher);
666 if (ret)
667 goto matcher_uninit;
668
669 mutex_unlock(&tbl->dmn->mutex);
670
671 return matcher;
672
673matcher_uninit:
674 dr_matcher_uninit(matcher);
675free_matcher:
676 mutex_unlock(&tbl->dmn->mutex);
677 kfree(matcher);
678dec_ref:
679 refcount_dec(&tbl->refcount);
680 return NULL;
681}
682
683static int dr_matcher_disconnect(struct mlx5dr_domain *dmn,
684 struct mlx5dr_table_rx_tx *nic_tbl,
685 struct mlx5dr_matcher_rx_tx *next_nic_matcher,
686 struct mlx5dr_matcher_rx_tx *prev_nic_matcher)
687{
688 struct mlx5dr_domain_rx_tx *nic_dmn = nic_tbl->nic_dmn;
689 struct mlx5dr_htbl_connect_info info;
690 struct mlx5dr_ste_htbl *prev_anchor;
691
692 if (prev_nic_matcher)
693 prev_anchor = prev_nic_matcher->e_anchor;
694 else
695 prev_anchor = nic_tbl->s_anchor;
696
697 /* Connect previous anchor hash table to next matcher or to the default address */
698 if (next_nic_matcher) {
699 info.type = CONNECT_HIT;
700 info.hit_next_htbl = next_nic_matcher->s_htbl;
701 next_nic_matcher->s_htbl->pointing_ste = prev_anchor->ste_arr;
702 prev_anchor->ste_arr[0].next_htbl = next_nic_matcher->s_htbl;
703 } else {
704 info.type = CONNECT_MISS;
705 info.miss_icm_addr = nic_tbl->default_icm_addr;
706 prev_anchor->ste_arr[0].next_htbl = NULL;
707 }
708
709 return mlx5dr_ste_htbl_init_and_postsend(dmn, nic_dmn, prev_anchor,
710 &info, true);
711}
712
713static int dr_matcher_remove_from_tbl(struct mlx5dr_matcher *matcher)
714{
715 struct mlx5dr_matcher *prev_matcher, *next_matcher;
716 struct mlx5dr_table *tbl = matcher->tbl;
717 struct mlx5dr_domain *dmn = tbl->dmn;
718 int ret = 0;
719
720 if (list_is_last(&matcher->matcher_list, &tbl->matcher_list))
721 next_matcher = NULL;
722 else
723 next_matcher = list_next_entry(matcher, matcher_list);
724
725 if (matcher->matcher_list.prev == &tbl->matcher_list)
726 prev_matcher = NULL;
727 else
728 prev_matcher = list_prev_entry(matcher, matcher_list);
729
730 if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB ||
731 dmn->type == MLX5DR_DOMAIN_TYPE_NIC_RX) {
732 ret = dr_matcher_disconnect(dmn, &tbl->rx,
733 next_matcher ? &next_matcher->rx : NULL,
734 prev_matcher ? &prev_matcher->rx : NULL);
735 if (ret)
736 return ret;
737 }
738
739 if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB ||
740 dmn->type == MLX5DR_DOMAIN_TYPE_NIC_TX) {
741 ret = dr_matcher_disconnect(dmn, &tbl->tx,
742 next_matcher ? &next_matcher->tx : NULL,
743 prev_matcher ? &prev_matcher->tx : NULL);
744 if (ret)
745 return ret;
746 }
747
748 list_del(&matcher->matcher_list);
749
750 return 0;
751}
752
753int mlx5dr_matcher_destroy(struct mlx5dr_matcher *matcher)
754{
755 struct mlx5dr_table *tbl = matcher->tbl;
756
757 if (refcount_read(&matcher->refcount) > 1)
758 return -EBUSY;
759
760 mutex_lock(&tbl->dmn->mutex);
761
762 dr_matcher_remove_from_tbl(matcher);
763 dr_matcher_uninit(matcher);
764 refcount_dec(&matcher->tbl->refcount);
765
766 mutex_unlock(&tbl->dmn->mutex);
767 kfree(matcher);
768
769 return 0;
770}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_rule.c+1243−0
@@ -0,0 +1,1243 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2/* Copyright (c) 2019 Mellanox Technologies. */
3
4#include "dr_types.h"
5
6#define DR_RULE_MAX_STE_CHAIN (DR_RULE_MAX_STES + DR_ACTION_MAX_STES)
7
8struct mlx5dr_rule_action_member {
9 struct mlx5dr_action *action;
10 struct list_head list;
11};
12
13static int dr_rule_append_to_miss_list(struct mlx5dr_ste *new_last_ste,
14 struct list_head *miss_list,
15 struct list_head *send_list)
16{
17 struct mlx5dr_ste_send_info *ste_info_last;
18 struct mlx5dr_ste *last_ste;
19
20 /* The new entry will be inserted after the last */
21 last_ste = list_entry(miss_list->prev, struct mlx5dr_ste, miss_list_node);
22 WARN_ON(!last_ste);
23
24 ste_info_last = kzalloc(sizeof(*ste_info_last), GFP_KERNEL);
25 if (!ste_info_last)
26 return -ENOMEM;
27
28 mlx5dr_ste_set_miss_addr(last_ste->hw_ste,
29 mlx5dr_ste_get_icm_addr(new_last_ste));
30 list_add_tail(&new_last_ste->miss_list_node, miss_list);
31
32 mlx5dr_send_fill_and_append_ste_send_info(last_ste, DR_STE_SIZE_REDUCED,
33 0, last_ste->hw_ste,
34 ste_info_last, send_list, true);
35
36 return 0;
37}
38
39static struct mlx5dr_ste *
40dr_rule_create_collision_htbl(struct mlx5dr_matcher *matcher,
41 struct mlx5dr_matcher_rx_tx *nic_matcher,
42 u8 *hw_ste)
43{
44 struct mlx5dr_domain *dmn = matcher->tbl->dmn;
45 struct mlx5dr_ste_htbl *new_htbl;
46 struct mlx5dr_ste *ste;
47
48 /* Create new table for miss entry */
49 new_htbl = mlx5dr_ste_htbl_alloc(dmn->ste_icm_pool,
50 DR_CHUNK_SIZE_1,
51 MLX5DR_STE_LU_TYPE_DONT_CARE,
52 0);
53 if (!new_htbl) {
54 mlx5dr_dbg(dmn, "Failed allocating collision table\n");
55 return NULL;
56 }
57
58 /* One and only entry, never grows */
59 ste = new_htbl->ste_arr;
60 mlx5dr_ste_set_miss_addr(hw_ste, nic_matcher->e_anchor->chunk->icm_addr);
61 mlx5dr_htbl_get(new_htbl);
62
63 return ste;
64}
65
66static struct mlx5dr_ste *
67dr_rule_create_collision_entry(struct mlx5dr_matcher *matcher,
68 struct mlx5dr_matcher_rx_tx *nic_matcher,
69 u8 *hw_ste,
70 struct mlx5dr_ste *orig_ste)
71{
72 struct mlx5dr_ste *ste;
73
74 ste = dr_rule_create_collision_htbl(matcher, nic_matcher, hw_ste);
75 if (!ste) {
76 mlx5dr_dbg(matcher->tbl->dmn, "Failed creating collision entry\n");
77 return NULL;
78 }
79
80 ste->ste_chain_location = orig_ste->ste_chain_location;
81
82 /* In collision entry, all members share the same miss_list_head */
83 ste->htbl->miss_list = mlx5dr_ste_get_miss_list(orig_ste);
84
85 /* Next table */
86 if (mlx5dr_ste_create_next_htbl(matcher, nic_matcher, ste, hw_ste,
87 DR_CHUNK_SIZE_1)) {
88 mlx5dr_dbg(matcher->tbl->dmn, "Failed allocating table\n");
89 goto free_tbl;
90 }
91
92 return ste;
93
94free_tbl:
95 mlx5dr_ste_free(ste, matcher, nic_matcher);
96 return NULL;
97}
98
99static int
100dr_rule_handle_one_ste_in_update_list(struct mlx5dr_ste_send_info *ste_info,
101 struct mlx5dr_domain *dmn)
102{
103 int ret;
104
105 list_del(&ste_info->send_list);
106 ret = mlx5dr_send_postsend_ste(dmn, ste_info->ste, ste_info->data,
107 ste_info->size, ste_info->offset);
108 if (ret)
109 goto out;
110 /* Copy data to ste, only reduced size, the last 16B (mask)
111 * is already written to the hw.
112 */
113 memcpy(ste_info->ste->hw_ste, ste_info->data, DR_STE_SIZE_REDUCED);
114
115out:
116 kfree(ste_info);
117 return ret;
118}
119
120static int dr_rule_send_update_list(struct list_head *send_ste_list,
121 struct mlx5dr_domain *dmn,
122 bool is_reverse)
123{
124 struct mlx5dr_ste_send_info *ste_info, *tmp_ste_info;
125 int ret;
126
127 if (is_reverse) {
128 list_for_each_entry_safe_reverse(ste_info, tmp_ste_info,
129 send_ste_list, send_list) {
130 ret = dr_rule_handle_one_ste_in_update_list(ste_info,
131 dmn);
132 if (ret)
133 return ret;
134 }
135 } else {
136 list_for_each_entry_safe(ste_info, tmp_ste_info,
137 send_ste_list, send_list) {
138 ret = dr_rule_handle_one_ste_in_update_list(ste_info,
139 dmn);
140 if (ret)
141 return ret;
142 }
143 }
144
145 return 0;
146}
147
148static struct mlx5dr_ste *
149dr_rule_find_ste_in_miss_list(struct list_head *miss_list, u8 *hw_ste)
150{
151 struct mlx5dr_ste *ste;
152
153 if (list_empty(miss_list))
154 return NULL;
155
156 /* Check if hw_ste is present in the list */
157 list_for_each_entry(ste, miss_list, miss_list_node) {
158 if (mlx5dr_ste_equal_tag(ste->hw_ste, hw_ste))
159 return ste;
160 }
161
162 return NULL;
163}
164
165static struct mlx5dr_ste *
166dr_rule_rehash_handle_collision(struct mlx5dr_matcher *matcher,
167 struct mlx5dr_matcher_rx_tx *nic_matcher,
168 struct list_head *update_list,
169 struct mlx5dr_ste *col_ste,
170 u8 *hw_ste)
171{
172 struct mlx5dr_ste *new_ste;
173 int ret;
174
175 new_ste = dr_rule_create_collision_htbl(matcher, nic_matcher, hw_ste);
176 if (!new_ste)
177 return NULL;
178
179 /* In collision entry, all members share the same miss_list_head */
180 new_ste->htbl->miss_list = mlx5dr_ste_get_miss_list(col_ste);
181
182 /* Update the previous from the list */
183 ret = dr_rule_append_to_miss_list(new_ste,
184 mlx5dr_ste_get_miss_list(col_ste),
185 update_list);
186 if (ret) {
187 mlx5dr_dbg(matcher->tbl->dmn, "Failed update dup entry\n");
188 goto err_exit;
189 }
190
191 return new_ste;
192
193err_exit:
194 mlx5dr_ste_free(new_ste, matcher, nic_matcher);
195 return NULL;
196}
197
198static void dr_rule_rehash_copy_ste_ctrl(struct mlx5dr_matcher *matcher,
199 struct mlx5dr_matcher_rx_tx *nic_matcher,
200 struct mlx5dr_ste *cur_ste,
201 struct mlx5dr_ste *new_ste)
202{
203 new_ste->next_htbl = cur_ste->next_htbl;
204 new_ste->ste_chain_location = cur_ste->ste_chain_location;
205
206 if (!mlx5dr_ste_is_last_in_rule(nic_matcher, new_ste->ste_chain_location))
207 new_ste->next_htbl->pointing_ste = new_ste;
208
209 /* We need to copy the refcount since this ste
210 * may have been traversed several times
211 */
212 refcount_set(&new_ste->refcount, refcount_read(&cur_ste->refcount));
213
214 /* Link old STEs rule_mem list to the new ste */
215 mlx5dr_rule_update_rule_member(cur_ste, new_ste);
216 INIT_LIST_HEAD(&new_ste->rule_list);
217 list_splice_tail_init(&cur_ste->rule_list, &new_ste->rule_list);
218}
219
220static struct mlx5dr_ste *
221dr_rule_rehash_copy_ste(struct mlx5dr_matcher *matcher,
222 struct mlx5dr_matcher_rx_tx *nic_matcher,
223 struct mlx5dr_ste *cur_ste,
224 struct mlx5dr_ste_htbl *new_htbl,
225 struct list_head *update_list)
226{
227 struct mlx5dr_ste_send_info *ste_info;
228 bool use_update_list = false;
229 u8 hw_ste[DR_STE_SIZE] = {};
230 struct mlx5dr_ste *new_ste;
231 int new_idx;
232 u8 sb_idx;
233
234 /* Copy STE mask from the matcher */
235 sb_idx = cur_ste->ste_chain_location - 1;
236 mlx5dr_ste_set_bit_mask(hw_ste, nic_matcher->ste_builder[sb_idx].bit_mask);
237
238 /* Copy STE control and tag */
239 memcpy(hw_ste, cur_ste->hw_ste, DR_STE_SIZE_REDUCED);
240 mlx5dr_ste_set_miss_addr(hw_ste, nic_matcher->e_anchor->chunk->icm_addr);
241
242 new_idx = mlx5dr_ste_calc_hash_index(hw_ste, new_htbl);
243 new_ste = &new_htbl->ste_arr[new_idx];
244
245 if (mlx5dr_ste_not_used_ste(new_ste)) {
246 mlx5dr_htbl_get(new_htbl);
247 list_add_tail(&new_ste->miss_list_node,
248 mlx5dr_ste_get_miss_list(new_ste));
249 } else {
250 new_ste = dr_rule_rehash_handle_collision(matcher,
251 nic_matcher,
252 update_list,
253 new_ste,
254 hw_ste);
255 if (!new_ste) {
256 mlx5dr_dbg(matcher->tbl->dmn, "Failed adding collision entry, index: %d\n",
257 new_idx);
258 return NULL;
259 }
260 new_htbl->ctrl.num_of_collisions++;
261 use_update_list = true;
262 }
263
264 memcpy(new_ste->hw_ste, hw_ste, DR_STE_SIZE_REDUCED);
265
266 new_htbl->ctrl.num_of_valid_entries++;
267
268 if (use_update_list) {
269 ste_info = kzalloc(sizeof(*ste_info), GFP_KERNEL);
270 if (!ste_info)
271 goto err_exit;
272
273 mlx5dr_send_fill_and_append_ste_send_info(new_ste, DR_STE_SIZE, 0,
274 hw_ste, ste_info,
275 update_list, true);
276 }
277
278 dr_rule_rehash_copy_ste_ctrl(matcher, nic_matcher, cur_ste, new_ste);
279
280 return new_ste;
281
282err_exit:
283 mlx5dr_ste_free(new_ste, matcher, nic_matcher);
284 return NULL;
285}
286
287static int dr_rule_rehash_copy_miss_list(struct mlx5dr_matcher *matcher,
288 struct mlx5dr_matcher_rx_tx *nic_matcher,
289 struct list_head *cur_miss_list,
290 struct mlx5dr_ste_htbl *new_htbl,
291 struct list_head *update_list)
292{
293 struct mlx5dr_ste *tmp_ste, *cur_ste, *new_ste;
294
295 if (list_empty(cur_miss_list))
296 return 0;
297
298 list_for_each_entry_safe(cur_ste, tmp_ste, cur_miss_list, miss_list_node) {
299 new_ste = dr_rule_rehash_copy_ste(matcher,
300 nic_matcher,
301 cur_ste,
302 new_htbl,
303 update_list);
304 if (!new_ste)
305 goto err_insert;
306
307 list_del(&cur_ste->miss_list_node);
308 mlx5dr_htbl_put(cur_ste->htbl);
309 }
310 return 0;
311
312err_insert:
313 mlx5dr_err(matcher->tbl->dmn, "Fatal error during resize\n");
314 WARN_ON(true);
315 return -EINVAL;
316}
317
318static int dr_rule_rehash_copy_htbl(struct mlx5dr_matcher *matcher,
319 struct mlx5dr_matcher_rx_tx *nic_matcher,
320 struct mlx5dr_ste_htbl *cur_htbl,
321 struct mlx5dr_ste_htbl *new_htbl,
322 struct list_head *update_list)
323{
324 struct mlx5dr_ste *cur_ste;
325 int cur_entries;
326 int err = 0;
327 int i;
328
329 cur_entries = mlx5dr_icm_pool_chunk_size_to_entries(cur_htbl->chunk_size);
330
331 if (cur_entries < 1) {
332 mlx5dr_dbg(matcher->tbl->dmn, "Invalid number of entries\n");
333 return -EINVAL;
334 }
335
336 for (i = 0; i < cur_entries; i++) {
337 cur_ste = &cur_htbl->ste_arr[i];
338 if (mlx5dr_ste_not_used_ste(cur_ste)) /* Empty, nothing to copy */
339 continue;
340
341 err = dr_rule_rehash_copy_miss_list(matcher,
342 nic_matcher,
343 mlx5dr_ste_get_miss_list(cur_ste),
344 new_htbl,
345 update_list);
346 if (err)
347 goto clean_copy;
348 }
349
350clean_copy:
351 return err;
352}
353
354static struct mlx5dr_ste_htbl *
355dr_rule_rehash_htbl(struct mlx5dr_rule *rule,
356 struct mlx5dr_rule_rx_tx *nic_rule,
357 struct mlx5dr_ste_htbl *cur_htbl,
358 u8 ste_location,
359 struct list_head *update_list,
360 enum mlx5dr_icm_chunk_size new_size)
361{
362 struct mlx5dr_ste_send_info *del_ste_info, *tmp_ste_info;
363 struct mlx5dr_matcher *matcher = rule->matcher;
364 struct mlx5dr_domain *dmn = matcher->tbl->dmn;
365 struct mlx5dr_matcher_rx_tx *nic_matcher;
366 struct mlx5dr_ste_send_info *ste_info;
367 struct mlx5dr_htbl_connect_info info;
368 struct mlx5dr_domain_rx_tx *nic_dmn;
369 u8 formatted_ste[DR_STE_SIZE] = {};
370 LIST_HEAD(rehash_table_send_list);
371 struct mlx5dr_ste *ste_to_update;
372 struct mlx5dr_ste_htbl *new_htbl;
373 int err;
374
375 nic_matcher = nic_rule->nic_matcher;
376 nic_dmn = nic_matcher->nic_tbl->nic_dmn;
377
378 ste_info = kzalloc(sizeof(*ste_info), GFP_KERNEL);
379 if (!ste_info)
380 return NULL;
381
382 new_htbl = mlx5dr_ste_htbl_alloc(dmn->ste_icm_pool,
383 new_size,
384 cur_htbl->lu_type,
385 cur_htbl->byte_mask);
386 if (!new_htbl) {
387 mlx5dr_err(dmn, "Failed to allocate new hash table\n");
388 goto free_ste_info;
389 }
390
391 /* Write new table to HW */
392 info.type = CONNECT_MISS;
393 info.miss_icm_addr = nic_matcher->e_anchor->chunk->icm_addr;
394 mlx5dr_ste_set_formatted_ste(dmn->info.caps.gvmi,
395 nic_dmn,
396 new_htbl,
397 formatted_ste,
398 &info);
399
400 new_htbl->pointing_ste = cur_htbl->pointing_ste;
401 new_htbl->pointing_ste->next_htbl = new_htbl;
402 err = dr_rule_rehash_copy_htbl(matcher,
403 nic_matcher,
404 cur_htbl,
405 new_htbl,
406 &rehash_table_send_list);
407 if (err)
408 goto free_new_htbl;
409
410 if (mlx5dr_send_postsend_htbl(dmn, new_htbl, formatted_ste,
411 nic_matcher->ste_builder[ste_location - 1].bit_mask)) {
412 mlx5dr_err(dmn, "Failed writing table to HW\n");
413 goto free_new_htbl;
414 }
415
416 /* Writing to the hw is done in regular order of rehash_table_send_list,
417 * in order to have the origin data written before the miss address of
418 * collision entries, if exists.
419 */
420 if (dr_rule_send_update_list(&rehash_table_send_list, dmn, false)) {
421 mlx5dr_err(dmn, "Failed updating table to HW\n");
422 goto free_ste_list;
423 }
424
425 /* Connect previous hash table to current */
426 if (ste_location == 1) {
427 /* The previous table is an anchor, anchors size is always one STE */
428 struct mlx5dr_ste_htbl *prev_htbl = cur_htbl->pointing_ste->htbl;
429
430 /* On matcher s_anchor we keep an extra refcount */
431 mlx5dr_htbl_get(new_htbl);
432 mlx5dr_htbl_put(cur_htbl);
433
434 nic_matcher->s_htbl = new_htbl;
435
436 /* It is safe to operate dr_ste_set_hit_addr on the hw_ste here
437 * (48B len) which works only on first 32B
438 */
439 mlx5dr_ste_set_hit_addr(prev_htbl->ste_arr[0].hw_ste,
440 new_htbl->chunk->icm_addr,
441 new_htbl->chunk->num_of_entries);
442
443 ste_to_update = &prev_htbl->ste_arr[0];
444 } else {
445 mlx5dr_ste_set_hit_addr_by_next_htbl(cur_htbl->pointing_ste->hw_ste,
446 new_htbl);
447 ste_to_update = cur_htbl->pointing_ste;
448 }
449
450 mlx5dr_send_fill_and_append_ste_send_info(ste_to_update, DR_STE_SIZE_REDUCED,
451 0, ste_to_update->hw_ste, ste_info,
452 update_list, false);
453
454 return new_htbl;
455
456free_ste_list:
457 /* Clean all ste_info's from the new table */
458 list_for_each_entry_safe(del_ste_info, tmp_ste_info,
459 &rehash_table_send_list, send_list) {
460 list_del(&del_ste_info->send_list);
461 kfree(del_ste_info);
462 }
463
464free_new_htbl:
465 mlx5dr_ste_htbl_free(new_htbl);
466free_ste_info:
467 kfree(ste_info);
468 mlx5dr_info(dmn, "Failed creating rehash table\n");
469 return NULL;
470}
471
472static struct mlx5dr_ste_htbl *dr_rule_rehash(struct mlx5dr_rule *rule,
473 struct mlx5dr_rule_rx_tx *nic_rule,
474 struct mlx5dr_ste_htbl *cur_htbl,
475 u8 ste_location,
476 struct list_head *update_list)
477{
478 struct mlx5dr_domain *dmn = rule->matcher->tbl->dmn;
479 enum mlx5dr_icm_chunk_size new_size;
480
481 new_size = mlx5dr_icm_next_higher_chunk(cur_htbl->chunk_size);
482 new_size = min_t(u32, new_size, dmn->info.max_log_sw_icm_sz);
483
484 if (new_size == cur_htbl->chunk_size)
485 return NULL; /* Skip rehash, we already at the max size */
486
487 return dr_rule_rehash_htbl(rule, nic_rule, cur_htbl, ste_location,
488 update_list, new_size);
489}
490
491static struct mlx5dr_ste *
492dr_rule_handle_collision(struct mlx5dr_matcher *matcher,
493 struct mlx5dr_matcher_rx_tx *nic_matcher,
494 struct mlx5dr_ste *ste,
495 u8 *hw_ste,
496 struct list_head *miss_list,
497 struct list_head *send_list)
498{
499 struct mlx5dr_ste_send_info *ste_info;
500 struct mlx5dr_ste *new_ste;
501
502 ste_info = kzalloc(sizeof(*ste_info), GFP_KERNEL);
503 if (!ste_info)
504 return NULL;
505
506 new_ste = dr_rule_create_collision_entry(matcher, nic_matcher, hw_ste, ste);
507 if (!new_ste)
508 goto free_send_info;
509
510 if (dr_rule_append_to_miss_list(new_ste, miss_list, send_list)) {
511 mlx5dr_dbg(matcher->tbl->dmn, "Failed to update prev miss_list\n");
512 goto err_exit;
513 }
514
515 mlx5dr_send_fill_and_append_ste_send_info(new_ste, DR_STE_SIZE, 0, hw_ste,
516 ste_info, send_list, false);
517
518 ste->htbl->ctrl.num_of_collisions++;
519 ste->htbl->ctrl.num_of_valid_entries++;
520
521 return new_ste;
522
523err_exit:
524 mlx5dr_ste_free(new_ste, matcher, nic_matcher);
525free_send_info:
526 kfree(ste_info);
527 return NULL;
528}
529
530static void dr_rule_remove_action_members(struct mlx5dr_rule *rule)
531{
532 struct mlx5dr_rule_action_member *action_mem;
533 struct mlx5dr_rule_action_member *tmp;
534
535 list_for_each_entry_safe(action_mem, tmp, &rule->rule_actions_list, list) {
536 list_del(&action_mem->list);
537 refcount_dec(&action_mem->action->refcount);
538 kvfree(action_mem);
539 }
540}
541
542static int dr_rule_add_action_members(struct mlx5dr_rule *rule,
543 size_t num_actions,
544 struct mlx5dr_action *actions[])
545{
546 struct mlx5dr_rule_action_member *action_mem;
547 int i;
548
549 for (i = 0; i < num_actions; i++) {
550 action_mem = kvzalloc(sizeof(*action_mem), GFP_KERNEL);
551 if (!action_mem)
552 goto free_action_members;
553
554 action_mem->action = actions[i];
555 INIT_LIST_HEAD(&action_mem->list);
556 list_add_tail(&action_mem->list, &rule->rule_actions_list);
557 refcount_inc(&action_mem->action->refcount);
558 }
559
560 return 0;
561
562free_action_members:
563 dr_rule_remove_action_members(rule);
564 return -ENOMEM;
565}
566
567/* While the pointer of ste is no longer valid, like while moving ste to be
568 * the first in the miss_list, and to be in the origin table,
569 * all rule-members that are attached to this ste should update their ste member
570 * to the new pointer
571 */
572void mlx5dr_rule_update_rule_member(struct mlx5dr_ste *ste,
573 struct mlx5dr_ste *new_ste)
574{
575 struct mlx5dr_rule_member *rule_mem;
576
577 if (!list_empty(&ste->rule_list))
578 list_for_each_entry(rule_mem, &ste->rule_list, use_ste_list)
579 rule_mem->ste = new_ste;
580}
581
582static void dr_rule_clean_rule_members(struct mlx5dr_rule *rule,
583 struct mlx5dr_rule_rx_tx *nic_rule)
584{
585 struct mlx5dr_rule_member *rule_mem;
586 struct mlx5dr_rule_member *tmp_mem;
587
588 if (list_empty(&nic_rule->rule_members_list))
589 return;
590 list_for_each_entry_safe(rule_mem, tmp_mem, &nic_rule->rule_members_list, list) {
591 list_del(&rule_mem->list);
592 list_del(&rule_mem->use_ste_list);
593 mlx5dr_ste_put(rule_mem->ste, rule->matcher, nic_rule->nic_matcher);
594 kvfree(rule_mem);
595 }
596}
597
598static bool dr_rule_need_enlarge_hash(struct mlx5dr_ste_htbl *htbl,
599 struct mlx5dr_domain *dmn,
600 struct mlx5dr_domain_rx_tx *nic_dmn)
601{
602 struct mlx5dr_ste_htbl_ctrl *ctrl = &htbl->ctrl;
603
604 if (dmn->info.max_log_sw_icm_sz <= htbl->chunk_size)
605 return false;
606
607 if (!ctrl->may_grow)
608 return false;
609
610 if (ctrl->num_of_collisions >= ctrl->increase_threshold &&
611 (ctrl->num_of_valid_entries - ctrl->num_of_collisions) >= ctrl->increase_threshold)
612 return true;
613
614 return false;
615}
616
617static int dr_rule_add_member(struct mlx5dr_rule_rx_tx *nic_rule,
618 struct mlx5dr_ste *ste)
619{
620 struct mlx5dr_rule_member *rule_mem;
621
622 rule_mem = kvzalloc(sizeof(*rule_mem), GFP_KERNEL);
623 if (!rule_mem)
624 return -ENOMEM;
625
626 rule_mem->ste = ste;
627 list_add_tail(&rule_mem->list, &nic_rule->rule_members_list);
628
629 list_add_tail(&rule_mem->use_ste_list, &ste->rule_list);
630
631 return 0;
632}
633
634static int dr_rule_handle_action_stes(struct mlx5dr_rule *rule,
635 struct mlx5dr_rule_rx_tx *nic_rule,
636 struct list_head *send_ste_list,
637 struct mlx5dr_ste *last_ste,
638 u8 *hw_ste_arr,
639 u32 new_hw_ste_arr_sz)
640{
641 struct mlx5dr_matcher_rx_tx *nic_matcher = nic_rule->nic_matcher;
642 struct mlx5dr_ste_send_info *ste_info_arr[DR_ACTION_MAX_STES];
643 u8 num_of_builders = nic_matcher->num_of_builders;
644 struct mlx5dr_matcher *matcher = rule->matcher;
645 u8 *curr_hw_ste, *prev_hw_ste;
646 struct mlx5dr_ste *action_ste;
647 int i, k, ret;
648
649 /* Two cases:
650 * 1. num_of_builders is equal to new_hw_ste_arr_sz, the action in the ste
651 * 2. num_of_builders is less then new_hw_ste_arr_sz, new ste was added
652 * to support the action.
653 */
654 if (num_of_builders == new_hw_ste_arr_sz)
655 return 0;
656
657 for (i = num_of_builders, k = 0; i < new_hw_ste_arr_sz; i++, k++) {
658 curr_hw_ste = hw_ste_arr + i * DR_STE_SIZE;
659 prev_hw_ste = (i == 0) ? curr_hw_ste : hw_ste_arr + ((i - 1) * DR_STE_SIZE);
660 action_ste = dr_rule_create_collision_htbl(matcher,
661 nic_matcher,
662 curr_hw_ste);
663 if (!action_ste)
664 return -ENOMEM;
665
666 mlx5dr_ste_get(action_ste);
667
668 /* While free ste we go over the miss list, so add this ste to the list */
669 list_add_tail(&action_ste->miss_list_node,
670 mlx5dr_ste_get_miss_list(action_ste));
671
672 ste_info_arr[k] = kzalloc(sizeof(*ste_info_arr[k]),
673 GFP_KERNEL);
674 if (!ste_info_arr[k])
675 goto err_exit;
676
677 /* Point current ste to the new action */
678 mlx5dr_ste_set_hit_addr_by_next_htbl(prev_hw_ste, action_ste->htbl);
679 ret = dr_rule_add_member(nic_rule, action_ste);
680 if (ret) {
681 mlx5dr_dbg(matcher->tbl->dmn, "Failed adding rule member\n");
682 goto free_ste_info;
683 }
684 mlx5dr_send_fill_and_append_ste_send_info(action_ste, DR_STE_SIZE, 0,
685 curr_hw_ste,
686 ste_info_arr[k],
687 send_ste_list, false);
688 }
689
690 return 0;
691
692free_ste_info:
693 kfree(ste_info_arr[k]);
694err_exit:
695 mlx5dr_ste_put(action_ste, matcher, nic_matcher);
696 return -ENOMEM;
697}
698
699static int dr_rule_handle_empty_entry(struct mlx5dr_matcher *matcher,
700 struct mlx5dr_matcher_rx_tx *nic_matcher,
701 struct mlx5dr_ste_htbl *cur_htbl,
702 struct mlx5dr_ste *ste,
703 u8 ste_location,
704 u8 *hw_ste,
705 struct list_head *miss_list,
706 struct list_head *send_list)
707{
708 struct mlx5dr_ste_send_info *ste_info;
709
710 /* Take ref on table, only on first time this ste is used */
711 mlx5dr_htbl_get(cur_htbl);
712
713 /* new entry -> new branch */
714 list_add_tail(&ste->miss_list_node, miss_list);
715
716 mlx5dr_ste_set_miss_addr(hw_ste, nic_matcher->e_anchor->chunk->icm_addr);
717
718 ste->ste_chain_location = ste_location;
719
720 ste_info = kzalloc(sizeof(*ste_info), GFP_KERNEL);
721 if (!ste_info)
722 goto clean_ste_setting;
723
724 if (mlx5dr_ste_create_next_htbl(matcher,
725 nic_matcher,
726 ste,
727 hw_ste,
728 DR_CHUNK_SIZE_1)) {
729 mlx5dr_dbg(matcher->tbl->dmn, "Failed allocating table\n");
730 goto clean_ste_info;
731 }
732
733 cur_htbl->ctrl.num_of_valid_entries++;
734
735 mlx5dr_send_fill_and_append_ste_send_info(ste, DR_STE_SIZE, 0, hw_ste,
736 ste_info, send_list, false);
737
738 return 0;
739
740clean_ste_info:
741 kfree(ste_info);
742clean_ste_setting:
743 list_del_init(&ste->miss_list_node);
744 mlx5dr_htbl_put(cur_htbl);
745
746 return -ENOMEM;
747}
748
749static struct mlx5dr_ste *
750dr_rule_handle_ste_branch(struct mlx5dr_rule *rule,
751 struct mlx5dr_rule_rx_tx *nic_rule,
752 struct list_head *send_ste_list,
753 struct mlx5dr_ste_htbl *cur_htbl,
754 u8 *hw_ste,
755 u8 ste_location,
756 struct mlx5dr_ste_htbl **put_htbl)
757{
758 struct mlx5dr_matcher *matcher = rule->matcher;
759 struct mlx5dr_domain *dmn = matcher->tbl->dmn;
760 struct mlx5dr_matcher_rx_tx *nic_matcher;
761 struct mlx5dr_domain_rx_tx *nic_dmn;
762 struct mlx5dr_ste_htbl *new_htbl;
763 struct mlx5dr_ste *matched_ste;
764 struct list_head *miss_list;
765 bool skip_rehash = false;
766 struct mlx5dr_ste *ste;
767 int index;
768
769 nic_matcher = nic_rule->nic_matcher;
770 nic_dmn = nic_matcher->nic_tbl->nic_dmn;
771
772again:
773 index = mlx5dr_ste_calc_hash_index(hw_ste, cur_htbl);
774 miss_list = &cur_htbl->chunk->miss_list[index];
775 ste = &cur_htbl->ste_arr[index];
776
777 if (mlx5dr_ste_not_used_ste(ste)) {
778 if (dr_rule_handle_empty_entry(matcher, nic_matcher, cur_htbl,
779 ste, ste_location,
780 hw_ste, miss_list,
781 send_ste_list))
782 return NULL;
783 } else {
784 /* Hash table index in use, check if this ste is in the miss list */
785 matched_ste = dr_rule_find_ste_in_miss_list(miss_list, hw_ste);
786 if (matched_ste) {
787 /* If it is last STE in the chain, and has the same tag
788 * it means that all the previous stes are the same,
789 * if so, this rule is duplicated.
790 */
791 if (mlx5dr_ste_is_last_in_rule(nic_matcher,
792 matched_ste->ste_chain_location)) {
793 mlx5dr_info(dmn, "Duplicate rule inserted, aborting!!\n");
794 return NULL;
795 }
796 return matched_ste;
797 }
798
799 if (!skip_rehash && dr_rule_need_enlarge_hash(cur_htbl, dmn, nic_dmn)) {
800 /* Hash table index in use, try to resize of the hash */
801 skip_rehash = true;
802
803 /* Hold the table till we update.
804 * Release in dr_rule_create_rule()
805 */
806 *put_htbl = cur_htbl;
807 mlx5dr_htbl_get(cur_htbl);
808
809 new_htbl = dr_rule_rehash(rule, nic_rule, cur_htbl,
810 ste_location, send_ste_list);
811 if (!new_htbl) {
812 mlx5dr_htbl_put(cur_htbl);
813 mlx5dr_info(dmn, "failed creating rehash table, htbl-log_size: %d\n",
814 cur_htbl->chunk_size);
815 } else {
816 cur_htbl = new_htbl;
817 }
818 goto again;
819 } else {
820 /* Hash table index in use, add another collision (miss) */
821 ste = dr_rule_handle_collision(matcher,
822 nic_matcher,
823 ste,
824 hw_ste,
825 miss_list,
826 send_ste_list);
827 if (!ste) {
828 mlx5dr_dbg(dmn, "failed adding collision entry, index: %d\n",
829 index);
830 return NULL;
831 }
832 }
833 }
834 return ste;
835}
836
837static bool dr_rule_cmp_value_to_mask(u8 *mask, u8 *value,
838 u32 s_idx, u32 e_idx)
839{
840 u32 i;
841
842 for (i = s_idx; i < e_idx; i++) {
843 if (value[i] & ~mask[i]) {
844 pr_info("Rule parameters contains a value not specified by mask\n");
845 return false;
846 }
847 }
848 return true;
849}
850
851static bool dr_rule_verify(struct mlx5dr_matcher *matcher,
852 struct mlx5dr_match_parameters *value,
853 struct mlx5dr_match_param *param)
854{
855 u8 match_criteria = matcher->match_criteria;
856 size_t value_size = value->match_sz;
857 u8 *mask_p = (u8 *)&matcher->mask;
858 u8 *param_p = (u8 *)param;
859 u32 s_idx, e_idx;
860
861 if (!value_size ||
862 (value_size > sizeof(struct mlx5dr_match_param) ||
863 (value_size % sizeof(u32)))) {
864 mlx5dr_dbg(matcher->tbl->dmn, "Rule parameters length is incorrect\n");
865 return false;
866 }
867
868 mlx5dr_ste_copy_param(matcher->match_criteria, param, value);
869
870 if (match_criteria & DR_MATCHER_CRITERIA_OUTER) {
871 s_idx = offsetof(struct mlx5dr_match_param, outer);
872 e_idx = min(s_idx + sizeof(param->outer), value_size);
873
874 if (!dr_rule_cmp_value_to_mask(mask_p, param_p, s_idx, e_idx)) {
875 mlx5dr_dbg(matcher->tbl->dmn, "Rule outer parameters contains a value not specified by mask\n");
876 return false;
877 }
878 }
879
880 if (match_criteria & DR_MATCHER_CRITERIA_MISC) {
881 s_idx = offsetof(struct mlx5dr_match_param, misc);
882 e_idx = min(s_idx + sizeof(param->misc), value_size);
883
884 if (!dr_rule_cmp_value_to_mask(mask_p, param_p, s_idx, e_idx)) {
885 mlx5dr_dbg(matcher->tbl->dmn, "Rule misc parameters contains a value not specified by mask\n");
886 return false;
887 }
888 }
889
890 if (match_criteria & DR_MATCHER_CRITERIA_INNER) {
891 s_idx = offsetof(struct mlx5dr_match_param, inner);
892 e_idx = min(s_idx + sizeof(param->inner), value_size);
893
894 if (!dr_rule_cmp_value_to_mask(mask_p, param_p, s_idx, e_idx)) {
895 mlx5dr_dbg(matcher->tbl->dmn, "Rule inner parameters contains a value not specified by mask\n");
896 return false;
897 }
898 }
899
900 if (match_criteria & DR_MATCHER_CRITERIA_MISC2) {
901 s_idx = offsetof(struct mlx5dr_match_param, misc2);
902 e_idx = min(s_idx + sizeof(param->misc2), value_size);
903
904 if (!dr_rule_cmp_value_to_mask(mask_p, param_p, s_idx, e_idx)) {
905 mlx5dr_dbg(matcher->tbl->dmn, "Rule misc2 parameters contains a value not specified by mask\n");
906 return false;
907 }
908 }
909
910 if (match_criteria & DR_MATCHER_CRITERIA_MISC3) {
911 s_idx = offsetof(struct mlx5dr_match_param, misc3);
912 e_idx = min(s_idx + sizeof(param->misc3), value_size);
913
914 if (!dr_rule_cmp_value_to_mask(mask_p, param_p, s_idx, e_idx)) {
915 mlx5dr_dbg(matcher->tbl->dmn, "Rule misc3 parameters contains a value not specified by mask\n");
916 return false;
917 }
918 }
919 return true;
920}
921
922static int dr_rule_destroy_rule_nic(struct mlx5dr_rule *rule,
923 struct mlx5dr_rule_rx_tx *nic_rule)
924{
925 dr_rule_clean_rule_members(rule, nic_rule);
926 return 0;
927}
928
929static int dr_rule_destroy_rule_fdb(struct mlx5dr_rule *rule)
930{
931 dr_rule_destroy_rule_nic(rule, &rule->rx);
932 dr_rule_destroy_rule_nic(rule, &rule->tx);
933 return 0;
934}
935
936static int dr_rule_destroy_rule(struct mlx5dr_rule *rule)
937{
938 struct mlx5dr_domain *dmn = rule->matcher->tbl->dmn;
939
940 switch (dmn->type) {
941 case MLX5DR_DOMAIN_TYPE_NIC_RX:
942 dr_rule_destroy_rule_nic(rule, &rule->rx);
943 break;
944 case MLX5DR_DOMAIN_TYPE_NIC_TX:
945 dr_rule_destroy_rule_nic(rule, &rule->tx);
946 break;
947 case MLX5DR_DOMAIN_TYPE_FDB:
948 dr_rule_destroy_rule_fdb(rule);
949 break;
950 default:
951 return -EINVAL;
952 }
953
954 dr_rule_remove_action_members(rule);
955 kfree(rule);
956 return 0;
957}
958
959static bool dr_rule_is_ipv6(struct mlx5dr_match_param *param)
960{
961 return (param->outer.ip_version == 6 ||
962 param->inner.ip_version == 6 ||
963 param->outer.ethertype == ETH_P_IPV6 ||
964 param->inner.ethertype == ETH_P_IPV6);
965}
966
967static bool dr_rule_skip(enum mlx5dr_domain_type domain,
968 enum mlx5dr_ste_entry_type ste_type,
969 struct mlx5dr_match_param *mask,
970 struct mlx5dr_match_param *value)
971{
972 if (domain != MLX5DR_DOMAIN_TYPE_FDB)
973 return false;
974
975 if (mask->misc.source_port) {
976 if (ste_type == MLX5DR_STE_TYPE_RX)
977 if (value->misc.source_port != WIRE_PORT)
978 return true;
979
980 if (ste_type == MLX5DR_STE_TYPE_TX)
981 if (value->misc.source_port == WIRE_PORT)
982 return true;
983 }
984
985 /* Metadata C can be used to describe the source vport */
986 if (mask->misc2.metadata_reg_c_0) {
987 if (ste_type == MLX5DR_STE_TYPE_RX)
988 if ((value->misc2.metadata_reg_c_0 & WIRE_PORT) != WIRE_PORT)
989 return true;
990
991 if (ste_type == MLX5DR_STE_TYPE_TX)
992 if ((value->misc2.metadata_reg_c_0 & WIRE_PORT) == WIRE_PORT)
993 return true;
994 }
995 return false;
996}
997
998static int
999dr_rule_create_rule_nic(struct mlx5dr_rule *rule,
1000 struct mlx5dr_rule_rx_tx *nic_rule,
1001 struct mlx5dr_match_param *param,
1002 size_t num_actions,
1003 struct mlx5dr_action *actions[])
1004{
1005 struct mlx5dr_ste_send_info *ste_info, *tmp_ste_info;
1006 struct mlx5dr_matcher *matcher = rule->matcher;
1007 struct mlx5dr_domain *dmn = matcher->tbl->dmn;
1008 struct mlx5dr_matcher_rx_tx *nic_matcher;
1009 struct mlx5dr_domain_rx_tx *nic_dmn;
1010 struct mlx5dr_ste_htbl *htbl = NULL;
1011 struct mlx5dr_ste_htbl *cur_htbl;
1012 struct mlx5dr_ste *ste = NULL;
1013 LIST_HEAD(send_ste_list);
1014 u8 *hw_ste_arr = NULL;
1015 u32 new_hw_ste_arr_sz;
1016 int ret, i;
1017
1018 nic_matcher = nic_rule->nic_matcher;
1019 nic_dmn = nic_matcher->nic_tbl->nic_dmn;
1020
1021 INIT_LIST_HEAD(&nic_rule->rule_members_list);
1022
1023 if (dr_rule_skip(dmn->type, nic_dmn->ste_type, &matcher->mask, param))
1024 return 0;
1025
1026 ret = mlx5dr_matcher_select_builders(matcher,
1027 nic_matcher,
1028 dr_rule_is_ipv6(param));
1029 if (ret)
1030 goto out_err;
1031
1032 hw_ste_arr = kzalloc(DR_RULE_MAX_STE_CHAIN * DR_STE_SIZE, GFP_KERNEL);
1033 if (!hw_ste_arr) {
1034 ret = -ENOMEM;
1035 goto out_err;
1036 }
1037
1038 /* Set the tag values inside the ste array */
1039 ret = mlx5dr_ste_build_ste_arr(matcher, nic_matcher, param, hw_ste_arr);
1040 if (ret)
1041 goto free_hw_ste;
1042
1043 /* Set the actions values/addresses inside the ste array */
1044 ret = mlx5dr_actions_build_ste_arr(matcher, nic_matcher, actions,
1045 num_actions, hw_ste_arr,
1046 &new_hw_ste_arr_sz);
1047 if (ret)
1048 goto free_hw_ste;
1049
1050 cur_htbl = nic_matcher->s_htbl;
1051
1052 /* Go over the array of STEs, and build dr_ste accordingly.
1053 * The loop is over only the builders which are equal or less to the
1054 * number of stes, in case we have actions that lives in other stes.
1055 */
1056 for (i = 0; i < nic_matcher->num_of_builders; i++) {
1057 /* Calculate CRC and keep new ste entry */
1058 u8 *cur_hw_ste_ent = hw_ste_arr + (i * DR_STE_SIZE);
1059
1060 ste = dr_rule_handle_ste_branch(rule,
1061 nic_rule,
1062 &send_ste_list,
1063 cur_htbl,
1064 cur_hw_ste_ent,
1065 i + 1,
1066 &htbl);
1067 if (!ste) {
1068 mlx5dr_err(dmn, "Failed creating next branch\n");
1069 ret = -ENOENT;
1070 goto free_rule;
1071 }
1072
1073 cur_htbl = ste->next_htbl;
1074
1075 /* Keep all STEs in the rule struct */
1076 ret = dr_rule_add_member(nic_rule, ste);
1077 if (ret) {
1078 mlx5dr_dbg(dmn, "Failed adding rule member index %d\n", i);
1079 goto free_ste;
1080 }
1081
1082 mlx5dr_ste_get(ste);
1083 }
1084
1085 /* Connect actions */
1086 ret = dr_rule_handle_action_stes(rule, nic_rule, &send_ste_list,
1087 ste, hw_ste_arr, new_hw_ste_arr_sz);
1088 if (ret) {
1089 mlx5dr_dbg(dmn, "Failed apply actions\n");
1090 goto free_rule;
1091 }
1092 ret = dr_rule_send_update_list(&send_ste_list, dmn, true);
1093 if (ret) {
1094 mlx5dr_err(dmn, "Failed sending ste!\n");
1095 goto free_rule;
1096 }
1097
1098 if (htbl)
1099 mlx5dr_htbl_put(htbl);
1100
1101 return 0;
1102
1103free_ste:
1104 mlx5dr_ste_put(ste, matcher, nic_matcher);
1105free_rule:
1106 dr_rule_clean_rule_members(rule, nic_rule);
1107 /* Clean all ste_info's */
1108 list_for_each_entry_safe(ste_info, tmp_ste_info, &send_ste_list, send_list) {
1109 list_del(&ste_info->send_list);
1110 kfree(ste_info);
1111 }
1112free_hw_ste:
1113 kfree(hw_ste_arr);
1114out_err:
1115 return ret;
1116}
1117
1118static int
1119dr_rule_create_rule_fdb(struct mlx5dr_rule *rule,
1120 struct mlx5dr_match_param *param,
1121 size_t num_actions,
1122 struct mlx5dr_action *actions[])
1123{
1124 struct mlx5dr_match_param copy_param = {};
1125 int ret;
1126
1127 /* Copy match_param since they will be consumed during the first
1128 * nic_rule insertion.
1129 */
1130 memcpy(&copy_param, param, sizeof(struct mlx5dr_match_param));
1131
1132 ret = dr_rule_create_rule_nic(rule, &rule->rx, param,
1133 num_actions, actions);
1134 if (ret)
1135 return ret;
1136
1137 ret = dr_rule_create_rule_nic(rule, &rule->tx, &copy_param,
1138 num_actions, actions);
1139 if (ret)
1140 goto destroy_rule_nic_rx;
1141
1142 return 0;
1143
1144destroy_rule_nic_rx:
1145 dr_rule_destroy_rule_nic(rule, &rule->rx);
1146 return ret;
1147}
1148
1149static struct mlx5dr_rule *
1150dr_rule_create_rule(struct mlx5dr_matcher *matcher,
1151 struct mlx5dr_match_parameters *value,
1152 size_t num_actions,
1153 struct mlx5dr_action *actions[])
1154{
1155 struct mlx5dr_domain *dmn = matcher->tbl->dmn;
1156 struct mlx5dr_match_param param = {};
1157 struct mlx5dr_rule *rule;
1158 int ret;
1159
1160 if (!dr_rule_verify(matcher, value, &param))
1161 return NULL;
1162
1163 rule = kzalloc(sizeof(*rule), GFP_KERNEL);
1164 if (!rule)
1165 return NULL;
1166
1167 rule->matcher = matcher;
1168 INIT_LIST_HEAD(&rule->rule_actions_list);
1169
1170 ret = dr_rule_add_action_members(rule, num_actions, actions);
1171 if (ret)
1172 goto free_rule;
1173
1174 switch (dmn->type) {
1175 case MLX5DR_DOMAIN_TYPE_NIC_RX:
1176 rule->rx.nic_matcher = &matcher->rx;
1177 ret = dr_rule_create_rule_nic(rule, &rule->rx, &param,
1178 num_actions, actions);
1179 break;
1180 case MLX5DR_DOMAIN_TYPE_NIC_TX:
1181 rule->tx.nic_matcher = &matcher->tx;
1182 ret = dr_rule_create_rule_nic(rule, &rule->tx, &param,
1183 num_actions, actions);
1184 break;
1185 case MLX5DR_DOMAIN_TYPE_FDB:
1186 rule->rx.nic_matcher = &matcher->rx;
1187 rule->tx.nic_matcher = &matcher->tx;
1188 ret = dr_rule_create_rule_fdb(rule, &param,
1189 num_actions, actions);
1190 break;
1191 default:
1192 ret = -EINVAL;
1193 break;
1194 }
1195
1196 if (ret)
1197 goto remove_action_members;
1198
1199 return rule;
1200
1201remove_action_members:
1202 dr_rule_remove_action_members(rule);
1203free_rule:
1204 kfree(rule);
1205 mlx5dr_info(dmn, "Failed creating rule\n");
1206 return NULL;
1207}
1208
1209struct mlx5dr_rule *mlx5dr_rule_create(struct mlx5dr_matcher *matcher,
1210 struct mlx5dr_match_parameters *value,
1211 size_t num_actions,
1212 struct mlx5dr_action *actions[])
1213{
1214 struct mlx5dr_rule *rule;
1215
1216 mutex_lock(&matcher->tbl->dmn->mutex);
1217 refcount_inc(&matcher->refcount);
1218
1219 rule = dr_rule_create_rule(matcher, value, num_actions, actions);
1220 if (!rule)
1221 refcount_dec(&matcher->refcount);
1222
1223 mutex_unlock(&matcher->tbl->dmn->mutex);
1224
1225 return rule;
1226}
1227
1228int mlx5dr_rule_destroy(struct mlx5dr_rule *rule)
1229{
1230 struct mlx5dr_matcher *matcher = rule->matcher;
1231 struct mlx5dr_table *tbl = rule->matcher->tbl;
1232 int ret;
1233
1234 mutex_lock(&tbl->dmn->mutex);
1235
1236 ret = dr_rule_destroy_rule(rule);
1237
1238 mutex_unlock(&tbl->dmn->mutex);
1239
1240 if (!ret)
1241 refcount_dec(&matcher->refcount);
1242 return ret;
1243}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c+976−0
@@ -0,0 +1,976 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2/* Copyright (c) 2019 Mellanox Technologies. */
3
4#include "dr_types.h"
5
6#define QUEUE_SIZE 128
7#define SIGNAL_PER_DIV_QUEUE 16
8#define TH_NUMS_TO_DRAIN 2
9
10enum { CQ_OK = 0, CQ_EMPTY = -1, CQ_POLL_ERR = -2 };
11
12struct dr_data_seg {
13 u64 addr;
14 u32 length;
15 u32 lkey;
16 unsigned int send_flags;
17};
18
19struct postsend_info {
20 struct dr_data_seg write;
21 struct dr_data_seg read;
22 u64 remote_addr;
23 u32 rkey;
24};
25
26struct dr_qp_rtr_attr {
27 struct mlx5dr_cmd_gid_attr dgid_attr;
28 enum ib_mtu mtu;
29 u32 qp_num;
30 u16 port_num;
31 u8 min_rnr_timer;
32 u8 sgid_index;
33 u16 udp_src_port;
34};
35
36struct dr_qp_rts_attr {
37 u8 timeout;
38 u8 retry_cnt;
39 u8 rnr_retry;
40};
41
42struct dr_qp_init_attr {
43 u32 cqn;
44 u32 pdn;
45 u32 max_send_wr;
46 struct mlx5_uars_page *uar;
47};
48
49static int dr_parse_cqe(struct mlx5dr_cq *dr_cq, struct mlx5_cqe64 *cqe64)
50{
51 unsigned int idx;
52 u8 opcode;
53
54 opcode = get_cqe_opcode(cqe64);
55 if (opcode == MLX5_CQE_REQ_ERR) {
56 idx = be16_to_cpu(cqe64->wqe_counter) &
57 (dr_cq->qp->sq.wqe_cnt - 1);
58 dr_cq->qp->sq.cc = dr_cq->qp->sq.wqe_head[idx] + 1;
59 } else if (opcode == MLX5_CQE_RESP_ERR) {
60 ++dr_cq->qp->sq.cc;
61 } else {
62 idx = be16_to_cpu(cqe64->wqe_counter) &
63 (dr_cq->qp->sq.wqe_cnt - 1);
64 dr_cq->qp->sq.cc = dr_cq->qp->sq.wqe_head[idx] + 1;
65
66 return CQ_OK;
67 }
68
69 return CQ_POLL_ERR;
70}
71
72static int dr_cq_poll_one(struct mlx5dr_cq *dr_cq)
73{
74 struct mlx5_cqe64 *cqe64;
75 int err;
76
77 cqe64 = mlx5_cqwq_get_cqe(&dr_cq->wq);
78 if (!cqe64)
79 return CQ_EMPTY;
80
81 mlx5_cqwq_pop(&dr_cq->wq);
82 err = dr_parse_cqe(dr_cq, cqe64);
83 mlx5_cqwq_update_db_record(&dr_cq->wq);
84
85 return err;
86}
87
88static int dr_poll_cq(struct mlx5dr_cq *dr_cq, int ne)
89{
90 int npolled;
91 int err = 0;
92
93 for (npolled = 0; npolled < ne; ++npolled) {
94 err = dr_cq_poll_one(dr_cq);
95 if (err != CQ_OK)
96 break;
97 }
98
99 return err == CQ_POLL_ERR ? err : npolled;
100}
101
102static void dr_qp_event(struct mlx5_core_qp *mqp, int event)
103{
104 pr_info("DR QP event %u on QP #%u\n", event, mqp->qpn);
105}
106
107static struct mlx5dr_qp *dr_create_rc_qp(struct mlx5_core_dev *mdev,
108 struct dr_qp_init_attr *attr)
109{
110 u32 temp_qpc[MLX5_ST_SZ_DW(qpc)] = {};
111 struct mlx5_wq_param wqp;
112 struct mlx5dr_qp *dr_qp;
113 int inlen;
114 void *qpc;
115 void *in;
116 int err;
117
118 dr_qp = kzalloc(sizeof(*dr_qp), GFP_KERNEL);
119 if (!dr_qp)
120 return NULL;
121
122 wqp.buf_numa_node = mdev->priv.numa_node;
123 wqp.db_numa_node = mdev->priv.numa_node;
124
125 dr_qp->rq.pc = 0;
126 dr_qp->rq.cc = 0;
127 dr_qp->rq.wqe_cnt = 4;
128 dr_qp->sq.pc = 0;
129 dr_qp->sq.cc = 0;
130 dr_qp->sq.wqe_cnt = roundup_pow_of_two(attr->max_send_wr);
131
132 MLX5_SET(qpc, temp_qpc, log_rq_stride, ilog2(MLX5_SEND_WQE_DS) - 4);
133 MLX5_SET(qpc, temp_qpc, log_rq_size, ilog2(dr_qp->rq.wqe_cnt));
134 MLX5_SET(qpc, temp_qpc, log_sq_size, ilog2(dr_qp->sq.wqe_cnt));
135 err = mlx5_wq_qp_create(mdev, &wqp, temp_qpc, &dr_qp->wq,
136 &dr_qp->wq_ctrl);
137 if (err) {
138 mlx5_core_info(mdev, "Can't create QP WQ\n");
139 goto err_wq;
140 }
141
142 dr_qp->sq.wqe_head = kcalloc(dr_qp->sq.wqe_cnt,
143 sizeof(dr_qp->sq.wqe_head[0]),
144 GFP_KERNEL);
145
146 if (!dr_qp->sq.wqe_head) {
147 mlx5_core_warn(mdev, "Can't allocate wqe head\n");
148 goto err_wqe_head;
149 }
150
151 inlen = MLX5_ST_SZ_BYTES(create_qp_in) +
152 MLX5_FLD_SZ_BYTES(create_qp_in, pas[0]) *
153 dr_qp->wq_ctrl.buf.npages;
154 in = kvzalloc(inlen, GFP_KERNEL);
155 if (!in) {
156 err = -ENOMEM;
157 goto err_in;
158 }
159
160 qpc = MLX5_ADDR_OF(create_qp_in, in, qpc);
161 MLX5_SET(qpc, qpc, st, MLX5_QP_ST_RC);
162 MLX5_SET(qpc, qpc, pm_state, MLX5_QP_PM_MIGRATED);
163 MLX5_SET(qpc, qpc, pd, attr->pdn);
164 MLX5_SET(qpc, qpc, uar_page, attr->uar->index);
165 MLX5_SET(qpc, qpc, log_page_size,
166 dr_qp->wq_ctrl.buf.page_shift - MLX5_ADAPTER_PAGE_SHIFT);
167 MLX5_SET(qpc, qpc, fre, 1);
168 MLX5_SET(qpc, qpc, rlky, 1);
169 MLX5_SET(qpc, qpc, cqn_snd, attr->cqn);
170 MLX5_SET(qpc, qpc, cqn_rcv, attr->cqn);
171 MLX5_SET(qpc, qpc, log_rq_stride, ilog2(MLX5_SEND_WQE_DS) - 4);
172 MLX5_SET(qpc, qpc, log_rq_size, ilog2(dr_qp->rq.wqe_cnt));
173 MLX5_SET(qpc, qpc, rq_type, MLX5_NON_ZERO_RQ);
174 MLX5_SET(qpc, qpc, log_sq_size, ilog2(dr_qp->sq.wqe_cnt));
175 MLX5_SET64(qpc, qpc, dbr_addr, dr_qp->wq_ctrl.db.dma);
176 if (MLX5_CAP_GEN(mdev, cqe_version) == 1)
177 MLX5_SET(qpc, qpc, user_index, 0xFFFFFF);
178 mlx5_fill_page_frag_array(&dr_qp->wq_ctrl.buf,
179 (__be64 *)MLX5_ADDR_OF(create_qp_in,
180 in, pas));
181
182 err = mlx5_core_create_qp(mdev, &dr_qp->mqp, in, inlen);
183 kfree(in);
184
185 if (err) {
186 mlx5_core_warn(mdev, " Can't create QP\n");
187 goto err_in;
188 }
189 dr_qp->mqp.event = dr_qp_event;
190 dr_qp->uar = attr->uar;
191
192 return dr_qp;
193
194err_in:
195 kfree(dr_qp->sq.wqe_head);
196err_wqe_head:
197 mlx5_wq_destroy(&dr_qp->wq_ctrl);
198err_wq:
199 kfree(dr_qp);
200 return NULL;
201}
202
203static void dr_destroy_qp(struct mlx5_core_dev *mdev,
204 struct mlx5dr_qp *dr_qp)
205{
206 mlx5_core_destroy_qp(mdev, &dr_qp->mqp);
207 kfree(dr_qp->sq.wqe_head);
208 mlx5_wq_destroy(&dr_qp->wq_ctrl);
209 kfree(dr_qp);
210}
211
212static void dr_cmd_notify_hw(struct mlx5dr_qp *dr_qp, void *ctrl)
213{
214 dma_wmb();
215 *dr_qp->wq.sq.db = cpu_to_be32(dr_qp->sq.pc & 0xfffff);
216
217 /* After wmb() the hw aware of new work */
218 wmb();
219
220 mlx5_write64(ctrl, dr_qp->uar->map + MLX5_BF_OFFSET);
221}
222
223static void dr_rdma_segments(struct mlx5dr_qp *dr_qp, u64 remote_addr,
224 u32 rkey, struct dr_data_seg *data_seg,
225 u32 opcode, int nreq)
226{
227 struct mlx5_wqe_raddr_seg *wq_raddr;
228 struct mlx5_wqe_ctrl_seg *wq_ctrl;
229 struct mlx5_wqe_data_seg *wq_dseg;
230 unsigned int size;
231 unsigned int idx;
232
233 size = sizeof(*wq_ctrl) / 16 + sizeof(*wq_dseg) / 16 +
234 sizeof(*wq_raddr) / 16;
235
236 idx = dr_qp->sq.pc & (dr_qp->sq.wqe_cnt - 1);
237
238 wq_ctrl = mlx5_wq_cyc_get_wqe(&dr_qp->wq.sq, idx);
239 wq_ctrl->imm = 0;
240 wq_ctrl->fm_ce_se = (data_seg->send_flags) ?
241 MLX5_WQE_CTRL_CQ_UPDATE : 0;
242 wq_ctrl->opmod_idx_opcode = cpu_to_be32(((dr_qp->sq.pc & 0xffff) << 8) |
243 opcode);
244 wq_ctrl->qpn_ds = cpu_to_be32(size | dr_qp->mqp.qpn << 8);
245 wq_raddr = (void *)(wq_ctrl + 1);
246 wq_raddr->raddr = cpu_to_be64(remote_addr);
247 wq_raddr->rkey = cpu_to_be32(rkey);
248 wq_raddr->reserved = 0;
249
250 wq_dseg = (void *)(wq_raddr + 1);
251 wq_dseg->byte_count = cpu_to_be32(data_seg->length);
252 wq_dseg->lkey = cpu_to_be32(data_seg->lkey);
253 wq_dseg->addr = cpu_to_be64(data_seg->addr);
254
255 dr_qp->sq.wqe_head[idx] = dr_qp->sq.pc++;
256
257 if (nreq)
258 dr_cmd_notify_hw(dr_qp, wq_ctrl);
259}
260
261static void dr_post_send(struct mlx5dr_qp *dr_qp, struct postsend_info *send_info)
262{
263 dr_rdma_segments(dr_qp, send_info->remote_addr, send_info->rkey,
264 &send_info->write, MLX5_OPCODE_RDMA_WRITE, 0);
265 dr_rdma_segments(dr_qp, send_info->remote_addr, send_info->rkey,
266 &send_info->read, MLX5_OPCODE_RDMA_READ, 1);
267}
268
269/**
270 * mlx5dr_send_fill_and_append_ste_send_info: Add data to be sent
271 * with send_list parameters:
272 *
273 * @ste: The data that attached to this specific ste
274 * @size: of data to write
275 * @offset: of the data from start of the hw_ste entry
276 * @data: data
277 * @ste_info: ste to be sent with send_list
278 * @send_list: to append into it
279 * @copy_data: if true indicates that the data should be kept because
280 * it's not backuped any where (like in re-hash).
281 * if false, it lets the data to be updated after
282 * it was added to the list.
283 */
284void mlx5dr_send_fill_and_append_ste_send_info(struct mlx5dr_ste *ste, u16 size,
285 u16 offset, u8 *data,
286 struct mlx5dr_ste_send_info *ste_info,
287 struct list_head *send_list,
288 bool copy_data)
289{
290 ste_info->size = size;
291 ste_info->ste = ste;
292 ste_info->offset = offset;
293
294 if (copy_data) {
295 memcpy(ste_info->data_cont, data, size);
296 ste_info->data = ste_info->data_cont;
297 } else {
298 ste_info->data = data;
299 }
300
301 list_add_tail(&ste_info->send_list, send_list);
302}
303
304/* The function tries to consume one wc each time, unless the queue is full, in
305 * that case, which means that the hw is behind the sw in a full queue len
306 * the function will drain the cq till it empty.
307 */
308static int dr_handle_pending_wc(struct mlx5dr_domain *dmn,
309 struct mlx5dr_send_ring *send_ring)
310{
311 bool is_drain = false;
312 int ne;
313
314 if (send_ring->pending_wqe < send_ring->signal_th)
315 return 0;
316
317 /* Queue is full start drain it */
318 if (send_ring->pending_wqe >=
319 dmn->send_ring->signal_th * TH_NUMS_TO_DRAIN)
320 is_drain = true;
321
322 do {
323 ne = dr_poll_cq(send_ring->cq, 1);
324 if (ne < 0)
325 return ne;
326 else if (ne == 1)
327 send_ring->pending_wqe -= send_ring->signal_th;
328 } while (is_drain && send_ring->pending_wqe);
329
330 return 0;
331}
332
333static void dr_fill_data_segs(struct mlx5dr_send_ring *send_ring,
334 struct postsend_info *send_info)
335{
336 send_ring->pending_wqe++;
337
338 if (send_ring->pending_wqe % send_ring->signal_th == 0)
339 send_info->write.send_flags |= IB_SEND_SIGNALED;
340
341 send_ring->pending_wqe++;
342 send_info->read.length = send_info->write.length;
343 /* Read into the same write area */
344 send_info->read.addr = (uintptr_t)send_info->write.addr;
345 send_info->read.lkey = send_ring->mr->mkey.key;
346
347 if (send_ring->pending_wqe % send_ring->signal_th == 0)
348 send_info->read.send_flags = IB_SEND_SIGNALED;
349 else
350 send_info->read.send_flags = 0;
351}
352
353static int dr_postsend_icm_data(struct mlx5dr_domain *dmn,
354 struct postsend_info *send_info)
355{
356 struct mlx5dr_send_ring *send_ring = dmn->send_ring;
357 u32 buff_offset;
358 int ret;
359
360 ret = dr_handle_pending_wc(dmn, send_ring);
361 if (ret)
362 return ret;
363
364 if (send_info->write.length > dmn->info.max_inline_size) {
365 buff_offset = (send_ring->tx_head &
366 (dmn->send_ring->signal_th - 1)) *
367 send_ring->max_post_send_size;
368 /* Copy to ring mr */
369 memcpy(send_ring->buf + buff_offset,
370 (void *)(uintptr_t)send_info->write.addr,
371 send_info->write.length);
372 send_info->write.addr = (uintptr_t)send_ring->mr->dma_addr + buff_offset;
373 send_info->write.lkey = send_ring->mr->mkey.key;
374 }
375
376 send_ring->tx_head++;
377 dr_fill_data_segs(send_ring, send_info);
378 dr_post_send(send_ring->qp, send_info);
379
380 return 0;
381}
382
383static int dr_get_tbl_copy_details(struct mlx5dr_domain *dmn,
384 struct mlx5dr_ste_htbl *htbl,
385 u8 **data,
386 u32 *byte_size,
387 int *iterations,
388 int *num_stes)
389{
390 int alloc_size;
391
392 if (htbl->chunk->byte_size > dmn->send_ring->max_post_send_size) {
393 *iterations = htbl->chunk->byte_size /
394 dmn->send_ring->max_post_send_size;
395 *byte_size = dmn->send_ring->max_post_send_size;
396 alloc_size = *byte_size;
397 *num_stes = *byte_size / DR_STE_SIZE;
398 } else {
399 *iterations = 1;
400 *num_stes = htbl->chunk->num_of_entries;
401 alloc_size = *num_stes * DR_STE_SIZE;
402 }
403
404 *data = kzalloc(alloc_size, GFP_KERNEL);
405 if (!*data)
406 return -ENOMEM;
407
408 return 0;
409}
410
411/**
412 * mlx5dr_send_postsend_ste: write size bytes into offset from the hw cm.
413 *
414 * @dmn: Domain
415 * @ste: The ste struct that contains the data (at
416 * least part of it)
417 * @data: The real data to send size data
418 * @size: for writing.
419 * @offset: The offset from the icm mapped data to
420 * start write to this for write only part of the
421 * buffer.
422 *
423 * Return: 0 on success.
424 */
425int mlx5dr_send_postsend_ste(struct mlx5dr_domain *dmn, struct mlx5dr_ste *ste,
426 u8 *data, u16 size, u16 offset)
427{
428 struct postsend_info send_info = {};
429
430 send_info.write.addr = (uintptr_t)data;
431 send_info.write.length = size;
432 send_info.write.lkey = 0;
433 send_info.remote_addr = mlx5dr_ste_get_mr_addr(ste) + offset;
434 send_info.rkey = ste->htbl->chunk->rkey;
435
436 return dr_postsend_icm_data(dmn, &send_info);
437}
438
439int mlx5dr_send_postsend_htbl(struct mlx5dr_domain *dmn,
440 struct mlx5dr_ste_htbl *htbl,
441 u8 *formatted_ste, u8 *mask)
442{
443 u32 byte_size = htbl->chunk->byte_size;
444 int num_stes_per_iter;
445 int iterations;
446 u8 *data;
447 int ret;
448 int i;
449 int j;
450
451 ret = dr_get_tbl_copy_details(dmn, htbl, &data, &byte_size,
452 &iterations, &num_stes_per_iter);
453 if (ret)
454 return ret;
455
456 /* Send the data iteration times */
457 for (i = 0; i < iterations; i++) {
458 u32 ste_index = i * (byte_size / DR_STE_SIZE);
459 struct postsend_info send_info = {};
460
461 /* Copy all ste's on the data buffer
462 * need to add the bit_mask
463 */
464 for (j = 0; j < num_stes_per_iter; j++) {
465 u8 *hw_ste = htbl->ste_arr[ste_index + j].hw_ste;
466 u32 ste_off = j * DR_STE_SIZE;
467
468 if (mlx5dr_ste_is_not_valid_entry(hw_ste)) {
469 memcpy(data + ste_off,
470 formatted_ste, DR_STE_SIZE);
471 } else {
472 /* Copy data */
473 memcpy(data + ste_off,
474 htbl->ste_arr[ste_index + j].hw_ste,
475 DR_STE_SIZE_REDUCED);
476 /* Copy bit_mask */
477 memcpy(data + ste_off + DR_STE_SIZE_REDUCED,
478 mask, DR_STE_SIZE_MASK);
479 }
480 }
481
482 send_info.write.addr = (uintptr_t)data;
483 send_info.write.length = byte_size;
484 send_info.write.lkey = 0;
485 send_info.remote_addr =
486 mlx5dr_ste_get_mr_addr(htbl->ste_arr + ste_index);
487 send_info.rkey = htbl->chunk->rkey;
488
489 ret = dr_postsend_icm_data(dmn, &send_info);
490 if (ret)
491 goto out_free;
492 }
493
494out_free:
495 kfree(data);
496 return ret;
497}
498
499/* Initialize htble with default STEs */
500int mlx5dr_send_postsend_formatted_htbl(struct mlx5dr_domain *dmn,
501 struct mlx5dr_ste_htbl *htbl,
502 u8 *ste_init_data,
503 bool update_hw_ste)
504{
505 u32 byte_size = htbl->chunk->byte_size;
506 int iterations;
507 int num_stes;
508 u8 *data;
509 int ret;
510 int i;
511
512 ret = dr_get_tbl_copy_details(dmn, htbl, &data, &byte_size,
513 &iterations, &num_stes);
514 if (ret)
515 return ret;
516
517 for (i = 0; i < num_stes; i++) {
518 u8 *copy_dst;
519
520 /* Copy the same ste on the data buffer */
521 copy_dst = data + i * DR_STE_SIZE;
522 memcpy(copy_dst, ste_init_data, DR_STE_SIZE);
523
524 if (update_hw_ste) {
525 /* Copy the reduced ste to hash table ste_arr */
526 copy_dst = htbl->hw_ste_arr + i * DR_STE_SIZE_REDUCED;
527 memcpy(copy_dst, ste_init_data, DR_STE_SIZE_REDUCED);
528 }
529 }
530
531 /* Send the data iteration times */
532 for (i = 0; i < iterations; i++) {
533 u8 ste_index = i * (byte_size / DR_STE_SIZE);
534 struct postsend_info send_info = {};
535
536 send_info.write.addr = (uintptr_t)data;
537 send_info.write.length = byte_size;
538 send_info.write.lkey = 0;
539 send_info.remote_addr =
540 mlx5dr_ste_get_mr_addr(htbl->ste_arr + ste_index);
541 send_info.rkey = htbl->chunk->rkey;
542
543 ret = dr_postsend_icm_data(dmn, &send_info);
544 if (ret)
545 goto out_free;
546 }
547
548out_free:
549 kfree(data);
550 return ret;
551}
552
553int mlx5dr_send_postsend_action(struct mlx5dr_domain *dmn,
554 struct mlx5dr_action *action)
555{
556 struct postsend_info send_info = {};
557 int ret;
558
559 send_info.write.addr = (uintptr_t)action->rewrite.data;
560 send_info.write.length = action->rewrite.chunk->byte_size;
561 send_info.write.lkey = 0;
562 send_info.remote_addr = action->rewrite.chunk->mr_addr;
563 send_info.rkey = action->rewrite.chunk->rkey;
564
565 mutex_lock(&dmn->mutex);
566 ret = dr_postsend_icm_data(dmn, &send_info);
567 mutex_unlock(&dmn->mutex);
568
569 return ret;
570}
571
572static int dr_modify_qp_rst2init(struct mlx5_core_dev *mdev,
573 struct mlx5dr_qp *dr_qp,
574 int port)
575{
576 u32 in[MLX5_ST_SZ_DW(rst2init_qp_in)] = {};
577 void *qpc;
578
579 qpc = MLX5_ADDR_OF(rst2init_qp_in, in, qpc);
580
581 MLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, port);
582 MLX5_SET(qpc, qpc, pm_state, MLX5_QPC_PM_STATE_MIGRATED);
583 MLX5_SET(qpc, qpc, rre, 1);
584 MLX5_SET(qpc, qpc, rwe, 1);
585
586 return mlx5_core_qp_modify(mdev, MLX5_CMD_OP_RST2INIT_QP, 0, qpc,
587 &dr_qp->mqp);
588}
589
590static int dr_cmd_modify_qp_rtr2rts(struct mlx5_core_dev *mdev,
591 struct mlx5dr_qp *dr_qp,
592 struct dr_qp_rts_attr *attr)
593{
594 u32 in[MLX5_ST_SZ_DW(rtr2rts_qp_in)] = {};
595 void *qpc;
596
597 qpc = MLX5_ADDR_OF(rtr2rts_qp_in, in, qpc);
598
599 MLX5_SET(rtr2rts_qp_in, in, qpn, dr_qp->mqp.qpn);
600
601 MLX5_SET(qpc, qpc, log_ack_req_freq, 0);
602 MLX5_SET(qpc, qpc, retry_count, attr->retry_cnt);
603 MLX5_SET(qpc, qpc, rnr_retry, attr->rnr_retry);
604
605 return mlx5_core_qp_modify(mdev, MLX5_CMD_OP_RTR2RTS_QP, 0, qpc,
606 &dr_qp->mqp);
607}
608
609static int dr_cmd_modify_qp_init2rtr(struct mlx5_core_dev *mdev,
610 struct mlx5dr_qp *dr_qp,
611 struct dr_qp_rtr_attr *attr)
612{
613 u32 in[MLX5_ST_SZ_DW(init2rtr_qp_in)] = {};
614 void *qpc;
615
616 qpc = MLX5_ADDR_OF(init2rtr_qp_in, in, qpc);
617
618 MLX5_SET(init2rtr_qp_in, in, qpn, dr_qp->mqp.qpn);
619
620 MLX5_SET(qpc, qpc, mtu, attr->mtu);
621 MLX5_SET(qpc, qpc, log_msg_max, DR_CHUNK_SIZE_MAX - 1);
622 MLX5_SET(qpc, qpc, remote_qpn, attr->qp_num);
623 memcpy(MLX5_ADDR_OF(qpc, qpc, primary_address_path.rmac_47_32),
624 attr->dgid_attr.mac, sizeof(attr->dgid_attr.mac));
625 memcpy(MLX5_ADDR_OF(qpc, qpc, primary_address_path.rgid_rip),
626 attr->dgid_attr.gid, sizeof(attr->dgid_attr.gid));
627 MLX5_SET(qpc, qpc, primary_address_path.src_addr_index,
628 attr->sgid_index);
629
630 if (attr->dgid_attr.roce_ver == MLX5_ROCE_VERSION_2)
631 MLX5_SET(qpc, qpc, primary_address_path.udp_sport,
632 attr->udp_src_port);
633
634 MLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, attr->port_num);
635 MLX5_SET(qpc, qpc, min_rnr_nak, 1);
636
637 return mlx5_core_qp_modify(mdev, MLX5_CMD_OP_INIT2RTR_QP, 0, qpc,
638 &dr_qp->mqp);
639}
640
641static int dr_prepare_qp_to_rts(struct mlx5dr_domain *dmn)
642{
643 struct mlx5dr_qp *dr_qp = dmn->send_ring->qp;
644 struct dr_qp_rts_attr rts_attr = {};
645 struct dr_qp_rtr_attr rtr_attr = {};
646 enum ib_mtu mtu = IB_MTU_1024;
647 u16 gid_index = 0;
648 int port = 1;
649 int ret;
650
651 /* Init */
652 ret = dr_modify_qp_rst2init(dmn->mdev, dr_qp, port);
653 if (ret)
654 return ret;
655
656 /* RTR */
657 ret = mlx5dr_cmd_query_gid(dmn->mdev, port, gid_index, &rtr_attr.dgid_attr);
658 if (ret)
659 return ret;
660
661 rtr_attr.mtu = mtu;
662 rtr_attr.qp_num = dr_qp->mqp.qpn;
663 rtr_attr.min_rnr_timer = 12;
664 rtr_attr.port_num = port;
665 rtr_attr.sgid_index = gid_index;
666 rtr_attr.udp_src_port = dmn->info.caps.roce_min_src_udp;
667
668 ret = dr_cmd_modify_qp_init2rtr(dmn->mdev, dr_qp, &rtr_attr);
669 if (ret)
670 return ret;
671
672 /* RTS */
673 rts_attr.timeout = 14;
674 rts_attr.retry_cnt = 7;
675 rts_attr.rnr_retry = 7;
676
677 ret = dr_cmd_modify_qp_rtr2rts(dmn->mdev, dr_qp, &rts_attr);
678 if (ret)
679 return ret;
680
681 return 0;
682}
683
684static void dr_cq_event(struct mlx5_core_cq *mcq,
685 enum mlx5_event event)
686{
687 pr_info("CQ event %u on CQ #%u\n", event, mcq->cqn);
688}
689
690static struct mlx5dr_cq *dr_create_cq(struct mlx5_core_dev *mdev,
691 struct mlx5_uars_page *uar,
692 size_t ncqe)
693{
694 u32 temp_cqc[MLX5_ST_SZ_DW(cqc)] = {};
695 u32 out[MLX5_ST_SZ_DW(create_cq_out)];
696 struct mlx5_wq_param wqp;
697 struct mlx5_cqe64 *cqe;
698 struct mlx5dr_cq *cq;
699 int inlen, err, eqn;
700 unsigned int irqn;
701 void *cqc, *in;
702 __be64 *pas;
703 u32 i;
704
705 cq = kzalloc(sizeof(*cq), GFP_KERNEL);
706 if (!cq)
707 return NULL;
708
709 ncqe = roundup_pow_of_two(ncqe);
710 MLX5_SET(cqc, temp_cqc, log_cq_size, ilog2(ncqe));
711
712 wqp.buf_numa_node = mdev->priv.numa_node;
713 wqp.db_numa_node = mdev->priv.numa_node;
714
715 err = mlx5_cqwq_create(mdev, &wqp, temp_cqc, &cq->wq,
716 &cq->wq_ctrl);
717 if (err)
718 goto out;
719
720 for (i = 0; i < mlx5_cqwq_get_size(&cq->wq); i++) {
721 cqe = mlx5_cqwq_get_wqe(&cq->wq, i);
722 cqe->op_own = MLX5_CQE_INVALID << 4 | MLX5_CQE_OWNER_MASK;
723 }
724
725 inlen = MLX5_ST_SZ_BYTES(create_cq_in) +
726 sizeof(u64) * cq->wq_ctrl.buf.npages;
727 in = kvzalloc(inlen, GFP_KERNEL);
728 if (!in)
729 goto err_cqwq;
730
731 err = mlx5_vector2eqn(mdev, smp_processor_id(), &eqn, &irqn);
732 if (err) {
733 kvfree(in);
734 goto err_cqwq;
735 }
736
737 cqc = MLX5_ADDR_OF(create_cq_in, in, cq_context);
738 MLX5_SET(cqc, cqc, log_cq_size, ilog2(ncqe));
739 MLX5_SET(cqc, cqc, c_eqn, eqn);
740 MLX5_SET(cqc, cqc, uar_page, uar->index);
741 MLX5_SET(cqc, cqc, log_page_size, cq->wq_ctrl.buf.page_shift -
742 MLX5_ADAPTER_PAGE_SHIFT);
743 MLX5_SET64(cqc, cqc, dbr_addr, cq->wq_ctrl.db.dma);
744
745 pas = (__be64 *)MLX5_ADDR_OF(create_cq_in, in, pas);
746 mlx5_fill_page_frag_array(&cq->wq_ctrl.buf, pas);
747
748 cq->mcq.event = dr_cq_event;
749
750 err = mlx5_core_create_cq(mdev, &cq->mcq, in, inlen, out, sizeof(out));
751 kvfree(in);
752
753 if (err)
754 goto err_cqwq;
755
756 cq->mcq.cqe_sz = 64;
757 cq->mcq.set_ci_db = cq->wq_ctrl.db.db;
758 cq->mcq.arm_db = cq->wq_ctrl.db.db + 1;
759 *cq->mcq.set_ci_db = 0;
760 *cq->mcq.arm_db = 0;
761 cq->mcq.vector = 0;
762 cq->mcq.irqn = irqn;
763 cq->mcq.uar = uar;
764
765 return cq;
766
767err_cqwq:
768 mlx5_wq_destroy(&cq->wq_ctrl);
769out:
770 kfree(cq);
771 return NULL;
772}
773
774static void dr_destroy_cq(struct mlx5_core_dev *mdev, struct mlx5dr_cq *cq)
775{
776 mlx5_core_destroy_cq(mdev, &cq->mcq);
777 mlx5_wq_destroy(&cq->wq_ctrl);
778 kfree(cq);
779}
780
781static int
782dr_create_mkey(struct mlx5_core_dev *mdev, u32 pdn, struct mlx5_core_mkey *mkey)
783{
784 u32 in[MLX5_ST_SZ_DW(create_mkey_in)] = {};
785 void *mkc;
786
787 mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry);
788 MLX5_SET(mkc, mkc, access_mode_1_0, MLX5_MKC_ACCESS_MODE_PA);
789 MLX5_SET(mkc, mkc, a, 1);
790 MLX5_SET(mkc, mkc, rw, 1);
791 MLX5_SET(mkc, mkc, rr, 1);
792 MLX5_SET(mkc, mkc, lw, 1);
793 MLX5_SET(mkc, mkc, lr, 1);
794
795 MLX5_SET(mkc, mkc, pd, pdn);
796 MLX5_SET(mkc, mkc, length64, 1);
797 MLX5_SET(mkc, mkc, qpn, 0xffffff);
798
799 return mlx5_core_create_mkey(mdev, mkey, in, sizeof(in));
800}
801
802static struct mlx5dr_mr *dr_reg_mr(struct mlx5_core_dev *mdev,
803 u32 pdn, void *buf, size_t size)
804{
805 struct mlx5dr_mr *mr = kzalloc(sizeof(*mr), GFP_KERNEL);
806 struct device *dma_device;
807 dma_addr_t dma_addr;
808 int err;
809
810 if (!mr)
811 return NULL;
812
813 dma_device = &mdev->pdev->dev;
814 dma_addr = dma_map_single(dma_device, buf, size,
815 DMA_BIDIRECTIONAL);
816 err = dma_mapping_error(dma_device, dma_addr);
817 if (err) {
818 mlx5_core_warn(mdev, "Can't dma buf\n");
819 kfree(mr);
820 return NULL;
821 }
822
823 err = dr_create_mkey(mdev, pdn, &mr->mkey);
824 if (err) {
825 mlx5_core_warn(mdev, "Can't create mkey\n");
826 dma_unmap_single(dma_device, dma_addr, size,
827 DMA_BIDIRECTIONAL);
828 kfree(mr);
829 return NULL;
830 }
831
832 mr->dma_addr = dma_addr;
833 mr->size = size;
834 mr->addr = buf;
835
836 return mr;
837}
838
839static void dr_dereg_mr(struct mlx5_core_dev *mdev, struct mlx5dr_mr *mr)
840{
841 mlx5_core_destroy_mkey(mdev, &mr->mkey);
842 dma_unmap_single(&mdev->pdev->dev, mr->dma_addr, mr->size,
843 DMA_BIDIRECTIONAL);
844 kfree(mr);
845}
846
847int mlx5dr_send_ring_alloc(struct mlx5dr_domain *dmn)
848{
849 struct dr_qp_init_attr init_attr = {};
850 int cq_size;
851 int size;
852 int ret;
853
854 dmn->send_ring = kzalloc(sizeof(*dmn->send_ring), GFP_KERNEL);
855 if (!dmn->send_ring)
856 return -ENOMEM;
857
858 cq_size = QUEUE_SIZE + 1;
859 dmn->send_ring->cq = dr_create_cq(dmn->mdev, dmn->uar, cq_size);
860 if (!dmn->send_ring->cq) {
861 ret = -ENOMEM;
862 goto free_send_ring;
863 }
864
865 init_attr.cqn = dmn->send_ring->cq->mcq.cqn;
866 init_attr.pdn = dmn->pdn;
867 init_attr.uar = dmn->uar;
868 init_attr.max_send_wr = QUEUE_SIZE;
869
870 dmn->send_ring->qp = dr_create_rc_qp(dmn->mdev, &init_attr);
871 if (!dmn->send_ring->qp) {
872 ret = -ENOMEM;
873 goto clean_cq;
874 }
875
876 dmn->send_ring->cq->qp = dmn->send_ring->qp;
877
878 dmn->info.max_send_wr = QUEUE_SIZE;
879 dmn->info.max_inline_size = min(dmn->send_ring->qp->max_inline_data,
880 DR_STE_SIZE);
881
882 dmn->send_ring->signal_th = dmn->info.max_send_wr /
883 SIGNAL_PER_DIV_QUEUE;
884
885 /* Prepare qp to be used */
886 ret = dr_prepare_qp_to_rts(dmn);
887 if (ret)
888 goto clean_qp;
889
890 dmn->send_ring->max_post_send_size =
891 mlx5dr_icm_pool_chunk_size_to_byte(DR_CHUNK_SIZE_1K,
892 DR_ICM_TYPE_STE);
893
894 /* Allocating the max size as a buffer for writing */
895 size = dmn->send_ring->signal_th * dmn->send_ring->max_post_send_size;
896 dmn->send_ring->buf = kzalloc(size, GFP_KERNEL);
897 if (!dmn->send_ring->buf) {
898 ret = -ENOMEM;
899 goto clean_qp;
900 }
901
902 memset(dmn->send_ring->buf, 0, size);
903 dmn->send_ring->buf_size = size;
904
905 dmn->send_ring->mr = dr_reg_mr(dmn->mdev,
906 dmn->pdn, dmn->send_ring->buf, size);
907 if (!dmn->send_ring->mr) {
908 ret = -ENOMEM;
909 goto free_mem;
910 }
911
912 dmn->send_ring->sync_mr = dr_reg_mr(dmn->mdev,
913 dmn->pdn, dmn->send_ring->sync_buff,
914 MIN_READ_SYNC);
915 if (!dmn->send_ring->sync_mr) {
916 ret = -ENOMEM;
917 goto clean_mr;
918 }
919
920 return 0;
921
922clean_mr:
923 dr_dereg_mr(dmn->mdev, dmn->send_ring->mr);
924free_mem:
925 kfree(dmn->send_ring->buf);
926clean_qp:
927 dr_destroy_qp(dmn->mdev, dmn->send_ring->qp);
928clean_cq:
929 dr_destroy_cq(dmn->mdev, dmn->send_ring->cq);
930free_send_ring:
931 kfree(dmn->send_ring);
932
933 return ret;
934}
935
936void mlx5dr_send_ring_free(struct mlx5dr_domain *dmn,
937 struct mlx5dr_send_ring *send_ring)
938{
939 dr_destroy_qp(dmn->mdev, send_ring->qp);
940 dr_destroy_cq(dmn->mdev, send_ring->cq);
941 dr_dereg_mr(dmn->mdev, send_ring->sync_mr);
942 dr_dereg_mr(dmn->mdev, send_ring->mr);
943 kfree(send_ring->buf);
944 kfree(send_ring);
945}
946
947int mlx5dr_send_ring_force_drain(struct mlx5dr_domain *dmn)
948{
949 struct mlx5dr_send_ring *send_ring = dmn->send_ring;
950 struct postsend_info send_info = {};
951 u8 data[DR_STE_SIZE];
952 int num_of_sends_req;
953 int ret;
954 int i;
955
956 /* Sending this amount of requests makes sure we will get drain */
957 num_of_sends_req = send_ring->signal_th * TH_NUMS_TO_DRAIN / 2;
958
959 /* Send fake requests forcing the last to be signaled */
960 send_info.write.addr = (uintptr_t)data;
961 send_info.write.length = DR_STE_SIZE;
962 send_info.write.lkey = 0;
963 /* Using the sync_mr in order to write/read */
964 send_info.remote_addr = (uintptr_t)send_ring->sync_mr->addr;
965 send_info.rkey = send_ring->sync_mr->mkey.key;
966
967 for (i = 0; i < num_of_sends_req; i++) {
968 ret = dr_postsend_icm_data(dmn, &send_info);
969 if (ret)
970 return ret;
971 }
972
973 ret = dr_handle_pending_wc(dmn, send_ring);
974
975 return ret;
976}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste.c+2308−0
@@ -0,0 +1,2308 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2/* Copyright (c) 2019 Mellanox Technologies. */
3
4#include <linux/types.h>
5#include "dr_types.h"
6
7#define DR_STE_CRC_POLY 0xEDB88320L
8#define STE_IPV4 0x1
9#define STE_IPV6 0x2
10#define STE_TCP 0x1
11#define STE_UDP 0x2
12#define STE_SPI 0x3
13#define IP_VERSION_IPV4 0x4
14#define IP_VERSION_IPV6 0x6
15#define STE_SVLAN 0x1
16#define STE_CVLAN 0x2
17
18#define DR_STE_ENABLE_FLOW_TAG BIT(31)
19
20/* Set to STE a specific value using DR_STE_SET */
21#define DR_STE_SET_VAL(lookup_type, tag, t_fname, spec, s_fname, value) do { \
22 if ((spec)->s_fname) { \
23 MLX5_SET(ste_##lookup_type, tag, t_fname, value); \
24 (spec)->s_fname = 0; \
25 } \
26} while (0)
27
28/* Set to STE spec->s_fname to tag->t_fname */
29#define DR_STE_SET_TAG(lookup_type, tag, t_fname, spec, s_fname) \
30 DR_STE_SET_VAL(lookup_type, tag, t_fname, spec, s_fname, spec->s_fname)
31
32/* Set to STE -1 to bit_mask->bm_fname and set spec->s_fname as used */
33#define DR_STE_SET_MASK(lookup_type, bit_mask, bm_fname, spec, s_fname) \
34 DR_STE_SET_VAL(lookup_type, bit_mask, bm_fname, spec, s_fname, -1)
35
36/* Set to STE spec->s_fname to bit_mask->bm_fname and set spec->s_fname as used */
37#define DR_STE_SET_MASK_V(lookup_type, bit_mask, bm_fname, spec, s_fname) \
38 DR_STE_SET_VAL(lookup_type, bit_mask, bm_fname, spec, s_fname, (spec)->s_fname)
39
40#define DR_STE_SET_TCP_FLAGS(lookup_type, tag, spec) do { \
41 MLX5_SET(ste_##lookup_type, tag, tcp_ns, !!((spec)->tcp_flags & (1 << 8))); \
42 MLX5_SET(ste_##lookup_type, tag, tcp_cwr, !!((spec)->tcp_flags & (1 << 7))); \
43 MLX5_SET(ste_##lookup_type, tag, tcp_ece, !!((spec)->tcp_flags & (1 << 6))); \
44 MLX5_SET(ste_##lookup_type, tag, tcp_urg, !!((spec)->tcp_flags & (1 << 5))); \
45 MLX5_SET(ste_##lookup_type, tag, tcp_ack, !!((spec)->tcp_flags & (1 << 4))); \
46 MLX5_SET(ste_##lookup_type, tag, tcp_psh, !!((spec)->tcp_flags & (1 << 3))); \
47 MLX5_SET(ste_##lookup_type, tag, tcp_rst, !!((spec)->tcp_flags & (1 << 2))); \
48 MLX5_SET(ste_##lookup_type, tag, tcp_syn, !!((spec)->tcp_flags & (1 << 1))); \
49 MLX5_SET(ste_##lookup_type, tag, tcp_fin, !!((spec)->tcp_flags & (1 << 0))); \
50} while (0)
51
52#define DR_STE_SET_MPLS_MASK(lookup_type, mask, in_out, bit_mask) do { \
53 DR_STE_SET_MASK_V(lookup_type, mask, mpls0_label, mask, \
54 in_out##_first_mpls_label);\
55 DR_STE_SET_MASK_V(lookup_type, mask, mpls0_s_bos, mask, \
56 in_out##_first_mpls_s_bos); \
57 DR_STE_SET_MASK_V(lookup_type, mask, mpls0_exp, mask, \
58 in_out##_first_mpls_exp); \
59 DR_STE_SET_MASK_V(lookup_type, mask, mpls0_ttl, mask, \
60 in_out##_first_mpls_ttl); \
61} while (0)
62
63#define DR_STE_SET_MPLS_TAG(lookup_type, mask, in_out, tag) do { \
64 DR_STE_SET_TAG(lookup_type, tag, mpls0_label, mask, \
65 in_out##_first_mpls_label);\
66 DR_STE_SET_TAG(lookup_type, tag, mpls0_s_bos, mask, \
67 in_out##_first_mpls_s_bos); \
68 DR_STE_SET_TAG(lookup_type, tag, mpls0_exp, mask, \
69 in_out##_first_mpls_exp); \
70 DR_STE_SET_TAG(lookup_type, tag, mpls0_ttl, mask, \
71 in_out##_first_mpls_ttl); \
72} while (0)
73
74#define DR_STE_IS_OUTER_MPLS_OVER_GRE_SET(_misc) (\
75 (_misc)->outer_first_mpls_over_gre_label || \
76 (_misc)->outer_first_mpls_over_gre_exp || \
77 (_misc)->outer_first_mpls_over_gre_s_bos || \
78 (_misc)->outer_first_mpls_over_gre_ttl)
79#define DR_STE_IS_OUTER_MPLS_OVER_UDP_SET(_misc) (\
80 (_misc)->outer_first_mpls_over_udp_label || \
81 (_misc)->outer_first_mpls_over_udp_exp || \
82 (_misc)->outer_first_mpls_over_udp_s_bos || \
83 (_misc)->outer_first_mpls_over_udp_ttl)
84
85#define DR_STE_CALC_LU_TYPE(lookup_type, rx, inner) \
86 ((inner) ? MLX5DR_STE_LU_TYPE_##lookup_type##_I : \
87 (rx) ? MLX5DR_STE_LU_TYPE_##lookup_type##_D : \
88 MLX5DR_STE_LU_TYPE_##lookup_type##_O)
89
90enum dr_ste_tunl_action {
91 DR_STE_TUNL_ACTION_NONE = 0,
92 DR_STE_TUNL_ACTION_ENABLE = 1,
93 DR_STE_TUNL_ACTION_DECAP = 2,
94 DR_STE_TUNL_ACTION_L3_DECAP = 3,
95 DR_STE_TUNL_ACTION_POP_VLAN = 4,
96};
97
98enum dr_ste_action_type {
99 DR_STE_ACTION_TYPE_PUSH_VLAN = 1,
100 DR_STE_ACTION_TYPE_ENCAP_L3 = 3,
101 DR_STE_ACTION_TYPE_ENCAP = 4,
102};
103
104struct dr_hw_ste_format {
105 u8 ctrl[DR_STE_SIZE_CTRL];
106 u8 tag[DR_STE_SIZE_TAG];
107 u8 mask[DR_STE_SIZE_MASK];
108};
109
110u32 mlx5dr_ste_calc_hash_index(u8 *hw_ste_p, struct mlx5dr_ste_htbl *htbl)
111{
112 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
113 u8 masked[DR_STE_SIZE_TAG] = {};
114 u32 crc32, index;
115 u16 bit;
116 int i;
117
118 /* Don't calculate CRC if the result is predicted */
119 if (htbl->chunk->num_of_entries == 1 || htbl->byte_mask == 0)
120 return 0;
121
122 /* Mask tag using byte mask, bit per byte */
123 bit = 1 << (DR_STE_SIZE_TAG - 1);
124 for (i = 0; i < DR_STE_SIZE_TAG; i++) {
125 if (htbl->byte_mask & bit)
126 masked[i] = hw_ste->tag[i];
127
128 bit = bit >> 1;
129 }
130
131 crc32 = mlx5dr_crc32_slice8_calc(masked, DR_STE_SIZE_TAG);
132 index = crc32 & (htbl->chunk->num_of_entries - 1);
133
134 return index;
135}
136
137static u16 dr_ste_conv_bit_to_byte_mask(u8 *bit_mask)
138{
139 u16 byte_mask = 0;
140 int i;
141
142 for (i = 0; i < DR_STE_SIZE_MASK; i++) {
143 byte_mask = byte_mask << 1;
144 if (bit_mask[i] == 0xff)
145 byte_mask |= 1;
146 }
147 return byte_mask;
148}
149
150void mlx5dr_ste_set_bit_mask(u8 *hw_ste_p, u8 *bit_mask)
151{
152 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
153
154 memcpy(hw_ste->mask, bit_mask, DR_STE_SIZE_MASK);
155}
156
157void mlx5dr_ste_rx_set_flow_tag(u8 *hw_ste_p, u32 flow_tag)
158{
159 MLX5_SET(ste_rx_steering_mult, hw_ste_p, qp_list_pointer,
160 DR_STE_ENABLE_FLOW_TAG | flow_tag);
161}
162
163void mlx5dr_ste_set_counter_id(u8 *hw_ste_p, u32 ctr_id)
164{
165 /* This can be used for both rx_steering_mult and for sx_transmit */
166 MLX5_SET(ste_rx_steering_mult, hw_ste_p, counter_trigger_15_0, ctr_id);
167 MLX5_SET(ste_rx_steering_mult, hw_ste_p, counter_trigger_23_16, ctr_id >> 16);
168}
169
170void mlx5dr_ste_set_go_back_bit(u8 *hw_ste_p)
171{
172 MLX5_SET(ste_sx_transmit, hw_ste_p, go_back, 1);
173}
174
175void mlx5dr_ste_set_tx_push_vlan(u8 *hw_ste_p, u32 vlan_hdr,
176 bool go_back)
177{
178 MLX5_SET(ste_sx_transmit, hw_ste_p, action_type,
179 DR_STE_ACTION_TYPE_PUSH_VLAN);
180 MLX5_SET(ste_sx_transmit, hw_ste_p, encap_pointer_vlan_data, vlan_hdr);
181 /* Due to HW limitation we need to set this bit, otherwise reforamt +
182 * push vlan will not work.
183 */
184 if (go_back)
185 mlx5dr_ste_set_go_back_bit(hw_ste_p);
186}
187
188void mlx5dr_ste_set_tx_encap(void *hw_ste_p, u32 reformat_id, int size, bool encap_l3)
189{
190 MLX5_SET(ste_sx_transmit, hw_ste_p, action_type,
191 encap_l3 ? DR_STE_ACTION_TYPE_ENCAP_L3 : DR_STE_ACTION_TYPE_ENCAP);
192 /* The hardware expects here size in words (2 byte) */
193 MLX5_SET(ste_sx_transmit, hw_ste_p, action_description, size / 2);
194 MLX5_SET(ste_sx_transmit, hw_ste_p, encap_pointer_vlan_data, reformat_id);
195}
196
197void mlx5dr_ste_set_rx_decap(u8 *hw_ste_p)
198{
199 MLX5_SET(ste_rx_steering_mult, hw_ste_p, tunneling_action,
200 DR_STE_TUNL_ACTION_DECAP);
201}
202
203void mlx5dr_ste_set_rx_pop_vlan(u8 *hw_ste_p)
204{
205 MLX5_SET(ste_rx_steering_mult, hw_ste_p, tunneling_action,
206 DR_STE_TUNL_ACTION_POP_VLAN);
207}
208
209void mlx5dr_ste_set_rx_decap_l3(u8 *hw_ste_p, bool vlan)
210{
211 MLX5_SET(ste_rx_steering_mult, hw_ste_p, tunneling_action,
212 DR_STE_TUNL_ACTION_L3_DECAP);
213 MLX5_SET(ste_modify_packet, hw_ste_p, action_description, vlan ? 1 : 0);
214}
215
216void mlx5dr_ste_set_entry_type(u8 *hw_ste_p, u8 entry_type)
217{
218 MLX5_SET(ste_general, hw_ste_p, entry_type, entry_type);
219}
220
221u8 mlx5dr_ste_get_entry_type(u8 *hw_ste_p)
222{
223 return MLX5_GET(ste_general, hw_ste_p, entry_type);
224}
225
226void mlx5dr_ste_set_rewrite_actions(u8 *hw_ste_p, u16 num_of_actions,
227 u32 re_write_index)
228{
229 MLX5_SET(ste_modify_packet, hw_ste_p, number_of_re_write_actions,
230 num_of_actions);
231 MLX5_SET(ste_modify_packet, hw_ste_p, header_re_write_actions_pointer,
232 re_write_index);
233}
234
235void mlx5dr_ste_set_hit_gvmi(u8 *hw_ste_p, u16 gvmi)
236{
237 MLX5_SET(ste_general, hw_ste_p, next_table_base_63_48, gvmi);
238}
239
240void mlx5dr_ste_init(u8 *hw_ste_p, u8 lu_type, u8 entry_type,
241 u16 gvmi)
242{
243 MLX5_SET(ste_general, hw_ste_p, entry_type, entry_type);
244 MLX5_SET(ste_general, hw_ste_p, entry_sub_type, lu_type);
245 MLX5_SET(ste_general, hw_ste_p, next_lu_type, MLX5DR_STE_LU_TYPE_DONT_CARE);
246
247 /* Set GVMI once, this is the same for RX/TX
248 * bits 63_48 of next table base / miss address encode the next GVMI
249 */
250 MLX5_SET(ste_rx_steering_mult, hw_ste_p, gvmi, gvmi);
251 MLX5_SET(ste_rx_steering_mult, hw_ste_p, next_table_base_63_48, gvmi);
252 MLX5_SET(ste_rx_steering_mult, hw_ste_p, miss_address_63_48, gvmi);
253}
254
255static void dr_ste_set_always_hit(struct dr_hw_ste_format *hw_ste)
256{
257 memset(&hw_ste->tag, 0, sizeof(hw_ste->tag));
258 memset(&hw_ste->mask, 0, sizeof(hw_ste->mask));
259}
260
261static void dr_ste_set_always_miss(struct dr_hw_ste_format *hw_ste)
262{
263 hw_ste->tag[0] = 0xdc;
264 hw_ste->mask[0] = 0;
265}
266
267u64 mlx5dr_ste_get_miss_addr(u8 *hw_ste)
268{
269 u64 index =
270 (MLX5_GET(ste_rx_steering_mult, hw_ste, miss_address_31_6) |
271 MLX5_GET(ste_rx_steering_mult, hw_ste, miss_address_39_32) << 26);
272
273 return index << 6;
274}
275
276void mlx5dr_ste_set_hit_addr(u8 *hw_ste, u64 icm_addr, u32 ht_size)
277{
278 u64 index = (icm_addr >> 5) | ht_size;
279
280 MLX5_SET(ste_general, hw_ste, next_table_base_39_32_size, index >> 27);
281 MLX5_SET(ste_general, hw_ste, next_table_base_31_5_size, index);
282}
283
284u64 mlx5dr_ste_get_icm_addr(struct mlx5dr_ste *ste)
285{
286 u32 index = ste - ste->htbl->ste_arr;
287
288 return ste->htbl->chunk->icm_addr + DR_STE_SIZE * index;
289}
290
291u64 mlx5dr_ste_get_mr_addr(struct mlx5dr_ste *ste)
292{
293 u32 index = ste - ste->htbl->ste_arr;
294
295 return ste->htbl->chunk->mr_addr + DR_STE_SIZE * index;
296}
297
298struct list_head *mlx5dr_ste_get_miss_list(struct mlx5dr_ste *ste)
299{
300 u32 index = ste - ste->htbl->ste_arr;
301
302 return &ste->htbl->miss_list[index];
303}
304
305static void dr_ste_always_hit_htbl(struct mlx5dr_ste *ste,
306 struct mlx5dr_ste_htbl *next_htbl)
307{
308 struct mlx5dr_icm_chunk *chunk = next_htbl->chunk;
309 u8 *hw_ste = ste->hw_ste;
310
311 MLX5_SET(ste_general, hw_ste, byte_mask, next_htbl->byte_mask);
312 MLX5_SET(ste_general, hw_ste, next_lu_type, next_htbl->lu_type);
313 mlx5dr_ste_set_hit_addr(hw_ste, chunk->icm_addr, chunk->num_of_entries);
314
315 dr_ste_set_always_hit((struct dr_hw_ste_format *)ste->hw_ste);
316}
317
318bool mlx5dr_ste_is_last_in_rule(struct mlx5dr_matcher_rx_tx *nic_matcher,
319 u8 ste_location)
320{
321 return ste_location == nic_matcher->num_of_builders;
322}
323
324/* Replace relevant fields, except of:
325 * htbl - keep the origin htbl
326 * miss_list + list - already took the src from the list.
327 * icm_addr/mr_addr - depends on the hosting table.
328 *
329 * Before:
330 * | a | -> | b | -> | c | ->
331 *
332 * After:
333 * | a | -> | c | ->
334 * While the data that was in b copied to a.
335 */
336static void dr_ste_replace(struct mlx5dr_ste *dst, struct mlx5dr_ste *src)
337{
338 memcpy(dst->hw_ste, src->hw_ste, DR_STE_SIZE_REDUCED);
339 dst->next_htbl = src->next_htbl;
340 if (dst->next_htbl)
341 dst->next_htbl->pointing_ste = dst;
342
343 refcount_set(&dst->refcount, refcount_read(&src->refcount));
344
345 INIT_LIST_HEAD(&dst->rule_list);
346 list_splice_tail_init(&src->rule_list, &dst->rule_list);
347}
348
349/* Free ste which is the head and the only one in miss_list */
350static void
351dr_ste_remove_head_ste(struct mlx5dr_ste *ste,
352 struct mlx5dr_matcher_rx_tx *nic_matcher,
353 struct mlx5dr_ste_send_info *ste_info_head,
354 struct list_head *send_ste_list,
355 struct mlx5dr_ste_htbl *stats_tbl)
356{
357 u8 tmp_data_ste[DR_STE_SIZE] = {};
358 struct mlx5dr_ste tmp_ste = {};
359 u64 miss_addr;
360
361 tmp_ste.hw_ste = tmp_data_ste;
362
363 /* Use temp ste because dr_ste_always_miss_addr
364 * touches bit_mask area which doesn't exist at ste->hw_ste.
365 */
366 memcpy(tmp_ste.hw_ste, ste->hw_ste, DR_STE_SIZE_REDUCED);
367 miss_addr = nic_matcher->e_anchor->chunk->icm_addr;
368 mlx5dr_ste_always_miss_addr(&tmp_ste, miss_addr);
369 memcpy(ste->hw_ste, tmp_ste.hw_ste, DR_STE_SIZE_REDUCED);
370
371 list_del_init(&ste->miss_list_node);
372
373 /* Write full STE size in order to have "always_miss" */
374 mlx5dr_send_fill_and_append_ste_send_info(ste, DR_STE_SIZE,
375 0, tmp_data_ste,
376 ste_info_head,
377 send_ste_list,
378 true /* Copy data */);
379
380 stats_tbl->ctrl.num_of_valid_entries--;
381}
382
383/* Free ste which is the head but NOT the only one in miss_list:
384 * |_ste_| --> |_next_ste_| -->|__| -->|__| -->/0
385 */
386static void
387dr_ste_replace_head_ste(struct mlx5dr_ste *ste, struct mlx5dr_ste *next_ste,
388 struct mlx5dr_ste_send_info *ste_info_head,
389 struct list_head *send_ste_list,
390 struct mlx5dr_ste_htbl *stats_tbl)
391
392{
393 struct mlx5dr_ste_htbl *next_miss_htbl;
394
395 next_miss_htbl = next_ste->htbl;
396
397 /* Remove from the miss_list the next_ste before copy */
398 list_del_init(&next_ste->miss_list_node);
399
400 /* All rule-members that use next_ste should know about that */
401 mlx5dr_rule_update_rule_member(next_ste, ste);
402
403 /* Move data from next into ste */
404 dr_ste_replace(ste, next_ste);
405
406 /* Del the htbl that contains the next_ste.
407 * The origin htbl stay with the same number of entries.
408 */
409 mlx5dr_htbl_put(next_miss_htbl);
410
411 mlx5dr_send_fill_and_append_ste_send_info(ste, DR_STE_SIZE_REDUCED,
412 0, ste->hw_ste,
413 ste_info_head,
414 send_ste_list,
415 true /* Copy data */);
416
417 stats_tbl->ctrl.num_of_collisions--;
418 stats_tbl->ctrl.num_of_valid_entries--;
419}
420
421/* Free ste that is located in the middle of the miss list:
422 * |__| -->|_prev_ste_|->|_ste_|-->|_next_ste_|
423 */
424static void dr_ste_remove_middle_ste(struct mlx5dr_ste *ste,
425 struct mlx5dr_ste_send_info *ste_info,
426 struct list_head *send_ste_list,
427 struct mlx5dr_ste_htbl *stats_tbl)
428{
429 struct mlx5dr_ste *prev_ste;
430 u64 miss_addr;
431
432 prev_ste = list_entry(mlx5dr_ste_get_miss_list(ste)->prev, struct mlx5dr_ste,
433 miss_list_node);
434 if (!prev_ste) {
435 WARN_ON(true);
436 return;
437 }
438
439 miss_addr = mlx5dr_ste_get_miss_addr(ste->hw_ste);
440 mlx5dr_ste_set_miss_addr(prev_ste->hw_ste, miss_addr);
441
442 mlx5dr_send_fill_and_append_ste_send_info(prev_ste, DR_STE_SIZE_REDUCED, 0,
443 prev_ste->hw_ste, ste_info,
444 send_ste_list, true /* Copy data*/);
445
446 list_del_init(&ste->miss_list_node);
447
448 stats_tbl->ctrl.num_of_valid_entries--;
449 stats_tbl->ctrl.num_of_collisions--;
450}
451
452void mlx5dr_ste_free(struct mlx5dr_ste *ste,
453 struct mlx5dr_matcher *matcher,
454 struct mlx5dr_matcher_rx_tx *nic_matcher)
455{
456 struct mlx5dr_ste_send_info *cur_ste_info, *tmp_ste_info;
457 struct mlx5dr_domain *dmn = matcher->tbl->dmn;
458 struct mlx5dr_ste_send_info ste_info_head;
459 struct mlx5dr_ste *next_ste, *first_ste;
460 bool put_on_origin_table = true;
461 struct mlx5dr_ste_htbl *stats_tbl;
462 LIST_HEAD(send_ste_list);
463
464 first_ste = list_entry(mlx5dr_ste_get_miss_list(ste)->next,
465 struct mlx5dr_ste, miss_list_node);
466 stats_tbl = first_ste->htbl;
467
468 /* Two options:
469 * 1. ste is head:
470 * a. head ste is the only ste in the miss list
471 * b. head ste is not the only ste in the miss-list
472 * 2. ste is not head
473 */
474 if (first_ste == ste) { /* Ste is the head */
475 struct mlx5dr_ste *last_ste;
476
477 last_ste = list_last_entry(mlx5dr_ste_get_miss_list(ste),
478 struct mlx5dr_ste, miss_list_node);
479 if (last_ste == first_ste)
480 next_ste = NULL;
481 else
482 next_ste = list_entry(ste->miss_list_node.next,
483 struct mlx5dr_ste, miss_list_node);
484
485 if (!next_ste) {
486 /* One and only entry in the list */
487 dr_ste_remove_head_ste(ste, nic_matcher,
488 &ste_info_head,
489 &send_ste_list,
490 stats_tbl);
491 } else {
492 /* First but not only entry in the list */
493 dr_ste_replace_head_ste(ste, next_ste, &ste_info_head,
494 &send_ste_list, stats_tbl);
495 put_on_origin_table = false;
496 }
497 } else { /* Ste in the middle of the list */
498 dr_ste_remove_middle_ste(ste, &ste_info_head, &send_ste_list, stats_tbl);
499 }
500
501 /* Update HW */
502 list_for_each_entry_safe(cur_ste_info, tmp_ste_info,
503 &send_ste_list, send_list) {
504 list_del(&cur_ste_info->send_list);
505 mlx5dr_send_postsend_ste(dmn, cur_ste_info->ste,
506 cur_ste_info->data, cur_ste_info->size,
507 cur_ste_info->offset);
508 }
509
510 if (put_on_origin_table)
511 mlx5dr_htbl_put(ste->htbl);
512}
513
514bool mlx5dr_ste_equal_tag(void *src, void *dst)
515{
516 struct dr_hw_ste_format *s_hw_ste = (struct dr_hw_ste_format *)src;
517 struct dr_hw_ste_format *d_hw_ste = (struct dr_hw_ste_format *)dst;
518
519 return !memcmp(s_hw_ste->tag, d_hw_ste->tag, DR_STE_SIZE_TAG);
520}
521
522void mlx5dr_ste_set_hit_addr_by_next_htbl(u8 *hw_ste,
523 struct mlx5dr_ste_htbl *next_htbl)
524{
525 struct mlx5dr_icm_chunk *chunk = next_htbl->chunk;
526
527 mlx5dr_ste_set_hit_addr(hw_ste, chunk->icm_addr, chunk->num_of_entries);
528}
529
530void mlx5dr_ste_set_miss_addr(u8 *hw_ste_p, u64 miss_addr)
531{
532 u64 index = miss_addr >> 6;
533
534 /* Miss address for TX and RX STEs located in the same offsets */
535 MLX5_SET(ste_rx_steering_mult, hw_ste_p, miss_address_39_32, index >> 26);
536 MLX5_SET(ste_rx_steering_mult, hw_ste_p, miss_address_31_6, index);
537}
538
539void mlx5dr_ste_always_miss_addr(struct mlx5dr_ste *ste, u64 miss_addr)
540{
541 u8 *hw_ste = ste->hw_ste;
542
543 MLX5_SET(ste_rx_steering_mult, hw_ste, next_lu_type, MLX5DR_STE_LU_TYPE_DONT_CARE);
544 mlx5dr_ste_set_miss_addr(hw_ste, miss_addr);
545 dr_ste_set_always_miss((struct dr_hw_ste_format *)ste->hw_ste);
546}
547
548/* The assumption here is that we don't update the ste->hw_ste if it is not
549 * used ste, so it will be all zero, checking the next_lu_type.
550 */
551bool mlx5dr_ste_is_not_valid_entry(u8 *p_hw_ste)
552{
553 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)p_hw_ste;
554
555 if (MLX5_GET(ste_general, hw_ste, next_lu_type) ==
556 MLX5DR_STE_LU_TYPE_NOP)
557 return true;
558
559 return false;
560}
561
562bool mlx5dr_ste_not_used_ste(struct mlx5dr_ste *ste)
563{
564 return !refcount_read(&ste->refcount);
565}
566
567static u16 get_bits_per_mask(u16 byte_mask)
568{
569 u16 bits = 0;
570
571 while (byte_mask) {
572 byte_mask = byte_mask & (byte_mask - 1);
573 bits++;
574 }
575
576 return bits;
577}
578
579/* Init one ste as a pattern for ste data array */
580void mlx5dr_ste_set_formatted_ste(u16 gvmi,
581 struct mlx5dr_domain_rx_tx *nic_dmn,
582 struct mlx5dr_ste_htbl *htbl,
583 u8 *formatted_ste,
584 struct mlx5dr_htbl_connect_info *connect_info)
585{
586 struct mlx5dr_ste ste = {};
587
588 mlx5dr_ste_init(formatted_ste, htbl->lu_type, nic_dmn->ste_type, gvmi);
589 ste.hw_ste = formatted_ste;
590
591 if (connect_info->type == CONNECT_HIT)
592 dr_ste_always_hit_htbl(&ste, connect_info->hit_next_htbl);
593 else
594 mlx5dr_ste_always_miss_addr(&ste, connect_info->miss_icm_addr);
595}
596
597int mlx5dr_ste_htbl_init_and_postsend(struct mlx5dr_domain *dmn,
598 struct mlx5dr_domain_rx_tx *nic_dmn,
599 struct mlx5dr_ste_htbl *htbl,
600 struct mlx5dr_htbl_connect_info *connect_info,
601 bool update_hw_ste)
602{
603 u8 formatted_ste[DR_STE_SIZE] = {};
604
605 mlx5dr_ste_set_formatted_ste(dmn->info.caps.gvmi,
606 nic_dmn,
607 htbl,
608 formatted_ste,
609 connect_info);
610
611 return mlx5dr_send_postsend_formatted_htbl(dmn, htbl, formatted_ste, update_hw_ste);
612}
613
614int mlx5dr_ste_create_next_htbl(struct mlx5dr_matcher *matcher,
615 struct mlx5dr_matcher_rx_tx *nic_matcher,
616 struct mlx5dr_ste *ste,
617 u8 *cur_hw_ste,
618 enum mlx5dr_icm_chunk_size log_table_size)
619{
620 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)cur_hw_ste;
621 struct mlx5dr_domain_rx_tx *nic_dmn = nic_matcher->nic_tbl->nic_dmn;
622 struct mlx5dr_domain *dmn = matcher->tbl->dmn;
623 struct mlx5dr_htbl_connect_info info;
624 struct mlx5dr_ste_htbl *next_htbl;
625
626 if (!mlx5dr_ste_is_last_in_rule(nic_matcher, ste->ste_chain_location)) {
627 u32 bits_in_mask;
628 u8 next_lu_type;
629 u16 byte_mask;
630
631 next_lu_type = MLX5_GET(ste_general, hw_ste, next_lu_type);
632 byte_mask = MLX5_GET(ste_general, hw_ste, byte_mask);
633
634 /* Don't allocate table more than required,
635 * the size of the table defined via the byte_mask, so no need
636 * to allocate more than that.
637 */
638 bits_in_mask = get_bits_per_mask(byte_mask) * BITS_PER_BYTE;
639 log_table_size = min(log_table_size, bits_in_mask);
640
641 next_htbl = mlx5dr_ste_htbl_alloc(dmn->ste_icm_pool,
642 log_table_size,
643 next_lu_type,
644 byte_mask);
645 if (!next_htbl) {
646 mlx5dr_dbg(dmn, "Failed allocating table\n");
647 return -ENOMEM;
648 }
649
650 /* Write new table to HW */
651 info.type = CONNECT_MISS;
652 info.miss_icm_addr = nic_matcher->e_anchor->chunk->icm_addr;
653 if (mlx5dr_ste_htbl_init_and_postsend(dmn, nic_dmn, next_htbl,
654 &info, false)) {
655 mlx5dr_info(dmn, "Failed writing table to HW\n");
656 goto free_table;
657 }
658
659 mlx5dr_ste_set_hit_addr_by_next_htbl(cur_hw_ste, next_htbl);
660 ste->next_htbl = next_htbl;
661 next_htbl->pointing_ste = ste;
662 }
663
664 return 0;
665
666free_table:
667 mlx5dr_ste_htbl_free(next_htbl);
668 return -ENOENT;
669}
670
671static void dr_ste_set_ctrl(struct mlx5dr_ste_htbl *htbl)
672{
673 struct mlx5dr_ste_htbl_ctrl *ctrl = &htbl->ctrl;
674 int num_of_entries;
675
676 htbl->ctrl.may_grow = true;
677
678 if (htbl->chunk_size == DR_CHUNK_SIZE_MAX - 1)
679 htbl->ctrl.may_grow = false;
680
681 /* Threshold is 50%, one is added to table of size 1 */
682 num_of_entries = mlx5dr_icm_pool_chunk_size_to_entries(htbl->chunk_size);
683 ctrl->increase_threshold = (num_of_entries + 1) / 2;
684}
685
686struct mlx5dr_ste_htbl *mlx5dr_ste_htbl_alloc(struct mlx5dr_icm_pool *pool,
687 enum mlx5dr_icm_chunk_size chunk_size,
688 u8 lu_type, u16 byte_mask)
689{
690 struct mlx5dr_icm_chunk *chunk;
691 struct mlx5dr_ste_htbl *htbl;
692 int i;
693
694 htbl = kzalloc(sizeof(*htbl), GFP_KERNEL);
695 if (!htbl)
696 return NULL;
697
698 chunk = mlx5dr_icm_alloc_chunk(pool, chunk_size);
699 if (!chunk)
700 goto out_free_htbl;
701
702 htbl->chunk = chunk;
703 htbl->lu_type = lu_type;
704 htbl->byte_mask = byte_mask;
705 htbl->ste_arr = chunk->ste_arr;
706 htbl->hw_ste_arr = chunk->hw_ste_arr;
707 htbl->miss_list = chunk->miss_list;
708 refcount_set(&htbl->refcount, 0);
709
710 for (i = 0; i < chunk->num_of_entries; i++) {
711 struct mlx5dr_ste *ste = &htbl->ste_arr[i];
712
713 ste->hw_ste = htbl->hw_ste_arr + i * DR_STE_SIZE_REDUCED;
714 ste->htbl = htbl;
715 refcount_set(&ste->refcount, 0);
716 INIT_LIST_HEAD(&ste->miss_list_node);
717 INIT_LIST_HEAD(&htbl->miss_list[i]);
718 INIT_LIST_HEAD(&ste->rule_list);
719 }
720
721 htbl->chunk_size = chunk_size;
722 dr_ste_set_ctrl(htbl);
723 return htbl;
724
725out_free_htbl:
726 kfree(htbl);
727 return NULL;
728}
729
730int mlx5dr_ste_htbl_free(struct mlx5dr_ste_htbl *htbl)
731{
732 if (refcount_read(&htbl->refcount))
733 return -EBUSY;
734
735 mlx5dr_icm_free_chunk(htbl->chunk);
736 kfree(htbl);
737 return 0;
738}
739
740int mlx5dr_ste_build_pre_check(struct mlx5dr_domain *dmn,
741 u8 match_criteria,
742 struct mlx5dr_match_param *mask,
743 struct mlx5dr_match_param *value)
744{
745 if (!value && (match_criteria & DR_MATCHER_CRITERIA_MISC)) {
746 if (mask->misc.source_port && mask->misc.source_port != 0xffff) {
747 mlx5dr_dbg(dmn, "Partial mask source_port is not supported\n");
748 return -EINVAL;
749 }
750 }
751
752 return 0;
753}
754
755int mlx5dr_ste_build_ste_arr(struct mlx5dr_matcher *matcher,
756 struct mlx5dr_matcher_rx_tx *nic_matcher,
757 struct mlx5dr_match_param *value,
758 u8 *ste_arr)
759{
760 struct mlx5dr_domain_rx_tx *nic_dmn = nic_matcher->nic_tbl->nic_dmn;
761 struct mlx5dr_domain *dmn = matcher->tbl->dmn;
762 struct mlx5dr_ste_build *sb;
763 int ret, i;
764
765 ret = mlx5dr_ste_build_pre_check(dmn, matcher->match_criteria,
766 &matcher->mask, value);
767 if (ret)
768 return ret;
769
770 sb = nic_matcher->ste_builder;
771 for (i = 0; i < nic_matcher->num_of_builders; i++) {
772 mlx5dr_ste_init(ste_arr,
773 sb->lu_type,
774 nic_dmn->ste_type,
775 dmn->info.caps.gvmi);
776
777 mlx5dr_ste_set_bit_mask(ste_arr, sb->bit_mask);
778
779 ret = sb->ste_build_tag_func(value, sb, ste_arr);
780 if (ret)
781 return ret;
782
783 /* Connect the STEs */
784 if (i < (nic_matcher->num_of_builders - 1)) {
785 /* Need the next builder for these fields,
786 * not relevant for the last ste in the chain.
787 */
788 sb++;
789 MLX5_SET(ste_general, ste_arr, next_lu_type, sb->lu_type);
790 MLX5_SET(ste_general, ste_arr, byte_mask, sb->byte_mask);
791 }
792 ste_arr += DR_STE_SIZE;
793 }
794 return 0;
795}
796
797static int dr_ste_build_eth_l2_src_des_bit_mask(struct mlx5dr_match_param *value,
798 bool inner, u8 *bit_mask)
799{
800 struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
801
802 DR_STE_SET_MASK_V(eth_l2_src_dst, bit_mask, dmac_47_16, mask, dmac_47_16);
803 DR_STE_SET_MASK_V(eth_l2_src_dst, bit_mask, dmac_15_0, mask, dmac_15_0);
804
805 if (mask->smac_47_16 || mask->smac_15_0) {
806 MLX5_SET(ste_eth_l2_src_dst, bit_mask, smac_47_32,
807 mask->smac_47_16 >> 16);
808 MLX5_SET(ste_eth_l2_src_dst, bit_mask, smac_31_0,
809 mask->smac_47_16 << 16 | mask->smac_15_0);
810 mask->smac_47_16 = 0;
811 mask->smac_15_0 = 0;
812 }
813
814 DR_STE_SET_MASK_V(eth_l2_src_dst, bit_mask, first_vlan_id, mask, first_vid);
815 DR_STE_SET_MASK_V(eth_l2_src_dst, bit_mask, first_cfi, mask, first_cfi);
816 DR_STE_SET_MASK_V(eth_l2_src_dst, bit_mask, first_priority, mask, first_prio);
817 DR_STE_SET_MASK(eth_l2_src_dst, bit_mask, l3_type, mask, ip_version);
818
819 if (mask->cvlan_tag) {
820 MLX5_SET(ste_eth_l2_src_dst, bit_mask, first_vlan_qualifier, -1);
821 mask->cvlan_tag = 0;
822 } else if (mask->svlan_tag) {
823 MLX5_SET(ste_eth_l2_src_dst, bit_mask, first_vlan_qualifier, -1);
824 mask->svlan_tag = 0;
825 }
826
827 if (mask->cvlan_tag || mask->svlan_tag) {
828 pr_info("Invalid c/svlan mask configuration\n");
829 return -EINVAL;
830 }
831
832 return 0;
833}
834
835static void dr_ste_copy_mask_misc(char *mask, struct mlx5dr_match_misc *spec)
836{
837 spec->gre_c_present = MLX5_GET(fte_match_set_misc, mask, gre_c_present);
838 spec->gre_k_present = MLX5_GET(fte_match_set_misc, mask, gre_k_present);
839 spec->gre_s_present = MLX5_GET(fte_match_set_misc, mask, gre_s_present);
840 spec->source_vhca_port = MLX5_GET(fte_match_set_misc, mask, source_vhca_port);
841 spec->source_sqn = MLX5_GET(fte_match_set_misc, mask, source_sqn);
842
843 spec->source_port = MLX5_GET(fte_match_set_misc, mask, source_port);
844
845 spec->outer_second_prio = MLX5_GET(fte_match_set_misc, mask, outer_second_prio);
846 spec->outer_second_cfi = MLX5_GET(fte_match_set_misc, mask, outer_second_cfi);
847 spec->outer_second_vid = MLX5_GET(fte_match_set_misc, mask, outer_second_vid);
848 spec->inner_second_prio = MLX5_GET(fte_match_set_misc, mask, inner_second_prio);
849 spec->inner_second_cfi = MLX5_GET(fte_match_set_misc, mask, inner_second_cfi);
850 spec->inner_second_vid = MLX5_GET(fte_match_set_misc, mask, inner_second_vid);
851
852 spec->outer_second_cvlan_tag =
853 MLX5_GET(fte_match_set_misc, mask, outer_second_cvlan_tag);
854 spec->inner_second_cvlan_tag =
855 MLX5_GET(fte_match_set_misc, mask, inner_second_cvlan_tag);
856 spec->outer_second_svlan_tag =
857 MLX5_GET(fte_match_set_misc, mask, outer_second_svlan_tag);
858 spec->inner_second_svlan_tag =
859 MLX5_GET(fte_match_set_misc, mask, inner_second_svlan_tag);
860
861 spec->gre_protocol = MLX5_GET(fte_match_set_misc, mask, gre_protocol);
862
863 spec->gre_key_h = MLX5_GET(fte_match_set_misc, mask, gre_key.nvgre.hi);
864 spec->gre_key_l = MLX5_GET(fte_match_set_misc, mask, gre_key.nvgre.lo);
865
866 spec->vxlan_vni = MLX5_GET(fte_match_set_misc, mask, vxlan_vni);
867
868 spec->geneve_vni = MLX5_GET(fte_match_set_misc, mask, geneve_vni);
869 spec->geneve_oam = MLX5_GET(fte_match_set_misc, mask, geneve_oam);
870
871 spec->outer_ipv6_flow_label =
872 MLX5_GET(fte_match_set_misc, mask, outer_ipv6_flow_label);
873
874 spec->inner_ipv6_flow_label =
875 MLX5_GET(fte_match_set_misc, mask, inner_ipv6_flow_label);
876
877 spec->geneve_opt_len = MLX5_GET(fte_match_set_misc, mask, geneve_opt_len);
878 spec->geneve_protocol_type =
879 MLX5_GET(fte_match_set_misc, mask, geneve_protocol_type);
880
881 spec->bth_dst_qp = MLX5_GET(fte_match_set_misc, mask, bth_dst_qp);
882}
883
884static void dr_ste_copy_mask_spec(char *mask, struct mlx5dr_match_spec *spec)
885{
886 u32 raw_ip[4];
887
888 spec->smac_47_16 = MLX5_GET(fte_match_set_lyr_2_4, mask, smac_47_16);
889
890 spec->smac_15_0 = MLX5_GET(fte_match_set_lyr_2_4, mask, smac_15_0);
891 spec->ethertype = MLX5_GET(fte_match_set_lyr_2_4, mask, ethertype);
892
893 spec->dmac_47_16 = MLX5_GET(fte_match_set_lyr_2_4, mask, dmac_47_16);
894
895 spec->dmac_15_0 = MLX5_GET(fte_match_set_lyr_2_4, mask, dmac_15_0);
896 spec->first_prio = MLX5_GET(fte_match_set_lyr_2_4, mask, first_prio);
897 spec->first_cfi = MLX5_GET(fte_match_set_lyr_2_4, mask, first_cfi);
898 spec->first_vid = MLX5_GET(fte_match_set_lyr_2_4, mask, first_vid);
899
900 spec->ip_protocol = MLX5_GET(fte_match_set_lyr_2_4, mask, ip_protocol);
901 spec->ip_dscp = MLX5_GET(fte_match_set_lyr_2_4, mask, ip_dscp);
902 spec->ip_ecn = MLX5_GET(fte_match_set_lyr_2_4, mask, ip_ecn);
903 spec->cvlan_tag = MLX5_GET(fte_match_set_lyr_2_4, mask, cvlan_tag);
904 spec->svlan_tag = MLX5_GET(fte_match_set_lyr_2_4, mask, svlan_tag);
905 spec->frag = MLX5_GET(fte_match_set_lyr_2_4, mask, frag);
906 spec->ip_version = MLX5_GET(fte_match_set_lyr_2_4, mask, ip_version);
907 spec->tcp_flags = MLX5_GET(fte_match_set_lyr_2_4, mask, tcp_flags);
908 spec->tcp_sport = MLX5_GET(fte_match_set_lyr_2_4, mask, tcp_sport);
909 spec->tcp_dport = MLX5_GET(fte_match_set_lyr_2_4, mask, tcp_dport);
910
911 spec->ttl_hoplimit = MLX5_GET(fte_match_set_lyr_2_4, mask, ttl_hoplimit);
912
913 spec->udp_sport = MLX5_GET(fte_match_set_lyr_2_4, mask, udp_sport);
914 spec->udp_dport = MLX5_GET(fte_match_set_lyr_2_4, mask, udp_dport);
915
916 memcpy(raw_ip, MLX5_ADDR_OF(fte_match_set_lyr_2_4, mask,
917 src_ipv4_src_ipv6.ipv6_layout.ipv6),
918 sizeof(raw_ip));
919
920 spec->src_ip_127_96 = be32_to_cpu(raw_ip[0]);
921 spec->src_ip_95_64 = be32_to_cpu(raw_ip[1]);
922 spec->src_ip_63_32 = be32_to_cpu(raw_ip[2]);
923 spec->src_ip_31_0 = be32_to_cpu(raw_ip[3]);
924
925 memcpy(raw_ip, MLX5_ADDR_OF(fte_match_set_lyr_2_4, mask,
926 dst_ipv4_dst_ipv6.ipv6_layout.ipv6),
927 sizeof(raw_ip));
928
929 spec->dst_ip_127_96 = be32_to_cpu(raw_ip[0]);
930 spec->dst_ip_95_64 = be32_to_cpu(raw_ip[1]);
931 spec->dst_ip_63_32 = be32_to_cpu(raw_ip[2]);
932 spec->dst_ip_31_0 = be32_to_cpu(raw_ip[3]);
933}
934
935static void dr_ste_copy_mask_misc2(char *mask, struct mlx5dr_match_misc2 *spec)
936{
937 spec->outer_first_mpls_label =
938 MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls.mpls_label);
939 spec->outer_first_mpls_exp =
940 MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls.mpls_exp);
941 spec->outer_first_mpls_s_bos =
942 MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls.mpls_s_bos);
943 spec->outer_first_mpls_ttl =
944 MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls.mpls_ttl);
945 spec->inner_first_mpls_label =
946 MLX5_GET(fte_match_set_misc2, mask, inner_first_mpls.mpls_label);
947 spec->inner_first_mpls_exp =
948 MLX5_GET(fte_match_set_misc2, mask, inner_first_mpls.mpls_exp);
949 spec->inner_first_mpls_s_bos =
950 MLX5_GET(fte_match_set_misc2, mask, inner_first_mpls.mpls_s_bos);
951 spec->inner_first_mpls_ttl =
952 MLX5_GET(fte_match_set_misc2, mask, inner_first_mpls.mpls_ttl);
953 spec->outer_first_mpls_over_gre_label =
954 MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_gre.mpls_label);
955 spec->outer_first_mpls_over_gre_exp =
956 MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_gre.mpls_exp);
957 spec->outer_first_mpls_over_gre_s_bos =
958 MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_gre.mpls_s_bos);
959 spec->outer_first_mpls_over_gre_ttl =
960 MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_gre.mpls_ttl);
961 spec->outer_first_mpls_over_udp_label =
962 MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_udp.mpls_label);
963 spec->outer_first_mpls_over_udp_exp =
964 MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_udp.mpls_exp);
965 spec->outer_first_mpls_over_udp_s_bos =
966 MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_udp.mpls_s_bos);
967 spec->outer_first_mpls_over_udp_ttl =
968 MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_udp.mpls_ttl);
969 spec->metadata_reg_c_7 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_7);
970 spec->metadata_reg_c_6 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_6);
971 spec->metadata_reg_c_5 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_5);
972 spec->metadata_reg_c_4 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_4);
973 spec->metadata_reg_c_3 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_3);
974 spec->metadata_reg_c_2 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_2);
975 spec->metadata_reg_c_1 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_1);
976 spec->metadata_reg_c_0 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_0);
977 spec->metadata_reg_a = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_a);
978 spec->metadata_reg_b = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_b);
979}
980
981static void dr_ste_copy_mask_misc3(char *mask, struct mlx5dr_match_misc3 *spec)
982{
983 spec->inner_tcp_seq_num = MLX5_GET(fte_match_set_misc3, mask, inner_tcp_seq_num);
984 spec->outer_tcp_seq_num = MLX5_GET(fte_match_set_misc3, mask, outer_tcp_seq_num);
985 spec->inner_tcp_ack_num = MLX5_GET(fte_match_set_misc3, mask, inner_tcp_ack_num);
986 spec->outer_tcp_ack_num = MLX5_GET(fte_match_set_misc3, mask, outer_tcp_ack_num);
987 spec->outer_vxlan_gpe_vni =
988 MLX5_GET(fte_match_set_misc3, mask, outer_vxlan_gpe_vni);
989 spec->outer_vxlan_gpe_next_protocol =
990 MLX5_GET(fte_match_set_misc3, mask, outer_vxlan_gpe_next_protocol);
991 spec->outer_vxlan_gpe_flags =
992 MLX5_GET(fte_match_set_misc3, mask, outer_vxlan_gpe_flags);
993 spec->icmpv4_header_data = MLX5_GET(fte_match_set_misc3, mask, icmp_header_data);
994 spec->icmpv6_header_data =
995 MLX5_GET(fte_match_set_misc3, mask, icmpv6_header_data);
996 spec->icmpv4_type = MLX5_GET(fte_match_set_misc3, mask, icmp_type);
997 spec->icmpv4_code = MLX5_GET(fte_match_set_misc3, mask, icmp_code);
998 spec->icmpv6_type = MLX5_GET(fte_match_set_misc3, mask, icmpv6_type);
999 spec->icmpv6_code = MLX5_GET(fte_match_set_misc3, mask, icmpv6_code);
1000}
1001
1002void mlx5dr_ste_copy_param(u8 match_criteria,
1003 struct mlx5dr_match_param *set_param,
1004 struct mlx5dr_match_parameters *mask)
1005{
1006 u8 tail_param[MLX5_ST_SZ_BYTES(fte_match_set_lyr_2_4)] = {};
1007 u8 *data = (u8 *)mask->match_buf;
1008 size_t param_location;
1009 void *buff;
1010
1011 if (match_criteria & DR_MATCHER_CRITERIA_OUTER) {
1012 if (mask->match_sz < sizeof(struct mlx5dr_match_spec)) {
1013 memcpy(tail_param, data, mask->match_sz);
1014 buff = tail_param;
1015 } else {
1016 buff = mask->match_buf;
1017 }
1018 dr_ste_copy_mask_spec(buff, &set_param->outer);
1019 }
1020 param_location = sizeof(struct mlx5dr_match_spec);
1021
1022 if (match_criteria & DR_MATCHER_CRITERIA_MISC) {
1023 if (mask->match_sz < param_location +
1024 sizeof(struct mlx5dr_match_misc)) {
1025 memcpy(tail_param, data + param_location,
1026 mask->match_sz - param_location);
1027 buff = tail_param;
1028 } else {
1029 buff = data + param_location;
1030 }
1031 dr_ste_copy_mask_misc(buff, &set_param->misc);
1032 }
1033 param_location += sizeof(struct mlx5dr_match_misc);
1034
1035 if (match_criteria & DR_MATCHER_CRITERIA_INNER) {
1036 if (mask->match_sz < param_location +
1037 sizeof(struct mlx5dr_match_spec)) {
1038 memcpy(tail_param, data + param_location,
1039 mask->match_sz - param_location);
1040 buff = tail_param;
1041 } else {
1042 buff = data + param_location;
1043 }
1044 dr_ste_copy_mask_spec(buff, &set_param->inner);
1045 }
1046 param_location += sizeof(struct mlx5dr_match_spec);
1047
1048 if (match_criteria & DR_MATCHER_CRITERIA_MISC2) {
1049 if (mask->match_sz < param_location +
1050 sizeof(struct mlx5dr_match_misc2)) {
1051 memcpy(tail_param, data + param_location,
1052 mask->match_sz - param_location);
1053 buff = tail_param;
1054 } else {
1055 buff = data + param_location;
1056 }
1057 dr_ste_copy_mask_misc2(buff, &set_param->misc2);
1058 }
1059
1060 param_location += sizeof(struct mlx5dr_match_misc2);
1061
1062 if (match_criteria & DR_MATCHER_CRITERIA_MISC3) {
1063 if (mask->match_sz < param_location +
1064 sizeof(struct mlx5dr_match_misc3)) {
1065 memcpy(tail_param, data + param_location,
1066 mask->match_sz - param_location);
1067 buff = tail_param;
1068 } else {
1069 buff = data + param_location;
1070 }
1071 dr_ste_copy_mask_misc3(buff, &set_param->misc3);
1072 }
1073}
1074
1075static int dr_ste_build_eth_l2_src_des_tag(struct mlx5dr_match_param *value,
1076 struct mlx5dr_ste_build *sb,
1077 u8 *hw_ste_p)
1078{
1079 struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1080 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1081 u8 *tag = hw_ste->tag;
1082
1083 DR_STE_SET_TAG(eth_l2_src_dst, tag, dmac_47_16, spec, dmac_47_16);
1084 DR_STE_SET_TAG(eth_l2_src_dst, tag, dmac_15_0, spec, dmac_15_0);
1085
1086 if (spec->smac_47_16 || spec->smac_15_0) {
1087 MLX5_SET(ste_eth_l2_src_dst, tag, smac_47_32,
1088 spec->smac_47_16 >> 16);
1089 MLX5_SET(ste_eth_l2_src_dst, tag, smac_31_0,
1090 spec->smac_47_16 << 16 | spec->smac_15_0);
1091 spec->smac_47_16 = 0;
1092 spec->smac_15_0 = 0;
1093 }
1094
1095 if (spec->ip_version) {
1096 if (spec->ip_version == IP_VERSION_IPV4) {
1097 MLX5_SET(ste_eth_l2_src_dst, tag, l3_type, STE_IPV4);
1098 spec->ip_version = 0;
1099 } else if (spec->ip_version == IP_VERSION_IPV6) {
1100 MLX5_SET(ste_eth_l2_src_dst, tag, l3_type, STE_IPV6);
1101 spec->ip_version = 0;
1102 } else {
1103 pr_info("Unsupported ip_version value\n");
1104 return -EINVAL;
1105 }
1106 }
1107
1108 DR_STE_SET_TAG(eth_l2_src_dst, tag, first_vlan_id, spec, first_vid);
1109 DR_STE_SET_TAG(eth_l2_src_dst, tag, first_cfi, spec, first_cfi);
1110 DR_STE_SET_TAG(eth_l2_src_dst, tag, first_priority, spec, first_prio);
1111
1112 if (spec->cvlan_tag) {
1113 MLX5_SET(ste_eth_l2_src_dst, tag, first_vlan_qualifier, DR_STE_CVLAN);
1114 spec->cvlan_tag = 0;
1115 } else if (spec->svlan_tag) {
1116 MLX5_SET(ste_eth_l2_src_dst, tag, first_vlan_qualifier, DR_STE_SVLAN);
1117 spec->svlan_tag = 0;
1118 }
1119 return 0;
1120}
1121
1122int mlx5dr_ste_build_eth_l2_src_des(struct mlx5dr_ste_build *sb,
1123 struct mlx5dr_match_param *mask,
1124 bool inner, bool rx)
1125{
1126 int ret;
1127
1128 ret = dr_ste_build_eth_l2_src_des_bit_mask(mask, inner, sb->bit_mask);
1129 if (ret)
1130 return ret;
1131
1132 sb->rx = rx;
1133 sb->inner = inner;
1134 sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL2_SRC_DST, rx, inner);
1135 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1136 sb->ste_build_tag_func = &dr_ste_build_eth_l2_src_des_tag;
1137
1138 return 0;
1139}
1140
1141static void dr_ste_build_eth_l3_ipv6_dst_bit_mask(struct mlx5dr_match_param *value,
1142 bool inner, u8 *bit_mask)
1143{
1144 struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1145
1146 DR_STE_SET_MASK_V(eth_l3_ipv6_dst, bit_mask, dst_ip_127_96, mask, dst_ip_127_96);
1147 DR_STE_SET_MASK_V(eth_l3_ipv6_dst, bit_mask, dst_ip_95_64, mask, dst_ip_95_64);
1148 DR_STE_SET_MASK_V(eth_l3_ipv6_dst, bit_mask, dst_ip_63_32, mask, dst_ip_63_32);
1149 DR_STE_SET_MASK_V(eth_l3_ipv6_dst, bit_mask, dst_ip_31_0, mask, dst_ip_31_0);
1150}
1151
1152static int dr_ste_build_eth_l3_ipv6_dst_tag(struct mlx5dr_match_param *value,
1153 struct mlx5dr_ste_build *sb,
1154 u8 *hw_ste_p)
1155{
1156 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1157 struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1158 u8 *tag = hw_ste->tag;
1159
1160 DR_STE_SET_TAG(eth_l3_ipv6_dst, tag, dst_ip_127_96, spec, dst_ip_127_96);
1161 DR_STE_SET_TAG(eth_l3_ipv6_dst, tag, dst_ip_95_64, spec, dst_ip_95_64);
1162 DR_STE_SET_TAG(eth_l3_ipv6_dst, tag, dst_ip_63_32, spec, dst_ip_63_32);
1163 DR_STE_SET_TAG(eth_l3_ipv6_dst, tag, dst_ip_31_0, spec, dst_ip_31_0);
1164
1165 return 0;
1166}
1167
1168void mlx5dr_ste_build_eth_l3_ipv6_dst(struct mlx5dr_ste_build *sb,
1169 struct mlx5dr_match_param *mask,
1170 bool inner, bool rx)
1171{
1172 dr_ste_build_eth_l3_ipv6_dst_bit_mask(mask, inner, sb->bit_mask);
1173
1174 sb->rx = rx;
1175 sb->inner = inner;
1176 sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL3_IPV6_DST, rx, inner);
1177 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1178 sb->ste_build_tag_func = &dr_ste_build_eth_l3_ipv6_dst_tag;
1179}
1180
1181static void dr_ste_build_eth_l3_ipv6_src_bit_mask(struct mlx5dr_match_param *value,
1182 bool inner, u8 *bit_mask)
1183{
1184 struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1185
1186 DR_STE_SET_MASK_V(eth_l3_ipv6_src, bit_mask, src_ip_127_96, mask, src_ip_127_96);
1187 DR_STE_SET_MASK_V(eth_l3_ipv6_src, bit_mask, src_ip_95_64, mask, src_ip_95_64);
1188 DR_STE_SET_MASK_V(eth_l3_ipv6_src, bit_mask, src_ip_63_32, mask, src_ip_63_32);
1189 DR_STE_SET_MASK_V(eth_l3_ipv6_src, bit_mask, src_ip_31_0, mask, src_ip_31_0);
1190}
1191
1192static int dr_ste_build_eth_l3_ipv6_src_tag(struct mlx5dr_match_param *value,
1193 struct mlx5dr_ste_build *sb,
1194 u8 *hw_ste_p)
1195{
1196 struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1197 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1198 u8 *tag = hw_ste->tag;
1199
1200 DR_STE_SET_TAG(eth_l3_ipv6_src, tag, src_ip_127_96, spec, src_ip_127_96);
1201 DR_STE_SET_TAG(eth_l3_ipv6_src, tag, src_ip_95_64, spec, src_ip_95_64);
1202 DR_STE_SET_TAG(eth_l3_ipv6_src, tag, src_ip_63_32, spec, src_ip_63_32);
1203 DR_STE_SET_TAG(eth_l3_ipv6_src, tag, src_ip_31_0, spec, src_ip_31_0);
1204
1205 return 0;
1206}
1207
1208void mlx5dr_ste_build_eth_l3_ipv6_src(struct mlx5dr_ste_build *sb,
1209 struct mlx5dr_match_param *mask,
1210 bool inner, bool rx)
1211{
1212 dr_ste_build_eth_l3_ipv6_src_bit_mask(mask, inner, sb->bit_mask);
1213
1214 sb->rx = rx;
1215 sb->inner = inner;
1216 sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL3_IPV6_SRC, rx, inner);
1217 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1218 sb->ste_build_tag_func = &dr_ste_build_eth_l3_ipv6_src_tag;
1219}
1220
1221static void dr_ste_build_eth_l3_ipv4_5_tuple_bit_mask(struct mlx5dr_match_param *value,
1222 bool inner,
1223 u8 *bit_mask)
1224{
1225 struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1226
1227 DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1228 destination_address, mask, dst_ip_31_0);
1229 DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1230 source_address, mask, src_ip_31_0);
1231 DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1232 destination_port, mask, tcp_dport);
1233 DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1234 destination_port, mask, udp_dport);
1235 DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1236 source_port, mask, tcp_sport);
1237 DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1238 source_port, mask, udp_sport);
1239 DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1240 protocol, mask, ip_protocol);
1241 DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1242 fragmented, mask, frag);
1243 DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1244 dscp, mask, ip_dscp);
1245 DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1246 ecn, mask, ip_ecn);
1247
1248 if (mask->tcp_flags) {
1249 DR_STE_SET_TCP_FLAGS(eth_l3_ipv4_5_tuple, bit_mask, mask);
1250 mask->tcp_flags = 0;
1251 }
1252}
1253
1254static int dr_ste_build_eth_l3_ipv4_5_tuple_tag(struct mlx5dr_match_param *value,
1255 struct mlx5dr_ste_build *sb,
1256 u8 *hw_ste_p)
1257{
1258 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1259 struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1260 u8 *tag = hw_ste->tag;
1261
1262 DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, destination_address, spec, dst_ip_31_0);
1263 DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, source_address, spec, src_ip_31_0);
1264 DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, destination_port, spec, tcp_dport);
1265 DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, destination_port, spec, udp_dport);
1266 DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, source_port, spec, tcp_sport);
1267 DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, source_port, spec, udp_sport);
1268 DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, protocol, spec, ip_protocol);
1269 DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, fragmented, spec, frag);
1270 DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, dscp, spec, ip_dscp);
1271 DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, ecn, spec, ip_ecn);
1272
1273 if (spec->tcp_flags) {
1274 DR_STE_SET_TCP_FLAGS(eth_l3_ipv4_5_tuple, tag, spec);
1275 spec->tcp_flags = 0;
1276 }
1277
1278 return 0;
1279}
1280
1281void mlx5dr_ste_build_eth_l3_ipv4_5_tuple(struct mlx5dr_ste_build *sb,
1282 struct mlx5dr_match_param *mask,
1283 bool inner, bool rx)
1284{
1285 dr_ste_build_eth_l3_ipv4_5_tuple_bit_mask(mask, inner, sb->bit_mask);
1286
1287 sb->rx = rx;
1288 sb->inner = inner;
1289 sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL3_IPV4_5_TUPLE, rx, inner);
1290 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1291 sb->ste_build_tag_func = &dr_ste_build_eth_l3_ipv4_5_tuple_tag;
1292}
1293
1294static void
1295dr_ste_build_eth_l2_src_or_dst_bit_mask(struct mlx5dr_match_param *value,
1296 bool inner, u8 *bit_mask)
1297{
1298 struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1299 struct mlx5dr_match_misc *misc_mask = &value->misc;
1300
1301 DR_STE_SET_MASK_V(eth_l2_src, bit_mask, first_vlan_id, mask, first_vid);
1302 DR_STE_SET_MASK_V(eth_l2_src, bit_mask, first_cfi, mask, first_cfi);
1303 DR_STE_SET_MASK_V(eth_l2_src, bit_mask, first_priority, mask, first_prio);
1304 DR_STE_SET_MASK_V(eth_l2_src, bit_mask, ip_fragmented, mask, frag);
1305 DR_STE_SET_MASK_V(eth_l2_src, bit_mask, l3_ethertype, mask, ethertype);
1306 DR_STE_SET_MASK(eth_l2_src, bit_mask, l3_type, mask, ip_version);
1307
1308 if (mask->svlan_tag || mask->cvlan_tag) {
1309 MLX5_SET(ste_eth_l2_src, bit_mask, first_vlan_qualifier, -1);
1310 mask->cvlan_tag = 0;
1311 mask->svlan_tag = 0;
1312 }
1313
1314 if (inner) {
1315 if (misc_mask->inner_second_cvlan_tag ||
1316 misc_mask->inner_second_svlan_tag) {
1317 MLX5_SET(ste_eth_l2_src, bit_mask, second_vlan_qualifier, -1);
1318 misc_mask->inner_second_cvlan_tag = 0;
1319 misc_mask->inner_second_svlan_tag = 0;
1320 }
1321
1322 DR_STE_SET_MASK_V(eth_l2_src, bit_mask,
1323 second_vlan_id, misc_mask, inner_second_vid);
1324 DR_STE_SET_MASK_V(eth_l2_src, bit_mask,
1325 second_cfi, misc_mask, inner_second_cfi);
1326 DR_STE_SET_MASK_V(eth_l2_src, bit_mask,
1327 second_priority, misc_mask, inner_second_prio);
1328 } else {
1329 if (misc_mask->outer_second_cvlan_tag ||
1330 misc_mask->outer_second_svlan_tag) {
1331 MLX5_SET(ste_eth_l2_src, bit_mask, second_vlan_qualifier, -1);
1332 misc_mask->outer_second_cvlan_tag = 0;
1333 misc_mask->outer_second_svlan_tag = 0;
1334 }
1335
1336 DR_STE_SET_MASK_V(eth_l2_src, bit_mask,
1337 second_vlan_id, misc_mask, outer_second_vid);
1338 DR_STE_SET_MASK_V(eth_l2_src, bit_mask,
1339 second_cfi, misc_mask, outer_second_cfi);
1340 DR_STE_SET_MASK_V(eth_l2_src, bit_mask,
1341 second_priority, misc_mask, outer_second_prio);
1342 }
1343}
1344
1345static int dr_ste_build_eth_l2_src_or_dst_tag(struct mlx5dr_match_param *value,
1346 bool inner, u8 *hw_ste_p)
1347{
1348 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1349 struct mlx5dr_match_spec *spec = inner ? &value->inner : &value->outer;
1350 struct mlx5dr_match_misc *misc_spec = &value->misc;
1351 u8 *tag = hw_ste->tag;
1352
1353 DR_STE_SET_TAG(eth_l2_src, tag, first_vlan_id, spec, first_vid);
1354 DR_STE_SET_TAG(eth_l2_src, tag, first_cfi, spec, first_cfi);
1355 DR_STE_SET_TAG(eth_l2_src, tag, first_priority, spec, first_prio);
1356 DR_STE_SET_TAG(eth_l2_src, tag, ip_fragmented, spec, frag);
1357 DR_STE_SET_TAG(eth_l2_src, tag, l3_ethertype, spec, ethertype);
1358
1359 if (spec->ip_version) {
1360 if (spec->ip_version == IP_VERSION_IPV4) {
1361 MLX5_SET(ste_eth_l2_src, tag, l3_type, STE_IPV4);
1362 spec->ip_version = 0;
1363 } else if (spec->ip_version == IP_VERSION_IPV6) {
1364 MLX5_SET(ste_eth_l2_src, tag, l3_type, STE_IPV6);
1365 spec->ip_version = 0;
1366 } else {
1367 pr_info("Unsupported ip_version value\n");
1368 return -EINVAL;
1369 }
1370 }
1371
1372 if (spec->cvlan_tag) {
1373 MLX5_SET(ste_eth_l2_src, tag, first_vlan_qualifier, DR_STE_CVLAN);
1374 spec->cvlan_tag = 0;
1375 } else if (spec->svlan_tag) {
1376 MLX5_SET(ste_eth_l2_src, tag, first_vlan_qualifier, DR_STE_SVLAN);
1377 spec->svlan_tag = 0;
1378 }
1379
1380 if (inner) {
1381 if (misc_spec->inner_second_cvlan_tag) {
1382 MLX5_SET(ste_eth_l2_src, tag, second_vlan_qualifier, DR_STE_CVLAN);
1383 misc_spec->inner_second_cvlan_tag = 0;
1384 } else if (misc_spec->inner_second_svlan_tag) {
1385 MLX5_SET(ste_eth_l2_src, tag, second_vlan_qualifier, DR_STE_SVLAN);
1386 misc_spec->inner_second_svlan_tag = 0;
1387 }
1388
1389 DR_STE_SET_TAG(eth_l2_src, tag, second_vlan_id, misc_spec, inner_second_vid);
1390 DR_STE_SET_TAG(eth_l2_src, tag, second_cfi, misc_spec, inner_second_cfi);
1391 DR_STE_SET_TAG(eth_l2_src, tag, second_priority, misc_spec, inner_second_prio);
1392 } else {
1393 if (misc_spec->outer_second_cvlan_tag) {
1394 MLX5_SET(ste_eth_l2_src, tag, second_vlan_qualifier, DR_STE_CVLAN);
1395 misc_spec->outer_second_cvlan_tag = 0;
1396 } else if (misc_spec->outer_second_svlan_tag) {
1397 MLX5_SET(ste_eth_l2_src, tag, second_vlan_qualifier, DR_STE_SVLAN);
1398 misc_spec->outer_second_svlan_tag = 0;
1399 }
1400 DR_STE_SET_TAG(eth_l2_src, tag, second_vlan_id, misc_spec, outer_second_vid);
1401 DR_STE_SET_TAG(eth_l2_src, tag, second_cfi, misc_spec, outer_second_cfi);
1402 DR_STE_SET_TAG(eth_l2_src, tag, second_priority, misc_spec, outer_second_prio);
1403 }
1404
1405 return 0;
1406}
1407
1408static void dr_ste_build_eth_l2_src_bit_mask(struct mlx5dr_match_param *value,
1409 bool inner, u8 *bit_mask)
1410{
1411 struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1412
1413 DR_STE_SET_MASK_V(eth_l2_src, bit_mask, smac_47_16, mask, smac_47_16);
1414 DR_STE_SET_MASK_V(eth_l2_src, bit_mask, smac_15_0, mask, smac_15_0);
1415
1416 dr_ste_build_eth_l2_src_or_dst_bit_mask(value, inner, bit_mask);
1417}
1418
1419static int dr_ste_build_eth_l2_src_tag(struct mlx5dr_match_param *value,
1420 struct mlx5dr_ste_build *sb,
1421 u8 *hw_ste_p)
1422{
1423 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1424 struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1425 u8 *tag = hw_ste->tag;
1426
1427 DR_STE_SET_TAG(eth_l2_src, tag, smac_47_16, spec, smac_47_16);
1428 DR_STE_SET_TAG(eth_l2_src, tag, smac_15_0, spec, smac_15_0);
1429
1430 return dr_ste_build_eth_l2_src_or_dst_tag(value, sb->inner, hw_ste_p);
1431}
1432
1433void mlx5dr_ste_build_eth_l2_src(struct mlx5dr_ste_build *sb,
1434 struct mlx5dr_match_param *mask,
1435 bool inner, bool rx)
1436{
1437 dr_ste_build_eth_l2_src_bit_mask(mask, inner, sb->bit_mask);
1438 sb->rx = rx;
1439 sb->inner = inner;
1440 sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL2_SRC, rx, inner);
1441 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1442 sb->ste_build_tag_func = &dr_ste_build_eth_l2_src_tag;
1443}
1444
1445static void dr_ste_build_eth_l2_dst_bit_mask(struct mlx5dr_match_param *value,
1446 bool inner, u8 *bit_mask)
1447{
1448 struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1449
1450 DR_STE_SET_MASK_V(eth_l2_dst, bit_mask, dmac_47_16, mask, dmac_47_16);
1451 DR_STE_SET_MASK_V(eth_l2_dst, bit_mask, dmac_15_0, mask, dmac_15_0);
1452
1453 dr_ste_build_eth_l2_src_or_dst_bit_mask(value, inner, bit_mask);
1454}
1455
1456static int dr_ste_build_eth_l2_dst_tag(struct mlx5dr_match_param *value,
1457 struct mlx5dr_ste_build *sb,
1458 u8 *hw_ste_p)
1459{
1460 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1461 struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1462 u8 *tag = hw_ste->tag;
1463
1464 DR_STE_SET_TAG(eth_l2_dst, tag, dmac_47_16, spec, dmac_47_16);
1465 DR_STE_SET_TAG(eth_l2_dst, tag, dmac_15_0, spec, dmac_15_0);
1466
1467 return dr_ste_build_eth_l2_src_or_dst_tag(value, sb->inner, hw_ste_p);
1468}
1469
1470void mlx5dr_ste_build_eth_l2_dst(struct mlx5dr_ste_build *sb,
1471 struct mlx5dr_match_param *mask,
1472 bool inner, bool rx)
1473{
1474 dr_ste_build_eth_l2_dst_bit_mask(mask, inner, sb->bit_mask);
1475
1476 sb->rx = rx;
1477 sb->inner = inner;
1478 sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL2_DST, rx, inner);
1479 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1480 sb->ste_build_tag_func = &dr_ste_build_eth_l2_dst_tag;
1481}
1482
1483static void dr_ste_build_eth_l2_tnl_bit_mask(struct mlx5dr_match_param *value,
1484 bool inner, u8 *bit_mask)
1485{
1486 struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1487 struct mlx5dr_match_misc *misc = &value->misc;
1488
1489 DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, dmac_47_16, mask, dmac_47_16);
1490 DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, dmac_15_0, mask, dmac_15_0);
1491 DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, first_vlan_id, mask, first_vid);
1492 DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, first_cfi, mask, first_cfi);
1493 DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, first_priority, mask, first_prio);
1494 DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, ip_fragmented, mask, frag);
1495 DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, l3_ethertype, mask, ethertype);
1496 DR_STE_SET_MASK(eth_l2_tnl, bit_mask, l3_type, mask, ip_version);
1497
1498 if (misc->vxlan_vni) {
1499 MLX5_SET(ste_eth_l2_tnl, bit_mask,
1500 l2_tunneling_network_id, (misc->vxlan_vni << 8));
1501 misc->vxlan_vni = 0;
1502 }
1503
1504 if (mask->svlan_tag || mask->cvlan_tag) {
1505 MLX5_SET(ste_eth_l2_tnl, bit_mask, first_vlan_qualifier, -1);
1506 mask->cvlan_tag = 0;
1507 mask->svlan_tag = 0;
1508 }
1509}
1510
1511static int dr_ste_build_eth_l2_tnl_tag(struct mlx5dr_match_param *value,
1512 struct mlx5dr_ste_build *sb,
1513 u8 *hw_ste_p)
1514{
1515 struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1516 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1517 struct mlx5dr_match_misc *misc = &value->misc;
1518 u8 *tag = hw_ste->tag;
1519
1520 DR_STE_SET_TAG(eth_l2_tnl, tag, dmac_47_16, spec, dmac_47_16);
1521 DR_STE_SET_TAG(eth_l2_tnl, tag, dmac_15_0, spec, dmac_15_0);
1522 DR_STE_SET_TAG(eth_l2_tnl, tag, first_vlan_id, spec, first_vid);
1523 DR_STE_SET_TAG(eth_l2_tnl, tag, first_cfi, spec, first_cfi);
1524 DR_STE_SET_TAG(eth_l2_tnl, tag, ip_fragmented, spec, frag);
1525 DR_STE_SET_TAG(eth_l2_tnl, tag, first_priority, spec, first_prio);
1526 DR_STE_SET_TAG(eth_l2_tnl, tag, l3_ethertype, spec, ethertype);
1527
1528 if (misc->vxlan_vni) {
1529 MLX5_SET(ste_eth_l2_tnl, tag, l2_tunneling_network_id,
1530 (misc->vxlan_vni << 8));
1531 misc->vxlan_vni = 0;
1532 }
1533
1534 if (spec->cvlan_tag) {
1535 MLX5_SET(ste_eth_l2_tnl, tag, first_vlan_qualifier, DR_STE_CVLAN);
1536 spec->cvlan_tag = 0;
1537 } else if (spec->svlan_tag) {
1538 MLX5_SET(ste_eth_l2_tnl, tag, first_vlan_qualifier, DR_STE_SVLAN);
1539 spec->svlan_tag = 0;
1540 }
1541
1542 if (spec->ip_version) {
1543 if (spec->ip_version == IP_VERSION_IPV4) {
1544 MLX5_SET(ste_eth_l2_tnl, tag, l3_type, STE_IPV4);
1545 spec->ip_version = 0;
1546 } else if (spec->ip_version == IP_VERSION_IPV6) {
1547 MLX5_SET(ste_eth_l2_tnl, tag, l3_type, STE_IPV6);
1548 spec->ip_version = 0;
1549 } else {
1550 return -EINVAL;
1551 }
1552 }
1553
1554 return 0;
1555}
1556
1557void mlx5dr_ste_build_eth_l2_tnl(struct mlx5dr_ste_build *sb,
1558 struct mlx5dr_match_param *mask, bool inner, bool rx)
1559{
1560 dr_ste_build_eth_l2_tnl_bit_mask(mask, inner, sb->bit_mask);
1561
1562 sb->rx = rx;
1563 sb->inner = inner;
1564 sb->lu_type = MLX5DR_STE_LU_TYPE_ETHL2_TUNNELING_I;
1565 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1566 sb->ste_build_tag_func = &dr_ste_build_eth_l2_tnl_tag;
1567}
1568
1569static void dr_ste_build_eth_l3_ipv4_misc_bit_mask(struct mlx5dr_match_param *value,
1570 bool inner, u8 *bit_mask)
1571{
1572 struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1573
1574 DR_STE_SET_MASK_V(eth_l3_ipv4_misc, bit_mask, time_to_live, mask, ttl_hoplimit);
1575}
1576
1577static int dr_ste_build_eth_l3_ipv4_misc_tag(struct mlx5dr_match_param *value,
1578 struct mlx5dr_ste_build *sb,
1579 u8 *hw_ste_p)
1580{
1581 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1582 struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1583 u8 *tag = hw_ste->tag;
1584
1585 DR_STE_SET_TAG(eth_l3_ipv4_misc, tag, time_to_live, spec, ttl_hoplimit);
1586
1587 return 0;
1588}
1589
1590void mlx5dr_ste_build_eth_l3_ipv4_misc(struct mlx5dr_ste_build *sb,
1591 struct mlx5dr_match_param *mask,
1592 bool inner, bool rx)
1593{
1594 dr_ste_build_eth_l3_ipv4_misc_bit_mask(mask, inner, sb->bit_mask);
1595
1596 sb->rx = rx;
1597 sb->inner = inner;
1598 sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL3_IPV4_MISC, rx, inner);
1599 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1600 sb->ste_build_tag_func = &dr_ste_build_eth_l3_ipv4_misc_tag;
1601}
1602
1603static void dr_ste_build_ipv6_l3_l4_bit_mask(struct mlx5dr_match_param *value,
1604 bool inner, u8 *bit_mask)
1605{
1606 struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1607
1608 DR_STE_SET_MASK_V(eth_l4, bit_mask, dst_port, mask, tcp_dport);
1609 DR_STE_SET_MASK_V(eth_l4, bit_mask, src_port, mask, tcp_sport);
1610 DR_STE_SET_MASK_V(eth_l4, bit_mask, dst_port, mask, udp_dport);
1611 DR_STE_SET_MASK_V(eth_l4, bit_mask, src_port, mask, udp_sport);
1612 DR_STE_SET_MASK_V(eth_l4, bit_mask, protocol, mask, ip_protocol);
1613 DR_STE_SET_MASK_V(eth_l4, bit_mask, fragmented, mask, frag);
1614 DR_STE_SET_MASK_V(eth_l4, bit_mask, dscp, mask, ip_dscp);
1615 DR_STE_SET_MASK_V(eth_l4, bit_mask, ecn, mask, ip_ecn);
1616 DR_STE_SET_MASK_V(eth_l4, bit_mask, ipv6_hop_limit, mask, ttl_hoplimit);
1617
1618 if (mask->tcp_flags) {
1619 DR_STE_SET_TCP_FLAGS(eth_l4, bit_mask, mask);
1620 mask->tcp_flags = 0;
1621 }
1622}
1623
1624static int dr_ste_build_ipv6_l3_l4_tag(struct mlx5dr_match_param *value,
1625 struct mlx5dr_ste_build *sb,
1626 u8 *hw_ste_p)
1627{
1628 struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1629 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1630 u8 *tag = hw_ste->tag;
1631
1632 DR_STE_SET_TAG(eth_l4, tag, dst_port, spec, tcp_dport);
1633 DR_STE_SET_TAG(eth_l4, tag, src_port, spec, tcp_sport);
1634 DR_STE_SET_TAG(eth_l4, tag, dst_port, spec, udp_dport);
1635 DR_STE_SET_TAG(eth_l4, tag, src_port, spec, udp_sport);
1636 DR_STE_SET_TAG(eth_l4, tag, protocol, spec, ip_protocol);
1637 DR_STE_SET_TAG(eth_l4, tag, fragmented, spec, frag);
1638 DR_STE_SET_TAG(eth_l4, tag, dscp, spec, ip_dscp);
1639 DR_STE_SET_TAG(eth_l4, tag, ecn, spec, ip_ecn);
1640 DR_STE_SET_TAG(eth_l4, tag, ipv6_hop_limit, spec, ttl_hoplimit);
1641
1642 if (spec->tcp_flags) {
1643 DR_STE_SET_TCP_FLAGS(eth_l4, tag, spec);
1644 spec->tcp_flags = 0;
1645 }
1646
1647 return 0;
1648}
1649
1650void mlx5dr_ste_build_ipv6_l3_l4(struct mlx5dr_ste_build *sb,
1651 struct mlx5dr_match_param *mask,
1652 bool inner, bool rx)
1653{
1654 dr_ste_build_ipv6_l3_l4_bit_mask(mask, inner, sb->bit_mask);
1655
1656 sb->rx = rx;
1657 sb->inner = inner;
1658 sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL4, rx, inner);
1659 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1660 sb->ste_build_tag_func = &dr_ste_build_ipv6_l3_l4_tag;
1661}
1662
1663static int dr_ste_build_empty_always_hit_tag(struct mlx5dr_match_param *value,
1664 struct mlx5dr_ste_build *sb,
1665 u8 *hw_ste_p)
1666{
1667 return 0;
1668}
1669
1670void mlx5dr_ste_build_empty_always_hit(struct mlx5dr_ste_build *sb, bool rx)
1671{
1672 sb->rx = rx;
1673 sb->lu_type = MLX5DR_STE_LU_TYPE_DONT_CARE;
1674 sb->byte_mask = 0;
1675 sb->ste_build_tag_func = &dr_ste_build_empty_always_hit_tag;
1676}
1677
1678static void dr_ste_build_mpls_bit_mask(struct mlx5dr_match_param *value,
1679 bool inner, u8 *bit_mask)
1680{
1681 struct mlx5dr_match_misc2 *misc2_mask = &value->misc2;
1682
1683 if (inner)
1684 DR_STE_SET_MPLS_MASK(mpls, misc2_mask, inner, bit_mask);
1685 else
1686 DR_STE_SET_MPLS_MASK(mpls, misc2_mask, outer, bit_mask);
1687}
1688
1689static int dr_ste_build_mpls_tag(struct mlx5dr_match_param *value,
1690 struct mlx5dr_ste_build *sb,
1691 u8 *hw_ste_p)
1692{
1693 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1694 struct mlx5dr_match_misc2 *misc2_mask = &value->misc2;
1695 u8 *tag = hw_ste->tag;
1696
1697 if (sb->inner)
1698 DR_STE_SET_MPLS_TAG(mpls, misc2_mask, inner, tag);
1699 else
1700 DR_STE_SET_MPLS_TAG(mpls, misc2_mask, outer, tag);
1701
1702 return 0;
1703}
1704
1705void mlx5dr_ste_build_mpls(struct mlx5dr_ste_build *sb,
1706 struct mlx5dr_match_param *mask,
1707 bool inner, bool rx)
1708{
1709 dr_ste_build_mpls_bit_mask(mask, inner, sb->bit_mask);
1710
1711 sb->rx = rx;
1712 sb->inner = inner;
1713 sb->lu_type = DR_STE_CALC_LU_TYPE(MPLS_FIRST, rx, inner);
1714 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1715 sb->ste_build_tag_func = &dr_ste_build_mpls_tag;
1716}
1717
1718static void dr_ste_build_gre_bit_mask(struct mlx5dr_match_param *value,
1719 bool inner, u8 *bit_mask)
1720{
1721 struct mlx5dr_match_misc *misc_mask = &value->misc;
1722
1723 DR_STE_SET_MASK_V(gre, bit_mask, gre_protocol, misc_mask, gre_protocol);
1724 DR_STE_SET_MASK_V(gre, bit_mask, gre_k_present, misc_mask, gre_k_present);
1725 DR_STE_SET_MASK_V(gre, bit_mask, gre_key_h, misc_mask, gre_key_h);
1726 DR_STE_SET_MASK_V(gre, bit_mask, gre_key_l, misc_mask, gre_key_l);
1727
1728 DR_STE_SET_MASK_V(gre, bit_mask, gre_c_present, misc_mask, gre_c_present);
1729 DR_STE_SET_MASK_V(gre, bit_mask, gre_s_present, misc_mask, gre_s_present);
1730}
1731
1732static int dr_ste_build_gre_tag(struct mlx5dr_match_param *value,
1733 struct mlx5dr_ste_build *sb,
1734 u8 *hw_ste_p)
1735{
1736 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1737 struct mlx5dr_match_misc *misc = &value->misc;
1738 u8 *tag = hw_ste->tag;
1739
1740 DR_STE_SET_TAG(gre, tag, gre_protocol, misc, gre_protocol);
1741
1742 DR_STE_SET_TAG(gre, tag, gre_k_present, misc, gre_k_present);
1743 DR_STE_SET_TAG(gre, tag, gre_key_h, misc, gre_key_h);
1744 DR_STE_SET_TAG(gre, tag, gre_key_l, misc, gre_key_l);
1745
1746 DR_STE_SET_TAG(gre, tag, gre_c_present, misc, gre_c_present);
1747
1748 DR_STE_SET_TAG(gre, tag, gre_s_present, misc, gre_s_present);
1749
1750 return 0;
1751}
1752
1753void mlx5dr_ste_build_gre(struct mlx5dr_ste_build *sb,
1754 struct mlx5dr_match_param *mask, bool inner, bool rx)
1755{
1756 dr_ste_build_gre_bit_mask(mask, inner, sb->bit_mask);
1757
1758 sb->rx = rx;
1759 sb->inner = inner;
1760 sb->lu_type = MLX5DR_STE_LU_TYPE_GRE;
1761 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1762 sb->ste_build_tag_func = &dr_ste_build_gre_tag;
1763}
1764
1765static void dr_ste_build_flex_parser_0_bit_mask(struct mlx5dr_match_param *value,
1766 bool inner, u8 *bit_mask)
1767{
1768 struct mlx5dr_match_misc2 *misc_2_mask = &value->misc2;
1769
1770 if (DR_STE_IS_OUTER_MPLS_OVER_GRE_SET(misc_2_mask)) {
1771 DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_label,
1772 misc_2_mask, outer_first_mpls_over_gre_label);
1773
1774 DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_exp,
1775 misc_2_mask, outer_first_mpls_over_gre_exp);
1776
1777 DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_s_bos,
1778 misc_2_mask, outer_first_mpls_over_gre_s_bos);
1779
1780 DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_ttl,
1781 misc_2_mask, outer_first_mpls_over_gre_ttl);
1782 } else {
1783 DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_label,
1784 misc_2_mask, outer_first_mpls_over_udp_label);
1785
1786 DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_exp,
1787 misc_2_mask, outer_first_mpls_over_udp_exp);
1788
1789 DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_s_bos,
1790 misc_2_mask, outer_first_mpls_over_udp_s_bos);
1791
1792 DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_ttl,
1793 misc_2_mask, outer_first_mpls_over_udp_ttl);
1794 }
1795}
1796
1797static int dr_ste_build_flex_parser_0_tag(struct mlx5dr_match_param *value,
1798 struct mlx5dr_ste_build *sb,
1799 u8 *hw_ste_p)
1800{
1801 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1802 struct mlx5dr_match_misc2 *misc_2_mask = &value->misc2;
1803 u8 *tag = hw_ste->tag;
1804
1805 if (DR_STE_IS_OUTER_MPLS_OVER_GRE_SET(misc_2_mask)) {
1806 DR_STE_SET_TAG(flex_parser_0, tag, parser_3_label,
1807 misc_2_mask, outer_first_mpls_over_gre_label);
1808
1809 DR_STE_SET_TAG(flex_parser_0, tag, parser_3_exp,
1810 misc_2_mask, outer_first_mpls_over_gre_exp);
1811
1812 DR_STE_SET_TAG(flex_parser_0, tag, parser_3_s_bos,
1813 misc_2_mask, outer_first_mpls_over_gre_s_bos);
1814
1815 DR_STE_SET_TAG(flex_parser_0, tag, parser_3_ttl,
1816 misc_2_mask, outer_first_mpls_over_gre_ttl);
1817 } else {
1818 DR_STE_SET_TAG(flex_parser_0, tag, parser_3_label,
1819 misc_2_mask, outer_first_mpls_over_udp_label);
1820
1821 DR_STE_SET_TAG(flex_parser_0, tag, parser_3_exp,
1822 misc_2_mask, outer_first_mpls_over_udp_exp);
1823
1824 DR_STE_SET_TAG(flex_parser_0, tag, parser_3_s_bos,
1825 misc_2_mask, outer_first_mpls_over_udp_s_bos);
1826
1827 DR_STE_SET_TAG(flex_parser_0, tag, parser_3_ttl,
1828 misc_2_mask, outer_first_mpls_over_udp_ttl);
1829 }
1830 return 0;
1831}
1832
1833void mlx5dr_ste_build_flex_parser_0(struct mlx5dr_ste_build *sb,
1834 struct mlx5dr_match_param *mask,
1835 bool inner, bool rx)
1836{
1837 dr_ste_build_flex_parser_0_bit_mask(mask, inner, sb->bit_mask);
1838
1839 sb->rx = rx;
1840 sb->inner = inner;
1841 sb->lu_type = MLX5DR_STE_LU_TYPE_FLEX_PARSER_0;
1842 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1843 sb->ste_build_tag_func = &dr_ste_build_flex_parser_0_tag;
1844}
1845
1846#define ICMP_TYPE_OFFSET_FIRST_DW 24
1847#define ICMP_CODE_OFFSET_FIRST_DW 16
1848#define ICMP_HEADER_DATA_OFFSET_SECOND_DW 0
1849
1850static int dr_ste_build_flex_parser_1_bit_mask(struct mlx5dr_match_param *mask,
1851 struct mlx5dr_cmd_caps *caps,
1852 u8 *bit_mask)
1853{
1854 struct mlx5dr_match_misc3 *misc_3_mask = &mask->misc3;
1855 bool is_ipv4_mask = DR_MASK_IS_FLEX_PARSER_ICMPV4_SET(misc_3_mask);
1856 u32 icmp_header_data_mask;
1857 u32 icmp_type_mask;
1858 u32 icmp_code_mask;
1859 int dw0_location;
1860 int dw1_location;
1861
1862 if (is_ipv4_mask) {
1863 icmp_header_data_mask = misc_3_mask->icmpv4_header_data;
1864 icmp_type_mask = misc_3_mask->icmpv4_type;
1865 icmp_code_mask = misc_3_mask->icmpv4_code;
1866 dw0_location = caps->flex_parser_id_icmp_dw0;
1867 dw1_location = caps->flex_parser_id_icmp_dw1;
1868 } else {
1869 icmp_header_data_mask = misc_3_mask->icmpv6_header_data;
1870 icmp_type_mask = misc_3_mask->icmpv6_type;
1871 icmp_code_mask = misc_3_mask->icmpv6_code;
1872 dw0_location = caps->flex_parser_id_icmpv6_dw0;
1873 dw1_location = caps->flex_parser_id_icmpv6_dw1;
1874 }
1875
1876 switch (dw0_location) {
1877 case 4:
1878 if (icmp_type_mask) {
1879 MLX5_SET(ste_flex_parser_1, bit_mask, flex_parser_4,
1880 (icmp_type_mask << ICMP_TYPE_OFFSET_FIRST_DW));
1881 if (is_ipv4_mask)
1882 misc_3_mask->icmpv4_type = 0;
1883 else
1884 misc_3_mask->icmpv6_type = 0;
1885 }
1886 if (icmp_code_mask) {
1887 u32 cur_val = MLX5_GET(ste_flex_parser_1, bit_mask,
1888 flex_parser_4);
1889 MLX5_SET(ste_flex_parser_1, bit_mask, flex_parser_4,
1890 cur_val | (icmp_code_mask << ICMP_CODE_OFFSET_FIRST_DW));
1891 if (is_ipv4_mask)
1892 misc_3_mask->icmpv4_code = 0;
1893 else
1894 misc_3_mask->icmpv6_code = 0;
1895 }
1896 break;
1897 default:
1898 return -EINVAL;
1899 }
1900
1901 switch (dw1_location) {
1902 case 5:
1903 if (icmp_header_data_mask) {
1904 MLX5_SET(ste_flex_parser_1, bit_mask, flex_parser_5,
1905 (icmp_header_data_mask << ICMP_HEADER_DATA_OFFSET_SECOND_DW));
1906 if (is_ipv4_mask)
1907 misc_3_mask->icmpv4_header_data = 0;
1908 else
1909 misc_3_mask->icmpv6_header_data = 0;
1910 }
1911 break;
1912 default:
1913 return -EINVAL;
1914 }
1915
1916 return 0;
1917}
1918
1919static int dr_ste_build_flex_parser_1_tag(struct mlx5dr_match_param *value,
1920 struct mlx5dr_ste_build *sb,
1921 u8 *hw_ste_p)
1922{
1923 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1924 struct mlx5dr_match_misc3 *misc_3 = &value->misc3;
1925 u8 *tag = hw_ste->tag;
1926 u32 icmp_header_data;
1927 int dw0_location;
1928 int dw1_location;
1929 u32 icmp_type;
1930 u32 icmp_code;
1931 bool is_ipv4;
1932
1933 is_ipv4 = DR_MASK_IS_FLEX_PARSER_ICMPV4_SET(misc_3);
1934 if (is_ipv4) {
1935 icmp_header_data = misc_3->icmpv4_header_data;
1936 icmp_type = misc_3->icmpv4_type;
1937 icmp_code = misc_3->icmpv4_code;
1938 dw0_location = sb->caps->flex_parser_id_icmp_dw0;
1939 dw1_location = sb->caps->flex_parser_id_icmp_dw1;
1940 } else {
1941 icmp_header_data = misc_3->icmpv6_header_data;
1942 icmp_type = misc_3->icmpv6_type;
1943 icmp_code = misc_3->icmpv6_code;
1944 dw0_location = sb->caps->flex_parser_id_icmpv6_dw0;
1945 dw1_location = sb->caps->flex_parser_id_icmpv6_dw1;
1946 }
1947
1948 switch (dw0_location) {
1949 case 4:
1950 if (icmp_type) {
1951 MLX5_SET(ste_flex_parser_1, tag, flex_parser_4,
1952 (icmp_type << ICMP_TYPE_OFFSET_FIRST_DW));
1953 if (is_ipv4)
1954 misc_3->icmpv4_type = 0;
1955 else
1956 misc_3->icmpv6_type = 0;
1957 }
1958
1959 if (icmp_code) {
1960 u32 cur_val = MLX5_GET(ste_flex_parser_1, tag,
1961 flex_parser_4);
1962 MLX5_SET(ste_flex_parser_1, tag, flex_parser_4,
1963 cur_val | (icmp_code << ICMP_CODE_OFFSET_FIRST_DW));
1964 if (is_ipv4)
1965 misc_3->icmpv4_code = 0;
1966 else
1967 misc_3->icmpv6_code = 0;
1968 }
1969 break;
1970 default:
1971 return -EINVAL;
1972 }
1973
1974 switch (dw1_location) {
1975 case 5:
1976 if (icmp_header_data) {
1977 MLX5_SET(ste_flex_parser_1, tag, flex_parser_5,
1978 (icmp_header_data << ICMP_HEADER_DATA_OFFSET_SECOND_DW));
1979 if (is_ipv4)
1980 misc_3->icmpv4_header_data = 0;
1981 else
1982 misc_3->icmpv6_header_data = 0;
1983 }
1984 break;
1985 default:
1986 return -EINVAL;
1987 }
1988
1989 return 0;
1990}
1991
1992int mlx5dr_ste_build_flex_parser_1(struct mlx5dr_ste_build *sb,
1993 struct mlx5dr_match_param *mask,
1994 struct mlx5dr_cmd_caps *caps,
1995 bool inner, bool rx)
1996{
1997 int ret;
1998
1999 ret = dr_ste_build_flex_parser_1_bit_mask(mask, caps, sb->bit_mask);
2000 if (ret)
2001 return ret;
2002
2003 sb->rx = rx;
2004 sb->inner = inner;
2005 sb->caps = caps;
2006 sb->lu_type = MLX5DR_STE_LU_TYPE_FLEX_PARSER_1;
2007 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2008 sb->ste_build_tag_func = &dr_ste_build_flex_parser_1_tag;
2009
2010 return 0;
2011}
2012
2013static void dr_ste_build_general_purpose_bit_mask(struct mlx5dr_match_param *value,
2014 bool inner, u8 *bit_mask)
2015{
2016 struct mlx5dr_match_misc2 *misc_2_mask = &value->misc2;
2017
2018 DR_STE_SET_MASK_V(general_purpose, bit_mask,
2019 general_purpose_lookup_field, misc_2_mask,
2020 metadata_reg_a);
2021}
2022
2023static int dr_ste_build_general_purpose_tag(struct mlx5dr_match_param *value,
2024 struct mlx5dr_ste_build *sb,
2025 u8 *hw_ste_p)
2026{
2027 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
2028 struct mlx5dr_match_misc2 *misc_2_mask = &value->misc2;
2029 u8 *tag = hw_ste->tag;
2030
2031 DR_STE_SET_TAG(general_purpose, tag, general_purpose_lookup_field,
2032 misc_2_mask, metadata_reg_a);
2033
2034 return 0;
2035}
2036
2037void mlx5dr_ste_build_general_purpose(struct mlx5dr_ste_build *sb,
2038 struct mlx5dr_match_param *mask,
2039 bool inner, bool rx)
2040{
2041 dr_ste_build_general_purpose_bit_mask(mask, inner, sb->bit_mask);
2042
2043 sb->rx = rx;
2044 sb->inner = inner;
2045 sb->lu_type = MLX5DR_STE_LU_TYPE_GENERAL_PURPOSE;
2046 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2047 sb->ste_build_tag_func = &dr_ste_build_general_purpose_tag;
2048}
2049
2050static void dr_ste_build_eth_l4_misc_bit_mask(struct mlx5dr_match_param *value,
2051 bool inner, u8 *bit_mask)
2052{
2053 struct mlx5dr_match_misc3 *misc_3_mask = &value->misc3;
2054
2055 if (inner) {
2056 DR_STE_SET_MASK_V(eth_l4_misc, bit_mask, seq_num, misc_3_mask,
2057 inner_tcp_seq_num);
2058 DR_STE_SET_MASK_V(eth_l4_misc, bit_mask, ack_num, misc_3_mask,
2059 inner_tcp_ack_num);
2060 } else {
2061 DR_STE_SET_MASK_V(eth_l4_misc, bit_mask, seq_num, misc_3_mask,
2062 outer_tcp_seq_num);
2063 DR_STE_SET_MASK_V(eth_l4_misc, bit_mask, ack_num, misc_3_mask,
2064 outer_tcp_ack_num);
2065 }
2066}
2067
2068static int dr_ste_build_eth_l4_misc_tag(struct mlx5dr_match_param *value,
2069 struct mlx5dr_ste_build *sb,
2070 u8 *hw_ste_p)
2071{
2072 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
2073 struct mlx5dr_match_misc3 *misc3 = &value->misc3;
2074 u8 *tag = hw_ste->tag;
2075
2076 if (sb->inner) {
2077 DR_STE_SET_TAG(eth_l4_misc, tag, seq_num, misc3, inner_tcp_seq_num);
2078 DR_STE_SET_TAG(eth_l4_misc, tag, ack_num, misc3, inner_tcp_ack_num);
2079 } else {
2080 DR_STE_SET_TAG(eth_l4_misc, tag, seq_num, misc3, outer_tcp_seq_num);
2081 DR_STE_SET_TAG(eth_l4_misc, tag, ack_num, misc3, outer_tcp_ack_num);
2082 }
2083
2084 return 0;
2085}
2086
2087void mlx5dr_ste_build_eth_l4_misc(struct mlx5dr_ste_build *sb,
2088 struct mlx5dr_match_param *mask,
2089 bool inner, bool rx)
2090{
2091 dr_ste_build_eth_l4_misc_bit_mask(mask, inner, sb->bit_mask);
2092
2093 sb->rx = rx;
2094 sb->inner = inner;
2095 sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL4_MISC, rx, inner);
2096 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2097 sb->ste_build_tag_func = &dr_ste_build_eth_l4_misc_tag;
2098}
2099
2100static void dr_ste_build_flex_parser_tnl_bit_mask(struct mlx5dr_match_param *value,
2101 bool inner, u8 *bit_mask)
2102{
2103 struct mlx5dr_match_misc3 *misc_3_mask = &value->misc3;
2104
2105 if (misc_3_mask->outer_vxlan_gpe_flags ||
2106 misc_3_mask->outer_vxlan_gpe_next_protocol) {
2107 MLX5_SET(ste_flex_parser_tnl, bit_mask,
2108 flex_parser_tunneling_header_63_32,
2109 (misc_3_mask->outer_vxlan_gpe_flags << 24) |
2110 (misc_3_mask->outer_vxlan_gpe_next_protocol));
2111 misc_3_mask->outer_vxlan_gpe_flags = 0;
2112 misc_3_mask->outer_vxlan_gpe_next_protocol = 0;
2113 }
2114
2115 if (misc_3_mask->outer_vxlan_gpe_vni) {
2116 MLX5_SET(ste_flex_parser_tnl, bit_mask,
2117 flex_parser_tunneling_header_31_0,
2118 misc_3_mask->outer_vxlan_gpe_vni << 8);
2119 misc_3_mask->outer_vxlan_gpe_vni = 0;
2120 }
2121}
2122
2123static int dr_ste_build_flex_parser_tnl_tag(struct mlx5dr_match_param *value,
2124 struct mlx5dr_ste_build *sb,
2125 u8 *hw_ste_p)
2126{
2127 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
2128 struct mlx5dr_match_misc3 *misc3 = &value->misc3;
2129 u8 *tag = hw_ste->tag;
2130
2131 if (misc3->outer_vxlan_gpe_flags ||
2132 misc3->outer_vxlan_gpe_next_protocol) {
2133 MLX5_SET(ste_flex_parser_tnl, tag,
2134 flex_parser_tunneling_header_63_32,
2135 (misc3->outer_vxlan_gpe_flags << 24) |
2136 (misc3->outer_vxlan_gpe_next_protocol));
2137 misc3->outer_vxlan_gpe_flags = 0;
2138 misc3->outer_vxlan_gpe_next_protocol = 0;
2139 }
2140
2141 if (misc3->outer_vxlan_gpe_vni) {
2142 MLX5_SET(ste_flex_parser_tnl, tag,
2143 flex_parser_tunneling_header_31_0,
2144 misc3->outer_vxlan_gpe_vni << 8);
2145 misc3->outer_vxlan_gpe_vni = 0;
2146 }
2147
2148 return 0;
2149}
2150
2151void mlx5dr_ste_build_flex_parser_tnl(struct mlx5dr_ste_build *sb,
2152 struct mlx5dr_match_param *mask,
2153 bool inner, bool rx)
2154{
2155 dr_ste_build_flex_parser_tnl_bit_mask(mask, inner, sb->bit_mask);
2156
2157 sb->rx = rx;
2158 sb->inner = inner;
2159 sb->lu_type = MLX5DR_STE_LU_TYPE_FLEX_PARSER_TNL_HEADER;
2160 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2161 sb->ste_build_tag_func = &dr_ste_build_flex_parser_tnl_tag;
2162}
2163
2164static void dr_ste_build_register_0_bit_mask(struct mlx5dr_match_param *value,
2165 u8 *bit_mask)
2166{
2167 struct mlx5dr_match_misc2 *misc_2_mask = &value->misc2;
2168
2169 DR_STE_SET_MASK_V(register_0, bit_mask, register_0_h,
2170 misc_2_mask, metadata_reg_c_0);
2171 DR_STE_SET_MASK_V(register_0, bit_mask, register_0_l,
2172 misc_2_mask, metadata_reg_c_1);
2173 DR_STE_SET_MASK_V(register_0, bit_mask, register_1_h,
2174 misc_2_mask, metadata_reg_c_2);
2175 DR_STE_SET_MASK_V(register_0, bit_mask, register_1_l,
2176 misc_2_mask, metadata_reg_c_3);
2177}
2178
2179static int dr_ste_build_register_0_tag(struct mlx5dr_match_param *value,
2180 struct mlx5dr_ste_build *sb,
2181 u8 *hw_ste_p)
2182{
2183 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
2184 struct mlx5dr_match_misc2 *misc2 = &value->misc2;
2185 u8 *tag = hw_ste->tag;
2186
2187 DR_STE_SET_TAG(register_0, tag, register_0_h, misc2, metadata_reg_c_0);
2188 DR_STE_SET_TAG(register_0, tag, register_0_l, misc2, metadata_reg_c_1);
2189 DR_STE_SET_TAG(register_0, tag, register_1_h, misc2, metadata_reg_c_2);
2190 DR_STE_SET_TAG(register_0, tag, register_1_l, misc2, metadata_reg_c_3);
2191
2192 return 0;
2193}
2194
2195void mlx5dr_ste_build_register_0(struct mlx5dr_ste_build *sb,
2196 struct mlx5dr_match_param *mask,
2197 bool inner, bool rx)
2198{
2199 dr_ste_build_register_0_bit_mask(mask, sb->bit_mask);
2200
2201 sb->rx = rx;
2202 sb->inner = inner;
2203 sb->lu_type = MLX5DR_STE_LU_TYPE_STEERING_REGISTERS_0;
2204 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2205 sb->ste_build_tag_func = &dr_ste_build_register_0_tag;
2206}
2207
2208static void dr_ste_build_register_1_bit_mask(struct mlx5dr_match_param *value,
2209 u8 *bit_mask)
2210{
2211 struct mlx5dr_match_misc2 *misc_2_mask = &value->misc2;
2212
2213 DR_STE_SET_MASK_V(register_1, bit_mask, register_2_h,
2214 misc_2_mask, metadata_reg_c_4);
2215 DR_STE_SET_MASK_V(register_1, bit_mask, register_2_l,
2216 misc_2_mask, metadata_reg_c_5);
2217 DR_STE_SET_MASK_V(register_1, bit_mask, register_3_h,
2218 misc_2_mask, metadata_reg_c_6);
2219 DR_STE_SET_MASK_V(register_1, bit_mask, register_3_l,
2220 misc_2_mask, metadata_reg_c_7);
2221}
2222
2223static int dr_ste_build_register_1_tag(struct mlx5dr_match_param *value,
2224 struct mlx5dr_ste_build *sb,
2225 u8 *hw_ste_p)
2226{
2227 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
2228 struct mlx5dr_match_misc2 *misc2 = &value->misc2;
2229 u8 *tag = hw_ste->tag;
2230
2231 DR_STE_SET_TAG(register_1, tag, register_2_h, misc2, metadata_reg_c_4);
2232 DR_STE_SET_TAG(register_1, tag, register_2_l, misc2, metadata_reg_c_5);
2233 DR_STE_SET_TAG(register_1, tag, register_3_h, misc2, metadata_reg_c_6);
2234 DR_STE_SET_TAG(register_1, tag, register_3_l, misc2, metadata_reg_c_7);
2235
2236 return 0;
2237}
2238
2239void mlx5dr_ste_build_register_1(struct mlx5dr_ste_build *sb,
2240 struct mlx5dr_match_param *mask,
2241 bool inner, bool rx)
2242{
2243 dr_ste_build_register_1_bit_mask(mask, sb->bit_mask);
2244
2245 sb->rx = rx;
2246 sb->inner = inner;
2247 sb->lu_type = MLX5DR_STE_LU_TYPE_STEERING_REGISTERS_1;
2248 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2249 sb->ste_build_tag_func = &dr_ste_build_register_1_tag;
2250}
2251
2252static int dr_ste_build_src_gvmi_qpn_bit_mask(struct mlx5dr_match_param *value,
2253 u8 *bit_mask)
2254{
2255 struct mlx5dr_match_misc *misc_mask = &value->misc;
2256
2257 if (misc_mask->source_port != 0xffff)
2258 return -EINVAL;
2259
2260 DR_STE_SET_MASK(src_gvmi_qp, bit_mask, source_gvmi, misc_mask, source_port);
2261 DR_STE_SET_MASK(src_gvmi_qp, bit_mask, source_qp, misc_mask, source_sqn);
2262
2263 return 0;
2264}
2265
2266static int dr_ste_build_src_gvmi_qpn_tag(struct mlx5dr_match_param *value,
2267 struct mlx5dr_ste_build *sb,
2268 u8 *hw_ste_p)
2269{
2270 struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
2271 struct mlx5dr_match_misc *misc = &value->misc;
2272 struct mlx5dr_cmd_vport_cap *vport_cap;
2273 u8 *tag = hw_ste->tag;
2274
2275 DR_STE_SET_TAG(src_gvmi_qp, tag, source_qp, misc, source_sqn);
2276
2277 vport_cap = mlx5dr_get_vport_cap(sb->caps, misc->source_port);
2278 if (!vport_cap)
2279 return -EINVAL;
2280
2281 if (vport_cap->vport_gvmi)
2282 MLX5_SET(ste_src_gvmi_qp, tag, source_gvmi, vport_cap->vport_gvmi);
2283
2284 misc->source_port = 0;
2285
2286 return 0;
2287}
2288
2289int mlx5dr_ste_build_src_gvmi_qpn(struct mlx5dr_ste_build *sb,
2290 struct mlx5dr_match_param *mask,
2291 struct mlx5dr_cmd_caps *caps,
2292 bool inner, bool rx)
2293{
2294 int ret;
2295
2296 ret = dr_ste_build_src_gvmi_qpn_bit_mask(mask, sb->bit_mask);
2297 if (ret)
2298 return ret;
2299
2300 sb->rx = rx;
2301 sb->caps = caps;
2302 sb->inner = inner;
2303 sb->lu_type = MLX5DR_STE_LU_TYPE_SRC_GVMI_AND_QP;
2304 sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2305 sb->ste_build_tag_func = &dr_ste_build_src_gvmi_qpn_tag;
2306
2307 return 0;
2308}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_table.c+294−0
@@ -0,0 +1,294 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2/* Copyright (c) 2019 Mellanox Technologies. */
3
4#include "dr_types.h"
5
6int mlx5dr_table_set_miss_action(struct mlx5dr_table *tbl,
7 struct mlx5dr_action *action)
8{
9 struct mlx5dr_matcher *last_matcher = NULL;
10 struct mlx5dr_htbl_connect_info info;
11 struct mlx5dr_ste_htbl *last_htbl;
12 int ret;
13
14 if (action && action->action_type != DR_ACTION_TYP_FT)
15 return -EOPNOTSUPP;
16
17 mutex_lock(&tbl->dmn->mutex);
18
19 if (!list_empty(&tbl->matcher_list))
20 last_matcher = list_last_entry(&tbl->matcher_list,
21 struct mlx5dr_matcher,
22 matcher_list);
23
24 if (tbl->dmn->type == MLX5DR_DOMAIN_TYPE_NIC_RX ||
25 tbl->dmn->type == MLX5DR_DOMAIN_TYPE_FDB) {
26 if (last_matcher)
27 last_htbl = last_matcher->rx.e_anchor;
28 else
29 last_htbl = tbl->rx.s_anchor;
30
31 tbl->rx.default_icm_addr = action ?
32 action->dest_tbl.tbl->rx.s_anchor->chunk->icm_addr :
33 tbl->rx.nic_dmn->default_icm_addr;
34
35 info.type = CONNECT_MISS;
36 info.miss_icm_addr = tbl->rx.default_icm_addr;
37
38 ret = mlx5dr_ste_htbl_init_and_postsend(tbl->dmn,
39 tbl->rx.nic_dmn,
40 last_htbl,
41 &info, true);
42 if (ret) {
43 mlx5dr_dbg(tbl->dmn, "Failed to set RX miss action, ret %d\n", ret);
44 goto out;
45 }
46 }
47
48 if (tbl->dmn->type == MLX5DR_DOMAIN_TYPE_NIC_TX ||
49 tbl->dmn->type == MLX5DR_DOMAIN_TYPE_FDB) {
50 if (last_matcher)
51 last_htbl = last_matcher->tx.e_anchor;
52 else
53 last_htbl = tbl->tx.s_anchor;
54
55 tbl->tx.default_icm_addr = action ?
56 action->dest_tbl.tbl->tx.s_anchor->chunk->icm_addr :
57 tbl->tx.nic_dmn->default_icm_addr;
58
59 info.type = CONNECT_MISS;
60 info.miss_icm_addr = tbl->tx.default_icm_addr;
61
62 ret = mlx5dr_ste_htbl_init_and_postsend(tbl->dmn,
63 tbl->tx.nic_dmn,
64 last_htbl, &info, true);
65 if (ret) {
66 mlx5dr_dbg(tbl->dmn, "Failed to set TX miss action, ret %d\n", ret);
67 goto out;
68 }
69 }
70
71 /* Release old action */
72 if (tbl->miss_action)
73 refcount_dec(&tbl->miss_action->refcount);
74
75 /* Set new miss action */
76 tbl->miss_action = action;
77 if (tbl->miss_action)
78 refcount_inc(&action->refcount);
79
80out:
81 mutex_unlock(&tbl->dmn->mutex);
82 return ret;
83}
84
85static void dr_table_uninit_nic(struct mlx5dr_table_rx_tx *nic_tbl)
86{
87 mlx5dr_htbl_put(nic_tbl->s_anchor);
88}
89
90static void dr_table_uninit_fdb(struct mlx5dr_table *tbl)
91{
92 dr_table_uninit_nic(&tbl->rx);
93 dr_table_uninit_nic(&tbl->tx);
94}
95
96static void dr_table_uninit(struct mlx5dr_table *tbl)
97{
98 mutex_lock(&tbl->dmn->mutex);
99
100 switch (tbl->dmn->type) {
101 case MLX5DR_DOMAIN_TYPE_NIC_RX:
102 dr_table_uninit_nic(&tbl->rx);
103 break;
104 case MLX5DR_DOMAIN_TYPE_NIC_TX:
105 dr_table_uninit_nic(&tbl->tx);
106 break;
107 case MLX5DR_DOMAIN_TYPE_FDB:
108 dr_table_uninit_fdb(tbl);
109 break;
110 default:
111 WARN_ON(true);
112 break;
113 }
114
115 mutex_unlock(&tbl->dmn->mutex);
116}
117
118static int dr_table_init_nic(struct mlx5dr_domain *dmn,
119 struct mlx5dr_table_rx_tx *nic_tbl)
120{
121 struct mlx5dr_domain_rx_tx *nic_dmn = nic_tbl->nic_dmn;
122 struct mlx5dr_htbl_connect_info info;
123 int ret;
124
125 nic_tbl->default_icm_addr = nic_dmn->default_icm_addr;
126
127 nic_tbl->s_anchor = mlx5dr_ste_htbl_alloc(dmn->ste_icm_pool,
128 DR_CHUNK_SIZE_1,
129 MLX5DR_STE_LU_TYPE_DONT_CARE,
130 0);
131 if (!nic_tbl->s_anchor)
132 return -ENOMEM;
133
134 info.type = CONNECT_MISS;
135 info.miss_icm_addr = nic_dmn->default_icm_addr;
136 ret = mlx5dr_ste_htbl_init_and_postsend(dmn, nic_dmn,
137 nic_tbl->s_anchor,
138 &info, true);
139 if (ret)
140 goto free_s_anchor;
141
142 mlx5dr_htbl_get(nic_tbl->s_anchor);
143
144 return 0;
145
146free_s_anchor:
147 mlx5dr_ste_htbl_free(nic_tbl->s_anchor);
148 return ret;
149}
150
151static int dr_table_init_fdb(struct mlx5dr_table *tbl)
152{
153 int ret;
154
155 ret = dr_table_init_nic(tbl->dmn, &tbl->rx);
156 if (ret)
157 return ret;
158
159 ret = dr_table_init_nic(tbl->dmn, &tbl->tx);
160 if (ret)
161 goto destroy_rx;
162
163 return 0;
164
165destroy_rx:
166 dr_table_uninit_nic(&tbl->rx);
167 return ret;
168}
169
170static int dr_table_init(struct mlx5dr_table *tbl)
171{
172 int ret = 0;
173
174 INIT_LIST_HEAD(&tbl->matcher_list);
175
176 mutex_lock(&tbl->dmn->mutex);
177
178 switch (tbl->dmn->type) {
179 case MLX5DR_DOMAIN_TYPE_NIC_RX:
180 tbl->table_type = MLX5_FLOW_TABLE_TYPE_NIC_RX;
181 tbl->rx.nic_dmn = &tbl->dmn->info.rx;
182 ret = dr_table_init_nic(tbl->dmn, &tbl->rx);
183 break;
184 case MLX5DR_DOMAIN_TYPE_NIC_TX:
185 tbl->table_type = MLX5_FLOW_TABLE_TYPE_NIC_TX;
186 tbl->tx.nic_dmn = &tbl->dmn->info.tx;
187 ret = dr_table_init_nic(tbl->dmn, &tbl->tx);
188 break;
189 case MLX5DR_DOMAIN_TYPE_FDB:
190 tbl->table_type = MLX5_FLOW_TABLE_TYPE_FDB;
191 tbl->rx.nic_dmn = &tbl->dmn->info.rx;
192 tbl->tx.nic_dmn = &tbl->dmn->info.tx;
193 ret = dr_table_init_fdb(tbl);
194 break;
195 default:
196 WARN_ON(true);
197 break;
198 }
199
200 mutex_unlock(&tbl->dmn->mutex);
201
202 return ret;
203}
204
205static int dr_table_destroy_sw_owned_tbl(struct mlx5dr_table *tbl)
206{
207 return mlx5dr_cmd_destroy_flow_table(tbl->dmn->mdev,
208 tbl->table_id,
209 tbl->table_type);
210}
211
212static int dr_table_create_sw_owned_tbl(struct mlx5dr_table *tbl)
213{
214 u64 icm_addr_rx = 0;
215 u64 icm_addr_tx = 0;
216 int ret;
217
218 if (tbl->rx.s_anchor)
219 icm_addr_rx = tbl->rx.s_anchor->chunk->icm_addr;
220
221 if (tbl->tx.s_anchor)
222 icm_addr_tx = tbl->tx.s_anchor->chunk->icm_addr;
223
224 ret = mlx5dr_cmd_create_flow_table(tbl->dmn->mdev,
225 tbl->table_type,
226 icm_addr_rx,
227 icm_addr_tx,
228 tbl->dmn->info.caps.max_ft_level - 1,
229 true, false, NULL,
230 &tbl->table_id);
231
232 return ret;
233}
234
235struct mlx5dr_table *mlx5dr_table_create(struct mlx5dr_domain *dmn, u32 level)
236{
237 struct mlx5dr_table *tbl;
238 int ret;
239
240 refcount_inc(&dmn->refcount);
241
242 tbl = kzalloc(sizeof(*tbl), GFP_KERNEL);
243 if (!tbl)
244 goto dec_ref;
245
246 tbl->dmn = dmn;
247 tbl->level = level;
248 refcount_set(&tbl->refcount, 1);
249
250 ret = dr_table_init(tbl);
251 if (ret)
252 goto free_tbl;
253
254 ret = dr_table_create_sw_owned_tbl(tbl);
255 if (ret)
256 goto uninit_tbl;
257
258 return tbl;
259
260uninit_tbl:
261 dr_table_uninit(tbl);
262free_tbl:
263 kfree(tbl);
264dec_ref:
265 refcount_dec(&dmn->refcount);
266 return NULL;
267}
268
269int mlx5dr_table_destroy(struct mlx5dr_table *tbl)
270{
271 int ret;
272
273 if (refcount_read(&tbl->refcount) > 1)
274 return -EBUSY;
275
276 ret = dr_table_destroy_sw_owned_tbl(tbl);
277 if (ret)
278 return ret;
279
280 dr_table_uninit(tbl);
281
282 if (tbl->miss_action)
283 refcount_dec(&tbl->miss_action->refcount);
284
285 refcount_dec(&tbl->dmn->refcount);
286 kfree(tbl);
287
288 return ret;
289}
290
291u32 mlx5dr_table_get_id(struct mlx5dr_table *tbl)
292{
293 return tbl->table_id;
294}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_types.h+1060−0
@@ -0,0 +1,1060 @@
1/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
2/* Copyright (c) 2019, Mellanox Technologies */
3
4#ifndef _DR_TYPES_
5#define _DR_TYPES_
6
7#include <linux/mlx5/driver.h>
8#include <linux/refcount.h>
9#include "fs_core.h"
10#include "wq.h"
11#include "lib/mlx5.h"
12#include "mlx5_ifc_dr.h"
13#include "mlx5dr.h"
14
15#define DR_RULE_MAX_STES 17
16#define DR_ACTION_MAX_STES 5
17#define WIRE_PORT 0xFFFF
18#define DR_STE_SVLAN 0x1
19#define DR_STE_CVLAN 0x2
20
21#define mlx5dr_err(dmn, arg...) mlx5_core_err((dmn)->mdev, ##arg)
22#define mlx5dr_info(dmn, arg...) mlx5_core_info((dmn)->mdev, ##arg)
23#define mlx5dr_dbg(dmn, arg...) mlx5_core_dbg((dmn)->mdev, ##arg)
24
25enum mlx5dr_icm_chunk_size {
26 DR_CHUNK_SIZE_1,
27 DR_CHUNK_SIZE_MIN = DR_CHUNK_SIZE_1, /* keep updated when changing */
28 DR_CHUNK_SIZE_2,
29 DR_CHUNK_SIZE_4,
30 DR_CHUNK_SIZE_8,
31 DR_CHUNK_SIZE_16,
32 DR_CHUNK_SIZE_32,
33 DR_CHUNK_SIZE_64,
34 DR_CHUNK_SIZE_128,
35 DR_CHUNK_SIZE_256,
36 DR_CHUNK_SIZE_512,
37 DR_CHUNK_SIZE_1K,
38 DR_CHUNK_SIZE_2K,
39 DR_CHUNK_SIZE_4K,
40 DR_CHUNK_SIZE_8K,
41 DR_CHUNK_SIZE_16K,
42 DR_CHUNK_SIZE_32K,
43 DR_CHUNK_SIZE_64K,
44 DR_CHUNK_SIZE_128K,
45 DR_CHUNK_SIZE_256K,
46 DR_CHUNK_SIZE_512K,
47 DR_CHUNK_SIZE_1024K,
48 DR_CHUNK_SIZE_2048K,
49 DR_CHUNK_SIZE_MAX,
50};
51
52enum mlx5dr_icm_type {
53 DR_ICM_TYPE_STE,
54 DR_ICM_TYPE_MODIFY_ACTION,
55};
56
57static inline enum mlx5dr_icm_chunk_size
58mlx5dr_icm_next_higher_chunk(enum mlx5dr_icm_chunk_size chunk)
59{
60 chunk += 2;
61 if (chunk < DR_CHUNK_SIZE_MAX)
62 return chunk;
63
64 return DR_CHUNK_SIZE_MAX;
65}
66
67enum {
68 DR_STE_SIZE = 64,
69 DR_STE_SIZE_CTRL = 32,
70 DR_STE_SIZE_TAG = 16,
71 DR_STE_SIZE_MASK = 16,
72};
73
74enum {
75 DR_STE_SIZE_REDUCED = DR_STE_SIZE - DR_STE_SIZE_MASK,
76};
77
78enum {
79 DR_MODIFY_ACTION_SIZE = 8,
80};
81
82enum mlx5dr_matcher_criteria {
83 DR_MATCHER_CRITERIA_EMPTY = 0,
84 DR_MATCHER_CRITERIA_OUTER = 1 << 0,
85 DR_MATCHER_CRITERIA_MISC = 1 << 1,
86 DR_MATCHER_CRITERIA_INNER = 1 << 2,
87 DR_MATCHER_CRITERIA_MISC2 = 1 << 3,
88 DR_MATCHER_CRITERIA_MISC3 = 1 << 4,
89 DR_MATCHER_CRITERIA_MAX = 1 << 5,
90};
91
92enum mlx5dr_action_type {
93 DR_ACTION_TYP_TNL_L2_TO_L2,
94 DR_ACTION_TYP_L2_TO_TNL_L2,
95 DR_ACTION_TYP_TNL_L3_TO_L2,
96 DR_ACTION_TYP_L2_TO_TNL_L3,
97 DR_ACTION_TYP_DROP,
98 DR_ACTION_TYP_QP,
99 DR_ACTION_TYP_FT,
100 DR_ACTION_TYP_CTR,
101 DR_ACTION_TYP_TAG,
102 DR_ACTION_TYP_MODIFY_HDR,
103 DR_ACTION_TYP_VPORT,
104 DR_ACTION_TYP_POP_VLAN,
105 DR_ACTION_TYP_PUSH_VLAN,
106 DR_ACTION_TYP_MAX,
107};
108
109struct mlx5dr_icm_pool;
110struct mlx5dr_icm_chunk;
111struct mlx5dr_icm_bucket;
112struct mlx5dr_ste_htbl;
113struct mlx5dr_match_param;
114struct mlx5dr_cmd_caps;
115struct mlx5dr_matcher_rx_tx;
116
117struct mlx5dr_ste {
118 u8 *hw_ste;
119 /* refcount: indicates the num of rules that using this ste */
120 refcount_t refcount;
121
122 /* attached to the miss_list head at each htbl entry */
123 struct list_head miss_list_node;
124
125 /* each rule member that uses this ste attached here */
126 struct list_head rule_list;
127
128 /* this ste is member of htbl */
129 struct mlx5dr_ste_htbl *htbl;
130
131 struct mlx5dr_ste_htbl *next_htbl;
132
133 /* this ste is part of a rule, located in ste's chain */
134 u8 ste_chain_location;
135};
136
137struct mlx5dr_ste_htbl_ctrl {
138 /* total number of valid entries belonging to this hash table. This
139 * includes the non collision and collision entries
140 */
141 unsigned int num_of_valid_entries;
142
143 /* total number of collisions entries attached to this table */
144 unsigned int num_of_collisions;
145 unsigned int increase_threshold;
146 u8 may_grow:1;
147};
148
149struct mlx5dr_ste_htbl {
150 u8 lu_type;
151 u16 byte_mask;
152 refcount_t refcount;
153 struct mlx5dr_icm_chunk *chunk;
154 struct mlx5dr_ste *ste_arr;
155 u8 *hw_ste_arr;
156
157 struct list_head *miss_list;
158
159 enum mlx5dr_icm_chunk_size chunk_size;
160 struct mlx5dr_ste *pointing_ste;
161
162 struct mlx5dr_ste_htbl_ctrl ctrl;
163};
164
165struct mlx5dr_ste_send_info {
166 struct mlx5dr_ste *ste;
167 struct list_head send_list;
168 u16 size;
169 u16 offset;
170 u8 data_cont[DR_STE_SIZE];
171 u8 *data;
172};
173
174void mlx5dr_send_fill_and_append_ste_send_info(struct mlx5dr_ste *ste, u16 size,
175 u16 offset, u8 *data,
176 struct mlx5dr_ste_send_info *ste_info,
177 struct list_head *send_list,
178 bool copy_data);
179
180struct mlx5dr_ste_build {
181 u8 inner:1;
182 u8 rx:1;
183 struct mlx5dr_cmd_caps *caps;
184 u8 lu_type;
185 u16 byte_mask;
186 u8 bit_mask[DR_STE_SIZE_MASK];
187 int (*ste_build_tag_func)(struct mlx5dr_match_param *spec,
188 struct mlx5dr_ste_build *sb,
189 u8 *hw_ste_p);
190};
191
192struct mlx5dr_ste_htbl *
193mlx5dr_ste_htbl_alloc(struct mlx5dr_icm_pool *pool,
194 enum mlx5dr_icm_chunk_size chunk_size,
195 u8 lu_type, u16 byte_mask);
196
197int mlx5dr_ste_htbl_free(struct mlx5dr_ste_htbl *htbl);
198
199static inline void mlx5dr_htbl_put(struct mlx5dr_ste_htbl *htbl)
200{
201 if (refcount_dec_and_test(&htbl->refcount))
202 mlx5dr_ste_htbl_free(htbl);
203}
204
205static inline void mlx5dr_htbl_get(struct mlx5dr_ste_htbl *htbl)
206{
207 refcount_inc(&htbl->refcount);
208}
209
210/* STE utils */
211u32 mlx5dr_ste_calc_hash_index(u8 *hw_ste_p, struct mlx5dr_ste_htbl *htbl);
212void mlx5dr_ste_init(u8 *hw_ste_p, u8 lu_type, u8 entry_type, u16 gvmi);
213void mlx5dr_ste_always_hit_htbl(struct mlx5dr_ste *ste,
214 struct mlx5dr_ste_htbl *next_htbl);
215void mlx5dr_ste_set_miss_addr(u8 *hw_ste, u64 miss_addr);
216u64 mlx5dr_ste_get_miss_addr(u8 *hw_ste);
217void mlx5dr_ste_set_hit_gvmi(u8 *hw_ste_p, u16 gvmi);
218void mlx5dr_ste_set_hit_addr(u8 *hw_ste, u64 icm_addr, u32 ht_size);
219void mlx5dr_ste_always_miss_addr(struct mlx5dr_ste *ste, u64 miss_addr);
220void mlx5dr_ste_set_bit_mask(u8 *hw_ste_p, u8 *bit_mask);
221bool mlx5dr_ste_not_used_ste(struct mlx5dr_ste *ste);
222bool mlx5dr_ste_is_last_in_rule(struct mlx5dr_matcher_rx_tx *nic_matcher,
223 u8 ste_location);
224void mlx5dr_ste_rx_set_flow_tag(u8 *hw_ste_p, u32 flow_tag);
225void mlx5dr_ste_set_counter_id(u8 *hw_ste_p, u32 ctr_id);
226void mlx5dr_ste_set_tx_encap(void *hw_ste_p, u32 reformat_id,
227 int size, bool encap_l3);
228void mlx5dr_ste_set_rx_decap(u8 *hw_ste_p);
229void mlx5dr_ste_set_rx_decap_l3(u8 *hw_ste_p, bool vlan);
230void mlx5dr_ste_set_rx_pop_vlan(u8 *hw_ste_p);
231void mlx5dr_ste_set_tx_push_vlan(u8 *hw_ste_p, u32 vlan_tpid_pcp_dei_vid,
232 bool go_back);
233void mlx5dr_ste_set_entry_type(u8 *hw_ste_p, u8 entry_type);
234u8 mlx5dr_ste_get_entry_type(u8 *hw_ste_p);
235void mlx5dr_ste_set_rewrite_actions(u8 *hw_ste_p, u16 num_of_actions,
236 u32 re_write_index);
237void mlx5dr_ste_set_go_back_bit(u8 *hw_ste_p);
238u64 mlx5dr_ste_get_icm_addr(struct mlx5dr_ste *ste);
239u64 mlx5dr_ste_get_mr_addr(struct mlx5dr_ste *ste);
240struct list_head *mlx5dr_ste_get_miss_list(struct mlx5dr_ste *ste);
241
242void mlx5dr_ste_free(struct mlx5dr_ste *ste,
243 struct mlx5dr_matcher *matcher,
244 struct mlx5dr_matcher_rx_tx *nic_matcher);
245static inline void mlx5dr_ste_put(struct mlx5dr_ste *ste,
246 struct mlx5dr_matcher *matcher,
247 struct mlx5dr_matcher_rx_tx *nic_matcher)
248{
249 if (refcount_dec_and_test(&ste->refcount))
250 mlx5dr_ste_free(ste, matcher, nic_matcher);
251}
252
253/* initial as 0, increased only when ste appears in a new rule */
254static inline void mlx5dr_ste_get(struct mlx5dr_ste *ste)
255{
256 refcount_inc(&ste->refcount);
257}
258
259void mlx5dr_ste_set_hit_addr_by_next_htbl(u8 *hw_ste,
260 struct mlx5dr_ste_htbl *next_htbl);
261bool mlx5dr_ste_equal_tag(void *src, void *dst);
262int mlx5dr_ste_create_next_htbl(struct mlx5dr_matcher *matcher,
263 struct mlx5dr_matcher_rx_tx *nic_matcher,
264 struct mlx5dr_ste *ste,
265 u8 *cur_hw_ste,
266 enum mlx5dr_icm_chunk_size log_table_size);
267
268/* STE build functions */
269int mlx5dr_ste_build_pre_check(struct mlx5dr_domain *dmn,
270 u8 match_criteria,
271 struct mlx5dr_match_param *mask,
272 struct mlx5dr_match_param *value);
273int mlx5dr_ste_build_ste_arr(struct mlx5dr_matcher *matcher,
274 struct mlx5dr_matcher_rx_tx *nic_matcher,
275 struct mlx5dr_match_param *value,
276 u8 *ste_arr);
277int mlx5dr_ste_build_eth_l2_src_des(struct mlx5dr_ste_build *builder,
278 struct mlx5dr_match_param *mask,
279 bool inner, bool rx);
280void mlx5dr_ste_build_eth_l3_ipv4_5_tuple(struct mlx5dr_ste_build *sb,
281 struct mlx5dr_match_param *mask,
282 bool inner, bool rx);
283void mlx5dr_ste_build_eth_l3_ipv4_misc(struct mlx5dr_ste_build *sb,
284 struct mlx5dr_match_param *mask,
285 bool inner, bool rx);
286void mlx5dr_ste_build_eth_l3_ipv6_dst(struct mlx5dr_ste_build *sb,
287 struct mlx5dr_match_param *mask,
288 bool inner, bool rx);
289void mlx5dr_ste_build_eth_l3_ipv6_src(struct mlx5dr_ste_build *sb,
290 struct mlx5dr_match_param *mask,
291 bool inner, bool rx);
292void mlx5dr_ste_build_eth_l2_src(struct mlx5dr_ste_build *sb,
293 struct mlx5dr_match_param *mask,
294 bool inner, bool rx);
295void mlx5dr_ste_build_eth_l2_dst(struct mlx5dr_ste_build *sb,
296 struct mlx5dr_match_param *mask,
297 bool inner, bool rx);
298void mlx5dr_ste_build_eth_l2_tnl(struct mlx5dr_ste_build *sb,
299 struct mlx5dr_match_param *mask,
300 bool inner, bool rx);
301void mlx5dr_ste_build_ipv6_l3_l4(struct mlx5dr_ste_build *sb,
302 struct mlx5dr_match_param *mask,
303 bool inner, bool rx);
304void mlx5dr_ste_build_eth_l4_misc(struct mlx5dr_ste_build *sb,
305 struct mlx5dr_match_param *mask,
306 bool inner, bool rx);
307void mlx5dr_ste_build_gre(struct mlx5dr_ste_build *sb,
308 struct mlx5dr_match_param *mask,
309 bool inner, bool rx);
310void mlx5dr_ste_build_mpls(struct mlx5dr_ste_build *sb,
311 struct mlx5dr_match_param *mask,
312 bool inner, bool rx);
313void mlx5dr_ste_build_flex_parser_0(struct mlx5dr_ste_build *sb,
314 struct mlx5dr_match_param *mask,
315 bool inner, bool rx);
316int mlx5dr_ste_build_flex_parser_1(struct mlx5dr_ste_build *sb,
317 struct mlx5dr_match_param *mask,
318 struct mlx5dr_cmd_caps *caps,
319 bool inner, bool rx);
320void mlx5dr_ste_build_flex_parser_tnl(struct mlx5dr_ste_build *sb,
321 struct mlx5dr_match_param *mask,
322 bool inner, bool rx);
323void mlx5dr_ste_build_general_purpose(struct mlx5dr_ste_build *sb,
324 struct mlx5dr_match_param *mask,
325 bool inner, bool rx);
326void mlx5dr_ste_build_register_0(struct mlx5dr_ste_build *sb,
327 struct mlx5dr_match_param *mask,
328 bool inner, bool rx);
329void mlx5dr_ste_build_register_1(struct mlx5dr_ste_build *sb,
330 struct mlx5dr_match_param *mask,
331 bool inner, bool rx);
332int mlx5dr_ste_build_src_gvmi_qpn(struct mlx5dr_ste_build *sb,
333 struct mlx5dr_match_param *mask,
334 struct mlx5dr_cmd_caps *caps,
335 bool inner, bool rx);
336void mlx5dr_ste_build_empty_always_hit(struct mlx5dr_ste_build *sb, bool rx);
337
338/* Actions utils */
339int mlx5dr_actions_build_ste_arr(struct mlx5dr_matcher *matcher,
340 struct mlx5dr_matcher_rx_tx *nic_matcher,
341 struct mlx5dr_action *actions[],
342 u32 num_actions,
343 u8 *ste_arr,
344 u32 *new_hw_ste_arr_sz);
345
346struct mlx5dr_match_spec {
347 u32 smac_47_16; /* Source MAC address of incoming packet */
348 /* Incoming packet Ethertype - this is the Ethertype
349 * following the last VLAN tag of the packet
350 */
351 u32 ethertype:16;
352 u32 smac_15_0:16; /* Source MAC address of incoming packet */
353 u32 dmac_47_16; /* Destination MAC address of incoming packet */
354 /* VLAN ID of first VLAN tag in the incoming packet.
355 * Valid only when cvlan_tag==1 or svlan_tag==1
356 */
357 u32 first_vid:12;
358 /* CFI bit of first VLAN tag in the incoming packet.
359 * Valid only when cvlan_tag==1 or svlan_tag==1
360 */
361 u32 first_cfi:1;
362 /* Priority of first VLAN tag in the incoming packet.
363 * Valid only when cvlan_tag==1 or svlan_tag==1
364 */
365 u32 first_prio:3;
366 u32 dmac_15_0:16; /* Destination MAC address of incoming packet */
367 /* TCP flags. ;Bit 0: FIN;Bit 1: SYN;Bit 2: RST;Bit 3: PSH;Bit 4: ACK;
368 * Bit 5: URG;Bit 6: ECE;Bit 7: CWR;Bit 8: NS
369 */
370 u32 tcp_flags:9;
371 u32 ip_version:4; /* IP version */
372 u32 frag:1; /* Packet is an IP fragment */
373 /* The first vlan in the packet is s-vlan (0x8a88).
374 * cvlan_tag and svlan_tag cannot be set together
375 */
376 u32 svlan_tag:1;
377 /* The first vlan in the packet is c-vlan (0x8100).
378 * cvlan_tag and svlan_tag cannot be set together
379 */
380 u32 cvlan_tag:1;
381 /* Explicit Congestion Notification derived from
382 * Traffic Class/TOS field of IPv6/v4
383 */
384 u32 ip_ecn:2;
385 /* Differentiated Services Code Point derived from
386 * Traffic Class/TOS field of IPv6/v4
387 */
388 u32 ip_dscp:6;
389 u32 ip_protocol:8; /* IP protocol */
390 /* TCP destination port.
391 * tcp and udp sport/dport are mutually exclusive
392 */
393 u32 tcp_dport:16;
394 /* TCP source port.;tcp and udp sport/dport are mutually exclusive */
395 u32 tcp_sport:16;
396 u32 ttl_hoplimit:8;
397 u32 reserved:24;
398 /* UDP destination port.;tcp and udp sport/dport are mutually exclusive */
399 u32 udp_dport:16;
400 /* UDP source port.;tcp and udp sport/dport are mutually exclusive */
401 u32 udp_sport:16;
402 /* IPv6 source address of incoming packets
403 * For IPv4 address use bits 31:0 (rest of the bits are reserved)
404 * This field should be qualified by an appropriate ethertype
405 */
406 u32 src_ip_127_96;
407 /* IPv6 source address of incoming packets
408 * For IPv4 address use bits 31:0 (rest of the bits are reserved)
409 * This field should be qualified by an appropriate ethertype
410 */
411 u32 src_ip_95_64;
412 /* IPv6 source address of incoming packets
413 * For IPv4 address use bits 31:0 (rest of the bits are reserved)
414 * This field should be qualified by an appropriate ethertype
415 */
416 u32 src_ip_63_32;
417 /* IPv6 source address of incoming packets
418 * For IPv4 address use bits 31:0 (rest of the bits are reserved)
419 * This field should be qualified by an appropriate ethertype
420 */
421 u32 src_ip_31_0;
422 /* IPv6 destination address of incoming packets
423 * For IPv4 address use bits 31:0 (rest of the bits are reserved)
424 * This field should be qualified by an appropriate ethertype
425 */
426 u32 dst_ip_127_96;
427 /* IPv6 destination address of incoming packets
428 * For IPv4 address use bits 31:0 (rest of the bits are reserved)
429 * This field should be qualified by an appropriate ethertype
430 */
431 u32 dst_ip_95_64;
432 /* IPv6 destination address of incoming packets
433 * For IPv4 address use bits 31:0 (rest of the bits are reserved)
434 * This field should be qualified by an appropriate ethertype
435 */
436 u32 dst_ip_63_32;
437 /* IPv6 destination address of incoming packets
438 * For IPv4 address use bits 31:0 (rest of the bits are reserved)
439 * This field should be qualified by an appropriate ethertype
440 */
441 u32 dst_ip_31_0;
442};
443
444struct mlx5dr_match_misc {
445 u32 source_sqn:24; /* Source SQN */
446 u32 source_vhca_port:4;
447 /* used with GRE, sequence number exist when gre_s_present == 1 */
448 u32 gre_s_present:1;
449 /* used with GRE, key exist when gre_k_present == 1 */
450 u32 gre_k_present:1;
451 u32 reserved_auto1:1;
452 /* used with GRE, checksum exist when gre_c_present == 1 */
453 u32 gre_c_present:1;
454 /* Source port.;0xffff determines wire port */
455 u32 source_port:16;
456 u32 reserved_auto2:16;
457 /* VLAN ID of first VLAN tag the inner header of the incoming packet.
458 * Valid only when inner_second_cvlan_tag ==1 or inner_second_svlan_tag ==1
459 */
460 u32 inner_second_vid:12;
461 /* CFI bit of first VLAN tag in the inner header of the incoming packet.
462 * Valid only when inner_second_cvlan_tag ==1 or inner_second_svlan_tag ==1
463 */
464 u32 inner_second_cfi:1;
465 /* Priority of second VLAN tag in the inner header of the incoming packet.
466 * Valid only when inner_second_cvlan_tag ==1 or inner_second_svlan_tag ==1
467 */
468 u32 inner_second_prio:3;
469 /* VLAN ID of first VLAN tag the outer header of the incoming packet.
470 * Valid only when outer_second_cvlan_tag ==1 or outer_second_svlan_tag ==1
471 */
472 u32 outer_second_vid:12;
473 /* CFI bit of first VLAN tag in the outer header of the incoming packet.
474 * Valid only when outer_second_cvlan_tag ==1 or outer_second_svlan_tag ==1
475 */
476 u32 outer_second_cfi:1;
477 /* Priority of second VLAN tag in the outer header of the incoming packet.
478 * Valid only when outer_second_cvlan_tag ==1 or outer_second_svlan_tag ==1
479 */
480 u32 outer_second_prio:3;
481 u32 gre_protocol:16; /* GRE Protocol (outer) */
482 u32 reserved_auto3:12;
483 /* The second vlan in the inner header of the packet is s-vlan (0x8a88).
484 * inner_second_cvlan_tag and inner_second_svlan_tag cannot be set together
485 */
486 u32 inner_second_svlan_tag:1;
487 /* The second vlan in the outer header of the packet is s-vlan (0x8a88).
488 * outer_second_cvlan_tag and outer_second_svlan_tag cannot be set together
489 */
490 u32 outer_second_svlan_tag:1;
491 /* The second vlan in the inner header of the packet is c-vlan (0x8100).
492 * inner_second_cvlan_tag and inner_second_svlan_tag cannot be set together
493 */
494 u32 inner_second_cvlan_tag:1;
495 /* The second vlan in the outer header of the packet is c-vlan (0x8100).
496 * outer_second_cvlan_tag and outer_second_svlan_tag cannot be set together
497 */
498 u32 outer_second_cvlan_tag:1;
499 u32 gre_key_l:8; /* GRE Key [7:0] (outer) */
500 u32 gre_key_h:24; /* GRE Key[31:8] (outer) */
501 u32 reserved_auto4:8;
502 u32 vxlan_vni:24; /* VXLAN VNI (outer) */
503 u32 geneve_oam:1; /* GENEVE OAM field (outer) */
504 u32 reserved_auto5:7;
505 u32 geneve_vni:24; /* GENEVE VNI field (outer) */
506 u32 outer_ipv6_flow_label:20; /* Flow label of incoming IPv6 packet (outer) */
507 u32 reserved_auto6:12;
508 u32 inner_ipv6_flow_label:20; /* Flow label of incoming IPv6 packet (inner) */
509 u32 reserved_auto7:12;
510 u32 geneve_protocol_type:16; /* GENEVE protocol type (outer) */
511 u32 geneve_opt_len:6; /* GENEVE OptLen (outer) */
512 u32 reserved_auto8:10;
513 u32 bth_dst_qp:24; /* Destination QP in BTH header */
514 u32 reserved_auto9:8;
515 u8 reserved_auto10[20];
516};
517
518struct mlx5dr_match_misc2 {
519 u32 outer_first_mpls_ttl:8; /* First MPLS TTL (outer) */
520 u32 outer_first_mpls_s_bos:1; /* First MPLS S_BOS (outer) */
521 u32 outer_first_mpls_exp:3; /* First MPLS EXP (outer) */
522 u32 outer_first_mpls_label:20; /* First MPLS LABEL (outer) */
523 u32 inner_first_mpls_ttl:8; /* First MPLS TTL (inner) */
524 u32 inner_first_mpls_s_bos:1; /* First MPLS S_BOS (inner) */
525 u32 inner_first_mpls_exp:3; /* First MPLS EXP (inner) */
526 u32 inner_first_mpls_label:20; /* First MPLS LABEL (inner) */
527 u32 outer_first_mpls_over_gre_ttl:8; /* last MPLS TTL (outer) */
528 u32 outer_first_mpls_over_gre_s_bos:1; /* last MPLS S_BOS (outer) */
529 u32 outer_first_mpls_over_gre_exp:3; /* last MPLS EXP (outer) */
530 u32 outer_first_mpls_over_gre_label:20; /* last MPLS LABEL (outer) */
531 u32 outer_first_mpls_over_udp_ttl:8; /* last MPLS TTL (outer) */
532 u32 outer_first_mpls_over_udp_s_bos:1; /* last MPLS S_BOS (outer) */
533 u32 outer_first_mpls_over_udp_exp:3; /* last MPLS EXP (outer) */
534 u32 outer_first_mpls_over_udp_label:20; /* last MPLS LABEL (outer) */
535 u32 metadata_reg_c_7; /* metadata_reg_c_7 */
536 u32 metadata_reg_c_6; /* metadata_reg_c_6 */
537 u32 metadata_reg_c_5; /* metadata_reg_c_5 */
538 u32 metadata_reg_c_4; /* metadata_reg_c_4 */
539 u32 metadata_reg_c_3; /* metadata_reg_c_3 */
540 u32 metadata_reg_c_2; /* metadata_reg_c_2 */
541 u32 metadata_reg_c_1; /* metadata_reg_c_1 */
542 u32 metadata_reg_c_0; /* metadata_reg_c_0 */
543 u32 metadata_reg_a; /* metadata_reg_a */
544 u32 metadata_reg_b; /* metadata_reg_b */
545 u8 reserved_auto2[8];
546};
547
548struct mlx5dr_match_misc3 {
549 u32 inner_tcp_seq_num;
550 u32 outer_tcp_seq_num;
551 u32 inner_tcp_ack_num;
552 u32 outer_tcp_ack_num;
553 u32 outer_vxlan_gpe_vni:24;
554 u32 reserved_auto1:8;
555 u32 reserved_auto2:16;
556 u32 outer_vxlan_gpe_flags:8;
557 u32 outer_vxlan_gpe_next_protocol:8;
558 u32 icmpv4_header_data;
559 u32 icmpv6_header_data;
560 u32 icmpv6_code:8;
561 u32 icmpv6_type:8;
562 u32 icmpv4_code:8;
563 u32 icmpv4_type:8;
564 u8 reserved_auto3[0x1c];
565};
566
567struct mlx5dr_match_param {
568 struct mlx5dr_match_spec outer;
569 struct mlx5dr_match_misc misc;
570 struct mlx5dr_match_spec inner;
571 struct mlx5dr_match_misc2 misc2;
572 struct mlx5dr_match_misc3 misc3;
573};
574
575#define DR_MASK_IS_FLEX_PARSER_ICMPV4_SET(_misc3) ((_misc3)->icmpv4_type || \
576 (_misc3)->icmpv4_code || \
577 (_misc3)->icmpv4_header_data)
578
579struct mlx5dr_esw_caps {
580 u64 drop_icm_address_rx;
581 u64 drop_icm_address_tx;
582 u64 uplink_icm_address_rx;
583 u64 uplink_icm_address_tx;
584 bool sw_owner;
585};
586
587struct mlx5dr_cmd_vport_cap {
588 u16 vport_gvmi;
589 u16 vhca_gvmi;
590 u64 icm_address_rx;
591 u64 icm_address_tx;
592 u32 num;
593};
594
595struct mlx5dr_cmd_caps {
596 u16 gvmi;
597 u64 nic_rx_drop_address;
598 u64 nic_tx_drop_address;
599 u64 nic_tx_allow_address;
600 u64 esw_rx_drop_address;
601 u64 esw_tx_drop_address;
602 u32 log_icm_size;
603 u64 hdr_modify_icm_addr;
604 u32 flex_protocols;
605 u8 flex_parser_id_icmp_dw0;
606 u8 flex_parser_id_icmp_dw1;
607 u8 flex_parser_id_icmpv6_dw0;
608 u8 flex_parser_id_icmpv6_dw1;
609 u8 max_ft_level;
610 u16 roce_min_src_udp;
611 u8 num_esw_ports;
612 bool eswitch_manager;
613 bool rx_sw_owner;
614 bool tx_sw_owner;
615 bool fdb_sw_owner;
616 u32 num_vports;
617 struct mlx5dr_esw_caps esw_caps;
618 struct mlx5dr_cmd_vport_cap *vports_caps;
619 bool prio_tag_required;
620};
621
622struct mlx5dr_domain_rx_tx {
623 u64 drop_icm_addr;
624 u64 default_icm_addr;
625 enum mlx5dr_ste_entry_type ste_type;
626};
627
628struct mlx5dr_domain_info {
629 bool supp_sw_steering;
630 u32 max_inline_size;
631 u32 max_send_wr;
632 u32 max_log_sw_icm_sz;
633 u32 max_log_action_icm_sz;
634 struct mlx5dr_domain_rx_tx rx;
635 struct mlx5dr_domain_rx_tx tx;
636 struct mlx5dr_cmd_caps caps;
637};
638
639struct mlx5dr_domain_cache {
640 struct mlx5dr_fw_recalc_cs_ft **recalc_cs_ft;
641};
642
643struct mlx5dr_domain {
644 struct mlx5dr_domain *peer_dmn;
645 struct mlx5_core_dev *mdev;
646 u32 pdn;
647 struct mlx5_uars_page *uar;
648 enum mlx5dr_domain_type type;
649 refcount_t refcount;
650 struct mutex mutex; /* protect domain */
651 struct mlx5dr_icm_pool *ste_icm_pool;
652 struct mlx5dr_icm_pool *action_icm_pool;
653 struct mlx5dr_send_ring *send_ring;
654 struct mlx5dr_domain_info info;
655 struct mlx5dr_domain_cache cache;
656};
657
658struct mlx5dr_table_rx_tx {
659 struct mlx5dr_ste_htbl *s_anchor;
660 struct mlx5dr_domain_rx_tx *nic_dmn;
661 u64 default_icm_addr;
662};
663
664struct mlx5dr_table {
665 struct mlx5dr_domain *dmn;
666 struct mlx5dr_table_rx_tx rx;
667 struct mlx5dr_table_rx_tx tx;
668 u32 level;
669 u32 table_type;
670 u32 table_id;
671 struct list_head matcher_list;
672 struct mlx5dr_action *miss_action;
673 refcount_t refcount;
674};
675
676struct mlx5dr_matcher_rx_tx {
677 struct mlx5dr_ste_htbl *s_htbl;
678 struct mlx5dr_ste_htbl *e_anchor;
679 struct mlx5dr_ste_build *ste_builder;
680 struct mlx5dr_ste_build ste_builder4[DR_RULE_MAX_STES];
681 struct mlx5dr_ste_build ste_builder6[DR_RULE_MAX_STES];
682 u8 num_of_builders;
683 u8 num_of_builders4;
684 u8 num_of_builders6;
685 u64 default_icm_addr;
686 struct mlx5dr_table_rx_tx *nic_tbl;
687};
688
689struct mlx5dr_matcher {
690 struct mlx5dr_table *tbl;
691 struct mlx5dr_matcher_rx_tx rx;
692 struct mlx5dr_matcher_rx_tx tx;
693 struct list_head matcher_list;
694 u16 prio;
695 struct mlx5dr_match_param mask;
696 u8 match_criteria;
697 refcount_t refcount;
698 struct mlx5dv_flow_matcher *dv_matcher;
699};
700
701struct mlx5dr_rule_member {
702 struct mlx5dr_ste *ste;
703 /* attached to mlx5dr_rule via this */
704 struct list_head list;
705 /* attached to mlx5dr_ste via this */
706 struct list_head use_ste_list;
707};
708
709struct mlx5dr_action {
710 enum mlx5dr_action_type action_type;
711 refcount_t refcount;
712 union {
713 struct {
714 struct mlx5dr_domain *dmn;
715 struct mlx5dr_icm_chunk *chunk;
716 u8 *data;
717 u32 data_size;
718 u16 num_of_actions;
719 u32 index;
720 u8 allow_rx:1;
721 u8 allow_tx:1;
722 u8 modify_ttl:1;
723 } rewrite;
724 struct {
725 struct mlx5dr_domain *dmn;
726 u32 reformat_id;
727 u32 reformat_size;
728 } reformat;
729 struct {
730 u8 is_fw_tbl:1;
731 union {
732 struct mlx5dr_table *tbl;
733 struct {
734 struct mlx5_flow_table *ft;
735 u64 rx_icm_addr;
736 u64 tx_icm_addr;
737 struct mlx5_core_dev *mdev;
738 } fw_tbl;
739 };
740 } dest_tbl;
741 struct {
742 u32 ctr_id;
743 u32 offeset;
744 } ctr;
745 struct {
746 struct mlx5dr_domain *dmn;
747 struct mlx5dr_cmd_vport_cap *caps;
748 u32 num;
749 } vport;
750 struct {
751 u32 vlan_hdr; /* tpid_pcp_dei_vid */
752 } push_vlan;
753 u32 flow_tag;
754 };
755};
756
757enum mlx5dr_connect_type {
758 CONNECT_HIT = 1,
759 CONNECT_MISS = 2,
760};
761
762struct mlx5dr_htbl_connect_info {
763 enum mlx5dr_connect_type type;
764 union {
765 struct mlx5dr_ste_htbl *hit_next_htbl;
766 u64 miss_icm_addr;
767 };
768};
769
770struct mlx5dr_rule_rx_tx {
771 struct list_head rule_members_list;
772 struct mlx5dr_matcher_rx_tx *nic_matcher;
773};
774
775struct mlx5dr_rule {
776 struct mlx5dr_matcher *matcher;
777 struct mlx5dr_rule_rx_tx rx;
778 struct mlx5dr_rule_rx_tx tx;
779 struct list_head rule_actions_list;
780};
781
782void mlx5dr_rule_update_rule_member(struct mlx5dr_ste *new_ste,
783 struct mlx5dr_ste *ste);
784
785struct mlx5dr_icm_chunk {
786 struct mlx5dr_icm_bucket *bucket;
787 struct list_head chunk_list;
788 u32 rkey;
789 u32 num_of_entries;
790 u32 byte_size;
791 u64 icm_addr;
792 u64 mr_addr;
793
794 /* Memory optimisation */
795 struct mlx5dr_ste *ste_arr;
796 u8 *hw_ste_arr;
797 struct list_head *miss_list;
798};
799
800static inline int
801mlx5dr_matcher_supp_flex_parser_icmp_v4(struct mlx5dr_cmd_caps *caps)
802{
803 return caps->flex_protocols & MLX5_FLEX_PARSER_ICMP_V4_ENABLED;
804}
805
806static inline int
807mlx5dr_matcher_supp_flex_parser_icmp_v6(struct mlx5dr_cmd_caps *caps)
808{
809 return caps->flex_protocols & MLX5_FLEX_PARSER_ICMP_V6_ENABLED;
810}
811
812int mlx5dr_matcher_select_builders(struct mlx5dr_matcher *matcher,
813 struct mlx5dr_matcher_rx_tx *nic_matcher,
814 bool ipv6);
815
816static inline u32
817mlx5dr_icm_pool_chunk_size_to_entries(enum mlx5dr_icm_chunk_size chunk_size)
818{
819 return 1 << chunk_size;
820}
821
822static inline int
823mlx5dr_icm_pool_chunk_size_to_byte(enum mlx5dr_icm_chunk_size chunk_size,
824 enum mlx5dr_icm_type icm_type)
825{
826 int num_of_entries;
827 int entry_size;
828
829 if (icm_type == DR_ICM_TYPE_STE)
830 entry_size = DR_STE_SIZE;
831 else
832 entry_size = DR_MODIFY_ACTION_SIZE;
833
834 num_of_entries = mlx5dr_icm_pool_chunk_size_to_entries(chunk_size);
835
836 return entry_size * num_of_entries;
837}
838
839static inline struct mlx5dr_cmd_vport_cap *
840mlx5dr_get_vport_cap(struct mlx5dr_cmd_caps *caps, u32 vport)
841{
842 if (!caps->vports_caps ||
843 (vport >= caps->num_vports && vport != WIRE_PORT))
844 return NULL;
845
846 if (vport == WIRE_PORT)
847 vport = caps->num_vports;
848
849 return &caps->vports_caps[vport];
850}
851
852struct mlx5dr_cmd_query_flow_table_details {
853 u8 status;
854 u8 level;
855 u64 sw_owner_icm_root_1;
856 u64 sw_owner_icm_root_0;
857};
858
859/* internal API functions */
860int mlx5dr_cmd_query_device(struct mlx5_core_dev *mdev,
861 struct mlx5dr_cmd_caps *caps);
862int mlx5dr_cmd_query_esw_vport_context(struct mlx5_core_dev *mdev,
863 bool other_vport, u16 vport_number,
864 u64 *icm_address_rx,
865 u64 *icm_address_tx);
866int mlx5dr_cmd_query_gvmi(struct mlx5_core_dev *mdev,
867 bool other_vport, u16 vport_number, u16 *gvmi);
868int mlx5dr_cmd_query_esw_caps(struct mlx5_core_dev *mdev,
869 struct mlx5dr_esw_caps *caps);
870int mlx5dr_cmd_sync_steering(struct mlx5_core_dev *mdev);
871int mlx5dr_cmd_set_fte_modify_and_vport(struct mlx5_core_dev *mdev,
872 u32 table_type,
873 u32 table_id,
874 u32 group_id,
875 u32 modify_header_id,
876 u32 vport_id);
877int mlx5dr_cmd_del_flow_table_entry(struct mlx5_core_dev *mdev,
878 u32 table_type,
879 u32 table_id);
880int mlx5dr_cmd_alloc_modify_header(struct mlx5_core_dev *mdev,
881 u32 table_type,
882 u8 num_of_actions,
883 u64 *actions,
884 u32 *modify_header_id);
885int mlx5dr_cmd_dealloc_modify_header(struct mlx5_core_dev *mdev,
886 u32 modify_header_id);
887int mlx5dr_cmd_create_empty_flow_group(struct mlx5_core_dev *mdev,
888 u32 table_type,
889 u32 table_id,
890 u32 *group_id);
891int mlx5dr_cmd_destroy_flow_group(struct mlx5_core_dev *mdev,
892 u32 table_type,
893 u32 table_id,
894 u32 group_id);
895int mlx5dr_cmd_create_flow_table(struct mlx5_core_dev *mdev,
896 u32 table_type,
897 u64 icm_addr_rx,
898 u64 icm_addr_tx,
899 u8 level,
900 bool sw_owner,
901 bool term_tbl,
902 u64 *fdb_rx_icm_addr,
903 u32 *table_id);
904int mlx5dr_cmd_destroy_flow_table(struct mlx5_core_dev *mdev,
905 u32 table_id,
906 u32 table_type);
907int mlx5dr_cmd_query_flow_table(struct mlx5_core_dev *dev,
908 enum fs_flow_table_type type,
909 u32 table_id,
910 struct mlx5dr_cmd_query_flow_table_details *output);
911int mlx5dr_cmd_create_reformat_ctx(struct mlx5_core_dev *mdev,
912 enum mlx5_reformat_ctx_type rt,
913 size_t reformat_size,
914 void *reformat_data,
915 u32 *reformat_id);
916void mlx5dr_cmd_destroy_reformat_ctx(struct mlx5_core_dev *mdev,
917 u32 reformat_id);
918
919struct mlx5dr_cmd_gid_attr {
920 u8 gid[16];
921 u8 mac[6];
922 u32 roce_ver;
923};
924
925struct mlx5dr_cmd_qp_create_attr {
926 u32 page_id;
927 u32 pdn;
928 u32 cqn;
929 u32 pm_state;
930 u32 service_type;
931 u32 buff_umem_id;
932 u32 db_umem_id;
933 u32 sq_wqe_cnt;
934 u32 rq_wqe_cnt;
935 u32 rq_wqe_shift;
936};
937
938int mlx5dr_cmd_query_gid(struct mlx5_core_dev *mdev, u8 vhca_port_num,
939 u16 index, struct mlx5dr_cmd_gid_attr *attr);
940
941struct mlx5dr_icm_pool *mlx5dr_icm_pool_create(struct mlx5dr_domain *dmn,
942 enum mlx5dr_icm_type icm_type);
943void mlx5dr_icm_pool_destroy(struct mlx5dr_icm_pool *pool);
944
945struct mlx5dr_icm_chunk *
946mlx5dr_icm_alloc_chunk(struct mlx5dr_icm_pool *pool,
947 enum mlx5dr_icm_chunk_size chunk_size);
948void mlx5dr_icm_free_chunk(struct mlx5dr_icm_chunk *chunk);
949bool mlx5dr_ste_is_not_valid_entry(u8 *p_hw_ste);
950int mlx5dr_ste_htbl_init_and_postsend(struct mlx5dr_domain *dmn,
951 struct mlx5dr_domain_rx_tx *nic_dmn,
952 struct mlx5dr_ste_htbl *htbl,
953 struct mlx5dr_htbl_connect_info *connect_info,
954 bool update_hw_ste);
955void mlx5dr_ste_set_formatted_ste(u16 gvmi,
956 struct mlx5dr_domain_rx_tx *nic_dmn,
957 struct mlx5dr_ste_htbl *htbl,
958 u8 *formatted_ste,
959 struct mlx5dr_htbl_connect_info *connect_info);
960void mlx5dr_ste_copy_param(u8 match_criteria,
961 struct mlx5dr_match_param *set_param,
962 struct mlx5dr_match_parameters *mask);
963
964void mlx5dr_crc32_init_table(void);
965u32 mlx5dr_crc32_slice8_calc(const void *input_data, size_t length);
966
967struct mlx5dr_qp {
968 struct mlx5_core_dev *mdev;
969 struct mlx5_wq_qp wq;
970 struct mlx5_uars_page *uar;
971 struct mlx5_wq_ctrl wq_ctrl;
972 struct mlx5_core_qp mqp;
973 struct {
974 unsigned int pc;
975 unsigned int cc;
976 unsigned int size;
977 unsigned int *wqe_head;
978 unsigned int wqe_cnt;
979 } sq;
980 struct {
981 unsigned int pc;
982 unsigned int cc;
983 unsigned int size;
984 unsigned int wqe_cnt;
985 } rq;
986 int max_inline_data;
987};
988
989struct mlx5dr_cq {
990 struct mlx5_core_dev *mdev;
991 struct mlx5_cqwq wq;
992 struct mlx5_wq_ctrl wq_ctrl;
993 struct mlx5_core_cq mcq;
994 struct mlx5dr_qp *qp;
995};
996
997struct mlx5dr_mr {
998 struct mlx5_core_dev *mdev;
999 struct mlx5_core_mkey mkey;
1000 dma_addr_t dma_addr;
1001 void *addr;
1002 size_t size;
1003};
1004
1005#define MAX_SEND_CQE 64
1006#define MIN_READ_SYNC 64
1007
1008struct mlx5dr_send_ring {
1009 struct mlx5dr_cq *cq;
1010 struct mlx5dr_qp *qp;
1011 struct mlx5dr_mr *mr;
1012 /* How much wqes are waiting for completion */
1013 u32 pending_wqe;
1014 /* Signal request per this trash hold value */
1015 u16 signal_th;
1016 /* Each post_send_size less than max_post_send_size */
1017 u32 max_post_send_size;
1018 /* manage the send queue */
1019 u32 tx_head;
1020 void *buf;
1021 u32 buf_size;
1022 struct ib_wc wc[MAX_SEND_CQE];
1023 u8 sync_buff[MIN_READ_SYNC];
1024 struct mlx5dr_mr *sync_mr;
1025};
1026
1027int mlx5dr_send_ring_alloc(struct mlx5dr_domain *dmn);
1028void mlx5dr_send_ring_free(struct mlx5dr_domain *dmn,
1029 struct mlx5dr_send_ring *send_ring);
1030int mlx5dr_send_ring_force_drain(struct mlx5dr_domain *dmn);
1031int mlx5dr_send_postsend_ste(struct mlx5dr_domain *dmn,
1032 struct mlx5dr_ste *ste,
1033 u8 *data,
1034 u16 size,
1035 u16 offset);
1036int mlx5dr_send_postsend_htbl(struct mlx5dr_domain *dmn,
1037 struct mlx5dr_ste_htbl *htbl,
1038 u8 *formatted_ste, u8 *mask);
1039int mlx5dr_send_postsend_formatted_htbl(struct mlx5dr_domain *dmn,
1040 struct mlx5dr_ste_htbl *htbl,
1041 u8 *ste_init_data,
1042 bool update_hw_ste);
1043int mlx5dr_send_postsend_action(struct mlx5dr_domain *dmn,
1044 struct mlx5dr_action *action);
1045
1046struct mlx5dr_fw_recalc_cs_ft {
1047 u64 rx_icm_addr;
1048 u32 table_id;
1049 u32 group_id;
1050 u32 modify_hdr_id;
1051};
1052
1053struct mlx5dr_fw_recalc_cs_ft *
1054mlx5dr_fw_create_recalc_cs_ft(struct mlx5dr_domain *dmn, u32 vport_num);
1055void mlx5dr_fw_destroy_recalc_cs_ft(struct mlx5dr_domain *dmn,
1056 struct mlx5dr_fw_recalc_cs_ft *recalc_cs_ft);
1057int mlx5dr_domain_cache_get_recalc_cs_ft_addr(struct mlx5dr_domain *dmn,
1058 u32 vport_num,
1059 u64 *rx_icm_addr);
1060#endif /* _DR_TYPES_H_ */
drivers/net/ethernet/mellanox/mlx5/core/steering/fs_dr.c+600−0
@@ -0,0 +1,600 @@
1// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2/* Copyright (c) 2019 Mellanox Technologies */
3
4#include "mlx5_core.h"
5#include "fs_core.h"
6#include "fs_cmd.h"
7#include "mlx5dr.h"
8#include "fs_dr.h"
9
10static bool mlx5_dr_is_fw_table(u32 flags)
11{
12 if (flags & MLX5_FLOW_TABLE_TERMINATION)
13 return true;
14
15 return false;
16}
17
18static int mlx5_cmd_dr_update_root_ft(struct mlx5_flow_root_namespace *ns,
19 struct mlx5_flow_table *ft,
20 u32 underlay_qpn,
21 bool disconnect)
22{
23 return mlx5_fs_cmd_get_fw_cmds()->update_root_ft(ns, ft, underlay_qpn,
24 disconnect);
25}
26
27static int set_miss_action(struct mlx5_flow_root_namespace *ns,
28 struct mlx5_flow_table *ft,
29 struct mlx5_flow_table *next_ft)
30{
31 struct mlx5dr_action *old_miss_action;
32 struct mlx5dr_action *action = NULL;
33 struct mlx5dr_table *next_tbl;
34 int err;
35
36 next_tbl = next_ft ? next_ft->fs_dr_table.dr_table : NULL;
37 if (next_tbl) {
38 action = mlx5dr_action_create_dest_table(next_tbl);
39 if (!action)
40 return -EINVAL;
41 }
42 old_miss_action = ft->fs_dr_table.miss_action;
43 err = mlx5dr_table_set_miss_action(ft->fs_dr_table.dr_table, action);
44 if (err && action) {
45 err = mlx5dr_action_destroy(action);
46 if (err) {
47 action = NULL;
48 mlx5_core_err(ns->dev, "Failed to destroy action (%d)\n",
49 err);
50 }
51 }
52 ft->fs_dr_table.miss_action = action;
53 if (old_miss_action) {
54 err = mlx5dr_action_destroy(old_miss_action);
55 if (err)
56 mlx5_core_err(ns->dev, "Failed to destroy action (%d)\n",
57 err);
58 }
59
60 return err;
61}
62
63static int mlx5_cmd_dr_create_flow_table(struct mlx5_flow_root_namespace *ns,
64 struct mlx5_flow_table *ft,
65 unsigned int log_size,
66 struct mlx5_flow_table *next_ft)
67{
68 struct mlx5dr_table *tbl;
69 int err;
70
71 if (mlx5_dr_is_fw_table(ft->flags))
72 return mlx5_fs_cmd_get_fw_cmds()->create_flow_table(ns, ft,
73 log_size,
74 next_ft);
75
76 tbl = mlx5dr_table_create(ns->fs_dr_domain.dr_domain,
77 ft->level);
78 if (!tbl) {
79 mlx5_core_err(ns->dev, "Failed creating dr flow_table\n");
80 return -EINVAL;
81 }
82
83 ft->fs_dr_table.dr_table = tbl;
84 ft->id = mlx5dr_table_get_id(tbl);
85
86 if (next_ft) {
87 err = set_miss_action(ns, ft, next_ft);
88 if (err) {
89 mlx5dr_table_destroy(tbl);
90 ft->fs_dr_table.dr_table = NULL;
91 return err;
92 }
93 }
94
95 return 0;
96}
97
98static int mlx5_cmd_dr_destroy_flow_table(struct mlx5_flow_root_namespace *ns,
99 struct mlx5_flow_table *ft)
100{
101 struct mlx5dr_action *action = ft->fs_dr_table.miss_action;
102 int err;
103
104 if (mlx5_dr_is_fw_table(ft->flags))
105 return mlx5_fs_cmd_get_fw_cmds()->destroy_flow_table(ns, ft);
106
107 err = mlx5dr_table_destroy(ft->fs_dr_table.dr_table);
108 if (err) {
109 mlx5_core_err(ns->dev, "Failed to destroy flow_table (%d)\n",
110 err);
111 return err;
112 }
113 if (action) {
114 err = mlx5dr_action_destroy(action);
115 if (err) {
116 mlx5_core_err(ns->dev, "Failed to destroy action(%d)\n",
117 err);
118 return err;
119 }
120 }
121
122 return err;
123}
124
125static int mlx5_cmd_dr_modify_flow_table(struct mlx5_flow_root_namespace *ns,
126 struct mlx5_flow_table *ft,
127 struct mlx5_flow_table *next_ft)
128{
129 return set_miss_action(ns, ft, next_ft);
130}
131
132static int mlx5_cmd_dr_create_flow_group(struct mlx5_flow_root_namespace *ns,
133 struct mlx5_flow_table *ft,
134 u32 *in,
135 struct mlx5_flow_group *fg)
136{
137 struct mlx5dr_matcher *matcher;
138 u16 priority = MLX5_GET(create_flow_group_in, in,
139 start_flow_index);
140 u8 match_criteria_enable = MLX5_GET(create_flow_group_in,
141 in,
142 match_criteria_enable);
143 struct mlx5dr_match_parameters mask;
144
145 if (mlx5_dr_is_fw_table(ft->flags))
146 return mlx5_fs_cmd_get_fw_cmds()->create_flow_group(ns, ft, in,
147 fg);
148
149 mask.match_buf = MLX5_ADDR_OF(create_flow_group_in,
150 in, match_criteria);
151 mask.match_sz = sizeof(fg->mask.match_criteria);
152
153 matcher = mlx5dr_matcher_create(ft->fs_dr_table.dr_table,
154 priority,
155 match_criteria_enable,
156 &mask);
157 if (!matcher) {
158 mlx5_core_err(ns->dev, "Failed creating matcher\n");
159 return -EINVAL;
160 }
161
162 fg->fs_dr_matcher.dr_matcher = matcher;
163 return 0;
164}
165
166static int mlx5_cmd_dr_destroy_flow_group(struct mlx5_flow_root_namespace *ns,
167 struct mlx5_flow_table *ft,
168 struct mlx5_flow_group *fg)
169{
170 if (mlx5_dr_is_fw_table(ft->flags))
171 return mlx5_fs_cmd_get_fw_cmds()->destroy_flow_group(ns, ft, fg);
172
173 return mlx5dr_matcher_destroy(fg->fs_dr_matcher.dr_matcher);
174}
175
176static struct mlx5dr_action *create_vport_action(struct mlx5dr_domain *domain,
177 struct mlx5_flow_rule *dst)
178{
179 struct mlx5_flow_destination *dest_attr = &dst->dest_attr;
180
181 return mlx5dr_action_create_dest_vport(domain, dest_attr->vport.num,
182 dest_attr->vport.flags &
183 MLX5_FLOW_DEST_VPORT_VHCA_ID,
184 dest_attr->vport.vhca_id);
185}
186
187static struct mlx5dr_action *create_ft_action(struct mlx5_core_dev *dev,
188 struct mlx5_flow_rule *dst)
189{
190 struct mlx5_flow_table *dest_ft = dst->dest_attr.ft;
191
192 if (mlx5_dr_is_fw_table(dest_ft->flags))
193 return mlx5dr_create_action_dest_flow_fw_table(dest_ft, dev);
194 return mlx5dr_action_create_dest_table(dest_ft->fs_dr_table.dr_table);
195}
196
197static struct mlx5dr_action *create_action_push_vlan(struct mlx5dr_domain *domain,
198 struct mlx5_fs_vlan *vlan)
199{
200 u16 n_ethtype = vlan->ethtype;
201 u8 prio = vlan->prio;
202 u16 vid = vlan->vid;
203 u32 vlan_hdr;
204
205 vlan_hdr = (u32)n_ethtype << 16 | (u32)(prio) << 12 | (u32)vid;
206 return mlx5dr_action_create_push_vlan(domain, htonl(vlan_hdr));
207}
208
209#define MLX5_FLOW_CONTEXT_ACTION_MAX 20
210static int mlx5_cmd_dr_create_fte(struct mlx5_flow_root_namespace *ns,
211 struct mlx5_flow_table *ft,
212 struct mlx5_flow_group *group,
213 struct fs_fte *fte)
214{
215 struct mlx5dr_domain *domain = ns->fs_dr_domain.dr_domain;
216 struct mlx5dr_action *term_action = NULL;
217 struct mlx5dr_match_parameters params;
218 struct mlx5_core_dev *dev = ns->dev;
219 struct mlx5dr_action **fs_dr_actions;
220 struct mlx5dr_action *tmp_action;
221 struct mlx5dr_action **actions;
222 bool delay_encap_set = false;
223 struct mlx5dr_rule *rule;
224 struct mlx5_flow_rule *dst;
225 int fs_dr_num_actions = 0;
226 int num_actions = 0;
227 size_t match_sz;
228 int err = 0;
229 int i;
230
231 if (mlx5_dr_is_fw_table(ft->flags))
232 return mlx5_fs_cmd_get_fw_cmds()->create_fte(ns, ft, group, fte);
233
234 actions = kcalloc(MLX5_FLOW_CONTEXT_ACTION_MAX, sizeof(*actions),
235 GFP_KERNEL);
236 if (!actions)
237 return -ENOMEM;
238
239 fs_dr_actions = kcalloc(MLX5_FLOW_CONTEXT_ACTION_MAX,
240 sizeof(*fs_dr_actions), GFP_KERNEL);
241 if (!fs_dr_actions) {
242 kfree(actions);
243 return -ENOMEM;
244 }
245
246 match_sz = sizeof(fte->val);
247
248 /* The order of the actions are must to be keep, only the following
249 * order is supported by SW steering:
250 * TX: push vlan -> modify header -> encap
251 * RX: decap -> pop vlan -> modify header
252 */
253 if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_VLAN_PUSH) {
254 tmp_action = create_action_push_vlan(domain, &fte->action.vlan[0]);
255 if (!tmp_action) {
256 err = -ENOMEM;
257 goto free_actions;
258 }
259 fs_dr_actions[fs_dr_num_actions++] = tmp_action;
260 actions[num_actions++] = tmp_action;
261 }
262
263 if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_VLAN_PUSH_2) {
264 tmp_action = create_action_push_vlan(domain, &fte->action.vlan[1]);
265 if (!tmp_action) {
266 err = -ENOMEM;
267 goto free_actions;
268 }
269 fs_dr_actions[fs_dr_num_actions++] = tmp_action;
270 actions[num_actions++] = tmp_action;
271 }
272
273 if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_DECAP) {
274 enum mlx5dr_action_reformat_type decap_type =
275 DR_ACTION_REFORMAT_TYP_TNL_L2_TO_L2;
276
277 tmp_action = mlx5dr_action_create_packet_reformat(domain,
278 decap_type, 0,
279 NULL);
280 if (!tmp_action) {
281 err = -ENOMEM;
282 goto free_actions;
283 }
284 fs_dr_actions[fs_dr_num_actions++] = tmp_action;
285 actions[num_actions++] = tmp_action;
286 }
287
288 if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT) {
289 bool is_decap = fte->action.pkt_reformat->reformat_type ==
290 MLX5_REFORMAT_TYPE_L3_TUNNEL_TO_L2;
291
292 if (is_decap)
293 actions[num_actions++] =
294 fte->action.pkt_reformat->action.dr_action;
295 else
296 delay_encap_set = true;
297 }
298
299 if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_VLAN_POP) {
300 tmp_action =
301 mlx5dr_action_create_pop_vlan();
302 if (!tmp_action) {
303 err = -ENOMEM;
304 goto free_actions;
305 }
306 fs_dr_actions[fs_dr_num_actions++] = tmp_action;
307 actions[num_actions++] = tmp_action;
308 }
309
310 if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_VLAN_POP_2) {
311 tmp_action =
312 mlx5dr_action_create_pop_vlan();
313 if (!tmp_action) {
314 err = -ENOMEM;
315 goto free_actions;
316 }
317 fs_dr_actions[fs_dr_num_actions++] = tmp_action;
318 actions[num_actions++] = tmp_action;
319 }
320
321 if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)
322 actions[num_actions++] =
323 fte->action.modify_hdr->action.dr_action;
324
325 if (delay_encap_set)
326 actions[num_actions++] =
327 fte->action.pkt_reformat->action.dr_action;
328
329 /* The order of the actions below is not important */
330
331 if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_DROP) {
332 tmp_action = mlx5dr_action_create_drop();
333 if (!tmp_action) {
334 err = -ENOMEM;
335 goto free_actions;
336 }
337 fs_dr_actions[fs_dr_num_actions++] = tmp_action;
338 term_action = tmp_action;
339 }
340
341 if (fte->flow_context.flow_tag) {
342 tmp_action =
343 mlx5dr_action_create_tag(fte->flow_context.flow_tag);
344 if (!tmp_action) {
345 err = -ENOMEM;
346 goto free_actions;
347 }
348 fs_dr_actions[fs_dr_num_actions++] = tmp_action;
349 actions[num_actions++] = tmp_action;
350 }
351
352 if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_FWD_DEST) {
353 list_for_each_entry(dst, &fte->node.children, node.list) {
354 enum mlx5_flow_destination_type type = dst->dest_attr.type;
355 u32 id;
356
357 if (num_actions == MLX5_FLOW_CONTEXT_ACTION_MAX) {
358 err = -ENOSPC;
359 goto free_actions;
360 }
361
362 switch (type) {
363 case MLX5_FLOW_DESTINATION_TYPE_COUNTER:
364 id = dst->dest_attr.counter_id;
365
366 tmp_action =
367 mlx5dr_action_create_flow_counter(id);
368 if (!tmp_action) {
369 err = -ENOMEM;
370 goto free_actions;
371 }
372 fs_dr_actions[fs_dr_num_actions++] = tmp_action;
373 actions[num_actions++] = tmp_action;
374 break;
375 case MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE:
376 tmp_action = create_ft_action(dev, dst);
377 if (!tmp_action) {
378 err = -ENOMEM;
379 goto free_actions;
380 }
381 fs_dr_actions[fs_dr_num_actions++] = tmp_action;
382 term_action = tmp_action;
383 break;
384 case MLX5_FLOW_DESTINATION_TYPE_VPORT:
385 tmp_action = create_vport_action(domain, dst);
386 if (!tmp_action) {
387 err = -ENOMEM;
388 goto free_actions;
389 }
390 fs_dr_actions[fs_dr_num_actions++] = tmp_action;
391 term_action = tmp_action;
392 break;
393 default:
394 err = -EOPNOTSUPP;
395 goto free_actions;
396 }
397 }
398 }
399
400 params.match_sz = match_sz;
401 params.match_buf = (u64 *)fte->val;
402
403 if (term_action)
404 actions[num_actions++] = term_action;
405
406 rule = mlx5dr_rule_create(group->fs_dr_matcher.dr_matcher,
407 &params,
408 num_actions,
409 actions);
410 if (!rule) {
411 err = -EINVAL;
412 goto free_actions;
413 }
414
415 kfree(actions);
416 fte->fs_dr_rule.dr_rule = rule;
417 fte->fs_dr_rule.num_actions = fs_dr_num_actions;
418 fte->fs_dr_rule.dr_actions = fs_dr_actions;
419
420 return 0;
421
422free_actions:
423 for (i = 0; i < fs_dr_num_actions; i++)
424 if (!IS_ERR_OR_NULL(fs_dr_actions[i]))
425 mlx5dr_action_destroy(fs_dr_actions[i]);
426
427 mlx5_core_err(dev, "Failed to create dr rule err(%d)\n", err);
428 kfree(actions);
429 kfree(fs_dr_actions);
430 return err;
431}
432
433static int mlx5_cmd_dr_packet_reformat_alloc(struct mlx5_flow_root_namespace *ns,
434 int reformat_type,
435 size_t size,
436 void *reformat_data,
437 enum mlx5_flow_namespace_type namespace,
438 struct mlx5_pkt_reformat *pkt_reformat)
439{
440 struct mlx5dr_domain *dr_domain = ns->fs_dr_domain.dr_domain;
441 struct mlx5dr_action *action;
442 int dr_reformat;
443
444 switch (reformat_type) {
445 case MLX5_REFORMAT_TYPE_L2_TO_VXLAN:
446 case MLX5_REFORMAT_TYPE_L2_TO_NVGRE:
447 case MLX5_REFORMAT_TYPE_L2_TO_L2_TUNNEL:
448 dr_reformat = DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L2;
449 break;
450 case MLX5_REFORMAT_TYPE_L3_TUNNEL_TO_L2:
451 dr_reformat = DR_ACTION_REFORMAT_TYP_TNL_L3_TO_L2;
452 break;
453 case MLX5_REFORMAT_TYPE_L2_TO_L3_TUNNEL:
454 dr_reformat = DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L3;
455 break;
456 default:
457 mlx5_core_err(ns->dev, "Packet-reformat not supported(%d)\n",
458 reformat_type);
459 return -EOPNOTSUPP;
460 }
461
462 action = mlx5dr_action_create_packet_reformat(dr_domain,
463 dr_reformat,
464 size,
465 reformat_data);
466 if (!action) {
467 mlx5_core_err(ns->dev, "Failed allocating packet-reformat action\n");
468 return -EINVAL;
469 }
470
471 pkt_reformat->action.dr_action = action;
472
473 return 0;
474}
475
476static void mlx5_cmd_dr_packet_reformat_dealloc(struct mlx5_flow_root_namespace *ns,
477 struct mlx5_pkt_reformat *pkt_reformat)
478{
479 mlx5dr_action_destroy(pkt_reformat->action.dr_action);
480}
481
482static int mlx5_cmd_dr_modify_header_alloc(struct mlx5_flow_root_namespace *ns,
483 u8 namespace, u8 num_actions,
484 void *modify_actions,
485 struct mlx5_modify_hdr *modify_hdr)
486{
487 struct mlx5dr_domain *dr_domain = ns->fs_dr_domain.dr_domain;
488 struct mlx5dr_action *action;
489 size_t actions_sz;
490
491 actions_sz = MLX5_UN_SZ_BYTES(set_action_in_add_action_in_auto) *
492 num_actions;
493 action = mlx5dr_action_create_modify_header(dr_domain, 0,
494 actions_sz,
495 modify_actions);
496 if (!action) {
497 mlx5_core_err(ns->dev, "Failed allocating modify-header action\n");
498 return -EINVAL;
499 }
500
501 modify_hdr->action.dr_action = action;
502
503 return 0;
504}
505
506static void mlx5_cmd_dr_modify_header_dealloc(struct mlx5_flow_root_namespace *ns,
507 struct mlx5_modify_hdr *modify_hdr)
508{
509 mlx5dr_action_destroy(modify_hdr->action.dr_action);
510}
511
512static int mlx5_cmd_dr_update_fte(struct mlx5_flow_root_namespace *ns,
513 struct mlx5_flow_table *ft,
514 struct mlx5_flow_group *group,
515 int modify_mask,
516 struct fs_fte *fte)
517{
518 return -EOPNOTSUPP;
519}
520
521static int mlx5_cmd_dr_delete_fte(struct mlx5_flow_root_namespace *ns,
522 struct mlx5_flow_table *ft,
523 struct fs_fte *fte)
524{
525 struct mlx5_fs_dr_rule *rule = &fte->fs_dr_rule;
526 int err;
527 int i;
528
529 if (mlx5_dr_is_fw_table(ft->flags))
530 return mlx5_fs_cmd_get_fw_cmds()->delete_fte(ns, ft, fte);
531
532 err = mlx5dr_rule_destroy(rule->dr_rule);
533 if (err)
534 return err;
535
536 for (i = 0; i < rule->num_actions; i++)
537 if (!IS_ERR_OR_NULL(rule->dr_actions[i]))
538 mlx5dr_action_destroy(rule->dr_actions[i]);
539
540 kfree(rule->dr_actions);
541 return 0;
542}
543
544static int mlx5_cmd_dr_set_peer(struct mlx5_flow_root_namespace *ns,
545 struct mlx5_flow_root_namespace *peer_ns)
546{
547 struct mlx5dr_domain *peer_domain = NULL;
548
549 if (peer_ns)
550 peer_domain = peer_ns->fs_dr_domain.dr_domain;
551 mlx5dr_domain_set_peer(ns->fs_dr_domain.dr_domain,
552 peer_domain);
553 return 0;
554}
555
556static int mlx5_cmd_dr_create_ns(struct mlx5_flow_root_namespace *ns)
557{
558 ns->fs_dr_domain.dr_domain =
559 mlx5dr_domain_create(ns->dev,
560 MLX5DR_DOMAIN_TYPE_FDB);
561 if (!ns->fs_dr_domain.dr_domain) {
562 mlx5_core_err(ns->dev, "Failed to create dr flow namespace\n");
563 return -EOPNOTSUPP;
564 }
565 return 0;
566}
567
568static int mlx5_cmd_dr_destroy_ns(struct mlx5_flow_root_namespace *ns)
569{
570 return mlx5dr_domain_destroy(ns->fs_dr_domain.dr_domain);
571}
572
573bool mlx5_fs_dr_is_supported(struct mlx5_core_dev *dev)
574{
575 return mlx5dr_is_supported(dev);
576}
577
578static const struct mlx5_flow_cmds mlx5_flow_cmds_dr = {
579 .create_flow_table = mlx5_cmd_dr_create_flow_table,
580 .destroy_flow_table = mlx5_cmd_dr_destroy_flow_table,
581 .modify_flow_table = mlx5_cmd_dr_modify_flow_table,
582 .create_flow_group = mlx5_cmd_dr_create_flow_group,
583 .destroy_flow_group = mlx5_cmd_dr_destroy_flow_group,
584 .create_fte = mlx5_cmd_dr_create_fte,
585 .update_fte = mlx5_cmd_dr_update_fte,
586 .delete_fte = mlx5_cmd_dr_delete_fte,
587 .update_root_ft = mlx5_cmd_dr_update_root_ft,
588 .packet_reformat_alloc = mlx5_cmd_dr_packet_reformat_alloc,
589 .packet_reformat_dealloc = mlx5_cmd_dr_packet_reformat_dealloc,
590 .modify_header_alloc = mlx5_cmd_dr_modify_header_alloc,
591 .modify_header_dealloc = mlx5_cmd_dr_modify_header_dealloc,
592 .set_peer = mlx5_cmd_dr_set_peer,
593 .create_ns = mlx5_cmd_dr_create_ns,
594 .destroy_ns = mlx5_cmd_dr_destroy_ns,
595};
596
597const struct mlx5_flow_cmds *mlx5_fs_cmd_get_dr_cmds(void)
598{
599 return &mlx5_flow_cmds_dr;
600}
drivers/net/ethernet/mellanox/mlx5/core/steering/fs_dr.h+60−0
@@ -0,0 +1,60 @@
1/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2 * Copyright (c) 2019 Mellanox Technologies
3 */
4
5#ifndef _MLX5_FS_DR_
6#define _MLX5_FS_DR_
7
8#include "mlx5dr.h"
9
10struct mlx5_flow_root_namespace;
11struct fs_fte;
12
13struct mlx5_fs_dr_action {
14 struct mlx5dr_action *dr_action;
15};
16
17struct mlx5_fs_dr_ns {
18 struct mlx5_dr_ns *dr_ns;
19};
20
21struct mlx5_fs_dr_rule {
22 struct mlx5dr_rule *dr_rule;
23 /* Only actions created by fs_dr */
24 struct mlx5dr_action **dr_actions;
25 int num_actions;
26};
27
28struct mlx5_fs_dr_domain {
29 struct mlx5dr_domain *dr_domain;
30};
31
32struct mlx5_fs_dr_matcher {
33 struct mlx5dr_matcher *dr_matcher;
34};
35
36struct mlx5_fs_dr_table {
37 struct mlx5dr_table *dr_table;
38 struct mlx5dr_action *miss_action;
39};
40
41#ifdef CONFIG_MLX5_SW_STEERING
42
43bool mlx5_fs_dr_is_supported(struct mlx5_core_dev *dev);
44
45const struct mlx5_flow_cmds *mlx5_fs_cmd_get_dr_cmds(void);
46
47#else
48
49static inline const struct mlx5_flow_cmds *mlx5_fs_cmd_get_dr_cmds(void)
50{
51 return NULL;
52}
53
54static inline bool mlx5_fs_dr_is_supported(struct mlx5_core_dev *dev)
55{
56 return false;
57}
58
59#endif /* CONFIG_MLX5_SW_STEERING */
60#endif
drivers/net/ethernet/mellanox/mlx5/core/steering/mlx5_ifc_dr.h+604−0
@@ -0,0 +1,604 @@
1/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
2/* Copyright (c) 2019, Mellanox Technologies */
3
4#ifndef MLX5_IFC_DR_H
5#define MLX5_IFC_DR_H
6
7enum {
8 MLX5DR_ACTION_MDFY_HW_FLD_L2_0 = 0,
9 MLX5DR_ACTION_MDFY_HW_FLD_L2_1 = 1,
10 MLX5DR_ACTION_MDFY_HW_FLD_L2_2 = 2,
11 MLX5DR_ACTION_MDFY_HW_FLD_L3_0 = 3,
12 MLX5DR_ACTION_MDFY_HW_FLD_L3_1 = 4,
13 MLX5DR_ACTION_MDFY_HW_FLD_L3_2 = 5,
14 MLX5DR_ACTION_MDFY_HW_FLD_L3_3 = 6,
15 MLX5DR_ACTION_MDFY_HW_FLD_L3_4 = 7,
16 MLX5DR_ACTION_MDFY_HW_FLD_L4_0 = 8,
17 MLX5DR_ACTION_MDFY_HW_FLD_L4_1 = 9,
18 MLX5DR_ACTION_MDFY_HW_FLD_MPLS = 10,
19 MLX5DR_ACTION_MDFY_HW_FLD_L2_TNL_0 = 11,
20 MLX5DR_ACTION_MDFY_HW_FLD_REG_0 = 12,
21 MLX5DR_ACTION_MDFY_HW_FLD_REG_1 = 13,
22 MLX5DR_ACTION_MDFY_HW_FLD_REG_2 = 14,
23 MLX5DR_ACTION_MDFY_HW_FLD_REG_3 = 15,
24 MLX5DR_ACTION_MDFY_HW_FLD_L4_2 = 16,
25 MLX5DR_ACTION_MDFY_HW_FLD_FLEX_0 = 17,
26 MLX5DR_ACTION_MDFY_HW_FLD_FLEX_1 = 18,
27 MLX5DR_ACTION_MDFY_HW_FLD_FLEX_2 = 19,
28 MLX5DR_ACTION_MDFY_HW_FLD_FLEX_3 = 20,
29 MLX5DR_ACTION_MDFY_HW_FLD_L2_TNL_1 = 21,
30 MLX5DR_ACTION_MDFY_HW_FLD_METADATA = 22,
31 MLX5DR_ACTION_MDFY_HW_FLD_RESERVED = 23,
32};
33
34enum {
35 MLX5DR_ACTION_MDFY_HW_OP_SET = 0x2,
36 MLX5DR_ACTION_MDFY_HW_OP_ADD = 0x3,
37};
38
39enum {
40 MLX5DR_ACTION_MDFY_HW_HDR_L3_NONE = 0x0,
41 MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV4 = 0x1,
42 MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6 = 0x2,
43};
44
45enum {
46 MLX5DR_ACTION_MDFY_HW_HDR_L4_NONE = 0x0,
47 MLX5DR_ACTION_MDFY_HW_HDR_L4_TCP = 0x1,
48 MLX5DR_ACTION_MDFY_HW_HDR_L4_UDP = 0x2,
49};
50
51enum {
52 MLX5DR_STE_LU_TYPE_NOP = 0x00,
53 MLX5DR_STE_LU_TYPE_SRC_GVMI_AND_QP = 0x05,
54 MLX5DR_STE_LU_TYPE_ETHL2_TUNNELING_I = 0x0a,
55 MLX5DR_STE_LU_TYPE_ETHL2_DST_O = 0x06,
56 MLX5DR_STE_LU_TYPE_ETHL2_DST_I = 0x07,
57 MLX5DR_STE_LU_TYPE_ETHL2_DST_D = 0x1b,
58 MLX5DR_STE_LU_TYPE_ETHL2_SRC_O = 0x08,
59 MLX5DR_STE_LU_TYPE_ETHL2_SRC_I = 0x09,
60 MLX5DR_STE_LU_TYPE_ETHL2_SRC_D = 0x1c,
61 MLX5DR_STE_LU_TYPE_ETHL2_SRC_DST_O = 0x36,
62 MLX5DR_STE_LU_TYPE_ETHL2_SRC_DST_I = 0x37,
63 MLX5DR_STE_LU_TYPE_ETHL2_SRC_DST_D = 0x38,
64 MLX5DR_STE_LU_TYPE_ETHL3_IPV6_DST_O = 0x0d,
65 MLX5DR_STE_LU_TYPE_ETHL3_IPV6_DST_I = 0x0e,
66 MLX5DR_STE_LU_TYPE_ETHL3_IPV6_DST_D = 0x1e,
67 MLX5DR_STE_LU_TYPE_ETHL3_IPV6_SRC_O = 0x0f,
68 MLX5DR_STE_LU_TYPE_ETHL3_IPV6_SRC_I = 0x10,
69 MLX5DR_STE_LU_TYPE_ETHL3_IPV6_SRC_D = 0x1f,
70 MLX5DR_STE_LU_TYPE_ETHL3_IPV4_5_TUPLE_O = 0x11,
71 MLX5DR_STE_LU_TYPE_ETHL3_IPV4_5_TUPLE_I = 0x12,
72 MLX5DR_STE_LU_TYPE_ETHL3_IPV4_5_TUPLE_D = 0x20,
73 MLX5DR_STE_LU_TYPE_ETHL3_IPV4_MISC_O = 0x29,
74 MLX5DR_STE_LU_TYPE_ETHL3_IPV4_MISC_I = 0x2a,
75 MLX5DR_STE_LU_TYPE_ETHL3_IPV4_MISC_D = 0x2b,
76 MLX5DR_STE_LU_TYPE_ETHL4_O = 0x13,
77 MLX5DR_STE_LU_TYPE_ETHL4_I = 0x14,
78 MLX5DR_STE_LU_TYPE_ETHL4_D = 0x21,
79 MLX5DR_STE_LU_TYPE_ETHL4_MISC_O = 0x2c,
80 MLX5DR_STE_LU_TYPE_ETHL4_MISC_I = 0x2d,
81 MLX5DR_STE_LU_TYPE_ETHL4_MISC_D = 0x2e,
82 MLX5DR_STE_LU_TYPE_MPLS_FIRST_O = 0x15,
83 MLX5DR_STE_LU_TYPE_MPLS_FIRST_I = 0x24,
84 MLX5DR_STE_LU_TYPE_MPLS_FIRST_D = 0x25,
85 MLX5DR_STE_LU_TYPE_GRE = 0x16,
86 MLX5DR_STE_LU_TYPE_FLEX_PARSER_0 = 0x22,
87 MLX5DR_STE_LU_TYPE_FLEX_PARSER_1 = 0x23,
88 MLX5DR_STE_LU_TYPE_FLEX_PARSER_TNL_HEADER = 0x19,
89 MLX5DR_STE_LU_TYPE_GENERAL_PURPOSE = 0x18,
90 MLX5DR_STE_LU_TYPE_STEERING_REGISTERS_0 = 0x2f,
91 MLX5DR_STE_LU_TYPE_STEERING_REGISTERS_1 = 0x30,
92 MLX5DR_STE_LU_TYPE_DONT_CARE = 0x0f,
93};
94
95enum mlx5dr_ste_entry_type {
96 MLX5DR_STE_TYPE_TX = 1,
97 MLX5DR_STE_TYPE_RX = 2,
98 MLX5DR_STE_TYPE_MODIFY_PKT = 6,
99};
100
101struct mlx5_ifc_ste_general_bits {
102 u8 entry_type[0x4];
103 u8 reserved_at_4[0x4];
104 u8 entry_sub_type[0x8];
105 u8 byte_mask[0x10];
106
107 u8 next_table_base_63_48[0x10];
108 u8 next_lu_type[0x8];
109 u8 next_table_base_39_32_size[0x8];
110
111 u8 next_table_base_31_5_size[0x1b];
112 u8 linear_hash_enable[0x1];
113 u8 reserved_at_5c[0x2];
114 u8 next_table_rank[0x2];
115
116 u8 reserved_at_60[0xa0];
117 u8 tag_value[0x60];
118 u8 bit_mask[0x60];
119};
120
121struct mlx5_ifc_ste_sx_transmit_bits {
122 u8 entry_type[0x4];
123 u8 reserved_at_4[0x4];
124 u8 entry_sub_type[0x8];
125 u8 byte_mask[0x10];
126
127 u8 next_table_base_63_48[0x10];
128 u8 next_lu_type[0x8];
129 u8 next_table_base_39_32_size[0x8];
130
131 u8 next_table_base_31_5_size[0x1b];
132 u8 linear_hash_enable[0x1];
133 u8 reserved_at_5c[0x2];
134 u8 next_table_rank[0x2];
135
136 u8 sx_wire[0x1];
137 u8 sx_func_lb[0x1];
138 u8 sx_sniffer[0x1];
139 u8 sx_wire_enable[0x1];
140 u8 sx_func_lb_enable[0x1];
141 u8 sx_sniffer_enable[0x1];
142 u8 action_type[0x3];
143 u8 reserved_at_69[0x1];
144 u8 action_description[0x6];
145 u8 gvmi[0x10];
146
147 u8 encap_pointer_vlan_data[0x20];
148
149 u8 loopback_syndome_en[0x8];
150 u8 loopback_syndome[0x8];
151 u8 counter_trigger[0x10];
152
153 u8 miss_address_63_48[0x10];
154 u8 counter_trigger_23_16[0x8];
155 u8 miss_address_39_32[0x8];
156
157 u8 miss_address_31_6[0x1a];
158 u8 learning_point[0x1];
159 u8 go_back[0x1];
160 u8 match_polarity[0x1];
161 u8 mask_mode[0x1];
162 u8 miss_rank[0x2];
163};
164
165struct mlx5_ifc_ste_rx_steering_mult_bits {
166 u8 entry_type[0x4];
167 u8 reserved_at_4[0x4];
168 u8 entry_sub_type[0x8];
169 u8 byte_mask[0x10];
170
171 u8 next_table_base_63_48[0x10];
172 u8 next_lu_type[0x8];
173 u8 next_table_base_39_32_size[0x8];
174
175 u8 next_table_base_31_5_size[0x1b];
176 u8 linear_hash_enable[0x1];
177 u8 reserved_at_[0x2];
178 u8 next_table_rank[0x2];
179
180 u8 member_count[0x10];
181 u8 gvmi[0x10];
182
183 u8 qp_list_pointer[0x20];
184
185 u8 reserved_at_a0[0x1];
186 u8 tunneling_action[0x3];
187 u8 action_description[0x4];
188 u8 reserved_at_a8[0x8];
189 u8 counter_trigger_15_0[0x10];
190
191 u8 miss_address_63_48[0x10];
192 u8 counter_trigger_23_16[0x08];
193 u8 miss_address_39_32[0x8];
194
195 u8 miss_address_31_6[0x1a];
196 u8 learning_point[0x1];
197 u8 fail_on_error[0x1];
198 u8 match_polarity[0x1];
199 u8 mask_mode[0x1];
200 u8 miss_rank[0x2];
201};
202
203struct mlx5_ifc_ste_modify_packet_bits {
204 u8 entry_type[0x4];
205 u8 reserved_at_4[0x4];
206 u8 entry_sub_type[0x8];
207 u8 byte_mask[0x10];
208
209 u8 next_table_base_63_48[0x10];
210 u8 next_lu_type[0x8];
211 u8 next_table_base_39_32_size[0x8];
212
213 u8 next_table_base_31_5_size[0x1b];
214 u8 linear_hash_enable[0x1];
215 u8 reserved_at_[0x2];
216 u8 next_table_rank[0x2];
217
218 u8 number_of_re_write_actions[0x10];
219 u8 gvmi[0x10];
220
221 u8 header_re_write_actions_pointer[0x20];
222
223 u8 reserved_at_a0[0x1];
224 u8 tunneling_action[0x3];
225 u8 action_description[0x4];
226 u8 reserved_at_a8[0x8];
227 u8 counter_trigger_15_0[0x10];
228
229 u8 miss_address_63_48[0x10];
230 u8 counter_trigger_23_16[0x08];
231 u8 miss_address_39_32[0x8];
232
233 u8 miss_address_31_6[0x1a];
234 u8 learning_point[0x1];
235 u8 fail_on_error[0x1];
236 u8 match_polarity[0x1];
237 u8 mask_mode[0x1];
238 u8 miss_rank[0x2];
239};
240
241struct mlx5_ifc_ste_eth_l2_src_bits {
242 u8 smac_47_16[0x20];
243
244 u8 smac_15_0[0x10];
245 u8 l3_ethertype[0x10];
246
247 u8 qp_type[0x2];
248 u8 ethertype_filter[0x1];
249 u8 reserved_at_43[0x1];
250 u8 sx_sniffer[0x1];
251 u8 force_lb[0x1];
252 u8 functional_lb[0x1];
253 u8 port[0x1];
254 u8 reserved_at_48[0x4];
255 u8 first_priority[0x3];
256 u8 first_cfi[0x1];
257 u8 first_vlan_qualifier[0x2];
258 u8 reserved_at_52[0x2];
259 u8 first_vlan_id[0xc];
260
261 u8 ip_fragmented[0x1];
262 u8 tcp_syn[0x1];
263 u8 encp_type[0x2];
264 u8 l3_type[0x2];
265 u8 l4_type[0x2];
266 u8 reserved_at_68[0x4];
267 u8 second_priority[0x3];
268 u8 second_cfi[0x1];
269 u8 second_vlan_qualifier[0x2];
270 u8 reserved_at_72[0x2];
271 u8 second_vlan_id[0xc];
272};
273
274struct mlx5_ifc_ste_eth_l2_dst_bits {
275 u8 dmac_47_16[0x20];
276
277 u8 dmac_15_0[0x10];
278 u8 l3_ethertype[0x10];
279
280 u8 qp_type[0x2];
281 u8 ethertype_filter[0x1];
282 u8 reserved_at_43[0x1];
283 u8 sx_sniffer[0x1];
284 u8 force_lb[0x1];
285 u8 functional_lb[0x1];
286 u8 port[0x1];
287 u8 reserved_at_48[0x4];
288 u8 first_priority[0x3];
289 u8 first_cfi[0x1];
290 u8 first_vlan_qualifier[0x2];
291 u8 reserved_at_52[0x2];
292 u8 first_vlan_id[0xc];
293
294 u8 ip_fragmented[0x1];
295 u8 tcp_syn[0x1];
296 u8 encp_type[0x2];
297 u8 l3_type[0x2];
298 u8 l4_type[0x2];
299 u8 reserved_at_68[0x4];
300 u8 second_priority[0x3];
301 u8 second_cfi[0x1];
302 u8 second_vlan_qualifier[0x2];
303 u8 reserved_at_72[0x2];
304 u8 second_vlan_id[0xc];
305};
306
307struct mlx5_ifc_ste_eth_l2_src_dst_bits {
308 u8 dmac_47_16[0x20];
309
310 u8 dmac_15_0[0x10];
311 u8 smac_47_32[0x10];
312
313 u8 smac_31_0[0x20];
314
315 u8 sx_sniffer[0x1];
316 u8 force_lb[0x1];
317 u8 functional_lb[0x1];
318 u8 port[0x1];
319 u8 l3_type[0x2];
320 u8 reserved_at_66[0x6];
321 u8 first_priority[0x3];
322 u8 first_cfi[0x1];
323 u8 first_vlan_qualifier[0x2];
324 u8 reserved_at_72[0x2];
325 u8 first_vlan_id[0xc];
326};
327
328struct mlx5_ifc_ste_eth_l3_ipv4_5_tuple_bits {
329 u8 destination_address[0x20];
330
331 u8 source_address[0x20];
332
333 u8 source_port[0x10];
334 u8 destination_port[0x10];
335
336 u8 fragmented[0x1];
337 u8 first_fragment[0x1];
338 u8 reserved_at_62[0x2];
339 u8 reserved_at_64[0x1];
340 u8 ecn[0x2];
341 u8 tcp_ns[0x1];
342 u8 tcp_cwr[0x1];
343 u8 tcp_ece[0x1];
344 u8 tcp_urg[0x1];
345 u8 tcp_ack[0x1];
346 u8 tcp_psh[0x1];
347 u8 tcp_rst[0x1];
348 u8 tcp_syn[0x1];
349 u8 tcp_fin[0x1];
350 u8 dscp[0x6];
351 u8 reserved_at_76[0x2];
352 u8 protocol[0x8];
353};
354
355struct mlx5_ifc_ste_eth_l3_ipv6_dst_bits {
356 u8 dst_ip_127_96[0x20];
357
358 u8 dst_ip_95_64[0x20];
359
360 u8 dst_ip_63_32[0x20];
361
362 u8 dst_ip_31_0[0x20];
363};
364
365struct mlx5_ifc_ste_eth_l2_tnl_bits {
366 u8 dmac_47_16[0x20];
367
368 u8 dmac_15_0[0x10];
369 u8 l3_ethertype[0x10];
370
371 u8 l2_tunneling_network_id[0x20];
372
373 u8 ip_fragmented[0x1];
374 u8 tcp_syn[0x1];
375 u8 encp_type[0x2];
376 u8 l3_type[0x2];
377 u8 l4_type[0x2];
378 u8 first_priority[0x3];
379 u8 first_cfi[0x1];
380 u8 reserved_at_6c[0x3];
381 u8 gre_key_flag[0x1];
382 u8 first_vlan_qualifier[0x2];
383 u8 reserved_at_72[0x2];
384 u8 first_vlan_id[0xc];
385};
386
387struct mlx5_ifc_ste_eth_l3_ipv6_src_bits {
388 u8 src_ip_127_96[0x20];
389
390 u8 src_ip_95_64[0x20];
391
392 u8 src_ip_63_32[0x20];
393
394 u8 src_ip_31_0[0x20];
395};
396
397struct mlx5_ifc_ste_eth_l3_ipv4_misc_bits {
398 u8 version[0x4];
399 u8 ihl[0x4];
400 u8 reserved_at_8[0x8];
401 u8 total_length[0x10];
402
403 u8 identification[0x10];
404 u8 flags[0x3];
405 u8 fragment_offset[0xd];
406
407 u8 time_to_live[0x8];
408 u8 reserved_at_48[0x8];
409 u8 checksum[0x10];
410
411 u8 reserved_at_60[0x20];
412};
413
414struct mlx5_ifc_ste_eth_l4_bits {
415 u8 fragmented[0x1];
416 u8 first_fragment[0x1];
417 u8 reserved_at_2[0x6];
418 u8 protocol[0x8];
419 u8 dst_port[0x10];
420
421 u8 ipv6_version[0x4];
422 u8 reserved_at_24[0x1];
423 u8 ecn[0x2];
424 u8 tcp_ns[0x1];
425 u8 tcp_cwr[0x1];
426 u8 tcp_ece[0x1];
427 u8 tcp_urg[0x1];
428 u8 tcp_ack[0x1];
429 u8 tcp_psh[0x1];
430 u8 tcp_rst[0x1];
431 u8 tcp_syn[0x1];
432 u8 tcp_fin[0x1];
433 u8 src_port[0x10];
434
435 u8 ipv6_payload_length[0x10];
436 u8 ipv6_hop_limit[0x8];
437 u8 dscp[0x6];
438 u8 reserved_at_5e[0x2];
439
440 u8 tcp_data_offset[0x4];
441 u8 reserved_at_64[0x8];
442 u8 flow_label[0x14];
443};
444
445struct mlx5_ifc_ste_eth_l4_misc_bits {
446 u8 checksum[0x10];
447 u8 length[0x10];
448
449 u8 seq_num[0x20];
450
451 u8 ack_num[0x20];
452
453 u8 urgent_pointer[0x10];
454 u8 window_size[0x10];
455};
456
457struct mlx5_ifc_ste_mpls_bits {
458 u8 mpls0_label[0x14];
459 u8 mpls0_exp[0x3];
460 u8 mpls0_s_bos[0x1];
461 u8 mpls0_ttl[0x8];
462
463 u8 mpls1_label[0x20];
464
465 u8 mpls2_label[0x20];
466
467 u8 reserved_at_60[0x16];
468 u8 mpls4_s_bit[0x1];
469 u8 mpls4_qualifier[0x1];
470 u8 mpls3_s_bit[0x1];
471 u8 mpls3_qualifier[0x1];
472 u8 mpls2_s_bit[0x1];
473 u8 mpls2_qualifier[0x1];
474 u8 mpls1_s_bit[0x1];
475 u8 mpls1_qualifier[0x1];
476 u8 mpls0_s_bit[0x1];
477 u8 mpls0_qualifier[0x1];
478};
479
480struct mlx5_ifc_ste_register_0_bits {
481 u8 register_0_h[0x20];
482
483 u8 register_0_l[0x20];
484
485 u8 register_1_h[0x20];
486
487 u8 register_1_l[0x20];
488};
489
490struct mlx5_ifc_ste_register_1_bits {
491 u8 register_2_h[0x20];
492
493 u8 register_2_l[0x20];
494
495 u8 register_3_h[0x20];
496
497 u8 register_3_l[0x20];
498};
499
500struct mlx5_ifc_ste_gre_bits {
501 u8 gre_c_present[0x1];
502 u8 reserved_at_30[0x1];
503 u8 gre_k_present[0x1];
504 u8 gre_s_present[0x1];
505 u8 strict_src_route[0x1];
506 u8 recur[0x3];
507 u8 flags[0x5];
508 u8 version[0x3];
509 u8 gre_protocol[0x10];
510
511 u8 checksum[0x10];
512 u8 offset[0x10];
513
514 u8 gre_key_h[0x18];
515 u8 gre_key_l[0x8];
516
517 u8 seq_num[0x20];
518};
519
520struct mlx5_ifc_ste_flex_parser_0_bits {
521 u8 parser_3_label[0x14];
522 u8 parser_3_exp[0x3];
523 u8 parser_3_s_bos[0x1];
524 u8 parser_3_ttl[0x8];
525
526 u8 flex_parser_2[0x20];
527
528 u8 flex_parser_1[0x20];
529
530 u8 flex_parser_0[0x20];
531};
532
533struct mlx5_ifc_ste_flex_parser_1_bits {
534 u8 flex_parser_7[0x20];
535
536 u8 flex_parser_6[0x20];
537
538 u8 flex_parser_5[0x20];
539
540 u8 flex_parser_4[0x20];
541};
542
543struct mlx5_ifc_ste_flex_parser_tnl_bits {
544 u8 flex_parser_tunneling_header_63_32[0x20];
545
546 u8 flex_parser_tunneling_header_31_0[0x20];
547
548 u8 reserved_at_40[0x40];
549};
550
551struct mlx5_ifc_ste_general_purpose_bits {
552 u8 general_purpose_lookup_field[0x20];
553
554 u8 reserved_at_20[0x20];
555
556 u8 reserved_at_40[0x20];
557
558 u8 reserved_at_60[0x20];
559};
560
561struct mlx5_ifc_ste_src_gvmi_qp_bits {
562 u8 loopback_syndrome[0x8];
563 u8 reserved_at_8[0x8];
564 u8 source_gvmi[0x10];
565
566 u8 reserved_at_20[0x5];
567 u8 force_lb[0x1];
568 u8 functional_lb[0x1];
569 u8 source_is_requestor[0x1];
570 u8 source_qp[0x18];
571
572 u8 reserved_at_40[0x20];
573
574 u8 reserved_at_60[0x20];
575};
576
577struct mlx5_ifc_l2_hdr_bits {
578 u8 dmac_47_16[0x20];
579
580 u8 dmac_15_0[0x10];
581 u8 smac_47_32[0x10];
582
583 u8 smac_31_0[0x20];
584
585 u8 ethertype[0x10];
586 u8 vlan_type[0x10];
587
588 u8 vlan[0x10];
589 u8 reserved_at_90[0x10];
590};
591
592/* Both HW set and HW add share the same HW format with different opcodes */
593struct mlx5_ifc_dr_action_hw_set_bits {
594 u8 opcode[0x8];
595 u8 destination_field_code[0x8];
596 u8 reserved_at_10[0x2];
597 u8 destination_left_shifter[0x6];
598 u8 reserved_at_18[0x3];
599 u8 destination_length[0x5];
600
601 u8 inline_data[0x20];
602};
603
604#endif /* MLX5_IFC_DR_H */
drivers/net/ethernet/mellanox/mlx5/core/steering/mlx5dr.h+212−0
@@ -0,0 +1,212 @@
1/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
2/* Copyright (c) 2019, Mellanox Technologies */
3
4#ifndef _MLX5DR_H_
5#define _MLX5DR_H_
6
7struct mlx5dr_domain;
8struct mlx5dr_table;
9struct mlx5dr_matcher;
10struct mlx5dr_rule;
11struct mlx5dr_action;
12
13enum mlx5dr_domain_type {
14 MLX5DR_DOMAIN_TYPE_NIC_RX,
15 MLX5DR_DOMAIN_TYPE_NIC_TX,
16 MLX5DR_DOMAIN_TYPE_FDB,
17};
18
19enum mlx5dr_domain_sync_flags {
20 MLX5DR_DOMAIN_SYNC_FLAGS_SW = 1 << 0,
21 MLX5DR_DOMAIN_SYNC_FLAGS_HW = 1 << 1,
22};
23
24enum mlx5dr_action_reformat_type {
25 DR_ACTION_REFORMAT_TYP_TNL_L2_TO_L2,
26 DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L2,
27 DR_ACTION_REFORMAT_TYP_TNL_L3_TO_L2,
28 DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L3,
29};
30
31struct mlx5dr_match_parameters {
32 size_t match_sz;
33 u64 *match_buf; /* Device spec format */
34};
35
36#ifdef CONFIG_MLX5_SW_STEERING
37
38struct mlx5dr_domain *
39mlx5dr_domain_create(struct mlx5_core_dev *mdev, enum mlx5dr_domain_type type);
40
41int mlx5dr_domain_destroy(struct mlx5dr_domain *domain);
42
43int mlx5dr_domain_sync(struct mlx5dr_domain *domain, u32 flags);
44
45void mlx5dr_domain_set_peer(struct mlx5dr_domain *dmn,
46 struct mlx5dr_domain *peer_dmn);
47
48struct mlx5dr_table *
49mlx5dr_table_create(struct mlx5dr_domain *domain, u32 level);
50
51int mlx5dr_table_destroy(struct mlx5dr_table *table);
52
53u32 mlx5dr_table_get_id(struct mlx5dr_table *table);
54
55struct mlx5dr_matcher *
56mlx5dr_matcher_create(struct mlx5dr_table *table,
57 u16 priority,
58 u8 match_criteria_enable,
59 struct mlx5dr_match_parameters *mask);
60
61int mlx5dr_matcher_destroy(struct mlx5dr_matcher *matcher);
62
63struct mlx5dr_rule *
64mlx5dr_rule_create(struct mlx5dr_matcher *matcher,
65 struct mlx5dr_match_parameters *value,
66 size_t num_actions,
67 struct mlx5dr_action *actions[]);
68
69int mlx5dr_rule_destroy(struct mlx5dr_rule *rule);
70
71int mlx5dr_table_set_miss_action(struct mlx5dr_table *tbl,
72 struct mlx5dr_action *action);
73
74struct mlx5dr_action *
75mlx5dr_action_create_dest_table(struct mlx5dr_table *table);
76
77struct mlx5dr_action *
78mlx5dr_create_action_dest_flow_fw_table(struct mlx5_flow_table *ft,
79 struct mlx5_core_dev *mdev);
80
81struct mlx5dr_action *
82mlx5dr_action_create_dest_vport(struct mlx5dr_domain *domain,
83 u32 vport, u8 vhca_id_valid,
84 u16 vhca_id);
85
86struct mlx5dr_action *mlx5dr_action_create_drop(void);
87
88struct mlx5dr_action *mlx5dr_action_create_tag(u32 tag_value);
89
90struct mlx5dr_action *
91mlx5dr_action_create_flow_counter(u32 counter_id);
92
93struct mlx5dr_action *
94mlx5dr_action_create_packet_reformat(struct mlx5dr_domain *dmn,
95 enum mlx5dr_action_reformat_type reformat_type,
96 size_t data_sz,
97 void *data);
98
99struct mlx5dr_action *
100mlx5dr_action_create_modify_header(struct mlx5dr_domain *domain,
101 u32 flags,
102 size_t actions_sz,
103 __be64 actions[]);
104
105struct mlx5dr_action *mlx5dr_action_create_pop_vlan(void);
106
107struct mlx5dr_action *
108mlx5dr_action_create_push_vlan(struct mlx5dr_domain *domain, __be32 vlan_hdr);
109
110int mlx5dr_action_destroy(struct mlx5dr_action *action);
111
112static inline bool
113mlx5dr_is_supported(struct mlx5_core_dev *dev)
114{
115 return MLX5_CAP_ESW_FLOWTABLE_FDB(dev, sw_owner);
116}
117
118#else /* CONFIG_MLX5_SW_STEERING */
119
120static inline struct mlx5dr_domain *
121mlx5dr_domain_create(struct mlx5_core_dev *mdev, enum mlx5dr_domain_type type) { return NULL; }
122
123static inline int
124mlx5dr_domain_destroy(struct mlx5dr_domain *domain) { return 0; }
125
126static inline int
127mlx5dr_domain_sync(struct mlx5dr_domain *domain, u32 flags) { return 0; }
128
129static inline void
130mlx5dr_domain_set_peer(struct mlx5dr_domain *dmn,
131 struct mlx5dr_domain *peer_dmn) { }
132
133static inline struct mlx5dr_table *
134mlx5dr_table_create(struct mlx5dr_domain *domain, u32 level) { return NULL; }
135
136static inline int
137mlx5dr_table_destroy(struct mlx5dr_table *table) { return 0; }
138
139static inline u32
140mlx5dr_table_get_id(struct mlx5dr_table *table) { return 0; }
141
142static inline struct mlx5dr_matcher *
143mlx5dr_matcher_create(struct mlx5dr_table *table,
144 u16 priority,
145 u8 match_criteria_enable,
146 struct mlx5dr_match_parameters *mask) { return NULL; }
147
148static inline int
149mlx5dr_matcher_destroy(struct mlx5dr_matcher *matcher) { return 0; }
150
151static inline struct mlx5dr_rule *
152mlx5dr_rule_create(struct mlx5dr_matcher *matcher,
153 struct mlx5dr_match_parameters *value,
154 size_t num_actions,
155 struct mlx5dr_action *actions[]) { return NULL; }
156
157static inline int
158mlx5dr_rule_destroy(struct mlx5dr_rule *rule) { return 0; }
159
160static inline int
161mlx5dr_table_set_miss_action(struct mlx5dr_table *tbl,
162 struct mlx5dr_action *action) { return 0; }
163
164static inline struct mlx5dr_action *
165mlx5dr_action_create_dest_table(struct mlx5dr_table *table) { return NULL; }
166
167static inline struct mlx5dr_action *
168mlx5dr_create_action_dest_flow_fw_table(struct mlx5_flow_table *ft,
169 struct mlx5_core_dev *mdev) { return NULL; }
170
171static inline struct mlx5dr_action *
172mlx5dr_action_create_dest_vport(struct mlx5dr_domain *domain,
173 u32 vport, u8 vhca_id_valid,
174 u16 vhca_id) { return NULL; }
175
176static inline struct mlx5dr_action *
177mlx5dr_action_create_drop(void) { return NULL; }
178
179static inline struct mlx5dr_action *
180mlx5dr_action_create_tag(u32 tag_value) { return NULL; }
181
182static inline struct mlx5dr_action *
183mlx5dr_action_create_flow_counter(u32 counter_id) { return NULL; }
184
185static inline struct mlx5dr_action *
186mlx5dr_action_create_packet_reformat(struct mlx5dr_domain *dmn,
187 enum mlx5dr_action_reformat_type reformat_type,
188 size_t data_sz,
189 void *data) { return NULL; }
190
191static inline struct mlx5dr_action *
192mlx5dr_action_create_modify_header(struct mlx5dr_domain *domain,
193 u32 flags,
194 size_t actions_sz,
195 __be64 actions[]) { return NULL; }
196
197static inline struct mlx5dr_action *
198mlx5dr_action_create_pop_vlan(void) { return NULL; }
199
200static inline struct mlx5dr_action *
201mlx5dr_action_create_push_vlan(struct mlx5dr_domain *domain,
202 __be32 vlan_hdr) { return NULL; }
203
204static inline int
205mlx5dr_action_destroy(struct mlx5dr_action *action) { return 0; }
206
207static inline bool
208mlx5dr_is_supported(struct mlx5_core_dev *dev) { return false; }
209
210#endif /* CONFIG_MLX5_SW_STEERING */
211
212#endif /* _MLX5DR_H_ */
include/linux/mlx5/device.h+7−0
@@ -1162,6 +1162,9 @@ enum mlx5_qcam_feature_groups { · 1161 unmodified lines
1162#define MLX5_CAP_FLOWTABLE(mdev, cap) \1162#define MLX5_CAP_FLOWTABLE(mdev, cap) \
1163 MLX5_GET(flow_table_nic_cap, mdev->caps.hca_cur[MLX5_CAP_FLOW_TABLE], cap)1163 MLX5_GET(flow_table_nic_cap, mdev->caps.hca_cur[MLX5_CAP_FLOW_TABLE], cap)
1164 1164
1165#define MLX5_CAP64_FLOWTABLE(mdev, cap) \
1166 MLX5_GET64(flow_table_nic_cap, (mdev)->caps.hca_cur[MLX5_CAP_FLOW_TABLE], cap)
1167
1165#define MLX5_CAP_FLOWTABLE_MAX(mdev, cap) \1168#define MLX5_CAP_FLOWTABLE_MAX(mdev, cap) \
1166 MLX5_GET(flow_table_nic_cap, mdev->caps.hca_max[MLX5_CAP_FLOW_TABLE], cap)1169 MLX5_GET(flow_table_nic_cap, mdev->caps.hca_max[MLX5_CAP_FLOW_TABLE], cap)
1167 1170
@@ -1225,6 +1228,10 @@ enum mlx5_qcam_feature_groups { · 57 unmodified lines
1225 MLX5_GET(e_switch_cap, \1228 MLX5_GET(e_switch_cap, \
1226 mdev->caps.hca_cur[MLX5_CAP_ESWITCH], cap)1229 mdev->caps.hca_cur[MLX5_CAP_ESWITCH], cap)
1227 1230
1231#define MLX5_CAP64_ESW_FLOWTABLE(mdev, cap) \
1232 MLX5_GET64(flow_table_eswitch_cap, \
1233 (mdev)->caps.hca_cur[MLX5_CAP_ESWITCH_FLOW_TABLE], cap)
1234
1228#define MLX5_CAP_ESW_MAX(mdev, cap) \1235#define MLX5_CAP_ESW_MAX(mdev, cap) \
1229 MLX5_GET(e_switch_cap, \1236 MLX5_GET(e_switch_cap, \
1230 mdev->caps.hca_max[MLX5_CAP_ESWITCH], cap)1237 mdev->caps.hca_max[MLX5_CAP_ESWITCH], cap)
include/linux/mlx5/driver.h+14−0
@@ -626,6 +626,11 @@ struct mlx5e_resources { · 625 unmodified lines
626 struct mlx5_sq_bfreg bfreg;626 struct mlx5_sq_bfreg bfreg;
627};627};
628 628
629enum mlx5_sw_icm_type {
630 MLX5_SW_ICM_TYPE_STEERING,
631 MLX5_SW_ICM_TYPE_HEADER_MODIFY,
632};
633
629#define MLX5_MAX_RESERVED_GIDS 8634#define MLX5_MAX_RESERVED_GIDS 8
630 635
631struct mlx5_rsvd_gids {636struct mlx5_rsvd_gids {
@@ -657,11 +662,15 @@ struct mlx5_clock { · 25 unmodified lines
657 struct mlx5_pps pps_info;662 struct mlx5_pps pps_info;
658};663};
659 664
665struct mlx5_dm;
660struct mlx5_fw_tracer;666struct mlx5_fw_tracer;
661struct mlx5_vxlan;667struct mlx5_vxlan;
662struct mlx5_geneve;668struct mlx5_geneve;
663struct mlx5_hv_vhca;669struct mlx5_hv_vhca;
664 670
671#define MLX5_LOG_SW_ICM_BLOCK_SIZE(dev) (MLX5_CAP_DEV_MEM(dev, log_sw_icm_alloc_granularity))
672#define MLX5_SW_ICM_BLOCK_SIZE(dev) (1 << MLX5_LOG_SW_ICM_BLOCK_SIZE(dev))
673
665struct mlx5_core_dev {674struct mlx5_core_dev {
666 struct device *device;675 struct device *device;
667 enum mlx5_coredev_type coredev_type;676 enum mlx5_coredev_type coredev_type;
@@ -695,6 +704,7 @@ struct mlx5_core_dev { · 27 unmodified lines
695 atomic_t num_qps;704 atomic_t num_qps;
696 u32 issi;705 u32 issi;
697 struct mlx5e_resources mlx5e_res;706 struct mlx5e_resources mlx5e_res;
707 struct mlx5_dm *dm;
698 struct mlx5_vxlan *vxlan;708 struct mlx5_vxlan *vxlan;
699 struct mlx5_geneve *geneve;709 struct mlx5_geneve *geneve;
700 struct {710 struct {
@@ -1078,6 +1088,10 @@ int mlx5_lag_query_cong_counters(struct mlx5_core_dev *dev, · 377 unmodified lines
1078 size_t *offsets);1088 size_t *offsets);
1079struct mlx5_uars_page *mlx5_get_uars_page(struct mlx5_core_dev *mdev);1089struct mlx5_uars_page *mlx5_get_uars_page(struct mlx5_core_dev *mdev);
1080void mlx5_put_uars_page(struct mlx5_core_dev *mdev, struct mlx5_uars_page *up);1090void mlx5_put_uars_page(struct mlx5_core_dev *mdev, struct mlx5_uars_page *up);
1091int mlx5_dm_sw_icm_alloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type,
1092 u64 length, u16 uid, phys_addr_t *addr, u32 *obj_id);
1093int mlx5_dm_sw_icm_dealloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type,
1094 u64 length, u16 uid, phys_addr_t addr, u32 obj_id);
1081 1095
1082#ifdef CONFIG_MLX5_CORE_IPOIB1096#ifdef CONFIG_MLX5_CORE_IPOIB
1083struct net_device *mlx5_rdma_netdev_alloc(struct mlx5_core_dev *mdev,1097struct net_device *mlx5_rdma_netdev_alloc(struct mlx5_core_dev *mdev,
include/linux/mlx5/eswitch.h+7−1
@@ -60,7 +60,6 @@ void *mlx5_eswitch_get_proto_dev(struct mlx5_eswitch *esw, · 59 unmodified lines
60struct mlx5_eswitch_rep *mlx5_eswitch_vport_rep(struct mlx5_eswitch *esw,60struct mlx5_eswitch_rep *mlx5_eswitch_vport_rep(struct mlx5_eswitch *esw,
61 u16 vport_num);61 u16 vport_num);
62void *mlx5_eswitch_uplink_get_proto_dev(struct mlx5_eswitch *esw, u8 rep_type);62void *mlx5_eswitch_uplink_get_proto_dev(struct mlx5_eswitch *esw, u8 rep_type);
63u8 mlx5_eswitch_mode(struct mlx5_eswitch *esw);
64struct mlx5_flow_handle *63struct mlx5_flow_handle *
65mlx5_eswitch_add_send_to_vport_rule(struct mlx5_eswitch *esw,64mlx5_eswitch_add_send_to_vport_rule(struct mlx5_eswitch *esw,
66 u16 vport_num, u32 sqn);65 u16 vport_num, u32 sqn);
@@ -74,7 +73,14 @@ mlx5_eswitch_get_encap_mode(const struct mlx5_core_dev *dev); · 7 unmodified lines
74bool mlx5_eswitch_vport_match_metadata_enabled(const struct mlx5_eswitch *esw);73bool mlx5_eswitch_vport_match_metadata_enabled(const struct mlx5_eswitch *esw);
75u32 mlx5_eswitch_get_vport_metadata_for_match(const struct mlx5_eswitch *esw,74u32 mlx5_eswitch_get_vport_metadata_for_match(const struct mlx5_eswitch *esw,
76 u16 vport_num);75 u16 vport_num);
76u8 mlx5_eswitch_mode(struct mlx5_eswitch *esw);
77#else /* CONFIG_MLX5_ESWITCH */77#else /* CONFIG_MLX5_ESWITCH */
78
79static inline u8 mlx5_eswitch_mode(struct mlx5_eswitch *esw)
80{
81 return MLX5_ESWITCH_NONE;
82}
83
78static inline enum devlink_eswitch_encap_mode84static inline enum devlink_eswitch_encap_mode
79mlx5_eswitch_get_encap_mode(const struct mlx5_core_dev *dev)85mlx5_eswitch_get_encap_mode(const struct mlx5_core_dev *dev)
80{86{
include/linux/mlx5/fs.h+16−17
@@ -84,6 +84,8 @@ enum { · 83 unmodified lines
84 FDB_SLOW_PATH,84 FDB_SLOW_PATH,
85};85};
86 86
87struct mlx5_pkt_reformat;
88struct mlx5_modify_hdr;
87struct mlx5_flow_table;89struct mlx5_flow_table;
88struct mlx5_flow_group;90struct mlx5_flow_group;
89struct mlx5_flow_namespace;91struct mlx5_flow_namespace;
@@ -121,7 +123,7 @@ struct mlx5_flow_destination { · 31 unmodified lines
121 struct {123 struct {
122 u16 num;124 u16 num;
123 u16 vhca_id;125 u16 vhca_id;
124 u32 reformat_id;126 struct mlx5_pkt_reformat *pkt_reformat;
125 u8 flags;127 u8 flags;
126 } vport;128 } vport;
127 };129 };
@@ -195,8 +197,8 @@ enum { · 67 unmodified lines
195 197
196struct mlx5_flow_act {198struct mlx5_flow_act {
197 u32 action;199 u32 action;
198 u32 reformat_id;200 struct mlx5_modify_hdr *modify_hdr;
199 u32 modify_id;201 struct mlx5_pkt_reformat *pkt_reformat;
200 uintptr_t esp_id;202 uintptr_t esp_id;
201 u32 flags;203 u32 flags;
202 struct mlx5_fs_vlan vlan[MLX5_FS_VLAN_DEPTH];204 struct mlx5_fs_vlan vlan[MLX5_FS_VLAN_DEPTH];
@@ -205,8 +207,6 @@ struct mlx5_flow_act { · 2 unmodified lines
205 207
206#define MLX5_DECLARE_FLOW_ACT(name) \208#define MLX5_DECLARE_FLOW_ACT(name) \
207 struct mlx5_flow_act name = { .action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,\209 struct mlx5_flow_act name = { .action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,\
208 .reformat_id = 0, \
209 .modify_id = 0, \
210 .flags = 0, }210 .flags = 0, }
211 211
212/* Single destination per rule.212/* Single destination per rule.
@@ -236,19 +236,18 @@ u32 mlx5_fc_id(struct mlx5_fc *counter); · 23 unmodified lines
236int mlx5_fs_add_rx_underlay_qpn(struct mlx5_core_dev *dev, u32 underlay_qpn);236int mlx5_fs_add_rx_underlay_qpn(struct mlx5_core_dev *dev, u32 underlay_qpn);
237int mlx5_fs_remove_rx_underlay_qpn(struct mlx5_core_dev *dev, u32 underlay_qpn);237int mlx5_fs_remove_rx_underlay_qpn(struct mlx5_core_dev *dev, u32 underlay_qpn);
238 238
239int mlx5_modify_header_alloc(struct mlx5_core_dev *dev,239struct mlx5_modify_hdr *mlx5_modify_header_alloc(struct mlx5_core_dev *dev,
240 u8 namespace, u8 num_actions,240 u8 ns_type, u8 num_actions,
241 void *modify_actions, u32 *modify_header_id);241 void *modify_actions);
242void mlx5_modify_header_dealloc(struct mlx5_core_dev *dev,242void mlx5_modify_header_dealloc(struct mlx5_core_dev *dev,
243 u32 modify_header_id);243 struct mlx5_modify_hdr *modify_hdr);
244 244
245int mlx5_packet_reformat_alloc(struct mlx5_core_dev *dev,245struct mlx5_pkt_reformat *mlx5_packet_reformat_alloc(struct mlx5_core_dev *dev,
246 int reformat_type,246 int reformat_type,
247 size_t size,247 size_t size,
248 void *reformat_data,248 void *reformat_data,
249 enum mlx5_flow_namespace_type namespace,249 enum mlx5_flow_namespace_type ns_type);
250 u32 *packet_reformat_id);
251void mlx5_packet_reformat_dealloc(struct mlx5_core_dev *dev,250void mlx5_packet_reformat_dealloc(struct mlx5_core_dev *dev,
252 u32 packet_reformat_id);251 struct mlx5_pkt_reformat *reformat);
253 252
254#endif253#endif
include/linux/mlx5/mlx5_ifc.h+198−37
@@ -282,6 +282,7 @@ enum { · 281 unmodified lines
282 MLX5_CMD_OP_ALLOC_MODIFY_HEADER_CONTEXT = 0x940,282 MLX5_CMD_OP_ALLOC_MODIFY_HEADER_CONTEXT = 0x940,
283 MLX5_CMD_OP_DEALLOC_MODIFY_HEADER_CONTEXT = 0x941,283 MLX5_CMD_OP_DEALLOC_MODIFY_HEADER_CONTEXT = 0x941,
284 MLX5_CMD_OP_QUERY_MODIFY_HEADER_CONTEXT = 0x942,284 MLX5_CMD_OP_QUERY_MODIFY_HEADER_CONTEXT = 0x942,
285 MLX5_CMD_OP_SYNC_STEERING = 0xb00,
285 MLX5_CMD_OP_FPGA_CREATE_QP = 0x960,286 MLX5_CMD_OP_FPGA_CREATE_QP = 0x960,
286 MLX5_CMD_OP_FPGA_MODIFY_QP = 0x961,287 MLX5_CMD_OP_FPGA_MODIFY_QP = 0x961,
287 MLX5_CMD_OP_FPGA_QUERY_QP = 0x962,288 MLX5_CMD_OP_FPGA_QUERY_QP = 0x962,
@@ -485,7 +486,11 @@ union mlx5_ifc_gre_key_bits { · 197 unmodified lines
485};486};
486 487
487struct mlx5_ifc_fte_match_set_misc_bits {488struct mlx5_ifc_fte_match_set_misc_bits {
488 u8 reserved_at_0[0x8];489 u8 gre_c_present[0x1];
490 u8 reserved_auto1[0x1];
491 u8 gre_k_present[0x1];
492 u8 gre_s_present[0x1];
493 u8 source_vhca_port[0x4];
489 u8 source_sqn[0x18];494 u8 source_sqn[0x18];
490 495
491 u8 source_eswitch_owner_vhca_id[0x10];496 u8 source_eswitch_owner_vhca_id[0x10];
@@ -565,12 +570,38 @@ struct mlx5_ifc_fte_match_set_misc2_bits { · 73 unmodified lines
565 570
566 u8 metadata_reg_a[0x20];571 u8 metadata_reg_a[0x20];
567 572
568 u8 reserved_at_1a0[0x60];573 u8 metadata_reg_b[0x20];
574
575 u8 reserved_at_1c0[0x40];
569};576};
570 577
571struct mlx5_ifc_fte_match_set_misc3_bits {578struct mlx5_ifc_fte_match_set_misc3_bits {
572 u8 reserved_at_0[0x120];579 u8 inner_tcp_seq_num[0x20];
580
581 u8 outer_tcp_seq_num[0x20];
582
583 u8 inner_tcp_ack_num[0x20];
584
585 u8 outer_tcp_ack_num[0x20];
586
587 u8 reserved_at_80[0x8];
588 u8 outer_vxlan_gpe_vni[0x18];
589
590 u8 outer_vxlan_gpe_next_protocol[0x8];
591 u8 outer_vxlan_gpe_flags[0x8];
592 u8 reserved_at_b0[0x10];
593
594 u8 icmp_header_data[0x20];
595
596 u8 icmpv6_header_data[0x20];
597
598 u8 icmp_type[0x8];
599 u8 icmp_code[0x8];
600 u8 icmpv6_type[0x8];
601 u8 icmpv6_code[0x8];
602
573 u8 geneve_tlv_option_0_data[0x20];603 u8 geneve_tlv_option_0_data[0x20];
604
574 u8 reserved_at_140[0xc0];605 u8 reserved_at_140[0xc0];
575};606};
576 607
@@ -666,7 +697,15 @@ struct mlx5_ifc_flow_table_nic_cap_bits { · 89 unmodified lines
666 697
667 struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_nic_transmit_sniffer;698 struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_nic_transmit_sniffer;
668 699
669 u8 reserved_at_e00[0x7200];700 u8 reserved_at_e00[0x1200];
701
702 u8 sw_steering_nic_rx_action_drop_icm_address[0x40];
703
704 u8 sw_steering_nic_tx_action_drop_icm_address[0x40];
705
706 u8 sw_steering_nic_tx_action_allow_icm_address[0x40];
707
708 u8 reserved_at_20c0[0x5f40];
670};709};
671 710
672enum {711enum {
@@ -698,7 +737,17 @@ struct mlx5_ifc_flow_table_eswitch_cap_bits { · 25 unmodified lines
698 737
699 struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_esw_acl_egress;738 struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_esw_acl_egress;
700 739
701 u8 reserved_at_800[0x7800];740 u8 reserved_at_800[0x1000];
741
742 u8 sw_steering_fdb_action_drop_icm_address_rx[0x40];
743
744 u8 sw_steering_fdb_action_drop_icm_address_tx[0x40];
745
746 u8 sw_steering_uplink_icm_address_rx[0x40];
747
748 u8 sw_steering_uplink_icm_address_tx[0x40];
749
750 u8 reserved_at_1900[0x6700];
702};751};
703 752
704enum {753enum {
@@ -849,6 +898,25 @@ struct mlx5_ifc_roce_cap_bits { · 144 unmodified lines
849 u8 reserved_at_100[0x700];898 u8 reserved_at_100[0x700];
850};899};
851 900
901struct mlx5_ifc_sync_steering_in_bits {
902 u8 opcode[0x10];
903 u8 uid[0x10];
904
905 u8 reserved_at_20[0x10];
906 u8 op_mod[0x10];
907
908 u8 reserved_at_40[0xc0];
909};
910
911struct mlx5_ifc_sync_steering_out_bits {
912 u8 status[0x8];
913 u8 reserved_at_8[0x18];
914
915 u8 syndrome[0x20];
916
917 u8 reserved_at_40[0x40];
918};
919
852struct mlx5_ifc_device_mem_cap_bits {920struct mlx5_ifc_device_mem_cap_bits {
853 u8 memic[0x1];921 u8 memic[0x1];
854 u8 reserved_at_1[0x1f];922 u8 reserved_at_1[0x1f];
@@ -1041,6 +1109,12 @@ enum { · 186 unmodified lines
1041 MLX5_CAP_UMR_FENCE_NONE = 0x2,1109 MLX5_CAP_UMR_FENCE_NONE = 0x2,
1042};1110};
1043 1111
1112enum {
1113 MLX5_FLEX_PARSER_VXLAN_GPE_ENABLED = 1 << 7,
1114 MLX5_FLEX_PARSER_ICMP_V4_ENABLED = 1 << 8,
1115 MLX5_FLEX_PARSER_ICMP_V6_ENABLED = 1 << 9,
1116};
1117
1044enum {1118enum {
1045 MLX5_UCTX_CAP_RAW_TX = 1UL << 0,1119 MLX5_UCTX_CAP_RAW_TX = 1UL << 0,
1046 MLX5_UCTX_CAP_INTERNAL_DEV_RES = 1UL << 1,1120 MLX5_UCTX_CAP_INTERNAL_DEV_RES = 1UL << 1,
@@ -1414,7 +1488,14 @@ struct mlx5_ifc_cmd_hca_cap_bits { · 367 unmodified lines
1414 1488
1415 u8 reserved_at_6c0[0x4];1489 u8 reserved_at_6c0[0x4];
1416 u8 flex_parser_id_geneve_tlv_option_0[0x4];1490 u8 flex_parser_id_geneve_tlv_option_0[0x4];
1417 u8 reserved_at_6c8[0x28];1491 u8 flex_parser_id_icmp_dw1[0x4];
1492 u8 flex_parser_id_icmp_dw0[0x4];
1493 u8 flex_parser_id_icmpv6_dw1[0x4];
1494 u8 flex_parser_id_icmpv6_dw0[0x4];
1495 u8 flex_parser_id_outer_first_mpls_over_gre[0x4];
1496 u8 flex_parser_id_outer_first_mpls_over_udp_label[0x4];
1497
1498 u8 reserved_at_6e0[0x10];
1418 u8 sf_base_id[0x10];1499 u8 sf_base_id[0x10];
1419 1500
1420 u8 reserved_at_700[0x80];1501 u8 reserved_at_700[0x80];
@@ -2652,6 +2733,7 @@ union mlx5_ifc_hca_cap_union_bits { · 1231 unmodified lines
2652 struct mlx5_ifc_debug_cap_bits debug_cap;2733 struct mlx5_ifc_debug_cap_bits debug_cap;
2653 struct mlx5_ifc_fpga_cap_bits fpga_cap;2734 struct mlx5_ifc_fpga_cap_bits fpga_cap;
2654 struct mlx5_ifc_tls_cap_bits tls_cap;2735 struct mlx5_ifc_tls_cap_bits tls_cap;
2736 struct mlx5_ifc_device_mem_cap_bits device_mem_cap;
2655 u8 reserved_at_0[0x8000];2737 u8 reserved_at_0[0x8000];
2656};2738};
2657 2739
@@ -3255,7 +3337,11 @@ struct mlx5_ifc_esw_vport_context_bits { · 597 unmodified lines
3255 u8 cvlan_pcp[0x3];3337 u8 cvlan_pcp[0x3];
3256 u8 cvlan_id[0xc];3338 u8 cvlan_id[0xc];
3257 3339
3258 u8 reserved_at_60[0x7a0];3340 u8 reserved_at_60[0x720];
3341
3342 u8 sw_steering_vport_icm_address_rx[0x40];
3343
3344 u8 sw_steering_vport_icm_address_tx[0x40];
3259};3345};
3260 3346
3261enum {3347enum {
@@ -4941,23 +5027,98 @@ struct mlx5_ifc_query_hca_cap_in_bits { · 1679 unmodified lines
4941 u8 reserved_at_20[0x10];5027 u8 reserved_at_20[0x10];
4942 u8 op_mod[0x10];5028 u8 op_mod[0x10];
4943 5029
4944 u8 reserved_at_40[0x40];5030 u8 other_function[0x1];
5031 u8 reserved_at_41[0xf];
5032 u8 function_id[0x10];
5033
5034 u8 reserved_at_60[0x20];
4945};5035};
4946 5036
4947struct mlx5_ifc_query_flow_table_out_bits {5037struct mlx5_ifc_other_hca_cap_bits {
5038 u8 roce[0x1];
5039 u8 reserved_0[0x27f];
5040};
5041
5042struct mlx5_ifc_query_other_hca_cap_out_bits {
4948 u8 status[0x8];5043 u8 status[0x8];
4949 u8 reserved_at_8[0x18];5044 u8 reserved_0[0x18];
4950 5045
4951 u8 syndrome[0x20];5046 u8 syndrome[0x20];
4952 5047
4953 u8 reserved_at_40[0x80];5048 u8 reserved_1[0x40];
4954 5049
4955 u8 reserved_at_c0[0x8];5050 struct mlx5_ifc_other_hca_cap_bits other_capability;
5051};
5052
5053struct mlx5_ifc_query_other_hca_cap_in_bits {
5054 u8 opcode[0x10];
5055 u8 reserved_0[0x10];
5056
5057 u8 reserved_1[0x10];
5058 u8 op_mod[0x10];
5059
5060 u8 reserved_2[0x10];
5061 u8 function_id[0x10];
5062
5063 u8 reserved_3[0x20];
5064};
5065
5066struct mlx5_ifc_modify_other_hca_cap_out_bits {
5067 u8 status[0x8];
5068 u8 reserved_0[0x18];
5069
5070 u8 syndrome[0x20];
5071
5072 u8 reserved_1[0x40];
5073};
5074
5075struct mlx5_ifc_modify_other_hca_cap_in_bits {
5076 u8 opcode[0x10];
5077 u8 reserved_0[0x10];
5078
5079 u8 reserved_1[0x10];
5080 u8 op_mod[0x10];
5081
5082 u8 reserved_2[0x10];
5083 u8 function_id[0x10];
5084 u8 field_select[0x20];
5085
5086 struct mlx5_ifc_other_hca_cap_bits other_capability;
5087};
5088
5089struct mlx5_ifc_flow_table_context_bits {
5090 u8 reformat_en[0x1];
5091 u8 decap_en[0x1];
5092 u8 sw_owner[0x1];
5093 u8 termination_table[0x1];
5094 u8 table_miss_action[0x4];
4956 u8 level[0x8];5095 u8 level[0x8];
4957 u8 reserved_at_d0[0x8];5096 u8 reserved_at_10[0x8];
4958 u8 log_size[0x8];5097 u8 log_size[0x8];
4959 5098
4960 u8 reserved_at_e0[0x120];5099 u8 reserved_at_20[0x8];
5100 u8 table_miss_id[0x18];
5101
5102 u8 reserved_at_40[0x8];
5103 u8 lag_master_next_table_id[0x18];
5104
5105 u8 reserved_at_60[0x60];
5106
5107 u8 sw_owner_icm_root_1[0x40];
5108
5109 u8 sw_owner_icm_root_0[0x40];
5110
5111};
5112
5113struct mlx5_ifc_query_flow_table_out_bits {
5114 u8 status[0x8];
5115 u8 reserved_at_8[0x18];
5116
5117 u8 syndrome[0x20];
5118
5119 u8 reserved_at_40[0x80];
5120
5121 struct mlx5_ifc_flow_table_context_bits flow_table_context;
4961};5122};
4962 5123
4963struct mlx5_ifc_query_flow_table_in_bits {5124struct mlx5_ifc_query_flow_table_in_bits {
@@ -5227,7 +5388,7 @@ struct mlx5_ifc_alloc_packet_reformat_context_out_bits { · 263 unmodified lines
5227 u8 reserved_at_60[0x20];5388 u8 reserved_at_60[0x20];
5228};5389};
5229 5390
5230enum {5391enum mlx5_reformat_ctx_type {
5231 MLX5_REFORMAT_TYPE_L2_TO_VXLAN = 0x0,5392 MLX5_REFORMAT_TYPE_L2_TO_VXLAN = 0x0,
5232 MLX5_REFORMAT_TYPE_L2_TO_NVGRE = 0x1,5393 MLX5_REFORMAT_TYPE_L2_TO_NVGRE = 0x1,
5233 MLX5_REFORMAT_TYPE_L2_TO_L2_TUNNEL = 0x2,5394 MLX5_REFORMAT_TYPE_L2_TO_L2_TUNNEL = 0x2,
@@ -5323,7 +5484,16 @@ enum { · 89 unmodified lines
5323 MLX5_ACTION_IN_FIELD_OUT_DIPV4 = 0x16,5484 MLX5_ACTION_IN_FIELD_OUT_DIPV4 = 0x16,
5324 MLX5_ACTION_IN_FIELD_OUT_FIRST_VID = 0x17,5485 MLX5_ACTION_IN_FIELD_OUT_FIRST_VID = 0x17,
5325 MLX5_ACTION_IN_FIELD_OUT_IPV6_HOPLIMIT = 0x47,5486 MLX5_ACTION_IN_FIELD_OUT_IPV6_HOPLIMIT = 0x47,
5487 MLX5_ACTION_IN_FIELD_METADATA_REG_A = 0x49,
5488 MLX5_ACTION_IN_FIELD_METADATA_REG_B = 0x50,
5326 MLX5_ACTION_IN_FIELD_METADATA_REG_C_0 = 0x51,5489 MLX5_ACTION_IN_FIELD_METADATA_REG_C_0 = 0x51,
5490 MLX5_ACTION_IN_FIELD_METADATA_REG_C_1 = 0x52,
5491 MLX5_ACTION_IN_FIELD_METADATA_REG_C_2 = 0x53,
5492 MLX5_ACTION_IN_FIELD_METADATA_REG_C_3 = 0x54,
5493 MLX5_ACTION_IN_FIELD_METADATA_REG_C_4 = 0x55,
5494 MLX5_ACTION_IN_FIELD_METADATA_REG_C_5 = 0x56,
5495 MLX5_ACTION_IN_FIELD_OUT_TCP_SEQ_NUM = 0x59,
5496 MLX5_ACTION_IN_FIELD_OUT_TCP_ACK_NUM = 0x5B,
5327};5497};
5328 5498
5329struct mlx5_ifc_alloc_modify_header_context_out_bits {5499struct mlx5_ifc_alloc_modify_header_context_out_bits {
@@ -7371,35 +7541,26 @@ struct mlx5_ifc_create_mkey_in_bits { · 2041 unmodified lines
7371 u8 klm_pas_mtt[0][0x20];7541 u8 klm_pas_mtt[0][0x20];
7372};7542};
7373 7543
7544enum {
7545 MLX5_FLOW_TABLE_TYPE_NIC_RX = 0x0,
7546 MLX5_FLOW_TABLE_TYPE_NIC_TX = 0x1,
7547 MLX5_FLOW_TABLE_TYPE_ESW_EGRESS_ACL = 0x2,
7548 MLX5_FLOW_TABLE_TYPE_ESW_INGRESS_ACL = 0x3,
7549 MLX5_FLOW_TABLE_TYPE_FDB = 0X4,
7550 MLX5_FLOW_TABLE_TYPE_SNIFFER_RX = 0X5,
7551 MLX5_FLOW_TABLE_TYPE_SNIFFER_TX = 0X6,
7552};
7553
7374struct mlx5_ifc_create_flow_table_out_bits {7554struct mlx5_ifc_create_flow_table_out_bits {
7375 u8 status[0x8];7555 u8 status[0x8];
7376 u8 reserved_at_8[0x18];7556 u8 icm_address_63_40[0x18];
7377 7557
7378 u8 syndrome[0x20];7558 u8 syndrome[0x20];
7379 7559
7380 u8 reserved_at_40[0x8];7560 u8 icm_address_39_32[0x8];
7381 u8 table_id[0x18];7561 u8 table_id[0x18];
7382 7562
7383 u8 reserved_at_60[0x20];7563 u8 icm_address_31_0[0x20];
7384};
7385
7386struct mlx5_ifc_flow_table_context_bits {
7387 u8 reformat_en[0x1];
7388 u8 decap_en[0x1];
7389 u8 reserved_at_2[0x1];
7390 u8 termination_table[0x1];
7391 u8 table_miss_action[0x4];
7392 u8 level[0x8];
7393 u8 reserved_at_10[0x8];
7394 u8 log_size[0x8];
7395
7396 u8 reserved_at_20[0x8];
7397 u8 table_miss_id[0x18];
7398
7399 u8 reserved_at_40[0x8];
7400 u8 lag_master_next_table_id[0x18];
7401
7402 u8 reserved_at_60[0xe0];
7403};7564};
7404 7565
7405struct mlx5_ifc_create_flow_table_in_bits {7566struct mlx5_ifc_create_flow_table_in_bits {

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t