History

Merge tag 'mlx5-updates-2019-09-01-v2' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux

Saeed Mahameed says: ==================== mlx5-updates-2019-09-01 (Software steering support) Abstract: -------- Mellanox ConnetX devices supports packet matching, packet modification and redirection. These functionalities are also referred to as flow-steering. To configure a steering rule, the rule is written to the device owned memory, this memory is accessed and cached by the device when processing a packet. Steering rules are constructed from multiple steering entries (STE). Rules are configured using the Firmware command interface. The Firmware processes the given driver command and translates them to STEs, then writes them to the device memory in the current steering tables. This process is slow due to the architecture of the command interface and the processing complexity of each rule. The highlight of this patchset is to cut the middle man (The firmware) and do steering rules programming into device directly from the driver, with no firmware intervention whatsoever. Motivation: ----------- Software (driver managed) steering allows for high rule insertion rates compared to the FW steering described above, this is achieved by using internal RDMA writes to the device owned memory instead of the slow command interface to program steering rules. Software (driver managed) steering, doesn't depend on new FW for new steering functionality, new implementations can be done in the driver skipping the FW layer. Performance: ------------ The insertion rate on a single core using the new approach allows programming ~300K rules per sec. (Done via direct raw test to the new mlx5 sw steering layer, without any kernel layer involved). Test: TC L2 rules 33K/s with Software steering (this patchset). 5K/s with FW and current driver. This will improve OVS based solution performance. Architecture and implementation details: ---------------------------------------- Software steering will be dynamically selected via devlink device parameter. Example: $ devlink dev param show pci/0000:06:00.0 name flow_steering_mode pci/0000:06:00.0: name flow_steering_mode type driver-specific values: cmode runtime value smfs mlx5 software steering module a.k.a (DR - Direct Rule) is implemented and contained in mlx5/core/steering directory and controlled by MLX5_SW_STEERING kconfig flag. mlx5 core steering layer (fs_core) already provides a shim layer for implementing different steering mechanisms, software steering will leverage that as seen at the end of this series. When Software Steering for a specific steering domain (NIC/RDMA/Vport/ESwitch, etc ..) is supported, it will cause rules targeting this domain to be created using SW steering instead of FW. The implementation includes: Domain - The steering domain is the object that all other object resides in. It holds the memory allocator, send engine, locks and other shared data needed by lower objects such as table, matcher, rule, action. Each domain can contain multiple tables. Domain is equivalent to namespaces e.g (NIC/RDMA/Vport/ESwitch, etc ..) as implemented currently in mlx5_core fs_core (flow steering core). Table - Table objects are used for holding multiple matchers, each table has a level used to prevent processing loops. Packets are being directed to this table once it is set as the root table, this is done by fs_core using a FW command. A packet is being processed inside the table matcher by matcher until a successful hit, otherwise the packet will perform the default action. Matcher - Matchers objects are used to specify the fields mask for matching when processing a packet. A matcher belongs to a table, each matcher can hold multiple rules, each rule with different matching values corresponding to the matcher mask. Each matcher has a priority used for rule processing order inside the table. Action - Action objects are created to specify different steering actions such as count, reformat (encapsulate, decapsulate, ...), modify header, forward to table and many other actions. When creating a rule a sequence of actions can be provided to be executed on a successful match. Rule - Rule objects are used to specify a specific match on packets as well as the actions that should be executed. A rule belongs to a matcher. STE - This layer is used to hold the specific STE format for the device and to convert the requested rule to STEs. Each rule is constructed of an STE chain, Multiple rules construct a steering graph. Each node in the graph is a hash table containing multiple STEs. The index of each STE in the hash table is being calculated using a CRC32 hash function. Memory pool - Used for managing and caching device owned memory for rule insertion. The memory is being allocated using DM (device memory) API. Communication with device - layer for standard RDMA operation using RC QP to configure the device steering. Command utility - This module holds all of the FW commands that are required for SW steering to function. Patch planning and files: ------------------------- 1) First patch, adds the support to Add flow steering actions to fs_cmd shim layer. 2) Next 12 patch will add a file per each Software steering functionality/module as described above. (See patches with title: DR, *) 3) Add CONFIG_MLX5_SW_STEERING for software steering support and enable build with the new files 4) Next two patches will add the support for software steering in mlx5 steering shim layer net/mlx5: Add API to set the namespace steering mode net/mlx5: Add direct rule fs_cmd implementation 5) Last two patches will add the new devlink parameter to select mlx5 steering mode, will be valid only for switchdev mode for now. Two modes are supported: 1. DMFS - Device managed flow steering 2. SMFS - Software/Driver managed flow steering. In the DMFS mode, the HW steering entities are created through the FW. In the SMFS mode this entities are created though the driver directly. The driver will use the devlink steering mode only if the steering domain supports it, for now SMFS will manages only the switchdev eswitch steering domain. User command examples: - Set SMFS flow steering mode:: $ devlink dev param set pci/0000:06:00.0 name flow_steering_mode value "smfs" cmode runtime - Read device flow steering mode:: $ devlink dev param show pci/0000:06:00.0 name flow_steering_mode pci/0000:06:00.0: name flow_steering_mode type driver-specific values: cmode runtime value smfs ====================

David S. Miller committed 7 years ago

Commit 94810bd2 parents 67538eb e890acd44 files · +12464 −371Browse files

Unified · Split

44 files
  1. Documentation/networking/device_drivers/mellanox/mlx5.rst +33 −0
  2. drivers/infiniband/hw/mlx5/cmd.c +0 −130
  3. drivers/infiniband/hw/mlx5/cmd.h +0 −4
  4. drivers/infiniband/hw/mlx5/flow.c +12 −9
  5. drivers/infiniband/hw/mlx5/main.c +34 −75
  6. drivers/infiniband/hw/mlx5/mlx5_ib.h +4 −3
  7. drivers/net/ethernet/mellanox/mlx5/core/Kconfig +7 −0
  8. drivers/net/ethernet/mellanox/mlx5/core/Makefile +8 −1
  9. drivers/net/ethernet/mellanox/mlx5/core/devlink.c +111 −1
  10. drivers/net/ethernet/mellanox/mlx5/core/en/tc_tun.c +14 −13
  11. drivers/net/ethernet/mellanox/mlx5/core/en_rep.h +1 −1
  12. drivers/net/ethernet/mellanox/mlx5/core/en_tc.c +26 −25
  13. drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +4 −3
  14. drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +68 −19
  15. drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c +89 −27
  16. drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.h +25 −0
  17. drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +159 −1
  18. drivers/net/ethernet/mellanox/mlx5/core/fs_core.h +38 −1
  19. drivers/net/ethernet/mellanox/mlx5/core/lib/dm.c +223 −0
  20. drivers/net/ethernet/mellanox/mlx5/core/main.c +5 −0
  21. drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +3 −0
  22. drivers/net/ethernet/mellanox/mlx5/core/rdma.c +5 −3
  23. drivers/net/ethernet/mellanox/mlx5/core/steering/Makefile +2 −0
  24. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_action.c +1588 −0
  25. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_cmd.c +480 −0
  26. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_crc32.c +98 −0
  27. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_domain.c +395 −0
  28. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_fw.c +93 −0
  29. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_icm_pool.c +570 −0
  30. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_matcher.c +770 −0
  31. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_rule.c +1243 −0
  32. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c +976 −0
  33. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste.c +2308 −0
  34. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_table.c +294 −0
  35. drivers/net/ethernet/mellanox/mlx5/core/steering/dr_types.h +1060 −0
  36. drivers/net/ethernet/mellanox/mlx5/core/steering/fs_dr.c +600 −0
  37. drivers/net/ethernet/mellanox/mlx5/core/steering/fs_dr.h +60 −0
  38. drivers/net/ethernet/mellanox/mlx5/core/steering/mlx5_ifc_dr.h +604 −0
  39. drivers/net/ethernet/mellanox/mlx5/core/steering/mlx5dr.h +212 −0
  40. include/linux/mlx5/device.h +7 −0
  41. include/linux/mlx5/driver.h +14 −0
  42. include/linux/mlx5/eswitch.h +7 −1
  43. include/linux/mlx5/fs.h +16 −17
  44. include/linux/mlx5/mlx5_ifc.h +198 −37
Documentation/networking/device_drivers/mellanox/mlx5.rst+33−0
@@ -11,6 +11,7 @@ Contents · 10 unmodified lines
1111
1212 - `Enabling the driver and kconfig options`_
1313 - `Devlink info`_
14+- `Devlink parameters`_
1415 - `Devlink health reporters`_
1516 - `mlx5 tracepoints`_
1617
@@ -122,6 +123,38 @@ User command example:: · 105 unmodified lines
122123 stored:
123124 fw.version 16.26.0100
124125
126+Devlink parameters
127+==================
128+
129+flow_steering_mode: Device flow steering mode
130+---------------------------------------------
131+The flow steering mode parameter controls the flow steering mode of the driver.
132+Two modes are supported:
133+1. 'dmfs' - Device managed flow steering.
134+2. 'smfs - Software/Driver managed flow steering.
135+
136+In DMFS mode, the HW steering entities are created and managed through the
137+Firmware.
138+In SMFS mode, the HW steering entities are created and managed though by
139+the driver directly into Hardware without firmware intervention.
140+
141+SMFS mode is faster and provides better rule inserstion rate compared to default DMFS mode.
142+
143+User command examples:
144+
145+- Set SMFS flow steering mode::
146+
147+ $ devlink dev param set pci/0000:06:00.0 name flow_steering_mode value "smfs" cmode runtime
148+
149+- Read device flow steering mode::
150+
151+ $ devlink dev param show pci/0000:06:00.0 name flow_steering_mode
152+ pci/0000:06:00.0:
153+ name flow_steering_mode type driver-specific
154+ values:
155+ cmode runtime value smfs
156+
157+
125158 Devlink health reporters
126159 ========================
127160
drivers/infiniband/hw/mlx5/cmd.c+0−130
@@ -186,136 +186,6 @@ int mlx5_cmd_dealloc_memic(struct mlx5_dm *dm, phys_addr_t addr, u64 length) · 185 unmodified lines
186186 return err;
187187 }
188188
189−int mlx5_cmd_alloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
190− u16 uid, phys_addr_t *addr, u32 *obj_id)
191−{
192− struct mlx5_core_dev *dev = dm->dev;
193− u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {};
194− u32 in[MLX5_ST_SZ_DW(create_sw_icm_in)] = {};
195− unsigned long *block_map;
196− u64 icm_start_addr;
197− u32 log_icm_size;
198− u32 num_blocks;
199− u32 max_blocks;
200− u64 block_idx;
201− void *sw_icm;
202− int ret;
203−
204− MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
205− MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
206− MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, MLX5_OBJ_TYPE_SW_ICM);
207− MLX5_SET(general_obj_in_cmd_hdr, in, uid, uid);
208−
209− switch (type) {
210− case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
211− icm_start_addr = MLX5_CAP64_DEV_MEM(dev,
212− steering_sw_icm_start_address);
213− log_icm_size = MLX5_CAP_DEV_MEM(dev, log_steering_sw_icm_size);
214− block_map = dm->steering_sw_icm_alloc_blocks;
215− break;
216− case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
217− icm_start_addr = MLX5_CAP64_DEV_MEM(dev,
218− header_modify_sw_icm_start_address);
219− log_icm_size = MLX5_CAP_DEV_MEM(dev,
220− log_header_modify_sw_icm_size);
221− block_map = dm->header_modify_sw_icm_alloc_blocks;
222− break;
223− default:
224− return -EINVAL;
225− }
226−
227− num_blocks = (length + MLX5_SW_ICM_BLOCK_SIZE(dev) - 1) >>
228− MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
229− max_blocks = BIT(log_icm_size - MLX5_LOG_SW_ICM_BLOCK_SIZE(dev));
230− spin_lock(&dm->lock);
231− block_idx = bitmap_find_next_zero_area(block_map,
232− max_blocks,
233− 0,
234− num_blocks, 0);
235−
236− if (block_idx < max_blocks)
237− bitmap_set(block_map,
238− block_idx, num_blocks);
239−
240− spin_unlock(&dm->lock);
241−
242− if (block_idx >= max_blocks)
243− return -ENOMEM;
244−
245− sw_icm = MLX5_ADDR_OF(create_sw_icm_in, in, sw_icm);
246− icm_start_addr += block_idx << MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
247− MLX5_SET64(sw_icm, sw_icm, sw_icm_start_addr,
248− icm_start_addr);
249− MLX5_SET(sw_icm, sw_icm, log_sw_icm_size, ilog2(length));
250−
251− ret = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
252− if (ret) {
253− spin_lock(&dm->lock);
254− bitmap_clear(block_map,
255− block_idx, num_blocks);
256− spin_unlock(&dm->lock);
257−
258− return ret;
259− }
260−
261− *addr = icm_start_addr;
262− *obj_id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
263−
264− return 0;
265−}
266−
267−int mlx5_cmd_dealloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
268− u16 uid, phys_addr_t addr, u32 obj_id)
269−{
270− struct mlx5_core_dev *dev = dm->dev;
271− u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {};
272− u32 in[MLX5_ST_SZ_DW(general_obj_in_cmd_hdr)] = {};
273− unsigned long *block_map;
274− u32 num_blocks;
275− u64 start_idx;
276− int err;
277−
278− num_blocks = (length + MLX5_SW_ICM_BLOCK_SIZE(dev) - 1) >>
279− MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
280−
281− switch (type) {
282− case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
283− start_idx =
284− (addr - MLX5_CAP64_DEV_MEM(
285− dev, steering_sw_icm_start_address)) >>
286− MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
287− block_map = dm->steering_sw_icm_alloc_blocks;
288− break;
289− case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
290− start_idx =
291− (addr -
292− MLX5_CAP64_DEV_MEM(
293− dev, header_modify_sw_icm_start_address)) >>
294− MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
295− block_map = dm->header_modify_sw_icm_alloc_blocks;
296− break;
297− default:
298− return -EINVAL;
299− }
300−
301− MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
302− MLX5_CMD_OP_DESTROY_GENERAL_OBJECT);
303− MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, MLX5_OBJ_TYPE_SW_ICM);
304− MLX5_SET(general_obj_in_cmd_hdr, in, obj_id, obj_id);
305− MLX5_SET(general_obj_in_cmd_hdr, in, uid, uid);
306−
307− err = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
308− if (err)
309− return err;
310−
311− spin_lock(&dm->lock);
312− bitmap_clear(block_map,
313− start_idx, num_blocks);
314− spin_unlock(&dm->lock);
315−
316− return 0;
317−}
318−
319189 int mlx5_cmd_query_ext_ppcnt_counters(struct mlx5_core_dev *dev, void *out)
320190 {
321191 u32 in[MLX5_ST_SZ_DW(ppcnt_reg)] = {};
drivers/infiniband/hw/mlx5/cmd.h+0−4
@@ -65,8 +65,4 @@ int mlx5_cmd_alloc_q_counter(struct mlx5_core_dev *dev, u16 *counter_id, · 64 unmodified lines
6565 u16 uid);
6666 int mlx5_cmd_mad_ifc(struct mlx5_core_dev *dev, const void *inb, void *outb,
6767 u16 opmod, u8 port);
68−int mlx5_cmd_alloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
69− u16 uid, phys_addr_t *addr, u32 *obj_id);
70−int mlx5_cmd_dealloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
71− u16 uid, phys_addr_t addr, u32 obj_id);
7268 #endif /* MLX5_IB_CMD_H */
drivers/infiniband/hw/mlx5/flow.c+12−9
@@ -322,11 +322,11 @@ void mlx5_ib_destroy_flow_action_raw(struct mlx5_ib_flow_action *maction) · 321 unmodified lines
322322 switch (maction->flow_action_raw.sub_type) {
323323 case MLX5_IB_FLOW_ACTION_MODIFY_HEADER:
324324 mlx5_modify_header_dealloc(maction->flow_action_raw.dev->mdev,
325− maction->flow_action_raw.action_id);
325+ maction->flow_action_raw.modify_hdr);
326326 break;
327327 case MLX5_IB_FLOW_ACTION_PACKET_REFORMAT:
328328 mlx5_packet_reformat_dealloc(maction->flow_action_raw.dev->mdev,
329− maction->flow_action_raw.action_id);
329+ maction->flow_action_raw.pkt_reformat);
330330 break;
331331 case MLX5_IB_FLOW_ACTION_DECAP:
332332 break;
@@ -352,10 +352,11 @@ mlx5_ib_create_modify_header(struct mlx5_ib_dev *dev, · 19 unmodified lines
352352 if (!maction)
353353 return ERR_PTR(-ENOMEM);
354354
355− ret = mlx5_modify_header_alloc(dev->mdev, namespace, num_actions, in,
356− &maction->flow_action_raw.action_id);
355+ maction->flow_action_raw.modify_hdr =
356+ mlx5_modify_header_alloc(dev->mdev, namespace, num_actions, in);
357357
358− if (ret) {
358+ if (IS_ERR(maction->flow_action_raw.modify_hdr)) {
359+ ret = PTR_ERR(maction->flow_action_raw.modify_hdr);
359360 kfree(maction);
360361 return ERR_PTR(ret);
361362 }
@@ -479,11 +480,13 @@ static int mlx5_ib_flow_action_create_packet_reformat_ctx( · 117 unmodified lines
479480 if (ret)
480481 return ret;
481482
482− ret = mlx5_packet_reformat_alloc(dev->mdev, prm_prt, len,
483− in, namespace,
484− &maction->flow_action_raw.action_id);
485− if (ret)
483+ maction->flow_action_raw.pkt_reformat =
484+ mlx5_packet_reformat_alloc(dev->mdev, prm_prt, len,
485+ in, namespace);
486+ if (IS_ERR(maction->flow_action_raw.pkt_reformat)) {
487+ ret = PTR_ERR(maction->flow_action_raw.pkt_reformat);
486488 return ret;
489+ }
487490
488491 maction->flow_action_raw.sub_type =
489492 MLX5_IB_FLOW_ACTION_PACKET_REFORMAT;
drivers/infiniband/hw/mlx5/main.c+34−75
@@ -2280,6 +2280,7 @@ static inline int check_dm_type_support(struct mlx5_ib_dev *dev, · 2279 unmodified lines
22802280 return -EOPNOTSUPP;
22812281 break;
22822282 case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
2283+ case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
22832284 if (!capable(CAP_SYS_RAWIO) ||
22842285 !capable(CAP_NET_RAW))
22852286 return -EPERM;
@@ -2344,20 +2345,20 @@ static int handle_alloc_dm_sw_icm(struct ib_ucontext *ctx, · 58 unmodified lines
23442345 struct uverbs_attr_bundle *attrs,
23452346 int type)
23462347 {
2347− struct mlx5_dm *dm_db = &to_mdev(ctx->device)->dm;
2348+ struct mlx5_core_dev *dev = to_mdev(ctx->device)->mdev;
23482349 u64 act_size;
23492350 int err;
23502351
23512352 /* Allocation size must a multiple of the basic block size
23522353 * and a power of 2.
23532354 */
2354− act_size = round_up(attr->length, MLX5_SW_ICM_BLOCK_SIZE(dm_db->dev));
2355+ act_size = round_up(attr->length, MLX5_SW_ICM_BLOCK_SIZE(dev));
23552356 act_size = roundup_pow_of_two(act_size);
23562357
23572358 dm->size = act_size;
2358− err = mlx5_cmd_alloc_sw_icm(dm_db, type, act_size,
2359− to_mucontext(ctx)->devx_uid, &dm->dev_addr,
2360− &dm->icm_dm.obj_id);
2359+ err = mlx5_dm_sw_icm_alloc(dev, type, act_size,
2360+ to_mucontext(ctx)->devx_uid, &dm->dev_addr,
2361+ &dm->icm_dm.obj_id);
23612362 if (err)
23622363 return err;
23632364
@@ -2365,9 +2366,9 @@ static int handle_alloc_dm_sw_icm(struct ib_ucontext *ctx, · 1 unmodified line
23652366 MLX5_IB_ATTR_ALLOC_DM_RESP_START_OFFSET,
23662367 &dm->dev_addr, sizeof(dm->dev_addr));
23672368 if (err)
2368− mlx5_cmd_dealloc_sw_icm(dm_db, type, dm->size,
2369− to_mucontext(ctx)->devx_uid,
2370− dm->dev_addr, dm->icm_dm.obj_id);
2369+ mlx5_dm_sw_icm_dealloc(dev, type, dm->size,
2370+ to_mucontext(ctx)->devx_uid, dm->dev_addr,
2371+ dm->icm_dm.obj_id);
23712372
23722373 return err;
23732374 }
@@ -2407,8 +2408,14 @@ struct ib_dm *mlx5_ib_alloc_dm(struct ib_device *ibdev, · 33 unmodified lines
24072408 attrs);
24082409 break;
24092410 case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
2411+ err = handle_alloc_dm_sw_icm(context, dm,
2412+ attr, attrs,
2413+ MLX5_SW_ICM_TYPE_STEERING);
2414+ break;
24102415 case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
2411− err = handle_alloc_dm_sw_icm(context, dm, attr, attrs, type);
2416+ err = handle_alloc_dm_sw_icm(context, dm,
2417+ attr, attrs,
2418+ MLX5_SW_ICM_TYPE_HEADER_MODIFY);
24122419 break;
24132420 default:
24142421 err = -EOPNOTSUPP;
@@ -2428,6 +2435,7 @@ int mlx5_ib_dealloc_dm(struct ib_dm *ibdm, struct uverbs_attr_bundle *attrs) · 13 unmodified lines
24282435 {
24292436 struct mlx5_ib_ucontext *ctx = rdma_udata_to_drv_context(
24302437 &attrs->driver_udata, struct mlx5_ib_ucontext, ibucontext);
2438+ struct mlx5_core_dev *dev = to_mdev(ibdm->device)->mdev;
24312439 struct mlx5_dm *dm_db = &to_mdev(ibdm->device)->dm;
24322440 struct mlx5_ib_dm *dm = to_mdm(ibdm);
24332441 u32 page_idx;
@@ -2439,19 +2447,23 @@ int mlx5_ib_dealloc_dm(struct ib_dm *ibdm, struct uverbs_attr_bundle *attrs) · 5 unmodified lines
24392447 if (ret)
24402448 return ret;
24412449
2442− page_idx = (dm->dev_addr -
2443− pci_resource_start(dm_db->dev->pdev, 0) -
2444− MLX5_CAP64_DEV_MEM(dm_db->dev,
2445− memic_bar_start_addr)) >>
2446− PAGE_SHIFT;
2450+ page_idx = (dm->dev_addr - pci_resource_start(dev->pdev, 0) -
2451+ MLX5_CAP64_DEV_MEM(dev, memic_bar_start_addr)) >>
2452+ PAGE_SHIFT;
24472453 bitmap_clear(ctx->dm_pages, page_idx,
24482454 DIV_ROUND_UP(dm->size, PAGE_SIZE));
24492455 break;
24502456 case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
2457+ ret = mlx5_dm_sw_icm_dealloc(dev, MLX5_SW_ICM_TYPE_STEERING,
2458+ dm->size, ctx->devx_uid, dm->dev_addr,
2459+ dm->icm_dm.obj_id);
2460+ if (ret)
2461+ return ret;
2462+ break;
24512463 case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
2452− ret = mlx5_cmd_dealloc_sw_icm(dm_db, dm->type, dm->size,
2453− ctx->devx_uid, dm->dev_addr,
2454− dm->icm_dm.obj_id);
2464+ ret = mlx5_dm_sw_icm_dealloc(dev, MLX5_SW_ICM_TYPE_HEADER_MODIFY,
2465+ dm->size, ctx->devx_uid, dm->dev_addr,
2466+ dm->icm_dm.obj_id);
24552467 if (ret)
24562468 return ret;
24572469 break;
@@ -2646,7 +2658,8 @@ int parse_flow_flow_action(struct mlx5_ib_flow_action *maction, · 188 unmodified lines
26462658 if (action->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)
26472659 return -EINVAL;
26482660 action->action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR;
2649− action->modify_id = maction->flow_action_raw.action_id;
2661+ action->modify_hdr =
2662+ maction->flow_action_raw.modify_hdr;
26502663 return 0;
26512664 }
26522665 if (maction->flow_action_raw.sub_type ==
@@ -2663,8 +2676,8 @@ int parse_flow_flow_action(struct mlx5_ib_flow_action *maction, · 10 unmodified lines
26632676 return -EINVAL;
26642677 action->action |=
26652678 MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;
2666− action->reformat_id =
2667− maction->flow_action_raw.action_id;
2679+ action->pkt_reformat =
2680+ maction->flow_action_raw.pkt_reformat;
26682681 return 0;
26692682 }
26702683 /* fall through */
@@ -6096,8 +6109,6 @@ static struct ib_counters *mlx5_ib_create_counters(struct ib_device *device, · 3425 unmodified lines
60966109
60976110 static void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev)
60986111 {
6099− struct mlx5_core_dev *mdev = dev->mdev;
6100−
61016112 mlx5_ib_cleanup_multiport_master(dev);
61026113 if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
61036114 srcu_barrier(&dev->mr_srcu);
@@ -6105,29 +6116,11 @@ static void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev) · 1 unmodified line
61056116 }
61066117
61076118 WARN_ON(!bitmap_empty(dev->dm.memic_alloc_pages, MLX5_MAX_MEMIC_PAGES));
6108−
6109− WARN_ON(dev->dm.steering_sw_icm_alloc_blocks &&
6110− !bitmap_empty(
6111− dev->dm.steering_sw_icm_alloc_blocks,
6112− BIT(MLX5_CAP_DEV_MEM(mdev, log_steering_sw_icm_size) -
6113− MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev))));
6114−
6115− kfree(dev->dm.steering_sw_icm_alloc_blocks);
6116−
6117− WARN_ON(dev->dm.header_modify_sw_icm_alloc_blocks &&
6118− !bitmap_empty(dev->dm.header_modify_sw_icm_alloc_blocks,
6119− BIT(MLX5_CAP_DEV_MEM(
6120− mdev, log_header_modify_sw_icm_size) -
6121− MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev))));
6122−
6123− kfree(dev->dm.header_modify_sw_icm_alloc_blocks);
61246119 }
61256120
61266121 static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
61276122 {
61286123 struct mlx5_core_dev *mdev = dev->mdev;
6129− u64 header_modify_icm_blocks = 0;
6130− u64 steering_icm_blocks = 0;
61316124 int err;
61326125 int i;
61336126
@@ -6174,51 +6167,17 @@ static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev) · 40 unmodified lines
61746167 INIT_LIST_HEAD(&dev->qp_list);
61756168 spin_lock_init(&dev->reset_flow_resource_lock);
61766169
6177− if (MLX5_CAP_GEN_64(mdev, general_obj_types) &
6178− MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM) {
6179− if (MLX5_CAP64_DEV_MEM(mdev, steering_sw_icm_start_address)) {
6180− steering_icm_blocks =
6181− BIT(MLX5_CAP_DEV_MEM(mdev,
6182− log_steering_sw_icm_size) -
6183− MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev));
6184−
6185− dev->dm.steering_sw_icm_alloc_blocks =
6186− kcalloc(BITS_TO_LONGS(steering_icm_blocks),
6187− sizeof(unsigned long), GFP_KERNEL);
6188− if (!dev->dm.steering_sw_icm_alloc_blocks)
6189− goto err_mp;
6190− }
6191−
6192− if (MLX5_CAP64_DEV_MEM(mdev,
6193− header_modify_sw_icm_start_address)) {
6194− header_modify_icm_blocks = BIT(
6195− MLX5_CAP_DEV_MEM(
6196− mdev, log_header_modify_sw_icm_size) -
6197− MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev));
6198−
6199− dev->dm.header_modify_sw_icm_alloc_blocks =
6200− kcalloc(BITS_TO_LONGS(header_modify_icm_blocks),
6201− sizeof(unsigned long), GFP_KERNEL);
6202− if (!dev->dm.header_modify_sw_icm_alloc_blocks)
6203− goto err_dm;
6204− }
6205− }
6206−
62076170 spin_lock_init(&dev->dm.lock);
62086171 dev->dm.dev = mdev;
62096172
62106173 if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
62116174 err = init_srcu_struct(&dev->mr_srcu);
62126175 if (err)
6213− goto err_dm;
6176+ goto err_mp;
62146177 }
62156178
62166179 return 0;
62176180
6218−err_dm:
6219− kfree(dev->dm.steering_sw_icm_alloc_blocks);
6220− kfree(dev->dm.header_modify_sw_icm_alloc_blocks);
6221−
62226181 err_mp:
62236182 mlx5_ib_cleanup_multiport_master(dev);
62246183
drivers/infiniband/hw/mlx5/mlx5_ib.h+4−3
@@ -868,7 +868,10 @@ struct mlx5_ib_flow_action { · 867 unmodified lines
868868 struct {
869869 struct mlx5_ib_dev *dev;
870870 u32 sub_type;
871− u32 action_id;
871+ union {
872+ struct mlx5_modify_hdr *modify_hdr;
873+ struct mlx5_pkt_reformat *pkt_reformat;
874+ };
872875 } flow_action_raw;
873876 };
874877 };
@@ -881,8 +884,6 @@ struct mlx5_dm { · 6 unmodified lines
881884 */
882885 spinlock_t lock;
883886 DECLARE_BITMAP(memic_alloc_pages, MLX5_MAX_MEMIC_PAGES);
884− unsigned long *steering_sw_icm_alloc_blocks;
885− unsigned long *header_modify_sw_icm_alloc_blocks;
886887 };
887888
888889 struct mlx5_read_counters_attr {
drivers/net/ethernet/mellanox/mlx5/core/Kconfig+7−0
@@ -154,3 +154,10 @@ config MLX5_EN_TLS · 153 unmodified lines
154154 Build support for TLS cryptography-offload accelaration in the NIC.
155155 Note: Support for hardware with this capability needs to be selected
156156 for this option to become available.
157+
158+config MLX5_SW_STEERING
159+ bool "Mellanox Technologies software-managed steering"
160+ depends on MLX5_CORE_EN && MLX5_ESWITCH
161+ default y
162+ help
163+ Build support for software-managed steering in the NIC.
drivers/net/ethernet/mellanox/mlx5/core/Makefile+8−1
@@ -15,7 +15,7 @@ mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \ · 14 unmodified lines
1515 health.o mcg.o cq.o alloc.o qp.o port.o mr.o pd.o \
1616 transobj.o vport.o sriov.o fs_cmd.o fs_core.o pci_irq.o \
1717 fs_counters.o rl.o lag.o dev.o events.o wq.o lib/gid.o \
18− lib/devcom.o lib/pci_vsc.o diag/fs_tracepoint.o \
18+ lib/devcom.o lib/pci_vsc.o lib/dm.o diag/fs_tracepoint.o \
1919 diag/fw_tracer.o diag/crdump.o devlink.o
2020
2121 #
@@ -67,3 +67,10 @@ mlx5_core-$(CONFIG_MLX5_EN_IPSEC) += en_accel/ipsec.o en_accel/ipsec_rxtx.o \ · 45 unmodified lines
6767
6868 mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/tls.o en_accel/tls_rxtx.o en_accel/tls_stats.o \
6969 en_accel/ktls.o en_accel/ktls_tx.o
70+
71+mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \
72+ steering/dr_matcher.o steering/dr_rule.o \
73+ steering/dr_icm_pool.o steering/dr_crc32.o \
74+ steering/dr_ste.o steering/dr_send.o \
75+ steering/dr_cmd.o steering/dr_fw.o \
76+ steering/dr_action.o steering/fs_dr.o
drivers/net/ethernet/mellanox/mlx5/core/devlink.c+111−1
@@ -4,6 +4,7 @@ · 3 unmodified lines
44 #include <devlink.h>
55
66 #include "mlx5_core.h"
7+#include "fs_core.h"
78 #include "eswitch.h"
89
910 static int mlx5_devlink_flash_update(struct devlink *devlink,
@@ -107,12 +108,121 @@ void mlx5_devlink_free(struct devlink *devlink) · 97 unmodified lines
107108 devlink_free(devlink);
108109 }
109110
111+static int mlx5_devlink_fs_mode_validate(struct devlink *devlink, u32 id,
112+ union devlink_param_value val,
113+ struct netlink_ext_ack *extack)
114+{
115+ struct mlx5_core_dev *dev = devlink_priv(devlink);
116+ char *value = val.vstr;
117+ int err = 0;
118+
119+ if (!strcmp(value, "dmfs")) {
120+ return 0;
121+ } else if (!strcmp(value, "smfs")) {
122+ u8 eswitch_mode;
123+ bool smfs_cap;
124+
125+ eswitch_mode = mlx5_eswitch_mode(dev->priv.eswitch);
126+ smfs_cap = mlx5_fs_dr_is_supported(dev);
127+
128+ if (!smfs_cap) {
129+ err = -EOPNOTSUPP;
130+ NL_SET_ERR_MSG_MOD(extack,
131+ "Software managed steering is not supported by current device");
132+ }
133+
134+ else if (eswitch_mode == MLX5_ESWITCH_OFFLOADS) {
135+ NL_SET_ERR_MSG_MOD(extack,
136+ "Software managed steering is not supported when eswitch offlaods enabled.");
137+ err = -EOPNOTSUPP;
138+ }
139+ } else {
140+ NL_SET_ERR_MSG_MOD(extack,
141+ "Bad parameter: supported values are [\"dmfs\", \"smfs\"]");
142+ err = -EINVAL;
143+ }
144+
145+ return err;
146+}
147+
148+static int mlx5_devlink_fs_mode_set(struct devlink *devlink, u32 id,
149+ struct devlink_param_gset_ctx *ctx)
150+{
151+ struct mlx5_core_dev *dev = devlink_priv(devlink);
152+ enum mlx5_flow_steering_mode mode;
153+
154+ if (!strcmp(ctx->val.vstr, "smfs"))
155+ mode = MLX5_FLOW_STEERING_MODE_SMFS;
156+ else
157+ mode = MLX5_FLOW_STEERING_MODE_DMFS;
158+ dev->priv.steering->mode = mode;
159+
160+ return 0;
161+}
162+
163+static int mlx5_devlink_fs_mode_get(struct devlink *devlink, u32 id,
164+ struct devlink_param_gset_ctx *ctx)
165+{
166+ struct mlx5_core_dev *dev = devlink_priv(devlink);
167+
168+ if (dev->priv.steering->mode == MLX5_FLOW_STEERING_MODE_SMFS)
169+ strcpy(ctx->val.vstr, "smfs");
170+ else
171+ strcpy(ctx->val.vstr, "dmfs");
172+ return 0;
173+}
174+
175+enum mlx5_devlink_param_id {
176+ MLX5_DEVLINK_PARAM_ID_BASE = DEVLINK_PARAM_GENERIC_ID_MAX,
177+ MLX5_DEVLINK_PARAM_FLOW_STEERING_MODE,
178+};
179+
180+static const struct devlink_param mlx5_devlink_params[] = {
181+ DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_FLOW_STEERING_MODE,
182+ "flow_steering_mode", DEVLINK_PARAM_TYPE_STRING,
183+ BIT(DEVLINK_PARAM_CMODE_RUNTIME),
184+ mlx5_devlink_fs_mode_get, mlx5_devlink_fs_mode_set,
185+ mlx5_devlink_fs_mode_validate),
186+};
187+
188+static void mlx5_devlink_set_params_init_values(struct devlink *devlink)
189+{
190+ struct mlx5_core_dev *dev = devlink_priv(devlink);
191+ union devlink_param_value value;
192+
193+ if (dev->priv.steering->mode == MLX5_FLOW_STEERING_MODE_DMFS)
194+ strcpy(value.vstr, "dmfs");
195+ else
196+ strcpy(value.vstr, "smfs");
197+ devlink_param_driverinit_value_set(devlink,
198+ MLX5_DEVLINK_PARAM_FLOW_STEERING_MODE,
199+ value);
200+}
201+
110202 int mlx5_devlink_register(struct devlink *devlink, struct device *dev)
111203 {
112− return devlink_register(devlink, dev);
204+ int err;
205+
206+ err = devlink_register(devlink, dev);
207+ if (err)
208+ return err;
209+
210+ err = devlink_params_register(devlink, mlx5_devlink_params,
211+ ARRAY_SIZE(mlx5_devlink_params));
212+ if (err)
213+ goto params_reg_err;
214+ mlx5_devlink_set_params_init_values(devlink);
215+ devlink_params_publish(devlink);
216+ return 0;
217+
218+params_reg_err:
219+ devlink_unregister(devlink);
220+ return err;
113221 }
114222
115223 void mlx5_devlink_unregister(struct devlink *devlink)
116224 {
225+ devlink_params_unregister(devlink, mlx5_devlink_params,
226+ ARRAY_SIZE(mlx5_devlink_params));
117227 devlink_unregister(devlink);
118228 }
drivers/net/ethernet/mellanox/mlx5/core/en/tc_tun.c+14−13
@@ -291,14 +291,14 @@ int mlx5e_tc_tun_create_header_ipv4(struct mlx5e_priv *priv, · 290 unmodified lines
291291 */
292292 goto out;
293293 }
294−
295− err = mlx5_packet_reformat_alloc(priv->mdev,
296− e->reformat_type,
297− ipv4_encap_size, encap_header,
298− MLX5_FLOW_NAMESPACE_FDB,
299− &e->encap_id);
300− if (err)
294+ e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev,
295+ e->reformat_type,
296+ ipv4_encap_size, encap_header,
297+ MLX5_FLOW_NAMESPACE_FDB);
298+ if (IS_ERR(e->pkt_reformat)) {
299+ err = PTR_ERR(e->pkt_reformat);
301300 goto destroy_neigh_entry;
301+ }
302302
303303 e->flags |= MLX5_ENCAP_ENTRY_VALID;
304304 mlx5e_rep_queue_neigh_stats_work(netdev_priv(out_dev));
@@ -407,13 +407,14 @@ int mlx5e_tc_tun_create_header_ipv6(struct mlx5e_priv *priv, · 102 unmodified lines
407407 goto out;
408408 }
409409
410− err = mlx5_packet_reformat_alloc(priv->mdev,
411− e->reformat_type,
412− ipv6_encap_size, encap_header,
413− MLX5_FLOW_NAMESPACE_FDB,
414− &e->encap_id);
415− if (err)
410+ e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev,
411+ e->reformat_type,
412+ ipv6_encap_size, encap_header,
413+ MLX5_FLOW_NAMESPACE_FDB);
414+ if (IS_ERR(e->pkt_reformat)) {
415+ err = PTR_ERR(e->pkt_reformat);
416416 goto destroy_neigh_entry;
417+ }
417418
418419 e->flags |= MLX5_ENCAP_ENTRY_VALID;
419420 mlx5e_rep_queue_neigh_stats_work(netdev_priv(out_dev));
drivers/net/ethernet/mellanox/mlx5/core/en_rep.h+1−1
@@ -161,7 +161,7 @@ struct mlx5e_encap_entry { · 160 unmodified lines
161161 */
162162 struct hlist_node encap_hlist;
163163 struct list_head flows;
164− u32 encap_id;
164+ struct mlx5_pkt_reformat *pkt_reformat;
165165 const struct ip_tunnel_info *tun_info;
166166 unsigned char h_dest[ETH_ALEN]; /* destination eth addr */
167167
drivers/net/ethernet/mellanox/mlx5/core/en_tc.c+26−25
@@ -61,7 +61,7 @@ · 60 unmodified lines
6161 struct mlx5_nic_flow_attr {
6262 u32 action;
6363 u32 flow_tag;
64− u32 mod_hdr_id;
64+ struct mlx5_modify_hdr *modify_hdr;
6565 u32 hairpin_tirn;
6666 u8 match_level;
6767 struct mlx5_flow_table *hairpin_ft;
@@ -201,7 +201,7 @@ struct mlx5e_mod_hdr_entry { · 133 unmodified lines
201201
202202 struct mod_hdr_key key;
203203
204− u32 mod_hdr_id;
204+ struct mlx5_modify_hdr *modify_hdr;
205205
206206 refcount_t refcnt;
207207 struct completion res_ready;
@@ -334,7 +334,7 @@ static void mlx5e_mod_hdr_put(struct mlx5e_priv *priv, · 126 unmodified lines
334334
335335 WARN_ON(!list_empty(&mh->flows));
336336 if (mh->compl_result > 0)
337− mlx5_modify_header_dealloc(priv->mdev, mh->mod_hdr_id);
337+ mlx5_modify_header_dealloc(priv->mdev, mh->modify_hdr);
338338
339339 kfree(mh);
340340 }
@@ -395,11 +395,11 @@ static int mlx5e_attach_mod_hdr(struct mlx5e_priv *priv, · 54 unmodified lines
395395 hash_add(tbl->hlist, &mh->mod_hdr_hlist, hash_key);
396396 mutex_unlock(&tbl->lock);
397397
398− err = mlx5_modify_header_alloc(priv->mdev, namespace,
399− mh->key.num_actions,
400− mh->key.actions,
401− &mh->mod_hdr_id);
402− if (err) {
398+ mh->modify_hdr = mlx5_modify_header_alloc(priv->mdev, namespace,
399+ mh->key.num_actions,
400+ mh->key.actions);
401+ if (IS_ERR(mh->modify_hdr)) {
402+ err = PTR_ERR(mh->modify_hdr);
403403 mh->compl_result = err;
404404 goto alloc_header_err;
405405 }
@@ -412,9 +412,9 @@ attach_flow: · 6 unmodified lines
412412 list_add(&flow->mod_hdr, &mh->flows);
413413 spin_unlock(&mh->flows_lock);
414414 if (mlx5e_is_eswitch_flow(flow))
415− flow->esw_attr->mod_hdr_id = mh->mod_hdr_id;
415+ flow->esw_attr->modify_hdr = mh->modify_hdr;
416416 else
417− flow->nic_attr->mod_hdr_id = mh->mod_hdr_id;
417+ flow->nic_attr->modify_hdr = mh->modify_hdr;
418418
419419 return 0;
420420
@@ -906,7 +906,6 @@ mlx5e_tc_add_nic_flow(struct mlx5e_priv *priv, · 485 unmodified lines
906906 struct mlx5_flow_destination dest[2] = {};
907907 struct mlx5_flow_act flow_act = {
908908 .action = attr->action,
909− .reformat_id = 0,
910909 .flags = FLOW_ACT_NO_APPEND,
911910 };
912911 struct mlx5_fc *counter = NULL;
@@ -947,7 +946,7 @@ mlx5e_tc_add_nic_flow(struct mlx5e_priv *priv, · 34 unmodified lines
947946
948947 if (attr->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR) {
949948 err = mlx5e_attach_mod_hdr(priv, flow, parse_attr);
950− flow_act.modify_id = attr->mod_hdr_id;
949+ flow_act.modify_hdr = attr->modify_hdr;
951950 kfree(parse_attr->mod_hdr_actions);
952951 if (err)
953952 return err;
@@ -1304,14 +1303,13 @@ void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv, · 350 unmodified lines
13041303 struct mlx5e_tc_flow *flow;
13051304 int err;
13061305
1307− err = mlx5_packet_reformat_alloc(priv->mdev,
1308− e->reformat_type,
1309− e->encap_size, e->encap_header,
1310− MLX5_FLOW_NAMESPACE_FDB,
1311− &e->encap_id);
1312− if (err) {
1313− mlx5_core_warn(priv->mdev, "Failed to offload cached encapsulation header, %d\n",
1314− err);
1306+ e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev,
1307+ e->reformat_type,
1308+ e->encap_size, e->encap_header,
1309+ MLX5_FLOW_NAMESPACE_FDB);
1310+ if (IS_ERR(e->pkt_reformat)) {
1311+ mlx5_core_warn(priv->mdev, "Failed to offload cached encapsulation header, %lu\n",
1312+ PTR_ERR(e->pkt_reformat));
13151313 return;
13161314 }
13171315 e->flags |= MLX5_ENCAP_ENTRY_VALID;
@@ -1326,7 +1324,7 @@ void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv, · 8 unmodified lines
13261324 esw_attr = flow->esw_attr;
13271325 spec = &esw_attr->parse_attr->spec;
13281326
1329− esw_attr->dests[flow->tmp_efi_index].encap_id = e->encap_id;
1327+ esw_attr->dests[flow->tmp_efi_index].pkt_reformat = e->pkt_reformat;
13301328 esw_attr->dests[flow->tmp_efi_index].flags |= MLX5_ESW_DEST_ENCAP_VALID;
13311329 /* Flow can be associated with multiple encap entries.
13321330 * Before offloading the flow verify that all of them have
@@ -1395,7 +1393,7 @@ void mlx5e_tc_encap_flows_del(struct mlx5e_priv *priv, · 62 unmodified lines
13951393
13961394 /* we know that the encap is valid */
13971395 e->flags &= ~MLX5_ENCAP_ENTRY_VALID;
1398− mlx5_packet_reformat_dealloc(priv->mdev, e->encap_id);
1396+ mlx5_packet_reformat_dealloc(priv->mdev, e->pkt_reformat);
13991397 }
14001398
14011399 static struct mlx5_fc *mlx5e_tc_get_counter(struct mlx5e_tc_flow *flow)
@@ -1561,7 +1559,7 @@ static void mlx5e_encap_dealloc(struct mlx5e_priv *priv, struct mlx5e_encap_entr · 159 unmodified lines
15611559 mlx5e_rep_encap_entry_detach(netdev_priv(e->out_dev), e);
15621560
15631561 if (e->flags & MLX5_ENCAP_ENTRY_VALID)
1564− mlx5_packet_reformat_dealloc(priv->mdev, e->encap_id);
1562+ mlx5_packet_reformat_dealloc(priv->mdev, e->pkt_reformat);
15651563 }
15661564
15671565 kfree(e->encap_header);
@@ -1896,7 +1894,10 @@ static int __parse_cls_flower(struct mlx5e_priv *priv, · 328 unmodified lines
18961894 *match_level = MLX5_MATCH_L2;
18971895 }
18981896 } else if (*match_level != MLX5_MATCH_NONE) {
1899− MLX5_SET(fte_match_set_lyr_2_4, headers_c, svlan_tag, 1);
1897+ /* cvlan_tag enabled in match criteria and
1898+ * disabled in match value means both S & C tags
1899+ * don't exist (untagged of both)
1900+ */
19001901 MLX5_SET(fte_match_set_lyr_2_4, headers_c, cvlan_tag, 1);
19011902 *match_level = MLX5_MATCH_L2;
19021903 }
@@ -3045,7 +3046,7 @@ attach_flow: · 1142 unmodified lines
30453046 flow->encaps[out_index].index = out_index;
30463047 *encap_dev = e->out_dev;
30473048 if (e->flags & MLX5_ENCAP_ENTRY_VALID) {
3048− attr->dests[out_index].encap_id = e->encap_id;
3049+ attr->dests[out_index].pkt_reformat = e->pkt_reformat;
30493050 attr->dests[out_index].flags |= MLX5_ESW_DEST_ENCAP_VALID;
30503051 *encap_valid = true;
30513052 } else {
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h+4−3
@@ -69,7 +69,7 @@ struct vport_ingress { · 68 unmodified lines
6969 struct mlx5_flow_group *allow_spoofchk_only_grp;
7070 struct mlx5_flow_group *allow_untagged_only_grp;
7171 struct mlx5_flow_group *drop_grp;
72− int modify_metadata_id;
72+ struct mlx5_modify_hdr *modify_metadata;
7373 struct mlx5_flow_handle *modify_metadata_rule;
7474 struct mlx5_flow_handle *allow_rule;
7575 struct mlx5_flow_handle *drop_rule;
@@ -153,6 +153,7 @@ struct mlx5_eswitch_fdb { · 77 unmodified lines
153153 } legacy;
154154
155155 struct offloads_fdb {
156+ struct mlx5_flow_namespace *ns;
156157 struct mlx5_flow_table *slow_fdb;
157158 struct mlx5_flow_group *send_to_vport_grp;
158159 struct mlx5_flow_group *peer_miss_grp;
@@ -385,11 +386,11 @@ struct mlx5_esw_flow_attr { · 226 unmodified lines
385386 struct {
386387 u32 flags;
387388 struct mlx5_eswitch_rep *rep;
389+ struct mlx5_pkt_reformat *pkt_reformat;
388390 struct mlx5_core_dev *mdev;
389− u32 encap_id;
390391 struct mlx5_termtbl_handle *termtbl;
391392 } dests[MLX5_MAX_FLOW_FWD_VPORTS];
392− u32 mod_hdr_id;
393+ struct mlx5_modify_hdr *modify_hdr;
393394 u8 inner_match_level;
394395 u8 outer_match_level;
395396 struct mlx5_fc *counter;
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c+68−19
@@ -190,10 +190,10 @@ mlx5_eswitch_add_offloaded_rule(struct mlx5_eswitch *esw, · 189 unmodified lines
190190 MLX5_FLOW_DEST_VPORT_VHCA_ID;
191191 if (attr->dests[j].flags & MLX5_ESW_DEST_ENCAP) {
192192 flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;
193− flow_act.reformat_id = attr->dests[j].encap_id;
193+ flow_act.pkt_reformat = attr->dests[j].pkt_reformat;
194194 dest[i].vport.flags |= MLX5_FLOW_DEST_VPORT_REFORMAT_ID;
195− dest[i].vport.reformat_id =
196− attr->dests[j].encap_id;
195+ dest[i].vport.pkt_reformat =
196+ attr->dests[j].pkt_reformat;
197197 }
198198 i++;
199199 }
@@ -213,7 +213,7 @@ mlx5_eswitch_add_offloaded_rule(struct mlx5_eswitch *esw, · 13 unmodified lines
213213 spec->match_criteria_enable |= MLX5_MATCH_INNER_HEADERS;
214214
215215 if (flow_act.action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)
216− flow_act.modify_id = attr->mod_hdr_id;
216+ flow_act.modify_hdr = attr->modify_hdr;
217217
218218 fdb = esw_get_prio_table(esw, attr->chain, attr->prio, !!split);
219219 if (IS_ERR(fdb)) {
@@ -276,7 +276,7 @@ mlx5_eswitch_add_fwd_rule(struct mlx5_eswitch *esw, · 56 unmodified lines
276276 dest[i].vport.flags |= MLX5_FLOW_DEST_VPORT_VHCA_ID;
277277 if (attr->dests[i].flags & MLX5_ESW_DEST_ENCAP) {
278278 dest[i].vport.flags |= MLX5_FLOW_DEST_VPORT_REFORMAT_ID;
279− dest[i].vport.reformat_id = attr->dests[i].encap_id;
279+ dest[i].vport.pkt_reformat = attr->dests[i].pkt_reformat;
280280 }
281281 }
282282 dest[i].type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
@@ -1068,6 +1068,13 @@ static int esw_create_offloads_fdb_tables(struct mlx5_eswitch *esw, int nvports) · 785 unmodified lines
10681068 err = -EOPNOTSUPP;
10691069 goto ns_err;
10701070 }
1071+ esw->fdb_table.offloads.ns = root_ns;
1072+ err = mlx5_flow_namespace_set_mode(root_ns,
1073+ esw->dev->priv.steering->mode);
1074+ if (err) {
1075+ esw_warn(dev, "Failed to set FDB namespace steering mode\n");
1076+ goto ns_err;
1077+ }
10711078
10721079 max_flow_counter = (MLX5_CAP_GEN(dev, max_flow_counter_31_16) << 16) |
10731080 MLX5_CAP_GEN(dev, max_flow_counter_15_0);
@@ -1207,6 +1214,8 @@ send_vport_err: · 133 unmodified lines
12071214 esw_destroy_offloads_fast_fdb_tables(esw);
12081215 mlx5_destroy_flow_table(esw->fdb_table.offloads.slow_fdb);
12091216 slow_fdb_err:
1217+ /* Holds true only as long as DMFS is the default */
1218+ mlx5_flow_namespace_set_mode(root_ns, MLX5_FLOW_STEERING_MODE_DMFS);
12101219 ns_err:
12111220 kvfree(flow_group_in);
12121221 return err;
@@ -1226,6 +1235,9 @@ static void esw_destroy_offloads_fdb_tables(struct mlx5_eswitch *esw) · 13 unmodified lines
12261235
12271236 mlx5_destroy_flow_table(esw->fdb_table.offloads.slow_fdb);
12281237 esw_destroy_offloads_fast_fdb_tables(esw);
1238+ /* Holds true only as long as DMFS is the default */
1239+ mlx5_flow_namespace_set_mode(esw->fdb_table.offloads.ns,
1240+ MLX5_FLOW_STEERING_MODE_DMFS);
12291241 }
12301242
12311243 static int esw_create_offloads_table(struct mlx5_eswitch *esw, int nvports)
@@ -1623,13 +1635,42 @@ static void mlx5_esw_offloads_unpair(struct mlx5_eswitch *esw) · 391 unmodified lines
16231635 esw_del_fdb_peer_miss_rules(esw);
16241636 }
16251637
1638+static int mlx5_esw_offloads_set_ns_peer(struct mlx5_eswitch *esw,
1639+ struct mlx5_eswitch *peer_esw,
1640+ bool pair)
1641+{
1642+ struct mlx5_flow_root_namespace *peer_ns;
1643+ struct mlx5_flow_root_namespace *ns;
1644+ int err;
1645+
1646+ peer_ns = peer_esw->dev->priv.steering->fdb_root_ns;
1647+ ns = esw->dev->priv.steering->fdb_root_ns;
1648+
1649+ if (pair) {
1650+ err = mlx5_flow_namespace_set_peer(ns, peer_ns);
1651+ if (err)
1652+ return err;
1653+
1654+ mlx5_flow_namespace_set_peer(peer_ns, ns);
1655+ if (err) {
1656+ mlx5_flow_namespace_set_peer(ns, NULL);
1657+ return err;
1658+ }
1659+ } else {
1660+ mlx5_flow_namespace_set_peer(ns, NULL);
1661+ mlx5_flow_namespace_set_peer(peer_ns, NULL);
1662+ }
1663+
1664+ return 0;
1665+}
1666+
16261667 static int mlx5_esw_offloads_devcom_event(int event,
16271668 void *my_data,
16281669 void *event_data)
16291670 {
16301671 struct mlx5_eswitch *esw = my_data;
1631− struct mlx5_eswitch *peer_esw = event_data;
16321672 struct mlx5_devcom *devcom = esw->dev->priv.devcom;
1673+ struct mlx5_eswitch *peer_esw = event_data;
16331674 int err;
16341675
16351676 switch (event) {
@@ -1638,9 +1679,12 @@ static int mlx5_esw_offloads_devcom_event(int event, · 2 unmodified lines
16381679 mlx5_eswitch_vport_match_metadata_enabled(peer_esw))
16391680 break;
16401681
1641− err = mlx5_esw_offloads_pair(esw, peer_esw);
1682+ err = mlx5_esw_offloads_set_ns_peer(esw, peer_esw, true);
16421683 if (err)
16431684 goto err_out;
1685+ err = mlx5_esw_offloads_pair(esw, peer_esw);
1686+ if (err)
1687+ goto err_peer;
16441688
16451689 err = mlx5_esw_offloads_pair(peer_esw, esw);
16461690 if (err)
@@ -1656,6 +1700,7 @@ static int mlx5_esw_offloads_devcom_event(int event, · 9 unmodified lines
16561700 mlx5_devcom_set_paired(devcom, MLX5_DEVCOM_ESW_OFFLOADS, false);
16571701 mlx5_esw_offloads_unpair(peer_esw);
16581702 mlx5_esw_offloads_unpair(esw);
1703+ mlx5_esw_offloads_set_ns_peer(esw, peer_esw, false);
16591704 break;
16601705 }
16611706
@@ -1663,7 +1708,8 @@ static int mlx5_esw_offloads_devcom_event(int event, · 1 unmodified line
16631708
16641709 err_pair:
16651710 mlx5_esw_offloads_unpair(esw);
1666−
1711+err_peer:
1712+ mlx5_esw_offloads_set_ns_peer(esw, peer_esw, false);
16671713 err_out:
16681714 mlx5_core_err(esw->dev, "esw offloads devcom event failure, event %u err %d",
16691715 event, err);
@@ -1734,7 +1780,7 @@ static int esw_vport_ingress_prio_tag_config(struct mlx5_eswitch *esw, · 64 unmodified lines
17341780
17351781 if (vport->ingress.modify_metadata_rule) {
17361782 flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR;
1737− flow_act.modify_id = vport->ingress.modify_metadata_id;
1783+ flow_act.modify_hdr = vport->ingress.modify_metadata;
17381784 }
17391785
17401786 vport->ingress.allow_rule =
@@ -1770,9 +1816,11 @@ static int esw_vport_add_ingress_acl_modify_metadata(struct mlx5_eswitch *esw, · 29 unmodified lines
17701816 MLX5_SET(set_action_in, action, data,
17711817 mlx5_eswitch_get_vport_metadata_for_match(esw, vport->vport));
17721818
1773− err = mlx5_modify_header_alloc(esw->dev, MLX5_FLOW_NAMESPACE_ESW_INGRESS,
1774− 1, action, &vport->ingress.modify_metadata_id);
1775− if (err) {
1819+ vport->ingress.modify_metadata =
1820+ mlx5_modify_header_alloc(esw->dev, MLX5_FLOW_NAMESPACE_ESW_INGRESS,
1821+ 1, action);
1822+ if (IS_ERR(vport->ingress.modify_metadata)) {
1823+ err = PTR_ERR(vport->ingress.modify_metadata);
17761824 esw_warn(esw->dev,
17771825 "failed to alloc modify header for vport %d ingress acl (%d)\n",
17781826 vport->vport, err);
@@ -1780,7 +1828,7 @@ static int esw_vport_add_ingress_acl_modify_metadata(struct mlx5_eswitch *esw, · 1 unmodified line
17801828 }
17811829
17821830 flow_act.action = MLX5_FLOW_CONTEXT_ACTION_MOD_HDR | MLX5_FLOW_CONTEXT_ACTION_ALLOW;
1783− flow_act.modify_id = vport->ingress.modify_metadata_id;
1831+ flow_act.modify_hdr = vport->ingress.modify_metadata;
17841832 vport->ingress.modify_metadata_rule = mlx5_add_flow_rules(vport->ingress.acl,
17851833 &spec, &flow_act, NULL, 0);
17861834 if (IS_ERR(vport->ingress.modify_metadata_rule)) {
@@ -1794,7 +1842,7 @@ static int esw_vport_add_ingress_acl_modify_metadata(struct mlx5_eswitch *esw, · 7 unmodified lines
17941842
17951843 out:
17961844 if (err)
1797− mlx5_modify_header_dealloc(esw->dev, vport->ingress.modify_metadata_id);
1845+ mlx5_modify_header_dealloc(esw->dev, vport->ingress.modify_metadata);
17981846 return err;
17991847 }
18001848
@@ -1803,7 +1851,7 @@ void esw_vport_del_ingress_acl_modify_metadata(struct mlx5_eswitch *esw, · 2 unmodified lines
18031851 {
18041852 if (vport->ingress.modify_metadata_rule) {
18051853 mlx5_del_flow_rules(vport->ingress.modify_metadata_rule);
1806− mlx5_modify_header_dealloc(esw->dev, vport->ingress.modify_metadata_id);
1854+ mlx5_modify_header_dealloc(esw->dev, vport->ingress.modify_metadata);
18071855
18081856 vport->ingress.modify_metadata_rule = NULL;
18091857 }
@@ -2113,9 +2161,10 @@ int esw_offloads_enable(struct mlx5_eswitch *esw) · 303 unmodified lines
21132161 else
21142162 esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_NONE;
21152163
2164+ mlx5_rdma_enable_roce(esw->dev);
21162165 err = esw_offloads_steering_init(esw);
21172166 if (err)
2118− return err;
2167+ goto err_steering_init;
21192168
21202169 err = esw_set_passing_vport_metadata(esw, true);
21212170 if (err)
@@ -2130,8 +2179,6 @@ int esw_offloads_enable(struct mlx5_eswitch *esw) · 8 unmodified lines
21302179 esw_offloads_devcom_init(esw);
21312180 mutex_init(&esw->offloads.termtbl_mutex);
21322181
2133− mlx5_rdma_enable_roce(esw->dev);
2134−
21352182 return 0;
21362183
21372184 err_reps:
@@ -2139,6 +2186,8 @@ err_reps: · 1 unmodified line
21392186 esw_set_passing_vport_metadata(esw, false);
21402187 err_vport_metadata:
21412188 esw_offloads_steering_cleanup(esw);
2189+err_steering_init:
2190+ mlx5_rdma_disable_roce(esw->dev);
21422191 return err;
21432192 }
21442193
@@ -2163,12 +2212,12 @@ static int esw_offloads_stop(struct mlx5_eswitch *esw, · 18 unmodified lines
21632212
21642213 void esw_offloads_disable(struct mlx5_eswitch *esw)
21652214 {
2166− mlx5_rdma_disable_roce(esw->dev);
21672215 esw_offloads_devcom_cleanup(esw);
21682216 esw_offloads_unload_all_reps(esw);
21692217 mlx5_eswitch_disable_pf_vf_vports(esw);
21702218 esw_set_passing_vport_metadata(esw, false);
21712219 esw_offloads_steering_cleanup(esw);
2220+ mlx5_rdma_disable_roce(esw->dev);
21722221 esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_NONE;
21732222 }
21742223
drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c+89−27
@@ -107,6 +107,50 @@ static int mlx5_cmd_stub_delete_fte(struct mlx5_flow_root_namespace *ns, · 106 unmodified lines
107107 return 0;
108108 }
109109
110+static int mlx5_cmd_stub_packet_reformat_alloc(struct mlx5_flow_root_namespace *ns,
111+ int reformat_type,
112+ size_t size,
113+ void *reformat_data,
114+ enum mlx5_flow_namespace_type namespace,
115+ struct mlx5_pkt_reformat *pkt_reformat)
116+{
117+ return 0;
118+}
119+
120+static void mlx5_cmd_stub_packet_reformat_dealloc(struct mlx5_flow_root_namespace *ns,
121+ struct mlx5_pkt_reformat *pkt_reformat)
122+{
123+}
124+
125+static int mlx5_cmd_stub_modify_header_alloc(struct mlx5_flow_root_namespace *ns,
126+ u8 namespace, u8 num_actions,
127+ void *modify_actions,
128+ struct mlx5_modify_hdr *modify_hdr)
129+{
130+ return 0;
131+}
132+
133+static void mlx5_cmd_stub_modify_header_dealloc(struct mlx5_flow_root_namespace *ns,
134+ struct mlx5_modify_hdr *modify_hdr)
135+{
136+}
137+
138+static int mlx5_cmd_stub_set_peer(struct mlx5_flow_root_namespace *ns,
139+ struct mlx5_flow_root_namespace *peer_ns)
140+{
141+ return 0;
142+}
143+
144+static int mlx5_cmd_stub_create_ns(struct mlx5_flow_root_namespace *ns)
145+{
146+ return 0;
147+}
148+
149+static int mlx5_cmd_stub_destroy_ns(struct mlx5_flow_root_namespace *ns)
150+{
151+ return 0;
152+}
153+
110154 static int mlx5_cmd_update_root_ft(struct mlx5_flow_root_namespace *ns,
111155 struct mlx5_flow_table *ft, u32 underlay_qpn,
112156 bool disconnect)
@@ -412,11 +456,13 @@ static int mlx5_cmd_set_fte(struct mlx5_core_dev *dev, · 299 unmodified lines
412456 } else {
413457 MLX5_SET(flow_context, in_flow_context, action,
414458 fte->action.action);
415− MLX5_SET(flow_context, in_flow_context, packet_reformat_id,
416− fte->action.reformat_id);
459+ if (fte->action.pkt_reformat)
460+ MLX5_SET(flow_context, in_flow_context, packet_reformat_id,
461+ fte->action.pkt_reformat->id);
417462 }
418− MLX5_SET(flow_context, in_flow_context, modify_header_id,
419− fte->action.modify_id);
463+ if (fte->action.modify_hdr)
464+ MLX5_SET(flow_context, in_flow_context, modify_header_id,
465+ fte->action.modify_hdr->id);
420466
421467 vlan = MLX5_ADDR_OF(flow_context, in_flow_context, push_vlan);
422468
@@ -468,7 +514,7 @@ static int mlx5_cmd_set_fte(struct mlx5_core_dev *dev, · 45 unmodified lines
468514 MLX5_FLOW_DEST_VPORT_REFORMAT_ID));
469515 MLX5_SET(extended_dest_format, in_dests,
470516 packet_reformat_id,
471− dst->dest_attr.vport.reformat_id);
517+ dst->dest_attr.vport.pkt_reformat->id);
472518 }
473519 break;
474520 default:
@@ -643,14 +689,15 @@ int mlx5_cmd_fc_bulk_query(struct mlx5_core_dev *dev, u32 base_id, int bulk_len, · 168 unmodified lines
643689 return mlx5_cmd_exec(dev, in, sizeof(in), out, outlen);
644690 }
645691
646−int mlx5_packet_reformat_alloc(struct mlx5_core_dev *dev,
647− int reformat_type,
648− size_t size,
649− void *reformat_data,
650− enum mlx5_flow_namespace_type namespace,
651− u32 *packet_reformat_id)
692+static int mlx5_cmd_packet_reformat_alloc(struct mlx5_flow_root_namespace *ns,
693+ int reformat_type,
694+ size_t size,
695+ void *reformat_data,
696+ enum mlx5_flow_namespace_type namespace,
697+ struct mlx5_pkt_reformat *pkt_reformat)
652698 {
653699 u32 out[MLX5_ST_SZ_DW(alloc_packet_reformat_context_out)];
700+ struct mlx5_core_dev *dev = ns->dev;
654701 void *packet_reformat_context_in;
655702 int max_encap_size;
656703 void *reformat;
@@ -693,35 +740,36 @@ int mlx5_packet_reformat_alloc(struct mlx5_core_dev *dev, · 36 unmodified lines
693740 memset(out, 0, sizeof(out));
694741 err = mlx5_cmd_exec(dev, in, inlen, out, sizeof(out));
695742
696− *packet_reformat_id = MLX5_GET(alloc_packet_reformat_context_out,
697− out, packet_reformat_id);
743+ pkt_reformat->id = MLX5_GET(alloc_packet_reformat_context_out,
744+ out, packet_reformat_id);
698745 kfree(in);
699746 return err;
700747 }
701−EXPORT_SYMBOL(mlx5_packet_reformat_alloc);
702748
703−void mlx5_packet_reformat_dealloc(struct mlx5_core_dev *dev,
704− u32 packet_reformat_id)
749+static void mlx5_cmd_packet_reformat_dealloc(struct mlx5_flow_root_namespace *ns,
750+ struct mlx5_pkt_reformat *pkt_reformat)
705751 {
706752 u32 in[MLX5_ST_SZ_DW(dealloc_packet_reformat_context_in)];
707753 u32 out[MLX5_ST_SZ_DW(dealloc_packet_reformat_context_out)];
754+ struct mlx5_core_dev *dev = ns->dev;
708755
709756 memset(in, 0, sizeof(in));
710757 MLX5_SET(dealloc_packet_reformat_context_in, in, opcode,
711758 MLX5_CMD_OP_DEALLOC_PACKET_REFORMAT_CONTEXT);
712759 MLX5_SET(dealloc_packet_reformat_context_in, in, packet_reformat_id,
713− packet_reformat_id);
760+ pkt_reformat->id);
714761
715762 mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
716763 }
717−EXPORT_SYMBOL(mlx5_packet_reformat_dealloc);
718764
719−int mlx5_modify_header_alloc(struct mlx5_core_dev *dev,
720− u8 namespace, u8 num_actions,
721− void *modify_actions, u32 *modify_header_id)
765+static int mlx5_cmd_modify_header_alloc(struct mlx5_flow_root_namespace *ns,
766+ u8 namespace, u8 num_actions,
767+ void *modify_actions,
768+ struct mlx5_modify_hdr *modify_hdr)
722769 {
723770 u32 out[MLX5_ST_SZ_DW(alloc_modify_header_context_out)];
724771 int max_actions, actions_size, inlen, err;
772+ struct mlx5_core_dev *dev = ns->dev;
725773 void *actions_in;
726774 u8 table_type;
727775 u32 *in;
@@ -772,26 +820,26 @@ int mlx5_modify_header_alloc(struct mlx5_core_dev *dev, · 44 unmodified lines
772820 memset(out, 0, sizeof(out));
773821 err = mlx5_cmd_exec(dev, in, inlen, out, sizeof(out));
774822
775− *modify_header_id = MLX5_GET(alloc_modify_header_context_out, out, modify_header_id);
823+ modify_hdr->id = MLX5_GET(alloc_modify_header_context_out, out, modify_header_id);
776824 kfree(in);
777825 return err;
778826 }
779−EXPORT_SYMBOL(mlx5_modify_header_alloc);
780827
781−void mlx5_modify_header_dealloc(struct mlx5_core_dev *dev, u32 modify_header_id)
828+static void mlx5_cmd_modify_header_dealloc(struct mlx5_flow_root_namespace *ns,
829+ struct mlx5_modify_hdr *modify_hdr)
782830 {
783831 u32 in[MLX5_ST_SZ_DW(dealloc_modify_header_context_in)];
784832 u32 out[MLX5_ST_SZ_DW(dealloc_modify_header_context_out)];
833+ struct mlx5_core_dev *dev = ns->dev;
785834
786835 memset(in, 0, sizeof(in));
787836 MLX5_SET(dealloc_modify_header_context_in, in, opcode,
788837 MLX5_CMD_OP_DEALLOC_MODIFY_HEADER_CONTEXT);
789838 MLX5_SET(dealloc_modify_header_context_in, in, modify_header_id,
790− modify_header_id);
839+ modify_hdr->id);
791840
792841 mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
793842 }
794−EXPORT_SYMBOL(mlx5_modify_header_dealloc);
795843
796844 static const struct mlx5_flow_cmds mlx5_flow_cmds = {
797845 .create_flow_table = mlx5_cmd_create_flow_table,
@@ -803,6 +851,13 @@ static const struct mlx5_flow_cmds mlx5_flow_cmds = { · 5 unmodified lines
803851 .update_fte = mlx5_cmd_update_fte,
804852 .delete_fte = mlx5_cmd_delete_fte,
805853 .update_root_ft = mlx5_cmd_update_root_ft,
854+ .packet_reformat_alloc = mlx5_cmd_packet_reformat_alloc,
855+ .packet_reformat_dealloc = mlx5_cmd_packet_reformat_dealloc,
856+ .modify_header_alloc = mlx5_cmd_modify_header_alloc,
857+ .modify_header_dealloc = mlx5_cmd_modify_header_dealloc,
858+ .set_peer = mlx5_cmd_stub_set_peer,
859+ .create_ns = mlx5_cmd_stub_create_ns,
860+ .destroy_ns = mlx5_cmd_stub_destroy_ns,
806861 };
807862
808863 static const struct mlx5_flow_cmds mlx5_flow_cmd_stubs = {
@@ -815,9 +870,16 @@ static const struct mlx5_flow_cmds mlx5_flow_cmd_stubs = { · 6 unmodified lines
815870 .update_fte = mlx5_cmd_stub_update_fte,
816871 .delete_fte = mlx5_cmd_stub_delete_fte,
817872 .update_root_ft = mlx5_cmd_stub_update_root_ft,
873+ .packet_reformat_alloc = mlx5_cmd_stub_packet_reformat_alloc,
874+ .packet_reformat_dealloc = mlx5_cmd_stub_packet_reformat_dealloc,
875+ .modify_header_alloc = mlx5_cmd_stub_modify_header_alloc,
876+ .modify_header_dealloc = mlx5_cmd_stub_modify_header_dealloc,
877+ .set_peer = mlx5_cmd_stub_set_peer,
878+ .create_ns = mlx5_cmd_stub_create_ns,
879+ .destroy_ns = mlx5_cmd_stub_destroy_ns,
818880 };
819881
820−static const struct mlx5_flow_cmds *mlx5_fs_cmd_get_fw_cmds(void)
882+const struct mlx5_flow_cmds *mlx5_fs_cmd_get_fw_cmds(void)
821883 {
822884 return &mlx5_flow_cmds;
823885 }
drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.h+25−0
@@ -75,6 +75,30 @@ struct mlx5_flow_cmds { · 74 unmodified lines
7575 struct mlx5_flow_table *ft,
7676 u32 underlay_qpn,
7777 bool disconnect);
78+
79+ int (*packet_reformat_alloc)(struct mlx5_flow_root_namespace *ns,
80+ int reformat_type,
81+ size_t size,
82+ void *reformat_data,
83+ enum mlx5_flow_namespace_type namespace,
84+ struct mlx5_pkt_reformat *pkt_reformat);
85+
86+ void (*packet_reformat_dealloc)(struct mlx5_flow_root_namespace *ns,
87+ struct mlx5_pkt_reformat *pkt_reformat);
88+
89+ int (*modify_header_alloc)(struct mlx5_flow_root_namespace *ns,
90+ u8 namespace, u8 num_actions,
91+ void *modify_actions,
92+ struct mlx5_modify_hdr *modify_hdr);
93+
94+ void (*modify_header_dealloc)(struct mlx5_flow_root_namespace *ns,
95+ struct mlx5_modify_hdr *modify_hdr);
96+
97+ int (*set_peer)(struct mlx5_flow_root_namespace *ns,
98+ struct mlx5_flow_root_namespace *peer_ns);
99+
100+ int (*create_ns)(struct mlx5_flow_root_namespace *ns);
101+ int (*destroy_ns)(struct mlx5_flow_root_namespace *ns);
78102 };
79103
80104 int mlx5_cmd_fc_alloc(struct mlx5_core_dev *dev, u32 *id);
@@ -90,5 +114,6 @@ int mlx5_cmd_fc_bulk_query(struct mlx5_core_dev *dev, u32 base_id, int bulk_len, · 9 unmodified lines
90114 u32 *out);
91115
92116 const struct mlx5_flow_cmds *mlx5_fs_cmd_get_default(enum fs_flow_table_type type);
117+const struct mlx5_flow_cmds *mlx5_fs_cmd_get_fw_cmds(void);
93118
94119 #endif
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c+159−1
@@ -1415,7 +1415,8 @@ static bool mlx5_flow_dests_cmp(struct mlx5_flow_destination *d1, · 1414 unmodified lines
14151415 ((d1->vport.flags & MLX5_FLOW_DEST_VPORT_VHCA_ID) ?
14161416 (d1->vport.vhca_id == d2->vport.vhca_id) : true) &&
14171417 ((d1->vport.flags & MLX5_FLOW_DEST_VPORT_REFORMAT_ID) ?
1418− (d1->vport.reformat_id == d2->vport.reformat_id) : true)) ||
1418+ (d1->vport.pkt_reformat->id ==
1419+ d2->vport.pkt_reformat->id) : true)) ||
14191420 (d1->type == MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE &&
14201421 d1->ft == d2->ft) ||
14211422 (d1->type == MLX5_FLOW_DESTINATION_TYPE_TIR &&
@@ -2888,3 +2889,160 @@ out: · 1466 unmodified lines
28882889 return err;
28892890 }
28902891 EXPORT_SYMBOL(mlx5_fs_remove_rx_underlay_qpn);
2892+
2893+static struct mlx5_flow_root_namespace
2894+*get_root_namespace(struct mlx5_core_dev *dev, enum mlx5_flow_namespace_type ns_type)
2895+{
2896+ struct mlx5_flow_namespace *ns;
2897+
2898+ if (ns_type == MLX5_FLOW_NAMESPACE_ESW_EGRESS ||
2899+ ns_type == MLX5_FLOW_NAMESPACE_ESW_INGRESS)
2900+ ns = mlx5_get_flow_vport_acl_namespace(dev, ns_type, 0);
2901+ else
2902+ ns = mlx5_get_flow_namespace(dev, ns_type);
2903+ if (!ns)
2904+ return NULL;
2905+
2906+ return find_root(&ns->node);
2907+}
2908+
2909+struct mlx5_modify_hdr *mlx5_modify_header_alloc(struct mlx5_core_dev *dev,
2910+ u8 ns_type, u8 num_actions,
2911+ void *modify_actions)
2912+{
2913+ struct mlx5_flow_root_namespace *root;
2914+ struct mlx5_modify_hdr *modify_hdr;
2915+ int err;
2916+
2917+ root = get_root_namespace(dev, ns_type);
2918+ if (!root)
2919+ return ERR_PTR(-EOPNOTSUPP);
2920+
2921+ modify_hdr = kzalloc(sizeof(*modify_hdr), GFP_KERNEL);
2922+ if (!modify_hdr)
2923+ return ERR_PTR(-ENOMEM);
2924+
2925+ modify_hdr->ns_type = ns_type;
2926+ err = root->cmds->modify_header_alloc(root, ns_type, num_actions,
2927+ modify_actions, modify_hdr);
2928+ if (err) {
2929+ kfree(modify_hdr);
2930+ return ERR_PTR(err);
2931+ }
2932+
2933+ return modify_hdr;
2934+}
2935+EXPORT_SYMBOL(mlx5_modify_header_alloc);
2936+
2937+void mlx5_modify_header_dealloc(struct mlx5_core_dev *dev,
2938+ struct mlx5_modify_hdr *modify_hdr)
2939+{
2940+ struct mlx5_flow_root_namespace *root;
2941+
2942+ root = get_root_namespace(dev, modify_hdr->ns_type);
2943+ if (WARN_ON(!root))
2944+ return;
2945+ root->cmds->modify_header_dealloc(root, modify_hdr);
2946+ kfree(modify_hdr);
2947+}
2948+EXPORT_SYMBOL(mlx5_modify_header_dealloc);
2949+
2950+struct mlx5_pkt_reformat *mlx5_packet_reformat_alloc(struct mlx5_core_dev *dev,
2951+ int reformat_type,
2952+ size_t size,
2953+ void *reformat_data,
2954+ enum mlx5_flow_namespace_type ns_type)
2955+{
2956+ struct mlx5_pkt_reformat *pkt_reformat;
2957+ struct mlx5_flow_root_namespace *root;
2958+ int err;
2959+
2960+ root = get_root_namespace(dev, ns_type);
2961+ if (!root)
2962+ return ERR_PTR(-EOPNOTSUPP);
2963+
2964+ pkt_reformat = kzalloc(sizeof(*pkt_reformat), GFP_KERNEL);
2965+ if (!pkt_reformat)
2966+ return ERR_PTR(-ENOMEM);
2967+
2968+ pkt_reformat->ns_type = ns_type;
2969+ pkt_reformat->reformat_type = reformat_type;
2970+ err = root->cmds->packet_reformat_alloc(root, reformat_type, size,
2971+ reformat_data, ns_type,
2972+ pkt_reformat);
2973+ if (err) {
2974+ kfree(pkt_reformat);
2975+ return ERR_PTR(err);
2976+ }
2977+
2978+ return pkt_reformat;
2979+}
2980+EXPORT_SYMBOL(mlx5_packet_reformat_alloc);
2981+
2982+void mlx5_packet_reformat_dealloc(struct mlx5_core_dev *dev,
2983+ struct mlx5_pkt_reformat *pkt_reformat)
2984+{
2985+ struct mlx5_flow_root_namespace *root;
2986+
2987+ root = get_root_namespace(dev, pkt_reformat->ns_type);
2988+ if (WARN_ON(!root))
2989+ return;
2990+ root->cmds->packet_reformat_dealloc(root, pkt_reformat);
2991+ kfree(pkt_reformat);
2992+}
2993+EXPORT_SYMBOL(mlx5_packet_reformat_dealloc);
2994+
2995+int mlx5_flow_namespace_set_peer(struct mlx5_flow_root_namespace *ns,
2996+ struct mlx5_flow_root_namespace *peer_ns)
2997+{
2998+ if (peer_ns && ns->mode != peer_ns->mode) {
2999+ mlx5_core_err(ns->dev,
3000+ "Can't peer namespace of different steering mode\n");
3001+ return -EINVAL;
3002+ }
3003+
3004+ return ns->cmds->set_peer(ns, peer_ns);
3005+}
3006+
3007+/* This function should be called only at init stage of the namespace.
3008+ * It is not safe to call this function while steering operations
3009+ * are executed in the namespace.
3010+ */
3011+int mlx5_flow_namespace_set_mode(struct mlx5_flow_namespace *ns,
3012+ enum mlx5_flow_steering_mode mode)
3013+{
3014+ struct mlx5_flow_root_namespace *root;
3015+ const struct mlx5_flow_cmds *cmds;
3016+ int err;
3017+
3018+ root = find_root(&ns->node);
3019+ if (&root->ns != ns)
3020+ /* Can't set cmds to non root namespace */
3021+ return -EINVAL;
3022+
3023+ if (root->table_type != FS_FT_FDB)
3024+ return -EOPNOTSUPP;
3025+
3026+ if (root->mode == mode)
3027+ return 0;
3028+
3029+ if (mode == MLX5_FLOW_STEERING_MODE_SMFS)
3030+ cmds = mlx5_fs_cmd_get_dr_cmds();
3031+ else
3032+ cmds = mlx5_fs_cmd_get_fw_cmds();
3033+ if (!cmds)
3034+ return -EOPNOTSUPP;
3035+
3036+ err = cmds->create_ns(root);
3037+ if (err) {
3038+ mlx5_core_err(root->dev, "Failed to create flow namespace (%d)\n",
3039+ err);
3040+ return err;
3041+ }
3042+
3043+ root->cmds->destroy_ns(root);
3044+ root->cmds = cmds;
3045+ root->mode = mode;
3046+
3047+ return 0;
3048+}
drivers/net/ethernet/mellanox/mlx5/core/fs_core.h+38−1
@@ -37,6 +37,24 @@ · 36 unmodified lines
3737 #include <linux/mlx5/fs.h>
3838 #include <linux/rhashtable.h>
3939 #include <linux/llist.h>
40+#include <steering/fs_dr.h>
41+
42+struct mlx5_modify_hdr {
43+ enum mlx5_flow_namespace_type ns_type;
44+ union {
45+ struct mlx5_fs_dr_action action;
46+ u32 id;
47+ };
48+};
49+
50+struct mlx5_pkt_reformat {
51+ enum mlx5_flow_namespace_type ns_type;
52+ int reformat_type; /* from mlx5_ifc */
53+ union {
54+ struct mlx5_fs_dr_action action;
55+ u32 id;
56+ };
57+};
4058
4159 /* FS_TYPE_PRIO_CHAINS is a PRIO that will have namespaces only,
4260 * and those are in parallel to one another when going over them to connect
@@ -80,9 +98,15 @@ enum fs_fte_status { · 37 unmodified lines
8098 FS_FTE_STATUS_EXISTING = 1UL << 0,
8199 };
82100
101+enum mlx5_flow_steering_mode {
102+ MLX5_FLOW_STEERING_MODE_DMFS,
103+ MLX5_FLOW_STEERING_MODE_SMFS
104+};
105+
83106 struct mlx5_flow_steering {
84107 struct mlx5_core_dev *dev;
85− struct kmem_cache *fgs_cache;
108+ enum mlx5_flow_steering_mode mode;
109+ struct kmem_cache *fgs_cache;
86110 struct kmem_cache *ftes_cache;
87111 struct mlx5_flow_root_namespace *root_ns;
88112 struct mlx5_flow_root_namespace *fdb_root_ns;
@@ -128,6 +152,7 @@ struct mlx5_flow_handle { · 39 unmodified lines
128152 /* Type of children is mlx5_flow_group */
129153 struct mlx5_flow_table {
130154 struct fs_node node;
155+ struct mlx5_fs_dr_table fs_dr_table;
131156 u32 id;
132157 u16 vport;
133158 unsigned int max_fte;
@@ -168,6 +193,7 @@ struct mlx5_ft_underlay_qp { · 34 unmodified lines
168193 /* Type of children is mlx5_flow_rule */
169194 struct fs_fte {
170195 struct fs_node node;
196+ struct mlx5_fs_dr_rule fs_dr_rule;
171197 u32 val[MLX5_ST_SZ_DW_MATCH_PARAM];
172198 u32 dests_size;
173199 u32 index;
@@ -203,6 +229,7 @@ struct mlx5_flow_group_mask { · 29 unmodified lines
203229 /* Type of children is fs_fte */
204230 struct mlx5_flow_group {
205231 struct fs_node node;
232+ struct mlx5_fs_dr_matcher fs_dr_matcher;
206233 struct mlx5_flow_group_mask mask;
207234 u32 start_index;
208235 u32 max_ftes;
@@ -214,6 +241,8 @@ struct mlx5_flow_group { · 5 unmodified lines
214241
215242 struct mlx5_flow_root_namespace {
216243 struct mlx5_flow_namespace ns;
244+ enum mlx5_flow_steering_mode mode;
245+ struct mlx5_fs_dr_domain fs_dr_domain;
217246 enum fs_flow_table_type table_type;
218247 struct mlx5_core_dev *dev;
219248 struct mlx5_flow_table *root_ft;
@@ -231,6 +260,14 @@ void mlx5_fc_queue_stats_work(struct mlx5_core_dev *dev, · 11 unmodified lines
231260 void mlx5_fc_update_sampling_interval(struct mlx5_core_dev *dev,
232261 unsigned long interval);
233262
263+const struct mlx5_flow_cmds *mlx5_fs_cmd_get_fw_cmds(void);
264+
265+int mlx5_flow_namespace_set_peer(struct mlx5_flow_root_namespace *ns,
266+ struct mlx5_flow_root_namespace *peer_ns);
267+
268+int mlx5_flow_namespace_set_mode(struct mlx5_flow_namespace *ns,
269+ enum mlx5_flow_steering_mode mode);
270+
234271 int mlx5_init_fs(struct mlx5_core_dev *dev);
235272 void mlx5_cleanup_fs(struct mlx5_core_dev *dev);
236273
drivers/net/ethernet/mellanox/mlx5/core/lib/dm.c+223−0
@@ -0,0 +1,223 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+// Copyright (c) 2019 Mellanox Technologies
3+
4+#include <linux/mlx5/driver.h>
5+#include <linux/mlx5/device.h>
6+
7+#include "mlx5_core.h"
8+#include "lib/mlx5.h"
9+
10+struct mlx5_dm {
11+ /* protect access to icm bitmask */
12+ spinlock_t lock;
13+ unsigned long *steering_sw_icm_alloc_blocks;
14+ unsigned long *header_modify_sw_icm_alloc_blocks;
15+};
16+
17+struct mlx5_dm *mlx5_dm_create(struct mlx5_core_dev *dev)
18+{
19+ u64 header_modify_icm_blocks = 0;
20+ u64 steering_icm_blocks = 0;
21+ struct mlx5_dm *dm;
22+
23+ if (!(MLX5_CAP_GEN_64(dev, general_obj_types) & MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM))
24+ return 0;
25+
26+ dm = kzalloc(sizeof(*dm), GFP_KERNEL);
27+ if (!dm)
28+ return ERR_PTR(-ENOMEM);
29+
30+ spin_lock_init(&dm->lock);
31+
32+ if (MLX5_CAP64_DEV_MEM(dev, steering_sw_icm_start_address)) {
33+ steering_icm_blocks =
34+ BIT(MLX5_CAP_DEV_MEM(dev, log_steering_sw_icm_size) -
35+ MLX5_LOG_SW_ICM_BLOCK_SIZE(dev));
36+
37+ dm->steering_sw_icm_alloc_blocks =
38+ kcalloc(BITS_TO_LONGS(steering_icm_blocks),
39+ sizeof(unsigned long), GFP_KERNEL);
40+ if (!dm->steering_sw_icm_alloc_blocks)
41+ goto err_steering;
42+ }
43+
44+ if (MLX5_CAP64_DEV_MEM(dev, header_modify_sw_icm_start_address)) {
45+ header_modify_icm_blocks =
46+ BIT(MLX5_CAP_DEV_MEM(dev, log_header_modify_sw_icm_size) -
47+ MLX5_LOG_SW_ICM_BLOCK_SIZE(dev));
48+
49+ dm->header_modify_sw_icm_alloc_blocks =
50+ kcalloc(BITS_TO_LONGS(header_modify_icm_blocks),
51+ sizeof(unsigned long), GFP_KERNEL);
52+ if (!dm->header_modify_sw_icm_alloc_blocks)
53+ goto err_modify_hdr;
54+ }
55+
56+ return dm;
57+
58+err_modify_hdr:
59+ kfree(dm->steering_sw_icm_alloc_blocks);
60+
61+err_steering:
62+ kfree(dm);
63+
64+ return ERR_PTR(-ENOMEM);
65+}
66+
67+void mlx5_dm_cleanup(struct mlx5_core_dev *dev)
68+{
69+ struct mlx5_dm *dm = dev->dm;
70+
71+ if (!dev->dm)
72+ return;
73+
74+ if (dm->steering_sw_icm_alloc_blocks) {
75+ WARN_ON(!bitmap_empty(dm->steering_sw_icm_alloc_blocks,
76+ BIT(MLX5_CAP_DEV_MEM(dev, log_steering_sw_icm_size) -
77+ MLX5_LOG_SW_ICM_BLOCK_SIZE(dev))));
78+ kfree(dm->steering_sw_icm_alloc_blocks);
79+ }
80+
81+ if (dm->header_modify_sw_icm_alloc_blocks) {
82+ WARN_ON(!bitmap_empty(dm->header_modify_sw_icm_alloc_blocks,
83+ BIT(MLX5_CAP_DEV_MEM(dev,
84+ log_header_modify_sw_icm_size) -
85+ MLX5_LOG_SW_ICM_BLOCK_SIZE(dev))));
86+ kfree(dm->header_modify_sw_icm_alloc_blocks);
87+ }
88+
89+ kfree(dm);
90+}
91+
92+int mlx5_dm_sw_icm_alloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type,
93+ u64 length, u16 uid, phys_addr_t *addr, u32 *obj_id)
94+{
95+ u32 num_blocks = DIV_ROUND_UP_ULL(length, MLX5_SW_ICM_BLOCK_SIZE(dev));
96+ u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {};
97+ u32 in[MLX5_ST_SZ_DW(create_sw_icm_in)] = {};
98+ struct mlx5_dm *dm = dev->dm;
99+ unsigned long *block_map;
100+ u64 icm_start_addr;
101+ u32 log_icm_size;
102+ u32 max_blocks;
103+ u64 block_idx;
104+ void *sw_icm;
105+ int ret;
106+
107+ if (!dev->dm)
108+ return -EOPNOTSUPP;
109+
110+ if (!length || (length & (length - 1)) ||
111+ length & (MLX5_SW_ICM_BLOCK_SIZE(dev) - 1))
112+ return -EINVAL;
113+
114+ MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
115+ MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
116+ MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, MLX5_OBJ_TYPE_SW_ICM);
117+ MLX5_SET(general_obj_in_cmd_hdr, in, uid, uid);
118+
119+ switch (type) {
120+ case MLX5_SW_ICM_TYPE_STEERING:
121+ icm_start_addr = MLX5_CAP64_DEV_MEM(dev, steering_sw_icm_start_address);
122+ log_icm_size = MLX5_CAP_DEV_MEM(dev, log_steering_sw_icm_size);
123+ block_map = dm->steering_sw_icm_alloc_blocks;
124+ break;
125+ case MLX5_SW_ICM_TYPE_HEADER_MODIFY:
126+ icm_start_addr = MLX5_CAP64_DEV_MEM(dev, header_modify_sw_icm_start_address);
127+ log_icm_size = MLX5_CAP_DEV_MEM(dev,
128+ log_header_modify_sw_icm_size);
129+ block_map = dm->header_modify_sw_icm_alloc_blocks;
130+ break;
131+ default:
132+ return -EINVAL;
133+ }
134+
135+ if (!block_map)
136+ return -EOPNOTSUPP;
137+
138+ max_blocks = BIT(log_icm_size - MLX5_LOG_SW_ICM_BLOCK_SIZE(dev));
139+ spin_lock(&dm->lock);
140+ block_idx = bitmap_find_next_zero_area(block_map,
141+ max_blocks,
142+ 0,
143+ num_blocks, 0);
144+
145+ if (block_idx < max_blocks)
146+ bitmap_set(block_map,
147+ block_idx, num_blocks);
148+
149+ spin_unlock(&dm->lock);
150+
151+ if (block_idx >= max_blocks)
152+ return -ENOMEM;
153+
154+ sw_icm = MLX5_ADDR_OF(create_sw_icm_in, in, sw_icm);
155+ icm_start_addr += block_idx << MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
156+ MLX5_SET64(sw_icm, sw_icm, sw_icm_start_addr,
157+ icm_start_addr);
158+ MLX5_SET(sw_icm, sw_icm, log_sw_icm_size, ilog2(length));
159+
160+ ret = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
161+ if (ret) {
162+ spin_lock(&dm->lock);
163+ bitmap_clear(block_map,
164+ block_idx, num_blocks);
165+ spin_unlock(&dm->lock);
166+
167+ return ret;
168+ }
169+
170+ *addr = icm_start_addr;
171+ *obj_id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
172+
173+ return 0;
174+}
175+EXPORT_SYMBOL_GPL(mlx5_dm_sw_icm_alloc);
176+
177+int mlx5_dm_sw_icm_dealloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type,
178+ u64 length, u16 uid, phys_addr_t addr, u32 obj_id)
179+{
180+ u32 num_blocks = DIV_ROUND_UP_ULL(length, MLX5_SW_ICM_BLOCK_SIZE(dev));
181+ u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {};
182+ u32 in[MLX5_ST_SZ_DW(general_obj_in_cmd_hdr)] = {};
183+ struct mlx5_dm *dm = dev->dm;
184+ unsigned long *block_map;
185+ u64 icm_start_addr;
186+ u64 start_idx;
187+ int err;
188+
189+ if (!dev->dm)
190+ return -EOPNOTSUPP;
191+
192+ switch (type) {
193+ case MLX5_SW_ICM_TYPE_STEERING:
194+ icm_start_addr = MLX5_CAP64_DEV_MEM(dev, steering_sw_icm_start_address);
195+ block_map = dm->steering_sw_icm_alloc_blocks;
196+ break;
197+ case MLX5_SW_ICM_TYPE_HEADER_MODIFY:
198+ icm_start_addr = MLX5_CAP64_DEV_MEM(dev, header_modify_sw_icm_start_address);
199+ block_map = dm->header_modify_sw_icm_alloc_blocks;
200+ break;
201+ default:
202+ return -EINVAL;
203+ }
204+
205+ MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
206+ MLX5_CMD_OP_DESTROY_GENERAL_OBJECT);
207+ MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, MLX5_OBJ_TYPE_SW_ICM);
208+ MLX5_SET(general_obj_in_cmd_hdr, in, obj_id, obj_id);
209+ MLX5_SET(general_obj_in_cmd_hdr, in, uid, uid);
210+
211+ err = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
212+ if (err)
213+ return err;
214+
215+ start_idx = (addr - icm_start_addr) >> MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
216+ spin_lock(&dm->lock);
217+ bitmap_clear(block_map,
218+ start_idx, num_blocks);
219+ spin_unlock(&dm->lock);
220+
221+ return 0;
222+}
223+EXPORT_SYMBOL_GPL(mlx5_dm_sw_icm_dealloc);
drivers/net/ethernet/mellanox/mlx5/core/main.c+5−0
@@ -876,6 +876,10 @@ static int mlx5_init_once(struct mlx5_core_dev *dev) · 875 unmodified lines
876876 goto err_eswitch_cleanup;
877877 }
878878
879+ dev->dm = mlx5_dm_create(dev);
880+ if (IS_ERR(dev->dm))
881+ mlx5_core_warn(dev, "Failed to init device memory%d\n", err);
882+
879883 dev->tracer = mlx5_fw_tracer_create(dev);
880884 dev->hv_vhca = mlx5_hv_vhca_create(dev);
881885
@@ -910,6 +914,7 @@ static void mlx5_cleanup_once(struct mlx5_core_dev *dev) · 28 unmodified lines
910914 {
911915 mlx5_hv_vhca_destroy(dev->hv_vhca);
912916 mlx5_fw_tracer_destroy(dev->tracer);
917+ mlx5_dm_cleanup(dev);
913918 mlx5_fpga_cleanup(dev);
914919 mlx5_eswitch_cleanup(dev->priv.eswitch);
915920 mlx5_sriov_cleanup(dev);
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h+3−0
@@ -198,6 +198,9 @@ int mlx5_set_mtpps(struct mlx5_core_dev *mdev, u32 *mtpps, u32 mtpps_size); · 197 unmodified lines
198198 int mlx5_query_mtppse(struct mlx5_core_dev *mdev, u8 pin, u8 *arm, u8 *mode);
199199 int mlx5_set_mtppse(struct mlx5_core_dev *mdev, u8 pin, u8 arm, u8 mode);
200200
201+struct mlx5_dm *mlx5_dm_create(struct mlx5_core_dev *dev);
202+void mlx5_dm_cleanup(struct mlx5_core_dev *dev);
203+
201204 #define MLX5_PPS_CAP(mdev) (MLX5_CAP_GEN((mdev), pps) && \
202205 MLX5_CAP_GEN((mdev), pps_modify) && \
203206 MLX5_CAP_MCAM_FEATURE((mdev), mtpps_fs) && \
drivers/net/ethernet/mellanox/mlx5/core/rdma.c+5−3
@@ -14,9 +14,6 @@ static void mlx5_rdma_disable_roce_steering(struct mlx5_core_dev *dev) · 13 unmodified lines
1414 {
1515 struct mlx5_core_roce *roce = &dev->priv.roce;
1616
17− if (!roce->ft)
18− return;
19−
2017 mlx5_del_flow_rules(roce->allow_rule);
2118 mlx5_destroy_flow_group(roce->fg);
2219 mlx5_destroy_flow_table(roce->ft);
@@ -145,6 +142,11 @@ static int mlx5_rdma_add_roce_addr(struct mlx5_core_dev *dev) · 122 unmodified lines
145142
146143 void mlx5_rdma_disable_roce(struct mlx5_core_dev *dev)
147144 {
145+ struct mlx5_core_roce *roce = &dev->priv.roce;
146+
147+ if (!roce->ft)
148+ return;
149+
148150 mlx5_rdma_disable_roce_steering(dev);
149151 mlx5_rdma_del_roce_addr(dev);
150152 mlx5_nic_vport_disable_roce(dev);
drivers/net/ethernet/mellanox/mlx5/core/steering/Makefile+2−0
@@ -0,0 +1,2 @@
1+# SPDX-License-Identifier: GPL-2.0-only
2+subdir-ccflags-y += -I$(src)/..
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_action.c+1588−0
@@ -0,0 +1,1588 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+/* Copyright (c) 2019 Mellanox Technologies. */
3+
4+#include "dr_types.h"
5+
6+enum dr_action_domain {
7+ DR_ACTION_DOMAIN_NIC_INGRESS,
8+ DR_ACTION_DOMAIN_NIC_EGRESS,
9+ DR_ACTION_DOMAIN_FDB_INGRESS,
10+ DR_ACTION_DOMAIN_FDB_EGRESS,
11+ DR_ACTION_DOMAIN_MAX,
12+};
13+
14+enum dr_action_valid_state {
15+ DR_ACTION_STATE_ERR,
16+ DR_ACTION_STATE_NO_ACTION,
17+ DR_ACTION_STATE_REFORMAT,
18+ DR_ACTION_STATE_MODIFY_HDR,
19+ DR_ACTION_STATE_MODIFY_VLAN,
20+ DR_ACTION_STATE_NON_TERM,
21+ DR_ACTION_STATE_TERM,
22+ DR_ACTION_STATE_MAX,
23+};
24+
25+static const enum dr_action_valid_state
26+next_action_state[DR_ACTION_DOMAIN_MAX][DR_ACTION_STATE_MAX][DR_ACTION_TYP_MAX] = {
27+ [DR_ACTION_DOMAIN_NIC_INGRESS] = {
28+ [DR_ACTION_STATE_NO_ACTION] = {
29+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
30+ [DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
31+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
32+ [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_NON_TERM,
33+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
34+ [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_REFORMAT,
35+ [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_REFORMAT,
36+ [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
37+ [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
38+ },
39+ [DR_ACTION_STATE_REFORMAT] = {
40+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
41+ [DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
42+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
43+ [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_REFORMAT,
44+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_REFORMAT,
45+ [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
46+ [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
47+ },
48+ [DR_ACTION_STATE_MODIFY_HDR] = {
49+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
50+ [DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
51+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
52+ [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_MODIFY_HDR,
53+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_HDR,
54+ },
55+ [DR_ACTION_STATE_MODIFY_VLAN] = {
56+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
57+ [DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
58+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
59+ [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_MODIFY_VLAN,
60+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_VLAN,
61+ [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
62+ [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
63+ },
64+ [DR_ACTION_STATE_NON_TERM] = {
65+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
66+ [DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
67+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
68+ [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_NON_TERM,
69+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
70+ [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_REFORMAT,
71+ [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_REFORMAT,
72+ [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
73+ [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
74+ },
75+ [DR_ACTION_STATE_TERM] = {
76+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_TERM,
77+ },
78+ },
79+ [DR_ACTION_DOMAIN_NIC_EGRESS] = {
80+ [DR_ACTION_STATE_NO_ACTION] = {
81+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
82+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
83+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
84+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
85+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
86+ [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
87+ [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
88+ },
89+ [DR_ACTION_STATE_REFORMAT] = {
90+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
91+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
92+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_REFORMAT,
93+ },
94+ [DR_ACTION_STATE_MODIFY_HDR] = {
95+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
96+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
97+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_HDR,
98+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
99+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
100+ [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
101+ },
102+ [DR_ACTION_STATE_MODIFY_VLAN] = {
103+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
104+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
105+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_VLAN,
106+ [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
107+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
108+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
109+ },
110+ [DR_ACTION_STATE_NON_TERM] = {
111+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
112+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
113+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
114+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
115+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
116+ [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
117+ [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
118+ },
119+ [DR_ACTION_STATE_TERM] = {
120+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_TERM,
121+ },
122+ },
123+ [DR_ACTION_DOMAIN_FDB_INGRESS] = {
124+ [DR_ACTION_STATE_NO_ACTION] = {
125+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
126+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
127+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
128+ [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_REFORMAT,
129+ [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_REFORMAT,
130+ [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
131+ [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
132+ [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
133+ },
134+ [DR_ACTION_STATE_REFORMAT] = {
135+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
136+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
137+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_REFORMAT,
138+ [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
139+ [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
140+ [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
141+ },
142+ [DR_ACTION_STATE_MODIFY_HDR] = {
143+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
144+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
145+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_HDR,
146+ [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
147+ },
148+ [DR_ACTION_STATE_MODIFY_VLAN] = {
149+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
150+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
151+ [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
152+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_VLAN,
153+ [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
154+ [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
155+ },
156+ [DR_ACTION_STATE_NON_TERM] = {
157+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
158+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
159+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
160+ [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_REFORMAT,
161+ [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_REFORMAT,
162+ [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
163+ [DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
164+ [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
165+ },
166+ [DR_ACTION_STATE_TERM] = {
167+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_TERM,
168+ },
169+ },
170+ [DR_ACTION_DOMAIN_FDB_EGRESS] = {
171+ [DR_ACTION_STATE_NO_ACTION] = {
172+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
173+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
174+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
175+ [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
176+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
177+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
178+ [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
179+ [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
180+ },
181+ [DR_ACTION_STATE_REFORMAT] = {
182+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
183+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
184+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_REFORMAT,
185+ [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
186+ },
187+ [DR_ACTION_STATE_MODIFY_HDR] = {
188+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
189+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
190+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_HDR,
191+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
192+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
193+ [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
194+ [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
195+ },
196+ [DR_ACTION_STATE_MODIFY_VLAN] = {
197+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
198+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
199+ [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
200+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_VLAN,
201+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
202+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
203+ [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
204+ },
205+ [DR_ACTION_STATE_NON_TERM] = {
206+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
207+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
208+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
209+ [DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
210+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
211+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
212+ [DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
213+ [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
214+ },
215+ [DR_ACTION_STATE_TERM] = {
216+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_TERM,
217+ },
218+ },
219+};
220+
221+struct dr_action_modify_field_conv {
222+ u16 hw_field;
223+ u8 start;
224+ u8 end;
225+ u8 l3_type;
226+ u8 l4_type;
227+};
228+
229+static const struct dr_action_modify_field_conv dr_action_conv_arr[] = {
230+ [MLX5_ACTION_IN_FIELD_OUT_SMAC_47_16] = {
231+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L2_1, .start = 16, .end = 47,
232+ },
233+ [MLX5_ACTION_IN_FIELD_OUT_SMAC_15_0] = {
234+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L2_1, .start = 0, .end = 15,
235+ },
236+ [MLX5_ACTION_IN_FIELD_OUT_ETHERTYPE] = {
237+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L2_2, .start = 32, .end = 47,
238+ },
239+ [MLX5_ACTION_IN_FIELD_OUT_DMAC_47_16] = {
240+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L2_0, .start = 16, .end = 47,
241+ },
242+ [MLX5_ACTION_IN_FIELD_OUT_DMAC_15_0] = {
243+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L2_0, .start = 0, .end = 15,
244+ },
245+ [MLX5_ACTION_IN_FIELD_OUT_IP_DSCP] = {
246+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_1, .start = 0, .end = 5,
247+ },
248+ [MLX5_ACTION_IN_FIELD_OUT_TCP_FLAGS] = {
249+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_0, .start = 48, .end = 56,
250+ .l4_type = MLX5DR_ACTION_MDFY_HW_HDR_L4_TCP,
251+ },
252+ [MLX5_ACTION_IN_FIELD_OUT_TCP_SPORT] = {
253+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_0, .start = 0, .end = 15,
254+ .l4_type = MLX5DR_ACTION_MDFY_HW_HDR_L4_TCP,
255+ },
256+ [MLX5_ACTION_IN_FIELD_OUT_TCP_DPORT] = {
257+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_0, .start = 16, .end = 31,
258+ .l4_type = MLX5DR_ACTION_MDFY_HW_HDR_L4_TCP,
259+ },
260+ [MLX5_ACTION_IN_FIELD_OUT_IP_TTL] = {
261+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_1, .start = 8, .end = 15,
262+ .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV4,
263+ },
264+ [MLX5_ACTION_IN_FIELD_OUT_IPV6_HOPLIMIT] = {
265+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_1, .start = 8, .end = 15,
266+ .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
267+ },
268+ [MLX5_ACTION_IN_FIELD_OUT_UDP_SPORT] = {
269+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_0, .start = 0, .end = 15,
270+ .l4_type = MLX5DR_ACTION_MDFY_HW_HDR_L4_UDP,
271+ },
272+ [MLX5_ACTION_IN_FIELD_OUT_UDP_DPORT] = {
273+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_0, .start = 16, .end = 31,
274+ .l4_type = MLX5DR_ACTION_MDFY_HW_HDR_L4_UDP,
275+ },
276+ [MLX5_ACTION_IN_FIELD_OUT_SIPV6_127_96] = {
277+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_3, .start = 32, .end = 63,
278+ .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
279+ },
280+ [MLX5_ACTION_IN_FIELD_OUT_SIPV6_95_64] = {
281+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_3, .start = 0, .end = 31,
282+ .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
283+ },
284+ [MLX5_ACTION_IN_FIELD_OUT_SIPV6_63_32] = {
285+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_4, .start = 32, .end = 63,
286+ .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
287+ },
288+ [MLX5_ACTION_IN_FIELD_OUT_SIPV6_31_0] = {
289+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_4, .start = 0, .end = 31,
290+ .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
291+ },
292+ [MLX5_ACTION_IN_FIELD_OUT_DIPV6_127_96] = {
293+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_0, .start = 32, .end = 63,
294+ .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
295+ },
296+ [MLX5_ACTION_IN_FIELD_OUT_DIPV6_95_64] = {
297+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_0, .start = 0, .end = 31,
298+ .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
299+ },
300+ [MLX5_ACTION_IN_FIELD_OUT_DIPV6_63_32] = {
301+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_2, .start = 32, .end = 63,
302+ .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
303+ },
304+ [MLX5_ACTION_IN_FIELD_OUT_DIPV6_31_0] = {
305+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_2, .start = 0, .end = 31,
306+ .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6,
307+ },
308+ [MLX5_ACTION_IN_FIELD_OUT_SIPV4] = {
309+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_0, .start = 0, .end = 31,
310+ .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV4,
311+ },
312+ [MLX5_ACTION_IN_FIELD_OUT_DIPV4] = {
313+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L3_0, .start = 32, .end = 63,
314+ .l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV4,
315+ },
316+ [MLX5_ACTION_IN_FIELD_METADATA_REG_A] = {
317+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_METADATA, .start = 0, .end = 31,
318+ },
319+ [MLX5_ACTION_IN_FIELD_METADATA_REG_B] = {
320+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_METADATA, .start = 32, .end = 63,
321+ },
322+ [MLX5_ACTION_IN_FIELD_METADATA_REG_C_0] = {
323+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_REG_0, .start = 32, .end = 63,
324+ },
325+ [MLX5_ACTION_IN_FIELD_METADATA_REG_C_1] = {
326+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_REG_0, .start = 0, .end = 31,
327+ },
328+ [MLX5_ACTION_IN_FIELD_METADATA_REG_C_2] = {
329+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_REG_1, .start = 32, .end = 63,
330+ },
331+ [MLX5_ACTION_IN_FIELD_METADATA_REG_C_3] = {
332+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_REG_1, .start = 0, .end = 31,
333+ },
334+ [MLX5_ACTION_IN_FIELD_METADATA_REG_C_4] = {
335+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_REG_2, .start = 32, .end = 63,
336+ },
337+ [MLX5_ACTION_IN_FIELD_METADATA_REG_C_5] = {
338+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_REG_2, .start = 0, .end = 31,
339+ },
340+ [MLX5_ACTION_IN_FIELD_OUT_TCP_SEQ_NUM] = {
341+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_1, .start = 32, .end = 63,
342+ },
343+ [MLX5_ACTION_IN_FIELD_OUT_TCP_ACK_NUM] = {
344+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L4_1, .start = 0, .end = 31,
345+ },
346+ [MLX5_ACTION_IN_FIELD_OUT_FIRST_VID] = {
347+ .hw_field = MLX5DR_ACTION_MDFY_HW_FLD_L2_2, .start = 0, .end = 15,
348+ },
349+};
350+
351+#define MAX_VLANS 2
352+struct dr_action_vlan_info {
353+ int count;
354+ u32 headers[MAX_VLANS];
355+};
356+
357+struct dr_action_apply_attr {
358+ u32 modify_index;
359+ u16 modify_actions;
360+ u32 decap_index;
361+ u16 decap_actions;
362+ u8 decap_with_vlan:1;
363+ u64 final_icm_addr;
364+ u32 flow_tag;
365+ u32 ctr_id;
366+ u16 gvmi;
367+ u16 hit_gvmi;
368+ u32 reformat_id;
369+ u32 reformat_size;
370+ struct dr_action_vlan_info vlans;
371+};
372+
373+static int
374+dr_action_reformat_to_action_type(enum mlx5dr_action_reformat_type reformat_type,
375+ enum mlx5dr_action_type *action_type)
376+{
377+ switch (reformat_type) {
378+ case DR_ACTION_REFORMAT_TYP_TNL_L2_TO_L2:
379+ *action_type = DR_ACTION_TYP_TNL_L2_TO_L2;
380+ break;
381+ case DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L2:
382+ *action_type = DR_ACTION_TYP_L2_TO_TNL_L2;
383+ break;
384+ case DR_ACTION_REFORMAT_TYP_TNL_L3_TO_L2:
385+ *action_type = DR_ACTION_TYP_TNL_L3_TO_L2;
386+ break;
387+ case DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L3:
388+ *action_type = DR_ACTION_TYP_L2_TO_TNL_L3;
389+ break;
390+ default:
391+ return -EINVAL;
392+ }
393+
394+ return 0;
395+}
396+
397+static void dr_actions_init_next_ste(u8 **last_ste,
398+ u32 *added_stes,
399+ enum mlx5dr_ste_entry_type entry_type,
400+ u16 gvmi)
401+{
402+ (*added_stes)++;
403+ *last_ste += DR_STE_SIZE;
404+ mlx5dr_ste_init(*last_ste, MLX5DR_STE_LU_TYPE_DONT_CARE, entry_type, gvmi);
405+}
406+
407+static void dr_actions_apply_tx(struct mlx5dr_domain *dmn,
408+ u8 *action_type_set,
409+ u8 *last_ste,
410+ struct dr_action_apply_attr *attr,
411+ u32 *added_stes)
412+{
413+ bool encap = action_type_set[DR_ACTION_TYP_L2_TO_TNL_L2] ||
414+ action_type_set[DR_ACTION_TYP_L2_TO_TNL_L3];
415+
416+ /* We want to make sure the modify header comes before L2
417+ * encapsulation. The reason for that is that we support
418+ * modify headers for outer headers only
419+ */
420+ if (action_type_set[DR_ACTION_TYP_MODIFY_HDR]) {
421+ mlx5dr_ste_set_entry_type(last_ste, MLX5DR_STE_TYPE_MODIFY_PKT);
422+ mlx5dr_ste_set_rewrite_actions(last_ste,
423+ attr->modify_actions,
424+ attr->modify_index);
425+ }
426+
427+ if (action_type_set[DR_ACTION_TYP_PUSH_VLAN]) {
428+ int i;
429+
430+ for (i = 0; i < attr->vlans.count; i++) {
431+ if (i || action_type_set[DR_ACTION_TYP_MODIFY_HDR])
432+ dr_actions_init_next_ste(&last_ste,
433+ added_stes,
434+ MLX5DR_STE_TYPE_TX,
435+ attr->gvmi);
436+
437+ mlx5dr_ste_set_tx_push_vlan(last_ste,
438+ attr->vlans.headers[i],
439+ encap);
440+ }
441+ }
442+
443+ if (encap) {
444+ /* Modify header and encapsulation require a different STEs.
445+ * Since modify header STE format doesn't support encapsulation
446+ * tunneling_action.
447+ */
448+ if (action_type_set[DR_ACTION_TYP_MODIFY_HDR] ||
449+ action_type_set[DR_ACTION_TYP_PUSH_VLAN])
450+ dr_actions_init_next_ste(&last_ste,
451+ added_stes,
452+ MLX5DR_STE_TYPE_TX,
453+ attr->gvmi);
454+
455+ mlx5dr_ste_set_tx_encap(last_ste,
456+ attr->reformat_id,
457+ attr->reformat_size,
458+ action_type_set[DR_ACTION_TYP_L2_TO_TNL_L3]);
459+ /* Whenever prio_tag_required enabled, we can be sure that the
460+ * previous table (ACL) already push vlan to our packet,
461+ * And due to HW limitation we need to set this bit, otherwise
462+ * push vlan + reformat will not work.
463+ */
464+ if (MLX5_CAP_GEN(dmn->mdev, prio_tag_required))
465+ mlx5dr_ste_set_go_back_bit(last_ste);
466+ }
467+
468+ if (action_type_set[DR_ACTION_TYP_CTR])
469+ mlx5dr_ste_set_counter_id(last_ste, attr->ctr_id);
470+}
471+
472+static void dr_actions_apply_rx(u8 *action_type_set,
473+ u8 *last_ste,
474+ struct dr_action_apply_attr *attr,
475+ u32 *added_stes)
476+{
477+ if (action_type_set[DR_ACTION_TYP_CTR])
478+ mlx5dr_ste_set_counter_id(last_ste, attr->ctr_id);
479+
480+ if (action_type_set[DR_ACTION_TYP_TNL_L3_TO_L2]) {
481+ mlx5dr_ste_set_entry_type(last_ste, MLX5DR_STE_TYPE_MODIFY_PKT);
482+ mlx5dr_ste_set_rx_decap_l3(last_ste, attr->decap_with_vlan);
483+ mlx5dr_ste_set_rewrite_actions(last_ste,
484+ attr->decap_actions,
485+ attr->decap_index);
486+ }
487+
488+ if (action_type_set[DR_ACTION_TYP_TNL_L2_TO_L2])
489+ mlx5dr_ste_set_rx_decap(last_ste);
490+
491+ if (action_type_set[DR_ACTION_TYP_POP_VLAN]) {
492+ int i;
493+
494+ for (i = 0; i < attr->vlans.count; i++) {
495+ if (i ||
496+ action_type_set[DR_ACTION_TYP_TNL_L2_TO_L2] ||
497+ action_type_set[DR_ACTION_TYP_TNL_L3_TO_L2])
498+ dr_actions_init_next_ste(&last_ste,
499+ added_stes,
500+ MLX5DR_STE_TYPE_RX,
501+ attr->gvmi);
502+
503+ mlx5dr_ste_set_rx_pop_vlan(last_ste);
504+ }
505+ }
506+
507+ if (action_type_set[DR_ACTION_TYP_MODIFY_HDR]) {
508+ if (mlx5dr_ste_get_entry_type(last_ste) == MLX5DR_STE_TYPE_MODIFY_PKT)
509+ dr_actions_init_next_ste(&last_ste,
510+ added_stes,
511+ MLX5DR_STE_TYPE_MODIFY_PKT,
512+ attr->gvmi);
513+ else
514+ mlx5dr_ste_set_entry_type(last_ste, MLX5DR_STE_TYPE_MODIFY_PKT);
515+
516+ mlx5dr_ste_set_rewrite_actions(last_ste,
517+ attr->modify_actions,
518+ attr->modify_index);
519+ }
520+
521+ if (action_type_set[DR_ACTION_TYP_TAG]) {
522+ if (mlx5dr_ste_get_entry_type(last_ste) == MLX5DR_STE_TYPE_MODIFY_PKT)
523+ dr_actions_init_next_ste(&last_ste,
524+ added_stes,
525+ MLX5DR_STE_TYPE_RX,
526+ attr->gvmi);
527+
528+ mlx5dr_ste_rx_set_flow_tag(last_ste, attr->flow_tag);
529+ }
530+}
531+
532+/* Apply the actions on the rule STE array starting from the last_ste.
533+ * Actions might require more than one STE, new_num_stes will return
534+ * the new size of the STEs array, rule with actions.
535+ */
536+static void dr_actions_apply(struct mlx5dr_domain *dmn,
537+ enum mlx5dr_ste_entry_type ste_type,
538+ u8 *action_type_set,
539+ u8 *last_ste,
540+ struct dr_action_apply_attr *attr,
541+ u32 *new_num_stes)
542+{
543+ u32 added_stes = 0;
544+
545+ if (ste_type == MLX5DR_STE_TYPE_RX)
546+ dr_actions_apply_rx(action_type_set, last_ste, attr, &added_stes);
547+ else
548+ dr_actions_apply_tx(dmn, action_type_set, last_ste, attr, &added_stes);
549+
550+ last_ste += added_stes * DR_STE_SIZE;
551+ *new_num_stes += added_stes;
552+
553+ mlx5dr_ste_set_hit_gvmi(last_ste, attr->hit_gvmi);
554+ mlx5dr_ste_set_hit_addr(last_ste, attr->final_icm_addr, 1);
555+}
556+
557+static enum dr_action_domain
558+dr_action_get_action_domain(enum mlx5dr_domain_type domain,
559+ enum mlx5dr_ste_entry_type ste_type)
560+{
561+ switch (domain) {
562+ case MLX5DR_DOMAIN_TYPE_NIC_RX:
563+ return DR_ACTION_DOMAIN_NIC_INGRESS;
564+ case MLX5DR_DOMAIN_TYPE_NIC_TX:
565+ return DR_ACTION_DOMAIN_NIC_EGRESS;
566+ case MLX5DR_DOMAIN_TYPE_FDB:
567+ if (ste_type == MLX5DR_STE_TYPE_RX)
568+ return DR_ACTION_DOMAIN_FDB_INGRESS;
569+ return DR_ACTION_DOMAIN_FDB_EGRESS;
570+ default:
571+ WARN_ON(true);
572+ return DR_ACTION_DOMAIN_MAX;
573+ }
574+}
575+
576+static
577+int dr_action_validate_and_get_next_state(enum dr_action_domain action_domain,
578+ u32 action_type,
579+ u32 *state)
580+{
581+ u32 cur_state = *state;
582+
583+ /* Check action state machine is valid */
584+ *state = next_action_state[action_domain][cur_state][action_type];
585+
586+ if (*state == DR_ACTION_STATE_ERR)
587+ return -EOPNOTSUPP;
588+
589+ return 0;
590+}
591+
592+static int dr_action_handle_cs_recalc(struct mlx5dr_domain *dmn,
593+ struct mlx5dr_action *dest_action,
594+ u64 *final_icm_addr)
595+{
596+ int ret;
597+
598+ switch (dest_action->action_type) {
599+ case DR_ACTION_TYP_FT:
600+ /* Allow destination flow table only if table is a terminating
601+ * table, since there is an *assumption* that in such case FW
602+ * will recalculate the CS.
603+ */
604+ if (dest_action->dest_tbl.is_fw_tbl) {
605+ *final_icm_addr = dest_action->dest_tbl.fw_tbl.rx_icm_addr;
606+ } else {
607+ mlx5dr_dbg(dmn,
608+ "Destination FT should be terminating when modify TTL is used\n");
609+ return -EINVAL;
610+ }
611+ break;
612+
613+ case DR_ACTION_TYP_VPORT:
614+ /* If destination is vport we will get the FW flow table
615+ * that recalculates the CS and forwards to the vport.
616+ */
617+ ret = mlx5dr_domain_cache_get_recalc_cs_ft_addr(dest_action->vport.dmn,
618+ dest_action->vport.num,
619+ final_icm_addr);
620+ if (ret) {
621+ mlx5dr_err(dmn, "Failed to get FW cs recalc flow table\n");
622+ return ret;
623+ }
624+ break;
625+
626+ default:
627+ break;
628+ }
629+
630+ return 0;
631+}
632+
633+#define WITH_VLAN_NUM_HW_ACTIONS 6
634+
635+int mlx5dr_actions_build_ste_arr(struct mlx5dr_matcher *matcher,
636+ struct mlx5dr_matcher_rx_tx *nic_matcher,
637+ struct mlx5dr_action *actions[],
638+ u32 num_actions,
639+ u8 *ste_arr,
640+ u32 *new_hw_ste_arr_sz)
641+{
642+ struct mlx5dr_domain_rx_tx *nic_dmn = nic_matcher->nic_tbl->nic_dmn;
643+ bool rx_rule = nic_dmn->ste_type == MLX5DR_STE_TYPE_RX;
644+ struct mlx5dr_domain *dmn = matcher->tbl->dmn;
645+ u8 action_type_set[DR_ACTION_TYP_MAX] = {};
646+ struct mlx5dr_action *dest_action = NULL;
647+ u32 state = DR_ACTION_STATE_NO_ACTION;
648+ struct dr_action_apply_attr attr = {};
649+ enum dr_action_domain action_domain;
650+ bool recalc_cs_required = false;
651+ u8 *last_ste;
652+ int i, ret;
653+
654+ attr.gvmi = dmn->info.caps.gvmi;
655+ attr.hit_gvmi = dmn->info.caps.gvmi;
656+ attr.final_icm_addr = nic_dmn->default_icm_addr;
657+ action_domain = dr_action_get_action_domain(dmn->type, nic_dmn->ste_type);
658+
659+ for (i = 0; i < num_actions; i++) {
660+ struct mlx5dr_action *action;
661+ int max_actions_type = 1;
662+ u32 action_type;
663+
664+ action = actions[i];
665+ action_type = action->action_type;
666+
667+ switch (action_type) {
668+ case DR_ACTION_TYP_DROP:
669+ attr.final_icm_addr = nic_dmn->drop_icm_addr;
670+ break;
671+ case DR_ACTION_TYP_FT:
672+ dest_action = action;
673+ if (!action->dest_tbl.is_fw_tbl) {
674+ if (action->dest_tbl.tbl->dmn != dmn) {
675+ mlx5dr_dbg(dmn,
676+ "Destination table belongs to a different domain\n");
677+ goto out_invalid_arg;
678+ }
679+ if (action->dest_tbl.tbl->level <= matcher->tbl->level) {
680+ mlx5dr_dbg(dmn,
681+ "Destination table level should be higher than source table\n");
682+ goto out_invalid_arg;
683+ }
684+ attr.final_icm_addr = rx_rule ?
685+ action->dest_tbl.tbl->rx.s_anchor->chunk->icm_addr :
686+ action->dest_tbl.tbl->tx.s_anchor->chunk->icm_addr;
687+ } else {
688+ struct mlx5dr_cmd_query_flow_table_details output;
689+ int ret;
690+
691+ /* get the relevant addresses */
692+ if (!action->dest_tbl.fw_tbl.rx_icm_addr) {
693+ ret = mlx5dr_cmd_query_flow_table(action->dest_tbl.fw_tbl.mdev,
694+ action->dest_tbl.fw_tbl.ft->type,
695+ action->dest_tbl.fw_tbl.ft->id,
696+ &output);
697+ if (!ret) {
698+ action->dest_tbl.fw_tbl.tx_icm_addr =
699+ output.sw_owner_icm_root_1;
700+ action->dest_tbl.fw_tbl.rx_icm_addr =
701+ output.sw_owner_icm_root_0;
702+ } else {
703+ mlx5dr_dbg(dmn,
704+ "Failed mlx5_cmd_query_flow_table ret: %d\n",
705+ ret);
706+ return ret;
707+ }
708+ }
709+ attr.final_icm_addr = rx_rule ?
710+ action->dest_tbl.fw_tbl.rx_icm_addr :
711+ action->dest_tbl.fw_tbl.tx_icm_addr;
712+ }
713+ break;
714+ case DR_ACTION_TYP_QP:
715+ mlx5dr_info(dmn, "Domain doesn't support QP\n");
716+ goto out_invalid_arg;
717+ case DR_ACTION_TYP_CTR:
718+ attr.ctr_id = action->ctr.ctr_id +
719+ action->ctr.offeset;
720+ break;
721+ case DR_ACTION_TYP_TAG:
722+ attr.flow_tag = action->flow_tag;
723+ break;
724+ case DR_ACTION_TYP_TNL_L2_TO_L2:
725+ break;
726+ case DR_ACTION_TYP_TNL_L3_TO_L2:
727+ attr.decap_index = action->rewrite.index;
728+ attr.decap_actions = action->rewrite.num_of_actions;
729+ attr.decap_with_vlan =
730+ attr.decap_actions == WITH_VLAN_NUM_HW_ACTIONS;
731+ break;
732+ case DR_ACTION_TYP_MODIFY_HDR:
733+ attr.modify_index = action->rewrite.index;
734+ attr.modify_actions = action->rewrite.num_of_actions;
735+ recalc_cs_required = action->rewrite.modify_ttl;
736+ break;
737+ case DR_ACTION_TYP_L2_TO_TNL_L2:
738+ case DR_ACTION_TYP_L2_TO_TNL_L3:
739+ attr.reformat_size = action->reformat.reformat_size;
740+ attr.reformat_id = action->reformat.reformat_id;
741+ break;
742+ case DR_ACTION_TYP_VPORT:
743+ attr.hit_gvmi = action->vport.caps->vhca_gvmi;
744+ dest_action = action;
745+ if (rx_rule) {
746+ /* Loopback on WIRE vport is not supported */
747+ if (action->vport.num == WIRE_PORT)
748+ goto out_invalid_arg;
749+
750+ attr.final_icm_addr = action->vport.caps->icm_address_rx;
751+ } else {
752+ attr.final_icm_addr = action->vport.caps->icm_address_tx;
753+ }
754+ break;
755+ case DR_ACTION_TYP_POP_VLAN:
756+ max_actions_type = MAX_VLANS;
757+ attr.vlans.count++;
758+ break;
759+ case DR_ACTION_TYP_PUSH_VLAN:
760+ max_actions_type = MAX_VLANS;
761+ if (attr.vlans.count == MAX_VLANS)
762+ return -EINVAL;
763+
764+ attr.vlans.headers[attr.vlans.count++] = action->push_vlan.vlan_hdr;
765+ break;
766+ default:
767+ goto out_invalid_arg;
768+ }
769+
770+ /* Check action duplication */
771+ if (++action_type_set[action_type] > max_actions_type) {
772+ mlx5dr_dbg(dmn, "Action type %d supports only max %d time(s)\n",
773+ action_type, max_actions_type);
774+ goto out_invalid_arg;
775+ }
776+
777+ /* Check action state machine is valid */
778+ if (dr_action_validate_and_get_next_state(action_domain,
779+ action_type,
780+ &state)) {
781+ mlx5dr_dbg(dmn, "Invalid action sequence provided\n");
782+ return -EOPNOTSUPP;
783+ }
784+ }
785+
786+ *new_hw_ste_arr_sz = nic_matcher->num_of_builders;
787+ last_ste = ste_arr + DR_STE_SIZE * (nic_matcher->num_of_builders - 1);
788+
789+ /* Due to a HW bug, modifying TTL on RX flows will cause an incorrect
790+ * checksum calculation. In this case we will use a FW table to
791+ * recalculate.
792+ */
793+ if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB &&
794+ rx_rule && recalc_cs_required && dest_action) {
795+ ret = dr_action_handle_cs_recalc(dmn, dest_action, &attr.final_icm_addr);
796+ if (ret) {
797+ mlx5dr_dbg(dmn,
798+ "Failed to handle checksum recalculation err %d\n",
799+ ret);
800+ return ret;
801+ }
802+ }
803+
804+ dr_actions_apply(dmn,
805+ nic_dmn->ste_type,
806+ action_type_set,
807+ last_ste,
808+ &attr,
809+ new_hw_ste_arr_sz);
810+
811+ return 0;
812+
813+out_invalid_arg:
814+ return -EINVAL;
815+}
816+
817+#define CVLAN_ETHERTYPE 0x8100
818+#define SVLAN_ETHERTYPE 0x88a8
819+#define HDR_LEN_L2_ONLY 14
820+#define HDR_LEN_L2_VLAN 18
821+#define REWRITE_HW_ACTION_NUM 6
822+
823+static int dr_actions_l2_rewrite(struct mlx5dr_domain *dmn,
824+ struct mlx5dr_action *action,
825+ void *data, size_t data_sz)
826+{
827+ struct mlx5_ifc_l2_hdr_bits *l2_hdr = data;
828+ u64 ops[REWRITE_HW_ACTION_NUM] = {};
829+ u32 hdr_fld_4b;
830+ u16 hdr_fld_2b;
831+ u16 vlan_type;
832+ bool vlan;
833+ int i = 0;
834+ int ret;
835+
836+ vlan = (data_sz != HDR_LEN_L2_ONLY);
837+
838+ /* dmac_47_16 */
839+ MLX5_SET(dr_action_hw_set, ops + i,
840+ opcode, MLX5DR_ACTION_MDFY_HW_OP_SET);
841+ MLX5_SET(dr_action_hw_set, ops + i,
842+ destination_length, 0);
843+ MLX5_SET(dr_action_hw_set, ops + i,
844+ destination_field_code, MLX5DR_ACTION_MDFY_HW_FLD_L2_0);
845+ MLX5_SET(dr_action_hw_set, ops + i,
846+ destination_left_shifter, 16);
847+ hdr_fld_4b = MLX5_GET(l2_hdr, l2_hdr, dmac_47_16);
848+ MLX5_SET(dr_action_hw_set, ops + i,
849+ inline_data, hdr_fld_4b);
850+ i++;
851+
852+ /* smac_47_16 */
853+ MLX5_SET(dr_action_hw_set, ops + i,
854+ opcode, MLX5DR_ACTION_MDFY_HW_OP_SET);
855+ MLX5_SET(dr_action_hw_set, ops + i,
856+ destination_length, 0);
857+ MLX5_SET(dr_action_hw_set, ops + i,
858+ destination_field_code, MLX5DR_ACTION_MDFY_HW_FLD_L2_1);
859+ MLX5_SET(dr_action_hw_set, ops + i,
860+ destination_left_shifter, 16);
861+ hdr_fld_4b = (MLX5_GET(l2_hdr, l2_hdr, smac_31_0) >> 16 |
862+ MLX5_GET(l2_hdr, l2_hdr, smac_47_32) << 16);
863+ MLX5_SET(dr_action_hw_set, ops + i,
864+ inline_data, hdr_fld_4b);
865+ i++;
866+
867+ /* dmac_15_0 */
868+ MLX5_SET(dr_action_hw_set, ops + i,
869+ opcode, MLX5DR_ACTION_MDFY_HW_OP_SET);
870+ MLX5_SET(dr_action_hw_set, ops + i,
871+ destination_length, 16);
872+ MLX5_SET(dr_action_hw_set, ops + i,
873+ destination_field_code, MLX5DR_ACTION_MDFY_HW_FLD_L2_0);
874+ MLX5_SET(dr_action_hw_set, ops + i,
875+ destination_left_shifter, 0);
876+ hdr_fld_2b = MLX5_GET(l2_hdr, l2_hdr, dmac_15_0);
877+ MLX5_SET(dr_action_hw_set, ops + i,
878+ inline_data, hdr_fld_2b);
879+ i++;
880+
881+ /* ethertype + (optional) vlan */
882+ MLX5_SET(dr_action_hw_set, ops + i,
883+ opcode, MLX5DR_ACTION_MDFY_HW_OP_SET);
884+ MLX5_SET(dr_action_hw_set, ops + i,
885+ destination_field_code, MLX5DR_ACTION_MDFY_HW_FLD_L2_2);
886+ MLX5_SET(dr_action_hw_set, ops + i,
887+ destination_left_shifter, 32);
888+ if (!vlan) {
889+ hdr_fld_2b = MLX5_GET(l2_hdr, l2_hdr, ethertype);
890+ MLX5_SET(dr_action_hw_set, ops + i, inline_data, hdr_fld_2b);
891+ MLX5_SET(dr_action_hw_set, ops + i, destination_length, 16);
892+ } else {
893+ hdr_fld_2b = MLX5_GET(l2_hdr, l2_hdr, ethertype);
894+ vlan_type = hdr_fld_2b == SVLAN_ETHERTYPE ? DR_STE_SVLAN : DR_STE_CVLAN;
895+ hdr_fld_2b = MLX5_GET(l2_hdr, l2_hdr, vlan);
896+ hdr_fld_4b = (vlan_type << 16) | hdr_fld_2b;
897+ MLX5_SET(dr_action_hw_set, ops + i, inline_data, hdr_fld_4b);
898+ MLX5_SET(dr_action_hw_set, ops + i, destination_length, 18);
899+ }
900+ i++;
901+
902+ /* smac_15_0 */
903+ MLX5_SET(dr_action_hw_set, ops + i,
904+ opcode, MLX5DR_ACTION_MDFY_HW_OP_SET);
905+ MLX5_SET(dr_action_hw_set, ops + i,
906+ destination_length, 16);
907+ MLX5_SET(dr_action_hw_set, ops + i,
908+ destination_field_code, MLX5DR_ACTION_MDFY_HW_FLD_L2_1);
909+ MLX5_SET(dr_action_hw_set, ops + i,
910+ destination_left_shifter, 0);
911+ hdr_fld_2b = MLX5_GET(l2_hdr, l2_hdr, smac_31_0);
912+ MLX5_SET(dr_action_hw_set, ops + i,
913+ inline_data, hdr_fld_2b);
914+ i++;
915+
916+ if (vlan) {
917+ MLX5_SET(dr_action_hw_set, ops + i,
918+ opcode, MLX5DR_ACTION_MDFY_HW_OP_SET);
919+ hdr_fld_2b = MLX5_GET(l2_hdr, l2_hdr, vlan_type);
920+ MLX5_SET(dr_action_hw_set, ops + i,
921+ inline_data, hdr_fld_2b);
922+ MLX5_SET(dr_action_hw_set, ops + i,
923+ destination_length, 16);
924+ MLX5_SET(dr_action_hw_set, ops + i,
925+ destination_field_code, MLX5DR_ACTION_MDFY_HW_FLD_L2_2);
926+ MLX5_SET(dr_action_hw_set, ops + i,
927+ destination_left_shifter, 0);
928+ i++;
929+ }
930+
931+ action->rewrite.data = (void *)ops;
932+ action->rewrite.num_of_actions = i;
933+ action->rewrite.chunk->byte_size = i * sizeof(*ops);
934+
935+ ret = mlx5dr_send_postsend_action(dmn, action);
936+ if (ret) {
937+ mlx5dr_dbg(dmn, "Writing encapsulation action to ICM failed\n");
938+ return ret;
939+ }
940+
941+ return 0;
942+}
943+
944+static struct mlx5dr_action *
945+dr_action_create_generic(enum mlx5dr_action_type action_type)
946+{
947+ struct mlx5dr_action *action;
948+
949+ action = kzalloc(sizeof(*action), GFP_KERNEL);
950+ if (!action)
951+ return NULL;
952+
953+ action->action_type = action_type;
954+ refcount_set(&action->refcount, 1);
955+
956+ return action;
957+}
958+
959+struct mlx5dr_action *mlx5dr_action_create_drop(void)
960+{
961+ return dr_action_create_generic(DR_ACTION_TYP_DROP);
962+}
963+
964+struct mlx5dr_action *
965+mlx5dr_action_create_dest_table(struct mlx5dr_table *tbl)
966+{
967+ struct mlx5dr_action *action;
968+
969+ refcount_inc(&tbl->refcount);
970+
971+ action = dr_action_create_generic(DR_ACTION_TYP_FT);
972+ if (!action)
973+ goto dec_ref;
974+
975+ action->dest_tbl.tbl = tbl;
976+
977+ return action;
978+
979+dec_ref:
980+ refcount_dec(&tbl->refcount);
981+ return NULL;
982+}
983+
984+struct mlx5dr_action *
985+mlx5dr_create_action_dest_flow_fw_table(struct mlx5_flow_table *ft,
986+ struct mlx5_core_dev *mdev)
987+{
988+ struct mlx5dr_action *action;
989+
990+ action = dr_action_create_generic(DR_ACTION_TYP_FT);
991+ if (!action)
992+ return NULL;
993+
994+ action->dest_tbl.is_fw_tbl = 1;
995+ action->dest_tbl.fw_tbl.ft = ft;
996+ action->dest_tbl.fw_tbl.mdev = mdev;
997+
998+ return action;
999+}
1000+
1001+struct mlx5dr_action *
1002+mlx5dr_action_create_flow_counter(u32 counter_id)
1003+{
1004+ struct mlx5dr_action *action;
1005+
1006+ action = dr_action_create_generic(DR_ACTION_TYP_CTR);
1007+ if (!action)
1008+ return NULL;
1009+
1010+ action->ctr.ctr_id = counter_id;
1011+
1012+ return action;
1013+}
1014+
1015+struct mlx5dr_action *mlx5dr_action_create_tag(u32 tag_value)
1016+{
1017+ struct mlx5dr_action *action;
1018+
1019+ action = dr_action_create_generic(DR_ACTION_TYP_TAG);
1020+ if (!action)
1021+ return NULL;
1022+
1023+ action->flow_tag = tag_value & 0xffffff;
1024+
1025+ return action;
1026+}
1027+
1028+static int
1029+dr_action_verify_reformat_params(enum mlx5dr_action_type reformat_type,
1030+ struct mlx5dr_domain *dmn,
1031+ size_t data_sz,
1032+ void *data)
1033+{
1034+ if ((!data && data_sz) || (data && !data_sz) || reformat_type >
1035+ DR_ACTION_TYP_L2_TO_TNL_L3) {
1036+ mlx5dr_dbg(dmn, "Invalid reformat parameter!\n");
1037+ goto out_err;
1038+ }
1039+
1040+ if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB)
1041+ return 0;
1042+
1043+ if (dmn->type == MLX5DR_DOMAIN_TYPE_NIC_RX) {
1044+ if (reformat_type != DR_ACTION_TYP_TNL_L2_TO_L2 &&
1045+ reformat_type != DR_ACTION_TYP_TNL_L3_TO_L2) {
1046+ mlx5dr_dbg(dmn, "Action reformat type not support on RX domain\n");
1047+ goto out_err;
1048+ }
1049+ } else if (dmn->type == MLX5DR_DOMAIN_TYPE_NIC_TX) {
1050+ if (reformat_type != DR_ACTION_TYP_L2_TO_TNL_L2 &&
1051+ reformat_type != DR_ACTION_TYP_L2_TO_TNL_L3) {
1052+ mlx5dr_dbg(dmn, "Action reformat type not support on TX domain\n");
1053+ goto out_err;
1054+ }
1055+ }
1056+
1057+ return 0;
1058+
1059+out_err:
1060+ return -EINVAL;
1061+}
1062+
1063+#define ACTION_CACHE_LINE_SIZE 64
1064+
1065+static int
1066+dr_action_create_reformat_action(struct mlx5dr_domain *dmn,
1067+ size_t data_sz, void *data,
1068+ struct mlx5dr_action *action)
1069+{
1070+ u32 reformat_id;
1071+ int ret;
1072+
1073+ switch (action->action_type) {
1074+ case DR_ACTION_TYP_L2_TO_TNL_L2:
1075+ case DR_ACTION_TYP_L2_TO_TNL_L3:
1076+ {
1077+ enum mlx5dr_action_type rt;
1078+
1079+ if (action->action_type == DR_ACTION_TYP_L2_TO_TNL_L2)
1080+ rt = MLX5_REFORMAT_TYPE_L2_TO_L2_TUNNEL;
1081+ else
1082+ rt = MLX5_REFORMAT_TYPE_L2_TO_L3_TUNNEL;
1083+
1084+ ret = mlx5dr_cmd_create_reformat_ctx(dmn->mdev, rt, data_sz, data,
1085+ &reformat_id);
1086+ if (ret)
1087+ return ret;
1088+
1089+ action->reformat.reformat_id = reformat_id;
1090+ action->reformat.reformat_size = data_sz;
1091+ return 0;
1092+ }
1093+ case DR_ACTION_TYP_TNL_L2_TO_L2:
1094+ {
1095+ return 0;
1096+ }
1097+ case DR_ACTION_TYP_TNL_L3_TO_L2:
1098+ {
1099+ /* Only Ethernet frame is supported, with VLAN (18) or without (14) */
1100+ if (data_sz != HDR_LEN_L2_ONLY && data_sz != HDR_LEN_L2_VLAN)
1101+ return -EINVAL;
1102+
1103+ action->rewrite.chunk = mlx5dr_icm_alloc_chunk(dmn->action_icm_pool,
1104+ DR_CHUNK_SIZE_8);
1105+ if (!action->rewrite.chunk)
1106+ return -ENOMEM;
1107+
1108+ action->rewrite.index = (action->rewrite.chunk->icm_addr -
1109+ dmn->info.caps.hdr_modify_icm_addr) /
1110+ ACTION_CACHE_LINE_SIZE;
1111+
1112+ ret = dr_actions_l2_rewrite(dmn, action, data, data_sz);
1113+ if (ret) {
1114+ mlx5dr_icm_free_chunk(action->rewrite.chunk);
1115+ return ret;
1116+ }
1117+ return 0;
1118+ }
1119+ default:
1120+ mlx5dr_info(dmn, "Reformat type is not supported %d\n", action->action_type);
1121+ return -EINVAL;
1122+ }
1123+}
1124+
1125+struct mlx5dr_action *mlx5dr_action_create_pop_vlan(void)
1126+{
1127+ return dr_action_create_generic(DR_ACTION_TYP_POP_VLAN);
1128+}
1129+
1130+struct mlx5dr_action *mlx5dr_action_create_push_vlan(struct mlx5dr_domain *dmn,
1131+ __be32 vlan_hdr)
1132+{
1133+ u32 vlan_hdr_h = ntohl(vlan_hdr);
1134+ u16 ethertype = vlan_hdr_h >> 16;
1135+ struct mlx5dr_action *action;
1136+
1137+ if (ethertype != SVLAN_ETHERTYPE && ethertype != CVLAN_ETHERTYPE) {
1138+ mlx5dr_dbg(dmn, "Invalid vlan ethertype\n");
1139+ return NULL;
1140+ }
1141+
1142+ action = dr_action_create_generic(DR_ACTION_TYP_PUSH_VLAN);
1143+ if (!action)
1144+ return NULL;
1145+
1146+ action->push_vlan.vlan_hdr = vlan_hdr_h;
1147+ return action;
1148+}
1149+
1150+struct mlx5dr_action *
1151+mlx5dr_action_create_packet_reformat(struct mlx5dr_domain *dmn,
1152+ enum mlx5dr_action_reformat_type reformat_type,
1153+ size_t data_sz,
1154+ void *data)
1155+{
1156+ enum mlx5dr_action_type action_type;
1157+ struct mlx5dr_action *action;
1158+ int ret;
1159+
1160+ refcount_inc(&dmn->refcount);
1161+
1162+ /* General checks */
1163+ ret = dr_action_reformat_to_action_type(reformat_type, &action_type);
1164+ if (ret) {
1165+ mlx5dr_dbg(dmn, "Invalid reformat_type provided\n");
1166+ goto dec_ref;
1167+ }
1168+
1169+ ret = dr_action_verify_reformat_params(action_type, dmn, data_sz, data);
1170+ if (ret)
1171+ goto dec_ref;
1172+
1173+ action = dr_action_create_generic(action_type);
1174+ if (!action)
1175+ goto dec_ref;
1176+
1177+ action->reformat.dmn = dmn;
1178+
1179+ ret = dr_action_create_reformat_action(dmn,
1180+ data_sz,
1181+ data,
1182+ action);
1183+ if (ret) {
1184+ mlx5dr_dbg(dmn, "Failed creating reformat action %d\n", ret);
1185+ goto free_action;
1186+ }
1187+
1188+ return action;
1189+
1190+free_action:
1191+ kfree(action);
1192+dec_ref:
1193+ refcount_dec(&dmn->refcount);
1194+ return NULL;
1195+}
1196+
1197+static const struct dr_action_modify_field_conv *
1198+dr_action_modify_get_hw_info(u16 sw_field)
1199+{
1200+ const struct dr_action_modify_field_conv *hw_action_info;
1201+
1202+ if (sw_field >= ARRAY_SIZE(dr_action_conv_arr))
1203+ goto not_found;
1204+
1205+ hw_action_info = &dr_action_conv_arr[sw_field];
1206+ if (!hw_action_info->end && !hw_action_info->start)
1207+ goto not_found;
1208+
1209+ return hw_action_info;
1210+
1211+not_found:
1212+ return NULL;
1213+}
1214+
1215+static int
1216+dr_action_modify_sw_to_hw(struct mlx5dr_domain *dmn,
1217+ __be64 *sw_action,
1218+ __be64 *hw_action,
1219+ const struct dr_action_modify_field_conv **ret_hw_info)
1220+{
1221+ const struct dr_action_modify_field_conv *hw_action_info;
1222+ u8 offset, length, max_length, action;
1223+ u16 sw_field;
1224+ u8 hw_opcode;
1225+ u32 data;
1226+
1227+ /* Get SW modify action data */
1228+ action = MLX5_GET(set_action_in, sw_action, action_type);
1229+ length = MLX5_GET(set_action_in, sw_action, length);
1230+ offset = MLX5_GET(set_action_in, sw_action, offset);
1231+ sw_field = MLX5_GET(set_action_in, sw_action, field);
1232+ data = MLX5_GET(set_action_in, sw_action, data);
1233+
1234+ /* Convert SW data to HW modify action format */
1235+ hw_action_info = dr_action_modify_get_hw_info(sw_field);
1236+ if (!hw_action_info) {
1237+ mlx5dr_dbg(dmn, "Modify action invalid field given\n");
1238+ return -EINVAL;
1239+ }
1240+
1241+ max_length = hw_action_info->end - hw_action_info->start + 1;
1242+
1243+ switch (action) {
1244+ case MLX5_ACTION_TYPE_SET:
1245+ hw_opcode = MLX5DR_ACTION_MDFY_HW_OP_SET;
1246+ /* PRM defines that length zero specific length of 32bits */
1247+ if (!length)
1248+ length = 32;
1249+
1250+ if (length + offset > max_length) {
1251+ mlx5dr_dbg(dmn, "Modify action length + offset exceeds limit\n");
1252+ return -EINVAL;
1253+ }
1254+ break;
1255+
1256+ case MLX5_ACTION_TYPE_ADD:
1257+ hw_opcode = MLX5DR_ACTION_MDFY_HW_OP_ADD;
1258+ offset = 0;
1259+ length = max_length;
1260+ break;
1261+
1262+ default:
1263+ mlx5dr_info(dmn, "Unsupported action_type for modify action\n");
1264+ return -EOPNOTSUPP;
1265+ }
1266+
1267+ MLX5_SET(dr_action_hw_set, hw_action, opcode, hw_opcode);
1268+
1269+ MLX5_SET(dr_action_hw_set, hw_action, destination_field_code,
1270+ hw_action_info->hw_field);
1271+
1272+ MLX5_SET(dr_action_hw_set, hw_action, destination_left_shifter,
1273+ hw_action_info->start + offset);
1274+
1275+ MLX5_SET(dr_action_hw_set, hw_action, destination_length,
1276+ length == 32 ? 0 : length);
1277+
1278+ MLX5_SET(dr_action_hw_set, hw_action, inline_data, data);
1279+
1280+ *ret_hw_info = hw_action_info;
1281+
1282+ return 0;
1283+}
1284+
1285+static int
1286+dr_action_modify_check_field_limitation(struct mlx5dr_domain *dmn,
1287+ const __be64 *sw_action)
1288+{
1289+ u16 sw_field;
1290+ u8 action;
1291+
1292+ sw_field = MLX5_GET(set_action_in, sw_action, field);
1293+ action = MLX5_GET(set_action_in, sw_action, action_type);
1294+
1295+ /* Check if SW field is supported in current domain (RX/TX) */
1296+ if (action == MLX5_ACTION_TYPE_SET) {
1297+ if (sw_field == MLX5_ACTION_IN_FIELD_METADATA_REG_A) {
1298+ if (dmn->type != MLX5DR_DOMAIN_TYPE_NIC_TX) {
1299+ mlx5dr_dbg(dmn, "Unsupported field %d for RX/FDB set action\n",
1300+ sw_field);
1301+ return -EINVAL;
1302+ }
1303+ }
1304+
1305+ if (sw_field == MLX5_ACTION_IN_FIELD_METADATA_REG_B) {
1306+ if (dmn->type != MLX5DR_DOMAIN_TYPE_NIC_RX) {
1307+ mlx5dr_dbg(dmn, "Unsupported field %d for TX/FDB set action\n",
1308+ sw_field);
1309+ return -EINVAL;
1310+ }
1311+ }
1312+ } else if (action == MLX5_ACTION_TYPE_ADD) {
1313+ if (sw_field != MLX5_ACTION_IN_FIELD_OUT_IP_TTL &&
1314+ sw_field != MLX5_ACTION_IN_FIELD_OUT_IPV6_HOPLIMIT &&
1315+ sw_field != MLX5_ACTION_IN_FIELD_OUT_TCP_SEQ_NUM &&
1316+ sw_field != MLX5_ACTION_IN_FIELD_OUT_TCP_ACK_NUM) {
1317+ mlx5dr_dbg(dmn, "Unsupported field %d for add action\n", sw_field);
1318+ return -EINVAL;
1319+ }
1320+ } else {
1321+ mlx5dr_info(dmn, "Unsupported action %d modify action\n", action);
1322+ return -EOPNOTSUPP;
1323+ }
1324+
1325+ return 0;
1326+}
1327+
1328+static bool
1329+dr_action_modify_check_is_ttl_modify(const u64 *sw_action)
1330+{
1331+ u16 sw_field = MLX5_GET(set_action_in, sw_action, field);
1332+
1333+ return sw_field == MLX5_ACTION_IN_FIELD_OUT_IP_TTL;
1334+}
1335+
1336+static int dr_actions_convert_modify_header(struct mlx5dr_domain *dmn,
1337+ u32 max_hw_actions,
1338+ u32 num_sw_actions,
1339+ __be64 sw_actions[],
1340+ __be64 hw_actions[],
1341+ u32 *num_hw_actions,
1342+ bool *modify_ttl)
1343+{
1344+ const struct dr_action_modify_field_conv *hw_action_info;
1345+ u16 hw_field = MLX5DR_ACTION_MDFY_HW_FLD_RESERVED;
1346+ u32 l3_type = MLX5DR_ACTION_MDFY_HW_HDR_L3_NONE;
1347+ u32 l4_type = MLX5DR_ACTION_MDFY_HW_HDR_L4_NONE;
1348+ int ret, i, hw_idx = 0;
1349+ __be64 *sw_action;
1350+ __be64 hw_action;
1351+
1352+ *modify_ttl = false;
1353+
1354+ for (i = 0; i < num_sw_actions; i++) {
1355+ sw_action = &sw_actions[i];
1356+
1357+ ret = dr_action_modify_check_field_limitation(dmn, sw_action);
1358+ if (ret)
1359+ return ret;
1360+
1361+ if (!(*modify_ttl))
1362+ *modify_ttl = dr_action_modify_check_is_ttl_modify(sw_action);
1363+
1364+ /* Convert SW action to HW action */
1365+ ret = dr_action_modify_sw_to_hw(dmn,
1366+ sw_action,
1367+ &hw_action,
1368+ &hw_action_info);
1369+ if (ret)
1370+ return ret;
1371+
1372+ /* Due to a HW limitation we cannot modify 2 different L3 types */
1373+ if (l3_type && hw_action_info->l3_type &&
1374+ hw_action_info->l3_type != l3_type) {
1375+ mlx5dr_dbg(dmn, "Action list can't support two different L3 types\n");
1376+ return -EINVAL;
1377+ }
1378+ if (hw_action_info->l3_type)
1379+ l3_type = hw_action_info->l3_type;
1380+
1381+ /* Due to a HW limitation we cannot modify two different L4 types */
1382+ if (l4_type && hw_action_info->l4_type &&
1383+ hw_action_info->l4_type != l4_type) {
1384+ mlx5dr_dbg(dmn, "Action list can't support two different L4 types\n");
1385+ return -EINVAL;
1386+ }
1387+ if (hw_action_info->l4_type)
1388+ l4_type = hw_action_info->l4_type;
1389+
1390+ /* HW reads and executes two actions at once this means we
1391+ * need to create a gap if two actions access the same field
1392+ */
1393+ if ((hw_idx % 2) && hw_field == hw_action_info->hw_field) {
1394+ /* Check if after gap insertion the total number of HW
1395+ * modify actions doesn't exceeds the limit
1396+ */
1397+ hw_idx++;
1398+ if ((num_sw_actions + hw_idx - i) >= max_hw_actions) {
1399+ mlx5dr_dbg(dmn, "Modify header action number exceeds HW limit\n");
1400+ return -EINVAL;
1401+ }
1402+ }
1403+ hw_field = hw_action_info->hw_field;
1404+
1405+ hw_actions[hw_idx] = hw_action;
1406+ hw_idx++;
1407+ }
1408+
1409+ *num_hw_actions = hw_idx;
1410+
1411+ return 0;
1412+}
1413+
1414+static int dr_action_create_modify_action(struct mlx5dr_domain *dmn,
1415+ size_t actions_sz,
1416+ __be64 actions[],
1417+ struct mlx5dr_action *action)
1418+{
1419+ struct mlx5dr_icm_chunk *chunk;
1420+ u32 max_hw_actions;
1421+ u32 num_hw_actions;
1422+ u32 num_sw_actions;
1423+ __be64 *hw_actions;
1424+ bool modify_ttl;
1425+ int ret;
1426+
1427+ num_sw_actions = actions_sz / DR_MODIFY_ACTION_SIZE;
1428+ max_hw_actions = mlx5dr_icm_pool_chunk_size_to_entries(DR_CHUNK_SIZE_16);
1429+
1430+ if (num_sw_actions > max_hw_actions) {
1431+ mlx5dr_dbg(dmn, "Max number of actions %d exceeds limit %d\n",
1432+ num_sw_actions, max_hw_actions);
1433+ return -EINVAL;
1434+ }
1435+
1436+ chunk = mlx5dr_icm_alloc_chunk(dmn->action_icm_pool, DR_CHUNK_SIZE_16);
1437+ if (!chunk)
1438+ return -ENOMEM;
1439+
1440+ hw_actions = kcalloc(1, max_hw_actions * DR_MODIFY_ACTION_SIZE, GFP_KERNEL);
1441+ if (!hw_actions) {
1442+ ret = -ENOMEM;
1443+ goto free_chunk;
1444+ }
1445+
1446+ ret = dr_actions_convert_modify_header(dmn,
1447+ max_hw_actions,
1448+ num_sw_actions,
1449+ actions,
1450+ hw_actions,
1451+ &num_hw_actions,
1452+ &modify_ttl);
1453+ if (ret)
1454+ goto free_hw_actions;
1455+
1456+ action->rewrite.chunk = chunk;
1457+ action->rewrite.modify_ttl = modify_ttl;
1458+ action->rewrite.data = (u8 *)hw_actions;
1459+ action->rewrite.num_of_actions = num_hw_actions;
1460+ action->rewrite.index = (chunk->icm_addr -
1461+ dmn->info.caps.hdr_modify_icm_addr) /
1462+ ACTION_CACHE_LINE_SIZE;
1463+
1464+ ret = mlx5dr_send_postsend_action(dmn, action);
1465+ if (ret)
1466+ goto free_hw_actions;
1467+
1468+ return 0;
1469+
1470+free_hw_actions:
1471+ kfree(hw_actions);
1472+free_chunk:
1473+ mlx5dr_icm_free_chunk(chunk);
1474+ return ret;
1475+}
1476+
1477+struct mlx5dr_action *
1478+mlx5dr_action_create_modify_header(struct mlx5dr_domain *dmn,
1479+ u32 flags,
1480+ size_t actions_sz,
1481+ __be64 actions[])
1482+{
1483+ struct mlx5dr_action *action;
1484+ int ret = 0;
1485+
1486+ refcount_inc(&dmn->refcount);
1487+
1488+ if (actions_sz % DR_MODIFY_ACTION_SIZE) {
1489+ mlx5dr_dbg(dmn, "Invalid modify actions size provided\n");
1490+ goto dec_ref;
1491+ }
1492+
1493+ action = dr_action_create_generic(DR_ACTION_TYP_MODIFY_HDR);
1494+ if (!action)
1495+ goto dec_ref;
1496+
1497+ action->rewrite.dmn = dmn;
1498+
1499+ ret = dr_action_create_modify_action(dmn,
1500+ actions_sz,
1501+ actions,
1502+ action);
1503+ if (ret) {
1504+ mlx5dr_dbg(dmn, "Failed creating modify header action %d\n", ret);
1505+ goto free_action;
1506+ }
1507+
1508+ return action;
1509+
1510+free_action:
1511+ kfree(action);
1512+dec_ref:
1513+ refcount_dec(&dmn->refcount);
1514+ return NULL;
1515+}
1516+
1517+struct mlx5dr_action *
1518+mlx5dr_action_create_dest_vport(struct mlx5dr_domain *dmn,
1519+ u32 vport, u8 vhca_id_valid,
1520+ u16 vhca_id)
1521+{
1522+ struct mlx5dr_cmd_vport_cap *vport_cap;
1523+ struct mlx5dr_domain *vport_dmn;
1524+ struct mlx5dr_action *action;
1525+ u8 peer_vport;
1526+
1527+ peer_vport = vhca_id_valid && (vhca_id != dmn->info.caps.gvmi);
1528+ vport_dmn = peer_vport ? dmn->peer_dmn : dmn;
1529+ if (!vport_dmn) {
1530+ mlx5dr_dbg(dmn, "No peer vport domain for given vhca_id\n");
1531+ return NULL;
1532+ }
1533+
1534+ if (vport_dmn->type != MLX5DR_DOMAIN_TYPE_FDB) {
1535+ mlx5dr_dbg(dmn, "Domain doesn't support vport actions\n");
1536+ return NULL;
1537+ }
1538+
1539+ vport_cap = mlx5dr_get_vport_cap(&vport_dmn->info.caps, vport);
1540+ if (!vport_cap) {
1541+ mlx5dr_dbg(dmn, "Failed to get vport %d caps\n", vport);
1542+ return NULL;
1543+ }
1544+
1545+ action = dr_action_create_generic(DR_ACTION_TYP_VPORT);
1546+ if (!action)
1547+ return NULL;
1548+
1549+ action->vport.dmn = vport_dmn;
1550+ action->vport.caps = vport_cap;
1551+
1552+ return action;
1553+}
1554+
1555+int mlx5dr_action_destroy(struct mlx5dr_action *action)
1556+{
1557+ if (refcount_read(&action->refcount) > 1)
1558+ return -EBUSY;
1559+
1560+ switch (action->action_type) {
1561+ case DR_ACTION_TYP_FT:
1562+ if (!action->dest_tbl.is_fw_tbl)
1563+ refcount_dec(&action->dest_tbl.tbl->refcount);
1564+ break;
1565+ case DR_ACTION_TYP_TNL_L2_TO_L2:
1566+ refcount_dec(&action->reformat.dmn->refcount);
1567+ break;
1568+ case DR_ACTION_TYP_TNL_L3_TO_L2:
1569+ mlx5dr_icm_free_chunk(action->rewrite.chunk);
1570+ refcount_dec(&action->reformat.dmn->refcount);
1571+ break;
1572+ case DR_ACTION_TYP_L2_TO_TNL_L2:
1573+ case DR_ACTION_TYP_L2_TO_TNL_L3:
1574+ mlx5dr_cmd_destroy_reformat_ctx((action->reformat.dmn)->mdev,
1575+ action->reformat.reformat_id);
1576+ refcount_dec(&action->reformat.dmn->refcount);
1577+ break;
1578+ case DR_ACTION_TYP_MODIFY_HDR:
1579+ mlx5dr_icm_free_chunk(action->rewrite.chunk);
1580+ refcount_dec(&action->rewrite.dmn->refcount);
1581+ break;
1582+ default:
1583+ break;
1584+ }
1585+
1586+ kfree(action);
1587+ return 0;
1588+}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_cmd.c+480−0
@@ -0,0 +1,480 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+/* Copyright (c) 2019 Mellanox Technologies. */
3+
4+#include "dr_types.h"
5+
6+int mlx5dr_cmd_query_esw_vport_context(struct mlx5_core_dev *mdev,
7+ bool other_vport,
8+ u16 vport_number,
9+ u64 *icm_address_rx,
10+ u64 *icm_address_tx)
11+{
12+ u32 out[MLX5_ST_SZ_DW(query_esw_vport_context_out)] = {};
13+ u32 in[MLX5_ST_SZ_DW(query_esw_vport_context_in)] = {};
14+ int err;
15+
16+ MLX5_SET(query_esw_vport_context_in, in, opcode,
17+ MLX5_CMD_OP_QUERY_ESW_VPORT_CONTEXT);
18+ MLX5_SET(query_esw_vport_context_in, in, other_vport, other_vport);
19+ MLX5_SET(query_esw_vport_context_in, in, vport_number, vport_number);
20+
21+ err = mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
22+ if (err)
23+ return err;
24+
25+ *icm_address_rx =
26+ MLX5_GET64(query_esw_vport_context_out, out,
27+ esw_vport_context.sw_steering_vport_icm_address_rx);
28+ *icm_address_tx =
29+ MLX5_GET64(query_esw_vport_context_out, out,
30+ esw_vport_context.sw_steering_vport_icm_address_tx);
31+ return 0;
32+}
33+
34+int mlx5dr_cmd_query_gvmi(struct mlx5_core_dev *mdev, bool other_vport,
35+ u16 vport_number, u16 *gvmi)
36+{
37+ u32 in[MLX5_ST_SZ_DW(query_hca_cap_in)] = {};
38+ int out_size;
39+ void *out;
40+ int err;
41+
42+ out_size = MLX5_ST_SZ_BYTES(query_hca_cap_out);
43+ out = kzalloc(out_size, GFP_KERNEL);
44+ if (!out)
45+ return -ENOMEM;
46+
47+ MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);
48+ MLX5_SET(query_hca_cap_in, in, other_function, other_vport);
49+ MLX5_SET(query_hca_cap_in, in, function_id, vport_number);
50+ MLX5_SET(query_hca_cap_in, in, op_mod,
51+ MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1 |
52+ HCA_CAP_OPMOD_GET_CUR);
53+
54+ err = mlx5_cmd_exec(mdev, in, sizeof(in), out, out_size);
55+ if (err) {
56+ kfree(out);
57+ return err;
58+ }
59+
60+ *gvmi = MLX5_GET(query_hca_cap_out, out, capability.cmd_hca_cap.vhca_id);
61+
62+ kfree(out);
63+ return 0;
64+}
65+
66+int mlx5dr_cmd_query_esw_caps(struct mlx5_core_dev *mdev,
67+ struct mlx5dr_esw_caps *caps)
68+{
69+ caps->drop_icm_address_rx =
70+ MLX5_CAP64_ESW_FLOWTABLE(mdev,
71+ sw_steering_fdb_action_drop_icm_address_rx);
72+ caps->drop_icm_address_tx =
73+ MLX5_CAP64_ESW_FLOWTABLE(mdev,
74+ sw_steering_fdb_action_drop_icm_address_tx);
75+ caps->uplink_icm_address_rx =
76+ MLX5_CAP64_ESW_FLOWTABLE(mdev,
77+ sw_steering_uplink_icm_address_rx);
78+ caps->uplink_icm_address_tx =
79+ MLX5_CAP64_ESW_FLOWTABLE(mdev,
80+ sw_steering_uplink_icm_address_tx);
81+ caps->sw_owner =
82+ MLX5_CAP_ESW_FLOWTABLE_FDB(mdev,
83+ sw_owner);
84+
85+ return 0;
86+}
87+
88+int mlx5dr_cmd_query_device(struct mlx5_core_dev *mdev,
89+ struct mlx5dr_cmd_caps *caps)
90+{
91+ caps->prio_tag_required = MLX5_CAP_GEN(mdev, prio_tag_required);
92+ caps->eswitch_manager = MLX5_CAP_GEN(mdev, eswitch_manager);
93+ caps->gvmi = MLX5_CAP_GEN(mdev, vhca_id);
94+ caps->flex_protocols = MLX5_CAP_GEN(mdev, flex_parser_protocols);
95+
96+ if (mlx5dr_matcher_supp_flex_parser_icmp_v4(caps)) {
97+ caps->flex_parser_id_icmp_dw0 = MLX5_CAP_GEN(mdev, flex_parser_id_icmp_dw0);
98+ caps->flex_parser_id_icmp_dw1 = MLX5_CAP_GEN(mdev, flex_parser_id_icmp_dw1);
99+ }
100+
101+ if (mlx5dr_matcher_supp_flex_parser_icmp_v6(caps)) {
102+ caps->flex_parser_id_icmpv6_dw0 =
103+ MLX5_CAP_GEN(mdev, flex_parser_id_icmpv6_dw0);
104+ caps->flex_parser_id_icmpv6_dw1 =
105+ MLX5_CAP_GEN(mdev, flex_parser_id_icmpv6_dw1);
106+ }
107+
108+ caps->nic_rx_drop_address =
109+ MLX5_CAP64_FLOWTABLE(mdev, sw_steering_nic_rx_action_drop_icm_address);
110+ caps->nic_tx_drop_address =
111+ MLX5_CAP64_FLOWTABLE(mdev, sw_steering_nic_tx_action_drop_icm_address);
112+ caps->nic_tx_allow_address =
113+ MLX5_CAP64_FLOWTABLE(mdev, sw_steering_nic_tx_action_allow_icm_address);
114+
115+ caps->rx_sw_owner = MLX5_CAP_FLOWTABLE_NIC_RX(mdev, sw_owner);
116+ caps->max_ft_level = MLX5_CAP_FLOWTABLE_NIC_RX(mdev, max_ft_level);
117+
118+ caps->tx_sw_owner = MLX5_CAP_FLOWTABLE_NIC_TX(mdev, sw_owner);
119+
120+ caps->log_icm_size = MLX5_CAP_DEV_MEM(mdev, log_steering_sw_icm_size);
121+ caps->hdr_modify_icm_addr =
122+ MLX5_CAP64_DEV_MEM(mdev, header_modify_sw_icm_start_address);
123+
124+ caps->roce_min_src_udp = MLX5_CAP_ROCE(mdev, r_roce_min_src_udp_port);
125+
126+ return 0;
127+}
128+
129+int mlx5dr_cmd_query_flow_table(struct mlx5_core_dev *dev,
130+ enum fs_flow_table_type type,
131+ u32 table_id,
132+ struct mlx5dr_cmd_query_flow_table_details *output)
133+{
134+ u32 out[MLX5_ST_SZ_DW(query_flow_table_out)] = {};
135+ u32 in[MLX5_ST_SZ_DW(query_flow_table_in)] = {};
136+ int err;
137+
138+ MLX5_SET(query_flow_table_in, in, opcode,
139+ MLX5_CMD_OP_QUERY_FLOW_TABLE);
140+
141+ MLX5_SET(query_flow_table_in, in, table_type, type);
142+ MLX5_SET(query_flow_table_in, in, table_id, table_id);
143+
144+ err = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
145+ if (err)
146+ return err;
147+
148+ output->status = MLX5_GET(query_flow_table_out, out, status);
149+ output->level = MLX5_GET(query_flow_table_out, out, flow_table_context.level);
150+
151+ output->sw_owner_icm_root_1 = MLX5_GET64(query_flow_table_out, out,
152+ flow_table_context.sw_owner_icm_root_1);
153+ output->sw_owner_icm_root_0 = MLX5_GET64(query_flow_table_out, out,
154+ flow_table_context.sw_owner_icm_root_0);
155+
156+ return 0;
157+}
158+
159+int mlx5dr_cmd_sync_steering(struct mlx5_core_dev *mdev)
160+{
161+ u32 out[MLX5_ST_SZ_DW(sync_steering_out)] = {};
162+ u32 in[MLX5_ST_SZ_DW(sync_steering_in)] = {};
163+
164+ MLX5_SET(sync_steering_in, in, opcode, MLX5_CMD_OP_SYNC_STEERING);
165+
166+ return mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
167+}
168+
169+int mlx5dr_cmd_set_fte_modify_and_vport(struct mlx5_core_dev *mdev,
170+ u32 table_type,
171+ u32 table_id,
172+ u32 group_id,
173+ u32 modify_header_id,
174+ u32 vport_id)
175+{
176+ u32 out[MLX5_ST_SZ_DW(set_fte_out)] = {};
177+ void *in_flow_context;
178+ unsigned int inlen;
179+ void *in_dests;
180+ u32 *in;
181+ int err;
182+
183+ inlen = MLX5_ST_SZ_BYTES(set_fte_in) +
184+ 1 * MLX5_ST_SZ_BYTES(dest_format_struct); /* One destination only */
185+
186+ in = kvzalloc(inlen, GFP_KERNEL);
187+ if (!in)
188+ return -ENOMEM;
189+
190+ MLX5_SET(set_fte_in, in, opcode, MLX5_CMD_OP_SET_FLOW_TABLE_ENTRY);
191+ MLX5_SET(set_fte_in, in, table_type, table_type);
192+ MLX5_SET(set_fte_in, in, table_id, table_id);
193+
194+ in_flow_context = MLX5_ADDR_OF(set_fte_in, in, flow_context);
195+ MLX5_SET(flow_context, in_flow_context, group_id, group_id);
196+ MLX5_SET(flow_context, in_flow_context, modify_header_id, modify_header_id);
197+ MLX5_SET(flow_context, in_flow_context, destination_list_size, 1);
198+ MLX5_SET(flow_context, in_flow_context, action,
199+ MLX5_FLOW_CONTEXT_ACTION_FWD_DEST |
200+ MLX5_FLOW_CONTEXT_ACTION_MOD_HDR);
201+
202+ in_dests = MLX5_ADDR_OF(flow_context, in_flow_context, destination);
203+ MLX5_SET(dest_format_struct, in_dests, destination_type,
204+ MLX5_FLOW_DESTINATION_TYPE_VPORT);
205+ MLX5_SET(dest_format_struct, in_dests, destination_id, vport_id);
206+
207+ err = mlx5_cmd_exec(mdev, in, inlen, out, sizeof(out));
208+ kvfree(in);
209+
210+ return err;
211+}
212+
213+int mlx5dr_cmd_del_flow_table_entry(struct mlx5_core_dev *mdev,
214+ u32 table_type,
215+ u32 table_id)
216+{
217+ u32 out[MLX5_ST_SZ_DW(delete_fte_out)] = {};
218+ u32 in[MLX5_ST_SZ_DW(delete_fte_in)] = {};
219+
220+ MLX5_SET(delete_fte_in, in, opcode, MLX5_CMD_OP_DELETE_FLOW_TABLE_ENTRY);
221+ MLX5_SET(delete_fte_in, in, table_type, table_type);
222+ MLX5_SET(delete_fte_in, in, table_id, table_id);
223+
224+ return mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
225+}
226+
227+int mlx5dr_cmd_alloc_modify_header(struct mlx5_core_dev *mdev,
228+ u32 table_type,
229+ u8 num_of_actions,
230+ u64 *actions,
231+ u32 *modify_header_id)
232+{
233+ u32 out[MLX5_ST_SZ_DW(alloc_modify_header_context_out)] = {};
234+ void *p_actions;
235+ u32 inlen;
236+ u32 *in;
237+ int err;
238+
239+ inlen = MLX5_ST_SZ_BYTES(alloc_modify_header_context_in) +
240+ num_of_actions * sizeof(u64);
241+ in = kvzalloc(inlen, GFP_KERNEL);
242+ if (!in)
243+ return -ENOMEM;
244+
245+ MLX5_SET(alloc_modify_header_context_in, in, opcode,
246+ MLX5_CMD_OP_ALLOC_MODIFY_HEADER_CONTEXT);
247+ MLX5_SET(alloc_modify_header_context_in, in, table_type, table_type);
248+ MLX5_SET(alloc_modify_header_context_in, in, num_of_actions, num_of_actions);
249+ p_actions = MLX5_ADDR_OF(alloc_modify_header_context_in, in, actions);
250+ memcpy(p_actions, actions, num_of_actions * sizeof(u64));
251+
252+ err = mlx5_cmd_exec(mdev, in, inlen, out, sizeof(out));
253+ if (err)
254+ goto out;
255+
256+ *modify_header_id = MLX5_GET(alloc_modify_header_context_out, out,
257+ modify_header_id);
258+out:
259+ kvfree(in);
260+ return err;
261+}
262+
263+int mlx5dr_cmd_dealloc_modify_header(struct mlx5_core_dev *mdev,
264+ u32 modify_header_id)
265+{
266+ u32 out[MLX5_ST_SZ_DW(dealloc_modify_header_context_out)] = {};
267+ u32 in[MLX5_ST_SZ_DW(dealloc_modify_header_context_in)] = {};
268+
269+ MLX5_SET(dealloc_modify_header_context_in, in, opcode,
270+ MLX5_CMD_OP_DEALLOC_MODIFY_HEADER_CONTEXT);
271+ MLX5_SET(dealloc_modify_header_context_in, in, modify_header_id,
272+ modify_header_id);
273+
274+ return mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
275+}
276+
277+int mlx5dr_cmd_create_empty_flow_group(struct mlx5_core_dev *mdev,
278+ u32 table_type,
279+ u32 table_id,
280+ u32 *group_id)
281+{
282+ u32 out[MLX5_ST_SZ_DW(create_flow_group_out)] = {};
283+ int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
284+ u32 *in;
285+ int err;
286+
287+ in = kzalloc(inlen, GFP_KERNEL);
288+ if (!in)
289+ return -ENOMEM;
290+
291+ MLX5_SET(create_flow_group_in, in, opcode, MLX5_CMD_OP_CREATE_FLOW_GROUP);
292+ MLX5_SET(create_flow_group_in, in, table_type, table_type);
293+ MLX5_SET(create_flow_group_in, in, table_id, table_id);
294+
295+ err = mlx5_cmd_exec(mdev, in, inlen, out, sizeof(out));
296+ if (err)
297+ goto out;
298+
299+ *group_id = MLX5_GET(create_flow_group_out, out, group_id);
300+
301+out:
302+ kfree(in);
303+ return err;
304+}
305+
306+int mlx5dr_cmd_destroy_flow_group(struct mlx5_core_dev *mdev,
307+ u32 table_type,
308+ u32 table_id,
309+ u32 group_id)
310+{
311+ u32 in[MLX5_ST_SZ_DW(destroy_flow_group_in)] = {};
312+ u32 out[MLX5_ST_SZ_DW(destroy_flow_group_out)] = {};
313+
314+ MLX5_SET(create_flow_group_in, in, opcode, MLX5_CMD_OP_DESTROY_FLOW_GROUP);
315+ MLX5_SET(destroy_flow_group_in, in, table_type, table_type);
316+ MLX5_SET(destroy_flow_group_in, in, table_id, table_id);
317+ MLX5_SET(destroy_flow_group_in, in, group_id, group_id);
318+
319+ return mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
320+}
321+
322+int mlx5dr_cmd_create_flow_table(struct mlx5_core_dev *mdev,
323+ u32 table_type,
324+ u64 icm_addr_rx,
325+ u64 icm_addr_tx,
326+ u8 level,
327+ bool sw_owner,
328+ bool term_tbl,
329+ u64 *fdb_rx_icm_addr,
330+ u32 *table_id)
331+{
332+ u32 out[MLX5_ST_SZ_DW(create_flow_table_out)] = {};
333+ u32 in[MLX5_ST_SZ_DW(create_flow_table_in)] = {};
334+ void *ft_mdev;
335+ int err;
336+
337+ MLX5_SET(create_flow_table_in, in, opcode, MLX5_CMD_OP_CREATE_FLOW_TABLE);
338+ MLX5_SET(create_flow_table_in, in, table_type, table_type);
339+
340+ ft_mdev = MLX5_ADDR_OF(create_flow_table_in, in, flow_table_context);
341+ MLX5_SET(flow_table_context, ft_mdev, termination_table, term_tbl);
342+ MLX5_SET(flow_table_context, ft_mdev, sw_owner, sw_owner);
343+ MLX5_SET(flow_table_context, ft_mdev, level, level);
344+
345+ if (sw_owner) {
346+ /* icm_addr_0 used for FDB RX / NIC TX / NIC_RX
347+ * icm_addr_1 used for FDB TX
348+ */
349+ if (table_type == MLX5_FLOW_TABLE_TYPE_NIC_RX) {
350+ MLX5_SET64(flow_table_context, ft_mdev,
351+ sw_owner_icm_root_0, icm_addr_rx);
352+ } else if (table_type == MLX5_FLOW_TABLE_TYPE_NIC_TX) {
353+ MLX5_SET64(flow_table_context, ft_mdev,
354+ sw_owner_icm_root_0, icm_addr_tx);
355+ } else if (table_type == MLX5_FLOW_TABLE_TYPE_FDB) {
356+ MLX5_SET64(flow_table_context, ft_mdev,
357+ sw_owner_icm_root_0, icm_addr_rx);
358+ MLX5_SET64(flow_table_context, ft_mdev,
359+ sw_owner_icm_root_1, icm_addr_tx);
360+ }
361+ }
362+
363+ err = mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
364+ if (err)
365+ return err;
366+
367+ *table_id = MLX5_GET(create_flow_table_out, out, table_id);
368+ if (!sw_owner && table_type == MLX5_FLOW_TABLE_TYPE_FDB)
369+ *fdb_rx_icm_addr =
370+ (u64)MLX5_GET(create_flow_table_out, out, icm_address_31_0) |
371+ (u64)MLX5_GET(create_flow_table_out, out, icm_address_39_32) << 32 |
372+ (u64)MLX5_GET(create_flow_table_out, out, icm_address_63_40) << 40;
373+
374+ return 0;
375+}
376+
377+int mlx5dr_cmd_destroy_flow_table(struct mlx5_core_dev *mdev,
378+ u32 table_id,
379+ u32 table_type)
380+{
381+ u32 out[MLX5_ST_SZ_DW(destroy_flow_table_out)] = {};
382+ u32 in[MLX5_ST_SZ_DW(destroy_flow_table_in)] = {};
383+
384+ MLX5_SET(destroy_flow_table_in, in, opcode,
385+ MLX5_CMD_OP_DESTROY_FLOW_TABLE);
386+ MLX5_SET(destroy_flow_table_in, in, table_type, table_type);
387+ MLX5_SET(destroy_flow_table_in, in, table_id, table_id);
388+
389+ return mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
390+}
391+
392+int mlx5dr_cmd_create_reformat_ctx(struct mlx5_core_dev *mdev,
393+ enum mlx5_reformat_ctx_type rt,
394+ size_t reformat_size,
395+ void *reformat_data,
396+ u32 *reformat_id)
397+{
398+ u32 out[MLX5_ST_SZ_DW(alloc_packet_reformat_context_out)] = {};
399+ size_t inlen, cmd_data_sz, cmd_total_sz;
400+ void *prctx;
401+ void *pdata;
402+ void *in;
403+ int err;
404+
405+ cmd_total_sz = MLX5_ST_SZ_BYTES(alloc_packet_reformat_context_in);
406+ cmd_data_sz = MLX5_FLD_SZ_BYTES(alloc_packet_reformat_context_in,
407+ packet_reformat_context.reformat_data);
408+ inlen = ALIGN(cmd_total_sz + reformat_size - cmd_data_sz, 4);
409+ in = kvzalloc(inlen, GFP_KERNEL);
410+ if (!in)
411+ return -ENOMEM;
412+
413+ MLX5_SET(alloc_packet_reformat_context_in, in, opcode,
414+ MLX5_CMD_OP_ALLOC_PACKET_REFORMAT_CONTEXT);
415+
416+ prctx = MLX5_ADDR_OF(alloc_packet_reformat_context_in, in, packet_reformat_context);
417+ pdata = MLX5_ADDR_OF(packet_reformat_context_in, prctx, reformat_data);
418+
419+ MLX5_SET(packet_reformat_context_in, prctx, reformat_type, rt);
420+ MLX5_SET(packet_reformat_context_in, prctx, reformat_data_size, reformat_size);
421+ memcpy(pdata, reformat_data, reformat_size);
422+
423+ err = mlx5_cmd_exec(mdev, in, inlen, out, sizeof(out));
424+ if (err)
425+ return err;
426+
427+ *reformat_id = MLX5_GET(alloc_packet_reformat_context_out, out, packet_reformat_id);
428+ kvfree(in);
429+
430+ return err;
431+}
432+
433+void mlx5dr_cmd_destroy_reformat_ctx(struct mlx5_core_dev *mdev,
434+ u32 reformat_id)
435+{
436+ u32 out[MLX5_ST_SZ_DW(dealloc_packet_reformat_context_out)] = {};
437+ u32 in[MLX5_ST_SZ_DW(dealloc_packet_reformat_context_in)] = {};
438+
439+ MLX5_SET(dealloc_packet_reformat_context_in, in, opcode,
440+ MLX5_CMD_OP_DEALLOC_PACKET_REFORMAT_CONTEXT);
441+ MLX5_SET(dealloc_packet_reformat_context_in, in, packet_reformat_id,
442+ reformat_id);
443+
444+ mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
445+}
446+
447+int mlx5dr_cmd_query_gid(struct mlx5_core_dev *mdev, u8 vhca_port_num,
448+ u16 index, struct mlx5dr_cmd_gid_attr *attr)
449+{
450+ u32 out[MLX5_ST_SZ_DW(query_roce_address_out)] = {};
451+ u32 in[MLX5_ST_SZ_DW(query_roce_address_in)] = {};
452+ int err;
453+
454+ MLX5_SET(query_roce_address_in, in, opcode,
455+ MLX5_CMD_OP_QUERY_ROCE_ADDRESS);
456+
457+ MLX5_SET(query_roce_address_in, in, roce_address_index, index);
458+ MLX5_SET(query_roce_address_in, in, vhca_port_num, vhca_port_num);
459+
460+ err = mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
461+ if (err)
462+ return err;
463+
464+ memcpy(&attr->gid,
465+ MLX5_ADDR_OF(query_roce_address_out,
466+ out, roce_address.source_l3_address),
467+ sizeof(attr->gid));
468+ memcpy(attr->mac,
469+ MLX5_ADDR_OF(query_roce_address_out, out,
470+ roce_address.source_mac_47_32),
471+ sizeof(attr->mac));
472+
473+ if (MLX5_GET(query_roce_address_out, out,
474+ roce_address.roce_version) == MLX5_ROCE_VERSION_2)
475+ attr->roce_ver = MLX5_ROCE_VERSION_2;
476+ else
477+ attr->roce_ver = MLX5_ROCE_VERSION_1;
478+
479+ return 0;
480+}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_crc32.c+98−0
@@ -0,0 +1,98 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+/* Copyright (c) 2019 Mellanox Technologies. */
3+
4+/* Copyright (c) 2011-2015 Stephan Brumme. All rights reserved.
5+ * Slicing-by-16 contributed by Bulat Ziganshin
6+ *
7+ * This software is provided 'as-is', without any express or implied warranty.
8+ * In no event will the author be held liable for any damages arising from the
9+ * of this software.
10+ *
11+ * Permission is granted to anyone to use this software for any purpose,
12+ * including commercial applications, and to alter it and redistribute it
13+ * freely, subject to the following restrictions:
14+ *
15+ * 1. The origin of this software must not be misrepresented; you must not
16+ * claim that you wrote the original software.
17+ * 2. If you use this software in a product, an acknowledgment in the product
18+ * documentation would be appreciated but is not required.
19+ * 3. Altered source versions must be plainly marked as such, and must not be
20+ * misrepresented as being the original software.
21+ *
22+ * Taken from http://create.stephan-brumme.com/crc32/ and adapted.
23+ */
24+
25+#include "dr_types.h"
26+
27+#define DR_STE_CRC_POLY 0xEDB88320L
28+
29+static u32 dr_ste_crc_tab32[8][256];
30+
31+static void dr_crc32_calc_lookup_entry(u32 (*tbl)[256], u8 i, u8 j)
32+{
33+ tbl[i][j] = (tbl[i - 1][j] >> 8) ^ tbl[0][tbl[i - 1][j] & 0xff];
34+}
35+
36+void mlx5dr_crc32_init_table(void)
37+{
38+ u32 crc, i, j;
39+
40+ for (i = 0; i < 256; i++) {
41+ crc = i;
42+ for (j = 0; j < 8; j++) {
43+ if (crc & 0x00000001L)
44+ crc = (crc >> 1) ^ DR_STE_CRC_POLY;
45+ else
46+ crc = crc >> 1;
47+ }
48+ dr_ste_crc_tab32[0][i] = crc;
49+ }
50+
51+ /* Init CRC lookup tables according to crc_slice_8 algorithm */
52+ for (i = 0; i < 256; i++) {
53+ dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 1, i);
54+ dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 2, i);
55+ dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 3, i);
56+ dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 4, i);
57+ dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 5, i);
58+ dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 6, i);
59+ dr_crc32_calc_lookup_entry(dr_ste_crc_tab32, 7, i);
60+ }
61+}
62+
63+/* Compute CRC32 (Slicing-by-8 algorithm) */
64+u32 mlx5dr_crc32_slice8_calc(const void *input_data, size_t length)
65+{
66+ const u32 *curr = (const u32 *)input_data;
67+ const u8 *curr_char;
68+ u32 crc = 0, one, two;
69+
70+ if (!input_data)
71+ return 0;
72+
73+ /* Process eight bytes at once (Slicing-by-8) */
74+ while (length >= 8) {
75+ one = *curr++ ^ crc;
76+ two = *curr++;
77+
78+ crc = dr_ste_crc_tab32[0][(two >> 24) & 0xff]
79+ ^ dr_ste_crc_tab32[1][(two >> 16) & 0xff]
80+ ^ dr_ste_crc_tab32[2][(two >> 8) & 0xff]
81+ ^ dr_ste_crc_tab32[3][two & 0xff]
82+ ^ dr_ste_crc_tab32[4][(one >> 24) & 0xff]
83+ ^ dr_ste_crc_tab32[5][(one >> 16) & 0xff]
84+ ^ dr_ste_crc_tab32[6][(one >> 8) & 0xff]
85+ ^ dr_ste_crc_tab32[7][one & 0xff];
86+
87+ length -= 8;
88+ }
89+
90+ curr_char = (const u8 *)curr;
91+ /* Remaining 1 to 7 bytes (standard algorithm) */
92+ while (length-- != 0)
93+ crc = (crc >> 8) ^ dr_ste_crc_tab32[0][(crc & 0xff)
94+ ^ *curr_char++];
95+
96+ return ((crc >> 24) & 0xff) | ((crc << 8) & 0xff0000) |
97+ ((crc >> 8) & 0xff00) | ((crc << 24) & 0xff000000);
98+}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_domain.c+395−0
@@ -0,0 +1,395 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+/* Copyright (c) 2019 Mellanox Technologies. */
3+
4+#include <linux/mlx5/eswitch.h>
5+#include "dr_types.h"
6+
7+static int dr_domain_init_cache(struct mlx5dr_domain *dmn)
8+{
9+ /* Per vport cached FW FT for checksum recalculation, this
10+ * recalculation is needed due to a HW bug.
11+ */
12+ dmn->cache.recalc_cs_ft = kcalloc(dmn->info.caps.num_vports,
13+ sizeof(dmn->cache.recalc_cs_ft[0]),
14+ GFP_KERNEL);
15+ if (!dmn->cache.recalc_cs_ft)
16+ return -ENOMEM;
17+
18+ return 0;
19+}
20+
21+static void dr_domain_uninit_cache(struct mlx5dr_domain *dmn)
22+{
23+ int i;
24+
25+ for (i = 0; i < dmn->info.caps.num_vports; i++) {
26+ if (!dmn->cache.recalc_cs_ft[i])
27+ continue;
28+
29+ mlx5dr_fw_destroy_recalc_cs_ft(dmn, dmn->cache.recalc_cs_ft[i]);
30+ }
31+
32+ kfree(dmn->cache.recalc_cs_ft);
33+}
34+
35+int mlx5dr_domain_cache_get_recalc_cs_ft_addr(struct mlx5dr_domain *dmn,
36+ u32 vport_num,
37+ u64 *rx_icm_addr)
38+{
39+ struct mlx5dr_fw_recalc_cs_ft *recalc_cs_ft;
40+
41+ recalc_cs_ft = dmn->cache.recalc_cs_ft[vport_num];
42+ if (!recalc_cs_ft) {
43+ /* Table not in cache, need to allocate a new one */
44+ recalc_cs_ft = mlx5dr_fw_create_recalc_cs_ft(dmn, vport_num);
45+ if (!recalc_cs_ft)
46+ return -EINVAL;
47+
48+ dmn->cache.recalc_cs_ft[vport_num] = recalc_cs_ft;
49+ }
50+
51+ *rx_icm_addr = recalc_cs_ft->rx_icm_addr;
52+
53+ return 0;
54+}
55+
56+static int dr_domain_init_resources(struct mlx5dr_domain *dmn)
57+{
58+ int ret;
59+
60+ ret = mlx5_core_alloc_pd(dmn->mdev, &dmn->pdn);
61+ if (ret) {
62+ mlx5dr_dbg(dmn, "Couldn't allocate PD\n");
63+ return ret;
64+ }
65+
66+ dmn->uar = mlx5_get_uars_page(dmn->mdev);
67+ if (!dmn->uar) {
68+ mlx5dr_err(dmn, "Couldn't allocate UAR\n");
69+ goto clean_pd;
70+ }
71+
72+ dmn->ste_icm_pool = mlx5dr_icm_pool_create(dmn, DR_ICM_TYPE_STE);
73+ if (!dmn->ste_icm_pool) {
74+ mlx5dr_err(dmn, "Couldn't get icm memory for %s\n",
75+ dev_name(dmn->mdev->device));
76+ goto clean_uar;
77+ }
78+
79+ dmn->action_icm_pool = mlx5dr_icm_pool_create(dmn, DR_ICM_TYPE_MODIFY_ACTION);
80+ if (!dmn->action_icm_pool) {
81+ mlx5dr_err(dmn, "Couldn't get action icm memory for %s\n",
82+ dev_name(dmn->mdev->device));
83+ goto free_ste_icm_pool;
84+ }
85+
86+ ret = mlx5dr_send_ring_alloc(dmn);
87+ if (ret) {
88+ mlx5dr_err(dmn, "Couldn't create send-ring for %s\n",
89+ dev_name(dmn->mdev->device));
90+ goto free_action_icm_pool;
91+ }
92+
93+ return 0;
94+
95+free_action_icm_pool:
96+ mlx5dr_icm_pool_destroy(dmn->action_icm_pool);
97+free_ste_icm_pool:
98+ mlx5dr_icm_pool_destroy(dmn->ste_icm_pool);
99+clean_uar:
100+ mlx5_put_uars_page(dmn->mdev, dmn->uar);
101+clean_pd:
102+ mlx5_core_dealloc_pd(dmn->mdev, dmn->pdn);
103+
104+ return ret;
105+}
106+
107+static void dr_domain_uninit_resources(struct mlx5dr_domain *dmn)
108+{
109+ mlx5dr_send_ring_free(dmn, dmn->send_ring);
110+ mlx5dr_icm_pool_destroy(dmn->action_icm_pool);
111+ mlx5dr_icm_pool_destroy(dmn->ste_icm_pool);
112+ mlx5_put_uars_page(dmn->mdev, dmn->uar);
113+ mlx5_core_dealloc_pd(dmn->mdev, dmn->pdn);
114+}
115+
116+static int dr_domain_query_vport(struct mlx5dr_domain *dmn,
117+ bool other_vport,
118+ u16 vport_number)
119+{
120+ struct mlx5dr_cmd_vport_cap *vport_caps;
121+ int ret;
122+
123+ vport_caps = &dmn->info.caps.vports_caps[vport_number];
124+
125+ ret = mlx5dr_cmd_query_esw_vport_context(dmn->mdev,
126+ other_vport,
127+ vport_number,
128+ &vport_caps->icm_address_rx,
129+ &vport_caps->icm_address_tx);
130+ if (ret)
131+ return ret;
132+
133+ ret = mlx5dr_cmd_query_gvmi(dmn->mdev,
134+ other_vport,
135+ vport_number,
136+ &vport_caps->vport_gvmi);
137+ if (ret)
138+ return ret;
139+
140+ vport_caps->num = vport_number;
141+ vport_caps->vhca_gvmi = dmn->info.caps.gvmi;
142+
143+ return 0;
144+}
145+
146+static int dr_domain_query_vports(struct mlx5dr_domain *dmn)
147+{
148+ struct mlx5dr_esw_caps *esw_caps = &dmn->info.caps.esw_caps;
149+ struct mlx5dr_cmd_vport_cap *wire_vport;
150+ int vport;
151+ int ret;
152+
153+ /* Query vports (except wire vport) */
154+ for (vport = 0; vport < dmn->info.caps.num_esw_ports - 1; vport++) {
155+ ret = dr_domain_query_vport(dmn, !!vport, vport);
156+ if (ret)
157+ return ret;
158+ }
159+
160+ /* Last vport is the wire port */
161+ wire_vport = &dmn->info.caps.vports_caps[vport];
162+ wire_vport->num = WIRE_PORT;
163+ wire_vport->icm_address_rx = esw_caps->uplink_icm_address_rx;
164+ wire_vport->icm_address_tx = esw_caps->uplink_icm_address_tx;
165+ wire_vport->vport_gvmi = 0;
166+ wire_vport->vhca_gvmi = dmn->info.caps.gvmi;
167+
168+ return 0;
169+}
170+
171+static int dr_domain_query_fdb_caps(struct mlx5_core_dev *mdev,
172+ struct mlx5dr_domain *dmn)
173+{
174+ int ret;
175+
176+ if (!dmn->info.caps.eswitch_manager)
177+ return -EOPNOTSUPP;
178+
179+ ret = mlx5dr_cmd_query_esw_caps(mdev, &dmn->info.caps.esw_caps);
180+ if (ret)
181+ return ret;
182+
183+ dmn->info.caps.fdb_sw_owner = dmn->info.caps.esw_caps.sw_owner;
184+ dmn->info.caps.esw_rx_drop_address = dmn->info.caps.esw_caps.drop_icm_address_rx;
185+ dmn->info.caps.esw_tx_drop_address = dmn->info.caps.esw_caps.drop_icm_address_tx;
186+
187+ dmn->info.caps.vports_caps = kcalloc(dmn->info.caps.num_esw_ports,
188+ sizeof(dmn->info.caps.vports_caps[0]),
189+ GFP_KERNEL);
190+ if (!dmn->info.caps.vports_caps)
191+ return -ENOMEM;
192+
193+ ret = dr_domain_query_vports(dmn);
194+ if (ret) {
195+ mlx5dr_dbg(dmn, "Failed to query vports caps\n");
196+ goto free_vports_caps;
197+ }
198+
199+ dmn->info.caps.num_vports = dmn->info.caps.num_esw_ports - 1;
200+
201+ return 0;
202+
203+free_vports_caps:
204+ kfree(dmn->info.caps.vports_caps);
205+ dmn->info.caps.vports_caps = NULL;
206+ return ret;
207+}
208+
209+static int dr_domain_caps_init(struct mlx5_core_dev *mdev,
210+ struct mlx5dr_domain *dmn)
211+{
212+ struct mlx5dr_cmd_vport_cap *vport_cap;
213+ int ret;
214+
215+ if (MLX5_CAP_GEN(mdev, port_type) != MLX5_CAP_PORT_TYPE_ETH) {
216+ mlx5dr_dbg(dmn, "Failed to allocate domain, bad link type\n");
217+ return -EOPNOTSUPP;
218+ }
219+
220+ dmn->info.caps.num_esw_ports = mlx5_eswitch_get_total_vports(mdev);
221+
222+ ret = mlx5dr_cmd_query_device(mdev, &dmn->info.caps);
223+ if (ret)
224+ return ret;
225+
226+ ret = dr_domain_query_fdb_caps(mdev, dmn);
227+ if (ret)
228+ return ret;
229+
230+ switch (dmn->type) {
231+ case MLX5DR_DOMAIN_TYPE_NIC_RX:
232+ if (!dmn->info.caps.rx_sw_owner)
233+ return -ENOTSUPP;
234+
235+ dmn->info.supp_sw_steering = true;
236+ dmn->info.rx.ste_type = MLX5DR_STE_TYPE_RX;
237+ dmn->info.rx.default_icm_addr = dmn->info.caps.nic_rx_drop_address;
238+ dmn->info.rx.drop_icm_addr = dmn->info.caps.nic_rx_drop_address;
239+ break;
240+ case MLX5DR_DOMAIN_TYPE_NIC_TX:
241+ if (!dmn->info.caps.tx_sw_owner)
242+ return -ENOTSUPP;
243+
244+ dmn->info.supp_sw_steering = true;
245+ dmn->info.tx.ste_type = MLX5DR_STE_TYPE_TX;
246+ dmn->info.tx.default_icm_addr = dmn->info.caps.nic_tx_allow_address;
247+ dmn->info.tx.drop_icm_addr = dmn->info.caps.nic_tx_drop_address;
248+ break;
249+ case MLX5DR_DOMAIN_TYPE_FDB:
250+ if (!dmn->info.caps.eswitch_manager)
251+ return -ENOTSUPP;
252+
253+ if (!dmn->info.caps.fdb_sw_owner)
254+ return -ENOTSUPP;
255+
256+ dmn->info.rx.ste_type = MLX5DR_STE_TYPE_RX;
257+ dmn->info.tx.ste_type = MLX5DR_STE_TYPE_TX;
258+ vport_cap = mlx5dr_get_vport_cap(&dmn->info.caps, 0);
259+ if (!vport_cap) {
260+ mlx5dr_dbg(dmn, "Failed to get esw manager vport\n");
261+ return -ENOENT;
262+ }
263+
264+ dmn->info.supp_sw_steering = true;
265+ dmn->info.tx.default_icm_addr = vport_cap->icm_address_tx;
266+ dmn->info.rx.default_icm_addr = vport_cap->icm_address_rx;
267+ dmn->info.rx.drop_icm_addr = dmn->info.caps.esw_rx_drop_address;
268+ dmn->info.tx.drop_icm_addr = dmn->info.caps.esw_tx_drop_address;
269+ break;
270+ default:
271+ mlx5dr_dbg(dmn, "Invalid domain\n");
272+ ret = -EINVAL;
273+ break;
274+ }
275+
276+ return ret;
277+}
278+
279+static void dr_domain_caps_uninit(struct mlx5dr_domain *dmn)
280+{
281+ kfree(dmn->info.caps.vports_caps);
282+}
283+
284+struct mlx5dr_domain *
285+mlx5dr_domain_create(struct mlx5_core_dev *mdev, enum mlx5dr_domain_type type)
286+{
287+ struct mlx5dr_domain *dmn;
288+ int ret;
289+
290+ if (type > MLX5DR_DOMAIN_TYPE_FDB)
291+ return NULL;
292+
293+ dmn = kzalloc(sizeof(*dmn), GFP_KERNEL);
294+ if (!dmn)
295+ return NULL;
296+
297+ dmn->mdev = mdev;
298+ dmn->type = type;
299+ refcount_set(&dmn->refcount, 1);
300+ mutex_init(&dmn->mutex);
301+
302+ if (dr_domain_caps_init(mdev, dmn)) {
303+ mlx5dr_dbg(dmn, "Failed init domain, no caps\n");
304+ goto free_domain;
305+ }
306+
307+ dmn->info.max_log_action_icm_sz = DR_CHUNK_SIZE_4K;
308+ dmn->info.max_log_sw_icm_sz = min_t(u32, DR_CHUNK_SIZE_1024K,
309+ dmn->info.caps.log_icm_size);
310+
311+ if (!dmn->info.supp_sw_steering) {
312+ mlx5dr_err(dmn, "SW steering not supported for %s\n",
313+ dev_name(mdev->device));
314+ goto uninit_caps;
315+ }
316+
317+ /* Allocate resources */
318+ ret = dr_domain_init_resources(dmn);
319+ if (ret) {
320+ mlx5dr_err(dmn, "Failed init domain resources for %s\n",
321+ dev_name(mdev->device));
322+ goto uninit_caps;
323+ }
324+
325+ ret = dr_domain_init_cache(dmn);
326+ if (ret) {
327+ mlx5dr_err(dmn, "Failed initialize domain cache\n");
328+ goto uninit_resourses;
329+ }
330+
331+ /* Init CRC table for htbl CRC calculation */
332+ mlx5dr_crc32_init_table();
333+
334+ return dmn;
335+
336+uninit_resourses:
337+ dr_domain_uninit_resources(dmn);
338+uninit_caps:
339+ dr_domain_caps_uninit(dmn);
340+free_domain:
341+ kfree(dmn);
342+ return NULL;
343+}
344+
345+/* Assure synchronization of the device steering tables with updates made by SW
346+ * insertion.
347+ */
348+int mlx5dr_domain_sync(struct mlx5dr_domain *dmn, u32 flags)
349+{
350+ int ret = 0;
351+
352+ if (flags & MLX5DR_DOMAIN_SYNC_FLAGS_SW) {
353+ mutex_lock(&dmn->mutex);
354+ ret = mlx5dr_send_ring_force_drain(dmn);
355+ mutex_unlock(&dmn->mutex);
356+ if (ret)
357+ return ret;
358+ }
359+
360+ if (flags & MLX5DR_DOMAIN_SYNC_FLAGS_HW)
361+ ret = mlx5dr_cmd_sync_steering(dmn->mdev);
362+
363+ return ret;
364+}
365+
366+int mlx5dr_domain_destroy(struct mlx5dr_domain *dmn)
367+{
368+ if (refcount_read(&dmn->refcount) > 1)
369+ return -EBUSY;
370+
371+ /* make sure resources are not used by the hardware */
372+ mlx5dr_cmd_sync_steering(dmn->mdev);
373+ dr_domain_uninit_cache(dmn);
374+ dr_domain_uninit_resources(dmn);
375+ dr_domain_caps_uninit(dmn);
376+ mutex_destroy(&dmn->mutex);
377+ kfree(dmn);
378+ return 0;
379+}
380+
381+void mlx5dr_domain_set_peer(struct mlx5dr_domain *dmn,
382+ struct mlx5dr_domain *peer_dmn)
383+{
384+ mutex_lock(&dmn->mutex);
385+
386+ if (dmn->peer_dmn)
387+ refcount_dec(&dmn->peer_dmn->refcount);
388+
389+ dmn->peer_dmn = peer_dmn;
390+
391+ if (dmn->peer_dmn)
392+ refcount_inc(&dmn->peer_dmn->refcount);
393+
394+ mutex_unlock(&dmn->mutex);
395+}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_fw.c+93−0
@@ -0,0 +1,93 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+/* Copyright (c) 2019 Mellanox Technologies. */
3+
4+#include <linux/types.h>
5+#include "dr_types.h"
6+
7+struct mlx5dr_fw_recalc_cs_ft *
8+mlx5dr_fw_create_recalc_cs_ft(struct mlx5dr_domain *dmn, u32 vport_num)
9+{
10+ struct mlx5dr_fw_recalc_cs_ft *recalc_cs_ft;
11+ u32 table_id, group_id, modify_hdr_id;
12+ u64 rx_icm_addr, modify_ttl_action;
13+ int ret;
14+
15+ recalc_cs_ft = kzalloc(sizeof(*recalc_cs_ft), GFP_KERNEL);
16+ if (!recalc_cs_ft)
17+ return NULL;
18+
19+ ret = mlx5dr_cmd_create_flow_table(dmn->mdev, MLX5_FLOW_TABLE_TYPE_FDB,
20+ 0, 0, dmn->info.caps.max_ft_level - 1,
21+ false, true, &rx_icm_addr, &table_id);
22+ if (ret) {
23+ mlx5dr_err(dmn, "Failed creating TTL W/A FW flow table %d\n", ret);
24+ goto free_ttl_tbl;
25+ }
26+
27+ ret = mlx5dr_cmd_create_empty_flow_group(dmn->mdev,
28+ MLX5_FLOW_TABLE_TYPE_FDB,
29+ table_id, &group_id);
30+ if (ret) {
31+ mlx5dr_err(dmn, "Failed creating TTL W/A FW flow group %d\n", ret);
32+ goto destroy_flow_table;
33+ }
34+
35+ /* Modify TTL action by adding zero to trigger CS recalculation */
36+ modify_ttl_action = 0;
37+ MLX5_SET(set_action_in, &modify_ttl_action, action_type, MLX5_ACTION_TYPE_ADD);
38+ MLX5_SET(set_action_in, &modify_ttl_action, field, MLX5_ACTION_IN_FIELD_OUT_IP_TTL);
39+
40+ ret = mlx5dr_cmd_alloc_modify_header(dmn->mdev, MLX5_FLOW_TABLE_TYPE_FDB, 1,
41+ &modify_ttl_action,
42+ &modify_hdr_id);
43+ if (ret) {
44+ mlx5dr_err(dmn, "Failed modify header TTL %d\n", ret);
45+ goto destroy_flow_group;
46+ }
47+
48+ ret = mlx5dr_cmd_set_fte_modify_and_vport(dmn->mdev,
49+ MLX5_FLOW_TABLE_TYPE_FDB,
50+ table_id, group_id, modify_hdr_id,
51+ vport_num);
52+ if (ret) {
53+ mlx5dr_err(dmn, "Failed setting TTL W/A flow table entry %d\n", ret);
54+ goto dealloc_modify_header;
55+ }
56+
57+ recalc_cs_ft->modify_hdr_id = modify_hdr_id;
58+ recalc_cs_ft->rx_icm_addr = rx_icm_addr;
59+ recalc_cs_ft->table_id = table_id;
60+ recalc_cs_ft->group_id = group_id;
61+
62+ return recalc_cs_ft;
63+
64+dealloc_modify_header:
65+ mlx5dr_cmd_dealloc_modify_header(dmn->mdev, modify_hdr_id);
66+destroy_flow_group:
67+ mlx5dr_cmd_destroy_flow_group(dmn->mdev,
68+ MLX5_FLOW_TABLE_TYPE_FDB,
69+ table_id, group_id);
70+destroy_flow_table:
71+ mlx5dr_cmd_destroy_flow_table(dmn->mdev, table_id, MLX5_FLOW_TABLE_TYPE_FDB);
72+free_ttl_tbl:
73+ kfree(recalc_cs_ft);
74+ return NULL;
75+}
76+
77+void mlx5dr_fw_destroy_recalc_cs_ft(struct mlx5dr_domain *dmn,
78+ struct mlx5dr_fw_recalc_cs_ft *recalc_cs_ft)
79+{
80+ mlx5dr_cmd_del_flow_table_entry(dmn->mdev,
81+ MLX5_FLOW_TABLE_TYPE_FDB,
82+ recalc_cs_ft->table_id);
83+ mlx5dr_cmd_dealloc_modify_header(dmn->mdev, recalc_cs_ft->modify_hdr_id);
84+ mlx5dr_cmd_destroy_flow_group(dmn->mdev,
85+ MLX5_FLOW_TABLE_TYPE_FDB,
86+ recalc_cs_ft->table_id,
87+ recalc_cs_ft->group_id);
88+ mlx5dr_cmd_destroy_flow_table(dmn->mdev,
89+ recalc_cs_ft->table_id,
90+ MLX5_FLOW_TABLE_TYPE_FDB);
91+
92+ kfree(recalc_cs_ft);
93+}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_icm_pool.c+570−0
@@ -0,0 +1,570 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+/* Copyright (c) 2019 Mellanox Technologies. */
3+
4+#include "dr_types.h"
5+
6+#define DR_ICM_MODIFY_HDR_ALIGN_BASE 64
7+#define DR_ICM_SYNC_THRESHOLD (64 * 1024 * 1024)
8+
9+struct mlx5dr_icm_pool;
10+
11+struct mlx5dr_icm_bucket {
12+ struct mlx5dr_icm_pool *pool;
13+
14+ /* Chunks that aren't visible to HW not directly and not in cache */
15+ struct list_head free_list;
16+ unsigned int free_list_count;
17+
18+ /* Used chunks, HW may be accessing this memory */
19+ struct list_head used_list;
20+ unsigned int used_list_count;
21+
22+ /* HW may be accessing this memory but at some future,
23+ * undetermined time, it might cease to do so. Before deciding to call
24+ * sync_ste, this list is moved to sync_list
25+ */
26+ struct list_head hot_list;
27+ unsigned int hot_list_count;
28+
29+ /* Pending sync list, entries from the hot list are moved to this list.
30+ * sync_ste is executed and then sync_list is concatenated to the free list
31+ */
32+ struct list_head sync_list;
33+ unsigned int sync_list_count;
34+
35+ u32 total_chunks;
36+ u32 num_of_entries;
37+ u32 entry_size;
38+ /* protect the ICM bucket */
39+ struct mutex mutex;
40+};
41+
42+struct mlx5dr_icm_pool {
43+ struct mlx5dr_icm_bucket *buckets;
44+ enum mlx5dr_icm_type icm_type;
45+ enum mlx5dr_icm_chunk_size max_log_chunk_sz;
46+ enum mlx5dr_icm_chunk_size num_of_buckets;
47+ struct list_head icm_mr_list;
48+ /* protect the ICM MR list */
49+ struct mutex mr_mutex;
50+ struct mlx5dr_domain *dmn;
51+};
52+
53+struct mlx5dr_icm_dm {
54+ u32 obj_id;
55+ enum mlx5_sw_icm_type type;
56+ u64 addr;
57+ size_t length;
58+};
59+
60+struct mlx5dr_icm_mr {
61+ struct mlx5dr_icm_pool *pool;
62+ struct mlx5_core_mkey mkey;
63+ struct mlx5dr_icm_dm dm;
64+ size_t used_length;
65+ size_t length;
66+ u64 icm_start_addr;
67+ struct list_head mr_list;
68+};
69+
70+static int dr_icm_create_dm_mkey(struct mlx5_core_dev *mdev,
71+ u32 pd, u64 length, u64 start_addr, int mode,
72+ struct mlx5_core_mkey *mkey)
73+{
74+ u32 inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
75+ u32 in[MLX5_ST_SZ_DW(create_mkey_in)] = {};
76+ void *mkc;
77+
78+ mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry);
79+
80+ MLX5_SET(mkc, mkc, access_mode_1_0, mode);
81+ MLX5_SET(mkc, mkc, access_mode_4_2, (mode >> 2) & 0x7);
82+ MLX5_SET(mkc, mkc, lw, 1);
83+ MLX5_SET(mkc, mkc, lr, 1);
84+ if (mode == MLX5_MKC_ACCESS_MODE_SW_ICM) {
85+ MLX5_SET(mkc, mkc, rw, 1);
86+ MLX5_SET(mkc, mkc, rr, 1);
87+ }
88+
89+ MLX5_SET64(mkc, mkc, len, length);
90+ MLX5_SET(mkc, mkc, pd, pd);
91+ MLX5_SET(mkc, mkc, qpn, 0xffffff);
92+ MLX5_SET64(mkc, mkc, start_addr, start_addr);
93+
94+ return mlx5_core_create_mkey(mdev, mkey, in, inlen);
95+}
96+
97+static struct mlx5dr_icm_mr *
98+dr_icm_pool_mr_create(struct mlx5dr_icm_pool *pool,
99+ enum mlx5_sw_icm_type type,
100+ size_t align_base)
101+{
102+ struct mlx5_core_dev *mdev = pool->dmn->mdev;
103+ struct mlx5dr_icm_mr *icm_mr;
104+ size_t align_diff;
105+ int err;
106+
107+ icm_mr = kvzalloc(sizeof(*icm_mr), GFP_KERNEL);
108+ if (!icm_mr)
109+ return NULL;
110+
111+ icm_mr->pool = pool;
112+ INIT_LIST_HEAD(&icm_mr->mr_list);
113+
114+ icm_mr->dm.type = type;
115+
116+ /* 2^log_biggest_table * entry-size * double-for-alignment */
117+ icm_mr->dm.length = mlx5dr_icm_pool_chunk_size_to_byte(pool->max_log_chunk_sz,
118+ pool->icm_type) * 2;
119+
120+ err = mlx5_dm_sw_icm_alloc(mdev, icm_mr->dm.type, icm_mr->dm.length, 0,
121+ &icm_mr->dm.addr, &icm_mr->dm.obj_id);
122+ if (err) {
123+ mlx5dr_err(pool->dmn, "Failed to allocate SW ICM memory, err (%d)\n", err);
124+ goto free_icm_mr;
125+ }
126+
127+ /* Register device memory */
128+ err = dr_icm_create_dm_mkey(mdev, pool->dmn->pdn,
129+ icm_mr->dm.length,
130+ icm_mr->dm.addr,
131+ MLX5_MKC_ACCESS_MODE_SW_ICM,
132+ &icm_mr->mkey);
133+ if (err) {
134+ mlx5dr_err(pool->dmn, "Failed to create SW ICM MKEY, err (%d)\n", err);
135+ goto free_dm;
136+ }
137+
138+ icm_mr->icm_start_addr = icm_mr->dm.addr;
139+
140+ align_diff = icm_mr->icm_start_addr % align_base;
141+ if (align_diff)
142+ icm_mr->used_length = align_base - align_diff;
143+
144+ list_add_tail(&icm_mr->mr_list, &pool->icm_mr_list);
145+
146+ return icm_mr;
147+
148+free_dm:
149+ mlx5_dm_sw_icm_dealloc(mdev, icm_mr->dm.type, icm_mr->dm.length, 0,
150+ icm_mr->dm.addr, icm_mr->dm.obj_id);
151+free_icm_mr:
152+ kvfree(icm_mr);
153+ return NULL;
154+}
155+
156+static void dr_icm_pool_mr_destroy(struct mlx5dr_icm_mr *icm_mr)
157+{
158+ struct mlx5_core_dev *mdev = icm_mr->pool->dmn->mdev;
159+ struct mlx5dr_icm_dm *dm = &icm_mr->dm;
160+
161+ list_del(&icm_mr->mr_list);
162+ mlx5_core_destroy_mkey(mdev, &icm_mr->mkey);
163+ mlx5_dm_sw_icm_dealloc(mdev, dm->type, dm->length, 0,
164+ dm->addr, dm->obj_id);
165+ kvfree(icm_mr);
166+}
167+
168+static int dr_icm_chunk_ste_init(struct mlx5dr_icm_chunk *chunk)
169+{
170+ struct mlx5dr_icm_bucket *bucket = chunk->bucket;
171+
172+ chunk->ste_arr = kvzalloc(bucket->num_of_entries *
173+ sizeof(chunk->ste_arr[0]), GFP_KERNEL);
174+ if (!chunk->ste_arr)
175+ return -ENOMEM;
176+
177+ chunk->hw_ste_arr = kvzalloc(bucket->num_of_entries *
178+ DR_STE_SIZE_REDUCED, GFP_KERNEL);
179+ if (!chunk->hw_ste_arr)
180+ goto out_free_ste_arr;
181+
182+ chunk->miss_list = kvmalloc(bucket->num_of_entries *
183+ sizeof(chunk->miss_list[0]), GFP_KERNEL);
184+ if (!chunk->miss_list)
185+ goto out_free_hw_ste_arr;
186+
187+ return 0;
188+
189+out_free_hw_ste_arr:
190+ kvfree(chunk->hw_ste_arr);
191+out_free_ste_arr:
192+ kvfree(chunk->ste_arr);
193+ return -ENOMEM;
194+}
195+
196+static int dr_icm_chunks_create(struct mlx5dr_icm_bucket *bucket)
197+{
198+ size_t mr_free_size, mr_req_size, mr_row_size;
199+ struct mlx5dr_icm_pool *pool = bucket->pool;
200+ struct mlx5dr_icm_mr *icm_mr = NULL;
201+ struct mlx5dr_icm_chunk *chunk;
202+ enum mlx5_sw_icm_type dm_type;
203+ size_t align_base;
204+ int i, err = 0;
205+
206+ mr_req_size = bucket->num_of_entries * bucket->entry_size;
207+ mr_row_size = mlx5dr_icm_pool_chunk_size_to_byte(pool->max_log_chunk_sz,
208+ pool->icm_type);
209+
210+ if (pool->icm_type == DR_ICM_TYPE_STE) {
211+ dm_type = MLX5_SW_ICM_TYPE_STEERING;
212+ /* Align base is the biggest chunk size / row size */
213+ align_base = mr_row_size;
214+ } else {
215+ dm_type = MLX5_SW_ICM_TYPE_HEADER_MODIFY;
216+ /* Align base is 64B */
217+ align_base = DR_ICM_MODIFY_HDR_ALIGN_BASE;
218+ }
219+
220+ mutex_lock(&pool->mr_mutex);
221+ if (!list_empty(&pool->icm_mr_list)) {
222+ icm_mr = list_last_entry(&pool->icm_mr_list,
223+ struct mlx5dr_icm_mr, mr_list);
224+
225+ if (icm_mr)
226+ mr_free_size = icm_mr->dm.length - icm_mr->used_length;
227+ }
228+
229+ if (!icm_mr || mr_free_size < mr_row_size) {
230+ icm_mr = dr_icm_pool_mr_create(pool, dm_type, align_base);
231+ if (!icm_mr) {
232+ err = -ENOMEM;
233+ goto out_err;
234+ }
235+ }
236+
237+ /* Create memory aligned chunks */
238+ for (i = 0; i < mr_row_size / mr_req_size; i++) {
239+ chunk = kvzalloc(sizeof(*chunk), GFP_KERNEL);
240+ if (!chunk) {
241+ err = -ENOMEM;
242+ goto out_err;
243+ }
244+
245+ chunk->bucket = bucket;
246+ chunk->rkey = icm_mr->mkey.key;
247+ /* mr start addr is zero based */
248+ chunk->mr_addr = icm_mr->used_length;
249+ chunk->icm_addr = (uintptr_t)icm_mr->icm_start_addr + icm_mr->used_length;
250+ icm_mr->used_length += mr_req_size;
251+ chunk->num_of_entries = bucket->num_of_entries;
252+ chunk->byte_size = chunk->num_of_entries * bucket->entry_size;
253+
254+ if (pool->icm_type == DR_ICM_TYPE_STE) {
255+ err = dr_icm_chunk_ste_init(chunk);
256+ if (err)
257+ goto out_free_chunk;
258+ }
259+
260+ INIT_LIST_HEAD(&chunk->chunk_list);
261+ list_add(&chunk->chunk_list, &bucket->free_list);
262+ bucket->free_list_count++;
263+ bucket->total_chunks++;
264+ }
265+ mutex_unlock(&pool->mr_mutex);
266+ return 0;
267+
268+out_free_chunk:
269+ kvfree(chunk);
270+out_err:
271+ mutex_unlock(&pool->mr_mutex);
272+ return err;
273+}
274+
275+static void dr_icm_chunk_ste_cleanup(struct mlx5dr_icm_chunk *chunk)
276+{
277+ kvfree(chunk->miss_list);
278+ kvfree(chunk->hw_ste_arr);
279+ kvfree(chunk->ste_arr);
280+}
281+
282+static void dr_icm_chunk_destroy(struct mlx5dr_icm_chunk *chunk)
283+{
284+ struct mlx5dr_icm_bucket *bucket = chunk->bucket;
285+
286+ list_del(&chunk->chunk_list);
287+ bucket->total_chunks--;
288+
289+ if (bucket->pool->icm_type == DR_ICM_TYPE_STE)
290+ dr_icm_chunk_ste_cleanup(chunk);
291+
292+ kvfree(chunk);
293+}
294+
295+static void dr_icm_bucket_init(struct mlx5dr_icm_pool *pool,
296+ struct mlx5dr_icm_bucket *bucket,
297+ enum mlx5dr_icm_chunk_size chunk_size)
298+{
299+ if (pool->icm_type == DR_ICM_TYPE_STE)
300+ bucket->entry_size = DR_STE_SIZE;
301+ else
302+ bucket->entry_size = DR_MODIFY_ACTION_SIZE;
303+
304+ bucket->num_of_entries = mlx5dr_icm_pool_chunk_size_to_entries(chunk_size);
305+ bucket->pool = pool;
306+ mutex_init(&bucket->mutex);
307+ INIT_LIST_HEAD(&bucket->free_list);
308+ INIT_LIST_HEAD(&bucket->used_list);
309+ INIT_LIST_HEAD(&bucket->hot_list);
310+ INIT_LIST_HEAD(&bucket->sync_list);
311+}
312+
313+static void dr_icm_bucket_cleanup(struct mlx5dr_icm_bucket *bucket)
314+{
315+ struct mlx5dr_icm_chunk *chunk, *next;
316+
317+ mutex_destroy(&bucket->mutex);
318+ list_splice_tail_init(&bucket->sync_list, &bucket->free_list);
319+ list_splice_tail_init(&bucket->hot_list, &bucket->free_list);
320+
321+ list_for_each_entry_safe(chunk, next, &bucket->free_list, chunk_list)
322+ dr_icm_chunk_destroy(chunk);
323+
324+ WARN_ON(bucket->total_chunks != 0);
325+
326+ /* Cleanup of unreturned chunks */
327+ list_for_each_entry_safe(chunk, next, &bucket->used_list, chunk_list)
328+ dr_icm_chunk_destroy(chunk);
329+}
330+
331+static u64 dr_icm_hot_mem_size(struct mlx5dr_icm_pool *pool)
332+{
333+ u64 hot_size = 0;
334+ int chunk_order;
335+
336+ for (chunk_order = 0; chunk_order < pool->num_of_buckets; chunk_order++)
337+ hot_size += pool->buckets[chunk_order].hot_list_count *
338+ mlx5dr_icm_pool_chunk_size_to_byte(chunk_order, pool->icm_type);
339+
340+ return hot_size;
341+}
342+
343+static bool dr_icm_reuse_hot_entries(struct mlx5dr_icm_pool *pool,
344+ struct mlx5dr_icm_bucket *bucket)
345+{
346+ u64 bytes_for_sync;
347+
348+ bytes_for_sync = dr_icm_hot_mem_size(pool);
349+ if (bytes_for_sync < DR_ICM_SYNC_THRESHOLD || !bucket->hot_list_count)
350+ return false;
351+
352+ return true;
353+}
354+
355+static void dr_icm_chill_bucket_start(struct mlx5dr_icm_bucket *bucket)
356+{
357+ list_splice_tail_init(&bucket->hot_list, &bucket->sync_list);
358+ bucket->sync_list_count += bucket->hot_list_count;
359+ bucket->hot_list_count = 0;
360+}
361+
362+static void dr_icm_chill_bucket_end(struct mlx5dr_icm_bucket *bucket)
363+{
364+ list_splice_tail_init(&bucket->sync_list, &bucket->free_list);
365+ bucket->free_list_count += bucket->sync_list_count;
366+ bucket->sync_list_count = 0;
367+}
368+
369+static void dr_icm_chill_bucket_abort(struct mlx5dr_icm_bucket *bucket)
370+{
371+ list_splice_tail_init(&bucket->sync_list, &bucket->hot_list);
372+ bucket->hot_list_count += bucket->sync_list_count;
373+ bucket->sync_list_count = 0;
374+}
375+
376+static void dr_icm_chill_buckets_start(struct mlx5dr_icm_pool *pool,
377+ struct mlx5dr_icm_bucket *cb,
378+ bool buckets[DR_CHUNK_SIZE_MAX])
379+{
380+ struct mlx5dr_icm_bucket *bucket;
381+ int i;
382+
383+ for (i = 0; i < pool->num_of_buckets; i++) {
384+ bucket = &pool->buckets[i];
385+ if (bucket == cb) {
386+ dr_icm_chill_bucket_start(bucket);
387+ continue;
388+ }
389+
390+ /* Freeing the mutex is done at the end of that process, after
391+ * sync_ste was executed at dr_icm_chill_buckets_end func.
392+ */
393+ if (mutex_trylock(&bucket->mutex)) {
394+ dr_icm_chill_bucket_start(bucket);
395+ buckets[i] = true;
396+ }
397+ }
398+}
399+
400+static void dr_icm_chill_buckets_end(struct mlx5dr_icm_pool *pool,
401+ struct mlx5dr_icm_bucket *cb,
402+ bool buckets[DR_CHUNK_SIZE_MAX])
403+{
404+ struct mlx5dr_icm_bucket *bucket;
405+ int i;
406+
407+ for (i = 0; i < pool->num_of_buckets; i++) {
408+ bucket = &pool->buckets[i];
409+ if (bucket == cb) {
410+ dr_icm_chill_bucket_end(bucket);
411+ continue;
412+ }
413+
414+ if (!buckets[i])
415+ continue;
416+
417+ dr_icm_chill_bucket_end(bucket);
418+ mutex_unlock(&bucket->mutex);
419+ }
420+}
421+
422+static void dr_icm_chill_buckets_abort(struct mlx5dr_icm_pool *pool,
423+ struct mlx5dr_icm_bucket *cb,
424+ bool buckets[DR_CHUNK_SIZE_MAX])
425+{
426+ struct mlx5dr_icm_bucket *bucket;
427+ int i;
428+
429+ for (i = 0; i < pool->num_of_buckets; i++) {
430+ bucket = &pool->buckets[i];
431+ if (bucket == cb) {
432+ dr_icm_chill_bucket_abort(bucket);
433+ continue;
434+ }
435+
436+ if (!buckets[i])
437+ continue;
438+
439+ dr_icm_chill_bucket_abort(bucket);
440+ mutex_unlock(&bucket->mutex);
441+ }
442+}
443+
444+/* Allocate an ICM chunk, each chunk holds a piece of ICM memory and
445+ * also memory used for HW STE management for optimizations.
446+ */
447+struct mlx5dr_icm_chunk *
448+mlx5dr_icm_alloc_chunk(struct mlx5dr_icm_pool *pool,
449+ enum mlx5dr_icm_chunk_size chunk_size)
450+{
451+ struct mlx5dr_icm_chunk *chunk = NULL; /* Fix compilation warning */
452+ bool buckets[DR_CHUNK_SIZE_MAX] = {};
453+ struct mlx5dr_icm_bucket *bucket;
454+ int err;
455+
456+ if (chunk_size > pool->max_log_chunk_sz)
457+ return NULL;
458+
459+ bucket = &pool->buckets[chunk_size];
460+
461+ mutex_lock(&bucket->mutex);
462+
463+ /* Take chunk from pool if available, otherwise allocate new chunks */
464+ if (list_empty(&bucket->free_list)) {
465+ if (dr_icm_reuse_hot_entries(pool, bucket)) {
466+ dr_icm_chill_buckets_start(pool, bucket, buckets);
467+ err = mlx5dr_cmd_sync_steering(pool->dmn->mdev);
468+ if (err) {
469+ dr_icm_chill_buckets_abort(pool, bucket, buckets);
470+ mlx5dr_dbg(pool->dmn, "Sync_steering failed\n");
471+ chunk = NULL;
472+ goto out;
473+ }
474+ dr_icm_chill_buckets_end(pool, bucket, buckets);
475+ } else {
476+ dr_icm_chunks_create(bucket);
477+ }
478+ }
479+
480+ if (!list_empty(&bucket->free_list)) {
481+ chunk = list_last_entry(&bucket->free_list,
482+ struct mlx5dr_icm_chunk,
483+ chunk_list);
484+ if (chunk) {
485+ list_del_init(&chunk->chunk_list);
486+ list_add_tail(&chunk->chunk_list, &bucket->used_list);
487+ bucket->free_list_count--;
488+ bucket->used_list_count++;
489+ }
490+ }
491+out:
492+ mutex_unlock(&bucket->mutex);
493+ return chunk;
494+}
495+
496+void mlx5dr_icm_free_chunk(struct mlx5dr_icm_chunk *chunk)
497+{
498+ struct mlx5dr_icm_bucket *bucket = chunk->bucket;
499+
500+ if (bucket->pool->icm_type == DR_ICM_TYPE_STE) {
501+ memset(chunk->ste_arr, 0,
502+ bucket->num_of_entries * sizeof(chunk->ste_arr[0]));
503+ memset(chunk->hw_ste_arr, 0,
504+ bucket->num_of_entries * DR_STE_SIZE_REDUCED);
505+ }
506+
507+ mutex_lock(&bucket->mutex);
508+ list_del_init(&chunk->chunk_list);
509+ list_add_tail(&chunk->chunk_list, &bucket->hot_list);
510+ bucket->hot_list_count++;
511+ bucket->used_list_count--;
512+ mutex_unlock(&bucket->mutex);
513+}
514+
515+struct mlx5dr_icm_pool *mlx5dr_icm_pool_create(struct mlx5dr_domain *dmn,
516+ enum mlx5dr_icm_type icm_type)
517+{
518+ enum mlx5dr_icm_chunk_size max_log_chunk_sz;
519+ struct mlx5dr_icm_pool *pool;
520+ int i;
521+
522+ if (icm_type == DR_ICM_TYPE_STE)
523+ max_log_chunk_sz = dmn->info.max_log_sw_icm_sz;
524+ else
525+ max_log_chunk_sz = dmn->info.max_log_action_icm_sz;
526+
527+ pool = kvzalloc(sizeof(*pool), GFP_KERNEL);
528+ if (!pool)
529+ return NULL;
530+
531+ pool->buckets = kcalloc(max_log_chunk_sz + 1,
532+ sizeof(pool->buckets[0]),
533+ GFP_KERNEL);
534+ if (!pool->buckets)
535+ goto free_pool;
536+
537+ pool->dmn = dmn;
538+ pool->icm_type = icm_type;
539+ pool->max_log_chunk_sz = max_log_chunk_sz;
540+ pool->num_of_buckets = max_log_chunk_sz + 1;
541+ INIT_LIST_HEAD(&pool->icm_mr_list);
542+
543+ for (i = 0; i < pool->num_of_buckets; i++)
544+ dr_icm_bucket_init(pool, &pool->buckets[i], i);
545+
546+ mutex_init(&pool->mr_mutex);
547+
548+ return pool;
549+
550+free_pool:
551+ kvfree(pool);
552+ return NULL;
553+}
554+
555+void mlx5dr_icm_pool_destroy(struct mlx5dr_icm_pool *pool)
556+{
557+ struct mlx5dr_icm_mr *icm_mr, *next;
558+ int i;
559+
560+ mutex_destroy(&pool->mr_mutex);
561+
562+ list_for_each_entry_safe(icm_mr, next, &pool->icm_mr_list, mr_list)
563+ dr_icm_pool_mr_destroy(icm_mr);
564+
565+ for (i = 0; i < pool->num_of_buckets; i++)
566+ dr_icm_bucket_cleanup(&pool->buckets[i]);
567+
568+ kfree(pool->buckets);
569+ kvfree(pool);
570+}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_matcher.c+770−0
@@ -0,0 +1,770 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+/* Copyright (c) 2019 Mellanox Technologies. */
3+
4+#include "dr_types.h"
5+
6+static bool dr_mask_is_smac_set(struct mlx5dr_match_spec *spec)
7+{
8+ return (spec->smac_47_16 || spec->smac_15_0);
9+}
10+
11+static bool dr_mask_is_dmac_set(struct mlx5dr_match_spec *spec)
12+{
13+ return (spec->dmac_47_16 || spec->dmac_15_0);
14+}
15+
16+static bool dr_mask_is_src_addr_set(struct mlx5dr_match_spec *spec)
17+{
18+ return (spec->src_ip_127_96 || spec->src_ip_95_64 ||
19+ spec->src_ip_63_32 || spec->src_ip_31_0);
20+}
21+
22+static bool dr_mask_is_dst_addr_set(struct mlx5dr_match_spec *spec)
23+{
24+ return (spec->dst_ip_127_96 || spec->dst_ip_95_64 ||
25+ spec->dst_ip_63_32 || spec->dst_ip_31_0);
26+}
27+
28+static bool dr_mask_is_l3_base_set(struct mlx5dr_match_spec *spec)
29+{
30+ return (spec->ip_protocol || spec->frag || spec->tcp_flags ||
31+ spec->ip_ecn || spec->ip_dscp);
32+}
33+
34+static bool dr_mask_is_tcp_udp_base_set(struct mlx5dr_match_spec *spec)
35+{
36+ return (spec->tcp_sport || spec->tcp_dport ||
37+ spec->udp_sport || spec->udp_dport);
38+}
39+
40+static bool dr_mask_is_ipv4_set(struct mlx5dr_match_spec *spec)
41+{
42+ return (spec->dst_ip_31_0 || spec->src_ip_31_0);
43+}
44+
45+static bool dr_mask_is_ipv4_5_tuple_set(struct mlx5dr_match_spec *spec)
46+{
47+ return (dr_mask_is_l3_base_set(spec) ||
48+ dr_mask_is_tcp_udp_base_set(spec) ||
49+ dr_mask_is_ipv4_set(spec));
50+}
51+
52+static bool dr_mask_is_eth_l2_tnl_set(struct mlx5dr_match_misc *misc)
53+{
54+ return misc->vxlan_vni;
55+}
56+
57+static bool dr_mask_is_ttl_set(struct mlx5dr_match_spec *spec)
58+{
59+ return spec->ttl_hoplimit;
60+}
61+
62+#define DR_MASK_IS_L2_DST(_spec, _misc, _inner_outer) (_spec.first_vid || \
63+ (_spec).first_cfi || (_spec).first_prio || (_spec).cvlan_tag || \
64+ (_spec).svlan_tag || (_spec).dmac_47_16 || (_spec).dmac_15_0 || \
65+ (_spec).ethertype || (_spec).ip_version || \
66+ (_misc)._inner_outer##_second_vid || \
67+ (_misc)._inner_outer##_second_cfi || \
68+ (_misc)._inner_outer##_second_prio || \
69+ (_misc)._inner_outer##_second_cvlan_tag || \
70+ (_misc)._inner_outer##_second_svlan_tag)
71+
72+#define DR_MASK_IS_ETH_L4_SET(_spec, _misc, _inner_outer) ( \
73+ dr_mask_is_l3_base_set(&(_spec)) || \
74+ dr_mask_is_tcp_udp_base_set(&(_spec)) || \
75+ dr_mask_is_ttl_set(&(_spec)) || \
76+ (_misc)._inner_outer##_ipv6_flow_label)
77+
78+#define DR_MASK_IS_ETH_L4_MISC_SET(_misc3, _inner_outer) ( \
79+ (_misc3)._inner_outer##_tcp_seq_num || \
80+ (_misc3)._inner_outer##_tcp_ack_num)
81+
82+#define DR_MASK_IS_FIRST_MPLS_SET(_misc2, _inner_outer) ( \
83+ (_misc2)._inner_outer##_first_mpls_label || \
84+ (_misc2)._inner_outer##_first_mpls_exp || \
85+ (_misc2)._inner_outer##_first_mpls_s_bos || \
86+ (_misc2)._inner_outer##_first_mpls_ttl)
87+
88+static bool dr_mask_is_gre_set(struct mlx5dr_match_misc *misc)
89+{
90+ return (misc->gre_key_h || misc->gre_key_l ||
91+ misc->gre_protocol || misc->gre_c_present ||
92+ misc->gre_k_present || misc->gre_s_present);
93+}
94+
95+#define DR_MASK_IS_OUTER_MPLS_OVER_GRE_UDP_SET(_misc2, gre_udp) ( \
96+ (_misc2).outer_first_mpls_over_##gre_udp##_label || \
97+ (_misc2).outer_first_mpls_over_##gre_udp##_exp || \
98+ (_misc2).outer_first_mpls_over_##gre_udp##_s_bos || \
99+ (_misc2).outer_first_mpls_over_##gre_udp##_ttl)
100+
101+#define DR_MASK_IS_FLEX_PARSER_0_SET(_misc2) ( \
102+ DR_MASK_IS_OUTER_MPLS_OVER_GRE_UDP_SET((_misc2), gre) || \
103+ DR_MASK_IS_OUTER_MPLS_OVER_GRE_UDP_SET((_misc2), udp))
104+
105+static bool dr_mask_is_flex_parser_tnl_set(struct mlx5dr_match_misc3 *misc3)
106+{
107+ return (misc3->outer_vxlan_gpe_vni ||
108+ misc3->outer_vxlan_gpe_next_protocol ||
109+ misc3->outer_vxlan_gpe_flags);
110+}
111+
112+static bool dr_mask_is_flex_parser_icmpv6_set(struct mlx5dr_match_misc3 *misc3)
113+{
114+ return (misc3->icmpv6_type || misc3->icmpv6_code ||
115+ misc3->icmpv6_header_data);
116+}
117+
118+static bool dr_mask_is_wqe_metadata_set(struct mlx5dr_match_misc2 *misc2)
119+{
120+ return misc2->metadata_reg_a;
121+}
122+
123+static bool dr_mask_is_reg_c_0_3_set(struct mlx5dr_match_misc2 *misc2)
124+{
125+ return (misc2->metadata_reg_c_0 || misc2->metadata_reg_c_1 ||
126+ misc2->metadata_reg_c_2 || misc2->metadata_reg_c_3);
127+}
128+
129+static bool dr_mask_is_reg_c_4_7_set(struct mlx5dr_match_misc2 *misc2)
130+{
131+ return (misc2->metadata_reg_c_4 || misc2->metadata_reg_c_5 ||
132+ misc2->metadata_reg_c_6 || misc2->metadata_reg_c_7);
133+}
134+
135+static bool dr_mask_is_gvmi_or_qpn_set(struct mlx5dr_match_misc *misc)
136+{
137+ return (misc->source_sqn || misc->source_port);
138+}
139+
140+static bool
141+dr_matcher_supp_flex_parser_vxlan_gpe(struct mlx5dr_domain *dmn)
142+{
143+ return dmn->info.caps.flex_protocols &
144+ MLX5_FLEX_PARSER_VXLAN_GPE_ENABLED;
145+}
146+
147+int mlx5dr_matcher_select_builders(struct mlx5dr_matcher *matcher,
148+ struct mlx5dr_matcher_rx_tx *nic_matcher,
149+ bool ipv6)
150+{
151+ if (ipv6) {
152+ nic_matcher->ste_builder = nic_matcher->ste_builder6;
153+ nic_matcher->num_of_builders = nic_matcher->num_of_builders6;
154+ } else {
155+ nic_matcher->ste_builder = nic_matcher->ste_builder4;
156+ nic_matcher->num_of_builders = nic_matcher->num_of_builders4;
157+ }
158+
159+ if (!nic_matcher->num_of_builders) {
160+ mlx5dr_dbg(matcher->tbl->dmn,
161+ "Rule not supported on this matcher due to IP related fields\n");
162+ return -EINVAL;
163+ }
164+
165+ return 0;
166+}
167+
168+static int dr_matcher_set_ste_builders(struct mlx5dr_matcher *matcher,
169+ struct mlx5dr_matcher_rx_tx *nic_matcher,
170+ bool ipv6)
171+{
172+ struct mlx5dr_domain_rx_tx *nic_dmn = nic_matcher->nic_tbl->nic_dmn;
173+ struct mlx5dr_domain *dmn = matcher->tbl->dmn;
174+ struct mlx5dr_match_param mask = {};
175+ struct mlx5dr_match_misc3 *misc3;
176+ struct mlx5dr_ste_build *sb;
177+ u8 *num_of_builders;
178+ bool inner, rx;
179+ int idx = 0;
180+ int ret, i;
181+
182+ if (ipv6) {
183+ sb = nic_matcher->ste_builder6;
184+ num_of_builders = &nic_matcher->num_of_builders6;
185+ } else {
186+ sb = nic_matcher->ste_builder4;
187+ num_of_builders = &nic_matcher->num_of_builders4;
188+ }
189+
190+ rx = nic_dmn->ste_type == MLX5DR_STE_TYPE_RX;
191+
192+ /* Create a temporary mask to track and clear used mask fields */
193+ if (matcher->match_criteria & DR_MATCHER_CRITERIA_OUTER)
194+ mask.outer = matcher->mask.outer;
195+
196+ if (matcher->match_criteria & DR_MATCHER_CRITERIA_MISC)
197+ mask.misc = matcher->mask.misc;
198+
199+ if (matcher->match_criteria & DR_MATCHER_CRITERIA_INNER)
200+ mask.inner = matcher->mask.inner;
201+
202+ if (matcher->match_criteria & DR_MATCHER_CRITERIA_MISC2)
203+ mask.misc2 = matcher->mask.misc2;
204+
205+ if (matcher->match_criteria & DR_MATCHER_CRITERIA_MISC3)
206+ mask.misc3 = matcher->mask.misc3;
207+
208+ ret = mlx5dr_ste_build_pre_check(dmn, matcher->match_criteria,
209+ &matcher->mask, NULL);
210+ if (ret)
211+ return ret;
212+
213+ /* Outer */
214+ if (matcher->match_criteria & (DR_MATCHER_CRITERIA_OUTER |
215+ DR_MATCHER_CRITERIA_MISC |
216+ DR_MATCHER_CRITERIA_MISC2 |
217+ DR_MATCHER_CRITERIA_MISC3)) {
218+ inner = false;
219+
220+ if (dr_mask_is_wqe_metadata_set(&mask.misc2))
221+ mlx5dr_ste_build_general_purpose(&sb[idx++], &mask, inner, rx);
222+
223+ if (dr_mask_is_reg_c_0_3_set(&mask.misc2))
224+ mlx5dr_ste_build_register_0(&sb[idx++], &mask, inner, rx);
225+
226+ if (dr_mask_is_reg_c_4_7_set(&mask.misc2))
227+ mlx5dr_ste_build_register_1(&sb[idx++], &mask, inner, rx);
228+
229+ if (dr_mask_is_gvmi_or_qpn_set(&mask.misc) &&
230+ (dmn->type == MLX5DR_DOMAIN_TYPE_FDB ||
231+ dmn->type == MLX5DR_DOMAIN_TYPE_NIC_RX)) {
232+ ret = mlx5dr_ste_build_src_gvmi_qpn(&sb[idx++], &mask,
233+ &dmn->info.caps,
234+ inner, rx);
235+ if (ret)
236+ return ret;
237+ }
238+
239+ if (dr_mask_is_smac_set(&mask.outer) &&
240+ dr_mask_is_dmac_set(&mask.outer)) {
241+ ret = mlx5dr_ste_build_eth_l2_src_des(&sb[idx++], &mask,
242+ inner, rx);
243+ if (ret)
244+ return ret;
245+ }
246+
247+ if (dr_mask_is_smac_set(&mask.outer))
248+ mlx5dr_ste_build_eth_l2_src(&sb[idx++], &mask, inner, rx);
249+
250+ if (DR_MASK_IS_L2_DST(mask.outer, mask.misc, outer))
251+ mlx5dr_ste_build_eth_l2_dst(&sb[idx++], &mask, inner, rx);
252+
253+ if (ipv6) {
254+ if (dr_mask_is_dst_addr_set(&mask.outer))
255+ mlx5dr_ste_build_eth_l3_ipv6_dst(&sb[idx++], &mask,
256+ inner, rx);
257+
258+ if (dr_mask_is_src_addr_set(&mask.outer))
259+ mlx5dr_ste_build_eth_l3_ipv6_src(&sb[idx++], &mask,
260+ inner, rx);
261+
262+ if (DR_MASK_IS_ETH_L4_SET(mask.outer, mask.misc, outer))
263+ mlx5dr_ste_build_ipv6_l3_l4(&sb[idx++], &mask,
264+ inner, rx);
265+ } else {
266+ if (dr_mask_is_ipv4_5_tuple_set(&mask.outer))
267+ mlx5dr_ste_build_eth_l3_ipv4_5_tuple(&sb[idx++], &mask,
268+ inner, rx);
269+
270+ if (dr_mask_is_ttl_set(&mask.outer))
271+ mlx5dr_ste_build_eth_l3_ipv4_misc(&sb[idx++], &mask,
272+ inner, rx);
273+ }
274+
275+ if (dr_mask_is_flex_parser_tnl_set(&mask.misc3) &&
276+ dr_matcher_supp_flex_parser_vxlan_gpe(dmn))
277+ mlx5dr_ste_build_flex_parser_tnl(&sb[idx++], &mask,
278+ inner, rx);
279+
280+ if (DR_MASK_IS_ETH_L4_MISC_SET(mask.misc3, outer))
281+ mlx5dr_ste_build_eth_l4_misc(&sb[idx++], &mask, inner, rx);
282+
283+ if (DR_MASK_IS_FIRST_MPLS_SET(mask.misc2, outer))
284+ mlx5dr_ste_build_mpls(&sb[idx++], &mask, inner, rx);
285+
286+ if (DR_MASK_IS_FLEX_PARSER_0_SET(mask.misc2))
287+ mlx5dr_ste_build_flex_parser_0(&sb[idx++], &mask,
288+ inner, rx);
289+
290+ misc3 = &mask.misc3;
291+ if ((DR_MASK_IS_FLEX_PARSER_ICMPV4_SET(misc3) &&
292+ mlx5dr_matcher_supp_flex_parser_icmp_v4(&dmn->info.caps)) ||
293+ (dr_mask_is_flex_parser_icmpv6_set(&mask.misc3) &&
294+ mlx5dr_matcher_supp_flex_parser_icmp_v6(&dmn->info.caps))) {
295+ ret = mlx5dr_ste_build_flex_parser_1(&sb[idx++],
296+ &mask, &dmn->info.caps,
297+ inner, rx);
298+ if (ret)
299+ return ret;
300+ }
301+ if (dr_mask_is_gre_set(&mask.misc))
302+ mlx5dr_ste_build_gre(&sb[idx++], &mask, inner, rx);
303+ }
304+
305+ /* Inner */
306+ if (matcher->match_criteria & (DR_MATCHER_CRITERIA_INNER |
307+ DR_MATCHER_CRITERIA_MISC |
308+ DR_MATCHER_CRITERIA_MISC2 |
309+ DR_MATCHER_CRITERIA_MISC3)) {
310+ inner = true;
311+
312+ if (dr_mask_is_eth_l2_tnl_set(&mask.misc))
313+ mlx5dr_ste_build_eth_l2_tnl(&sb[idx++], &mask, inner, rx);
314+
315+ if (dr_mask_is_smac_set(&mask.inner) &&
316+ dr_mask_is_dmac_set(&mask.inner)) {
317+ ret = mlx5dr_ste_build_eth_l2_src_des(&sb[idx++],
318+ &mask, inner, rx);
319+ if (ret)
320+ return ret;
321+ }
322+
323+ if (dr_mask_is_smac_set(&mask.inner))
324+ mlx5dr_ste_build_eth_l2_src(&sb[idx++], &mask, inner, rx);
325+
326+ if (DR_MASK_IS_L2_DST(mask.inner, mask.misc, inner))
327+ mlx5dr_ste_build_eth_l2_dst(&sb[idx++], &mask, inner, rx);
328+
329+ if (ipv6) {
330+ if (dr_mask_is_dst_addr_set(&mask.inner))
331+ mlx5dr_ste_build_eth_l3_ipv6_dst(&sb[idx++], &mask,
332+ inner, rx);
333+
334+ if (dr_mask_is_src_addr_set(&mask.inner))
335+ mlx5dr_ste_build_eth_l3_ipv6_src(&sb[idx++], &mask,
336+ inner, rx);
337+
338+ if (DR_MASK_IS_ETH_L4_SET(mask.inner, mask.misc, inner))
339+ mlx5dr_ste_build_ipv6_l3_l4(&sb[idx++], &mask,
340+ inner, rx);
341+ } else {
342+ if (dr_mask_is_ipv4_5_tuple_set(&mask.inner))
343+ mlx5dr_ste_build_eth_l3_ipv4_5_tuple(&sb[idx++], &mask,
344+ inner, rx);
345+
346+ if (dr_mask_is_ttl_set(&mask.inner))
347+ mlx5dr_ste_build_eth_l3_ipv4_misc(&sb[idx++], &mask,
348+ inner, rx);
349+ }
350+
351+ if (DR_MASK_IS_ETH_L4_MISC_SET(mask.misc3, inner))
352+ mlx5dr_ste_build_eth_l4_misc(&sb[idx++], &mask, inner, rx);
353+
354+ if (DR_MASK_IS_FIRST_MPLS_SET(mask.misc2, inner))
355+ mlx5dr_ste_build_mpls(&sb[idx++], &mask, inner, rx);
356+
357+ if (DR_MASK_IS_FLEX_PARSER_0_SET(mask.misc2))
358+ mlx5dr_ste_build_flex_parser_0(&sb[idx++], &mask, inner, rx);
359+ }
360+ /* Empty matcher, takes all */
361+ if (matcher->match_criteria == DR_MATCHER_CRITERIA_EMPTY)
362+ mlx5dr_ste_build_empty_always_hit(&sb[idx++], rx);
363+
364+ if (idx == 0) {
365+ mlx5dr_dbg(dmn, "Cannot generate any valid rules from mask\n");
366+ return -EINVAL;
367+ }
368+
369+ /* Check that all mask fields were consumed */
370+ for (i = 0; i < sizeof(struct mlx5dr_match_param); i++) {
371+ if (((u8 *)&mask)[i] != 0) {
372+ mlx5dr_info(dmn, "Mask contains unsupported parameters\n");
373+ return -EOPNOTSUPP;
374+ }
375+ }
376+
377+ *num_of_builders = idx;
378+
379+ return 0;
380+}
381+
382+static int dr_matcher_connect(struct mlx5dr_domain *dmn,
383+ struct mlx5dr_matcher_rx_tx *curr_nic_matcher,
384+ struct mlx5dr_matcher_rx_tx *next_nic_matcher,
385+ struct mlx5dr_matcher_rx_tx *prev_nic_matcher)
386+{
387+ struct mlx5dr_table_rx_tx *nic_tbl = curr_nic_matcher->nic_tbl;
388+ struct mlx5dr_domain_rx_tx *nic_dmn = nic_tbl->nic_dmn;
389+ struct mlx5dr_htbl_connect_info info;
390+ struct mlx5dr_ste_htbl *prev_htbl;
391+ int ret;
392+
393+ /* Connect end anchor hash table to next_htbl or to the default address */
394+ if (next_nic_matcher) {
395+ info.type = CONNECT_HIT;
396+ info.hit_next_htbl = next_nic_matcher->s_htbl;
397+ } else {
398+ info.type = CONNECT_MISS;
399+ info.miss_icm_addr = nic_tbl->default_icm_addr;
400+ }
401+ ret = mlx5dr_ste_htbl_init_and_postsend(dmn, nic_dmn,
402+ curr_nic_matcher->e_anchor,
403+ &info, info.type == CONNECT_HIT);
404+ if (ret)
405+ return ret;
406+
407+ /* Connect start hash table to end anchor */
408+ info.type = CONNECT_MISS;
409+ info.miss_icm_addr = curr_nic_matcher->e_anchor->chunk->icm_addr;
410+ ret = mlx5dr_ste_htbl_init_and_postsend(dmn, nic_dmn,
411+ curr_nic_matcher->s_htbl,
412+ &info, false);
413+ if (ret)
414+ return ret;
415+
416+ /* Connect previous hash table to matcher start hash table */
417+ if (prev_nic_matcher)
418+ prev_htbl = prev_nic_matcher->e_anchor;
419+ else
420+ prev_htbl = nic_tbl->s_anchor;
421+
422+ info.type = CONNECT_HIT;
423+ info.hit_next_htbl = curr_nic_matcher->s_htbl;
424+ ret = mlx5dr_ste_htbl_init_and_postsend(dmn, nic_dmn, prev_htbl,
425+ &info, true);
426+ if (ret)
427+ return ret;
428+
429+ /* Update the pointing ste and next hash table */
430+ curr_nic_matcher->s_htbl->pointing_ste = prev_htbl->ste_arr;
431+ prev_htbl->ste_arr[0].next_htbl = curr_nic_matcher->s_htbl;
432+
433+ if (next_nic_matcher) {
434+ next_nic_matcher->s_htbl->pointing_ste = curr_nic_matcher->e_anchor->ste_arr;
435+ curr_nic_matcher->e_anchor->ste_arr[0].next_htbl = next_nic_matcher->s_htbl;
436+ }
437+
438+ return 0;
439+}
440+
441+static int dr_matcher_add_to_tbl(struct mlx5dr_matcher *matcher)
442+{
443+ struct mlx5dr_matcher *next_matcher, *prev_matcher, *tmp_matcher;
444+ struct mlx5dr_table *tbl = matcher->tbl;
445+ struct mlx5dr_domain *dmn = tbl->dmn;
446+ bool first = true;
447+ int ret;
448+
449+ next_matcher = NULL;
450+ if (!list_empty(&tbl->matcher_list))
451+ list_for_each_entry(tmp_matcher, &tbl->matcher_list, matcher_list) {
452+ if (tmp_matcher->prio >= matcher->prio) {
453+ next_matcher = tmp_matcher;
454+ break;
455+ }
456+ first = false;
457+ }
458+
459+ prev_matcher = NULL;
460+ if (next_matcher && !first)
461+ prev_matcher = list_entry(next_matcher->matcher_list.prev,
462+ struct mlx5dr_matcher,
463+ matcher_list);
464+ else if (!first)
465+ prev_matcher = list_entry(tbl->matcher_list.prev,
466+ struct mlx5dr_matcher,
467+ matcher_list);
468+
469+ if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB ||
470+ dmn->type == MLX5DR_DOMAIN_TYPE_NIC_RX) {
471+ ret = dr_matcher_connect(dmn, &matcher->rx,
472+ next_matcher ? &next_matcher->rx : NULL,
473+ prev_matcher ? &prev_matcher->rx : NULL);
474+ if (ret)
475+ return ret;
476+ }
477+
478+ if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB ||
479+ dmn->type == MLX5DR_DOMAIN_TYPE_NIC_TX) {
480+ ret = dr_matcher_connect(dmn, &matcher->tx,
481+ next_matcher ? &next_matcher->tx : NULL,
482+ prev_matcher ? &prev_matcher->tx : NULL);
483+ if (ret)
484+ return ret;
485+ }
486+
487+ if (prev_matcher)
488+ list_add(&matcher->matcher_list, &prev_matcher->matcher_list);
489+ else if (next_matcher)
490+ list_add_tail(&matcher->matcher_list,
491+ &next_matcher->matcher_list);
492+ else
493+ list_add(&matcher->matcher_list, &tbl->matcher_list);
494+
495+ return 0;
496+}
497+
498+static void dr_matcher_uninit_nic(struct mlx5dr_matcher_rx_tx *nic_matcher)
499+{
500+ mlx5dr_htbl_put(nic_matcher->s_htbl);
501+ mlx5dr_htbl_put(nic_matcher->e_anchor);
502+}
503+
504+static void dr_matcher_uninit_fdb(struct mlx5dr_matcher *matcher)
505+{
506+ dr_matcher_uninit_nic(&matcher->rx);
507+ dr_matcher_uninit_nic(&matcher->tx);
508+}
509+
510+static void dr_matcher_uninit(struct mlx5dr_matcher *matcher)
511+{
512+ struct mlx5dr_domain *dmn = matcher->tbl->dmn;
513+
514+ switch (dmn->type) {
515+ case MLX5DR_DOMAIN_TYPE_NIC_RX:
516+ dr_matcher_uninit_nic(&matcher->rx);
517+ break;
518+ case MLX5DR_DOMAIN_TYPE_NIC_TX:
519+ dr_matcher_uninit_nic(&matcher->tx);
520+ break;
521+ case MLX5DR_DOMAIN_TYPE_FDB:
522+ dr_matcher_uninit_fdb(matcher);
523+ break;
524+ default:
525+ WARN_ON(true);
526+ break;
527+ }
528+}
529+
530+static int dr_matcher_init_nic(struct mlx5dr_matcher *matcher,
531+ struct mlx5dr_matcher_rx_tx *nic_matcher)
532+{
533+ struct mlx5dr_domain *dmn = matcher->tbl->dmn;
534+ int ret, ret_v4, ret_v6;
535+
536+ ret_v4 = dr_matcher_set_ste_builders(matcher, nic_matcher, false);
537+ ret_v6 = dr_matcher_set_ste_builders(matcher, nic_matcher, true);
538+
539+ if (ret_v4 && ret_v6) {
540+ mlx5dr_dbg(dmn, "Cannot generate IPv4 or IPv6 rules with given mask\n");
541+ return -EINVAL;
542+ }
543+
544+ if (!ret_v4)
545+ nic_matcher->ste_builder = nic_matcher->ste_builder4;
546+ else
547+ nic_matcher->ste_builder = nic_matcher->ste_builder6;
548+
549+ nic_matcher->e_anchor = mlx5dr_ste_htbl_alloc(dmn->ste_icm_pool,
550+ DR_CHUNK_SIZE_1,
551+ MLX5DR_STE_LU_TYPE_DONT_CARE,
552+ 0);
553+ if (!nic_matcher->e_anchor)
554+ return -ENOMEM;
555+
556+ nic_matcher->s_htbl = mlx5dr_ste_htbl_alloc(dmn->ste_icm_pool,
557+ DR_CHUNK_SIZE_1,
558+ nic_matcher->ste_builder[0].lu_type,
559+ nic_matcher->ste_builder[0].byte_mask);
560+ if (!nic_matcher->s_htbl) {
561+ ret = -ENOMEM;
562+ goto free_e_htbl;
563+ }
564+
565+ /* make sure the tables exist while empty */
566+ mlx5dr_htbl_get(nic_matcher->s_htbl);
567+ mlx5dr_htbl_get(nic_matcher->e_anchor);
568+
569+ return 0;
570+
571+free_e_htbl:
572+ mlx5dr_ste_htbl_free(nic_matcher->e_anchor);
573+ return ret;
574+}
575+
576+static int dr_matcher_init_fdb(struct mlx5dr_matcher *matcher)
577+{
578+ int ret;
579+
580+ ret = dr_matcher_init_nic(matcher, &matcher->rx);
581+ if (ret)
582+ return ret;
583+
584+ ret = dr_matcher_init_nic(matcher, &matcher->tx);
585+ if (ret)
586+ goto uninit_nic_rx;
587+
588+ return 0;
589+
590+uninit_nic_rx:
591+ dr_matcher_uninit_nic(&matcher->rx);
592+ return ret;
593+}
594+
595+static int dr_matcher_init(struct mlx5dr_matcher *matcher,
596+ struct mlx5dr_match_parameters *mask)
597+{
598+ struct mlx5dr_table *tbl = matcher->tbl;
599+ struct mlx5dr_domain *dmn = tbl->dmn;
600+ int ret;
601+
602+ if (matcher->match_criteria >= DR_MATCHER_CRITERIA_MAX) {
603+ mlx5dr_info(dmn, "Invalid match criteria attribute\n");
604+ return -EINVAL;
605+ }
606+
607+ if (mask) {
608+ if (mask->match_sz > sizeof(struct mlx5dr_match_param)) {
609+ mlx5dr_info(dmn, "Invalid match size attribute\n");
610+ return -EINVAL;
611+ }
612+ mlx5dr_ste_copy_param(matcher->match_criteria,
613+ &matcher->mask, mask);
614+ }
615+
616+ switch (dmn->type) {
617+ case MLX5DR_DOMAIN_TYPE_NIC_RX:
618+ matcher->rx.nic_tbl = &tbl->rx;
619+ ret = dr_matcher_init_nic(matcher, &matcher->rx);
620+ break;
621+ case MLX5DR_DOMAIN_TYPE_NIC_TX:
622+ matcher->tx.nic_tbl = &tbl->tx;
623+ ret = dr_matcher_init_nic(matcher, &matcher->tx);
624+ break;
625+ case MLX5DR_DOMAIN_TYPE_FDB:
626+ matcher->rx.nic_tbl = &tbl->rx;
627+ matcher->tx.nic_tbl = &tbl->tx;
628+ ret = dr_matcher_init_fdb(matcher);
629+ break;
630+ default:
631+ WARN_ON(true);
632+ return -EINVAL;
633+ }
634+
635+ return ret;
636+}
637+
638+struct mlx5dr_matcher *
639+mlx5dr_matcher_create(struct mlx5dr_table *tbl,
640+ u16 priority,
641+ u8 match_criteria_enable,
642+ struct mlx5dr_match_parameters *mask)
643+{
644+ struct mlx5dr_matcher *matcher;
645+ int ret;
646+
647+ refcount_inc(&tbl->refcount);
648+
649+ matcher = kzalloc(sizeof(*matcher), GFP_KERNEL);
650+ if (!matcher)
651+ goto dec_ref;
652+
653+ matcher->tbl = tbl;
654+ matcher->prio = priority;
655+ matcher->match_criteria = match_criteria_enable;
656+ refcount_set(&matcher->refcount, 1);
657+ INIT_LIST_HEAD(&matcher->matcher_list);
658+
659+ mutex_lock(&tbl->dmn->mutex);
660+
661+ ret = dr_matcher_init(matcher, mask);
662+ if (ret)
663+ goto free_matcher;
664+
665+ ret = dr_matcher_add_to_tbl(matcher);
666+ if (ret)
667+ goto matcher_uninit;
668+
669+ mutex_unlock(&tbl->dmn->mutex);
670+
671+ return matcher;
672+
673+matcher_uninit:
674+ dr_matcher_uninit(matcher);
675+free_matcher:
676+ mutex_unlock(&tbl->dmn->mutex);
677+ kfree(matcher);
678+dec_ref:
679+ refcount_dec(&tbl->refcount);
680+ return NULL;
681+}
682+
683+static int dr_matcher_disconnect(struct mlx5dr_domain *dmn,
684+ struct mlx5dr_table_rx_tx *nic_tbl,
685+ struct mlx5dr_matcher_rx_tx *next_nic_matcher,
686+ struct mlx5dr_matcher_rx_tx *prev_nic_matcher)
687+{
688+ struct mlx5dr_domain_rx_tx *nic_dmn = nic_tbl->nic_dmn;
689+ struct mlx5dr_htbl_connect_info info;
690+ struct mlx5dr_ste_htbl *prev_anchor;
691+
692+ if (prev_nic_matcher)
693+ prev_anchor = prev_nic_matcher->e_anchor;
694+ else
695+ prev_anchor = nic_tbl->s_anchor;
696+
697+ /* Connect previous anchor hash table to next matcher or to the default address */
698+ if (next_nic_matcher) {
699+ info.type = CONNECT_HIT;
700+ info.hit_next_htbl = next_nic_matcher->s_htbl;
701+ next_nic_matcher->s_htbl->pointing_ste = prev_anchor->ste_arr;
702+ prev_anchor->ste_arr[0].next_htbl = next_nic_matcher->s_htbl;
703+ } else {
704+ info.type = CONNECT_MISS;
705+ info.miss_icm_addr = nic_tbl->default_icm_addr;
706+ prev_anchor->ste_arr[0].next_htbl = NULL;
707+ }
708+
709+ return mlx5dr_ste_htbl_init_and_postsend(dmn, nic_dmn, prev_anchor,
710+ &info, true);
711+}
712+
713+static int dr_matcher_remove_from_tbl(struct mlx5dr_matcher *matcher)
714+{
715+ struct mlx5dr_matcher *prev_matcher, *next_matcher;
716+ struct mlx5dr_table *tbl = matcher->tbl;
717+ struct mlx5dr_domain *dmn = tbl->dmn;
718+ int ret = 0;
719+
720+ if (list_is_last(&matcher->matcher_list, &tbl->matcher_list))
721+ next_matcher = NULL;
722+ else
723+ next_matcher = list_next_entry(matcher, matcher_list);
724+
725+ if (matcher->matcher_list.prev == &tbl->matcher_list)
726+ prev_matcher = NULL;
727+ else
728+ prev_matcher = list_prev_entry(matcher, matcher_list);
729+
730+ if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB ||
731+ dmn->type == MLX5DR_DOMAIN_TYPE_NIC_RX) {
732+ ret = dr_matcher_disconnect(dmn, &tbl->rx,
733+ next_matcher ? &next_matcher->rx : NULL,
734+ prev_matcher ? &prev_matcher->rx : NULL);
735+ if (ret)
736+ return ret;
737+ }
738+
739+ if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB ||
740+ dmn->type == MLX5DR_DOMAIN_TYPE_NIC_TX) {
741+ ret = dr_matcher_disconnect(dmn, &tbl->tx,
742+ next_matcher ? &next_matcher->tx : NULL,
743+ prev_matcher ? &prev_matcher->tx : NULL);
744+ if (ret)
745+ return ret;
746+ }
747+
748+ list_del(&matcher->matcher_list);
749+
750+ return 0;
751+}
752+
753+int mlx5dr_matcher_destroy(struct mlx5dr_matcher *matcher)
754+{
755+ struct mlx5dr_table *tbl = matcher->tbl;
756+
757+ if (refcount_read(&matcher->refcount) > 1)
758+ return -EBUSY;
759+
760+ mutex_lock(&tbl->dmn->mutex);
761+
762+ dr_matcher_remove_from_tbl(matcher);
763+ dr_matcher_uninit(matcher);
764+ refcount_dec(&matcher->tbl->refcount);
765+
766+ mutex_unlock(&tbl->dmn->mutex);
767+ kfree(matcher);
768+
769+ return 0;
770+}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_rule.c+1243−0
@@ -0,0 +1,1243 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+/* Copyright (c) 2019 Mellanox Technologies. */
3+
4+#include "dr_types.h"
5+
6+#define DR_RULE_MAX_STE_CHAIN (DR_RULE_MAX_STES + DR_ACTION_MAX_STES)
7+
8+struct mlx5dr_rule_action_member {
9+ struct mlx5dr_action *action;
10+ struct list_head list;
11+};
12+
13+static int dr_rule_append_to_miss_list(struct mlx5dr_ste *new_last_ste,
14+ struct list_head *miss_list,
15+ struct list_head *send_list)
16+{
17+ struct mlx5dr_ste_send_info *ste_info_last;
18+ struct mlx5dr_ste *last_ste;
19+
20+ /* The new entry will be inserted after the last */
21+ last_ste = list_entry(miss_list->prev, struct mlx5dr_ste, miss_list_node);
22+ WARN_ON(!last_ste);
23+
24+ ste_info_last = kzalloc(sizeof(*ste_info_last), GFP_KERNEL);
25+ if (!ste_info_last)
26+ return -ENOMEM;
27+
28+ mlx5dr_ste_set_miss_addr(last_ste->hw_ste,
29+ mlx5dr_ste_get_icm_addr(new_last_ste));
30+ list_add_tail(&new_last_ste->miss_list_node, miss_list);
31+
32+ mlx5dr_send_fill_and_append_ste_send_info(last_ste, DR_STE_SIZE_REDUCED,
33+ 0, last_ste->hw_ste,
34+ ste_info_last, send_list, true);
35+
36+ return 0;
37+}
38+
39+static struct mlx5dr_ste *
40+dr_rule_create_collision_htbl(struct mlx5dr_matcher *matcher,
41+ struct mlx5dr_matcher_rx_tx *nic_matcher,
42+ u8 *hw_ste)
43+{
44+ struct mlx5dr_domain *dmn = matcher->tbl->dmn;
45+ struct mlx5dr_ste_htbl *new_htbl;
46+ struct mlx5dr_ste *ste;
47+
48+ /* Create new table for miss entry */
49+ new_htbl = mlx5dr_ste_htbl_alloc(dmn->ste_icm_pool,
50+ DR_CHUNK_SIZE_1,
51+ MLX5DR_STE_LU_TYPE_DONT_CARE,
52+ 0);
53+ if (!new_htbl) {
54+ mlx5dr_dbg(dmn, "Failed allocating collision table\n");
55+ return NULL;
56+ }
57+
58+ /* One and only entry, never grows */
59+ ste = new_htbl->ste_arr;
60+ mlx5dr_ste_set_miss_addr(hw_ste, nic_matcher->e_anchor->chunk->icm_addr);
61+ mlx5dr_htbl_get(new_htbl);
62+
63+ return ste;
64+}
65+
66+static struct mlx5dr_ste *
67+dr_rule_create_collision_entry(struct mlx5dr_matcher *matcher,
68+ struct mlx5dr_matcher_rx_tx *nic_matcher,
69+ u8 *hw_ste,
70+ struct mlx5dr_ste *orig_ste)
71+{
72+ struct mlx5dr_ste *ste;
73+
74+ ste = dr_rule_create_collision_htbl(matcher, nic_matcher, hw_ste);
75+ if (!ste) {
76+ mlx5dr_dbg(matcher->tbl->dmn, "Failed creating collision entry\n");
77+ return NULL;
78+ }
79+
80+ ste->ste_chain_location = orig_ste->ste_chain_location;
81+
82+ /* In collision entry, all members share the same miss_list_head */
83+ ste->htbl->miss_list = mlx5dr_ste_get_miss_list(orig_ste);
84+
85+ /* Next table */
86+ if (mlx5dr_ste_create_next_htbl(matcher, nic_matcher, ste, hw_ste,
87+ DR_CHUNK_SIZE_1)) {
88+ mlx5dr_dbg(matcher->tbl->dmn, "Failed allocating table\n");
89+ goto free_tbl;
90+ }
91+
92+ return ste;
93+
94+free_tbl:
95+ mlx5dr_ste_free(ste, matcher, nic_matcher);
96+ return NULL;
97+}
98+
99+static int
100+dr_rule_handle_one_ste_in_update_list(struct mlx5dr_ste_send_info *ste_info,
101+ struct mlx5dr_domain *dmn)
102+{
103+ int ret;
104+
105+ list_del(&ste_info->send_list);
106+ ret = mlx5dr_send_postsend_ste(dmn, ste_info->ste, ste_info->data,
107+ ste_info->size, ste_info->offset);
108+ if (ret)
109+ goto out;
110+ /* Copy data to ste, only reduced size, the last 16B (mask)
111+ * is already written to the hw.
112+ */
113+ memcpy(ste_info->ste->hw_ste, ste_info->data, DR_STE_SIZE_REDUCED);
114+
115+out:
116+ kfree(ste_info);
117+ return ret;
118+}
119+
120+static int dr_rule_send_update_list(struct list_head *send_ste_list,
121+ struct mlx5dr_domain *dmn,
122+ bool is_reverse)
123+{
124+ struct mlx5dr_ste_send_info *ste_info, *tmp_ste_info;
125+ int ret;
126+
127+ if (is_reverse) {
128+ list_for_each_entry_safe_reverse(ste_info, tmp_ste_info,
129+ send_ste_list, send_list) {
130+ ret = dr_rule_handle_one_ste_in_update_list(ste_info,
131+ dmn);
132+ if (ret)
133+ return ret;
134+ }
135+ } else {
136+ list_for_each_entry_safe(ste_info, tmp_ste_info,
137+ send_ste_list, send_list) {
138+ ret = dr_rule_handle_one_ste_in_update_list(ste_info,
139+ dmn);
140+ if (ret)
141+ return ret;
142+ }
143+ }
144+
145+ return 0;
146+}
147+
148+static struct mlx5dr_ste *
149+dr_rule_find_ste_in_miss_list(struct list_head *miss_list, u8 *hw_ste)
150+{
151+ struct mlx5dr_ste *ste;
152+
153+ if (list_empty(miss_list))
154+ return NULL;
155+
156+ /* Check if hw_ste is present in the list */
157+ list_for_each_entry(ste, miss_list, miss_list_node) {
158+ if (mlx5dr_ste_equal_tag(ste->hw_ste, hw_ste))
159+ return ste;
160+ }
161+
162+ return NULL;
163+}
164+
165+static struct mlx5dr_ste *
166+dr_rule_rehash_handle_collision(struct mlx5dr_matcher *matcher,
167+ struct mlx5dr_matcher_rx_tx *nic_matcher,
168+ struct list_head *update_list,
169+ struct mlx5dr_ste *col_ste,
170+ u8 *hw_ste)
171+{
172+ struct mlx5dr_ste *new_ste;
173+ int ret;
174+
175+ new_ste = dr_rule_create_collision_htbl(matcher, nic_matcher, hw_ste);
176+ if (!new_ste)
177+ return NULL;
178+
179+ /* In collision entry, all members share the same miss_list_head */
180+ new_ste->htbl->miss_list = mlx5dr_ste_get_miss_list(col_ste);
181+
182+ /* Update the previous from the list */
183+ ret = dr_rule_append_to_miss_list(new_ste,
184+ mlx5dr_ste_get_miss_list(col_ste),
185+ update_list);
186+ if (ret) {
187+ mlx5dr_dbg(matcher->tbl->dmn, "Failed update dup entry\n");
188+ goto err_exit;
189+ }
190+
191+ return new_ste;
192+
193+err_exit:
194+ mlx5dr_ste_free(new_ste, matcher, nic_matcher);
195+ return NULL;
196+}
197+
198+static void dr_rule_rehash_copy_ste_ctrl(struct mlx5dr_matcher *matcher,
199+ struct mlx5dr_matcher_rx_tx *nic_matcher,
200+ struct mlx5dr_ste *cur_ste,
201+ struct mlx5dr_ste *new_ste)
202+{
203+ new_ste->next_htbl = cur_ste->next_htbl;
204+ new_ste->ste_chain_location = cur_ste->ste_chain_location;
205+
206+ if (!mlx5dr_ste_is_last_in_rule(nic_matcher, new_ste->ste_chain_location))
207+ new_ste->next_htbl->pointing_ste = new_ste;
208+
209+ /* We need to copy the refcount since this ste
210+ * may have been traversed several times
211+ */
212+ refcount_set(&new_ste->refcount, refcount_read(&cur_ste->refcount));
213+
214+ /* Link old STEs rule_mem list to the new ste */
215+ mlx5dr_rule_update_rule_member(cur_ste, new_ste);
216+ INIT_LIST_HEAD(&new_ste->rule_list);
217+ list_splice_tail_init(&cur_ste->rule_list, &new_ste->rule_list);
218+}
219+
220+static struct mlx5dr_ste *
221+dr_rule_rehash_copy_ste(struct mlx5dr_matcher *matcher,
222+ struct mlx5dr_matcher_rx_tx *nic_matcher,
223+ struct mlx5dr_ste *cur_ste,
224+ struct mlx5dr_ste_htbl *new_htbl,
225+ struct list_head *update_list)
226+{
227+ struct mlx5dr_ste_send_info *ste_info;
228+ bool use_update_list = false;
229+ u8 hw_ste[DR_STE_SIZE] = {};
230+ struct mlx5dr_ste *new_ste;
231+ int new_idx;
232+ u8 sb_idx;
233+
234+ /* Copy STE mask from the matcher */
235+ sb_idx = cur_ste->ste_chain_location - 1;
236+ mlx5dr_ste_set_bit_mask(hw_ste, nic_matcher->ste_builder[sb_idx].bit_mask);
237+
238+ /* Copy STE control and tag */
239+ memcpy(hw_ste, cur_ste->hw_ste, DR_STE_SIZE_REDUCED);
240+ mlx5dr_ste_set_miss_addr(hw_ste, nic_matcher->e_anchor->chunk->icm_addr);
241+
242+ new_idx = mlx5dr_ste_calc_hash_index(hw_ste, new_htbl);
243+ new_ste = &new_htbl->ste_arr[new_idx];
244+
245+ if (mlx5dr_ste_not_used_ste(new_ste)) {
246+ mlx5dr_htbl_get(new_htbl);
247+ list_add_tail(&new_ste->miss_list_node,
248+ mlx5dr_ste_get_miss_list(new_ste));
249+ } else {
250+ new_ste = dr_rule_rehash_handle_collision(matcher,
251+ nic_matcher,
252+ update_list,
253+ new_ste,
254+ hw_ste);
255+ if (!new_ste) {
256+ mlx5dr_dbg(matcher->tbl->dmn, "Failed adding collision entry, index: %d\n",
257+ new_idx);
258+ return NULL;
259+ }
260+ new_htbl->ctrl.num_of_collisions++;
261+ use_update_list = true;
262+ }
263+
264+ memcpy(new_ste->hw_ste, hw_ste, DR_STE_SIZE_REDUCED);
265+
266+ new_htbl->ctrl.num_of_valid_entries++;
267+
268+ if (use_update_list) {
269+ ste_info = kzalloc(sizeof(*ste_info), GFP_KERNEL);
270+ if (!ste_info)
271+ goto err_exit;
272+
273+ mlx5dr_send_fill_and_append_ste_send_info(new_ste, DR_STE_SIZE, 0,
274+ hw_ste, ste_info,
275+ update_list, true);
276+ }
277+
278+ dr_rule_rehash_copy_ste_ctrl(matcher, nic_matcher, cur_ste, new_ste);
279+
280+ return new_ste;
281+
282+err_exit:
283+ mlx5dr_ste_free(new_ste, matcher, nic_matcher);
284+ return NULL;
285+}
286+
287+static int dr_rule_rehash_copy_miss_list(struct mlx5dr_matcher *matcher,
288+ struct mlx5dr_matcher_rx_tx *nic_matcher,
289+ struct list_head *cur_miss_list,
290+ struct mlx5dr_ste_htbl *new_htbl,
291+ struct list_head *update_list)
292+{
293+ struct mlx5dr_ste *tmp_ste, *cur_ste, *new_ste;
294+
295+ if (list_empty(cur_miss_list))
296+ return 0;
297+
298+ list_for_each_entry_safe(cur_ste, tmp_ste, cur_miss_list, miss_list_node) {
299+ new_ste = dr_rule_rehash_copy_ste(matcher,
300+ nic_matcher,
301+ cur_ste,
302+ new_htbl,
303+ update_list);
304+ if (!new_ste)
305+ goto err_insert;
306+
307+ list_del(&cur_ste->miss_list_node);
308+ mlx5dr_htbl_put(cur_ste->htbl);
309+ }
310+ return 0;
311+
312+err_insert:
313+ mlx5dr_err(matcher->tbl->dmn, "Fatal error during resize\n");
314+ WARN_ON(true);
315+ return -EINVAL;
316+}
317+
318+static int dr_rule_rehash_copy_htbl(struct mlx5dr_matcher *matcher,
319+ struct mlx5dr_matcher_rx_tx *nic_matcher,
320+ struct mlx5dr_ste_htbl *cur_htbl,
321+ struct mlx5dr_ste_htbl *new_htbl,
322+ struct list_head *update_list)
323+{
324+ struct mlx5dr_ste *cur_ste;
325+ int cur_entries;
326+ int err = 0;
327+ int i;
328+
329+ cur_entries = mlx5dr_icm_pool_chunk_size_to_entries(cur_htbl->chunk_size);
330+
331+ if (cur_entries < 1) {
332+ mlx5dr_dbg(matcher->tbl->dmn, "Invalid number of entries\n");
333+ return -EINVAL;
334+ }
335+
336+ for (i = 0; i < cur_entries; i++) {
337+ cur_ste = &cur_htbl->ste_arr[i];
338+ if (mlx5dr_ste_not_used_ste(cur_ste)) /* Empty, nothing to copy */
339+ continue;
340+
341+ err = dr_rule_rehash_copy_miss_list(matcher,
342+ nic_matcher,
343+ mlx5dr_ste_get_miss_list(cur_ste),
344+ new_htbl,
345+ update_list);
346+ if (err)
347+ goto clean_copy;
348+ }
349+
350+clean_copy:
351+ return err;
352+}
353+
354+static struct mlx5dr_ste_htbl *
355+dr_rule_rehash_htbl(struct mlx5dr_rule *rule,
356+ struct mlx5dr_rule_rx_tx *nic_rule,
357+ struct mlx5dr_ste_htbl *cur_htbl,
358+ u8 ste_location,
359+ struct list_head *update_list,
360+ enum mlx5dr_icm_chunk_size new_size)
361+{
362+ struct mlx5dr_ste_send_info *del_ste_info, *tmp_ste_info;
363+ struct mlx5dr_matcher *matcher = rule->matcher;
364+ struct mlx5dr_domain *dmn = matcher->tbl->dmn;
365+ struct mlx5dr_matcher_rx_tx *nic_matcher;
366+ struct mlx5dr_ste_send_info *ste_info;
367+ struct mlx5dr_htbl_connect_info info;
368+ struct mlx5dr_domain_rx_tx *nic_dmn;
369+ u8 formatted_ste[DR_STE_SIZE] = {};
370+ LIST_HEAD(rehash_table_send_list);
371+ struct mlx5dr_ste *ste_to_update;
372+ struct mlx5dr_ste_htbl *new_htbl;
373+ int err;
374+
375+ nic_matcher = nic_rule->nic_matcher;
376+ nic_dmn = nic_matcher->nic_tbl->nic_dmn;
377+
378+ ste_info = kzalloc(sizeof(*ste_info), GFP_KERNEL);
379+ if (!ste_info)
380+ return NULL;
381+
382+ new_htbl = mlx5dr_ste_htbl_alloc(dmn->ste_icm_pool,
383+ new_size,
384+ cur_htbl->lu_type,
385+ cur_htbl->byte_mask);
386+ if (!new_htbl) {
387+ mlx5dr_err(dmn, "Failed to allocate new hash table\n");
388+ goto free_ste_info;
389+ }
390+
391+ /* Write new table to HW */
392+ info.type = CONNECT_MISS;
393+ info.miss_icm_addr = nic_matcher->e_anchor->chunk->icm_addr;
394+ mlx5dr_ste_set_formatted_ste(dmn->info.caps.gvmi,
395+ nic_dmn,
396+ new_htbl,
397+ formatted_ste,
398+ &info);
399+
400+ new_htbl->pointing_ste = cur_htbl->pointing_ste;
401+ new_htbl->pointing_ste->next_htbl = new_htbl;
402+ err = dr_rule_rehash_copy_htbl(matcher,
403+ nic_matcher,
404+ cur_htbl,
405+ new_htbl,
406+ &rehash_table_send_list);
407+ if (err)
408+ goto free_new_htbl;
409+
410+ if (mlx5dr_send_postsend_htbl(dmn, new_htbl, formatted_ste,
411+ nic_matcher->ste_builder[ste_location - 1].bit_mask)) {
412+ mlx5dr_err(dmn, "Failed writing table to HW\n");
413+ goto free_new_htbl;
414+ }
415+
416+ /* Writing to the hw is done in regular order of rehash_table_send_list,
417+ * in order to have the origin data written before the miss address of
418+ * collision entries, if exists.
419+ */
420+ if (dr_rule_send_update_list(&rehash_table_send_list, dmn, false)) {
421+ mlx5dr_err(dmn, "Failed updating table to HW\n");
422+ goto free_ste_list;
423+ }
424+
425+ /* Connect previous hash table to current */
426+ if (ste_location == 1) {
427+ /* The previous table is an anchor, anchors size is always one STE */
428+ struct mlx5dr_ste_htbl *prev_htbl = cur_htbl->pointing_ste->htbl;
429+
430+ /* On matcher s_anchor we keep an extra refcount */
431+ mlx5dr_htbl_get(new_htbl);
432+ mlx5dr_htbl_put(cur_htbl);
433+
434+ nic_matcher->s_htbl = new_htbl;
435+
436+ /* It is safe to operate dr_ste_set_hit_addr on the hw_ste here
437+ * (48B len) which works only on first 32B
438+ */
439+ mlx5dr_ste_set_hit_addr(prev_htbl->ste_arr[0].hw_ste,
440+ new_htbl->chunk->icm_addr,
441+ new_htbl->chunk->num_of_entries);
442+
443+ ste_to_update = &prev_htbl->ste_arr[0];
444+ } else {
445+ mlx5dr_ste_set_hit_addr_by_next_htbl(cur_htbl->pointing_ste->hw_ste,
446+ new_htbl);
447+ ste_to_update = cur_htbl->pointing_ste;
448+ }
449+
450+ mlx5dr_send_fill_and_append_ste_send_info(ste_to_update, DR_STE_SIZE_REDUCED,
451+ 0, ste_to_update->hw_ste, ste_info,
452+ update_list, false);
453+
454+ return new_htbl;
455+
456+free_ste_list:
457+ /* Clean all ste_info's from the new table */
458+ list_for_each_entry_safe(del_ste_info, tmp_ste_info,
459+ &rehash_table_send_list, send_list) {
460+ list_del(&del_ste_info->send_list);
461+ kfree(del_ste_info);
462+ }
463+
464+free_new_htbl:
465+ mlx5dr_ste_htbl_free(new_htbl);
466+free_ste_info:
467+ kfree(ste_info);
468+ mlx5dr_info(dmn, "Failed creating rehash table\n");
469+ return NULL;
470+}
471+
472+static struct mlx5dr_ste_htbl *dr_rule_rehash(struct mlx5dr_rule *rule,
473+ struct mlx5dr_rule_rx_tx *nic_rule,
474+ struct mlx5dr_ste_htbl *cur_htbl,
475+ u8 ste_location,
476+ struct list_head *update_list)
477+{
478+ struct mlx5dr_domain *dmn = rule->matcher->tbl->dmn;
479+ enum mlx5dr_icm_chunk_size new_size;
480+
481+ new_size = mlx5dr_icm_next_higher_chunk(cur_htbl->chunk_size);
482+ new_size = min_t(u32, new_size, dmn->info.max_log_sw_icm_sz);
483+
484+ if (new_size == cur_htbl->chunk_size)
485+ return NULL; /* Skip rehash, we already at the max size */
486+
487+ return dr_rule_rehash_htbl(rule, nic_rule, cur_htbl, ste_location,
488+ update_list, new_size);
489+}
490+
491+static struct mlx5dr_ste *
492+dr_rule_handle_collision(struct mlx5dr_matcher *matcher,
493+ struct mlx5dr_matcher_rx_tx *nic_matcher,
494+ struct mlx5dr_ste *ste,
495+ u8 *hw_ste,
496+ struct list_head *miss_list,
497+ struct list_head *send_list)
498+{
499+ struct mlx5dr_ste_send_info *ste_info;
500+ struct mlx5dr_ste *new_ste;
501+
502+ ste_info = kzalloc(sizeof(*ste_info), GFP_KERNEL);
503+ if (!ste_info)
504+ return NULL;
505+
506+ new_ste = dr_rule_create_collision_entry(matcher, nic_matcher, hw_ste, ste);
507+ if (!new_ste)
508+ goto free_send_info;
509+
510+ if (dr_rule_append_to_miss_list(new_ste, miss_list, send_list)) {
511+ mlx5dr_dbg(matcher->tbl->dmn, "Failed to update prev miss_list\n");
512+ goto err_exit;
513+ }
514+
515+ mlx5dr_send_fill_and_append_ste_send_info(new_ste, DR_STE_SIZE, 0, hw_ste,
516+ ste_info, send_list, false);
517+
518+ ste->htbl->ctrl.num_of_collisions++;
519+ ste->htbl->ctrl.num_of_valid_entries++;
520+
521+ return new_ste;
522+
523+err_exit:
524+ mlx5dr_ste_free(new_ste, matcher, nic_matcher);
525+free_send_info:
526+ kfree(ste_info);
527+ return NULL;
528+}
529+
530+static void dr_rule_remove_action_members(struct mlx5dr_rule *rule)
531+{
532+ struct mlx5dr_rule_action_member *action_mem;
533+ struct mlx5dr_rule_action_member *tmp;
534+
535+ list_for_each_entry_safe(action_mem, tmp, &rule->rule_actions_list, list) {
536+ list_del(&action_mem->list);
537+ refcount_dec(&action_mem->action->refcount);
538+ kvfree(action_mem);
539+ }
540+}
541+
542+static int dr_rule_add_action_members(struct mlx5dr_rule *rule,
543+ size_t num_actions,
544+ struct mlx5dr_action *actions[])
545+{
546+ struct mlx5dr_rule_action_member *action_mem;
547+ int i;
548+
549+ for (i = 0; i < num_actions; i++) {
550+ action_mem = kvzalloc(sizeof(*action_mem), GFP_KERNEL);
551+ if (!action_mem)
552+ goto free_action_members;
553+
554+ action_mem->action = actions[i];
555+ INIT_LIST_HEAD(&action_mem->list);
556+ list_add_tail(&action_mem->list, &rule->rule_actions_list);
557+ refcount_inc(&action_mem->action->refcount);
558+ }
559+
560+ return 0;
561+
562+free_action_members:
563+ dr_rule_remove_action_members(rule);
564+ return -ENOMEM;
565+}
566+
567+/* While the pointer of ste is no longer valid, like while moving ste to be
568+ * the first in the miss_list, and to be in the origin table,
569+ * all rule-members that are attached to this ste should update their ste member
570+ * to the new pointer
571+ */
572+void mlx5dr_rule_update_rule_member(struct mlx5dr_ste *ste,
573+ struct mlx5dr_ste *new_ste)
574+{
575+ struct mlx5dr_rule_member *rule_mem;
576+
577+ if (!list_empty(&ste->rule_list))
578+ list_for_each_entry(rule_mem, &ste->rule_list, use_ste_list)
579+ rule_mem->ste = new_ste;
580+}
581+
582+static void dr_rule_clean_rule_members(struct mlx5dr_rule *rule,
583+ struct mlx5dr_rule_rx_tx *nic_rule)
584+{
585+ struct mlx5dr_rule_member *rule_mem;
586+ struct mlx5dr_rule_member *tmp_mem;
587+
588+ if (list_empty(&nic_rule->rule_members_list))
589+ return;
590+ list_for_each_entry_safe(rule_mem, tmp_mem, &nic_rule->rule_members_list, list) {
591+ list_del(&rule_mem->list);
592+ list_del(&rule_mem->use_ste_list);
593+ mlx5dr_ste_put(rule_mem->ste, rule->matcher, nic_rule->nic_matcher);
594+ kvfree(rule_mem);
595+ }
596+}
597+
598+static bool dr_rule_need_enlarge_hash(struct mlx5dr_ste_htbl *htbl,
599+ struct mlx5dr_domain *dmn,
600+ struct mlx5dr_domain_rx_tx *nic_dmn)
601+{
602+ struct mlx5dr_ste_htbl_ctrl *ctrl = &htbl->ctrl;
603+
604+ if (dmn->info.max_log_sw_icm_sz <= htbl->chunk_size)
605+ return false;
606+
607+ if (!ctrl->may_grow)
608+ return false;
609+
610+ if (ctrl->num_of_collisions >= ctrl->increase_threshold &&
611+ (ctrl->num_of_valid_entries - ctrl->num_of_collisions) >= ctrl->increase_threshold)
612+ return true;
613+
614+ return false;
615+}
616+
617+static int dr_rule_add_member(struct mlx5dr_rule_rx_tx *nic_rule,
618+ struct mlx5dr_ste *ste)
619+{
620+ struct mlx5dr_rule_member *rule_mem;
621+
622+ rule_mem = kvzalloc(sizeof(*rule_mem), GFP_KERNEL);
623+ if (!rule_mem)
624+ return -ENOMEM;
625+
626+ rule_mem->ste = ste;
627+ list_add_tail(&rule_mem->list, &nic_rule->rule_members_list);
628+
629+ list_add_tail(&rule_mem->use_ste_list, &ste->rule_list);
630+
631+ return 0;
632+}
633+
634+static int dr_rule_handle_action_stes(struct mlx5dr_rule *rule,
635+ struct mlx5dr_rule_rx_tx *nic_rule,
636+ struct list_head *send_ste_list,
637+ struct mlx5dr_ste *last_ste,
638+ u8 *hw_ste_arr,
639+ u32 new_hw_ste_arr_sz)
640+{
641+ struct mlx5dr_matcher_rx_tx *nic_matcher = nic_rule->nic_matcher;
642+ struct mlx5dr_ste_send_info *ste_info_arr[DR_ACTION_MAX_STES];
643+ u8 num_of_builders = nic_matcher->num_of_builders;
644+ struct mlx5dr_matcher *matcher = rule->matcher;
645+ u8 *curr_hw_ste, *prev_hw_ste;
646+ struct mlx5dr_ste *action_ste;
647+ int i, k, ret;
648+
649+ /* Two cases:
650+ * 1. num_of_builders is equal to new_hw_ste_arr_sz, the action in the ste
651+ * 2. num_of_builders is less then new_hw_ste_arr_sz, new ste was added
652+ * to support the action.
653+ */
654+ if (num_of_builders == new_hw_ste_arr_sz)
655+ return 0;
656+
657+ for (i = num_of_builders, k = 0; i < new_hw_ste_arr_sz; i++, k++) {
658+ curr_hw_ste = hw_ste_arr + i * DR_STE_SIZE;
659+ prev_hw_ste = (i == 0) ? curr_hw_ste : hw_ste_arr + ((i - 1) * DR_STE_SIZE);
660+ action_ste = dr_rule_create_collision_htbl(matcher,
661+ nic_matcher,
662+ curr_hw_ste);
663+ if (!action_ste)
664+ return -ENOMEM;
665+
666+ mlx5dr_ste_get(action_ste);
667+
668+ /* While free ste we go over the miss list, so add this ste to the list */
669+ list_add_tail(&action_ste->miss_list_node,
670+ mlx5dr_ste_get_miss_list(action_ste));
671+
672+ ste_info_arr[k] = kzalloc(sizeof(*ste_info_arr[k]),
673+ GFP_KERNEL);
674+ if (!ste_info_arr[k])
675+ goto err_exit;
676+
677+ /* Point current ste to the new action */
678+ mlx5dr_ste_set_hit_addr_by_next_htbl(prev_hw_ste, action_ste->htbl);
679+ ret = dr_rule_add_member(nic_rule, action_ste);
680+ if (ret) {
681+ mlx5dr_dbg(matcher->tbl->dmn, "Failed adding rule member\n");
682+ goto free_ste_info;
683+ }
684+ mlx5dr_send_fill_and_append_ste_send_info(action_ste, DR_STE_SIZE, 0,
685+ curr_hw_ste,
686+ ste_info_arr[k],
687+ send_ste_list, false);
688+ }
689+
690+ return 0;
691+
692+free_ste_info:
693+ kfree(ste_info_arr[k]);
694+err_exit:
695+ mlx5dr_ste_put(action_ste, matcher, nic_matcher);
696+ return -ENOMEM;
697+}
698+
699+static int dr_rule_handle_empty_entry(struct mlx5dr_matcher *matcher,
700+ struct mlx5dr_matcher_rx_tx *nic_matcher,
701+ struct mlx5dr_ste_htbl *cur_htbl,
702+ struct mlx5dr_ste *ste,
703+ u8 ste_location,
704+ u8 *hw_ste,
705+ struct list_head *miss_list,
706+ struct list_head *send_list)
707+{
708+ struct mlx5dr_ste_send_info *ste_info;
709+
710+ /* Take ref on table, only on first time this ste is used */
711+ mlx5dr_htbl_get(cur_htbl);
712+
713+ /* new entry -> new branch */
714+ list_add_tail(&ste->miss_list_node, miss_list);
715+
716+ mlx5dr_ste_set_miss_addr(hw_ste, nic_matcher->e_anchor->chunk->icm_addr);
717+
718+ ste->ste_chain_location = ste_location;
719+
720+ ste_info = kzalloc(sizeof(*ste_info), GFP_KERNEL);
721+ if (!ste_info)
722+ goto clean_ste_setting;
723+
724+ if (mlx5dr_ste_create_next_htbl(matcher,
725+ nic_matcher,
726+ ste,
727+ hw_ste,
728+ DR_CHUNK_SIZE_1)) {
729+ mlx5dr_dbg(matcher->tbl->dmn, "Failed allocating table\n");
730+ goto clean_ste_info;
731+ }
732+
733+ cur_htbl->ctrl.num_of_valid_entries++;
734+
735+ mlx5dr_send_fill_and_append_ste_send_info(ste, DR_STE_SIZE, 0, hw_ste,
736+ ste_info, send_list, false);
737+
738+ return 0;
739+
740+clean_ste_info:
741+ kfree(ste_info);
742+clean_ste_setting:
743+ list_del_init(&ste->miss_list_node);
744+ mlx5dr_htbl_put(cur_htbl);
745+
746+ return -ENOMEM;
747+}
748+
749+static struct mlx5dr_ste *
750+dr_rule_handle_ste_branch(struct mlx5dr_rule *rule,
751+ struct mlx5dr_rule_rx_tx *nic_rule,
752+ struct list_head *send_ste_list,
753+ struct mlx5dr_ste_htbl *cur_htbl,
754+ u8 *hw_ste,
755+ u8 ste_location,
756+ struct mlx5dr_ste_htbl **put_htbl)
757+{
758+ struct mlx5dr_matcher *matcher = rule->matcher;
759+ struct mlx5dr_domain *dmn = matcher->tbl->dmn;
760+ struct mlx5dr_matcher_rx_tx *nic_matcher;
761+ struct mlx5dr_domain_rx_tx *nic_dmn;
762+ struct mlx5dr_ste_htbl *new_htbl;
763+ struct mlx5dr_ste *matched_ste;
764+ struct list_head *miss_list;
765+ bool skip_rehash = false;
766+ struct mlx5dr_ste *ste;
767+ int index;
768+
769+ nic_matcher = nic_rule->nic_matcher;
770+ nic_dmn = nic_matcher->nic_tbl->nic_dmn;
771+
772+again:
773+ index = mlx5dr_ste_calc_hash_index(hw_ste, cur_htbl);
774+ miss_list = &cur_htbl->chunk->miss_list[index];
775+ ste = &cur_htbl->ste_arr[index];
776+
777+ if (mlx5dr_ste_not_used_ste(ste)) {
778+ if (dr_rule_handle_empty_entry(matcher, nic_matcher, cur_htbl,
779+ ste, ste_location,
780+ hw_ste, miss_list,
781+ send_ste_list))
782+ return NULL;
783+ } else {
784+ /* Hash table index in use, check if this ste is in the miss list */
785+ matched_ste = dr_rule_find_ste_in_miss_list(miss_list, hw_ste);
786+ if (matched_ste) {
787+ /* If it is last STE in the chain, and has the same tag
788+ * it means that all the previous stes are the same,
789+ * if so, this rule is duplicated.
790+ */
791+ if (mlx5dr_ste_is_last_in_rule(nic_matcher,
792+ matched_ste->ste_chain_location)) {
793+ mlx5dr_info(dmn, "Duplicate rule inserted, aborting!!\n");
794+ return NULL;
795+ }
796+ return matched_ste;
797+ }
798+
799+ if (!skip_rehash && dr_rule_need_enlarge_hash(cur_htbl, dmn, nic_dmn)) {
800+ /* Hash table index in use, try to resize of the hash */
801+ skip_rehash = true;
802+
803+ /* Hold the table till we update.
804+ * Release in dr_rule_create_rule()
805+ */
806+ *put_htbl = cur_htbl;
807+ mlx5dr_htbl_get(cur_htbl);
808+
809+ new_htbl = dr_rule_rehash(rule, nic_rule, cur_htbl,
810+ ste_location, send_ste_list);
811+ if (!new_htbl) {
812+ mlx5dr_htbl_put(cur_htbl);
813+ mlx5dr_info(dmn, "failed creating rehash table, htbl-log_size: %d\n",
814+ cur_htbl->chunk_size);
815+ } else {
816+ cur_htbl = new_htbl;
817+ }
818+ goto again;
819+ } else {
820+ /* Hash table index in use, add another collision (miss) */
821+ ste = dr_rule_handle_collision(matcher,
822+ nic_matcher,
823+ ste,
824+ hw_ste,
825+ miss_list,
826+ send_ste_list);
827+ if (!ste) {
828+ mlx5dr_dbg(dmn, "failed adding collision entry, index: %d\n",
829+ index);
830+ return NULL;
831+ }
832+ }
833+ }
834+ return ste;
835+}
836+
837+static bool dr_rule_cmp_value_to_mask(u8 *mask, u8 *value,
838+ u32 s_idx, u32 e_idx)
839+{
840+ u32 i;
841+
842+ for (i = s_idx; i < e_idx; i++) {
843+ if (value[i] & ~mask[i]) {
844+ pr_info("Rule parameters contains a value not specified by mask\n");
845+ return false;
846+ }
847+ }
848+ return true;
849+}
850+
851+static bool dr_rule_verify(struct mlx5dr_matcher *matcher,
852+ struct mlx5dr_match_parameters *value,
853+ struct mlx5dr_match_param *param)
854+{
855+ u8 match_criteria = matcher->match_criteria;
856+ size_t value_size = value->match_sz;
857+ u8 *mask_p = (u8 *)&matcher->mask;
858+ u8 *param_p = (u8 *)param;
859+ u32 s_idx, e_idx;
860+
861+ if (!value_size ||
862+ (value_size > sizeof(struct mlx5dr_match_param) ||
863+ (value_size % sizeof(u32)))) {
864+ mlx5dr_dbg(matcher->tbl->dmn, "Rule parameters length is incorrect\n");
865+ return false;
866+ }
867+
868+ mlx5dr_ste_copy_param(matcher->match_criteria, param, value);
869+
870+ if (match_criteria & DR_MATCHER_CRITERIA_OUTER) {
871+ s_idx = offsetof(struct mlx5dr_match_param, outer);
872+ e_idx = min(s_idx + sizeof(param->outer), value_size);
873+
874+ if (!dr_rule_cmp_value_to_mask(mask_p, param_p, s_idx, e_idx)) {
875+ mlx5dr_dbg(matcher->tbl->dmn, "Rule outer parameters contains a value not specified by mask\n");
876+ return false;
877+ }
878+ }
879+
880+ if (match_criteria & DR_MATCHER_CRITERIA_MISC) {
881+ s_idx = offsetof(struct mlx5dr_match_param, misc);
882+ e_idx = min(s_idx + sizeof(param->misc), value_size);
883+
884+ if (!dr_rule_cmp_value_to_mask(mask_p, param_p, s_idx, e_idx)) {
885+ mlx5dr_dbg(matcher->tbl->dmn, "Rule misc parameters contains a value not specified by mask\n");
886+ return false;
887+ }
888+ }
889+
890+ if (match_criteria & DR_MATCHER_CRITERIA_INNER) {
891+ s_idx = offsetof(struct mlx5dr_match_param, inner);
892+ e_idx = min(s_idx + sizeof(param->inner), value_size);
893+
894+ if (!dr_rule_cmp_value_to_mask(mask_p, param_p, s_idx, e_idx)) {
895+ mlx5dr_dbg(matcher->tbl->dmn, "Rule inner parameters contains a value not specified by mask\n");
896+ return false;
897+ }
898+ }
899+
900+ if (match_criteria & DR_MATCHER_CRITERIA_MISC2) {
901+ s_idx = offsetof(struct mlx5dr_match_param, misc2);
902+ e_idx = min(s_idx + sizeof(param->misc2), value_size);
903+
904+ if (!dr_rule_cmp_value_to_mask(mask_p, param_p, s_idx, e_idx)) {
905+ mlx5dr_dbg(matcher->tbl->dmn, "Rule misc2 parameters contains a value not specified by mask\n");
906+ return false;
907+ }
908+ }
909+
910+ if (match_criteria & DR_MATCHER_CRITERIA_MISC3) {
911+ s_idx = offsetof(struct mlx5dr_match_param, misc3);
912+ e_idx = min(s_idx + sizeof(param->misc3), value_size);
913+
914+ if (!dr_rule_cmp_value_to_mask(mask_p, param_p, s_idx, e_idx)) {
915+ mlx5dr_dbg(matcher->tbl->dmn, "Rule misc3 parameters contains a value not specified by mask\n");
916+ return false;
917+ }
918+ }
919+ return true;
920+}
921+
922+static int dr_rule_destroy_rule_nic(struct mlx5dr_rule *rule,
923+ struct mlx5dr_rule_rx_tx *nic_rule)
924+{
925+ dr_rule_clean_rule_members(rule, nic_rule);
926+ return 0;
927+}
928+
929+static int dr_rule_destroy_rule_fdb(struct mlx5dr_rule *rule)
930+{
931+ dr_rule_destroy_rule_nic(rule, &rule->rx);
932+ dr_rule_destroy_rule_nic(rule, &rule->tx);
933+ return 0;
934+}
935+
936+static int dr_rule_destroy_rule(struct mlx5dr_rule *rule)
937+{
938+ struct mlx5dr_domain *dmn = rule->matcher->tbl->dmn;
939+
940+ switch (dmn->type) {
941+ case MLX5DR_DOMAIN_TYPE_NIC_RX:
942+ dr_rule_destroy_rule_nic(rule, &rule->rx);
943+ break;
944+ case MLX5DR_DOMAIN_TYPE_NIC_TX:
945+ dr_rule_destroy_rule_nic(rule, &rule->tx);
946+ break;
947+ case MLX5DR_DOMAIN_TYPE_FDB:
948+ dr_rule_destroy_rule_fdb(rule);
949+ break;
950+ default:
951+ return -EINVAL;
952+ }
953+
954+ dr_rule_remove_action_members(rule);
955+ kfree(rule);
956+ return 0;
957+}
958+
959+static bool dr_rule_is_ipv6(struct mlx5dr_match_param *param)
960+{
961+ return (param->outer.ip_version == 6 ||
962+ param->inner.ip_version == 6 ||
963+ param->outer.ethertype == ETH_P_IPV6 ||
964+ param->inner.ethertype == ETH_P_IPV6);
965+}
966+
967+static bool dr_rule_skip(enum mlx5dr_domain_type domain,
968+ enum mlx5dr_ste_entry_type ste_type,
969+ struct mlx5dr_match_param *mask,
970+ struct mlx5dr_match_param *value)
971+{
972+ if (domain != MLX5DR_DOMAIN_TYPE_FDB)
973+ return false;
974+
975+ if (mask->misc.source_port) {
976+ if (ste_type == MLX5DR_STE_TYPE_RX)
977+ if (value->misc.source_port != WIRE_PORT)
978+ return true;
979+
980+ if (ste_type == MLX5DR_STE_TYPE_TX)
981+ if (value->misc.source_port == WIRE_PORT)
982+ return true;
983+ }
984+
985+ /* Metadata C can be used to describe the source vport */
986+ if (mask->misc2.metadata_reg_c_0) {
987+ if (ste_type == MLX5DR_STE_TYPE_RX)
988+ if ((value->misc2.metadata_reg_c_0 & WIRE_PORT) != WIRE_PORT)
989+ return true;
990+
991+ if (ste_type == MLX5DR_STE_TYPE_TX)
992+ if ((value->misc2.metadata_reg_c_0 & WIRE_PORT) == WIRE_PORT)
993+ return true;
994+ }
995+ return false;
996+}
997+
998+static int
999+dr_rule_create_rule_nic(struct mlx5dr_rule *rule,
1000+ struct mlx5dr_rule_rx_tx *nic_rule,
1001+ struct mlx5dr_match_param *param,
1002+ size_t num_actions,
1003+ struct mlx5dr_action *actions[])
1004+{
1005+ struct mlx5dr_ste_send_info *ste_info, *tmp_ste_info;
1006+ struct mlx5dr_matcher *matcher = rule->matcher;
1007+ struct mlx5dr_domain *dmn = matcher->tbl->dmn;
1008+ struct mlx5dr_matcher_rx_tx *nic_matcher;
1009+ struct mlx5dr_domain_rx_tx *nic_dmn;
1010+ struct mlx5dr_ste_htbl *htbl = NULL;
1011+ struct mlx5dr_ste_htbl *cur_htbl;
1012+ struct mlx5dr_ste *ste = NULL;
1013+ LIST_HEAD(send_ste_list);
1014+ u8 *hw_ste_arr = NULL;
1015+ u32 new_hw_ste_arr_sz;
1016+ int ret, i;
1017+
1018+ nic_matcher = nic_rule->nic_matcher;
1019+ nic_dmn = nic_matcher->nic_tbl->nic_dmn;
1020+
1021+ INIT_LIST_HEAD(&nic_rule->rule_members_list);
1022+
1023+ if (dr_rule_skip(dmn->type, nic_dmn->ste_type, &matcher->mask, param))
1024+ return 0;
1025+
1026+ ret = mlx5dr_matcher_select_builders(matcher,
1027+ nic_matcher,
1028+ dr_rule_is_ipv6(param));
1029+ if (ret)
1030+ goto out_err;
1031+
1032+ hw_ste_arr = kzalloc(DR_RULE_MAX_STE_CHAIN * DR_STE_SIZE, GFP_KERNEL);
1033+ if (!hw_ste_arr) {
1034+ ret = -ENOMEM;
1035+ goto out_err;
1036+ }
1037+
1038+ /* Set the tag values inside the ste array */
1039+ ret = mlx5dr_ste_build_ste_arr(matcher, nic_matcher, param, hw_ste_arr);
1040+ if (ret)
1041+ goto free_hw_ste;
1042+
1043+ /* Set the actions values/addresses inside the ste array */
1044+ ret = mlx5dr_actions_build_ste_arr(matcher, nic_matcher, actions,
1045+ num_actions, hw_ste_arr,
1046+ &new_hw_ste_arr_sz);
1047+ if (ret)
1048+ goto free_hw_ste;
1049+
1050+ cur_htbl = nic_matcher->s_htbl;
1051+
1052+ /* Go over the array of STEs, and build dr_ste accordingly.
1053+ * The loop is over only the builders which are equal or less to the
1054+ * number of stes, in case we have actions that lives in other stes.
1055+ */
1056+ for (i = 0; i < nic_matcher->num_of_builders; i++) {
1057+ /* Calculate CRC and keep new ste entry */
1058+ u8 *cur_hw_ste_ent = hw_ste_arr + (i * DR_STE_SIZE);
1059+
1060+ ste = dr_rule_handle_ste_branch(rule,
1061+ nic_rule,
1062+ &send_ste_list,
1063+ cur_htbl,
1064+ cur_hw_ste_ent,
1065+ i + 1,
1066+ &htbl);
1067+ if (!ste) {
1068+ mlx5dr_err(dmn, "Failed creating next branch\n");
1069+ ret = -ENOENT;
1070+ goto free_rule;
1071+ }
1072+
1073+ cur_htbl = ste->next_htbl;
1074+
1075+ /* Keep all STEs in the rule struct */
1076+ ret = dr_rule_add_member(nic_rule, ste);
1077+ if (ret) {
1078+ mlx5dr_dbg(dmn, "Failed adding rule member index %d\n", i);
1079+ goto free_ste;
1080+ }
1081+
1082+ mlx5dr_ste_get(ste);
1083+ }
1084+
1085+ /* Connect actions */
1086+ ret = dr_rule_handle_action_stes(rule, nic_rule, &send_ste_list,
1087+ ste, hw_ste_arr, new_hw_ste_arr_sz);
1088+ if (ret) {
1089+ mlx5dr_dbg(dmn, "Failed apply actions\n");
1090+ goto free_rule;
1091+ }
1092+ ret = dr_rule_send_update_list(&send_ste_list, dmn, true);
1093+ if (ret) {
1094+ mlx5dr_err(dmn, "Failed sending ste!\n");
1095+ goto free_rule;
1096+ }
1097+
1098+ if (htbl)
1099+ mlx5dr_htbl_put(htbl);
1100+
1101+ return 0;
1102+
1103+free_ste:
1104+ mlx5dr_ste_put(ste, matcher, nic_matcher);
1105+free_rule:
1106+ dr_rule_clean_rule_members(rule, nic_rule);
1107+ /* Clean all ste_info's */
1108+ list_for_each_entry_safe(ste_info, tmp_ste_info, &send_ste_list, send_list) {
1109+ list_del(&ste_info->send_list);
1110+ kfree(ste_info);
1111+ }
1112+free_hw_ste:
1113+ kfree(hw_ste_arr);
1114+out_err:
1115+ return ret;
1116+}
1117+
1118+static int
1119+dr_rule_create_rule_fdb(struct mlx5dr_rule *rule,
1120+ struct mlx5dr_match_param *param,
1121+ size_t num_actions,
1122+ struct mlx5dr_action *actions[])
1123+{
1124+ struct mlx5dr_match_param copy_param = {};
1125+ int ret;
1126+
1127+ /* Copy match_param since they will be consumed during the first
1128+ * nic_rule insertion.
1129+ */
1130+ memcpy(&copy_param, param, sizeof(struct mlx5dr_match_param));
1131+
1132+ ret = dr_rule_create_rule_nic(rule, &rule->rx, param,
1133+ num_actions, actions);
1134+ if (ret)
1135+ return ret;
1136+
1137+ ret = dr_rule_create_rule_nic(rule, &rule->tx, &copy_param,
1138+ num_actions, actions);
1139+ if (ret)
1140+ goto destroy_rule_nic_rx;
1141+
1142+ return 0;
1143+
1144+destroy_rule_nic_rx:
1145+ dr_rule_destroy_rule_nic(rule, &rule->rx);
1146+ return ret;
1147+}
1148+
1149+static struct mlx5dr_rule *
1150+dr_rule_create_rule(struct mlx5dr_matcher *matcher,
1151+ struct mlx5dr_match_parameters *value,
1152+ size_t num_actions,
1153+ struct mlx5dr_action *actions[])
1154+{
1155+ struct mlx5dr_domain *dmn = matcher->tbl->dmn;
1156+ struct mlx5dr_match_param param = {};
1157+ struct mlx5dr_rule *rule;
1158+ int ret;
1159+
1160+ if (!dr_rule_verify(matcher, value, &param))
1161+ return NULL;
1162+
1163+ rule = kzalloc(sizeof(*rule), GFP_KERNEL);
1164+ if (!rule)
1165+ return NULL;
1166+
1167+ rule->matcher = matcher;
1168+ INIT_LIST_HEAD(&rule->rule_actions_list);
1169+
1170+ ret = dr_rule_add_action_members(rule, num_actions, actions);
1171+ if (ret)
1172+ goto free_rule;
1173+
1174+ switch (dmn->type) {
1175+ case MLX5DR_DOMAIN_TYPE_NIC_RX:
1176+ rule->rx.nic_matcher = &matcher->rx;
1177+ ret = dr_rule_create_rule_nic(rule, &rule->rx, &param,
1178+ num_actions, actions);
1179+ break;
1180+ case MLX5DR_DOMAIN_TYPE_NIC_TX:
1181+ rule->tx.nic_matcher = &matcher->tx;
1182+ ret = dr_rule_create_rule_nic(rule, &rule->tx, &param,
1183+ num_actions, actions);
1184+ break;
1185+ case MLX5DR_DOMAIN_TYPE_FDB:
1186+ rule->rx.nic_matcher = &matcher->rx;
1187+ rule->tx.nic_matcher = &matcher->tx;
1188+ ret = dr_rule_create_rule_fdb(rule, &param,
1189+ num_actions, actions);
1190+ break;
1191+ default:
1192+ ret = -EINVAL;
1193+ break;
1194+ }
1195+
1196+ if (ret)
1197+ goto remove_action_members;
1198+
1199+ return rule;
1200+
1201+remove_action_members:
1202+ dr_rule_remove_action_members(rule);
1203+free_rule:
1204+ kfree(rule);
1205+ mlx5dr_info(dmn, "Failed creating rule\n");
1206+ return NULL;
1207+}
1208+
1209+struct mlx5dr_rule *mlx5dr_rule_create(struct mlx5dr_matcher *matcher,
1210+ struct mlx5dr_match_parameters *value,
1211+ size_t num_actions,
1212+ struct mlx5dr_action *actions[])
1213+{
1214+ struct mlx5dr_rule *rule;
1215+
1216+ mutex_lock(&matcher->tbl->dmn->mutex);
1217+ refcount_inc(&matcher->refcount);
1218+
1219+ rule = dr_rule_create_rule(matcher, value, num_actions, actions);
1220+ if (!rule)
1221+ refcount_dec(&matcher->refcount);
1222+
1223+ mutex_unlock(&matcher->tbl->dmn->mutex);
1224+
1225+ return rule;
1226+}
1227+
1228+int mlx5dr_rule_destroy(struct mlx5dr_rule *rule)
1229+{
1230+ struct mlx5dr_matcher *matcher = rule->matcher;
1231+ struct mlx5dr_table *tbl = rule->matcher->tbl;
1232+ int ret;
1233+
1234+ mutex_lock(&tbl->dmn->mutex);
1235+
1236+ ret = dr_rule_destroy_rule(rule);
1237+
1238+ mutex_unlock(&tbl->dmn->mutex);
1239+
1240+ if (!ret)
1241+ refcount_dec(&matcher->refcount);
1242+ return ret;
1243+}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c+976−0
@@ -0,0 +1,976 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+/* Copyright (c) 2019 Mellanox Technologies. */
3+
4+#include "dr_types.h"
5+
6+#define QUEUE_SIZE 128
7+#define SIGNAL_PER_DIV_QUEUE 16
8+#define TH_NUMS_TO_DRAIN 2
9+
10+enum { CQ_OK = 0, CQ_EMPTY = -1, CQ_POLL_ERR = -2 };
11+
12+struct dr_data_seg {
13+ u64 addr;
14+ u32 length;
15+ u32 lkey;
16+ unsigned int send_flags;
17+};
18+
19+struct postsend_info {
20+ struct dr_data_seg write;
21+ struct dr_data_seg read;
22+ u64 remote_addr;
23+ u32 rkey;
24+};
25+
26+struct dr_qp_rtr_attr {
27+ struct mlx5dr_cmd_gid_attr dgid_attr;
28+ enum ib_mtu mtu;
29+ u32 qp_num;
30+ u16 port_num;
31+ u8 min_rnr_timer;
32+ u8 sgid_index;
33+ u16 udp_src_port;
34+};
35+
36+struct dr_qp_rts_attr {
37+ u8 timeout;
38+ u8 retry_cnt;
39+ u8 rnr_retry;
40+};
41+
42+struct dr_qp_init_attr {
43+ u32 cqn;
44+ u32 pdn;
45+ u32 max_send_wr;
46+ struct mlx5_uars_page *uar;
47+};
48+
49+static int dr_parse_cqe(struct mlx5dr_cq *dr_cq, struct mlx5_cqe64 *cqe64)
50+{
51+ unsigned int idx;
52+ u8 opcode;
53+
54+ opcode = get_cqe_opcode(cqe64);
55+ if (opcode == MLX5_CQE_REQ_ERR) {
56+ idx = be16_to_cpu(cqe64->wqe_counter) &
57+ (dr_cq->qp->sq.wqe_cnt - 1);
58+ dr_cq->qp->sq.cc = dr_cq->qp->sq.wqe_head[idx] + 1;
59+ } else if (opcode == MLX5_CQE_RESP_ERR) {
60+ ++dr_cq->qp->sq.cc;
61+ } else {
62+ idx = be16_to_cpu(cqe64->wqe_counter) &
63+ (dr_cq->qp->sq.wqe_cnt - 1);
64+ dr_cq->qp->sq.cc = dr_cq->qp->sq.wqe_head[idx] + 1;
65+
66+ return CQ_OK;
67+ }
68+
69+ return CQ_POLL_ERR;
70+}
71+
72+static int dr_cq_poll_one(struct mlx5dr_cq *dr_cq)
73+{
74+ struct mlx5_cqe64 *cqe64;
75+ int err;
76+
77+ cqe64 = mlx5_cqwq_get_cqe(&dr_cq->wq);
78+ if (!cqe64)
79+ return CQ_EMPTY;
80+
81+ mlx5_cqwq_pop(&dr_cq->wq);
82+ err = dr_parse_cqe(dr_cq, cqe64);
83+ mlx5_cqwq_update_db_record(&dr_cq->wq);
84+
85+ return err;
86+}
87+
88+static int dr_poll_cq(struct mlx5dr_cq *dr_cq, int ne)
89+{
90+ int npolled;
91+ int err = 0;
92+
93+ for (npolled = 0; npolled < ne; ++npolled) {
94+ err = dr_cq_poll_one(dr_cq);
95+ if (err != CQ_OK)
96+ break;
97+ }
98+
99+ return err == CQ_POLL_ERR ? err : npolled;
100+}
101+
102+static void dr_qp_event(struct mlx5_core_qp *mqp, int event)
103+{
104+ pr_info("DR QP event %u on QP #%u\n", event, mqp->qpn);
105+}
106+
107+static struct mlx5dr_qp *dr_create_rc_qp(struct mlx5_core_dev *mdev,
108+ struct dr_qp_init_attr *attr)
109+{
110+ u32 temp_qpc[MLX5_ST_SZ_DW(qpc)] = {};
111+ struct mlx5_wq_param wqp;
112+ struct mlx5dr_qp *dr_qp;
113+ int inlen;
114+ void *qpc;
115+ void *in;
116+ int err;
117+
118+ dr_qp = kzalloc(sizeof(*dr_qp), GFP_KERNEL);
119+ if (!dr_qp)
120+ return NULL;
121+
122+ wqp.buf_numa_node = mdev->priv.numa_node;
123+ wqp.db_numa_node = mdev->priv.numa_node;
124+
125+ dr_qp->rq.pc = 0;
126+ dr_qp->rq.cc = 0;
127+ dr_qp->rq.wqe_cnt = 4;
128+ dr_qp->sq.pc = 0;
129+ dr_qp->sq.cc = 0;
130+ dr_qp->sq.wqe_cnt = roundup_pow_of_two(attr->max_send_wr);
131+
132+ MLX5_SET(qpc, temp_qpc, log_rq_stride, ilog2(MLX5_SEND_WQE_DS) - 4);
133+ MLX5_SET(qpc, temp_qpc, log_rq_size, ilog2(dr_qp->rq.wqe_cnt));
134+ MLX5_SET(qpc, temp_qpc, log_sq_size, ilog2(dr_qp->sq.wqe_cnt));
135+ err = mlx5_wq_qp_create(mdev, &wqp, temp_qpc, &dr_qp->wq,
136+ &dr_qp->wq_ctrl);
137+ if (err) {
138+ mlx5_core_info(mdev, "Can't create QP WQ\n");
139+ goto err_wq;
140+ }
141+
142+ dr_qp->sq.wqe_head = kcalloc(dr_qp->sq.wqe_cnt,
143+ sizeof(dr_qp->sq.wqe_head[0]),
144+ GFP_KERNEL);
145+
146+ if (!dr_qp->sq.wqe_head) {
147+ mlx5_core_warn(mdev, "Can't allocate wqe head\n");
148+ goto err_wqe_head;
149+ }
150+
151+ inlen = MLX5_ST_SZ_BYTES(create_qp_in) +
152+ MLX5_FLD_SZ_BYTES(create_qp_in, pas[0]) *
153+ dr_qp->wq_ctrl.buf.npages;
154+ in = kvzalloc(inlen, GFP_KERNEL);
155+ if (!in) {
156+ err = -ENOMEM;
157+ goto err_in;
158+ }
159+
160+ qpc = MLX5_ADDR_OF(create_qp_in, in, qpc);
161+ MLX5_SET(qpc, qpc, st, MLX5_QP_ST_RC);
162+ MLX5_SET(qpc, qpc, pm_state, MLX5_QP_PM_MIGRATED);
163+ MLX5_SET(qpc, qpc, pd, attr->pdn);
164+ MLX5_SET(qpc, qpc, uar_page, attr->uar->index);
165+ MLX5_SET(qpc, qpc, log_page_size,
166+ dr_qp->wq_ctrl.buf.page_shift - MLX5_ADAPTER_PAGE_SHIFT);
167+ MLX5_SET(qpc, qpc, fre, 1);
168+ MLX5_SET(qpc, qpc, rlky, 1);
169+ MLX5_SET(qpc, qpc, cqn_snd, attr->cqn);
170+ MLX5_SET(qpc, qpc, cqn_rcv, attr->cqn);
171+ MLX5_SET(qpc, qpc, log_rq_stride, ilog2(MLX5_SEND_WQE_DS) - 4);
172+ MLX5_SET(qpc, qpc, log_rq_size, ilog2(dr_qp->rq.wqe_cnt));
173+ MLX5_SET(qpc, qpc, rq_type, MLX5_NON_ZERO_RQ);
174+ MLX5_SET(qpc, qpc, log_sq_size, ilog2(dr_qp->sq.wqe_cnt));
175+ MLX5_SET64(qpc, qpc, dbr_addr, dr_qp->wq_ctrl.db.dma);
176+ if (MLX5_CAP_GEN(mdev, cqe_version) == 1)
177+ MLX5_SET(qpc, qpc, user_index, 0xFFFFFF);
178+ mlx5_fill_page_frag_array(&dr_qp->wq_ctrl.buf,
179+ (__be64 *)MLX5_ADDR_OF(create_qp_in,
180+ in, pas));
181+
182+ err = mlx5_core_create_qp(mdev, &dr_qp->mqp, in, inlen);
183+ kfree(in);
184+
185+ if (err) {
186+ mlx5_core_warn(mdev, " Can't create QP\n");
187+ goto err_in;
188+ }
189+ dr_qp->mqp.event = dr_qp_event;
190+ dr_qp->uar = attr->uar;
191+
192+ return dr_qp;
193+
194+err_in:
195+ kfree(dr_qp->sq.wqe_head);
196+err_wqe_head:
197+ mlx5_wq_destroy(&dr_qp->wq_ctrl);
198+err_wq:
199+ kfree(dr_qp);
200+ return NULL;
201+}
202+
203+static void dr_destroy_qp(struct mlx5_core_dev *mdev,
204+ struct mlx5dr_qp *dr_qp)
205+{
206+ mlx5_core_destroy_qp(mdev, &dr_qp->mqp);
207+ kfree(dr_qp->sq.wqe_head);
208+ mlx5_wq_destroy(&dr_qp->wq_ctrl);
209+ kfree(dr_qp);
210+}
211+
212+static void dr_cmd_notify_hw(struct mlx5dr_qp *dr_qp, void *ctrl)
213+{
214+ dma_wmb();
215+ *dr_qp->wq.sq.db = cpu_to_be32(dr_qp->sq.pc & 0xfffff);
216+
217+ /* After wmb() the hw aware of new work */
218+ wmb();
219+
220+ mlx5_write64(ctrl, dr_qp->uar->map + MLX5_BF_OFFSET);
221+}
222+
223+static void dr_rdma_segments(struct mlx5dr_qp *dr_qp, u64 remote_addr,
224+ u32 rkey, struct dr_data_seg *data_seg,
225+ u32 opcode, int nreq)
226+{
227+ struct mlx5_wqe_raddr_seg *wq_raddr;
228+ struct mlx5_wqe_ctrl_seg *wq_ctrl;
229+ struct mlx5_wqe_data_seg *wq_dseg;
230+ unsigned int size;
231+ unsigned int idx;
232+
233+ size = sizeof(*wq_ctrl) / 16 + sizeof(*wq_dseg) / 16 +
234+ sizeof(*wq_raddr) / 16;
235+
236+ idx = dr_qp->sq.pc & (dr_qp->sq.wqe_cnt - 1);
237+
238+ wq_ctrl = mlx5_wq_cyc_get_wqe(&dr_qp->wq.sq, idx);
239+ wq_ctrl->imm = 0;
240+ wq_ctrl->fm_ce_se = (data_seg->send_flags) ?
241+ MLX5_WQE_CTRL_CQ_UPDATE : 0;
242+ wq_ctrl->opmod_idx_opcode = cpu_to_be32(((dr_qp->sq.pc & 0xffff) << 8) |
243+ opcode);
244+ wq_ctrl->qpn_ds = cpu_to_be32(size | dr_qp->mqp.qpn << 8);
245+ wq_raddr = (void *)(wq_ctrl + 1);
246+ wq_raddr->raddr = cpu_to_be64(remote_addr);
247+ wq_raddr->rkey = cpu_to_be32(rkey);
248+ wq_raddr->reserved = 0;
249+
250+ wq_dseg = (void *)(wq_raddr + 1);
251+ wq_dseg->byte_count = cpu_to_be32(data_seg->length);
252+ wq_dseg->lkey = cpu_to_be32(data_seg->lkey);
253+ wq_dseg->addr = cpu_to_be64(data_seg->addr);
254+
255+ dr_qp->sq.wqe_head[idx] = dr_qp->sq.pc++;
256+
257+ if (nreq)
258+ dr_cmd_notify_hw(dr_qp, wq_ctrl);
259+}
260+
261+static void dr_post_send(struct mlx5dr_qp *dr_qp, struct postsend_info *send_info)
262+{
263+ dr_rdma_segments(dr_qp, send_info->remote_addr, send_info->rkey,
264+ &send_info->write, MLX5_OPCODE_RDMA_WRITE, 0);
265+ dr_rdma_segments(dr_qp, send_info->remote_addr, send_info->rkey,
266+ &send_info->read, MLX5_OPCODE_RDMA_READ, 1);
267+}
268+
269+/**
270+ * mlx5dr_send_fill_and_append_ste_send_info: Add data to be sent
271+ * with send_list parameters:
272+ *
273+ * @ste: The data that attached to this specific ste
274+ * @size: of data to write
275+ * @offset: of the data from start of the hw_ste entry
276+ * @data: data
277+ * @ste_info: ste to be sent with send_list
278+ * @send_list: to append into it
279+ * @copy_data: if true indicates that the data should be kept because
280+ * it's not backuped any where (like in re-hash).
281+ * if false, it lets the data to be updated after
282+ * it was added to the list.
283+ */
284+void mlx5dr_send_fill_and_append_ste_send_info(struct mlx5dr_ste *ste, u16 size,
285+ u16 offset, u8 *data,
286+ struct mlx5dr_ste_send_info *ste_info,
287+ struct list_head *send_list,
288+ bool copy_data)
289+{
290+ ste_info->size = size;
291+ ste_info->ste = ste;
292+ ste_info->offset = offset;
293+
294+ if (copy_data) {
295+ memcpy(ste_info->data_cont, data, size);
296+ ste_info->data = ste_info->data_cont;
297+ } else {
298+ ste_info->data = data;
299+ }
300+
301+ list_add_tail(&ste_info->send_list, send_list);
302+}
303+
304+/* The function tries to consume one wc each time, unless the queue is full, in
305+ * that case, which means that the hw is behind the sw in a full queue len
306+ * the function will drain the cq till it empty.
307+ */
308+static int dr_handle_pending_wc(struct mlx5dr_domain *dmn,
309+ struct mlx5dr_send_ring *send_ring)
310+{
311+ bool is_drain = false;
312+ int ne;
313+
314+ if (send_ring->pending_wqe < send_ring->signal_th)
315+ return 0;
316+
317+ /* Queue is full start drain it */
318+ if (send_ring->pending_wqe >=
319+ dmn->send_ring->signal_th * TH_NUMS_TO_DRAIN)
320+ is_drain = true;
321+
322+ do {
323+ ne = dr_poll_cq(send_ring->cq, 1);
324+ if (ne < 0)
325+ return ne;
326+ else if (ne == 1)
327+ send_ring->pending_wqe -= send_ring->signal_th;
328+ } while (is_drain && send_ring->pending_wqe);
329+
330+ return 0;
331+}
332+
333+static void dr_fill_data_segs(struct mlx5dr_send_ring *send_ring,
334+ struct postsend_info *send_info)
335+{
336+ send_ring->pending_wqe++;
337+
338+ if (send_ring->pending_wqe % send_ring->signal_th == 0)
339+ send_info->write.send_flags |= IB_SEND_SIGNALED;
340+
341+ send_ring->pending_wqe++;
342+ send_info->read.length = send_info->write.length;
343+ /* Read into the same write area */
344+ send_info->read.addr = (uintptr_t)send_info->write.addr;
345+ send_info->read.lkey = send_ring->mr->mkey.key;
346+
347+ if (send_ring->pending_wqe % send_ring->signal_th == 0)
348+ send_info->read.send_flags = IB_SEND_SIGNALED;
349+ else
350+ send_info->read.send_flags = 0;
351+}
352+
353+static int dr_postsend_icm_data(struct mlx5dr_domain *dmn,
354+ struct postsend_info *send_info)
355+{
356+ struct mlx5dr_send_ring *send_ring = dmn->send_ring;
357+ u32 buff_offset;
358+ int ret;
359+
360+ ret = dr_handle_pending_wc(dmn, send_ring);
361+ if (ret)
362+ return ret;
363+
364+ if (send_info->write.length > dmn->info.max_inline_size) {
365+ buff_offset = (send_ring->tx_head &
366+ (dmn->send_ring->signal_th - 1)) *
367+ send_ring->max_post_send_size;
368+ /* Copy to ring mr */
369+ memcpy(send_ring->buf + buff_offset,
370+ (void *)(uintptr_t)send_info->write.addr,
371+ send_info->write.length);
372+ send_info->write.addr = (uintptr_t)send_ring->mr->dma_addr + buff_offset;
373+ send_info->write.lkey = send_ring->mr->mkey.key;
374+ }
375+
376+ send_ring->tx_head++;
377+ dr_fill_data_segs(send_ring, send_info);
378+ dr_post_send(send_ring->qp, send_info);
379+
380+ return 0;
381+}
382+
383+static int dr_get_tbl_copy_details(struct mlx5dr_domain *dmn,
384+ struct mlx5dr_ste_htbl *htbl,
385+ u8 **data,
386+ u32 *byte_size,
387+ int *iterations,
388+ int *num_stes)
389+{
390+ int alloc_size;
391+
392+ if (htbl->chunk->byte_size > dmn->send_ring->max_post_send_size) {
393+ *iterations = htbl->chunk->byte_size /
394+ dmn->send_ring->max_post_send_size;
395+ *byte_size = dmn->send_ring->max_post_send_size;
396+ alloc_size = *byte_size;
397+ *num_stes = *byte_size / DR_STE_SIZE;
398+ } else {
399+ *iterations = 1;
400+ *num_stes = htbl->chunk->num_of_entries;
401+ alloc_size = *num_stes * DR_STE_SIZE;
402+ }
403+
404+ *data = kzalloc(alloc_size, GFP_KERNEL);
405+ if (!*data)
406+ return -ENOMEM;
407+
408+ return 0;
409+}
410+
411+/**
412+ * mlx5dr_send_postsend_ste: write size bytes into offset from the hw cm.
413+ *
414+ * @dmn: Domain
415+ * @ste: The ste struct that contains the data (at
416+ * least part of it)
417+ * @data: The real data to send size data
418+ * @size: for writing.
419+ * @offset: The offset from the icm mapped data to
420+ * start write to this for write only part of the
421+ * buffer.
422+ *
423+ * Return: 0 on success.
424+ */
425+int mlx5dr_send_postsend_ste(struct mlx5dr_domain *dmn, struct mlx5dr_ste *ste,
426+ u8 *data, u16 size, u16 offset)
427+{
428+ struct postsend_info send_info = {};
429+
430+ send_info.write.addr = (uintptr_t)data;
431+ send_info.write.length = size;
432+ send_info.write.lkey = 0;
433+ send_info.remote_addr = mlx5dr_ste_get_mr_addr(ste) + offset;
434+ send_info.rkey = ste->htbl->chunk->rkey;
435+
436+ return dr_postsend_icm_data(dmn, &send_info);
437+}
438+
439+int mlx5dr_send_postsend_htbl(struct mlx5dr_domain *dmn,
440+ struct mlx5dr_ste_htbl *htbl,
441+ u8 *formatted_ste, u8 *mask)
442+{
443+ u32 byte_size = htbl->chunk->byte_size;
444+ int num_stes_per_iter;
445+ int iterations;
446+ u8 *data;
447+ int ret;
448+ int i;
449+ int j;
450+
451+ ret = dr_get_tbl_copy_details(dmn, htbl, &data, &byte_size,
452+ &iterations, &num_stes_per_iter);
453+ if (ret)
454+ return ret;
455+
456+ /* Send the data iteration times */
457+ for (i = 0; i < iterations; i++) {
458+ u32 ste_index = i * (byte_size / DR_STE_SIZE);
459+ struct postsend_info send_info = {};
460+
461+ /* Copy all ste's on the data buffer
462+ * need to add the bit_mask
463+ */
464+ for (j = 0; j < num_stes_per_iter; j++) {
465+ u8 *hw_ste = htbl->ste_arr[ste_index + j].hw_ste;
466+ u32 ste_off = j * DR_STE_SIZE;
467+
468+ if (mlx5dr_ste_is_not_valid_entry(hw_ste)) {
469+ memcpy(data + ste_off,
470+ formatted_ste, DR_STE_SIZE);
471+ } else {
472+ /* Copy data */
473+ memcpy(data + ste_off,
474+ htbl->ste_arr[ste_index + j].hw_ste,
475+ DR_STE_SIZE_REDUCED);
476+ /* Copy bit_mask */
477+ memcpy(data + ste_off + DR_STE_SIZE_REDUCED,
478+ mask, DR_STE_SIZE_MASK);
479+ }
480+ }
481+
482+ send_info.write.addr = (uintptr_t)data;
483+ send_info.write.length = byte_size;
484+ send_info.write.lkey = 0;
485+ send_info.remote_addr =
486+ mlx5dr_ste_get_mr_addr(htbl->ste_arr + ste_index);
487+ send_info.rkey = htbl->chunk->rkey;
488+
489+ ret = dr_postsend_icm_data(dmn, &send_info);
490+ if (ret)
491+ goto out_free;
492+ }
493+
494+out_free:
495+ kfree(data);
496+ return ret;
497+}
498+
499+/* Initialize htble with default STEs */
500+int mlx5dr_send_postsend_formatted_htbl(struct mlx5dr_domain *dmn,
501+ struct mlx5dr_ste_htbl *htbl,
502+ u8 *ste_init_data,
503+ bool update_hw_ste)
504+{
505+ u32 byte_size = htbl->chunk->byte_size;
506+ int iterations;
507+ int num_stes;
508+ u8 *data;
509+ int ret;
510+ int i;
511+
512+ ret = dr_get_tbl_copy_details(dmn, htbl, &data, &byte_size,
513+ &iterations, &num_stes);
514+ if (ret)
515+ return ret;
516+
517+ for (i = 0; i < num_stes; i++) {
518+ u8 *copy_dst;
519+
520+ /* Copy the same ste on the data buffer */
521+ copy_dst = data + i * DR_STE_SIZE;
522+ memcpy(copy_dst, ste_init_data, DR_STE_SIZE);
523+
524+ if (update_hw_ste) {
525+ /* Copy the reduced ste to hash table ste_arr */
526+ copy_dst = htbl->hw_ste_arr + i * DR_STE_SIZE_REDUCED;
527+ memcpy(copy_dst, ste_init_data, DR_STE_SIZE_REDUCED);
528+ }
529+ }
530+
531+ /* Send the data iteration times */
532+ for (i = 0; i < iterations; i++) {
533+ u8 ste_index = i * (byte_size / DR_STE_SIZE);
534+ struct postsend_info send_info = {};
535+
536+ send_info.write.addr = (uintptr_t)data;
537+ send_info.write.length = byte_size;
538+ send_info.write.lkey = 0;
539+ send_info.remote_addr =
540+ mlx5dr_ste_get_mr_addr(htbl->ste_arr + ste_index);
541+ send_info.rkey = htbl->chunk->rkey;
542+
543+ ret = dr_postsend_icm_data(dmn, &send_info);
544+ if (ret)
545+ goto out_free;
546+ }
547+
548+out_free:
549+ kfree(data);
550+ return ret;
551+}
552+
553+int mlx5dr_send_postsend_action(struct mlx5dr_domain *dmn,
554+ struct mlx5dr_action *action)
555+{
556+ struct postsend_info send_info = {};
557+ int ret;
558+
559+ send_info.write.addr = (uintptr_t)action->rewrite.data;
560+ send_info.write.length = action->rewrite.chunk->byte_size;
561+ send_info.write.lkey = 0;
562+ send_info.remote_addr = action->rewrite.chunk->mr_addr;
563+ send_info.rkey = action->rewrite.chunk->rkey;
564+
565+ mutex_lock(&dmn->mutex);
566+ ret = dr_postsend_icm_data(dmn, &send_info);
567+ mutex_unlock(&dmn->mutex);
568+
569+ return ret;
570+}
571+
572+static int dr_modify_qp_rst2init(struct mlx5_core_dev *mdev,
573+ struct mlx5dr_qp *dr_qp,
574+ int port)
575+{
576+ u32 in[MLX5_ST_SZ_DW(rst2init_qp_in)] = {};
577+ void *qpc;
578+
579+ qpc = MLX5_ADDR_OF(rst2init_qp_in, in, qpc);
580+
581+ MLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, port);
582+ MLX5_SET(qpc, qpc, pm_state, MLX5_QPC_PM_STATE_MIGRATED);
583+ MLX5_SET(qpc, qpc, rre, 1);
584+ MLX5_SET(qpc, qpc, rwe, 1);
585+
586+ return mlx5_core_qp_modify(mdev, MLX5_CMD_OP_RST2INIT_QP, 0, qpc,
587+ &dr_qp->mqp);
588+}
589+
590+static int dr_cmd_modify_qp_rtr2rts(struct mlx5_core_dev *mdev,
591+ struct mlx5dr_qp *dr_qp,
592+ struct dr_qp_rts_attr *attr)
593+{
594+ u32 in[MLX5_ST_SZ_DW(rtr2rts_qp_in)] = {};
595+ void *qpc;
596+
597+ qpc = MLX5_ADDR_OF(rtr2rts_qp_in, in, qpc);
598+
599+ MLX5_SET(rtr2rts_qp_in, in, qpn, dr_qp->mqp.qpn);
600+
601+ MLX5_SET(qpc, qpc, log_ack_req_freq, 0);
602+ MLX5_SET(qpc, qpc, retry_count, attr->retry_cnt);
603+ MLX5_SET(qpc, qpc, rnr_retry, attr->rnr_retry);
604+
605+ return mlx5_core_qp_modify(mdev, MLX5_CMD_OP_RTR2RTS_QP, 0, qpc,
606+ &dr_qp->mqp);
607+}
608+
609+static int dr_cmd_modify_qp_init2rtr(struct mlx5_core_dev *mdev,
610+ struct mlx5dr_qp *dr_qp,
611+ struct dr_qp_rtr_attr *attr)
612+{
613+ u32 in[MLX5_ST_SZ_DW(init2rtr_qp_in)] = {};
614+ void *qpc;
615+
616+ qpc = MLX5_ADDR_OF(init2rtr_qp_in, in, qpc);
617+
618+ MLX5_SET(init2rtr_qp_in, in, qpn, dr_qp->mqp.qpn);
619+
620+ MLX5_SET(qpc, qpc, mtu, attr->mtu);
621+ MLX5_SET(qpc, qpc, log_msg_max, DR_CHUNK_SIZE_MAX - 1);
622+ MLX5_SET(qpc, qpc, remote_qpn, attr->qp_num);
623+ memcpy(MLX5_ADDR_OF(qpc, qpc, primary_address_path.rmac_47_32),
624+ attr->dgid_attr.mac, sizeof(attr->dgid_attr.mac));
625+ memcpy(MLX5_ADDR_OF(qpc, qpc, primary_address_path.rgid_rip),
626+ attr->dgid_attr.gid, sizeof(attr->dgid_attr.gid));
627+ MLX5_SET(qpc, qpc, primary_address_path.src_addr_index,
628+ attr->sgid_index);
629+
630+ if (attr->dgid_attr.roce_ver == MLX5_ROCE_VERSION_2)
631+ MLX5_SET(qpc, qpc, primary_address_path.udp_sport,
632+ attr->udp_src_port);
633+
634+ MLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, attr->port_num);
635+ MLX5_SET(qpc, qpc, min_rnr_nak, 1);
636+
637+ return mlx5_core_qp_modify(mdev, MLX5_CMD_OP_INIT2RTR_QP, 0, qpc,
638+ &dr_qp->mqp);
639+}
640+
641+static int dr_prepare_qp_to_rts(struct mlx5dr_domain *dmn)
642+{
643+ struct mlx5dr_qp *dr_qp = dmn->send_ring->qp;
644+ struct dr_qp_rts_attr rts_attr = {};
645+ struct dr_qp_rtr_attr rtr_attr = {};
646+ enum ib_mtu mtu = IB_MTU_1024;
647+ u16 gid_index = 0;
648+ int port = 1;
649+ int ret;
650+
651+ /* Init */
652+ ret = dr_modify_qp_rst2init(dmn->mdev, dr_qp, port);
653+ if (ret)
654+ return ret;
655+
656+ /* RTR */
657+ ret = mlx5dr_cmd_query_gid(dmn->mdev, port, gid_index, &rtr_attr.dgid_attr);
658+ if (ret)
659+ return ret;
660+
661+ rtr_attr.mtu = mtu;
662+ rtr_attr.qp_num = dr_qp->mqp.qpn;
663+ rtr_attr.min_rnr_timer = 12;
664+ rtr_attr.port_num = port;
665+ rtr_attr.sgid_index = gid_index;
666+ rtr_attr.udp_src_port = dmn->info.caps.roce_min_src_udp;
667+
668+ ret = dr_cmd_modify_qp_init2rtr(dmn->mdev, dr_qp, &rtr_attr);
669+ if (ret)
670+ return ret;
671+
672+ /* RTS */
673+ rts_attr.timeout = 14;
674+ rts_attr.retry_cnt = 7;
675+ rts_attr.rnr_retry = 7;
676+
677+ ret = dr_cmd_modify_qp_rtr2rts(dmn->mdev, dr_qp, &rts_attr);
678+ if (ret)
679+ return ret;
680+
681+ return 0;
682+}
683+
684+static void dr_cq_event(struct mlx5_core_cq *mcq,
685+ enum mlx5_event event)
686+{
687+ pr_info("CQ event %u on CQ #%u\n", event, mcq->cqn);
688+}
689+
690+static struct mlx5dr_cq *dr_create_cq(struct mlx5_core_dev *mdev,
691+ struct mlx5_uars_page *uar,
692+ size_t ncqe)
693+{
694+ u32 temp_cqc[MLX5_ST_SZ_DW(cqc)] = {};
695+ u32 out[MLX5_ST_SZ_DW(create_cq_out)];
696+ struct mlx5_wq_param wqp;
697+ struct mlx5_cqe64 *cqe;
698+ struct mlx5dr_cq *cq;
699+ int inlen, err, eqn;
700+ unsigned int irqn;
701+ void *cqc, *in;
702+ __be64 *pas;
703+ u32 i;
704+
705+ cq = kzalloc(sizeof(*cq), GFP_KERNEL);
706+ if (!cq)
707+ return NULL;
708+
709+ ncqe = roundup_pow_of_two(ncqe);
710+ MLX5_SET(cqc, temp_cqc, log_cq_size, ilog2(ncqe));
711+
712+ wqp.buf_numa_node = mdev->priv.numa_node;
713+ wqp.db_numa_node = mdev->priv.numa_node;
714+
715+ err = mlx5_cqwq_create(mdev, &wqp, temp_cqc, &cq->wq,
716+ &cq->wq_ctrl);
717+ if (err)
718+ goto out;
719+
720+ for (i = 0; i < mlx5_cqwq_get_size(&cq->wq); i++) {
721+ cqe = mlx5_cqwq_get_wqe(&cq->wq, i);
722+ cqe->op_own = MLX5_CQE_INVALID << 4 | MLX5_CQE_OWNER_MASK;
723+ }
724+
725+ inlen = MLX5_ST_SZ_BYTES(create_cq_in) +
726+ sizeof(u64) * cq->wq_ctrl.buf.npages;
727+ in = kvzalloc(inlen, GFP_KERNEL);
728+ if (!in)
729+ goto err_cqwq;
730+
731+ err = mlx5_vector2eqn(mdev, smp_processor_id(), &eqn, &irqn);
732+ if (err) {
733+ kvfree(in);
734+ goto err_cqwq;
735+ }
736+
737+ cqc = MLX5_ADDR_OF(create_cq_in, in, cq_context);
738+ MLX5_SET(cqc, cqc, log_cq_size, ilog2(ncqe));
739+ MLX5_SET(cqc, cqc, c_eqn, eqn);
740+ MLX5_SET(cqc, cqc, uar_page, uar->index);
741+ MLX5_SET(cqc, cqc, log_page_size, cq->wq_ctrl.buf.page_shift -
742+ MLX5_ADAPTER_PAGE_SHIFT);
743+ MLX5_SET64(cqc, cqc, dbr_addr, cq->wq_ctrl.db.dma);
744+
745+ pas = (__be64 *)MLX5_ADDR_OF(create_cq_in, in, pas);
746+ mlx5_fill_page_frag_array(&cq->wq_ctrl.buf, pas);
747+
748+ cq->mcq.event = dr_cq_event;
749+
750+ err = mlx5_core_create_cq(mdev, &cq->mcq, in, inlen, out, sizeof(out));
751+ kvfree(in);
752+
753+ if (err)
754+ goto err_cqwq;
755+
756+ cq->mcq.cqe_sz = 64;
757+ cq->mcq.set_ci_db = cq->wq_ctrl.db.db;
758+ cq->mcq.arm_db = cq->wq_ctrl.db.db + 1;
759+ *cq->mcq.set_ci_db = 0;
760+ *cq->mcq.arm_db = 0;
761+ cq->mcq.vector = 0;
762+ cq->mcq.irqn = irqn;
763+ cq->mcq.uar = uar;
764+
765+ return cq;
766+
767+err_cqwq:
768+ mlx5_wq_destroy(&cq->wq_ctrl);
769+out:
770+ kfree(cq);
771+ return NULL;
772+}
773+
774+static void dr_destroy_cq(struct mlx5_core_dev *mdev, struct mlx5dr_cq *cq)
775+{
776+ mlx5_core_destroy_cq(mdev, &cq->mcq);
777+ mlx5_wq_destroy(&cq->wq_ctrl);
778+ kfree(cq);
779+}
780+
781+static int
782+dr_create_mkey(struct mlx5_core_dev *mdev, u32 pdn, struct mlx5_core_mkey *mkey)
783+{
784+ u32 in[MLX5_ST_SZ_DW(create_mkey_in)] = {};
785+ void *mkc;
786+
787+ mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry);
788+ MLX5_SET(mkc, mkc, access_mode_1_0, MLX5_MKC_ACCESS_MODE_PA);
789+ MLX5_SET(mkc, mkc, a, 1);
790+ MLX5_SET(mkc, mkc, rw, 1);
791+ MLX5_SET(mkc, mkc, rr, 1);
792+ MLX5_SET(mkc, mkc, lw, 1);
793+ MLX5_SET(mkc, mkc, lr, 1);
794+
795+ MLX5_SET(mkc, mkc, pd, pdn);
796+ MLX5_SET(mkc, mkc, length64, 1);
797+ MLX5_SET(mkc, mkc, qpn, 0xffffff);
798+
799+ return mlx5_core_create_mkey(mdev, mkey, in, sizeof(in));
800+}
801+
802+static struct mlx5dr_mr *dr_reg_mr(struct mlx5_core_dev *mdev,
803+ u32 pdn, void *buf, size_t size)
804+{
805+ struct mlx5dr_mr *mr = kzalloc(sizeof(*mr), GFP_KERNEL);
806+ struct device *dma_device;
807+ dma_addr_t dma_addr;
808+ int err;
809+
810+ if (!mr)
811+ return NULL;
812+
813+ dma_device = &mdev->pdev->dev;
814+ dma_addr = dma_map_single(dma_device, buf, size,
815+ DMA_BIDIRECTIONAL);
816+ err = dma_mapping_error(dma_device, dma_addr);
817+ if (err) {
818+ mlx5_core_warn(mdev, "Can't dma buf\n");
819+ kfree(mr);
820+ return NULL;
821+ }
822+
823+ err = dr_create_mkey(mdev, pdn, &mr->mkey);
824+ if (err) {
825+ mlx5_core_warn(mdev, "Can't create mkey\n");
826+ dma_unmap_single(dma_device, dma_addr, size,
827+ DMA_BIDIRECTIONAL);
828+ kfree(mr);
829+ return NULL;
830+ }
831+
832+ mr->dma_addr = dma_addr;
833+ mr->size = size;
834+ mr->addr = buf;
835+
836+ return mr;
837+}
838+
839+static void dr_dereg_mr(struct mlx5_core_dev *mdev, struct mlx5dr_mr *mr)
840+{
841+ mlx5_core_destroy_mkey(mdev, &mr->mkey);
842+ dma_unmap_single(&mdev->pdev->dev, mr->dma_addr, mr->size,
843+ DMA_BIDIRECTIONAL);
844+ kfree(mr);
845+}
846+
847+int mlx5dr_send_ring_alloc(struct mlx5dr_domain *dmn)
848+{
849+ struct dr_qp_init_attr init_attr = {};
850+ int cq_size;
851+ int size;
852+ int ret;
853+
854+ dmn->send_ring = kzalloc(sizeof(*dmn->send_ring), GFP_KERNEL);
855+ if (!dmn->send_ring)
856+ return -ENOMEM;
857+
858+ cq_size = QUEUE_SIZE + 1;
859+ dmn->send_ring->cq = dr_create_cq(dmn->mdev, dmn->uar, cq_size);
860+ if (!dmn->send_ring->cq) {
861+ ret = -ENOMEM;
862+ goto free_send_ring;
863+ }
864+
865+ init_attr.cqn = dmn->send_ring->cq->mcq.cqn;
866+ init_attr.pdn = dmn->pdn;
867+ init_attr.uar = dmn->uar;
868+ init_attr.max_send_wr = QUEUE_SIZE;
869+
870+ dmn->send_ring->qp = dr_create_rc_qp(dmn->mdev, &init_attr);
871+ if (!dmn->send_ring->qp) {
872+ ret = -ENOMEM;
873+ goto clean_cq;
874+ }
875+
876+ dmn->send_ring->cq->qp = dmn->send_ring->qp;
877+
878+ dmn->info.max_send_wr = QUEUE_SIZE;
879+ dmn->info.max_inline_size = min(dmn->send_ring->qp->max_inline_data,
880+ DR_STE_SIZE);
881+
882+ dmn->send_ring->signal_th = dmn->info.max_send_wr /
883+ SIGNAL_PER_DIV_QUEUE;
884+
885+ /* Prepare qp to be used */
886+ ret = dr_prepare_qp_to_rts(dmn);
887+ if (ret)
888+ goto clean_qp;
889+
890+ dmn->send_ring->max_post_send_size =
891+ mlx5dr_icm_pool_chunk_size_to_byte(DR_CHUNK_SIZE_1K,
892+ DR_ICM_TYPE_STE);
893+
894+ /* Allocating the max size as a buffer for writing */
895+ size = dmn->send_ring->signal_th * dmn->send_ring->max_post_send_size;
896+ dmn->send_ring->buf = kzalloc(size, GFP_KERNEL);
897+ if (!dmn->send_ring->buf) {
898+ ret = -ENOMEM;
899+ goto clean_qp;
900+ }
901+
902+ memset(dmn->send_ring->buf, 0, size);
903+ dmn->send_ring->buf_size = size;
904+
905+ dmn->send_ring->mr = dr_reg_mr(dmn->mdev,
906+ dmn->pdn, dmn->send_ring->buf, size);
907+ if (!dmn->send_ring->mr) {
908+ ret = -ENOMEM;
909+ goto free_mem;
910+ }
911+
912+ dmn->send_ring->sync_mr = dr_reg_mr(dmn->mdev,
913+ dmn->pdn, dmn->send_ring->sync_buff,
914+ MIN_READ_SYNC);
915+ if (!dmn->send_ring->sync_mr) {
916+ ret = -ENOMEM;
917+ goto clean_mr;
918+ }
919+
920+ return 0;
921+
922+clean_mr:
923+ dr_dereg_mr(dmn->mdev, dmn->send_ring->mr);
924+free_mem:
925+ kfree(dmn->send_ring->buf);
926+clean_qp:
927+ dr_destroy_qp(dmn->mdev, dmn->send_ring->qp);
928+clean_cq:
929+ dr_destroy_cq(dmn->mdev, dmn->send_ring->cq);
930+free_send_ring:
931+ kfree(dmn->send_ring);
932+
933+ return ret;
934+}
935+
936+void mlx5dr_send_ring_free(struct mlx5dr_domain *dmn,
937+ struct mlx5dr_send_ring *send_ring)
938+{
939+ dr_destroy_qp(dmn->mdev, send_ring->qp);
940+ dr_destroy_cq(dmn->mdev, send_ring->cq);
941+ dr_dereg_mr(dmn->mdev, send_ring->sync_mr);
942+ dr_dereg_mr(dmn->mdev, send_ring->mr);
943+ kfree(send_ring->buf);
944+ kfree(send_ring);
945+}
946+
947+int mlx5dr_send_ring_force_drain(struct mlx5dr_domain *dmn)
948+{
949+ struct mlx5dr_send_ring *send_ring = dmn->send_ring;
950+ struct postsend_info send_info = {};
951+ u8 data[DR_STE_SIZE];
952+ int num_of_sends_req;
953+ int ret;
954+ int i;
955+
956+ /* Sending this amount of requests makes sure we will get drain */
957+ num_of_sends_req = send_ring->signal_th * TH_NUMS_TO_DRAIN / 2;
958+
959+ /* Send fake requests forcing the last to be signaled */
960+ send_info.write.addr = (uintptr_t)data;
961+ send_info.write.length = DR_STE_SIZE;
962+ send_info.write.lkey = 0;
963+ /* Using the sync_mr in order to write/read */
964+ send_info.remote_addr = (uintptr_t)send_ring->sync_mr->addr;
965+ send_info.rkey = send_ring->sync_mr->mkey.key;
966+
967+ for (i = 0; i < num_of_sends_req; i++) {
968+ ret = dr_postsend_icm_data(dmn, &send_info);
969+ if (ret)
970+ return ret;
971+ }
972+
973+ ret = dr_handle_pending_wc(dmn, send_ring);
974+
975+ return ret;
976+}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste.c+2308−0
@@ -0,0 +1,2308 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+/* Copyright (c) 2019 Mellanox Technologies. */
3+
4+#include <linux/types.h>
5+#include "dr_types.h"
6+
7+#define DR_STE_CRC_POLY 0xEDB88320L
8+#define STE_IPV4 0x1
9+#define STE_IPV6 0x2
10+#define STE_TCP 0x1
11+#define STE_UDP 0x2
12+#define STE_SPI 0x3
13+#define IP_VERSION_IPV4 0x4
14+#define IP_VERSION_IPV6 0x6
15+#define STE_SVLAN 0x1
16+#define STE_CVLAN 0x2
17+
18+#define DR_STE_ENABLE_FLOW_TAG BIT(31)
19+
20+/* Set to STE a specific value using DR_STE_SET */
21+#define DR_STE_SET_VAL(lookup_type, tag, t_fname, spec, s_fname, value) do { \
22+ if ((spec)->s_fname) { \
23+ MLX5_SET(ste_##lookup_type, tag, t_fname, value); \
24+ (spec)->s_fname = 0; \
25+ } \
26+} while (0)
27+
28+/* Set to STE spec->s_fname to tag->t_fname */
29+#define DR_STE_SET_TAG(lookup_type, tag, t_fname, spec, s_fname) \
30+ DR_STE_SET_VAL(lookup_type, tag, t_fname, spec, s_fname, spec->s_fname)
31+
32+/* Set to STE -1 to bit_mask->bm_fname and set spec->s_fname as used */
33+#define DR_STE_SET_MASK(lookup_type, bit_mask, bm_fname, spec, s_fname) \
34+ DR_STE_SET_VAL(lookup_type, bit_mask, bm_fname, spec, s_fname, -1)
35+
36+/* Set to STE spec->s_fname to bit_mask->bm_fname and set spec->s_fname as used */
37+#define DR_STE_SET_MASK_V(lookup_type, bit_mask, bm_fname, spec, s_fname) \
38+ DR_STE_SET_VAL(lookup_type, bit_mask, bm_fname, spec, s_fname, (spec)->s_fname)
39+
40+#define DR_STE_SET_TCP_FLAGS(lookup_type, tag, spec) do { \
41+ MLX5_SET(ste_##lookup_type, tag, tcp_ns, !!((spec)->tcp_flags & (1 << 8))); \
42+ MLX5_SET(ste_##lookup_type, tag, tcp_cwr, !!((spec)->tcp_flags & (1 << 7))); \
43+ MLX5_SET(ste_##lookup_type, tag, tcp_ece, !!((spec)->tcp_flags & (1 << 6))); \
44+ MLX5_SET(ste_##lookup_type, tag, tcp_urg, !!((spec)->tcp_flags & (1 << 5))); \
45+ MLX5_SET(ste_##lookup_type, tag, tcp_ack, !!((spec)->tcp_flags & (1 << 4))); \
46+ MLX5_SET(ste_##lookup_type, tag, tcp_psh, !!((spec)->tcp_flags & (1 << 3))); \
47+ MLX5_SET(ste_##lookup_type, tag, tcp_rst, !!((spec)->tcp_flags & (1 << 2))); \
48+ MLX5_SET(ste_##lookup_type, tag, tcp_syn, !!((spec)->tcp_flags & (1 << 1))); \
49+ MLX5_SET(ste_##lookup_type, tag, tcp_fin, !!((spec)->tcp_flags & (1 << 0))); \
50+} while (0)
51+
52+#define DR_STE_SET_MPLS_MASK(lookup_type, mask, in_out, bit_mask) do { \
53+ DR_STE_SET_MASK_V(lookup_type, mask, mpls0_label, mask, \
54+ in_out##_first_mpls_label);\
55+ DR_STE_SET_MASK_V(lookup_type, mask, mpls0_s_bos, mask, \
56+ in_out##_first_mpls_s_bos); \
57+ DR_STE_SET_MASK_V(lookup_type, mask, mpls0_exp, mask, \
58+ in_out##_first_mpls_exp); \
59+ DR_STE_SET_MASK_V(lookup_type, mask, mpls0_ttl, mask, \
60+ in_out##_first_mpls_ttl); \
61+} while (0)
62+
63+#define DR_STE_SET_MPLS_TAG(lookup_type, mask, in_out, tag) do { \
64+ DR_STE_SET_TAG(lookup_type, tag, mpls0_label, mask, \
65+ in_out##_first_mpls_label);\
66+ DR_STE_SET_TAG(lookup_type, tag, mpls0_s_bos, mask, \
67+ in_out##_first_mpls_s_bos); \
68+ DR_STE_SET_TAG(lookup_type, tag, mpls0_exp, mask, \
69+ in_out##_first_mpls_exp); \
70+ DR_STE_SET_TAG(lookup_type, tag, mpls0_ttl, mask, \
71+ in_out##_first_mpls_ttl); \
72+} while (0)
73+
74+#define DR_STE_IS_OUTER_MPLS_OVER_GRE_SET(_misc) (\
75+ (_misc)->outer_first_mpls_over_gre_label || \
76+ (_misc)->outer_first_mpls_over_gre_exp || \
77+ (_misc)->outer_first_mpls_over_gre_s_bos || \
78+ (_misc)->outer_first_mpls_over_gre_ttl)
79+#define DR_STE_IS_OUTER_MPLS_OVER_UDP_SET(_misc) (\
80+ (_misc)->outer_first_mpls_over_udp_label || \
81+ (_misc)->outer_first_mpls_over_udp_exp || \
82+ (_misc)->outer_first_mpls_over_udp_s_bos || \
83+ (_misc)->outer_first_mpls_over_udp_ttl)
84+
85+#define DR_STE_CALC_LU_TYPE(lookup_type, rx, inner) \
86+ ((inner) ? MLX5DR_STE_LU_TYPE_##lookup_type##_I : \
87+ (rx) ? MLX5DR_STE_LU_TYPE_##lookup_type##_D : \
88+ MLX5DR_STE_LU_TYPE_##lookup_type##_O)
89+
90+enum dr_ste_tunl_action {
91+ DR_STE_TUNL_ACTION_NONE = 0,
92+ DR_STE_TUNL_ACTION_ENABLE = 1,
93+ DR_STE_TUNL_ACTION_DECAP = 2,
94+ DR_STE_TUNL_ACTION_L3_DECAP = 3,
95+ DR_STE_TUNL_ACTION_POP_VLAN = 4,
96+};
97+
98+enum dr_ste_action_type {
99+ DR_STE_ACTION_TYPE_PUSH_VLAN = 1,
100+ DR_STE_ACTION_TYPE_ENCAP_L3 = 3,
101+ DR_STE_ACTION_TYPE_ENCAP = 4,
102+};
103+
104+struct dr_hw_ste_format {
105+ u8 ctrl[DR_STE_SIZE_CTRL];
106+ u8 tag[DR_STE_SIZE_TAG];
107+ u8 mask[DR_STE_SIZE_MASK];
108+};
109+
110+u32 mlx5dr_ste_calc_hash_index(u8 *hw_ste_p, struct mlx5dr_ste_htbl *htbl)
111+{
112+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
113+ u8 masked[DR_STE_SIZE_TAG] = {};
114+ u32 crc32, index;
115+ u16 bit;
116+ int i;
117+
118+ /* Don't calculate CRC if the result is predicted */
119+ if (htbl->chunk->num_of_entries == 1 || htbl->byte_mask == 0)
120+ return 0;
121+
122+ /* Mask tag using byte mask, bit per byte */
123+ bit = 1 << (DR_STE_SIZE_TAG - 1);
124+ for (i = 0; i < DR_STE_SIZE_TAG; i++) {
125+ if (htbl->byte_mask & bit)
126+ masked[i] = hw_ste->tag[i];
127+
128+ bit = bit >> 1;
129+ }
130+
131+ crc32 = mlx5dr_crc32_slice8_calc(masked, DR_STE_SIZE_TAG);
132+ index = crc32 & (htbl->chunk->num_of_entries - 1);
133+
134+ return index;
135+}
136+
137+static u16 dr_ste_conv_bit_to_byte_mask(u8 *bit_mask)
138+{
139+ u16 byte_mask = 0;
140+ int i;
141+
142+ for (i = 0; i < DR_STE_SIZE_MASK; i++) {
143+ byte_mask = byte_mask << 1;
144+ if (bit_mask[i] == 0xff)
145+ byte_mask |= 1;
146+ }
147+ return byte_mask;
148+}
149+
150+void mlx5dr_ste_set_bit_mask(u8 *hw_ste_p, u8 *bit_mask)
151+{
152+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
153+
154+ memcpy(hw_ste->mask, bit_mask, DR_STE_SIZE_MASK);
155+}
156+
157+void mlx5dr_ste_rx_set_flow_tag(u8 *hw_ste_p, u32 flow_tag)
158+{
159+ MLX5_SET(ste_rx_steering_mult, hw_ste_p, qp_list_pointer,
160+ DR_STE_ENABLE_FLOW_TAG | flow_tag);
161+}
162+
163+void mlx5dr_ste_set_counter_id(u8 *hw_ste_p, u32 ctr_id)
164+{
165+ /* This can be used for both rx_steering_mult and for sx_transmit */
166+ MLX5_SET(ste_rx_steering_mult, hw_ste_p, counter_trigger_15_0, ctr_id);
167+ MLX5_SET(ste_rx_steering_mult, hw_ste_p, counter_trigger_23_16, ctr_id >> 16);
168+}
169+
170+void mlx5dr_ste_set_go_back_bit(u8 *hw_ste_p)
171+{
172+ MLX5_SET(ste_sx_transmit, hw_ste_p, go_back, 1);
173+}
174+
175+void mlx5dr_ste_set_tx_push_vlan(u8 *hw_ste_p, u32 vlan_hdr,
176+ bool go_back)
177+{
178+ MLX5_SET(ste_sx_transmit, hw_ste_p, action_type,
179+ DR_STE_ACTION_TYPE_PUSH_VLAN);
180+ MLX5_SET(ste_sx_transmit, hw_ste_p, encap_pointer_vlan_data, vlan_hdr);
181+ /* Due to HW limitation we need to set this bit, otherwise reforamt +
182+ * push vlan will not work.
183+ */
184+ if (go_back)
185+ mlx5dr_ste_set_go_back_bit(hw_ste_p);
186+}
187+
188+void mlx5dr_ste_set_tx_encap(void *hw_ste_p, u32 reformat_id, int size, bool encap_l3)
189+{
190+ MLX5_SET(ste_sx_transmit, hw_ste_p, action_type,
191+ encap_l3 ? DR_STE_ACTION_TYPE_ENCAP_L3 : DR_STE_ACTION_TYPE_ENCAP);
192+ /* The hardware expects here size in words (2 byte) */
193+ MLX5_SET(ste_sx_transmit, hw_ste_p, action_description, size / 2);
194+ MLX5_SET(ste_sx_transmit, hw_ste_p, encap_pointer_vlan_data, reformat_id);
195+}
196+
197+void mlx5dr_ste_set_rx_decap(u8 *hw_ste_p)
198+{
199+ MLX5_SET(ste_rx_steering_mult, hw_ste_p, tunneling_action,
200+ DR_STE_TUNL_ACTION_DECAP);
201+}
202+
203+void mlx5dr_ste_set_rx_pop_vlan(u8 *hw_ste_p)
204+{
205+ MLX5_SET(ste_rx_steering_mult, hw_ste_p, tunneling_action,
206+ DR_STE_TUNL_ACTION_POP_VLAN);
207+}
208+
209+void mlx5dr_ste_set_rx_decap_l3(u8 *hw_ste_p, bool vlan)
210+{
211+ MLX5_SET(ste_rx_steering_mult, hw_ste_p, tunneling_action,
212+ DR_STE_TUNL_ACTION_L3_DECAP);
213+ MLX5_SET(ste_modify_packet, hw_ste_p, action_description, vlan ? 1 : 0);
214+}
215+
216+void mlx5dr_ste_set_entry_type(u8 *hw_ste_p, u8 entry_type)
217+{
218+ MLX5_SET(ste_general, hw_ste_p, entry_type, entry_type);
219+}
220+
221+u8 mlx5dr_ste_get_entry_type(u8 *hw_ste_p)
222+{
223+ return MLX5_GET(ste_general, hw_ste_p, entry_type);
224+}
225+
226+void mlx5dr_ste_set_rewrite_actions(u8 *hw_ste_p, u16 num_of_actions,
227+ u32 re_write_index)
228+{
229+ MLX5_SET(ste_modify_packet, hw_ste_p, number_of_re_write_actions,
230+ num_of_actions);
231+ MLX5_SET(ste_modify_packet, hw_ste_p, header_re_write_actions_pointer,
232+ re_write_index);
233+}
234+
235+void mlx5dr_ste_set_hit_gvmi(u8 *hw_ste_p, u16 gvmi)
236+{
237+ MLX5_SET(ste_general, hw_ste_p, next_table_base_63_48, gvmi);
238+}
239+
240+void mlx5dr_ste_init(u8 *hw_ste_p, u8 lu_type, u8 entry_type,
241+ u16 gvmi)
242+{
243+ MLX5_SET(ste_general, hw_ste_p, entry_type, entry_type);
244+ MLX5_SET(ste_general, hw_ste_p, entry_sub_type, lu_type);
245+ MLX5_SET(ste_general, hw_ste_p, next_lu_type, MLX5DR_STE_LU_TYPE_DONT_CARE);
246+
247+ /* Set GVMI once, this is the same for RX/TX
248+ * bits 63_48 of next table base / miss address encode the next GVMI
249+ */
250+ MLX5_SET(ste_rx_steering_mult, hw_ste_p, gvmi, gvmi);
251+ MLX5_SET(ste_rx_steering_mult, hw_ste_p, next_table_base_63_48, gvmi);
252+ MLX5_SET(ste_rx_steering_mult, hw_ste_p, miss_address_63_48, gvmi);
253+}
254+
255+static void dr_ste_set_always_hit(struct dr_hw_ste_format *hw_ste)
256+{
257+ memset(&hw_ste->tag, 0, sizeof(hw_ste->tag));
258+ memset(&hw_ste->mask, 0, sizeof(hw_ste->mask));
259+}
260+
261+static void dr_ste_set_always_miss(struct dr_hw_ste_format *hw_ste)
262+{
263+ hw_ste->tag[0] = 0xdc;
264+ hw_ste->mask[0] = 0;
265+}
266+
267+u64 mlx5dr_ste_get_miss_addr(u8 *hw_ste)
268+{
269+ u64 index =
270+ (MLX5_GET(ste_rx_steering_mult, hw_ste, miss_address_31_6) |
271+ MLX5_GET(ste_rx_steering_mult, hw_ste, miss_address_39_32) << 26);
272+
273+ return index << 6;
274+}
275+
276+void mlx5dr_ste_set_hit_addr(u8 *hw_ste, u64 icm_addr, u32 ht_size)
277+{
278+ u64 index = (icm_addr >> 5) | ht_size;
279+
280+ MLX5_SET(ste_general, hw_ste, next_table_base_39_32_size, index >> 27);
281+ MLX5_SET(ste_general, hw_ste, next_table_base_31_5_size, index);
282+}
283+
284+u64 mlx5dr_ste_get_icm_addr(struct mlx5dr_ste *ste)
285+{
286+ u32 index = ste - ste->htbl->ste_arr;
287+
288+ return ste->htbl->chunk->icm_addr + DR_STE_SIZE * index;
289+}
290+
291+u64 mlx5dr_ste_get_mr_addr(struct mlx5dr_ste *ste)
292+{
293+ u32 index = ste - ste->htbl->ste_arr;
294+
295+ return ste->htbl->chunk->mr_addr + DR_STE_SIZE * index;
296+}
297+
298+struct list_head *mlx5dr_ste_get_miss_list(struct mlx5dr_ste *ste)
299+{
300+ u32 index = ste - ste->htbl->ste_arr;
301+
302+ return &ste->htbl->miss_list[index];
303+}
304+
305+static void dr_ste_always_hit_htbl(struct mlx5dr_ste *ste,
306+ struct mlx5dr_ste_htbl *next_htbl)
307+{
308+ struct mlx5dr_icm_chunk *chunk = next_htbl->chunk;
309+ u8 *hw_ste = ste->hw_ste;
310+
311+ MLX5_SET(ste_general, hw_ste, byte_mask, next_htbl->byte_mask);
312+ MLX5_SET(ste_general, hw_ste, next_lu_type, next_htbl->lu_type);
313+ mlx5dr_ste_set_hit_addr(hw_ste, chunk->icm_addr, chunk->num_of_entries);
314+
315+ dr_ste_set_always_hit((struct dr_hw_ste_format *)ste->hw_ste);
316+}
317+
318+bool mlx5dr_ste_is_last_in_rule(struct mlx5dr_matcher_rx_tx *nic_matcher,
319+ u8 ste_location)
320+{
321+ return ste_location == nic_matcher->num_of_builders;
322+}
323+
324+/* Replace relevant fields, except of:
325+ * htbl - keep the origin htbl
326+ * miss_list + list - already took the src from the list.
327+ * icm_addr/mr_addr - depends on the hosting table.
328+ *
329+ * Before:
330+ * | a | -> | b | -> | c | ->
331+ *
332+ * After:
333+ * | a | -> | c | ->
334+ * While the data that was in b copied to a.
335+ */
336+static void dr_ste_replace(struct mlx5dr_ste *dst, struct mlx5dr_ste *src)
337+{
338+ memcpy(dst->hw_ste, src->hw_ste, DR_STE_SIZE_REDUCED);
339+ dst->next_htbl = src->next_htbl;
340+ if (dst->next_htbl)
341+ dst->next_htbl->pointing_ste = dst;
342+
343+ refcount_set(&dst->refcount, refcount_read(&src->refcount));
344+
345+ INIT_LIST_HEAD(&dst->rule_list);
346+ list_splice_tail_init(&src->rule_list, &dst->rule_list);
347+}
348+
349+/* Free ste which is the head and the only one in miss_list */
350+static void
351+dr_ste_remove_head_ste(struct mlx5dr_ste *ste,
352+ struct mlx5dr_matcher_rx_tx *nic_matcher,
353+ struct mlx5dr_ste_send_info *ste_info_head,
354+ struct list_head *send_ste_list,
355+ struct mlx5dr_ste_htbl *stats_tbl)
356+{
357+ u8 tmp_data_ste[DR_STE_SIZE] = {};
358+ struct mlx5dr_ste tmp_ste = {};
359+ u64 miss_addr;
360+
361+ tmp_ste.hw_ste = tmp_data_ste;
362+
363+ /* Use temp ste because dr_ste_always_miss_addr
364+ * touches bit_mask area which doesn't exist at ste->hw_ste.
365+ */
366+ memcpy(tmp_ste.hw_ste, ste->hw_ste, DR_STE_SIZE_REDUCED);
367+ miss_addr = nic_matcher->e_anchor->chunk->icm_addr;
368+ mlx5dr_ste_always_miss_addr(&tmp_ste, miss_addr);
369+ memcpy(ste->hw_ste, tmp_ste.hw_ste, DR_STE_SIZE_REDUCED);
370+
371+ list_del_init(&ste->miss_list_node);
372+
373+ /* Write full STE size in order to have "always_miss" */
374+ mlx5dr_send_fill_and_append_ste_send_info(ste, DR_STE_SIZE,
375+ 0, tmp_data_ste,
376+ ste_info_head,
377+ send_ste_list,
378+ true /* Copy data */);
379+
380+ stats_tbl->ctrl.num_of_valid_entries--;
381+}
382+
383+/* Free ste which is the head but NOT the only one in miss_list:
384+ * |_ste_| --> |_next_ste_| -->|__| -->|__| -->/0
385+ */
386+static void
387+dr_ste_replace_head_ste(struct mlx5dr_ste *ste, struct mlx5dr_ste *next_ste,
388+ struct mlx5dr_ste_send_info *ste_info_head,
389+ struct list_head *send_ste_list,
390+ struct mlx5dr_ste_htbl *stats_tbl)
391+
392+{
393+ struct mlx5dr_ste_htbl *next_miss_htbl;
394+
395+ next_miss_htbl = next_ste->htbl;
396+
397+ /* Remove from the miss_list the next_ste before copy */
398+ list_del_init(&next_ste->miss_list_node);
399+
400+ /* All rule-members that use next_ste should know about that */
401+ mlx5dr_rule_update_rule_member(next_ste, ste);
402+
403+ /* Move data from next into ste */
404+ dr_ste_replace(ste, next_ste);
405+
406+ /* Del the htbl that contains the next_ste.
407+ * The origin htbl stay with the same number of entries.
408+ */
409+ mlx5dr_htbl_put(next_miss_htbl);
410+
411+ mlx5dr_send_fill_and_append_ste_send_info(ste, DR_STE_SIZE_REDUCED,
412+ 0, ste->hw_ste,
413+ ste_info_head,
414+ send_ste_list,
415+ true /* Copy data */);
416+
417+ stats_tbl->ctrl.num_of_collisions--;
418+ stats_tbl->ctrl.num_of_valid_entries--;
419+}
420+
421+/* Free ste that is located in the middle of the miss list:
422+ * |__| -->|_prev_ste_|->|_ste_|-->|_next_ste_|
423+ */
424+static void dr_ste_remove_middle_ste(struct mlx5dr_ste *ste,
425+ struct mlx5dr_ste_send_info *ste_info,
426+ struct list_head *send_ste_list,
427+ struct mlx5dr_ste_htbl *stats_tbl)
428+{
429+ struct mlx5dr_ste *prev_ste;
430+ u64 miss_addr;
431+
432+ prev_ste = list_entry(mlx5dr_ste_get_miss_list(ste)->prev, struct mlx5dr_ste,
433+ miss_list_node);
434+ if (!prev_ste) {
435+ WARN_ON(true);
436+ return;
437+ }
438+
439+ miss_addr = mlx5dr_ste_get_miss_addr(ste->hw_ste);
440+ mlx5dr_ste_set_miss_addr(prev_ste->hw_ste, miss_addr);
441+
442+ mlx5dr_send_fill_and_append_ste_send_info(prev_ste, DR_STE_SIZE_REDUCED, 0,
443+ prev_ste->hw_ste, ste_info,
444+ send_ste_list, true /* Copy data*/);
445+
446+ list_del_init(&ste->miss_list_node);
447+
448+ stats_tbl->ctrl.num_of_valid_entries--;
449+ stats_tbl->ctrl.num_of_collisions--;
450+}
451+
452+void mlx5dr_ste_free(struct mlx5dr_ste *ste,
453+ struct mlx5dr_matcher *matcher,
454+ struct mlx5dr_matcher_rx_tx *nic_matcher)
455+{
456+ struct mlx5dr_ste_send_info *cur_ste_info, *tmp_ste_info;
457+ struct mlx5dr_domain *dmn = matcher->tbl->dmn;
458+ struct mlx5dr_ste_send_info ste_info_head;
459+ struct mlx5dr_ste *next_ste, *first_ste;
460+ bool put_on_origin_table = true;
461+ struct mlx5dr_ste_htbl *stats_tbl;
462+ LIST_HEAD(send_ste_list);
463+
464+ first_ste = list_entry(mlx5dr_ste_get_miss_list(ste)->next,
465+ struct mlx5dr_ste, miss_list_node);
466+ stats_tbl = first_ste->htbl;
467+
468+ /* Two options:
469+ * 1. ste is head:
470+ * a. head ste is the only ste in the miss list
471+ * b. head ste is not the only ste in the miss-list
472+ * 2. ste is not head
473+ */
474+ if (first_ste == ste) { /* Ste is the head */
475+ struct mlx5dr_ste *last_ste;
476+
477+ last_ste = list_last_entry(mlx5dr_ste_get_miss_list(ste),
478+ struct mlx5dr_ste, miss_list_node);
479+ if (last_ste == first_ste)
480+ next_ste = NULL;
481+ else
482+ next_ste = list_entry(ste->miss_list_node.next,
483+ struct mlx5dr_ste, miss_list_node);
484+
485+ if (!next_ste) {
486+ /* One and only entry in the list */
487+ dr_ste_remove_head_ste(ste, nic_matcher,
488+ &ste_info_head,
489+ &send_ste_list,
490+ stats_tbl);
491+ } else {
492+ /* First but not only entry in the list */
493+ dr_ste_replace_head_ste(ste, next_ste, &ste_info_head,
494+ &send_ste_list, stats_tbl);
495+ put_on_origin_table = false;
496+ }
497+ } else { /* Ste in the middle of the list */
498+ dr_ste_remove_middle_ste(ste, &ste_info_head, &send_ste_list, stats_tbl);
499+ }
500+
501+ /* Update HW */
502+ list_for_each_entry_safe(cur_ste_info, tmp_ste_info,
503+ &send_ste_list, send_list) {
504+ list_del(&cur_ste_info->send_list);
505+ mlx5dr_send_postsend_ste(dmn, cur_ste_info->ste,
506+ cur_ste_info->data, cur_ste_info->size,
507+ cur_ste_info->offset);
508+ }
509+
510+ if (put_on_origin_table)
511+ mlx5dr_htbl_put(ste->htbl);
512+}
513+
514+bool mlx5dr_ste_equal_tag(void *src, void *dst)
515+{
516+ struct dr_hw_ste_format *s_hw_ste = (struct dr_hw_ste_format *)src;
517+ struct dr_hw_ste_format *d_hw_ste = (struct dr_hw_ste_format *)dst;
518+
519+ return !memcmp(s_hw_ste->tag, d_hw_ste->tag, DR_STE_SIZE_TAG);
520+}
521+
522+void mlx5dr_ste_set_hit_addr_by_next_htbl(u8 *hw_ste,
523+ struct mlx5dr_ste_htbl *next_htbl)
524+{
525+ struct mlx5dr_icm_chunk *chunk = next_htbl->chunk;
526+
527+ mlx5dr_ste_set_hit_addr(hw_ste, chunk->icm_addr, chunk->num_of_entries);
528+}
529+
530+void mlx5dr_ste_set_miss_addr(u8 *hw_ste_p, u64 miss_addr)
531+{
532+ u64 index = miss_addr >> 6;
533+
534+ /* Miss address for TX and RX STEs located in the same offsets */
535+ MLX5_SET(ste_rx_steering_mult, hw_ste_p, miss_address_39_32, index >> 26);
536+ MLX5_SET(ste_rx_steering_mult, hw_ste_p, miss_address_31_6, index);
537+}
538+
539+void mlx5dr_ste_always_miss_addr(struct mlx5dr_ste *ste, u64 miss_addr)
540+{
541+ u8 *hw_ste = ste->hw_ste;
542+
543+ MLX5_SET(ste_rx_steering_mult, hw_ste, next_lu_type, MLX5DR_STE_LU_TYPE_DONT_CARE);
544+ mlx5dr_ste_set_miss_addr(hw_ste, miss_addr);
545+ dr_ste_set_always_miss((struct dr_hw_ste_format *)ste->hw_ste);
546+}
547+
548+/* The assumption here is that we don't update the ste->hw_ste if it is not
549+ * used ste, so it will be all zero, checking the next_lu_type.
550+ */
551+bool mlx5dr_ste_is_not_valid_entry(u8 *p_hw_ste)
552+{
553+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)p_hw_ste;
554+
555+ if (MLX5_GET(ste_general, hw_ste, next_lu_type) ==
556+ MLX5DR_STE_LU_TYPE_NOP)
557+ return true;
558+
559+ return false;
560+}
561+
562+bool mlx5dr_ste_not_used_ste(struct mlx5dr_ste *ste)
563+{
564+ return !refcount_read(&ste->refcount);
565+}
566+
567+static u16 get_bits_per_mask(u16 byte_mask)
568+{
569+ u16 bits = 0;
570+
571+ while (byte_mask) {
572+ byte_mask = byte_mask & (byte_mask - 1);
573+ bits++;
574+ }
575+
576+ return bits;
577+}
578+
579+/* Init one ste as a pattern for ste data array */
580+void mlx5dr_ste_set_formatted_ste(u16 gvmi,
581+ struct mlx5dr_domain_rx_tx *nic_dmn,
582+ struct mlx5dr_ste_htbl *htbl,
583+ u8 *formatted_ste,
584+ struct mlx5dr_htbl_connect_info *connect_info)
585+{
586+ struct mlx5dr_ste ste = {};
587+
588+ mlx5dr_ste_init(formatted_ste, htbl->lu_type, nic_dmn->ste_type, gvmi);
589+ ste.hw_ste = formatted_ste;
590+
591+ if (connect_info->type == CONNECT_HIT)
592+ dr_ste_always_hit_htbl(&ste, connect_info->hit_next_htbl);
593+ else
594+ mlx5dr_ste_always_miss_addr(&ste, connect_info->miss_icm_addr);
595+}
596+
597+int mlx5dr_ste_htbl_init_and_postsend(struct mlx5dr_domain *dmn,
598+ struct mlx5dr_domain_rx_tx *nic_dmn,
599+ struct mlx5dr_ste_htbl *htbl,
600+ struct mlx5dr_htbl_connect_info *connect_info,
601+ bool update_hw_ste)
602+{
603+ u8 formatted_ste[DR_STE_SIZE] = {};
604+
605+ mlx5dr_ste_set_formatted_ste(dmn->info.caps.gvmi,
606+ nic_dmn,
607+ htbl,
608+ formatted_ste,
609+ connect_info);
610+
611+ return mlx5dr_send_postsend_formatted_htbl(dmn, htbl, formatted_ste, update_hw_ste);
612+}
613+
614+int mlx5dr_ste_create_next_htbl(struct mlx5dr_matcher *matcher,
615+ struct mlx5dr_matcher_rx_tx *nic_matcher,
616+ struct mlx5dr_ste *ste,
617+ u8 *cur_hw_ste,
618+ enum mlx5dr_icm_chunk_size log_table_size)
619+{
620+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)cur_hw_ste;
621+ struct mlx5dr_domain_rx_tx *nic_dmn = nic_matcher->nic_tbl->nic_dmn;
622+ struct mlx5dr_domain *dmn = matcher->tbl->dmn;
623+ struct mlx5dr_htbl_connect_info info;
624+ struct mlx5dr_ste_htbl *next_htbl;
625+
626+ if (!mlx5dr_ste_is_last_in_rule(nic_matcher, ste->ste_chain_location)) {
627+ u32 bits_in_mask;
628+ u8 next_lu_type;
629+ u16 byte_mask;
630+
631+ next_lu_type = MLX5_GET(ste_general, hw_ste, next_lu_type);
632+ byte_mask = MLX5_GET(ste_general, hw_ste, byte_mask);
633+
634+ /* Don't allocate table more than required,
635+ * the size of the table defined via the byte_mask, so no need
636+ * to allocate more than that.
637+ */
638+ bits_in_mask = get_bits_per_mask(byte_mask) * BITS_PER_BYTE;
639+ log_table_size = min(log_table_size, bits_in_mask);
640+
641+ next_htbl = mlx5dr_ste_htbl_alloc(dmn->ste_icm_pool,
642+ log_table_size,
643+ next_lu_type,
644+ byte_mask);
645+ if (!next_htbl) {
646+ mlx5dr_dbg(dmn, "Failed allocating table\n");
647+ return -ENOMEM;
648+ }
649+
650+ /* Write new table to HW */
651+ info.type = CONNECT_MISS;
652+ info.miss_icm_addr = nic_matcher->e_anchor->chunk->icm_addr;
653+ if (mlx5dr_ste_htbl_init_and_postsend(dmn, nic_dmn, next_htbl,
654+ &info, false)) {
655+ mlx5dr_info(dmn, "Failed writing table to HW\n");
656+ goto free_table;
657+ }
658+
659+ mlx5dr_ste_set_hit_addr_by_next_htbl(cur_hw_ste, next_htbl);
660+ ste->next_htbl = next_htbl;
661+ next_htbl->pointing_ste = ste;
662+ }
663+
664+ return 0;
665+
666+free_table:
667+ mlx5dr_ste_htbl_free(next_htbl);
668+ return -ENOENT;
669+}
670+
671+static void dr_ste_set_ctrl(struct mlx5dr_ste_htbl *htbl)
672+{
673+ struct mlx5dr_ste_htbl_ctrl *ctrl = &htbl->ctrl;
674+ int num_of_entries;
675+
676+ htbl->ctrl.may_grow = true;
677+
678+ if (htbl->chunk_size == DR_CHUNK_SIZE_MAX - 1)
679+ htbl->ctrl.may_grow = false;
680+
681+ /* Threshold is 50%, one is added to table of size 1 */
682+ num_of_entries = mlx5dr_icm_pool_chunk_size_to_entries(htbl->chunk_size);
683+ ctrl->increase_threshold = (num_of_entries + 1) / 2;
684+}
685+
686+struct mlx5dr_ste_htbl *mlx5dr_ste_htbl_alloc(struct mlx5dr_icm_pool *pool,
687+ enum mlx5dr_icm_chunk_size chunk_size,
688+ u8 lu_type, u16 byte_mask)
689+{
690+ struct mlx5dr_icm_chunk *chunk;
691+ struct mlx5dr_ste_htbl *htbl;
692+ int i;
693+
694+ htbl = kzalloc(sizeof(*htbl), GFP_KERNEL);
695+ if (!htbl)
696+ return NULL;
697+
698+ chunk = mlx5dr_icm_alloc_chunk(pool, chunk_size);
699+ if (!chunk)
700+ goto out_free_htbl;
701+
702+ htbl->chunk = chunk;
703+ htbl->lu_type = lu_type;
704+ htbl->byte_mask = byte_mask;
705+ htbl->ste_arr = chunk->ste_arr;
706+ htbl->hw_ste_arr = chunk->hw_ste_arr;
707+ htbl->miss_list = chunk->miss_list;
708+ refcount_set(&htbl->refcount, 0);
709+
710+ for (i = 0; i < chunk->num_of_entries; i++) {
711+ struct mlx5dr_ste *ste = &htbl->ste_arr[i];
712+
713+ ste->hw_ste = htbl->hw_ste_arr + i * DR_STE_SIZE_REDUCED;
714+ ste->htbl = htbl;
715+ refcount_set(&ste->refcount, 0);
716+ INIT_LIST_HEAD(&ste->miss_list_node);
717+ INIT_LIST_HEAD(&htbl->miss_list[i]);
718+ INIT_LIST_HEAD(&ste->rule_list);
719+ }
720+
721+ htbl->chunk_size = chunk_size;
722+ dr_ste_set_ctrl(htbl);
723+ return htbl;
724+
725+out_free_htbl:
726+ kfree(htbl);
727+ return NULL;
728+}
729+
730+int mlx5dr_ste_htbl_free(struct mlx5dr_ste_htbl *htbl)
731+{
732+ if (refcount_read(&htbl->refcount))
733+ return -EBUSY;
734+
735+ mlx5dr_icm_free_chunk(htbl->chunk);
736+ kfree(htbl);
737+ return 0;
738+}
739+
740+int mlx5dr_ste_build_pre_check(struct mlx5dr_domain *dmn,
741+ u8 match_criteria,
742+ struct mlx5dr_match_param *mask,
743+ struct mlx5dr_match_param *value)
744+{
745+ if (!value && (match_criteria & DR_MATCHER_CRITERIA_MISC)) {
746+ if (mask->misc.source_port && mask->misc.source_port != 0xffff) {
747+ mlx5dr_dbg(dmn, "Partial mask source_port is not supported\n");
748+ return -EINVAL;
749+ }
750+ }
751+
752+ return 0;
753+}
754+
755+int mlx5dr_ste_build_ste_arr(struct mlx5dr_matcher *matcher,
756+ struct mlx5dr_matcher_rx_tx *nic_matcher,
757+ struct mlx5dr_match_param *value,
758+ u8 *ste_arr)
759+{
760+ struct mlx5dr_domain_rx_tx *nic_dmn = nic_matcher->nic_tbl->nic_dmn;
761+ struct mlx5dr_domain *dmn = matcher->tbl->dmn;
762+ struct mlx5dr_ste_build *sb;
763+ int ret, i;
764+
765+ ret = mlx5dr_ste_build_pre_check(dmn, matcher->match_criteria,
766+ &matcher->mask, value);
767+ if (ret)
768+ return ret;
769+
770+ sb = nic_matcher->ste_builder;
771+ for (i = 0; i < nic_matcher->num_of_builders; i++) {
772+ mlx5dr_ste_init(ste_arr,
773+ sb->lu_type,
774+ nic_dmn->ste_type,
775+ dmn->info.caps.gvmi);
776+
777+ mlx5dr_ste_set_bit_mask(ste_arr, sb->bit_mask);
778+
779+ ret = sb->ste_build_tag_func(value, sb, ste_arr);
780+ if (ret)
781+ return ret;
782+
783+ /* Connect the STEs */
784+ if (i < (nic_matcher->num_of_builders - 1)) {
785+ /* Need the next builder for these fields,
786+ * not relevant for the last ste in the chain.
787+ */
788+ sb++;
789+ MLX5_SET(ste_general, ste_arr, next_lu_type, sb->lu_type);
790+ MLX5_SET(ste_general, ste_arr, byte_mask, sb->byte_mask);
791+ }
792+ ste_arr += DR_STE_SIZE;
793+ }
794+ return 0;
795+}
796+
797+static int dr_ste_build_eth_l2_src_des_bit_mask(struct mlx5dr_match_param *value,
798+ bool inner, u8 *bit_mask)
799+{
800+ struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
801+
802+ DR_STE_SET_MASK_V(eth_l2_src_dst, bit_mask, dmac_47_16, mask, dmac_47_16);
803+ DR_STE_SET_MASK_V(eth_l2_src_dst, bit_mask, dmac_15_0, mask, dmac_15_0);
804+
805+ if (mask->smac_47_16 || mask->smac_15_0) {
806+ MLX5_SET(ste_eth_l2_src_dst, bit_mask, smac_47_32,
807+ mask->smac_47_16 >> 16);
808+ MLX5_SET(ste_eth_l2_src_dst, bit_mask, smac_31_0,
809+ mask->smac_47_16 << 16 | mask->smac_15_0);
810+ mask->smac_47_16 = 0;
811+ mask->smac_15_0 = 0;
812+ }
813+
814+ DR_STE_SET_MASK_V(eth_l2_src_dst, bit_mask, first_vlan_id, mask, first_vid);
815+ DR_STE_SET_MASK_V(eth_l2_src_dst, bit_mask, first_cfi, mask, first_cfi);
816+ DR_STE_SET_MASK_V(eth_l2_src_dst, bit_mask, first_priority, mask, first_prio);
817+ DR_STE_SET_MASK(eth_l2_src_dst, bit_mask, l3_type, mask, ip_version);
818+
819+ if (mask->cvlan_tag) {
820+ MLX5_SET(ste_eth_l2_src_dst, bit_mask, first_vlan_qualifier, -1);
821+ mask->cvlan_tag = 0;
822+ } else if (mask->svlan_tag) {
823+ MLX5_SET(ste_eth_l2_src_dst, bit_mask, first_vlan_qualifier, -1);
824+ mask->svlan_tag = 0;
825+ }
826+
827+ if (mask->cvlan_tag || mask->svlan_tag) {
828+ pr_info("Invalid c/svlan mask configuration\n");
829+ return -EINVAL;
830+ }
831+
832+ return 0;
833+}
834+
835+static void dr_ste_copy_mask_misc(char *mask, struct mlx5dr_match_misc *spec)
836+{
837+ spec->gre_c_present = MLX5_GET(fte_match_set_misc, mask, gre_c_present);
838+ spec->gre_k_present = MLX5_GET(fte_match_set_misc, mask, gre_k_present);
839+ spec->gre_s_present = MLX5_GET(fte_match_set_misc, mask, gre_s_present);
840+ spec->source_vhca_port = MLX5_GET(fte_match_set_misc, mask, source_vhca_port);
841+ spec->source_sqn = MLX5_GET(fte_match_set_misc, mask, source_sqn);
842+
843+ spec->source_port = MLX5_GET(fte_match_set_misc, mask, source_port);
844+
845+ spec->outer_second_prio = MLX5_GET(fte_match_set_misc, mask, outer_second_prio);
846+ spec->outer_second_cfi = MLX5_GET(fte_match_set_misc, mask, outer_second_cfi);
847+ spec->outer_second_vid = MLX5_GET(fte_match_set_misc, mask, outer_second_vid);
848+ spec->inner_second_prio = MLX5_GET(fte_match_set_misc, mask, inner_second_prio);
849+ spec->inner_second_cfi = MLX5_GET(fte_match_set_misc, mask, inner_second_cfi);
850+ spec->inner_second_vid = MLX5_GET(fte_match_set_misc, mask, inner_second_vid);
851+
852+ spec->outer_second_cvlan_tag =
853+ MLX5_GET(fte_match_set_misc, mask, outer_second_cvlan_tag);
854+ spec->inner_second_cvlan_tag =
855+ MLX5_GET(fte_match_set_misc, mask, inner_second_cvlan_tag);
856+ spec->outer_second_svlan_tag =
857+ MLX5_GET(fte_match_set_misc, mask, outer_second_svlan_tag);
858+ spec->inner_second_svlan_tag =
859+ MLX5_GET(fte_match_set_misc, mask, inner_second_svlan_tag);
860+
861+ spec->gre_protocol = MLX5_GET(fte_match_set_misc, mask, gre_protocol);
862+
863+ spec->gre_key_h = MLX5_GET(fte_match_set_misc, mask, gre_key.nvgre.hi);
864+ spec->gre_key_l = MLX5_GET(fte_match_set_misc, mask, gre_key.nvgre.lo);
865+
866+ spec->vxlan_vni = MLX5_GET(fte_match_set_misc, mask, vxlan_vni);
867+
868+ spec->geneve_vni = MLX5_GET(fte_match_set_misc, mask, geneve_vni);
869+ spec->geneve_oam = MLX5_GET(fte_match_set_misc, mask, geneve_oam);
870+
871+ spec->outer_ipv6_flow_label =
872+ MLX5_GET(fte_match_set_misc, mask, outer_ipv6_flow_label);
873+
874+ spec->inner_ipv6_flow_label =
875+ MLX5_GET(fte_match_set_misc, mask, inner_ipv6_flow_label);
876+
877+ spec->geneve_opt_len = MLX5_GET(fte_match_set_misc, mask, geneve_opt_len);
878+ spec->geneve_protocol_type =
879+ MLX5_GET(fte_match_set_misc, mask, geneve_protocol_type);
880+
881+ spec->bth_dst_qp = MLX5_GET(fte_match_set_misc, mask, bth_dst_qp);
882+}
883+
884+static void dr_ste_copy_mask_spec(char *mask, struct mlx5dr_match_spec *spec)
885+{
886+ u32 raw_ip[4];
887+
888+ spec->smac_47_16 = MLX5_GET(fte_match_set_lyr_2_4, mask, smac_47_16);
889+
890+ spec->smac_15_0 = MLX5_GET(fte_match_set_lyr_2_4, mask, smac_15_0);
891+ spec->ethertype = MLX5_GET(fte_match_set_lyr_2_4, mask, ethertype);
892+
893+ spec->dmac_47_16 = MLX5_GET(fte_match_set_lyr_2_4, mask, dmac_47_16);
894+
895+ spec->dmac_15_0 = MLX5_GET(fte_match_set_lyr_2_4, mask, dmac_15_0);
896+ spec->first_prio = MLX5_GET(fte_match_set_lyr_2_4, mask, first_prio);
897+ spec->first_cfi = MLX5_GET(fte_match_set_lyr_2_4, mask, first_cfi);
898+ spec->first_vid = MLX5_GET(fte_match_set_lyr_2_4, mask, first_vid);
899+
900+ spec->ip_protocol = MLX5_GET(fte_match_set_lyr_2_4, mask, ip_protocol);
901+ spec->ip_dscp = MLX5_GET(fte_match_set_lyr_2_4, mask, ip_dscp);
902+ spec->ip_ecn = MLX5_GET(fte_match_set_lyr_2_4, mask, ip_ecn);
903+ spec->cvlan_tag = MLX5_GET(fte_match_set_lyr_2_4, mask, cvlan_tag);
904+ spec->svlan_tag = MLX5_GET(fte_match_set_lyr_2_4, mask, svlan_tag);
905+ spec->frag = MLX5_GET(fte_match_set_lyr_2_4, mask, frag);
906+ spec->ip_version = MLX5_GET(fte_match_set_lyr_2_4, mask, ip_version);
907+ spec->tcp_flags = MLX5_GET(fte_match_set_lyr_2_4, mask, tcp_flags);
908+ spec->tcp_sport = MLX5_GET(fte_match_set_lyr_2_4, mask, tcp_sport);
909+ spec->tcp_dport = MLX5_GET(fte_match_set_lyr_2_4, mask, tcp_dport);
910+
911+ spec->ttl_hoplimit = MLX5_GET(fte_match_set_lyr_2_4, mask, ttl_hoplimit);
912+
913+ spec->udp_sport = MLX5_GET(fte_match_set_lyr_2_4, mask, udp_sport);
914+ spec->udp_dport = MLX5_GET(fte_match_set_lyr_2_4, mask, udp_dport);
915+
916+ memcpy(raw_ip, MLX5_ADDR_OF(fte_match_set_lyr_2_4, mask,
917+ src_ipv4_src_ipv6.ipv6_layout.ipv6),
918+ sizeof(raw_ip));
919+
920+ spec->src_ip_127_96 = be32_to_cpu(raw_ip[0]);
921+ spec->src_ip_95_64 = be32_to_cpu(raw_ip[1]);
922+ spec->src_ip_63_32 = be32_to_cpu(raw_ip[2]);
923+ spec->src_ip_31_0 = be32_to_cpu(raw_ip[3]);
924+
925+ memcpy(raw_ip, MLX5_ADDR_OF(fte_match_set_lyr_2_4, mask,
926+ dst_ipv4_dst_ipv6.ipv6_layout.ipv6),
927+ sizeof(raw_ip));
928+
929+ spec->dst_ip_127_96 = be32_to_cpu(raw_ip[0]);
930+ spec->dst_ip_95_64 = be32_to_cpu(raw_ip[1]);
931+ spec->dst_ip_63_32 = be32_to_cpu(raw_ip[2]);
932+ spec->dst_ip_31_0 = be32_to_cpu(raw_ip[3]);
933+}
934+
935+static void dr_ste_copy_mask_misc2(char *mask, struct mlx5dr_match_misc2 *spec)
936+{
937+ spec->outer_first_mpls_label =
938+ MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls.mpls_label);
939+ spec->outer_first_mpls_exp =
940+ MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls.mpls_exp);
941+ spec->outer_first_mpls_s_bos =
942+ MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls.mpls_s_bos);
943+ spec->outer_first_mpls_ttl =
944+ MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls.mpls_ttl);
945+ spec->inner_first_mpls_label =
946+ MLX5_GET(fte_match_set_misc2, mask, inner_first_mpls.mpls_label);
947+ spec->inner_first_mpls_exp =
948+ MLX5_GET(fte_match_set_misc2, mask, inner_first_mpls.mpls_exp);
949+ spec->inner_first_mpls_s_bos =
950+ MLX5_GET(fte_match_set_misc2, mask, inner_first_mpls.mpls_s_bos);
951+ spec->inner_first_mpls_ttl =
952+ MLX5_GET(fte_match_set_misc2, mask, inner_first_mpls.mpls_ttl);
953+ spec->outer_first_mpls_over_gre_label =
954+ MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_gre.mpls_label);
955+ spec->outer_first_mpls_over_gre_exp =
956+ MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_gre.mpls_exp);
957+ spec->outer_first_mpls_over_gre_s_bos =
958+ MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_gre.mpls_s_bos);
959+ spec->outer_first_mpls_over_gre_ttl =
960+ MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_gre.mpls_ttl);
961+ spec->outer_first_mpls_over_udp_label =
962+ MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_udp.mpls_label);
963+ spec->outer_first_mpls_over_udp_exp =
964+ MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_udp.mpls_exp);
965+ spec->outer_first_mpls_over_udp_s_bos =
966+ MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_udp.mpls_s_bos);
967+ spec->outer_first_mpls_over_udp_ttl =
968+ MLX5_GET(fte_match_set_misc2, mask, outer_first_mpls_over_udp.mpls_ttl);
969+ spec->metadata_reg_c_7 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_7);
970+ spec->metadata_reg_c_6 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_6);
971+ spec->metadata_reg_c_5 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_5);
972+ spec->metadata_reg_c_4 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_4);
973+ spec->metadata_reg_c_3 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_3);
974+ spec->metadata_reg_c_2 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_2);
975+ spec->metadata_reg_c_1 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_1);
976+ spec->metadata_reg_c_0 = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_c_0);
977+ spec->metadata_reg_a = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_a);
978+ spec->metadata_reg_b = MLX5_GET(fte_match_set_misc2, mask, metadata_reg_b);
979+}
980+
981+static void dr_ste_copy_mask_misc3(char *mask, struct mlx5dr_match_misc3 *spec)
982+{
983+ spec->inner_tcp_seq_num = MLX5_GET(fte_match_set_misc3, mask, inner_tcp_seq_num);
984+ spec->outer_tcp_seq_num = MLX5_GET(fte_match_set_misc3, mask, outer_tcp_seq_num);
985+ spec->inner_tcp_ack_num = MLX5_GET(fte_match_set_misc3, mask, inner_tcp_ack_num);
986+ spec->outer_tcp_ack_num = MLX5_GET(fte_match_set_misc3, mask, outer_tcp_ack_num);
987+ spec->outer_vxlan_gpe_vni =
988+ MLX5_GET(fte_match_set_misc3, mask, outer_vxlan_gpe_vni);
989+ spec->outer_vxlan_gpe_next_protocol =
990+ MLX5_GET(fte_match_set_misc3, mask, outer_vxlan_gpe_next_protocol);
991+ spec->outer_vxlan_gpe_flags =
992+ MLX5_GET(fte_match_set_misc3, mask, outer_vxlan_gpe_flags);
993+ spec->icmpv4_header_data = MLX5_GET(fte_match_set_misc3, mask, icmp_header_data);
994+ spec->icmpv6_header_data =
995+ MLX5_GET(fte_match_set_misc3, mask, icmpv6_header_data);
996+ spec->icmpv4_type = MLX5_GET(fte_match_set_misc3, mask, icmp_type);
997+ spec->icmpv4_code = MLX5_GET(fte_match_set_misc3, mask, icmp_code);
998+ spec->icmpv6_type = MLX5_GET(fte_match_set_misc3, mask, icmpv6_type);
999+ spec->icmpv6_code = MLX5_GET(fte_match_set_misc3, mask, icmpv6_code);
1000+}
1001+
1002+void mlx5dr_ste_copy_param(u8 match_criteria,
1003+ struct mlx5dr_match_param *set_param,
1004+ struct mlx5dr_match_parameters *mask)
1005+{
1006+ u8 tail_param[MLX5_ST_SZ_BYTES(fte_match_set_lyr_2_4)] = {};
1007+ u8 *data = (u8 *)mask->match_buf;
1008+ size_t param_location;
1009+ void *buff;
1010+
1011+ if (match_criteria & DR_MATCHER_CRITERIA_OUTER) {
1012+ if (mask->match_sz < sizeof(struct mlx5dr_match_spec)) {
1013+ memcpy(tail_param, data, mask->match_sz);
1014+ buff = tail_param;
1015+ } else {
1016+ buff = mask->match_buf;
1017+ }
1018+ dr_ste_copy_mask_spec(buff, &set_param->outer);
1019+ }
1020+ param_location = sizeof(struct mlx5dr_match_spec);
1021+
1022+ if (match_criteria & DR_MATCHER_CRITERIA_MISC) {
1023+ if (mask->match_sz < param_location +
1024+ sizeof(struct mlx5dr_match_misc)) {
1025+ memcpy(tail_param, data + param_location,
1026+ mask->match_sz - param_location);
1027+ buff = tail_param;
1028+ } else {
1029+ buff = data + param_location;
1030+ }
1031+ dr_ste_copy_mask_misc(buff, &set_param->misc);
1032+ }
1033+ param_location += sizeof(struct mlx5dr_match_misc);
1034+
1035+ if (match_criteria & DR_MATCHER_CRITERIA_INNER) {
1036+ if (mask->match_sz < param_location +
1037+ sizeof(struct mlx5dr_match_spec)) {
1038+ memcpy(tail_param, data + param_location,
1039+ mask->match_sz - param_location);
1040+ buff = tail_param;
1041+ } else {
1042+ buff = data + param_location;
1043+ }
1044+ dr_ste_copy_mask_spec(buff, &set_param->inner);
1045+ }
1046+ param_location += sizeof(struct mlx5dr_match_spec);
1047+
1048+ if (match_criteria & DR_MATCHER_CRITERIA_MISC2) {
1049+ if (mask->match_sz < param_location +
1050+ sizeof(struct mlx5dr_match_misc2)) {
1051+ memcpy(tail_param, data + param_location,
1052+ mask->match_sz - param_location);
1053+ buff = tail_param;
1054+ } else {
1055+ buff = data + param_location;
1056+ }
1057+ dr_ste_copy_mask_misc2(buff, &set_param->misc2);
1058+ }
1059+
1060+ param_location += sizeof(struct mlx5dr_match_misc2);
1061+
1062+ if (match_criteria & DR_MATCHER_CRITERIA_MISC3) {
1063+ if (mask->match_sz < param_location +
1064+ sizeof(struct mlx5dr_match_misc3)) {
1065+ memcpy(tail_param, data + param_location,
1066+ mask->match_sz - param_location);
1067+ buff = tail_param;
1068+ } else {
1069+ buff = data + param_location;
1070+ }
1071+ dr_ste_copy_mask_misc3(buff, &set_param->misc3);
1072+ }
1073+}
1074+
1075+static int dr_ste_build_eth_l2_src_des_tag(struct mlx5dr_match_param *value,
1076+ struct mlx5dr_ste_build *sb,
1077+ u8 *hw_ste_p)
1078+{
1079+ struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1080+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1081+ u8 *tag = hw_ste->tag;
1082+
1083+ DR_STE_SET_TAG(eth_l2_src_dst, tag, dmac_47_16, spec, dmac_47_16);
1084+ DR_STE_SET_TAG(eth_l2_src_dst, tag, dmac_15_0, spec, dmac_15_0);
1085+
1086+ if (spec->smac_47_16 || spec->smac_15_0) {
1087+ MLX5_SET(ste_eth_l2_src_dst, tag, smac_47_32,
1088+ spec->smac_47_16 >> 16);
1089+ MLX5_SET(ste_eth_l2_src_dst, tag, smac_31_0,
1090+ spec->smac_47_16 << 16 | spec->smac_15_0);
1091+ spec->smac_47_16 = 0;
1092+ spec->smac_15_0 = 0;
1093+ }
1094+
1095+ if (spec->ip_version) {
1096+ if (spec->ip_version == IP_VERSION_IPV4) {
1097+ MLX5_SET(ste_eth_l2_src_dst, tag, l3_type, STE_IPV4);
1098+ spec->ip_version = 0;
1099+ } else if (spec->ip_version == IP_VERSION_IPV6) {
1100+ MLX5_SET(ste_eth_l2_src_dst, tag, l3_type, STE_IPV6);
1101+ spec->ip_version = 0;
1102+ } else {
1103+ pr_info("Unsupported ip_version value\n");
1104+ return -EINVAL;
1105+ }
1106+ }
1107+
1108+ DR_STE_SET_TAG(eth_l2_src_dst, tag, first_vlan_id, spec, first_vid);
1109+ DR_STE_SET_TAG(eth_l2_src_dst, tag, first_cfi, spec, first_cfi);
1110+ DR_STE_SET_TAG(eth_l2_src_dst, tag, first_priority, spec, first_prio);
1111+
1112+ if (spec->cvlan_tag) {
1113+ MLX5_SET(ste_eth_l2_src_dst, tag, first_vlan_qualifier, DR_STE_CVLAN);
1114+ spec->cvlan_tag = 0;
1115+ } else if (spec->svlan_tag) {
1116+ MLX5_SET(ste_eth_l2_src_dst, tag, first_vlan_qualifier, DR_STE_SVLAN);
1117+ spec->svlan_tag = 0;
1118+ }
1119+ return 0;
1120+}
1121+
1122+int mlx5dr_ste_build_eth_l2_src_des(struct mlx5dr_ste_build *sb,
1123+ struct mlx5dr_match_param *mask,
1124+ bool inner, bool rx)
1125+{
1126+ int ret;
1127+
1128+ ret = dr_ste_build_eth_l2_src_des_bit_mask(mask, inner, sb->bit_mask);
1129+ if (ret)
1130+ return ret;
1131+
1132+ sb->rx = rx;
1133+ sb->inner = inner;
1134+ sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL2_SRC_DST, rx, inner);
1135+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1136+ sb->ste_build_tag_func = &dr_ste_build_eth_l2_src_des_tag;
1137+
1138+ return 0;
1139+}
1140+
1141+static void dr_ste_build_eth_l3_ipv6_dst_bit_mask(struct mlx5dr_match_param *value,
1142+ bool inner, u8 *bit_mask)
1143+{
1144+ struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1145+
1146+ DR_STE_SET_MASK_V(eth_l3_ipv6_dst, bit_mask, dst_ip_127_96, mask, dst_ip_127_96);
1147+ DR_STE_SET_MASK_V(eth_l3_ipv6_dst, bit_mask, dst_ip_95_64, mask, dst_ip_95_64);
1148+ DR_STE_SET_MASK_V(eth_l3_ipv6_dst, bit_mask, dst_ip_63_32, mask, dst_ip_63_32);
1149+ DR_STE_SET_MASK_V(eth_l3_ipv6_dst, bit_mask, dst_ip_31_0, mask, dst_ip_31_0);
1150+}
1151+
1152+static int dr_ste_build_eth_l3_ipv6_dst_tag(struct mlx5dr_match_param *value,
1153+ struct mlx5dr_ste_build *sb,
1154+ u8 *hw_ste_p)
1155+{
1156+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1157+ struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1158+ u8 *tag = hw_ste->tag;
1159+
1160+ DR_STE_SET_TAG(eth_l3_ipv6_dst, tag, dst_ip_127_96, spec, dst_ip_127_96);
1161+ DR_STE_SET_TAG(eth_l3_ipv6_dst, tag, dst_ip_95_64, spec, dst_ip_95_64);
1162+ DR_STE_SET_TAG(eth_l3_ipv6_dst, tag, dst_ip_63_32, spec, dst_ip_63_32);
1163+ DR_STE_SET_TAG(eth_l3_ipv6_dst, tag, dst_ip_31_0, spec, dst_ip_31_0);
1164+
1165+ return 0;
1166+}
1167+
1168+void mlx5dr_ste_build_eth_l3_ipv6_dst(struct mlx5dr_ste_build *sb,
1169+ struct mlx5dr_match_param *mask,
1170+ bool inner, bool rx)
1171+{
1172+ dr_ste_build_eth_l3_ipv6_dst_bit_mask(mask, inner, sb->bit_mask);
1173+
1174+ sb->rx = rx;
1175+ sb->inner = inner;
1176+ sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL3_IPV6_DST, rx, inner);
1177+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1178+ sb->ste_build_tag_func = &dr_ste_build_eth_l3_ipv6_dst_tag;
1179+}
1180+
1181+static void dr_ste_build_eth_l3_ipv6_src_bit_mask(struct mlx5dr_match_param *value,
1182+ bool inner, u8 *bit_mask)
1183+{
1184+ struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1185+
1186+ DR_STE_SET_MASK_V(eth_l3_ipv6_src, bit_mask, src_ip_127_96, mask, src_ip_127_96);
1187+ DR_STE_SET_MASK_V(eth_l3_ipv6_src, bit_mask, src_ip_95_64, mask, src_ip_95_64);
1188+ DR_STE_SET_MASK_V(eth_l3_ipv6_src, bit_mask, src_ip_63_32, mask, src_ip_63_32);
1189+ DR_STE_SET_MASK_V(eth_l3_ipv6_src, bit_mask, src_ip_31_0, mask, src_ip_31_0);
1190+}
1191+
1192+static int dr_ste_build_eth_l3_ipv6_src_tag(struct mlx5dr_match_param *value,
1193+ struct mlx5dr_ste_build *sb,
1194+ u8 *hw_ste_p)
1195+{
1196+ struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1197+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1198+ u8 *tag = hw_ste->tag;
1199+
1200+ DR_STE_SET_TAG(eth_l3_ipv6_src, tag, src_ip_127_96, spec, src_ip_127_96);
1201+ DR_STE_SET_TAG(eth_l3_ipv6_src, tag, src_ip_95_64, spec, src_ip_95_64);
1202+ DR_STE_SET_TAG(eth_l3_ipv6_src, tag, src_ip_63_32, spec, src_ip_63_32);
1203+ DR_STE_SET_TAG(eth_l3_ipv6_src, tag, src_ip_31_0, spec, src_ip_31_0);
1204+
1205+ return 0;
1206+}
1207+
1208+void mlx5dr_ste_build_eth_l3_ipv6_src(struct mlx5dr_ste_build *sb,
1209+ struct mlx5dr_match_param *mask,
1210+ bool inner, bool rx)
1211+{
1212+ dr_ste_build_eth_l3_ipv6_src_bit_mask(mask, inner, sb->bit_mask);
1213+
1214+ sb->rx = rx;
1215+ sb->inner = inner;
1216+ sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL3_IPV6_SRC, rx, inner);
1217+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1218+ sb->ste_build_tag_func = &dr_ste_build_eth_l3_ipv6_src_tag;
1219+}
1220+
1221+static void dr_ste_build_eth_l3_ipv4_5_tuple_bit_mask(struct mlx5dr_match_param *value,
1222+ bool inner,
1223+ u8 *bit_mask)
1224+{
1225+ struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1226+
1227+ DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1228+ destination_address, mask, dst_ip_31_0);
1229+ DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1230+ source_address, mask, src_ip_31_0);
1231+ DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1232+ destination_port, mask, tcp_dport);
1233+ DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1234+ destination_port, mask, udp_dport);
1235+ DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1236+ source_port, mask, tcp_sport);
1237+ DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1238+ source_port, mask, udp_sport);
1239+ DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1240+ protocol, mask, ip_protocol);
1241+ DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1242+ fragmented, mask, frag);
1243+ DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1244+ dscp, mask, ip_dscp);
1245+ DR_STE_SET_MASK_V(eth_l3_ipv4_5_tuple, bit_mask,
1246+ ecn, mask, ip_ecn);
1247+
1248+ if (mask->tcp_flags) {
1249+ DR_STE_SET_TCP_FLAGS(eth_l3_ipv4_5_tuple, bit_mask, mask);
1250+ mask->tcp_flags = 0;
1251+ }
1252+}
1253+
1254+static int dr_ste_build_eth_l3_ipv4_5_tuple_tag(struct mlx5dr_match_param *value,
1255+ struct mlx5dr_ste_build *sb,
1256+ u8 *hw_ste_p)
1257+{
1258+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1259+ struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1260+ u8 *tag = hw_ste->tag;
1261+
1262+ DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, destination_address, spec, dst_ip_31_0);
1263+ DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, source_address, spec, src_ip_31_0);
1264+ DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, destination_port, spec, tcp_dport);
1265+ DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, destination_port, spec, udp_dport);
1266+ DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, source_port, spec, tcp_sport);
1267+ DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, source_port, spec, udp_sport);
1268+ DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, protocol, spec, ip_protocol);
1269+ DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, fragmented, spec, frag);
1270+ DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, dscp, spec, ip_dscp);
1271+ DR_STE_SET_TAG(eth_l3_ipv4_5_tuple, tag, ecn, spec, ip_ecn);
1272+
1273+ if (spec->tcp_flags) {
1274+ DR_STE_SET_TCP_FLAGS(eth_l3_ipv4_5_tuple, tag, spec);
1275+ spec->tcp_flags = 0;
1276+ }
1277+
1278+ return 0;
1279+}
1280+
1281+void mlx5dr_ste_build_eth_l3_ipv4_5_tuple(struct mlx5dr_ste_build *sb,
1282+ struct mlx5dr_match_param *mask,
1283+ bool inner, bool rx)
1284+{
1285+ dr_ste_build_eth_l3_ipv4_5_tuple_bit_mask(mask, inner, sb->bit_mask);
1286+
1287+ sb->rx = rx;
1288+ sb->inner = inner;
1289+ sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL3_IPV4_5_TUPLE, rx, inner);
1290+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1291+ sb->ste_build_tag_func = &dr_ste_build_eth_l3_ipv4_5_tuple_tag;
1292+}
1293+
1294+static void
1295+dr_ste_build_eth_l2_src_or_dst_bit_mask(struct mlx5dr_match_param *value,
1296+ bool inner, u8 *bit_mask)
1297+{
1298+ struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1299+ struct mlx5dr_match_misc *misc_mask = &value->misc;
1300+
1301+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask, first_vlan_id, mask, first_vid);
1302+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask, first_cfi, mask, first_cfi);
1303+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask, first_priority, mask, first_prio);
1304+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask, ip_fragmented, mask, frag);
1305+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask, l3_ethertype, mask, ethertype);
1306+ DR_STE_SET_MASK(eth_l2_src, bit_mask, l3_type, mask, ip_version);
1307+
1308+ if (mask->svlan_tag || mask->cvlan_tag) {
1309+ MLX5_SET(ste_eth_l2_src, bit_mask, first_vlan_qualifier, -1);
1310+ mask->cvlan_tag = 0;
1311+ mask->svlan_tag = 0;
1312+ }
1313+
1314+ if (inner) {
1315+ if (misc_mask->inner_second_cvlan_tag ||
1316+ misc_mask->inner_second_svlan_tag) {
1317+ MLX5_SET(ste_eth_l2_src, bit_mask, second_vlan_qualifier, -1);
1318+ misc_mask->inner_second_cvlan_tag = 0;
1319+ misc_mask->inner_second_svlan_tag = 0;
1320+ }
1321+
1322+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask,
1323+ second_vlan_id, misc_mask, inner_second_vid);
1324+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask,
1325+ second_cfi, misc_mask, inner_second_cfi);
1326+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask,
1327+ second_priority, misc_mask, inner_second_prio);
1328+ } else {
1329+ if (misc_mask->outer_second_cvlan_tag ||
1330+ misc_mask->outer_second_svlan_tag) {
1331+ MLX5_SET(ste_eth_l2_src, bit_mask, second_vlan_qualifier, -1);
1332+ misc_mask->outer_second_cvlan_tag = 0;
1333+ misc_mask->outer_second_svlan_tag = 0;
1334+ }
1335+
1336+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask,
1337+ second_vlan_id, misc_mask, outer_second_vid);
1338+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask,
1339+ second_cfi, misc_mask, outer_second_cfi);
1340+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask,
1341+ second_priority, misc_mask, outer_second_prio);
1342+ }
1343+}
1344+
1345+static int dr_ste_build_eth_l2_src_or_dst_tag(struct mlx5dr_match_param *value,
1346+ bool inner, u8 *hw_ste_p)
1347+{
1348+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1349+ struct mlx5dr_match_spec *spec = inner ? &value->inner : &value->outer;
1350+ struct mlx5dr_match_misc *misc_spec = &value->misc;
1351+ u8 *tag = hw_ste->tag;
1352+
1353+ DR_STE_SET_TAG(eth_l2_src, tag, first_vlan_id, spec, first_vid);
1354+ DR_STE_SET_TAG(eth_l2_src, tag, first_cfi, spec, first_cfi);
1355+ DR_STE_SET_TAG(eth_l2_src, tag, first_priority, spec, first_prio);
1356+ DR_STE_SET_TAG(eth_l2_src, tag, ip_fragmented, spec, frag);
1357+ DR_STE_SET_TAG(eth_l2_src, tag, l3_ethertype, spec, ethertype);
1358+
1359+ if (spec->ip_version) {
1360+ if (spec->ip_version == IP_VERSION_IPV4) {
1361+ MLX5_SET(ste_eth_l2_src, tag, l3_type, STE_IPV4);
1362+ spec->ip_version = 0;
1363+ } else if (spec->ip_version == IP_VERSION_IPV6) {
1364+ MLX5_SET(ste_eth_l2_src, tag, l3_type, STE_IPV6);
1365+ spec->ip_version = 0;
1366+ } else {
1367+ pr_info("Unsupported ip_version value\n");
1368+ return -EINVAL;
1369+ }
1370+ }
1371+
1372+ if (spec->cvlan_tag) {
1373+ MLX5_SET(ste_eth_l2_src, tag, first_vlan_qualifier, DR_STE_CVLAN);
1374+ spec->cvlan_tag = 0;
1375+ } else if (spec->svlan_tag) {
1376+ MLX5_SET(ste_eth_l2_src, tag, first_vlan_qualifier, DR_STE_SVLAN);
1377+ spec->svlan_tag = 0;
1378+ }
1379+
1380+ if (inner) {
1381+ if (misc_spec->inner_second_cvlan_tag) {
1382+ MLX5_SET(ste_eth_l2_src, tag, second_vlan_qualifier, DR_STE_CVLAN);
1383+ misc_spec->inner_second_cvlan_tag = 0;
1384+ } else if (misc_spec->inner_second_svlan_tag) {
1385+ MLX5_SET(ste_eth_l2_src, tag, second_vlan_qualifier, DR_STE_SVLAN);
1386+ misc_spec->inner_second_svlan_tag = 0;
1387+ }
1388+
1389+ DR_STE_SET_TAG(eth_l2_src, tag, second_vlan_id, misc_spec, inner_second_vid);
1390+ DR_STE_SET_TAG(eth_l2_src, tag, second_cfi, misc_spec, inner_second_cfi);
1391+ DR_STE_SET_TAG(eth_l2_src, tag, second_priority, misc_spec, inner_second_prio);
1392+ } else {
1393+ if (misc_spec->outer_second_cvlan_tag) {
1394+ MLX5_SET(ste_eth_l2_src, tag, second_vlan_qualifier, DR_STE_CVLAN);
1395+ misc_spec->outer_second_cvlan_tag = 0;
1396+ } else if (misc_spec->outer_second_svlan_tag) {
1397+ MLX5_SET(ste_eth_l2_src, tag, second_vlan_qualifier, DR_STE_SVLAN);
1398+ misc_spec->outer_second_svlan_tag = 0;
1399+ }
1400+ DR_STE_SET_TAG(eth_l2_src, tag, second_vlan_id, misc_spec, outer_second_vid);
1401+ DR_STE_SET_TAG(eth_l2_src, tag, second_cfi, misc_spec, outer_second_cfi);
1402+ DR_STE_SET_TAG(eth_l2_src, tag, second_priority, misc_spec, outer_second_prio);
1403+ }
1404+
1405+ return 0;
1406+}
1407+
1408+static void dr_ste_build_eth_l2_src_bit_mask(struct mlx5dr_match_param *value,
1409+ bool inner, u8 *bit_mask)
1410+{
1411+ struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1412+
1413+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask, smac_47_16, mask, smac_47_16);
1414+ DR_STE_SET_MASK_V(eth_l2_src, bit_mask, smac_15_0, mask, smac_15_0);
1415+
1416+ dr_ste_build_eth_l2_src_or_dst_bit_mask(value, inner, bit_mask);
1417+}
1418+
1419+static int dr_ste_build_eth_l2_src_tag(struct mlx5dr_match_param *value,
1420+ struct mlx5dr_ste_build *sb,
1421+ u8 *hw_ste_p)
1422+{
1423+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1424+ struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1425+ u8 *tag = hw_ste->tag;
1426+
1427+ DR_STE_SET_TAG(eth_l2_src, tag, smac_47_16, spec, smac_47_16);
1428+ DR_STE_SET_TAG(eth_l2_src, tag, smac_15_0, spec, smac_15_0);
1429+
1430+ return dr_ste_build_eth_l2_src_or_dst_tag(value, sb->inner, hw_ste_p);
1431+}
1432+
1433+void mlx5dr_ste_build_eth_l2_src(struct mlx5dr_ste_build *sb,
1434+ struct mlx5dr_match_param *mask,
1435+ bool inner, bool rx)
1436+{
1437+ dr_ste_build_eth_l2_src_bit_mask(mask, inner, sb->bit_mask);
1438+ sb->rx = rx;
1439+ sb->inner = inner;
1440+ sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL2_SRC, rx, inner);
1441+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1442+ sb->ste_build_tag_func = &dr_ste_build_eth_l2_src_tag;
1443+}
1444+
1445+static void dr_ste_build_eth_l2_dst_bit_mask(struct mlx5dr_match_param *value,
1446+ bool inner, u8 *bit_mask)
1447+{
1448+ struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1449+
1450+ DR_STE_SET_MASK_V(eth_l2_dst, bit_mask, dmac_47_16, mask, dmac_47_16);
1451+ DR_STE_SET_MASK_V(eth_l2_dst, bit_mask, dmac_15_0, mask, dmac_15_0);
1452+
1453+ dr_ste_build_eth_l2_src_or_dst_bit_mask(value, inner, bit_mask);
1454+}
1455+
1456+static int dr_ste_build_eth_l2_dst_tag(struct mlx5dr_match_param *value,
1457+ struct mlx5dr_ste_build *sb,
1458+ u8 *hw_ste_p)
1459+{
1460+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1461+ struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1462+ u8 *tag = hw_ste->tag;
1463+
1464+ DR_STE_SET_TAG(eth_l2_dst, tag, dmac_47_16, spec, dmac_47_16);
1465+ DR_STE_SET_TAG(eth_l2_dst, tag, dmac_15_0, spec, dmac_15_0);
1466+
1467+ return dr_ste_build_eth_l2_src_or_dst_tag(value, sb->inner, hw_ste_p);
1468+}
1469+
1470+void mlx5dr_ste_build_eth_l2_dst(struct mlx5dr_ste_build *sb,
1471+ struct mlx5dr_match_param *mask,
1472+ bool inner, bool rx)
1473+{
1474+ dr_ste_build_eth_l2_dst_bit_mask(mask, inner, sb->bit_mask);
1475+
1476+ sb->rx = rx;
1477+ sb->inner = inner;
1478+ sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL2_DST, rx, inner);
1479+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1480+ sb->ste_build_tag_func = &dr_ste_build_eth_l2_dst_tag;
1481+}
1482+
1483+static void dr_ste_build_eth_l2_tnl_bit_mask(struct mlx5dr_match_param *value,
1484+ bool inner, u8 *bit_mask)
1485+{
1486+ struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1487+ struct mlx5dr_match_misc *misc = &value->misc;
1488+
1489+ DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, dmac_47_16, mask, dmac_47_16);
1490+ DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, dmac_15_0, mask, dmac_15_0);
1491+ DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, first_vlan_id, mask, first_vid);
1492+ DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, first_cfi, mask, first_cfi);
1493+ DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, first_priority, mask, first_prio);
1494+ DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, ip_fragmented, mask, frag);
1495+ DR_STE_SET_MASK_V(eth_l2_tnl, bit_mask, l3_ethertype, mask, ethertype);
1496+ DR_STE_SET_MASK(eth_l2_tnl, bit_mask, l3_type, mask, ip_version);
1497+
1498+ if (misc->vxlan_vni) {
1499+ MLX5_SET(ste_eth_l2_tnl, bit_mask,
1500+ l2_tunneling_network_id, (misc->vxlan_vni << 8));
1501+ misc->vxlan_vni = 0;
1502+ }
1503+
1504+ if (mask->svlan_tag || mask->cvlan_tag) {
1505+ MLX5_SET(ste_eth_l2_tnl, bit_mask, first_vlan_qualifier, -1);
1506+ mask->cvlan_tag = 0;
1507+ mask->svlan_tag = 0;
1508+ }
1509+}
1510+
1511+static int dr_ste_build_eth_l2_tnl_tag(struct mlx5dr_match_param *value,
1512+ struct mlx5dr_ste_build *sb,
1513+ u8 *hw_ste_p)
1514+{
1515+ struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1516+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1517+ struct mlx5dr_match_misc *misc = &value->misc;
1518+ u8 *tag = hw_ste->tag;
1519+
1520+ DR_STE_SET_TAG(eth_l2_tnl, tag, dmac_47_16, spec, dmac_47_16);
1521+ DR_STE_SET_TAG(eth_l2_tnl, tag, dmac_15_0, spec, dmac_15_0);
1522+ DR_STE_SET_TAG(eth_l2_tnl, tag, first_vlan_id, spec, first_vid);
1523+ DR_STE_SET_TAG(eth_l2_tnl, tag, first_cfi, spec, first_cfi);
1524+ DR_STE_SET_TAG(eth_l2_tnl, tag, ip_fragmented, spec, frag);
1525+ DR_STE_SET_TAG(eth_l2_tnl, tag, first_priority, spec, first_prio);
1526+ DR_STE_SET_TAG(eth_l2_tnl, tag, l3_ethertype, spec, ethertype);
1527+
1528+ if (misc->vxlan_vni) {
1529+ MLX5_SET(ste_eth_l2_tnl, tag, l2_tunneling_network_id,
1530+ (misc->vxlan_vni << 8));
1531+ misc->vxlan_vni = 0;
1532+ }
1533+
1534+ if (spec->cvlan_tag) {
1535+ MLX5_SET(ste_eth_l2_tnl, tag, first_vlan_qualifier, DR_STE_CVLAN);
1536+ spec->cvlan_tag = 0;
1537+ } else if (spec->svlan_tag) {
1538+ MLX5_SET(ste_eth_l2_tnl, tag, first_vlan_qualifier, DR_STE_SVLAN);
1539+ spec->svlan_tag = 0;
1540+ }
1541+
1542+ if (spec->ip_version) {
1543+ if (spec->ip_version == IP_VERSION_IPV4) {
1544+ MLX5_SET(ste_eth_l2_tnl, tag, l3_type, STE_IPV4);
1545+ spec->ip_version = 0;
1546+ } else if (spec->ip_version == IP_VERSION_IPV6) {
1547+ MLX5_SET(ste_eth_l2_tnl, tag, l3_type, STE_IPV6);
1548+ spec->ip_version = 0;
1549+ } else {
1550+ return -EINVAL;
1551+ }
1552+ }
1553+
1554+ return 0;
1555+}
1556+
1557+void mlx5dr_ste_build_eth_l2_tnl(struct mlx5dr_ste_build *sb,
1558+ struct mlx5dr_match_param *mask, bool inner, bool rx)
1559+{
1560+ dr_ste_build_eth_l2_tnl_bit_mask(mask, inner, sb->bit_mask);
1561+
1562+ sb->rx = rx;
1563+ sb->inner = inner;
1564+ sb->lu_type = MLX5DR_STE_LU_TYPE_ETHL2_TUNNELING_I;
1565+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1566+ sb->ste_build_tag_func = &dr_ste_build_eth_l2_tnl_tag;
1567+}
1568+
1569+static void dr_ste_build_eth_l3_ipv4_misc_bit_mask(struct mlx5dr_match_param *value,
1570+ bool inner, u8 *bit_mask)
1571+{
1572+ struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1573+
1574+ DR_STE_SET_MASK_V(eth_l3_ipv4_misc, bit_mask, time_to_live, mask, ttl_hoplimit);
1575+}
1576+
1577+static int dr_ste_build_eth_l3_ipv4_misc_tag(struct mlx5dr_match_param *value,
1578+ struct mlx5dr_ste_build *sb,
1579+ u8 *hw_ste_p)
1580+{
1581+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1582+ struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1583+ u8 *tag = hw_ste->tag;
1584+
1585+ DR_STE_SET_TAG(eth_l3_ipv4_misc, tag, time_to_live, spec, ttl_hoplimit);
1586+
1587+ return 0;
1588+}
1589+
1590+void mlx5dr_ste_build_eth_l3_ipv4_misc(struct mlx5dr_ste_build *sb,
1591+ struct mlx5dr_match_param *mask,
1592+ bool inner, bool rx)
1593+{
1594+ dr_ste_build_eth_l3_ipv4_misc_bit_mask(mask, inner, sb->bit_mask);
1595+
1596+ sb->rx = rx;
1597+ sb->inner = inner;
1598+ sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL3_IPV4_MISC, rx, inner);
1599+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1600+ sb->ste_build_tag_func = &dr_ste_build_eth_l3_ipv4_misc_tag;
1601+}
1602+
1603+static void dr_ste_build_ipv6_l3_l4_bit_mask(struct mlx5dr_match_param *value,
1604+ bool inner, u8 *bit_mask)
1605+{
1606+ struct mlx5dr_match_spec *mask = inner ? &value->inner : &value->outer;
1607+
1608+ DR_STE_SET_MASK_V(eth_l4, bit_mask, dst_port, mask, tcp_dport);
1609+ DR_STE_SET_MASK_V(eth_l4, bit_mask, src_port, mask, tcp_sport);
1610+ DR_STE_SET_MASK_V(eth_l4, bit_mask, dst_port, mask, udp_dport);
1611+ DR_STE_SET_MASK_V(eth_l4, bit_mask, src_port, mask, udp_sport);
1612+ DR_STE_SET_MASK_V(eth_l4, bit_mask, protocol, mask, ip_protocol);
1613+ DR_STE_SET_MASK_V(eth_l4, bit_mask, fragmented, mask, frag);
1614+ DR_STE_SET_MASK_V(eth_l4, bit_mask, dscp, mask, ip_dscp);
1615+ DR_STE_SET_MASK_V(eth_l4, bit_mask, ecn, mask, ip_ecn);
1616+ DR_STE_SET_MASK_V(eth_l4, bit_mask, ipv6_hop_limit, mask, ttl_hoplimit);
1617+
1618+ if (mask->tcp_flags) {
1619+ DR_STE_SET_TCP_FLAGS(eth_l4, bit_mask, mask);
1620+ mask->tcp_flags = 0;
1621+ }
1622+}
1623+
1624+static int dr_ste_build_ipv6_l3_l4_tag(struct mlx5dr_match_param *value,
1625+ struct mlx5dr_ste_build *sb,
1626+ u8 *hw_ste_p)
1627+{
1628+ struct mlx5dr_match_spec *spec = sb->inner ? &value->inner : &value->outer;
1629+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1630+ u8 *tag = hw_ste->tag;
1631+
1632+ DR_STE_SET_TAG(eth_l4, tag, dst_port, spec, tcp_dport);
1633+ DR_STE_SET_TAG(eth_l4, tag, src_port, spec, tcp_sport);
1634+ DR_STE_SET_TAG(eth_l4, tag, dst_port, spec, udp_dport);
1635+ DR_STE_SET_TAG(eth_l4, tag, src_port, spec, udp_sport);
1636+ DR_STE_SET_TAG(eth_l4, tag, protocol, spec, ip_protocol);
1637+ DR_STE_SET_TAG(eth_l4, tag, fragmented, spec, frag);
1638+ DR_STE_SET_TAG(eth_l4, tag, dscp, spec, ip_dscp);
1639+ DR_STE_SET_TAG(eth_l4, tag, ecn, spec, ip_ecn);
1640+ DR_STE_SET_TAG(eth_l4, tag, ipv6_hop_limit, spec, ttl_hoplimit);
1641+
1642+ if (spec->tcp_flags) {
1643+ DR_STE_SET_TCP_FLAGS(eth_l4, tag, spec);
1644+ spec->tcp_flags = 0;
1645+ }
1646+
1647+ return 0;
1648+}
1649+
1650+void mlx5dr_ste_build_ipv6_l3_l4(struct mlx5dr_ste_build *sb,
1651+ struct mlx5dr_match_param *mask,
1652+ bool inner, bool rx)
1653+{
1654+ dr_ste_build_ipv6_l3_l4_bit_mask(mask, inner, sb->bit_mask);
1655+
1656+ sb->rx = rx;
1657+ sb->inner = inner;
1658+ sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL4, rx, inner);
1659+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1660+ sb->ste_build_tag_func = &dr_ste_build_ipv6_l3_l4_tag;
1661+}
1662+
1663+static int dr_ste_build_empty_always_hit_tag(struct mlx5dr_match_param *value,
1664+ struct mlx5dr_ste_build *sb,
1665+ u8 *hw_ste_p)
1666+{
1667+ return 0;
1668+}
1669+
1670+void mlx5dr_ste_build_empty_always_hit(struct mlx5dr_ste_build *sb, bool rx)
1671+{
1672+ sb->rx = rx;
1673+ sb->lu_type = MLX5DR_STE_LU_TYPE_DONT_CARE;
1674+ sb->byte_mask = 0;
1675+ sb->ste_build_tag_func = &dr_ste_build_empty_always_hit_tag;
1676+}
1677+
1678+static void dr_ste_build_mpls_bit_mask(struct mlx5dr_match_param *value,
1679+ bool inner, u8 *bit_mask)
1680+{
1681+ struct mlx5dr_match_misc2 *misc2_mask = &value->misc2;
1682+
1683+ if (inner)
1684+ DR_STE_SET_MPLS_MASK(mpls, misc2_mask, inner, bit_mask);
1685+ else
1686+ DR_STE_SET_MPLS_MASK(mpls, misc2_mask, outer, bit_mask);
1687+}
1688+
1689+static int dr_ste_build_mpls_tag(struct mlx5dr_match_param *value,
1690+ struct mlx5dr_ste_build *sb,
1691+ u8 *hw_ste_p)
1692+{
1693+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1694+ struct mlx5dr_match_misc2 *misc2_mask = &value->misc2;
1695+ u8 *tag = hw_ste->tag;
1696+
1697+ if (sb->inner)
1698+ DR_STE_SET_MPLS_TAG(mpls, misc2_mask, inner, tag);
1699+ else
1700+ DR_STE_SET_MPLS_TAG(mpls, misc2_mask, outer, tag);
1701+
1702+ return 0;
1703+}
1704+
1705+void mlx5dr_ste_build_mpls(struct mlx5dr_ste_build *sb,
1706+ struct mlx5dr_match_param *mask,
1707+ bool inner, bool rx)
1708+{
1709+ dr_ste_build_mpls_bit_mask(mask, inner, sb->bit_mask);
1710+
1711+ sb->rx = rx;
1712+ sb->inner = inner;
1713+ sb->lu_type = DR_STE_CALC_LU_TYPE(MPLS_FIRST, rx, inner);
1714+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1715+ sb->ste_build_tag_func = &dr_ste_build_mpls_tag;
1716+}
1717+
1718+static void dr_ste_build_gre_bit_mask(struct mlx5dr_match_param *value,
1719+ bool inner, u8 *bit_mask)
1720+{
1721+ struct mlx5dr_match_misc *misc_mask = &value->misc;
1722+
1723+ DR_STE_SET_MASK_V(gre, bit_mask, gre_protocol, misc_mask, gre_protocol);
1724+ DR_STE_SET_MASK_V(gre, bit_mask, gre_k_present, misc_mask, gre_k_present);
1725+ DR_STE_SET_MASK_V(gre, bit_mask, gre_key_h, misc_mask, gre_key_h);
1726+ DR_STE_SET_MASK_V(gre, bit_mask, gre_key_l, misc_mask, gre_key_l);
1727+
1728+ DR_STE_SET_MASK_V(gre, bit_mask, gre_c_present, misc_mask, gre_c_present);
1729+ DR_STE_SET_MASK_V(gre, bit_mask, gre_s_present, misc_mask, gre_s_present);
1730+}
1731+
1732+static int dr_ste_build_gre_tag(struct mlx5dr_match_param *value,
1733+ struct mlx5dr_ste_build *sb,
1734+ u8 *hw_ste_p)
1735+{
1736+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1737+ struct mlx5dr_match_misc *misc = &value->misc;
1738+ u8 *tag = hw_ste->tag;
1739+
1740+ DR_STE_SET_TAG(gre, tag, gre_protocol, misc, gre_protocol);
1741+
1742+ DR_STE_SET_TAG(gre, tag, gre_k_present, misc, gre_k_present);
1743+ DR_STE_SET_TAG(gre, tag, gre_key_h, misc, gre_key_h);
1744+ DR_STE_SET_TAG(gre, tag, gre_key_l, misc, gre_key_l);
1745+
1746+ DR_STE_SET_TAG(gre, tag, gre_c_present, misc, gre_c_present);
1747+
1748+ DR_STE_SET_TAG(gre, tag, gre_s_present, misc, gre_s_present);
1749+
1750+ return 0;
1751+}
1752+
1753+void mlx5dr_ste_build_gre(struct mlx5dr_ste_build *sb,
1754+ struct mlx5dr_match_param *mask, bool inner, bool rx)
1755+{
1756+ dr_ste_build_gre_bit_mask(mask, inner, sb->bit_mask);
1757+
1758+ sb->rx = rx;
1759+ sb->inner = inner;
1760+ sb->lu_type = MLX5DR_STE_LU_TYPE_GRE;
1761+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1762+ sb->ste_build_tag_func = &dr_ste_build_gre_tag;
1763+}
1764+
1765+static void dr_ste_build_flex_parser_0_bit_mask(struct mlx5dr_match_param *value,
1766+ bool inner, u8 *bit_mask)
1767+{
1768+ struct mlx5dr_match_misc2 *misc_2_mask = &value->misc2;
1769+
1770+ if (DR_STE_IS_OUTER_MPLS_OVER_GRE_SET(misc_2_mask)) {
1771+ DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_label,
1772+ misc_2_mask, outer_first_mpls_over_gre_label);
1773+
1774+ DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_exp,
1775+ misc_2_mask, outer_first_mpls_over_gre_exp);
1776+
1777+ DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_s_bos,
1778+ misc_2_mask, outer_first_mpls_over_gre_s_bos);
1779+
1780+ DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_ttl,
1781+ misc_2_mask, outer_first_mpls_over_gre_ttl);
1782+ } else {
1783+ DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_label,
1784+ misc_2_mask, outer_first_mpls_over_udp_label);
1785+
1786+ DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_exp,
1787+ misc_2_mask, outer_first_mpls_over_udp_exp);
1788+
1789+ DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_s_bos,
1790+ misc_2_mask, outer_first_mpls_over_udp_s_bos);
1791+
1792+ DR_STE_SET_MASK_V(flex_parser_0, bit_mask, parser_3_ttl,
1793+ misc_2_mask, outer_first_mpls_over_udp_ttl);
1794+ }
1795+}
1796+
1797+static int dr_ste_build_flex_parser_0_tag(struct mlx5dr_match_param *value,
1798+ struct mlx5dr_ste_build *sb,
1799+ u8 *hw_ste_p)
1800+{
1801+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1802+ struct mlx5dr_match_misc2 *misc_2_mask = &value->misc2;
1803+ u8 *tag = hw_ste->tag;
1804+
1805+ if (DR_STE_IS_OUTER_MPLS_OVER_GRE_SET(misc_2_mask)) {
1806+ DR_STE_SET_TAG(flex_parser_0, tag, parser_3_label,
1807+ misc_2_mask, outer_first_mpls_over_gre_label);
1808+
1809+ DR_STE_SET_TAG(flex_parser_0, tag, parser_3_exp,
1810+ misc_2_mask, outer_first_mpls_over_gre_exp);
1811+
1812+ DR_STE_SET_TAG(flex_parser_0, tag, parser_3_s_bos,
1813+ misc_2_mask, outer_first_mpls_over_gre_s_bos);
1814+
1815+ DR_STE_SET_TAG(flex_parser_0, tag, parser_3_ttl,
1816+ misc_2_mask, outer_first_mpls_over_gre_ttl);
1817+ } else {
1818+ DR_STE_SET_TAG(flex_parser_0, tag, parser_3_label,
1819+ misc_2_mask, outer_first_mpls_over_udp_label);
1820+
1821+ DR_STE_SET_TAG(flex_parser_0, tag, parser_3_exp,
1822+ misc_2_mask, outer_first_mpls_over_udp_exp);
1823+
1824+ DR_STE_SET_TAG(flex_parser_0, tag, parser_3_s_bos,
1825+ misc_2_mask, outer_first_mpls_over_udp_s_bos);
1826+
1827+ DR_STE_SET_TAG(flex_parser_0, tag, parser_3_ttl,
1828+ misc_2_mask, outer_first_mpls_over_udp_ttl);
1829+ }
1830+ return 0;
1831+}
1832+
1833+void mlx5dr_ste_build_flex_parser_0(struct mlx5dr_ste_build *sb,
1834+ struct mlx5dr_match_param *mask,
1835+ bool inner, bool rx)
1836+{
1837+ dr_ste_build_flex_parser_0_bit_mask(mask, inner, sb->bit_mask);
1838+
1839+ sb->rx = rx;
1840+ sb->inner = inner;
1841+ sb->lu_type = MLX5DR_STE_LU_TYPE_FLEX_PARSER_0;
1842+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
1843+ sb->ste_build_tag_func = &dr_ste_build_flex_parser_0_tag;
1844+}
1845+
1846+#define ICMP_TYPE_OFFSET_FIRST_DW 24
1847+#define ICMP_CODE_OFFSET_FIRST_DW 16
1848+#define ICMP_HEADER_DATA_OFFSET_SECOND_DW 0
1849+
1850+static int dr_ste_build_flex_parser_1_bit_mask(struct mlx5dr_match_param *mask,
1851+ struct mlx5dr_cmd_caps *caps,
1852+ u8 *bit_mask)
1853+{
1854+ struct mlx5dr_match_misc3 *misc_3_mask = &mask->misc3;
1855+ bool is_ipv4_mask = DR_MASK_IS_FLEX_PARSER_ICMPV4_SET(misc_3_mask);
1856+ u32 icmp_header_data_mask;
1857+ u32 icmp_type_mask;
1858+ u32 icmp_code_mask;
1859+ int dw0_location;
1860+ int dw1_location;
1861+
1862+ if (is_ipv4_mask) {
1863+ icmp_header_data_mask = misc_3_mask->icmpv4_header_data;
1864+ icmp_type_mask = misc_3_mask->icmpv4_type;
1865+ icmp_code_mask = misc_3_mask->icmpv4_code;
1866+ dw0_location = caps->flex_parser_id_icmp_dw0;
1867+ dw1_location = caps->flex_parser_id_icmp_dw1;
1868+ } else {
1869+ icmp_header_data_mask = misc_3_mask->icmpv6_header_data;
1870+ icmp_type_mask = misc_3_mask->icmpv6_type;
1871+ icmp_code_mask = misc_3_mask->icmpv6_code;
1872+ dw0_location = caps->flex_parser_id_icmpv6_dw0;
1873+ dw1_location = caps->flex_parser_id_icmpv6_dw1;
1874+ }
1875+
1876+ switch (dw0_location) {
1877+ case 4:
1878+ if (icmp_type_mask) {
1879+ MLX5_SET(ste_flex_parser_1, bit_mask, flex_parser_4,
1880+ (icmp_type_mask << ICMP_TYPE_OFFSET_FIRST_DW));
1881+ if (is_ipv4_mask)
1882+ misc_3_mask->icmpv4_type = 0;
1883+ else
1884+ misc_3_mask->icmpv6_type = 0;
1885+ }
1886+ if (icmp_code_mask) {
1887+ u32 cur_val = MLX5_GET(ste_flex_parser_1, bit_mask,
1888+ flex_parser_4);
1889+ MLX5_SET(ste_flex_parser_1, bit_mask, flex_parser_4,
1890+ cur_val | (icmp_code_mask << ICMP_CODE_OFFSET_FIRST_DW));
1891+ if (is_ipv4_mask)
1892+ misc_3_mask->icmpv4_code = 0;
1893+ else
1894+ misc_3_mask->icmpv6_code = 0;
1895+ }
1896+ break;
1897+ default:
1898+ return -EINVAL;
1899+ }
1900+
1901+ switch (dw1_location) {
1902+ case 5:
1903+ if (icmp_header_data_mask) {
1904+ MLX5_SET(ste_flex_parser_1, bit_mask, flex_parser_5,
1905+ (icmp_header_data_mask << ICMP_HEADER_DATA_OFFSET_SECOND_DW));
1906+ if (is_ipv4_mask)
1907+ misc_3_mask->icmpv4_header_data = 0;
1908+ else
1909+ misc_3_mask->icmpv6_header_data = 0;
1910+ }
1911+ break;
1912+ default:
1913+ return -EINVAL;
1914+ }
1915+
1916+ return 0;
1917+}
1918+
1919+static int dr_ste_build_flex_parser_1_tag(struct mlx5dr_match_param *value,
1920+ struct mlx5dr_ste_build *sb,
1921+ u8 *hw_ste_p)
1922+{
1923+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
1924+ struct mlx5dr_match_misc3 *misc_3 = &value->misc3;
1925+ u8 *tag = hw_ste->tag;
1926+ u32 icmp_header_data;
1927+ int dw0_location;
1928+ int dw1_location;
1929+ u32 icmp_type;
1930+ u32 icmp_code;
1931+ bool is_ipv4;
1932+
1933+ is_ipv4 = DR_MASK_IS_FLEX_PARSER_ICMPV4_SET(misc_3);
1934+ if (is_ipv4) {
1935+ icmp_header_data = misc_3->icmpv4_header_data;
1936+ icmp_type = misc_3->icmpv4_type;
1937+ icmp_code = misc_3->icmpv4_code;
1938+ dw0_location = sb->caps->flex_parser_id_icmp_dw0;
1939+ dw1_location = sb->caps->flex_parser_id_icmp_dw1;
1940+ } else {
1941+ icmp_header_data = misc_3->icmpv6_header_data;
1942+ icmp_type = misc_3->icmpv6_type;
1943+ icmp_code = misc_3->icmpv6_code;
1944+ dw0_location = sb->caps->flex_parser_id_icmpv6_dw0;
1945+ dw1_location = sb->caps->flex_parser_id_icmpv6_dw1;
1946+ }
1947+
1948+ switch (dw0_location) {
1949+ case 4:
1950+ if (icmp_type) {
1951+ MLX5_SET(ste_flex_parser_1, tag, flex_parser_4,
1952+ (icmp_type << ICMP_TYPE_OFFSET_FIRST_DW));
1953+ if (is_ipv4)
1954+ misc_3->icmpv4_type = 0;
1955+ else
1956+ misc_3->icmpv6_type = 0;
1957+ }
1958+
1959+ if (icmp_code) {
1960+ u32 cur_val = MLX5_GET(ste_flex_parser_1, tag,
1961+ flex_parser_4);
1962+ MLX5_SET(ste_flex_parser_1, tag, flex_parser_4,
1963+ cur_val | (icmp_code << ICMP_CODE_OFFSET_FIRST_DW));
1964+ if (is_ipv4)
1965+ misc_3->icmpv4_code = 0;
1966+ else
1967+ misc_3->icmpv6_code = 0;
1968+ }
1969+ break;
1970+ default:
1971+ return -EINVAL;
1972+ }
1973+
1974+ switch (dw1_location) {
1975+ case 5:
1976+ if (icmp_header_data) {
1977+ MLX5_SET(ste_flex_parser_1, tag, flex_parser_5,
1978+ (icmp_header_data << ICMP_HEADER_DATA_OFFSET_SECOND_DW));
1979+ if (is_ipv4)
1980+ misc_3->icmpv4_header_data = 0;
1981+ else
1982+ misc_3->icmpv6_header_data = 0;
1983+ }
1984+ break;
1985+ default:
1986+ return -EINVAL;
1987+ }
1988+
1989+ return 0;
1990+}
1991+
1992+int mlx5dr_ste_build_flex_parser_1(struct mlx5dr_ste_build *sb,
1993+ struct mlx5dr_match_param *mask,
1994+ struct mlx5dr_cmd_caps *caps,
1995+ bool inner, bool rx)
1996+{
1997+ int ret;
1998+
1999+ ret = dr_ste_build_flex_parser_1_bit_mask(mask, caps, sb->bit_mask);
2000+ if (ret)
2001+ return ret;
2002+
2003+ sb->rx = rx;
2004+ sb->inner = inner;
2005+ sb->caps = caps;
2006+ sb->lu_type = MLX5DR_STE_LU_TYPE_FLEX_PARSER_1;
2007+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2008+ sb->ste_build_tag_func = &dr_ste_build_flex_parser_1_tag;
2009+
2010+ return 0;
2011+}
2012+
2013+static void dr_ste_build_general_purpose_bit_mask(struct mlx5dr_match_param *value,
2014+ bool inner, u8 *bit_mask)
2015+{
2016+ struct mlx5dr_match_misc2 *misc_2_mask = &value->misc2;
2017+
2018+ DR_STE_SET_MASK_V(general_purpose, bit_mask,
2019+ general_purpose_lookup_field, misc_2_mask,
2020+ metadata_reg_a);
2021+}
2022+
2023+static int dr_ste_build_general_purpose_tag(struct mlx5dr_match_param *value,
2024+ struct mlx5dr_ste_build *sb,
2025+ u8 *hw_ste_p)
2026+{
2027+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
2028+ struct mlx5dr_match_misc2 *misc_2_mask = &value->misc2;
2029+ u8 *tag = hw_ste->tag;
2030+
2031+ DR_STE_SET_TAG(general_purpose, tag, general_purpose_lookup_field,
2032+ misc_2_mask, metadata_reg_a);
2033+
2034+ return 0;
2035+}
2036+
2037+void mlx5dr_ste_build_general_purpose(struct mlx5dr_ste_build *sb,
2038+ struct mlx5dr_match_param *mask,
2039+ bool inner, bool rx)
2040+{
2041+ dr_ste_build_general_purpose_bit_mask(mask, inner, sb->bit_mask);
2042+
2043+ sb->rx = rx;
2044+ sb->inner = inner;
2045+ sb->lu_type = MLX5DR_STE_LU_TYPE_GENERAL_PURPOSE;
2046+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2047+ sb->ste_build_tag_func = &dr_ste_build_general_purpose_tag;
2048+}
2049+
2050+static void dr_ste_build_eth_l4_misc_bit_mask(struct mlx5dr_match_param *value,
2051+ bool inner, u8 *bit_mask)
2052+{
2053+ struct mlx5dr_match_misc3 *misc_3_mask = &value->misc3;
2054+
2055+ if (inner) {
2056+ DR_STE_SET_MASK_V(eth_l4_misc, bit_mask, seq_num, misc_3_mask,
2057+ inner_tcp_seq_num);
2058+ DR_STE_SET_MASK_V(eth_l4_misc, bit_mask, ack_num, misc_3_mask,
2059+ inner_tcp_ack_num);
2060+ } else {
2061+ DR_STE_SET_MASK_V(eth_l4_misc, bit_mask, seq_num, misc_3_mask,
2062+ outer_tcp_seq_num);
2063+ DR_STE_SET_MASK_V(eth_l4_misc, bit_mask, ack_num, misc_3_mask,
2064+ outer_tcp_ack_num);
2065+ }
2066+}
2067+
2068+static int dr_ste_build_eth_l4_misc_tag(struct mlx5dr_match_param *value,
2069+ struct mlx5dr_ste_build *sb,
2070+ u8 *hw_ste_p)
2071+{
2072+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
2073+ struct mlx5dr_match_misc3 *misc3 = &value->misc3;
2074+ u8 *tag = hw_ste->tag;
2075+
2076+ if (sb->inner) {
2077+ DR_STE_SET_TAG(eth_l4_misc, tag, seq_num, misc3, inner_tcp_seq_num);
2078+ DR_STE_SET_TAG(eth_l4_misc, tag, ack_num, misc3, inner_tcp_ack_num);
2079+ } else {
2080+ DR_STE_SET_TAG(eth_l4_misc, tag, seq_num, misc3, outer_tcp_seq_num);
2081+ DR_STE_SET_TAG(eth_l4_misc, tag, ack_num, misc3, outer_tcp_ack_num);
2082+ }
2083+
2084+ return 0;
2085+}
2086+
2087+void mlx5dr_ste_build_eth_l4_misc(struct mlx5dr_ste_build *sb,
2088+ struct mlx5dr_match_param *mask,
2089+ bool inner, bool rx)
2090+{
2091+ dr_ste_build_eth_l4_misc_bit_mask(mask, inner, sb->bit_mask);
2092+
2093+ sb->rx = rx;
2094+ sb->inner = inner;
2095+ sb->lu_type = DR_STE_CALC_LU_TYPE(ETHL4_MISC, rx, inner);
2096+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2097+ sb->ste_build_tag_func = &dr_ste_build_eth_l4_misc_tag;
2098+}
2099+
2100+static void dr_ste_build_flex_parser_tnl_bit_mask(struct mlx5dr_match_param *value,
2101+ bool inner, u8 *bit_mask)
2102+{
2103+ struct mlx5dr_match_misc3 *misc_3_mask = &value->misc3;
2104+
2105+ if (misc_3_mask->outer_vxlan_gpe_flags ||
2106+ misc_3_mask->outer_vxlan_gpe_next_protocol) {
2107+ MLX5_SET(ste_flex_parser_tnl, bit_mask,
2108+ flex_parser_tunneling_header_63_32,
2109+ (misc_3_mask->outer_vxlan_gpe_flags << 24) |
2110+ (misc_3_mask->outer_vxlan_gpe_next_protocol));
2111+ misc_3_mask->outer_vxlan_gpe_flags = 0;
2112+ misc_3_mask->outer_vxlan_gpe_next_protocol = 0;
2113+ }
2114+
2115+ if (misc_3_mask->outer_vxlan_gpe_vni) {
2116+ MLX5_SET(ste_flex_parser_tnl, bit_mask,
2117+ flex_parser_tunneling_header_31_0,
2118+ misc_3_mask->outer_vxlan_gpe_vni << 8);
2119+ misc_3_mask->outer_vxlan_gpe_vni = 0;
2120+ }
2121+}
2122+
2123+static int dr_ste_build_flex_parser_tnl_tag(struct mlx5dr_match_param *value,
2124+ struct mlx5dr_ste_build *sb,
2125+ u8 *hw_ste_p)
2126+{
2127+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
2128+ struct mlx5dr_match_misc3 *misc3 = &value->misc3;
2129+ u8 *tag = hw_ste->tag;
2130+
2131+ if (misc3->outer_vxlan_gpe_flags ||
2132+ misc3->outer_vxlan_gpe_next_protocol) {
2133+ MLX5_SET(ste_flex_parser_tnl, tag,
2134+ flex_parser_tunneling_header_63_32,
2135+ (misc3->outer_vxlan_gpe_flags << 24) |
2136+ (misc3->outer_vxlan_gpe_next_protocol));
2137+ misc3->outer_vxlan_gpe_flags = 0;
2138+ misc3->outer_vxlan_gpe_next_protocol = 0;
2139+ }
2140+
2141+ if (misc3->outer_vxlan_gpe_vni) {
2142+ MLX5_SET(ste_flex_parser_tnl, tag,
2143+ flex_parser_tunneling_header_31_0,
2144+ misc3->outer_vxlan_gpe_vni << 8);
2145+ misc3->outer_vxlan_gpe_vni = 0;
2146+ }
2147+
2148+ return 0;
2149+}
2150+
2151+void mlx5dr_ste_build_flex_parser_tnl(struct mlx5dr_ste_build *sb,
2152+ struct mlx5dr_match_param *mask,
2153+ bool inner, bool rx)
2154+{
2155+ dr_ste_build_flex_parser_tnl_bit_mask(mask, inner, sb->bit_mask);
2156+
2157+ sb->rx = rx;
2158+ sb->inner = inner;
2159+ sb->lu_type = MLX5DR_STE_LU_TYPE_FLEX_PARSER_TNL_HEADER;
2160+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2161+ sb->ste_build_tag_func = &dr_ste_build_flex_parser_tnl_tag;
2162+}
2163+
2164+static void dr_ste_build_register_0_bit_mask(struct mlx5dr_match_param *value,
2165+ u8 *bit_mask)
2166+{
2167+ struct mlx5dr_match_misc2 *misc_2_mask = &value->misc2;
2168+
2169+ DR_STE_SET_MASK_V(register_0, bit_mask, register_0_h,
2170+ misc_2_mask, metadata_reg_c_0);
2171+ DR_STE_SET_MASK_V(register_0, bit_mask, register_0_l,
2172+ misc_2_mask, metadata_reg_c_1);
2173+ DR_STE_SET_MASK_V(register_0, bit_mask, register_1_h,
2174+ misc_2_mask, metadata_reg_c_2);
2175+ DR_STE_SET_MASK_V(register_0, bit_mask, register_1_l,
2176+ misc_2_mask, metadata_reg_c_3);
2177+}
2178+
2179+static int dr_ste_build_register_0_tag(struct mlx5dr_match_param *value,
2180+ struct mlx5dr_ste_build *sb,
2181+ u8 *hw_ste_p)
2182+{
2183+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
2184+ struct mlx5dr_match_misc2 *misc2 = &value->misc2;
2185+ u8 *tag = hw_ste->tag;
2186+
2187+ DR_STE_SET_TAG(register_0, tag, register_0_h, misc2, metadata_reg_c_0);
2188+ DR_STE_SET_TAG(register_0, tag, register_0_l, misc2, metadata_reg_c_1);
2189+ DR_STE_SET_TAG(register_0, tag, register_1_h, misc2, metadata_reg_c_2);
2190+ DR_STE_SET_TAG(register_0, tag, register_1_l, misc2, metadata_reg_c_3);
2191+
2192+ return 0;
2193+}
2194+
2195+void mlx5dr_ste_build_register_0(struct mlx5dr_ste_build *sb,
2196+ struct mlx5dr_match_param *mask,
2197+ bool inner, bool rx)
2198+{
2199+ dr_ste_build_register_0_bit_mask(mask, sb->bit_mask);
2200+
2201+ sb->rx = rx;
2202+ sb->inner = inner;
2203+ sb->lu_type = MLX5DR_STE_LU_TYPE_STEERING_REGISTERS_0;
2204+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2205+ sb->ste_build_tag_func = &dr_ste_build_register_0_tag;
2206+}
2207+
2208+static void dr_ste_build_register_1_bit_mask(struct mlx5dr_match_param *value,
2209+ u8 *bit_mask)
2210+{
2211+ struct mlx5dr_match_misc2 *misc_2_mask = &value->misc2;
2212+
2213+ DR_STE_SET_MASK_V(register_1, bit_mask, register_2_h,
2214+ misc_2_mask, metadata_reg_c_4);
2215+ DR_STE_SET_MASK_V(register_1, bit_mask, register_2_l,
2216+ misc_2_mask, metadata_reg_c_5);
2217+ DR_STE_SET_MASK_V(register_1, bit_mask, register_3_h,
2218+ misc_2_mask, metadata_reg_c_6);
2219+ DR_STE_SET_MASK_V(register_1, bit_mask, register_3_l,
2220+ misc_2_mask, metadata_reg_c_7);
2221+}
2222+
2223+static int dr_ste_build_register_1_tag(struct mlx5dr_match_param *value,
2224+ struct mlx5dr_ste_build *sb,
2225+ u8 *hw_ste_p)
2226+{
2227+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
2228+ struct mlx5dr_match_misc2 *misc2 = &value->misc2;
2229+ u8 *tag = hw_ste->tag;
2230+
2231+ DR_STE_SET_TAG(register_1, tag, register_2_h, misc2, metadata_reg_c_4);
2232+ DR_STE_SET_TAG(register_1, tag, register_2_l, misc2, metadata_reg_c_5);
2233+ DR_STE_SET_TAG(register_1, tag, register_3_h, misc2, metadata_reg_c_6);
2234+ DR_STE_SET_TAG(register_1, tag, register_3_l, misc2, metadata_reg_c_7);
2235+
2236+ return 0;
2237+}
2238+
2239+void mlx5dr_ste_build_register_1(struct mlx5dr_ste_build *sb,
2240+ struct mlx5dr_match_param *mask,
2241+ bool inner, bool rx)
2242+{
2243+ dr_ste_build_register_1_bit_mask(mask, sb->bit_mask);
2244+
2245+ sb->rx = rx;
2246+ sb->inner = inner;
2247+ sb->lu_type = MLX5DR_STE_LU_TYPE_STEERING_REGISTERS_1;
2248+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2249+ sb->ste_build_tag_func = &dr_ste_build_register_1_tag;
2250+}
2251+
2252+static int dr_ste_build_src_gvmi_qpn_bit_mask(struct mlx5dr_match_param *value,
2253+ u8 *bit_mask)
2254+{
2255+ struct mlx5dr_match_misc *misc_mask = &value->misc;
2256+
2257+ if (misc_mask->source_port != 0xffff)
2258+ return -EINVAL;
2259+
2260+ DR_STE_SET_MASK(src_gvmi_qp, bit_mask, source_gvmi, misc_mask, source_port);
2261+ DR_STE_SET_MASK(src_gvmi_qp, bit_mask, source_qp, misc_mask, source_sqn);
2262+
2263+ return 0;
2264+}
2265+
2266+static int dr_ste_build_src_gvmi_qpn_tag(struct mlx5dr_match_param *value,
2267+ struct mlx5dr_ste_build *sb,
2268+ u8 *hw_ste_p)
2269+{
2270+ struct dr_hw_ste_format *hw_ste = (struct dr_hw_ste_format *)hw_ste_p;
2271+ struct mlx5dr_match_misc *misc = &value->misc;
2272+ struct mlx5dr_cmd_vport_cap *vport_cap;
2273+ u8 *tag = hw_ste->tag;
2274+
2275+ DR_STE_SET_TAG(src_gvmi_qp, tag, source_qp, misc, source_sqn);
2276+
2277+ vport_cap = mlx5dr_get_vport_cap(sb->caps, misc->source_port);
2278+ if (!vport_cap)
2279+ return -EINVAL;
2280+
2281+ if (vport_cap->vport_gvmi)
2282+ MLX5_SET(ste_src_gvmi_qp, tag, source_gvmi, vport_cap->vport_gvmi);
2283+
2284+ misc->source_port = 0;
2285+
2286+ return 0;
2287+}
2288+
2289+int mlx5dr_ste_build_src_gvmi_qpn(struct mlx5dr_ste_build *sb,
2290+ struct mlx5dr_match_param *mask,
2291+ struct mlx5dr_cmd_caps *caps,
2292+ bool inner, bool rx)
2293+{
2294+ int ret;
2295+
2296+ ret = dr_ste_build_src_gvmi_qpn_bit_mask(mask, sb->bit_mask);
2297+ if (ret)
2298+ return ret;
2299+
2300+ sb->rx = rx;
2301+ sb->caps = caps;
2302+ sb->inner = inner;
2303+ sb->lu_type = MLX5DR_STE_LU_TYPE_SRC_GVMI_AND_QP;
2304+ sb->byte_mask = dr_ste_conv_bit_to_byte_mask(sb->bit_mask);
2305+ sb->ste_build_tag_func = &dr_ste_build_src_gvmi_qpn_tag;
2306+
2307+ return 0;
2308+}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_table.c+294−0
@@ -0,0 +1,294 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+/* Copyright (c) 2019 Mellanox Technologies. */
3+
4+#include "dr_types.h"
5+
6+int mlx5dr_table_set_miss_action(struct mlx5dr_table *tbl,
7+ struct mlx5dr_action *action)
8+{
9+ struct mlx5dr_matcher *last_matcher = NULL;
10+ struct mlx5dr_htbl_connect_info info;
11+ struct mlx5dr_ste_htbl *last_htbl;
12+ int ret;
13+
14+ if (action && action->action_type != DR_ACTION_TYP_FT)
15+ return -EOPNOTSUPP;
16+
17+ mutex_lock(&tbl->dmn->mutex);
18+
19+ if (!list_empty(&tbl->matcher_list))
20+ last_matcher = list_last_entry(&tbl->matcher_list,
21+ struct mlx5dr_matcher,
22+ matcher_list);
23+
24+ if (tbl->dmn->type == MLX5DR_DOMAIN_TYPE_NIC_RX ||
25+ tbl->dmn->type == MLX5DR_DOMAIN_TYPE_FDB) {
26+ if (last_matcher)
27+ last_htbl = last_matcher->rx.e_anchor;
28+ else
29+ last_htbl = tbl->rx.s_anchor;
30+
31+ tbl->rx.default_icm_addr = action ?
32+ action->dest_tbl.tbl->rx.s_anchor->chunk->icm_addr :
33+ tbl->rx.nic_dmn->default_icm_addr;
34+
35+ info.type = CONNECT_MISS;
36+ info.miss_icm_addr = tbl->rx.default_icm_addr;
37+
38+ ret = mlx5dr_ste_htbl_init_and_postsend(tbl->dmn,
39+ tbl->rx.nic_dmn,
40+ last_htbl,
41+ &info, true);
42+ if (ret) {
43+ mlx5dr_dbg(tbl->dmn, "Failed to set RX miss action, ret %d\n", ret);
44+ goto out;
45+ }
46+ }
47+
48+ if (tbl->dmn->type == MLX5DR_DOMAIN_TYPE_NIC_TX ||
49+ tbl->dmn->type == MLX5DR_DOMAIN_TYPE_FDB) {
50+ if (last_matcher)
51+ last_htbl = last_matcher->tx.e_anchor;
52+ else
53+ last_htbl = tbl->tx.s_anchor;
54+
55+ tbl->tx.default_icm_addr = action ?
56+ action->dest_tbl.tbl->tx.s_anchor->chunk->icm_addr :
57+ tbl->tx.nic_dmn->default_icm_addr;
58+
59+ info.type = CONNECT_MISS;
60+ info.miss_icm_addr = tbl->tx.default_icm_addr;
61+
62+ ret = mlx5dr_ste_htbl_init_and_postsend(tbl->dmn,
63+ tbl->tx.nic_dmn,
64+ last_htbl, &info, true);
65+ if (ret) {
66+ mlx5dr_dbg(tbl->dmn, "Failed to set TX miss action, ret %d\n", ret);
67+ goto out;
68+ }
69+ }
70+
71+ /* Release old action */
72+ if (tbl->miss_action)
73+ refcount_dec(&tbl->miss_action->refcount);
74+
75+ /* Set new miss action */
76+ tbl->miss_action = action;
77+ if (tbl->miss_action)
78+ refcount_inc(&action->refcount);
79+
80+out:
81+ mutex_unlock(&tbl->dmn->mutex);
82+ return ret;
83+}
84+
85+static void dr_table_uninit_nic(struct mlx5dr_table_rx_tx *nic_tbl)
86+{
87+ mlx5dr_htbl_put(nic_tbl->s_anchor);
88+}
89+
90+static void dr_table_uninit_fdb(struct mlx5dr_table *tbl)
91+{
92+ dr_table_uninit_nic(&tbl->rx);
93+ dr_table_uninit_nic(&tbl->tx);
94+}
95+
96+static void dr_table_uninit(struct mlx5dr_table *tbl)
97+{
98+ mutex_lock(&tbl->dmn->mutex);
99+
100+ switch (tbl->dmn->type) {
101+ case MLX5DR_DOMAIN_TYPE_NIC_RX:
102+ dr_table_uninit_nic(&tbl->rx);
103+ break;
104+ case MLX5DR_DOMAIN_TYPE_NIC_TX:
105+ dr_table_uninit_nic(&tbl->tx);
106+ break;
107+ case MLX5DR_DOMAIN_TYPE_FDB:
108+ dr_table_uninit_fdb(tbl);
109+ break;
110+ default:
111+ WARN_ON(true);
112+ break;
113+ }
114+
115+ mutex_unlock(&tbl->dmn->mutex);
116+}
117+
118+static int dr_table_init_nic(struct mlx5dr_domain *dmn,
119+ struct mlx5dr_table_rx_tx *nic_tbl)
120+{
121+ struct mlx5dr_domain_rx_tx *nic_dmn = nic_tbl->nic_dmn;
122+ struct mlx5dr_htbl_connect_info info;
123+ int ret;
124+
125+ nic_tbl->default_icm_addr = nic_dmn->default_icm_addr;
126+
127+ nic_tbl->s_anchor = mlx5dr_ste_htbl_alloc(dmn->ste_icm_pool,
128+ DR_CHUNK_SIZE_1,
129+ MLX5DR_STE_LU_TYPE_DONT_CARE,
130+ 0);
131+ if (!nic_tbl->s_anchor)
132+ return -ENOMEM;
133+
134+ info.type = CONNECT_MISS;
135+ info.miss_icm_addr = nic_dmn->default_icm_addr;
136+ ret = mlx5dr_ste_htbl_init_and_postsend(dmn, nic_dmn,
137+ nic_tbl->s_anchor,
138+ &info, true);
139+ if (ret)
140+ goto free_s_anchor;
141+
142+ mlx5dr_htbl_get(nic_tbl->s_anchor);
143+
144+ return 0;
145+
146+free_s_anchor:
147+ mlx5dr_ste_htbl_free(nic_tbl->s_anchor);
148+ return ret;
149+}
150+
151+static int dr_table_init_fdb(struct mlx5dr_table *tbl)
152+{
153+ int ret;
154+
155+ ret = dr_table_init_nic(tbl->dmn, &tbl->rx);
156+ if (ret)
157+ return ret;
158+
159+ ret = dr_table_init_nic(tbl->dmn, &tbl->tx);
160+ if (ret)
161+ goto destroy_rx;
162+
163+ return 0;
164+
165+destroy_rx:
166+ dr_table_uninit_nic(&tbl->rx);
167+ return ret;
168+}
169+
170+static int dr_table_init(struct mlx5dr_table *tbl)
171+{
172+ int ret = 0;
173+
174+ INIT_LIST_HEAD(&tbl->matcher_list);
175+
176+ mutex_lock(&tbl->dmn->mutex);
177+
178+ switch (tbl->dmn->type) {
179+ case MLX5DR_DOMAIN_TYPE_NIC_RX:
180+ tbl->table_type = MLX5_FLOW_TABLE_TYPE_NIC_RX;
181+ tbl->rx.nic_dmn = &tbl->dmn->info.rx;
182+ ret = dr_table_init_nic(tbl->dmn, &tbl->rx);
183+ break;
184+ case MLX5DR_DOMAIN_TYPE_NIC_TX:
185+ tbl->table_type = MLX5_FLOW_TABLE_TYPE_NIC_TX;
186+ tbl->tx.nic_dmn = &tbl->dmn->info.tx;
187+ ret = dr_table_init_nic(tbl->dmn, &tbl->tx);
188+ break;
189+ case MLX5DR_DOMAIN_TYPE_FDB:
190+ tbl->table_type = MLX5_FLOW_TABLE_TYPE_FDB;
191+ tbl->rx.nic_dmn = &tbl->dmn->info.rx;
192+ tbl->tx.nic_dmn = &tbl->dmn->info.tx;
193+ ret = dr_table_init_fdb(tbl);
194+ break;
195+ default:
196+ WARN_ON(true);
197+ break;
198+ }
199+
200+ mutex_unlock(&tbl->dmn->mutex);
201+
202+ return ret;
203+}
204+
205+static int dr_table_destroy_sw_owned_tbl(struct mlx5dr_table *tbl)
206+{
207+ return mlx5dr_cmd_destroy_flow_table(tbl->dmn->mdev,
208+ tbl->table_id,
209+ tbl->table_type);
210+}
211+
212+static int dr_table_create_sw_owned_tbl(struct mlx5dr_table *tbl)
213+{
214+ u64 icm_addr_rx = 0;
215+ u64 icm_addr_tx = 0;
216+ int ret;
217+
218+ if (tbl->rx.s_anchor)
219+ icm_addr_rx = tbl->rx.s_anchor->chunk->icm_addr;
220+
221+ if (tbl->tx.s_anchor)
222+ icm_addr_tx = tbl->tx.s_anchor->chunk->icm_addr;
223+
224+ ret = mlx5dr_cmd_create_flow_table(tbl->dmn->mdev,
225+ tbl->table_type,
226+ icm_addr_rx,
227+ icm_addr_tx,
228+ tbl->dmn->info.caps.max_ft_level - 1,
229+ true, false, NULL,
230+ &tbl->table_id);
231+
232+ return ret;
233+}
234+
235+struct mlx5dr_table *mlx5dr_table_create(struct mlx5dr_domain *dmn, u32 level)
236+{
237+ struct mlx5dr_table *tbl;
238+ int ret;
239+
240+ refcount_inc(&dmn->refcount);
241+
242+ tbl = kzalloc(sizeof(*tbl), GFP_KERNEL);
243+ if (!tbl)
244+ goto dec_ref;
245+
246+ tbl->dmn = dmn;
247+ tbl->level = level;
248+ refcount_set(&tbl->refcount, 1);
249+
250+ ret = dr_table_init(tbl);
251+ if (ret)
252+ goto free_tbl;
253+
254+ ret = dr_table_create_sw_owned_tbl(tbl);
255+ if (ret)
256+ goto uninit_tbl;
257+
258+ return tbl;
259+
260+uninit_tbl:
261+ dr_table_uninit(tbl);
262+free_tbl:
263+ kfree(tbl);
264+dec_ref:
265+ refcount_dec(&dmn->refcount);
266+ return NULL;
267+}
268+
269+int mlx5dr_table_destroy(struct mlx5dr_table *tbl)
270+{
271+ int ret;
272+
273+ if (refcount_read(&tbl->refcount) > 1)
274+ return -EBUSY;
275+
276+ ret = dr_table_destroy_sw_owned_tbl(tbl);
277+ if (ret)
278+ return ret;
279+
280+ dr_table_uninit(tbl);
281+
282+ if (tbl->miss_action)
283+ refcount_dec(&tbl->miss_action->refcount);
284+
285+ refcount_dec(&tbl->dmn->refcount);
286+ kfree(tbl);
287+
288+ return ret;
289+}
290+
291+u32 mlx5dr_table_get_id(struct mlx5dr_table *tbl)
292+{
293+ return tbl->table_id;
294+}
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_types.h+1060−0
@@ -0,0 +1,1060 @@
1+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
2+/* Copyright (c) 2019, Mellanox Technologies */
3+
4+#ifndef _DR_TYPES_
5+#define _DR_TYPES_
6+
7+#include <linux/mlx5/driver.h>
8+#include <linux/refcount.h>
9+#include "fs_core.h"
10+#include "wq.h"
11+#include "lib/mlx5.h"
12+#include "mlx5_ifc_dr.h"
13+#include "mlx5dr.h"
14+
15+#define DR_RULE_MAX_STES 17
16+#define DR_ACTION_MAX_STES 5
17+#define WIRE_PORT 0xFFFF
18+#define DR_STE_SVLAN 0x1
19+#define DR_STE_CVLAN 0x2
20+
21+#define mlx5dr_err(dmn, arg...) mlx5_core_err((dmn)->mdev, ##arg)
22+#define mlx5dr_info(dmn, arg...) mlx5_core_info((dmn)->mdev, ##arg)
23+#define mlx5dr_dbg(dmn, arg...) mlx5_core_dbg((dmn)->mdev, ##arg)
24+
25+enum mlx5dr_icm_chunk_size {
26+ DR_CHUNK_SIZE_1,
27+ DR_CHUNK_SIZE_MIN = DR_CHUNK_SIZE_1, /* keep updated when changing */
28+ DR_CHUNK_SIZE_2,
29+ DR_CHUNK_SIZE_4,
30+ DR_CHUNK_SIZE_8,
31+ DR_CHUNK_SIZE_16,
32+ DR_CHUNK_SIZE_32,
33+ DR_CHUNK_SIZE_64,
34+ DR_CHUNK_SIZE_128,
35+ DR_CHUNK_SIZE_256,
36+ DR_CHUNK_SIZE_512,
37+ DR_CHUNK_SIZE_1K,
38+ DR_CHUNK_SIZE_2K,
39+ DR_CHUNK_SIZE_4K,
40+ DR_CHUNK_SIZE_8K,
41+ DR_CHUNK_SIZE_16K,
42+ DR_CHUNK_SIZE_32K,
43+ DR_CHUNK_SIZE_64K,
44+ DR_CHUNK_SIZE_128K,
45+ DR_CHUNK_SIZE_256K,
46+ DR_CHUNK_SIZE_512K,
47+ DR_CHUNK_SIZE_1024K,
48+ DR_CHUNK_SIZE_2048K,
49+ DR_CHUNK_SIZE_MAX,
50+};
51+
52+enum mlx5dr_icm_type {
53+ DR_ICM_TYPE_STE,
54+ DR_ICM_TYPE_MODIFY_ACTION,
55+};
56+
57+static inline enum mlx5dr_icm_chunk_size
58+mlx5dr_icm_next_higher_chunk(enum mlx5dr_icm_chunk_size chunk)
59+{
60+ chunk += 2;
61+ if (chunk < DR_CHUNK_SIZE_MAX)
62+ return chunk;
63+
64+ return DR_CHUNK_SIZE_MAX;
65+}
66+
67+enum {
68+ DR_STE_SIZE = 64,
69+ DR_STE_SIZE_CTRL = 32,
70+ DR_STE_SIZE_TAG = 16,
71+ DR_STE_SIZE_MASK = 16,
72+};
73+
74+enum {
75+ DR_STE_SIZE_REDUCED = DR_STE_SIZE - DR_STE_SIZE_MASK,
76+};
77+
78+enum {
79+ DR_MODIFY_ACTION_SIZE = 8,
80+};
81+
82+enum mlx5dr_matcher_criteria {
83+ DR_MATCHER_CRITERIA_EMPTY = 0,
84+ DR_MATCHER_CRITERIA_OUTER = 1 << 0,
85+ DR_MATCHER_CRITERIA_MISC = 1 << 1,
86+ DR_MATCHER_CRITERIA_INNER = 1 << 2,
87+ DR_MATCHER_CRITERIA_MISC2 = 1 << 3,
88+ DR_MATCHER_CRITERIA_MISC3 = 1 << 4,
89+ DR_MATCHER_CRITERIA_MAX = 1 << 5,
90+};
91+
92+enum mlx5dr_action_type {
93+ DR_ACTION_TYP_TNL_L2_TO_L2,
94+ DR_ACTION_TYP_L2_TO_TNL_L2,
95+ DR_ACTION_TYP_TNL_L3_TO_L2,
96+ DR_ACTION_TYP_L2_TO_TNL_L3,
97+ DR_ACTION_TYP_DROP,
98+ DR_ACTION_TYP_QP,
99+ DR_ACTION_TYP_FT,
100+ DR_ACTION_TYP_CTR,
101+ DR_ACTION_TYP_TAG,
102+ DR_ACTION_TYP_MODIFY_HDR,
103+ DR_ACTION_TYP_VPORT,
104+ DR_ACTION_TYP_POP_VLAN,
105+ DR_ACTION_TYP_PUSH_VLAN,
106+ DR_ACTION_TYP_MAX,
107+};
108+
109+struct mlx5dr_icm_pool;
110+struct mlx5dr_icm_chunk;
111+struct mlx5dr_icm_bucket;
112+struct mlx5dr_ste_htbl;
113+struct mlx5dr_match_param;
114+struct mlx5dr_cmd_caps;
115+struct mlx5dr_matcher_rx_tx;
116+
117+struct mlx5dr_ste {
118+ u8 *hw_ste;
119+ /* refcount: indicates the num of rules that using this ste */
120+ refcount_t refcount;
121+
122+ /* attached to the miss_list head at each htbl entry */
123+ struct list_head miss_list_node;
124+
125+ /* each rule member that uses this ste attached here */
126+ struct list_head rule_list;
127+
128+ /* this ste is member of htbl */
129+ struct mlx5dr_ste_htbl *htbl;
130+
131+ struct mlx5dr_ste_htbl *next_htbl;
132+
133+ /* this ste is part of a rule, located in ste's chain */
134+ u8 ste_chain_location;
135+};
136+
137+struct mlx5dr_ste_htbl_ctrl {
138+ /* total number of valid entries belonging to this hash table. This
139+ * includes the non collision and collision entries
140+ */
141+ unsigned int num_of_valid_entries;
142+
143+ /* total number of collisions entries attached to this table */
144+ unsigned int num_of_collisions;
145+ unsigned int increase_threshold;
146+ u8 may_grow:1;
147+};
148+
149+struct mlx5dr_ste_htbl {
150+ u8 lu_type;
151+ u16 byte_mask;
152+ refcount_t refcount;
153+ struct mlx5dr_icm_chunk *chunk;
154+ struct mlx5dr_ste *ste_arr;
155+ u8 *hw_ste_arr;
156+
157+ struct list_head *miss_list;
158+
159+ enum mlx5dr_icm_chunk_size chunk_size;
160+ struct mlx5dr_ste *pointing_ste;
161+
162+ struct mlx5dr_ste_htbl_ctrl ctrl;
163+};
164+
165+struct mlx5dr_ste_send_info {
166+ struct mlx5dr_ste *ste;
167+ struct list_head send_list;
168+ u16 size;
169+ u16 offset;
170+ u8 data_cont[DR_STE_SIZE];
171+ u8 *data;
172+};
173+
174+void mlx5dr_send_fill_and_append_ste_send_info(struct mlx5dr_ste *ste, u16 size,
175+ u16 offset, u8 *data,
176+ struct mlx5dr_ste_send_info *ste_info,
177+ struct list_head *send_list,
178+ bool copy_data);
179+
180+struct mlx5dr_ste_build {
181+ u8 inner:1;
182+ u8 rx:1;
183+ struct mlx5dr_cmd_caps *caps;
184+ u8 lu_type;
185+ u16 byte_mask;
186+ u8 bit_mask[DR_STE_SIZE_MASK];
187+ int (*ste_build_tag_func)(struct mlx5dr_match_param *spec,
188+ struct mlx5dr_ste_build *sb,
189+ u8 *hw_ste_p);
190+};
191+
192+struct mlx5dr_ste_htbl *
193+mlx5dr_ste_htbl_alloc(struct mlx5dr_icm_pool *pool,
194+ enum mlx5dr_icm_chunk_size chunk_size,
195+ u8 lu_type, u16 byte_mask);
196+
197+int mlx5dr_ste_htbl_free(struct mlx5dr_ste_htbl *htbl);
198+
199+static inline void mlx5dr_htbl_put(struct mlx5dr_ste_htbl *htbl)
200+{
201+ if (refcount_dec_and_test(&htbl->refcount))
202+ mlx5dr_ste_htbl_free(htbl);
203+}
204+
205+static inline void mlx5dr_htbl_get(struct mlx5dr_ste_htbl *htbl)
206+{
207+ refcount_inc(&htbl->refcount);
208+}
209+
210+/* STE utils */
211+u32 mlx5dr_ste_calc_hash_index(u8 *hw_ste_p, struct mlx5dr_ste_htbl *htbl);
212+void mlx5dr_ste_init(u8 *hw_ste_p, u8 lu_type, u8 entry_type, u16 gvmi);
213+void mlx5dr_ste_always_hit_htbl(struct mlx5dr_ste *ste,
214+ struct mlx5dr_ste_htbl *next_htbl);
215+void mlx5dr_ste_set_miss_addr(u8 *hw_ste, u64 miss_addr);
216+u64 mlx5dr_ste_get_miss_addr(u8 *hw_ste);
217+void mlx5dr_ste_set_hit_gvmi(u8 *hw_ste_p, u16 gvmi);
218+void mlx5dr_ste_set_hit_addr(u8 *hw_ste, u64 icm_addr, u32 ht_size);
219+void mlx5dr_ste_always_miss_addr(struct mlx5dr_ste *ste, u64 miss_addr);
220+void mlx5dr_ste_set_bit_mask(u8 *hw_ste_p, u8 *bit_mask);
221+bool mlx5dr_ste_not_used_ste(struct mlx5dr_ste *ste);
222+bool mlx5dr_ste_is_last_in_rule(struct mlx5dr_matcher_rx_tx *nic_matcher,
223+ u8 ste_location);
224+void mlx5dr_ste_rx_set_flow_tag(u8 *hw_ste_p, u32 flow_tag);
225+void mlx5dr_ste_set_counter_id(u8 *hw_ste_p, u32 ctr_id);
226+void mlx5dr_ste_set_tx_encap(void *hw_ste_p, u32 reformat_id,
227+ int size, bool encap_l3);
228+void mlx5dr_ste_set_rx_decap(u8 *hw_ste_p);
229+void mlx5dr_ste_set_rx_decap_l3(u8 *hw_ste_p, bool vlan);
230+void mlx5dr_ste_set_rx_pop_vlan(u8 *hw_ste_p);
231+void mlx5dr_ste_set_tx_push_vlan(u8 *hw_ste_p, u32 vlan_tpid_pcp_dei_vid,
232+ bool go_back);
233+void mlx5dr_ste_set_entry_type(u8 *hw_ste_p, u8 entry_type);
234+u8 mlx5dr_ste_get_entry_type(u8 *hw_ste_p);
235+void mlx5dr_ste_set_rewrite_actions(u8 *hw_ste_p, u16 num_of_actions,
236+ u32 re_write_index);
237+void mlx5dr_ste_set_go_back_bit(u8 *hw_ste_p);
238+u64 mlx5dr_ste_get_icm_addr(struct mlx5dr_ste *ste);
239+u64 mlx5dr_ste_get_mr_addr(struct mlx5dr_ste *ste);
240+struct list_head *mlx5dr_ste_get_miss_list(struct mlx5dr_ste *ste);
241+
242+void mlx5dr_ste_free(struct mlx5dr_ste *ste,
243+ struct mlx5dr_matcher *matcher,
244+ struct mlx5dr_matcher_rx_tx *nic_matcher);
245+static inline void mlx5dr_ste_put(struct mlx5dr_ste *ste,
246+ struct mlx5dr_matcher *matcher,
247+ struct mlx5dr_matcher_rx_tx *nic_matcher)
248+{
249+ if (refcount_dec_and_test(&ste->refcount))
250+ mlx5dr_ste_free(ste, matcher, nic_matcher);
251+}
252+
253+/* initial as 0, increased only when ste appears in a new rule */
254+static inline void mlx5dr_ste_get(struct mlx5dr_ste *ste)
255+{
256+ refcount_inc(&ste->refcount);
257+}
258+
259+void mlx5dr_ste_set_hit_addr_by_next_htbl(u8 *hw_ste,
260+ struct mlx5dr_ste_htbl *next_htbl);
261+bool mlx5dr_ste_equal_tag(void *src, void *dst);
262+int mlx5dr_ste_create_next_htbl(struct mlx5dr_matcher *matcher,
263+ struct mlx5dr_matcher_rx_tx *nic_matcher,
264+ struct mlx5dr_ste *ste,
265+ u8 *cur_hw_ste,
266+ enum mlx5dr_icm_chunk_size log_table_size);
267+
268+/* STE build functions */
269+int mlx5dr_ste_build_pre_check(struct mlx5dr_domain *dmn,
270+ u8 match_criteria,
271+ struct mlx5dr_match_param *mask,
272+ struct mlx5dr_match_param *value);
273+int mlx5dr_ste_build_ste_arr(struct mlx5dr_matcher *matcher,
274+ struct mlx5dr_matcher_rx_tx *nic_matcher,
275+ struct mlx5dr_match_param *value,
276+ u8 *ste_arr);
277+int mlx5dr_ste_build_eth_l2_src_des(struct mlx5dr_ste_build *builder,
278+ struct mlx5dr_match_param *mask,
279+ bool inner, bool rx);
280+void mlx5dr_ste_build_eth_l3_ipv4_5_tuple(struct mlx5dr_ste_build *sb,
281+ struct mlx5dr_match_param *mask,
282+ bool inner, bool rx);
283+void mlx5dr_ste_build_eth_l3_ipv4_misc(struct mlx5dr_ste_build *sb,
284+ struct mlx5dr_match_param *mask,
285+ bool inner, bool rx);
286+void mlx5dr_ste_build_eth_l3_ipv6_dst(struct mlx5dr_ste_build *sb,
287+ struct mlx5dr_match_param *mask,
288+ bool inner, bool rx);
289+void mlx5dr_ste_build_eth_l3_ipv6_src(struct mlx5dr_ste_build *sb,
290+ struct mlx5dr_match_param *mask,
291+ bool inner, bool rx);
292+void mlx5dr_ste_build_eth_l2_src(struct mlx5dr_ste_build *sb,
293+ struct mlx5dr_match_param *mask,
294+ bool inner, bool rx);
295+void mlx5dr_ste_build_eth_l2_dst(struct mlx5dr_ste_build *sb,
296+ struct mlx5dr_match_param *mask,
297+ bool inner, bool rx);
298+void mlx5dr_ste_build_eth_l2_tnl(struct mlx5dr_ste_build *sb,
299+ struct mlx5dr_match_param *mask,
300+ bool inner, bool rx);
301+void mlx5dr_ste_build_ipv6_l3_l4(struct mlx5dr_ste_build *sb,
302+ struct mlx5dr_match_param *mask,
303+ bool inner, bool rx);
304+void mlx5dr_ste_build_eth_l4_misc(struct mlx5dr_ste_build *sb,
305+ struct mlx5dr_match_param *mask,
306+ bool inner, bool rx);
307+void mlx5dr_ste_build_gre(struct mlx5dr_ste_build *sb,
308+ struct mlx5dr_match_param *mask,
309+ bool inner, bool rx);
310+void mlx5dr_ste_build_mpls(struct mlx5dr_ste_build *sb,
311+ struct mlx5dr_match_param *mask,
312+ bool inner, bool rx);
313+void mlx5dr_ste_build_flex_parser_0(struct mlx5dr_ste_build *sb,
314+ struct mlx5dr_match_param *mask,
315+ bool inner, bool rx);
316+int mlx5dr_ste_build_flex_parser_1(struct mlx5dr_ste_build *sb,
317+ struct mlx5dr_match_param *mask,
318+ struct mlx5dr_cmd_caps *caps,
319+ bool inner, bool rx);
320+void mlx5dr_ste_build_flex_parser_tnl(struct mlx5dr_ste_build *sb,
321+ struct mlx5dr_match_param *mask,
322+ bool inner, bool rx);
323+void mlx5dr_ste_build_general_purpose(struct mlx5dr_ste_build *sb,
324+ struct mlx5dr_match_param *mask,
325+ bool inner, bool rx);
326+void mlx5dr_ste_build_register_0(struct mlx5dr_ste_build *sb,
327+ struct mlx5dr_match_param *mask,
328+ bool inner, bool rx);
329+void mlx5dr_ste_build_register_1(struct mlx5dr_ste_build *sb,
330+ struct mlx5dr_match_param *mask,
331+ bool inner, bool rx);
332+int mlx5dr_ste_build_src_gvmi_qpn(struct mlx5dr_ste_build *sb,
333+ struct mlx5dr_match_param *mask,
334+ struct mlx5dr_cmd_caps *caps,
335+ bool inner, bool rx);
336+void mlx5dr_ste_build_empty_always_hit(struct mlx5dr_ste_build *sb, bool rx);
337+
338+/* Actions utils */
339+int mlx5dr_actions_build_ste_arr(struct mlx5dr_matcher *matcher,
340+ struct mlx5dr_matcher_rx_tx *nic_matcher,
341+ struct mlx5dr_action *actions[],
342+ u32 num_actions,
343+ u8 *ste_arr,
344+ u32 *new_hw_ste_arr_sz);
345+
346+struct mlx5dr_match_spec {
347+ u32 smac_47_16; /* Source MAC address of incoming packet */
348+ /* Incoming packet Ethertype - this is the Ethertype
349+ * following the last VLAN tag of the packet
350+ */
351+ u32 ethertype:16;
352+ u32 smac_15_0:16; /* Source MAC address of incoming packet */
353+ u32 dmac_47_16; /* Destination MAC address of incoming packet */
354+ /* VLAN ID of first VLAN tag in the incoming packet.
355+ * Valid only when cvlan_tag==1 or svlan_tag==1
356+ */
357+ u32 first_vid:12;
358+ /* CFI bit of first VLAN tag in the incoming packet.
359+ * Valid only when cvlan_tag==1 or svlan_tag==1
360+ */
361+ u32 first_cfi:1;
362+ /* Priority of first VLAN tag in the incoming packet.
363+ * Valid only when cvlan_tag==1 or svlan_tag==1
364+ */
365+ u32 first_prio:3;
366+ u32 dmac_15_0:16; /* Destination MAC address of incoming packet */
367+ /* TCP flags. ;Bit 0: FIN;Bit 1: SYN;Bit 2: RST;Bit 3: PSH;Bit 4: ACK;
368+ * Bit 5: URG;Bit 6: ECE;Bit 7: CWR;Bit 8: NS
369+ */
370+ u32 tcp_flags:9;
371+ u32 ip_version:4; /* IP version */
372+ u32 frag:1; /* Packet is an IP fragment */
373+ /* The first vlan in the packet is s-vlan (0x8a88).
374+ * cvlan_tag and svlan_tag cannot be set together
375+ */
376+ u32 svlan_tag:1;
377+ /* The first vlan in the packet is c-vlan (0x8100).
378+ * cvlan_tag and svlan_tag cannot be set together
379+ */
380+ u32 cvlan_tag:1;
381+ /* Explicit Congestion Notification derived from
382+ * Traffic Class/TOS field of IPv6/v4
383+ */
384+ u32 ip_ecn:2;
385+ /* Differentiated Services Code Point derived from
386+ * Traffic Class/TOS field of IPv6/v4
387+ */
388+ u32 ip_dscp:6;
389+ u32 ip_protocol:8; /* IP protocol */
390+ /* TCP destination port.
391+ * tcp and udp sport/dport are mutually exclusive
392+ */
393+ u32 tcp_dport:16;
394+ /* TCP source port.;tcp and udp sport/dport are mutually exclusive */
395+ u32 tcp_sport:16;
396+ u32 ttl_hoplimit:8;
397+ u32 reserved:24;
398+ /* UDP destination port.;tcp and udp sport/dport are mutually exclusive */
399+ u32 udp_dport:16;
400+ /* UDP source port.;tcp and udp sport/dport are mutually exclusive */
401+ u32 udp_sport:16;
402+ /* IPv6 source address of incoming packets
403+ * For IPv4 address use bits 31:0 (rest of the bits are reserved)
404+ * This field should be qualified by an appropriate ethertype
405+ */
406+ u32 src_ip_127_96;
407+ /* IPv6 source address of incoming packets
408+ * For IPv4 address use bits 31:0 (rest of the bits are reserved)
409+ * This field should be qualified by an appropriate ethertype
410+ */
411+ u32 src_ip_95_64;
412+ /* IPv6 source address of incoming packets
413+ * For IPv4 address use bits 31:0 (rest of the bits are reserved)
414+ * This field should be qualified by an appropriate ethertype
415+ */
416+ u32 src_ip_63_32;
417+ /* IPv6 source address of incoming packets
418+ * For IPv4 address use bits 31:0 (rest of the bits are reserved)
419+ * This field should be qualified by an appropriate ethertype
420+ */
421+ u32 src_ip_31_0;
422+ /* IPv6 destination address of incoming packets
423+ * For IPv4 address use bits 31:0 (rest of the bits are reserved)
424+ * This field should be qualified by an appropriate ethertype
425+ */
426+ u32 dst_ip_127_96;
427+ /* IPv6 destination address of incoming packets
428+ * For IPv4 address use bits 31:0 (rest of the bits are reserved)
429+ * This field should be qualified by an appropriate ethertype
430+ */
431+ u32 dst_ip_95_64;
432+ /* IPv6 destination address of incoming packets
433+ * For IPv4 address use bits 31:0 (rest of the bits are reserved)
434+ * This field should be qualified by an appropriate ethertype
435+ */
436+ u32 dst_ip_63_32;
437+ /* IPv6 destination address of incoming packets
438+ * For IPv4 address use bits 31:0 (rest of the bits are reserved)
439+ * This field should be qualified by an appropriate ethertype
440+ */
441+ u32 dst_ip_31_0;
442+};
443+
444+struct mlx5dr_match_misc {
445+ u32 source_sqn:24; /* Source SQN */
446+ u32 source_vhca_port:4;
447+ /* used with GRE, sequence number exist when gre_s_present == 1 */
448+ u32 gre_s_present:1;
449+ /* used with GRE, key exist when gre_k_present == 1 */
450+ u32 gre_k_present:1;
451+ u32 reserved_auto1:1;
452+ /* used with GRE, checksum exist when gre_c_present == 1 */
453+ u32 gre_c_present:1;
454+ /* Source port.;0xffff determines wire port */
455+ u32 source_port:16;
456+ u32 reserved_auto2:16;
457+ /* VLAN ID of first VLAN tag the inner header of the incoming packet.
458+ * Valid only when inner_second_cvlan_tag ==1 or inner_second_svlan_tag ==1
459+ */
460+ u32 inner_second_vid:12;
461+ /* CFI bit of first VLAN tag in the inner header of the incoming packet.
462+ * Valid only when inner_second_cvlan_tag ==1 or inner_second_svlan_tag ==1
463+ */
464+ u32 inner_second_cfi:1;
465+ /* Priority of second VLAN tag in the inner header of the incoming packet.
466+ * Valid only when inner_second_cvlan_tag ==1 or inner_second_svlan_tag ==1
467+ */
468+ u32 inner_second_prio:3;
469+ /* VLAN ID of first VLAN tag the outer header of the incoming packet.
470+ * Valid only when outer_second_cvlan_tag ==1 or outer_second_svlan_tag ==1
471+ */
472+ u32 outer_second_vid:12;
473+ /* CFI bit of first VLAN tag in the outer header of the incoming packet.
474+ * Valid only when outer_second_cvlan_tag ==1 or outer_second_svlan_tag ==1
475+ */
476+ u32 outer_second_cfi:1;
477+ /* Priority of second VLAN tag in the outer header of the incoming packet.
478+ * Valid only when outer_second_cvlan_tag ==1 or outer_second_svlan_tag ==1
479+ */
480+ u32 outer_second_prio:3;
481+ u32 gre_protocol:16; /* GRE Protocol (outer) */
482+ u32 reserved_auto3:12;
483+ /* The second vlan in the inner header of the packet is s-vlan (0x8a88).
484+ * inner_second_cvlan_tag and inner_second_svlan_tag cannot be set together
485+ */
486+ u32 inner_second_svlan_tag:1;
487+ /* The second vlan in the outer header of the packet is s-vlan (0x8a88).
488+ * outer_second_cvlan_tag and outer_second_svlan_tag cannot be set together
489+ */
490+ u32 outer_second_svlan_tag:1;
491+ /* The second vlan in the inner header of the packet is c-vlan (0x8100).
492+ * inner_second_cvlan_tag and inner_second_svlan_tag cannot be set together
493+ */
494+ u32 inner_second_cvlan_tag:1;
495+ /* The second vlan in the outer header of the packet is c-vlan (0x8100).
496+ * outer_second_cvlan_tag and outer_second_svlan_tag cannot be set together
497+ */
498+ u32 outer_second_cvlan_tag:1;
499+ u32 gre_key_l:8; /* GRE Key [7:0] (outer) */
500+ u32 gre_key_h:24; /* GRE Key[31:8] (outer) */
501+ u32 reserved_auto4:8;
502+ u32 vxlan_vni:24; /* VXLAN VNI (outer) */
503+ u32 geneve_oam:1; /* GENEVE OAM field (outer) */
504+ u32 reserved_auto5:7;
505+ u32 geneve_vni:24; /* GENEVE VNI field (outer) */
506+ u32 outer_ipv6_flow_label:20; /* Flow label of incoming IPv6 packet (outer) */
507+ u32 reserved_auto6:12;
508+ u32 inner_ipv6_flow_label:20; /* Flow label of incoming IPv6 packet (inner) */
509+ u32 reserved_auto7:12;
510+ u32 geneve_protocol_type:16; /* GENEVE protocol type (outer) */
511+ u32 geneve_opt_len:6; /* GENEVE OptLen (outer) */
512+ u32 reserved_auto8:10;
513+ u32 bth_dst_qp:24; /* Destination QP in BTH header */
514+ u32 reserved_auto9:8;
515+ u8 reserved_auto10[20];
516+};
517+
518+struct mlx5dr_match_misc2 {
519+ u32 outer_first_mpls_ttl:8; /* First MPLS TTL (outer) */
520+ u32 outer_first_mpls_s_bos:1; /* First MPLS S_BOS (outer) */
521+ u32 outer_first_mpls_exp:3; /* First MPLS EXP (outer) */
522+ u32 outer_first_mpls_label:20; /* First MPLS LABEL (outer) */
523+ u32 inner_first_mpls_ttl:8; /* First MPLS TTL (inner) */
524+ u32 inner_first_mpls_s_bos:1; /* First MPLS S_BOS (inner) */
525+ u32 inner_first_mpls_exp:3; /* First MPLS EXP (inner) */
526+ u32 inner_first_mpls_label:20; /* First MPLS LABEL (inner) */
527+ u32 outer_first_mpls_over_gre_ttl:8; /* last MPLS TTL (outer) */
528+ u32 outer_first_mpls_over_gre_s_bos:1; /* last MPLS S_BOS (outer) */
529+ u32 outer_first_mpls_over_gre_exp:3; /* last MPLS EXP (outer) */
530+ u32 outer_first_mpls_over_gre_label:20; /* last MPLS LABEL (outer) */
531+ u32 outer_first_mpls_over_udp_ttl:8; /* last MPLS TTL (outer) */
532+ u32 outer_first_mpls_over_udp_s_bos:1; /* last MPLS S_BOS (outer) */
533+ u32 outer_first_mpls_over_udp_exp:3; /* last MPLS EXP (outer) */
534+ u32 outer_first_mpls_over_udp_label:20; /* last MPLS LABEL (outer) */
535+ u32 metadata_reg_c_7; /* metadata_reg_c_7 */
536+ u32 metadata_reg_c_6; /* metadata_reg_c_6 */
537+ u32 metadata_reg_c_5; /* metadata_reg_c_5 */
538+ u32 metadata_reg_c_4; /* metadata_reg_c_4 */
539+ u32 metadata_reg_c_3; /* metadata_reg_c_3 */
540+ u32 metadata_reg_c_2; /* metadata_reg_c_2 */
541+ u32 metadata_reg_c_1; /* metadata_reg_c_1 */
542+ u32 metadata_reg_c_0; /* metadata_reg_c_0 */
543+ u32 metadata_reg_a; /* metadata_reg_a */
544+ u32 metadata_reg_b; /* metadata_reg_b */
545+ u8 reserved_auto2[8];
546+};
547+
548+struct mlx5dr_match_misc3 {
549+ u32 inner_tcp_seq_num;
550+ u32 outer_tcp_seq_num;
551+ u32 inner_tcp_ack_num;
552+ u32 outer_tcp_ack_num;
553+ u32 outer_vxlan_gpe_vni:24;
554+ u32 reserved_auto1:8;
555+ u32 reserved_auto2:16;
556+ u32 outer_vxlan_gpe_flags:8;
557+ u32 outer_vxlan_gpe_next_protocol:8;
558+ u32 icmpv4_header_data;
559+ u32 icmpv6_header_data;
560+ u32 icmpv6_code:8;
561+ u32 icmpv6_type:8;
562+ u32 icmpv4_code:8;
563+ u32 icmpv4_type:8;
564+ u8 reserved_auto3[0x1c];
565+};
566+
567+struct mlx5dr_match_param {
568+ struct mlx5dr_match_spec outer;
569+ struct mlx5dr_match_misc misc;
570+ struct mlx5dr_match_spec inner;
571+ struct mlx5dr_match_misc2 misc2;
572+ struct mlx5dr_match_misc3 misc3;
573+};
574+
575+#define DR_MASK_IS_FLEX_PARSER_ICMPV4_SET(_misc3) ((_misc3)->icmpv4_type || \
576+ (_misc3)->icmpv4_code || \
577+ (_misc3)->icmpv4_header_data)
578+
579+struct mlx5dr_esw_caps {
580+ u64 drop_icm_address_rx;
581+ u64 drop_icm_address_tx;
582+ u64 uplink_icm_address_rx;
583+ u64 uplink_icm_address_tx;
584+ bool sw_owner;
585+};
586+
587+struct mlx5dr_cmd_vport_cap {
588+ u16 vport_gvmi;
589+ u16 vhca_gvmi;
590+ u64 icm_address_rx;
591+ u64 icm_address_tx;
592+ u32 num;
593+};
594+
595+struct mlx5dr_cmd_caps {
596+ u16 gvmi;
597+ u64 nic_rx_drop_address;
598+ u64 nic_tx_drop_address;
599+ u64 nic_tx_allow_address;
600+ u64 esw_rx_drop_address;
601+ u64 esw_tx_drop_address;
602+ u32 log_icm_size;
603+ u64 hdr_modify_icm_addr;
604+ u32 flex_protocols;
605+ u8 flex_parser_id_icmp_dw0;
606+ u8 flex_parser_id_icmp_dw1;
607+ u8 flex_parser_id_icmpv6_dw0;
608+ u8 flex_parser_id_icmpv6_dw1;
609+ u8 max_ft_level;
610+ u16 roce_min_src_udp;
611+ u8 num_esw_ports;
612+ bool eswitch_manager;
613+ bool rx_sw_owner;
614+ bool tx_sw_owner;
615+ bool fdb_sw_owner;
616+ u32 num_vports;
617+ struct mlx5dr_esw_caps esw_caps;
618+ struct mlx5dr_cmd_vport_cap *vports_caps;
619+ bool prio_tag_required;
620+};
621+
622+struct mlx5dr_domain_rx_tx {
623+ u64 drop_icm_addr;
624+ u64 default_icm_addr;
625+ enum mlx5dr_ste_entry_type ste_type;
626+};
627+
628+struct mlx5dr_domain_info {
629+ bool supp_sw_steering;
630+ u32 max_inline_size;
631+ u32 max_send_wr;
632+ u32 max_log_sw_icm_sz;
633+ u32 max_log_action_icm_sz;
634+ struct mlx5dr_domain_rx_tx rx;
635+ struct mlx5dr_domain_rx_tx tx;
636+ struct mlx5dr_cmd_caps caps;
637+};
638+
639+struct mlx5dr_domain_cache {
640+ struct mlx5dr_fw_recalc_cs_ft **recalc_cs_ft;
641+};
642+
643+struct mlx5dr_domain {
644+ struct mlx5dr_domain *peer_dmn;
645+ struct mlx5_core_dev *mdev;
646+ u32 pdn;
647+ struct mlx5_uars_page *uar;
648+ enum mlx5dr_domain_type type;
649+ refcount_t refcount;
650+ struct mutex mutex; /* protect domain */
651+ struct mlx5dr_icm_pool *ste_icm_pool;
652+ struct mlx5dr_icm_pool *action_icm_pool;
653+ struct mlx5dr_send_ring *send_ring;
654+ struct mlx5dr_domain_info info;
655+ struct mlx5dr_domain_cache cache;
656+};
657+
658+struct mlx5dr_table_rx_tx {
659+ struct mlx5dr_ste_htbl *s_anchor;
660+ struct mlx5dr_domain_rx_tx *nic_dmn;
661+ u64 default_icm_addr;
662+};
663+
664+struct mlx5dr_table {
665+ struct mlx5dr_domain *dmn;
666+ struct mlx5dr_table_rx_tx rx;
667+ struct mlx5dr_table_rx_tx tx;
668+ u32 level;
669+ u32 table_type;
670+ u32 table_id;
671+ struct list_head matcher_list;
672+ struct mlx5dr_action *miss_action;
673+ refcount_t refcount;
674+};
675+
676+struct mlx5dr_matcher_rx_tx {
677+ struct mlx5dr_ste_htbl *s_htbl;
678+ struct mlx5dr_ste_htbl *e_anchor;
679+ struct mlx5dr_ste_build *ste_builder;
680+ struct mlx5dr_ste_build ste_builder4[DR_RULE_MAX_STES];
681+ struct mlx5dr_ste_build ste_builder6[DR_RULE_MAX_STES];
682+ u8 num_of_builders;
683+ u8 num_of_builders4;
684+ u8 num_of_builders6;
685+ u64 default_icm_addr;
686+ struct mlx5dr_table_rx_tx *nic_tbl;
687+};
688+
689+struct mlx5dr_matcher {
690+ struct mlx5dr_table *tbl;
691+ struct mlx5dr_matcher_rx_tx rx;
692+ struct mlx5dr_matcher_rx_tx tx;
693+ struct list_head matcher_list;
694+ u16 prio;
695+ struct mlx5dr_match_param mask;
696+ u8 match_criteria;
697+ refcount_t refcount;
698+ struct mlx5dv_flow_matcher *dv_matcher;
699+};
700+
701+struct mlx5dr_rule_member {
702+ struct mlx5dr_ste *ste;
703+ /* attached to mlx5dr_rule via this */
704+ struct list_head list;
705+ /* attached to mlx5dr_ste via this */
706+ struct list_head use_ste_list;
707+};
708+
709+struct mlx5dr_action {
710+ enum mlx5dr_action_type action_type;
711+ refcount_t refcount;
712+ union {
713+ struct {
714+ struct mlx5dr_domain *dmn;
715+ struct mlx5dr_icm_chunk *chunk;
716+ u8 *data;
717+ u32 data_size;
718+ u16 num_of_actions;
719+ u32 index;
720+ u8 allow_rx:1;
721+ u8 allow_tx:1;
722+ u8 modify_ttl:1;
723+ } rewrite;
724+ struct {
725+ struct mlx5dr_domain *dmn;
726+ u32 reformat_id;
727+ u32 reformat_size;
728+ } reformat;
729+ struct {
730+ u8 is_fw_tbl:1;
731+ union {
732+ struct mlx5dr_table *tbl;
733+ struct {
734+ struct mlx5_flow_table *ft;
735+ u64 rx_icm_addr;
736+ u64 tx_icm_addr;
737+ struct mlx5_core_dev *mdev;
738+ } fw_tbl;
739+ };
740+ } dest_tbl;
741+ struct {
742+ u32 ctr_id;
743+ u32 offeset;
744+ } ctr;
745+ struct {
746+ struct mlx5dr_domain *dmn;
747+ struct mlx5dr_cmd_vport_cap *caps;
748+ u32 num;
749+ } vport;
750+ struct {
751+ u32 vlan_hdr; /* tpid_pcp_dei_vid */
752+ } push_vlan;
753+ u32 flow_tag;
754+ };
755+};
756+
757+enum mlx5dr_connect_type {
758+ CONNECT_HIT = 1,
759+ CONNECT_MISS = 2,
760+};
761+
762+struct mlx5dr_htbl_connect_info {
763+ enum mlx5dr_connect_type type;
764+ union {
765+ struct mlx5dr_ste_htbl *hit_next_htbl;
766+ u64 miss_icm_addr;
767+ };
768+};
769+
770+struct mlx5dr_rule_rx_tx {
771+ struct list_head rule_members_list;
772+ struct mlx5dr_matcher_rx_tx *nic_matcher;
773+};
774+
775+struct mlx5dr_rule {
776+ struct mlx5dr_matcher *matcher;
777+ struct mlx5dr_rule_rx_tx rx;
778+ struct mlx5dr_rule_rx_tx tx;
779+ struct list_head rule_actions_list;
780+};
781+
782+void mlx5dr_rule_update_rule_member(struct mlx5dr_ste *new_ste,
783+ struct mlx5dr_ste *ste);
784+
785+struct mlx5dr_icm_chunk {
786+ struct mlx5dr_icm_bucket *bucket;
787+ struct list_head chunk_list;
788+ u32 rkey;
789+ u32 num_of_entries;
790+ u32 byte_size;
791+ u64 icm_addr;
792+ u64 mr_addr;
793+
794+ /* Memory optimisation */
795+ struct mlx5dr_ste *ste_arr;
796+ u8 *hw_ste_arr;
797+ struct list_head *miss_list;
798+};
799+
800+static inline int
801+mlx5dr_matcher_supp_flex_parser_icmp_v4(struct mlx5dr_cmd_caps *caps)
802+{
803+ return caps->flex_protocols & MLX5_FLEX_PARSER_ICMP_V4_ENABLED;
804+}
805+
806+static inline int
807+mlx5dr_matcher_supp_flex_parser_icmp_v6(struct mlx5dr_cmd_caps *caps)
808+{
809+ return caps->flex_protocols & MLX5_FLEX_PARSER_ICMP_V6_ENABLED;
810+}
811+
812+int mlx5dr_matcher_select_builders(struct mlx5dr_matcher *matcher,
813+ struct mlx5dr_matcher_rx_tx *nic_matcher,
814+ bool ipv6);
815+
816+static inline u32
817+mlx5dr_icm_pool_chunk_size_to_entries(enum mlx5dr_icm_chunk_size chunk_size)
818+{
819+ return 1 << chunk_size;
820+}
821+
822+static inline int
823+mlx5dr_icm_pool_chunk_size_to_byte(enum mlx5dr_icm_chunk_size chunk_size,
824+ enum mlx5dr_icm_type icm_type)
825+{
826+ int num_of_entries;
827+ int entry_size;
828+
829+ if (icm_type == DR_ICM_TYPE_STE)
830+ entry_size = DR_STE_SIZE;
831+ else
832+ entry_size = DR_MODIFY_ACTION_SIZE;
833+
834+ num_of_entries = mlx5dr_icm_pool_chunk_size_to_entries(chunk_size);
835+
836+ return entry_size * num_of_entries;
837+}
838+
839+static inline struct mlx5dr_cmd_vport_cap *
840+mlx5dr_get_vport_cap(struct mlx5dr_cmd_caps *caps, u32 vport)
841+{
842+ if (!caps->vports_caps ||
843+ (vport >= caps->num_vports && vport != WIRE_PORT))
844+ return NULL;
845+
846+ if (vport == WIRE_PORT)
847+ vport = caps->num_vports;
848+
849+ return &caps->vports_caps[vport];
850+}
851+
852+struct mlx5dr_cmd_query_flow_table_details {
853+ u8 status;
854+ u8 level;
855+ u64 sw_owner_icm_root_1;
856+ u64 sw_owner_icm_root_0;
857+};
858+
859+/* internal API functions */
860+int mlx5dr_cmd_query_device(struct mlx5_core_dev *mdev,
861+ struct mlx5dr_cmd_caps *caps);
862+int mlx5dr_cmd_query_esw_vport_context(struct mlx5_core_dev *mdev,
863+ bool other_vport, u16 vport_number,
864+ u64 *icm_address_rx,
865+ u64 *icm_address_tx);
866+int mlx5dr_cmd_query_gvmi(struct mlx5_core_dev *mdev,
867+ bool other_vport, u16 vport_number, u16 *gvmi);
868+int mlx5dr_cmd_query_esw_caps(struct mlx5_core_dev *mdev,
869+ struct mlx5dr_esw_caps *caps);
870+int mlx5dr_cmd_sync_steering(struct mlx5_core_dev *mdev);
871+int mlx5dr_cmd_set_fte_modify_and_vport(struct mlx5_core_dev *mdev,
872+ u32 table_type,
873+ u32 table_id,
874+ u32 group_id,
875+ u32 modify_header_id,
876+ u32 vport_id);
877+int mlx5dr_cmd_del_flow_table_entry(struct mlx5_core_dev *mdev,
878+ u32 table_type,
879+ u32 table_id);
880+int mlx5dr_cmd_alloc_modify_header(struct mlx5_core_dev *mdev,
881+ u32 table_type,
882+ u8 num_of_actions,
883+ u64 *actions,
884+ u32 *modify_header_id);
885+int mlx5dr_cmd_dealloc_modify_header(struct mlx5_core_dev *mdev,
886+ u32 modify_header_id);
887+int mlx5dr_cmd_create_empty_flow_group(struct mlx5_core_dev *mdev,
888+ u32 table_type,
889+ u32 table_id,
890+ u32 *group_id);
891+int mlx5dr_cmd_destroy_flow_group(struct mlx5_core_dev *mdev,
892+ u32 table_type,
893+ u32 table_id,
894+ u32 group_id);
895+int mlx5dr_cmd_create_flow_table(struct mlx5_core_dev *mdev,
896+ u32 table_type,
897+ u64 icm_addr_rx,
898+ u64 icm_addr_tx,
899+ u8 level,
900+ bool sw_owner,
901+ bool term_tbl,
902+ u64 *fdb_rx_icm_addr,
903+ u32 *table_id);
904+int mlx5dr_cmd_destroy_flow_table(struct mlx5_core_dev *mdev,
905+ u32 table_id,
906+ u32 table_type);
907+int mlx5dr_cmd_query_flow_table(struct mlx5_core_dev *dev,
908+ enum fs_flow_table_type type,
909+ u32 table_id,
910+ struct mlx5dr_cmd_query_flow_table_details *output);
911+int mlx5dr_cmd_create_reformat_ctx(struct mlx5_core_dev *mdev,
912+ enum mlx5_reformat_ctx_type rt,
913+ size_t reformat_size,
914+ void *reformat_data,
915+ u32 *reformat_id);
916+void mlx5dr_cmd_destroy_reformat_ctx(struct mlx5_core_dev *mdev,
917+ u32 reformat_id);
918+
919+struct mlx5dr_cmd_gid_attr {
920+ u8 gid[16];
921+ u8 mac[6];
922+ u32 roce_ver;
923+};
924+
925+struct mlx5dr_cmd_qp_create_attr {
926+ u32 page_id;
927+ u32 pdn;
928+ u32 cqn;
929+ u32 pm_state;
930+ u32 service_type;
931+ u32 buff_umem_id;
932+ u32 db_umem_id;
933+ u32 sq_wqe_cnt;
934+ u32 rq_wqe_cnt;
935+ u32 rq_wqe_shift;
936+};
937+
938+int mlx5dr_cmd_query_gid(struct mlx5_core_dev *mdev, u8 vhca_port_num,
939+ u16 index, struct mlx5dr_cmd_gid_attr *attr);
940+
941+struct mlx5dr_icm_pool *mlx5dr_icm_pool_create(struct mlx5dr_domain *dmn,
942+ enum mlx5dr_icm_type icm_type);
943+void mlx5dr_icm_pool_destroy(struct mlx5dr_icm_pool *pool);
944+
945+struct mlx5dr_icm_chunk *
946+mlx5dr_icm_alloc_chunk(struct mlx5dr_icm_pool *pool,
947+ enum mlx5dr_icm_chunk_size chunk_size);
948+void mlx5dr_icm_free_chunk(struct mlx5dr_icm_chunk *chunk);
949+bool mlx5dr_ste_is_not_valid_entry(u8 *p_hw_ste);
950+int mlx5dr_ste_htbl_init_and_postsend(struct mlx5dr_domain *dmn,
951+ struct mlx5dr_domain_rx_tx *nic_dmn,
952+ struct mlx5dr_ste_htbl *htbl,
953+ struct mlx5dr_htbl_connect_info *connect_info,
954+ bool update_hw_ste);
955+void mlx5dr_ste_set_formatted_ste(u16 gvmi,
956+ struct mlx5dr_domain_rx_tx *nic_dmn,
957+ struct mlx5dr_ste_htbl *htbl,
958+ u8 *formatted_ste,
959+ struct mlx5dr_htbl_connect_info *connect_info);
960+void mlx5dr_ste_copy_param(u8 match_criteria,
961+ struct mlx5dr_match_param *set_param,
962+ struct mlx5dr_match_parameters *mask);
963+
964+void mlx5dr_crc32_init_table(void);
965+u32 mlx5dr_crc32_slice8_calc(const void *input_data, size_t length);
966+
967+struct mlx5dr_qp {
968+ struct mlx5_core_dev *mdev;
969+ struct mlx5_wq_qp wq;
970+ struct mlx5_uars_page *uar;
971+ struct mlx5_wq_ctrl wq_ctrl;
972+ struct mlx5_core_qp mqp;
973+ struct {
974+ unsigned int pc;
975+ unsigned int cc;
976+ unsigned int size;
977+ unsigned int *wqe_head;
978+ unsigned int wqe_cnt;
979+ } sq;
980+ struct {
981+ unsigned int pc;
982+ unsigned int cc;
983+ unsigned int size;
984+ unsigned int wqe_cnt;
985+ } rq;
986+ int max_inline_data;
987+};
988+
989+struct mlx5dr_cq {
990+ struct mlx5_core_dev *mdev;
991+ struct mlx5_cqwq wq;
992+ struct mlx5_wq_ctrl wq_ctrl;
993+ struct mlx5_core_cq mcq;
994+ struct mlx5dr_qp *qp;
995+};
996+
997+struct mlx5dr_mr {
998+ struct mlx5_core_dev *mdev;
999+ struct mlx5_core_mkey mkey;
1000+ dma_addr_t dma_addr;
1001+ void *addr;
1002+ size_t size;
1003+};
1004+
1005+#define MAX_SEND_CQE 64
1006+#define MIN_READ_SYNC 64
1007+
1008+struct mlx5dr_send_ring {
1009+ struct mlx5dr_cq *cq;
1010+ struct mlx5dr_qp *qp;
1011+ struct mlx5dr_mr *mr;
1012+ /* How much wqes are waiting for completion */
1013+ u32 pending_wqe;
1014+ /* Signal request per this trash hold value */
1015+ u16 signal_th;
1016+ /* Each post_send_size less than max_post_send_size */
1017+ u32 max_post_send_size;
1018+ /* manage the send queue */
1019+ u32 tx_head;
1020+ void *buf;
1021+ u32 buf_size;
1022+ struct ib_wc wc[MAX_SEND_CQE];
1023+ u8 sync_buff[MIN_READ_SYNC];
1024+ struct mlx5dr_mr *sync_mr;
1025+};
1026+
1027+int mlx5dr_send_ring_alloc(struct mlx5dr_domain *dmn);
1028+void mlx5dr_send_ring_free(struct mlx5dr_domain *dmn,
1029+ struct mlx5dr_send_ring *send_ring);
1030+int mlx5dr_send_ring_force_drain(struct mlx5dr_domain *dmn);
1031+int mlx5dr_send_postsend_ste(struct mlx5dr_domain *dmn,
1032+ struct mlx5dr_ste *ste,
1033+ u8 *data,
1034+ u16 size,
1035+ u16 offset);
1036+int mlx5dr_send_postsend_htbl(struct mlx5dr_domain *dmn,
1037+ struct mlx5dr_ste_htbl *htbl,
1038+ u8 *formatted_ste, u8 *mask);
1039+int mlx5dr_send_postsend_formatted_htbl(struct mlx5dr_domain *dmn,
1040+ struct mlx5dr_ste_htbl *htbl,
1041+ u8 *ste_init_data,
1042+ bool update_hw_ste);
1043+int mlx5dr_send_postsend_action(struct mlx5dr_domain *dmn,
1044+ struct mlx5dr_action *action);
1045+
1046+struct mlx5dr_fw_recalc_cs_ft {
1047+ u64 rx_icm_addr;
1048+ u32 table_id;
1049+ u32 group_id;
1050+ u32 modify_hdr_id;
1051+};
1052+
1053+struct mlx5dr_fw_recalc_cs_ft *
1054+mlx5dr_fw_create_recalc_cs_ft(struct mlx5dr_domain *dmn, u32 vport_num);
1055+void mlx5dr_fw_destroy_recalc_cs_ft(struct mlx5dr_domain *dmn,
1056+ struct mlx5dr_fw_recalc_cs_ft *recalc_cs_ft);
1057+int mlx5dr_domain_cache_get_recalc_cs_ft_addr(struct mlx5dr_domain *dmn,
1058+ u32 vport_num,
1059+ u64 *rx_icm_addr);
1060+#endif /* _DR_TYPES_H_ */
drivers/net/ethernet/mellanox/mlx5/core/steering/fs_dr.c+600−0
@@ -0,0 +1,600 @@
1+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+/* Copyright (c) 2019 Mellanox Technologies */
3+
4+#include "mlx5_core.h"
5+#include "fs_core.h"
6+#include "fs_cmd.h"
7+#include "mlx5dr.h"
8+#include "fs_dr.h"
9+
10+static bool mlx5_dr_is_fw_table(u32 flags)
11+{
12+ if (flags & MLX5_FLOW_TABLE_TERMINATION)
13+ return true;
14+
15+ return false;
16+}
17+
18+static int mlx5_cmd_dr_update_root_ft(struct mlx5_flow_root_namespace *ns,
19+ struct mlx5_flow_table *ft,
20+ u32 underlay_qpn,
21+ bool disconnect)
22+{
23+ return mlx5_fs_cmd_get_fw_cmds()->update_root_ft(ns, ft, underlay_qpn,
24+ disconnect);
25+}
26+
27+static int set_miss_action(struct mlx5_flow_root_namespace *ns,
28+ struct mlx5_flow_table *ft,
29+ struct mlx5_flow_table *next_ft)
30+{
31+ struct mlx5dr_action *old_miss_action;
32+ struct mlx5dr_action *action = NULL;
33+ struct mlx5dr_table *next_tbl;
34+ int err;
35+
36+ next_tbl = next_ft ? next_ft->fs_dr_table.dr_table : NULL;
37+ if (next_tbl) {
38+ action = mlx5dr_action_create_dest_table(next_tbl);
39+ if (!action)
40+ return -EINVAL;
41+ }
42+ old_miss_action = ft->fs_dr_table.miss_action;
43+ err = mlx5dr_table_set_miss_action(ft->fs_dr_table.dr_table, action);
44+ if (err && action) {
45+ err = mlx5dr_action_destroy(action);
46+ if (err) {
47+ action = NULL;
48+ mlx5_core_err(ns->dev, "Failed to destroy action (%d)\n",
49+ err);
50+ }
51+ }
52+ ft->fs_dr_table.miss_action = action;
53+ if (old_miss_action) {
54+ err = mlx5dr_action_destroy(old_miss_action);
55+ if (err)
56+ mlx5_core_err(ns->dev, "Failed to destroy action (%d)\n",
57+ err);
58+ }
59+
60+ return err;
61+}
62+
63+static int mlx5_cmd_dr_create_flow_table(struct mlx5_flow_root_namespace *ns,
64+ struct mlx5_flow_table *ft,
65+ unsigned int log_size,
66+ struct mlx5_flow_table *next_ft)
67+{
68+ struct mlx5dr_table *tbl;
69+ int err;
70+
71+ if (mlx5_dr_is_fw_table(ft->flags))
72+ return mlx5_fs_cmd_get_fw_cmds()->create_flow_table(ns, ft,
73+ log_size,
74+ next_ft);
75+
76+ tbl = mlx5dr_table_create(ns->fs_dr_domain.dr_domain,
77+ ft->level);
78+ if (!tbl) {
79+ mlx5_core_err(ns->dev, "Failed creating dr flow_table\n");
80+ return -EINVAL;
81+ }
82+
83+ ft->fs_dr_table.dr_table = tbl;
84+ ft->id = mlx5dr_table_get_id(tbl);
85+
86+ if (next_ft) {
87+ err = set_miss_action(ns, ft, next_ft);
88+ if (err) {
89+ mlx5dr_table_destroy(tbl);
90+ ft->fs_dr_table.dr_table = NULL;
91+ return err;
92+ }
93+ }
94+
95+ return 0;
96+}
97+
98+static int mlx5_cmd_dr_destroy_flow_table(struct mlx5_flow_root_namespace *ns,
99+ struct mlx5_flow_table *ft)
100+{
101+ struct mlx5dr_action *action = ft->fs_dr_table.miss_action;
102+ int err;
103+
104+ if (mlx5_dr_is_fw_table(ft->flags))
105+ return mlx5_fs_cmd_get_fw_cmds()->destroy_flow_table(ns, ft);
106+
107+ err = mlx5dr_table_destroy(ft->fs_dr_table.dr_table);
108+ if (err) {
109+ mlx5_core_err(ns->dev, "Failed to destroy flow_table (%d)\n",
110+ err);
111+ return err;
112+ }
113+ if (action) {
114+ err = mlx5dr_action_destroy(action);
115+ if (err) {
116+ mlx5_core_err(ns->dev, "Failed to destroy action(%d)\n",
117+ err);
118+ return err;
119+ }
120+ }
121+
122+ return err;
123+}
124+
125+static int mlx5_cmd_dr_modify_flow_table(struct mlx5_flow_root_namespace *ns,
126+ struct mlx5_flow_table *ft,
127+ struct mlx5_flow_table *next_ft)
128+{
129+ return set_miss_action(ns, ft, next_ft);
130+}
131+
132+static int mlx5_cmd_dr_create_flow_group(struct mlx5_flow_root_namespace *ns,
133+ struct mlx5_flow_table *ft,
134+ u32 *in,
135+ struct mlx5_flow_group *fg)
136+{
137+ struct mlx5dr_matcher *matcher;
138+ u16 priority = MLX5_GET(create_flow_group_in, in,
139+ start_flow_index);
140+ u8 match_criteria_enable = MLX5_GET(create_flow_group_in,
141+ in,
142+ match_criteria_enable);
143+ struct mlx5dr_match_parameters mask;
144+
145+ if (mlx5_dr_is_fw_table(ft->flags))
146+ return mlx5_fs_cmd_get_fw_cmds()->create_flow_group(ns, ft, in,
147+ fg);
148+
149+ mask.match_buf = MLX5_ADDR_OF(create_flow_group_in,
150+ in, match_criteria);
151+ mask.match_sz = sizeof(fg->mask.match_criteria);
152+
153+ matcher = mlx5dr_matcher_create(ft->fs_dr_table.dr_table,
154+ priority,
155+ match_criteria_enable,
156+ &mask);
157+ if (!matcher) {
158+ mlx5_core_err(ns->dev, "Failed creating matcher\n");
159+ return -EINVAL;
160+ }
161+
162+ fg->fs_dr_matcher.dr_matcher = matcher;
163+ return 0;
164+}
165+
166+static int mlx5_cmd_dr_destroy_flow_group(struct mlx5_flow_root_namespace *ns,
167+ struct mlx5_flow_table *ft,
168+ struct mlx5_flow_group *fg)
169+{
170+ if (mlx5_dr_is_fw_table(ft->flags))
171+ return mlx5_fs_cmd_get_fw_cmds()->destroy_flow_group(ns, ft, fg);
172+
173+ return mlx5dr_matcher_destroy(fg->fs_dr_matcher.dr_matcher);
174+}
175+
176+static struct mlx5dr_action *create_vport_action(struct mlx5dr_domain *domain,
177+ struct mlx5_flow_rule *dst)
178+{
179+ struct mlx5_flow_destination *dest_attr = &dst->dest_attr;
180+
181+ return mlx5dr_action_create_dest_vport(domain, dest_attr->vport.num,
182+ dest_attr->vport.flags &
183+ MLX5_FLOW_DEST_VPORT_VHCA_ID,
184+ dest_attr->vport.vhca_id);
185+}
186+
187+static struct mlx5dr_action *create_ft_action(struct mlx5_core_dev *dev,
188+ struct mlx5_flow_rule *dst)
189+{
190+ struct mlx5_flow_table *dest_ft = dst->dest_attr.ft;
191+
192+ if (mlx5_dr_is_fw_table(dest_ft->flags))
193+ return mlx5dr_create_action_dest_flow_fw_table(dest_ft, dev);
194+ return mlx5dr_action_create_dest_table(dest_ft->fs_dr_table.dr_table);
195+}
196+
197+static struct mlx5dr_action *create_action_push_vlan(struct mlx5dr_domain *domain,
198+ struct mlx5_fs_vlan *vlan)
199+{
200+ u16 n_ethtype = vlan->ethtype;
201+ u8 prio = vlan->prio;
202+ u16 vid = vlan->vid;
203+ u32 vlan_hdr;
204+
205+ vlan_hdr = (u32)n_ethtype << 16 | (u32)(prio) << 12 | (u32)vid;
206+ return mlx5dr_action_create_push_vlan(domain, htonl(vlan_hdr));
207+}
208+
209+#define MLX5_FLOW_CONTEXT_ACTION_MAX 20
210+static int mlx5_cmd_dr_create_fte(struct mlx5_flow_root_namespace *ns,
211+ struct mlx5_flow_table *ft,
212+ struct mlx5_flow_group *group,
213+ struct fs_fte *fte)
214+{
215+ struct mlx5dr_domain *domain = ns->fs_dr_domain.dr_domain;
216+ struct mlx5dr_action *term_action = NULL;
217+ struct mlx5dr_match_parameters params;
218+ struct mlx5_core_dev *dev = ns->dev;
219+ struct mlx5dr_action **fs_dr_actions;
220+ struct mlx5dr_action *tmp_action;
221+ struct mlx5dr_action **actions;
222+ bool delay_encap_set = false;
223+ struct mlx5dr_rule *rule;
224+ struct mlx5_flow_rule *dst;
225+ int fs_dr_num_actions = 0;
226+ int num_actions = 0;
227+ size_t match_sz;
228+ int err = 0;
229+ int i;
230+
231+ if (mlx5_dr_is_fw_table(ft->flags))
232+ return mlx5_fs_cmd_get_fw_cmds()->create_fte(ns, ft, group, fte);
233+
234+ actions = kcalloc(MLX5_FLOW_CONTEXT_ACTION_MAX, sizeof(*actions),
235+ GFP_KERNEL);
236+ if (!actions)
237+ return -ENOMEM;
238+
239+ fs_dr_actions = kcalloc(MLX5_FLOW_CONTEXT_ACTION_MAX,
240+ sizeof(*fs_dr_actions), GFP_KERNEL);
241+ if (!fs_dr_actions) {
242+ kfree(actions);
243+ return -ENOMEM;
244+ }
245+
246+ match_sz = sizeof(fte->val);
247+
248+ /* The order of the actions are must to be keep, only the following
249+ * order is supported by SW steering:
250+ * TX: push vlan -> modify header -> encap
251+ * RX: decap -> pop vlan -> modify header
252+ */
253+ if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_VLAN_PUSH) {
254+ tmp_action = create_action_push_vlan(domain, &fte->action.vlan[0]);
255+ if (!tmp_action) {
256+ err = -ENOMEM;
257+ goto free_actions;
258+ }
259+ fs_dr_actions[fs_dr_num_actions++] = tmp_action;
260+ actions[num_actions++] = tmp_action;
261+ }
262+
263+ if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_VLAN_PUSH_2) {
264+ tmp_action = create_action_push_vlan(domain, &fte->action.vlan[1]);
265+ if (!tmp_action) {
266+ err = -ENOMEM;
267+ goto free_actions;
268+ }
269+ fs_dr_actions[fs_dr_num_actions++] = tmp_action;
270+ actions[num_actions++] = tmp_action;
271+ }
272+
273+ if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_DECAP) {
274+ enum mlx5dr_action_reformat_type decap_type =
275+ DR_ACTION_REFORMAT_TYP_TNL_L2_TO_L2;
276+
277+ tmp_action = mlx5dr_action_create_packet_reformat(domain,
278+ decap_type, 0,
279+ NULL);
280+ if (!tmp_action) {
281+ err = -ENOMEM;
282+ goto free_actions;
283+ }
284+ fs_dr_actions[fs_dr_num_actions++] = tmp_action;
285+ actions[num_actions++] = tmp_action;
286+ }
287+
288+ if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT) {
289+ bool is_decap = fte->action.pkt_reformat->reformat_type ==
290+ MLX5_REFORMAT_TYPE_L3_TUNNEL_TO_L2;
291+
292+ if (is_decap)
293+ actions[num_actions++] =
294+ fte->action.pkt_reformat->action.dr_action;
295+ else
296+ delay_encap_set = true;
297+ }
298+
299+ if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_VLAN_POP) {
300+ tmp_action =
301+ mlx5dr_action_create_pop_vlan();
302+ if (!tmp_action) {
303+ err = -ENOMEM;
304+ goto free_actions;
305+ }
306+ fs_dr_actions[fs_dr_num_actions++] = tmp_action;
307+ actions[num_actions++] = tmp_action;
308+ }
309+
310+ if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_VLAN_POP_2) {
311+ tmp_action =
312+ mlx5dr_action_create_pop_vlan();
313+ if (!tmp_action) {
314+ err = -ENOMEM;
315+ goto free_actions;
316+ }
317+ fs_dr_actions[fs_dr_num_actions++] = tmp_action;
318+ actions[num_actions++] = tmp_action;
319+ }
320+
321+ if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)
322+ actions[num_actions++] =
323+ fte->action.modify_hdr->action.dr_action;
324+
325+ if (delay_encap_set)
326+ actions[num_actions++] =
327+ fte->action.pkt_reformat->action.dr_action;
328+
329+ /* The order of the actions below is not important */
330+
331+ if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_DROP) {
332+ tmp_action = mlx5dr_action_create_drop();
333+ if (!tmp_action) {
334+ err = -ENOMEM;
335+ goto free_actions;
336+ }
337+ fs_dr_actions[fs_dr_num_actions++] = tmp_action;
338+ term_action = tmp_action;
339+ }
340+
341+ if (fte->flow_context.flow_tag) {
342+ tmp_action =
343+ mlx5dr_action_create_tag(fte->flow_context.flow_tag);
344+ if (!tmp_action) {
345+ err = -ENOMEM;
346+ goto free_actions;
347+ }
348+ fs_dr_actions[fs_dr_num_actions++] = tmp_action;
349+ actions[num_actions++] = tmp_action;
350+ }
351+
352+ if (fte->action.action & MLX5_FLOW_CONTEXT_ACTION_FWD_DEST) {
353+ list_for_each_entry(dst, &fte->node.children, node.list) {
354+ enum mlx5_flow_destination_type type = dst->dest_attr.type;
355+ u32 id;
356+
357+ if (num_actions == MLX5_FLOW_CONTEXT_ACTION_MAX) {
358+ err = -ENOSPC;
359+ goto free_actions;
360+ }
361+
362+ switch (type) {
363+ case MLX5_FLOW_DESTINATION_TYPE_COUNTER:
364+ id = dst->dest_attr.counter_id;
365+
366+ tmp_action =
367+ mlx5dr_action_create_flow_counter(id);
368+ if (!tmp_action) {
369+ err = -ENOMEM;
370+ goto free_actions;
371+ }
372+ fs_dr_actions[fs_dr_num_actions++] = tmp_action;
373+ actions[num_actions++] = tmp_action;
374+ break;
375+ case MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE:
376+ tmp_action = create_ft_action(dev, dst);
377+ if (!tmp_action) {
378+ err = -ENOMEM;
379+ goto free_actions;
380+ }
381+ fs_dr_actions[fs_dr_num_actions++] = tmp_action;
382+ term_action = tmp_action;
383+ break;
384+ case MLX5_FLOW_DESTINATION_TYPE_VPORT:
385+ tmp_action = create_vport_action(domain, dst);
386+ if (!tmp_action) {
387+ err = -ENOMEM;
388+ goto free_actions;
389+ }
390+ fs_dr_actions[fs_dr_num_actions++] = tmp_action;
391+ term_action = tmp_action;
392+ break;
393+ default:
394+ err = -EOPNOTSUPP;
395+ goto free_actions;
396+ }
397+ }
398+ }
399+
400+ params.match_sz = match_sz;
401+ params.match_buf = (u64 *)fte->val;
402+
403+ if (term_action)
404+ actions[num_actions++] = term_action;
405+
406+ rule = mlx5dr_rule_create(group->fs_dr_matcher.dr_matcher,
407+ &params,
408+ num_actions,
409+ actions);
410+ if (!rule) {
411+ err = -EINVAL;
412+ goto free_actions;
413+ }
414+
415+ kfree(actions);
416+ fte->fs_dr_rule.dr_rule = rule;
417+ fte->fs_dr_rule.num_actions = fs_dr_num_actions;
418+ fte->fs_dr_rule.dr_actions = fs_dr_actions;
419+
420+ return 0;
421+
422+free_actions:
423+ for (i = 0; i < fs_dr_num_actions; i++)
424+ if (!IS_ERR_OR_NULL(fs_dr_actions[i]))
425+ mlx5dr_action_destroy(fs_dr_actions[i]);
426+
427+ mlx5_core_err(dev, "Failed to create dr rule err(%d)\n", err);
428+ kfree(actions);
429+ kfree(fs_dr_actions);
430+ return err;
431+}
432+
433+static int mlx5_cmd_dr_packet_reformat_alloc(struct mlx5_flow_root_namespace *ns,
434+ int reformat_type,
435+ size_t size,
436+ void *reformat_data,
437+ enum mlx5_flow_namespace_type namespace,
438+ struct mlx5_pkt_reformat *pkt_reformat)
439+{
440+ struct mlx5dr_domain *dr_domain = ns->fs_dr_domain.dr_domain;
441+ struct mlx5dr_action *action;
442+ int dr_reformat;
443+
444+ switch (reformat_type) {
445+ case MLX5_REFORMAT_TYPE_L2_TO_VXLAN:
446+ case MLX5_REFORMAT_TYPE_L2_TO_NVGRE:
447+ case MLX5_REFORMAT_TYPE_L2_TO_L2_TUNNEL:
448+ dr_reformat = DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L2;
449+ break;
450+ case MLX5_REFORMAT_TYPE_L3_TUNNEL_TO_L2:
451+ dr_reformat = DR_ACTION_REFORMAT_TYP_TNL_L3_TO_L2;
452+ break;
453+ case MLX5_REFORMAT_TYPE_L2_TO_L3_TUNNEL:
454+ dr_reformat = DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L3;
455+ break;
456+ default:
457+ mlx5_core_err(ns->dev, "Packet-reformat not supported(%d)\n",
458+ reformat_type);
459+ return -EOPNOTSUPP;
460+ }
461+
462+ action = mlx5dr_action_create_packet_reformat(dr_domain,
463+ dr_reformat,
464+ size,
465+ reformat_data);
466+ if (!action) {
467+ mlx5_core_err(ns->dev, "Failed allocating packet-reformat action\n");
468+ return -EINVAL;
469+ }
470+
471+ pkt_reformat->action.dr_action = action;
472+
473+ return 0;
474+}
475+
476+static void mlx5_cmd_dr_packet_reformat_dealloc(struct mlx5_flow_root_namespace *ns,
477+ struct mlx5_pkt_reformat *pkt_reformat)
478+{
479+ mlx5dr_action_destroy(pkt_reformat->action.dr_action);
480+}
481+
482+static int mlx5_cmd_dr_modify_header_alloc(struct mlx5_flow_root_namespace *ns,
483+ u8 namespace, u8 num_actions,
484+ void *modify_actions,
485+ struct mlx5_modify_hdr *modify_hdr)
486+{
487+ struct mlx5dr_domain *dr_domain = ns->fs_dr_domain.dr_domain;
488+ struct mlx5dr_action *action;
489+ size_t actions_sz;
490+
491+ actions_sz = MLX5_UN_SZ_BYTES(set_action_in_add_action_in_auto) *
492+ num_actions;
493+ action = mlx5dr_action_create_modify_header(dr_domain, 0,
494+ actions_sz,
495+ modify_actions);
496+ if (!action) {
497+ mlx5_core_err(ns->dev, "Failed allocating modify-header action\n");
498+ return -EINVAL;
499+ }
500+
501+ modify_hdr->action.dr_action = action;
502+
503+ return 0;
504+}
505+
506+static void mlx5_cmd_dr_modify_header_dealloc(struct mlx5_flow_root_namespace *ns,
507+ struct mlx5_modify_hdr *modify_hdr)
508+{
509+ mlx5dr_action_destroy(modify_hdr->action.dr_action);
510+}
511+
512+static int mlx5_cmd_dr_update_fte(struct mlx5_flow_root_namespace *ns,
513+ struct mlx5_flow_table *ft,
514+ struct mlx5_flow_group *group,
515+ int modify_mask,
516+ struct fs_fte *fte)
517+{
518+ return -EOPNOTSUPP;
519+}
520+
521+static int mlx5_cmd_dr_delete_fte(struct mlx5_flow_root_namespace *ns,
522+ struct mlx5_flow_table *ft,
523+ struct fs_fte *fte)
524+{
525+ struct mlx5_fs_dr_rule *rule = &fte->fs_dr_rule;
526+ int err;
527+ int i;
528+
529+ if (mlx5_dr_is_fw_table(ft->flags))
530+ return mlx5_fs_cmd_get_fw_cmds()->delete_fte(ns, ft, fte);
531+
532+ err = mlx5dr_rule_destroy(rule->dr_rule);
533+ if (err)
534+ return err;
535+
536+ for (i = 0; i < rule->num_actions; i++)
537+ if (!IS_ERR_OR_NULL(rule->dr_actions[i]))
538+ mlx5dr_action_destroy(rule->dr_actions[i]);
539+
540+ kfree(rule->dr_actions);
541+ return 0;
542+}
543+
544+static int mlx5_cmd_dr_set_peer(struct mlx5_flow_root_namespace *ns,
545+ struct mlx5_flow_root_namespace *peer_ns)
546+{
547+ struct mlx5dr_domain *peer_domain = NULL;
548+
549+ if (peer_ns)
550+ peer_domain = peer_ns->fs_dr_domain.dr_domain;
551+ mlx5dr_domain_set_peer(ns->fs_dr_domain.dr_domain,
552+ peer_domain);
553+ return 0;
554+}
555+
556+static int mlx5_cmd_dr_create_ns(struct mlx5_flow_root_namespace *ns)
557+{
558+ ns->fs_dr_domain.dr_domain =
559+ mlx5dr_domain_create(ns->dev,
560+ MLX5DR_DOMAIN_TYPE_FDB);
561+ if (!ns->fs_dr_domain.dr_domain) {
562+ mlx5_core_err(ns->dev, "Failed to create dr flow namespace\n");
563+ return -EOPNOTSUPP;
564+ }
565+ return 0;
566+}
567+
568+static int mlx5_cmd_dr_destroy_ns(struct mlx5_flow_root_namespace *ns)
569+{
570+ return mlx5dr_domain_destroy(ns->fs_dr_domain.dr_domain);
571+}
572+
573+bool mlx5_fs_dr_is_supported(struct mlx5_core_dev *dev)
574+{
575+ return mlx5dr_is_supported(dev);
576+}
577+
578+static const struct mlx5_flow_cmds mlx5_flow_cmds_dr = {
579+ .create_flow_table = mlx5_cmd_dr_create_flow_table,
580+ .destroy_flow_table = mlx5_cmd_dr_destroy_flow_table,
581+ .modify_flow_table = mlx5_cmd_dr_modify_flow_table,
582+ .create_flow_group = mlx5_cmd_dr_create_flow_group,
583+ .destroy_flow_group = mlx5_cmd_dr_destroy_flow_group,
584+ .create_fte = mlx5_cmd_dr_create_fte,
585+ .update_fte = mlx5_cmd_dr_update_fte,
586+ .delete_fte = mlx5_cmd_dr_delete_fte,
587+ .update_root_ft = mlx5_cmd_dr_update_root_ft,
588+ .packet_reformat_alloc = mlx5_cmd_dr_packet_reformat_alloc,
589+ .packet_reformat_dealloc = mlx5_cmd_dr_packet_reformat_dealloc,
590+ .modify_header_alloc = mlx5_cmd_dr_modify_header_alloc,
591+ .modify_header_dealloc = mlx5_cmd_dr_modify_header_dealloc,
592+ .set_peer = mlx5_cmd_dr_set_peer,
593+ .create_ns = mlx5_cmd_dr_create_ns,
594+ .destroy_ns = mlx5_cmd_dr_destroy_ns,
595+};
596+
597+const struct mlx5_flow_cmds *mlx5_fs_cmd_get_dr_cmds(void)
598+{
599+ return &mlx5_flow_cmds_dr;
600+}
drivers/net/ethernet/mellanox/mlx5/core/steering/fs_dr.h+60−0
@@ -0,0 +1,60 @@
1+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2+ * Copyright (c) 2019 Mellanox Technologies
3+ */
4+
5+#ifndef _MLX5_FS_DR_
6+#define _MLX5_FS_DR_
7+
8+#include "mlx5dr.h"
9+
10+struct mlx5_flow_root_namespace;
11+struct fs_fte;
12+
13+struct mlx5_fs_dr_action {
14+ struct mlx5dr_action *dr_action;
15+};
16+
17+struct mlx5_fs_dr_ns {
18+ struct mlx5_dr_ns *dr_ns;
19+};
20+
21+struct mlx5_fs_dr_rule {
22+ struct mlx5dr_rule *dr_rule;
23+ /* Only actions created by fs_dr */
24+ struct mlx5dr_action **dr_actions;
25+ int num_actions;
26+};
27+
28+struct mlx5_fs_dr_domain {
29+ struct mlx5dr_domain *dr_domain;
30+};
31+
32+struct mlx5_fs_dr_matcher {
33+ struct mlx5dr_matcher *dr_matcher;
34+};
35+
36+struct mlx5_fs_dr_table {
37+ struct mlx5dr_table *dr_table;
38+ struct mlx5dr_action *miss_action;
39+};
40+
41+#ifdef CONFIG_MLX5_SW_STEERING
42+
43+bool mlx5_fs_dr_is_supported(struct mlx5_core_dev *dev);
44+
45+const struct mlx5_flow_cmds *mlx5_fs_cmd_get_dr_cmds(void);
46+
47+#else
48+
49+static inline const struct mlx5_flow_cmds *mlx5_fs_cmd_get_dr_cmds(void)
50+{
51+ return NULL;
52+}
53+
54+static inline bool mlx5_fs_dr_is_supported(struct mlx5_core_dev *dev)
55+{
56+ return false;
57+}
58+
59+#endif /* CONFIG_MLX5_SW_STEERING */
60+#endif
drivers/net/ethernet/mellanox/mlx5/core/steering/mlx5_ifc_dr.h+604−0
@@ -0,0 +1,604 @@
1+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
2+/* Copyright (c) 2019, Mellanox Technologies */
3+
4+#ifndef MLX5_IFC_DR_H
5+#define MLX5_IFC_DR_H
6+
7+enum {
8+ MLX5DR_ACTION_MDFY_HW_FLD_L2_0 = 0,
9+ MLX5DR_ACTION_MDFY_HW_FLD_L2_1 = 1,
10+ MLX5DR_ACTION_MDFY_HW_FLD_L2_2 = 2,
11+ MLX5DR_ACTION_MDFY_HW_FLD_L3_0 = 3,
12+ MLX5DR_ACTION_MDFY_HW_FLD_L3_1 = 4,
13+ MLX5DR_ACTION_MDFY_HW_FLD_L3_2 = 5,
14+ MLX5DR_ACTION_MDFY_HW_FLD_L3_3 = 6,
15+ MLX5DR_ACTION_MDFY_HW_FLD_L3_4 = 7,
16+ MLX5DR_ACTION_MDFY_HW_FLD_L4_0 = 8,
17+ MLX5DR_ACTION_MDFY_HW_FLD_L4_1 = 9,
18+ MLX5DR_ACTION_MDFY_HW_FLD_MPLS = 10,
19+ MLX5DR_ACTION_MDFY_HW_FLD_L2_TNL_0 = 11,
20+ MLX5DR_ACTION_MDFY_HW_FLD_REG_0 = 12,
21+ MLX5DR_ACTION_MDFY_HW_FLD_REG_1 = 13,
22+ MLX5DR_ACTION_MDFY_HW_FLD_REG_2 = 14,
23+ MLX5DR_ACTION_MDFY_HW_FLD_REG_3 = 15,
24+ MLX5DR_ACTION_MDFY_HW_FLD_L4_2 = 16,
25+ MLX5DR_ACTION_MDFY_HW_FLD_FLEX_0 = 17,
26+ MLX5DR_ACTION_MDFY_HW_FLD_FLEX_1 = 18,
27+ MLX5DR_ACTION_MDFY_HW_FLD_FLEX_2 = 19,
28+ MLX5DR_ACTION_MDFY_HW_FLD_FLEX_3 = 20,
29+ MLX5DR_ACTION_MDFY_HW_FLD_L2_TNL_1 = 21,
30+ MLX5DR_ACTION_MDFY_HW_FLD_METADATA = 22,
31+ MLX5DR_ACTION_MDFY_HW_FLD_RESERVED = 23,
32+};
33+
34+enum {
35+ MLX5DR_ACTION_MDFY_HW_OP_SET = 0x2,
36+ MLX5DR_ACTION_MDFY_HW_OP_ADD = 0x3,
37+};
38+
39+enum {
40+ MLX5DR_ACTION_MDFY_HW_HDR_L3_NONE = 0x0,
41+ MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV4 = 0x1,
42+ MLX5DR_ACTION_MDFY_HW_HDR_L3_IPV6 = 0x2,
43+};
44+
45+enum {
46+ MLX5DR_ACTION_MDFY_HW_HDR_L4_NONE = 0x0,
47+ MLX5DR_ACTION_MDFY_HW_HDR_L4_TCP = 0x1,
48+ MLX5DR_ACTION_MDFY_HW_HDR_L4_UDP = 0x2,
49+};
50+
51+enum {
52+ MLX5DR_STE_LU_TYPE_NOP = 0x00,
53+ MLX5DR_STE_LU_TYPE_SRC_GVMI_AND_QP = 0x05,
54+ MLX5DR_STE_LU_TYPE_ETHL2_TUNNELING_I = 0x0a,
55+ MLX5DR_STE_LU_TYPE_ETHL2_DST_O = 0x06,
56+ MLX5DR_STE_LU_TYPE_ETHL2_DST_I = 0x07,
57+ MLX5DR_STE_LU_TYPE_ETHL2_DST_D = 0x1b,
58+ MLX5DR_STE_LU_TYPE_ETHL2_SRC_O = 0x08,
59+ MLX5DR_STE_LU_TYPE_ETHL2_SRC_I = 0x09,
60+ MLX5DR_STE_LU_TYPE_ETHL2_SRC_D = 0x1c,
61+ MLX5DR_STE_LU_TYPE_ETHL2_SRC_DST_O = 0x36,
62+ MLX5DR_STE_LU_TYPE_ETHL2_SRC_DST_I = 0x37,
63+ MLX5DR_STE_LU_TYPE_ETHL2_SRC_DST_D = 0x38,
64+ MLX5DR_STE_LU_TYPE_ETHL3_IPV6_DST_O = 0x0d,
65+ MLX5DR_STE_LU_TYPE_ETHL3_IPV6_DST_I = 0x0e,
66+ MLX5DR_STE_LU_TYPE_ETHL3_IPV6_DST_D = 0x1e,
67+ MLX5DR_STE_LU_TYPE_ETHL3_IPV6_SRC_O = 0x0f,
68+ MLX5DR_STE_LU_TYPE_ETHL3_IPV6_SRC_I = 0x10,
69+ MLX5DR_STE_LU_TYPE_ETHL3_IPV6_SRC_D = 0x1f,
70+ MLX5DR_STE_LU_TYPE_ETHL3_IPV4_5_TUPLE_O = 0x11,
71+ MLX5DR_STE_LU_TYPE_ETHL3_IPV4_5_TUPLE_I = 0x12,
72+ MLX5DR_STE_LU_TYPE_ETHL3_IPV4_5_TUPLE_D = 0x20,
73+ MLX5DR_STE_LU_TYPE_ETHL3_IPV4_MISC_O = 0x29,
74+ MLX5DR_STE_LU_TYPE_ETHL3_IPV4_MISC_I = 0x2a,
75+ MLX5DR_STE_LU_TYPE_ETHL3_IPV4_MISC_D = 0x2b,
76+ MLX5DR_STE_LU_TYPE_ETHL4_O = 0x13,
77+ MLX5DR_STE_LU_TYPE_ETHL4_I = 0x14,
78+ MLX5DR_STE_LU_TYPE_ETHL4_D = 0x21,
79+ MLX5DR_STE_LU_TYPE_ETHL4_MISC_O = 0x2c,
80+ MLX5DR_STE_LU_TYPE_ETHL4_MISC_I = 0x2d,
81+ MLX5DR_STE_LU_TYPE_ETHL4_MISC_D = 0x2e,
82+ MLX5DR_STE_LU_TYPE_MPLS_FIRST_O = 0x15,
83+ MLX5DR_STE_LU_TYPE_MPLS_FIRST_I = 0x24,
84+ MLX5DR_STE_LU_TYPE_MPLS_FIRST_D = 0x25,
85+ MLX5DR_STE_LU_TYPE_GRE = 0x16,
86+ MLX5DR_STE_LU_TYPE_FLEX_PARSER_0 = 0x22,
87+ MLX5DR_STE_LU_TYPE_FLEX_PARSER_1 = 0x23,
88+ MLX5DR_STE_LU_TYPE_FLEX_PARSER_TNL_HEADER = 0x19,
89+ MLX5DR_STE_LU_TYPE_GENERAL_PURPOSE = 0x18,
90+ MLX5DR_STE_LU_TYPE_STEERING_REGISTERS_0 = 0x2f,
91+ MLX5DR_STE_LU_TYPE_STEERING_REGISTERS_1 = 0x30,
92+ MLX5DR_STE_LU_TYPE_DONT_CARE = 0x0f,
93+};
94+
95+enum mlx5dr_ste_entry_type {
96+ MLX5DR_STE_TYPE_TX = 1,
97+ MLX5DR_STE_TYPE_RX = 2,
98+ MLX5DR_STE_TYPE_MODIFY_PKT = 6,
99+};
100+
101+struct mlx5_ifc_ste_general_bits {
102+ u8 entry_type[0x4];
103+ u8 reserved_at_4[0x4];
104+ u8 entry_sub_type[0x8];
105+ u8 byte_mask[0x10];
106+
107+ u8 next_table_base_63_48[0x10];
108+ u8 next_lu_type[0x8];
109+ u8 next_table_base_39_32_size[0x8];
110+
111+ u8 next_table_base_31_5_size[0x1b];
112+ u8 linear_hash_enable[0x1];
113+ u8 reserved_at_5c[0x2];
114+ u8 next_table_rank[0x2];
115+
116+ u8 reserved_at_60[0xa0];
117+ u8 tag_value[0x60];
118+ u8 bit_mask[0x60];
119+};
120+
121+struct mlx5_ifc_ste_sx_transmit_bits {
122+ u8 entry_type[0x4];
123+ u8 reserved_at_4[0x4];
124+ u8 entry_sub_type[0x8];
125+ u8 byte_mask[0x10];
126+
127+ u8 next_table_base_63_48[0x10];
128+ u8 next_lu_type[0x8];
129+ u8 next_table_base_39_32_size[0x8];
130+
131+ u8 next_table_base_31_5_size[0x1b];
132+ u8 linear_hash_enable[0x1];
133+ u8 reserved_at_5c[0x2];
134+ u8 next_table_rank[0x2];
135+
136+ u8 sx_wire[0x1];
137+ u8 sx_func_lb[0x1];
138+ u8 sx_sniffer[0x1];
139+ u8 sx_wire_enable[0x1];
140+ u8 sx_func_lb_enable[0x1];
141+ u8 sx_sniffer_enable[0x1];
142+ u8 action_type[0x3];
143+ u8 reserved_at_69[0x1];
144+ u8 action_description[0x6];
145+ u8 gvmi[0x10];
146+
147+ u8 encap_pointer_vlan_data[0x20];
148+
149+ u8 loopback_syndome_en[0x8];
150+ u8 loopback_syndome[0x8];
151+ u8 counter_trigger[0x10];
152+
153+ u8 miss_address_63_48[0x10];
154+ u8 counter_trigger_23_16[0x8];
155+ u8 miss_address_39_32[0x8];
156+
157+ u8 miss_address_31_6[0x1a];
158+ u8 learning_point[0x1];
159+ u8 go_back[0x1];
160+ u8 match_polarity[0x1];
161+ u8 mask_mode[0x1];
162+ u8 miss_rank[0x2];
163+};
164+
165+struct mlx5_ifc_ste_rx_steering_mult_bits {
166+ u8 entry_type[0x4];
167+ u8 reserved_at_4[0x4];
168+ u8 entry_sub_type[0x8];
169+ u8 byte_mask[0x10];
170+
171+ u8 next_table_base_63_48[0x10];
172+ u8 next_lu_type[0x8];
173+ u8 next_table_base_39_32_size[0x8];
174+
175+ u8 next_table_base_31_5_size[0x1b];
176+ u8 linear_hash_enable[0x1];
177+ u8 reserved_at_[0x2];
178+ u8 next_table_rank[0x2];
179+
180+ u8 member_count[0x10];
181+ u8 gvmi[0x10];
182+
183+ u8 qp_list_pointer[0x20];
184+
185+ u8 reserved_at_a0[0x1];
186+ u8 tunneling_action[0x3];
187+ u8 action_description[0x4];
188+ u8 reserved_at_a8[0x8];
189+ u8 counter_trigger_15_0[0x10];
190+
191+ u8 miss_address_63_48[0x10];
192+ u8 counter_trigger_23_16[0x08];
193+ u8 miss_address_39_32[0x8];
194+
195+ u8 miss_address_31_6[0x1a];
196+ u8 learning_point[0x1];
197+ u8 fail_on_error[0x1];
198+ u8 match_polarity[0x1];
199+ u8 mask_mode[0x1];
200+ u8 miss_rank[0x2];
201+};
202+
203+struct mlx5_ifc_ste_modify_packet_bits {
204+ u8 entry_type[0x4];
205+ u8 reserved_at_4[0x4];
206+ u8 entry_sub_type[0x8];
207+ u8 byte_mask[0x10];
208+
209+ u8 next_table_base_63_48[0x10];
210+ u8 next_lu_type[0x8];
211+ u8 next_table_base_39_32_size[0x8];
212+
213+ u8 next_table_base_31_5_size[0x1b];
214+ u8 linear_hash_enable[0x1];
215+ u8 reserved_at_[0x2];
216+ u8 next_table_rank[0x2];
217+
218+ u8 number_of_re_write_actions[0x10];
219+ u8 gvmi[0x10];
220+
221+ u8 header_re_write_actions_pointer[0x20];
222+
223+ u8 reserved_at_a0[0x1];
224+ u8 tunneling_action[0x3];
225+ u8 action_description[0x4];
226+ u8 reserved_at_a8[0x8];
227+ u8 counter_trigger_15_0[0x10];
228+
229+ u8 miss_address_63_48[0x10];
230+ u8 counter_trigger_23_16[0x08];
231+ u8 miss_address_39_32[0x8];
232+
233+ u8 miss_address_31_6[0x1a];
234+ u8 learning_point[0x1];
235+ u8 fail_on_error[0x1];
236+ u8 match_polarity[0x1];
237+ u8 mask_mode[0x1];
238+ u8 miss_rank[0x2];
239+};
240+
241+struct mlx5_ifc_ste_eth_l2_src_bits {
242+ u8 smac_47_16[0x20];
243+
244+ u8 smac_15_0[0x10];
245+ u8 l3_ethertype[0x10];
246+
247+ u8 qp_type[0x2];
248+ u8 ethertype_filter[0x1];
249+ u8 reserved_at_43[0x1];
250+ u8 sx_sniffer[0x1];
251+ u8 force_lb[0x1];
252+ u8 functional_lb[0x1];
253+ u8 port[0x1];
254+ u8 reserved_at_48[0x4];
255+ u8 first_priority[0x3];
256+ u8 first_cfi[0x1];
257+ u8 first_vlan_qualifier[0x2];
258+ u8 reserved_at_52[0x2];
259+ u8 first_vlan_id[0xc];
260+
261+ u8 ip_fragmented[0x1];
262+ u8 tcp_syn[0x1];
263+ u8 encp_type[0x2];
264+ u8 l3_type[0x2];
265+ u8 l4_type[0x2];
266+ u8 reserved_at_68[0x4];
267+ u8 second_priority[0x3];
268+ u8 second_cfi[0x1];
269+ u8 second_vlan_qualifier[0x2];
270+ u8 reserved_at_72[0x2];
271+ u8 second_vlan_id[0xc];
272+};
273+
274+struct mlx5_ifc_ste_eth_l2_dst_bits {
275+ u8 dmac_47_16[0x20];
276+
277+ u8 dmac_15_0[0x10];
278+ u8 l3_ethertype[0x10];
279+
280+ u8 qp_type[0x2];
281+ u8 ethertype_filter[0x1];
282+ u8 reserved_at_43[0x1];
283+ u8 sx_sniffer[0x1];
284+ u8 force_lb[0x1];
285+ u8 functional_lb[0x1];
286+ u8 port[0x1];
287+ u8 reserved_at_48[0x4];
288+ u8 first_priority[0x3];
289+ u8 first_cfi[0x1];
290+ u8 first_vlan_qualifier[0x2];
291+ u8 reserved_at_52[0x2];
292+ u8 first_vlan_id[0xc];
293+
294+ u8 ip_fragmented[0x1];
295+ u8 tcp_syn[0x1];
296+ u8 encp_type[0x2];
297+ u8 l3_type[0x2];
298+ u8 l4_type[0x2];
299+ u8 reserved_at_68[0x4];
300+ u8 second_priority[0x3];
301+ u8 second_cfi[0x1];
302+ u8 second_vlan_qualifier[0x2];
303+ u8 reserved_at_72[0x2];
304+ u8 second_vlan_id[0xc];
305+};
306+
307+struct mlx5_ifc_ste_eth_l2_src_dst_bits {
308+ u8 dmac_47_16[0x20];
309+
310+ u8 dmac_15_0[0x10];
311+ u8 smac_47_32[0x10];
312+
313+ u8 smac_31_0[0x20];
314+
315+ u8 sx_sniffer[0x1];
316+ u8 force_lb[0x1];
317+ u8 functional_lb[0x1];
318+ u8 port[0x1];
319+ u8 l3_type[0x2];
320+ u8 reserved_at_66[0x6];
321+ u8 first_priority[0x3];
322+ u8 first_cfi[0x1];
323+ u8 first_vlan_qualifier[0x2];
324+ u8 reserved_at_72[0x2];
325+ u8 first_vlan_id[0xc];
326+};
327+
328+struct mlx5_ifc_ste_eth_l3_ipv4_5_tuple_bits {
329+ u8 destination_address[0x20];
330+
331+ u8 source_address[0x20];
332+
333+ u8 source_port[0x10];
334+ u8 destination_port[0x10];
335+
336+ u8 fragmented[0x1];
337+ u8 first_fragment[0x1];
338+ u8 reserved_at_62[0x2];
339+ u8 reserved_at_64[0x1];
340+ u8 ecn[0x2];
341+ u8 tcp_ns[0x1];
342+ u8 tcp_cwr[0x1];
343+ u8 tcp_ece[0x1];
344+ u8 tcp_urg[0x1];
345+ u8 tcp_ack[0x1];
346+ u8 tcp_psh[0x1];
347+ u8 tcp_rst[0x1];
348+ u8 tcp_syn[0x1];
349+ u8 tcp_fin[0x1];
350+ u8 dscp[0x6];
351+ u8 reserved_at_76[0x2];
352+ u8 protocol[0x8];
353+};
354+
355+struct mlx5_ifc_ste_eth_l3_ipv6_dst_bits {
356+ u8 dst_ip_127_96[0x20];
357+
358+ u8 dst_ip_95_64[0x20];
359+
360+ u8 dst_ip_63_32[0x20];
361+
362+ u8 dst_ip_31_0[0x20];
363+};
364+
365+struct mlx5_ifc_ste_eth_l2_tnl_bits {
366+ u8 dmac_47_16[0x20];
367+
368+ u8 dmac_15_0[0x10];
369+ u8 l3_ethertype[0x10];
370+
371+ u8 l2_tunneling_network_id[0x20];
372+
373+ u8 ip_fragmented[0x1];
374+ u8 tcp_syn[0x1];
375+ u8 encp_type[0x2];
376+ u8 l3_type[0x2];
377+ u8 l4_type[0x2];
378+ u8 first_priority[0x3];
379+ u8 first_cfi[0x1];
380+ u8 reserved_at_6c[0x3];
381+ u8 gre_key_flag[0x1];
382+ u8 first_vlan_qualifier[0x2];
383+ u8 reserved_at_72[0x2];
384+ u8 first_vlan_id[0xc];
385+};
386+
387+struct mlx5_ifc_ste_eth_l3_ipv6_src_bits {
388+ u8 src_ip_127_96[0x20];
389+
390+ u8 src_ip_95_64[0x20];
391+
392+ u8 src_ip_63_32[0x20];
393+
394+ u8 src_ip_31_0[0x20];
395+};
396+
397+struct mlx5_ifc_ste_eth_l3_ipv4_misc_bits {
398+ u8 version[0x4];
399+ u8 ihl[0x4];
400+ u8 reserved_at_8[0x8];
401+ u8 total_length[0x10];
402+
403+ u8 identification[0x10];
404+ u8 flags[0x3];
405+ u8 fragment_offset[0xd];
406+
407+ u8 time_to_live[0x8];
408+ u8 reserved_at_48[0x8];
409+ u8 checksum[0x10];
410+
411+ u8 reserved_at_60[0x20];
412+};
413+
414+struct mlx5_ifc_ste_eth_l4_bits {
415+ u8 fragmented[0x1];
416+ u8 first_fragment[0x1];
417+ u8 reserved_at_2[0x6];
418+ u8 protocol[0x8];
419+ u8 dst_port[0x10];
420+
421+ u8 ipv6_version[0x4];
422+ u8 reserved_at_24[0x1];
423+ u8 ecn[0x2];
424+ u8 tcp_ns[0x1];
425+ u8 tcp_cwr[0x1];
426+ u8 tcp_ece[0x1];
427+ u8 tcp_urg[0x1];
428+ u8 tcp_ack[0x1];
429+ u8 tcp_psh[0x1];
430+ u8 tcp_rst[0x1];
431+ u8 tcp_syn[0x1];
432+ u8 tcp_fin[0x1];
433+ u8 src_port[0x10];
434+
435+ u8 ipv6_payload_length[0x10];
436+ u8 ipv6_hop_limit[0x8];
437+ u8 dscp[0x6];
438+ u8 reserved_at_5e[0x2];
439+
440+ u8 tcp_data_offset[0x4];
441+ u8 reserved_at_64[0x8];
442+ u8 flow_label[0x14];
443+};
444+
445+struct mlx5_ifc_ste_eth_l4_misc_bits {
446+ u8 checksum[0x10];
447+ u8 length[0x10];
448+
449+ u8 seq_num[0x20];
450+
451+ u8 ack_num[0x20];
452+
453+ u8 urgent_pointer[0x10];
454+ u8 window_size[0x10];
455+};
456+
457+struct mlx5_ifc_ste_mpls_bits {
458+ u8 mpls0_label[0x14];
459+ u8 mpls0_exp[0x3];
460+ u8 mpls0_s_bos[0x1];
461+ u8 mpls0_ttl[0x8];
462+
463+ u8 mpls1_label[0x20];
464+
465+ u8 mpls2_label[0x20];
466+
467+ u8 reserved_at_60[0x16];
468+ u8 mpls4_s_bit[0x1];
469+ u8 mpls4_qualifier[0x1];
470+ u8 mpls3_s_bit[0x1];
471+ u8 mpls3_qualifier[0x1];
472+ u8 mpls2_s_bit[0x1];
473+ u8 mpls2_qualifier[0x1];
474+ u8 mpls1_s_bit[0x1];
475+ u8 mpls1_qualifier[0x1];
476+ u8 mpls0_s_bit[0x1];
477+ u8 mpls0_qualifier[0x1];
478+};
479+
480+struct mlx5_ifc_ste_register_0_bits {
481+ u8 register_0_h[0x20];
482+
483+ u8 register_0_l[0x20];
484+
485+ u8 register_1_h[0x20];
486+
487+ u8 register_1_l[0x20];
488+};
489+
490+struct mlx5_ifc_ste_register_1_bits {
491+ u8 register_2_h[0x20];
492+
493+ u8 register_2_l[0x20];
494+
495+ u8 register_3_h[0x20];
496+
497+ u8 register_3_l[0x20];
498+};
499+
500+struct mlx5_ifc_ste_gre_bits {
501+ u8 gre_c_present[0x1];
502+ u8 reserved_at_30[0x1];
503+ u8 gre_k_present[0x1];
504+ u8 gre_s_present[0x1];
505+ u8 strict_src_route[0x1];
506+ u8 recur[0x3];
507+ u8 flags[0x5];
508+ u8 version[0x3];
509+ u8 gre_protocol[0x10];
510+
511+ u8 checksum[0x10];
512+ u8 offset[0x10];
513+
514+ u8 gre_key_h[0x18];
515+ u8 gre_key_l[0x8];
516+
517+ u8 seq_num[0x20];
518+};
519+
520+struct mlx5_ifc_ste_flex_parser_0_bits {
521+ u8 parser_3_label[0x14];
522+ u8 parser_3_exp[0x3];
523+ u8 parser_3_s_bos[0x1];
524+ u8 parser_3_ttl[0x8];
525+
526+ u8 flex_parser_2[0x20];
527+
528+ u8 flex_parser_1[0x20];
529+
530+ u8 flex_parser_0[0x20];
531+};
532+
533+struct mlx5_ifc_ste_flex_parser_1_bits {
534+ u8 flex_parser_7[0x20];
535+
536+ u8 flex_parser_6[0x20];
537+
538+ u8 flex_parser_5[0x20];
539+
540+ u8 flex_parser_4[0x20];
541+};
542+
543+struct mlx5_ifc_ste_flex_parser_tnl_bits {
544+ u8 flex_parser_tunneling_header_63_32[0x20];
545+
546+ u8 flex_parser_tunneling_header_31_0[0x20];
547+
548+ u8 reserved_at_40[0x40];
549+};
550+
551+struct mlx5_ifc_ste_general_purpose_bits {
552+ u8 general_purpose_lookup_field[0x20];
553+
554+ u8 reserved_at_20[0x20];
555+
556+ u8 reserved_at_40[0x20];
557+
558+ u8 reserved_at_60[0x20];
559+};
560+
561+struct mlx5_ifc_ste_src_gvmi_qp_bits {
562+ u8 loopback_syndrome[0x8];
563+ u8 reserved_at_8[0x8];
564+ u8 source_gvmi[0x10];
565+
566+ u8 reserved_at_20[0x5];
567+ u8 force_lb[0x1];
568+ u8 functional_lb[0x1];
569+ u8 source_is_requestor[0x1];
570+ u8 source_qp[0x18];
571+
572+ u8 reserved_at_40[0x20];
573+
574+ u8 reserved_at_60[0x20];
575+};
576+
577+struct mlx5_ifc_l2_hdr_bits {
578+ u8 dmac_47_16[0x20];
579+
580+ u8 dmac_15_0[0x10];
581+ u8 smac_47_32[0x10];
582+
583+ u8 smac_31_0[0x20];
584+
585+ u8 ethertype[0x10];
586+ u8 vlan_type[0x10];
587+
588+ u8 vlan[0x10];
589+ u8 reserved_at_90[0x10];
590+};
591+
592+/* Both HW set and HW add share the same HW format with different opcodes */
593+struct mlx5_ifc_dr_action_hw_set_bits {
594+ u8 opcode[0x8];
595+ u8 destination_field_code[0x8];
596+ u8 reserved_at_10[0x2];
597+ u8 destination_left_shifter[0x6];
598+ u8 reserved_at_18[0x3];
599+ u8 destination_length[0x5];
600+
601+ u8 inline_data[0x20];
602+};
603+
604+#endif /* MLX5_IFC_DR_H */
drivers/net/ethernet/mellanox/mlx5/core/steering/mlx5dr.h+212−0
@@ -0,0 +1,212 @@
1+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
2+/* Copyright (c) 2019, Mellanox Technologies */
3+
4+#ifndef _MLX5DR_H_
5+#define _MLX5DR_H_
6+
7+struct mlx5dr_domain;
8+struct mlx5dr_table;
9+struct mlx5dr_matcher;
10+struct mlx5dr_rule;
11+struct mlx5dr_action;
12+
13+enum mlx5dr_domain_type {
14+ MLX5DR_DOMAIN_TYPE_NIC_RX,
15+ MLX5DR_DOMAIN_TYPE_NIC_TX,
16+ MLX5DR_DOMAIN_TYPE_FDB,
17+};
18+
19+enum mlx5dr_domain_sync_flags {
20+ MLX5DR_DOMAIN_SYNC_FLAGS_SW = 1 << 0,
21+ MLX5DR_DOMAIN_SYNC_FLAGS_HW = 1 << 1,
22+};
23+
24+enum mlx5dr_action_reformat_type {
25+ DR_ACTION_REFORMAT_TYP_TNL_L2_TO_L2,
26+ DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L2,
27+ DR_ACTION_REFORMAT_TYP_TNL_L3_TO_L2,
28+ DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L3,
29+};
30+
31+struct mlx5dr_match_parameters {
32+ size_t match_sz;
33+ u64 *match_buf; /* Device spec format */
34+};
35+
36+#ifdef CONFIG_MLX5_SW_STEERING
37+
38+struct mlx5dr_domain *
39+mlx5dr_domain_create(struct mlx5_core_dev *mdev, enum mlx5dr_domain_type type);
40+
41+int mlx5dr_domain_destroy(struct mlx5dr_domain *domain);
42+
43+int mlx5dr_domain_sync(struct mlx5dr_domain *domain, u32 flags);
44+
45+void mlx5dr_domain_set_peer(struct mlx5dr_domain *dmn,
46+ struct mlx5dr_domain *peer_dmn);
47+
48+struct mlx5dr_table *
49+mlx5dr_table_create(struct mlx5dr_domain *domain, u32 level);
50+
51+int mlx5dr_table_destroy(struct mlx5dr_table *table);
52+
53+u32 mlx5dr_table_get_id(struct mlx5dr_table *table);
54+
55+struct mlx5dr_matcher *
56+mlx5dr_matcher_create(struct mlx5dr_table *table,
57+ u16 priority,
58+ u8 match_criteria_enable,
59+ struct mlx5dr_match_parameters *mask);
60+
61+int mlx5dr_matcher_destroy(struct mlx5dr_matcher *matcher);
62+
63+struct mlx5dr_rule *
64+mlx5dr_rule_create(struct mlx5dr_matcher *matcher,
65+ struct mlx5dr_match_parameters *value,
66+ size_t num_actions,
67+ struct mlx5dr_action *actions[]);
68+
69+int mlx5dr_rule_destroy(struct mlx5dr_rule *rule);
70+
71+int mlx5dr_table_set_miss_action(struct mlx5dr_table *tbl,
72+ struct mlx5dr_action *action);
73+
74+struct mlx5dr_action *
75+mlx5dr_action_create_dest_table(struct mlx5dr_table *table);
76+
77+struct mlx5dr_action *
78+mlx5dr_create_action_dest_flow_fw_table(struct mlx5_flow_table *ft,
79+ struct mlx5_core_dev *mdev);
80+
81+struct mlx5dr_action *
82+mlx5dr_action_create_dest_vport(struct mlx5dr_domain *domain,
83+ u32 vport, u8 vhca_id_valid,
84+ u16 vhca_id);
85+
86+struct mlx5dr_action *mlx5dr_action_create_drop(void);
87+
88+struct mlx5dr_action *mlx5dr_action_create_tag(u32 tag_value);
89+
90+struct mlx5dr_action *
91+mlx5dr_action_create_flow_counter(u32 counter_id);
92+
93+struct mlx5dr_action *
94+mlx5dr_action_create_packet_reformat(struct mlx5dr_domain *dmn,
95+ enum mlx5dr_action_reformat_type reformat_type,
96+ size_t data_sz,
97+ void *data);
98+
99+struct mlx5dr_action *
100+mlx5dr_action_create_modify_header(struct mlx5dr_domain *domain,
101+ u32 flags,
102+ size_t actions_sz,
103+ __be64 actions[]);
104+
105+struct mlx5dr_action *mlx5dr_action_create_pop_vlan(void);
106+
107+struct mlx5dr_action *
108+mlx5dr_action_create_push_vlan(struct mlx5dr_domain *domain, __be32 vlan_hdr);
109+
110+int mlx5dr_action_destroy(struct mlx5dr_action *action);
111+
112+static inline bool
113+mlx5dr_is_supported(struct mlx5_core_dev *dev)
114+{
115+ return MLX5_CAP_ESW_FLOWTABLE_FDB(dev, sw_owner);
116+}
117+
118+#else /* CONFIG_MLX5_SW_STEERING */
119+
120+static inline struct mlx5dr_domain *
121+mlx5dr_domain_create(struct mlx5_core_dev *mdev, enum mlx5dr_domain_type type) { return NULL; }
122+
123+static inline int
124+mlx5dr_domain_destroy(struct mlx5dr_domain *domain) { return 0; }
125+
126+static inline int
127+mlx5dr_domain_sync(struct mlx5dr_domain *domain, u32 flags) { return 0; }
128+
129+static inline void
130+mlx5dr_domain_set_peer(struct mlx5dr_domain *dmn,
131+ struct mlx5dr_domain *peer_dmn) { }
132+
133+static inline struct mlx5dr_table *
134+mlx5dr_table_create(struct mlx5dr_domain *domain, u32 level) { return NULL; }
135+
136+static inline int
137+mlx5dr_table_destroy(struct mlx5dr_table *table) { return 0; }
138+
139+static inline u32
140+mlx5dr_table_get_id(struct mlx5dr_table *table) { return 0; }
141+
142+static inline struct mlx5dr_matcher *
143+mlx5dr_matcher_create(struct mlx5dr_table *table,
144+ u16 priority,
145+ u8 match_criteria_enable,
146+ struct mlx5dr_match_parameters *mask) { return NULL; }
147+
148+static inline int
149+mlx5dr_matcher_destroy(struct mlx5dr_matcher *matcher) { return 0; }
150+
151+static inline struct mlx5dr_rule *
152+mlx5dr_rule_create(struct mlx5dr_matcher *matcher,
153+ struct mlx5dr_match_parameters *value,
154+ size_t num_actions,
155+ struct mlx5dr_action *actions[]) { return NULL; }
156+
157+static inline int
158+mlx5dr_rule_destroy(struct mlx5dr_rule *rule) { return 0; }
159+
160+static inline int
161+mlx5dr_table_set_miss_action(struct mlx5dr_table *tbl,
162+ struct mlx5dr_action *action) { return 0; }
163+
164+static inline struct mlx5dr_action *
165+mlx5dr_action_create_dest_table(struct mlx5dr_table *table) { return NULL; }
166+
167+static inline struct mlx5dr_action *
168+mlx5dr_create_action_dest_flow_fw_table(struct mlx5_flow_table *ft,
169+ struct mlx5_core_dev *mdev) { return NULL; }
170+
171+static inline struct mlx5dr_action *
172+mlx5dr_action_create_dest_vport(struct mlx5dr_domain *domain,
173+ u32 vport, u8 vhca_id_valid,
174+ u16 vhca_id) { return NULL; }
175+
176+static inline struct mlx5dr_action *
177+mlx5dr_action_create_drop(void) { return NULL; }
178+
179+static inline struct mlx5dr_action *
180+mlx5dr_action_create_tag(u32 tag_value) { return NULL; }
181+
182+static inline struct mlx5dr_action *
183+mlx5dr_action_create_flow_counter(u32 counter_id) { return NULL; }
184+
185+static inline struct mlx5dr_action *
186+mlx5dr_action_create_packet_reformat(struct mlx5dr_domain *dmn,
187+ enum mlx5dr_action_reformat_type reformat_type,
188+ size_t data_sz,
189+ void *data) { return NULL; }
190+
191+static inline struct mlx5dr_action *
192+mlx5dr_action_create_modify_header(struct mlx5dr_domain *domain,
193+ u32 flags,
194+ size_t actions_sz,
195+ __be64 actions[]) { return NULL; }
196+
197+static inline struct mlx5dr_action *
198+mlx5dr_action_create_pop_vlan(void) { return NULL; }
199+
200+static inline struct mlx5dr_action *
201+mlx5dr_action_create_push_vlan(struct mlx5dr_domain *domain,
202+ __be32 vlan_hdr) { return NULL; }
203+
204+static inline int
205+mlx5dr_action_destroy(struct mlx5dr_action *action) { return 0; }
206+
207+static inline bool
208+mlx5dr_is_supported(struct mlx5_core_dev *dev) { return false; }
209+
210+#endif /* CONFIG_MLX5_SW_STEERING */
211+
212+#endif /* _MLX5DR_H_ */
include/linux/mlx5/device.h+7−0
@@ -1162,6 +1162,9 @@ enum mlx5_qcam_feature_groups { · 1161 unmodified lines
11621162 #define MLX5_CAP_FLOWTABLE(mdev, cap) \
11631163 MLX5_GET(flow_table_nic_cap, mdev->caps.hca_cur[MLX5_CAP_FLOW_TABLE], cap)
11641164
1165+#define MLX5_CAP64_FLOWTABLE(mdev, cap) \
1166+ MLX5_GET64(flow_table_nic_cap, (mdev)->caps.hca_cur[MLX5_CAP_FLOW_TABLE], cap)
1167+
11651168 #define MLX5_CAP_FLOWTABLE_MAX(mdev, cap) \
11661169 MLX5_GET(flow_table_nic_cap, mdev->caps.hca_max[MLX5_CAP_FLOW_TABLE], cap)
11671170
@@ -1225,6 +1228,10 @@ enum mlx5_qcam_feature_groups { · 57 unmodified lines
12251228 MLX5_GET(e_switch_cap, \
12261229 mdev->caps.hca_cur[MLX5_CAP_ESWITCH], cap)
12271230
1231+#define MLX5_CAP64_ESW_FLOWTABLE(mdev, cap) \
1232+ MLX5_GET64(flow_table_eswitch_cap, \
1233+ (mdev)->caps.hca_cur[MLX5_CAP_ESWITCH_FLOW_TABLE], cap)
1234+
12281235 #define MLX5_CAP_ESW_MAX(mdev, cap) \
12291236 MLX5_GET(e_switch_cap, \
12301237 mdev->caps.hca_max[MLX5_CAP_ESWITCH], cap)
include/linux/mlx5/driver.h+14−0
@@ -626,6 +626,11 @@ struct mlx5e_resources { · 625 unmodified lines
626626 struct mlx5_sq_bfreg bfreg;
627627 };
628628
629+enum mlx5_sw_icm_type {
630+ MLX5_SW_ICM_TYPE_STEERING,
631+ MLX5_SW_ICM_TYPE_HEADER_MODIFY,
632+};
633+
629634 #define MLX5_MAX_RESERVED_GIDS 8
630635
631636 struct mlx5_rsvd_gids {
@@ -657,11 +662,15 @@ struct mlx5_clock { · 25 unmodified lines
657662 struct mlx5_pps pps_info;
658663 };
659664
665+struct mlx5_dm;
660666 struct mlx5_fw_tracer;
661667 struct mlx5_vxlan;
662668 struct mlx5_geneve;
663669 struct mlx5_hv_vhca;
664670
671+#define MLX5_LOG_SW_ICM_BLOCK_SIZE(dev) (MLX5_CAP_DEV_MEM(dev, log_sw_icm_alloc_granularity))
672+#define MLX5_SW_ICM_BLOCK_SIZE(dev) (1 << MLX5_LOG_SW_ICM_BLOCK_SIZE(dev))
673+
665674 struct mlx5_core_dev {
666675 struct device *device;
667676 enum mlx5_coredev_type coredev_type;
@@ -695,6 +704,7 @@ struct mlx5_core_dev { · 27 unmodified lines
695704 atomic_t num_qps;
696705 u32 issi;
697706 struct mlx5e_resources mlx5e_res;
707+ struct mlx5_dm *dm;
698708 struct mlx5_vxlan *vxlan;
699709 struct mlx5_geneve *geneve;
700710 struct {
@@ -1078,6 +1088,10 @@ int mlx5_lag_query_cong_counters(struct mlx5_core_dev *dev, · 377 unmodified lines
10781088 size_t *offsets);
10791089 struct mlx5_uars_page *mlx5_get_uars_page(struct mlx5_core_dev *mdev);
10801090 void mlx5_put_uars_page(struct mlx5_core_dev *mdev, struct mlx5_uars_page *up);
1091+int mlx5_dm_sw_icm_alloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type,
1092+ u64 length, u16 uid, phys_addr_t *addr, u32 *obj_id);
1093+int mlx5_dm_sw_icm_dealloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type,
1094+ u64 length, u16 uid, phys_addr_t addr, u32 obj_id);
10811095
10821096 #ifdef CONFIG_MLX5_CORE_IPOIB
10831097 struct net_device *mlx5_rdma_netdev_alloc(struct mlx5_core_dev *mdev,
include/linux/mlx5/eswitch.h+7−1
@@ -60,7 +60,6 @@ void *mlx5_eswitch_get_proto_dev(struct mlx5_eswitch *esw, · 59 unmodified lines
6060 struct mlx5_eswitch_rep *mlx5_eswitch_vport_rep(struct mlx5_eswitch *esw,
6161 u16 vport_num);
6262 void *mlx5_eswitch_uplink_get_proto_dev(struct mlx5_eswitch *esw, u8 rep_type);
63−u8 mlx5_eswitch_mode(struct mlx5_eswitch *esw);
6463 struct mlx5_flow_handle *
6564 mlx5_eswitch_add_send_to_vport_rule(struct mlx5_eswitch *esw,
6665 u16 vport_num, u32 sqn);
@@ -74,7 +73,14 @@ mlx5_eswitch_get_encap_mode(const struct mlx5_core_dev *dev); · 7 unmodified lines
7473 bool mlx5_eswitch_vport_match_metadata_enabled(const struct mlx5_eswitch *esw);
7574 u32 mlx5_eswitch_get_vport_metadata_for_match(const struct mlx5_eswitch *esw,
7675 u16 vport_num);
76+u8 mlx5_eswitch_mode(struct mlx5_eswitch *esw);
7777 #else /* CONFIG_MLX5_ESWITCH */
78+
79+static inline u8 mlx5_eswitch_mode(struct mlx5_eswitch *esw)
80+{
81+ return MLX5_ESWITCH_NONE;
82+}
83+
7884 static inline enum devlink_eswitch_encap_mode
7985 mlx5_eswitch_get_encap_mode(const struct mlx5_core_dev *dev)
8086 {
include/linux/mlx5/fs.h+16−17
@@ -84,6 +84,8 @@ enum { · 83 unmodified lines
8484 FDB_SLOW_PATH,
8585 };
8686
87+struct mlx5_pkt_reformat;
88+struct mlx5_modify_hdr;
8789 struct mlx5_flow_table;
8890 struct mlx5_flow_group;
8991 struct mlx5_flow_namespace;
@@ -121,7 +123,7 @@ struct mlx5_flow_destination { · 31 unmodified lines
121123 struct {
122124 u16 num;
123125 u16 vhca_id;
124− u32 reformat_id;
126+ struct mlx5_pkt_reformat *pkt_reformat;
125127 u8 flags;
126128 } vport;
127129 };
@@ -195,8 +197,8 @@ enum { · 67 unmodified lines
195197
196198 struct mlx5_flow_act {
197199 u32 action;
198− u32 reformat_id;
199− u32 modify_id;
200+ struct mlx5_modify_hdr *modify_hdr;
201+ struct mlx5_pkt_reformat *pkt_reformat;
200202 uintptr_t esp_id;
201203 u32 flags;
202204 struct mlx5_fs_vlan vlan[MLX5_FS_VLAN_DEPTH];
@@ -205,8 +207,6 @@ struct mlx5_flow_act { · 2 unmodified lines
205207
206208 #define MLX5_DECLARE_FLOW_ACT(name) \
207209 struct mlx5_flow_act name = { .action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,\
208− .reformat_id = 0, \
209− .modify_id = 0, \
210210 .flags = 0, }
211211
212212 /* Single destination per rule.
@@ -236,19 +236,18 @@ u32 mlx5_fc_id(struct mlx5_fc *counter); · 23 unmodified lines
236236 int mlx5_fs_add_rx_underlay_qpn(struct mlx5_core_dev *dev, u32 underlay_qpn);
237237 int mlx5_fs_remove_rx_underlay_qpn(struct mlx5_core_dev *dev, u32 underlay_qpn);
238238
239−int mlx5_modify_header_alloc(struct mlx5_core_dev *dev,
240− u8 namespace, u8 num_actions,
241− void *modify_actions, u32 *modify_header_id);
239+struct mlx5_modify_hdr *mlx5_modify_header_alloc(struct mlx5_core_dev *dev,
240+ u8 ns_type, u8 num_actions,
241+ void *modify_actions);
242242 void mlx5_modify_header_dealloc(struct mlx5_core_dev *dev,
243− u32 modify_header_id);
244−
245−int mlx5_packet_reformat_alloc(struct mlx5_core_dev *dev,
246− int reformat_type,
247− size_t size,
248− void *reformat_data,
249− enum mlx5_flow_namespace_type namespace,
250− u32 *packet_reformat_id);
243+ struct mlx5_modify_hdr *modify_hdr);
244+
245+struct mlx5_pkt_reformat *mlx5_packet_reformat_alloc(struct mlx5_core_dev *dev,
246+ int reformat_type,
247+ size_t size,
248+ void *reformat_data,
249+ enum mlx5_flow_namespace_type ns_type);
251250 void mlx5_packet_reformat_dealloc(struct mlx5_core_dev *dev,
252− u32 packet_reformat_id);
251+ struct mlx5_pkt_reformat *reformat);
253252
254253 #endif
include/linux/mlx5/mlx5_ifc.h+198−37
@@ -282,6 +282,7 @@ enum { · 281 unmodified lines
282282 MLX5_CMD_OP_ALLOC_MODIFY_HEADER_CONTEXT = 0x940,
283283 MLX5_CMD_OP_DEALLOC_MODIFY_HEADER_CONTEXT = 0x941,
284284 MLX5_CMD_OP_QUERY_MODIFY_HEADER_CONTEXT = 0x942,
285+ MLX5_CMD_OP_SYNC_STEERING = 0xb00,
285286 MLX5_CMD_OP_FPGA_CREATE_QP = 0x960,
286287 MLX5_CMD_OP_FPGA_MODIFY_QP = 0x961,
287288 MLX5_CMD_OP_FPGA_QUERY_QP = 0x962,
@@ -485,7 +486,11 @@ union mlx5_ifc_gre_key_bits { · 197 unmodified lines
485486 };
486487
487488 struct mlx5_ifc_fte_match_set_misc_bits {
488− u8 reserved_at_0[0x8];
489+ u8 gre_c_present[0x1];
490+ u8 reserved_auto1[0x1];
491+ u8 gre_k_present[0x1];
492+ u8 gre_s_present[0x1];
493+ u8 source_vhca_port[0x4];
489494 u8 source_sqn[0x18];
490495
491496 u8 source_eswitch_owner_vhca_id[0x10];
@@ -565,12 +570,38 @@ struct mlx5_ifc_fte_match_set_misc2_bits { · 73 unmodified lines
565570
566571 u8 metadata_reg_a[0x20];
567572
568− u8 reserved_at_1a0[0x60];
573+ u8 metadata_reg_b[0x20];
574+
575+ u8 reserved_at_1c0[0x40];
569576 };
570577
571578 struct mlx5_ifc_fte_match_set_misc3_bits {
572− u8 reserved_at_0[0x120];
579+ u8 inner_tcp_seq_num[0x20];
580+
581+ u8 outer_tcp_seq_num[0x20];
582+
583+ u8 inner_tcp_ack_num[0x20];
584+
585+ u8 outer_tcp_ack_num[0x20];
586+
587+ u8 reserved_at_80[0x8];
588+ u8 outer_vxlan_gpe_vni[0x18];
589+
590+ u8 outer_vxlan_gpe_next_protocol[0x8];
591+ u8 outer_vxlan_gpe_flags[0x8];
592+ u8 reserved_at_b0[0x10];
593+
594+ u8 icmp_header_data[0x20];
595+
596+ u8 icmpv6_header_data[0x20];
597+
598+ u8 icmp_type[0x8];
599+ u8 icmp_code[0x8];
600+ u8 icmpv6_type[0x8];
601+ u8 icmpv6_code[0x8];
602+
573603 u8 geneve_tlv_option_0_data[0x20];
604+
574605 u8 reserved_at_140[0xc0];
575606 };
576607
@@ -666,7 +697,15 @@ struct mlx5_ifc_flow_table_nic_cap_bits { · 89 unmodified lines
666697
667698 struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_nic_transmit_sniffer;
668699
669− u8 reserved_at_e00[0x7200];
700+ u8 reserved_at_e00[0x1200];
701+
702+ u8 sw_steering_nic_rx_action_drop_icm_address[0x40];
703+
704+ u8 sw_steering_nic_tx_action_drop_icm_address[0x40];
705+
706+ u8 sw_steering_nic_tx_action_allow_icm_address[0x40];
707+
708+ u8 reserved_at_20c0[0x5f40];
670709 };
671710
672711 enum {
@@ -698,7 +737,17 @@ struct mlx5_ifc_flow_table_eswitch_cap_bits { · 25 unmodified lines
698737
699738 struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_esw_acl_egress;
700739
701− u8 reserved_at_800[0x7800];
740+ u8 reserved_at_800[0x1000];
741+
742+ u8 sw_steering_fdb_action_drop_icm_address_rx[0x40];
743+
744+ u8 sw_steering_fdb_action_drop_icm_address_tx[0x40];
745+
746+ u8 sw_steering_uplink_icm_address_rx[0x40];
747+
748+ u8 sw_steering_uplink_icm_address_tx[0x40];
749+
750+ u8 reserved_at_1900[0x6700];
702751 };
703752
704753 enum {
@@ -849,6 +898,25 @@ struct mlx5_ifc_roce_cap_bits { · 144 unmodified lines
849898 u8 reserved_at_100[0x700];
850899 };
851900
901+struct mlx5_ifc_sync_steering_in_bits {
902+ u8 opcode[0x10];
903+ u8 uid[0x10];
904+
905+ u8 reserved_at_20[0x10];
906+ u8 op_mod[0x10];
907+
908+ u8 reserved_at_40[0xc0];
909+};
910+
911+struct mlx5_ifc_sync_steering_out_bits {
912+ u8 status[0x8];
913+ u8 reserved_at_8[0x18];
914+
915+ u8 syndrome[0x20];
916+
917+ u8 reserved_at_40[0x40];
918+};
919+
852920 struct mlx5_ifc_device_mem_cap_bits {
853921 u8 memic[0x1];
854922 u8 reserved_at_1[0x1f];
@@ -1041,6 +1109,12 @@ enum { · 186 unmodified lines
10411109 MLX5_CAP_UMR_FENCE_NONE = 0x2,
10421110 };
10431111
1112+enum {
1113+ MLX5_FLEX_PARSER_VXLAN_GPE_ENABLED = 1 << 7,
1114+ MLX5_FLEX_PARSER_ICMP_V4_ENABLED = 1 << 8,
1115+ MLX5_FLEX_PARSER_ICMP_V6_ENABLED = 1 << 9,
1116+};
1117+
10441118 enum {
10451119 MLX5_UCTX_CAP_RAW_TX = 1UL << 0,
10461120 MLX5_UCTX_CAP_INTERNAL_DEV_RES = 1UL << 1,
@@ -1414,7 +1488,14 @@ struct mlx5_ifc_cmd_hca_cap_bits { · 367 unmodified lines
14141488
14151489 u8 reserved_at_6c0[0x4];
14161490 u8 flex_parser_id_geneve_tlv_option_0[0x4];
1417− u8 reserved_at_6c8[0x28];
1491+ u8 flex_parser_id_icmp_dw1[0x4];
1492+ u8 flex_parser_id_icmp_dw0[0x4];
1493+ u8 flex_parser_id_icmpv6_dw1[0x4];
1494+ u8 flex_parser_id_icmpv6_dw0[0x4];
1495+ u8 flex_parser_id_outer_first_mpls_over_gre[0x4];
1496+ u8 flex_parser_id_outer_first_mpls_over_udp_label[0x4];
1497+
1498+ u8 reserved_at_6e0[0x10];
14181499 u8 sf_base_id[0x10];
14191500
14201501 u8 reserved_at_700[0x80];
@@ -2652,6 +2733,7 @@ union mlx5_ifc_hca_cap_union_bits { · 1231 unmodified lines
26522733 struct mlx5_ifc_debug_cap_bits debug_cap;
26532734 struct mlx5_ifc_fpga_cap_bits fpga_cap;
26542735 struct mlx5_ifc_tls_cap_bits tls_cap;
2736+ struct mlx5_ifc_device_mem_cap_bits device_mem_cap;
26552737 u8 reserved_at_0[0x8000];
26562738 };
26572739
@@ -3255,7 +3337,11 @@ struct mlx5_ifc_esw_vport_context_bits { · 597 unmodified lines
32553337 u8 cvlan_pcp[0x3];
32563338 u8 cvlan_id[0xc];
32573339
3258− u8 reserved_at_60[0x7a0];
3340+ u8 reserved_at_60[0x720];
3341+
3342+ u8 sw_steering_vport_icm_address_rx[0x40];
3343+
3344+ u8 sw_steering_vport_icm_address_tx[0x40];
32593345 };
32603346
32613347 enum {
@@ -4941,23 +5027,98 @@ struct mlx5_ifc_query_hca_cap_in_bits { · 1679 unmodified lines
49415027 u8 reserved_at_20[0x10];
49425028 u8 op_mod[0x10];
49435029
4944− u8 reserved_at_40[0x40];
5030+ u8 other_function[0x1];
5031+ u8 reserved_at_41[0xf];
5032+ u8 function_id[0x10];
5033+
5034+ u8 reserved_at_60[0x20];
49455035 };
49465036
4947−struct mlx5_ifc_query_flow_table_out_bits {
5037+struct mlx5_ifc_other_hca_cap_bits {
5038+ u8 roce[0x1];
5039+ u8 reserved_0[0x27f];
5040+};
5041+
5042+struct mlx5_ifc_query_other_hca_cap_out_bits {
49485043 u8 status[0x8];
4949− u8 reserved_at_8[0x18];
5044+ u8 reserved_0[0x18];
49505045
49515046 u8 syndrome[0x20];
49525047
4953− u8 reserved_at_40[0x80];
5048+ u8 reserved_1[0x40];
49545049
4955− u8 reserved_at_c0[0x8];
5050+ struct mlx5_ifc_other_hca_cap_bits other_capability;
5051+};
5052+
5053+struct mlx5_ifc_query_other_hca_cap_in_bits {
5054+ u8 opcode[0x10];
5055+ u8 reserved_0[0x10];
5056+
5057+ u8 reserved_1[0x10];
5058+ u8 op_mod[0x10];
5059+
5060+ u8 reserved_2[0x10];
5061+ u8 function_id[0x10];
5062+
5063+ u8 reserved_3[0x20];
5064+};
5065+
5066+struct mlx5_ifc_modify_other_hca_cap_out_bits {
5067+ u8 status[0x8];
5068+ u8 reserved_0[0x18];
5069+
5070+ u8 syndrome[0x20];
5071+
5072+ u8 reserved_1[0x40];
5073+};
5074+
5075+struct mlx5_ifc_modify_other_hca_cap_in_bits {
5076+ u8 opcode[0x10];
5077+ u8 reserved_0[0x10];
5078+
5079+ u8 reserved_1[0x10];
5080+ u8 op_mod[0x10];
5081+
5082+ u8 reserved_2[0x10];
5083+ u8 function_id[0x10];
5084+ u8 field_select[0x20];
5085+
5086+ struct mlx5_ifc_other_hca_cap_bits other_capability;
5087+};
5088+
5089+struct mlx5_ifc_flow_table_context_bits {
5090+ u8 reformat_en[0x1];
5091+ u8 decap_en[0x1];
5092+ u8 sw_owner[0x1];
5093+ u8 termination_table[0x1];
5094+ u8 table_miss_action[0x4];
49565095 u8 level[0x8];
4957− u8 reserved_at_d0[0x8];
5096+ u8 reserved_at_10[0x8];
49585097 u8 log_size[0x8];
49595098
4960− u8 reserved_at_e0[0x120];
5099+ u8 reserved_at_20[0x8];
5100+ u8 table_miss_id[0x18];
5101+
5102+ u8 reserved_at_40[0x8];
5103+ u8 lag_master_next_table_id[0x18];
5104+
5105+ u8 reserved_at_60[0x60];
5106+
5107+ u8 sw_owner_icm_root_1[0x40];
5108+
5109+ u8 sw_owner_icm_root_0[0x40];
5110+
5111+};
5112+
5113+struct mlx5_ifc_query_flow_table_out_bits {
5114+ u8 status[0x8];
5115+ u8 reserved_at_8[0x18];
5116+
5117+ u8 syndrome[0x20];
5118+
5119+ u8 reserved_at_40[0x80];
5120+
5121+ struct mlx5_ifc_flow_table_context_bits flow_table_context;
49615122 };
49625123
49635124 struct mlx5_ifc_query_flow_table_in_bits {
@@ -5227,7 +5388,7 @@ struct mlx5_ifc_alloc_packet_reformat_context_out_bits { · 263 unmodified lines
52275388 u8 reserved_at_60[0x20];
52285389 };
52295390
5230−enum {
5391+enum mlx5_reformat_ctx_type {
52315392 MLX5_REFORMAT_TYPE_L2_TO_VXLAN = 0x0,
52325393 MLX5_REFORMAT_TYPE_L2_TO_NVGRE = 0x1,
52335394 MLX5_REFORMAT_TYPE_L2_TO_L2_TUNNEL = 0x2,
@@ -5323,7 +5484,16 @@ enum { · 89 unmodified lines
53235484 MLX5_ACTION_IN_FIELD_OUT_DIPV4 = 0x16,
53245485 MLX5_ACTION_IN_FIELD_OUT_FIRST_VID = 0x17,
53255486 MLX5_ACTION_IN_FIELD_OUT_IPV6_HOPLIMIT = 0x47,
5487+ MLX5_ACTION_IN_FIELD_METADATA_REG_A = 0x49,
5488+ MLX5_ACTION_IN_FIELD_METADATA_REG_B = 0x50,
53265489 MLX5_ACTION_IN_FIELD_METADATA_REG_C_0 = 0x51,
5490+ MLX5_ACTION_IN_FIELD_METADATA_REG_C_1 = 0x52,
5491+ MLX5_ACTION_IN_FIELD_METADATA_REG_C_2 = 0x53,
5492+ MLX5_ACTION_IN_FIELD_METADATA_REG_C_3 = 0x54,
5493+ MLX5_ACTION_IN_FIELD_METADATA_REG_C_4 = 0x55,
5494+ MLX5_ACTION_IN_FIELD_METADATA_REG_C_5 = 0x56,
5495+ MLX5_ACTION_IN_FIELD_OUT_TCP_SEQ_NUM = 0x59,
5496+ MLX5_ACTION_IN_FIELD_OUT_TCP_ACK_NUM = 0x5B,
53275497 };
53285498
53295499 struct mlx5_ifc_alloc_modify_header_context_out_bits {
@@ -7371,35 +7541,26 @@ struct mlx5_ifc_create_mkey_in_bits { · 2041 unmodified lines
73717541 u8 klm_pas_mtt[0][0x20];
73727542 };
73737543
7544+enum {
7545+ MLX5_FLOW_TABLE_TYPE_NIC_RX = 0x0,
7546+ MLX5_FLOW_TABLE_TYPE_NIC_TX = 0x1,
7547+ MLX5_FLOW_TABLE_TYPE_ESW_EGRESS_ACL = 0x2,
7548+ MLX5_FLOW_TABLE_TYPE_ESW_INGRESS_ACL = 0x3,
7549+ MLX5_FLOW_TABLE_TYPE_FDB = 0X4,
7550+ MLX5_FLOW_TABLE_TYPE_SNIFFER_RX = 0X5,
7551+ MLX5_FLOW_TABLE_TYPE_SNIFFER_TX = 0X6,
7552+};
7553+
73747554 struct mlx5_ifc_create_flow_table_out_bits {
73757555 u8 status[0x8];
7376− u8 reserved_at_8[0x18];
7556+ u8 icm_address_63_40[0x18];
73777557
73787558 u8 syndrome[0x20];
73797559
7380− u8 reserved_at_40[0x8];
7560+ u8 icm_address_39_32[0x8];
73817561 u8 table_id[0x18];
73827562
7383− u8 reserved_at_60[0x20];
7384−};
7385−
7386−struct mlx5_ifc_flow_table_context_bits {
7387− u8 reformat_en[0x1];
7388− u8 decap_en[0x1];
7389− u8 reserved_at_2[0x1];
7390− u8 termination_table[0x1];
7391− u8 table_miss_action[0x4];
7392− u8 level[0x8];
7393− u8 reserved_at_10[0x8];
7394− u8 log_size[0x8];
7395−
7396− u8 reserved_at_20[0x8];
7397− u8 table_miss_id[0x18];
7398−
7399− u8 reserved_at_40[0x8];
7400− u8 lag_master_next_table_id[0x18];
7401−
7402− u8 reserved_at_60[0xe0];
7563+ u8 icm_address_31_0[0x20];
74037564 };
74047565
74057566 struct mlx5_ifc_create_flow_table_in_bits {

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t