History

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma

Pull rdma updates from Jason Gunthorpe: "A new Pensando ionic driver, a new Gen 3 HW support for Intel irdma, and lots of small bnxt_re improvements. - Small bug fixes and improves to hfi1, efa, mlx5, erdma, rdmarvt, siw - Allow userspace access to IB service records through the rdmacm - Optimize dma mapping for erdma - Fix shutdown of the GSI QP in mana - Support relaxed ordering MR and fix a corruption bug with mlx5 DMA Data Direct - Many improvement to bnxt_re: - Debugging features and counters - Improve performance of some commands - Change flow_label reporting in completions - Mirror vnic - RDMA flow support - New RDMA driver for Pensando Ethernet devices: ionic - Gen 3 hardware support for the Intel irdma driver - Fix rdma routing resolution with VRFs" * tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma: (85 commits) RDMA/ionic: Fix memory leak of admin q_wr RDMA/siw: Always report immediate post SQ errors RDMA/bnxt_re: improve clarity in ALLOC_PAGE handler RDMA/irdma: Remove unused struct irdma_cq fields RDMA/irdma: Fix positive vs negative error codes in irdma_post_send() RDMA/bnxt_re: Remove non-statistics counters from hw_counters RDMA/bnxt_re: Add debugfs info entry for device and resource information RDMA/bnxt_re: Fix incorrect errno used in function comments RDMA: Use %pe format specifier for error pointers RDMA/ionic: Use ether_addr_copy instead of memcpy RDMA/ionic: Fix build failure on SPARC due to xchg() operand size RDMA/rxe: Fix race in do_task() when draining IB/sa: Fix sa_local_svc_timeout_ms read race IB/ipoib: Ignore L3 master device RDMA/core: Use route entry flag to decide on loopback traffic RDMA/core: Resolve MAC of next-hop device without ARP support RDMA/core: Squash a single user static function RDMA/irdma: Update Kconfig RDMA/irdma: Extend CQE Error and Flush Handling for GEN3 Devices RDMA/irdma: Add Atomic Operations support ...

Linus Torvalds committed 1 year ago

Commit 2ccb4d22 parents e56ebe2 e6d736b128 files · +16186 −1486Browse files

Unified · Split

128 files
  1. Documentation/networking/device_drivers/ethernet/index.rst +1 −0
  2. Documentation/networking/device_drivers/ethernet/pensando/ionic.rst +10 −0
  3. Documentation/networking/device_drivers/ethernet/pensando/ionic_rdma.rst +52 −0
  4. MAINTAINERS +9 −0
  5. drivers/infiniband/Kconfig +1 −0
  6. drivers/infiniband/core/addr.c +39 −44
  7. drivers/infiniband/core/agent.c +1 −2
  8. drivers/infiniband/core/cm.c +2 −2
  9. drivers/infiniband/core/cma.c +134 −2
  10. drivers/infiniband/core/cma_priv.h +3 −1
  11. drivers/infiniband/core/device.c +1 −1
  12. drivers/infiniband/core/sa_query.c +270 −13
  13. drivers/infiniband/core/ucma.c +119 −1
  14. drivers/infiniband/hw/Makefile +1 −0
  15. drivers/infiniband/hw/bnxt_re/bnxt_re.h +15 −4
  16. drivers/infiniband/hw/bnxt_re/debugfs.c +37 −0
  17. drivers/infiniband/hw/bnxt_re/hw_counters.c +34 −75
  18. drivers/infiniband/hw/bnxt_re/hw_counters.h +3 −23
  19. drivers/infiniband/hw/bnxt_re/ib_verbs.c +149 −7
  20. drivers/infiniband/hw/bnxt_re/ib_verbs.h +10 −0
  21. drivers/infiniband/hw/bnxt_re/main.c +230 −148
  22. drivers/infiniband/hw/bnxt_re/qplib_fp.c +10 −3
  23. drivers/infiniband/hw/bnxt_re/qplib_fp.h +2 −0
  24. drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +8 −2
  25. drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +1 −0
  26. drivers/infiniband/hw/bnxt_re/qplib_res.c +11 −27
  27. drivers/infiniband/hw/bnxt_re/qplib_res.h +16 −5
  28. drivers/infiniband/hw/bnxt_re/qplib_sp.c +49 −49
  29. drivers/infiniband/hw/bnxt_re/qplib_sp.h +5 −1
  30. drivers/infiniband/hw/bnxt_re/roce_hsi.h +42 −2
  31. drivers/infiniband/hw/cxgb4/device.c +2 −3
  32. drivers/infiniband/hw/efa/efa_com.c +11 −7
  33. drivers/infiniband/hw/efa/efa_verbs.c +4 −2
  34. drivers/infiniband/hw/erdma/erdma_verbs.c +69 −41
  35. drivers/infiniband/hw/erdma/erdma_verbs.h +2 −2
  36. drivers/infiniband/hw/hfi1/device.c +2 −2
  37. drivers/infiniband/hw/hfi1/sdma.c +1 −1
  38. drivers/infiniband/hw/hfi1/user_sdma.c +2 −2
  39. drivers/infiniband/hw/hns/hns_roce_mr.c +4 −4
  40. drivers/infiniband/hw/ionic/Kconfig +15 −0
  41. drivers/infiniband/hw/ionic/Makefile +9 −0
  42. drivers/infiniband/hw/ionic/ionic_admin.c +1229 −0
  43. drivers/infiniband/hw/ionic/ionic_controlpath.c +2679 −0
  44. drivers/infiniband/hw/ionic/ionic_datapath.c +1399 −0
  45. drivers/infiniband/hw/ionic/ionic_fw.h +1029 −0
  46. drivers/infiniband/hw/ionic/ionic_hw_stats.c +484 −0
  47. drivers/infiniband/hw/ionic/ionic_ibdev.c +440 −0
  48. drivers/infiniband/hw/ionic/ionic_ibdev.h +517 −0
  49. drivers/infiniband/hw/ionic/ionic_lif_cfg.c +111 −0
  50. drivers/infiniband/hw/ionic/ionic_lif_cfg.h +66 −0
  51. drivers/infiniband/hw/ionic/ionic_pgtbl.c +143 −0
  52. drivers/infiniband/hw/ionic/ionic_queue.c +52 −0
  53. drivers/infiniband/hw/ionic/ionic_queue.h +234 −0
  54. drivers/infiniband/hw/ionic/ionic_res.h +154 −0
  55. drivers/infiniband/hw/irdma/Kconfig +4 −3
  56. drivers/infiniband/hw/irdma/Makefile +4 −0
  57. drivers/infiniband/hw/irdma/ctrl.c +1338 −130
  58. drivers/infiniband/hw/irdma/defs.h +145 −119
  59. drivers/infiniband/hw/irdma/hmc.c +15 −3
  60. drivers/infiniband/hw/irdma/hmc.h +18 −1
  61. drivers/infiniband/hw/irdma/hw.c +216 −147
  62. drivers/infiniband/hw/irdma/i40iw_hw.c +2 −0
  63. drivers/infiniband/hw/irdma/i40iw_hw.h +2 −0
  64. drivers/infiniband/hw/irdma/i40iw_if.c +3 −0
  65. drivers/infiniband/hw/irdma/icrdma_hw.c +3 −0
  66. drivers/infiniband/hw/irdma/icrdma_hw.h +3 −2
  67. drivers/infiniband/hw/irdma/icrdma_if.c +343 −0
  68. drivers/infiniband/hw/irdma/ig3rdma_hw.c +170 −0
  69. drivers/infiniband/hw/irdma/ig3rdma_hw.h +32 −0
  70. drivers/infiniband/hw/irdma/ig3rdma_if.c +232 −0
  71. drivers/infiniband/hw/irdma/irdma.h +20 −2
  72. drivers/infiniband/hw/irdma/main.c +81 −290
  73. drivers/infiniband/hw/irdma/main.h +27 −8
  74. drivers/infiniband/hw/irdma/pble.c +14 −6
  75. drivers/infiniband/hw/irdma/protos.h +1 −0
  76. drivers/infiniband/hw/irdma/puda.h +2 −2
  77. drivers/infiniband/hw/irdma/type.h +203 −18
  78. drivers/infiniband/hw/irdma/uda_d.h +2 −3
  79. drivers/infiniband/hw/irdma/uk.c +291 −12
  80. drivers/infiniband/hw/irdma/user.h +264 −3
  81. drivers/infiniband/hw/irdma/utils.c +107 −5
  82. drivers/infiniband/hw/irdma/verbs.c +728 −106
  83. drivers/infiniband/hw/irdma/verbs.h +44 −6
  84. drivers/infiniband/hw/irdma/virtchnl.c +618 −0
  85. drivers/infiniband/hw/irdma/virtchnl.h +176 −0
  86. drivers/infiniband/hw/mana/cq.c +26 −0
  87. drivers/infiniband/hw/mana/device.c +3 −0
  88. drivers/infiniband/hw/mana/main.c +2 −3
  89. drivers/infiniband/hw/mana/mana_ib.h +12 −2
  90. drivers/infiniband/hw/mana/mr.c +4 −2
  91. drivers/infiniband/hw/mana/qp.c +9 −0
  92. drivers/infiniband/hw/mlx4/mad.c +4 −4
  93. drivers/infiniband/hw/mlx4/qp.c +2 −1
  94. drivers/infiniband/hw/mlx5/data_direct.c +1 −1
  95. drivers/infiniband/hw/mlx5/gsi.c +9 −6
  96. drivers/infiniband/hw/mlx5/main.c +102 −11
  97. drivers/infiniband/hw/mlx5/mlx5_ib.h +7 −0
  98. drivers/infiniband/hw/mlx5/mr.c +5 −6
  99. drivers/infiniband/hw/mlx5/umr.c +5 −1
  100. drivers/infiniband/sw/rdmavt/qp.c +6 −7
  101. drivers/infiniband/sw/rxe/rxe_task.c +6 −2
  102. drivers/infiniband/sw/siw/siw_verbs.c +14 −11
  103. drivers/infiniband/ulp/ipoib/ipoib_main.c +11 −10
  104. drivers/infiniband/ulp/srpt/ib_srpt.c +7 −9
  105. drivers/net/ethernet/broadcom/bnxt/bnxt.c +2 −0
  106. drivers/net/ethernet/broadcom/bnxt/bnxt.h +3 −0
  107. drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +6 −0
  108. drivers/net/ethernet/pensando/Kconfig +1 −0
  109. drivers/net/ethernet/pensando/ionic/Makefile +1 −1
  110. drivers/net/ethernet/pensando/ionic/ionic.h +0 −7
  111. drivers/net/ethernet/pensando/ionic/ionic_api.h +131 −0
  112. drivers/net/ethernet/pensando/ionic/ionic_aux.c +102 −0
  113. drivers/net/ethernet/pensando/ionic/ionic_aux.h +10 −0
  114. drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +7 −0
  115. drivers/net/ethernet/pensando/ionic/ionic_dev.c +255 −15
  116. drivers/net/ethernet/pensando/ionic/ionic_dev.h +13 −15
  117. drivers/net/ethernet/pensando/ionic/ionic_if.h +113 −5
  118. drivers/net/ethernet/pensando/ionic/ionic_lif.c +27 −20
  119. drivers/net/ethernet/pensando/ionic/ionic_lif.h +3 −0
  120. drivers/net/ethernet/pensando/ionic/ionic_main.c +3 −1
  121. include/rdma/ib_mad.h +1 −0
  122. include/rdma/ib_sa.h +37 −0
  123. include/rdma/rdma_cm.h +20 −1
  124. include/uapi/rdma/ib_user_ioctl_verbs.h +1 −0
  125. include/uapi/rdma/ib_user_sa.h +14 −0
  126. include/uapi/rdma/ionic-abi.h +115 −0
  127. include/uapi/rdma/irdma-abi.h +15 −1
  128. include/uapi/rdma/rdma_user_cm.h +40 −2
Documentation/networking/device_drivers/ethernet/index.rst+1−0
@@ -50,6 +50,7 @@ Contents: · 49 unmodified lines
5050 neterion/s2io
5151 netronome/nfp
5252 pensando/ionic
53+ pensando/ionic_rdma
5354 qualcomm/ppe/ppe
5455 smsc/smc9
5556 stmicro/stmmac
Documentation/networking/device_drivers/ethernet/pensando/ionic.rst+10−0
@@ -13,6 +13,7 @@ Contents · 12 unmodified lines
1313 - Identifying the Adapter
1414 - Enabling the driver
1515 - Configuring the driver
16+- RDMA Support via Auxiliary Device
1617 - Statistics
1718 - Support
1819
@@ -105,6 +106,15 @@ XDP · 86 unmodified lines
105106 Support for XDP includes the basics, plus Jumbo frames, Redirect and
106107 ndo_xmit. There is no current support for zero-copy sockets or HW offload.
107108
109+RDMA Support via Auxiliary Device
110+=================================
111+
112+The ionic driver supports RDMA (Remote Direct Memory Access) functionality
113+through the Linux auxiliary device framework when advertised by the firmware.
114+RDMA capability is detected during device initialization, and if supported,
115+the ethernet driver will create an auxiliary device that allows the RDMA
116+driver to bind and provide InfiniBand/RoCE functionality.
117+
108118 Statistics
109119 ==========
110120
Documentation/networking/device_drivers/ethernet/pensando/ionic_rdma.rst+52−0
@@ -0,0 +1,52 @@
1+.. SPDX-License-Identifier: GPL-2.0+
2+
3+===========================================================
4+RDMA Driver for the AMD Pensando(R) Ethernet adapter family
5+===========================================================
6+
7+AMD Pensando RDMA driver.
8+Copyright (C) 2018-2025, Advanced Micro Devices, Inc.
9+
10+Overview
11+========
12+
13+The ionic_rdma driver provides Remote Direct Memory Access functionality
14+for AMD Pensando DSC (Distributed Services Card) devices. This driver
15+implements RDMA capabilities as an auxiliary driver that operates in
16+conjunction with the ionic ethernet driver.
17+
18+The ionic ethernet driver detects RDMA capability during device
19+initialization and creates auxiliary devices that the ionic_rdma driver
20+binds to, establishing the RDMA data path and control interfaces.
21+
22+Identifying the Adapter
23+=======================
24+
25+See Documentation/networking/device_drivers/ethernet/pensando/ionic.rst
26+for more information on identifying the adapter.
27+
28+Enabling the driver
29+===================
30+
31+The ionic_rdma driver depends on the ionic ethernet driver.
32+See Documentation/networking/device_drivers/ethernet/pensando/ionic.rst
33+for detailed information on enabling and configuring the ionic driver.
34+
35+The ionic_rdma driver is enabled via the standard kernel configuration system,
36+using the make command::
37+
38+ make oldconfig/menuconfig/etc.
39+
40+The driver is located in the menu structure at:
41+
42+ -> Device Drivers
43+ -> InfiniBand support
44+ -> AMD Pensando DSC RDMA/RoCE Support
45+
46+Support
47+=======
48+
49+For general Linux RDMA support, please use the RDMA mailing
50+list, which is monitored by AMD Pensando personnel::
51+
52+ linux-rdma@vger.kernel.org
MAINTAINERS+9−0
@@ -1177,6 +1177,15 @@ F: Documentation/networking/device_drivers/ethernet/amd/pds_core.rst · 1176 unmodified lines
11771177 F: drivers/net/ethernet/amd/pds_core/
11781178 F: include/linux/pds/
11791179
1180+AMD PENSANDO RDMA DRIVER
1181+M: Abhijit Gangurde <abhijit.gangurde@amd.com>
1182+M: Allen Hubbe <allen.hubbe@amd.com>
1183+L: linux-rdma@vger.kernel.org
1184+S: Maintained
1185+F: Documentation/networking/device_drivers/ethernet/pensando/ionic_rdma.rst
1186+F: drivers/infiniband/hw/ionic/
1187+F: include/uapi/rdma/ionic-abi.h
1188+
11801189 AMD PMC DRIVER
11811190 M: Shyam Sundar S K <Shyam-sundar.S-k@amd.com>
11821191 L: platform-driver-x86@vger.kernel.org
drivers/infiniband/Kconfig+1−0
@@ -85,6 +85,7 @@ source "drivers/infiniband/hw/efa/Kconfig" · 84 unmodified lines
8585 source "drivers/infiniband/hw/erdma/Kconfig"
8686 source "drivers/infiniband/hw/hfi1/Kconfig"
8787 source "drivers/infiniband/hw/hns/Kconfig"
88+source "drivers/infiniband/hw/ionic/Kconfig"
8889 source "drivers/infiniband/hw/irdma/Kconfig"
8990 source "drivers/infiniband/hw/mana/Kconfig"
9091 source "drivers/infiniband/hw/mlx4/Kconfig"
drivers/infiniband/core/addr.c+39−44
@@ -446,63 +446,41 @@ static int addr6_resolve(struct sockaddr *src_sock, · 445 unmodified lines
446446 }
447447 #endif
448448
449+static bool is_dst_local(const struct dst_entry *dst)
450+{
451+ if (dst->ops->family == AF_INET)
452+ return !!(dst_rtable(dst)->rt_type & RTN_LOCAL);
453+ else if (dst->ops->family == AF_INET6)
454+ return !!(dst_rt6_info(dst)->rt6i_flags & RTF_LOCAL);
455+ else
456+ return false;
457+}
458+
449459 static int addr_resolve_neigh(const struct dst_entry *dst,
450460 const struct sockaddr *dst_in,
451461 struct rdma_dev_addr *addr,
452− unsigned int ndev_flags,
453462 u32 seq)
454463 {
455− int ret = 0;
456−
457− if (ndev_flags & IFF_LOOPBACK) {
464+ if (is_dst_local(dst)) {
465+ /* When the destination is local entry, source and destination
466+ * are same. Skip the neighbour lookup.
467+ */
458468 memcpy(addr->dst_dev_addr, addr->src_dev_addr, MAX_ADDR_LEN);
459− } else {
460− if (!(ndev_flags & IFF_NOARP)) {
461− /* If the device doesn't do ARP internally */
462− ret = fetch_ha(dst, addr, dst_in, seq);
463− }
469+ return 0;
464470 }
465− return ret;
466−}
467−
468−static int copy_src_l2_addr(struct rdma_dev_addr *dev_addr,
469− const struct sockaddr *dst_in,
470− const struct dst_entry *dst,
471− const struct net_device *ndev)
472−{
473− int ret = 0;
474−
475− if (dst->dev->flags & IFF_LOOPBACK)
476− ret = rdma_translate_ip(dst_in, dev_addr);
477− else
478− rdma_copy_src_l2_addr(dev_addr, dst->dev);
479−
480− /*
481− * If there's a gateway and type of device not ARPHRD_INFINIBAND,
482− * we're definitely in RoCE v2 (as RoCE v1 isn't routable) set the
483− * network type accordingly.
484− */
485− if (has_gateway(dst, dst_in->sa_family) &&
486− ndev->type != ARPHRD_INFINIBAND)
487− dev_addr->network = dst_in->sa_family == AF_INET ?
488− RDMA_NETWORK_IPV4 :
489− RDMA_NETWORK_IPV6;
490− else
491− dev_addr->network = RDMA_NETWORK_IB;
492471
493− return ret;
472+ return fetch_ha(dst, addr, dst_in, seq);
494473 }
495474
496475 static int rdma_set_src_addr_rcu(struct rdma_dev_addr *dev_addr,
497− unsigned int *ndev_flags,
498476 const struct sockaddr *dst_in,
499477 const struct dst_entry *dst)
500478 {
501479 struct net_device *ndev = READ_ONCE(dst->dev);
502480
503− *ndev_flags = ndev->flags;
504481 /* A physical device must be the RDMA device to use */
505− if (ndev->flags & IFF_LOOPBACK) {
482+ if (is_dst_local(dst)) {
483+ int ret;
506484 /*
507485 * RDMA (IB/RoCE, iWarp) doesn't run on lo interface or
508486 * loopback IP address. So if route is resolved to loopback
@@ -512,9 +490,27 @@ static int rdma_set_src_addr_rcu(struct rdma_dev_addr *dev_addr, · 3 unmodified lines
512490 ndev = rdma_find_ndev_for_src_ip_rcu(dev_net(ndev), dst_in);
513491 if (IS_ERR(ndev))
514492 return -ENODEV;
493+ ret = rdma_translate_ip(dst_in, dev_addr);
494+ if (ret)
495+ return ret;
496+ } else {
497+ rdma_copy_src_l2_addr(dev_addr, dst->dev);
515498 }
516499
517− return copy_src_l2_addr(dev_addr, dst_in, dst, ndev);
500+ /*
501+ * If there's a gateway and type of device not ARPHRD_INFINIBAND,
502+ * we're definitely in RoCE v2 (as RoCE v1 isn't routable) set the
503+ * network type accordingly.
504+ */
505+ if (has_gateway(dst, dst_in->sa_family) &&
506+ ndev->type != ARPHRD_INFINIBAND)
507+ dev_addr->network = dst_in->sa_family == AF_INET ?
508+ RDMA_NETWORK_IPV4 :
509+ RDMA_NETWORK_IPV6;
510+ else
511+ dev_addr->network = RDMA_NETWORK_IB;
512+
513+ return 0;
518514 }
519515
520516 static int set_addr_netns_by_gid_rcu(struct rdma_dev_addr *addr)
@@ -551,7 +547,6 @@ static int addr_resolve(struct sockaddr *src_in, · 30 unmodified lines
551547 u32 seq)
552548 {
553549 struct dst_entry *dst = NULL;
554− unsigned int ndev_flags = 0;
555550 struct rtable *rt = NULL;
556551 int ret;
557552
@@ -588,7 +583,7 @@ static int addr_resolve(struct sockaddr *src_in, · 30 unmodified lines
588583 rcu_read_unlock();
589584 goto done;
590585 }
591− ret = rdma_set_src_addr_rcu(addr, &ndev_flags, dst_in, dst);
586+ ret = rdma_set_src_addr_rcu(addr, dst_in, dst);
592587 rcu_read_unlock();
593588
594589 /*
@@ -596,7 +591,7 @@ static int addr_resolve(struct sockaddr *src_in, · 1 unmodified line
596591 * only if src addr translation didn't fail.
597592 */
598593 if (!ret && resolve_neigh)
599− ret = addr_resolve_neigh(dst, dst_in, addr, ndev_flags, seq);
594+ ret = addr_resolve_neigh(dst, dst_in, addr, seq);
600595
601596 if (src_in->sa_family == AF_INET)
602597 ip_rt_put(rt);
drivers/infiniband/core/agent.c+1−2
@@ -110,8 +110,7 @@ void agent_send_response(const struct ib_mad_hdr *mad_hdr, const struct ib_grh * · 109 unmodified lines
110110 agent = port_priv->agent[qpn];
111111 ah = ib_create_ah_from_wc(agent->qp->pd, wc, grh, port_num);
112112 if (IS_ERR(ah)) {
113− dev_err(&device->dev, "ib_create_ah_from_wc error %ld\n",
114− PTR_ERR(ah));
113+ dev_err(&device->dev, "ib_create_ah_from_wc error %pe\n", ah);
115114 return;
116115 }
117116
drivers/infiniband/core/cm.c+2−2
@@ -1049,8 +1049,8 @@ static noinline void cm_destroy_id_wait_timeout(struct ib_cm_id *cm_id, · 1048 unmodified lines
10491049 struct cm_id_private *cm_id_priv;
10501050
10511051 cm_id_priv = container_of(cm_id, struct cm_id_private, id);
1052− pr_err("%s: cm_id=%p timed out. state %d -> %d, refcnt=%d\n", __func__,
1053− cm_id, old_state, cm_id->state, refcount_read(&cm_id_priv->refcount));
1052+ pr_err_ratelimited("%s: cm_id=%p timed out. state %d -> %d, refcnt=%d\n", __func__,
1053+ cm_id, old_state, cm_id->state, refcount_read(&cm_id_priv->refcount));
10541054 }
10551055
10561056 static void cm_destroy_id(struct ib_cm_id *cm_id, int err)
drivers/infiniband/core/cma.c+134−2
@@ -2076,6 +2076,7 @@ static void _destroy_id(struct rdma_id_private *id_priv, · 2075 unmodified lines
20762076 kfree(id_priv->id.route.path_rec);
20772077 kfree(id_priv->id.route.path_rec_inbound);
20782078 kfree(id_priv->id.route.path_rec_outbound);
2079+ kfree(id_priv->id.route.service_recs);
20792080
20802081 put_net(id_priv->id.route.addr.dev_addr.net);
20812082 kfree(id_priv);
@@ -3382,13 +3383,18 @@ err1: · 1300 unmodified lines
33823383 int rdma_resolve_route(struct rdma_cm_id *id, unsigned long timeout_ms)
33833384 {
33843385 struct rdma_id_private *id_priv;
3386+ enum rdma_cm_state state;
33853387 int ret;
33863388
33873389 if (!timeout_ms)
33883390 return -EINVAL;
33893391
33903392 id_priv = container_of(id, struct rdma_id_private, id);
3391− if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_RESOLVED, RDMA_CM_ROUTE_QUERY))
3393+ state = id_priv->state;
3394+ if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_RESOLVED,
3395+ RDMA_CM_ROUTE_QUERY) &&
3396+ !cma_comp_exch(id_priv, RDMA_CM_ADDRINFO_RESOLVED,
3397+ RDMA_CM_ROUTE_QUERY))
33923398 return -EINVAL;
33933399
33943400 cma_id_get(id_priv);
@@ -3409,7 +3415,7 @@ int rdma_resolve_route(struct rdma_cm_id *id, unsigned long timeout_ms) · 14 unmodified lines
34093415
34103416 return 0;
34113417 err:
3412− cma_comp_exch(id_priv, RDMA_CM_ROUTE_QUERY, RDMA_CM_ADDR_RESOLVED);
3418+ cma_comp_exch(id_priv, RDMA_CM_ROUTE_QUERY, state);
34133419 cma_id_put(id_priv);
34143420 return ret;
34153421 }
@@ -5506,3 +5512,129 @@ static void __exit cma_cleanup(void) · 2090 unmodified lines
55065512
55075513 module_init(cma_init);
55085514 module_exit(cma_cleanup);
5515+
5516+static void cma_query_ib_service_handler(int status,
5517+ struct sa_service_rec *recs,
5518+ unsigned int num_recs, void *context)
5519+{
5520+ struct cma_work *work = context;
5521+ struct rdma_id_private *id_priv = work->id;
5522+ struct sockaddr_ib *addr;
5523+
5524+ if (status)
5525+ goto fail;
5526+
5527+ if (!num_recs) {
5528+ status = -ENOENT;
5529+ goto fail;
5530+ }
5531+
5532+ if (id_priv->id.route.service_recs) {
5533+ status = -EALREADY;
5534+ goto fail;
5535+ }
5536+
5537+ id_priv->id.route.service_recs =
5538+ kmalloc_array(num_recs, sizeof(*recs), GFP_KERNEL);
5539+ if (!id_priv->id.route.service_recs) {
5540+ status = -ENOMEM;
5541+ goto fail;
5542+ }
5543+
5544+ id_priv->id.route.num_service_recs = num_recs;
5545+ memcpy(id_priv->id.route.service_recs, recs, sizeof(*recs) * num_recs);
5546+
5547+ addr = (struct sockaddr_ib *)&id_priv->id.route.addr.dst_addr;
5548+ addr->sib_family = AF_IB;
5549+ addr->sib_addr = *(struct ib_addr *)&recs->gid;
5550+ addr->sib_pkey = recs->pkey;
5551+ addr->sib_sid = recs->id;
5552+ rdma_addr_set_dgid(&id_priv->id.route.addr.dev_addr,
5553+ (union ib_gid *)&addr->sib_addr);
5554+ ib_addr_set_pkey(&id_priv->id.route.addr.dev_addr,
5555+ ntohs(addr->sib_pkey));
5556+
5557+ queue_work(cma_wq, &work->work);
5558+ return;
5559+
5560+fail:
5561+ work->old_state = RDMA_CM_ADDRINFO_QUERY;
5562+ work->new_state = RDMA_CM_ADDR_BOUND;
5563+ work->event.event = RDMA_CM_EVENT_ADDRINFO_ERROR;
5564+ work->event.status = status;
5565+ pr_debug_ratelimited(
5566+ "RDMA CM: SERVICE_ERROR: failed to query service record. status %d\n",
5567+ status);
5568+ queue_work(cma_wq, &work->work);
5569+}
5570+
5571+static int cma_resolve_ib_service(struct rdma_id_private *id_priv,
5572+ struct rdma_ucm_ib_service *ibs)
5573+{
5574+ struct sa_service_rec sr = {};
5575+ ib_sa_comp_mask mask = 0;
5576+ struct cma_work *work;
5577+
5578+ work = kzalloc(sizeof(*work), GFP_KERNEL);
5579+ if (!work)
5580+ return -ENOMEM;
5581+
5582+ cma_id_get(id_priv);
5583+
5584+ work->id = id_priv;
5585+ INIT_WORK(&work->work, cma_work_handler);
5586+ work->old_state = RDMA_CM_ADDRINFO_QUERY;
5587+ work->new_state = RDMA_CM_ADDRINFO_RESOLVED;
5588+ work->event.event = RDMA_CM_EVENT_ADDRINFO_RESOLVED;
5589+
5590+ if (ibs->flags & RDMA_USER_CM_IB_SERVICE_FLAG_ID) {
5591+ sr.id = cpu_to_be64(ibs->service_id);
5592+ mask |= IB_SA_SERVICE_REC_SERVICE_ID;
5593+ }
5594+ if (ibs->flags & RDMA_USER_CM_IB_SERVICE_FLAG_NAME) {
5595+ strscpy(sr.name, ibs->service_name, sizeof(sr.name));
5596+ mask |= IB_SA_SERVICE_REC_SERVICE_NAME;
5597+ }
5598+
5599+ id_priv->query_id = ib_sa_service_rec_get(&sa_client,
5600+ id_priv->id.device,
5601+ id_priv->id.port_num,
5602+ &sr, mask,
5603+ 2000, GFP_KERNEL,
5604+ cma_query_ib_service_handler,
5605+ work, &id_priv->query);
5606+
5607+ if (id_priv->query_id < 0) {
5608+ cma_id_put(id_priv);
5609+ kfree(work);
5610+ return id_priv->query_id;
5611+ }
5612+
5613+ return 0;
5614+}
5615+
5616+int rdma_resolve_ib_service(struct rdma_cm_id *id,
5617+ struct rdma_ucm_ib_service *ibs)
5618+{
5619+ struct rdma_id_private *id_priv;
5620+ int ret;
5621+
5622+ id_priv = container_of(id, struct rdma_id_private, id);
5623+ if (!id_priv->cma_dev ||
5624+ !cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_ADDRINFO_QUERY))
5625+ return -EINVAL;
5626+
5627+ if (rdma_cap_ib_sa(id->device, id->port_num))
5628+ ret = cma_resolve_ib_service(id_priv, ibs);
5629+ else
5630+ ret = -EOPNOTSUPP;
5631+
5632+ if (ret)
5633+ goto err;
5634+
5635+ return 0;
5636+err:
5637+ cma_comp_exch(id_priv, RDMA_CM_ADDRINFO_QUERY, RDMA_CM_ADDR_BOUND);
5638+ return ret;
5639+}
5640+EXPORT_SYMBOL(rdma_resolve_ib_service);
drivers/infiniband/core/cma_priv.h+3−1
@@ -47,7 +47,9 @@ enum rdma_cm_state { · 46 unmodified lines
4747 RDMA_CM_ADDR_BOUND,
4848 RDMA_CM_LISTEN,
4949 RDMA_CM_DEVICE_REMOVAL,
50− RDMA_CM_DESTROYING
50+ RDMA_CM_DESTROYING,
51+ RDMA_CM_ADDRINFO_QUERY,
52+ RDMA_CM_ADDRINFO_RESOLVED
5153 };
5254
5355 struct rdma_id_private {
drivers/infiniband/core/device.c+1−1
@@ -1543,7 +1543,7 @@ static void __ib_unregister_device(struct ib_device *ib_dev) · 1542 unmodified lines
15431543
15441544 /*
15451545 * We have a registration lock so that all the calls to unregister are
1546− * fully fenced, once any unregister returns the device is truely
1546+ * fully fenced, once any unregister returns the device is truly
15471547 * unregistered even if multiple callers are unregistering it at the
15481548 * same time. This also interacts with the registration flow and
15491549 * provides sane semantics if register and unregister are racing.
drivers/infiniband/core/sa_query.c+270−13
@@ -107,6 +107,8 @@ struct ib_sa_device { · 106 unmodified lines
107107 struct ib_sa_query {
108108 void (*callback)(struct ib_sa_query *sa_query, int status,
109109 struct ib_sa_mad *mad);
110+ void (*rmpp_callback)(struct ib_sa_query *sa_query, int status,
111+ struct ib_mad_recv_wc *mad);
110112 void (*release)(struct ib_sa_query *);
111113 struct ib_sa_client *client;
112114 struct ib_sa_port *port;
@@ -150,6 +152,13 @@ struct ib_sa_mcmember_query { · 37 unmodified lines
150152 struct ib_sa_query sa_query;
151153 };
152154
155+struct ib_sa_service_query {
156+ void (*callback)(int status, struct sa_service_rec *rec,
157+ unsigned int num_services, void *context);
158+ void *context;
159+ struct ib_sa_query sa_query;
160+};
161+
153162 static LIST_HEAD(ib_nl_request_list);
154163 static DEFINE_SPINLOCK(ib_nl_request_lock);
155164 static atomic_t ib_nl_sa_request_seq;
@@ -684,6 +693,58 @@ static const struct ib_field guidinfo_rec_table[] = { · 528 unmodified lines
684693 .size_bits = 512 },
685694 };
686695
696+#define SERVICE_REC_FIELD(field) \
697+ .struct_offset_bytes = offsetof(struct sa_service_rec, field), \
698+ .struct_size_bytes = sizeof_field(struct sa_service_rec, field), \
699+ .field_name = "sa_service_rec:" #field
700+
701+static const struct ib_field service_rec_table[] = {
702+ { SERVICE_REC_FIELD(id),
703+ .offset_words = 0,
704+ .offset_bits = 0,
705+ .size_bits = 64 },
706+ { SERVICE_REC_FIELD(gid),
707+ .offset_words = 2,
708+ .offset_bits = 0,
709+ .size_bits = 128 },
710+ { SERVICE_REC_FIELD(pkey),
711+ .offset_words = 6,
712+ .offset_bits = 0,
713+ .size_bits = 16 },
714+ { RESERVED,
715+ .offset_words = 6,
716+ .offset_bits = 16,
717+ .size_bits = 16 },
718+ { SERVICE_REC_FIELD(lease),
719+ .offset_words = 7,
720+ .offset_bits = 0,
721+ .size_bits = 32 },
722+ { SERVICE_REC_FIELD(key),
723+ .offset_words = 8,
724+ .offset_bits = 0,
725+ .size_bits = 128 },
726+ { SERVICE_REC_FIELD(name),
727+ .offset_words = 12,
728+ .offset_bits = 0,
729+ .size_bits = 512 },
730+ { SERVICE_REC_FIELD(data_8),
731+ .offset_words = 28,
732+ .offset_bits = 0,
733+ .size_bits = 128 },
734+ { SERVICE_REC_FIELD(data_16),
735+ .offset_words = 32,
736+ .offset_bits = 0,
737+ .size_bits = 128 },
738+ { SERVICE_REC_FIELD(data_32),
739+ .offset_words = 36,
740+ .offset_bits = 0,
741+ .size_bits = 128 },
742+ { SERVICE_REC_FIELD(data_64),
743+ .offset_words = 40,
744+ .offset_bits = 0,
745+ .size_bits = 128 },
746+};
747+
687748 #define RDMA_PRIMARY_PATH_MAX_REC_NUM 3
688749
689750 static inline void ib_sa_disable_local_svc(struct ib_sa_query *query)
@@ -1013,6 +1074,8 @@ int ib_nl_handle_set_timeout(struct sk_buff *skb, · 323 unmodified lines
10131074 if (timeout > IB_SA_LOCAL_SVC_TIMEOUT_MAX)
10141075 timeout = IB_SA_LOCAL_SVC_TIMEOUT_MAX;
10151076
1077+ spin_lock_irqsave(&ib_nl_request_lock, flags);
1078+
10161079 delta = timeout - sa_local_svc_timeout_ms;
10171080 if (delta < 0)
10181081 abs_delta = -delta;
@@ -1020,7 +1083,6 @@ int ib_nl_handle_set_timeout(struct sk_buff *skb, · 1 unmodified line
10201083 abs_delta = delta;
10211084
10221085 if (delta != 0) {
1023− spin_lock_irqsave(&ib_nl_request_lock, flags);
10241086 sa_local_svc_timeout_ms = timeout;
10251087 list_for_each_entry(query, &ib_nl_request_list, list) {
10261088 if (delta < 0 && abs_delta > query->timeout)
@@ -1038,9 +1100,10 @@ int ib_nl_handle_set_timeout(struct sk_buff *skb, · 11 unmodified lines
10381100 if (delay)
10391101 mod_delayed_work(ib_nl_wq, &ib_nl_timed_work,
10401102 (unsigned long)delay);
1041− spin_unlock_irqrestore(&ib_nl_request_lock, flags);
10421103 }
10431104
1105+ spin_unlock_irqrestore(&ib_nl_request_lock, flags);
1106+
10441107 settimeout_out:
10451108 return 0;
10461109 }
@@ -1390,6 +1453,20 @@ void ib_sa_pack_path(struct sa_path_rec *rec, void *attribute) · 343 unmodified lines
13901453 }
13911454 EXPORT_SYMBOL(ib_sa_pack_path);
13921455
1456+void ib_sa_pack_service(struct sa_service_rec *rec, void *attribute)
1457+{
1458+ ib_pack(service_rec_table, ARRAY_SIZE(service_rec_table), rec,
1459+ attribute);
1460+}
1461+EXPORT_SYMBOL(ib_sa_pack_service);
1462+
1463+void ib_sa_unpack_service(void *attribute, struct sa_service_rec *rec)
1464+{
1465+ ib_unpack(service_rec_table, ARRAY_SIZE(service_rec_table), attribute,
1466+ rec);
1467+}
1468+EXPORT_SYMBOL(ib_sa_unpack_service);
1469+
13931470 static bool ib_sa_opa_pathrecord_support(struct ib_sa_client *client,
13941471 struct ib_sa_device *sa_dev,
13951472 u32 port_num)
@@ -1479,6 +1556,68 @@ static void ib_sa_path_rec_callback(struct ib_sa_query *sa_query, · 83 unmodified lines
14791556 }
14801557 }
14811558
1559+#define IB_SA_DATA_OFFS 56
1560+#define IB_SERVICE_REC_SZ 176
1561+
1562+static void ib_unpack_service_rmpp(struct sa_service_rec *rec,
1563+ struct ib_mad_recv_wc *mad_wc,
1564+ int num_services)
1565+{
1566+ unsigned int cp_sz, data_i, data_size, rec_i = 0, buf_i = 0;
1567+ struct ib_mad_recv_buf *mad_buf;
1568+ u8 buf[IB_SERVICE_REC_SZ];
1569+ u8 *data;
1570+
1571+ data_size = sizeof(((struct ib_sa_mad *) mad_buf->mad)->data);
1572+
1573+ list_for_each_entry(mad_buf, &mad_wc->rmpp_list, list) {
1574+ data = ((struct ib_sa_mad *) mad_buf->mad)->data;
1575+ data_i = 0;
1576+ while (data_i < data_size && rec_i < num_services) {
1577+ cp_sz = min(IB_SERVICE_REC_SZ - buf_i,
1578+ data_size - data_i);
1579+ memcpy(buf + buf_i, data + data_i, cp_sz);
1580+ data_i += cp_sz;
1581+ buf_i += cp_sz;
1582+ if (buf_i == IB_SERVICE_REC_SZ) {
1583+ ib_sa_unpack_service(buf, rec + rec_i);
1584+ buf_i = 0;
1585+ rec_i++;
1586+ }
1587+ }
1588+ }
1589+}
1590+
1591+static void ib_sa_service_rec_callback(struct ib_sa_query *sa_query, int status,
1592+ struct ib_mad_recv_wc *mad_wc)
1593+{
1594+ struct ib_sa_service_query *query =
1595+ container_of(sa_query, struct ib_sa_service_query, sa_query);
1596+ struct sa_service_rec *rec;
1597+ int num_services;
1598+
1599+ if (!mad_wc || !mad_wc->recv_buf.mad) {
1600+ query->callback(status, NULL, 0, query->context);
1601+ return;
1602+ }
1603+
1604+ num_services = (mad_wc->mad_len - IB_SA_DATA_OFFS) / IB_SERVICE_REC_SZ;
1605+ if (!num_services) {
1606+ query->callback(-ENODATA, NULL, 0, query->context);
1607+ return;
1608+ }
1609+
1610+ rec = kmalloc_array(num_services, sizeof(*rec), GFP_KERNEL);
1611+ if (!rec) {
1612+ query->callback(-ENOMEM, NULL, 0, query->context);
1613+ return;
1614+ }
1615+
1616+ ib_unpack_service_rmpp(rec, mad_wc, num_services);
1617+ query->callback(status, rec, num_services, query->context);
1618+ kfree(rec);
1619+}
1620+
14821621 static void ib_sa_path_rec_release(struct ib_sa_query *sa_query)
14831622 {
14841623 struct ib_sa_path_query *query =
@@ -1488,6 +1627,14 @@ static void ib_sa_path_rec_release(struct ib_sa_query *sa_query) · 3 unmodified lines
14881627 kfree(query);
14891628 }
14901629
1630+static void ib_sa_service_rec_release(struct ib_sa_query *sa_query)
1631+{
1632+ struct ib_sa_service_query *query =
1633+ container_of(sa_query, struct ib_sa_service_query, sa_query);
1634+
1635+ kfree(query);
1636+}
1637+
14911638 /**
14921639 * ib_sa_path_rec_get - Start a Path get query
14931640 * @client:SA client
@@ -1618,6 +1765,101 @@ err1: · 124 unmodified lines
16181765 }
16191766 EXPORT_SYMBOL(ib_sa_path_rec_get);
16201767
1768+/**
1769+ * ib_sa_service_rec_get - Start a Service get query
1770+ * @client: SA client
1771+ * @device: device to send query on
1772+ * @port_num: port number to send query on
1773+ * @rec: Service Record to send in query
1774+ * @comp_mask: component mask to send in query
1775+ * @timeout_ms: time to wait for response
1776+ * @gfp_mask: GFP mask to use for internal allocations
1777+ * @callback: function called when query completes, times out or is
1778+ * canceled
1779+ * @context: opaque user context passed to callback
1780+ * @sa_query: query context, used to cancel query
1781+ *
1782+ * Send a Service Record Get query to the SA to look up a path. The
1783+ * callback function will be called when the query completes (or
1784+ * fails); status is 0 for a successful response, -EINTR if the query
1785+ * is canceled, -ETIMEDOUT is the query timed out, or -EIO if an error
1786+ * occurred sending the query. The resp parameter of the callback is
1787+ * only valid if status is 0.
1788+ *
1789+ * If the return value of ib_sa_service_rec_get() is negative, it is an
1790+ * error code. Otherwise it is a query ID that can be used to cancel
1791+ * the query.
1792+ */
1793+int ib_sa_service_rec_get(struct ib_sa_client *client,
1794+ struct ib_device *device, u32 port_num,
1795+ struct sa_service_rec *rec,
1796+ ib_sa_comp_mask comp_mask,
1797+ unsigned long timeout_ms, gfp_t gfp_mask,
1798+ void (*callback)(int status,
1799+ struct sa_service_rec *resp,
1800+ unsigned int num_services,
1801+ void *context),
1802+ void *context, struct ib_sa_query **sa_query)
1803+{
1804+ struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client);
1805+ struct ib_sa_service_query *query;
1806+ struct ib_mad_agent *agent;
1807+ struct ib_sa_port *port;
1808+ struct ib_sa_mad *mad;
1809+ int ret;
1810+
1811+ if (!sa_dev)
1812+ return -ENODEV;
1813+
1814+ port = &sa_dev->port[port_num - sa_dev->start_port];
1815+ agent = port->agent;
1816+
1817+ query = kzalloc(sizeof(*query), gfp_mask);
1818+ if (!query)
1819+ return -ENOMEM;
1820+
1821+ query->sa_query.port = port;
1822+
1823+ ret = alloc_mad(&query->sa_query, gfp_mask);
1824+ if (ret)
1825+ goto err1;
1826+
1827+ ib_sa_client_get(client);
1828+ query->sa_query.client = client;
1829+ query->callback = callback;
1830+ query->context = context;
1831+
1832+ mad = query->sa_query.mad_buf->mad;
1833+ init_mad(&query->sa_query, agent);
1834+
1835+ query->sa_query.rmpp_callback = callback ? ib_sa_service_rec_callback :
1836+ NULL;
1837+ query->sa_query.release = ib_sa_service_rec_release;
1838+ mad->mad_hdr.method = IB_MGMT_METHOD_GET_TABLE;
1839+ mad->mad_hdr.attr_id = cpu_to_be16(IB_SA_ATTR_SERVICE_REC);
1840+ mad->sa_hdr.comp_mask = comp_mask;
1841+
1842+ ib_sa_pack_service(rec, mad->data);
1843+
1844+ *sa_query = &query->sa_query;
1845+ query->sa_query.mad_buf->context[1] = rec;
1846+
1847+ ret = send_mad(&query->sa_query, timeout_ms, gfp_mask);
1848+ if (ret < 0)
1849+ goto err2;
1850+
1851+ return ret;
1852+
1853+err2:
1854+ *sa_query = NULL;
1855+ ib_sa_client_put(query->sa_query.client);
1856+ free_mad(&query->sa_query);
1857+err1:
1858+ kfree(query);
1859+ return ret;
1860+}
1861+EXPORT_SYMBOL(ib_sa_service_rec_get);
1862+
16211863 static void ib_sa_mcmember_rec_callback(struct ib_sa_query *sa_query,
16221864 int status, struct ib_sa_mad *mad)
16231865 {
@@ -1987,23 +2229,29 @@ static void send_handler(struct ib_mad_agent *agent, · 363 unmodified lines
19872229 {
19882230 struct ib_sa_query *query = mad_send_wc->send_buf->context[0];
19892231 unsigned long flags;
2232+ int status = 0;
19902233
1991− if (query->callback)
2234+ if (query->callback || query->rmpp_callback) {
19922235 switch (mad_send_wc->status) {
19932236 case IB_WC_SUCCESS:
19942237 /* No callback -- already got recv */
19952238 break;
19962239 case IB_WC_RESP_TIMEOUT_ERR:
1997− query->callback(query, -ETIMEDOUT, NULL);
2240+ status = -ETIMEDOUT;
19982241 break;
19992242 case IB_WC_WR_FLUSH_ERR:
2000− query->callback(query, -EINTR, NULL);
2243+ status = -EINTR;
20012244 break;
20022245 default:
2003− query->callback(query, -EIO, NULL);
2246+ status = -EIO;
20042247 break;
20052248 }
20062249
2250+ if (status)
2251+ query->callback ? query->callback(query, status, NULL) :
2252+ query->rmpp_callback(query, status, NULL);
2253+ }
2254+
20072255 xa_lock_irqsave(&queries, flags);
20082256 __xa_erase(&queries, query->id);
20092257 xa_unlock_irqrestore(&queries, flags);
@@ -2019,17 +2267,25 @@ static void recv_handler(struct ib_mad_agent *mad_agent, · 9 unmodified lines
20192267 struct ib_mad_recv_wc *mad_recv_wc)
20202268 {
20212269 struct ib_sa_query *query;
2270+ struct ib_mad *mad;
2271+
20222272
20232273 if (!send_buf)
20242274 return;
20252275
20262276 query = send_buf->context[0];
2027− if (query->callback) {
2277+ mad = mad_recv_wc->recv_buf.mad;
2278+
2279+ if (query->rmpp_callback) {
2280+ if (mad_recv_wc->wc->status == IB_WC_SUCCESS)
2281+ query->rmpp_callback(query, mad->mad_hdr.status ?
2282+ -EINVAL : 0, mad_recv_wc);
2283+ else
2284+ query->rmpp_callback(query, -EIO, NULL);
2285+ } else if (query->callback) {
20282286 if (mad_recv_wc->wc->status == IB_WC_SUCCESS)
2029− query->callback(query,
2030− mad_recv_wc->recv_buf.mad->mad_hdr.status ?
2031− -EINVAL : 0,
2032− (struct ib_sa_mad *) mad_recv_wc->recv_buf.mad);
2287+ query->callback(query, mad->mad_hdr.status ?
2288+ -EINVAL : 0, (struct ib_sa_mad *)mad);
20332289 else
20342290 query->callback(query, -EIO, NULL);
20352291 }
@@ -2181,8 +2437,9 @@ static int ib_sa_add_one(struct ib_device *device) · 145 unmodified lines
21812437
21822438 sa_dev->port[i].agent =
21832439 ib_register_mad_agent(device, i + s, IB_QPT_GSI,
2184− NULL, 0, send_handler,
2185− recv_handler, sa_dev, 0);
2440+ NULL, IB_MGMT_RMPP_VERSION,
2441+ send_handler, recv_handler,
2442+ sa_dev, 0);
21862443 if (IS_ERR(sa_dev->port[i].agent)) {
21872444 ret = PTR_ERR(sa_dev->port[i].agent);
21882445 goto err;
drivers/infiniband/core/ucma.c+119−1
@@ -282,6 +282,10 @@ static struct ucma_event *ucma_create_uevent(struct ucma_context *ctx, · 281 unmodified lines
282282 }
283283 uevent->resp.event = event->event;
284284 uevent->resp.status = event->status;
285+
286+ if (event->event == RDMA_CM_EVENT_ADDRINFO_RESOLVED)
287+ goto out;
288+
285289 if (ctx->cm_id->qp_type == IB_QPT_UD)
286290 ucma_copy_ud_event(ctx->cm_id->device, &uevent->resp.param.ud,
287291 &event->param.ud);
@@ -289,6 +293,7 @@ static struct ucma_event *ucma_create_uevent(struct ucma_context *ctx, · 1 unmodified line
289293 ucma_copy_conn_event(&uevent->resp.param.conn,
290294 &event->param.conn);
291295
296+out:
292297 uevent->resp.ece.vendor_id = event->ece.vendor_id;
293298 uevent->resp.ece.attr_mod = event->ece.attr_mod;
294299 return uevent;
@@ -728,6 +733,28 @@ static ssize_t ucma_resolve_addr(struct ucma_file *file, · 433 unmodified lines
728733 return ret;
729734 }
730735
736+static ssize_t ucma_resolve_ib_service(struct ucma_file *file,
737+ const char __user *inbuf, int in_len,
738+ int out_len)
739+{
740+ struct rdma_ucm_resolve_ib_service cmd;
741+ struct ucma_context *ctx;
742+ int ret;
743+
744+ if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
745+ return -EFAULT;
746+
747+ ctx = ucma_get_ctx(file, cmd.id);
748+ if (IS_ERR(ctx))
749+ return PTR_ERR(ctx);
750+
751+ mutex_lock(&ctx->mutex);
752+ ret = rdma_resolve_ib_service(ctx->cm_id, &cmd.ibs);
753+ mutex_unlock(&ctx->mutex);
754+ ucma_put_ctx(ctx);
755+ return ret;
756+}
757+
731758 static ssize_t ucma_resolve_route(struct ucma_file *file,
732759 const char __user *inbuf,
733760 int in_len, int out_len)
@@ -994,6 +1021,43 @@ static ssize_t ucma_query_gid(struct ucma_context *ctx, · 260 unmodified lines
9941021 return ret;
9951022 }
9961023
1024+static ssize_t ucma_query_ib_service(struct ucma_context *ctx,
1025+ void __user *response, int out_len)
1026+{
1027+ struct rdma_ucm_query_ib_service_resp *resp;
1028+ int n, ret = 0;
1029+
1030+ if (out_len < sizeof(struct rdma_ucm_query_ib_service_resp))
1031+ return -ENOSPC;
1032+
1033+ if (!ctx->cm_id->route.service_recs)
1034+ return -ENODATA;
1035+
1036+ resp = kzalloc(out_len, GFP_KERNEL);
1037+ if (!resp)
1038+ return -ENOMEM;
1039+
1040+ resp->num_service_recs = ctx->cm_id->route.num_service_recs;
1041+
1042+ n = (out_len - sizeof(struct rdma_ucm_query_ib_service_resp)) /
1043+ sizeof(struct ib_user_service_rec);
1044+
1045+ if (!n)
1046+ goto out;
1047+
1048+ if (n > ctx->cm_id->route.num_service_recs)
1049+ n = ctx->cm_id->route.num_service_recs;
1050+
1051+ memcpy(resp->recs, ctx->cm_id->route.service_recs,
1052+ sizeof(*resp->recs) * n);
1053+ if (copy_to_user(response, resp, struct_size(resp, recs, n)))
1054+ ret = -EFAULT;
1055+
1056+out:
1057+ kfree(resp);
1058+ return ret;
1059+}
1060+
9971061 static ssize_t ucma_query(struct ucma_file *file,
9981062 const char __user *inbuf,
9991063 int in_len, int out_len)
@@ -1022,6 +1086,9 @@ static ssize_t ucma_query(struct ucma_file *file, · 22 unmodified lines
10221086 case RDMA_USER_CM_QUERY_GID:
10231087 ret = ucma_query_gid(ctx, response, out_len);
10241088 break;
1089+ case RDMA_USER_CM_QUERY_IB_SERVICE:
1090+ ret = ucma_query_ib_service(ctx, response, out_len);
1091+ break;
10251092 default:
10261093 ret = -ENOSYS;
10271094 break;
@@ -1678,6 +1745,55 @@ err_unlock: · 650 unmodified lines
16781745 return ret;
16791746 }
16801747
1748+static ssize_t ucma_write_cm_event(struct ucma_file *file,
1749+ const char __user *inbuf, int in_len,
1750+ int out_len)
1751+{
1752+ struct rdma_ucm_write_cm_event cmd;
1753+ struct rdma_cm_event event = {};
1754+ struct ucma_event *uevent;
1755+ struct ucma_context *ctx;
1756+ int ret = 0;
1757+
1758+ if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
1759+ return -EFAULT;
1760+
1761+ if ((cmd.event != RDMA_CM_EVENT_USER) &&
1762+ (cmd.event != RDMA_CM_EVENT_INTERNAL))
1763+ return -EINVAL;
1764+
1765+ ctx = ucma_get_ctx(file, cmd.id);
1766+ if (IS_ERR(ctx))
1767+ return PTR_ERR(ctx);
1768+
1769+ event.event = cmd.event;
1770+ event.status = cmd.status;
1771+ event.param.arg = cmd.param.arg;
1772+
1773+ uevent = kzalloc(sizeof(*uevent), GFP_KERNEL);
1774+ if (!uevent) {
1775+ ret = -ENOMEM;
1776+ goto out;
1777+ }
1778+
1779+ uevent->ctx = ctx;
1780+ uevent->resp.uid = ctx->uid;
1781+ uevent->resp.id = ctx->id;
1782+ uevent->resp.event = event.event;
1783+ uevent->resp.status = event.status;
1784+ memcpy(uevent->resp.param.arg32, &event.param.arg,
1785+ sizeof(event.param.arg));
1786+
1787+ mutex_lock(&ctx->file->mut);
1788+ list_add_tail(&uevent->list, &ctx->file->event_list);
1789+ mutex_unlock(&ctx->file->mut);
1790+ wake_up_interruptible(&ctx->file->poll_wait);
1791+
1792+out:
1793+ ucma_put_ctx(ctx);
1794+ return ret;
1795+}
1796+
16811797 static ssize_t (*ucma_cmd_table[])(struct ucma_file *file,
16821798 const char __user *inbuf,
16831799 int in_len, int out_len) = {
@@ -1703,7 +1819,9 @@ static ssize_t (*ucma_cmd_table[])(struct ucma_file *file, · 19 unmodified lines
17031819 [RDMA_USER_CM_CMD_QUERY] = ucma_query,
17041820 [RDMA_USER_CM_CMD_BIND] = ucma_bind,
17051821 [RDMA_USER_CM_CMD_RESOLVE_ADDR] = ucma_resolve_addr,
1706− [RDMA_USER_CM_CMD_JOIN_MCAST] = ucma_join_multicast
1822+ [RDMA_USER_CM_CMD_JOIN_MCAST] = ucma_join_multicast,
1823+ [RDMA_USER_CM_CMD_RESOLVE_IB_SERVICE] = ucma_resolve_ib_service,
1824+ [RDMA_USER_CM_CMD_WRITE_CM_EVENT] = ucma_write_cm_event,
17071825 };
17081826
17091827 static ssize_t ucma_write(struct file *filp, const char __user *buf,
drivers/infiniband/hw/Makefile+1−0
@@ -14,3 +14,4 @@ obj-$(CONFIG_INFINIBAND_HNS_HIP08) += hns/ · 13 unmodified lines
1414 obj-$(CONFIG_INFINIBAND_QEDR) += qedr/
1515 obj-$(CONFIG_INFINIBAND_BNXT_RE) += bnxt_re/
1616 obj-$(CONFIG_INFINIBAND_ERDMA) += erdma/
17+obj-$(CONFIG_INFINIBAND_IONIC) += ionic/
drivers/infiniband/hw/bnxt_re/bnxt_re.h+15−4
@@ -172,9 +172,9 @@ struct bnxt_re_dev { · 171 unmodified lines
172172 struct list_head list;
173173 unsigned long flags;
174174 #define BNXT_RE_FLAG_NETDEV_REGISTERED 0
175+#define BNXT_RE_FLAG_STATS_CTX3_ALLOC 1
175176 #define BNXT_RE_FLAG_HAVE_L2_REF 3
176177 #define BNXT_RE_FLAG_RCFW_CHANNEL_EN 4
177−#define BNXT_RE_FLAG_QOS_WORK_REG 5
178178 #define BNXT_RE_FLAG_RESOURCES_ALLOCATED 7
179179 #define BNXT_RE_FLAG_RESOURCES_INITIALIZED 8
180180 #define BNXT_RE_FLAG_ERR_DEVICE_DETACHED 17
@@ -187,9 +187,6 @@ struct bnxt_re_dev { · 6 unmodified lines
187187
188188 int id;
189189
190− struct delayed_work worker;
191− u8 cur_prio_map;
192−
193190 /* RCFW Channel */
194191 struct bnxt_qplib_rcfw rcfw;
195192
@@ -227,6 +224,13 @@ struct bnxt_re_dev { · 31 unmodified lines
227224 struct workqueue_struct *dcb_wq;
228225 struct dentry *cc_config;
229226 struct bnxt_re_dbg_cc_config_params *cc_config_params;
227+#define BNXT_VPD_FLD_LEN 32
228+ char board_partno[BNXT_VPD_FLD_LEN];
229+ /* RoCE mirror */
230+ u16 mirror_vnic_id;
231+ union ib_gid ugid;
232+ u32 ugid_index;
233+ u8 sniffer_flow_created : 1;
230234 };
231235
232236 #define to_bnxt_re_dev(ptr, member) \
@@ -243,6 +247,10 @@ int bnxt_re_assign_pma_port_counters(struct bnxt_re_dev *rdev, struct ib_mad *ou · 10 unmodified lines
243247 int bnxt_re_assign_pma_port_ext_counters(struct bnxt_re_dev *rdev,
244248 struct ib_mad *out_mad);
245249
250+void bnxt_re_hwrm_free_vnic(struct bnxt_re_dev *rdev);
251+int bnxt_re_hwrm_alloc_vnic(struct bnxt_re_dev *rdev);
252+int bnxt_re_hwrm_cfg_vnic(struct bnxt_re_dev *rdev, u32 qp_id);
253+
246254 static inline struct device *rdev_to_dev(struct bnxt_re_dev *rdev)
247255 {
248256 if (rdev)
@@ -276,4 +284,7 @@ static inline int bnxt_re_read_context_allowed(struct bnxt_re_dev *rdev) · 27 unmodified lines
276284 #define BNXT_RE_CONTEXT_TYPE_MRW_SIZE_P7 192
277285 #define BNXT_RE_CONTEXT_TYPE_SRQ_SIZE_P7 192
278286
287+#define BNXT_RE_HWRM_CMD_TIMEOUT(rdev) \
288+ ((rdev)->chip_ctx->hwrm_cmd_max_timeout * 1000)
289+
279290 #endif
drivers/infiniband/hw/bnxt_re/debugfs.c+37−0
@@ -8,6 +8,7 @@ · 7 unmodified lines
88
99 #include <linux/debugfs.h>
1010 #include <linux/pci.h>
11+#include <linux/seq_file.h>
1112 #include <rdma/ib_addr.h>
1213
1314 #include "bnxt_ulp.h"
@@ -314,6 +315,40 @@ static const struct file_operations bnxt_re_cc_config_ops = { · 300 unmodified lines
314315 .write = bnxt_re_cc_config_set,
315316 };
316317
318+static int info_show(struct seq_file *m, void *unused)
319+{
320+ struct bnxt_re_dev *rdev = m->private;
321+ struct bnxt_re_res_cntrs *res_s = &rdev->stats.res;
322+
323+ seq_puts(m, "Info:\n");
324+ seq_printf(m, "Device Name\t\t: %s\n", dev_name(&rdev->ibdev.dev));
325+ seq_printf(m, "PD Watermark\t\t: %llu\n", res_s->pd_watermark);
326+ seq_printf(m, "AH Watermark\t\t: %llu\n", res_s->ah_watermark);
327+ seq_printf(m, "QP Watermark\t\t: %llu\n", res_s->qp_watermark);
328+ seq_printf(m, "RC QP Watermark\t\t: %llu\n", res_s->rc_qp_watermark);
329+ seq_printf(m, "UD QP Watermark\t\t: %llu\n", res_s->ud_qp_watermark);
330+ seq_printf(m, "SRQ Watermark\t\t: %llu\n", res_s->srq_watermark);
331+ seq_printf(m, "CQ Watermark\t\t: %llu\n", res_s->cq_watermark);
332+ seq_printf(m, "MR Watermark\t\t: %llu\n", res_s->mr_watermark);
333+ seq_printf(m, "MW Watermark\t\t: %llu\n", res_s->mw_watermark);
334+ seq_printf(m, "CQ Resize Count\t\t: %d\n", atomic_read(&res_s->resize_count));
335+ if (rdev->pacing.dbr_pacing) {
336+ seq_printf(m, "DB Pacing Reschedule\t: %llu\n", rdev->stats.pacing.resched);
337+ seq_printf(m, "DB Pacing Complete\t: %llu\n", rdev->stats.pacing.complete);
338+ seq_printf(m, "DB Pacing Alerts\t: %llu\n", rdev->stats.pacing.alerts);
339+ seq_printf(m, "DB FIFO Register\t: 0x%x\n",
340+ readl(rdev->en_dev->bar0 + rdev->pacing.dbr_db_fifo_reg_off));
341+ }
342+
343+ return 0;
344+}
345+DEFINE_SHOW_ATTRIBUTE(info);
346+
347+static void bnxt_re_debugfs_add_info(struct bnxt_re_dev *rdev)
348+{
349+ debugfs_create_file("info", 0400, rdev->dbg_root, rdev, &info_fops);
350+}
351+
317352 void bnxt_re_debugfs_add_pdev(struct bnxt_re_dev *rdev)
318353 {
319354 struct pci_dev *pdev = rdev->en_dev->pdev;
@@ -325,6 +360,8 @@ void bnxt_re_debugfs_add_pdev(struct bnxt_re_dev *rdev) · 5 unmodified lines
325360 rdev->qp_debugfs = debugfs_create_dir("QPs", rdev->dbg_root);
326361 rdev->cc_config = debugfs_create_dir("cc_config", rdev->dbg_root);
327362
363+ bnxt_re_debugfs_add_info(rdev);
364+
328365 rdev->cc_config_params = kzalloc(sizeof(*cc_params), GFP_KERNEL);
329366
330367 for (i = 0; i < BNXT_RE_CC_PARAM_GEN0; i++) {
drivers/infiniband/hw/bnxt_re/hw_counters.c+34−75
@@ -51,25 +51,6 @@ · 50 unmodified lines
5151 #include "hw_counters.h"
5252
5353 static const struct rdma_stat_desc bnxt_re_stat_descs[] = {
54− [BNXT_RE_ACTIVE_PD].name = "active_pds",
55− [BNXT_RE_ACTIVE_AH].name = "active_ahs",
56− [BNXT_RE_ACTIVE_QP].name = "active_qps",
57− [BNXT_RE_ACTIVE_RC_QP].name = "active_rc_qps",
58− [BNXT_RE_ACTIVE_UD_QP].name = "active_ud_qps",
59− [BNXT_RE_ACTIVE_SRQ].name = "active_srqs",
60− [BNXT_RE_ACTIVE_CQ].name = "active_cqs",
61− [BNXT_RE_ACTIVE_MR].name = "active_mrs",
62− [BNXT_RE_ACTIVE_MW].name = "active_mws",
63− [BNXT_RE_WATERMARK_PD].name = "watermark_pds",
64− [BNXT_RE_WATERMARK_AH].name = "watermark_ahs",
65− [BNXT_RE_WATERMARK_QP].name = "watermark_qps",
66− [BNXT_RE_WATERMARK_RC_QP].name = "watermark_rc_qps",
67− [BNXT_RE_WATERMARK_UD_QP].name = "watermark_ud_qps",
68− [BNXT_RE_WATERMARK_SRQ].name = "watermark_srqs",
69− [BNXT_RE_WATERMARK_CQ].name = "watermark_cqs",
70− [BNXT_RE_WATERMARK_MR].name = "watermark_mrs",
71− [BNXT_RE_WATERMARK_MW].name = "watermark_mws",
72− [BNXT_RE_RESIZE_CQ_CNT].name = "resize_cq_cnt",
7354 [BNXT_RE_RX_PKTS].name = "rx_pkts",
7455 [BNXT_RE_RX_BYTES].name = "rx_bytes",
7556 [BNXT_RE_TX_PKTS].name = "tx_pkts",
@@ -79,22 +60,22 @@ static const struct rdma_stat_desc bnxt_re_stat_descs[] = { · 3 unmodified lines
7960 [BNXT_RE_TX_DISCARDS].name = "tx_roce_discards",
8061 [BNXT_RE_RX_ERRORS].name = "rx_roce_errors",
8162 [BNXT_RE_RX_DISCARDS].name = "rx_roce_discards",
82− [BNXT_RE_TO_RETRANSMITS].name = "to_retransmits",
83− [BNXT_RE_SEQ_ERR_NAKS_RCVD].name = "seq_err_naks_rcvd",
84− [BNXT_RE_MAX_RETRY_EXCEEDED].name = "max_retry_exceeded",
85− [BNXT_RE_RNR_NAKS_RCVD].name = "rnr_naks_rcvd",
86− [BNXT_RE_MISSING_RESP].name = "missing_resp",
63+ [BNXT_RE_TO_RETRANSMITS].name = "local_ack_timeout_err",
64+ [BNXT_RE_SEQ_ERR_NAKS_RCVD].name = "packet_seq_err",
65+ [BNXT_RE_MAX_RETRY_EXCEEDED].name = "max_retry_exceeded",
66+ [BNXT_RE_RNR_NAKS_RCVD].name = "rnr_nak_retry_err",
67+ [BNXT_RE_MISSING_RESP].name = "implied_nak_seq_err",
8768 [BNXT_RE_UNRECOVERABLE_ERR].name = "unrecoverable_err",
8869 [BNXT_RE_BAD_RESP_ERR].name = "bad_resp_err",
8970 [BNXT_RE_LOCAL_QP_OP_ERR].name = "local_qp_op_err",
9071 [BNXT_RE_LOCAL_PROTECTION_ERR].name = "local_protection_err",
9172 [BNXT_RE_MEM_MGMT_OP_ERR].name = "mem_mgmt_op_err",
92− [BNXT_RE_REMOTE_INVALID_REQ_ERR].name = "remote_invalid_req_err",
93− [BNXT_RE_REMOTE_ACCESS_ERR].name = "remote_access_err",
73+ [BNXT_RE_REMOTE_INVALID_REQ_ERR].name = "req_remote_invalid_request",
74+ [BNXT_RE_REMOTE_ACCESS_ERR].name = "req_remote_access_errors",
9475 [BNXT_RE_REMOTE_OP_ERR].name = "remote_op_err",
95− [BNXT_RE_DUP_REQ].name = "dup_req",
76+ [BNXT_RE_DUP_REQ].name = "duplicate_request",
9677 [BNXT_RE_RES_EXCEED_MAX].name = "res_exceed_max",
97− [BNXT_RE_RES_LENGTH_MISMATCH].name = "res_length_mismatch",
78+ [BNXT_RE_RES_LENGTH_MISMATCH].name = "resp_local_length_error",
9879 [BNXT_RE_RES_EXCEEDS_WQE].name = "res_exceeds_wqe",
9980 [BNXT_RE_RES_OPCODE_ERR].name = "res_opcode_err",
10081 [BNXT_RE_RES_RX_INVALID_RKEY].name = "res_rx_invalid_rkey",
@@ -118,7 +99,7 @@ static const struct rdma_stat_desc bnxt_re_stat_descs[] = { · 17 unmodified lines
11899 [BNXT_RE_RES_SRQ_LOAD_ERR].name = "res_srq_load_err",
119100 [BNXT_RE_RES_TX_PCI_ERR].name = "res_tx_pci_err",
120101 [BNXT_RE_RES_RX_PCI_ERR].name = "res_rx_pci_err",
121− [BNXT_RE_OUT_OF_SEQ_ERR].name = "oos_drop_count",
102+ [BNXT_RE_OUT_OF_SEQ_ERR].name = "out_of_sequence",
122103 [BNXT_RE_TX_ATOMIC_REQ].name = "tx_atomic_req",
123104 [BNXT_RE_TX_READ_REQ].name = "tx_read_req",
124105 [BNXT_RE_TX_READ_RES].name = "tx_read_resp",
@@ -126,23 +107,22 @@ static const struct rdma_stat_desc bnxt_re_stat_descs[] = { · 1 unmodified line
126107 [BNXT_RE_TX_SEND_REQ].name = "tx_send_req",
127108 [BNXT_RE_TX_ROCE_PKTS].name = "tx_roce_only_pkts",
128109 [BNXT_RE_TX_ROCE_BYTES].name = "tx_roce_only_bytes",
129− [BNXT_RE_RX_ATOMIC_REQ].name = "rx_atomic_req",
130− [BNXT_RE_RX_READ_REQ].name = "rx_read_req",
110+ [BNXT_RE_RX_ATOMIC_REQ].name = "rx_atomic_requests",
111+ [BNXT_RE_RX_READ_REQ].name = "rx_read_requests",
131112 [BNXT_RE_RX_READ_RESP].name = "rx_read_resp",
132− [BNXT_RE_RX_WRITE_REQ].name = "rx_write_req",
113+ [BNXT_RE_RX_WRITE_REQ].name = "rx_write_requests",
133114 [BNXT_RE_RX_SEND_REQ].name = "rx_send_req",
134115 [BNXT_RE_RX_ROCE_PKTS].name = "rx_roce_only_pkts",
135116 [BNXT_RE_RX_ROCE_BYTES].name = "rx_roce_only_bytes",
136117 [BNXT_RE_RX_ROCE_GOOD_PKTS].name = "rx_roce_good_pkts",
137118 [BNXT_RE_RX_ROCE_GOOD_BYTES].name = "rx_roce_good_bytes",
138− [BNXT_RE_OOB].name = "rx_out_of_buffer",
139− [BNXT_RE_TX_CNP].name = "tx_cnp_pkts",
140− [BNXT_RE_RX_CNP].name = "rx_cnp_pkts",
141− [BNXT_RE_RX_ECN].name = "rx_ecn_marked_pkts",
142− [BNXT_RE_PACING_RESCHED].name = "pacing_reschedule",
143− [BNXT_RE_PACING_CMPL].name = "pacing_complete",
144− [BNXT_RE_PACING_ALERT].name = "pacing_alerts",
145− [BNXT_RE_DB_FIFO_REG].name = "db_fifo_register",
119+ [BNXT_RE_OOB].name = "out_of_buffer",
120+ [BNXT_RE_TX_CNP].name = "np_cnp_pkts",
121+ [BNXT_RE_RX_CNP].name = "rp_cnp_handled",
122+ [BNXT_RE_RX_ECN].name = "np_ecn_marked_roce_packets",
123+ [BNXT_RE_REQ_CQE_ERROR].name = "req_cqe_error",
124+ [BNXT_RE_RESP_CQE_ERROR].name = "resp_cqe_error",
125+ [BNXT_RE_RESP_REMOTE_ACCESS_ERRS].name = "resp_remote_access_errors",
146126 };
147127
148128 static void bnxt_re_copy_ext_stats(struct bnxt_re_dev *rdev,
@@ -273,18 +253,20 @@ static void bnxt_re_copy_err_stats(struct bnxt_re_dev *rdev, · 124 unmodified lines
273253 err_s->res_rx_pci_err;
274254 stats->value[BNXT_RE_OUT_OF_SEQ_ERR] =
275255 err_s->res_oos_drop_count;
276−}
277−
278−static void bnxt_re_copy_db_pacing_stats(struct bnxt_re_dev *rdev,
279− struct rdma_hw_stats *stats)
280−{
281− struct bnxt_re_db_pacing_stats *pacing_s = &rdev->stats.pacing;
282−
283− stats->value[BNXT_RE_PACING_RESCHED] = pacing_s->resched;
284− stats->value[BNXT_RE_PACING_CMPL] = pacing_s->complete;
285− stats->value[BNXT_RE_PACING_ALERT] = pacing_s->alerts;
286− stats->value[BNXT_RE_DB_FIFO_REG] =
287− readl(rdev->en_dev->bar0 + rdev->pacing.dbr_db_fifo_reg_off);
256+ stats->value[BNXT_RE_REQ_CQE_ERROR] =
257+ err_s->bad_resp_err +
258+ err_s->local_qp_op_err +
259+ err_s->local_protection_err +
260+ err_s->mem_mgmt_op_err +
261+ err_s->remote_invalid_req_err +
262+ err_s->remote_access_err +
263+ err_s->remote_op_err;
264+ stats->value[BNXT_RE_RESP_CQE_ERROR] =
265+ err_s->res_cmp_err +
266+ err_s->res_cq_load_err;
267+ stats->value[BNXT_RE_RESP_REMOTE_ACCESS_ERRS] =
268+ err_s->res_rx_no_perm +
269+ err_s->res_tx_no_perm;
288270 }
289271
290272 int bnxt_re_assign_pma_port_ext_counters(struct bnxt_re_dev *rdev, struct ib_mad *out_mad)
@@ -382,7 +364,6 @@ int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev, · 91 unmodified lines
382364 u32 port, int index)
383365 {
384366 struct bnxt_re_dev *rdev = to_bnxt_re_dev(ibdev, ibdev);
385− struct bnxt_re_res_cntrs *res_s = &rdev->stats.res;
386367 struct bnxt_qplib_roce_stats *err_s = NULL;
387368 struct ctx_hw_stats *hw_stats = NULL;
388369 int rc = 0;
@@ -391,26 +372,6 @@ int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev, · 2 unmodified lines
391372 if (!port || !stats)
392373 return -EINVAL;
393374
394− stats->value[BNXT_RE_ACTIVE_QP] = atomic_read(&res_s->qp_count);
395− stats->value[BNXT_RE_ACTIVE_RC_QP] = atomic_read(&res_s->rc_qp_count);
396− stats->value[BNXT_RE_ACTIVE_UD_QP] = atomic_read(&res_s->ud_qp_count);
397− stats->value[BNXT_RE_ACTIVE_SRQ] = atomic_read(&res_s->srq_count);
398− stats->value[BNXT_RE_ACTIVE_CQ] = atomic_read(&res_s->cq_count);
399− stats->value[BNXT_RE_ACTIVE_MR] = atomic_read(&res_s->mr_count);
400− stats->value[BNXT_RE_ACTIVE_MW] = atomic_read(&res_s->mw_count);
401− stats->value[BNXT_RE_ACTIVE_PD] = atomic_read(&res_s->pd_count);
402− stats->value[BNXT_RE_ACTIVE_AH] = atomic_read(&res_s->ah_count);
403− stats->value[BNXT_RE_WATERMARK_QP] = res_s->qp_watermark;
404− stats->value[BNXT_RE_WATERMARK_RC_QP] = res_s->rc_qp_watermark;
405− stats->value[BNXT_RE_WATERMARK_UD_QP] = res_s->ud_qp_watermark;
406− stats->value[BNXT_RE_WATERMARK_SRQ] = res_s->srq_watermark;
407− stats->value[BNXT_RE_WATERMARK_CQ] = res_s->cq_watermark;
408− stats->value[BNXT_RE_WATERMARK_MR] = res_s->mr_watermark;
409− stats->value[BNXT_RE_WATERMARK_MW] = res_s->mw_watermark;
410− stats->value[BNXT_RE_WATERMARK_PD] = res_s->pd_watermark;
411− stats->value[BNXT_RE_WATERMARK_AH] = res_s->ah_watermark;
412− stats->value[BNXT_RE_RESIZE_CQ_CNT] = atomic_read(&res_s->resize_count);
413−
414375 if (hw_stats) {
415376 stats->value[BNXT_RE_RECOVERABLE_ERRORS] =
416377 le64_to_cpu(hw_stats->tx_bcast_pkts);
@@ -449,8 +410,6 @@ int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev, · 32 unmodified lines
449410 goto done;
450411 }
451412 }
452− if (rdev->pacing.dbr_pacing && bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx))
453− bnxt_re_copy_db_pacing_stats(rdev, stats);
454413 }
455414
456415 done:
drivers/infiniband/hw/bnxt_re/hw_counters.h+3−23
@@ -41,25 +41,6 @@ · 40 unmodified lines
4141 #define __BNXT_RE_HW_STATS_H__
4242
4343 enum bnxt_re_hw_stats {
44− BNXT_RE_ACTIVE_PD,
45− BNXT_RE_ACTIVE_AH,
46− BNXT_RE_ACTIVE_QP,
47− BNXT_RE_ACTIVE_RC_QP,
48− BNXT_RE_ACTIVE_UD_QP,
49− BNXT_RE_ACTIVE_SRQ,
50− BNXT_RE_ACTIVE_CQ,
51− BNXT_RE_ACTIVE_MR,
52− BNXT_RE_ACTIVE_MW,
53− BNXT_RE_WATERMARK_PD,
54− BNXT_RE_WATERMARK_AH,
55− BNXT_RE_WATERMARK_QP,
56− BNXT_RE_WATERMARK_RC_QP,
57− BNXT_RE_WATERMARK_UD_QP,
58− BNXT_RE_WATERMARK_SRQ,
59− BNXT_RE_WATERMARK_CQ,
60− BNXT_RE_WATERMARK_MR,
61− BNXT_RE_WATERMARK_MW,
62− BNXT_RE_RESIZE_CQ_CNT,
6344 BNXT_RE_RX_PKTS,
6445 BNXT_RE_RX_BYTES,
6546 BNXT_RE_TX_PKTS,
@@ -129,10 +110,9 @@ enum bnxt_re_hw_stats { · 63 unmodified lines
129110 BNXT_RE_TX_CNP,
130111 BNXT_RE_RX_CNP,
131112 BNXT_RE_RX_ECN,
132− BNXT_RE_PACING_RESCHED,
133− BNXT_RE_PACING_CMPL,
134− BNXT_RE_PACING_ALERT,
135− BNXT_RE_DB_FIFO_REG,
113+ BNXT_RE_REQ_CQE_ERROR,
114+ BNXT_RE_RESP_CQE_ERROR,
115+ BNXT_RE_RESP_REMOTE_ACCESS_ERRS,
136116 BNXT_RE_NUM_EXT_COUNTERS
137117 };
138118
drivers/infiniband/hw/bnxt_re/ib_verbs.c+149−7
@@ -288,7 +288,9 @@ int bnxt_re_query_port(struct ib_device *ibdev, u32 port_num, · 287 unmodified lines
288288 }
289289 port_attr->max_mtu = IB_MTU_4096;
290290 port_attr->active_mtu = iboe_get_mtu(rdev->netdev->mtu);
291− port_attr->gid_tbl_len = dev_attr->max_sgid;
291+ /* One GID is reserved for RawEth QP. Report one less */
292+ port_attr->gid_tbl_len = (rdev->rcfw.roce_mirror ? (dev_attr->max_sgid - 1) :
293+ dev_attr->max_sgid);
292294 port_attr->port_cap_flags = IB_PORT_CM_SUP | IB_PORT_REINIT_SUP |
293295 IB_PORT_DEVICE_MGMT_SUP |
294296 IB_PORT_VENDOR_CLASS_SUP;
@@ -375,7 +377,7 @@ int bnxt_re_del_gid(const struct ib_gid_attr *attr, void **context) · 80 unmodified lines
375377 if (!ctx)
376378 return -EINVAL;
377379
378− if (sgid_tbl && sgid_tbl->active) {
380+ if (sgid_tbl->active) {
379381 if (ctx->idx >= sgid_tbl->max)
380382 return -EINVAL;
381383 gid_to_del = &sgid_tbl->tbl[ctx->idx].gid;
@@ -429,7 +431,7 @@ int bnxt_re_add_gid(const struct ib_gid_attr *attr, void **context) · 47 unmodified lines
429431
430432 rc = bnxt_qplib_add_sgid(sgid_tbl, (struct bnxt_qplib_gid *)&attr->gid,
431433 rdev->qplib_res.netdev->dev_addr,
432− vlan_id, true, &tbl_idx);
434+ vlan_id, true, &tbl_idx, false, 0);
433435 if (rc == -EALREADY) {
434436 ctx_tbl = sgid_tbl->ctx;
435437 ctx_tbl[tbl_idx]->refcnt++;
@@ -955,6 +957,20 @@ fail: · 519 unmodified lines
955957 return rc;
956958 }
957959
960+static void bnxt_re_del_unique_gid(struct bnxt_re_dev *rdev)
961+{
962+ int rc;
963+
964+ if (!rdev->rcfw.roce_mirror)
965+ return;
966+
967+ rc = bnxt_qplib_del_sgid(&rdev->qplib_res.sgid_tbl,
968+ (struct bnxt_qplib_gid *)&rdev->ugid,
969+ 0xFFFF, true);
970+ if (rc)
971+ dev_err(rdev_to_dev(rdev), "Failed to delete unique GID, rc: %d\n", rc);
972+}
973+
958974 /* Queue Pairs */
959975 int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata)
960976 {
@@ -994,6 +1010,9 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata) · 33 unmodified lines
9941010 else if (qp->qplib_qp.type == CMDQ_CREATE_QP_TYPE_UD)
9951011 atomic_dec(&rdev->stats.res.ud_qp_count);
9961012
1013+ if (qp->qplib_qp.type == CMDQ_CREATE_QP_TYPE_RAW_ETHERTYPE)
1014+ bnxt_re_del_unique_gid(rdev);
1015+
9971016 ib_umem_release(qp->rumem);
9981017 ib_umem_release(qp->sumem);
9991018
@@ -1018,6 +1037,8 @@ static u8 __from_ib_qp_type(enum ib_qp_type type) · 18 unmodified lines
10181037 return CMDQ_CREATE_QP_TYPE_RC;
10191038 case IB_QPT_UD:
10201039 return CMDQ_CREATE_QP_TYPE_UD;
1040+ case IB_QPT_RAW_PACKET:
1041+ return CMDQ_CREATE_QP_TYPE_RAW_ETHERTYPE;
10211042 default:
10221043 return IB_QPT_MAX;
10231044 }
@@ -1595,6 +1616,29 @@ static bool bnxt_re_test_qp_limits(struct bnxt_re_dev *rdev, · 571 unmodified lines
15951616 return rc;
15961617 }
15971618
1619+static int bnxt_re_add_unique_gid(struct bnxt_re_dev *rdev)
1620+{
1621+ struct bnxt_qplib_ctx *hctx = &rdev->qplib_ctx;
1622+ struct bnxt_qplib_res *res = &rdev->qplib_res;
1623+ int rc;
1624+
1625+ if (!rdev->rcfw.roce_mirror)
1626+ return 0;
1627+
1628+ rdev->ugid.global.subnet_prefix = cpu_to_be64(0xfe8000000000abcdLL);
1629+ addrconf_ifid_eui48(&rdev->ugid.raw[8], rdev->netdev);
1630+
1631+ rc = bnxt_qplib_add_sgid(&res->sgid_tbl,
1632+ (struct bnxt_qplib_gid *)&rdev->ugid,
1633+ rdev->qplib_res.netdev->dev_addr,
1634+ 0xFFFF, true, &rdev->ugid_index, true,
1635+ hctx->stats3.fw_id);
1636+ if (rc)
1637+ dev_err(rdev_to_dev(rdev), "Failed to add unique GID. rc = %d\n", rc);
1638+
1639+ return rc;
1640+}
1641+
15981642 int bnxt_re_create_qp(struct ib_qp *ib_qp, struct ib_qp_init_attr *qp_init_attr,
15991643 struct ib_udata *udata)
16001644 {
@@ -1656,6 +1700,17 @@ int bnxt_re_create_qp(struct ib_qp *ib_qp, struct ib_qp_init_attr *qp_init_attr, · 55 unmodified lines
16561700 }
16571701 }
16581702
1703+ /* Support for RawEth QP is added to capture TCP pkt dump.
1704+ * So unique SGID is used to avoid incorrect statistics on per
1705+ * function stats_ctx
1706+ */
1707+ if (qp->qplib_qp.type == CMDQ_CREATE_QP_TYPE_RAW_ETHERTYPE) {
1708+ rc = bnxt_re_add_unique_gid(rdev);
1709+ if (rc)
1710+ goto qp_destroy;
1711+ qp->qplib_qp.ugid_index = rdev->ugid_index;
1712+ }
1713+
16591714 qp->ib_qp.qp_num = qp->qplib_qp.id;
16601715 if (qp_init_attr->qp_type == IB_QPT_GSI)
16611716 rdev->gsi_ctx.gsi_qp = qp;
@@ -2301,7 +2356,7 @@ int bnxt_re_query_qp(struct ib_qp *ib_qp, struct ib_qp_attr *qp_attr, · 639 unmodified lines
23012356 qp_attr->pkey_index = qplib_qp->pkey_index;
23022357 qp_attr->qkey = qplib_qp->qkey;
23032358 qp_attr->ah_attr.type = RDMA_AH_ATTR_TYPE_ROCE;
2304− rdma_ah_set_grh(&qp_attr->ah_attr, NULL, qplib_qp->ah.flow_label,
2359+ rdma_ah_set_grh(&qp_attr->ah_attr, NULL, qplib_qp->udp_sport,
23052360 qplib_qp->ah.host_sgid_index,
23062361 qplib_qp->ah.hop_limit,
23072362 qplib_qp->ah.traffic_class);
@@ -3248,9 +3303,9 @@ int bnxt_re_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata) · 940 unmodified lines
32483303 IB_ACCESS_LOCAL_WRITE);
32493304 if (IS_ERR(cq->resize_umem)) {
32503305 rc = PTR_ERR(cq->resize_umem);
3306+ ibdev_err(&rdev->ibdev, "%s: ib_umem_get failed! rc = %pe\n",
3307+ __func__, cq->resize_umem);
32513308 cq->resize_umem = NULL;
3252− ibdev_err(&rdev->ibdev, "%s: ib_umem_get failed! rc = %d\n",
3253− __func__, rc);
32543309 goto fail;
32553310 }
32563311 cq->resize_cqe = entries;
@@ -4392,6 +4447,93 @@ void bnxt_re_dealloc_ucontext(struct ib_ucontext *ib_uctx) · 1135 unmodified lines
43924447 }
43934448 }
43944449
4450+static int bnxt_re_setup_vnic(struct bnxt_re_dev *rdev, struct bnxt_re_qp *qp)
4451+{
4452+ int rc;
4453+
4454+ rc = bnxt_re_hwrm_alloc_vnic(rdev);
4455+ if (rc)
4456+ return rc;
4457+
4458+ rc = bnxt_re_hwrm_cfg_vnic(rdev, qp->qplib_qp.id);
4459+ if (rc)
4460+ goto out_free_vnic;
4461+
4462+ return 0;
4463+out_free_vnic:
4464+ bnxt_re_hwrm_free_vnic(rdev);
4465+ return rc;
4466+}
4467+
4468+struct ib_flow *bnxt_re_create_flow(struct ib_qp *ib_qp,
4469+ struct ib_flow_attr *attr,
4470+ struct ib_udata *udata)
4471+{
4472+ struct bnxt_re_qp *qp = container_of(ib_qp, struct bnxt_re_qp, ib_qp);
4473+ struct bnxt_re_dev *rdev = qp->rdev;
4474+ struct bnxt_re_flow *flow;
4475+ int rc;
4476+
4477+ if (attr->type != IB_FLOW_ATTR_SNIFFER ||
4478+ !rdev->rcfw.roce_mirror)
4479+ return ERR_PTR(-EOPNOTSUPP);
4480+
4481+ mutex_lock(&rdev->qp_lock);
4482+ if (rdev->sniffer_flow_created) {
4483+ ibdev_err(&rdev->ibdev, "RoCE Mirroring is already Configured\n");
4484+ mutex_unlock(&rdev->qp_lock);
4485+ return ERR_PTR(-EBUSY);
4486+ }
4487+
4488+ flow = kzalloc(sizeof(*flow), GFP_KERNEL);
4489+ if (!flow) {
4490+ mutex_unlock(&rdev->qp_lock);
4491+ return ERR_PTR(-ENOMEM);
4492+ }
4493+
4494+ flow->rdev = rdev;
4495+
4496+ rc = bnxt_re_setup_vnic(rdev, qp);
4497+ if (rc)
4498+ goto out_free_flow;
4499+
4500+ rc = bnxt_qplib_create_flow(&rdev->qplib_res);
4501+ if (rc)
4502+ goto out_free_vnic;
4503+
4504+ rdev->sniffer_flow_created = 1;
4505+ mutex_unlock(&rdev->qp_lock);
4506+
4507+ return &flow->ib_flow;
4508+
4509+out_free_vnic:
4510+ bnxt_re_hwrm_free_vnic(rdev);
4511+out_free_flow:
4512+ mutex_unlock(&rdev->qp_lock);
4513+ kfree(flow);
4514+ return ERR_PTR(rc);
4515+}
4516+
4517+int bnxt_re_destroy_flow(struct ib_flow *flow_id)
4518+{
4519+ struct bnxt_re_flow *flow =
4520+ container_of(flow_id, struct bnxt_re_flow, ib_flow);
4521+ struct bnxt_re_dev *rdev = flow->rdev;
4522+ int rc;
4523+
4524+ mutex_lock(&rdev->qp_lock);
4525+ rc = bnxt_qplib_destroy_flow(&rdev->qplib_res);
4526+ if (rc)
4527+ ibdev_dbg(&rdev->ibdev, "failed to destroy_flow rc = %d\n", rc);
4528+ rdev->sniffer_flow_created = 0;
4529+
4530+ bnxt_re_hwrm_free_vnic(rdev);
4531+ mutex_unlock(&rdev->qp_lock);
4532+ kfree(flow);
4533+
4534+ return rc;
4535+}
4536+
43954537 static struct bnxt_re_cq *bnxt_re_search_for_cq(struct bnxt_re_dev *rdev, u32 cq_id)
43964538 {
43974539 struct bnxt_re_cq *cq = NULL, *tmp_cq;
@@ -4604,7 +4746,7 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_ALLOC_PAGE)(struct uverbs_attr_bundle * · 206 unmodified lines
46044746 return err;
46054747
46064748 err = uverbs_copy_to(attrs, BNXT_RE_ALLOC_PAGE_DPI,
4607− &dpi, sizeof(length));
4749+ &dpi, sizeof(dpi));
46084750 if (err)
46094751 return err;
46104752
drivers/infiniband/hw/bnxt_re/ib_verbs.h+10−0
@@ -164,6 +164,11 @@ struct bnxt_re_user_mmap_entry { · 163 unmodified lines
164164 u8 mmap_flag;
165165 };
166166
167+struct bnxt_re_flow {
168+ struct ib_flow ib_flow;
169+ struct bnxt_re_dev *rdev;
170+};
171+
167172 static inline u16 bnxt_re_get_swqe_size(int nsge)
168173 {
169174 return sizeof(struct sq_send_hdr) + nsge * sizeof(struct sq_sge);
@@ -267,6 +272,11 @@ struct ib_mr *bnxt_re_reg_user_mr_dmabuf(struct ib_pd *ib_pd, u64 start, · 97 unmodified lines
267272 struct uverbs_attr_bundle *attrs);
268273 int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata);
269274 void bnxt_re_dealloc_ucontext(struct ib_ucontext *context);
275+struct ib_flow *bnxt_re_create_flow(struct ib_qp *ib_qp,
276+ struct ib_flow_attr *attr,
277+ struct ib_udata *udata);
278+int bnxt_re_destroy_flow(struct ib_flow *flow_id);
279+
270280 int bnxt_re_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
271281 void bnxt_re_mmap_free(struct rdma_user_mmap_entry *rdma_entry);
272282
drivers/infiniband/hw/bnxt_re/main.c+230−148
@@ -80,6 +80,7 @@ MODULE_LICENSE("Dual BSD/GPL"); · 79 unmodified lines
8080 static DEFINE_MUTEX(bnxt_re_mutex);
8181
8282 static int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev);
83+static int bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev);
8384
8485 static int bnxt_re_hwrm_qcfg(struct bnxt_re_dev *rdev, u32 *db_len,
8586 u32 *offset);
@@ -188,6 +189,10 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev) · 102 unmodified lines
188189 rdev->qplib_res.is_vf = BNXT_EN_VF(en_dev);
189190 rdev->qplib_res.en_dev = en_dev;
190191
192+ rc = bnxt_re_query_hwrm_intf_version(rdev);
193+ if (rc)
194+ goto free_dev_attr;
195+
191196 bnxt_re_set_drv_mode(rdev);
192197
193198 bnxt_re_set_db_offset(rdev);
@@ -540,6 +545,72 @@ static void bnxt_re_fill_fw_msg(struct bnxt_fw_msg *fw_msg, void *msg, · 346 unmodified lines
540545 fw_msg->timeout = timeout;
541546 }
542547
548+void bnxt_re_hwrm_free_vnic(struct bnxt_re_dev *rdev)
549+{
550+ struct bnxt_en_dev *en_dev = rdev->en_dev;
551+ struct hwrm_vnic_free_input req = {};
552+ struct bnxt_fw_msg fw_msg = {};
553+ int rc;
554+
555+ bnxt_re_init_hwrm_hdr((void *)&req, HWRM_VNIC_FREE);
556+
557+ req.vnic_id = cpu_to_le32(rdev->mirror_vnic_id);
558+ bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), NULL,
559+ 0, BNXT_RE_HWRM_CMD_TIMEOUT(rdev));
560+ rc = bnxt_send_msg(en_dev, &fw_msg);
561+ if (rc)
562+ ibdev_dbg(&rdev->ibdev,
563+ "Failed to free vnic, rc = %d\n", rc);
564+}
565+
566+int bnxt_re_hwrm_alloc_vnic(struct bnxt_re_dev *rdev)
567+{
568+ struct bnxt_en_dev *en_dev = rdev->en_dev;
569+ struct hwrm_vnic_alloc_output resp = {};
570+ struct hwrm_vnic_alloc_input req = {};
571+ struct bnxt_fw_msg fw_msg = {};
572+ int rc;
573+
574+ bnxt_re_init_hwrm_hdr((void *)&req, HWRM_VNIC_ALLOC);
575+
576+ req.vnic_id = cpu_to_le16(rdev->mirror_vnic_id);
577+ req.flags = cpu_to_le32(VNIC_ALLOC_REQ_FLAGS_VNIC_ID_VALID);
578+ bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp,
579+ sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev));
580+ rc = bnxt_send_msg(en_dev, &fw_msg);
581+ if (rc)
582+ ibdev_dbg(&rdev->ibdev,
583+ "Failed to alloc vnic, rc = %d\n", rc);
584+
585+ return rc;
586+}
587+
588+int bnxt_re_hwrm_cfg_vnic(struct bnxt_re_dev *rdev, u32 qp_id)
589+{
590+ struct bnxt_en_dev *en_dev = rdev->en_dev;
591+ struct hwrm_vnic_cfg_input req = {};
592+ struct bnxt_fw_msg fw_msg = {};
593+ int rc;
594+
595+ bnxt_re_init_hwrm_hdr((void *)&req, HWRM_VNIC_CFG);
596+
597+ req.flags = cpu_to_le32(VNIC_CFG_REQ_FLAGS_ROCE_ONLY_VNIC_MODE);
598+ req.enables = cpu_to_le32(VNIC_CFG_REQ_ENABLES_RAW_QP_ID |
599+ VNIC_CFG_REQ_ENABLES_MRU);
600+ req.vnic_id = cpu_to_le16(rdev->mirror_vnic_id);
601+ req.raw_qp_id = cpu_to_le32(qp_id);
602+ req.mru = cpu_to_le16(rdev->netdev->mtu + VLAN_ETH_HLEN);
603+
604+ bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), NULL,
605+ 0, BNXT_RE_HWRM_CMD_TIMEOUT(rdev));
606+ rc = bnxt_send_msg(en_dev, &fw_msg);
607+ if (rc)
608+ ibdev_dbg(&rdev->ibdev,
609+ "Failed to cfg vnic, rc = %d\n", rc);
610+
611+ return rc;
612+}
613+
543614 /* Query device config using common hwrm */
544615 static int bnxt_re_hwrm_qcfg(struct bnxt_re_dev *rdev, u32 *db_len,
545616 u32 *offset)
@@ -553,11 +624,12 @@ static int bnxt_re_hwrm_qcfg(struct bnxt_re_dev *rdev, u32 *db_len, · 7 unmodified lines
553624 bnxt_re_init_hwrm_hdr((void *)&req, HWRM_FUNC_QCFG);
554625 req.fid = cpu_to_le16(0xffff);
555626 bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp,
556− sizeof(resp), DFLT_HWRM_CMD_TIMEOUT);
627+ sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev));
557628 rc = bnxt_send_msg(en_dev, &fw_msg);
558629 if (!rc) {
559630 *db_len = PAGE_ALIGN(le16_to_cpu(resp.l2_doorbell_bar_size_kb) * 1024);
560631 *offset = PAGE_ALIGN(le16_to_cpu(resp.legacy_l2_db_size_kb) * 1024);
632+ rdev->mirror_vnic_id = le16_to_cpu(resp.mirror_vnic_id);
561633 }
562634 return rc;
563635 }
@@ -577,7 +649,7 @@ int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev) · 13 unmodified lines
577649 bnxt_re_init_hwrm_hdr((void *)&req, HWRM_FUNC_QCAPS);
578650 req.fid = cpu_to_le16(0xffff);
579651 bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp,
580− sizeof(resp), DFLT_HWRM_CMD_TIMEOUT);
652+ sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev));
581653
582654 rc = bnxt_send_msg(en_dev, &fw_msg);
583655 if (rc)
@@ -587,6 +659,8 @@ int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev) · 3 unmodified lines
587659 flags_ext2 = le32_to_cpu(resp.flags_ext2);
588660 cctx->modes.dbr_pacing = flags_ext2 & FUNC_QCAPS_RESP_FLAGS_EXT2_DBR_PACING_EXT_SUPPORTED ||
589661 flags_ext2 & FUNC_QCAPS_RESP_FLAGS_EXT2_DBR_PACING_V0_SUPPORTED;
662+ cctx->modes.roce_mirror = !!(le32_to_cpu(resp.flags_ext3) &
663+ FUNC_QCAPS_RESP_FLAGS_EXT3_MIRROR_ON_ROCE_SUPPORTED);
590664 return 0;
591665 }
592666
@@ -603,7 +677,7 @@ static int bnxt_re_hwrm_dbr_pacing_qcfg(struct bnxt_re_dev *rdev) · 10 unmodified lines
603677 cctx = rdev->chip_ctx;
604678 bnxt_re_init_hwrm_hdr((void *)&req, HWRM_FUNC_DBR_PACING_QCFG);
605679 bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp,
606− sizeof(resp), DFLT_HWRM_CMD_TIMEOUT);
680+ sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev));
607681 rc = bnxt_send_msg(en_dev, &fw_msg);
608682 if (rc)
609683 return rc;
@@ -842,20 +916,12 @@ static void bnxt_re_deinitialize_dbr_pacing(struct bnxt_re_dev *rdev) · 232 unmodified lines
842916 static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev,
843917 u16 fw_ring_id, int type)
844918 {
845− struct bnxt_en_dev *en_dev;
919+ struct bnxt_en_dev *en_dev = rdev->en_dev;
846920 struct hwrm_ring_free_input req = {};
847921 struct hwrm_ring_free_output resp;
848922 struct bnxt_fw_msg fw_msg = {};
849923 int rc = -EINVAL;
850924
851− if (!rdev)
852− return rc;
853−
854− en_dev = rdev->en_dev;
855−
856− if (!en_dev)
857− return rc;
858−
859925 if (test_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags))
860926 return 0;
861927
@@ -863,7 +929,7 @@ static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, · 1 unmodified line
863929 req.ring_type = type;
864930 req.ring_id = cpu_to_le16(fw_ring_id);
865931 bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp,
866− sizeof(resp), DFLT_HWRM_CMD_TIMEOUT);
932+ sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev));
867933 rc = bnxt_send_msg(en_dev, &fw_msg);
868934 if (rc)
869935 ibdev_err(&rdev->ibdev, "Failed to free HW ring:%d :%#x",
@@ -881,9 +947,6 @@ static int bnxt_re_net_ring_alloc(struct bnxt_re_dev *rdev, · 11 unmodified lines
881947 struct bnxt_fw_msg fw_msg = {};
882948 int rc = -EINVAL;
883949
884− if (!en_dev)
885− return rc;
886−
887950 bnxt_re_init_hwrm_hdr((void *)&req, HWRM_RING_ALLOC);
888951 req.enables = 0;
889952 req.page_tbl_addr = cpu_to_le64(ring_attr->dma_arr[0]);
@@ -899,7 +962,7 @@ static int bnxt_re_net_ring_alloc(struct bnxt_re_dev *rdev, · 9 unmodified lines
899962 req.ring_type = ring_attr->type;
900963 req.int_mode = ring_attr->mode;
901964 bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp,
902− sizeof(resp), DFLT_HWRM_CMD_TIMEOUT);
965+ sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev));
903966 rc = bnxt_send_msg(en_dev, &fw_msg);
904967 if (!rc)
905968 *fw_ring_id = le16_to_cpu(resp.ring_id);
@@ -916,16 +979,13 @@ static int bnxt_re_net_stats_ctx_free(struct bnxt_re_dev *rdev, · 10 unmodified lines
916979 struct bnxt_fw_msg fw_msg = {};
917980 int rc = -EINVAL;
918981
919− if (!en_dev)
920− return rc;
921−
922982 if (test_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags))
923983 return 0;
924984
925985 bnxt_re_init_hwrm_hdr((void *)&req, HWRM_STAT_CTX_FREE);
926986 req.stat_ctx_id = cpu_to_le32(fw_stats_ctx_id);
927987 bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp,
928− sizeof(resp), DFLT_HWRM_CMD_TIMEOUT);
988+ sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev));
929989 rc = bnxt_send_msg(en_dev, &fw_msg);
930990 if (rc)
931991 ibdev_err(&rdev->ibdev, "Failed to free HW stats context %#x",
@@ -935,8 +995,7 @@ static int bnxt_re_net_stats_ctx_free(struct bnxt_re_dev *rdev, · 3 unmodified lines
935995 }
936996
937997 static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev,
938− dma_addr_t dma_map,
939− u32 *fw_stats_ctx_id)
998+ struct bnxt_qplib_stats *stats)
940999 {
9411000 struct bnxt_qplib_chip_ctx *chip_ctx = rdev->chip_ctx;
9421001 struct hwrm_stat_ctx_alloc_output resp = {};
@@ -945,21 +1004,18 @@ static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, · 2 unmodified lines
9451004 struct bnxt_fw_msg fw_msg = {};
9461005 int rc = -EINVAL;
9471006
948− *fw_stats_ctx_id = INVALID_STATS_CTX_ID;
949−
950− if (!en_dev)
951− return rc;
1007+ stats->fw_id = INVALID_STATS_CTX_ID;
9521008
9531009 bnxt_re_init_hwrm_hdr((void *)&req, HWRM_STAT_CTX_ALLOC);
9541010 req.update_period_ms = cpu_to_le32(1000);
955− req.stats_dma_addr = cpu_to_le64(dma_map);
1011+ req.stats_dma_addr = cpu_to_le64(stats->dma_map);
9561012 req.stats_dma_length = cpu_to_le16(chip_ctx->hw_stats_size);
9571013 req.stat_ctx_flags = STAT_CTX_ALLOC_REQ_STAT_CTX_FLAGS_ROCE;
9581014 bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp,
959− sizeof(resp), DFLT_HWRM_CMD_TIMEOUT);
1015+ sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev));
9601016 rc = bnxt_send_msg(en_dev, &fw_msg);
9611017 if (!rc)
962− *fw_stats_ctx_id = le32_to_cpu(resp.stat_ctx_id);
1018+ stats->fw_id = le32_to_cpu(resp.stat_ctx_id);
9631019
9641020 return rc;
9651021 }
@@ -975,7 +1031,7 @@ static ssize_t hw_rev_show(struct device *device, struct device_attribute *attr, · 9 unmodified lines
9751031 struct bnxt_re_dev *rdev =
9761032 rdma_device_to_drv_device(device, struct bnxt_re_dev, ibdev);
9771033
978− return sysfs_emit(buf, "0x%x\n", rdev->en_dev->pdev->vendor);
1034+ return sysfs_emit(buf, "0x%x\n", rdev->en_dev->pdev->revision);
9791035 }
9801036 static DEVICE_ATTR_RO(hw_rev);
9811037
@@ -985,13 +1041,31 @@ static ssize_t hca_type_show(struct device *device, · 3 unmodified lines
9851041 struct bnxt_re_dev *rdev =
9861042 rdma_device_to_drv_device(device, struct bnxt_re_dev, ibdev);
9871043
988− return sysfs_emit(buf, "%s\n", rdev->ibdev.node_desc);
1044+ return sysfs_emit(buf, "0x%x\n", rdev->en_dev->pdev->device);
9891045 }
9901046 static DEVICE_ATTR_RO(hca_type);
9911047
1048+static ssize_t board_id_show(struct device *device, struct device_attribute *attr,
1049+ char *buf)
1050+{
1051+ struct bnxt_re_dev *rdev = rdma_device_to_drv_device(device,
1052+ struct bnxt_re_dev, ibdev);
1053+ char buffer[BNXT_VPD_FLD_LEN] = {};
1054+
1055+ if (!rdev->is_virtfn)
1056+ memcpy(buffer, rdev->board_partno, BNXT_VPD_FLD_LEN - 1);
1057+ else
1058+ scnprintf(buffer, BNXT_VPD_FLD_LEN, "0x%x-VF",
1059+ rdev->en_dev->pdev->device);
1060+
1061+ return sysfs_emit(buf, "%s\n", buffer);
1062+}
1063+static DEVICE_ATTR_RO(board_id);
1064+
9921065 static struct attribute *bnxt_re_attributes[] = {
9931066 &dev_attr_hw_rev.attr,
9941067 &dev_attr_hca_type.attr,
1068+ &dev_attr_board_id.attr,
9951069 NULL
9961070 };
9971071
@@ -1207,6 +1281,8 @@ static int bnxt_re_fill_res_srq_entry(struct sk_buff *msg, struct ib_srq *ib_srq · 209 unmodified lines
12071281 goto err;
12081282 if (rdma_nl_put_driver_u32_hex(msg, "max_sge", srq->qplib_srq.max_sge))
12091283 goto err;
1284+ if (rdma_nl_put_driver_u32_hex(msg, "srq_limit", srq->qplib_srq.threshold))
1285+ goto err;
12101286
12111287 nla_nest_end(msg, table_attr);
12121288 return 0;
@@ -1297,6 +1373,8 @@ static const struct ib_device_ops bnxt_re_dev_ops = { · 84 unmodified lines
12971373 .reg_user_mr_dmabuf = bnxt_re_reg_user_mr_dmabuf,
12981374 .req_notify_cq = bnxt_re_req_notify_cq,
12991375 .resize_cq = bnxt_re_resize_cq,
1376+ .create_flow = bnxt_re_create_flow,
1377+ .destroy_flow = bnxt_re_destroy_flow,
13001378 INIT_RDMA_OBJ_SIZE(ib_ah, bnxt_re_ah, ib_ah),
13011379 INIT_RDMA_OBJ_SIZE(ib_cq, bnxt_re_cq, ib_cq),
13021380 INIT_RDMA_OBJ_SIZE(ib_pd, bnxt_re_pd, ib_pd),
@@ -1323,8 +1401,7 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev) · 20 unmodified lines
13231401
13241402 /* ib device init */
13251403 ibdev->node_type = RDMA_NODE_IB_CA;
1326− strscpy(ibdev->node_desc, BNXT_RE_DESC " HCA",
1327− strlen(BNXT_RE_DESC) + 5);
1404+ strscpy(ibdev->node_desc, BNXT_RE_DESC " HCA");
13281405 ibdev->phys_port_cnt = 1;
13291406
13301407 addrconf_addr_eui48((u8 *)&ibdev->node_guid, rdev->netdev->dev_addr);
@@ -1850,81 +1927,6 @@ static void bnxt_re_dev_stop(struct bnxt_re_dev *rdev) · 519 unmodified lines
18501927 mutex_unlock(&rdev->qp_lock);
18511928 }
18521929
1853−static int bnxt_re_update_gid(struct bnxt_re_dev *rdev)
1854−{
1855− struct bnxt_qplib_sgid_tbl *sgid_tbl = &rdev->qplib_res.sgid_tbl;
1856− struct bnxt_qplib_gid gid;
1857− u16 gid_idx, index;
1858− int rc = 0;
1859−
1860− if (!ib_device_try_get(&rdev->ibdev))
1861− return 0;
1862−
1863− for (index = 0; index < sgid_tbl->active; index++) {
1864− gid_idx = sgid_tbl->hw_id[index];
1865−
1866− if (!memcmp(&sgid_tbl->tbl[index], &bnxt_qplib_gid_zero,
1867− sizeof(bnxt_qplib_gid_zero)))
1868− continue;
1869− /* need to modify the VLAN enable setting of non VLAN GID only
1870− * as setting is done for VLAN GID while adding GID
1871− */
1872− if (sgid_tbl->vlan[index])
1873− continue;
1874−
1875− memcpy(&gid, &sgid_tbl->tbl[index], sizeof(gid));
1876−
1877− rc = bnxt_qplib_update_sgid(sgid_tbl, &gid, gid_idx,
1878− rdev->qplib_res.netdev->dev_addr);
1879− }
1880−
1881− ib_device_put(&rdev->ibdev);
1882− return rc;
1883−}
1884−
1885−static u32 bnxt_re_get_priority_mask(struct bnxt_re_dev *rdev)
1886−{
1887− u32 prio_map = 0, tmp_map = 0;
1888− struct net_device *netdev;
1889− struct dcb_app app = {};
1890−
1891− netdev = rdev->netdev;
1892−
1893− app.selector = IEEE_8021QAZ_APP_SEL_ETHERTYPE;
1894− app.protocol = ETH_P_IBOE;
1895− tmp_map = dcb_ieee_getapp_mask(netdev, &app);
1896− prio_map = tmp_map;
1897−
1898− app.selector = IEEE_8021QAZ_APP_SEL_DGRAM;
1899− app.protocol = ROCE_V2_UDP_DPORT;
1900− tmp_map = dcb_ieee_getapp_mask(netdev, &app);
1901− prio_map |= tmp_map;
1902−
1903− return prio_map;
1904−}
1905−
1906−static int bnxt_re_setup_qos(struct bnxt_re_dev *rdev)
1907−{
1908− u8 prio_map = 0;
1909−
1910− /* Get priority for roce */
1911− prio_map = bnxt_re_get_priority_mask(rdev);
1912−
1913− if (prio_map == rdev->cur_prio_map)
1914− return 0;
1915− rdev->cur_prio_map = prio_map;
1916− /* Actual priorities are not programmed as they are already
1917− * done by L2 driver; just enable or disable priority vlan tagging
1918− */
1919− if ((prio_map == 0 && rdev->qplib_res.prio) ||
1920− (prio_map != 0 && !rdev->qplib_res.prio)) {
1921− rdev->qplib_res.prio = prio_map;
1922− bnxt_re_update_gid(rdev);
1923− }
1924−
1925− return 0;
1926−}
1927−
19281930 static void bnxt_re_net_unregister_async_event(struct bnxt_re_dev *rdev)
19291931 {
19301932 if (rdev->is_virtfn)
@@ -1945,7 +1947,31 @@ static void bnxt_re_net_register_async_event(struct bnxt_re_dev *rdev) · 14 unmodified lines
19451947 ASYNC_EVENT_CMPL_EVENT_ID_DCB_CONFIG_CHANGE);
19461948 }
19471949
1948−static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev)
1950+static void bnxt_re_read_vpd_info(struct bnxt_re_dev *rdev)
1951+{
1952+ struct pci_dev *pdev = rdev->en_dev->pdev;
1953+ unsigned int vpd_size, kw_len;
1954+ int pos, size;
1955+ u8 *vpd_data;
1956+
1957+ vpd_data = pci_vpd_alloc(pdev, &vpd_size);
1958+ if (IS_ERR(vpd_data)) {
1959+ pci_warn(pdev, "Unable to read VPD, err=%pe\n", vpd_data);
1960+ return;
1961+ }
1962+
1963+ pos = pci_vpd_find_ro_info_keyword(vpd_data, vpd_size,
1964+ PCI_VPD_RO_KEYWORD_PARTNO, &kw_len);
1965+ if (pos < 0)
1966+ goto free;
1967+
1968+ size = min_t(int, kw_len, BNXT_VPD_FLD_LEN - 1);
1969+ memcpy(rdev->board_partno, &vpd_data[pos], size);
1970+free:
1971+ kfree(vpd_data);
1972+}
1973+
1974+static int bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev)
19491975 {
19501976 struct bnxt_en_dev *en_dev = rdev->en_dev;
19511977 struct hwrm_ver_get_output resp = {};
@@ -1964,7 +1990,7 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) · 12 unmodified lines
19641990 if (rc) {
19651991 ibdev_err(&rdev->ibdev, "Failed to query HW version, rc = 0x%x",
19661992 rc);
1967− return;
1993+ return rc;
19681994 }
19691995
19701996 cctx = rdev->chip_ctx;
@@ -1978,6 +2004,8 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) · 7 unmodified lines
19782004
19792005 if (!cctx->hwrm_cmd_max_timeout)
19802006 cctx->hwrm_cmd_max_timeout = RCFW_FW_STALL_MAX_TIMEOUT;
2007+
2008+ return 0;
19812009 }
19822010
19832011 static int bnxt_re_ib_init(struct bnxt_re_dev *rdev)
@@ -2039,6 +2067,72 @@ static void bnxt_re_free_gid_ctx(struct bnxt_re_dev *rdev) · 55 unmodified lines
20392067 }
20402068 }
20412069
2070+static int bnxt_re_get_stats_ctx(struct bnxt_re_dev *rdev)
2071+{
2072+ struct bnxt_qplib_ctx *hctx = &rdev->qplib_ctx;
2073+ struct bnxt_qplib_res *res = &rdev->qplib_res;
2074+ int rc;
2075+
2076+ rc = bnxt_qplib_alloc_stats_ctx(res->pdev, res->cctx, &hctx->stats);
2077+ if (rc)
2078+ return rc;
2079+
2080+ rc = bnxt_re_net_stats_ctx_alloc(rdev, &hctx->stats);
2081+ if (rc)
2082+ goto free_stat_mem;
2083+
2084+ return 0;
2085+free_stat_mem:
2086+ bnxt_qplib_free_stats_ctx(res->pdev, &hctx->stats);
2087+
2088+ return rc;
2089+}
2090+
2091+static int bnxt_re_get_stats3_ctx(struct bnxt_re_dev *rdev)
2092+{
2093+ struct bnxt_qplib_ctx *hctx = &rdev->qplib_ctx;
2094+ struct bnxt_qplib_res *res = &rdev->qplib_res;
2095+ int rc;
2096+
2097+ if (!rdev->rcfw.roce_mirror)
2098+ return 0;
2099+
2100+ rc = bnxt_qplib_alloc_stats_ctx(res->pdev, res->cctx, &hctx->stats3);
2101+ if (rc)
2102+ return rc;
2103+
2104+ rc = bnxt_re_net_stats_ctx_alloc(rdev, &hctx->stats3);
2105+ if (rc)
2106+ goto free_stat_mem;
2107+
2108+ return 0;
2109+free_stat_mem:
2110+ bnxt_qplib_free_stats_ctx(res->pdev, &hctx->stats3);
2111+
2112+ return rc;
2113+}
2114+
2115+static void bnxt_re_put_stats3_ctx(struct bnxt_re_dev *rdev)
2116+{
2117+ struct bnxt_qplib_ctx *hctx = &rdev->qplib_ctx;
2118+ struct bnxt_qplib_res *res = &rdev->qplib_res;
2119+
2120+ if (!rdev->rcfw.roce_mirror)
2121+ return;
2122+
2123+ bnxt_re_net_stats_ctx_free(rdev, hctx->stats3.fw_id);
2124+ bnxt_qplib_free_stats_ctx(res->pdev, &hctx->stats3);
2125+}
2126+
2127+static void bnxt_re_put_stats_ctx(struct bnxt_re_dev *rdev)
2128+{
2129+ struct bnxt_qplib_ctx *hctx = &rdev->qplib_ctx;
2130+ struct bnxt_qplib_res *res = &rdev->qplib_res;
2131+
2132+ bnxt_re_net_stats_ctx_free(rdev, hctx->stats.fw_id);
2133+ bnxt_qplib_free_stats_ctx(res->pdev, &hctx->stats);
2134+}
2135+
20422136 static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type)
20432137 {
20442138 u8 type;
@@ -2049,8 +2143,7 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · 4 unmodified lines
20492143 bnxt_re_net_unregister_async_event(rdev);
20502144 bnxt_re_uninit_dcb_wq(rdev);
20512145
2052− if (test_and_clear_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags))
2053− cancel_delayed_work_sync(&rdev->worker);
2146+ bnxt_re_put_stats3_ctx(rdev);
20542147
20552148 bnxt_re_free_gid_ctx(rdev);
20562149 if (test_and_clear_bit(BNXT_RE_FLAG_RESOURCES_INITIALIZED,
@@ -2064,8 +2157,8 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · 7 unmodified lines
20642157 if (rc)
20652158 ibdev_warn(&rdev->ibdev,
20662159 "Failed to deinitialize RCFW: %#x", rc);
2067− bnxt_re_net_stats_ctx_free(rdev, rdev->qplib_ctx.stats.fw_id);
2068− bnxt_qplib_free_ctx(&rdev->qplib_res, &rdev->qplib_ctx);
2160+ bnxt_re_put_stats_ctx(rdev);
2161+ bnxt_qplib_free_hwctx(&rdev->qplib_res, &rdev->qplib_ctx);
20692162 bnxt_qplib_disable_rcfw_channel(&rdev->rcfw);
20702163 type = bnxt_qplib_get_ring_type(rdev->chip_ctx);
20712164 bnxt_re_net_ring_free(rdev, rdev->rcfw.creq.ring_id, type);
@@ -2085,16 +2178,6 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · 13 unmodified lines
20852178 }
20862179 }
20872180
2088−/* worker thread for polling periodic events. Now used for QoS programming*/
2089−static void bnxt_re_worker(struct work_struct *work)
2090−{
2091− struct bnxt_re_dev *rdev = container_of(work, struct bnxt_re_dev,
2092− worker.work);
2093−
2094− bnxt_re_setup_qos(rdev);
2095− schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000));
2096−}
2097−
20982181 static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type)
20992182 {
21002183 struct bnxt_re_ring_attr rattr = {};
@@ -2109,8 +2192,9 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) · 8 unmodified lines
21092192 rc = bnxt_re_register_netdev(rdev);
21102193 if (rc) {
21112194 ibdev_err(&rdev->ibdev,
2112− "Failed to register with netedev: %#x\n", rc);
2113− return -EINVAL;
2195+ "Failed to register with Ethernet driver, rc %d\n",
2196+ rc);
2197+ return rc;
21142198 }
21152199 }
21162200 set_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags);
@@ -2148,8 +2232,6 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) · 31 unmodified lines
21482232 /* Check whether VF or PF */
21492233 bnxt_re_get_sriov_func_type(rdev);
21502234
2151− bnxt_re_query_hwrm_intf_version(rdev);
2152−
21532235 /* Establish RCFW Communication Channel to initialize the context
21542236 * memory for the function and all child VFs
21552237 */
@@ -2199,18 +2281,20 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) · 43 unmodified lines
21992281 if (rc)
22002282 goto disable_rcfw;
22012283
2284+ bnxt_qplib_query_version(&rdev->rcfw);
22022285 bnxt_re_set_resource_limits(rdev);
22032286
2204− rc = bnxt_qplib_alloc_ctx(&rdev->qplib_res, &rdev->qplib_ctx, 0,
2205− bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx));
2206− if (rc) {
2207− ibdev_err(&rdev->ibdev,
2208− "Failed to allocate QPLIB context: %#x\n", rc);
2209− goto disable_rcfw;
2287+ if (!rdev->is_virtfn &&
2288+ !bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx)) {
2289+ rc = bnxt_qplib_alloc_hwctx(&rdev->qplib_res, &rdev->qplib_ctx);
2290+ if (rc) {
2291+ ibdev_err(&rdev->ibdev,
2292+ "Failed to allocate hw context: %#x\n", rc);
2293+ goto disable_rcfw;
2294+ }
22102295 }
2211− rc = bnxt_re_net_stats_ctx_alloc(rdev,
2212− rdev->qplib_ctx.stats.dma_map,
2213− &rdev->qplib_ctx.stats.fw_id);
2296+
2297+ rc = bnxt_re_get_stats_ctx(rdev);
22142298 if (rc) {
22152299 ibdev_err(&rdev->ibdev,
22162300 "Failed to allocate stats context: %#x\n", rc);
@@ -2249,15 +2333,6 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) · 32 unmodified lines
22492333 if (rc)
22502334 ibdev_warn(&rdev->ibdev, "Failed to query CC defaults\n");
22512335
2252− rc = bnxt_re_setup_qos(rdev);
2253− if (rc)
2254− ibdev_info(&rdev->ibdev,
2255− "RoCE priority not yet configured\n");
2256−
2257− INIT_DELAYED_WORK(&rdev->worker, bnxt_re_worker);
2258− set_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags);
2259− schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000));
2260−
22612336 if (!(rdev->qplib_res.en_dev->flags & BNXT_EN_FLAG_ROCE_VF_RES_MGMT))
22622337 bnxt_re_vf_res_config(rdev);
22632338 }
@@ -2270,11 +2345,18 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) · 6 unmodified lines
22702345 bnxt_re_init_dcb_wq(rdev);
22712346 bnxt_re_net_register_async_event(rdev);
22722347
2348+ if (!rdev->is_virtfn)
2349+ bnxt_re_read_vpd_info(rdev);
2350+
2351+ rc = bnxt_re_get_stats3_ctx(rdev);
2352+ if (rc)
2353+ goto fail;
2354+
22732355 return 0;
22742356 free_sctx:
22752357 bnxt_re_net_stats_ctx_free(rdev, rdev->qplib_ctx.stats.fw_id);
22762358 free_ctx:
2277− bnxt_qplib_free_ctx(&rdev->qplib_res, &rdev->qplib_ctx);
2359+ bnxt_qplib_free_hwctx(&rdev->qplib_res, &rdev->qplib_ctx);
22782360 disable_rcfw:
22792361 bnxt_qplib_disable_rcfw_channel(&rdev->rcfw);
22802362 free_ring:
drivers/infiniband/hw/bnxt_re/qplib_fp.c+10−3
@@ -1307,6 +1307,7 @@ static bool is_optimized_state_transition(struct bnxt_qplib_qp *qp) · 1306 unmodified lines
13071307
13081308 int bnxt_qplib_modify_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp)
13091309 {
1310+ struct bnxt_qplib_sgid_tbl *sgid_tbl = &res->sgid_tbl;
13101311 struct bnxt_qplib_rcfw *rcfw = res->rcfw;
13111312 struct creq_modify_qp_resp resp = {};
13121313 struct bnxt_qplib_cmdqmsg msg = {};
@@ -1358,9 +1359,14 @@ int bnxt_qplib_modify_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp) · 45 unmodified lines
13581359 if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_FLOW_LABEL)
13591360 req.flow_label = cpu_to_le32(qp->ah.flow_label);
13601361
1361− if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_SGID_INDEX)
1362− req.sgid_index = cpu_to_le16(res->sgid_tbl.hw_id
1363− [qp->ah.sgid_index]);
1362+ if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_SGID_INDEX) {
1363+ if (qp->type == CMDQ_CREATE_QP_TYPE_RAW_ETHERTYPE)
1364+ req.sgid_index =
1365+ cpu_to_le16(sgid_tbl->hw_id[qp->ugid_index]);
1366+ else
1367+ req.sgid_index =
1368+ cpu_to_le16(sgid_tbl->hw_id[qp->ah.sgid_index]);
1369+ }
13641370
13651371 if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_HOP_LIMIT)
13661372 req.hop_limit = qp->ah.hop_limit;
@@ -1464,6 +1470,7 @@ int bnxt_qplib_query_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp) · 97 unmodified lines
14641470 qp->access = sb->access;
14651471 qp->pkey_index = le16_to_cpu(sb->pkey);
14661472 qp->qkey = le32_to_cpu(sb->qkey);
1473+ qp->udp_sport = le16_to_cpu(sb->udp_src_port);
14671474
14681475 temp32[0] = le32_to_cpu(sb->dgid[0]);
14691476 temp32[1] = le32_to_cpu(sb->dgid[1]);
drivers/infiniband/hw/bnxt_re/qplib_fp.h+2−0
@@ -299,6 +299,7 @@ struct bnxt_qplib_qp { · 298 unmodified lines
299299 u8 smac[6];
300300 u16 vlan_id;
301301 u16 port_id;
302+ u16 udp_sport;
302303 u8 nw_type;
303304 struct bnxt_qplib_ah ah;
304305
@@ -344,6 +345,7 @@ struct bnxt_qplib_qp { · 39 unmodified lines
344345 u32 msn_tbl_sz;
345346 bool is_host_msn_tbl;
346347 u8 tos_dscp;
348+ u32 ugid_index;
347349 };
348350
349351 #define BNXT_RE_MAX_MSG_SIZE 0x80000000
drivers/infiniband/hw/bnxt_re/qplib_rcfw.c+8−2
@@ -186,7 +186,7 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) · 185 unmodified lines
186186 * wait for command completion. Maximum holding interval is 8 second.
187187 *
188188 * Returns:
189− * -ETIMEOUT if command is not completed in specific time interval.
189+ * -ETIMEDOUT if command is not completed in specific time interval.
190190 * 0 if command is completed by firmware.
191191 */
192192 static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie)
@@ -366,6 +366,7 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, · 173 unmodified lines
366366 wmb();
367367 writel(cmdq_prod, cmdq->cmdq_mbox.prod);
368368 writel(RCFW_CMDQ_TRIG_VAL, cmdq->cmdq_mbox.db);
369+ print_hex_dump_bytes("req: ", DUMP_PREFIX_OFFSET, msg->req, msg->req_sz);
369370 spin_unlock_bh(&hwq->lock);
370371 /* Return the CREQ response pointer */
371372 return 0;
@@ -381,7 +382,7 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, · 9 unmodified lines
381382 * This function can not be called from non-sleepable context.
382383 *
383384 * Returns:
384− * -ETIMEOUT if command is not completed in specific time interval.
385+ * -ETIMEDOUT if command is not completed in specific time interval.
385386 * 0 if command is completed by firmware.
386387 */
387388 static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie)
@@ -631,6 +632,7 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, · 243 unmodified lines
631632 int rc = 0;
632633
633634 pdev = rcfw->pdev;
635+ print_hex_dump_bytes("event: ", DUMP_PREFIX_OFFSET, qp_event, sizeof(*qp_event));
634636 switch (qp_event->event) {
635637 case CREQ_QP_EVENT_EVENT_QP_ERROR_NOTIFICATION:
636638 err_event = (struct creq_qp_error_notification *)qp_event;
@@ -903,6 +905,10 @@ skip_ctx_setup: · 266 unmodified lines
903905 flags |= CMDQ_INITIALIZE_FW_FLAGS_OPTIMIZE_MODIFY_QP_SUPPORTED;
904906 if (rcfw->res->en_dev->flags & BNXT_EN_FLAG_ROCE_VF_RES_MGMT)
905907 flags |= CMDQ_INITIALIZE_FW_FLAGS_L2_VF_RESOURCE_MGMT;
908+ if (bnxt_qplib_roce_mirror_supported(rcfw->res->cctx)) {
909+ flags |= CMDQ_INITIALIZE_FW_FLAGS_MIRROR_ON_ROCE_SUPPORTED;
910+ rcfw->roce_mirror = true;
911+ }
906912 req.flags |= cpu_to_le16(flags);
907913 req.stat_ctx_id = cpu_to_le32(ctx->stats.fw_id);
908914 bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), sizeof(resp), 0);
drivers/infiniband/hw/bnxt_re/qplib_rcfw.h+1−0
@@ -236,6 +236,7 @@ struct bnxt_qplib_rcfw { · 235 unmodified lines
236236 atomic_t timeout_send;
237237 /* cached from chip cctx for quick reference in slow path */
238238 u16 max_timeout;
239+ bool roce_mirror;
239240 };
240241
241242 struct bnxt_qplib_cmdqmsg {
drivers/infiniband/hw/bnxt_re/qplib_res.c+11−27
@@ -53,12 +53,6 @@ · 52 unmodified lines
5353 #include "qplib_sp.h"
5454 #include "qplib_rcfw.h"
5555
56−static void bnxt_qplib_free_stats_ctx(struct pci_dev *pdev,
57− struct bnxt_qplib_stats *stats);
58−static int bnxt_qplib_alloc_stats_ctx(struct pci_dev *pdev,
59− struct bnxt_qplib_chip_ctx *cctx,
60− struct bnxt_qplib_stats *stats);
61−
6256 /* PBL */
6357 static void __free_pbl(struct bnxt_qplib_res *res, struct bnxt_qplib_pbl *pbl,
6458 bool is_umem)
@@ -352,8 +346,8 @@ fail: · 287 unmodified lines
352346 }
353347
354348 /* Context Tables */
355−void bnxt_qplib_free_ctx(struct bnxt_qplib_res *res,
356− struct bnxt_qplib_ctx *ctx)
349+void bnxt_qplib_free_hwctx(struct bnxt_qplib_res *res,
350+ struct bnxt_qplib_ctx *ctx)
357351 {
358352 int i;
359353
@@ -367,7 +361,6 @@ void bnxt_qplib_free_ctx(struct bnxt_qplib_res *res, · 7 unmodified lines
367361 /* restore original pde level before destroy */
368362 ctx->tqm_ctx.pde.level = ctx->tqm_ctx.pde_level;
369363 bnxt_qplib_free_hwq(res, &ctx->tqm_ctx.pde);
370− bnxt_qplib_free_stats_ctx(res->pdev, &ctx->stats);
371364 }
372365
373366 static int bnxt_qplib_alloc_tqm_rings(struct bnxt_qplib_res *res,
@@ -466,7 +459,7 @@ fail: · 92 unmodified lines
466459 }
467460
468461 /*
469− * Routine: bnxt_qplib_alloc_ctx
462+ * Routine: bnxt_qplib_alloc_hwctx
470463 * Description:
471464 * Context tables are memories which are used by the chip fw.
472465 * The 6 tables defined are:
@@ -486,17 +479,13 @@ fail: · 13 unmodified lines
486479 * Returns:
487480 * 0 if success, else -ERRORS
488481 */
489−int bnxt_qplib_alloc_ctx(struct bnxt_qplib_res *res,
490− struct bnxt_qplib_ctx *ctx,
491− bool virt_fn, bool is_p5)
482+int bnxt_qplib_alloc_hwctx(struct bnxt_qplib_res *res,
483+ struct bnxt_qplib_ctx *ctx)
492484 {
493485 struct bnxt_qplib_hwq_attr hwq_attr = {};
494486 struct bnxt_qplib_sg_info sginfo = {};
495487 int rc;
496488
497− if (virt_fn || is_p5)
498− goto stats_alloc;
499−
500489 /* QPC Tables */
501490 sginfo.pgsize = PAGE_SIZE;
502491 sginfo.pgshft = PAGE_SHIFT;
@@ -542,16 +531,11 @@ int bnxt_qplib_alloc_ctx(struct bnxt_qplib_res *res, · 39 unmodified lines
542531 rc = bnxt_qplib_alloc_init_hwq(&ctx->tim_tbl, &hwq_attr);
543532 if (rc)
544533 goto fail;
545−stats_alloc:
546− /* Stats */
547− rc = bnxt_qplib_alloc_stats_ctx(res->pdev, res->cctx, &ctx->stats);
548− if (rc)
549− goto fail;
550534
551535 return 0;
552536
553537 fail:
554− bnxt_qplib_free_ctx(res, ctx);
538+ bnxt_qplib_free_hwctx(res, ctx);
555539 return rc;
556540 }
557541
@@ -832,8 +816,8 @@ static int bnxt_qplib_alloc_dpi_tbl(struct bnxt_qplib_res *res, · 274 unmodified lines
832816 }
833817
834818 /* Stats */
835−static void bnxt_qplib_free_stats_ctx(struct pci_dev *pdev,
836− struct bnxt_qplib_stats *stats)
819+void bnxt_qplib_free_stats_ctx(struct pci_dev *pdev,
820+ struct bnxt_qplib_stats *stats)
837821 {
838822 if (stats->dma) {
839823 dma_free_coherent(&pdev->dev, stats->size,
@@ -843,9 +827,9 @@ static void bnxt_qplib_free_stats_ctx(struct pci_dev *pdev, · 3 unmodified lines
843827 stats->fw_id = -1;
844828 }
845829
846−static int bnxt_qplib_alloc_stats_ctx(struct pci_dev *pdev,
847− struct bnxt_qplib_chip_ctx *cctx,
848− struct bnxt_qplib_stats *stats)
830+int bnxt_qplib_alloc_stats_ctx(struct pci_dev *pdev,
831+ struct bnxt_qplib_chip_ctx *cctx,
832+ struct bnxt_qplib_stats *stats)
849833 {
850834 memset(stats, 0, sizeof(*stats));
851835 stats->fw_id = -1;
drivers/infiniband/hw/bnxt_re/qplib_res.h+16−5
@@ -65,6 +65,7 @@ struct bnxt_qplib_drv_modes { · 64 unmodified lines
6565 bool db_push;
6666 bool dbr_pacing;
6767 u32 toggle_bits;
68+ u8 roce_mirror;
6869 };
6970
7071 enum bnxt_re_toggle_modes {
@@ -303,6 +304,7 @@ struct bnxt_qplib_ctx { · 232 unmodified lines
303304 struct bnxt_qplib_hwq tim_tbl;
304305 struct bnxt_qplib_tqm_ctx tqm_ctx;
305306 struct bnxt_qplib_stats stats;
307+ struct bnxt_qplib_stats stats3;
306308 struct bnxt_qplib_vf_res vf_res;
307309 };
308310
@@ -432,15 +434,19 @@ void bnxt_qplib_cleanup_res(struct bnxt_qplib_res *res); · 123 unmodified lines
432434 int bnxt_qplib_init_res(struct bnxt_qplib_res *res);
433435 void bnxt_qplib_free_res(struct bnxt_qplib_res *res);
434436 int bnxt_qplib_alloc_res(struct bnxt_qplib_res *res, struct net_device *netdev);
435−void bnxt_qplib_free_ctx(struct bnxt_qplib_res *res,
436− struct bnxt_qplib_ctx *ctx);
437−int bnxt_qplib_alloc_ctx(struct bnxt_qplib_res *res,
438− struct bnxt_qplib_ctx *ctx,
439− bool virt_fn, bool is_p5);
437+void bnxt_qplib_free_hwctx(struct bnxt_qplib_res *res,
438+ struct bnxt_qplib_ctx *ctx);
439+int bnxt_qplib_alloc_hwctx(struct bnxt_qplib_res *res,
440+ struct bnxt_qplib_ctx *ctx);
440441 int bnxt_qplib_map_db_bar(struct bnxt_qplib_res *res);
441442 void bnxt_qplib_unmap_db_bar(struct bnxt_qplib_res *res);
442443
443444 int bnxt_qplib_determine_atomics(struct pci_dev *dev);
445+int bnxt_qplib_alloc_stats_ctx(struct pci_dev *pdev,
446+ struct bnxt_qplib_chip_ctx *cctx,
447+ struct bnxt_qplib_stats *stats);
448+void bnxt_qplib_free_stats_ctx(struct pci_dev *pdev,
449+ struct bnxt_qplib_stats *stats);
444450
445451 static inline void bnxt_qplib_hwq_incr_prod(struct bnxt_qplib_db_info *dbinfo,
446452 struct bnxt_qplib_hwq *hwq, u32 cnt)
@@ -582,6 +588,11 @@ static inline u8 bnxt_qplib_dbr_pacing_en(struct bnxt_qplib_chip_ctx *cctx) · 135 unmodified lines
582588 return cctx->modes.dbr_pacing;
583589 }
584590
591+static inline u8 bnxt_qplib_roce_mirror_supported(struct bnxt_qplib_chip_ctx *cctx)
592+{
593+ return cctx->modes.roce_mirror;
594+}
595+
585596 static inline bool _is_alloc_mr_unified(u16 dev_cap_flags)
586597 {
587598 return dev_cap_flags & CREQ_QUERY_FUNC_RESP_SB_MR_REGISTER_ALLOC;
drivers/infiniband/hw/bnxt_re/qplib_sp.c+49−49
@@ -66,14 +66,15 @@ static bool bnxt_qplib_is_atomic_cap(struct bnxt_qplib_rcfw *rcfw) · 65 unmodified lines
6666 return (pcie_ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ);
6767 }
6868
69−static void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw,
70− char *fw_ver)
69+void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw)
7170 {
7271 struct creq_query_version_resp resp = {};
7372 struct bnxt_qplib_cmdqmsg msg = {};
7473 struct cmdq_query_version req = {};
74+ struct bnxt_qplib_dev_attr *attr;
7575 int rc;
7676
77+ attr = rcfw->res->dattr;
7778 bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req,
7879 CMDQ_BASE_OPCODE_QUERY_VERSION,
7980 sizeof(req));
@@ -82,10 +83,10 @@ static void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw, · 2 unmodified lines
8283 rc = bnxt_qplib_rcfw_send_message(rcfw, &msg);
8384 if (rc)
8485 return;
85− fw_ver[0] = resp.fw_maj;
86− fw_ver[1] = resp.fw_minor;
87− fw_ver[2] = resp.fw_bld;
88− fw_ver[3] = resp.fw_rsvd;
86+ attr->fw_ver[0] = resp.fw_maj;
87+ attr->fw_ver[1] = resp.fw_minor;
88+ attr->fw_ver[2] = resp.fw_bld;
89+ attr->fw_ver[3] = resp.fw_rsvd;
8990 }
9091
9192 int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw)
@@ -179,8 +180,6 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw) · 87 unmodified lines
179180 if (_is_max_srq_ext_supported(attr->dev_cap_flags2))
180181 attr->max_srq += le16_to_cpu(sb->max_srq_ext);
181182
182− bnxt_qplib_query_version(rcfw, attr->fw_ver);
183−
184183 for (i = 0; i < MAX_TQM_ALLOC_REQ / 4; i++) {
185184 temp = le32_to_cpu(sb->tqm_alloc_reqs[i]);
186185 tqm_alloc = (u8 *)&temp;
@@ -309,7 +308,8 @@ int bnxt_qplib_del_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, · 122 unmodified lines
309308
310309 int bnxt_qplib_add_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl,
311310 struct bnxt_qplib_gid *gid, const u8 *smac,
312− u16 vlan_id, bool update, u32 *index)
311+ u16 vlan_id, bool update, u32 *index,
312+ bool is_ugid, u32 stats_ctx_id)
313313 {
314314 struct bnxt_qplib_res *res = to_bnxt_qplib(sgid_tbl,
315315 struct bnxt_qplib_res,
@@ -374,6 +374,9 @@ int bnxt_qplib_add_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, · 58 unmodified lines
374374 req.src_mac[1] = cpu_to_be16(((u16 *)smac)[1]);
375375 req.src_mac[2] = cpu_to_be16(((u16 *)smac)[2]);
376376
377+ req.stats_ctx = cpu_to_le16(CMDQ_ADD_GID_STATS_CTX_STATS_CTX_VALID |
378+ (u16)stats_ctx_id);
379+
377380 bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req),
378381 sizeof(resp), 0);
379382 rc = bnxt_qplib_rcfw_send_message(rcfw, &msg);
@@ -397,46 +400,6 @@ int bnxt_qplib_add_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, · 17 unmodified lines
397400 return 0;
398401 }
399402
400−int bnxt_qplib_update_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl,
401− struct bnxt_qplib_gid *gid, u16 gid_idx,
402− const u8 *smac)
403−{
404− struct bnxt_qplib_res *res = to_bnxt_qplib(sgid_tbl,
405− struct bnxt_qplib_res,
406− sgid_tbl);
407− struct bnxt_qplib_rcfw *rcfw = res->rcfw;
408− struct creq_modify_gid_resp resp = {};
409− struct bnxt_qplib_cmdqmsg msg = {};
410− struct cmdq_modify_gid req = {};
411− int rc;
412−
413− bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req,
414− CMDQ_BASE_OPCODE_MODIFY_GID,
415− sizeof(req));
416−
417− req.gid[0] = cpu_to_be32(((u32 *)gid->data)[3]);
418− req.gid[1] = cpu_to_be32(((u32 *)gid->data)[2]);
419− req.gid[2] = cpu_to_be32(((u32 *)gid->data)[1]);
420− req.gid[3] = cpu_to_be32(((u32 *)gid->data)[0]);
421− if (res->prio) {
422− req.vlan |= cpu_to_le16
423− (CMDQ_ADD_GID_VLAN_TPID_TPID_8100 |
424− CMDQ_ADD_GID_VLAN_VLAN_EN);
425− }
426−
427− /* MAC in network format */
428− req.src_mac[0] = cpu_to_be16(((u16 *)smac)[0]);
429− req.src_mac[1] = cpu_to_be16(((u16 *)smac)[1]);
430− req.src_mac[2] = cpu_to_be16(((u16 *)smac)[2]);
431−
432− req.gid_index = cpu_to_le16(gid_idx);
433−
434− bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req),
435− sizeof(resp), 0);
436− rc = bnxt_qplib_rcfw_send_message(rcfw, &msg);
437− return rc;
438−}
439−
440403 /* AH */
441404 int bnxt_qplib_create_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah,
442405 bool block)
@@ -1143,3 +1106,40 @@ out: · 700 unmodified lines
11431106 dma_free_coherent(&rcfw->pdev->dev, sbuf.size, sbuf.sb, sbuf.dma_addr);
11441107 return rc;
11451108 }
1109+
1110+int bnxt_qplib_create_flow(struct bnxt_qplib_res *res)
1111+{
1112+ struct creq_roce_mirror_cfg_resp resp = {};
1113+ struct bnxt_qplib_rcfw *rcfw = res->rcfw;
1114+ struct cmdq_roce_mirror_cfg req = {};
1115+ struct bnxt_qplib_cmdqmsg msg = {};
1116+
1117+ bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req,
1118+ CMDQ_BASE_OPCODE_ROCE_MIRROR_CFG,
1119+ sizeof(req));
1120+
1121+ req.mirror_flags = (u8)CMDQ_ROCE_MIRROR_CFG_MIRROR_ENABLE;
1122+
1123+ bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req),
1124+ sizeof(resp), 0);
1125+ return bnxt_qplib_rcfw_send_message(rcfw, &msg);
1126+}
1127+
1128+int bnxt_qplib_destroy_flow(struct bnxt_qplib_res *res)
1129+{
1130+ struct creq_roce_mirror_cfg_resp resp = {};
1131+ struct bnxt_qplib_rcfw *rcfw = res->rcfw;
1132+ struct cmdq_roce_mirror_cfg req = {};
1133+ struct bnxt_qplib_cmdqmsg msg = {};
1134+
1135+ bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req,
1136+ CMDQ_BASE_OPCODE_ROCE_MIRROR_CFG,
1137+ sizeof(req));
1138+
1139+ req.mirror_flags &= ~((u8)CMDQ_ROCE_MIRROR_CFG_MIRROR_ENABLE);
1140+
1141+ bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req),
1142+ sizeof(resp), 0);
1143+
1144+ return bnxt_qplib_rcfw_send_message(rcfw, &msg);
1145+}
drivers/infiniband/hw/bnxt_re/qplib_sp.h+5−1
@@ -323,7 +323,8 @@ int bnxt_qplib_del_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, · 322 unmodified lines
323323 struct bnxt_qplib_gid *gid, u16 vlan_id, bool update);
324324 int bnxt_qplib_add_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl,
325325 struct bnxt_qplib_gid *gid, const u8 *mac, u16 vlan_id,
326− bool update, u32 *index);
326+ bool update, u32 *index,
327+ bool is_ugid, u32 stats_ctx_id);
327328 int bnxt_qplib_update_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl,
328329 struct bnxt_qplib_gid *gid, u16 gid_idx,
329330 const u8 *smac);
@@ -358,6 +359,9 @@ int bnxt_qplib_read_context(struct bnxt_qplib_rcfw *rcfw, u8 type, u32 xid, · 28 unmodified lines
358359 u32 resp_size, void *resp_va);
359360 int bnxt_qplib_query_cc_param(struct bnxt_qplib_res *res,
360361 struct bnxt_qplib_cc_param *cc_param);
362+void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw);
363+int bnxt_qplib_create_flow(struct bnxt_qplib_res *res);
364+int bnxt_qplib_destroy_flow(struct bnxt_qplib_res *res);
361365
362366 #define BNXT_VAR_MAX_WQE 4352
363367 #define BNXT_VAR_MAX_SLOT_ALIGN 256
drivers/infiniband/hw/bnxt_re/roce_hsi.h+42−2
@@ -144,7 +144,8 @@ struct cmdq_base { · 143 unmodified lines
144144 #define CMDQ_BASE_OPCODE_MODIFY_CQ 0x90UL
145145 #define CMDQ_BASE_OPCODE_QUERY_QP_EXTEND 0x91UL
146146 #define CMDQ_BASE_OPCODE_QUERY_ROCE_STATS_EXT 0x92UL
147− #define CMDQ_BASE_OPCODE_LAST CMDQ_BASE_OPCODE_QUERY_ROCE_STATS_EXT
147+ #define CMDQ_BASE_OPCODE_ROCE_MIRROR_CFG 0x99UL
148+ #define CMDQ_BASE_OPCODE_LAST CMDQ_BASE_OPCODE_ROCE_MIRROR_CFG
148149 u8 cmd_size;
149150 __le16 flags;
150151 __le16 cookie;
@@ -218,6 +219,7 @@ struct cmdq_initialize_fw { · 67 unmodified lines
218219 #define CMDQ_INITIALIZE_FW_FLAGS_HW_REQUESTER_RETX_SUPPORTED 0x2UL
219220 #define CMDQ_INITIALIZE_FW_FLAGS_OPTIMIZE_MODIFY_QP_SUPPORTED 0x8UL
220221 #define CMDQ_INITIALIZE_FW_FLAGS_L2_VF_RESOURCE_MGMT 0x10UL
222+ #define CMDQ_INITIALIZE_FW_FLAGS_MIRROR_ON_ROCE_SUPPORTED 0x80UL
221223 __le16 cookie;
222224 u8 resp_size;
223225 u8 reserved8;
@@ -788,7 +790,8 @@ struct creq_query_qp_resp_sb { · 564 unmodified lines
788790 #define CREQ_QUERY_QP_RESP_SB_ACCESS_REMOTE_ATOMIC 0x8UL
789791 __le16 pkey;
790792 __le32 qkey;
791− __le32 reserved32;
793+ __le16 udp_src_port;
794+ __le16 reserved16;
792795 __le32 dgid[4];
793796 __le32 flow_label;
794797 __le16 sgid_index;
@@ -2108,6 +2111,43 @@ struct creq_query_roce_stats_ext_resp_sb { · 1313 unmodified lines
21082111 __le64 dup_req;
21092112 };
21102113
2114+/* cmdq_roce_mirror_cfg (size:192b/24B) */
2115+struct cmdq_roce_mirror_cfg {
2116+ u8 opcode;
2117+ #define CMDQ_ROCE_MIRROR_CFG_OPCODE_ROCE_MIRROR_CFG 0x99UL
2118+ #define CMDQ_ROCE_MIRROR_CFG_OPCODE_LAST \
2119+ CMDQ_ROCE_MIRROR_CFG_OPCODE_ROCE_MIRROR_CFG
2120+ u8 cmd_size;
2121+ __le16 flags;
2122+ __le16 cookie;
2123+ u8 resp_size;
2124+ u8 reserved8;
2125+ __le64 resp_addr;
2126+ u8 mirror_flags;
2127+ #define CMDQ_ROCE_MIRROR_CFG_MIRROR_ENABLE 0x1UL
2128+ u8 rsvd[7];
2129+};
2130+
2131+/* creq_roce_mirror_cfg_resp (size:128b/16B) */
2132+struct creq_roce_mirror_cfg_resp {
2133+ u8 type;
2134+ #define CREQ_ROCE_MIRROR_CFG_RESP_TYPE_MASK 0x3fUL
2135+ #define CREQ_ROCE_MIRROR_CFG_RESP_TYPE_SFT 0
2136+ #define CREQ_ROCE_MIRROR_CFG_RESP_TYPE_QP_EVENT 0x38UL
2137+ #define CREQ_ROCE_MIRROR_CFG_RESP_TYPE_LAST \
2138+ CREQ_ROCE_MIRROR_CFG_RESP_TYPE_QP_EVENT
2139+ u8 status;
2140+ __le16 cookie;
2141+ __le32 reserved32;
2142+ u8 v;
2143+ #define CREQ_ROCE_MIRROR_CFG_RESP_V 0x1UL
2144+ u8 event;
2145+ #define CREQ_ROCE_MIRROR_CFG_RESP_EVENT_ROCE_MIRROR_CFG 0x99UL
2146+ #define CREQ_ROCE_MIRROR_CFG_RESP_EVENT_LAST \
2147+ CREQ_ROCE_MIRROR_CFG_RESP_EVENT_ROCE_MIRROR_CFG
2148+ u8 reserved48[6];
2149+};
2150+
21112151 /* cmdq_query_func (size:128b/16B) */
21122152 struct cmdq_query_func {
21132153 u8 opcode;
drivers/infiniband/hw/cxgb4/device.c+2−3
@@ -1228,9 +1228,8 @@ static int c4iw_uld_state_change(void *handle, enum cxgb4_state new_state) · 1227 unmodified lines
12281228 if (!ctx->dev) {
12291229 ctx->dev = c4iw_alloc(&ctx->lldi);
12301230 if (IS_ERR(ctx->dev)) {
1231− pr_err("%s: initialization failed: %ld\n",
1232− pci_name(ctx->lldi.pdev),
1233− PTR_ERR(ctx->dev));
1231+ pr_err("%s: initialization failed: %pe\n",
1232+ pci_name(ctx->lldi.pdev), ctx->dev);
12341233 ctx->dev = NULL;
12351234 break;
12361235 }
drivers/infiniband/hw/efa/efa_com.c+11−7
@@ -1,6 +1,6 @@
11 // SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
22 /*
3− * Copyright 2018-2024 Amazon.com, Inc. or its affiliates. All rights reserved.
3+ * Copyright 2018-2025 Amazon.com, Inc. or its affiliates. All rights reserved.
44 */
55
66 #include "efa_com.h"
@@ -30,6 +30,7 @@ struct efa_comp_ctx { · 23 unmodified lines
3030 struct efa_admin_acq_entry *user_cqe;
3131 u32 comp_size;
3232 enum efa_cmd_status status;
33+ u16 cmd_id;
3334 u8 cmd_opcode;
3435 u8 occupied;
3536 };
@@ -333,6 +334,7 @@ static struct efa_comp_ctx *__efa_com_submit_admin_cmd(struct efa_com_admin_queu · 297 unmodified lines
333334 comp_ctx->comp_size = comp_size_in_bytes;
334335 comp_ctx->user_cqe = comp;
335336 comp_ctx->cmd_opcode = cmd->aq_common_descriptor.opcode;
337+ comp_ctx->cmd_id = cmd_id;
336338
337339 reinit_completion(&comp_ctx->wait_event);
338340
@@ -557,17 +559,19 @@ static int efa_com_wait_and_process_admin_cq_interrupts(struct efa_comp_ctx *com · 218 unmodified lines
557559 if (comp_ctx->status == EFA_CMD_COMPLETED)
558560 ibdev_err_ratelimited(
559561 aq->efa_dev,
560− "The device sent a completion but the driver didn't receive any MSI-X interrupt for admin cmd %s(%d) status %d (ctx: 0x%p, sq producer: %d, sq consumer: %d, cq consumer: %d)\n",
562+ "The device sent a completion but the driver didn't receive any MSI-X interrupt for admin cmd %s(%d) status %d (id: %d, sq producer: %d, sq consumer: %d, cq consumer: %d)\n",
561563 efa_com_cmd_str(comp_ctx->cmd_opcode),
562564 comp_ctx->cmd_opcode, comp_ctx->status,
563− comp_ctx, aq->sq.pc, aq->sq.cc, aq->cq.cc);
565+ comp_ctx->cmd_id, aq->sq.pc, aq->sq.cc,
566+ aq->cq.cc);
564567 else
565568 ibdev_err_ratelimited(
566569 aq->efa_dev,
567− "The device didn't send any completion for admin cmd %s(%d) status %d (ctx 0x%p, sq producer: %d, sq consumer: %d, cq consumer: %d)\n",
570+ "The device didn't send any completion for admin cmd %s(%d) status %d (id: %d, sq producer: %d, sq consumer: %d, cq consumer: %d)\n",
568571 efa_com_cmd_str(comp_ctx->cmd_opcode),
569572 comp_ctx->cmd_opcode, comp_ctx->status,
570− comp_ctx, aq->sq.pc, aq->sq.cc, aq->cq.cc);
573+ comp_ctx->cmd_id, aq->sq.pc, aq->sq.cc,
574+ aq->cq.cc);
571575
572576 clear_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state);
573577 err = -ETIME;
@@ -631,9 +635,9 @@ int efa_com_cmd_exec(struct efa_com_admin_queue *aq, · 57 unmodified lines
631635 if (IS_ERR(comp_ctx)) {
632636 ibdev_err_ratelimited(
633637 aq->efa_dev,
634− "Failed to submit command %s (opcode %u) err %ld\n",
638+ "Failed to submit command %s (opcode %u) err %pe\n",
635639 efa_com_cmd_str(cmd->aq_common_descriptor.opcode),
636− cmd->aq_common_descriptor.opcode, PTR_ERR(comp_ctx));
640+ cmd->aq_common_descriptor.opcode, comp_ctx);
637641
638642 up(&aq->avail_cmds);
639643 atomic64_inc(&aq->stats.cmd_err);
drivers/infiniband/hw/efa/efa_verbs.c+4−2
@@ -1788,7 +1788,8 @@ struct ib_mr *efa_reg_user_mr_dmabuf(struct ib_pd *ibpd, u64 start, · 1787 unmodified lines
17881788 access_flags);
17891789 if (IS_ERR(umem_dmabuf)) {
17901790 err = PTR_ERR(umem_dmabuf);
1791− ibdev_dbg(&dev->ibdev, "Failed to get dmabuf umem[%d]\n", err);
1791+ ibdev_dbg(&dev->ibdev, "Failed to get dmabuf umem[%pe]\n",
1792+ umem_dmabuf);
17921793 goto err_free;
17931794 }
17941795
@@ -1832,7 +1833,8 @@ struct ib_mr *efa_reg_mr(struct ib_pd *ibpd, u64 start, u64 length, · 37 unmodified lines
18321833 if (IS_ERR(mr->umem)) {
18331834 err = PTR_ERR(mr->umem);
18341835 ibdev_dbg(&dev->ibdev,
1835− "Failed to pin and map user space memory[%d]\n", err);
1836+ "Failed to pin and map user space memory[%pe]\n",
1837+ mr->umem);
18361838 goto err_free;
18371839 }
18381840
drivers/infiniband/hw/erdma/erdma_verbs.c+69−41
@@ -149,7 +149,7 @@ static int regmr_cmd(struct erdma_dev *dev, struct erdma_mr *mr) · 148 unmodified lines
149149 req.phy_addr[0] = mr->mem.mtt->buf_dma;
150150 mtt_level = ERDMA_MR_MTT_1LEVEL;
151151 } else {
152− req.phy_addr[0] = sg_dma_address(mr->mem.mtt->sglist);
152+ req.phy_addr[0] = mr->mem.mtt->dma_addrs[0];
153153 mtt_level = mr->mem.mtt->level;
154154 }
155155 } else if (mr->type != ERDMA_MR_TYPE_DMA) {
@@ -626,18 +626,27 @@ err_free_mtt: · 470 unmodified lines
626626 return ERR_PTR(-ENOMEM);
627627 }
628628
629−static void erdma_destroy_mtt_buf_sg(struct erdma_dev *dev,
630− struct erdma_mtt *mtt)
629+static void erdma_unmap_page_list(struct erdma_dev *dev, dma_addr_t *pg_dma,
630+ u32 npages)
631631 {
632− dma_unmap_sg(&dev->pdev->dev, mtt->sglist,
633− DIV_ROUND_UP(mtt->size, PAGE_SIZE), DMA_TO_DEVICE);
634− vfree(mtt->sglist);
632+ u32 i;
633+
634+ for (i = 0; i < npages; i++)
635+ dma_unmap_page(&dev->pdev->dev, pg_dma[i], PAGE_SIZE,
636+ DMA_TO_DEVICE);
637+}
638+
639+static void erdma_destroy_mtt_buf_dma_addrs(struct erdma_dev *dev,
640+ struct erdma_mtt *mtt)
641+{
642+ erdma_unmap_page_list(dev, mtt->dma_addrs, mtt->npages);
643+ vfree(mtt->dma_addrs);
635644 }
636645
637646 static void erdma_destroy_scatter_mtt(struct erdma_dev *dev,
638647 struct erdma_mtt *mtt)
639648 {
640− erdma_destroy_mtt_buf_sg(dev, mtt);
649+ erdma_destroy_mtt_buf_dma_addrs(dev, mtt);
641650 vfree(mtt->buf);
642651 kfree(mtt);
643652 }
@@ -645,50 +654,69 @@ static void erdma_destroy_scatter_mtt(struct erdma_dev *dev, · 1 unmodified line
645654 static void erdma_init_middle_mtt(struct erdma_mtt *mtt,
646655 struct erdma_mtt *low_mtt)
647656 {
648− struct scatterlist *sg;
649− u32 idx = 0, i;
657+ dma_addr_t *pg_addr = mtt->buf;
658+ u32 i;
650659
651− for_each_sg(low_mtt->sglist, sg, low_mtt->nsg, i)
652− mtt->buf[idx++] = sg_dma_address(sg);
660+ for (i = 0; i < low_mtt->npages; i++)
661+ pg_addr[i] = low_mtt->dma_addrs[i];
653662 }
654663
655−static int erdma_create_mtt_buf_sg(struct erdma_dev *dev, struct erdma_mtt *mtt)
664+static u32 vmalloc_to_dma_addrs(struct erdma_dev *dev, dma_addr_t **dma_addrs,
665+ void *buf, u64 len)
656666 {
657− struct scatterlist *sglist;
658− void *buf = mtt->buf;
659− u32 npages, i, nsg;
667+ dma_addr_t *pg_dma;
660668 struct page *pg;
669+ u32 npages, i;
670+ void *addr;
661671
662− /* Failed if buf is not page aligned */
663− if ((uintptr_t)buf & ~PAGE_MASK)
664− return -EINVAL;
665−
666− npages = DIV_ROUND_UP(mtt->size, PAGE_SIZE);
667− sglist = vzalloc(npages * sizeof(*sglist));
668− if (!sglist)
669− return -ENOMEM;
672+ npages = (PAGE_ALIGN((u64)buf + len) - PAGE_ALIGN_DOWN((u64)buf)) >>
673+ PAGE_SHIFT;
674+ pg_dma = vcalloc(npages, sizeof(*pg_dma));
675+ if (!pg_dma)
676+ return 0;
670677
671− sg_init_table(sglist, npages);
678+ addr = buf;
672679 for (i = 0; i < npages; i++) {
673− pg = vmalloc_to_page(buf);
680+ pg = vmalloc_to_page(addr);
674681 if (!pg)
675682 goto err;
676− sg_set_page(&sglist[i], pg, PAGE_SIZE, 0);
677− buf += PAGE_SIZE;
683+
684+ pg_dma[i] = dma_map_page(&dev->pdev->dev, pg, 0, PAGE_SIZE,
685+ DMA_TO_DEVICE);
686+ if (dma_mapping_error(&dev->pdev->dev, pg_dma[i]))
687+ goto err;
688+
689+ addr += PAGE_SIZE;
678690 }
679691
680− nsg = dma_map_sg(&dev->pdev->dev, sglist, npages, DMA_TO_DEVICE);
681− if (!nsg)
682− goto err;
692+ *dma_addrs = pg_dma;
683693
684− mtt->sglist = sglist;
685− mtt->nsg = nsg;
694+ return npages;
695+err:
696+ erdma_unmap_page_list(dev, pg_dma, i);
697+ vfree(pg_dma);
686698
687699 return 0;
688−err:
689− vfree(sglist);
700+}
690701
691− return -ENOMEM;
702+static int erdma_create_mtt_buf_dma_addrs(struct erdma_dev *dev,
703+ struct erdma_mtt *mtt)
704+{
705+ dma_addr_t *addrs;
706+ u32 npages;
707+
708+ /* Failed if buf is not page aligned */
709+ if ((uintptr_t)mtt->buf & ~PAGE_MASK)
710+ return -EINVAL;
711+
712+ npages = vmalloc_to_dma_addrs(dev, &addrs, mtt->buf, mtt->size);
713+ if (!npages)
714+ return -ENOMEM;
715+
716+ mtt->dma_addrs = addrs;
717+ mtt->npages = npages;
718+
719+ return 0;
692720 }
693721
694722 static struct erdma_mtt *erdma_create_scatter_mtt(struct erdma_dev *dev,
@@ -707,12 +735,12 @@ static struct erdma_mtt *erdma_create_scatter_mtt(struct erdma_dev *dev, · 12 unmodified lines
707735 if (!mtt->buf)
708736 goto err_free_mtt;
709737
710− ret = erdma_create_mtt_buf_sg(dev, mtt);
738+ ret = erdma_create_mtt_buf_dma_addrs(dev, mtt);
711739 if (ret)
712740 goto err_free_mtt_buf;
713741
714− ibdev_dbg(&dev->ibdev, "create scatter mtt, size:%lu, nsg:%u\n",
715− mtt->size, mtt->nsg);
742+ ibdev_dbg(&dev->ibdev, "create scatter mtt, size:%lu, npages:%u\n",
743+ mtt->size, mtt->npages);
716744
717745 return mtt;
718746
@@ -746,8 +774,8 @@ static struct erdma_mtt *erdma_create_mtt(struct erdma_dev *dev, size_t size, · 27 unmodified lines
746774 level = 1;
747775
748776 /* convergence the mtt table. */
749− while (mtt->nsg != 1 && level <= 3) {
750− tmp_mtt = erdma_create_scatter_mtt(dev, MTT_SIZE(mtt->nsg));
777+ while (mtt->npages != 1 && level <= 3) {
778+ tmp_mtt = erdma_create_scatter_mtt(dev, MTT_SIZE(mtt->npages));
751779 if (IS_ERR(tmp_mtt)) {
752780 ret = PTR_ERR(tmp_mtt);
753781 goto err_free_mtt;
@@ -765,7 +793,7 @@ static struct erdma_mtt *erdma_create_mtt(struct erdma_dev *dev, size_t size, · 11 unmodified lines
765793
766794 mtt->level = level;
767795 ibdev_dbg(&dev->ibdev, "top mtt: level:%d, dma_addr 0x%llx\n",
768− mtt->level, mtt->sglist[0].dma_address);
796+ mtt->level, mtt->dma_addrs[0]);
769797
770798 return mtt;
771799 err_free_mtt:
drivers/infiniband/hw/erdma/erdma_verbs.h+2−2
@@ -99,8 +99,8 @@ struct erdma_mtt { · 98 unmodified lines
9999 union {
100100 dma_addr_t buf_dma;
101101 struct {
102− struct scatterlist *sglist;
103− u32 nsg;
102+ dma_addr_t *dma_addrs;
103+ u32 npages;
104104 u32 level;
105105 };
106106 };
drivers/infiniband/hw/hfi1/device.c+2−2
@@ -64,9 +64,9 @@ int hfi1_cdev_init(int minor, const char *name, · 63 unmodified lines
6464
6565 if (IS_ERR(device)) {
6666 ret = PTR_ERR(device);
67+ pr_err("Could not create device for minor %d, %s (err %pe)\n",
68+ minor, name, device);
6769 device = NULL;
68− pr_err("Could not create device for minor %d, %s (err %d)\n",
69− minor, name, -ret);
7070 cdev_del(cdev);
7171 }
7272 done:
drivers/infiniband/hw/hfi1/sdma.c+1−1
@@ -990,7 +990,7 @@ ssize_t sdma_set_cpu_to_sde_map(struct sdma_engine *sde, const char *buf, · 989 unmodified lines
990990 }
991991
992992 /* Clean up old mappings */
993− for_each_cpu(cpu, cpu_online_mask) {
993+ for_each_online_cpu(cpu) {
994994 struct sdma_rht_node *rht_node;
995995
996996 /* Don't cleanup sdes that are set in the new mask */
drivers/infiniband/hw/hfi1/user_sdma.c+2−2
@@ -498,8 +498,8 @@ int hfi1_user_sdma_process_request(struct hfi1_filedata *fd, · 497 unmodified lines
498498 ntids, sizeof(*req->tids));
499499 if (IS_ERR(tmp)) {
500500 ret = PTR_ERR(tmp);
501− SDMA_DBG(req, "Failed to copy %d TIDs (%d)",
502− ntids, ret);
501+ SDMA_DBG(req, "Failed to copy %d TIDs (%pe)", ntids,
502+ tmp);
503503 goto free_req;
504504 }
505505 req->tids = tmp;
drivers/infiniband/hw/hns/hns_roce_mr.c+4−4
@@ -594,8 +594,8 @@ static int mtr_alloc_bufs(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr, · 593 unmodified lines
594594 mtr->umem = ib_umem_get(ibdev, user_addr, total_size,
595595 buf_attr->user_access);
596596 if (IS_ERR(mtr->umem)) {
597− ibdev_err(ibdev, "failed to get umem, ret = %ld.\n",
598− PTR_ERR(mtr->umem));
597+ ibdev_err(ibdev, "failed to get umem, ret = %pe.\n",
598+ mtr->umem);
599599 return -ENOMEM;
600600 }
601601 } else {
@@ -605,8 +605,8 @@ static int mtr_alloc_bufs(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr, · 3 unmodified lines
605605 !mtr_has_mtt(buf_attr) ?
606606 HNS_ROCE_BUF_DIRECT : 0);
607607 if (IS_ERR(mtr->kmem)) {
608− ibdev_err(ibdev, "failed to alloc kmem, ret = %ld.\n",
609− PTR_ERR(mtr->kmem));
608+ ibdev_err(ibdev, "failed to alloc kmem, ret = %pe.\n",
609+ mtr->kmem);
610610 return PTR_ERR(mtr->kmem);
611611 }
612612 }
drivers/infiniband/hw/ionic/Kconfig+15−0
@@ -0,0 +1,15 @@
1+# SPDX-License-Identifier: GPL-2.0
2+# Copyright (C) 2018-2025, Advanced Micro Devices, Inc.
3+
4+config INFINIBAND_IONIC
5+ tristate "AMD Pensando DSC RDMA/RoCE Support"
6+ depends on NETDEVICES && ETHERNET && PCI && INET && IONIC
7+ help
8+ This enables RDMA/RoCE support for the AMD Pensando family of
9+ Distributed Services Cards (DSCs).
10+
11+ To learn more, visit our website at
12+ <https://www.amd.com/en/products/accelerators/pensando.html>.
13+
14+ To compile this driver as a module, choose M here. The module
15+ will be called ionic_rdma.
drivers/infiniband/hw/ionic/Makefile+9−0
@@ -0,0 +1,9 @@
1+# SPDX-License-Identifier: GPL-2.0
2+
3+ccflags-y := -I $(srctree)/drivers/net/ethernet/pensando/ionic
4+
5+obj-$(CONFIG_INFINIBAND_IONIC) += ionic_rdma.o
6+
7+ionic_rdma-y := \
8+ ionic_ibdev.o ionic_lif_cfg.o ionic_queue.o ionic_pgtbl.o ionic_admin.o \
9+ ionic_controlpath.o ionic_datapath.o ionic_hw_stats.o
drivers/infiniband/hw/ionic/ionic_admin.c+1229−0
@@ -0,0 +1,1229 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#include <linux/interrupt.h>
5+#include <linux/module.h>
6+#include <linux/printk.h>
7+
8+#include "ionic_fw.h"
9+#include "ionic_ibdev.h"
10+
11+#define IONIC_EQ_COUNT_MIN 4
12+#define IONIC_AQ_COUNT_MIN 1
13+
14+/* not a valid queue position or negative error status */
15+#define IONIC_ADMIN_POSTED 0x10000
16+
17+/* cpu can be held with irq disabled for COUNT * MS (for create/destroy_ah) */
18+#define IONIC_ADMIN_BUSY_RETRY_COUNT 2000
19+#define IONIC_ADMIN_BUSY_RETRY_MS 1
20+
21+/* admin queue will be considered failed if a command takes longer */
22+#define IONIC_ADMIN_TIMEOUT (HZ * 2)
23+#define IONIC_ADMIN_WARN (HZ / 8)
24+
25+/* will poll for admin cq to tolerate and report from missed event */
26+#define IONIC_ADMIN_DELAY (HZ / 8)
27+
28+/* work queue for polling the event queue and admin cq */
29+struct workqueue_struct *ionic_evt_workq;
30+
31+static void ionic_admin_timedout(struct ionic_aq *aq)
32+{
33+ struct ionic_ibdev *dev = aq->dev;
34+ unsigned long irqflags;
35+ u16 pos;
36+
37+ spin_lock_irqsave(&aq->lock, irqflags);
38+ if (ionic_queue_empty(&aq->q))
39+ goto out;
40+
41+ /* Reset ALL adminq if any one times out */
42+ if (atomic_read(&aq->admin_state) < IONIC_ADMIN_KILLED)
43+ queue_work(ionic_evt_workq, &dev->reset_work);
44+
45+ ibdev_err(&dev->ibdev, "admin command timed out, aq %d after: %ums\n",
46+ aq->aqid, (u32)jiffies_to_msecs(jiffies - aq->stamp));
47+
48+ pos = (aq->q.prod - 1) & aq->q.mask;
49+ if (pos == aq->q.cons)
50+ goto out;
51+
52+ ibdev_warn(&dev->ibdev, "admin pos %u (last posted)\n", pos);
53+ print_hex_dump(KERN_WARNING, "cmd ", DUMP_PREFIX_OFFSET, 16, 1,
54+ ionic_queue_at(&aq->q, pos),
55+ BIT(aq->q.stride_log2), true);
56+
57+out:
58+ spin_unlock_irqrestore(&aq->lock, irqflags);
59+}
60+
61+static void ionic_admin_reset_dwork(struct ionic_ibdev *dev)
62+{
63+ if (atomic_read(&dev->admin_state) == IONIC_ADMIN_KILLED)
64+ return;
65+
66+ queue_delayed_work(ionic_evt_workq, &dev->admin_dwork,
67+ IONIC_ADMIN_DELAY);
68+}
69+
70+static void ionic_admin_reset_wdog(struct ionic_aq *aq)
71+{
72+ if (atomic_read(&aq->admin_state) == IONIC_ADMIN_KILLED)
73+ return;
74+
75+ aq->stamp = jiffies;
76+ ionic_admin_reset_dwork(aq->dev);
77+}
78+
79+static bool ionic_admin_next_cqe(struct ionic_ibdev *dev, struct ionic_cq *cq,
80+ struct ionic_v1_cqe **cqe)
81+{
82+ struct ionic_v1_cqe *qcqe = ionic_queue_at_prod(&cq->q);
83+
84+ if (unlikely(cq->color != ionic_v1_cqe_color(qcqe)))
85+ return false;
86+
87+ /* Prevent out-of-order reads of the CQE */
88+ dma_rmb();
89+ *cqe = qcqe;
90+
91+ return true;
92+}
93+
94+static void ionic_admin_poll_locked(struct ionic_aq *aq)
95+{
96+ struct ionic_cq *cq = &aq->vcq->cq[0];
97+ struct ionic_admin_wr *wr, *wr_next;
98+ struct ionic_ibdev *dev = aq->dev;
99+ u32 wr_strides, avlbl_strides;
100+ struct ionic_v1_cqe *cqe;
101+ u32 qtf, qid;
102+ u16 old_prod;
103+ u8 type;
104+
105+ lockdep_assert_held(&aq->lock);
106+
107+ if (atomic_read(&aq->admin_state) == IONIC_ADMIN_KILLED) {
108+ list_for_each_entry_safe(wr, wr_next, &aq->wr_prod, aq_ent) {
109+ INIT_LIST_HEAD(&wr->aq_ent);
110+ aq->q_wr[wr->status].wr = NULL;
111+ wr->status = atomic_read(&aq->admin_state);
112+ complete_all(&wr->work);
113+ }
114+ INIT_LIST_HEAD(&aq->wr_prod);
115+
116+ list_for_each_entry_safe(wr, wr_next, &aq->wr_post, aq_ent) {
117+ INIT_LIST_HEAD(&wr->aq_ent);
118+ wr->status = atomic_read(&aq->admin_state);
119+ complete_all(&wr->work);
120+ }
121+ INIT_LIST_HEAD(&aq->wr_post);
122+
123+ return;
124+ }
125+
126+ old_prod = cq->q.prod;
127+
128+ while (ionic_admin_next_cqe(dev, cq, &cqe)) {
129+ qtf = ionic_v1_cqe_qtf(cqe);
130+ qid = ionic_v1_cqe_qtf_qid(qtf);
131+ type = ionic_v1_cqe_qtf_type(qtf);
132+
133+ if (unlikely(type != IONIC_V1_CQE_TYPE_ADMIN)) {
134+ ibdev_warn_ratelimited(&dev->ibdev,
135+ "bad cqe type %u\n", type);
136+ goto cq_next;
137+ }
138+
139+ if (unlikely(qid != aq->aqid)) {
140+ ibdev_warn_ratelimited(&dev->ibdev,
141+ "bad cqe qid %u\n", qid);
142+ goto cq_next;
143+ }
144+
145+ if (unlikely(be16_to_cpu(cqe->admin.cmd_idx) != aq->q.cons)) {
146+ ibdev_warn_ratelimited(&dev->ibdev,
147+ "bad idx %u cons %u qid %u\n",
148+ be16_to_cpu(cqe->admin.cmd_idx),
149+ aq->q.cons, qid);
150+ goto cq_next;
151+ }
152+
153+ if (unlikely(ionic_queue_empty(&aq->q))) {
154+ ibdev_warn_ratelimited(&dev->ibdev,
155+ "bad cqe for empty adminq\n");
156+ goto cq_next;
157+ }
158+
159+ wr = aq->q_wr[aq->q.cons].wr;
160+ if (wr) {
161+ aq->q_wr[aq->q.cons].wr = NULL;
162+ list_del_init(&wr->aq_ent);
163+
164+ wr->cqe = *cqe;
165+ wr->status = atomic_read(&aq->admin_state);
166+ complete_all(&wr->work);
167+ }
168+
169+ ionic_queue_consume_entries(&aq->q,
170+ aq->q_wr[aq->q.cons].wqe_strides);
171+
172+cq_next:
173+ ionic_queue_produce(&cq->q);
174+ cq->color = ionic_color_wrap(cq->q.prod, cq->color);
175+ }
176+
177+ if (old_prod != cq->q.prod) {
178+ ionic_admin_reset_wdog(aq);
179+ cq->q.cons = cq->q.prod;
180+ ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.cq_qtype,
181+ ionic_queue_dbell_val(&cq->q));
182+ queue_work(ionic_evt_workq, &aq->work);
183+ } else if (!aq->armed) {
184+ aq->armed = true;
185+ cq->arm_any_prod = ionic_queue_next(&cq->q, cq->arm_any_prod);
186+ ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.cq_qtype,
187+ cq->q.dbell | IONIC_CQ_RING_ARM |
188+ cq->arm_any_prod);
189+ queue_work(ionic_evt_workq, &aq->work);
190+ }
191+
192+ if (atomic_read(&aq->admin_state) != IONIC_ADMIN_ACTIVE)
193+ return;
194+
195+ old_prod = aq->q.prod;
196+
197+ if (ionic_queue_empty(&aq->q) && !list_empty(&aq->wr_post))
198+ ionic_admin_reset_wdog(aq);
199+
200+ if (list_empty(&aq->wr_post))
201+ return;
202+
203+ do {
204+ u8 *src;
205+ int i, src_len;
206+ size_t stride_len;
207+
208+ wr = list_first_entry(&aq->wr_post, struct ionic_admin_wr,
209+ aq_ent);
210+ wr_strides = (le16_to_cpu(wr->wqe.len) + ADMIN_WQE_HDR_LEN +
211+ (ADMIN_WQE_STRIDE - 1)) >> aq->q.stride_log2;
212+ avlbl_strides = ionic_queue_length_remaining(&aq->q);
213+
214+ if (wr_strides > avlbl_strides)
215+ break;
216+
217+ list_move(&wr->aq_ent, &aq->wr_prod);
218+ wr->status = aq->q.prod;
219+ aq->q_wr[aq->q.prod].wr = wr;
220+ aq->q_wr[aq->q.prod].wqe_strides = wr_strides;
221+
222+ src_len = le16_to_cpu(wr->wqe.len);
223+ src = (uint8_t *)&wr->wqe.cmd;
224+
225+ /* First stride */
226+ memcpy(ionic_queue_at_prod(&aq->q), &wr->wqe,
227+ ADMIN_WQE_HDR_LEN);
228+ stride_len = ADMIN_WQE_STRIDE - ADMIN_WQE_HDR_LEN;
229+ if (stride_len > src_len)
230+ stride_len = src_len;
231+ memcpy(ionic_queue_at_prod(&aq->q) + ADMIN_WQE_HDR_LEN,
232+ src, stride_len);
233+ ibdev_dbg(&dev->ibdev, "post admin prod %u (%u strides)\n",
234+ aq->q.prod, wr_strides);
235+ print_hex_dump_debug("wqe ", DUMP_PREFIX_OFFSET, 16, 1,
236+ ionic_queue_at_prod(&aq->q),
237+ BIT(aq->q.stride_log2), true);
238+ ionic_queue_produce(&aq->q);
239+
240+ /* Remaining strides */
241+ for (i = stride_len; i < src_len; i += stride_len) {
242+ stride_len = ADMIN_WQE_STRIDE;
243+
244+ if (i + stride_len > src_len)
245+ stride_len = src_len - i;
246+
247+ memcpy(ionic_queue_at_prod(&aq->q), src + i,
248+ stride_len);
249+ print_hex_dump_debug("wqe ", DUMP_PREFIX_OFFSET, 16, 1,
250+ ionic_queue_at_prod(&aq->q),
251+ BIT(aq->q.stride_log2), true);
252+ ionic_queue_produce(&aq->q);
253+ }
254+ } while (!list_empty(&aq->wr_post));
255+
256+ if (old_prod != aq->q.prod)
257+ ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.aq_qtype,
258+ ionic_queue_dbell_val(&aq->q));
259+}
260+
261+static void ionic_admin_dwork(struct work_struct *ws)
262+{
263+ struct ionic_ibdev *dev =
264+ container_of(ws, struct ionic_ibdev, admin_dwork.work);
265+ struct ionic_aq *aq, *bad_aq = NULL;
266+ bool do_reschedule = false;
267+ unsigned long irqflags;
268+ bool do_reset = false;
269+ u16 pos;
270+ int i;
271+
272+ for (i = 0; i < dev->lif_cfg.aq_count; i++) {
273+ aq = dev->aq_vec[i];
274+
275+ spin_lock_irqsave(&aq->lock, irqflags);
276+
277+ if (ionic_queue_empty(&aq->q))
278+ goto next_aq;
279+
280+ /* Reschedule if any queue has outstanding work */
281+ do_reschedule = true;
282+
283+ if (time_is_after_eq_jiffies(aq->stamp + IONIC_ADMIN_WARN))
284+ /* Warning threshold not met, nothing to do */
285+ goto next_aq;
286+
287+ /* See if polling now makes some progress */
288+ pos = aq->q.cons;
289+ ionic_admin_poll_locked(aq);
290+ if (pos != aq->q.cons) {
291+ ibdev_dbg(&dev->ibdev,
292+ "missed event for acq %d\n", aq->cqid);
293+ goto next_aq;
294+ }
295+
296+ if (time_is_after_eq_jiffies(aq->stamp +
297+ IONIC_ADMIN_TIMEOUT)) {
298+ /* Timeout threshold not met */
299+ ibdev_dbg(&dev->ibdev, "no progress after %ums\n",
300+ (u32)jiffies_to_msecs(jiffies - aq->stamp));
301+ goto next_aq;
302+ }
303+
304+ /* Queue timed out */
305+ bad_aq = aq;
306+ do_reset = true;
307+next_aq:
308+ spin_unlock_irqrestore(&aq->lock, irqflags);
309+ }
310+
311+ if (do_reset)
312+ /* Reset RDMA lif on a timeout */
313+ ionic_admin_timedout(bad_aq);
314+ else if (do_reschedule)
315+ /* Try to poll again later */
316+ ionic_admin_reset_dwork(dev);
317+}
318+
319+static void ionic_admin_work(struct work_struct *ws)
320+{
321+ struct ionic_aq *aq = container_of(ws, struct ionic_aq, work);
322+ unsigned long irqflags;
323+
324+ spin_lock_irqsave(&aq->lock, irqflags);
325+ ionic_admin_poll_locked(aq);
326+ spin_unlock_irqrestore(&aq->lock, irqflags);
327+}
328+
329+static void ionic_admin_post_aq(struct ionic_aq *aq, struct ionic_admin_wr *wr)
330+{
331+ unsigned long irqflags;
332+ bool poll;
333+
334+ wr->status = IONIC_ADMIN_POSTED;
335+ wr->aq = aq;
336+
337+ spin_lock_irqsave(&aq->lock, irqflags);
338+ poll = list_empty(&aq->wr_post);
339+ list_add(&wr->aq_ent, &aq->wr_post);
340+ if (poll)
341+ ionic_admin_poll_locked(aq);
342+ spin_unlock_irqrestore(&aq->lock, irqflags);
343+}
344+
345+void ionic_admin_post(struct ionic_ibdev *dev, struct ionic_admin_wr *wr)
346+{
347+ int aq_idx;
348+
349+ /* Use cpu id for the adminq selection */
350+ aq_idx = raw_smp_processor_id() % dev->lif_cfg.aq_count;
351+ ionic_admin_post_aq(dev->aq_vec[aq_idx], wr);
352+}
353+
354+static void ionic_admin_cancel(struct ionic_admin_wr *wr)
355+{
356+ struct ionic_aq *aq = wr->aq;
357+ unsigned long irqflags;
358+
359+ spin_lock_irqsave(&aq->lock, irqflags);
360+
361+ if (!list_empty(&wr->aq_ent)) {
362+ list_del(&wr->aq_ent);
363+ if (wr->status != IONIC_ADMIN_POSTED)
364+ aq->q_wr[wr->status].wr = NULL;
365+ }
366+
367+ spin_unlock_irqrestore(&aq->lock, irqflags);
368+}
369+
370+static int ionic_admin_busy_wait(struct ionic_admin_wr *wr)
371+{
372+ struct ionic_aq *aq = wr->aq;
373+ unsigned long irqflags;
374+ int try_i;
375+
376+ for (try_i = 0; try_i < IONIC_ADMIN_BUSY_RETRY_COUNT; ++try_i) {
377+ if (completion_done(&wr->work))
378+ return 0;
379+
380+ mdelay(IONIC_ADMIN_BUSY_RETRY_MS);
381+
382+ spin_lock_irqsave(&aq->lock, irqflags);
383+ ionic_admin_poll_locked(aq);
384+ spin_unlock_irqrestore(&aq->lock, irqflags);
385+ }
386+
387+ /*
388+ * we timed out. Initiate RDMA LIF reset and indicate
389+ * error to caller.
390+ */
391+ ionic_admin_timedout(aq);
392+ return -ETIMEDOUT;
393+}
394+
395+int ionic_admin_wait(struct ionic_ibdev *dev, struct ionic_admin_wr *wr,
396+ enum ionic_admin_flags flags)
397+{
398+ int rc, timo;
399+
400+ if (flags & IONIC_ADMIN_F_BUSYWAIT) {
401+ /* Spin */
402+ rc = ionic_admin_busy_wait(wr);
403+ } else if (flags & IONIC_ADMIN_F_INTERRUPT) {
404+ /*
405+ * Interruptible sleep, 1s timeout
406+ * This is used for commands which are safe for the caller
407+ * to clean up without killing and resetting the adminq.
408+ */
409+ timo = wait_for_completion_interruptible_timeout(&wr->work,
410+ HZ);
411+ if (timo > 0)
412+ rc = 0;
413+ else if (timo == 0)
414+ rc = -ETIMEDOUT;
415+ else
416+ rc = timo;
417+ } else {
418+ /*
419+ * Uninterruptible sleep
420+ * This is used for commands which are NOT safe for the
421+ * caller to clean up. Cleanup must be handled by the
422+ * adminq kill and reset process so that host memory is
423+ * not corrupted by the device.
424+ */
425+ wait_for_completion(&wr->work);
426+ rc = 0;
427+ }
428+
429+ if (rc) {
430+ ibdev_warn(&dev->ibdev, "wait status %d\n", rc);
431+ ionic_admin_cancel(wr);
432+ } else if (wr->status == IONIC_ADMIN_KILLED) {
433+ ibdev_dbg(&dev->ibdev, "admin killed\n");
434+
435+ /* No error if admin already killed during teardown */
436+ rc = (flags & IONIC_ADMIN_F_TEARDOWN) ? 0 : -ENODEV;
437+ } else if (ionic_v1_cqe_error(&wr->cqe)) {
438+ ibdev_warn(&dev->ibdev, "opcode %u error %u\n",
439+ wr->wqe.op,
440+ be32_to_cpu(wr->cqe.status_length));
441+ rc = -EINVAL;
442+ }
443+ return rc;
444+}
445+
446+static int ionic_rdma_devcmd(struct ionic_ibdev *dev,
447+ struct ionic_admin_ctx *admin)
448+{
449+ int rc;
450+
451+ rc = ionic_adminq_post_wait(dev->lif_cfg.lif, admin);
452+ if (rc)
453+ return rc;
454+
455+ return ionic_error_to_errno(admin->comp.comp.status);
456+}
457+
458+int ionic_rdma_reset_devcmd(struct ionic_ibdev *dev)
459+{
460+ struct ionic_admin_ctx admin = {
461+ .work = COMPLETION_INITIALIZER_ONSTACK(admin.work),
462+ .cmd.rdma_reset = {
463+ .opcode = IONIC_CMD_RDMA_RESET_LIF,
464+ .lif_index = cpu_to_le16(dev->lif_cfg.lif_index),
465+ },
466+ };
467+
468+ return ionic_rdma_devcmd(dev, &admin);
469+}
470+
471+static int ionic_rdma_queue_devcmd(struct ionic_ibdev *dev,
472+ struct ionic_queue *q,
473+ u32 qid, u32 cid, u16 opcode)
474+{
475+ struct ionic_admin_ctx admin = {
476+ .work = COMPLETION_INITIALIZER_ONSTACK(admin.work),
477+ .cmd.rdma_queue = {
478+ .opcode = opcode,
479+ .lif_index = cpu_to_le16(dev->lif_cfg.lif_index),
480+ .qid_ver = cpu_to_le32(qid),
481+ .cid = cpu_to_le32(cid),
482+ .dbid = cpu_to_le16(dev->lif_cfg.dbid),
483+ .depth_log2 = q->depth_log2,
484+ .stride_log2 = q->stride_log2,
485+ .dma_addr = cpu_to_le64(q->dma),
486+ },
487+ };
488+
489+ return ionic_rdma_devcmd(dev, &admin);
490+}
491+
492+static void ionic_rdma_admincq_comp(struct ib_cq *ibcq, void *cq_context)
493+{
494+ struct ionic_aq *aq = cq_context;
495+ unsigned long irqflags;
496+
497+ spin_lock_irqsave(&aq->lock, irqflags);
498+ aq->armed = false;
499+ if (atomic_read(&aq->admin_state) < IONIC_ADMIN_KILLED)
500+ queue_work(ionic_evt_workq, &aq->work);
501+ spin_unlock_irqrestore(&aq->lock, irqflags);
502+}
503+
504+static void ionic_rdma_admincq_event(struct ib_event *event, void *cq_context)
505+{
506+ struct ionic_aq *aq = cq_context;
507+
508+ ibdev_err(&aq->dev->ibdev, "admincq event %d\n", event->event);
509+}
510+
511+static struct ionic_vcq *ionic_create_rdma_admincq(struct ionic_ibdev *dev,
512+ int comp_vector)
513+{
514+ struct ib_cq_init_attr attr = {
515+ .cqe = IONIC_AQ_DEPTH,
516+ .comp_vector = comp_vector,
517+ };
518+ struct ionic_tbl_buf buf = {};
519+ struct ionic_vcq *vcq;
520+ struct ionic_cq *cq;
521+ int rc;
522+
523+ vcq = kzalloc(sizeof(*vcq), GFP_KERNEL);
524+ if (!vcq)
525+ return ERR_PTR(-ENOMEM);
526+
527+ vcq->ibcq.device = &dev->ibdev;
528+ vcq->ibcq.comp_handler = ionic_rdma_admincq_comp;
529+ vcq->ibcq.event_handler = ionic_rdma_admincq_event;
530+ atomic_set(&vcq->ibcq.usecnt, 0);
531+
532+ vcq->udma_mask = 1;
533+ cq = &vcq->cq[0];
534+
535+ rc = ionic_create_cq_common(vcq, &buf, &attr, NULL, NULL,
536+ NULL, NULL, 0);
537+ if (rc)
538+ goto err_init;
539+
540+ rc = ionic_rdma_queue_devcmd(dev, &cq->q, cq->cqid, cq->eqid,
541+ IONIC_CMD_RDMA_CREATE_CQ);
542+ if (rc)
543+ goto err_cmd;
544+
545+ return vcq;
546+
547+err_cmd:
548+ ionic_destroy_cq_common(dev, cq);
549+err_init:
550+ kfree(vcq);
551+
552+ return ERR_PTR(rc);
553+}
554+
555+static struct ionic_aq *__ionic_create_rdma_adminq(struct ionic_ibdev *dev,
556+ u32 aqid, u32 cqid)
557+{
558+ struct ionic_aq *aq;
559+ int rc;
560+
561+ aq = kzalloc(sizeof(*aq), GFP_KERNEL);
562+ if (!aq)
563+ return ERR_PTR(-ENOMEM);
564+
565+ atomic_set(&aq->admin_state, IONIC_ADMIN_KILLED);
566+ aq->dev = dev;
567+ aq->aqid = aqid;
568+ aq->cqid = cqid;
569+ spin_lock_init(&aq->lock);
570+
571+ rc = ionic_queue_init(&aq->q, dev->lif_cfg.hwdev, IONIC_EQ_DEPTH,
572+ ADMIN_WQE_STRIDE);
573+ if (rc)
574+ goto err_q;
575+
576+ ionic_queue_dbell_init(&aq->q, aq->aqid);
577+
578+ aq->q_wr = kcalloc((u32)aq->q.mask + 1, sizeof(*aq->q_wr), GFP_KERNEL);
579+ if (!aq->q_wr) {
580+ rc = -ENOMEM;
581+ goto err_wr;
582+ }
583+
584+ INIT_LIST_HEAD(&aq->wr_prod);
585+ INIT_LIST_HEAD(&aq->wr_post);
586+
587+ INIT_WORK(&aq->work, ionic_admin_work);
588+ aq->armed = false;
589+
590+ return aq;
591+
592+err_wr:
593+ ionic_queue_destroy(&aq->q, dev->lif_cfg.hwdev);
594+err_q:
595+ kfree(aq);
596+
597+ return ERR_PTR(rc);
598+}
599+
600+static void __ionic_destroy_rdma_adminq(struct ionic_ibdev *dev,
601+ struct ionic_aq *aq)
602+{
603+ kfree(aq->q_wr);
604+ ionic_queue_destroy(&aq->q, dev->lif_cfg.hwdev);
605+ kfree(aq);
606+}
607+
608+static struct ionic_aq *ionic_create_rdma_adminq(struct ionic_ibdev *dev,
609+ u32 aqid, u32 cqid)
610+{
611+ struct ionic_aq *aq;
612+ int rc;
613+
614+ aq = __ionic_create_rdma_adminq(dev, aqid, cqid);
615+ if (IS_ERR(aq))
616+ return aq;
617+
618+ rc = ionic_rdma_queue_devcmd(dev, &aq->q, aq->aqid, aq->cqid,
619+ IONIC_CMD_RDMA_CREATE_ADMINQ);
620+ if (rc)
621+ goto err_cmd;
622+
623+ return aq;
624+
625+err_cmd:
626+ __ionic_destroy_rdma_adminq(dev, aq);
627+
628+ return ERR_PTR(rc);
629+}
630+
631+static void ionic_flush_qs(struct ionic_ibdev *dev)
632+{
633+ struct ionic_qp *qp, *qp_tmp;
634+ struct ionic_cq *cq, *cq_tmp;
635+ LIST_HEAD(flush_list);
636+ unsigned long index;
637+
638+ WARN_ON(!irqs_disabled());
639+
640+ /* Flush qp send and recv */
641+ xa_lock(&dev->qp_tbl);
642+ xa_for_each(&dev->qp_tbl, index, qp) {
643+ kref_get(&qp->qp_kref);
644+ list_add_tail(&qp->ibkill_flush_ent, &flush_list);
645+ }
646+ xa_unlock(&dev->qp_tbl);
647+
648+ list_for_each_entry_safe(qp, qp_tmp, &flush_list, ibkill_flush_ent) {
649+ ionic_flush_qp(dev, qp);
650+ kref_put(&qp->qp_kref, ionic_qp_complete);
651+ list_del(&qp->ibkill_flush_ent);
652+ }
653+
654+ /* Notify completions */
655+ xa_lock(&dev->cq_tbl);
656+ xa_for_each(&dev->cq_tbl, index, cq) {
657+ kref_get(&cq->cq_kref);
658+ list_add_tail(&cq->ibkill_flush_ent, &flush_list);
659+ }
660+ xa_unlock(&dev->cq_tbl);
661+
662+ list_for_each_entry_safe(cq, cq_tmp, &flush_list, ibkill_flush_ent) {
663+ ionic_notify_flush_cq(cq);
664+ kref_put(&cq->cq_kref, ionic_cq_complete);
665+ list_del(&cq->ibkill_flush_ent);
666+ }
667+}
668+
669+static void ionic_kill_ibdev(struct ionic_ibdev *dev, bool fatal_path)
670+{
671+ unsigned long irqflags;
672+ bool do_flush = false;
673+ int i;
674+
675+ /* Mark AQs for drain and flush the QPs while irq is disabled */
676+ local_irq_save(irqflags);
677+
678+ /* Mark the admin queue, flushing at most once */
679+ for (i = 0; i < dev->lif_cfg.aq_count; i++) {
680+ struct ionic_aq *aq = dev->aq_vec[i];
681+
682+ spin_lock(&aq->lock);
683+ if (atomic_read(&aq->admin_state) != IONIC_ADMIN_KILLED) {
684+ atomic_set(&aq->admin_state, IONIC_ADMIN_KILLED);
685+ /* Flush incomplete admin commands */
686+ ionic_admin_poll_locked(aq);
687+ do_flush = true;
688+ }
689+ spin_unlock(&aq->lock);
690+ }
691+
692+ if (do_flush)
693+ ionic_flush_qs(dev);
694+
695+ local_irq_restore(irqflags);
696+
697+ /* Post a fatal event if requested */
698+ if (fatal_path) {
699+ struct ib_event ev;
700+
701+ ev.device = &dev->ibdev;
702+ ev.element.port_num = 1;
703+ ev.event = IB_EVENT_DEVICE_FATAL;
704+
705+ ib_dispatch_event(&ev);
706+ }
707+
708+ atomic_set(&dev->admin_state, IONIC_ADMIN_KILLED);
709+}
710+
711+void ionic_kill_rdma_admin(struct ionic_ibdev *dev, bool fatal_path)
712+{
713+ enum ionic_admin_state old_state;
714+ unsigned long irqflags = 0;
715+ int i, rc;
716+
717+ if (!dev->aq_vec)
718+ return;
719+
720+ /*
721+ * Admin queues are transitioned from active to paused to killed state.
722+ * When in paused state, no new commands are issued to the device,
723+ * nor are any completed locally. After resetting the lif, it will be
724+ * safe to resume the rdma admin queues in the killed state. Commands
725+ * will not be issued to the device, but will complete locally with status
726+ * IONIC_ADMIN_KILLED. Handling completion will ensure that creating or
727+ * modifying resources fails, but destroying resources succeeds.
728+ * If there was a failure resetting the lif using this strategy,
729+ * then the state of the device is unknown.
730+ */
731+ old_state = atomic_cmpxchg(&dev->admin_state, IONIC_ADMIN_ACTIVE,
732+ IONIC_ADMIN_PAUSED);
733+ if (old_state != IONIC_ADMIN_ACTIVE)
734+ return;
735+
736+ /* Pause all the AQs */
737+ local_irq_save(irqflags);
738+ for (i = 0; i < dev->lif_cfg.aq_count; i++) {
739+ struct ionic_aq *aq = dev->aq_vec[i];
740+
741+ spin_lock(&aq->lock);
742+ /* pause rdma admin queues to reset lif */
743+ if (atomic_read(&aq->admin_state) == IONIC_ADMIN_ACTIVE)
744+ atomic_set(&aq->admin_state, IONIC_ADMIN_PAUSED);
745+ spin_unlock(&aq->lock);
746+ }
747+ local_irq_restore(irqflags);
748+
749+ rc = ionic_rdma_reset_devcmd(dev);
750+ if (unlikely(rc)) {
751+ ibdev_err(&dev->ibdev, "failed to reset rdma %d\n", rc);
752+ ionic_request_rdma_reset(dev->lif_cfg.lif);
753+ }
754+
755+ ionic_kill_ibdev(dev, fatal_path);
756+}
757+
758+static void ionic_reset_work(struct work_struct *ws)
759+{
760+ struct ionic_ibdev *dev =
761+ container_of(ws, struct ionic_ibdev, reset_work);
762+
763+ ionic_kill_rdma_admin(dev, true);
764+}
765+
766+static bool ionic_next_eqe(struct ionic_eq *eq, struct ionic_v1_eqe *eqe)
767+{
768+ struct ionic_v1_eqe *qeqe;
769+ bool color;
770+
771+ qeqe = ionic_queue_at_prod(&eq->q);
772+ color = ionic_v1_eqe_color(qeqe);
773+
774+ /* cons is color for eq */
775+ if (eq->q.cons != color)
776+ return false;
777+
778+ /* Prevent out-of-order reads of the EQE */
779+ dma_rmb();
780+
781+ ibdev_dbg(&eq->dev->ibdev, "poll eq prod %u\n", eq->q.prod);
782+ print_hex_dump_debug("eqe ", DUMP_PREFIX_OFFSET, 16, 1,
783+ qeqe, BIT(eq->q.stride_log2), true);
784+ *eqe = *qeqe;
785+
786+ return true;
787+}
788+
789+static void ionic_cq_event(struct ionic_ibdev *dev, u32 cqid, u8 code)
790+{
791+ unsigned long irqflags;
792+ struct ib_event ibev;
793+ struct ionic_cq *cq;
794+
795+ xa_lock_irqsave(&dev->cq_tbl, irqflags);
796+ cq = xa_load(&dev->cq_tbl, cqid);
797+ if (cq)
798+ kref_get(&cq->cq_kref);
799+ xa_unlock_irqrestore(&dev->cq_tbl, irqflags);
800+
801+ if (!cq) {
802+ ibdev_dbg(&dev->ibdev,
803+ "missing cqid %#x code %u\n", cqid, code);
804+ return;
805+ }
806+
807+ switch (code) {
808+ case IONIC_V1_EQE_CQ_NOTIFY:
809+ if (cq->vcq->ibcq.comp_handler)
810+ cq->vcq->ibcq.comp_handler(&cq->vcq->ibcq,
811+ cq->vcq->ibcq.cq_context);
812+ break;
813+
814+ case IONIC_V1_EQE_CQ_ERR:
815+ if (cq->vcq->ibcq.event_handler) {
816+ ibev.event = IB_EVENT_CQ_ERR;
817+ ibev.device = &dev->ibdev;
818+ ibev.element.cq = &cq->vcq->ibcq;
819+
820+ cq->vcq->ibcq.event_handler(&ibev,
821+ cq->vcq->ibcq.cq_context);
822+ }
823+ break;
824+
825+ default:
826+ ibdev_dbg(&dev->ibdev,
827+ "unrecognized cqid %#x code %u\n", cqid, code);
828+ break;
829+ }
830+
831+ kref_put(&cq->cq_kref, ionic_cq_complete);
832+}
833+
834+static void ionic_qp_event(struct ionic_ibdev *dev, u32 qpid, u8 code)
835+{
836+ unsigned long irqflags;
837+ struct ib_event ibev;
838+ struct ionic_qp *qp;
839+
840+ xa_lock_irqsave(&dev->qp_tbl, irqflags);
841+ qp = xa_load(&dev->qp_tbl, qpid);
842+ if (qp)
843+ kref_get(&qp->qp_kref);
844+ xa_unlock_irqrestore(&dev->qp_tbl, irqflags);
845+
846+ if (!qp) {
847+ ibdev_dbg(&dev->ibdev,
848+ "missing qpid %#x code %u\n", qpid, code);
849+ return;
850+ }
851+
852+ ibev.device = &dev->ibdev;
853+ ibev.element.qp = &qp->ibqp;
854+
855+ switch (code) {
856+ case IONIC_V1_EQE_SQ_DRAIN:
857+ ibev.event = IB_EVENT_SQ_DRAINED;
858+ break;
859+
860+ case IONIC_V1_EQE_QP_COMM_EST:
861+ ibev.event = IB_EVENT_COMM_EST;
862+ break;
863+
864+ case IONIC_V1_EQE_QP_LAST_WQE:
865+ ibev.event = IB_EVENT_QP_LAST_WQE_REACHED;
866+ break;
867+
868+ case IONIC_V1_EQE_QP_ERR:
869+ ibev.event = IB_EVENT_QP_FATAL;
870+ break;
871+
872+ case IONIC_V1_EQE_QP_ERR_REQUEST:
873+ ibev.event = IB_EVENT_QP_REQ_ERR;
874+ break;
875+
876+ case IONIC_V1_EQE_QP_ERR_ACCESS:
877+ ibev.event = IB_EVENT_QP_ACCESS_ERR;
878+ break;
879+
880+ default:
881+ ibdev_dbg(&dev->ibdev,
882+ "unrecognized qpid %#x code %u\n", qpid, code);
883+ goto out;
884+ }
885+
886+ if (qp->ibqp.event_handler)
887+ qp->ibqp.event_handler(&ibev, qp->ibqp.qp_context);
888+
889+out:
890+ kref_put(&qp->qp_kref, ionic_qp_complete);
891+}
892+
893+static u16 ionic_poll_eq(struct ionic_eq *eq, u16 budget)
894+{
895+ struct ionic_ibdev *dev = eq->dev;
896+ struct ionic_v1_eqe eqe;
897+ u16 npolled = 0;
898+ u8 type, code;
899+ u32 evt, qid;
900+
901+ while (npolled < budget) {
902+ if (!ionic_next_eqe(eq, &eqe))
903+ break;
904+
905+ ionic_queue_produce(&eq->q);
906+
907+ /* cons is color for eq */
908+ eq->q.cons = ionic_color_wrap(eq->q.prod, eq->q.cons);
909+
910+ ++npolled;
911+
912+ evt = ionic_v1_eqe_evt(&eqe);
913+ type = ionic_v1_eqe_evt_type(evt);
914+ code = ionic_v1_eqe_evt_code(evt);
915+ qid = ionic_v1_eqe_evt_qid(evt);
916+
917+ switch (type) {
918+ case IONIC_V1_EQE_TYPE_CQ:
919+ ionic_cq_event(dev, qid, code);
920+ break;
921+
922+ case IONIC_V1_EQE_TYPE_QP:
923+ ionic_qp_event(dev, qid, code);
924+ break;
925+
926+ default:
927+ ibdev_dbg(&dev->ibdev,
928+ "unknown event %#x type %u\n", evt, type);
929+ }
930+ }
931+
932+ return npolled;
933+}
934+
935+static void ionic_poll_eq_work(struct work_struct *work)
936+{
937+ struct ionic_eq *eq = container_of(work, struct ionic_eq, work);
938+ u32 npolled;
939+
940+ if (unlikely(!eq->enable) || WARN_ON(eq->armed))
941+ return;
942+
943+ npolled = ionic_poll_eq(eq, IONIC_EQ_WORK_BUDGET);
944+ if (npolled == IONIC_EQ_WORK_BUDGET) {
945+ ionic_intr_credits(eq->dev->lif_cfg.intr_ctrl, eq->intr,
946+ npolled, 0);
947+ queue_work(ionic_evt_workq, &eq->work);
948+ } else {
949+ xchg(&eq->armed, 1);
950+ ionic_intr_credits(eq->dev->lif_cfg.intr_ctrl, eq->intr,
951+ 0, IONIC_INTR_CRED_UNMASK);
952+ }
953+}
954+
955+static irqreturn_t ionic_poll_eq_isr(int irq, void *eqptr)
956+{
957+ struct ionic_eq *eq = eqptr;
958+ int was_armed;
959+ u32 npolled;
960+
961+ was_armed = xchg(&eq->armed, 0);
962+
963+ if (unlikely(!eq->enable) || !was_armed)
964+ return IRQ_HANDLED;
965+
966+ npolled = ionic_poll_eq(eq, IONIC_EQ_ISR_BUDGET);
967+ if (npolled == IONIC_EQ_ISR_BUDGET) {
968+ ionic_intr_credits(eq->dev->lif_cfg.intr_ctrl, eq->intr,
969+ npolled, 0);
970+ queue_work(ionic_evt_workq, &eq->work);
971+ } else {
972+ xchg(&eq->armed, 1);
973+ ionic_intr_credits(eq->dev->lif_cfg.intr_ctrl, eq->intr,
974+ 0, IONIC_INTR_CRED_UNMASK);
975+ }
976+
977+ return IRQ_HANDLED;
978+}
979+
980+static struct ionic_eq *ionic_create_eq(struct ionic_ibdev *dev, int eqid)
981+{
982+ struct ionic_intr_info intr_obj = { };
983+ struct ionic_eq *eq;
984+ int rc;
985+
986+ eq = kzalloc(sizeof(*eq), GFP_KERNEL);
987+ if (!eq)
988+ return ERR_PTR(-ENOMEM);
989+
990+ eq->dev = dev;
991+
992+ rc = ionic_queue_init(&eq->q, dev->lif_cfg.hwdev, IONIC_EQ_DEPTH,
993+ sizeof(struct ionic_v1_eqe));
994+ if (rc)
995+ goto err_q;
996+
997+ eq->eqid = eqid;
998+
999+ eq->armed = true;
1000+ eq->enable = false;
1001+ INIT_WORK(&eq->work, ionic_poll_eq_work);
1002+
1003+ rc = ionic_intr_alloc(dev->lif_cfg.lif, &intr_obj);
1004+ if (rc < 0)
1005+ goto err_intr;
1006+
1007+ eq->irq = intr_obj.vector;
1008+ eq->intr = intr_obj.index;
1009+
1010+ ionic_queue_dbell_init(&eq->q, eq->eqid);
1011+
1012+ /* cons is color for eq */
1013+ eq->q.cons = true;
1014+
1015+ snprintf(eq->name, sizeof(eq->name), "%s-%d-%d-eq",
1016+ "ionr", dev->lif_cfg.lif_index, eq->eqid);
1017+
1018+ ionic_intr_mask(dev->lif_cfg.intr_ctrl, eq->intr, IONIC_INTR_MASK_SET);
1019+ ionic_intr_mask_assert(dev->lif_cfg.intr_ctrl, eq->intr, IONIC_INTR_MASK_SET);
1020+ ionic_intr_coal_init(dev->lif_cfg.intr_ctrl, eq->intr, 0);
1021+ ionic_intr_clean(dev->lif_cfg.intr_ctrl, eq->intr);
1022+
1023+ eq->enable = true;
1024+
1025+ rc = request_irq(eq->irq, ionic_poll_eq_isr, 0, eq->name, eq);
1026+ if (rc)
1027+ goto err_irq;
1028+
1029+ rc = ionic_rdma_queue_devcmd(dev, &eq->q, eq->eqid, eq->intr,
1030+ IONIC_CMD_RDMA_CREATE_EQ);
1031+ if (rc)
1032+ goto err_cmd;
1033+
1034+ ionic_intr_mask(dev->lif_cfg.intr_ctrl, eq->intr, IONIC_INTR_MASK_CLEAR);
1035+
1036+ return eq;
1037+
1038+err_cmd:
1039+ eq->enable = false;
1040+ free_irq(eq->irq, eq);
1041+ flush_work(&eq->work);
1042+err_irq:
1043+ ionic_intr_free(dev->lif_cfg.lif, eq->intr);
1044+err_intr:
1045+ ionic_queue_destroy(&eq->q, dev->lif_cfg.hwdev);
1046+err_q:
1047+ kfree(eq);
1048+
1049+ return ERR_PTR(rc);
1050+}
1051+
1052+static void ionic_destroy_eq(struct ionic_eq *eq)
1053+{
1054+ struct ionic_ibdev *dev = eq->dev;
1055+
1056+ eq->enable = false;
1057+ free_irq(eq->irq, eq);
1058+ flush_work(&eq->work);
1059+
1060+ ionic_intr_free(dev->lif_cfg.lif, eq->intr);
1061+ ionic_queue_destroy(&eq->q, dev->lif_cfg.hwdev);
1062+ kfree(eq);
1063+}
1064+
1065+int ionic_create_rdma_admin(struct ionic_ibdev *dev)
1066+{
1067+ int eq_i = 0, aq_i = 0, rc = 0;
1068+ struct ionic_vcq *vcq;
1069+ struct ionic_aq *aq;
1070+ struct ionic_eq *eq;
1071+
1072+ dev->eq_vec = NULL;
1073+ dev->aq_vec = NULL;
1074+
1075+ INIT_WORK(&dev->reset_work, ionic_reset_work);
1076+ INIT_DELAYED_WORK(&dev->admin_dwork, ionic_admin_dwork);
1077+ atomic_set(&dev->admin_state, IONIC_ADMIN_KILLED);
1078+
1079+ if (dev->lif_cfg.aq_count > IONIC_AQ_COUNT) {
1080+ ibdev_dbg(&dev->ibdev, "limiting adminq count to %d\n",
1081+ IONIC_AQ_COUNT);
1082+ dev->lif_cfg.aq_count = IONIC_AQ_COUNT;
1083+ }
1084+
1085+ if (dev->lif_cfg.eq_count > IONIC_EQ_COUNT) {
1086+ dev_dbg(&dev->ibdev.dev, "limiting eventq count to %d\n",
1087+ IONIC_EQ_COUNT);
1088+ dev->lif_cfg.eq_count = IONIC_EQ_COUNT;
1089+ }
1090+
1091+ /* need at least two eq and one aq */
1092+ if (dev->lif_cfg.eq_count < IONIC_EQ_COUNT_MIN ||
1093+ dev->lif_cfg.aq_count < IONIC_AQ_COUNT_MIN) {
1094+ rc = -EINVAL;
1095+ goto out;
1096+ }
1097+
1098+ dev->eq_vec = kmalloc_array(dev->lif_cfg.eq_count, sizeof(*dev->eq_vec),
1099+ GFP_KERNEL);
1100+ if (!dev->eq_vec) {
1101+ rc = -ENOMEM;
1102+ goto out;
1103+ }
1104+
1105+ for (eq_i = 0; eq_i < dev->lif_cfg.eq_count; ++eq_i) {
1106+ eq = ionic_create_eq(dev, eq_i + dev->lif_cfg.eq_base);
1107+ if (IS_ERR(eq)) {
1108+ rc = PTR_ERR(eq);
1109+
1110+ if (eq_i < IONIC_EQ_COUNT_MIN) {
1111+ ibdev_err(&dev->ibdev,
1112+ "fail create eq %pe\n", eq);
1113+ goto out;
1114+ }
1115+
1116+ /* ok, just fewer eq than device supports */
1117+ ibdev_dbg(&dev->ibdev, "eq count %d want %d rc %pe\n",
1118+ eq_i, dev->lif_cfg.eq_count, eq);
1119+
1120+ rc = 0;
1121+ break;
1122+ }
1123+
1124+ dev->eq_vec[eq_i] = eq;
1125+ }
1126+
1127+ dev->lif_cfg.eq_count = eq_i;
1128+
1129+ dev->aq_vec = kmalloc_array(dev->lif_cfg.aq_count, sizeof(*dev->aq_vec),
1130+ GFP_KERNEL);
1131+ if (!dev->aq_vec) {
1132+ rc = -ENOMEM;
1133+ goto out;
1134+ }
1135+
1136+ /* Create one CQ per AQ */
1137+ for (aq_i = 0; aq_i < dev->lif_cfg.aq_count; ++aq_i) {
1138+ vcq = ionic_create_rdma_admincq(dev, aq_i % eq_i);
1139+ if (IS_ERR(vcq)) {
1140+ rc = PTR_ERR(vcq);
1141+
1142+ if (!aq_i) {
1143+ ibdev_err(&dev->ibdev,
1144+ "failed to create acq %pe\n", vcq);
1145+ goto out;
1146+ }
1147+
1148+ /* ok, just fewer adminq than device supports */
1149+ ibdev_dbg(&dev->ibdev, "acq count %d want %d rc %pe\n",
1150+ aq_i, dev->lif_cfg.aq_count, vcq);
1151+ break;
1152+ }
1153+
1154+ aq = ionic_create_rdma_adminq(dev, aq_i + dev->lif_cfg.aq_base,
1155+ vcq->cq[0].cqid);
1156+ if (IS_ERR(aq)) {
1157+ /* Clean up the dangling CQ */
1158+ ionic_destroy_cq_common(dev, &vcq->cq[0]);
1159+ kfree(vcq);
1160+
1161+ rc = PTR_ERR(aq);
1162+
1163+ if (!aq_i) {
1164+ ibdev_err(&dev->ibdev,
1165+ "failed to create aq %pe\n", aq);
1166+ goto out;
1167+ }
1168+
1169+ /* ok, just fewer adminq than device supports */
1170+ ibdev_dbg(&dev->ibdev, "aq count %d want %d rc %pe\n",
1171+ aq_i, dev->lif_cfg.aq_count, aq);
1172+ break;
1173+ }
1174+
1175+ vcq->ibcq.cq_context = aq;
1176+ aq->vcq = vcq;
1177+
1178+ atomic_set(&aq->admin_state, IONIC_ADMIN_ACTIVE);
1179+ dev->aq_vec[aq_i] = aq;
1180+ }
1181+
1182+ atomic_set(&dev->admin_state, IONIC_ADMIN_ACTIVE);
1183+out:
1184+ dev->lif_cfg.eq_count = eq_i;
1185+ dev->lif_cfg.aq_count = aq_i;
1186+
1187+ return rc;
1188+}
1189+
1190+void ionic_destroy_rdma_admin(struct ionic_ibdev *dev)
1191+{
1192+ struct ionic_vcq *vcq;
1193+ struct ionic_aq *aq;
1194+ struct ionic_eq *eq;
1195+
1196+ /*
1197+ * Killing the admin before destroy makes sure all admin and
1198+ * completions are flushed. admin_state = IONIC_ADMIN_KILLED
1199+ * stops queueing up further works.
1200+ */
1201+ cancel_delayed_work_sync(&dev->admin_dwork);
1202+ cancel_work_sync(&dev->reset_work);
1203+
1204+ if (dev->aq_vec) {
1205+ while (dev->lif_cfg.aq_count > 0) {
1206+ aq = dev->aq_vec[--dev->lif_cfg.aq_count];
1207+ vcq = aq->vcq;
1208+
1209+ cancel_work_sync(&aq->work);
1210+
1211+ __ionic_destroy_rdma_adminq(dev, aq);
1212+ if (vcq) {
1213+ ionic_destroy_cq_common(dev, &vcq->cq[0]);
1214+ kfree(vcq);
1215+ }
1216+ }
1217+
1218+ kfree(dev->aq_vec);
1219+ }
1220+
1221+ if (dev->eq_vec) {
1222+ while (dev->lif_cfg.eq_count > 0) {
1223+ eq = dev->eq_vec[--dev->lif_cfg.eq_count];
1224+ ionic_destroy_eq(eq);
1225+ }
1226+
1227+ kfree(dev->eq_vec);
1228+ }
1229+}
drivers/infiniband/hw/ionic/ionic_controlpath.c+2679−0
@@ -0,0 +1,2679 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#include <linux/module.h>
5+#include <linux/printk.h>
6+#include <rdma/ib_addr.h>
7+#include <rdma/ib_cache.h>
8+#include <rdma/ib_user_verbs.h>
9+#include <ionic_api.h>
10+
11+#include "ionic_fw.h"
12+#include "ionic_ibdev.h"
13+
14+#define ionic_set_ecn(tos) (((tos) | 2u) & ~1u)
15+#define ionic_clear_ecn(tos) ((tos) & ~3u)
16+
17+static int ionic_validate_qdesc(struct ionic_qdesc *q)
18+{
19+ if (!q->addr || !q->size || !q->mask ||
20+ !q->depth_log2 || !q->stride_log2)
21+ return -EINVAL;
22+
23+ if (q->addr & (PAGE_SIZE - 1))
24+ return -EINVAL;
25+
26+ if (q->mask != BIT(q->depth_log2) - 1)
27+ return -EINVAL;
28+
29+ if (q->size < BIT_ULL(q->depth_log2 + q->stride_log2))
30+ return -EINVAL;
31+
32+ return 0;
33+}
34+
35+static u32 ionic_get_eqid(struct ionic_ibdev *dev, u32 comp_vector, u8 udma_idx)
36+{
37+ /* EQ per vector per udma, and the first eqs reserved for async events.
38+ * The rest of the vectors can be requested for completions.
39+ */
40+ u32 comp_vec_count = dev->lif_cfg.eq_count / dev->lif_cfg.udma_count - 1;
41+
42+ return (comp_vector % comp_vec_count + 1) * dev->lif_cfg.udma_count + udma_idx;
43+}
44+
45+static int ionic_get_cqid(struct ionic_ibdev *dev, u32 *cqid, u8 udma_idx)
46+{
47+ unsigned int size, base, bound;
48+ int rc;
49+
50+ size = dev->lif_cfg.cq_count / dev->lif_cfg.udma_count;
51+ base = size * udma_idx;
52+ bound = base + size;
53+
54+ rc = ionic_resid_get_shared(&dev->inuse_cqid, base, bound);
55+ if (rc >= 0) {
56+ /* cq_base is zero or a multiple of two queue groups */
57+ *cqid = dev->lif_cfg.cq_base +
58+ ionic_bitid_to_qid(rc, dev->lif_cfg.udma_qgrp_shift,
59+ dev->half_cqid_udma_shift);
60+
61+ rc = 0;
62+ }
63+
64+ return rc;
65+}
66+
67+static void ionic_put_cqid(struct ionic_ibdev *dev, u32 cqid)
68+{
69+ u32 bitid = ionic_qid_to_bitid(cqid - dev->lif_cfg.cq_base,
70+ dev->lif_cfg.udma_qgrp_shift,
71+ dev->half_cqid_udma_shift);
72+
73+ ionic_resid_put(&dev->inuse_cqid, bitid);
74+}
75+
76+int ionic_create_cq_common(struct ionic_vcq *vcq,
77+ struct ionic_tbl_buf *buf,
78+ const struct ib_cq_init_attr *attr,
79+ struct ionic_ctx *ctx,
80+ struct ib_udata *udata,
81+ struct ionic_qdesc *req_cq,
82+ __u32 *resp_cqid,
83+ int udma_idx)
84+{
85+ struct ionic_ibdev *dev = to_ionic_ibdev(vcq->ibcq.device);
86+ struct ionic_cq *cq = &vcq->cq[udma_idx];
87+ void *entry;
88+ int rc;
89+
90+ cq->vcq = vcq;
91+
92+ if (attr->cqe < 1 || attr->cqe + IONIC_CQ_GRACE > 0xffff) {
93+ rc = -EINVAL;
94+ goto err_args;
95+ }
96+
97+ rc = ionic_get_cqid(dev, &cq->cqid, udma_idx);
98+ if (rc)
99+ goto err_args;
100+
101+ cq->eqid = ionic_get_eqid(dev, attr->comp_vector, udma_idx);
102+
103+ spin_lock_init(&cq->lock);
104+ INIT_LIST_HEAD(&cq->poll_sq);
105+ INIT_LIST_HEAD(&cq->flush_sq);
106+ INIT_LIST_HEAD(&cq->flush_rq);
107+
108+ if (udata) {
109+ rc = ionic_validate_qdesc(req_cq);
110+ if (rc)
111+ goto err_qdesc;
112+
113+ cq->umem = ib_umem_get(&dev->ibdev, req_cq->addr, req_cq->size,
114+ IB_ACCESS_LOCAL_WRITE);
115+ if (IS_ERR(cq->umem)) {
116+ rc = PTR_ERR(cq->umem);
117+ goto err_qdesc;
118+ }
119+
120+ cq->q.ptr = NULL;
121+ cq->q.size = req_cq->size;
122+ cq->q.mask = req_cq->mask;
123+ cq->q.depth_log2 = req_cq->depth_log2;
124+ cq->q.stride_log2 = req_cq->stride_log2;
125+
126+ *resp_cqid = cq->cqid;
127+ } else {
128+ rc = ionic_queue_init(&cq->q, dev->lif_cfg.hwdev,
129+ attr->cqe + IONIC_CQ_GRACE,
130+ sizeof(struct ionic_v1_cqe));
131+ if (rc)
132+ goto err_q_init;
133+
134+ ionic_queue_dbell_init(&cq->q, cq->cqid);
135+ cq->color = true;
136+ cq->credit = cq->q.mask;
137+ }
138+
139+ rc = ionic_pgtbl_init(dev, buf, cq->umem, cq->q.dma, 1, PAGE_SIZE);
140+ if (rc)
141+ goto err_pgtbl_init;
142+
143+ init_completion(&cq->cq_rel_comp);
144+ kref_init(&cq->cq_kref);
145+
146+ entry = xa_store_irq(&dev->cq_tbl, cq->cqid, cq, GFP_KERNEL);
147+ if (entry) {
148+ if (!xa_is_err(entry))
149+ rc = -EINVAL;
150+ else
151+ rc = xa_err(entry);
152+
153+ goto err_xa;
154+ }
155+
156+ return 0;
157+
158+err_xa:
159+ ionic_pgtbl_unbuf(dev, buf);
160+err_pgtbl_init:
161+ if (!udata)
162+ ionic_queue_destroy(&cq->q, dev->lif_cfg.hwdev);
163+err_q_init:
164+ if (cq->umem)
165+ ib_umem_release(cq->umem);
166+err_qdesc:
167+ ionic_put_cqid(dev, cq->cqid);
168+err_args:
169+ cq->vcq = NULL;
170+
171+ return rc;
172+}
173+
174+void ionic_destroy_cq_common(struct ionic_ibdev *dev, struct ionic_cq *cq)
175+{
176+ if (!cq->vcq)
177+ return;
178+
179+ xa_erase_irq(&dev->cq_tbl, cq->cqid);
180+
181+ kref_put(&cq->cq_kref, ionic_cq_complete);
182+ wait_for_completion(&cq->cq_rel_comp);
183+
184+ if (cq->umem)
185+ ib_umem_release(cq->umem);
186+ else
187+ ionic_queue_destroy(&cq->q, dev->lif_cfg.hwdev);
188+
189+ ionic_put_cqid(dev, cq->cqid);
190+
191+ cq->vcq = NULL;
192+}
193+
194+static int ionic_validate_qdesc_zero(struct ionic_qdesc *q)
195+{
196+ if (q->addr || q->size || q->mask || q->depth_log2 || q->stride_log2)
197+ return -EINVAL;
198+
199+ return 0;
200+}
201+
202+static int ionic_get_pdid(struct ionic_ibdev *dev, u32 *pdid)
203+{
204+ int rc;
205+
206+ rc = ionic_resid_get(&dev->inuse_pdid);
207+ if (rc < 0)
208+ return rc;
209+
210+ *pdid = rc;
211+ return 0;
212+}
213+
214+static int ionic_get_ahid(struct ionic_ibdev *dev, u32 *ahid)
215+{
216+ int rc;
217+
218+ rc = ionic_resid_get(&dev->inuse_ahid);
219+ if (rc < 0)
220+ return rc;
221+
222+ *ahid = rc;
223+ return 0;
224+}
225+
226+static int ionic_get_mrid(struct ionic_ibdev *dev, u32 *mrid)
227+{
228+ int rc;
229+
230+ /* wrap to 1, skip reserved lkey */
231+ rc = ionic_resid_get_shared(&dev->inuse_mrid, 1,
232+ dev->inuse_mrid.inuse_size);
233+ if (rc < 0)
234+ return rc;
235+
236+ *mrid = ionic_mrid(rc, dev->next_mrkey++);
237+ return 0;
238+}
239+
240+static int ionic_get_gsi_qpid(struct ionic_ibdev *dev, u32 *qpid)
241+{
242+ int rc = 0;
243+
244+ rc = ionic_resid_get_shared(&dev->inuse_qpid, IB_QPT_GSI, IB_QPT_GSI + 1);
245+ if (rc < 0)
246+ return rc;
247+
248+ *qpid = IB_QPT_GSI;
249+ return 0;
250+}
251+
252+static int ionic_get_qpid(struct ionic_ibdev *dev, u32 *qpid,
253+ u8 *udma_idx, u8 udma_mask)
254+{
255+ unsigned int size, base, bound;
256+ int udma_i, udma_x, udma_ix;
257+ int rc = -EINVAL;
258+
259+ udma_x = dev->next_qpid_udma_idx;
260+
261+ dev->next_qpid_udma_idx ^= dev->lif_cfg.udma_count - 1;
262+
263+ for (udma_i = 0; udma_i < dev->lif_cfg.udma_count; ++udma_i) {
264+ udma_ix = udma_i ^ udma_x;
265+
266+ if (!(udma_mask & BIT(udma_ix)))
267+ continue;
268+
269+ size = dev->lif_cfg.qp_count / dev->lif_cfg.udma_count;
270+ base = size * udma_ix;
271+ bound = base + size;
272+
273+ /* skip reserved SMI and GSI qpids in group zero */
274+ if (!base)
275+ base = 2;
276+
277+ rc = ionic_resid_get_shared(&dev->inuse_qpid, base, bound);
278+ if (rc >= 0) {
279+ *qpid = ionic_bitid_to_qid(rc,
280+ dev->lif_cfg.udma_qgrp_shift,
281+ dev->half_qpid_udma_shift);
282+ *udma_idx = udma_ix;
283+
284+ rc = 0;
285+ break;
286+ }
287+ }
288+
289+ return rc;
290+}
291+
292+static int ionic_get_dbid(struct ionic_ibdev *dev, u32 *dbid, phys_addr_t *addr)
293+{
294+ int rc, dbpage_num;
295+
296+ /* wrap to 1, skip kernel reserved */
297+ rc = ionic_resid_get_shared(&dev->inuse_dbid, 1,
298+ dev->inuse_dbid.inuse_size);
299+ if (rc < 0)
300+ return rc;
301+
302+ dbpage_num = (dev->lif_cfg.lif_hw_index * dev->lif_cfg.dbid_count) + rc;
303+ *addr = dev->lif_cfg.db_phys + ((phys_addr_t)dbpage_num << PAGE_SHIFT);
304+
305+ *dbid = rc;
306+
307+ return 0;
308+}
309+
310+static void ionic_put_pdid(struct ionic_ibdev *dev, u32 pdid)
311+{
312+ ionic_resid_put(&dev->inuse_pdid, pdid);
313+}
314+
315+static void ionic_put_ahid(struct ionic_ibdev *dev, u32 ahid)
316+{
317+ ionic_resid_put(&dev->inuse_ahid, ahid);
318+}
319+
320+static void ionic_put_mrid(struct ionic_ibdev *dev, u32 mrid)
321+{
322+ ionic_resid_put(&dev->inuse_mrid, ionic_mrid_index(mrid));
323+}
324+
325+static void ionic_put_qpid(struct ionic_ibdev *dev, u32 qpid)
326+{
327+ u32 bitid = ionic_qid_to_bitid(qpid,
328+ dev->lif_cfg.udma_qgrp_shift,
329+ dev->half_qpid_udma_shift);
330+
331+ ionic_resid_put(&dev->inuse_qpid, bitid);
332+}
333+
334+static void ionic_put_dbid(struct ionic_ibdev *dev, u32 dbid)
335+{
336+ ionic_resid_put(&dev->inuse_dbid, dbid);
337+}
338+
339+static struct rdma_user_mmap_entry*
340+ionic_mmap_entry_insert(struct ionic_ctx *ctx, unsigned long size,
341+ unsigned long pfn, u8 mmap_flags, u64 *offset)
342+{
343+ struct ionic_mmap_entry *entry;
344+ int rc;
345+
346+ entry = kzalloc(sizeof(*entry), GFP_KERNEL);
347+ if (!entry)
348+ return NULL;
349+
350+ entry->size = size;
351+ entry->pfn = pfn;
352+ entry->mmap_flags = mmap_flags;
353+
354+ rc = rdma_user_mmap_entry_insert(&ctx->ibctx, &entry->rdma_entry,
355+ entry->size);
356+ if (rc) {
357+ kfree(entry);
358+ return NULL;
359+ }
360+
361+ if (offset)
362+ *offset = rdma_user_mmap_get_offset(&entry->rdma_entry);
363+
364+ return &entry->rdma_entry;
365+}
366+
367+int ionic_alloc_ucontext(struct ib_ucontext *ibctx, struct ib_udata *udata)
368+{
369+ struct ionic_ibdev *dev = to_ionic_ibdev(ibctx->device);
370+ struct ionic_ctx *ctx = to_ionic_ctx(ibctx);
371+ struct ionic_ctx_resp resp = {};
372+ struct ionic_ctx_req req;
373+ phys_addr_t db_phys = 0;
374+ int rc;
375+
376+ rc = ib_copy_from_udata(&req, udata, sizeof(req));
377+ if (rc)
378+ return rc;
379+
380+ /* try to allocate dbid for user ctx */
381+ rc = ionic_get_dbid(dev, &ctx->dbid, &db_phys);
382+ if (rc < 0)
383+ return rc;
384+
385+ ibdev_dbg(&dev->ibdev, "user space dbid %u\n", ctx->dbid);
386+
387+ ctx->mmap_dbell = ionic_mmap_entry_insert(ctx, PAGE_SIZE,
388+ PHYS_PFN(db_phys), 0, NULL);
389+ if (!ctx->mmap_dbell) {
390+ rc = -ENOMEM;
391+ goto err_mmap_dbell;
392+ }
393+
394+ resp.page_shift = PAGE_SHIFT;
395+
396+ resp.dbell_offset = db_phys & ~PAGE_MASK;
397+
398+ resp.version = dev->lif_cfg.rdma_version;
399+ resp.qp_opcodes = dev->lif_cfg.qp_opcodes;
400+ resp.admin_opcodes = dev->lif_cfg.admin_opcodes;
401+
402+ resp.sq_qtype = dev->lif_cfg.sq_qtype;
403+ resp.rq_qtype = dev->lif_cfg.rq_qtype;
404+ resp.cq_qtype = dev->lif_cfg.cq_qtype;
405+ resp.admin_qtype = dev->lif_cfg.aq_qtype;
406+ resp.max_stride = dev->lif_cfg.max_stride;
407+ resp.max_spec = IONIC_SPEC_HIGH;
408+
409+ resp.udma_count = dev->lif_cfg.udma_count;
410+ resp.expdb_mask = dev->lif_cfg.expdb_mask;
411+
412+ if (dev->lif_cfg.sq_expdb)
413+ resp.expdb_qtypes |= IONIC_EXPDB_SQ;
414+ if (dev->lif_cfg.rq_expdb)
415+ resp.expdb_qtypes |= IONIC_EXPDB_RQ;
416+
417+ rc = ib_copy_to_udata(udata, &resp, sizeof(resp));
418+ if (rc)
419+ goto err_resp;
420+
421+ return 0;
422+
423+err_resp:
424+ rdma_user_mmap_entry_remove(ctx->mmap_dbell);
425+err_mmap_dbell:
426+ ionic_put_dbid(dev, ctx->dbid);
427+
428+ return rc;
429+}
430+
431+void ionic_dealloc_ucontext(struct ib_ucontext *ibctx)
432+{
433+ struct ionic_ibdev *dev = to_ionic_ibdev(ibctx->device);
434+ struct ionic_ctx *ctx = to_ionic_ctx(ibctx);
435+
436+ rdma_user_mmap_entry_remove(ctx->mmap_dbell);
437+ ionic_put_dbid(dev, ctx->dbid);
438+}
439+
440+int ionic_mmap(struct ib_ucontext *ibctx, struct vm_area_struct *vma)
441+{
442+ struct ionic_ibdev *dev = to_ionic_ibdev(ibctx->device);
443+ struct ionic_ctx *ctx = to_ionic_ctx(ibctx);
444+ struct rdma_user_mmap_entry *rdma_entry;
445+ struct ionic_mmap_entry *ionic_entry;
446+ int rc = 0;
447+
448+ rdma_entry = rdma_user_mmap_entry_get(&ctx->ibctx, vma);
449+ if (!rdma_entry) {
450+ ibdev_dbg(&dev->ibdev, "not found %#lx\n",
451+ vma->vm_pgoff << PAGE_SHIFT);
452+ return -EINVAL;
453+ }
454+
455+ ionic_entry = container_of(rdma_entry, struct ionic_mmap_entry,
456+ rdma_entry);
457+
458+ ibdev_dbg(&dev->ibdev, "writecombine? %d\n",
459+ ionic_entry->mmap_flags & IONIC_MMAP_WC);
460+ if (ionic_entry->mmap_flags & IONIC_MMAP_WC)
461+ vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
462+ else
463+ vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
464+
465+ ibdev_dbg(&dev->ibdev, "remap st %#lx pf %#lx sz %#lx\n",
466+ vma->vm_start, ionic_entry->pfn, ionic_entry->size);
467+ rc = rdma_user_mmap_io(&ctx->ibctx, vma, ionic_entry->pfn,
468+ ionic_entry->size, vma->vm_page_prot,
469+ rdma_entry);
470+ if (rc)
471+ ibdev_dbg(&dev->ibdev, "remap failed %d\n", rc);
472+
473+ rdma_user_mmap_entry_put(rdma_entry);
474+ return rc;
475+}
476+
477+void ionic_mmap_free(struct rdma_user_mmap_entry *rdma_entry)
478+{
479+ struct ionic_mmap_entry *ionic_entry;
480+
481+ ionic_entry = container_of(rdma_entry, struct ionic_mmap_entry,
482+ rdma_entry);
483+ kfree(ionic_entry);
484+}
485+
486+int ionic_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
487+{
488+ struct ionic_ibdev *dev = to_ionic_ibdev(ibpd->device);
489+ struct ionic_pd *pd = to_ionic_pd(ibpd);
490+
491+ return ionic_get_pdid(dev, &pd->pdid);
492+}
493+
494+int ionic_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
495+{
496+ struct ionic_ibdev *dev = to_ionic_ibdev(ibpd->device);
497+ struct ionic_pd *pd = to_ionic_pd(ibpd);
498+
499+ ionic_put_pdid(dev, pd->pdid);
500+
501+ return 0;
502+}
503+
504+static int ionic_build_hdr(struct ionic_ibdev *dev,
505+ struct ib_ud_header *hdr,
506+ const struct rdma_ah_attr *attr,
507+ u16 sport, bool want_ecn)
508+{
509+ const struct ib_global_route *grh;
510+ enum rdma_network_type net;
511+ u16 vlan;
512+ int rc;
513+
514+ if (attr->ah_flags != IB_AH_GRH)
515+ return -EINVAL;
516+ if (attr->type != RDMA_AH_ATTR_TYPE_ROCE)
517+ return -EINVAL;
518+
519+ grh = rdma_ah_read_grh(attr);
520+
521+ rc = rdma_read_gid_l2_fields(grh->sgid_attr, &vlan, &hdr->eth.smac_h[0]);
522+ if (rc)
523+ return rc;
524+
525+ net = rdma_gid_attr_network_type(grh->sgid_attr);
526+
527+ rc = ib_ud_header_init(0, /* no payload */
528+ 0, /* no lrh */
529+ 1, /* yes eth */
530+ vlan != 0xffff,
531+ 0, /* no grh */
532+ net == RDMA_NETWORK_IPV4 ? 4 : 6,
533+ 1, /* yes udp */
534+ 0, /* no imm */
535+ hdr);
536+ if (rc)
537+ return rc;
538+
539+ ether_addr_copy(hdr->eth.dmac_h, attr->roce.dmac);
540+
541+ if (net == RDMA_NETWORK_IPV4) {
542+ hdr->eth.type = cpu_to_be16(ETH_P_IP);
543+ hdr->ip4.frag_off = cpu_to_be16(0x4000); /* don't fragment */
544+ hdr->ip4.ttl = grh->hop_limit;
545+ hdr->ip4.tot_len = cpu_to_be16(0xffff);
546+ hdr->ip4.saddr =
547+ *(const __be32 *)(grh->sgid_attr->gid.raw + 12);
548+ hdr->ip4.daddr = *(const __be32 *)(grh->dgid.raw + 12);
549+
550+ if (want_ecn)
551+ hdr->ip4.tos = ionic_set_ecn(grh->traffic_class);
552+ else
553+ hdr->ip4.tos = ionic_clear_ecn(grh->traffic_class);
554+ } else {
555+ hdr->eth.type = cpu_to_be16(ETH_P_IPV6);
556+ hdr->grh.flow_label = cpu_to_be32(grh->flow_label);
557+ hdr->grh.hop_limit = grh->hop_limit;
558+ hdr->grh.source_gid = grh->sgid_attr->gid;
559+ hdr->grh.destination_gid = grh->dgid;
560+
561+ if (want_ecn)
562+ hdr->grh.traffic_class =
563+ ionic_set_ecn(grh->traffic_class);
564+ else
565+ hdr->grh.traffic_class =
566+ ionic_clear_ecn(grh->traffic_class);
567+ }
568+
569+ if (vlan != 0xffff) {
570+ vlan |= rdma_ah_get_sl(attr) << VLAN_PRIO_SHIFT;
571+ hdr->vlan.tag = cpu_to_be16(vlan);
572+ hdr->vlan.type = hdr->eth.type;
573+ hdr->eth.type = cpu_to_be16(ETH_P_8021Q);
574+ }
575+
576+ hdr->udp.sport = cpu_to_be16(sport);
577+ hdr->udp.dport = cpu_to_be16(ROCE_V2_UDP_DPORT);
578+
579+ return 0;
580+}
581+
582+static void ionic_set_ah_attr(struct ionic_ibdev *dev,
583+ struct rdma_ah_attr *ah_attr,
584+ struct ib_ud_header *hdr,
585+ int sgid_index)
586+{
587+ u32 flow_label;
588+ u16 vlan = 0;
589+ u8 tos, ttl;
590+
591+ if (hdr->vlan_present)
592+ vlan = be16_to_cpu(hdr->vlan.tag);
593+
594+ if (hdr->ipv4_present) {
595+ flow_label = 0;
596+ ttl = hdr->ip4.ttl;
597+ tos = hdr->ip4.tos;
598+ *(__be16 *)(hdr->grh.destination_gid.raw + 10) = cpu_to_be16(0xffff);
599+ *(__be32 *)(hdr->grh.destination_gid.raw + 12) = hdr->ip4.daddr;
600+ } else {
601+ flow_label = be32_to_cpu(hdr->grh.flow_label);
602+ ttl = hdr->grh.hop_limit;
603+ tos = hdr->grh.traffic_class;
604+ }
605+
606+ memset(ah_attr, 0, sizeof(*ah_attr));
607+ ah_attr->type = RDMA_AH_ATTR_TYPE_ROCE;
608+ if (hdr->eth_present)
609+ ether_addr_copy(ah_attr->roce.dmac, hdr->eth.dmac_h);
610+ rdma_ah_set_sl(ah_attr, vlan >> VLAN_PRIO_SHIFT);
611+ rdma_ah_set_port_num(ah_attr, 1);
612+ rdma_ah_set_grh(ah_attr, NULL, flow_label, sgid_index, ttl, tos);
613+ rdma_ah_set_dgid_raw(ah_attr, &hdr->grh.destination_gid);
614+}
615+
616+static int ionic_create_ah_cmd(struct ionic_ibdev *dev,
617+ struct ionic_ah *ah,
618+ struct ionic_pd *pd,
619+ struct rdma_ah_attr *attr,
620+ u32 flags)
621+{
622+ struct ionic_admin_wr wr = {
623+ .work = COMPLETION_INITIALIZER_ONSTACK(wr.work),
624+ .wqe = {
625+ .op = IONIC_V1_ADMIN_CREATE_AH,
626+ .len = cpu_to_le16(IONIC_ADMIN_CREATE_AH_IN_V1_LEN),
627+ .cmd.create_ah = {
628+ .pd_id = cpu_to_le32(pd->pdid),
629+ .dbid_flags = cpu_to_le16(dev->lif_cfg.dbid),
630+ .id_ver = cpu_to_le32(ah->ahid),
631+ }
632+ }
633+ };
634+ enum ionic_admin_flags admin_flags = 0;
635+ dma_addr_t hdr_dma = 0;
636+ void *hdr_buf;
637+ gfp_t gfp = GFP_ATOMIC;
638+ int rc, hdr_len = 0;
639+
640+ if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_CREATE_AH)
641+ return -EBADRQC;
642+
643+ if (flags & RDMA_CREATE_AH_SLEEPABLE)
644+ gfp = GFP_KERNEL;
645+ else
646+ admin_flags |= IONIC_ADMIN_F_BUSYWAIT;
647+
648+ rc = ionic_build_hdr(dev, &ah->hdr, attr, IONIC_ROCE_UDP_SPORT, false);
649+ if (rc)
650+ return rc;
651+
652+ if (ah->hdr.eth.type == cpu_to_be16(ETH_P_8021Q)) {
653+ if (ah->hdr.vlan.type == cpu_to_be16(ETH_P_IP))
654+ wr.wqe.cmd.create_ah.csum_profile =
655+ IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_UDP;
656+ else
657+ wr.wqe.cmd.create_ah.csum_profile =
658+ IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV6_UDP;
659+ } else {
660+ if (ah->hdr.eth.type == cpu_to_be16(ETH_P_IP))
661+ wr.wqe.cmd.create_ah.csum_profile =
662+ IONIC_TFP_CSUM_PROF_ETH_IPV4_UDP;
663+ else
664+ wr.wqe.cmd.create_ah.csum_profile =
665+ IONIC_TFP_CSUM_PROF_ETH_IPV6_UDP;
666+ }
667+
668+ ah->sgid_index = rdma_ah_read_grh(attr)->sgid_index;
669+
670+ hdr_buf = kmalloc(PAGE_SIZE, gfp);
671+ if (!hdr_buf)
672+ return -ENOMEM;
673+
674+ hdr_len = ib_ud_header_pack(&ah->hdr, hdr_buf);
675+ hdr_len -= IB_BTH_BYTES;
676+ hdr_len -= IB_DETH_BYTES;
677+ ibdev_dbg(&dev->ibdev, "roce packet header template\n");
678+ print_hex_dump_debug("hdr ", DUMP_PREFIX_OFFSET, 16, 1,
679+ hdr_buf, hdr_len, true);
680+
681+ hdr_dma = dma_map_single(dev->lif_cfg.hwdev, hdr_buf, hdr_len,
682+ DMA_TO_DEVICE);
683+
684+ rc = dma_mapping_error(dev->lif_cfg.hwdev, hdr_dma);
685+ if (rc)
686+ goto err_dma;
687+
688+ wr.wqe.cmd.create_ah.dma_addr = cpu_to_le64(hdr_dma);
689+ wr.wqe.cmd.create_ah.length = cpu_to_le32(hdr_len);
690+
691+ ionic_admin_post(dev, &wr);
692+ rc = ionic_admin_wait(dev, &wr, admin_flags);
693+
694+ dma_unmap_single(dev->lif_cfg.hwdev, hdr_dma, hdr_len,
695+ DMA_TO_DEVICE);
696+err_dma:
697+ kfree(hdr_buf);
698+
699+ return rc;
700+}
701+
702+static int ionic_destroy_ah_cmd(struct ionic_ibdev *dev, u32 ahid, u32 flags)
703+{
704+ struct ionic_admin_wr wr = {
705+ .work = COMPLETION_INITIALIZER_ONSTACK(wr.work),
706+ .wqe = {
707+ .op = IONIC_V1_ADMIN_DESTROY_AH,
708+ .len = cpu_to_le16(IONIC_ADMIN_DESTROY_AH_IN_V1_LEN),
709+ .cmd.destroy_ah = {
710+ .ah_id = cpu_to_le32(ahid),
711+ },
712+ }
713+ };
714+ enum ionic_admin_flags admin_flags = IONIC_ADMIN_F_TEARDOWN;
715+
716+ if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_DESTROY_AH)
717+ return -EBADRQC;
718+
719+ if (!(flags & RDMA_CREATE_AH_SLEEPABLE))
720+ admin_flags |= IONIC_ADMIN_F_BUSYWAIT;
721+
722+ ionic_admin_post(dev, &wr);
723+ ionic_admin_wait(dev, &wr, admin_flags);
724+
725+ /* No host-memory resource is associated with ah, so it is ok
726+ * to "succeed" and complete this destroy ah on the host.
727+ */
728+ return 0;
729+}
730+
731+int ionic_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *init_attr,
732+ struct ib_udata *udata)
733+{
734+ struct ionic_ibdev *dev = to_ionic_ibdev(ibah->device);
735+ struct rdma_ah_attr *attr = init_attr->ah_attr;
736+ struct ionic_pd *pd = to_ionic_pd(ibah->pd);
737+ struct ionic_ah *ah = to_ionic_ah(ibah);
738+ struct ionic_ah_resp resp = {};
739+ u32 flags = init_attr->flags;
740+ int rc;
741+
742+ rc = ionic_get_ahid(dev, &ah->ahid);
743+ if (rc)
744+ return rc;
745+
746+ rc = ionic_create_ah_cmd(dev, ah, pd, attr, flags);
747+ if (rc)
748+ goto err_cmd;
749+
750+ if (udata) {
751+ resp.ahid = ah->ahid;
752+
753+ rc = ib_copy_to_udata(udata, &resp, sizeof(resp));
754+ if (rc)
755+ goto err_resp;
756+ }
757+
758+ return 0;
759+
760+err_resp:
761+ ionic_destroy_ah_cmd(dev, ah->ahid, flags);
762+err_cmd:
763+ ionic_put_ahid(dev, ah->ahid);
764+ return rc;
765+}
766+
767+int ionic_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr)
768+{
769+ struct ionic_ibdev *dev = to_ionic_ibdev(ibah->device);
770+ struct ionic_ah *ah = to_ionic_ah(ibah);
771+
772+ ionic_set_ah_attr(dev, ah_attr, &ah->hdr, ah->sgid_index);
773+
774+ return 0;
775+}
776+
777+int ionic_destroy_ah(struct ib_ah *ibah, u32 flags)
778+{
779+ struct ionic_ibdev *dev = to_ionic_ibdev(ibah->device);
780+ struct ionic_ah *ah = to_ionic_ah(ibah);
781+ int rc;
782+
783+ rc = ionic_destroy_ah_cmd(dev, ah->ahid, flags);
784+ if (rc)
785+ return rc;
786+
787+ ionic_put_ahid(dev, ah->ahid);
788+
789+ return 0;
790+}
791+
792+static int ionic_create_mr_cmd(struct ionic_ibdev *dev,
793+ struct ionic_pd *pd,
794+ struct ionic_mr *mr,
795+ u64 addr,
796+ u64 length)
797+{
798+ struct ionic_admin_wr wr = {
799+ .work = COMPLETION_INITIALIZER_ONSTACK(wr.work),
800+ .wqe = {
801+ .op = IONIC_V1_ADMIN_CREATE_MR,
802+ .len = cpu_to_le16(IONIC_ADMIN_CREATE_MR_IN_V1_LEN),
803+ .cmd.create_mr = {
804+ .va = cpu_to_le64(addr),
805+ .length = cpu_to_le64(length),
806+ .pd_id = cpu_to_le32(pd->pdid),
807+ .page_size_log2 = mr->buf.page_size_log2,
808+ .tbl_index = cpu_to_le32(~0),
809+ .map_count = cpu_to_le32(mr->buf.tbl_pages),
810+ .dma_addr = ionic_pgtbl_dma(&mr->buf, addr),
811+ .dbid_flags = cpu_to_le16(mr->flags),
812+ .id_ver = cpu_to_le32(mr->mrid),
813+ }
814+ }
815+ };
816+ int rc;
817+
818+ if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_CREATE_MR)
819+ return -EBADRQC;
820+
821+ ionic_admin_post(dev, &wr);
822+ rc = ionic_admin_wait(dev, &wr, 0);
823+ if (!rc)
824+ mr->created = true;
825+
826+ return rc;
827+}
828+
829+static int ionic_destroy_mr_cmd(struct ionic_ibdev *dev, u32 mrid)
830+{
831+ struct ionic_admin_wr wr = {
832+ .work = COMPLETION_INITIALIZER_ONSTACK(wr.work),
833+ .wqe = {
834+ .op = IONIC_V1_ADMIN_DESTROY_MR,
835+ .len = cpu_to_le16(IONIC_ADMIN_DESTROY_MR_IN_V1_LEN),
836+ .cmd.destroy_mr = {
837+ .mr_id = cpu_to_le32(mrid),
838+ },
839+ }
840+ };
841+
842+ if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_DESTROY_MR)
843+ return -EBADRQC;
844+
845+ ionic_admin_post(dev, &wr);
846+
847+ return ionic_admin_wait(dev, &wr, IONIC_ADMIN_F_TEARDOWN);
848+}
849+
850+struct ib_mr *ionic_get_dma_mr(struct ib_pd *ibpd, int access)
851+{
852+ struct ionic_pd *pd = to_ionic_pd(ibpd);
853+ struct ionic_mr *mr;
854+
855+ mr = kzalloc(sizeof(*mr), GFP_KERNEL);
856+ if (!mr)
857+ return ERR_PTR(-ENOMEM);
858+
859+ mr->ibmr.lkey = IONIC_DMA_LKEY;
860+ mr->ibmr.rkey = IONIC_DMA_RKEY;
861+
862+ if (pd)
863+ pd->flags |= IONIC_QPF_PRIVILEGED;
864+
865+ return &mr->ibmr;
866+}
867+
868+struct ib_mr *ionic_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length,
869+ u64 addr, int access, struct ib_dmah *dmah,
870+ struct ib_udata *udata)
871+{
872+ struct ionic_ibdev *dev = to_ionic_ibdev(ibpd->device);
873+ struct ionic_pd *pd = to_ionic_pd(ibpd);
874+ struct ionic_mr *mr;
875+ unsigned long pg_sz;
876+ int rc;
877+
878+ if (dmah)
879+ return ERR_PTR(-EOPNOTSUPP);
880+
881+ mr = kzalloc(sizeof(*mr), GFP_KERNEL);
882+ if (!mr)
883+ return ERR_PTR(-ENOMEM);
884+
885+ rc = ionic_get_mrid(dev, &mr->mrid);
886+ if (rc)
887+ goto err_mrid;
888+
889+ mr->ibmr.lkey = mr->mrid;
890+ mr->ibmr.rkey = mr->mrid;
891+ mr->ibmr.iova = addr;
892+ mr->ibmr.length = length;
893+
894+ mr->flags = IONIC_MRF_USER_MR | to_ionic_mr_flags(access);
895+
896+ mr->umem = ib_umem_get(&dev->ibdev, start, length, access);
897+ if (IS_ERR(mr->umem)) {
898+ rc = PTR_ERR(mr->umem);
899+ goto err_umem;
900+ }
901+
902+ pg_sz = ib_umem_find_best_pgsz(mr->umem,
903+ dev->lif_cfg.page_size_supported,
904+ addr);
905+ if (!pg_sz) {
906+ rc = -EINVAL;
907+ goto err_pgtbl;
908+ }
909+
910+ rc = ionic_pgtbl_init(dev, &mr->buf, mr->umem, 0, 1, pg_sz);
911+ if (rc)
912+ goto err_pgtbl;
913+
914+ rc = ionic_create_mr_cmd(dev, pd, mr, addr, length);
915+ if (rc)
916+ goto err_cmd;
917+
918+ ionic_pgtbl_unbuf(dev, &mr->buf);
919+
920+ return &mr->ibmr;
921+
922+err_cmd:
923+ ionic_pgtbl_unbuf(dev, &mr->buf);
924+err_pgtbl:
925+ ib_umem_release(mr->umem);
926+err_umem:
927+ ionic_put_mrid(dev, mr->mrid);
928+err_mrid:
929+ kfree(mr);
930+ return ERR_PTR(rc);
931+}
932+
933+struct ib_mr *ionic_reg_user_mr_dmabuf(struct ib_pd *ibpd, u64 offset,
934+ u64 length, u64 addr, int fd, int access,
935+ struct ib_dmah *dmah,
936+ struct uverbs_attr_bundle *attrs)
937+{
938+ struct ionic_ibdev *dev = to_ionic_ibdev(ibpd->device);
939+ struct ionic_pd *pd = to_ionic_pd(ibpd);
940+ struct ib_umem_dmabuf *umem_dmabuf;
941+ struct ionic_mr *mr;
942+ u64 pg_sz;
943+ int rc;
944+
945+ if (dmah)
946+ return ERR_PTR(-EOPNOTSUPP);
947+
948+ mr = kzalloc(sizeof(*mr), GFP_KERNEL);
949+ if (!mr)
950+ return ERR_PTR(-ENOMEM);
951+
952+ rc = ionic_get_mrid(dev, &mr->mrid);
953+ if (rc)
954+ goto err_mrid;
955+
956+ mr->ibmr.lkey = mr->mrid;
957+ mr->ibmr.rkey = mr->mrid;
958+ mr->ibmr.iova = addr;
959+ mr->ibmr.length = length;
960+
961+ mr->flags = IONIC_MRF_USER_MR | to_ionic_mr_flags(access);
962+
963+ umem_dmabuf = ib_umem_dmabuf_get_pinned(&dev->ibdev, offset, length,
964+ fd, access);
965+ if (IS_ERR(umem_dmabuf)) {
966+ rc = PTR_ERR(umem_dmabuf);
967+ goto err_umem;
968+ }
969+
970+ mr->umem = &umem_dmabuf->umem;
971+
972+ pg_sz = ib_umem_find_best_pgsz(mr->umem,
973+ dev->lif_cfg.page_size_supported,
974+ addr);
975+ if (!pg_sz) {
976+ rc = -EINVAL;
977+ goto err_pgtbl;
978+ }
979+
980+ rc = ionic_pgtbl_init(dev, &mr->buf, mr->umem, 0, 1, pg_sz);
981+ if (rc)
982+ goto err_pgtbl;
983+
984+ rc = ionic_create_mr_cmd(dev, pd, mr, addr, length);
985+ if (rc)
986+ goto err_cmd;
987+
988+ ionic_pgtbl_unbuf(dev, &mr->buf);
989+
990+ return &mr->ibmr;
991+
992+err_cmd:
993+ ionic_pgtbl_unbuf(dev, &mr->buf);
994+err_pgtbl:
995+ ib_umem_release(mr->umem);
996+err_umem:
997+ ionic_put_mrid(dev, mr->mrid);
998+err_mrid:
999+ kfree(mr);
1000+ return ERR_PTR(rc);
1001+}
1002+
1003+int ionic_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
1004+{
1005+ struct ionic_ibdev *dev = to_ionic_ibdev(ibmr->device);
1006+ struct ionic_mr *mr = to_ionic_mr(ibmr);
1007+ int rc;
1008+
1009+ if (!mr->ibmr.lkey)
1010+ goto out;
1011+
1012+ if (mr->created) {
1013+ rc = ionic_destroy_mr_cmd(dev, mr->mrid);
1014+ if (rc)
1015+ return rc;
1016+ }
1017+
1018+ ionic_pgtbl_unbuf(dev, &mr->buf);
1019+
1020+ if (mr->umem)
1021+ ib_umem_release(mr->umem);
1022+
1023+ ionic_put_mrid(dev, mr->mrid);
1024+
1025+out:
1026+ kfree(mr);
1027+
1028+ return 0;
1029+}
1030+
1031+struct ib_mr *ionic_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type type,
1032+ u32 max_sg)
1033+{
1034+ struct ionic_ibdev *dev = to_ionic_ibdev(ibpd->device);
1035+ struct ionic_pd *pd = to_ionic_pd(ibpd);
1036+ struct ionic_mr *mr;
1037+ int rc;
1038+
1039+ if (type != IB_MR_TYPE_MEM_REG)
1040+ return ERR_PTR(-EINVAL);
1041+
1042+ mr = kzalloc(sizeof(*mr), GFP_KERNEL);
1043+ if (!mr)
1044+ return ERR_PTR(-ENOMEM);
1045+
1046+ rc = ionic_get_mrid(dev, &mr->mrid);
1047+ if (rc)
1048+ goto err_mrid;
1049+
1050+ mr->ibmr.lkey = mr->mrid;
1051+ mr->ibmr.rkey = mr->mrid;
1052+
1053+ mr->flags = IONIC_MRF_PHYS_MR;
1054+
1055+ rc = ionic_pgtbl_init(dev, &mr->buf, mr->umem, 0, max_sg, PAGE_SIZE);
1056+ if (rc)
1057+ goto err_pgtbl;
1058+
1059+ mr->buf.tbl_pages = 0;
1060+
1061+ rc = ionic_create_mr_cmd(dev, pd, mr, 0, 0);
1062+ if (rc)
1063+ goto err_cmd;
1064+
1065+ return &mr->ibmr;
1066+
1067+err_cmd:
1068+ ionic_pgtbl_unbuf(dev, &mr->buf);
1069+err_pgtbl:
1070+ ionic_put_mrid(dev, mr->mrid);
1071+err_mrid:
1072+ kfree(mr);
1073+ return ERR_PTR(rc);
1074+}
1075+
1076+static int ionic_map_mr_page(struct ib_mr *ibmr, u64 dma)
1077+{
1078+ struct ionic_ibdev *dev = to_ionic_ibdev(ibmr->device);
1079+ struct ionic_mr *mr = to_ionic_mr(ibmr);
1080+
1081+ ibdev_dbg(&dev->ibdev, "dma %p\n", (void *)dma);
1082+ return ionic_pgtbl_page(&mr->buf, dma);
1083+}
1084+
1085+int ionic_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
1086+ unsigned int *sg_offset)
1087+{
1088+ struct ionic_ibdev *dev = to_ionic_ibdev(ibmr->device);
1089+ struct ionic_mr *mr = to_ionic_mr(ibmr);
1090+ int rc;
1091+
1092+ /* mr must be allocated using ib_alloc_mr() */
1093+ if (unlikely(!mr->buf.tbl_limit))
1094+ return -EINVAL;
1095+
1096+ mr->buf.tbl_pages = 0;
1097+
1098+ if (mr->buf.tbl_buf)
1099+ dma_sync_single_for_cpu(dev->lif_cfg.hwdev, mr->buf.tbl_dma,
1100+ mr->buf.tbl_size, DMA_TO_DEVICE);
1101+
1102+ ibdev_dbg(&dev->ibdev, "sg %p nent %d\n", sg, sg_nents);
1103+ rc = ib_sg_to_pages(ibmr, sg, sg_nents, sg_offset, ionic_map_mr_page);
1104+
1105+ mr->buf.page_size_log2 = order_base_2(ibmr->page_size);
1106+
1107+ if (mr->buf.tbl_buf)
1108+ dma_sync_single_for_device(dev->lif_cfg.hwdev, mr->buf.tbl_dma,
1109+ mr->buf.tbl_size, DMA_TO_DEVICE);
1110+
1111+ return rc;
1112+}
1113+
1114+int ionic_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata)
1115+{
1116+ struct ionic_ibdev *dev = to_ionic_ibdev(ibmw->device);
1117+ struct ionic_pd *pd = to_ionic_pd(ibmw->pd);
1118+ struct ionic_mr *mr = to_ionic_mw(ibmw);
1119+ int rc;
1120+
1121+ rc = ionic_get_mrid(dev, &mr->mrid);
1122+ if (rc)
1123+ return rc;
1124+
1125+ mr->ibmw.rkey = mr->mrid;
1126+
1127+ if (mr->ibmw.type == IB_MW_TYPE_1)
1128+ mr->flags = IONIC_MRF_MW_1;
1129+ else
1130+ mr->flags = IONIC_MRF_MW_2;
1131+
1132+ rc = ionic_create_mr_cmd(dev, pd, mr, 0, 0);
1133+ if (rc)
1134+ goto err_cmd;
1135+
1136+ return 0;
1137+
1138+err_cmd:
1139+ ionic_put_mrid(dev, mr->mrid);
1140+ return rc;
1141+}
1142+
1143+int ionic_dealloc_mw(struct ib_mw *ibmw)
1144+{
1145+ struct ionic_ibdev *dev = to_ionic_ibdev(ibmw->device);
1146+ struct ionic_mr *mr = to_ionic_mw(ibmw);
1147+ int rc;
1148+
1149+ rc = ionic_destroy_mr_cmd(dev, mr->mrid);
1150+ if (rc)
1151+ return rc;
1152+
1153+ ionic_put_mrid(dev, mr->mrid);
1154+
1155+ return 0;
1156+}
1157+
1158+static int ionic_create_cq_cmd(struct ionic_ibdev *dev,
1159+ struct ionic_ctx *ctx,
1160+ struct ionic_cq *cq,
1161+ struct ionic_tbl_buf *buf)
1162+{
1163+ const u16 dbid = ionic_ctx_dbid(dev, ctx);
1164+ struct ionic_admin_wr wr = {
1165+ .work = COMPLETION_INITIALIZER_ONSTACK(wr.work),
1166+ .wqe = {
1167+ .op = IONIC_V1_ADMIN_CREATE_CQ,
1168+ .len = cpu_to_le16(IONIC_ADMIN_CREATE_CQ_IN_V1_LEN),
1169+ .cmd.create_cq = {
1170+ .eq_id = cpu_to_le32(cq->eqid),
1171+ .depth_log2 = cq->q.depth_log2,
1172+ .stride_log2 = cq->q.stride_log2,
1173+ .page_size_log2 = buf->page_size_log2,
1174+ .tbl_index = cpu_to_le32(~0),
1175+ .map_count = cpu_to_le32(buf->tbl_pages),
1176+ .dma_addr = ionic_pgtbl_dma(buf, 0),
1177+ .dbid_flags = cpu_to_le16(dbid),
1178+ .id_ver = cpu_to_le32(cq->cqid),
1179+ }
1180+ }
1181+ };
1182+
1183+ if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_CREATE_CQ)
1184+ return -EBADRQC;
1185+
1186+ ionic_admin_post(dev, &wr);
1187+
1188+ return ionic_admin_wait(dev, &wr, 0);
1189+}
1190+
1191+static int ionic_destroy_cq_cmd(struct ionic_ibdev *dev, u32 cqid)
1192+{
1193+ struct ionic_admin_wr wr = {
1194+ .work = COMPLETION_INITIALIZER_ONSTACK(wr.work),
1195+ .wqe = {
1196+ .op = IONIC_V1_ADMIN_DESTROY_CQ,
1197+ .len = cpu_to_le16(IONIC_ADMIN_DESTROY_CQ_IN_V1_LEN),
1198+ .cmd.destroy_cq = {
1199+ .cq_id = cpu_to_le32(cqid),
1200+ },
1201+ }
1202+ };
1203+
1204+ if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_DESTROY_CQ)
1205+ return -EBADRQC;
1206+
1207+ ionic_admin_post(dev, &wr);
1208+
1209+ return ionic_admin_wait(dev, &wr, IONIC_ADMIN_F_TEARDOWN);
1210+}
1211+
1212+int ionic_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
1213+ struct uverbs_attr_bundle *attrs)
1214+{
1215+ struct ionic_ibdev *dev = to_ionic_ibdev(ibcq->device);
1216+ struct ib_udata *udata = &attrs->driver_udata;
1217+ struct ionic_ctx *ctx =
1218+ rdma_udata_to_drv_context(udata, struct ionic_ctx, ibctx);
1219+ struct ionic_vcq *vcq = to_ionic_vcq(ibcq);
1220+ struct ionic_tbl_buf buf = {};
1221+ struct ionic_cq_resp resp;
1222+ struct ionic_cq_req req;
1223+ int udma_idx = 0, rc;
1224+
1225+ if (udata) {
1226+ rc = ib_copy_from_udata(&req, udata, sizeof(req));
1227+ if (rc)
1228+ return rc;
1229+ }
1230+
1231+ vcq->udma_mask = BIT(dev->lif_cfg.udma_count) - 1;
1232+
1233+ if (udata)
1234+ vcq->udma_mask &= req.udma_mask;
1235+
1236+ if (!vcq->udma_mask) {
1237+ rc = -EINVAL;
1238+ goto err_init;
1239+ }
1240+
1241+ for (; udma_idx < dev->lif_cfg.udma_count; ++udma_idx) {
1242+ if (!(vcq->udma_mask & BIT(udma_idx)))
1243+ continue;
1244+
1245+ rc = ionic_create_cq_common(vcq, &buf, attr, ctx, udata,
1246+ &req.cq[udma_idx],
1247+ &resp.cqid[udma_idx],
1248+ udma_idx);
1249+ if (rc)
1250+ goto err_init;
1251+
1252+ rc = ionic_create_cq_cmd(dev, ctx, &vcq->cq[udma_idx], &buf);
1253+ if (rc)
1254+ goto err_cmd;
1255+
1256+ ionic_pgtbl_unbuf(dev, &buf);
1257+ }
1258+
1259+ vcq->ibcq.cqe = attr->cqe;
1260+
1261+ if (udata) {
1262+ resp.udma_mask = vcq->udma_mask;
1263+
1264+ rc = ib_copy_to_udata(udata, &resp, sizeof(resp));
1265+ if (rc)
1266+ goto err_resp;
1267+ }
1268+
1269+ return 0;
1270+
1271+err_resp:
1272+ while (udma_idx) {
1273+ --udma_idx;
1274+ if (!(vcq->udma_mask & BIT(udma_idx)))
1275+ continue;
1276+ ionic_destroy_cq_cmd(dev, vcq->cq[udma_idx].cqid);
1277+err_cmd:
1278+ ionic_pgtbl_unbuf(dev, &buf);
1279+ ionic_destroy_cq_common(dev, &vcq->cq[udma_idx]);
1280+err_init:
1281+ ;
1282+ }
1283+
1284+ return rc;
1285+}
1286+
1287+int ionic_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
1288+{
1289+ struct ionic_ibdev *dev = to_ionic_ibdev(ibcq->device);
1290+ struct ionic_vcq *vcq = to_ionic_vcq(ibcq);
1291+ int udma_idx, rc_tmp, rc = 0;
1292+
1293+ for (udma_idx = dev->lif_cfg.udma_count; udma_idx; ) {
1294+ --udma_idx;
1295+
1296+ if (!(vcq->udma_mask & BIT(udma_idx)))
1297+ continue;
1298+
1299+ rc_tmp = ionic_destroy_cq_cmd(dev, vcq->cq[udma_idx].cqid);
1300+ if (rc_tmp) {
1301+ if (!rc)
1302+ rc = rc_tmp;
1303+
1304+ continue;
1305+ }
1306+
1307+ ionic_destroy_cq_common(dev, &vcq->cq[udma_idx]);
1308+ }
1309+
1310+ return rc;
1311+}
1312+
1313+static bool pd_remote_privileged(struct ib_pd *pd)
1314+{
1315+ return pd->flags & IB_PD_UNSAFE_GLOBAL_RKEY;
1316+}
1317+
1318+static int ionic_create_qp_cmd(struct ionic_ibdev *dev,
1319+ struct ionic_pd *pd,
1320+ struct ionic_cq *send_cq,
1321+ struct ionic_cq *recv_cq,
1322+ struct ionic_qp *qp,
1323+ struct ionic_tbl_buf *sq_buf,
1324+ struct ionic_tbl_buf *rq_buf,
1325+ struct ib_qp_init_attr *attr)
1326+{
1327+ const u16 dbid = ionic_obj_dbid(dev, pd->ibpd.uobject);
1328+ const u32 flags = to_ionic_qp_flags(0, 0,
1329+ qp->sq_cmb & IONIC_CMB_ENABLE,
1330+ qp->rq_cmb & IONIC_CMB_ENABLE,
1331+ qp->sq_spec, qp->rq_spec,
1332+ pd->flags & IONIC_QPF_PRIVILEGED,
1333+ pd_remote_privileged(&pd->ibpd));
1334+ struct ionic_admin_wr wr = {
1335+ .work = COMPLETION_INITIALIZER_ONSTACK(wr.work),
1336+ .wqe = {
1337+ .op = IONIC_V1_ADMIN_CREATE_QP,
1338+ .len = cpu_to_le16(IONIC_ADMIN_CREATE_QP_IN_V1_LEN),
1339+ .cmd.create_qp = {
1340+ .pd_id = cpu_to_le32(pd->pdid),
1341+ .priv_flags = cpu_to_be32(flags),
1342+ .type_state = to_ionic_qp_type(attr->qp_type),
1343+ .dbid_flags = cpu_to_le16(dbid),
1344+ .id_ver = cpu_to_le32(qp->qpid),
1345+ }
1346+ }
1347+ };
1348+
1349+ if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_CREATE_QP)
1350+ return -EBADRQC;
1351+
1352+ if (qp->has_sq) {
1353+ wr.wqe.cmd.create_qp.sq_cq_id = cpu_to_le32(send_cq->cqid);
1354+ wr.wqe.cmd.create_qp.sq_depth_log2 = qp->sq.depth_log2;
1355+ wr.wqe.cmd.create_qp.sq_stride_log2 = qp->sq.stride_log2;
1356+ wr.wqe.cmd.create_qp.sq_page_size_log2 = sq_buf->page_size_log2;
1357+ wr.wqe.cmd.create_qp.sq_tbl_index_xrcd_id = cpu_to_le32(~0);
1358+ wr.wqe.cmd.create_qp.sq_map_count =
1359+ cpu_to_le32(sq_buf->tbl_pages);
1360+ wr.wqe.cmd.create_qp.sq_dma_addr = ionic_pgtbl_dma(sq_buf, 0);
1361+ }
1362+
1363+ if (qp->has_rq) {
1364+ wr.wqe.cmd.create_qp.rq_cq_id = cpu_to_le32(recv_cq->cqid);
1365+ wr.wqe.cmd.create_qp.rq_depth_log2 = qp->rq.depth_log2;
1366+ wr.wqe.cmd.create_qp.rq_stride_log2 = qp->rq.stride_log2;
1367+ wr.wqe.cmd.create_qp.rq_page_size_log2 = rq_buf->page_size_log2;
1368+ wr.wqe.cmd.create_qp.rq_tbl_index_srq_id = cpu_to_le32(~0);
1369+ wr.wqe.cmd.create_qp.rq_map_count =
1370+ cpu_to_le32(rq_buf->tbl_pages);
1371+ wr.wqe.cmd.create_qp.rq_dma_addr = ionic_pgtbl_dma(rq_buf, 0);
1372+ }
1373+
1374+ ionic_admin_post(dev, &wr);
1375+
1376+ return ionic_admin_wait(dev, &wr, 0);
1377+}
1378+
1379+static int ionic_modify_qp_cmd(struct ionic_ibdev *dev,
1380+ struct ionic_pd *pd,
1381+ struct ionic_qp *qp,
1382+ struct ib_qp_attr *attr,
1383+ int mask)
1384+{
1385+ const u32 flags = to_ionic_qp_flags(attr->qp_access_flags,
1386+ attr->en_sqd_async_notify,
1387+ qp->sq_cmb & IONIC_CMB_ENABLE,
1388+ qp->rq_cmb & IONIC_CMB_ENABLE,
1389+ qp->sq_spec, qp->rq_spec,
1390+ pd->flags & IONIC_QPF_PRIVILEGED,
1391+ pd_remote_privileged(qp->ibqp.pd));
1392+ const u8 state = to_ionic_qp_modify_state(attr->qp_state,
1393+ attr->cur_qp_state);
1394+ struct ionic_admin_wr wr = {
1395+ .work = COMPLETION_INITIALIZER_ONSTACK(wr.work),
1396+ .wqe = {
1397+ .op = IONIC_V1_ADMIN_MODIFY_QP,
1398+ .len = cpu_to_le16(IONIC_ADMIN_MODIFY_QP_IN_V1_LEN),
1399+ .cmd.mod_qp = {
1400+ .attr_mask = cpu_to_be32(mask),
1401+ .access_flags = cpu_to_be16(flags),
1402+ .rq_psn = cpu_to_le32(attr->rq_psn),
1403+ .sq_psn = cpu_to_le32(attr->sq_psn),
1404+ .rate_limit_kbps =
1405+ cpu_to_le32(attr->rate_limit),
1406+ .pmtu = (attr->path_mtu + 7),
1407+ .retry = (attr->retry_cnt |
1408+ (attr->rnr_retry << 4)),
1409+ .rnr_timer = attr->min_rnr_timer,
1410+ .retry_timeout = attr->timeout,
1411+ .type_state = state,
1412+ .id_ver = cpu_to_le32(qp->qpid),
1413+ }
1414+ }
1415+ };
1416+ const struct ib_global_route *grh = rdma_ah_read_grh(&attr->ah_attr);
1417+ void *hdr_buf = NULL;
1418+ dma_addr_t hdr_dma = 0;
1419+ int rc, hdr_len = 0;
1420+ u16 sport;
1421+
1422+ if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_MODIFY_QP)
1423+ return -EBADRQC;
1424+
1425+ if ((mask & IB_QP_MAX_DEST_RD_ATOMIC) && attr->max_dest_rd_atomic) {
1426+ /* Note, round up/down was already done for allocating
1427+ * resources on the device. The allocation order is in cache
1428+ * line size. We can't use the order of the resource
1429+ * allocation to determine the order wqes here, because for
1430+ * queue length <= one cache line it is not distinct.
1431+ *
1432+ * Therefore, order wqes is computed again here.
1433+ *
1434+ * Account for hole and round up to the next order.
1435+ */
1436+ wr.wqe.cmd.mod_qp.rsq_depth =
1437+ order_base_2(attr->max_dest_rd_atomic + 1);
1438+ wr.wqe.cmd.mod_qp.rsq_index = cpu_to_le32(~0);
1439+ }
1440+
1441+ if ((mask & IB_QP_MAX_QP_RD_ATOMIC) && attr->max_rd_atomic) {
1442+ /* Account for hole and round down to the next order */
1443+ wr.wqe.cmd.mod_qp.rrq_depth =
1444+ order_base_2(attr->max_rd_atomic + 2) - 1;
1445+ wr.wqe.cmd.mod_qp.rrq_index = cpu_to_le32(~0);
1446+ }
1447+
1448+ if (qp->ibqp.qp_type == IB_QPT_RC || qp->ibqp.qp_type == IB_QPT_UC)
1449+ wr.wqe.cmd.mod_qp.qkey_dest_qpn =
1450+ cpu_to_le32(attr->dest_qp_num);
1451+ else
1452+ wr.wqe.cmd.mod_qp.qkey_dest_qpn = cpu_to_le32(attr->qkey);
1453+
1454+ if (mask & IB_QP_AV) {
1455+ if (!qp->hdr)
1456+ return -ENOMEM;
1457+
1458+ sport = rdma_get_udp_sport(grh->flow_label,
1459+ qp->qpid,
1460+ attr->dest_qp_num);
1461+
1462+ rc = ionic_build_hdr(dev, qp->hdr, &attr->ah_attr, sport, true);
1463+ if (rc)
1464+ return rc;
1465+
1466+ qp->sgid_index = grh->sgid_index;
1467+
1468+ hdr_buf = kmalloc(PAGE_SIZE, GFP_KERNEL);
1469+ if (!hdr_buf)
1470+ return -ENOMEM;
1471+
1472+ hdr_len = ib_ud_header_pack(qp->hdr, hdr_buf);
1473+ hdr_len -= IB_BTH_BYTES;
1474+ hdr_len -= IB_DETH_BYTES;
1475+ ibdev_dbg(&dev->ibdev, "roce packet header template\n");
1476+ print_hex_dump_debug("hdr ", DUMP_PREFIX_OFFSET, 16, 1,
1477+ hdr_buf, hdr_len, true);
1478+
1479+ hdr_dma = dma_map_single(dev->lif_cfg.hwdev, hdr_buf, hdr_len,
1480+ DMA_TO_DEVICE);
1481+
1482+ rc = dma_mapping_error(dev->lif_cfg.hwdev, hdr_dma);
1483+ if (rc)
1484+ goto err_dma;
1485+
1486+ if (qp->hdr->ipv4_present) {
1487+ wr.wqe.cmd.mod_qp.tfp_csum_profile =
1488+ qp->hdr->vlan_present ?
1489+ IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_UDP :
1490+ IONIC_TFP_CSUM_PROF_ETH_IPV4_UDP;
1491+ } else {
1492+ wr.wqe.cmd.mod_qp.tfp_csum_profile =
1493+ qp->hdr->vlan_present ?
1494+ IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV6_UDP :
1495+ IONIC_TFP_CSUM_PROF_ETH_IPV6_UDP;
1496+ }
1497+
1498+ wr.wqe.cmd.mod_qp.ah_id_len =
1499+ cpu_to_le32(qp->ahid | (hdr_len << 24));
1500+ wr.wqe.cmd.mod_qp.dma_addr = cpu_to_le64(hdr_dma);
1501+
1502+ wr.wqe.cmd.mod_qp.en_pcp = attr->ah_attr.sl;
1503+ wr.wqe.cmd.mod_qp.ip_dscp = grh->traffic_class >> 2;
1504+ }
1505+
1506+ ionic_admin_post(dev, &wr);
1507+
1508+ rc = ionic_admin_wait(dev, &wr, 0);
1509+
1510+ if (mask & IB_QP_AV)
1511+ dma_unmap_single(dev->lif_cfg.hwdev, hdr_dma, hdr_len,
1512+ DMA_TO_DEVICE);
1513+err_dma:
1514+ if (mask & IB_QP_AV)
1515+ kfree(hdr_buf);
1516+
1517+ return rc;
1518+}
1519+
1520+static int ionic_query_qp_cmd(struct ionic_ibdev *dev,
1521+ struct ionic_qp *qp,
1522+ struct ib_qp_attr *attr,
1523+ int mask)
1524+{
1525+ struct ionic_admin_wr wr = {
1526+ .work = COMPLETION_INITIALIZER_ONSTACK(wr.work),
1527+ .wqe = {
1528+ .op = IONIC_V1_ADMIN_QUERY_QP,
1529+ .len = cpu_to_le16(IONIC_ADMIN_QUERY_QP_IN_V1_LEN),
1530+ .cmd.query_qp = {
1531+ .id_ver = cpu_to_le32(qp->qpid),
1532+ },
1533+ }
1534+ };
1535+ struct ionic_v1_admin_query_qp_sq *query_sqbuf;
1536+ struct ionic_v1_admin_query_qp_rq *query_rqbuf;
1537+ dma_addr_t query_sqdma;
1538+ dma_addr_t query_rqdma;
1539+ dma_addr_t hdr_dma = 0;
1540+ void *hdr_buf = NULL;
1541+ int flags, rc;
1542+
1543+ if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_QUERY_QP)
1544+ return -EBADRQC;
1545+
1546+ if (qp->has_sq) {
1547+ bool expdb = !!(qp->sq_cmb & IONIC_CMB_EXPDB);
1548+
1549+ attr->cap.max_send_sge =
1550+ ionic_v1_send_wqe_max_sge(qp->sq.stride_log2,
1551+ qp->sq_spec,
1552+ expdb);
1553+ attr->cap.max_inline_data =
1554+ ionic_v1_send_wqe_max_data(qp->sq.stride_log2, expdb);
1555+ }
1556+
1557+ if (qp->has_rq) {
1558+ attr->cap.max_recv_sge =
1559+ ionic_v1_recv_wqe_max_sge(qp->rq.stride_log2,
1560+ qp->rq_spec,
1561+ qp->rq_cmb & IONIC_CMB_EXPDB);
1562+ }
1563+
1564+ query_sqbuf = kzalloc(PAGE_SIZE, GFP_KERNEL);
1565+ if (!query_sqbuf)
1566+ return -ENOMEM;
1567+
1568+ query_rqbuf = kzalloc(PAGE_SIZE, GFP_KERNEL);
1569+ if (!query_rqbuf) {
1570+ rc = -ENOMEM;
1571+ goto err_rqbuf;
1572+ }
1573+
1574+ query_sqdma = dma_map_single(dev->lif_cfg.hwdev, query_sqbuf, PAGE_SIZE,
1575+ DMA_FROM_DEVICE);
1576+ rc = dma_mapping_error(dev->lif_cfg.hwdev, query_sqdma);
1577+ if (rc)
1578+ goto err_sqdma;
1579+
1580+ query_rqdma = dma_map_single(dev->lif_cfg.hwdev, query_rqbuf, PAGE_SIZE,
1581+ DMA_FROM_DEVICE);
1582+ rc = dma_mapping_error(dev->lif_cfg.hwdev, query_rqdma);
1583+ if (rc)
1584+ goto err_rqdma;
1585+
1586+ if (mask & IB_QP_AV) {
1587+ hdr_buf = kmalloc(PAGE_SIZE, GFP_KERNEL);
1588+ if (!hdr_buf) {
1589+ rc = -ENOMEM;
1590+ goto err_hdrbuf;
1591+ }
1592+
1593+ hdr_dma = dma_map_single(dev->lif_cfg.hwdev, hdr_buf,
1594+ PAGE_SIZE, DMA_FROM_DEVICE);
1595+ rc = dma_mapping_error(dev->lif_cfg.hwdev, hdr_dma);
1596+ if (rc)
1597+ goto err_hdrdma;
1598+ }
1599+
1600+ wr.wqe.cmd.query_qp.sq_dma_addr = cpu_to_le64(query_sqdma);
1601+ wr.wqe.cmd.query_qp.rq_dma_addr = cpu_to_le64(query_rqdma);
1602+ wr.wqe.cmd.query_qp.hdr_dma_addr = cpu_to_le64(hdr_dma);
1603+ wr.wqe.cmd.query_qp.ah_id = cpu_to_le32(qp->ahid);
1604+
1605+ ionic_admin_post(dev, &wr);
1606+
1607+ rc = ionic_admin_wait(dev, &wr, 0);
1608+
1609+ if (rc)
1610+ goto err_hdrdma;
1611+
1612+ flags = be16_to_cpu(query_sqbuf->access_perms_flags |
1613+ query_rqbuf->access_perms_flags);
1614+
1615+ print_hex_dump_debug("sqbuf ", DUMP_PREFIX_OFFSET, 16, 1,
1616+ query_sqbuf, sizeof(*query_sqbuf), true);
1617+ print_hex_dump_debug("rqbuf ", DUMP_PREFIX_OFFSET, 16, 1,
1618+ query_rqbuf, sizeof(*query_rqbuf), true);
1619+ ibdev_dbg(&dev->ibdev, "query qp %u state_pmtu %#x flags %#x",
1620+ qp->qpid, query_rqbuf->state_pmtu, flags);
1621+
1622+ attr->qp_state = from_ionic_qp_state(query_rqbuf->state_pmtu >> 4);
1623+ attr->cur_qp_state = attr->qp_state;
1624+ attr->path_mtu = (query_rqbuf->state_pmtu & 0xf) - 7;
1625+ attr->path_mig_state = IB_MIG_MIGRATED;
1626+ attr->qkey = be32_to_cpu(query_sqbuf->qkey_dest_qpn);
1627+ attr->rq_psn = be32_to_cpu(query_sqbuf->rq_psn);
1628+ attr->sq_psn = be32_to_cpu(query_rqbuf->sq_psn);
1629+ attr->dest_qp_num = attr->qkey;
1630+ attr->qp_access_flags = from_ionic_qp_flags(flags);
1631+ attr->pkey_index = 0;
1632+ attr->alt_pkey_index = 0;
1633+ attr->en_sqd_async_notify = !!(flags & IONIC_QPF_SQD_NOTIFY);
1634+ attr->sq_draining = !!(flags & IONIC_QPF_SQ_DRAINING);
1635+ attr->max_rd_atomic = BIT(query_rqbuf->rrq_depth) - 1;
1636+ attr->max_dest_rd_atomic = BIT(query_rqbuf->rsq_depth) - 1;
1637+ attr->min_rnr_timer = query_sqbuf->rnr_timer;
1638+ attr->port_num = 0;
1639+ attr->timeout = query_sqbuf->retry_timeout;
1640+ attr->retry_cnt = query_rqbuf->retry_rnrtry & 0xf;
1641+ attr->rnr_retry = query_rqbuf->retry_rnrtry >> 4;
1642+ attr->alt_port_num = 0;
1643+ attr->alt_timeout = 0;
1644+ attr->rate_limit = be32_to_cpu(query_sqbuf->rate_limit_kbps);
1645+
1646+ if (mask & IB_QP_AV)
1647+ ionic_set_ah_attr(dev, &attr->ah_attr,
1648+ qp->hdr, qp->sgid_index);
1649+
1650+err_hdrdma:
1651+ if (mask & IB_QP_AV) {
1652+ dma_unmap_single(dev->lif_cfg.hwdev, hdr_dma,
1653+ PAGE_SIZE, DMA_FROM_DEVICE);
1654+ kfree(hdr_buf);
1655+ }
1656+err_hdrbuf:
1657+ dma_unmap_single(dev->lif_cfg.hwdev, query_rqdma, sizeof(*query_rqbuf),
1658+ DMA_FROM_DEVICE);
1659+err_rqdma:
1660+ dma_unmap_single(dev->lif_cfg.hwdev, query_sqdma, sizeof(*query_sqbuf),
1661+ DMA_FROM_DEVICE);
1662+err_sqdma:
1663+ kfree(query_rqbuf);
1664+err_rqbuf:
1665+ kfree(query_sqbuf);
1666+
1667+ return rc;
1668+}
1669+
1670+static int ionic_destroy_qp_cmd(struct ionic_ibdev *dev, u32 qpid)
1671+{
1672+ struct ionic_admin_wr wr = {
1673+ .work = COMPLETION_INITIALIZER_ONSTACK(wr.work),
1674+ .wqe = {
1675+ .op = IONIC_V1_ADMIN_DESTROY_QP,
1676+ .len = cpu_to_le16(IONIC_ADMIN_DESTROY_QP_IN_V1_LEN),
1677+ .cmd.destroy_qp = {
1678+ .qp_id = cpu_to_le32(qpid),
1679+ },
1680+ }
1681+ };
1682+
1683+ if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_DESTROY_QP)
1684+ return -EBADRQC;
1685+
1686+ ionic_admin_post(dev, &wr);
1687+
1688+ return ionic_admin_wait(dev, &wr, IONIC_ADMIN_F_TEARDOWN);
1689+}
1690+
1691+static bool ionic_expdb_wqe_size_supported(struct ionic_ibdev *dev,
1692+ uint32_t wqe_size)
1693+{
1694+ switch (wqe_size) {
1695+ case 64: return dev->lif_cfg.expdb_mask & IONIC_EXPDB_64;
1696+ case 128: return dev->lif_cfg.expdb_mask & IONIC_EXPDB_128;
1697+ case 256: return dev->lif_cfg.expdb_mask & IONIC_EXPDB_256;
1698+ case 512: return dev->lif_cfg.expdb_mask & IONIC_EXPDB_512;
1699+ }
1700+
1701+ return false;
1702+}
1703+
1704+static void ionic_qp_sq_init_cmb(struct ionic_ibdev *dev,
1705+ struct ionic_qp *qp,
1706+ struct ib_udata *udata,
1707+ int max_data)
1708+{
1709+ u8 expdb_stride_log2 = 0;
1710+ bool expdb;
1711+ int rc;
1712+
1713+ if (!(qp->sq_cmb & IONIC_CMB_ENABLE))
1714+ goto not_in_cmb;
1715+
1716+ if (qp->sq_cmb & ~IONIC_CMB_SUPPORTED) {
1717+ if (qp->sq_cmb & IONIC_CMB_REQUIRE)
1718+ goto not_in_cmb;
1719+
1720+ qp->sq_cmb &= IONIC_CMB_SUPPORTED;
1721+ }
1722+
1723+ if ((qp->sq_cmb & IONIC_CMB_EXPDB) && !dev->lif_cfg.sq_expdb) {
1724+ if (qp->sq_cmb & IONIC_CMB_REQUIRE)
1725+ goto not_in_cmb;
1726+
1727+ qp->sq_cmb &= ~IONIC_CMB_EXPDB;
1728+ }
1729+
1730+ qp->sq_cmb_order = order_base_2(qp->sq.size / PAGE_SIZE);
1731+
1732+ if (qp->sq_cmb_order >= IONIC_SQCMB_ORDER)
1733+ goto not_in_cmb;
1734+
1735+ if (qp->sq_cmb & IONIC_CMB_EXPDB)
1736+ expdb_stride_log2 = qp->sq.stride_log2;
1737+
1738+ rc = ionic_get_cmb(dev->lif_cfg.lif, &qp->sq_cmb_pgid,
1739+ &qp->sq_cmb_addr, qp->sq_cmb_order,
1740+ expdb_stride_log2, &expdb);
1741+ if (rc)
1742+ goto not_in_cmb;
1743+
1744+ if ((qp->sq_cmb & IONIC_CMB_EXPDB) && !expdb) {
1745+ if (qp->sq_cmb & IONIC_CMB_REQUIRE)
1746+ goto err_map;
1747+
1748+ qp->sq_cmb &= ~IONIC_CMB_EXPDB;
1749+ }
1750+
1751+ return;
1752+
1753+err_map:
1754+ ionic_put_cmb(dev->lif_cfg.lif, qp->sq_cmb_pgid, qp->sq_cmb_order);
1755+not_in_cmb:
1756+ if (qp->sq_cmb & IONIC_CMB_REQUIRE)
1757+ ibdev_dbg(&dev->ibdev, "could not place sq in cmb as required\n");
1758+
1759+ qp->sq_cmb = 0;
1760+ qp->sq_cmb_order = IONIC_RES_INVALID;
1761+ qp->sq_cmb_pgid = 0;
1762+ qp->sq_cmb_addr = 0;
1763+}
1764+
1765+static void ionic_qp_sq_destroy_cmb(struct ionic_ibdev *dev,
1766+ struct ionic_ctx *ctx,
1767+ struct ionic_qp *qp)
1768+{
1769+ if (!(qp->sq_cmb & IONIC_CMB_ENABLE))
1770+ return;
1771+
1772+ if (ctx)
1773+ rdma_user_mmap_entry_remove(qp->mmap_sq_cmb);
1774+
1775+ ionic_put_cmb(dev->lif_cfg.lif, qp->sq_cmb_pgid, qp->sq_cmb_order);
1776+}
1777+
1778+static int ionic_qp_sq_init(struct ionic_ibdev *dev, struct ionic_ctx *ctx,
1779+ struct ionic_qp *qp, struct ionic_qdesc *sq,
1780+ struct ionic_tbl_buf *buf, int max_wr, int max_sge,
1781+ int max_data, int sq_spec, struct ib_udata *udata)
1782+{
1783+ u32 wqe_size;
1784+ int rc = 0;
1785+
1786+ qp->sq_msn_prod = 0;
1787+ qp->sq_msn_cons = 0;
1788+
1789+ if (!qp->has_sq) {
1790+ if (buf) {
1791+ buf->tbl_buf = NULL;
1792+ buf->tbl_limit = 0;
1793+ buf->tbl_pages = 0;
1794+ }
1795+ if (udata)
1796+ rc = ionic_validate_qdesc_zero(sq);
1797+
1798+ return rc;
1799+ }
1800+
1801+ rc = -EINVAL;
1802+
1803+ if (max_wr < 0 || max_wr > 0xffff)
1804+ return rc;
1805+
1806+ if (max_sge < 1)
1807+ return rc;
1808+
1809+ if (max_sge > min(ionic_v1_send_wqe_max_sge(dev->lif_cfg.max_stride, 0,
1810+ qp->sq_cmb &
1811+ IONIC_CMB_EXPDB),
1812+ IONIC_SPEC_HIGH))
1813+ return rc;
1814+
1815+ if (max_data < 0)
1816+ return rc;
1817+
1818+ if (max_data > ionic_v1_send_wqe_max_data(dev->lif_cfg.max_stride,
1819+ qp->sq_cmb & IONIC_CMB_EXPDB))
1820+ return rc;
1821+
1822+ if (udata) {
1823+ rc = ionic_validate_qdesc(sq);
1824+ if (rc)
1825+ return rc;
1826+
1827+ qp->sq_spec = sq_spec;
1828+
1829+ qp->sq.ptr = NULL;
1830+ qp->sq.size = sq->size;
1831+ qp->sq.mask = sq->mask;
1832+ qp->sq.depth_log2 = sq->depth_log2;
1833+ qp->sq.stride_log2 = sq->stride_log2;
1834+
1835+ qp->sq_meta = NULL;
1836+ qp->sq_msn_idx = NULL;
1837+
1838+ qp->sq_umem = ib_umem_get(&dev->ibdev, sq->addr, sq->size, 0);
1839+ if (IS_ERR(qp->sq_umem))
1840+ return PTR_ERR(qp->sq_umem);
1841+ } else {
1842+ qp->sq_umem = NULL;
1843+
1844+ qp->sq_spec = ionic_v1_use_spec_sge(max_sge, sq_spec);
1845+ if (sq_spec && !qp->sq_spec)
1846+ ibdev_dbg(&dev->ibdev,
1847+ "init sq: max_sge %u disables spec\n",
1848+ max_sge);
1849+
1850+ if (qp->sq_cmb & IONIC_CMB_EXPDB) {
1851+ wqe_size = ionic_v1_send_wqe_min_size(max_sge, max_data,
1852+ qp->sq_spec,
1853+ true);
1854+
1855+ if (!ionic_expdb_wqe_size_supported(dev, wqe_size))
1856+ qp->sq_cmb &= ~IONIC_CMB_EXPDB;
1857+ }
1858+
1859+ if (!(qp->sq_cmb & IONIC_CMB_EXPDB))
1860+ wqe_size = ionic_v1_send_wqe_min_size(max_sge, max_data,
1861+ qp->sq_spec,
1862+ false);
1863+
1864+ rc = ionic_queue_init(&qp->sq, dev->lif_cfg.hwdev,
1865+ max_wr, wqe_size);
1866+ if (rc)
1867+ return rc;
1868+
1869+ ionic_queue_dbell_init(&qp->sq, qp->qpid);
1870+
1871+ qp->sq_meta = kmalloc_array((u32)qp->sq.mask + 1,
1872+ sizeof(*qp->sq_meta),
1873+ GFP_KERNEL);
1874+ if (!qp->sq_meta) {
1875+ rc = -ENOMEM;
1876+ goto err_sq_meta;
1877+ }
1878+
1879+ qp->sq_msn_idx = kmalloc_array((u32)qp->sq.mask + 1,
1880+ sizeof(*qp->sq_msn_idx),
1881+ GFP_KERNEL);
1882+ if (!qp->sq_msn_idx) {
1883+ rc = -ENOMEM;
1884+ goto err_sq_msn;
1885+ }
1886+ }
1887+
1888+ ionic_qp_sq_init_cmb(dev, qp, udata, max_data);
1889+
1890+ if (qp->sq_cmb & IONIC_CMB_ENABLE)
1891+ rc = ionic_pgtbl_init(dev, buf, NULL,
1892+ (u64)qp->sq_cmb_pgid << PAGE_SHIFT,
1893+ 1, PAGE_SIZE);
1894+ else
1895+ rc = ionic_pgtbl_init(dev, buf,
1896+ qp->sq_umem, qp->sq.dma, 1, PAGE_SIZE);
1897+ if (rc)
1898+ goto err_sq_tbl;
1899+
1900+ return 0;
1901+
1902+err_sq_tbl:
1903+ ionic_qp_sq_destroy_cmb(dev, ctx, qp);
1904+ kfree(qp->sq_msn_idx);
1905+err_sq_msn:
1906+ kfree(qp->sq_meta);
1907+err_sq_meta:
1908+ if (qp->sq_umem)
1909+ ib_umem_release(qp->sq_umem);
1910+ else
1911+ ionic_queue_destroy(&qp->sq, dev->lif_cfg.hwdev);
1912+ return rc;
1913+}
1914+
1915+static void ionic_qp_sq_destroy(struct ionic_ibdev *dev,
1916+ struct ionic_ctx *ctx,
1917+ struct ionic_qp *qp)
1918+{
1919+ if (!qp->has_sq)
1920+ return;
1921+
1922+ ionic_qp_sq_destroy_cmb(dev, ctx, qp);
1923+
1924+ kfree(qp->sq_msn_idx);
1925+ kfree(qp->sq_meta);
1926+
1927+ if (qp->sq_umem)
1928+ ib_umem_release(qp->sq_umem);
1929+ else
1930+ ionic_queue_destroy(&qp->sq, dev->lif_cfg.hwdev);
1931+}
1932+
1933+static void ionic_qp_rq_init_cmb(struct ionic_ibdev *dev,
1934+ struct ionic_qp *qp,
1935+ struct ib_udata *udata)
1936+{
1937+ u8 expdb_stride_log2 = 0;
1938+ bool expdb;
1939+ int rc;
1940+
1941+ if (!(qp->rq_cmb & IONIC_CMB_ENABLE))
1942+ goto not_in_cmb;
1943+
1944+ if (qp->rq_cmb & ~IONIC_CMB_SUPPORTED) {
1945+ if (qp->rq_cmb & IONIC_CMB_REQUIRE)
1946+ goto not_in_cmb;
1947+
1948+ qp->rq_cmb &= IONIC_CMB_SUPPORTED;
1949+ }
1950+
1951+ if ((qp->rq_cmb & IONIC_CMB_EXPDB) && !dev->lif_cfg.rq_expdb) {
1952+ if (qp->rq_cmb & IONIC_CMB_REQUIRE)
1953+ goto not_in_cmb;
1954+
1955+ qp->rq_cmb &= ~IONIC_CMB_EXPDB;
1956+ }
1957+
1958+ qp->rq_cmb_order = order_base_2(qp->rq.size / PAGE_SIZE);
1959+
1960+ if (qp->rq_cmb_order >= IONIC_RQCMB_ORDER)
1961+ goto not_in_cmb;
1962+
1963+ if (qp->rq_cmb & IONIC_CMB_EXPDB)
1964+ expdb_stride_log2 = qp->rq.stride_log2;
1965+
1966+ rc = ionic_get_cmb(dev->lif_cfg.lif, &qp->rq_cmb_pgid,
1967+ &qp->rq_cmb_addr, qp->rq_cmb_order,
1968+ expdb_stride_log2, &expdb);
1969+ if (rc)
1970+ goto not_in_cmb;
1971+
1972+ if ((qp->rq_cmb & IONIC_CMB_EXPDB) && !expdb) {
1973+ if (qp->rq_cmb & IONIC_CMB_REQUIRE)
1974+ goto err_map;
1975+
1976+ qp->rq_cmb &= ~IONIC_CMB_EXPDB;
1977+ }
1978+
1979+ return;
1980+
1981+err_map:
1982+ ionic_put_cmb(dev->lif_cfg.lif, qp->rq_cmb_pgid, qp->rq_cmb_order);
1983+not_in_cmb:
1984+ if (qp->rq_cmb & IONIC_CMB_REQUIRE)
1985+ ibdev_dbg(&dev->ibdev, "could not place rq in cmb as required\n");
1986+
1987+ qp->rq_cmb = 0;
1988+ qp->rq_cmb_order = IONIC_RES_INVALID;
1989+ qp->rq_cmb_pgid = 0;
1990+ qp->rq_cmb_addr = 0;
1991+}
1992+
1993+static void ionic_qp_rq_destroy_cmb(struct ionic_ibdev *dev,
1994+ struct ionic_ctx *ctx,
1995+ struct ionic_qp *qp)
1996+{
1997+ if (!(qp->rq_cmb & IONIC_CMB_ENABLE))
1998+ return;
1999+
2000+ if (ctx)
2001+ rdma_user_mmap_entry_remove(qp->mmap_rq_cmb);
2002+
2003+ ionic_put_cmb(dev->lif_cfg.lif, qp->rq_cmb_pgid, qp->rq_cmb_order);
2004+}
2005+
2006+static int ionic_qp_rq_init(struct ionic_ibdev *dev, struct ionic_ctx *ctx,
2007+ struct ionic_qp *qp, struct ionic_qdesc *rq,
2008+ struct ionic_tbl_buf *buf, int max_wr, int max_sge,
2009+ int rq_spec, struct ib_udata *udata)
2010+{
2011+ int rc = 0, i;
2012+ u32 wqe_size;
2013+
2014+ if (!qp->has_rq) {
2015+ if (buf) {
2016+ buf->tbl_buf = NULL;
2017+ buf->tbl_limit = 0;
2018+ buf->tbl_pages = 0;
2019+ }
2020+ if (udata)
2021+ rc = ionic_validate_qdesc_zero(rq);
2022+
2023+ return rc;
2024+ }
2025+
2026+ rc = -EINVAL;
2027+
2028+ if (max_wr < 0 || max_wr > 0xffff)
2029+ return rc;
2030+
2031+ if (max_sge < 1)
2032+ return rc;
2033+
2034+ if (max_sge > min(ionic_v1_recv_wqe_max_sge(dev->lif_cfg.max_stride, 0, false),
2035+ IONIC_SPEC_HIGH))
2036+ return rc;
2037+
2038+ if (udata) {
2039+ rc = ionic_validate_qdesc(rq);
2040+ if (rc)
2041+ return rc;
2042+
2043+ qp->rq_spec = rq_spec;
2044+
2045+ qp->rq.ptr = NULL;
2046+ qp->rq.size = rq->size;
2047+ qp->rq.mask = rq->mask;
2048+ qp->rq.depth_log2 = rq->depth_log2;
2049+ qp->rq.stride_log2 = rq->stride_log2;
2050+
2051+ qp->rq_meta = NULL;
2052+
2053+ qp->rq_umem = ib_umem_get(&dev->ibdev, rq->addr, rq->size, 0);
2054+ if (IS_ERR(qp->rq_umem))
2055+ return PTR_ERR(qp->rq_umem);
2056+ } else {
2057+ qp->rq_umem = NULL;
2058+
2059+ qp->rq_spec = ionic_v1_use_spec_sge(max_sge, rq_spec);
2060+ if (rq_spec && !qp->rq_spec)
2061+ ibdev_dbg(&dev->ibdev,
2062+ "init rq: max_sge %u disables spec\n",
2063+ max_sge);
2064+
2065+ if (qp->rq_cmb & IONIC_CMB_EXPDB) {
2066+ wqe_size = ionic_v1_recv_wqe_min_size(max_sge,
2067+ qp->rq_spec,
2068+ true);
2069+
2070+ if (!ionic_expdb_wqe_size_supported(dev, wqe_size))
2071+ qp->rq_cmb &= ~IONIC_CMB_EXPDB;
2072+ }
2073+
2074+ if (!(qp->rq_cmb & IONIC_CMB_EXPDB))
2075+ wqe_size = ionic_v1_recv_wqe_min_size(max_sge,
2076+ qp->rq_spec,
2077+ false);
2078+
2079+ rc = ionic_queue_init(&qp->rq, dev->lif_cfg.hwdev,
2080+ max_wr, wqe_size);
2081+ if (rc)
2082+ return rc;
2083+
2084+ ionic_queue_dbell_init(&qp->rq, qp->qpid);
2085+
2086+ qp->rq_meta = kmalloc_array((u32)qp->rq.mask + 1,
2087+ sizeof(*qp->rq_meta),
2088+ GFP_KERNEL);
2089+ if (!qp->rq_meta) {
2090+ rc = -ENOMEM;
2091+ goto err_rq_meta;
2092+ }
2093+
2094+ for (i = 0; i < qp->rq.mask; ++i)
2095+ qp->rq_meta[i].next = &qp->rq_meta[i + 1];
2096+ qp->rq_meta[i].next = IONIC_META_LAST;
2097+ qp->rq_meta_head = &qp->rq_meta[0];
2098+ }
2099+
2100+ ionic_qp_rq_init_cmb(dev, qp, udata);
2101+
2102+ if (qp->rq_cmb & IONIC_CMB_ENABLE)
2103+ rc = ionic_pgtbl_init(dev, buf, NULL,
2104+ (u64)qp->rq_cmb_pgid << PAGE_SHIFT,
2105+ 1, PAGE_SIZE);
2106+ else
2107+ rc = ionic_pgtbl_init(dev, buf,
2108+ qp->rq_umem, qp->rq.dma, 1, PAGE_SIZE);
2109+ if (rc)
2110+ goto err_rq_tbl;
2111+
2112+ return 0;
2113+
2114+err_rq_tbl:
2115+ ionic_qp_rq_destroy_cmb(dev, ctx, qp);
2116+ kfree(qp->rq_meta);
2117+err_rq_meta:
2118+ if (qp->rq_umem)
2119+ ib_umem_release(qp->rq_umem);
2120+ else
2121+ ionic_queue_destroy(&qp->rq, dev->lif_cfg.hwdev);
2122+ return rc;
2123+}
2124+
2125+static void ionic_qp_rq_destroy(struct ionic_ibdev *dev,
2126+ struct ionic_ctx *ctx,
2127+ struct ionic_qp *qp)
2128+{
2129+ if (!qp->has_rq)
2130+ return;
2131+
2132+ ionic_qp_rq_destroy_cmb(dev, ctx, qp);
2133+
2134+ kfree(qp->rq_meta);
2135+
2136+ if (qp->rq_umem)
2137+ ib_umem_release(qp->rq_umem);
2138+ else
2139+ ionic_queue_destroy(&qp->rq, dev->lif_cfg.hwdev);
2140+}
2141+
2142+int ionic_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *attr,
2143+ struct ib_udata *udata)
2144+{
2145+ struct ionic_ibdev *dev = to_ionic_ibdev(ibqp->device);
2146+ struct ionic_tbl_buf sq_buf = {}, rq_buf = {};
2147+ struct ionic_pd *pd = to_ionic_pd(ibqp->pd);
2148+ struct ionic_qp *qp = to_ionic_qp(ibqp);
2149+ struct ionic_ctx *ctx =
2150+ rdma_udata_to_drv_context(udata, struct ionic_ctx, ibctx);
2151+ struct ionic_qp_resp resp = {};
2152+ struct ionic_qp_req req = {};
2153+ struct ionic_cq *cq;
2154+ u8 udma_mask;
2155+ void *entry;
2156+ int rc;
2157+
2158+ if (udata) {
2159+ rc = ib_copy_from_udata(&req, udata, sizeof(req));
2160+ if (rc)
2161+ return rc;
2162+ } else {
2163+ req.sq_spec = IONIC_SPEC_HIGH;
2164+ req.rq_spec = IONIC_SPEC_HIGH;
2165+ }
2166+
2167+ if (attr->qp_type == IB_QPT_SMI || attr->qp_type > IB_QPT_UD)
2168+ return -EOPNOTSUPP;
2169+
2170+ qp->state = IB_QPS_RESET;
2171+
2172+ INIT_LIST_HEAD(&qp->cq_poll_sq);
2173+ INIT_LIST_HEAD(&qp->cq_flush_sq);
2174+ INIT_LIST_HEAD(&qp->cq_flush_rq);
2175+
2176+ spin_lock_init(&qp->sq_lock);
2177+ spin_lock_init(&qp->rq_lock);
2178+
2179+ qp->has_sq = 1;
2180+ qp->has_rq = 1;
2181+
2182+ if (attr->qp_type == IB_QPT_GSI) {
2183+ rc = ionic_get_gsi_qpid(dev, &qp->qpid);
2184+ } else {
2185+ udma_mask = BIT(dev->lif_cfg.udma_count) - 1;
2186+
2187+ if (qp->has_sq)
2188+ udma_mask &= to_ionic_vcq(attr->send_cq)->udma_mask;
2189+
2190+ if (qp->has_rq)
2191+ udma_mask &= to_ionic_vcq(attr->recv_cq)->udma_mask;
2192+
2193+ if (udata && req.udma_mask)
2194+ udma_mask &= req.udma_mask;
2195+
2196+ if (!udma_mask)
2197+ return -EINVAL;
2198+
2199+ rc = ionic_get_qpid(dev, &qp->qpid, &qp->udma_idx, udma_mask);
2200+ }
2201+ if (rc)
2202+ return rc;
2203+
2204+ qp->sig_all = attr->sq_sig_type == IB_SIGNAL_ALL_WR;
2205+ qp->has_ah = attr->qp_type == IB_QPT_RC;
2206+
2207+ if (qp->has_ah) {
2208+ qp->hdr = kzalloc(sizeof(*qp->hdr), GFP_KERNEL);
2209+ if (!qp->hdr) {
2210+ rc = -ENOMEM;
2211+ goto err_ah_alloc;
2212+ }
2213+
2214+ rc = ionic_get_ahid(dev, &qp->ahid);
2215+ if (rc)
2216+ goto err_ahid;
2217+ }
2218+
2219+ if (udata) {
2220+ if (req.rq_cmb & IONIC_CMB_ENABLE)
2221+ qp->rq_cmb = req.rq_cmb;
2222+
2223+ if (req.sq_cmb & IONIC_CMB_ENABLE)
2224+ qp->sq_cmb = req.sq_cmb;
2225+ }
2226+
2227+ rc = ionic_qp_sq_init(dev, ctx, qp, &req.sq, &sq_buf,
2228+ attr->cap.max_send_wr, attr->cap.max_send_sge,
2229+ attr->cap.max_inline_data, req.sq_spec, udata);
2230+ if (rc)
2231+ goto err_sq;
2232+
2233+ rc = ionic_qp_rq_init(dev, ctx, qp, &req.rq, &rq_buf,
2234+ attr->cap.max_recv_wr, attr->cap.max_recv_sge,
2235+ req.rq_spec, udata);
2236+ if (rc)
2237+ goto err_rq;
2238+
2239+ rc = ionic_create_qp_cmd(dev, pd,
2240+ to_ionic_vcq_cq(attr->send_cq, qp->udma_idx),
2241+ to_ionic_vcq_cq(attr->recv_cq, qp->udma_idx),
2242+ qp, &sq_buf, &rq_buf, attr);
2243+ if (rc)
2244+ goto err_cmd;
2245+
2246+ if (udata) {
2247+ resp.qpid = qp->qpid;
2248+ resp.udma_idx = qp->udma_idx;
2249+
2250+ if (qp->sq_cmb & IONIC_CMB_ENABLE) {
2251+ bool wc;
2252+
2253+ if ((qp->sq_cmb & (IONIC_CMB_WC | IONIC_CMB_UC)) ==
2254+ (IONIC_CMB_WC | IONIC_CMB_UC)) {
2255+ ibdev_dbg(&dev->ibdev,
2256+ "Both sq_cmb flags IONIC_CMB_WC and IONIC_CMB_UC are set, using default driver mapping\n");
2257+ qp->sq_cmb &= ~(IONIC_CMB_WC | IONIC_CMB_UC);
2258+ }
2259+
2260+ wc = (qp->sq_cmb & (IONIC_CMB_WC | IONIC_CMB_UC))
2261+ != IONIC_CMB_UC;
2262+
2263+ /* let userspace know the mapping */
2264+ if (wc)
2265+ qp->sq_cmb |= IONIC_CMB_WC;
2266+ else
2267+ qp->sq_cmb |= IONIC_CMB_UC;
2268+
2269+ qp->mmap_sq_cmb =
2270+ ionic_mmap_entry_insert(ctx,
2271+ qp->sq.size,
2272+ PHYS_PFN(qp->sq_cmb_addr),
2273+ wc ? IONIC_MMAP_WC : 0,
2274+ &resp.sq_cmb_offset);
2275+ if (!qp->mmap_sq_cmb) {
2276+ rc = -ENOMEM;
2277+ goto err_mmap_sq;
2278+ }
2279+
2280+ resp.sq_cmb = qp->sq_cmb;
2281+ }
2282+
2283+ if (qp->rq_cmb & IONIC_CMB_ENABLE) {
2284+ bool wc;
2285+
2286+ if ((qp->rq_cmb & (IONIC_CMB_WC | IONIC_CMB_UC)) ==
2287+ (IONIC_CMB_WC | IONIC_CMB_UC)) {
2288+ ibdev_dbg(&dev->ibdev,
2289+ "Both rq_cmb flags IONIC_CMB_WC and IONIC_CMB_UC are set, using default driver mapping\n");
2290+ qp->rq_cmb &= ~(IONIC_CMB_WC | IONIC_CMB_UC);
2291+ }
2292+
2293+ if (qp->rq_cmb & IONIC_CMB_EXPDB)
2294+ wc = (qp->rq_cmb & (IONIC_CMB_WC | IONIC_CMB_UC))
2295+ == IONIC_CMB_WC;
2296+ else
2297+ wc = (qp->rq_cmb & (IONIC_CMB_WC | IONIC_CMB_UC))
2298+ != IONIC_CMB_UC;
2299+
2300+ /* let userspace know the mapping */
2301+ if (wc)
2302+ qp->rq_cmb |= IONIC_CMB_WC;
2303+ else
2304+ qp->rq_cmb |= IONIC_CMB_UC;
2305+
2306+ qp->mmap_rq_cmb =
2307+ ionic_mmap_entry_insert(ctx,
2308+ qp->rq.size,
2309+ PHYS_PFN(qp->rq_cmb_addr),
2310+ wc ? IONIC_MMAP_WC : 0,
2311+ &resp.rq_cmb_offset);
2312+ if (!qp->mmap_rq_cmb) {
2313+ rc = -ENOMEM;
2314+ goto err_mmap_rq;
2315+ }
2316+
2317+ resp.rq_cmb = qp->rq_cmb;
2318+ }
2319+
2320+ rc = ib_copy_to_udata(udata, &resp, sizeof(resp));
2321+ if (rc)
2322+ goto err_resp;
2323+ }
2324+
2325+ ionic_pgtbl_unbuf(dev, &rq_buf);
2326+ ionic_pgtbl_unbuf(dev, &sq_buf);
2327+
2328+ qp->ibqp.qp_num = qp->qpid;
2329+
2330+ init_completion(&qp->qp_rel_comp);
2331+ kref_init(&qp->qp_kref);
2332+
2333+ entry = xa_store_irq(&dev->qp_tbl, qp->qpid, qp, GFP_KERNEL);
2334+ if (entry) {
2335+ if (!xa_is_err(entry))
2336+ rc = -EINVAL;
2337+ else
2338+ rc = xa_err(entry);
2339+
2340+ goto err_resp;
2341+ }
2342+
2343+ if (qp->has_sq) {
2344+ cq = to_ionic_vcq_cq(attr->send_cq, qp->udma_idx);
2345+
2346+ attr->cap.max_send_wr = qp->sq.mask;
2347+ attr->cap.max_send_sge =
2348+ ionic_v1_send_wqe_max_sge(qp->sq.stride_log2,
2349+ qp->sq_spec,
2350+ qp->sq_cmb & IONIC_CMB_EXPDB);
2351+ attr->cap.max_inline_data =
2352+ ionic_v1_send_wqe_max_data(qp->sq.stride_log2,
2353+ qp->sq_cmb &
2354+ IONIC_CMB_EXPDB);
2355+ qp->sq_cqid = cq->cqid;
2356+ }
2357+
2358+ if (qp->has_rq) {
2359+ cq = to_ionic_vcq_cq(attr->recv_cq, qp->udma_idx);
2360+
2361+ attr->cap.max_recv_wr = qp->rq.mask;
2362+ attr->cap.max_recv_sge =
2363+ ionic_v1_recv_wqe_max_sge(qp->rq.stride_log2,
2364+ qp->rq_spec,
2365+ qp->rq_cmb & IONIC_CMB_EXPDB);
2366+ qp->rq_cqid = cq->cqid;
2367+ }
2368+
2369+ return 0;
2370+
2371+err_resp:
2372+ if (udata && (qp->rq_cmb & IONIC_CMB_ENABLE))
2373+ rdma_user_mmap_entry_remove(qp->mmap_rq_cmb);
2374+err_mmap_rq:
2375+ if (udata && (qp->sq_cmb & IONIC_CMB_ENABLE))
2376+ rdma_user_mmap_entry_remove(qp->mmap_sq_cmb);
2377+err_mmap_sq:
2378+ ionic_destroy_qp_cmd(dev, qp->qpid);
2379+err_cmd:
2380+ ionic_pgtbl_unbuf(dev, &rq_buf);
2381+ ionic_qp_rq_destroy(dev, ctx, qp);
2382+err_rq:
2383+ ionic_pgtbl_unbuf(dev, &sq_buf);
2384+ ionic_qp_sq_destroy(dev, ctx, qp);
2385+err_sq:
2386+ if (qp->has_ah)
2387+ ionic_put_ahid(dev, qp->ahid);
2388+err_ahid:
2389+ kfree(qp->hdr);
2390+err_ah_alloc:
2391+ ionic_put_qpid(dev, qp->qpid);
2392+ return rc;
2393+}
2394+
2395+void ionic_notify_flush_cq(struct ionic_cq *cq)
2396+{
2397+ if (cq->flush && cq->vcq->ibcq.comp_handler)
2398+ cq->vcq->ibcq.comp_handler(&cq->vcq->ibcq,
2399+ cq->vcq->ibcq.cq_context);
2400+}
2401+
2402+static void ionic_notify_qp_cqs(struct ionic_ibdev *dev, struct ionic_qp *qp)
2403+{
2404+ if (qp->ibqp.send_cq)
2405+ ionic_notify_flush_cq(to_ionic_vcq_cq(qp->ibqp.send_cq,
2406+ qp->udma_idx));
2407+ if (qp->ibqp.recv_cq && qp->ibqp.recv_cq != qp->ibqp.send_cq)
2408+ ionic_notify_flush_cq(to_ionic_vcq_cq(qp->ibqp.recv_cq,
2409+ qp->udma_idx));
2410+}
2411+
2412+void ionic_flush_qp(struct ionic_ibdev *dev, struct ionic_qp *qp)
2413+{
2414+ unsigned long irqflags;
2415+ struct ionic_cq *cq;
2416+
2417+ if (qp->ibqp.send_cq) {
2418+ cq = to_ionic_vcq_cq(qp->ibqp.send_cq, qp->udma_idx);
2419+
2420+ /* Hold the CQ lock and QP sq_lock to set up flush */
2421+ spin_lock_irqsave(&cq->lock, irqflags);
2422+ spin_lock(&qp->sq_lock);
2423+ qp->sq_flush = true;
2424+ if (!ionic_queue_empty(&qp->sq)) {
2425+ cq->flush = true;
2426+ list_move_tail(&qp->cq_flush_sq, &cq->flush_sq);
2427+ }
2428+ spin_unlock(&qp->sq_lock);
2429+ spin_unlock_irqrestore(&cq->lock, irqflags);
2430+ }
2431+
2432+ if (qp->ibqp.recv_cq) {
2433+ cq = to_ionic_vcq_cq(qp->ibqp.recv_cq, qp->udma_idx);
2434+
2435+ /* Hold the CQ lock and QP rq_lock to set up flush */
2436+ spin_lock_irqsave(&cq->lock, irqflags);
2437+ spin_lock(&qp->rq_lock);
2438+ qp->rq_flush = true;
2439+ if (!ionic_queue_empty(&qp->rq)) {
2440+ cq->flush = true;
2441+ list_move_tail(&qp->cq_flush_rq, &cq->flush_rq);
2442+ }
2443+ spin_unlock(&qp->rq_lock);
2444+ spin_unlock_irqrestore(&cq->lock, irqflags);
2445+ }
2446+}
2447+
2448+static void ionic_clean_cq(struct ionic_cq *cq, u32 qpid)
2449+{
2450+ struct ionic_v1_cqe *qcqe;
2451+ int prod, qtf, qid, type;
2452+ bool color;
2453+
2454+ if (!cq->q.ptr)
2455+ return;
2456+
2457+ color = cq->color;
2458+ prod = cq->q.prod;
2459+ qcqe = ionic_queue_at(&cq->q, prod);
2460+
2461+ while (color == ionic_v1_cqe_color(qcqe)) {
2462+ qtf = ionic_v1_cqe_qtf(qcqe);
2463+ qid = ionic_v1_cqe_qtf_qid(qtf);
2464+ type = ionic_v1_cqe_qtf_type(qtf);
2465+
2466+ if (qid == qpid && type != IONIC_V1_CQE_TYPE_ADMIN)
2467+ ionic_v1_cqe_clean(qcqe);
2468+
2469+ prod = ionic_queue_next(&cq->q, prod);
2470+ qcqe = ionic_queue_at(&cq->q, prod);
2471+ color = ionic_color_wrap(prod, color);
2472+ }
2473+}
2474+
2475+static void ionic_reset_qp(struct ionic_ibdev *dev, struct ionic_qp *qp)
2476+{
2477+ unsigned long irqflags;
2478+ struct ionic_cq *cq;
2479+ int i;
2480+
2481+ local_irq_save(irqflags);
2482+
2483+ if (qp->ibqp.send_cq) {
2484+ cq = to_ionic_vcq_cq(qp->ibqp.send_cq, qp->udma_idx);
2485+ spin_lock(&cq->lock);
2486+ ionic_clean_cq(cq, qp->qpid);
2487+ spin_unlock(&cq->lock);
2488+ }
2489+
2490+ if (qp->ibqp.recv_cq) {
2491+ cq = to_ionic_vcq_cq(qp->ibqp.recv_cq, qp->udma_idx);
2492+ spin_lock(&cq->lock);
2493+ ionic_clean_cq(cq, qp->qpid);
2494+ spin_unlock(&cq->lock);
2495+ }
2496+
2497+ if (qp->has_sq) {
2498+ spin_lock(&qp->sq_lock);
2499+ qp->sq_flush = false;
2500+ qp->sq_flush_rcvd = false;
2501+ qp->sq_msn_prod = 0;
2502+ qp->sq_msn_cons = 0;
2503+ qp->sq.prod = 0;
2504+ qp->sq.cons = 0;
2505+ spin_unlock(&qp->sq_lock);
2506+ }
2507+
2508+ if (qp->has_rq) {
2509+ spin_lock(&qp->rq_lock);
2510+ qp->rq_flush = false;
2511+ qp->rq.prod = 0;
2512+ qp->rq.cons = 0;
2513+ if (qp->rq_meta) {
2514+ for (i = 0; i < qp->rq.mask; ++i)
2515+ qp->rq_meta[i].next = &qp->rq_meta[i + 1];
2516+ qp->rq_meta[i].next = IONIC_META_LAST;
2517+ }
2518+ qp->rq_meta_head = &qp->rq_meta[0];
2519+ spin_unlock(&qp->rq_lock);
2520+ }
2521+
2522+ local_irq_restore(irqflags);
2523+}
2524+
2525+static bool ionic_qp_cur_state_is_ok(enum ib_qp_state q_state,
2526+ enum ib_qp_state attr_state)
2527+{
2528+ if (q_state == attr_state)
2529+ return true;
2530+
2531+ if (attr_state == IB_QPS_ERR)
2532+ return true;
2533+
2534+ if (attr_state == IB_QPS_SQE)
2535+ return q_state == IB_QPS_RTS || q_state == IB_QPS_SQD;
2536+
2537+ return false;
2538+}
2539+
2540+static int ionic_check_modify_qp(struct ionic_qp *qp, struct ib_qp_attr *attr,
2541+ int mask)
2542+{
2543+ enum ib_qp_state cur_state = (mask & IB_QP_CUR_STATE) ?
2544+ attr->cur_qp_state : qp->state;
2545+ enum ib_qp_state next_state = (mask & IB_QP_STATE) ?
2546+ attr->qp_state : cur_state;
2547+
2548+ if ((mask & IB_QP_CUR_STATE) &&
2549+ !ionic_qp_cur_state_is_ok(qp->state, attr->cur_qp_state))
2550+ return -EINVAL;
2551+
2552+ if (!ib_modify_qp_is_ok(cur_state, next_state, qp->ibqp.qp_type, mask))
2553+ return -EINVAL;
2554+
2555+ /* unprivileged qp not allowed privileged qkey */
2556+ if ((mask & IB_QP_QKEY) && (attr->qkey & 0x80000000) &&
2557+ qp->ibqp.uobject)
2558+ return -EPERM;
2559+
2560+ return 0;
2561+}
2562+
2563+int ionic_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int mask,
2564+ struct ib_udata *udata)
2565+{
2566+ struct ionic_ibdev *dev = to_ionic_ibdev(ibqp->device);
2567+ struct ionic_pd *pd = to_ionic_pd(ibqp->pd);
2568+ struct ionic_qp *qp = to_ionic_qp(ibqp);
2569+ int rc;
2570+
2571+ rc = ionic_check_modify_qp(qp, attr, mask);
2572+ if (rc)
2573+ return rc;
2574+
2575+ if (mask & IB_QP_CAP)
2576+ return -EINVAL;
2577+
2578+ rc = ionic_modify_qp_cmd(dev, pd, qp, attr, mask);
2579+ if (rc)
2580+ return rc;
2581+
2582+ if (mask & IB_QP_STATE) {
2583+ qp->state = attr->qp_state;
2584+
2585+ if (attr->qp_state == IB_QPS_ERR) {
2586+ ionic_flush_qp(dev, qp);
2587+ ionic_notify_qp_cqs(dev, qp);
2588+ } else if (attr->qp_state == IB_QPS_RESET) {
2589+ ionic_reset_qp(dev, qp);
2590+ }
2591+ }
2592+
2593+ return 0;
2594+}
2595+
2596+int ionic_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
2597+ int mask, struct ib_qp_init_attr *init_attr)
2598+{
2599+ struct ionic_ibdev *dev = to_ionic_ibdev(ibqp->device);
2600+ struct ionic_qp *qp = to_ionic_qp(ibqp);
2601+ int rc;
2602+
2603+ memset(attr, 0, sizeof(*attr));
2604+ memset(init_attr, 0, sizeof(*init_attr));
2605+
2606+ rc = ionic_query_qp_cmd(dev, qp, attr, mask);
2607+ if (rc)
2608+ return rc;
2609+
2610+ if (qp->has_sq)
2611+ attr->cap.max_send_wr = qp->sq.mask;
2612+
2613+ if (qp->has_rq)
2614+ attr->cap.max_recv_wr = qp->rq.mask;
2615+
2616+ init_attr->event_handler = ibqp->event_handler;
2617+ init_attr->qp_context = ibqp->qp_context;
2618+ init_attr->send_cq = ibqp->send_cq;
2619+ init_attr->recv_cq = ibqp->recv_cq;
2620+ init_attr->srq = ibqp->srq;
2621+ init_attr->xrcd = ibqp->xrcd;
2622+ init_attr->cap = attr->cap;
2623+ init_attr->sq_sig_type = qp->sig_all ?
2624+ IB_SIGNAL_ALL_WR : IB_SIGNAL_REQ_WR;
2625+ init_attr->qp_type = ibqp->qp_type;
2626+ init_attr->create_flags = 0;
2627+ init_attr->port_num = 0;
2628+ init_attr->rwq_ind_tbl = ibqp->rwq_ind_tbl;
2629+ init_attr->source_qpn = 0;
2630+
2631+ return rc;
2632+}
2633+
2634+int ionic_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
2635+{
2636+ struct ionic_ctx *ctx =
2637+ rdma_udata_to_drv_context(udata, struct ionic_ctx, ibctx);
2638+ struct ionic_ibdev *dev = to_ionic_ibdev(ibqp->device);
2639+ struct ionic_qp *qp = to_ionic_qp(ibqp);
2640+ unsigned long irqflags;
2641+ struct ionic_cq *cq;
2642+ int rc;
2643+
2644+ rc = ionic_destroy_qp_cmd(dev, qp->qpid);
2645+ if (rc)
2646+ return rc;
2647+
2648+ xa_erase_irq(&dev->qp_tbl, qp->qpid);
2649+
2650+ kref_put(&qp->qp_kref, ionic_qp_complete);
2651+ wait_for_completion(&qp->qp_rel_comp);
2652+
2653+ if (qp->ibqp.send_cq) {
2654+ cq = to_ionic_vcq_cq(qp->ibqp.send_cq, qp->udma_idx);
2655+ spin_lock_irqsave(&cq->lock, irqflags);
2656+ ionic_clean_cq(cq, qp->qpid);
2657+ list_del(&qp->cq_poll_sq);
2658+ list_del(&qp->cq_flush_sq);
2659+ spin_unlock_irqrestore(&cq->lock, irqflags);
2660+ }
2661+
2662+ if (qp->ibqp.recv_cq) {
2663+ cq = to_ionic_vcq_cq(qp->ibqp.recv_cq, qp->udma_idx);
2664+ spin_lock_irqsave(&cq->lock, irqflags);
2665+ ionic_clean_cq(cq, qp->qpid);
2666+ list_del(&qp->cq_flush_rq);
2667+ spin_unlock_irqrestore(&cq->lock, irqflags);
2668+ }
2669+
2670+ ionic_qp_rq_destroy(dev, ctx, qp);
2671+ ionic_qp_sq_destroy(dev, ctx, qp);
2672+ if (qp->has_ah) {
2673+ ionic_put_ahid(dev, qp->ahid);
2674+ kfree(qp->hdr);
2675+ }
2676+ ionic_put_qpid(dev, qp->qpid);
2677+
2678+ return 0;
2679+}
drivers/infiniband/hw/ionic/ionic_datapath.c+1399−0
@@ -0,0 +1,1399 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#include <linux/module.h>
5+#include <linux/printk.h>
6+#include <rdma/ib_addr.h>
7+#include <rdma/ib_user_verbs.h>
8+
9+#include "ionic_fw.h"
10+#include "ionic_ibdev.h"
11+
12+#define IONIC_OP(version, opname) \
13+ ((version) < 2 ? IONIC_V1_OP_##opname : IONIC_V2_OP_##opname)
14+
15+static bool ionic_next_cqe(struct ionic_ibdev *dev, struct ionic_cq *cq,
16+ struct ionic_v1_cqe **cqe)
17+{
18+ struct ionic_v1_cqe *qcqe = ionic_queue_at_prod(&cq->q);
19+
20+ if (unlikely(cq->color != ionic_v1_cqe_color(qcqe)))
21+ return false;
22+
23+ /* Prevent out-of-order reads of the CQE */
24+ dma_rmb();
25+
26+ *cqe = qcqe;
27+
28+ return true;
29+}
30+
31+static int ionic_flush_recv(struct ionic_qp *qp, struct ib_wc *wc)
32+{
33+ struct ionic_rq_meta *meta;
34+ struct ionic_v1_wqe *wqe;
35+
36+ if (!qp->rq_flush)
37+ return 0;
38+
39+ if (ionic_queue_empty(&qp->rq))
40+ return 0;
41+
42+ wqe = ionic_queue_at_cons(&qp->rq);
43+
44+ /* wqe_id must be a valid queue index */
45+ if (unlikely(wqe->base.wqe_id >> qp->rq.depth_log2)) {
46+ ibdev_warn(qp->ibqp.device,
47+ "flush qp %u recv index %llu invalid\n",
48+ qp->qpid, (unsigned long long)wqe->base.wqe_id);
49+ return -EIO;
50+ }
51+
52+ /* wqe_id must indicate a request that is outstanding */
53+ meta = &qp->rq_meta[wqe->base.wqe_id];
54+ if (unlikely(meta->next != IONIC_META_POSTED)) {
55+ ibdev_warn(qp->ibqp.device,
56+ "flush qp %u recv index %llu not posted\n",
57+ qp->qpid, (unsigned long long)wqe->base.wqe_id);
58+ return -EIO;
59+ }
60+
61+ ionic_queue_consume(&qp->rq);
62+
63+ memset(wc, 0, sizeof(*wc));
64+
65+ wc->status = IB_WC_WR_FLUSH_ERR;
66+ wc->wr_id = meta->wrid;
67+ wc->qp = &qp->ibqp;
68+
69+ meta->next = qp->rq_meta_head;
70+ qp->rq_meta_head = meta;
71+
72+ return 1;
73+}
74+
75+static int ionic_flush_recv_many(struct ionic_qp *qp,
76+ struct ib_wc *wc, int nwc)
77+{
78+ int rc = 0, npolled = 0;
79+
80+ while (npolled < nwc) {
81+ rc = ionic_flush_recv(qp, wc + npolled);
82+ if (rc <= 0)
83+ break;
84+
85+ npolled += rc;
86+ }
87+
88+ return npolled ?: rc;
89+}
90+
91+static int ionic_flush_send(struct ionic_qp *qp, struct ib_wc *wc)
92+{
93+ struct ionic_sq_meta *meta;
94+
95+ if (!qp->sq_flush)
96+ return 0;
97+
98+ if (ionic_queue_empty(&qp->sq))
99+ return 0;
100+
101+ meta = &qp->sq_meta[qp->sq.cons];
102+
103+ ionic_queue_consume(&qp->sq);
104+
105+ memset(wc, 0, sizeof(*wc));
106+
107+ wc->status = IB_WC_WR_FLUSH_ERR;
108+ wc->wr_id = meta->wrid;
109+ wc->qp = &qp->ibqp;
110+
111+ return 1;
112+}
113+
114+static int ionic_flush_send_many(struct ionic_qp *qp,
115+ struct ib_wc *wc, int nwc)
116+{
117+ int rc = 0, npolled = 0;
118+
119+ while (npolled < nwc) {
120+ rc = ionic_flush_send(qp, wc + npolled);
121+ if (rc <= 0)
122+ break;
123+
124+ npolled += rc;
125+ }
126+
127+ return npolled ?: rc;
128+}
129+
130+static int ionic_poll_recv(struct ionic_ibdev *dev, struct ionic_cq *cq,
131+ struct ionic_qp *cqe_qp, struct ionic_v1_cqe *cqe,
132+ struct ib_wc *wc)
133+{
134+ struct ionic_qp *qp = NULL;
135+ struct ionic_rq_meta *meta;
136+ u32 src_qpn, st_len;
137+ u16 vlan_tag;
138+ u8 op;
139+
140+ if (cqe_qp->rq_flush)
141+ return 0;
142+
143+ qp = cqe_qp;
144+
145+ st_len = be32_to_cpu(cqe->status_length);
146+
147+ /* ignore wqe_id in case of flush error */
148+ if (ionic_v1_cqe_error(cqe) && st_len == IONIC_STS_WQE_FLUSHED_ERR) {
149+ cqe_qp->rq_flush = true;
150+ cq->flush = true;
151+ list_move_tail(&qp->cq_flush_rq, &cq->flush_rq);
152+
153+ /* posted recvs (if any) flushed by ionic_flush_recv */
154+ return 0;
155+ }
156+
157+ /* there had better be something in the recv queue to complete */
158+ if (ionic_queue_empty(&qp->rq)) {
159+ ibdev_warn(&dev->ibdev, "qp %u is empty\n", qp->qpid);
160+ return -EIO;
161+ }
162+
163+ /* wqe_id must be a valid queue index */
164+ if (unlikely(cqe->recv.wqe_id >> qp->rq.depth_log2)) {
165+ ibdev_warn(&dev->ibdev,
166+ "qp %u recv index %llu invalid\n",
167+ qp->qpid, (unsigned long long)cqe->recv.wqe_id);
168+ return -EIO;
169+ }
170+
171+ /* wqe_id must indicate a request that is outstanding */
172+ meta = &qp->rq_meta[cqe->recv.wqe_id];
173+ if (unlikely(meta->next != IONIC_META_POSTED)) {
174+ ibdev_warn(&dev->ibdev,
175+ "qp %u recv index %llu not posted\n",
176+ qp->qpid, (unsigned long long)cqe->recv.wqe_id);
177+ return -EIO;
178+ }
179+
180+ meta->next = qp->rq_meta_head;
181+ qp->rq_meta_head = meta;
182+
183+ memset(wc, 0, sizeof(*wc));
184+
185+ wc->wr_id = meta->wrid;
186+
187+ wc->qp = &cqe_qp->ibqp;
188+
189+ if (ionic_v1_cqe_error(cqe)) {
190+ wc->vendor_err = st_len;
191+ wc->status = ionic_to_ib_status(st_len);
192+
193+ cqe_qp->rq_flush = true;
194+ cq->flush = true;
195+ list_move_tail(&qp->cq_flush_rq, &cq->flush_rq);
196+
197+ ibdev_warn(&dev->ibdev,
198+ "qp %d recv cqe with error\n", qp->qpid);
199+ print_hex_dump(KERN_WARNING, "cqe ", DUMP_PREFIX_OFFSET, 16, 1,
200+ cqe, BIT(cq->q.stride_log2), true);
201+ goto out;
202+ }
203+
204+ wc->vendor_err = 0;
205+ wc->status = IB_WC_SUCCESS;
206+
207+ src_qpn = be32_to_cpu(cqe->recv.src_qpn_op);
208+ op = src_qpn >> IONIC_V1_CQE_RECV_OP_SHIFT;
209+
210+ src_qpn &= IONIC_V1_CQE_RECV_QPN_MASK;
211+ op &= IONIC_V1_CQE_RECV_OP_MASK;
212+
213+ wc->opcode = IB_WC_RECV;
214+ switch (op) {
215+ case IONIC_V1_CQE_RECV_OP_RDMA_IMM:
216+ wc->opcode = IB_WC_RECV_RDMA_WITH_IMM;
217+ wc->wc_flags |= IB_WC_WITH_IMM;
218+ wc->ex.imm_data = cqe->recv.imm_data_rkey; /* be32 in wc */
219+ break;
220+ case IONIC_V1_CQE_RECV_OP_SEND_IMM:
221+ wc->wc_flags |= IB_WC_WITH_IMM;
222+ wc->ex.imm_data = cqe->recv.imm_data_rkey; /* be32 in wc */
223+ break;
224+ case IONIC_V1_CQE_RECV_OP_SEND_INV:
225+ wc->wc_flags |= IB_WC_WITH_INVALIDATE;
226+ wc->ex.invalidate_rkey = be32_to_cpu(cqe->recv.imm_data_rkey);
227+ break;
228+ }
229+
230+ wc->byte_len = st_len;
231+ wc->src_qp = src_qpn;
232+
233+ if (qp->ibqp.qp_type == IB_QPT_UD ||
234+ qp->ibqp.qp_type == IB_QPT_GSI) {
235+ wc->wc_flags |= IB_WC_GRH | IB_WC_WITH_SMAC;
236+ ether_addr_copy(wc->smac, cqe->recv.src_mac);
237+
238+ wc->wc_flags |= IB_WC_WITH_NETWORK_HDR_TYPE;
239+ if (ionic_v1_cqe_recv_is_ipv4(cqe))
240+ wc->network_hdr_type = RDMA_NETWORK_IPV4;
241+ else
242+ wc->network_hdr_type = RDMA_NETWORK_IPV6;
243+
244+ if (ionic_v1_cqe_recv_is_vlan(cqe))
245+ wc->wc_flags |= IB_WC_WITH_VLAN;
246+
247+ /* vlan_tag in cqe will be valid from dpath even if no vlan */
248+ vlan_tag = be16_to_cpu(cqe->recv.vlan_tag);
249+ wc->vlan_id = vlan_tag & 0xfff; /* 802.1q VID */
250+ wc->sl = vlan_tag >> VLAN_PRIO_SHIFT; /* 802.1q PCP */
251+ }
252+
253+ wc->pkey_index = 0;
254+ wc->port_num = 1;
255+
256+out:
257+ ionic_queue_consume(&qp->rq);
258+
259+ return 1;
260+}
261+
262+static bool ionic_peek_send(struct ionic_qp *qp)
263+{
264+ struct ionic_sq_meta *meta;
265+
266+ if (qp->sq_flush)
267+ return false;
268+
269+ /* completed all send queue requests */
270+ if (ionic_queue_empty(&qp->sq))
271+ return false;
272+
273+ meta = &qp->sq_meta[qp->sq.cons];
274+
275+ /* waiting for remote completion */
276+ if (meta->remote && meta->seq == qp->sq_msn_cons)
277+ return false;
278+
279+ /* waiting for local completion */
280+ if (!meta->remote && !meta->local_comp)
281+ return false;
282+
283+ return true;
284+}
285+
286+static int ionic_poll_send(struct ionic_ibdev *dev, struct ionic_cq *cq,
287+ struct ionic_qp *qp, struct ib_wc *wc)
288+{
289+ struct ionic_sq_meta *meta;
290+
291+ if (qp->sq_flush)
292+ return 0;
293+
294+ do {
295+ /* completed all send queue requests */
296+ if (ionic_queue_empty(&qp->sq))
297+ goto out_empty;
298+
299+ meta = &qp->sq_meta[qp->sq.cons];
300+
301+ /* waiting for remote completion */
302+ if (meta->remote && meta->seq == qp->sq_msn_cons)
303+ goto out_empty;
304+
305+ /* waiting for local completion */
306+ if (!meta->remote && !meta->local_comp)
307+ goto out_empty;
308+
309+ ionic_queue_consume(&qp->sq);
310+
311+ /* produce wc only if signaled or error status */
312+ } while (!meta->signal && meta->ibsts == IB_WC_SUCCESS);
313+
314+ memset(wc, 0, sizeof(*wc));
315+
316+ wc->status = meta->ibsts;
317+ wc->wr_id = meta->wrid;
318+ wc->qp = &qp->ibqp;
319+
320+ if (meta->ibsts == IB_WC_SUCCESS) {
321+ wc->byte_len = meta->len;
322+ wc->opcode = meta->ibop;
323+ } else {
324+ wc->vendor_err = meta->len;
325+
326+ qp->sq_flush = true;
327+ cq->flush = true;
328+ list_move_tail(&qp->cq_flush_sq, &cq->flush_sq);
329+ }
330+
331+ return 1;
332+
333+out_empty:
334+ if (qp->sq_flush_rcvd) {
335+ qp->sq_flush = true;
336+ cq->flush = true;
337+ list_move_tail(&qp->cq_flush_sq, &cq->flush_sq);
338+ }
339+ return 0;
340+}
341+
342+static int ionic_poll_send_many(struct ionic_ibdev *dev, struct ionic_cq *cq,
343+ struct ionic_qp *qp, struct ib_wc *wc, int nwc)
344+{
345+ int rc = 0, npolled = 0;
346+
347+ while (npolled < nwc) {
348+ rc = ionic_poll_send(dev, cq, qp, wc + npolled);
349+ if (rc <= 0)
350+ break;
351+
352+ npolled += rc;
353+ }
354+
355+ return npolled ?: rc;
356+}
357+
358+static int ionic_validate_cons(u16 prod, u16 cons,
359+ u16 comp, u16 mask)
360+{
361+ if (((prod - cons) & mask) <= ((comp - cons) & mask))
362+ return -EIO;
363+
364+ return 0;
365+}
366+
367+static int ionic_comp_msn(struct ionic_qp *qp, struct ionic_v1_cqe *cqe)
368+{
369+ struct ionic_sq_meta *meta;
370+ u16 cqe_seq, cqe_idx;
371+ int rc;
372+
373+ if (qp->sq_flush)
374+ return 0;
375+
376+ cqe_seq = be32_to_cpu(cqe->send.msg_msn) & qp->sq.mask;
377+
378+ rc = ionic_validate_cons(qp->sq_msn_prod,
379+ qp->sq_msn_cons,
380+ cqe_seq - 1,
381+ qp->sq.mask);
382+ if (rc) {
383+ ibdev_warn(qp->ibqp.device,
384+ "qp %u bad msn %#x seq %u for prod %u cons %u\n",
385+ qp->qpid, be32_to_cpu(cqe->send.msg_msn),
386+ cqe_seq, qp->sq_msn_prod, qp->sq_msn_cons);
387+ return rc;
388+ }
389+
390+ qp->sq_msn_cons = cqe_seq;
391+
392+ if (ionic_v1_cqe_error(cqe)) {
393+ cqe_idx = qp->sq_msn_idx[(cqe_seq - 1) & qp->sq.mask];
394+
395+ meta = &qp->sq_meta[cqe_idx];
396+ meta->len = be32_to_cpu(cqe->status_length);
397+ meta->ibsts = ionic_to_ib_status(meta->len);
398+
399+ ibdev_warn(qp->ibqp.device,
400+ "qp %d msn cqe with error\n", qp->qpid);
401+ print_hex_dump(KERN_WARNING, "cqe ", DUMP_PREFIX_OFFSET, 16, 1,
402+ cqe, sizeof(*cqe), true);
403+ }
404+
405+ return 0;
406+}
407+
408+static int ionic_comp_npg(struct ionic_qp *qp, struct ionic_v1_cqe *cqe)
409+{
410+ struct ionic_sq_meta *meta;
411+ u16 cqe_idx;
412+ u32 st_len;
413+
414+ if (qp->sq_flush)
415+ return 0;
416+
417+ st_len = be32_to_cpu(cqe->status_length);
418+
419+ if (ionic_v1_cqe_error(cqe) && st_len == IONIC_STS_WQE_FLUSHED_ERR) {
420+ /*
421+ * Flush cqe does not consume a wqe on the device, and maybe
422+ * no such work request is posted.
423+ *
424+ * The driver should begin flushing after the last indicated
425+ * normal or error completion. Here, only set a hint that the
426+ * flush request was indicated. In poll_send, if nothing more
427+ * can be polled normally, then begin flushing.
428+ */
429+ qp->sq_flush_rcvd = true;
430+ return 0;
431+ }
432+
433+ cqe_idx = cqe->send.npg_wqe_id & qp->sq.mask;
434+ meta = &qp->sq_meta[cqe_idx];
435+ meta->local_comp = true;
436+
437+ if (ionic_v1_cqe_error(cqe)) {
438+ meta->len = st_len;
439+ meta->ibsts = ionic_to_ib_status(st_len);
440+ meta->remote = false;
441+ ibdev_warn(qp->ibqp.device,
442+ "qp %d npg cqe with error\n", qp->qpid);
443+ print_hex_dump(KERN_WARNING, "cqe ", DUMP_PREFIX_OFFSET, 16, 1,
444+ cqe, sizeof(*cqe), true);
445+ }
446+
447+ return 0;
448+}
449+
450+static void ionic_reserve_sync_cq(struct ionic_ibdev *dev, struct ionic_cq *cq)
451+{
452+ if (!ionic_queue_empty(&cq->q)) {
453+ cq->credit += ionic_queue_length(&cq->q);
454+ cq->q.cons = cq->q.prod;
455+
456+ ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.cq_qtype,
457+ ionic_queue_dbell_val(&cq->q));
458+ }
459+}
460+
461+static void ionic_reserve_cq(struct ionic_ibdev *dev, struct ionic_cq *cq,
462+ int spend)
463+{
464+ cq->credit -= spend;
465+
466+ if (cq->credit <= 0)
467+ ionic_reserve_sync_cq(dev, cq);
468+}
469+
470+static int ionic_poll_vcq_cq(struct ionic_ibdev *dev,
471+ struct ionic_cq *cq,
472+ int nwc, struct ib_wc *wc)
473+{
474+ struct ionic_qp *qp, *qp_next;
475+ struct ionic_v1_cqe *cqe;
476+ int rc = 0, npolled = 0;
477+ unsigned long irqflags;
478+ u32 qtf, qid;
479+ bool peek;
480+ u8 type;
481+
482+ if (nwc < 1)
483+ return 0;
484+
485+ spin_lock_irqsave(&cq->lock, irqflags);
486+
487+ /* poll already indicated work completions for send queue */
488+ list_for_each_entry_safe(qp, qp_next, &cq->poll_sq, cq_poll_sq) {
489+ if (npolled == nwc)
490+ goto out;
491+
492+ spin_lock(&qp->sq_lock);
493+ rc = ionic_poll_send_many(dev, cq, qp, wc + npolled,
494+ nwc - npolled);
495+ spin_unlock(&qp->sq_lock);
496+
497+ if (rc > 0)
498+ npolled += rc;
499+
500+ if (npolled < nwc)
501+ list_del_init(&qp->cq_poll_sq);
502+ }
503+
504+ /* poll for more work completions */
505+ while (likely(ionic_next_cqe(dev, cq, &cqe))) {
506+ if (npolled == nwc)
507+ goto out;
508+
509+ qtf = ionic_v1_cqe_qtf(cqe);
510+ qid = ionic_v1_cqe_qtf_qid(qtf);
511+ type = ionic_v1_cqe_qtf_type(qtf);
512+
513+ /*
514+ * Safe to access QP without additional reference here as,
515+ * 1. We hold cq->lock throughout
516+ * 2. ionic_destroy_qp() acquires the same cq->lock before cleanup
517+ * 3. QP is removed from qp_tbl before any cleanup begins
518+ * This ensures no concurrent access between polling and destruction.
519+ */
520+ qp = xa_load(&dev->qp_tbl, qid);
521+ if (unlikely(!qp)) {
522+ ibdev_dbg(&dev->ibdev, "missing qp for qid %u\n", qid);
523+ goto cq_next;
524+ }
525+
526+ switch (type) {
527+ case IONIC_V1_CQE_TYPE_RECV:
528+ spin_lock(&qp->rq_lock);
529+ rc = ionic_poll_recv(dev, cq, qp, cqe, wc + npolled);
530+ spin_unlock(&qp->rq_lock);
531+
532+ if (rc < 0)
533+ goto out;
534+
535+ npolled += rc;
536+
537+ break;
538+
539+ case IONIC_V1_CQE_TYPE_SEND_MSN:
540+ spin_lock(&qp->sq_lock);
541+ rc = ionic_comp_msn(qp, cqe);
542+ if (!rc) {
543+ rc = ionic_poll_send_many(dev, cq, qp,
544+ wc + npolled,
545+ nwc - npolled);
546+ peek = ionic_peek_send(qp);
547+ }
548+ spin_unlock(&qp->sq_lock);
549+
550+ if (rc < 0)
551+ goto out;
552+
553+ npolled += rc;
554+
555+ if (peek)
556+ list_move_tail(&qp->cq_poll_sq, &cq->poll_sq);
557+ break;
558+
559+ case IONIC_V1_CQE_TYPE_SEND_NPG:
560+ spin_lock(&qp->sq_lock);
561+ rc = ionic_comp_npg(qp, cqe);
562+ if (!rc) {
563+ rc = ionic_poll_send_many(dev, cq, qp,
564+ wc + npolled,
565+ nwc - npolled);
566+ peek = ionic_peek_send(qp);
567+ }
568+ spin_unlock(&qp->sq_lock);
569+
570+ if (rc < 0)
571+ goto out;
572+
573+ npolled += rc;
574+
575+ if (peek)
576+ list_move_tail(&qp->cq_poll_sq, &cq->poll_sq);
577+ break;
578+
579+ default:
580+ ibdev_warn(&dev->ibdev,
581+ "unexpected cqe type %u\n", type);
582+ rc = -EIO;
583+ goto out;
584+ }
585+
586+cq_next:
587+ ionic_queue_produce(&cq->q);
588+ cq->color = ionic_color_wrap(cq->q.prod, cq->color);
589+ }
590+
591+ /* lastly, flush send and recv queues */
592+ if (likely(!cq->flush))
593+ goto out;
594+
595+ cq->flush = false;
596+
597+ list_for_each_entry_safe(qp, qp_next, &cq->flush_sq, cq_flush_sq) {
598+ if (npolled == nwc)
599+ goto out;
600+
601+ spin_lock(&qp->sq_lock);
602+ rc = ionic_flush_send_many(qp, wc + npolled, nwc - npolled);
603+ spin_unlock(&qp->sq_lock);
604+
605+ if (rc > 0)
606+ npolled += rc;
607+
608+ if (npolled < nwc)
609+ list_del_init(&qp->cq_flush_sq);
610+ else
611+ cq->flush = true;
612+ }
613+
614+ list_for_each_entry_safe(qp, qp_next, &cq->flush_rq, cq_flush_rq) {
615+ if (npolled == nwc)
616+ goto out;
617+
618+ spin_lock(&qp->rq_lock);
619+ rc = ionic_flush_recv_many(qp, wc + npolled, nwc - npolled);
620+ spin_unlock(&qp->rq_lock);
621+
622+ if (rc > 0)
623+ npolled += rc;
624+
625+ if (npolled < nwc)
626+ list_del_init(&qp->cq_flush_rq);
627+ else
628+ cq->flush = true;
629+ }
630+
631+out:
632+ /* in case credit was depleted (more work posted than cq depth) */
633+ if (cq->credit <= 0)
634+ ionic_reserve_sync_cq(dev, cq);
635+
636+ spin_unlock_irqrestore(&cq->lock, irqflags);
637+
638+ return npolled ?: rc;
639+}
640+
641+int ionic_poll_cq(struct ib_cq *ibcq, int nwc, struct ib_wc *wc)
642+{
643+ struct ionic_ibdev *dev = to_ionic_ibdev(ibcq->device);
644+ struct ionic_vcq *vcq = to_ionic_vcq(ibcq);
645+ int rc_tmp, rc = 0, npolled = 0;
646+ int cq_i, cq_x, cq_ix;
647+
648+ cq_x = vcq->poll_idx;
649+ vcq->poll_idx ^= dev->lif_cfg.udma_count - 1;
650+
651+ for (cq_i = 0; npolled < nwc && cq_i < dev->lif_cfg.udma_count; ++cq_i) {
652+ cq_ix = cq_i ^ cq_x;
653+
654+ if (!(vcq->udma_mask & BIT(cq_ix)))
655+ continue;
656+
657+ rc_tmp = ionic_poll_vcq_cq(dev, &vcq->cq[cq_ix],
658+ nwc - npolled,
659+ wc + npolled);
660+
661+ if (rc_tmp >= 0)
662+ npolled += rc_tmp;
663+ else if (!rc)
664+ rc = rc_tmp;
665+ }
666+
667+ return npolled ?: rc;
668+}
669+
670+static int ionic_req_notify_vcq_cq(struct ionic_ibdev *dev, struct ionic_cq *cq,
671+ enum ib_cq_notify_flags flags)
672+{
673+ u64 dbell_val = cq->q.dbell;
674+
675+ if (flags & IB_CQ_SOLICITED) {
676+ cq->arm_sol_prod = ionic_queue_next(&cq->q, cq->arm_sol_prod);
677+ dbell_val |= cq->arm_sol_prod | IONIC_CQ_RING_SOL;
678+ } else {
679+ cq->arm_any_prod = ionic_queue_next(&cq->q, cq->arm_any_prod);
680+ dbell_val |= cq->arm_any_prod | IONIC_CQ_RING_ARM;
681+ }
682+
683+ ionic_reserve_sync_cq(dev, cq);
684+
685+ ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.cq_qtype, dbell_val);
686+
687+ /*
688+ * IB_CQ_REPORT_MISSED_EVENTS:
689+ *
690+ * The queue index in ring zero guarantees no missed events.
691+ *
692+ * Here, we check if the color bit in the next cqe is flipped. If it
693+ * is flipped, then progress can be made by immediately polling the cq.
694+ * Still, the cq will be armed, and an event will be generated. The cq
695+ * may be empty when polled after the event, because the next poll
696+ * after arming the cq can empty it.
697+ */
698+ return (flags & IB_CQ_REPORT_MISSED_EVENTS) &&
699+ cq->color == ionic_v1_cqe_color(ionic_queue_at_prod(&cq->q));
700+}
701+
702+int ionic_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags)
703+{
704+ struct ionic_ibdev *dev = to_ionic_ibdev(ibcq->device);
705+ struct ionic_vcq *vcq = to_ionic_vcq(ibcq);
706+ int rc = 0, cq_i;
707+
708+ for (cq_i = 0; cq_i < dev->lif_cfg.udma_count; ++cq_i) {
709+ if (!(vcq->udma_mask & BIT(cq_i)))
710+ continue;
711+
712+ if (ionic_req_notify_vcq_cq(dev, &vcq->cq[cq_i], flags))
713+ rc = 1;
714+ }
715+
716+ return rc;
717+}
718+
719+static s64 ionic_prep_inline(void *data, u32 max_data,
720+ const struct ib_sge *ib_sgl, int num_sge)
721+{
722+ static const s64 bit_31 = 1u << 31;
723+ s64 len = 0, sg_len;
724+ int sg_i;
725+
726+ for (sg_i = 0; sg_i < num_sge; ++sg_i) {
727+ sg_len = ib_sgl[sg_i].length;
728+
729+ /* sge length zero means 2GB */
730+ if (unlikely(sg_len == 0))
731+ sg_len = bit_31;
732+
733+ /* greater than max inline data is invalid */
734+ if (unlikely(len + sg_len > max_data))
735+ return -EINVAL;
736+
737+ memcpy(data + len, (void *)ib_sgl[sg_i].addr, sg_len);
738+
739+ len += sg_len;
740+ }
741+
742+ return len;
743+}
744+
745+static s64 ionic_prep_pld(struct ionic_v1_wqe *wqe,
746+ union ionic_v1_pld *pld,
747+ int spec, u32 max_sge,
748+ const struct ib_sge *ib_sgl,
749+ int num_sge)
750+{
751+ static const s64 bit_31 = 1l << 31;
752+ struct ionic_sge *sgl;
753+ __be32 *spec32 = NULL;
754+ __be16 *spec16 = NULL;
755+ s64 len = 0, sg_len;
756+ int sg_i = 0;
757+
758+ if (unlikely(num_sge < 0 || (u32)num_sge > max_sge))
759+ return -EINVAL;
760+
761+ if (spec && num_sge > IONIC_V1_SPEC_FIRST_SGE) {
762+ sg_i = IONIC_V1_SPEC_FIRST_SGE;
763+
764+ if (num_sge > 8) {
765+ wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_SPEC16);
766+ spec16 = pld->spec16;
767+ } else {
768+ wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_SPEC32);
769+ spec32 = pld->spec32;
770+ }
771+ }
772+
773+ sgl = &pld->sgl[sg_i];
774+
775+ for (sg_i = 0; sg_i < num_sge; ++sg_i) {
776+ sg_len = ib_sgl[sg_i].length;
777+
778+ /* sge length zero means 2GB */
779+ if (unlikely(sg_len == 0))
780+ sg_len = bit_31;
781+
782+ /* greater than 2GB data is invalid */
783+ if (unlikely(len + sg_len > bit_31))
784+ return -EINVAL;
785+
786+ sgl[sg_i].va = cpu_to_be64(ib_sgl[sg_i].addr);
787+ sgl[sg_i].len = cpu_to_be32(sg_len);
788+ sgl[sg_i].lkey = cpu_to_be32(ib_sgl[sg_i].lkey);
789+
790+ if (spec32) {
791+ spec32[sg_i] = sgl[sg_i].len;
792+ } else if (spec16) {
793+ if (unlikely(sg_len > U16_MAX))
794+ return -EINVAL;
795+ spec16[sg_i] = cpu_to_be16(sg_len);
796+ }
797+
798+ len += sg_len;
799+ }
800+
801+ return len;
802+}
803+
804+static void ionic_prep_base(struct ionic_qp *qp,
805+ const struct ib_send_wr *wr,
806+ struct ionic_sq_meta *meta,
807+ struct ionic_v1_wqe *wqe)
808+{
809+ meta->wrid = wr->wr_id;
810+ meta->ibsts = IB_WC_SUCCESS;
811+ meta->signal = false;
812+ meta->local_comp = false;
813+
814+ wqe->base.wqe_id = qp->sq.prod;
815+
816+ if (wr->send_flags & IB_SEND_FENCE)
817+ wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_FENCE);
818+
819+ if (wr->send_flags & IB_SEND_SOLICITED)
820+ wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_SOL);
821+
822+ if (qp->sig_all || wr->send_flags & IB_SEND_SIGNALED) {
823+ wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_SIG);
824+ meta->signal = true;
825+ }
826+
827+ meta->seq = qp->sq_msn_prod;
828+ meta->remote =
829+ qp->ibqp.qp_type != IB_QPT_UD &&
830+ qp->ibqp.qp_type != IB_QPT_GSI &&
831+ !ionic_ibop_is_local(wr->opcode);
832+
833+ if (meta->remote) {
834+ qp->sq_msn_idx[meta->seq] = qp->sq.prod;
835+ qp->sq_msn_prod = ionic_queue_next(&qp->sq, qp->sq_msn_prod);
836+ }
837+
838+ ionic_queue_produce(&qp->sq);
839+}
840+
841+static int ionic_prep_common(struct ionic_qp *qp,
842+ const struct ib_send_wr *wr,
843+ struct ionic_sq_meta *meta,
844+ struct ionic_v1_wqe *wqe)
845+{
846+ s64 signed_len;
847+ u32 mval;
848+
849+ if (wr->send_flags & IB_SEND_INLINE) {
850+ wqe->base.num_sge_key = 0;
851+ wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_INL);
852+ mval = ionic_v1_send_wqe_max_data(qp->sq.stride_log2, false);
853+ signed_len = ionic_prep_inline(wqe->common.pld.data, mval,
854+ wr->sg_list, wr->num_sge);
855+ } else {
856+ wqe->base.num_sge_key = wr->num_sge;
857+ mval = ionic_v1_send_wqe_max_sge(qp->sq.stride_log2,
858+ qp->sq_spec,
859+ false);
860+ signed_len = ionic_prep_pld(wqe, &wqe->common.pld,
861+ qp->sq_spec, mval,
862+ wr->sg_list, wr->num_sge);
863+ }
864+
865+ if (unlikely(signed_len < 0))
866+ return signed_len;
867+
868+ meta->len = signed_len;
869+ wqe->common.length = cpu_to_be32(signed_len);
870+
871+ ionic_prep_base(qp, wr, meta, wqe);
872+
873+ return 0;
874+}
875+
876+static void ionic_prep_sq_wqe(struct ionic_qp *qp, void *wqe)
877+{
878+ memset(wqe, 0, 1u << qp->sq.stride_log2);
879+}
880+
881+static void ionic_prep_rq_wqe(struct ionic_qp *qp, void *wqe)
882+{
883+ memset(wqe, 0, 1u << qp->rq.stride_log2);
884+}
885+
886+static int ionic_prep_send(struct ionic_qp *qp,
887+ const struct ib_send_wr *wr)
888+{
889+ struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device);
890+ struct ionic_sq_meta *meta;
891+ struct ionic_v1_wqe *wqe;
892+
893+ meta = &qp->sq_meta[qp->sq.prod];
894+ wqe = ionic_queue_at_prod(&qp->sq);
895+
896+ ionic_prep_sq_wqe(qp, wqe);
897+
898+ meta->ibop = IB_WC_SEND;
899+
900+ switch (wr->opcode) {
901+ case IB_WR_SEND:
902+ wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, SEND);
903+ break;
904+ case IB_WR_SEND_WITH_IMM:
905+ wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, SEND_IMM);
906+ wqe->base.imm_data_key = wr->ex.imm_data;
907+ break;
908+ case IB_WR_SEND_WITH_INV:
909+ wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, SEND_INV);
910+ wqe->base.imm_data_key =
911+ cpu_to_be32(wr->ex.invalidate_rkey);
912+ break;
913+ default:
914+ return -EINVAL;
915+ }
916+
917+ return ionic_prep_common(qp, wr, meta, wqe);
918+}
919+
920+static int ionic_prep_send_ud(struct ionic_qp *qp,
921+ const struct ib_ud_wr *wr)
922+{
923+ struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device);
924+ struct ionic_sq_meta *meta;
925+ struct ionic_v1_wqe *wqe;
926+ struct ionic_ah *ah;
927+
928+ if (unlikely(!wr->ah))
929+ return -EINVAL;
930+
931+ ah = to_ionic_ah(wr->ah);
932+
933+ meta = &qp->sq_meta[qp->sq.prod];
934+ wqe = ionic_queue_at_prod(&qp->sq);
935+
936+ ionic_prep_sq_wqe(qp, wqe);
937+
938+ wqe->common.send.ah_id = cpu_to_be32(ah->ahid);
939+ wqe->common.send.dest_qpn = cpu_to_be32(wr->remote_qpn);
940+ wqe->common.send.dest_qkey = cpu_to_be32(wr->remote_qkey);
941+
942+ meta->ibop = IB_WC_SEND;
943+
944+ switch (wr->wr.opcode) {
945+ case IB_WR_SEND:
946+ wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, SEND);
947+ break;
948+ case IB_WR_SEND_WITH_IMM:
949+ wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, SEND_IMM);
950+ wqe->base.imm_data_key = wr->wr.ex.imm_data;
951+ break;
952+ default:
953+ return -EINVAL;
954+ }
955+
956+ return ionic_prep_common(qp, &wr->wr, meta, wqe);
957+}
958+
959+static int ionic_prep_rdma(struct ionic_qp *qp,
960+ const struct ib_rdma_wr *wr)
961+{
962+ struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device);
963+ struct ionic_sq_meta *meta;
964+ struct ionic_v1_wqe *wqe;
965+
966+ meta = &qp->sq_meta[qp->sq.prod];
967+ wqe = ionic_queue_at_prod(&qp->sq);
968+
969+ ionic_prep_sq_wqe(qp, wqe);
970+
971+ meta->ibop = IB_WC_RDMA_WRITE;
972+
973+ switch (wr->wr.opcode) {
974+ case IB_WR_RDMA_READ:
975+ if (wr->wr.send_flags & (IB_SEND_SOLICITED | IB_SEND_INLINE))
976+ return -EINVAL;
977+ meta->ibop = IB_WC_RDMA_READ;
978+ wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, RDMA_READ);
979+ break;
980+ case IB_WR_RDMA_WRITE:
981+ if (wr->wr.send_flags & IB_SEND_SOLICITED)
982+ return -EINVAL;
983+ wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, RDMA_WRITE);
984+ break;
985+ case IB_WR_RDMA_WRITE_WITH_IMM:
986+ wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, RDMA_WRITE_IMM);
987+ wqe->base.imm_data_key = wr->wr.ex.imm_data;
988+ break;
989+ default:
990+ return -EINVAL;
991+ }
992+
993+ wqe->common.rdma.remote_va_high = cpu_to_be32(wr->remote_addr >> 32);
994+ wqe->common.rdma.remote_va_low = cpu_to_be32(wr->remote_addr);
995+ wqe->common.rdma.remote_rkey = cpu_to_be32(wr->rkey);
996+
997+ return ionic_prep_common(qp, &wr->wr, meta, wqe);
998+}
999+
1000+static int ionic_prep_atomic(struct ionic_qp *qp,
1001+ const struct ib_atomic_wr *wr)
1002+{
1003+ struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device);
1004+ struct ionic_sq_meta *meta;
1005+ struct ionic_v1_wqe *wqe;
1006+
1007+ if (wr->wr.num_sge != 1 || wr->wr.sg_list[0].length != 8)
1008+ return -EINVAL;
1009+
1010+ if (wr->wr.send_flags & (IB_SEND_SOLICITED | IB_SEND_INLINE))
1011+ return -EINVAL;
1012+
1013+ meta = &qp->sq_meta[qp->sq.prod];
1014+ wqe = ionic_queue_at_prod(&qp->sq);
1015+
1016+ ionic_prep_sq_wqe(qp, wqe);
1017+
1018+ meta->ibop = IB_WC_RDMA_WRITE;
1019+
1020+ switch (wr->wr.opcode) {
1021+ case IB_WR_ATOMIC_CMP_AND_SWP:
1022+ meta->ibop = IB_WC_COMP_SWAP;
1023+ wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, ATOMIC_CS);
1024+ wqe->atomic.swap_add_high = cpu_to_be32(wr->swap >> 32);
1025+ wqe->atomic.swap_add_low = cpu_to_be32(wr->swap);
1026+ wqe->atomic.compare_high = cpu_to_be32(wr->compare_add >> 32);
1027+ wqe->atomic.compare_low = cpu_to_be32(wr->compare_add);
1028+ break;
1029+ case IB_WR_ATOMIC_FETCH_AND_ADD:
1030+ meta->ibop = IB_WC_FETCH_ADD;
1031+ wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, ATOMIC_FA);
1032+ wqe->atomic.swap_add_high = cpu_to_be32(wr->compare_add >> 32);
1033+ wqe->atomic.swap_add_low = cpu_to_be32(wr->compare_add);
1034+ break;
1035+ default:
1036+ return -EINVAL;
1037+ }
1038+
1039+ wqe->atomic.remote_va_high = cpu_to_be32(wr->remote_addr >> 32);
1040+ wqe->atomic.remote_va_low = cpu_to_be32(wr->remote_addr);
1041+ wqe->atomic.remote_rkey = cpu_to_be32(wr->rkey);
1042+
1043+ wqe->base.num_sge_key = 1;
1044+ wqe->atomic.sge.va = cpu_to_be64(wr->wr.sg_list[0].addr);
1045+ wqe->atomic.sge.len = cpu_to_be32(8);
1046+ wqe->atomic.sge.lkey = cpu_to_be32(wr->wr.sg_list[0].lkey);
1047+
1048+ return ionic_prep_common(qp, &wr->wr, meta, wqe);
1049+}
1050+
1051+static int ionic_prep_inv(struct ionic_qp *qp,
1052+ const struct ib_send_wr *wr)
1053+{
1054+ struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device);
1055+ struct ionic_sq_meta *meta;
1056+ struct ionic_v1_wqe *wqe;
1057+
1058+ if (wr->send_flags & (IB_SEND_SOLICITED | IB_SEND_INLINE))
1059+ return -EINVAL;
1060+
1061+ meta = &qp->sq_meta[qp->sq.prod];
1062+ wqe = ionic_queue_at_prod(&qp->sq);
1063+
1064+ ionic_prep_sq_wqe(qp, wqe);
1065+
1066+ wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, LOCAL_INV);
1067+ wqe->base.imm_data_key = cpu_to_be32(wr->ex.invalidate_rkey);
1068+
1069+ meta->len = 0;
1070+ meta->ibop = IB_WC_LOCAL_INV;
1071+
1072+ ionic_prep_base(qp, wr, meta, wqe);
1073+
1074+ return 0;
1075+}
1076+
1077+static int ionic_prep_reg(struct ionic_qp *qp,
1078+ const struct ib_reg_wr *wr)
1079+{
1080+ struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device);
1081+ struct ionic_mr *mr = to_ionic_mr(wr->mr);
1082+ struct ionic_sq_meta *meta;
1083+ struct ionic_v1_wqe *wqe;
1084+ __le64 dma_addr;
1085+ int flags;
1086+
1087+ if (wr->wr.send_flags & (IB_SEND_SOLICITED | IB_SEND_INLINE))
1088+ return -EINVAL;
1089+
1090+ /* must call ib_map_mr_sg before posting reg wr */
1091+ if (!mr->buf.tbl_pages)
1092+ return -EINVAL;
1093+
1094+ meta = &qp->sq_meta[qp->sq.prod];
1095+ wqe = ionic_queue_at_prod(&qp->sq);
1096+
1097+ ionic_prep_sq_wqe(qp, wqe);
1098+
1099+ flags = to_ionic_mr_flags(wr->access);
1100+
1101+ wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, REG_MR);
1102+ wqe->base.num_sge_key = wr->key;
1103+ wqe->base.imm_data_key = cpu_to_be32(mr->ibmr.lkey);
1104+ wqe->reg_mr.va = cpu_to_be64(mr->ibmr.iova);
1105+ wqe->reg_mr.length = cpu_to_be64(mr->ibmr.length);
1106+ wqe->reg_mr.offset = ionic_pgtbl_off(&mr->buf, mr->ibmr.iova);
1107+ dma_addr = ionic_pgtbl_dma(&mr->buf, mr->ibmr.iova);
1108+ wqe->reg_mr.dma_addr = cpu_to_be64(le64_to_cpu(dma_addr));
1109+
1110+ wqe->reg_mr.map_count = cpu_to_be32(mr->buf.tbl_pages);
1111+ wqe->reg_mr.flags = cpu_to_be16(flags);
1112+ wqe->reg_mr.dir_size_log2 = 0;
1113+ wqe->reg_mr.page_size_log2 = order_base_2(mr->ibmr.page_size);
1114+
1115+ meta->len = 0;
1116+ meta->ibop = IB_WC_REG_MR;
1117+
1118+ ionic_prep_base(qp, &wr->wr, meta, wqe);
1119+
1120+ return 0;
1121+}
1122+
1123+static int ionic_prep_one_rc(struct ionic_qp *qp,
1124+ const struct ib_send_wr *wr)
1125+{
1126+ struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device);
1127+ int rc = 0;
1128+
1129+ switch (wr->opcode) {
1130+ case IB_WR_SEND:
1131+ case IB_WR_SEND_WITH_IMM:
1132+ case IB_WR_SEND_WITH_INV:
1133+ rc = ionic_prep_send(qp, wr);
1134+ break;
1135+ case IB_WR_RDMA_READ:
1136+ case IB_WR_RDMA_WRITE:
1137+ case IB_WR_RDMA_WRITE_WITH_IMM:
1138+ rc = ionic_prep_rdma(qp, rdma_wr(wr));
1139+ break;
1140+ case IB_WR_ATOMIC_CMP_AND_SWP:
1141+ case IB_WR_ATOMIC_FETCH_AND_ADD:
1142+ rc = ionic_prep_atomic(qp, atomic_wr(wr));
1143+ break;
1144+ case IB_WR_LOCAL_INV:
1145+ rc = ionic_prep_inv(qp, wr);
1146+ break;
1147+ case IB_WR_REG_MR:
1148+ rc = ionic_prep_reg(qp, reg_wr(wr));
1149+ break;
1150+ default:
1151+ ibdev_dbg(&dev->ibdev, "invalid opcode %d\n", wr->opcode);
1152+ rc = -EINVAL;
1153+ }
1154+
1155+ return rc;
1156+}
1157+
1158+static int ionic_prep_one_ud(struct ionic_qp *qp,
1159+ const struct ib_send_wr *wr)
1160+{
1161+ struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device);
1162+ int rc = 0;
1163+
1164+ switch (wr->opcode) {
1165+ case IB_WR_SEND:
1166+ case IB_WR_SEND_WITH_IMM:
1167+ rc = ionic_prep_send_ud(qp, ud_wr(wr));
1168+ break;
1169+ default:
1170+ ibdev_dbg(&dev->ibdev, "invalid opcode %d\n", wr->opcode);
1171+ rc = -EINVAL;
1172+ }
1173+
1174+ return rc;
1175+}
1176+
1177+static int ionic_prep_recv(struct ionic_qp *qp,
1178+ const struct ib_recv_wr *wr)
1179+{
1180+ struct ionic_rq_meta *meta;
1181+ struct ionic_v1_wqe *wqe;
1182+ s64 signed_len;
1183+ u32 mval;
1184+
1185+ wqe = ionic_queue_at_prod(&qp->rq);
1186+
1187+ /* if wqe is owned by device, caller can try posting again soon */
1188+ if (wqe->base.flags & cpu_to_be16(IONIC_V1_FLAG_FENCE))
1189+ return -EAGAIN;
1190+
1191+ meta = qp->rq_meta_head;
1192+ if (unlikely(meta == IONIC_META_LAST) ||
1193+ unlikely(meta == IONIC_META_POSTED))
1194+ return -EIO;
1195+
1196+ ionic_prep_rq_wqe(qp, wqe);
1197+
1198+ mval = ionic_v1_recv_wqe_max_sge(qp->rq.stride_log2, qp->rq_spec,
1199+ false);
1200+ signed_len = ionic_prep_pld(wqe, &wqe->recv.pld,
1201+ qp->rq_spec, mval,
1202+ wr->sg_list, wr->num_sge);
1203+ if (signed_len < 0)
1204+ return signed_len;
1205+
1206+ meta->wrid = wr->wr_id;
1207+
1208+ wqe->base.wqe_id = meta - qp->rq_meta;
1209+ wqe->base.num_sge_key = wr->num_sge;
1210+
1211+ /* total length for recv goes in base imm_data_key */
1212+ wqe->base.imm_data_key = cpu_to_be32(signed_len);
1213+
1214+ ionic_queue_produce(&qp->rq);
1215+
1216+ qp->rq_meta_head = meta->next;
1217+ meta->next = IONIC_META_POSTED;
1218+
1219+ return 0;
1220+}
1221+
1222+static int ionic_post_send_common(struct ionic_ibdev *dev,
1223+ struct ionic_vcq *vcq,
1224+ struct ionic_cq *cq,
1225+ struct ionic_qp *qp,
1226+ const struct ib_send_wr *wr,
1227+ const struct ib_send_wr **bad)
1228+{
1229+ unsigned long irqflags;
1230+ bool notify = false;
1231+ int spend, rc = 0;
1232+
1233+ if (!bad)
1234+ return -EINVAL;
1235+
1236+ if (!qp->has_sq) {
1237+ *bad = wr;
1238+ return -EINVAL;
1239+ }
1240+
1241+ if (qp->state < IB_QPS_RTS) {
1242+ *bad = wr;
1243+ return -EINVAL;
1244+ }
1245+
1246+ spin_lock_irqsave(&qp->sq_lock, irqflags);
1247+
1248+ while (wr) {
1249+ if (ionic_queue_full(&qp->sq)) {
1250+ ibdev_dbg(&dev->ibdev, "queue full");
1251+ rc = -ENOMEM;
1252+ goto out;
1253+ }
1254+
1255+ if (qp->ibqp.qp_type == IB_QPT_UD ||
1256+ qp->ibqp.qp_type == IB_QPT_GSI)
1257+ rc = ionic_prep_one_ud(qp, wr);
1258+ else
1259+ rc = ionic_prep_one_rc(qp, wr);
1260+ if (rc)
1261+ goto out;
1262+
1263+ wr = wr->next;
1264+ }
1265+
1266+out:
1267+ spin_unlock_irqrestore(&qp->sq_lock, irqflags);
1268+
1269+ spin_lock_irqsave(&cq->lock, irqflags);
1270+ spin_lock(&qp->sq_lock);
1271+
1272+ if (likely(qp->sq.prod != qp->sq_old_prod)) {
1273+ /* ring cq doorbell just in time */
1274+ spend = (qp->sq.prod - qp->sq_old_prod) & qp->sq.mask;
1275+ ionic_reserve_cq(dev, cq, spend);
1276+
1277+ qp->sq_old_prod = qp->sq.prod;
1278+
1279+ ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.sq_qtype,
1280+ ionic_queue_dbell_val(&qp->sq));
1281+ }
1282+
1283+ if (qp->sq_flush) {
1284+ notify = true;
1285+ cq->flush = true;
1286+ list_move_tail(&qp->cq_flush_sq, &cq->flush_sq);
1287+ }
1288+
1289+ spin_unlock(&qp->sq_lock);
1290+ spin_unlock_irqrestore(&cq->lock, irqflags);
1291+
1292+ if (notify && vcq->ibcq.comp_handler)
1293+ vcq->ibcq.comp_handler(&vcq->ibcq, vcq->ibcq.cq_context);
1294+
1295+ *bad = wr;
1296+ return rc;
1297+}
1298+
1299+static int ionic_post_recv_common(struct ionic_ibdev *dev,
1300+ struct ionic_vcq *vcq,
1301+ struct ionic_cq *cq,
1302+ struct ionic_qp *qp,
1303+ const struct ib_recv_wr *wr,
1304+ const struct ib_recv_wr **bad)
1305+{
1306+ unsigned long irqflags;
1307+ bool notify = false;
1308+ int spend, rc = 0;
1309+
1310+ if (!bad)
1311+ return -EINVAL;
1312+
1313+ if (!qp->has_rq) {
1314+ *bad = wr;
1315+ return -EINVAL;
1316+ }
1317+
1318+ if (qp->state < IB_QPS_INIT) {
1319+ *bad = wr;
1320+ return -EINVAL;
1321+ }
1322+
1323+ spin_lock_irqsave(&qp->rq_lock, irqflags);
1324+
1325+ while (wr) {
1326+ if (ionic_queue_full(&qp->rq)) {
1327+ ibdev_dbg(&dev->ibdev, "queue full");
1328+ rc = -ENOMEM;
1329+ goto out;
1330+ }
1331+
1332+ rc = ionic_prep_recv(qp, wr);
1333+ if (rc)
1334+ goto out;
1335+
1336+ wr = wr->next;
1337+ }
1338+
1339+out:
1340+ if (!cq) {
1341+ spin_unlock_irqrestore(&qp->rq_lock, irqflags);
1342+ goto out_unlocked;
1343+ }
1344+ spin_unlock_irqrestore(&qp->rq_lock, irqflags);
1345+
1346+ spin_lock_irqsave(&cq->lock, irqflags);
1347+ spin_lock(&qp->rq_lock);
1348+
1349+ if (likely(qp->rq.prod != qp->rq_old_prod)) {
1350+ /* ring cq doorbell just in time */
1351+ spend = (qp->rq.prod - qp->rq_old_prod) & qp->rq.mask;
1352+ ionic_reserve_cq(dev, cq, spend);
1353+
1354+ qp->rq_old_prod = qp->rq.prod;
1355+
1356+ ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.rq_qtype,
1357+ ionic_queue_dbell_val(&qp->rq));
1358+ }
1359+
1360+ if (qp->rq_flush) {
1361+ notify = true;
1362+ cq->flush = true;
1363+ list_move_tail(&qp->cq_flush_rq, &cq->flush_rq);
1364+ }
1365+
1366+ spin_unlock(&qp->rq_lock);
1367+ spin_unlock_irqrestore(&cq->lock, irqflags);
1368+
1369+ if (notify && vcq->ibcq.comp_handler)
1370+ vcq->ibcq.comp_handler(&vcq->ibcq, vcq->ibcq.cq_context);
1371+
1372+out_unlocked:
1373+ *bad = wr;
1374+ return rc;
1375+}
1376+
1377+int ionic_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,
1378+ const struct ib_send_wr **bad)
1379+{
1380+ struct ionic_ibdev *dev = to_ionic_ibdev(ibqp->device);
1381+ struct ionic_vcq *vcq = to_ionic_vcq(ibqp->send_cq);
1382+ struct ionic_qp *qp = to_ionic_qp(ibqp);
1383+ struct ionic_cq *cq =
1384+ to_ionic_vcq_cq(ibqp->send_cq, qp->udma_idx);
1385+
1386+ return ionic_post_send_common(dev, vcq, cq, qp, wr, bad);
1387+}
1388+
1389+int ionic_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,
1390+ const struct ib_recv_wr **bad)
1391+{
1392+ struct ionic_ibdev *dev = to_ionic_ibdev(ibqp->device);
1393+ struct ionic_vcq *vcq = to_ionic_vcq(ibqp->recv_cq);
1394+ struct ionic_qp *qp = to_ionic_qp(ibqp);
1395+ struct ionic_cq *cq =
1396+ to_ionic_vcq_cq(ibqp->recv_cq, qp->udma_idx);
1397+
1398+ return ionic_post_recv_common(dev, vcq, cq, qp, wr, bad);
1399+}
drivers/infiniband/hw/ionic/ionic_fw.h+1029−0
@@ -0,0 +1,1029 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#ifndef _IONIC_FW_H_
5+#define _IONIC_FW_H_
6+
7+#include <linux/kernel.h>
8+#include <rdma/ib_verbs.h>
9+
10+/* common for ib spec */
11+
12+#define IONIC_EXP_DBELL_SZ 8
13+
14+enum ionic_mrid_bits {
15+ IONIC_MRID_INDEX_SHIFT = 8,
16+};
17+
18+static inline u32 ionic_mrid(u32 index, u8 key)
19+{
20+ return (index << IONIC_MRID_INDEX_SHIFT) | key;
21+}
22+
23+static inline u32 ionic_mrid_index(u32 lrkey)
24+{
25+ return lrkey >> IONIC_MRID_INDEX_SHIFT;
26+}
27+
28+/* common to all versions */
29+
30+/* wqe scatter gather element */
31+struct ionic_sge {
32+ __be64 va;
33+ __be32 len;
34+ __be32 lkey;
35+};
36+
37+/* admin queue mr type */
38+enum ionic_mr_flags {
39+ /* bits that determine mr access */
40+ IONIC_MRF_LOCAL_WRITE = BIT(0),
41+ IONIC_MRF_REMOTE_WRITE = BIT(1),
42+ IONIC_MRF_REMOTE_READ = BIT(2),
43+ IONIC_MRF_REMOTE_ATOMIC = BIT(3),
44+ IONIC_MRF_MW_BIND = BIT(4),
45+ IONIC_MRF_ZERO_BASED = BIT(5),
46+ IONIC_MRF_ON_DEMAND = BIT(6),
47+ IONIC_MRF_PB = BIT(7),
48+ IONIC_MRF_ACCESS_MASK = BIT(12) - 1,
49+
50+ /* bits that determine mr type */
51+ IONIC_MRF_UKEY_EN = BIT(13),
52+ IONIC_MRF_IS_MW = BIT(14),
53+ IONIC_MRF_INV_EN = BIT(15),
54+
55+ /* base flags combinations for mr types */
56+ IONIC_MRF_USER_MR = 0,
57+ IONIC_MRF_PHYS_MR = (IONIC_MRF_UKEY_EN |
58+ IONIC_MRF_INV_EN),
59+ IONIC_MRF_MW_1 = (IONIC_MRF_UKEY_EN |
60+ IONIC_MRF_IS_MW),
61+ IONIC_MRF_MW_2 = (IONIC_MRF_UKEY_EN |
62+ IONIC_MRF_IS_MW |
63+ IONIC_MRF_INV_EN),
64+};
65+
66+static inline int to_ionic_mr_flags(int access)
67+{
68+ int flags = 0;
69+
70+ if (access & IB_ACCESS_LOCAL_WRITE)
71+ flags |= IONIC_MRF_LOCAL_WRITE;
72+
73+ if (access & IB_ACCESS_REMOTE_READ)
74+ flags |= IONIC_MRF_REMOTE_READ;
75+
76+ if (access & IB_ACCESS_REMOTE_WRITE)
77+ flags |= IONIC_MRF_REMOTE_WRITE;
78+
79+ if (access & IB_ACCESS_REMOTE_ATOMIC)
80+ flags |= IONIC_MRF_REMOTE_ATOMIC;
81+
82+ if (access & IB_ACCESS_MW_BIND)
83+ flags |= IONIC_MRF_MW_BIND;
84+
85+ if (access & IB_ZERO_BASED)
86+ flags |= IONIC_MRF_ZERO_BASED;
87+
88+ return flags;
89+}
90+
91+enum ionic_qp_flags {
92+ /* bits that determine qp access */
93+ IONIC_QPF_REMOTE_WRITE = BIT(0),
94+ IONIC_QPF_REMOTE_READ = BIT(1),
95+ IONIC_QPF_REMOTE_ATOMIC = BIT(2),
96+
97+ /* bits that determine other qp behavior */
98+ IONIC_QPF_SQ_PB = BIT(6),
99+ IONIC_QPF_RQ_PB = BIT(7),
100+ IONIC_QPF_SQ_SPEC = BIT(8),
101+ IONIC_QPF_RQ_SPEC = BIT(9),
102+ IONIC_QPF_REMOTE_PRIVILEGED = BIT(10),
103+ IONIC_QPF_SQ_DRAINING = BIT(11),
104+ IONIC_QPF_SQD_NOTIFY = BIT(12),
105+ IONIC_QPF_SQ_CMB = BIT(13),
106+ IONIC_QPF_RQ_CMB = BIT(14),
107+ IONIC_QPF_PRIVILEGED = BIT(15),
108+};
109+
110+static inline int from_ionic_qp_flags(int flags)
111+{
112+ int access_flags = 0;
113+
114+ if (flags & IONIC_QPF_REMOTE_WRITE)
115+ access_flags |= IB_ACCESS_REMOTE_WRITE;
116+
117+ if (flags & IONIC_QPF_REMOTE_READ)
118+ access_flags |= IB_ACCESS_REMOTE_READ;
119+
120+ if (flags & IONIC_QPF_REMOTE_ATOMIC)
121+ access_flags |= IB_ACCESS_REMOTE_ATOMIC;
122+
123+ return access_flags;
124+}
125+
126+static inline int to_ionic_qp_flags(int access, bool sqd_notify,
127+ bool sq_is_cmb, bool rq_is_cmb,
128+ bool sq_spec, bool rq_spec,
129+ bool privileged, bool remote_privileged)
130+{
131+ int flags = 0;
132+
133+ if (access & IB_ACCESS_REMOTE_WRITE)
134+ flags |= IONIC_QPF_REMOTE_WRITE;
135+
136+ if (access & IB_ACCESS_REMOTE_READ)
137+ flags |= IONIC_QPF_REMOTE_READ;
138+
139+ if (access & IB_ACCESS_REMOTE_ATOMIC)
140+ flags |= IONIC_QPF_REMOTE_ATOMIC;
141+
142+ if (sqd_notify)
143+ flags |= IONIC_QPF_SQD_NOTIFY;
144+
145+ if (sq_is_cmb)
146+ flags |= IONIC_QPF_SQ_CMB;
147+
148+ if (rq_is_cmb)
149+ flags |= IONIC_QPF_RQ_CMB;
150+
151+ if (sq_spec)
152+ flags |= IONIC_QPF_SQ_SPEC;
153+
154+ if (rq_spec)
155+ flags |= IONIC_QPF_RQ_SPEC;
156+
157+ if (privileged)
158+ flags |= IONIC_QPF_PRIVILEGED;
159+
160+ if (remote_privileged)
161+ flags |= IONIC_QPF_REMOTE_PRIVILEGED;
162+
163+ return flags;
164+}
165+
166+/* cqe non-admin status indicated in status_length field when err bit is set */
167+enum ionic_status {
168+ IONIC_STS_OK,
169+ IONIC_STS_LOCAL_LEN_ERR,
170+ IONIC_STS_LOCAL_QP_OPER_ERR,
171+ IONIC_STS_LOCAL_PROT_ERR,
172+ IONIC_STS_WQE_FLUSHED_ERR,
173+ IONIC_STS_MEM_MGMT_OPER_ERR,
174+ IONIC_STS_BAD_RESP_ERR,
175+ IONIC_STS_LOCAL_ACC_ERR,
176+ IONIC_STS_REMOTE_INV_REQ_ERR,
177+ IONIC_STS_REMOTE_ACC_ERR,
178+ IONIC_STS_REMOTE_OPER_ERR,
179+ IONIC_STS_RETRY_EXCEEDED,
180+ IONIC_STS_RNR_RETRY_EXCEEDED,
181+ IONIC_STS_XRC_VIO_ERR,
182+ IONIC_STS_LOCAL_SGL_INV_ERR,
183+};
184+
185+static inline int ionic_to_ib_status(int sts)
186+{
187+ switch (sts) {
188+ case IONIC_STS_OK:
189+ return IB_WC_SUCCESS;
190+ case IONIC_STS_LOCAL_LEN_ERR:
191+ return IB_WC_LOC_LEN_ERR;
192+ case IONIC_STS_LOCAL_QP_OPER_ERR:
193+ case IONIC_STS_LOCAL_SGL_INV_ERR:
194+ return IB_WC_LOC_QP_OP_ERR;
195+ case IONIC_STS_LOCAL_PROT_ERR:
196+ return IB_WC_LOC_PROT_ERR;
197+ case IONIC_STS_WQE_FLUSHED_ERR:
198+ return IB_WC_WR_FLUSH_ERR;
199+ case IONIC_STS_MEM_MGMT_OPER_ERR:
200+ return IB_WC_MW_BIND_ERR;
201+ case IONIC_STS_BAD_RESP_ERR:
202+ return IB_WC_BAD_RESP_ERR;
203+ case IONIC_STS_LOCAL_ACC_ERR:
204+ return IB_WC_LOC_ACCESS_ERR;
205+ case IONIC_STS_REMOTE_INV_REQ_ERR:
206+ return IB_WC_REM_INV_REQ_ERR;
207+ case IONIC_STS_REMOTE_ACC_ERR:
208+ return IB_WC_REM_ACCESS_ERR;
209+ case IONIC_STS_REMOTE_OPER_ERR:
210+ return IB_WC_REM_OP_ERR;
211+ case IONIC_STS_RETRY_EXCEEDED:
212+ return IB_WC_RETRY_EXC_ERR;
213+ case IONIC_STS_RNR_RETRY_EXCEEDED:
214+ return IB_WC_RNR_RETRY_EXC_ERR;
215+ case IONIC_STS_XRC_VIO_ERR:
216+ default:
217+ return IB_WC_GENERAL_ERR;
218+ }
219+}
220+
221+/* admin queue qp type */
222+enum ionic_qp_type {
223+ IONIC_QPT_RC,
224+ IONIC_QPT_UC,
225+ IONIC_QPT_RD,
226+ IONIC_QPT_UD,
227+ IONIC_QPT_SRQ,
228+ IONIC_QPT_XRC_INI,
229+ IONIC_QPT_XRC_TGT,
230+ IONIC_QPT_XRC_SRQ,
231+};
232+
233+static inline int to_ionic_qp_type(enum ib_qp_type type)
234+{
235+ switch (type) {
236+ case IB_QPT_GSI:
237+ case IB_QPT_UD:
238+ return IONIC_QPT_UD;
239+ case IB_QPT_RC:
240+ return IONIC_QPT_RC;
241+ case IB_QPT_UC:
242+ return IONIC_QPT_UC;
243+ case IB_QPT_XRC_INI:
244+ return IONIC_QPT_XRC_INI;
245+ case IB_QPT_XRC_TGT:
246+ return IONIC_QPT_XRC_TGT;
247+ default:
248+ return -EINVAL;
249+ }
250+}
251+
252+/* admin queue qp state */
253+enum ionic_qp_state {
254+ IONIC_QPS_RESET,
255+ IONIC_QPS_INIT,
256+ IONIC_QPS_RTR,
257+ IONIC_QPS_RTS,
258+ IONIC_QPS_SQD,
259+ IONIC_QPS_SQE,
260+ IONIC_QPS_ERR,
261+};
262+
263+static inline int from_ionic_qp_state(enum ionic_qp_state state)
264+{
265+ switch (state) {
266+ case IONIC_QPS_RESET:
267+ return IB_QPS_RESET;
268+ case IONIC_QPS_INIT:
269+ return IB_QPS_INIT;
270+ case IONIC_QPS_RTR:
271+ return IB_QPS_RTR;
272+ case IONIC_QPS_RTS:
273+ return IB_QPS_RTS;
274+ case IONIC_QPS_SQD:
275+ return IB_QPS_SQD;
276+ case IONIC_QPS_SQE:
277+ return IB_QPS_SQE;
278+ case IONIC_QPS_ERR:
279+ return IB_QPS_ERR;
280+ default:
281+ return -EINVAL;
282+ }
283+}
284+
285+static inline int to_ionic_qp_state(enum ib_qp_state state)
286+{
287+ switch (state) {
288+ case IB_QPS_RESET:
289+ return IONIC_QPS_RESET;
290+ case IB_QPS_INIT:
291+ return IONIC_QPS_INIT;
292+ case IB_QPS_RTR:
293+ return IONIC_QPS_RTR;
294+ case IB_QPS_RTS:
295+ return IONIC_QPS_RTS;
296+ case IB_QPS_SQD:
297+ return IONIC_QPS_SQD;
298+ case IB_QPS_SQE:
299+ return IONIC_QPS_SQE;
300+ case IB_QPS_ERR:
301+ return IONIC_QPS_ERR;
302+ default:
303+ return 0;
304+ }
305+}
306+
307+static inline int to_ionic_qp_modify_state(enum ib_qp_state to_state,
308+ enum ib_qp_state from_state)
309+{
310+ return to_ionic_qp_state(to_state) |
311+ (to_ionic_qp_state(from_state) << 4);
312+}
313+
314+/* fw abi v1 */
315+
316+/* data payload part of v1 wqe */
317+union ionic_v1_pld {
318+ struct ionic_sge sgl[2];
319+ __be32 spec32[8];
320+ __be16 spec16[16];
321+ __u8 data[32];
322+};
323+
324+/* completion queue v1 cqe */
325+struct ionic_v1_cqe {
326+ union {
327+ struct {
328+ __be16 cmd_idx;
329+ __u8 cmd_op;
330+ __u8 rsvd[17];
331+ __le16 old_sq_cindex;
332+ __le16 old_rq_cq_cindex;
333+ } admin;
334+ struct {
335+ __u64 wqe_id;
336+ __be32 src_qpn_op;
337+ __u8 src_mac[6];
338+ __be16 vlan_tag;
339+ __be32 imm_data_rkey;
340+ } recv;
341+ struct {
342+ __u8 rsvd[4];
343+ __be32 msg_msn;
344+ __u8 rsvd2[8];
345+ __u64 npg_wqe_id;
346+ } send;
347+ };
348+ __be32 status_length;
349+ __be32 qid_type_flags;
350+};
351+
352+/* bits for cqe recv */
353+enum ionic_v1_cqe_src_qpn_bits {
354+ IONIC_V1_CQE_RECV_QPN_MASK = 0xffffff,
355+ IONIC_V1_CQE_RECV_OP_SHIFT = 24,
356+
357+ /* MASK could be 0x3, but need 0x1f for makeshift values:
358+ * OP_TYPE_RDMA_OPER_WITH_IMM, OP_TYPE_SEND_RCVD
359+ */
360+ IONIC_V1_CQE_RECV_OP_MASK = 0x1f,
361+ IONIC_V1_CQE_RECV_OP_SEND = 0,
362+ IONIC_V1_CQE_RECV_OP_SEND_INV = 1,
363+ IONIC_V1_CQE_RECV_OP_SEND_IMM = 2,
364+ IONIC_V1_CQE_RECV_OP_RDMA_IMM = 3,
365+
366+ IONIC_V1_CQE_RECV_IS_IPV4 = BIT(7 + IONIC_V1_CQE_RECV_OP_SHIFT),
367+ IONIC_V1_CQE_RECV_IS_VLAN = BIT(6 + IONIC_V1_CQE_RECV_OP_SHIFT),
368+};
369+
370+/* bits for cqe qid_type_flags */
371+enum ionic_v1_cqe_qtf_bits {
372+ IONIC_V1_CQE_COLOR = BIT(0),
373+ IONIC_V1_CQE_ERROR = BIT(1),
374+ IONIC_V1_CQE_TYPE_SHIFT = 5,
375+ IONIC_V1_CQE_TYPE_MASK = 0x7,
376+ IONIC_V1_CQE_QID_SHIFT = 8,
377+
378+ IONIC_V1_CQE_TYPE_ADMIN = 0,
379+ IONIC_V1_CQE_TYPE_RECV = 1,
380+ IONIC_V1_CQE_TYPE_SEND_MSN = 2,
381+ IONIC_V1_CQE_TYPE_SEND_NPG = 3,
382+};
383+
384+static inline bool ionic_v1_cqe_color(struct ionic_v1_cqe *cqe)
385+{
386+ return cqe->qid_type_flags & cpu_to_be32(IONIC_V1_CQE_COLOR);
387+}
388+
389+static inline bool ionic_v1_cqe_error(struct ionic_v1_cqe *cqe)
390+{
391+ return cqe->qid_type_flags & cpu_to_be32(IONIC_V1_CQE_ERROR);
392+}
393+
394+static inline bool ionic_v1_cqe_recv_is_ipv4(struct ionic_v1_cqe *cqe)
395+{
396+ return cqe->recv.src_qpn_op & cpu_to_be32(IONIC_V1_CQE_RECV_IS_IPV4);
397+}
398+
399+static inline bool ionic_v1_cqe_recv_is_vlan(struct ionic_v1_cqe *cqe)
400+{
401+ return cqe->recv.src_qpn_op & cpu_to_be32(IONIC_V1_CQE_RECV_IS_VLAN);
402+}
403+
404+static inline void ionic_v1_cqe_clean(struct ionic_v1_cqe *cqe)
405+{
406+ cqe->qid_type_flags |= cpu_to_be32(~0u << IONIC_V1_CQE_QID_SHIFT);
407+}
408+
409+static inline u32 ionic_v1_cqe_qtf(struct ionic_v1_cqe *cqe)
410+{
411+ return be32_to_cpu(cqe->qid_type_flags);
412+}
413+
414+static inline u8 ionic_v1_cqe_qtf_type(u32 qtf)
415+{
416+ return (qtf >> IONIC_V1_CQE_TYPE_SHIFT) & IONIC_V1_CQE_TYPE_MASK;
417+}
418+
419+static inline u32 ionic_v1_cqe_qtf_qid(u32 qtf)
420+{
421+ return qtf >> IONIC_V1_CQE_QID_SHIFT;
422+}
423+
424+/* v1 base wqe header */
425+struct ionic_v1_base_hdr {
426+ __u64 wqe_id;
427+ __u8 op;
428+ __u8 num_sge_key;
429+ __be16 flags;
430+ __be32 imm_data_key;
431+};
432+
433+/* v1 receive wqe body */
434+struct ionic_v1_recv_bdy {
435+ __u8 rsvd[16];
436+ union ionic_v1_pld pld;
437+};
438+
439+/* v1 send/rdma wqe body (common, has sgl) */
440+struct ionic_v1_common_bdy {
441+ union {
442+ struct {
443+ __be32 ah_id;
444+ __be32 dest_qpn;
445+ __be32 dest_qkey;
446+ } send;
447+ struct {
448+ __be32 remote_va_high;
449+ __be32 remote_va_low;
450+ __be32 remote_rkey;
451+ } rdma;
452+ };
453+ __be32 length;
454+ union ionic_v1_pld pld;
455+};
456+
457+/* v1 atomic wqe body */
458+struct ionic_v1_atomic_bdy {
459+ __be32 remote_va_high;
460+ __be32 remote_va_low;
461+ __be32 remote_rkey;
462+ __be32 swap_add_high;
463+ __be32 swap_add_low;
464+ __be32 compare_high;
465+ __be32 compare_low;
466+ __u8 rsvd[4];
467+ struct ionic_sge sge;
468+};
469+
470+/* v1 reg mr wqe body */
471+struct ionic_v1_reg_mr_bdy {
472+ __be64 va;
473+ __be64 length;
474+ __be64 offset;
475+ __be64 dma_addr;
476+ __be32 map_count;
477+ __be16 flags;
478+ __u8 dir_size_log2;
479+ __u8 page_size_log2;
480+ __u8 rsvd[8];
481+};
482+
483+/* v1 bind mw wqe body */
484+struct ionic_v1_bind_mw_bdy {
485+ __be64 va;
486+ __be64 length;
487+ __be32 lkey;
488+ __be16 flags;
489+ __u8 rsvd[26];
490+};
491+
492+/* v1 send/recv wqe */
493+struct ionic_v1_wqe {
494+ struct ionic_v1_base_hdr base;
495+ union {
496+ struct ionic_v1_recv_bdy recv;
497+ struct ionic_v1_common_bdy common;
498+ struct ionic_v1_atomic_bdy atomic;
499+ struct ionic_v1_reg_mr_bdy reg_mr;
500+ struct ionic_v1_bind_mw_bdy bind_mw;
501+ };
502+};
503+
504+/* queue pair v1 send opcodes */
505+enum ionic_v1_op {
506+ IONIC_V1_OP_SEND,
507+ IONIC_V1_OP_SEND_INV,
508+ IONIC_V1_OP_SEND_IMM,
509+ IONIC_V1_OP_RDMA_READ,
510+ IONIC_V1_OP_RDMA_WRITE,
511+ IONIC_V1_OP_RDMA_WRITE_IMM,
512+ IONIC_V1_OP_ATOMIC_CS,
513+ IONIC_V1_OP_ATOMIC_FA,
514+ IONIC_V1_OP_REG_MR,
515+ IONIC_V1_OP_LOCAL_INV,
516+ IONIC_V1_OP_BIND_MW,
517+
518+ /* flags */
519+ IONIC_V1_FLAG_FENCE = BIT(0),
520+ IONIC_V1_FLAG_SOL = BIT(1),
521+ IONIC_V1_FLAG_INL = BIT(2),
522+ IONIC_V1_FLAG_SIG = BIT(3),
523+
524+ /* flags last four bits for sgl spec format */
525+ IONIC_V1_FLAG_SPEC32 = (1u << 12),
526+ IONIC_V1_FLAG_SPEC16 = (2u << 12),
527+ IONIC_V1_SPEC_FIRST_SGE = 2,
528+};
529+
530+/* queue pair v2 send opcodes */
531+enum ionic_v2_op {
532+ IONIC_V2_OPSL_OUT = 0x20,
533+ IONIC_V2_OPSL_IMM = 0x40,
534+ IONIC_V2_OPSL_INV = 0x80,
535+
536+ IONIC_V2_OP_SEND = 0x0 | IONIC_V2_OPSL_OUT,
537+ IONIC_V2_OP_SEND_IMM = IONIC_V2_OP_SEND | IONIC_V2_OPSL_IMM,
538+ IONIC_V2_OP_SEND_INV = IONIC_V2_OP_SEND | IONIC_V2_OPSL_INV,
539+
540+ IONIC_V2_OP_RDMA_WRITE = 0x1 | IONIC_V2_OPSL_OUT,
541+ IONIC_V2_OP_RDMA_WRITE_IMM = IONIC_V2_OP_RDMA_WRITE | IONIC_V2_OPSL_IMM,
542+
543+ IONIC_V2_OP_RDMA_READ = 0x2,
544+
545+ IONIC_V2_OP_ATOMIC_CS = 0x4,
546+ IONIC_V2_OP_ATOMIC_FA = 0x5,
547+ IONIC_V2_OP_REG_MR = 0x6,
548+ IONIC_V2_OP_LOCAL_INV = 0x7,
549+ IONIC_V2_OP_BIND_MW = 0x8,
550+};
551+
552+static inline size_t ionic_v1_send_wqe_min_size(int min_sge, int min_data,
553+ int spec, bool expdb)
554+{
555+ size_t sz_wqe, sz_sgl, sz_data;
556+
557+ if (spec > IONIC_V1_SPEC_FIRST_SGE)
558+ min_sge += IONIC_V1_SPEC_FIRST_SGE;
559+
560+ if (expdb) {
561+ min_sge += 1;
562+ min_data += IONIC_EXP_DBELL_SZ;
563+ }
564+
565+ sz_wqe = sizeof(struct ionic_v1_wqe);
566+ sz_sgl = offsetof(struct ionic_v1_wqe, common.pld.sgl[min_sge]);
567+ sz_data = offsetof(struct ionic_v1_wqe, common.pld.data[min_data]);
568+
569+ if (sz_sgl > sz_wqe)
570+ sz_wqe = sz_sgl;
571+
572+ if (sz_data > sz_wqe)
573+ sz_wqe = sz_data;
574+
575+ return sz_wqe;
576+}
577+
578+static inline int ionic_v1_send_wqe_max_sge(u8 stride_log2, int spec,
579+ bool expdb)
580+{
581+ struct ionic_sge *sge = (void *)(1ull << stride_log2);
582+ struct ionic_v1_wqe *wqe = (void *)0;
583+ int num_sge = 0;
584+
585+ if (expdb)
586+ sge -= 1;
587+
588+ if (spec > IONIC_V1_SPEC_FIRST_SGE)
589+ num_sge = IONIC_V1_SPEC_FIRST_SGE;
590+
591+ num_sge = sge - &wqe->common.pld.sgl[num_sge];
592+
593+ if (spec && num_sge > spec)
594+ num_sge = spec;
595+
596+ return num_sge;
597+}
598+
599+static inline int ionic_v1_send_wqe_max_data(u8 stride_log2, bool expdb)
600+{
601+ struct ionic_v1_wqe *wqe = (void *)0;
602+ __u8 *data = (void *)(1ull << stride_log2);
603+
604+ if (expdb)
605+ data -= IONIC_EXP_DBELL_SZ;
606+
607+ return data - wqe->common.pld.data;
608+}
609+
610+static inline size_t ionic_v1_recv_wqe_min_size(int min_sge, int spec,
611+ bool expdb)
612+{
613+ size_t sz_wqe, sz_sgl;
614+
615+ if (spec > IONIC_V1_SPEC_FIRST_SGE)
616+ min_sge += IONIC_V1_SPEC_FIRST_SGE;
617+
618+ if (expdb)
619+ min_sge += 1;
620+
621+ sz_wqe = sizeof(struct ionic_v1_wqe);
622+ sz_sgl = offsetof(struct ionic_v1_wqe, recv.pld.sgl[min_sge]);
623+
624+ if (sz_sgl > sz_wqe)
625+ sz_wqe = sz_sgl;
626+
627+ return sz_wqe;
628+}
629+
630+static inline int ionic_v1_recv_wqe_max_sge(u8 stride_log2, int spec,
631+ bool expdb)
632+{
633+ struct ionic_sge *sge = (void *)(1ull << stride_log2);
634+ struct ionic_v1_wqe *wqe = (void *)0;
635+ int num_sge = 0;
636+
637+ if (expdb)
638+ sge -= 1;
639+
640+ if (spec > IONIC_V1_SPEC_FIRST_SGE)
641+ num_sge = IONIC_V1_SPEC_FIRST_SGE;
642+
643+ num_sge = sge - &wqe->recv.pld.sgl[num_sge];
644+
645+ if (spec && num_sge > spec)
646+ num_sge = spec;
647+
648+ return num_sge;
649+}
650+
651+static inline int ionic_v1_use_spec_sge(int min_sge, int spec)
652+{
653+ if (!spec || min_sge > spec)
654+ return 0;
655+
656+ if (min_sge <= IONIC_V1_SPEC_FIRST_SGE)
657+ return IONIC_V1_SPEC_FIRST_SGE;
658+
659+ return spec;
660+}
661+
662+struct ionic_admin_stats_hdr {
663+ __le64 dma_addr;
664+ __le32 length;
665+ __le32 id_ver;
666+ __u8 type_state;
667+} __packed;
668+
669+#define IONIC_ADMIN_STATS_HDRS_IN_V1_LEN 17
670+static_assert(sizeof(struct ionic_admin_stats_hdr) ==
671+ IONIC_ADMIN_STATS_HDRS_IN_V1_LEN);
672+
673+struct ionic_admin_create_ah {
674+ __le64 dma_addr;
675+ __le32 length;
676+ __le32 pd_id;
677+ __le32 id_ver;
678+ __le16 dbid_flags;
679+ __u8 csum_profile;
680+ __u8 crypto;
681+} __packed;
682+
683+#define IONIC_ADMIN_CREATE_AH_IN_V1_LEN 24
684+static_assert(sizeof(struct ionic_admin_create_ah) ==
685+ IONIC_ADMIN_CREATE_AH_IN_V1_LEN);
686+
687+struct ionic_admin_destroy_ah {
688+ __le32 ah_id;
689+} __packed;
690+
691+#define IONIC_ADMIN_DESTROY_AH_IN_V1_LEN 4
692+static_assert(sizeof(struct ionic_admin_destroy_ah) ==
693+ IONIC_ADMIN_DESTROY_AH_IN_V1_LEN);
694+
695+struct ionic_admin_query_ah {
696+ __le64 dma_addr;
697+} __packed;
698+
699+#define IONIC_ADMIN_QUERY_AH_IN_V1_LEN 8
700+static_assert(sizeof(struct ionic_admin_query_ah) ==
701+ IONIC_ADMIN_QUERY_AH_IN_V1_LEN);
702+
703+struct ionic_admin_create_mr {
704+ __le64 va;
705+ __le64 length;
706+ __le32 pd_id;
707+ __le32 id_ver;
708+ __le32 tbl_index;
709+ __le32 map_count;
710+ __le64 dma_addr;
711+ __le16 dbid_flags;
712+ __u8 pt_type;
713+ __u8 dir_size_log2;
714+ __u8 page_size_log2;
715+} __packed;
716+
717+#define IONIC_ADMIN_CREATE_MR_IN_V1_LEN 45
718+static_assert(sizeof(struct ionic_admin_create_mr) ==
719+ IONIC_ADMIN_CREATE_MR_IN_V1_LEN);
720+
721+struct ionic_admin_destroy_mr {
722+ __le32 mr_id;
723+} __packed;
724+
725+#define IONIC_ADMIN_DESTROY_MR_IN_V1_LEN 4
726+static_assert(sizeof(struct ionic_admin_destroy_mr) ==
727+ IONIC_ADMIN_DESTROY_MR_IN_V1_LEN);
728+
729+struct ionic_admin_create_cq {
730+ __le32 eq_id;
731+ __u8 depth_log2;
732+ __u8 stride_log2;
733+ __u8 dir_size_log2_rsvd;
734+ __u8 page_size_log2;
735+ __le32 cq_flags;
736+ __le32 id_ver;
737+ __le32 tbl_index;
738+ __le32 map_count;
739+ __le64 dma_addr;
740+ __le16 dbid_flags;
741+} __packed;
742+
743+#define IONIC_ADMIN_CREATE_CQ_IN_V1_LEN 34
744+static_assert(sizeof(struct ionic_admin_create_cq) ==
745+ IONIC_ADMIN_CREATE_CQ_IN_V1_LEN);
746+
747+struct ionic_admin_destroy_cq {
748+ __le32 cq_id;
749+} __packed;
750+
751+#define IONIC_ADMIN_DESTROY_CQ_IN_V1_LEN 4
752+static_assert(sizeof(struct ionic_admin_destroy_cq) ==
753+ IONIC_ADMIN_DESTROY_CQ_IN_V1_LEN);
754+
755+struct ionic_admin_create_qp {
756+ __le32 pd_id;
757+ __be32 priv_flags;
758+ __le32 sq_cq_id;
759+ __u8 sq_depth_log2;
760+ __u8 sq_stride_log2;
761+ __u8 sq_dir_size_log2_rsvd;
762+ __u8 sq_page_size_log2;
763+ __le32 sq_tbl_index_xrcd_id;
764+ __le32 sq_map_count;
765+ __le64 sq_dma_addr;
766+ __le32 rq_cq_id;
767+ __u8 rq_depth_log2;
768+ __u8 rq_stride_log2;
769+ __u8 rq_dir_size_log2_rsvd;
770+ __u8 rq_page_size_log2;
771+ __le32 rq_tbl_index_srq_id;
772+ __le32 rq_map_count;
773+ __le64 rq_dma_addr;
774+ __le32 id_ver;
775+ __le16 dbid_flags;
776+ __u8 type_state;
777+ __u8 rsvd;
778+} __packed;
779+
780+#define IONIC_ADMIN_CREATE_QP_IN_V1_LEN 64
781+static_assert(sizeof(struct ionic_admin_create_qp) ==
782+ IONIC_ADMIN_CREATE_QP_IN_V1_LEN);
783+
784+struct ionic_admin_destroy_qp {
785+ __le32 qp_id;
786+} __packed;
787+
788+#define IONIC_ADMIN_DESTROY_QP_IN_V1_LEN 4
789+static_assert(sizeof(struct ionic_admin_destroy_qp) ==
790+ IONIC_ADMIN_DESTROY_QP_IN_V1_LEN);
791+
792+struct ionic_admin_mod_qp {
793+ __be32 attr_mask;
794+ __u8 dcqcn_profile;
795+ __u8 tfp_csum_profile;
796+ __be16 access_flags;
797+ __le32 rq_psn;
798+ __le32 sq_psn;
799+ __le32 qkey_dest_qpn;
800+ __le32 rate_limit_kbps;
801+ __u8 pmtu;
802+ __u8 retry;
803+ __u8 rnr_timer;
804+ __u8 retry_timeout;
805+ __u8 rsq_depth;
806+ __u8 rrq_depth;
807+ __le16 pkey_id;
808+ __le32 ah_id_len;
809+ __u8 en_pcp;
810+ __u8 ip_dscp;
811+ __u8 rsvd2;
812+ __u8 type_state;
813+ union {
814+ struct {
815+ __le16 rsvd1;
816+ };
817+ __le32 rrq_index;
818+ };
819+ __le32 rsq_index;
820+ __le64 dma_addr;
821+ __le32 id_ver;
822+} __packed;
823+
824+#define IONIC_ADMIN_MODIFY_QP_IN_V1_LEN 60
825+static_assert(sizeof(struct ionic_admin_mod_qp) ==
826+ IONIC_ADMIN_MODIFY_QP_IN_V1_LEN);
827+
828+struct ionic_admin_query_qp {
829+ __le64 hdr_dma_addr;
830+ __le64 sq_dma_addr;
831+ __le64 rq_dma_addr;
832+ __le32 ah_id;
833+ __le32 id_ver;
834+ __le16 dbid_flags;
835+} __packed;
836+
837+#define IONIC_ADMIN_QUERY_QP_IN_V1_LEN 34
838+static_assert(sizeof(struct ionic_admin_query_qp) ==
839+ IONIC_ADMIN_QUERY_QP_IN_V1_LEN);
840+
841+#define ADMIN_WQE_STRIDE 64
842+#define ADMIN_WQE_HDR_LEN 4
843+
844+/* admin queue v1 wqe */
845+struct ionic_v1_admin_wqe {
846+ __u8 op;
847+ __u8 rsvd;
848+ __le16 len;
849+
850+ union {
851+ struct ionic_admin_stats_hdr stats;
852+ struct ionic_admin_create_ah create_ah;
853+ struct ionic_admin_destroy_ah destroy_ah;
854+ struct ionic_admin_query_ah query_ah;
855+ struct ionic_admin_create_mr create_mr;
856+ struct ionic_admin_destroy_mr destroy_mr;
857+ struct ionic_admin_create_cq create_cq;
858+ struct ionic_admin_destroy_cq destroy_cq;
859+ struct ionic_admin_create_qp create_qp;
860+ struct ionic_admin_destroy_qp destroy_qp;
861+ struct ionic_admin_mod_qp mod_qp;
862+ struct ionic_admin_query_qp query_qp;
863+ } cmd;
864+};
865+
866+/* side data for query qp */
867+struct ionic_v1_admin_query_qp_sq {
868+ __u8 rnr_timer;
869+ __u8 retry_timeout;
870+ __be16 access_perms_flags;
871+ __be16 rsvd;
872+ __be16 pkey_id;
873+ __be32 qkey_dest_qpn;
874+ __be32 rate_limit_kbps;
875+ __be32 rq_psn;
876+};
877+
878+struct ionic_v1_admin_query_qp_rq {
879+ __u8 state_pmtu;
880+ __u8 retry_rnrtry;
881+ __u8 rrq_depth;
882+ __u8 rsq_depth;
883+ __be32 sq_psn;
884+ __be16 access_perms_flags;
885+ __be16 rsvd;
886+};
887+
888+/* admin queue v1 opcodes */
889+enum ionic_v1_admin_op {
890+ IONIC_V1_ADMIN_NOOP,
891+ IONIC_V1_ADMIN_CREATE_CQ,
892+ IONIC_V1_ADMIN_CREATE_QP,
893+ IONIC_V1_ADMIN_CREATE_MR,
894+ IONIC_V1_ADMIN_STATS_HDRS,
895+ IONIC_V1_ADMIN_STATS_VALS,
896+ IONIC_V1_ADMIN_DESTROY_MR,
897+ IONIC_V1_ADMIN_RSVD_7, /* RESIZE_CQ */
898+ IONIC_V1_ADMIN_DESTROY_CQ,
899+ IONIC_V1_ADMIN_MODIFY_QP,
900+ IONIC_V1_ADMIN_QUERY_QP,
901+ IONIC_V1_ADMIN_DESTROY_QP,
902+ IONIC_V1_ADMIN_DEBUG,
903+ IONIC_V1_ADMIN_CREATE_AH,
904+ IONIC_V1_ADMIN_QUERY_AH,
905+ IONIC_V1_ADMIN_MODIFY_DCQCN,
906+ IONIC_V1_ADMIN_DESTROY_AH,
907+ IONIC_V1_ADMIN_QP_STATS_HDRS,
908+ IONIC_V1_ADMIN_QP_STATS_VALS,
909+ IONIC_V1_ADMIN_OPCODES_MAX,
910+};
911+
912+/* admin queue v1 cqe status */
913+enum ionic_v1_admin_status {
914+ IONIC_V1_ASTS_OK,
915+ IONIC_V1_ASTS_BAD_CMD,
916+ IONIC_V1_ASTS_BAD_INDEX,
917+ IONIC_V1_ASTS_BAD_STATE,
918+ IONIC_V1_ASTS_BAD_TYPE,
919+ IONIC_V1_ASTS_BAD_ATTR,
920+ IONIC_V1_ASTS_MSG_TOO_BIG,
921+};
922+
923+/* event queue v1 eqe */
924+struct ionic_v1_eqe {
925+ __be32 evt;
926+};
927+
928+/* bits for cqe queue_type_flags */
929+enum ionic_v1_eqe_evt_bits {
930+ IONIC_V1_EQE_COLOR = BIT(0),
931+ IONIC_V1_EQE_TYPE_SHIFT = 1,
932+ IONIC_V1_EQE_TYPE_MASK = 0x7,
933+ IONIC_V1_EQE_CODE_SHIFT = 4,
934+ IONIC_V1_EQE_CODE_MASK = 0xf,
935+ IONIC_V1_EQE_QID_SHIFT = 8,
936+
937+ /* cq events */
938+ IONIC_V1_EQE_TYPE_CQ = 0,
939+ /* cq normal events */
940+ IONIC_V1_EQE_CQ_NOTIFY = 0,
941+ /* cq error events */
942+ IONIC_V1_EQE_CQ_ERR = 8,
943+
944+ /* qp and srq events */
945+ IONIC_V1_EQE_TYPE_QP = 1,
946+ /* qp normal events */
947+ IONIC_V1_EQE_SRQ_LEVEL = 0,
948+ IONIC_V1_EQE_SQ_DRAIN = 1,
949+ IONIC_V1_EQE_QP_COMM_EST = 2,
950+ IONIC_V1_EQE_QP_LAST_WQE = 3,
951+ /* qp error events */
952+ IONIC_V1_EQE_QP_ERR = 8,
953+ IONIC_V1_EQE_QP_ERR_REQUEST = 9,
954+ IONIC_V1_EQE_QP_ERR_ACCESS = 10,
955+};
956+
957+enum ionic_tfp_csum_profiles {
958+ IONIC_TFP_CSUM_PROF_ETH_IPV4_UDP = 0,
959+ IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_UDP = 1,
960+ IONIC_TFP_CSUM_PROF_ETH_IPV6_UDP = 2,
961+ IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV6_UDP = 3,
962+ IONIC_TFP_CSUM_PROF_IPV4_UDP_VXLAN_ETH_QTAG_IPV4_UDP = 4,
963+ IONIC_TFP_CSUM_PROF_IPV4_UDP_VXLAN_ETH_QTAG_IPV6_UDP = 5,
964+ IONIC_TFP_CSUM_PROF_QTAG_IPV4_UDP_VXLAN_ETH_QTAG_IPV4_UDP = 6,
965+ IONIC_TFP_CSUM_PROF_QTAG_IPV4_UDP_VXLAN_ETH_QTAG_IPV6_UDP = 7,
966+ IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_UDP_ESP_IPV4_UDP = 8,
967+ IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_ESP_UDP = 9,
968+ IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_UDP_ESP_UDP = 10,
969+ IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV6_ESP_UDP = 11,
970+ IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_UDP_CSUM = 12,
971+};
972+
973+static inline bool ionic_v1_eqe_color(struct ionic_v1_eqe *eqe)
974+{
975+ return eqe->evt & cpu_to_be32(IONIC_V1_EQE_COLOR);
976+}
977+
978+static inline u32 ionic_v1_eqe_evt(struct ionic_v1_eqe *eqe)
979+{
980+ return be32_to_cpu(eqe->evt);
981+}
982+
983+static inline u8 ionic_v1_eqe_evt_type(u32 evt)
984+{
985+ return (evt >> IONIC_V1_EQE_TYPE_SHIFT) & IONIC_V1_EQE_TYPE_MASK;
986+}
987+
988+static inline u8 ionic_v1_eqe_evt_code(u32 evt)
989+{
990+ return (evt >> IONIC_V1_EQE_CODE_SHIFT) & IONIC_V1_EQE_CODE_MASK;
991+}
992+
993+static inline u32 ionic_v1_eqe_evt_qid(u32 evt)
994+{
995+ return evt >> IONIC_V1_EQE_QID_SHIFT;
996+}
997+
998+enum ionic_v1_stat_bits {
999+ IONIC_V1_STAT_TYPE_SHIFT = 28,
1000+ IONIC_V1_STAT_TYPE_NONE = 0,
1001+ IONIC_V1_STAT_TYPE_8 = 1,
1002+ IONIC_V1_STAT_TYPE_LE16 = 2,
1003+ IONIC_V1_STAT_TYPE_LE32 = 3,
1004+ IONIC_V1_STAT_TYPE_LE64 = 4,
1005+ IONIC_V1_STAT_TYPE_BE16 = 5,
1006+ IONIC_V1_STAT_TYPE_BE32 = 6,
1007+ IONIC_V1_STAT_TYPE_BE64 = 7,
1008+ IONIC_V1_STAT_OFF_MASK = BIT(IONIC_V1_STAT_TYPE_SHIFT) - 1,
1009+};
1010+
1011+struct ionic_v1_stat {
1012+ union {
1013+ __be32 be_type_off;
1014+ u32 type_off;
1015+ };
1016+ char name[28];
1017+};
1018+
1019+static inline int ionic_v1_stat_type(struct ionic_v1_stat *hdr)
1020+{
1021+ return hdr->type_off >> IONIC_V1_STAT_TYPE_SHIFT;
1022+}
1023+
1024+static inline unsigned int ionic_v1_stat_off(struct ionic_v1_stat *hdr)
1025+{
1026+ return hdr->type_off & IONIC_V1_STAT_OFF_MASK;
1027+}
1028+
1029+#endif /* _IONIC_FW_H_ */
drivers/infiniband/hw/ionic/ionic_hw_stats.c+484−0
@@ -0,0 +1,484 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#include <linux/dma-mapping.h>
5+
6+#include "ionic_fw.h"
7+#include "ionic_ibdev.h"
8+
9+static int ionic_v1_stat_normalize(struct ionic_v1_stat *hw_stats,
10+ int hw_stats_count)
11+{
12+ int hw_stat_i;
13+
14+ for (hw_stat_i = 0; hw_stat_i < hw_stats_count; ++hw_stat_i) {
15+ struct ionic_v1_stat *stat = &hw_stats[hw_stat_i];
16+
17+ stat->type_off = be32_to_cpu(stat->be_type_off);
18+ stat->name[sizeof(stat->name) - 1] = 0;
19+ if (ionic_v1_stat_type(stat) == IONIC_V1_STAT_TYPE_NONE)
20+ break;
21+ }
22+
23+ return hw_stat_i;
24+}
25+
26+static void ionic_fill_stats_desc(struct rdma_stat_desc *hw_stats_hdrs,
27+ struct ionic_v1_stat *hw_stats,
28+ int hw_stats_count)
29+{
30+ int hw_stat_i;
31+
32+ for (hw_stat_i = 0; hw_stat_i < hw_stats_count; ++hw_stat_i) {
33+ struct ionic_v1_stat *stat = &hw_stats[hw_stat_i];
34+
35+ hw_stats_hdrs[hw_stat_i].name = stat->name;
36+ }
37+}
38+
39+static u64 ionic_v1_stat_val(struct ionic_v1_stat *stat,
40+ void *vals_buf, size_t vals_len)
41+{
42+ unsigned int off = ionic_v1_stat_off(stat);
43+ int type = ionic_v1_stat_type(stat);
44+
45+#define __ionic_v1_stat_validate(__type) \
46+ ((off + sizeof(__type) <= vals_len) && \
47+ (IS_ALIGNED(off, sizeof(__type))))
48+
49+ switch (type) {
50+ case IONIC_V1_STAT_TYPE_8:
51+ if (__ionic_v1_stat_validate(u8))
52+ return *(u8 *)(vals_buf + off);
53+ break;
54+ case IONIC_V1_STAT_TYPE_LE16:
55+ if (__ionic_v1_stat_validate(__le16))
56+ return le16_to_cpu(*(__le16 *)(vals_buf + off));
57+ break;
58+ case IONIC_V1_STAT_TYPE_LE32:
59+ if (__ionic_v1_stat_validate(__le32))
60+ return le32_to_cpu(*(__le32 *)(vals_buf + off));
61+ break;
62+ case IONIC_V1_STAT_TYPE_LE64:
63+ if (__ionic_v1_stat_validate(__le64))
64+ return le64_to_cpu(*(__le64 *)(vals_buf + off));
65+ break;
66+ case IONIC_V1_STAT_TYPE_BE16:
67+ if (__ionic_v1_stat_validate(__be16))
68+ return be16_to_cpu(*(__be16 *)(vals_buf + off));
69+ break;
70+ case IONIC_V1_STAT_TYPE_BE32:
71+ if (__ionic_v1_stat_validate(__be32))
72+ return be32_to_cpu(*(__be32 *)(vals_buf + off));
73+ break;
74+ case IONIC_V1_STAT_TYPE_BE64:
75+ if (__ionic_v1_stat_validate(__be64))
76+ return be64_to_cpu(*(__be64 *)(vals_buf + off));
77+ break;
78+ }
79+
80+ return ~0ull;
81+#undef __ionic_v1_stat_validate
82+}
83+
84+static int ionic_hw_stats_cmd(struct ionic_ibdev *dev,
85+ dma_addr_t dma, size_t len, int qid, int op)
86+{
87+ struct ionic_admin_wr wr = {
88+ .work = COMPLETION_INITIALIZER_ONSTACK(wr.work),
89+ .wqe = {
90+ .op = op,
91+ .len = cpu_to_le16(IONIC_ADMIN_STATS_HDRS_IN_V1_LEN),
92+ .cmd.stats = {
93+ .dma_addr = cpu_to_le64(dma),
94+ .length = cpu_to_le32(len),
95+ .id_ver = cpu_to_le32(qid),
96+ },
97+ }
98+ };
99+
100+ if (dev->lif_cfg.admin_opcodes <= op)
101+ return -EBADRQC;
102+
103+ ionic_admin_post(dev, &wr);
104+
105+ return ionic_admin_wait(dev, &wr, IONIC_ADMIN_F_INTERRUPT);
106+}
107+
108+static int ionic_init_hw_stats(struct ionic_ibdev *dev)
109+{
110+ dma_addr_t hw_stats_dma;
111+ int rc, hw_stats_count;
112+
113+ if (dev->hw_stats_hdrs)
114+ return 0;
115+
116+ dev->hw_stats_count = 0;
117+
118+ /* buffer for current values from the device */
119+ dev->hw_stats_buf = kzalloc(PAGE_SIZE, GFP_KERNEL);
120+ if (!dev->hw_stats_buf) {
121+ rc = -ENOMEM;
122+ goto err_buf;
123+ }
124+
125+ /* buffer for names, sizes, offsets of values */
126+ dev->hw_stats = kzalloc(PAGE_SIZE, GFP_KERNEL);
127+ if (!dev->hw_stats) {
128+ rc = -ENOMEM;
129+ goto err_hw_stats;
130+ }
131+
132+ /* request the names, sizes, offsets */
133+ hw_stats_dma = dma_map_single(dev->lif_cfg.hwdev, dev->hw_stats,
134+ PAGE_SIZE, DMA_FROM_DEVICE);
135+ rc = dma_mapping_error(dev->lif_cfg.hwdev, hw_stats_dma);
136+ if (rc)
137+ goto err_dma;
138+
139+ rc = ionic_hw_stats_cmd(dev, hw_stats_dma, PAGE_SIZE, 0,
140+ IONIC_V1_ADMIN_STATS_HDRS);
141+ if (rc)
142+ goto err_cmd;
143+
144+ dma_unmap_single(dev->lif_cfg.hwdev, hw_stats_dma, PAGE_SIZE, DMA_FROM_DEVICE);
145+
146+ /* normalize and count the number of hw_stats */
147+ hw_stats_count =
148+ ionic_v1_stat_normalize(dev->hw_stats,
149+ PAGE_SIZE / sizeof(*dev->hw_stats));
150+ if (!hw_stats_count) {
151+ rc = -ENODATA;
152+ goto err_dma;
153+ }
154+
155+ dev->hw_stats_count = hw_stats_count;
156+
157+ /* alloc and init array of names, for alloc_hw_stats */
158+ dev->hw_stats_hdrs = kcalloc(hw_stats_count,
159+ sizeof(*dev->hw_stats_hdrs),
160+ GFP_KERNEL);
161+ if (!dev->hw_stats_hdrs) {
162+ rc = -ENOMEM;
163+ goto err_dma;
164+ }
165+
166+ ionic_fill_stats_desc(dev->hw_stats_hdrs, dev->hw_stats,
167+ hw_stats_count);
168+
169+ return 0;
170+
171+err_cmd:
172+ dma_unmap_single(dev->lif_cfg.hwdev, hw_stats_dma, PAGE_SIZE, DMA_FROM_DEVICE);
173+err_dma:
174+ kfree(dev->hw_stats);
175+err_hw_stats:
176+ kfree(dev->hw_stats_buf);
177+err_buf:
178+ dev->hw_stats_count = 0;
179+ dev->hw_stats = NULL;
180+ dev->hw_stats_buf = NULL;
181+ dev->hw_stats_hdrs = NULL;
182+ return rc;
183+}
184+
185+static struct rdma_hw_stats *ionic_alloc_hw_stats(struct ib_device *ibdev,
186+ u32 port)
187+{
188+ struct ionic_ibdev *dev = to_ionic_ibdev(ibdev);
189+
190+ if (port != 1)
191+ return NULL;
192+
193+ return rdma_alloc_hw_stats_struct(dev->hw_stats_hdrs,
194+ dev->hw_stats_count,
195+ RDMA_HW_STATS_DEFAULT_LIFESPAN);
196+}
197+
198+static int ionic_get_hw_stats(struct ib_device *ibdev,
199+ struct rdma_hw_stats *hw_stats,
200+ u32 port, int index)
201+{
202+ struct ionic_ibdev *dev = to_ionic_ibdev(ibdev);
203+ dma_addr_t hw_stats_dma;
204+ int rc, hw_stat_i;
205+
206+ if (port != 1)
207+ return -EINVAL;
208+
209+ hw_stats_dma = dma_map_single(dev->lif_cfg.hwdev, dev->hw_stats_buf,
210+ PAGE_SIZE, DMA_FROM_DEVICE);
211+ rc = dma_mapping_error(dev->lif_cfg.hwdev, hw_stats_dma);
212+ if (rc)
213+ goto err_dma;
214+
215+ rc = ionic_hw_stats_cmd(dev, hw_stats_dma, PAGE_SIZE,
216+ 0, IONIC_V1_ADMIN_STATS_VALS);
217+ if (rc)
218+ goto err_cmd;
219+
220+ dma_unmap_single(dev->lif_cfg.hwdev, hw_stats_dma,
221+ PAGE_SIZE, DMA_FROM_DEVICE);
222+
223+ for (hw_stat_i = 0; hw_stat_i < dev->hw_stats_count; ++hw_stat_i)
224+ hw_stats->value[hw_stat_i] =
225+ ionic_v1_stat_val(&dev->hw_stats[hw_stat_i],
226+ dev->hw_stats_buf, PAGE_SIZE);
227+
228+ return hw_stat_i;
229+
230+err_cmd:
231+ dma_unmap_single(dev->lif_cfg.hwdev, hw_stats_dma,
232+ PAGE_SIZE, DMA_FROM_DEVICE);
233+err_dma:
234+ return rc;
235+}
236+
237+static struct rdma_hw_stats *
238+ionic_counter_alloc_stats(struct rdma_counter *counter)
239+{
240+ struct ionic_ibdev *dev = to_ionic_ibdev(counter->device);
241+ struct ionic_counter *cntr;
242+ int err;
243+
244+ cntr = kzalloc(sizeof(*cntr), GFP_KERNEL);
245+ if (!cntr)
246+ return NULL;
247+
248+ /* buffer for current values from the device */
249+ cntr->vals = kzalloc(PAGE_SIZE, GFP_KERNEL);
250+ if (!cntr->vals)
251+ goto err_vals;
252+
253+ err = xa_alloc(&dev->counter_stats->xa_counters, &counter->id,
254+ cntr,
255+ XA_LIMIT(0, IONIC_MAX_QPID),
256+ GFP_KERNEL);
257+ if (err)
258+ goto err_xa;
259+
260+ INIT_LIST_HEAD(&cntr->qp_list);
261+
262+ return rdma_alloc_hw_stats_struct(dev->counter_stats->stats_hdrs,
263+ dev->counter_stats->queue_stats_count,
264+ RDMA_HW_STATS_DEFAULT_LIFESPAN);
265+err_xa:
266+ kfree(cntr->vals);
267+err_vals:
268+ kfree(cntr);
269+
270+ return NULL;
271+}
272+
273+static int ionic_counter_dealloc(struct rdma_counter *counter)
274+{
275+ struct ionic_ibdev *dev = to_ionic_ibdev(counter->device);
276+ struct ionic_counter *cntr;
277+
278+ cntr = xa_erase(&dev->counter_stats->xa_counters, counter->id);
279+ if (!cntr)
280+ return -EINVAL;
281+
282+ kfree(cntr->vals);
283+ kfree(cntr);
284+
285+ return 0;
286+}
287+
288+static int ionic_counter_bind_qp(struct rdma_counter *counter,
289+ struct ib_qp *ibqp,
290+ u32 port)
291+{
292+ struct ionic_ibdev *dev = to_ionic_ibdev(counter->device);
293+ struct ionic_qp *qp = to_ionic_qp(ibqp);
294+ struct ionic_counter *cntr;
295+
296+ cntr = xa_load(&dev->counter_stats->xa_counters, counter->id);
297+ if (!cntr)
298+ return -EINVAL;
299+
300+ list_add_tail(&qp->qp_list_counter, &cntr->qp_list);
301+ ibqp->counter = counter;
302+
303+ return 0;
304+}
305+
306+static int ionic_counter_unbind_qp(struct ib_qp *ibqp, u32 port)
307+{
308+ struct ionic_qp *qp = to_ionic_qp(ibqp);
309+
310+ if (ibqp->counter) {
311+ list_del(&qp->qp_list_counter);
312+ ibqp->counter = NULL;
313+ }
314+
315+ return 0;
316+}
317+
318+static int ionic_get_qp_stats(struct ib_device *ibdev,
319+ struct rdma_hw_stats *hw_stats,
320+ u32 counter_id)
321+{
322+ struct ionic_ibdev *dev = to_ionic_ibdev(ibdev);
323+ struct ionic_counter_stats *cs;
324+ struct ionic_counter *cntr;
325+ dma_addr_t hw_stats_dma;
326+ struct ionic_qp *qp;
327+ int rc, stat_i = 0;
328+
329+ cs = dev->counter_stats;
330+ cntr = xa_load(&cs->xa_counters, counter_id);
331+ if (!cntr)
332+ return -EINVAL;
333+
334+ hw_stats_dma = dma_map_single(dev->lif_cfg.hwdev, cntr->vals,
335+ PAGE_SIZE, DMA_FROM_DEVICE);
336+ rc = dma_mapping_error(dev->lif_cfg.hwdev, hw_stats_dma);
337+ if (rc)
338+ return rc;
339+
340+ memset(hw_stats->value, 0, sizeof(u64) * hw_stats->num_counters);
341+
342+ list_for_each_entry(qp, &cntr->qp_list, qp_list_counter) {
343+ rc = ionic_hw_stats_cmd(dev, hw_stats_dma, PAGE_SIZE,
344+ qp->qpid,
345+ IONIC_V1_ADMIN_QP_STATS_VALS);
346+ if (rc)
347+ goto err_cmd;
348+
349+ for (stat_i = 0; stat_i < cs->queue_stats_count; ++stat_i)
350+ hw_stats->value[stat_i] +=
351+ ionic_v1_stat_val(&cs->hdr[stat_i],
352+ cntr->vals,
353+ PAGE_SIZE);
354+ }
355+
356+ dma_unmap_single(dev->lif_cfg.hwdev, hw_stats_dma, PAGE_SIZE, DMA_FROM_DEVICE);
357+ return stat_i;
358+
359+err_cmd:
360+ dma_unmap_single(dev->lif_cfg.hwdev, hw_stats_dma, PAGE_SIZE, DMA_FROM_DEVICE);
361+
362+ return rc;
363+}
364+
365+static int ionic_counter_update_stats(struct rdma_counter *counter)
366+{
367+ return ionic_get_qp_stats(counter->device, counter->stats, counter->id);
368+}
369+
370+static int ionic_alloc_counters(struct ionic_ibdev *dev)
371+{
372+ struct ionic_counter_stats *cs = dev->counter_stats;
373+ int rc, hw_stats_count;
374+ dma_addr_t hdr_dma;
375+
376+ /* buffer for names, sizes, offsets of values */
377+ cs->hdr = kzalloc(PAGE_SIZE, GFP_KERNEL);
378+ if (!cs->hdr)
379+ return -ENOMEM;
380+
381+ hdr_dma = dma_map_single(dev->lif_cfg.hwdev, cs->hdr,
382+ PAGE_SIZE, DMA_FROM_DEVICE);
383+ rc = dma_mapping_error(dev->lif_cfg.hwdev, hdr_dma);
384+ if (rc)
385+ goto err_dma;
386+
387+ rc = ionic_hw_stats_cmd(dev, hdr_dma, PAGE_SIZE, 0,
388+ IONIC_V1_ADMIN_QP_STATS_HDRS);
389+ if (rc)
390+ goto err_cmd;
391+
392+ dma_unmap_single(dev->lif_cfg.hwdev, hdr_dma, PAGE_SIZE, DMA_FROM_DEVICE);
393+
394+ /* normalize and count the number of hw_stats */
395+ hw_stats_count = ionic_v1_stat_normalize(cs->hdr,
396+ PAGE_SIZE / sizeof(*cs->hdr));
397+ if (!hw_stats_count) {
398+ rc = -ENODATA;
399+ goto err_dma;
400+ }
401+
402+ cs->queue_stats_count = hw_stats_count;
403+
404+ /* alloc and init array of names */
405+ cs->stats_hdrs = kcalloc(hw_stats_count, sizeof(*cs->stats_hdrs),
406+ GFP_KERNEL);
407+ if (!cs->stats_hdrs) {
408+ rc = -ENOMEM;
409+ goto err_dma;
410+ }
411+
412+ ionic_fill_stats_desc(cs->stats_hdrs, cs->hdr, hw_stats_count);
413+
414+ return 0;
415+
416+err_cmd:
417+ dma_unmap_single(dev->lif_cfg.hwdev, hdr_dma, PAGE_SIZE, DMA_FROM_DEVICE);
418+err_dma:
419+ kfree(cs->hdr);
420+
421+ return rc;
422+}
423+
424+static const struct ib_device_ops ionic_hw_stats_ops = {
425+ .driver_id = RDMA_DRIVER_IONIC,
426+ .alloc_hw_port_stats = ionic_alloc_hw_stats,
427+ .get_hw_stats = ionic_get_hw_stats,
428+};
429+
430+static const struct ib_device_ops ionic_counter_stats_ops = {
431+ .counter_alloc_stats = ionic_counter_alloc_stats,
432+ .counter_dealloc = ionic_counter_dealloc,
433+ .counter_bind_qp = ionic_counter_bind_qp,
434+ .counter_unbind_qp = ionic_counter_unbind_qp,
435+ .counter_update_stats = ionic_counter_update_stats,
436+};
437+
438+void ionic_stats_init(struct ionic_ibdev *dev)
439+{
440+ u16 stats_type = dev->lif_cfg.stats_type;
441+ int rc;
442+
443+ if (stats_type & IONIC_LIF_RDMA_STAT_GLOBAL) {
444+ rc = ionic_init_hw_stats(dev);
445+ if (rc)
446+ ibdev_dbg(&dev->ibdev, "Failed to init hw stats\n");
447+ else
448+ ib_set_device_ops(&dev->ibdev, &ionic_hw_stats_ops);
449+ }
450+
451+ if (stats_type & IONIC_LIF_RDMA_STAT_QP) {
452+ dev->counter_stats = kzalloc(sizeof(*dev->counter_stats),
453+ GFP_KERNEL);
454+ if (!dev->counter_stats)
455+ return;
456+
457+ rc = ionic_alloc_counters(dev);
458+ if (rc) {
459+ ibdev_dbg(&dev->ibdev, "Failed to init counter stats\n");
460+ kfree(dev->counter_stats);
461+ dev->counter_stats = NULL;
462+ return;
463+ }
464+
465+ xa_init_flags(&dev->counter_stats->xa_counters, XA_FLAGS_ALLOC);
466+
467+ ib_set_device_ops(&dev->ibdev, &ionic_counter_stats_ops);
468+ }
469+}
470+
471+void ionic_stats_cleanup(struct ionic_ibdev *dev)
472+{
473+ if (dev->counter_stats) {
474+ xa_destroy(&dev->counter_stats->xa_counters);
475+ kfree(dev->counter_stats->hdr);
476+ kfree(dev->counter_stats->stats_hdrs);
477+ kfree(dev->counter_stats);
478+ dev->counter_stats = NULL;
479+ }
480+
481+ kfree(dev->hw_stats);
482+ kfree(dev->hw_stats_buf);
483+ kfree(dev->hw_stats_hdrs);
484+}
drivers/infiniband/hw/ionic/ionic_ibdev.c+440−0
@@ -0,0 +1,440 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#include <linux/module.h>
5+#include <linux/printk.h>
6+#include <linux/pci.h>
7+#include <linux/irq.h>
8+#include <net/addrconf.h>
9+#include <rdma/ib_addr.h>
10+#include <rdma/ib_mad.h>
11+
12+#include "ionic_ibdev.h"
13+
14+#define DRIVER_DESCRIPTION "AMD Pensando RoCE HCA driver"
15+#define DEVICE_DESCRIPTION "AMD Pensando RoCE HCA"
16+
17+MODULE_AUTHOR("Allen Hubbe <allen.hubbe@amd.com>");
18+MODULE_DESCRIPTION(DRIVER_DESCRIPTION);
19+MODULE_LICENSE("GPL");
20+MODULE_IMPORT_NS("NET_IONIC");
21+
22+static int ionic_query_device(struct ib_device *ibdev,
23+ struct ib_device_attr *attr,
24+ struct ib_udata *udata)
25+{
26+ struct ionic_ibdev *dev = to_ionic_ibdev(ibdev);
27+ struct net_device *ndev;
28+
29+ ndev = ib_device_get_netdev(ibdev, 1);
30+ addrconf_ifid_eui48((u8 *)&attr->sys_image_guid, ndev);
31+ dev_put(ndev);
32+ attr->max_mr_size = dev->lif_cfg.npts_per_lif * PAGE_SIZE / 2;
33+ attr->page_size_cap = dev->lif_cfg.page_size_supported;
34+
35+ attr->vendor_id = to_pci_dev(dev->lif_cfg.hwdev)->vendor;
36+ attr->vendor_part_id = to_pci_dev(dev->lif_cfg.hwdev)->device;
37+
38+ attr->hw_ver = ionic_lif_asic_rev(dev->lif_cfg.lif);
39+ attr->fw_ver = 0;
40+ attr->max_qp = dev->lif_cfg.qp_count;
41+ attr->max_qp_wr = IONIC_MAX_DEPTH;
42+ attr->device_cap_flags =
43+ IB_DEVICE_MEM_WINDOW |
44+ IB_DEVICE_MEM_MGT_EXTENSIONS |
45+ IB_DEVICE_MEM_WINDOW_TYPE_2B |
46+ 0;
47+ attr->max_send_sge =
48+ min(ionic_v1_send_wqe_max_sge(dev->lif_cfg.max_stride, 0, false),
49+ IONIC_SPEC_HIGH);
50+ attr->max_recv_sge =
51+ min(ionic_v1_recv_wqe_max_sge(dev->lif_cfg.max_stride, 0, false),
52+ IONIC_SPEC_HIGH);
53+ attr->max_sge_rd = attr->max_send_sge;
54+ attr->max_cq = dev->lif_cfg.cq_count / dev->lif_cfg.udma_count;
55+ attr->max_cqe = IONIC_MAX_CQ_DEPTH - IONIC_CQ_GRACE;
56+ attr->max_mr = dev->lif_cfg.nmrs_per_lif;
57+ attr->max_pd = IONIC_MAX_PD;
58+ attr->max_qp_rd_atom = IONIC_MAX_RD_ATOM;
59+ attr->max_ee_rd_atom = 0;
60+ attr->max_res_rd_atom = IONIC_MAX_RD_ATOM;
61+ attr->max_qp_init_rd_atom = IONIC_MAX_RD_ATOM;
62+ attr->max_ee_init_rd_atom = 0;
63+ attr->atomic_cap = IB_ATOMIC_GLOB;
64+ attr->masked_atomic_cap = IB_ATOMIC_GLOB;
65+ attr->max_mw = dev->lif_cfg.nmrs_per_lif;
66+ attr->max_mcast_grp = 0;
67+ attr->max_mcast_qp_attach = 0;
68+ attr->max_ah = dev->lif_cfg.nahs_per_lif;
69+ attr->max_fast_reg_page_list_len = dev->lif_cfg.npts_per_lif / 2;
70+ attr->max_pkeys = IONIC_PKEY_TBL_LEN;
71+
72+ return 0;
73+}
74+
75+static int ionic_query_port(struct ib_device *ibdev, u32 port,
76+ struct ib_port_attr *attr)
77+{
78+ struct net_device *ndev;
79+
80+ if (port != 1)
81+ return -EINVAL;
82+
83+ ndev = ib_device_get_netdev(ibdev, port);
84+
85+ if (netif_running(ndev) && netif_carrier_ok(ndev)) {
86+ attr->state = IB_PORT_ACTIVE;
87+ attr->phys_state = IB_PORT_PHYS_STATE_LINK_UP;
88+ } else if (netif_running(ndev)) {
89+ attr->state = IB_PORT_DOWN;
90+ attr->phys_state = IB_PORT_PHYS_STATE_POLLING;
91+ } else {
92+ attr->state = IB_PORT_DOWN;
93+ attr->phys_state = IB_PORT_PHYS_STATE_DISABLED;
94+ }
95+
96+ attr->max_mtu = iboe_get_mtu(ndev->max_mtu);
97+ attr->active_mtu = min(attr->max_mtu, iboe_get_mtu(ndev->mtu));
98+ attr->gid_tbl_len = IONIC_GID_TBL_LEN;
99+ attr->ip_gids = true;
100+ attr->port_cap_flags = 0;
101+ attr->max_msg_sz = 0x80000000;
102+ attr->pkey_tbl_len = IONIC_PKEY_TBL_LEN;
103+ attr->max_vl_num = 1;
104+ attr->subnet_prefix = 0xfe80000000000000ull;
105+
106+ dev_put(ndev);
107+
108+ return ib_get_eth_speed(ibdev, port,
109+ &attr->active_speed,
110+ &attr->active_width);
111+}
112+
113+static enum rdma_link_layer ionic_get_link_layer(struct ib_device *ibdev,
114+ u32 port)
115+{
116+ return IB_LINK_LAYER_ETHERNET;
117+}
118+
119+static int ionic_query_pkey(struct ib_device *ibdev, u32 port, u16 index,
120+ u16 *pkey)
121+{
122+ if (port != 1)
123+ return -EINVAL;
124+
125+ if (index != 0)
126+ return -EINVAL;
127+
128+ *pkey = IB_DEFAULT_PKEY_FULL;
129+
130+ return 0;
131+}
132+
133+static int ionic_modify_device(struct ib_device *ibdev, int mask,
134+ struct ib_device_modify *attr)
135+{
136+ struct ionic_ibdev *dev = to_ionic_ibdev(ibdev);
137+
138+ if (mask & ~IB_DEVICE_MODIFY_NODE_DESC)
139+ return -EOPNOTSUPP;
140+
141+ if (mask & IB_DEVICE_MODIFY_NODE_DESC)
142+ memcpy(dev->ibdev.node_desc, attr->node_desc,
143+ IB_DEVICE_NODE_DESC_MAX);
144+
145+ return 0;
146+}
147+
148+static int ionic_get_port_immutable(struct ib_device *ibdev, u32 port,
149+ struct ib_port_immutable *attr)
150+{
151+ if (port != 1)
152+ return -EINVAL;
153+
154+ attr->core_cap_flags = RDMA_CORE_PORT_IBA_ROCE_UDP_ENCAP;
155+
156+ attr->pkey_tbl_len = IONIC_PKEY_TBL_LEN;
157+ attr->gid_tbl_len = IONIC_GID_TBL_LEN;
158+ attr->max_mad_size = IB_MGMT_MAD_SIZE;
159+
160+ return 0;
161+}
162+
163+static void ionic_get_dev_fw_str(struct ib_device *ibdev, char *str)
164+{
165+ struct ionic_ibdev *dev = to_ionic_ibdev(ibdev);
166+
167+ ionic_lif_fw_version(dev->lif_cfg.lif, str, IB_FW_VERSION_NAME_MAX);
168+}
169+
170+static ssize_t hw_rev_show(struct device *device, struct device_attribute *attr,
171+ char *buf)
172+{
173+ struct ionic_ibdev *dev =
174+ rdma_device_to_drv_device(device, struct ionic_ibdev, ibdev);
175+
176+ return sysfs_emit(buf, "0x%x\n", ionic_lif_asic_rev(dev->lif_cfg.lif));
177+}
178+static DEVICE_ATTR_RO(hw_rev);
179+
180+static ssize_t hca_type_show(struct device *device,
181+ struct device_attribute *attr, char *buf)
182+{
183+ struct ionic_ibdev *dev =
184+ rdma_device_to_drv_device(device, struct ionic_ibdev, ibdev);
185+
186+ return sysfs_emit(buf, "%s\n", dev->ibdev.node_desc);
187+}
188+static DEVICE_ATTR_RO(hca_type);
189+
190+static struct attribute *ionic_rdma_attributes[] = {
191+ &dev_attr_hw_rev.attr,
192+ &dev_attr_hca_type.attr,
193+ NULL
194+};
195+
196+static const struct attribute_group ionic_rdma_attr_group = {
197+ .attrs = ionic_rdma_attributes,
198+};
199+
200+static void ionic_disassociate_ucontext(struct ib_ucontext *ibcontext)
201+{
202+ /*
203+ * Dummy define disassociate_ucontext so that it does not
204+ * wait for user context before cleaning up hw resources.
205+ */
206+}
207+
208+static const struct ib_device_ops ionic_dev_ops = {
209+ .owner = THIS_MODULE,
210+ .driver_id = RDMA_DRIVER_IONIC,
211+ .uverbs_abi_ver = IONIC_ABI_VERSION,
212+
213+ .alloc_ucontext = ionic_alloc_ucontext,
214+ .dealloc_ucontext = ionic_dealloc_ucontext,
215+ .mmap = ionic_mmap,
216+ .mmap_free = ionic_mmap_free,
217+ .alloc_pd = ionic_alloc_pd,
218+ .dealloc_pd = ionic_dealloc_pd,
219+ .create_ah = ionic_create_ah,
220+ .query_ah = ionic_query_ah,
221+ .destroy_ah = ionic_destroy_ah,
222+ .create_user_ah = ionic_create_ah,
223+ .get_dma_mr = ionic_get_dma_mr,
224+ .reg_user_mr = ionic_reg_user_mr,
225+ .reg_user_mr_dmabuf = ionic_reg_user_mr_dmabuf,
226+ .dereg_mr = ionic_dereg_mr,
227+ .alloc_mr = ionic_alloc_mr,
228+ .map_mr_sg = ionic_map_mr_sg,
229+ .alloc_mw = ionic_alloc_mw,
230+ .dealloc_mw = ionic_dealloc_mw,
231+ .create_cq = ionic_create_cq,
232+ .destroy_cq = ionic_destroy_cq,
233+ .create_qp = ionic_create_qp,
234+ .modify_qp = ionic_modify_qp,
235+ .query_qp = ionic_query_qp,
236+ .destroy_qp = ionic_destroy_qp,
237+
238+ .post_send = ionic_post_send,
239+ .post_recv = ionic_post_recv,
240+ .poll_cq = ionic_poll_cq,
241+ .req_notify_cq = ionic_req_notify_cq,
242+
243+ .query_device = ionic_query_device,
244+ .query_port = ionic_query_port,
245+ .get_link_layer = ionic_get_link_layer,
246+ .query_pkey = ionic_query_pkey,
247+ .modify_device = ionic_modify_device,
248+ .get_port_immutable = ionic_get_port_immutable,
249+ .get_dev_fw_str = ionic_get_dev_fw_str,
250+ .device_group = &ionic_rdma_attr_group,
251+ .disassociate_ucontext = ionic_disassociate_ucontext,
252+
253+ INIT_RDMA_OBJ_SIZE(ib_ucontext, ionic_ctx, ibctx),
254+ INIT_RDMA_OBJ_SIZE(ib_pd, ionic_pd, ibpd),
255+ INIT_RDMA_OBJ_SIZE(ib_ah, ionic_ah, ibah),
256+ INIT_RDMA_OBJ_SIZE(ib_cq, ionic_vcq, ibcq),
257+ INIT_RDMA_OBJ_SIZE(ib_qp, ionic_qp, ibqp),
258+ INIT_RDMA_OBJ_SIZE(ib_mw, ionic_mr, ibmw),
259+};
260+
261+static void ionic_init_resids(struct ionic_ibdev *dev)
262+{
263+ ionic_resid_init(&dev->inuse_cqid, dev->lif_cfg.cq_count);
264+ dev->half_cqid_udma_shift =
265+ order_base_2(dev->lif_cfg.cq_count / dev->lif_cfg.udma_count);
266+ ionic_resid_init(&dev->inuse_pdid, IONIC_MAX_PD);
267+ ionic_resid_init(&dev->inuse_ahid, dev->lif_cfg.nahs_per_lif);
268+ ionic_resid_init(&dev->inuse_mrid, dev->lif_cfg.nmrs_per_lif);
269+ /* skip reserved lkey */
270+ dev->next_mrkey = 1;
271+ ionic_resid_init(&dev->inuse_qpid, dev->lif_cfg.qp_count);
272+ /* skip reserved SMI and GSI qpids */
273+ dev->half_qpid_udma_shift =
274+ order_base_2(dev->lif_cfg.qp_count / dev->lif_cfg.udma_count);
275+ ionic_resid_init(&dev->inuse_dbid, dev->lif_cfg.dbid_count);
276+}
277+
278+static void ionic_destroy_resids(struct ionic_ibdev *dev)
279+{
280+ ionic_resid_destroy(&dev->inuse_cqid);
281+ ionic_resid_destroy(&dev->inuse_pdid);
282+ ionic_resid_destroy(&dev->inuse_ahid);
283+ ionic_resid_destroy(&dev->inuse_mrid);
284+ ionic_resid_destroy(&dev->inuse_qpid);
285+ ionic_resid_destroy(&dev->inuse_dbid);
286+}
287+
288+static void ionic_destroy_ibdev(struct ionic_ibdev *dev)
289+{
290+ ionic_kill_rdma_admin(dev, false);
291+ ib_unregister_device(&dev->ibdev);
292+ ionic_stats_cleanup(dev);
293+ ionic_destroy_rdma_admin(dev);
294+ ionic_destroy_resids(dev);
295+ WARN_ON(!xa_empty(&dev->qp_tbl));
296+ xa_destroy(&dev->qp_tbl);
297+ WARN_ON(!xa_empty(&dev->cq_tbl));
298+ xa_destroy(&dev->cq_tbl);
299+ ib_dealloc_device(&dev->ibdev);
300+}
301+
302+static struct ionic_ibdev *ionic_create_ibdev(struct ionic_aux_dev *ionic_adev)
303+{
304+ struct ib_device *ibdev;
305+ struct ionic_ibdev *dev;
306+ struct net_device *ndev;
307+ int rc;
308+
309+ dev = ib_alloc_device(ionic_ibdev, ibdev);
310+ if (!dev)
311+ return ERR_PTR(-EINVAL);
312+
313+ ionic_fill_lif_cfg(ionic_adev->lif, &dev->lif_cfg);
314+
315+ xa_init_flags(&dev->qp_tbl, GFP_ATOMIC);
316+ xa_init_flags(&dev->cq_tbl, GFP_ATOMIC);
317+
318+ ionic_init_resids(dev);
319+
320+ rc = ionic_rdma_reset_devcmd(dev);
321+ if (rc)
322+ goto err_reset;
323+
324+ rc = ionic_create_rdma_admin(dev);
325+ if (rc)
326+ goto err_admin;
327+
328+ ibdev = &dev->ibdev;
329+ ibdev->dev.parent = dev->lif_cfg.hwdev;
330+
331+ strscpy(ibdev->name, "ionic_%d", IB_DEVICE_NAME_MAX);
332+ strscpy(ibdev->node_desc, DEVICE_DESCRIPTION, IB_DEVICE_NODE_DESC_MAX);
333+
334+ ibdev->node_type = RDMA_NODE_IB_CA;
335+ ibdev->phys_port_cnt = 1;
336+
337+ /* the first two eq are reserved for async events */
338+ ibdev->num_comp_vectors = dev->lif_cfg.eq_count - 2;
339+
340+ ndev = ionic_lif_netdev(ionic_adev->lif);
341+ addrconf_ifid_eui48((u8 *)&ibdev->node_guid, ndev);
342+ rc = ib_device_set_netdev(ibdev, ndev, 1);
343+ /* ionic_lif_netdev() returns ndev with refcount held */
344+ dev_put(ndev);
345+ if (rc)
346+ goto err_admin;
347+
348+ ib_set_device_ops(&dev->ibdev, &ionic_dev_ops);
349+
350+ ionic_stats_init(dev);
351+
352+ rc = ib_register_device(ibdev, "ionic_%d", ibdev->dev.parent);
353+ if (rc)
354+ goto err_register;
355+
356+ return dev;
357+
358+err_register:
359+ ionic_stats_cleanup(dev);
360+err_admin:
361+ ionic_kill_rdma_admin(dev, false);
362+ ionic_destroy_rdma_admin(dev);
363+err_reset:
364+ ionic_destroy_resids(dev);
365+ xa_destroy(&dev->qp_tbl);
366+ xa_destroy(&dev->cq_tbl);
367+ ib_dealloc_device(&dev->ibdev);
368+
369+ return ERR_PTR(rc);
370+}
371+
372+static int ionic_aux_probe(struct auxiliary_device *adev,
373+ const struct auxiliary_device_id *id)
374+{
375+ struct ionic_aux_dev *ionic_adev;
376+ struct ionic_ibdev *dev;
377+
378+ ionic_adev = container_of(adev, struct ionic_aux_dev, adev);
379+ dev = ionic_create_ibdev(ionic_adev);
380+ if (IS_ERR(dev))
381+ return dev_err_probe(&adev->dev, PTR_ERR(dev),
382+ "Failed to register ibdev\n");
383+
384+ auxiliary_set_drvdata(adev, dev);
385+ ibdev_dbg(&dev->ibdev, "registered\n");
386+
387+ return 0;
388+}
389+
390+static void ionic_aux_remove(struct auxiliary_device *adev)
391+{
392+ struct ionic_ibdev *dev = auxiliary_get_drvdata(adev);
393+
394+ dev_dbg(&adev->dev, "unregister ibdev\n");
395+ ionic_destroy_ibdev(dev);
396+ dev_dbg(&adev->dev, "unregistered\n");
397+}
398+
399+static const struct auxiliary_device_id ionic_aux_id_table[] = {
400+ { .name = "ionic.rdma", },
401+ {},
402+};
403+
404+MODULE_DEVICE_TABLE(auxiliary, ionic_aux_id_table);
405+
406+static struct auxiliary_driver ionic_aux_r_driver = {
407+ .name = "rdma",
408+ .probe = ionic_aux_probe,
409+ .remove = ionic_aux_remove,
410+ .id_table = ionic_aux_id_table,
411+};
412+
413+static int __init ionic_mod_init(void)
414+{
415+ int rc;
416+
417+ ionic_evt_workq = create_workqueue(KBUILD_MODNAME "-evt");
418+ if (!ionic_evt_workq)
419+ return -ENOMEM;
420+
421+ rc = auxiliary_driver_register(&ionic_aux_r_driver);
422+ if (rc)
423+ goto err_aux;
424+
425+ return 0;
426+
427+err_aux:
428+ destroy_workqueue(ionic_evt_workq);
429+
430+ return rc;
431+}
432+
433+static void __exit ionic_mod_exit(void)
434+{
435+ auxiliary_driver_unregister(&ionic_aux_r_driver);
436+ destroy_workqueue(ionic_evt_workq);
437+}
438+
439+module_init(ionic_mod_init);
440+module_exit(ionic_mod_exit);
drivers/infiniband/hw/ionic/ionic_ibdev.h+517−0
@@ -0,0 +1,517 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#ifndef _IONIC_IBDEV_H_
5+#define _IONIC_IBDEV_H_
6+
7+#include <rdma/ib_umem.h>
8+#include <rdma/ib_verbs.h>
9+#include <rdma/ib_pack.h>
10+#include <rdma/uverbs_ioctl.h>
11+
12+#include <rdma/ionic-abi.h>
13+#include <ionic_api.h>
14+#include <ionic_regs.h>
15+
16+#include "ionic_fw.h"
17+#include "ionic_queue.h"
18+#include "ionic_res.h"
19+
20+#include "ionic_lif_cfg.h"
21+
22+/* Config knobs */
23+#define IONIC_EQ_DEPTH 511
24+#define IONIC_EQ_COUNT 32
25+#define IONIC_AQ_DEPTH 63
26+#define IONIC_AQ_COUNT 4
27+#define IONIC_EQ_ISR_BUDGET 10
28+#define IONIC_EQ_WORK_BUDGET 1000
29+#define IONIC_MAX_RD_ATOM 16
30+#define IONIC_PKEY_TBL_LEN 1
31+#define IONIC_GID_TBL_LEN 256
32+
33+#define IONIC_MAX_QPID 0xffffff
34+#define IONIC_SPEC_HIGH 8
35+#define IONIC_MAX_PD 1024
36+#define IONIC_SPEC_HIGH 8
37+#define IONIC_SQCMB_ORDER 5
38+#define IONIC_RQCMB_ORDER 0
39+
40+#define IONIC_META_LAST ((void *)1ul)
41+#define IONIC_META_POSTED ((void *)2ul)
42+
43+#define IONIC_CQ_GRACE 100
44+
45+#define IONIC_ROCE_UDP_SPORT 28272
46+#define IONIC_DMA_LKEY 0
47+#define IONIC_DMA_RKEY IONIC_DMA_LKEY
48+
49+#define IONIC_CMB_SUPPORTED \
50+ (IONIC_CMB_ENABLE | IONIC_CMB_REQUIRE | IONIC_CMB_EXPDB | \
51+ IONIC_CMB_WC | IONIC_CMB_UC)
52+
53+/* resource is not reserved on the device, indicated in tbl_order */
54+#define IONIC_RES_INVALID -1
55+
56+struct ionic_aq;
57+struct ionic_cq;
58+struct ionic_eq;
59+struct ionic_vcq;
60+
61+enum ionic_admin_state {
62+ IONIC_ADMIN_ACTIVE, /* submitting admin commands to queue */
63+ IONIC_ADMIN_PAUSED, /* not submitting, but may complete normally */
64+ IONIC_ADMIN_KILLED, /* not submitting, locally completed */
65+};
66+
67+enum ionic_admin_flags {
68+ IONIC_ADMIN_F_BUSYWAIT = BIT(0), /* Don't sleep */
69+ IONIC_ADMIN_F_TEARDOWN = BIT(1), /* In destroy path */
70+ IONIC_ADMIN_F_INTERRUPT = BIT(2), /* Interruptible w/timeout */
71+};
72+
73+enum ionic_mmap_flag {
74+ IONIC_MMAP_WC = BIT(0),
75+};
76+
77+struct ionic_mmap_entry {
78+ struct rdma_user_mmap_entry rdma_entry;
79+ unsigned long size;
80+ unsigned long pfn;
81+ u8 mmap_flags;
82+};
83+
84+struct ionic_ibdev {
85+ struct ib_device ibdev;
86+
87+ struct ionic_lif_cfg lif_cfg;
88+
89+ struct xarray qp_tbl;
90+ struct xarray cq_tbl;
91+
92+ struct ionic_resid_bits inuse_dbid;
93+ struct ionic_resid_bits inuse_pdid;
94+ struct ionic_resid_bits inuse_ahid;
95+ struct ionic_resid_bits inuse_mrid;
96+ struct ionic_resid_bits inuse_qpid;
97+ struct ionic_resid_bits inuse_cqid;
98+
99+ u8 half_cqid_udma_shift;
100+ u8 half_qpid_udma_shift;
101+ u8 next_qpid_udma_idx;
102+ u8 next_mrkey;
103+
104+ struct work_struct reset_work;
105+ bool reset_posted;
106+ u32 reset_cnt;
107+
108+ struct delayed_work admin_dwork;
109+ struct ionic_aq **aq_vec;
110+ atomic_t admin_state;
111+
112+ struct ionic_eq **eq_vec;
113+
114+ struct ionic_v1_stat *hw_stats;
115+ void *hw_stats_buf;
116+ struct rdma_stat_desc *hw_stats_hdrs;
117+ struct ionic_counter_stats *counter_stats;
118+ int hw_stats_count;
119+};
120+
121+struct ionic_eq {
122+ struct ionic_ibdev *dev;
123+
124+ u32 eqid;
125+ u32 intr;
126+
127+ struct ionic_queue q;
128+
129+ int armed;
130+ bool enable;
131+
132+ struct work_struct work;
133+
134+ int irq;
135+ char name[32];
136+};
137+
138+struct ionic_admin_wr {
139+ struct completion work;
140+ struct list_head aq_ent;
141+ struct ionic_v1_admin_wqe wqe;
142+ struct ionic_v1_cqe cqe;
143+ struct ionic_aq *aq;
144+ int status;
145+};
146+
147+struct ionic_admin_wr_q {
148+ struct ionic_admin_wr *wr;
149+ int wqe_strides;
150+};
151+
152+struct ionic_aq {
153+ struct ionic_ibdev *dev;
154+ struct ionic_vcq *vcq;
155+
156+ struct work_struct work;
157+
158+ atomic_t admin_state;
159+ unsigned long stamp;
160+ bool armed;
161+
162+ u32 aqid;
163+ u32 cqid;
164+
165+ spinlock_t lock; /* for posting */
166+ struct ionic_queue q;
167+ struct ionic_admin_wr_q *q_wr;
168+ struct list_head wr_prod;
169+ struct list_head wr_post;
170+};
171+
172+struct ionic_ctx {
173+ struct ib_ucontext ibctx;
174+ u32 dbid;
175+ struct rdma_user_mmap_entry *mmap_dbell;
176+};
177+
178+struct ionic_tbl_buf {
179+ u32 tbl_limit;
180+ u32 tbl_pages;
181+ size_t tbl_size;
182+ __le64 *tbl_buf;
183+ dma_addr_t tbl_dma;
184+ u8 page_size_log2;
185+};
186+
187+struct ionic_pd {
188+ struct ib_pd ibpd;
189+
190+ u32 pdid;
191+ u32 flags;
192+};
193+
194+struct ionic_cq {
195+ struct ionic_vcq *vcq;
196+
197+ u32 cqid;
198+ u32 eqid;
199+
200+ spinlock_t lock; /* for polling */
201+ struct list_head poll_sq;
202+ bool flush;
203+ struct list_head flush_sq;
204+ struct list_head flush_rq;
205+ struct list_head ibkill_flush_ent;
206+
207+ struct ionic_queue q;
208+ bool color;
209+ int credit;
210+ u16 arm_any_prod;
211+ u16 arm_sol_prod;
212+
213+ struct kref cq_kref;
214+ struct completion cq_rel_comp;
215+
216+ /* infrequently accessed, keep at end */
217+ struct ib_umem *umem;
218+};
219+
220+struct ionic_vcq {
221+ struct ib_cq ibcq;
222+ struct ionic_cq cq[2];
223+ u8 udma_mask;
224+ u8 poll_idx;
225+};
226+
227+struct ionic_sq_meta {
228+ u64 wrid;
229+ u32 len;
230+ u16 seq;
231+ u8 ibop;
232+ u8 ibsts;
233+ u8 remote:1;
234+ u8 signal:1;
235+ u8 local_comp:1;
236+};
237+
238+struct ionic_rq_meta {
239+ struct ionic_rq_meta *next;
240+ u64 wrid;
241+};
242+
243+struct ionic_qp {
244+ struct ib_qp ibqp;
245+ enum ib_qp_state state;
246+
247+ u32 qpid;
248+ u32 ahid;
249+ u32 sq_cqid;
250+ u32 rq_cqid;
251+ u8 udma_idx;
252+ u8 has_ah:1;
253+ u8 has_sq:1;
254+ u8 has_rq:1;
255+ u8 sig_all:1;
256+
257+ struct list_head qp_list_counter;
258+
259+ struct list_head cq_poll_sq;
260+ struct list_head cq_flush_sq;
261+ struct list_head cq_flush_rq;
262+ struct list_head ibkill_flush_ent;
263+
264+ spinlock_t sq_lock; /* for posting and polling */
265+ struct ionic_queue sq;
266+ struct ionic_sq_meta *sq_meta;
267+ u16 *sq_msn_idx;
268+ int sq_spec;
269+ u16 sq_old_prod;
270+ u16 sq_msn_prod;
271+ u16 sq_msn_cons;
272+ u8 sq_cmb;
273+ bool sq_flush;
274+ bool sq_flush_rcvd;
275+
276+ spinlock_t rq_lock; /* for posting and polling */
277+ struct ionic_queue rq;
278+ struct ionic_rq_meta *rq_meta;
279+ struct ionic_rq_meta *rq_meta_head;
280+ int rq_spec;
281+ u16 rq_old_prod;
282+ u8 rq_cmb;
283+ bool rq_flush;
284+
285+ struct kref qp_kref;
286+ struct completion qp_rel_comp;
287+
288+ /* infrequently accessed, keep at end */
289+ int sgid_index;
290+ int sq_cmb_order;
291+ u32 sq_cmb_pgid;
292+ phys_addr_t sq_cmb_addr;
293+ struct rdma_user_mmap_entry *mmap_sq_cmb;
294+
295+ struct ib_umem *sq_umem;
296+
297+ int rq_cmb_order;
298+ u32 rq_cmb_pgid;
299+ phys_addr_t rq_cmb_addr;
300+ struct rdma_user_mmap_entry *mmap_rq_cmb;
301+
302+ struct ib_umem *rq_umem;
303+
304+ int dcqcn_profile;
305+
306+ struct ib_ud_header *hdr;
307+};
308+
309+struct ionic_ah {
310+ struct ib_ah ibah;
311+ u32 ahid;
312+ int sgid_index;
313+ struct ib_ud_header hdr;
314+};
315+
316+struct ionic_mr {
317+ union {
318+ struct ib_mr ibmr;
319+ struct ib_mw ibmw;
320+ };
321+
322+ u32 mrid;
323+ int flags;
324+
325+ struct ib_umem *umem;
326+ struct ionic_tbl_buf buf;
327+ bool created;
328+};
329+
330+struct ionic_counter_stats {
331+ int queue_stats_count;
332+ struct ionic_v1_stat *hdr;
333+ struct rdma_stat_desc *stats_hdrs;
334+ struct xarray xa_counters;
335+};
336+
337+struct ionic_counter {
338+ void *vals;
339+ struct list_head qp_list;
340+};
341+
342+static inline struct ionic_ibdev *to_ionic_ibdev(struct ib_device *ibdev)
343+{
344+ return container_of(ibdev, struct ionic_ibdev, ibdev);
345+}
346+
347+static inline struct ionic_ctx *to_ionic_ctx(struct ib_ucontext *ibctx)
348+{
349+ return container_of(ibctx, struct ionic_ctx, ibctx);
350+}
351+
352+static inline struct ionic_ctx *to_ionic_ctx_uobj(struct ib_uobject *uobj)
353+{
354+ if (!uobj)
355+ return NULL;
356+
357+ if (!uobj->context)
358+ return NULL;
359+
360+ return to_ionic_ctx(uobj->context);
361+}
362+
363+static inline struct ionic_pd *to_ionic_pd(struct ib_pd *ibpd)
364+{
365+ return container_of(ibpd, struct ionic_pd, ibpd);
366+}
367+
368+static inline struct ionic_mr *to_ionic_mr(struct ib_mr *ibmr)
369+{
370+ return container_of(ibmr, struct ionic_mr, ibmr);
371+}
372+
373+static inline struct ionic_mr *to_ionic_mw(struct ib_mw *ibmw)
374+{
375+ return container_of(ibmw, struct ionic_mr, ibmw);
376+}
377+
378+static inline struct ionic_vcq *to_ionic_vcq(struct ib_cq *ibcq)
379+{
380+ return container_of(ibcq, struct ionic_vcq, ibcq);
381+}
382+
383+static inline struct ionic_cq *to_ionic_vcq_cq(struct ib_cq *ibcq,
384+ uint8_t udma_idx)
385+{
386+ return &to_ionic_vcq(ibcq)->cq[udma_idx];
387+}
388+
389+static inline struct ionic_qp *to_ionic_qp(struct ib_qp *ibqp)
390+{
391+ return container_of(ibqp, struct ionic_qp, ibqp);
392+}
393+
394+static inline struct ionic_ah *to_ionic_ah(struct ib_ah *ibah)
395+{
396+ return container_of(ibah, struct ionic_ah, ibah);
397+}
398+
399+static inline u32 ionic_ctx_dbid(struct ionic_ibdev *dev,
400+ struct ionic_ctx *ctx)
401+{
402+ if (!ctx)
403+ return dev->lif_cfg.dbid;
404+
405+ return ctx->dbid;
406+}
407+
408+static inline u32 ionic_obj_dbid(struct ionic_ibdev *dev,
409+ struct ib_uobject *uobj)
410+{
411+ return ionic_ctx_dbid(dev, to_ionic_ctx_uobj(uobj));
412+}
413+
414+static inline bool ionic_ibop_is_local(enum ib_wr_opcode op)
415+{
416+ return op == IB_WR_LOCAL_INV || op == IB_WR_REG_MR;
417+}
418+
419+static inline void ionic_qp_complete(struct kref *kref)
420+{
421+ struct ionic_qp *qp = container_of(kref, struct ionic_qp, qp_kref);
422+
423+ complete(&qp->qp_rel_comp);
424+}
425+
426+static inline void ionic_cq_complete(struct kref *kref)
427+{
428+ struct ionic_cq *cq = container_of(kref, struct ionic_cq, cq_kref);
429+
430+ complete(&cq->cq_rel_comp);
431+}
432+
433+/* ionic_admin.c */
434+extern struct workqueue_struct *ionic_evt_workq;
435+void ionic_admin_post(struct ionic_ibdev *dev, struct ionic_admin_wr *wr);
436+int ionic_admin_wait(struct ionic_ibdev *dev, struct ionic_admin_wr *wr,
437+ enum ionic_admin_flags);
438+
439+int ionic_rdma_reset_devcmd(struct ionic_ibdev *dev);
440+
441+int ionic_create_rdma_admin(struct ionic_ibdev *dev);
442+void ionic_destroy_rdma_admin(struct ionic_ibdev *dev);
443+void ionic_kill_rdma_admin(struct ionic_ibdev *dev, bool fatal_path);
444+
445+/* ionic_controlpath.c */
446+int ionic_create_cq_common(struct ionic_vcq *vcq,
447+ struct ionic_tbl_buf *buf,
448+ const struct ib_cq_init_attr *attr,
449+ struct ionic_ctx *ctx,
450+ struct ib_udata *udata,
451+ struct ionic_qdesc *req_cq,
452+ __u32 *resp_cqid,
453+ int udma_idx);
454+void ionic_destroy_cq_common(struct ionic_ibdev *dev, struct ionic_cq *cq);
455+void ionic_flush_qp(struct ionic_ibdev *dev, struct ionic_qp *qp);
456+void ionic_notify_flush_cq(struct ionic_cq *cq);
457+
458+int ionic_alloc_ucontext(struct ib_ucontext *ibctx, struct ib_udata *udata);
459+void ionic_dealloc_ucontext(struct ib_ucontext *ibctx);
460+int ionic_mmap(struct ib_ucontext *ibctx, struct vm_area_struct *vma);
461+void ionic_mmap_free(struct rdma_user_mmap_entry *rdma_entry);
462+int ionic_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata);
463+int ionic_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata);
464+int ionic_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *init_attr,
465+ struct ib_udata *udata);
466+int ionic_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr);
467+int ionic_destroy_ah(struct ib_ah *ibah, u32 flags);
468+struct ib_mr *ionic_get_dma_mr(struct ib_pd *ibpd, int access);
469+struct ib_mr *ionic_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length,
470+ u64 addr, int access, struct ib_dmah *dmah,
471+ struct ib_udata *udata);
472+struct ib_mr *ionic_reg_user_mr_dmabuf(struct ib_pd *ibpd, u64 offset,
473+ u64 length, u64 addr, int fd, int access,
474+ struct ib_dmah *dmah,
475+ struct uverbs_attr_bundle *attrs);
476+int ionic_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata);
477+struct ib_mr *ionic_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type type,
478+ u32 max_sg);
479+int ionic_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
480+ unsigned int *sg_offset);
481+int ionic_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata);
482+int ionic_dealloc_mw(struct ib_mw *ibmw);
483+int ionic_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
484+ struct uverbs_attr_bundle *attrs);
485+int ionic_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata);
486+int ionic_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *attr,
487+ struct ib_udata *udata);
488+int ionic_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int mask,
489+ struct ib_udata *udata);
490+int ionic_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int mask,
491+ struct ib_qp_init_attr *init_attr);
492+int ionic_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata);
493+
494+/* ionic_datapath.c */
495+int ionic_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,
496+ const struct ib_send_wr **bad);
497+int ionic_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,
498+ const struct ib_recv_wr **bad);
499+int ionic_poll_cq(struct ib_cq *ibcq, int nwc, struct ib_wc *wc);
500+int ionic_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags);
501+
502+/* ionic_hw_stats.c */
503+void ionic_stats_init(struct ionic_ibdev *dev);
504+void ionic_stats_cleanup(struct ionic_ibdev *dev);
505+
506+/* ionic_pgtbl.c */
507+__le64 ionic_pgtbl_dma(struct ionic_tbl_buf *buf, u64 va);
508+__be64 ionic_pgtbl_off(struct ionic_tbl_buf *buf, u64 va);
509+int ionic_pgtbl_page(struct ionic_tbl_buf *buf, u64 dma);
510+int ionic_pgtbl_init(struct ionic_ibdev *dev,
511+ struct ionic_tbl_buf *buf,
512+ struct ib_umem *umem,
513+ dma_addr_t dma,
514+ int limit,
515+ u64 page_size);
516+void ionic_pgtbl_unbuf(struct ionic_ibdev *dev, struct ionic_tbl_buf *buf);
517+#endif /* _IONIC_IBDEV_H_ */
drivers/infiniband/hw/ionic/ionic_lif_cfg.c+111−0
@@ -0,0 +1,111 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#include <linux/kernel.h>
5+
6+#include <ionic.h>
7+#include <ionic_lif.h>
8+
9+#include "ionic_lif_cfg.h"
10+
11+#define IONIC_MIN_RDMA_VERSION 0
12+#define IONIC_MAX_RDMA_VERSION 2
13+
14+static u8 ionic_get_expdb(struct ionic_lif *lif)
15+{
16+ u8 expdb_support = 0;
17+
18+ if (lif->ionic->idev.phy_cmb_expdb64_pages)
19+ expdb_support |= IONIC_EXPDB_64B_WQE;
20+ if (lif->ionic->idev.phy_cmb_expdb128_pages)
21+ expdb_support |= IONIC_EXPDB_128B_WQE;
22+ if (lif->ionic->idev.phy_cmb_expdb256_pages)
23+ expdb_support |= IONIC_EXPDB_256B_WQE;
24+ if (lif->ionic->idev.phy_cmb_expdb512_pages)
25+ expdb_support |= IONIC_EXPDB_512B_WQE;
26+
27+ return expdb_support;
28+}
29+
30+void ionic_fill_lif_cfg(struct ionic_lif *lif, struct ionic_lif_cfg *cfg)
31+{
32+ union ionic_lif_identity *ident = &lif->ionic->ident.lif;
33+
34+ cfg->lif = lif;
35+ cfg->hwdev = &lif->ionic->pdev->dev;
36+ cfg->lif_index = lif->index;
37+ cfg->lif_hw_index = lif->hw_index;
38+
39+ cfg->dbid = lif->kern_pid;
40+ cfg->dbid_count = le32_to_cpu(lif->ionic->ident.dev.ndbpgs_per_lif);
41+ cfg->dbpage = lif->kern_dbpage;
42+ cfg->intr_ctrl = lif->ionic->idev.intr_ctrl;
43+
44+ cfg->db_phys = lif->ionic->bars[IONIC_PCI_BAR_DBELL].bus_addr;
45+
46+ if (IONIC_VERSION(ident->rdma.version, ident->rdma.minor_version) >=
47+ IONIC_VERSION(2, 1))
48+ cfg->page_size_supported =
49+ le64_to_cpu(ident->rdma.page_size_cap);
50+ else
51+ cfg->page_size_supported = IONIC_PAGE_SIZE_SUPPORTED;
52+
53+ cfg->rdma_version = ident->rdma.version;
54+ cfg->qp_opcodes = ident->rdma.qp_opcodes;
55+ cfg->admin_opcodes = ident->rdma.admin_opcodes;
56+
57+ cfg->stats_type = le16_to_cpu(ident->rdma.stats_type);
58+ cfg->npts_per_lif = le32_to_cpu(ident->rdma.npts_per_lif);
59+ cfg->nmrs_per_lif = le32_to_cpu(ident->rdma.nmrs_per_lif);
60+ cfg->nahs_per_lif = le32_to_cpu(ident->rdma.nahs_per_lif);
61+
62+ cfg->aq_base = le32_to_cpu(ident->rdma.aq_qtype.qid_base);
63+ cfg->cq_base = le32_to_cpu(ident->rdma.cq_qtype.qid_base);
64+ cfg->eq_base = le32_to_cpu(ident->rdma.eq_qtype.qid_base);
65+
66+ /*
67+ * ionic_create_rdma_admin() may reduce aq_count or eq_count if
68+ * it is unable to allocate all that were requested.
69+ * aq_count is tunable; see ionic_aq_count
70+ * eq_count is tunable; see ionic_eq_count
71+ */
72+ cfg->aq_count = le32_to_cpu(ident->rdma.aq_qtype.qid_count);
73+ cfg->eq_count = le32_to_cpu(ident->rdma.eq_qtype.qid_count);
74+ cfg->cq_count = le32_to_cpu(ident->rdma.cq_qtype.qid_count);
75+ cfg->qp_count = le32_to_cpu(ident->rdma.sq_qtype.qid_count);
76+ cfg->dbid_count = le32_to_cpu(lif->ionic->ident.dev.ndbpgs_per_lif);
77+
78+ cfg->aq_qtype = ident->rdma.aq_qtype.qtype;
79+ cfg->sq_qtype = ident->rdma.sq_qtype.qtype;
80+ cfg->rq_qtype = ident->rdma.rq_qtype.qtype;
81+ cfg->cq_qtype = ident->rdma.cq_qtype.qtype;
82+ cfg->eq_qtype = ident->rdma.eq_qtype.qtype;
83+ cfg->udma_qgrp_shift = ident->rdma.udma_shift;
84+ cfg->udma_count = 2;
85+
86+ cfg->max_stride = ident->rdma.max_stride;
87+ cfg->expdb_mask = ionic_get_expdb(lif);
88+
89+ cfg->sq_expdb =
90+ !!(lif->qtype_info[IONIC_QTYPE_TXQ].features & IONIC_QIDENT_F_EXPDB);
91+ cfg->rq_expdb =
92+ !!(lif->qtype_info[IONIC_QTYPE_RXQ].features & IONIC_QIDENT_F_EXPDB);
93+}
94+
95+struct net_device *ionic_lif_netdev(struct ionic_lif *lif)
96+{
97+ struct net_device *netdev = lif->netdev;
98+
99+ dev_hold(netdev);
100+ return netdev;
101+}
102+
103+void ionic_lif_fw_version(struct ionic_lif *lif, char *str, size_t len)
104+{
105+ strscpy(str, lif->ionic->idev.dev_info.fw_version, len);
106+}
107+
108+u8 ionic_lif_asic_rev(struct ionic_lif *lif)
109+{
110+ return lif->ionic->idev.dev_info.asic_rev;
111+}
drivers/infiniband/hw/ionic/ionic_lif_cfg.h+66−0
@@ -0,0 +1,66 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#ifndef _IONIC_LIF_CFG_H_
5+
6+#define IONIC_VERSION(a, b) (((a) << 16) + ((b) << 8))
7+#define IONIC_PAGE_SIZE_SUPPORTED 0x40201000 /* 4kb, 2Mb, 1Gb */
8+
9+#define IONIC_EXPDB_64B_WQE BIT(0)
10+#define IONIC_EXPDB_128B_WQE BIT(1)
11+#define IONIC_EXPDB_256B_WQE BIT(2)
12+#define IONIC_EXPDB_512B_WQE BIT(3)
13+
14+struct ionic_lif_cfg {
15+ struct device *hwdev;
16+ struct ionic_lif *lif;
17+
18+ int lif_index;
19+ int lif_hw_index;
20+
21+ u32 dbid;
22+ int dbid_count;
23+ u64 __iomem *dbpage;
24+ struct ionic_intr __iomem *intr_ctrl;
25+ phys_addr_t db_phys;
26+
27+ u64 page_size_supported;
28+ u32 npts_per_lif;
29+ u32 nmrs_per_lif;
30+ u32 nahs_per_lif;
31+
32+ u32 aq_base;
33+ u32 cq_base;
34+ u32 eq_base;
35+
36+ int aq_count;
37+ int eq_count;
38+ int cq_count;
39+ int qp_count;
40+
41+ u16 stats_type;
42+ u8 aq_qtype;
43+ u8 sq_qtype;
44+ u8 rq_qtype;
45+ u8 cq_qtype;
46+ u8 eq_qtype;
47+
48+ u8 udma_count;
49+ u8 udma_qgrp_shift;
50+
51+ u8 rdma_version;
52+ u8 qp_opcodes;
53+ u8 admin_opcodes;
54+
55+ u8 max_stride;
56+ bool sq_expdb;
57+ bool rq_expdb;
58+ u8 expdb_mask;
59+};
60+
61+void ionic_fill_lif_cfg(struct ionic_lif *lif, struct ionic_lif_cfg *cfg);
62+struct net_device *ionic_lif_netdev(struct ionic_lif *lif);
63+void ionic_lif_fw_version(struct ionic_lif *lif, char *str, size_t len);
64+u8 ionic_lif_asic_rev(struct ionic_lif *lif);
65+
66+#endif /* _IONIC_LIF_CFG_H_ */
drivers/infiniband/hw/ionic/ionic_pgtbl.c+143−0
@@ -0,0 +1,143 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#include <linux/mman.h>
5+#include <linux/dma-mapping.h>
6+
7+#include "ionic_fw.h"
8+#include "ionic_ibdev.h"
9+
10+__le64 ionic_pgtbl_dma(struct ionic_tbl_buf *buf, u64 va)
11+{
12+ u64 pg_mask = BIT_ULL(buf->page_size_log2) - 1;
13+ u64 dma;
14+
15+ if (!buf->tbl_pages)
16+ return cpu_to_le64(0);
17+
18+ if (buf->tbl_pages > 1)
19+ return cpu_to_le64(buf->tbl_dma);
20+
21+ if (buf->tbl_buf)
22+ dma = le64_to_cpu(buf->tbl_buf[0]);
23+ else
24+ dma = buf->tbl_dma;
25+
26+ return cpu_to_le64(dma + (va & pg_mask));
27+}
28+
29+__be64 ionic_pgtbl_off(struct ionic_tbl_buf *buf, u64 va)
30+{
31+ if (buf->tbl_pages > 1) {
32+ u64 pg_mask = BIT_ULL(buf->page_size_log2) - 1;
33+
34+ return cpu_to_be64(va & pg_mask);
35+ }
36+
37+ return 0;
38+}
39+
40+int ionic_pgtbl_page(struct ionic_tbl_buf *buf, u64 dma)
41+{
42+ if (unlikely(buf->tbl_pages == buf->tbl_limit))
43+ return -ENOMEM;
44+
45+ if (buf->tbl_buf)
46+ buf->tbl_buf[buf->tbl_pages] = cpu_to_le64(dma);
47+ else
48+ buf->tbl_dma = dma;
49+
50+ ++buf->tbl_pages;
51+
52+ return 0;
53+}
54+
55+static int ionic_tbl_buf_alloc(struct ionic_ibdev *dev,
56+ struct ionic_tbl_buf *buf)
57+{
58+ int rc;
59+
60+ buf->tbl_size = buf->tbl_limit * sizeof(*buf->tbl_buf);
61+ buf->tbl_buf = kmalloc(buf->tbl_size, GFP_KERNEL);
62+ if (!buf->tbl_buf)
63+ return -ENOMEM;
64+
65+ buf->tbl_dma = dma_map_single(dev->lif_cfg.hwdev, buf->tbl_buf,
66+ buf->tbl_size, DMA_TO_DEVICE);
67+ rc = dma_mapping_error(dev->lif_cfg.hwdev, buf->tbl_dma);
68+ if (rc) {
69+ kfree(buf->tbl_buf);
70+ return rc;
71+ }
72+
73+ return 0;
74+}
75+
76+static int ionic_pgtbl_umem(struct ionic_tbl_buf *buf, struct ib_umem *umem)
77+{
78+ struct ib_block_iter biter;
79+ u64 page_dma;
80+ int rc;
81+
82+ rdma_umem_for_each_dma_block(umem, &biter, BIT_ULL(buf->page_size_log2)) {
83+ page_dma = rdma_block_iter_dma_address(&biter);
84+ rc = ionic_pgtbl_page(buf, page_dma);
85+ if (rc)
86+ return rc;
87+ }
88+
89+ return 0;
90+}
91+
92+void ionic_pgtbl_unbuf(struct ionic_ibdev *dev, struct ionic_tbl_buf *buf)
93+{
94+ if (buf->tbl_buf)
95+ dma_unmap_single(dev->lif_cfg.hwdev, buf->tbl_dma,
96+ buf->tbl_size, DMA_TO_DEVICE);
97+
98+ kfree(buf->tbl_buf);
99+ memset(buf, 0, sizeof(*buf));
100+}
101+
102+int ionic_pgtbl_init(struct ionic_ibdev *dev,
103+ struct ionic_tbl_buf *buf,
104+ struct ib_umem *umem,
105+ dma_addr_t dma,
106+ int limit,
107+ u64 page_size)
108+{
109+ int rc;
110+
111+ memset(buf, 0, sizeof(*buf));
112+
113+ if (umem) {
114+ limit = ib_umem_num_dma_blocks(umem, page_size);
115+ buf->page_size_log2 = order_base_2(page_size);
116+ }
117+
118+ if (limit < 1)
119+ return -EINVAL;
120+
121+ buf->tbl_limit = limit;
122+
123+ /* skip pgtbl if contiguous / direct translation */
124+ if (limit > 1) {
125+ rc = ionic_tbl_buf_alloc(dev, buf);
126+ if (rc)
127+ return rc;
128+ }
129+
130+ if (umem)
131+ rc = ionic_pgtbl_umem(buf, umem);
132+ else
133+ rc = ionic_pgtbl_page(buf, dma);
134+
135+ if (rc)
136+ goto err_unbuf;
137+
138+ return 0;
139+
140+err_unbuf:
141+ ionic_pgtbl_unbuf(dev, buf);
142+ return rc;
143+}
drivers/infiniband/hw/ionic/ionic_queue.c+52−0
@@ -0,0 +1,52 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#include <linux/dma-mapping.h>
5+
6+#include "ionic_queue.h"
7+
8+int ionic_queue_init(struct ionic_queue *q, struct device *dma_dev,
9+ int depth, size_t stride)
10+{
11+ if (depth < 0 || depth > 0xffff)
12+ return -EINVAL;
13+
14+ if (stride == 0 || stride > 0x10000)
15+ return -EINVAL;
16+
17+ if (depth == 0)
18+ depth = 1;
19+
20+ q->depth_log2 = order_base_2(depth + 1);
21+ q->stride_log2 = order_base_2(stride);
22+
23+ if (q->depth_log2 + q->stride_log2 < PAGE_SHIFT)
24+ q->depth_log2 = PAGE_SHIFT - q->stride_log2;
25+
26+ if (q->depth_log2 > 16 || q->stride_log2 > 16)
27+ return -EINVAL;
28+
29+ q->size = BIT_ULL(q->depth_log2 + q->stride_log2);
30+ q->mask = BIT(q->depth_log2) - 1;
31+
32+ q->ptr = dma_alloc_coherent(dma_dev, q->size, &q->dma, GFP_KERNEL);
33+ if (!q->ptr)
34+ return -ENOMEM;
35+
36+ /* it will always be page aligned, but just to be sure... */
37+ if (!PAGE_ALIGNED(q->ptr)) {
38+ dma_free_coherent(dma_dev, q->size, q->ptr, q->dma);
39+ return -ENOMEM;
40+ }
41+
42+ q->prod = 0;
43+ q->cons = 0;
44+ q->dbell = 0;
45+
46+ return 0;
47+}
48+
49+void ionic_queue_destroy(struct ionic_queue *q, struct device *dma_dev)
50+{
51+ dma_free_coherent(dma_dev, q->size, q->ptr, q->dma);
52+}
drivers/infiniband/hw/ionic/ionic_queue.h+234−0
@@ -0,0 +1,234 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#ifndef _IONIC_QUEUE_H_
5+#define _IONIC_QUEUE_H_
6+
7+#include <linux/io.h>
8+#include <ionic_regs.h>
9+
10+#define IONIC_MAX_DEPTH 0xffff
11+#define IONIC_MAX_CQ_DEPTH 0xffff
12+#define IONIC_CQ_RING_ARM IONIC_DBELL_RING_1
13+#define IONIC_CQ_RING_SOL IONIC_DBELL_RING_2
14+
15+/**
16+ * struct ionic_queue - Ring buffer used between device and driver
17+ * @size: Size of the buffer, in bytes
18+ * @dma: Dma address of the buffer
19+ * @ptr: Buffer virtual address
20+ * @prod: Driver position in the queue
21+ * @cons: Device position in the queue
22+ * @mask: Capacity of the queue, subtracting the hole
23+ * This value is equal to ((1 << depth_log2) - 1)
24+ * @depth_log2: Log base two size depth of the queue
25+ * @stride_log2: Log base two size of an element in the queue
26+ * @dbell: Doorbell identifying bits
27+ */
28+struct ionic_queue {
29+ size_t size;
30+ dma_addr_t dma;
31+ void *ptr;
32+ u16 prod;
33+ u16 cons;
34+ u16 mask;
35+ u8 depth_log2;
36+ u8 stride_log2;
37+ u64 dbell;
38+};
39+
40+/**
41+ * ionic_queue_init() - Initialize user space queue
42+ * @q: Uninitialized queue structure
43+ * @dma_dev: DMA device for mapping
44+ * @depth: Depth of the queue
45+ * @stride: Size of each element of the queue
46+ *
47+ * Return: status code
48+ */
49+int ionic_queue_init(struct ionic_queue *q, struct device *dma_dev,
50+ int depth, size_t stride);
51+
52+/**
53+ * ionic_queue_destroy() - Destroy user space queue
54+ * @q: Queue structure
55+ * @dma_dev: DMA device for mapping
56+ *
57+ * Return: status code
58+ */
59+void ionic_queue_destroy(struct ionic_queue *q, struct device *dma_dev);
60+
61+/**
62+ * ionic_queue_empty() - Test if queue is empty
63+ * @q: Queue structure
64+ *
65+ * This is only valid for to-device queues.
66+ *
67+ * Return: is empty
68+ */
69+static inline bool ionic_queue_empty(struct ionic_queue *q)
70+{
71+ return q->prod == q->cons;
72+}
73+
74+/**
75+ * ionic_queue_length() - Get the current length of the queue
76+ * @q: Queue structure
77+ *
78+ * This is only valid for to-device queues.
79+ *
80+ * Return: length
81+ */
82+static inline u16 ionic_queue_length(struct ionic_queue *q)
83+{
84+ return (q->prod - q->cons) & q->mask;
85+}
86+
87+/**
88+ * ionic_queue_length_remaining() - Get the remaining length of the queue
89+ * @q: Queue structure
90+ *
91+ * This is only valid for to-device queues.
92+ *
93+ * Return: length remaining
94+ */
95+static inline u16 ionic_queue_length_remaining(struct ionic_queue *q)
96+{
97+ return q->mask - ionic_queue_length(q);
98+}
99+
100+/**
101+ * ionic_queue_full() - Test if queue is full
102+ * @q: Queue structure
103+ *
104+ * This is only valid for to-device queues.
105+ *
106+ * Return: is full
107+ */
108+static inline bool ionic_queue_full(struct ionic_queue *q)
109+{
110+ return q->mask == ionic_queue_length(q);
111+}
112+
113+/**
114+ * ionic_color_wrap() - Flip the color if prod is wrapped
115+ * @prod: Queue index just after advancing
116+ * @color: Queue color just prior to advancing the index
117+ *
118+ * Return: color after advancing the index
119+ */
120+static inline bool ionic_color_wrap(u16 prod, bool color)
121+{
122+ /* logical xor color with (prod == 0) */
123+ return color != (prod == 0);
124+}
125+
126+/**
127+ * ionic_queue_at() - Get the element at the given index
128+ * @q: Queue structure
129+ * @idx: Index in the queue
130+ *
131+ * The index must be within the bounds of the queue. It is not checked here.
132+ *
133+ * Return: pointer to element at index
134+ */
135+static inline void *ionic_queue_at(struct ionic_queue *q, u16 idx)
136+{
137+ return q->ptr + ((unsigned long)idx << q->stride_log2);
138+}
139+
140+/**
141+ * ionic_queue_at_prod() - Get the element at the producer index
142+ * @q: Queue structure
143+ *
144+ * Return: pointer to element at producer index
145+ */
146+static inline void *ionic_queue_at_prod(struct ionic_queue *q)
147+{
148+ return ionic_queue_at(q, q->prod);
149+}
150+
151+/**
152+ * ionic_queue_at_cons() - Get the element at the consumer index
153+ * @q: Queue structure
154+ *
155+ * Return: pointer to element at consumer index
156+ */
157+static inline void *ionic_queue_at_cons(struct ionic_queue *q)
158+{
159+ return ionic_queue_at(q, q->cons);
160+}
161+
162+/**
163+ * ionic_queue_next() - Compute the next index
164+ * @q: Queue structure
165+ * @idx: Index
166+ *
167+ * Return: next index after idx
168+ */
169+static inline u16 ionic_queue_next(struct ionic_queue *q, u16 idx)
170+{
171+ return (idx + 1) & q->mask;
172+}
173+
174+/**
175+ * ionic_queue_produce() - Increase the producer index
176+ * @q: Queue structure
177+ *
178+ * Caller must ensure that the queue is not full. It is not checked here.
179+ */
180+static inline void ionic_queue_produce(struct ionic_queue *q)
181+{
182+ q->prod = ionic_queue_next(q, q->prod);
183+}
184+
185+/**
186+ * ionic_queue_consume() - Increase the consumer index
187+ * @q: Queue structure
188+ *
189+ * Caller must ensure that the queue is not empty. It is not checked here.
190+ *
191+ * This is only valid for to-device queues.
192+ */
193+static inline void ionic_queue_consume(struct ionic_queue *q)
194+{
195+ q->cons = ionic_queue_next(q, q->cons);
196+}
197+
198+/**
199+ * ionic_queue_consume_entries() - Increase the consumer index by entries
200+ * @q: Queue structure
201+ * @entries: Number of entries to increment
202+ *
203+ * Caller must ensure that the queue is not empty. It is not checked here.
204+ *
205+ * This is only valid for to-device queues.
206+ */
207+static inline void ionic_queue_consume_entries(struct ionic_queue *q,
208+ u16 entries)
209+{
210+ q->cons = (q->cons + entries) & q->mask;
211+}
212+
213+/**
214+ * ionic_queue_dbell_init() - Initialize doorbell bits for queue id
215+ * @q: Queue structure
216+ * @qid: Queue identifying number
217+ */
218+static inline void ionic_queue_dbell_init(struct ionic_queue *q, u32 qid)
219+{
220+ q->dbell = IONIC_DBELL_QID(qid);
221+}
222+
223+/**
224+ * ionic_queue_dbell_val() - Get current doorbell update value
225+ * @q: Queue structure
226+ *
227+ * Return: current doorbell update value
228+ */
229+static inline u64 ionic_queue_dbell_val(struct ionic_queue *q)
230+{
231+ return q->dbell | q->prod;
232+}
233+
234+#endif /* _IONIC_QUEUE_H_ */
drivers/infiniband/hw/ionic/ionic_res.h+154−0
@@ -0,0 +1,154 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+/* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */
3+
4+#ifndef _IONIC_RES_H_
5+#define _IONIC_RES_H_
6+
7+#include <linux/kernel.h>
8+#include <linux/idr.h>
9+
10+/**
11+ * struct ionic_resid_bits - Number allocator based on IDA
12+ *
13+ * @inuse: IDA handle
14+ * @inuse_size: Highest ID limit for IDA
15+ */
16+struct ionic_resid_bits {
17+ struct ida inuse;
18+ unsigned int inuse_size;
19+};
20+
21+/**
22+ * ionic_resid_init() - Initialize a resid allocator
23+ * @resid: Uninitialized resid allocator
24+ * @size: Capacity of the allocator
25+ *
26+ * Return: Zero on success, or negative error number
27+ */
28+static inline void ionic_resid_init(struct ionic_resid_bits *resid,
29+ unsigned int size)
30+{
31+ resid->inuse_size = size;
32+ ida_init(&resid->inuse);
33+}
34+
35+/**
36+ * ionic_resid_destroy() - Destroy a resid allocator
37+ * @resid: Resid allocator
38+ */
39+static inline void ionic_resid_destroy(struct ionic_resid_bits *resid)
40+{
41+ ida_destroy(&resid->inuse);
42+}
43+
44+/**
45+ * ionic_resid_get_shared() - Allocate an available shared resource id
46+ * @resid: Resid allocator
47+ * @min: Smallest valid resource id
48+ * @size: One after largest valid resource id
49+ *
50+ * Return: Resource id, or negative error number
51+ */
52+static inline int ionic_resid_get_shared(struct ionic_resid_bits *resid,
53+ unsigned int min,
54+ unsigned int size)
55+{
56+ return ida_alloc_range(&resid->inuse, min, size - 1, GFP_KERNEL);
57+}
58+
59+/**
60+ * ionic_resid_get() - Allocate an available resource id
61+ * @resid: Resid allocator
62+ *
63+ * Return: Resource id, or negative error number
64+ */
65+static inline int ionic_resid_get(struct ionic_resid_bits *resid)
66+{
67+ return ionic_resid_get_shared(resid, 0, resid->inuse_size);
68+}
69+
70+/**
71+ * ionic_resid_put() - Free a resource id
72+ * @resid: Resid allocator
73+ * @id: Resource id
74+ */
75+static inline void ionic_resid_put(struct ionic_resid_bits *resid, int id)
76+{
77+ ida_free(&resid->inuse, id);
78+}
79+
80+/**
81+ * ionic_bitid_to_qid() - Transform a resource bit index into a queue id
82+ * @bitid: Bit index
83+ * @qgrp_shift: Log2 number of queues per queue group
84+ * @half_qid_shift: Log2 of half the total number of queues
85+ *
86+ * Return: Queue id
87+ *
88+ * Udma-constrained queues (QPs and CQs) are associated with their udma by
89+ * queue group. Even queue groups are associated with udma0, and odd queue
90+ * groups with udma1.
91+ *
92+ * For allocating queue ids, we want to arrange the bits into two halves,
93+ * with the even queue groups of udma0 in the lower half of the bitset,
94+ * and the odd queue groups of udma1 in the upper half of the bitset.
95+ * Then, one or two calls of find_next_zero_bit can examine all the bits
96+ * for queues of an entire udma.
97+ *
98+ * For example, assuming eight queue groups with qgrp qids per group:
99+ *
100+ * bitid 0*qgrp..1*qgrp-1 : qid 0*qgrp..1*qgrp-1
101+ * bitid 1*qgrp..2*qgrp-1 : qid 2*qgrp..3*qgrp-1
102+ * bitid 2*qgrp..3*qgrp-1 : qid 4*qgrp..5*qgrp-1
103+ * bitid 3*qgrp..4*qgrp-1 : qid 6*qgrp..7*qgrp-1
104+ * bitid 4*qgrp..5*qgrp-1 : qid 1*qgrp..2*qgrp-1
105+ * bitid 5*qgrp..6*qgrp-1 : qid 3*qgrp..4*qgrp-1
106+ * bitid 6*qgrp..7*qgrp-1 : qid 5*qgrp..6*qgrp-1
107+ * bitid 7*qgrp..8*qgrp-1 : qid 7*qgrp..8*qgrp-1
108+ *
109+ * There are three important ranges of bits in the qid. There is the udma
110+ * bit "U" at qgrp_shift, which is the least significant bit of the group
111+ * index, and determines which udma a queue is associated with.
112+ * The bits of lesser significance we can call the idx bits "I", which are
113+ * the index of the queue within the group. The bits of greater significance
114+ * we can call the grp bits "G", which are other bits of the group index that
115+ * do not determine the udma. Those bits are just rearranged in the bit index
116+ * in the bitset. A bitid has the udma bit in the most significant place,
117+ * then the grp bits, then the idx bits.
118+ *
119+ * bitid: 00000000000000 U GGG IIIIII
120+ * qid: 00000000000000 GGG U IIIIII
121+ *
122+ * Transforming from bit index to qid, or from qid to bit index, can be
123+ * accomplished by rearranging the bits by masking and shifting.
124+ */
125+static inline u32 ionic_bitid_to_qid(u32 bitid, u8 qgrp_shift,
126+ u8 half_qid_shift)
127+{
128+ u32 udma_bit =
129+ (bitid & BIT(half_qid_shift)) >> (half_qid_shift - qgrp_shift);
130+ u32 grp_bits = (bitid & GENMASK(half_qid_shift - 1, qgrp_shift)) << 1;
131+ u32 idx_bits = bitid & (BIT(qgrp_shift) - 1);
132+
133+ return grp_bits | udma_bit | idx_bits;
134+}
135+
136+/**
137+ * ionic_qid_to_bitid() - Transform a queue id into a resource bit index
138+ * @qid: queue index
139+ * @qgrp_shift: Log2 number of queues per queue group
140+ * @half_qid_shift: Log2 of half the total number of queues
141+ *
142+ * Return: Resource bit index
143+ *
144+ * This is the inverse of ionic_bitid_to_qid().
145+ */
146+static inline u32 ionic_qid_to_bitid(u32 qid, u8 qgrp_shift, u8 half_qid_shift)
147+{
148+ u32 udma_bit = (qid & BIT(qgrp_shift)) << (half_qid_shift - qgrp_shift);
149+ u32 grp_bits = (qid & GENMASK(half_qid_shift, qgrp_shift + 1)) >> 1;
150+ u32 idx_bits = qid & (BIT(qgrp_shift) - 1);
151+
152+ return udma_bit | grp_bits | idx_bits;
153+}
154+#endif /* _IONIC_RES_H_ */
drivers/infiniband/hw/irdma/Kconfig+4−3
@@ -4,10 +4,11 @@ config INFINIBAND_IRDMA · 3 unmodified lines
44 depends on INET
55 depends on IPV6 || !IPV6
66 depends on PCI
7− depends on ICE && I40E
7+ depends on IDPF && ICE && I40E
88 select GENERIC_ALLOCATOR
99 select AUXILIARY_BUS
1010 select CRC32
1111 help
12− This is an Intel(R) Ethernet Protocol Driver for RDMA driver
13− that support E810 (iWARP/RoCE) and X722 (iWARP) network devices.
12+ This is an Intel(R) Ethernet Protocol Driver for RDMA that
13+ supports IPU E2000 (RoCEv2), E810 (iWARP/RoCEv2) and X722 (iWARP)
14+ network devices.
drivers/infiniband/hw/irdma/Makefile+4−0
@@ -13,7 +13,10 @@ irdma-objs := cm.o \ · 12 unmodified lines
1313 hw.o \
1414 i40iw_hw.o \
1515 i40iw_if.o \
16+ ig3rdma_if.o\
17+ icrdma_if.o \
1618 icrdma_hw.o \
19+ ig3rdma_hw.o\
1720 main.o \
1821 pble.o \
1922 puda.o \
@@ -22,6 +25,7 @@ irdma-objs := cm.o \ · 2 unmodified lines
2225 uk.o \
2326 utils.o \
2427 verbs.o \
28+ virtchnl.o \
2529 ws.o \
2630
2731 CFLAGS_trace.o = -I$(src)
drivers/infiniband/hw/irdma/ctrl.c+1338−130
@@ -74,6 +74,14 @@ static void irdma_set_qos_info(struct irdma_sc_vsi *vsi, · 73 unmodified lines
7474 {
7575 u8 i;
7676
77+ if (vsi->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) {
78+ for (i = 0; i < IRDMA_MAX_USER_PRIORITY; i++) {
79+ vsi->qos[i].qs_handle = vsi->dev->qos[i].qs_handle;
80+ vsi->qos[i].valid = true;
81+ }
82+
83+ return;
84+ }
7785 vsi->qos_rel_bw = l2p->vsi_rel_bw;
7886 vsi->qos_prio_type = l2p->vsi_prio_type;
7987 vsi->dscp_mode = l2p->dscp_mode;
@@ -404,7 +412,8 @@ int irdma_sc_qp_init(struct irdma_sc_qp *qp, struct irdma_qp_init_info *info) · 324 unmodified lines
404412 pble_obj_cnt = info->pd->dev->hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].cnt;
405413
406414 if ((info->virtual_map && info->sq_pa >= pble_obj_cnt) ||
407− (info->virtual_map && info->rq_pa >= pble_obj_cnt))
415+ (!info->qp_uk_init_info.srq_uk &&
416+ info->virtual_map && info->rq_pa >= pble_obj_cnt))
408417 return -EINVAL;
409418
410419 qp->llp_stream_handle = (void *)(-1);
@@ -438,6 +447,208 @@ int irdma_sc_qp_init(struct irdma_sc_qp *qp, struct irdma_qp_init_info *info) · 27 unmodified lines
438447 return 0;
439448 }
440449
450+/**
451+ * irdma_sc_srq_init - init sc_srq structure
452+ * @srq: srq sc struct
453+ * @info: parameters for srq init
454+ */
455+int irdma_sc_srq_init(struct irdma_sc_srq *srq,
456+ struct irdma_srq_init_info *info)
457+{
458+ u32 srq_size_quanta;
459+ int ret_code;
460+
461+ ret_code = irdma_uk_srq_init(&srq->srq_uk, &info->srq_uk_init_info);
462+ if (ret_code)
463+ return ret_code;
464+
465+ srq->dev = info->pd->dev;
466+ srq->pd = info->pd;
467+ srq->vsi = info->vsi;
468+ srq->srq_pa = info->srq_pa;
469+ srq->first_pm_pbl_idx = info->first_pm_pbl_idx;
470+ srq->pasid = info->pasid;
471+ srq->pasid_valid = info->pasid_valid;
472+ srq->srq_limit = info->srq_limit;
473+ srq->leaf_pbl_size = info->leaf_pbl_size;
474+ srq->virtual_map = info->virtual_map;
475+ srq->tph_en = info->tph_en;
476+ srq->arm_limit_event = info->arm_limit_event;
477+ srq->tph_val = info->tph_value;
478+ srq->shadow_area_pa = info->shadow_area_pa;
479+
480+ /* Smallest SRQ size is 256B i.e. 8 quanta */
481+ srq_size_quanta = max((u32)IRDMA_SRQ_MIN_QUANTA,
482+ srq->srq_uk.srq_size *
483+ srq->srq_uk.wqe_size_multiplier);
484+ srq->hw_srq_size = irdma_get_encoded_wqe_size(srq_size_quanta,
485+ IRDMA_QUEUE_TYPE_SRQ);
486+
487+ return 0;
488+}
489+
490+/**
491+ * irdma_sc_srq_create - send srq create CQP WQE
492+ * @srq: srq sc struct
493+ * @scratch: u64 saved to be used during cqp completion
494+ * @post_sq: flag for cqp db to ring
495+ */
496+static int irdma_sc_srq_create(struct irdma_sc_srq *srq, u64 scratch,
497+ bool post_sq)
498+{
499+ struct irdma_sc_cqp *cqp;
500+ __le64 *wqe;
501+ u64 hdr;
502+
503+ cqp = srq->pd->dev->cqp;
504+ if (srq->srq_uk.srq_id < cqp->dev->hw_attrs.min_hw_srq_id ||
505+ srq->srq_uk.srq_id >
506+ (cqp->dev->hmc_info->hmc_obj[IRDMA_HMC_IW_SRQ].max_cnt - 1))
507+ return -EINVAL;
508+
509+ wqe = irdma_sc_cqp_get_next_send_wqe(cqp, scratch);
510+ if (!wqe)
511+ return -ENOMEM;
512+
513+ set_64bit_val(wqe, 0,
514+ FIELD_PREP(IRDMA_CQPSQ_SRQ_SRQ_LIMIT, srq->srq_limit) |
515+ FIELD_PREP(IRDMA_CQPSQ_SRQ_RQSIZE, srq->hw_srq_size) |
516+ FIELD_PREP(IRDMA_CQPSQ_SRQ_RQ_WQE_SIZE, srq->srq_uk.wqe_size));
517+ set_64bit_val(wqe, 8, (uintptr_t)srq);
518+ set_64bit_val(wqe, 16,
519+ FIELD_PREP(IRDMA_CQPSQ_SRQ_PD_ID, srq->pd->pd_id));
520+ set_64bit_val(wqe, 32,
521+ FIELD_PREP(IRDMA_CQPSQ_SRQ_PHYSICAL_BUFFER_ADDR,
522+ srq->srq_pa >>
523+ IRDMA_CQPSQ_SRQ_PHYSICAL_BUFFER_ADDR_S));
524+ set_64bit_val(wqe, 40,
525+ FIELD_PREP(IRDMA_CQPSQ_SRQ_DB_SHADOW_ADDR,
526+ srq->shadow_area_pa >>
527+ IRDMA_CQPSQ_SRQ_DB_SHADOW_ADDR_S));
528+ set_64bit_val(wqe, 48,
529+ FIELD_PREP(IRDMA_CQPSQ_SRQ_FIRST_PM_PBL_IDX,
530+ srq->first_pm_pbl_idx));
531+
532+ hdr = srq->srq_uk.srq_id |
533+ FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_CREATE_SRQ) |
534+ FIELD_PREP(IRDMA_CQPSQ_SRQ_LEAF_PBL_SIZE, srq->leaf_pbl_size) |
535+ FIELD_PREP(IRDMA_CQPSQ_SRQ_VIRTMAP, srq->virtual_map) |
536+ FIELD_PREP(IRDMA_CQPSQ_SRQ_ARM_LIMIT_EVENT,
537+ srq->arm_limit_event) |
538+ FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity);
539+
540+ dma_wmb(); /* make sure WQE is written before valid bit is set */
541+
542+ set_64bit_val(wqe, 24, hdr);
543+
544+ print_hex_dump_debug("WQE: SRQ_CREATE WQE", DUMP_PREFIX_OFFSET, 16, 8,
545+ wqe, IRDMA_CQP_WQE_SIZE * 8, false);
546+ if (post_sq)
547+ irdma_sc_cqp_post_sq(cqp);
548+
549+ return 0;
550+}
551+
552+/**
553+ * irdma_sc_srq_modify - send modify_srq CQP WQE
554+ * @srq: srq sc struct
555+ * @info: parameters for srq modification
556+ * @scratch: u64 saved to be used during cqp completion
557+ * @post_sq: flag for cqp db to ring
558+ */
559+static int irdma_sc_srq_modify(struct irdma_sc_srq *srq,
560+ struct irdma_modify_srq_info *info, u64 scratch,
561+ bool post_sq)
562+{
563+ struct irdma_sc_cqp *cqp;
564+ __le64 *wqe;
565+ u64 hdr;
566+
567+ cqp = srq->dev->cqp;
568+ if (srq->srq_uk.srq_id < cqp->dev->hw_attrs.min_hw_srq_id ||
569+ srq->srq_uk.srq_id >
570+ (cqp->dev->hmc_info->hmc_obj[IRDMA_HMC_IW_SRQ].max_cnt - 1))
571+ return -EINVAL;
572+
573+ wqe = irdma_sc_cqp_get_next_send_wqe(cqp, scratch);
574+ if (!wqe)
575+ return -ENOMEM;
576+
577+ set_64bit_val(wqe, 0,
578+ FIELD_PREP(IRDMA_CQPSQ_SRQ_SRQ_LIMIT, info->srq_limit) |
579+ FIELD_PREP(IRDMA_CQPSQ_SRQ_RQSIZE, srq->hw_srq_size) |
580+ FIELD_PREP(IRDMA_CQPSQ_SRQ_RQ_WQE_SIZE, srq->srq_uk.wqe_size));
581+ set_64bit_val(wqe, 8,
582+ FIELD_PREP(IRDMA_CQPSQ_SRQ_SRQCTX, srq->srq_uk.srq_id));
583+ set_64bit_val(wqe, 16,
584+ FIELD_PREP(IRDMA_CQPSQ_SRQ_PD_ID, srq->pd->pd_id));
585+ set_64bit_val(wqe, 32,
586+ FIELD_PREP(IRDMA_CQPSQ_SRQ_PHYSICAL_BUFFER_ADDR,
587+ srq->srq_pa >>
588+ IRDMA_CQPSQ_SRQ_PHYSICAL_BUFFER_ADDR_S));
589+ set_64bit_val(wqe, 40,
590+ FIELD_PREP(IRDMA_CQPSQ_SRQ_DB_SHADOW_ADDR,
591+ srq->shadow_area_pa >>
592+ IRDMA_CQPSQ_SRQ_DB_SHADOW_ADDR_S));
593+ set_64bit_val(wqe, 48,
594+ FIELD_PREP(IRDMA_CQPSQ_SRQ_FIRST_PM_PBL_IDX,
595+ srq->first_pm_pbl_idx));
596+
597+ hdr = srq->srq_uk.srq_id |
598+ FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_MODIFY_SRQ) |
599+ FIELD_PREP(IRDMA_CQPSQ_SRQ_LEAF_PBL_SIZE, srq->leaf_pbl_size) |
600+ FIELD_PREP(IRDMA_CQPSQ_SRQ_VIRTMAP, srq->virtual_map) |
601+ FIELD_PREP(IRDMA_CQPSQ_SRQ_ARM_LIMIT_EVENT,
602+ info->arm_limit_event) |
603+ FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity);
604+ dma_wmb(); /* make sure WQE is written before valid bit is set */
605+
606+ set_64bit_val(wqe, 24, hdr);
607+
608+ print_hex_dump_debug("WQE: SRQ_MODIFY WQE", DUMP_PREFIX_OFFSET, 16, 8,
609+ wqe, IRDMA_CQP_WQE_SIZE * 8, false);
610+ if (post_sq)
611+ irdma_sc_cqp_post_sq(cqp);
612+
613+ return 0;
614+}
615+
616+/**
617+ * irdma_sc_srq_destroy - send srq_destroy CQP WQE
618+ * @srq: srq sc struct
619+ * @scratch: u64 saved to be used during cqp completion
620+ * @post_sq: flag for cqp db to ring
621+ */
622+static int irdma_sc_srq_destroy(struct irdma_sc_srq *srq, u64 scratch,
623+ bool post_sq)
624+{
625+ struct irdma_sc_cqp *cqp;
626+ __le64 *wqe;
627+ u64 hdr;
628+
629+ cqp = srq->dev->cqp;
630+
631+ wqe = irdma_sc_cqp_get_next_send_wqe(cqp, scratch);
632+ if (!wqe)
633+ return -ENOMEM;
634+
635+ set_64bit_val(wqe, 8, (uintptr_t)srq);
636+
637+ hdr = srq->srq_uk.srq_id |
638+ FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_DESTROY_SRQ) |
639+ FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity);
640+ dma_wmb(); /* make sure WQE is written before valid bit is set */
641+
642+ set_64bit_val(wqe, 24, hdr);
643+
644+ print_hex_dump_debug("WQE: SRQ_DESTROY WQE", DUMP_PREFIX_OFFSET, 16,
645+ 8, wqe, IRDMA_CQP_WQE_SIZE * 8, false);
646+ if (post_sq)
647+ irdma_sc_cqp_post_sq(cqp);
648+
649+ return 0;
650+}
651+
441652 /**
442653 * irdma_sc_qp_create - create qp
443654 * @qp: sc qp
@@ -629,13 +840,14 @@ static u8 irdma_sc_get_encoded_ird_size(u16 ird_size) · 185 unmodified lines
629840 }
630841
631842 /**
632− * irdma_sc_qp_setctx_roce - set qp's context
843+ * irdma_sc_qp_setctx_roce_gen_2 - set qp's context
633844 * @qp: sc qp
634845 * @qp_ctx: context ptr
635846 * @info: ctx info
636847 */
637−void irdma_sc_qp_setctx_roce(struct irdma_sc_qp *qp, __le64 *qp_ctx,
638− struct irdma_qp_host_ctx_info *info)
848+static void irdma_sc_qp_setctx_roce_gen_2(struct irdma_sc_qp *qp,
849+ __le64 *qp_ctx,
850+ struct irdma_qp_host_ctx_info *info)
639851 {
640852 struct irdma_roce_offload_info *roce_info;
641853 struct irdma_udp_offload_info *udp;
@@ -753,6 +965,189 @@ void irdma_sc_qp_setctx_roce(struct irdma_sc_qp *qp, __le64 *qp_ctx, · 111 unmodified lines
753965 8, qp_ctx, IRDMA_QP_CTX_SIZE, false);
754966 }
755967
968+/**
969+ * irdma_sc_get_encoded_ird_size_gen_3 - get encoded IRD size for GEN 3
970+ * @ird_size: IRD size
971+ * The ird from the connection is rounded to a supported HW setting and then encoded
972+ * for ird_size field of qp_ctx. Consumers are expected to provide valid ird size based
973+ * on hardware attributes. IRD size defaults to a value of 4 in case of invalid input.
974+ */
975+static u8 irdma_sc_get_encoded_ird_size_gen_3(u16 ird_size)
976+{
977+ switch (ird_size ?
978+ roundup_pow_of_two(2 * ird_size) : 4) {
979+ case 4096:
980+ return IRDMA_IRD_HW_SIZE_4096_GEN3;
981+ case 2048:
982+ return IRDMA_IRD_HW_SIZE_2048_GEN3;
983+ case 1024:
984+ return IRDMA_IRD_HW_SIZE_1024_GEN3;
985+ case 512:
986+ return IRDMA_IRD_HW_SIZE_512_GEN3;
987+ case 256:
988+ return IRDMA_IRD_HW_SIZE_256_GEN3;
989+ case 128:
990+ return IRDMA_IRD_HW_SIZE_128_GEN3;
991+ case 64:
992+ return IRDMA_IRD_HW_SIZE_64_GEN3;
993+ case 32:
994+ return IRDMA_IRD_HW_SIZE_32_GEN3;
995+ case 16:
996+ return IRDMA_IRD_HW_SIZE_16_GEN3;
997+ case 8:
998+ return IRDMA_IRD_HW_SIZE_8_GEN3;
999+ case 4:
1000+ default:
1001+ break;
1002+ }
1003+
1004+ return IRDMA_IRD_HW_SIZE_4_GEN3;
1005+}
1006+
1007+/**
1008+ * irdma_sc_qp_setctx_roce_gen_3 - set qp's context
1009+ * @qp: sc qp
1010+ * @qp_ctx: context ptr
1011+ * @info: ctx info
1012+ */
1013+static void irdma_sc_qp_setctx_roce_gen_3(struct irdma_sc_qp *qp,
1014+ __le64 *qp_ctx,
1015+ struct irdma_qp_host_ctx_info *info)
1016+{
1017+ struct irdma_roce_offload_info *roce_info = info->roce_info;
1018+ struct irdma_udp_offload_info *udp = info->udp_info;
1019+ u64 qw0, qw3, qw7 = 0, qw8 = 0;
1020+ u8 push_mode_en;
1021+ u32 push_idx;
1022+
1023+ qp->user_pri = info->user_pri;
1024+ if (qp->push_idx == IRDMA_INVALID_PUSH_PAGE_INDEX) {
1025+ push_mode_en = 0;
1026+ push_idx = 0;
1027+ } else {
1028+ push_mode_en = 1;
1029+ push_idx = qp->push_idx;
1030+ }
1031+
1032+ qw0 = FIELD_PREP(IRDMAQPC_RQWQESIZE, qp->qp_uk.rq_wqe_size) |
1033+ FIELD_PREP(IRDMAQPC_RCVTPHEN, qp->rcv_tph_en) |
1034+ FIELD_PREP(IRDMAQPC_XMITTPHEN, qp->xmit_tph_en) |
1035+ FIELD_PREP(IRDMAQPC_RQTPHEN, qp->rq_tph_en) |
1036+ FIELD_PREP(IRDMAQPC_SQTPHEN, qp->sq_tph_en) |
1037+ FIELD_PREP(IRDMAQPC_PPIDX, push_idx) |
1038+ FIELD_PREP(IRDMAQPC_PMENA, push_mode_en) |
1039+ FIELD_PREP(IRDMAQPC_DC_TCP_EN, roce_info->dctcp_en) |
1040+ FIELD_PREP(IRDMAQPC_ISQP1, roce_info->is_qp1) |
1041+ FIELD_PREP(IRDMAQPC_ROCE_TVER, roce_info->roce_tver) |
1042+ FIELD_PREP(IRDMAQPC_IPV4, udp->ipv4) |
1043+ FIELD_PREP(IRDMAQPC_USE_SRQ, !qp->qp_uk.srq_uk ? 0 : 1) |
1044+ FIELD_PREP(IRDMAQPC_INSERTVLANTAG, udp->insert_vlan_tag);
1045+ set_64bit_val(qp_ctx, 0, qw0);
1046+ set_64bit_val(qp_ctx, 8, qp->sq_pa);
1047+ set_64bit_val(qp_ctx, 16, qp->rq_pa);
1048+ qw3 = FIELD_PREP(IRDMAQPC_RQSIZE, qp->hw_rq_size) |
1049+ FIELD_PREP(IRDMAQPC_SQSIZE, qp->hw_sq_size) |
1050+ FIELD_PREP(IRDMAQPC_TTL, udp->ttl) |
1051+ FIELD_PREP(IRDMAQPC_TOS, udp->tos) |
1052+ FIELD_PREP(IRDMAQPC_SRCPORTNUM, udp->src_port) |
1053+ FIELD_PREP(IRDMAQPC_DESTPORTNUM, udp->dst_port);
1054+ set_64bit_val(qp_ctx, 24, qw3);
1055+ set_64bit_val(qp_ctx, 32,
1056+ FIELD_PREP(IRDMAQPC_DESTIPADDR2, udp->dest_ip_addr[2]) |
1057+ FIELD_PREP(IRDMAQPC_DESTIPADDR3, udp->dest_ip_addr[3]));
1058+ set_64bit_val(qp_ctx, 40,
1059+ FIELD_PREP(IRDMAQPC_DESTIPADDR0, udp->dest_ip_addr[0]) |
1060+ FIELD_PREP(IRDMAQPC_DESTIPADDR1, udp->dest_ip_addr[1]));
1061+ set_64bit_val(qp_ctx, 48,
1062+ FIELD_PREP(IRDMAQPC_SNDMSS, udp->snd_mss) |
1063+ FIELD_PREP(IRDMAQPC_VLANTAG, udp->vlan_tag) |
1064+ FIELD_PREP(IRDMAQPC_ARPIDX, udp->arp_idx));
1065+ qw7 = FIELD_PREP(IRDMAQPC_PKEY, roce_info->p_key) |
1066+ FIELD_PREP(IRDMAQPC_ACKCREDITS, roce_info->ack_credits) |
1067+ FIELD_PREP(IRDMAQPC_FLOWLABEL, udp->flow_label);
1068+ set_64bit_val(qp_ctx, 56, qw7);
1069+ qw8 = FIELD_PREP(IRDMAQPC_QKEY, roce_info->qkey) |
1070+ FIELD_PREP(IRDMAQPC_DESTQP, roce_info->dest_qp);
1071+ set_64bit_val(qp_ctx, 64, qw8);
1072+ set_64bit_val(qp_ctx, 80,
1073+ FIELD_PREP(IRDMAQPC_PSNNXT, udp->psn_nxt) |
1074+ FIELD_PREP(IRDMAQPC_LSN, udp->lsn));
1075+ set_64bit_val(qp_ctx, 88,
1076+ FIELD_PREP(IRDMAQPC_EPSN, udp->epsn));
1077+ set_64bit_val(qp_ctx, 96,
1078+ FIELD_PREP(IRDMAQPC_PSNMAX, udp->psn_max) |
1079+ FIELD_PREP(IRDMAQPC_PSNUNA, udp->psn_una));
1080+ set_64bit_val(qp_ctx, 112,
1081+ FIELD_PREP(IRDMAQPC_CWNDROCE, udp->cwnd));
1082+ set_64bit_val(qp_ctx, 128,
1083+ FIELD_PREP(IRDMAQPC_MINRNR_TIMER, udp->min_rnr_timer) |
1084+ FIELD_PREP(IRDMAQPC_RNRNAK_THRESH, udp->rnr_nak_thresh) |
1085+ FIELD_PREP(IRDMAQPC_REXMIT_THRESH, udp->rexmit_thresh) |
1086+ FIELD_PREP(IRDMAQPC_RNRNAK_TMR, udp->rnr_nak_tmr) |
1087+ FIELD_PREP(IRDMAQPC_RTOMIN, roce_info->rtomin));
1088+ set_64bit_val(qp_ctx, 136,
1089+ FIELD_PREP(IRDMAQPC_TXCQNUM, info->send_cq_num) |
1090+ FIELD_PREP(IRDMAQPC_RXCQNUM, info->rcv_cq_num));
1091+ set_64bit_val(qp_ctx, 152,
1092+ FIELD_PREP(IRDMAQPC_MACADDRESS,
1093+ ether_addr_to_u64(roce_info->mac_addr)) |
1094+ FIELD_PREP(IRDMAQPC_LOCALACKTIMEOUT,
1095+ roce_info->local_ack_timeout));
1096+ set_64bit_val(qp_ctx, 160,
1097+ FIELD_PREP(IRDMAQPC_ORDSIZE_GEN3, roce_info->ord_size) |
1098+ FIELD_PREP(IRDMAQPC_IRDSIZE_GEN3,
1099+ irdma_sc_get_encoded_ird_size_gen_3(roce_info->ird_size)) |
1100+ FIELD_PREP(IRDMAQPC_WRRDRSPOK, roce_info->wr_rdresp_en) |
1101+ FIELD_PREP(IRDMAQPC_RDOK, roce_info->rd_en) |
1102+ FIELD_PREP(IRDMAQPC_USESTATSINSTANCE,
1103+ info->stats_idx_valid) |
1104+ FIELD_PREP(IRDMAQPC_BINDEN, roce_info->bind_en) |
1105+ FIELD_PREP(IRDMAQPC_FASTREGEN, roce_info->fast_reg_en) |
1106+ FIELD_PREP(IRDMAQPC_DCQCNENABLE, roce_info->dcqcn_en) |
1107+ FIELD_PREP(IRDMAQPC_RCVNOICRC, roce_info->rcv_no_icrc) |
1108+ FIELD_PREP(IRDMAQPC_FW_CC_ENABLE,
1109+ roce_info->fw_cc_enable) |
1110+ FIELD_PREP(IRDMAQPC_UDPRIVCQENABLE,
1111+ roce_info->udprivcq_en) |
1112+ FIELD_PREP(IRDMAQPC_PRIVEN, roce_info->priv_mode_en) |
1113+ FIELD_PREP(IRDMAQPC_REMOTE_ATOMIC_EN,
1114+ info->remote_atomics_en) |
1115+ FIELD_PREP(IRDMAQPC_TIMELYENABLE, roce_info->timely_en));
1116+ set_64bit_val(qp_ctx, 168,
1117+ FIELD_PREP(IRDMAQPC_QPCOMPCTX, info->qp_compl_ctx));
1118+ set_64bit_val(qp_ctx, 176,
1119+ FIELD_PREP(IRDMAQPC_SQTPHVAL, qp->sq_tph_val) |
1120+ FIELD_PREP(IRDMAQPC_RQTPHVAL, qp->rq_tph_val) |
1121+ FIELD_PREP(IRDMAQPC_QSHANDLE, qp->qs_handle));
1122+ set_64bit_val(qp_ctx, 184,
1123+ FIELD_PREP(IRDMAQPC_LOCAL_IPADDR3, udp->local_ipaddr[3]) |
1124+ FIELD_PREP(IRDMAQPC_LOCAL_IPADDR2, udp->local_ipaddr[2]));
1125+ set_64bit_val(qp_ctx, 192,
1126+ FIELD_PREP(IRDMAQPC_LOCAL_IPADDR1, udp->local_ipaddr[1]) |
1127+ FIELD_PREP(IRDMAQPC_LOCAL_IPADDR0, udp->local_ipaddr[0]));
1128+ set_64bit_val(qp_ctx, 200,
1129+ FIELD_PREP(IRDMAQPC_THIGH, roce_info->t_high) |
1130+ FIELD_PREP(IRDMAQPC_SRQ_ID,
1131+ !qp->qp_uk.srq_uk ?
1132+ 0 : qp->qp_uk.srq_uk->srq_id) |
1133+ FIELD_PREP(IRDMAQPC_TLOW, roce_info->t_low));
1134+ set_64bit_val(qp_ctx, 208, roce_info->pd_id |
1135+ FIELD_PREP(IRDMAQPC_STAT_INDEX_GEN3, info->stats_idx) |
1136+ FIELD_PREP(IRDMAQPC_PKT_LIMIT, qp->pkt_limit));
1137+
1138+ print_hex_dump_debug("WQE: QP_HOST ROCE CTX WQE", DUMP_PREFIX_OFFSET,
1139+ 16, 8, qp_ctx, IRDMA_QP_CTX_SIZE, false);
1140+}
1141+
1142+void irdma_sc_qp_setctx_roce(struct irdma_sc_qp *qp, __le64 *qp_ctx,
1143+ struct irdma_qp_host_ctx_info *info)
1144+{
1145+ if (qp->dev->hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_2)
1146+ irdma_sc_qp_setctx_roce_gen_2(qp, qp_ctx, info);
1147+ else
1148+ irdma_sc_qp_setctx_roce_gen_3(qp, qp_ctx, info);
1149+}
1150+
7561151 /* irdma_sc_alloc_local_mac_entry - allocate a mac entry
7571152 * @cqp: struct for cqp hw
7581153 * @scratch: u64 saved to be used during cqp completion
@@ -1080,7 +1475,8 @@ static int irdma_sc_alloc_stag(struct irdma_sc_dev *dev, · 321 unmodified lines
10801475 FLD_LS_64(dev, info->pd_id, IRDMA_CQPSQ_STAG_PDID) |
10811476 FIELD_PREP(IRDMA_CQPSQ_STAG_STAGLEN, info->total_len));
10821477 set_64bit_val(wqe, 16,
1083− FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->stag_idx));
1478+ FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->stag_idx) |
1479+ FIELD_PREP(IRDMA_CQPSQ_STAG_PDID_HI, info->pd_id >> 18));
10841480 set_64bit_val(wqe, 40,
10851481 FIELD_PREP(IRDMA_CQPSQ_STAG_HMCFNIDX, info->hmc_fcn_index));
10861482
@@ -1096,6 +1492,8 @@ static int irdma_sc_alloc_stag(struct irdma_sc_dev *dev, · 9 unmodified lines
10961492 FIELD_PREP(IRDMA_CQPSQ_STAG_REMACCENABLED, info->remote_access) |
10971493 FIELD_PREP(IRDMA_CQPSQ_STAG_USEHMCFNIDX, info->use_hmc_fcn_index) |
10981494 FIELD_PREP(IRDMA_CQPSQ_STAG_USEPFRID, info->use_pf_rid) |
1495+ FIELD_PREP(IRDMA_CQPSQ_STAG_REMOTE_ATOMIC_EN,
1496+ info->remote_atomics_en) |
10991497 FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity);
11001498 dma_wmb(); /* make sure WQE is written before valid bit is set */
11011499
@@ -1165,6 +1563,7 @@ static int irdma_sc_mr_reg_non_shared(struct irdma_sc_dev *dev, · 63 unmodified lines
11651563 FLD_LS_64(dev, info->pd_id, IRDMA_CQPSQ_STAG_PDID));
11661564 set_64bit_val(wqe, 16,
11671565 FIELD_PREP(IRDMA_CQPSQ_STAG_KEY, info->stag_key) |
1566+ FIELD_PREP(IRDMA_CQPSQ_STAG_PDID_HI, info->pd_id >> 18) |
11681567 FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->stag_idx));
11691568 if (!info->chunk_size) {
11701569 set_64bit_val(wqe, 32, info->reg_addr_pa);
@@ -1187,6 +1586,8 @@ static int irdma_sc_mr_reg_non_shared(struct irdma_sc_dev *dev, · 16 unmodified lines
11871586 FIELD_PREP(IRDMA_CQPSQ_STAG_VABASEDTO, addr_type) |
11881587 FIELD_PREP(IRDMA_CQPSQ_STAG_USEHMCFNIDX, info->use_hmc_fcn_index) |
11891588 FIELD_PREP(IRDMA_CQPSQ_STAG_USEPFRID, info->use_pf_rid) |
1589+ FIELD_PREP(IRDMA_CQPSQ_STAG_REMOTE_ATOMIC_EN,
1590+ info->remote_atomics_en) |
11901591 FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity);
11911592 dma_wmb(); /* make sure WQE is written before valid bit is set */
11921593
@@ -1223,7 +1624,8 @@ static int irdma_sc_dealloc_stag(struct irdma_sc_dev *dev, · 30 unmodified lines
12231624 set_64bit_val(wqe, 8,
12241625 FLD_LS_64(dev, info->pd_id, IRDMA_CQPSQ_STAG_PDID));
12251626 set_64bit_val(wqe, 16,
1226− FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->stag_idx));
1627+ FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->stag_idx) |
1628+ FIELD_PREP(IRDMA_CQPSQ_STAG_PDID_HI, info->pd_id >> 18));
12271629
12281630 hdr = FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_DEALLOC_STAG) |
12291631 FIELD_PREP(IRDMA_CQPSQ_STAG_MR, info->mr) |
@@ -1263,7 +1665,8 @@ static int irdma_sc_mw_alloc(struct irdma_sc_dev *dev, · 33 unmodified lines
12631665 set_64bit_val(wqe, 8,
12641666 FLD_LS_64(dev, info->pd_id, IRDMA_CQPSQ_STAG_PDID));
12651667 set_64bit_val(wqe, 16,
1266− FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->mw_stag_index));
1668+ FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->mw_stag_index) |
1669+ FIELD_PREP(IRDMA_CQPSQ_STAG_PDID_HI, info->pd_id >> 18));
12671670
12681671 hdr = FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_ALLOC_STAG) |
12691672 FIELD_PREP(IRDMA_CQPSQ_STAG_MWTYPE, info->mw_wide) |
@@ -1343,6 +1746,7 @@ int irdma_sc_mr_fast_register(struct irdma_sc_qp *qp, · 73 unmodified lines
13431746 FIELD_PREP(IRDMAQPSQ_READFENCE, info->read_fence) |
13441747 FIELD_PREP(IRDMAQPSQ_LOCALFENCE, info->local_fence) |
13451748 FIELD_PREP(IRDMAQPSQ_SIGCOMPL, info->signaled) |
1749+ FIELD_PREP(IRDMAQPSQ_REMOTE_ATOMICS_EN, info->remote_atomics_en) |
13461750 FIELD_PREP(IRDMAQPSQ_VALID, qp->qp_uk.swqe_polarity);
13471751 dma_wmb(); /* make sure WQE is written before valid bit is set */
13481752
@@ -1873,7 +2277,7 @@ void irdma_sc_vsi_init(struct irdma_sc_vsi *vsi, · 524 unmodified lines
18732277 mutex_init(&vsi->qos[i].qos_mutex);
18742278 INIT_LIST_HEAD(&vsi->qos[i].qplist);
18752279 }
1876− if (vsi->register_qset) {
2280+ if (vsi->dev->hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_2) {
18772281 vsi->dev->ws_add = irdma_ws_add;
18782282 vsi->dev->ws_remove = irdma_ws_remove;
18792283 vsi->dev->ws_reset = irdma_ws_reset;
@@ -1888,7 +2292,7 @@ void irdma_sc_vsi_init(struct irdma_sc_vsi *vsi, · 8 unmodified lines
18882292 * irdma_get_stats_idx - Return stats index
18892293 * @vsi: pointer to the vsi
18902294 */
1891−static u8 irdma_get_stats_idx(struct irdma_sc_vsi *vsi)
2295+static u16 irdma_get_stats_idx(struct irdma_sc_vsi *vsi)
18922296 {
18932297 struct irdma_stats_inst_info stats_info = {};
18942298 struct irdma_sc_dev *dev = vsi->dev;
@@ -1964,12 +2368,13 @@ int irdma_vsi_stats_init(struct irdma_sc_vsi *vsi, · 69 unmodified lines
19642368 (void *)((uintptr_t)stats_buff_mem->va +
19652369 IRDMA_GATHER_STATS_BUF_SIZE);
19662370
1967− irdma_hw_stats_start_timer(vsi);
2371+ if (vsi->dev->hw_attrs.uk_attrs.hw_rev < IRDMA_GEN_3)
2372+ irdma_hw_stats_start_timer(vsi);
19682373
19692374 /* when stat allocation is not required default to fcn_id. */
19702375 vsi->stats_idx = info->fcn_id;
19712376 if (info->alloc_stats_inst) {
1972− u8 stats_idx = irdma_get_stats_idx(vsi);
2377+ u16 stats_idx = irdma_get_stats_idx(vsi);
19732378
19742379 if (stats_idx != IRDMA_INVALID_STATS_IDX) {
19752380 vsi->stats_inst_alloc = true;
@@ -1993,7 +2398,7 @@ void irdma_vsi_stats_free(struct irdma_sc_vsi *vsi) · 17 unmodified lines
19932398 {
19942399 struct irdma_stats_inst_info stats_info = {};
19952400 struct irdma_sc_dev *dev = vsi->dev;
1996− u8 stats_idx = vsi->stats_idx;
2401+ u16 stats_idx = vsi->stats_idx;
19972402
19982403 if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_2) {
19992404 if (vsi->stats_inst_alloc) {
@@ -2009,7 +2414,9 @@ void irdma_vsi_stats_free(struct irdma_sc_vsi *vsi) · 9 unmodified lines
20092414
20102415 if (!vsi->pestat)
20112416 return;
2012− irdma_hw_stats_stop_timer(vsi);
2417+
2418+ if (dev->hw_attrs.uk_attrs.hw_rev < IRDMA_GEN_3)
2419+ irdma_hw_stats_stop_timer(vsi);
20132420 dma_free_coherent(vsi->pestat->hw->device,
20142421 vsi->pestat->gather_info.stats_buff_mem.size,
20152422 vsi->pestat->gather_info.stats_buff_mem.va,
@@ -2026,6 +2433,14 @@ u8 irdma_get_encoded_wqe_size(u32 wqsize, enum irdma_queue_type queue_type) · 10 unmodified lines
20262433 {
20272434 u8 encoded_size = 0;
20282435
2436+ if (queue_type == IRDMA_QUEUE_TYPE_SRQ) {
2437+ /* Smallest SRQ size is 256B (8 quanta) that gets
2438+ * encoded to 0.
2439+ */
2440+ encoded_size = ilog2(wqsize) - 3;
2441+
2442+ return encoded_size;
2443+ }
20292444 /* cqp sq's hw coded value starts from 1 for size of 4
20302445 * while it starts from 0 for qp' wq's.
20312446 */
@@ -2259,6 +2674,12 @@ int irdma_sc_qp_flush_wqes(struct irdma_sc_qp *qp, · 227 unmodified lines
22592674 info->ae_code | FIELD_PREP(IRDMA_CQPSQ_FWQE_AESOURCE,
22602675 info->ae_src) : 0;
22612676 set_64bit_val(wqe, 8, temp);
2677+ if (cqp->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) {
2678+ set_64bit_val(wqe, 40,
2679+ FIELD_PREP(IRDMA_CQPSQ_FWQE_ERR_SQ_IDX, info->err_sq_idx));
2680+ set_64bit_val(wqe, 48,
2681+ FIELD_PREP(IRDMA_CQPSQ_FWQE_ERR_RQ_IDX, info->err_rq_idx));
2682+ }
22622683
22632684 hdr = qp->qp_uk.qp_id |
22642685 FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_FLUSH_WQES) |
@@ -2267,6 +2688,9 @@ int irdma_sc_qp_flush_wqes(struct irdma_sc_qp *qp, · 2 unmodified lines
22672688 FIELD_PREP(IRDMA_CQPSQ_FWQE_FLUSHSQ, flush_sq) |
22682689 FIELD_PREP(IRDMA_CQPSQ_FWQE_FLUSHRQ, flush_rq) |
22692690 FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity);
2691+ if (cqp->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3)
2692+ hdr |= FIELD_PREP(IRDMA_CQPSQ_FWQE_ERR_SQ_IDX_VALID, info->err_sq_idx_valid) |
2693+ FIELD_PREP(IRDMA_CQPSQ_FWQE_ERR_RQ_IDX_VALID, info->err_rq_idx_valid);
22702694 dma_wmb(); /* make sure WQE is written before valid bit is set */
22712695
22722696 set_64bit_val(wqe, 24, hdr);
@@ -2562,6 +2986,9 @@ static int irdma_sc_cq_create(struct irdma_sc_cq *cq, u64 scratch, · 289 unmodified lines
25622986 FIELD_PREP(IRDMA_CQPSQ_CQ_LPBLSIZE, cq->pbl_chunk_size) |
25632987 FIELD_PREP(IRDMA_CQPSQ_CQ_CHKOVERFLOW, check_overflow) |
25642988 FIELD_PREP(IRDMA_CQPSQ_CQ_VIRTMAP, cq->virtual_map) |
2989+ FIELD_PREP(IRDMA_CQPSQ_CQ_CQID_HIGH, cq->cq_uk.cq_id >> 22) |
2990+ FIELD_PREP(IRDMA_CQPSQ_CQ_CEQID_HIGH,
2991+ (cq->ceq_id_valid ? cq->ceq_id : 0) >> 10) |
25652992 FIELD_PREP(IRDMA_CQPSQ_CQ_ENCEQEMASK, cq->ceqe_mask) |
25662993 FIELD_PREP(IRDMA_CQPSQ_CQ_CEQIDVALID, cq->ceq_id_valid) |
25672994 FIELD_PREP(IRDMA_CQPSQ_TPHEN, cq->tph_en) |
@@ -2705,6 +3132,41 @@ static int irdma_sc_cq_modify(struct irdma_sc_cq *cq, · 137 unmodified lines
27053132 return 0;
27063133 }
27073134
3135+/**
3136+ * irdma_sc_get_decoded_ird_size_gen_3 - get decoded IRD size for GEN 3
3137+ * @ird_enc: IRD encoding
3138+ * IRD size defaults to a value of 4 in case of invalid input.
3139+ */
3140+static u16 irdma_sc_get_decoded_ird_size_gen_3(u8 ird_enc)
3141+{
3142+ switch (ird_enc) {
3143+ case IRDMA_IRD_HW_SIZE_4096_GEN3:
3144+ return 4096;
3145+ case IRDMA_IRD_HW_SIZE_2048_GEN3:
3146+ return 2048;
3147+ case IRDMA_IRD_HW_SIZE_1024_GEN3:
3148+ return 1024;
3149+ case IRDMA_IRD_HW_SIZE_512_GEN3:
3150+ return 512;
3151+ case IRDMA_IRD_HW_SIZE_256_GEN3:
3152+ return 256;
3153+ case IRDMA_IRD_HW_SIZE_128_GEN3:
3154+ return 128;
3155+ case IRDMA_IRD_HW_SIZE_64_GEN3:
3156+ return 64;
3157+ case IRDMA_IRD_HW_SIZE_32_GEN3:
3158+ return 32;
3159+ case IRDMA_IRD_HW_SIZE_16_GEN3:
3160+ return 16;
3161+ case IRDMA_IRD_HW_SIZE_8_GEN3:
3162+ return 8;
3163+ case IRDMA_IRD_HW_SIZE_4_GEN3:
3164+ return 4;
3165+ default:
3166+ return 4;
3167+ }
3168+}
3169+
27083170 /**
27093171 * irdma_check_cqp_progress - check cqp processing progress
27103172 * @timeout: timeout info struct
@@ -2738,23 +3200,106 @@ static inline void irdma_get_cqp_reg_info(struct irdma_sc_cqp *cqp, u32 *val, · 27 unmodified lines
27383200 }
27393201
27403202 /**
2741− * irdma_cqp_poll_registers - poll cqp registers
2742− * @cqp: struct for cqp hw
2743− * @tail: wqtail register value
2744− * @count: how many times to try for completion
3203+ * irdma_sc_cqp_def_cmpl_ae_handler - remove completed requests from pending list
3204+ * @dev: sc device struct
3205+ * @info: AE entry info
3206+ * @first: true if this is the first call to this handler for given AEQE
3207+ * @scratch: (out) scratch entry pointer
3208+ * @sw_def_info: (in/out) SW ticket value for this AE
3209+ *
3210+ * In case of AE_DEF_CMPL event, this function should be called in a loop
3211+ * until it returns NULL-ptr via scratch.
3212+ * For each call, it looks for a matching CQP request on pending list,
3213+ * removes it from the list and returns the pointer to the associated scratch
3214+ * entry.
3215+ * If this is the first call to this function for given AEQE, sw_def_info
3216+ * value is not used to find matching requests. Instead, it is populated
3217+ * with the value from the first matching cqp_request on the list.
3218+ * For subsequent calls, ooo_op->sw_def_info need to match the value passed
3219+ * by a caller.
3220+ *
3221+ * Return: scratch entry pointer for cqp_request to be released or NULL
3222+ * if no matching request is found.
27453223 */
2746−static int irdma_cqp_poll_registers(struct irdma_sc_cqp *cqp, u32 tail,
2747− u32 count)
3224+void irdma_sc_cqp_def_cmpl_ae_handler(struct irdma_sc_dev *dev,
3225+ struct irdma_aeqe_info *info,
3226+ bool first, u64 *scratch,
3227+ u32 *sw_def_info)
27483228 {
2749− u32 i = 0;
2750− u32 newtail, error, val;
3229+ struct irdma_ooo_cqp_op *ooo_op;
3230+ unsigned long flags;
27513231
2752− while (i++ < count) {
2753− irdma_get_cqp_reg_info(cqp, &val, &newtail, &error);
2754− if (error) {
2755− error = readl(cqp->dev->hw_regs[IRDMA_CQPERRCODES]);
2756− ibdev_dbg(to_ibdev(cqp->dev),
2757− "CQP: CQPERRCODES error_code[x%08X]\n",
3232+ *scratch = 0;
3233+
3234+ spin_lock_irqsave(&dev->cqp->ooo_list_lock, flags);
3235+ list_for_each_entry(ooo_op, &dev->cqp->ooo_pnd, list_entry) {
3236+ if (ooo_op->deferred &&
3237+ ((first && ooo_op->def_info == info->def_info) ||
3238+ (!first && ooo_op->sw_def_info == *sw_def_info))) {
3239+ *sw_def_info = ooo_op->sw_def_info;
3240+ *scratch = ooo_op->scratch;
3241+
3242+ list_move(&ooo_op->list_entry, &dev->cqp->ooo_avail);
3243+ atomic64_inc(&dev->cqp->completed_ops);
3244+
3245+ break;
3246+ }
3247+ }
3248+ spin_unlock_irqrestore(&dev->cqp->ooo_list_lock, flags);
3249+
3250+ if (first && !*scratch)
3251+ ibdev_dbg(to_ibdev(dev),
3252+ "AEQ: deferred completion with unknown ticket: def_info 0x%x\n",
3253+ info->def_info);
3254+}
3255+
3256+/**
3257+ * irdma_sc_cqp_cleanup_handler - remove requests from pending list
3258+ * @dev: sc device struct
3259+ *
3260+ * This function should be called in a loop from irdma_cleanup_pending_cqp_op.
3261+ * For each call, it returns first CQP request on pending list, removes it
3262+ * from the list and returns the pointer to the associated scratch entry.
3263+ *
3264+ * Return: scratch entry pointer for cqp_request to be released or NULL
3265+ * if pending list is empty.
3266+ */
3267+u64 irdma_sc_cqp_cleanup_handler(struct irdma_sc_dev *dev)
3268+{
3269+ struct irdma_ooo_cqp_op *ooo_op;
3270+ u64 scratch = 0;
3271+
3272+ list_for_each_entry(ooo_op, &dev->cqp->ooo_pnd, list_entry) {
3273+ scratch = ooo_op->scratch;
3274+
3275+ list_del(&ooo_op->list_entry);
3276+ list_add(&ooo_op->list_entry, &dev->cqp->ooo_avail);
3277+ atomic64_inc(&dev->cqp->completed_ops);
3278+
3279+ break;
3280+ }
3281+
3282+ return scratch;
3283+}
3284+
3285+/**
3286+ * irdma_cqp_poll_registers - poll cqp registers
3287+ * @cqp: struct for cqp hw
3288+ * @tail: wqtail register value
3289+ * @count: how many times to try for completion
3290+ */
3291+static int irdma_cqp_poll_registers(struct irdma_sc_cqp *cqp, u32 tail,
3292+ u32 count)
3293+{
3294+ u32 i = 0;
3295+ u32 newtail, error, val;
3296+
3297+ while (i++ < count) {
3298+ irdma_get_cqp_reg_info(cqp, &val, &newtail, &error);
3299+ if (error) {
3300+ error = readl(cqp->dev->hw_regs[IRDMA_CQPERRCODES]);
3301+ ibdev_dbg(to_ibdev(cqp->dev),
3302+ "CQP: CQPERRCODES error_code[x%08X]\n",
27583303 error);
27593304 return -EIO;
27603305 }
@@ -2794,7 +3339,10 @@ static u64 irdma_sc_decode_fpm_commit(struct irdma_sc_dev *dev, __le64 *buf, · 33 unmodified lines
27943339 obj_info[rsrc_idx].cnt = (u32)FLD_RS_64(dev, temp, IRDMA_COMMIT_FPM_CQCNT);
27953340 break;
27963341 case IRDMA_HMC_IW_APBVT_ENTRY:
2797− obj_info[rsrc_idx].cnt = 1;
3342+ if (dev->hw_attrs.uk_attrs.hw_rev <= IRDMA_GEN_2)
3343+ obj_info[rsrc_idx].cnt = 1;
3344+ else
3345+ obj_info[rsrc_idx].cnt = 0;
27983346 break;
27993347 default:
28003348 obj_info[rsrc_idx].cnt = (u32)temp;
@@ -2829,7 +3377,8 @@ irdma_sc_parse_fpm_commit_buf(struct irdma_sc_dev *dev, __le64 *buf, · 28 unmodified lines
28293377 IRDMA_HMC_IW_QP);
28303378 irdma_sc_decode_fpm_commit(dev, buf, 8, info,
28313379 IRDMA_HMC_IW_CQ);
2832− /* skiping RSRVD */
3380+ irdma_sc_decode_fpm_commit(dev, buf, 16, info,
3381+ IRDMA_HMC_IW_SRQ);
28333382 irdma_sc_decode_fpm_commit(dev, buf, 24, info,
28343383 IRDMA_HMC_IW_HTE);
28353384 irdma_sc_decode_fpm_commit(dev, buf, 32, info,
@@ -2864,15 +3413,17 @@ irdma_sc_parse_fpm_commit_buf(struct irdma_sc_dev *dev, __le64 *buf, · 28 unmodified lines
28643413 IRDMA_HMC_IW_HDR);
28653414 irdma_sc_decode_fpm_commit(dev, buf, 152, info,
28663415 IRDMA_HMC_IW_MD);
2867− irdma_sc_decode_fpm_commit(dev, buf, 160, info,
2868− IRDMA_HMC_IW_OOISC);
2869− irdma_sc_decode_fpm_commit(dev, buf, 168, info,
2870− IRDMA_HMC_IW_OOISCFFL);
3416+ if (dev->cqp->protocol_used == IRDMA_IWARP_PROTOCOL_ONLY) {
3417+ irdma_sc_decode_fpm_commit(dev, buf, 160, info,
3418+ IRDMA_HMC_IW_OOISC);
3419+ irdma_sc_decode_fpm_commit(dev, buf, 168, info,
3420+ IRDMA_HMC_IW_OOISCFFL);
3421+ }
28713422 }
28723423
28733424 /* searching for the last object in HMC to find the size of the HMC area. */
28743425 for (i = IRDMA_HMC_IW_QP; i < IRDMA_HMC_IW_MAX; i++) {
2875− if (info[i].base > max_base) {
3426+ if (info[i].base > max_base && info[i].cnt) {
28763427 max_base = info[i].base;
28773428 last_hmc_obj = i;
28783429 }
@@ -2927,6 +3478,7 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, · 48 unmodified lines
29273478 struct irdma_hmc_fpm_misc *hmc_fpm_misc)
29283479 {
29293480 struct irdma_hmc_obj_info *obj_info;
3481+ u8 ird_encoding;
29303482 u64 temp;
29313483 u32 size;
29323484 u16 max_pe_sds;
@@ -2935,7 +3487,19 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, · 2 unmodified lines
29353487
29363488 get_64bit_val(buf, 0, &temp);
29373489 hmc_info->first_sd_index = (u16)FIELD_GET(IRDMA_QUERY_FPM_FIRST_PE_SD_INDEX, temp);
2938− max_pe_sds = (u16)FIELD_GET(IRDMA_QUERY_FPM_MAX_PE_SDS, temp);
3490+
3491+ if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3)
3492+ max_pe_sds = (u16)FIELD_GET(IRDMA_QUERY_FPM_MAX_PE_SDS_GEN3, temp);
3493+ else
3494+ max_pe_sds = (u16)FIELD_GET(IRDMA_QUERY_FPM_MAX_PE_SDS, temp);
3495+
3496+ /* Reduce SD count for unprivleged functions by 1 to account for PBLE
3497+ * backing page rounding
3498+ */
3499+ if (dev->hw_attrs.uk_attrs.hw_rev <= IRDMA_GEN_2 &&
3500+ (hmc_info->hmc_fn_id >= dev->hw_attrs.first_hw_vf_fpm_id ||
3501+ !dev->privileged))
3502+ max_pe_sds--;
29393503
29403504 hmc_fpm_misc->max_sds = max_pe_sds;
29413505 hmc_info->sd_table.sd_cnt = max_pe_sds + hmc_info->first_sd_index;
@@ -2949,11 +3513,17 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, · 7 unmodified lines
29493513 size = (u32)(temp >> 32);
29503514 obj_info[IRDMA_HMC_IW_CQ].size = BIT_ULL(size);
29513515
3516+ irdma_sc_decode_fpm_query(buf, 24, obj_info, IRDMA_HMC_IW_SRQ);
29523517 irdma_sc_decode_fpm_query(buf, 32, obj_info, IRDMA_HMC_IW_HTE);
29533518 irdma_sc_decode_fpm_query(buf, 40, obj_info, IRDMA_HMC_IW_ARP);
29543519
2955− obj_info[IRDMA_HMC_IW_APBVT_ENTRY].size = 8192;
2956− obj_info[IRDMA_HMC_IW_APBVT_ENTRY].max_cnt = 1;
3520+ if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) {
3521+ obj_info[IRDMA_HMC_IW_APBVT_ENTRY].size = 0;
3522+ obj_info[IRDMA_HMC_IW_APBVT_ENTRY].max_cnt = 0;
3523+ } else {
3524+ obj_info[IRDMA_HMC_IW_APBVT_ENTRY].size = 8192;
3525+ obj_info[IRDMA_HMC_IW_APBVT_ENTRY].max_cnt = 1;
3526+ }
29573527
29583528 irdma_sc_decode_fpm_query(buf, 48, obj_info, IRDMA_HMC_IW_MR);
29593529 irdma_sc_decode_fpm_query(buf, 56, obj_info, IRDMA_HMC_IW_XF);
@@ -2962,7 +3532,7 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, · 2 unmodified lines
29623532 obj_info[IRDMA_HMC_IW_XFFL].max_cnt = (u32)temp;
29633533 obj_info[IRDMA_HMC_IW_XFFL].size = 4;
29643534 hmc_fpm_misc->xf_block_size = FIELD_GET(IRDMA_QUERY_FPM_XFBLOCKSIZE, temp);
2965− if (!hmc_fpm_misc->xf_block_size)
3535+ if (obj_info[IRDMA_HMC_IW_XF].max_cnt && !hmc_fpm_misc->xf_block_size)
29663536 return -EINVAL;
29673537
29683538 irdma_sc_decode_fpm_query(buf, 72, obj_info, IRDMA_HMC_IW_Q1);
@@ -2984,6 +3554,14 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, · 15 unmodified lines
29843554 hmc_fpm_misc->max_ceqs = FIELD_GET(IRDMA_QUERY_FPM_MAX_CEQS, temp);
29853555 hmc_fpm_misc->ht_multiplier = FIELD_GET(IRDMA_QUERY_FPM_HTMULTIPLIER, temp);
29863556 hmc_fpm_misc->timer_bucket = FIELD_GET(IRDMA_QUERY_FPM_TIMERBUCKET, temp);
3557+ if (FIELD_GET(IRDMA_MANAGE_RSRC_VER2,
3558+ dev->feature_info[IRDMA_FTN_FLAGS])) {
3559+ ird_encoding = (u8)FIELD_GET(IRDMA_QUERY_FPM_MAX_IRD, temp);
3560+ hmc_fpm_misc->ird =
3561+ irdma_sc_get_decoded_ird_size_gen_3(ird_encoding) / 2;
3562+ dev->hw_attrs.max_hw_ird = hmc_fpm_misc->ird;
3563+ dev->hw_attrs.max_hw_ord = hmc_fpm_misc->ird;
3564+ }
29873565 if (dev->hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_1)
29883566 return 0;
29893567 irdma_sc_decode_fpm_query(buf, 96, obj_info, IRDMA_HMC_IW_FSIMC);
@@ -3000,15 +3578,25 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, · 10 unmodified lines
30003578
30013579 irdma_sc_decode_fpm_query(buf, 144, obj_info, IRDMA_HMC_IW_HDR);
30023580 irdma_sc_decode_fpm_query(buf, 152, obj_info, IRDMA_HMC_IW_MD);
3003− irdma_sc_decode_fpm_query(buf, 160, obj_info, IRDMA_HMC_IW_OOISC);
3004−
3005− get_64bit_val(buf, 168, &temp);
3006− obj_info[IRDMA_HMC_IW_OOISCFFL].max_cnt = (u32)temp;
3007− obj_info[IRDMA_HMC_IW_OOISCFFL].size = 4;
3008− hmc_fpm_misc->ooiscf_block_size = FIELD_GET(IRDMA_QUERY_FPM_OOISCFBLOCKSIZE, temp);
3009− if (!hmc_fpm_misc->ooiscf_block_size &&
3010− obj_info[IRDMA_HMC_IW_OOISCFFL].max_cnt)
3011− return -EINVAL;
3581+
3582+ if (dev->cqp->protocol_used == IRDMA_IWARP_PROTOCOL_ONLY) {
3583+ irdma_sc_decode_fpm_query(buf, 160, obj_info, IRDMA_HMC_IW_OOISC);
3584+
3585+ get_64bit_val(buf, 168, &temp);
3586+ obj_info[IRDMA_HMC_IW_OOISCFFL].max_cnt = (u32)temp;
3587+ obj_info[IRDMA_HMC_IW_OOISCFFL].size = 4;
3588+ hmc_fpm_misc->ooiscf_block_size = FIELD_GET(IRDMA_QUERY_FPM_OOISCFBLOCKSIZE, temp);
3589+ if (!hmc_fpm_misc->ooiscf_block_size &&
3590+ obj_info[IRDMA_HMC_IW_OOISCFFL].max_cnt)
3591+ return -EINVAL;
3592+ }
3593+
3594+ if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) {
3595+ get_64bit_val(buf, 176, &temp);
3596+ hmc_fpm_misc->loc_mem_pages = (u32)FIELD_GET(IRDMA_QUERY_FPM_LOC_MEM_PAGES, temp);
3597+ if (!hmc_fpm_misc->loc_mem_pages)
3598+ return -EINVAL;
3599+ }
30123600
30133601 return 0;
30143602 }
@@ -3088,6 +3676,8 @@ exit: · 73 unmodified lines
30883676 int irdma_sc_cqp_init(struct irdma_sc_cqp *cqp,
30893677 struct irdma_cqp_init_info *info)
30903678 {
3679+ struct irdma_ooo_cqp_op *ooo_op;
3680+ u32 num_ooo_ops;
30913681 u8 hw_sq_size;
30923682
30933683 if (info->sq_size > IRDMA_CQP_SW_SQSIZE_2048 ||
@@ -3118,17 +3708,43 @@ int irdma_sc_cqp_init(struct irdma_sc_cqp *cqp, · 24 unmodified lines
31183708 cqp->rocev2_rto_policy = info->rocev2_rto_policy;
31193709 cqp->protocol_used = info->protocol_used;
31203710 memcpy(&cqp->dcqcn_params, &info->dcqcn_params, sizeof(cqp->dcqcn_params));
3711+ if (cqp->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) {
3712+ cqp->ooisc_blksize = info->ooisc_blksize;
3713+ cqp->rrsp_blksize = info->rrsp_blksize;
3714+ cqp->q1_blksize = info->q1_blksize;
3715+ cqp->xmit_blksize = info->xmit_blksize;
3716+ cqp->blksizes_valid = info->blksizes_valid;
3717+ cqp->ts_shift = info->ts_shift;
3718+ cqp->ts_override = info->ts_override;
3719+ cqp->en_fine_grained_timers = info->en_fine_grained_timers;
3720+ cqp->pe_en_vf_cnt = info->pe_en_vf_cnt;
3721+ cqp->ooo_op_array = info->ooo_op_array;
3722+ /* initialize the OOO lists */
3723+ INIT_LIST_HEAD(&cqp->ooo_avail);
3724+ INIT_LIST_HEAD(&cqp->ooo_pnd);
3725+ if (cqp->ooo_op_array) {
3726+ /* Populate avail list entries */
3727+ for (num_ooo_ops = 0, ooo_op = info->ooo_op_array;
3728+ num_ooo_ops < cqp->sq_size;
3729+ num_ooo_ops++, ooo_op++)
3730+ list_add(&ooo_op->list_entry, &cqp->ooo_avail);
3731+ }
3732+ }
31213733 info->dev->cqp = cqp;
31223734
31233735 IRDMA_RING_INIT(cqp->sq_ring, cqp->sq_size);
3736+ cqp->last_def_cmpl_ticket = 0;
3737+ cqp->sw_def_cmpl_ticket = 0;
31243738 cqp->requested_ops = 0;
31253739 atomic64_set(&cqp->completed_ops, 0);
31263740 /* for the cqp commands backlog. */
31273741 INIT_LIST_HEAD(&cqp->dev->cqp_cmd_head);
31283742
31293743 writel(0, cqp->dev->hw_regs[IRDMA_CQPTAIL]);
3130− writel(0, cqp->dev->hw_regs[IRDMA_CQPDB]);
3131− writel(0, cqp->dev->hw_regs[IRDMA_CCQPSTATUS]);
3744+ if (cqp->dev->hw_attrs.uk_attrs.hw_rev <= IRDMA_GEN_2) {
3745+ writel(0, cqp->dev->hw_regs[IRDMA_CQPDB]);
3746+ writel(0, cqp->dev->hw_regs[IRDMA_CCQPSTATUS]);
3747+ }
31323748
31333749 ibdev_dbg(to_ibdev(cqp->dev),
31343750 "WQE: sq_size[%04d] hw_sq_size[%04d] sq_base[%p] sq_pa[%p] cqp[%p] polarity[x%04x]\n",
@@ -3160,6 +3776,7 @@ int irdma_sc_cqp_create(struct irdma_sc_cqp *cqp, u16 *maj_err, u16 *min_err) · 25 unmodified lines
31603776 return -ENOMEM;
31613777
31623778 spin_lock_init(&cqp->dev->cqp_lock);
3779+ spin_lock_init(&cqp->ooo_list_lock);
31633780
31643781 temp = FIELD_PREP(IRDMA_CQPHC_SQSIZE, cqp->hw_sq_size) |
31653782 FIELD_PREP(IRDMA_CQPHC_SVER, cqp->struct_ver) |
@@ -3171,12 +3788,29 @@ int irdma_sc_cqp_create(struct irdma_sc_cqp *cqp, u16 *maj_err, u16 *min_err) · 5 unmodified lines
31713788 FIELD_PREP(IRDMA_CQPHC_PROTOCOL_USED,
31723789 cqp->protocol_used);
31733790 }
3791+ if (hw_rev >= IRDMA_GEN_3)
3792+ temp |= FIELD_PREP(IRDMA_CQPHC_EN_FINE_GRAINED_TIMERS,
3793+ cqp->en_fine_grained_timers);
31743794
31753795 set_64bit_val(cqp->host_ctx, 0, temp);
31763796 set_64bit_val(cqp->host_ctx, 8, cqp->sq_pa);
31773797
31783798 temp = FIELD_PREP(IRDMA_CQPHC_ENABLED_VFS, cqp->ena_vf_count) |
31793799 FIELD_PREP(IRDMA_CQPHC_HMC_PROFILE, cqp->hmc_profile);
3800+
3801+ if (hw_rev >= IRDMA_GEN_3)
3802+ temp |= FIELD_PREP(IRDMA_CQPHC_OOISC_BLKSIZE,
3803+ cqp->ooisc_blksize) |
3804+ FIELD_PREP(IRDMA_CQPHC_RRSP_BLKSIZE,
3805+ cqp->rrsp_blksize) |
3806+ FIELD_PREP(IRDMA_CQPHC_Q1_BLKSIZE, cqp->q1_blksize) |
3807+ FIELD_PREP(IRDMA_CQPHC_XMIT_BLKSIZE,
3808+ cqp->xmit_blksize) |
3809+ FIELD_PREP(IRDMA_CQPHC_BLKSIZES_VALID,
3810+ cqp->blksizes_valid) |
3811+ FIELD_PREP(IRDMA_CQPHC_TIMESTAMP_OVERRIDE,
3812+ cqp->ts_override) |
3813+ FIELD_PREP(IRDMA_CQPHC_TS_SHIFT, cqp->ts_shift);
31803814 set_64bit_val(cqp->host_ctx, 16, temp);
31813815 set_64bit_val(cqp->host_ctx, 24, (uintptr_t)cqp);
31823816 temp = FIELD_PREP(IRDMA_CQPHC_HW_MAJVER, cqp->hw_maj_ver) |
@@ -3337,6 +3971,87 @@ void irdma_sc_ccq_arm(struct irdma_sc_cq *ccq) · 154 unmodified lines
33373971 writel(ccq->cq_uk.cq_id, ccq->dev->cq_arm_db);
33383972 }
33393973
3974+/**
3975+ * irdma_sc_process_def_cmpl - process deferred or pending completion
3976+ * @cqp: CQP sc struct
3977+ * @info: CQP CQE info
3978+ * @wqe_idx: CQP WQE descriptor index
3979+ * @def_info: deferred op ticket value or out-of-order completion id
3980+ * @def_cmpl: true for deferred completion, false for pending (RCA)
3981+ */
3982+static void irdma_sc_process_def_cmpl(struct irdma_sc_cqp *cqp,
3983+ struct irdma_ccq_cqe_info *info,
3984+ u32 wqe_idx, u32 def_info, bool def_cmpl)
3985+{
3986+ struct irdma_ooo_cqp_op *ooo_op;
3987+ unsigned long flags;
3988+
3989+ /* Deferred and out-of-order completions share the same list of pending
3990+ * completions. Since the list can be also accessed from AE handler,
3991+ * it must be protected by a lock.
3992+ */
3993+ spin_lock_irqsave(&cqp->ooo_list_lock, flags);
3994+
3995+ /* For deferred completions bump up SW completion ticket value. */
3996+ if (def_cmpl) {
3997+ cqp->last_def_cmpl_ticket = def_info;
3998+ cqp->sw_def_cmpl_ticket++;
3999+ }
4000+ if (!list_empty(&cqp->ooo_avail)) {
4001+ ooo_op = (struct irdma_ooo_cqp_op *)
4002+ list_entry(cqp->ooo_avail.next,
4003+ struct irdma_ooo_cqp_op, list_entry);
4004+
4005+ list_del(&ooo_op->list_entry);
4006+ ooo_op->scratch = info->scratch;
4007+ ooo_op->def_info = def_info;
4008+ ooo_op->sw_def_info = cqp->sw_def_cmpl_ticket;
4009+ ooo_op->deferred = def_cmpl;
4010+ ooo_op->wqe_idx = wqe_idx;
4011+ /* Pending completions must be chronologically ordered,
4012+ * so adding at the end of list.
4013+ */
4014+ list_add_tail(&ooo_op->list_entry, &cqp->ooo_pnd);
4015+ }
4016+ spin_unlock_irqrestore(&cqp->ooo_list_lock, flags);
4017+
4018+ info->pending = true;
4019+}
4020+
4021+/**
4022+ * irdma_sc_process_ooo_cmpl - process out-of-order (final) completion
4023+ * @cqp: CQP sc struct
4024+ * @info: CQP CQE info
4025+ * @def_info: out-of-order completion id
4026+ */
4027+static void irdma_sc_process_ooo_cmpl(struct irdma_sc_cqp *cqp,
4028+ struct irdma_ccq_cqe_info *info,
4029+ u32 def_info)
4030+{
4031+ struct irdma_ooo_cqp_op *ooo_op_tmp;
4032+ struct irdma_ooo_cqp_op *ooo_op;
4033+ unsigned long flags;
4034+
4035+ info->scratch = 0;
4036+
4037+ spin_lock_irqsave(&cqp->ooo_list_lock, flags);
4038+ list_for_each_entry_safe(ooo_op, ooo_op_tmp, &cqp->ooo_pnd,
4039+ list_entry) {
4040+ if (!ooo_op->deferred && ooo_op->def_info == def_info) {
4041+ list_del(&ooo_op->list_entry);
4042+ info->scratch = ooo_op->scratch;
4043+ list_add(&ooo_op->list_entry, &cqp->ooo_avail);
4044+ break;
4045+ }
4046+ }
4047+ spin_unlock_irqrestore(&cqp->ooo_list_lock, flags);
4048+
4049+ if (!info->scratch)
4050+ ibdev_dbg(to_ibdev(cqp->dev),
4051+ "CQP: DEBUG_FW_OOO out-of-order completion with unknown def_info = 0x%x\n",
4052+ def_info);
4053+}
4054+
33404055 /**
33414056 * irdma_sc_ccq_get_cqe_info - get ccq's cq entry
33424057 * @ccq: ccq sc struct
@@ -3345,6 +4060,10 @@ void irdma_sc_ccq_arm(struct irdma_sc_cq *ccq) · 2 unmodified lines
33454060 int irdma_sc_ccq_get_cqe_info(struct irdma_sc_cq *ccq,
33464061 struct irdma_ccq_cqe_info *info)
33474062 {
4063+ u32 def_info;
4064+ bool def_cmpl = false;
4065+ bool pend_cmpl = false;
4066+ bool ooo_final_cmpl = false;
33484067 u64 qp_ctx, temp, temp1;
33494068 __le64 *cqe;
33504069 struct irdma_sc_cqp *cqp;
@@ -3352,6 +4071,7 @@ int irdma_sc_ccq_get_cqe_info(struct irdma_sc_cq *ccq, · 1 unmodified line
33524071 u32 error;
33534072 u8 polarity;
33544073 int ret_code = 0;
4074+ unsigned long flags;
33554075
33564076 if (ccq->cq_uk.avoid_mem_cflct)
33574077 cqe = IRDMA_GET_CURRENT_EXTENDED_CQ_ELEM(&ccq->cq_uk);
@@ -3383,6 +4103,25 @@ int irdma_sc_ccq_get_cqe_info(struct irdma_sc_cq *ccq, · 25 unmodified lines
33834103
33844104 get_64bit_val(cqe, 16, &temp1);
33854105 info->op_ret_val = (u32)FIELD_GET(IRDMA_CCQ_OPRETVAL, temp1);
4106+ if (cqp->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) {
4107+ def_cmpl = info->maj_err_code == IRDMA_CQPSQ_MAJ_NO_ERROR &&
4108+ info->min_err_code == IRDMA_CQPSQ_MIN_DEF_CMPL;
4109+ def_info = (u32)FIELD_GET(IRDMA_CCQ_DEFINFO, temp1);
4110+
4111+ pend_cmpl = info->maj_err_code == IRDMA_CQPSQ_MAJ_NO_ERROR &&
4112+ info->min_err_code == IRDMA_CQPSQ_MIN_OOO_CMPL;
4113+
4114+ ooo_final_cmpl = (bool)FIELD_GET(IRDMA_OOO_CMPL, temp);
4115+
4116+ if (def_cmpl || pend_cmpl || ooo_final_cmpl) {
4117+ if (ooo_final_cmpl)
4118+ irdma_sc_process_ooo_cmpl(cqp, info, def_info);
4119+ else
4120+ irdma_sc_process_def_cmpl(cqp, info, wqe_idx,
4121+ def_info, def_cmpl);
4122+ }
4123+ }
4124+
33864125 get_64bit_val(cqp->sq_base[wqe_idx].elem, 24, &temp1);
33874126 info->op_code = (u8)FIELD_GET(IRDMA_CQPSQ_OPCODE, temp1);
33884127 info->cqp = cqp;
@@ -3399,7 +4138,16 @@ int irdma_sc_ccq_get_cqe_info(struct irdma_sc_cq *ccq, · 10 unmodified lines
33994138
34004139 dma_wmb(); /* make sure shadow area is updated before moving tail */
34014140
3402− IRDMA_RING_MOVE_TAIL(cqp->sq_ring);
4141+ spin_lock_irqsave(&cqp->dev->cqp_lock, flags);
4142+ if (!ooo_final_cmpl)
4143+ IRDMA_RING_MOVE_TAIL(cqp->sq_ring);
4144+ spin_unlock_irqrestore(&cqp->dev->cqp_lock, flags);
4145+
4146+ /* Do not increment completed_ops counter on pending or deferred
4147+ * completions.
4148+ */
4149+ if (pend_cmpl || def_cmpl)
4150+ return ret_code;
34034151 atomic64_inc(&cqp->completed_ops);
34044152
34054153 return ret_code;
@@ -3647,7 +4395,7 @@ int irdma_sc_ceq_init(struct irdma_sc_ceq *ceq, · 241 unmodified lines
36474395 ceq->pbl_list = (ceq->virtual_map ? info->pbl_list : NULL);
36484396 ceq->tph_en = info->tph_en;
36494397 ceq->tph_val = info->tph_val;
3650− ceq->vsi = info->vsi;
4398+ ceq->vsi_idx = info->vsi_idx;
36514399 ceq->polarity = 1;
36524400 IRDMA_RING_INIT(ceq->ceq_ring, ceq->elem_cnt);
36534401 ceq->dev->ceq[info->ceq_id] = ceq;
@@ -3680,13 +4428,16 @@ static int irdma_sc_ceq_create(struct irdma_sc_ceq *ceq, u64 scratch, · 26 unmodified lines
36804428 (ceq->virtual_map ? ceq->first_pm_pbl_idx : 0));
36814429 set_64bit_val(wqe, 56,
36824430 FIELD_PREP(IRDMA_CQPSQ_TPHVAL, ceq->tph_val) |
3683− FIELD_PREP(IRDMA_CQPSQ_VSIIDX, ceq->vsi->vsi_idx));
4431+ FIELD_PREP(IRDMA_CQPSQ_PASID, ceq->pasid) |
4432+ FIELD_PREP(IRDMA_CQPSQ_VSIIDX, ceq->vsi_idx));
36844433 hdr = FIELD_PREP(IRDMA_CQPSQ_CEQ_CEQID, ceq->ceq_id) |
4434+ FIELD_PREP(IRDMA_CQPSQ_CEQ_CEQID_HIGH, ceq->ceq_id >> 10) |
36854435 FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_CREATE_CEQ) |
36864436 FIELD_PREP(IRDMA_CQPSQ_CEQ_LPBLSIZE, ceq->pbl_chunk_size) |
36874437 FIELD_PREP(IRDMA_CQPSQ_CEQ_VMAP, ceq->virtual_map) |
36884438 FIELD_PREP(IRDMA_CQPSQ_CEQ_ITRNOEXPIRE, ceq->itr_no_expire) |
36894439 FIELD_PREP(IRDMA_CQPSQ_TPHEN, ceq->tph_en) |
4440+ FIELD_PREP(IRDMA_CQPSQ_PASID_VALID, ceq->pasid_valid) |
36904441 FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity);
36914442 dma_wmb(); /* make sure WQE is written before valid bit is set */
36924443
@@ -3741,7 +4492,7 @@ int irdma_sc_cceq_create(struct irdma_sc_ceq *ceq, u64 scratch) · 48 unmodified lines
37414492 int ret_code;
37424493 struct irdma_sc_dev *dev = ceq->dev;
37434494
3744− dev->ccq->vsi = ceq->vsi;
4495+ dev->ccq->vsi_idx = ceq->vsi_idx;
37454496 if (ceq->reg_cq) {
37464497 ret_code = irdma_sc_add_cq_ctx(ceq, ceq->dev->ccq);
37474498 if (ret_code)
@@ -3774,11 +4525,14 @@ int irdma_sc_ceq_destroy(struct irdma_sc_ceq *ceq, u64 scratch, bool post_sq) · 26 unmodified lines
37744525
37754526 set_64bit_val(wqe, 16, ceq->elem_cnt);
37764527 set_64bit_val(wqe, 48, ceq->first_pm_pbl_idx);
4528+ set_64bit_val(wqe, 56,
4529+ FIELD_PREP(IRDMA_CQPSQ_PASID, ceq->pasid));
37774530 hdr = ceq->ceq_id |
37784531 FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_DESTROY_CEQ) |
37794532 FIELD_PREP(IRDMA_CQPSQ_CEQ_LPBLSIZE, ceq->pbl_chunk_size) |
37804533 FIELD_PREP(IRDMA_CQPSQ_CEQ_VMAP, ceq->virtual_map) |
37814534 FIELD_PREP(IRDMA_CQPSQ_TPHEN, ceq->tph_en) |
4535+ FIELD_PREP(IRDMA_CQPSQ_PASID_VALID, ceq->pasid_valid) |
37824536 FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity);
37834537 dma_wmb(); /* make sure WQE is written before valid bit is set */
37844538
@@ -3942,10 +4696,13 @@ static int irdma_sc_aeq_create(struct irdma_sc_aeq *aeq, u64 scratch, · 157 unmodified lines
39424696 (aeq->virtual_map ? 0 : aeq->aeq_elem_pa));
39434697 set_64bit_val(wqe, 48,
39444698 (aeq->virtual_map ? aeq->first_pm_pbl_idx : 0));
4699+ set_64bit_val(wqe, 56,
4700+ FIELD_PREP(IRDMA_CQPSQ_PASID, aeq->pasid));
39454701
39464702 hdr = FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_CREATE_AEQ) |
39474703 FIELD_PREP(IRDMA_CQPSQ_AEQ_LPBLSIZE, aeq->pbl_chunk_size) |
39484704 FIELD_PREP(IRDMA_CQPSQ_AEQ_VMAP, aeq->virtual_map) |
4705+ FIELD_PREP(IRDMA_CQPSQ_PASID_VALID, aeq->pasid_valid) |
39494706 FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity);
39504707 dma_wmb(); /* make sure WQE is written before valid bit is set */
39514708
@@ -3974,7 +4731,8 @@ static int irdma_sc_aeq_destroy(struct irdma_sc_aeq *aeq, u64 scratch, · 22 unmodified lines
39744731 u64 hdr;
39754732
39764733 dev = aeq->dev;
3977− writel(0, dev->hw_regs[IRDMA_PFINT_AEQCTL]);
4734+ if (dev->privileged)
4735+ writel(0, dev->hw_regs[IRDMA_PFINT_AEQCTL]);
39784736
39794737 cqp = dev->cqp;
39804738 wqe = irdma_sc_cqp_get_next_send_wqe(cqp, scratch);
@@ -3982,9 +4740,12 @@ static int irdma_sc_aeq_destroy(struct irdma_sc_aeq *aeq, u64 scratch, · 1 unmodified line
39824740 return -ENOMEM;
39834741 set_64bit_val(wqe, 16, aeq->elem_cnt);
39844742 set_64bit_val(wqe, 48, aeq->first_pm_pbl_idx);
4743+ set_64bit_val(wqe, 56,
4744+ FIELD_PREP(IRDMA_CQPSQ_PASID, aeq->pasid));
39854745 hdr = FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_DESTROY_AEQ) |
39864746 FIELD_PREP(IRDMA_CQPSQ_AEQ_LPBLSIZE, aeq->pbl_chunk_size) |
39874747 FIELD_PREP(IRDMA_CQPSQ_AEQ_VMAP, aeq->virtual_map) |
4748+ FIELD_PREP(IRDMA_CQPSQ_PASID_VALID, aeq->pasid_valid) |
39884749 FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity);
39894750 dma_wmb(); /* make sure WQE is written before valid bit is set */
39904751
@@ -4025,18 +4786,39 @@ int irdma_sc_get_next_aeqe(struct irdma_sc_aeq *aeq, · 34 unmodified lines
40254786 print_hex_dump_debug("WQE: AEQ_ENTRY WQE", DUMP_PREFIX_OFFSET, 16, 8,
40264787 aeqe, 16, false);
40274788
4028− ae_src = (u8)FIELD_GET(IRDMA_AEQE_AESRC, temp);
4029− info->wqe_idx = (u16)FIELD_GET(IRDMA_AEQE_WQDESCIDX, temp);
4030− info->qp_cq_id = (u32)FIELD_GET(IRDMA_AEQE_QPCQID_LOW, temp) |
4789+ if (aeq->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) {
4790+ ae_src = (u8)FIELD_GET(IRDMA_AEQE_AESRC_GEN_3, temp);
4791+ info->wqe_idx = (u16)FIELD_GET(IRDMA_AEQE_WQDESCIDX_GEN_3,
4792+ temp);
4793+ info->qp_cq_id = (u32)FIELD_GET(IRDMA_AEQE_QPCQID_GEN_3, temp);
4794+ info->ae_id = (u16)FIELD_GET(IRDMA_AEQE_AECODE_GEN_3, temp);
4795+ info->tcp_state = (u8)FIELD_GET(IRDMA_AEQE_TCPSTATE_GEN_3, compl_ctx);
4796+ info->iwarp_state = (u8)FIELD_GET(IRDMA_AEQE_IWSTATE_GEN_3, temp);
4797+ info->q2_data_written = (u8)FIELD_GET(IRDMA_AEQE_Q2DATA_GEN_3, compl_ctx);
4798+ info->aeqe_overflow = (bool)FIELD_GET(IRDMA_AEQE_OVERFLOW_GEN_3, temp);
4799+ info->compl_ctx = FIELD_GET(IRDMA_AEQE_CMPL_CTXT, compl_ctx);
4800+ compl_ctx = FIELD_GET(IRDMA_AEQE_CMPL_CTXT, compl_ctx) << IRDMA_AEQE_CMPL_CTXT_S;
4801+ } else {
4802+ ae_src = (u8)FIELD_GET(IRDMA_AEQE_AESRC, temp);
4803+ info->wqe_idx = (u16)FIELD_GET(IRDMA_AEQE_WQDESCIDX, temp);
4804+ info->qp_cq_id = (u32)FIELD_GET(IRDMA_AEQE_QPCQID_LOW, temp) |
40314805 ((u32)FIELD_GET(IRDMA_AEQE_QPCQID_HI, temp) << 18);
4032− info->ae_id = (u16)FIELD_GET(IRDMA_AEQE_AECODE, temp);
4033− info->tcp_state = (u8)FIELD_GET(IRDMA_AEQE_TCPSTATE, temp);
4034− info->iwarp_state = (u8)FIELD_GET(IRDMA_AEQE_IWSTATE, temp);
4035− info->q2_data_written = (u8)FIELD_GET(IRDMA_AEQE_Q2DATA, temp);
4036− info->aeqe_overflow = (bool)FIELD_GET(IRDMA_AEQE_OVERFLOW, temp);
4806+ info->ae_id = (u16)FIELD_GET(IRDMA_AEQE_AECODE, temp);
4807+ info->tcp_state = (u8)FIELD_GET(IRDMA_AEQE_TCPSTATE, temp);
4808+ info->iwarp_state = (u8)FIELD_GET(IRDMA_AEQE_IWSTATE, temp);
4809+ info->q2_data_written = (u8)FIELD_GET(IRDMA_AEQE_Q2DATA, temp);
4810+ info->aeqe_overflow = (bool)FIELD_GET(IRDMA_AEQE_OVERFLOW,
4811+ temp);
4812+ }
40374813
40384814 info->ae_src = ae_src;
40394815 switch (info->ae_id) {
4816+ case IRDMA_AE_SRQ_LIMIT:
4817+ info->srq = true;
4818+ /* [63:6] from CMPL_CTXT, [5:0] from WQDESCIDX. */
4819+ info->compl_ctx = compl_ctx;
4820+ ae_src = IRDMA_AE_SOURCE_RSVD;
4821+ break;
40404822 case IRDMA_AE_PRIV_OPERATION_DENIED:
40414823 case IRDMA_AE_AMP_INVALIDATE_TYPE1_MW:
40424824 case IRDMA_AE_AMP_MWBIND_ZERO_BASED_TYPE1_MW:
@@ -4069,6 +4851,10 @@ int irdma_sc_get_next_aeqe(struct irdma_sc_aeq *aeq, · 26 unmodified lines
40694851 case IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR:
40704852 case IRDMA_AE_LLP_SEGMENT_TOO_SMALL:
40714853 case IRDMA_AE_LLP_TOO_MANY_RETRIES:
4854+ case IRDMA_AE_LLP_TOO_MANY_RNRS:
4855+ case IRDMA_AE_REMOTE_QP_CATASTROPHIC:
4856+ case IRDMA_AE_LOCAL_QP_CATASTROPHIC:
4857+ case IRDMA_AE_RCE_QP_CATASTROPHIC:
40724858 case IRDMA_AE_LLP_DOUBT_REACHABILITY:
40734859 case IRDMA_AE_LLP_CONNECTION_ESTABLISHED:
40744860 case IRDMA_AE_RESET_SENT:
@@ -4085,6 +4871,10 @@ int irdma_sc_get_next_aeqe(struct irdma_sc_aeq *aeq, · 10 unmodified lines
40854871 info->compl_ctx = compl_ctx << 1;
40864872 ae_src = IRDMA_AE_SOURCE_RSVD;
40874873 break;
4874+ case IRDMA_AE_CQP_DEFERRED_COMPLETE:
4875+ info->def_info = info->wqe_idx;
4876+ ae_src = IRDMA_AE_SOURCE_RSVD;
4877+ break;
40884878 case IRDMA_AE_ROCE_EMPTY_MCG:
40894879 case IRDMA_AE_ROCE_BAD_MC_IP_ADDR:
40904880 case IRDMA_AE_ROCE_BAD_MC_QPID:
@@ -4110,6 +4900,7 @@ int irdma_sc_get_next_aeqe(struct irdma_sc_aeq *aeq, · 19 unmodified lines
41104900 info->qp = true;
41114901 info->rq = true;
41124902 info->compl_ctx = compl_ctx;
4903+ info->err_rq_idx_valid = true;
41134904 break;
41144905 case IRDMA_AE_SOURCE_CQ:
41154906 case IRDMA_AE_SOURCE_CQ_0110:
@@ -4125,8 +4916,18 @@ int irdma_sc_get_next_aeqe(struct irdma_sc_aeq *aeq, · 9 unmodified lines
41254916 info->compl_ctx = compl_ctx;
41264917 break;
41274918 case IRDMA_AE_SOURCE_IN_RR_WR:
4919+ info->qp = true;
4920+ if (aeq->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3)
4921+ info->err_rq_idx_valid = true;
4922+ info->compl_ctx = compl_ctx;
4923+ info->in_rdrsp_wr = true;
4924+ break;
41284925 case IRDMA_AE_SOURCE_IN_RR_WR_1011:
41294926 info->qp = true;
4927+ if (aeq->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) {
4928+ info->sq = true;
4929+ info->err_rq_idx_valid = true;
4930+ }
41304931 info->compl_ctx = compl_ctx;
41314932 info->in_rdrsp_wr = true;
41324933 break;
@@ -4335,6 +5136,26 @@ int irdma_sc_init_iw_hmc(struct irdma_sc_dev *dev, u8 hmc_fn_id) · 202 unmodified lines
43355136 return ret_code;
43365137 }
43375138
5139+/**
5140+ * irdma_set_loc_mem() - set a local memory bit field
5141+ * @buf: ptr to a buffer where local memory gets enabled
5142+ */
5143+static void irdma_set_loc_mem(__le64 *buf)
5144+{
5145+ u64 loc_mem_en = BIT_ULL(ENABLE_LOC_MEM);
5146+ u32 offset;
5147+ u64 temp;
5148+
5149+ for (offset = 0; offset < IRDMA_COMMIT_FPM_BUF_SIZE;
5150+ offset += sizeof(__le64)) {
5151+ if (offset == IRDMA_PBLE_COMMIT_OFFSET)
5152+ continue;
5153+ get_64bit_val(buf, offset, &temp);
5154+ if (temp)
5155+ set_64bit_val(buf, offset, temp | loc_mem_en);
5156+ }
5157+}
5158+
43385159 /**
43395160 * irdma_sc_cfg_iw_fpm() - commits hmc obj cnt values using cqp
43405161 * command and populates fpm base address in hmc_info
@@ -4356,7 +5177,7 @@ static int irdma_sc_cfg_iw_fpm(struct irdma_sc_dev *dev, u8 hmc_fn_id) · 15 unmodified lines
43565177
43575178 set_64bit_val(buf, 0, (u64)obj_info[IRDMA_HMC_IW_QP].cnt);
43585179 set_64bit_val(buf, 8, (u64)obj_info[IRDMA_HMC_IW_CQ].cnt);
4359− set_64bit_val(buf, 16, (u64)0); /* RSRVD */
5180+ set_64bit_val(buf, 16, (u64)obj_info[IRDMA_HMC_IW_SRQ].cnt);
43605181 set_64bit_val(buf, 24, (u64)obj_info[IRDMA_HMC_IW_HTE].cnt);
43615182 set_64bit_val(buf, 32, (u64)obj_info[IRDMA_HMC_IW_ARP].cnt);
43625183 set_64bit_val(buf, 40, (u64)0); /* RSVD */
@@ -4383,7 +5204,9 @@ static int irdma_sc_cfg_iw_fpm(struct irdma_sc_dev *dev, u8 hmc_fn_id) · 20 unmodified lines
43835204 (u64)obj_info[IRDMA_HMC_IW_OOISC].cnt);
43845205 set_64bit_val(buf, 168,
43855206 (u64)obj_info[IRDMA_HMC_IW_OOISCFFL].cnt);
4386−
5207+ if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3 &&
5208+ dev->hmc_fpm_misc.loc_mem_pages)
5209+ irdma_set_loc_mem(buf);
43875210 commit_fpm_mem.pa = dev->fpm_commit_buf_pa;
43885211 commit_fpm_mem.va = dev->fpm_commit_buf;
43895212
@@ -4592,6 +5415,7 @@ static bool irdma_cqp_ring_full(struct irdma_sc_cqp *cqp) · 202 unmodified lines
45925415 static u32 irdma_est_sd(struct irdma_sc_dev *dev,
45935416 struct irdma_hmc_info *hmc_info)
45945417 {
5418+ struct irdma_hmc_obj_info *pble_info;
45955419 int i;
45965420 u64 size = 0;
45975421 u64 sd;
@@ -4600,12 +5424,22 @@ static u32 irdma_est_sd(struct irdma_sc_dev *dev, · 2 unmodified lines
46005424 if (i != IRDMA_HMC_IW_PBLE)
46015425 size += round_up(hmc_info->hmc_obj[i].cnt *
46025426 hmc_info->hmc_obj[i].size, 512);
4603− size += round_up(hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].cnt *
4604− hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].size, 512);
5427+
5428+ pble_info = &hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE];
5429+ if (dev->privileged)
5430+ size += round_up(pble_info->cnt * pble_info->size, 512);
46055431 if (size & 0x1FFFFF)
46065432 sd = (size >> 21) + 1; /* add 1 for remainder */
46075433 else
46085434 sd = size >> 21;
5435+ if (!dev->privileged && !dev->hmc_fpm_misc.loc_mem_pages) {
5436+ /* 2MB alignment for VF PBLE HMC */
5437+ size = pble_info->cnt * pble_info->size;
5438+ if (size & 0x1FFFFF)
5439+ sd += (size >> 21) + 1; /* add 1 for remainder */
5440+ else
5441+ sd += size >> 21;
5442+ }
46095443 if (sd > 0xFFFFFFFF) {
46105444 ibdev_dbg(to_ibdev(dev), "HMC: sd overflow[%lld]\n", sd);
46115445 sd = 0xFFFFFFFF - 1;
@@ -4614,17 +5448,6 @@ static u32 irdma_est_sd(struct irdma_sc_dev *dev, · 2 unmodified lines
46145448 return (u32)sd;
46155449 }
46165450
4617−/**
4618− * irdma_sc_query_rdma_features_done - poll cqp for query features done
4619− * @cqp: struct for cqp hw
4620− */
4621−static int irdma_sc_query_rdma_features_done(struct irdma_sc_cqp *cqp)
4622−{
4623− return irdma_sc_poll_for_cqp_op_done(cqp,
4624− IRDMA_CQP_OP_QUERY_RDMA_FEATURES,
4625− NULL);
4626−}
4627−
46285451 /**
46295452 * irdma_sc_query_rdma_features - query RDMA features and FW ver
46305453 * @cqp: struct for cqp hw
@@ -4634,7 +5457,9 @@ static int irdma_sc_query_rdma_features_done(struct irdma_sc_cqp *cqp) · 3 unmodified lines
46345457 static int irdma_sc_query_rdma_features(struct irdma_sc_cqp *cqp,
46355458 struct irdma_dma_mem *buf, u64 scratch)
46365459 {
5460+ u32 tail, val, error;
46375461 __le64 *wqe;
5462+ int status;
46385463 u64 temp;
46395464
46405465 wqe = irdma_sc_cqp_get_next_send_wqe(cqp, scratch);
@@ -4654,9 +5479,15 @@ static int irdma_sc_query_rdma_features(struct irdma_sc_cqp *cqp, · 13 unmodified lines
46545479
46555480 print_hex_dump_debug("WQE: QUERY RDMA FEATURES", DUMP_PREFIX_OFFSET,
46565481 16, 8, wqe, IRDMA_CQP_WQE_SIZE * 8, false);
5482+ irdma_get_cqp_reg_info(cqp, &val, &tail, &error);
5483+
46575484 irdma_sc_cqp_post_sq(cqp);
5485+ status = irdma_cqp_poll_registers(cqp, tail,
5486+ cqp->dev->hw_attrs.max_done_count);
5487+ if (error || status)
5488+ status = -EINVAL;
46585489
4659− return 0;
5490+ return status;
46605491 }
46615492
46625493 /**
@@ -4678,8 +5509,6 @@ int irdma_get_rdma_features(struct irdma_sc_dev *dev) · 15 unmodified lines
46785509 return -ENOMEM;
46795510
46805511 ret_code = irdma_sc_query_rdma_features(dev->cqp, &feat_buf, 0);
4681− if (!ret_code)
4682− ret_code = irdma_sc_query_rdma_features_done(dev->cqp);
46835512 if (ret_code)
46845513 goto exit;
46855514
@@ -4703,8 +5532,6 @@ int irdma_get_rdma_features(struct irdma_sc_dev *dev) · 17 unmodified lines
47035532 return -ENOMEM;
47045533
47055534 ret_code = irdma_sc_query_rdma_features(dev->cqp, &feat_buf, 0);
4706− if (!ret_code)
4707− ret_code = irdma_sc_query_rdma_features_done(dev->cqp);
47085535 if (ret_code)
47095536 goto exit;
47105537
@@ -4731,6 +5558,10 @@ int irdma_get_rdma_features(struct irdma_sc_dev *dev) · 20 unmodified lines
47315558 }
47325559 dev->feature_info[feat_type] = temp;
47335560 }
5561+
5562+ if (dev->feature_info[IRDMA_FTN_FLAGS] & IRDMA_ATOMICS_ALLOWED_BIT)
5563+ dev->hw_attrs.uk_attrs.feature_flags |= IRDMA_FEATURE_ATOMIC_OPS;
5564+
47345565 exit:
47355566 dma_free_coherent(dev->hw->device, feat_buf.size, feat_buf.va,
47365567 feat_buf.pa);
@@ -4785,6 +5616,339 @@ static void cfg_fpm_value_gen_2(struct irdma_sc_dev *dev, · 48 unmodified lines
47855616 hmc_fpm_misc->ooiscf_block_size;
47865617 }
47875618
5619+/**
5620+ * irdma_get_rsrc_mem_config - configure resources if local memory or host
5621+ * @dev: sc device struct
5622+ * @is_mrte_loc_mem: if true, MR's to be in local memory because sd=loc pages
5623+ *
5624+ * Only mr can be configured host or local memory if qp's are in local memory.
5625+ * If qp is in local memory, then all resource object will be in local memory
5626+ * except mr which can be either host or local memory. The only exception
5627+ * is pble's which are always in host memory.
5628+ */
5629+static void irdma_get_rsrc_mem_config(struct irdma_sc_dev *dev, bool is_mrte_loc_mem)
5630+{
5631+ struct irdma_hmc_info *hmc_info = dev->hmc_info;
5632+ int i;
5633+
5634+ for (i = IRDMA_HMC_IW_QP; i < IRDMA_HMC_IW_MAX; i++)
5635+ hmc_info->hmc_obj[i].mem_loc = IRDMA_LOC_MEM;
5636+
5637+ if (dev->feature_info[IRDMA_OBJ_1] && !is_mrte_loc_mem) {
5638+ u8 mem_type;
5639+
5640+ mem_type = (u8)FIELD_GET(IRDMA_MR_MEM_LOC, dev->feature_info[IRDMA_OBJ_1]);
5641+
5642+ hmc_info->hmc_obj[IRDMA_HMC_IW_MR].mem_loc =
5643+ (mem_type & IRDMA_OBJ_LOC_MEM_BIT) ?
5644+ IRDMA_LOC_MEM : IRDMA_HOST_MEM;
5645+ } else {
5646+ hmc_info->hmc_obj[IRDMA_HMC_IW_MR].mem_loc = IRDMA_LOC_MEM;
5647+ }
5648+
5649+ hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].mem_loc = IRDMA_HOST_MEM;
5650+
5651+ ibdev_dbg(to_ibdev(dev), "HMC: INFO: mrte_mem_loc = %d pble = %d\n",
5652+ hmc_info->hmc_obj[IRDMA_HMC_IW_MR].mem_loc,
5653+ hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].mem_loc);
5654+}
5655+
5656+/**
5657+ * irdma_cfg_sd_mem - allocate sd memory
5658+ * @dev: sc device struct
5659+ * @hmc_info: ptr to irdma_hmc_obj_info struct
5660+ */
5661+static int irdma_cfg_sd_mem(struct irdma_sc_dev *dev,
5662+ struct irdma_hmc_info *hmc_info)
5663+{
5664+ struct irdma_virt_mem virt_mem;
5665+ u32 mem_size;
5666+
5667+ mem_size = sizeof(struct irdma_hmc_sd_entry) * hmc_info->sd_table.sd_cnt;
5668+ virt_mem.size = mem_size;
5669+ virt_mem.va = kzalloc(virt_mem.size, GFP_KERNEL);
5670+ if (!virt_mem.va)
5671+ return -ENOMEM;
5672+ hmc_info->sd_table.sd_entry = virt_mem.va;
5673+
5674+ return 0;
5675+}
5676+
5677+/**
5678+ * irdma_get_objs_pages - get number of 2M pages needed
5679+ * @dev: sc device struct
5680+ * @hmc_info: pointer to the HMC configuration information struct
5681+ * @mem_loc: pages for local or host memory
5682+ */
5683+static u32 irdma_get_objs_pages(struct irdma_sc_dev *dev,
5684+ struct irdma_hmc_info *hmc_info,
5685+ enum irdma_hmc_obj_mem mem_loc)
5686+{
5687+ u64 size = 0;
5688+ int i;
5689+
5690+ for (i = IRDMA_HMC_IW_QP; i < IRDMA_HMC_IW_MAX; i++) {
5691+ if (hmc_info->hmc_obj[i].mem_loc == mem_loc) {
5692+ size += round_up(hmc_info->hmc_obj[i].cnt *
5693+ hmc_info->hmc_obj[i].size, 512);
5694+ }
5695+ }
5696+
5697+ return DIV_ROUND_UP(size, IRDMA_HMC_PAGE_SIZE);
5698+}
5699+
5700+/**
5701+ * irdma_set_host_hmc_rsrc_gen_3 - calculate host hmc resources for gen 3
5702+ * @dev: sc device struct
5703+ */
5704+static void irdma_set_host_hmc_rsrc_gen_3(struct irdma_sc_dev *dev)
5705+{
5706+ struct irdma_hmc_fpm_misc *hmc_fpm_misc;
5707+ struct irdma_hmc_info *hmc_info;
5708+ enum irdma_hmc_obj_mem mrte_loc;
5709+ u32 mrwanted, pblewanted;
5710+ u32 avail_sds, mr_sds;
5711+
5712+ hmc_info = dev->hmc_info;
5713+ hmc_fpm_misc = &dev->hmc_fpm_misc;
5714+ avail_sds = hmc_fpm_misc->max_sds;
5715+ mrte_loc = hmc_info->hmc_obj[IRDMA_HMC_IW_MR].mem_loc;
5716+ mrwanted = hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt;
5717+ pblewanted = hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].max_cnt;
5718+
5719+ if (mrte_loc == IRDMA_HOST_MEM && avail_sds > IRDMA_MIN_PBLE_PAGES) {
5720+ mr_sds = avail_sds - IRDMA_MIN_PBLE_PAGES;
5721+ mrwanted = min(mrwanted, mr_sds * MAX_MR_PER_SD);
5722+ hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt = mrwanted;
5723+ avail_sds -= DIV_ROUND_UP(mrwanted, MAX_MR_PER_SD);
5724+ }
5725+
5726+ if (FIELD_GET(IRDMA_MANAGE_RSRC_VER2, dev->feature_info[IRDMA_FTN_FLAGS]) &&
5727+ pblewanted > avail_sds * MAX_PBLE_PER_SD)
5728+ ibdev_dbg(to_ibdev(dev),
5729+ "HMC: Warn: Resource version 2: pble wanted = 0x%x available = 0x%x\n",
5730+ pblewanted, avail_sds * MAX_PBLE_PER_SD);
5731+
5732+ pblewanted = min(pblewanted, avail_sds * MAX_PBLE_PER_SD);
5733+ hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].cnt = pblewanted;
5734+}
5735+
5736+/**
5737+ * irdma_verify_commit_fpm_gen_3 - verify query fpm values
5738+ * @dev: sc device struct
5739+ * @max_pages: max local memory available
5740+ * @qpwanted: number of qp's wanted
5741+ */
5742+static int irdma_verify_commit_fpm_gen_3(struct irdma_sc_dev *dev,
5743+ u32 max_pages,
5744+ u32 qpwanted)
5745+{
5746+ struct irdma_hmc_fpm_misc *hmc_fpm_misc;
5747+ u32 rrf_cnt, xf_cnt, timer_cnt, pages_needed;
5748+ struct irdma_hmc_info *hmc_info;
5749+ u32 rrffl_cnt = 0;
5750+ u32 xffl_cnt = 0;
5751+ u32 q1fl_cnt;
5752+
5753+ hmc_info = dev->hmc_info;
5754+ hmc_fpm_misc = &dev->hmc_fpm_misc;
5755+
5756+ rrf_cnt = roundup_pow_of_two(IRDMA_RRF_MULTIPLIER * qpwanted);
5757+
5758+ if (hmc_info->hmc_obj[IRDMA_HMC_IW_RRFFL].max_cnt)
5759+ rrffl_cnt =
5760+ hmc_info->hmc_obj[IRDMA_HMC_IW_RRF].cnt /
5761+ hmc_fpm_misc->rrf_block_size;
5762+
5763+ xf_cnt = roundup_pow_of_two(IRDMA_XF_MULTIPLIER * qpwanted);
5764+
5765+ if (xf_cnt)
5766+ xffl_cnt = xf_cnt / hmc_fpm_misc->xf_block_size;
5767+
5768+ timer_cnt = (round_up(qpwanted, 512) / 512 + 1) *
5769+ hmc_fpm_misc->timer_bucket;
5770+
5771+ q1fl_cnt = hmc_info->hmc_obj[IRDMA_HMC_IW_Q1].cnt / hmc_fpm_misc->q1_block_size;
5772+
5773+ pages_needed = irdma_get_objs_pages(dev, hmc_info, IRDMA_LOC_MEM);
5774+ if (pages_needed > max_pages) {
5775+ ibdev_dbg(to_ibdev(dev),
5776+ "HMC: FAIL: SW counts rrf_cnt = %u rrffl_cnt = %u timer_cnt = %u",
5777+ rrf_cnt, rrffl_cnt, timer_cnt);
5778+ ibdev_dbg(to_ibdev(dev),
5779+ "HMC: FAIL: SW counts xf_cnt = %u xffl_cnt = %u q1fl_cnt = %u",
5780+ xf_cnt, xffl_cnt, q1fl_cnt);
5781+
5782+ return -EINVAL;
5783+ }
5784+
5785+ hmc_fpm_misc->max_sds -= pages_needed;
5786+ hmc_fpm_misc->loc_mem_pages -= pages_needed;
5787+
5788+ return 0;
5789+}
5790+
5791+/**
5792+ * irdma_set_loc_hmc_rsrc_gen_3 - calculate hmc resources for gen 3
5793+ * @dev: sc device struct
5794+ * @max_pages: max local memory available
5795+ * @qpwanted: number of qp's wanted
5796+ */
5797+static int irdma_set_loc_hmc_rsrc_gen_3(struct irdma_sc_dev *dev,
5798+ u32 max_pages,
5799+ u32 qpwanted)
5800+{
5801+ struct irdma_hmc_fpm_misc *hmc_fpm_misc;
5802+ u32 rrf_cnt, xf_cnt, timer_cnt, pages_needed;
5803+ struct irdma_hmc_info *hmc_info;
5804+ u32 ird, ord;
5805+
5806+ if (FIELD_GET(IRDMA_MANAGE_RSRC_VER2, dev->feature_info[IRDMA_FTN_FLAGS]))
5807+ return irdma_verify_commit_fpm_gen_3(dev, max_pages, qpwanted);
5808+
5809+ hmc_info = dev->hmc_info;
5810+ hmc_fpm_misc = &dev->hmc_fpm_misc;
5811+ ird = dev->hw_attrs.max_hw_ird;
5812+ ord = dev->hw_attrs.max_hw_ord;
5813+
5814+ hmc_info->hmc_obj[IRDMA_HMC_IW_HDR].cnt = qpwanted;
5815+ hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt = qpwanted;
5816+
5817+ hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt =
5818+ min(hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt, qpwanted * 2);
5819+
5820+ hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].cnt =
5821+ min(qpwanted * 8, hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].max_cnt);
5822+
5823+ rrf_cnt = roundup_pow_of_two(IRDMA_RRF_MULTIPLIER * qpwanted);
5824+ hmc_info->hmc_obj[IRDMA_HMC_IW_RRF].cnt =
5825+ min(hmc_info->hmc_obj[IRDMA_HMC_IW_RRF].max_cnt, rrf_cnt);
5826+
5827+ if (hmc_info->hmc_obj[IRDMA_HMC_IW_RRFFL].max_cnt)
5828+ hmc_info->hmc_obj[IRDMA_HMC_IW_RRFFL].cnt =
5829+ hmc_info->hmc_obj[IRDMA_HMC_IW_RRF].cnt /
5830+ hmc_fpm_misc->rrf_block_size;
5831+
5832+ xf_cnt = roundup_pow_of_two(IRDMA_XF_MULTIPLIER * qpwanted);
5833+ hmc_info->hmc_obj[IRDMA_HMC_IW_XF].cnt =
5834+ min(hmc_info->hmc_obj[IRDMA_HMC_IW_XF].max_cnt, xf_cnt);
5835+ hmc_info->hmc_obj[IRDMA_HMC_IW_XFFL].cnt =
5836+ xf_cnt / hmc_fpm_misc->xf_block_size;
5837+
5838+ timer_cnt = (round_up(qpwanted, 512) / 512 + 1) *
5839+ hmc_fpm_misc->timer_bucket;
5840+ hmc_info->hmc_obj[IRDMA_HMC_IW_TIMER].cnt =
5841+ min(timer_cnt, hmc_info->hmc_obj[IRDMA_HMC_IW_TIMER].cnt);
5842+
5843+ do {
5844+ hmc_info->hmc_obj[IRDMA_HMC_IW_Q1].cnt = roundup_pow_of_two(ird * 2 * qpwanted);
5845+ hmc_info->hmc_obj[IRDMA_HMC_IW_Q1FL].cnt =
5846+ hmc_info->hmc_obj[IRDMA_HMC_IW_Q1].cnt / hmc_fpm_misc->q1_block_size;
5847+
5848+ pages_needed = irdma_get_objs_pages(dev, hmc_info, IRDMA_LOC_MEM);
5849+ if (pages_needed <= max_pages)
5850+ break;
5851+
5852+ ird /= 2;
5853+ ord /= 2;
5854+ } while (ird >= IRDMA_MIN_IRD);
5855+
5856+ if (ird < IRDMA_MIN_IRD) {
5857+ ibdev_dbg(to_ibdev(dev), "HMC: FAIL: IRD=%u Q1 CNT = %u\n",
5858+ ird, hmc_info->hmc_obj[IRDMA_HMC_IW_Q1].cnt);
5859+ return -EINVAL;
5860+ }
5861+
5862+ dev->hw_attrs.max_hw_ird = ird;
5863+ dev->hw_attrs.max_hw_ord = ord;
5864+ hmc_fpm_misc->max_sds -= pages_needed;
5865+
5866+ return 0;
5867+}
5868+
5869+/**
5870+ * cfg_fpm_value_gen_3 - configure fpm for gen 3
5871+ * @dev: sc device struct
5872+ * @hmc_info: ptr to irdma_hmc_obj_info struct
5873+ * @hmc_fpm_misc: ptr to fpm data
5874+ */
5875+static int cfg_fpm_value_gen_3(struct irdma_sc_dev *dev,
5876+ struct irdma_hmc_info *hmc_info,
5877+ struct irdma_hmc_fpm_misc *hmc_fpm_misc)
5878+{
5879+ enum irdma_hmc_obj_mem mrte_loc;
5880+ u32 mrwanted, qpwanted;
5881+ int i, ret_code = 0;
5882+ u32 loc_mem_pages;
5883+ bool is_mrte_loc_mem;
5884+
5885+ loc_mem_pages = hmc_fpm_misc->loc_mem_pages;
5886+ is_mrte_loc_mem = hmc_fpm_misc->loc_mem_pages == hmc_fpm_misc->max_sds ?
5887+ true : false;
5888+
5889+ irdma_get_rsrc_mem_config(dev, is_mrte_loc_mem);
5890+ mrte_loc = hmc_info->hmc_obj[IRDMA_HMC_IW_MR].mem_loc;
5891+
5892+ if (is_mrte_loc_mem)
5893+ loc_mem_pages -= IRDMA_MIN_PBLE_PAGES;
5894+
5895+ ibdev_dbg(to_ibdev(dev),
5896+ "HMC: mrte_loc %d loc_mem %u fpm max sds %u host_obj %d\n",
5897+ hmc_info->hmc_obj[IRDMA_HMC_IW_MR].mem_loc,
5898+ hmc_fpm_misc->loc_mem_pages, hmc_fpm_misc->max_sds,
5899+ is_mrte_loc_mem);
5900+
5901+ mrwanted = hmc_info->hmc_obj[IRDMA_HMC_IW_MR].max_cnt;
5902+ qpwanted = hmc_info->hmc_obj[IRDMA_HMC_IW_QP].max_cnt;
5903+ hmc_info->hmc_obj[IRDMA_HMC_IW_HDR].cnt = qpwanted;
5904+
5905+ hmc_info->hmc_obj[IRDMA_HMC_IW_OOISC].max_cnt = 0;
5906+ hmc_info->hmc_obj[IRDMA_HMC_IW_OOISCFFL].max_cnt = 0;
5907+ hmc_info->hmc_obj[IRDMA_HMC_IW_HTE].max_cnt = 0;
5908+ hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].max_cnt = 0;
5909+
5910+ if (!FIELD_GET(IRDMA_MANAGE_RSRC_VER2, dev->feature_info[IRDMA_FTN_FLAGS]))
5911+ hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].max_cnt =
5912+ min(hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].max_cnt,
5913+ (u32)IRDMA_FSIAV_CNT_MAX);
5914+
5915+ for (i = IRDMA_HMC_IW_QP; i < IRDMA_HMC_IW_MAX; i++)
5916+ hmc_info->hmc_obj[i].cnt = hmc_info->hmc_obj[i].max_cnt;
5917+
5918+ while (qpwanted >= IRDMA_MIN_QP_CNT) {
5919+ if (!irdma_set_loc_hmc_rsrc_gen_3(dev, loc_mem_pages, qpwanted))
5920+ break;
5921+
5922+ if (FIELD_GET(IRDMA_MANAGE_RSRC_VER2, dev->feature_info[IRDMA_FTN_FLAGS]))
5923+ return -EINVAL;
5924+
5925+ qpwanted /= 2;
5926+ if (mrte_loc == IRDMA_LOC_MEM) {
5927+ mrwanted = qpwanted * IRDMA_MIN_MR_PER_QP;
5928+ hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt =
5929+ min(hmc_info->hmc_obj[IRDMA_HMC_IW_MR].max_cnt, mrwanted);
5930+ }
5931+ }
5932+
5933+ if (qpwanted < IRDMA_MIN_QP_CNT) {
5934+ ibdev_dbg(to_ibdev(dev),
5935+ "HMC: ERROR: could not allocate fpm resources\n");
5936+ return -EINVAL;
5937+ }
5938+
5939+ irdma_set_host_hmc_rsrc_gen_3(dev);
5940+ ret_code = irdma_sc_cfg_iw_fpm(dev, dev->hmc_fn_id);
5941+ if (ret_code) {
5942+ ibdev_dbg(to_ibdev(dev),
5943+ "HMC: cfg_iw_fpm returned error_code[x%08X]\n",
5944+ readl(dev->hw_regs[IRDMA_CQPERRCODES]));
5945+
5946+ return ret_code;
5947+ }
5948+
5949+ return irdma_cfg_sd_mem(dev, hmc_info);
5950+}
5951+
47885952 /**
47895953 * irdma_cfg_fpm_val - configure HMC objects
47905954 * @dev: sc device struct
@@ -4792,16 +5956,15 @@ static void cfg_fpm_value_gen_2(struct irdma_sc_dev *dev, · 1 unmodified line
47925956 */
47935957 int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count)
47945958 {
4795− struct irdma_virt_mem virt_mem;
4796− u32 i, mem_size;
47975959 u32 qpwanted, mrwanted, pblewanted;
4798− u32 powerof2, hte;
5960+ u32 powerof2, hte, i;
47995961 u32 sd_needed;
48005962 u32 sd_diff;
48015963 u32 loop_count = 0;
48025964 struct irdma_hmc_info *hmc_info;
48035965 struct irdma_hmc_fpm_misc *hmc_fpm_misc;
48045966 int ret_code = 0;
5967+ u32 max_sds;
48055968
48065969 hmc_info = dev->hmc_info;
48075970 hmc_fpm_misc = &dev->hmc_fpm_misc;
@@ -4814,14 +5977,16 @@ int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) · 6 unmodified lines
48145977 return ret_code;
48155978 }
48165979
5980+ max_sds = hmc_fpm_misc->max_sds;
5981+
5982+ if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3)
5983+ return cfg_fpm_value_gen_3(dev, hmc_info, hmc_fpm_misc);
5984+
48175985 for (i = IRDMA_HMC_IW_QP; i < IRDMA_HMC_IW_MAX; i++)
48185986 hmc_info->hmc_obj[i].cnt = hmc_info->hmc_obj[i].max_cnt;
48195987 sd_needed = irdma_est_sd(dev, hmc_info);
4820− ibdev_dbg(to_ibdev(dev),
4821− "HMC: FW max resources sd_needed[%08d] first_sd_index[%04d]\n",
4822− sd_needed, hmc_info->first_sd_index);
4823− ibdev_dbg(to_ibdev(dev), "HMC: sd count %d where max sd is %d\n",
4824− hmc_info->sd_table.sd_cnt, hmc_fpm_misc->max_sds);
5988+ ibdev_dbg(to_ibdev(dev), "HMC: sd count %u where max sd is %u\n",
5989+ hmc_info->sd_table.sd_cnt, max_sds);
48255990
48265991 qpwanted = min(qp_count, hmc_info->hmc_obj[IRDMA_HMC_IW_QP].max_cnt);
48275992
@@ -4835,21 +6000,21 @@ int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) · 7 unmodified lines
48356000 pblewanted = hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].max_cnt;
48366001
48376002 ibdev_dbg(to_ibdev(dev),
4838− "HMC: req_qp=%d max_sd=%d, max_qp = %d, max_cq=%d, max_mr=%d, max_pble=%d, mc=%d, av=%d\n",
4839− qp_count, hmc_fpm_misc->max_sds,
6003+ "HMC: req_qp=%d max_sd=%u, max_qp = %u, max_cq=%u, max_mr=%u, max_pble=%u, mc=%d, av=%u\n",
6004+ qp_count, max_sds,
48406005 hmc_info->hmc_obj[IRDMA_HMC_IW_QP].max_cnt,
48416006 hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].max_cnt,
48426007 hmc_info->hmc_obj[IRDMA_HMC_IW_MR].max_cnt,
48436008 hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].max_cnt,
48446009 hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].max_cnt,
48456010 hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].max_cnt);
6011+
48466012 hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].cnt =
48476013 hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].max_cnt;
48486014 hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].cnt =
48496015 hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].max_cnt;
48506016 hmc_info->hmc_obj[IRDMA_HMC_IW_ARP].cnt =
48516017 hmc_info->hmc_obj[IRDMA_HMC_IW_ARP].max_cnt;
4852−
48536018 hmc_info->hmc_obj[IRDMA_HMC_IW_APBVT_ENTRY].cnt = 1;
48546019
48556020 while (irdma_q1_cnt(dev, hmc_info, qpwanted) > hmc_info->hmc_obj[IRDMA_HMC_IW_Q1].max_cnt)
@@ -4860,7 +6025,7 @@ int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) · 4 unmodified lines
48606025 hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt = qpwanted;
48616026 hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt =
48626027 min(2 * qpwanted, hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt);
4863− hmc_info->hmc_obj[IRDMA_HMC_IW_RESERVED].cnt = 0; /* Reserved */
6028+ hmc_info->hmc_obj[IRDMA_HMC_IW_SRQ].cnt = 0; /* Reserved */
48646029 hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt = mrwanted;
48656030
48666031 hte = round_up(qpwanted + hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].cnt, 512);
@@ -4898,11 +6063,12 @@ int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) · 31 unmodified lines
48986063 if (!(loop_count % 2) && qpwanted > 128) {
48996064 qpwanted /= 2;
49006065 } else {
4901− mrwanted /= 2;
49026066 pblewanted /= 2;
6067+ mrwanted /= 2;
49036068 }
49046069 continue;
49056070 }
6071+
49066072 if (dev->cqp->hmc_profile != IRDMA_HMC_PROFILE_FAVOR_VF &&
49076073 pblewanted > (512 * FPM_MULTIPLIER * sd_diff)) {
49086074 pblewanted -= 256 * FPM_MULTIPLIER * sd_diff;
@@ -4928,14 +6094,13 @@ int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) · 19 unmodified lines
49286094
49296095 if (sd_needed > hmc_fpm_misc->max_sds) {
49306096 ibdev_dbg(to_ibdev(dev),
4931− "HMC: cfg_fpm failed loop_cnt=%d, sd_needed=%d, max sd count %d\n",
6097+ "HMC: cfg_fpm failed loop_cnt=%u, sd_needed=%u, max sd count %u\n",
49326098 loop_count, sd_needed, hmc_info->sd_table.sd_cnt);
49336099 return -EINVAL;
49346100 }
49356101
4936− if (loop_count > 1 && sd_needed < hmc_fpm_misc->max_sds) {
4937− pblewanted += (hmc_fpm_misc->max_sds - sd_needed) * 256 *
4938− FPM_MULTIPLIER;
6102+ if (loop_count > 1 && sd_needed < max_sds) {
6103+ pblewanted += (max_sds - sd_needed) * 256 * FPM_MULTIPLIER;
49396104 hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].cnt = pblewanted;
49406105 sd_needed = irdma_est_sd(dev, hmc_info);
49416106 }
@@ -4959,18 +6124,7 @@ int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) · 17 unmodified lines
49596124 return ret_code;
49606125 }
49616126
4962− mem_size = sizeof(struct irdma_hmc_sd_entry) *
4963− (hmc_info->sd_table.sd_cnt + hmc_info->first_sd_index + 1);
4964− virt_mem.size = mem_size;
4965− virt_mem.va = kzalloc(virt_mem.size, GFP_KERNEL);
4966− if (!virt_mem.va) {
4967− ibdev_dbg(to_ibdev(dev),
4968− "HMC: failed to allocate memory for sd_entry buffer\n");
4969− return -ENOMEM;
4970− }
4971− hmc_info->sd_table.sd_entry = virt_mem.va;
4972−
4973− return ret_code;
6127+ return irdma_cfg_sd_mem(dev, hmc_info);
49746128 }
49756129
49766130 /**
@@ -5242,6 +6396,22 @@ static int irdma_exec_cqp_cmd(struct irdma_sc_dev *dev, · 265 unmodified lines
52426396 &pcmdinfo->in.u.mc_modify.info,
52436397 pcmdinfo->in.u.mc_modify.scratch);
52446398 break;
6399+ case IRDMA_OP_SRQ_CREATE:
6400+ status = irdma_sc_srq_create(pcmdinfo->in.u.srq_create.srq,
6401+ pcmdinfo->in.u.srq_create.scratch,
6402+ pcmdinfo->post_sq);
6403+ break;
6404+ case IRDMA_OP_SRQ_MODIFY:
6405+ status = irdma_sc_srq_modify(pcmdinfo->in.u.srq_modify.srq,
6406+ &pcmdinfo->in.u.srq_modify.info,
6407+ pcmdinfo->in.u.srq_modify.scratch,
6408+ pcmdinfo->post_sq);
6409+ break;
6410+ case IRDMA_OP_SRQ_DESTROY:
6411+ status = irdma_sc_srq_destroy(pcmdinfo->in.u.srq_destroy.srq,
6412+ pcmdinfo->in.u.srq_destroy.scratch,
6413+ pcmdinfo->post_sq);
6414+ break;
52456415 default:
52466416 status = -EOPNOTSUPP;
52476417 break;
@@ -5314,14 +6484,26 @@ void irdma_cfg_aeq(struct irdma_sc_dev *dev, u32 idx, bool enable) · 66 unmodified lines
53146484 */
53156485 void sc_vsi_update_stats(struct irdma_sc_vsi *vsi)
53166486 {
5317− struct irdma_gather_stats *gather_stats;
5318− struct irdma_gather_stats *last_gather_stats;
6487+ struct irdma_dev_hw_stats *hw_stats = &vsi->pestat->hw_stats;
6488+ struct irdma_gather_stats *gather_stats =
6489+ vsi->pestat->gather_info.gather_stats_va;
6490+ struct irdma_gather_stats *last_gather_stats =
6491+ vsi->pestat->gather_info.last_gather_stats_va;
6492+ const struct irdma_hw_stat_map *map = vsi->dev->hw_stats_map;
6493+ u16 max_stat_idx = vsi->dev->hw_attrs.max_stat_idx;
6494+ u16 i;
6495+
6496+ if (vsi->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) {
6497+ for (i = 0; i < max_stat_idx; i++) {
6498+ u16 idx = map[i].byteoff / sizeof(u64);
53196499
5320− gather_stats = vsi->pestat->gather_info.gather_stats_va;
5321− last_gather_stats = vsi->pestat->gather_info.last_gather_stats_va;
5322− irdma_update_stats(&vsi->pestat->hw_stats, gather_stats,
5323− last_gather_stats, vsi->dev->hw_stats_map,
5324− vsi->dev->hw_attrs.max_stat_idx);
6500+ hw_stats->stats_val[i] = gather_stats->val[idx];
6501+ }
6502+ return;
6503+ }
6504+
6505+ irdma_update_stats(hw_stats, gather_stats, last_gather_stats,
6506+ map, max_stat_idx);
53256507 }
53266508
53276509 /**
@@ -5356,6 +6538,9 @@ static inline void irdma_sc_init_hw(struct irdma_sc_dev *dev) · 28 unmodified lines
53566538 case IRDMA_GEN_2:
53576539 icrdma_init_hw(dev);
53586540 break;
6541+ case IRDMA_GEN_3:
6542+ ig3rdma_init_hw(dev);
6543+ break;
53596544 }
53606545 }
53616546
@@ -5381,10 +6566,15 @@ int irdma_sc_dev_init(enum irdma_vers ver, struct irdma_sc_dev *dev, · 19 unmodified lines
53816566 dev->fpm_commit_buf = info->fpm_commit_buf;
53826567 dev->hw = info->hw;
53836568 dev->hw->hw_addr = info->bar0;
6569+ dev->protocol_used = info->protocol_used;
53846570 /* Setup the hardware limits, hmc may limit further */
53856571 dev->hw_attrs.min_hw_qp_id = IRDMA_MIN_IW_QP_ID;
6572+ dev->hw_attrs.min_hw_srq_id = IRDMA_MIN_IW_SRQ_ID;
53866573 dev->hw_attrs.min_hw_aeq_size = IRDMA_MIN_AEQ_ENTRIES;
5387− dev->hw_attrs.max_hw_aeq_size = IRDMA_MAX_AEQ_ENTRIES;
6574+ if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3)
6575+ dev->hw_attrs.max_hw_aeq_size = IRDMA_MAX_AEQ_ENTRIES_GEN_3;
6576+ else
6577+ dev->hw_attrs.max_hw_aeq_size = IRDMA_MAX_AEQ_ENTRIES;
53886578 dev->hw_attrs.min_hw_ceq_size = IRDMA_MIN_CEQ_ENTRIES;
53896579 dev->hw_attrs.max_hw_ceq_size = IRDMA_MAX_CEQ_ENTRIES;
53906580 dev->hw_attrs.uk_attrs.min_hw_cq_size = IRDMA_MIN_CQ_SIZE;
@@ -5409,21 +6599,39 @@ int irdma_sc_dev_init(enum irdma_vers ver, struct irdma_sc_dev *dev, · 18 unmodified lines
54096599 dev->hw_attrs.max_sleep_count = IRDMA_SLEEP_COUNT;
54106600 dev->hw_attrs.max_cqp_compl_wait_time_ms = CQP_COMPL_WAIT_TIME_MS;
54116601
5412− dev->hw_attrs.uk_attrs.hw_rev = ver;
6602+ if (!dev->privileged) {
6603+ ret_code = irdma_vchnl_req_get_hmc_fcn(dev);
6604+ if (ret_code) {
6605+ ibdev_dbg(to_ibdev(dev),
6606+ "DEV: Get HMC function ret = %d\n",
6607+ ret_code);
6608+
6609+ return ret_code;
6610+ }
6611+ }
6612+
54136613 irdma_sc_init_hw(dev);
54146614
5415− if (irdma_wait_pe_ready(dev))
5416− return -ETIMEDOUT;
6615+ if (dev->privileged) {
6616+ if (irdma_wait_pe_ready(dev))
6617+ return -ETIMEDOUT;
54176618
5418− val = readl(dev->hw_regs[IRDMA_GLPCI_LBARCTRL]);
5419− db_size = (u8)FIELD_GET(IRDMA_GLPCI_LBARCTRL_PE_DB_SIZE, val);
5420− if (db_size != IRDMA_PE_DB_SIZE_4M && db_size != IRDMA_PE_DB_SIZE_8M) {
5421− ibdev_dbg(to_ibdev(dev),
5422− "DEV: RDMA PE doorbell is not enabled in CSR val 0x%x db_size=%d\n",
5423− val, db_size);
5424− return -ENODEV;
6619+ val = readl(dev->hw_regs[IRDMA_GLPCI_LBARCTRL]);
6620+ db_size = (u8)FIELD_GET(IRDMA_GLPCI_LBARCTRL_PE_DB_SIZE, val);
6621+ if (db_size != IRDMA_PE_DB_SIZE_4M &&
6622+ db_size != IRDMA_PE_DB_SIZE_8M) {
6623+ ibdev_dbg(to_ibdev(dev),
6624+ "DEV: RDMA PE doorbell is not enabled in CSR val 0x%x db_size=%d\n",
6625+ val, db_size);
6626+ return -ENODEV;
6627+ }
6628+ } else {
6629+ ret_code = irdma_vchnl_req_get_reg_layout(dev);
6630+ if (ret_code)
6631+ ibdev_dbg(to_ibdev(dev),
6632+ "DEV: Get Register layout failed ret = %d\n",
6633+ ret_code);
54256634 }
5426− dev->db_addr = dev->hw->hw_addr + (uintptr_t)dev->hw_regs[IRDMA_DB_ADDR_OFFSET];
54276635
54286636 return ret_code;
54296637 }
drivers/infiniband/hw/irdma/defs.h+145−119
@@ -14,6 +14,18 @@ · 13 unmodified lines
1414 #define IRDMA_PE_DB_SIZE_4M 1
1515 #define IRDMA_PE_DB_SIZE_8M 2
1616
17+#define IRDMA_IRD_HW_SIZE_4_GEN3 0
18+#define IRDMA_IRD_HW_SIZE_8_GEN3 1
19+#define IRDMA_IRD_HW_SIZE_16_GEN3 2
20+#define IRDMA_IRD_HW_SIZE_32_GEN3 3
21+#define IRDMA_IRD_HW_SIZE_64_GEN3 4
22+#define IRDMA_IRD_HW_SIZE_128_GEN3 5
23+#define IRDMA_IRD_HW_SIZE_256_GEN3 6
24+#define IRDMA_IRD_HW_SIZE_512_GEN3 7
25+#define IRDMA_IRD_HW_SIZE_1024_GEN3 8
26+#define IRDMA_IRD_HW_SIZE_2048_GEN3 9
27+#define IRDMA_IRD_HW_SIZE_4096_GEN3 10
28+
1729 #define IRDMA_IRD_HW_SIZE_4 0
1830 #define IRDMA_IRD_HW_SIZE_16 1
1931 #define IRDMA_IRD_HW_SIZE_64 2
@@ -114,6 +126,13 @@ enum irdma_protocol_used { · 94 unmodified lines
114126 #define IRDMA_UPDATE_SD_BUFF_SIZE 128
115127 #define IRDMA_FEATURE_BUF_SIZE (8 * IRDMA_MAX_FEATURES)
116128
129+#define ENABLE_LOC_MEM 63
130+#define IRDMA_ATOMICS_ALLOWED_BIT 1
131+#define MAX_PBLE_PER_SD 0x40000
132+#define MAX_PBLE_SD_PER_FCN 0x400
133+#define MAX_MR_PER_SD 0x8000
134+#define MAX_MR_SD_PER_FCN 0x80
135+#define IRDMA_PBLE_COMMIT_OFFSET 112
117136 #define IRDMA_MAX_QUANTA_PER_WR 8
118137
119138 #define IRDMA_QP_SW_MAX_WQ_QUANTA 32768
@@ -121,6 +140,10 @@ enum irdma_protocol_used { · 1 unmodified line
121140 #define IRDMA_QP_SW_MAX_RQ_QUANTA 32768
122141 #define IRDMA_MAX_QP_WRS(max_quanta_per_wr) \
123142 ((IRDMA_QP_SW_MAX_WQ_QUANTA - IRDMA_SQ_RSVD) / (max_quanta_per_wr))
143+#define IRDMA_SRQ_MIN_QUANTA 8
144+#define IRDMA_SRQ_MAX_QUANTA 262144
145+#define IRDMA_MAX_SRQ_WRS \
146+ ((IRDMA_SRQ_MAX_QUANTA - IRDMA_RQ_RSVD) / IRDMA_MAX_QUANTA_PER_WR)
124147
125148 #define IRDMAQP_TERM_SEND_TERM_AND_FIN 0
126149 #define IRDMAQP_TERM_SEND_TERM_ONLY 1
@@ -147,8 +170,13 @@ enum irdma_protocol_used { · 20 unmodified lines
147170 #define IRDMA_SQ_RSVD 258
148171 #define IRDMA_RQ_RSVD 1
149172
150−#define IRDMA_FEATURE_RTS_AE 1ULL
151−#define IRDMA_FEATURE_CQ_RESIZE 2ULL
173+#define IRDMA_FEATURE_RTS_AE BIT_ULL(0)
174+#define IRDMA_FEATURE_CQ_RESIZE BIT_ULL(1)
175+#define IRDMA_FEATURE_64_BYTE_CQE BIT_ULL(5)
176+#define IRDMA_FEATURE_ATOMIC_OPS BIT_ULL(6)
177+#define IRDMA_FEATURE_SRQ BIT_ULL(7)
178+#define IRDMA_FEATURE_CQE_TIMESTAMPING BIT_ULL(8)
179+
152180 #define IRDMAQP_OP_RDMA_WRITE 0x00
153181 #define IRDMAQP_OP_RDMA_READ 0x01
154182 #define IRDMAQP_OP_RDMA_SEND 0x03
@@ -161,6 +189,8 @@ enum irdma_protocol_used { · 6 unmodified lines
161189 #define IRDMAQP_OP_RDMA_READ_LOC_INV 0x0b
162190 #define IRDMAQP_OP_NOP 0x0c
163191 #define IRDMAQP_OP_RDMA_WRITE_SOL 0x0d
192+#define IRDMAQP_OP_ATOMIC_FETCH_ADD 0x0f
193+#define IRDMAQP_OP_ATOMIC_COMPARE_SWAP_ADD 0x11
164194 #define IRDMAQP_OP_GEN_RTS_AE 0x30
165195
166196 enum irdma_cqp_op_type {
@@ -212,9 +242,12 @@ enum irdma_cqp_op_type { · 45 unmodified lines
212242 IRDMA_OP_ADD_LOCAL_MAC_ENTRY = 46,
213243 IRDMA_OP_DELETE_LOCAL_MAC_ENTRY = 47,
214244 IRDMA_OP_CQ_MODIFY = 48,
245+ IRDMA_OP_SRQ_CREATE = 49,
246+ IRDMA_OP_SRQ_MODIFY = 50,
247+ IRDMA_OP_SRQ_DESTROY = 51,
215248
216249 /* Must be last entry*/
217− IRDMA_MAX_CQP_OPS = 49,
250+ IRDMA_MAX_CQP_OPS = 52,
218251 };
219252
220253 /* CQP SQ WQES */
@@ -224,6 +257,9 @@ enum irdma_cqp_op_type { · 3 unmodified lines
224257 #define IRDMA_CQP_OP_CREATE_CQ 0x03
225258 #define IRDMA_CQP_OP_MODIFY_CQ 0x04
226259 #define IRDMA_CQP_OP_DESTROY_CQ 0x05
260+#define IRDMA_CQP_OP_CREATE_SRQ 0x06
261+#define IRDMA_CQP_OP_MODIFY_SRQ 0x07
262+#define IRDMA_CQP_OP_DESTROY_SRQ 0x08
227263 #define IRDMA_CQP_OP_ALLOC_STAG 0x09
228264 #define IRDMA_CQP_OP_REG_MR 0x0a
229265 #define IRDMA_CQP_OP_QUERY_STAG 0x0b
@@ -265,97 +301,6 @@ enum irdma_cqp_op_type { · 35 unmodified lines
265301 #define IRDMA_CQP_OP_GATHER_STATS 0x2e
266302 #define IRDMA_CQP_OP_UP_MAP 0x2f
267303
268−/* Async Events codes */
269−#define IRDMA_AE_AMP_UNALLOCATED_STAG 0x0102
270−#define IRDMA_AE_AMP_INVALID_STAG 0x0103
271−#define IRDMA_AE_AMP_BAD_QP 0x0104
272−#define IRDMA_AE_AMP_BAD_PD 0x0105
273−#define IRDMA_AE_AMP_BAD_STAG_KEY 0x0106
274−#define IRDMA_AE_AMP_BAD_STAG_INDEX 0x0107
275−#define IRDMA_AE_AMP_BOUNDS_VIOLATION 0x0108
276−#define IRDMA_AE_AMP_RIGHTS_VIOLATION 0x0109
277−#define IRDMA_AE_AMP_TO_WRAP 0x010a
278−#define IRDMA_AE_AMP_FASTREG_VALID_STAG 0x010c
279−#define IRDMA_AE_AMP_FASTREG_MW_STAG 0x010d
280−#define IRDMA_AE_AMP_FASTREG_INVALID_RIGHTS 0x010e
281−#define IRDMA_AE_AMP_FASTREG_INVALID_LENGTH 0x0110
282−#define IRDMA_AE_AMP_INVALIDATE_SHARED 0x0111
283−#define IRDMA_AE_AMP_INVALIDATE_NO_REMOTE_ACCESS_RIGHTS 0x0112
284−#define IRDMA_AE_AMP_INVALIDATE_MR_WITH_BOUND_WINDOWS 0x0113
285−#define IRDMA_AE_AMP_MWBIND_VALID_STAG 0x0114
286−#define IRDMA_AE_AMP_MWBIND_OF_MR_STAG 0x0115
287−#define IRDMA_AE_AMP_MWBIND_TO_ZERO_BASED_STAG 0x0116
288−#define IRDMA_AE_AMP_MWBIND_TO_MW_STAG 0x0117
289−#define IRDMA_AE_AMP_MWBIND_INVALID_RIGHTS 0x0118
290−#define IRDMA_AE_AMP_MWBIND_INVALID_BOUNDS 0x0119
291−#define IRDMA_AE_AMP_MWBIND_TO_INVALID_PARENT 0x011a
292−#define IRDMA_AE_AMP_MWBIND_BIND_DISABLED 0x011b
293−#define IRDMA_AE_PRIV_OPERATION_DENIED 0x011c
294−#define IRDMA_AE_AMP_INVALIDATE_TYPE1_MW 0x011d
295−#define IRDMA_AE_AMP_MWBIND_ZERO_BASED_TYPE1_MW 0x011e
296−#define IRDMA_AE_AMP_FASTREG_INVALID_PBL_HPS_CFG 0x011f
297−#define IRDMA_AE_AMP_MWBIND_WRONG_TYPE 0x0120
298−#define IRDMA_AE_AMP_FASTREG_PBLE_MISMATCH 0x0121
299−#define IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG 0x0132
300−#define IRDMA_AE_UDA_XMIT_BAD_PD 0x0133
301−#define IRDMA_AE_UDA_XMIT_DGRAM_TOO_SHORT 0x0134
302−#define IRDMA_AE_UDA_L4LEN_INVALID 0x0135
303−#define IRDMA_AE_BAD_CLOSE 0x0201
304−#define IRDMA_AE_RDMAP_ROE_BAD_LLP_CLOSE 0x0202
305−#define IRDMA_AE_CQ_OPERATION_ERROR 0x0203
306−#define IRDMA_AE_RDMA_READ_WHILE_ORD_ZERO 0x0205
307−#define IRDMA_AE_STAG_ZERO_INVALID 0x0206
308−#define IRDMA_AE_IB_RREQ_AND_Q1_FULL 0x0207
309−#define IRDMA_AE_IB_INVALID_REQUEST 0x0208
310−#define IRDMA_AE_WQE_UNEXPECTED_OPCODE 0x020a
311−#define IRDMA_AE_WQE_INVALID_PARAMETER 0x020b
312−#define IRDMA_AE_WQE_INVALID_FRAG_DATA 0x020c
313−#define IRDMA_AE_IB_REMOTE_ACCESS_ERROR 0x020d
314−#define IRDMA_AE_IB_REMOTE_OP_ERROR 0x020e
315−#define IRDMA_AE_WQE_LSMM_TOO_LONG 0x0220
316−#define IRDMA_AE_INVALID_REQUEST 0x0223
317−#define IRDMA_AE_DDP_INVALID_MSN_GAP_IN_MSN 0x0301
318−#define IRDMA_AE_DDP_UBE_DDP_MESSAGE_TOO_LONG_FOR_AVAILABLE_BUFFER 0x0303
319−#define IRDMA_AE_DDP_UBE_INVALID_DDP_VERSION 0x0304
320−#define IRDMA_AE_DDP_UBE_INVALID_MO 0x0305
321−#define IRDMA_AE_DDP_UBE_INVALID_MSN_NO_BUFFER_AVAILABLE 0x0306
322−#define IRDMA_AE_DDP_UBE_INVALID_QN 0x0307
323−#define IRDMA_AE_DDP_NO_L_BIT 0x0308
324−#define IRDMA_AE_RDMAP_ROE_INVALID_RDMAP_VERSION 0x0311
325−#define IRDMA_AE_RDMAP_ROE_UNEXPECTED_OPCODE 0x0312
326−#define IRDMA_AE_ROE_INVALID_RDMA_READ_REQUEST 0x0313
327−#define IRDMA_AE_ROE_INVALID_RDMA_WRITE_OR_READ_RESP 0x0314
328−#define IRDMA_AE_ROCE_RSP_LENGTH_ERROR 0x0316
329−#define IRDMA_AE_ROCE_EMPTY_MCG 0x0380
330−#define IRDMA_AE_ROCE_BAD_MC_IP_ADDR 0x0381
331−#define IRDMA_AE_ROCE_BAD_MC_QPID 0x0382
332−#define IRDMA_AE_MCG_QP_PROTOCOL_MISMATCH 0x0383
333−#define IRDMA_AE_INVALID_ARP_ENTRY 0x0401
334−#define IRDMA_AE_INVALID_TCP_OPTION_RCVD 0x0402
335−#define IRDMA_AE_STALE_ARP_ENTRY 0x0403
336−#define IRDMA_AE_INVALID_AH_ENTRY 0x0406
337−#define IRDMA_AE_LLP_CLOSE_COMPLETE 0x0501
338−#define IRDMA_AE_LLP_CONNECTION_RESET 0x0502
339−#define IRDMA_AE_LLP_FIN_RECEIVED 0x0503
340−#define IRDMA_AE_LLP_RECEIVED_MARKER_AND_LENGTH_FIELDS_DONT_MATCH 0x0504
341−#define IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR 0x0505
342−#define IRDMA_AE_LLP_SEGMENT_TOO_SMALL 0x0507
343−#define IRDMA_AE_LLP_SYN_RECEIVED 0x0508
344−#define IRDMA_AE_LLP_TERMINATE_RECEIVED 0x0509
345−#define IRDMA_AE_LLP_TOO_MANY_RETRIES 0x050a
346−#define IRDMA_AE_LLP_TOO_MANY_KEEPALIVE_RETRIES 0x050b
347−#define IRDMA_AE_LLP_DOUBT_REACHABILITY 0x050c
348−#define IRDMA_AE_LLP_CONNECTION_ESTABLISHED 0x050e
349−#define IRDMA_AE_LLP_TOO_MANY_RNRS 0x050f
350−#define IRDMA_AE_RESOURCE_EXHAUSTION 0x0520
351−#define IRDMA_AE_RESET_SENT 0x0601
352−#define IRDMA_AE_TERMINATE_SENT 0x0602
353−#define IRDMA_AE_RESET_NOT_SENT 0x0603
354−#define IRDMA_AE_LCE_QP_CATASTROPHIC 0x0700
355−#define IRDMA_AE_LCE_FUNCTION_CATASTROPHIC 0x0701
356−#define IRDMA_AE_LCE_CQ_CATASTROPHIC 0x0702
357−#define IRDMA_AE_QP_SUSPEND_COMPLETE 0x0900
358−
359304 #define FLD_LS_64(dev, val, field) \
360305 (((u64)(val) << (dev)->hw_shifts[field ## _S]) & (dev)->hw_masks[field ## _M])
361306 #define FLD_RS_64(dev, val, field) \
@@ -393,9 +338,13 @@ enum irdma_cqp_op_type { · 31 unmodified lines
393338 #define IRDMA_CQPSQ_STATS_USE_INST BIT_ULL(61)
394339 #define IRDMA_CQPSQ_STATS_OP GENMASK_ULL(37, 32)
395340 #define IRDMA_CQPSQ_STATS_INST_INDEX GENMASK_ULL(6, 0)
396−#define IRDMA_CQPSQ_STATS_HMC_FCN_INDEX GENMASK_ULL(5, 0)
341+#define IRDMA_CQPSQ_STATS_HMC_FCN_INDEX GENMASK_ULL(15, 0)
397342 #define IRDMA_CQPSQ_WS_WQEVALID BIT_ULL(63)
398−#define IRDMA_CQPSQ_WS_NODEOP GENMASK_ULL(53, 52)
343+#define IRDMA_CQPSQ_WS_NODEOP GENMASK_ULL(55, 52)
344+#define IRDMA_SD_MAX GENMASK_ULL(15, 0)
345+#define IRDMA_MEM_MAX GENMASK_ULL(15, 0)
346+#define IRDMA_QP_MEM_LOC GENMASK_ULL(47, 44)
347+#define IRDMA_MR_MEM_LOC GENMASK_ULL(27, 24)
399348
400349 #define IRDMA_CQPSQ_WS_ENABLENODE BIT_ULL(62)
401350 #define IRDMA_CQPSQ_WS_NODETYPE BIT_ULL(61)
@@ -404,16 +353,16 @@ enum irdma_cqp_op_type { · 2 unmodified lines
404353 #define IRDMA_CQPSQ_WS_VMVFTYPE GENMASK_ULL(55, 54)
405354 #define IRDMA_CQPSQ_WS_VMVFNUM GENMASK_ULL(51, 42)
406355 #define IRDMA_CQPSQ_WS_OP GENMASK_ULL(37, 32)
407−#define IRDMA_CQPSQ_WS_PARENTID GENMASK_ULL(25, 16)
408−#define IRDMA_CQPSQ_WS_NODEID GENMASK_ULL(9, 0)
409−#define IRDMA_CQPSQ_WS_VSI GENMASK_ULL(57, 48)
356+#define IRDMA_CQPSQ_WS_PARENTID GENMASK_ULL(29, 16)
357+#define IRDMA_CQPSQ_WS_NODEID GENMASK_ULL(13, 0)
358+#define IRDMA_CQPSQ_WS_VSI GENMASK_ULL(63, 48)
410359 #define IRDMA_CQPSQ_WS_WEIGHT GENMASK_ULL(38, 32)
411360
412361 #define IRDMA_CQPSQ_UP_WQEVALID BIT_ULL(63)
413362 #define IRDMA_CQPSQ_UP_USEVLAN BIT_ULL(62)
414363 #define IRDMA_CQPSQ_UP_USEOVERRIDE BIT_ULL(61)
415364 #define IRDMA_CQPSQ_UP_OP GENMASK_ULL(37, 32)
416−#define IRDMA_CQPSQ_UP_HMCFCNIDX GENMASK_ULL(5, 0)
365+#define IRDMA_CQPSQ_UP_HMCFCNIDX GENMASK_ULL(15, 0)
417366 #define IRDMA_CQPSQ_UP_CNPOVERRIDE GENMASK_ULL(37, 32)
418367 #define IRDMA_CQPSQ_QUERY_RDMA_FEATURES_WQEVALID BIT_ULL(63)
419368 #define IRDMA_CQPSQ_QUERY_RDMA_FEATURES_BUF_LEN GENMASK_ULL(31, 0)
@@ -448,6 +397,16 @@ enum irdma_cqp_op_type { · 28 unmodified lines
448397 #define IRDMA_CQPHC_SVER GENMASK_ULL(31, 24)
449398 #define IRDMA_CQPHC_SQBASE GENMASK_ULL(63, 9)
450399
400+#define IRDMA_CQPHC_TIMESTAMP_OVERRIDE BIT_ULL(5)
401+#define IRDMA_CQPHC_TS_SHIFT GENMASK_ULL(12, 8)
402+#define IRDMA_CQPHC_EN_FINE_GRAINED_TIMERS BIT_ULL(0)
403+
404+#define IRDMA_CQPHC_OOISC_BLKSIZE GENMASK_ULL(63, 60)
405+#define IRDMA_CQPHC_RRSP_BLKSIZE GENMASK_ULL(59, 56)
406+#define IRDMA_CQPHC_Q1_BLKSIZE GENMASK_ULL(55, 52)
407+#define IRDMA_CQPHC_XMIT_BLKSIZE GENMASK_ULL(51, 48)
408+#define IRDMA_CQPHC_BLKSIZES_VALID BIT_ULL(4)
409+
451410 #define IRDMA_CQPHC_QPCTX GENMASK_ULL(63, 0)
452411 #define IRDMA_QP_DBSA_HW_SQ_TAIL GENMASK_ULL(14, 0)
453412 #define IRDMA_CQ_DBSA_CQEIDX GENMASK_ULL(19, 0)
@@ -461,6 +420,8 @@ enum irdma_cqp_op_type { · 7 unmodified lines
461420
462421 #define IRDMA_CCQ_OPRETVAL GENMASK_ULL(31, 0)
463422
423+#define IRDMA_CCQ_DEFINFO GENMASK_ULL(63, 32)
424+
464425 #define IRDMA_CQ_MINERR GENMASK_ULL(15, 0)
465426 #define IRDMA_CQ_MAJERR GENMASK_ULL(31, 16)
466427 #define IRDMA_CQ_WQEIDX GENMASK_ULL(46, 32)
@@ -469,6 +430,7 @@ enum irdma_cqp_op_type { · 2 unmodified lines
469430 #define IRDMA_CQ_ERROR BIT_ULL(55)
470431 #define IRDMA_CQ_SQ BIT_ULL(62)
471432
433+#define IRDMA_CQ_SRQ BIT_ULL(52)
472434 #define IRDMA_CQ_VALID BIT_ULL(63)
473435 #define IRDMA_CQ_IMMVALID BIT_ULL(62)
474436 #define IRDMA_CQ_UDSMACVALID BIT_ULL(61)
@@ -476,8 +438,6 @@ enum irdma_cqp_op_type { · 1 unmodified line
476438 #define IRDMA_CQ_UDSMAC GENMASK_ULL(47, 0)
477439 #define IRDMA_CQ_UDVLAN GENMASK_ULL(63, 48)
478440
479−#define IRDMA_CQ_IMMDATA_S 0
480−#define IRDMA_CQ_IMMDATA_M (0xffffffffffffffffULL << IRDMA_CQ_IMMVALID_S)
481441 #define IRDMA_CQ_IMMDATALOW32 GENMASK_ULL(31, 0)
482442 #define IRDMA_CQ_IMMDATAUP32 GENMASK_ULL(63, 32)
483443 #define IRDMACQ_PAYLDLEN GENMASK_ULL(31, 0)
@@ -508,6 +468,17 @@ enum irdma_cqp_op_type { · 24 unmodified lines
508468 #define IRDMA_AEQE_Q2DATA GENMASK_ULL(62, 61)
509469 #define IRDMA_AEQE_VALID BIT_ULL(63)
510470
471+#define IRDMA_AEQE_Q2DATA_GEN_3 GENMASK_ULL(5, 4)
472+#define IRDMA_AEQE_TCPSTATE_GEN_3 GENMASK_ULL(3, 0)
473+#define IRDMA_AEQE_QPCQID_GEN_3 GENMASK_ULL(24, 0)
474+#define IRDMA_AEQE_AECODE_GEN_3 GENMASK_ULL(61, 50)
475+#define IRDMA_AEQE_OVERFLOW_GEN_3 BIT_ULL(62)
476+#define IRDMA_AEQE_WQDESCIDX_GEN_3 GENMASK_ULL(49, 32)
477+#define IRDMA_AEQE_IWSTATE_GEN_3 GENMASK_ULL(31, 29)
478+#define IRDMA_AEQE_AESRC_GEN_3 GENMASK_ULL(28, 25)
479+#define IRDMA_AEQE_CMPL_CTXT_S 6
480+#define IRDMA_AEQE_CMPL_CTXT GENMASK_ULL(63, 6)
481+
511482 #define IRDMA_UDA_QPSQ_NEXT_HDR GENMASK_ULL(23, 16)
512483 #define IRDMA_UDA_QPSQ_OPCODE GENMASK_ULL(37, 32)
513484 #define IRDMA_UDA_QPSQ_L4LEN GENMASK_ULL(45, 42)
@@ -530,11 +501,14 @@ enum irdma_cqp_op_type { · 16 unmodified lines
530501 #define IRDMA_CQPSQ_WQEVALID BIT_ULL(63)
531502 #define IRDMA_CQPSQ_TPHVAL GENMASK_ULL(7, 0)
532503
533−#define IRDMA_CQPSQ_VSIIDX GENMASK_ULL(17, 8)
504+#define IRDMA_CQPSQ_VSIIDX GENMASK_ULL(23, 8)
534505 #define IRDMA_CQPSQ_TPHEN BIT_ULL(60)
535506
536507 #define IRDMA_CQPSQ_PBUFADDR IRDMA_CQPHC_QPCTX
537508
509+#define IRDMA_CQPSQ_PASID GENMASK_ULL(51, 32)
510+#define IRDMA_CQPSQ_PASID_VALID BIT_ULL(62)
511+
538512 /* Create/Modify/Destroy QP */
539513
540514 #define IRDMA_CQPSQ_QP_NEWMSS GENMASK_ULL(45, 32)
@@ -566,10 +540,30 @@ enum irdma_cqp_op_type { · 25 unmodified lines
566540
567541 #define IRDMA_CQPSQ_QP_DBSHADOWADDR IRDMA_CQPHC_QPCTX
568542
543+#define IRDMA_CQPSQ_SRQ_RQSIZE GENMASK_ULL(3, 0)
544+#define IRDMA_CQPSQ_SRQ_RQ_WQE_SIZE GENMASK_ULL(5, 4)
545+#define IRDMA_CQPSQ_SRQ_SRQ_LIMIT GENMASK_ULL(43, 32)
546+#define IRDMA_CQPSQ_SRQ_SRQCTX GENMASK_ULL(63, 6)
547+#define IRDMA_CQPSQ_SRQ_PD_ID GENMASK_ULL(39, 16)
548+#define IRDMA_CQPSQ_SRQ_SRQ_ID GENMASK_ULL(15, 0)
549+#define IRDMA_CQPSQ_SRQ_OP GENMASK_ULL(37, 32)
550+#define IRDMA_CQPSQ_SRQ_LEAF_PBL_SIZE GENMASK_ULL(45, 44)
551+#define IRDMA_CQPSQ_SRQ_VIRTMAP BIT_ULL(47)
552+#define IRDMA_CQPSQ_SRQ_TPH_EN BIT_ULL(60)
553+#define IRDMA_CQPSQ_SRQ_ARM_LIMIT_EVENT BIT_ULL(61)
554+#define IRDMA_CQPSQ_SRQ_FIRST_PM_PBL_IDX GENMASK_ULL(27, 0)
555+#define IRDMA_CQPSQ_SRQ_TPH_VALUE GENMASK_ULL(7, 0)
556+#define IRDMA_CQPSQ_SRQ_PHYSICAL_BUFFER_ADDR_S 8
557+#define IRDMA_CQPSQ_SRQ_PHYSICAL_BUFFER_ADDR GENMASK_ULL(63, 8)
558+#define IRDMA_CQPSQ_SRQ_DB_SHADOW_ADDR_S 6
559+#define IRDMA_CQPSQ_SRQ_DB_SHADOW_ADDR GENMASK_ULL(63, 6)
560+
569561 #define IRDMA_CQPSQ_CQ_CQSIZE GENMASK_ULL(20, 0)
570562 #define IRDMA_CQPSQ_CQ_CQCTX GENMASK_ULL(62, 0)
571563 #define IRDMA_CQPSQ_CQ_SHADOW_READ_THRESHOLD GENMASK(17, 0)
572564
565+#define IRDMA_CQPSQ_CQ_CQID_HIGH GENMASK_ULL(52, 50)
566+#define IRDMA_CQPSQ_CQ_CEQID_HIGH GENMASK_ULL(59, 54)
573567 #define IRDMA_CQPSQ_CQ_OP GENMASK_ULL(37, 32)
574568 #define IRDMA_CQPSQ_CQ_CQRESIZE BIT_ULL(43)
575569 #define IRDMA_CQPSQ_CQ_LPBLSIZE GENMASK_ULL(45, 44)
@@ -590,6 +584,7 @@ enum irdma_cqp_op_type { · 14 unmodified lines
590584 #define IRDMA_CQPSQ_STAG_MR BIT_ULL(43)
591585 #define IRDMA_CQPSQ_STAG_MWTYPE BIT_ULL(42)
592586 #define IRDMA_CQPSQ_STAG_MW1_BIND_DONT_VLDT_KEY BIT_ULL(58)
587+#define IRDMA_CQPSQ_STAG_PDID_HI GENMASK_ULL(59, 54)
593588
594589 #define IRDMA_CQPSQ_STAG_LPBLSIZE IRDMA_CQPSQ_CQ_LPBLSIZE
595590 #define IRDMA_CQPSQ_STAG_HPAGESIZE GENMASK_ULL(47, 46)
@@ -600,7 +595,8 @@ enum irdma_cqp_op_type { · 4 unmodified lines
600595 #define IRDMA_CQPSQ_STAG_USEPFRID BIT_ULL(61)
601596
602597 #define IRDMA_CQPSQ_STAG_PBA IRDMA_CQPHC_QPCTX
603−#define IRDMA_CQPSQ_STAG_HMCFNIDX GENMASK_ULL(5, 0)
598+#define IRDMA_CQPSQ_STAG_HMCFNIDX GENMASK_ULL(15, 0)
599+#define IRDMA_CQPSQ_STAG_REMOTE_ATOMIC_EN BIT_ULL(61)
604600
605601 #define IRDMA_CQPSQ_STAG_FIRSTPMPBLIDX GENMASK_ULL(27, 0)
606602 #define IRDMA_CQPSQ_QUERYSTAG_IDX IRDMA_CQPSQ_STAG_IDX
@@ -628,11 +624,8 @@ enum irdma_cqp_op_type { · 21 unmodified lines
628624 /* Manage Push Page - MPP */
629625 #define IRDMA_INVALID_PUSH_PAGE_INDEX_GEN_1 0xffff
630626 #define IRDMA_INVALID_PUSH_PAGE_INDEX 0xffffffff
631−
632−#define IRDMA_CQPSQ_MPP_QS_HANDLE GENMASK_ULL(9, 0)
633−#define IRDMA_CQPSQ_MPP_PPIDX GENMASK_ULL(9, 0)
627+#define IRDMA_CQPSQ_MPP_PPIDX GENMASK_ULL(31, 0)
634628 #define IRDMA_CQPSQ_MPP_PPTYPE GENMASK_ULL(61, 60)
635−
636629 #define IRDMA_CQPSQ_MPP_FREE_PAGE BIT_ULL(62)
637630
638631 /* Upload Context - UCTX */
@@ -651,6 +644,8 @@ enum irdma_cqp_op_type { · 12 unmodified lines
651644 #define IRDMA_CQPSQ_CEQ_CEQSIZE GENMASK_ULL(21, 0)
652645 #define IRDMA_CQPSQ_CEQ_CEQID GENMASK_ULL(9, 0)
653646
647+#define IRDMA_CQPSQ_CEQ_CEQID_HIGH GENMASK_ULL(15, 10)
648+
654649 #define IRDMA_CQPSQ_CEQ_LPBLSIZE IRDMA_CQPSQ_CQ_LPBLSIZE
655650 #define IRDMA_CQPSQ_CEQ_VMAP BIT_ULL(47)
656651 #define IRDMA_CQPSQ_CEQ_ITRNOEXPIRE BIT_ULL(46)
@@ -660,10 +655,10 @@ enum irdma_cqp_op_type { · 3 unmodified lines
660655 #define IRDMA_CQPSQ_AEQ_VMAP BIT_ULL(47)
661656 #define IRDMA_CQPSQ_AEQ_FIRSTPMPBLIDX GENMASK_ULL(27, 0)
662657
663−#define IRDMA_COMMIT_FPM_QPCNT GENMASK_ULL(18, 0)
664−
658+#define IRDMA_COMMIT_FPM_QPCNT GENMASK_ULL(20, 0)
665659 #define IRDMA_COMMIT_FPM_BASE_S 32
666−#define IRDMA_CQPSQ_CFPM_HMCFNID GENMASK_ULL(5, 0)
660+#define IRDMA_CQPSQ_CFPM_HMCFNID GENMASK_ULL(15, 0)
661+
667662 #define IRDMA_CQPSQ_FWQE_AECODE GENMASK_ULL(15, 0)
668663 #define IRDMA_CQPSQ_FWQE_AESOURCE GENMASK_ULL(19, 16)
669664 #define IRDMA_CQPSQ_FWQE_RQMNERR GENMASK_ULL(15, 0)
@@ -675,6 +670,10 @@ enum irdma_cqp_op_type { · 5 unmodified lines
675670 #define IRDMA_CQPSQ_FWQE_USERFLCODE BIT_ULL(60)
676671 #define IRDMA_CQPSQ_FWQE_FLUSHSQ BIT_ULL(61)
677672 #define IRDMA_CQPSQ_FWQE_FLUSHRQ BIT_ULL(62)
673+#define IRDMA_CQPSQ_FWQE_ERR_SQ_IDX_VALID BIT_ULL(42)
674+#define IRDMA_CQPSQ_FWQE_ERR_SQ_IDX GENMASK_ULL(49, 32)
675+#define IRDMA_CQPSQ_FWQE_ERR_RQ_IDX_VALID BIT_ULL(43)
676+#define IRDMA_CQPSQ_FWQE_ERR_RQ_IDX GENMASK_ULL(46, 32)
678677 #define IRDMA_CQPSQ_MAPT_PORT GENMASK_ULL(15, 0)
679678 #define IRDMA_CQPSQ_MAPT_ADDPORT BIT_ULL(62)
680679 #define IRDMA_CQPSQ_UPESD_SDCMD GENMASK_ULL(31, 0)
@@ -693,9 +692,12 @@ enum irdma_cqp_op_type { · 12 unmodified lines
693692 #define IRDMA_CQPSQ_SUSPENDQP_QPID GENMASK_ULL(23, 0)
694693 #define IRDMA_CQPSQ_RESUMEQP_QSHANDLE GENMASK_ULL(31, 0)
695694 #define IRDMA_CQPSQ_RESUMEQP_QPID GENMASK(23, 0)
695+#define IRDMA_MANAGE_RSRC_VER2 BIT_ULL(2)
696696
697697 #define IRDMA_CQPSQ_MIN_STAG_INVALID 0x0001
698698 #define IRDMA_CQPSQ_MIN_SUSPEND_PND 0x0005
699+#define IRDMA_CQPSQ_MIN_DEF_CMPL 0x0006
700+#define IRDMA_CQPSQ_MIN_OOO_CMPL 0x0007
699701
700702 #define IRDMA_CQPSQ_MAJ_NO_ERROR 0x0000
701703 #define IRDMA_CQPSQ_MAJ_OBJCACHE_ERROR 0xF000
@@ -712,6 +714,11 @@ enum irdma_cqp_op_type { · 10 unmodified lines
712714 #define IRDMAQPC_INSERTL2TAG2 BIT_ULL(11)
713715 #define IRDMAQPC_LIMIT GENMASK_ULL(13, 12)
714716
717+#define IRDMAQPC_USE_SRQ BIT_ULL(10)
718+#define IRDMAQPC_SRQ_ID GENMASK_ULL(15, 0)
719+#define IRDMAQPC_PASID GENMASK_ULL(19, 0)
720+#define IRDMAQPC_PASID_VALID BIT_ULL(11)
721+
715722 #define IRDMAQPC_ECN_EN BIT_ULL(14)
716723 #define IRDMAQPC_DROPOOOSEG BIT_ULL(15)
717724 #define IRDMAQPC_DUPACK_THRESH GENMASK_ULL(18, 16)
@@ -782,21 +789,31 @@ enum irdma_cqp_op_type { · 64 unmodified lines
782789 #define IRDMAQPC_CWNDROCE GENMASK_ULL(55, 32)
783790 #define IRDMAQPC_SNDWL1 GENMASK_ULL(31, 0)
784791 #define IRDMAQPC_SNDWL2 GENMASK_ULL(63, 32)
785−#define IRDMAQPC_ERR_RQ_IDX GENMASK_ULL(45, 32)
792+#define IRDMAQPC_MINRNR_TIMER GENMASK_ULL(4, 0)
793+#define IRDMAQPC_ERR_RQ_IDX GENMASK_ULL(46, 32)
786794 #define IRDMAQPC_RTOMIN GENMASK_ULL(63, 57)
787795 #define IRDMAQPC_MAXSNDWND GENMASK_ULL(31, 0)
788796 #define IRDMAQPC_REXMIT_THRESH GENMASK_ULL(53, 48)
789797 #define IRDMAQPC_RNRNAK_THRESH GENMASK_ULL(56, 54)
790−#define IRDMAQPC_TXCQNUM GENMASK_ULL(18, 0)
791−#define IRDMAQPC_RXCQNUM GENMASK_ULL(50, 32)
798+#define IRDMAQPC_TXCQNUM GENMASK_ULL(24, 0)
799+#define IRDMAQPC_RXCQNUM GENMASK_ULL(56, 32)
792800 #define IRDMAQPC_STAT_INDEX GENMASK_ULL(6, 0)
793801 #define IRDMAQPC_Q2ADDR GENMASK_ULL(63, 8)
794802 #define IRDMAQPC_LASTBYTESENT GENMASK_ULL(7, 0)
795803 #define IRDMAQPC_MACADDRESS GENMASK_ULL(63, 16)
796804 #define IRDMAQPC_ORDSIZE GENMASK_ULL(7, 0)
797805
806+#define IRDMAQPC_LOCALACKTIMEOUT GENMASK_ULL(12, 8)
807+#define IRDMAQPC_RNRNAK_TMR GENMASK_ULL(4, 0)
808+#define IRDMAQPC_ORDSIZE_GEN3 GENMASK_ULL(10, 0)
809+#define IRDMAQPC_REMOTE_ATOMIC_EN BIT_ULL(18)
810+#define IRDMAQPC_STAT_INDEX_GEN3 GENMASK_ULL(47, 32)
811+#define IRDMAQPC_PKT_LIMIT GENMASK_ULL(55, 48)
812+
798813 #define IRDMAQPC_IRDSIZE GENMASK_ULL(18, 16)
799814
815+#define IRDMAQPC_IRDSIZE_GEN3 GENMASK_ULL(17, 14)
816+
800817 #define IRDMAQPC_UDPRIVCQENABLE BIT_ULL(19)
801818 #define IRDMAQPC_WRRDRSPOK BIT_ULL(20)
802819 #define IRDMAQPC_RDOK BIT_ULL(21)
@@ -833,6 +850,7 @@ enum irdma_cqp_op_type { · 30 unmodified lines
833850 #define IRDMA_FEATURE_INFO GENMASK_ULL(47, 0)
834851 #define IRDMA_FEATURE_CNT GENMASK_ULL(47, 32)
835852 #define IRDMA_FEATURE_TYPE GENMASK_ULL(63, 48)
853+#define IRDMA_FEATURE_RSRC_MAX GENMASK_ULL(31, 0)
836854
837855 #define IRDMAQPSQ_OPCODE GENMASK_ULL(37, 32)
838856 #define IRDMAQPSQ_COPY_HOST_PBL BIT_ULL(43)
@@ -856,7 +874,7 @@ enum irdma_cqp_op_type { · 17 unmodified lines
856874 #define IRDMAQPSQ_REMSTAGINV GENMASK_ULL(31, 0)
857875 #define IRDMAQPSQ_DESTQKEY GENMASK_ULL(31, 0)
858876 #define IRDMAQPSQ_DESTQPN GENMASK_ULL(55, 32)
859−#define IRDMAQPSQ_AHID GENMASK_ULL(16, 0)
877+#define IRDMAQPSQ_AHID GENMASK_ULL(24, 0)
860878 #define IRDMAQPSQ_INLINEDATAFLAG BIT_ULL(57)
861879
862880 #define IRDMA_INLINE_VALID_S 7
@@ -869,6 +887,9 @@ enum irdma_cqp_op_type { · 6 unmodified lines
869887
870888 #define IRDMAQPSQ_REMTO IRDMA_CQPHC_QPCTX
871889
890+#define IRDMAQPSQ_STAG GENMASK_ULL(31, 0)
891+#define IRDMAQPSQ_REMOTE_STAG GENMASK_ULL(31, 0)
892+
872893 #define IRDMAQPSQ_STAGRIGHTS GENMASK_ULL(52, 48)
873894 #define IRDMAQPSQ_VABASEDTO BIT_ULL(53)
874895 #define IRDMAQPSQ_MEMWINDOWTYPE BIT_ULL(54)
@@ -879,6 +900,8 @@ enum irdma_cqp_op_type { · 4 unmodified lines
879900
880901 #define IRDMAQPSQ_BASEVA_TO_FBO IRDMA_CQPHC_QPCTX
881902
903+#define IRDMAQPSQ_REMOTE_ATOMICS_EN BIT_ULL(55)
904+
882905 #define IRDMAQPSQ_LOCSTAG GENMASK_ULL(31, 0)
883906
884907 #define IRDMAQPSQ_STAGKEY GENMASK_ULL(7, 0)
@@ -903,11 +926,14 @@ enum irdma_cqp_op_type { · 18 unmodified lines
903926 #define IRDMAPFINT_OICR_PE_PUSH_M BIT(27)
904927 #define IRDMAPFINT_OICR_PE_CRITERR_M BIT(28)
905928
906−#define IRDMA_QUERY_FPM_MAX_QPS GENMASK_ULL(18, 0)
907−#define IRDMA_QUERY_FPM_MAX_CQS GENMASK_ULL(19, 0)
929+#define IRDMA_QUERY_FPM_LOC_MEM_PAGES GENMASK_ULL(63, 32)
930+#define IRDMA_QUERY_FPM_MAX_QPS GENMASK_ULL(31, 0)
931+#define IRDMA_QUERY_FPM_MAX_CQS GENMASK_ULL(31, 0)
908932 #define IRDMA_QUERY_FPM_FIRST_PE_SD_INDEX GENMASK_ULL(13, 0)
909−#define IRDMA_QUERY_FPM_MAX_PE_SDS GENMASK_ULL(45, 32)
933+#define IRDMA_QUERY_FPM_MAX_PE_SDS GENMASK_ULL(44, 32)
934+#define IRDMA_QUERY_FPM_MAX_PE_SDS_GEN3 GENMASK_ULL(47, 32)
910935 #define IRDMA_QUERY_FPM_MAX_CEQS GENMASK_ULL(9, 0)
936+#define IRDMA_QUERY_FPM_MAX_IRD GENMASK_ULL(53, 50)
911937 #define IRDMA_QUERY_FPM_XFBLOCKSIZE GENMASK_ULL(63, 32)
912938 #define IRDMA_QUERY_FPM_Q1BLOCKSIZE GENMASK_ULL(63, 32)
913939 #define IRDMA_QUERY_FPM_HTMULTIPLIER GENMASK_ULL(19, 16)
@@ -1103,7 +1129,7 @@ enum irdma_alignment { · 189 unmodified lines
11031129 IRDMA_CEQ_ALIGNMENT = 0x100,
11041130 IRDMA_CQ0_ALIGNMENT = 0x100,
11051131 IRDMA_SD_BUF_ALIGNMENT = 0x80,
1106− IRDMA_FEATURE_BUF_ALIGNMENT = 0x8,
1132+ IRDMA_FEATURE_BUF_ALIGNMENT = 0x10,
11071133 };
11081134
11091135 enum icrdma_protocol_used {
drivers/infiniband/hw/irdma/hmc.c+15−3
@@ -5,6 +5,7 @@ · 4 unmodified lines
55 #include "defs.h"
66 #include "type.h"
77 #include "protos.h"
8+#include "virtchnl.h"
89
910 /**
1011 * irdma_find_sd_index_limit - finds segment descriptor index limit
@@ -228,6 +229,10 @@ int irdma_sc_create_hmc_obj(struct irdma_sc_dev *dev, · 217 unmodified lines
228229 bool pd_error = false;
229230 int ret_code = 0;
230231
232+ if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3 &&
233+ dev->hmc_info->hmc_obj[info->rsrc_type].mem_loc == IRDMA_LOC_MEM)
234+ return 0;
235+
231236 if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt)
232237 return -EINVAL;
233238
@@ -330,7 +335,7 @@ static int irdma_finish_del_sd_reg(struct irdma_sc_dev *dev, · 96 unmodified lines
330335 u32 i, sd_idx;
331336 struct irdma_dma_mem *mem;
332337
333− if (!reset)
338+ if (dev->privileged && !reset)
334339 ret_code = irdma_hmc_sd_grp(dev, info->hmc_info,
335340 info->hmc_info->sd_indexes[0],
336341 info->del_sd_cnt, false);
@@ -376,6 +381,9 @@ int irdma_sc_del_hmc_obj(struct irdma_sc_dev *dev, · 39 unmodified lines
376381 u32 i, j;
377382 int ret_code = 0;
378383
384+ if (dev->hmc_info->hmc_obj[info->rsrc_type].mem_loc == IRDMA_LOC_MEM)
385+ return 0;
386+
379387 if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt) {
380388 ibdev_dbg(to_ibdev(dev),
381389 "HMC: error start_idx[%04d] >= [type %04d].cnt[%04d]\n",
@@ -589,7 +597,10 @@ int irdma_add_pd_table_entry(struct irdma_sc_dev *dev, · 207 unmodified lines
589597 pd_entry->sd_index = sd_idx;
590598 pd_entry->valid = true;
591599 pd_table->use_cnt++;
592− irdma_invalidate_pf_hmc_pd(dev, sd_idx, rel_pd_idx);
600+
601+ if (hmc_info->hmc_fn_id < dev->hw_attrs.first_hw_vf_fpm_id &&
602+ dev->privileged)
603+ irdma_invalidate_pf_hmc_pd(dev, sd_idx, rel_pd_idx);
593604 }
594605 pd_entry->bp.use_cnt++;
595606
@@ -640,7 +651,8 @@ int irdma_remove_pd_bp(struct irdma_sc_dev *dev, · 44 unmodified lines
640651 pd_addr = pd_table->pd_page_addr.va;
641652 pd_addr += rel_pd_idx;
642653 memset(pd_addr, 0, sizeof(u64));
643− irdma_invalidate_pf_hmc_pd(dev, sd_idx, idx);
654+ if (dev->privileged && dev->hmc_fn_id == hmc_info->hmc_fn_id)
655+ irdma_invalidate_pf_hmc_pd(dev, sd_idx, idx);
644656
645657 if (!pd_entry->rsrc_pg) {
646658 mem = &pd_entry->bp.addr;
drivers/infiniband/hw/irdma/hmc.h+18−1
@@ -16,11 +16,21 @@ · 15 unmodified lines
1616 #define IRDMA_HMC_PD_BP_BUF_ALIGNMENT 4096
1717 #define IRDMA_FIRST_VF_FPM_ID 8
1818 #define FPM_MULTIPLIER 1024
19+#define IRDMA_OBJ_LOC_MEM_BIT 0x4
20+#define IRDMA_XF_MULTIPLIER 16
21+#define IRDMA_RRF_MULTIPLIER 8
22+#define IRDMA_MIN_PBLE_PAGES 3
23+#define IRDMA_HMC_PAGE_SIZE 2097152
24+#define IRDMA_MIN_MR_PER_QP 4
25+#define IRDMA_MIN_QP_CNT 64
26+#define IRDMA_FSIAV_CNT_MAX 1048576
27+#define IRDMA_MIN_IRD 8
28+#define IRDMA_HMC_MIN_RRF 16
1929
2030 enum irdma_hmc_rsrc_type {
2131 IRDMA_HMC_IW_QP = 0,
2232 IRDMA_HMC_IW_CQ = 1,
23− IRDMA_HMC_IW_RESERVED = 2,
33+ IRDMA_HMC_IW_SRQ = 2,
2434 IRDMA_HMC_IW_HTE = 3,
2535 IRDMA_HMC_IW_ARP = 4,
2636 IRDMA_HMC_IW_APBVT_ENTRY = 5,
@@ -48,11 +58,17 @@ enum irdma_sd_entry_type { · 21 unmodified lines
4858 IRDMA_SD_TYPE_DIRECT = 2,
4959 };
5060
61+enum irdma_hmc_obj_mem {
62+ IRDMA_HOST_MEM = 0,
63+ IRDMA_LOC_MEM = 1,
64+};
65+
5166 struct irdma_hmc_obj_info {
5267 u64 base;
5368 u32 max_cnt;
5469 u32 cnt;
5570 u64 size;
71+ enum irdma_hmc_obj_mem mem_loc;
5672 };
5773
5874 struct irdma_hmc_bp {
@@ -117,6 +133,7 @@ struct irdma_update_sds_info { · 58 unmodified lines
117133 struct irdma_ccq_cqe_info;
118134 struct irdma_hmc_fcn_info {
119135 u32 vf_id;
136+ u8 protocol_used;
120137 u8 free_fcn;
121138 };
122139
drivers/infiniband/hw/irdma/hw.c+216−147
@@ -33,6 +33,7 @@ static struct irdma_rsrc_limits rsrc_limits_table[] = { · 32 unmodified lines
3333 static enum irdma_hmc_rsrc_type iw_hmc_obj_types[] = {
3434 IRDMA_HMC_IW_QP,
3535 IRDMA_HMC_IW_CQ,
36+ IRDMA_HMC_IW_SRQ,
3637 IRDMA_HMC_IW_HTE,
3738 IRDMA_HMC_IW_ARP,
3839 IRDMA_HMC_IW_APBVT_ENTRY,
@@ -134,75 +135,68 @@ static void irdma_process_ceq(struct irdma_pci_f *rf, struct irdma_ceq *ceq) · 95 unmodified lines
134135 static void irdma_set_flush_fields(struct irdma_sc_qp *qp,
135136 struct irdma_aeqe_info *info)
136137 {
138+ struct qp_err_code qp_err;
139+
137140 qp->sq_flush_code = info->sq;
138141 qp->rq_flush_code = info->rq;
139− qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC;
140−
141− switch (info->ae_id) {
142− case IRDMA_AE_AMP_BOUNDS_VIOLATION:
143− case IRDMA_AE_AMP_INVALID_STAG:
144− case IRDMA_AE_AMP_RIGHTS_VIOLATION:
145− case IRDMA_AE_AMP_UNALLOCATED_STAG:
146− case IRDMA_AE_AMP_BAD_PD:
147− case IRDMA_AE_AMP_BAD_QP:
148− case IRDMA_AE_AMP_BAD_STAG_KEY:
149− case IRDMA_AE_AMP_BAD_STAG_INDEX:
150− case IRDMA_AE_AMP_TO_WRAP:
151− case IRDMA_AE_PRIV_OPERATION_DENIED:
152− qp->flush_code = FLUSH_PROT_ERR;
153− qp->event_type = IRDMA_QP_EVENT_ACCESS_ERR;
154− break;
155− case IRDMA_AE_UDA_XMIT_BAD_PD:
156− case IRDMA_AE_WQE_UNEXPECTED_OPCODE:
157− qp->flush_code = FLUSH_LOC_QP_OP_ERR;
158− qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC;
159− break;
160− case IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG:
161− case IRDMA_AE_UDA_XMIT_DGRAM_TOO_SHORT:
162− case IRDMA_AE_UDA_L4LEN_INVALID:
163− case IRDMA_AE_DDP_UBE_INVALID_MO:
164− case IRDMA_AE_DDP_UBE_DDP_MESSAGE_TOO_LONG_FOR_AVAILABLE_BUFFER:
165− qp->flush_code = FLUSH_LOC_LEN_ERR;
166− qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC;
167− break;
168− case IRDMA_AE_AMP_INVALIDATE_NO_REMOTE_ACCESS_RIGHTS:
169− case IRDMA_AE_IB_REMOTE_ACCESS_ERROR:
170− qp->flush_code = FLUSH_REM_ACCESS_ERR;
171− qp->event_type = IRDMA_QP_EVENT_ACCESS_ERR;
172− break;
173− case IRDMA_AE_LLP_SEGMENT_TOO_SMALL:
174− case IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR:
175− case IRDMA_AE_ROCE_RSP_LENGTH_ERROR:
176− case IRDMA_AE_IB_REMOTE_OP_ERROR:
177− qp->flush_code = FLUSH_REM_OP_ERR;
178− qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC;
179− break;
180− case IRDMA_AE_LCE_QP_CATASTROPHIC:
181− qp->flush_code = FLUSH_FATAL_ERR;
182− qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC;
183− break;
184− case IRDMA_AE_IB_RREQ_AND_Q1_FULL:
185− qp->flush_code = FLUSH_GENERAL_ERR;
186− break;
187− case IRDMA_AE_LLP_TOO_MANY_RETRIES:
188− qp->flush_code = FLUSH_RETRY_EXC_ERR;
189− qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC;
190− break;
191− case IRDMA_AE_AMP_MWBIND_INVALID_RIGHTS:
192− case IRDMA_AE_AMP_MWBIND_BIND_DISABLED:
193− case IRDMA_AE_AMP_MWBIND_INVALID_BOUNDS:
194− case IRDMA_AE_AMP_MWBIND_VALID_STAG:
195− qp->flush_code = FLUSH_MW_BIND_ERR;
196− qp->event_type = IRDMA_QP_EVENT_ACCESS_ERR;
197− break;
198− case IRDMA_AE_IB_INVALID_REQUEST:
199− qp->flush_code = FLUSH_REM_INV_REQ_ERR;
200− qp->event_type = IRDMA_QP_EVENT_REQ_ERR;
201− break;
202− default:
203− qp->flush_code = FLUSH_GENERAL_ERR;
204− qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC;
205− break;
142+ if (qp->qp_uk.uk_attrs->hw_rev >= IRDMA_GEN_3) {
143+ if (info->sq) {
144+ qp->err_sq_idx_valid = true;
145+ qp->err_sq_idx = info->wqe_idx;
146+ }
147+ if (info->rq) {
148+ qp->err_rq_idx_valid = true;
149+ qp->err_rq_idx = info->wqe_idx;
150+ }
151+ }
152+
153+ qp_err = irdma_ae_to_qp_err_code(info->ae_id);
154+ qp->flush_code = qp_err.flush_code;
155+ qp->event_type = qp_err.event_type;
156+}
157+
158+/**
159+ * irdma_complete_cqp_request - perform post-completion cleanup
160+ * @cqp: device CQP
161+ * @cqp_request: CQP request
162+ *
163+ * Mark CQP request as done, wake up waiting thread or invoke
164+ * callback function and release/free CQP request.
165+ */
166+static void irdma_complete_cqp_request(struct irdma_cqp *cqp,
167+ struct irdma_cqp_request *cqp_request)
168+{
169+ if (cqp_request->waiting) {
170+ WRITE_ONCE(cqp_request->request_done, true);
171+ wake_up(&cqp_request->waitq);
172+ } else if (cqp_request->callback_fcn) {
173+ cqp_request->callback_fcn(cqp_request);
174+ }
175+ irdma_put_cqp_request(cqp, cqp_request);
176+}
177+
178+/**
179+ * irdma_process_ae_def_cmpl - handle IRDMA_AE_CQP_DEFERRED_COMPLETE event
180+ * @rf: RDMA PCI function
181+ * @info: AEQ entry info
182+ */
183+static void irdma_process_ae_def_cmpl(struct irdma_pci_f *rf,
184+ struct irdma_aeqe_info *info)
185+{
186+ u32 sw_def_info;
187+ u64 scratch;
188+
189+ irdma_cqp_ce_handler(rf, &rf->ccq.sc_cq);
190+
191+ irdma_sc_cqp_def_cmpl_ae_handler(&rf->sc_dev, info, true,
192+ &scratch, &sw_def_info);
193+ while (scratch) {
194+ struct irdma_cqp_request *cqp_request =
195+ (struct irdma_cqp_request *)(uintptr_t)scratch;
196+
197+ irdma_complete_cqp_request(&rf->cqp, cqp_request);
198+ irdma_sc_cqp_def_cmpl_ae_handler(&rf->sc_dev, info, false,
199+ &scratch, &sw_def_info);
206200 }
207201 }
208202
@@ -223,6 +217,7 @@ static void irdma_process_aeq(struct irdma_pci_f *rf) · 14 unmodified lines
223217 struct irdma_sc_qp *qp = NULL;
224218 struct irdma_qp_host_ctx_info *ctx_info = NULL;
225219 struct irdma_device *iwdev = rf->iwdev;
220+ struct irdma_sc_srq *srq;
226221 unsigned long flags;
227222
228223 u32 aeqcnt = 0;
@@ -236,6 +231,13 @@ static void irdma_process_aeq(struct irdma_pci_f *rf) · 7 unmodified lines
236231 if (ret)
237232 break;
238233
234+ if (info->aeqe_overflow) {
235+ ibdev_err(&iwdev->ibdev, "AEQ has overflowed\n");
236+ rf->reset = true;
237+ rf->gen_ops.request_reset(rf);
238+ return;
239+ }
240+
239241 aeqcnt++;
240242 ibdev_dbg(&iwdev->ibdev,
241243 "AEQ: ae_id = 0x%x bool qp=%d qp_id = %d tcp_state=%d iwarp_state=%d ae_src=%d\n",
@@ -266,9 +268,12 @@ static void irdma_process_aeq(struct irdma_pci_f *rf) · 24 unmodified lines
266268 if (info->ae_id != IRDMA_AE_QP_SUSPEND_COMPLETE)
267269 iwqp->last_aeq = info->ae_id;
268270 spin_unlock_irqrestore(&iwqp->lock, flags);
269− ctx_info = &iwqp->ctx_info;
271+ } else if (info->srq) {
272+ if (info->ae_id != IRDMA_AE_SRQ_LIMIT)
273+ continue;
270274 } else {
271− if (info->ae_id != IRDMA_AE_CQ_OPERATION_ERROR)
275+ if (info->ae_id != IRDMA_AE_CQ_OPERATION_ERROR &&
276+ info->ae_id != IRDMA_AE_CQP_DEFERRED_COMPLETE)
272277 continue;
273278 }
274279
@@ -363,6 +368,18 @@ static void irdma_process_aeq(struct irdma_pci_f *rf) · 88 unmodified lines
363368 }
364369 irdma_cq_rem_ref(&iwcq->ibcq);
365370 break;
371+ case IRDMA_AE_SRQ_LIMIT:
372+ srq = (struct irdma_sc_srq *)(uintptr_t)info->compl_ctx;
373+ irdma_srq_event(srq);
374+ break;
375+ case IRDMA_AE_SRQ_CATASTROPHIC_ERROR:
376+ break;
377+ case IRDMA_AE_CQP_DEFERRED_COMPLETE:
378+ /* Remove completed CQP requests from pending list
379+ * and notify about those CQP ops completion.
380+ */
381+ irdma_process_ae_def_cmpl(rf, info);
382+ break;
366383 case IRDMA_AE_RESET_NOT_SENT:
367384 case IRDMA_AE_LLP_DOUBT_REACHABILITY:
368385 case IRDMA_AE_RESOURCE_EXHAUSTION:
@@ -389,13 +406,18 @@ static void irdma_process_aeq(struct irdma_pci_f *rf) · 20 unmodified lines
389406 case IRDMA_AE_LCE_FUNCTION_CATASTROPHIC:
390407 case IRDMA_AE_LLP_TOO_MANY_RNRS:
391408 case IRDMA_AE_LCE_CQ_CATASTROPHIC:
409+ case IRDMA_AE_REMOTE_QP_CATASTROPHIC:
410+ case IRDMA_AE_LOCAL_QP_CATASTROPHIC:
411+ case IRDMA_AE_RCE_QP_CATASTROPHIC:
392412 case IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG:
393413 default:
394414 ibdev_err(&iwdev->ibdev, "abnormal ae_id = 0x%x bool qp=%d qp_id = %d, ae_src=%d\n",
395415 info->ae_id, info->qp, info->qp_cq_id, info->ae_src);
396− if (rdma_protocol_roce(&iwdev->ibdev, 1)) {
397− ctx_info->roce_info->err_rq_idx_valid = info->rq;
398− if (info->rq) {
416+ ctx_info = &iwqp->ctx_info;
417+ if (rdma_protocol_roce(&iwqp->iwdev->ibdev, 1)) {
418+ ctx_info->roce_info->err_rq_idx_valid =
419+ ctx_info->srq_valid ? false : info->err_rq_idx_valid;
420+ if (ctx_info->roce_info->err_rq_idx_valid) {
399421 ctx_info->roce_info->err_rq_idx = info->wqe_idx;
400422 irdma_sc_qp_setctx_roce(&iwqp->sc_qp, iwqp->host_ctx.va,
401423 ctx_info);
@@ -599,6 +621,8 @@ static void irdma_destroy_cqp(struct irdma_pci_f *rf) · 197 unmodified lines
599621 dma_free_coherent(dev->hw->device, cqp->sq.size, cqp->sq.va,
600622 cqp->sq.pa);
601623 cqp->sq.va = NULL;
624+ kfree(cqp->oop_op_array);
625+ cqp->oop_op_array = NULL;
602626 kfree(cqp->scratch_array);
603627 cqp->scratch_array = NULL;
604628 kfree(cqp->cqp_requests);
@@ -631,7 +655,9 @@ static void irdma_destroy_aeq(struct irdma_pci_f *rf) · 26 unmodified lines
631655 int status = -EBUSY;
632656
633657 if (!rf->msix_shared) {
634− rf->sc_dev.irq_ops->irdma_cfg_aeq(&rf->sc_dev, rf->iw_msixtbl->idx, false);
658+ if (rf->sc_dev.privileged)
659+ rf->sc_dev.irq_ops->irdma_cfg_aeq(&rf->sc_dev,
660+ rf->iw_msixtbl->idx, false);
635661 irdma_destroy_irq(rf, rf->iw_msixtbl, rf);
636662 }
637663 if (rf->reset)
@@ -697,9 +723,10 @@ static void irdma_del_ceq_0(struct irdma_pci_f *rf) · 59 unmodified lines
697723
698724 if (rf->msix_shared) {
699725 msix_vec = &rf->iw_msixtbl[0];
700− rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev,
701− msix_vec->ceq_id,
702− msix_vec->idx, false);
726+ if (rf->sc_dev.privileged)
727+ rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev,
728+ msix_vec->ceq_id,
729+ msix_vec->idx, false);
703730 irdma_destroy_irq(rf, msix_vec, rf);
704731 } else {
705732 msix_vec = &rf->iw_msixtbl[1];
@@ -730,8 +757,10 @@ static void irdma_del_ceqs(struct irdma_pci_f *rf) · 24 unmodified lines
730757 msix_vec = &rf->iw_msixtbl[2];
731758
732759 for (i = 1; i < rf->ceqs_count; i++, msix_vec++, iwceq++) {
733− rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev, msix_vec->ceq_id,
734− msix_vec->idx, false);
760+ if (rf->sc_dev.privileged)
761+ rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev,
762+ msix_vec->ceq_id,
763+ msix_vec->idx, false);
735764 irdma_destroy_irq(rf, msix_vec, iwceq);
736765 irdma_cqp_ceq_cmd(&rf->sc_dev, &iwceq->sc_ceq,
737766 IRDMA_OP_CEQ_DESTROY);
@@ -942,6 +971,13 @@ static int irdma_create_cqp(struct irdma_pci_f *rf) · 204 unmodified lines
942971 goto err_scratch;
943972 }
944973
974+ cqp->oop_op_array = kcalloc(sqsize, sizeof(*cqp->oop_op_array),
975+ GFP_KERNEL);
976+ if (!cqp->oop_op_array) {
977+ status = -ENOMEM;
978+ goto err_oop;
979+ }
980+ cqp_init_info.ooo_op_array = cqp->oop_op_array;
945981 dev->cqp = &cqp->sc_cqp;
946982 dev->cqp->dev = dev;
947983 cqp->sq.size = ALIGN(sizeof(struct irdma_cqp_sq_wqe) * sqsize,
@@ -978,6 +1014,10 @@ static int irdma_create_cqp(struct irdma_pci_f *rf) · 30 unmodified lines
9781014 case IRDMA_GEN_2:
9791015 cqp_init_info.hw_maj_ver = IRDMA_CQPHC_HW_MAJVER_GEN_2;
9801016 break;
1017+ case IRDMA_GEN_3:
1018+ cqp_init_info.hw_maj_ver = IRDMA_CQPHC_HW_MAJVER_GEN_3;
1019+ cqp_init_info.ts_override = 1;
1020+ break;
9811021 }
9821022 status = irdma_sc_cqp_init(dev->cqp, &cqp_init_info);
9831023 if (status) {
@@ -1012,6 +1052,9 @@ err_ctx: · 28 unmodified lines
10121052 cqp->sq.va, cqp->sq.pa);
10131053 cqp->sq.va = NULL;
10141054 err_sq:
1055+ kfree(cqp->oop_op_array);
1056+ cqp->oop_op_array = NULL;
1057+err_oop:
10151058 kfree(cqp->scratch_array);
10161059 cqp->scratch_array = NULL;
10171060 err_scratch:
@@ -1033,13 +1076,15 @@ static int irdma_create_ccq(struct irdma_pci_f *rf) · 15 unmodified lines
10331076 struct irdma_sc_dev *dev = &rf->sc_dev;
10341077 struct irdma_ccq_init_info info = {};
10351078 struct irdma_ccq *ccq = &rf->ccq;
1079+ int ccq_size;
10361080 int status;
10371081
10381082 dev->ccq = &ccq->sc_cq;
10391083 dev->ccq->dev = dev;
10401084 info.dev = dev;
1085+ ccq_size = (rf->rdma_ver >= IRDMA_GEN_3) ? IW_GEN_3_CCQ_SIZE : IW_CCQ_SIZE;
10411086 ccq->shadow_area.size = sizeof(struct irdma_cq_shadow_area);
1042− ccq->mem_cq.size = ALIGN(sizeof(struct irdma_cqe) * IW_CCQ_SIZE,
1087+ ccq->mem_cq.size = ALIGN(sizeof(struct irdma_cqe) * ccq_size,
10431088 IRDMA_CQ0_ALIGNMENT);
10441089 ccq->mem_cq.va = dma_alloc_coherent(dev->hw->device, ccq->mem_cq.size,
10451090 &ccq->mem_cq.pa, GFP_KERNEL);
@@ -1056,7 +1101,7 @@ static int irdma_create_ccq(struct irdma_pci_f *rf) · 10 unmodified lines
10561101 /* populate the ccq init info */
10571102 info.cq_base = ccq->mem_cq.va;
10581103 info.cq_pa = ccq->mem_cq.pa;
1059− info.num_elem = IW_CCQ_SIZE;
1104+ info.num_elem = ccq_size;
10601105 info.shadow_area = ccq->shadow_area.va;
10611106 info.shadow_area_pa = ccq->shadow_area.pa;
10621107 info.ceqe_mask = false;
@@ -1140,9 +1185,13 @@ static int irdma_cfg_ceq_vector(struct irdma_pci_f *rf, struct irdma_ceq *iwceq, · 77 unmodified lines
11401185 }
11411186
11421187 msix_vec->ceq_id = ceq_id;
1143− rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev, ceq_id, msix_vec->idx, true);
1144−
1145− return 0;
1188+ if (rf->sc_dev.privileged)
1189+ rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev, ceq_id,
1190+ msix_vec->idx, true);
1191+ else
1192+ status = irdma_vchnl_req_ceq_vec_map(&rf->sc_dev, ceq_id,
1193+ msix_vec->idx);
1194+ return status;
11461195 }
11471196
11481197 /**
@@ -1155,7 +1204,7 @@ static int irdma_cfg_ceq_vector(struct irdma_pci_f *rf, struct irdma_ceq *iwceq, · 6 unmodified lines
11551204 static int irdma_cfg_aeq_vector(struct irdma_pci_f *rf)
11561205 {
11571206 struct irdma_msix_vector *msix_vec = rf->iw_msixtbl;
1158− u32 ret = 0;
1207+ int ret = 0;
11591208
11601209 if (!rf->msix_shared) {
11611210 snprintf(msix_vec->name, sizeof(msix_vec->name) - 1,
@@ -1166,12 +1215,16 @@ static int irdma_cfg_aeq_vector(struct irdma_pci_f *rf) · 4 unmodified lines
11661215 }
11671216 if (ret) {
11681217 ibdev_dbg(&rf->iwdev->ibdev, "ERR: aeq irq config fail\n");
1169− return -EINVAL;
1218+ return ret;
11701219 }
11711220
1172− rf->sc_dev.irq_ops->irdma_cfg_aeq(&rf->sc_dev, msix_vec->idx, true);
1221+ if (rf->sc_dev.privileged)
1222+ rf->sc_dev.irq_ops->irdma_cfg_aeq(&rf->sc_dev, msix_vec->idx,
1223+ true);
1224+ else
1225+ ret = irdma_vchnl_req_aeq_vec_map(&rf->sc_dev, msix_vec->idx);
11731226
1174− return 0;
1227+ return ret;
11751228 }
11761229
11771230 /**
@@ -1179,13 +1232,13 @@ static int irdma_cfg_aeq_vector(struct irdma_pci_f *rf) · 1 unmodified line
11791232 * @rf: RDMA PCI function
11801233 * @iwceq: pointer to the ceq resources to be created
11811234 * @ceq_id: the id number of the iwceq
1182− * @vsi: SC vsi struct
1235+ * @vsi_idx: vsi idx
11831236 *
11841237 * Return 0, if the ceq and the resources associated with it
11851238 * are successfully created, otherwise return error
11861239 */
11871240 static int irdma_create_ceq(struct irdma_pci_f *rf, struct irdma_ceq *iwceq,
1188− u32 ceq_id, struct irdma_sc_vsi *vsi)
1241+ u32 ceq_id, u16 vsi_idx)
11891242 {
11901243 int status;
11911244 struct irdma_ceq_init_info info = {};
@@ -1209,7 +1262,7 @@ static int irdma_create_ceq(struct irdma_pci_f *rf, struct irdma_ceq *iwceq, · 17 unmodified lines
12091262 info.elem_cnt = ceq_size;
12101263 iwceq->sc_ceq.ceq_id = ceq_id;
12111264 info.dev = dev;
1212− info.vsi = vsi;
1265+ info.vsi_idx = vsi_idx;
12131266 status = irdma_sc_ceq_init(&iwceq->sc_ceq, &info);
12141267 if (!status) {
12151268 if (dev->ceq_valid)
@@ -1252,7 +1305,7 @@ static int irdma_setup_ceq_0(struct irdma_pci_f *rf) · 36 unmodified lines
12521305 }
12531306
12541307 iwceq = &rf->ceqlist[0];
1255− status = irdma_create_ceq(rf, iwceq, 0, &rf->default_vsi);
1308+ status = irdma_create_ceq(rf, iwceq, 0, rf->default_vsi.vsi_idx);
12561309 if (status) {
12571310 ibdev_dbg(&rf->iwdev->ibdev, "ERR: create ceq status = %d\n",
12581311 status);
@@ -1287,13 +1340,13 @@ exit: · 28 unmodified lines
12871340 /**
12881341 * irdma_setup_ceqs - manage the device ceq's and their interrupt resources
12891342 * @rf: RDMA PCI function
1290− * @vsi: VSI structure for this CEQ
1343+ * @vsi_idx: vsi_idx for this CEQ
12911344 *
12921345 * Allocate a list for all device completion event queues
12931346 * Create the ceq's and configure their msix interrupt vectors
12941347 * Return 0, if ceqs are successfully set up, otherwise return error
12951348 */
1296−static int irdma_setup_ceqs(struct irdma_pci_f *rf, struct irdma_sc_vsi *vsi)
1349+static int irdma_setup_ceqs(struct irdma_pci_f *rf, u16 vsi_idx)
12971350 {
12981351 u32 i;
12991352 u32 ceq_id;
@@ -1306,7 +1359,7 @@ static int irdma_setup_ceqs(struct irdma_pci_f *rf, struct irdma_sc_vsi *vsi) · 6 unmodified lines
13061359 i = (rf->msix_shared) ? 1 : 2;
13071360 for (ceq_id = 1; i < num_ceqs; i++, ceq_id++) {
13081361 iwceq = &rf->ceqlist[ceq_id];
1309− status = irdma_create_ceq(rf, iwceq, ceq_id, vsi);
1362+ status = irdma_create_ceq(rf, iwceq, ceq_id, vsi_idx);
13101363 if (status) {
13111364 ibdev_dbg(&rf->iwdev->ibdev,
13121365 "ERR: create ceq status = %d\n", status);
@@ -1387,7 +1440,10 @@ static int irdma_create_aeq(struct irdma_pci_f *rf) · 74 unmodified lines
13871440 aeq_size = multiplier * hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt +
13881441 hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt;
13891442 aeq_size = min(aeq_size, dev->hw_attrs.max_hw_aeq_size);
1390−
1443+ /* GEN_3 does not support virtual AEQ. Cap at max Kernel alloc size */
1444+ if (rf->rdma_ver == IRDMA_GEN_3)
1445+ aeq_size = min(aeq_size, (u32)((PAGE_SIZE << MAX_PAGE_ORDER) /
1446+ sizeof(struct irdma_sc_aeqe)));
13911447 aeq->mem.size = ALIGN(sizeof(struct irdma_sc_aeqe) * aeq_size,
13921448 IRDMA_AEQ_ALIGNMENT);
13931449 aeq->mem.va = dma_alloc_coherent(dev->hw->device, aeq->mem.size,
@@ -1395,6 +1451,8 @@ static int irdma_create_aeq(struct irdma_pci_f *rf) · 1 unmodified line
13951451 GFP_KERNEL | __GFP_NOWARN);
13961452 if (aeq->mem.va)
13971453 goto skip_virt_aeq;
1454+ else if (rf->rdma_ver == IRDMA_GEN_3)
1455+ return -ENOMEM;
13981456
13991457 /* physically mapped aeq failed. setup virtual aeq */
14001458 status = irdma_create_virt_aeq(rf, aeq_size);
@@ -1569,6 +1627,8 @@ static void irdma_del_init_mem(struct irdma_pci_f *rf) · 168 unmodified lines
15691627 {
15701628 struct irdma_sc_dev *dev = &rf->sc_dev;
15711629
1630+ if (!rf->sc_dev.privileged)
1631+ irdma_vchnl_req_put_hmc_fcn(&rf->sc_dev);
15721632 kfree(dev->hmc_info->sd_table.sd_entry);
15731633 dev->hmc_info->sd_table.sd_entry = NULL;
15741634 vfree(rf->mem_rsrc);
@@ -1635,6 +1695,7 @@ static int irdma_initialize_dev(struct irdma_pci_f *rf) · 60 unmodified lines
16351695
16361696 info.bar0 = rf->hw.hw_addr;
16371697 info.hmc_fn_id = rf->pf_id;
1698+ info.protocol_used = rf->protocol_used;
16381699 info.hw = &rf->hw;
16391700 status = irdma_sc_dev_init(rf->rdma_ver, &rf->sc_dev, &info);
16401701 if (status)
@@ -1665,9 +1726,6 @@ void irdma_rt_deinit_hw(struct irdma_device *iwdev) · 24 unmodified lines
16651726 irdma_del_local_mac_entry(iwdev->rf,
16661727 (u8)iwdev->mac_ip_table_idx);
16671728 fallthrough;
1668− case AEQ_CREATED:
1669− case PBLE_CHUNK_MEM:
1670− case CEQS_CREATED:
16711729 case IEQ_CREATED:
16721730 if (!iwdev->roce_mode)
16731731 irdma_puda_dele_rsrc(&iwdev->vsi, IRDMA_PUDA_RSRC_TYPE_IEQ,
@@ -1740,7 +1798,9 @@ static void irdma_get_used_rsrc(struct irdma_device *iwdev) · 66 unmodified lines
17401798 iwdev->rf->used_qps = find_first_zero_bit(iwdev->rf->allocated_qps,
17411799 iwdev->rf->max_qp);
17421800 iwdev->rf->used_cqs = find_first_zero_bit(iwdev->rf->allocated_cqs,
1743− iwdev->rf->max_cq);
1801+ iwdev->rf->max_cq);
1802+ iwdev->rf->used_srqs = find_first_zero_bit(iwdev->rf->allocated_srqs,
1803+ iwdev->rf->max_srq);
17441804 iwdev->rf->used_mrs = find_first_zero_bit(iwdev->rf->allocated_mrs,
17451805 iwdev->rf->max_mr);
17461806 }
@@ -1750,13 +1810,17 @@ void irdma_ctrl_deinit_hw(struct irdma_pci_f *rf) · 3 unmodified lines
17501810 enum init_completion_state state = rf->init_state;
17511811
17521812 rf->init_state = INVALID_STATE;
1753− if (rf->rsrc_created) {
1813+
1814+ switch (state) {
1815+ case AEQ_CREATED:
17541816 irdma_destroy_aeq(rf);
1817+ fallthrough;
1818+ case PBLE_CHUNK_MEM:
17551819 irdma_destroy_pble_prm(rf->pble_rsrc);
1820+ fallthrough;
1821+ case CEQS_CREATED:
17561822 irdma_del_ceqs(rf);
1757− rf->rsrc_created = false;
1758− }
1759− switch (state) {
1823+ fallthrough;
17601824 case CEQ0_CREATED:
17611825 irdma_del_ceq_0(rf);
17621826 fallthrough;
@@ -1835,32 +1899,6 @@ int irdma_rt_init_hw(struct irdma_device *iwdev, · 72 unmodified lines
18351899 break;
18361900 iwdev->init_state = IEQ_CREATED;
18371901 }
1838− if (!rf->rsrc_created) {
1839− status = irdma_setup_ceqs(rf, &iwdev->vsi);
1840− if (status)
1841− break;
1842−
1843− iwdev->init_state = CEQS_CREATED;
1844−
1845− status = irdma_hmc_init_pble(&rf->sc_dev,
1846− rf->pble_rsrc);
1847− if (status) {
1848− irdma_del_ceqs(rf);
1849− break;
1850− }
1851−
1852− iwdev->init_state = PBLE_CHUNK_MEM;
1853−
1854− status = irdma_setup_aeq(rf);
1855− if (status) {
1856− irdma_destroy_pble_prm(rf->pble_rsrc);
1857− irdma_del_ceqs(rf);
1858− break;
1859− }
1860− iwdev->init_state = AEQ_CREATED;
1861− rf->rsrc_created = true;
1862− }
1863−
18641902 if (iwdev->rf->sc_dev.hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_1)
18651903 irdma_alloc_set_mac(iwdev);
18661904 irdma_add_ip(iwdev);
@@ -1907,6 +1945,13 @@ int irdma_ctrl_init_hw(struct irdma_pci_f *rf) · 40 unmodified lines
19071945 break;
19081946 rf->init_state = CQP_CREATED;
19091947
1948+ dev->feature_info[IRDMA_FEATURE_FW_INFO] = IRDMA_FW_VER_DEFAULT;
1949+ if (rf->rdma_ver != IRDMA_GEN_1) {
1950+ status = irdma_get_rdma_features(dev);
1951+ if (status)
1952+ break;
1953+ }
1954+
19101955 status = irdma_hmc_setup(rf);
19111956 if (status)
19121957 break;
@@ -1922,13 +1967,6 @@ int irdma_ctrl_init_hw(struct irdma_pci_f *rf) · 9 unmodified lines
19221967 break;
19231968 rf->init_state = CCQ_CREATED;
19241969
1925− dev->feature_info[IRDMA_FEATURE_FW_INFO] = IRDMA_FW_VER_DEFAULT;
1926− if (rf->rdma_ver != IRDMA_GEN_1) {
1927− status = irdma_get_rdma_features(dev);
1928− if (status)
1929− break;
1930− }
1931−
19321970 status = irdma_setup_ceq_0(rf);
19331971 if (status)
19341972 break;
@@ -1942,6 +1980,25 @@ int irdma_ctrl_init_hw(struct irdma_pci_f *rf) · 7 unmodified lines
19421980 }
19431981 INIT_WORK(&rf->cqp_cmpl_work, cqp_compl_worker);
19441982 irdma_sc_ccq_arm(dev->ccq);
1983+
1984+ status = irdma_setup_ceqs(rf, rf->iwdev ? rf->iwdev->vsi_num : 0);
1985+ if (status)
1986+ break;
1987+
1988+ rf->init_state = CEQS_CREATED;
1989+
1990+ status = irdma_hmc_init_pble(&rf->sc_dev,
1991+ rf->pble_rsrc);
1992+ if (status)
1993+ break;
1994+
1995+ rf->init_state = PBLE_CHUNK_MEM;
1996+
1997+ status = irdma_setup_aeq(rf);
1998+ if (status)
1999+ break;
2000+ rf->init_state = AEQ_CREATED;
2001+
19452002 return 0;
19462003 } while (0);
19472004
@@ -1960,7 +2017,8 @@ static void irdma_set_hw_rsrc(struct irdma_pci_f *rf) · 12 unmodified lines
19602017 rf->allocated_qps = (void *)(rf->mem_rsrc +
19612018 (sizeof(struct irdma_arp_entry) * rf->arp_table_size));
19622019 rf->allocated_cqs = &rf->allocated_qps[BITS_TO_LONGS(rf->max_qp)];
1963− rf->allocated_mrs = &rf->allocated_cqs[BITS_TO_LONGS(rf->max_cq)];
2020+ rf->allocated_srqs = &rf->allocated_cqs[BITS_TO_LONGS(rf->max_cq)];
2021+ rf->allocated_mrs = &rf->allocated_srqs[BITS_TO_LONGS(rf->max_srq)];
19642022 rf->allocated_pds = &rf->allocated_mrs[BITS_TO_LONGS(rf->max_mr)];
19652023 rf->allocated_ahs = &rf->allocated_pds[BITS_TO_LONGS(rf->max_pd)];
19662024 rf->allocated_mcgs = &rf->allocated_ahs[BITS_TO_LONGS(rf->max_ah)];
@@ -1988,12 +2046,14 @@ static u32 irdma_calc_mem_rsrc_size(struct irdma_pci_f *rf) · 21 unmodified lines
19882046 rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_qp);
19892047 rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_mr);
19902048 rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_cq);
2049+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_srq);
19912050 rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_pd);
19922051 rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->arp_table_size);
19932052 rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_ah);
19942053 rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_mcg);
19952054 rsrc_size += sizeof(struct irdma_qp **) * rf->max_qp;
19962055 rsrc_size += sizeof(struct irdma_cq **) * rf->max_cq;
2056+ rsrc_size += sizeof(struct irdma_srq **) * rf->max_srq;
19972057
19982058 return rsrc_size;
19992059 }
@@ -2021,6 +2081,7 @@ u32 irdma_initialize_hw_rsrc(struct irdma_pci_f *rf) · 21 unmodified lines
20212081 rf->max_qp = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt;
20222082 rf->max_mr = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt;
20232083 rf->max_cq = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt;
2084+ rf->max_srq = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_SRQ].cnt;
20242085 rf->max_pd = rf->sc_dev.hw_attrs.max_hw_pds;
20252086 rf->arp_table_size = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_ARP].cnt;
20262087 rf->max_ah = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].cnt;
@@ -2040,6 +2101,7 @@ u32 irdma_initialize_hw_rsrc(struct irdma_pci_f *rf) · 13 unmodified lines
20402101 set_bit(0, rf->allocated_mrs);
20412102 set_bit(0, rf->allocated_qps);
20422103 set_bit(0, rf->allocated_cqs);
2104+ set_bit(0, rf->allocated_srqs);
20432105 set_bit(0, rf->allocated_pds);
20442106 set_bit(0, rf->allocated_arps);
20452107 set_bit(0, rf->allocated_ahs);
@@ -2100,15 +2162,16 @@ void irdma_cqp_ce_handler(struct irdma_pci_f *rf, struct irdma_sc_cq *cq) · 54 unmodified lines
21002162 cqp_request->compl_info.op_ret_val = info.op_ret_val;
21012163 cqp_request->compl_info.error = info.error;
21022164
2103− if (cqp_request->waiting) {
2104− WRITE_ONCE(cqp_request->request_done, true);
2105− wake_up(&cqp_request->waitq);
2106− irdma_put_cqp_request(&rf->cqp, cqp_request);
2107− } else {
2108− if (cqp_request->callback_fcn)
2109− cqp_request->callback_fcn(cqp_request);
2110− irdma_put_cqp_request(&rf->cqp, cqp_request);
2111− }
2165+ /*
2166+ * If this is deferred or pending completion, then mark
2167+ * CQP request as pending to not block the CQ, but don't
2168+ * release CQP request, as it is still on the OOO list.
2169+ */
2170+ if (info.pending)
2171+ cqp_request->pending = true;
2172+ else
2173+ irdma_complete_cqp_request(&rf->cqp,
2174+ cqp_request);
21122175 }
21132176
21142177 cqe_count++;
@@ -2718,7 +2781,9 @@ void irdma_flush_wqes(struct irdma_qp *iwqp, u32 flush_mask) · 603 unmodified lines
27182781 struct irdma_pci_f *rf = iwqp->iwdev->rf;
27192782 u8 flush_code = iwqp->sc_qp.flush_code;
27202783
2721− if (!(flush_mask & IRDMA_FLUSH_SQ) && !(flush_mask & IRDMA_FLUSH_RQ))
2784+ if ((!(flush_mask & IRDMA_FLUSH_SQ) &&
2785+ !(flush_mask & IRDMA_FLUSH_RQ)) ||
2786+ ((flush_mask & IRDMA_REFLUSH) && rf->rdma_ver >= IRDMA_GEN_3))
27222787 return;
27232788
27242789 /* Set flush info fields*/
@@ -2731,6 +2796,10 @@ void irdma_flush_wqes(struct irdma_qp *iwqp, u32 flush_mask) · 6 unmodified lines
27312796 info.rq_major_code = IRDMA_FLUSH_MAJOR_ERR;
27322797 info.rq_minor_code = FLUSH_GENERAL_ERR;
27332798 info.userflushcode = true;
2799+ info.err_sq_idx_valid = iwqp->sc_qp.err_sq_idx_valid;
2800+ info.err_sq_idx = iwqp->sc_qp.err_sq_idx;
2801+ info.err_rq_idx_valid = iwqp->sc_qp.err_rq_idx_valid;
2802+ info.err_rq_idx = iwqp->sc_qp.err_rq_idx;
27342803
27352804 if (flush_mask & IRDMA_REFLUSH) {
27362805 if (info.sq)
drivers/infiniband/hw/irdma/i40iw_hw.c+2−0
@@ -85,6 +85,7 @@ static u64 i40iw_masks[IRDMA_MAX_MASKS] = { · 84 unmodified lines
8585 I40E_CQPSQ_CQ_CEQID,
8686 I40E_CQPSQ_CQ_CQID,
8787 I40E_COMMIT_FPM_CQCNT,
88+ I40E_CQPSQ_UPESD_HMCFNID,
8889 };
8990
9091 static u64 i40iw_shifts[IRDMA_MAX_SHIFTS] = {
@@ -94,6 +95,7 @@ static u64 i40iw_shifts[IRDMA_MAX_SHIFTS] = { · 3 unmodified lines
9495 I40E_CQPSQ_CQ_CEQID_S,
9596 I40E_CQPSQ_CQ_CQID_S,
9697 I40E_COMMIT_FPM_CQCNT_S,
98+ I40E_CQPSQ_UPESD_HMCFNID_S,
9799 };
98100
99101 /**
drivers/infiniband/hw/irdma/i40iw_hw.h+2−0
@@ -123,6 +123,8 @@ · 122 unmodified lines
123123 #define I40E_CQPSQ_CQ_CQID GENMASK_ULL(15, 0)
124124 #define I40E_COMMIT_FPM_CQCNT_S 0
125125 #define I40E_COMMIT_FPM_CQCNT GENMASK_ULL(17, 0)
126+#define I40E_CQPSQ_UPESD_HMCFNID_S 0
127+#define I40E_CQPSQ_UPESD_HMCFNID GENMASK_ULL(5, 0)
126128
127129 #define I40E_VSIQF_CTL(_VSI) (0x0020D800 + ((_VSI) * 4))
128130
drivers/infiniband/hw/irdma/i40iw_if.c+3−0
@@ -75,6 +75,9 @@ static void i40iw_fill_device_info(struct irdma_device *iwdev, struct i40e_info · 74 unmodified lines
7575 struct irdma_pci_f *rf = iwdev->rf;
7676
7777 rf->rdma_ver = IRDMA_GEN_1;
78+ rf->sc_dev.hw = &rf->hw;
79+ rf->sc_dev.hw_attrs.uk_attrs.hw_rev = IRDMA_GEN_1;
80+ rf->sc_dev.privileged = true;
7881 rf->gen_ops.request_reset = i40iw_request_reset;
7982 rf->pcidev = cdev_info->pcidev;
8083 rf->pf_id = cdev_info->fid;
drivers/infiniband/hw/irdma/icrdma_hw.c+3−0
@@ -38,6 +38,7 @@ static u64 icrdma_masks[IRDMA_MAX_MASKS] = { · 37 unmodified lines
3838 ICRDMA_CQPSQ_CQ_CEQID,
3939 ICRDMA_CQPSQ_CQ_CQID,
4040 ICRDMA_COMMIT_FPM_CQCNT,
41+ ICRDMA_CQPSQ_UPESD_HMCFNID,
4142 };
4243
4344 static u64 icrdma_shifts[IRDMA_MAX_SHIFTS] = {
@@ -47,6 +48,7 @@ static u64 icrdma_shifts[IRDMA_MAX_SHIFTS] = { · 3 unmodified lines
4748 ICRDMA_CQPSQ_CQ_CEQID_S,
4849 ICRDMA_CQPSQ_CQ_CQID_S,
4950 ICRDMA_COMMIT_FPM_CQCNT_S,
51+ ICRDMA_CQPSQ_UPESD_HMCFNID_S,
5052 };
5153
5254 /**
@@ -194,6 +196,7 @@ void icrdma_init_hw(struct irdma_sc_dev *dev) · 141 unmodified lines
194196 dev->hw_attrs.max_hw_ord = ICRDMA_MAX_ORD_SIZE;
195197 dev->hw_attrs.max_stat_inst = ICRDMA_MAX_STATS_COUNT;
196198 dev->hw_attrs.max_stat_idx = IRDMA_HW_STAT_INDEX_MAX_GEN_2;
199+ dev->hw_attrs.max_hw_device_pages = ICRDMA_MAX_PUSH_PAGE_COUNT;
197200
198201 dev->hw_attrs.uk_attrs.min_hw_wq_size = ICRDMA_MIN_WQ_SIZE;
199202 dev->hw_attrs.uk_attrs.max_hw_sq_chunk = IRDMA_MAX_QUANTA_PER_WR;
drivers/infiniband/hw/irdma/icrdma_hw.h+3−2
@@ -58,14 +58,15 @@ · 57 unmodified lines
5858 #define ICRDMA_CQPSQ_CQ_CQID GENMASK_ULL(18, 0)
5959 #define ICRDMA_COMMIT_FPM_CQCNT_S 0
6060 #define ICRDMA_COMMIT_FPM_CQCNT GENMASK_ULL(19, 0)
61−
61+#define ICRDMA_CQPSQ_UPESD_HMCFNID_S 0
62+#define ICRDMA_CQPSQ_UPESD_HMCFNID GENMASK_ULL(5, 0)
6263 enum icrdma_device_caps_const {
6364 ICRDMA_MAX_STATS_COUNT = 128,
6465
6566 ICRDMA_MAX_IRD_SIZE = 127,
6667 ICRDMA_MAX_ORD_SIZE = 255,
6768 ICRDMA_MIN_WQ_SIZE = 8 /* WQEs */,
68−
69+ ICRDMA_MAX_PUSH_PAGE_COUNT = 256,
6970 };
7071
7172 void icrdma_init_hw(struct irdma_sc_dev *dev);
drivers/infiniband/hw/irdma/icrdma_if.c+343−0
@@ -0,0 +1,343 @@
1+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
2+/* Copyright (c) 2015 - 2024 Intel Corporation */
3+
4+#include "main.h"
5+#include <linux/net/intel/iidc_rdma_ice.h>
6+
7+static void icrdma_prep_tc_change(struct irdma_device *iwdev)
8+{
9+ iwdev->vsi.tc_change_pending = true;
10+ irdma_sc_suspend_resume_qps(&iwdev->vsi, IRDMA_OP_SUSPEND);
11+
12+ /* Wait for all qp's to suspend */
13+ wait_event_timeout(iwdev->suspend_wq,
14+ !atomic_read(&iwdev->vsi.qp_suspend_reqs),
15+ msecs_to_jiffies(IRDMA_EVENT_TIMEOUT_MS));
16+ irdma_ws_reset(&iwdev->vsi);
17+}
18+
19+static void icrdma_fill_qos_info(struct irdma_l2params *l2params,
20+ struct iidc_rdma_qos_params *qos_info)
21+{
22+ int i;
23+
24+ l2params->num_tc = qos_info->num_tc;
25+ l2params->vsi_prio_type = qos_info->vport_priority_type;
26+ l2params->vsi_rel_bw = qos_info->vport_relative_bw;
27+ for (i = 0; i < l2params->num_tc; i++) {
28+ l2params->tc_info[i].egress_virt_up =
29+ qos_info->tc_info[i].egress_virt_up;
30+ l2params->tc_info[i].ingress_virt_up =
31+ qos_info->tc_info[i].ingress_virt_up;
32+ l2params->tc_info[i].prio_type = qos_info->tc_info[i].prio_type;
33+ l2params->tc_info[i].rel_bw = qos_info->tc_info[i].rel_bw;
34+ l2params->tc_info[i].tc_ctx = qos_info->tc_info[i].tc_ctx;
35+ }
36+ for (i = 0; i < IIDC_MAX_USER_PRIORITY; i++)
37+ l2params->up2tc[i] = qos_info->up2tc[i];
38+ if (qos_info->pfc_mode == IIDC_DSCP_PFC_MODE) {
39+ l2params->dscp_mode = true;
40+ memcpy(l2params->dscp_map, qos_info->dscp_map, sizeof(l2params->dscp_map));
41+ }
42+}
43+
44+static void icrdma_iidc_event_handler(struct iidc_rdma_core_dev_info *cdev_info,
45+ struct iidc_rdma_event *event)
46+{
47+ struct irdma_device *iwdev = dev_get_drvdata(&cdev_info->adev->dev);
48+ struct irdma_l2params l2params = {};
49+
50+ if (*event->type & BIT(IIDC_RDMA_EVENT_AFTER_MTU_CHANGE)) {
51+ ibdev_dbg(&iwdev->ibdev, "CLNT: new MTU = %d\n", iwdev->netdev->mtu);
52+ if (iwdev->vsi.mtu != iwdev->netdev->mtu) {
53+ l2params.mtu = iwdev->netdev->mtu;
54+ l2params.mtu_changed = true;
55+ irdma_log_invalid_mtu(l2params.mtu, &iwdev->rf->sc_dev);
56+ irdma_change_l2params(&iwdev->vsi, &l2params);
57+ }
58+ } else if (*event->type & BIT(IIDC_RDMA_EVENT_BEFORE_TC_CHANGE)) {
59+ if (iwdev->vsi.tc_change_pending)
60+ return;
61+
62+ icrdma_prep_tc_change(iwdev);
63+ } else if (*event->type & BIT(IIDC_RDMA_EVENT_AFTER_TC_CHANGE)) {
64+ struct iidc_rdma_priv_dev_info *idc_priv = cdev_info->iidc_priv;
65+
66+ if (!iwdev->vsi.tc_change_pending)
67+ return;
68+
69+ l2params.tc_changed = true;
70+ ibdev_dbg(&iwdev->ibdev, "CLNT: TC Change\n");
71+
72+ icrdma_fill_qos_info(&l2params, &idc_priv->qos_info);
73+ if (iwdev->rf->protocol_used != IRDMA_IWARP_PROTOCOL_ONLY)
74+ iwdev->dcb_vlan_mode =
75+ l2params.num_tc > 1 && !l2params.dscp_mode;
76+ irdma_change_l2params(&iwdev->vsi, &l2params);
77+ } else if (*event->type & BIT(IIDC_RDMA_EVENT_CRIT_ERR)) {
78+ ibdev_warn(&iwdev->ibdev, "ICE OICR event notification: oicr = 0x%08x\n",
79+ event->reg);
80+ if (event->reg & IRDMAPFINT_OICR_PE_CRITERR_M) {
81+ u32 pe_criterr;
82+
83+ pe_criterr = readl(iwdev->rf->sc_dev.hw_regs[IRDMA_GLPE_CRITERR]);
84+#define IRDMA_Q1_RESOURCE_ERR 0x0001024d
85+ if (pe_criterr != IRDMA_Q1_RESOURCE_ERR) {
86+ ibdev_err(&iwdev->ibdev, "critical PE Error, GLPE_CRITERR=0x%08x\n",
87+ pe_criterr);
88+ iwdev->rf->reset = true;
89+ } else {
90+ ibdev_warn(&iwdev->ibdev, "Q1 Resource Check\n");
91+ }
92+ }
93+ if (event->reg & IRDMAPFINT_OICR_HMC_ERR_M) {
94+ ibdev_err(&iwdev->ibdev, "HMC Error\n");
95+ iwdev->rf->reset = true;
96+ }
97+ if (event->reg & IRDMAPFINT_OICR_PE_PUSH_M) {
98+ ibdev_err(&iwdev->ibdev, "PE Push Error\n");
99+ iwdev->rf->reset = true;
100+ }
101+ if (iwdev->rf->reset)
102+ iwdev->rf->gen_ops.request_reset(iwdev->rf);
103+ }
104+}
105+
106+/**
107+ * icrdma_lan_register_qset - Register qset with LAN driver
108+ * @vsi: vsi structure
109+ * @tc_node: Traffic class node
110+ */
111+static int icrdma_lan_register_qset(struct irdma_sc_vsi *vsi,
112+ struct irdma_ws_node *tc_node)
113+{
114+ struct irdma_device *iwdev = vsi->back_vsi;
115+ struct iidc_rdma_core_dev_info *cdev_info = iwdev->rf->cdev;
116+ struct iidc_rdma_qset_params qset = {};
117+ int ret;
118+
119+ qset.qs_handle = tc_node->qs_handle;
120+ qset.tc = tc_node->traffic_class;
121+ qset.vport_id = vsi->vsi_idx;
122+ ret = ice_add_rdma_qset(cdev_info, &qset);
123+ if (ret) {
124+ ibdev_dbg(&iwdev->ibdev, "WS: LAN alloc_res for rdma qset failed.\n");
125+ return ret;
126+ }
127+
128+ tc_node->l2_sched_node_id = qset.teid;
129+ vsi->qos[tc_node->user_pri].l2_sched_node_id = qset.teid;
130+
131+ return 0;
132+}
133+
134+/**
135+ * icrdma_lan_unregister_qset - Unregister qset with LAN driver
136+ * @vsi: vsi structure
137+ * @tc_node: Traffic class node
138+ */
139+static void icrdma_lan_unregister_qset(struct irdma_sc_vsi *vsi,
140+ struct irdma_ws_node *tc_node)
141+{
142+ struct irdma_device *iwdev = vsi->back_vsi;
143+ struct iidc_rdma_core_dev_info *cdev_info = iwdev->rf->cdev;
144+ struct iidc_rdma_qset_params qset = {};
145+
146+ qset.qs_handle = tc_node->qs_handle;
147+ qset.tc = tc_node->traffic_class;
148+ qset.vport_id = vsi->vsi_idx;
149+ qset.teid = tc_node->l2_sched_node_id;
150+
151+ if (ice_del_rdma_qset(cdev_info, &qset))
152+ ibdev_dbg(&iwdev->ibdev, "WS: LAN free_res for rdma qset failed.\n");
153+}
154+
155+/**
156+ * icrdma_request_reset - Request a reset
157+ * @rf: RDMA PCI function
158+ */
159+static void icrdma_request_reset(struct irdma_pci_f *rf)
160+{
161+ ibdev_warn(&rf->iwdev->ibdev, "Requesting a reset\n");
162+ ice_rdma_request_reset(rf->cdev, IIDC_FUNC_RESET);
163+}
164+
165+static int icrdma_init_interrupts(struct irdma_pci_f *rf, struct iidc_rdma_core_dev_info *cdev)
166+{
167+ int i;
168+
169+ rf->msix_count = num_online_cpus() + IRDMA_NUM_AEQ_MSIX;
170+ rf->msix_entries = kcalloc(rf->msix_count, sizeof(*rf->msix_entries),
171+ GFP_KERNEL);
172+ if (!rf->msix_entries)
173+ return -ENOMEM;
174+
175+ for (i = 0; i < rf->msix_count; i++)
176+ if (ice_alloc_rdma_qvector(cdev, &rf->msix_entries[i]))
177+ break;
178+
179+ if (i < IRDMA_MIN_MSIX) {
180+ while (--i >= 0)
181+ ice_free_rdma_qvector(cdev, &rf->msix_entries[i]);
182+
183+ kfree(rf->msix_entries);
184+ return -ENOMEM;
185+ }
186+
187+ rf->msix_count = i;
188+
189+ return 0;
190+}
191+
192+static void icrdma_deinit_interrupts(struct irdma_pci_f *rf, struct iidc_rdma_core_dev_info *cdev)
193+{
194+ int i;
195+
196+ for (i = 0; i < rf->msix_count; i++)
197+ ice_free_rdma_qvector(cdev, &rf->msix_entries[i]);
198+
199+ kfree(rf->msix_entries);
200+}
201+
202+static void icrdma_fill_device_info(struct irdma_device *iwdev,
203+ struct iidc_rdma_core_dev_info *cdev_info)
204+{
205+ struct iidc_rdma_priv_dev_info *idc_priv = cdev_info->iidc_priv;
206+ struct irdma_pci_f *rf = iwdev->rf;
207+
208+ rf->sc_dev.hw = &rf->hw;
209+ rf->iwdev = iwdev;
210+ rf->cdev = cdev_info;
211+ rf->hw.hw_addr = idc_priv->hw_addr;
212+ rf->pcidev = cdev_info->pdev;
213+ rf->hw.device = &rf->pcidev->dev;
214+ rf->pf_id = idc_priv->pf_id;
215+ rf->rdma_ver = IRDMA_GEN_2;
216+ rf->sc_dev.hw_attrs.uk_attrs.hw_rev = IRDMA_GEN_2;
217+ rf->sc_dev.is_pf = true;
218+ rf->sc_dev.privileged = true;
219+
220+ rf->gen_ops.register_qset = icrdma_lan_register_qset;
221+ rf->gen_ops.unregister_qset = icrdma_lan_unregister_qset;
222+
223+ rf->default_vsi.vsi_idx = idc_priv->vport_id;
224+ rf->protocol_used =
225+ cdev_info->rdma_protocol == IIDC_RDMA_PROTOCOL_ROCEV2 ?
226+ IRDMA_ROCE_PROTOCOL_ONLY : IRDMA_IWARP_PROTOCOL_ONLY;
227+ rf->rsrc_profile = IRDMA_HMC_PROFILE_DEFAULT;
228+ rf->rst_to = IRDMA_RST_TIMEOUT_HZ;
229+ rf->gen_ops.request_reset = icrdma_request_reset;
230+ rf->limits_sel = 7;
231+ mutex_init(&rf->ah_tbl_lock);
232+
233+ iwdev->netdev = idc_priv->netdev;
234+ iwdev->vsi_num = idc_priv->vport_id;
235+ iwdev->init_state = INITIAL_STATE;
236+ iwdev->roce_cwnd = IRDMA_ROCE_CWND_DEFAULT;
237+ iwdev->roce_ackcreds = IRDMA_ROCE_ACKCREDS_DEFAULT;
238+ iwdev->rcv_wnd = IRDMA_CM_DEFAULT_RCV_WND_SCALED;
239+ iwdev->rcv_wscale = IRDMA_CM_DEFAULT_RCV_WND_SCALE;
240+ if (iwdev->rf->protocol_used != IRDMA_IWARP_PROTOCOL_ONLY)
241+ iwdev->roce_mode = true;
242+}
243+
244+static int icrdma_probe(struct auxiliary_device *aux_dev, const struct auxiliary_device_id *id)
245+{
246+ struct iidc_rdma_core_auxiliary_dev *iidc_adev;
247+ struct iidc_rdma_core_dev_info *cdev_info;
248+ struct iidc_rdma_priv_dev_info *idc_priv;
249+ struct irdma_l2params l2params = {};
250+ struct irdma_device *iwdev;
251+ struct irdma_pci_f *rf;
252+ int err;
253+
254+ iidc_adev = container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev);
255+ cdev_info = iidc_adev->cdev_info;
256+ idc_priv = cdev_info->iidc_priv;
257+
258+ iwdev = ib_alloc_device(irdma_device, ibdev);
259+ if (!iwdev)
260+ return -ENOMEM;
261+ iwdev->rf = kzalloc(sizeof(*rf), GFP_KERNEL);
262+ if (!iwdev->rf) {
263+ ib_dealloc_device(&iwdev->ibdev);
264+ return -ENOMEM;
265+ }
266+
267+ icrdma_fill_device_info(iwdev, cdev_info);
268+ rf = iwdev->rf;
269+
270+ err = icrdma_init_interrupts(rf, cdev_info);
271+ if (err)
272+ goto err_init_interrupts;
273+
274+ err = irdma_ctrl_init_hw(rf);
275+ if (err)
276+ goto err_ctrl_init;
277+
278+ l2params.mtu = iwdev->netdev->mtu;
279+ icrdma_fill_qos_info(&l2params, &idc_priv->qos_info);
280+ if (iwdev->rf->protocol_used != IRDMA_IWARP_PROTOCOL_ONLY)
281+ iwdev->dcb_vlan_mode = l2params.num_tc > 1 && !l2params.dscp_mode;
282+
283+ err = irdma_rt_init_hw(iwdev, &l2params);
284+ if (err)
285+ goto err_rt_init;
286+
287+ err = irdma_ib_register_device(iwdev);
288+ if (err)
289+ goto err_ibreg;
290+
291+ ice_rdma_update_vsi_filter(cdev_info, iwdev->vsi_num, true);
292+
293+ ibdev_dbg(&iwdev->ibdev, "INIT: Gen2 PF[%d] device probe success\n", PCI_FUNC(rf->pcidev->devfn));
294+ auxiliary_set_drvdata(aux_dev, iwdev);
295+
296+ return 0;
297+
298+err_ibreg:
299+ irdma_rt_deinit_hw(iwdev);
300+err_rt_init:
301+ irdma_ctrl_deinit_hw(rf);
302+err_ctrl_init:
303+ icrdma_deinit_interrupts(rf, cdev_info);
304+err_init_interrupts:
305+ kfree(iwdev->rf);
306+ ib_dealloc_device(&iwdev->ibdev);
307+
308+ return err;
309+}
310+
311+static void icrdma_remove(struct auxiliary_device *aux_dev)
312+{
313+ struct iidc_rdma_core_auxiliary_dev *idc_adev =
314+ container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev);
315+ struct iidc_rdma_core_dev_info *cdev_info = idc_adev->cdev_info;
316+ struct irdma_device *iwdev = auxiliary_get_drvdata(aux_dev);
317+ u8 rdma_ver = iwdev->rf->rdma_ver;
318+
319+ ice_rdma_update_vsi_filter(cdev_info, iwdev->vsi_num, false);
320+ irdma_ib_unregister_device(iwdev);
321+ icrdma_deinit_interrupts(iwdev->rf, cdev_info);
322+
323+ pr_debug("INIT: Gen[%d] func[%d] device remove success\n",
324+ rdma_ver, PCI_FUNC(cdev_info->pdev->devfn));
325+}
326+
327+static const struct auxiliary_device_id icrdma_auxiliary_id_table[] = {
328+ {.name = "ice.iwarp", },
329+ {.name = "ice.roce", },
330+ {},
331+};
332+
333+MODULE_DEVICE_TABLE(auxiliary, icrdma_auxiliary_id_table);
334+
335+struct iidc_rdma_core_auxiliary_drv icrdma_core_auxiliary_drv = {
336+ .adrv = {
337+ .name = "gen_2",
338+ .id_table = icrdma_auxiliary_id_table,
339+ .probe = icrdma_probe,
340+ .remove = icrdma_remove,
341+ },
342+ .event_handler = icrdma_iidc_event_handler,
343+};
drivers/infiniband/hw/irdma/ig3rdma_hw.c+170−0
@@ -0,0 +1,170 @@
1+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
2+/* Copyright (c) 2018 - 2024 Intel Corporation */
3+#include "osdep.h"
4+#include "type.h"
5+#include "protos.h"
6+#include "ig3rdma_hw.h"
7+
8+/**
9+ * ig3rdma_ena_irq - Enable interrupt
10+ * @dev: pointer to the device structure
11+ * @idx: vector index
12+ */
13+static void ig3rdma_ena_irq(struct irdma_sc_dev *dev, u32 idx)
14+{
15+ u32 val;
16+ u32 int_stride = 1; /* one u32 per register */
17+
18+ if (dev->is_pf)
19+ int_stride = 0x400;
20+ else
21+ idx--; /* VFs use DYN_CTL_N */
22+
23+ val = FIELD_PREP(IRDMA_GLINT_DYN_CTL_INTENA, 1) |
24+ FIELD_PREP(IRDMA_GLINT_DYN_CTL_CLEARPBA, 1);
25+
26+ writel(val, dev->hw_regs[IRDMA_GLINT_DYN_CTL] + (idx * int_stride));
27+}
28+
29+/**
30+ * ig3rdma_disable_irq - Disable interrupt
31+ * @dev: pointer to the device structure
32+ * @idx: vector index
33+ */
34+static void ig3rdma_disable_irq(struct irdma_sc_dev *dev, u32 idx)
35+{
36+ u32 int_stride = 1; /* one u32 per register */
37+
38+ if (dev->is_pf)
39+ int_stride = 0x400;
40+ else
41+ idx--; /* VFs use DYN_CTL_N */
42+
43+ writel(0, dev->hw_regs[IRDMA_GLINT_DYN_CTL] + (idx * int_stride));
44+}
45+
46+static const struct irdma_irq_ops ig3rdma_irq_ops = {
47+ .irdma_dis_irq = ig3rdma_disable_irq,
48+ .irdma_en_irq = ig3rdma_ena_irq,
49+};
50+
51+static const struct irdma_hw_stat_map ig3rdma_hw_stat_map[] = {
52+ [IRDMA_HW_STAT_INDEX_RXVLANERR] = { 0, 0, 0 },
53+ [IRDMA_HW_STAT_INDEX_IP4RXOCTS] = { 8, 0, 0 },
54+ [IRDMA_HW_STAT_INDEX_IP4RXPKTS] = { 16, 0, 0 },
55+ [IRDMA_HW_STAT_INDEX_IP4RXDISCARD] = { 24, 0, 0 },
56+ [IRDMA_HW_STAT_INDEX_IP4RXTRUNC] = { 32, 0, 0 },
57+ [IRDMA_HW_STAT_INDEX_IP4RXFRAGS] = { 40, 0, 0 },
58+ [IRDMA_HW_STAT_INDEX_IP4RXMCOCTS] = { 48, 0, 0 },
59+ [IRDMA_HW_STAT_INDEX_IP4RXMCPKTS] = { 56, 0, 0 },
60+ [IRDMA_HW_STAT_INDEX_IP6RXOCTS] = { 64, 0, 0 },
61+ [IRDMA_HW_STAT_INDEX_IP6RXPKTS] = { 72, 0, 0 },
62+ [IRDMA_HW_STAT_INDEX_IP6RXDISCARD] = { 80, 0, 0 },
63+ [IRDMA_HW_STAT_INDEX_IP6RXTRUNC] = { 88, 0, 0 },
64+ [IRDMA_HW_STAT_INDEX_IP6RXFRAGS] = { 96, 0, 0 },
65+ [IRDMA_HW_STAT_INDEX_IP6RXMCOCTS] = { 104, 0, 0 },
66+ [IRDMA_HW_STAT_INDEX_IP6RXMCPKTS] = { 112, 0, 0 },
67+ [IRDMA_HW_STAT_INDEX_IP4TXOCTS] = { 120, 0, 0 },
68+ [IRDMA_HW_STAT_INDEX_IP4TXPKTS] = { 128, 0, 0 },
69+ [IRDMA_HW_STAT_INDEX_IP4TXFRAGS] = { 136, 0, 0 },
70+ [IRDMA_HW_STAT_INDEX_IP4TXMCOCTS] = { 144, 0, 0 },
71+ [IRDMA_HW_STAT_INDEX_IP4TXMCPKTS] = { 152, 0, 0 },
72+ [IRDMA_HW_STAT_INDEX_IP6TXOCTS] = { 160, 0, 0 },
73+ [IRDMA_HW_STAT_INDEX_IP6TXPKTS] = { 168, 0, 0 },
74+ [IRDMA_HW_STAT_INDEX_IP6TXFRAGS] = { 176, 0, 0 },
75+ [IRDMA_HW_STAT_INDEX_IP6TXMCOCTS] = { 184, 0, 0 },
76+ [IRDMA_HW_STAT_INDEX_IP6TXMCPKTS] = { 192, 0, 0 },
77+ [IRDMA_HW_STAT_INDEX_IP4TXNOROUTE] = { 200, 0, 0 },
78+ [IRDMA_HW_STAT_INDEX_IP6TXNOROUTE] = { 208, 0, 0 },
79+ [IRDMA_HW_STAT_INDEX_TCPRTXSEG] = { 216, 0, 0 },
80+ [IRDMA_HW_STAT_INDEX_TCPRXOPTERR] = { 224, 0, 0 },
81+ [IRDMA_HW_STAT_INDEX_TCPRXPROTOERR] = { 232, 0, 0 },
82+ [IRDMA_HW_STAT_INDEX_TCPTXSEG] = { 240, 0, 0 },
83+ [IRDMA_HW_STAT_INDEX_TCPRXSEGS] = { 248, 0, 0 },
84+ [IRDMA_HW_STAT_INDEX_UDPRXPKTS] = { 256, 0, 0 },
85+ [IRDMA_HW_STAT_INDEX_UDPTXPKTS] = { 264, 0, 0 },
86+ [IRDMA_HW_STAT_INDEX_RDMARXWRS] = { 272, 0, 0 },
87+ [IRDMA_HW_STAT_INDEX_RDMARXRDS] = { 280, 0, 0 },
88+ [IRDMA_HW_STAT_INDEX_RDMARXSNDS] = { 288, 0, 0 },
89+ [IRDMA_HW_STAT_INDEX_RDMATXWRS] = { 296, 0, 0 },
90+ [IRDMA_HW_STAT_INDEX_RDMATXRDS] = { 304, 0, 0 },
91+ [IRDMA_HW_STAT_INDEX_RDMATXSNDS] = { 312, 0, 0 },
92+ [IRDMA_HW_STAT_INDEX_RDMAVBND] = { 320, 0, 0 },
93+ [IRDMA_HW_STAT_INDEX_RDMAVINV] = { 328, 0, 0 },
94+ [IRDMA_HW_STAT_INDEX_RXNPECNMARKEDPKTS] = { 336, 0, 0 },
95+ [IRDMA_HW_STAT_INDEX_RXRPCNPHANDLED] = { 344, 0, 0 },
96+ [IRDMA_HW_STAT_INDEX_RXRPCNPIGNORED] = { 352, 0, 0 },
97+ [IRDMA_HW_STAT_INDEX_TXNPCNPSENT] = { 360, 0, 0 },
98+ [IRDMA_HW_STAT_INDEX_RNR_SENT] = { 368, 0, 0 },
99+ [IRDMA_HW_STAT_INDEX_RNR_RCVD] = { 376, 0, 0 },
100+ [IRDMA_HW_STAT_INDEX_RDMAORDLMTCNT] = { 384, 0, 0 },
101+ [IRDMA_HW_STAT_INDEX_RDMAIRDLMTCNT] = { 392, 0, 0 },
102+ [IRDMA_HW_STAT_INDEX_RDMARXATS] = { 408, 0, 0 },
103+ [IRDMA_HW_STAT_INDEX_RDMATXATS] = { 416, 0, 0 },
104+ [IRDMA_HW_STAT_INDEX_NAKSEQERR] = { 424, 0, 0 },
105+ [IRDMA_HW_STAT_INDEX_NAKSEQERR_IMPLIED] = { 432, 0, 0 },
106+ [IRDMA_HW_STAT_INDEX_RTO] = { 440, 0, 0 },
107+ [IRDMA_HW_STAT_INDEX_RXOOOPKTS] = { 448, 0, 0 },
108+ [IRDMA_HW_STAT_INDEX_ICRCERR] = { 456, 0, 0 },
109+};
110+
111+void ig3rdma_init_hw(struct irdma_sc_dev *dev)
112+{
113+ dev->irq_ops = &ig3rdma_irq_ops;
114+ dev->hw_stats_map = ig3rdma_hw_stat_map;
115+
116+ dev->hw_attrs.uk_attrs.hw_rev = IRDMA_GEN_3;
117+ dev->hw_attrs.uk_attrs.max_hw_wq_frags = IG3RDMA_MAX_WQ_FRAGMENT_COUNT;
118+ dev->hw_attrs.uk_attrs.max_hw_read_sges = IG3RDMA_MAX_SGE_RD;
119+ dev->hw_attrs.uk_attrs.max_hw_sq_chunk = IRDMA_MAX_QUANTA_PER_WR;
120+ dev->hw_attrs.first_hw_vf_fpm_id = 0;
121+ dev->hw_attrs.max_hw_vf_fpm_id = IG3_MAX_APFS + IG3_MAX_AVFS;
122+ dev->hw_attrs.uk_attrs.feature_flags |= IRDMA_FEATURE_64_BYTE_CQE;
123+ dev->hw_attrs.uk_attrs.feature_flags |= IRDMA_FEATURE_CQE_TIMESTAMPING;
124+
125+ dev->hw_attrs.uk_attrs.feature_flags |= IRDMA_FEATURE_SRQ;
126+ dev->hw_attrs.uk_attrs.feature_flags |= IRDMA_FEATURE_RTS_AE |
127+ IRDMA_FEATURE_CQ_RESIZE;
128+ dev->hw_attrs.page_size_cap = SZ_4K | SZ_2M | SZ_1G;
129+ dev->hw_attrs.max_hw_ird = IG3RDMA_MAX_IRD_SIZE;
130+ dev->hw_attrs.max_hw_ord = IG3RDMA_MAX_ORD_SIZE;
131+ dev->hw_attrs.max_stat_inst = IG3RDMA_MAX_STATS_COUNT;
132+ dev->hw_attrs.max_stat_idx = IRDMA_HW_STAT_INDEX_MAX_GEN_3;
133+ dev->hw_attrs.uk_attrs.min_hw_wq_size = IG3RDMA_MIN_WQ_SIZE;
134+ dev->hw_attrs.uk_attrs.max_hw_srq_quanta = IRDMA_SRQ_MAX_QUANTA;
135+ dev->hw_attrs.uk_attrs.max_hw_inline = IG3RDMA_MAX_INLINE_DATA_SIZE;
136+ dev->hw_attrs.max_hw_device_pages =
137+ dev->is_pf ? IG3RDMA_MAX_PF_PUSH_PAGE_COUNT : IG3RDMA_MAX_VF_PUSH_PAGE_COUNT;
138+}
139+
140+static void __iomem *__ig3rdma_get_reg_addr(struct irdma_mmio_region *region, u64 reg_offset)
141+{
142+ if (reg_offset >= region->offset &&
143+ reg_offset < (region->offset + region->len)) {
144+ reg_offset -= region->offset;
145+
146+ return region->addr + reg_offset;
147+ }
148+
149+ return NULL;
150+}
151+
152+void __iomem *ig3rdma_get_reg_addr(struct irdma_hw *hw, u64 reg_offset)
153+{
154+ u8 __iomem *reg_addr;
155+ int i;
156+
157+ reg_addr = __ig3rdma_get_reg_addr(&hw->rdma_reg, reg_offset);
158+ if (reg_addr)
159+ return reg_addr;
160+
161+ for (i = 0; i < hw->num_io_regions; i++) {
162+ reg_addr = __ig3rdma_get_reg_addr(&hw->io_regs[i], reg_offset);
163+ if (reg_addr)
164+ return reg_addr;
165+ }
166+
167+ WARN_ON_ONCE(1);
168+
169+ return NULL;
170+}
drivers/infiniband/hw/irdma/ig3rdma_hw.h+32−0
@@ -0,0 +1,32 @@
1+/* SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB */
2+/* Copyright (c) 2021 - 2024 Intel Corporation */
3+#ifndef IG3RDMA_HW_H
4+#define IG3RDMA_HW_H
5+
6+#define IG3_MAX_APFS 1
7+#define IG3_MAX_AVFS 0
8+
9+#define IG3_PF_RDMA_REGION_OFFSET 0xBC00000
10+#define IG3_PF_RDMA_REGION_LEN 0x401000
11+#define IG3_VF_RDMA_REGION_OFFSET 0x8C00
12+#define IG3_VF_RDMA_REGION_LEN 0x8400
13+
14+enum ig3rdma_device_caps_const {
15+ IG3RDMA_MAX_WQ_FRAGMENT_COUNT = 14,
16+ IG3RDMA_MAX_SGE_RD = 14,
17+
18+ IG3RDMA_MAX_STATS_COUNT = 128,
19+
20+ IG3RDMA_MAX_IRD_SIZE = 64,
21+ IG3RDMA_MAX_ORD_SIZE = 64,
22+ IG3RDMA_MIN_WQ_SIZE = 16 /* WQEs */,
23+ IG3RDMA_MAX_INLINE_DATA_SIZE = 216,
24+ IG3RDMA_MAX_PF_PUSH_PAGE_COUNT = 8192,
25+ IG3RDMA_MAX_VF_PUSH_PAGE_COUNT = 16,
26+};
27+
28+void __iomem *ig3rdma_get_reg_addr(struct irdma_hw *hw, u64 reg_offset);
29+int ig3rdma_vchnl_send_sync(struct irdma_sc_dev *dev, u8 *msg, u16 len,
30+ u8 *recv_msg, u16 *recv_len);
31+
32+#endif /* IG3RDMA_HW_H*/
drivers/infiniband/hw/irdma/ig3rdma_if.c+232−0
@@ -0,0 +1,232 @@
1+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
2+/* Copyright (c) 2023 - 2024 Intel Corporation */
3+
4+#include "main.h"
5+#include <linux/net/intel/iidc_rdma_idpf.h>
6+#include "ig3rdma_hw.h"
7+
8+static void ig3rdma_idc_core_event_handler(struct iidc_rdma_core_dev_info *cdev_info,
9+ struct iidc_rdma_event *event)
10+{
11+ struct irdma_pci_f *rf = auxiliary_get_drvdata(cdev_info->adev);
12+
13+ if (*event->type & BIT(IIDC_RDMA_EVENT_WARN_RESET)) {
14+ rf->reset = true;
15+ rf->sc_dev.vchnl_up = false;
16+ }
17+}
18+
19+int ig3rdma_vchnl_send_sync(struct irdma_sc_dev *dev, u8 *msg, u16 len,
20+ u8 *recv_msg, u16 *recv_len)
21+{
22+ struct iidc_rdma_core_dev_info *cdev_info = dev_to_rf(dev)->cdev;
23+ int ret;
24+
25+ ret = idpf_idc_rdma_vc_send_sync(cdev_info, msg, len, recv_msg,
26+ recv_len);
27+ if (ret == -ETIMEDOUT) {
28+ ibdev_err(&(dev_to_rf(dev)->iwdev->ibdev),
29+ "Virtual channel Req <-> Resp completion timeout\n");
30+ dev->vchnl_up = false;
31+ }
32+
33+ return ret;
34+}
35+
36+static int ig3rdma_vchnl_init(struct irdma_pci_f *rf,
37+ struct iidc_rdma_core_dev_info *cdev_info,
38+ u8 *rdma_ver)
39+{
40+ struct iidc_rdma_priv_dev_info *idc_priv = cdev_info->iidc_priv;
41+ struct irdma_vchnl_init_info virt_info;
42+ u8 gen = rf->rdma_ver;
43+ int ret;
44+
45+ rf->vchnl_wq = alloc_ordered_workqueue("irdma-virtchnl-wq", 0);
46+ if (!rf->vchnl_wq)
47+ return -ENOMEM;
48+
49+ mutex_init(&rf->sc_dev.vchnl_mutex);
50+
51+ virt_info.is_pf = !idc_priv->ftype;
52+ virt_info.hw_rev = gen;
53+ virt_info.privileged = gen == IRDMA_GEN_2;
54+ virt_info.vchnl_wq = rf->vchnl_wq;
55+ ret = irdma_sc_vchnl_init(&rf->sc_dev, &virt_info);
56+ if (ret) {
57+ destroy_workqueue(rf->vchnl_wq);
58+ return ret;
59+ }
60+
61+ *rdma_ver = rf->sc_dev.hw_attrs.uk_attrs.hw_rev;
62+
63+ return 0;
64+}
65+
66+/**
67+ * ig3rdma_request_reset - Request a reset
68+ * @rf: RDMA PCI function
69+ */
70+static void ig3rdma_request_reset(struct irdma_pci_f *rf)
71+{
72+ ibdev_warn(&rf->iwdev->ibdev, "Requesting a reset\n");
73+ idpf_idc_request_reset(rf->cdev, IIDC_FUNC_RESET);
74+}
75+
76+static int ig3rdma_cfg_regions(struct irdma_hw *hw,
77+ struct iidc_rdma_core_dev_info *cdev_info)
78+{
79+ struct iidc_rdma_priv_dev_info *idc_priv = cdev_info->iidc_priv;
80+ struct pci_dev *pdev = cdev_info->pdev;
81+ int i;
82+
83+ switch (idc_priv->ftype) {
84+ case IIDC_FUNCTION_TYPE_PF:
85+ hw->rdma_reg.len = IG3_PF_RDMA_REGION_LEN;
86+ hw->rdma_reg.offset = IG3_PF_RDMA_REGION_OFFSET;
87+ break;
88+ case IIDC_FUNCTION_TYPE_VF:
89+ hw->rdma_reg.len = IG3_VF_RDMA_REGION_LEN;
90+ hw->rdma_reg.offset = IG3_VF_RDMA_REGION_OFFSET;
91+ break;
92+ default:
93+ return -ENODEV;
94+ }
95+
96+ hw->rdma_reg.addr = ioremap(pci_resource_start(pdev, 0) + hw->rdma_reg.offset,
97+ hw->rdma_reg.len);
98+
99+ if (!hw->rdma_reg.addr)
100+ return -ENOMEM;
101+
102+ hw->num_io_regions = le16_to_cpu(idc_priv->num_memory_regions);
103+ hw->io_regs = kcalloc(hw->num_io_regions,
104+ sizeof(struct irdma_mmio_region), GFP_KERNEL);
105+
106+ if (!hw->io_regs) {
107+ iounmap(hw->rdma_reg.addr);
108+ return -ENOMEM;
109+ }
110+
111+ for (i = 0; i < hw->num_io_regions; i++) {
112+ hw->io_regs[i].addr =
113+ idc_priv->mapped_mem_regions[i].region_addr;
114+ hw->io_regs[i].len =
115+ le64_to_cpu(idc_priv->mapped_mem_regions[i].size);
116+ hw->io_regs[i].offset =
117+ le64_to_cpu(idc_priv->mapped_mem_regions[i].start_offset);
118+ }
119+
120+ return 0;
121+}
122+
123+static void ig3rdma_decfg_rf(struct irdma_pci_f *rf)
124+{
125+ struct irdma_hw *hw = &rf->hw;
126+
127+ destroy_workqueue(rf->vchnl_wq);
128+ kfree(hw->io_regs);
129+ iounmap(hw->rdma_reg.addr);
130+}
131+
132+static int ig3rdma_cfg_rf(struct irdma_pci_f *rf,
133+ struct iidc_rdma_core_dev_info *cdev_info)
134+{
135+ struct iidc_rdma_priv_dev_info *idc_priv = cdev_info->iidc_priv;
136+ int err;
137+
138+ rf->sc_dev.hw = &rf->hw;
139+ rf->cdev = cdev_info;
140+ rf->pcidev = cdev_info->pdev;
141+ rf->hw.device = &rf->pcidev->dev;
142+ rf->msix_count = idc_priv->msix_count;
143+ rf->msix_entries = idc_priv->msix_entries;
144+
145+ err = ig3rdma_vchnl_init(rf, cdev_info, &rf->rdma_ver);
146+ if (err)
147+ return err;
148+
149+ err = ig3rdma_cfg_regions(&rf->hw, cdev_info);
150+ if (err) {
151+ destroy_workqueue(rf->vchnl_wq);
152+ return err;
153+ }
154+
155+ rf->protocol_used = IRDMA_ROCE_PROTOCOL_ONLY;
156+ rf->rsrc_profile = IRDMA_HMC_PROFILE_DEFAULT;
157+ rf->rst_to = IRDMA_RST_TIMEOUT_HZ;
158+ rf->gen_ops.request_reset = ig3rdma_request_reset;
159+ rf->limits_sel = 7;
160+ mutex_init(&rf->ah_tbl_lock);
161+
162+ return 0;
163+}
164+
165+static int ig3rdma_core_probe(struct auxiliary_device *aux_dev,
166+ const struct auxiliary_device_id *id)
167+{
168+ struct iidc_rdma_core_auxiliary_dev *idc_adev =
169+ container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev);
170+ struct iidc_rdma_core_dev_info *cdev_info = idc_adev->cdev_info;
171+ struct irdma_pci_f *rf;
172+ int err;
173+
174+ rf = kzalloc(sizeof(*rf), GFP_KERNEL);
175+ if (!rf)
176+ return -ENOMEM;
177+
178+ err = ig3rdma_cfg_rf(rf, cdev_info);
179+ if (err)
180+ goto err_cfg_rf;
181+
182+ err = irdma_ctrl_init_hw(rf);
183+ if (err)
184+ goto err_ctrl_init;
185+
186+ auxiliary_set_drvdata(aux_dev, rf);
187+
188+ err = idpf_idc_vport_dev_ctrl(cdev_info, true);
189+ if (err)
190+ goto err_vport_ctrl;
191+
192+ return 0;
193+
194+err_vport_ctrl:
195+ irdma_ctrl_deinit_hw(rf);
196+err_ctrl_init:
197+ ig3rdma_decfg_rf(rf);
198+err_cfg_rf:
199+ kfree(rf);
200+
201+ return err;
202+}
203+
204+static void ig3rdma_core_remove(struct auxiliary_device *aux_dev)
205+{
206+ struct iidc_rdma_core_auxiliary_dev *idc_adev =
207+ container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev);
208+ struct iidc_rdma_core_dev_info *cdev_info = idc_adev->cdev_info;
209+ struct irdma_pci_f *rf = auxiliary_get_drvdata(aux_dev);
210+
211+ idpf_idc_vport_dev_ctrl(cdev_info, false);
212+ irdma_ctrl_deinit_hw(rf);
213+ ig3rdma_decfg_rf(rf);
214+ kfree(rf);
215+}
216+
217+static const struct auxiliary_device_id ig3rdma_core_auxiliary_id_table[] = {
218+ {.name = "idpf.8086.rdma.core", },
219+ {},
220+};
221+
222+MODULE_DEVICE_TABLE(auxiliary, ig3rdma_core_auxiliary_id_table);
223+
224+struct iidc_rdma_core_auxiliary_drv ig3rdma_core_auxiliary_drv = {
225+ .adrv = {
226+ .name = "core",
227+ .id_table = ig3rdma_core_auxiliary_id_table,
228+ .probe = ig3rdma_core_probe,
229+ .remove = ig3rdma_core_remove,
230+ },
231+ .event_handler = ig3rdma_idc_core_event_handler,
232+};
drivers/infiniband/hw/irdma/irdma.h+20−2
@@ -32,7 +32,16 @@ · 31 unmodified lines
3232 #define IRDMA_PFHMC_SDDATALOW_PMSDDATALOW GENMASK(31, 12)
3333 #define IRDMA_PFHMC_SDCMD_PMSDWR BIT(31)
3434
35−#define IRDMA_INVALID_CQ_IDX 0xffffffff
35+#define IRDMA_INVALID_CQ_IDX 0xffffffff
36+#define IRDMA_Q_INVALID_IDX 0xffff
37+
38+enum irdma_dyn_idx_t {
39+ IRDMA_IDX_ITR0 = 0,
40+ IRDMA_IDX_ITR1 = 1,
41+ IRDMA_IDX_ITR2 = 2,
42+ IRDMA_IDX_NOITR = 3,
43+};
44+
3645 enum irdma_registers {
3746 IRDMA_CQPTAIL,
3847 IRDMA_CQPDB,
@@ -67,6 +76,7 @@ enum irdma_shifts { · 28 unmodified lines
6776 IRDMA_CQPSQ_CQ_CEQID_S,
6877 IRDMA_CQPSQ_CQ_CQID_S,
6978 IRDMA_COMMIT_FPM_CQCNT_S,
79+ IRDMA_CQPSQ_UPESD_HMCFNID_S,
7080 IRDMA_MAX_SHIFTS,
7181 };
7282
@@ -77,6 +87,7 @@ enum irdma_masks { · 4 unmodified lines
7787 IRDMA_CQPSQ_CQ_CEQID_M,
7888 IRDMA_CQPSQ_CQ_CQID_M,
7989 IRDMA_COMMIT_FPM_CQCNT_M,
90+ IRDMA_CQPSQ_UPESD_HMCFNID_M,
8091 IRDMA_MAX_MASKS, /* Must be last entry */
8192 };
8293
@@ -92,7 +103,7 @@ struct irdma_mcast_grp_ctx_entry_info { · 9 unmodified lines
92103 struct irdma_mcast_grp_info {
93104 u8 dest_mac_addr[ETH_ALEN];
94105 u16 vlan_id;
95− u8 hmc_fcn_id;
106+ u16 hmc_fcn_id;
96107 bool ipv4_valid:1;
97108 bool vlan_valid:1;
98109 u16 mg_id;
@@ -107,6 +118,9 @@ enum irdma_vers { · 8 unmodified lines
107118 IRDMA_GEN_RSVD,
108119 IRDMA_GEN_1,
109120 IRDMA_GEN_2,
121+ IRDMA_GEN_3,
122+ IRDMA_GEN_NEXT,
123+ IRDMA_GEN_MAX = IRDMA_GEN_NEXT-1
110124 };
111125
112126 struct irdma_uk_attrs {
@@ -118,6 +132,7 @@ struct irdma_uk_attrs { · 5 unmodified lines
118132 u32 max_hw_wq_quanta;
119133 u32 min_hw_cq_size;
120134 u32 max_hw_cq_size;
135+ u32 max_hw_srq_quanta;
121136 u16 max_hw_sq_chunk;
122137 u16 min_hw_wq_size;
123138 u8 hw_rev;
@@ -147,10 +162,13 @@ struct irdma_hw_attrs { · 23 unmodified lines
147162 u32 max_done_count;
148163 u32 max_sleep_count;
149164 u32 max_cqp_compl_wait_time_ms;
165+ u32 min_hw_srq_id;
150166 u16 max_stat_inst;
151167 u16 max_stat_idx;
152168 };
153169
154170 void i40iw_init_hw(struct irdma_sc_dev *dev);
155171 void icrdma_init_hw(struct irdma_sc_dev *dev);
172+void ig3rdma_init_hw(struct irdma_sc_dev *dev);
173+void __iomem *ig3rdma_get_reg_addr(struct irdma_hw *hw, u64 reg_offset);
156174 #endif /* IRDMA_H*/
drivers/infiniband/hw/irdma/main.c+81−290
@@ -1,6 +1,7 @@
11 // SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
22 /* Copyright (c) 2015 - 2021 Intel Corporation */
33 #include "main.h"
4+#include <linux/net/intel/iidc_rdma_idpf.h>
45
56 MODULE_ALIAS("i40iw");
67 MODULE_DESCRIPTION("Intel(R) Ethernet Protocol Driver for RDMA");
@@ -38,19 +39,7 @@ static void irdma_unregister_notifiers(void) · 31 unmodified lines
3839 unregister_netdevice_notifier(&irdma_netdevice_notifier);
3940 }
4041
41−static void irdma_prep_tc_change(struct irdma_device *iwdev)
42−{
43− iwdev->vsi.tc_change_pending = true;
44− irdma_sc_suspend_resume_qps(&iwdev->vsi, IRDMA_OP_SUSPEND);
45−
46− /* Wait for all qp's to suspend */
47− wait_event_timeout(iwdev->suspend_wq,
48− !atomic_read(&iwdev->vsi.qp_suspend_reqs),
49− msecs_to_jiffies(IRDMA_EVENT_TIMEOUT_MS));
50− irdma_ws_reset(&iwdev->vsi);
51−}
52−
53−static void irdma_log_invalid_mtu(u16 mtu, struct irdma_sc_dev *dev)
42+void irdma_log_invalid_mtu(u16 mtu, struct irdma_sc_dev *dev)
5443 {
5544 if (mtu < IRDMA_MIN_MTU_IPV4)
5645 ibdev_warn(to_ibdev(dev), "MTU setting [%d] too low for RDMA traffic. Minimum MTU is 576 for IPv4\n", mtu);
@@ -58,35 +47,10 @@ static void irdma_log_invalid_mtu(u16 mtu, struct irdma_sc_dev *dev) · 1 unmodified line
5847 ibdev_warn(to_ibdev(dev), "MTU setting [%d] too low for RDMA traffic. Minimum MTU is 1280 for IPv6\\n", mtu);
5948 }
6049
61−static void irdma_fill_qos_info(struct irdma_l2params *l2params,
62− struct iidc_rdma_qos_params *qos_info)
50+static void ig3rdma_idc_vport_event_handler(struct iidc_rdma_vport_dev_info *cdev_info,
51+ struct iidc_rdma_event *event)
6352 {
64− int i;
65−
66− l2params->num_tc = qos_info->num_tc;
67− l2params->vsi_prio_type = qos_info->vport_priority_type;
68− l2params->vsi_rel_bw = qos_info->vport_relative_bw;
69− for (i = 0; i < l2params->num_tc; i++) {
70− l2params->tc_info[i].egress_virt_up =
71− qos_info->tc_info[i].egress_virt_up;
72− l2params->tc_info[i].ingress_virt_up =
73− qos_info->tc_info[i].ingress_virt_up;
74− l2params->tc_info[i].prio_type = qos_info->tc_info[i].prio_type;
75− l2params->tc_info[i].rel_bw = qos_info->tc_info[i].rel_bw;
76− l2params->tc_info[i].tc_ctx = qos_info->tc_info[i].tc_ctx;
77− }
78− for (i = 0; i < IIDC_MAX_USER_PRIORITY; i++)
79− l2params->up2tc[i] = qos_info->up2tc[i];
80− if (qos_info->pfc_mode == IIDC_DSCP_PFC_MODE) {
81− l2params->dscp_mode = true;
82− memcpy(l2params->dscp_map, qos_info->dscp_map, sizeof(l2params->dscp_map));
83− }
84−}
85−
86−static void irdma_iidc_event_handler(struct iidc_rdma_core_dev_info *cdev_info,
87− struct iidc_rdma_event *event)
88−{
89− struct irdma_device *iwdev = dev_get_drvdata(&cdev_info->adev->dev);
53+ struct irdma_device *iwdev = auxiliary_get_drvdata(cdev_info->adev);
9054 struct irdma_l2params l2params = {};
9155
9256 if (*event->type & BIT(IIDC_RDMA_EVENT_AFTER_MTU_CHANGE)) {
@@ -97,248 +61,39 @@ static void irdma_iidc_event_handler(struct iidc_rdma_core_dev_info *cdev_info, · 4 unmodified lines
9761 irdma_log_invalid_mtu(l2params.mtu, &iwdev->rf->sc_dev);
9862 irdma_change_l2params(&iwdev->vsi, &l2params);
9963 }
100− } else if (*event->type & BIT(IIDC_RDMA_EVENT_BEFORE_TC_CHANGE)) {
101− if (iwdev->vsi.tc_change_pending)
102− return;
103−
104− irdma_prep_tc_change(iwdev);
105− } else if (*event->type & BIT(IIDC_RDMA_EVENT_AFTER_TC_CHANGE)) {
106− struct iidc_rdma_priv_dev_info *iidc_priv = cdev_info->iidc_priv;
107−
108− if (!iwdev->vsi.tc_change_pending)
109− return;
110−
111− l2params.tc_changed = true;
112− ibdev_dbg(&iwdev->ibdev, "CLNT: TC Change\n");
113−
114− irdma_fill_qos_info(&l2params, &iidc_priv->qos_info);
115− if (iwdev->rf->protocol_used != IRDMA_IWARP_PROTOCOL_ONLY)
116− iwdev->dcb_vlan_mode =
117− l2params.num_tc > 1 && !l2params.dscp_mode;
118− irdma_change_l2params(&iwdev->vsi, &l2params);
119− } else if (*event->type & BIT(IIDC_RDMA_EVENT_CRIT_ERR)) {
120− ibdev_warn(&iwdev->ibdev, "ICE OICR event notification: oicr = 0x%08x\n",
121− event->reg);
122− if (event->reg & IRDMAPFINT_OICR_PE_CRITERR_M) {
123− u32 pe_criterr;
124−
125− pe_criterr = readl(iwdev->rf->sc_dev.hw_regs[IRDMA_GLPE_CRITERR]);
126−#define IRDMA_Q1_RESOURCE_ERR 0x0001024d
127− if (pe_criterr != IRDMA_Q1_RESOURCE_ERR) {
128− ibdev_err(&iwdev->ibdev, "critical PE Error, GLPE_CRITERR=0x%08x\n",
129− pe_criterr);
130− iwdev->rf->reset = true;
131− } else {
132− ibdev_warn(&iwdev->ibdev, "Q1 Resource Check\n");
133− }
134− }
135− if (event->reg & IRDMAPFINT_OICR_HMC_ERR_M) {
136− ibdev_err(&iwdev->ibdev, "HMC Error\n");
137− iwdev->rf->reset = true;
138− }
139− if (event->reg & IRDMAPFINT_OICR_PE_PUSH_M) {
140− ibdev_err(&iwdev->ibdev, "PE Push Error\n");
141− iwdev->rf->reset = true;
142− }
143− if (iwdev->rf->reset)
144− iwdev->rf->gen_ops.request_reset(iwdev->rf);
14564 }
14665 }
14766
148−/**
149− * irdma_request_reset - Request a reset
150− * @rf: RDMA PCI function
151− */
152−static void irdma_request_reset(struct irdma_pci_f *rf)
153−{
154− ibdev_warn(&rf->iwdev->ibdev, "Requesting a reset\n");
155− ice_rdma_request_reset(rf->cdev, IIDC_FUNC_RESET);
156−}
157−
158−/**
159− * irdma_lan_register_qset - Register qset with LAN driver
160− * @vsi: vsi structure
161− * @tc_node: Traffic class node
162− */
163−static int irdma_lan_register_qset(struct irdma_sc_vsi *vsi,
164− struct irdma_ws_node *tc_node)
165−{
166− struct irdma_device *iwdev = vsi->back_vsi;
167− struct iidc_rdma_core_dev_info *cdev_info;
168− struct iidc_rdma_qset_params qset = {};
169− int ret;
170−
171− cdev_info = iwdev->rf->cdev;
172− qset.qs_handle = tc_node->qs_handle;
173− qset.tc = tc_node->traffic_class;
174− qset.vport_id = vsi->vsi_idx;
175− ret = ice_add_rdma_qset(cdev_info, &qset);
176− if (ret) {
177− ibdev_dbg(&iwdev->ibdev, "WS: LAN alloc_res for rdma qset failed.\n");
178− return ret;
179− }
180−
181− tc_node->l2_sched_node_id = qset.teid;
182− vsi->qos[tc_node->user_pri].l2_sched_node_id = qset.teid;
183−
184− return 0;
185−}
186−
187−/**
188− * irdma_lan_unregister_qset - Unregister qset with LAN driver
189− * @vsi: vsi structure
190− * @tc_node: Traffic class node
191− */
192−static void irdma_lan_unregister_qset(struct irdma_sc_vsi *vsi,
193− struct irdma_ws_node *tc_node)
67+static int ig3rdma_vport_probe(struct auxiliary_device *aux_dev,
68+ const struct auxiliary_device_id *id)
19469 {
195− struct irdma_device *iwdev = vsi->back_vsi;
196− struct iidc_rdma_core_dev_info *cdev_info;
197− struct iidc_rdma_qset_params qset = {};
198−
199− cdev_info = iwdev->rf->cdev;
200− qset.qs_handle = tc_node->qs_handle;
201− qset.tc = tc_node->traffic_class;
202− qset.vport_id = vsi->vsi_idx;
203− qset.teid = tc_node->l2_sched_node_id;
204−
205− if (ice_del_rdma_qset(cdev_info, &qset))
206− ibdev_dbg(&iwdev->ibdev, "WS: LAN free_res for rdma qset failed.\n");
207−}
208−
209−static int irdma_init_interrupts(struct irdma_pci_f *rf, struct iidc_rdma_core_dev_info *cdev)
210−{
211− int i;
212−
213− rf->msix_count = num_online_cpus() + IRDMA_NUM_AEQ_MSIX;
214− rf->msix_entries = kcalloc(rf->msix_count, sizeof(*rf->msix_entries),
215− GFP_KERNEL);
216− if (!rf->msix_entries)
217− return -ENOMEM;
218−
219− for (i = 0; i < rf->msix_count; i++)
220− if (ice_alloc_rdma_qvector(cdev, &rf->msix_entries[i]))
221− break;
222−
223− if (i < IRDMA_MIN_MSIX) {
224− while (--i >= 0)
225− ice_free_rdma_qvector(cdev, &rf->msix_entries[i]);
70+ struct iidc_rdma_vport_auxiliary_dev *idc_adev =
71+ container_of(aux_dev, struct iidc_rdma_vport_auxiliary_dev, adev);
72+ struct auxiliary_device *aux_core_dev = idc_adev->vdev_info->core_adev;
73+ struct irdma_pci_f *rf = auxiliary_get_drvdata(aux_core_dev);
74+ struct irdma_l2params l2params = {};
75+ struct irdma_device *iwdev;
76+ int err;
22677
227− kfree(rf->msix_entries);
78+ if (!rf) {
79+ WARN_ON_ONCE(1);
22880 return -ENOMEM;
22981 }
230−
231− rf->msix_count = i;
232−
233− return 0;
234−}
235−
236−static void irdma_deinit_interrupts(struct irdma_pci_f *rf, struct iidc_rdma_core_dev_info *cdev)
237−{
238− int i;
239−
240− for (i = 0; i < rf->msix_count; i++)
241− ice_free_rdma_qvector(cdev, &rf->msix_entries[i]);
242−
243− kfree(rf->msix_entries);
244−}
245−
246−static void irdma_remove(struct auxiliary_device *aux_dev)
247−{
248− struct irdma_device *iwdev = auxiliary_get_drvdata(aux_dev);
249− struct iidc_rdma_core_auxiliary_dev *iidc_adev;
250− struct iidc_rdma_core_dev_info *cdev_info;
251−
252− iidc_adev = container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev);
253− cdev_info = iidc_adev->cdev_info;
254−
255− ice_rdma_update_vsi_filter(cdev_info, iwdev->vsi_num, false);
256− irdma_ib_unregister_device(iwdev);
257− irdma_deinit_interrupts(iwdev->rf, cdev_info);
258−
259− kfree(iwdev->rf);
260−
261− pr_debug("INIT: Gen2 PF[%d] device remove success\n", PCI_FUNC(cdev_info->pdev->devfn));
262−}
263−
264−static void irdma_fill_device_info(struct irdma_device *iwdev,
265− struct iidc_rdma_core_dev_info *cdev_info)
266−{
267− struct iidc_rdma_priv_dev_info *iidc_priv = cdev_info->iidc_priv;
268− struct irdma_pci_f *rf = iwdev->rf;
269−
270− rf->sc_dev.hw = &rf->hw;
271− rf->iwdev = iwdev;
272− rf->cdev = cdev_info;
273− rf->hw.hw_addr = iidc_priv->hw_addr;
274− rf->pcidev = cdev_info->pdev;
275− rf->hw.device = &rf->pcidev->dev;
276− rf->pf_id = iidc_priv->pf_id;
277− rf->gen_ops.register_qset = irdma_lan_register_qset;
278− rf->gen_ops.unregister_qset = irdma_lan_unregister_qset;
279−
280− rf->default_vsi.vsi_idx = iidc_priv->vport_id;
281− rf->protocol_used =
282− cdev_info->rdma_protocol == IIDC_RDMA_PROTOCOL_ROCEV2 ?
283− IRDMA_ROCE_PROTOCOL_ONLY : IRDMA_IWARP_PROTOCOL_ONLY;
284− rf->rdma_ver = IRDMA_GEN_2;
285− rf->rsrc_profile = IRDMA_HMC_PROFILE_DEFAULT;
286− rf->rst_to = IRDMA_RST_TIMEOUT_HZ;
287− rf->gen_ops.request_reset = irdma_request_reset;
288− rf->limits_sel = 7;
289− rf->iwdev = iwdev;
290−
291− mutex_init(&iwdev->ah_tbl_lock);
292−
293− iwdev->netdev = iidc_priv->netdev;
294− iwdev->vsi_num = iidc_priv->vport_id;
82+ iwdev = ib_alloc_device(irdma_device, ibdev);
83+ /* Fill iwdev info */
84+ iwdev->is_vport = true;
85+ iwdev->rf = rf;
86+ iwdev->vport_id = idc_adev->vdev_info->vport_id;
87+ iwdev->netdev = idc_adev->vdev_info->netdev;
29588 iwdev->init_state = INITIAL_STATE;
29689 iwdev->roce_cwnd = IRDMA_ROCE_CWND_DEFAULT;
29790 iwdev->roce_ackcreds = IRDMA_ROCE_ACKCREDS_DEFAULT;
29891 iwdev->rcv_wnd = IRDMA_CM_DEFAULT_RCV_WND_SCALED;
29992 iwdev->rcv_wscale = IRDMA_CM_DEFAULT_RCV_WND_SCALE;
300− if (rf->protocol_used == IRDMA_ROCE_PROTOCOL_ONLY)
301− iwdev->roce_mode = true;
302−}
303−
304−static int irdma_probe(struct auxiliary_device *aux_dev, const struct auxiliary_device_id *id)
305−{
306− struct iidc_rdma_core_auxiliary_dev *iidc_adev;
307− struct iidc_rdma_core_dev_info *cdev_info;
308− struct iidc_rdma_priv_dev_info *iidc_priv;
309− struct irdma_l2params l2params = {};
310− struct irdma_device *iwdev;
311− struct irdma_pci_f *rf;
312− int err;
313−
314− iidc_adev = container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev);
315− cdev_info = iidc_adev->cdev_info;
316− iidc_priv = cdev_info->iidc_priv;
317−
318− iwdev = ib_alloc_device(irdma_device, ibdev);
319− if (!iwdev)
320− return -ENOMEM;
321− iwdev->rf = kzalloc(sizeof(*rf), GFP_KERNEL);
322− if (!iwdev->rf) {
323− ib_dealloc_device(&iwdev->ibdev);
324− return -ENOMEM;
325− }
326−
327− irdma_fill_device_info(iwdev, cdev_info);
328− rf = iwdev->rf;
329−
330− err = irdma_init_interrupts(rf, cdev_info);
331− if (err)
332− goto err_init_interrupts;
333−
334− err = irdma_ctrl_init_hw(rf);
335− if (err)
336− goto err_ctrl_init;
93+ iwdev->roce_mode = true;
94+ iwdev->push_mode = false;
33795
33896 l2params.mtu = iwdev->netdev->mtu;
339− irdma_fill_qos_info(&l2params, &iidc_priv->qos_info);
340− if (iwdev->rf->protocol_used != IRDMA_IWARP_PROTOCOL_ONLY)
341− iwdev->dcb_vlan_mode = l2params.num_tc > 1 && !l2params.dscp_mode;
34297
34398 err = irdma_rt_init_hw(iwdev, &l2params);
34499 if (err)
@@ -348,43 +103,57 @@ static int irdma_probe(struct auxiliary_device *aux_dev, const struct auxiliary_ · 3 unmodified lines
348103 if (err)
349104 goto err_ibreg;
350105
351− ice_rdma_update_vsi_filter(cdev_info, iwdev->vsi_num, true);
352−
353− ibdev_dbg(&iwdev->ibdev, "INIT: Gen2 PF[%d] device probe success\n", PCI_FUNC(rf->pcidev->devfn));
354106 auxiliary_set_drvdata(aux_dev, iwdev);
355107
356− return 0;
108+ ibdev_dbg(&iwdev->ibdev,
109+ "INIT: Gen[%d] vport[%d] probe success. dev_name = %s, core_dev_name = %s, netdev=%s\n",
110+ rf->rdma_ver, idc_adev->vdev_info->vport_id,
111+ dev_name(&aux_dev->dev),
112+ dev_name(&idc_adev->vdev_info->core_adev->dev),
113+ netdev_name(idc_adev->vdev_info->netdev));
357114
115+ return 0;
358116 err_ibreg:
359117 irdma_rt_deinit_hw(iwdev);
360118 err_rt_init:
361− irdma_ctrl_deinit_hw(rf);
362−err_ctrl_init:
363− irdma_deinit_interrupts(rf, cdev_info);
364−err_init_interrupts:
365− kfree(iwdev->rf);
366119 ib_dealloc_device(&iwdev->ibdev);
367120
368121 return err;
369122 }
370123
371−static const struct auxiliary_device_id irdma_auxiliary_id_table[] = {
372− {.name = "ice.iwarp", },
373− {.name = "ice.roce", },
124+static void ig3rdma_vport_remove(struct auxiliary_device *aux_dev)
125+{
126+ struct iidc_rdma_vport_auxiliary_dev *idc_adev =
127+ container_of(aux_dev, struct iidc_rdma_vport_auxiliary_dev, adev);
128+ struct irdma_device *iwdev = auxiliary_get_drvdata(aux_dev);
129+
130+ ibdev_dbg(&iwdev->ibdev,
131+ "INIT: Gen[%d] dev_name = %s, core_dev_name = %s, netdev=%s\n",
132+ iwdev->rf->rdma_ver, dev_name(&aux_dev->dev),
133+ dev_name(&idc_adev->vdev_info->core_adev->dev),
134+ netdev_name(idc_adev->vdev_info->netdev));
135+
136+ irdma_ib_unregister_device(iwdev);
137+}
138+
139+static const struct auxiliary_device_id ig3rdma_vport_auxiliary_id_table[] = {
140+ {.name = "idpf.8086.rdma.vdev", },
374141 {},
375142 };
376143
377−MODULE_DEVICE_TABLE(auxiliary, irdma_auxiliary_id_table);
144+MODULE_DEVICE_TABLE(auxiliary, ig3rdma_vport_auxiliary_id_table);
378145
379−static struct iidc_rdma_core_auxiliary_drv irdma_auxiliary_drv = {
146+static struct iidc_rdma_vport_auxiliary_drv ig3rdma_vport_auxiliary_drv = {
380147 .adrv = {
381− .id_table = irdma_auxiliary_id_table,
382− .probe = irdma_probe,
383− .remove = irdma_remove,
148+ .name = "vdev",
149+ .id_table = ig3rdma_vport_auxiliary_id_table,
150+ .probe = ig3rdma_vport_probe,
151+ .remove = ig3rdma_vport_remove,
384152 },
385− .event_handler = irdma_iidc_event_handler,
153+ .event_handler = ig3rdma_idc_vport_event_handler,
386154 };
387155
156+
388157 static int __init irdma_init_module(void)
389158 {
390159 int ret;
@@ -396,14 +165,34 @@ static int __init irdma_init_module(void) · 5 unmodified lines
396165 return ret;
397166 }
398167
399− ret = auxiliary_driver_register(&irdma_auxiliary_drv.adrv);
168+ ret = auxiliary_driver_register(&icrdma_core_auxiliary_drv.adrv);
169+ if (ret) {
170+ auxiliary_driver_unregister(&i40iw_auxiliary_drv);
171+ pr_err("Failed icrdma(gen_2) auxiliary_driver_register() ret=%d\n",
172+ ret);
173+ return ret;
174+ }
175+
176+ ret = auxiliary_driver_register(&ig3rdma_core_auxiliary_drv.adrv);
400177 if (ret) {
178+ auxiliary_driver_unregister(&icrdma_core_auxiliary_drv.adrv);
401179 auxiliary_driver_unregister(&i40iw_auxiliary_drv);
402− pr_err("Failed irdma auxiliary_driver_register() ret=%d\n",
180+ pr_err("Failed ig3rdma(gen_3) core auxiliary_driver_register() ret=%d\n",
403181 ret);
182+
404183 return ret;
405184 }
406185
186+ ret = auxiliary_driver_register(&ig3rdma_vport_auxiliary_drv.adrv);
187+ if (ret) {
188+ auxiliary_driver_unregister(&ig3rdma_core_auxiliary_drv.adrv);
189+ auxiliary_driver_unregister(&icrdma_core_auxiliary_drv.adrv);
190+ auxiliary_driver_unregister(&i40iw_auxiliary_drv);
191+ pr_err("Failed ig3rdma vport auxiliary_driver_register() ret=%d\n",
192+ ret);
193+
194+ return ret;
195+ }
407196 irdma_register_notifiers();
408197
409198 return 0;
@@ -412,8 +201,10 @@ static int __init irdma_init_module(void) · 2 unmodified lines
412201 static void __exit irdma_exit_module(void)
413202 {
414203 irdma_unregister_notifiers();
415− auxiliary_driver_unregister(&irdma_auxiliary_drv.adrv);
204+ auxiliary_driver_unregister(&icrdma_core_auxiliary_drv.adrv);
416205 auxiliary_driver_unregister(&i40iw_auxiliary_drv);
206+ auxiliary_driver_unregister(&ig3rdma_core_auxiliary_drv.adrv);
207+ auxiliary_driver_unregister(&ig3rdma_vport_auxiliary_drv.adrv);
417208 }
418209
419210 module_init(irdma_init_module);
drivers/infiniband/hw/irdma/main.h+27−8
@@ -30,7 +30,6 @@ · 29 unmodified lines
3030 #endif
3131 #include <linux/auxiliary_bus.h>
3232 #include <linux/net/intel/iidc_rdma.h>
33−#include <linux/net/intel/iidc_rdma_ice.h>
3433 #include <rdma/ib_smi.h>
3534 #include <rdma/ib_verbs.h>
3635 #include <rdma/ib_pack.h>
@@ -54,6 +53,8 @@ · 17 unmodified lines
5453 #include "puda.h"
5554
5655 extern struct auxiliary_driver i40iw_auxiliary_drv;
56+extern struct iidc_rdma_core_auxiliary_drv icrdma_core_auxiliary_drv;
57+extern struct iidc_rdma_core_auxiliary_drv ig3rdma_core_auxiliary_drv;
5758
5859 #define IRDMA_FW_VER_DEFAULT 2
5960 #define IRDMA_HW_VER 2
@@ -65,7 +66,8 @@ extern struct auxiliary_driver i40iw_auxiliary_drv; · 5 unmodified lines
6566 #define IRDMA_MACIP_ADD 1
6667 #define IRDMA_MACIP_DELETE 2
6768
68−#define IW_CCQ_SIZE (IRDMA_CQP_SW_SQSIZE_2048 + 1)
69+#define IW_GEN_3_CCQ_SIZE (2 * IRDMA_CQP_SW_SQSIZE_2048 + 2)
70+#define IW_CCQ_SIZE (IRDMA_CQP_SW_SQSIZE_2048 + 2)
6971 #define IW_CEQ_SIZE 2048
7072 #define IW_AEQ_SIZE 2048
7173
@@ -127,12 +129,12 @@ enum init_completion_state { · 55 unmodified lines
127129 HMC_OBJS_CREATED,
128130 HW_RSRC_INITIALIZED,
129131 CCQ_CREATED,
130− CEQ0_CREATED, /* Last state of probe */
131− ILQ_CREATED,
132− IEQ_CREATED,
132+ CEQ0_CREATED,
133133 CEQS_CREATED,
134134 PBLE_CHUNK_MEM,
135135 AEQ_CREATED,
136+ ILQ_CREATED,
137+ IEQ_CREATED, /* Last state of probe */
136138 IP_ADDR_REGISTERED, /* Last state of open */
137139 };
138140
@@ -167,6 +169,7 @@ struct irdma_cqp_request { · 28 unmodified lines
167169 bool request_done; /* READ/WRITE_ONCE macros operate on it */
168170 bool waiting:1;
169171 bool dynamic:1;
172+ bool pending:1;
170173 };
171174
172175 struct irdma_cqp {
@@ -179,6 +182,7 @@ struct irdma_cqp { · 6 unmodified lines
179182 struct irdma_dma_mem host_ctx;
180183 u64 *scratch_array;
181184 struct irdma_cqp_request *cqp_requests;
185+ struct irdma_ooo_cqp_op *oop_op_array;
182186 struct list_head cqp_avail_reqs;
183187 struct list_head cqp_pending_reqs;
184188 };
@@ -257,6 +261,7 @@ struct irdma_pci_f { · 72 unmodified lines
257261 bool reset:1;
258262 bool rsrc_created:1;
259263 bool msix_shared:1;
264+ bool hwqp1_rsvd:1;
260265 u8 rsrc_profile;
261266 u8 *hmc_info_mem;
262267 u8 *mem_rsrc;
@@ -269,6 +274,8 @@ struct irdma_pci_f { · 6 unmodified lines
269274 u32 max_mr;
270275 u32 max_qp;
271276 u32 max_cq;
277+ u32 max_srq;
278+ u32 next_srq;
272279 u32 max_ah;
273280 u32 next_ah;
274281 u32 max_mcg;
@@ -282,6 +289,7 @@ struct irdma_pci_f { · 7 unmodified lines
282289 u32 mr_stagmask;
283290 u32 used_pds;
284291 u32 used_cqs;
292+ u32 used_srqs;
285293 u32 used_mrs;
286294 u32 used_qps;
287295 u32 arp_table_size;
@@ -293,6 +301,7 @@ struct irdma_pci_f { · 5 unmodified lines
293301 unsigned long *allocated_ws_nodes;
294302 unsigned long *allocated_qps;
295303 unsigned long *allocated_cqs;
304+ unsigned long *allocated_srqs;
296305 unsigned long *allocated_mrs;
297306 unsigned long *allocated_pds;
298307 unsigned long *allocated_mcgs;
@@ -327,10 +336,13 @@ struct irdma_pci_f { · 28 unmodified lines
327336 wait_queue_head_t vchnl_waitq;
328337 struct workqueue_struct *cqp_cmpl_wq;
329338 struct work_struct cqp_cmpl_work;
339+ struct workqueue_struct *vchnl_wq;
330340 struct irdma_sc_vsi default_vsi;
331341 void *back_fcn;
332342 struct irdma_gen_ops gen_ops;
333343 struct irdma_device *iwdev;
344+ DECLARE_HASHTABLE(ah_hash_tbl, 8);
345+ struct mutex ah_tbl_lock; /* protect AH hash table access */
334346 };
335347
336348 struct irdma_device {
@@ -340,8 +352,6 @@ struct irdma_device { · 3 unmodified lines
340352 struct workqueue_struct *cleanup_wq;
341353 struct irdma_sc_vsi vsi;
342354 struct irdma_cm_core cm_core;
343− DECLARE_HASHTABLE(ah_hash_tbl, 8);
344− struct mutex ah_tbl_lock; /* protect AH hash table access */
345355 u32 roce_cwnd;
346356 u32 roce_ackcreds;
347357 u32 vendor_id;
@@ -350,12 +360,14 @@ struct irdma_device { · 2 unmodified lines
350360 u32 rcv_wnd;
351361 u16 mac_ip_table_idx;
352362 u16 vsi_num;
363+ u16 vport_id;
353364 u8 rcv_wscale;
354365 u8 iw_status;
355366 bool roce_mode:1;
356367 bool roce_dcqcn_en:1;
357368 bool dcb_vlan_mode:1;
358369 bool iw_ooo:1;
370+ bool is_vport:1;
359371 enum init_completion_state init_state;
360372
361373 wait_queue_head_t suspend_wq;
@@ -413,6 +425,11 @@ static inline struct irdma_pci_f *dev_to_rf(struct irdma_sc_dev *dev) · 51 unmodified lines
413425 return container_of(dev, struct irdma_pci_f, sc_dev);
414426 }
415427
428+static inline struct irdma_srq *to_iwsrq(struct ib_srq *ibsrq)
429+{
430+ return container_of(ibsrq, struct irdma_srq, ibsrq);
431+}
432+
416433 /**
417434 * irdma_alloc_resource - allocate a resource
418435 * @iwdev: device pointer
@@ -508,7 +525,8 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 89 unmodified lines
508525 void irdma_cq_add_ref(struct ib_cq *ibcq);
509526 void irdma_cq_rem_ref(struct ib_cq *ibcq);
510527 void irdma_cq_wq_destroy(struct irdma_pci_f *rf, struct irdma_sc_cq *cq);
511−
528+void irdma_srq_event(struct irdma_sc_srq *srq);
529+void irdma_srq_wq_destroy(struct irdma_pci_f *rf, struct irdma_sc_srq *srq);
512530 void irdma_cleanup_pending_cqp_op(struct irdma_pci_f *rf);
513531 int irdma_hw_modify_qp(struct irdma_device *iwdev, struct irdma_qp *iwqp,
514532 struct irdma_modify_qp_info *info, bool wait);
@@ -557,4 +575,5 @@ int irdma_netdevice_event(struct notifier_block *notifier, unsigned long event, · 42 unmodified lines
557575 void *ptr);
558576 void irdma_add_ip(struct irdma_device *iwdev);
559577 void cqp_compl_worker(struct work_struct *work);
578+void irdma_log_invalid_mtu(u16 mtu, struct irdma_sc_dev *dev);
560579 #endif /* IRDMA_MAIN_H */
drivers/infiniband/hw/irdma/pble.c+14−6
@@ -193,8 +193,15 @@ static enum irdma_sd_entry_type irdma_get_type(struct irdma_sc_dev *dev, · 192 unmodified lines
193193 {
194194 enum irdma_sd_entry_type sd_entry_type;
195195
196− sd_entry_type = !idx->rel_pd_idx && pages == IRDMA_HMC_PD_CNT_IN_SD ?
197− IRDMA_SD_TYPE_DIRECT : IRDMA_SD_TYPE_PAGED;
196+ if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3)
197+ sd_entry_type = (!idx->rel_pd_idx &&
198+ pages == IRDMA_HMC_PD_CNT_IN_SD) ?
199+ IRDMA_SD_TYPE_DIRECT : IRDMA_SD_TYPE_PAGED;
200+ else
201+ sd_entry_type = (!idx->rel_pd_idx &&
202+ pages == IRDMA_HMC_PD_CNT_IN_SD &&
203+ dev->privileged) ?
204+ IRDMA_SD_TYPE_DIRECT : IRDMA_SD_TYPE_PAGED;
198205 return sd_entry_type;
199206 }
200207
@@ -279,10 +286,11 @@ static int add_pble_prm(struct irdma_hmc_pble_rsrc *pble_rsrc) · 78 unmodified lines
279286 sd_reg_val = (sd_entry_type == IRDMA_SD_TYPE_PAGED) ?
280287 sd_entry->u.pd_table.pd_page_addr.pa :
281288 sd_entry->u.bp.addr.pa;
282−
283− if (!sd_entry->valid) {
284− ret_code = irdma_hmc_sd_one(dev, hmc_info->hmc_fn_id, sd_reg_val,
285− idx->sd_idx, sd_entry->entry_type, true);
289+ if ((dev->privileged && !sd_entry->valid) ||
290+ dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) {
291+ ret_code = irdma_hmc_sd_one(dev, hmc_info->hmc_fn_id,
292+ sd_reg_val, idx->sd_idx,
293+ sd_entry->entry_type, true);
286294 if (ret_code)
287295 goto error;
288296 }
drivers/infiniband/hw/irdma/protos.h+1−0
@@ -10,6 +10,7 @@ · 9 unmodified lines
1010 #define ALL_TC2PFC 0xff
1111 #define CQP_COMPL_WAIT_TIME_MS 10
1212 #define CQP_TIMEOUT_THRESHOLD 500
13+#define CQP_DEF_CMPL_TIMEOUT_THRESHOLD 2500
1314
1415 /* init operations */
1516 int irdma_sc_dev_init(enum irdma_vers ver, struct irdma_sc_dev *dev,
drivers/infiniband/hw/irdma/puda.h+2−2
@@ -91,7 +91,7 @@ struct irdma_puda_rsrc_info { · 90 unmodified lines
9191 u32 rq_size;
9292 u32 tx_buf_cnt; /* total bufs allocated will be rq_size + tx_buf_cnt */
9393 u16 buf_size;
94− u8 stats_idx;
94+ u16 stats_idx;
9595 bool stats_idx_valid:1;
9696 int abi_ver;
9797 };
@@ -140,7 +140,7 @@ struct irdma_puda_rsrc { · 42 unmodified lines
140140 u64 crc_err;
141141 u64 pmode_count;
142142 u64 partials_handled;
143− u8 stats_idx;
143+ u16 stats_idx;
144144 bool check_crc:1;
145145 bool stats_idx_valid:1;
146146 };
drivers/infiniband/hw/irdma/type.h+203−18
@@ -8,6 +8,8 @@ · 7 unmodified lines
88 #include "hmc.h"
99 #include "uda.h"
1010 #include "ws.h"
11+#include "virtchnl.h"
12+
1113 #define IRDMA_DEBUG_ERR "ERR"
1214 #define IRDMA_DEBUG_INIT "INIT"
1315 #define IRDMA_DEBUG_DEV "DEV"
@@ -95,12 +97,6 @@ enum irdma_term_mpa_errors { · 81 unmodified lines
9597 MPA_REQ_RSP = 0x04,
9698 };
9799
98−enum irdma_qp_event_type {
99− IRDMA_QP_EVENT_CATASTROPHIC,
100− IRDMA_QP_EVENT_ACCESS_ERR,
101− IRDMA_QP_EVENT_REQ_ERR,
102−};
103−
104100 enum irdma_hw_stats_index {
105101 /* gen1 - 32-bit */
106102 IRDMA_HW_STAT_INDEX_IP4RXDISCARD = 0,
@@ -154,12 +150,46 @@ enum irdma_hw_stats_index { · 47 unmodified lines
154150 IRDMA_HW_STAT_INDEX_RXRPCNPIGNORED = 44,
155151 IRDMA_HW_STAT_INDEX_TXNPCNPSENT = 45,
156152 IRDMA_HW_STAT_INDEX_MAX_GEN_2 = 46,
153+
154+ /* gen3 */
155+ IRDMA_HW_STAT_INDEX_RNR_SENT = 46,
156+ IRDMA_HW_STAT_INDEX_RNR_RCVD = 47,
157+ IRDMA_HW_STAT_INDEX_RDMAORDLMTCNT = 48,
158+ IRDMA_HW_STAT_INDEX_RDMAIRDLMTCNT = 49,
159+ IRDMA_HW_STAT_INDEX_RDMARXATS = 50,
160+ IRDMA_HW_STAT_INDEX_RDMATXATS = 51,
161+ IRDMA_HW_STAT_INDEX_NAKSEQERR = 52,
162+ IRDMA_HW_STAT_INDEX_NAKSEQERR_IMPLIED = 53,
163+ IRDMA_HW_STAT_INDEX_RTO = 54,
164+ IRDMA_HW_STAT_INDEX_RXOOOPKTS = 55,
165+ IRDMA_HW_STAT_INDEX_ICRCERR = 56,
166+
167+ IRDMA_HW_STAT_INDEX_MAX_GEN_3 = 57,
157168 };
158169
159170 enum irdma_feature_type {
160171 IRDMA_FEATURE_FW_INFO = 0,
161172 IRDMA_HW_VERSION_INFO = 1,
173+ IRDMA_QP_MAX_INCR = 2,
174+ IRDMA_CQ_MAX_INCR = 3,
175+ IRDMA_CEQ_MAX_INCR = 4,
176+ IRDMA_SD_MAX_INCR = 5,
177+ IRDMA_MR_MAX_INCR = 6,
178+ IRDMA_Q1_MAX_INCR = 7,
179+ IRDMA_AH_MAX_INCR = 8,
180+ IRDMA_SRQ_MAX_INCR = 9,
181+ IRDMA_TIMER_MAX_INCR = 10,
182+ IRDMA_XF_MAX_INCR = 11,
183+ IRDMA_RRF_MAX_INCR = 12,
184+ IRDMA_PBLE_MAX_INCR = 13,
185+ IRDMA_OBJ_1 = 22,
186+ IRDMA_OBJ_2 = 23,
187+ IRDMA_ENDPT_TRK = 24,
188+ IRDMA_FTN_INLINE_MAX = 25,
162189 IRDMA_QSETS_MAX = 26,
190+ IRDMA_ASO = 27,
191+ IRDMA_FTN_FLAGS = 32,
192+ IRDMA_FTN_NOP = 33,
163193 IRDMA_MAX_FEATURES, /* Must be last entry */
164194 };
165195
@@ -206,6 +236,7 @@ enum irdma_syn_rst_handling { · 40 unmodified lines
206236 enum irdma_queue_type {
207237 IRDMA_QUEUE_TYPE_SQ_RQ = 0,
208238 IRDMA_QUEUE_TYPE_CQP,
239+ IRDMA_QUEUE_TYPE_SRQ,
209240 };
210241
211242 struct irdma_sc_dev;
@@ -233,12 +264,22 @@ struct irdma_cqp_init_info { · 21 unmodified lines
233264 __le64 *host_ctx;
234265 u64 *scratch_array;
235266 u32 sq_size;
267+ struct irdma_ooo_cqp_op *ooo_op_array;
268+ u32 pe_en_vf_cnt;
236269 u16 hw_maj_ver;
237270 u16 hw_min_ver;
238271 u8 struct_ver;
239272 u8 hmc_profile;
240273 u8 ena_vf_count;
241274 u8 ceqs_per_vf;
275+ u8 ooisc_blksize;
276+ u8 rrsp_blksize;
277+ u8 q1_blksize;
278+ u8 xmit_blksize;
279+ u8 ts_override;
280+ u8 ts_shift;
281+ u8 en_fine_grained_timers;
282+ u8 blksizes_valid;
242283 bool en_datacenter_tcp:1;
243284 bool disable_packed:1;
244285 bool rocev2_rto_policy:1;
@@ -310,9 +351,21 @@ struct irdma_vsi_pestat { · 65 unmodified lines
310351 spinlock_t lock; /* rdma stats lock */
311352 };
312353
354+struct irdma_mmio_region {
355+ u8 __iomem *addr;
356+ resource_size_t len;
357+ resource_size_t offset;
358+};
359+
313360 struct irdma_hw {
314− u8 __iomem *hw_addr;
315− u8 __iomem *priv_hw_addr;
361+ union {
362+ u8 __iomem *hw_addr;
363+ struct {
364+ struct irdma_mmio_region rdma_reg; /* RDMA region */
365+ struct irdma_mmio_region *io_regs; /* Non-RDMA MMIO regions */
366+ u16 num_io_regions; /* Number of Non-RDMA MMIO regions */
367+ };
368+ };
316369 struct device *device;
317370 struct irdma_hmc_info hmc;
318371 };
@@ -351,7 +404,21 @@ struct irdma_cqp_quanta { · 32 unmodified lines
351404 __le64 elem[IRDMA_CQP_WQE_SIZE];
352405 };
353406
407+struct irdma_ooo_cqp_op {
408+ struct list_head list_entry;
409+ u64 scratch;
410+ u32 def_info;
411+ u32 sw_def_info;
412+ u32 wqe_idx;
413+ bool deferred:1;
414+};
415+
354416 struct irdma_sc_cqp {
417+ spinlock_t ooo_list_lock; /* protects list of pending completions */
418+ struct list_head ooo_avail;
419+ struct list_head ooo_pnd;
420+ u32 last_def_cmpl_ticket;
421+ u32 sw_def_cmpl_ticket;
355422 u32 size;
356423 u64 sq_pa;
357424 u64 host_ctx_pa;
@@ -367,8 +434,10 @@ struct irdma_sc_cqp { · 9 unmodified lines
367434 u64 *scratch_array;
368435 u64 requested_ops;
369436 atomic64_t completed_ops;
437+ struct irdma_ooo_cqp_op *ooo_op_array;
370438 u32 cqp_id;
371439 u32 sq_size;
440+ u32 pe_en_vf_cnt;
372441 u32 hw_sq_size;
373442 u16 hw_maj_ver;
374443 u16 hw_min_ver;
@@ -378,6 +447,14 @@ struct irdma_sc_cqp { · 3 unmodified lines
378447 u8 ena_vf_count;
379448 u8 timeout_count;
380449 u8 ceqs_per_vf;
450+ u8 ooisc_blksize;
451+ u8 rrsp_blksize;
452+ u8 q1_blksize;
453+ u8 xmit_blksize;
454+ u8 ts_override;
455+ u8 ts_shift;
456+ u8 en_fine_grained_timers;
457+ u8 blksizes_valid;
381458 bool en_datacenter_tcp:1;
382459 bool disable_packed:1;
383460 bool rocev2_rto_policy:1;
@@ -397,6 +474,8 @@ struct irdma_sc_aeq { · 13 unmodified lines
397474 u32 msix_idx;
398475 u8 polarity;
399476 bool virtual_map:1;
477+ bool pasid_valid:1;
478+ u32 pasid;
400479 };
401480
402481 struct irdma_sc_ceq {
@@ -412,13 +491,15 @@ struct irdma_sc_ceq { · 9 unmodified lines
412491 u8 tph_val;
413492 u32 first_pm_pbl_idx;
414493 u8 polarity;
415− struct irdma_sc_vsi *vsi;
494+ u16 vsi_idx;
416495 struct irdma_sc_cq **reg_cq;
417496 u32 reg_cq_size;
418497 spinlock_t req_cq_lock; /* protect access to reg_cq array */
419498 bool virtual_map:1;
420499 bool tph_en:1;
421500 bool itr_no_expire:1;
501+ bool pasid_valid:1;
502+ u32 pasid;
422503 };
423504
424505 struct irdma_sc_cq {
@@ -426,6 +507,7 @@ struct irdma_sc_cq { · 1 unmodified line
426507 u64 cq_pa;
427508 u64 shadow_area_pa;
428509 struct irdma_sc_dev *dev;
510+ u16 vsi_idx;
429511 struct irdma_sc_vsi *vsi;
430512 void *pbl_list;
431513 void *back_cq;
@@ -477,8 +559,13 @@ struct irdma_sc_qp { · 45 unmodified lines
477559 bool virtual_map:1;
478560 bool flush_sq:1;
479561 bool flush_rq:1;
562+ bool err_sq_idx_valid:1;
563+ bool err_rq_idx_valid:1;
564+ u32 err_sq_idx;
565+ u32 err_rq_idx;
480566 bool sq_flush_code:1;
481567 bool rq_flush_code:1;
568+ u32 pkt_limit;
482569 enum irdma_flush_opcode flush_code;
483570 enum irdma_qp_event_type event_type;
484571 u8 term_flags;
@@ -489,13 +576,13 @@ struct irdma_sc_qp { · 4 unmodified lines
489576 struct irdma_stats_inst_info {
490577 bool use_hmc_fcn_index;
491578 u8 hmc_fn_id;
492− u8 stats_idx;
579+ u16 stats_idx;
493580 };
494581
495582 struct irdma_up_info {
496583 u8 map[8];
497584 u8 cnp_up_override;
498− u8 hmc_fcn_idx;
585+ u16 hmc_fcn_idx;
499586 bool use_vlan:1;
500587 bool use_cnp_up_override:1;
501588 };
@@ -518,6 +605,8 @@ struct irdma_ws_node_info { · 16 unmodified lines
518605 struct irdma_hmc_fpm_misc {
519606 u32 max_ceqs;
520607 u32 max_sds;
608+ u32 loc_mem_pages;
609+ u8 ird;
521610 u32 xf_block_size;
522611 u32 q1_block_size;
523612 u32 ht_multiplier;
@@ -526,6 +615,7 @@ struct irdma_hmc_fpm_misc { · 2 unmodified lines
526615 u32 ooiscf_block_size;
527616 };
528617
618+#define IRDMA_VCHNL_MAX_MSG_SIZE 512
529619 #define IRDMA_LEAF_DEFAULT_REL_BW 64
530620 #define IRDMA_PARENT_DEFAULT_REL_BW 1
531621
@@ -601,19 +691,28 @@ struct irdma_sc_dev { · 69 unmodified lines
601691 u64 cqp_cmd_stats[IRDMA_MAX_CQP_OPS];
602692 struct irdma_hw_attrs hw_attrs;
603693 struct irdma_hmc_info *hmc_info;
694+ struct irdma_vchnl_rdma_caps vc_caps;
695+ u8 vc_recv_buf[IRDMA_VCHNL_MAX_MSG_SIZE];
696+ u16 vc_recv_len;
604697 struct irdma_sc_cqp *cqp;
605698 struct irdma_sc_aeq *aeq;
606699 struct irdma_sc_ceq *ceq[IRDMA_CEQ_MAX_COUNT];
607700 struct irdma_sc_cq *ccq;
608701 const struct irdma_irq_ops *irq_ops;
702+ struct irdma_qos qos[IRDMA_MAX_USER_PRIORITY];
609703 struct irdma_hmc_fpm_misc hmc_fpm_misc;
610704 struct irdma_ws_node *ws_tree_root;
611705 struct mutex ws_mutex; /* ws tree mutex */
706+ u32 vchnl_ver;
612707 u16 num_vfs;
613− u8 hmc_fn_id;
708+ u16 hmc_fn_id;
614709 u8 vf_id;
710+ bool privileged:1;
615711 bool vchnl_up:1;
616712 bool ceq_valid:1;
713+ bool is_pf:1;
714+ u8 protocol_used;
715+ struct mutex vchnl_mutex; /* mutex to synchronize RDMA virtual channel messages */
617716 u8 pci_rev;
618717 int (*ws_add)(struct irdma_sc_vsi *vsi, u8 user_pri);
619718 void (*ws_remove)(struct irdma_sc_vsi *vsi, u8 user_pri);
@@ -632,6 +731,51 @@ struct irdma_modify_cq_info { · 12 unmodified lines
632731 bool cq_resize:1;
633732 };
634733
734+struct irdma_srq_init_info {
735+ struct irdma_sc_pd *pd;
736+ struct irdma_sc_vsi *vsi;
737+ u64 srq_pa;
738+ u64 shadow_area_pa;
739+ u32 first_pm_pbl_idx;
740+ u32 pasid;
741+ u32 srq_size;
742+ u16 srq_limit;
743+ u8 pasid_valid;
744+ u8 wqe_size;
745+ u8 leaf_pbl_size;
746+ u8 virtual_map;
747+ u8 tph_en;
748+ u8 arm_limit_event;
749+ u8 tph_value;
750+ u8 pbl_chunk_size;
751+ struct irdma_srq_uk_init_info srq_uk_init_info;
752+};
753+
754+struct irdma_sc_srq {
755+ struct irdma_sc_dev *dev;
756+ struct irdma_sc_vsi *vsi;
757+ struct irdma_sc_pd *pd;
758+ struct irdma_srq_uk srq_uk;
759+ void *back_srq;
760+ u64 srq_pa;
761+ u64 shadow_area_pa;
762+ u32 first_pm_pbl_idx;
763+ u32 pasid;
764+ u32 hw_srq_size;
765+ u16 srq_limit;
766+ u8 pasid_valid;
767+ u8 leaf_pbl_size;
768+ u8 virtual_map;
769+ u8 tph_en;
770+ u8 arm_limit_event;
771+ u8 tph_val;
772+};
773+
774+struct irdma_modify_srq_info {
775+ u16 srq_limit;
776+ u8 arm_limit_event;
777+};
778+
635779 struct irdma_create_qp_info {
636780 bool ord_valid:1;
637781 bool tcp_ctx_valid:1;
@@ -671,7 +815,8 @@ struct irdma_ccq_cqe_info { · 33 unmodified lines
671815 u16 maj_err_code;
672816 u16 min_err_code;
673817 u8 op_code;
674− bool error;
818+ bool error:1;
819+ bool pending:1;
675820 };
676821
677822 struct irdma_dcb_app_info {
@@ -720,7 +865,7 @@ struct irdma_vsi_init_info { · 42 unmodified lines
720865
721866 struct irdma_vsi_stats_info {
722867 struct irdma_vsi_pestat *pestat;
723− u8 fcn_id;
868+ u16 fcn_id;
724869 bool alloc_stats_inst;
725870 };
726871
@@ -731,7 +876,8 @@ struct irdma_device_init_info { · 4 unmodified lines
731876 __le64 *fpm_commit_buf;
732877 struct irdma_hw *hw;
733878 void __iomem *bar0;
734− u8 hmc_fn_id;
879+ enum irdma_protocol_used protocol_used;
880+ u16 hmc_fn_id;
735881 };
736882
737883 struct irdma_ceq_init_info {
@@ -746,8 +892,8 @@ struct irdma_ceq_init_info { · 8 unmodified lines
746892 bool itr_no_expire:1;
747893 u8 pbl_chunk_size;
748894 u8 tph_val;
895+ u16 vsi_idx;
749896 u32 first_pm_pbl_idx;
750− struct irdma_sc_vsi *vsi;
751897 struct irdma_sc_cq **reg_cq;
752898 u32 reg_cq_idx;
753899 };
@@ -807,6 +953,8 @@ struct irdma_udp_offload_info { · 53 unmodified lines
807953 u32 cwnd;
808954 u8 rexmit_thresh;
809955 u8 rnr_nak_thresh;
956+ u8 rnr_nak_tmr;
957+ u8 min_rnr_timer;
810958 };
811959
812960 struct irdma_roce_offload_info {
@@ -833,6 +981,7 @@ struct irdma_roce_offload_info { · 20 unmodified lines
833981 bool dctcp_en:1;
834982 bool fw_cc_enable:1;
835983 bool use_stats_inst:1;
984+ u8 local_ack_timeout;
836985 u16 t_high;
837986 u16 t_low;
838987 u8 last_byte_sent;
@@ -933,8 +1082,10 @@ struct irdma_qp_host_ctx_info { · 94 unmodified lines
9331082 };
9341083 u32 send_cq_num;
9351084 u32 rcv_cq_num;
1085+ u32 srq_id;
9361086 u32 rem_endpoint_idx;
937− u8 stats_idx;
1087+ u16 stats_idx;
1088+ bool remote_atomics_en:1;
9381089 bool srq_valid:1;
9391090 bool tcp_info_valid:1;
9401091 bool iwarp_info_valid:1;
@@ -945,6 +1096,7 @@ struct irdma_qp_host_ctx_info { · 4 unmodified lines
9451096 struct irdma_aeqe_info {
9461097 u64 compl_ctx;
9471098 u32 qp_cq_id;
1099+ u32 def_info; /* only valid for DEF_CMPL */
9481100 u16 ae_id;
9491101 u16 wqe_idx;
9501102 u8 tcp_state;
@@ -953,9 +1105,11 @@ struct irdma_aeqe_info { · 2 unmodified lines
9531105 bool cq:1;
9541106 bool sq:1;
9551107 bool rq:1;
1108+ bool srq:1;
9561109 bool in_rdrsp_wr:1;
9571110 bool out_rdrsp:1;
9581111 bool aeqe_overflow:1;
1112+ bool err_rq_idx_valid:1;
9591113 u8 q2_data_written;
9601114 u8 ae_src;
9611115 };
@@ -972,7 +1126,8 @@ struct irdma_allocate_stag_info { · 10 unmodified lines
9721126 bool use_hmc_fcn_index:1;
9731127 bool use_pf_rid:1;
9741128 bool all_memory:1;
975− u8 hmc_fcn_index;
1129+ bool remote_atomics_en:1;
1130+ u16 hmc_fcn_index;
9761131 };
9771132
9781133 struct irdma_mw_alloc_info {
@@ -1000,6 +1155,7 @@ struct irdma_reg_ns_stag_info { · 21 unmodified lines
10001155 u8 hmc_fcn_index;
10011156 bool use_pf_rid:1;
10021157 bool all_memory:1;
1158+ bool remote_atomics_en:1;
10031159 };
10041160
10051161 struct irdma_fast_reg_stag_info {
@@ -1023,6 +1179,7 @@ struct irdma_fast_reg_stag_info { · 17 unmodified lines
10231179 u8 hmc_fcn_index;
10241180 bool use_pf_rid:1;
10251181 bool defer_flag:1;
1182+ bool remote_atomics_en:1;
10261183 };
10271184
10281185 struct irdma_dealloc_stag_info {
@@ -1130,6 +1287,8 @@ struct irdma_cqp_manage_push_page_info { · 101 unmodified lines
11301287 };
11311288
11321289 struct irdma_qp_flush_info {
1290+ u32 err_sq_idx;
1291+ u32 err_rq_idx;
11331292 u16 sq_minor_code;
11341293 u16 sq_major_code;
11351294 u16 rq_minor_code;
@@ -1140,6 +1299,8 @@ struct irdma_qp_flush_info { · 4 unmodified lines
11401299 bool rq:1;
11411300 bool userflushcode:1;
11421301 bool generate_ae:1;
1302+ bool err_sq_idx_valid:1;
1303+ bool err_rq_idx_valid:1;
11431304 };
11441305
11451306 struct irdma_gen_ae_info {
@@ -1189,6 +1350,11 @@ void irdma_sc_pd_init(struct irdma_sc_dev *dev, struct irdma_sc_pd *pd, u32 pd_i · 43 unmodified lines
11891350 void irdma_cfg_aeq(struct irdma_sc_dev *dev, u32 idx, bool enable);
11901351 void irdma_check_cqp_progress(struct irdma_cqp_timeout *cqp_timeout,
11911352 struct irdma_sc_dev *dev);
1353+void irdma_sc_cqp_def_cmpl_ae_handler(struct irdma_sc_dev *dev,
1354+ struct irdma_aeqe_info *info,
1355+ bool first, u64 *scratch,
1356+ u32 *sw_def_info);
1357+u64 irdma_sc_cqp_cleanup_handler(struct irdma_sc_dev *dev);
11921358 int irdma_sc_cqp_create(struct irdma_sc_cqp *cqp, u16 *maj_err, u16 *min_err);
11931359 int irdma_sc_cqp_destroy(struct irdma_sc_cqp *cqp);
11941360 int irdma_sc_cqp_init(struct irdma_sc_cqp *cqp,
@@ -1224,6 +1390,8 @@ void irdma_sc_cq_resize(struct irdma_sc_cq *cq, struct irdma_modify_cq_info *inf · 29 unmodified lines
12241390 int irdma_sc_static_hmc_pages_allocated(struct irdma_sc_cqp *cqp, u64 scratch,
12251391 u8 hmc_fn_id, bool post_sq,
12261392 bool poll_registers);
1393+int irdma_sc_srq_init(struct irdma_sc_srq *srq,
1394+ struct irdma_srq_init_info *info);
12271395
12281396 void sc_vsi_update_stats(struct irdma_sc_vsi *vsi);
12291397 struct cqp_info {
@@ -1467,6 +1635,23 @@ struct cqp_info { · 237 unmodified lines
14671635 struct irdma_dma_mem query_buff_mem;
14681636 u64 scratch;
14691637 } query_rdma;
1638+
1639+ struct {
1640+ struct irdma_sc_srq *srq;
1641+ u64 scratch;
1642+ } srq_create;
1643+
1644+ struct {
1645+ struct irdma_sc_srq *srq;
1646+ struct irdma_modify_srq_info info;
1647+ u64 scratch;
1648+ } srq_modify;
1649+
1650+ struct {
1651+ struct irdma_sc_srq *srq;
1652+ u64 scratch;
1653+ } srq_destroy;
1654+
14701655 } u;
14711656 };
14721657
drivers/infiniband/hw/irdma/uda_d.h+2−3
@@ -78,8 +78,7 @@ · 77 unmodified lines
7878 #define IRDMA_UDAQPC_IPID GENMASK_ULL(47, 32)
7979 #define IRDMA_UDAQPC_SNDMSS GENMASK_ULL(29, 16)
8080 #define IRDMA_UDAQPC_VLANTAG GENMASK_ULL(15, 0)
81−
82−#define IRDMA_UDA_CQPSQ_MAV_PDINDEXHI GENMASK_ULL(21, 20)
81+#define IRDMA_UDA_CQPSQ_MAV_PDINDEXHI GENMASK_ULL(27, 20)
8382 #define IRDMA_UDA_CQPSQ_MAV_PDINDEXLO GENMASK_ULL(63, 48)
8483 #define IRDMA_UDA_CQPSQ_MAV_SRCMACADDRINDEX GENMASK_ULL(29, 24)
8584 #define IRDMA_UDA_CQPSQ_MAV_ARPINDEX GENMASK_ULL(63, 48)
@@ -94,7 +93,7 @@ · 8 unmodified lines
9493 #define IRDMA_UDA_CQPSQ_MAV_OPCODE GENMASK_ULL(37, 32)
9594 #define IRDMA_UDA_CQPSQ_MAV_DOLOOPBACKK BIT_ULL(62)
9695 #define IRDMA_UDA_CQPSQ_MAV_IPV4VALID BIT_ULL(59)
97−#define IRDMA_UDA_CQPSQ_MAV_AVIDX GENMASK_ULL(16, 0)
96+#define IRDMA_UDA_CQPSQ_MAV_AVIDX GENMASK_ULL(23, 0)
9897 #define IRDMA_UDA_CQPSQ_MAV_INSERTVLANTAG BIT_ULL(60)
9998 #define IRDMA_UDA_MGCTX_VFFLAG BIT_ULL(29)
10099 #define IRDMA_UDA_MGCTX_DESTPORT GENMASK_ULL(47, 32)
drivers/infiniband/hw/irdma/uk.c+291−12
@@ -198,6 +198,26 @@ __le64 *irdma_qp_get_next_send_wqe(struct irdma_qp_uk *qp, u32 *wqe_idx, · 197 unmodified lines
198198 return wqe;
199199 }
200200
201+__le64 *irdma_srq_get_next_recv_wqe(struct irdma_srq_uk *srq, u32 *wqe_idx)
202+{
203+ int ret_code;
204+ __le64 *wqe;
205+
206+ if (IRDMA_RING_FULL_ERR(srq->srq_ring))
207+ return NULL;
208+
209+ IRDMA_ATOMIC_RING_MOVE_HEAD(srq->srq_ring, *wqe_idx, ret_code);
210+ if (ret_code)
211+ return NULL;
212+
213+ if (!*wqe_idx)
214+ srq->srwqe_polarity = !srq->srwqe_polarity;
215+ /* rq_wqe_size_multiplier is no of 32 byte quanta in one rq wqe */
216+ wqe = srq->srq_base[*wqe_idx * (srq->wqe_size_multiplier)].elem;
217+
218+ return wqe;
219+}
220+
201221 /**
202222 * irdma_qp_get_next_recv_wqe - get next qp's rcv wqe
203223 * @qp: hw qp ptr
@@ -317,6 +337,160 @@ int irdma_uk_rdma_write(struct irdma_qp_uk *qp, struct irdma_post_sq_info *info, · 113 unmodified lines
317337 return 0;
318338 }
319339
340+/**
341+ * irdma_uk_atomic_fetch_add - atomic fetch and add operation
342+ * @qp: hw qp ptr
343+ * @info: post sq information
344+ * @post_sq: flag to post sq
345+ */
346+int irdma_uk_atomic_fetch_add(struct irdma_qp_uk *qp,
347+ struct irdma_post_sq_info *info, bool post_sq)
348+{
349+ struct irdma_atomic_fetch_add *op_info;
350+ u32 total_size = 0;
351+ u16 quanta = 2;
352+ u32 wqe_idx;
353+ __le64 *wqe;
354+ u64 hdr;
355+
356+ op_info = &info->op.atomic_fetch_add;
357+ wqe = irdma_qp_get_next_send_wqe(qp, &wqe_idx, quanta, total_size,
358+ info);
359+ if (!wqe)
360+ return -ENOMEM;
361+
362+ set_64bit_val(wqe, 0, op_info->tagged_offset);
363+ set_64bit_val(wqe, 8,
364+ FIELD_PREP(IRDMAQPSQ_STAG, op_info->stag));
365+ set_64bit_val(wqe, 16, op_info->remote_tagged_offset);
366+
367+ hdr = FIELD_PREP(IRDMAQPSQ_ADDFRAGCNT, 1) |
368+ FIELD_PREP(IRDMAQPSQ_REMOTE_STAG, op_info->remote_stag) |
369+ FIELD_PREP(IRDMAQPSQ_OPCODE, IRDMAQP_OP_ATOMIC_FETCH_ADD) |
370+ FIELD_PREP(IRDMAQPSQ_READFENCE, info->read_fence) |
371+ FIELD_PREP(IRDMAQPSQ_LOCALFENCE, info->local_fence) |
372+ FIELD_PREP(IRDMAQPSQ_SIGCOMPL, info->signaled) |
373+ FIELD_PREP(IRDMAQPSQ_VALID, qp->swqe_polarity);
374+
375+ set_64bit_val(wqe, 32, op_info->fetch_add_data_bytes);
376+ set_64bit_val(wqe, 40, 0);
377+ set_64bit_val(wqe, 48, 0);
378+ set_64bit_val(wqe, 56,
379+ FIELD_PREP(IRDMAQPSQ_VALID, qp->swqe_polarity));
380+
381+ dma_wmb(); /* make sure WQE is populated before valid bit is set */
382+
383+ set_64bit_val(wqe, 24, hdr);
384+
385+ if (post_sq)
386+ irdma_uk_qp_post_wr(qp);
387+
388+ return 0;
389+}
390+
391+/**
392+ * irdma_uk_atomic_compare_swap - atomic compare and swap operation
393+ * @qp: hw qp ptr
394+ * @info: post sq information
395+ * @post_sq: flag to post sq
396+ */
397+int irdma_uk_atomic_compare_swap(struct irdma_qp_uk *qp,
398+ struct irdma_post_sq_info *info, bool post_sq)
399+{
400+ struct irdma_atomic_compare_swap *op_info;
401+ u32 total_size = 0;
402+ u16 quanta = 2;
403+ u32 wqe_idx;
404+ __le64 *wqe;
405+ u64 hdr;
406+
407+ op_info = &info->op.atomic_compare_swap;
408+ wqe = irdma_qp_get_next_send_wqe(qp, &wqe_idx, quanta, total_size,
409+ info);
410+ if (!wqe)
411+ return -ENOMEM;
412+
413+ set_64bit_val(wqe, 0, op_info->tagged_offset);
414+ set_64bit_val(wqe, 8,
415+ FIELD_PREP(IRDMAQPSQ_STAG, op_info->stag));
416+ set_64bit_val(wqe, 16, op_info->remote_tagged_offset);
417+
418+ hdr = FIELD_PREP(IRDMAQPSQ_ADDFRAGCNT, 1) |
419+ FIELD_PREP(IRDMAQPSQ_REMOTE_STAG, op_info->remote_stag) |
420+ FIELD_PREP(IRDMAQPSQ_OPCODE, IRDMAQP_OP_ATOMIC_COMPARE_SWAP_ADD) |
421+ FIELD_PREP(IRDMAQPSQ_READFENCE, info->read_fence) |
422+ FIELD_PREP(IRDMAQPSQ_LOCALFENCE, info->local_fence) |
423+ FIELD_PREP(IRDMAQPSQ_SIGCOMPL, info->signaled) |
424+ FIELD_PREP(IRDMAQPSQ_VALID, qp->swqe_polarity);
425+
426+ set_64bit_val(wqe, 32, op_info->swap_data_bytes);
427+ set_64bit_val(wqe, 40, op_info->compare_data_bytes);
428+ set_64bit_val(wqe, 48, 0);
429+ set_64bit_val(wqe, 56,
430+ FIELD_PREP(IRDMAQPSQ_VALID, qp->swqe_polarity));
431+
432+ dma_wmb(); /* make sure WQE is populated before valid bit is set */
433+
434+ set_64bit_val(wqe, 24, hdr);
435+
436+ if (post_sq)
437+ irdma_uk_qp_post_wr(qp);
438+
439+ return 0;
440+}
441+
442+/**
443+ * irdma_uk_srq_post_receive - post a receive wqe to a shared rq
444+ * @srq: shared rq ptr
445+ * @info: post rq information
446+ */
447+int irdma_uk_srq_post_receive(struct irdma_srq_uk *srq,
448+ struct irdma_post_rq_info *info)
449+{
450+ u32 wqe_idx, i, byte_off;
451+ u32 addl_frag_cnt;
452+ __le64 *wqe;
453+ u64 hdr;
454+
455+ if (srq->max_srq_frag_cnt < info->num_sges)
456+ return -EINVAL;
457+
458+ wqe = irdma_srq_get_next_recv_wqe(srq, &wqe_idx);
459+ if (!wqe)
460+ return -ENOMEM;
461+
462+ addl_frag_cnt = info->num_sges > 1 ? info->num_sges - 1 : 0;
463+ srq->wqe_ops.iw_set_fragment(wqe, 0, info->sg_list,
464+ srq->srwqe_polarity);
465+
466+ for (i = 1, byte_off = 32; i < info->num_sges; i++) {
467+ srq->wqe_ops.iw_set_fragment(wqe, byte_off, &info->sg_list[i],
468+ srq->srwqe_polarity);
469+ byte_off += 16;
470+ }
471+
472+ /* if not an odd number set valid bit in next fragment */
473+ if (srq->uk_attrs->hw_rev >= IRDMA_GEN_2 && !(info->num_sges & 0x01) &&
474+ info->num_sges) {
475+ srq->wqe_ops.iw_set_fragment(wqe, byte_off, NULL,
476+ srq->srwqe_polarity);
477+ if (srq->uk_attrs->hw_rev == IRDMA_GEN_2)
478+ ++addl_frag_cnt;
479+ }
480+
481+ set_64bit_val(wqe, 16, (u64)info->wr_id);
482+ hdr = FIELD_PREP(IRDMAQPSQ_ADDFRAGCNT, addl_frag_cnt) |
483+ FIELD_PREP(IRDMAQPSQ_VALID, srq->srwqe_polarity);
484+
485+ dma_wmb(); /* make sure WQE is populated before valid bit is set */
486+
487+ set_64bit_val(wqe, 24, hdr);
488+
489+ set_64bit_val(srq->shadow_area, 0, (wqe_idx + 1) % srq->srq_ring.size);
490+
491+ return 0;
492+}
493+
320494 /**
321495 * irdma_uk_rdma_read - rdma read command
322496 * @qp: hw qp ptr
@@ -973,6 +1147,9 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq, · 650 unmodified lines
9731147 u64 comp_ctx, qword0, qword2, qword3;
9741148 __le64 *cqe;
9751149 struct irdma_qp_uk *qp;
1150+ struct irdma_srq_uk *srq;
1151+ struct qp_err_code qp_err;
1152+ u8 is_srq;
9761153 struct irdma_ring *pring = NULL;
9771154 u32 wqe_idx;
9781155 int ret_code;
@@ -1046,21 +1223,46 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq, · 67 unmodified lines
10461223 }
10471224
10481225 info->q_type = (u8)FIELD_GET(IRDMA_CQ_SQ, qword3);
1226+ is_srq = (u8)FIELD_GET(IRDMA_CQ_SRQ, qword3);
10491227 info->error = (bool)FIELD_GET(IRDMA_CQ_ERROR, qword3);
10501228 info->ipv4 = (bool)FIELD_GET(IRDMACQ_IPV4, qword3);
1229+ get_64bit_val(cqe, 8, &comp_ctx);
1230+ if (is_srq)
1231+ get_64bit_val(cqe, 40, (u64 *)&qp);
1232+ else
1233+ qp = (struct irdma_qp_uk *)(unsigned long)comp_ctx;
10511234 if (info->error) {
10521235 info->major_err = FIELD_GET(IRDMA_CQ_MAJERR, qword3);
10531236 info->minor_err = FIELD_GET(IRDMA_CQ_MINERR, qword3);
1054− if (info->major_err == IRDMA_FLUSH_MAJOR_ERR) {
1055− info->comp_status = IRDMA_COMPL_STATUS_FLUSHED;
1237+ switch (info->major_err) {
1238+ case IRDMA_SRQFLUSH_RSVD_MAJOR_ERR:
1239+ qp_err = irdma_ae_to_qp_err_code(info->minor_err);
1240+ info->minor_err = qp_err.flush_code;
1241+ fallthrough;
1242+ case IRDMA_FLUSH_MAJOR_ERR:
10561243 /* Set the min error to standard flush error code for remaining cqes */
10571244 if (info->minor_err != FLUSH_GENERAL_ERR) {
10581245 qword3 &= ~IRDMA_CQ_MINERR;
10591246 qword3 |= FIELD_PREP(IRDMA_CQ_MINERR, FLUSH_GENERAL_ERR);
10601247 set_64bit_val(cqe, 24, qword3);
10611248 }
1062− } else {
1063− info->comp_status = IRDMA_COMPL_STATUS_UNKNOWN;
1249+ info->comp_status = IRDMA_COMPL_STATUS_FLUSHED;
1250+ break;
1251+ default:
1252+#define IRDMA_CIE_SIGNATURE 0xE
1253+#define IRDMA_CQMAJERR_HIGH_NIBBLE GENMASK(15, 12)
1254+ if (info->q_type == IRDMA_CQE_QTYPE_SQ &&
1255+ qp->qp_type == IRDMA_QP_TYPE_ROCE_UD &&
1256+ FIELD_GET(IRDMA_CQMAJERR_HIGH_NIBBLE, info->major_err)
1257+ == IRDMA_CIE_SIGNATURE) {
1258+ info->error = 0;
1259+ info->major_err = 0;
1260+ info->minor_err = 0;
1261+ info->comp_status = IRDMA_COMPL_STATUS_SUCCESS;
1262+ } else {
1263+ info->comp_status = IRDMA_COMPL_STATUS_UNKNOWN;
1264+ }
1265+ break;
10641266 }
10651267 } else {
10661268 info->comp_status = IRDMA_COMPL_STATUS_SUCCESS;
@@ -1069,7 +1271,6 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq, · 2 unmodified lines
10691271 get_64bit_val(cqe, 0, &qword0);
10701272 get_64bit_val(cqe, 16, &qword2);
10711273
1072− info->tcp_seq_num_rtt = (u32)FIELD_GET(IRDMACQ_TCPSEQNUMRTT, qword0);
10731274 info->qp_id = (u32)FIELD_GET(IRDMACQ_QPID, qword2);
10741275 info->ud_src_qpn = (u32)FIELD_GET(IRDMACQ_UDSRCQPN, qword2);
10751276
@@ -1085,7 +1286,22 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq, · 9 unmodified lines
10851286 info->qp_handle = (irdma_qp_handle)(unsigned long)qp;
10861287 info->op_type = (u8)FIELD_GET(IRDMACQ_OP, qword3);
10871288
1088− if (info->q_type == IRDMA_CQE_QTYPE_RQ) {
1289+ if (info->q_type == IRDMA_CQE_QTYPE_RQ && is_srq) {
1290+ srq = qp->srq_uk;
1291+
1292+ get_64bit_val(cqe, 8, &info->wr_id);
1293+ info->bytes_xfered = (u32)FIELD_GET(IRDMACQ_PAYLDLEN, qword0);
1294+
1295+ if (qword3 & IRDMACQ_STAG) {
1296+ info->stag_invalid_set = true;
1297+ info->inv_stag = (u32)FIELD_GET(IRDMACQ_INVSTAG,
1298+ qword2);
1299+ } else {
1300+ info->stag_invalid_set = false;
1301+ }
1302+ IRDMA_RING_MOVE_TAIL(srq->srq_ring);
1303+ pring = &srq->srq_ring;
1304+ } else if (info->q_type == IRDMA_CQE_QTYPE_RQ && !is_srq) {
10891305 u32 array_idx;
10901306
10911307 array_idx = wqe_idx / qp->rq_wqe_size_multiplier;
@@ -1180,9 +1396,15 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq, · 88 unmodified lines
11801396 ret_code = 0;
11811397
11821398 exit:
1183− if (!ret_code && info->comp_status == IRDMA_COMPL_STATUS_FLUSHED)
1399+ if (!ret_code && info->comp_status == IRDMA_COMPL_STATUS_FLUSHED) {
11841400 if (pring && IRDMA_RING_MORE_WORK(*pring))
1185− move_cq_head = false;
1401+ /* Park CQ head during a flush to generate additional CQEs
1402+ * from SW for all unprocessed WQEs. For GEN3 and beyond
1403+ * FW will generate/flush these CQEs so move to the next CQE
1404+ */
1405+ move_cq_head = qp->uk_attrs->hw_rev <= IRDMA_GEN_2 ?
1406+ false : true;
1407+ }
11861408
11871409 if (move_cq_head) {
11881410 IRDMA_RING_MOVE_HEAD_NOCHECK(cq->cq_ring);
@@ -1210,10 +1432,10 @@ exit: · 21 unmodified lines
12101432 }
12111433
12121434 /**
1213− * irdma_qp_round_up - return round up qp wq depth
1435+ * irdma_round_up_wq - return round up qp wq depth
12141436 * @wqdepth: wq depth in quanta to round up
12151437 */
1216−static int irdma_qp_round_up(u32 wqdepth)
1438+static int irdma_round_up_wq(u32 wqdepth)
12171439 {
12181440 int scount = 1;
12191441
@@ -1268,7 +1490,7 @@ int irdma_get_sqdepth(struct irdma_uk_attrs *uk_attrs, u32 sq_size, u8 shift, · 48 unmodified lines
12681490 {
12691491 u32 min_size = (u32)uk_attrs->min_hw_wq_size << shift;
12701492
1271− *sqdepth = irdma_qp_round_up((sq_size << shift) + IRDMA_SQ_RSVD);
1493+ *sqdepth = irdma_round_up_wq((sq_size << shift) + IRDMA_SQ_RSVD);
12721494
12731495 if (*sqdepth < min_size)
12741496 *sqdepth = min_size;
@@ -1290,7 +1512,7 @@ int irdma_get_rqdepth(struct irdma_uk_attrs *uk_attrs, u32 rq_size, u8 shift, · 15 unmodified lines
12901512 {
12911513 u32 min_size = (u32)uk_attrs->min_hw_wq_size << shift;
12921514
1293− *rqdepth = irdma_qp_round_up((rq_size << shift) + IRDMA_RQ_RSVD);
1515+ *rqdepth = irdma_round_up_wq((rq_size << shift) + IRDMA_RQ_RSVD);
12941516
12951517 if (*rqdepth < min_size)
12961518 *rqdepth = min_size;
@@ -1300,6 +1522,26 @@ int irdma_get_rqdepth(struct irdma_uk_attrs *uk_attrs, u32 rq_size, u8 shift, · 3 unmodified lines
13001522 return 0;
13011523 }
13021524
1525+/*
1526+ * irdma_get_srqdepth - get SRQ depth (quanta)
1527+ * @uk_attrs: qp HW attributes
1528+ * @srq_size: SRQ size
1529+ * @shift: shift which determines size of WQE
1530+ * @srqdepth: depth of SRQ
1531+ */
1532+int irdma_get_srqdepth(struct irdma_uk_attrs *uk_attrs, u32 srq_size, u8 shift,
1533+ u32 *srqdepth)
1534+{
1535+ *srqdepth = irdma_round_up_wq((srq_size << shift) + IRDMA_RQ_RSVD);
1536+
1537+ if (*srqdepth < ((u32)uk_attrs->min_hw_wq_size << shift))
1538+ *srqdepth = uk_attrs->min_hw_wq_size << shift;
1539+ else if (*srqdepth > uk_attrs->max_hw_srq_quanta)
1540+ return -EINVAL;
1541+
1542+ return 0;
1543+}
1544+
13031545 static const struct irdma_wqe_uk_ops iw_wqe_uk_ops = {
13041546 .iw_copy_inline_data = irdma_copy_inline_data,
13051547 .iw_inline_data_size_to_quanta = irdma_inline_data_size_to_quanta,
@@ -1335,6 +1577,42 @@ static void irdma_setup_connection_wqes(struct irdma_qp_uk *qp, · 29 unmodified lines
13351577 IRDMA_RING_MOVE_HEAD_BY_COUNT_NOCHECK(qp->initial_ring, move_cnt);
13361578 }
13371579
1580+/**
1581+ * irdma_uk_srq_init - initialize shared qp
1582+ * @srq: hw srq (user and kernel)
1583+ * @info: srq initialization info
1584+ *
1585+ * Initializes the vars used in both user and kernel mode.
1586+ * The size of the wqe depends on number of max fragments
1587+ * allowed. Then size of wqe * the number of wqes should be the
1588+ * amount of memory allocated for srq.
1589+ */
1590+int irdma_uk_srq_init(struct irdma_srq_uk *srq,
1591+ struct irdma_srq_uk_init_info *info)
1592+{
1593+ u8 rqshift;
1594+
1595+ srq->uk_attrs = info->uk_attrs;
1596+ if (info->max_srq_frag_cnt > srq->uk_attrs->max_hw_wq_frags)
1597+ return -EINVAL;
1598+
1599+ irdma_get_wqe_shift(srq->uk_attrs, info->max_srq_frag_cnt, 0, &rqshift);
1600+ srq->srq_caps = info->srq_caps;
1601+ srq->srq_base = info->srq;
1602+ srq->shadow_area = info->shadow_area;
1603+ srq->srq_id = info->srq_id;
1604+ srq->srwqe_polarity = 0;
1605+ srq->srq_size = info->srq_size;
1606+ srq->wqe_size = rqshift;
1607+ srq->max_srq_frag_cnt = min(srq->uk_attrs->max_hw_wq_frags,
1608+ ((u32)2 << rqshift) - 1);
1609+ IRDMA_RING_INIT(srq->srq_ring, srq->srq_size);
1610+ srq->wqe_size_multiplier = 1 << rqshift;
1611+ srq->wqe_ops = iw_wqe_uk_ops;
1612+
1613+ return 0;
1614+}
1615+
13381616 /**
13391617 * irdma_uk_calc_shift_wq - calculate WQE shift for both SQ and RQ
13401618 * @ukinfo: qp initialization info
@@ -1461,6 +1739,7 @@ int irdma_uk_qp_init(struct irdma_qp_uk *qp, struct irdma_qp_uk_init_info *info) · 120 unmodified lines
14611739 qp->wqe_ops = iw_wqe_uk_ops_gen_1;
14621740 else
14631741 qp->wqe_ops = iw_wqe_uk_ops;
1742+ qp->srq_uk = info->srq_uk;
14641743 return ret_code;
14651744 }
14661745
drivers/infiniband/hw/irdma/user.h+264−3
@@ -41,10 +41,114 @@ · 40 unmodified lines
4141 #define IRDMA_OP_TYPE_INV_STAG 0x0a
4242 #define IRDMA_OP_TYPE_RDMA_READ_INV_STAG 0x0b
4343 #define IRDMA_OP_TYPE_NOP 0x0c
44+#define IRDMA_OP_TYPE_ATOMIC_FETCH_AND_ADD 0x0f
45+#define IRDMA_OP_TYPE_ATOMIC_COMPARE_AND_SWAP 0x11
4446 #define IRDMA_OP_TYPE_REC 0x3e
4547 #define IRDMA_OP_TYPE_REC_IMM 0x3f
4648
47−#define IRDMA_FLUSH_MAJOR_ERR 1
49+#define IRDMA_FLUSH_MAJOR_ERR 1
50+#define IRDMA_SRQFLUSH_RSVD_MAJOR_ERR 0xfffe
51+
52+/* Async Events codes */
53+#define IRDMA_AE_AMP_UNALLOCATED_STAG 0x0102
54+#define IRDMA_AE_AMP_INVALID_STAG 0x0103
55+#define IRDMA_AE_AMP_BAD_QP 0x0104
56+#define IRDMA_AE_AMP_BAD_PD 0x0105
57+#define IRDMA_AE_AMP_BAD_STAG_KEY 0x0106
58+#define IRDMA_AE_AMP_BAD_STAG_INDEX 0x0107
59+#define IRDMA_AE_AMP_BOUNDS_VIOLATION 0x0108
60+#define IRDMA_AE_AMP_RIGHTS_VIOLATION 0x0109
61+#define IRDMA_AE_AMP_TO_WRAP 0x010a
62+#define IRDMA_AE_AMP_FASTREG_VALID_STAG 0x010c
63+#define IRDMA_AE_AMP_FASTREG_MW_STAG 0x010d
64+#define IRDMA_AE_AMP_FASTREG_INVALID_RIGHTS 0x010e
65+#define IRDMA_AE_AMP_FASTREG_INVALID_LENGTH 0x0110
66+#define IRDMA_AE_AMP_INVALIDATE_SHARED 0x0111
67+#define IRDMA_AE_AMP_INVALIDATE_NO_REMOTE_ACCESS_RIGHTS 0x0112
68+#define IRDMA_AE_AMP_INVALIDATE_MR_WITH_BOUND_WINDOWS 0x0113
69+#define IRDMA_AE_AMP_MWBIND_VALID_STAG 0x0114
70+#define IRDMA_AE_AMP_MWBIND_OF_MR_STAG 0x0115
71+#define IRDMA_AE_AMP_MWBIND_TO_ZERO_BASED_STAG 0x0116
72+#define IRDMA_AE_AMP_MWBIND_TO_MW_STAG 0x0117
73+#define IRDMA_AE_AMP_MWBIND_INVALID_RIGHTS 0x0118
74+#define IRDMA_AE_AMP_MWBIND_INVALID_BOUNDS 0x0119
75+#define IRDMA_AE_AMP_MWBIND_TO_INVALID_PARENT 0x011a
76+#define IRDMA_AE_AMP_MWBIND_BIND_DISABLED 0x011b
77+#define IRDMA_AE_PRIV_OPERATION_DENIED 0x011c
78+#define IRDMA_AE_AMP_INVALIDATE_TYPE1_MW 0x011d
79+#define IRDMA_AE_AMP_MWBIND_ZERO_BASED_TYPE1_MW 0x011e
80+#define IRDMA_AE_AMP_FASTREG_INVALID_PBL_HPS_CFG 0x011f
81+#define IRDMA_AE_AMP_MWBIND_WRONG_TYPE 0x0120
82+#define IRDMA_AE_AMP_FASTREG_PBLE_MISMATCH 0x0121
83+#define IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG 0x0132
84+#define IRDMA_AE_UDA_XMIT_BAD_PD 0x0133
85+#define IRDMA_AE_UDA_XMIT_DGRAM_TOO_SHORT 0x0134
86+#define IRDMA_AE_UDA_L4LEN_INVALID 0x0135
87+#define IRDMA_AE_BAD_CLOSE 0x0201
88+#define IRDMA_AE_RDMAP_ROE_BAD_LLP_CLOSE 0x0202
89+#define IRDMA_AE_CQ_OPERATION_ERROR 0x0203
90+#define IRDMA_AE_RDMA_READ_WHILE_ORD_ZERO 0x0205
91+#define IRDMA_AE_STAG_ZERO_INVALID 0x0206
92+#define IRDMA_AE_IB_RREQ_AND_Q1_FULL 0x0207
93+#define IRDMA_AE_IB_INVALID_REQUEST 0x0208
94+#define IRDMA_AE_SRQ_LIMIT 0x0209
95+#define IRDMA_AE_WQE_UNEXPECTED_OPCODE 0x020a
96+#define IRDMA_AE_WQE_INVALID_PARAMETER 0x020b
97+#define IRDMA_AE_WQE_INVALID_FRAG_DATA 0x020c
98+#define IRDMA_AE_IB_REMOTE_ACCESS_ERROR 0x020d
99+#define IRDMA_AE_IB_REMOTE_OP_ERROR 0x020e
100+#define IRDMA_AE_SRQ_CATASTROPHIC_ERROR 0x020f
101+#define IRDMA_AE_WQE_LSMM_TOO_LONG 0x0220
102+#define IRDMA_AE_ATOMIC_ALIGNMENT 0x0221
103+#define IRDMA_AE_ATOMIC_MASK 0x0222
104+#define IRDMA_AE_INVALID_REQUEST 0x0223
105+#define IRDMA_AE_PCIE_ATOMIC_DISABLE 0x0224
106+#define IRDMA_AE_DDP_INVALID_MSN_GAP_IN_MSN 0x0301
107+#define IRDMA_AE_DDP_UBE_DDP_MESSAGE_TOO_LONG_FOR_AVAILABLE_BUFFER 0x0303
108+#define IRDMA_AE_DDP_UBE_INVALID_DDP_VERSION 0x0304
109+#define IRDMA_AE_DDP_UBE_INVALID_MO 0x0305
110+#define IRDMA_AE_DDP_UBE_INVALID_MSN_NO_BUFFER_AVAILABLE 0x0306
111+#define IRDMA_AE_DDP_UBE_INVALID_QN 0x0307
112+#define IRDMA_AE_DDP_NO_L_BIT 0x0308
113+#define IRDMA_AE_RDMAP_ROE_INVALID_RDMAP_VERSION 0x0311
114+#define IRDMA_AE_RDMAP_ROE_UNEXPECTED_OPCODE 0x0312
115+#define IRDMA_AE_ROE_INVALID_RDMA_READ_REQUEST 0x0313
116+#define IRDMA_AE_ROE_INVALID_RDMA_WRITE_OR_READ_RESP 0x0314
117+#define IRDMA_AE_ROCE_RSP_LENGTH_ERROR 0x0316
118+#define IRDMA_AE_ROCE_EMPTY_MCG 0x0380
119+#define IRDMA_AE_ROCE_BAD_MC_IP_ADDR 0x0381
120+#define IRDMA_AE_ROCE_BAD_MC_QPID 0x0382
121+#define IRDMA_AE_MCG_QP_PROTOCOL_MISMATCH 0x0383
122+#define IRDMA_AE_INVALID_ARP_ENTRY 0x0401
123+#define IRDMA_AE_INVALID_TCP_OPTION_RCVD 0x0402
124+#define IRDMA_AE_STALE_ARP_ENTRY 0x0403
125+#define IRDMA_AE_INVALID_AH_ENTRY 0x0406
126+#define IRDMA_AE_LLP_CLOSE_COMPLETE 0x0501
127+#define IRDMA_AE_LLP_CONNECTION_RESET 0x0502
128+#define IRDMA_AE_LLP_FIN_RECEIVED 0x0503
129+#define IRDMA_AE_LLP_RECEIVED_MARKER_AND_LENGTH_FIELDS_DONT_MATCH 0x0504
130+#define IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR 0x0505
131+#define IRDMA_AE_LLP_SEGMENT_TOO_SMALL 0x0507
132+#define IRDMA_AE_LLP_SYN_RECEIVED 0x0508
133+#define IRDMA_AE_LLP_TERMINATE_RECEIVED 0x0509
134+#define IRDMA_AE_LLP_TOO_MANY_RETRIES 0x050a
135+#define IRDMA_AE_LLP_TOO_MANY_KEEPALIVE_RETRIES 0x050b
136+#define IRDMA_AE_LLP_DOUBT_REACHABILITY 0x050c
137+#define IRDMA_AE_LLP_CONNECTION_ESTABLISHED 0x050e
138+#define IRDMA_AE_LLP_TOO_MANY_RNRS 0x050f
139+#define IRDMA_AE_RESOURCE_EXHAUSTION 0x0520
140+#define IRDMA_AE_RESET_SENT 0x0601
141+#define IRDMA_AE_TERMINATE_SENT 0x0602
142+#define IRDMA_AE_RESET_NOT_SENT 0x0603
143+#define IRDMA_AE_LCE_QP_CATASTROPHIC 0x0700
144+#define IRDMA_AE_LCE_FUNCTION_CATASTROPHIC 0x0701
145+#define IRDMA_AE_LCE_CQ_CATASTROPHIC 0x0702
146+#define IRDMA_AE_REMOTE_QP_CATASTROPHIC 0x0703
147+#define IRDMA_AE_LOCAL_QP_CATASTROPHIC 0x0704
148+#define IRDMA_AE_RCE_QP_CATASTROPHIC 0x0705
149+#define IRDMA_AE_QP_SUSPEND_COMPLETE 0x0900
150+#define IRDMA_AE_CQP_DEFERRED_COMPLETE 0x0901
151+#define IRDMA_AE_ADAPTER_CATASTROPHIC 0x0B0B
48152
49153 enum irdma_device_caps_const {
50154 IRDMA_WQE_SIZE = 4,
@@ -55,11 +159,12 @@ enum irdma_device_caps_const { · 4 unmodified lines
55159 IRDMA_CEQE_SIZE = 1,
56160 IRDMA_CQP_CTX_SIZE = 8,
57161 IRDMA_SHADOW_AREA_SIZE = 8,
58− IRDMA_QUERY_FPM_BUF_SIZE = 176,
59− IRDMA_COMMIT_FPM_BUF_SIZE = 176,
162+ IRDMA_QUERY_FPM_BUF_SIZE = 192,
163+ IRDMA_COMMIT_FPM_BUF_SIZE = 192,
60164 IRDMA_GATHER_STATS_BUF_SIZE = 1024,
61165 IRDMA_MIN_IW_QP_ID = 0,
62166 IRDMA_MAX_IW_QP_ID = 262143,
167+ IRDMA_MIN_IW_SRQ_ID = 0,
63168 IRDMA_MIN_CEQID = 0,
64169 IRDMA_MAX_CEQID = 1023,
65170 IRDMA_CEQ_MAX_COUNT = IRDMA_MAX_CEQID + 1,
@@ -67,6 +172,7 @@ enum irdma_device_caps_const { · 1 unmodified line
67172 IRDMA_MAX_CQID = 524287,
68173 IRDMA_MIN_AEQ_ENTRIES = 1,
69174 IRDMA_MAX_AEQ_ENTRIES = 524287,
175+ IRDMA_MAX_AEQ_ENTRIES_GEN_3 = 262144,
70176 IRDMA_MIN_CEQ_ENTRIES = 1,
71177 IRDMA_MAX_CEQ_ENTRIES = 262143,
72178 IRDMA_MIN_CQ_SIZE = 1,
@@ -105,6 +211,13 @@ enum irdma_flush_opcode { · 32 unmodified lines
105211 FLUSH_RETRY_EXC_ERR,
106212 FLUSH_MW_BIND_ERR,
107213 FLUSH_REM_INV_REQ_ERR,
214+ FLUSH_RNR_RETRY_EXC_ERR,
215+};
216+
217+enum irdma_qp_event_type {
218+ IRDMA_QP_EVENT_CATASTROPHIC,
219+ IRDMA_QP_EVENT_ACCESS_ERR,
220+ IRDMA_QP_EVENT_REQ_ERR,
108221 };
109222
110223 enum irdma_cmpl_status {
@@ -147,6 +260,8 @@ enum irdma_qp_caps { · 36 unmodified lines
147260 IRDMA_PUSH_MODE = 8,
148261 };
149262
263+struct irdma_srq_uk;
264+struct irdma_srq_uk_init_info;
150265 struct irdma_qp_uk;
151266 struct irdma_cq_uk;
152267 struct irdma_qp_uk_init_info;
@@ -201,6 +316,24 @@ struct irdma_bind_window { · 48 unmodified lines
201316 bool ena_writes:1;
202317 irdma_stag mw_stag;
203318 bool mem_window_type_1:1;
319+ bool remote_atomics_en:1;
320+};
321+
322+struct irdma_atomic_fetch_add {
323+ u64 tagged_offset;
324+ u64 remote_tagged_offset;
325+ u64 fetch_add_data_bytes;
326+ u32 stag;
327+ u32 remote_stag;
328+};
329+
330+struct irdma_atomic_compare_swap {
331+ u64 tagged_offset;
332+ u64 remote_tagged_offset;
333+ u64 swap_data_bytes;
334+ u64 compare_data_bytes;
335+ u32 stag;
336+ u32 remote_stag;
204337 };
205338
206339 struct irdma_inv_local_stag {
@@ -219,6 +352,7 @@ struct irdma_post_sq_info { · 12 unmodified lines
219352 bool report_rtt:1;
220353 bool udp_hdr:1;
221354 bool defer_flag:1;
355+ bool remote_atomic_en:1;
222356 u32 imm_data;
223357 u32 stag_to_inv;
224358 union {
@@ -227,6 +361,8 @@ struct irdma_post_sq_info { · 2 unmodified lines
227361 struct irdma_rdma_read rdma_read;
228362 struct irdma_bind_window bind_window;
229363 struct irdma_inv_local_stag inv_local_stag;
364+ struct irdma_atomic_fetch_add atomic_fetch_add;
365+ struct irdma_atomic_compare_swap atomic_compare_swap;
230366 } op;
231367 };
232368
@@ -255,6 +391,15 @@ struct irdma_cq_poll_info { · 22 unmodified lines
255391 bool imm_valid:1;
256392 };
257393
394+struct qp_err_code {
395+ enum irdma_flush_opcode flush_code;
396+ enum irdma_qp_event_type event_type;
397+};
398+
399+int irdma_uk_atomic_compare_swap(struct irdma_qp_uk *qp,
400+ struct irdma_post_sq_info *info, bool post_sq);
401+int irdma_uk_atomic_fetch_add(struct irdma_qp_uk *qp,
402+ struct irdma_post_sq_info *info, bool post_sq);
258403 int irdma_uk_inline_rdma_write(struct irdma_qp_uk *qp,
259404 struct irdma_post_sq_info *info, bool post_sq);
260405 int irdma_uk_inline_send(struct irdma_qp_uk *qp,
@@ -300,6 +445,39 @@ int irdma_uk_calc_depth_shift_sq(struct irdma_qp_uk_init_info *ukinfo, · 39 unmodified lines
300445 u32 *sq_depth, u8 *sq_shift);
301446 int irdma_uk_calc_depth_shift_rq(struct irdma_qp_uk_init_info *ukinfo,
302447 u32 *rq_depth, u8 *rq_shift);
448+int irdma_uk_srq_init(struct irdma_srq_uk *srq,
449+ struct irdma_srq_uk_init_info *info);
450+int irdma_uk_srq_post_receive(struct irdma_srq_uk *srq,
451+ struct irdma_post_rq_info *info);
452+
453+struct irdma_srq_uk {
454+ u32 srq_caps;
455+ struct irdma_qp_quanta *srq_base;
456+ struct irdma_uk_attrs *uk_attrs;
457+ __le64 *shadow_area;
458+ struct irdma_ring srq_ring;
459+ struct irdma_ring initial_ring;
460+ u32 srq_id;
461+ u32 srq_size;
462+ u32 max_srq_frag_cnt;
463+ struct irdma_wqe_uk_ops wqe_ops;
464+ u8 srwqe_polarity;
465+ u8 wqe_size;
466+ u8 wqe_size_multiplier;
467+ u8 deferred_flag;
468+};
469+
470+struct irdma_srq_uk_init_info {
471+ struct irdma_qp_quanta *srq;
472+ struct irdma_uk_attrs *uk_attrs;
473+ __le64 *shadow_area;
474+ u64 *srq_wrid_array;
475+ u32 srq_id;
476+ u32 srq_caps;
477+ u32 srq_size;
478+ u32 max_srq_frag_cnt;
479+};
480+
303481 struct irdma_sq_uk_wr_trk_info {
304482 u64 wrid;
305483 u32 wr_len;
@@ -344,6 +522,7 @@ struct irdma_qp_uk { · 38 unmodified lines
344522 bool destroy_pending:1; /* Indicates the QP is being destroyed */
345523 void *back_qp;
346524 u8 dbg_rq_flushed;
525+ struct irdma_srq_uk *srq_uk;
347526 u8 sq_flush_seen;
348527 u8 rq_flush_seen;
349528 };
@@ -383,6 +562,7 @@ struct irdma_qp_uk_init_info { · 33 unmodified lines
383562 u8 rq_shift;
384563 int abi_ver;
385564 bool legacy_mode;
565+ struct irdma_srq_uk *srq_uk;
386566 };
387567
388568 struct irdma_cq_uk_init_info {
@@ -398,6 +578,7 @@ struct irdma_cq_uk_init_info { · 9 unmodified lines
398578 __le64 *irdma_qp_get_next_send_wqe(struct irdma_qp_uk *qp, u32 *wqe_idx,
399579 u16 quanta, u32 total_size,
400580 struct irdma_post_sq_info *info);
581+__le64 *irdma_srq_get_next_recv_wqe(struct irdma_srq_uk *srq, u32 *wqe_idx);
401582 __le64 *irdma_qp_get_next_recv_wqe(struct irdma_qp_uk *qp, u32 *wqe_idx);
402583 void irdma_uk_clean_cq(void *q, struct irdma_cq_uk *cq);
403584 int irdma_nop(struct irdma_qp_uk *qp, u64 wr_id, bool signaled, bool post_sq);
@@ -409,5 +590,85 @@ int irdma_get_sqdepth(struct irdma_uk_attrs *uk_attrs, u32 sq_size, u8 shift, · 5 unmodified lines
409590 u32 *wqdepth);
410591 int irdma_get_rqdepth(struct irdma_uk_attrs *uk_attrs, u32 rq_size, u8 shift,
411592 u32 *wqdepth);
593+int irdma_get_srqdepth(struct irdma_uk_attrs *uk_attrs, u32 srq_size, u8 shift,
594+ u32 *srqdepth);
412595 void irdma_clr_wqes(struct irdma_qp_uk *qp, u32 qp_wqe_idx);
596+
597+static inline struct qp_err_code irdma_ae_to_qp_err_code(u16 ae_id)
598+{
599+ struct qp_err_code qp_err = {};
600+
601+ switch (ae_id) {
602+ case IRDMA_AE_AMP_BOUNDS_VIOLATION:
603+ case IRDMA_AE_AMP_INVALID_STAG:
604+ case IRDMA_AE_AMP_RIGHTS_VIOLATION:
605+ case IRDMA_AE_AMP_UNALLOCATED_STAG:
606+ case IRDMA_AE_AMP_BAD_PD:
607+ case IRDMA_AE_AMP_BAD_QP:
608+ case IRDMA_AE_AMP_BAD_STAG_KEY:
609+ case IRDMA_AE_AMP_BAD_STAG_INDEX:
610+ case IRDMA_AE_AMP_TO_WRAP:
611+ case IRDMA_AE_PRIV_OPERATION_DENIED:
612+ qp_err.flush_code = FLUSH_PROT_ERR;
613+ qp_err.event_type = IRDMA_QP_EVENT_ACCESS_ERR;
614+ break;
615+ case IRDMA_AE_UDA_XMIT_BAD_PD:
616+ case IRDMA_AE_WQE_UNEXPECTED_OPCODE:
617+ qp_err.flush_code = FLUSH_LOC_QP_OP_ERR;
618+ qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC;
619+ break;
620+ case IRDMA_AE_UDA_XMIT_DGRAM_TOO_SHORT:
621+ case IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG:
622+ case IRDMA_AE_UDA_L4LEN_INVALID:
623+ case IRDMA_AE_DDP_UBE_INVALID_MO:
624+ case IRDMA_AE_DDP_UBE_DDP_MESSAGE_TOO_LONG_FOR_AVAILABLE_BUFFER:
625+ qp_err.flush_code = FLUSH_LOC_LEN_ERR;
626+ qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC;
627+ break;
628+ case IRDMA_AE_AMP_INVALIDATE_NO_REMOTE_ACCESS_RIGHTS:
629+ case IRDMA_AE_IB_REMOTE_ACCESS_ERROR:
630+ qp_err.flush_code = FLUSH_REM_ACCESS_ERR;
631+ qp_err.event_type = IRDMA_QP_EVENT_ACCESS_ERR;
632+ break;
633+ case IRDMA_AE_AMP_MWBIND_INVALID_RIGHTS:
634+ case IRDMA_AE_AMP_MWBIND_BIND_DISABLED:
635+ case IRDMA_AE_AMP_MWBIND_INVALID_BOUNDS:
636+ case IRDMA_AE_AMP_MWBIND_VALID_STAG:
637+ qp_err.flush_code = FLUSH_MW_BIND_ERR;
638+ qp_err.event_type = IRDMA_QP_EVENT_ACCESS_ERR;
639+ break;
640+ case IRDMA_AE_LLP_TOO_MANY_RETRIES:
641+ qp_err.flush_code = FLUSH_RETRY_EXC_ERR;
642+ qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC;
643+ break;
644+ case IRDMA_AE_IB_INVALID_REQUEST:
645+ qp_err.flush_code = FLUSH_REM_INV_REQ_ERR;
646+ qp_err.event_type = IRDMA_QP_EVENT_REQ_ERR;
647+ break;
648+ case IRDMA_AE_LLP_SEGMENT_TOO_SMALL:
649+ case IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR:
650+ case IRDMA_AE_ROCE_RSP_LENGTH_ERROR:
651+ case IRDMA_AE_IB_REMOTE_OP_ERROR:
652+ qp_err.flush_code = FLUSH_REM_OP_ERR;
653+ qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC;
654+ break;
655+ case IRDMA_AE_LLP_TOO_MANY_RNRS:
656+ qp_err.flush_code = FLUSH_RNR_RETRY_EXC_ERR;
657+ qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC;
658+ break;
659+ case IRDMA_AE_LCE_QP_CATASTROPHIC:
660+ case IRDMA_AE_REMOTE_QP_CATASTROPHIC:
661+ case IRDMA_AE_LOCAL_QP_CATASTROPHIC:
662+ case IRDMA_AE_RCE_QP_CATASTROPHIC:
663+ qp_err.flush_code = FLUSH_FATAL_ERR;
664+ qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC;
665+ break;
666+ default:
667+ qp_err.flush_code = FLUSH_GENERAL_ERR;
668+ qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC;
669+ break;
670+ }
671+
672+ return qp_err;
673+}
413674 #endif /* IRDMA_USER_H */
drivers/infiniband/hw/irdma/utils.c+107−5
@@ -481,6 +481,7 @@ void irdma_free_cqp_request(struct irdma_cqp *cqp, · 480 unmodified lines
481481 WRITE_ONCE(cqp_request->request_done, false);
482482 cqp_request->callback_fcn = NULL;
483483 cqp_request->waiting = false;
484+ cqp_request->pending = false;
484485
485486 spin_lock_irqsave(&cqp->req_lock, flags);
486487 list_add_tail(&cqp_request->list, &cqp->cqp_avail_reqs);
@@ -520,6 +521,22 @@ irdma_free_pending_cqp_request(struct irdma_cqp *cqp, · 33 unmodified lines
520521 irdma_put_cqp_request(cqp, cqp_request);
521522 }
522523
524+/**
525+ * irdma_cleanup_deferred_cqp_ops - clean-up cqp with no completions
526+ * @dev: sc_dev
527+ * @cqp: cqp
528+ */
529+static void irdma_cleanup_deferred_cqp_ops(struct irdma_sc_dev *dev,
530+ struct irdma_cqp *cqp)
531+{
532+ u64 scratch;
533+
534+ /* process all CQP requests with deferred/pending completions */
535+ while ((scratch = irdma_sc_cqp_cleanup_handler(dev)))
536+ irdma_free_pending_cqp_request(cqp, (struct irdma_cqp_request *)
537+ (uintptr_t)scratch);
538+}
539+
523540 /**
524541 * irdma_cleanup_pending_cqp_op - clean-up cqp with no
525542 * completions
@@ -533,6 +550,8 @@ void irdma_cleanup_pending_cqp_op(struct irdma_pci_f *rf) · 7 unmodified lines
533550 struct cqp_cmds_info *pcmdinfo = NULL;
534551 u32 i, pending_work, wqe_idx;
535552
553+ if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3)
554+ irdma_cleanup_deferred_cqp_ops(dev, cqp);
536555 pending_work = IRDMA_RING_USED_QUANTA(cqp->sc_cqp.sq_ring);
537556 wqe_idx = IRDMA_RING_CURRENT_TAIL(cqp->sc_cqp.sq_ring);
538557 for (i = 0; i < pending_work; i++) {
@@ -552,6 +571,26 @@ void irdma_cleanup_pending_cqp_op(struct irdma_pci_f *rf) · 13 unmodified lines
552571 }
553572 }
554573
574+static int irdma_get_timeout_threshold(struct irdma_sc_dev *dev)
575+{
576+ u16 time_s = dev->vc_caps.cqp_timeout_s;
577+
578+ if (!time_s)
579+ return CQP_TIMEOUT_THRESHOLD;
580+
581+ return time_s * 1000 / dev->hw_attrs.max_cqp_compl_wait_time_ms;
582+}
583+
584+static int irdma_get_def_timeout_threshold(struct irdma_sc_dev *dev)
585+{
586+ u16 time_s = dev->vc_caps.cqp_def_timeout_s;
587+
588+ if (!time_s)
589+ return CQP_DEF_CMPL_TIMEOUT_THRESHOLD;
590+
591+ return time_s * 1000 / dev->hw_attrs.max_cqp_compl_wait_time_ms;
592+}
593+
555594 /**
556595 * irdma_wait_event - wait for completion
557596 * @rf: RDMA PCI function
@@ -561,6 +600,7 @@ static int irdma_wait_event(struct irdma_pci_f *rf, · 3 unmodified lines
561600 struct irdma_cqp_request *cqp_request)
562601 {
563602 struct irdma_cqp_timeout cqp_timeout = {};
603+ int timeout_threshold = irdma_get_timeout_threshold(&rf->sc_dev);
564604 bool cqp_error = false;
565605 int err_code = 0;
566606
@@ -572,9 +612,17 @@ static int irdma_wait_event(struct irdma_pci_f *rf, · 5 unmodified lines
572612 msecs_to_jiffies(CQP_COMPL_WAIT_TIME_MS)))
573613 break;
574614
615+ if (cqp_request->pending)
616+ /* There was a deferred or pending completion
617+ * received for this CQP request, so we need
618+ * to wait longer than usual.
619+ */
620+ timeout_threshold =
621+ irdma_get_def_timeout_threshold(&rf->sc_dev);
622+
575623 irdma_check_cqp_progress(&cqp_timeout, &rf->sc_dev);
576624
577− if (cqp_timeout.count < CQP_TIMEOUT_THRESHOLD)
625+ if (cqp_timeout.count < timeout_threshold)
578626 continue;
579627
580628 if (!rf->reset) {
@@ -649,6 +697,9 @@ static const char *const irdma_cqp_cmd_names[IRDMA_MAX_CQP_OPS] = { · 68 unmodified lines
649697 [IRDMA_OP_ADD_LOCAL_MAC_ENTRY] = "Add Local MAC Entry Cmd",
650698 [IRDMA_OP_DELETE_LOCAL_MAC_ENTRY] = "Delete Local MAC Entry Cmd",
651699 [IRDMA_OP_CQ_MODIFY] = "CQ Modify Cmd",
700+ [IRDMA_OP_SRQ_CREATE] = "Create SRQ Cmd",
701+ [IRDMA_OP_SRQ_MODIFY] = "Modify SRQ Cmd",
702+ [IRDMA_OP_SRQ_DESTROY] = "Destroy SRQ Cmd",
652703 };
653704
654705 static const struct irdma_cqp_err_info irdma_noncrit_err_list[] = {
@@ -1065,6 +1116,26 @@ static void irdma_dealloc_push_page(struct irdma_pci_f *rf, · 410 unmodified lines
10651116 irdma_put_cqp_request(&rf->cqp, cqp_request);
10661117 }
10671118
1119+static void irdma_free_gsi_qp_rsrc(struct irdma_qp *iwqp, u32 qp_num)
1120+{
1121+ struct irdma_device *iwdev = iwqp->iwdev;
1122+ struct irdma_pci_f *rf = iwdev->rf;
1123+ unsigned long flags;
1124+
1125+ if (rf->sc_dev.hw_attrs.uk_attrs.hw_rev < IRDMA_GEN_3)
1126+ return;
1127+
1128+ irdma_vchnl_req_del_vport(&rf->sc_dev, iwdev->vport_id, qp_num);
1129+
1130+ if (qp_num == 1) {
1131+ spin_lock_irqsave(&rf->rsrc_lock, flags);
1132+ rf->hwqp1_rsvd = false;
1133+ spin_unlock_irqrestore(&rf->rsrc_lock, flags);
1134+ } else if (qp_num > 2) {
1135+ irdma_free_rsrc(rf, rf->allocated_qps, qp_num);
1136+ }
1137+}
1138+
10681139 /**
10691140 * irdma_free_qp_rsrc - free up memory resources for qp
10701141 * @iwqp: qp ptr (user or kernel)
@@ -1073,7 +1144,7 @@ void irdma_free_qp_rsrc(struct irdma_qp *iwqp) · 2 unmodified lines
10731144 {
10741145 struct irdma_device *iwdev = iwqp->iwdev;
10751146 struct irdma_pci_f *rf = iwdev->rf;
1076− u32 qp_num = iwqp->ibqp.qp_num;
1147+ u32 qp_num = iwqp->sc_qp.qp_uk.qp_id;
10771148
10781149 irdma_ieq_cleanup_qp(iwdev->vsi.ieq, &iwqp->sc_qp);
10791150 irdma_dealloc_push_page(rf, &iwqp->sc_qp);
@@ -1083,8 +1154,12 @@ void irdma_free_qp_rsrc(struct irdma_qp *iwqp) · 3 unmodified lines
10831154 iwqp->sc_qp.user_pri);
10841155 }
10851156
1086− if (qp_num > 2)
1087− irdma_free_rsrc(rf, rf->allocated_qps, qp_num);
1157+ if (iwqp->ibqp.qp_type == IB_QPT_GSI) {
1158+ irdma_free_gsi_qp_rsrc(iwqp, qp_num);
1159+ } else {
1160+ if (qp_num > 2)
1161+ irdma_free_rsrc(rf, rf->allocated_qps, qp_num);
1162+ }
10881163 dma_free_coherent(rf->sc_dev.hw->device, iwqp->q2_ctx_mem.size,
10891164 iwqp->q2_ctx_mem.va, iwqp->q2_ctx_mem.pa);
10901165 iwqp->q2_ctx_mem.va = NULL;
@@ -1095,6 +1170,30 @@ void irdma_free_qp_rsrc(struct irdma_qp *iwqp) · 4 unmodified lines
10951170 kfree(iwqp->kqp.rq_wrid_mem);
10961171 }
10971172
1173+/**
1174+ * irdma_srq_wq_destroy - send srq destroy cqp
1175+ * @rf: RDMA PCI function
1176+ * @srq: hardware control srq
1177+ */
1178+void irdma_srq_wq_destroy(struct irdma_pci_f *rf, struct irdma_sc_srq *srq)
1179+{
1180+ struct irdma_cqp_request *cqp_request;
1181+ struct cqp_cmds_info *cqp_info;
1182+
1183+ cqp_request = irdma_alloc_and_get_cqp_request(&rf->cqp, true);
1184+ if (!cqp_request)
1185+ return;
1186+
1187+ cqp_info = &cqp_request->info;
1188+ cqp_info->cqp_cmd = IRDMA_OP_SRQ_DESTROY;
1189+ cqp_info->post_sq = 1;
1190+ cqp_info->in.u.srq_destroy.srq = srq;
1191+ cqp_info->in.u.srq_destroy.scratch = (uintptr_t)cqp_request;
1192+
1193+ irdma_handle_cqp_op(rf, cqp_request);
1194+ irdma_put_cqp_request(&rf->cqp, cqp_request);
1195+}
1196+
10981197 /**
10991198 * irdma_cq_wq_destroy - send cq destroy cqp
11001199 * @rf: RDMA PCI function
@@ -2266,7 +2365,10 @@ bool irdma_cq_empty(struct irdma_cq *iwcq) · 1165 unmodified lines
22662365 u8 polarity;
22672366
22682367 ukcq = &iwcq->sc_cq.cq_uk;
2269− cqe = IRDMA_GET_CURRENT_CQ_ELEM(ukcq);
2368+ if (ukcq->avoid_mem_cflct)
2369+ cqe = IRDMA_GET_CURRENT_EXTENDED_CQ_ELEM(ukcq);
2370+ else
2371+ cqe = IRDMA_GET_CURRENT_CQ_ELEM(ukcq);
22702372 get_64bit_val(cqe, 24, &qword3);
22712373 polarity = (u8)FIELD_GET(IRDMA_CQ_VALID, qword3);
22722374
drivers/infiniband/hw/irdma/verbs.c+728−106
@@ -41,7 +41,8 @@ static int irdma_query_device(struct ib_device *ibdev, · 40 unmodified lines
4141 props->max_cq = rf->max_cq - rf->used_cqs;
4242 props->max_cqe = rf->max_cqe - 1;
4343 props->max_mr = rf->max_mr - rf->used_mrs;
44− props->max_mw = props->max_mr;
44+ if (hw_attrs->uk_attrs.hw_rev >= IRDMA_GEN_3)
45+ props->max_mw = props->max_mr;
4546 props->max_pd = rf->max_pd - rf->used_pds;
4647 props->max_sge_rd = hw_attrs->uk_attrs.max_hw_read_sges;
4748 props->max_qp_rd_atom = hw_attrs->max_hw_ird;
@@ -56,9 +57,21 @@ static int irdma_query_device(struct ib_device *ibdev, · 8 unmodified lines
5657 props->max_mcast_qp_attach = IRDMA_MAX_MGS_PER_CTX;
5758 props->max_total_mcast_qp_attach = rf->max_qp * IRDMA_MAX_MGS_PER_CTX;
5859 props->max_fast_reg_page_list_len = IRDMA_MAX_PAGES_PER_FMR;
59−#define HCA_CLOCK_TIMESTAMP_MASK 0x1ffff
60− if (hw_attrs->uk_attrs.hw_rev >= IRDMA_GEN_2)
61− props->timestamp_mask = HCA_CLOCK_TIMESTAMP_MASK;
60+ props->max_srq = rf->max_srq - rf->used_srqs;
61+ props->max_srq_wr = IRDMA_MAX_SRQ_WRS;
62+ props->max_srq_sge = hw_attrs->uk_attrs.max_hw_wq_frags;
63+ if (hw_attrs->uk_attrs.feature_flags & IRDMA_FEATURE_ATOMIC_OPS)
64+ props->atomic_cap = IB_ATOMIC_HCA;
65+ else
66+ props->atomic_cap = IB_ATOMIC_NONE;
67+ props->masked_atomic_cap = props->atomic_cap;
68+ if (hw_attrs->uk_attrs.hw_rev >= IRDMA_GEN_3) {
69+#define HCA_CORE_CLOCK_KHZ 1000000UL
70+ props->timestamp_mask = GENMASK(31, 0);
71+ props->hca_core_clock = HCA_CORE_CLOCK_KHZ;
72+ }
73+ if (hw_attrs->uk_attrs.hw_rev >= IRDMA_GEN_3)
74+ props->device_cap_flags |= IB_DEVICE_MEM_WINDOW_TYPE_2B;
6275
6376 return 0;
6477 }
@@ -292,6 +305,10 @@ static int irdma_alloc_ucontext(struct ib_ucontext *uctx, · 227 unmodified lines
292305 ucontext->iwdev = iwdev;
293306 ucontext->abi_ver = req.userspace_ver;
294307
308+ if (!(req.comp_mask & IRDMA_SUPPORT_WQE_FORMAT_V2) &&
309+ uk_attrs->hw_rev >= IRDMA_GEN_3)
310+ return -EOPNOTSUPP;
311+
295312 if (req.comp_mask & IRDMA_ALLOC_UCTX_USE_RAW_ATTR)
296313 ucontext->use_raw_attrs = true;
297314
@@ -332,6 +349,8 @@ static int irdma_alloc_ucontext(struct ib_ucontext *uctx, · 34 unmodified lines
332349 uresp.comp_mask |= IRDMA_ALLOC_UCTX_USE_RAW_ATTR;
333350 uresp.min_hw_wq_size = uk_attrs->min_hw_wq_size;
334351 uresp.comp_mask |= IRDMA_ALLOC_UCTX_MIN_HW_WQ_SIZE;
352+ uresp.max_hw_srq_quanta = uk_attrs->max_hw_srq_quanta;
353+ uresp.comp_mask |= IRDMA_ALLOC_UCTX_MAX_HW_SRQ_QUANTA;
335354 if (ib_copy_to_udata(udata, &uresp,
336355 min(sizeof(uresp), udata->outlen))) {
337356 rdma_user_mmap_entry_remove(ucontext->db_mmap_entry);
@@ -343,6 +362,8 @@ static int irdma_alloc_ucontext(struct ib_ucontext *uctx, · 5 unmodified lines
343362 spin_lock_init(&ucontext->cq_reg_mem_list_lock);
344363 INIT_LIST_HEAD(&ucontext->qp_reg_mem_list);
345364 spin_lock_init(&ucontext->qp_reg_mem_list_lock);
365+ INIT_LIST_HEAD(&ucontext->srq_reg_mem_list);
366+ spin_lock_init(&ucontext->srq_reg_mem_list_lock);
346367
347368 return 0;
348369
@@ -521,7 +542,7 @@ static int irdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) · 172 unmodified lines
521542
522543 iwqp->sc_qp.qp_uk.destroy_pending = true;
523544
524− if (iwqp->iwarp_state == IRDMA_QP_STATE_RTS)
545+ if (iwqp->iwarp_state >= IRDMA_QP_STATE_IDLE)
525546 irdma_modify_qp_to_err(&iwqp->sc_qp);
526547
527548 if (!iwqp->user_mode)
@@ -541,6 +562,9 @@ static int irdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) · 13 unmodified lines
541562 irdma_cqp_qp_destroy_cmd(&iwdev->rf->sc_dev, &iwqp->sc_qp);
542563
543564 irdma_remove_push_mmap_entries(iwqp);
565+
566+ if (iwqp->sc_qp.qp_uk.qp_id == 1)
567+ iwdev->rf->hwqp1_rsvd = false;
544568 irdma_free_qp_rsrc(iwqp);
545569
546570 return 0;
@@ -564,7 +588,11 @@ static void irdma_setup_virt_qp(struct irdma_device *iwdev, · 17 unmodified lines
564588 if (iwpbl->pbl_allocated) {
565589 init_info->virtual_map = true;
566590 init_info->sq_pa = qpmr->sq_pbl.idx;
567− init_info->rq_pa = qpmr->rq_pbl.idx;
591+ /* Need to use contiguous buffer for RQ of QP
592+ * in case it is associated with SRQ.
593+ */
594+ init_info->rq_pa = init_info->qp_uk_init_info.srq_uk ?
595+ qpmr->rq_pa : qpmr->rq_pbl.idx;
568596 } else {
569597 init_info->sq_pa = qpmr->sq_pbl.addr;
570598 init_info->rq_pa = qpmr->rq_pbl.addr;
@@ -719,6 +747,7 @@ static int irdma_setup_kmode_qp(struct irdma_device *iwdev, · 148 unmodified lines
719747 info->rq_pa + (ukinfo->rq_depth * IRDMA_QP_WQE_MIN_SIZE);
720748 ukinfo->sq_size = ukinfo->sq_depth >> ukinfo->sq_shift;
721749 ukinfo->rq_size = ukinfo->rq_depth >> ukinfo->rq_shift;
750+ ukinfo->qp_id = info->qp_uk_init_info.qp_id;
722751
723752 iwqp->max_send_wr = (ukinfo->sq_depth - IRDMA_SQ_RSVD) >> ukinfo->sq_shift;
724753 iwqp->max_recv_wr = (ukinfo->rq_depth - IRDMA_RQ_RSVD) >> ukinfo->rq_shift;
@@ -775,9 +804,12 @@ static void irdma_roce_fill_and_set_qpctx_info(struct irdma_qp *iwqp, · 50 unmodified lines
775804 roce_info = &iwqp->roce_info;
776805 ether_addr_copy(roce_info->mac_addr, iwdev->netdev->dev_addr);
777806
807+ if (iwqp->ibqp.qp_type == IB_QPT_GSI && iwqp->ibqp.qp_num != 1)
808+ roce_info->is_qp1 = true;
778809 roce_info->rd_en = true;
779810 roce_info->wr_rdresp_en = true;
780− roce_info->bind_en = true;
811+ if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3)
812+ roce_info->bind_en = true;
781813 roce_info->dcqcn_en = false;
782814 roce_info->rtomin = 5;
783815
@@ -808,7 +840,6 @@ static void irdma_iw_fill_and_set_qpctx_info(struct irdma_qp *iwqp, · 24 unmodified lines
808840 ether_addr_copy(iwarp_info->mac_addr, iwdev->netdev->dev_addr);
809841 iwarp_info->rd_en = true;
810842 iwarp_info->wr_rdresp_en = true;
811− iwarp_info->bind_en = true;
812843 iwarp_info->ecn_en = true;
813844 iwarp_info->rtomin = 5;
814845
@@ -864,6 +895,47 @@ static void irdma_flush_worker(struct work_struct *work) · 49 unmodified lines
864895 irdma_generate_flush_completions(iwqp);
865896 }
866897
898+static int irdma_setup_gsi_qp_rsrc(struct irdma_qp *iwqp, u32 *qp_num)
899+{
900+ struct irdma_device *iwdev = iwqp->iwdev;
901+ struct irdma_pci_f *rf = iwdev->rf;
902+ unsigned long flags;
903+ int ret;
904+
905+ if (rf->rdma_ver <= IRDMA_GEN_2) {
906+ *qp_num = 1;
907+ return 0;
908+ }
909+
910+ spin_lock_irqsave(&rf->rsrc_lock, flags);
911+ if (!rf->hwqp1_rsvd) {
912+ *qp_num = 1;
913+ rf->hwqp1_rsvd = true;
914+ spin_unlock_irqrestore(&rf->rsrc_lock, flags);
915+ } else {
916+ spin_unlock_irqrestore(&rf->rsrc_lock, flags);
917+ ret = irdma_alloc_rsrc(rf, rf->allocated_qps, rf->max_qp,
918+ qp_num, &rf->next_qp);
919+ if (ret)
920+ return ret;
921+ }
922+
923+ ret = irdma_vchnl_req_add_vport(&rf->sc_dev, iwdev->vport_id, *qp_num,
924+ (&iwdev->vsi)->qos);
925+ if (ret) {
926+ if (*qp_num != 1) {
927+ irdma_free_rsrc(rf, rf->allocated_qps, *qp_num);
928+ } else {
929+ spin_lock_irqsave(&rf->rsrc_lock, flags);
930+ rf->hwqp1_rsvd = false;
931+ spin_unlock_irqrestore(&rf->rsrc_lock, flags);
932+ }
933+ return ret;
934+ }
935+
936+ return 0;
937+}
938+
867939 /**
868940 * irdma_create_qp - create qp
869941 * @ibqp: ptr of qp
@@ -889,6 +961,18 @@ static int irdma_create_qp(struct ib_qp *ibqp, · 19 unmodified lines
889961 struct irdma_uk_attrs *uk_attrs = &dev->hw_attrs.uk_attrs;
890962 struct irdma_qp_init_info init_info = {};
891963 struct irdma_qp_host_ctx_info *ctx_info;
964+ struct irdma_srq *iwsrq;
965+ bool srq_valid = false;
966+ u32 srq_id = 0;
967+
968+ if (init_attr->srq) {
969+ iwsrq = to_iwsrq(init_attr->srq);
970+ srq_valid = true;
971+ srq_id = iwsrq->srq_num;
972+ init_attr->cap.max_recv_sge = uk_attrs->max_hw_wq_frags;
973+ init_attr->cap.max_recv_wr = 4;
974+ init_info.qp_uk_init_info.srq_uk = &iwsrq->sc_srq.srq_uk;
975+ }
892976
893977 err_code = irdma_validate_qp_attrs(init_attr, iwdev);
894978 if (err_code)
@@ -925,16 +1009,20 @@ static int irdma_create_qp(struct ib_qp *ibqp, · 30 unmodified lines
9251009 init_info.host_ctx = (__le64 *)(init_info.q2 + IRDMA_Q2_BUF_SIZE);
9261010 init_info.host_ctx_pa = init_info.q2_pa + IRDMA_Q2_BUF_SIZE;
9271011
928− if (init_attr->qp_type == IB_QPT_GSI)
929− qp_num = 1;
930− else
1012+ if (init_attr->qp_type == IB_QPT_GSI) {
1013+ err_code = irdma_setup_gsi_qp_rsrc(iwqp, &qp_num);
1014+ if (err_code)
1015+ goto error;
1016+ iwqp->ibqp.qp_num = 1;
1017+ } else {
9311018 err_code = irdma_alloc_rsrc(rf, rf->allocated_qps, rf->max_qp,
9321019 &qp_num, &rf->next_qp);
933− if (err_code)
934− goto error;
1020+ if (err_code)
1021+ goto error;
1022+ iwqp->ibqp.qp_num = qp_num;
1023+ }
9351024
9361025 iwqp->iwpd = iwpd;
937− iwqp->ibqp.qp_num = qp_num;
9381026 qp = &iwqp->sc_qp;
9391027 iwqp->iwscq = to_iwcq(init_attr->send_cq);
9401028 iwqp->iwrcq = to_iwcq(init_attr->recv_cq);
@@ -991,13 +1079,22 @@ static int irdma_create_qp(struct ib_qp *ibqp, · 50 unmodified lines
9911079 }
9921080
9931081 ctx_info = &iwqp->ctx_info;
1082+ ctx_info->srq_valid = srq_valid;
1083+ ctx_info->srq_id = srq_id;
9941084 ctx_info->send_cq_num = iwqp->iwscq->sc_cq.cq_uk.cq_id;
9951085 ctx_info->rcv_cq_num = iwqp->iwrcq->sc_cq.cq_uk.cq_id;
9961086
997− if (rdma_protocol_roce(&iwdev->ibdev, 1))
1087+ if (rdma_protocol_roce(&iwdev->ibdev, 1)) {
1088+ if (dev->ws_add(&iwdev->vsi, 0)) {
1089+ irdma_cqp_qp_destroy_cmd(&rf->sc_dev, &iwqp->sc_qp);
1090+ err_code = -EINVAL;
1091+ goto error;
1092+ }
1093+ irdma_qp_add_qos(&iwqp->sc_qp);
9981094 irdma_roce_fill_and_set_qpctx_info(iwqp, ctx_info);
999− else
1095+ } else {
10001096 irdma_iw_fill_and_set_qpctx_info(iwqp, ctx_info);
1097+ }
10011098
10021099 err_code = irdma_cqp_create_qp_cmd(iwqp);
10031100 if (err_code)
@@ -1009,16 +1106,6 @@ static int irdma_create_qp(struct ib_qp *ibqp, · 5 unmodified lines
10091106 iwqp->sig_all = init_attr->sq_sig_type == IB_SIGNAL_ALL_WR;
10101107 rf->qp_table[qp_num] = iwqp;
10111108
1012− if (rdma_protocol_roce(&iwdev->ibdev, 1)) {
1013− if (dev->ws_add(&iwdev->vsi, 0)) {
1014− irdma_cqp_qp_destroy_cmd(&rf->sc_dev, &iwqp->sc_qp);
1015− err_code = -EINVAL;
1016− goto error;
1017− }
1018−
1019− irdma_qp_add_qos(&iwqp->sc_qp);
1020− }
1021−
10221109 if (udata) {
10231110 /* GEN_1 legacy support with libi40iw does not have expanded uresp struct */
10241111 if (udata->outlen < sizeof(uresp)) {
@@ -1063,6 +1150,8 @@ static int irdma_get_ib_acc_flags(struct irdma_qp *iwqp) · 38 unmodified lines
10631150 acc_flags |= IB_ACCESS_REMOTE_READ;
10641151 if (iwqp->roce_info.bind_en)
10651152 acc_flags |= IB_ACCESS_MW_BIND;
1153+ if (iwqp->ctx_info.remote_atomics_en)
1154+ acc_flags |= IB_ACCESS_REMOTE_ATOMIC;
10661155 } else {
10671156 if (iwqp->iwarp_info.wr_rdresp_en) {
10681157 acc_flags |= IB_ACCESS_LOCAL_WRITE;
@@ -1070,8 +1159,8 @@ static int irdma_get_ib_acc_flags(struct irdma_qp *iwqp) · 1 unmodified line
10701159 }
10711160 if (iwqp->iwarp_info.rd_en)
10721161 acc_flags |= IB_ACCESS_REMOTE_READ;
1073− if (iwqp->iwarp_info.bind_en)
1074− acc_flags |= IB_ACCESS_MW_BIND;
1162+ if (iwqp->ctx_info.remote_atomics_en)
1163+ acc_flags |= IB_ACCESS_REMOTE_ATOMIC;
10751164 }
10761165 return acc_flags;
10771166 }
@@ -1110,6 +1199,7 @@ static int irdma_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 32 unmodified lines
11101199 attr->pkey_index = iwqp->roce_info.p_key;
11111200 attr->retry_cnt = iwqp->udp_info.rexmit_thresh;
11121201 attr->rnr_retry = iwqp->udp_info.rnr_nak_thresh;
1202+ attr->min_rnr_timer = iwqp->udp_info.min_rnr_timer;
11131203 attr->max_rd_atomic = iwqp->roce_info.ord_size;
11141204 attr->max_dest_rd_atomic = iwqp->roce_info.ird_size;
11151205 }
@@ -1118,6 +1208,7 @@ static int irdma_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 2 unmodified lines
11181208 init_attr->qp_context = iwqp->ibqp.qp_context;
11191209 init_attr->send_cq = iwqp->ibqp.send_cq;
11201210 init_attr->recv_cq = iwqp->ibqp.recv_cq;
1211+ init_attr->srq = iwqp->ibqp.srq;
11211212 init_attr->cap = attr->cap;
11221213
11231214 return 0;
@@ -1242,6 +1333,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 118 unmodified lines
12421333 if (attr_mask & IB_QP_RNR_RETRY)
12431334 udp_info->rnr_nak_thresh = attr->rnr_retry;
12441335
1336+ if (attr_mask & IB_QP_MIN_RNR_TIMER &&
1337+ dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3)
1338+ udp_info->min_rnr_timer = attr->min_rnr_timer;
1339+
12451340 if (attr_mask & IB_QP_RETRY_CNT)
12461341 udp_info->rexmit_thresh = attr->retry_cnt;
12471342
@@ -1362,6 +1457,9 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 114 unmodified lines
13621457 roce_info->wr_rdresp_en = true;
13631458 if (attr->qp_access_flags & IB_ACCESS_REMOTE_READ)
13641459 roce_info->rd_en = true;
1460+ if (dev->hw_attrs.uk_attrs.feature_flags & IRDMA_FEATURE_ATOMIC_OPS)
1461+ if (attr->qp_access_flags & IB_ACCESS_REMOTE_ATOMIC)
1462+ ctx_info->remote_atomics_en = true;
13651463 }
13661464
13671465 wait_event(iwqp->mod_qp_waitq, !atomic_read(&iwqp->hw_mod_qp_pend));
@@ -1776,6 +1874,24 @@ exit: · 408 unmodified lines
17761874 return err;
17771875 }
17781876
1877+/**
1878+ * irdma_srq_free_rsrc - free up resources for srq
1879+ * @rf: RDMA PCI function
1880+ * @iwsrq: srq ptr
1881+ */
1882+static void irdma_srq_free_rsrc(struct irdma_pci_f *rf, struct irdma_srq *iwsrq)
1883+{
1884+ struct irdma_sc_srq *srq = &iwsrq->sc_srq;
1885+
1886+ if (!iwsrq->user_mode) {
1887+ dma_free_coherent(rf->sc_dev.hw->device, iwsrq->kmem.size,
1888+ iwsrq->kmem.va, iwsrq->kmem.pa);
1889+ iwsrq->kmem.va = NULL;
1890+ }
1891+
1892+ irdma_free_rsrc(rf, rf->allocated_srqs, srq->srq_uk.srq_id);
1893+}
1894+
17791895 /**
17801896 * irdma_cq_free_rsrc - free up resources for cq
17811897 * @rf: RDMA PCI function
@@ -1839,6 +1955,22 @@ static int irdma_process_resize_list(struct irdma_cq *iwcq, · 57 unmodified lines
18391955 return cnt;
18401956 }
18411957
1958+/**
1959+ * irdma_destroy_srq - destroy srq
1960+ * @ibsrq: srq pointer
1961+ * @udata: user data
1962+ */
1963+static int irdma_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)
1964+{
1965+ struct irdma_device *iwdev = to_iwdev(ibsrq->device);
1966+ struct irdma_srq *iwsrq = to_iwsrq(ibsrq);
1967+ struct irdma_sc_srq *srq = &iwsrq->sc_srq;
1968+
1969+ irdma_srq_wq_destroy(iwdev->rf, srq);
1970+ irdma_srq_free_rsrc(iwdev->rf, iwsrq);
1971+ return 0;
1972+}
1973+
18421974 /**
18431975 * irdma_destroy_cq - destroy cq
18441976 * @ib_cq: cq pointer
@@ -1914,8 +2046,13 @@ static int irdma_resize_cq(struct ib_cq *ibcq, int entries, · 69 unmodified lines
19142046
19152047 if (!iwcq->user_mode) {
19162048 entries++;
1917− if (rf->sc_dev.hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_2)
2049+
2050+ if (!iwcq->sc_cq.cq_uk.avoid_mem_cflct &&
2051+ dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_2)
19182052 entries *= 2;
2053+
2054+ if (entries & 1)
2055+ entries += 1; /* cq size must be an even number */
19192056 }
19202057
19212058 info.cq_size = max(entries, 4);
@@ -2022,10 +2159,297 @@ error: · 100 unmodified lines
20222159 return ret;
20232160 }
20242161
2162+/**
2163+ * irdma_srq_event - event notification for srq limit
2164+ * @srq: shared srq struct
2165+ */
2166+void irdma_srq_event(struct irdma_sc_srq *srq)
2167+{
2168+ struct irdma_srq *iwsrq = container_of(srq, struct irdma_srq, sc_srq);
2169+ struct ib_srq *ibsrq = &iwsrq->ibsrq;
2170+ struct ib_event event;
2171+
2172+ srq->srq_limit = 0;
2173+
2174+ if (!ibsrq->event_handler)
2175+ return;
2176+
2177+ event.device = ibsrq->device;
2178+ event.element.port_num = 1;
2179+ event.element.srq = ibsrq;
2180+ event.event = IB_EVENT_SRQ_LIMIT_REACHED;
2181+ ibsrq->event_handler(&event, ibsrq->srq_context);
2182+}
2183+
2184+/**
2185+ * irdma_modify_srq - modify srq request
2186+ * @ibsrq: srq's pointer for modify
2187+ * @attr: access attributes
2188+ * @attr_mask: state mask
2189+ * @udata: user data
2190+ */
2191+static int irdma_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
2192+ enum ib_srq_attr_mask attr_mask,
2193+ struct ib_udata *udata)
2194+{
2195+ struct irdma_device *iwdev = to_iwdev(ibsrq->device);
2196+ struct irdma_srq *iwsrq = to_iwsrq(ibsrq);
2197+ struct irdma_cqp_request *cqp_request;
2198+ struct irdma_pci_f *rf = iwdev->rf;
2199+ struct irdma_modify_srq_info *info;
2200+ struct cqp_cmds_info *cqp_info;
2201+ int status;
2202+
2203+ if (attr_mask & IB_SRQ_MAX_WR)
2204+ return -EINVAL;
2205+
2206+ if (!(attr_mask & IB_SRQ_LIMIT))
2207+ return 0;
2208+
2209+ if (attr->srq_limit > iwsrq->sc_srq.srq_uk.srq_size)
2210+ return -EINVAL;
2211+
2212+ /* Execute this cqp op synchronously, so we can update srq_limit
2213+ * upon successful completion.
2214+ */
2215+ cqp_request = irdma_alloc_and_get_cqp_request(&rf->cqp, true);
2216+ if (!cqp_request)
2217+ return -ENOMEM;
2218+
2219+ cqp_info = &cqp_request->info;
2220+ info = &cqp_info->in.u.srq_modify.info;
2221+ info->srq_limit = attr->srq_limit;
2222+ if (info->srq_limit > 0xFFF)
2223+ info->srq_limit = 0xFFF;
2224+ info->arm_limit_event = 1;
2225+
2226+ cqp_info->cqp_cmd = IRDMA_OP_SRQ_MODIFY;
2227+ cqp_info->post_sq = 1;
2228+ cqp_info->in.u.srq_modify.srq = &iwsrq->sc_srq;
2229+ cqp_info->in.u.srq_modify.scratch = (uintptr_t)cqp_request;
2230+ status = irdma_handle_cqp_op(rf, cqp_request);
2231+ irdma_put_cqp_request(&rf->cqp, cqp_request);
2232+ if (status)
2233+ return status;
2234+
2235+ iwsrq->sc_srq.srq_limit = info->srq_limit;
2236+
2237+ return 0;
2238+}
2239+
2240+static int irdma_setup_umode_srq(struct irdma_device *iwdev,
2241+ struct irdma_srq *iwsrq,
2242+ struct irdma_srq_init_info *info,
2243+ struct ib_udata *udata)
2244+{
2245+#define IRDMA_CREATE_SRQ_MIN_REQ_LEN \
2246+ offsetofend(struct irdma_create_srq_req, user_shadow_area)
2247+ struct irdma_create_srq_req req = {};
2248+ struct irdma_ucontext *ucontext;
2249+ struct irdma_srq_mr *srqmr;
2250+ struct irdma_pbl *iwpbl;
2251+ unsigned long flags;
2252+
2253+ iwsrq->user_mode = true;
2254+ ucontext = rdma_udata_to_drv_context(udata, struct irdma_ucontext,
2255+ ibucontext);
2256+
2257+ if (udata->inlen < IRDMA_CREATE_SRQ_MIN_REQ_LEN)
2258+ return -EINVAL;
2259+
2260+ if (ib_copy_from_udata(&req, udata,
2261+ min(sizeof(req), udata->inlen)))
2262+ return -EFAULT;
2263+
2264+ spin_lock_irqsave(&ucontext->srq_reg_mem_list_lock, flags);
2265+ iwpbl = irdma_get_pbl((unsigned long)req.user_srq_buf,
2266+ &ucontext->srq_reg_mem_list);
2267+ spin_unlock_irqrestore(&ucontext->srq_reg_mem_list_lock, flags);
2268+ if (!iwpbl)
2269+ return -EPROTO;
2270+
2271+ iwsrq->iwpbl = iwpbl;
2272+ srqmr = &iwpbl->srq_mr;
2273+
2274+ if (iwpbl->pbl_allocated) {
2275+ info->virtual_map = true;
2276+ info->pbl_chunk_size = 1;
2277+ info->first_pm_pbl_idx = srqmr->srq_pbl.idx;
2278+ info->leaf_pbl_size = 1;
2279+ } else {
2280+ info->srq_pa = srqmr->srq_pbl.addr;
2281+ }
2282+ info->shadow_area_pa = srqmr->shadow;
2283+
2284+ return 0;
2285+}
2286+
2287+static int irdma_setup_kmode_srq(struct irdma_device *iwdev,
2288+ struct irdma_srq *iwsrq,
2289+ struct irdma_srq_init_info *info, u32 depth,
2290+ u8 shift)
2291+{
2292+ struct irdma_srq_uk_init_info *ukinfo = &info->srq_uk_init_info;
2293+ struct irdma_dma_mem *mem = &iwsrq->kmem;
2294+ u32 size, ring_size;
2295+
2296+ ring_size = depth * IRDMA_QP_WQE_MIN_SIZE;
2297+ size = ring_size + (IRDMA_SHADOW_AREA_SIZE << 3);
2298+
2299+ mem->size = ALIGN(size, 256);
2300+ mem->va = dma_alloc_coherent(iwdev->rf->hw.device, mem->size,
2301+ &mem->pa, GFP_KERNEL);
2302+ if (!mem->va)
2303+ return -ENOMEM;
2304+
2305+ ukinfo->srq = mem->va;
2306+ ukinfo->srq_size = depth >> shift;
2307+ ukinfo->shadow_area = mem->va + ring_size;
2308+
2309+ info->shadow_area_pa = info->srq_pa + ring_size;
2310+ info->srq_pa = mem->pa;
2311+
2312+ return 0;
2313+}
2314+
2315+/**
2316+ * irdma_create_srq - create srq
2317+ * @ibsrq: ib's srq pointer
2318+ * @initattrs: attributes for srq
2319+ * @udata: user data for create srq
2320+ */
2321+static int irdma_create_srq(struct ib_srq *ibsrq,
2322+ struct ib_srq_init_attr *initattrs,
2323+ struct ib_udata *udata)
2324+{
2325+ struct irdma_device *iwdev = to_iwdev(ibsrq->device);
2326+ struct ib_srq_attr *attr = &initattrs->attr;
2327+ struct irdma_pd *iwpd = to_iwpd(ibsrq->pd);
2328+ struct irdma_srq *iwsrq = to_iwsrq(ibsrq);
2329+ struct irdma_srq_uk_init_info *ukinfo;
2330+ struct irdma_cqp_request *cqp_request;
2331+ struct irdma_srq_init_info info = {};
2332+ struct irdma_pci_f *rf = iwdev->rf;
2333+ struct irdma_uk_attrs *uk_attrs;
2334+ struct cqp_cmds_info *cqp_info;
2335+ int err_code = 0;
2336+ u32 depth;
2337+ u8 shift;
2338+
2339+ uk_attrs = &rf->sc_dev.hw_attrs.uk_attrs;
2340+ ukinfo = &info.srq_uk_init_info;
2341+
2342+ if (initattrs->srq_type != IB_SRQT_BASIC)
2343+ return -EOPNOTSUPP;
2344+
2345+ if (!(uk_attrs->feature_flags & IRDMA_FEATURE_SRQ) ||
2346+ attr->max_sge > uk_attrs->max_hw_wq_frags)
2347+ return -EINVAL;
2348+
2349+ refcount_set(&iwsrq->refcnt, 1);
2350+ spin_lock_init(&iwsrq->lock);
2351+ err_code = irdma_alloc_rsrc(rf, rf->allocated_srqs, rf->max_srq,
2352+ &iwsrq->srq_num, &rf->next_srq);
2353+ if (err_code)
2354+ return err_code;
2355+
2356+ ukinfo->max_srq_frag_cnt = attr->max_sge;
2357+ ukinfo->uk_attrs = uk_attrs;
2358+ ukinfo->srq_id = iwsrq->srq_num;
2359+
2360+ irdma_get_wqe_shift(ukinfo->uk_attrs, ukinfo->max_srq_frag_cnt, 0,
2361+ &shift);
2362+
2363+ err_code = irdma_get_srqdepth(ukinfo->uk_attrs, attr->max_wr,
2364+ shift, &depth);
2365+ if (err_code)
2366+ return err_code;
2367+
2368+ /* Actual SRQ size in WRs for ring and HW */
2369+ ukinfo->srq_size = depth >> shift;
2370+
2371+ /* Max postable WRs to SRQ */
2372+ iwsrq->max_wr = (depth - IRDMA_RQ_RSVD) >> shift;
2373+ attr->max_wr = iwsrq->max_wr;
2374+
2375+ if (udata)
2376+ err_code = irdma_setup_umode_srq(iwdev, iwsrq, &info, udata);
2377+ else
2378+ err_code = irdma_setup_kmode_srq(iwdev, iwsrq, &info, depth,
2379+ shift);
2380+
2381+ if (err_code)
2382+ goto free_rsrc;
2383+
2384+ info.vsi = &iwdev->vsi;
2385+ info.pd = &iwpd->sc_pd;
2386+
2387+ err_code = irdma_sc_srq_init(&iwsrq->sc_srq, &info);
2388+ if (err_code)
2389+ goto free_dmem;
2390+
2391+ cqp_request = irdma_alloc_and_get_cqp_request(&rf->cqp, true);
2392+ if (!cqp_request) {
2393+ err_code = -ENOMEM;
2394+ goto free_dmem;
2395+ }
2396+
2397+ cqp_info = &cqp_request->info;
2398+ cqp_info->cqp_cmd = IRDMA_OP_SRQ_CREATE;
2399+ cqp_info->post_sq = 1;
2400+ cqp_info->in.u.srq_create.srq = &iwsrq->sc_srq;
2401+ cqp_info->in.u.srq_create.scratch = (uintptr_t)cqp_request;
2402+ err_code = irdma_handle_cqp_op(rf, cqp_request);
2403+ irdma_put_cqp_request(&rf->cqp, cqp_request);
2404+ if (err_code)
2405+ goto free_dmem;
2406+
2407+ if (udata) {
2408+ struct irdma_create_srq_resp resp = {};
2409+
2410+ resp.srq_id = iwsrq->srq_num;
2411+ resp.srq_size = ukinfo->srq_size;
2412+ if (ib_copy_to_udata(udata, &resp,
2413+ min(sizeof(resp), udata->outlen))) {
2414+ err_code = -EPROTO;
2415+ goto srq_destroy;
2416+ }
2417+ }
2418+
2419+ return 0;
2420+
2421+srq_destroy:
2422+ irdma_srq_wq_destroy(rf, &iwsrq->sc_srq);
2423+
2424+free_dmem:
2425+ if (!iwsrq->user_mode)
2426+ dma_free_coherent(rf->hw.device, iwsrq->kmem.size,
2427+ iwsrq->kmem.va, iwsrq->kmem.pa);
2428+free_rsrc:
2429+ irdma_free_rsrc(rf, rf->allocated_srqs, iwsrq->srq_num);
2430+ return err_code;
2431+}
2432+
2433+/**
2434+ * irdma_query_srq - get SRQ attributes
2435+ * @ibsrq: the SRQ to query
2436+ * @attr: the attributes of the SRQ
2437+ */
2438+static int irdma_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr)
2439+{
2440+ struct irdma_srq *iwsrq = to_iwsrq(ibsrq);
2441+
2442+ attr->max_wr = iwsrq->max_wr;
2443+ attr->max_sge = iwsrq->sc_srq.srq_uk.max_srq_frag_cnt;
2444+ attr->srq_limit = iwsrq->sc_srq.srq_limit;
2445+
2446+ return 0;
2447+}
2448+
20252449 static inline int cq_validate_flags(u32 flags, u8 hw_rev)
20262450 {
2027− /* GEN1 does not support CQ create flags */
2028− if (hw_rev == IRDMA_GEN_1)
2451+ /* GEN1/2 does not support CQ create flags */
2452+ if (hw_rev <= IRDMA_GEN_2)
20292453 return flags ? -EOPNOTSUPP : 0;
20302454
20312455 return flags & ~IB_UVERBS_CQ_FLAGS_TIMESTAMP_COMPLETION ? -EOPNOTSUPP : 0;
@@ -2058,6 +2482,7 @@ static int irdma_create_cq(struct ib_cq *ibcq, · 26 unmodified lines
20582482 unsigned long flags;
20592483 int err_code;
20602484 int entries = attr->cqe;
2485+ bool cqe_64byte_ena;
20612486
20622487 err_code = cq_validate_flags(attr->flags, dev->hw_attrs.uk_attrs.hw_rev);
20632488 if (err_code)
@@ -2081,6 +2506,9 @@ static int irdma_create_cq(struct ib_cq *ibcq, · 17 unmodified lines
20812506 info.dev = dev;
20822507 ukinfo->cq_size = max(entries, 4);
20832508 ukinfo->cq_id = cq_num;
2509+ cqe_64byte_ena = dev->hw_attrs.uk_attrs.feature_flags & IRDMA_FEATURE_64_BYTE_CQE ?
2510+ true : false;
2511+ ukinfo->avoid_mem_cflct = cqe_64byte_ena;
20842512 iwcq->ibcq.cqe = info.cq_uk_init_info.cq_size;
20852513 if (attr->comp_vector < rf->ceqs_count)
20862514 info.ceq_id = attr->comp_vector;
@@ -2116,8 +2544,6 @@ static int irdma_create_cq(struct ib_cq *ibcq, · 29 unmodified lines
21162544 goto cq_free_rsrc;
21172545 }
21182546
2119− iwcq->iwpbl = iwpbl;
2120− iwcq->cq_mem_size = 0;
21212547 cqmr = &iwpbl->cq_mr;
21222548
21232549 if (rf->sc_dev.hw_attrs.uk_attrs.feature_flags &
@@ -2132,7 +2558,6 @@ static int irdma_create_cq(struct ib_cq *ibcq, · 8 unmodified lines
21322558 err_code = -EPROTO;
21332559 goto cq_free_rsrc;
21342560 }
2135− iwcq->iwpbl_shadow = iwpbl_shadow;
21362561 cqmr_shadow = &iwpbl_shadow->cq_mr;
21372562 info.shadow_area_pa = cqmr_shadow->cq_pbl.addr;
21382563 cqmr->split = true;
@@ -2156,11 +2581,18 @@ static int irdma_create_cq(struct ib_cq *ibcq, · 17 unmodified lines
21562581 }
21572582
21582583 entries++;
2159− if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_2)
2584+ if (!cqe_64byte_ena && dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_2)
21602585 entries *= 2;
2586+
2587+ if (entries & 1)
2588+ entries += 1; /* cq size must be an even number */
2589+
21612590 ukinfo->cq_size = entries;
21622591
2163− rsize = info.cq_uk_init_info.cq_size * sizeof(struct irdma_cqe);
2592+ if (cqe_64byte_ena)
2593+ rsize = info.cq_uk_init_info.cq_size * sizeof(struct irdma_extended_cqe);
2594+ else
2595+ rsize = info.cq_uk_init_info.cq_size * sizeof(struct irdma_cqe);
21642596 iwcq->kmem.size = ALIGN(round_up(rsize, 256), 256);
21652597 iwcq->kmem.va = dma_alloc_coherent(dev->hw->device,
21662598 iwcq->kmem.size,
@@ -2240,8 +2672,9 @@ cq_free_rsrc: · 73 unmodified lines
22402672 /**
22412673 * irdma_get_mr_access - get hw MR access permissions from IB access flags
22422674 * @access: IB access flags
2675+ * @hw_rev: Hardware version
22432676 */
2244−static inline u16 irdma_get_mr_access(int access)
2677+static inline u16 irdma_get_mr_access(int access, u8 hw_rev)
22452678 {
22462679 u16 hw_access = 0;
22472680
@@ -2251,8 +2684,10 @@ static inline u16 irdma_get_mr_access(int access) · 3 unmodified lines
22512684 IRDMA_ACCESS_FLAGS_REMOTEWRITE : 0;
22522685 hw_access |= (access & IB_ACCESS_REMOTE_READ) ?
22532686 IRDMA_ACCESS_FLAGS_REMOTEREAD : 0;
2254− hw_access |= (access & IB_ACCESS_MW_BIND) ?
2255− IRDMA_ACCESS_FLAGS_BIND_WINDOW : 0;
2687+ if (hw_rev >= IRDMA_GEN_3) {
2688+ hw_access |= (access & IB_ACCESS_MW_BIND) ?
2689+ IRDMA_ACCESS_FLAGS_BIND_WINDOW : 0;
2690+ }
22562691 hw_access |= (access & IB_ZERO_BASED) ?
22572692 IRDMA_ACCESS_FLAGS_ZERO_BASED : 0;
22582693 hw_access |= IRDMA_ACCESS_FLAGS_LOCALREAD;
@@ -2463,6 +2898,7 @@ static int irdma_handle_q_mem(struct irdma_device *iwdev, · 204 unmodified lines
24632898 struct irdma_mr *iwmr = iwpbl->iwmr;
24642899 struct irdma_qp_mr *qpmr = &iwpbl->qp_mr;
24652900 struct irdma_cq_mr *cqmr = &iwpbl->cq_mr;
2901+ struct irdma_srq_mr *srqmr = &iwpbl->srq_mr;
24662902 struct irdma_hmc_pble *hmc_p;
24672903 u64 *arr = iwmr->pgaddrmem;
24682904 u32 pg_size, total;
@@ -2482,7 +2918,10 @@ static int irdma_handle_q_mem(struct irdma_device *iwdev, · 13 unmodified lines
24822918 total = req->sq_pages + req->rq_pages;
24832919 hmc_p = &qpmr->sq_pbl;
24842920 qpmr->shadow = (dma_addr_t)arr[total];
2485−
2921+ /* Need to use physical address for RQ of QP
2922+ * in case it is associated with SRQ.
2923+ */
2924+ qpmr->rq_pa = (dma_addr_t)arr[req->sq_pages];
24862925 if (lvl) {
24872926 ret = irdma_check_mem_contiguous(arr, req->sq_pages,
24882927 pg_size);
@@ -2502,6 +2941,18 @@ static int irdma_handle_q_mem(struct irdma_device *iwdev, · 13 unmodified lines
25022941 hmc_p->addr = arr[req->sq_pages];
25032942 }
25042943 break;
2944+ case IRDMA_MEMREG_TYPE_SRQ:
2945+ hmc_p = &srqmr->srq_pbl;
2946+ srqmr->shadow = (dma_addr_t)arr[req->rq_pages];
2947+ if (lvl)
2948+ ret = irdma_check_mem_contiguous(arr, req->rq_pages,
2949+ pg_size);
2950+
2951+ if (!ret)
2952+ hmc_p->idx = palloc->level1.idx;
2953+ else
2954+ hmc_p->addr = arr[0];
2955+ break;
25052956 case IRDMA_MEMREG_TYPE_CQ:
25062957 hmc_p = &cqmr->cq_pbl;
25072958
@@ -2806,7 +3257,10 @@ static int irdma_hwreg_mr(struct irdma_device *iwdev, struct irdma_mr *iwmr, · 298 unmodified lines
28063257 stag_info->stag_idx = iwmr->stag >> IRDMA_CQPSQ_STAG_IDX_S;
28073258 stag_info->stag_key = (u8)iwmr->stag;
28083259 stag_info->total_len = iwmr->len;
2809− stag_info->access_rights = irdma_get_mr_access(access);
3260+ stag_info->access_rights = irdma_get_mr_access(access,
3261+ iwdev->rf->sc_dev.hw_attrs.uk_attrs.hw_rev);
3262+ if (iwdev->rf->sc_dev.hw_attrs.uk_attrs.feature_flags & IRDMA_FEATURE_ATOMIC_OPS)
3263+ stag_info->remote_atomics_en = (access & IB_ACCESS_REMOTE_ATOMIC) ? 1 : 0;
28103264 stag_info->pd_id = iwpd->sc_pd.pd_id;
28113265 stag_info->all_memory = pd->flags & IB_PD_UNSAFE_GLOBAL_RKEY;
28123266 if (stag_info->access_rights & IRDMA_ACCESS_FLAGS_ZERO_BASED)
@@ -2972,6 +3426,37 @@ static int irdma_reg_user_mr_type_qp(struct irdma_mem_reg_req req, · 159 unmodified lines
29723426 return 0;
29733427 }
29743428
3429+static int irdma_reg_user_mr_type_srq(struct irdma_mem_reg_req req,
3430+ struct ib_udata *udata,
3431+ struct irdma_mr *iwmr)
3432+{
3433+ struct irdma_device *iwdev = to_iwdev(iwmr->ibmr.device);
3434+ struct irdma_pbl *iwpbl = &iwmr->iwpbl;
3435+ struct irdma_ucontext *ucontext;
3436+ unsigned long flags;
3437+ u32 total;
3438+ int err;
3439+ u8 lvl;
3440+
3441+ total = req.rq_pages + IRDMA_SHADOW_PGCNT;
3442+ if (total > iwmr->page_cnt)
3443+ return -EINVAL;
3444+
3445+ lvl = req.rq_pages > 1 ? PBLE_LEVEL_1 : PBLE_LEVEL_0;
3446+ err = irdma_handle_q_mem(iwdev, &req, iwpbl, lvl);
3447+ if (err)
3448+ return err;
3449+
3450+ ucontext = rdma_udata_to_drv_context(udata, struct irdma_ucontext,
3451+ ibucontext);
3452+ spin_lock_irqsave(&ucontext->srq_reg_mem_list_lock, flags);
3453+ list_add_tail(&iwpbl->list, &ucontext->srq_reg_mem_list);
3454+ iwpbl->on_list = true;
3455+ spin_unlock_irqrestore(&ucontext->srq_reg_mem_list_lock, flags);
3456+
3457+ return 0;
3458+}
3459+
29753460 static int irdma_reg_user_mr_type_cq(struct irdma_mem_reg_req req,
29763461 struct ib_udata *udata,
29773462 struct irdma_mr *iwmr)
@@ -3062,6 +3547,12 @@ static struct ib_mr *irdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 len, · 84 unmodified lines
30623547 if (err)
30633548 goto error;
30643549
3550+ break;
3551+ case IRDMA_MEMREG_TYPE_SRQ:
3552+ err = irdma_reg_user_mr_type_srq(req, udata, iwmr);
3553+ if (err)
3554+ goto error;
3555+
30653556 break;
30663557 case IRDMA_MEMREG_TYPE_CQ:
30673558 err = irdma_reg_user_mr_type_cq(req, udata, iwmr);
@@ -3106,9 +3597,9 @@ static struct ib_mr *irdma_reg_user_mr_dmabuf(struct ib_pd *pd, u64 start, · 38 unmodified lines
31063597
31073598 umem_dmabuf = ib_umem_dmabuf_get_pinned(pd->device, start, len, fd, access);
31083599 if (IS_ERR(umem_dmabuf)) {
3109− err = PTR_ERR(umem_dmabuf);
3110− ibdev_dbg(&iwdev->ibdev, "Failed to get dmabuf umem[%d]\n", err);
3111− return ERR_PTR(err);
3600+ ibdev_dbg(&iwdev->ibdev, "Failed to get dmabuf umem[%pe]\n",
3601+ umem_dmabuf);
3602+ return ERR_CAST(umem_dmabuf);
31123603 }
31133604
31143605 iwmr = irdma_alloc_iwmr(&umem_dmabuf->umem, pd, virt, IRDMA_MEMREG_TYPE_MEM);
@@ -3382,6 +3873,14 @@ static void irdma_del_memlist(struct irdma_mr *iwmr, · 267 unmodified lines
33823873 }
33833874 spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags);
33843875 break;
3876+ case IRDMA_MEMREG_TYPE_SRQ:
3877+ spin_lock_irqsave(&ucontext->srq_reg_mem_list_lock, flags);
3878+ if (iwpbl->on_list) {
3879+ iwpbl->on_list = false;
3880+ list_del(&iwpbl->list);
3881+ }
3882+ spin_unlock_irqrestore(&ucontext->srq_reg_mem_list_lock, flags);
3883+ break;
33853884 default:
33863885 break;
33873886 }
@@ -3461,6 +3960,40 @@ static int irdma_post_send(struct ib_qp *ibqp, · 73 unmodified lines
34613960 if (ib_wr->send_flags & IB_SEND_FENCE)
34623961 info.read_fence = true;
34633962 switch (ib_wr->opcode) {
3963+ case IB_WR_ATOMIC_CMP_AND_SWP:
3964+ if (unlikely(!(dev->hw_attrs.uk_attrs.feature_flags &
3965+ IRDMA_FEATURE_ATOMIC_OPS))) {
3966+ err = -EINVAL;
3967+ break;
3968+ }
3969+ info.op_type = IRDMA_OP_TYPE_ATOMIC_COMPARE_AND_SWAP;
3970+ info.op.atomic_compare_swap.tagged_offset = ib_wr->sg_list[0].addr;
3971+ info.op.atomic_compare_swap.remote_tagged_offset =
3972+ atomic_wr(ib_wr)->remote_addr;
3973+ info.op.atomic_compare_swap.swap_data_bytes = atomic_wr(ib_wr)->swap;
3974+ info.op.atomic_compare_swap.compare_data_bytes =
3975+ atomic_wr(ib_wr)->compare_add;
3976+ info.op.atomic_compare_swap.stag = ib_wr->sg_list[0].lkey;
3977+ info.op.atomic_compare_swap.remote_stag = atomic_wr(ib_wr)->rkey;
3978+ err = irdma_uk_atomic_compare_swap(ukqp, &info, false);
3979+ break;
3980+ case IB_WR_ATOMIC_FETCH_AND_ADD:
3981+ if (unlikely(!(dev->hw_attrs.uk_attrs.feature_flags &
3982+ IRDMA_FEATURE_ATOMIC_OPS))) {
3983+ err = -EINVAL;
3984+ break;
3985+ }
3986+ info.op_type = IRDMA_OP_TYPE_ATOMIC_FETCH_AND_ADD;
3987+ info.op.atomic_fetch_add.tagged_offset = ib_wr->sg_list[0].addr;
3988+ info.op.atomic_fetch_add.remote_tagged_offset =
3989+ atomic_wr(ib_wr)->remote_addr;
3990+ info.op.atomic_fetch_add.fetch_add_data_bytes =
3991+ atomic_wr(ib_wr)->compare_add;
3992+ info.op.atomic_fetch_add.stag = ib_wr->sg_list[0].lkey;
3993+ info.op.atomic_fetch_add.remote_stag =
3994+ atomic_wr(ib_wr)->rkey;
3995+ err = irdma_uk_atomic_fetch_add(ukqp, &info, false);
3996+ break;
34643997 case IB_WR_SEND_WITH_IMM:
34653998 if (ukqp->qp_caps & IRDMA_SEND_WITH_IMM) {
34663999 info.imm_data_valid = true;
@@ -3555,7 +4088,9 @@ static int irdma_post_send(struct ib_qp *ibqp, · 88 unmodified lines
35554088
35564089 stag_info.signaled = info.signaled;
35574090 stag_info.read_fence = info.read_fence;
3558− stag_info.access_rights = irdma_get_mr_access(reg_wr(ib_wr)->access);
4091+ stag_info.access_rights =
4092+ irdma_get_mr_access(reg_wr(ib_wr)->access,
4093+ dev->hw_attrs.uk_attrs.hw_rev);
35594094 stag_info.stag_key = reg_wr(ib_wr)->key & 0xff;
35604095 stag_info.stag_idx = reg_wr(ib_wr)->key >> 8;
35614096 stag_info.page_size = reg_wr(ib_wr)->mr->page_size;
@@ -3594,6 +4129,48 @@ static int irdma_post_send(struct ib_qp *ibqp, · 32 unmodified lines
35944129 mod_delayed_work(iwqp->iwdev->cleanup_wq, &iwqp->dwork_flush,
35954130 msecs_to_jiffies(IRDMA_FLUSH_DELAY_MS));
35964131 }
4132+
4133+ if (err)
4134+ *bad_wr = ib_wr;
4135+
4136+ return err;
4137+}
4138+
4139+/**
4140+ * irdma_post_srq_recv - post receive wr for kernel application
4141+ * @ibsrq: ib srq pointer
4142+ * @ib_wr: work request for receive
4143+ * @bad_wr: bad wr caused an error
4144+ */
4145+static int irdma_post_srq_recv(struct ib_srq *ibsrq,
4146+ const struct ib_recv_wr *ib_wr,
4147+ const struct ib_recv_wr **bad_wr)
4148+{
4149+ struct irdma_srq *iwsrq = to_iwsrq(ibsrq);
4150+ struct irdma_srq_uk *uksrq = &iwsrq->sc_srq.srq_uk;
4151+ struct irdma_post_rq_info post_recv = {};
4152+ unsigned long flags;
4153+ int err = 0;
4154+
4155+ spin_lock_irqsave(&iwsrq->lock, flags);
4156+ while (ib_wr) {
4157+ if (ib_wr->num_sge > uksrq->max_srq_frag_cnt) {
4158+ err = -EINVAL;
4159+ goto out;
4160+ }
4161+ post_recv.num_sges = ib_wr->num_sge;
4162+ post_recv.wr_id = ib_wr->wr_id;
4163+ post_recv.sg_list = ib_wr->sg_list;
4164+ err = irdma_uk_srq_post_receive(uksrq, &post_recv);
4165+ if (err)
4166+ goto out;
4167+
4168+ ib_wr = ib_wr->next;
4169+ }
4170+
4171+out:
4172+ spin_unlock_irqrestore(&iwsrq->lock, flags);
4173+
35974174 if (err)
35984175 *bad_wr = ib_wr;
35994176
@@ -3619,6 +4196,11 @@ static int irdma_post_recv(struct ib_qp *ibqp, · 19 unmodified lines
36194196 iwqp = to_iwqp(ibqp);
36204197 ukqp = &iwqp->sc_qp.qp_uk;
36214198
4199+ if (ukqp->srq_uk) {
4200+ *bad_wr = ib_wr;
4201+ return -EINVAL;
4202+ }
4203+
36224204 spin_lock_irqsave(&iwqp->lock, flags);
36234205 while (ib_wr) {
36244206 post_recv.num_sges = ib_wr->num_sge;
@@ -3671,6 +4253,8 @@ static enum ib_wc_status irdma_flush_err_to_ib_wc_status(enum irdma_flush_opcode · 46 unmodified lines
36714253 return IB_WC_MW_BIND_ERR;
36724254 case FLUSH_REM_INV_REQ_ERR:
36734255 return IB_WC_REM_INV_REQ_ERR;
4256+ case FLUSH_RNR_RETRY_EXC_ERR:
4257+ return IB_WC_RNR_RETRY_EXC_ERR;
36744258 case FLUSH_FATAL_ERR:
36754259 default:
36764260 return IB_WC_FATAL_ERR;
@@ -3727,8 +4311,12 @@ static void irdma_process_cqe(struct ib_wc *entry, · 50 unmodified lines
37274311 if (cq_poll_info->q_type == IRDMA_CQE_QTYPE_SQ) {
37284312 set_ib_wc_op_sq(cq_poll_info, entry);
37294313 } else {
3730− set_ib_wc_op_rq(cq_poll_info, entry,
3731− qp->qp_uk.qp_caps & IRDMA_SEND_WITH_IMM);
4314+ if (qp->dev->hw_attrs.uk_attrs.hw_rev <= IRDMA_GEN_2)
4315+ set_ib_wc_op_rq(cq_poll_info, entry,
4316+ qp->qp_uk.qp_caps & IRDMA_SEND_WITH_IMM ?
4317+ true : false);
4318+ else
4319+ set_ib_wc_op_rq_gen_3(cq_poll_info, entry);
37324320 if (qp->qp_uk.qp_type != IRDMA_QP_TYPE_ROCE_UD &&
37334321 cq_poll_info->stag_invalid_set) {
37344322 entry->ex.invalidate_rkey = cq_poll_info->inv_stag;
@@ -3923,40 +4511,7 @@ static int irdma_req_notify_cq(struct ib_cq *ibcq, · 188 unmodified lines
39234511 return ret;
39244512 }
39254513
3926−static int irdma_roce_port_immutable(struct ib_device *ibdev, u32 port_num,
3927− struct ib_port_immutable *immutable)
3928−{
3929− struct ib_port_attr attr;
3930− int err;
3931−
3932− immutable->core_cap_flags = RDMA_CORE_PORT_IBA_ROCE_UDP_ENCAP;
3933− err = ib_query_port(ibdev, port_num, &attr);
3934− if (err)
3935− return err;
3936−
3937− immutable->max_mad_size = IB_MGMT_MAD_SIZE;
3938− immutable->pkey_tbl_len = attr.pkey_tbl_len;
3939− immutable->gid_tbl_len = attr.gid_tbl_len;
3940−
3941− return 0;
3942−}
3943−
3944−static int irdma_iw_port_immutable(struct ib_device *ibdev, u32 port_num,
3945− struct ib_port_immutable *immutable)
3946−{
3947− struct ib_port_attr attr;
3948− int err;
3949−
3950− immutable->core_cap_flags = RDMA_CORE_PORT_IWARP;
3951− err = ib_query_port(ibdev, port_num, &attr);
3952− if (err)
3953− return err;
3954− immutable->gid_tbl_len = attr.gid_tbl_len;
3955−
3956− return 0;
3957−}
3958−
3959−static const struct rdma_stat_desc irdma_hw_stat_names[] = {
4514+static const struct rdma_stat_desc irdma_hw_stat_descs[] = {
39604515 /* gen1 - 32-bit */
39614516 [IRDMA_HW_STAT_INDEX_IP4RXDISCARD].name = "ip4InDiscards",
39624517 [IRDMA_HW_STAT_INDEX_IP4RXTRUNC].name = "ip4InTruncatedPkts",
@@ -3964,9 +4519,6 @@ static const struct rdma_stat_desc irdma_hw_stat_names[] = { · 1 unmodified line
39644519 [IRDMA_HW_STAT_INDEX_IP6RXDISCARD].name = "ip6InDiscards",
39654520 [IRDMA_HW_STAT_INDEX_IP6RXTRUNC].name = "ip6InTruncatedPkts",
39664521 [IRDMA_HW_STAT_INDEX_IP6TXNOROUTE].name = "ip6OutNoRoutes",
3967− [IRDMA_HW_STAT_INDEX_TCPRTXSEG].name = "tcpRetransSegs",
3968− [IRDMA_HW_STAT_INDEX_TCPRXOPTERR].name = "tcpInOptErrors",
3969− [IRDMA_HW_STAT_INDEX_TCPRXPROTOERR].name = "tcpInProtoErrors",
39704522 [IRDMA_HW_STAT_INDEX_RXVLANERR].name = "rxVlanErrors",
39714523 /* gen1 - 64-bit */
39724524 [IRDMA_HW_STAT_INDEX_IP4RXOCTS].name = "ip4InOctets",
@@ -3985,16 +4537,14 @@ static const struct rdma_stat_desc irdma_hw_stat_names[] = { · 12 unmodified lines
39854537 [IRDMA_HW_STAT_INDEX_IP6TXPKTS].name = "ip6OutPkts",
39864538 [IRDMA_HW_STAT_INDEX_IP6TXFRAGS].name = "ip6OutSegRqd",
39874539 [IRDMA_HW_STAT_INDEX_IP6TXMCPKTS].name = "ip6OutMcastPkts",
3988− [IRDMA_HW_STAT_INDEX_TCPRXSEGS].name = "tcpInSegs",
3989− [IRDMA_HW_STAT_INDEX_TCPTXSEG].name = "tcpOutSegs",
3990− [IRDMA_HW_STAT_INDEX_RDMARXRDS].name = "iwInRdmaReads",
3991− [IRDMA_HW_STAT_INDEX_RDMARXSNDS].name = "iwInRdmaSends",
3992− [IRDMA_HW_STAT_INDEX_RDMARXWRS].name = "iwInRdmaWrites",
3993− [IRDMA_HW_STAT_INDEX_RDMATXRDS].name = "iwOutRdmaReads",
3994− [IRDMA_HW_STAT_INDEX_RDMATXSNDS].name = "iwOutRdmaSends",
3995− [IRDMA_HW_STAT_INDEX_RDMATXWRS].name = "iwOutRdmaWrites",
3996− [IRDMA_HW_STAT_INDEX_RDMAVBND].name = "iwRdmaBnd",
3997− [IRDMA_HW_STAT_INDEX_RDMAVINV].name = "iwRdmaInv",
4540+ [IRDMA_HW_STAT_INDEX_RDMARXRDS].name = "InRdmaReads",
4541+ [IRDMA_HW_STAT_INDEX_RDMARXSNDS].name = "InRdmaSends",
4542+ [IRDMA_HW_STAT_INDEX_RDMARXWRS].name = "InRdmaWrites",
4543+ [IRDMA_HW_STAT_INDEX_RDMATXRDS].name = "OutRdmaReads",
4544+ [IRDMA_HW_STAT_INDEX_RDMATXSNDS].name = "OutRdmaSends",
4545+ [IRDMA_HW_STAT_INDEX_RDMATXWRS].name = "OutRdmaWrites",
4546+ [IRDMA_HW_STAT_INDEX_RDMAVBND].name = "RdmaBnd",
4547+ [IRDMA_HW_STAT_INDEX_RDMAVINV].name = "RdmaInv",
39984548
39994549 /* gen2 - 32-bit */
40004550 [IRDMA_HW_STAT_INDEX_RXRPCNPHANDLED].name = "cnpHandled",
@@ -4008,9 +4558,59 @@ static const struct rdma_stat_desc irdma_hw_stat_names[] = { · 7 unmodified lines
40084558 [IRDMA_HW_STAT_INDEX_UDPRXPKTS].name = "RxUDP",
40094559 [IRDMA_HW_STAT_INDEX_UDPTXPKTS].name = "TxUDP",
40104560 [IRDMA_HW_STAT_INDEX_RXNPECNMARKEDPKTS].name = "RxECNMrkd",
4011−
4561+ [IRDMA_HW_STAT_INDEX_TCPRTXSEG].name = "RetransSegs",
4562+ [IRDMA_HW_STAT_INDEX_TCPRXOPTERR].name = "InOptErrors",
4563+ [IRDMA_HW_STAT_INDEX_TCPRXPROTOERR].name = "InProtoErrors",
4564+ [IRDMA_HW_STAT_INDEX_TCPRXSEGS].name = "InSegs",
4565+ [IRDMA_HW_STAT_INDEX_TCPTXSEG].name = "OutSegs",
4566+
4567+ /* gen3 */
4568+ [IRDMA_HW_STAT_INDEX_RNR_SENT].name = "RNR sent",
4569+ [IRDMA_HW_STAT_INDEX_RNR_RCVD].name = "RNR received",
4570+ [IRDMA_HW_STAT_INDEX_RDMAORDLMTCNT].name = "ord limit count",
4571+ [IRDMA_HW_STAT_INDEX_RDMAIRDLMTCNT].name = "ird limit count",
4572+ [IRDMA_HW_STAT_INDEX_RDMARXATS].name = "Rx atomics",
4573+ [IRDMA_HW_STAT_INDEX_RDMATXATS].name = "Tx atomics",
4574+ [IRDMA_HW_STAT_INDEX_NAKSEQERR].name = "Nak Sequence Error",
4575+ [IRDMA_HW_STAT_INDEX_NAKSEQERR_IMPLIED].name = "Nak Sequence Error Implied",
4576+ [IRDMA_HW_STAT_INDEX_RTO].name = "RTO",
4577+ [IRDMA_HW_STAT_INDEX_RXOOOPKTS].name = "Rcvd Out of order packets",
4578+ [IRDMA_HW_STAT_INDEX_ICRCERR].name = "CRC errors",
40124579 };
40134580
4581+static int irdma_roce_port_immutable(struct ib_device *ibdev, u32 port_num,
4582+ struct ib_port_immutable *immutable)
4583+{
4584+ struct ib_port_attr attr;
4585+ int err;
4586+
4587+ immutable->core_cap_flags = RDMA_CORE_PORT_IBA_ROCE_UDP_ENCAP;
4588+ err = ib_query_port(ibdev, port_num, &attr);
4589+ if (err)
4590+ return err;
4591+
4592+ immutable->max_mad_size = IB_MGMT_MAD_SIZE;
4593+ immutable->pkey_tbl_len = attr.pkey_tbl_len;
4594+ immutable->gid_tbl_len = attr.gid_tbl_len;
4595+
4596+ return 0;
4597+}
4598+
4599+static int irdma_iw_port_immutable(struct ib_device *ibdev, u32 port_num,
4600+ struct ib_port_immutable *immutable)
4601+{
4602+ struct ib_port_attr attr;
4603+ int err;
4604+
4605+ immutable->core_cap_flags = RDMA_CORE_PORT_IWARP;
4606+ err = ib_query_port(ibdev, port_num, &attr);
4607+ if (err)
4608+ return err;
4609+ immutable->gid_tbl_len = attr.gid_tbl_len;
4610+
4611+ return 0;
4612+}
4613+
40144614 static void irdma_get_dev_fw_str(struct ib_device *dev, char *str)
40154615 {
40164616 struct irdma_device *iwdev = to_iwdev(dev);
@@ -4034,7 +4634,7 @@ static struct rdma_hw_stats *irdma_alloc_hw_port_stats(struct ib_device *ibdev, · 17 unmodified lines
40344634 int num_counters = dev->hw_attrs.max_stat_idx;
40354635 unsigned long lifespan = RDMA_HW_STATS_DEFAULT_LIFESPAN;
40364636
4037− return rdma_alloc_hw_stats_struct(irdma_hw_stat_names, num_counters,
4637+ return rdma_alloc_hw_stats_struct(irdma_hw_stat_descs, num_counters,
40384638 lifespan);
40394639 }
40404640
@@ -4539,7 +5139,7 @@ static bool irdma_ah_exists(struct irdma_device *iwdev, · 498 unmodified lines
45395139 new_ah->sc_ah.ah_info.dest_ip_addr[2] ^
45405140 new_ah->sc_ah.ah_info.dest_ip_addr[3];
45415141
4542− hash_for_each_possible(iwdev->ah_hash_tbl, ah, list, key) {
5142+ hash_for_each_possible(iwdev->rf->ah_hash_tbl, ah, list, key) {
45435143 /* Set ah_valid and ah_id the same so memcmp can work */
45445144 new_ah->sc_ah.ah_info.ah_idx = ah->sc_ah.ah_info.ah_idx;
45455145 new_ah->sc_ah.ah_info.ah_valid = ah->sc_ah.ah_info.ah_valid;
@@ -4565,14 +5165,14 @@ static int irdma_destroy_ah(struct ib_ah *ibah, u32 ah_flags) · 19 unmodified lines
45655165 struct irdma_ah *ah = to_iwah(ibah);
45665166
45675167 if ((ah_flags & RDMA_DESTROY_AH_SLEEPABLE) && ah->parent_ah) {
4568− mutex_lock(&iwdev->ah_tbl_lock);
5168+ mutex_lock(&iwdev->rf->ah_tbl_lock);
45695169 if (!refcount_dec_and_test(&ah->parent_ah->refcnt)) {
4570− mutex_unlock(&iwdev->ah_tbl_lock);
5170+ mutex_unlock(&iwdev->rf->ah_tbl_lock);
45715171 return 0;
45725172 }
45735173 hash_del(&ah->parent_ah->list);
45745174 kfree(ah->parent_ah);
4575− mutex_unlock(&iwdev->ah_tbl_lock);
5175+ mutex_unlock(&iwdev->rf->ah_tbl_lock);
45765176 }
45775177
45785178 irdma_ah_cqp_op(iwdev->rf, &ah->sc_ah, IRDMA_OP_AH_DESTROY,
@@ -4609,11 +5209,11 @@ static int irdma_create_user_ah(struct ib_ah *ibah, · 30 unmodified lines
46095209 err = irdma_setup_ah(ibah, attr);
46105210 if (err)
46115211 return err;
4612− mutex_lock(&iwdev->ah_tbl_lock);
5212+ mutex_lock(&iwdev->rf->ah_tbl_lock);
46135213 if (!irdma_ah_exists(iwdev, ah)) {
46145214 err = irdma_create_hw_ah(iwdev, ah, true);
46155215 if (err) {
4616− mutex_unlock(&iwdev->ah_tbl_lock);
5216+ mutex_unlock(&iwdev->rf->ah_tbl_lock);
46175217 return err;
46185218 }
46195219 /* Add new AH to list */
@@ -4625,11 +5225,11 @@ static int irdma_create_user_ah(struct ib_ah *ibah, · 5 unmodified lines
46255225 parent_ah->sc_ah.ah_info.dest_ip_addr[3];
46265226
46275227 ah->parent_ah = parent_ah;
4628− hash_add(iwdev->ah_hash_tbl, &parent_ah->list, key);
5228+ hash_add(iwdev->rf->ah_hash_tbl, &parent_ah->list, key);
46295229 refcount_set(&parent_ah->refcnt, 1);
46305230 }
46315231 }
4632− mutex_unlock(&iwdev->ah_tbl_lock);
5232+ mutex_unlock(&iwdev->rf->ah_tbl_lock);
46335233
46345234 uresp.ah_id = ah->sc_ah.ah_info.ah_idx;
46355235 err = ib_copy_to_udata(udata, &uresp, min(sizeof(uresp), udata->outlen));
@@ -4691,6 +5291,20 @@ static enum rdma_link_layer irdma_get_link_layer(struct ib_device *ibdev, · 55 unmodified lines
46915291 return IB_LINK_LAYER_ETHERNET;
46925292 }
46935293
5294+static const struct ib_device_ops irdma_gen1_dev_ops = {
5295+ .dealloc_driver = irdma_ib_dealloc_device,
5296+};
5297+
5298+static const struct ib_device_ops irdma_gen3_dev_ops = {
5299+ .alloc_mw = irdma_alloc_mw,
5300+ .create_srq = irdma_create_srq,
5301+ .dealloc_mw = irdma_dealloc_mw,
5302+ .destroy_srq = irdma_destroy_srq,
5303+ .modify_srq = irdma_modify_srq,
5304+ .post_srq_recv = irdma_post_srq_recv,
5305+ .query_srq = irdma_query_srq,
5306+};
5307+
46945308 static const struct ib_device_ops irdma_roce_dev_ops = {
46955309 .attach_mcast = irdma_attach_mcast,
46965310 .create_ah = irdma_create_ah,
@@ -4725,7 +5339,6 @@ static const struct ib_device_ops irdma_dev_ops = { · 28 unmodified lines
47255339
47265340 .alloc_hw_port_stats = irdma_alloc_hw_port_stats,
47275341 .alloc_mr = irdma_alloc_mr,
4728− .alloc_mw = irdma_alloc_mw,
47295342 .alloc_pd = irdma_alloc_pd,
47305343 .alloc_ucontext = irdma_alloc_ucontext,
47315344 .create_cq = irdma_create_cq,
@@ -4761,6 +5374,7 @@ static const struct ib_device_ops irdma_dev_ops = { · 29 unmodified lines
47615374 INIT_RDMA_OBJ_SIZE(ib_cq, irdma_cq, ibcq),
47625375 INIT_RDMA_OBJ_SIZE(ib_mw, irdma_mr, ibmw),
47635376 INIT_RDMA_OBJ_SIZE(ib_qp, irdma_qp, ibqp),
5377+ INIT_RDMA_OBJ_SIZE(ib_srq, irdma_srq, ibsrq),
47645378 };
47655379
47665380 /**
@@ -4808,6 +5422,10 @@ static void irdma_init_rdma_device(struct irdma_device *iwdev) · 41 unmodified lines
48085422 iwdev->ibdev.num_comp_vectors = iwdev->rf->ceqs_count;
48095423 iwdev->ibdev.dev.parent = &pcidev->dev;
48105424 ib_set_device_ops(&iwdev->ibdev, &irdma_dev_ops);
5425+ if (iwdev->rf->rdma_ver == IRDMA_GEN_1)
5426+ ib_set_device_ops(&iwdev->ibdev, &irdma_gen1_dev_ops);
5427+ if (iwdev->rf->rdma_ver >= IRDMA_GEN_3)
5428+ ib_set_device_ops(&iwdev->ibdev, &irdma_gen3_dev_ops);
48115429 }
48125430
48135431 /**
@@ -4879,5 +5497,9 @@ void irdma_ib_dealloc_device(struct ib_device *ibdev) · 65 unmodified lines
48795497 struct irdma_device *iwdev = to_iwdev(ibdev);
48805498
48815499 irdma_rt_deinit_hw(iwdev);
4882− irdma_ctrl_deinit_hw(iwdev->rf);
5500+ if (!iwdev->is_vport) {
5501+ irdma_ctrl_deinit_hw(iwdev->rf);
5502+ if (iwdev->rf->vchnl_wq)
5503+ destroy_workqueue(iwdev->rf->vchnl_wq);
5504+ }
48835505 }
drivers/infiniband/hw/irdma/verbs.h+44−6
@@ -8,6 +8,7 @@ · 7 unmodified lines
88
99 #define IRDMA_PKEY_TBL_SZ 1
1010 #define IRDMA_DEFAULT_PKEY 0xFFFF
11+#define IRDMA_SHADOW_PGCNT 1
1112
1213 struct irdma_ucontext {
1314 struct ib_ucontext ibucontext;
@@ -17,6 +18,8 @@ struct irdma_ucontext { · 3 unmodified lines
1718 spinlock_t cq_reg_mem_list_lock; /* protect CQ memory list */
1819 struct list_head qp_reg_mem_list;
1920 spinlock_t qp_reg_mem_list_lock; /* protect QP memory list */
21+ struct list_head srq_reg_mem_list;
22+ spinlock_t srq_reg_mem_list_lock; /* protect SRQ memory list */
2023 int abi_ver;
2124 u8 legacy_mode : 1;
2225 u8 use_raw_attrs : 1;
@@ -65,10 +68,16 @@ struct irdma_cq_mr { · 42 unmodified lines
6568 bool split;
6669 };
6770
71+struct irdma_srq_mr {
72+ struct irdma_hmc_pble srq_pbl;
73+ dma_addr_t shadow;
74+};
75+
6876 struct irdma_qp_mr {
6977 struct irdma_hmc_pble sq_pbl;
7078 struct irdma_hmc_pble rq_pbl;
7179 dma_addr_t shadow;
80+ dma_addr_t rq_pa;
7281 struct page *sq_page;
7382 };
7483
@@ -85,6 +94,7 @@ struct irdma_pbl { · 10 unmodified lines
8594 union {
8695 struct irdma_qp_mr qp_mr;
8796 struct irdma_cq_mr cq_mr;
97+ struct irdma_srq_mr srq_mr;
8898 };
8999
90100 bool pbl_allocated:1;
@@ -112,24 +122,33 @@ struct irdma_mr { · 21 unmodified lines
112122 struct irdma_pbl iwpbl;
113123 };
114124
125+struct irdma_srq {
126+ struct ib_srq ibsrq;
127+ struct irdma_sc_srq sc_srq __aligned(64);
128+ struct irdma_dma_mem kmem;
129+ u64 *srq_wrid_mem;
130+ refcount_t refcnt;
131+ spinlock_t lock; /* for poll srq */
132+ struct irdma_pbl *iwpbl;
133+ struct irdma_sge *sg_list;
134+ u16 srq_head;
135+ u32 srq_num;
136+ u32 max_wr;
137+ bool user_mode:1;
138+};
139+
115140 struct irdma_cq {
116141 struct ib_cq ibcq;
117142 struct irdma_sc_cq sc_cq;
118− u16 cq_head;
119− u16 cq_size;
120143 u16 cq_num;
121144 bool user_mode;
122145 atomic_t armed;
123146 enum irdma_cmpl_notify last_notify;
124− u32 polled_cmpls;
125− u32 cq_mem_size;
126147 struct irdma_dma_mem kmem;
127148 struct irdma_dma_mem kmem_shadow;
128149 struct completion free_cq;
129150 refcount_t refcnt;
130151 spinlock_t lock; /* for poll cq */
131− struct irdma_pbl *iwpbl;
132− struct irdma_pbl *iwpbl_shadow;
133152 struct list_head resize_list;
134153 struct irdma_cq_poll_info cur_cqe;
135154 struct list_head cmpl_generated;
@@ -259,6 +278,12 @@ static inline void set_ib_wc_op_sq(struct irdma_cq_poll_info *cq_poll_info, · 123 unmodified lines
259278 case IRDMA_OP_TYPE_FAST_REG_NSMR:
260279 entry->opcode = IB_WC_REG_MR;
261280 break;
281+ case IRDMA_OP_TYPE_ATOMIC_COMPARE_AND_SWAP:
282+ entry->opcode = IB_WC_COMP_SWAP;
283+ break;
284+ case IRDMA_OP_TYPE_ATOMIC_FETCH_AND_ADD:
285+ entry->opcode = IB_WC_FETCH_ADD;
286+ break;
262287 case IRDMA_OP_TYPE_INV_STAG:
263288 entry->opcode = IB_WC_LOCAL_INV;
264289 break;
@@ -267,6 +292,19 @@ static inline void set_ib_wc_op_sq(struct irdma_cq_poll_info *cq_poll_info, · 2 unmodified lines
267292 }
268293 }
269294
295+static inline void set_ib_wc_op_rq_gen_3(struct irdma_cq_poll_info *info,
296+ struct ib_wc *entry)
297+{
298+ switch (info->op_type) {
299+ case IRDMA_OP_TYPE_RDMA_WRITE:
300+ case IRDMA_OP_TYPE_RDMA_WRITE_SOL:
301+ entry->opcode = IB_WC_RECV_RDMA_WITH_IMM;
302+ break;
303+ default:
304+ entry->opcode = IB_WC_RECV;
305+ }
306+}
307+
270308 static inline void set_ib_wc_op_rq(struct irdma_cq_poll_info *cq_poll_info,
271309 struct ib_wc *entry, bool send_imm_support)
272310 {
drivers/infiniband/hw/irdma/virtchnl.c+618−0
@@ -0,0 +1,618 @@
1+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
2+/* Copyright (c) 2015 - 2024 Intel Corporation */
3+
4+#include "osdep.h"
5+#include "hmc.h"
6+#include "defs.h"
7+#include "type.h"
8+#include "protos.h"
9+#include "virtchnl.h"
10+#include "ws.h"
11+#include "i40iw_hw.h"
12+#include "ig3rdma_hw.h"
13+
14+struct vchnl_reg_map_elem {
15+ u16 reg_id;
16+ u16 reg_idx;
17+ bool pg_rel;
18+};
19+
20+struct vchnl_regfld_map_elem {
21+ u16 regfld_id;
22+ u16 regfld_idx;
23+};
24+
25+static struct vchnl_reg_map_elem vchnl_reg_map[] = {
26+ {IRDMA_VCHNL_REG_ID_CQPTAIL, IRDMA_CQPTAIL, false},
27+ {IRDMA_VCHNL_REG_ID_CQPDB, IRDMA_CQPDB, false},
28+ {IRDMA_VCHNL_REG_ID_CCQPSTATUS, IRDMA_CCQPSTATUS, false},
29+ {IRDMA_VCHNL_REG_ID_CCQPHIGH, IRDMA_CCQPHIGH, false},
30+ {IRDMA_VCHNL_REG_ID_CCQPLOW, IRDMA_CCQPLOW, false},
31+ {IRDMA_VCHNL_REG_ID_CQARM, IRDMA_CQARM, false},
32+ {IRDMA_VCHNL_REG_ID_CQACK, IRDMA_CQACK, false},
33+ {IRDMA_VCHNL_REG_ID_AEQALLOC, IRDMA_AEQALLOC, false},
34+ {IRDMA_VCHNL_REG_ID_CQPERRCODES, IRDMA_CQPERRCODES, false},
35+ {IRDMA_VCHNL_REG_ID_WQEALLOC, IRDMA_WQEALLOC, false},
36+ {IRDMA_VCHNL_REG_ID_DB_ADDR_OFFSET, IRDMA_DB_ADDR_OFFSET, false },
37+ {IRDMA_VCHNL_REG_ID_DYN_CTL, IRDMA_GLINT_DYN_CTL, false },
38+ {IRDMA_VCHNL_REG_INV_ID, IRDMA_VCHNL_REG_INV_ID, false }
39+};
40+
41+static struct vchnl_regfld_map_elem vchnl_regfld_map[] = {
42+ {IRDMA_VCHNL_REGFLD_ID_CCQPSTATUS_CQP_OP_ERR, IRDMA_CCQPSTATUS_CCQP_ERR_M},
43+ {IRDMA_VCHNL_REGFLD_ID_CCQPSTATUS_CCQP_DONE, IRDMA_CCQPSTATUS_CCQP_DONE_M},
44+ {IRDMA_VCHNL_REGFLD_ID_CQPSQ_STAG_PDID, IRDMA_CQPSQ_STAG_PDID_M},
45+ {IRDMA_VCHNL_REGFLD_ID_CQPSQ_CQ_CEQID, IRDMA_CQPSQ_CQ_CEQID_M},
46+ {IRDMA_VCHNL_REGFLD_ID_CQPSQ_CQ_CQID, IRDMA_CQPSQ_CQ_CQID_M},
47+ {IRDMA_VCHNL_REGFLD_ID_COMMIT_FPM_CQCNT, IRDMA_COMMIT_FPM_CQCNT_M},
48+ {IRDMA_VCHNL_REGFLD_ID_UPESD_HMCN_ID, IRDMA_CQPSQ_UPESD_HMCFNID_M},
49+ {IRDMA_VCHNL_REGFLD_INV_ID, IRDMA_VCHNL_REGFLD_INV_ID}
50+};
51+
52+#define IRDMA_VCHNL_REG_COUNT ARRAY_SIZE(vchnl_reg_map)
53+#define IRDMA_VCHNL_REGFLD_COUNT ARRAY_SIZE(vchnl_regfld_map)
54+#define IRDMA_VCHNL_REGFLD_BUF_SIZE \
55+ (IRDMA_VCHNL_REG_COUNT * sizeof(struct irdma_vchnl_reg_info) + \
56+ IRDMA_VCHNL_REGFLD_COUNT * sizeof(struct irdma_vchnl_reg_field_info))
57+#define IRDMA_REGMAP_RESP_BUF_SIZE (IRDMA_VCHNL_RESP_MIN_SIZE + IRDMA_VCHNL_REGFLD_BUF_SIZE)
58+
59+/**
60+ * irdma_sc_vchnl_init - Initialize dev virtchannel and get hw_rev
61+ * @dev: dev structure to update
62+ * @info: virtchannel info parameters to fill into the dev structure
63+ */
64+int irdma_sc_vchnl_init(struct irdma_sc_dev *dev,
65+ struct irdma_vchnl_init_info *info)
66+{
67+ dev->vchnl_up = true;
68+ dev->privileged = info->privileged;
69+ dev->is_pf = info->is_pf;
70+ dev->hw_attrs.uk_attrs.hw_rev = info->hw_rev;
71+
72+ if (!dev->privileged) {
73+ int ret = irdma_vchnl_req_get_ver(dev, IRDMA_VCHNL_CHNL_VER_MAX,
74+ &dev->vchnl_ver);
75+
76+ ibdev_dbg(to_ibdev(dev),
77+ "DEV: Get Channel version ret = %d, version is %u\n",
78+ ret, dev->vchnl_ver);
79+
80+ if (ret)
81+ return ret;
82+
83+ ret = irdma_vchnl_req_get_caps(dev);
84+ if (ret)
85+ return ret;
86+
87+ dev->hw_attrs.uk_attrs.hw_rev = dev->vc_caps.hw_rev;
88+ }
89+
90+ return 0;
91+}
92+
93+/**
94+ * irdma_vchnl_req_verify_resp - Verify requested response size
95+ * @vchnl_req: vchnl message requested
96+ * @resp_len: response length sent from vchnl peer
97+ */
98+static int irdma_vchnl_req_verify_resp(struct irdma_vchnl_req *vchnl_req,
99+ u16 resp_len)
100+{
101+ switch (vchnl_req->vchnl_msg->op_code) {
102+ case IRDMA_VCHNL_OP_GET_VER:
103+ case IRDMA_VCHNL_OP_GET_HMC_FCN:
104+ case IRDMA_VCHNL_OP_PUT_HMC_FCN:
105+ if (resp_len != vchnl_req->parm_len)
106+ return -EBADMSG;
107+ break;
108+ case IRDMA_VCHNL_OP_GET_RDMA_CAPS:
109+ if (resp_len < IRDMA_VCHNL_OP_GET_RDMA_CAPS_MIN_SIZE)
110+ return -EBADMSG;
111+ break;
112+ case IRDMA_VCHNL_OP_GET_REG_LAYOUT:
113+ case IRDMA_VCHNL_OP_QUEUE_VECTOR_MAP:
114+ case IRDMA_VCHNL_OP_QUEUE_VECTOR_UNMAP:
115+ case IRDMA_VCHNL_OP_ADD_VPORT:
116+ case IRDMA_VCHNL_OP_DEL_VPORT:
117+ break;
118+ default:
119+ return -EOPNOTSUPP;
120+ }
121+
122+ return 0;
123+}
124+
125+static void irdma_free_vchnl_req_msg(struct irdma_vchnl_req *vchnl_req)
126+{
127+ kfree(vchnl_req->vchnl_msg);
128+}
129+
130+static int irdma_alloc_vchnl_req_msg(struct irdma_vchnl_req *vchnl_req,
131+ struct irdma_vchnl_req_init_info *info)
132+{
133+ struct irdma_vchnl_op_buf *vchnl_msg;
134+
135+ vchnl_msg = kzalloc(IRDMA_VCHNL_MAX_MSG_SIZE, GFP_KERNEL);
136+
137+ if (!vchnl_msg)
138+ return -ENOMEM;
139+
140+ vchnl_msg->op_ctx = (uintptr_t)vchnl_req;
141+ vchnl_msg->buf_len = sizeof(*vchnl_msg) + info->req_parm_len;
142+ if (info->req_parm_len)
143+ memcpy(vchnl_msg->buf, info->req_parm, info->req_parm_len);
144+ vchnl_msg->op_code = info->op_code;
145+ vchnl_msg->op_ver = info->op_ver;
146+
147+ vchnl_req->vchnl_msg = vchnl_msg;
148+ vchnl_req->parm = info->resp_parm;
149+ vchnl_req->parm_len = info->resp_parm_len;
150+
151+ return 0;
152+}
153+
154+static int irdma_vchnl_req_send_sync(struct irdma_sc_dev *dev,
155+ struct irdma_vchnl_req_init_info *info)
156+{
157+ u16 resp_len = sizeof(dev->vc_recv_buf);
158+ struct irdma_vchnl_req vchnl_req = {};
159+ u16 msg_len;
160+ u8 *msg;
161+ int ret;
162+
163+ ret = irdma_alloc_vchnl_req_msg(&vchnl_req, info);
164+ if (ret)
165+ return ret;
166+
167+ msg_len = vchnl_req.vchnl_msg->buf_len;
168+ msg = (u8 *)vchnl_req.vchnl_msg;
169+
170+ mutex_lock(&dev->vchnl_mutex);
171+ ret = ig3rdma_vchnl_send_sync(dev, msg, msg_len, dev->vc_recv_buf,
172+ &resp_len);
173+ dev->vc_recv_len = resp_len;
174+ if (ret)
175+ goto exit;
176+
177+ ret = irdma_vchnl_req_get_resp(dev, &vchnl_req);
178+exit:
179+ mutex_unlock(&dev->vchnl_mutex);
180+ ibdev_dbg(to_ibdev(dev),
181+ "VIRT: virtual channel send %s caller: %pS ret=%d op=%u op_ver=%u req_len=%u parm_len=%u resp_len=%u\n",
182+ !ret ? "SUCCEEDS" : "FAILS", __builtin_return_address(0),
183+ ret, vchnl_req.vchnl_msg->op_code,
184+ vchnl_req.vchnl_msg->op_ver, vchnl_req.vchnl_msg->buf_len,
185+ vchnl_req.parm_len, vchnl_req.resp_len);
186+ irdma_free_vchnl_req_msg(&vchnl_req);
187+
188+ return ret;
189+}
190+
191+/**
192+ * irdma_vchnl_req_get_reg_layout - Get Register Layout
193+ * @dev: RDMA device pointer
194+ */
195+int irdma_vchnl_req_get_reg_layout(struct irdma_sc_dev *dev)
196+{
197+ u16 reg_idx, reg_id, tmp_reg_id, regfld_idx, regfld_id, tmp_regfld_id;
198+ struct irdma_vchnl_reg_field_info *regfld_array = NULL;
199+ u8 resp_buffer[IRDMA_REGMAP_RESP_BUF_SIZE] = {};
200+ struct vchnl_regfld_map_elem *regfld_map_array;
201+ struct irdma_vchnl_req_init_info info = {};
202+ struct vchnl_reg_map_elem *reg_map_array;
203+ struct irdma_vchnl_reg_info *reg_array;
204+ u8 num_bits, shift_cnt;
205+ u16 buf_len = 0;
206+ u64 bitmask;
207+ u32 rindex;
208+ int ret;
209+
210+ if (!dev->vchnl_up)
211+ return -EBUSY;
212+
213+ info.op_code = IRDMA_VCHNL_OP_GET_REG_LAYOUT;
214+ info.op_ver = IRDMA_VCHNL_OP_GET_REG_LAYOUT_V0;
215+ info.resp_parm = resp_buffer;
216+ info.resp_parm_len = sizeof(resp_buffer);
217+
218+ ret = irdma_vchnl_req_send_sync(dev, &info);
219+
220+ if (ret)
221+ return ret;
222+
223+ /* parse the response buffer and update reg info*/
224+ /* Parse registers till invalid */
225+ /* Parse register fields till invalid */
226+ reg_array = (struct irdma_vchnl_reg_info *)resp_buffer;
227+ for (rindex = 0; rindex < IRDMA_VCHNL_REG_COUNT; rindex++) {
228+ buf_len += sizeof(struct irdma_vchnl_reg_info);
229+ if (buf_len >= sizeof(resp_buffer))
230+ return -ENOMEM;
231+
232+ regfld_array =
233+ (struct irdma_vchnl_reg_field_info *)&reg_array[rindex + 1];
234+ reg_id = reg_array[rindex].reg_id;
235+ if (reg_id == IRDMA_VCHNL_REG_INV_ID)
236+ break;
237+
238+ reg_id &= ~IRDMA_VCHNL_REG_PAGE_REL;
239+ if (reg_id >= IRDMA_VCHNL_REG_COUNT)
240+ return -EINVAL;
241+
242+ /* search regmap for register index in hw_regs.*/
243+ reg_map_array = vchnl_reg_map;
244+ do {
245+ tmp_reg_id = reg_map_array->reg_id;
246+ if (tmp_reg_id == reg_id)
247+ break;
248+
249+ reg_map_array++;
250+ } while (tmp_reg_id != IRDMA_VCHNL_REG_INV_ID);
251+ if (tmp_reg_id != reg_id)
252+ continue;
253+
254+ reg_idx = reg_map_array->reg_idx;
255+
256+ /* Page relative, DB Offset do not need bar offset */
257+ if (reg_idx == IRDMA_DB_ADDR_OFFSET ||
258+ (reg_array[rindex].reg_id & IRDMA_VCHNL_REG_PAGE_REL)) {
259+ dev->hw_regs[reg_idx] =
260+ (u32 __iomem *)(uintptr_t)reg_array[rindex].reg_offset;
261+ continue;
262+ }
263+
264+ /* Update the local HW struct */
265+ dev->hw_regs[reg_idx] = ig3rdma_get_reg_addr(dev->hw,
266+ reg_array[rindex].reg_offset);
267+ if (!dev->hw_regs[reg_idx])
268+ return -EINVAL;
269+ }
270+
271+ if (!regfld_array)
272+ return -ENOMEM;
273+
274+ /* set up doorbell variables using mapped DB page */
275+ dev->wqe_alloc_db = dev->hw_regs[IRDMA_WQEALLOC];
276+ dev->cq_arm_db = dev->hw_regs[IRDMA_CQARM];
277+ dev->aeq_alloc_db = dev->hw_regs[IRDMA_AEQALLOC];
278+ dev->cqp_db = dev->hw_regs[IRDMA_CQPDB];
279+ dev->cq_ack_db = dev->hw_regs[IRDMA_CQACK];
280+
281+ for (rindex = 0; rindex < IRDMA_VCHNL_REGFLD_COUNT; rindex++) {
282+ buf_len += sizeof(struct irdma_vchnl_reg_field_info);
283+ if ((buf_len - 1) > sizeof(resp_buffer))
284+ break;
285+
286+ if (regfld_array[rindex].fld_id == IRDMA_VCHNL_REGFLD_INV_ID)
287+ break;
288+
289+ regfld_id = regfld_array[rindex].fld_id;
290+ regfld_map_array = vchnl_regfld_map;
291+ do {
292+ tmp_regfld_id = regfld_map_array->regfld_id;
293+ if (tmp_regfld_id == regfld_id)
294+ break;
295+
296+ regfld_map_array++;
297+ } while (tmp_regfld_id != IRDMA_VCHNL_REGFLD_INV_ID);
298+
299+ if (tmp_regfld_id != regfld_id)
300+ continue;
301+
302+ regfld_idx = regfld_map_array->regfld_idx;
303+
304+ num_bits = regfld_array[rindex].fld_bits;
305+ shift_cnt = regfld_array[rindex].fld_shift;
306+ if ((num_bits + shift_cnt > 64) || !num_bits) {
307+ ibdev_dbg(to_ibdev(dev),
308+ "ERR: Invalid field mask id %d bits %d shift %d",
309+ regfld_id, num_bits, shift_cnt);
310+
311+ continue;
312+ }
313+
314+ bitmask = (1ULL << num_bits) - 1;
315+ dev->hw_masks[regfld_idx] = bitmask << shift_cnt;
316+ dev->hw_shifts[regfld_idx] = shift_cnt;
317+ }
318+
319+ return 0;
320+}
321+
322+int irdma_vchnl_req_add_vport(struct irdma_sc_dev *dev, u16 vport_id,
323+ u32 qp1_id, struct irdma_qos *qos)
324+{
325+ struct irdma_vchnl_resp_vport_info resp_vport = { 0 };
326+ struct irdma_vchnl_req_vport_info req_vport = { 0 };
327+ struct irdma_vchnl_req_init_info info = { 0 };
328+ int ret, i;
329+
330+ if (!dev->vchnl_up)
331+ return -EBUSY;
332+
333+ info.op_code = IRDMA_VCHNL_OP_ADD_VPORT;
334+ info.op_ver = IRDMA_VCHNL_OP_ADD_VPORT_V0;
335+ req_vport.vport_id = vport_id;
336+ req_vport.qp1_id = qp1_id;
337+ info.req_parm_len = sizeof(req_vport);
338+ info.req_parm = &req_vport;
339+ info.resp_parm = &resp_vport;
340+ info.resp_parm_len = sizeof(resp_vport);

278 more lines aren’t shown here.

drivers/infiniband/hw/irdma/virtchnl.h+176−0

This diff is large: its 177 lines aren’t shown here.

drivers/infiniband/hw/mana/cq.c+26−0

This diff is large: its 33 lines aren’t shown here.

drivers/infiniband/hw/mana/device.c+3−0

This diff is large: its 10 lines aren’t shown here.

drivers/infiniband/hw/mana/main.c+2−3

This diff is large: its 12 lines aren’t shown here.

drivers/infiniband/hw/mana/mana_ib.h+12−2

This diff is large: its 43 lines aren’t shown here.

drivers/infiniband/hw/mana/mr.c+4−2

This diff is large: its 20 lines aren’t shown here.

drivers/infiniband/hw/mana/qp.c+9−0

This diff is large: its 30 lines aren’t shown here.

drivers/infiniband/hw/mlx4/mad.c+4−4

This diff is large: its 29 lines aren’t shown here.

drivers/infiniband/hw/mlx4/qp.c+2−1

This diff is large: its 10 lines aren’t shown here.

drivers/infiniband/hw/mlx5/data_direct.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/infiniband/hw/mlx5/gsi.c+9−6

This diff is large: its 36 lines aren’t shown here.

drivers/infiniband/hw/mlx5/main.c+102−11

This diff is large: its 239 lines aren’t shown here.

drivers/infiniband/hw/mlx5/mlx5_ib.h+7−0

This diff is large: its 28 lines aren’t shown here.

drivers/infiniband/hw/mlx5/mr.c+5−6

This diff is large: its 26 lines aren’t shown here.

drivers/infiniband/hw/mlx5/umr.c+5−1

This diff is large: its 13 lines aren’t shown here.

drivers/infiniband/sw/rdmavt/qp.c+6−7

This diff is large: its 41 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_task.c+6−2

This diff is large: its 15 lines aren’t shown here.

drivers/infiniband/sw/siw/siw_verbs.c+14−11

This diff is large: its 48 lines aren’t shown here.

drivers/infiniband/ulp/ipoib/ipoib_main.c+11−10

This diff is large: its 54 lines aren’t shown here.

drivers/infiniband/ulp/srpt/ib_srpt.c+7−9

This diff is large: its 52 lines aren’t shown here.

drivers/net/ethernet/broadcom/bnxt/bnxt.c+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/net/ethernet/broadcom/bnxt/bnxt.h+3−0

This diff is large: its 17 lines aren’t shown here.

drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c+6−0

This diff is large: its 13 lines aren’t shown here.

drivers/net/ethernet/pensando/Kconfig+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/net/ethernet/pensando/ionic/Makefile+1−1

This diff is large: its 7 lines aren’t shown here.

drivers/net/ethernet/pensando/ionic/ionic.h+0−7

This diff is large: its 17 lines aren’t shown here.

drivers/net/ethernet/pensando/ionic/ionic_api.h+131−0

This diff is large: its 132 lines aren’t shown here.

drivers/net/ethernet/pensando/ionic/ionic_aux.c+102−0

This diff is large: its 103 lines aren’t shown here.

drivers/net/ethernet/pensando/ionic/ionic_aux.h+10−0

This diff is large: its 11 lines aren’t shown here.

drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c+7−0

This diff is large: its 42 lines aren’t shown here.

drivers/net/ethernet/pensando/ionic/ionic_dev.c+255−15

This diff is large: its 329 lines aren’t shown here.

drivers/net/ethernet/pensando/ionic/ionic_dev.h+13−15

This diff is large: its 63 lines aren’t shown here.

drivers/net/ethernet/pensando/ionic/ionic_if.h+113−5

This diff is large: its 231 lines aren’t shown here.

drivers/net/ethernet/pensando/ionic/ionic_lif.c+27−20

This diff is large: its 145 lines aren’t shown here.

drivers/net/ethernet/pensando/ionic/ionic_lif.h+3−0

This diff is large: its 17 lines aren’t shown here.

drivers/net/ethernet/pensando/ionic/ionic_main.c+3−1

This diff is large: its 25 lines aren’t shown here.

include/rdma/ib_mad.h+1−0

This diff is large: its 8 lines aren’t shown here.

include/rdma/ib_sa.h+37−0

This diff is large: its 58 lines aren’t shown here.

include/rdma/rdma_cm.h+20−1

This diff is large: its 49 lines aren’t shown here.

include/uapi/rdma/ib_user_ioctl_verbs.h+1−0

This diff is large: its 8 lines aren’t shown here.

include/uapi/rdma/ib_user_sa.h+14−0

This diff is large: its 19 lines aren’t shown here.

include/uapi/rdma/ionic-abi.h+115−0

This diff is large: its 116 lines aren’t shown here.

include/uapi/rdma/irdma-abi.h+15−1

This diff is large: its 42 lines aren’t shown here.

include/uapi/rdma/rdma_user_cm.h+40−2

This diff is large: its 75 lines aren’t shown here.