History

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma

Pull RDMA subsystem updates from Jason Gunthorpe: "Overall this cycle did not have any major excitement, and did not require any shared branch with netdev. Lots of driver updates, particularly of the scale-up and performance variety. The largest body of core work was Parav's patches fixing and restructing some of the core code to make way for future RDMA containerization. Summary: - misc small driver fixups to bnxt_re/hfi1/qib/hns/ocrdma/rdmavt/vmw_pvrdma/nes - several major feature adds to bnxt_re driver: SRIOV VF RoCE support, HugePages support, extended hardware stats support, and SRQ support - a notable number of fixes to the i40iw driver from debugging scale up testing - more work to enable the new hip08 chip in the hns driver - misc small ULP fixups to srp/srpt//ipoib - preparation for srp initiator and target to support the RDMA-CM protocol for connections - add RDMA-CM support to srp initiator, srp target is still a WIP - fixes for a couple of places where ipoib could spam the dmesg log - fix encode/decode of FDR/EDR data rates in the core - many patches from Parav with ongoing work to clean up inconsistencies and bugs in RoCE support around the rdma_cm - mlx5 driver support for the userspace features 'thread domain', 'wallclock timestamps' and 'DV Direct Connected transport'. Support for the firmware dual port rocee capability - core support for more than 32 rdma devices in the char dev allocation - kernel doc updates from Randy Dunlap - new netlink uAPI for inspecting RDMA objects similar in spirit to 'ss' - one minor change to the kobject code acked by Greg KH" * tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma: (259 commits) RDMA/nldev: Provide detailed QP information RDMA/nldev: Provide global resource utilization RDMA/core: Add resource tracking for create and destroy PDs RDMA/core: Add resource tracking for create and destroy CQs RDMA/core: Add resource tracking for create and destroy QPs RDMA/restrack: Add general infrastructure to track RDMA resources RDMA/core: Save kernel caller name when creating PD and CQ objects RDMA/core: Use the MODNAME instead of the function name for pd callers RDMA: Move enum ib_cq_creation_flags to uapi headers IB/rxe: Change RDMA_RXE kconfig to use select IB/qib: remove qib_keys.c IB/mthca: remove mthca_user.h RDMA/cm: Fix access to uninitialized variable RDMA/cma: Use existing netif_is_bond_master function IB/core: Avoid SGID attributes query while converting GID from OPA to IB RDMA/mlx5: Avoid memory leak in case of XRCD dealloc failure IB/umad: Fix use of unprotected device pointer IB/iser: Combine substrings for three messages IB/iser: Delete an unnecessary variable initialisation in iser_send_data_out() IB/iser: Delete an error message for a failed memory allocation in iser_send_data_out() ...

Linus Torvalds committed 8 years ago

Commit 7b1cd952 parents 2155e69 e7996a9187 files · +10240 −4028Browse files

Unified · Split

187 files
  1. MAINTAINERS +4 −3
  2. drivers/infiniband/core/Makefile +1 −1
  3. drivers/infiniband/core/addr.c +8 −57
  4. drivers/infiniband/core/cache.c +7 −16
  5. drivers/infiniband/core/cm.c +159 −68
  6. drivers/infiniband/core/cma.c +156 −96
  7. drivers/infiniband/core/cma_configfs.c +1 −1
  8. drivers/infiniband/core/core_priv.h +31 −21
  9. drivers/infiniband/core/cq.c +25 −14
  10. drivers/infiniband/core/device.c +22 −20
  11. drivers/infiniband/core/fmr_pool.c +5 −7
  12. drivers/infiniband/core/iwpm_util.c +1 −0
  13. drivers/infiniband/core/mad.c +0 −1
  14. drivers/infiniband/core/netlink.c +3 −7
  15. drivers/infiniband/core/nldev.c +388 −6
  16. drivers/infiniband/core/restrack.c +164 −0
  17. drivers/infiniband/core/roce_gid_mgmt.c +8 −5
  18. drivers/infiniband/core/sa_query.c +14 −4
  19. drivers/infiniband/core/security.c +4 −6
  20. drivers/infiniband/core/sysfs.c +0 −1
  21. drivers/infiniband/core/ucm.c +28 −45
  22. drivers/infiniband/core/ucma.c +10 −9
  23. drivers/infiniband/core/umem.c +1 −1
  24. drivers/infiniband/core/user_mad.c +57 −66
  25. drivers/infiniband/core/uverbs_cmd.c +7 −7
  26. drivers/infiniband/core/uverbs_ioctl.c +13 −6
  27. drivers/infiniband/core/uverbs_main.c +34 −61
  28. drivers/infiniband/core/uverbs_std_types.c +3 −0
  29. drivers/infiniband/core/verbs.c +177 −135
  30. drivers/infiniband/hw/bnxt_re/bnxt_re.h +34 −9
  31. drivers/infiniband/hw/bnxt_re/hw_counters.c +135 −10
  32. drivers/infiniband/hw/bnxt_re/hw_counters.h +39 −0
  33. drivers/infiniband/hw/bnxt_re/ib_verbs.c +348 −56
  34. drivers/infiniband/hw/bnxt_re/ib_verbs.h +20 −0
  35. drivers/infiniband/hw/bnxt_re/main.c +209 −42
  36. drivers/infiniband/hw/bnxt_re/qplib_fp.c +400 −63
  37. drivers/infiniband/hw/bnxt_re/qplib_fp.h +54 −24
  38. drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +3 −2
  39. drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +3 −4
  40. drivers/infiniband/hw/bnxt_re/qplib_res.c +4 −5
  41. drivers/infiniband/hw/bnxt_re/qplib_sp.c +136 −5
  42. drivers/infiniband/hw/bnxt_re/qplib_sp.h +88 −3
  43. drivers/infiniband/hw/bnxt_re/roce_hsi.h +126 −1
  44. drivers/infiniband/hw/cxgb4/cm.c +16 −11
  45. drivers/infiniband/hw/cxgb4/device.c +16 −20
  46. drivers/infiniband/hw/cxgb4/ev.c +1 −1
  47. drivers/infiniband/hw/cxgb4/iw_cxgb4.h +2 −2
  48. drivers/infiniband/hw/cxgb4/qp.c +3 −3
  49. drivers/infiniband/hw/cxgb4/t4.h +2 −2
  50. drivers/infiniband/hw/hfi1/chip.c +31 −56
  51. drivers/infiniband/hw/hfi1/chip.h +1 −1
  52. drivers/infiniband/hw/hfi1/driver.c +0 −16
  53. drivers/infiniband/hw/hfi1/firmware.c +16 −48
  54. drivers/infiniband/hw/hfi1/hfi.h +13 −12
  55. drivers/infiniband/hw/hfi1/init.c +2 −0
  56. drivers/infiniband/hw/hfi1/mad.c +1 −5
  57. drivers/infiniband/hw/hfi1/qp.c +8 −2
  58. drivers/infiniband/hw/hfi1/rc.c +4 −4
  59. drivers/infiniband/hw/hfi1/verbs.c +1 −5
  60. drivers/infiniband/hw/hns/Makefile +1 −1
  61. drivers/infiniband/hw/hns/hns_roce_cmd.c +1 −0
  62. drivers/infiniband/hw/hns/hns_roce_cmd.h +10 −0
  63. drivers/infiniband/hw/hns/hns_roce_common.h +11 −0
  64. drivers/infiniband/hw/hns/hns_roce_cq.c +10 −9
  65. drivers/infiniband/hw/hns/hns_roce_device.h +96 −7
  66. drivers/infiniband/hw/hns/hns_roce_eq.c +0 −759
  67. drivers/infiniband/hw/hns/hns_roce_eq.h +0 −134
  68. drivers/infiniband/hw/hns/hns_roce_hw_v1.c +741 −17
  69. drivers/infiniband/hw/hns/hns_roce_hw_v1.h +42 −2
  70. drivers/infiniband/hw/hns/hns_roce_hw_v2.c +1655 −182
  71. drivers/infiniband/hw/hns/hns_roce_hw_v2.h +278 −5
  72. drivers/infiniband/hw/hns/hns_roce_main.c +6 −10
  73. drivers/infiniband/hw/hns/hns_roce_qp.c +63 −9
  74. drivers/infiniband/hw/i40iw/Kconfig +0 −1
  75. drivers/infiniband/hw/i40iw/i40iw.h +3 −0
  76. drivers/infiniband/hw/i40iw/i40iw_cm.c +39 −29
  77. drivers/infiniband/hw/i40iw/i40iw_cm.h +4 −4
  78. drivers/infiniband/hw/i40iw/i40iw_ctrl.c +8 −17
  79. drivers/infiniband/hw/i40iw/i40iw_d.h +1 −0
  80. drivers/infiniband/hw/i40iw/i40iw_hw.c +2 −1
  81. drivers/infiniband/hw/i40iw/i40iw_main.c +10 −3
  82. drivers/infiniband/hw/i40iw/i40iw_puda.c +2 −3
  83. drivers/infiniband/hw/i40iw/i40iw_puda.h +1 −0
  84. drivers/infiniband/hw/i40iw/i40iw_uk.c +2 −16
  85. drivers/infiniband/hw/i40iw/i40iw_user.h +1 −2
  86. drivers/infiniband/hw/i40iw/i40iw_utils.c +47 −3
  87. drivers/infiniband/hw/i40iw/i40iw_verbs.c +2 −3
  88. drivers/infiniband/hw/mlx4/cq.c +2 −2
  89. drivers/infiniband/hw/mlx4/main.c +11 −8
  90. drivers/infiniband/hw/mlx4/qp.c +18 −2
  91. drivers/infiniband/hw/mlx5/cong.c +59 −24
  92. drivers/infiniband/hw/mlx5/cq.c +1 −1
  93. drivers/infiniband/hw/mlx5/mad.c +15 −8
  94. drivers/infiniband/hw/mlx5/main.c +1115 −296
  95. drivers/infiniband/hw/mlx5/mlx5_ib.h +91 −20
  96. drivers/infiniband/hw/mlx5/mr.c +3 −0
  97. drivers/infiniband/hw/mlx5/odp.c +0 −9
  98. drivers/infiniband/hw/mlx5/qp.c +405 −27
  99. drivers/infiniband/hw/mthca/mthca_memfree.c +3 −4
  100. drivers/infiniband/hw/mthca/mthca_user.h +0 −112
  101. drivers/infiniband/hw/nes/nes_cm.c +1 −1
  102. drivers/infiniband/hw/nes/nes_cm.h +1 −2
  103. drivers/infiniband/hw/ocrdma/ocrdma_hw.c +7 −12
  104. drivers/infiniband/hw/ocrdma/ocrdma_stats.c +3 −5
  105. drivers/infiniband/hw/ocrdma/ocrdma_verbs.c +4 −6
  106. drivers/infiniband/hw/qedr/verbs.c +7 −8
  107. drivers/infiniband/hw/qib/qib.h +3 −5
  108. drivers/infiniband/hw/qib/qib_driver.c +0 −16
  109. drivers/infiniband/hw/qib/qib_eeprom.c +1 −2
  110. drivers/infiniband/hw/qib/qib_file_ops.c +25 −43
  111. drivers/infiniband/hw/qib/qib_init.c +2 −0
  112. drivers/infiniband/hw/qib/qib_keys.c +0 −235
  113. drivers/infiniband/hw/qib/qib_rc.c +3 −3
  114. drivers/infiniband/hw/qib/qib_verbs.c +0 −2
  115. drivers/infiniband/hw/usnic/usnic_ib_sysfs.c +0 −1
  116. drivers/infiniband/hw/usnic/usnic_ib_verbs.c +0 −1
  117. drivers/infiniband/hw/vmw_pvrdma/pvrdma.h +2 −2
  118. drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c +6 −7
  119. drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c +10 −11
  120. drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c +4 −11
  121. drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c +5 −6
  122. drivers/infiniband/sw/rdmavt/cq.c +9 −6
  123. drivers/infiniband/sw/rdmavt/mcast.c +2 −2
  124. drivers/infiniband/sw/rdmavt/mr.c +1 −1
  125. drivers/infiniband/sw/rdmavt/qp.c +13 −14
  126. drivers/infiniband/sw/rdmavt/srq.c +9 −7
  127. drivers/infiniband/sw/rdmavt/trace.h +2 −2
  128. drivers/infiniband/sw/rdmavt/trace_qp.h +42 −0
  129. drivers/infiniband/sw/rdmavt/vt.c +5 −5
  130. drivers/infiniband/sw/rdmavt/vt.h +3 −3
  131. drivers/infiniband/sw/rxe/Kconfig +2 −2
  132. drivers/infiniband/sw/rxe/rxe.c +0 −6
  133. drivers/infiniband/sw/rxe/rxe.h +5 −1
  134. drivers/infiniband/sw/rxe/rxe_loc.h +0 −1
  135. drivers/infiniband/sw/rxe/rxe_net.c +6 −12
  136. drivers/infiniband/sw/rxe/rxe_net.h +0 −1
  137. drivers/infiniband/sw/rxe/rxe_qp.c +10 −2
  138. drivers/infiniband/sw/rxe/rxe_recv.c +1 −2
  139. drivers/infiniband/sw/rxe/rxe_req.c +1 −8
  140. drivers/infiniband/sw/rxe/rxe_resp.c +2 −3
  141. drivers/infiniband/sw/rxe/rxe_verbs.c +2 −0
  142. drivers/infiniband/sw/rxe/rxe_verbs.h +3 −0
  143. drivers/infiniband/ulp/ipoib/ipoib_cm.c +10 −8
  144. drivers/infiniband/ulp/ipoib/ipoib_ib.c +2 −3
  145. drivers/infiniband/ulp/ipoib/ipoib_main.c +67 −31
  146. drivers/infiniband/ulp/ipoib/ipoib_verbs.c +3 −3
  147. drivers/infiniband/ulp/iser/iser_initiator.c +6 −10
  148. drivers/infiniband/ulp/isert/ib_isert.c +7 −0
  149. drivers/infiniband/ulp/isert/ib_isert.h +1 −0
  150. drivers/infiniband/ulp/opa_vnic/opa_vnic_vema.c +1 −1
  151. drivers/infiniband/ulp/srp/ib_srp.c +624 −171
  152. drivers/infiniband/ulp/srp/ib_srp.h +36 −7
  153. drivers/infiniband/ulp/srpt/ib_srpt.c +585 −377
  154. drivers/infiniband/ulp/srpt/ib_srpt.h +66 −34
  155. drivers/net/ethernet/mellanox/mlx4/qp.c +3 −0
  156. drivers/net/ethernet/mellanox/mlx5/core/eq.c +8 −1
  157. drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c +6 −5
  158. drivers/net/ethernet/mellanox/mlx5/core/fw.c +9 −1
  159. drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c +1 −1
  160. drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c +55 −0
  161. drivers/net/ethernet/mellanox/mlx5/core/lib/gid.c +4 −1
  162. drivers/net/ethernet/mellanox/mlx5/core/main.c +14 −1
  163. drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +1 −1
  164. drivers/net/ethernet/mellanox/mlx5/core/qp.c +113 −12
  165. drivers/net/ethernet/mellanox/mlx5/core/vport.c +85 −6
  166. include/linux/mlx5/device.h +16 −0
  167. include/linux/mlx5/driver.h +41 −2
  168. include/linux/mlx5/mlx5_ifc.h +25 −7
  169. include/linux/mlx5/qp.h +12 −0
  170. include/linux/mlx5/vport.h +4 −0
  171. include/rdma/ib_addr.h +9 −29
  172. include/rdma/ib_sa.h +5 −5
  173. include/rdma/ib_verbs.h +46 −18
  174. include/rdma/opa_addr.h +16 −0
  175. include/rdma/rdma_cm.h +19 −0
  176. include/rdma/rdma_cm_ib.h +4 −4
  177. include/rdma/rdma_vt.h +24 −7
  178. include/rdma/restrack.h +157 −0
  179. include/scsi/srp.h +17 −0
  180. include/uapi/rdma/bnxt_re-abi.h +9 −0
  181. include/uapi/rdma/ib_user_verbs.h +8 −3
  182. include/uapi/rdma/mlx4-abi.h +4 −3
  183. include/uapi/rdma/mlx5-abi.h +49 −4
  184. include/uapi/rdma/rdma_netlink.h +49 −0
  185. include/uapi/rdma/vmw_pvrdma-abi.h +7 −5
  186. lib/kobject.c +2 −0
  187. net/rds/ib.c +2 −4
MAINTAINERS+4−3
@@ -6892,7 +6892,7 @@ M: Jason Gunthorpe <jgg@mellanox.com> · 6891 unmodified lines
68926892 L: linux-rdma@vger.kernel.org
68936893 W: http://www.openfabrics.org/
68946894 Q: http://patchwork.kernel.org/project/linux-rdma/list/
6895−T: git git://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma.git
6895+T: git git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git
68966896 S: Supported
68976897 F: Documentation/devicetree/bindings/infiniband/
68986898 F: Documentation/infiniband/
@@ -11218,7 +11218,8 @@ S: Maintained · 4319 unmodified lines
1121811218 F: drivers/firmware/qemu_fw_cfg.c
1121911219
1122011220 QIB DRIVER
11221−M: Mike Marciniszyn <infinipath@intel.com>
11221+M: Dennis Dalessandro <dennis.dalessandro@intel.com>
11222+M: Mike Marciniszyn <mike.marciniszyn@intel.com>
1122211223 L: linux-rdma@vger.kernel.org
1122311224 S: Supported
1122411225 F: drivers/infiniband/hw/qib/
@@ -11245,7 +11246,6 @@ F: include/linux/qed/ · 20 unmodified lines
1124511246 F: drivers/net/ethernet/qlogic/qede/
1124611247
1124711248 QLOGIC QL4xxx RDMA DRIVER
11248−M: Ram Amrani <Ram.Amrani@cavium.com>
1124911249 M: Michal Kalderon <Michal.Kalderon@cavium.com>
1125011250 M: Ariel Elior <Ariel.Elior@cavium.com>
1125111251 L: linux-rdma@vger.kernel.org
@@ -11507,6 +11507,7 @@ F: drivers/net/ethernet/rdc/r6040.c · 255 unmodified lines
1150711507
1150811508 RDMAVT - RDMA verbs software
1150911509 M: Dennis Dalessandro <dennis.dalessandro@intel.com>
11510+M: Mike Marciniszyn <mike.marciniszyn@intel.com>
1151011511 L: linux-rdma@vger.kernel.org
1151111512 S: Supported
1151211513 F: drivers/infiniband/sw/rdmavt
drivers/infiniband/core/Makefile+1−1
@@ -12,7 +12,7 @@ ib_core-y := packer.o ud_header.o verbs.o cq.o rw.o sysfs.o \ · 11 unmodified lines
1212 device.o fmr_pool.o cache.o netlink.o \
1313 roce_gid_mgmt.o mr_pool.o addr.o sa_query.o \
1414 multicast.o mad.o smi.o agent.o mad_rmpp.o \
15− security.o nldev.o
15+ security.o nldev.o restrack.o
1616
1717 ib_core-$(CONFIG_INFINIBAND_USER_MEM) += umem.o
1818 ib_core-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += umem_odp.o
drivers/infiniband/core/addr.c+8−57
@@ -243,8 +243,7 @@ void rdma_copy_addr(struct rdma_dev_addr *dev_addr, · 242 unmodified lines
243243 EXPORT_SYMBOL(rdma_copy_addr);
244244
245245 int rdma_translate_ip(const struct sockaddr *addr,
246− struct rdma_dev_addr *dev_addr,
247− u16 *vlan_id)
246+ struct rdma_dev_addr *dev_addr)
248247 {
249248 struct net_device *dev;
250249
@@ -266,9 +265,6 @@ int rdma_translate_ip(const struct sockaddr *addr, · 15 unmodified lines
266265 return -EADDRNOTAVAIL;
267266
268267 rdma_copy_addr(dev_addr, dev, NULL);
269− dev_addr->bound_dev_if = dev->ifindex;
270− if (vlan_id)
271− *vlan_id = rdma_vlan_dev_vlan_id(dev);
272268 dev_put(dev);
273269 break;
274270 #if IS_ENABLED(CONFIG_IPV6)
@@ -279,9 +275,6 @@ int rdma_translate_ip(const struct sockaddr *addr, · 4 unmodified lines
279275 &((const struct sockaddr_in6 *)addr)->sin6_addr,
280276 dev, 1)) {
281277 rdma_copy_addr(dev_addr, dev, NULL);
282− dev_addr->bound_dev_if = dev->ifindex;
283− if (vlan_id)
284− *vlan_id = rdma_vlan_dev_vlan_id(dev);
285278 break;
286279 }
287280 }
@@ -481,7 +474,7 @@ static int addr_resolve_neigh(struct dst_entry *dst, · 193 unmodified lines
481474 if (dst->dev->flags & IFF_LOOPBACK) {
482475 int ret;
483476
484− ret = rdma_translate_ip(dst_in, addr, NULL);
477+ ret = rdma_translate_ip(dst_in, addr);
485478 if (!ret)
486479 memcpy(addr->dst_dev_addr, addr->src_dev_addr,
487480 MAX_ADDR_LEN);
@@ -558,7 +551,7 @@ static int addr_resolve(struct sockaddr *src_in, · 70 unmodified lines
558551 }
559552
560553 if (ndev->flags & IFF_LOOPBACK) {
561− ret = rdma_translate_ip(dst_in, addr, NULL);
554+ ret = rdma_translate_ip(dst_in, addr);
562555 /*
563556 * Put the loopback device and get the translated
564557 * device instead.
@@ -744,7 +737,6 @@ void rdma_addr_cancel(struct rdma_dev_addr *addr) · 179 unmodified lines
744737 EXPORT_SYMBOL(rdma_addr_cancel);
745738
746739 struct resolve_cb_context {
747− struct rdma_dev_addr *addr;
748740 struct completion comp;
749741 int status;
750742 };
@@ -752,39 +744,31 @@ struct resolve_cb_context { · 1 unmodified line
752744 static void resolve_cb(int status, struct sockaddr *src_addr,
753745 struct rdma_dev_addr *addr, void *context)
754746 {
755− if (!status)
756− memcpy(((struct resolve_cb_context *)context)->addr,
757− addr, sizeof(struct rdma_dev_addr));
758747 ((struct resolve_cb_context *)context)->status = status;
759748 complete(&((struct resolve_cb_context *)context)->comp);
760749 }
761750
762751 int rdma_addr_find_l2_eth_by_grh(const union ib_gid *sgid,
763752 const union ib_gid *dgid,
764− u8 *dmac, u16 *vlan_id, int *if_index,
753+ u8 *dmac, const struct net_device *ndev,
765754 int *hoplimit)
766755 {
767− int ret = 0;
768756 struct rdma_dev_addr dev_addr;
769757 struct resolve_cb_context ctx;
770− struct net_device *dev;
771−
772758 union {
773759 struct sockaddr _sockaddr;
774760 struct sockaddr_in _sockaddr_in;
775761 struct sockaddr_in6 _sockaddr_in6;
776762 } sgid_addr, dgid_addr;
777−
763+ int ret;
778764
779765 rdma_gid2ip(&sgid_addr._sockaddr, sgid);
780766 rdma_gid2ip(&dgid_addr._sockaddr, dgid);
781767
782768 memset(&dev_addr, 0, sizeof(dev_addr));
783− if (if_index)
784− dev_addr.bound_dev_if = *if_index;
769+ dev_addr.bound_dev_if = ndev->ifindex;
785770 dev_addr.net = &init_net;
786771
787− ctx.addr = &dev_addr;
788772 init_completion(&ctx.comp);
789773 ret = rdma_resolve_ip(&self, &sgid_addr._sockaddr, &dgid_addr._sockaddr,
790774 &dev_addr, 1000, resolve_cb, &ctx);
@@ -798,42 +782,9 @@ int rdma_addr_find_l2_eth_by_grh(const union ib_gid *sgid, · 7 unmodified lines
798782 return ret;
799783
800784 memcpy(dmac, dev_addr.dst_dev_addr, ETH_ALEN);
801− dev = dev_get_by_index(&init_net, dev_addr.bound_dev_if);
802− if (!dev)
803− return -ENODEV;
804− if (if_index)
805− *if_index = dev_addr.bound_dev_if;
806− if (vlan_id)
807− *vlan_id = rdma_vlan_dev_vlan_id(dev);
808− if (hoplimit)
809− *hoplimit = dev_addr.hoplimit;
810− dev_put(dev);
811− return ret;
812−}
813−EXPORT_SYMBOL(rdma_addr_find_l2_eth_by_grh);
814−
815−int rdma_addr_find_smac_by_sgid(union ib_gid *sgid, u8 *smac, u16 *vlan_id)
816−{
817− int ret = 0;
818− struct rdma_dev_addr dev_addr;
819− union {
820− struct sockaddr _sockaddr;
821− struct sockaddr_in _sockaddr_in;
822− struct sockaddr_in6 _sockaddr_in6;
823− } gid_addr;
824−
825− rdma_gid2ip(&gid_addr._sockaddr, sgid);
826−
827− memset(&dev_addr, 0, sizeof(dev_addr));
828− dev_addr.net = &init_net;
829− ret = rdma_translate_ip(&gid_addr._sockaddr, &dev_addr, vlan_id);
830− if (ret)
831− return ret;
832−
833− memcpy(smac, dev_addr.src_dev_addr, ETH_ALEN);
834− return ret;
785+ *hoplimit = dev_addr.hoplimit;
786+ return 0;
835787 }
836−EXPORT_SYMBOL(rdma_addr_find_smac_by_sgid);
837788
838789 static int netevent_callback(struct notifier_block *self, unsigned long event,
839790 void *ctx)
drivers/infiniband/core/cache.c+7−16
@@ -573,27 +573,24 @@ static int ib_cache_gid_find_by_filter(struct ib_device *ib_dev, · 572 unmodified lines
573573 struct ib_gid_attr attr;
574574
575575 if (table->data_vec[i].props & GID_TABLE_ENTRY_INVALID)
576− goto next;
576+ continue;
577577
578578 if (memcmp(gid, &table->data_vec[i].gid, sizeof(*gid)))
579− goto next;
579+ continue;
580580
581581 memcpy(&attr, &table->data_vec[i].attr, sizeof(attr));
582582
583− if (filter(gid, &attr, context))
583+ if (filter(gid, &attr, context)) {
584584 found = true;
585−
586−next:
587− if (found)
585+ if (index)
586+ *index = i;
588587 break;
588+ }
589589 }
590590 read_unlock_irqrestore(&table->rwlock, flags);
591591
592592 if (!found)
593593 return -ENOENT;
594−
595− if (index)
596− *index = i;
597594 return 0;
598595 }
599596
@@ -824,12 +821,7 @@ static int gid_table_setup_one(struct ib_device *ib_dev) · 224 unmodified lines
824821 if (err)
825822 return err;
826823
827− err = roce_rescan_device(ib_dev);
828−
829− if (err) {
830− gid_table_cleanup_one(ib_dev);
831− gid_table_release_one(ib_dev);
832− }
824+ rdma_roce_rescan_device(ib_dev);
833825
834826 return err;
835827 }
@@ -883,7 +875,6 @@ int ib_find_gid_by_filter(struct ib_device *device, · 47 unmodified lines
883875 port_num, filter,
884876 context, index);
885877 }
886−EXPORT_SYMBOL(ib_find_gid_by_filter);
887878
888879 int ib_get_cached_pkey(struct ib_device *device,
889880 u8 port_num,
drivers/infiniband/core/cm.c+159−68
@@ -452,13 +452,14 @@ static void cm_set_private_data(struct cm_id_private *cm_id_priv, · 451 unmodified lines
452452 cm_id_priv->private_data_len = private_data_len;
453453 }
454454
455−static void cm_init_av_for_response(struct cm_port *port, struct ib_wc *wc,
456− struct ib_grh *grh, struct cm_av *av)
455+static int cm_init_av_for_response(struct cm_port *port, struct ib_wc *wc,
456+ struct ib_grh *grh, struct cm_av *av)
457457 {
458458 av->port = port;
459459 av->pkey_index = wc->pkey_index;
460− ib_init_ah_from_wc(port->cm_dev->ib_device, port->port_num, wc,
461− grh, &av->ah_attr);
460+ return ib_init_ah_attr_from_wc(port->cm_dev->ib_device,
461+ port->port_num, wc,
462+ grh, &av->ah_attr);
462463 }
463464
464465 static int cm_init_av_by_path(struct sa_path_rec *path, struct cm_av *av,
@@ -494,8 +495,11 @@ static int cm_init_av_by_path(struct sa_path_rec *path, struct cm_av *av, · 29 unmodified lines
494495 return ret;
495496
496497 av->port = port;
497− ib_init_ah_from_path(cm_dev->ib_device, port->port_num, path,
498− &av->ah_attr);
498+ ret = ib_init_ah_attr_from_path(cm_dev->ib_device, port->port_num, path,
499+ &av->ah_attr);
500+ if (ret)
501+ return ret;
502+
499503 av->timeout = path->packet_life_time + 1;
500504
501505 spin_lock_irqsave(&cm.lock, flags);
@@ -1560,6 +1564,35 @@ static u16 cm_get_bth_pkey(struct cm_work *work) · 1058 unmodified lines
15601564 return pkey;
15611565 }
15621566
1567+/**
1568+ * Convert OPA SGID to IB SGID
1569+ * ULPs (such as IPoIB) do not understand OPA GIDs and will
1570+ * reject them as the local_gid will not match the sgid. Therefore,
1571+ * change the pathrec's SGID to an IB SGID.
1572+ *
1573+ * @work: Work completion
1574+ * @path: Path record
1575+ */
1576+static void cm_opa_to_ib_sgid(struct cm_work *work,
1577+ struct sa_path_rec *path)
1578+{
1579+ struct ib_device *dev = work->port->cm_dev->ib_device;
1580+ u8 port_num = work->port->port_num;
1581+
1582+ if (rdma_cap_opa_ah(dev, port_num) &&
1583+ (ib_is_opa_gid(&path->sgid))) {
1584+ union ib_gid sgid;
1585+
1586+ if (ib_get_cached_gid(dev, port_num, 0, &sgid, NULL)) {
1587+ dev_warn(&dev->dev,
1588+ "Error updating sgid in CM request\n");
1589+ return;
1590+ }
1591+
1592+ path->sgid = sgid;
1593+ }
1594+}
1595+
15631596 static void cm_format_req_event(struct cm_work *work,
15641597 struct cm_id_private *cm_id_priv,
15651598 struct ib_cm_id *listen_id)
@@ -1573,10 +1606,13 @@ static void cm_format_req_event(struct cm_work *work, · 7 unmodified lines
15731606 param->bth_pkey = cm_get_bth_pkey(work);
15741607 param->port = cm_id_priv->av.port->port_num;
15751608 param->primary_path = &work->path[0];
1576− if (cm_req_has_alt_path(req_msg))
1609+ cm_opa_to_ib_sgid(work, param->primary_path);
1610+ if (cm_req_has_alt_path(req_msg)) {
15771611 param->alternate_path = &work->path[1];
1578− else
1612+ cm_opa_to_ib_sgid(work, param->alternate_path);
1613+ } else {
15791614 param->alternate_path = NULL;
1615+ }
15801616 param->remote_ca_guid = req_msg->local_ca_guid;
15811617 param->remote_qkey = be32_to_cpu(req_msg->local_qkey);
15821618 param->remote_qpn = be32_to_cpu(cm_req_get_local_qpn(req_msg));
@@ -1826,9 +1862,11 @@ static int cm_req_handler(struct cm_work *work) · 243 unmodified lines
18261862
18271863 cm_id_priv = container_of(cm_id, struct cm_id_private, id);
18281864 cm_id_priv->id.remote_id = req_msg->local_comm_id;
1829− cm_init_av_for_response(work->port, work->mad_recv_wc->wc,
1830− work->mad_recv_wc->recv_buf.grh,
1831− &cm_id_priv->av);
1865+ ret = cm_init_av_for_response(work->port, work->mad_recv_wc->wc,
1866+ work->mad_recv_wc->recv_buf.grh,
1867+ &cm_id_priv->av);
1868+ if (ret)
1869+ goto destroy;
18321870 cm_id_priv->timewait_info = cm_create_timewait_info(cm_id_priv->
18331871 id.local_id);
18341872 if (IS_ERR(cm_id_priv->timewait_info)) {
@@ -1841,9 +1879,10 @@ static int cm_req_handler(struct cm_work *work) · 6 unmodified lines
18411879
18421880 listen_cm_id_priv = cm_match_req(work, cm_id_priv);
18431881 if (!listen_cm_id_priv) {
1882+ pr_debug("%s: local_id %d, no listen_cm_id_priv\n", __func__,
1883+ be32_to_cpu(cm_id->local_id));
18441884 ret = -EINVAL;
1845− kfree(cm_id_priv->timewait_info);
1846− goto destroy;
1885+ goto free_timeinfo;
18471886 }
18481887
18491888 cm_id_priv->id.cm_handler = listen_cm_id_priv->id.cm_handler;
@@ -1861,56 +1900,50 @@ static int cm_req_handler(struct cm_work *work) · 11 unmodified lines
18611900 work->port->port_num,
18621901 grh->sgid_index,
18631902 &gid, &gid_attr);
1864− if (!ret) {
1865− if (gid_attr.ndev) {
1866− work->path[0].rec_type =
1867− sa_conv_gid_to_pathrec_type(gid_attr.gid_type);
1868− sa_path_set_ifindex(&work->path[0],
1869− gid_attr.ndev->ifindex);
1870− sa_path_set_ndev(&work->path[0],
1871− dev_net(gid_attr.ndev));
1872− dev_put(gid_attr.ndev);
1873− } else {
1874− cm_path_set_rec_type(work->port->cm_dev->ib_device,
1875− work->port->port_num,
1876− &work->path[0],
1877− &req_msg->primary_local_gid);
1878− }
1879− if (cm_req_has_alt_path(req_msg))
1880− work->path[1].rec_type = work->path[0].rec_type;
1881− cm_format_paths_from_req(req_msg, &work->path[0],
1882− &work->path[1]);
1883− if (cm_id_priv->av.ah_attr.type == RDMA_AH_ATTR_TYPE_ROCE)
1884− sa_path_set_dmac(&work->path[0],
1885− cm_id_priv->av.ah_attr.roce.dmac);
1886− work->path[0].hop_limit = grh->hop_limit;
1887− ret = cm_init_av_by_path(&work->path[0], &cm_id_priv->av,
1888− cm_id_priv);
1903+ if (ret) {
1904+ ib_send_cm_rej(cm_id, IB_CM_REJ_UNSUPPORTED, NULL, 0, NULL, 0);
1905+ goto rejected;
1906+ }
1907+
1908+ if (gid_attr.ndev) {
1909+ work->path[0].rec_type =
1910+ sa_conv_gid_to_pathrec_type(gid_attr.gid_type);
1911+ sa_path_set_ifindex(&work->path[0],
1912+ gid_attr.ndev->ifindex);
1913+ sa_path_set_ndev(&work->path[0],
1914+ dev_net(gid_attr.ndev));
1915+ dev_put(gid_attr.ndev);
1916+ } else {
1917+ cm_path_set_rec_type(work->port->cm_dev->ib_device,
1918+ work->port->port_num,
1919+ &work->path[0],
1920+ &req_msg->primary_local_gid);
18891921 }
1922+ if (cm_req_has_alt_path(req_msg))
1923+ work->path[1].rec_type = work->path[0].rec_type;
1924+ cm_format_paths_from_req(req_msg, &work->path[0],
1925+ &work->path[1]);
1926+ if (cm_id_priv->av.ah_attr.type == RDMA_AH_ATTR_TYPE_ROCE)
1927+ sa_path_set_dmac(&work->path[0],
1928+ cm_id_priv->av.ah_attr.roce.dmac);
1929+ work->path[0].hop_limit = grh->hop_limit;
1930+ ret = cm_init_av_by_path(&work->path[0], &cm_id_priv->av,
1931+ cm_id_priv);
18901932 if (ret) {
1891− int err = ib_get_cached_gid(work->port->cm_dev->ib_device,
1892− work->port->port_num, 0,
1893− &work->path[0].sgid,
1894− &gid_attr);
1895− if (!err && gid_attr.ndev) {
1896− work->path[0].rec_type =
1897− sa_conv_gid_to_pathrec_type(gid_attr.gid_type);
1898− sa_path_set_ifindex(&work->path[0],
1899− gid_attr.ndev->ifindex);
1900− sa_path_set_ndev(&work->path[0],
1901− dev_net(gid_attr.ndev));
1902− dev_put(gid_attr.ndev);
1903− } else {
1904− cm_path_set_rec_type(work->port->cm_dev->ib_device,
1905− work->port->port_num,
1906− &work->path[0],
1907− &req_msg->primary_local_gid);
1908− }
1909− if (cm_req_has_alt_path(req_msg))
1910− work->path[1].rec_type = work->path[0].rec_type;
1911− ib_send_cm_rej(cm_id, IB_CM_REJ_INVALID_GID,
1912− &work->path[0].sgid, sizeof work->path[0].sgid,
1913− NULL, 0);
1933+ int err;
1934+
1935+ err = ib_get_cached_gid(work->port->cm_dev->ib_device,
1936+ work->port->port_num, 0,
1937+ &work->path[0].sgid,
1938+ NULL);
1939+ if (err)
1940+ ib_send_cm_rej(cm_id, IB_CM_REJ_INVALID_GID,
1941+ NULL, 0, NULL, 0);
1942+ else
1943+ ib_send_cm_rej(cm_id, IB_CM_REJ_INVALID_GID,
1944+ &work->path[0].sgid,
1945+ sizeof(work->path[0].sgid),
1946+ NULL, 0);
19141947 goto rejected;
19151948 }
19161949 if (cm_req_has_alt_path(req_msg)) {
@@ -1919,7 +1952,7 @@ static int cm_req_handler(struct cm_work *work) · 2 unmodified lines
19191952 if (ret) {
19201953 ib_send_cm_rej(cm_id, IB_CM_REJ_INVALID_ALT_GID,
19211954 &work->path[0].sgid,
1922− sizeof work->path[0].sgid, NULL, 0);
1955+ sizeof(work->path[0].sgid), NULL, 0);
19231956 goto rejected;
19241957 }
19251958 }
@@ -1945,6 +1978,8 @@ static int cm_req_handler(struct cm_work *work) · 19 unmodified lines
19451978 rejected:
19461979 atomic_dec(&cm_id_priv->refcount);
19471980 cm_deref_id(listen_cm_id_priv);
1981+free_timeinfo:
1982+ kfree(cm_id_priv->timewait_info);
19481983 destroy:
19491984 ib_destroy_cm_id(cm_id);
19501985 return ret;
@@ -1997,6 +2032,8 @@ int ib_send_cm_rep(struct ib_cm_id *cm_id, · 46 unmodified lines
19972032 spin_lock_irqsave(&cm_id_priv->lock, flags);
19982033 if (cm_id->state != IB_CM_REQ_RCVD &&
19992034 cm_id->state != IB_CM_MRA_REQ_SENT) {
2035+ pr_debug("%s: local_comm_id %d, cm_id->state: %d\n", __func__,
2036+ be32_to_cpu(cm_id_priv->id.local_id), cm_id->state);
20002037 ret = -EINVAL;
20012038 goto out;
20022039 }
@@ -2063,6 +2100,8 @@ int ib_send_cm_rtu(struct ib_cm_id *cm_id, · 60 unmodified lines
20632100 spin_lock_irqsave(&cm_id_priv->lock, flags);
20642101 if (cm_id->state != IB_CM_REP_RCVD &&
20652102 cm_id->state != IB_CM_MRA_REP_SENT) {
2103+ pr_debug("%s: local_id %d, cm_id->state %d\n", __func__,
2104+ be32_to_cpu(cm_id->local_id), cm_id->state);
20662105 ret = -EINVAL;
20672106 goto error;
20682107 }
@@ -2170,6 +2209,8 @@ static int cm_rep_handler(struct cm_work *work) · 101 unmodified lines
21702209 cm_id_priv = cm_acquire_id(rep_msg->remote_comm_id, 0);
21712210 if (!cm_id_priv) {
21722211 cm_dup_rep_handler(work);
2212+ pr_debug("%s: remote_comm_id %d, no cm_id_priv\n", __func__,
2213+ be32_to_cpu(rep_msg->remote_comm_id));
21732214 return -EINVAL;
21742215 }
21752216
@@ -2183,6 +2224,10 @@ static int cm_rep_handler(struct cm_work *work) · 7 unmodified lines
21832224 default:
21842225 spin_unlock_irq(&cm_id_priv->lock);
21852226 ret = -EINVAL;
2227+ pr_debug("%s: cm_id_priv->id.state: %d, local_comm_id %d, remote_comm_id %d\n",
2228+ __func__, cm_id_priv->id.state,
2229+ be32_to_cpu(rep_msg->local_comm_id),
2230+ be32_to_cpu(rep_msg->remote_comm_id));
21862231 goto error;
21872232 }
21882233
@@ -2196,6 +2241,8 @@ static int cm_rep_handler(struct cm_work *work) · 7 unmodified lines
21962241 spin_unlock(&cm.lock);
21972242 spin_unlock_irq(&cm_id_priv->lock);
21982243 ret = -EINVAL;
2244+ pr_debug("%s: Failed to insert remote id %d\n", __func__,
2245+ be32_to_cpu(rep_msg->remote_comm_id));
21992246 goto error;
22002247 }
22012248 /* Check for a stale connection. */
@@ -2213,6 +2260,10 @@ static int cm_rep_handler(struct cm_work *work) · 11 unmodified lines
22132260 IB_CM_REJ_STALE_CONN, CM_MSG_RESPONSE_REP,
22142261 NULL, 0);
22152262 ret = -EINVAL;
2263+ pr_debug("%s: Stale connection. local_comm_id %d, remote_comm_id %d\n",
2264+ __func__, be32_to_cpu(rep_msg->local_comm_id),
2265+ be32_to_cpu(rep_msg->remote_comm_id));
2266+
22162267 if (cur_cm_id_priv) {
22172268 cm_id = &cur_cm_id_priv->id;
22182269 ib_send_cm_dreq(cm_id, NULL, 0);
@@ -2359,6 +2410,8 @@ int ib_send_cm_dreq(struct ib_cm_id *cm_id, · 140 unmodified lines
23592410 cm_id_priv = container_of(cm_id, struct cm_id_private, id);
23602411 spin_lock_irqsave(&cm_id_priv->lock, flags);
23612412 if (cm_id->state != IB_CM_ESTABLISHED) {
2413+ pr_debug("%s: local_id %d, cm_id->state: %d\n", __func__,
2414+ be32_to_cpu(cm_id->local_id), cm_id->state);
23622415 ret = -EINVAL;
23632416 goto out;
23642417 }
@@ -2428,6 +2481,8 @@ int ib_send_cm_drep(struct ib_cm_id *cm_id, · 63 unmodified lines
24282481 if (cm_id->state != IB_CM_DREQ_RCVD) {
24292482 spin_unlock_irqrestore(&cm_id_priv->lock, flags);
24302483 kfree(data);
2484+ pr_debug("%s: local_id %d, cm_idcm_id->state(%d) != IB_CM_DREQ_RCVD\n",
2485+ __func__, be32_to_cpu(cm_id->local_id), cm_id->state);
24312486 return -EINVAL;
24322487 }
24332488
@@ -2493,6 +2548,9 @@ static int cm_dreq_handler(struct cm_work *work) · 59 unmodified lines
24932548 atomic_long_inc(&work->port->counter_group[CM_RECV_DUPLICATES].
24942549 counter[CM_DREQ_COUNTER]);
24952550 cm_issue_drep(work->port, work->mad_recv_wc);
2551+ pr_debug("%s: no cm_id_priv, local_comm_id %d, remote_comm_id %d\n",
2552+ __func__, be32_to_cpu(dreq_msg->local_comm_id),
2553+ be32_to_cpu(dreq_msg->remote_comm_id));
24962554 return -EINVAL;
24972555 }
24982556
@@ -2535,6 +2593,9 @@ static int cm_dreq_handler(struct cm_work *work) · 36 unmodified lines
25352593 counter[CM_DREQ_COUNTER]);
25362594 goto unlock;
25372595 default:
2596+ pr_debug("%s: local_id %d, cm_id_priv->id.state: %d\n",
2597+ __func__, be32_to_cpu(cm_id_priv->id.local_id),
2598+ cm_id_priv->id.state);
25382599 goto unlock;
25392600 }
25402601 cm_id_priv->id.state = IB_CM_DREQ_RCVD;
@@ -2638,6 +2699,8 @@ int ib_send_cm_rej(struct ib_cm_id *cm_id, · 97 unmodified lines
26382699 cm_enter_timewait(cm_id_priv);
26392700 break;
26402701 default:
2702+ pr_debug("%s: local_id %d, cm_id->state: %d\n", __func__,
2703+ be32_to_cpu(cm_id_priv->id.local_id), cm_id->state);
26412704 ret = -EINVAL;
26422705 goto out;
26432706 }
@@ -2748,6 +2811,9 @@ static int cm_rej_handler(struct cm_work *work) · 104 unmodified lines
27482811 /* fall through */
27492812 default:
27502813 spin_unlock_irq(&cm_id_priv->lock);
2814+ pr_debug("%s: local_id %d, cm_id_priv->id.state: %d\n",
2815+ __func__, be32_to_cpu(cm_id_priv->id.local_id),
2816+ cm_id_priv->id.state);
27512817 ret = -EINVAL;
27522818 goto out;
27532819 }
@@ -2811,6 +2877,9 @@ int ib_send_cm_mra(struct ib_cm_id *cm_id, · 57 unmodified lines
28112877 }
28122878 /* fall through */
28132879 default:
2880+ pr_debug("%s: local_id %d, cm_id_priv->id.state: %d\n",
2881+ __func__, be32_to_cpu(cm_id_priv->id.local_id),
2882+ cm_id_priv->id.state);
28142883 ret = -EINVAL;
28152884 goto error1;
28162885 }
@@ -2912,6 +2981,9 @@ static int cm_mra_handler(struct cm_work *work) · 95 unmodified lines
29122981 counter[CM_MRA_COUNTER]);
29132982 /* fall through */
29142983 default:
2984+ pr_debug("%s local_id %d, cm_id_priv->id.state: %d\n",
2985+ __func__, be32_to_cpu(cm_id_priv->id.local_id),
2986+ cm_id_priv->id.state);
29152987 goto out;
29162988 }
29172989
@@ -3085,6 +3157,12 @@ static int cm_lap_handler(struct cm_work *work) · 167 unmodified lines
30853157 if (!cm_id_priv)
30863158 return -EINVAL;
30873159
3160+ ret = cm_init_av_for_response(work->port, work->mad_recv_wc->wc,
3161+ work->mad_recv_wc->recv_buf.grh,
3162+ &cm_id_priv->av);
3163+ if (ret)
3164+ goto deref;
3165+
30883166 param = &work->cm_event.param.lap_rcvd;
30893167 memset(&work->path[0], 0, sizeof(work->path[1]));
30903168 cm_path_set_rec_type(work->port->cm_dev->ib_device,
@@ -3131,9 +3209,6 @@ static int cm_lap_handler(struct cm_work *work) · 40 unmodified lines
31313209
31323210 cm_id_priv->id.lap_state = IB_CM_LAP_RCVD;
31333211 cm_id_priv->tid = lap_msg->hdr.tid;
3134− cm_init_av_for_response(work->port, work->mad_recv_wc->wc,
3135− work->mad_recv_wc->recv_buf.grh,
3136− &cm_id_priv->av);
31373212 cm_init_av_by_path(param->alternate_path, &cm_id_priv->alt_av,
31383213 cm_id_priv);
31393214 ret = atomic_inc_and_test(&cm_id_priv->work_count);
@@ -3386,6 +3461,7 @@ static int cm_sidr_req_handler(struct cm_work *work) · 246 unmodified lines
33863461 struct cm_id_private *cm_id_priv, *cur_cm_id_priv;
33873462 struct cm_sidr_req_msg *sidr_req_msg;
33883463 struct ib_wc *wc;
3464+ int ret;
33893465
33903466 cm_id = ib_create_cm_id(work->port->cm_dev->ib_device, NULL, NULL);
33913467 if (IS_ERR(cm_id))
@@ -3398,9 +3474,12 @@ static int cm_sidr_req_handler(struct cm_work *work) · 6 unmodified lines
33983474 wc = work->mad_recv_wc->wc;
33993475 cm_id_priv->av.dgid.global.subnet_prefix = cpu_to_be64(wc->slid);
34003476 cm_id_priv->av.dgid.global.interface_id = 0;
3401− cm_init_av_for_response(work->port, work->mad_recv_wc->wc,
3402− work->mad_recv_wc->recv_buf.grh,
3403− &cm_id_priv->av);
3477+ ret = cm_init_av_for_response(work->port, work->mad_recv_wc->wc,
3478+ work->mad_recv_wc->recv_buf.grh,
3479+ &cm_id_priv->av);
3480+ if (ret)
3481+ goto out;
3482+
34043483 cm_id_priv->id.remote_id = sidr_req_msg->request_id;
34053484 cm_id_priv->tid = sidr_req_msg->hdr.tid;
34063485 atomic_inc(&cm_id_priv->work_count);
@@ -3692,6 +3771,7 @@ static void cm_work_handler(struct work_struct *_work) · 285 unmodified lines
36923771 ret = cm_timewait_handler(work);
36933772 break;
36943773 default:
3774+ pr_debug("cm_event.event: 0x%x\n", work->cm_event.event);
36953775 ret = -EINVAL;
36963776 break;
36973777 }
@@ -3727,6 +3807,8 @@ static int cm_establish(struct ib_cm_id *cm_id) · 29 unmodified lines
37273807 ret = -EISCONN;
37283808 break;
37293809 default:
3810+ pr_debug("%s: local_id %d, cm_id->state: %d\n", __func__,
3811+ be32_to_cpu(cm_id->local_id), cm_id->state);
37303812 ret = -EINVAL;
37313813 break;
37323814 }
@@ -3924,6 +4006,9 @@ static int cm_init_qp_init_attr(struct cm_id_private *cm_id_priv, · 191 unmodified lines
39244006 ret = 0;
39254007 break;
39264008 default:
4009+ pr_debug("%s: local_id %d, cm_id_priv->id.state: %d\n",
4010+ __func__, be32_to_cpu(cm_id_priv->id.local_id),
4011+ cm_id_priv->id.state);
39274012 ret = -EINVAL;
39284013 break;
39294014 }
@@ -3971,6 +4056,9 @@ static int cm_init_qp_rtr_attr(struct cm_id_private *cm_id_priv, · 41 unmodified lines
39714056 ret = 0;
39724057 break;
39734058 default:
4059+ pr_debug("%s: local_id %d, cm_id_priv->id.state: %d\n",
4060+ __func__, be32_to_cpu(cm_id_priv->id.local_id),
4061+ cm_id_priv->id.state);
39744062 ret = -EINVAL;
39754063 break;
39764064 }
@@ -4030,6 +4118,9 @@ static int cm_init_qp_rts_attr(struct cm_id_private *cm_id_priv, · 53 unmodified lines
40304118 ret = 0;
40314119 break;
40324120 default:
4121+ pr_debug("%s: local_id %d, cm_id_priv->id.state: %d\n",
4122+ __func__, be32_to_cpu(cm_id_priv->id.local_id),
4123+ cm_id_priv->id.state);
40334124 ret = -EINVAL;
40344125 break;
40354126 }
drivers/infiniband/core/cma.c+156−96
@@ -601,7 +601,7 @@ static int cma_translate_addr(struct sockaddr *addr, struct rdma_dev_addr *dev_a · 600 unmodified lines
601601 int ret;
602602
603603 if (addr->sa_family != AF_IB) {
604− ret = rdma_translate_ip(addr, dev_addr, NULL);
604+ ret = rdma_translate_ip(addr, dev_addr);
605605 } else {
606606 cma_translate_ib((struct sockaddr_ib *) addr, dev_addr);
607607 ret = 0;
@@ -612,11 +612,14 @@ static int cma_translate_addr(struct sockaddr *addr, struct rdma_dev_addr *dev_a · 4 unmodified lines
612612
613613 static inline int cma_validate_port(struct ib_device *device, u8 port,
614614 enum ib_gid_type gid_type,
615− union ib_gid *gid, int dev_type,
616− int bound_if_index)
615+ union ib_gid *gid,
616+ struct rdma_id_private *id_priv)
617617 {
618− int ret = -ENODEV;
618+ struct rdma_dev_addr *dev_addr = &id_priv->id.route.addr.dev_addr;
619+ int bound_if_index = dev_addr->bound_dev_if;
620+ int dev_type = dev_addr->dev_type;
619621 struct net_device *ndev = NULL;
622+ int ret = -ENODEV;
620623
621624 if ((dev_type == ARPHRD_INFINIBAND) && !rdma_protocol_ib(device, port))
622625 return ret;
@@ -624,11 +627,13 @@ static inline int cma_validate_port(struct ib_device *device, u8 port, · 1 unmodified line
624627 if ((dev_type != ARPHRD_INFINIBAND) && rdma_protocol_ib(device, port))
625628 return ret;
626629
627− if (dev_type == ARPHRD_ETHER && rdma_protocol_roce(device, port))
628− ndev = dev_get_by_index(&init_net, bound_if_index);
629− else
630+ if (dev_type == ARPHRD_ETHER && rdma_protocol_roce(device, port)) {
631+ ndev = dev_get_by_index(dev_addr->net, bound_if_index);
632+ if (!ndev)
633+ return ret;
634+ } else {
630635 gid_type = IB_GID_TYPE_IB;
631−
636+ }
632637
633638 ret = ib_find_cached_gid_by_port(device, gid, gid_type, port,
634639 ndev, NULL);
@@ -669,8 +674,7 @@ static int cma_acquire_dev(struct rdma_id_private *id_priv, · 34 unmodified lines
669674 rdma_protocol_ib(cma_dev->device, port) ?
670675 IB_GID_TYPE_IB :
671676 listen_id_priv->gid_type, gidp,
672− dev_addr->dev_type,
673− dev_addr->bound_dev_if);
677+ id_priv);
674678 if (!ret) {
675679 id_priv->id.port_num = port;
676680 goto out;
@@ -691,8 +695,7 @@ static int cma_acquire_dev(struct rdma_id_private *id_priv, · 14 unmodified lines
691695 rdma_protocol_ib(cma_dev->device, port) ?
692696 IB_GID_TYPE_IB :
693697 cma_dev->default_gid_type[port - 1],
694− gidp, dev_addr->dev_type,
695− dev_addr->bound_dev_if);
698+ gidp, id_priv);
696699 if (!ret) {
697700 id_priv->id.port_num = port;
698701 goto out;
@@ -2036,6 +2039,33 @@ __be64 rdma_get_service_id(struct rdma_cm_id *id, struct sockaddr *addr) · 1337 unmodified lines
20362039 }
20372040 EXPORT_SYMBOL(rdma_get_service_id);
20382041
2042+void rdma_read_gids(struct rdma_cm_id *cm_id, union ib_gid *sgid,
2043+ union ib_gid *dgid)
2044+{
2045+ struct rdma_addr *addr = &cm_id->route.addr;
2046+
2047+ if (!cm_id->device) {
2048+ if (sgid)
2049+ memset(sgid, 0, sizeof(*sgid));
2050+ if (dgid)
2051+ memset(dgid, 0, sizeof(*dgid));
2052+ return;
2053+ }
2054+
2055+ if (rdma_protocol_roce(cm_id->device, cm_id->port_num)) {
2056+ if (sgid)
2057+ rdma_ip2gid((struct sockaddr *)&addr->src_addr, sgid);
2058+ if (dgid)
2059+ rdma_ip2gid((struct sockaddr *)&addr->dst_addr, dgid);
2060+ } else {
2061+ if (sgid)
2062+ rdma_addr_get_sgid(&addr->dev_addr, sgid);
2063+ if (dgid)
2064+ rdma_addr_get_dgid(&addr->dev_addr, dgid);
2065+ }
2066+}
2067+EXPORT_SYMBOL(rdma_read_gids);
2068+
20392069 static int cma_iw_handler(struct iw_cm_id *iw_id, struct iw_cm_event *iw_event)
20402070 {
20412071 struct rdma_id_private *id_priv = iw_id->context;
@@ -2132,7 +2162,7 @@ static int iw_conn_req_handler(struct iw_cm_id *cm_id, · 90 unmodified lines
21322162 mutex_lock_nested(&conn_id->handler_mutex, SINGLE_DEPTH_NESTING);
21332163 conn_id->state = RDMA_CM_CONNECT;
21342164
2135− ret = rdma_translate_ip(laddr, &conn_id->id.route.addr.dev_addr, NULL);
2165+ ret = rdma_translate_ip(laddr, &conn_id->id.route.addr.dev_addr);
21362166 if (ret) {
21372167 mutex_unlock(&conn_id->handler_mutex);
21382168 rdma_destroy_id(new_cm_id);
@@ -2414,6 +2444,26 @@ out: · 275 unmodified lines
24142444 kfree(work);
24152445 }
24162446
2447+static void cma_init_resolve_route_work(struct cma_work *work,
2448+ struct rdma_id_private *id_priv)
2449+{
2450+ work->id = id_priv;
2451+ INIT_WORK(&work->work, cma_work_handler);
2452+ work->old_state = RDMA_CM_ROUTE_QUERY;
2453+ work->new_state = RDMA_CM_ROUTE_RESOLVED;
2454+ work->event.event = RDMA_CM_EVENT_ROUTE_RESOLVED;
2455+}
2456+
2457+static void cma_init_resolve_addr_work(struct cma_work *work,
2458+ struct rdma_id_private *id_priv)
2459+{
2460+ work->id = id_priv;
2461+ INIT_WORK(&work->work, cma_work_handler);
2462+ work->old_state = RDMA_CM_ADDR_QUERY;
2463+ work->new_state = RDMA_CM_ADDR_RESOLVED;
2464+ work->event.event = RDMA_CM_EVENT_ADDR_RESOLVED;
2465+}
2466+
24172467 static int cma_resolve_ib_route(struct rdma_id_private *id_priv, int timeout_ms)
24182468 {
24192469 struct rdma_route *route = &id_priv->id.route;
@@ -2424,11 +2474,7 @@ static int cma_resolve_ib_route(struct rdma_id_private *id_priv, int timeout_ms) · 4 unmodified lines
24242474 if (!work)
24252475 return -ENOMEM;
24262476
2427− work->id = id_priv;
2428− INIT_WORK(&work->work, cma_work_handler);
2429− work->old_state = RDMA_CM_ROUTE_QUERY;
2430− work->new_state = RDMA_CM_ROUTE_RESOLVED;
2431− work->event.event = RDMA_CM_EVENT_ROUTE_RESOLVED;
2477+ cma_init_resolve_route_work(work, id_priv);
24322478
24332479 route->path_rec = kmalloc(sizeof *route->path_rec, GFP_KERNEL);
24342480 if (!route->path_rec) {
@@ -2449,10 +2495,63 @@ err1: · 14 unmodified lines
24492495 return ret;
24502496 }
24512497
2452−int rdma_set_ib_paths(struct rdma_cm_id *id,
2453− struct sa_path_rec *path_rec, int num_paths)
2498+static enum ib_gid_type cma_route_gid_type(enum rdma_network_type network_type,
2499+ unsigned long supported_gids,
2500+ enum ib_gid_type default_gid)
2501+{
2502+ if ((network_type == RDMA_NETWORK_IPV4 ||
2503+ network_type == RDMA_NETWORK_IPV6) &&
2504+ test_bit(IB_GID_TYPE_ROCE_UDP_ENCAP, &supported_gids))
2505+ return IB_GID_TYPE_ROCE_UDP_ENCAP;
2506+
2507+ return default_gid;
2508+}
2509+
2510+/*
2511+ * cma_iboe_set_path_rec_l2_fields() is helper function which sets
2512+ * path record type based on GID type.
2513+ * It also sets up other L2 fields which includes destination mac address
2514+ * netdev ifindex, of the path record.
2515+ * It returns the netdev of the bound interface for this path record entry.
2516+ */
2517+static struct net_device *
2518+cma_iboe_set_path_rec_l2_fields(struct rdma_id_private *id_priv)
2519+{
2520+ struct rdma_route *route = &id_priv->id.route;
2521+ enum ib_gid_type gid_type = IB_GID_TYPE_ROCE;
2522+ struct rdma_addr *addr = &route->addr;
2523+ unsigned long supported_gids;
2524+ struct net_device *ndev;
2525+
2526+ if (!addr->dev_addr.bound_dev_if)
2527+ return NULL;
2528+
2529+ ndev = dev_get_by_index(addr->dev_addr.net,
2530+ addr->dev_addr.bound_dev_if);
2531+ if (!ndev)
2532+ return NULL;
2533+
2534+ supported_gids = roce_gid_type_mask_support(id_priv->id.device,
2535+ id_priv->id.port_num);
2536+ gid_type = cma_route_gid_type(addr->dev_addr.network,
2537+ supported_gids,
2538+ id_priv->gid_type);
2539+ /* Use the hint from IP Stack to select GID Type */
2540+ if (gid_type < ib_network_to_gid_type(addr->dev_addr.network))
2541+ gid_type = ib_network_to_gid_type(addr->dev_addr.network);
2542+ route->path_rec->rec_type = sa_conv_gid_to_pathrec_type(gid_type);
2543+
2544+ sa_path_set_ndev(route->path_rec, addr->dev_addr.net);
2545+ sa_path_set_ifindex(route->path_rec, ndev->ifindex);
2546+ sa_path_set_dmac(route->path_rec, addr->dev_addr.dst_dev_addr);
2547+ return ndev;
2548+}
2549+
2550+int rdma_set_ib_path(struct rdma_cm_id *id,
2551+ struct sa_path_rec *path_rec)
24542552 {
24552553 struct rdma_id_private *id_priv;
2554+ struct net_device *ndev;
24562555 int ret;
24572556
24582557 id_priv = container_of(id, struct rdma_id_private, id);
@@ -2460,20 +2559,33 @@ int rdma_set_ib_paths(struct rdma_cm_id *id, · 1 unmodified line
24602559 RDMA_CM_ROUTE_RESOLVED))
24612560 return -EINVAL;
24622561
2463− id->route.path_rec = kmemdup(path_rec, sizeof *path_rec * num_paths,
2562+ id->route.path_rec = kmemdup(path_rec, sizeof(*path_rec),
24642563 GFP_KERNEL);
24652564 if (!id->route.path_rec) {
24662565 ret = -ENOMEM;
24672566 goto err;
24682567 }
24692568
2470− id->route.num_paths = num_paths;
2569+ if (rdma_protocol_roce(id->device, id->port_num)) {
2570+ ndev = cma_iboe_set_path_rec_l2_fields(id_priv);
2571+ if (!ndev) {
2572+ ret = -ENODEV;
2573+ goto err_free;
2574+ }
2575+ dev_put(ndev);
2576+ }
2577+
2578+ id->route.num_paths = 1;
24712579 return 0;
2580+
2581+err_free:
2582+ kfree(id->route.path_rec);
2583+ id->route.path_rec = NULL;
24722584 err:
24732585 cma_comp_exch(id_priv, RDMA_CM_ROUTE_RESOLVED, RDMA_CM_ADDR_RESOLVED);
24742586 return ret;
24752587 }
2476−EXPORT_SYMBOL(rdma_set_ib_paths);
2588+EXPORT_SYMBOL(rdma_set_ib_path);
24772589
24782590 static int cma_resolve_iw_route(struct rdma_id_private *id_priv, int timeout_ms)
24792591 {
@@ -2483,11 +2595,7 @@ static int cma_resolve_iw_route(struct rdma_id_private *id_priv, int timeout_ms) · 3 unmodified lines
24832595 if (!work)
24842596 return -ENOMEM;
24852597
2486− work->id = id_priv;
2487− INIT_WORK(&work->work, cma_work_handler);
2488− work->old_state = RDMA_CM_ROUTE_QUERY;
2489− work->new_state = RDMA_CM_ROUTE_RESOLVED;
2490− work->event.event = RDMA_CM_EVENT_ROUTE_RESOLVED;
2598+ cma_init_resolve_route_work(work, id_priv);
24912599 queue_work(cma_wq, &work->work);
24922600 return 0;
24932601 }
@@ -2510,26 +2618,14 @@ static int iboe_tos_to_sl(struct net_device *ndev, int tos) · 16 unmodified lines
25102618 return 0;
25112619 }
25122620
2513−static enum ib_gid_type cma_route_gid_type(enum rdma_network_type network_type,
2514− unsigned long supported_gids,
2515− enum ib_gid_type default_gid)
2516−{
2517− if ((network_type == RDMA_NETWORK_IPV4 ||
2518− network_type == RDMA_NETWORK_IPV6) &&
2519− test_bit(IB_GID_TYPE_ROCE_UDP_ENCAP, &supported_gids))
2520− return IB_GID_TYPE_ROCE_UDP_ENCAP;
2521−
2522− return default_gid;
2523−}
2524−
25252621 static int cma_resolve_iboe_route(struct rdma_id_private *id_priv)
25262622 {
25272623 struct rdma_route *route = &id_priv->id.route;
25282624 struct rdma_addr *addr = &route->addr;
25292625 struct cma_work *work;
25302626 int ret;
2531− struct net_device *ndev = NULL;
2532− enum ib_gid_type gid_type = IB_GID_TYPE_IB;
2627+ struct net_device *ndev;
2628+
25332629 u8 default_roce_tos = id_priv->cma_dev->default_roce_tos[id_priv->id.port_num -
25342630 rdma_start_port(id_priv->cma_dev->device)];
25352631 u8 tos = id_priv->tos_set ? id_priv->tos : default_roce_tos;
@@ -2539,9 +2635,6 @@ static int cma_resolve_iboe_route(struct rdma_id_private *id_priv) · 3 unmodified lines
25392635 if (!work)
25402636 return -ENOMEM;
25412637
2542− work->id = id_priv;
2543− INIT_WORK(&work->work, cma_work_handler);
2544−
25452638 route->path_rec = kzalloc(sizeof *route->path_rec, GFP_KERNEL);
25462639 if (!route->path_rec) {
25472640 ret = -ENOMEM;
@@ -2550,42 +2643,17 @@ static int cma_resolve_iboe_route(struct rdma_id_private *id_priv) · 2 unmodified lines
25502643
25512644 route->num_paths = 1;
25522645
2553− if (addr->dev_addr.bound_dev_if) {
2554− unsigned long supported_gids;
2555−
2556− ndev = dev_get_by_index(&init_net, addr->dev_addr.bound_dev_if);
2557− if (!ndev) {
2558− ret = -ENODEV;
2559− goto err2;
2560− }
2561−
2562− supported_gids = roce_gid_type_mask_support(id_priv->id.device,
2563− id_priv->id.port_num);
2564− gid_type = cma_route_gid_type(addr->dev_addr.network,
2565− supported_gids,
2566− id_priv->gid_type);
2567− route->path_rec->rec_type =
2568− sa_conv_gid_to_pathrec_type(gid_type);
2569− sa_path_set_ndev(route->path_rec, &init_net);
2570− sa_path_set_ifindex(route->path_rec, ndev->ifindex);
2571− }
2646+ ndev = cma_iboe_set_path_rec_l2_fields(id_priv);
25722647 if (!ndev) {
25732648 ret = -ENODEV;
25742649 goto err2;
25752650 }
25762651
2577− sa_path_set_dmac(route->path_rec, addr->dev_addr.dst_dev_addr);
2578−
25792652 rdma_ip2gid((struct sockaddr *)&id_priv->id.route.addr.src_addr,
25802653 &route->path_rec->sgid);
25812654 rdma_ip2gid((struct sockaddr *)&id_priv->id.route.addr.dst_addr,
25822655 &route->path_rec->dgid);
25832656
2584− /* Use the hint from IP Stack to select GID Type */
2585− if (gid_type < ib_network_to_gid_type(addr->dev_addr.network))
2586− gid_type = ib_network_to_gid_type(addr->dev_addr.network);
2587− route->path_rec->rec_type = sa_conv_gid_to_pathrec_type(gid_type);
2588−
25892657 if (((struct sockaddr *)&id_priv->id.route.addr.dst_addr)->sa_family != AF_IB)
25902658 /* TODO: get the hoplimit from the inet/inet6 device */
25912659 route->path_rec->hop_limit = addr->dev_addr.hoplimit;
@@ -2607,11 +2675,7 @@ static int cma_resolve_iboe_route(struct rdma_id_private *id_priv) · 15 unmodified lines
26072675 goto err2;
26082676 }
26092677
2610− work->old_state = RDMA_CM_ROUTE_QUERY;
2611− work->new_state = RDMA_CM_ROUTE_RESOLVED;
2612− work->event.event = RDMA_CM_EVENT_ROUTE_RESOLVED;
2613− work->event.status = 0;
2614−
2678+ cma_init_resolve_route_work(work, id_priv);
26152679 queue_work(cma_wq, &work->work);
26162680
26172681 return 0;
@@ -2791,11 +2855,7 @@ static int cma_resolve_loopback(struct rdma_id_private *id_priv) · 173 unmodified lines
27912855 rdma_addr_get_sgid(&id_priv->id.route.addr.dev_addr, &gid);
27922856 rdma_addr_set_dgid(&id_priv->id.route.addr.dev_addr, &gid);
27932857
2794− work->id = id_priv;
2795− INIT_WORK(&work->work, cma_work_handler);
2796− work->old_state = RDMA_CM_ADDR_QUERY;
2797− work->new_state = RDMA_CM_ADDR_RESOLVED;
2798− work->event.event = RDMA_CM_EVENT_ADDR_RESOLVED;
2858+ cma_init_resolve_addr_work(work, id_priv);
27992859 queue_work(cma_wq, &work->work);
28002860 return 0;
28012861 err:
@@ -2821,11 +2881,7 @@ static int cma_resolve_ib_addr(struct rdma_id_private *id_priv) · 19 unmodified lines
28212881 rdma_addr_set_dgid(&id_priv->id.route.addr.dev_addr, (union ib_gid *)
28222882 &(((struct sockaddr_ib *) &id_priv->id.route.addr.dst_addr)->sib_addr));
28232883
2824− work->id = id_priv;
2825− INIT_WORK(&work->work, cma_work_handler);
2826− work->old_state = RDMA_CM_ADDR_QUERY;
2827− work->new_state = RDMA_CM_ADDR_RESOLVED;
2828− work->event.event = RDMA_CM_EVENT_ADDR_RESOLVED;
2884+ cma_init_resolve_addr_work(work, id_priv);
28292885 queue_work(cma_wq, &work->work);
28302886 return 0;
28312887 err:
@@ -3404,9 +3460,10 @@ static int cma_sidr_rep_handler(struct ib_cm_id *cm_id, · 572 unmodified lines
34043460 event.status = ret;
34053461 break;
34063462 }
3407− ib_init_ah_from_path(id_priv->id.device, id_priv->id.port_num,
3408− id_priv->id.route.path_rec,
3409− &event.param.ud.ah_attr);
3463+ ib_init_ah_attr_from_path(id_priv->id.device,
3464+ id_priv->id.port_num,
3465+ id_priv->id.route.path_rec,
3466+ &event.param.ud.ah_attr);
34103467 event.param.ud.qp_num = rep->qpn;
34113468 event.param.ud.qkey = rep->qkey;
34123469 event.event = RDMA_CM_EVENT_ESTABLISHED;
@@ -3873,7 +3930,7 @@ static int cma_ib_mc_handler(int status, struct ib_sa_multicast *multicast) · 460 unmodified lines
38733930 struct rdma_dev_addr *dev_addr =
38743931 &id_priv->id.route.addr.dev_addr;
38753932 struct net_device *ndev =
3876− dev_get_by_index(&init_net, dev_addr->bound_dev_if);
3933+ dev_get_by_index(dev_addr->net, dev_addr->bound_dev_if);
38773934 enum ib_gid_type gid_type =
38783935 id_priv->cma_dev->default_gid_type[id_priv->id.port_num -
38793936 rdma_start_port(id_priv->cma_dev->device)];
@@ -4010,8 +4067,10 @@ static void cma_iboe_set_mgid(struct sockaddr *addr, union ib_gid *mgid, · 130 unmodified lines
40104067 } else if (addr->sa_family == AF_INET6) {
40114068 memcpy(mgid, &sin6->sin6_addr, sizeof *mgid);
40124069 } else {
4013− mgid->raw[0] = (gid_type == IB_GID_TYPE_IB) ? 0xff : 0;
4014− mgid->raw[1] = (gid_type == IB_GID_TYPE_IB) ? 0x0e : 0;
4070+ mgid->raw[0] =
4071+ (gid_type == IB_GID_TYPE_ROCE_UDP_ENCAP) ? 0 : 0xff;
4072+ mgid->raw[1] =
4073+ (gid_type == IB_GID_TYPE_ROCE_UDP_ENCAP) ? 0 : 0x0e;
40154074 mgid->raw[2] = 0;
40164075 mgid->raw[3] = 0;
40174076 mgid->raw[4] = 0;
@@ -4061,7 +4120,7 @@ static int cma_iboe_join_multicast(struct rdma_id_private *id_priv, · 43 unmodified lines
40614120 mc->multicast.ib->rec.qkey = cpu_to_be32(RDMA_UDP_QKEY);
40624121
40634122 if (dev_addr->bound_dev_if)
4064− ndev = dev_get_by_index(&init_net, dev_addr->bound_dev_if);
4123+ ndev = dev_get_by_index(dev_addr->net, dev_addr->bound_dev_if);
40654124 if (!ndev) {
40664125 err = -ENODEV;
40674126 goto out2;
@@ -4179,7 +4238,7 @@ void rdma_leave_multicast(struct rdma_cm_id *id, struct sockaddr *addr) · 111 unmodified lines
41794238 struct net_device *ndev = NULL;
41804239
41814240 if (dev_addr->bound_dev_if)
4182− ndev = dev_get_by_index(&init_net,
4241+ ndev = dev_get_by_index(dev_addr->net,
41834242 dev_addr->bound_dev_if);
41844243 if (ndev) {
41854244 cma_igmp_send(ndev,
@@ -4235,7 +4294,7 @@ static int cma_netdev_callback(struct notifier_block *self, unsigned long event, · 49 unmodified lines
42354294 if (event != NETDEV_BONDING_FAILOVER)
42364295 return NOTIFY_DONE;
42374296
4238− if (!(ndev->flags & IFF_MASTER) || !(ndev->priv_flags & IFF_BONDING))
4297+ if (!netif_is_bond_master(ndev))
42394298 return NOTIFY_DONE;
42404299
42414300 mutex_lock(&lock);
@@ -4432,7 +4491,7 @@ static int cma_get_id_stats(struct sk_buff *skb, struct netlink_callback *cb) · 190 unmodified lines
44324491 RDMA_NL_RDMA_CM_ATTR_SRC_ADDR))
44334492 goto out;
44344493 if (ibnl_put_attr(skb, nlh,
4435− rdma_addr_size(cma_src_addr(id_priv)),
4494+ rdma_addr_size(cma_dst_addr(id_priv)),
44364495 cma_dst_addr(id_priv),
44374496 RDMA_NL_RDMA_CM_ATTR_DST_ADDR))
44384497 goto out;
@@ -4444,6 +4503,7 @@ static int cma_get_id_stats(struct sk_buff *skb, struct netlink_callback *cb) · 5 unmodified lines
44444503 id_stats->qp_type = id->qp_type;
44454504
44464505 i_id++;
4506+ nlmsg_end(skb, nlh);
44474507 }
44484508
44494509 cb->args[1] = 0;
drivers/infiniband/core/cma_configfs.c+1−1
@@ -295,7 +295,7 @@ static struct config_group *make_cma_dev(struct config_group *group, · 294 unmodified lines
295295 goto fail;
296296 }
297297
298− strncpy(cma_dev_group->name, name, sizeof(cma_dev_group->name));
298+ strlcpy(cma_dev_group->name, name, sizeof(cma_dev_group->name));
299299
300300 config_group_init_type_name(&cma_dev_group->ports_group, "ports",
301301 &cma_ports_group_type);
drivers/infiniband/core/core_priv.h+31−21
@@ -40,8 +40,12 @@ · 39 unmodified lines
4040 #include <rdma/ib_verbs.h>
4141 #include <rdma/opa_addr.h>
4242 #include <rdma/ib_mad.h>
43+#include <rdma/restrack.h>
4344 #include "mad_priv.h"
4445
46+/* Total number of ports combined across all struct ib_devices's */
47+#define RDMA_MAX_PORTS 1024
48+
4549 struct pkey_index_qp_list {
4650 struct list_head pkey_index_list;
4751 u16 pkey_index;
@@ -137,7 +141,6 @@ int ib_cache_gid_del_all_netdev_gids(struct ib_device *ib_dev, u8 port, · 89 unmodified lines
137141 int roce_gid_mgmt_init(void);
138142 void roce_gid_mgmt_cleanup(void);
139143
140−int roce_rescan_device(struct ib_device *ib_dev);
141144 unsigned long roce_gid_type_mask_support(struct ib_device *ib_dev, u8 port);
142145
143146 int ib_cache_setup_one(struct ib_device *device);
@@ -191,13 +194,6 @@ void ib_sa_cleanup(void); · 47 unmodified lines
191194 int rdma_nl_init(void);
192195 void rdma_nl_exit(void);
193196
194−/**
195− * Check if there are any listeners to the netlink group
196− * @group: the netlink group ID
197− * Returns 0 on success or a negative for no listeners.
198− */
199−int ibnl_chk_listeners(unsigned int group);
200−
201197 int ib_nl_handle_resolve_resp(struct sk_buff *skb,
202198 struct nlmsghdr *nlh,
203199 struct netlink_ext_ack *extack);
@@ -213,11 +209,6 @@ int ib_get_cached_subnet_prefix(struct ib_device *device, · 9 unmodified lines
213209 u64 *sn_pfx);
214210
215211 #ifdef CONFIG_SECURITY_INFINIBAND
216−int ib_security_pkey_access(struct ib_device *dev,
217− u8 port_num,
218− u16 pkey_index,
219− void *sec);
220−
221212 void ib_security_destroy_port_pkey_list(struct ib_device *device);
222213
223214 void ib_security_cache_change(struct ib_device *device,
@@ -240,14 +231,6 @@ int ib_mad_agent_security_setup(struct ib_mad_agent *agent, · 16 unmodified lines
240231 void ib_mad_agent_security_cleanup(struct ib_mad_agent *agent);
241232 int ib_mad_enforce_security(struct ib_mad_agent_private *map, u16 pkey_index);
242233 #else
243−static inline int ib_security_pkey_access(struct ib_device *dev,
244− u8 port_num,
245− u16 pkey_index,
246− void *sec)
247−{
248− return 0;
249−}
250−
251234 static inline void ib_security_destroy_port_pkey_list(struct ib_device *device)
252235 {
253236 }
@@ -318,4 +301,31 @@ struct ib_device *ib_device_get_by_index(u32 ifindex); · 64 unmodified lines
318301 /* RDMA device netlink */
319302 void nldev_init(void);
320303 void nldev_exit(void);
304+
305+static inline struct ib_qp *_ib_create_qp(struct ib_device *dev,
306+ struct ib_pd *pd,
307+ struct ib_qp_init_attr *attr,
308+ struct ib_udata *udata)
309+{
310+ struct ib_qp *qp;
311+
312+ qp = dev->create_qp(pd, attr, udata);
313+ if (IS_ERR(qp))
314+ return qp;
315+
316+ qp->device = dev;
317+ qp->pd = pd;
318+ /*
319+ * We don't track XRC QPs for now, because they don't have PD
320+ * and more importantly they are created internaly by driver,
321+ * see mlx5 create_dev_resources() as an example.
322+ */
323+ if (attr->qp_type < IB_QPT_XRC_INI) {
324+ qp->res.type = RDMA_RESTRACK_QP;
325+ rdma_restrack_add(&qp->res);
326+ } else
327+ qp->res.valid = false;
328+
329+ return qp;
330+}
321331 #endif /* _CORE_PRIV_H */
drivers/infiniband/core/cq.c+25−14
@@ -25,9 +25,10 @@ · 24 unmodified lines
2525 #define IB_POLL_FLAGS \
2626 (IB_CQ_NEXT_COMP | IB_CQ_REPORT_MISSED_EVENTS)
2727
28−static int __ib_process_cq(struct ib_cq *cq, int budget)
28+static int __ib_process_cq(struct ib_cq *cq, int budget, struct ib_wc *poll_wc)
2929 {
3030 int i, n, completed = 0;
31+ struct ib_wc *wcs = poll_wc ? : cq->wc;
3132
3233 /*
3334 * budget might be (-1) if the caller does not
@@ -35,9 +36,9 @@ static int __ib_process_cq(struct ib_cq *cq, int budget) · 1 unmodified line
3536 * minimum here.
3637 */
3738 while ((n = ib_poll_cq(cq, min_t(u32, IB_POLL_BATCH,
38− budget - completed), cq->wc)) > 0) {
39+ budget - completed), wcs)) > 0) {
3940 for (i = 0; i < n; i++) {
40− struct ib_wc *wc = &cq->wc[i];
41+ struct ib_wc *wc = &wcs[i];
4142
4243 if (wc->wr_cqe)
4344 wc->wr_cqe->done(cq, wc);
@@ -60,18 +61,20 @@ static int __ib_process_cq(struct ib_cq *cq, int budget) · 16 unmodified lines
6061 * @cq: CQ to process
6162 * @budget: number of CQEs to poll for
6263 *
63− * This function is used to process all outstanding CQ entries on a
64− * %IB_POLL_DIRECT CQ. It does not offload CQ processing to a different
65− * context and does not ask for completion interrupts from the HCA.
64+ * This function is used to process all outstanding CQ entries.
65+ * It does not offload CQ processing to a different context and does
66+ * not ask for completion interrupts from the HCA.
67+ * Using direct processing on CQ with non IB_POLL_DIRECT type may trigger
68+ * concurrent processing.
6669 *
6770 * Note: do not pass -1 as %budget unless it is guaranteed that the number
6871 * of completions that will be processed is small.
6972 */
7073 int ib_process_cq_direct(struct ib_cq *cq, int budget)
7174 {
72− WARN_ON_ONCE(cq->poll_ctx != IB_POLL_DIRECT);
75+ struct ib_wc wcs[IB_POLL_BATCH];
7376
74− return __ib_process_cq(cq, budget);
77+ return __ib_process_cq(cq, budget, wcs);
7578 }
7679 EXPORT_SYMBOL(ib_process_cq_direct);
7780
@@ -85,7 +88,7 @@ static int ib_poll_handler(struct irq_poll *iop, int budget) · 7 unmodified lines
8588 struct ib_cq *cq = container_of(iop, struct ib_cq, iop);
8689 int completed;
8790
88− completed = __ib_process_cq(cq, budget);
91+ completed = __ib_process_cq(cq, budget, NULL);
8992 if (completed < budget) {
9093 irq_poll_complete(&cq->iop);
9194 if (ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0)
@@ -105,7 +108,7 @@ static void ib_cq_poll_work(struct work_struct *work) · 13 unmodified lines
105108 struct ib_cq *cq = container_of(work, struct ib_cq, work);
106109 int completed;
107110
108− completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE);
111+ completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE, NULL);
109112 if (completed >= IB_POLL_BUDGET_WORKQUEUE ||
110113 ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0)
111114 queue_work(ib_comp_wq, &cq->work);
@@ -117,20 +120,22 @@ static void ib_cq_completion_workqueue(struct ib_cq *cq, void *private) · 5 unmodified lines
117120 }
118121
119122 /**
120− * ib_alloc_cq - allocate a completion queue
123+ * __ib_alloc_cq - allocate a completion queue
121124 * @dev: device to allocate the CQ for
122125 * @private: driver private data, accessible from cq->cq_context
123126 * @nr_cqe: number of CQEs to allocate
124127 * @comp_vector: HCA completion vectors for this CQ
125128 * @poll_ctx: context to poll the CQ from.
129+ * @caller: module owner name.
126130 *
127131 * This is the proper interface to allocate a CQ for in-kernel users. A
128132 * CQ allocated with this interface will automatically be polled from the
129133 * specified context. The ULP must use wr->wr_cqe instead of wr->wr_id
130134 * to use this CQ abstraction.
131135 */
132−struct ib_cq *ib_alloc_cq(struct ib_device *dev, void *private,
133− int nr_cqe, int comp_vector, enum ib_poll_context poll_ctx)
136+struct ib_cq *__ib_alloc_cq(struct ib_device *dev, void *private,
137+ int nr_cqe, int comp_vector,
138+ enum ib_poll_context poll_ctx, const char *caller)
134139 {
135140 struct ib_cq_init_attr cq_attr = {
136141 .cqe = nr_cqe,
@@ -154,6 +159,10 @@ struct ib_cq *ib_alloc_cq(struct ib_device *dev, void *private, · 17 unmodified lines
154159 if (!cq->wc)
155160 goto out_destroy_cq;
156161
162+ cq->res.type = RDMA_RESTRACK_CQ;
163+ cq->res.kern_name = caller;
164+ rdma_restrack_add(&cq->res);
165+
157166 switch (cq->poll_ctx) {
158167 case IB_POLL_DIRECT:
159168 cq->comp_handler = ib_cq_completion_direct;
@@ -178,11 +187,12 @@ struct ib_cq *ib_alloc_cq(struct ib_device *dev, void *private, · 18 unmodified lines
178187
179188 out_free_wc:
180189 kfree(cq->wc);
190+ rdma_restrack_del(&cq->res);
181191 out_destroy_cq:
182192 cq->device->destroy_cq(cq);
183193 return ERR_PTR(ret);
184194 }
185−EXPORT_SYMBOL(ib_alloc_cq);
195+EXPORT_SYMBOL(__ib_alloc_cq);
186196
187197 /**
188198 * ib_free_cq - free a completion queue
@@ -209,6 +219,7 @@ void ib_free_cq(struct ib_cq *cq) · 20 unmodified lines
209219 }
210220
211221 kfree(cq->wc);
222+ rdma_restrack_del(&cq->res);
212223 ret = cq->device->destroy_cq(cq);
213224 WARN_ON_ONCE(ret);
214225 }
drivers/infiniband/core/device.c+22−20
@@ -263,6 +263,8 @@ struct ib_device *ib_alloc_device(size_t size) · 262 unmodified lines
263263 if (!device)
264264 return NULL;
265265
266+ rdma_restrack_init(&device->res);
267+
266268 device->dev.class = &ib_class;
267269 device_initialize(&device->dev);
268270
@@ -288,7 +290,7 @@ void ib_dealloc_device(struct ib_device *device) · 19 unmodified lines
288290 {
289291 WARN_ON(device->reg_state != IB_DEV_UNREGISTERED &&
290292 device->reg_state != IB_DEV_UNINITIALIZED);
291− kobject_put(&device->dev.kobj);
293+ put_device(&device->dev);
292294 }
293295 EXPORT_SYMBOL(ib_dealloc_device);
294296
@@ -462,7 +464,6 @@ int ib_register_device(struct ib_device *device, · 167 unmodified lines
462464 struct ib_udata uhw = {.outlen = 0, .inlen = 0};
463465 struct device *parent = device->dev.parent;
464466
465− WARN_ON_ONCE(!parent);
466467 WARN_ON_ONCE(device->dma_device);
467468 if (device->dev.dma_ops) {
468469 /*
@@ -471,16 +472,25 @@ int ib_register_device(struct ib_device *device, · 2 unmodified lines
471472 * into device->dev.
472473 */
473474 device->dma_device = &device->dev;
474− if (!device->dev.dma_mask)
475− device->dev.dma_mask = parent->dma_mask;
476− if (!device->dev.coherent_dma_mask)
477− device->dev.coherent_dma_mask =
478− parent->coherent_dma_mask;
475+ if (!device->dev.dma_mask) {
476+ if (parent)
477+ device->dev.dma_mask = parent->dma_mask;
478+ else
479+ WARN_ON_ONCE(true);
480+ }
481+ if (!device->dev.coherent_dma_mask) {
482+ if (parent)
483+ device->dev.coherent_dma_mask =
484+ parent->coherent_dma_mask;
485+ else
486+ WARN_ON_ONCE(true);
487+ }
479488 } else {
480489 /*
481490 * The caller did not provide custom DMA operations. Use the
482491 * DMA mapping operations of the parent device.
483492 */
493+ WARN_ON_ONCE(!parent);
484494 device->dma_device = parent;
485495 }
486496
@@ -588,6 +598,8 @@ void ib_unregister_device(struct ib_device *device) · 101 unmodified lines
588598 }
589599 up_read(&lists_rwsem);
590600
601+ rdma_restrack_clean(&device->res);
602+
591603 ib_device_unregister_rdmacg(device);
592604 ib_device_unregister_sysfs(device);
593605
@@ -1033,32 +1045,22 @@ EXPORT_SYMBOL(ib_modify_port); · 439 unmodified lines
10331045
10341046 /**
10351047 * ib_find_gid - Returns the port number and GID table index where
1036− * a specified GID value occurs.
1048+ * a specified GID value occurs. Its searches only for IB link layer.
10371049 * @device: The device to query.
10381050 * @gid: The GID value to search for.
1039− * @gid_type: Type of GID.
10401051 * @ndev: The ndev related to the GID to search for.
10411052 * @port_num: The port number of the device where the GID value was found.
10421053 * @index: The index into the GID table where the GID was found. This
10431054 * parameter may be NULL.
10441055 */
10451056 int ib_find_gid(struct ib_device *device, union ib_gid *gid,
1046− enum ib_gid_type gid_type, struct net_device *ndev,
1047− u8 *port_num, u16 *index)
1057+ struct net_device *ndev, u8 *port_num, u16 *index)
10481058 {
10491059 union ib_gid tmp_gid;
10501060 int ret, port, i;
10511061
10521062 for (port = rdma_start_port(device); port <= rdma_end_port(device); ++port) {
1053− if (rdma_cap_roce_gid_table(device, port)) {
1054− if (!ib_find_cached_gid_by_port(device, gid, gid_type, port,
1055− ndev, index)) {
1056− *port_num = port;
1057− return 0;
1058− }
1059− }
1060−
1061− if (gid_type != IB_GID_TYPE_IB)
1063+ if (rdma_cap_roce_gid_table(device, port))
10621064 continue;
10631065
10641066 for (i = 0; i < device->port_immutable[port].gid_tbl_len; ++i) {
drivers/infiniband/core/fmr_pool.c+5−7
@@ -388,13 +388,11 @@ int ib_flush_fmr_pool(struct ib_fmr_pool *pool) · 387 unmodified lines
388388 EXPORT_SYMBOL(ib_flush_fmr_pool);
389389
390390 /**
391− * ib_fmr_pool_map_phys -
392− * @pool:FMR pool to allocate FMR from
393− * @page_list:List of pages to map
394− * @list_len:Number of pages in @page_list
395− * @io_virtual_address:I/O virtual address for new FMR
396− *
397− * Map an FMR from an FMR pool.
391+ * ib_fmr_pool_map_phys - Map an FMR from an FMR pool.
392+ * @pool_handle: FMR pool to allocate FMR from
393+ * @page_list: List of pages to map
394+ * @list_len: Number of pages in @page_list
395+ * @io_virtual_address: I/O virtual address for new FMR
398396 */
399397 struct ib_pool_fmr *ib_fmr_pool_map_phys(struct ib_fmr_pool *pool_handle,
400398 u64 *page_list,
drivers/infiniband/core/iwpm_util.c+1−0
@@ -654,6 +654,7 @@ int iwpm_send_mapinfo(u8 nl_client, int iwpm_pid) · 653 unmodified lines
654654 }
655655 skb_num++;
656656 spin_lock_irqsave(&iwpm_mapinfo_lock, flags);
657+ ret = -EINVAL;
657658 for (i = 0; i < IWPM_MAPINFO_HASH_SIZE; i++) {
658659 hlist_for_each_entry(map_info, &iwpm_hash_bucket[i],
659660 hlist_node) {
drivers/infiniband/core/mad.c+0−1
@@ -49,7 +49,6 @@ · 48 unmodified lines
4949 #include "smi.h"
5050 #include "opa_smi.h"
5151 #include "agent.h"
52−#include "core_priv.h"
5352
5453 static int mad_sendq_size = IB_MAD_QP_SEND_SIZE;
5554 static int mad_recvq_size = IB_MAD_QP_RECV_SIZE;
drivers/infiniband/core/netlink.c+3−7
@@ -41,8 +41,6 @@ · 40 unmodified lines
4141 #include <linux/module.h>
4242 #include "core_priv.h"
4343
44−#include "core_priv.h"
45−
4644 static DEFINE_MUTEX(rdma_nl_mutex);
4745 static struct sock *nls;
4846 static struct {
@@ -83,15 +81,13 @@ static bool is_nl_valid(unsigned int type, unsigned int op) · 34 unmodified lines
8381 if (!is_nl_msg_valid(type, op))
8482 return false;
8583
86− cb_table = rdma_nl_types[type].cb_table;
87−#ifdef CONFIG_MODULES
88− if (!cb_table) {
84+ if (!rdma_nl_types[type].cb_table) {
8985 mutex_unlock(&rdma_nl_mutex);
9086 request_module("rdma-netlink-subsys-%d", type);
9187 mutex_lock(&rdma_nl_mutex);
92− cb_table = rdma_nl_types[type].cb_table;
9388 }
94−#endif
89+
90+ cb_table = rdma_nl_types[type].cb_table;
9591
9692 if (!cb_table || (!cb_table[op].dump && !cb_table[op].doit))
9793 return false;
drivers/infiniband/core/nldev.c+388−6
@@ -31,6 +31,8 @@ · 30 unmodified lines
3131 */
3232
3333 #include <linux/module.h>
34+#include <linux/pid.h>
35+#include <linux/pid_namespace.h>
3436 #include <net/netlink.h>
3537 #include <rdma/rdma_netlink.h>
3638
@@ -52,16 +54,42 @@ static const struct nla_policy nldev_policy[RDMA_NLDEV_ATTR_MAX] = { · 15 unmodified lines
5254 [RDMA_NLDEV_ATTR_PORT_STATE] = { .type = NLA_U8 },
5355 [RDMA_NLDEV_ATTR_PORT_PHYS_STATE] = { .type = NLA_U8 },
5456 [RDMA_NLDEV_ATTR_DEV_NODE_TYPE] = { .type = NLA_U8 },
57+ [RDMA_NLDEV_ATTR_RES_SUMMARY] = { .type = NLA_NESTED },
58+ [RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY] = { .type = NLA_NESTED },
59+ [RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_NAME] = { .type = NLA_NUL_STRING,
60+ .len = 16 },
61+ [RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_CURR] = { .type = NLA_U64 },
62+ [RDMA_NLDEV_ATTR_RES_QP] = { .type = NLA_NESTED },
63+ [RDMA_NLDEV_ATTR_RES_QP_ENTRY] = { .type = NLA_NESTED },
64+ [RDMA_NLDEV_ATTR_RES_LQPN] = { .type = NLA_U32 },
65+ [RDMA_NLDEV_ATTR_RES_RQPN] = { .type = NLA_U32 },
66+ [RDMA_NLDEV_ATTR_RES_RQ_PSN] = { .type = NLA_U32 },
67+ [RDMA_NLDEV_ATTR_RES_SQ_PSN] = { .type = NLA_U32 },
68+ [RDMA_NLDEV_ATTR_RES_PATH_MIG_STATE] = { .type = NLA_U8 },
69+ [RDMA_NLDEV_ATTR_RES_TYPE] = { .type = NLA_U8 },
70+ [RDMA_NLDEV_ATTR_RES_STATE] = { .type = NLA_U8 },
71+ [RDMA_NLDEV_ATTR_RES_PID] = { .type = NLA_U32 },
72+ [RDMA_NLDEV_ATTR_RES_KERN_NAME] = { .type = NLA_NUL_STRING,
73+ .len = TASK_COMM_LEN },
5574 };
5675
57−static int fill_dev_info(struct sk_buff *msg, struct ib_device *device)
76+static int fill_nldev_handle(struct sk_buff *msg, struct ib_device *device)
5877 {
59− char fw[IB_FW_VERSION_NAME_MAX];
60−
6178 if (nla_put_u32(msg, RDMA_NLDEV_ATTR_DEV_INDEX, device->index))
6279 return -EMSGSIZE;
6380 if (nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_NAME, device->name))
6481 return -EMSGSIZE;
82+
83+ return 0;
84+}
85+
86+static int fill_dev_info(struct sk_buff *msg, struct ib_device *device)
87+{
88+ char fw[IB_FW_VERSION_NAME_MAX];
89+
90+ if (fill_nldev_handle(msg, device))
91+ return -EMSGSIZE;
92+
6593 if (nla_put_u32(msg, RDMA_NLDEV_ATTR_PORT_INDEX, rdma_end_port(device)))
6694 return -EMSGSIZE;
6795
@@ -92,10 +120,9 @@ static int fill_port_info(struct sk_buff *msg, · 24 unmodified lines
92120 struct ib_port_attr attr;
93121 int ret;
94122
95− if (nla_put_u32(msg, RDMA_NLDEV_ATTR_DEV_INDEX, device->index))
96− return -EMSGSIZE;
97− if (nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_NAME, device->name))
123+ if (fill_nldev_handle(msg, device))
98124 return -EMSGSIZE;
125+
99126 if (nla_put_u32(msg, RDMA_NLDEV_ATTR_PORT_INDEX, port))
100127 return -EMSGSIZE;
101128
@@ -126,6 +153,137 @@ static int fill_port_info(struct sk_buff *msg, · 24 unmodified lines
126153 return 0;
127154 }
128155
156+static int fill_res_info_entry(struct sk_buff *msg,
157+ const char *name, u64 curr)
158+{
159+ struct nlattr *entry_attr;
160+
161+ entry_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY);
162+ if (!entry_attr)
163+ return -EMSGSIZE;
164+
165+ if (nla_put_string(msg, RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_NAME, name))
166+ goto err;
167+ if (nla_put_u64_64bit(msg,
168+ RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_CURR, curr, 0))
169+ goto err;
170+
171+ nla_nest_end(msg, entry_attr);
172+ return 0;
173+
174+err:
175+ nla_nest_cancel(msg, entry_attr);
176+ return -EMSGSIZE;
177+}
178+
179+static int fill_res_info(struct sk_buff *msg, struct ib_device *device)
180+{
181+ static const char * const names[RDMA_RESTRACK_MAX] = {
182+ [RDMA_RESTRACK_PD] = "pd",
183+ [RDMA_RESTRACK_CQ] = "cq",
184+ [RDMA_RESTRACK_QP] = "qp",
185+ };
186+
187+ struct rdma_restrack_root *res = &device->res;
188+ struct nlattr *table_attr;
189+ int ret, i, curr;
190+
191+ if (fill_nldev_handle(msg, device))
192+ return -EMSGSIZE;
193+
194+ table_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_RES_SUMMARY);
195+ if (!table_attr)
196+ return -EMSGSIZE;
197+
198+ for (i = 0; i < RDMA_RESTRACK_MAX; i++) {
199+ if (!names[i])
200+ continue;
201+ curr = rdma_restrack_count(res, i, task_active_pid_ns(current));
202+ ret = fill_res_info_entry(msg, names[i], curr);
203+ if (ret)
204+ goto err;
205+ }
206+
207+ nla_nest_end(msg, table_attr);
208+ return 0;
209+
210+err:
211+ nla_nest_cancel(msg, table_attr);
212+ return ret;
213+}
214+
215+static int fill_res_qp_entry(struct sk_buff *msg,
216+ struct ib_qp *qp, uint32_t port)
217+{
218+ struct rdma_restrack_entry *res = &qp->res;
219+ struct ib_qp_init_attr qp_init_attr;
220+ struct nlattr *entry_attr;
221+ struct ib_qp_attr qp_attr;
222+ int ret;
223+
224+ ret = ib_query_qp(qp, &qp_attr, 0, &qp_init_attr);
225+ if (ret)
226+ return ret;
227+
228+ if (port && port != qp_attr.port_num)
229+ return 0;
230+
231+ entry_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_RES_QP_ENTRY);
232+ if (!entry_attr)
233+ goto out;
234+
235+ /* In create_qp() port is not set yet */
236+ if (qp_attr.port_num &&
237+ nla_put_u32(msg, RDMA_NLDEV_ATTR_PORT_INDEX, qp_attr.port_num))
238+ goto err;
239+
240+ if (nla_put_u32(msg, RDMA_NLDEV_ATTR_RES_LQPN, qp->qp_num))
241+ goto err;
242+ if (qp->qp_type == IB_QPT_RC || qp->qp_type == IB_QPT_UC) {
243+ if (nla_put_u32(msg, RDMA_NLDEV_ATTR_RES_RQPN,
244+ qp_attr.dest_qp_num))
245+ goto err;
246+ if (nla_put_u32(msg, RDMA_NLDEV_ATTR_RES_RQ_PSN,
247+ qp_attr.rq_psn))
248+ goto err;
249+ }
250+
251+ if (nla_put_u32(msg, RDMA_NLDEV_ATTR_RES_SQ_PSN, qp_attr.sq_psn))
252+ goto err;
253+
254+ if (qp->qp_type == IB_QPT_RC || qp->qp_type == IB_QPT_UC ||
255+ qp->qp_type == IB_QPT_XRC_INI || qp->qp_type == IB_QPT_XRC_TGT) {
256+ if (nla_put_u8(msg, RDMA_NLDEV_ATTR_RES_PATH_MIG_STATE,
257+ qp_attr.path_mig_state))
258+ goto err;
259+ }
260+ if (nla_put_u8(msg, RDMA_NLDEV_ATTR_RES_TYPE, qp->qp_type))
261+ goto err;
262+ if (nla_put_u8(msg, RDMA_NLDEV_ATTR_RES_STATE, qp_attr.qp_state))
263+ goto err;
264+
265+ /*
266+ * Existence of task means that it is user QP and netlink
267+ * user is invited to go and read /proc/PID/comm to get name
268+ * of the task file and res->task_com should be NULL.
269+ */
270+ if (rdma_is_kernel_res(res)) {
271+ if (nla_put_string(msg, RDMA_NLDEV_ATTR_RES_KERN_NAME, res->kern_name))
272+ goto err;
273+ } else {
274+ if (nla_put_u32(msg, RDMA_NLDEV_ATTR_RES_PID, task_pid_vnr(res->task)))
275+ goto err;
276+ }
277+
278+ nla_nest_end(msg, entry_attr);
279+ return 0;
280+
281+err:
282+ nla_nest_cancel(msg, entry_attr);
283+out:
284+ return -EMSGSIZE;
285+}
286+
129287 static int nldev_get_doit(struct sk_buff *skb, struct nlmsghdr *nlh,
130288 struct netlink_ext_ack *extack)
131289 {
@@ -321,6 +479,213 @@ out: · 189 unmodified lines
321479 return skb->len;
322480 }
323481
482+static int nldev_res_get_doit(struct sk_buff *skb, struct nlmsghdr *nlh,
483+ struct netlink_ext_ack *extack)
484+{
485+ struct nlattr *tb[RDMA_NLDEV_ATTR_MAX];
486+ struct ib_device *device;
487+ struct sk_buff *msg;
488+ u32 index;
489+ int ret;
490+
491+ ret = nlmsg_parse(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1,
492+ nldev_policy, extack);
493+ if (ret || !tb[RDMA_NLDEV_ATTR_DEV_INDEX])
494+ return -EINVAL;
495+
496+ index = nla_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
497+ device = ib_device_get_by_index(index);
498+ if (!device)
499+ return -EINVAL;
500+
501+ msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
502+ if (!msg)
503+ goto err;
504+
505+ nlh = nlmsg_put(msg, NETLINK_CB(skb).portid, nlh->nlmsg_seq,
506+ RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_RES_GET),
507+ 0, 0);
508+
509+ ret = fill_res_info(msg, device);
510+ if (ret)
511+ goto err_free;
512+
513+ nlmsg_end(msg, nlh);
514+ put_device(&device->dev);
515+ return rdma_nl_unicast(msg, NETLINK_CB(skb).portid);
516+
517+err_free:
518+ nlmsg_free(msg);
519+err:
520+ put_device(&device->dev);
521+ return ret;
522+}
523+
524+static int _nldev_res_get_dumpit(struct ib_device *device,
525+ struct sk_buff *skb,
526+ struct netlink_callback *cb,
527+ unsigned int idx)
528+{
529+ int start = cb->args[0];
530+ struct nlmsghdr *nlh;
531+
532+ if (idx < start)
533+ return 0;
534+
535+ nlh = nlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq,
536+ RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_RES_GET),
537+ 0, NLM_F_MULTI);
538+
539+ if (fill_res_info(skb, device)) {
540+ nlmsg_cancel(skb, nlh);
541+ goto out;
542+ }
543+
544+ nlmsg_end(skb, nlh);
545+
546+ idx++;
547+
548+out:
549+ cb->args[0] = idx;
550+ return skb->len;
551+}
552+
553+static int nldev_res_get_dumpit(struct sk_buff *skb,
554+ struct netlink_callback *cb)
555+{
556+ return ib_enum_all_devs(_nldev_res_get_dumpit, skb, cb);
557+}
558+
559+static int nldev_res_get_qp_dumpit(struct sk_buff *skb,
560+ struct netlink_callback *cb)
561+{
562+ struct nlattr *tb[RDMA_NLDEV_ATTR_MAX];
563+ struct rdma_restrack_entry *res;
564+ int err, ret = 0, idx = 0;
565+ struct nlattr *table_attr;
566+ struct ib_device *device;
567+ int start = cb->args[0];
568+ struct ib_qp *qp = NULL;
569+ struct nlmsghdr *nlh;
570+ u32 index, port = 0;
571+
572+ err = nlmsg_parse(cb->nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1,
573+ nldev_policy, NULL);
574+ /*
575+ * Right now, we are expecting the device index to get QP information,
576+ * but it is possible to extend this code to return all devices in
577+ * one shot by checking the existence of RDMA_NLDEV_ATTR_DEV_INDEX.
578+ * if it doesn't exist, we will iterate over all devices.
579+ *
580+ * But it is not needed for now.
581+ */
582+ if (err || !tb[RDMA_NLDEV_ATTR_DEV_INDEX])
583+ return -EINVAL;
584+
585+ index = nla_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
586+ device = ib_device_get_by_index(index);
587+ if (!device)
588+ return -EINVAL;
589+
590+ /*
591+ * If no PORT_INDEX is supplied, we will return all QPs from that device
592+ */
593+ if (tb[RDMA_NLDEV_ATTR_PORT_INDEX]) {
594+ port = nla_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]);
595+ if (!rdma_is_port_valid(device, port)) {
596+ ret = -EINVAL;
597+ goto err_index;
598+ }
599+ }
600+
601+ nlh = nlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq,
602+ RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_RES_QP_GET),
603+ 0, NLM_F_MULTI);
604+
605+ if (fill_nldev_handle(skb, device)) {
606+ ret = -EMSGSIZE;
607+ goto err;
608+ }
609+
610+ table_attr = nla_nest_start(skb, RDMA_NLDEV_ATTR_RES_QP);
611+ if (!table_attr) {
612+ ret = -EMSGSIZE;
613+ goto err;
614+ }
615+
616+ down_read(&device->res.rwsem);
617+ hash_for_each_possible(device->res.hash, res, node, RDMA_RESTRACK_QP) {
618+ if (idx < start)
619+ goto next;
620+
621+ if ((rdma_is_kernel_res(res) &&
622+ task_active_pid_ns(current) != &init_pid_ns) ||
623+ (!rdma_is_kernel_res(res) &&
624+ task_active_pid_ns(current) != task_active_pid_ns(res->task)))
625+ /*
626+ * 1. Kernel QPs should be visible in init namspace only
627+ * 2. Present only QPs visible in the current namespace
628+ */
629+ goto next;
630+
631+ if (!rdma_restrack_get(res))
632+ /*
633+ * Resource is under release now, but we are not
634+ * relesing lock now, so it will be released in
635+ * our next pass, once we will get ->next pointer.
636+ */
637+ goto next;
638+
639+ qp = container_of(res, struct ib_qp, res);
640+
641+ up_read(&device->res.rwsem);
642+ ret = fill_res_qp_entry(skb, qp, port);
643+ down_read(&device->res.rwsem);
644+ /*
645+ * Return resource back, but it won't be released till
646+ * the &device->res.rwsem will be released for write.
647+ */
648+ rdma_restrack_put(res);
649+
650+ if (ret == -EMSGSIZE)
651+ /*
652+ * There is a chance to optimize here.
653+ * It can be done by using list_prepare_entry
654+ * and list_for_each_entry_continue afterwards.
655+ */
656+ break;
657+ if (ret)
658+ goto res_err;
659+next: idx++;
660+ }
661+ up_read(&device->res.rwsem);
662+
663+ nla_nest_end(skb, table_attr);
664+ nlmsg_end(skb, nlh);
665+ cb->args[0] = idx;
666+
667+ /*
668+ * No more QPs to fill, cancel the message and
669+ * return 0 to mark end of dumpit.
670+ */
671+ if (!qp)
672+ goto err;
673+
674+ put_device(&device->dev);
675+ return skb->len;
676+
677+res_err:
678+ nla_nest_cancel(skb, table_attr);
679+ up_read(&device->res.rwsem);
680+
681+err:
682+ nlmsg_cancel(skb, nlh);
683+
684+err_index:
685+ put_device(&device->dev);
686+ return ret;
687+}
688+
324689 static const struct rdma_nl_cbs nldev_cb_table[RDMA_NLDEV_NUM_OPS] = {
325690 [RDMA_NLDEV_CMD_GET] = {
326691 .doit = nldev_get_doit,
@@ -330,6 +695,23 @@ static const struct rdma_nl_cbs nldev_cb_table[RDMA_NLDEV_NUM_OPS] = { · 3 unmodified lines
330695 .doit = nldev_port_get_doit,
331696 .dump = nldev_port_get_dumpit,
332697 },
698+ [RDMA_NLDEV_CMD_RES_GET] = {
699+ .doit = nldev_res_get_doit,
700+ .dump = nldev_res_get_dumpit,
701+ },
702+ [RDMA_NLDEV_CMD_RES_QP_GET] = {
703+ .dump = nldev_res_get_qp_dumpit,
704+ /*
705+ * .doit is not implemented yet for two reasons:
706+ * 1. It is not needed yet.
707+ * 2. There is a need to provide identifier, while it is easy
708+ * for the QPs (device index + port index + LQPN), it is not
709+ * the case for the rest of resources (PD and CQ). Because it
710+ * is better to provide similar interface for all resources,
711+ * let's wait till we will have other resources implemented
712+ * too.
713+ */
714+ },
333715 };
334716
335717 void __init nldev_init(void)
drivers/infiniband/core/restrack.c+164−0
@@ -0,0 +1,164 @@
1+/* SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause) */
2+/*
3+ * Copyright (c) 2017-2018 Mellanox Technologies. All rights reserved.
4+ */
5+
6+#include <rdma/ib_verbs.h>
7+#include <rdma/restrack.h>
8+#include <linux/mutex.h>
9+#include <linux/sched/task.h>
10+#include <linux/uaccess.h>
11+#include <linux/pid_namespace.h>
12+
13+void rdma_restrack_init(struct rdma_restrack_root *res)
14+{
15+ init_rwsem(&res->rwsem);
16+}
17+
18+void rdma_restrack_clean(struct rdma_restrack_root *res)
19+{
20+ WARN_ON_ONCE(!hash_empty(res->hash));
21+}
22+
23+int rdma_restrack_count(struct rdma_restrack_root *res,
24+ enum rdma_restrack_type type,
25+ struct pid_namespace *ns)
26+{
27+ struct rdma_restrack_entry *e;
28+ u32 cnt = 0;
29+
30+ down_read(&res->rwsem);
31+ hash_for_each_possible(res->hash, e, node, type) {
32+ if (ns == &init_pid_ns ||
33+ (!rdma_is_kernel_res(e) &&
34+ ns == task_active_pid_ns(e->task)))
35+ cnt++;
36+ }
37+ up_read(&res->rwsem);
38+ return cnt;
39+}
40+EXPORT_SYMBOL(rdma_restrack_count);
41+
42+static void set_kern_name(struct rdma_restrack_entry *res)
43+{
44+ enum rdma_restrack_type type = res->type;
45+ struct ib_qp *qp;
46+
47+ if (type != RDMA_RESTRACK_QP)
48+ /* PD and CQ types already have this name embedded in */
49+ return;
50+
51+ qp = container_of(res, struct ib_qp, res);
52+ if (!qp->pd) {
53+ WARN_ONCE(true, "XRC QPs are not supported\n");
54+ /* Survive, despite the programmer's error */
55+ res->kern_name = " ";
56+ return;
57+ }
58+
59+ res->kern_name = qp->pd->res.kern_name;
60+}
61+
62+static struct ib_device *res_to_dev(struct rdma_restrack_entry *res)
63+{
64+ enum rdma_restrack_type type = res->type;
65+ struct ib_device *dev;
66+ struct ib_xrcd *xrcd;
67+ struct ib_pd *pd;
68+ struct ib_cq *cq;
69+ struct ib_qp *qp;
70+
71+ switch (type) {
72+ case RDMA_RESTRACK_PD:
73+ pd = container_of(res, struct ib_pd, res);
74+ dev = pd->device;
75+ break;
76+ case RDMA_RESTRACK_CQ:
77+ cq = container_of(res, struct ib_cq, res);
78+ dev = cq->device;
79+ break;
80+ case RDMA_RESTRACK_QP:
81+ qp = container_of(res, struct ib_qp, res);
82+ dev = qp->device;
83+ break;
84+ case RDMA_RESTRACK_XRCD:
85+ xrcd = container_of(res, struct ib_xrcd, res);
86+ dev = xrcd->device;
87+ break;
88+ default:
89+ WARN_ONCE(true, "Wrong resource tracking type %u\n", type);
90+ return NULL;
91+ }
92+
93+ return dev;
94+}
95+
96+void rdma_restrack_add(struct rdma_restrack_entry *res)
97+{
98+ struct ib_device *dev = res_to_dev(res);
99+
100+ if (!dev)
101+ return;
102+
103+ if (!uaccess_kernel()) {
104+ get_task_struct(current);
105+ res->task = current;
106+ res->kern_name = NULL;
107+ } else {
108+ set_kern_name(res);
109+ res->task = NULL;
110+ }
111+
112+ kref_init(&res->kref);
113+ init_completion(&res->comp);
114+ res->valid = true;
115+
116+ down_write(&dev->res.rwsem);
117+ hash_add(dev->res.hash, &res->node, res->type);
118+ up_write(&dev->res.rwsem);
119+}
120+EXPORT_SYMBOL(rdma_restrack_add);
121+
122+int __must_check rdma_restrack_get(struct rdma_restrack_entry *res)
123+{
124+ return kref_get_unless_zero(&res->kref);
125+}
126+EXPORT_SYMBOL(rdma_restrack_get);
127+
128+static void restrack_release(struct kref *kref)
129+{
130+ struct rdma_restrack_entry *res;
131+
132+ res = container_of(kref, struct rdma_restrack_entry, kref);
133+ complete(&res->comp);
134+}
135+
136+int rdma_restrack_put(struct rdma_restrack_entry *res)
137+{
138+ return kref_put(&res->kref, restrack_release);
139+}
140+EXPORT_SYMBOL(rdma_restrack_put);
141+
142+void rdma_restrack_del(struct rdma_restrack_entry *res)
143+{
144+ struct ib_device *dev;
145+
146+ if (!res->valid)
147+ return;
148+
149+ dev = res_to_dev(res);
150+ if (!dev)
151+ return;
152+
153+ rdma_restrack_put(res);
154+
155+ wait_for_completion(&res->comp);
156+
157+ down_write(&dev->res.rwsem);
158+ hash_del(&res->node);
159+ res->valid = false;
160+ if (res->task)
161+ put_task_struct(res->task);
162+ up_write(&dev->res.rwsem);
163+}
164+EXPORT_SYMBOL(rdma_restrack_del);
drivers/infiniband/core/roce_gid_mgmt.c+8−5
@@ -410,15 +410,18 @@ static void enum_all_gids_of_dev_cb(struct ib_device *ib_dev, · 409 unmodified lines
410410 rtnl_unlock();
411411 }
412412
413−/* This function will rescan all of the network devices in the system
414− * and add their gids, as needed, to the relevant RoCE devices. */
415−int roce_rescan_device(struct ib_device *ib_dev)
413+/**
414+ * rdma_roce_rescan_device - Rescan all of the network devices in the system
415+ * and add their gids, as needed, to the relevant RoCE devices.
416+ *
417+ * @device: the rdma device
418+ */
419+void rdma_roce_rescan_device(struct ib_device *ib_dev)
416420 {
417421 ib_enum_roce_netdev(ib_dev, pass_all_filter, NULL,
418422 enum_all_gids_of_dev_cb, NULL);
419−
420− return 0;
421423 }
424+EXPORT_SYMBOL(rdma_roce_rescan_device);
422425
423426 static void callback_for_addr_gid_device_scan(struct ib_device *device,
424427 u8 port,
drivers/infiniband/core/sa_query.c+14−4
@@ -1227,9 +1227,9 @@ static u8 get_src_path_mask(struct ib_device *device, u8 port_num) · 1226 unmodified lines
12271227 return src_path_mask;
12281228 }
12291229
1230−int ib_init_ah_from_path(struct ib_device *device, u8 port_num,
1231− struct sa_path_rec *rec,
1232− struct rdma_ah_attr *ah_attr)
1230+int ib_init_ah_attr_from_path(struct ib_device *device, u8 port_num,
1231+ struct sa_path_rec *rec,
1232+ struct rdma_ah_attr *ah_attr)
12331233 {
12341234 int ret;
12351235 u16 gid_index;
@@ -1341,10 +1341,11 @@ int ib_init_ah_from_path(struct ib_device *device, u8 port_num, · 105 unmodified lines
13411341
13421342 return 0;
13431343 }
1344−EXPORT_SYMBOL(ib_init_ah_from_path);
1344+EXPORT_SYMBOL(ib_init_ah_attr_from_path);
13451345
13461346 static int alloc_mad(struct ib_sa_query *query, gfp_t gfp_mask)
13471347 {
1348+ struct rdma_ah_attr ah_attr;
13481349 unsigned long flags;
13491350
13501351 spin_lock_irqsave(&query->port->ah_lock, flags);
@@ -1356,6 +1357,15 @@ static int alloc_mad(struct ib_sa_query *query, gfp_t gfp_mask) · 5 unmodified lines
13561357 query->sm_ah = query->port->sm_ah;
13571358 spin_unlock_irqrestore(&query->port->ah_lock, flags);
13581359
1360+ /*
1361+ * Always check if sm_ah has valid dlid assigned,
1362+ * before querying for class port info
1363+ */
1364+ if ((rdma_query_ah(query->sm_ah->ah, &ah_attr) < 0) ||
1365+ !rdma_is_valid_unicast_lid(&ah_attr)) {
1366+ kref_put(&query->sm_ah->ref, free_sm_ah);
1367+ return -EAGAIN;
1368+ }
13591369 query->mad_buf = ib_create_send_mad(query->port->agent, 1,
13601370 query->sm_ah->pkey_index,
13611371 0, IB_MGMT_SA_HDR, IB_MGMT_SA_DATA,
drivers/infiniband/core/security.c+4−6
@@ -653,12 +653,11 @@ int ib_security_modify_qp(struct ib_qp *qp, · 652 unmodified lines
653653 }
654654 return ret;
655655 }
656−EXPORT_SYMBOL(ib_security_modify_qp);
657656
658−int ib_security_pkey_access(struct ib_device *dev,
659− u8 port_num,
660− u16 pkey_index,
661− void *sec)
657+static int ib_security_pkey_access(struct ib_device *dev,
658+ u8 port_num,
659+ u16 pkey_index,
660+ void *sec)
662661 {
663662 u64 subnet_prefix;
664663 u16 pkey;
@@ -678,7 +677,6 @@ int ib_security_pkey_access(struct ib_device *dev, · 13 unmodified lines
678677
679678 return security_ib_pkey_access(sec, subnet_prefix, pkey);
680679 }
681−EXPORT_SYMBOL(ib_security_pkey_access);
682680
683681 static int ib_mad_agent_security_change(struct notifier_block *nb,
684682 unsigned long event,
drivers/infiniband/core/sysfs.c+0−1
@@ -1276,7 +1276,6 @@ int ib_device_register_sysfs(struct ib_device *device, · 1275 unmodified lines
12761276 int ret;
12771277 int i;
12781278
1279− WARN_ON_ONCE(!device->dev.parent);
12801279 ret = dev_set_name(class_dev, "%s", device->name);
12811280 if (ret)
12821281 return ret;
drivers/infiniband/core/ucm.c+28−45
@@ -53,6 +53,8 @@ · 52 unmodified lines
5353 #include <rdma/ib_user_cm.h>
5454 #include <rdma/ib_marshall.h>
5555
56+#include "core_priv.h"
57+
5658 MODULE_AUTHOR("Libor Michalek");
5759 MODULE_DESCRIPTION("InfiniBand userspace Connection Manager access");
5860 MODULE_LICENSE("Dual BSD/GPL");
@@ -104,10 +106,13 @@ struct ib_ucm_event { · 45 unmodified lines
104106 enum {
105107 IB_UCM_MAJOR = 231,
106108 IB_UCM_BASE_MINOR = 224,
107− IB_UCM_MAX_DEVICES = 32
109+ IB_UCM_MAX_DEVICES = RDMA_MAX_PORTS,
110+ IB_UCM_NUM_FIXED_MINOR = 32,
111+ IB_UCM_NUM_DYNAMIC_MINOR = IB_UCM_MAX_DEVICES - IB_UCM_NUM_FIXED_MINOR,
108112 };
109113
110114 #define IB_UCM_BASE_DEV MKDEV(IB_UCM_MAJOR, IB_UCM_BASE_MINOR)
115+static dev_t dynamic_ucm_dev;
111116
112117 static void ib_ucm_add_one(struct ib_device *device);
113118 static void ib_ucm_remove_one(struct ib_device *device, void *client_data);
@@ -1199,7 +1204,6 @@ static int ib_ucm_close(struct inode *inode, struct file *filp) · 1085 unmodified lines
11991204 return 0;
12001205 }
12011206
1202−static DECLARE_BITMAP(overflow_map, IB_UCM_MAX_DEVICES);
12031207 static void ib_ucm_release_dev(struct device *dev)
12041208 {
12051209 struct ib_ucm_device *ucm_dev;
@@ -1210,10 +1214,7 @@ static void ib_ucm_release_dev(struct device *dev) · 4 unmodified lines
12101214
12111215 static void ib_ucm_free_dev(struct ib_ucm_device *ucm_dev)
12121216 {
1213− if (ucm_dev->devnum < IB_UCM_MAX_DEVICES)
1214− clear_bit(ucm_dev->devnum, dev_map);
1215− else
1216− clear_bit(ucm_dev->devnum - IB_UCM_MAX_DEVICES, overflow_map);
1217+ clear_bit(ucm_dev->devnum, dev_map);
12171218 }
12181219
12191220 static const struct file_operations ucm_fops = {
@@ -1235,27 +1236,6 @@ static ssize_t show_ibdev(struct device *dev, struct device_attribute *attr, · 15 unmodified lines
12351236 }
12361237 static DEVICE_ATTR(ibdev, S_IRUGO, show_ibdev, NULL);
12371238
1238−static dev_t overflow_maj;
1239−static int find_overflow_devnum(void)
1240−{
1241− int ret;
1242−
1243− if (!overflow_maj) {
1244− ret = alloc_chrdev_region(&overflow_maj, 0, IB_UCM_MAX_DEVICES,
1245− "infiniband_cm");
1246− if (ret) {
1247− pr_err("ucm: couldn't register dynamic device number\n");
1248− return ret;
1249− }
1250− }
1251−
1252− ret = find_first_zero_bit(overflow_map, IB_UCM_MAX_DEVICES);
1253− if (ret >= IB_UCM_MAX_DEVICES)
1254− return -1;
1255−
1256− return ret;
1257−}
1258−
12591239 static void ib_ucm_add_one(struct ib_device *device)
12601240 {
12611241 int devnum;
@@ -1274,19 +1254,14 @@ static void ib_ucm_add_one(struct ib_device *device) · 12 unmodified lines
12741254 ucm_dev->dev.release = ib_ucm_release_dev;
12751255
12761256 devnum = find_first_zero_bit(dev_map, IB_UCM_MAX_DEVICES);
1277− if (devnum >= IB_UCM_MAX_DEVICES) {
1278− devnum = find_overflow_devnum();
1279− if (devnum < 0)
1280− goto err;
1281−
1282− ucm_dev->devnum = devnum + IB_UCM_MAX_DEVICES;
1283− base = devnum + overflow_maj;
1284− set_bit(devnum, overflow_map);
1285− } else {
1286− ucm_dev->devnum = devnum;
1287− base = devnum + IB_UCM_BASE_DEV;
1288− set_bit(devnum, dev_map);
1289− }
1257+ if (devnum >= IB_UCM_MAX_DEVICES)
1258+ goto err;
1259+ ucm_dev->devnum = devnum;
1260+ set_bit(devnum, dev_map);
1261+ if (devnum >= IB_UCM_NUM_FIXED_MINOR)
1262+ base = dynamic_ucm_dev + devnum - IB_UCM_NUM_FIXED_MINOR;
1263+ else
1264+ base = IB_UCM_BASE_DEV + devnum;
12901265
12911266 cdev_init(&ucm_dev->cdev, &ucm_fops);
12921267 ucm_dev->cdev.owner = THIS_MODULE;
@@ -1334,13 +1309,20 @@ static int __init ib_ucm_init(void) · 41 unmodified lines
13341309 {
13351310 int ret;
13361311
1337− ret = register_chrdev_region(IB_UCM_BASE_DEV, IB_UCM_MAX_DEVICES,
1312+ ret = register_chrdev_region(IB_UCM_BASE_DEV, IB_UCM_NUM_FIXED_MINOR,
13381313 "infiniband_cm");
13391314 if (ret) {
13401315 pr_err("ucm: couldn't register device number\n");
13411316 goto error1;
13421317 }
13431318
1319+ ret = alloc_chrdev_region(&dynamic_ucm_dev, 0, IB_UCM_NUM_DYNAMIC_MINOR,
1320+ "infiniband_cm");
1321+ if (ret) {
1322+ pr_err("ucm: couldn't register dynamic device number\n");
1323+ goto err_alloc;
1324+ }
1325+
13441326 ret = class_create_file(&cm_class, &class_attr_abi_version.attr);
13451327 if (ret) {
13461328 pr_err("ucm: couldn't create abi_version attribute\n");
@@ -1357,7 +1339,9 @@ static int __init ib_ucm_init(void) · 10 unmodified lines
13571339 error3:
13581340 class_remove_file(&cm_class, &class_attr_abi_version.attr);
13591341 error2:
1360− unregister_chrdev_region(IB_UCM_BASE_DEV, IB_UCM_MAX_DEVICES);
1342+ unregister_chrdev_region(dynamic_ucm_dev, IB_UCM_NUM_DYNAMIC_MINOR);
1343+err_alloc:
1344+ unregister_chrdev_region(IB_UCM_BASE_DEV, IB_UCM_NUM_FIXED_MINOR);
13611345 error1:
13621346 return ret;
13631347 }
@@ -1366,9 +1350,8 @@ static void __exit ib_ucm_cleanup(void) · 2 unmodified lines
13661350 {
13671351 ib_unregister_client(&ucm_client);
13681352 class_remove_file(&cm_class, &class_attr_abi_version.attr);
1369− unregister_chrdev_region(IB_UCM_BASE_DEV, IB_UCM_MAX_DEVICES);
1370− if (overflow_maj)
1371− unregister_chrdev_region(overflow_maj, IB_UCM_MAX_DEVICES);
1353+ unregister_chrdev_region(IB_UCM_BASE_DEV, IB_UCM_NUM_FIXED_MINOR);
1354+ unregister_chrdev_region(dynamic_ucm_dev, IB_UCM_NUM_DYNAMIC_MINOR);
13721355 idr_destroy(&ctx_id_table);
13731356 }
13741357
drivers/infiniband/core/ucma.c+10−9
@@ -904,13 +904,14 @@ static ssize_t ucma_query_path(struct ucma_context *ctx, · 903 unmodified lines
904904
905905 resp->path_data[i].flags = IB_PATH_GMP | IB_PATH_PRIMARY |
906906 IB_PATH_BIDIRECTIONAL;
907− if (rec->rec_type == SA_PATH_REC_TYPE_IB) {
908− ib_sa_pack_path(rec, &resp->path_data[i].path_rec);
909− } else {
907+ if (rec->rec_type == SA_PATH_REC_TYPE_OPA) {
910908 struct sa_path_rec ib;
911909
912910 sa_convert_path_opa_to_ib(&ib, rec);
913911 ib_sa_pack_path(&ib, &resp->path_data[i].path_rec);
912+
913+ } else {
914+ ib_sa_pack_path(rec, &resp->path_data[i].path_rec);
914915 }
915916 }
916917
@@ -943,8 +944,8 @@ static ssize_t ucma_query_gid(struct ucma_context *ctx, · 26 unmodified lines
943944 } else {
944945 addr->sib_family = AF_IB;
945946 addr->sib_pkey = (__force __be16) resp.pkey;
946− rdma_addr_get_sgid(&ctx->cm_id->route.addr.dev_addr,
947− (union ib_gid *) &addr->sib_addr);
947+ rdma_read_gids(ctx->cm_id, (union ib_gid *)&addr->sib_addr,
948+ NULL);
948949 addr->sib_sid = rdma_get_service_id(ctx->cm_id, (struct sockaddr *)
949950 &ctx->cm_id->route.addr.src_addr);
950951 }
@@ -956,8 +957,8 @@ static ssize_t ucma_query_gid(struct ucma_context *ctx, · 5 unmodified lines
956957 } else {
957958 addr->sib_family = AF_IB;
958959 addr->sib_pkey = (__force __be16) resp.pkey;
959− rdma_addr_get_dgid(&ctx->cm_id->route.addr.dev_addr,
960− (union ib_gid *) &addr->sib_addr);
960+ rdma_read_gids(ctx->cm_id, NULL,
961+ (union ib_gid *)&addr->sib_addr);
961962 addr->sib_sid = rdma_get_service_id(ctx->cm_id, (struct sockaddr *)
962963 &ctx->cm_id->route.addr.dst_addr);
963964 }
@@ -1231,9 +1232,9 @@ static int ucma_set_ib_path(struct ucma_context *ctx, · 267 unmodified lines
12311232 struct sa_path_rec opa;
12321233
12331234 sa_convert_path_ib_to_opa(&opa, &sa_path);
1234− ret = rdma_set_ib_paths(ctx->cm_id, &opa, 1);
1235+ ret = rdma_set_ib_path(ctx->cm_id, &opa);
12351236 } else {
1236− ret = rdma_set_ib_paths(ctx->cm_id, &sa_path, 1);
1237+ ret = rdma_set_ib_path(ctx->cm_id, &sa_path);
12371238 }
12381239 if (ret)
12391240 return ret;
drivers/infiniband/core/umem.c+1−1
@@ -352,7 +352,7 @@ int ib_umem_copy_from(void *dst, struct ib_umem *umem, size_t offset, · 351 unmodified lines
352352 return -EINVAL;
353353 }
354354
355− ret = sg_pcopy_to_buffer(umem->sg_head.sgl, umem->nmap, dst, length,
355+ ret = sg_pcopy_to_buffer(umem->sg_head.sgl, umem->npages, dst, length,
356356 offset + ib_umem_offset(umem));
357357
358358 if (ret < 0)
drivers/infiniband/core/user_mad.c+57−66
@@ -55,16 +55,21 @@ · 54 unmodified lines
5555 #include <rdma/ib_mad.h>
5656 #include <rdma/ib_user_mad.h>
5757
58+#include "core_priv.h"
59+
5860 MODULE_AUTHOR("Roland Dreier");
5961 MODULE_DESCRIPTION("InfiniBand userspace MAD packet access");
6062 MODULE_LICENSE("Dual BSD/GPL");
6163
6264 enum {
63− IB_UMAD_MAX_PORTS = 64,
65+ IB_UMAD_MAX_PORTS = RDMA_MAX_PORTS,
6466 IB_UMAD_MAX_AGENTS = 32,
6567
6668 IB_UMAD_MAJOR = 231,
67− IB_UMAD_MINOR_BASE = 0
69+ IB_UMAD_MINOR_BASE = 0,
70+ IB_UMAD_NUM_FIXED_MINOR = 64,
71+ IB_UMAD_NUM_DYNAMIC_MINOR = IB_UMAD_MAX_PORTS - IB_UMAD_NUM_FIXED_MINOR,
72+ IB_ISSM_MINOR_BASE = IB_UMAD_NUM_FIXED_MINOR,
6873 };
6974
7075 /*
@@ -127,9 +132,12 @@ struct ib_umad_packet { · 56 unmodified lines
127132
128133 static struct class *umad_class;
129134
130−static const dev_t base_dev = MKDEV(IB_UMAD_MAJOR, IB_UMAD_MINOR_BASE);
135+static const dev_t base_umad_dev = MKDEV(IB_UMAD_MAJOR, IB_UMAD_MINOR_BASE);
136+static const dev_t base_issm_dev = MKDEV(IB_UMAD_MAJOR, IB_UMAD_MINOR_BASE) +
137+ IB_UMAD_NUM_FIXED_MINOR;
138+static dev_t dynamic_umad_dev;
139+static dev_t dynamic_issm_dev;
131140
132−static DEFINE_SPINLOCK(port_lock);
133141 static DECLARE_BITMAP(dev_map, IB_UMAD_MAX_PORTS);
134142
135143 static void ib_umad_add_one(struct ib_device *device);
@@ -233,8 +241,7 @@ static void recv_handler(struct ib_mad_agent *agent, · 97 unmodified lines
233241 * On OPA devices it is okay to lose the upper 16 bits of LID as this
234242 * information is obtained elsewhere. Mask off the upper 16 bits.
235243 */
236− if (agent->device->port_immutable[agent->port_num].core_cap_flags &
237− RDMA_CORE_PORT_INTEL_OPA)
244+ if (rdma_cap_opa_mad(agent->device, agent->port_num))
238245 packet->mad.hdr.lid = ib_lid_be16(0xFFFF &
239246 mad_recv_wc->wc->slid);
240247 else
@@ -246,10 +253,14 @@ static void recv_handler(struct ib_mad_agent *agent, · 5 unmodified lines
246253 if (packet->mad.hdr.grh_present) {
247254 struct rdma_ah_attr ah_attr;
248255 const struct ib_global_route *grh;
256+ int ret;
249257
250− ib_init_ah_from_wc(agent->device, agent->port_num,
251− mad_recv_wc->wc, mad_recv_wc->recv_buf.grh,
252− &ah_attr);
258+ ret = ib_init_ah_attr_from_wc(agent->device, agent->port_num,
259+ mad_recv_wc->wc,
260+ mad_recv_wc->recv_buf.grh,
261+ &ah_attr);
262+ if (ret)
263+ goto err2;
253264
254265 grh = rdma_ah_read_grh(&ah_attr);
255266 packet->mad.hdr.gid_index = grh->sgid_index;
@@ -500,7 +511,7 @@ static ssize_t ib_umad_write(struct file *filp, const char __user *buf, · 244 unmodified lines
500511 }
501512
502513 memset(&ah_attr, 0, sizeof ah_attr);
503− ah_attr.type = rdma_ah_find_type(file->port->ib_dev,
514+ ah_attr.type = rdma_ah_find_type(agent->device,
504515 file->port->port_num);
505516 rdma_ah_set_dlid(&ah_attr, be16_to_cpu(packet->mad.hdr.lid));
506517 rdma_ah_set_sl(&ah_attr, packet->mad.hdr.sl);
@@ -1139,54 +1150,26 @@ static DEVICE_ATTR(port, S_IRUGO, show_port, NULL); · 632 unmodified lines
11391150 static CLASS_ATTR_STRING(abi_version, S_IRUGO,
11401151 __stringify(IB_USER_MAD_ABI_VERSION));
11411152
1142−static dev_t overflow_maj;
1143−static DECLARE_BITMAP(overflow_map, IB_UMAD_MAX_PORTS);
1144−static int find_overflow_devnum(struct ib_device *device)
1145−{
1146− int ret;
1147−
1148− if (!overflow_maj) {
1149− ret = alloc_chrdev_region(&overflow_maj, 0, IB_UMAD_MAX_PORTS * 2,
1150− "infiniband_mad");
1151− if (ret) {
1152− dev_err(&device->dev,
1153− "couldn't register dynamic device number\n");
1154− return ret;
1155− }
1156− }
1157−
1158− ret = find_first_zero_bit(overflow_map, IB_UMAD_MAX_PORTS);
1159− if (ret >= IB_UMAD_MAX_PORTS)
1160− return -1;
1161−
1162− return ret;
1163−}
1164−
11651153 static int ib_umad_init_port(struct ib_device *device, int port_num,
11661154 struct ib_umad_device *umad_dev,
11671155 struct ib_umad_port *port)
11681156 {
11691157 int devnum;
1170− dev_t base;
1158+ dev_t base_umad;
1159+ dev_t base_issm;
11711160
1172− spin_lock(&port_lock);
11731161 devnum = find_first_zero_bit(dev_map, IB_UMAD_MAX_PORTS);
1174− if (devnum >= IB_UMAD_MAX_PORTS) {
1175− spin_unlock(&port_lock);
1176− devnum = find_overflow_devnum(device);
1177− if (devnum < 0)
1178− return -1;
1179−
1180− spin_lock(&port_lock);
1181− port->dev_num = devnum + IB_UMAD_MAX_PORTS;
1182− base = devnum + overflow_maj;
1183− set_bit(devnum, overflow_map);
1162+ if (devnum >= IB_UMAD_MAX_PORTS)
1163+ return -1;
1164+ port->dev_num = devnum;
1165+ set_bit(devnum, dev_map);
1166+ if (devnum >= IB_UMAD_NUM_FIXED_MINOR) {
1167+ base_umad = dynamic_umad_dev + devnum - IB_UMAD_NUM_FIXED_MINOR;
1168+ base_issm = dynamic_issm_dev + devnum - IB_UMAD_NUM_FIXED_MINOR;
11841169 } else {
1185− port->dev_num = devnum;
1186− base = devnum + base_dev;
1187− set_bit(devnum, dev_map);
1170+ base_umad = devnum + base_umad_dev;
1171+ base_issm = devnum + base_issm_dev;
11881172 }
1189− spin_unlock(&port_lock);
11901173
11911174 port->ib_dev = device;
11921175 port->port_num = port_num;
@@ -1198,7 +1181,7 @@ static int ib_umad_init_port(struct ib_device *device, int port_num, · 5 unmodified lines
11981181 port->cdev.owner = THIS_MODULE;
11991182 cdev_set_parent(&port->cdev, &umad_dev->kobj);
12001183 kobject_set_name(&port->cdev.kobj, "umad%d", port->dev_num);
1201− if (cdev_add(&port->cdev, base, 1))
1184+ if (cdev_add(&port->cdev, base_umad, 1))
12021185 goto err_cdev;
12031186
12041187 port->dev = device_create(umad_class, device->dev.parent,
@@ -1212,12 +1195,11 @@ static int ib_umad_init_port(struct ib_device *device, int port_num, · 7 unmodified lines
12121195 if (device_create_file(port->dev, &dev_attr_port))
12131196 goto err_dev;
12141197
1215− base += IB_UMAD_MAX_PORTS;
12161198 cdev_init(&port->sm_cdev, &umad_sm_fops);
12171199 port->sm_cdev.owner = THIS_MODULE;
12181200 cdev_set_parent(&port->sm_cdev, &umad_dev->kobj);
12191201 kobject_set_name(&port->sm_cdev.kobj, "issm%d", port->dev_num);
1220− if (cdev_add(&port->sm_cdev, base, 1))
1202+ if (cdev_add(&port->sm_cdev, base_issm, 1))
12211203 goto err_sm_cdev;
12221204
12231205 port->sm_dev = device_create(umad_class, device->dev.parent,
@@ -1244,10 +1226,7 @@ err_dev: · 20 unmodified lines
12441226
12451227 err_cdev:
12461228 cdev_del(&port->cdev);
1247− if (port->dev_num < IB_UMAD_MAX_PORTS)
1248− clear_bit(devnum, dev_map);
1249− else
1250− clear_bit(devnum, overflow_map);
1229+ clear_bit(devnum, dev_map);
12511230
12521231 return -1;
12531232 }
@@ -1281,11 +1260,7 @@ static void ib_umad_kill_port(struct ib_umad_port *port) · 27 unmodified lines
12811260 }
12821261
12831262 mutex_unlock(&port->file_mutex);
1284−
1285− if (port->dev_num < IB_UMAD_MAX_PORTS)
1286− clear_bit(port->dev_num, dev_map);
1287− else
1288− clear_bit(port->dev_num - IB_UMAD_MAX_PORTS, overflow_map);
1263+ clear_bit(port->dev_num, dev_map);
12891264 }
12901265
12911266 static void ib_umad_add_one(struct ib_device *device)
@@ -1361,13 +1336,23 @@ static int __init ib_umad_init(void) · 69 unmodified lines
13611336 {
13621337 int ret;
13631338
1364− ret = register_chrdev_region(base_dev, IB_UMAD_MAX_PORTS * 2,
1339+ ret = register_chrdev_region(base_umad_dev,
1340+ IB_UMAD_NUM_FIXED_MINOR * 2,
13651341 "infiniband_mad");
13661342 if (ret) {
13671343 pr_err("couldn't register device number\n");
13681344 goto out;
13691345 }
13701346
1347+ ret = alloc_chrdev_region(&dynamic_umad_dev, 0,
1348+ IB_UMAD_NUM_DYNAMIC_MINOR * 2,
1349+ "infiniband_mad");
1350+ if (ret) {
1351+ pr_err("couldn't register dynamic device number\n");
1352+ goto out_alloc;
1353+ }
1354+ dynamic_issm_dev = dynamic_umad_dev + IB_UMAD_NUM_DYNAMIC_MINOR;
1355+
13711356 umad_class = class_create(THIS_MODULE, "infiniband_mad");
13721357 if (IS_ERR(umad_class)) {
13731358 ret = PTR_ERR(umad_class);
@@ -1395,7 +1380,12 @@ out_class: · 21 unmodified lines
13951380 class_destroy(umad_class);
13961381
13971382 out_chrdev:
1398− unregister_chrdev_region(base_dev, IB_UMAD_MAX_PORTS * 2);
1383+ unregister_chrdev_region(dynamic_umad_dev,
1384+ IB_UMAD_NUM_DYNAMIC_MINOR * 2);
1385+
1386+out_alloc:
1387+ unregister_chrdev_region(base_umad_dev,
1388+ IB_UMAD_NUM_FIXED_MINOR * 2);
13991389
14001390 out:
14011391 return ret;
@@ -1405,9 +1395,10 @@ static void __exit ib_umad_cleanup(void) · 3 unmodified lines
14051395 {
14061396 ib_unregister_client(&umad_client);
14071397 class_destroy(umad_class);
1408− unregister_chrdev_region(base_dev, IB_UMAD_MAX_PORTS * 2);
1409− if (overflow_maj)
1410− unregister_chrdev_region(overflow_maj, IB_UMAD_MAX_PORTS * 2);
1398+ unregister_chrdev_region(base_umad_dev,
1399+ IB_UMAD_NUM_FIXED_MINOR * 2);
1400+ unregister_chrdev_region(dynamic_umad_dev,
1401+ IB_UMAD_NUM_DYNAMIC_MINOR * 2);
14111402 }
14121403
14131404 module_init(ib_umad_init);
drivers/infiniband/core/uverbs_cmd.c+7−7
@@ -340,6 +340,8 @@ ssize_t ib_uverbs_alloc_pd(struct ib_uverbs_file *file, · 339 unmodified lines
340340 uobj->object = pd;
341341 memset(&resp, 0, sizeof resp);
342342 resp.pd_handle = uobj->id;
343+ pd->res.type = RDMA_RESTRACK_PD;
344+ rdma_restrack_add(&pd->res);
343345
344346 if (copy_to_user(u64_to_user_ptr(cmd.response), &resp, sizeof resp)) {
345347 ret = -EFAULT;
@@ -1033,6 +1035,8 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file, · 687 unmodified lines
10331035 goto err_cb;
10341036
10351037 uobj_alloc_commit(&obj->uobject);
1038+ cq->res.type = RDMA_RESTRACK_CQ;
1039+ rdma_restrack_add(&cq->res);
10361040
10371041 return obj;
10381042
@@ -1145,10 +1149,7 @@ int ib_uverbs_ex_create_cq(struct ib_uverbs_file *file, · 106 unmodified lines
11451149 min(ucore->inlen, sizeof(cmd)),
11461150 ib_uverbs_ex_create_cq_cb, NULL);
11471151
1148− if (IS_ERR(obj))
1149− return PTR_ERR(obj);
1150−
1151− return 0;
1152+ return PTR_ERR_OR_ZERO(obj);
11521153 }
11531154
11541155 ssize_t ib_uverbs_resize_cq(struct ib_uverbs_file *file,
@@ -1199,7 +1200,7 @@ static int copy_wc_to_user(struct ib_device *ib_dev, void __user *dest, · 44 unmodified lines
11991200 tmp.opcode = wc->opcode;
12001201 tmp.vendor_err = wc->vendor_err;
12011202 tmp.byte_len = wc->byte_len;
1202− tmp.ex.imm_data = (__u32 __force) wc->ex.imm_data;
1203+ tmp.ex.imm_data = wc->ex.imm_data;
12031204 tmp.qp_num = wc->qp->qp_num;
12041205 tmp.src_qp = wc->src_qp;
12051206 tmp.wc_flags = wc->wc_flags;
@@ -1517,7 +1518,7 @@ static int create_qp(struct ib_uverbs_file *file, · 311 unmodified lines
15171518 if (cmd->qp_type == IB_QPT_XRC_TGT)
15181519 qp = ib_create_qp(pd, &attr);
15191520 else
1520− qp = device->create_qp(pd, &attr, uhw);
1521+ qp = _ib_create_qp(device, pd, &attr, uhw);
15211522
15221523 if (IS_ERR(qp)) {
15231524 ret = PTR_ERR(qp);
@@ -1530,7 +1531,6 @@ static int create_qp(struct ib_uverbs_file *file, · 6 unmodified lines
15301531 goto err_cb;
15311532
15321533 qp->real_qp = qp;
1533− qp->device = device;
15341534 qp->pd = pd;
15351535 qp->send_cq = attr.send_cq;
15361536 qp->recv_cq = attr.recv_cq;
drivers/infiniband/core/uverbs_ioctl.c+13−6
@@ -243,16 +243,13 @@ static long ib_uverbs_cmd_verbs(struct ib_device *ib_dev, · 242 unmodified lines
243243 size_t ctx_size;
244244 uintptr_t data[UVERBS_OPTIMIZE_USING_STACK_SZ / sizeof(uintptr_t)];
245245
246− if (hdr->reserved)
247− return -EINVAL;
248−
249246 object_spec = uverbs_get_object(ib_dev, hdr->object_id);
250247 if (!object_spec)
251− return -EOPNOTSUPP;
248+ return -EPROTONOSUPPORT;
252249
253250 method_spec = uverbs_get_method(object_spec, hdr->method_id);
254251 if (!method_spec)
255− return -EOPNOTSUPP;
252+ return -EPROTONOSUPPORT;
256253
257254 if ((method_spec->flags & UVERBS_ACTION_FLAG_CREATE_ROOT) ^ !file->ucontext)
258255 return -EINVAL;
@@ -305,6 +302,16 @@ static long ib_uverbs_cmd_verbs(struct ib_device *ib_dev, · 46 unmodified lines
305302
306303 err = uverbs_handle_method(buf, ctx->uattrs, hdr->num_attrs, ib_dev,
307304 file, method_spec, ctx->uverbs_attr_bundle);
305+
306+ /*
307+ * EPROTONOSUPPORT is ONLY to be returned if the ioctl framework can
308+ * not invoke the method because the request is not supported. No
309+ * other cases should return this code.
310+ */
311+ if (unlikely(err == -EPROTONOSUPPORT)) {
312+ WARN_ON_ONCE(err == -EPROTONOSUPPORT);
313+ err = -EINVAL;
314+ }
308315 out:
309316 if (ctx != (void *)data)
310317 kfree(ctx);
@@ -341,7 +348,7 @@ long ib_uverbs_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) · 30 unmodified lines
341348 }
342349
343350 if (hdr.reserved) {
344− err = -EOPNOTSUPP;
351+ err = -EPROTONOSUPPORT;
345352 goto out;
346353 }
347354
drivers/infiniband/core/uverbs_main.c+34−61
@@ -62,14 +62,16 @@ MODULE_LICENSE("Dual BSD/GPL"); · 61 unmodified lines
6262 enum {
6363 IB_UVERBS_MAJOR = 231,
6464 IB_UVERBS_BASE_MINOR = 192,
65− IB_UVERBS_MAX_DEVICES = 32
65+ IB_UVERBS_MAX_DEVICES = RDMA_MAX_PORTS,
66+ IB_UVERBS_NUM_FIXED_MINOR = 32,
67+ IB_UVERBS_NUM_DYNAMIC_MINOR = IB_UVERBS_MAX_DEVICES - IB_UVERBS_NUM_FIXED_MINOR,
6668 };
6769
6870 #define IB_UVERBS_BASE_DEV MKDEV(IB_UVERBS_MAJOR, IB_UVERBS_BASE_MINOR)
6971
72+static dev_t dynamic_uverbs_dev;
7073 static struct class *uverbs_class;
7174
72−static DEFINE_SPINLOCK(map_lock);
7375 static DECLARE_BITMAP(dev_map, IB_UVERBS_MAX_DEVICES);
7476
7577 static ssize_t (*uverbs_cmd_table[])(struct ib_uverbs_file *file,
@@ -1005,34 +1007,6 @@ static DEVICE_ATTR(abi_version, S_IRUGO, show_dev_abi_version, NULL); · 929 unmodified lines
10051007 static CLASS_ATTR_STRING(abi_version, S_IRUGO,
10061008 __stringify(IB_USER_VERBS_ABI_VERSION));
10071009
1008−static dev_t overflow_maj;
1009−static DECLARE_BITMAP(overflow_map, IB_UVERBS_MAX_DEVICES);
1010−
1011−/*
1012− * If we have more than IB_UVERBS_MAX_DEVICES, dynamically overflow by
1013− * requesting a new major number and doubling the number of max devices we
1014− * support. It's stupid, but simple.
1015− */
1016−static int find_overflow_devnum(void)
1017−{
1018− int ret;
1019−
1020− if (!overflow_maj) {
1021− ret = alloc_chrdev_region(&overflow_maj, 0, IB_UVERBS_MAX_DEVICES,
1022− "infiniband_verbs");
1023− if (ret) {
1024− pr_err("user_verbs: couldn't register dynamic device number\n");
1025− return ret;
1026− }
1027− }
1028−
1029− ret = find_first_zero_bit(overflow_map, IB_UVERBS_MAX_DEVICES);
1030− if (ret >= IB_UVERBS_MAX_DEVICES)
1031− return -1;
1032−
1033− return ret;
1034−}
1035−
10361010 static void ib_uverbs_add_one(struct ib_device *device)
10371011 {
10381012 int devnum;
@@ -1062,24 +1036,15 @@ static void ib_uverbs_add_one(struct ib_device *device) · 23 unmodified lines
10621036 INIT_LIST_HEAD(&uverbs_dev->uverbs_file_list);
10631037 INIT_LIST_HEAD(&uverbs_dev->uverbs_events_file_list);
10641038
1065− spin_lock(&map_lock);
10661039 devnum = find_first_zero_bit(dev_map, IB_UVERBS_MAX_DEVICES);
1067− if (devnum >= IB_UVERBS_MAX_DEVICES) {
1068− spin_unlock(&map_lock);
1069− devnum = find_overflow_devnum();
1070− if (devnum < 0)
1071− goto err;
1072−
1073− spin_lock(&map_lock);
1074− uverbs_dev->devnum = devnum + IB_UVERBS_MAX_DEVICES;
1075− base = devnum + overflow_maj;
1076− set_bit(devnum, overflow_map);
1077− } else {
1078− uverbs_dev->devnum = devnum;
1079− base = devnum + IB_UVERBS_BASE_DEV;
1080− set_bit(devnum, dev_map);
1081− }
1082− spin_unlock(&map_lock);
1040+ if (devnum >= IB_UVERBS_MAX_DEVICES)
1041+ goto err;
1042+ uverbs_dev->devnum = devnum;
1043+ set_bit(devnum, dev_map);
1044+ if (devnum >= IB_UVERBS_NUM_FIXED_MINOR)
1045+ base = dynamic_uverbs_dev + devnum - IB_UVERBS_NUM_FIXED_MINOR;
1046+ else
1047+ base = IB_UVERBS_BASE_DEV + devnum;
10831048
10841049 rcu_assign_pointer(uverbs_dev->ib_dev, device);
10851050 uverbs_dev->num_comp_vectors = device->num_comp_vectors;
@@ -1124,10 +1089,7 @@ err_class: · 38 unmodified lines
11241089
11251090 err_cdev:
11261091 cdev_del(&uverbs_dev->cdev);
1127− if (uverbs_dev->devnum < IB_UVERBS_MAX_DEVICES)
1128− clear_bit(devnum, dev_map);
1129− else
1130− clear_bit(devnum, overflow_map);
1092+ clear_bit(devnum, dev_map);
11311093
11321094 err:
11331095 if (atomic_dec_and_test(&uverbs_dev->refcount))
@@ -1219,11 +1181,7 @@ static void ib_uverbs_remove_one(struct ib_device *device, void *client_data) · 85 unmodified lines
12191181 dev_set_drvdata(uverbs_dev->dev, NULL);
12201182 device_destroy(uverbs_class, uverbs_dev->cdev.dev);
12211183 cdev_del(&uverbs_dev->cdev);
1222−
1223− if (uverbs_dev->devnum < IB_UVERBS_MAX_DEVICES)
1224− clear_bit(uverbs_dev->devnum, dev_map);
1225− else
1226− clear_bit(uverbs_dev->devnum - IB_UVERBS_MAX_DEVICES, overflow_map);
1184+ clear_bit(uverbs_dev->devnum, dev_map);
12271185
12281186 if (device->disassociate_ucontext) {
12291187 /* We disassociate HW resources and immediately return.
@@ -1265,13 +1223,22 @@ static int __init ib_uverbs_init(void) · 35 unmodified lines
12651223 {
12661224 int ret;
12671225
1268− ret = register_chrdev_region(IB_UVERBS_BASE_DEV, IB_UVERBS_MAX_DEVICES,
1226+ ret = register_chrdev_region(IB_UVERBS_BASE_DEV,
1227+ IB_UVERBS_NUM_FIXED_MINOR,
12691228 "infiniband_verbs");
12701229 if (ret) {
12711230 pr_err("user_verbs: couldn't register device number\n");
12721231 goto out;
12731232 }
12741233
1234+ ret = alloc_chrdev_region(&dynamic_uverbs_dev, 0,
1235+ IB_UVERBS_NUM_DYNAMIC_MINOR,
1236+ "infiniband_verbs");
1237+ if (ret) {
1238+ pr_err("couldn't register dynamic device number\n");
1239+ goto out_alloc;
1240+ }
1241+
12751242 uverbs_class = class_create(THIS_MODULE, "infiniband_verbs");
12761243 if (IS_ERR(uverbs_class)) {
12771244 ret = PTR_ERR(uverbs_class);
@@ -1299,7 +1266,12 @@ out_class: · 21 unmodified lines
12991266 class_destroy(uverbs_class);
13001267
13011268 out_chrdev:
1302− unregister_chrdev_region(IB_UVERBS_BASE_DEV, IB_UVERBS_MAX_DEVICES);
1269+ unregister_chrdev_region(dynamic_uverbs_dev,
1270+ IB_UVERBS_NUM_DYNAMIC_MINOR);
1271+
1272+out_alloc:
1273+ unregister_chrdev_region(IB_UVERBS_BASE_DEV,
1274+ IB_UVERBS_NUM_FIXED_MINOR);
13031275
13041276 out:
13051277 return ret;
@@ -1309,9 +1281,10 @@ static void __exit ib_uverbs_cleanup(void) · 3 unmodified lines
13091281 {
13101282 ib_unregister_client(&uverbs_client);
13111283 class_destroy(uverbs_class);
1312− unregister_chrdev_region(IB_UVERBS_BASE_DEV, IB_UVERBS_MAX_DEVICES);
1313− if (overflow_maj)
1314− unregister_chrdev_region(overflow_maj, IB_UVERBS_MAX_DEVICES);
1284+ unregister_chrdev_region(IB_UVERBS_BASE_DEV,
1285+ IB_UVERBS_NUM_FIXED_MINOR);
1286+ unregister_chrdev_region(dynamic_uverbs_dev,
1287+ IB_UVERBS_NUM_DYNAMIC_MINOR);
13151288 }
13161289
13171290 module_init(ib_uverbs_init);
drivers/infiniband/core/uverbs_std_types.c+3−0
@@ -35,6 +35,7 @@ · 34 unmodified lines
3535 #include <rdma/ib_verbs.h>
3636 #include <linux/bug.h>
3737 #include <linux/file.h>
38+#include <rdma/restrack.h>
3839 #include "rdma_core.h"
3940 #include "uverbs.h"
4041
@@ -319,6 +320,8 @@ static int uverbs_create_cq_handler(struct ib_device *ib_dev, · 278 unmodified lines
319320 obj->uobject.object = cq;
320321 obj->uobject.user_handle = user_handle;
321322 atomic_set(&cq->usecnt, 0);
323+ cq->res.type = RDMA_RESTRACK_CQ;
324+ rdma_restrack_add(&cq->res);
322325
323326 ret = uverbs_copy_to(attrs, CREATE_CQ_RESP_CQE, &cq->cqe);
324327 if (ret)
drivers/infiniband/core/verbs.c+177−135
@@ -124,16 +124,24 @@ EXPORT_SYMBOL(ib_wc_status_msg); · 123 unmodified lines
124124 __attribute_const__ int ib_rate_to_mult(enum ib_rate rate)
125125 {
126126 switch (rate) {
127− case IB_RATE_2_5_GBPS: return 1;
128− case IB_RATE_5_GBPS: return 2;
129− case IB_RATE_10_GBPS: return 4;
130− case IB_RATE_20_GBPS: return 8;
131− case IB_RATE_30_GBPS: return 12;
132− case IB_RATE_40_GBPS: return 16;
133− case IB_RATE_60_GBPS: return 24;
134− case IB_RATE_80_GBPS: return 32;
135− case IB_RATE_120_GBPS: return 48;
136− default: return -1;
127+ case IB_RATE_2_5_GBPS: return 1;
128+ case IB_RATE_5_GBPS: return 2;
129+ case IB_RATE_10_GBPS: return 4;
130+ case IB_RATE_20_GBPS: return 8;
131+ case IB_RATE_30_GBPS: return 12;
132+ case IB_RATE_40_GBPS: return 16;
133+ case IB_RATE_60_GBPS: return 24;
134+ case IB_RATE_80_GBPS: return 32;
135+ case IB_RATE_120_GBPS: return 48;
136+ case IB_RATE_14_GBPS: return 6;
137+ case IB_RATE_56_GBPS: return 22;
138+ case IB_RATE_112_GBPS: return 45;
139+ case IB_RATE_168_GBPS: return 67;
140+ case IB_RATE_25_GBPS: return 10;
141+ case IB_RATE_100_GBPS: return 40;
142+ case IB_RATE_200_GBPS: return 80;
143+ case IB_RATE_300_GBPS: return 120;
144+ default: return -1;
137145 }
138146 }
139147 EXPORT_SYMBOL(ib_rate_to_mult);
@@ -141,16 +149,24 @@ EXPORT_SYMBOL(ib_rate_to_mult); · 1 unmodified line
141149 __attribute_const__ enum ib_rate mult_to_ib_rate(int mult)
142150 {
143151 switch (mult) {
144− case 1: return IB_RATE_2_5_GBPS;
145− case 2: return IB_RATE_5_GBPS;
146− case 4: return IB_RATE_10_GBPS;
147− case 8: return IB_RATE_20_GBPS;
148− case 12: return IB_RATE_30_GBPS;
149− case 16: return IB_RATE_40_GBPS;
150− case 24: return IB_RATE_60_GBPS;
151− case 32: return IB_RATE_80_GBPS;
152− case 48: return IB_RATE_120_GBPS;
153− default: return IB_RATE_PORT_CURRENT;
152+ case 1: return IB_RATE_2_5_GBPS;
153+ case 2: return IB_RATE_5_GBPS;
154+ case 4: return IB_RATE_10_GBPS;
155+ case 8: return IB_RATE_20_GBPS;
156+ case 12: return IB_RATE_30_GBPS;
157+ case 16: return IB_RATE_40_GBPS;
158+ case 24: return IB_RATE_60_GBPS;
159+ case 32: return IB_RATE_80_GBPS;
160+ case 48: return IB_RATE_120_GBPS;
161+ case 6: return IB_RATE_14_GBPS;
162+ case 22: return IB_RATE_56_GBPS;
163+ case 45: return IB_RATE_112_GBPS;
164+ case 67: return IB_RATE_168_GBPS;
165+ case 10: return IB_RATE_25_GBPS;
166+ case 40: return IB_RATE_100_GBPS;
167+ case 80: return IB_RATE_200_GBPS;
168+ case 120: return IB_RATE_300_GBPS;
169+ default: return IB_RATE_PORT_CURRENT;
154170 }
155171 }
156172 EXPORT_SYMBOL(mult_to_ib_rate);
@@ -247,6 +263,10 @@ struct ib_pd *__ib_alloc_pd(struct ib_device *device, unsigned int flags, · 90 unmodified lines
247263 mr_access_flags |= IB_ACCESS_REMOTE_READ | IB_ACCESS_REMOTE_WRITE;
248264 }
249265
266+ pd->res.type = RDMA_RESTRACK_PD;
267+ pd->res.kern_name = caller;
268+ rdma_restrack_add(&pd->res);
269+
250270 if (mr_access_flags) {
251271 struct ib_mr *mr;
252272
@@ -296,6 +316,7 @@ void ib_dealloc_pd(struct ib_pd *pd) · 43 unmodified lines
296316 requires the caller to guarantee we can't race here. */
297317 WARN_ON(atomic_read(&pd->usecnt));
298318
319+ rdma_restrack_del(&pd->res);
299320 /* Making delalloc_pd a void return is a WIP, no driver should return
300321 an error here. */
301322 ret = pd->device->dealloc_pd(pd);
@@ -421,8 +442,7 @@ static bool find_gid_index(const union ib_gid *gid, · 119 unmodified lines
421442 const struct ib_gid_attr *gid_attr,
422443 void *context)
423444 {
424− struct find_gid_index_context *ctx =
425− (struct find_gid_index_context *)context;
445+ struct find_gid_index_context *ctx = context;
426446
427447 if (ctx->gid_type != gid_attr->gid_type)
428448 return false;
@@ -481,8 +501,53 @@ int ib_get_gids_from_rdma_hdr(const union rdma_network_hdr *hdr, · 52 unmodified lines
481501 }
482502 EXPORT_SYMBOL(ib_get_gids_from_rdma_hdr);
483503
504+/* Resolve destination mac address and hop limit for unicast destination
505+ * GID entry, considering the source GID entry as well.
506+ * ah_attribute must have have valid port_num, sgid_index.
507+ */
508+static int ib_resolve_unicast_gid_dmac(struct ib_device *device,
509+ struct rdma_ah_attr *ah_attr)
510+{
511+ struct ib_gid_attr sgid_attr;
512+ struct ib_global_route *grh;
513+ int hop_limit = 0xff;
514+ union ib_gid sgid;
515+ int ret;
516+
517+ grh = rdma_ah_retrieve_grh(ah_attr);
518+
519+ ret = ib_query_gid(device,
520+ rdma_ah_get_port_num(ah_attr),
521+ grh->sgid_index,
522+ &sgid, &sgid_attr);
523+ if (ret || !sgid_attr.ndev) {
524+ if (!ret)
525+ ret = -ENXIO;
526+ return ret;
527+ }
528+
529+ /* If destination is link local and source GID is RoCEv1,
530+ * IP stack is not used.
531+ */
532+ if (rdma_link_local_addr((struct in6_addr *)grh->dgid.raw) &&
533+ sgid_attr.gid_type == IB_GID_TYPE_ROCE) {
534+ rdma_get_ll_mac((struct in6_addr *)grh->dgid.raw,
535+ ah_attr->roce.dmac);
536+ goto done;
537+ }
538+
539+ ret = rdma_addr_find_l2_eth_by_grh(&sgid, &grh->dgid,
540+ ah_attr->roce.dmac,
541+ sgid_attr.ndev, &hop_limit);
542+done:
543+ dev_put(sgid_attr.ndev);
544+
545+ grh->hop_limit = hop_limit;
546+ return ret;
547+}
548+
484549 /*
485− * This function creates ah from the incoming packet.
550+ * This function initializes address handle attributes from the incoming packet.
486551 * Incoming packet has dgid of the receiver node on which this code is
487552 * getting executed and, sgid contains the GID of the sender.
488553 *
@@ -490,13 +555,10 @@ EXPORT_SYMBOL(ib_get_gids_from_rdma_hdr); · 1 unmodified line
490555 * as sgid and, sgid is used as dgid because sgid contains destinations
491556 * GID whom to respond to.
492557 *
493− * This is why when calling rdma_addr_find_l2_eth_by_grh() function, the
494− * position of arguments dgid and sgid do not match the order of the
495− * parameters.
496558 */
497−int ib_init_ah_from_wc(struct ib_device *device, u8 port_num,
498− const struct ib_wc *wc, const struct ib_grh *grh,
499− struct rdma_ah_attr *ah_attr)
559+int ib_init_ah_attr_from_wc(struct ib_device *device, u8 port_num,
560+ const struct ib_wc *wc, const struct ib_grh *grh,
561+ struct rdma_ah_attr *ah_attr)
500562 {
501563 u32 flow_class;
502564 u16 gid_index;
@@ -523,57 +585,33 @@ int ib_init_ah_from_wc(struct ib_device *device, u8 port_num, · 20 unmodified lines
523585 if (ret)
524586 return ret;
525587
588+ rdma_ah_set_sl(ah_attr, wc->sl);
589+ rdma_ah_set_port_num(ah_attr, port_num);
590+
526591 if (rdma_protocol_roce(device, port_num)) {
527− int if_index = 0;
528592 u16 vlan_id = wc->wc_flags & IB_WC_WITH_VLAN ?
529593 wc->vlan_id : 0xffff;
530− struct net_device *idev;
531− struct net_device *resolved_dev;
532594
533595 if (!(wc->wc_flags & IB_WC_GRH))
534596 return -EPROTOTYPE;
535597
536− if (!device->get_netdev)
537− return -EOPNOTSUPP;
538−
539− idev = device->get_netdev(device, port_num);
540− if (!idev)
541− return -ENODEV;
542−
543− ret = rdma_addr_find_l2_eth_by_grh(&dgid, &sgid,
544− ah_attr->roce.dmac,
545− wc->wc_flags & IB_WC_WITH_VLAN ?
546− NULL : &vlan_id,
547− &if_index, &hoplimit);
548− if (ret) {
549− dev_put(idev);
550− return ret;
551− }
552−
553− resolved_dev = dev_get_by_index(&init_net, if_index);
554− rcu_read_lock();
555− if (resolved_dev != idev && !rdma_is_upper_dev_rcu(idev,
556− resolved_dev))
557− ret = -EHOSTUNREACH;
558− rcu_read_unlock();
559− dev_put(idev);
560− dev_put(resolved_dev);
598+ ret = get_sgid_index_from_eth(device, port_num,
599+ vlan_id, &dgid,
600+ gid_type, &gid_index);
561601 if (ret)
562602 return ret;
563603
564− ret = get_sgid_index_from_eth(device, port_num, vlan_id,
565− &dgid, gid_type, &gid_index);
566− if (ret)
567− return ret;
568− }
569−
570− rdma_ah_set_dlid(ah_attr, wc->slid);
571− rdma_ah_set_sl(ah_attr, wc->sl);
572− rdma_ah_set_path_bits(ah_attr, wc->dlid_path_bits);
573− rdma_ah_set_port_num(ah_attr, port_num);
604+ flow_class = be32_to_cpu(grh->version_tclass_flow);
605+ rdma_ah_set_grh(ah_attr, &sgid,
606+ flow_class & 0xFFFFF,
607+ (u8)gid_index, hoplimit,
608+ (flow_class >> 20) & 0xFF);
609+ return ib_resolve_unicast_gid_dmac(device, ah_attr);
610+ } else {
611+ rdma_ah_set_dlid(ah_attr, wc->slid);
612+ rdma_ah_set_path_bits(ah_attr, wc->dlid_path_bits);
574613
575− if (wc->wc_flags & IB_WC_GRH) {
576− if (!rdma_cap_eth_ah(device, port_num)) {
614+ if (wc->wc_flags & IB_WC_GRH) {
577615 if (dgid.global.interface_id != cpu_to_be64(IB_SA_WELL_KNOWN_GUID)) {
578616 ret = ib_find_cached_gid_by_port(device, &dgid,
579617 IB_GID_TYPE_IB,
@@ -584,18 +622,17 @@ int ib_init_ah_from_wc(struct ib_device *device, u8 port_num, · 4 unmodified lines
584622 } else {
585623 gid_index = 0;
586624 }
587− }
588−
589− flow_class = be32_to_cpu(grh->version_tclass_flow);
590− rdma_ah_set_grh(ah_attr, &sgid,
591− flow_class & 0xFFFFF,
592− (u8)gid_index, hoplimit,
593− (flow_class >> 20) & 0xFF);
594625
626+ flow_class = be32_to_cpu(grh->version_tclass_flow);
627+ rdma_ah_set_grh(ah_attr, &sgid,
628+ flow_class & 0xFFFFF,
629+ (u8)gid_index, hoplimit,
630+ (flow_class >> 20) & 0xFF);
631+ }
632+ return 0;
595633 }
596− return 0;
597634 }
598−EXPORT_SYMBOL(ib_init_ah_from_wc);
635+EXPORT_SYMBOL(ib_init_ah_attr_from_wc);
599636
600637 struct ib_ah *ib_create_ah_from_wc(struct ib_pd *pd, const struct ib_wc *wc,
601638 const struct ib_grh *grh, u8 port_num)
@@ -603,7 +640,7 @@ struct ib_ah *ib_create_ah_from_wc(struct ib_pd *pd, const struct ib_wc *wc, · 1 unmodified line
603640 struct rdma_ah_attr ah_attr;
604641 int ret;
605642
606− ret = ib_init_ah_from_wc(pd->device, port_num, wc, grh, &ah_attr);
643+ ret = ib_init_ah_attr_from_wc(pd->device, port_num, wc, grh, &ah_attr);
607644 if (ret)
608645 return ERR_PTR(ret);
609646
@@ -850,7 +887,7 @@ struct ib_qp *ib_create_qp(struct ib_pd *pd, · 240 unmodified lines
850887 if (qp_init_attr->cap.max_rdma_ctxs)
851888 rdma_rw_init_qp(device, qp_init_attr);
852889
853− qp = device->create_qp(pd, qp_init_attr, NULL);
890+ qp = _ib_create_qp(device, pd, qp_init_attr, NULL);
854891 if (IS_ERR(qp))
855892 return qp;
856893
@@ -860,7 +897,6 @@ struct ib_qp *ib_create_qp(struct ib_pd *pd, · 3 unmodified lines
860897 return ERR_PTR(ret);
861898 }
862899
863− qp->device = device;
864900 qp->real_qp = qp;
865901 qp->uobject = NULL;
866902 qp->qp_type = qp_init_attr->qp_type;
@@ -890,7 +926,6 @@ struct ib_qp *ib_create_qp(struct ib_pd *pd, · 23 unmodified lines
890926 atomic_inc(&qp_init_attr->srq->usecnt);
891927 }
892928
893− qp->pd = pd;
894929 qp->send_cq = qp_init_attr->send_cq;
895930 qp->xrcd = NULL;
896931
@@ -1269,16 +1304,8 @@ static int ib_resolve_eth_dmac(struct ib_device *device, · 372 unmodified lines
12691304 if (!rdma_is_port_valid(device, rdma_ah_get_port_num(ah_attr)))
12701305 return -EINVAL;
12711306
1272− if (ah_attr->type != RDMA_AH_ATTR_TYPE_ROCE)
1273− return 0;
1274−
12751307 grh = rdma_ah_retrieve_grh(ah_attr);
12761308
1277− if (rdma_link_local_addr((struct in6_addr *)grh->dgid.raw)) {
1278− rdma_get_ll_mac((struct in6_addr *)grh->dgid.raw,
1279− ah_attr->roce.dmac);
1280− return 0;
1281− }
12821309 if (rdma_is_multicast_addr((struct in6_addr *)ah_attr->grh.dgid.raw)) {
12831310 if (ipv6_addr_v4mapped((struct in6_addr *)ah_attr->grh.dgid.raw)) {
12841311 __be32 addr = 0;
@@ -1290,40 +1317,52 @@ static int ib_resolve_eth_dmac(struct ib_device *device, · 5 unmodified lines
12901317 (char *)ah_attr->roce.dmac);
12911318 }
12921319 } else {
1293− union ib_gid sgid;
1294− struct ib_gid_attr sgid_attr;
1295− int ifindex;
1296− int hop_limit;
1297−
1298− ret = ib_query_gid(device,
1299− rdma_ah_get_port_num(ah_attr),
1300− grh->sgid_index,
1301− &sgid, &sgid_attr);
1302−
1303− if (ret || !sgid_attr.ndev) {
1304− if (!ret)
1305− ret = -ENXIO;
1306− goto out;
1307− }
1308−
1309− ifindex = sgid_attr.ndev->ifindex;
1320+ ret = ib_resolve_unicast_gid_dmac(device, ah_attr);
1321+ }
1322+ return ret;
1323+}
13101324
1311− ret =
1312− rdma_addr_find_l2_eth_by_grh(&sgid, &grh->dgid,
1313− ah_attr->roce.dmac,
1314− NULL, &ifindex, &hop_limit);
1325+/**
1326+ * IB core internal function to perform QP attributes modification.
1327+ */
1328+static int _ib_modify_qp(struct ib_qp *qp, struct ib_qp_attr *attr,
1329+ int attr_mask, struct ib_udata *udata)
1330+{
1331+ u8 port = attr_mask & IB_QP_PORT ? attr->port_num : qp->port;
1332+ int ret;
13151333
1316− dev_put(sgid_attr.ndev);
1334+ if (rdma_ib_or_roce(qp->device, port)) {
1335+ if (attr_mask & IB_QP_RQ_PSN && attr->rq_psn & ~0xffffff) {
1336+ pr_warn("%s: %s rq_psn overflow, masking to 24 bits\n",
1337+ __func__, qp->device->name);
1338+ attr->rq_psn &= 0xffffff;
1339+ }
13171340
1318− grh->hop_limit = hop_limit;
1341+ if (attr_mask & IB_QP_SQ_PSN && attr->sq_psn & ~0xffffff) {
1342+ pr_warn("%s: %s sq_psn overflow, masking to 24 bits\n",
1343+ __func__, qp->device->name);
1344+ attr->sq_psn &= 0xffffff;
1345+ }
13191346 }
1320−out:
1347+
1348+ ret = ib_security_modify_qp(qp, attr, attr_mask, udata);
1349+ if (!ret && (attr_mask & IB_QP_PORT))
1350+ qp->port = attr->port_num;
1351+
13211352 return ret;
13221353 }
13231354
1355+static bool is_qp_type_connected(const struct ib_qp *qp)
1356+{
1357+ return (qp->qp_type == IB_QPT_UC ||
1358+ qp->qp_type == IB_QPT_RC ||
1359+ qp->qp_type == IB_QPT_XRC_INI ||
1360+ qp->qp_type == IB_QPT_XRC_TGT);
1361+}
1362+
13241363 /**
13251364 * ib_modify_qp_with_udata - Modifies the attributes for the specified QP.
1326− * @qp: The QP to modify.
1365+ * @ib_qp: The QP to modify.
13271366 * @attr: On input, specifies the QP attributes to modify. On output,
13281367 * the current values of selected QP attributes are returned.
13291368 * @attr_mask: A bit-mask used to specify which attributes of the QP
@@ -1332,21 +1371,20 @@ out: · 2 unmodified lines
13321371 * are being modified.
13331372 * It returns 0 on success and returns appropriate error code on error.
13341373 */
1335−int ib_modify_qp_with_udata(struct ib_qp *qp, struct ib_qp_attr *attr,
1374+int ib_modify_qp_with_udata(struct ib_qp *ib_qp, struct ib_qp_attr *attr,
13361375 int attr_mask, struct ib_udata *udata)
13371376 {
1377+ struct ib_qp *qp = ib_qp->real_qp;
13381378 int ret;
13391379
1340− if (attr_mask & IB_QP_AV) {
1380+ if (attr_mask & IB_QP_AV &&
1381+ attr->ah_attr.type == RDMA_AH_ATTR_TYPE_ROCE &&
1382+ is_qp_type_connected(qp)) {
13411383 ret = ib_resolve_eth_dmac(qp->device, &attr->ah_attr);
13421384 if (ret)
13431385 return ret;
13441386 }
1345− ret = ib_security_modify_qp(qp, attr, attr_mask, udata);
1346− if (!ret && (attr_mask & IB_QP_PORT))
1347− qp->port = attr->port_num;
1348−
1349− return ret;
1387+ return _ib_modify_qp(qp, attr, attr_mask, udata);
13501388 }
13511389 EXPORT_SYMBOL(ib_modify_qp_with_udata);
13521390
@@ -1409,7 +1447,7 @@ int ib_modify_qp(struct ib_qp *qp, · 56 unmodified lines
14091447 struct ib_qp_attr *qp_attr,
14101448 int qp_attr_mask)
14111449 {
1412− return ib_modify_qp_with_udata(qp, qp_attr, qp_attr_mask, NULL);
1450+ return _ib_modify_qp(qp->real_qp, qp_attr, qp_attr_mask, NULL);
14131451 }
14141452 EXPORT_SYMBOL(ib_modify_qp);
14151453
@@ -1503,6 +1541,7 @@ int ib_destroy_qp(struct ib_qp *qp) · 87 unmodified lines
15031541 if (!qp->uobject)
15041542 rdma_rw_cleanup_mrs(qp);
15051543
1544+ rdma_restrack_del(&qp->res);
15061545 ret = qp->device->destroy_qp(qp);
15071546 if (!ret) {
15081547 if (pd)
@@ -1545,6 +1584,8 @@ struct ib_cq *ib_create_cq(struct ib_device *device, · 36 unmodified lines
15451584 cq->event_handler = event_handler;
15461585 cq->cq_context = cq_context;
15471586 atomic_set(&cq->usecnt, 0);
1587+ cq->res.type = RDMA_RESTRACK_CQ;
1588+ rdma_restrack_add(&cq->res);
15481589 }
15491590
15501591 return cq;
@@ -1563,6 +1604,7 @@ int ib_destroy_cq(struct ib_cq *cq) · 12 unmodified lines
15631604 if (atomic_read(&cq->usecnt))
15641605 return -EBUSY;
15651606
1607+ rdma_restrack_del(&cq->res);
15661608 return cq->device->destroy_cq(cq);
15671609 }
15681610 EXPORT_SYMBOL(ib_destroy_cq);
@@ -1747,7 +1789,7 @@ int ib_detach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid) · 178 unmodified lines
17471789 }
17481790 EXPORT_SYMBOL(ib_detach_mcast);
17491791
1750−struct ib_xrcd *ib_alloc_xrcd(struct ib_device *device)
1792+struct ib_xrcd *__ib_alloc_xrcd(struct ib_device *device, const char *caller)
17511793 {
17521794 struct ib_xrcd *xrcd;
17531795
@@ -1765,7 +1807,7 @@ struct ib_xrcd *ib_alloc_xrcd(struct ib_device *device) · 11 unmodified lines
17651807
17661808 return xrcd;
17671809 }
1768−EXPORT_SYMBOL(ib_alloc_xrcd);
1810+EXPORT_SYMBOL(__ib_alloc_xrcd);
17691811
17701812 int ib_dealloc_xrcd(struct ib_xrcd *xrcd)
17711813 {
@@ -1790,11 +1832,11 @@ EXPORT_SYMBOL(ib_dealloc_xrcd); · 18 unmodified lines
17901832 * ib_create_wq - Creates a WQ associated with the specified protection
17911833 * domain.
17921834 * @pd: The protection domain associated with the WQ.
1793− * @wq_init_attr: A list of initial attributes required to create the
1835+ * @wq_attr: A list of initial attributes required to create the
17941836 * WQ. If WQ creation succeeds, then the attributes are updated to
17951837 * the actual capabilities of the created WQ.
17961838 *
1797− * wq_init_attr->max_wr and wq_init_attr->max_sge determine
1839+ * wq_attr->max_wr and wq_attr->max_sge determine
17981840 * the requested size of the WQ, and set to the actual values allocated
17991841 * on return.
18001842 * If ib_create_wq() succeeds, then max_wr and max_sge will always be
@@ -2156,16 +2198,16 @@ static void __ib_drain_sq(struct ib_qp *qp) · 355 unmodified lines
21562198 struct ib_send_wr swr = {}, *bad_swr;
21572199 int ret;
21582200
2159− swr.wr_cqe = &sdrain.cqe;
2160− sdrain.cqe.done = ib_drain_qp_done;
2161− init_completion(&sdrain.done);
2162−
21632201 ret = ib_modify_qp(qp, &attr, IB_QP_STATE);
21642202 if (ret) {
21652203 WARN_ONCE(ret, "failed to drain send queue: %d\n", ret);
21662204 return;
21672205 }
21682206
2207+ swr.wr_cqe = &sdrain.cqe;
2208+ sdrain.cqe.done = ib_drain_qp_done;
2209+ init_completion(&sdrain.done);
2210+
21692211 ret = ib_post_send(qp, &swr, &bad_swr);
21702212 if (ret) {
21712213 WARN_ONCE(ret, "failed to drain send queue: %d\n", ret);
@@ -2190,16 +2232,16 @@ static void __ib_drain_rq(struct ib_qp *qp) · 18 unmodified lines
21902232 struct ib_recv_wr rwr = {}, *bad_rwr;
21912233 int ret;
21922234
2193− rwr.wr_cqe = &rdrain.cqe;
2194− rdrain.cqe.done = ib_drain_qp_done;
2195− init_completion(&rdrain.done);
2196−
21972235 ret = ib_modify_qp(qp, &attr, IB_QP_STATE);
21982236 if (ret) {
21992237 WARN_ONCE(ret, "failed to drain recv queue: %d\n", ret);
22002238 return;
22012239 }
22022240
2241+ rwr.wr_cqe = &rdrain.cqe;
2242+ rdrain.cqe.done = ib_drain_qp_done;
2243+ init_completion(&rdrain.done);
2244+
22032245 ret = ib_post_recv(qp, &rwr, &bad_rwr);
22042246 if (ret) {
22052247 WARN_ONCE(ret, "failed to drain recv queue: %d\n", ret);
drivers/infiniband/hw/bnxt_re/bnxt_re.h+34−9
@@ -43,20 +43,41 @@ · 42 unmodified lines
4343 #define ROCE_DRV_MODULE_VERSION "1.0.0"
4444
4545 #define BNXT_RE_DESC "Broadcom NetXtreme-C/E RoCE Driver"
46−
47−#define BNXT_RE_PAGE_SIZE_4K BIT(12)
48−#define BNXT_RE_PAGE_SIZE_8K BIT(13)
49−#define BNXT_RE_PAGE_SIZE_64K BIT(16)
50−#define BNXT_RE_PAGE_SIZE_2M BIT(21)
51−#define BNXT_RE_PAGE_SIZE_8M BIT(23)
52−#define BNXT_RE_PAGE_SIZE_1G BIT(30)
53−
54−#define BNXT_RE_MAX_MR_SIZE BIT(30)
46+#define BNXT_RE_PAGE_SHIFT_4K (12)
47+#define BNXT_RE_PAGE_SHIFT_8K (13)
48+#define BNXT_RE_PAGE_SHIFT_64K (16)
49+#define BNXT_RE_PAGE_SHIFT_2M (21)
50+#define BNXT_RE_PAGE_SHIFT_8M (23)
51+#define BNXT_RE_PAGE_SHIFT_1G (30)
52+
53+#define BNXT_RE_PAGE_SIZE_4K BIT(BNXT_RE_PAGE_SHIFT_4K)
54+#define BNXT_RE_PAGE_SIZE_8K BIT(BNXT_RE_PAGE_SHIFT_8K)
55+#define BNXT_RE_PAGE_SIZE_64K BIT(BNXT_RE_PAGE_SHIFT_64K)
56+#define BNXT_RE_PAGE_SIZE_2M BIT(BNXT_RE_PAGE_SHIFT_2M)
57+#define BNXT_RE_PAGE_SIZE_8M BIT(BNXT_RE_PAGE_SHIFT_8M)
58+#define BNXT_RE_PAGE_SIZE_1G BIT(BNXT_RE_PAGE_SHIFT_1G)
59+
60+#define BNXT_RE_MAX_MR_SIZE_LOW BIT(BNXT_RE_PAGE_SHIFT_1G)
61+#define BNXT_RE_MAX_MR_SIZE_HIGH BIT(39)
62+#define BNXT_RE_MAX_MR_SIZE BNXT_RE_MAX_MR_SIZE_HIGH
5563
5664 #define BNXT_RE_MAX_QPC_COUNT (64 * 1024)
5765 #define BNXT_RE_MAX_MRW_COUNT (64 * 1024)
5866 #define BNXT_RE_MAX_SRQC_COUNT (64 * 1024)
5967 #define BNXT_RE_MAX_CQ_COUNT (64 * 1024)
68+#define BNXT_RE_MAX_MRW_COUNT_64K (64 * 1024)
69+#define BNXT_RE_MAX_MRW_COUNT_256K (256 * 1024)
70+
71+/* Number of MRs to reserve for PF, leaving remainder for VFs */
72+#define BNXT_RE_RESVD_MR_FOR_PF (32 * 1024)
73+#define BNXT_RE_MAX_GID_PER_VF 128
74+
75+/*
76+ * Percentage of resources of each type reserved for PF.
77+ * Remaining resources are divided equally among VFs.
78+ * [0, 100]
79+ */
80+#define BNXT_RE_PCT_RSVD_FOR_PF 50
6081
6182 #define BNXT_RE_UD_QP_HW_STALL 0x400000
6283
@@ -100,6 +121,7 @@ struct bnxt_re_dev { · 37 unmodified lines
100121 #define BNXT_RE_FLAG_RCFW_CHANNEL_EN 4
101122 #define BNXT_RE_FLAG_QOS_WORK_REG 5
102123 #define BNXT_RE_FLAG_TASK_IN_PROG 6
124+#define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29
103125 struct net_device *netdev;
104126 unsigned int version, major, minor;
105127 struct bnxt_en_dev *en_dev;
@@ -145,6 +167,9 @@ struct bnxt_re_dev { · 39 unmodified lines
145167 struct bnxt_re_ah *sqp_ah;
146168 struct bnxt_re_sqp_entries sqp_tbl[1024];
147169 atomic_t nq_alloc_cnt;
170+ u32 is_virtfn;
171+ u32 num_vfs;
172+ struct bnxt_qplib_roce_stats stats;
148173 };
149174
150175 #define to_bnxt_re_dev(ptr, member) \
drivers/infiniband/hw/bnxt_re/hw_counters.c+135−10
@@ -58,16 +58,55 @@ · 57 unmodified lines
5858 #include "hw_counters.h"
5959
6060 static const char * const bnxt_re_stat_name[] = {
61− [BNXT_RE_ACTIVE_QP] = "active_qps",
62− [BNXT_RE_ACTIVE_SRQ] = "active_srqs",
63− [BNXT_RE_ACTIVE_CQ] = "active_cqs",
64− [BNXT_RE_ACTIVE_MR] = "active_mrs",
65− [BNXT_RE_ACTIVE_MW] = "active_mws",
66− [BNXT_RE_RX_PKTS] = "rx_pkts",
67− [BNXT_RE_RX_BYTES] = "rx_bytes",
68− [BNXT_RE_TX_PKTS] = "tx_pkts",
69− [BNXT_RE_TX_BYTES] = "tx_bytes",
70− [BNXT_RE_RECOVERABLE_ERRORS] = "recoverable_errors"
61+ [BNXT_RE_ACTIVE_QP] = "active_qps",
62+ [BNXT_RE_ACTIVE_SRQ] = "active_srqs",
63+ [BNXT_RE_ACTIVE_CQ] = "active_cqs",
64+ [BNXT_RE_ACTIVE_MR] = "active_mrs",
65+ [BNXT_RE_ACTIVE_MW] = "active_mws",
66+ [BNXT_RE_RX_PKTS] = "rx_pkts",
67+ [BNXT_RE_RX_BYTES] = "rx_bytes",
68+ [BNXT_RE_TX_PKTS] = "tx_pkts",
69+ [BNXT_RE_TX_BYTES] = "tx_bytes",
70+ [BNXT_RE_RECOVERABLE_ERRORS] = "recoverable_errors",
71+ [BNXT_RE_TO_RETRANSMITS] = "to_retransmits",
72+ [BNXT_RE_SEQ_ERR_NAKS_RCVD] = "seq_err_naks_rcvd",
73+ [BNXT_RE_MAX_RETRY_EXCEEDED] = "max_retry_exceeded",
74+ [BNXT_RE_RNR_NAKS_RCVD] = "rnr_naks_rcvd",
75+ [BNXT_RE_MISSING_RESP] = "missin_resp",
76+ [BNXT_RE_UNRECOVERABLE_ERR] = "unrecoverable_err",
77+ [BNXT_RE_BAD_RESP_ERR] = "bad_resp_err",
78+ [BNXT_RE_LOCAL_QP_OP_ERR] = "local_qp_op_err",
79+ [BNXT_RE_LOCAL_PROTECTION_ERR] = "local_protection_err",
80+ [BNXT_RE_MEM_MGMT_OP_ERR] = "mem_mgmt_op_err",
81+ [BNXT_RE_REMOTE_INVALID_REQ_ERR] = "remote_invalid_req_err",
82+ [BNXT_RE_REMOTE_ACCESS_ERR] = "remote_access_err",
83+ [BNXT_RE_REMOTE_OP_ERR] = "remote_op_err",
84+ [BNXT_RE_DUP_REQ] = "dup_req",
85+ [BNXT_RE_RES_EXCEED_MAX] = "res_exceed_max",
86+ [BNXT_RE_RES_LENGTH_MISMATCH] = "res_length_mismatch",
87+ [BNXT_RE_RES_EXCEEDS_WQE] = "res_exceeds_wqe",
88+ [BNXT_RE_RES_OPCODE_ERR] = "res_opcode_err",
89+ [BNXT_RE_RES_RX_INVALID_RKEY] = "res_rx_invalid_rkey",
90+ [BNXT_RE_RES_RX_DOMAIN_ERR] = "res_rx_domain_err",
91+ [BNXT_RE_RES_RX_NO_PERM] = "res_rx_no_perm",
92+ [BNXT_RE_RES_RX_RANGE_ERR] = "res_rx_range_err",
93+ [BNXT_RE_RES_TX_INVALID_RKEY] = "res_tx_invalid_rkey",
94+ [BNXT_RE_RES_TX_DOMAIN_ERR] = "res_tx_domain_err",
95+ [BNXT_RE_RES_TX_NO_PERM] = "res_tx_no_perm",
96+ [BNXT_RE_RES_TX_RANGE_ERR] = "res_tx_range_err",
97+ [BNXT_RE_RES_IRRQ_OFLOW] = "res_irrq_oflow",
98+ [BNXT_RE_RES_UNSUP_OPCODE] = "res_unsup_opcode",
99+ [BNXT_RE_RES_UNALIGNED_ATOMIC] = "res_unaligned_atomic",
100+ [BNXT_RE_RES_REM_INV_ERR] = "res_rem_inv_err",
101+ [BNXT_RE_RES_MEM_ERROR] = "res_mem_err",
102+ [BNXT_RE_RES_SRQ_ERR] = "res_srq_err",
103+ [BNXT_RE_RES_CMP_ERR] = "res_cmp_err",
104+ [BNXT_RE_RES_INVALID_DUP_RKEY] = "res_invalid_dup_rkey",
105+ [BNXT_RE_RES_WQE_FORMAT_ERR] = "res_wqe_format_err",
106+ [BNXT_RE_RES_CQ_LOAD_ERR] = "res_cq_load_err",
107+ [BNXT_RE_RES_SRQ_LOAD_ERR] = "res_srq_load_err",
108+ [BNXT_RE_RES_TX_PCI_ERR] = "res_tx_pci_err",
109+ [BNXT_RE_RES_RX_PCI_ERR] = "res_rx_pci_err"
71110 };
72111
73112 int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev,
@@ -76,6 +115,7 @@ int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev, · 2 unmodified lines
76115 {
77116 struct bnxt_re_dev *rdev = to_bnxt_re_dev(ibdev, ibdev);
78117 struct ctx_hw_stats *bnxt_re_stats = rdev->qplib_ctx.stats.dma;
118+ int rc = 0;
79119
80120 if (!port || !stats)
81121 return -EINVAL;
@@ -97,6 +137,91 @@ int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev, · 15 unmodified lines
97137 stats->value[BNXT_RE_TX_BYTES] =
98138 le64_to_cpu(bnxt_re_stats->tx_ucast_bytes);
99139 }
140+ if (test_bit(BNXT_RE_FLAG_ISSUE_ROCE_STATS, &rdev->flags)) {
141+ rc = bnxt_qplib_get_roce_stats(&rdev->rcfw, &rdev->stats);
142+ if (rc)
143+ clear_bit(BNXT_RE_FLAG_ISSUE_ROCE_STATS,
144+ &rdev->flags);
145+ stats->value[BNXT_RE_TO_RETRANSMITS] =
146+ rdev->stats.to_retransmits;
147+ stats->value[BNXT_RE_SEQ_ERR_NAKS_RCVD] =
148+ rdev->stats.seq_err_naks_rcvd;
149+ stats->value[BNXT_RE_MAX_RETRY_EXCEEDED] =
150+ rdev->stats.max_retry_exceeded;
151+ stats->value[BNXT_RE_RNR_NAKS_RCVD] =
152+ rdev->stats.rnr_naks_rcvd;
153+ stats->value[BNXT_RE_MISSING_RESP] =
154+ rdev->stats.missing_resp;
155+ stats->value[BNXT_RE_UNRECOVERABLE_ERR] =
156+ rdev->stats.unrecoverable_err;
157+ stats->value[BNXT_RE_BAD_RESP_ERR] =
158+ rdev->stats.bad_resp_err;
159+ stats->value[BNXT_RE_LOCAL_QP_OP_ERR] =
160+ rdev->stats.local_qp_op_err;
161+ stats->value[BNXT_RE_LOCAL_PROTECTION_ERR] =
162+ rdev->stats.local_protection_err;
163+ stats->value[BNXT_RE_MEM_MGMT_OP_ERR] =
164+ rdev->stats.mem_mgmt_op_err;
165+ stats->value[BNXT_RE_REMOTE_INVALID_REQ_ERR] =
166+ rdev->stats.remote_invalid_req_err;
167+ stats->value[BNXT_RE_REMOTE_ACCESS_ERR] =
168+ rdev->stats.remote_access_err;
169+ stats->value[BNXT_RE_REMOTE_OP_ERR] =
170+ rdev->stats.remote_op_err;
171+ stats->value[BNXT_RE_DUP_REQ] =
172+ rdev->stats.dup_req;
173+ stats->value[BNXT_RE_RES_EXCEED_MAX] =
174+ rdev->stats.res_exceed_max;
175+ stats->value[BNXT_RE_RES_LENGTH_MISMATCH] =
176+ rdev->stats.res_length_mismatch;
177+ stats->value[BNXT_RE_RES_EXCEEDS_WQE] =
178+ rdev->stats.res_exceeds_wqe;
179+ stats->value[BNXT_RE_RES_OPCODE_ERR] =
180+ rdev->stats.res_opcode_err;
181+ stats->value[BNXT_RE_RES_RX_INVALID_RKEY] =
182+ rdev->stats.res_rx_invalid_rkey;
183+ stats->value[BNXT_RE_RES_RX_DOMAIN_ERR] =
184+ rdev->stats.res_rx_domain_err;
185+ stats->value[BNXT_RE_RES_RX_NO_PERM] =
186+ rdev->stats.res_rx_no_perm;
187+ stats->value[BNXT_RE_RES_RX_RANGE_ERR] =
188+ rdev->stats.res_rx_range_err;
189+ stats->value[BNXT_RE_RES_TX_INVALID_RKEY] =
190+ rdev->stats.res_tx_invalid_rkey;
191+ stats->value[BNXT_RE_RES_TX_DOMAIN_ERR] =
192+ rdev->stats.res_tx_domain_err;
193+ stats->value[BNXT_RE_RES_TX_NO_PERM] =
194+ rdev->stats.res_tx_no_perm;
195+ stats->value[BNXT_RE_RES_TX_RANGE_ERR] =
196+ rdev->stats.res_tx_range_err;
197+ stats->value[BNXT_RE_RES_IRRQ_OFLOW] =
198+ rdev->stats.res_irrq_oflow;
199+ stats->value[BNXT_RE_RES_UNSUP_OPCODE] =
200+ rdev->stats.res_unsup_opcode;
201+ stats->value[BNXT_RE_RES_UNALIGNED_ATOMIC] =
202+ rdev->stats.res_unaligned_atomic;
203+ stats->value[BNXT_RE_RES_REM_INV_ERR] =
204+ rdev->stats.res_rem_inv_err;
205+ stats->value[BNXT_RE_RES_MEM_ERROR] =
206+ rdev->stats.res_mem_error;
207+ stats->value[BNXT_RE_RES_SRQ_ERR] =
208+ rdev->stats.res_srq_err;
209+ stats->value[BNXT_RE_RES_CMP_ERR] =
210+ rdev->stats.res_cmp_err;
211+ stats->value[BNXT_RE_RES_INVALID_DUP_RKEY] =
212+ rdev->stats.res_invalid_dup_rkey;
213+ stats->value[BNXT_RE_RES_WQE_FORMAT_ERR] =
214+ rdev->stats.res_wqe_format_err;
215+ stats->value[BNXT_RE_RES_CQ_LOAD_ERR] =
216+ rdev->stats.res_cq_load_err;
217+ stats->value[BNXT_RE_RES_SRQ_LOAD_ERR] =
218+ rdev->stats.res_srq_load_err;
219+ stats->value[BNXT_RE_RES_TX_PCI_ERR] =
220+ rdev->stats.res_tx_pci_err;
221+ stats->value[BNXT_RE_RES_RX_PCI_ERR] =
222+ rdev->stats.res_rx_pci_err;
223+ }
224+
100225 return ARRAY_SIZE(bnxt_re_stat_name);
101226 }
102227
drivers/infiniband/hw/bnxt_re/hw_counters.h+39−0
@@ -51,6 +51,45 @@ enum bnxt_re_hw_stats { · 50 unmodified lines
5151 BNXT_RE_TX_PKTS,
5252 BNXT_RE_TX_BYTES,
5353 BNXT_RE_RECOVERABLE_ERRORS,
54+ BNXT_RE_TO_RETRANSMITS,
55+ BNXT_RE_SEQ_ERR_NAKS_RCVD,
56+ BNXT_RE_MAX_RETRY_EXCEEDED,
57+ BNXT_RE_RNR_NAKS_RCVD,
58+ BNXT_RE_MISSING_RESP,
59+ BNXT_RE_UNRECOVERABLE_ERR,
60+ BNXT_RE_BAD_RESP_ERR,
61+ BNXT_RE_LOCAL_QP_OP_ERR,
62+ BNXT_RE_LOCAL_PROTECTION_ERR,
63+ BNXT_RE_MEM_MGMT_OP_ERR,
64+ BNXT_RE_REMOTE_INVALID_REQ_ERR,
65+ BNXT_RE_REMOTE_ACCESS_ERR,
66+ BNXT_RE_REMOTE_OP_ERR,
67+ BNXT_RE_DUP_REQ,
68+ BNXT_RE_RES_EXCEED_MAX,
69+ BNXT_RE_RES_LENGTH_MISMATCH,
70+ BNXT_RE_RES_EXCEEDS_WQE,
71+ BNXT_RE_RES_OPCODE_ERR,
72+ BNXT_RE_RES_RX_INVALID_RKEY,
73+ BNXT_RE_RES_RX_DOMAIN_ERR,
74+ BNXT_RE_RES_RX_NO_PERM,
75+ BNXT_RE_RES_RX_RANGE_ERR,
76+ BNXT_RE_RES_TX_INVALID_RKEY,
77+ BNXT_RE_RES_TX_DOMAIN_ERR,
78+ BNXT_RE_RES_TX_NO_PERM,
79+ BNXT_RE_RES_TX_RANGE_ERR,
80+ BNXT_RE_RES_IRRQ_OFLOW,
81+ BNXT_RE_RES_UNSUP_OPCODE,
82+ BNXT_RE_RES_UNALIGNED_ATOMIC,
83+ BNXT_RE_RES_REM_INV_ERR,
84+ BNXT_RE_RES_MEM_ERROR,
85+ BNXT_RE_RES_SRQ_ERR,
86+ BNXT_RE_RES_CMP_ERR,
87+ BNXT_RE_RES_INVALID_DUP_RKEY,
88+ BNXT_RE_RES_WQE_FORMAT_ERR,
89+ BNXT_RE_RES_CQ_LOAD_ERR,
90+ BNXT_RE_RES_SRQ_LOAD_ERR,
91+ BNXT_RE_RES_TX_PCI_ERR,
92+ BNXT_RE_RES_RX_PCI_ERR,
5493 BNXT_RE_NUM_COUNTERS
5594 };
5695
drivers/infiniband/hw/bnxt_re/ib_verbs.c+348−56
@@ -141,12 +141,13 @@ int bnxt_re_query_device(struct ib_device *ibdev, · 140 unmodified lines
141141 struct bnxt_qplib_dev_attr *dev_attr = &rdev->dev_attr;
142142
143143 memset(ib_attr, 0, sizeof(*ib_attr));
144−
145− ib_attr->fw_ver = (u64)(unsigned long)(dev_attr->fw_ver);
144+ memcpy(&ib_attr->fw_ver, dev_attr->fw_ver,
145+ min(sizeof(dev_attr->fw_ver),
146+ sizeof(ib_attr->fw_ver)));
146147 bnxt_qplib_get_guid(rdev->netdev->dev_addr,
147148 (u8 *)&ib_attr->sys_image_guid);
148149 ib_attr->max_mr_size = BNXT_RE_MAX_MR_SIZE;
149− ib_attr->page_size_cap = BNXT_RE_PAGE_SIZE_4K;
150+ ib_attr->page_size_cap = BNXT_RE_PAGE_SIZE_4K | BNXT_RE_PAGE_SIZE_2M;
150151
151152 ib_attr->vendor_id = rdev->en_dev->pdev->vendor;
152153 ib_attr->vendor_part_id = rdev->en_dev->pdev->device;
@@ -247,8 +248,7 @@ int bnxt_re_query_port(struct ib_device *ibdev, u8 port_num, · 94 unmodified lines
247248 IB_PORT_VENDOR_CLASS_SUP |
248249 IB_PORT_IP_BASED_GIDS;
249250
250− /* Max MSG size set to 2G for now */
251− port_attr->max_msg_sz = 0x80000000;
251+ port_attr->max_msg_sz = (u32)BNXT_RE_MAX_MR_SIZE_LOW;
252252 port_attr->bad_pkey_cntr = 0;
253253 port_attr->qkey_viol_cntr = 0;
254254 port_attr->pkey_tbl_len = dev_attr->max_pkey;
@@ -281,6 +281,15 @@ int bnxt_re_get_port_immutable(struct ib_device *ibdev, u8 port_num, · 26 unmodified lines
281281 return 0;
282282 }
283283
284+void bnxt_re_query_fw_str(struct ib_device *ibdev, char *str)
285+{
286+ struct bnxt_re_dev *rdev = to_bnxt_re_dev(ibdev, ibdev);
287+
288+ snprintf(str, IB_FW_VERSION_NAME_MAX, "%d.%d.%d.%d",
289+ rdev->dev_attr.fw_ver[0], rdev->dev_attr.fw_ver[1],
290+ rdev->dev_attr.fw_ver[2], rdev->dev_attr.fw_ver[3]);
291+}
292+
284293 int bnxt_re_query_pkey(struct ib_device *ibdev, u8 port_num,
285294 u16 index, u16 *pkey)
286295 {
@@ -532,7 +541,7 @@ static int bnxt_re_create_fence_mr(struct bnxt_re_pd *pd) · 245 unmodified lines
532541 mr->qplib_mr.total_size = BNXT_RE_FENCE_BYTES;
533542 pbl_tbl = dma_addr;
534543 rc = bnxt_qplib_reg_mr(&rdev->qplib_res, &mr->qplib_mr, &pbl_tbl,
535− BNXT_RE_FENCE_PBL_SIZE, false);
544+ BNXT_RE_FENCE_PBL_SIZE, false, PAGE_SIZE);
536545 if (rc) {
537546 dev_err(rdev_to_dev(rdev), "Failed to register fence-MR\n");
538547 goto fail;
@@ -1018,6 +1027,7 @@ struct ib_qp *bnxt_re_create_qp(struct ib_pd *ib_pd, · 479 unmodified lines
10181027 struct bnxt_qplib_dev_attr *dev_attr = &rdev->dev_attr;
10191028 struct bnxt_re_qp *qp;
10201029 struct bnxt_re_cq *cq;
1030+ struct bnxt_re_srq *srq;
10211031 int rc, entries;
10221032
10231033 if ((qp_init_attr->cap.max_send_wr > dev_attr->max_qp_wqes) ||
@@ -1073,9 +1083,15 @@ struct ib_qp *bnxt_re_create_qp(struct ib_pd *ib_pd, · 49 unmodified lines
10731083 }
10741084
10751085 if (qp_init_attr->srq) {
1076− dev_err(rdev_to_dev(rdev), "SRQ not supported");
1077− rc = -ENOTSUPP;
1078− goto fail;
1086+ srq = container_of(qp_init_attr->srq, struct bnxt_re_srq,
1087+ ib_srq);
1088+ if (!srq) {
1089+ dev_err(rdev_to_dev(rdev), "SRQ not found");
1090+ rc = -EINVAL;
1091+ goto fail;
1092+ }
1093+ qp->qplib_qp.srq = &srq->qplib_srq;
1094+ qp->qplib_qp.rq.max_wqe = 0;
10791095 } else {
10801096 /* Allocate 1 more than what's provided so posting max doesn't
10811097 * mean empty
@@ -1280,6 +1296,237 @@ static enum ib_mtu __to_ib_mtu(u32 mtu) · 198 unmodified lines
12801296 }
12811297 }
12821298
1299+/* Shared Receive Queues */
1300+int bnxt_re_destroy_srq(struct ib_srq *ib_srq)
1301+{
1302+ struct bnxt_re_srq *srq = container_of(ib_srq, struct bnxt_re_srq,
1303+ ib_srq);
1304+ struct bnxt_re_dev *rdev = srq->rdev;
1305+ struct bnxt_qplib_srq *qplib_srq = &srq->qplib_srq;
1306+ struct bnxt_qplib_nq *nq = NULL;
1307+ int rc;
1308+
1309+ if (qplib_srq->cq)
1310+ nq = qplib_srq->cq->nq;
1311+ rc = bnxt_qplib_destroy_srq(&rdev->qplib_res, qplib_srq);
1312+ if (rc) {
1313+ dev_err(rdev_to_dev(rdev), "Destroy HW SRQ failed!");
1314+ return rc;
1315+ }
1316+
1317+ if (srq->umem && !IS_ERR(srq->umem))
1318+ ib_umem_release(srq->umem);
1319+ kfree(srq);
1320+ atomic_dec(&rdev->srq_count);
1321+ if (nq)
1322+ nq->budget--;
1323+ return 0;
1324+}
1325+
1326+static int bnxt_re_init_user_srq(struct bnxt_re_dev *rdev,
1327+ struct bnxt_re_pd *pd,
1328+ struct bnxt_re_srq *srq,
1329+ struct ib_udata *udata)
1330+{
1331+ struct bnxt_re_srq_req ureq;
1332+ struct bnxt_qplib_srq *qplib_srq = &srq->qplib_srq;
1333+ struct ib_umem *umem;
1334+ int bytes = 0;
1335+ struct ib_ucontext *context = pd->ib_pd.uobject->context;
1336+ struct bnxt_re_ucontext *cntx = container_of(context,
1337+ struct bnxt_re_ucontext,
1338+ ib_uctx);
1339+ if (ib_copy_from_udata(&ureq, udata, sizeof(ureq)))
1340+ return -EFAULT;
1341+
1342+ bytes = (qplib_srq->max_wqe * BNXT_QPLIB_MAX_RQE_ENTRY_SIZE);
1343+ bytes = PAGE_ALIGN(bytes);
1344+ umem = ib_umem_get(context, ureq.srqva, bytes,
1345+ IB_ACCESS_LOCAL_WRITE, 1);
1346+ if (IS_ERR(umem))
1347+ return PTR_ERR(umem);
1348+
1349+ srq->umem = umem;
1350+ qplib_srq->nmap = umem->nmap;
1351+ qplib_srq->sglist = umem->sg_head.sgl;
1352+ qplib_srq->srq_handle = ureq.srq_handle;
1353+ qplib_srq->dpi = &cntx->dpi;
1354+
1355+ return 0;
1356+}
1357+
1358+struct ib_srq *bnxt_re_create_srq(struct ib_pd *ib_pd,
1359+ struct ib_srq_init_attr *srq_init_attr,
1360+ struct ib_udata *udata)
1361+{
1362+ struct bnxt_re_pd *pd = container_of(ib_pd, struct bnxt_re_pd, ib_pd);
1363+ struct bnxt_re_dev *rdev = pd->rdev;
1364+ struct bnxt_qplib_dev_attr *dev_attr = &rdev->dev_attr;
1365+ struct bnxt_re_srq *srq;
1366+ struct bnxt_qplib_nq *nq = NULL;
1367+ int rc, entries;
1368+
1369+ if (srq_init_attr->attr.max_wr >= dev_attr->max_srq_wqes) {
1370+ dev_err(rdev_to_dev(rdev), "Create CQ failed - max exceeded");
1371+ rc = -EINVAL;
1372+ goto exit;
1373+ }
1374+
1375+ if (srq_init_attr->srq_type != IB_SRQT_BASIC) {
1376+ rc = -ENOTSUPP;
1377+ goto exit;
1378+ }
1379+
1380+ srq = kzalloc(sizeof(*srq), GFP_KERNEL);
1381+ if (!srq) {
1382+ rc = -ENOMEM;
1383+ goto exit;
1384+ }
1385+ srq->rdev = rdev;
1386+ srq->qplib_srq.pd = &pd->qplib_pd;
1387+ srq->qplib_srq.dpi = &rdev->dpi_privileged;
1388+ /* Allocate 1 more than what's provided so posting max doesn't
1389+ * mean empty
1390+ */
1391+ entries = roundup_pow_of_two(srq_init_attr->attr.max_wr + 1);
1392+ if (entries > dev_attr->max_srq_wqes + 1)
1393+ entries = dev_attr->max_srq_wqes + 1;
1394+
1395+ srq->qplib_srq.max_wqe = entries;
1396+ srq->qplib_srq.max_sge = srq_init_attr->attr.max_sge;
1397+ srq->qplib_srq.threshold = srq_init_attr->attr.srq_limit;
1398+ srq->srq_limit = srq_init_attr->attr.srq_limit;
1399+ srq->qplib_srq.eventq_hw_ring_id = rdev->nq[0].ring_id;
1400+ nq = &rdev->nq[0];
1401+
1402+ if (udata) {
1403+ rc = bnxt_re_init_user_srq(rdev, pd, srq, udata);
1404+ if (rc)
1405+ goto fail;
1406+ }
1407+
1408+ rc = bnxt_qplib_create_srq(&rdev->qplib_res, &srq->qplib_srq);
1409+ if (rc) {
1410+ dev_err(rdev_to_dev(rdev), "Create HW SRQ failed!");
1411+ goto fail;
1412+ }
1413+
1414+ if (udata) {
1415+ struct bnxt_re_srq_resp resp;
1416+
1417+ resp.srqid = srq->qplib_srq.id;
1418+ rc = ib_copy_to_udata(udata, &resp, sizeof(resp));
1419+ if (rc) {
1420+ dev_err(rdev_to_dev(rdev), "SRQ copy to udata failed!");
1421+ bnxt_qplib_destroy_srq(&rdev->qplib_res,
1422+ &srq->qplib_srq);
1423+ goto exit;
1424+ }
1425+ }
1426+ if (nq)
1427+ nq->budget++;
1428+ atomic_inc(&rdev->srq_count);
1429+
1430+ return &srq->ib_srq;
1431+
1432+fail:
1433+ if (udata && srq->umem && !IS_ERR(srq->umem)) {
1434+ ib_umem_release(srq->umem);
1435+ srq->umem = NULL;
1436+ }
1437+
1438+ kfree(srq);
1439+exit:
1440+ return ERR_PTR(rc);
1441+}
1442+
1443+int bnxt_re_modify_srq(struct ib_srq *ib_srq, struct ib_srq_attr *srq_attr,
1444+ enum ib_srq_attr_mask srq_attr_mask,
1445+ struct ib_udata *udata)
1446+{
1447+ struct bnxt_re_srq *srq = container_of(ib_srq, struct bnxt_re_srq,
1448+ ib_srq);
1449+ struct bnxt_re_dev *rdev = srq->rdev;
1450+ int rc;
1451+
1452+ switch (srq_attr_mask) {
1453+ case IB_SRQ_MAX_WR:
1454+ /* SRQ resize is not supported */
1455+ break;
1456+ case IB_SRQ_LIMIT:
1457+ /* Change the SRQ threshold */
1458+ if (srq_attr->srq_limit > srq->qplib_srq.max_wqe)
1459+ return -EINVAL;
1460+
1461+ srq->qplib_srq.threshold = srq_attr->srq_limit;
1462+ rc = bnxt_qplib_modify_srq(&rdev->qplib_res, &srq->qplib_srq);
1463+ if (rc) {
1464+ dev_err(rdev_to_dev(rdev), "Modify HW SRQ failed!");
1465+ return rc;
1466+ }
1467+ /* On success, update the shadow */
1468+ srq->srq_limit = srq_attr->srq_limit;
1469+ /* No need to Build and send response back to udata */
1470+ break;
1471+ default:
1472+ dev_err(rdev_to_dev(rdev),
1473+ "Unsupported srq_attr_mask 0x%x", srq_attr_mask);
1474+ return -EINVAL;
1475+ }
1476+ return 0;
1477+}
1478+
1479+int bnxt_re_query_srq(struct ib_srq *ib_srq, struct ib_srq_attr *srq_attr)
1480+{
1481+ struct bnxt_re_srq *srq = container_of(ib_srq, struct bnxt_re_srq,
1482+ ib_srq);
1483+ struct bnxt_re_srq tsrq;
1484+ struct bnxt_re_dev *rdev = srq->rdev;
1485+ int rc;
1486+
1487+ /* Get live SRQ attr */
1488+ tsrq.qplib_srq.id = srq->qplib_srq.id;
1489+ rc = bnxt_qplib_query_srq(&rdev->qplib_res, &tsrq.qplib_srq);
1490+ if (rc) {
1491+ dev_err(rdev_to_dev(rdev), "Query HW SRQ failed!");
1492+ return rc;
1493+ }
1494+ srq_attr->max_wr = srq->qplib_srq.max_wqe;
1495+ srq_attr->max_sge = srq->qplib_srq.max_sge;
1496+ srq_attr->srq_limit = tsrq.qplib_srq.threshold;
1497+
1498+ return 0;
1499+}
1500+
1501+int bnxt_re_post_srq_recv(struct ib_srq *ib_srq, struct ib_recv_wr *wr,
1502+ struct ib_recv_wr **bad_wr)
1503+{
1504+ struct bnxt_re_srq *srq = container_of(ib_srq, struct bnxt_re_srq,
1505+ ib_srq);
1506+ struct bnxt_qplib_swqe wqe;
1507+ unsigned long flags;
1508+ int rc = 0, payload_sz = 0;
1509+
1510+ spin_lock_irqsave(&srq->lock, flags);
1511+ while (wr) {
1512+ /* Transcribe each ib_recv_wr to qplib_swqe */
1513+ wqe.num_sge = wr->num_sge;
1514+ payload_sz = bnxt_re_build_sgl(wr->sg_list, wqe.sg_list,
1515+ wr->num_sge);
1516+ wqe.wr_id = wr->wr_id;
1517+ wqe.type = BNXT_QPLIB_SWQE_TYPE_RECV;
1518+
1519+ rc = bnxt_qplib_post_srq_recv(&srq->qplib_srq, &wqe);
1520+ if (rc) {
1521+ *bad_wr = wr;
1522+ break;
1523+ }
1524+ wr = wr->next;
1525+ }
1526+ spin_unlock_irqrestore(&srq->lock, flags);
1527+
1528+ return rc;
1529+}
12831530 static int bnxt_re_modify_shadow_qp(struct bnxt_re_dev *rdev,
12841531 struct bnxt_re_qp *qp1_qp,
12851532 int qp_attr_mask)
@@ -2295,10 +2542,14 @@ int bnxt_re_post_recv(struct ib_qp *ib_qp, struct ib_recv_wr *wr, · 1009 unmodified lines
22952542 /* Completion Queues */
22962543 int bnxt_re_destroy_cq(struct ib_cq *ib_cq)
22972544 {
2298− struct bnxt_re_cq *cq = container_of(ib_cq, struct bnxt_re_cq, ib_cq);
2299− struct bnxt_re_dev *rdev = cq->rdev;
23002545 int rc;
2301− struct bnxt_qplib_nq *nq = cq->qplib_cq.nq;
2546+ struct bnxt_re_cq *cq;
2547+ struct bnxt_qplib_nq *nq;
2548+ struct bnxt_re_dev *rdev;
2549+
2550+ cq = container_of(ib_cq, struct bnxt_re_cq, ib_cq);
2551+ rdev = cq->rdev;
2552+ nq = cq->qplib_cq.nq;
23022553
23032554 rc = bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq);
23042555 if (rc) {
@@ -2308,12 +2559,11 @@ int bnxt_re_destroy_cq(struct ib_cq *ib_cq) · 3 unmodified lines
23082559 if (!IS_ERR_OR_NULL(cq->umem))
23092560 ib_umem_release(cq->umem);
23102561
2311− if (cq) {
2312− kfree(cq->cql);
2313− kfree(cq);
2314− }
23152562 atomic_dec(&rdev->cq_count);
23162563 nq->budget--;
2564+ kfree(cq->cql);
2565+ kfree(cq);
2566+
23172567 return 0;
23182568 }
23192569
@@ -3078,7 +3328,8 @@ struct ib_mr *bnxt_re_get_dma_mr(struct ib_pd *ib_pd, int mr_access_flags) · 758 unmodified lines
30783328
30793329 mr->qplib_mr.hwq.level = PBL_LVL_MAX;
30803330 mr->qplib_mr.total_size = -1; /* Infinte length */
3081− rc = bnxt_qplib_reg_mr(&rdev->qplib_res, &mr->qplib_mr, &pbl, 0, false);
3331+ rc = bnxt_qplib_reg_mr(&rdev->qplib_res, &mr->qplib_mr, &pbl, 0, false,
3332+ PAGE_SIZE);
30823333 if (rc)
30833334 goto fail_mr;
30843335
@@ -3104,10 +3355,8 @@ int bnxt_re_dereg_mr(struct ib_mr *ib_mr) · 19 unmodified lines
31043355 int rc;
31053356
31063357 rc = bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr);
3107− if (rc) {
3358+ if (rc)
31083359 dev_err(rdev_to_dev(rdev), "Dereg MR failed: %#x\n", rc);
3109− return rc;
3110− }
31113360
31123361 if (mr->pages) {
31133362 rc = bnxt_qplib_free_fast_reg_page_list(&rdev->qplib_res,
@@ -3170,7 +3419,7 @@ struct ib_mr *bnxt_re_alloc_mr(struct ib_pd *ib_pd, enum ib_mr_type type, · 56 unmodified lines
31703419
31713420 rc = bnxt_qplib_alloc_mrw(&rdev->qplib_res, &mr->qplib_mr);
31723421 if (rc)
3173− goto fail;
3422+ goto bail;
31743423
31753424 mr->ib_mr.lkey = mr->qplib_mr.lkey;
31763425 mr->ib_mr.rkey = mr->ib_mr.lkey;
@@ -3192,9 +3441,10 @@ struct ib_mr *bnxt_re_alloc_mr(struct ib_pd *ib_pd, enum ib_mr_type type, · 15 unmodified lines
31923441 return &mr->ib_mr;
31933442
31943443 fail_mr:
3195− bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr);
3196−fail:
31973444 kfree(mr->pages);
3445+fail:
3446+ bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr);
3447+bail:
31983448 kfree(mr);
31993449 return ERR_PTR(rc);
32003450 }
@@ -3248,6 +3498,46 @@ int bnxt_re_dealloc_mw(struct ib_mw *ib_mw) · 47 unmodified lines
32483498 return rc;
32493499 }
32503500
3501+static int bnxt_re_page_size_ok(int page_shift)
3502+{
3503+ switch (page_shift) {
3504+ case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_4K:
3505+ case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_8K:
3506+ case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_64K:
3507+ case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_2M:
3508+ case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_256K:
3509+ case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_1M:
3510+ case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_4M:
3511+ case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_1G:
3512+ return 1;
3513+ default:
3514+ return 0;
3515+ }
3516+}
3517+
3518+static int fill_umem_pbl_tbl(struct ib_umem *umem, u64 *pbl_tbl_orig,
3519+ int page_shift)
3520+{
3521+ u64 *pbl_tbl = pbl_tbl_orig;
3522+ u64 paddr;
3523+ u64 page_mask = (1ULL << page_shift) - 1;
3524+ int i, pages;
3525+ struct scatterlist *sg;
3526+ int entry;
3527+
3528+ for_each_sg(umem->sg_head.sgl, sg, umem->nmap, entry) {
3529+ pages = sg_dma_len(sg) >> PAGE_SHIFT;
3530+ for (i = 0; i < pages; i++) {
3531+ paddr = sg_dma_address(sg) + (i << PAGE_SHIFT);
3532+ if (pbl_tbl == pbl_tbl_orig)
3533+ *pbl_tbl++ = paddr & ~page_mask;
3534+ else if ((paddr & page_mask) == 0)
3535+ *pbl_tbl++ = paddr;
3536+ }
3537+ }
3538+ return pbl_tbl - pbl_tbl_orig;
3539+}
3540+
32513541 /* uverbs */
32523542 struct ib_mr *bnxt_re_reg_user_mr(struct ib_pd *ib_pd, u64 start, u64 length,
32533543 u64 virt_addr, int mr_access_flags,
@@ -3257,10 +3547,8 @@ struct ib_mr *bnxt_re_reg_user_mr(struct ib_pd *ib_pd, u64 start, u64 length, · 3 unmodified lines
32573547 struct bnxt_re_dev *rdev = pd->rdev;
32583548 struct bnxt_re_mr *mr;
32593549 struct ib_umem *umem;
3260− u64 *pbl_tbl, *pbl_tbl_orig;
3261− int i, umem_pgs, pages, rc;
3262− struct scatterlist *sg;
3263− int entry;
3550+ u64 *pbl_tbl = NULL;
3551+ int umem_pgs, page_shift, rc;
32643552
32653553 if (length > BNXT_RE_MAX_MR_SIZE) {
32663554 dev_err(rdev_to_dev(rdev), "MR Size: %lld > Max supported:%ld\n",
@@ -3277,64 +3565,68 @@ struct ib_mr *bnxt_re_reg_user_mr(struct ib_pd *ib_pd, u64 start, u64 length, · 10 unmodified lines
32773565 mr->qplib_mr.flags = __from_ib_access_flags(mr_access_flags);
32783566 mr->qplib_mr.type = CMDQ_ALLOCATE_MRW_MRW_FLAGS_MR;
32793567
3568+ rc = bnxt_qplib_alloc_mrw(&rdev->qplib_res, &mr->qplib_mr);
3569+ if (rc) {
3570+ dev_err(rdev_to_dev(rdev), "Failed to allocate MR");
3571+ goto free_mr;
3572+ }
3573+ /* The fixed portion of the rkey is the same as the lkey */
3574+ mr->ib_mr.rkey = mr->qplib_mr.rkey;
3575+
32803576 umem = ib_umem_get(ib_pd->uobject->context, start, length,
32813577 mr_access_flags, 0);
32823578 if (IS_ERR(umem)) {
32833579 dev_err(rdev_to_dev(rdev), "Failed to get umem");
32843580 rc = -EFAULT;
3285− goto free_mr;
3581+ goto free_mrw;
32863582 }
32873583 mr->ib_umem = umem;
32883584
3289− rc = bnxt_qplib_alloc_mrw(&rdev->qplib_res, &mr->qplib_mr);
3290− if (rc) {
3291− dev_err(rdev_to_dev(rdev), "Failed to allocate MR");
3292− goto release_umem;
3293− }
3294− /* The fixed portion of the rkey is the same as the lkey */
3295− mr->ib_mr.rkey = mr->qplib_mr.rkey;
3296−
32973585 mr->qplib_mr.va = virt_addr;
32983586 umem_pgs = ib_umem_page_count(umem);
32993587 if (!umem_pgs) {
33003588 dev_err(rdev_to_dev(rdev), "umem is invalid!");
33013589 rc = -EINVAL;
3302− goto free_mrw;
3590+ goto free_umem;
33033591 }
33043592 mr->qplib_mr.total_size = length;
33053593
33063594 pbl_tbl = kcalloc(umem_pgs, sizeof(u64 *), GFP_KERNEL);
33073595 if (!pbl_tbl) {
3308− rc = -EINVAL;
3309− goto free_mrw;
3596+ rc = -ENOMEM;
3597+ goto free_umem;
33103598 }
3311− pbl_tbl_orig = pbl_tbl;
33123599
3313− if (umem->hugetlb) {
3314− dev_err(rdev_to_dev(rdev), "umem hugetlb not supported!");
3600+ page_shift = umem->page_shift;
3601+
3602+ if (!bnxt_re_page_size_ok(page_shift)) {
3603+ dev_err(rdev_to_dev(rdev), "umem page size unsupported!");
33153604 rc = -EFAULT;
33163605 goto fail;
33173606 }
33183607
3319− if (umem->page_shift != PAGE_SHIFT) {
3320− dev_err(rdev_to_dev(rdev), "umem page shift unsupported!");
3321− rc = -EFAULT;
3608+ if (!umem->hugetlb && length > BNXT_RE_MAX_MR_SIZE_LOW) {
3609+ dev_err(rdev_to_dev(rdev), "Requested MR Sz:%llu Max sup:%llu",
3610+ length, (u64)BNXT_RE_MAX_MR_SIZE_LOW);
3611+ rc = -EINVAL;
33223612 goto fail;
33233613 }
3324− /* Map umem buf ptrs to the PBL */
3325− for_each_sg(umem->sg_head.sgl, sg, umem->nmap, entry) {
3326− pages = sg_dma_len(sg) >> umem->page_shift;
3327− for (i = 0; i < pages; i++, pbl_tbl++)
3328− *pbl_tbl = sg_dma_address(sg) + (i << umem->page_shift);
3614+ if (umem->hugetlb && length > BNXT_RE_PAGE_SIZE_2M) {
3615+ page_shift = BNXT_RE_PAGE_SHIFT_2M;
3616+ dev_warn(rdev_to_dev(rdev), "umem hugetlb set page_size %x",
3617+ 1 << page_shift);
33293618 }
3330− rc = bnxt_qplib_reg_mr(&rdev->qplib_res, &mr->qplib_mr, pbl_tbl_orig,
3331− umem_pgs, false);
3619+
3620+ /* Map umem buf ptrs to the PBL */
3621+ umem_pgs = fill_umem_pbl_tbl(umem, pbl_tbl, page_shift);
3622+ rc = bnxt_qplib_reg_mr(&rdev->qplib_res, &mr->qplib_mr, pbl_tbl,
3623+ umem_pgs, false, 1 << page_shift);
33323624 if (rc) {
33333625 dev_err(rdev_to_dev(rdev), "Failed to register user MR");
33343626 goto fail;
33353627 }
33363628
3337− kfree(pbl_tbl_orig);
3629+ kfree(pbl_tbl);
33383630
33393631 mr->ib_mr.lkey = mr->qplib_mr.lkey;
33403632 mr->ib_mr.rkey = mr->qplib_mr.lkey;
@@ -3342,11 +3634,11 @@ struct ib_mr *bnxt_re_reg_user_mr(struct ib_pd *ib_pd, u64 start, u64 length, · 1 unmodified line
33423634
33433635 return &mr->ib_mr;
33443636 fail:
3345− kfree(pbl_tbl_orig);
3637+ kfree(pbl_tbl);
3638+free_umem:
3639+ ib_umem_release(umem);
33463640 free_mrw:
33473641 bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr);
3348−release_umem:
3349− ib_umem_release(umem);
33503642 free_mr:
33513643 kfree(mr);
33523644 return ERR_PTR(rc);
drivers/infiniband/hw/bnxt_re/ib_verbs.h+20−0
@@ -68,6 +68,15 @@ struct bnxt_re_ah { · 67 unmodified lines
6868 struct bnxt_qplib_ah qplib_ah;
6969 };
7070
71+struct bnxt_re_srq {
72+ struct bnxt_re_dev *rdev;
73+ u32 srq_limit;
74+ struct ib_srq ib_srq;
75+ struct bnxt_qplib_srq qplib_srq;
76+ struct ib_umem *umem;
77+ spinlock_t lock; /* protect srq */
78+};
79+
7180 struct bnxt_re_qp {
7281 struct list_head list;
7382 struct bnxt_re_dev *rdev;
@@ -143,6 +152,7 @@ int bnxt_re_query_port(struct ib_device *ibdev, u8 port_num, · 69 unmodified lines
143152 struct ib_port_attr *port_attr);
144153 int bnxt_re_get_port_immutable(struct ib_device *ibdev, u8 port_num,
145154 struct ib_port_immutable *immutable);
155+void bnxt_re_query_fw_str(struct ib_device *ibdev, char *str);
146156 int bnxt_re_query_pkey(struct ib_device *ibdev, u8 port_num,
147157 u16 index, u16 *pkey);
148158 int bnxt_re_del_gid(struct ib_device *ibdev, u8 port_num,
@@ -164,6 +174,16 @@ struct ib_ah *bnxt_re_create_ah(struct ib_pd *pd, · 15 unmodified lines
164174 int bnxt_re_modify_ah(struct ib_ah *ah, struct rdma_ah_attr *ah_attr);
165175 int bnxt_re_query_ah(struct ib_ah *ah, struct rdma_ah_attr *ah_attr);
166176 int bnxt_re_destroy_ah(struct ib_ah *ah);
177+struct ib_srq *bnxt_re_create_srq(struct ib_pd *pd,
178+ struct ib_srq_init_attr *srq_init_attr,
179+ struct ib_udata *udata);
180+int bnxt_re_modify_srq(struct ib_srq *srq, struct ib_srq_attr *srq_attr,
181+ enum ib_srq_attr_mask srq_attr_mask,
182+ struct ib_udata *udata);
183+int bnxt_re_query_srq(struct ib_srq *srq, struct ib_srq_attr *srq_attr);
184+int bnxt_re_destroy_srq(struct ib_srq *srq);
185+int bnxt_re_post_srq_recv(struct ib_srq *srq, struct ib_recv_wr *recv_wr,
186+ struct ib_recv_wr **bad_recv_wr);
167187 struct ib_qp *bnxt_re_create_qp(struct ib_pd *pd,
168188 struct ib_qp_init_attr *qp_init_attr,
169189 struct ib_udata *udata);
drivers/infiniband/hw/bnxt_re/main.c+209−42
@@ -80,6 +80,79 @@ static DEFINE_MUTEX(bnxt_re_dev_lock); · 79 unmodified lines
8080 static struct workqueue_struct *bnxt_re_wq;
8181 static void bnxt_re_ib_unreg(struct bnxt_re_dev *rdev, bool lock_wait);
8282
83+/* SR-IOV helper functions */
84+
85+static void bnxt_re_get_sriov_func_type(struct bnxt_re_dev *rdev)
86+{
87+ struct bnxt *bp;
88+
89+ bp = netdev_priv(rdev->en_dev->net);
90+ if (BNXT_VF(bp))
91+ rdev->is_virtfn = 1;
92+}
93+
94+/* Set the maximum number of each resource that the driver actually wants
95+ * to allocate. This may be up to the maximum number the firmware has
96+ * reserved for the function. The driver may choose to allocate fewer
97+ * resources than the firmware maximum.
98+ */
99+static void bnxt_re_set_resource_limits(struct bnxt_re_dev *rdev)
100+{
101+ u32 vf_qps = 0, vf_srqs = 0, vf_cqs = 0, vf_mrws = 0, vf_gids = 0;
102+ u32 i;
103+ u32 vf_pct;
104+ u32 num_vfs;
105+ struct bnxt_qplib_dev_attr *dev_attr = &rdev->dev_attr;
106+
107+ rdev->qplib_ctx.qpc_count = min_t(u32, BNXT_RE_MAX_QPC_COUNT,
108+ dev_attr->max_qp);
109+
110+ rdev->qplib_ctx.mrw_count = BNXT_RE_MAX_MRW_COUNT_256K;
111+ /* Use max_mr from fw since max_mrw does not get set */
112+ rdev->qplib_ctx.mrw_count = min_t(u32, rdev->qplib_ctx.mrw_count,
113+ dev_attr->max_mr);
114+ rdev->qplib_ctx.srqc_count = min_t(u32, BNXT_RE_MAX_SRQC_COUNT,
115+ dev_attr->max_srq);
116+ rdev->qplib_ctx.cq_count = min_t(u32, BNXT_RE_MAX_CQ_COUNT,
117+ dev_attr->max_cq);
118+
119+ for (i = 0; i < MAX_TQM_ALLOC_REQ; i++)
120+ rdev->qplib_ctx.tqm_count[i] =
121+ rdev->dev_attr.tqm_alloc_reqs[i];
122+
123+ if (rdev->num_vfs) {
124+ /*
125+ * Reserve a set of resources for the PF. Divide the remaining
126+ * resources among the VFs
127+ */
128+ vf_pct = 100 - BNXT_RE_PCT_RSVD_FOR_PF;
129+ num_vfs = 100 * rdev->num_vfs;
130+ vf_qps = (rdev->qplib_ctx.qpc_count * vf_pct) / num_vfs;
131+ vf_srqs = (rdev->qplib_ctx.srqc_count * vf_pct) / num_vfs;
132+ vf_cqs = (rdev->qplib_ctx.cq_count * vf_pct) / num_vfs;
133+ /*
134+ * The driver allows many more MRs than other resources. If the
135+ * firmware does also, then reserve a fixed amount for the PF
136+ * and divide the rest among VFs. VFs may use many MRs for NFS
137+ * mounts, ISER, NVME applications, etc. If the firmware
138+ * severely restricts the number of MRs, then let PF have
139+ * half and divide the rest among VFs, as for the other
140+ * resource types.
141+ */
142+ if (rdev->qplib_ctx.mrw_count < BNXT_RE_MAX_MRW_COUNT_64K)
143+ vf_mrws = rdev->qplib_ctx.mrw_count * vf_pct / num_vfs;
144+ else
145+ vf_mrws = (rdev->qplib_ctx.mrw_count -
146+ BNXT_RE_RESVD_MR_FOR_PF) / rdev->num_vfs;
147+ vf_gids = BNXT_RE_MAX_GID_PER_VF;
148+ }
149+ rdev->qplib_ctx.vf_res.max_mrw_per_vf = vf_mrws;
150+ rdev->qplib_ctx.vf_res.max_gid_per_vf = vf_gids;
151+ rdev->qplib_ctx.vf_res.max_qp_per_vf = vf_qps;
152+ rdev->qplib_ctx.vf_res.max_srq_per_vf = vf_srqs;
153+ rdev->qplib_ctx.vf_res.max_cq_per_vf = vf_cqs;
154+}
155+
83156 /* for handling bnxt_en callbacks later */
84157 static void bnxt_re_stop(void *p)
85158 {
@@ -91,6 +164,15 @@ static void bnxt_re_start(void *p) · 5 unmodified lines
91164
92165 static void bnxt_re_sriov_config(void *p, int num_vfs)
93166 {
167+ struct bnxt_re_dev *rdev = p;
168+
169+ if (!rdev)
170+ return;
171+
172+ rdev->num_vfs = num_vfs;
173+ bnxt_re_set_resource_limits(rdev);
174+ bnxt_qplib_set_func_resources(&rdev->qplib_res, &rdev->rcfw,
175+ &rdev->qplib_ctx);
94176 }
95177
96178 static void bnxt_re_shutdown(void *p)
@@ -417,7 +499,7 @@ static struct bnxt_en_dev *bnxt_re_dev_probe(struct net_device *netdev) · 320 unmodified lines
417499 return ERR_PTR(-EINVAL);
418500
419501 if (!(en_dev->flags & BNXT_EN_FLAG_ROCE_CAP)) {
420− dev_dbg(&pdev->dev,
502+ dev_info(&pdev->dev,
421503 "%s: probe error: RoCE is not supported on this device",
422504 ROCE_DRV_MODULE_NAME);
423505 return ERR_PTR(-ENODEV);
@@ -490,6 +572,7 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev) · 66 unmodified lines
490572
491573 ibdev->query_port = bnxt_re_query_port;
492574 ibdev->get_port_immutable = bnxt_re_get_port_immutable;
575+ ibdev->get_dev_fw_str = bnxt_re_query_fw_str;
493576 ibdev->query_pkey = bnxt_re_query_pkey;
494577 ibdev->query_gid = bnxt_re_query_gid;
495578 ibdev->get_netdev = bnxt_re_get_netdev;
@@ -505,6 +588,12 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev) · 9 unmodified lines
505588 ibdev->query_ah = bnxt_re_query_ah;
506589 ibdev->destroy_ah = bnxt_re_destroy_ah;
507590
591+ ibdev->create_srq = bnxt_re_create_srq;
592+ ibdev->modify_srq = bnxt_re_modify_srq;
593+ ibdev->query_srq = bnxt_re_query_srq;
594+ ibdev->destroy_srq = bnxt_re_destroy_srq;
595+ ibdev->post_srq_recv = bnxt_re_post_srq_recv;
596+
508597 ibdev->create_qp = bnxt_re_create_qp;
509598 ibdev->modify_qp = bnxt_re_modify_qp;
510599 ibdev->query_qp = bnxt_re_query_qp;
@@ -541,14 +630,6 @@ static ssize_t show_rev(struct device *device, struct device_attribute *attr, · 30 unmodified lines
541630 return scnprintf(buf, PAGE_SIZE, "0x%x\n", rdev->en_dev->pdev->vendor);
542631 }
543632
544−static ssize_t show_fw_ver(struct device *device, struct device_attribute *attr,
545− char *buf)
546−{
547− struct bnxt_re_dev *rdev = to_bnxt_re_dev(device, ibdev.dev);
548−
549− return scnprintf(buf, PAGE_SIZE, "%s\n", rdev->dev_attr.fw_ver);
550−}
551−
552633 static ssize_t show_hca(struct device *device, struct device_attribute *attr,
553634 char *buf)
554635 {
@@ -558,12 +639,10 @@ static ssize_t show_hca(struct device *device, struct device_attribute *attr, · 3 unmodified lines
558639 }
559640
560641 static DEVICE_ATTR(hw_rev, 0444, show_rev, NULL);
561−static DEVICE_ATTR(fw_rev, 0444, show_fw_ver, NULL);
562642 static DEVICE_ATTR(hca_type, 0444, show_hca, NULL);
563643
564644 static struct device_attribute *bnxt_re_attributes[] = {
565645 &dev_attr_hw_rev,
566− &dev_attr_fw_rev,
567646 &dev_attr_hca_type
568647 };
569648
@@ -616,10 +695,10 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, · 46 unmodified lines
616695 return rdev;
617696 }
618697
619−static int bnxt_re_aeq_handler(struct bnxt_qplib_rcfw *rcfw,
620− struct creq_func_event *aeqe)
698+static int bnxt_re_handle_unaffi_async_event(struct creq_func_event
699+ *unaffi_async)
621700 {
622− switch (aeqe->event) {
701+ switch (unaffi_async->event) {
623702 case CREQ_FUNC_EVENT_EVENT_TX_WQE_ERROR:
624703 break;
625704 case CREQ_FUNC_EVENT_EVENT_TX_DATA_ERROR:
@@ -648,6 +727,93 @@ static int bnxt_re_aeq_handler(struct bnxt_qplib_rcfw *rcfw, · 22 unmodified lines
648727 return 0;
649728 }
650729
730+static int bnxt_re_handle_qp_async_event(struct creq_qp_event *qp_event,
731+ struct bnxt_re_qp *qp)
732+{
733+ struct ib_event event;
734+
735+ memset(&event, 0, sizeof(event));
736+ if (qp->qplib_qp.srq) {
737+ event.device = &qp->rdev->ibdev;
738+ event.element.qp = &qp->ib_qp;
739+ event.event = IB_EVENT_QP_LAST_WQE_REACHED;
740+ }
741+
742+ if (event.device && qp->ib_qp.event_handler)
743+ qp->ib_qp.event_handler(&event, qp->ib_qp.qp_context);
744+
745+ return 0;
746+}
747+
748+static int bnxt_re_handle_affi_async_event(struct creq_qp_event *affi_async,
749+ void *obj)
750+{
751+ int rc = 0;
752+ u8 event;
753+
754+ if (!obj)
755+ return rc; /* QP was already dead, still return success */
756+
757+ event = affi_async->event;
758+ if (event == CREQ_QP_EVENT_EVENT_QP_ERROR_NOTIFICATION) {
759+ struct bnxt_qplib_qp *lib_qp = obj;
760+ struct bnxt_re_qp *qp = container_of(lib_qp, struct bnxt_re_qp,
761+ qplib_qp);
762+ rc = bnxt_re_handle_qp_async_event(affi_async, qp);
763+ }
764+ return rc;
765+}
766+
767+static int bnxt_re_aeq_handler(struct bnxt_qplib_rcfw *rcfw,
768+ void *aeqe, void *obj)
769+{
770+ struct creq_qp_event *affi_async;
771+ struct creq_func_event *unaffi_async;
772+ u8 type;
773+ int rc;
774+
775+ type = ((struct creq_base *)aeqe)->type;
776+ if (type == CREQ_BASE_TYPE_FUNC_EVENT) {
777+ unaffi_async = aeqe;
778+ rc = bnxt_re_handle_unaffi_async_event(unaffi_async);
779+ } else {
780+ affi_async = aeqe;
781+ rc = bnxt_re_handle_affi_async_event(affi_async, obj);
782+ }
783+
784+ return rc;
785+}
786+
787+static int bnxt_re_srqn_handler(struct bnxt_qplib_nq *nq,
788+ struct bnxt_qplib_srq *handle, u8 event)
789+{
790+ struct bnxt_re_srq *srq = container_of(handle, struct bnxt_re_srq,
791+ qplib_srq);
792+ struct ib_event ib_event;
793+ int rc = 0;
794+
795+ if (!srq) {
796+ dev_err(NULL, "%s: SRQ is NULL, SRQN not handled",
797+ ROCE_DRV_MODULE_NAME);
798+ rc = -EINVAL;
799+ goto done;
800+ }
801+ ib_event.device = &srq->rdev->ibdev;
802+ ib_event.element.srq = &srq->ib_srq;
803+ if (event == NQ_SRQ_EVENT_EVENT_SRQ_THRESHOLD_EVENT)
804+ ib_event.event = IB_EVENT_SRQ_LIMIT_REACHED;
805+ else
806+ ib_event.event = IB_EVENT_SRQ_ERR;
807+
808+ if (srq->ib_srq.event_handler) {
809+ /* Lock event_handler? */
810+ (*srq->ib_srq.event_handler)(&ib_event,
811+ srq->ib_srq.srq_context);
812+ }
813+done:
814+ return rc;
815+}
816+
651817 static int bnxt_re_cqn_handler(struct bnxt_qplib_nq *nq,
652818 struct bnxt_qplib_cq *handle)
653819 {
@@ -690,7 +856,8 @@ static int bnxt_re_init_res(struct bnxt_re_dev *rdev) · 36 unmodified lines
690856 rc = bnxt_qplib_enable_nq(rdev->en_dev->pdev, &rdev->nq[i - 1],
691857 i - 1, rdev->msix_entries[i].vector,
692858 rdev->msix_entries[i].db_offset,
693− &bnxt_re_cqn_handler, NULL);
859+ &bnxt_re_cqn_handler,
860+ &bnxt_re_srqn_handler);
694861
695862 if (rc) {
696863 dev_err(rdev_to_dev(rdev),
@@ -734,7 +901,8 @@ static int bnxt_re_alloc_res(struct bnxt_re_dev *rdev) · 37 unmodified lines
734901
735902 /* Configure and allocate resources for qplib */
736903 rdev->qplib_res.rcfw = &rdev->rcfw;
737− rc = bnxt_qplib_get_dev_attr(&rdev->rcfw, &rdev->dev_attr);
904+ rc = bnxt_qplib_get_dev_attr(&rdev->rcfw, &rdev->dev_attr,
905+ rdev->is_virtfn);
738906 if (rc)
739907 goto fail;
740908
@@ -1035,19 +1203,6 @@ static void bnxt_re_ib_unreg(struct bnxt_re_dev *rdev, bool lock_wait) · 294 unmodified lines
10351203 }
10361204 }
10371205
1038−static void bnxt_re_set_resource_limits(struct bnxt_re_dev *rdev)
1039−{
1040− u32 i;
1041−
1042− rdev->qplib_ctx.qpc_count = BNXT_RE_MAX_QPC_COUNT;
1043− rdev->qplib_ctx.mrw_count = BNXT_RE_MAX_MRW_COUNT;
1044− rdev->qplib_ctx.srqc_count = BNXT_RE_MAX_SRQC_COUNT;
1045− rdev->qplib_ctx.cq_count = BNXT_RE_MAX_CQ_COUNT;
1046− for (i = 0; i < MAX_TQM_ALLOC_REQ; i++)
1047− rdev->qplib_ctx.tqm_count[i] =
1048− rdev->dev_attr.tqm_alloc_reqs[i];
1049−}
1050−
10511206 /* worker thread for polling periodic events. Now used for QoS programming*/
10521207 static void bnxt_re_worker(struct work_struct *work)
10531208 {
@@ -1070,6 +1225,9 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) · 16 unmodified lines
10701225 }
10711226 set_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags);
10721227
1228+ /* Check whether VF or PF */
1229+ bnxt_re_get_sriov_func_type(rdev);
1230+
10731231 rc = bnxt_re_request_msix(rdev);
10741232 if (rc) {
10751233 pr_err("Failed to get MSI-X vectors: %#x\n", rc);
@@ -1101,16 +1259,18 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) · 25 unmodified lines
11011259 (rdev->en_dev->pdev, &rdev->rcfw,
11021260 rdev->msix_entries[BNXT_RE_AEQ_IDX].vector,
11031261 rdev->msix_entries[BNXT_RE_AEQ_IDX].db_offset,
1104− 0, &bnxt_re_aeq_handler);
1262+ rdev->is_virtfn, &bnxt_re_aeq_handler);
11051263 if (rc) {
11061264 pr_err("Failed to enable RCFW channel: %#x\n", rc);
11071265 goto free_ring;
11081266 }
11091267
1110− rc = bnxt_qplib_get_dev_attr(&rdev->rcfw, &rdev->dev_attr);
1268+ rc = bnxt_qplib_get_dev_attr(&rdev->rcfw, &rdev->dev_attr,
1269+ rdev->is_virtfn);
11111270 if (rc)
11121271 goto disable_rcfw;
1113− bnxt_re_set_resource_limits(rdev);
1272+ if (!rdev->is_virtfn)
1273+ bnxt_re_set_resource_limits(rdev);
11141274
11151275 rc = bnxt_qplib_alloc_ctx(rdev->en_dev->pdev, &rdev->qplib_ctx, 0);
11161276 if (rc) {
@@ -1125,7 +1285,8 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) · 8 unmodified lines
11251285 goto free_ctx;
11261286 }
11271287
1128− rc = bnxt_qplib_init_rcfw(&rdev->rcfw, &rdev->qplib_ctx, 0);
1288+ rc = bnxt_qplib_init_rcfw(&rdev->rcfw, &rdev->qplib_ctx,
1289+ rdev->is_virtfn);
11291290 if (rc) {
11301291 pr_err("Failed to initialize RCFW: %#x\n", rc);
11311292 goto free_sctx;
@@ -1144,13 +1305,15 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) · 12 unmodified lines
11441305 goto fail;
11451306 }
11461307
1147− rc = bnxt_re_setup_qos(rdev);
1148− if (rc)
1149− pr_info("RoCE priority not yet configured\n");
1308+ if (!rdev->is_virtfn) {
1309+ rc = bnxt_re_setup_qos(rdev);
1310+ if (rc)
1311+ pr_info("RoCE priority not yet configured\n");
11501312
1151− INIT_DELAYED_WORK(&rdev->worker, bnxt_re_worker);
1152− set_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags);
1153− schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000));
1313+ INIT_DELAYED_WORK(&rdev->worker, bnxt_re_worker);
1314+ set_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags);
1315+ schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000));
1316+ }
11541317
11551318 /* Register ib dev */
11561319 rc = bnxt_re_register_ib(rdev);
@@ -1176,6 +1339,7 @@ static int bnxt_re_ib_reg(struct bnxt_re_dev *rdev) · 19 unmodified lines
11761339 set_bit(BNXT_RE_FLAG_IBDEV_REGISTERED, &rdev->flags);
11771340 ib_get_eth_speed(&rdev->ibdev, 1, &rdev->active_speed,
11781341 &rdev->active_width);
1342+ set_bit(BNXT_RE_FLAG_ISSUE_ROCE_STATS, &rdev->flags);
11791343 bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, IB_EVENT_PORT_ACTIVE);
11801344 bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, IB_EVENT_GID_CHANGE);
11811345
@@ -1400,7 +1564,7 @@ err_netdev: · 218 unmodified lines
14001564
14011565 static void __exit bnxt_re_mod_exit(void)
14021566 {
1403− struct bnxt_re_dev *rdev;
1567+ struct bnxt_re_dev *rdev, *next;
14041568 LIST_HEAD(to_be_deleted);
14051569
14061570 mutex_lock(&bnxt_re_dev_lock);
@@ -1408,8 +1572,11 @@ static void __exit bnxt_re_mod_exit(void) · 1 unmodified line
14081572 if (!list_empty(&bnxt_re_dev_list))
14091573 list_splice_init(&bnxt_re_dev_list, &to_be_deleted);
14101574 mutex_unlock(&bnxt_re_dev_lock);
1411−
1412− list_for_each_entry(rdev, &to_be_deleted, list) {
1575+ /*
1576+ * Cleanup the devices in reverse order so that the VF device
1577+ * cleanup is done before PF cleanup
1578+ */
1579+ list_for_each_entry_safe_reverse(rdev, next, &to_be_deleted, list) {
14131580 dev_info(rdev_to_dev(rdev), "Unregistering Device");
14141581 bnxt_re_dev_stop(rdev);
14151582 bnxt_re_ib_unreg(rdev, true);
drivers/infiniband/hw/bnxt_re/qplib_fp.c+400−63
@@ -52,6 +52,7 @@ · 51 unmodified lines
5252
5353 static void bnxt_qplib_arm_cq_enable(struct bnxt_qplib_cq *cq);
5454 static void __clean_cq(struct bnxt_qplib_cq *cq, u64 qp);
55+static void bnxt_qplib_arm_srq(struct bnxt_qplib_srq *srq, u32 arm_type);
5556
5657 static void bnxt_qplib_cancel_phantom_processing(struct bnxt_qplib_qp *qp)
5758 {
@@ -278,6 +279,7 @@ static void bnxt_qplib_service_nq(unsigned long data) · 220 unmodified lines
278279 struct nq_base *nqe, **nq_ptr;
279280 struct bnxt_qplib_cq *cq;
280281 int num_cqne_processed = 0;
282+ int num_srqne_processed = 0;
281283 u32 sw_cons, raw_cons;
282284 u16 type;
283285 int budget = nq->budget;
@@ -320,6 +322,26 @@ static void bnxt_qplib_service_nq(unsigned long data) · 36 unmodified lines
320322 spin_unlock_bh(&cq->compl_lock);
321323 break;
322324 }
325+ case NQ_BASE_TYPE_SRQ_EVENT:
326+ {
327+ struct nq_srq_event *nqsrqe =
328+ (struct nq_srq_event *)nqe;
329+
330+ q_handle = le32_to_cpu(nqsrqe->srq_handle_low);
331+ q_handle |= (u64)le32_to_cpu(nqsrqe->srq_handle_high)
332+ << 32;
333+ bnxt_qplib_arm_srq((struct bnxt_qplib_srq *)q_handle,
334+ DBR_DBR_TYPE_SRQ_ARMENA);
335+ if (!nq->srqn_handler(nq,
336+ (struct bnxt_qplib_srq *)q_handle,
337+ nqsrqe->event))
338+ num_srqne_processed++;
339+ else
340+ dev_warn(&nq->pdev->dev,
341+ "QPLIB: SRQ event 0x%x not handled",
342+ nqsrqe->event);
343+ break;
344+ }
323345 case NQ_BASE_TYPE_DBQ_EVENT:
324346 break;
325347 default:
@@ -384,17 +406,19 @@ int bnxt_qplib_enable_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq, · 58 unmodified lines
384406 int (*cqn_handler)(struct bnxt_qplib_nq *nq,
385407 struct bnxt_qplib_cq *),
386408 int (*srqn_handler)(struct bnxt_qplib_nq *nq,
387− void *, u8 event))
409+ struct bnxt_qplib_srq *,
410+ u8 event))
388411 {
389412 resource_size_t nq_base;
390413 int rc = -1;
391414
392415 nq->pdev = pdev;
393416 nq->vector = msix_vector;
417+ if (cqn_handler)
418+ nq->cqn_handler = cqn_handler;
394419
395− nq->cqn_handler = cqn_handler;
396−
397− nq->srqn_handler = srqn_handler;
420+ if (srqn_handler)
421+ nq->srqn_handler = srqn_handler;
398422
399423 tasklet_init(&nq->worker, bnxt_qplib_service_nq, (unsigned long)nq);
400424
@@ -410,7 +434,6 @@ int bnxt_qplib_enable_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq, · 9 unmodified lines
410434 if (rc) {
411435 dev_err(&nq->pdev->dev,
412436 "Failed to request IRQ for NQ: %#x", rc);
413− bnxt_qplib_disable_nq(nq);
414437 goto fail;
415438 }
416439
@@ -469,6 +492,238 @@ int bnxt_qplib_alloc_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq) · 52 unmodified lines
469492 return 0;
470493 }
471494
495+/* SRQ */
496+static void bnxt_qplib_arm_srq(struct bnxt_qplib_srq *srq, u32 arm_type)
497+{
498+ struct bnxt_qplib_hwq *srq_hwq = &srq->hwq;
499+ struct dbr_dbr db_msg = { 0 };
500+ void __iomem *db;
501+ u32 sw_prod = 0;
502+
503+ /* Ring DB */
504+ sw_prod = (arm_type == DBR_DBR_TYPE_SRQ_ARM) ? srq->threshold :
505+ HWQ_CMP(srq_hwq->prod, srq_hwq);
506+ db_msg.index = cpu_to_le32((sw_prod << DBR_DBR_INDEX_SFT) &
507+ DBR_DBR_INDEX_MASK);
508+ db_msg.type_xid = cpu_to_le32(((srq->id << DBR_DBR_XID_SFT) &
509+ DBR_DBR_XID_MASK) | arm_type);
510+ db = (arm_type == DBR_DBR_TYPE_SRQ_ARMENA) ?
511+ srq->dbr_base : srq->dpi->dbr;
512+ wmb(); /* barrier before db ring */
513+ __iowrite64_copy(db, &db_msg, sizeof(db_msg) / sizeof(u64));
514+}
515+
516+int bnxt_qplib_destroy_srq(struct bnxt_qplib_res *res,
517+ struct bnxt_qplib_srq *srq)
518+{
519+ struct bnxt_qplib_rcfw *rcfw = res->rcfw;
520+ struct cmdq_destroy_srq req;
521+ struct creq_destroy_srq_resp resp;
522+ u16 cmd_flags = 0;
523+ int rc;
524+
525+ RCFW_CMD_PREP(req, DESTROY_SRQ, cmd_flags);
526+
527+ /* Configure the request */
528+ req.srq_cid = cpu_to_le32(srq->id);
529+
530+ rc = bnxt_qplib_rcfw_send_message(rcfw, (void *)&req,
531+ (void *)&resp, NULL, 0);
532+ if (rc)
533+ return rc;
534+
535+ bnxt_qplib_free_hwq(res->pdev, &srq->hwq);
536+ kfree(srq->swq);
537+ return 0;
538+}
539+
540+int bnxt_qplib_create_srq(struct bnxt_qplib_res *res,
541+ struct bnxt_qplib_srq *srq)
542+{
543+ struct bnxt_qplib_rcfw *rcfw = res->rcfw;
544+ struct cmdq_create_srq req;
545+ struct creq_create_srq_resp resp;
546+ struct bnxt_qplib_pbl *pbl;
547+ u16 cmd_flags = 0;
548+ int rc, idx;
549+
550+ srq->hwq.max_elements = srq->max_wqe;
551+ rc = bnxt_qplib_alloc_init_hwq(res->pdev, &srq->hwq, srq->sglist,
552+ srq->nmap, &srq->hwq.max_elements,
553+ BNXT_QPLIB_MAX_RQE_ENTRY_SIZE, 0,
554+ PAGE_SIZE, HWQ_TYPE_QUEUE);
555+ if (rc)
556+ goto exit;
557+
558+ srq->swq = kcalloc(srq->hwq.max_elements, sizeof(*srq->swq),
559+ GFP_KERNEL);
560+ if (!srq->swq)
561+ goto fail;
562+
563+ RCFW_CMD_PREP(req, CREATE_SRQ, cmd_flags);
564+
565+ /* Configure the request */
566+ req.dpi = cpu_to_le32(srq->dpi->dpi);
567+ req.srq_handle = cpu_to_le64(srq);
568+
569+ req.srq_size = cpu_to_le16((u16)srq->hwq.max_elements);
570+ pbl = &srq->hwq.pbl[PBL_LVL_0];
571+ req.pg_size_lvl = cpu_to_le16((((u16)srq->hwq.level &
572+ CMDQ_CREATE_SRQ_LVL_MASK) <<
573+ CMDQ_CREATE_SRQ_LVL_SFT) |
574+ (pbl->pg_size == ROCE_PG_SIZE_4K ?
575+ CMDQ_CREATE_SRQ_PG_SIZE_PG_4K :
576+ pbl->pg_size == ROCE_PG_SIZE_8K ?
577+ CMDQ_CREATE_SRQ_PG_SIZE_PG_8K :
578+ pbl->pg_size == ROCE_PG_SIZE_64K ?
579+ CMDQ_CREATE_SRQ_PG_SIZE_PG_64K :
580+ pbl->pg_size == ROCE_PG_SIZE_2M ?
581+ CMDQ_CREATE_SRQ_PG_SIZE_PG_2M :
582+ pbl->pg_size == ROCE_PG_SIZE_8M ?
583+ CMDQ_CREATE_SRQ_PG_SIZE_PG_8M :
584+ pbl->pg_size == ROCE_PG_SIZE_1G ?
585+ CMDQ_CREATE_SRQ_PG_SIZE_PG_1G :
586+ CMDQ_CREATE_SRQ_PG_SIZE_PG_4K));
587+ req.pbl = cpu_to_le64(pbl->pg_map_arr[0]);
588+ req.pd_id = cpu_to_le32(srq->pd->id);
589+ req.eventq_id = cpu_to_le16(srq->eventq_hw_ring_id);
590+
591+ rc = bnxt_qplib_rcfw_send_message(rcfw, (void *)&req,
592+ (void *)&resp, NULL, 0);
593+ if (rc)
594+ goto fail;
595+
596+ spin_lock_init(&srq->lock);
597+ srq->start_idx = 0;
598+ srq->last_idx = srq->hwq.max_elements - 1;
599+ for (idx = 0; idx < srq->hwq.max_elements; idx++)
600+ srq->swq[idx].next_idx = idx + 1;
601+ srq->swq[srq->last_idx].next_idx = -1;
602+
603+ srq->id = le32_to_cpu(resp.xid);
604+ srq->dbr_base = res->dpi_tbl.dbr_bar_reg_iomem;
605+ if (srq->threshold)
606+ bnxt_qplib_arm_srq(srq, DBR_DBR_TYPE_SRQ_ARMENA);
607+ srq->arm_req = false;
608+
609+ return 0;
610+fail:
611+ bnxt_qplib_free_hwq(res->pdev, &srq->hwq);
612+ kfree(srq->swq);
613+exit:
614+ return rc;
615+}
616+
617+int bnxt_qplib_modify_srq(struct bnxt_qplib_res *res,
618+ struct bnxt_qplib_srq *srq)
619+{
620+ struct bnxt_qplib_hwq *srq_hwq = &srq->hwq;
621+ u32 sw_prod, sw_cons, count = 0;
622+
623+ sw_prod = HWQ_CMP(srq_hwq->prod, srq_hwq);
624+ sw_cons = HWQ_CMP(srq_hwq->cons, srq_hwq);
625+
626+ count = sw_prod > sw_cons ? sw_prod - sw_cons :
627+ srq_hwq->max_elements - sw_cons + sw_prod;
628+ if (count > srq->threshold) {
629+ srq->arm_req = false;
630+ bnxt_qplib_arm_srq(srq, DBR_DBR_TYPE_SRQ_ARM);
631+ } else {
632+ /* Deferred arming */
633+ srq->arm_req = true;
634+ }
635+
636+ return 0;
637+}
638+
639+int bnxt_qplib_query_srq(struct bnxt_qplib_res *res,
640+ struct bnxt_qplib_srq *srq)
641+{
642+ struct bnxt_qplib_rcfw *rcfw = res->rcfw;
643+ struct cmdq_query_srq req;
644+ struct creq_query_srq_resp resp;
645+ struct bnxt_qplib_rcfw_sbuf *sbuf;
646+ struct creq_query_srq_resp_sb *sb;
647+ u16 cmd_flags = 0;
648+ int rc = 0;
649+
650+ RCFW_CMD_PREP(req, QUERY_SRQ, cmd_flags);
651+ req.srq_cid = cpu_to_le32(srq->id);
652+
653+ /* Configure the request */
654+ sbuf = bnxt_qplib_rcfw_alloc_sbuf(rcfw, sizeof(*sb));
655+ if (!sbuf)
656+ return -ENOMEM;
657+ sb = sbuf->sb;
658+ rc = bnxt_qplib_rcfw_send_message(rcfw, (void *)&req, (void *)&resp,
659+ (void *)sbuf, 0);
660+ srq->threshold = le16_to_cpu(sb->srq_limit);
661+ bnxt_qplib_rcfw_free_sbuf(rcfw, sbuf);
662+
663+ return rc;
664+}
665+
666+int bnxt_qplib_post_srq_recv(struct bnxt_qplib_srq *srq,
667+ struct bnxt_qplib_swqe *wqe)
668+{
669+ struct bnxt_qplib_hwq *srq_hwq = &srq->hwq;
670+ struct rq_wqe *srqe, **srqe_ptr;
671+ struct sq_sge *hw_sge;
672+ u32 sw_prod, sw_cons, count = 0;
673+ int i, rc = 0, next;
674+
675+ spin_lock(&srq_hwq->lock);
676+ if (srq->start_idx == srq->last_idx) {
677+ dev_err(&srq_hwq->pdev->dev, "QPLIB: FP: SRQ (0x%x) is full!",
678+ srq->id);
679+ rc = -EINVAL;
680+ spin_unlock(&srq_hwq->lock);
681+ goto done;
682+ }
683+ next = srq->start_idx;
684+ srq->start_idx = srq->swq[next].next_idx;
685+ spin_unlock(&srq_hwq->lock);
686+
687+ sw_prod = HWQ_CMP(srq_hwq->prod, srq_hwq);
688+ srqe_ptr = (struct rq_wqe **)srq_hwq->pbl_ptr;
689+ srqe = &srqe_ptr[RQE_PG(sw_prod)][RQE_IDX(sw_prod)];
690+ memset(srqe, 0, BNXT_QPLIB_MAX_RQE_ENTRY_SIZE);
691+ /* Calculate wqe_size16 and data_len */
692+ for (i = 0, hw_sge = (struct sq_sge *)srqe->data;
693+ i < wqe->num_sge; i++, hw_sge++) {
694+ hw_sge->va_or_pa = cpu_to_le64(wqe->sg_list[i].addr);
695+ hw_sge->l_key = cpu_to_le32(wqe->sg_list[i].lkey);
696+ hw_sge->size = cpu_to_le32(wqe->sg_list[i].size);
697+ }
698+ srqe->wqe_type = wqe->type;
699+ srqe->flags = wqe->flags;
700+ srqe->wqe_size = wqe->num_sge +
701+ ((offsetof(typeof(*srqe), data) + 15) >> 4);
702+ srqe->wr_id[0] = cpu_to_le32((u32)next);
703+ srq->swq[next].wr_id = wqe->wr_id;
704+
705+ srq_hwq->prod++;
706+
707+ spin_lock(&srq_hwq->lock);
708+ sw_prod = HWQ_CMP(srq_hwq->prod, srq_hwq);
709+ /* retaining srq_hwq->cons for this logic
710+ * actually the lock is only required to
711+ * read srq_hwq->cons.
712+ */
713+ sw_cons = HWQ_CMP(srq_hwq->cons, srq_hwq);
714+ count = sw_prod > sw_cons ? sw_prod - sw_cons :
715+ srq_hwq->max_elements - sw_cons + sw_prod;
716+ spin_unlock(&srq_hwq->lock);
717+ /* Ring DB */
718+ bnxt_qplib_arm_srq(srq, DBR_DBR_TYPE_SRQ);
719+ if (srq->arm_req == true && count > srq->threshold) {
720+ srq->arm_req = false;
721+ bnxt_qplib_arm_srq(srq, DBR_DBR_TYPE_SRQ_ARM);
722+ }
723+done:
724+ return rc;
725+}
726+
472727 /* QP */
473728 int bnxt_qplib_create_qp1(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp)
474729 {
@@ -737,6 +992,12 @@ int bnxt_qplib_create_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp) · 262 unmodified lines
737992 pbl->pg_size == ROCE_PG_SIZE_1G ?
738993 CMDQ_CREATE_QP_RQ_PG_SIZE_PG_1G :
739994 CMDQ_CREATE_QP_RQ_PG_SIZE_PG_4K);
995+ } else {
996+ /* SRQ */
997+ if (qp->srq) {
998+ qp_flags |= CMDQ_CREATE_QP_QP_FLAGS_SRQ_USED;
999+ req.srq_cid = cpu_to_le32(qp->srq->id);
1000+ }
7401001 }
7411002
7421003 if (qp->rcq)
@@ -2068,6 +2329,16 @@ done: · 1325 unmodified lines
20682329 return rc;
20692330 }
20702331
2332+static void bnxt_qplib_release_srqe(struct bnxt_qplib_srq *srq, u32 tag)
2333+{
2334+ spin_lock(&srq->hwq.lock);
2335+ srq->swq[srq->last_idx].next_idx = (int)tag;
2336+ srq->last_idx = (int)tag;
2337+ srq->swq[srq->last_idx].next_idx = -1;
2338+ srq->hwq.cons++; /* Support for SRQE counter */
2339+ spin_unlock(&srq->hwq.lock);
2340+}
2341+
20712342 static int bnxt_qplib_cq_process_res_rc(struct bnxt_qplib_cq *cq,
20722343 struct cq_res_rc *hwcqe,
20732344 struct bnxt_qplib_cqe **pcqe,
@@ -2075,6 +2346,7 @@ static int bnxt_qplib_cq_process_res_rc(struct bnxt_qplib_cq *cq, · 1 unmodified line
20752346 {
20762347 struct bnxt_qplib_qp *qp;
20772348 struct bnxt_qplib_q *rq;
2349+ struct bnxt_qplib_srq *srq;
20782350 struct bnxt_qplib_cqe *cqe;
20792351 u32 wr_id_idx;
20802352 int rc = 0;
@@ -2102,27 +2374,46 @@ static int bnxt_qplib_cq_process_res_rc(struct bnxt_qplib_cq *cq, · 21 unmodified lines
21022374
21032375 wr_id_idx = le32_to_cpu(hwcqe->srq_or_rq_wr_id) &
21042376 CQ_RES_RC_SRQ_OR_RQ_WR_ID_MASK;
2105− rq = &qp->rq;
2106− if (wr_id_idx > rq->hwq.max_elements) {
2107− dev_err(&cq->hwq.pdev->dev, "QPLIB: FP: CQ Process RC ");
2108− dev_err(&cq->hwq.pdev->dev,
2109− "QPLIB: wr_id idx 0x%x exceeded RQ max 0x%x",
2110− wr_id_idx, rq->hwq.max_elements);
2111− return -EINVAL;
2112− }
2113−
2114− cqe->wr_id = rq->swq[wr_id_idx].wr_id;
2115− cqe++;
2116− (*budget)--;
2117− rq->hwq.cons++;
2118− *pcqe = cqe;
2377+ if (cqe->flags & CQ_RES_RC_FLAGS_SRQ_SRQ) {
2378+ srq = qp->srq;
2379+ if (!srq)
2380+ return -EINVAL;
2381+ if (wr_id_idx > srq->hwq.max_elements) {
2382+ dev_err(&cq->hwq.pdev->dev,
2383+ "QPLIB: FP: CQ Process RC ");
2384+ dev_err(&cq->hwq.pdev->dev,
2385+ "QPLIB: wr_id idx 0x%x exceeded SRQ max 0x%x",
2386+ wr_id_idx, srq->hwq.max_elements);
2387+ return -EINVAL;
2388+ }
2389+ cqe->wr_id = srq->swq[wr_id_idx].wr_id;
2390+ bnxt_qplib_release_srqe(srq, wr_id_idx);
2391+ cqe++;
2392+ (*budget)--;
2393+ *pcqe = cqe;
2394+ } else {
2395+ rq = &qp->rq;
2396+ if (wr_id_idx > rq->hwq.max_elements) {
2397+ dev_err(&cq->hwq.pdev->dev,
2398+ "QPLIB: FP: CQ Process RC ");
2399+ dev_err(&cq->hwq.pdev->dev,
2400+ "QPLIB: wr_id idx 0x%x exceeded RQ max 0x%x",
2401+ wr_id_idx, rq->hwq.max_elements);
2402+ return -EINVAL;
2403+ }
2404+ cqe->wr_id = rq->swq[wr_id_idx].wr_id;
2405+ cqe++;
2406+ (*budget)--;
2407+ rq->hwq.cons++;
2408+ *pcqe = cqe;
21192409
2120− if (hwcqe->status != CQ_RES_RC_STATUS_OK) {
2121− qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR;
2122− /* Add qp to flush list of the CQ */
2123− bnxt_qplib_lock_buddy_cq(qp, cq);
2124− __bnxt_qplib_add_flush_qp(qp);
2125− bnxt_qplib_unlock_buddy_cq(qp, cq);
2410+ if (hwcqe->status != CQ_RES_RC_STATUS_OK) {
2411+ qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR;
2412+ /* Add qp to flush list of the CQ */
2413+ bnxt_qplib_lock_buddy_cq(qp, cq);
2414+ __bnxt_qplib_add_flush_qp(qp);
2415+ bnxt_qplib_unlock_buddy_cq(qp, cq);
2416+ }
21262417 }
21272418
21282419 done:
@@ -2136,6 +2427,7 @@ static int bnxt_qplib_cq_process_res_ud(struct bnxt_qplib_cq *cq, · 7 unmodified lines
21362427 {
21372428 struct bnxt_qplib_qp *qp;
21382429 struct bnxt_qplib_q *rq;
2430+ struct bnxt_qplib_srq *srq;
21392431 struct bnxt_qplib_cqe *cqe;
21402432 u32 wr_id_idx;
21412433 int rc = 0;
@@ -2166,27 +2458,48 @@ static int bnxt_qplib_cq_process_res_ud(struct bnxt_qplib_cq *cq, · 24 unmodified lines
21662458 hwcqe->src_qp_high_srq_or_rq_wr_id) &
21672459 CQ_RES_UD_SRC_QP_HIGH_MASK) >> 8);
21682460
2169− rq = &qp->rq;
2170− if (wr_id_idx > rq->hwq.max_elements) {
2171− dev_err(&cq->hwq.pdev->dev, "QPLIB: FP: CQ Process UD ");
2172− dev_err(&cq->hwq.pdev->dev,
2173− "QPLIB: wr_id idx %#x exceeded RQ max %#x",
2174− wr_id_idx, rq->hwq.max_elements);
2175− return -EINVAL;
2176− }
2461+ if (cqe->flags & CQ_RES_RC_FLAGS_SRQ_SRQ) {
2462+ srq = qp->srq;
2463+ if (!srq)
2464+ return -EINVAL;
21772465
2178− cqe->wr_id = rq->swq[wr_id_idx].wr_id;
2179− cqe++;
2180− (*budget)--;
2181− rq->hwq.cons++;
2182− *pcqe = cqe;
2466+ if (wr_id_idx > srq->hwq.max_elements) {
2467+ dev_err(&cq->hwq.pdev->dev,
2468+ "QPLIB: FP: CQ Process UD ");
2469+ dev_err(&cq->hwq.pdev->dev,
2470+ "QPLIB: wr_id idx 0x%x exceeded SRQ max 0x%x",
2471+ wr_id_idx, srq->hwq.max_elements);
2472+ return -EINVAL;
2473+ }
2474+ cqe->wr_id = srq->swq[wr_id_idx].wr_id;
2475+ bnxt_qplib_release_srqe(srq, wr_id_idx);
2476+ cqe++;
2477+ (*budget)--;
2478+ *pcqe = cqe;
2479+ } else {
2480+ rq = &qp->rq;
2481+ if (wr_id_idx > rq->hwq.max_elements) {
2482+ dev_err(&cq->hwq.pdev->dev,
2483+ "QPLIB: FP: CQ Process UD ");
2484+ dev_err(&cq->hwq.pdev->dev,
2485+ "QPLIB: wr_id idx 0x%x exceeded RQ max 0x%x",
2486+ wr_id_idx, rq->hwq.max_elements);
2487+ return -EINVAL;
2488+ }
21832489
2184− if (hwcqe->status != CQ_RES_RC_STATUS_OK) {
2185− qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR;
2186− /* Add qp to flush list of the CQ */
2187− bnxt_qplib_lock_buddy_cq(qp, cq);
2188− __bnxt_qplib_add_flush_qp(qp);
2189− bnxt_qplib_unlock_buddy_cq(qp, cq);
2490+ cqe->wr_id = rq->swq[wr_id_idx].wr_id;
2491+ cqe++;
2492+ (*budget)--;
2493+ rq->hwq.cons++;
2494+ *pcqe = cqe;
2495+
2496+ if (hwcqe->status != CQ_RES_RC_STATUS_OK) {
2497+ qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR;
2498+ /* Add qp to flush list of the CQ */
2499+ bnxt_qplib_lock_buddy_cq(qp, cq);
2500+ __bnxt_qplib_add_flush_qp(qp);
2501+ bnxt_qplib_unlock_buddy_cq(qp, cq);
2502+ }
21902503 }
21912504 done:
21922505 return rc;
@@ -2218,6 +2531,7 @@ static int bnxt_qplib_cq_process_res_raweth_qp1(struct bnxt_qplib_cq *cq, · 25 unmodified lines
22182531 {
22192532 struct bnxt_qplib_qp *qp;
22202533 struct bnxt_qplib_q *rq;
2534+ struct bnxt_qplib_srq *srq;
22212535 struct bnxt_qplib_cqe *cqe;
22222536 u32 wr_id_idx;
22232537 int rc = 0;
@@ -2256,26 +2570,49 @@ static int bnxt_qplib_cq_process_res_raweth_qp1(struct bnxt_qplib_cq *cq, · 32 unmodified lines
22562570 cqe->raweth_qp1_flags2 = le32_to_cpu(hwcqe->raweth_qp1_flags2);
22572571 cqe->raweth_qp1_metadata = le32_to_cpu(hwcqe->raweth_qp1_metadata);
22582572
2259− rq = &qp->rq;
2260− if (wr_id_idx > rq->hwq.max_elements) {
2261− dev_err(&cq->hwq.pdev->dev, "QPLIB: FP: CQ Process Raw/QP1 RQ wr_id ");
2262− dev_err(&cq->hwq.pdev->dev, "QPLIB: ix 0x%x exceeded RQ max 0x%x",
2263− wr_id_idx, rq->hwq.max_elements);
2264− return -EINVAL;
2265− }
2266−
2267− cqe->wr_id = rq->swq[wr_id_idx].wr_id;
2268− cqe++;
2269− (*budget)--;
2270− rq->hwq.cons++;
2271− *pcqe = cqe;
2573+ if (cqe->flags & CQ_RES_RAWETH_QP1_FLAGS_SRQ_SRQ) {
2574+ srq = qp->srq;
2575+ if (!srq) {
2576+ dev_err(&cq->hwq.pdev->dev,
2577+ "QPLIB: FP: SRQ used but not defined??");
2578+ return -EINVAL;
2579+ }
2580+ if (wr_id_idx > srq->hwq.max_elements) {
2581+ dev_err(&cq->hwq.pdev->dev,
2582+ "QPLIB: FP: CQ Process Raw/QP1 ");
2583+ dev_err(&cq->hwq.pdev->dev,
2584+ "QPLIB: wr_id idx 0x%x exceeded SRQ max 0x%x",
2585+ wr_id_idx, srq->hwq.max_elements);
2586+ return -EINVAL;
2587+ }
2588+ cqe->wr_id = srq->swq[wr_id_idx].wr_id;
2589+ bnxt_qplib_release_srqe(srq, wr_id_idx);
2590+ cqe++;
2591+ (*budget)--;
2592+ *pcqe = cqe;
2593+ } else {
2594+ rq = &qp->rq;
2595+ if (wr_id_idx > rq->hwq.max_elements) {
2596+ dev_err(&cq->hwq.pdev->dev,
2597+ "QPLIB: FP: CQ Process Raw/QP1 RQ wr_id ");
2598+ dev_err(&cq->hwq.pdev->dev,
2599+ "QPLIB: ix 0x%x exceeded RQ max 0x%x",
2600+ wr_id_idx, rq->hwq.max_elements);
2601+ return -EINVAL;
2602+ }
2603+ cqe->wr_id = rq->swq[wr_id_idx].wr_id;
2604+ cqe++;
2605+ (*budget)--;
2606+ rq->hwq.cons++;
2607+ *pcqe = cqe;
22722608
2273− if (hwcqe->status != CQ_RES_RC_STATUS_OK) {
2274− qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR;
2275− /* Add qp to flush list of the CQ */
2276− bnxt_qplib_lock_buddy_cq(qp, cq);
2277− __bnxt_qplib_add_flush_qp(qp);
2278− bnxt_qplib_unlock_buddy_cq(qp, cq);
2609+ if (hwcqe->status != CQ_RES_RC_STATUS_OK) {
2610+ qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR;
2611+ /* Add qp to flush list of the CQ */
2612+ bnxt_qplib_lock_buddy_cq(qp, cq);
2613+ __bnxt_qplib_add_flush_qp(qp);
2614+ bnxt_qplib_unlock_buddy_cq(qp, cq);
2615+ }
22792616 }
22802617
22812618 done:
drivers/infiniband/hw/bnxt_re/qplib_fp.h+54−24
@@ -39,6 +39,27 @@ · 38 unmodified lines
3939 #ifndef __BNXT_QPLIB_FP_H__
4040 #define __BNXT_QPLIB_FP_H__
4141
42+struct bnxt_qplib_srq {
43+ struct bnxt_qplib_pd *pd;
44+ struct bnxt_qplib_dpi *dpi;
45+ void __iomem *dbr_base;
46+ u64 srq_handle;
47+ u32 id;
48+ u32 max_wqe;
49+ u32 max_sge;
50+ u32 threshold;
51+ bool arm_req;
52+ struct bnxt_qplib_cq *cq;
53+ struct bnxt_qplib_hwq hwq;
54+ struct bnxt_qplib_swq *swq;
55+ struct scatterlist *sglist;
56+ int start_idx;
57+ int last_idx;
58+ u32 nmap;
59+ u16 eventq_hw_ring_id;
60+ spinlock_t lock; /* protect SRQE link list */
61+};
62+
4263 struct bnxt_qplib_sge {
4364 u64 addr;
4465 u32 lkey;
@@ -79,6 +100,7 @@ static inline u32 get_psne_idx(u32 val) · 34 unmodified lines
79100
80101 struct bnxt_qplib_swq {
81102 u64 wr_id;
103+ int next_idx;
82104 u8 type;
83105 u8 flags;
84106 u32 start_psn;
@@ -404,29 +426,27 @@ struct bnxt_qplib_cq { · 319 unmodified lines
404426 writel(NQ_DB_CP_FLAGS | ((raw_cons) & ((cp_bit) - 1)), db)
405427
406428 struct bnxt_qplib_nq {
407− struct pci_dev *pdev;
408−
409− int vector;
410− cpumask_t mask;
411− int budget;
412− bool requested;
413− struct tasklet_struct worker;
414− struct bnxt_qplib_hwq hwq;
415−
416− u16 bar_reg;
417− u16 bar_reg_off;
418− u16 ring_id;
419− void __iomem *bar_reg_iomem;
420−
421− int (*cqn_handler)
422− (struct bnxt_qplib_nq *nq,
423− struct bnxt_qplib_cq *cq);
424− int (*srqn_handler)
425− (struct bnxt_qplib_nq *nq,
426− void *srq,
427− u8 event);
428− struct workqueue_struct *cqn_wq;
429− char name[32];
429+ struct pci_dev *pdev;
430+
431+ int vector;
432+ cpumask_t mask;
433+ int budget;
434+ bool requested;
435+ struct tasklet_struct worker;
436+ struct bnxt_qplib_hwq hwq;
437+
438+ u16 bar_reg;
439+ u16 bar_reg_off;
440+ u16 ring_id;
441+ void __iomem *bar_reg_iomem;
442+
443+ int (*cqn_handler)(struct bnxt_qplib_nq *nq,
444+ struct bnxt_qplib_cq *cq);
445+ int (*srqn_handler)(struct bnxt_qplib_nq *nq,
446+ struct bnxt_qplib_srq *srq,
447+ u8 event);
448+ struct workqueue_struct *cqn_wq;
449+ char name[32];
430450 };
431451
432452 struct bnxt_qplib_nq_work {
@@ -441,8 +461,18 @@ int bnxt_qplib_enable_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq, · 8 unmodified lines
441461 int (*cqn_handler)(struct bnxt_qplib_nq *nq,
442462 struct bnxt_qplib_cq *cq),
443463 int (*srqn_handler)(struct bnxt_qplib_nq *nq,
444− void *srq,
464+ struct bnxt_qplib_srq *srq,
445465 u8 event));
466+int bnxt_qplib_create_srq(struct bnxt_qplib_res *res,
467+ struct bnxt_qplib_srq *srq);
468+int bnxt_qplib_modify_srq(struct bnxt_qplib_res *res,
469+ struct bnxt_qplib_srq *srq);
470+int bnxt_qplib_query_srq(struct bnxt_qplib_res *res,
471+ struct bnxt_qplib_srq *srq);
472+int bnxt_qplib_destroy_srq(struct bnxt_qplib_res *res,
473+ struct bnxt_qplib_srq *srq);
474+int bnxt_qplib_post_srq_recv(struct bnxt_qplib_srq *srq,
475+ struct bnxt_qplib_swqe *wqe);
446476 int bnxt_qplib_create_qp1(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
447477 int bnxt_qplib_create_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
448478 int bnxt_qplib_modify_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
drivers/infiniband/hw/bnxt_re/qplib_rcfw.c+3−2
@@ -93,7 +93,8 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, struct cmdq_base *req, · 92 unmodified lines
9393 opcode = req->opcode;
9494 if (!test_bit(FIRMWARE_INITIALIZED_FLAG, &rcfw->flags) &&
9595 (opcode != CMDQ_BASE_OPCODE_QUERY_FUNC &&
96− opcode != CMDQ_BASE_OPCODE_INITIALIZE_FW)) {
96+ opcode != CMDQ_BASE_OPCODE_INITIALIZE_FW &&
97+ opcode != CMDQ_BASE_OPCODE_QUERY_VERSION)) {
9798 dev_err(&rcfw->pdev->dev,
9899 "QPLIB: RCFW not initialized, reject opcode 0x%x",
99100 opcode);
@@ -615,7 +616,7 @@ int bnxt_qplib_enable_rcfw_channel(struct pci_dev *pdev, · 515 unmodified lines
615616 int msix_vector,
616617 int cp_bar_reg_off, int virt_fn,
617618 int (*aeq_handler)(struct bnxt_qplib_rcfw *,
618− struct creq_func_event *))
619+ void *, void *))
619620 {
620621 resource_size_t res_base;
621622 struct cmdq_init init;
drivers/infiniband/hw/bnxt_re/qplib_rcfw.h+3−4
@@ -167,7 +167,7 @@ struct bnxt_qplib_rcfw { · 166 unmodified lines
167167 #define FIRMWARE_TIMED_OUT 3
168168 wait_queue_head_t waitq;
169169 int (*aeq_handler)(struct bnxt_qplib_rcfw *,
170− struct creq_func_event *);
170+ void *, void *);
171171 u32 seq_num;
172172
173173 /* Bar region info */
@@ -199,9 +199,8 @@ int bnxt_qplib_enable_rcfw_channel(struct pci_dev *pdev, · 25 unmodified lines
199199 struct bnxt_qplib_rcfw *rcfw,
200200 int msix_vector,
201201 int cp_bar_reg_off, int virt_fn,
202− int (*aeq_handler)
203− (struct bnxt_qplib_rcfw *,
204− struct creq_func_event *));
202+ int (*aeq_handler)(struct bnxt_qplib_rcfw *,
203+ void *aeqe, void *obj));
205204
206205 struct bnxt_qplib_rcfw_sbuf *bnxt_qplib_rcfw_alloc_sbuf(
207206 struct bnxt_qplib_rcfw *rcfw,
drivers/infiniband/hw/bnxt_re/qplib_res.c+4−5
@@ -104,13 +104,12 @@ static int __alloc_pbl(struct pci_dev *pdev, struct bnxt_qplib_pbl *pbl, · 103 unmodified lines
104104
105105 if (!sghead) {
106106 for (i = 0; i < pages; i++) {
107− pbl->pg_arr[i] = dma_alloc_coherent(&pdev->dev,
108− pbl->pg_size,
109− &pbl->pg_map_arr[i],
110− GFP_KERNEL);
107+ pbl->pg_arr[i] = dma_zalloc_coherent(&pdev->dev,
108+ pbl->pg_size,
109+ &pbl->pg_map_arr[i],
110+ GFP_KERNEL);
111111 if (!pbl->pg_arr[i])
112112 goto fail;
113− memset(pbl->pg_arr[i], 0, pbl->pg_size);
114113 pbl->pg_count++;
115114 }
116115 } else {
drivers/infiniband/hw/bnxt_re/qplib_sp.c+136−5
@@ -64,8 +64,28 @@ static bool bnxt_qplib_is_atomic_cap(struct bnxt_qplib_rcfw *rcfw) · 63 unmodified lines
6464 return !!(pcie_ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ);
6565 }
6666
67+static void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw,
68+ char *fw_ver)
69+{
70+ struct cmdq_query_version req;
71+ struct creq_query_version_resp resp;
72+ u16 cmd_flags = 0;
73+ int rc = 0;
74+
75+ RCFW_CMD_PREP(req, QUERY_VERSION, cmd_flags);
76+
77+ rc = bnxt_qplib_rcfw_send_message(rcfw, (void *)&req,
78+ (void *)&resp, NULL, 0);
79+ if (rc)
80+ return;
81+ fw_ver[0] = resp.fw_maj;
82+ fw_ver[1] = resp.fw_minor;
83+ fw_ver[2] = resp.fw_bld;
84+ fw_ver[3] = resp.fw_rsvd;
85+}
86+
6787 int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw,
68− struct bnxt_qplib_dev_attr *attr)
88+ struct bnxt_qplib_dev_attr *attr, bool vf)
6989 {
7090 struct cmdq_query_func req;
7191 struct creq_query_func_resp resp;
@@ -95,7 +115,8 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw, · 23 unmodified lines
95115 /* Extract the context from the side buffer */
96116 attr->max_qp = le32_to_cpu(sb->max_qp);
97117 /* max_qp value reported by FW for PF doesn't include the QP1 for PF */
98− attr->max_qp += 1;
118+ if (!vf)
119+ attr->max_qp += 1;
99120 attr->max_qp_rd_atom =
100121 sb->max_qp_rd_atom > BNXT_QPLIB_MAX_OUT_RD_ATOM ?
101122 BNXT_QPLIB_MAX_OUT_RD_ATOM : sb->max_qp_rd_atom;
@@ -133,7 +154,7 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw, · 31 unmodified lines
133154 attr->l2_db_size = (sb->l2_db_space_size + 1) * PAGE_SIZE;
134155 attr->max_sgid = le32_to_cpu(sb->max_gid);
135156
136− strlcpy(attr->fw_ver, "20.6.28.0", sizeof(attr->fw_ver));
157+ bnxt_qplib_query_version(rcfw, attr->fw_ver);
137158
138159 for (i = 0; i < MAX_TQM_ALLOC_REQ / 4; i++) {
139160 temp = le32_to_cpu(sb->tqm_alloc_reqs[i]);
@@ -150,6 +171,38 @@ bail: · 10 unmodified lines
150171 return rc;
151172 }
152173
174+int bnxt_qplib_set_func_resources(struct bnxt_qplib_res *res,
175+ struct bnxt_qplib_rcfw *rcfw,
176+ struct bnxt_qplib_ctx *ctx)
177+{
178+ struct cmdq_set_func_resources req;
179+ struct creq_set_func_resources_resp resp;
180+ u16 cmd_flags = 0;
181+ int rc = 0;
182+
183+ RCFW_CMD_PREP(req, SET_FUNC_RESOURCES, cmd_flags);
184+
185+ req.number_of_qp = cpu_to_le32(ctx->qpc_count);
186+ req.number_of_mrw = cpu_to_le32(ctx->mrw_count);
187+ req.number_of_srq = cpu_to_le32(ctx->srqc_count);
188+ req.number_of_cq = cpu_to_le32(ctx->cq_count);
189+
190+ req.max_qp_per_vf = cpu_to_le32(ctx->vf_res.max_qp_per_vf);
191+ req.max_mrw_per_vf = cpu_to_le32(ctx->vf_res.max_mrw_per_vf);
192+ req.max_srq_per_vf = cpu_to_le32(ctx->vf_res.max_srq_per_vf);
193+ req.max_cq_per_vf = cpu_to_le32(ctx->vf_res.max_cq_per_vf);
194+ req.max_gid_per_vf = cpu_to_le32(ctx->vf_res.max_gid_per_vf);
195+
196+ rc = bnxt_qplib_rcfw_send_message(rcfw, (void *)&req,
197+ (void *)&resp,
198+ NULL, 0);
199+ if (rc) {
200+ dev_err(&res->pdev->dev,
201+ "QPLIB: Failed to set function resources");
202+ }
203+ return rc;
204+}
205+
153206 /* SGID */
154207 int bnxt_qplib_get_sgid(struct bnxt_qplib_res *res,
155208 struct bnxt_qplib_sgid_tbl *sgid_tbl, int index,
@@ -604,7 +657,7 @@ int bnxt_qplib_dereg_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw, · 448 unmodified lines
604657 }
605658
606659 int bnxt_qplib_reg_mr(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr,
607− u64 *pbl_tbl, int num_pbls, bool block)
660+ u64 *pbl_tbl, int num_pbls, bool block, u32 buf_pg_size)
608661 {
609662 struct bnxt_qplib_rcfw *rcfw = res->rcfw;
610663 struct cmdq_register_mr req;
@@ -615,6 +668,9 @@ int bnxt_qplib_reg_mr(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr, · 4 unmodified lines
615668 u32 pg_size;
616669
617670 if (num_pbls) {
671+ /* Allocate memory for the non-leaf pages to store buf ptrs.
672+ * Non-leaf pages always uses system PAGE_SIZE
673+ */
618674 pg_ptrs = roundup_pow_of_two(num_pbls);
619675 pages = pg_ptrs >> MAX_PBL_LVL_1_PGS_SHIFT;
620676 if (!pages)
@@ -632,6 +688,7 @@ int bnxt_qplib_reg_mr(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr, · 11 unmodified lines
632688 bnxt_qplib_free_hwq(res->pdev, &mr->hwq);
633689
634690 mr->hwq.max_elements = pages;
691+ /* Use system PAGE_SIZE */
635692 rc = bnxt_qplib_alloc_init_hwq(res->pdev, &mr->hwq, NULL, 0,
636693 &mr->hwq.max_elements,
637694 PAGE_SIZE, 0, PAGE_SIZE,
@@ -652,18 +709,22 @@ int bnxt_qplib_reg_mr(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr, · 14 unmodified lines
652709
653710 /* Configure the request */
654711 if (mr->hwq.level == PBL_LVL_MAX) {
712+ /* No PBL provided, just use system PAGE_SIZE */
655713 level = 0;
656714 req.pbl = 0;
657715 pg_size = PAGE_SIZE;
658716 } else {
659717 level = mr->hwq.level + 1;
660718 req.pbl = cpu_to_le64(mr->hwq.pbl[PBL_LVL_0].pg_map_arr[0]);
661− pg_size = mr->hwq.pbl[PBL_LVL_0].pg_size;
662719 }
720+ pg_size = buf_pg_size ? buf_pg_size : PAGE_SIZE;
663721 req.log2_pg_size_lvl = (level << CMDQ_REGISTER_MR_LVL_SFT) |
664722 ((ilog2(pg_size) <<
665723 CMDQ_REGISTER_MR_LOG2_PG_SIZE_SFT) &
666724 CMDQ_REGISTER_MR_LOG2_PG_SIZE_MASK);
725+ req.log2_pbl_pg_size = cpu_to_le16(((ilog2(PAGE_SIZE) <<
726+ CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_SFT) &
727+ CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_MASK));
667728 req.access = (mr->flags & 0xFFFF);
668729 req.va = cpu_to_le64(mr->va);
669730 req.key = cpu_to_le32(mr->lkey);
@@ -729,3 +790,73 @@ int bnxt_qplib_map_tc2cos(struct bnxt_qplib_res *res, u16 *cids) · 59 unmodified lines
729790 0);
730791 return 0;
731792 }
793+
794+int bnxt_qplib_get_roce_stats(struct bnxt_qplib_rcfw *rcfw,
795+ struct bnxt_qplib_roce_stats *stats)
796+{
797+ struct cmdq_query_roce_stats req;
798+ struct creq_query_roce_stats_resp resp;
799+ struct bnxt_qplib_rcfw_sbuf *sbuf;
800+ struct creq_query_roce_stats_resp_sb *sb;
801+ u16 cmd_flags = 0;
802+ int rc = 0;
803+
804+ RCFW_CMD_PREP(req, QUERY_ROCE_STATS, cmd_flags);
805+
806+ sbuf = bnxt_qplib_rcfw_alloc_sbuf(rcfw, sizeof(*sb));
807+ if (!sbuf) {
808+ dev_err(&rcfw->pdev->dev,
809+ "QPLIB: SP: QUERY_ROCE_STATS alloc side buffer failed");
810+ return -ENOMEM;
811+ }
812+
813+ sb = sbuf->sb;
814+ req.resp_size = sizeof(*sb) / BNXT_QPLIB_CMDQE_UNITS;
815+ rc = bnxt_qplib_rcfw_send_message(rcfw, (void *)&req, (void *)&resp,
816+ (void *)sbuf, 0);
817+ if (rc)
818+ goto bail;
819+ /* Extract the context from the side buffer */
820+ stats->to_retransmits = le64_to_cpu(sb->to_retransmits);
821+ stats->seq_err_naks_rcvd = le64_to_cpu(sb->seq_err_naks_rcvd);
822+ stats->max_retry_exceeded = le64_to_cpu(sb->max_retry_exceeded);
823+ stats->rnr_naks_rcvd = le64_to_cpu(sb->rnr_naks_rcvd);
824+ stats->missing_resp = le64_to_cpu(sb->missing_resp);
825+ stats->unrecoverable_err = le64_to_cpu(sb->unrecoverable_err);
826+ stats->bad_resp_err = le64_to_cpu(sb->bad_resp_err);
827+ stats->local_qp_op_err = le64_to_cpu(sb->local_qp_op_err);
828+ stats->local_protection_err = le64_to_cpu(sb->local_protection_err);
829+ stats->mem_mgmt_op_err = le64_to_cpu(sb->mem_mgmt_op_err);
830+ stats->remote_invalid_req_err = le64_to_cpu(sb->remote_invalid_req_err);
831+ stats->remote_access_err = le64_to_cpu(sb->remote_access_err);
832+ stats->remote_op_err = le64_to_cpu(sb->remote_op_err);
833+ stats->dup_req = le64_to_cpu(sb->dup_req);
834+ stats->res_exceed_max = le64_to_cpu(sb->res_exceed_max);
835+ stats->res_length_mismatch = le64_to_cpu(sb->res_length_mismatch);
836+ stats->res_exceeds_wqe = le64_to_cpu(sb->res_exceeds_wqe);
837+ stats->res_opcode_err = le64_to_cpu(sb->res_opcode_err);
838+ stats->res_rx_invalid_rkey = le64_to_cpu(sb->res_rx_invalid_rkey);
839+ stats->res_rx_domain_err = le64_to_cpu(sb->res_rx_domain_err);
840+ stats->res_rx_no_perm = le64_to_cpu(sb->res_rx_no_perm);
841+ stats->res_rx_range_err = le64_to_cpu(sb->res_rx_range_err);
842+ stats->res_tx_invalid_rkey = le64_to_cpu(sb->res_tx_invalid_rkey);
843+ stats->res_tx_domain_err = le64_to_cpu(sb->res_tx_domain_err);
844+ stats->res_tx_no_perm = le64_to_cpu(sb->res_tx_no_perm);
845+ stats->res_tx_range_err = le64_to_cpu(sb->res_tx_range_err);
846+ stats->res_irrq_oflow = le64_to_cpu(sb->res_irrq_oflow);
847+ stats->res_unsup_opcode = le64_to_cpu(sb->res_unsup_opcode);
848+ stats->res_unaligned_atomic = le64_to_cpu(sb->res_unaligned_atomic);
849+ stats->res_rem_inv_err = le64_to_cpu(sb->res_rem_inv_err);
850+ stats->res_mem_error = le64_to_cpu(sb->res_mem_error);
851+ stats->res_srq_err = le64_to_cpu(sb->res_srq_err);
852+ stats->res_cmp_err = le64_to_cpu(sb->res_cmp_err);
853+ stats->res_invalid_dup_rkey = le64_to_cpu(sb->res_invalid_dup_rkey);
854+ stats->res_wqe_format_err = le64_to_cpu(sb->res_wqe_format_err);
855+ stats->res_cq_load_err = le64_to_cpu(sb->res_cq_load_err);
856+ stats->res_srq_load_err = le64_to_cpu(sb->res_srq_load_err);
857+ stats->res_tx_pci_err = le64_to_cpu(sb->res_tx_pci_err);
858+ stats->res_rx_pci_err = le64_to_cpu(sb->res_rx_pci_err);
859+bail:
860+ bnxt_qplib_rcfw_free_sbuf(rcfw, sbuf);
861+ return rc;
862+}
drivers/infiniband/hw/bnxt_re/qplib_sp.h+88−3
@@ -45,7 +45,8 @@ · 44 unmodified lines
4545 #define PCI_EXP_DEVCTL2_ATOMIC_REQ 0x0040
4646
4747 struct bnxt_qplib_dev_attr {
48− char fw_ver[32];
48+#define FW_VER_ARR_LEN 4
49+ u8 fw_ver[FW_VER_ARR_LEN];
4950 u16 max_sgid;
5051 u16 max_mrw;
5152 u32 max_qp;
@@ -127,6 +128,85 @@ struct bnxt_qplib_frpl { · 75 unmodified lines
127128 #define BNXT_QPLIB_ACCESS_ZERO_BASED BIT(5)
128129 #define BNXT_QPLIB_ACCESS_ON_DEMAND BIT(6)
129130
131+struct bnxt_qplib_roce_stats {
132+ u64 to_retransmits;
133+ u64 seq_err_naks_rcvd;
134+ /* seq_err_naks_rcvd is 64 b */
135+ u64 max_retry_exceeded;
136+ /* max_retry_exceeded is 64 b */
137+ u64 rnr_naks_rcvd;
138+ /* rnr_naks_rcvd is 64 b */
139+ u64 missing_resp;
140+ u64 unrecoverable_err;
141+ /* unrecoverable_err is 64 b */
142+ u64 bad_resp_err;
143+ /* bad_resp_err is 64 b */
144+ u64 local_qp_op_err;
145+ /* local_qp_op_err is 64 b */
146+ u64 local_protection_err;
147+ /* local_protection_err is 64 b */
148+ u64 mem_mgmt_op_err;
149+ /* mem_mgmt_op_err is 64 b */
150+ u64 remote_invalid_req_err;
151+ /* remote_invalid_req_err is 64 b */
152+ u64 remote_access_err;
153+ /* remote_access_err is 64 b */
154+ u64 remote_op_err;
155+ /* remote_op_err is 64 b */
156+ u64 dup_req;
157+ /* dup_req is 64 b */
158+ u64 res_exceed_max;
159+ /* res_exceed_max is 64 b */
160+ u64 res_length_mismatch;
161+ /* res_length_mismatch is 64 b */
162+ u64 res_exceeds_wqe;
163+ /* res_exceeds_wqe is 64 b */
164+ u64 res_opcode_err;
165+ /* res_opcode_err is 64 b */
166+ u64 res_rx_invalid_rkey;
167+ /* res_rx_invalid_rkey is 64 b */
168+ u64 res_rx_domain_err;
169+ /* res_rx_domain_err is 64 b */
170+ u64 res_rx_no_perm;
171+ /* res_rx_no_perm is 64 b */
172+ u64 res_rx_range_err;
173+ /* res_rx_range_err is 64 b */
174+ u64 res_tx_invalid_rkey;
175+ /* res_tx_invalid_rkey is 64 b */
176+ u64 res_tx_domain_err;
177+ /* res_tx_domain_err is 64 b */
178+ u64 res_tx_no_perm;
179+ /* res_tx_no_perm is 64 b */
180+ u64 res_tx_range_err;
181+ /* res_tx_range_err is 64 b */
182+ u64 res_irrq_oflow;
183+ /* res_irrq_oflow is 64 b */
184+ u64 res_unsup_opcode;
185+ /* res_unsup_opcode is 64 b */
186+ u64 res_unaligned_atomic;
187+ /* res_unaligned_atomic is 64 b */
188+ u64 res_rem_inv_err;
189+ /* res_rem_inv_err is 64 b */
190+ u64 res_mem_error;
191+ /* res_mem_error is 64 b */
192+ u64 res_srq_err;
193+ /* res_srq_err is 64 b */
194+ u64 res_cmp_err;
195+ /* res_cmp_err is 64 b */
196+ u64 res_invalid_dup_rkey;
197+ /* res_invalid_dup_rkey is 64 b */
198+ u64 res_wqe_format_err;
199+ /* res_wqe_format_err is 64 b */
200+ u64 res_cq_load_err;
201+ /* res_cq_load_err is 64 b */
202+ u64 res_srq_load_err;
203+ /* res_srq_load_err is 64 b */
204+ u64 res_tx_pci_err;
205+ /* res_tx_pci_err is 64 b */
206+ u64 res_rx_pci_err;
207+ /* res_rx_pci_err is 64 b */
208+};
209+
130210 int bnxt_qplib_get_sgid(struct bnxt_qplib_res *res,
131211 struct bnxt_qplib_sgid_tbl *sgid_tbl, int index,
132212 struct bnxt_qplib_gid *gid);
@@ -147,7 +227,10 @@ int bnxt_qplib_add_pkey(struct bnxt_qplib_res *res, · 14 unmodified lines
147227 struct bnxt_qplib_pkey_tbl *pkey_tbl, u16 *pkey,
148228 bool update);
149229 int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw,
150− struct bnxt_qplib_dev_attr *attr);
230+ struct bnxt_qplib_dev_attr *attr, bool vf);
231+int bnxt_qplib_set_func_resources(struct bnxt_qplib_res *res,
232+ struct bnxt_qplib_rcfw *rcfw,
233+ struct bnxt_qplib_ctx *ctx);
151234 int bnxt_qplib_create_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah);
152235 int bnxt_qplib_destroy_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah);
153236 int bnxt_qplib_alloc_mrw(struct bnxt_qplib_res *res,
@@ -155,7 +238,7 @@ int bnxt_qplib_alloc_mrw(struct bnxt_qplib_res *res, · 1 unmodified line
155238 int bnxt_qplib_dereg_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw,
156239 bool block);
157240 int bnxt_qplib_reg_mr(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr,
158− u64 *pbl_tbl, int num_pbls, bool block);
241+ u64 *pbl_tbl, int num_pbls, bool block, u32 buf_pg_size);
159242 int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr);
160243 int bnxt_qplib_alloc_fast_reg_mr(struct bnxt_qplib_res *res,
161244 struct bnxt_qplib_mrw *mr, int max);
@@ -164,4 +247,6 @@ int bnxt_qplib_alloc_fast_reg_page_list(struct bnxt_qplib_res *res, · 2 unmodified lines
164247 int bnxt_qplib_free_fast_reg_page_list(struct bnxt_qplib_res *res,
165248 struct bnxt_qplib_frpl *frpl);
166249 int bnxt_qplib_map_tc2cos(struct bnxt_qplib_res *res, u16 *cids);
250+int bnxt_qplib_get_roce_stats(struct bnxt_qplib_rcfw *rcfw,
251+ struct bnxt_qplib_roce_stats *stats);
167252 #endif /* __BNXT_QPLIB_SP_H__*/
drivers/infiniband/hw/bnxt_re/roce_hsi.h+126−1
@@ -954,6 +954,7 @@ struct cmdq_base { · 953 unmodified lines
954954 #define CMDQ_BASE_OPCODE_QUERY_VERSION 0x8bUL
955955 #define CMDQ_BASE_OPCODE_MODIFY_CC 0x8cUL
956956 #define CMDQ_BASE_OPCODE_QUERY_CC 0x8dUL
957+ #define CMDQ_BASE_OPCODE_QUERY_ROCE_STATS 0x8eUL
957958 u8 cmd_size;
958959 __le16 flags;
959960 __le16 cookie;
@@ -1383,8 +1384,20 @@ struct cmdq_register_mr { · 423 unmodified lines
13831384 #define CMDQ_REGISTER_MR_LVL_LVL_0 0x0UL
13841385 #define CMDQ_REGISTER_MR_LVL_LVL_1 0x1UL
13851386 #define CMDQ_REGISTER_MR_LVL_LVL_2 0x2UL
1387+ #define CMDQ_REGISTER_MR_LVL_LAST CMDQ_REGISTER_MR_LVL_LVL_2
13861388 #define CMDQ_REGISTER_MR_LOG2_PG_SIZE_MASK 0x7cUL
13871389 #define CMDQ_REGISTER_MR_LOG2_PG_SIZE_SFT 2
1390+ #define CMDQ_REGISTER_MR_LOG2_PG_SIZE_PG_4K (0xcUL << 2)
1391+ #define CMDQ_REGISTER_MR_LOG2_PG_SIZE_PG_8K (0xdUL << 2)
1392+ #define CMDQ_REGISTER_MR_LOG2_PG_SIZE_PG_64K (0x10UL << 2)
1393+ #define CMDQ_REGISTER_MR_LOG2_PG_SIZE_PG_256K (0x12UL << 2)
1394+ #define CMDQ_REGISTER_MR_LOG2_PG_SIZE_PG_1M (0x14UL << 2)
1395+ #define CMDQ_REGISTER_MR_LOG2_PG_SIZE_PG_2M (0x15UL << 2)
1396+ #define CMDQ_REGISTER_MR_LOG2_PG_SIZE_PG_4M (0x16UL << 2)
1397+ #define CMDQ_REGISTER_MR_LOG2_PG_SIZE_PG_1G (0x1eUL << 2)
1398+ #define CMDQ_REGISTER_MR_LOG2_PG_SIZE_LAST \
1399+ CMDQ_REGISTER_MR_LOG2_PG_SIZE_PG_1G
1400+ #define CMDQ_REGISTER_MR_UNUSED1 0x80UL
13881401 u8 access;
13891402 #define CMDQ_REGISTER_MR_ACCESS_LOCAL_WRITE 0x1UL
13901403 #define CMDQ_REGISTER_MR_ACCESS_REMOTE_READ 0x2UL
@@ -1392,7 +1405,21 @@ struct cmdq_register_mr { · 1 unmodified line
13921405 #define CMDQ_REGISTER_MR_ACCESS_REMOTE_ATOMIC 0x8UL
13931406 #define CMDQ_REGISTER_MR_ACCESS_MW_BIND 0x10UL
13941407 #define CMDQ_REGISTER_MR_ACCESS_ZERO_BASED 0x20UL
1395− __le16 unused_1;
1408+ __le16 log2_pbl_pg_size;
1409+ #define CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_MASK 0x1fUL
1410+ #define CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_SFT 0
1411+ #define CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_4K 0xcUL
1412+ #define CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_8K 0xdUL
1413+ #define CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_64K 0x10UL
1414+ #define CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_256K 0x12UL
1415+ #define CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_1M 0x14UL
1416+ #define CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_2M 0x15UL
1417+ #define CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_4M 0x16UL
1418+ #define CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_1G 0x1eUL
1419+ #define CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_LAST \
1420+ CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_1G
1421+ #define CMDQ_REGISTER_MR_UNUSED11_MASK 0xffe0UL
1422+ #define CMDQ_REGISTER_MR_UNUSED11_SFT 5
13961423 __le32 key;
13971424 __le64 pbl;
13981425 __le64 va;
@@ -1799,6 +1826,16 @@ struct cmdq_set_func_resources { · 400 unmodified lines
17991826 u8 resp_size;
18001827 u8 reserved8;
18011828 __le64 resp_addr;
1829+ __le32 number_of_qp;
1830+ __le32 number_of_mrw;
1831+ __le32 number_of_srq;
1832+ __le32 number_of_cq;
1833+ __le32 max_qp_per_vf;
1834+ __le32 max_mrw_per_vf;
1835+ __le32 max_srq_per_vf;
1836+ __le32 max_cq_per_vf;
1837+ __le32 max_gid_per_vf;
1838+ __le32 stat_ctx_id;
18021839 };
18031840
18041841 /* Read hardware resource context command (24 bytes) */
@@ -2013,6 +2050,20 @@ struct creq_modify_qp_resp { · 208 unmodified lines
20132050 __le16 reserved48[3];
20142051 };
20152052
2053+/* cmdq_query_roce_stats (size:128b/16B) */
2054+struct cmdq_query_roce_stats {
2055+ u8 opcode;
2056+ #define CMDQ_QUERY_ROCE_STATS_OPCODE_QUERY_ROCE_STATS 0x8eUL
2057+ #define CMDQ_QUERY_ROCE_STATS_OPCODE_LAST \
2058+ CMDQ_QUERY_ROCE_STATS_OPCODE_QUERY_ROCE_STATS
2059+ u8 cmd_size;
2060+ __le16 flags;
2061+ __le16 cookie;
2062+ u8 resp_size;
2063+ u8 reserved8;
2064+ __le64 resp_addr;
2065+};
2066+
20162067 /* Query QP command response (16 bytes) */
20172068 struct creq_query_qp_resp {
20182069 u8 type;
@@ -2783,6 +2834,80 @@ struct creq_query_cc_resp_sb { · 764 unmodified lines
27832834 __le64 reserved64_1;
27842835 };
27852836
2837+/* creq_query_roce_stats_resp (size:128b/16B) */
2838+struct creq_query_roce_stats_resp {
2839+ u8 type;
2840+ #define CREQ_QUERY_ROCE_STATS_RESP_TYPE_MASK 0x3fUL
2841+ #define CREQ_QUERY_ROCE_STATS_RESP_TYPE_SFT 0
2842+ #define CREQ_QUERY_ROCE_STATS_RESP_TYPE_QP_EVENT 0x38UL
2843+ #define CREQ_QUERY_ROCE_STATS_RESP_TYPE_LAST \
2844+ CREQ_QUERY_ROCE_STATS_RESP_TYPE_QP_EVENT
2845+ u8 status;
2846+ __le16 cookie;
2847+ __le32 size;
2848+ u8 v;
2849+ #define CREQ_QUERY_ROCE_STATS_RESP_V 0x1UL
2850+ u8 event;
2851+ #define CREQ_QUERY_ROCE_STATS_RESP_EVENT_QUERY_ROCE_STATS 0x8eUL
2852+ #define CREQ_QUERY_ROCE_STATS_RESP_EVENT_LAST \
2853+ CREQ_QUERY_ROCE_STATS_RESP_EVENT_QUERY_ROCE_STATS
2854+ u8 reserved48[6];
2855+};
2856+
2857+/* creq_query_roce_stats_resp_sb (size:2624b/328B) */
2858+struct creq_query_roce_stats_resp_sb {
2859+ u8 opcode;
2860+ #define CREQ_QUERY_ROCE_STATS_RESP_SB_OPCODE_QUERY_ROCE_STATS 0x8eUL
2861+ #define CREQ_QUERY_ROCE_STATS_RESP_SB_OPCODE_LAST \
2862+ CREQ_QUERY_ROCE_STATS_RESP_SB_OPCODE_QUERY_ROCE_STATS
2863+ u8 status;
2864+ __le16 cookie;
2865+ __le16 flags;
2866+ u8 resp_size;
2867+ u8 rsvd;
2868+ __le32 num_counters;
2869+ __le32 rsvd1;
2870+ __le64 to_retransmits;
2871+ __le64 seq_err_naks_rcvd;
2872+ __le64 max_retry_exceeded;
2873+ __le64 rnr_naks_rcvd;
2874+ __le64 missing_resp;
2875+ __le64 unrecoverable_err;
2876+ __le64 bad_resp_err;
2877+ __le64 local_qp_op_err;
2878+ __le64 local_protection_err;
2879+ __le64 mem_mgmt_op_err;
2880+ __le64 remote_invalid_req_err;
2881+ __le64 remote_access_err;
2882+ __le64 remote_op_err;
2883+ __le64 dup_req;
2884+ __le64 res_exceed_max;
2885+ __le64 res_length_mismatch;
2886+ __le64 res_exceeds_wqe;
2887+ __le64 res_opcode_err;
2888+ __le64 res_rx_invalid_rkey;
2889+ __le64 res_rx_domain_err;
2890+ __le64 res_rx_no_perm;
2891+ __le64 res_rx_range_err;
2892+ __le64 res_tx_invalid_rkey;
2893+ __le64 res_tx_domain_err;
2894+ __le64 res_tx_no_perm;
2895+ __le64 res_tx_range_err;
2896+ __le64 res_irrq_oflow;
2897+ __le64 res_unsup_opcode;
2898+ __le64 res_unaligned_atomic;
2899+ __le64 res_rem_inv_err;
2900+ __le64 res_mem_error;
2901+ __le64 res_srq_err;
2902+ __le64 res_cmp_err;
2903+ __le64 res_invalid_dup_rkey;
2904+ __le64 res_wqe_format_err;
2905+ __le64 res_cq_load_err;
2906+ __le64 res_srq_load_err;
2907+ __le64 res_tx_pci_err;
2908+ __le64 res_rx_pci_err;
2909+};
2910+
27862911 /* QP error notification event (16 bytes) */
27872912 struct creq_qp_error_notification {
27882913 u8 type;
drivers/infiniband/hw/cxgb4/cm.c+16−11
@@ -257,8 +257,8 @@ static void set_emss(struct c4iw_ep *ep, u16 opt) · 256 unmodified lines
257257 if (ep->emss < 128)
258258 ep->emss = 128;
259259 if (ep->emss & 7)
260− pr_warn("Warning: misaligned mtu idx %u mss %u emss=%u\n",
261− TCPOPT_MSS_G(opt), ep->mss, ep->emss);
260+ pr_debug("Warning: misaligned mtu idx %u mss %u emss=%u\n",
261+ TCPOPT_MSS_G(opt), ep->mss, ep->emss);
262262 pr_debug("mss_idx %u mss %u emss=%u\n", TCPOPT_MSS_G(opt), ep->mss,
263263 ep->emss);
264264 }
@@ -2733,9 +2733,8 @@ static int peer_abort(struct c4iw_dev *dev, struct sk_buff *skb) · 2468 unmodified lines
27332733 return 0;
27342734
27352735 if (cxgb_is_neg_adv(req->status)) {
2736− pr_warn("%s Negative advice on abort- tid %u status %d (%s)\n",
2737− __func__, ep->hwtid, req->status,
2738− neg_adv_str(req->status));
2736+ pr_debug("Negative advice on abort- tid %u status %d (%s)\n",
2737+ ep->hwtid, req->status, neg_adv_str(req->status));
27392738 ep->stats.abort_neg_adv++;
27402739 mutex_lock(&dev->rdev.stats.lock);
27412740 dev->rdev.stats.neg_adv++;
@@ -3567,8 +3566,8 @@ int c4iw_ep_disconnect(struct c4iw_ep *ep, int abrupt, gfp_t gfp) · 825 unmodified lines
35673566 case MORIBUND:
35683567 case ABORTING:
35693568 case DEAD:
3570− pr_info("%s ignoring disconnect ep %p state %u\n",
3571− __func__, ep, ep->com.state);
3569+ pr_debug("ignoring disconnect ep %p state %u\n",
3570+ ep, ep->com.state);
35723571 break;
35733572 default:
35743573 WARN_ONCE(1, "Bad endpoint state %u\n", ep->com.state);
@@ -4097,9 +4096,15 @@ static void process_work(struct work_struct *work) · 522 unmodified lines
40974096 dev = *((struct c4iw_dev **) (skb->cb + sizeof(void *)));
40984097 opcode = rpl->ot.opcode;
40994098
4100− ret = work_handlers[opcode](dev, skb);
4101− if (!ret)
4099+ if (opcode >= ARRAY_SIZE(work_handlers) ||
4100+ !work_handlers[opcode]) {
4101+ pr_err("No handler for opcode 0x%x.\n", opcode);
41024102 kfree_skb(skb);
4103+ } else {
4104+ ret = work_handlers[opcode](dev, skb);
4105+ if (!ret)
4106+ kfree_skb(skb);
4107+ }
41034108 process_timedout_eps();
41044109 }
41054110 }
@@ -4201,8 +4206,8 @@ static int peer_abort_intr(struct c4iw_dev *dev, struct sk_buff *skb) · 95 unmodified lines
42014206 return 0;
42024207 }
42034208 if (cxgb_is_neg_adv(req->status)) {
4204− pr_warn("%s Negative advice on abort- tid %u status %d (%s)\n",
4205− __func__, ep->hwtid, req->status,
4209+ pr_debug("Negative advice on abort- tid %u status %d (%s)\n",
4210+ ep->hwtid, req->status,
42064211 neg_adv_str(req->status));
42074212 goto out;
42084213 }
drivers/infiniband/hw/cxgb4/device.c+16−20
@@ -66,7 +66,7 @@ MODULE_PARM_DESC(c4iw_wr_log_size_order, · 65 unmodified lines
6666
6767 static LIST_HEAD(uld_ctx_list);
6868 static DEFINE_MUTEX(dev_mutex);
69−struct workqueue_struct *reg_workq;
69+static struct workqueue_struct *reg_workq;
7070
7171 #define DB_FC_RESUME_SIZE 64
7272 #define DB_FC_RESUME_DELAY 1
@@ -108,19 +108,19 @@ void c4iw_log_wr_stats(struct t4_wq *wq, struct t4_cqe *cqe) · 35 unmodified lines
108108 idx = (atomic_inc_return(&wq->rdev->wr_log_idx) - 1) &
109109 (wq->rdev->wr_log_size - 1);
110110 le.poll_sge_ts = cxgb4_read_sge_timestamp(wq->rdev->lldi.ports[0]);
111− getnstimeofday(&le.poll_host_ts);
111+ le.poll_host_time = ktime_get();
112112 le.valid = 1;
113113 le.cqe_sge_ts = CQE_TS(cqe);
114114 if (SQ_TYPE(cqe)) {
115115 le.qid = wq->sq.qid;
116116 le.opcode = CQE_OPCODE(cqe);
117− le.post_host_ts = wq->sq.sw_sq[wq->sq.cidx].host_ts;
117+ le.post_host_time = wq->sq.sw_sq[wq->sq.cidx].host_time;
118118 le.post_sge_ts = wq->sq.sw_sq[wq->sq.cidx].sge_ts;
119119 le.wr_id = CQE_WRID_SQ_IDX(cqe);
120120 } else {
121121 le.qid = wq->rq.qid;
122122 le.opcode = FW_RI_RECEIVE;
123− le.post_host_ts = wq->rq.sw_rq[wq->rq.cidx].host_ts;
123+ le.post_host_time = wq->rq.sw_rq[wq->rq.cidx].host_time;
124124 le.post_sge_ts = wq->rq.sw_rq[wq->rq.cidx].sge_ts;
125125 le.wr_id = CQE_WRID_MSN(cqe);
126126 }
@@ -130,9 +130,9 @@ void c4iw_log_wr_stats(struct t4_wq *wq, struct t4_cqe *cqe) · 3 unmodified lines
130130 static int wr_log_show(struct seq_file *seq, void *v)
131131 {
132132 struct c4iw_dev *dev = seq->private;
133− struct timespec prev_ts = {0, 0};
133+ ktime_t prev_time;
134134 struct wr_log_entry *lep;
135− int prev_ts_set = 0;
135+ int prev_time_set = 0;
136136 int idx, end;
137137
138138 #define ts2ns(ts) div64_u64((ts) * dev->rdev.lldi.cclk_ps, 1000)
@@ -145,33 +145,29 @@ static int wr_log_show(struct seq_file *seq, void *v) · 6 unmodified lines
145145 lep = &dev->rdev.wr_log[idx];
146146 while (idx != end) {
147147 if (lep->valid) {
148− if (!prev_ts_set) {
149− prev_ts_set = 1;
150− prev_ts = lep->poll_host_ts;
148+ if (!prev_time_set) {
149+ prev_time_set = 1;
150+ prev_time = lep->poll_host_time;
151151 }
152− seq_printf(seq, "%04u: sec %lu nsec %lu qid %u opcode "
153− "%u %s 0x%x host_wr_delta sec %lu nsec %lu "
152+ seq_printf(seq, "%04u: nsec %llu qid %u opcode "
153+ "%u %s 0x%x host_wr_delta nsec %llu "
154154 "post_sge_ts 0x%llx cqe_sge_ts 0x%llx "
155155 "poll_sge_ts 0x%llx post_poll_delta_ns %llu "
156156 "cqe_poll_delta_ns %llu\n",
157157 idx,
158− timespec_sub(lep->poll_host_ts,
159− prev_ts).tv_sec,
160− timespec_sub(lep->poll_host_ts,
161− prev_ts).tv_nsec,
158+ ktime_to_ns(ktime_sub(lep->poll_host_time,
159+ prev_time)),
162160 lep->qid, lep->opcode,
163161 lep->opcode == FW_RI_RECEIVE ?
164162 "msn" : "wrid",
165163 lep->wr_id,
166− timespec_sub(lep->poll_host_ts,
167− lep->post_host_ts).tv_sec,
168− timespec_sub(lep->poll_host_ts,
169− lep->post_host_ts).tv_nsec,
164+ ktime_to_ns(ktime_sub(lep->poll_host_time,
165+ lep->post_host_time)),
170166 lep->post_sge_ts, lep->cqe_sge_ts,
171167 lep->poll_sge_ts,
172168 ts2ns(lep->poll_sge_ts - lep->post_sge_ts),
173169 ts2ns(lep->poll_sge_ts - lep->cqe_sge_ts));
174− prev_ts = lep->poll_host_ts;
170+ prev_time = lep->poll_host_time;
175171 }
176172 idx++;
177173 if (idx > (dev->rdev.wr_log_size - 1))
drivers/infiniband/hw/cxgb4/ev.c+1−1
@@ -236,7 +236,7 @@ int c4iw_ev_handler(struct c4iw_dev *dev, u32 qid) · 235 unmodified lines
236236 if (atomic_dec_and_test(&chp->refcnt))
237237 wake_up(&chp->wait);
238238 } else {
239− pr_warn("%s unknown cqid 0x%x\n", __func__, qid);
239+ pr_debug("unknown cqid 0x%x\n", qid);
240240 spin_unlock_irqrestore(&dev->lock, flag);
241241 }
242242 return 0;
drivers/infiniband/hw/cxgb4/iw_cxgb4.h+2−2
@@ -153,8 +153,8 @@ struct c4iw_hw_queue { · 152 unmodified lines
153153 };
154154
155155 struct wr_log_entry {
156− struct timespec post_host_ts;
157− struct timespec poll_host_ts;
156+ ktime_t post_host_time;
157+ ktime_t poll_host_time;
158158 u64 post_sge_ts;
159159 u64 cqe_sge_ts;
160160 u64 poll_sge_ts;
drivers/infiniband/hw/cxgb4/qp.c+3−3
@@ -1042,7 +1042,7 @@ int c4iw_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, · 1041 unmodified lines
10421042 if (c4iw_wr_log) {
10431043 swsqe->sge_ts = cxgb4_read_sge_timestamp(
10441044 qhp->rhp->rdev.lldi.ports[0]);
1045− getnstimeofday(&swsqe->host_ts);
1045+ swsqe->host_time = ktime_get();
10461046 }
10471047
10481048 init_wr_hdr(wqe, qhp->wq.sq.pidx, fw_opcode, fw_flags, len16);
@@ -1117,8 +1117,8 @@ int c4iw_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr, · 68 unmodified lines
11171117 qhp->wq.rq.sw_rq[qhp->wq.rq.pidx].sge_ts =
11181118 cxgb4_read_sge_timestamp(
11191119 qhp->rhp->rdev.lldi.ports[0]);
1120− getnstimeofday(
1121− &qhp->wq.rq.sw_rq[qhp->wq.rq.pidx].host_ts);
1120+ qhp->wq.rq.sw_rq[qhp->wq.rq.pidx].host_time =
1121+ ktime_get();
11221122 }
11231123
11241124 wqe->recv.opcode = FW_RI_RECV_WR;
drivers/infiniband/hw/cxgb4/t4.h+2−2
@@ -277,7 +277,7 @@ struct t4_swsqe { · 276 unmodified lines
277277 int signaled;
278278 u16 idx;
279279 int flushed;
280− struct timespec host_ts;
280+ ktime_t host_time;
281281 u64 sge_ts;
282282 };
283283
@@ -318,7 +318,7 @@ struct t4_sq { · 34 unmodified lines
318318
319319 struct t4_swrqe {
320320 u64 wr_id;
321− struct timespec host_ts;
321+ ktime_t host_time;
322322 u64 sge_ts;
323323 };
324324
drivers/infiniband/hw/hfi1/chip.c+31−56
@@ -6518,11 +6518,12 @@ static void _dc_start(struct hfi1_devdata *dd) · 6517 unmodified lines
65186518 if (!dd->dc_shutdown)
65196519 return;
65206520
6521− /*
6522− * Take the 8051 out of reset, wait until 8051 is ready, and set host
6523− * version bit.
6524− */
6525− release_and_wait_ready_8051_firmware(dd);
6521+ /* Take the 8051 out of reset */
6522+ write_csr(dd, DC_DC8051_CFG_RST, 0ull);
6523+ /* Wait until 8051 is ready */
6524+ if (wait_fm_ready(dd, TIMEOUT_8051_START))
6525+ dd_dev_err(dd, "%s: timeout starting 8051 firmware\n",
6526+ __func__);
65266527
65276528 /* Take away reset for LCB and RX FPE (set in lcb_shutdown). */
65286529 write_csr(dd, DCC_CFG_RESET, 0x10);
@@ -8564,23 +8565,27 @@ int write_lcb_csr(struct hfi1_devdata *dd, u32 addr, u64 data) · 2035 unmodified lines
85648565 }
85658566
85668567 /*
8567− * If the 8051 is in reset mode (dd->dc_shutdown == 1), this function
8568− * will still continue executing.
8569− *
85708568 * Returns:
85718569 * < 0 = Linux error, not able to get access
85728570 * > 0 = 8051 command RETURN_CODE
85738571 */
8574−static int _do_8051_command(struct hfi1_devdata *dd, u32 type, u64 in_data,
8575− u64 *out_data)
8572+static int do_8051_command(struct hfi1_devdata *dd, u32 type, u64 in_data,
8573+ u64 *out_data)
85768574 {
85778575 u64 reg, completed;
85788576 int return_code;
85798577 unsigned long timeout;
85808578
8581− lockdep_assert_held(&dd->dc8051_lock);
85828579 hfi1_cdbg(DC8051, "type %d, data 0x%012llx", type, in_data);
85838580
8581+ mutex_lock(&dd->dc8051_lock);
8582+
8583+ /* We can't send any commands to the 8051 if it's in reset */
8584+ if (dd->dc_shutdown) {
8585+ return_code = -ENODEV;
8586+ goto fail;
8587+ }
8588+
85848589 /*
85858590 * If an 8051 host command timed out previously, then the 8051 is
85868591 * stuck.
@@ -8680,29 +8685,6 @@ static int _do_8051_command(struct hfi1_devdata *dd, u32 type, u64 in_data, · 93 unmodified lines
86808685 */
86818686 write_csr(dd, DC_DC8051_CFG_HOST_CMD_0, 0);
86828687
8683−fail:
8684− return return_code;
8685−}
8686−
8687−/*
8688− * Returns:
8689− * < 0 = Linux error, not able to get access
8690− * > 0 = 8051 command RETURN_CODE
8691− */
8692−static int do_8051_command(struct hfi1_devdata *dd, u32 type, u64 in_data,
8693− u64 *out_data)
8694−{
8695− int return_code;
8696−
8697− mutex_lock(&dd->dc8051_lock);
8698− /* We can't send any commands to the 8051 if it's in reset */
8699− if (dd->dc_shutdown) {
8700− return_code = -ENODEV;
8701− goto fail;
8702− }
8703−
8704− return_code = _do_8051_command(dd, type, in_data, out_data);
8705−
87068688 fail:
87078689 mutex_unlock(&dd->dc8051_lock);
87088690 return return_code;
@@ -8713,17 +8695,16 @@ static int set_physical_link_state(struct hfi1_devdata *dd, u64 state) · 4 unmodified lines
87138695 return do_8051_command(dd, HCMD_CHANGE_PHY_STATE, state, NULL);
87148696 }
87158697
8716−static int _load_8051_config(struct hfi1_devdata *dd, u8 field_id,
8717− u8 lane_id, u32 config_data)
8698+int load_8051_config(struct hfi1_devdata *dd, u8 field_id,
8699+ u8 lane_id, u32 config_data)
87188700 {
87198701 u64 data;
87208702 int ret;
87218703
8722− lockdep_assert_held(&dd->dc8051_lock);
87238704 data = (u64)field_id << LOAD_DATA_FIELD_ID_SHIFT
87248705 | (u64)lane_id << LOAD_DATA_LANE_ID_SHIFT
87258706 | (u64)config_data << LOAD_DATA_DATA_SHIFT;
8726− ret = _do_8051_command(dd, HCMD_LOAD_CONFIG_DATA, data, NULL);
8707+ ret = do_8051_command(dd, HCMD_LOAD_CONFIG_DATA, data, NULL);
87278708 if (ret != HCMD_SUCCESS) {
87288709 dd_dev_err(dd,
87298710 "load 8051 config: field id %d, lane %d, err %d\n",
@@ -8732,18 +8713,6 @@ static int _load_8051_config(struct hfi1_devdata *dd, u8 field_id, · 2 unmodified lines
87328713 return ret;
87338714 }
87348715
8735−int load_8051_config(struct hfi1_devdata *dd, u8 field_id,
8736− u8 lane_id, u32 config_data)
8737−{
8738− int return_code;
8739−
8740− mutex_lock(&dd->dc8051_lock);
8741− return_code = _load_8051_config(dd, field_id, lane_id, config_data);
8742− mutex_unlock(&dd->dc8051_lock);
8743−
8744− return return_code;
8745−}
8746−
87478716 /*
87488717 * Read the 8051 firmware "registers". Use the RAM directly. Always
87498718 * set the result, even on error.
@@ -8859,14 +8828,13 @@ int write_host_interface_version(struct hfi1_devdata *dd, u8 version) · 109 unmodified lines
88598828 u32 frame;
88608829 u32 mask;
88618830
8862− lockdep_assert_held(&dd->dc8051_lock);
88638831 mask = (HOST_INTERFACE_VERSION_MASK << HOST_INTERFACE_VERSION_SHIFT);
88648832 read_8051_config(dd, RESERVED_REGISTERS, GENERAL_CONFIG, &frame);
88658833 /* Clear, then set field */
88668834 frame &= ~mask;
88678835 frame |= ((u32)version << HOST_INTERFACE_VERSION_SHIFT);
8868− return _load_8051_config(dd, RESERVED_REGISTERS, GENERAL_CONFIG,
8869− frame);
8836+ return load_8051_config(dd, RESERVED_REGISTERS, GENERAL_CONFIG,
8837+ frame);
88708838 }
88718839
88728840 void read_misc_status(struct hfi1_devdata *dd, u8 *ver_major, u8 *ver_minor,
@@ -9270,6 +9238,14 @@ static int set_local_link_attributes(struct hfi1_pportdata *ppd) · 397 unmodified lines
92709238 if (ret != HCMD_SUCCESS)
92719239 goto set_local_link_attributes_fail;
92729240
9241+ ret = write_host_interface_version(dd, HOST_INTERFACE_VERSION);
9242+ if (ret != HCMD_SUCCESS) {
9243+ dd_dev_err(dd,
9244+ "Failed to set host interface version, return 0x%x\n",
9245+ ret);
9246+ goto set_local_link_attributes_fail;
9247+ }
9248+
92739249 /*
92749250 * DC supports continuous updates.
92759251 */
@@ -14944,9 +14920,8 @@ struct hfi1_devdata *hfi1_init_dd(struct pci_dev *pdev, · 5668 unmodified lines
1494414920
1494514921 if (num_vls < HFI1_MIN_VLS_SUPPORTED ||
1494614922 num_vls > HFI1_MAX_VLS_SUPPORTED) {
14947− hfi1_early_err(&pdev->dev,
14948− "Invalid num_vls %u, using %u VLs\n",
14949− num_vls, HFI1_MAX_VLS_SUPPORTED);
14923+ dd_dev_err(dd, "Invalid num_vls %u, using %u VLs\n",
14924+ num_vls, HFI1_MAX_VLS_SUPPORTED);
1495014925 num_vls = HFI1_MAX_VLS_SUPPORTED;
1495114926 }
1495214927 ppd->vls_supported = num_vls;
drivers/infiniband/hw/hfi1/chip.h+1−1
@@ -508,6 +508,7 @@ · 507 unmodified lines
508508 #define DOWN_REMOTE_REASON_SHIFT 16
509509 #define DOWN_REMOTE_REASON_MASK 0xff
510510
511+#define HOST_INTERFACE_VERSION 1
511512 #define HOST_INTERFACE_VERSION_SHIFT 16
512513 #define HOST_INTERFACE_VERSION_MASK 0xff
513514
@@ -713,7 +714,6 @@ void read_misc_status(struct hfi1_devdata *dd, u8 *ver_major, u8 *ver_minor, · 199 unmodified lines
713714 u8 *ver_patch);
714715 int write_host_interface_version(struct hfi1_devdata *dd, u8 version);
715716 void read_guid(struct hfi1_devdata *dd);
716−int release_and_wait_ready_8051_firmware(struct hfi1_devdata *dd);
717717 int wait_fm_ready(struct hfi1_devdata *dd, u32 mstimeout);
718718 void set_link_down_reason(struct hfi1_pportdata *ppd, u8 lcl_reason,
719719 u8 neigh_reason, u8 rem_reason);
drivers/infiniband/hw/hfi1/driver.c+0−16
@@ -159,22 +159,6 @@ static int hfi1_caps_get(char *buffer, const struct kernel_param *kp) · 158 unmodified lines
159159 return scnprintf(buffer, PAGE_SIZE, "0x%lx", cap_mask);
160160 }
161161
162−const char *get_unit_name(int unit)
163−{
164− static char iname[16];
165−
166− snprintf(iname, sizeof(iname), DRIVER_NAME "_%u", unit);
167− return iname;
168−}
169−
170−const char *get_card_name(struct rvt_dev_info *rdi)
171−{
172− struct hfi1_ibdev *ibdev = container_of(rdi, struct hfi1_ibdev, rdi);
173− struct hfi1_devdata *dd = container_of(ibdev,
174− struct hfi1_devdata, verbs_dev);
175− return get_unit_name(dd->unit);
176−}
177−
178162 struct pci_dev *get_pci_dev(struct rvt_dev_info *rdi)
179163 {
180164 struct hfi1_ibdev *ibdev = container_of(rdi, struct hfi1_ibdev, rdi);
drivers/infiniband/hw/hfi1/firmware.c+16−48
@@ -68,7 +68,6 @@ · 67 unmodified lines
6868 #define ALT_FW_FABRIC_NAME "hfi1_fabric_d.fw"
6969 #define ALT_FW_SBUS_NAME "hfi1_sbus_d.fw"
7070 #define ALT_FW_PCIE_NAME "hfi1_pcie_d.fw"
71−#define HOST_INTERFACE_VERSION 1
7271
7372 MODULE_FIRMWARE(DEFAULT_FW_8051_NAME_ASIC);
7473 MODULE_FIRMWARE(DEFAULT_FW_FABRIC_NAME);
@@ -975,46 +974,6 @@ int wait_fm_ready(struct hfi1_devdata *dd, u32 mstimeout) · 900 unmodified lines
975974 }
976975 }
977976
978−/*
979− * Clear all reset bits, releasing the 8051.
980− * Wait for firmware to be ready to accept host requests.
981− * Then, set host version bit.
982− *
983− * This function executes even if the 8051 is in reset mode when
984− * dd->dc_shutdown == 1.
985− *
986− * Expects dd->dc8051_lock to be held.
987− */
988−int release_and_wait_ready_8051_firmware(struct hfi1_devdata *dd)
989−{
990− int ret;
991−
992− lockdep_assert_held(&dd->dc8051_lock);
993− /* clear all reset bits, releasing the 8051 */
994− write_csr(dd, DC_DC8051_CFG_RST, 0ull);
995−
996− /*
997− * Wait for firmware to be ready to accept host
998− * requests.
999− */
1000− ret = wait_fm_ready(dd, TIMEOUT_8051_START);
1001− if (ret) {
1002− dd_dev_err(dd, "8051 start timeout, current FW state 0x%x\n",
1003− get_firmware_state(dd));
1004− return ret;
1005− }
1006−
1007− ret = write_host_interface_version(dd, HOST_INTERFACE_VERSION);
1008− if (ret != HCMD_SUCCESS) {
1009− dd_dev_err(dd,
1010− "Failed to set host interface version, return 0x%x\n",
1011− ret);
1012− return -EIO;
1013− }
1014−
1015− return 0;
1016−}
1017−
1018977 /*
1019978 * Load the 8051 firmware.
1020979 */
@@ -1080,22 +1039,31 @@ static int load_8051_firmware(struct hfi1_devdata *dd, · 59 unmodified lines
10801039 if (ret)
10811040 return ret;
10821041
1042+ /* clear all reset bits, releasing the 8051 */
1043+ write_csr(dd, DC_DC8051_CFG_RST, 0ull);
1044+
10831045 /*
1084− * Clear all reset bits, releasing the 8051.
10851046 * DC reset step 5. Wait for firmware to be ready to accept host
10861047 * requests.
1087− * Then, set host version bit.
10881048 */
1089− mutex_lock(&dd->dc8051_lock);
1090− ret = release_and_wait_ready_8051_firmware(dd);
1091− mutex_unlock(&dd->dc8051_lock);
1092− if (ret)
1093− return ret;
1049+ ret = wait_fm_ready(dd, TIMEOUT_8051_START);
1050+ if (ret) { /* timed out */
1051+ dd_dev_err(dd, "8051 start timeout, current state 0x%x\n",
1052+ get_firmware_state(dd));
1053+ return -ETIMEDOUT;
1054+ }
10941055
10951056 read_misc_status(dd, &ver_major, &ver_minor, &ver_patch);
10961057 dd_dev_info(dd, "8051 firmware version %d.%d.%d\n",
10971058 (int)ver_major, (int)ver_minor, (int)ver_patch);
10981059 dd->dc8051_ver = dc8051_ver(ver_major, ver_minor, ver_patch);
1060+ ret = write_host_interface_version(dd, HOST_INTERFACE_VERSION);
1061+ if (ret != HCMD_SUCCESS) {
1062+ dd_dev_err(dd,
1063+ "Failed to set host interface version, return 0x%x\n",
1064+ ret);
1065+ return -EIO;
1066+ }
10991067
11001068 return 0;
11011069 }
drivers/infiniband/hw/hfi1/hfi.h+13−12
@@ -1623,7 +1623,7 @@ static int ingress_pkey_table_search(struct hfi1_pportdata *ppd, u16 pkey) · 1622 unmodified lines
16231623 * the 'error info' for this failure.
16241624 */
16251625 static void ingress_pkey_table_fail(struct hfi1_pportdata *ppd, u16 pkey,
1626− u16 slid)
1626+ u32 slid)
16271627 {
16281628 struct hfi1_devdata *dd = ppd->dd;
16291629
@@ -1971,8 +1971,6 @@ int get_platform_config_field(struct hfi1_devdata *dd, · 341 unmodified lines
19711971 table_type, int table_index, int field_index,
19721972 u32 *data, u32 len);
19731973
1974−const char *get_unit_name(int unit);
1975−const char *get_card_name(struct rvt_dev_info *rdi);
19761974 struct pci_dev *get_pci_dev(struct rvt_dev_info *rdi);
19771975
19781976 /*
@@ -2122,39 +2120,42 @@ static inline u64 hfi1_pkt_base_sdma_integrity(struct hfi1_devdata *dd) · 143 unmodified lines
21222120
21232121 #define dd_dev_emerg(dd, fmt, ...) \
21242122 dev_emerg(&(dd)->pcidev->dev, "%s: " fmt, \
2125− get_unit_name((dd)->unit), ##__VA_ARGS__)
2123+ rvt_get_ibdev_name(&(dd)->verbs_dev.rdi), ##__VA_ARGS__)
21262124
21272125 #define dd_dev_err(dd, fmt, ...) \
21282126 dev_err(&(dd)->pcidev->dev, "%s: " fmt, \
2129− get_unit_name((dd)->unit), ##__VA_ARGS__)
2127+ rvt_get_ibdev_name(&(dd)->verbs_dev.rdi), ##__VA_ARGS__)
21302128
21312129 #define dd_dev_err_ratelimited(dd, fmt, ...) \
21322130 dev_err_ratelimited(&(dd)->pcidev->dev, "%s: " fmt, \
2133− get_unit_name((dd)->unit), ##__VA_ARGS__)
2131+ rvt_get_ibdev_name(&(dd)->verbs_dev.rdi), \
2132+ ##__VA_ARGS__)
21342133
21352134 #define dd_dev_warn(dd, fmt, ...) \
21362135 dev_warn(&(dd)->pcidev->dev, "%s: " fmt, \
2137− get_unit_name((dd)->unit), ##__VA_ARGS__)
2136+ rvt_get_ibdev_name(&(dd)->verbs_dev.rdi), ##__VA_ARGS__)
21382137
21392138 #define dd_dev_warn_ratelimited(dd, fmt, ...) \
21402139 dev_warn_ratelimited(&(dd)->pcidev->dev, "%s: " fmt, \
2141− get_unit_name((dd)->unit), ##__VA_ARGS__)
2140+ rvt_get_ibdev_name(&(dd)->verbs_dev.rdi), \
2141+ ##__VA_ARGS__)
21422142
21432143 #define dd_dev_info(dd, fmt, ...) \
21442144 dev_info(&(dd)->pcidev->dev, "%s: " fmt, \
2145− get_unit_name((dd)->unit), ##__VA_ARGS__)
2145+ rvt_get_ibdev_name(&(dd)->verbs_dev.rdi), ##__VA_ARGS__)
21462146
21472147 #define dd_dev_info_ratelimited(dd, fmt, ...) \
21482148 dev_info_ratelimited(&(dd)->pcidev->dev, "%s: " fmt, \
2149− get_unit_name((dd)->unit), ##__VA_ARGS__)
2149+ rvt_get_ibdev_name(&(dd)->verbs_dev.rdi), \
2150+ ##__VA_ARGS__)
21502151
21512152 #define dd_dev_dbg(dd, fmt, ...) \
21522153 dev_dbg(&(dd)->pcidev->dev, "%s: " fmt, \
2153− get_unit_name((dd)->unit), ##__VA_ARGS__)
2154+ rvt_get_ibdev_name(&(dd)->verbs_dev.rdi), ##__VA_ARGS__)
21542155
21552156 #define hfi1_dev_porterr(dd, port, fmt, ...) \
21562157 dev_err(&(dd)->pcidev->dev, "%s: port %u: " fmt, \
2157− get_unit_name((dd)->unit), (port), ##__VA_ARGS__)
2158+ rvt_get_ibdev_name(&(dd)->verbs_dev.rdi), (port), ##__VA_ARGS__)
21582159
21592160 /*
21602161 * this is used for formatting hw error messages...
drivers/infiniband/hw/hfi1/init.c+2−0
@@ -1272,6 +1272,8 @@ struct hfi1_devdata *hfi1_alloc_devdata(struct pci_dev *pdev, size_t extra) · 1271 unmodified lines
12721272 "Could not allocate unit ID: error %d\n", -ret);
12731273 goto bail;
12741274 }
1275+ rvt_set_ibdev_name(&dd->verbs_dev.rdi, "%s_%d", class_name(), dd->unit);
1276+
12751277 /*
12761278 * Initialize all locks for the device. This needs to be as early as
12771279 * possible so locks are usable.
drivers/infiniband/hw/hfi1/mad.c+1−5
@@ -4348,11 +4348,7 @@ static int opa_local_smp_check(struct hfi1_ibport *ibp, · 4347 unmodified lines
43484348 */
43494349 if (pkey == LIM_MGMT_P_KEY || pkey == FULL_MGMT_P_KEY)
43504350 return 0;
4351− /*
4352− * On OPA devices it is okay to lose the upper 16 bits of LID as this
4353− * information is obtained elsewhere. Mask off the upper 16 bits.
4354− */
4355− ingress_pkey_table_fail(ppd, pkey, ib_lid_cpu16(0xFFFF & in_wc->slid));
4351+ ingress_pkey_table_fail(ppd, pkey, in_wc->slid);
43564352 return 1;
43574353 }
43584354
drivers/infiniband/hw/hfi1/qp.c+8−2
@@ -556,6 +556,8 @@ void qp_iter_print(struct seq_file *s, struct rvt_qp_iter *iter) · 555 unmodified lines
556556 struct sdma_engine *sde;
557557 struct send_context *send_context;
558558 struct rvt_ack_entry *e = NULL;
559+ struct rvt_srq *srq = qp->ibqp.srq ?
560+ ibsrq_to_rvtsrq(qp->ibqp.srq) : NULL;
559561
560562 sde = qp_to_sdma_engine(qp, priv->s_sc);
561563 wqe = rvt_get_swqe_ptr(qp, qp->s_last);
@@ -563,7 +565,7 @@ void qp_iter_print(struct seq_file *s, struct rvt_qp_iter *iter) · 1 unmodified line
563565 if (qp->s_ack_queue)
564566 e = &qp->s_ack_queue[qp->s_tail_ack_queue];
565567 seq_printf(s,
566− "N %d %s QP %x R %u %s %u %u %u f=%x %u %u %u %u %u %u SPSN %x %x %x %x %x RPSN %x S(%u %u %u %u %u %u %u) R(%u %u %u) RQP %x LID %x SL %u MTU %u %u %u %u %u SDE %p,%u SC %p,%u SCQ %u %u PID %d OS %x %x E %x %x %x\n",
568+ "N %d %s QP %x R %u %s %u %u %u f=%x %u %u %u %u %u %u SPSN %x %x %x %x %x RPSN %x S(%u %u %u %u %u %u %u) R(%u %u %u) RQP %x LID %x SL %u MTU %u %u %u %u %u SDE %p,%u SC %p,%u SCQ %u %u PID %d OS %x %x E %x %x %x RNR %d %s %d\n",
567569 iter->n,
568570 qp_idle(qp) ? "I" : "B",
569571 qp->ibqp.qp_num,
@@ -610,7 +612,11 @@ void qp_iter_print(struct seq_file *s, struct rvt_qp_iter *iter) · 40 unmodified lines
610612 /* ack queue information */
611613 e ? e->opcode : 0,
612614 e ? e->psn : 0,
613− e ? e->lpsn : 0);
615+ e ? e->lpsn : 0,
616+ qp->r_min_rnr_timer,
617+ srq ? "SRQ" : "RQ",
618+ srq ? srq->rq.size : qp->r_rq.size
619+ );
614620 }
615621
616622 void *qp_priv_alloc(struct rvt_dev_info *rdi, struct rvt_qp *qp)
drivers/infiniband/hw/hfi1/rc.c+4−4
@@ -841,11 +841,11 @@ static inline void hfi1_make_rc_ack_16B(struct rvt_qp *qp, · 840 unmodified lines
841841 /* Convert dwords to flits */
842842 len = (*hwords + *nwords) >> 1;
843843
844− hfi1_make_16b_hdr(hdr,
845− ppd->lid | rdma_ah_get_path_bits(&qp->remote_ah_attr),
844+ hfi1_make_16b_hdr(hdr, ppd->lid |
845+ (rdma_ah_get_path_bits(&qp->remote_ah_attr) &
846+ ((1 << ppd->lmc) - 1)),
846847 opa_get_lid(rdma_ah_get_dlid(&qp->remote_ah_attr),
847− 16B),
848− len, pkey, becn, 0, l4, sc5);
848+ 16B), len, pkey, becn, 0, l4, sc5);
849849
850850 bth0 = pkey | (OP(ACKNOWLEDGE) << 24);
851851 bth0 |= extra_bytes << 20;
drivers/infiniband/hw/hfi1/verbs.c+1−5
@@ -1486,7 +1486,7 @@ static int query_port(struct rvt_dev_info *rdi, u8 port_num, · 1485 unmodified lines
14861486 props->max_mtu = mtu_to_enum((!valid_ib_mtu(hfi1_max_mtu) ?
14871487 4096 : hfi1_max_mtu), IB_MTU_4096);
14881488 props->active_mtu = !valid_ib_mtu(ppd->ibmtu) ? props->max_mtu :
1489− mtu_to_enum(ppd->ibmtu, IB_MTU_2048);
1489+ mtu_to_enum(ppd->ibmtu, IB_MTU_4096);
14901490
14911491 /*
14921492 * sm_lid of 0xFFFF needs special handling so that it can
@@ -1844,7 +1844,6 @@ int hfi1_register_ib_device(struct hfi1_devdata *dd) · 351 unmodified lines
18441844 struct hfi1_ibport *ibp = &ppd->ibport_data;
18451845 unsigned i;
18461846 int ret;
1847− size_t lcpysz = IB_DEVICE_NAME_MAX;
18481847
18491848 for (i = 0; i < dd->num_pports; i++)
18501849 init_ibport(ppd + i);
@@ -1872,8 +1871,6 @@ int hfi1_register_ib_device(struct hfi1_devdata *dd) · 21 unmodified lines
18721871 */
18731872 if (!ib_hfi1_sys_image_guid)
18741873 ib_hfi1_sys_image_guid = ibdev->node_guid;
1875− lcpysz = strlcpy(ibdev->name, class_name(), lcpysz);
1876− strlcpy(ibdev->name + lcpysz, "_%d", IB_DEVICE_NAME_MAX - lcpysz);
18771874 ibdev->owner = THIS_MODULE;
18781875 ibdev->phys_port_cnt = dd->num_pports;
18791876 ibdev->dev.parent = &dd->pcidev->dev;
@@ -1893,7 +1890,6 @@ int hfi1_register_ib_device(struct hfi1_devdata *dd) · 13 unmodified lines
18931890 * Fill in rvt info object.
18941891 */
18951892 dd->verbs_dev.rdi.driver_f.port_callback = hfi1_create_port_files;
1896− dd->verbs_dev.rdi.driver_f.get_card_name = get_card_name;
18971893 dd->verbs_dev.rdi.driver_f.get_pci_dev = get_pci_dev;
18981894 dd->verbs_dev.rdi.driver_f.check_ah = hfi1_check_ah;
18991895 dd->verbs_dev.rdi.driver_f.notify_new_ah = hfi1_notify_new_ah;
drivers/infiniband/hw/hns/Makefile+1−1
@@ -5,7 +5,7 @@ · 4 unmodified lines
55 ccflags-y := -Idrivers/net/ethernet/hisilicon/hns3
66
77 obj-$(CONFIG_INFINIBAND_HNS) += hns-roce.o
8−hns-roce-objs := hns_roce_main.o hns_roce_cmd.o hns_roce_eq.o hns_roce_pd.o \
8+hns-roce-objs := hns_roce_main.o hns_roce_cmd.o hns_roce_pd.o \
99 hns_roce_ah.o hns_roce_hem.o hns_roce_mr.o hns_roce_qp.o \
1010 hns_roce_cq.o hns_roce_alloc.o
1111 obj-$(CONFIG_INFINIBAND_HNS_HIP06) += hns-roce-hw-v1.o
drivers/infiniband/hw/hns/hns_roce_cmd.c+1−0
@@ -103,6 +103,7 @@ void hns_roce_cmd_event(struct hns_roce_dev *hr_dev, u16 token, u8 status, · 102 unmodified lines
103103 context->out_param = out_param;
104104 complete(&context->done);
105105 }
106+EXPORT_SYMBOL_GPL(hns_roce_cmd_event);
106107
107108 /* this should be called with "use_events" */
108109 static int __hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param,
drivers/infiniband/hw/hns/hns_roce_cmd.h+10−0
@@ -88,6 +88,16 @@ enum { · 87 unmodified lines
8888 HNS_ROCE_CMD_DESTROY_SRQC_BT0 = 0x38,
8989 HNS_ROCE_CMD_DESTROY_SRQC_BT1 = 0x39,
9090 HNS_ROCE_CMD_DESTROY_SRQC_BT2 = 0x3a,
91+
92+ /* EQC commands */
93+ HNS_ROCE_CMD_CREATE_AEQC = 0x80,
94+ HNS_ROCE_CMD_MODIFY_AEQC = 0x81,
95+ HNS_ROCE_CMD_QUERY_AEQC = 0x82,
96+ HNS_ROCE_CMD_DESTROY_AEQC = 0x83,
97+ HNS_ROCE_CMD_CREATE_CEQC = 0x90,
98+ HNS_ROCE_CMD_MODIFY_CEQC = 0x91,
99+ HNS_ROCE_CMD_QUERY_CEQC = 0x92,
100+ HNS_ROCE_CMD_DESTROY_CEQC = 0x93,
91101 };
92102
93103 enum {
drivers/infiniband/hw/hns/hns_roce_common.h+11−0
@@ -376,6 +376,12 @@ · 375 unmodified lines
376376 #define ROCEE_RX_CMQ_TAIL_REG 0x07024
377377 #define ROCEE_RX_CMQ_HEAD_REG 0x07028
378378
379+#define ROCEE_VF_MB_CFG0_REG 0x40
380+#define ROCEE_VF_MB_STATUS_REG 0x58
381+
382+#define ROCEE_VF_EQ_DB_CFG0_REG 0x238
383+#define ROCEE_VF_EQ_DB_CFG1_REG 0x23C
384+
379385 #define ROCEE_VF_SMAC_CFG0_REG 0x12000
380386 #define ROCEE_VF_SMAC_CFG1_REG 0x12004
381387
@@ -385,4 +391,9 @@ · 3 unmodified lines
385391 #define ROCEE_VF_SGID_CFG3_REG 0x1000c
386392 #define ROCEE_VF_SGID_CFG4_REG 0x10010
387393
394+#define ROCEE_VF_ABN_INT_CFG_REG 0x13000
395+#define ROCEE_VF_ABN_INT_ST_REG 0x13004
396+#define ROCEE_VF_ABN_INT_EN_REG 0x13008
397+#define ROCEE_VF_EVENT_INT_EN_REG 0x1300c
398+
388399 #endif /* _HNS_ROCE_COMMON_H */
drivers/infiniband/hw/hns/hns_roce_cq.c+10−9
@@ -196,15 +196,14 @@ void hns_roce_free_cq(struct hns_roce_dev *hr_dev, struct hns_roce_cq *hr_cq) · 195 unmodified lines
196196 if (ret)
197197 dev_err(dev, "HW2SW_CQ failed (%d) for CQN %06lx\n", ret,
198198 hr_cq->cqn);
199− if (hr_dev->eq_table.eq) {
200− /* Waiting interrupt process procedure carried out */
201− synchronize_irq(hr_dev->eq_table.eq[hr_cq->vector].irq);
202−
203− /* wait for all interrupt processed */
204− if (atomic_dec_and_test(&hr_cq->refcount))
205− complete(&hr_cq->free);
206− wait_for_completion(&hr_cq->free);
207− }
199+
200+ /* Waiting interrupt process procedure carried out */
201+ synchronize_irq(hr_dev->eq_table.eq[hr_cq->vector].irq);
202+
203+ /* wait for all interrupt processed */
204+ if (atomic_dec_and_test(&hr_cq->refcount))
205+ complete(&hr_cq->free);
206+ wait_for_completion(&hr_cq->free);
208207
209208 spin_lock_irq(&cq_table->lock);
210209 radix_tree_delete(&cq_table->tree, hr_cq->cqn);
@@ -460,6 +459,7 @@ void hns_roce_cq_completion(struct hns_roce_dev *hr_dev, u32 cqn) · 249 unmodified lines
460459 ++cq->arm_sn;
461460 cq->comp(cq);
462461 }
462+EXPORT_SYMBOL_GPL(hns_roce_cq_completion);
463463
464464 void hns_roce_cq_event(struct hns_roce_dev *hr_dev, u32 cqn, int event_type)
465465 {
@@ -482,6 +482,7 @@ void hns_roce_cq_event(struct hns_roce_dev *hr_dev, u32 cqn, int event_type) · 16 unmodified lines
482482 if (atomic_dec_and_test(&cq->refcount))
483483 complete(&cq->free);
484484 }
485+EXPORT_SYMBOL_GPL(hns_roce_cq_event);
485486
486487 int hns_roce_init_cq_table(struct hns_roce_dev *hr_dev)
487488 {
drivers/infiniband/hw/hns/hns_roce_device.h+96−7
@@ -62,12 +62,16 @@ · 61 unmodified lines
6262 #define HNS_ROCE_CQE_WCMD_EMPTY_BIT 0x2
6363 #define HNS_ROCE_MIN_CQE_CNT 16
6464
65−#define HNS_ROCE_MAX_IRQ_NUM 34
65+#define HNS_ROCE_MAX_IRQ_NUM 128
6666
67−#define HNS_ROCE_COMP_VEC_NUM 32
67+#define EQ_ENABLE 1
68+#define EQ_DISABLE 0
6869
69−#define HNS_ROCE_AEQE_VEC_NUM 1
70−#define HNS_ROCE_AEQE_OF_VEC_NUM 1
70+#define HNS_ROCE_CEQ 0
71+#define HNS_ROCE_AEQ 1
72+
73+#define HNS_ROCE_CEQ_ENTRY_SIZE 0x4
74+#define HNS_ROCE_AEQ_ENTRY_SIZE 0x10
7175
7276 /* 4G/4K = 1M */
7377 #define HNS_ROCE_SL_SHIFT 28
@@ -130,6 +134,7 @@ enum hns_roce_event { · 56 unmodified lines
130134 HNS_ROCE_EVENT_TYPE_DB_OVERFLOW = 0x12,
131135 HNS_ROCE_EVENT_TYPE_MB = 0x13,
132136 HNS_ROCE_EVENT_TYPE_CEQ_OVERFLOW = 0x14,
137+ HNS_ROCE_EVENT_TYPE_FLR = 0x15,
133138 };
134139
135140 /* Local Work Queue Catastrophic Error,SUBTYPE 0x5 */
@@ -173,6 +178,7 @@ enum { · 37 unmodified lines
173178 enum {
174179 HNS_ROCE_CAP_FLAG_REREG_MR = BIT(0),
175180 HNS_ROCE_CAP_FLAG_ROCE_V1_V2 = BIT(1),
181+ HNS_ROCE_CAP_FLAG_RQ_INLINE = BIT(2)
176182 };
177183
178184 enum hns_roce_mtt_type {
@@ -441,6 +447,21 @@ struct hns_roce_cmd_mailbox { · 262 unmodified lines
441447
442448 struct hns_roce_dev;
443449
450+struct hns_roce_rinl_sge {
451+ void *addr;
452+ u32 len;
453+};
454+
455+struct hns_roce_rinl_wqe {
456+ struct hns_roce_rinl_sge *sg_list;
457+ u32 sge_cnt;
458+};
459+
460+struct hns_roce_rinl_buf {
461+ struct hns_roce_rinl_wqe *wqe_list;
462+ u32 wqe_cnt;
463+};
464+
444465 struct hns_roce_qp {
445466 struct ib_qp ibqp;
446467 struct hns_roce_buf hr_buf;
@@ -462,7 +483,9 @@ struct hns_roce_qp { · 15 unmodified lines
462483 u8 resp_depth;
463484 u8 state;
464485 u32 access_flags;
486+ u32 atomic_rd_en;
465487 u32 pkey_index;
488+ u32 qkey;
466489 void (*event)(struct hns_roce_qp *,
467490 enum hns_roce_event);
468491 unsigned long qpn;
@@ -472,6 +495,8 @@ struct hns_roce_qp { · 3 unmodified lines
472495
473496 struct hns_roce_sge sge;
474497 u32 next_sge;
498+
499+ struct hns_roce_rinl_buf rq_inl_buf;
475500 };
476501
477502 struct hns_roce_sqp {
@@ -485,6 +510,45 @@ struct hns_roce_ib_iboe { · 7 unmodified lines
485510 u8 phy_port[HNS_ROCE_MAX_PORTS];
486511 };
487512
513+enum {
514+ HNS_ROCE_EQ_STAT_INVALID = 0,
515+ HNS_ROCE_EQ_STAT_VALID = 2,
516+};
517+
518+struct hns_roce_ceqe {
519+ u32 comp;
520+};
521+
522+struct hns_roce_aeqe {
523+ u32 asyn;
524+ union {
525+ struct {
526+ u32 qp;
527+ u32 rsv0;
528+ u32 rsv1;
529+ } qp_event;
530+
531+ struct {
532+ u32 cq;
533+ u32 rsv0;
534+ u32 rsv1;
535+ } cq_event;
536+
537+ struct {
538+ u32 ceqe;
539+ u32 rsv0;
540+ u32 rsv1;
541+ } ce_event;
542+
543+ struct {
544+ __le64 out_param;
545+ __le16 token;
546+ u8 status;
547+ u8 rsv0;
548+ } __packed cmd;
549+ } event;
550+};
551+
488552 struct hns_roce_eq {
489553 struct hns_roce_dev *hr_dev;
490554 void __iomem *doorbell;
@@ -498,11 +562,31 @@ struct hns_roce_eq { · 7 unmodified lines
498562 int log_page_size;
499563 int cons_index;
500564 struct hns_roce_buf_list *buf_list;
565+ int over_ignore;
566+ int coalesce;
567+ int arm_st;
568+ u64 eqe_ba;
569+ int eqe_ba_pg_sz;
570+ int eqe_buf_pg_sz;
571+ int hop_num;
572+ u64 *bt_l0; /* Base address table for L0 */
573+ u64 **bt_l1; /* Base address table for L1 */
574+ u64 **buf;
575+ dma_addr_t l0_dma;
576+ dma_addr_t *l1_dma;
577+ dma_addr_t *buf_dma;
578+ u32 l0_last_num; /* L0 last chunk num */
579+ u32 l1_last_num; /* L1 last chunk num */
580+ int eq_max_cnt;
581+ int eq_period;
582+ int shift;
583+ dma_addr_t cur_eqe_ba;
584+ dma_addr_t nxt_eqe_ba;
501585 };
502586
503587 struct hns_roce_eq_table {
504588 struct hns_roce_eq *eq;
505− void __iomem **eqc_base;
589+ void __iomem **eqc_base; /* only for hw v1 */
506590 };
507591
508592 struct hns_roce_caps {
@@ -528,7 +612,7 @@ struct hns_roce_caps { · 19 unmodified lines
528612 u32 min_wqes;
529613 int reserved_cqs;
530614 int num_aeq_vectors; /* 1 */
531− int num_comp_vectors; /* 32 ceq */
615+ int num_comp_vectors;
532616 int num_other_vectors;
533617 int num_mtpts;
534618 u32 num_mtt_segs;
@@ -550,7 +634,7 @@ struct hns_roce_caps { · 15 unmodified lines
550634 u32 pbl_buf_pg_sz;
551635 u32 pbl_hop_num;
552636 int aeqe_depth;
553− int ceqe_depth[HNS_ROCE_COMP_VEC_NUM];
637+ int ceqe_depth;
554638 enum ib_mtu max_mtu;
555639 u32 qpc_bt_num;
556640 u32 srqc_bt_num;
@@ -574,6 +658,9 @@ struct hns_roce_caps { · 17 unmodified lines
574658 u32 cqe_ba_pg_sz;
575659 u32 cqe_buf_pg_sz;
576660 u32 cqe_hop_num;
661+ u32 eqe_ba_pg_sz;
662+ u32 eqe_buf_pg_sz;
663+ u32 eqe_hop_num;
577664 u32 chunk_sz; /* chunk size in non multihop mode*/
578665 u64 flags;
579666 };
@@ -623,6 +710,8 @@ struct hns_roce_hw { · 43 unmodified lines
623710 int (*dereg_mr)(struct hns_roce_dev *hr_dev, struct hns_roce_mr *mr);
624711 int (*destroy_cq)(struct ib_cq *ibcq);
625712 int (*modify_cq)(struct ib_cq *cq, u16 cq_count, u16 cq_period);
713+ int (*init_eq)(struct hns_roce_dev *hr_dev);
714+ void (*cleanup_eq)(struct hns_roce_dev *hr_dev);
626715 };
627716
628717 struct hns_roce_dev {
drivers/infiniband/hw/hns/hns_roce_eq.c+0−759
@@ -1,759 +0,0 @@
1−/*
2− * Copyright (c) 2016 Hisilicon Limited.
3− *
4− * This software is available to you under a choice of one of two
5− * licenses. You may choose to be licensed under the terms of the GNU
6− * General Public License (GPL) Version 2, available from the file
7− * COPYING in the main directory of this source tree, or the
8− * OpenIB.org BSD license below:
9− *
10− * Redistribution and use in source and binary forms, with or
11− * without modification, are permitted provided that the following
12− * conditions are met:
13− *
14− * - Redistributions of source code must retain the above
15− * copyright notice, this list of conditions and the following
16− * disclaimer.
17− *
18− * - Redistributions in binary form must reproduce the above
19− * copyright notice, this list of conditions and the following
20− * disclaimer in the documentation and/or other materials
21− * provided with the distribution.
22− *
23− * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
24− * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
25− * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
26− * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
27− * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
28− * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
29− * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
30− * SOFTWARE.
31− */
32−
33−#include <linux/platform_device.h>
34−#include <linux/interrupt.h>
35−#include "hns_roce_common.h"
36−#include "hns_roce_device.h"
37−#include "hns_roce_eq.h"
38−
39−static void eq_set_cons_index(struct hns_roce_eq *eq, int req_not)
40−{
41− roce_raw_write((eq->cons_index & CONS_INDEX_MASK) |
42− (req_not << eq->log_entries), eq->doorbell);
43− /* Memory barrier */
44− mb();
45−}
46−
47−static struct hns_roce_aeqe *get_aeqe(struct hns_roce_eq *eq, u32 entry)
48−{
49− unsigned long off = (entry & (eq->entries - 1)) *
50− HNS_ROCE_AEQ_ENTRY_SIZE;
51−
52− return (struct hns_roce_aeqe *)((u8 *)
53− (eq->buf_list[off / HNS_ROCE_BA_SIZE].buf) +
54− off % HNS_ROCE_BA_SIZE);
55−}
56−
57−static struct hns_roce_aeqe *next_aeqe_sw(struct hns_roce_eq *eq)
58−{
59− struct hns_roce_aeqe *aeqe = get_aeqe(eq, eq->cons_index);
60−
61− return (roce_get_bit(aeqe->asyn, HNS_ROCE_AEQE_U32_4_OWNER_S) ^
62− !!(eq->cons_index & eq->entries)) ? aeqe : NULL;
63−}
64−
65−static void hns_roce_wq_catas_err_handle(struct hns_roce_dev *hr_dev,
66− struct hns_roce_aeqe *aeqe, int qpn)
67−{
68− struct device *dev = &hr_dev->pdev->dev;
69−
70− dev_warn(dev, "Local Work Queue Catastrophic Error.\n");
71− switch (roce_get_field(aeqe->asyn, HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_M,
72− HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_S)) {
73− case HNS_ROCE_LWQCE_QPC_ERROR:
74− dev_warn(dev, "QP %d, QPC error.\n", qpn);
75− break;
76− case HNS_ROCE_LWQCE_MTU_ERROR:
77− dev_warn(dev, "QP %d, MTU error.\n", qpn);
78− break;
79− case HNS_ROCE_LWQCE_WQE_BA_ADDR_ERROR:
80− dev_warn(dev, "QP %d, WQE BA addr error.\n", qpn);
81− break;
82− case HNS_ROCE_LWQCE_WQE_ADDR_ERROR:
83− dev_warn(dev, "QP %d, WQE addr error.\n", qpn);
84− break;
85− case HNS_ROCE_LWQCE_SQ_WQE_SHIFT_ERROR:
86− dev_warn(dev, "QP %d, WQE shift error\n", qpn);
87− break;
88− case HNS_ROCE_LWQCE_SL_ERROR:
89− dev_warn(dev, "QP %d, SL error.\n", qpn);
90− break;
91− case HNS_ROCE_LWQCE_PORT_ERROR:
92− dev_warn(dev, "QP %d, port error.\n", qpn);
93− break;
94− default:
95− break;
96− }
97−}
98−
99−static void hns_roce_local_wq_access_err_handle(struct hns_roce_dev *hr_dev,
100− struct hns_roce_aeqe *aeqe,
101− int qpn)
102−{
103− struct device *dev = &hr_dev->pdev->dev;
104−
105− dev_warn(dev, "Local Access Violation Work Queue Error.\n");
106− switch (roce_get_field(aeqe->asyn, HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_M,
107− HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_S)) {
108− case HNS_ROCE_LAVWQE_R_KEY_VIOLATION:
109− dev_warn(dev, "QP %d, R_key violation.\n", qpn);
110− break;
111− case HNS_ROCE_LAVWQE_LENGTH_ERROR:
112− dev_warn(dev, "QP %d, length error.\n", qpn);
113− break;
114− case HNS_ROCE_LAVWQE_VA_ERROR:
115− dev_warn(dev, "QP %d, VA error.\n", qpn);
116− break;
117− case HNS_ROCE_LAVWQE_PD_ERROR:
118− dev_err(dev, "QP %d, PD error.\n", qpn);
119− break;
120− case HNS_ROCE_LAVWQE_RW_ACC_ERROR:
121− dev_warn(dev, "QP %d, rw acc error.\n", qpn);
122− break;
123− case HNS_ROCE_LAVWQE_KEY_STATE_ERROR:
124− dev_warn(dev, "QP %d, key state error.\n", qpn);
125− break;
126− case HNS_ROCE_LAVWQE_MR_OPERATION_ERROR:
127− dev_warn(dev, "QP %d, MR operation error.\n", qpn);
128− break;
129− default:
130− break;
131− }
132−}
133−
134−static void hns_roce_qp_err_handle(struct hns_roce_dev *hr_dev,
135− struct hns_roce_aeqe *aeqe,
136− int event_type)
137−{
138− struct device *dev = &hr_dev->pdev->dev;
139− int phy_port;
140− int qpn;
141−
142− qpn = roce_get_field(aeqe->event.qp_event.qp,
143− HNS_ROCE_AEQE_EVENT_QP_EVENT_QP_QPN_M,
144− HNS_ROCE_AEQE_EVENT_QP_EVENT_QP_QPN_S);
145− phy_port = roce_get_field(aeqe->event.qp_event.qp,
146− HNS_ROCE_AEQE_EVENT_QP_EVENT_PORT_NUM_M,
147− HNS_ROCE_AEQE_EVENT_QP_EVENT_PORT_NUM_S);
148− if (qpn <= 1)
149− qpn = HNS_ROCE_MAX_PORTS * qpn + phy_port;
150−
151− switch (event_type) {
152− case HNS_ROCE_EVENT_TYPE_INV_REQ_LOCAL_WQ_ERROR:
153− dev_warn(dev, "Invalid Req Local Work Queue Error.\n"
154− "QP %d, phy_port %d.\n", qpn, phy_port);
155− break;
156− case HNS_ROCE_EVENT_TYPE_WQ_CATAS_ERROR:
157− hns_roce_wq_catas_err_handle(hr_dev, aeqe, qpn);
158− break;
159− case HNS_ROCE_EVENT_TYPE_LOCAL_WQ_ACCESS_ERROR:
160− hns_roce_local_wq_access_err_handle(hr_dev, aeqe, qpn);
161− break;
162− default:
163− break;
164− }
165−
166− hns_roce_qp_event(hr_dev, qpn, event_type);
167−}
168−
169−static void hns_roce_cq_err_handle(struct hns_roce_dev *hr_dev,
170− struct hns_roce_aeqe *aeqe,
171− int event_type)
172−{
173− struct device *dev = &hr_dev->pdev->dev;
174− u32 cqn;
175−
176− cqn = le32_to_cpu(roce_get_field(aeqe->event.cq_event.cq,
177− HNS_ROCE_AEQE_EVENT_CQ_EVENT_CQ_CQN_M,
178− HNS_ROCE_AEQE_EVENT_CQ_EVENT_CQ_CQN_S));
179−
180− switch (event_type) {
181− case HNS_ROCE_EVENT_TYPE_CQ_ACCESS_ERROR:
182− dev_warn(dev, "CQ 0x%x access err.\n", cqn);
183− break;
184− case HNS_ROCE_EVENT_TYPE_CQ_OVERFLOW:
185− dev_warn(dev, "CQ 0x%x overflow\n", cqn);
186− break;
187− case HNS_ROCE_EVENT_TYPE_CQ_ID_INVALID:
188− dev_warn(dev, "CQ 0x%x ID invalid.\n", cqn);
189− break;
190− default:
191− break;
192− }
193−
194− hns_roce_cq_event(hr_dev, cqn, event_type);
195−}
196−
197−static void hns_roce_db_overflow_handle(struct hns_roce_dev *hr_dev,
198− struct hns_roce_aeqe *aeqe)
199−{
200− struct device *dev = &hr_dev->pdev->dev;
201−
202− switch (roce_get_field(aeqe->asyn, HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_M,
203− HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_S)) {
204− case HNS_ROCE_DB_SUBTYPE_SDB_OVF:
205− dev_warn(dev, "SDB overflow.\n");
206− break;
207− case HNS_ROCE_DB_SUBTYPE_SDB_ALM_OVF:
208− dev_warn(dev, "SDB almost overflow.\n");
209− break;
210− case HNS_ROCE_DB_SUBTYPE_SDB_ALM_EMP:
211− dev_warn(dev, "SDB almost empty.\n");
212− break;
213− case HNS_ROCE_DB_SUBTYPE_ODB_OVF:
214− dev_warn(dev, "ODB overflow.\n");
215− break;
216− case HNS_ROCE_DB_SUBTYPE_ODB_ALM_OVF:
217− dev_warn(dev, "ODB almost overflow.\n");
218− break;
219− case HNS_ROCE_DB_SUBTYPE_ODB_ALM_EMP:
220− dev_warn(dev, "SDB almost empty.\n");
221− break;
222− default:
223− break;
224− }
225−}
226−
227−static int hns_roce_aeq_int(struct hns_roce_dev *hr_dev, struct hns_roce_eq *eq)
228−{
229− struct device *dev = &hr_dev->pdev->dev;
230− struct hns_roce_aeqe *aeqe;
231− int aeqes_found = 0;
232− int event_type;
233−
234− while ((aeqe = next_aeqe_sw(eq))) {
235− dev_dbg(dev, "aeqe = %p, aeqe->asyn.event_type = 0x%lx\n", aeqe,
236− roce_get_field(aeqe->asyn,
237− HNS_ROCE_AEQE_U32_4_EVENT_TYPE_M,
238− HNS_ROCE_AEQE_U32_4_EVENT_TYPE_S));
239− /* Memory barrier */
240− rmb();
241−
242− event_type = roce_get_field(aeqe->asyn,
243− HNS_ROCE_AEQE_U32_4_EVENT_TYPE_M,
244− HNS_ROCE_AEQE_U32_4_EVENT_TYPE_S);
245− switch (event_type) {
246− case HNS_ROCE_EVENT_TYPE_PATH_MIG:
247− dev_warn(dev, "PATH MIG not supported\n");
248− break;
249− case HNS_ROCE_EVENT_TYPE_COMM_EST:
250− dev_warn(dev, "COMMUNICATION established\n");
251− break;
252− case HNS_ROCE_EVENT_TYPE_SQ_DRAINED:
253− dev_warn(dev, "SQ DRAINED not supported\n");
254− break;
255− case HNS_ROCE_EVENT_TYPE_PATH_MIG_FAILED:
256− dev_warn(dev, "PATH MIG failed\n");
257− break;
258− case HNS_ROCE_EVENT_TYPE_INV_REQ_LOCAL_WQ_ERROR:
259− case HNS_ROCE_EVENT_TYPE_WQ_CATAS_ERROR:
260− case HNS_ROCE_EVENT_TYPE_LOCAL_WQ_ACCESS_ERROR:
261− hns_roce_qp_err_handle(hr_dev, aeqe, event_type);
262− break;
263− case HNS_ROCE_EVENT_TYPE_SRQ_LIMIT_REACH:
264− case HNS_ROCE_EVENT_TYPE_SRQ_CATAS_ERROR:
265− case HNS_ROCE_EVENT_TYPE_SRQ_LAST_WQE_REACH:
266− dev_warn(dev, "SRQ not support!\n");
267− break;
268− case HNS_ROCE_EVENT_TYPE_CQ_ACCESS_ERROR:
269− case HNS_ROCE_EVENT_TYPE_CQ_OVERFLOW:
270− case HNS_ROCE_EVENT_TYPE_CQ_ID_INVALID:
271− hns_roce_cq_err_handle(hr_dev, aeqe, event_type);
272− break;
273− case HNS_ROCE_EVENT_TYPE_PORT_CHANGE:
274− dev_warn(dev, "port change.\n");
275− break;
276− case HNS_ROCE_EVENT_TYPE_MB:
277− hns_roce_cmd_event(hr_dev,
278− le16_to_cpu(aeqe->event.cmd.token),
279− aeqe->event.cmd.status,
280− le64_to_cpu(aeqe->event.cmd.out_param
281− ));
282− break;
283− case HNS_ROCE_EVENT_TYPE_DB_OVERFLOW:
284− hns_roce_db_overflow_handle(hr_dev, aeqe);
285− break;
286− case HNS_ROCE_EVENT_TYPE_CEQ_OVERFLOW:
287− dev_warn(dev, "CEQ 0x%lx overflow.\n",
288− roce_get_field(aeqe->event.ce_event.ceqe,
289− HNS_ROCE_AEQE_EVENT_CE_EVENT_CEQE_CEQN_M,
290− HNS_ROCE_AEQE_EVENT_CE_EVENT_CEQE_CEQN_S));
291− break;
292− default:
293− dev_warn(dev, "Unhandled event %d on EQ %d at index %u\n",
294− event_type, eq->eqn, eq->cons_index);
295− break;
296− }
297−
298− eq->cons_index++;
299− aeqes_found = 1;
300−
301− if (eq->cons_index > 2 * hr_dev->caps.aeqe_depth - 1) {
302− dev_warn(dev, "cons_index overflow, set back to zero\n"
303− );
304− eq->cons_index = 0;
305− }
306− }
307−
308− eq_set_cons_index(eq, 0);
309−
310− return aeqes_found;
311−}
312−
313−static struct hns_roce_ceqe *get_ceqe(struct hns_roce_eq *eq, u32 entry)
314−{
315− unsigned long off = (entry & (eq->entries - 1)) *
316− HNS_ROCE_CEQ_ENTRY_SIZE;
317−
318− return (struct hns_roce_ceqe *)((u8 *)
319− (eq->buf_list[off / HNS_ROCE_BA_SIZE].buf) +
320− off % HNS_ROCE_BA_SIZE);
321−}
322−
323−static struct hns_roce_ceqe *next_ceqe_sw(struct hns_roce_eq *eq)
324−{
325− struct hns_roce_ceqe *ceqe = get_ceqe(eq, eq->cons_index);
326−
327− return (!!(roce_get_bit(ceqe->ceqe.comp,
328− HNS_ROCE_CEQE_CEQE_COMP_OWNER_S))) ^
329− (!!(eq->cons_index & eq->entries)) ? ceqe : NULL;
330−}
331−
332−static int hns_roce_ceq_int(struct hns_roce_dev *hr_dev, struct hns_roce_eq *eq)
333−{
334− struct hns_roce_ceqe *ceqe;
335− int ceqes_found = 0;
336− u32 cqn;
337−
338− while ((ceqe = next_ceqe_sw(eq))) {
339− /* Memory barrier */
340− rmb();
341− cqn = roce_get_field(ceqe->ceqe.comp,
342− HNS_ROCE_CEQE_CEQE_COMP_CQN_M,
343− HNS_ROCE_CEQE_CEQE_COMP_CQN_S);
344− hns_roce_cq_completion(hr_dev, cqn);
345−
346− ++eq->cons_index;
347− ceqes_found = 1;
348−
349− if (eq->cons_index > 2 * hr_dev->caps.ceqe_depth[eq->eqn] - 1) {
350− dev_warn(&eq->hr_dev->pdev->dev,
351− "cons_index overflow, set back to zero\n");
352− eq->cons_index = 0;
353− }
354− }
355−
356− eq_set_cons_index(eq, 0);
357−
358− return ceqes_found;
359−}
360−
361−static int hns_roce_aeq_ovf_int(struct hns_roce_dev *hr_dev,
362− struct hns_roce_eq *eq)
363−{
364− struct device *dev = &eq->hr_dev->pdev->dev;
365− int eqovf_found = 0;
366− u32 caepaemask_val;
367− u32 cealmovf_val;
368− u32 caepaest_val;
369− u32 aeshift_val;
370− u32 ceshift_val;
371− u32 cemask_val;
372− int i = 0;
373−
374− /**
375− * AEQ overflow ECC mult bit err CEQ overflow alarm
376− * must clear interrupt, mask irq, clear irq, cancel mask operation
377− */
378− aeshift_val = roce_read(hr_dev, ROCEE_CAEP_AEQC_AEQE_SHIFT_REG);
379−
380− if (roce_get_bit(aeshift_val,
381− ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQ_ALM_OVF_INT_ST_S) == 1) {
382− dev_warn(dev, "AEQ overflow!\n");
383−
384− /* Set mask */
385− caepaemask_val = roce_read(hr_dev, ROCEE_CAEP_AE_MASK_REG);
386− roce_set_bit(caepaemask_val,
387− ROCEE_CAEP_AE_MASK_CAEP_AEQ_ALM_OVF_MASK_S,
388− HNS_ROCE_INT_MASK_ENABLE);
389− roce_write(hr_dev, ROCEE_CAEP_AE_MASK_REG, caepaemask_val);
390−
391− /* Clear int state(INT_WC : write 1 clear) */
392− caepaest_val = roce_read(hr_dev, ROCEE_CAEP_AE_ST_REG);
393− roce_set_bit(caepaest_val,
394− ROCEE_CAEP_AE_ST_CAEP_AEQ_ALM_OVF_S, 1);
395− roce_write(hr_dev, ROCEE_CAEP_AE_ST_REG, caepaest_val);
396−
397− /* Clear mask */
398− caepaemask_val = roce_read(hr_dev, ROCEE_CAEP_AE_MASK_REG);
399− roce_set_bit(caepaemask_val,
400− ROCEE_CAEP_AE_MASK_CAEP_AEQ_ALM_OVF_MASK_S,
401− HNS_ROCE_INT_MASK_DISABLE);
402− roce_write(hr_dev, ROCEE_CAEP_AE_MASK_REG, caepaemask_val);
403− }
404−
405− /* CEQ almost overflow */
406− for (i = 0; i < hr_dev->caps.num_comp_vectors; i++) {
407− ceshift_val = roce_read(hr_dev, ROCEE_CAEP_CEQC_SHIFT_0_REG +
408− i * CEQ_REG_OFFSET);
409−
410− if (roce_get_bit(ceshift_val,
411− ROCEE_CAEP_CEQC_SHIFT_CAEP_CEQ_ALM_OVF_INT_ST_S) == 1) {
412− dev_warn(dev, "CEQ[%d] almost overflow!\n", i);
413− eqovf_found++;
414−
415− /* Set mask */
416− cemask_val = roce_read(hr_dev,
417− ROCEE_CAEP_CE_IRQ_MASK_0_REG +
418− i * CEQ_REG_OFFSET);
419− roce_set_bit(cemask_val,
420− ROCEE_CAEP_CE_IRQ_MASK_CAEP_CEQ_ALM_OVF_MASK_S,
421− HNS_ROCE_INT_MASK_ENABLE);
422− roce_write(hr_dev, ROCEE_CAEP_CE_IRQ_MASK_0_REG +
423− i * CEQ_REG_OFFSET, cemask_val);
424−
425− /* Clear int state(INT_WC : write 1 clear) */
426− cealmovf_val = roce_read(hr_dev,
427− ROCEE_CAEP_CEQ_ALM_OVF_0_REG +
428− i * CEQ_REG_OFFSET);
429− roce_set_bit(cealmovf_val,
430− ROCEE_CAEP_CEQ_ALM_OVF_CAEP_CEQ_ALM_OVF_S,
431− 1);
432− roce_write(hr_dev, ROCEE_CAEP_CEQ_ALM_OVF_0_REG +
433− i * CEQ_REG_OFFSET, cealmovf_val);
434−
435− /* Clear mask */
436− cemask_val = roce_read(hr_dev,
437− ROCEE_CAEP_CE_IRQ_MASK_0_REG +
438− i * CEQ_REG_OFFSET);
439− roce_set_bit(cemask_val,
440− ROCEE_CAEP_CE_IRQ_MASK_CAEP_CEQ_ALM_OVF_MASK_S,
441− HNS_ROCE_INT_MASK_DISABLE);
442− roce_write(hr_dev, ROCEE_CAEP_CE_IRQ_MASK_0_REG +
443− i * CEQ_REG_OFFSET, cemask_val);
444− }
445− }
446−
447− /* ECC multi-bit error alarm */
448− dev_warn(dev, "ECC UCERR ALARM: 0x%x, 0x%x, 0x%x\n",
449− roce_read(hr_dev, ROCEE_ECC_UCERR_ALM0_REG),
450− roce_read(hr_dev, ROCEE_ECC_UCERR_ALM1_REG),
451− roce_read(hr_dev, ROCEE_ECC_UCERR_ALM2_REG));
452−
453− dev_warn(dev, "ECC CERR ALARM: 0x%x, 0x%x, 0x%x\n",
454− roce_read(hr_dev, ROCEE_ECC_CERR_ALM0_REG),
455− roce_read(hr_dev, ROCEE_ECC_CERR_ALM1_REG),
456− roce_read(hr_dev, ROCEE_ECC_CERR_ALM2_REG));
457−
458− return eqovf_found;
459−}
460−
461−static int hns_roce_eq_int(struct hns_roce_dev *hr_dev, struct hns_roce_eq *eq)
462−{
463− int eqes_found = 0;
464−
465− if (likely(eq->type_flag == HNS_ROCE_CEQ))
466− /* CEQ irq routine, CEQ is pulse irq, not clear */
467− eqes_found = hns_roce_ceq_int(hr_dev, eq);
468− else if (likely(eq->type_flag == HNS_ROCE_AEQ))
469− /* AEQ irq routine, AEQ is pulse irq, not clear */
470− eqes_found = hns_roce_aeq_int(hr_dev, eq);
471− else
472− /* AEQ queue overflow irq */
473− eqes_found = hns_roce_aeq_ovf_int(hr_dev, eq);
474−
475− return eqes_found;
476−}
477−
478−static irqreturn_t hns_roce_msi_x_interrupt(int irq, void *eq_ptr)
479−{
480− int int_work = 0;
481− struct hns_roce_eq *eq = eq_ptr;
482− struct hns_roce_dev *hr_dev = eq->hr_dev;
483−
484− int_work = hns_roce_eq_int(hr_dev, eq);
485−
486− return IRQ_RETVAL(int_work);
487−}
488−
489−static void hns_roce_enable_eq(struct hns_roce_dev *hr_dev, int eq_num,
490− int enable_flag)
491−{
492− void __iomem *eqc = hr_dev->eq_table.eqc_base[eq_num];
493− u32 val;
494−
495− val = readl(eqc);
496−
497− if (enable_flag)
498− roce_set_field(val,
499− ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_M,
500− ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_S,
501− HNS_ROCE_EQ_STAT_VALID);
502− else
503− roce_set_field(val,
504− ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_M,
505− ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_S,
506− HNS_ROCE_EQ_STAT_INVALID);
507− writel(val, eqc);
508−}
509−
510−static int hns_roce_create_eq(struct hns_roce_dev *hr_dev,
511− struct hns_roce_eq *eq)
512−{
513− void __iomem *eqc = hr_dev->eq_table.eqc_base[eq->eqn];
514− struct device *dev = &hr_dev->pdev->dev;
515− dma_addr_t tmp_dma_addr;
516− u32 eqconsindx_val = 0;
517− u32 eqcuridx_val = 0;
518− u32 eqshift_val = 0;
519− int num_bas = 0;
520− int ret;
521− int i;
522−
523− num_bas = (PAGE_ALIGN(eq->entries * eq->eqe_size) +
524− HNS_ROCE_BA_SIZE - 1) / HNS_ROCE_BA_SIZE;
525−
526− if ((eq->entries * eq->eqe_size) > HNS_ROCE_BA_SIZE) {
527− dev_err(dev, "[error]eq buf %d gt ba size(%d) need bas=%d\n",
528− (eq->entries * eq->eqe_size), HNS_ROCE_BA_SIZE,
529− num_bas);
530− return -EINVAL;
531− }
532−
533− eq->buf_list = kcalloc(num_bas, sizeof(*eq->buf_list), GFP_KERNEL);
534− if (!eq->buf_list)
535− return -ENOMEM;
536−
537− for (i = 0; i < num_bas; ++i) {
538− eq->buf_list[i].buf = dma_alloc_coherent(dev, HNS_ROCE_BA_SIZE,
539− &tmp_dma_addr,
540− GFP_KERNEL);
541− if (!eq->buf_list[i].buf) {
542− ret = -ENOMEM;
543− goto err_out_free_pages;
544− }
545−
546− eq->buf_list[i].map = tmp_dma_addr;
547− memset(eq->buf_list[i].buf, 0, HNS_ROCE_BA_SIZE);
548− }
549− eq->cons_index = 0;
550− roce_set_field(eqshift_val,
551− ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_M,
552− ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_S,
553− HNS_ROCE_EQ_STAT_INVALID);
554− roce_set_field(eqshift_val,
555− ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_AEQE_SHIFT_M,
556− ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_AEQE_SHIFT_S,
557− eq->log_entries);
558− writel(eqshift_val, eqc);
559−
560− /* Configure eq extended address 12~44bit */
561− writel((u32)(eq->buf_list[0].map >> 12), eqc + 4);
562−
563− /*
564− * Configure eq extended address 45~49 bit.
565− * 44 = 32 + 12, When evaluating addr to hardware, shift 12 because of
566− * using 4K page, and shift more 32 because of
567− * caculating the high 32 bit value evaluated to hardware.
568− */
569− roce_set_field(eqcuridx_val, ROCEE_CAEP_AEQE_CUR_IDX_CAEP_AEQ_BT_H_M,
570− ROCEE_CAEP_AEQE_CUR_IDX_CAEP_AEQ_BT_H_S,
571− eq->buf_list[0].map >> 44);
572− roce_set_field(eqcuridx_val,
573− ROCEE_CAEP_AEQE_CUR_IDX_CAEP_AEQE_CUR_IDX_M,
574− ROCEE_CAEP_AEQE_CUR_IDX_CAEP_AEQE_CUR_IDX_S, 0);
575− writel(eqcuridx_val, eqc + 8);
576−
577− /* Configure eq consumer index */
578− roce_set_field(eqconsindx_val,
579− ROCEE_CAEP_AEQE_CONS_IDX_CAEP_AEQE_CONS_IDX_M,
580− ROCEE_CAEP_AEQE_CONS_IDX_CAEP_AEQE_CONS_IDX_S, 0);
581− writel(eqconsindx_val, eqc + 0xc);
582−
583− return 0;
584−
585−err_out_free_pages:
586− for (i = i - 1; i >= 0; i--)
587− dma_free_coherent(dev, HNS_ROCE_BA_SIZE, eq->buf_list[i].buf,
588− eq->buf_list[i].map);
589−
590− kfree(eq->buf_list);
591− return ret;
592−}
593−
594−static void hns_roce_free_eq(struct hns_roce_dev *hr_dev,
595− struct hns_roce_eq *eq)
596−{
597− int i = 0;
598− int npages = (PAGE_ALIGN(eq->eqe_size * eq->entries) +
599− HNS_ROCE_BA_SIZE - 1) / HNS_ROCE_BA_SIZE;
600−
601− if (!eq->buf_list)
602− return;
603−
604− for (i = 0; i < npages; ++i)
605− dma_free_coherent(&hr_dev->pdev->dev, HNS_ROCE_BA_SIZE,
606− eq->buf_list[i].buf, eq->buf_list[i].map);
607−
608− kfree(eq->buf_list);
609−}
610−
611−static void hns_roce_int_mask_en(struct hns_roce_dev *hr_dev)
612−{
613− int i = 0;
614− u32 aemask_val;
615− int masken = 0;
616−
617− /* AEQ INT */
618− aemask_val = roce_read(hr_dev, ROCEE_CAEP_AE_MASK_REG);
619− roce_set_bit(aemask_val, ROCEE_CAEP_AE_MASK_CAEP_AEQ_ALM_OVF_MASK_S,
620− masken);
621− roce_set_bit(aemask_val, ROCEE_CAEP_AE_MASK_CAEP_AE_IRQ_MASK_S, masken);
622− roce_write(hr_dev, ROCEE_CAEP_AE_MASK_REG, aemask_val);
623−
624− /* CEQ INT */
625− for (i = 0; i < hr_dev->caps.num_comp_vectors; i++) {
626− /* IRQ mask */
627− roce_write(hr_dev, ROCEE_CAEP_CE_IRQ_MASK_0_REG +
628− i * CEQ_REG_OFFSET, masken);
629− }
630−}
631−
632−static void hns_roce_ce_int_default_cfg(struct hns_roce_dev *hr_dev)
633−{
634− /* Configure ce int interval */
635− roce_write(hr_dev, ROCEE_CAEP_CE_INTERVAL_CFG_REG,
636− HNS_ROCE_CEQ_DEFAULT_INTERVAL);
637−
638− /* Configure ce int burst num */
639− roce_write(hr_dev, ROCEE_CAEP_CE_BURST_NUM_CFG_REG,
640− HNS_ROCE_CEQ_DEFAULT_BURST_NUM);
641−}
642−
643−int hns_roce_init_eq_table(struct hns_roce_dev *hr_dev)
644−{
645− struct hns_roce_eq_table *eq_table = &hr_dev->eq_table;
646− struct device *dev = &hr_dev->pdev->dev;
647− struct hns_roce_eq *eq = NULL;
648− int eq_num = 0;
649− int ret = 0;
650− int i = 0;
651− int j = 0;
652−
653− eq_num = hr_dev->caps.num_comp_vectors + hr_dev->caps.num_aeq_vectors;
654− eq_table->eq = kcalloc(eq_num, sizeof(*eq_table->eq), GFP_KERNEL);
655− if (!eq_table->eq)
656− return -ENOMEM;
657−
658− eq_table->eqc_base = kcalloc(eq_num, sizeof(*eq_table->eqc_base),
659− GFP_KERNEL);
660− if (!eq_table->eqc_base) {
661− ret = -ENOMEM;
662− goto err_eqc_base_alloc_fail;
663− }
664−
665− for (i = 0; i < eq_num; i++) {
666− eq = &eq_table->eq[i];
667− eq->hr_dev = hr_dev;
668− eq->eqn = i;
669− eq->irq = hr_dev->irq[i];
670− eq->log_page_size = PAGE_SHIFT;
671−
672− if (i < hr_dev->caps.num_comp_vectors) {
673− /* CEQ */
674− eq_table->eqc_base[i] = hr_dev->reg_base +
675− ROCEE_CAEP_CEQC_SHIFT_0_REG +
676− HNS_ROCE_CEQC_REG_OFFSET * i;
677− eq->type_flag = HNS_ROCE_CEQ;
678− eq->doorbell = hr_dev->reg_base +
679− ROCEE_CAEP_CEQC_CONS_IDX_0_REG +
680− HNS_ROCE_CEQC_REG_OFFSET * i;
681− eq->entries = hr_dev->caps.ceqe_depth[i];
682− eq->log_entries = ilog2(eq->entries);
683− eq->eqe_size = sizeof(struct hns_roce_ceqe);
684− } else {
685− /* AEQ */
686− eq_table->eqc_base[i] = hr_dev->reg_base +
687− ROCEE_CAEP_AEQC_AEQE_SHIFT_REG;
688− eq->type_flag = HNS_ROCE_AEQ;
689− eq->doorbell = hr_dev->reg_base +
690− ROCEE_CAEP_AEQE_CONS_IDX_REG;
691− eq->entries = hr_dev->caps.aeqe_depth;
692− eq->log_entries = ilog2(eq->entries);
693− eq->eqe_size = sizeof(struct hns_roce_aeqe);
694− }
695− }
696−
697− /* Disable irq */
698− hns_roce_int_mask_en(hr_dev);
699−
700− /* Configure CE irq interval and burst num */
701− hns_roce_ce_int_default_cfg(hr_dev);
702−
703− for (i = 0; i < eq_num; i++) {
704− ret = hns_roce_create_eq(hr_dev, &eq_table->eq[i]);
705− if (ret) {
706− dev_err(dev, "eq create failed\n");
707− goto err_create_eq_fail;
708− }
709− }
710−
711− for (j = 0; j < eq_num; j++) {
712− ret = request_irq(eq_table->eq[j].irq, hns_roce_msi_x_interrupt,
713− 0, hr_dev->irq_names[j], eq_table->eq + j);
714− if (ret) {
715− dev_err(dev, "request irq error!\n");
716− goto err_request_irq_fail;
717− }
718− }
719−
720− for (i = 0; i < eq_num; i++)
721− hns_roce_enable_eq(hr_dev, i, EQ_ENABLE);
722−
723− return 0;
724−
725−err_request_irq_fail:
726− for (j = j - 1; j >= 0; j--)
727− free_irq(eq_table->eq[j].irq, eq_table->eq + j);
728−
729−err_create_eq_fail:
730− for (i = i - 1; i >= 0; i--)
731− hns_roce_free_eq(hr_dev, &eq_table->eq[i]);
732−
733− kfree(eq_table->eqc_base);
734−
735−err_eqc_base_alloc_fail:
736− kfree(eq_table->eq);
737−
738− return ret;
739−}
740−
741−void hns_roce_cleanup_eq_table(struct hns_roce_dev *hr_dev)
742−{
743− int i;
744− int eq_num;
745− struct hns_roce_eq_table *eq_table = &hr_dev->eq_table;
746−
747− eq_num = hr_dev->caps.num_comp_vectors + hr_dev->caps.num_aeq_vectors;
748− for (i = 0; i < eq_num; i++) {
749− /* Disable EQ */
750− hns_roce_enable_eq(hr_dev, i, EQ_DISABLE);
751−
752− free_irq(eq_table->eq[i].irq, eq_table->eq + i);
753−
754− hns_roce_free_eq(hr_dev, &eq_table->eq[i]);
755− }
756−
757− kfree(eq_table->eqc_base);
758− kfree(eq_table->eq);
759−}
drivers/infiniband/hw/hns/hns_roce_eq.h+0−134
@@ -1,134 +0,0 @@
1−/*
2− * Copyright (c) 2016 Hisilicon Limited.
3− *
4− * This software is available to you under a choice of one of two
5− * licenses. You may choose to be licensed under the terms of the GNU
6− * General Public License (GPL) Version 2, available from the file
7− * COPYING in the main directory of this source tree, or the
8− * OpenIB.org BSD license below:
9− *
10− * Redistribution and use in source and binary forms, with or
11− * without modification, are permitted provided that the following
12− * conditions are met:
13− *
14− * - Redistributions of source code must retain the above
15− * copyright notice, this list of conditions and the following
16− * disclaimer.
17− *
18− * - Redistributions in binary form must reproduce the above
19− * copyright notice, this list of conditions and the following
20− * disclaimer in the documentation and/or other materials
21− * provided with the distribution.
22− *
23− * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
24− * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
25− * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
26− * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
27− * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
28− * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
29− * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
30− * SOFTWARE.
31− */
32−
33−#ifndef _HNS_ROCE_EQ_H
34−#define _HNS_ROCE_EQ_H
35−
36−#define HNS_ROCE_CEQ 1
37−#define HNS_ROCE_AEQ 2
38−
39−#define HNS_ROCE_CEQ_ENTRY_SIZE 0x4
40−#define HNS_ROCE_AEQ_ENTRY_SIZE 0x10
41−#define HNS_ROCE_CEQC_REG_OFFSET 0x18
42−
43−#define HNS_ROCE_CEQ_DEFAULT_INTERVAL 0x10
44−#define HNS_ROCE_CEQ_DEFAULT_BURST_NUM 0x10
45−
46−#define HNS_ROCE_INT_MASK_DISABLE 0
47−#define HNS_ROCE_INT_MASK_ENABLE 1
48−
49−#define EQ_ENABLE 1
50−#define EQ_DISABLE 0
51−#define CONS_INDEX_MASK 0xffff
52−
53−#define CEQ_REG_OFFSET 0x18
54−
55−enum {
56− HNS_ROCE_EQ_STAT_INVALID = 0,
57− HNS_ROCE_EQ_STAT_VALID = 2,
58−};
59−
60−struct hns_roce_aeqe {
61− u32 asyn;
62− union {
63− struct {
64− u32 qp;
65− u32 rsv0;
66− u32 rsv1;
67− } qp_event;
68−
69− struct {
70− u32 cq;
71− u32 rsv0;
72− u32 rsv1;
73− } cq_event;
74−
75− struct {
76− u32 port;
77− u32 rsv0;
78− u32 rsv1;
79− } port_event;
80−
81− struct {
82− u32 ceqe;
83− u32 rsv0;
84− u32 rsv1;
85− } ce_event;
86−
87− struct {
88− __le64 out_param;
89− __le16 token;
90− u8 status;
91− u8 rsv0;
92− } __packed cmd;
93− } event;
94−};
95−
96−#define HNS_ROCE_AEQE_U32_4_EVENT_TYPE_S 16
97−#define HNS_ROCE_AEQE_U32_4_EVENT_TYPE_M \
98− (((1UL << 8) - 1) << HNS_ROCE_AEQE_U32_4_EVENT_TYPE_S)
99−
100−#define HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_S 24
101−#define HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_M \
102− (((1UL << 7) - 1) << HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_S)
103−
104−#define HNS_ROCE_AEQE_U32_4_OWNER_S 31
105−
106−#define HNS_ROCE_AEQE_EVENT_QP_EVENT_QP_QPN_S 0
107−#define HNS_ROCE_AEQE_EVENT_QP_EVENT_QP_QPN_M \
108− (((1UL << 24) - 1) << HNS_ROCE_AEQE_EVENT_QP_EVENT_QP_QPN_S)
109−
110−#define HNS_ROCE_AEQE_EVENT_QP_EVENT_PORT_NUM_S 25
111−#define HNS_ROCE_AEQE_EVENT_QP_EVENT_PORT_NUM_M \
112− (((1UL << 3) - 1) << HNS_ROCE_AEQE_EVENT_QP_EVENT_PORT_NUM_S)
113−
114−#define HNS_ROCE_AEQE_EVENT_CQ_EVENT_CQ_CQN_S 0
115−#define HNS_ROCE_AEQE_EVENT_CQ_EVENT_CQ_CQN_M \
116− (((1UL << 16) - 1) << HNS_ROCE_AEQE_EVENT_CQ_EVENT_CQ_CQN_S)
117−
118−#define HNS_ROCE_AEQE_EVENT_CE_EVENT_CEQE_CEQN_S 0
119−#define HNS_ROCE_AEQE_EVENT_CE_EVENT_CEQE_CEQN_M \
120− (((1UL << 5) - 1) << HNS_ROCE_AEQE_EVENT_CE_EVENT_CEQE_CEQN_S)
121−
122−struct hns_roce_ceqe {
123− union {
124− int comp;
125− } ceqe;
126−};
127−
128−#define HNS_ROCE_CEQE_CEQE_COMP_OWNER_S 0
129−
130−#define HNS_ROCE_CEQE_CEQE_COMP_CQN_S 16
131−#define HNS_ROCE_CEQE_CEQE_COMP_CQN_M \
132− (((1UL << 16) - 1) << HNS_ROCE_CEQE_CEQE_COMP_CQN_S)
133−
134−#endif /* _HNS_ROCE_EQ_H */
drivers/infiniband/hw/hns/hns_roce_hw_v1.c+741−17
@@ -33,6 +33,7 @@ · 32 unmodified lines
3333 #include <linux/platform_device.h>
3434 #include <linux/acpi.h>
3535 #include <linux/etherdevice.h>
36+#include <linux/interrupt.h>
3637 #include <linux/of.h>
3738 #include <linux/of_platform.h>
3839 #include <rdma/ib_umem.h>
@@ -774,7 +775,7 @@ static int hns_roce_v1_rsv_lp_qp(struct hns_roce_dev *hr_dev) · 735 unmodified lines
774775 goto create_lp_qp_failed;
775776 }
776777
777− ret = hr_dev->hw->modify_qp(&hr_qp->ibqp, &attr, attr_mask,
778+ ret = hr_dev->hw->modify_qp(&hr_qp->ibqp, &attr, IB_QP_DEST_QPN,
778779 IB_QPS_INIT, IB_QPS_RTR);
779780 if (ret) {
780781 dev_err(dev, "modify qp failed(%d)!\n", ret);
@@ -1492,9 +1493,9 @@ static int hns_roce_v1_profile(struct hns_roce_dev *hr_dev) · 711 unmodified lines
14921493 caps->max_sq_inline = HNS_ROCE_V1_INLINE_SIZE;
14931494 caps->num_uars = HNS_ROCE_V1_UAR_NUM;
14941495 caps->phy_num_uars = HNS_ROCE_V1_PHY_UAR_NUM;
1495− caps->num_aeq_vectors = HNS_ROCE_AEQE_VEC_NUM;
1496− caps->num_comp_vectors = HNS_ROCE_COMP_VEC_NUM;
1497− caps->num_other_vectors = HNS_ROCE_AEQE_OF_VEC_NUM;
1496+ caps->num_aeq_vectors = HNS_ROCE_V1_AEQE_VEC_NUM;
1497+ caps->num_comp_vectors = HNS_ROCE_V1_COMP_VEC_NUM;
1498+ caps->num_other_vectors = HNS_ROCE_V1_ABNORMAL_VEC_NUM;
14981499 caps->num_mtpts = HNS_ROCE_V1_MAX_MTPT_NUM;
14991500 caps->num_mtt_segs = HNS_ROCE_V1_MAX_MTT_SEGS;
15001501 caps->num_pds = HNS_ROCE_V1_MAX_PD_NUM;
@@ -1529,10 +1530,8 @@ static int hns_roce_v1_profile(struct hns_roce_dev *hr_dev) · 28 unmodified lines
15291530 caps->num_ports + 1;
15301531 }
15311532
1532− for (i = 0; i < caps->num_comp_vectors; i++)
1533− caps->ceqe_depth[i] = HNS_ROCE_V1_NUM_COMP_EQE;
1534−
1535− caps->aeqe_depth = HNS_ROCE_V1_NUM_ASYNC_EQE;
1533+ caps->ceqe_depth = HNS_ROCE_V1_COMP_EQE_NUM;
1534+ caps->aeqe_depth = HNS_ROCE_V1_ASYNC_EQE_NUM;
15361535 caps->local_ca_ack_delay = le32_to_cpu(roce_read(hr_dev,
15371536 ROCEE_ACK_DELAY_REG));
15381537 caps->max_mtu = IB_MTU_2048;
@@ -2312,15 +2311,16 @@ static int hns_roce_v1_poll_one(struct hns_roce_cq *hr_cq, · 773 unmodified lines
23122311 case HNS_ROCE_OPCODE_RDMA_WITH_IMM_RECEIVE:
23132312 wc->opcode = IB_WC_RECV_RDMA_WITH_IMM;
23142313 wc->wc_flags = IB_WC_WITH_IMM;
2315− wc->ex.imm_data = le32_to_cpu(cqe->immediate_data);
2314+ wc->ex.imm_data =
2315+ cpu_to_be32(le32_to_cpu(cqe->immediate_data));
23162316 break;
23172317 case HNS_ROCE_OPCODE_SEND_DATA_RECEIVE:
23182318 if (roce_get_bit(cqe->cqe_byte_4,
23192319 CQE_BYTE_4_IMM_INDICATOR_S)) {
23202320 wc->opcode = IB_WC_RECV;
23212321 wc->wc_flags = IB_WC_WITH_IMM;
2322− wc->ex.imm_data = le32_to_cpu(
2323− cqe->immediate_data);
2322+ wc->ex.imm_data = cpu_to_be32(
2323+ le32_to_cpu(cqe->immediate_data));
23242324 } else {
23252325 wc->opcode = IB_WC_RECV;
23262326 wc->wc_flags = 0;
@@ -3960,6 +3960,732 @@ static int hns_roce_v1_destroy_cq(struct ib_cq *ibcq) · 1633 unmodified lines
39603960 return ret;
39613961 }
39623962
3963+static void set_eq_cons_index_v1(struct hns_roce_eq *eq, int req_not)
3964+{
3965+ roce_raw_write((eq->cons_index & HNS_ROCE_V1_CONS_IDX_M) |
3966+ (req_not << eq->log_entries), eq->doorbell);
3967+}
3968+
3969+static void hns_roce_v1_wq_catas_err_handle(struct hns_roce_dev *hr_dev,
3970+ struct hns_roce_aeqe *aeqe, int qpn)
3971+{
3972+ struct device *dev = &hr_dev->pdev->dev;
3973+
3974+ dev_warn(dev, "Local Work Queue Catastrophic Error.\n");
3975+ switch (roce_get_field(aeqe->asyn, HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_M,
3976+ HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_S)) {
3977+ case HNS_ROCE_LWQCE_QPC_ERROR:
3978+ dev_warn(dev, "QP %d, QPC error.\n", qpn);
3979+ break;
3980+ case HNS_ROCE_LWQCE_MTU_ERROR:
3981+ dev_warn(dev, "QP %d, MTU error.\n", qpn);
3982+ break;
3983+ case HNS_ROCE_LWQCE_WQE_BA_ADDR_ERROR:
3984+ dev_warn(dev, "QP %d, WQE BA addr error.\n", qpn);
3985+ break;
3986+ case HNS_ROCE_LWQCE_WQE_ADDR_ERROR:
3987+ dev_warn(dev, "QP %d, WQE addr error.\n", qpn);
3988+ break;
3989+ case HNS_ROCE_LWQCE_SQ_WQE_SHIFT_ERROR:
3990+ dev_warn(dev, "QP %d, WQE shift error\n", qpn);
3991+ break;
3992+ case HNS_ROCE_LWQCE_SL_ERROR:
3993+ dev_warn(dev, "QP %d, SL error.\n", qpn);
3994+ break;
3995+ case HNS_ROCE_LWQCE_PORT_ERROR:
3996+ dev_warn(dev, "QP %d, port error.\n", qpn);
3997+ break;
3998+ default:
3999+ break;
4000+ }
4001+}
4002+
4003+static void hns_roce_v1_local_wq_access_err_handle(struct hns_roce_dev *hr_dev,
4004+ struct hns_roce_aeqe *aeqe,
4005+ int qpn)
4006+{
4007+ struct device *dev = &hr_dev->pdev->dev;
4008+
4009+ dev_warn(dev, "Local Access Violation Work Queue Error.\n");
4010+ switch (roce_get_field(aeqe->asyn, HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_M,
4011+ HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_S)) {
4012+ case HNS_ROCE_LAVWQE_R_KEY_VIOLATION:
4013+ dev_warn(dev, "QP %d, R_key violation.\n", qpn);
4014+ break;
4015+ case HNS_ROCE_LAVWQE_LENGTH_ERROR:
4016+ dev_warn(dev, "QP %d, length error.\n", qpn);
4017+ break;
4018+ case HNS_ROCE_LAVWQE_VA_ERROR:
4019+ dev_warn(dev, "QP %d, VA error.\n", qpn);
4020+ break;
4021+ case HNS_ROCE_LAVWQE_PD_ERROR:
4022+ dev_err(dev, "QP %d, PD error.\n", qpn);
4023+ break;
4024+ case HNS_ROCE_LAVWQE_RW_ACC_ERROR:
4025+ dev_warn(dev, "QP %d, rw acc error.\n", qpn);
4026+ break;
4027+ case HNS_ROCE_LAVWQE_KEY_STATE_ERROR:
4028+ dev_warn(dev, "QP %d, key state error.\n", qpn);
4029+ break;
4030+ case HNS_ROCE_LAVWQE_MR_OPERATION_ERROR:
4031+ dev_warn(dev, "QP %d, MR operation error.\n", qpn);
4032+ break;
4033+ default:
4034+ break;
4035+ }
4036+}
4037+
4038+static void hns_roce_v1_qp_err_handle(struct hns_roce_dev *hr_dev,
4039+ struct hns_roce_aeqe *aeqe,
4040+ int event_type)
4041+{
4042+ struct device *dev = &hr_dev->pdev->dev;
4043+ int phy_port;
4044+ int qpn;
4045+
4046+ qpn = roce_get_field(aeqe->event.qp_event.qp,
4047+ HNS_ROCE_AEQE_EVENT_QP_EVENT_QP_QPN_M,
4048+ HNS_ROCE_AEQE_EVENT_QP_EVENT_QP_QPN_S);
4049+ phy_port = roce_get_field(aeqe->event.qp_event.qp,
4050+ HNS_ROCE_AEQE_EVENT_QP_EVENT_PORT_NUM_M,
4051+ HNS_ROCE_AEQE_EVENT_QP_EVENT_PORT_NUM_S);
4052+ if (qpn <= 1)
4053+ qpn = HNS_ROCE_MAX_PORTS * qpn + phy_port;
4054+
4055+ switch (event_type) {
4056+ case HNS_ROCE_EVENT_TYPE_INV_REQ_LOCAL_WQ_ERROR:
4057+ dev_warn(dev, "Invalid Req Local Work Queue Error.\n"
4058+ "QP %d, phy_port %d.\n", qpn, phy_port);
4059+ break;
4060+ case HNS_ROCE_EVENT_TYPE_WQ_CATAS_ERROR:
4061+ hns_roce_v1_wq_catas_err_handle(hr_dev, aeqe, qpn);
4062+ break;
4063+ case HNS_ROCE_EVENT_TYPE_LOCAL_WQ_ACCESS_ERROR:
4064+ hns_roce_v1_local_wq_access_err_handle(hr_dev, aeqe, qpn);
4065+ break;
4066+ default:
4067+ break;
4068+ }
4069+
4070+ hns_roce_qp_event(hr_dev, qpn, event_type);
4071+}
4072+
4073+static void hns_roce_v1_cq_err_handle(struct hns_roce_dev *hr_dev,
4074+ struct hns_roce_aeqe *aeqe,
4075+ int event_type)
4076+{
4077+ struct device *dev = &hr_dev->pdev->dev;
4078+ u32 cqn;
4079+
4080+ cqn = le32_to_cpu(roce_get_field(aeqe->event.cq_event.cq,
4081+ HNS_ROCE_AEQE_EVENT_CQ_EVENT_CQ_CQN_M,
4082+ HNS_ROCE_AEQE_EVENT_CQ_EVENT_CQ_CQN_S));
4083+
4084+ switch (event_type) {
4085+ case HNS_ROCE_EVENT_TYPE_CQ_ACCESS_ERROR:
4086+ dev_warn(dev, "CQ 0x%x access err.\n", cqn);
4087+ break;
4088+ case HNS_ROCE_EVENT_TYPE_CQ_OVERFLOW:
4089+ dev_warn(dev, "CQ 0x%x overflow\n", cqn);
4090+ break;
4091+ case HNS_ROCE_EVENT_TYPE_CQ_ID_INVALID:
4092+ dev_warn(dev, "CQ 0x%x ID invalid.\n", cqn);
4093+ break;
4094+ default:
4095+ break;
4096+ }
4097+
4098+ hns_roce_cq_event(hr_dev, cqn, event_type);
4099+}
4100+
4101+static void hns_roce_v1_db_overflow_handle(struct hns_roce_dev *hr_dev,
4102+ struct hns_roce_aeqe *aeqe)
4103+{
4104+ struct device *dev = &hr_dev->pdev->dev;
4105+
4106+ switch (roce_get_field(aeqe->asyn, HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_M,
4107+ HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_S)) {
4108+ case HNS_ROCE_DB_SUBTYPE_SDB_OVF:
4109+ dev_warn(dev, "SDB overflow.\n");
4110+ break;
4111+ case HNS_ROCE_DB_SUBTYPE_SDB_ALM_OVF:
4112+ dev_warn(dev, "SDB almost overflow.\n");
4113+ break;
4114+ case HNS_ROCE_DB_SUBTYPE_SDB_ALM_EMP:
4115+ dev_warn(dev, "SDB almost empty.\n");
4116+ break;
4117+ case HNS_ROCE_DB_SUBTYPE_ODB_OVF:
4118+ dev_warn(dev, "ODB overflow.\n");
4119+ break;
4120+ case HNS_ROCE_DB_SUBTYPE_ODB_ALM_OVF:
4121+ dev_warn(dev, "ODB almost overflow.\n");
4122+ break;
4123+ case HNS_ROCE_DB_SUBTYPE_ODB_ALM_EMP:
4124+ dev_warn(dev, "SDB almost empty.\n");
4125+ break;
4126+ default:
4127+ break;
4128+ }
4129+}
4130+
4131+static struct hns_roce_aeqe *get_aeqe_v1(struct hns_roce_eq *eq, u32 entry)
4132+{
4133+ unsigned long off = (entry & (eq->entries - 1)) *
4134+ HNS_ROCE_AEQ_ENTRY_SIZE;
4135+
4136+ return (struct hns_roce_aeqe *)((u8 *)
4137+ (eq->buf_list[off / HNS_ROCE_BA_SIZE].buf) +
4138+ off % HNS_ROCE_BA_SIZE);
4139+}
4140+
4141+static struct hns_roce_aeqe *next_aeqe_sw_v1(struct hns_roce_eq *eq)
4142+{
4143+ struct hns_roce_aeqe *aeqe = get_aeqe_v1(eq, eq->cons_index);
4144+
4145+ return (roce_get_bit(aeqe->asyn, HNS_ROCE_AEQE_U32_4_OWNER_S) ^
4146+ !!(eq->cons_index & eq->entries)) ? aeqe : NULL;
4147+}
4148+
4149+static int hns_roce_v1_aeq_int(struct hns_roce_dev *hr_dev,
4150+ struct hns_roce_eq *eq)
4151+{
4152+ struct device *dev = &hr_dev->pdev->dev;
4153+ struct hns_roce_aeqe *aeqe;
4154+ int aeqes_found = 0;
4155+ int event_type;
4156+
4157+ while ((aeqe = next_aeqe_sw_v1(eq))) {
4158+
4159+ /* Make sure we read the AEQ entry after we have checked the
4160+ * ownership bit
4161+ */
4162+ dma_rmb();
4163+
4164+ dev_dbg(dev, "aeqe = %p, aeqe->asyn.event_type = 0x%lx\n", aeqe,
4165+ roce_get_field(aeqe->asyn,
4166+ HNS_ROCE_AEQE_U32_4_EVENT_TYPE_M,
4167+ HNS_ROCE_AEQE_U32_4_EVENT_TYPE_S));
4168+ event_type = roce_get_field(aeqe->asyn,
4169+ HNS_ROCE_AEQE_U32_4_EVENT_TYPE_M,
4170+ HNS_ROCE_AEQE_U32_4_EVENT_TYPE_S);
4171+ switch (event_type) {
4172+ case HNS_ROCE_EVENT_TYPE_PATH_MIG:
4173+ dev_warn(dev, "PATH MIG not supported\n");
4174+ break;
4175+ case HNS_ROCE_EVENT_TYPE_COMM_EST:
4176+ dev_warn(dev, "COMMUNICATION established\n");
4177+ break;
4178+ case HNS_ROCE_EVENT_TYPE_SQ_DRAINED:
4179+ dev_warn(dev, "SQ DRAINED not supported\n");
4180+ break;
4181+ case HNS_ROCE_EVENT_TYPE_PATH_MIG_FAILED:
4182+ dev_warn(dev, "PATH MIG failed\n");
4183+ break;
4184+ case HNS_ROCE_EVENT_TYPE_INV_REQ_LOCAL_WQ_ERROR:
4185+ case HNS_ROCE_EVENT_TYPE_WQ_CATAS_ERROR:
4186+ case HNS_ROCE_EVENT_TYPE_LOCAL_WQ_ACCESS_ERROR:
4187+ hns_roce_v1_qp_err_handle(hr_dev, aeqe, event_type);
4188+ break;
4189+ case HNS_ROCE_EVENT_TYPE_SRQ_LIMIT_REACH:
4190+ case HNS_ROCE_EVENT_TYPE_SRQ_CATAS_ERROR:
4191+ case HNS_ROCE_EVENT_TYPE_SRQ_LAST_WQE_REACH:
4192+ dev_warn(dev, "SRQ not support!\n");
4193+ break;
4194+ case HNS_ROCE_EVENT_TYPE_CQ_ACCESS_ERROR:
4195+ case HNS_ROCE_EVENT_TYPE_CQ_OVERFLOW:
4196+ case HNS_ROCE_EVENT_TYPE_CQ_ID_INVALID:
4197+ hns_roce_v1_cq_err_handle(hr_dev, aeqe, event_type);
4198+ break;
4199+ case HNS_ROCE_EVENT_TYPE_PORT_CHANGE:
4200+ dev_warn(dev, "port change.\n");
4201+ break;
4202+ case HNS_ROCE_EVENT_TYPE_MB:
4203+ hns_roce_cmd_event(hr_dev,
4204+ le16_to_cpu(aeqe->event.cmd.token),
4205+ aeqe->event.cmd.status,
4206+ le64_to_cpu(aeqe->event.cmd.out_param
4207+ ));
4208+ break;
4209+ case HNS_ROCE_EVENT_TYPE_DB_OVERFLOW:
4210+ hns_roce_v1_db_overflow_handle(hr_dev, aeqe);
4211+ break;
4212+ case HNS_ROCE_EVENT_TYPE_CEQ_OVERFLOW:
4213+ dev_warn(dev, "CEQ 0x%lx overflow.\n",
4214+ roce_get_field(aeqe->event.ce_event.ceqe,
4215+ HNS_ROCE_AEQE_EVENT_CE_EVENT_CEQE_CEQN_M,
4216+ HNS_ROCE_AEQE_EVENT_CE_EVENT_CEQE_CEQN_S));
4217+ break;
4218+ default:
4219+ dev_warn(dev, "Unhandled event %d on EQ %d at idx %u.\n",
4220+ event_type, eq->eqn, eq->cons_index);
4221+ break;
4222+ }
4223+
4224+ eq->cons_index++;
4225+ aeqes_found = 1;
4226+
4227+ if (eq->cons_index > 2 * hr_dev->caps.aeqe_depth - 1) {
4228+ dev_warn(dev, "cons_index overflow, set back to 0.\n");
4229+ eq->cons_index = 0;
4230+ }
4231+ }
4232+
4233+ set_eq_cons_index_v1(eq, 0);
4234+
4235+ return aeqes_found;
4236+}
4237+
4238+static struct hns_roce_ceqe *get_ceqe_v1(struct hns_roce_eq *eq, u32 entry)
4239+{
4240+ unsigned long off = (entry & (eq->entries - 1)) *
4241+ HNS_ROCE_CEQ_ENTRY_SIZE;
4242+
4243+ return (struct hns_roce_ceqe *)((u8 *)
4244+ (eq->buf_list[off / HNS_ROCE_BA_SIZE].buf) +
4245+ off % HNS_ROCE_BA_SIZE);
4246+}
4247+
4248+static struct hns_roce_ceqe *next_ceqe_sw_v1(struct hns_roce_eq *eq)
4249+{
4250+ struct hns_roce_ceqe *ceqe = get_ceqe_v1(eq, eq->cons_index);
4251+
4252+ return (!!(roce_get_bit(ceqe->comp,
4253+ HNS_ROCE_CEQE_CEQE_COMP_OWNER_S))) ^
4254+ (!!(eq->cons_index & eq->entries)) ? ceqe : NULL;
4255+}
4256+
4257+static int hns_roce_v1_ceq_int(struct hns_roce_dev *hr_dev,
4258+ struct hns_roce_eq *eq)
4259+{
4260+ struct hns_roce_ceqe *ceqe;
4261+ int ceqes_found = 0;
4262+ u32 cqn;
4263+
4264+ while ((ceqe = next_ceqe_sw_v1(eq))) {
4265+
4266+ /* Make sure we read CEQ entry after we have checked the
4267+ * ownership bit
4268+ */
4269+ dma_rmb();
4270+
4271+ cqn = roce_get_field(ceqe->comp,
4272+ HNS_ROCE_CEQE_CEQE_COMP_CQN_M,
4273+ HNS_ROCE_CEQE_CEQE_COMP_CQN_S);
4274+ hns_roce_cq_completion(hr_dev, cqn);
4275+
4276+ ++eq->cons_index;
4277+ ceqes_found = 1;
4278+
4279+ if (eq->cons_index > 2 * hr_dev->caps.ceqe_depth - 1) {
4280+ dev_warn(&eq->hr_dev->pdev->dev,
4281+ "cons_index overflow, set back to 0.\n");
4282+ eq->cons_index = 0;
4283+ }
4284+ }
4285+
4286+ set_eq_cons_index_v1(eq, 0);
4287+
4288+ return ceqes_found;
4289+}
4290+
4291+static irqreturn_t hns_roce_v1_msix_interrupt_eq(int irq, void *eq_ptr)
4292+{
4293+ struct hns_roce_eq *eq = eq_ptr;
4294+ struct hns_roce_dev *hr_dev = eq->hr_dev;
4295+ int int_work = 0;
4296+
4297+ if (eq->type_flag == HNS_ROCE_CEQ)
4298+ /* CEQ irq routine, CEQ is pulse irq, not clear */
4299+ int_work = hns_roce_v1_ceq_int(hr_dev, eq);
4300+ else
4301+ /* AEQ irq routine, AEQ is pulse irq, not clear */
4302+ int_work = hns_roce_v1_aeq_int(hr_dev, eq);
4303+
4304+ return IRQ_RETVAL(int_work);
4305+}
4306+
4307+static irqreturn_t hns_roce_v1_msix_interrupt_abn(int irq, void *dev_id)
4308+{
4309+ struct hns_roce_dev *hr_dev = dev_id;
4310+ struct device *dev = &hr_dev->pdev->dev;
4311+ int int_work = 0;
4312+ u32 caepaemask_val;
4313+ u32 cealmovf_val;
4314+ u32 caepaest_val;
4315+ u32 aeshift_val;
4316+ u32 ceshift_val;
4317+ u32 cemask_val;
4318+ int i;
4319+
4320+ /*
4321+ * Abnormal interrupt:
4322+ * AEQ overflow, ECC multi-bit err, CEQ overflow must clear
4323+ * interrupt, mask irq, clear irq, cancel mask operation
4324+ */
4325+ aeshift_val = roce_read(hr_dev, ROCEE_CAEP_AEQC_AEQE_SHIFT_REG);
4326+
4327+ /* AEQE overflow */
4328+ if (roce_get_bit(aeshift_val,
4329+ ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQ_ALM_OVF_INT_ST_S) == 1) {
4330+ dev_warn(dev, "AEQ overflow!\n");
4331+
4332+ /* Set mask */
4333+ caepaemask_val = roce_read(hr_dev, ROCEE_CAEP_AE_MASK_REG);
4334+ roce_set_bit(caepaemask_val,
4335+ ROCEE_CAEP_AE_MASK_CAEP_AEQ_ALM_OVF_MASK_S,
4336+ HNS_ROCE_INT_MASK_ENABLE);
4337+ roce_write(hr_dev, ROCEE_CAEP_AE_MASK_REG, caepaemask_val);
4338+
4339+ /* Clear int state(INT_WC : write 1 clear) */
4340+ caepaest_val = roce_read(hr_dev, ROCEE_CAEP_AE_ST_REG);
4341+ roce_set_bit(caepaest_val,
4342+ ROCEE_CAEP_AE_ST_CAEP_AEQ_ALM_OVF_S, 1);
4343+ roce_write(hr_dev, ROCEE_CAEP_AE_ST_REG, caepaest_val);
4344+
4345+ /* Clear mask */
4346+ caepaemask_val = roce_read(hr_dev, ROCEE_CAEP_AE_MASK_REG);
4347+ roce_set_bit(caepaemask_val,
4348+ ROCEE_CAEP_AE_MASK_CAEP_AEQ_ALM_OVF_MASK_S,
4349+ HNS_ROCE_INT_MASK_DISABLE);
4350+ roce_write(hr_dev, ROCEE_CAEP_AE_MASK_REG, caepaemask_val);
4351+ }
4352+
4353+ /* CEQ almost overflow */
4354+ for (i = 0; i < hr_dev->caps.num_comp_vectors; i++) {
4355+ ceshift_val = roce_read(hr_dev, ROCEE_CAEP_CEQC_SHIFT_0_REG +
4356+ i * CEQ_REG_OFFSET);
4357+
4358+ if (roce_get_bit(ceshift_val,
4359+ ROCEE_CAEP_CEQC_SHIFT_CAEP_CEQ_ALM_OVF_INT_ST_S) == 1) {
4360+ dev_warn(dev, "CEQ[%d] almost overflow!\n", i);
4361+ int_work++;
4362+
4363+ /* Set mask */
4364+ cemask_val = roce_read(hr_dev,
4365+ ROCEE_CAEP_CE_IRQ_MASK_0_REG +
4366+ i * CEQ_REG_OFFSET);
4367+ roce_set_bit(cemask_val,
4368+ ROCEE_CAEP_CE_IRQ_MASK_CAEP_CEQ_ALM_OVF_MASK_S,
4369+ HNS_ROCE_INT_MASK_ENABLE);
4370+ roce_write(hr_dev, ROCEE_CAEP_CE_IRQ_MASK_0_REG +
4371+ i * CEQ_REG_OFFSET, cemask_val);
4372+
4373+ /* Clear int state(INT_WC : write 1 clear) */
4374+ cealmovf_val = roce_read(hr_dev,
4375+ ROCEE_CAEP_CEQ_ALM_OVF_0_REG +
4376+ i * CEQ_REG_OFFSET);
4377+ roce_set_bit(cealmovf_val,
4378+ ROCEE_CAEP_CEQ_ALM_OVF_CAEP_CEQ_ALM_OVF_S,
4379+ 1);
4380+ roce_write(hr_dev, ROCEE_CAEP_CEQ_ALM_OVF_0_REG +
4381+ i * CEQ_REG_OFFSET, cealmovf_val);
4382+
4383+ /* Clear mask */
4384+ cemask_val = roce_read(hr_dev,
4385+ ROCEE_CAEP_CE_IRQ_MASK_0_REG +
4386+ i * CEQ_REG_OFFSET);
4387+ roce_set_bit(cemask_val,
4388+ ROCEE_CAEP_CE_IRQ_MASK_CAEP_CEQ_ALM_OVF_MASK_S,
4389+ HNS_ROCE_INT_MASK_DISABLE);
4390+ roce_write(hr_dev, ROCEE_CAEP_CE_IRQ_MASK_0_REG +
4391+ i * CEQ_REG_OFFSET, cemask_val);
4392+ }
4393+ }
4394+
4395+ /* ECC multi-bit error alarm */
4396+ dev_warn(dev, "ECC UCERR ALARM: 0x%x, 0x%x, 0x%x\n",
4397+ roce_read(hr_dev, ROCEE_ECC_UCERR_ALM0_REG),
4398+ roce_read(hr_dev, ROCEE_ECC_UCERR_ALM1_REG),
4399+ roce_read(hr_dev, ROCEE_ECC_UCERR_ALM2_REG));
4400+
4401+ dev_warn(dev, "ECC CERR ALARM: 0x%x, 0x%x, 0x%x\n",
4402+ roce_read(hr_dev, ROCEE_ECC_CERR_ALM0_REG),
4403+ roce_read(hr_dev, ROCEE_ECC_CERR_ALM1_REG),
4404+ roce_read(hr_dev, ROCEE_ECC_CERR_ALM2_REG));
4405+
4406+ return IRQ_RETVAL(int_work);
4407+}
4408+
4409+static void hns_roce_v1_int_mask_enable(struct hns_roce_dev *hr_dev)
4410+{
4411+ u32 aemask_val;
4412+ int masken = 0;
4413+ int i;
4414+
4415+ /* AEQ INT */
4416+ aemask_val = roce_read(hr_dev, ROCEE_CAEP_AE_MASK_REG);
4417+ roce_set_bit(aemask_val, ROCEE_CAEP_AE_MASK_CAEP_AEQ_ALM_OVF_MASK_S,
4418+ masken);
4419+ roce_set_bit(aemask_val, ROCEE_CAEP_AE_MASK_CAEP_AE_IRQ_MASK_S, masken);
4420+ roce_write(hr_dev, ROCEE_CAEP_AE_MASK_REG, aemask_val);
4421+
4422+ /* CEQ INT */
4423+ for (i = 0; i < hr_dev->caps.num_comp_vectors; i++) {
4424+ /* IRQ mask */
4425+ roce_write(hr_dev, ROCEE_CAEP_CE_IRQ_MASK_0_REG +
4426+ i * CEQ_REG_OFFSET, masken);
4427+ }
4428+}
4429+
4430+static void hns_roce_v1_free_eq(struct hns_roce_dev *hr_dev,
4431+ struct hns_roce_eq *eq)
4432+{
4433+ int npages = (PAGE_ALIGN(eq->eqe_size * eq->entries) +
4434+ HNS_ROCE_BA_SIZE - 1) / HNS_ROCE_BA_SIZE;
4435+ int i;
4436+
4437+ if (!eq->buf_list)
4438+ return;
4439+
4440+ for (i = 0; i < npages; ++i)
4441+ dma_free_coherent(&hr_dev->pdev->dev, HNS_ROCE_BA_SIZE,
4442+ eq->buf_list[i].buf, eq->buf_list[i].map);
4443+
4444+ kfree(eq->buf_list);
4445+}
4446+
4447+static void hns_roce_v1_enable_eq(struct hns_roce_dev *hr_dev, int eq_num,
4448+ int enable_flag)
4449+{
4450+ void __iomem *eqc = hr_dev->eq_table.eqc_base[eq_num];
4451+ u32 val;
4452+
4453+ val = readl(eqc);
4454+
4455+ if (enable_flag)
4456+ roce_set_field(val,
4457+ ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_M,
4458+ ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_S,
4459+ HNS_ROCE_EQ_STAT_VALID);
4460+ else
4461+ roce_set_field(val,
4462+ ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_M,
4463+ ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_S,
4464+ HNS_ROCE_EQ_STAT_INVALID);
4465+ writel(val, eqc);
4466+}
4467+
4468+static int hns_roce_v1_create_eq(struct hns_roce_dev *hr_dev,
4469+ struct hns_roce_eq *eq)
4470+{
4471+ void __iomem *eqc = hr_dev->eq_table.eqc_base[eq->eqn];
4472+ struct device *dev = &hr_dev->pdev->dev;
4473+ dma_addr_t tmp_dma_addr;
4474+ u32 eqconsindx_val = 0;
4475+ u32 eqcuridx_val = 0;
4476+ u32 eqshift_val = 0;
4477+ int num_bas;
4478+ int ret;
4479+ int i;
4480+
4481+ num_bas = (PAGE_ALIGN(eq->entries * eq->eqe_size) +
4482+ HNS_ROCE_BA_SIZE - 1) / HNS_ROCE_BA_SIZE;
4483+
4484+ if ((eq->entries * eq->eqe_size) > HNS_ROCE_BA_SIZE) {
4485+ dev_err(dev, "[error]eq buf %d gt ba size(%d) need bas=%d\n",
4486+ (eq->entries * eq->eqe_size), HNS_ROCE_BA_SIZE,
4487+ num_bas);
4488+ return -EINVAL;
4489+ }
4490+
4491+ eq->buf_list = kcalloc(num_bas, sizeof(*eq->buf_list), GFP_KERNEL);
4492+ if (!eq->buf_list)
4493+ return -ENOMEM;
4494+
4495+ for (i = 0; i < num_bas; ++i) {
4496+ eq->buf_list[i].buf = dma_alloc_coherent(dev, HNS_ROCE_BA_SIZE,
4497+ &tmp_dma_addr,
4498+ GFP_KERNEL);
4499+ if (!eq->buf_list[i].buf) {
4500+ ret = -ENOMEM;
4501+ goto err_out_free_pages;
4502+ }
4503+
4504+ eq->buf_list[i].map = tmp_dma_addr;
4505+ memset(eq->buf_list[i].buf, 0, HNS_ROCE_BA_SIZE);
4506+ }
4507+ eq->cons_index = 0;
4508+ roce_set_field(eqshift_val,
4509+ ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_M,
4510+ ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_S,
4511+ HNS_ROCE_EQ_STAT_INVALID);
4512+ roce_set_field(eqshift_val,
4513+ ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_AEQE_SHIFT_M,
4514+ ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_AEQE_SHIFT_S,
4515+ eq->log_entries);
4516+ writel(eqshift_val, eqc);
4517+
4518+ /* Configure eq extended address 12~44bit */
4519+ writel((u32)(eq->buf_list[0].map >> 12), eqc + 4);
4520+
4521+ /*
4522+ * Configure eq extended address 45~49 bit.
4523+ * 44 = 32 + 12, When evaluating addr to hardware, shift 12 because of
4524+ * using 4K page, and shift more 32 because of
4525+ * caculating the high 32 bit value evaluated to hardware.
4526+ */
4527+ roce_set_field(eqcuridx_val, ROCEE_CAEP_AEQE_CUR_IDX_CAEP_AEQ_BT_H_M,
4528+ ROCEE_CAEP_AEQE_CUR_IDX_CAEP_AEQ_BT_H_S,
4529+ eq->buf_list[0].map >> 44);
4530+ roce_set_field(eqcuridx_val,
4531+ ROCEE_CAEP_AEQE_CUR_IDX_CAEP_AEQE_CUR_IDX_M,
4532+ ROCEE_CAEP_AEQE_CUR_IDX_CAEP_AEQE_CUR_IDX_S, 0);
4533+ writel(eqcuridx_val, eqc + 8);
4534+
4535+ /* Configure eq consumer index */
4536+ roce_set_field(eqconsindx_val,
4537+ ROCEE_CAEP_AEQE_CONS_IDX_CAEP_AEQE_CONS_IDX_M,
4538+ ROCEE_CAEP_AEQE_CONS_IDX_CAEP_AEQE_CONS_IDX_S, 0);
4539+ writel(eqconsindx_val, eqc + 0xc);
4540+
4541+ return 0;
4542+
4543+err_out_free_pages:
4544+ for (i -= 1; i >= 0; i--)
4545+ dma_free_coherent(dev, HNS_ROCE_BA_SIZE, eq->buf_list[i].buf,
4546+ eq->buf_list[i].map);
4547+
4548+ kfree(eq->buf_list);
4549+ return ret;
4550+}
4551+
4552+static int hns_roce_v1_init_eq_table(struct hns_roce_dev *hr_dev)
4553+{
4554+ struct hns_roce_eq_table *eq_table = &hr_dev->eq_table;
4555+ struct device *dev = &hr_dev->pdev->dev;
4556+ struct hns_roce_eq *eq;
4557+ int irq_num;
4558+ int eq_num;
4559+ int ret;
4560+ int i, j;
4561+
4562+ eq_num = hr_dev->caps.num_comp_vectors + hr_dev->caps.num_aeq_vectors;
4563+ irq_num = eq_num + hr_dev->caps.num_other_vectors;
4564+
4565+ eq_table->eq = kcalloc(eq_num, sizeof(*eq_table->eq), GFP_KERNEL);
4566+ if (!eq_table->eq)
4567+ return -ENOMEM;
4568+
4569+ eq_table->eqc_base = kcalloc(eq_num, sizeof(*eq_table->eqc_base),
4570+ GFP_KERNEL);
4571+ if (!eq_table->eqc_base) {
4572+ ret = -ENOMEM;
4573+ goto err_eqc_base_alloc_fail;
4574+ }
4575+
4576+ for (i = 0; i < eq_num; i++) {
4577+ eq = &eq_table->eq[i];
4578+ eq->hr_dev = hr_dev;
4579+ eq->eqn = i;
4580+ eq->irq = hr_dev->irq[i];
4581+ eq->log_page_size = PAGE_SHIFT;
4582+
4583+ if (i < hr_dev->caps.num_comp_vectors) {
4584+ /* CEQ */
4585+ eq_table->eqc_base[i] = hr_dev->reg_base +
4586+ ROCEE_CAEP_CEQC_SHIFT_0_REG +
4587+ CEQ_REG_OFFSET * i;
4588+ eq->type_flag = HNS_ROCE_CEQ;
4589+ eq->doorbell = hr_dev->reg_base +
4590+ ROCEE_CAEP_CEQC_CONS_IDX_0_REG +
4591+ CEQ_REG_OFFSET * i;
4592+ eq->entries = hr_dev->caps.ceqe_depth;
4593+ eq->log_entries = ilog2(eq->entries);
4594+ eq->eqe_size = HNS_ROCE_CEQ_ENTRY_SIZE;
4595+ } else {
4596+ /* AEQ */
4597+ eq_table->eqc_base[i] = hr_dev->reg_base +
4598+ ROCEE_CAEP_AEQC_AEQE_SHIFT_REG;
4599+ eq->type_flag = HNS_ROCE_AEQ;
4600+ eq->doorbell = hr_dev->reg_base +
4601+ ROCEE_CAEP_AEQE_CONS_IDX_REG;
4602+ eq->entries = hr_dev->caps.aeqe_depth;
4603+ eq->log_entries = ilog2(eq->entries);
4604+ eq->eqe_size = HNS_ROCE_AEQ_ENTRY_SIZE;
4605+ }
4606+ }
4607+
4608+ /* Disable irq */
4609+ hns_roce_v1_int_mask_enable(hr_dev);
4610+
4611+ /* Configure ce int interval */
4612+ roce_write(hr_dev, ROCEE_CAEP_CE_INTERVAL_CFG_REG,
4613+ HNS_ROCE_CEQ_DEFAULT_INTERVAL);
4614+
4615+ /* Configure ce int burst num */
4616+ roce_write(hr_dev, ROCEE_CAEP_CE_BURST_NUM_CFG_REG,
4617+ HNS_ROCE_CEQ_DEFAULT_BURST_NUM);
4618+
4619+ for (i = 0; i < eq_num; i++) {
4620+ ret = hns_roce_v1_create_eq(hr_dev, &eq_table->eq[i]);
4621+ if (ret) {
4622+ dev_err(dev, "eq create failed\n");
4623+ goto err_create_eq_fail;
4624+ }
4625+ }
4626+
4627+ for (j = 0; j < irq_num; j++) {
4628+ if (j < eq_num)
4629+ ret = request_irq(hr_dev->irq[j],
4630+ hns_roce_v1_msix_interrupt_eq, 0,
4631+ hr_dev->irq_names[j],
4632+ &eq_table->eq[j]);
4633+ else
4634+ ret = request_irq(hr_dev->irq[j],
4635+ hns_roce_v1_msix_interrupt_abn, 0,
4636+ hr_dev->irq_names[j], hr_dev);
4637+
4638+ if (ret) {
4639+ dev_err(dev, "request irq error!\n");
4640+ goto err_request_irq_fail;
4641+ }
4642+ }
4643+
4644+ for (i = 0; i < eq_num; i++)
4645+ hns_roce_v1_enable_eq(hr_dev, i, EQ_ENABLE);
4646+
4647+ return 0;
4648+
4649+err_request_irq_fail:
4650+ for (j -= 1; j >= 0; j--)
4651+ free_irq(hr_dev->irq[j], &eq_table->eq[j]);
4652+
4653+err_create_eq_fail:
4654+ for (i -= 1; i >= 0; i--)
4655+ hns_roce_v1_free_eq(hr_dev, &eq_table->eq[i]);
4656+
4657+ kfree(eq_table->eqc_base);
4658+
4659+err_eqc_base_alloc_fail:
4660+ kfree(eq_table->eq);
4661+
4662+ return ret;
4663+}
4664+
4665+static void hns_roce_v1_cleanup_eq_table(struct hns_roce_dev *hr_dev)
4666+{
4667+ struct hns_roce_eq_table *eq_table = &hr_dev->eq_table;
4668+ int irq_num;
4669+ int eq_num;
4670+ int i;
4671+
4672+ eq_num = hr_dev->caps.num_comp_vectors + hr_dev->caps.num_aeq_vectors;
4673+ irq_num = eq_num + hr_dev->caps.num_other_vectors;
4674+ for (i = 0; i < eq_num; i++) {
4675+ /* Disable EQ */
4676+ hns_roce_v1_enable_eq(hr_dev, i, EQ_DISABLE);
4677+
4678+ free_irq(hr_dev->irq[i], &eq_table->eq[i]);
4679+
4680+ hns_roce_v1_free_eq(hr_dev, &eq_table->eq[i]);
4681+ }
4682+ for (i = eq_num; i < irq_num; i++)
4683+ free_irq(hr_dev->irq[i], hr_dev);
4684+
4685+ kfree(eq_table->eqc_base);
4686+ kfree(eq_table->eq);
4687+}
4688+
39634689 static const struct hns_roce_hw hns_roce_hw_v1 = {
39644690 .reset = hns_roce_v1_reset,
39654691 .hw_profile = hns_roce_v1_profile,
@@ -3983,6 +4709,8 @@ static const struct hns_roce_hw hns_roce_hw_v1 = { · 17 unmodified lines
39834709 .poll_cq = hns_roce_v1_poll_cq,
39844710 .dereg_mr = hns_roce_v1_dereg_mr,
39854711 .destroy_cq = hns_roce_v1_destroy_cq,
4712+ .init_eq = hns_roce_v1_init_eq_table,
4713+ .cleanup_eq = hns_roce_v1_cleanup_eq_table,
39864714 };
39874715
39884716 static const struct of_device_id hns_roce_of_match[] = {
@@ -4060,10 +4788,6 @@ static int hns_roce_get_cfg(struct hns_roce_dev *hr_dev) · 71 unmodified lines
40604788
40614789 /* get the mapped register base address */
40624790 res = platform_get_resource(hr_dev->pdev, IORESOURCE_MEM, 0);
4063− if (!res) {
4064− dev_err(dev, "memory resource not found!\n");
4065− return -EINVAL;
4066− }
40674791 hr_dev->reg_base = devm_ioremap_resource(dev, res);
40684792 if (IS_ERR(hr_dev->reg_base))
40694793 return PTR_ERR(hr_dev->reg_base);
@@ -4132,14 +4856,14 @@ static int hns_roce_get_cfg(struct hns_roce_dev *hr_dev) · 62 unmodified lines
41324856 /* read the interrupt names from the DT or ACPI */
41334857 ret = device_property_read_string_array(dev, "interrupt-names",
41344858 hr_dev->irq_names,
4135− HNS_ROCE_MAX_IRQ_NUM);
4859+ HNS_ROCE_V1_MAX_IRQ_NUM);
41364860 if (ret < 0) {
41374861 dev_err(dev, "couldn't get interrupt names from DT or ACPI!\n");
41384862 return ret;
41394863 }
41404864
41414865 /* fetch the interrupt numbers */
4142− for (i = 0; i < HNS_ROCE_MAX_IRQ_NUM; i++) {
4866+ for (i = 0; i < HNS_ROCE_V1_MAX_IRQ_NUM; i++) {
41434867 hr_dev->irq[i] = platform_get_irq(hr_dev->pdev, i);
41444868 if (hr_dev->irq[i] <= 0) {
41454869 dev_err(dev, "platform get of irq[=%d] failed!\n", i);
drivers/infiniband/hw/hns/hns_roce_hw_v1.h+42−2
@@ -60,8 +60,13 @@ · 59 unmodified lines
6060 #define HNS_ROCE_V1_GID_NUM 16
6161 #define HNS_ROCE_V1_RESV_QP 8
6262
63−#define HNS_ROCE_V1_NUM_COMP_EQE 0x8000
64−#define HNS_ROCE_V1_NUM_ASYNC_EQE 0x400
63+#define HNS_ROCE_V1_MAX_IRQ_NUM 34
64+#define HNS_ROCE_V1_COMP_VEC_NUM 32
65+#define HNS_ROCE_V1_AEQE_VEC_NUM 1
66+#define HNS_ROCE_V1_ABNORMAL_VEC_NUM 1
67+
68+#define HNS_ROCE_V1_COMP_EQE_NUM 0x8000
69+#define HNS_ROCE_V1_ASYNC_EQE_NUM 0x400
6570
6671 #define HNS_ROCE_V1_QPC_ENTRY_SIZE 256
6772 #define HNS_ROCE_V1_IRRL_ENTRY_SIZE 8
@@ -159,6 +164,41 @@ · 91 unmodified lines
159164 #define SDB_INV_CNT_OFFSET 8
160165 #define SDB_ST_CMP_VAL 8
161166
167+#define HNS_ROCE_CEQ_DEFAULT_INTERVAL 0x10
168+#define HNS_ROCE_CEQ_DEFAULT_BURST_NUM 0x10
169+
170+#define HNS_ROCE_INT_MASK_DISABLE 0
171+#define HNS_ROCE_INT_MASK_ENABLE 1
172+
173+#define CEQ_REG_OFFSET 0x18
174+
175+#define HNS_ROCE_CEQE_CEQE_COMP_OWNER_S 0
176+
177+#define HNS_ROCE_V1_CONS_IDX_M GENMASK(15, 0)
178+
179+#define HNS_ROCE_CEQE_CEQE_COMP_CQN_S 16
180+#define HNS_ROCE_CEQE_CEQE_COMP_CQN_M GENMASK(31, 16)
181+
182+#define HNS_ROCE_AEQE_U32_4_EVENT_TYPE_S 16
183+#define HNS_ROCE_AEQE_U32_4_EVENT_TYPE_M GENMASK(23, 16)
184+
185+#define HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_S 24
186+#define HNS_ROCE_AEQE_U32_4_EVENT_SUB_TYPE_M GENMASK(30, 24)
187+
188+#define HNS_ROCE_AEQE_U32_4_OWNER_S 31
189+
190+#define HNS_ROCE_AEQE_EVENT_QP_EVENT_QP_QPN_S 0
191+#define HNS_ROCE_AEQE_EVENT_QP_EVENT_QP_QPN_M GENMASK(23, 0)
192+
193+#define HNS_ROCE_AEQE_EVENT_QP_EVENT_PORT_NUM_S 25
194+#define HNS_ROCE_AEQE_EVENT_QP_EVENT_PORT_NUM_M GENMASK(27, 25)
195+
196+#define HNS_ROCE_AEQE_EVENT_CQ_EVENT_CQ_CQN_S 0
197+#define HNS_ROCE_AEQE_EVENT_CQ_EVENT_CQ_CQN_M GENMASK(15, 0)
198+
199+#define HNS_ROCE_AEQE_EVENT_CE_EVENT_CEQE_CEQN_S 0
200+#define HNS_ROCE_AEQE_EVENT_CE_EVENT_CEQE_CEQN_M GENMASK(4, 0)
201+
162202 struct hns_roce_cq_context {
163203 u32 cqc_byte_4;
164204 u32 cq_bt_l;
drivers/infiniband/hw/hns/hns_roce_hw_v2.c+1655−182
@@ -34,6 +34,7 @@ · 33 unmodified lines
3434 #include <linux/etherdevice.h>
3535 #include <linux/interrupt.h>
3636 #include <linux/kernel.h>
37+#include <net/addrconf.h>
3738 #include <rdma/ib_umem.h>
3839
3940 #include "hnae3.h"
@@ -51,32 +52,106 @@ static void set_data_seg_v2(struct hns_roce_v2_wqe_data_seg *dseg, · 11 unmodified lines
5152 dseg->len = cpu_to_le32(sg->length);
5253 }
5354
55+static int set_rwqe_data_seg(struct ib_qp *ibqp, struct ib_send_wr *wr,
56+ struct hns_roce_v2_rc_send_wqe *rc_sq_wqe,
57+ void *wqe, unsigned int *sge_ind,
58+ struct ib_send_wr **bad_wr)
59+{
60+ struct hns_roce_dev *hr_dev = to_hr_dev(ibqp->device);
61+ struct hns_roce_v2_wqe_data_seg *dseg = wqe;
62+ struct hns_roce_qp *qp = to_hr_qp(ibqp);
63+ int i;
64+
65+ if (wr->send_flags & IB_SEND_INLINE && wr->num_sge) {
66+ if (rc_sq_wqe->msg_len > hr_dev->caps.max_sq_inline) {
67+ *bad_wr = wr;
68+ dev_err(hr_dev->dev, "inline len(1-%d)=%d, illegal",
69+ rc_sq_wqe->msg_len, hr_dev->caps.max_sq_inline);
70+ return -EINVAL;
71+ }
72+
73+ for (i = 0; i < wr->num_sge; i++) {
74+ memcpy(wqe, ((void *)wr->sg_list[i].addr),
75+ wr->sg_list[i].length);
76+ wqe += wr->sg_list[i].length;
77+ }
78+
79+ roce_set_bit(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_INLINE_S,
80+ 1);
81+ } else {
82+ if (wr->num_sge <= 2) {
83+ for (i = 0; i < wr->num_sge; i++) {
84+ if (likely(wr->sg_list[i].length)) {
85+ set_data_seg_v2(dseg, wr->sg_list + i);
86+ dseg++;
87+ }
88+ }
89+ } else {
90+ roce_set_field(rc_sq_wqe->byte_20,
91+ V2_RC_SEND_WQE_BYTE_20_MSG_START_SGE_IDX_M,
92+ V2_RC_SEND_WQE_BYTE_20_MSG_START_SGE_IDX_S,
93+ (*sge_ind) & (qp->sge.sge_cnt - 1));
94+
95+ for (i = 0; i < 2; i++) {
96+ if (likely(wr->sg_list[i].length)) {
97+ set_data_seg_v2(dseg, wr->sg_list + i);
98+ dseg++;
99+ }
100+ }
101+
102+ dseg = get_send_extend_sge(qp,
103+ (*sge_ind) & (qp->sge.sge_cnt - 1));
104+
105+ for (i = 0; i < wr->num_sge - 2; i++) {
106+ if (likely(wr->sg_list[i + 2].length)) {
107+ set_data_seg_v2(dseg,
108+ wr->sg_list + 2 + i);
109+ dseg++;
110+ (*sge_ind)++;
111+ }
112+ }
113+ }
114+
115+ roce_set_field(rc_sq_wqe->byte_16,
116+ V2_RC_SEND_WQE_BYTE_16_SGE_NUM_M,
117+ V2_RC_SEND_WQE_BYTE_16_SGE_NUM_S, wr->num_sge);
118+ }
119+
120+ return 0;
121+}
122+
54123 static int hns_roce_v2_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
55124 struct ib_send_wr **bad_wr)
56125 {
57126 struct hns_roce_dev *hr_dev = to_hr_dev(ibqp->device);
127+ struct hns_roce_ah *ah = to_hr_ah(ud_wr(wr)->ah);
128+ struct hns_roce_v2_ud_send_wqe *ud_sq_wqe;
58129 struct hns_roce_v2_rc_send_wqe *rc_sq_wqe;
59130 struct hns_roce_qp *qp = to_hr_qp(ibqp);
60131 struct hns_roce_v2_wqe_data_seg *dseg;
61132 struct device *dev = hr_dev->dev;
62133 struct hns_roce_v2_db sq_db;
63134 unsigned int sge_ind = 0;
64− unsigned int wqe_sz = 0;
65135 unsigned int owner_bit;
66136 unsigned long flags;
67137 unsigned int ind;
68138 void *wqe = NULL;
139+ bool loopback;
69140 int ret = 0;
141+ u8 *smac;
70142 int nreq;
71143 int i;
72144
73− if (unlikely(ibqp->qp_type != IB_QPT_RC)) {
145+ if (unlikely(ibqp->qp_type != IB_QPT_RC &&
146+ ibqp->qp_type != IB_QPT_GSI &&
147+ ibqp->qp_type != IB_QPT_UD)) {
74148 dev_err(dev, "Not supported QP(0x%x)type!\n", ibqp->qp_type);
75149 *bad_wr = NULL;
76150 return -EOPNOTSUPP;
77151 }
78152
79− if (unlikely(qp->state != IB_QPS_RTS && qp->state != IB_QPS_SQD)) {
153+ if (unlikely(qp->state == IB_QPS_RESET || qp->state == IB_QPS_INIT ||
154+ qp->state == IB_QPS_RTR)) {
80155 dev_err(dev, "Post WQE fail, QP state %d err!\n", qp->state);
81156 *bad_wr = wr;
82157 return -EINVAL;
@@ -106,161 +181,255 @@ static int hns_roce_v2_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, · 23 unmodified lines
106181 wr->wr_id;
107182
108183 owner_bit = ~(qp->sq.head >> ilog2(qp->sq.wqe_cnt)) & 0x1;
109− rc_sq_wqe = wqe;
110− memset(rc_sq_wqe, 0, sizeof(*rc_sq_wqe));
111− for (i = 0; i < wr->num_sge; i++)
112− rc_sq_wqe->msg_len += wr->sg_list[i].length;
113184
114− rc_sq_wqe->inv_key_immtdata = send_ieth(wr);
185+ /* Corresponding to the QP type, wqe process separately */
186+ if (ibqp->qp_type == IB_QPT_GSI) {
187+ ud_sq_wqe = wqe;
188+ memset(ud_sq_wqe, 0, sizeof(*ud_sq_wqe));
189+
190+ roce_set_field(ud_sq_wqe->dmac, V2_UD_SEND_WQE_DMAC_0_M,
191+ V2_UD_SEND_WQE_DMAC_0_S, ah->av.mac[0]);
192+ roce_set_field(ud_sq_wqe->dmac, V2_UD_SEND_WQE_DMAC_1_M,
193+ V2_UD_SEND_WQE_DMAC_1_S, ah->av.mac[1]);
194+ roce_set_field(ud_sq_wqe->dmac, V2_UD_SEND_WQE_DMAC_2_M,
195+ V2_UD_SEND_WQE_DMAC_2_S, ah->av.mac[2]);
196+ roce_set_field(ud_sq_wqe->dmac, V2_UD_SEND_WQE_DMAC_3_M,
197+ V2_UD_SEND_WQE_DMAC_3_S, ah->av.mac[3]);
198+ roce_set_field(ud_sq_wqe->byte_48,
199+ V2_UD_SEND_WQE_BYTE_48_DMAC_4_M,
200+ V2_UD_SEND_WQE_BYTE_48_DMAC_4_S,
201+ ah->av.mac[4]);
202+ roce_set_field(ud_sq_wqe->byte_48,
203+ V2_UD_SEND_WQE_BYTE_48_DMAC_5_M,
204+ V2_UD_SEND_WQE_BYTE_48_DMAC_5_S,
205+ ah->av.mac[5]);
206+
207+ /* MAC loopback */
208+ smac = (u8 *)hr_dev->dev_addr[qp->port];
209+ loopback = ether_addr_equal_unaligned(ah->av.mac,
210+ smac) ? 1 : 0;
211+
212+ roce_set_bit(ud_sq_wqe->byte_40,
213+ V2_UD_SEND_WQE_BYTE_40_LBI_S, loopback);
214+
215+ roce_set_field(ud_sq_wqe->byte_4,
216+ V2_UD_SEND_WQE_BYTE_4_OPCODE_M,
217+ V2_UD_SEND_WQE_BYTE_4_OPCODE_S,
218+ HNS_ROCE_V2_WQE_OP_SEND);
115219
116− roce_set_bit(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_FENCE_S,
117− (wr->send_flags & IB_SEND_FENCE) ? 1 : 0);
220+ for (i = 0; i < wr->num_sge; i++)
221+ ud_sq_wqe->msg_len += wr->sg_list[i].length;
118222
119− roce_set_bit(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_SE_S,
120− (wr->send_flags & IB_SEND_SOLICITED) ? 1 : 0);
223+ ud_sq_wqe->immtdata = send_ieth(wr);
121224
122− roce_set_bit(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_CQE_S,
123− (wr->send_flags & IB_SEND_SIGNALED) ? 1 : 0);
225+ /* Set sig attr */
226+ roce_set_bit(ud_sq_wqe->byte_4,
227+ V2_UD_SEND_WQE_BYTE_4_CQE_S,
228+ (wr->send_flags & IB_SEND_SIGNALED) ? 1 : 0);
124229
125− roce_set_bit(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_OWNER_S,
126− owner_bit);
230+ /* Set se attr */
231+ roce_set_bit(ud_sq_wqe->byte_4,
232+ V2_UD_SEND_WQE_BYTE_4_SE_S,
233+ (wr->send_flags & IB_SEND_SOLICITED) ? 1 : 0);
127234
128− switch (wr->opcode) {
129− case IB_WR_RDMA_READ:
130− roce_set_field(rc_sq_wqe->byte_4,
131− V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
132− V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
133− HNS_ROCE_V2_WQE_OP_RDMA_READ);
134− rc_sq_wqe->rkey = cpu_to_le32(rdma_wr(wr)->rkey);
135− rc_sq_wqe->va = cpu_to_le64(rdma_wr(wr)->remote_addr);
136− break;
137− case IB_WR_RDMA_WRITE:
138− roce_set_field(rc_sq_wqe->byte_4,
139− V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
140− V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
141− HNS_ROCE_V2_WQE_OP_RDMA_WRITE);
142− rc_sq_wqe->rkey = cpu_to_le32(rdma_wr(wr)->rkey);
143− rc_sq_wqe->va = cpu_to_le64(rdma_wr(wr)->remote_addr);
144− break;
145− case IB_WR_RDMA_WRITE_WITH_IMM:
146− roce_set_field(rc_sq_wqe->byte_4,
235+ roce_set_bit(ud_sq_wqe->byte_4,
236+ V2_UD_SEND_WQE_BYTE_4_OWNER_S, owner_bit);
237+
238+ roce_set_field(ud_sq_wqe->byte_16,
239+ V2_UD_SEND_WQE_BYTE_16_PD_M,
240+ V2_UD_SEND_WQE_BYTE_16_PD_S,
241+ to_hr_pd(ibqp->pd)->pdn);
242+
243+ roce_set_field(ud_sq_wqe->byte_16,
244+ V2_UD_SEND_WQE_BYTE_16_SGE_NUM_M,
245+ V2_UD_SEND_WQE_BYTE_16_SGE_NUM_S,
246+ wr->num_sge);
247+
248+ roce_set_field(ud_sq_wqe->byte_20,
249+ V2_UD_SEND_WQE_BYTE_20_MSG_START_SGE_IDX_M,
250+ V2_UD_SEND_WQE_BYTE_20_MSG_START_SGE_IDX_S,
251+ sge_ind & (qp->sge.sge_cnt - 1));
252+
253+ roce_set_field(ud_sq_wqe->byte_24,
254+ V2_UD_SEND_WQE_BYTE_24_UDPSPN_M,
255+ V2_UD_SEND_WQE_BYTE_24_UDPSPN_S, 0);
256+ ud_sq_wqe->qkey =
257+ cpu_to_be32(ud_wr(wr)->remote_qkey & 0x80000000) ?
258+ qp->qkey : ud_wr(wr)->remote_qkey;
259+ roce_set_field(ud_sq_wqe->byte_32,
260+ V2_UD_SEND_WQE_BYTE_32_DQPN_M,
261+ V2_UD_SEND_WQE_BYTE_32_DQPN_S,
262+ ud_wr(wr)->remote_qpn);
263+
264+ roce_set_field(ud_sq_wqe->byte_36,
265+ V2_UD_SEND_WQE_BYTE_36_VLAN_M,
266+ V2_UD_SEND_WQE_BYTE_36_VLAN_S,
267+ ah->av.vlan);
268+ roce_set_field(ud_sq_wqe->byte_36,
269+ V2_UD_SEND_WQE_BYTE_36_HOPLIMIT_M,
270+ V2_UD_SEND_WQE_BYTE_36_HOPLIMIT_S,
271+ ah->av.hop_limit);
272+ roce_set_field(ud_sq_wqe->byte_36,
273+ V2_UD_SEND_WQE_BYTE_36_TCLASS_M,
274+ V2_UD_SEND_WQE_BYTE_36_TCLASS_S,
275+ 0);
276+ roce_set_field(ud_sq_wqe->byte_36,
277+ V2_UD_SEND_WQE_BYTE_36_TCLASS_M,
278+ V2_UD_SEND_WQE_BYTE_36_TCLASS_S,
279+ 0);
280+ roce_set_field(ud_sq_wqe->byte_40,
281+ V2_UD_SEND_WQE_BYTE_40_FLOW_LABEL_M,
282+ V2_UD_SEND_WQE_BYTE_40_FLOW_LABEL_S, 0);
283+ roce_set_field(ud_sq_wqe->byte_40,
284+ V2_UD_SEND_WQE_BYTE_40_SL_M,
285+ V2_UD_SEND_WQE_BYTE_40_SL_S,
286+ ah->av.sl_tclass_flowlabel >>
287+ HNS_ROCE_SL_SHIFT);
288+ roce_set_field(ud_sq_wqe->byte_40,
289+ V2_UD_SEND_WQE_BYTE_40_PORTN_M,
290+ V2_UD_SEND_WQE_BYTE_40_PORTN_S,
291+ qp->port);
292+
293+ roce_set_field(ud_sq_wqe->byte_48,
294+ V2_UD_SEND_WQE_BYTE_48_SGID_INDX_M,
295+ V2_UD_SEND_WQE_BYTE_48_SGID_INDX_S,
296+ hns_get_gid_index(hr_dev, qp->phy_port,
297+ ah->av.gid_index));
298+
299+ memcpy(&ud_sq_wqe->dgid[0], &ah->av.dgid[0],
300+ GID_LEN_V2);
301+
302+ dseg = get_send_extend_sge(qp,
303+ sge_ind & (qp->sge.sge_cnt - 1));
304+ for (i = 0; i < wr->num_sge; i++) {
305+ set_data_seg_v2(dseg + i, wr->sg_list + i);
306+ sge_ind++;
307+ }
308+
309+ ind++;
310+ } else if (ibqp->qp_type == IB_QPT_RC) {
311+ rc_sq_wqe = wqe;
312+ memset(rc_sq_wqe, 0, sizeof(*rc_sq_wqe));
313+ for (i = 0; i < wr->num_sge; i++)
314+ rc_sq_wqe->msg_len += wr->sg_list[i].length;
315+
316+ rc_sq_wqe->inv_key_immtdata = send_ieth(wr);
317+
318+ roce_set_bit(rc_sq_wqe->byte_4,
319+ V2_RC_SEND_WQE_BYTE_4_FENCE_S,
320+ (wr->send_flags & IB_SEND_FENCE) ? 1 : 0);
321+
322+ roce_set_bit(rc_sq_wqe->byte_4,
323+ V2_RC_SEND_WQE_BYTE_4_SE_S,
324+ (wr->send_flags & IB_SEND_SOLICITED) ? 1 : 0);
325+
326+ roce_set_bit(rc_sq_wqe->byte_4,
327+ V2_RC_SEND_WQE_BYTE_4_CQE_S,
328+ (wr->send_flags & IB_SEND_SIGNALED) ? 1 : 0);
329+
330+ roce_set_bit(rc_sq_wqe->byte_4,
331+ V2_RC_SEND_WQE_BYTE_4_OWNER_S, owner_bit);
332+
333+ switch (wr->opcode) {
334+ case IB_WR_RDMA_READ:
335+ roce_set_field(rc_sq_wqe->byte_4,
336+ V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
337+ V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
338+ HNS_ROCE_V2_WQE_OP_RDMA_READ);
339+ rc_sq_wqe->rkey =
340+ cpu_to_le32(rdma_wr(wr)->rkey);
341+ rc_sq_wqe->va =
342+ cpu_to_le64(rdma_wr(wr)->remote_addr);
343+ break;
344+ case IB_WR_RDMA_WRITE:
345+ roce_set_field(rc_sq_wqe->byte_4,
346+ V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
347+ V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
348+ HNS_ROCE_V2_WQE_OP_RDMA_WRITE);
349+ rc_sq_wqe->rkey =
350+ cpu_to_le32(rdma_wr(wr)->rkey);
351+ rc_sq_wqe->va =
352+ cpu_to_le64(rdma_wr(wr)->remote_addr);
353+ break;
354+ case IB_WR_RDMA_WRITE_WITH_IMM:
355+ roce_set_field(rc_sq_wqe->byte_4,
147356 V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
148357 V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
149358 HNS_ROCE_V2_WQE_OP_RDMA_WRITE_WITH_IMM);
150− rc_sq_wqe->rkey = cpu_to_le32(rdma_wr(wr)->rkey);
151− rc_sq_wqe->va = cpu_to_le64(rdma_wr(wr)->remote_addr);
152− break;
153− case IB_WR_SEND:
154− roce_set_field(rc_sq_wqe->byte_4,
155− V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
156− V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
157− HNS_ROCE_V2_WQE_OP_SEND);
158− break;
159− case IB_WR_SEND_WITH_INV:
160− roce_set_field(rc_sq_wqe->byte_4,
359+ rc_sq_wqe->rkey =
360+ cpu_to_le32(rdma_wr(wr)->rkey);
361+ rc_sq_wqe->va =
362+ cpu_to_le64(rdma_wr(wr)->remote_addr);
363+ break;
364+ case IB_WR_SEND:
365+ roce_set_field(rc_sq_wqe->byte_4,
366+ V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
367+ V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
368+ HNS_ROCE_V2_WQE_OP_SEND);
369+ break;
370+ case IB_WR_SEND_WITH_INV:
371+ roce_set_field(rc_sq_wqe->byte_4,
161372 V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
162373 V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
163374 HNS_ROCE_V2_WQE_OP_SEND_WITH_INV);
164− break;
165− case IB_WR_SEND_WITH_IMM:
166− roce_set_field(rc_sq_wqe->byte_4,
167− V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
168− V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
169− HNS_ROCE_V2_WQE_OP_SEND_WITH_IMM);
170− break;
171− case IB_WR_LOCAL_INV:
172− roce_set_field(rc_sq_wqe->byte_4,
173− V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
174− V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
175− HNS_ROCE_V2_WQE_OP_LOCAL_INV);
176− break;
177− case IB_WR_ATOMIC_CMP_AND_SWP:
178− roce_set_field(rc_sq_wqe->byte_4,
179− V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
180− V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
181− HNS_ROCE_V2_WQE_OP_ATOM_CMP_AND_SWAP);
182− break;
183− case IB_WR_ATOMIC_FETCH_AND_ADD:
184− roce_set_field(rc_sq_wqe->byte_4,
185− V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
186− V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
187− HNS_ROCE_V2_WQE_OP_ATOM_FETCH_AND_ADD);
188− break;
189− case IB_WR_MASKED_ATOMIC_CMP_AND_SWP:
190− roce_set_field(rc_sq_wqe->byte_4,
375+ break;
376+ case IB_WR_SEND_WITH_IMM:
377+ roce_set_field(rc_sq_wqe->byte_4,
378+ V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
379+ V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
380+ HNS_ROCE_V2_WQE_OP_SEND_WITH_IMM);
381+ break;
382+ case IB_WR_LOCAL_INV:
383+ roce_set_field(rc_sq_wqe->byte_4,
384+ V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
385+ V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
386+ HNS_ROCE_V2_WQE_OP_LOCAL_INV);
387+ break;
388+ case IB_WR_ATOMIC_CMP_AND_SWP:
389+ roce_set_field(rc_sq_wqe->byte_4,
390+ V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
391+ V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
392+ HNS_ROCE_V2_WQE_OP_ATOM_CMP_AND_SWAP);
393+ break;
394+ case IB_WR_ATOMIC_FETCH_AND_ADD:
395+ roce_set_field(rc_sq_wqe->byte_4,
396+ V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
397+ V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
398+ HNS_ROCE_V2_WQE_OP_ATOM_FETCH_AND_ADD);
399+ break;
400+ case IB_WR_MASKED_ATOMIC_CMP_AND_SWP:
401+ roce_set_field(rc_sq_wqe->byte_4,
191402 V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
192403 V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
193404 HNS_ROCE_V2_WQE_OP_ATOM_MSK_CMP_AND_SWAP);
194− break;
195− case IB_WR_MASKED_ATOMIC_FETCH_AND_ADD:
196− roce_set_field(rc_sq_wqe->byte_4,
405+ break;
406+ case IB_WR_MASKED_ATOMIC_FETCH_AND_ADD:
407+ roce_set_field(rc_sq_wqe->byte_4,
197408 V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
198409 V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
199410 HNS_ROCE_V2_WQE_OP_ATOM_MSK_FETCH_AND_ADD);
200− break;
201− default:
202− roce_set_field(rc_sq_wqe->byte_4,
203− V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
204− V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
205− HNS_ROCE_V2_WQE_OP_MASK);
206− break;
207− }
208−
209− wqe += sizeof(struct hns_roce_v2_rc_send_wqe);
210− dseg = wqe;
211− if (wr->send_flags & IB_SEND_INLINE && wr->num_sge) {
212− if (rc_sq_wqe->msg_len >
213− hr_dev->caps.max_sq_inline) {
214− ret = -EINVAL;
215− *bad_wr = wr;
216− dev_err(dev, "inline len(1-%d)=%d, illegal",
217− rc_sq_wqe->msg_len,
218− hr_dev->caps.max_sq_inline);
219− goto out;
411+ break;
412+ default:
413+ roce_set_field(rc_sq_wqe->byte_4,
414+ V2_RC_SEND_WQE_BYTE_4_OPCODE_M,
415+ V2_RC_SEND_WQE_BYTE_4_OPCODE_S,
416+ HNS_ROCE_V2_WQE_OP_MASK);
417+ break;
220418 }
221419
222− for (i = 0; i < wr->num_sge; i++) {
223− memcpy(wqe, ((void *)wr->sg_list[i].addr),
224− wr->sg_list[i].length);
225− wqe += wr->sg_list[i].length;
226− wqe_sz += wr->sg_list[i].length;
227− }
420+ wqe += sizeof(struct hns_roce_v2_rc_send_wqe);
421+ dseg = wqe;
228422
229− roce_set_bit(rc_sq_wqe->byte_4,
230− V2_RC_SEND_WQE_BYTE_4_INLINE_S, 1);
423+ ret = set_rwqe_data_seg(ibqp, wr, rc_sq_wqe, wqe,
424+ &sge_ind, bad_wr);
425+ if (ret)
426+ goto out;
427+ ind++;
231428 } else {
232− if (wr->num_sge <= 2) {
233− for (i = 0; i < wr->num_sge; i++)
234− set_data_seg_v2(dseg + i,
235− wr->sg_list + i);
236− } else {
237− roce_set_field(rc_sq_wqe->byte_20,
238− V2_RC_SEND_WQE_BYTE_20_MSG_START_SGE_IDX_M,
239− V2_RC_SEND_WQE_BYTE_20_MSG_START_SGE_IDX_S,
240− sge_ind & (qp->sge.sge_cnt - 1));
241−
242− for (i = 0; i < 2; i++)
243− set_data_seg_v2(dseg + i,
244− wr->sg_list + i);
245−
246− dseg = get_send_extend_sge(qp,
247− sge_ind & (qp->sge.sge_cnt - 1));
248−
249− for (i = 0; i < wr->num_sge - 2; i++) {
250− set_data_seg_v2(dseg + i,
251− wr->sg_list + 2 + i);
252− sge_ind++;
253− }
254− }
255−
256− roce_set_field(rc_sq_wqe->byte_16,
257− V2_RC_SEND_WQE_BYTE_16_SGE_NUM_M,
258− V2_RC_SEND_WQE_BYTE_16_SGE_NUM_S,
259− wr->num_sge);
260− wqe_sz += wr->num_sge *
261− sizeof(struct hns_roce_v2_wqe_data_seg);
429+ dev_err(dev, "Illegal qp_type(0x%x)\n", ibqp->qp_type);
430+ spin_unlock_irqrestore(&qp->sq.lock, flags);
431+ return -EOPNOTSUPP;
262432 }
263− ind++;
264433 }
265434
266435 out:
@@ -299,6 +468,7 @@ static int hns_roce_v2_post_recv(struct ib_qp *ibqp, struct ib_recv_wr *wr, · 32 unmodified lines
299468 struct hns_roce_dev *hr_dev = to_hr_dev(ibqp->device);
300469 struct hns_roce_qp *hr_qp = to_hr_qp(ibqp);
301470 struct hns_roce_v2_wqe_data_seg *dseg;
471+ struct hns_roce_rinl_sge *sge_list;
302472 struct device *dev = hr_dev->dev;
303473 struct hns_roce_v2_db rq_db;
304474 unsigned long flags;
@@ -347,6 +517,14 @@ static int hns_roce_v2_post_recv(struct ib_qp *ibqp, struct ib_recv_wr *wr, · 42 unmodified lines
347517 dseg[i].addr = 0;
348518 }
349519
520+ /* rq support inline data */
521+ sge_list = hr_qp->rq_inl_buf.wqe_list[ind].sg_list;
522+ hr_qp->rq_inl_buf.wqe_list[ind].sge_cnt = (u32)wr->num_sge;
523+ for (i = 0; i < wr->num_sge; i++) {
524+ sge_list[i].addr = (void *)(u64)wr->sg_list[i].addr;
525+ sge_list[i].len = wr->sg_list[i].length;
526+ }
527+
350528 hr_qp->rq.wrid[ind] = wr->wr_id;
351529
352530 ind = (ind + 1) & (hr_qp->rq.wqe_cnt - 1);
@@ -908,9 +1086,9 @@ static int hns_roce_v2_profile(struct hns_roce_dev *hr_dev) · 555 unmodified lines
9081086 caps->max_sq_inline = HNS_ROCE_V2_MAX_SQ_INLINE;
9091087 caps->num_uars = HNS_ROCE_V2_UAR_NUM;
9101088 caps->phy_num_uars = HNS_ROCE_V2_PHY_UAR_NUM;
911− caps->num_aeq_vectors = 1;
912− caps->num_comp_vectors = 63;
913− caps->num_other_vectors = 0;
1089+ caps->num_aeq_vectors = HNS_ROCE_V2_AEQE_VEC_NUM;
1090+ caps->num_comp_vectors = HNS_ROCE_V2_COMP_VEC_NUM;
1091+ caps->num_other_vectors = HNS_ROCE_V2_ABNORMAL_VEC_NUM;
9141092 caps->num_mtpts = HNS_ROCE_V2_MAX_MTPT_NUM;
9151093 caps->num_mtt_segs = HNS_ROCE_V2_MAX_MTT_SEGS;
9161094 caps->num_cqe_segs = HNS_ROCE_V2_MAX_CQE_SEGS;
@@ -955,12 +1133,18 @@ static int hns_roce_v2_profile(struct hns_roce_dev *hr_dev) · 38 unmodified lines
9551133 caps->cqe_ba_pg_sz = 0;
9561134 caps->cqe_buf_pg_sz = 0;
9571135 caps->cqe_hop_num = HNS_ROCE_CQE_HOP_NUM;
1136+ caps->eqe_ba_pg_sz = 0;
1137+ caps->eqe_buf_pg_sz = 0;
1138+ caps->eqe_hop_num = HNS_ROCE_EQE_HOP_NUM;
9581139 caps->chunk_sz = HNS_ROCE_V2_TABLE_CHUNK_SIZE;
9591140
9601141 caps->flags = HNS_ROCE_CAP_FLAG_REREG_MR |
961− HNS_ROCE_CAP_FLAG_ROCE_V1_V2;
1142+ HNS_ROCE_CAP_FLAG_ROCE_V1_V2 |
1143+ HNS_ROCE_CAP_FLAG_RQ_INLINE;
9621144 caps->pkey_table_len[0] = 1;
9631145 caps->gid_table_len[0] = HNS_ROCE_V2_GID_INDEX_NUM;
1146+ caps->ceqe_depth = HNS_ROCE_V2_COMP_EQE_NUM;
1147+ caps->aeqe_depth = HNS_ROCE_V2_ASYNC_EQE_NUM;
9641148 caps->local_ca_ack_delay = 0;
9651149 caps->max_mtu = IB_MTU_4096;
9661150
@@ -1382,6 +1566,8 @@ static void hns_roce_v2_write_cqc(struct hns_roce_dev *hr_dev, · 415 unmodified lines
13821566
13831567 roce_set_field(cq_context->byte_4_pg_ceqn, V2_CQC_BYTE_4_CQ_ST_M,
13841568 V2_CQC_BYTE_4_CQ_ST_S, V2_CQ_STATE_VALID);
1569+ roce_set_field(cq_context->byte_4_pg_ceqn, V2_CQC_BYTE_4_ARM_ST_M,
1570+ V2_CQC_BYTE_4_ARM_ST_S, REG_NXT_CEQE);
13851571 roce_set_field(cq_context->byte_4_pg_ceqn, V2_CQC_BYTE_4_SHIFT_M,
13861572 V2_CQC_BYTE_4_SHIFT_S, ilog2((unsigned int)nent));
13871573 roce_set_field(cq_context->byte_4_pg_ceqn, V2_CQC_BYTE_4_CEQN_M,
@@ -1422,6 +1608,15 @@ static void hns_roce_v2_write_cqc(struct hns_roce_dev *hr_dev, · 34 unmodified lines
14221608
14231609 roce_set_field(cq_context->byte_40_cqe_ba, V2_CQC_BYTE_40_CQE_BA_M,
14241610 V2_CQC_BYTE_40_CQE_BA_S, (dma_handle >> (32 + 3)));
1611+
1612+ roce_set_field(cq_context->byte_56_cqe_period_maxcnt,
1613+ V2_CQC_BYTE_56_CQ_MAX_CNT_M,
1614+ V2_CQC_BYTE_56_CQ_MAX_CNT_S,
1615+ HNS_ROCE_V2_CQ_DEFAULT_BURST_NUM);
1616+ roce_set_field(cq_context->byte_56_cqe_period_maxcnt,
1617+ V2_CQC_BYTE_56_CQ_PERIOD_M,
1618+ V2_CQC_BYTE_56_CQ_PERIOD_S,
1619+ HNS_ROCE_V2_CQ_DEFAULT_INTERVAL);
14251620 }
14261621
14271622 static int hns_roce_v2_req_notify_cq(struct ib_cq *ibcq,
@@ -1457,6 +1652,40 @@ static int hns_roce_v2_req_notify_cq(struct ib_cq *ibcq, · 29 unmodified lines
14571652 return 0;
14581653 }
14591654
1655+static int hns_roce_handle_recv_inl_wqe(struct hns_roce_v2_cqe *cqe,
1656+ struct hns_roce_qp **cur_qp,
1657+ struct ib_wc *wc)
1658+{
1659+ struct hns_roce_rinl_sge *sge_list;
1660+ u32 wr_num, wr_cnt, sge_num;
1661+ u32 sge_cnt, data_len, size;
1662+ void *wqe_buf;
1663+
1664+ wr_num = roce_get_field(cqe->byte_4, V2_CQE_BYTE_4_WQE_INDX_M,
1665+ V2_CQE_BYTE_4_WQE_INDX_S) & 0xffff;
1666+ wr_cnt = wr_num & ((*cur_qp)->rq.wqe_cnt - 1);
1667+
1668+ sge_list = (*cur_qp)->rq_inl_buf.wqe_list[wr_cnt].sg_list;
1669+ sge_num = (*cur_qp)->rq_inl_buf.wqe_list[wr_cnt].sge_cnt;
1670+ wqe_buf = get_recv_wqe(*cur_qp, wr_cnt);
1671+ data_len = wc->byte_len;
1672+
1673+ for (sge_cnt = 0; (sge_cnt < sge_num) && (data_len); sge_cnt++) {
1674+ size = min(sge_list[sge_cnt].len, data_len);
1675+ memcpy((void *)sge_list[sge_cnt].addr, wqe_buf, size);
1676+
1677+ data_len -= size;
1678+ wqe_buf += size;
1679+ }
1680+
1681+ if (data_len) {
1682+ wc->status = IB_WC_LOC_LEN_ERR;
1683+ return -EAGAIN;
1684+ }
1685+
1686+ return 0;
1687+}
1688+
14601689 static int hns_roce_v2_poll_one(struct hns_roce_cq *hr_cq,
14611690 struct hns_roce_qp **cur_qp, struct ib_wc *wc)
14621691 {
@@ -1469,6 +1698,7 @@ static int hns_roce_v2_poll_one(struct hns_roce_cq *hr_cq, · 6 unmodified lines
14691698 u32 opcode;
14701699 u32 status;
14711700 int qpn;
1701+ int ret;
14721702
14731703 /* Find cqe according to consumer index */
14741704 cqe = next_cqe_sw_v2(hr_cq);
@@ -1636,7 +1866,7 @@ static int hns_roce_v2_poll_one(struct hns_roce_cq *hr_cq, · 161 unmodified lines
16361866 case HNS_ROCE_V2_OPCODE_RDMA_WRITE_IMM:
16371867 wc->opcode = IB_WC_RECV_RDMA_WITH_IMM;
16381868 wc->wc_flags = IB_WC_WITH_IMM;
1639− wc->ex.imm_data = le32_to_cpu(cqe->rkey_immtdata);
1869+ wc->ex.imm_data = cqe->immtdata;
16401870 break;
16411871 case HNS_ROCE_V2_OPCODE_SEND:
16421872 wc->opcode = IB_WC_RECV;
@@ -1645,18 +1875,29 @@ static int hns_roce_v2_poll_one(struct hns_roce_cq *hr_cq, · 2 unmodified lines
16451875 case HNS_ROCE_V2_OPCODE_SEND_WITH_IMM:
16461876 wc->opcode = IB_WC_RECV;
16471877 wc->wc_flags = IB_WC_WITH_IMM;
1648− wc->ex.imm_data = le32_to_cpu(cqe->rkey_immtdata);
1878+ wc->ex.imm_data = cqe->immtdata;
16491879 break;
16501880 case HNS_ROCE_V2_OPCODE_SEND_WITH_INV:
16511881 wc->opcode = IB_WC_RECV;
16521882 wc->wc_flags = IB_WC_WITH_INVALIDATE;
1653− wc->ex.invalidate_rkey = cqe->rkey_immtdata;
1883+ wc->ex.invalidate_rkey = le32_to_cpu(cqe->rkey);
16541884 break;
16551885 default:
16561886 wc->status = IB_WC_GENERAL_ERR;
16571887 break;
16581888 }
16591889
1890+ if ((wc->qp->qp_type == IB_QPT_RC ||
1891+ wc->qp->qp_type == IB_QPT_UC) &&
1892+ (opcode == HNS_ROCE_V2_OPCODE_SEND ||
1893+ opcode == HNS_ROCE_V2_OPCODE_SEND_WITH_IMM ||
1894+ opcode == HNS_ROCE_V2_OPCODE_SEND_WITH_INV) &&
1895+ (roce_get_bit(cqe->byte_4, V2_CQE_BYTE_4_RQ_INLINE_S))) {
1896+ ret = hns_roce_handle_recv_inl_wqe(cqe, cur_qp, wc);
1897+ if (ret)
1898+ return -EAGAIN;
1899+ }
1900+
16601901 /* Update tail pointer, record wr_id */
16611902 wq = &(*cur_qp)->rq;
16621903 wc->wr_id = wq->wrid[wq->tail & (wq->wqe_cnt - 1)];
@@ -1670,6 +1911,21 @@ static int hns_roce_v2_poll_one(struct hns_roce_cq *hr_cq, · 7 unmodified lines
16701911 wc->wc_flags |= (roce_get_bit(cqe->byte_32,
16711912 V2_CQE_BYTE_32_GRH_S) ?
16721913 IB_WC_GRH : 0);
1914+ wc->port_num = roce_get_field(cqe->byte_32,
1915+ V2_CQE_BYTE_32_PORTN_M, V2_CQE_BYTE_32_PORTN_S);
1916+ wc->pkey_index = 0;
1917+ memcpy(wc->smac, cqe->smac, 4);
1918+ wc->smac[4] = roce_get_field(cqe->byte_28,
1919+ V2_CQE_BYTE_28_SMAC_4_M,
1920+ V2_CQE_BYTE_28_SMAC_4_S);
1921+ wc->smac[5] = roce_get_field(cqe->byte_28,
1922+ V2_CQE_BYTE_28_SMAC_5_M,
1923+ V2_CQE_BYTE_28_SMAC_5_S);
1924+ wc->vlan_id = 0xffff;
1925+ wc->wc_flags |= (IB_WC_WITH_VLAN | IB_WC_WITH_SMAC);
1926+ wc->network_hdr_type = roce_get_field(cqe->byte_28,
1927+ V2_CQE_BYTE_28_PORT_TYPE_M,
1928+ V2_CQE_BYTE_28_PORT_TYPE_S);
16731929 }
16741930
16751931 return 0;
@@ -1859,8 +2115,39 @@ static int hns_roce_v2_qp_modify(struct hns_roce_dev *hr_dev, · 183 unmodified lines
18592115 return ret;
18602116 }
18612117
2118+static void set_access_flags(struct hns_roce_qp *hr_qp,
2119+ struct hns_roce_v2_qp_context *context,
2120+ struct hns_roce_v2_qp_context *qpc_mask,
2121+ const struct ib_qp_attr *attr, int attr_mask)
2122+{
2123+ u8 dest_rd_atomic;
2124+ u32 access_flags;
2125+
2126+ dest_rd_atomic = !!(attr_mask & IB_QP_MAX_DEST_RD_ATOMIC) ?
2127+ attr->max_dest_rd_atomic : hr_qp->resp_depth;
2128+
2129+ access_flags = !!(attr_mask & IB_QP_ACCESS_FLAGS) ?
2130+ attr->qp_access_flags : hr_qp->atomic_rd_en;
2131+
2132+ if (!dest_rd_atomic)
2133+ access_flags &= IB_ACCESS_REMOTE_WRITE;
2134+
2135+ roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_RRE_S,
2136+ !!(access_flags & IB_ACCESS_REMOTE_READ));
2137+ roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_RRE_S, 0);
2138+
2139+ roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_RWE_S,
2140+ !!(access_flags & IB_ACCESS_REMOTE_WRITE));
2141+ roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_RWE_S, 0);
2142+
2143+ roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
2144+ !!(access_flags & IB_ACCESS_REMOTE_ATOMIC));
2145+ roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S, 0);
2146+}
2147+
18622148 static void modify_qp_reset_to_init(struct ib_qp *ibqp,
18632149 const struct ib_qp_attr *attr,
2150+ int attr_mask,
18642151 struct hns_roce_v2_qp_context *context,
18652152 struct hns_roce_v2_qp_context *qpc_mask)
18662153 {
@@ -1877,9 +2164,18 @@ static void modify_qp_reset_to_init(struct ib_qp *ibqp, · 10 unmodified lines
18772164 roce_set_field(qpc_mask->byte_4_sqpn_tst, V2_QPC_BYTE_4_TST_M,
18782165 V2_QPC_BYTE_4_TST_S, 0);
18792166
1880− roce_set_field(context->byte_4_sqpn_tst, V2_QPC_BYTE_4_SGE_SHIFT_M,
1881− V2_QPC_BYTE_4_SGE_SHIFT_S, hr_qp->sq.max_gs > 2 ?
1882− ilog2((unsigned int)hr_qp->sge.sge_cnt) : 0);
2167+ if (ibqp->qp_type == IB_QPT_GSI)
2168+ roce_set_field(context->byte_4_sqpn_tst,
2169+ V2_QPC_BYTE_4_SGE_SHIFT_M,
2170+ V2_QPC_BYTE_4_SGE_SHIFT_S,
2171+ ilog2((unsigned int)hr_qp->sge.sge_cnt));
2172+ else
2173+ roce_set_field(context->byte_4_sqpn_tst,
2174+ V2_QPC_BYTE_4_SGE_SHIFT_M,
2175+ V2_QPC_BYTE_4_SGE_SHIFT_S,
2176+ hr_qp->sq.max_gs > 2 ?
2177+ ilog2((unsigned int)hr_qp->sge.sge_cnt) : 0);
2178+
18832179 roce_set_field(qpc_mask->byte_4_sqpn_tst, V2_QPC_BYTE_4_SGE_SHIFT_M,
18842180 V2_QPC_BYTE_4_SGE_SHIFT_S, 0);
18852181
@@ -1944,18 +2240,13 @@ static void modify_qp_reset_to_init(struct ib_qp *ibqp, · 58 unmodified lines
19442240 roce_set_bit(qpc_mask->byte_28_at_fl, V2_QPC_BYTE_28_CNP_TX_FLAG_S, 0);
19452241 roce_set_bit(qpc_mask->byte_28_at_fl, V2_QPC_BYTE_28_CE_FLAG_S, 0);
19462242
1947− roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_RRE_S,
1948− !!(attr->qp_access_flags & IB_ACCESS_REMOTE_READ));
1949− roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_RRE_S, 0);
1950−
1951− roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_RWE_S,
1952− !!(attr->qp_access_flags & IB_ACCESS_REMOTE_WRITE));
1953− roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_RWE_S, 0);
1954−
1955− roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
1956− !!(attr->qp_access_flags & IB_ACCESS_REMOTE_ATOMIC));
1957− roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S, 0);
2243+ if (attr_mask & IB_QP_QKEY) {
2244+ context->qkey_xrcd = attr->qkey;
2245+ qpc_mask->qkey_xrcd = 0;
2246+ hr_qp->qkey = attr->qkey;
2247+ }
19582248
2249+ roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_RQIE_S, 1);
19592250 roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_RQIE_S, 0);
19602251
19612252 roce_set_field(context->byte_80_rnr_rx_cqn, V2_QPC_BYTE_80_RX_CQN_M,
@@ -2176,9 +2467,17 @@ static void modify_qp_init_to_init(struct ib_qp *ibqp, · 214 unmodified lines
21762467 roce_set_field(qpc_mask->byte_4_sqpn_tst, V2_QPC_BYTE_4_TST_M,
21772468 V2_QPC_BYTE_4_TST_S, 0);
21782469
2179− roce_set_field(context->byte_4_sqpn_tst, V2_QPC_BYTE_4_SGE_SHIFT_M,
2180− V2_QPC_BYTE_4_SGE_SHIFT_S, hr_qp->sq.max_gs > 2 ?
2181− ilog2((unsigned int)hr_qp->sge.sge_cnt) : 0);
2470+ if (ibqp->qp_type == IB_QPT_GSI)
2471+ roce_set_field(context->byte_4_sqpn_tst,
2472+ V2_QPC_BYTE_4_SGE_SHIFT_M,
2473+ V2_QPC_BYTE_4_SGE_SHIFT_S,
2474+ ilog2((unsigned int)hr_qp->sge.sge_cnt));
2475+ else
2476+ roce_set_field(context->byte_4_sqpn_tst,
2477+ V2_QPC_BYTE_4_SGE_SHIFT_M,
2478+ V2_QPC_BYTE_4_SGE_SHIFT_S, hr_qp->sq.max_gs > 2 ?
2479+ ilog2((unsigned int)hr_qp->sge.sge_cnt) : 0);
2480+
21822481 roce_set_field(qpc_mask->byte_4_sqpn_tst, V2_QPC_BYTE_4_SGE_SHIFT_M,
21832482 V2_QPC_BYTE_4_SGE_SHIFT_S, 0);
21842483
@@ -2239,7 +2538,7 @@ static void modify_qp_init_to_init(struct ib_qp *ibqp, · 54 unmodified lines
22392538 V2_QPC_BYTE_80_RX_CQN_S, 0);
22402539
22412540 roce_set_field(context->byte_252_err_txcqn, V2_QPC_BYTE_252_TX_CQN_M,
2242− V2_QPC_BYTE_252_TX_CQN_S, to_hr_cq(ibqp->recv_cq)->cqn);
2541+ V2_QPC_BYTE_252_TX_CQN_S, to_hr_cq(ibqp->send_cq)->cqn);
22432542 roce_set_field(qpc_mask->byte_252_err_txcqn, V2_QPC_BYTE_252_TX_CQN_M,
22442543 V2_QPC_BYTE_252_TX_CQN_S, 0);
22452544
@@ -2255,10 +2554,10 @@ static void modify_qp_init_to_init(struct ib_qp *ibqp, · 9 unmodified lines
22552554 V2_QPC_BYTE_76_SRQN_M, V2_QPC_BYTE_76_SRQN_S, 0);
22562555 }
22572556
2258− if (attr_mask & IB_QP_PKEY_INDEX)
2259− context->qkey_xrcd = attr->pkey_index;
2260− else
2261− context->qkey_xrcd = hr_qp->pkey_index;
2557+ if (attr_mask & IB_QP_QKEY) {
2558+ context->qkey_xrcd = attr->qkey;
2559+ qpc_mask->qkey_xrcd = 0;
2560+ }
22622561
22632562 roce_set_field(context->byte_4_sqpn_tst, V2_QPC_BYTE_4_SQPN_M,
22642563 V2_QPC_BYTE_4_SQPN_S, hr_qp->qpn);
@@ -2354,7 +2653,8 @@ static int modify_qp_init_to_rtr(struct ib_qp *ibqp, · 89 unmodified lines
23542653 roce_set_field(context->byte_20_smac_sgid_idx,
23552654 V2_QPC_BYTE_20_SGE_HOP_NUM_M,
23562655 V2_QPC_BYTE_20_SGE_HOP_NUM_S,
2357− hr_qp->sq.max_gs > 2 ? hr_dev->caps.mtt_hop_num : 0);
2656+ ((ibqp->qp_type == IB_QPT_GSI) || hr_qp->sq.max_gs > 2) ?
2657+ hr_dev->caps.mtt_hop_num : 0);
23582658 roce_set_field(qpc_mask->byte_20_smac_sgid_idx,
23592659 V2_QPC_BYTE_20_SGE_HOP_NUM_M,
23602660 V2_QPC_BYTE_20_SGE_HOP_NUM_S, 0);
@@ -2463,11 +2763,14 @@ static int modify_qp_init_to_rtr(struct ib_qp *ibqp, · 102 unmodified lines
24632763 roce_set_bit(qpc_mask->byte_28_at_fl, V2_QPC_BYTE_28_LBI_S, 0);
24642764 }
24652765
2466− roce_set_field(context->byte_140_raq, V2_QPC_BYTE_140_RR_MAX_M,
2467− V2_QPC_BYTE_140_RR_MAX_S,
2468− ilog2((unsigned int)attr->max_dest_rd_atomic));
2469− roce_set_field(qpc_mask->byte_140_raq, V2_QPC_BYTE_140_RR_MAX_M,
2470− V2_QPC_BYTE_140_RR_MAX_S, 0);
2766+ if ((attr_mask & IB_QP_MAX_DEST_RD_ATOMIC) &&
2767+ attr->max_dest_rd_atomic) {
2768+ roce_set_field(context->byte_140_raq, V2_QPC_BYTE_140_RR_MAX_M,
2769+ V2_QPC_BYTE_140_RR_MAX_S,
2770+ fls(attr->max_dest_rd_atomic - 1));
2771+ roce_set_field(qpc_mask->byte_140_raq, V2_QPC_BYTE_140_RR_MAX_M,
2772+ V2_QPC_BYTE_140_RR_MAX_S, 0);
2773+ }
24712774
24722775 roce_set_field(context->byte_56_dqpn_err, V2_QPC_BYTE_56_DQPN_M,
24732776 V2_QPC_BYTE_56_DQPN_S, attr->dest_qp_num);
@@ -2511,8 +2814,13 @@ static int modify_qp_init_to_rtr(struct ib_qp *ibqp, · 37 unmodified lines
25112814 roce_set_field(qpc_mask->byte_24_mtu_tc, V2_QPC_BYTE_24_TC_M,
25122815 V2_QPC_BYTE_24_TC_S, 0);
25132816
2514− roce_set_field(context->byte_24_mtu_tc, V2_QPC_BYTE_24_MTU_M,
2515− V2_QPC_BYTE_24_MTU_S, attr->path_mtu);
2817+ if (ibqp->qp_type == IB_QPT_GSI || ibqp->qp_type == IB_QPT_UD)
2818+ roce_set_field(context->byte_24_mtu_tc, V2_QPC_BYTE_24_MTU_M,
2819+ V2_QPC_BYTE_24_MTU_S, IB_MTU_4096);
2820+ else
2821+ roce_set_field(context->byte_24_mtu_tc, V2_QPC_BYTE_24_MTU_M,
2822+ V2_QPC_BYTE_24_MTU_S, attr->path_mtu);
2823+
25162824 roce_set_field(qpc_mask->byte_24_mtu_tc, V2_QPC_BYTE_24_MTU_M,
25172825 V2_QPC_BYTE_24_MTU_S, 0);
25182826
@@ -2557,12 +2865,6 @@ static int modify_qp_init_to_rtr(struct ib_qp *ibqp, · 38 unmodified lines
25572865 V2_QPC_BYTE_168_LP_SGEN_INI_M,
25582866 V2_QPC_BYTE_168_LP_SGEN_INI_S, 0);
25592867
2560− roce_set_field(context->byte_208_irrl, V2_QPC_BYTE_208_SR_MAX_M,
2561− V2_QPC_BYTE_208_SR_MAX_S,
2562− ilog2((unsigned int)attr->max_rd_atomic));
2563− roce_set_field(qpc_mask->byte_208_irrl, V2_QPC_BYTE_208_SR_MAX_M,
2564− V2_QPC_BYTE_208_SR_MAX_S, 0);
2565−
25662868 roce_set_field(context->byte_28_at_fl, V2_QPC_BYTE_28_SL_M,
25672869 V2_QPC_BYTE_28_SL_S, rdma_ah_get_sl(&attr->ah_attr));
25682870 roce_set_field(qpc_mask->byte_28_at_fl, V2_QPC_BYTE_28_SL_M,
@@ -2625,13 +2927,14 @@ static int modify_qp_rtr_to_rts(struct ib_qp *ibqp, · 56 unmodified lines
26252927 V2_QPC_BYTE_168_SQ_CUR_BLK_ADDR_S, 0);
26262928
26272929 page_size = 1 << (hr_dev->caps.mtt_buf_pg_sz + PAGE_SHIFT);
2628− context->sq_cur_sge_blk_addr = hr_qp->sq.max_gs > 2 ?
2930+ context->sq_cur_sge_blk_addr =
2931+ ((ibqp->qp_type == IB_QPT_GSI) || hr_qp->sq.max_gs > 2) ?
26292932 ((u32)(mtts[hr_qp->sge.offset / page_size]
26302933 >> PAGE_ADDR_SHIFT)) : 0;
26312934 roce_set_field(context->byte_184_irrl_idx,
26322935 V2_QPC_BYTE_184_SQ_CUR_SGE_BLK_ADDR_M,
26332936 V2_QPC_BYTE_184_SQ_CUR_SGE_BLK_ADDR_S,
2634− hr_qp->sq.max_gs > 2 ?
2937+ ((ibqp->qp_type == IB_QPT_GSI) || hr_qp->sq.max_gs > 2) ?
26352938 (mtts[hr_qp->sge.offset / page_size] >>
26362939 (32 + PAGE_ADDR_SHIFT)) : 0);
26372940 qpc_mask->sq_cur_sge_blk_addr = 0;
@@ -2766,6 +3069,14 @@ static int modify_qp_rtr_to_rts(struct ib_qp *ibqp, · 128 unmodified lines
27663069 roce_set_field(qpc_mask->byte_196_sq_psn, V2_QPC_BYTE_196_SQ_MAX_PSN_M,
27673070 V2_QPC_BYTE_196_SQ_MAX_PSN_S, 0);
27683071
3072+ if ((attr_mask & IB_QP_MAX_QP_RD_ATOMIC) && attr->max_rd_atomic) {
3073+ roce_set_field(context->byte_208_irrl, V2_QPC_BYTE_208_SR_MAX_M,
3074+ V2_QPC_BYTE_208_SR_MAX_S,
3075+ fls(attr->max_rd_atomic - 1));
3076+ roce_set_field(qpc_mask->byte_208_irrl,
3077+ V2_QPC_BYTE_208_SR_MAX_M,
3078+ V2_QPC_BYTE_208_SR_MAX_S, 0);
3079+ }
27693080 return 0;
27703081 }
27713082
@@ -2794,7 +3105,8 @@ static int hns_roce_v2_modify_qp(struct ib_qp *ibqp, · 22 unmodified lines
27943105 */
27953106 memset(qpc_mask, 0xff, sizeof(*qpc_mask));
27963107 if (cur_state == IB_QPS_RESET && new_state == IB_QPS_INIT) {
2797− modify_qp_reset_to_init(ibqp, attr, context, qpc_mask);
3108+ modify_qp_reset_to_init(ibqp, attr, attr_mask, context,
3109+ qpc_mask);
27983110 } else if (cur_state == IB_QPS_INIT && new_state == IB_QPS_INIT) {
27993111 modify_qp_init_to_init(ibqp, attr, attr_mask, context,
28003112 qpc_mask);
@@ -2829,6 +3141,9 @@ static int hns_roce_v2_modify_qp(struct ib_qp *ibqp, · 28 unmodified lines
28293141 goto out;
28303142 }
28313143
3144+ if (attr_mask & (IB_QP_ACCESS_FLAGS | IB_QP_MAX_DEST_RD_ATOMIC))
3145+ set_access_flags(hr_qp, context, qpc_mask, attr, attr_mask);
3146+
28323147 /* Every status migrate must change state */
28333148 roce_set_field(context->byte_60_qpst_mapid, V2_QPC_BYTE_60_QP_ST_M,
28343149 V2_QPC_BYTE_60_QP_ST_S, new_state);
@@ -2845,6 +3160,9 @@ static int hns_roce_v2_modify_qp(struct ib_qp *ibqp, · 10 unmodified lines
28453160
28463161 hr_qp->state = new_state;
28473162
3163+ if (attr_mask & IB_QP_ACCESS_FLAGS)
3164+ hr_qp->atomic_rd_en = attr->qp_access_flags;
3165+
28483166 if (attr_mask & IB_QP_MAX_DEST_RD_ATOMIC)
28493167 hr_qp->resp_depth = attr->max_dest_rd_atomic;
28503168 if (attr_mask & IB_QP_PORT) {
@@ -3098,6 +3416,11 @@ static int hns_roce_v2_destroy_qp_common(struct hns_roce_dev *hr_dev, · 247 unmodified lines
30983416 hns_roce_buf_free(hr_dev, hr_qp->buff_size, &hr_qp->hr_buf);
30993417 }
31003418
3419+ if (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_RQ_INLINE) {
3420+ kfree(hr_qp->rq_inl_buf.wqe_list[0].sg_list);
3421+ kfree(hr_qp->rq_inl_buf.wqe_list);
3422+ }
3423+
31013424 return 0;
31023425 }
31033426
@@ -3162,6 +3485,1146 @@ static int hns_roce_v2_modify_cq(struct ib_cq *cq, u16 cq_count, u16 cq_period) · 58 unmodified lines
31623485 return ret;
31633486 }
31643487
3488+static void set_eq_cons_index_v2(struct hns_roce_eq *eq)
3489+{
3490+ u32 doorbell[2];
3491+
3492+ doorbell[0] = 0;
3493+ doorbell[1] = 0;
3494+
3495+ if (eq->type_flag == HNS_ROCE_AEQ) {
3496+ roce_set_field(doorbell[0], HNS_ROCE_V2_EQ_DB_CMD_M,
3497+ HNS_ROCE_V2_EQ_DB_CMD_S,
3498+ eq->arm_st == HNS_ROCE_V2_EQ_ALWAYS_ARMED ?
3499+ HNS_ROCE_EQ_DB_CMD_AEQ :
3500+ HNS_ROCE_EQ_DB_CMD_AEQ_ARMED);
3501+ } else {
3502+ roce_set_field(doorbell[0], HNS_ROCE_V2_EQ_DB_TAG_M,
3503+ HNS_ROCE_V2_EQ_DB_TAG_S, eq->eqn);
3504+
3505+ roce_set_field(doorbell[0], HNS_ROCE_V2_EQ_DB_CMD_M,
3506+ HNS_ROCE_V2_EQ_DB_CMD_S,
3507+ eq->arm_st == HNS_ROCE_V2_EQ_ALWAYS_ARMED ?
3508+ HNS_ROCE_EQ_DB_CMD_CEQ :
3509+ HNS_ROCE_EQ_DB_CMD_CEQ_ARMED);
3510+ }
3511+
3512+ roce_set_field(doorbell[1], HNS_ROCE_V2_EQ_DB_PARA_M,
3513+ HNS_ROCE_V2_EQ_DB_PARA_S,
3514+ (eq->cons_index & HNS_ROCE_V2_CONS_IDX_M));
3515+
3516+ hns_roce_write64_k(doorbell, eq->doorbell);
3517+}
3518+
3519+static void hns_roce_v2_wq_catas_err_handle(struct hns_roce_dev *hr_dev,
3520+ struct hns_roce_aeqe *aeqe,
3521+ u32 qpn)
3522+{
3523+ struct device *dev = hr_dev->dev;
3524+ int sub_type;
3525+
3526+ dev_warn(dev, "Local work queue catastrophic error.\n");
3527+ sub_type = roce_get_field(aeqe->asyn, HNS_ROCE_V2_AEQE_SUB_TYPE_M,
3528+ HNS_ROCE_V2_AEQE_SUB_TYPE_S);
3529+ switch (sub_type) {
3530+ case HNS_ROCE_LWQCE_QPC_ERROR:
3531+ dev_warn(dev, "QP %d, QPC error.\n", qpn);
3532+ break;
3533+ case HNS_ROCE_LWQCE_MTU_ERROR:
3534+ dev_warn(dev, "QP %d, MTU error.\n", qpn);
3535+ break;
3536+ case HNS_ROCE_LWQCE_WQE_BA_ADDR_ERROR:
3537+ dev_warn(dev, "QP %d, WQE BA addr error.\n", qpn);
3538+ break;
3539+ case HNS_ROCE_LWQCE_WQE_ADDR_ERROR:
3540+ dev_warn(dev, "QP %d, WQE addr error.\n", qpn);
3541+ break;
3542+ case HNS_ROCE_LWQCE_SQ_WQE_SHIFT_ERROR:
3543+ dev_warn(dev, "QP %d, WQE shift error.\n", qpn);
3544+ break;
3545+ default:
3546+ dev_err(dev, "Unhandled sub_event type %d.\n", sub_type);
3547+ break;
3548+ }
3549+}
3550+
3551+static void hns_roce_v2_local_wq_access_err_handle(struct hns_roce_dev *hr_dev,
3552+ struct hns_roce_aeqe *aeqe, u32 qpn)
3553+{
3554+ struct device *dev = hr_dev->dev;
3555+ int sub_type;
3556+
3557+ dev_warn(dev, "Local access violation work queue error.\n");
3558+ sub_type = roce_get_field(aeqe->asyn, HNS_ROCE_V2_AEQE_SUB_TYPE_M,
3559+ HNS_ROCE_V2_AEQE_SUB_TYPE_S);
3560+ switch (sub_type) {
3561+ case HNS_ROCE_LAVWQE_R_KEY_VIOLATION:
3562+ dev_warn(dev, "QP %d, R_key violation.\n", qpn);
3563+ break;
3564+ case HNS_ROCE_LAVWQE_LENGTH_ERROR:
3565+ dev_warn(dev, "QP %d, length error.\n", qpn);
3566+ break;
3567+ case HNS_ROCE_LAVWQE_VA_ERROR:
3568+ dev_warn(dev, "QP %d, VA error.\n", qpn);
3569+ break;
3570+ case HNS_ROCE_LAVWQE_PD_ERROR:
3571+ dev_err(dev, "QP %d, PD error.\n", qpn);
3572+ break;
3573+ case HNS_ROCE_LAVWQE_RW_ACC_ERROR:
3574+ dev_warn(dev, "QP %d, rw acc error.\n", qpn);
3575+ break;
3576+ case HNS_ROCE_LAVWQE_KEY_STATE_ERROR:
3577+ dev_warn(dev, "QP %d, key state error.\n", qpn);
3578+ break;
3579+ case HNS_ROCE_LAVWQE_MR_OPERATION_ERROR:
3580+ dev_warn(dev, "QP %d, MR operation error.\n", qpn);
3581+ break;
3582+ default:
3583+ dev_err(dev, "Unhandled sub_event type %d.\n", sub_type);
3584+ break;
3585+ }
3586+}
3587+
3588+static void hns_roce_v2_qp_err_handle(struct hns_roce_dev *hr_dev,
3589+ struct hns_roce_aeqe *aeqe,
3590+ int event_type)
3591+{
3592+ struct device *dev = hr_dev->dev;
3593+ u32 qpn;
3594+
3595+ qpn = roce_get_field(aeqe->event.qp_event.qp,
3596+ HNS_ROCE_V2_AEQE_EVENT_QUEUE_NUM_M,
3597+ HNS_ROCE_V2_AEQE_EVENT_QUEUE_NUM_S);
3598+
3599+ switch (event_type) {
3600+ case HNS_ROCE_EVENT_TYPE_COMM_EST:
3601+ dev_warn(dev, "Communication established.\n");
3602+ break;
3603+ case HNS_ROCE_EVENT_TYPE_SQ_DRAINED:
3604+ dev_warn(dev, "Send queue drained.\n");
3605+ break;
3606+ case HNS_ROCE_EVENT_TYPE_WQ_CATAS_ERROR:
3607+ hns_roce_v2_wq_catas_err_handle(hr_dev, aeqe, qpn);
3608+ break;
3609+ case HNS_ROCE_EVENT_TYPE_INV_REQ_LOCAL_WQ_ERROR:
3610+ dev_warn(dev, "Invalid request local work queue error.\n");
3611+ break;
3612+ case HNS_ROCE_EVENT_TYPE_LOCAL_WQ_ACCESS_ERROR:
3613+ hns_roce_v2_local_wq_access_err_handle(hr_dev, aeqe, qpn);
3614+ break;
3615+ default:
3616+ break;
3617+ }
3618+
3619+ hns_roce_qp_event(hr_dev, qpn, event_type);
3620+}
3621+
3622+static void hns_roce_v2_cq_err_handle(struct hns_roce_dev *hr_dev,
3623+ struct hns_roce_aeqe *aeqe,
3624+ int event_type)
3625+{
3626+ struct device *dev = hr_dev->dev;
3627+ u32 cqn;
3628+
3629+ cqn = roce_get_field(aeqe->event.cq_event.cq,
3630+ HNS_ROCE_V2_AEQE_EVENT_QUEUE_NUM_M,
3631+ HNS_ROCE_V2_AEQE_EVENT_QUEUE_NUM_S);
3632+
3633+ switch (event_type) {
3634+ case HNS_ROCE_EVENT_TYPE_CQ_ACCESS_ERROR:
3635+ dev_warn(dev, "CQ 0x%x access err.\n", cqn);
3636+ break;
3637+ case HNS_ROCE_EVENT_TYPE_CQ_OVERFLOW:
3638+ dev_warn(dev, "CQ 0x%x overflow\n", cqn);
3639+ break;
3640+ default:
3641+ break;
3642+ }
3643+
3644+ hns_roce_cq_event(hr_dev, cqn, event_type);
3645+}
3646+
3647+static struct hns_roce_aeqe *get_aeqe_v2(struct hns_roce_eq *eq, u32 entry)
3648+{
3649+ u32 buf_chk_sz;
3650+ unsigned long off;
3651+
3652+ buf_chk_sz = 1 << (eq->eqe_buf_pg_sz + PAGE_SHIFT);
3653+ off = (entry & (eq->entries - 1)) * HNS_ROCE_AEQ_ENTRY_SIZE;
3654+
3655+ return (struct hns_roce_aeqe *)((char *)(eq->buf_list->buf) +
3656+ off % buf_chk_sz);
3657+}
3658+
3659+static struct hns_roce_aeqe *mhop_get_aeqe(struct hns_roce_eq *eq, u32 entry)
3660+{
3661+ u32 buf_chk_sz;
3662+ unsigned long off;
3663+
3664+ buf_chk_sz = 1 << (eq->eqe_buf_pg_sz + PAGE_SHIFT);
3665+
3666+ off = (entry & (eq->entries - 1)) * HNS_ROCE_AEQ_ENTRY_SIZE;
3667+
3668+ if (eq->hop_num == HNS_ROCE_HOP_NUM_0)
3669+ return (struct hns_roce_aeqe *)((u8 *)(eq->bt_l0) +
3670+ off % buf_chk_sz);
3671+ else
3672+ return (struct hns_roce_aeqe *)((u8 *)
3673+ (eq->buf[off / buf_chk_sz]) + off % buf_chk_sz);
3674+}
3675+
3676+static struct hns_roce_aeqe *next_aeqe_sw_v2(struct hns_roce_eq *eq)
3677+{
3678+ struct hns_roce_aeqe *aeqe;
3679+
3680+ if (!eq->hop_num)
3681+ aeqe = get_aeqe_v2(eq, eq->cons_index);
3682+ else
3683+ aeqe = mhop_get_aeqe(eq, eq->cons_index);
3684+
3685+ return (roce_get_bit(aeqe->asyn, HNS_ROCE_V2_AEQ_AEQE_OWNER_S) ^
3686+ !!(eq->cons_index & eq->entries)) ? aeqe : NULL;
3687+}
3688+
3689+static int hns_roce_v2_aeq_int(struct hns_roce_dev *hr_dev,
3690+ struct hns_roce_eq *eq)
3691+{
3692+ struct device *dev = hr_dev->dev;
3693+ struct hns_roce_aeqe *aeqe;
3694+ int aeqe_found = 0;
3695+ int event_type;
3696+
3697+ while ((aeqe = next_aeqe_sw_v2(eq))) {
3698+
3699+ /* Make sure we read AEQ entry after we have checked the
3700+ * ownership bit
3701+ */
3702+ dma_rmb();
3703+
3704+ event_type = roce_get_field(aeqe->asyn,
3705+ HNS_ROCE_V2_AEQE_EVENT_TYPE_M,
3706+ HNS_ROCE_V2_AEQE_EVENT_TYPE_S);
3707+
3708+ switch (event_type) {
3709+ case HNS_ROCE_EVENT_TYPE_PATH_MIG:
3710+ dev_warn(dev, "Path migrated succeeded.\n");
3711+ break;
3712+ case HNS_ROCE_EVENT_TYPE_PATH_MIG_FAILED:
3713+ dev_warn(dev, "Path migration failed.\n");
3714+ break;
3715+ case HNS_ROCE_EVENT_TYPE_COMM_EST:
3716+ case HNS_ROCE_EVENT_TYPE_SQ_DRAINED:
3717+ case HNS_ROCE_EVENT_TYPE_WQ_CATAS_ERROR:
3718+ case HNS_ROCE_EVENT_TYPE_INV_REQ_LOCAL_WQ_ERROR:
3719+ case HNS_ROCE_EVENT_TYPE_LOCAL_WQ_ACCESS_ERROR:
3720+ hns_roce_v2_qp_err_handle(hr_dev, aeqe, event_type);
3721+ break;
3722+ case HNS_ROCE_EVENT_TYPE_SRQ_LIMIT_REACH:
3723+ case HNS_ROCE_EVENT_TYPE_SRQ_LAST_WQE_REACH:
3724+ case HNS_ROCE_EVENT_TYPE_SRQ_CATAS_ERROR:
3725+ dev_warn(dev, "SRQ not support.\n");
3726+ break;
3727+ case HNS_ROCE_EVENT_TYPE_CQ_ACCESS_ERROR:
3728+ case HNS_ROCE_EVENT_TYPE_CQ_OVERFLOW:
3729+ hns_roce_v2_cq_err_handle(hr_dev, aeqe, event_type);
3730+ break;
3731+ case HNS_ROCE_EVENT_TYPE_DB_OVERFLOW:
3732+ dev_warn(dev, "DB overflow.\n");
3733+ break;
3734+ case HNS_ROCE_EVENT_TYPE_MB:
3735+ hns_roce_cmd_event(hr_dev,
3736+ le16_to_cpu(aeqe->event.cmd.token),
3737+ aeqe->event.cmd.status,
3738+ le64_to_cpu(aeqe->event.cmd.out_param));
3739+ break;
3740+ case HNS_ROCE_EVENT_TYPE_CEQ_OVERFLOW:
3741+ dev_warn(dev, "CEQ overflow.\n");
3742+ break;
3743+ case HNS_ROCE_EVENT_TYPE_FLR:
3744+ dev_warn(dev, "Function level reset.\n");
3745+ break;
3746+ default:
3747+ dev_err(dev, "Unhandled event %d on EQ %d at idx %u.\n",
3748+ event_type, eq->eqn, eq->cons_index);
3749+ break;
3750+ };
3751+
3752+ ++eq->cons_index;
3753+ aeqe_found = 1;
3754+
3755+ if (eq->cons_index > (2 * eq->entries - 1)) {
3756+ dev_warn(dev, "cons_index overflow, set back to 0.\n");
3757+ eq->cons_index = 0;
3758+ }
3759+ }
3760+
3761+ set_eq_cons_index_v2(eq);
3762+ return aeqe_found;
3763+}
3764+
3765+static struct hns_roce_ceqe *get_ceqe_v2(struct hns_roce_eq *eq, u32 entry)
3766+{
3767+ u32 buf_chk_sz;
3768+ unsigned long off;
3769+
3770+ buf_chk_sz = 1 << (eq->eqe_buf_pg_sz + PAGE_SHIFT);
3771+ off = (entry & (eq->entries - 1)) * HNS_ROCE_CEQ_ENTRY_SIZE;
3772+
3773+ return (struct hns_roce_ceqe *)((char *)(eq->buf_list->buf) +
3774+ off % buf_chk_sz);
3775+}
3776+
3777+static struct hns_roce_ceqe *mhop_get_ceqe(struct hns_roce_eq *eq, u32 entry)
3778+{
3779+ u32 buf_chk_sz;
3780+ unsigned long off;
3781+
3782+ buf_chk_sz = 1 << (eq->eqe_buf_pg_sz + PAGE_SHIFT);
3783+
3784+ off = (entry & (eq->entries - 1)) * HNS_ROCE_CEQ_ENTRY_SIZE;
3785+
3786+ if (eq->hop_num == HNS_ROCE_HOP_NUM_0)
3787+ return (struct hns_roce_ceqe *)((u8 *)(eq->bt_l0) +
3788+ off % buf_chk_sz);
3789+ else
3790+ return (struct hns_roce_ceqe *)((u8 *)(eq->buf[off /
3791+ buf_chk_sz]) + off % buf_chk_sz);
3792+}
3793+
3794+static struct hns_roce_ceqe *next_ceqe_sw_v2(struct hns_roce_eq *eq)
3795+{
3796+ struct hns_roce_ceqe *ceqe;
3797+
3798+ if (!eq->hop_num)
3799+ ceqe = get_ceqe_v2(eq, eq->cons_index);
3800+ else
3801+ ceqe = mhop_get_ceqe(eq, eq->cons_index);
3802+
3803+ return (!!(roce_get_bit(ceqe->comp, HNS_ROCE_V2_CEQ_CEQE_OWNER_S))) ^
3804+ (!!(eq->cons_index & eq->entries)) ? ceqe : NULL;
3805+}
3806+
3807+static int hns_roce_v2_ceq_int(struct hns_roce_dev *hr_dev,
3808+ struct hns_roce_eq *eq)
3809+{
3810+ struct device *dev = hr_dev->dev;
3811+ struct hns_roce_ceqe *ceqe;
3812+ int ceqe_found = 0;
3813+ u32 cqn;
3814+
3815+ while ((ceqe = next_ceqe_sw_v2(eq))) {
3816+
3817+ /* Make sure we read CEQ entry after we have checked the
3818+ * ownership bit
3819+ */
3820+ dma_rmb();
3821+
3822+ cqn = roce_get_field(ceqe->comp,
3823+ HNS_ROCE_V2_CEQE_COMP_CQN_M,
3824+ HNS_ROCE_V2_CEQE_COMP_CQN_S);
3825+
3826+ hns_roce_cq_completion(hr_dev, cqn);
3827+
3828+ ++eq->cons_index;
3829+ ceqe_found = 1;
3830+
3831+ if (eq->cons_index > (2 * eq->entries - 1)) {
3832+ dev_warn(dev, "cons_index overflow, set back to 0.\n");
3833+ eq->cons_index = 0;
3834+ }
3835+ }
3836+
3837+ set_eq_cons_index_v2(eq);
3838+
3839+ return ceqe_found;
3840+}
3841+
3842+static irqreturn_t hns_roce_v2_msix_interrupt_eq(int irq, void *eq_ptr)
3843+{
3844+ struct hns_roce_eq *eq = eq_ptr;
3845+ struct hns_roce_dev *hr_dev = eq->hr_dev;
3846+ int int_work = 0;
3847+
3848+ if (eq->type_flag == HNS_ROCE_CEQ)
3849+ /* Completion event interrupt */
3850+ int_work = hns_roce_v2_ceq_int(hr_dev, eq);
3851+ else
3852+ /* Asychronous event interrupt */
3853+ int_work = hns_roce_v2_aeq_int(hr_dev, eq);
3854+
3855+ return IRQ_RETVAL(int_work);
3856+}
3857+
3858+static irqreturn_t hns_roce_v2_msix_interrupt_abn(int irq, void *dev_id)
3859+{
3860+ struct hns_roce_dev *hr_dev = dev_id;
3861+ struct device *dev = hr_dev->dev;
3862+ int int_work = 0;
3863+ u32 int_st;
3864+ u32 int_en;
3865+
3866+ /* Abnormal interrupt */
3867+ int_st = roce_read(hr_dev, ROCEE_VF_ABN_INT_ST_REG);
3868+ int_en = roce_read(hr_dev, ROCEE_VF_ABN_INT_EN_REG);
3869+
3870+ if (roce_get_bit(int_st, HNS_ROCE_V2_VF_INT_ST_AEQ_OVERFLOW_S)) {
3871+ dev_err(dev, "AEQ overflow!\n");
3872+
3873+ roce_set_bit(int_st, HNS_ROCE_V2_VF_INT_ST_AEQ_OVERFLOW_S, 1);
3874+ roce_write(hr_dev, ROCEE_VF_ABN_INT_ST_REG, int_st);
3875+
3876+ roce_set_bit(int_en, HNS_ROCE_V2_VF_ABN_INT_EN_S, 1);
3877+ roce_write(hr_dev, ROCEE_VF_ABN_INT_EN_REG, int_en);
3878+
3879+ int_work = 1;
3880+ } else if (roce_get_bit(int_st, HNS_ROCE_V2_VF_INT_ST_BUS_ERR_S)) {
3881+ dev_err(dev, "BUS ERR!\n");
3882+
3883+ roce_set_bit(int_st, HNS_ROCE_V2_VF_INT_ST_BUS_ERR_S, 1);
3884+ roce_write(hr_dev, ROCEE_VF_ABN_INT_ST_REG, int_st);
3885+
3886+ roce_set_bit(int_en, HNS_ROCE_V2_VF_ABN_INT_EN_S, 1);
3887+ roce_write(hr_dev, ROCEE_VF_ABN_INT_EN_REG, int_en);
3888+
3889+ int_work = 1;
3890+ } else if (roce_get_bit(int_st, HNS_ROCE_V2_VF_INT_ST_OTHER_ERR_S)) {
3891+ dev_err(dev, "OTHER ERR!\n");
3892+
3893+ roce_set_bit(int_st, HNS_ROCE_V2_VF_INT_ST_OTHER_ERR_S, 1);
3894+ roce_write(hr_dev, ROCEE_VF_ABN_INT_ST_REG, int_st);
3895+
3896+ roce_set_bit(int_en, HNS_ROCE_V2_VF_ABN_INT_EN_S, 1);
3897+ roce_write(hr_dev, ROCEE_VF_ABN_INT_EN_REG, int_en);
3898+
3899+ int_work = 1;
3900+ } else
3901+ dev_err(dev, "There is no abnormal irq found!\n");
3902+
3903+ return IRQ_RETVAL(int_work);
3904+}
3905+
3906+static void hns_roce_v2_int_mask_enable(struct hns_roce_dev *hr_dev,
3907+ int eq_num, int enable_flag)
3908+{
3909+ int i;
3910+
3911+ if (enable_flag == EQ_ENABLE) {
3912+ for (i = 0; i < eq_num; i++)
3913+ roce_write(hr_dev, ROCEE_VF_EVENT_INT_EN_REG +
3914+ i * EQ_REG_OFFSET,
3915+ HNS_ROCE_V2_VF_EVENT_INT_EN_M);
3916+
3917+ roce_write(hr_dev, ROCEE_VF_ABN_INT_EN_REG,
3918+ HNS_ROCE_V2_VF_ABN_INT_EN_M);
3919+ roce_write(hr_dev, ROCEE_VF_ABN_INT_CFG_REG,
3920+ HNS_ROCE_V2_VF_ABN_INT_CFG_M);
3921+ } else {
3922+ for (i = 0; i < eq_num; i++)
3923+ roce_write(hr_dev, ROCEE_VF_EVENT_INT_EN_REG +
3924+ i * EQ_REG_OFFSET,
3925+ HNS_ROCE_V2_VF_EVENT_INT_EN_M & 0x0);
3926+
3927+ roce_write(hr_dev, ROCEE_VF_ABN_INT_EN_REG,
3928+ HNS_ROCE_V2_VF_ABN_INT_EN_M & 0x0);
3929+ roce_write(hr_dev, ROCEE_VF_ABN_INT_CFG_REG,
3930+ HNS_ROCE_V2_VF_ABN_INT_CFG_M & 0x0);
3931+ }
3932+}
3933+
3934+static void hns_roce_v2_destroy_eqc(struct hns_roce_dev *hr_dev, int eqn)
3935+{
3936+ struct device *dev = hr_dev->dev;
3937+ int ret;
3938+
3939+ if (eqn < hr_dev->caps.num_comp_vectors)
3940+ ret = hns_roce_cmd_mbox(hr_dev, 0, 0, eqn & HNS_ROCE_V2_EQN_M,
3941+ 0, HNS_ROCE_CMD_DESTROY_CEQC,
3942+ HNS_ROCE_CMD_TIMEOUT_MSECS);
3943+ else
3944+ ret = hns_roce_cmd_mbox(hr_dev, 0, 0, eqn & HNS_ROCE_V2_EQN_M,
3945+ 0, HNS_ROCE_CMD_DESTROY_AEQC,
3946+ HNS_ROCE_CMD_TIMEOUT_MSECS);
3947+ if (ret)
3948+ dev_err(dev, "[mailbox cmd] destroy eqc(%d) failed.\n", eqn);
3949+}
3950+
3951+static void hns_roce_mhop_free_eq(struct hns_roce_dev *hr_dev,
3952+ struct hns_roce_eq *eq)
3953+{
3954+ struct device *dev = hr_dev->dev;
3955+ u64 idx;
3956+ u64 size;
3957+ u32 buf_chk_sz;
3958+ u32 bt_chk_sz;
3959+ u32 mhop_num;
3960+ int eqe_alloc;
3961+ int ba_num;
3962+ int i = 0;
3963+ int j = 0;
3964+
3965+ mhop_num = hr_dev->caps.eqe_hop_num;
3966+ buf_chk_sz = 1 << (hr_dev->caps.eqe_buf_pg_sz + PAGE_SHIFT);
3967+ bt_chk_sz = 1 << (hr_dev->caps.eqe_ba_pg_sz + PAGE_SHIFT);
3968+ ba_num = (PAGE_ALIGN(eq->entries * eq->eqe_size) + buf_chk_sz - 1) /
3969+ buf_chk_sz;
3970+
3971+ /* hop_num = 0 */
3972+ if (mhop_num == HNS_ROCE_HOP_NUM_0) {
3973+ dma_free_coherent(dev, (unsigned int)(eq->entries *
3974+ eq->eqe_size), eq->bt_l0, eq->l0_dma);
3975+ return;
3976+ }
3977+
3978+ /* hop_num = 1 or hop = 2 */
3979+ dma_free_coherent(dev, bt_chk_sz, eq->bt_l0, eq->l0_dma);
3980+ if (mhop_num == 1) {
3981+ for (i = 0; i < eq->l0_last_num; i++) {
3982+ if (i == eq->l0_last_num - 1) {
3983+ eqe_alloc = i * (buf_chk_sz / eq->eqe_size);
3984+ size = (eq->entries - eqe_alloc) * eq->eqe_size;
3985+ dma_free_coherent(dev, size, eq->buf[i],
3986+ eq->buf_dma[i]);
3987+ break;
3988+ }
3989+ dma_free_coherent(dev, buf_chk_sz, eq->buf[i],
3990+ eq->buf_dma[i]);
3991+ }
3992+ } else if (mhop_num == 2) {
3993+ for (i = 0; i < eq->l0_last_num; i++) {
3994+ dma_free_coherent(dev, bt_chk_sz, eq->bt_l1[i],
3995+ eq->l1_dma[i]);
3996+
3997+ for (j = 0; j < bt_chk_sz / 8; j++) {
3998+ idx = i * (bt_chk_sz / 8) + j;
3999+ if ((i == eq->l0_last_num - 1)
4000+ && j == eq->l1_last_num - 1) {
4001+ eqe_alloc = (buf_chk_sz / eq->eqe_size)
4002+ * idx;
4003+ size = (eq->entries - eqe_alloc)
4004+ * eq->eqe_size;
4005+ dma_free_coherent(dev, size,
4006+ eq->buf[idx],
4007+ eq->buf_dma[idx]);
4008+ break;
4009+ }
4010+ dma_free_coherent(dev, buf_chk_sz, eq->buf[idx],
4011+ eq->buf_dma[idx]);
4012+ }
4013+ }
4014+ }
4015+ kfree(eq->buf_dma);
4016+ kfree(eq->buf);
4017+ kfree(eq->l1_dma);
4018+ kfree(eq->bt_l1);
4019+ eq->buf_dma = NULL;
4020+ eq->buf = NULL;
4021+ eq->l1_dma = NULL;
4022+ eq->bt_l1 = NULL;
4023+}
4024+
4025+static void hns_roce_v2_free_eq(struct hns_roce_dev *hr_dev,
4026+ struct hns_roce_eq *eq)
4027+{
4028+ u32 buf_chk_sz;
4029+
4030+ buf_chk_sz = 1 << (eq->eqe_buf_pg_sz + PAGE_SHIFT);
4031+
4032+ if (hr_dev->caps.eqe_hop_num) {
4033+ hns_roce_mhop_free_eq(hr_dev, eq);
4034+ return;
4035+ }
4036+
4037+ if (eq->buf_list)
4038+ dma_free_coherent(hr_dev->dev, buf_chk_sz,
4039+ eq->buf_list->buf, eq->buf_list->map);
4040+}
4041+
4042+static void hns_roce_config_eqc(struct hns_roce_dev *hr_dev,
4043+ struct hns_roce_eq *eq,
4044+ void *mb_buf)
4045+{
4046+ struct hns_roce_eq_context *eqc;
4047+
4048+ eqc = mb_buf;
4049+ memset(eqc, 0, sizeof(struct hns_roce_eq_context));
4050+
4051+ /* init eqc */
4052+ eq->doorbell = hr_dev->reg_base + ROCEE_VF_EQ_DB_CFG0_REG;
4053+ eq->hop_num = hr_dev->caps.eqe_hop_num;
4054+ eq->cons_index = 0;
4055+ eq->over_ignore = HNS_ROCE_V2_EQ_OVER_IGNORE_0;
4056+ eq->coalesce = HNS_ROCE_V2_EQ_COALESCE_0;
4057+ eq->arm_st = HNS_ROCE_V2_EQ_ALWAYS_ARMED;
4058+ eq->eqe_ba_pg_sz = hr_dev->caps.eqe_ba_pg_sz;
4059+ eq->eqe_buf_pg_sz = hr_dev->caps.eqe_buf_pg_sz;
4060+ eq->shift = ilog2((unsigned int)eq->entries);
4061+
4062+ if (!eq->hop_num)
4063+ eq->eqe_ba = eq->buf_list->map;
4064+ else
4065+ eq->eqe_ba = eq->l0_dma;
4066+
4067+ /* set eqc state */
4068+ roce_set_field(eqc->byte_4,
4069+ HNS_ROCE_EQC_EQ_ST_M,
4070+ HNS_ROCE_EQC_EQ_ST_S,
4071+ HNS_ROCE_V2_EQ_STATE_VALID);
4072+
4073+ /* set eqe hop num */
4074+ roce_set_field(eqc->byte_4,
4075+ HNS_ROCE_EQC_HOP_NUM_M,
4076+ HNS_ROCE_EQC_HOP_NUM_S, eq->hop_num);
4077+
4078+ /* set eqc over_ignore */
4079+ roce_set_field(eqc->byte_4,
4080+ HNS_ROCE_EQC_OVER_IGNORE_M,
4081+ HNS_ROCE_EQC_OVER_IGNORE_S, eq->over_ignore);
4082+
4083+ /* set eqc coalesce */
4084+ roce_set_field(eqc->byte_4,
4085+ HNS_ROCE_EQC_COALESCE_M,
4086+ HNS_ROCE_EQC_COALESCE_S, eq->coalesce);
4087+
4088+ /* set eqc arm_state */
4089+ roce_set_field(eqc->byte_4,
4090+ HNS_ROCE_EQC_ARM_ST_M,
4091+ HNS_ROCE_EQC_ARM_ST_S, eq->arm_st);
4092+
4093+ /* set eqn */
4094+ roce_set_field(eqc->byte_4,
4095+ HNS_ROCE_EQC_EQN_M,
4096+ HNS_ROCE_EQC_EQN_S, eq->eqn);
4097+
4098+ /* set eqe_cnt */
4099+ roce_set_field(eqc->byte_4,
4100+ HNS_ROCE_EQC_EQE_CNT_M,
4101+ HNS_ROCE_EQC_EQE_CNT_S,
4102+ HNS_ROCE_EQ_INIT_EQE_CNT);
4103+
4104+ /* set eqe_ba_pg_sz */
4105+ roce_set_field(eqc->byte_8,
4106+ HNS_ROCE_EQC_BA_PG_SZ_M,
4107+ HNS_ROCE_EQC_BA_PG_SZ_S, eq->eqe_ba_pg_sz);
4108+
4109+ /* set eqe_buf_pg_sz */
4110+ roce_set_field(eqc->byte_8,
4111+ HNS_ROCE_EQC_BUF_PG_SZ_M,
4112+ HNS_ROCE_EQC_BUF_PG_SZ_S, eq->eqe_buf_pg_sz);
4113+
4114+ /* set eq_producer_idx */
4115+ roce_set_field(eqc->byte_8,
4116+ HNS_ROCE_EQC_PROD_INDX_M,
4117+ HNS_ROCE_EQC_PROD_INDX_S,
4118+ HNS_ROCE_EQ_INIT_PROD_IDX);
4119+
4120+ /* set eq_max_cnt */
4121+ roce_set_field(eqc->byte_12,
4122+ HNS_ROCE_EQC_MAX_CNT_M,
4123+ HNS_ROCE_EQC_MAX_CNT_S, eq->eq_max_cnt);
4124+
4125+ /* set eq_period */
4126+ roce_set_field(eqc->byte_12,
4127+ HNS_ROCE_EQC_PERIOD_M,
4128+ HNS_ROCE_EQC_PERIOD_S, eq->eq_period);
4129+
4130+ /* set eqe_report_timer */
4131+ roce_set_field(eqc->eqe_report_timer,
4132+ HNS_ROCE_EQC_REPORT_TIMER_M,
4133+ HNS_ROCE_EQC_REPORT_TIMER_S,
4134+ HNS_ROCE_EQ_INIT_REPORT_TIMER);
4135+
4136+ /* set eqe_ba [34:3] */
4137+ roce_set_field(eqc->eqe_ba0,
4138+ HNS_ROCE_EQC_EQE_BA_L_M,
4139+ HNS_ROCE_EQC_EQE_BA_L_S, eq->eqe_ba >> 3);
4140+
4141+ /* set eqe_ba [64:35] */
4142+ roce_set_field(eqc->eqe_ba1,
4143+ HNS_ROCE_EQC_EQE_BA_H_M,
4144+ HNS_ROCE_EQC_EQE_BA_H_S, eq->eqe_ba >> 35);
4145+
4146+ /* set eq shift */
4147+ roce_set_field(eqc->byte_28,
4148+ HNS_ROCE_EQC_SHIFT_M,
4149+ HNS_ROCE_EQC_SHIFT_S, eq->shift);
4150+
4151+ /* set eq MSI_IDX */
4152+ roce_set_field(eqc->byte_28,
4153+ HNS_ROCE_EQC_MSI_INDX_M,
4154+ HNS_ROCE_EQC_MSI_INDX_S,
4155+ HNS_ROCE_EQ_INIT_MSI_IDX);
4156+
4157+ /* set cur_eqe_ba [27:12] */
4158+ roce_set_field(eqc->byte_28,
4159+ HNS_ROCE_EQC_CUR_EQE_BA_L_M,
4160+ HNS_ROCE_EQC_CUR_EQE_BA_L_S, eq->cur_eqe_ba >> 12);
4161+
4162+ /* set cur_eqe_ba [59:28] */
4163+ roce_set_field(eqc->byte_32,
4164+ HNS_ROCE_EQC_CUR_EQE_BA_M_M,
4165+ HNS_ROCE_EQC_CUR_EQE_BA_M_S, eq->cur_eqe_ba >> 28);
4166+
4167+ /* set cur_eqe_ba [63:60] */
4168+ roce_set_field(eqc->byte_36,
4169+ HNS_ROCE_EQC_CUR_EQE_BA_H_M,
4170+ HNS_ROCE_EQC_CUR_EQE_BA_H_S, eq->cur_eqe_ba >> 60);
4171+
4172+ /* set eq consumer idx */
4173+ roce_set_field(eqc->byte_36,
4174+ HNS_ROCE_EQC_CONS_INDX_M,
4175+ HNS_ROCE_EQC_CONS_INDX_S,
4176+ HNS_ROCE_EQ_INIT_CONS_IDX);
4177+
4178+ /* set nex_eqe_ba[43:12] */
4179+ roce_set_field(eqc->nxt_eqe_ba0,
4180+ HNS_ROCE_EQC_NXT_EQE_BA_L_M,
4181+ HNS_ROCE_EQC_NXT_EQE_BA_L_S, eq->nxt_eqe_ba >> 12);
4182+
4183+ /* set nex_eqe_ba[63:44] */
4184+ roce_set_field(eqc->nxt_eqe_ba1,
4185+ HNS_ROCE_EQC_NXT_EQE_BA_H_M,
4186+ HNS_ROCE_EQC_NXT_EQE_BA_H_S, eq->nxt_eqe_ba >> 44);
4187+}
4188+
4189+static int hns_roce_mhop_alloc_eq(struct hns_roce_dev *hr_dev,
4190+ struct hns_roce_eq *eq)
4191+{
4192+ struct device *dev = hr_dev->dev;
4193+ int eq_alloc_done = 0;
4194+ int eq_buf_cnt = 0;
4195+ int eqe_alloc;
4196+ u32 buf_chk_sz;
4197+ u32 bt_chk_sz;
4198+ u32 mhop_num;
4199+ u64 size;
4200+ u64 idx;
4201+ int ba_num;
4202+ int bt_num;
4203+ int record_i;
4204+ int record_j;
4205+ int i = 0;
4206+ int j = 0;
4207+
4208+ mhop_num = hr_dev->caps.eqe_hop_num;
4209+ buf_chk_sz = 1 << (hr_dev->caps.eqe_buf_pg_sz + PAGE_SHIFT);
4210+ bt_chk_sz = 1 << (hr_dev->caps.eqe_ba_pg_sz + PAGE_SHIFT);
4211+
4212+ ba_num = (PAGE_ALIGN(eq->entries * eq->eqe_size) + buf_chk_sz - 1)
4213+ / buf_chk_sz;
4214+ bt_num = (ba_num + bt_chk_sz / 8 - 1) / (bt_chk_sz / 8);
4215+
4216+ /* hop_num = 0 */
4217+ if (mhop_num == HNS_ROCE_HOP_NUM_0) {
4218+ if (eq->entries > buf_chk_sz / eq->eqe_size) {
4219+ dev_err(dev, "eq entries %d is larger than buf_pg_sz!",
4220+ eq->entries);
4221+ return -EINVAL;
4222+ }
4223+ eq->bt_l0 = dma_alloc_coherent(dev, eq->entries * eq->eqe_size,
4224+ &(eq->l0_dma), GFP_KERNEL);
4225+ if (!eq->bt_l0)
4226+ return -ENOMEM;
4227+
4228+ eq->cur_eqe_ba = eq->l0_dma;
4229+ eq->nxt_eqe_ba = 0;
4230+
4231+ memset(eq->bt_l0, 0, eq->entries * eq->eqe_size);
4232+
4233+ return 0;
4234+ }
4235+
4236+ eq->buf_dma = kcalloc(ba_num, sizeof(*eq->buf_dma), GFP_KERNEL);
4237+ if (!eq->buf_dma)
4238+ return -ENOMEM;
4239+ eq->buf = kcalloc(ba_num, sizeof(*eq->buf), GFP_KERNEL);
4240+ if (!eq->buf)
4241+ goto err_kcalloc_buf;
4242+
4243+ if (mhop_num == 2) {
4244+ eq->l1_dma = kcalloc(bt_num, sizeof(*eq->l1_dma), GFP_KERNEL);
4245+ if (!eq->l1_dma)
4246+ goto err_kcalloc_l1_dma;
4247+
4248+ eq->bt_l1 = kcalloc(bt_num, sizeof(*eq->bt_l1), GFP_KERNEL);
4249+ if (!eq->bt_l1)
4250+ goto err_kcalloc_bt_l1;
4251+ }
4252+
4253+ /* alloc L0 BT */
4254+ eq->bt_l0 = dma_alloc_coherent(dev, bt_chk_sz, &eq->l0_dma, GFP_KERNEL);
4255+ if (!eq->bt_l0)
4256+ goto err_dma_alloc_l0;
4257+
4258+ if (mhop_num == 1) {
4259+ if (ba_num > (bt_chk_sz / 8))
4260+ dev_err(dev, "ba_num %d is too large for 1 hop\n",
4261+ ba_num);
4262+
4263+ /* alloc buf */
4264+ for (i = 0; i < bt_chk_sz / 8; i++) {
4265+ if (eq_buf_cnt + 1 < ba_num) {
4266+ size = buf_chk_sz;
4267+ } else {
4268+ eqe_alloc = i * (buf_chk_sz / eq->eqe_size);
4269+ size = (eq->entries - eqe_alloc) * eq->eqe_size;
4270+ }
4271+ eq->buf[i] = dma_alloc_coherent(dev, size,
4272+ &(eq->buf_dma[i]),
4273+ GFP_KERNEL);
4274+ if (!eq->buf[i])
4275+ goto err_dma_alloc_buf;
4276+
4277+ memset(eq->buf[i], 0, size);
4278+ *(eq->bt_l0 + i) = eq->buf_dma[i];
4279+
4280+ eq_buf_cnt++;
4281+ if (eq_buf_cnt >= ba_num)
4282+ break;
4283+ }
4284+ eq->cur_eqe_ba = eq->buf_dma[0];
4285+ eq->nxt_eqe_ba = eq->buf_dma[1];
4286+
4287+ } else if (mhop_num == 2) {
4288+ /* alloc L1 BT and buf */
4289+ for (i = 0; i < bt_chk_sz / 8; i++) {
4290+ eq->bt_l1[i] = dma_alloc_coherent(dev, bt_chk_sz,
4291+ &(eq->l1_dma[i]),
4292+ GFP_KERNEL);
4293+ if (!eq->bt_l1[i])
4294+ goto err_dma_alloc_l1;
4295+ *(eq->bt_l0 + i) = eq->l1_dma[i];
4296+
4297+ for (j = 0; j < bt_chk_sz / 8; j++) {
4298+ idx = i * bt_chk_sz / 8 + j;
4299+ if (eq_buf_cnt + 1 < ba_num) {
4300+ size = buf_chk_sz;
4301+ } else {
4302+ eqe_alloc = (buf_chk_sz / eq->eqe_size)
4303+ * idx;
4304+ size = (eq->entries - eqe_alloc)
4305+ * eq->eqe_size;
4306+ }
4307+ eq->buf[idx] = dma_alloc_coherent(dev, size,
4308+ &(eq->buf_dma[idx]),
4309+ GFP_KERNEL);
4310+ if (!eq->buf[idx])
4311+ goto err_dma_alloc_buf;
4312+
4313+ memset(eq->buf[idx], 0, size);
4314+ *(eq->bt_l1[i] + j) = eq->buf_dma[idx];
4315+
4316+ eq_buf_cnt++;
4317+ if (eq_buf_cnt >= ba_num) {
4318+ eq_alloc_done = 1;
4319+ break;
4320+ }
4321+ }
4322+
4323+ if (eq_alloc_done)
4324+ break;
4325+ }
4326+ eq->cur_eqe_ba = eq->buf_dma[0];
4327+ eq->nxt_eqe_ba = eq->buf_dma[1];
4328+ }
4329+
4330+ eq->l0_last_num = i + 1;
4331+ if (mhop_num == 2)
4332+ eq->l1_last_num = j + 1;
4333+
4334+ return 0;
4335+
4336+err_dma_alloc_l1:
4337+ dma_free_coherent(dev, bt_chk_sz, eq->bt_l0, eq->l0_dma);
4338+ eq->bt_l0 = NULL;
4339+ eq->l0_dma = 0;
4340+ for (i -= 1; i >= 0; i--) {
4341+ dma_free_coherent(dev, bt_chk_sz, eq->bt_l1[i],
4342+ eq->l1_dma[i]);
4343+
4344+ for (j = 0; j < bt_chk_sz / 8; j++) {
4345+ idx = i * bt_chk_sz / 8 + j;
4346+ dma_free_coherent(dev, buf_chk_sz, eq->buf[idx],
4347+ eq->buf_dma[idx]);
4348+ }
4349+ }
4350+ goto err_dma_alloc_l0;
4351+
4352+err_dma_alloc_buf:
4353+ dma_free_coherent(dev, bt_chk_sz, eq->bt_l0, eq->l0_dma);
4354+ eq->bt_l0 = NULL;
4355+ eq->l0_dma = 0;
4356+
4357+ if (mhop_num == 1)
4358+ for (i -= i; i >= 0; i--)
4359+ dma_free_coherent(dev, buf_chk_sz, eq->buf[i],
4360+ eq->buf_dma[i]);
4361+ else if (mhop_num == 2) {
4362+ record_i = i;
4363+ record_j = j;
4364+ for (; i >= 0; i--) {
4365+ dma_free_coherent(dev, bt_chk_sz, eq->bt_l1[i],
4366+ eq->l1_dma[i]);
4367+
4368+ for (j = 0; j < bt_chk_sz / 8; j++) {
4369+ if (i == record_i && j >= record_j)
4370+ break;
4371+
4372+ idx = i * bt_chk_sz / 8 + j;
4373+ dma_free_coherent(dev, buf_chk_sz,
4374+ eq->buf[idx],
4375+ eq->buf_dma[idx]);
4376+ }
4377+ }
4378+ }
4379+
4380+err_dma_alloc_l0:
4381+ kfree(eq->bt_l1);
4382+ eq->bt_l1 = NULL;
4383+
4384+err_kcalloc_bt_l1:
4385+ kfree(eq->l1_dma);
4386+ eq->l1_dma = NULL;
4387+
4388+err_kcalloc_l1_dma:
4389+ kfree(eq->buf);
4390+ eq->buf = NULL;
4391+
4392+err_kcalloc_buf:
4393+ kfree(eq->buf_dma);
4394+ eq->buf_dma = NULL;
4395+
4396+ return -ENOMEM;
4397+}
4398+
4399+static int hns_roce_v2_create_eq(struct hns_roce_dev *hr_dev,
4400+ struct hns_roce_eq *eq,
4401+ unsigned int eq_cmd)
4402+{
4403+ struct device *dev = hr_dev->dev;
4404+ struct hns_roce_cmd_mailbox *mailbox;
4405+ u32 buf_chk_sz = 0;
4406+ int ret;
4407+
4408+ /* Allocate mailbox memory */
4409+ mailbox = hns_roce_alloc_cmd_mailbox(hr_dev);
4410+ if (IS_ERR(mailbox))
4411+ return PTR_ERR(mailbox);
4412+
4413+ if (!hr_dev->caps.eqe_hop_num) {
4414+ buf_chk_sz = 1 << (hr_dev->caps.eqe_buf_pg_sz + PAGE_SHIFT);
4415+
4416+ eq->buf_list = kzalloc(sizeof(struct hns_roce_buf_list),
4417+ GFP_KERNEL);
4418+ if (!eq->buf_list) {
4419+ ret = -ENOMEM;
4420+ goto free_cmd_mbox;
4421+ }
4422+
4423+ eq->buf_list->buf = dma_alloc_coherent(dev, buf_chk_sz,
4424+ &(eq->buf_list->map),
4425+ GFP_KERNEL);
4426+ if (!eq->buf_list->buf) {
4427+ ret = -ENOMEM;
4428+ goto err_alloc_buf;
4429+ }
4430+
4431+ memset(eq->buf_list->buf, 0, buf_chk_sz);
4432+ } else {
4433+ ret = hns_roce_mhop_alloc_eq(hr_dev, eq);
4434+ if (ret) {
4435+ ret = -ENOMEM;
4436+ goto free_cmd_mbox;
4437+ }
4438+ }
4439+
4440+ hns_roce_config_eqc(hr_dev, eq, mailbox->buf);
4441+
4442+ ret = hns_roce_cmd_mbox(hr_dev, mailbox->dma, 0, eq->eqn, 0,
4443+ eq_cmd, HNS_ROCE_CMD_TIMEOUT_MSECS);
4444+ if (ret) {
4445+ dev_err(dev, "[mailbox cmd] creat eqc failed.\n");
4446+ goto err_cmd_mbox;
4447+ }
4448+
4449+ hns_roce_free_cmd_mailbox(hr_dev, mailbox);
4450+
4451+ return 0;
4452+
4453+err_cmd_mbox:
4454+ if (!hr_dev->caps.eqe_hop_num)
4455+ dma_free_coherent(dev, buf_chk_sz, eq->buf_list->buf,
4456+ eq->buf_list->map);
4457+ else {
4458+ hns_roce_mhop_free_eq(hr_dev, eq);
4459+ goto free_cmd_mbox;
4460+ }
4461+
4462+err_alloc_buf:
4463+ kfree(eq->buf_list);
4464+
4465+free_cmd_mbox:
4466+ hns_roce_free_cmd_mailbox(hr_dev, mailbox);
4467+
4468+ return ret;
4469+}
4470+
4471+static int hns_roce_v2_init_eq_table(struct hns_roce_dev *hr_dev)
4472+{
4473+ struct hns_roce_eq_table *eq_table = &hr_dev->eq_table;
4474+ struct device *dev = hr_dev->dev;
4475+ struct hns_roce_eq *eq;
4476+ unsigned int eq_cmd;
4477+ int irq_num;
4478+ int eq_num;
4479+ int other_num;
4480+ int comp_num;
4481+ int aeq_num;
4482+ int i, j, k;
4483+ int ret;
4484+
4485+ other_num = hr_dev->caps.num_other_vectors;
4486+ comp_num = hr_dev->caps.num_comp_vectors;
4487+ aeq_num = hr_dev->caps.num_aeq_vectors;
4488+
4489+ eq_num = comp_num + aeq_num;
4490+ irq_num = eq_num + other_num;
4491+
4492+ eq_table->eq = kcalloc(eq_num, sizeof(*eq_table->eq), GFP_KERNEL);
4493+ if (!eq_table->eq)
4494+ return -ENOMEM;
4495+
4496+ for (i = 0; i < irq_num; i++) {
4497+ hr_dev->irq_names[i] = kzalloc(HNS_ROCE_INT_NAME_LEN,
4498+ GFP_KERNEL);
4499+ if (!hr_dev->irq_names[i]) {
4500+ ret = -ENOMEM;
4501+ goto err_failed_kzalloc;
4502+ }
4503+ }
4504+
4505+ /* create eq */
4506+ for (j = 0; j < eq_num; j++) {
4507+ eq = &eq_table->eq[j];
4508+ eq->hr_dev = hr_dev;
4509+ eq->eqn = j;
4510+ if (j < comp_num) {
4511+ /* CEQ */
4512+ eq_cmd = HNS_ROCE_CMD_CREATE_CEQC;
4513+ eq->type_flag = HNS_ROCE_CEQ;
4514+ eq->entries = hr_dev->caps.ceqe_depth;
4515+ eq->eqe_size = HNS_ROCE_CEQ_ENTRY_SIZE;
4516+ eq->irq = hr_dev->irq[j + other_num + aeq_num];
4517+ eq->eq_max_cnt = HNS_ROCE_CEQ_DEFAULT_BURST_NUM;
4518+ eq->eq_period = HNS_ROCE_CEQ_DEFAULT_INTERVAL;
4519+ } else {
4520+ /* AEQ */
4521+ eq_cmd = HNS_ROCE_CMD_CREATE_AEQC;
4522+ eq->type_flag = HNS_ROCE_AEQ;
4523+ eq->entries = hr_dev->caps.aeqe_depth;
4524+ eq->eqe_size = HNS_ROCE_AEQ_ENTRY_SIZE;
4525+ eq->irq = hr_dev->irq[j - comp_num + other_num];
4526+ eq->eq_max_cnt = HNS_ROCE_AEQ_DEFAULT_BURST_NUM;
4527+ eq->eq_period = HNS_ROCE_AEQ_DEFAULT_INTERVAL;
4528+ }
4529+
4530+ ret = hns_roce_v2_create_eq(hr_dev, eq, eq_cmd);
4531+ if (ret) {
4532+ dev_err(dev, "eq create failed.\n");
4533+ goto err_create_eq_fail;
4534+ }
4535+ }
4536+
4537+ /* enable irq */
4538+ hns_roce_v2_int_mask_enable(hr_dev, eq_num, EQ_ENABLE);
4539+
4540+ /* irq contains: abnormal + AEQ + CEQ*/
4541+ for (k = 0; k < irq_num; k++)
4542+ if (k < other_num)
4543+ snprintf((char *)hr_dev->irq_names[k],
4544+ HNS_ROCE_INT_NAME_LEN, "hns-abn-%d", k);
4545+ else if (k < (other_num + aeq_num))
4546+ snprintf((char *)hr_dev->irq_names[k],
4547+ HNS_ROCE_INT_NAME_LEN, "hns-aeq-%d",
4548+ k - other_num);
4549+ else
4550+ snprintf((char *)hr_dev->irq_names[k],
4551+ HNS_ROCE_INT_NAME_LEN, "hns-ceq-%d",
4552+ k - other_num - aeq_num);
4553+
4554+ for (k = 0; k < irq_num; k++) {
4555+ if (k < other_num)
4556+ ret = request_irq(hr_dev->irq[k],
4557+ hns_roce_v2_msix_interrupt_abn,
4558+ 0, hr_dev->irq_names[k], hr_dev);
4559+
4560+ else if (k < (other_num + comp_num))
4561+ ret = request_irq(eq_table->eq[k - other_num].irq,
4562+ hns_roce_v2_msix_interrupt_eq,
4563+ 0, hr_dev->irq_names[k + aeq_num],
4564+ &eq_table->eq[k - other_num]);
4565+ else
4566+ ret = request_irq(eq_table->eq[k - other_num].irq,
4567+ hns_roce_v2_msix_interrupt_eq,
4568+ 0, hr_dev->irq_names[k - comp_num],
4569+ &eq_table->eq[k - other_num]);
4570+ if (ret) {
4571+ dev_err(dev, "Request irq error!\n");
4572+ goto err_request_irq_fail;
4573+ }
4574+ }
4575+
4576+ return 0;
4577+
4578+err_request_irq_fail:
4579+ for (k -= 1; k >= 0; k--)
4580+ if (k < other_num)
4581+ free_irq(hr_dev->irq[k], hr_dev);
4582+ else
4583+ free_irq(eq_table->eq[k - other_num].irq,
4584+ &eq_table->eq[k - other_num]);
4585+
4586+err_create_eq_fail:
4587+ for (j -= 1; j >= 0; j--)
4588+ hns_roce_v2_free_eq(hr_dev, &eq_table->eq[j]);
4589+
4590+err_failed_kzalloc:
4591+ for (i -= 1; i >= 0; i--)
4592+ kfree(hr_dev->irq_names[i]);
4593+ kfree(eq_table->eq);
4594+
4595+ return ret;
4596+}
4597+
4598+static void hns_roce_v2_cleanup_eq_table(struct hns_roce_dev *hr_dev)
4599+{
4600+ struct hns_roce_eq_table *eq_table = &hr_dev->eq_table;
4601+ int irq_num;
4602+ int eq_num;
4603+ int i;
4604+
4605+ eq_num = hr_dev->caps.num_comp_vectors + hr_dev->caps.num_aeq_vectors;
4606+ irq_num = eq_num + hr_dev->caps.num_other_vectors;
4607+
4608+ /* Disable irq */
4609+ hns_roce_v2_int_mask_enable(hr_dev, eq_num, EQ_DISABLE);
4610+
4611+ for (i = 0; i < hr_dev->caps.num_other_vectors; i++)
4612+ free_irq(hr_dev->irq[i], hr_dev);
4613+
4614+ for (i = 0; i < eq_num; i++) {
4615+ hns_roce_v2_destroy_eqc(hr_dev, i);
4616+
4617+ free_irq(eq_table->eq[i].irq, &eq_table->eq[i]);
4618+
4619+ hns_roce_v2_free_eq(hr_dev, &eq_table->eq[i]);
4620+ }
4621+
4622+ for (i = 0; i < irq_num; i++)
4623+ kfree(hr_dev->irq_names[i]);
4624+
4625+ kfree(eq_table->eq);
4626+}
4627+
31654628 static const struct hns_roce_hw hns_roce_hw_v2 = {
31664629 .cmq_init = hns_roce_v2_cmq_init,
31674630 .cmq_exit = hns_roce_v2_cmq_exit,
@@ -3183,6 +4646,8 @@ static const struct hns_roce_hw hns_roce_hw_v2 = { · 15 unmodified lines
31834646 .post_recv = hns_roce_v2_post_recv,
31844647 .req_notify_cq = hns_roce_v2_req_notify_cq,
31854648 .poll_cq = hns_roce_v2_poll_cq,
4649+ .init_eq = hns_roce_v2_init_eq_table,
4650+ .cleanup_eq = hns_roce_v2_cleanup_eq_table,
31864651 };
31874652
31884653 static const struct pci_device_id hns_roce_hw_v2_pci_tbl[] = {
@@ -3197,6 +4662,7 @@ static int hns_roce_hw_v2_get_cfg(struct hns_roce_dev *hr_dev, · 8 unmodified lines
31974662 struct hnae3_handle *handle)
31984663 {
31994664 const struct pci_device_id *id;
4665+ int i;
32004666
32014667 id = pci_match_id(hns_roce_hw_v2_pci_tbl, hr_dev->pci_dev);
32024668 if (!id) {
@@ -3214,8 +4680,15 @@ static int hns_roce_hw_v2_get_cfg(struct hns_roce_dev *hr_dev, · 11 unmodified lines
32144680 hr_dev->iboe.netdevs[0] = handle->rinfo.netdev;
32154681 hr_dev->iboe.phy_port[0] = 0;
32164682
4683+ addrconf_addr_eui48((u8 *)&hr_dev->ib_dev.node_guid,
4684+ hr_dev->iboe.netdevs[0]->dev_addr);
4685+
4686+ for (i = 0; i < HNS_ROCE_V2_MAX_IRQ_NUM; i++)
4687+ hr_dev->irq[i] = pci_irq_vector(handle->pdev,
4688+ i + handle->rinfo.base_vector);
4689+
32174690 /* cmd issue mode: 0 is poll, 1 is event */
3218− hr_dev->cmd_mod = 0;
4691+ hr_dev->cmd_mod = 1;
32194692 hr_dev->loop_idc = 0;
32204693
32214694 return 0;
drivers/infiniband/hw/hns/hns_roce_hw_v2.h+278−5
@@ -53,6 +53,10 @@ · 52 unmodified lines
5353 #define HNS_ROCE_V2_MAX_SQ_INLINE 0x20
5454 #define HNS_ROCE_V2_UAR_NUM 256
5555 #define HNS_ROCE_V2_PHY_UAR_NUM 1
56+#define HNS_ROCE_V2_MAX_IRQ_NUM 65
57+#define HNS_ROCE_V2_COMP_VEC_NUM 63
58+#define HNS_ROCE_V2_AEQE_VEC_NUM 1
59+#define HNS_ROCE_V2_ABNORMAL_VEC_NUM 1
5660 #define HNS_ROCE_V2_MAX_MTPT_NUM 0x8000
5761 #define HNS_ROCE_V2_MAX_MTT_SEGS 0x1000000
5862 #define HNS_ROCE_V2_MAX_CQE_SEGS 0x1000000
@@ -78,6 +82,8 @@ · 19 unmodified lines
7882 #define HNS_ROCE_MTT_HOP_NUM 1
7983 #define HNS_ROCE_CQE_HOP_NUM 1
8084 #define HNS_ROCE_PBL_HOP_NUM 2
85+#define HNS_ROCE_EQE_HOP_NUM 2
86+
8187 #define HNS_ROCE_V2_GID_INDEX_NUM 256
8288
8389 #define HNS_ROCE_V2_TABLE_CHUNK_SIZE (1 << 18)
@@ -105,6 +111,12 @@ · 21 unmodified lines
105111 (step_idx == 1 && hop_num == 1) || \
106112 (step_idx == 2 && hop_num == 2))
107113
114+enum {
115+ NO_ARMED = 0x0,
116+ REG_NXT_CEQE = 0x2,
117+ REG_NXT_SE_CEQE = 0x3
118+};
119+
108120 #define V2_CQ_DB_REQ_NOT_SOL 0
109121 #define V2_CQ_DB_REQ_NOT 1
110122
@@ -229,6 +241,9 @@ struct hns_roce_v2_cq_context { · 118 unmodified lines
229241 u32 cqe_report_timer;
230242 u32 byte_64_se_cqe_idx;
231243 };
244+#define HNS_ROCE_V2_CQ_DEFAULT_BURST_NUM 0x0
245+#define HNS_ROCE_V2_CQ_DEFAULT_INTERVAL 0x0
246+
232247 #define V2_CQC_BYTE_4_CQ_ST_S 0
233248 #define V2_CQC_BYTE_4_CQ_ST_M GENMASK(1, 0)
234249
@@ -747,11 +762,14 @@ struct hns_roce_v2_qp_context { · 512 unmodified lines
747762
748763 struct hns_roce_v2_cqe {
749764 u32 byte_4;
750− u32 rkey_immtdata;
765+ union {
766+ __le32 rkey;
767+ __be32 immtdata;
768+ };
751769 u32 byte_12;
752770 u32 byte_16;
753771 u32 byte_cnt;
754− u32 smac;
772+ u8 smac[4];
755773 u32 byte_28;
756774 u32 byte_32;
757775 };
@@ -901,6 +919,90 @@ struct hns_roce_v2_cq_db { · 143 unmodified lines
901919
902920 #define V2_CQ_DB_PARAMETER_NOTIFY_S 24
903921
922+struct hns_roce_v2_ud_send_wqe {
923+ u32 byte_4;
924+ u32 msg_len;
925+ u32 immtdata;
926+ u32 byte_16;
927+ u32 byte_20;
928+ u32 byte_24;
929+ u32 qkey;
930+ u32 byte_32;
931+ u32 byte_36;
932+ u32 byte_40;
933+ u32 dmac;
934+ u32 byte_48;
935+ u8 dgid[GID_LEN_V2];
936+
937+};
938+#define V2_UD_SEND_WQE_BYTE_4_OPCODE_S 0
939+#define V2_UD_SEND_WQE_BYTE_4_OPCODE_M GENMASK(4, 0)
940+
941+#define V2_UD_SEND_WQE_BYTE_4_OWNER_S 7
942+
943+#define V2_UD_SEND_WQE_BYTE_4_CQE_S 8
944+
945+#define V2_UD_SEND_WQE_BYTE_4_SE_S 11
946+
947+#define V2_UD_SEND_WQE_BYTE_16_PD_S 0
948+#define V2_UD_SEND_WQE_BYTE_16_PD_M GENMASK(23, 0)
949+
950+#define V2_UD_SEND_WQE_BYTE_16_SGE_NUM_S 24
951+#define V2_UD_SEND_WQE_BYTE_16_SGE_NUM_M GENMASK(31, 24)
952+
953+#define V2_UD_SEND_WQE_BYTE_20_MSG_START_SGE_IDX_S 0
954+#define V2_UD_SEND_WQE_BYTE_20_MSG_START_SGE_IDX_M GENMASK(23, 0)
955+
956+#define V2_UD_SEND_WQE_BYTE_24_UDPSPN_S 16
957+#define V2_UD_SEND_WQE_BYTE_24_UDPSPN_M GENMASK(31, 16)
958+
959+#define V2_UD_SEND_WQE_BYTE_32_DQPN_S 0
960+#define V2_UD_SEND_WQE_BYTE_32_DQPN_M GENMASK(23, 0)
961+
962+#define V2_UD_SEND_WQE_BYTE_36_VLAN_S 0
963+#define V2_UD_SEND_WQE_BYTE_36_VLAN_M GENMASK(15, 0)
964+
965+#define V2_UD_SEND_WQE_BYTE_36_HOPLIMIT_S 16
966+#define V2_UD_SEND_WQE_BYTE_36_HOPLIMIT_M GENMASK(23, 16)
967+
968+#define V2_UD_SEND_WQE_BYTE_36_TCLASS_S 24
969+#define V2_UD_SEND_WQE_BYTE_36_TCLASS_M GENMASK(31, 24)
970+
971+#define V2_UD_SEND_WQE_BYTE_40_FLOW_LABEL_S 0
972+#define V2_UD_SEND_WQE_BYTE_40_FLOW_LABEL_M GENMASK(19, 0)
973+
974+#define V2_UD_SEND_WQE_BYTE_40_SL_S 20
975+#define V2_UD_SEND_WQE_BYTE_40_SL_M GENMASK(23, 20)
976+
977+#define V2_UD_SEND_WQE_BYTE_40_PORTN_S 24
978+#define V2_UD_SEND_WQE_BYTE_40_PORTN_M GENMASK(26, 24)
979+
980+#define V2_UD_SEND_WQE_BYTE_40_LBI_S 31
981+
982+#define V2_UD_SEND_WQE_DMAC_0_S 0
983+#define V2_UD_SEND_WQE_DMAC_0_M GENMASK(7, 0)
984+
985+#define V2_UD_SEND_WQE_DMAC_1_S 8
986+#define V2_UD_SEND_WQE_DMAC_1_M GENMASK(15, 8)
987+
988+#define V2_UD_SEND_WQE_DMAC_2_S 16
989+#define V2_UD_SEND_WQE_DMAC_2_M GENMASK(23, 16)
990+
991+#define V2_UD_SEND_WQE_DMAC_3_S 24
992+#define V2_UD_SEND_WQE_DMAC_3_M GENMASK(31, 24)
993+
994+#define V2_UD_SEND_WQE_BYTE_48_DMAC_4_S 0
995+#define V2_UD_SEND_WQE_BYTE_48_DMAC_4_M GENMASK(7, 0)
996+
997+#define V2_UD_SEND_WQE_BYTE_48_DMAC_5_S 8
998+#define V2_UD_SEND_WQE_BYTE_48_DMAC_5_M GENMASK(15, 8)
999+
1000+#define V2_UD_SEND_WQE_BYTE_48_SGID_INDX_S 16
1001+#define V2_UD_SEND_WQE_BYTE_48_SGID_INDX_M GENMASK(23, 16)
1002+
1003+#define V2_UD_SEND_WQE_BYTE_48_SMAC_INDX_S 24
1004+#define V2_UD_SEND_WQE_BYTE_48_SMAC_INDX_M GENMASK(31, 24)
1005+
9041006 struct hns_roce_v2_rc_send_wqe {
9051007 u32 byte_4;
9061008 u32 msg_len;
@@ -1129,9 +1231,6 @@ struct hns_roce_cmq_desc { · 222 unmodified lines
11291231 u32 data[6];
11301232 };
11311233
1132−#define ROCEE_VF_MB_CFG0_REG 0x40
1133−#define ROCEE_VF_MB_STATUS_REG 0x58
1134−
11351234 #define HNS_ROCE_V2_GO_BIT_TIMEOUT_MSECS 10000
11361235
11371236 #define HNS_ROCE_HW_RUN_BIT_SHIFT 31
@@ -1174,4 +1273,178 @@ struct hns_roce_v2_priv { · 36 unmodified lines
11741273 struct hns_roce_v2_cmq cmq;
11751274 };
11761275
1276+struct hns_roce_eq_context {
1277+ u32 byte_4;
1278+ u32 byte_8;
1279+ u32 byte_12;
1280+ u32 eqe_report_timer;
1281+ u32 eqe_ba0;
1282+ u32 eqe_ba1;
1283+ u32 byte_28;
1284+ u32 byte_32;
1285+ u32 byte_36;
1286+ u32 nxt_eqe_ba0;
1287+ u32 nxt_eqe_ba1;
1288+ u32 rsv[5];
1289+};
1290+
1291+#define HNS_ROCE_AEQ_DEFAULT_BURST_NUM 0x0
1292+#define HNS_ROCE_AEQ_DEFAULT_INTERVAL 0x0
1293+#define HNS_ROCE_CEQ_DEFAULT_BURST_NUM 0x0
1294+#define HNS_ROCE_CEQ_DEFAULT_INTERVAL 0x0
1295+
1296+#define HNS_ROCE_V2_EQ_STATE_INVALID 0
1297+#define HNS_ROCE_V2_EQ_STATE_VALID 1
1298+#define HNS_ROCE_V2_EQ_STATE_OVERFLOW 2
1299+#define HNS_ROCE_V2_EQ_STATE_FAILURE 3
1300+
1301+#define HNS_ROCE_V2_EQ_OVER_IGNORE_0 0
1302+#define HNS_ROCE_V2_EQ_OVER_IGNORE_1 1
1303+
1304+#define HNS_ROCE_V2_EQ_COALESCE_0 0
1305+#define HNS_ROCE_V2_EQ_COALESCE_1 1
1306+
1307+#define HNS_ROCE_V2_EQ_FIRED 0
1308+#define HNS_ROCE_V2_EQ_ARMED 1
1309+#define HNS_ROCE_V2_EQ_ALWAYS_ARMED 3
1310+
1311+#define HNS_ROCE_EQ_INIT_EQE_CNT 0
1312+#define HNS_ROCE_EQ_INIT_PROD_IDX 0
1313+#define HNS_ROCE_EQ_INIT_REPORT_TIMER 0
1314+#define HNS_ROCE_EQ_INIT_MSI_IDX 0
1315+#define HNS_ROCE_EQ_INIT_CONS_IDX 0
1316+#define HNS_ROCE_EQ_INIT_NXT_EQE_BA 0
1317+
1318+#define HNS_ROCE_V2_CEQ_CEQE_OWNER_S 31
1319+#define HNS_ROCE_V2_AEQ_AEQE_OWNER_S 31
1320+
1321+#define HNS_ROCE_V2_COMP_EQE_NUM 0x1000
1322+#define HNS_ROCE_V2_ASYNC_EQE_NUM 0x1000
1323+
1324+#define HNS_ROCE_V2_VF_INT_ST_AEQ_OVERFLOW_S 0
1325+#define HNS_ROCE_V2_VF_INT_ST_BUS_ERR_S 1
1326+#define HNS_ROCE_V2_VF_INT_ST_OTHER_ERR_S 2
1327+
1328+#define HNS_ROCE_EQ_DB_CMD_AEQ 0x0
1329+#define HNS_ROCE_EQ_DB_CMD_AEQ_ARMED 0x1
1330+#define HNS_ROCE_EQ_DB_CMD_CEQ 0x2
1331+#define HNS_ROCE_EQ_DB_CMD_CEQ_ARMED 0x3
1332+
1333+#define EQ_ENABLE 1
1334+#define EQ_DISABLE 0
1335+
1336+#define EQ_REG_OFFSET 0x4
1337+
1338+#define HNS_ROCE_INT_NAME_LEN 32
1339+#define HNS_ROCE_V2_EQN_M GENMASK(23, 0)
1340+
1341+#define HNS_ROCE_V2_CONS_IDX_M GENMASK(23, 0)
1342+
1343+#define HNS_ROCE_V2_VF_ABN_INT_EN_S 0
1344+#define HNS_ROCE_V2_VF_ABN_INT_EN_M GENMASK(0, 0)
1345+#define HNS_ROCE_V2_VF_ABN_INT_ST_M GENMASK(2, 0)
1346+#define HNS_ROCE_V2_VF_ABN_INT_CFG_M GENMASK(2, 0)
1347+#define HNS_ROCE_V2_VF_EVENT_INT_EN_M GENMASK(0, 0)
1348+
1349+/* WORD0 */
1350+#define HNS_ROCE_EQC_EQ_ST_S 0
1351+#define HNS_ROCE_EQC_EQ_ST_M GENMASK(1, 0)
1352+
1353+#define HNS_ROCE_EQC_HOP_NUM_S 2
1354+#define HNS_ROCE_EQC_HOP_NUM_M GENMASK(3, 2)
1355+
1356+#define HNS_ROCE_EQC_OVER_IGNORE_S 4
1357+#define HNS_ROCE_EQC_OVER_IGNORE_M GENMASK(4, 4)
1358+
1359+#define HNS_ROCE_EQC_COALESCE_S 5
1360+#define HNS_ROCE_EQC_COALESCE_M GENMASK(5, 5)
1361+
1362+#define HNS_ROCE_EQC_ARM_ST_S 6
1363+#define HNS_ROCE_EQC_ARM_ST_M GENMASK(7, 6)
1364+
1365+#define HNS_ROCE_EQC_EQN_S 8
1366+#define HNS_ROCE_EQC_EQN_M GENMASK(15, 8)
1367+
1368+#define HNS_ROCE_EQC_EQE_CNT_S 16
1369+#define HNS_ROCE_EQC_EQE_CNT_M GENMASK(31, 16)
1370+
1371+/* WORD1 */
1372+#define HNS_ROCE_EQC_BA_PG_SZ_S 0
1373+#define HNS_ROCE_EQC_BA_PG_SZ_M GENMASK(3, 0)
1374+
1375+#define HNS_ROCE_EQC_BUF_PG_SZ_S 4
1376+#define HNS_ROCE_EQC_BUF_PG_SZ_M GENMASK(7, 4)
1377+
1378+#define HNS_ROCE_EQC_PROD_INDX_S 8
1379+#define HNS_ROCE_EQC_PROD_INDX_M GENMASK(31, 8)
1380+
1381+/* WORD2 */
1382+#define HNS_ROCE_EQC_MAX_CNT_S 0
1383+#define HNS_ROCE_EQC_MAX_CNT_M GENMASK(15, 0)
1384+
1385+#define HNS_ROCE_EQC_PERIOD_S 16
1386+#define HNS_ROCE_EQC_PERIOD_M GENMASK(31, 16)
1387+
1388+/* WORD3 */
1389+#define HNS_ROCE_EQC_REPORT_TIMER_S 0
1390+#define HNS_ROCE_EQC_REPORT_TIMER_M GENMASK(31, 0)
1391+
1392+/* WORD4 */
1393+#define HNS_ROCE_EQC_EQE_BA_L_S 0
1394+#define HNS_ROCE_EQC_EQE_BA_L_M GENMASK(31, 0)
1395+
1396+/* WORD5 */
1397+#define HNS_ROCE_EQC_EQE_BA_H_S 0
1398+#define HNS_ROCE_EQC_EQE_BA_H_M GENMASK(28, 0)
1399+
1400+/* WORD6 */
1401+#define HNS_ROCE_EQC_SHIFT_S 0
1402+#define HNS_ROCE_EQC_SHIFT_M GENMASK(7, 0)
1403+
1404+#define HNS_ROCE_EQC_MSI_INDX_S 8
1405+#define HNS_ROCE_EQC_MSI_INDX_M GENMASK(15, 8)
1406+
1407+#define HNS_ROCE_EQC_CUR_EQE_BA_L_S 16
1408+#define HNS_ROCE_EQC_CUR_EQE_BA_L_M GENMASK(31, 16)
1409+
1410+/* WORD7 */
1411+#define HNS_ROCE_EQC_CUR_EQE_BA_M_S 0
1412+#define HNS_ROCE_EQC_CUR_EQE_BA_M_M GENMASK(31, 0)
1413+
1414+/* WORD8 */
1415+#define HNS_ROCE_EQC_CUR_EQE_BA_H_S 0
1416+#define HNS_ROCE_EQC_CUR_EQE_BA_H_M GENMASK(3, 0)
1417+
1418+#define HNS_ROCE_EQC_CONS_INDX_S 8
1419+#define HNS_ROCE_EQC_CONS_INDX_M GENMASK(31, 8)
1420+
1421+/* WORD9 */
1422+#define HNS_ROCE_EQC_NXT_EQE_BA_L_S 0
1423+#define HNS_ROCE_EQC_NXT_EQE_BA_L_M GENMASK(31, 0)
1424+
1425+/* WORD10 */
1426+#define HNS_ROCE_EQC_NXT_EQE_BA_H_S 0
1427+#define HNS_ROCE_EQC_NXT_EQE_BA_H_M GENMASK(19, 0)
1428+
1429+#define HNS_ROCE_V2_CEQE_COMP_CQN_S 0
1430+#define HNS_ROCE_V2_CEQE_COMP_CQN_M GENMASK(23, 0)
1431+
1432+#define HNS_ROCE_V2_AEQE_EVENT_TYPE_S 0
1433+#define HNS_ROCE_V2_AEQE_EVENT_TYPE_M GENMASK(7, 0)
1434+
1435+#define HNS_ROCE_V2_AEQE_SUB_TYPE_S 8
1436+#define HNS_ROCE_V2_AEQE_SUB_TYPE_M GENMASK(15, 8)
1437+
1438+#define HNS_ROCE_V2_EQ_DB_CMD_S 16
1439+#define HNS_ROCE_V2_EQ_DB_CMD_M GENMASK(17, 16)
1440+
1441+#define HNS_ROCE_V2_EQ_DB_TAG_S 0
1442+#define HNS_ROCE_V2_EQ_DB_TAG_M GENMASK(7, 0)
1443+
1444+#define HNS_ROCE_V2_EQ_DB_PARA_S 0
1445+#define HNS_ROCE_V2_EQ_DB_PARA_M GENMASK(23, 0)
1446+
1447+#define HNS_ROCE_V2_AEQE_EVENT_QUEUE_NUM_S 0
1448+#define HNS_ROCE_V2_AEQE_EVENT_QUEUE_NUM_M GENMASK(23, 0)
1449+
11771450 #endif
drivers/infiniband/hw/hns/hns_roce_main.c+6−10
@@ -748,12 +748,10 @@ int hns_roce_init(struct hns_roce_dev *hr_dev) · 747 unmodified lines
748748 goto error_failed_cmd_init;
749749 }
750750
751− if (hr_dev->cmd_mod) {
752− ret = hns_roce_init_eq_table(hr_dev);
753− if (ret) {
754− dev_err(dev, "eq init failed!\n");
755− goto error_failed_eq_table;
756− }
751+ ret = hr_dev->hw->init_eq(hr_dev);
752+ if (ret) {
753+ dev_err(dev, "eq init failed!\n");
754+ goto error_failed_eq_table;
757755 }
758756
759757 if (hr_dev->cmd_mod) {
@@ -805,8 +803,7 @@ error_failed_init_hem: · 45 unmodified lines
805803 hns_roce_cmd_use_polling(hr_dev);
806804
807805 error_failed_use_event:
808− if (hr_dev->cmd_mod)
809− hns_roce_cleanup_eq_table(hr_dev);
806+ hr_dev->hw->cleanup_eq(hr_dev);
810807
811808 error_failed_eq_table:
812809 hns_roce_cmd_cleanup(hr_dev);
@@ -837,8 +834,7 @@ void hns_roce_exit(struct hns_roce_dev *hr_dev) · 24 unmodified lines
837834 if (hr_dev->cmd_mod)
838835 hns_roce_cmd_use_polling(hr_dev);
839836
840− if (hr_dev->cmd_mod)
841− hns_roce_cleanup_eq_table(hr_dev);
837+ hr_dev->hw->cleanup_eq(hr_dev);
842838 hns_roce_cmd_cleanup(hr_dev);
843839 if (hr_dev->hw->cmq_exit)
844840 hr_dev->hw->cmq_exit(hr_dev);
drivers/infiniband/hw/hns/hns_roce_qp.c+63−9
@@ -65,6 +65,7 @@ void hns_roce_qp_event(struct hns_roce_dev *hr_dev, u32 qpn, int event_type) · 64 unmodified lines
6565 if (atomic_dec_and_test(&qp->refcount))
6666 complete(&qp->free);
6767 }
68+EXPORT_SYMBOL_GPL(hns_roce_qp_event);
6869
6970 static void hns_roce_ib_qp_event(struct hns_roce_qp *hr_qp,
7071 enum hns_roce_event type)
@@ -454,6 +455,13 @@ static int hns_roce_set_kernel_sq_size(struct hns_roce_dev *hr_dev, · 383 unmodified lines
454455 hr_qp->sge.sge_shift = 4;
455456 }
456457
458+ /* ud sqwqe's sge use extend sge */
459+ if (hr_dev->caps.max_sq_sg > 2 && hr_qp->ibqp.qp_type == IB_QPT_GSI) {
460+ hr_qp->sge.sge_cnt = roundup_pow_of_two(hr_qp->sq.wqe_cnt *
461+ hr_qp->sq.max_gs);
462+ hr_qp->sge.sge_shift = 4;
463+ }
464+
457465 /* Get buf size, SQ and RQ are aligned to PAGE_SIZE */
458466 page_size = 1 << (hr_dev->caps.mtt_buf_pg_sz + PAGE_SHIFT);
459467 hr_qp->sq.offset = 0;
@@ -493,6 +501,7 @@ static int hns_roce_create_qp_common(struct hns_roce_dev *hr_dev, · 33 unmodified lines
493501 int ret = 0;
494502 u32 page_shift;
495503 u32 npages;
504+ int i;
496505
497506 mutex_init(&hr_qp->mutex);
498507 spin_lock_init(&hr_qp->sq.lock);
@@ -500,6 +509,8 @@ static int hns_roce_create_qp_common(struct hns_roce_dev *hr_dev, · 1 unmodified line
500509
501510 hr_qp->state = IB_QPS_RESET;
502511
512+ hr_qp->ibqp.qp_type = init_attr->qp_type;
513+
503514 if (init_attr->sq_sig_type == IB_SIGNAL_ALL_WR)
504515 hr_qp->sq_signal_bits = IB_SIGNAL_ALL_WR;
505516 else
@@ -512,18 +523,48 @@ static int hns_roce_create_qp_common(struct hns_roce_dev *hr_dev, · 6 unmodified lines
512523 goto err_out;
513524 }
514525
526+ if (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_RQ_INLINE) {
527+ /* allocate recv inline buf */
528+ hr_qp->rq_inl_buf.wqe_list = kcalloc(hr_qp->rq.wqe_cnt,
529+ sizeof(struct hns_roce_rinl_wqe),
530+ GFP_KERNEL);
531+ if (!hr_qp->rq_inl_buf.wqe_list) {
532+ ret = -ENOMEM;
533+ goto err_out;
534+ }
535+
536+ hr_qp->rq_inl_buf.wqe_cnt = hr_qp->rq.wqe_cnt;
537+
538+ /* Firstly, allocate a list of sge space buffer */
539+ hr_qp->rq_inl_buf.wqe_list[0].sg_list =
540+ kcalloc(hr_qp->rq_inl_buf.wqe_cnt,
541+ init_attr->cap.max_recv_sge *
542+ sizeof(struct hns_roce_rinl_sge),
543+ GFP_KERNEL);
544+ if (!hr_qp->rq_inl_buf.wqe_list[0].sg_list) {
545+ ret = -ENOMEM;
546+ goto err_wqe_list;
547+ }
548+
549+ for (i = 1; i < hr_qp->rq_inl_buf.wqe_cnt; i++)
550+ /* Secondly, reallocate the buffer */
551+ hr_qp->rq_inl_buf.wqe_list[i].sg_list =
552+ &hr_qp->rq_inl_buf.wqe_list[0].sg_list[i *
553+ init_attr->cap.max_recv_sge];
554+ }
555+
515556 if (ib_pd->uobject) {
516557 if (ib_copy_from_udata(&ucmd, udata, sizeof(ucmd))) {
517558 dev_err(dev, "ib_copy_from_udata error for create qp\n");
518559 ret = -EFAULT;
519− goto err_out;
560+ goto err_rq_sge_list;
520561 }
521562
522563 ret = hns_roce_set_user_sq_size(hr_dev, &init_attr->cap, hr_qp,
523564 &ucmd);
524565 if (ret) {
525566 dev_err(dev, "hns_roce_set_user_sq_size error for create qp\n");
526− goto err_out;
567+ goto err_rq_sge_list;
527568 }
528569
529570 hr_qp->umem = ib_umem_get(ib_pd->uobject->context,
@@ -532,7 +573,7 @@ static int hns_roce_create_qp_common(struct hns_roce_dev *hr_dev, · 2 unmodified lines
532573 if (IS_ERR(hr_qp->umem)) {
533574 dev_err(dev, "ib_umem_get error for create qp\n");
534575 ret = PTR_ERR(hr_qp->umem);
535− goto err_out;
576+ goto err_rq_sge_list;
536577 }
537578
538579 hr_qp->mtt.mtt_type = MTT_TYPE_WQE;
@@ -566,13 +607,13 @@ static int hns_roce_create_qp_common(struct hns_roce_dev *hr_dev, · 27 unmodified lines
566607 IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK) {
567608 dev_err(dev, "init_attr->create_flags error!\n");
568609 ret = -EINVAL;
569− goto err_out;
610+ goto err_rq_sge_list;
570611 }
571612
572613 if (init_attr->create_flags & IB_QP_CREATE_IPOIB_UD_LSO) {
573614 dev_err(dev, "init_attr->create_flags error!\n");
574615 ret = -EINVAL;
575− goto err_out;
616+ goto err_rq_sge_list;
576617 }
577618
578619 /* Set SQ size */
@@ -580,7 +621,7 @@ static int hns_roce_create_qp_common(struct hns_roce_dev *hr_dev, · 1 unmodified line
580621 hr_qp);
581622 if (ret) {
582623 dev_err(dev, "hns_roce_set_kernel_sq_size error!\n");
583− goto err_out;
624+ goto err_rq_sge_list;
584625 }
585626
586627 /* QP doorbell register address */
@@ -596,7 +637,7 @@ static int hns_roce_create_qp_common(struct hns_roce_dev *hr_dev, · 9 unmodified lines
596637 &hr_qp->hr_buf, page_shift)) {
597638 dev_err(dev, "hns_roce_buf_alloc error!\n");
598639 ret = -ENOMEM;
599− goto err_out;
640+ goto err_rq_sge_list;
600641 }
601642
602643 hr_qp->mtt.mtt_type = MTT_TYPE_WQE;
@@ -678,6 +719,14 @@ err_buf: · 75 unmodified lines
678719 else
679720 hns_roce_buf_free(hr_dev, hr_qp->buff_size, &hr_qp->hr_buf);
680721
722+err_rq_sge_list:
723+ if (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_RQ_INLINE)
724+ kfree(hr_qp->rq_inl_buf.wqe_list[0].sg_list);
725+
726+err_wqe_list:
727+ if (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_RQ_INLINE)
728+ kfree(hr_qp->rq_inl_buf.wqe_list);
729+
681730 err_out:
682731 return ret;
683732 }
@@ -724,8 +773,13 @@ struct ib_qp *hns_roce_create_qp(struct ib_pd *pd, · 40 unmodified lines
724773 hr_qp = &hr_sqp->hr_qp;
725774 hr_qp->port = init_attr->port_num - 1;
726775 hr_qp->phy_port = hr_dev->iboe.phy_port[hr_qp->port];
727− hr_qp->ibqp.qp_num = HNS_ROCE_MAX_PORTS +
728− hr_dev->iboe.phy_port[hr_qp->port];
776+
777+ /* when hw version is v1, the sqpn is allocated */
778+ if (hr_dev->caps.max_sq_sg <= 2)
779+ hr_qp->ibqp.qp_num = HNS_ROCE_MAX_PORTS +
780+ hr_dev->iboe.phy_port[hr_qp->port];
781+ else
782+ hr_qp->ibqp.qp_num = 1;
729783
730784 ret = hns_roce_create_qp_common(hr_dev, pd, init_attr, udata,
731785 hr_qp->ibqp.qp_num, hr_qp);
drivers/infiniband/hw/i40iw/Kconfig+0−1
@@ -5,4 +5,3 @@ config INFINIBAND_I40IW · 4 unmodified lines
55 select GENERIC_ALLOCATOR
66 ---help---
77 Intel(R) Ethernet X722 iWARP Driver
8− INET && I40IW && INFINIBAND && I40E
drivers/infiniband/hw/i40iw/i40iw.h+3−0
@@ -587,5 +587,8 @@ int i40iw_inet6addr_event(struct notifier_block *notifier, · 586 unmodified lines
587587 int i40iw_net_event(struct notifier_block *notifier,
588588 unsigned long event,
589589 void *ptr);
590+int i40iw_netdevice_event(struct notifier_block *notifier,
591+ unsigned long event,
592+ void *ptr);
590593
591594 #endif
drivers/infiniband/hw/i40iw/i40iw_cm.c+39−29
@@ -92,14 +92,9 @@ void i40iw_free_sqbuf(struct i40iw_sc_vsi *vsi, void *bufp) · 91 unmodified lines
9292 static u8 i40iw_derive_hw_ird_setting(u16 cm_ird)
9393 {
9494 u8 encoded_ird_size;
95− u8 pof2_cm_ird = 1;
96−
97− /* round-off to next powerof2 */
98− while (pof2_cm_ird < cm_ird)
99− pof2_cm_ird *= 2;
10095
10196 /* ird_size field is encoded in qp_ctx */
102− switch (pof2_cm_ird) {
97+ switch (cm_ird ? roundup_pow_of_two(cm_ird) : 0) {
10398 case I40IW_HW_IRD_SETTING_64:
10499 encoded_ird_size = 3;
105100 break;
@@ -125,13 +120,16 @@ static u8 i40iw_derive_hw_ird_setting(u16 cm_ird) · 19 unmodified lines
125120 * @conn_ird: connection IRD
126121 * @conn_ord: connection ORD
127122 */
128−static void i40iw_record_ird_ord(struct i40iw_cm_node *cm_node, u16 conn_ird, u16 conn_ord)
123+static void i40iw_record_ird_ord(struct i40iw_cm_node *cm_node, u32 conn_ird,
124+ u32 conn_ord)
129125 {
130126 if (conn_ird > I40IW_MAX_IRD_SIZE)
131127 conn_ird = I40IW_MAX_IRD_SIZE;
132128
133129 if (conn_ord > I40IW_MAX_ORD_SIZE)
134130 conn_ord = I40IW_MAX_ORD_SIZE;
131+ else if (!conn_ord && cm_node->send_rdma0_op == SEND_RDMA_READ_ZERO)
132+ conn_ord = 1;
135133
136134 cm_node->ird_size = conn_ird;
137135 cm_node->ord_size = conn_ord;
@@ -2878,15 +2876,13 @@ static struct i40iw_cm_listener *i40iw_make_listen_node( · 2740 unmodified lines
28782876 * i40iw_create_cm_node - make a connection node with params
28792877 * @cm_core: cm's core
28802878 * @iwdev: iwarp device structure
2881− * @private_data_len: len to provate data for mpa request
2882− * @private_data: pointer to private data for connection
2879+ * @conn_param: upper layer connection parameters
28832880 * @cm_info: quad info for connection
28842881 */
28852882 static struct i40iw_cm_node *i40iw_create_cm_node(
28862883 struct i40iw_cm_core *cm_core,
28872884 struct i40iw_device *iwdev,
2888− u16 private_data_len,
2889− void *private_data,
2885+ struct iw_cm_conn_param *conn_param,
28902886 struct i40iw_cm_info *cm_info)
28912887 {
28922888 struct i40iw_cm_node *cm_node;
@@ -2894,6 +2890,9 @@ static struct i40iw_cm_node *i40iw_create_cm_node( · 1 unmodified line
28942890 struct i40iw_cm_node *loopback_remotenode;
28952891 struct i40iw_cm_info loopback_cm_info;
28962892
2893+ u16 private_data_len = conn_param->private_data_len;
2894+ const void *private_data = conn_param->private_data;
2895+
28972896 /* create a CM connection node */
28982897 cm_node = i40iw_make_cm_node(cm_core, iwdev, cm_info, NULL);
28992898 if (!cm_node)
@@ -2902,6 +2901,8 @@ static struct i40iw_cm_node *i40iw_create_cm_node( · 2 unmodified lines
29022901 cm_node->tcp_cntxt.client = 1;
29032902 cm_node->tcp_cntxt.rcv_wscale = I40IW_CM_DEFAULT_RCV_WND_SCALE;
29042903
2904+ i40iw_record_ird_ord(cm_node, conn_param->ird, conn_param->ord);
2905+
29052906 if (!memcmp(cm_info->loc_addr, cm_info->rem_addr, sizeof(cm_info->loc_addr))) {
29062907 loopback_remotelistener = i40iw_find_listener(
29072908 cm_core,
@@ -2935,6 +2936,10 @@ static struct i40iw_cm_node *i40iw_create_cm_node( · 27 unmodified lines
29352936 private_data_len);
29362937 loopback_remotenode->pdata.size = private_data_len;
29372938
2939+ if (loopback_remotenode->ord_size > cm_node->ird_size)
2940+ loopback_remotenode->ord_size =
2941+ cm_node->ird_size;
2942+
29382943 cm_node->state = I40IW_CM_STATE_OFFLOADED;
29392944 cm_node->tcp_cntxt.rcv_nxt =
29402945 loopback_remotenode->tcp_cntxt.loc_seq_num;
@@ -3691,7 +3696,7 @@ int i40iw_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) · 750 unmodified lines
36913696 cm_node->qhash_set = false;
36923697 i40iw_modify_qp(&iwqp->ibqp, &attr, IB_QP_STATE, NULL);
36933698
3694− cm_node->accelerated = 1;
3699+ cm_node->accelerated = true;
36953700 status =
36963701 i40iw_send_cm_event(cm_node, cm_id, IW_CM_EVENT_ESTABLISHED, 0);
36973702 if (status)
@@ -3815,9 +3820,7 @@ int i40iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) · 117 unmodified lines
38153820 __func__, cm_id->tos, cm_info.user_pri);
38163821 cm_id->add_ref(cm_id);
38173822 cm_node = i40iw_create_cm_node(&iwdev->cm_core, iwdev,
3818− conn_param->private_data_len,
3819− (void *)conn_param->private_data,
3820− &cm_info);
3823+ conn_param, &cm_info);
38213824
38223825 if (IS_ERR(cm_node)) {
38233826 ret = PTR_ERR(cm_node);
@@ -3849,11 +3852,6 @@ int i40iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) · 25 unmodified lines
38493852 }
38503853
38513854 cm_node->apbvt_set = true;
3852− i40iw_record_ird_ord(cm_node, (u16)conn_param->ird, (u16)conn_param->ord);
3853− if (cm_node->send_rdma0_op == SEND_RDMA_READ_ZERO &&
3854− !cm_node->ord_size)
3855− cm_node->ord_size = 1;
3856−
38573855 iwqp->cm_node = cm_node;
38583856 cm_node->iwqp = iwqp;
38593857 iwqp->cm_id = cm_id;
@@ -4058,7 +4056,7 @@ static void i40iw_cm_event_connected(struct i40iw_cm_event *event) · 198 unmodified lines
40584056 cm_node->qhash_set = false;
40594057 i40iw_modify_qp(&iwqp->ibqp, &attr, IB_QP_STATE, NULL);
40604058
4061− cm_node->accelerated = 1;
4059+ cm_node->accelerated = true;
40624060 status = i40iw_send_cm_event(cm_node, cm_id, IW_CM_EVENT_CONNECT_REPLY,
40634061 0);
40644062 if (status)
@@ -4242,10 +4240,16 @@ set_qhash: · 177 unmodified lines
42424240 }
42434241
42444242 /**
4245− * i40iw_cm_disconnect_all - disconnect all connected qp's
4243+ * i40iw_cm_teardown_connections - teardown QPs
42464244 * @iwdev: device pointer
4245+ * @ipaddr: Pointer to IPv4 or IPv6 address
4246+ * @ipv4: flag indicating IPv4 when true
4247+ * @disconnect_all: flag indicating disconnect all QPs
4248+ * teardown QPs where source or destination addr matches ip addr
42474249 */
4248−void i40iw_cm_disconnect_all(struct i40iw_device *iwdev)
4250+void i40iw_cm_teardown_connections(struct i40iw_device *iwdev, u32 *ipaddr,
4251+ struct i40iw_cm_info *nfo,
4252+ bool disconnect_all)
42494253 {
42504254 struct i40iw_cm_core *cm_core = &iwdev->cm_core;
42514255 struct list_head *list_core_temp;
@@ -4259,8 +4263,13 @@ void i40iw_cm_disconnect_all(struct i40iw_device *iwdev) · 7 unmodified lines
42594263 spin_lock_irqsave(&cm_core->ht_lock, flags);
42604264 list_for_each_safe(list_node, list_core_temp, &cm_core->connected_nodes) {
42614265 cm_node = container_of(list_node, struct i40iw_cm_node, list);
4262− atomic_inc(&cm_node->ref_count);
4263− list_add(&cm_node->connected_entry, &connected_list);
4266+ if (disconnect_all ||
4267+ (nfo->vlan_id == cm_node->vlan_id &&
4268+ (!memcmp(cm_node->loc_addr, ipaddr, nfo->ipv4 ? 4 : 16) ||
4269+ !memcmp(cm_node->rem_addr, ipaddr, nfo->ipv4 ? 4 : 16)))) {
4270+ atomic_inc(&cm_node->ref_count);
4271+ list_add(&cm_node->connected_entry, &connected_list);
4272+ }
42644273 }
42654274 spin_unlock_irqrestore(&cm_core->ht_lock, flags);
42664275
@@ -4294,6 +4303,9 @@ void i40iw_if_notify(struct i40iw_device *iwdev, struct net_device *netdev, · 27 unmodified lines
42944303 enum i40iw_quad_hash_manage_type op =
42954304 ifup ? I40IW_QHASH_MANAGE_TYPE_ADD : I40IW_QHASH_MANAGE_TYPE_DELETE;
42964305
4306+ nfo.vlan_id = vlan_id;
4307+ nfo.ipv4 = ipv4;
4308+
42974309 /* Disable or enable qhash for listeners */
42984310 spin_lock_irqsave(&cm_core->listen_list_lock, flags);
42994311 list_for_each_entry(listen_node, &cm_core->listen_nodes, list) {
@@ -4303,8 +4315,6 @@ void i40iw_if_notify(struct i40iw_device *iwdev, struct net_device *netdev, · 3 unmodified lines
43034315 memcpy(nfo.loc_addr, listen_node->loc_addr,
43044316 sizeof(nfo.loc_addr));
43054317 nfo.loc_port = listen_node->loc_port;
4306− nfo.ipv4 = listen_node->ipv4;
4307− nfo.vlan_id = listen_node->vlan_id;
43084318 nfo.user_pri = listen_node->user_pri;
43094319 if (!list_empty(&listen_node->child_listen_list)) {
43104320 i40iw_qhash_ctrl(iwdev,
@@ -4326,7 +4336,7 @@ void i40iw_if_notify(struct i40iw_device *iwdev, struct net_device *netdev, · 15 unmodified lines
43264336 }
43274337 spin_unlock_irqrestore(&cm_core->listen_list_lock, flags);
43284338
4329− /* disconnect any connected qp's on ifdown */
4339+ /* teardown connected qp's on ifdown */
43304340 if (!ifup)
4331− i40iw_cm_disconnect_all(iwdev);
4341+ i40iw_cm_teardown_connections(iwdev, ipaddr, &nfo, false);
43324342 }
drivers/infiniband/hw/i40iw/i40iw_cm.h+4−4
@@ -276,8 +276,6 @@ struct i40iw_cm_tcp_context { · 275 unmodified lines
276276 u32 mss;
277277 u8 snd_wscale;
278278 u8 rcv_wscale;
279−
280− struct timeval sent_ts;
281279 };
282280
283281 enum i40iw_cm_listener_state {
@@ -337,7 +335,7 @@ struct i40iw_cm_node { · 53 unmodified lines
337335 u16 mpav2_ird_ord;
338336 struct iw_cm_id *cm_id;
339337 struct list_head list;
340− int accelerated;
338+ bool accelerated;
341339 struct i40iw_cm_listener *listener;
342340 int apbvt_set;
343341 int accept_pend;
@@ -455,5 +453,7 @@ int i40iw_arp_table(struct i40iw_device *iwdev, · 111 unmodified lines
455453
456454 void i40iw_if_notify(struct i40iw_device *iwdev, struct net_device *netdev,
457455 u32 *ipaddr, bool ipv4, bool ifup);
458−void i40iw_cm_disconnect_all(struct i40iw_device *iwdev);
456+void i40iw_cm_teardown_connections(struct i40iw_device *iwdev, u32 *ipaddr,
457+ struct i40iw_cm_info *nfo,
458+ bool disconnect_all);
459459 #endif /* I40IW_CM_H */
drivers/infiniband/hw/i40iw/i40iw_ctrl.c+8−17
@@ -1893,8 +1893,6 @@ static enum i40iw_status_code i40iw_sc_get_next_aeqe(struct i40iw_sc_aeq *aeq, · 1892 unmodified lines
18931893 static enum i40iw_status_code i40iw_sc_repost_aeq_entries(struct i40iw_sc_dev *dev,
18941894 u32 count)
18951895 {
1896− if (count > I40IW_MAX_AEQ_ALLOCATE_COUNT)
1897− return I40IW_ERR_INVALID_SIZE;
18981896
18991897 if (dev->is_pf)
19001898 i40iw_wr32(dev->hw, I40E_PFPE_AEQALLOC, count);
@@ -3872,7 +3870,6 @@ enum i40iw_status_code i40iw_config_fpm_values(struct i40iw_sc_dev *dev, u32 qp_ · 1971 unmodified lines
38723870 struct i40iw_virt_mem virt_mem;
38733871 u32 i, mem_size;
38743872 u32 qpwantedoriginal, qpwanted, mrwanted, pblewanted;
3875− u32 powerof2;
38763873 u64 sd_needed;
38773874 u32 loop_count = 0;
38783875
@@ -3928,8 +3925,10 @@ enum i40iw_status_code i40iw_config_fpm_values(struct i40iw_sc_dev *dev, u32 qp_ · 49 unmodified lines
39283925 hmc_info->hmc_obj[I40IW_HMC_IW_APBVT_ENTRY].cnt = 1;
39293926 hmc_info->hmc_obj[I40IW_HMC_IW_MR].cnt = mrwanted;
39303927
3931− hmc_info->hmc_obj[I40IW_HMC_IW_XF].cnt = I40IW_MAX_WQ_ENTRIES * qpwanted;
3932− hmc_info->hmc_obj[I40IW_HMC_IW_Q1].cnt = 4 * I40IW_MAX_IRD_SIZE * qpwanted;
3928+ hmc_info->hmc_obj[I40IW_HMC_IW_XF].cnt =
3929+ roundup_pow_of_two(I40IW_MAX_WQ_ENTRIES * qpwanted);
3930+ hmc_info->hmc_obj[I40IW_HMC_IW_Q1].cnt =
3931+ roundup_pow_of_two(2 * I40IW_MAX_IRD_SIZE * qpwanted);
39333932 hmc_info->hmc_obj[I40IW_HMC_IW_XFFL].cnt =
39343933 hmc_info->hmc_obj[I40IW_HMC_IW_XF].cnt / hmc_fpm_misc->xf_block_size;
39353934 hmc_info->hmc_obj[I40IW_HMC_IW_Q1FL].cnt =
@@ -3945,16 +3944,10 @@ enum i40iw_status_code i40iw_config_fpm_values(struct i40iw_sc_dev *dev, u32 qp_ · 9 unmodified lines
39453944 if ((loop_count > 1000) ||
39463945 ((!(loop_count % 10)) &&
39473946 (qpwanted > qpwantedoriginal * 2 / 3))) {
3948− if (qpwanted > FPM_MULTIPLIER) {
3949− qpwanted -= FPM_MULTIPLIER;
3950− powerof2 = 1;
3951− while (powerof2 < qpwanted)
3952− powerof2 *= 2;
3953− powerof2 /= 2;
3954− qpwanted = powerof2;
3955− } else {
3956− qpwanted /= 2;
3957− }
3947+ if (qpwanted > FPM_MULTIPLIER)
3948+ qpwanted = roundup_pow_of_two(qpwanted -
3949+ FPM_MULTIPLIER);
3950+ qpwanted >>= 1;
39583951 }
39593952 if (mrwanted > FPM_MULTIPLIER * 10)
39603953 mrwanted -= FPM_MULTIPLIER * 10;
@@ -3962,8 +3955,6 @@ enum i40iw_status_code i40iw_config_fpm_values(struct i40iw_sc_dev *dev, u32 qp_ · 1 unmodified line
39623955 pblewanted -= FPM_MULTIPLIER * 1000;
39633956 } while (sd_needed > hmc_fpm_misc->max_sds && loop_count < 2000);
39643957
3965− sd_needed = i40iw_est_sd(dev, hmc_info);
3966−
39673958 i40iw_debug(dev, I40IW_DEBUG_HMC,
39683959 "loop_cnt=%d, sd_needed=%lld, qpcnt = %d, cqcnt=%d, mrcnt=%d, pblecnt=%d\n",
39693960 loop_count, sd_needed,
drivers/infiniband/hw/i40iw/i40iw_d.h+1−0
@@ -97,6 +97,7 @@ · 96 unmodified lines
9797 #define RDMA_OPCODE_MASK 0x0f
9898 #define RDMA_READ_REQ_OPCODE 1
9999 #define Q2_BAD_FRAME_OFFSET 72
100+#define Q2_FPSN_OFFSET 64
100101 #define CQE_MAJOR_DRV 0x8000
101102
102103 #define I40IW_TERM_SENT 0x01
drivers/infiniband/hw/i40iw/i40iw_hw.c+2−1
@@ -385,6 +385,8 @@ void i40iw_process_aeq(struct i40iw_device *iwdev) · 384 unmodified lines
385385 iwcq->ibcq.event_handler(&ibevent, iwcq->ibcq.cq_context);
386386 }
387387 break;
388+ case I40IW_AE_LLP_DOUBT_REACHABILITY:
389+ break;
388390 case I40IW_AE_PRIV_OPERATION_DENIED:
389391 case I40IW_AE_STAG_ZERO_INVALID:
390392 case I40IW_AE_IB_RREQ_AND_Q1_FULL:
@@ -403,7 +405,6 @@ void i40iw_process_aeq(struct i40iw_device *iwdev) · 12 unmodified lines
403405 case I40IW_AE_LLP_SEGMENT_TOO_SMALL:
404406 case I40IW_AE_LLP_SYN_RECEIVED:
405407 case I40IW_AE_LLP_TOO_MANY_RETRIES:
406− case I40IW_AE_LLP_DOUBT_REACHABILITY:
407408 case I40IW_AE_LCE_QP_CATASTROPHIC:
408409 case I40IW_AE_LCE_FUNCTION_CATASTROPHIC:
409410 case I40IW_AE_LCE_CQ_CATASTROPHIC:
drivers/infiniband/hw/i40iw/i40iw_main.c+10−3
@@ -99,6 +99,10 @@ static struct notifier_block i40iw_net_notifier = { · 98 unmodified lines
9999 .notifier_call = i40iw_net_event
100100 };
101101
102+static struct notifier_block i40iw_netdevice_notifier = {
103+ .notifier_call = i40iw_netdevice_event
104+};
105+
102106 /**
103107 * i40iw_find_i40e_handler - find a handler given a client info
104108 * @ldev: pointer to a client info
@@ -483,6 +487,7 @@ static enum i40iw_status_code i40iw_create_hmc_objs(struct i40iw_device *iwdev, · 378 unmodified lines
483487 for (i = 0; i < IW_HMC_OBJ_TYPE_NUM; i++) {
484488 info.rsrc_type = iw_hmc_obj_types[i];
485489 info.count = dev->hmc_info->hmc_obj[info.rsrc_type].cnt;
490+ info.add_sd_cnt = 0;
486491 status = i40iw_create_hmc_obj_type(dev, &info);
487492 if (status) {
488493 i40iw_pr_err("create obj type %d status = %d\n",
@@ -607,7 +612,7 @@ static enum i40iw_status_code i40iw_create_cqp(struct i40iw_device *iwdev) · 118 unmodified lines
607612 INIT_LIST_HEAD(&cqp->cqp_avail_reqs);
608613 INIT_LIST_HEAD(&cqp->cqp_pending_reqs);
609614 /* init the waitq of the cqp_requests and add them to the list */
610− for (i = 0; i < I40IW_CQP_SW_SQSIZE_2048; i++) {
615+ for (i = 0; i < sqsize; i++) {
611616 init_waitqueue_head(&cqp->cqp_requests[i].waitq);
612617 list_add_tail(&cqp->cqp_requests[i].list, &cqp->cqp_avail_reqs);
613618 }
@@ -1285,7 +1290,7 @@ static void i40iw_wait_pe_ready(struct i40iw_hw *hw) · 671 unmodified lines
12851290 __LINE__, statuscpu2);
12861291 if ((statuscpu0 == 0x80) && (statuscpu1 == 0x80) && (statuscpu2 == 0x80))
12871292 break; /* SUCCESS */
1288− mdelay(1000);
1293+ msleep(1000);
12891294 retrycount++;
12901295 } while (retrycount < 14);
12911296 i40iw_wr32(hw, 0xb4040, 0x4C104C5);
@@ -1393,6 +1398,7 @@ static void i40iw_register_notifiers(void) · 101 unmodified lines
13931398 register_inetaddr_notifier(&i40iw_inetaddr_notifier);
13941399 register_inet6addr_notifier(&i40iw_inetaddr6_notifier);
13951400 register_netevent_notifier(&i40iw_net_notifier);
1401+ register_netdevice_notifier(&i40iw_netdevice_notifier);
13961402 }
13971403
13981404 /**
@@ -1404,6 +1410,7 @@ static void i40iw_unregister_notifiers(void) · 5 unmodified lines
14041410 unregister_netevent_notifier(&i40iw_net_notifier);
14051411 unregister_inetaddr_notifier(&i40iw_inetaddr_notifier);
14061412 unregister_inet6addr_notifier(&i40iw_inetaddr6_notifier);
1413+ unregister_netdevice_notifier(&i40iw_netdevice_notifier);
14071414 }
14081415
14091416 /**
@@ -1793,7 +1800,7 @@ static void i40iw_close(struct i40e_info *ldev, struct i40e_client *client, bool · 383 unmodified lines
17931800 if (reset)
17941801 iwdev->reset = true;
17951802
1796− i40iw_cm_disconnect_all(iwdev);
1803+ i40iw_cm_teardown_connections(iwdev, NULL, NULL, true);
17971804 destroy_workqueue(iwdev->virtchnl_wq);
17981805 i40iw_deinit_device(iwdev);
17991806 }
drivers/infiniband/hw/i40iw/i40iw_puda.c+2−3
@@ -48,7 +48,6 @@ static void i40iw_ieq_tx_compl(struct i40iw_sc_vsi *vsi, void *sqwrid); · 47 unmodified lines
4848 static void i40iw_ilq_putback_rcvbuf(struct i40iw_sc_qp *qp, u32 wqe_idx);
4949 static enum i40iw_status_code i40iw_puda_replenish_rq(struct i40iw_puda_rsrc
5050 *rsrc, bool initial);
51−static void i40iw_ieq_cleanup_qp(struct i40iw_puda_rsrc *ieq, struct i40iw_sc_qp *qp);
5251 /**
5352 * i40iw_puda_get_listbuf - get buffer from puda list
5453 * @list: list to use for buffers (ILQ or IEQ)
@@ -1378,7 +1377,7 @@ static void i40iw_ieq_handle_exception(struct i40iw_puda_rsrc *ieq, · 1323 unmodified lines
13781377 u32 *hw_host_ctx = (u32 *)qp->hw_host_ctx;
13791378 u32 rcv_wnd = hw_host_ctx[23];
13801379 /* first partial seq # in q2 */
1381− u32 fps = qp->q2_buf[16];
1380+ u32 fps = *(u32 *)(qp->q2_buf + Q2_FPSN_OFFSET);
13821381 struct list_head *rxlist = &pfpdu->rxlist;
13831382 struct list_head *plist;
13841383
@@ -1483,7 +1482,7 @@ static void i40iw_ieq_tx_compl(struct i40iw_sc_vsi *vsi, void *sqwrid) · 98 unmodified lines
14831482 * @ieq: ieq resource
14841483 * @qp: all pending fpdu buffers
14851484 */
1486−static void i40iw_ieq_cleanup_qp(struct i40iw_puda_rsrc *ieq, struct i40iw_sc_qp *qp)
1485+void i40iw_ieq_cleanup_qp(struct i40iw_puda_rsrc *ieq, struct i40iw_sc_qp *qp)
14871486 {
14881487 struct i40iw_puda_buf *buf;
14891488 struct i40iw_pfpdu *pfpdu = &qp->pfpdu;
drivers/infiniband/hw/i40iw/i40iw_puda.h+1−0
@@ -184,4 +184,5 @@ enum i40iw_status_code i40iw_cqp_qp_create_cmd(struct i40iw_sc_dev *dev, struct · 183 unmodified lines
184184 enum i40iw_status_code i40iw_cqp_cq_create_cmd(struct i40iw_sc_dev *dev, struct i40iw_sc_cq *cq);
185185 void i40iw_cqp_qp_destroy_cmd(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp);
186186 void i40iw_cqp_cq_destroy_cmd(struct i40iw_sc_dev *dev, struct i40iw_sc_cq *cq);
187+void i40iw_ieq_cleanup_qp(struct i40iw_puda_rsrc *ieq, struct i40iw_sc_qp *qp);
187188 #endif
drivers/infiniband/hw/i40iw/i40iw_uk.c+2−16
@@ -893,20 +893,6 @@ exit: · 892 unmodified lines
893893 return ret_code;
894894 }
895895
896−/**
897− * i40iw_qp_roundup - return round up QP WQ depth
898− * @wqdepth: WQ depth in quantas to round up
899− */
900−static int i40iw_qp_round_up(u32 wqdepth)
901−{
902− int scount = 1;
903−
904− for (wqdepth--; scount <= 16; scount *= 2)
905− wqdepth |= wqdepth >> scount;
906−
907− return ++wqdepth;
908−}
909−
910896 /**
911897 * i40iw_get_wqe_shift - get shift count for maximum wqe size
912898 * @sge: Maximum Scatter Gather Elements wqe
@@ -934,7 +920,7 @@ void i40iw_get_wqe_shift(u32 sge, u32 inline_data, u8 *shift) · 21 unmodified lines
934920 */
935921 enum i40iw_status_code i40iw_get_sqdepth(u32 sq_size, u8 shift, u32 *sqdepth)
936922 {
937− *sqdepth = i40iw_qp_round_up((sq_size << shift) + I40IW_SQ_RSVD);
923+ *sqdepth = roundup_pow_of_two((sq_size << shift) + I40IW_SQ_RSVD);
938924
939925 if (*sqdepth < (I40IW_QP_SW_MIN_WQSIZE << shift))
940926 *sqdepth = I40IW_QP_SW_MIN_WQSIZE << shift;
@@ -953,7 +939,7 @@ enum i40iw_status_code i40iw_get_sqdepth(u32 sq_size, u8 shift, u32 *sqdepth) · 12 unmodified lines
953939 */
954940 enum i40iw_status_code i40iw_get_rqdepth(u32 rq_size, u8 shift, u32 *rqdepth)
955941 {
956− *rqdepth = i40iw_qp_round_up((rq_size << shift) + I40IW_RQ_RSVD);
942+ *rqdepth = roundup_pow_of_two((rq_size << shift) + I40IW_RQ_RSVD);
957943
958944 if (*rqdepth < (I40IW_QP_SW_MIN_WQSIZE << shift))
959945 *rqdepth = I40IW_QP_SW_MIN_WQSIZE << shift;
drivers/infiniband/hw/i40iw/i40iw_user.h+1−2
@@ -59,7 +59,6 @@ enum i40iw_device_capabilities_const { · 58 unmodified lines
5959 I40IW_MAX_CEQ_ENTRIES = 131071,
6060 I40IW_MIN_CQ_SIZE = 1,
6161 I40IW_MAX_CQ_SIZE = 1048575,
62− I40IW_MAX_AEQ_ALLOCATE_COUNT = 255,
6362 I40IW_DB_ID_ZERO = 0,
6463 I40IW_MAX_WQ_FRAGMENT_COUNT = 3,
6564 I40IW_MAX_SGE_RD = 1,
@@ -72,7 +71,7 @@ enum i40iw_device_capabilities_const { · 6 unmodified lines
7271 I40IW_MAX_SQ_PAYLOAD_SIZE = 2145386496,
7372 I40IW_MAX_INLINE_DATA_SIZE = 48,
7473 I40IW_MAX_PUSHMODE_INLINE_DATA_SIZE = 48,
75− I40IW_MAX_IRD_SIZE = 63,
74+ I40IW_MAX_IRD_SIZE = 64,
7675 I40IW_MAX_ORD_SIZE = 127,
7776 I40IW_MAX_WQ_ENTRIES = 2048,
7877 I40IW_Q2_BUFFER_SIZE = (248 + 100),
drivers/infiniband/hw/i40iw/i40iw_utils.c+47−3
@@ -137,7 +137,7 @@ inline u32 i40iw_rd32(struct i40iw_hw *hw, u32 reg) · 136 unmodified lines
137137 }
138138
139139 /**
140− * i40iw_inetaddr_event - system notifier for netdev events
140+ * i40iw_inetaddr_event - system notifier for ipv4 addr events
141141 * @notfier: not used
142142 * @event: event for notifier
143143 * @ptr: if address
@@ -200,7 +200,7 @@ int i40iw_inetaddr_event(struct notifier_block *notifier, · 56 unmodified lines
200200 }
201201
202202 /**
203− * i40iw_inet6addr_event - system notifier for ipv6 netdev events
203+ * i40iw_inet6addr_event - system notifier for ipv6 addr events
204204 * @notfier: not used
205205 * @event: event for notifier
206206 * @ptr: if address
@@ -252,7 +252,7 @@ int i40iw_inet6addr_event(struct notifier_block *notifier, · 45 unmodified lines
252252 }
253253
254254 /**
255− * i40iw_net_event - system notifier for net events
255+ * i40iw_net_event - system notifier for netevents
256256 * @notfier: not used
257257 * @event: event for notifier
258258 * @ptr: neighbor
@@ -296,6 +296,50 @@ int i40iw_net_event(struct notifier_block *notifier, unsigned long event, void * · 37 unmodified lines
296296 return NOTIFY_DONE;
297297 }
298298
299+/**
300+ * i40iw_netdevice_event - system notifier for netdev events
301+ * @notfier: not used
302+ * @event: event for notifier
303+ * @ptr: netdev
304+ */
305+int i40iw_netdevice_event(struct notifier_block *notifier,
306+ unsigned long event,
307+ void *ptr)
308+{
309+ struct net_device *event_netdev;
310+ struct net_device *netdev;
311+ struct i40iw_device *iwdev;
312+ struct i40iw_handler *hdl;
313+
314+ event_netdev = netdev_notifier_info_to_dev(ptr);
315+
316+ hdl = i40iw_find_netdev(event_netdev);
317+ if (!hdl)
318+ return NOTIFY_DONE;
319+
320+ iwdev = &hdl->device;
321+ if (iwdev->init_state < RDMA_DEV_REGISTERED || iwdev->closing)
322+ return NOTIFY_DONE;
323+
324+ netdev = iwdev->ldev->netdev;
325+ if (netdev != event_netdev)
326+ return NOTIFY_DONE;
327+
328+ iwdev->iw_status = 1;
329+
330+ switch (event) {
331+ case NETDEV_DOWN:
332+ iwdev->iw_status = 0;
333+ /* Fall through */
334+ case NETDEV_UP:
335+ i40iw_port_ibevent(iwdev);
336+ break;
337+ default:
338+ break;
339+ }
340+ return NOTIFY_DONE;
341+}
342+
299343 /**
300344 * i40iw_get_cqp_request - get cqp struct
301345 * @cqp: device cqp ptr
drivers/infiniband/hw/i40iw/i40iw_verbs.c+2−3
@@ -412,6 +412,7 @@ void i40iw_free_qp_resources(struct i40iw_device *iwdev, · 411 unmodified lines
412412 {
413413 struct i40iw_pbl *iwpbl = &iwqp->iwpbl;
414414
415+ i40iw_ieq_cleanup_qp(iwdev->vsi.ieq, &iwqp->sc_qp);
415416 i40iw_dealloc_push_page(iwdev, &iwqp->sc_qp);
416417 if (qp_num)
417418 i40iw_free_resource(iwdev, iwdev->allocated_qps, qp_num);
@@ -1637,6 +1638,7 @@ static struct ib_mr *i40iw_alloc_mr(struct ib_pd *pd, · 1219 unmodified lines
16371638 err_code = -EOVERFLOW;
16381639 goto err;
16391640 }
1641+ stag &= ~I40IW_CQPSQ_STAG_KEY_MASK;
16401642 iwmr->stag = stag;
16411643 iwmr->ibmr.rkey = stag;
16421644 iwmr->ibmr.lkey = stag;
@@ -2242,14 +2244,12 @@ static int i40iw_post_send(struct ib_qp *ibqp, · 599 unmodified lines
22422244 info.op.inline_rdma_write.len = ib_wr->sg_list[0].length;
22432245 info.op.inline_rdma_write.rem_addr.tag_off = rdma_wr(ib_wr)->remote_addr;
22442246 info.op.inline_rdma_write.rem_addr.stag = rdma_wr(ib_wr)->rkey;
2245− info.op.inline_rdma_write.rem_addr.len = ib_wr->sg_list->length;
22462247 ret = ukqp->ops.iw_inline_rdma_write(ukqp, &info, false);
22472248 } else {
22482249 info.op.rdma_write.lo_sg_list = (void *)ib_wr->sg_list;
22492250 info.op.rdma_write.num_lo_sges = ib_wr->num_sge;
22502251 info.op.rdma_write.rem_addr.tag_off = rdma_wr(ib_wr)->remote_addr;
22512252 info.op.rdma_write.rem_addr.stag = rdma_wr(ib_wr)->rkey;
2252− info.op.rdma_write.rem_addr.len = ib_wr->sg_list->length;
22532253 ret = ukqp->ops.iw_rdma_write(ukqp, &info, false);
22542254 }
22552255
@@ -2271,7 +2271,6 @@ static int i40iw_post_send(struct ib_qp *ibqp, · 15 unmodified lines
22712271 info.op_type = I40IW_OP_TYPE_RDMA_READ;
22722272 info.op.rdma_read.rem_addr.tag_off = rdma_wr(ib_wr)->remote_addr;
22732273 info.op.rdma_read.rem_addr.stag = rdma_wr(ib_wr)->rkey;
2274− info.op.rdma_read.rem_addr.len = ib_wr->sg_list->length;
22752274 info.op.rdma_read.lo_addr.tag_off = ib_wr->sg_list->addr;
22762275 info.op.rdma_read.lo_addr.stag = ib_wr->sg_list->lkey;
22772276 info.op.rdma_read.lo_addr.len = ib_wr->sg_list->length;
drivers/infiniband/hw/mlx4/cq.c+2−2
@@ -170,7 +170,7 @@ err_buf: · 169 unmodified lines
170170 return err;
171171 }
172172
173−#define CQ_CREATE_FLAGS_SUPPORTED IB_CQ_FLAGS_TIMESTAMP_COMPLETION
173+#define CQ_CREATE_FLAGS_SUPPORTED IB_UVERBS_CQ_FLAGS_TIMESTAMP_COMPLETION
174174 struct ib_cq *mlx4_ib_create_cq(struct ib_device *ibdev,
175175 const struct ib_cq_init_attr *attr,
176176 struct ib_ucontext *context,
@@ -246,7 +246,7 @@ struct ib_cq *mlx4_ib_create_cq(struct ib_device *ibdev, · 69 unmodified lines
246246
247247 err = mlx4_cq_alloc(dev->dev, entries, &cq->buf.mtt, uar,
248248 cq->db.dma, &cq->mcq, vector, 0,
249− !!(cq->create_flags & IB_CQ_FLAGS_TIMESTAMP_COMPLETION));
249+ !!(cq->create_flags & IB_UVERBS_CQ_FLAGS_TIMESTAMP_COMPLETION));
250250 if (err)
251251 goto err_dbmap;
252252
drivers/infiniband/hw/mlx4/main.c+11−8
@@ -589,6 +589,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev, · 588 unmodified lines
589589 if (props->rss_caps.supported_qpts) {
590590 resp.rss_caps.rx_hash_function =
591591 MLX4_IB_RX_HASH_FUNC_TOEPLITZ;
592+
592593 resp.rss_caps.rx_hash_fields_mask =
593594 MLX4_IB_RX_HASH_SRC_IPV4 |
594595 MLX4_IB_RX_HASH_DST_IPV4 |
@@ -598,6 +599,11 @@ static int mlx4_ib_query_device(struct ib_device *ibdev, · 3 unmodified lines
598599 MLX4_IB_RX_HASH_DST_PORT_TCP |
599600 MLX4_IB_RX_HASH_SRC_PORT_UDP |
600601 MLX4_IB_RX_HASH_DST_PORT_UDP;
602+
603+ if (dev->dev->caps.tunnel_offload_mode ==
604+ MLX4_TUNNEL_OFFLOAD_MODE_VXLAN)
605+ resp.rss_caps.rx_hash_fields_mask |=
606+ MLX4_IB_RX_HASH_INNER;
601607 }
602608 }
603609
@@ -2995,9 +3001,8 @@ err_steer_free_bitmap: · 2391 unmodified lines
29953001 kfree(ibdev->ib_uc_qpns_bitmap);
29963002
29973003 err_steer_qp_release:
2998− if (ibdev->steering_support == MLX4_STEERING_MODE_DEVICE_MANAGED)
2999− mlx4_qp_release_range(dev, ibdev->steer_qpn_base,
3000− ibdev->steer_qpn_count);
3004+ mlx4_qp_release_range(dev, ibdev->steer_qpn_base,
3005+ ibdev->steer_qpn_count);
30013006 err_counter:
30023007 for (i = 0; i < ibdev->num_ports; ++i)
30033008 mlx4_ib_delete_counters_table(ibdev, &ibdev->counters_table[i]);
@@ -3102,11 +3107,9 @@ static void mlx4_ib_remove(struct mlx4_dev *dev, void *ibdev_ptr) · 98 unmodified lines
31023107 ibdev->iboe.nb.notifier_call = NULL;
31033108 }
31043109
3105− if (ibdev->steering_support == MLX4_STEERING_MODE_DEVICE_MANAGED) {
3106− mlx4_qp_release_range(dev, ibdev->steer_qpn_base,
3107− ibdev->steer_qpn_count);
3108− kfree(ibdev->ib_uc_qpns_bitmap);
3109− }
3110+ mlx4_qp_release_range(dev, ibdev->steer_qpn_base,
3111+ ibdev->steer_qpn_count);
3112+ kfree(ibdev->ib_uc_qpns_bitmap);
31103113
31113114 iounmap(ibdev->uar_map);
31123115 for (p = 0; p < ibdev->num_ports; ++p)
drivers/infiniband/hw/mlx4/qp.c+18−2
@@ -734,10 +734,24 @@ static int set_qp_rss(struct mlx4_ib_dev *dev, struct mlx4_ib_rss *rss_ctx, · 733 unmodified lines
734734 return (-EOPNOTSUPP);
735735 }
736736
737+ if (ucmd->rx_hash_fields_mask & MLX4_IB_RX_HASH_INNER) {
738+ if (dev->dev->caps.tunnel_offload_mode ==
739+ MLX4_TUNNEL_OFFLOAD_MODE_VXLAN) {
740+ /*
741+ * Hash according to inner headers if exist, otherwise
742+ * according to outer headers.
743+ */
744+ rss_ctx->flags |= MLX4_RSS_BY_INNER_HEADERS_IPONLY;
745+ } else {
746+ pr_debug("RSS Hash for inner headers isn't supported\n");
747+ return (-EOPNOTSUPP);
748+ }
749+ }
750+
737751 return 0;
738752 }
739753
740−static int create_qp_rss(struct mlx4_ib_dev *dev, struct ib_pd *ibpd,
754+static int create_qp_rss(struct mlx4_ib_dev *dev,
741755 struct ib_qp_init_attr *init_attr,
742756 struct mlx4_ib_create_qp_rss *ucmd,
743757 struct mlx4_ib_qp *qp)
@@ -860,7 +874,7 @@ static struct ib_qp *_mlx4_ib_create_qp_rss(struct ib_pd *pd, · 116 unmodified lines
860874 qp->pri.vid = 0xFFFF;
861875 qp->alt.vid = 0xFFFF;
862876
863− err = create_qp_rss(to_mdev(pd->device), pd, init_attr, &ucmd, qp);
877+ err = create_qp_rss(to_mdev(pd->device), init_attr, &ucmd, qp);
864878 if (err) {
865879 kfree(qp);
866880 return ERR_PTR(err);
@@ -1836,6 +1850,8 @@ static int _mlx4_set_path(struct mlx4_ib_dev *dev, · 969 unmodified lines
18361850 mlx4_ib_gid_index_to_real_index(dev, port,
18371851 grh->sgid_index);
18381852
1853+ if (real_sgid_index < 0)
1854+ return real_sgid_index;
18391855 if (real_sgid_index >= dev->dev->caps.gid_table_len[port]) {
18401856 pr_err("sgid_index (%u) too large. max is %d\n",
18411857 real_sgid_index, dev->dev->caps.gid_table_len[port] - 1);
drivers/infiniband/hw/mlx5/cong.c+59−24
@@ -247,21 +247,30 @@ static void mlx5_ib_set_cc_param_mask_val(void *field, int offset, · 246 unmodified lines
247247 }
248248 }
249249
250−static int mlx5_ib_get_cc_params(struct mlx5_ib_dev *dev, int offset, u32 *var)
250+static int mlx5_ib_get_cc_params(struct mlx5_ib_dev *dev, u8 port_num,
251+ int offset, u32 *var)
251252 {
252253 int outlen = MLX5_ST_SZ_BYTES(query_cong_params_out);
253254 void *out;
254255 void *field;
255256 int err;
256257 enum mlx5_ib_cong_node_type node;
258+ struct mlx5_core_dev *mdev;
259+
260+ /* Takes a 1-based port number */
261+ mdev = mlx5_ib_get_native_port_mdev(dev, port_num + 1, NULL);
262+ if (!mdev)
263+ return -ENODEV;
257264
258265 out = kvzalloc(outlen, GFP_KERNEL);
259− if (!out)
260− return -ENOMEM;
266+ if (!out) {
267+ err = -ENOMEM;
268+ goto alloc_err;
269+ }
261270
262271 node = mlx5_ib_param_to_node(offset);
263272
264− err = mlx5_cmd_query_cong_params(dev->mdev, node, out, outlen);
273+ err = mlx5_cmd_query_cong_params(mdev, node, out, outlen);
265274 if (err)
266275 goto free;
267276
@@ -270,21 +279,32 @@ static int mlx5_ib_get_cc_params(struct mlx5_ib_dev *dev, int offset, u32 *var) · 2 unmodified lines
270279
271280 free:
272281 kvfree(out);
282+alloc_err:
283+ mlx5_ib_put_native_port_mdev(dev, port_num + 1);
273284 return err;
274285 }
275286
276−static int mlx5_ib_set_cc_params(struct mlx5_ib_dev *dev, int offset, u32 var)
287+static int mlx5_ib_set_cc_params(struct mlx5_ib_dev *dev, u8 port_num,
288+ int offset, u32 var)
277289 {
278290 int inlen = MLX5_ST_SZ_BYTES(modify_cong_params_in);
279291 void *in;
280292 void *field;
281293 enum mlx5_ib_cong_node_type node;
294+ struct mlx5_core_dev *mdev;
282295 u32 attr_mask = 0;
283296 int err;
284297
298+ /* Takes a 1-based port number */
299+ mdev = mlx5_ib_get_native_port_mdev(dev, port_num + 1, NULL);
300+ if (!mdev)
301+ return -ENODEV;
302+
285303 in = kvzalloc(inlen, GFP_KERNEL);
286− if (!in)
287− return -ENOMEM;
304+ if (!in) {
305+ err = -ENOMEM;
306+ goto alloc_err;
307+ }
288308
289309 MLX5_SET(modify_cong_params_in, in, opcode,
290310 MLX5_CMD_OP_MODIFY_CONG_PARAMS);
@@ -299,8 +319,10 @@ static int mlx5_ib_set_cc_params(struct mlx5_ib_dev *dev, int offset, u32 var) · 8 unmodified lines
299319 MLX5_SET(field_select_r_roce_rp, field, field_select_r_roce_rp,
300320 attr_mask);
301321
302− err = mlx5_cmd_modify_cong_params(dev->mdev, in, inlen);
322+ err = mlx5_cmd_modify_cong_params(mdev, in, inlen);
303323 kvfree(in);
324+alloc_err:
325+ mlx5_ib_put_native_port_mdev(dev, port_num + 1);
304326 return err;
305327 }
306328
@@ -324,7 +346,7 @@ static ssize_t set_param(struct file *filp, const char __user *buf, · 17 unmodified lines
324346 if (kstrtou32(lbuf, 0, &var))
325347 return -EINVAL;
326348
327− ret = mlx5_ib_set_cc_params(param->dev, offset, var);
349+ ret = mlx5_ib_set_cc_params(param->dev, param->port_num, offset, var);
328350 return ret ? ret : count;
329351 }
330352
@@ -340,7 +362,7 @@ static ssize_t get_param(struct file *filp, char __user *buf, size_t count, · 9 unmodified lines
340362 if (*pos)
341363 return 0;
342364
343− ret = mlx5_ib_get_cc_params(param->dev, offset, &var);
365+ ret = mlx5_ib_get_cc_params(param->dev, param->port_num, offset, &var);
344366 if (ret)
345367 return ret;
346368
@@ -362,44 +384,51 @@ static const struct file_operations dbg_cc_fops = { · 15 unmodified lines
362384 .read = get_param,
363385 };
364386
365−void mlx5_ib_cleanup_cong_debugfs(struct mlx5_ib_dev *dev)
387+void mlx5_ib_cleanup_cong_debugfs(struct mlx5_ib_dev *dev, u8 port_num)
366388 {
367389 if (!mlx5_debugfs_root ||
368− !dev->dbg_cc_params ||
369− !dev->dbg_cc_params->root)
390+ !dev->port[port_num].dbg_cc_params ||
391+ !dev->port[port_num].dbg_cc_params->root)
370392 return;
371393
372− debugfs_remove_recursive(dev->dbg_cc_params->root);
373− kfree(dev->dbg_cc_params);
374− dev->dbg_cc_params = NULL;
394+ debugfs_remove_recursive(dev->port[port_num].dbg_cc_params->root);
395+ kfree(dev->port[port_num].dbg_cc_params);
396+ dev->port[port_num].dbg_cc_params = NULL;
375397 }
376398
377−int mlx5_ib_init_cong_debugfs(struct mlx5_ib_dev *dev)
399+int mlx5_ib_init_cong_debugfs(struct mlx5_ib_dev *dev, u8 port_num)
378400 {
379401 struct mlx5_ib_dbg_cc_params *dbg_cc_params;
402+ struct mlx5_core_dev *mdev;
380403 int i;
381404
382405 if (!mlx5_debugfs_root)
383406 goto out;
384407
385− if (!MLX5_CAP_GEN(dev->mdev, cc_query_allowed) ||
386− !MLX5_CAP_GEN(dev->mdev, cc_modify_allowed))
408+ /* Takes a 1-based port number */
409+ mdev = mlx5_ib_get_native_port_mdev(dev, port_num + 1, NULL);
410+ if (!mdev)
387411 goto out;
388412
413+ if (!MLX5_CAP_GEN(mdev, cc_query_allowed) ||
414+ !MLX5_CAP_GEN(mdev, cc_modify_allowed))
415+ goto put_mdev;
416+
389417 dbg_cc_params = kzalloc(sizeof(*dbg_cc_params), GFP_KERNEL);
390418 if (!dbg_cc_params)
391− goto out;
419+ goto err;
392420
393− dev->dbg_cc_params = dbg_cc_params;
421+ dev->port[port_num].dbg_cc_params = dbg_cc_params;
394422
395423 dbg_cc_params->root = debugfs_create_dir("cc_params",
396− dev->mdev->priv.dbg_root);
424+ mdev->priv.dbg_root);
397425 if (!dbg_cc_params->root)
398426 goto err;
399427
400428 for (i = 0; i < MLX5_IB_DBG_CC_MAX; i++) {
401429 dbg_cc_params->params[i].offset = i;
402430 dbg_cc_params->params[i].dev = dev;
431+ dbg_cc_params->params[i].port_num = port_num;
403432 dbg_cc_params->params[i].dentry =
404433 debugfs_create_file(mlx5_ib_dbg_cc_name[i],
405434 0600, dbg_cc_params->root,
@@ -408,11 +437,17 @@ int mlx5_ib_init_cong_debugfs(struct mlx5_ib_dev *dev) · 2 unmodified lines
408437 if (!dbg_cc_params->params[i].dentry)
409438 goto err;
410439 }
411−out: return 0;
440+
441+put_mdev:
442+ mlx5_ib_put_native_port_mdev(dev, port_num + 1);
443+out:
444+ return 0;
412445
413446 err:
414447 mlx5_ib_warn(dev, "cong debugfs failure\n");
415− mlx5_ib_cleanup_cong_debugfs(dev);
448+ mlx5_ib_cleanup_cong_debugfs(dev, port_num);
449+ mlx5_ib_put_native_port_mdev(dev, port_num + 1);
450+
416451 /*
417452 * We don't want to fail driver if debugfs failed to initialize,
418453 * so we are not forwarding error to the user.
drivers/infiniband/hw/mlx5/cq.c+1−1
@@ -1010,7 +1010,7 @@ struct ib_cq *mlx5_ib_create_cq(struct ib_device *ibdev, · 1009 unmodified lines
10101010 MLX5_SET(cqc, cqc, uar_page, index);
10111011 MLX5_SET(cqc, cqc, c_eqn, eqn);
10121012 MLX5_SET64(cqc, cqc, dbr_addr, cq->db.dma);
1013− if (cq->create_flags & IB_CQ_FLAGS_IGNORE_OVERRUN)
1013+ if (cq->create_flags & IB_UVERBS_CQ_FLAGS_IGNORE_OVERRUN)
10141014 MLX5_SET(cqc, cqc, oi, 1);
10151015
10161016 err = mlx5_core_create_cq(dev->mdev, &cq->mcq, cqb, inlen);
drivers/infiniband/hw/mlx5/mad.c+15−8
@@ -197,10 +197,9 @@ static void pma_cnt_assign(struct ib_pma_portcounters *pma_cnt, · 196 unmodified lines
197197 vl_15_dropped);
198198 }
199199
200−static int process_pma_cmd(struct ib_device *ibdev, u8 port_num,
200+static int process_pma_cmd(struct mlx5_core_dev *mdev, u8 port_num,
201201 const struct ib_mad *in_mad, struct ib_mad *out_mad)
202202 {
203− struct mlx5_ib_dev *dev = to_mdev(ibdev);
204203 int err;
205204 void *out_cnt;
206205
@@ -222,7 +221,7 @@ static int process_pma_cmd(struct ib_device *ibdev, u8 port_num, · 15 unmodified lines
222221 if (!out_cnt)
223222 return IB_MAD_RESULT_FAILURE;
224223
225− err = mlx5_core_query_vport_counter(dev->mdev, 0, 0,
224+ err = mlx5_core_query_vport_counter(mdev, 0, 0,
226225 port_num, out_cnt, sz);
227226 if (!err)
228227 pma_cnt_ext_assign(pma_cnt_ext, out_cnt);
@@ -235,7 +234,7 @@ static int process_pma_cmd(struct ib_device *ibdev, u8 port_num, · 6 unmodified lines
235234 if (!out_cnt)
236235 return IB_MAD_RESULT_FAILURE;
237236
238− err = mlx5_core_query_ib_ppcnt(dev->mdev, port_num,
237+ err = mlx5_core_query_ib_ppcnt(mdev, port_num,
239238 out_cnt, sz);
240239 if (!err)
241240 pma_cnt_assign(pma_cnt, out_cnt);
@@ -255,9 +254,11 @@ int mlx5_ib_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num, · 13 unmodified lines
255254 u16 *out_mad_pkey_index)
256255 {
257256 struct mlx5_ib_dev *dev = to_mdev(ibdev);
258− struct mlx5_core_dev *mdev = dev->mdev;
259257 const struct ib_mad *in_mad = (const struct ib_mad *)in;
260258 struct ib_mad *out_mad = (struct ib_mad *)out;
259+ struct mlx5_core_dev *mdev;
260+ u8 mdev_port_num;
261+ int ret;
261262
262263 if (WARN_ON_ONCE(in_mad_size != sizeof(*in_mad) ||
263264 *out_mad_size != sizeof(*out_mad)))
@@ -265,14 +266,20 @@ int mlx5_ib_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num, · 1 unmodified line
265266
266267 memset(out_mad->data, 0, sizeof(out_mad->data));
267268
269+ mdev = mlx5_ib_get_native_port_mdev(dev, port_num, &mdev_port_num);
270+ if (!mdev)
271+ return IB_MAD_RESULT_FAILURE;
272+
268273 if (MLX5_CAP_GEN(mdev, vport_counters) &&
269274 in_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_PERF_MGMT &&
270275 in_mad->mad_hdr.method == IB_MGMT_METHOD_GET) {
271− return process_pma_cmd(ibdev, port_num, in_mad, out_mad);
276+ ret = process_pma_cmd(mdev, mdev_port_num, in_mad, out_mad);
272277 } else {
273− return process_mad(ibdev, mad_flags, port_num, in_wc, in_grh,
278+ ret = process_mad(ibdev, mad_flags, port_num, in_wc, in_grh,
274279 in_mad, out_mad);
275280 }
281+ mlx5_ib_put_native_port_mdev(dev, port_num);
282+ return ret;
276283 }
277284
278285 int mlx5_query_ext_port_caps(struct mlx5_ib_dev *dev, u8 port)
@@ -519,7 +526,7 @@ int mlx5_query_mad_ifc_port(struct ib_device *ibdev, u8 port, · 240 unmodified lines
519526 int ext_active_speed;
520527 int err = -ENOMEM;
521528
522− if (port < 1 || port > MLX5_CAP_GEN(mdev, num_ports)) {
529+ if (port < 1 || port > dev->num_ports) {
523530 mlx5_ib_warn(dev, "invalid port number %d\n", port);
524531 return -EINVAL;
525532 }
drivers/infiniband/hw/mlx5/main.c+1115−296
@@ -50,16 +50,14 @@ · 49 unmodified lines
5050 #include <rdma/ib_cache.h>
5151 #include <linux/mlx5/port.h>
5252 #include <linux/mlx5/vport.h>
53+#include <linux/mlx5/fs.h>
5354 #include <linux/list.h>
5455 #include <rdma/ib_smi.h>
5556 #include <rdma/ib_umem.h>
5657 #include <linux/in.h>
5758 #include <linux/etherdevice.h>
58−#include <linux/mlx5/fs.h>
59−#include <linux/mlx5/vport.h>
6059 #include "mlx5_ib.h"
6160 #include "cmd.h"
62−#include <linux/mlx5/vport.h>
6361
6462 #define DRIVER_NAME "mlx5_ib"
6563 #define DRIVER_VERSION "5.0-0"
@@ -72,10 +70,36 @@ static char mlx5_version[] = · 6 unmodified lines
7270 DRIVER_NAME ": Mellanox Connect-IB Infiniband driver v"
7371 DRIVER_VERSION "\n";
7472
73+struct mlx5_ib_event_work {
74+ struct work_struct work;
75+ struct mlx5_core_dev *dev;
76+ void *context;
77+ enum mlx5_dev_event event;
78+ unsigned long param;
79+};
80+
7581 enum {
7682 MLX5_ATOMIC_SIZE_QP_8BYTES = 1 << 3,
7783 };
7884
85+static struct workqueue_struct *mlx5_ib_event_wq;
86+static LIST_HEAD(mlx5_ib_unaffiliated_port_list);
87+static LIST_HEAD(mlx5_ib_dev_list);
88+/*
89+ * This mutex should be held when accessing either of the above lists
90+ */
91+static DEFINE_MUTEX(mlx5_ib_multiport_mutex);
92+
93+struct mlx5_ib_dev *mlx5_ib_get_ibdev_from_mpi(struct mlx5_ib_multiport_info *mpi)
94+{
95+ struct mlx5_ib_dev *dev;
96+
97+ mutex_lock(&mlx5_ib_multiport_mutex);
98+ dev = mpi->ibdev;
99+ mutex_unlock(&mlx5_ib_multiport_mutex);
100+ return dev;
101+}
102+
79103 static enum rdma_link_layer
80104 mlx5_port_type_cap_to_rdma_ll(int port_type_cap)
81105 {
@@ -115,24 +139,32 @@ static int get_port_state(struct ib_device *ibdev, · 33 unmodified lines
115139 static int mlx5_netdev_event(struct notifier_block *this,
116140 unsigned long event, void *ptr)
117141 {
142+ struct mlx5_roce *roce = container_of(this, struct mlx5_roce, nb);
118143 struct net_device *ndev = netdev_notifier_info_to_dev(ptr);
119− struct mlx5_ib_dev *ibdev = container_of(this, struct mlx5_ib_dev,
120− roce.nb);
144+ u8 port_num = roce->native_port_num;
145+ struct mlx5_core_dev *mdev;
146+ struct mlx5_ib_dev *ibdev;
147+
148+ ibdev = roce->dev;
149+ mdev = mlx5_ib_get_native_port_mdev(ibdev, port_num, NULL);
150+ if (!mdev)
151+ return NOTIFY_DONE;
121152
122153 switch (event) {
123154 case NETDEV_REGISTER:
124155 case NETDEV_UNREGISTER:
125− write_lock(&ibdev->roce.netdev_lock);
126− if (ndev->dev.parent == &ibdev->mdev->pdev->dev)
127− ibdev->roce.netdev = (event == NETDEV_UNREGISTER) ?
128− NULL : ndev;
129− write_unlock(&ibdev->roce.netdev_lock);
156+ write_lock(&roce->netdev_lock);
157+
158+ if (ndev->dev.parent == &mdev->pdev->dev)
159+ roce->netdev = (event == NETDEV_UNREGISTER) ?
160+ NULL : ndev;
161+ write_unlock(&roce->netdev_lock);
130162 break;
131163
132164 case NETDEV_CHANGE:
133165 case NETDEV_UP:
134166 case NETDEV_DOWN: {
135− struct net_device *lag_ndev = mlx5_lag_get_roce_netdev(ibdev->mdev);
167+ struct net_device *lag_ndev = mlx5_lag_get_roce_netdev(mdev);
136168 struct net_device *upper = NULL;
137169
138170 if (lag_ndev) {
@@ -140,27 +172,28 @@ static int mlx5_netdev_event(struct notifier_block *this, · 1 unmodified line
140172 dev_put(lag_ndev);
141173 }
142174
143− if ((upper == ndev || (!upper && ndev == ibdev->roce.netdev))
175+ if ((upper == ndev || (!upper && ndev == roce->netdev))
144176 && ibdev->ib_active) {
145177 struct ib_event ibev = { };
146178 enum ib_port_state port_state;
147179
148− if (get_port_state(&ibdev->ib_dev, 1, &port_state))
149− return NOTIFY_DONE;
180+ if (get_port_state(&ibdev->ib_dev, port_num,
181+ &port_state))
182+ goto done;
150183
151− if (ibdev->roce.last_port_state == port_state)
152− return NOTIFY_DONE;
184+ if (roce->last_port_state == port_state)
185+ goto done;
153186
154− ibdev->roce.last_port_state = port_state;
187+ roce->last_port_state = port_state;
155188 ibev.device = &ibdev->ib_dev;
156189 if (port_state == IB_PORT_DOWN)
157190 ibev.event = IB_EVENT_PORT_ERR;
158191 else if (port_state == IB_PORT_ACTIVE)
159192 ibev.event = IB_EVENT_PORT_ACTIVE;
160193 else
161− return NOTIFY_DONE;
194+ goto done;
162195
163− ibev.element.port_num = 1;
196+ ibev.element.port_num = port_num;
164197 ib_dispatch_event(&ibev);
165198 }
166199 break;
@@ -169,7 +202,8 @@ static int mlx5_netdev_event(struct notifier_block *this, · 2 unmodified lines
169202 default:
170203 break;
171204 }
172−
205+done:
206+ mlx5_ib_put_native_port_mdev(ibdev, port_num);
173207 return NOTIFY_DONE;
174208 }
175209
@@ -178,22 +212,88 @@ static struct net_device *mlx5_ib_get_netdev(struct ib_device *device, · 2 unmodified lines
178212 {
179213 struct mlx5_ib_dev *ibdev = to_mdev(device);
180214 struct net_device *ndev;
215+ struct mlx5_core_dev *mdev;
216+
217+ mdev = mlx5_ib_get_native_port_mdev(ibdev, port_num, NULL);
218+ if (!mdev)
219+ return NULL;
181220
182− ndev = mlx5_lag_get_roce_netdev(ibdev->mdev);
221+ ndev = mlx5_lag_get_roce_netdev(mdev);
183222 if (ndev)
184− return ndev;
223+ goto out;
185224
186225 /* Ensure ndev does not disappear before we invoke dev_hold()
187226 */
188− read_lock(&ibdev->roce.netdev_lock);
189− ndev = ibdev->roce.netdev;
227+ read_lock(&ibdev->roce[port_num - 1].netdev_lock);
228+ ndev = ibdev->roce[port_num - 1].netdev;
190229 if (ndev)
191230 dev_hold(ndev);
192− read_unlock(&ibdev->roce.netdev_lock);
231+ read_unlock(&ibdev->roce[port_num - 1].netdev_lock);
193232
233+out:
234+ mlx5_ib_put_native_port_mdev(ibdev, port_num);
194235 return ndev;
195236 }
196237
238+struct mlx5_core_dev *mlx5_ib_get_native_port_mdev(struct mlx5_ib_dev *ibdev,
239+ u8 ib_port_num,
240+ u8 *native_port_num)
241+{
242+ enum rdma_link_layer ll = mlx5_ib_port_link_layer(&ibdev->ib_dev,
243+ ib_port_num);
244+ struct mlx5_core_dev *mdev = NULL;
245+ struct mlx5_ib_multiport_info *mpi;
246+ struct mlx5_ib_port *port;
247+
248+ if (native_port_num)
249+ *native_port_num = 1;
250+
251+ if (!mlx5_core_mp_enabled(ibdev->mdev) || ll != IB_LINK_LAYER_ETHERNET)
252+ return ibdev->mdev;
253+
254+ port = &ibdev->port[ib_port_num - 1];
255+ if (!port)
256+ return NULL;
257+
258+ spin_lock(&port->mp.mpi_lock);
259+ mpi = ibdev->port[ib_port_num - 1].mp.mpi;
260+ if (mpi && !mpi->unaffiliate) {
261+ mdev = mpi->mdev;
262+ /* If it's the master no need to refcount, it'll exist
263+ * as long as the ib_dev exists.
264+ */
265+ if (!mpi->is_master)
266+ mpi->mdev_refcnt++;
267+ }
268+ spin_unlock(&port->mp.mpi_lock);
269+
270+ return mdev;
271+}
272+
273+void mlx5_ib_put_native_port_mdev(struct mlx5_ib_dev *ibdev, u8 port_num)
274+{
275+ enum rdma_link_layer ll = mlx5_ib_port_link_layer(&ibdev->ib_dev,
276+ port_num);
277+ struct mlx5_ib_multiport_info *mpi;
278+ struct mlx5_ib_port *port;
279+
280+ if (!mlx5_core_mp_enabled(ibdev->mdev) || ll != IB_LINK_LAYER_ETHERNET)
281+ return;
282+
283+ port = &ibdev->port[port_num - 1];
284+
285+ spin_lock(&port->mp.mpi_lock);
286+ mpi = ibdev->port[port_num - 1].mp.mpi;
287+ if (mpi->is_master)
288+ goto out;
289+
290+ mpi->mdev_refcnt--;
291+ if (mpi->unaffiliate)
292+ complete(&mpi->unref_comp);
293+out:
294+ spin_unlock(&port->mp.mpi_lock);
295+}
296+
197297 static int translate_eth_proto_oper(u32 eth_proto_oper, u8 *active_speed,
198298 u8 *active_width)
199299 {
@@ -256,19 +356,33 @@ static int mlx5_query_port_roce(struct ib_device *device, u8 port_num, · 56 unmodified lines
256356 struct ib_port_attr *props)
257357 {
258358 struct mlx5_ib_dev *dev = to_mdev(device);
259− struct mlx5_core_dev *mdev = dev->mdev;
359+ struct mlx5_core_dev *mdev;
260360 struct net_device *ndev, *upper;
261361 enum ib_mtu ndev_ib_mtu;
362+ bool put_mdev = true;
262363 u16 qkey_viol_cntr;
263364 u32 eth_prot_oper;
365+ u8 mdev_port_num;
264366 int err;
265367
368+ mdev = mlx5_ib_get_native_port_mdev(dev, port_num, &mdev_port_num);
369+ if (!mdev) {
370+ /* This means the port isn't affiliated yet. Get the
371+ * info for the master port instead.
372+ */
373+ put_mdev = false;
374+ mdev = dev->mdev;
375+ mdev_port_num = 1;
376+ port_num = 1;
377+ }
378+
266379 /* Possible bad flows are checked before filling out props so in case
267380 * of an error it will still be zeroed out.
268381 */
269− err = mlx5_query_port_eth_proto_oper(mdev, &eth_prot_oper, port_num);
382+ err = mlx5_query_port_eth_proto_oper(mdev, &eth_prot_oper,
383+ mdev_port_num);
270384 if (err)
271− return err;
385+ goto out;
272386
273387 translate_eth_proto_oper(eth_prot_oper, &props->active_speed,
274388 &props->active_width);
@@ -284,12 +398,16 @@ static int mlx5_query_port_roce(struct ib_device *device, u8 port_num, · 9 unmodified lines
284398 props->state = IB_PORT_DOWN;
285399 props->phys_state = 3;
286400
287− mlx5_query_nic_vport_qkey_viol_cntr(dev->mdev, &qkey_viol_cntr);
401+ mlx5_query_nic_vport_qkey_viol_cntr(mdev, &qkey_viol_cntr);
288402 props->qkey_viol_cntr = qkey_viol_cntr;
289403
404+ /* If this is a stub query for an unaffiliated port stop here */
405+ if (!put_mdev)
406+ goto out;
407+
290408 ndev = mlx5_ib_get_netdev(device, port_num);
291409 if (!ndev)
292− return 0;
410+ goto out;
293411
294412 if (mlx5_lag_is_active(dev->mdev)) {
295413 rcu_read_lock();
@@ -312,7 +430,10 @@ static int mlx5_query_port_roce(struct ib_device *device, u8 port_num, · 16 unmodified lines
312430 dev_put(ndev);
313431
314432 props->active_mtu = min(props->max_mtu, ndev_ib_mtu);
315− return 0;
433+out:
434+ if (put_mdev)
435+ mlx5_ib_put_native_port_mdev(dev, port_num);
436+ return err;
316437 }
317438
318439 static int set_roce_addr(struct mlx5_ib_dev *dev, u8 port_num,
@@ -354,7 +475,7 @@ static int set_roce_addr(struct mlx5_ib_dev *dev, u8 port_num, · 35 unmodified lines
354475
355476 return mlx5_core_roce_gid_set(dev->mdev, index, roce_version,
356477 roce_l3_type, gid->raw, mac, vlan,
357− vlan_id);
478+ vlan_id, port_num);
358479 }
359480
360481 static int mlx5_ib_add_gid(struct ib_device *device, u8 port_num,
@@ -438,11 +559,11 @@ static int mlx5_get_vport_access_method(struct ib_device *ibdev) · 77 unmodified lines
438559 }
439560
440561 static void get_atomic_caps(struct mlx5_ib_dev *dev,
562+ u8 atomic_size_qp,
441563 struct ib_device_attr *props)
442564 {
443565 u8 tmp;
444566 u8 atomic_operations = MLX5_CAP_ATOMIC(dev->mdev, atomic_operations);
445− u8 atomic_size_qp = MLX5_CAP_ATOMIC(dev->mdev, atomic_size_qp);
446567 u8 atomic_req_8B_endianness_mode =
447568 MLX5_CAP_ATOMIC(dev->mdev, atomic_req_8B_endianness_mode);
448569
@@ -459,6 +580,29 @@ static void get_atomic_caps(struct mlx5_ib_dev *dev, · 10 unmodified lines
459580 }
460581 }
461582
583+static void get_atomic_caps_qp(struct mlx5_ib_dev *dev,
584+ struct ib_device_attr *props)
585+{
586+ u8 atomic_size_qp = MLX5_CAP_ATOMIC(dev->mdev, atomic_size_qp);
587+
588+ get_atomic_caps(dev, atomic_size_qp, props);
589+}
590+
591+static void get_atomic_caps_dc(struct mlx5_ib_dev *dev,
592+ struct ib_device_attr *props)
593+{
594+ u8 atomic_size_qp = MLX5_CAP_ATOMIC(dev->mdev, atomic_size_dc);
595+
596+ get_atomic_caps(dev, atomic_size_qp, props);
597+}
598+
599+bool mlx5_ib_dc_atomic_is_supported(struct mlx5_ib_dev *dev)
600+{
601+ struct ib_device_attr props = {};
602+
603+ get_atomic_caps_dc(dev, &props);
604+ return (props.atomic_cap == IB_ATOMIC_HCA) ? true : false;
605+}
462606 static int mlx5_query_system_image_guid(struct ib_device *ibdev,
463607 __be64 *sys_image_guid)
464608 {
@@ -587,6 +731,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 122 unmodified lines
587731 int max_rq_sg;
588732 int max_sq_sg;
589733 u64 min_page_size = 1ull << MLX5_CAP_GEN(mdev, log_pg_sz);
734+ bool raw_support = !mlx5_core_mp_enabled(mdev);
590735 struct mlx5_ib_query_device_resp resp = {};
591736 size_t resp_len;
592737 u64 max_tso;
@@ -650,7 +795,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 57 unmodified lines
650795 if (MLX5_CAP_GEN(mdev, block_lb_mc))
651796 props->device_cap_flags |= IB_DEVICE_BLOCK_MULTICAST_LOOPBACK;
652797
653− if (MLX5_CAP_GEN(dev->mdev, eth_net_offloads)) {
798+ if (MLX5_CAP_GEN(dev->mdev, eth_net_offloads) && raw_support) {
654799 if (MLX5_CAP_ETH(mdev, csum_cap)) {
655800 /* Legacy bit to support old userspace libraries */
656801 props->device_cap_flags |= IB_DEVICE_RAW_IP_CSUM;
@@ -682,7 +827,8 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 25 unmodified lines
682827 MLX5_RX_HASH_SRC_PORT_TCP |
683828 MLX5_RX_HASH_DST_PORT_TCP |
684829 MLX5_RX_HASH_SRC_PORT_UDP |
685− MLX5_RX_HASH_DST_PORT_UDP;
830+ MLX5_RX_HASH_DST_PORT_UDP |
831+ MLX5_RX_HASH_INNER;
686832 resp.response_length += sizeof(resp.rss_caps);
687833 }
688834 } else {
@@ -698,7 +844,8 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 9 unmodified lines
698844 }
699845
700846 if (MLX5_CAP_GEN(dev->mdev, rq_delay_drop) &&
701− MLX5_CAP_GEN(dev->mdev, general_notification_event))
847+ MLX5_CAP_GEN(dev->mdev, general_notification_event) &&
848+ raw_support)
702849 props->raw_packet_caps |= IB_RAW_PACKET_CAP_DELAY_DROP;
703850
704851 if (MLX5_CAP_GEN(mdev, ipoib_enhanced_offloads) &&
@@ -706,7 +853,8 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 1 unmodified line
706853 props->device_cap_flags |= IB_DEVICE_UD_IP_CSUM;
707854
708855 if (MLX5_CAP_GEN(dev->mdev, eth_net_offloads) &&
709− MLX5_CAP_ETH(dev->mdev, scatter_fcs)) {
856+ MLX5_CAP_ETH(dev->mdev, scatter_fcs) &&
857+ raw_support) {
710858 /* Legacy bit to support old userspace libraries */
711859 props->device_cap_flags |= IB_DEVICE_RAW_SCATTER_FCS;
712860 props->raw_packet_caps |= IB_RAW_PACKET_CAP_SCATTER_FCS;
@@ -746,7 +894,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 33 unmodified lines
746894 props->max_srq_sge = max_rq_sg - 1;
747895 props->max_fast_reg_page_list_len =
748896 1 << MLX5_CAP_GEN(mdev, log_max_klm_list_size);
749− get_atomic_caps(dev, props);
897+ get_atomic_caps_qp(dev, props);
750898 props->masked_atomic_cap = IB_ATOMIC_NONE;
751899 props->max_mcast_grp = 1 << MLX5_CAP_GEN(mdev, log_max_mcg);
752900 props->max_mcast_qp_attach = MLX5_CAP_GEN(mdev, max_qp_mcg);
@@ -770,7 +918,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 17 unmodified lines
770918 props->device_cap_flags |= IB_DEVICE_VIRTUAL_FUNCTION;
771919
772920 if (mlx5_ib_port_link_layer(ibdev, 1) ==
773− IB_LINK_LAYER_ETHERNET) {
921+ IB_LINK_LAYER_ETHERNET && raw_support) {
774922 props->rss_caps.max_rwq_indirection_tables =
775923 1 << MLX5_CAP_GEN(dev->mdev, log_max_rqt);
776924 props->rss_caps.max_rwq_indirection_table_size =
@@ -807,7 +955,8 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 30 unmodified lines
807955 resp.response_length += sizeof(resp.cqe_comp_caps);
808956 }
809957
810− if (field_avail(typeof(resp), packet_pacing_caps, uhw->outlen)) {
958+ if (field_avail(typeof(resp), packet_pacing_caps, uhw->outlen) &&
959+ raw_support) {
811960 if (MLX5_CAP_QOS(mdev, packet_pacing) &&
812961 MLX5_CAP_GEN(mdev, qos)) {
813962 resp.packet_pacing_caps.qp_rate_limit_max =
@@ -866,7 +1015,8 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 52 unmodified lines
8661015 }
8671016 }
8681017
869− if (field_avail(typeof(resp), striding_rq_caps, uhw->outlen)) {
1018+ if (field_avail(typeof(resp), striding_rq_caps, uhw->outlen) &&
1019+ raw_support) {
8701020 resp.response_length += sizeof(resp.striding_rq_caps);
8711021 if (MLX5_CAP_GEN(mdev, striding_rq)) {
8721022 resp.striding_rq_caps.min_single_stride_log_num_of_bytes =
@@ -1097,7 +1247,22 @@ int mlx5_ib_query_port(struct ib_device *ibdev, u8 port, · 224 unmodified lines
10971247 }
10981248
10991249 if (!ret && props) {
1100− count = mlx5_core_reserved_gids_count(to_mdev(ibdev)->mdev);
1250+ struct mlx5_ib_dev *dev = to_mdev(ibdev);
1251+ struct mlx5_core_dev *mdev;
1252+ bool put_mdev = true;
1253+
1254+ mdev = mlx5_ib_get_native_port_mdev(dev, port, NULL);
1255+ if (!mdev) {
1256+ /* If the port isn't affiliated yet query the master.
1257+ * The master and slave will have the same values.
1258+ */
1259+ mdev = dev->mdev;
1260+ port = 1;
1261+ put_mdev = false;
1262+ }
1263+ count = mlx5_core_reserved_gids_count(mdev);
1264+ if (put_mdev)
1265+ mlx5_ib_put_native_port_mdev(dev, port);
11011266 props->gid_tbl_len -= count;
11021267 }
11031268 return ret;
@@ -1122,20 +1287,43 @@ static int mlx5_ib_query_gid(struct ib_device *ibdev, u8 port, int index, · 18 unmodified lines
11221287
11231288 }
11241289
1125−static int mlx5_ib_query_pkey(struct ib_device *ibdev, u8 port, u16 index,
1126− u16 *pkey)
1290+static int mlx5_query_hca_nic_pkey(struct ib_device *ibdev, u8 port,
1291+ u16 index, u16 *pkey)
11271292 {
11281293 struct mlx5_ib_dev *dev = to_mdev(ibdev);
1129− struct mlx5_core_dev *mdev = dev->mdev;
1294+ struct mlx5_core_dev *mdev;
1295+ bool put_mdev = true;
1296+ u8 mdev_port_num;
1297+ int err;
1298+
1299+ mdev = mlx5_ib_get_native_port_mdev(dev, port, &mdev_port_num);
1300+ if (!mdev) {
1301+ /* The port isn't affiliated yet, get the PKey from the master
1302+ * port. For RoCE the PKey tables will be the same.
1303+ */
1304+ put_mdev = false;
1305+ mdev = dev->mdev;
1306+ mdev_port_num = 1;
1307+ }
1308+
1309+ err = mlx5_query_hca_vport_pkey(mdev, 0, mdev_port_num, 0,
1310+ index, pkey);
1311+ if (put_mdev)
1312+ mlx5_ib_put_native_port_mdev(dev, port);
1313+
1314+ return err;
1315+}
11301316
1317+static int mlx5_ib_query_pkey(struct ib_device *ibdev, u8 port, u16 index,
1318+ u16 *pkey)
1319+{
11311320 switch (mlx5_get_vport_access_method(ibdev)) {
11321321 case MLX5_VPORT_ACCESS_METHOD_MAD:
11331322 return mlx5_query_mad_ifc_pkey(ibdev, port, index, pkey);
11341323
11351324 case MLX5_VPORT_ACCESS_METHOD_HCA:
11361325 case MLX5_VPORT_ACCESS_METHOD_NIC:
1137− return mlx5_query_hca_vport_pkey(mdev, 0, port, 0, index,
1138− pkey);
1326+ return mlx5_query_hca_nic_pkey(ibdev, port, index, pkey);
11391327 default:
11401328 return -EINVAL;
11411329 }
@@ -1174,23 +1362,32 @@ static int set_port_caps_atomic(struct mlx5_ib_dev *dev, u8 port_num, u32 mask, · 32 unmodified lines
11741362 u32 value)
11751363 {
11761364 struct mlx5_hca_vport_context ctx = {};
1365+ struct mlx5_core_dev *mdev;
1366+ u8 mdev_port_num;
11771367 int err;
11781368
1179− err = mlx5_query_hca_vport_context(dev->mdev, 0,
1180− port_num, 0, &ctx);
1369+ mdev = mlx5_ib_get_native_port_mdev(dev, port_num, &mdev_port_num);
1370+ if (!mdev)
1371+ return -ENODEV;
1372+
1373+ err = mlx5_query_hca_vport_context(mdev, 0, mdev_port_num, 0, &ctx);
11811374 if (err)
1182− return err;
1375+ goto out;
11831376
11841377 if (~ctx.cap_mask1_perm & mask) {
11851378 mlx5_ib_warn(dev, "trying to change bitmask 0x%X but change supported 0x%X\n",
11861379 mask, ctx.cap_mask1_perm);
1187− return -EINVAL;
1380+ err = -EINVAL;
1381+ goto out;
11881382 }
11891383
11901384 ctx.cap_mask1 = value;
11911385 ctx.cap_mask1_perm = mask;
1192− err = mlx5_core_modify_hca_vport_context(dev->mdev, 0,
1193− port_num, 0, &ctx);
1386+ err = mlx5_core_modify_hca_vport_context(mdev, 0, mdev_port_num,
1387+ 0, &ctx);
1388+
1389+out:
1390+ mlx5_ib_put_native_port_mdev(dev, port_num);
11941391
11951392 return err;
11961393 }
@@ -1241,9 +1438,18 @@ static void print_lib_caps(struct mlx5_ib_dev *dev, u64 caps) · 44 unmodified lines
12411438 caps & MLX5_LIB_CAP_4K_UAR ? "y" : "n");
12421439 }
12431440
1441+static u16 calc_dynamic_bfregs(int uars_per_sys_page)
1442+{
1443+ /* Large page with non 4k uar support might limit the dynamic size */
1444+ if (uars_per_sys_page == 1 && PAGE_SIZE > 4096)
1445+ return MLX5_MIN_DYN_BFREGS;
1446+
1447+ return MLX5_MAX_DYN_BFREGS;
1448+}
1449+
12441450 static int calc_total_bfregs(struct mlx5_ib_dev *dev, bool lib_uar_4k,
12451451 struct mlx5_ib_alloc_ucontext_req_v2 *req,
1246− u32 *num_sys_pages)
1452+ struct mlx5_bfreg_info *bfregi)
12471453 {
12481454 int uars_per_sys_page;
12491455 int bfregs_per_sys_page;
@@ -1260,16 +1466,21 @@ static int calc_total_bfregs(struct mlx5_ib_dev *dev, bool lib_uar_4k, · 10 unmodified lines
12601466
12611467 uars_per_sys_page = get_uars_per_sys_page(dev, lib_uar_4k);
12621468 bfregs_per_sys_page = uars_per_sys_page * MLX5_NON_FP_BFREGS_PER_UAR;
1469+ /* This holds the required static allocation asked by the user */
12631470 req->total_num_bfregs = ALIGN(req->total_num_bfregs, bfregs_per_sys_page);
1264− *num_sys_pages = req->total_num_bfregs / bfregs_per_sys_page;
1265−
12661471 if (req->num_low_latency_bfregs > req->total_num_bfregs - 1)
12671472 return -EINVAL;
12681473
1269− mlx5_ib_dbg(dev, "uar_4k: fw support %s, lib support %s, user requested %d bfregs, allocated %d, using %d sys pages\n",
1474+ bfregi->num_static_sys_pages = req->total_num_bfregs / bfregs_per_sys_page;
1475+ bfregi->num_dyn_bfregs = ALIGN(calc_dynamic_bfregs(uars_per_sys_page), bfregs_per_sys_page);
1476+ bfregi->total_num_bfregs = req->total_num_bfregs + bfregi->num_dyn_bfregs;
1477+ bfregi->num_sys_pages = bfregi->total_num_bfregs / bfregs_per_sys_page;
1478+
1479+ mlx5_ib_dbg(dev, "uar_4k: fw support %s, lib support %s, user requested %d bfregs, allocated %d, total bfregs %d, using %d sys pages\n",
12701480 MLX5_CAP_GEN(dev->mdev, uar_4k) ? "yes" : "no",
12711481 lib_uar_4k ? "yes" : "no", ref_bfregs,
1272− req->total_num_bfregs, *num_sys_pages);
1482+ req->total_num_bfregs, bfregi->total_num_bfregs,
1483+ bfregi->num_sys_pages);
12731484
12741485 return 0;
12751486 }
@@ -1281,13 +1492,17 @@ static int allocate_uars(struct mlx5_ib_dev *dev, struct mlx5_ib_ucontext *conte · 5 unmodified lines
12811492 int i;
12821493
12831494 bfregi = &context->bfregi;
1284− for (i = 0; i < bfregi->num_sys_pages; i++) {
1495+ for (i = 0; i < bfregi->num_static_sys_pages; i++) {
12851496 err = mlx5_cmd_alloc_uar(dev->mdev, &bfregi->sys_pages[i]);
12861497 if (err)
12871498 goto error;
12881499
12891500 mlx5_ib_dbg(dev, "allocated uar %d\n", bfregi->sys_pages[i]);
12901501 }
1502+
1503+ for (i = bfregi->num_static_sys_pages; i < bfregi->num_sys_pages; i++)
1504+ bfregi->sys_pages[i] = MLX5_IB_INVALID_UAR_INDEX;
1505+
12911506 return 0;
12921507
12931508 error:
@@ -1306,12 +1521,16 @@ static int deallocate_uars(struct mlx5_ib_dev *dev, struct mlx5_ib_ucontext *con · 12 unmodified lines
13061521
13071522 bfregi = &context->bfregi;
13081523 for (i = 0; i < bfregi->num_sys_pages; i++) {
1309− err = mlx5_cmd_free_uar(dev->mdev, bfregi->sys_pages[i]);
1310− if (err) {
1311− mlx5_ib_warn(dev, "failed to free uar %d\n", i);
1312− return err;
1524+ if (i < bfregi->num_static_sys_pages ||
1525+ bfregi->sys_pages[i] != MLX5_IB_INVALID_UAR_INDEX) {
1526+ err = mlx5_cmd_free_uar(dev->mdev, bfregi->sys_pages[i]);
1527+ if (err) {
1528+ mlx5_ib_warn(dev, "failed to free uar %d, err=%d\n", i, err);
1529+ return err;
1530+ }
13131531 }
13141532 }
1533+
13151534 return 0;
13161535 }
13171536
@@ -1362,6 +1581,7 @@ static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev, · 44 unmodified lines
13621581 struct mlx5_ib_dev *dev = to_mdev(ibdev);
13631582 struct mlx5_ib_alloc_ucontext_req_v2 req = {};
13641583 struct mlx5_ib_alloc_ucontext_resp resp = {};
1584+ struct mlx5_core_dev *mdev = dev->mdev;
13651585 struct mlx5_ib_ucontext *context;
13661586 struct mlx5_bfreg_info *bfregi;
13671587 int ver;
@@ -1422,13 +1642,13 @@ static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev, · 54 unmodified lines
14221642 bfregi = &context->bfregi;
14231643
14241644 /* updates req->total_num_bfregs */
1425− err = calc_total_bfregs(dev, lib_uar_4k, &req, &bfregi->num_sys_pages);
1645+ err = calc_total_bfregs(dev, lib_uar_4k, &req, bfregi);
14261646 if (err)
14271647 goto out_ctx;
14281648
14291649 mutex_init(&bfregi->lock);
14301650 bfregi->lib_uar_4k = lib_uar_4k;
1431− bfregi->count = kcalloc(req.total_num_bfregs, sizeof(*bfregi->count),
1651+ bfregi->count = kcalloc(bfregi->total_num_bfregs, sizeof(*bfregi->count),
14321652 GFP_KERNEL);
14331653 if (!bfregi->count) {
14341654 err = -ENOMEM;
@@ -1470,7 +1690,7 @@ static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev, · 35 unmodified lines
14701690 mutex_init(&context->db_page_mutex);
14711691
14721692 resp.tot_bfregs = req.total_num_bfregs;
1473− resp.num_ports = MLX5_CAP_GEN(dev->mdev, num_ports);
1693+ resp.num_ports = dev->num_ports;
14741694
14751695 if (field_avail(typeof(resp), cqe_version, udata->outlen))
14761696 resp.response_length += sizeof(resp.cqe_version);
@@ -1489,6 +1709,12 @@ static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev, · 12 unmodified lines
14891709 resp.response_length += sizeof(resp.eth_min_inline);
14901710 }
14911711
1712+ if (field_avail(typeof(resp), clock_info_versions, udata->outlen)) {
1713+ if (mdev->clock_info)
1714+ resp.clock_info_versions = BIT(MLX5_IB_CLOCK_INFO_V1);
1715+ resp.response_length += sizeof(resp.clock_info_versions);
1716+ }
1717+
14921718 /*
14931719 * We don't want to expose information from the PCI bar that is located
14941720 * after 4096 bytes, so if the arch only supports larger pages, let's
@@ -1502,8 +1728,7 @@ static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev, · 7 unmodified lines
15021728 resp.hca_core_clock_offset =
15031729 offsetof(struct mlx5_init_seg, internal_timer_h) % PAGE_SIZE;
15041730 }
1505− resp.response_length += sizeof(resp.hca_core_clock_offset) +
1506− sizeof(resp.reserved2);
1731+ resp.response_length += sizeof(resp.hca_core_clock_offset);
15071732 }
15081733
15091734 if (field_avail(typeof(resp), log_uar_size, udata->outlen))
@@ -1512,6 +1737,11 @@ static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev, · 2 unmodified lines
15121737 if (field_avail(typeof(resp), num_uars_per_page, udata->outlen))
15131738 resp.response_length += sizeof(resp.num_uars_per_page);
15141739
1740+ if (field_avail(typeof(resp), num_dyn_bfregs, udata->outlen)) {
1741+ resp.num_dyn_bfregs = bfregi->num_dyn_bfregs;
1742+ resp.response_length += sizeof(resp.num_dyn_bfregs);
1743+ }
1744+
15151745 err = ib_copy_to_udata(udata, &resp, resp.response_length);
15161746 if (err)
15171747 goto out_td;
@@ -1566,15 +1796,13 @@ static int mlx5_ib_dealloc_ucontext(struct ib_ucontext *ibcontext) · 48 unmodified lines
15661796 }
15671797
15681798 static phys_addr_t uar_index2pfn(struct mlx5_ib_dev *dev,
1569− struct mlx5_bfreg_info *bfregi,
1570− int idx)
1799+ int uar_idx)
15711800 {
15721801 int fw_uars_per_page;
15731802
15741803 fw_uars_per_page = MLX5_CAP_GEN(dev->mdev, uar_4k) ? MLX5_UARS_IN_PAGE : 1;
15751804
1576− return (pci_resource_start(dev->mdev->pdev, 0) >> PAGE_SHIFT) +
1577− bfregi->sys_pages[idx] / fw_uars_per_page;
1805+ return (pci_resource_start(dev->mdev->pdev, 0) >> PAGE_SHIFT) + uar_idx / fw_uars_per_page;
15781806 }
15791807
15801808 static int get_command(unsigned long offset)
@@ -1592,6 +1820,12 @@ static int get_index(unsigned long offset) · 11 unmodified lines
15921820 return get_arg(offset);
15931821 }
15941822
1823+/* Index resides in an extra byte to enable larger values than 255 */
1824+static int get_extended_index(unsigned long offset)
1825+{
1826+ return get_arg(offset) | ((offset >> 16) & 0xff) << 8;
1827+}
1828+
15951829 static void mlx5_ib_vma_open(struct vm_area_struct *area)
15961830 {
15971831 /* vma_open is called when a new VMA is created on top of our VMA. This
@@ -1733,6 +1967,38 @@ static inline char *mmap_cmd2str(enum mlx5_ib_mmap_cmd cmd) · 135 unmodified lines
17331967 }
17341968 }
17351969
1970+static int mlx5_ib_mmap_clock_info_page(struct mlx5_ib_dev *dev,
1971+ struct vm_area_struct *vma,
1972+ struct mlx5_ib_ucontext *context)
1973+{
1974+ phys_addr_t pfn;
1975+ int err;
1976+
1977+ if (vma->vm_end - vma->vm_start != PAGE_SIZE)
1978+ return -EINVAL;
1979+
1980+ if (get_index(vma->vm_pgoff) != MLX5_IB_CLOCK_INFO_V1)
1981+ return -EOPNOTSUPP;
1982+
1983+ if (vma->vm_flags & VM_WRITE)
1984+ return -EPERM;
1985+
1986+ if (!dev->mdev->clock_info_page)
1987+ return -EOPNOTSUPP;
1988+
1989+ pfn = page_to_pfn(dev->mdev->clock_info_page);
1990+ err = remap_pfn_range(vma, vma->vm_start, pfn, PAGE_SIZE,
1991+ vma->vm_page_prot);
1992+ if (err)
1993+ return err;
1994+
1995+ mlx5_ib_dbg(dev, "mapped clock info at 0x%lx, PA 0x%llx\n",
1996+ vma->vm_start,
1997+ (unsigned long long)pfn << PAGE_SHIFT);
1998+
1999+ return mlx5_ib_set_vma_data(vma, context);
2000+}
2001+
17362002 static int uar_mmap(struct mlx5_ib_dev *dev, enum mlx5_ib_mmap_cmd cmd,
17372003 struct vm_area_struct *vma,
17382004 struct mlx5_ib_ucontext *context)
@@ -1742,21 +2008,29 @@ static int uar_mmap(struct mlx5_ib_dev *dev, enum mlx5_ib_mmap_cmd cmd, · 3 unmodified lines
17422008 unsigned long idx;
17432009 phys_addr_t pfn, pa;
17442010 pgprot_t prot;
1745− int uars_per_page;
2011+ u32 bfreg_dyn_idx = 0;
2012+ u32 uar_index;
2013+ int dyn_uar = (cmd == MLX5_IB_MMAP_ALLOC_WC);
2014+ int max_valid_idx = dyn_uar ? bfregi->num_sys_pages :
2015+ bfregi->num_static_sys_pages;
17462016
17472017 if (vma->vm_end - vma->vm_start != PAGE_SIZE)
17482018 return -EINVAL;
17492019
1750− uars_per_page = get_uars_per_sys_page(dev, bfregi->lib_uar_4k);
1751− idx = get_index(vma->vm_pgoff);
1752− if (idx % uars_per_page ||
1753− idx * uars_per_page >= bfregi->num_sys_pages) {
1754− mlx5_ib_warn(dev, "invalid uar index %lu\n", idx);
2020+ if (dyn_uar)
2021+ idx = get_extended_index(vma->vm_pgoff) + bfregi->num_static_sys_pages;
2022+ else
2023+ idx = get_index(vma->vm_pgoff);
2024+
2025+ if (idx >= max_valid_idx) {
2026+ mlx5_ib_warn(dev, "invalid uar index %lu, max=%d\n",
2027+ idx, max_valid_idx);
17552028 return -EINVAL;
17562029 }
17572030
17582031 switch (cmd) {
17592032 case MLX5_IB_MMAP_WC_PAGE:
2033+ case MLX5_IB_MMAP_ALLOC_WC:
17602034 /* Some architectures don't support WC memory */
17612035 #if defined(CONFIG_X86)
17622036 if (!pat_enabled())
@@ -1776,7 +2050,40 @@ static int uar_mmap(struct mlx5_ib_dev *dev, enum mlx5_ib_mmap_cmd cmd, · 13 unmodified lines
17762050 return -EINVAL;
17772051 }
17782052
1779− pfn = uar_index2pfn(dev, bfregi, idx);
2053+ if (dyn_uar) {
2054+ int uars_per_page;
2055+
2056+ uars_per_page = get_uars_per_sys_page(dev, bfregi->lib_uar_4k);
2057+ bfreg_dyn_idx = idx * (uars_per_page * MLX5_NON_FP_BFREGS_PER_UAR);
2058+ if (bfreg_dyn_idx >= bfregi->total_num_bfregs) {
2059+ mlx5_ib_warn(dev, "invalid bfreg_dyn_idx %u, max=%u\n",
2060+ bfreg_dyn_idx, bfregi->total_num_bfregs);
2061+ return -EINVAL;
2062+ }
2063+
2064+ mutex_lock(&bfregi->lock);
2065+ /* Fail if uar already allocated, first bfreg index of each
2066+ * page holds its count.
2067+ */
2068+ if (bfregi->count[bfreg_dyn_idx]) {
2069+ mlx5_ib_warn(dev, "wrong offset, idx %lu is busy, bfregn=%u\n", idx, bfreg_dyn_idx);
2070+ mutex_unlock(&bfregi->lock);
2071+ return -EINVAL;
2072+ }
2073+
2074+ bfregi->count[bfreg_dyn_idx]++;
2075+ mutex_unlock(&bfregi->lock);
2076+
2077+ err = mlx5_cmd_alloc_uar(dev->mdev, &uar_index);
2078+ if (err) {
2079+ mlx5_ib_warn(dev, "UAR alloc failed\n");
2080+ goto free_bfreg;
2081+ }
2082+ } else {
2083+ uar_index = bfregi->sys_pages[idx];
2084+ }
2085+
2086+ pfn = uar_index2pfn(dev, uar_index);
17802087 mlx5_ib_dbg(dev, "uar idx 0x%lx, pfn %pa\n", idx, &pfn);
17812088
17822089 vma->vm_page_prot = prot;
@@ -1785,14 +2092,32 @@ static int uar_mmap(struct mlx5_ib_dev *dev, enum mlx5_ib_mmap_cmd cmd, · 2 unmodified lines
17852092 if (err) {
17862093 mlx5_ib_err(dev, "io_remap_pfn_range failed with error=%d, vm_start=0x%lx, pfn=%pa, mmap_cmd=%s\n",
17872094 err, vma->vm_start, &pfn, mmap_cmd2str(cmd));
1788− return -EAGAIN;
2095+ err = -EAGAIN;
2096+ goto err;
17892097 }
17902098
17912099 pa = pfn << PAGE_SHIFT;
17922100 mlx5_ib_dbg(dev, "mapped %s at 0x%lx, PA %pa\n", mmap_cmd2str(cmd),
17932101 vma->vm_start, &pa);
17942102
1795− return mlx5_ib_set_vma_data(vma, context);
2103+ err = mlx5_ib_set_vma_data(vma, context);
2104+ if (err)
2105+ goto err;
2106+
2107+ if (dyn_uar)
2108+ bfregi->sys_pages[idx] = uar_index;
2109+ return 0;
2110+
2111+err:
2112+ if (!dyn_uar)
2113+ return err;
2114+
2115+ mlx5_cmd_free_uar(dev->mdev, idx);
2116+
2117+free_bfreg:
2118+ mlx5_ib_free_bfreg(dev, bfregi, bfreg_dyn_idx);
2119+
2120+ return err;
17962121 }
17972122
17982123 static int mlx5_ib_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vma)
@@ -1807,6 +2132,7 @@ static int mlx5_ib_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vm · 8 unmodified lines
18072132 case MLX5_IB_MMAP_WC_PAGE:
18082133 case MLX5_IB_MMAP_NC_PAGE:
18092134 case MLX5_IB_MMAP_REGULAR_PAGE:
2135+ case MLX5_IB_MMAP_ALLOC_WC:
18102136 return uar_mmap(dev, command, vma, context);
18112137
18122138 case MLX5_IB_MMAP_GET_CONTIGUOUS_PAGES:
@@ -1835,6 +2161,8 @@ static int mlx5_ib_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vm · 22 unmodified lines
18352161 vma->vm_start,
18362162 (unsigned long long)pfn << PAGE_SHIFT);
18372163 break;
2164+ case MLX5_IB_MMAP_CLOCK_INFO:
2165+ return mlx5_ib_mmap_clock_info_page(dev, vma, context);
18382166
18392167 default:
18402168 return -EINVAL;
@@ -2663,7 +2991,7 @@ static struct ib_flow *mlx5_ib_create_flow(struct ib_qp *qp, · 822 unmodified lines
26632991 return ERR_PTR(-ENOMEM);
26642992
26652993 if (domain != IB_FLOW_DOMAIN_USER ||
2666− flow_attr->port > MLX5_CAP_GEN(dev->mdev, num_ports) ||
2994+ flow_attr->port > dev->num_ports ||
26672995 (flow_attr->flags & ~IB_FLOW_ATTR_FLAGS_DONT_TRAP))
26682996 return ERR_PTR(-EINVAL);
26692997
@@ -2928,15 +3256,24 @@ static void delay_drop_handler(struct work_struct *work) · 258 unmodified lines
29283256 mutex_unlock(&delay_drop->lock);
29293257 }
29303258
2931−static void mlx5_ib_event(struct mlx5_core_dev *dev, void *context,
2932− enum mlx5_dev_event event, unsigned long param)
3259+static void mlx5_ib_handle_event(struct work_struct *_work)
29333260 {
2934− struct mlx5_ib_dev *ibdev = (struct mlx5_ib_dev *)context;
3261+ struct mlx5_ib_event_work *work =
3262+ container_of(_work, struct mlx5_ib_event_work, work);
3263+ struct mlx5_ib_dev *ibdev;
29353264 struct ib_event ibev;
29363265 bool fatal = false;
29373266 u8 port = 0;
29383267
2939− switch (event) {
3268+ if (mlx5_core_is_mp_slave(work->dev)) {
3269+ ibdev = mlx5_ib_get_ibdev_from_mpi(work->context);
3270+ if (!ibdev)
3271+ goto out;
3272+ } else {
3273+ ibdev = work->context;
3274+ }
3275+
3276+ switch (work->event) {
29403277 case MLX5_DEV_EVENT_SYS_ERROR:
29413278 ibev.event = IB_EVENT_DEVICE_FATAL;
29423279 mlx5_ib_handle_internal_error(ibdev);
@@ -2946,39 +3283,39 @@ static void mlx5_ib_event(struct mlx5_core_dev *dev, void *context, · 3 unmodified lines
29463283 case MLX5_DEV_EVENT_PORT_UP:
29473284 case MLX5_DEV_EVENT_PORT_DOWN:
29483285 case MLX5_DEV_EVENT_PORT_INITIALIZED:
2949− port = (u8)param;
3286+ port = (u8)work->param;
29503287
29513288 /* In RoCE, port up/down events are handled in
29523289 * mlx5_netdev_event().
29533290 */
29543291 if (mlx5_ib_port_link_layer(&ibdev->ib_dev, port) ==
29553292 IB_LINK_LAYER_ETHERNET)
2956− return;
3293+ goto out;
29573294
2958− ibev.event = (event == MLX5_DEV_EVENT_PORT_UP) ?
3295+ ibev.event = (work->event == MLX5_DEV_EVENT_PORT_UP) ?
29593296 IB_EVENT_PORT_ACTIVE : IB_EVENT_PORT_ERR;
29603297 break;
29613298
29623299 case MLX5_DEV_EVENT_LID_CHANGE:
29633300 ibev.event = IB_EVENT_LID_CHANGE;
2964− port = (u8)param;
3301+ port = (u8)work->param;
29653302 break;
29663303
29673304 case MLX5_DEV_EVENT_PKEY_CHANGE:
29683305 ibev.event = IB_EVENT_PKEY_CHANGE;
2969− port = (u8)param;
3306+ port = (u8)work->param;
29703307
29713308 schedule_work(&ibdev->devr.ports[port - 1].pkey_change_work);
29723309 break;
29733310
29743311 case MLX5_DEV_EVENT_GUID_CHANGE:
29753312 ibev.event = IB_EVENT_GID_CHANGE;
2976− port = (u8)param;
3313+ port = (u8)work->param;
29773314 break;
29783315
29793316 case MLX5_DEV_EVENT_CLIENT_REREG:
29803317 ibev.event = IB_EVENT_CLIENT_REREGISTER;
2981− port = (u8)param;
3318+ port = (u8)work->param;
29823319 break;
29833320 case MLX5_DEV_EVENT_DELAY_DROP_TIMEOUT:
29843321 schedule_work(&ibdev->delay_drop.delay_drop_work);
@@ -3000,9 +3337,26 @@ static void mlx5_ib_event(struct mlx5_core_dev *dev, void *context, · 15 unmodified lines
30003337
30013338 if (fatal)
30023339 ibdev->ib_active = false;
3003−
30043340 out:
3005− return;
3341+ kfree(work);
3342+}
3343+
3344+static void mlx5_ib_event(struct mlx5_core_dev *dev, void *context,
3345+ enum mlx5_dev_event event, unsigned long param)
3346+{
3347+ struct mlx5_ib_event_work *work;
3348+
3349+ work = kmalloc(sizeof(*work), GFP_ATOMIC);
3350+ if (!work)
3351+ return;
3352+
3353+ INIT_WORK(&work->work, mlx5_ib_handle_event);
3354+ work->dev = dev;
3355+ work->param = param;
3356+ work->context = context;
3357+ work->event = event;
3358+
3359+ queue_work(mlx5_ib_event_wq, &work->work);
30063360 }
30073361
30083362 static int set_has_smi_cap(struct mlx5_ib_dev *dev)
@@ -3011,7 +3365,7 @@ static int set_has_smi_cap(struct mlx5_ib_dev *dev) · 2 unmodified lines
30113365 int err;
30123366 int port;
30133367
3014− for (port = 1; port <= MLX5_CAP_GEN(dev->mdev, num_ports); port++) {
3368+ for (port = 1; port <= dev->num_ports; port++) {
30153369 dev->mdev->port_caps[port - 1].has_smi = false;
30163370 if (MLX5_CAP_GEN(dev->mdev, port_type) ==
30173371 MLX5_CAP_PORT_TYPE_IB) {
@@ -3038,16 +3392,15 @@ static void get_ext_port_caps(struct mlx5_ib_dev *dev) · 20 unmodified lines
30383392 {
30393393 int port;
30403394
3041− for (port = 1; port <= MLX5_CAP_GEN(dev->mdev, num_ports); port++)
3395+ for (port = 1; port <= dev->num_ports; port++)
30423396 mlx5_query_ext_port_caps(dev, port);
30433397 }
30443398
3045−static int get_port_caps(struct mlx5_ib_dev *dev)
3399+static int get_port_caps(struct mlx5_ib_dev *dev, u8 port)
30463400 {
30473401 struct ib_device_attr *dprops = NULL;
30483402 struct ib_port_attr *pprops = NULL;
30493403 int err = -ENOMEM;
3050− int port;
30513404 struct ib_udata uhw = {.inlen = 0, .outlen = 0};
30523405
30533406 pprops = kmalloc(sizeof(*pprops), GFP_KERNEL);
@@ -3068,22 +3421,21 @@ static int get_port_caps(struct mlx5_ib_dev *dev) · 14 unmodified lines
30683421 goto out;
30693422 }
30703423
3071− for (port = 1; port <= MLX5_CAP_GEN(dev->mdev, num_ports); port++) {
3072− memset(pprops, 0, sizeof(*pprops));
3073− err = mlx5_ib_query_port(&dev->ib_dev, port, pprops);
3074− if (err) {
3075− mlx5_ib_warn(dev, "query_port %d failed %d\n",
3076− port, err);
3077− break;
3078− }
3079− dev->mdev->port_caps[port - 1].pkey_table_len =
3080− dprops->max_pkeys;
3081− dev->mdev->port_caps[port - 1].gid_table_len =
3082− pprops->gid_tbl_len;
3083− mlx5_ib_dbg(dev, "pkey_table_len %d, gid_table_len %d\n",
3084− dprops->max_pkeys, pprops->gid_tbl_len);
3424+ memset(pprops, 0, sizeof(*pprops));
3425+ err = mlx5_ib_query_port(&dev->ib_dev, port, pprops);
3426+ if (err) {
3427+ mlx5_ib_warn(dev, "query_port %d failed %d\n",
3428+ port, err);
3429+ goto out;
30853430 }
30863431
3432+ dev->mdev->port_caps[port - 1].pkey_table_len =
3433+ dprops->max_pkeys;
3434+ dev->mdev->port_caps[port - 1].gid_table_len =
3435+ pprops->gid_tbl_len;
3436+ mlx5_ib_dbg(dev, "port %d: pkey_table_len %d, gid_table_len %d\n",
3437+ port, dprops->max_pkeys, pprops->gid_tbl_len);
3438+
30873439 out:
30883440 kfree(pprops);
30893441 kfree(dprops);
@@ -3373,12 +3725,14 @@ static u32 get_core_cap_flags(struct ib_device *ibdev) · 283 unmodified lines
33733725 enum rdma_link_layer ll = mlx5_ib_port_link_layer(ibdev, 1);
33743726 u8 l3_type_cap = MLX5_CAP_ROCE(dev->mdev, l3_type);
33753727 u8 roce_version_cap = MLX5_CAP_ROCE(dev->mdev, roce_version);
3728+ bool raw_support = !mlx5_core_mp_enabled(dev->mdev);
33763729 u32 ret = 0;
33773730
33783731 if (ll == IB_LINK_LAYER_INFINIBAND)
33793732 return RDMA_CORE_PORT_IBA_IB;
33803733
3381− ret = RDMA_CORE_PORT_RAW_PACKET;
3734+ if (raw_support)
3735+ ret = RDMA_CORE_PORT_RAW_PACKET;
33823736
33833737 if (!(l3_type_cap & MLX5_ROCE_L3_TYPE_IPV4_CAP))
33843738 return ret;
@@ -3468,33 +3822,33 @@ static void mlx5_eth_lag_cleanup(struct mlx5_ib_dev *dev) · 83 unmodified lines
34683822 }
34693823 }
34703824
3471−static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev)
3825+static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)
34723826 {
34733827 int err;
34743828
3475− dev->roce.nb.notifier_call = mlx5_netdev_event;
3476− err = register_netdevice_notifier(&dev->roce.nb);
3829+ dev->roce[port_num].nb.notifier_call = mlx5_netdev_event;
3830+ err = register_netdevice_notifier(&dev->roce[port_num].nb);
34773831 if (err) {
3478− dev->roce.nb.notifier_call = NULL;
3832+ dev->roce[port_num].nb.notifier_call = NULL;
34793833 return err;
34803834 }
34813835
34823836 return 0;
34833837 }
34843838
3485−static void mlx5_remove_netdev_notifier(struct mlx5_ib_dev *dev)
3839+static void mlx5_remove_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)
34863840 {
3487− if (dev->roce.nb.notifier_call) {
3488− unregister_netdevice_notifier(&dev->roce.nb);
3489− dev->roce.nb.notifier_call = NULL;
3841+ if (dev->roce[port_num].nb.notifier_call) {
3842+ unregister_netdevice_notifier(&dev->roce[port_num].nb);
3843+ dev->roce[port_num].nb.notifier_call = NULL;
34903844 }
34913845 }
34923846
3493−static int mlx5_enable_eth(struct mlx5_ib_dev *dev)
3847+static int mlx5_enable_eth(struct mlx5_ib_dev *dev, u8 port_num)
34943848 {
34953849 int err;
34963850
3497− err = mlx5_add_netdev_notifier(dev);
3851+ err = mlx5_add_netdev_notifier(dev, port_num);
34983852 if (err)
34993853 return err;
35003854
@@ -3515,7 +3869,7 @@ err_disable_roce: · 14 unmodified lines
35153869 mlx5_nic_vport_disable_roce(dev->mdev);
35163870
35173871 err_unregister_netdevice_notifier:
3518− mlx5_remove_netdev_notifier(dev);
3872+ mlx5_remove_netdev_notifier(dev, port_num);
35193873 return err;
35203874 }
35213875
@@ -3577,11 +3931,12 @@ static const struct mlx5_ib_counter extended_err_cnts[] = { · 55 unmodified lines
35773931
35783932 static void mlx5_ib_dealloc_counters(struct mlx5_ib_dev *dev)
35793933 {
3580− unsigned int i;
3934+ int i;
35813935
35823936 for (i = 0; i < dev->num_ports; i++) {
3583− mlx5_core_dealloc_q_counter(dev->mdev,
3584− dev->port[i].cnts.set_id);
3937+ if (dev->port[i].cnts.set_id)
3938+ mlx5_core_dealloc_q_counter(dev->mdev,
3939+ dev->port[i].cnts.set_id);
35853940 kfree(dev->port[i].cnts.names);
35863941 kfree(dev->port[i].cnts.offsets);
35873942 }
@@ -3623,6 +3978,7 @@ static int __mlx5_ib_alloc_counters(struct mlx5_ib_dev *dev, · 35 unmodified lines
36233978
36243979 err_names:
36253980 kfree(cnts->names);
3981+ cnts->names = NULL;
36263982 return -ENOMEM;
36273983 }
36283984
@@ -3669,37 +4025,33 @@ static void mlx5_ib_fill_counters(struct mlx5_ib_dev *dev, · 40 unmodified lines
36694025
36704026 static int mlx5_ib_alloc_counters(struct mlx5_ib_dev *dev)
36714027 {
4028+ int err = 0;
36724029 int i;
3673− int ret;
36744030
36754031 for (i = 0; i < dev->num_ports; i++) {
3676− struct mlx5_ib_port *port = &dev->port[i];
4032+ err = __mlx5_ib_alloc_counters(dev, &dev->port[i].cnts);
4033+ if (err)
4034+ goto err_alloc;
36774035
3678− ret = mlx5_core_alloc_q_counter(dev->mdev,
3679− &port->cnts.set_id);
3680− if (ret) {
4036+ mlx5_ib_fill_counters(dev, dev->port[i].cnts.names,
4037+ dev->port[i].cnts.offsets);
4038+
4039+ err = mlx5_core_alloc_q_counter(dev->mdev,
4040+ &dev->port[i].cnts.set_id);
4041+ if (err) {
36814042 mlx5_ib_warn(dev,
36824043 "couldn't allocate queue counter for port %d, err %d\n",
3683− i + 1, ret);
3684− goto dealloc_counters;
4044+ i + 1, err);
4045+ goto err_alloc;
36854046 }
3686−
3687− ret = __mlx5_ib_alloc_counters(dev, &port->cnts);
3688− if (ret)
3689− goto dealloc_counters;
3690−
3691− mlx5_ib_fill_counters(dev, port->cnts.names,
3692− port->cnts.offsets);
4047+ dev->port[i].cnts.set_id_valid = true;
36934048 }
36944049
36954050 return 0;
36964051
3697−dealloc_counters:
3698− while (--i >= 0)
3699− mlx5_core_dealloc_q_counter(dev->mdev,
3700− dev->port[i].cnts.set_id);
3701−
3702− return ret;
4052+err_alloc:
4053+ mlx5_ib_dealloc_counters(dev);
4054+ return err;
37034055 }
37044056
37054057 static struct rdma_hw_stats *mlx5_ib_alloc_hw_stats(struct ib_device *ibdev,
@@ -3718,7 +4070,7 @@ static struct rdma_hw_stats *mlx5_ib_alloc_hw_stats(struct ib_device *ibdev, · 12 unmodified lines
37184070 RDMA_HW_STATS_DEFAULT_LIFESPAN);
37194071 }
37204072
3721−static int mlx5_ib_query_q_counters(struct mlx5_ib_dev *dev,
4073+static int mlx5_ib_query_q_counters(struct mlx5_core_dev *mdev,
37224074 struct mlx5_ib_port *port,
37234075 struct rdma_hw_stats *stats)
37244076 {
@@ -3731,7 +4083,7 @@ static int mlx5_ib_query_q_counters(struct mlx5_ib_dev *dev, · 6 unmodified lines
37314083 if (!out)
37324084 return -ENOMEM;
37334085
3734− ret = mlx5_core_query_q_counter(dev->mdev,
4086+ ret = mlx5_core_query_q_counter(mdev,
37354087 port->cnts.set_id, 0,
37364088 out, outlen);
37374089 if (ret)
@@ -3753,28 +4105,43 @@ static int mlx5_ib_get_hw_stats(struct ib_device *ibdev, · 15 unmodified lines
37534105 {
37544106 struct mlx5_ib_dev *dev = to_mdev(ibdev);
37554107 struct mlx5_ib_port *port = &dev->port[port_num - 1];
4108+ struct mlx5_core_dev *mdev;
37564109 int ret, num_counters;
4110+ u8 mdev_port_num;
37574111
37584112 if (!stats)
37594113 return -EINVAL;
37604114
3761− ret = mlx5_ib_query_q_counters(dev, port, stats);
4115+ num_counters = port->cnts.num_q_counters + port->cnts.num_cong_counters;
4116+
4117+ /* q_counters are per IB device, query the master mdev */
4118+ ret = mlx5_ib_query_q_counters(dev->mdev, port, stats);
37624119 if (ret)
37634120 return ret;
3764− num_counters = port->cnts.num_q_counters;
37654121
37664122 if (MLX5_CAP_GEN(dev->mdev, cc_query_allowed)) {
4123+ mdev = mlx5_ib_get_native_port_mdev(dev, port_num,
4124+ &mdev_port_num);
4125+ if (!mdev) {
4126+ /* If port is not affiliated yet, its in down state
4127+ * which doesn't have any counters yet, so it would be
4128+ * zero. So no need to read from the HCA.
4129+ */
4130+ goto done;
4131+ }
37674132 ret = mlx5_lag_query_cong_counters(dev->mdev,
37684133 stats->value +
37694134 port->cnts.num_q_counters,
37704135 port->cnts.num_cong_counters,
37714136 port->cnts.offsets +
37724137 port->cnts.num_q_counters);
4138+
4139+ mlx5_ib_put_native_port_mdev(dev, port_num);
37734140 if (ret)
37744141 return ret;
3775− num_counters += port->cnts.num_cong_counters;
37764142 }
37774143
4144+done:
37784145 return num_counters;
37794146 }
37804147
@@ -3936,67 +4303,306 @@ mlx5_ib_get_vector_affinity(struct ib_device *ibdev, int comp_vector) · 155 unmodified lines
39364303 return mlx5_get_vector_affinity(dev->mdev, comp_vector);
39374304 }
39384305
3939−static void *mlx5_ib_add(struct mlx5_core_dev *mdev)
4306+/* The mlx5_ib_multiport_mutex should be held when calling this function */
4307+static void mlx5_ib_unbind_slave_port(struct mlx5_ib_dev *ibdev,
4308+ struct mlx5_ib_multiport_info *mpi)
39404309 {
3941− struct mlx5_ib_dev *dev;
3942− enum rdma_link_layer ll;
3943− int port_type_cap;
3944− const char *name;
4310+ u8 port_num = mlx5_core_native_port_num(mpi->mdev) - 1;
4311+ struct mlx5_ib_port *port = &ibdev->port[port_num];
4312+ int comps;
39454313 int err;
39464314 int i;
39474315
3948− port_type_cap = MLX5_CAP_GEN(mdev, port_type);
3949− ll = mlx5_port_type_cap_to_rdma_ll(port_type_cap);
4316+ mlx5_ib_cleanup_cong_debugfs(ibdev, port_num);
39504317
3951− printk_once(KERN_INFO "%s", mlx5_version);
4318+ spin_lock(&port->mp.mpi_lock);
4319+ if (!mpi->ibdev) {
4320+ spin_unlock(&port->mp.mpi_lock);
4321+ return;
4322+ }
4323+ mpi->ibdev = NULL;
39524324
3953− dev = (struct mlx5_ib_dev *)ib_alloc_device(sizeof(*dev));
3954− if (!dev)
3955− return NULL;
4325+ spin_unlock(&port->mp.mpi_lock);
4326+ mlx5_remove_netdev_notifier(ibdev, port_num);
4327+ spin_lock(&port->mp.mpi_lock);
39564328
3957− dev->mdev = mdev;
4329+ comps = mpi->mdev_refcnt;
4330+ if (comps) {
4331+ mpi->unaffiliate = true;
4332+ init_completion(&mpi->unref_comp);
4333+ spin_unlock(&port->mp.mpi_lock);
39584334
3959− dev->port = kcalloc(MLX5_CAP_GEN(mdev, num_ports), sizeof(*dev->port),
3960− GFP_KERNEL);
3961− if (!dev->port)
3962− goto err_dealloc;
4335+ for (i = 0; i < comps; i++)
4336+ wait_for_completion(&mpi->unref_comp);
4337+
4338+ spin_lock(&port->mp.mpi_lock);
4339+ mpi->unaffiliate = false;
4340+ }
4341+
4342+ port->mp.mpi = NULL;
39634343
3964− rwlock_init(&dev->roce.netdev_lock);
3965− err = get_port_caps(dev);
4344+ list_add_tail(&mpi->list, &mlx5_ib_unaffiliated_port_list);
4345+
4346+ spin_unlock(&port->mp.mpi_lock);
4347+
4348+ err = mlx5_nic_vport_unaffiliate_multiport(mpi->mdev);
4349+
4350+ mlx5_ib_dbg(ibdev, "unaffiliated port %d\n", port_num + 1);
4351+ /* Log an error, still needed to cleanup the pointers and add
4352+ * it back to the list.
4353+ */
39664354 if (err)
3967− goto err_free_port;
4355+ mlx5_ib_err(ibdev, "Failed to unaffiliate port %u\n",
4356+ port_num + 1);
39684357
3969− if (mlx5_use_mad_ifc(dev))
3970− get_ext_port_caps(dev);
4358+ ibdev->roce[port_num].last_port_state = IB_PORT_DOWN;
4359+}
39714360
3972− if (!mlx5_lag_is_active(mdev))
3973− name = "mlx5_%d";
3974− else
3975− name = "mlx5_bond_%d";
4361+/* The mlx5_ib_multiport_mutex should be held when calling this function */
4362+static bool mlx5_ib_bind_slave_port(struct mlx5_ib_dev *ibdev,
4363+ struct mlx5_ib_multiport_info *mpi)
4364+{
4365+ u8 port_num = mlx5_core_native_port_num(mpi->mdev) - 1;
4366+ int err;
39764367
3977− strlcpy(dev->ib_dev.name, name, IB_DEVICE_NAME_MAX);
3978− dev->ib_dev.owner = THIS_MODULE;
3979− dev->ib_dev.node_type = RDMA_NODE_IB_CA;
3980− dev->ib_dev.local_dma_lkey = 0 /* not supported for now */;
3981− dev->num_ports = MLX5_CAP_GEN(mdev, num_ports);
3982− dev->ib_dev.phys_port_cnt = dev->num_ports;
3983− dev->ib_dev.num_comp_vectors =
3984− dev->mdev->priv.eq_table.num_comp_vectors;
3985− dev->ib_dev.dev.parent = &mdev->pdev->dev;
4368+ spin_lock(&ibdev->port[port_num].mp.mpi_lock);
4369+ if (ibdev->port[port_num].mp.mpi) {
4370+ mlx5_ib_warn(ibdev, "port %d already affiliated.\n",
4371+ port_num + 1);
4372+ spin_unlock(&ibdev->port[port_num].mp.mpi_lock);
4373+ return false;
4374+ }
39864375
3987− dev->ib_dev.uverbs_abi_ver = MLX5_IB_UVERBS_ABI_VERSION;
3988− dev->ib_dev.uverbs_cmd_mask =
3989− (1ull << IB_USER_VERBS_CMD_GET_CONTEXT) |
3990− (1ull << IB_USER_VERBS_CMD_QUERY_DEVICE) |
3991− (1ull << IB_USER_VERBS_CMD_QUERY_PORT) |
3992− (1ull << IB_USER_VERBS_CMD_ALLOC_PD) |
3993− (1ull << IB_USER_VERBS_CMD_DEALLOC_PD) |
3994− (1ull << IB_USER_VERBS_CMD_CREATE_AH) |
3995− (1ull << IB_USER_VERBS_CMD_DESTROY_AH) |
3996− (1ull << IB_USER_VERBS_CMD_REG_MR) |
3997− (1ull << IB_USER_VERBS_CMD_REREG_MR) |
3998− (1ull << IB_USER_VERBS_CMD_DEREG_MR) |
3999− (1ull << IB_USER_VERBS_CMD_CREATE_COMP_CHANNEL) |
4376+ ibdev->port[port_num].mp.mpi = mpi;
4377+ mpi->ibdev = ibdev;
4378+ spin_unlock(&ibdev->port[port_num].mp.mpi_lock);
4379+
4380+ err = mlx5_nic_vport_affiliate_multiport(ibdev->mdev, mpi->mdev);
4381+ if (err)
4382+ goto unbind;
4383+
4384+ err = get_port_caps(ibdev, mlx5_core_native_port_num(mpi->mdev));
4385+ if (err)
4386+ goto unbind;
4387+
4388+ err = mlx5_add_netdev_notifier(ibdev, port_num);
4389+ if (err) {
4390+ mlx5_ib_err(ibdev, "failed adding netdev notifier for port %u\n",
4391+ port_num + 1);
4392+ goto unbind;
4393+ }
4394+
4395+ err = mlx5_ib_init_cong_debugfs(ibdev, port_num);
4396+ if (err)
4397+ goto unbind;
4398+
4399+ return true;
4400+
4401+unbind:
4402+ mlx5_ib_unbind_slave_port(ibdev, mpi);
4403+ return false;
4404+}
4405+
4406+static int mlx5_ib_init_multiport_master(struct mlx5_ib_dev *dev)
4407+{
4408+ int port_num = mlx5_core_native_port_num(dev->mdev) - 1;
4409+ enum rdma_link_layer ll = mlx5_ib_port_link_layer(&dev->ib_dev,
4410+ port_num + 1);
4411+ struct mlx5_ib_multiport_info *mpi;
4412+ int err;
4413+ int i;
4414+
4415+ if (!mlx5_core_is_mp_master(dev->mdev) || ll != IB_LINK_LAYER_ETHERNET)
4416+ return 0;
4417+
4418+ err = mlx5_query_nic_vport_system_image_guid(dev->mdev,
4419+ &dev->sys_image_guid);
4420+ if (err)
4421+ return err;
4422+
4423+ err = mlx5_nic_vport_enable_roce(dev->mdev);
4424+ if (err)
4425+ return err;
4426+
4427+ mutex_lock(&mlx5_ib_multiport_mutex);
4428+ for (i = 0; i < dev->num_ports; i++) {
4429+ bool bound = false;
4430+
4431+ /* build a stub multiport info struct for the native port. */
4432+ if (i == port_num) {
4433+ mpi = kzalloc(sizeof(*mpi), GFP_KERNEL);
4434+ if (!mpi) {
4435+ mutex_unlock(&mlx5_ib_multiport_mutex);
4436+ mlx5_nic_vport_disable_roce(dev->mdev);
4437+ return -ENOMEM;
4438+ }
4439+
4440+ mpi->is_master = true;
4441+ mpi->mdev = dev->mdev;
4442+ mpi->sys_image_guid = dev->sys_image_guid;
4443+ dev->port[i].mp.mpi = mpi;
4444+ mpi->ibdev = dev;
4445+ mpi = NULL;
4446+ continue;
4447+ }
4448+
4449+ list_for_each_entry(mpi, &mlx5_ib_unaffiliated_port_list,
4450+ list) {
4451+ if (dev->sys_image_guid == mpi->sys_image_guid &&
4452+ (mlx5_core_native_port_num(mpi->mdev) - 1) == i) {
4453+ bound = mlx5_ib_bind_slave_port(dev, mpi);
4454+ }
4455+
4456+ if (bound) {
4457+ dev_dbg(&mpi->mdev->pdev->dev, "removing port from unaffiliated list.\n");
4458+ mlx5_ib_dbg(dev, "port %d bound\n", i + 1);
4459+ list_del(&mpi->list);
4460+ break;
4461+ }
4462+ }
4463+ if (!bound) {
4464+ get_port_caps(dev, i + 1);
4465+ mlx5_ib_dbg(dev, "no free port found for port %d\n",
4466+ i + 1);
4467+ }
4468+ }
4469+
4470+ list_add_tail(&dev->ib_dev_list, &mlx5_ib_dev_list);
4471+ mutex_unlock(&mlx5_ib_multiport_mutex);
4472+ return err;
4473+}
4474+
4475+static void mlx5_ib_cleanup_multiport_master(struct mlx5_ib_dev *dev)
4476+{
4477+ int port_num = mlx5_core_native_port_num(dev->mdev) - 1;
4478+ enum rdma_link_layer ll = mlx5_ib_port_link_layer(&dev->ib_dev,
4479+ port_num + 1);
4480+ int i;
4481+
4482+ if (!mlx5_core_is_mp_master(dev->mdev) || ll != IB_LINK_LAYER_ETHERNET)
4483+ return;
4484+
4485+ mutex_lock(&mlx5_ib_multiport_mutex);
4486+ for (i = 0; i < dev->num_ports; i++) {
4487+ if (dev->port[i].mp.mpi) {
4488+ /* Destroy the native port stub */
4489+ if (i == port_num) {
4490+ kfree(dev->port[i].mp.mpi);
4491+ dev->port[i].mp.mpi = NULL;
4492+ } else {
4493+ mlx5_ib_dbg(dev, "unbinding port_num: %d\n", i + 1);
4494+ mlx5_ib_unbind_slave_port(dev, dev->port[i].mp.mpi);
4495+ }
4496+ }
4497+ }
4498+
4499+ mlx5_ib_dbg(dev, "removing from devlist\n");
4500+ list_del(&dev->ib_dev_list);
4501+ mutex_unlock(&mlx5_ib_multiport_mutex);
4502+
4503+ mlx5_nic_vport_disable_roce(dev->mdev);
4504+}
4505+
4506+static void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev)
4507+{
4508+ mlx5_ib_cleanup_multiport_master(dev);
4509+#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
4510+ cleanup_srcu_struct(&dev->mr_srcu);
4511+#endif
4512+ kfree(dev->port);
4513+}
4514+
4515+static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
4516+{
4517+ struct mlx5_core_dev *mdev = dev->mdev;
4518+ const char *name;
4519+ int err;
4520+ int i;
4521+
4522+ dev->port = kcalloc(dev->num_ports, sizeof(*dev->port),
4523+ GFP_KERNEL);
4524+ if (!dev->port)
4525+ return -ENOMEM;
4526+
4527+ for (i = 0; i < dev->num_ports; i++) {
4528+ spin_lock_init(&dev->port[i].mp.mpi_lock);
4529+ rwlock_init(&dev->roce[i].netdev_lock);
4530+ }
4531+
4532+ err = mlx5_ib_init_multiport_master(dev);
4533+ if (err)
4534+ goto err_free_port;
4535+
4536+ if (!mlx5_core_mp_enabled(mdev)) {
4537+ int i;
4538+
4539+ for (i = 1; i <= dev->num_ports; i++) {
4540+ err = get_port_caps(dev, i);
4541+ if (err)
4542+ break;
4543+ }
4544+ } else {
4545+ err = get_port_caps(dev, mlx5_core_native_port_num(mdev));
4546+ }
4547+ if (err)
4548+ goto err_mp;
4549+
4550+ if (mlx5_use_mad_ifc(dev))
4551+ get_ext_port_caps(dev);
4552+
4553+ if (!mlx5_lag_is_active(mdev))
4554+ name = "mlx5_%d";
4555+ else
4556+ name = "mlx5_bond_%d";
4557+
4558+ strlcpy(dev->ib_dev.name, name, IB_DEVICE_NAME_MAX);
4559+ dev->ib_dev.owner = THIS_MODULE;
4560+ dev->ib_dev.node_type = RDMA_NODE_IB_CA;
4561+ dev->ib_dev.local_dma_lkey = 0 /* not supported for now */;
4562+ dev->ib_dev.phys_port_cnt = dev->num_ports;
4563+ dev->ib_dev.num_comp_vectors =
4564+ dev->mdev->priv.eq_table.num_comp_vectors;
4565+ dev->ib_dev.dev.parent = &mdev->pdev->dev;
4566+
4567+ mutex_init(&dev->flow_db.lock);
4568+ mutex_init(&dev->cap_mask_mutex);
4569+ INIT_LIST_HEAD(&dev->qp_list);
4570+ spin_lock_init(&dev->reset_flow_resource_lock);
4571+
4572+#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
4573+ err = init_srcu_struct(&dev->mr_srcu);
4574+ if (err)
4575+ goto err_free_port;
4576+#endif
4577+
4578+ return 0;
4579+err_mp:
4580+ mlx5_ib_cleanup_multiport_master(dev);
4581+
4582+err_free_port:
4583+ kfree(dev->port);
4584+
4585+ return -ENOMEM;
4586+}
4587+
4588+static int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev)
4589+{
4590+ struct mlx5_core_dev *mdev = dev->mdev;
4591+ int err;
4592+
4593+ dev->ib_dev.uverbs_abi_ver = MLX5_IB_UVERBS_ABI_VERSION;
4594+ dev->ib_dev.uverbs_cmd_mask =
4595+ (1ull << IB_USER_VERBS_CMD_GET_CONTEXT) |
4596+ (1ull << IB_USER_VERBS_CMD_QUERY_DEVICE) |
4597+ (1ull << IB_USER_VERBS_CMD_QUERY_PORT) |
4598+ (1ull << IB_USER_VERBS_CMD_ALLOC_PD) |
4599+ (1ull << IB_USER_VERBS_CMD_DEALLOC_PD) |
4600+ (1ull << IB_USER_VERBS_CMD_CREATE_AH) |
4601+ (1ull << IB_USER_VERBS_CMD_DESTROY_AH) |
4602+ (1ull << IB_USER_VERBS_CMD_REG_MR) |
4603+ (1ull << IB_USER_VERBS_CMD_REREG_MR) |
4604+ (1ull << IB_USER_VERBS_CMD_DEREG_MR) |
4605+ (1ull << IB_USER_VERBS_CMD_CREATE_COMP_CHANNEL) |
40004606 (1ull << IB_USER_VERBS_CMD_CREATE_CQ) |
40014607 (1ull << IB_USER_VERBS_CMD_RESIZE_CQ) |
40024608 (1ull << IB_USER_VERBS_CMD_DESTROY_CQ) |
@@ -4022,8 +4628,6 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev) · 19 unmodified lines
40224628 dev->ib_dev.query_device = mlx5_ib_query_device;
40234629 dev->ib_dev.query_port = mlx5_ib_query_port;
40244630 dev->ib_dev.get_link_layer = mlx5_ib_port_link_layer;
4025− if (ll == IB_LINK_LAYER_ETHERNET)
4026− dev->ib_dev.get_netdev = mlx5_ib_get_netdev;
40274631 dev->ib_dev.query_gid = mlx5_ib_query_gid;
40284632 dev->ib_dev.add_gid = mlx5_ib_add_gid;
40294633 dev->ib_dev.del_gid = mlx5_ib_del_gid;
@@ -4080,8 +4684,6 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev) · 50 unmodified lines
40804684
40814685 dev->ib_dev.disassociate_ucontext = mlx5_ib_disassociate_ucontext;
40824686
4083− mlx5_ib_internal_fill_odp_caps(dev);
4084−
40854687 dev->umr_fence = mlx5_get_umr_fence(MLX5_CAP_GEN(mdev, umr_fence));
40864688
40874689 if (MLX5_CAP_GEN(mdev, imaicl)) {
@@ -4092,11 +4694,6 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev) · 4 unmodified lines
40924694 (1ull << IB_USER_VERBS_CMD_DEALLOC_MW);
40934695 }
40944696
4095− if (MLX5_CAP_GEN(dev->mdev, max_qp_cnt)) {
4096− dev->ib_dev.get_hw_stats = mlx5_ib_get_hw_stats;
4097− dev->ib_dev.alloc_hw_stats = mlx5_ib_alloc_hw_stats;
4098− }
4099−
41004697 if (MLX5_CAP_GEN(mdev, xrc)) {
41014698 dev->ib_dev.alloc_xrcd = mlx5_ib_alloc_xrcd;
41024699 dev->ib_dev.dealloc_xrcd = mlx5_ib_dealloc_xrcd;
@@ -4111,8 +4708,39 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev) · 8 unmodified lines
41114708 (1ull << IB_USER_VERBS_EX_CMD_CREATE_FLOW) |
41124709 (1ull << IB_USER_VERBS_EX_CMD_DESTROY_FLOW);
41134710
4114− if (mlx5_ib_port_link_layer(&dev->ib_dev, 1) ==
4115− IB_LINK_LAYER_ETHERNET) {
4711+ err = init_node_data(dev);
4712+ if (err)
4713+ return err;
4714+
4715+ if ((MLX5_CAP_GEN(dev->mdev, port_type) == MLX5_CAP_PORT_TYPE_ETH) &&
4716+ (MLX5_CAP_GEN(dev->mdev, disable_local_lb_uc) ||
4717+ MLX5_CAP_GEN(dev->mdev, disable_local_lb_mc)))
4718+ mutex_init(&dev->lb_mutex);
4719+
4720+ return 0;
4721+}
4722+
4723+static int mlx5_ib_stage_roce_init(struct mlx5_ib_dev *dev)
4724+{
4725+ struct mlx5_core_dev *mdev = dev->mdev;
4726+ enum rdma_link_layer ll;
4727+ int port_type_cap;
4728+ u8 port_num;
4729+ int err;
4730+ int i;
4731+
4732+ port_num = mlx5_core_native_port_num(dev->mdev) - 1;
4733+ port_type_cap = MLX5_CAP_GEN(mdev, port_type);
4734+ ll = mlx5_port_type_cap_to_rdma_ll(port_type_cap);
4735+
4736+ if (ll == IB_LINK_LAYER_ETHERNET) {
4737+ for (i = 0; i < dev->num_ports; i++) {
4738+ dev->roce[i].dev = dev;
4739+ dev->roce[i].native_port_num = i + 1;
4740+ dev->roce[i].last_port_state = IB_PORT_DOWN;
4741+ }
4742+
4743+ dev->ib_dev.get_netdev = mlx5_ib_get_netdev;
41164744 dev->ib_dev.create_wq = mlx5_ib_create_wq;
41174745 dev->ib_dev.modify_wq = mlx5_ib_modify_wq;
41184746 dev->ib_dev.destroy_wq = mlx5_ib_destroy_wq;
@@ -4124,143 +4752,329 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev) · 5 unmodified lines
41244752 (1ull << IB_USER_VERBS_EX_CMD_DESTROY_WQ) |
41254753 (1ull << IB_USER_VERBS_EX_CMD_CREATE_RWQ_IND_TBL) |
41264754 (1ull << IB_USER_VERBS_EX_CMD_DESTROY_RWQ_IND_TBL);
4755+ err = mlx5_enable_eth(dev, port_num);
4756+ if (err)
4757+ return err;
41274758 }
4128− err = init_node_data(dev);
4129− if (err)
4130− goto err_free_port;
41314759
4132− mutex_init(&dev->flow_db.lock);
4133− mutex_init(&dev->cap_mask_mutex);
4134− INIT_LIST_HEAD(&dev->qp_list);
4135− spin_lock_init(&dev->reset_flow_resource_lock);
4760+ return 0;
4761+}
4762+
4763+static void mlx5_ib_stage_roce_cleanup(struct mlx5_ib_dev *dev)
4764+{
4765+ struct mlx5_core_dev *mdev = dev->mdev;
4766+ enum rdma_link_layer ll;
4767+ int port_type_cap;
4768+ u8 port_num;
4769+
4770+ port_num = mlx5_core_native_port_num(dev->mdev) - 1;
4771+ port_type_cap = MLX5_CAP_GEN(mdev, port_type);
4772+ ll = mlx5_port_type_cap_to_rdma_ll(port_type_cap);
41364773
41374774 if (ll == IB_LINK_LAYER_ETHERNET) {
4138− err = mlx5_enable_eth(dev);
4139− if (err)
4140− goto err_free_port;
4141− dev->roce.last_port_state = IB_PORT_DOWN;
4775+ mlx5_disable_eth(dev);
4776+ mlx5_remove_netdev_notifier(dev, port_num);
41424777 }
4778+}
41434779
4144− err = create_dev_resources(&dev->devr);
4145− if (err)
4146− goto err_disable_eth;
4780+static int mlx5_ib_stage_dev_res_init(struct mlx5_ib_dev *dev)
4781+{
4782+ return create_dev_resources(&dev->devr);
4783+}
41474784
4148− err = mlx5_ib_odp_init_one(dev);
4149− if (err)
4150− goto err_rsrc;
4785+static void mlx5_ib_stage_dev_res_cleanup(struct mlx5_ib_dev *dev)
4786+{
4787+ destroy_dev_resources(&dev->devr);
4788+}
4789+
4790+static int mlx5_ib_stage_odp_init(struct mlx5_ib_dev *dev)
4791+{
4792+ mlx5_ib_internal_fill_odp_caps(dev);
4793+
4794+ return mlx5_ib_odp_init_one(dev);
4795+}
41514796
4797+static int mlx5_ib_stage_counters_init(struct mlx5_ib_dev *dev)
4798+{
41524799 if (MLX5_CAP_GEN(dev->mdev, max_qp_cnt)) {
4153− err = mlx5_ib_alloc_counters(dev);
4154− if (err)
4155− goto err_odp;
4800+ dev->ib_dev.get_hw_stats = mlx5_ib_get_hw_stats;
4801+ dev->ib_dev.alloc_hw_stats = mlx5_ib_alloc_hw_stats;
4802+
4803+ return mlx5_ib_alloc_counters(dev);
41564804 }
41574805
4158− err = mlx5_ib_init_cong_debugfs(dev);
4159− if (err)
4160− goto err_cnt;
4806+ return 0;
4807+}
4808+
4809+static void mlx5_ib_stage_counters_cleanup(struct mlx5_ib_dev *dev)
4810+{
4811+ if (MLX5_CAP_GEN(dev->mdev, max_qp_cnt))
4812+ mlx5_ib_dealloc_counters(dev);
4813+}
4814+
4815+static int mlx5_ib_stage_cong_debugfs_init(struct mlx5_ib_dev *dev)
4816+{
4817+ return mlx5_ib_init_cong_debugfs(dev,
4818+ mlx5_core_native_port_num(dev->mdev) - 1);
4819+}
4820+
4821+static void mlx5_ib_stage_cong_debugfs_cleanup(struct mlx5_ib_dev *dev)
4822+{
4823+ mlx5_ib_cleanup_cong_debugfs(dev,
4824+ mlx5_core_native_port_num(dev->mdev) - 1);
4825+}
41614826
4827+static int mlx5_ib_stage_uar_init(struct mlx5_ib_dev *dev)
4828+{
41624829 dev->mdev->priv.uar = mlx5_get_uars_page(dev->mdev);
4163− if (IS_ERR(dev->mdev->priv.uar))
4164− goto err_cong;
4830+ if (!dev->mdev->priv.uar)
4831+ return -ENOMEM;
4832+ return 0;
4833+}
4834+
4835+static void mlx5_ib_stage_uar_cleanup(struct mlx5_ib_dev *dev)
4836+{
4837+ mlx5_put_uars_page(dev->mdev, dev->mdev->priv.uar);
4838+}
4839+
4840+static int mlx5_ib_stage_bfrag_init(struct mlx5_ib_dev *dev)
4841+{
4842+ int err;
41654843
41664844 err = mlx5_alloc_bfreg(dev->mdev, &dev->bfreg, false, false);
41674845 if (err)
4168− goto err_uar_page;
4846+ return err;
41694847
41704848 err = mlx5_alloc_bfreg(dev->mdev, &dev->fp_bfreg, false, true);
41714849 if (err)
4172− goto err_bfreg;
4850+ mlx5_free_bfreg(dev->mdev, &dev->fp_bfreg);
41734851
4174− err = ib_register_device(&dev->ib_dev, NULL);
4175− if (err)
4176− goto err_fp_bfreg;
4852+ return err;
4853+}
41774854
4178− err = create_umr_res(dev);
4179− if (err)
4180− goto err_dev;
4855+static void mlx5_ib_stage_bfrag_cleanup(struct mlx5_ib_dev *dev)
4856+{
4857+ mlx5_free_bfreg(dev->mdev, &dev->fp_bfreg);
4858+ mlx5_free_bfreg(dev->mdev, &dev->bfreg);
4859+}
41814860
4861+static int mlx5_ib_stage_ib_reg_init(struct mlx5_ib_dev *dev)
4862+{
4863+ return ib_register_device(&dev->ib_dev, NULL);
4864+}
4865+
4866+static void mlx5_ib_stage_ib_reg_cleanup(struct mlx5_ib_dev *dev)
4867+{
4868+ ib_unregister_device(&dev->ib_dev);
4869+}
4870+
4871+static int mlx5_ib_stage_umr_res_init(struct mlx5_ib_dev *dev)
4872+{
4873+ return create_umr_res(dev);
4874+}
4875+
4876+static void mlx5_ib_stage_umr_res_cleanup(struct mlx5_ib_dev *dev)
4877+{
4878+ destroy_umrc_res(dev);
4879+}
4880+
4881+static int mlx5_ib_stage_delay_drop_init(struct mlx5_ib_dev *dev)
4882+{
41824883 init_delay_drop(dev);
41834884
4885+ return 0;
4886+}
4887+
4888+static void mlx5_ib_stage_delay_drop_cleanup(struct mlx5_ib_dev *dev)
4889+{
4890+ cancel_delay_drop(dev);
4891+}
4892+
4893+static int mlx5_ib_stage_class_attr_init(struct mlx5_ib_dev *dev)
4894+{
4895+ int err;
4896+ int i;
4897+
41844898 for (i = 0; i < ARRAY_SIZE(mlx5_class_attributes); i++) {
41854899 err = device_create_file(&dev->ib_dev.dev,
41864900 mlx5_class_attributes[i]);
41874901 if (err)
4188− goto err_delay_drop;
4902+ return err;
41894903 }
41904904
4191− if ((MLX5_CAP_GEN(mdev, port_type) == MLX5_CAP_PORT_TYPE_ETH) &&
4192− (MLX5_CAP_GEN(mdev, disable_local_lb_uc) ||
4193− MLX5_CAP_GEN(mdev, disable_local_lb_mc)))
4194− mutex_init(&dev->lb_mutex);
4905+ return 0;
4906+}
4907+
4908+static void __mlx5_ib_remove(struct mlx5_ib_dev *dev,
4909+ const struct mlx5_ib_profile *profile,
4910+ int stage)
4911+{
4912+ /* Number of stages to cleanup */
4913+ while (stage) {
4914+ stage--;
4915+ if (profile->stage[stage].cleanup)
4916+ profile->stage[stage].cleanup(dev);
4917+ }
4918+
4919+ ib_dealloc_device((struct ib_device *)dev);
4920+}
4921+
4922+static void *mlx5_ib_add_slave_port(struct mlx5_core_dev *mdev, u8 port_num);
41954923
4924+static void *__mlx5_ib_add(struct mlx5_core_dev *mdev,
4925+ const struct mlx5_ib_profile *profile)
4926+{
4927+ struct mlx5_ib_dev *dev;
4928+ int err;
4929+ int i;
4930+
4931+ printk_once(KERN_INFO "%s", mlx5_version);
4932+
4933+ dev = (struct mlx5_ib_dev *)ib_alloc_device(sizeof(*dev));
4934+ if (!dev)
4935+ return NULL;
4936+
4937+ dev->mdev = mdev;
4938+ dev->num_ports = max(MLX5_CAP_GEN(mdev, num_ports),
4939+ MLX5_CAP_GEN(mdev, num_vhca_ports));
4940+
4941+ for (i = 0; i < MLX5_IB_STAGE_MAX; i++) {
4942+ if (profile->stage[i].init) {
4943+ err = profile->stage[i].init(dev);
4944+ if (err)
4945+ goto err_out;
4946+ }
4947+ }
4948+
4949+ dev->profile = profile;
41964950 dev->ib_active = true;
41974951
41984952 return dev;
41994953
4200−err_delay_drop:
4201− cancel_delay_drop(dev);
4202− destroy_umrc_res(dev);
4954+err_out:
4955+ __mlx5_ib_remove(dev, profile, i);
42034956
4204−err_dev:
4205− ib_unregister_device(&dev->ib_dev);
4957+ return NULL;
4958+}
42064959
4207−err_fp_bfreg:
4208− mlx5_free_bfreg(dev->mdev, &dev->fp_bfreg);
4960+static const struct mlx5_ib_profile pf_profile = {
4961+ STAGE_CREATE(MLX5_IB_STAGE_INIT,
4962+ mlx5_ib_stage_init_init,
4963+ mlx5_ib_stage_init_cleanup),
4964+ STAGE_CREATE(MLX5_IB_STAGE_CAPS,
4965+ mlx5_ib_stage_caps_init,
4966+ NULL),
4967+ STAGE_CREATE(MLX5_IB_STAGE_ROCE,
4968+ mlx5_ib_stage_roce_init,
4969+ mlx5_ib_stage_roce_cleanup),
4970+ STAGE_CREATE(MLX5_IB_STAGE_DEVICE_RESOURCES,
4971+ mlx5_ib_stage_dev_res_init,
4972+ mlx5_ib_stage_dev_res_cleanup),
4973+ STAGE_CREATE(MLX5_IB_STAGE_ODP,
4974+ mlx5_ib_stage_odp_init,
4975+ NULL),
4976+ STAGE_CREATE(MLX5_IB_STAGE_COUNTERS,
4977+ mlx5_ib_stage_counters_init,
4978+ mlx5_ib_stage_counters_cleanup),
4979+ STAGE_CREATE(MLX5_IB_STAGE_CONG_DEBUGFS,
4980+ mlx5_ib_stage_cong_debugfs_init,
4981+ mlx5_ib_stage_cong_debugfs_cleanup),
4982+ STAGE_CREATE(MLX5_IB_STAGE_UAR,
4983+ mlx5_ib_stage_uar_init,
4984+ mlx5_ib_stage_uar_cleanup),
4985+ STAGE_CREATE(MLX5_IB_STAGE_BFREG,
4986+ mlx5_ib_stage_bfrag_init,
4987+ mlx5_ib_stage_bfrag_cleanup),
4988+ STAGE_CREATE(MLX5_IB_STAGE_IB_REG,
4989+ mlx5_ib_stage_ib_reg_init,
4990+ mlx5_ib_stage_ib_reg_cleanup),
4991+ STAGE_CREATE(MLX5_IB_STAGE_UMR_RESOURCES,
4992+ mlx5_ib_stage_umr_res_init,
4993+ mlx5_ib_stage_umr_res_cleanup),
4994+ STAGE_CREATE(MLX5_IB_STAGE_DELAY_DROP,
4995+ mlx5_ib_stage_delay_drop_init,
4996+ mlx5_ib_stage_delay_drop_cleanup),
4997+ STAGE_CREATE(MLX5_IB_STAGE_CLASS_ATTR,
4998+ mlx5_ib_stage_class_attr_init,
4999+ NULL),
5000+};
42095001
4210−err_bfreg:
4211− mlx5_free_bfreg(dev->mdev, &dev->bfreg);
5002+static void *mlx5_ib_add_slave_port(struct mlx5_core_dev *mdev, u8 port_num)
5003+{
5004+ struct mlx5_ib_multiport_info *mpi;
5005+ struct mlx5_ib_dev *dev;
5006+ bool bound = false;
5007+ int err;
42125008
4213−err_uar_page:
4214− mlx5_put_uars_page(dev->mdev, dev->mdev->priv.uar);
5009+ mpi = kzalloc(sizeof(*mpi), GFP_KERNEL);
5010+ if (!mpi)
5011+ return NULL;
42155012
4216−err_cong:
4217− mlx5_ib_cleanup_cong_debugfs(dev);
4218−err_cnt:
4219− if (MLX5_CAP_GEN(dev->mdev, max_qp_cnt))
4220− mlx5_ib_dealloc_counters(dev);
5013+ mpi->mdev = mdev;
42215014
4222−err_odp:
4223− mlx5_ib_odp_remove_one(dev);
5015+ err = mlx5_query_nic_vport_system_image_guid(mdev,
5016+ &mpi->sys_image_guid);
5017+ if (err) {
5018+ kfree(mpi);
5019+ return NULL;
5020+ }
42245021
4225−err_rsrc:
4226− destroy_dev_resources(&dev->devr);
5022+ mutex_lock(&mlx5_ib_multiport_mutex);
5023+ list_for_each_entry(dev, &mlx5_ib_dev_list, ib_dev_list) {
5024+ if (dev->sys_image_guid == mpi->sys_image_guid)
5025+ bound = mlx5_ib_bind_slave_port(dev, mpi);
42275026
4228−err_disable_eth:
4229− if (ll == IB_LINK_LAYER_ETHERNET) {
4230− mlx5_disable_eth(dev);
4231− mlx5_remove_netdev_notifier(dev);
5027+ if (bound) {
5028+ rdma_roce_rescan_device(&dev->ib_dev);
5029+ break;
5030+ }
42325031 }
42335032
4234−err_free_port:
4235− kfree(dev->port);
5033+ if (!bound) {
5034+ list_add_tail(&mpi->list, &mlx5_ib_unaffiliated_port_list);
5035+ dev_dbg(&mdev->pdev->dev, "no suitable IB device found to bind to, added to unaffiliated list.\n");
5036+ } else {
5037+ mlx5_ib_dbg(dev, "bound port %u\n", port_num + 1);
5038+ }
5039+ mutex_unlock(&mlx5_ib_multiport_mutex);
42365040
4237−err_dealloc:
4238− ib_dealloc_device((struct ib_device *)dev);
5041+ return mpi;
5042+}
42395043
4240− return NULL;
5044+static void *mlx5_ib_add(struct mlx5_core_dev *mdev)
5045+{
5046+ enum rdma_link_layer ll;
5047+ int port_type_cap;
5048+
5049+ port_type_cap = MLX5_CAP_GEN(mdev, port_type);
5050+ ll = mlx5_port_type_cap_to_rdma_ll(port_type_cap);
5051+
5052+ if (mlx5_core_is_mp_slave(mdev) && ll == IB_LINK_LAYER_ETHERNET) {
5053+ u8 port_num = mlx5_core_native_port_num(mdev) - 1;
5054+
5055+ return mlx5_ib_add_slave_port(mdev, port_num);
5056+ }
5057+
5058+ return __mlx5_ib_add(mdev, &pf_profile);
42415059 }
42425060
42435061 static void mlx5_ib_remove(struct mlx5_core_dev *mdev, void *context)
42445062 {
4245− struct mlx5_ib_dev *dev = context;
4246− enum rdma_link_layer ll = mlx5_ib_port_link_layer(&dev->ib_dev, 1);
5063+ struct mlx5_ib_multiport_info *mpi;
5064+ struct mlx5_ib_dev *dev;
42475065
4248− cancel_delay_drop(dev);
4249− mlx5_remove_netdev_notifier(dev);
4250− ib_unregister_device(&dev->ib_dev);
4251− mlx5_free_bfreg(dev->mdev, &dev->fp_bfreg);
4252− mlx5_free_bfreg(dev->mdev, &dev->bfreg);
4253− mlx5_put_uars_page(dev->mdev, mdev->priv.uar);
4254− mlx5_ib_cleanup_cong_debugfs(dev);
4255− if (MLX5_CAP_GEN(dev->mdev, max_qp_cnt))
4256− mlx5_ib_dealloc_counters(dev);
4257− destroy_umrc_res(dev);
4258− mlx5_ib_odp_remove_one(dev);
4259− destroy_dev_resources(&dev->devr);
4260− if (ll == IB_LINK_LAYER_ETHERNET)
4261− mlx5_disable_eth(dev);
4262− kfree(dev->port);
4263− ib_dealloc_device(&dev->ib_dev);
5066+ if (mlx5_core_is_mp_slave(mdev)) {
5067+ mpi = context;
5068+ mutex_lock(&mlx5_ib_multiport_mutex);
5069+ if (mpi->ibdev)
5070+ mlx5_ib_unbind_slave_port(mpi->ibdev, mpi);
5071+ list_del(&mpi->list);
5072+ mutex_unlock(&mlx5_ib_multiport_mutex);
5073+ return;
5074+ }
5075+
5076+ dev = context;
5077+ __mlx5_ib_remove(dev, dev->profile, MLX5_IB_STAGE_MAX);
42645078 }
42655079
42665080 static struct mlx5_interface mlx5_ib_interface = {
@@ -4277,6 +5091,10 @@ static int __init mlx5_ib_init(void) · 10 unmodified lines
42775091 {
42785092 int err;
42795093
5094+ mlx5_ib_event_wq = alloc_ordered_workqueue("mlx5_ib_event_wq", 0);
5095+ if (!mlx5_ib_event_wq)
5096+ return -ENOMEM;
5097+
42805098 mlx5_ib_odp_init();
42815099
42825100 err = mlx5_register_interface(&mlx5_ib_interface);
@@ -4287,6 +5105,7 @@ static int __init mlx5_ib_init(void) · 4 unmodified lines
42875105 static void __exit mlx5_ib_cleanup(void)
42885106 {
42895107 mlx5_unregister_interface(&mlx5_ib_interface);
5108+ destroy_workqueue(mlx5_ib_event_wq);
42905109 }
42915110
42925111 module_init(mlx5_ib_init);
drivers/infiniband/hw/mlx5/mlx5_ib.h+91−20
@@ -70,15 +70,6 @@ enum { · 69 unmodified lines
7070 MLX5_IB_MMAP_CMD_MASK = 0xff,
7171 };
7272
73−enum mlx5_ib_mmap_cmd {
74− MLX5_IB_MMAP_REGULAR_PAGE = 0,
75− MLX5_IB_MMAP_GET_CONTIGUOUS_PAGES = 1,
76− MLX5_IB_MMAP_WC_PAGE = 2,
77− MLX5_IB_MMAP_NC_PAGE = 3,
78− /* 5 is chosen in order to be compatible with old versions of libmlx5 */
79− MLX5_IB_MMAP_CORE_CLOCK = 5,
80−};
81−
8273 enum {
8374 MLX5_RES_SCAT_DATA32_CQE = 0x1,
8475 MLX5_RES_SCAT_DATA64_CQE = 0x2,
@@ -112,6 +103,11 @@ enum { · 27 unmodified lines
112103 MLX5_TM_MAX_SGE = 1,
113104 };
114105
106+enum {
107+ MLX5_IB_INVALID_UAR_INDEX = BIT(31),
108+ MLX5_IB_INVALID_BFREG = BIT(31),
109+};
110+
115111 struct mlx5_ib_vma_private_data {
116112 struct list_head list;
117113 struct vm_area_struct *vma;
@@ -200,6 +196,8 @@ struct mlx5_ib_flow_db { · 82 unmodified lines
200196 * creates the actual hardware QP.
201197 */
202198 #define MLX5_IB_QPT_HW_GSI IB_QPT_RESERVED2
199+#define MLX5_IB_QPT_DCI IB_QPT_RESERVED3
200+#define MLX5_IB_QPT_DCT IB_QPT_RESERVED4
203201 #define MLX5_IB_WR_UMR IB_WR_RESERVED1
204202
205203 #define MLX5_IB_UMR_OCTOWORD 16
@@ -360,12 +358,18 @@ struct mlx5_bf { · 154 unmodified lines
360358 struct mlx5_sq_bfreg *bfreg;
361359 };
362360
361+struct mlx5_ib_dct {
362+ struct mlx5_core_dct mdct;
363+ u32 *in;
364+};
365+
363366 struct mlx5_ib_qp {
364367 struct ib_qp ibqp;
365368 union {
366369 struct mlx5_ib_qp_trans trans_qp;
367370 struct mlx5_ib_raw_packet_qp raw_packet_qp;
368371 struct mlx5_ib_rss_qp rss_qp;
372+ struct mlx5_ib_dct dct;
369373 };
370374 struct mlx5_buf buf;
371375
@@ -404,6 +408,8 @@ struct mlx5_ib_qp { · 32 unmodified lines
404408 u32 rate_limit;
405409 u32 underlay_qpn;
406410 bool tunnel_offload_en;
411+ /* storage for qp sub type when core qp type is IB_QPT_DRIVER */
412+ enum ib_qp_type qp_sub_type;
407413 };
408414
409415 struct mlx5_ib_cq_buf {
@@ -636,10 +642,21 @@ struct mlx5_ib_counters { · 226 unmodified lines
636642 u32 num_q_counters;
637643 u32 num_cong_counters;
638644 u16 set_id;
645+ bool set_id_valid;
646+};
647+
648+struct mlx5_ib_multiport_info;
649+
650+struct mlx5_ib_multiport {
651+ struct mlx5_ib_multiport_info *mpi;
652+ /* To be held when accessing the multiport info */
653+ spinlock_t mpi_lock;
639654 };
640655
641656 struct mlx5_ib_port {
642657 struct mlx5_ib_counters cnts;
658+ struct mlx5_ib_multiport mp;
659+ struct mlx5_ib_dbg_cc_params *dbg_cc_params;
643660 };
644661
645662 struct mlx5_roce {
@@ -651,12 +668,15 @@ struct mlx5_roce { · 5 unmodified lines
651668 struct notifier_block nb;
652669 atomic_t next_port;
653670 enum ib_port_state last_port_state;
671+ struct mlx5_ib_dev *dev;
672+ u8 native_port_num;
654673 };
655674
656675 struct mlx5_ib_dbg_param {
657676 int offset;
658677 struct mlx5_ib_dev *dev;
659678 struct dentry *dentry;
679+ u8 port_num;
660680 };
661681
662682 enum mlx5_ib_dbg_cc_types {
@@ -709,10 +729,50 @@ struct mlx5_ib_delay_drop { · 46 unmodified lines
709729 struct mlx5_ib_dbg_delay_drop *dbg;
710730 };
711731
732+enum mlx5_ib_stages {
733+ MLX5_IB_STAGE_INIT,
734+ MLX5_IB_STAGE_CAPS,
735+ MLX5_IB_STAGE_ROCE,
736+ MLX5_IB_STAGE_DEVICE_RESOURCES,
737+ MLX5_IB_STAGE_ODP,
738+ MLX5_IB_STAGE_COUNTERS,
739+ MLX5_IB_STAGE_CONG_DEBUGFS,
740+ MLX5_IB_STAGE_UAR,
741+ MLX5_IB_STAGE_BFREG,
742+ MLX5_IB_STAGE_IB_REG,
743+ MLX5_IB_STAGE_UMR_RESOURCES,
744+ MLX5_IB_STAGE_DELAY_DROP,
745+ MLX5_IB_STAGE_CLASS_ATTR,
746+ MLX5_IB_STAGE_MAX,
747+};
748+
749+struct mlx5_ib_stage {
750+ int (*init)(struct mlx5_ib_dev *dev);
751+ void (*cleanup)(struct mlx5_ib_dev *dev);
752+};
753+
754+#define STAGE_CREATE(_stage, _init, _cleanup) \
755+ .stage[_stage] = {.init = _init, .cleanup = _cleanup}
756+
757+struct mlx5_ib_profile {
758+ struct mlx5_ib_stage stage[MLX5_IB_STAGE_MAX];
759+};
760+
761+struct mlx5_ib_multiport_info {
762+ struct list_head list;
763+ struct mlx5_ib_dev *ibdev;
764+ struct mlx5_core_dev *mdev;
765+ struct completion unref_comp;
766+ u64 sys_image_guid;
767+ u32 mdev_refcnt;
768+ bool is_master;
769+ bool unaffiliate;
770+};
771+
712772 struct mlx5_ib_dev {
713773 struct ib_device ib_dev;
714774 struct mlx5_core_dev *mdev;
715− struct mlx5_roce roce;
775+ struct mlx5_roce roce[MLX5_MAX_PORTS];
716776 int num_ports;
717777 /* serialize update of capability mask
718778 */
@@ -746,12 +806,14 @@ struct mlx5_ib_dev { · 27 unmodified lines
746806 struct mlx5_sq_bfreg bfreg;
747807 struct mlx5_sq_bfreg fp_bfreg;
748808 struct mlx5_ib_delay_drop delay_drop;
749− struct mlx5_ib_dbg_cc_params *dbg_cc_params;
809+ const struct mlx5_ib_profile *profile;
750810
751811 /* protect the user_td */
752812 struct mutex lb_mutex;
753813 u32 user_td;
754814 u8 umr_fence;
815+ struct list_head ib_dev_list;
816+ u64 sys_image_guid;
755817 };
756818
757819 static inline struct mlx5_ib_cq *to_mibcq(struct mlx5_core_cq *mcq)
@@ -956,13 +1018,14 @@ struct ib_rwq_ind_table *mlx5_ib_create_rwq_ind_table(struct ib_device *device, · 198 unmodified lines
9561018 struct ib_rwq_ind_table_init_attr *init_attr,
9571019 struct ib_udata *udata);
9581020 int mlx5_ib_destroy_rwq_ind_table(struct ib_rwq_ind_table *wq_ind_table);
1021+bool mlx5_ib_dc_atomic_is_supported(struct mlx5_ib_dev *dev);
1022+
9591023
9601024 #ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
9611025 void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev);
9621026 void mlx5_ib_pfault(struct mlx5_core_dev *mdev, void *context,
9631027 struct mlx5_pagefault *pfault);
9641028 int mlx5_ib_odp_init_one(struct mlx5_ib_dev *ibdev);
965−void mlx5_ib_odp_remove_one(struct mlx5_ib_dev *ibdev);
9661029 int __init mlx5_ib_odp_init(void);
9671030 void mlx5_ib_odp_cleanup(void);
9681031 void mlx5_ib_invalidate_range(struct ib_umem *umem, unsigned long start,
@@ -977,7 +1040,6 @@ static inline void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev) · 8 unmodified lines
9771040 }
9781041
9791042 static inline int mlx5_ib_odp_init_one(struct mlx5_ib_dev *ibdev) { return 0; }
980−static inline void mlx5_ib_odp_remove_one(struct mlx5_ib_dev *ibdev) {}
9811043 static inline int mlx5_ib_odp_init(void) { return 0; }
9821044 static inline void mlx5_ib_odp_cleanup(void) {}
9831045 static inline void mlx5_odp_init_mr_cache_entry(struct mlx5_cache_ent *ent) {}
@@ -1001,8 +1063,8 @@ __be16 mlx5_get_roce_udp_sport(struct mlx5_ib_dev *dev, u8 port_num, · 17 unmodified lines
10011063 int mlx5_get_roce_gid_type(struct mlx5_ib_dev *dev, u8 port_num,
10021064 int index, enum ib_gid_type *gid_type);
10031065
1004−void mlx5_ib_cleanup_cong_debugfs(struct mlx5_ib_dev *dev);
1005−int mlx5_ib_init_cong_debugfs(struct mlx5_ib_dev *dev);
1066+void mlx5_ib_cleanup_cong_debugfs(struct mlx5_ib_dev *dev, u8 port_num);
1067+int mlx5_ib_init_cong_debugfs(struct mlx5_ib_dev *dev, u8 port_num);
10061068
10071069 /* GSI QP helper functions */
10081070 struct ib_qp *mlx5_ib_gsi_create_qp(struct ib_pd *pd,
@@ -1021,6 +1083,15 @@ void mlx5_ib_gsi_pkey_change(struct mlx5_ib_gsi_qp *gsi); · 12 unmodified lines
10211083
10221084 int mlx5_ib_generate_wc(struct ib_cq *ibcq, struct ib_wc *wc);
10231085
1086+void mlx5_ib_free_bfreg(struct mlx5_ib_dev *dev, struct mlx5_bfreg_info *bfregi,
1087+ int bfregn);
1088+struct mlx5_ib_dev *mlx5_ib_get_ibdev_from_mpi(struct mlx5_ib_multiport_info *mpi);
1089+struct mlx5_core_dev *mlx5_ib_get_native_port_mdev(struct mlx5_ib_dev *dev,
1090+ u8 ib_port_num,
1091+ u8 *native_port_num);
1092+void mlx5_ib_put_native_port_mdev(struct mlx5_ib_dev *dev,
1093+ u8 port_num);
1094+
10241095 static inline void init_query_mad(struct ib_smp *mad)
10251096 {
10261097 mad->base_version = 1;
@@ -1052,8 +1123,8 @@ static inline u32 check_cq_create_flags(u32 flags) · 25 unmodified lines
10521123 * It returns non-zero value for unsupported CQ
10531124 * create flags, otherwise it returns zero.
10541125 */
1055− return (flags & ~(IB_CQ_FLAGS_IGNORE_OVERRUN |
1056− IB_CQ_FLAGS_TIMESTAMP_COMPLETION));
1126+ return (flags & ~(IB_UVERBS_CQ_FLAGS_IGNORE_OVERRUN |
1127+ IB_UVERBS_CQ_FLAGS_TIMESTAMP_COMPLETION));
10571128 }
10581129
10591130 static inline int verify_assign_uidx(u8 cqe_version, u32 cmd_uidx,
@@ -1113,10 +1184,10 @@ static inline int get_uars_per_sys_page(struct mlx5_ib_dev *dev, bool lib_suppor · 53 unmodified lines
11131184 MLX5_UARS_IN_PAGE : 1;
11141185 }
11151186
1116−static inline int get_num_uars(struct mlx5_ib_dev *dev,
1117− struct mlx5_bfreg_info *bfregi)
1187+static inline int get_num_static_uars(struct mlx5_ib_dev *dev,
1188+ struct mlx5_bfreg_info *bfregi)
11181189 {
1119− return get_uars_per_sys_page(dev, bfregi->lib_uar_4k) * bfregi->num_sys_pages;
1190+ return get_uars_per_sys_page(dev, bfregi->lib_uar_4k) * bfregi->num_static_sys_pages;
11201191 }
11211192
11221193 #endif /* MLX5_IB_H */
drivers/infiniband/hw/mlx5/mr.c+3−0
@@ -1206,6 +1206,9 @@ struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, · 1205 unmodified lines
12061206 int err;
12071207 bool use_umr = true;
12081208
1209+ if (!IS_ENABLED(CONFIG_INFINIBAND_USER_MEM))
1210+ return ERR_PTR(-EINVAL);
1211+
12091212 mlx5_ib_dbg(dev, "start 0x%llx, virt_addr 0x%llx, length 0x%llx, access_flags 0x%x\n",
12101213 start, virt_addr, length, access_flags);
12111214
drivers/infiniband/hw/mlx5/odp.c+0−9
@@ -1207,10 +1207,6 @@ int mlx5_ib_odp_init_one(struct mlx5_ib_dev *dev) · 1206 unmodified lines
12071207 {
12081208 int ret;
12091209
1210− ret = init_srcu_struct(&dev->mr_srcu);
1211− if (ret)
1212− return ret;
1213−
12141210 if (dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT) {
12151211 ret = mlx5_cmd_null_mkey(dev->mdev, &dev->null_mkey);
12161212 if (ret) {
@@ -1222,11 +1218,6 @@ int mlx5_ib_odp_init_one(struct mlx5_ib_dev *dev) · 5 unmodified lines
12221218 return 0;
12231219 }
12241220
1225−void mlx5_ib_odp_remove_one(struct mlx5_ib_dev *dev)
1226−{
1227− cleanup_srcu_struct(&dev->mr_srcu);
1228−}
1229−
12301221 int mlx5_ib_odp_init(void)
12311222 {
12321223 mlx5_imr_ksm_entries = BIT_ULL(get_order(TASK_SIZE) -
drivers/infiniband/hw/mlx5/qp.c+405−27
@@ -493,7 +493,7 @@ enum { · 492 unmodified lines
493493
494494 static int max_bfregs(struct mlx5_ib_dev *dev, struct mlx5_bfreg_info *bfregi)
495495 {
496− return get_num_uars(dev, bfregi) * MLX5_NON_FP_BFREGS_PER_UAR;
496+ return get_num_static_uars(dev, bfregi) * MLX5_NON_FP_BFREGS_PER_UAR;
497497 }
498498
499499 static int num_med_bfreg(struct mlx5_ib_dev *dev,
@@ -581,7 +581,7 @@ static int alloc_bfreg(struct mlx5_ib_dev *dev, · 81 unmodified lines
581581 return bfregn;
582582 }
583583
584−static void free_bfreg(struct mlx5_ib_dev *dev, struct mlx5_bfreg_info *bfregi, int bfregn)
584+void mlx5_ib_free_bfreg(struct mlx5_ib_dev *dev, struct mlx5_bfreg_info *bfregi, int bfregn)
585585 {
586586 mutex_lock(&bfregi->lock);
587587 bfregi->count[bfregn]--;
@@ -613,6 +613,7 @@ static int to_mlx5_st(enum ib_qp_type type) · 25 unmodified lines
613613 case IB_QPT_XRC_TGT: return MLX5_QP_ST_XRC;
614614 case IB_QPT_SMI: return MLX5_QP_ST_QP0;
615615 case MLX5_IB_QPT_HW_GSI: return MLX5_QP_ST_QP1;
616+ case MLX5_IB_QPT_DCI: return MLX5_QP_ST_DCI;
616617 case IB_QPT_RAW_IPV6: return MLX5_QP_ST_RAW_IPV6;
617618 case IB_QPT_RAW_PACKET:
618619 case IB_QPT_RAW_ETHERTYPE: return MLX5_QP_ST_RAW_ETHERTYPE;
@@ -627,7 +628,8 @@ static void mlx5_ib_unlock_cqs(struct mlx5_ib_cq *send_cq, · 8 unmodified lines
627628 struct mlx5_ib_cq *recv_cq);
628629
629630 static int bfregn_to_uar_index(struct mlx5_ib_dev *dev,
630− struct mlx5_bfreg_info *bfregi, int bfregn)
631+ struct mlx5_bfreg_info *bfregi, int bfregn,
632+ bool dyn_bfreg)
631633 {
632634 int bfregs_per_sys_page;
633635 int index_of_sys_page;
@@ -637,8 +639,16 @@ static int bfregn_to_uar_index(struct mlx5_ib_dev *dev, · 3 unmodified lines
637639 MLX5_NON_FP_BFREGS_PER_UAR;
638640 index_of_sys_page = bfregn / bfregs_per_sys_page;
639641
640− offset = bfregn % bfregs_per_sys_page / MLX5_NON_FP_BFREGS_PER_UAR;
642+ if (dyn_bfreg) {
643+ index_of_sys_page += bfregi->num_static_sys_pages;
644+ if (bfregn > bfregi->num_dyn_bfregs ||
645+ bfregi->sys_pages[index_of_sys_page] == MLX5_IB_INVALID_UAR_INDEX) {
646+ mlx5_ib_dbg(dev, "Invalid dynamic uar index\n");
647+ return -EINVAL;
648+ }
649+ }
641650
651+ offset = bfregn % bfregs_per_sys_page / MLX5_NON_FP_BFREGS_PER_UAR;
642652 return bfregi->sys_pages[index_of_sys_page] + offset;
643653 }
644654
@@ -764,7 +774,7 @@ static int create_user_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd, · 119 unmodified lines
764774 struct mlx5_ib_create_qp ucmd;
765775 struct mlx5_ib_ubuffer *ubuffer = &base->ubuffer;
766776 int page_shift = 0;
767− int uar_index;
777+ int uar_index = 0;
768778 int npages;
769779 u32 offset = 0;
770780 int bfregn;
@@ -780,12 +790,20 @@ static int create_user_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd, · 9 unmodified lines
780790 }
781791
782792 context = to_mucontext(pd->uobject->context);
783− /*
784− * TBD: should come from the verbs when we have the API
785− */
786− if (qp->flags & MLX5_IB_QP_CROSS_CHANNEL)
793+ if (ucmd.flags & MLX5_QP_FLAG_BFREG_INDEX) {
794+ uar_index = bfregn_to_uar_index(dev, &context->bfregi,
795+ ucmd.bfreg_index, true);
796+ if (uar_index < 0)
797+ return uar_index;
798+
799+ bfregn = MLX5_IB_INVALID_BFREG;
800+ } else if (qp->flags & MLX5_IB_QP_CROSS_CHANNEL) {
801+ /*
802+ * TBD: should come from the verbs when we have the API
803+ */
787804 /* In CROSS_CHANNEL CQ and QP must use the same UAR */
788805 bfregn = MLX5_CROSS_CHANNEL_BFREG;
806+ }
789807 else {
790808 bfregn = alloc_bfreg(dev, &context->bfregi, MLX5_IB_LATENCY_CLASS_HIGH);
791809 if (bfregn < 0) {
@@ -804,8 +822,10 @@ static int create_user_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd, · 12 unmodified lines
804822 }
805823 }
806824
807− uar_index = bfregn_to_uar_index(dev, &context->bfregi, bfregn);
808825 mlx5_ib_dbg(dev, "bfregn 0x%x, uar_index 0x%x\n", bfregn, uar_index);
826+ if (bfregn != MLX5_IB_INVALID_BFREG)
827+ uar_index = bfregn_to_uar_index(dev, &context->bfregi, bfregn,
828+ false);
809829
810830 qp->rq.offset = 0;
811831 qp->sq.wqe_shift = ilog2(MLX5_SEND_WQE_BB);
@@ -845,7 +865,10 @@ static int create_user_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd, · 33 unmodified lines
845865 MLX5_SET(qpc, qpc, page_offset, offset);
846866
847867 MLX5_SET(qpc, qpc, uar_page, uar_index);
848− resp->bfreg_index = adjust_bfregn(dev, &context->bfregi, bfregn);
868+ if (bfregn != MLX5_IB_INVALID_BFREG)
869+ resp->bfreg_index = adjust_bfregn(dev, &context->bfregi, bfregn);
870+ else
871+ resp->bfreg_index = MLX5_IB_INVALID_BFREG;
849872 qp->bfregn = bfregn;
850873
851874 err = mlx5_ib_db_map_user(context, ucmd.db_addr, &qp->db);
@@ -874,7 +897,8 @@ err_umem: · 22 unmodified lines
874897 ib_umem_release(ubuffer->umem);
875898
876899 err_bfreg:
877− free_bfreg(dev, &context->bfregi, bfregn);
900+ if (bfregn != MLX5_IB_INVALID_BFREG)
901+ mlx5_ib_free_bfreg(dev, &context->bfregi, bfregn);
878902 return err;
879903 }
880904
@@ -887,7 +911,13 @@ static void destroy_qp_user(struct mlx5_ib_dev *dev, struct ib_pd *pd, · 6 unmodified lines
887911 mlx5_ib_db_unmap_user(context, &qp->db);
888912 if (base->ubuffer.umem)
889913 ib_umem_release(base->ubuffer.umem);
890− free_bfreg(dev, &context->bfregi, qp->bfregn);
914+
915+ /*
916+ * Free only the BFREGs which are handled by the kernel.
917+ * BFREGs of UARs allocated dynamically are handled by user.
918+ */
919+ if (qp->bfregn != MLX5_IB_INVALID_BFREG)
920+ mlx5_ib_free_bfreg(dev, &context->bfregi, qp->bfregn);
891921 }
892922
893923 static int create_kernel_qp(struct mlx5_ib_dev *dev,
@@ -1015,6 +1045,7 @@ static void destroy_qp_kernel(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp) · 121 unmodified lines
10151045 static u32 get_rx_type(struct mlx5_ib_qp *qp, struct ib_qp_init_attr *attr)
10161046 {
10171047 if (attr->srq || (attr->qp_type == IB_QPT_XRC_TGT) ||
1048+ (attr->qp_type == MLX5_IB_QPT_DCI) ||
10181049 (attr->qp_type == IB_QPT_XRC_INI))
10191050 return MLX5_SRQ_RQ;
10201051 else if (!qp->has_rq)
@@ -2086,20 +2117,108 @@ static const char *ib_qp_type_str(enum ib_qp_type type) · 1065 unmodified lines
20862117 return "IB_QPT_RAW_PACKET";
20872118 case MLX5_IB_QPT_REG_UMR:
20882119 return "MLX5_IB_QPT_REG_UMR";
2120+ case IB_QPT_DRIVER:
2121+ return "IB_QPT_DRIVER";
20892122 case IB_QPT_MAX:
20902123 default:
20912124 return "Invalid QP type";
20922125 }
20932126 }
20942127
2128+static struct ib_qp *mlx5_ib_create_dct(struct ib_pd *pd,
2129+ struct ib_qp_init_attr *attr,
2130+ struct mlx5_ib_create_qp *ucmd)
2131+{
2132+ struct mlx5_ib_dev *dev;
2133+ struct mlx5_ib_qp *qp;
2134+ int err = 0;
2135+ u32 uidx = MLX5_IB_DEFAULT_UIDX;
2136+ void *dctc;
2137+
2138+ if (!attr->srq || !attr->recv_cq)
2139+ return ERR_PTR(-EINVAL);
2140+
2141+ dev = to_mdev(pd->device);
2142+
2143+ err = get_qp_user_index(to_mucontext(pd->uobject->context),
2144+ ucmd, sizeof(*ucmd), &uidx);
2145+ if (err)
2146+ return ERR_PTR(err);
2147+
2148+ qp = kzalloc(sizeof(*qp), GFP_KERNEL);
2149+ if (!qp)
2150+ return ERR_PTR(-ENOMEM);
2151+
2152+ qp->dct.in = kzalloc(MLX5_ST_SZ_BYTES(create_dct_in), GFP_KERNEL);
2153+ if (!qp->dct.in) {
2154+ err = -ENOMEM;
2155+ goto err_free;
2156+ }
2157+
2158+ dctc = MLX5_ADDR_OF(create_dct_in, qp->dct.in, dct_context_entry);
2159+ qp->qp_sub_type = MLX5_IB_QPT_DCT;
2160+ MLX5_SET(dctc, dctc, pd, to_mpd(pd)->pdn);
2161+ MLX5_SET(dctc, dctc, srqn_xrqn, to_msrq(attr->srq)->msrq.srqn);
2162+ MLX5_SET(dctc, dctc, cqn, to_mcq(attr->recv_cq)->mcq.cqn);
2163+ MLX5_SET64(dctc, dctc, dc_access_key, ucmd->access_key);
2164+ MLX5_SET(dctc, dctc, user_index, uidx);
2165+
2166+ qp->state = IB_QPS_RESET;
2167+
2168+ return &qp->ibqp;
2169+err_free:
2170+ kfree(qp);
2171+ return ERR_PTR(err);
2172+}
2173+
2174+static int set_mlx_qp_type(struct mlx5_ib_dev *dev,
2175+ struct ib_qp_init_attr *init_attr,
2176+ struct mlx5_ib_create_qp *ucmd,
2177+ struct ib_udata *udata)
2178+{
2179+ enum { MLX_QP_FLAGS = MLX5_QP_FLAG_TYPE_DCT | MLX5_QP_FLAG_TYPE_DCI };
2180+ int err;
2181+
2182+ if (!udata)
2183+ return -EINVAL;
2184+
2185+ if (udata->inlen < sizeof(*ucmd)) {
2186+ mlx5_ib_dbg(dev, "create_qp user command is smaller than expected\n");
2187+ return -EINVAL;
2188+ }
2189+ err = ib_copy_from_udata(ucmd, udata, sizeof(*ucmd));
2190+ if (err)
2191+ return err;
2192+
2193+ if ((ucmd->flags & MLX_QP_FLAGS) == MLX5_QP_FLAG_TYPE_DCI) {
2194+ init_attr->qp_type = MLX5_IB_QPT_DCI;
2195+ } else {
2196+ if ((ucmd->flags & MLX_QP_FLAGS) == MLX5_QP_FLAG_TYPE_DCT) {
2197+ init_attr->qp_type = MLX5_IB_QPT_DCT;
2198+ } else {
2199+ mlx5_ib_dbg(dev, "Invalid QP flags\n");
2200+ return -EINVAL;
2201+ }
2202+ }
2203+
2204+ if (!MLX5_CAP_GEN(dev->mdev, dct)) {
2205+ mlx5_ib_dbg(dev, "DC transport is not supported\n");
2206+ return -EOPNOTSUPP;
2207+ }
2208+
2209+ return 0;
2210+}
2211+
20952212 struct ib_qp *mlx5_ib_create_qp(struct ib_pd *pd,
2096− struct ib_qp_init_attr *init_attr,
2213+ struct ib_qp_init_attr *verbs_init_attr,
20972214 struct ib_udata *udata)
20982215 {
20992216 struct mlx5_ib_dev *dev;
21002217 struct mlx5_ib_qp *qp;
21012218 u16 xrcdn = 0;
21022219 int err;
2220+ struct ib_qp_init_attr mlx_init_attr;
2221+ struct ib_qp_init_attr *init_attr = verbs_init_attr;
21032222
21042223 if (pd) {
21052224 dev = to_mdev(pd->device);
@@ -2124,6 +2243,26 @@ struct ib_qp *mlx5_ib_create_qp(struct ib_pd *pd, · 18 unmodified lines
21242243 dev = to_mdev(to_mxrcd(init_attr->xrcd)->ibxrcd.device);
21252244 }
21262245
2246+ if (init_attr->qp_type == IB_QPT_DRIVER) {
2247+ struct mlx5_ib_create_qp ucmd;
2248+
2249+ init_attr = &mlx_init_attr;
2250+ memcpy(init_attr, verbs_init_attr, sizeof(*verbs_init_attr));
2251+ err = set_mlx_qp_type(dev, init_attr, &ucmd, udata);
2252+ if (err)
2253+ return ERR_PTR(err);
2254+
2255+ if (init_attr->qp_type == MLX5_IB_QPT_DCI) {
2256+ if (init_attr->cap.max_recv_wr ||
2257+ init_attr->cap.max_recv_sge) {
2258+ mlx5_ib_dbg(dev, "DCI QP requires zero size receive queue\n");
2259+ return ERR_PTR(-EINVAL);
2260+ }
2261+ } else {
2262+ return mlx5_ib_create_dct(pd, init_attr, &ucmd);
2263+ }
2264+ }
2265+
21272266 switch (init_attr->qp_type) {
21282267 case IB_QPT_XRC_TGT:
21292268 case IB_QPT_XRC_INI:
@@ -2145,6 +2284,7 @@ struct ib_qp *mlx5_ib_create_qp(struct ib_pd *pd, · 15 unmodified lines
21452284 case IB_QPT_SMI:
21462285 case MLX5_IB_QPT_HW_GSI:
21472286 case MLX5_IB_QPT_REG_UMR:
2287+ case MLX5_IB_QPT_DCI:
21482288 qp = kzalloc(sizeof(*qp), GFP_KERNEL);
21492289 if (!qp)
21502290 return ERR_PTR(-ENOMEM);
@@ -2185,9 +2325,31 @@ struct ib_qp *mlx5_ib_create_qp(struct ib_pd *pd, · 34 unmodified lines
21852325 return ERR_PTR(-EINVAL);
21862326 }
21872327
2328+ if (verbs_init_attr->qp_type == IB_QPT_DRIVER)
2329+ qp->qp_sub_type = init_attr->qp_type;
2330+
21882331 return &qp->ibqp;
21892332 }
21902333
2334+static int mlx5_ib_destroy_dct(struct mlx5_ib_qp *mqp)
2335+{
2336+ struct mlx5_ib_dev *dev = to_mdev(mqp->ibqp.device);
2337+
2338+ if (mqp->state == IB_QPS_RTR) {
2339+ int err;
2340+
2341+ err = mlx5_core_destroy_dct(dev->mdev, &mqp->dct.mdct);
2342+ if (err) {
2343+ mlx5_ib_warn(dev, "failed to destroy DCT %d\n", err);
2344+ return err;
2345+ }
2346+ }
2347+
2348+ kfree(mqp->dct.in);
2349+ kfree(mqp);
2350+ return 0;
2351+}
2352+
21912353 int mlx5_ib_destroy_qp(struct ib_qp *qp)
21922354 {
21932355 struct mlx5_ib_dev *dev = to_mdev(qp->device);
@@ -2196,6 +2358,9 @@ int mlx5_ib_destroy_qp(struct ib_qp *qp) · 2 unmodified lines
21962358 if (unlikely(qp->qp_type == IB_QPT_GSI))
21972359 return mlx5_ib_gsi_destroy_qp(qp);
21982360
2361+ if (mqp->qp_sub_type == MLX5_IB_QPT_DCT)
2362+ return mlx5_ib_destroy_dct(mqp);
2363+
21992364 destroy_qp_common(dev, mqp);
22002365
22012366 kfree(mqp);
@@ -2763,7 +2928,8 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, · 561 unmodified lines
27632928 if (!context)
27642929 return -ENOMEM;
27652930
2766− err = to_mlx5_st(ibqp->qp_type);
2931+ err = to_mlx5_st(ibqp->qp_type == IB_QPT_DRIVER ?
2932+ qp->qp_sub_type : ibqp->qp_type);
27672933 if (err < 0) {
27682934 mlx5_ib_dbg(dev, "unsupported qp type %d\n", ibqp->qp_type);
27692935 goto out;
@@ -2796,8 +2962,9 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, · 26 unmodified lines
27962962 (ibqp->qp_type == IB_QPT_XRC_INI) ||
27972963 (ibqp->qp_type == IB_QPT_XRC_TGT)) {
27982964 if (mlx5_lag_is_active(dev->mdev)) {
2965+ u8 p = mlx5_core_native_port_num(dev->mdev);
27992966 tx_affinity = (unsigned int)atomic_add_return(1,
2800− &dev->roce.next_port) %
2967+ &dev->roce[p].next_port) %
28012968 MLX5_MAX_PORTS + 1;
28022969 context->flags |= cpu_to_be32(tx_affinity << 24);
28032970 }
@@ -2922,7 +3089,8 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, · 118 unmodified lines
29223089
29233090 mlx5_cur = to_mlx5_state(cur_state);
29243091 mlx5_new = to_mlx5_state(new_state);
2925− mlx5_st = to_mlx5_st(ibqp->qp_type);
3092+ mlx5_st = to_mlx5_st(ibqp->qp_type == IB_QPT_DRIVER ?
3093+ qp->qp_sub_type : ibqp->qp_type);
29263094 if (mlx5_st < 0)
29273095 goto out;
29283096
@@ -2994,6 +3162,139 @@ out: · 65 unmodified lines
29943162 return err;
29953163 }
29963164
3165+static inline bool is_valid_mask(int mask, int req, int opt)
3166+{
3167+ if ((mask & req) != req)
3168+ return false;
3169+
3170+ if (mask & ~(req | opt))
3171+ return false;
3172+
3173+ return true;
3174+}
3175+
3176+/* check valid transition for driver QP types
3177+ * for now the only QP type that this function supports is DCI
3178+ */
3179+static bool modify_dci_qp_is_ok(enum ib_qp_state cur_state, enum ib_qp_state new_state,
3180+ enum ib_qp_attr_mask attr_mask)
3181+{
3182+ int req = IB_QP_STATE;
3183+ int opt = 0;
3184+
3185+ if (cur_state == IB_QPS_RESET && new_state == IB_QPS_INIT) {
3186+ req |= IB_QP_PKEY_INDEX | IB_QP_PORT;
3187+ return is_valid_mask(attr_mask, req, opt);
3188+ } else if (cur_state == IB_QPS_INIT && new_state == IB_QPS_INIT) {
3189+ opt = IB_QP_PKEY_INDEX | IB_QP_PORT;
3190+ return is_valid_mask(attr_mask, req, opt);
3191+ } else if (cur_state == IB_QPS_INIT && new_state == IB_QPS_RTR) {
3192+ req |= IB_QP_PATH_MTU;
3193+ opt = IB_QP_PKEY_INDEX;
3194+ return is_valid_mask(attr_mask, req, opt);
3195+ } else if (cur_state == IB_QPS_RTR && new_state == IB_QPS_RTS) {
3196+ req |= IB_QP_TIMEOUT | IB_QP_RETRY_CNT | IB_QP_RNR_RETRY |
3197+ IB_QP_MAX_QP_RD_ATOMIC | IB_QP_SQ_PSN;
3198+ opt = IB_QP_MIN_RNR_TIMER;
3199+ return is_valid_mask(attr_mask, req, opt);
3200+ } else if (cur_state == IB_QPS_RTS && new_state == IB_QPS_RTS) {
3201+ opt = IB_QP_MIN_RNR_TIMER;
3202+ return is_valid_mask(attr_mask, req, opt);
3203+ } else if (cur_state != IB_QPS_RESET && new_state == IB_QPS_ERR) {
3204+ return is_valid_mask(attr_mask, req, opt);
3205+ }
3206+ return false;
3207+}
3208+
3209+/* mlx5_ib_modify_dct: modify a DCT QP
3210+ * valid transitions are:
3211+ * RESET to INIT: must set access_flags, pkey_index and port
3212+ * INIT to RTR : must set min_rnr_timer, tclass, flow_label,
3213+ * mtu, gid_index and hop_limit
3214+ * Other transitions and attributes are illegal
3215+ */
3216+static int mlx5_ib_modify_dct(struct ib_qp *ibqp, struct ib_qp_attr *attr,
3217+ int attr_mask, struct ib_udata *udata)
3218+{
3219+ struct mlx5_ib_qp *qp = to_mqp(ibqp);
3220+ struct mlx5_ib_dev *dev = to_mdev(ibqp->device);
3221+ enum ib_qp_state cur_state, new_state;
3222+ int err = 0;
3223+ int required = IB_QP_STATE;
3224+ void *dctc;
3225+
3226+ if (!(attr_mask & IB_QP_STATE))
3227+ return -EINVAL;
3228+
3229+ cur_state = qp->state;
3230+ new_state = attr->qp_state;
3231+
3232+ dctc = MLX5_ADDR_OF(create_dct_in, qp->dct.in, dct_context_entry);
3233+ if (cur_state == IB_QPS_RESET && new_state == IB_QPS_INIT) {
3234+ required |= IB_QP_ACCESS_FLAGS | IB_QP_PKEY_INDEX | IB_QP_PORT;
3235+ if (!is_valid_mask(attr_mask, required, 0))
3236+ return -EINVAL;
3237+
3238+ if (attr->port_num == 0 ||
3239+ attr->port_num > MLX5_CAP_GEN(dev->mdev, num_ports)) {
3240+ mlx5_ib_dbg(dev, "invalid port number %d. number of ports is %d\n",
3241+ attr->port_num, dev->num_ports);
3242+ return -EINVAL;
3243+ }
3244+ if (attr->qp_access_flags & IB_ACCESS_REMOTE_READ)
3245+ MLX5_SET(dctc, dctc, rre, 1);
3246+ if (attr->qp_access_flags & IB_ACCESS_REMOTE_WRITE)
3247+ MLX5_SET(dctc, dctc, rwe, 1);
3248+ if (attr->qp_access_flags & IB_ACCESS_REMOTE_ATOMIC) {
3249+ if (!mlx5_ib_dc_atomic_is_supported(dev))
3250+ return -EOPNOTSUPP;
3251+ MLX5_SET(dctc, dctc, rae, 1);
3252+ MLX5_SET(dctc, dctc, atomic_mode, MLX5_ATOMIC_MODE_DCT_CX);
3253+ }
3254+ MLX5_SET(dctc, dctc, pkey_index, attr->pkey_index);
3255+ MLX5_SET(dctc, dctc, port, attr->port_num);
3256+ MLX5_SET(dctc, dctc, counter_set_id, dev->port[attr->port_num - 1].cnts.set_id);
3257+
3258+ } else if (cur_state == IB_QPS_INIT && new_state == IB_QPS_RTR) {
3259+ struct mlx5_ib_modify_qp_resp resp = {};
3260+ u32 min_resp_len = offsetof(typeof(resp), dctn) +
3261+ sizeof(resp.dctn);
3262+
3263+ if (udata->outlen < min_resp_len)
3264+ return -EINVAL;
3265+ resp.response_length = min_resp_len;
3266+
3267+ required |= IB_QP_MIN_RNR_TIMER | IB_QP_AV | IB_QP_PATH_MTU;
3268+ if (!is_valid_mask(attr_mask, required, 0))
3269+ return -EINVAL;
3270+ MLX5_SET(dctc, dctc, min_rnr_nak, attr->min_rnr_timer);
3271+ MLX5_SET(dctc, dctc, tclass, attr->ah_attr.grh.traffic_class);
3272+ MLX5_SET(dctc, dctc, flow_label, attr->ah_attr.grh.flow_label);
3273+ MLX5_SET(dctc, dctc, mtu, attr->path_mtu);
3274+ MLX5_SET(dctc, dctc, my_addr_index, attr->ah_attr.grh.sgid_index);
3275+ MLX5_SET(dctc, dctc, hop_limit, attr->ah_attr.grh.hop_limit);
3276+
3277+ err = mlx5_core_create_dct(dev->mdev, &qp->dct.mdct, qp->dct.in,
3278+ MLX5_ST_SZ_BYTES(create_dct_in));
3279+ if (err)
3280+ return err;
3281+ resp.dctn = qp->dct.mdct.mqp.qpn;
3282+ err = ib_copy_to_udata(udata, &resp, resp.response_length);
3283+ if (err) {
3284+ mlx5_core_destroy_dct(dev->mdev, &qp->dct.mdct);
3285+ return err;
3286+ }
3287+ } else {
3288+ mlx5_ib_warn(dev, "Modify DCT: Invalid transition from %d to %d\n", cur_state, new_state);
3289+ return -EINVAL;
3290+ }
3291+ if (err)
3292+ qp->state = IB_QPS_ERR;
3293+ else
3294+ qp->state = new_state;
3295+ return err;
3296+}
3297+
29973298 int mlx5_ib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
29983299 int attr_mask, struct ib_udata *udata)
29993300 {
@@ -3011,8 +3312,14 @@ int mlx5_ib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 11 unmodified lines
30113312 if (unlikely(ibqp->qp_type == IB_QPT_GSI))
30123313 return mlx5_ib_gsi_modify_qp(ibqp, attr, attr_mask);
30133314
3014− qp_type = (unlikely(ibqp->qp_type == MLX5_IB_QPT_HW_GSI)) ?
3015− IB_QPT_GSI : ibqp->qp_type;
3315+ if (ibqp->qp_type == IB_QPT_DRIVER)
3316+ qp_type = qp->qp_sub_type;
3317+ else
3318+ qp_type = (unlikely(ibqp->qp_type == MLX5_IB_QPT_HW_GSI)) ?
3319+ IB_QPT_GSI : ibqp->qp_type;
3320+
3321+ if (qp_type == MLX5_IB_QPT_DCT)
3322+ return mlx5_ib_modify_dct(ibqp, attr, attr_mask, udata);
30163323
30173324 mutex_lock(&qp->mutex);
30183325
@@ -3031,15 +3338,21 @@ int mlx5_ib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 12 unmodified lines
30313338 goto out;
30323339 }
30333340 } else if (qp_type != MLX5_IB_QPT_REG_UMR &&
3034− !ib_modify_qp_is_ok(cur_state, new_state, qp_type, attr_mask, ll)) {
3341+ qp_type != MLX5_IB_QPT_DCI &&
3342+ !ib_modify_qp_is_ok(cur_state, new_state, qp_type, attr_mask, ll)) {
30353343 mlx5_ib_dbg(dev, "invalid QP state transition from %d to %d, qp_type %d, attr_mask 0x%x\n",
30363344 cur_state, new_state, ibqp->qp_type, attr_mask);
30373345 goto out;
3346+ } else if (qp_type == MLX5_IB_QPT_DCI &&
3347+ !modify_dci_qp_is_ok(cur_state, new_state, attr_mask)) {
3348+ mlx5_ib_dbg(dev, "invalid QP state transition from %d to %d, qp_type %d, attr_mask 0x%x\n",
3349+ cur_state, new_state, qp_type, attr_mask);
3350+ goto out;
30383351 }
30393352
30403353 if ((attr_mask & IB_QP_PORT) &&
30413354 (attr->port_num == 0 ||
3042− attr->port_num > MLX5_CAP_GEN(dev->mdev, num_ports))) {
3355+ attr->port_num > dev->num_ports)) {
30433356 mlx5_ib_dbg(dev, "invalid port number %d. number of ports is %d\n",
30443357 attr->port_num, dev->num_ports);
30453358 goto out;
@@ -4358,11 +4671,10 @@ static void to_rdma_ah_attr(struct mlx5_ib_dev *ibdev, · 1312 unmodified lines
43584671 struct rdma_ah_attr *ah_attr,
43594672 struct mlx5_qp_path *path)
43604673 {
4361− struct mlx5_core_dev *dev = ibdev->mdev;
43624674
43634675 memset(ah_attr, 0, sizeof(*ah_attr));
43644676
4365− if (!path->port || path->port > MLX5_CAP_GEN(dev, num_ports))
4677+ if (!path->port || path->port > ibdev->num_ports)
43664678 return;
43674679
43684680 ah_attr->type = rdma_ah_find_type(&ibdev->ib_dev, path->port);
@@ -4577,6 +4889,71 @@ out: · 208 unmodified lines
45774889 return err;
45784890 }
45794891
4892+static int mlx5_ib_dct_query_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *mqp,
4893+ struct ib_qp_attr *qp_attr, int qp_attr_mask,
4894+ struct ib_qp_init_attr *qp_init_attr)
4895+{
4896+ struct mlx5_core_dct *dct = &mqp->dct.mdct;
4897+ u32 *out;
4898+ u32 access_flags = 0;
4899+ int outlen = MLX5_ST_SZ_BYTES(query_dct_out);
4900+ void *dctc;
4901+ int err;
4902+ int supported_mask = IB_QP_STATE |
4903+ IB_QP_ACCESS_FLAGS |
4904+ IB_QP_PORT |
4905+ IB_QP_MIN_RNR_TIMER |
4906+ IB_QP_AV |
4907+ IB_QP_PATH_MTU |
4908+ IB_QP_PKEY_INDEX;
4909+
4910+ if (qp_attr_mask & ~supported_mask)
4911+ return -EINVAL;
4912+ if (mqp->state != IB_QPS_RTR)
4913+ return -EINVAL;
4914+
4915+ out = kzalloc(outlen, GFP_KERNEL);
4916+ if (!out)
4917+ return -ENOMEM;
4918+
4919+ err = mlx5_core_dct_query(dev->mdev, dct, out, outlen);
4920+ if (err)
4921+ goto out;
4922+
4923+ dctc = MLX5_ADDR_OF(query_dct_out, out, dct_context_entry);
4924+
4925+ if (qp_attr_mask & IB_QP_STATE)
4926+ qp_attr->qp_state = IB_QPS_RTR;
4927+
4928+ if (qp_attr_mask & IB_QP_ACCESS_FLAGS) {
4929+ if (MLX5_GET(dctc, dctc, rre))
4930+ access_flags |= IB_ACCESS_REMOTE_READ;
4931+ if (MLX5_GET(dctc, dctc, rwe))
4932+ access_flags |= IB_ACCESS_REMOTE_WRITE;
4933+ if (MLX5_GET(dctc, dctc, rae))
4934+ access_flags |= IB_ACCESS_REMOTE_ATOMIC;
4935+ qp_attr->qp_access_flags = access_flags;
4936+ }
4937+
4938+ if (qp_attr_mask & IB_QP_PORT)
4939+ qp_attr->port_num = MLX5_GET(dctc, dctc, port);
4940+ if (qp_attr_mask & IB_QP_MIN_RNR_TIMER)
4941+ qp_attr->min_rnr_timer = MLX5_GET(dctc, dctc, min_rnr_nak);
4942+ if (qp_attr_mask & IB_QP_AV) {
4943+ qp_attr->ah_attr.grh.traffic_class = MLX5_GET(dctc, dctc, tclass);
4944+ qp_attr->ah_attr.grh.flow_label = MLX5_GET(dctc, dctc, flow_label);
4945+ qp_attr->ah_attr.grh.sgid_index = MLX5_GET(dctc, dctc, my_addr_index);
4946+ qp_attr->ah_attr.grh.hop_limit = MLX5_GET(dctc, dctc, hop_limit);
4947+ }
4948+ if (qp_attr_mask & IB_QP_PATH_MTU)
4949+ qp_attr->path_mtu = MLX5_GET(dctc, dctc, mtu);
4950+ if (qp_attr_mask & IB_QP_PKEY_INDEX)
4951+ qp_attr->pkey_index = MLX5_GET(dctc, dctc, pkey_index);
4952+out:
4953+ kfree(out);
4954+ return err;
4955+}
4956+
45804957 int mlx5_ib_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
45814958 int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr)
45824959 {
@@ -4596,6 +4973,10 @@ int mlx5_ib_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr, · 13 unmodified lines
45964973 memset(qp_init_attr, 0, sizeof(*qp_init_attr));
45974974 memset(qp_attr, 0, sizeof(*qp_attr));
45984975
4976+ if (unlikely(qp->qp_sub_type == MLX5_IB_QPT_DCT))
4977+ return mlx5_ib_dct_query_qp(dev, qp, qp_attr,
4978+ qp_attr_mask, qp_init_attr);
4979+
45994980 mutex_lock(&qp->mutex);
46004981
46014982 if (qp->ibqp.qp_type == IB_QPT_RAW_PACKET ||
@@ -4685,13 +5066,10 @@ int mlx5_ib_dealloc_xrcd(struct ib_xrcd *xrcd) · 83 unmodified lines
46855066 int err;
46865067
46875068 err = mlx5_core_xrcd_dealloc(dev->mdev, xrcdn);
4688− if (err) {
5069+ if (err)
46895070 mlx5_ib_warn(dev, "failed to dealloc xrcdn 0x%x\n", xrcdn);
4690− return err;
4691− }
46925071
46935072 kfree(xrcd);
4694−
46955073 return 0;
46965074 }
46975075
drivers/infiniband/hw/mthca/mthca_memfree.c+3−4
@@ -472,7 +472,7 @@ int mthca_map_user_db(struct mthca_dev *dev, struct mthca_uar *uar, · 471 unmodified lines
472472 goto out;
473473 }
474474
475− ret = get_user_pages(uaddr & PAGE_MASK, 1, FOLL_WRITE, pages, NULL);
475+ ret = get_user_pages_fast(uaddr & PAGE_MASK, 1, FOLL_WRITE, pages);
476476 if (ret < 0)
477477 goto out;
478478
@@ -623,13 +623,12 @@ int mthca_alloc_db(struct mthca_dev *dev, enum mthca_db_type type, · 144 unmodified lines
623623 page = dev->db_tab->page + end;
624624
625625 alloc:
626− page->db_rec = dma_alloc_coherent(&dev->pdev->dev, MTHCA_ICM_PAGE_SIZE,
627− &page->mapping, GFP_KERNEL);
626+ page->db_rec = dma_zalloc_coherent(&dev->pdev->dev, MTHCA_ICM_PAGE_SIZE,
627+ &page->mapping, GFP_KERNEL);
628628 if (!page->db_rec) {
629629 ret = -ENOMEM;
630630 goto out;
631631 }
632− memset(page->db_rec, 0, MTHCA_ICM_PAGE_SIZE);
633632
634633 ret = mthca_MAP_ICM_page(dev, page->mapping,
635634 mthca_uarc_virt(dev, &dev->driver_uar, i));
drivers/infiniband/hw/mthca/mthca_user.h+0−112
@@ -1,112 +0,0 @@
1−/*
2− * Copyright (c) 2005 Topspin Communications. All rights reserved.
3− * Copyright (c) 2005, 2006 Cisco Systems. All rights reserved.
4− *
5− * This software is available to you under a choice of one of two
6− * licenses. You may choose to be licensed under the terms of the GNU
7− * General Public License (GPL) Version 2, available from the file
8− * COPYING in the main directory of this source tree, or the
9− * OpenIB.org BSD license below:
10− *
11− * Redistribution and use in source and binary forms, with or
12− * without modification, are permitted provided that the following
13− * conditions are met:
14− *
15− * - Redistributions of source code must retain the above
16− * copyright notice, this list of conditions and the following
17− * disclaimer.
18− *
19− * - Redistributions in binary form must reproduce the above
20− * copyright notice, this list of conditions and the following
21− * disclaimer in the documentation and/or other materials
22− * provided with the distribution.
23− *
24− * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
25− * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
26− * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
27− * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
28− * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
29− * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
30− * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
31− * SOFTWARE.
32− */
33−
34−#ifndef MTHCA_USER_H
35−#define MTHCA_USER_H
36−
37−#include <linux/types.h>
38−
39−/*
40− * Increment this value if any changes that break userspace ABI
41− * compatibility are made.
42− */
43−#define MTHCA_UVERBS_ABI_VERSION 1
44−
45−/*
46− * Make sure that all structs defined in this file remain laid out so
47− * that they pack the same way on 32-bit and 64-bit architectures (to
48− * avoid incompatibility between 32-bit userspace and 64-bit kernels).
49− * In particular do not use pointer types -- pass pointers in __u64
50− * instead.
51− */
52−
53−struct mthca_alloc_ucontext_resp {
54− __u32 qp_tab_size;
55− __u32 uarc_size;
56−};
57−
58−struct mthca_alloc_pd_resp {
59− __u32 pdn;
60− __u32 reserved;
61−};
62−
63−struct mthca_reg_mr {
64−/*
65− * Mark the memory region with a DMA attribute that causes
66− * in-flight DMA to be flushed when the region is written to:
67− */
68−#define MTHCA_MR_DMASYNC 0x1
69− __u32 mr_attrs;
70− __u32 reserved;
71−};
72−
73−struct mthca_create_cq {
74− __u32 lkey;
75− __u32 pdn;
76− __u64 arm_db_page;
77− __u64 set_db_page;
78− __u32 arm_db_index;
79− __u32 set_db_index;
80−};
81−
82−struct mthca_create_cq_resp {
83− __u32 cqn;
84− __u32 reserved;
85−};
86−
87−struct mthca_resize_cq {
88− __u32 lkey;
89− __u32 reserved;
90−};
91−
92−struct mthca_create_srq {
93− __u32 lkey;
94− __u32 db_index;
95− __u64 db_page;
96−};
97−
98−struct mthca_create_srq_resp {
99− __u32 srqn;
100− __u32 reserved;
101−};
102−
103−struct mthca_create_qp {
104− __u32 lkey;
105− __u32 reserved;
106− __u64 sq_db_page;
107− __u64 rq_db_page;
108− __u32 sq_db_index;
109− __u32 rq_db_index;
110−};
111−
112−#endif /* MTHCA_USER_H */
drivers/infiniband/hw/nes/nes_cm.c+1−1
@@ -1365,7 +1365,7 @@ static int mini_cm_del_listen(struct nes_cm_core *cm_core, · 1364 unmodified lines
13651365 static inline int mini_cm_accelerated(struct nes_cm_core *cm_core,
13661366 struct nes_cm_node *cm_node)
13671367 {
1368− cm_node->accelerated = 1;
1368+ cm_node->accelerated = true;
13691369
13701370 if (cm_node->accept_pend) {
13711371 BUG_ON(!cm_node->listener);
drivers/infiniband/hw/nes/nes_cm.h+1−2
@@ -279,7 +279,6 @@ struct nes_cm_tcp_context { · 278 unmodified lines
279279 u8 rcv_wscale;
280280
281281 struct nes_cm_tsa_context tsa_cntxt;
282− struct timeval sent_ts;
283282 };
284283
285284
@@ -341,7 +340,7 @@ struct nes_cm_node { · 55 unmodified lines
341340 u16 mpa_frame_size;
342341 struct iw_cm_id *cm_id;
343342 struct list_head list;
344− int accelerated;
343+ bool accelerated;
345344 struct nes_cm_listener *listener;
346345 enum nes_cm_conn_type conn_type;
347346 struct nes_vnic *nesvnic;
drivers/infiniband/hw/ocrdma/ocrdma_hw.c+7−12
@@ -380,11 +380,10 @@ static int ocrdma_alloc_q(struct ocrdma_dev *dev, · 379 unmodified lines
380380 q->len = len;
381381 q->entry_size = entry_size;
382382 q->size = len * entry_size;
383− q->va = dma_alloc_coherent(&dev->nic_info.pdev->dev, q->size,
384− &q->dma, GFP_KERNEL);
383+ q->va = dma_zalloc_coherent(&dev->nic_info.pdev->dev, q->size,
384+ &q->dma, GFP_KERNEL);
385385 if (!q->va)
386386 return -ENOMEM;
387− memset(q->va, 0, q->size);
388387 return 0;
389388 }
390389
@@ -1819,12 +1818,11 @@ int ocrdma_mbx_create_cq(struct ocrdma_dev *dev, struct ocrdma_cq *cq, · 1428 unmodified lines
18191818 return -ENOMEM;
18201819 ocrdma_init_mch(&cmd->cmd.req, OCRDMA_CMD_CREATE_CQ,
18211820 OCRDMA_SUBSYS_COMMON, sizeof(*cmd));
1822− cq->va = dma_alloc_coherent(&pdev->dev, cq->len, &cq->pa, GFP_KERNEL);
1821+ cq->va = dma_zalloc_coherent(&pdev->dev, cq->len, &cq->pa, GFP_KERNEL);
18231822 if (!cq->va) {
18241823 status = -ENOMEM;
18251824 goto mem_err;
18261825 }
1827− memset(cq->va, 0, cq->len);
18281826 page_size = cq->len / hw_pages;
18291827 cmd->cmd.pgsz_pgcnt = (page_size / OCRDMA_MIN_Q_PAGE_SIZE) <<
18301828 OCRDMA_CREATE_CQ_PAGE_SIZE_SHIFT;
@@ -2212,10 +2210,9 @@ static int ocrdma_set_create_qp_sq_cmd(struct ocrdma_create_qp_req *cmd, · 381 unmodified lines
22122210 qp->sq.max_cnt = max_wqe_allocated;
22132211 len = (hw_pages * hw_page_size);
22142212
2215− qp->sq.va = dma_alloc_coherent(&pdev->dev, len, &pa, GFP_KERNEL);
2213+ qp->sq.va = dma_zalloc_coherent(&pdev->dev, len, &pa, GFP_KERNEL);
22162214 if (!qp->sq.va)
22172215 return -EINVAL;
2218− memset(qp->sq.va, 0, len);
22192216 qp->sq.len = len;
22202217 qp->sq.pa = pa;
22212218 qp->sq.entry_size = dev->attr.wqe_size;
@@ -2263,10 +2260,9 @@ static int ocrdma_set_create_qp_rq_cmd(struct ocrdma_create_qp_req *cmd, · 41 unmodified lines
22632260 qp->rq.max_cnt = max_rqe_allocated;
22642261 len = (hw_pages * hw_page_size);
22652262
2266− qp->rq.va = dma_alloc_coherent(&pdev->dev, len, &pa, GFP_KERNEL);
2263+ qp->rq.va = dma_zalloc_coherent(&pdev->dev, len, &pa, GFP_KERNEL);
22672264 if (!qp->rq.va)
22682265 return -ENOMEM;
2269− memset(qp->rq.va, 0, len);
22702266 qp->rq.pa = pa;
22712267 qp->rq.len = len;
22722268 qp->rq.entry_size = dev->attr.rqe_size;
@@ -2320,11 +2316,10 @@ static int ocrdma_set_create_qp_ird_cmd(struct ocrdma_create_qp_req *cmd, · 47 unmodified lines
23202316 if (dev->attr.ird == 0)
23212317 return 0;
23222318
2323− qp->ird_q_va = dma_alloc_coherent(&pdev->dev, ird_q_len,
2324− &pa, GFP_KERNEL);
2319+ qp->ird_q_va = dma_zalloc_coherent(&pdev->dev, ird_q_len, &pa,
2320+ GFP_KERNEL);
23252321 if (!qp->ird_q_va)
23262322 return -ENOMEM;
2327− memset(qp->ird_q_va, 0, ird_q_len);
23282323 ocrdma_build_q_pages(&cmd->ird_addr[0], dev->attr.num_ird_pages,
23292324 pa, ird_page_size);
23302325 for (; i < ird_q_len / dev->attr.rqe_size; i++) {
drivers/infiniband/hw/ocrdma/ocrdma_stats.c+3−5
@@ -73,15 +73,13 @@ bool ocrdma_alloc_stats_resources(struct ocrdma_dev *dev) · 72 unmodified lines
7373 mem->size = max_t(u32, sizeof(struct ocrdma_rdma_stats_req),
7474 sizeof(struct ocrdma_rdma_stats_resp));
7575
76− mem->va = dma_alloc_coherent(&dev->nic_info.pdev->dev, mem->size,
77− &mem->pa, GFP_KERNEL);
76+ mem->va = dma_zalloc_coherent(&dev->nic_info.pdev->dev, mem->size,
77+ &mem->pa, GFP_KERNEL);
7878 if (!mem->va) {
7979 pr_err("%s: stats mbox allocation failed\n", __func__);
8080 return false;
8181 }
8282
83− memset(mem->va, 0, mem->size);
84−
8583 /* Alloc debugfs mem */
8684 mem->debugfs_mem = kzalloc(OCRDMA_MAX_DBGFS_MEM, GFP_KERNEL);
8785 if (!mem->debugfs_mem)
@@ -834,7 +832,7 @@ void ocrdma_add_port_stats(struct ocrdma_dev *dev) · 746 unmodified lines
834832
835833 dev->reset_stats.type = OCRDMA_RESET_STATS;
836834 dev->reset_stats.dev = dev;
837− if (!debugfs_create_file("reset_stats", S_IRUSR, dev->dir,
835+ if (!debugfs_create_file("reset_stats", 0200, dev->dir,
838836 &dev->reset_stats, &ocrdma_dbg_ops))
839837 goto err;
840838
drivers/infiniband/hw/ocrdma/ocrdma_verbs.c+4−6
@@ -461,7 +461,7 @@ retry: · 460 unmodified lines
461461 static inline int is_ucontext_pd(struct ocrdma_ucontext *uctx,
462462 struct ocrdma_pd *pd)
463463 {
464− return (uctx->cntxt_pd == pd ? true : false);
464+ return (uctx->cntxt_pd == pd);
465465 }
466466
467467 static int _ocrdma_dealloc_pd(struct ocrdma_dev *dev,
@@ -550,13 +550,12 @@ struct ib_ucontext *ocrdma_alloc_ucontext(struct ib_device *ibdev, · 82 unmodified lines
550550 INIT_LIST_HEAD(&ctx->mm_head);
551551 mutex_init(&ctx->mm_list_lock);
552552
553− ctx->ah_tbl.va = dma_alloc_coherent(&pdev->dev, map_len,
554− &ctx->ah_tbl.pa, GFP_KERNEL);
553+ ctx->ah_tbl.va = dma_zalloc_coherent(&pdev->dev, map_len,
554+ &ctx->ah_tbl.pa, GFP_KERNEL);
555555 if (!ctx->ah_tbl.va) {
556556 kfree(ctx);
557557 return ERR_PTR(-ENOMEM);
558558 }
559− memset(ctx->ah_tbl.va, 0, map_len);
560559 ctx->ah_tbl.len = map_len;
561560
562561 memset(&resp, 0, sizeof(resp));
@@ -885,13 +884,12 @@ static int ocrdma_build_pbl_tbl(struct ocrdma_dev *dev, struct ocrdma_hw_mr *mr) · 322 unmodified lines
885884 return -ENOMEM;
886885
887886 for (i = 0; i < mr->num_pbls; i++) {
888− va = dma_alloc_coherent(&pdev->dev, dma_len, &pa, GFP_KERNEL);
887+ va = dma_zalloc_coherent(&pdev->dev, dma_len, &pa, GFP_KERNEL);
889888 if (!va) {
890889 ocrdma_free_mr_pbl_tbl(dev, mr);
891890 status = -ENOMEM;
892891 break;
893892 }
894− memset(va, 0, dma_len);
895893 mr->pbl_table[i].va = va;
896894 mr->pbl_table[i].pa = pa;
897895 }
drivers/infiniband/hw/qedr/verbs.c+7−8
@@ -604,12 +604,11 @@ static struct qedr_pbl *qedr_alloc_pbl_tbl(struct qedr_dev *dev, · 603 unmodified lines
604604 return ERR_PTR(-ENOMEM);
605605
606606 for (i = 0; i < pbl_info->num_pbls; i++) {
607− va = dma_alloc_coherent(&pdev->dev, pbl_info->pbl_size,
608− &pa, flags);
607+ va = dma_zalloc_coherent(&pdev->dev, pbl_info->pbl_size,
608+ &pa, flags);
609609 if (!va)
610610 goto err;
611611
612− memset(va, 0, pbl_info->pbl_size);
613612 pbl_table[i].va = va;
614613 pbl_table[i].pa = pa;
615614 }
@@ -3040,7 +3039,7 @@ static int __qedr_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, · 2424 unmodified lines
30403039 swqe->wqe_size = 2;
30413040 swqe2 = qed_chain_produce(&qp->sq.pbl);
30423041
3043− swqe->inv_key_or_imm_data = cpu_to_le32(wr->ex.imm_data);
3042+ swqe->inv_key_or_imm_data = cpu_to_le32(be32_to_cpu(wr->ex.imm_data));
30443043 length = qedr_prepare_sq_send_data(dev, qp, swqe, swqe2,
30453044 wr, bad_wr);
30463045 swqe->length = cpu_to_le32(length);
@@ -3471,9 +3470,9 @@ static int qedr_poll_cq_req(struct qedr_dev *dev, · 424 unmodified lines
34713470 break;
34723471 case RDMA_CQE_REQ_STS_WORK_REQUEST_FLUSHED_ERR:
34733472 if (qp->state != QED_ROCE_QP_STATE_ERR)
3474− DP_ERR(dev,
3475− "Error: POLL CQ with RDMA_CQE_REQ_STS_WORK_REQUEST_FLUSHED_ERR. CQ icid=0x%x, QP icid=0x%x\n",
3476− cq->icid, qp->icid);
3473+ DP_DEBUG(dev, QEDR_MSG_CQ,
3474+ "Error: POLL CQ with RDMA_CQE_REQ_STS_WORK_REQUEST_FLUSHED_ERR. CQ icid=0x%x, QP icid=0x%x\n",
3475+ cq->icid, qp->icid);
34773476 cnt = process_req(dev, qp, cq, num_entries, wc, req->sq_cons,
34783477 IB_WC_WR_FLUSH_ERR, 1);
34793478 break;
@@ -3591,7 +3590,7 @@ static inline int qedr_set_ok_cqe_resp_wc(struct rdma_cqe_responder *resp, · 111 unmodified lines
35913590 wc->byte_len = le32_to_cpu(resp->length);
35923591
35933592 if (resp->flags & QEDR_RESP_IMM) {
3594− wc->ex.imm_data = le32_to_cpu(resp->imm_data_or_inv_r_Key);
3593+ wc->ex.imm_data = cpu_to_be32(le32_to_cpu(resp->imm_data_or_inv_r_Key));
35953594 wc->wc_flags |= IB_WC_WITH_IMM;
35963595
35973596 if (resp->flags & QEDR_RESP_RDMA)
drivers/infiniband/hw/qib/qib.h+3−5
@@ -1428,8 +1428,6 @@ u64 qib_sps_ints(void); · 1427 unmodified lines
14281428 */
14291429 dma_addr_t qib_map_page(struct pci_dev *, struct page *, unsigned long,
14301430 size_t, int);
1431−const char *qib_get_unit_name(int unit);
1432−const char *qib_get_card_name(struct rvt_dev_info *rdi);
14331431 struct pci_dev *qib_get_pci_dev(struct rvt_dev_info *rdi);
14341432
14351433 /*
@@ -1488,15 +1486,15 @@ extern struct mutex qib_mutex; · 52 unmodified lines
14881486
14891487 #define qib_dev_err(dd, fmt, ...) \
14901488 dev_err(&(dd)->pcidev->dev, "%s: " fmt, \
1491− qib_get_unit_name((dd)->unit), ##__VA_ARGS__)
1489+ rvt_get_ibdev_name(&(dd)->verbs_dev.rdi), ##__VA_ARGS__)
14921490
14931491 #define qib_dev_warn(dd, fmt, ...) \
14941492 dev_warn(&(dd)->pcidev->dev, "%s: " fmt, \
1495− qib_get_unit_name((dd)->unit), ##__VA_ARGS__)
1493+ rvt_get_ibdev_name(&(dd)->verbs_dev.rdi), ##__VA_ARGS__)
14961494
14971495 #define qib_dev_porterr(dd, port, fmt, ...) \
14981496 dev_err(&(dd)->pcidev->dev, "%s: IB%u:%u " fmt, \
1499− qib_get_unit_name((dd)->unit), (dd)->unit, (port), \
1497+ rvt_get_ibdev_name(&(dd)->verbs_dev.rdi), (dd)->unit, (port), \
15001498 ##__VA_ARGS__)
15011499
15021500 #define qib_devinfo(pcidev, fmt, ...) \
drivers/infiniband/hw/qib/qib_driver.c+0−16
@@ -81,22 +81,6 @@ MODULE_DESCRIPTION("Intel IB driver"); · 80 unmodified lines
8181
8282 struct qlogic_ib_stats qib_stats;
8383
84−const char *qib_get_unit_name(int unit)
85−{
86− static char iname[16];
87−
88− snprintf(iname, sizeof(iname), "infinipath%u", unit);
89− return iname;
90−}
91−
92−const char *qib_get_card_name(struct rvt_dev_info *rdi)
93−{
94− struct qib_ibdev *ibdev = container_of(rdi, struct qib_ibdev, rdi);
95− struct qib_devdata *dd = container_of(ibdev,
96− struct qib_devdata, verbs_dev);
97− return qib_get_unit_name(dd->unit);
98−}
99−
10084 struct pci_dev *qib_get_pci_dev(struct rvt_dev_info *rdi)
10185 {
10286 struct qib_ibdev *ibdev = container_of(rdi, struct qib_ibdev, rdi);
drivers/infiniband/hw/qib/qib_eeprom.c+1−2
@@ -163,8 +163,7 @@ void qib_get_eeprom_info(struct qib_devdata *dd) · 162 unmodified lines
163163 if (bguid[6] == 0xff) {
164164 if (bguid[5] == 0xff) {
165165 qib_dev_err(dd,
166− "Can't set %s GUID from base, wraps to OUI!\n",
167− qib_get_unit_name(t));
166+ "Can't set GUID from base, wraps to OUI!\n");
168167 dd->base_guid = 0;
169168 goto bail;
170169 }
drivers/infiniband/hw/qib/qib_file_ops.c+25−43
@@ -568,20 +568,16 @@ done: · 567 unmodified lines
568568 static int qib_set_part_key(struct qib_ctxtdata *rcd, u16 key)
569569 {
570570 struct qib_pportdata *ppd = rcd->ppd;
571− int i, any = 0, pidx = -1;
571+ int i, pidx = -1;
572+ bool any = false;
572573 u16 lkey = key & 0x7FFF;
573− int ret;
574574
575− if (lkey == (QIB_DEFAULT_P_KEY & 0x7FFF)) {
575+ if (lkey == (QIB_DEFAULT_P_KEY & 0x7FFF))
576576 /* nothing to do; this key always valid */
577− ret = 0;
578− goto bail;
579− }
577+ return 0;
580578
581− if (!lkey) {
582− ret = -EINVAL;
583− goto bail;
584− }
579+ if (!lkey)
580+ return -EINVAL;
585581
586582 /*
587583 * Set the full membership bit, because it has to be
@@ -594,18 +590,14 @@ static int qib_set_part_key(struct qib_ctxtdata *rcd, u16 key) · 6 unmodified lines
594590 for (i = 0; i < ARRAY_SIZE(rcd->pkeys); i++) {
595591 if (!rcd->pkeys[i] && pidx == -1)
596592 pidx = i;
597− if (rcd->pkeys[i] == key) {
598− ret = -EEXIST;
599− goto bail;
600− }
593+ if (rcd->pkeys[i] == key)
594+ return -EEXIST;
601595 }
602− if (pidx == -1) {
603− ret = -EBUSY;
604− goto bail;
605− }
606− for (any = i = 0; i < ARRAY_SIZE(ppd->pkeys); i++) {
596+ if (pidx == -1)
597+ return -EBUSY;
598+ for (i = 0; i < ARRAY_SIZE(ppd->pkeys); i++) {
607599 if (!ppd->pkeys[i]) {
608− any++;
600+ any = true;
609601 continue;
610602 }
611603 if (ppd->pkeys[i] == key) {
@@ -613,44 +605,34 @@ static int qib_set_part_key(struct qib_ctxtdata *rcd, u16 key) · 1 unmodified line
613605
614606 if (atomic_inc_return(pkrefs) > 1) {
615607 rcd->pkeys[pidx] = key;
616− ret = 0;
617− goto bail;
618− } else {
619− /*
620− * lost race, decrement count, catch below
621− */
622− atomic_dec(pkrefs);
623− any++;
608+ return 0;
624609 }
610+ /*
611+ * lost race, decrement count, catch below
612+ */
613+ atomic_dec(pkrefs);
614+ any = true;
625615 }
626− if ((ppd->pkeys[i] & 0x7FFF) == lkey) {
616+ if ((ppd->pkeys[i] & 0x7FFF) == lkey)
627617 /*
628618 * It makes no sense to have both the limited and
629619 * full membership PKEY set at the same time since
630620 * the unlimited one will disable the limited one.
631621 */
632− ret = -EEXIST;
633− goto bail;
634− }
635− }
636− if (!any) {
637− ret = -EBUSY;
638− goto bail;
622+ return -EEXIST;
639623 }
640− for (any = i = 0; i < ARRAY_SIZE(ppd->pkeys); i++) {
624+ if (!any)
625+ return -EBUSY;
626+ for (i = 0; i < ARRAY_SIZE(ppd->pkeys); i++) {
641627 if (!ppd->pkeys[i] &&
642628 atomic_inc_return(&ppd->pkeyrefs[i]) == 1) {
643629 rcd->pkeys[pidx] = key;
644630 ppd->pkeys[i] = key;
645631 (void) ppd->dd->f_set_ib_cfg(ppd, QIB_IB_CFG_PKEYS, 0);
646− ret = 0;
647− goto bail;
632+ return 0;
648633 }
649634 }
650− ret = -EBUSY;
651−
652−bail:
653− return ret;
635+ return -EBUSY;
654636 }
655637
656638 /**
drivers/infiniband/hw/qib/qib_init.c+2−0
@@ -1119,6 +1119,8 @@ struct qib_devdata *qib_alloc_devdata(struct pci_dev *pdev, size_t extra) · 1118 unmodified lines
11191119 "Could not allocate unit ID: error %d\n", -ret);
11201120 goto bail;
11211121 }
1122+ rvt_set_ibdev_name(&dd->verbs_dev.rdi, "%s%d", "qib", dd->unit);
1123+
11221124 dd->int_counter = alloc_percpu(u64);
11231125 if (!dd->int_counter) {
11241126 ret = -ENOMEM;
drivers/infiniband/hw/qib/qib_keys.c+0−235
@@ -1,235 +0,0 @@
1−/*
2− * Copyright (c) 2006, 2007, 2009 QLogic Corporation. All rights reserved.
3− * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
4− *
5− * This software is available to you under a choice of one of two
6− * licenses. You may choose to be licensed under the terms of the GNU
7− * General Public License (GPL) Version 2, available from the file
8− * COPYING in the main directory of this source tree, or the
9− * OpenIB.org BSD license below:
10− *
11− * Redistribution and use in source and binary forms, with or
12− * without modification, are permitted provided that the following
13− * conditions are met:
14− *
15− * - Redistributions of source code must retain the above
16− * copyright notice, this list of conditions and the following
17− * disclaimer.
18− *
19− * - Redistributions in binary form must reproduce the above
20− * copyright notice, this list of conditions and the following
21− * disclaimer in the documentation and/or other materials
22− * provided with the distribution.
23− *
24− * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
25− * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
26− * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
27− * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
28− * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
29− * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
30− * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
31− * SOFTWARE.
32− */
33−
34−#include "qib.h"
35−
36−/**
37− * qib_alloc_lkey - allocate an lkey
38− * @mr: memory region that this lkey protects
39− * @dma_region: 0->normal key, 1->restricted DMA key
40− *
41− * Returns 0 if successful, otherwise returns -errno.
42− *
43− * Increments mr reference count as required.
44− *
45− * Sets the lkey field mr for non-dma regions.
46− *
47− */
48−
49−int qib_alloc_lkey(struct rvt_mregion *mr, int dma_region)
50−{
51− unsigned long flags;
52− u32 r;
53− u32 n;
54− int ret = 0;
55− struct qib_ibdev *dev = to_idev(mr->pd->device);
56− struct rvt_lkey_table *rkt = &dev->lk_table;
57−
58− spin_lock_irqsave(&rkt->lock, flags);
59−
60− /* special case for dma_mr lkey == 0 */
61− if (dma_region) {
62− struct rvt_mregion *tmr;
63−
64− tmr = rcu_access_pointer(dev->dma_mr);
65− if (!tmr) {
66− qib_get_mr(mr);
67− rcu_assign_pointer(dev->dma_mr, mr);
68− mr->lkey_published = 1;
69− }
70− goto success;
71− }
72−
73− /* Find the next available LKEY */
74− r = rkt->next;
75− n = r;
76− for (;;) {
77− if (rkt->table[r] == NULL)
78− break;
79− r = (r + 1) & (rkt->max - 1);
80− if (r == n)
81− goto bail;
82− }
83− rkt->next = (r + 1) & (rkt->max - 1);
84− /*
85− * Make sure lkey is never zero which is reserved to indicate an
86− * unrestricted LKEY.
87− */
88− rkt->gen++;
89− /*
90− * bits are capped in qib_verbs.c to insure enough bits
91− * for generation number
92− */
93− mr->lkey = (r << (32 - ib_rvt_lkey_table_size)) |
94− ((((1 << (24 - ib_rvt_lkey_table_size)) - 1) & rkt->gen)
95− << 8);
96− if (mr->lkey == 0) {
97− mr->lkey |= 1 << 8;
98− rkt->gen++;
99− }
100− qib_get_mr(mr);
101− rcu_assign_pointer(rkt->table[r], mr);
102− mr->lkey_published = 1;
103−success:
104− spin_unlock_irqrestore(&rkt->lock, flags);
105−out:
106− return ret;
107−bail:
108− spin_unlock_irqrestore(&rkt->lock, flags);
109− ret = -ENOMEM;
110− goto out;
111−}
112−
113−/**
114− * qib_free_lkey - free an lkey
115− * @mr: mr to free from tables
116− */
117−void qib_free_lkey(struct rvt_mregion *mr)
118−{
119− unsigned long flags;
120− u32 lkey = mr->lkey;
121− u32 r;
122− struct qib_ibdev *dev = to_idev(mr->pd->device);
123− struct rvt_lkey_table *rkt = &dev->lk_table;
124−
125− spin_lock_irqsave(&rkt->lock, flags);
126− if (!mr->lkey_published)
127− goto out;
128− if (lkey == 0)
129− RCU_INIT_POINTER(dev->dma_mr, NULL);
130− else {
131− r = lkey >> (32 - ib_rvt_lkey_table_size);
132− RCU_INIT_POINTER(rkt->table[r], NULL);
133− }
134− qib_put_mr(mr);
135− mr->lkey_published = 0;
136−out:
137− spin_unlock_irqrestore(&rkt->lock, flags);
138−}
139−
140−/**
141− * qib_rkey_ok - check the IB virtual address, length, and RKEY
142− * @qp: qp for validation
143− * @sge: SGE state
144− * @len: length of data
145− * @vaddr: virtual address to place data
146− * @rkey: rkey to check
147− * @acc: access flags
148− *
149− * Return 1 if successful, otherwise 0.
150− *
151− * increments the reference count upon success
152− */
153−int qib_rkey_ok(struct rvt_qp *qp, struct rvt_sge *sge,
154− u32 len, u64 vaddr, u32 rkey, int acc)
155−{
156− struct rvt_lkey_table *rkt = &to_idev(qp->ibqp.device)->lk_table;
157− struct rvt_mregion *mr;
158− unsigned n, m;
159− size_t off;
160−
161− /* We use RKEY == zero for kernel virtual addresses */
162− rcu_read_lock();
163− if (rkey == 0) {
164− struct rvt_pd *pd = ibpd_to_rvtpd(qp->ibqp.pd);
165− struct qib_ibdev *dev = to_idev(pd->ibpd.device);
166−
167− if (pd->user)
168− goto bail;
169− mr = rcu_dereference(dev->dma_mr);
170− if (!mr)
171− goto bail;
172− if (unlikely(!atomic_inc_not_zero(&mr->refcount)))
173− goto bail;
174− rcu_read_unlock();
175−
176− sge->mr = mr;
177− sge->vaddr = (void *) vaddr;
178− sge->length = len;
179− sge->sge_length = len;
180− sge->m = 0;
181− sge->n = 0;
182− goto ok;
183− }
184−
185− mr = rcu_dereference(
186− rkt->table[(rkey >> (32 - ib_rvt_lkey_table_size))]);
187− if (unlikely(!mr || mr->lkey != rkey || qp->ibqp.pd != mr->pd))
188− goto bail;
189−
190− off = vaddr - mr->iova;
191− if (unlikely(vaddr < mr->iova || off + len > mr->length ||
192− (mr->access_flags & acc) == 0))
193− goto bail;
194− if (unlikely(!atomic_inc_not_zero(&mr->refcount)))
195− goto bail;
196− rcu_read_unlock();
197−
198− off += mr->offset;
199− if (mr->page_shift) {
200− /*
201− page sizes are uniform power of 2 so no loop is necessary
202− entries_spanned_by_off is the number of times the loop below
203− would have executed.
204− */
205− size_t entries_spanned_by_off;
206−
207− entries_spanned_by_off = off >> mr->page_shift;
208− off -= (entries_spanned_by_off << mr->page_shift);
209− m = entries_spanned_by_off / RVT_SEGSZ;
210− n = entries_spanned_by_off % RVT_SEGSZ;
211− } else {
212− m = 0;
213− n = 0;
214− while (off >= mr->map[m]->segs[n].length) {
215− off -= mr->map[m]->segs[n].length;
216− n++;
217− if (n >= RVT_SEGSZ) {
218− m++;
219− n = 0;
220− }
221− }
222− }
223− sge->mr = mr;
224− sge->vaddr = mr->map[m]->segs[n].vaddr + off;
225− sge->length = mr->map[m]->segs[n].length - off;
226− sge->sge_length = len;
227− sge->m = m;
228− sge->n = n;
229−ok:
230− return 1;
231−bail:
232− rcu_read_unlock();
233− return 0;
234−}
235−
drivers/infiniband/hw/qib/qib_rc.c+3−3
@@ -432,13 +432,13 @@ no_flow_control: · 431 unmodified lines
432432 qp->s_state = OP(COMPARE_SWAP);
433433 put_ib_ateth_swap(wqe->atomic_wr.swap,
434434 &ohdr->u.atomic_eth);
435− put_ib_ateth_swap(wqe->atomic_wr.compare_add,
436− &ohdr->u.atomic_eth);
435+ put_ib_ateth_compare(wqe->atomic_wr.compare_add,
436+ &ohdr->u.atomic_eth);
437437 } else {
438438 qp->s_state = OP(FETCH_ADD);
439439 put_ib_ateth_swap(wqe->atomic_wr.compare_add,
440440 &ohdr->u.atomic_eth);
441− put_ib_ateth_swap(0, &ohdr->u.atomic_eth);
441+ put_ib_ateth_compare(0, &ohdr->u.atomic_eth);
442442 }
443443 put_ib_ateth_vaddr(wqe->atomic_wr.remote_addr,
444444 &ohdr->u.atomic_eth);
drivers/infiniband/hw/qib/qib_verbs.c+0−2
@@ -1571,7 +1571,6 @@ int qib_register_ib_device(struct qib_devdata *dd) · 1570 unmodified lines
15711571 if (!ib_qib_sys_image_guid)
15721572 ib_qib_sys_image_guid = ppd->guid;
15731573
1574− strlcpy(ibdev->name, "qib%d", IB_DEVICE_NAME_MAX);
15751574 ibdev->owner = THIS_MODULE;
15761575 ibdev->node_guid = ppd->guid;
15771576 ibdev->phys_port_cnt = dd->num_pports;
@@ -1586,7 +1585,6 @@ int qib_register_ib_device(struct qib_devdata *dd) · 8 unmodified lines
15861585 * Fill in rvt info object.
15871586 */
15881587 dd->verbs_dev.rdi.driver_f.port_callback = qib_create_port_files;
1589− dd->verbs_dev.rdi.driver_f.get_card_name = qib_get_card_name;
15901588 dd->verbs_dev.rdi.driver_f.get_pci_dev = qib_get_pci_dev;
15911589 dd->verbs_dev.rdi.driver_f.check_ah = qib_check_ah;
15921590 dd->verbs_dev.rdi.driver_f.check_send_wqe = qib_check_send_wqe;
drivers/infiniband/hw/usnic/usnic_ib_sysfs.c+0−1
@@ -45,7 +45,6 @@ · 44 unmodified lines
4545 #include "usnic_ib_verbs.h"
4646 #include "usnic_ib_sysfs.h"
4747 #include "usnic_log.h"
48−#include "usnic_ib_sysfs.h"
4948
5049 static ssize_t usnic_ib_show_board(struct device *device,
5150 struct device_attribute *attr,
drivers/infiniband/hw/usnic/usnic_ib_verbs.c+0−1
@@ -47,7 +47,6 @@ · 46 unmodified lines
4747 #include "usnic_log.h"
4848 #include "usnic_uiom.h"
4949 #include "usnic_transport.h"
50−#include "usnic_ib_verbs.h"
5150
5251 #define USNIC_DEFAULT_TRANSPORT USNIC_TRANSPORT_ROCE_CUSTOM
5352
drivers/infiniband/hw/vmw_pvrdma/pvrdma.h+2−2
@@ -93,7 +93,7 @@ struct pvrdma_cq { · 92 unmodified lines
9393 struct pvrdma_page_dir pdir;
9494 u32 cq_handle;
9595 bool is_kernel;
96− atomic_t refcnt;
96+ refcount_t refcnt;
9797 struct completion free;
9898 };
9999
@@ -196,7 +196,7 @@ struct pvrdma_qp { · 96 unmodified lines
196196 u8 state;
197197 bool is_kernel;
198198 struct mutex mutex; /* QP state mutex. */
199− atomic_t refcnt;
199+ refcount_t refcnt;
200200 struct completion free;
201201 };
202202
drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c+6−7
@@ -132,8 +132,9 @@ struct ib_cq *pvrdma_create_cq(struct ib_device *ibdev, · 131 unmodified lines
132132 }
133133
134134 cq->ibcq.cqe = entries;
135+ cq->is_kernel = !context;
135136
136− if (context) {
137+ if (!cq->is_kernel) {
137138 if (ib_copy_from_udata(&ucmd, udata, sizeof(ucmd))) {
138139 ret = -EFAULT;
139140 goto err_cq;
@@ -148,8 +149,6 @@ struct ib_cq *pvrdma_create_cq(struct ib_device *ibdev, · 8 unmodified lines
148149
149150 npages = ib_umem_page_count(cq->umem);
150151 } else {
151− cq->is_kernel = true;
152−
153152 /* One extra page for shared ring state */
154153 npages = 1 + (entries * sizeof(struct pvrdma_cqe) +
155154 PAGE_SIZE - 1) / PAGE_SIZE;
@@ -178,7 +177,7 @@ struct ib_cq *pvrdma_create_cq(struct ib_device *ibdev, · 22 unmodified lines
178177 else
179178 pvrdma_page_dir_insert_umem(&cq->pdir, cq->umem, 0);
180179
181− atomic_set(&cq->refcnt, 1);
180+ refcount_set(&cq->refcnt, 1);
182181 init_completion(&cq->free);
183182 spin_lock_init(&cq->cq_lock);
184183
@@ -202,7 +201,7 @@ struct ib_cq *pvrdma_create_cq(struct ib_device *ibdev, · 17 unmodified lines
202201 dev->cq_tbl[cq->cq_handle % dev->dsr->caps.max_cq] = cq;
203202 spin_unlock_irqrestore(&dev->cq_tbl_lock, flags);
204203
205− if (context) {
204+ if (!cq->is_kernel) {
206205 cq->uar = &(to_vucontext(context)->uar);
207206
208207 /* Copy udata back. */
@@ -219,7 +218,7 @@ struct ib_cq *pvrdma_create_cq(struct ib_device *ibdev, · 10 unmodified lines
219218 err_page_dir:
220219 pvrdma_page_dir_cleanup(dev, &cq->pdir);
221220 err_umem:
222− if (context)
221+ if (!cq->is_kernel)
223222 ib_umem_release(cq->umem);
224223 err_cq:
225224 atomic_dec(&dev->num_cqs);
@@ -230,7 +229,7 @@ err_cq: · 4 unmodified lines
230229
231230 static void pvrdma_free_cq(struct pvrdma_dev *dev, struct pvrdma_cq *cq)
232231 {
233− if (atomic_dec_and_test(&cq->refcnt))
232+ if (refcount_dec_and_test(&cq->refcnt))
234233 complete(&cq->free);
235234 wait_for_completion(&cq->free);
236235
drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c+10−11
@@ -243,13 +243,13 @@ static int pvrdma_register_device(struct pvrdma_dev *dev) · 242 unmodified lines
243243 mutex_init(&dev->port_mutex);
244244 spin_lock_init(&dev->desc_lock);
245245
246− dev->cq_tbl = kcalloc(dev->dsr->caps.max_cq, sizeof(void *),
246+ dev->cq_tbl = kcalloc(dev->dsr->caps.max_cq, sizeof(struct pvrdma_cq *),
247247 GFP_KERNEL);
248248 if (!dev->cq_tbl)
249249 return ret;
250250 spin_lock_init(&dev->cq_tbl_lock);
251251
252− dev->qp_tbl = kcalloc(dev->dsr->caps.max_qp, sizeof(void *),
252+ dev->qp_tbl = kcalloc(dev->dsr->caps.max_qp, sizeof(struct pvrdma_qp *),
253253 GFP_KERNEL);
254254 if (!dev->qp_tbl)
255255 goto err_cq_free;
@@ -333,7 +333,7 @@ static void pvrdma_qp_event(struct pvrdma_dev *dev, u32 qpn, int type) · 77 unmodified lines
333333 spin_lock_irqsave(&dev->qp_tbl_lock, flags);
334334 qp = dev->qp_tbl[qpn % dev->dsr->caps.max_qp];
335335 if (qp)
336− atomic_inc(&qp->refcnt);
336+ refcount_inc(&qp->refcnt);
337337 spin_unlock_irqrestore(&dev->qp_tbl_lock, flags);
338338
339339 if (qp && qp->ibqp.event_handler) {
@@ -346,7 +346,7 @@ static void pvrdma_qp_event(struct pvrdma_dev *dev, u32 qpn, int type) · 6 unmodified lines
346346 ibqp->event_handler(&e, ibqp->qp_context);
347347 }
348348 if (qp) {
349− if (atomic_dec_and_test(&qp->refcnt))
349+ if (refcount_dec_and_test(&qp->refcnt))
350350 complete(&qp->free);
351351 }
352352 }
@@ -359,7 +359,7 @@ static void pvrdma_cq_event(struct pvrdma_dev *dev, u32 cqn, int type) · 6 unmodified lines
359359 spin_lock_irqsave(&dev->cq_tbl_lock, flags);
360360 cq = dev->cq_tbl[cqn % dev->dsr->caps.max_cq];
361361 if (cq)
362− atomic_inc(&cq->refcnt);
362+ refcount_inc(&cq->refcnt);
363363 spin_unlock_irqrestore(&dev->cq_tbl_lock, flags);
364364
365365 if (cq && cq->ibcq.event_handler) {
@@ -372,7 +372,7 @@ static void pvrdma_cq_event(struct pvrdma_dev *dev, u32 cqn, int type) · 6 unmodified lines
372372 ibcq->event_handler(&e, ibcq->cq_context);
373373 }
374374 if (cq) {
375− if (atomic_dec_and_test(&cq->refcnt))
375+ if (refcount_dec_and_test(&cq->refcnt))
376376 complete(&cq->free);
377377 }
378378 }
@@ -531,13 +531,13 @@ static irqreturn_t pvrdma_intrx_handler(int irq, void *dev_id) · 152 unmodified lines
531531 spin_lock_irqsave(&dev->cq_tbl_lock, flags);
532532 cq = dev->cq_tbl[cqne->info % dev->dsr->caps.max_cq];
533533 if (cq)
534− atomic_inc(&cq->refcnt);
534+ refcount_inc(&cq->refcnt);
535535 spin_unlock_irqrestore(&dev->cq_tbl_lock, flags);
536536
537537 if (cq && cq->ibcq.comp_handler)
538538 cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context);
539539 if (cq) {
540− if (atomic_dec_and_test(&cq->refcnt))
540+ if (refcount_dec_and_test(&cq->refcnt))
541541 complete(&cq->free);
542542 }
543543 pvrdma_idx_ring_inc(&ring->cons_head, ring_slots);
@@ -882,8 +882,8 @@ static int pvrdma_pci_probe(struct pci_dev *pdev, · 338 unmodified lines
882882 dev_info(&pdev->dev, "device version %d, driver version %d\n",
883883 dev->dsr_version, PVRDMA_VERSION);
884884
885− dev->dsr = dma_alloc_coherent(&pdev->dev, sizeof(*dev->dsr),
886− &dev->dsrbase, GFP_KERNEL);
885+ dev->dsr = dma_zalloc_coherent(&pdev->dev, sizeof(*dev->dsr),
886+ &dev->dsrbase, GFP_KERNEL);
887887 if (!dev->dsr) {
888888 dev_err(&pdev->dev, "failed to allocate shared region\n");
889889 ret = -ENOMEM;
@@ -891,7 +891,6 @@ static int pvrdma_pci_probe(struct pci_dev *pdev, · 1 unmodified line
891891 }
892892
893893 /* Setup the shared region */
894− memset(dev->dsr, 0, sizeof(*dev->dsr));
895894 dev->dsr->driver_version = PVRDMA_VERSION;
896895 dev->dsr->gos_info.gos_bits = sizeof(void *) == 4 ?
897896 PVRDMA_GOS_BITS_32 :
drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c+4−11
@@ -119,10 +119,7 @@ struct ib_mr *pvrdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, · 118 unmodified lines
119119 union pvrdma_cmd_resp rsp;
120120 struct pvrdma_cmd_create_mr *cmd = &req.create_mr;
121121 struct pvrdma_cmd_create_mr_resp *resp = &rsp.create_mr_resp;
122− int nchunks;
123122 int ret;
124− int entry;
125− struct scatterlist *sg;
126123
127124 if (length == 0 || length > dev->dsr->caps.max_mr_size) {
128125 dev_warn(&dev->pdev->dev, "invalid mem region length\n");
@@ -137,13 +134,9 @@ struct ib_mr *pvrdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, · 8 unmodified lines
137134 return ERR_CAST(umem);
138135 }
139136
140− nchunks = 0;
141− for_each_sg(umem->sg_head.sgl, sg, umem->nmap, entry)
142− nchunks += sg_dma_len(sg) >> PAGE_SHIFT;
143−
144− if (nchunks < 0 || nchunks > PVRDMA_PAGE_DIR_MAX_PAGES) {
137+ if (umem->npages < 0 || umem->npages > PVRDMA_PAGE_DIR_MAX_PAGES) {
145138 dev_warn(&dev->pdev->dev, "overflow %d pages in mem region\n",
146− nchunks);
139+ umem->npages);
147140 ret = -EINVAL;
148141 goto err_umem;
149142 }
@@ -158,7 +151,7 @@ struct ib_mr *pvrdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, · 8 unmodified lines
158151 mr->mmr.size = length;
159152 mr->umem = umem;
160153
161− ret = pvrdma_page_dir_init(dev, &mr->pdir, nchunks, false);
154+ ret = pvrdma_page_dir_init(dev, &mr->pdir, umem->npages, false);
162155 if (ret) {
163156 dev_warn(&dev->pdev->dev,
164157 "could not allocate page directory\n");
@@ -175,7 +168,7 @@ struct ib_mr *pvrdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, · 10 unmodified lines
175168 cmd->length = length;
176169 cmd->pd_handle = to_vpd(pd)->pd_handle;
177170 cmd->access_flags = access_flags;
178− cmd->nchunks = nchunks;
171+ cmd->nchunks = umem->npages;
179172 cmd->pdir_dma = mr->pdir.dir_dma;
180173
181174 ret = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_CREATE_MR_RESP);
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c+5−6
@@ -245,12 +245,13 @@ struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, · 244 unmodified lines
245245 spin_lock_init(&qp->sq.lock);
246246 spin_lock_init(&qp->rq.lock);
247247 mutex_init(&qp->mutex);
248− atomic_set(&qp->refcnt, 1);
248+ refcount_set(&qp->refcnt, 1);
249249 init_completion(&qp->free);
250250
251251 qp->state = IB_QPS_RESET;
252+ qp->is_kernel = !(pd->uobject && udata);
252253
253− if (pd->uobject && udata) {
254+ if (!qp->is_kernel) {
254255 dev_dbg(&dev->pdev->dev,
255256 "create queuepair from user space\n");
256257
@@ -291,8 +292,6 @@ struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, · 34 unmodified lines
291292 qp->npages_recv = 0;
292293 qp->npages = qp->npages_send + qp->npages_recv;
293294 } else {
294− qp->is_kernel = true;
295−
296295 ret = pvrdma_set_sq_size(to_vdev(pd->device),
297296 &init_attr->cap, qp);
298297 if (ret)
@@ -394,7 +393,7 @@ struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, · 95 unmodified lines
394393 err_pdir:
395394 pvrdma_page_dir_cleanup(dev, &qp->pdir);
396395 err_umem:
397− if (pd->uobject && udata) {
396+ if (!qp->is_kernel) {
398397 if (qp->rumem)
399398 ib_umem_release(qp->rumem);
400399 if (qp->sumem)
@@ -428,7 +427,7 @@ static void pvrdma_free_qp(struct pvrdma_qp *qp) · 27 unmodified lines
428427
429428 pvrdma_unlock_cqs(scq, rcq, &scq_flags, &rcq_flags);
430429
431− if (atomic_dec_and_test(&qp->refcnt))
430+ if (refcount_dec_and_test(&qp->refcnt))
432431 complete(&qp->free);
433432 wait_for_completion(&qp->free);
434433
drivers/infiniband/sw/rdmavt/cq.c+9−6
@@ -56,7 +56,7 @@ · 55 unmodified lines
5656 * rvt_cq_enter - add a new entry to the completion queue
5757 * @cq: completion queue
5858 * @entry: work completion entry to add
59− * @sig: true if @entry is solicited
59+ * @solicited: true if @entry is solicited
6060 *
6161 * This may be called with qp->s_lock held.
6262 */
@@ -101,8 +101,7 @@ void rvt_cq_enter(struct rvt_cq *cq, struct ib_wc *entry, bool solicited) · 38 unmodified lines
101101 wc->uqueue[head].opcode = entry->opcode;
102102 wc->uqueue[head].vendor_err = entry->vendor_err;
103103 wc->uqueue[head].byte_len = entry->byte_len;
104− wc->uqueue[head].ex.imm_data =
105− (__u32 __force)entry->ex.imm_data;
104+ wc->uqueue[head].ex.imm_data = entry->ex.imm_data;
106105 wc->uqueue[head].qp_num = entry->qp->qp_num;
107106 wc->uqueue[head].src_qp = entry->src_qp;
108107 wc->uqueue[head].wc_flags = entry->wc_flags;
@@ -198,7 +197,7 @@ struct ib_cq *rvt_create_cq(struct ib_device *ibdev, · 89 unmodified lines
198197 return ERR_PTR(-EINVAL);
199198
200199 /* Allocate the completion queue structure. */
201− cq = kzalloc(sizeof(*cq), GFP_KERNEL);
200+ cq = kzalloc_node(sizeof(*cq), GFP_KERNEL, rdi->dparms.node);
202201 if (!cq)
203202 return ERR_PTR(-ENOMEM);
204203
@@ -214,7 +213,9 @@ struct ib_cq *rvt_create_cq(struct ib_device *ibdev, · 9 unmodified lines
214213 sz += sizeof(struct ib_uverbs_wc) * (entries + 1);
215214 else
216215 sz += sizeof(struct ib_wc) * (entries + 1);
217− wc = vmalloc_user(sz);
216+ wc = udata ?
217+ vmalloc_user(sz) :
218+ vzalloc_node(sz, rdi->dparms.node);
218219 if (!wc) {
219220 ret = ERR_PTR(-ENOMEM);
220221 goto bail_cq;
@@ -369,7 +370,9 @@ int rvt_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata) · 148 unmodified lines
369370 sz += sizeof(struct ib_uverbs_wc) * (cqe + 1);
370371 else
371372 sz += sizeof(struct ib_wc) * (cqe + 1);
372− wc = vmalloc_user(sz);
373+ wc = udata ?
374+ vmalloc_user(sz) :
375+ vzalloc_node(sz, rdi->dparms.node);
373376 if (!wc)
374377 return -ENOMEM;
375378
drivers/infiniband/sw/rdmavt/mcast.c+2−2
@@ -272,7 +272,7 @@ bail: · 271 unmodified lines
272272 /**
273273 * rvt_attach_mcast - attach a qp to a multicast group
274274 * @ibqp: Infiniband qp
275− * @igd: multicast guid
275+ * @gid: multicast guid
276276 * @lid: multicast lid
277277 *
278278 * Return: 0 on success
@@ -335,7 +335,7 @@ bail_mcast: · 56 unmodified lines
335335 /**
336336 * rvt_detach_mcast - remove a qp from a multicast group
337337 * @ibqp: Infiniband qp
338− * @igd: multicast guid
338+ * @gid: multicast guid
339339 * @lid: multicast lid
340340 *
341341 * Return: 0 on success
drivers/infiniband/sw/rdmavt/mr.c+1−1
@@ -768,7 +768,7 @@ bail: · 767 unmodified lines
768768
769769 /**
770770 * rvt_map_phys_fmr - set up a fast memory region
771− * @ibmfr: the fast memory region to set up
771+ * @ibfmr: the fast memory region to set up
772772 * @page_list: the list of pages to associate with the fast memory region
773773 * @list_len: the number of pages to associate with the fast memory region
774774 * @iova: the virtual address of the start of the fast memory region
drivers/infiniband/sw/rdmavt/qp.c+13−14
@@ -269,7 +269,7 @@ no_qp_table: · 268 unmodified lines
269269
270270 /**
271271 * free_all_qps - check for QPs still in use
272− * @qpt: the QP table to empty
272+ * @rdi: rvt device info structure
273273 *
274274 * There should not be any QPs still in use.
275275 * Free memory for table.
@@ -335,9 +335,9 @@ static inline unsigned mk_qpn(struct rvt_qpn_table *qpt, · 59 unmodified lines
335335 /**
336336 * alloc_qpn - Allocate the next available qpn or zero/one for QP type
337337 * IB_QPT_SMI/IB_QPT_GSI
338− *@rdi: rvt device info structure
339− *@qpt: queue pair number table pointer
340− *@port_num: IB port number, 1 based, comes from core
338+ * @rdi: rvt device info structure
339+ * @qpt: queue pair number table pointer
340+ * @port_num: IB port number, 1 based, comes from core
341341 *
342342 * Return: The queue pair number
343343 */
@@ -1650,9 +1650,9 @@ static inline int rvt_qp_valid_operation( · 1306 unmodified lines
16501650
16511651 /**
16521652 * rvt_qp_is_avail - determine queue capacity
1653− * @qp - the qp
1654− * @rdi - the rdmavt device
1655− * @reserved_op - is reserved operation
1653+ * @qp: the qp
1654+ * @rdi: the rdmavt device
1655+ * @reserved_op: is reserved operation
16561656 *
16571657 * This assumes the s_hlock is held but the s_last
16581658 * qp variable is uncontrolled.
@@ -2074,6 +2074,7 @@ void rvt_add_rnr_timer(struct rvt_qp *qp, u32 aeth) · 415 unmodified lines
20742074 lockdep_assert_held(&qp->s_lock);
20752075 qp->s_flags |= RVT_S_WAIT_RNR;
20762076 to = rvt_aeth_to_usec(aeth);
2077+ trace_rvt_rnrnak_add(qp, to);
20772078 hrtimer_start(&qp->s_rnr_timer,
20782079 ns_to_ktime(1000 * to), HRTIMER_MODE_REL);
20792080 }
@@ -2103,17 +2104,14 @@ EXPORT_SYMBOL(rvt_stop_rc_timers); · 23 unmodified lines
21032104 * stop an rnr timer and return if the timer
21042105 * had been pending.
21052106 */
2106−static int rvt_stop_rnr_timer(struct rvt_qp *qp)
2107+static void rvt_stop_rnr_timer(struct rvt_qp *qp)
21072108 {
2108− int rval = 0;
2109−
21102109 lockdep_assert_held(&qp->s_lock);
21112110 /* Remove QP from rnr timer */
21122111 if (qp->s_flags & RVT_S_WAIT_RNR) {
21132112 qp->s_flags &= ~RVT_S_WAIT_RNR;
2114− rval = hrtimer_try_to_cancel(&qp->s_rnr_timer);
2113+ trace_rvt_rnrnak_stop(qp, 0);
21152114 }
2116− return rval;
21172115 }
21182116
21192117 /**
@@ -2166,6 +2164,7 @@ enum hrtimer_restart rvt_rc_rnr_retry(struct hrtimer *t) · 46 unmodified lines
21662164
21672165 spin_lock_irqsave(&qp->s_lock, flags);
21682166 rvt_stop_rnr_timer(qp);
2167+ trace_rvt_rnrnak_timeout(qp, 0);
21692168 rdi->driver_f.schedule_send(qp);
21702169 spin_unlock_irqrestore(&qp->s_lock, flags);
21712170 return HRTIMER_NORESTART;
@@ -2174,8 +2173,8 @@ EXPORT_SYMBOL(rvt_rc_rnr_retry); · 2 unmodified lines
21742173
21752174 /**
21762175 * rvt_qp_iter_init - initial for QP iteration
2177− * @rdi - rvt devinfo
2178− * @v - u64 value
2176+ * @rdi: rvt devinfo
2177+ * @v: u64 value
21792178 *
21802179 * This returns an iterator suitable for iterating QPs
21812180 * in the system.
drivers/infiniband/sw/rdmavt/srq.c+9−7
@@ -90,7 +90,7 @@ struct ib_srq *rvt_create_srq(struct ib_pd *ibpd, · 89 unmodified lines
9090 srq_init_attr->attr.max_wr > dev->dparms.props.max_srq_wr)
9191 return ERR_PTR(-EINVAL);
9292
93− srq = kmalloc(sizeof(*srq), GFP_KERNEL);
93+ srq = kzalloc_node(sizeof(*srq), GFP_KERNEL, dev->dparms.node);
9494 if (!srq)
9595 return ERR_PTR(-ENOMEM);
9696
@@ -101,7 +101,10 @@ struct ib_srq *rvt_create_srq(struct ib_pd *ibpd, · 4 unmodified lines
101101 srq->rq.max_sge = srq_init_attr->attr.max_sge;
102102 sz = sizeof(struct ib_sge) * srq->rq.max_sge +
103103 sizeof(struct rvt_rwqe);
104− srq->rq.wq = vmalloc_user(sizeof(struct rvt_rwq) + srq->rq.size * sz);
104+ srq->rq.wq = udata ?
105+ vmalloc_user(sizeof(struct rvt_rwq) + srq->rq.size * sz) :
106+ vzalloc_node(sizeof(struct rvt_rwq) + srq->rq.size * sz,
107+ dev->dparms.node);
105108 if (!srq->rq.wq) {
106109 ret = ERR_PTR(-ENOMEM);
107110 goto bail_srq;
@@ -129,16 +132,12 @@ struct ib_srq *rvt_create_srq(struct ib_pd *ibpd, · 21 unmodified lines
129132 ret = ERR_PTR(err);
130133 goto bail_ip;
131134 }
132− } else {
133− srq->ip = NULL;
134135 }
135136
136137 /*
137138 * ib_create_srq() will initialize srq->ibsrq.
138139 */
139140 spin_lock_init(&srq->rq.lock);
140− srq->rq.wq->head = 0;
141− srq->rq.wq->tail = 0;
142141 srq->limit = srq_init_attr->attr.srq_limit;
143142
144143 spin_lock(&dev->n_srqs_lock);
@@ -200,7 +199,10 @@ int rvt_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr, · 55 unmodified lines
200199 sz = sizeof(struct rvt_rwqe) +
201200 srq->rq.max_sge * sizeof(struct ib_sge);
202201 size = attr->max_wr + 1;
203− wq = vmalloc_user(sizeof(struct rvt_rwq) + size * sz);
202+ wq = udata ?
203+ vmalloc_user(sizeof(struct rvt_rwq) + size * sz) :
204+ vzalloc_node(sizeof(struct rvt_rwq) + size * sz,
205+ dev->dparms.node);
204206 if (!wq)
205207 return -ENOMEM;
206208
drivers/infiniband/sw/rdmavt/trace.h+2−2
@@ -45,8 +45,8 @@ · 44 unmodified lines
4545 *
4646 */
4747
48−#define RDI_DEV_ENTRY(rdi) __string(dev, rdi->driver_f.get_card_name(rdi))
49−#define RDI_DEV_ASSIGN(rdi) __assign_str(dev, rdi->driver_f.get_card_name(rdi))
48+#define RDI_DEV_ENTRY(rdi) __string(dev, rvt_get_ibdev_name(rdi))
49+#define RDI_DEV_ASSIGN(rdi) __assign_str(dev, rvt_get_ibdev_name(rdi))
5050
5151 #include "trace_rvt.h"
5252 #include "trace_qp.h"
drivers/infiniband/sw/rdmavt/trace_qp.h+42−0
@@ -85,6 +85,48 @@ DEFINE_EVENT(rvt_qphash_template, rvt_qpremove, · 84 unmodified lines
8585 TP_PROTO(struct rvt_qp *qp, u32 bucket),
8686 TP_ARGS(qp, bucket));
8787
88+DECLARE_EVENT_CLASS(
89+ rvt_rnrnak_template,
90+ TP_PROTO(struct rvt_qp *qp, u32 to),
91+ TP_ARGS(qp, to),
92+ TP_STRUCT__entry(
93+ RDI_DEV_ENTRY(ib_to_rvt(qp->ibqp.device))
94+ __field(u32, qpn)
95+ __field(void *, hrtimer)
96+ __field(u32, s_flags)
97+ __field(u32, to)
98+ ),
99+ TP_fast_assign(
100+ RDI_DEV_ASSIGN(ib_to_rvt(qp->ibqp.device))
101+ __entry->qpn = qp->ibqp.qp_num;
102+ __entry->hrtimer = &qp->s_rnr_timer;
103+ __entry->s_flags = qp->s_flags;
104+ __entry->to = to;
105+ ),
106+ TP_printk(
107+ "[%s] qpn 0x%x hrtimer 0x%p s_flags 0x%x timeout %u us",
108+ __get_str(dev),
109+ __entry->qpn,
110+ __entry->hrtimer,
111+ __entry->s_flags,
112+ __entry->to
113+ )
114+);
115+
116+DEFINE_EVENT(
117+ rvt_rnrnak_template, rvt_rnrnak_add,
118+ TP_PROTO(struct rvt_qp *qp, u32 to),
119+ TP_ARGS(qp, to));
120+
121+DEFINE_EVENT(
122+ rvt_rnrnak_template, rvt_rnrnak_timeout,
123+ TP_PROTO(struct rvt_qp *qp, u32 to),
124+ TP_ARGS(qp, to));
125+
126+DEFINE_EVENT(
127+ rvt_rnrnak_template, rvt_rnrnak_stop,
128+ TP_PROTO(struct rvt_qp *qp, u32 to),
129+ TP_ARGS(qp, to));
88130
89131 #endif /* __RVT_TRACE_QP_H */
90132
drivers/infiniband/sw/rdmavt/vt.c+5−5
@@ -224,7 +224,8 @@ static int rvt_modify_port(struct ib_device *ibdev, u8 port_num, · 223 unmodified lines
224224 * rvt_query_pkey - Return a pkey from the table at a given index
225225 * @ibdev: Verbs IB dev
226226 * @port_num: Port number, 1 based from ib core
227− * @intex: Index into pkey table
227+ * @index: Index into pkey table
228+ * @pkey: returned pkey from the port pkey table
228229 *
229230 * Return: 0 on failure pkey otherwise
230231 */
@@ -255,7 +256,7 @@ static int rvt_query_pkey(struct ib_device *ibdev, u8 port_num, u16 index, · 24 unmodified lines
255256 * rvt_query_gid - Return a gid from the table
256257 * @ibdev: Verbs IB dev
257258 * @port_num: Port number, 1 based from ib core
258− * @index: = Index in table
259+ * @guid_index: Index in table
259260 * @gid: Gid to return
260261 *
261262 * Return: 0 on success
@@ -297,8 +298,8 @@ static inline struct rvt_ucontext *to_iucontext(struct ib_ucontext · 35 unmodified lines
297298
298299 /**
299300 * rvt_alloc_ucontext - Allocate a user context
300− * @ibdev: Vers IB dev
301− * @data: User data allocated
301+ * @ibdev: Verbs IB dev
302+ * @udata: User data allocated
302303 */
303304 static struct ib_ucontext *rvt_alloc_ucontext(struct ib_device *ibdev,
304305 struct ib_udata *udata)
@@ -413,7 +414,6 @@ static noinline int check_support(struct rvt_dev_info *rdi, int verb) · 108 unmodified lines
413414 * required for rdmavt to function.
414415 */
415416 if ((!rdi->driver_f.port_callback) ||
416− (!rdi->driver_f.get_card_name) ||
417417 (!rdi->driver_f.get_pci_dev))
418418 return -EINVAL;
419419 break;
drivers/infiniband/sw/rdmavt/vt.h+3−3
@@ -63,19 +63,19 @@ · 62 unmodified lines
6363
6464 #define rvt_pr_info(rdi, fmt, ...) \
6565 __rvt_pr_info(rdi->driver_f.get_pci_dev(rdi), \
66− rdi->driver_f.get_card_name(rdi), \
66+ rvt_get_ibdev_name(rdi), \
6767 fmt, \
6868 ##__VA_ARGS__)
6969
7070 #define rvt_pr_warn(rdi, fmt, ...) \
7171 __rvt_pr_warn(rdi->driver_f.get_pci_dev(rdi), \
72− rdi->driver_f.get_card_name(rdi), \
72+ rvt_get_ibdev_name(rdi), \
7373 fmt, \
7474 ##__VA_ARGS__)
7575
7676 #define rvt_pr_err(rdi, fmt, ...) \
7777 __rvt_pr_err(rdi->driver_f.get_pci_dev(rdi), \
78− rdi->driver_f.get_card_name(rdi), \
78+ rvt_get_ibdev_name(rdi), \
7979 fmt, \
8080 ##__VA_ARGS__)
8181
drivers/infiniband/sw/rxe/Kconfig+2−2
@@ -1,8 +1,8 @@
11 config RDMA_RXE
22 tristate "Software RDMA over Ethernet (RoCE) driver"
33 depends on INET && PCI && INFINIBAND
4− depends on NET_UDP_TUNNEL
5− depends on CRYPTO_CRC32
4+ select NET_UDP_TUNNEL
5+ select CRYPTO_CRC32
66 select DMA_VIRT_OPS
77 ---help---
88 This driver implements the InfiniBand RDMA transport over
drivers/infiniband/sw/rxe/rxe.c+0−6
@@ -77,12 +77,6 @@ void rxe_release(struct kref *kref) · 76 unmodified lines
7777 ib_dealloc_device(&rxe->ib_dev);
7878 }
7979
80−void rxe_dev_put(struct rxe_dev *rxe)
81−{
82− kref_put(&rxe->ref_cnt, rxe_release);
83−}
84−EXPORT_SYMBOL_GPL(rxe_dev_put);
85−
8680 /* initialize rxe device parameters */
8781 static int rxe_init_device_param(struct rxe_dev *rxe)
8882 {
drivers/infiniband/sw/rxe/rxe.h+5−1
@@ -57,6 +57,7 @@ · 56 unmodified lines
5757 #include "rxe_hdr.h"
5858 #include "rxe_param.h"
5959 #include "rxe_verbs.h"
60+#include "rxe_loc.h"
6061
6162 #define RXE_UVERBS_ABI_VERSION (1)
6263
@@ -95,7 +96,10 @@ void rxe_remove_all(void); · 32 unmodified lines
9596
9697 int rxe_rcv(struct sk_buff *skb);
9798
98−void rxe_dev_put(struct rxe_dev *rxe);
99+static inline void rxe_dev_put(struct rxe_dev *rxe)
100+{
101+ kref_put(&rxe->ref_cnt, rxe_release);
102+}
99103 struct rxe_dev *net_to_rxe(struct net_device *ndev);
100104 struct rxe_dev *get_rxe_by_name(const char *name);
101105
drivers/infiniband/sw/rxe/rxe_loc.h+0−1
@@ -237,7 +237,6 @@ int rxe_srq_from_attr(struct rxe_dev *rxe, struct rxe_srq *srq, · 236 unmodified lines
237237
238238 void rxe_release(struct kref *kref);
239239
240−void rxe_drain_req_pkts(struct rxe_qp *qp, bool notify);
241240 int rxe_completer(void *arg);
242241 int rxe_requester(void *arg);
243242 int rxe_responder(void *arg);
drivers/infiniband/sw/rxe/rxe_net.c+6−12
@@ -82,7 +82,7 @@ struct rxe_dev *get_rxe_by_name(const char *name) · 81 unmodified lines
8282 }
8383
8484
85−struct rxe_recv_sockets recv_sockets;
85+static struct rxe_recv_sockets recv_sockets;
8686
8787 struct device *rxe_dma_device(struct rxe_dev *rxe)
8888 {
@@ -452,31 +452,26 @@ static void rxe_skb_tx_dtor(struct sk_buff *skb) · 363 unmodified lines
452452
453453 int rxe_send(struct rxe_dev *rxe, struct rxe_pkt_info *pkt, struct sk_buff *skb)
454454 {
455− struct sk_buff *nskb;
456455 struct rxe_av *av;
457456 int err;
458457
459458 av = rxe_get_av(pkt);
460459
461− nskb = skb_clone(skb, GFP_ATOMIC);
462− if (!nskb)
463− return -ENOMEM;
464−
465− nskb->destructor = rxe_skb_tx_dtor;
466− nskb->sk = pkt->qp->sk->sk;
460+ skb->destructor = rxe_skb_tx_dtor;
461+ skb->sk = pkt->qp->sk->sk;
467462
468463 rxe_add_ref(pkt->qp);
469464 atomic_inc(&pkt->qp->skb_out);
470465
471466 if (av->network_type == RDMA_NETWORK_IPV4) {
472− err = ip_local_out(dev_net(skb_dst(skb)->dev), nskb->sk, nskb);
467+ err = ip_local_out(dev_net(skb_dst(skb)->dev), skb->sk, skb);
473468 } else if (av->network_type == RDMA_NETWORK_IPV6) {
474− err = ip6_local_out(dev_net(skb_dst(skb)->dev), nskb->sk, nskb);
469+ err = ip6_local_out(dev_net(skb_dst(skb)->dev), skb->sk, skb);
475470 } else {
476471 pr_err("Unknown layer 3 protocol: %d\n", av->network_type);
477472 atomic_dec(&pkt->qp->skb_out);
478473 rxe_drop_ref(pkt->qp);
479− kfree_skb(nskb);
474+ kfree_skb(skb);
480475 return -EINVAL;
481476 }
482477
@@ -485,7 +480,6 @@ int rxe_send(struct rxe_dev *rxe, struct rxe_pkt_info *pkt, struct sk_buff *skb) · 2 unmodified lines
485480 return -EAGAIN;
486481 }
487482
488− kfree_skb(skb);
489483 return 0;
490484 }
491485
drivers/infiniband/sw/rxe/rxe_net.h+0−1
@@ -43,7 +43,6 @@ struct rxe_recv_sockets { · 42 unmodified lines
4343 struct socket *sk6;
4444 };
4545
46−extern struct rxe_recv_sockets recv_sockets;
4746 extern struct notifier_block rxe_net_notifier;
4847 void rxe_release_udp_tunnel(struct socket *sk);
4948
drivers/infiniband/sw/rxe/rxe_qp.c+10−2
@@ -824,9 +824,9 @@ void rxe_qp_destroy(struct rxe_qp *qp) · 823 unmodified lines
824824 }
825825
826826 /* called when the last reference to the qp is dropped */
827−void rxe_qp_cleanup(struct rxe_pool_entry *arg)
827+static void rxe_qp_do_cleanup(struct work_struct *work)
828828 {
829− struct rxe_qp *qp = container_of(arg, typeof(*qp), pelem);
829+ struct rxe_qp *qp = container_of(work, typeof(*qp), cleanup_work.work);
830830
831831 rxe_drop_all_mcast_groups(qp);
832832
@@ -859,3 +859,11 @@ void rxe_qp_cleanup(struct rxe_pool_entry *arg) · 26 unmodified lines
859859 kernel_sock_shutdown(qp->sk, SHUT_RDWR);
860860 sock_release(qp->sk);
861861 }
862+
863+/* called when the last reference to the qp is dropped */
864+void rxe_qp_cleanup(struct rxe_pool_entry *arg)
865+{
866+ struct rxe_qp *qp = container_of(arg, typeof(*qp), pelem);
867+
868+ execute_in_process_context(rxe_qp_do_cleanup, &qp->cleanup_work);
869+}
drivers/infiniband/sw/rxe/rxe_recv.c+1−2
@@ -336,7 +336,6 @@ static int rxe_match_dgid(struct rxe_dev *rxe, struct sk_buff *skb) · 335 unmodified lines
336336 {
337337 union ib_gid dgid;
338338 union ib_gid *pdgid;
339− u16 index;
340339
341340 if (skb->protocol == htons(ETH_P_IP)) {
342341 ipv6_addr_set_v4mapped(ip_hdr(skb)->daddr,
@@ -348,7 +347,7 @@ static int rxe_match_dgid(struct rxe_dev *rxe, struct sk_buff *skb) · 5 unmodified lines
348347
349348 return ib_find_cached_gid_by_port(&rxe->ib_dev, pdgid,
350349 IB_GID_TYPE_ROCE_UDP_ENCAP,
351− 1, rxe->ndev, &index);
350+ 1, rxe->ndev, NULL);
352351 }
353352
354353 /* rxe_rcv is called from the interface driver */
drivers/infiniband/sw/rxe/rxe_req.c+1−8
@@ -594,15 +594,8 @@ int rxe_requester(void *arg) · 593 unmodified lines
594594 rxe_add_ref(qp);
595595
596596 next_wqe:
597− if (unlikely(!qp->valid)) {
598− rxe_drain_req_pkts(qp, true);
597+ if (unlikely(!qp->valid || qp->req.state == QP_STATE_ERROR))
599598 goto exit;
600− }
601−
602− if (unlikely(qp->req.state == QP_STATE_ERROR)) {
603− rxe_drain_req_pkts(qp, true);
604− goto exit;
605− }
606599
607600 if (unlikely(qp->req.state == QP_STATE_RESET)) {
608601 qp->req.wqe_index = consumer_index(qp->sq.queue);
drivers/infiniband/sw/rxe/rxe_resp.c+2−3
@@ -863,8 +863,7 @@ static enum resp_states do_complete(struct rxe_qp *qp, · 862 unmodified lines
863863
864864 if (pkt->mask & RXE_IMMDT_MASK) {
865865 uwc->wc_flags |= IB_WC_WITH_IMM;
866− uwc->ex.imm_data =
867− (__u32 __force)immdt_imm(pkt);
866+ uwc->ex.imm_data = immdt_imm(pkt);
868867 }
869868
870869 if (pkt->mask & RXE_IETH_MASK) {
@@ -1210,7 +1209,7 @@ static enum resp_states do_class_d1e_error(struct rxe_qp *qp) · 339 unmodified lines
12101209 }
12111210 }
12121211
1213−void rxe_drain_req_pkts(struct rxe_qp *qp, bool notify)
1212+static void rxe_drain_req_pkts(struct rxe_qp *qp, bool notify)
12141213 {
12151214 struct sk_buff *skb;
12161215
drivers/infiniband/sw/rxe/rxe_verbs.c+2−0
@@ -814,6 +814,8 @@ static int rxe_post_send_kernel(struct rxe_qp *qp, struct ib_send_wr *wr, · 813 unmodified lines
814814 (queue_count(qp->sq.queue) > 1);
815815
816816 rxe_run_task(&qp->req.task, must_sched);
817+ if (unlikely(qp->req.state == QP_STATE_ERROR))
818+ rxe_run_task(&qp->comp.task, 1);
817819
818820 return err;
819821 }
drivers/infiniband/sw/rxe/rxe_verbs.h+3−0
@@ -35,6 +35,7 @@ · 34 unmodified lines
3535 #define RXE_VERBS_H
3636
3737 #include <linux/interrupt.h>
38+#include <linux/workqueue.h>
3839 #include <rdma/rdma_user_rxe.h>
3940 #include "rxe_pool.h"
4041 #include "rxe_task.h"
@@ -281,6 +282,8 @@ struct rxe_qp { · 240 unmodified lines
281282 struct timer_list rnr_nak_timer;
282283
283284 spinlock_t state_lock; /* guard requester and completer */
285+
286+ struct execute_work cleanup_work;
284287 };
285288
286289 enum rxe_mem_state {
drivers/infiniband/ulp/ipoib/ipoib_cm.c+10−8
@@ -766,12 +766,14 @@ void ipoib_cm_send(struct net_device *dev, struct sk_buff *skb, struct ipoib_cm_ · 765 unmodified lines
766766 skb_orphan(skb);
767767 skb_dst_drop(skb);
768768
769− if (netif_queue_stopped(dev))
770− if (ib_req_notify_cq(priv->send_cq, IB_CQ_NEXT_COMP |
771− IB_CQ_REPORT_MISSED_EVENTS)) {
769+ if (netif_queue_stopped(dev)) {
770+ rc = ib_req_notify_cq(priv->send_cq, IB_CQ_NEXT_COMP |
771+ IB_CQ_REPORT_MISSED_EVENTS);
772+ if (unlikely(rc < 0))
772773 ipoib_warn(priv, "IPoIB/CM:request notify on send CQ failed\n");
774+ else if (rc)
773775 napi_schedule(&priv->send_napi);
774− }
776+ }
775777
776778 rc = post_send(priv, tx, tx->tx_head & (ipoib_sendq_size - 1), tx_req);
777779 if (unlikely(rc)) {
@@ -876,7 +878,7 @@ int ipoib_cm_dev_open(struct net_device *dev) · 98 unmodified lines
876878
877879 priv->cm.id = ib_create_cm_id(priv->ca, ipoib_cm_rx_handler, dev);
878880 if (IS_ERR(priv->cm.id)) {
879− printk(KERN_WARNING "%s: failed to create CM ID\n", priv->ca->name);
881+ pr_warn("%s: failed to create CM ID\n", priv->ca->name);
880882 ret = PTR_ERR(priv->cm.id);
881883 goto err_cm;
882884 }
@@ -884,8 +886,8 @@ int ipoib_cm_dev_open(struct net_device *dev) · 1 unmodified line
884886 ret = ib_cm_listen(priv->cm.id, cpu_to_be64(IPOIB_CM_IETF_ID | priv->qp->qp_num),
885887 0);
886888 if (ret) {
887− printk(KERN_WARNING "%s: failed to listen on ID 0x%llx\n", priv->ca->name,
888− IPOIB_CM_IETF_ID | priv->qp->qp_num);
889+ pr_warn("%s: failed to listen on ID 0x%llx\n", priv->ca->name,
890+ IPOIB_CM_IETF_ID | priv->qp->qp_num);
889891 goto err_listen;
890892 }
891893
@@ -1562,7 +1564,7 @@ static void ipoib_cm_create_srq(struct net_device *dev, int max_sge) · 670 unmodified lines
15621564 priv->cm.srq = ib_create_srq(priv->pd, &srq_init_attr);
15631565 if (IS_ERR(priv->cm.srq)) {
15641566 if (PTR_ERR(priv->cm.srq) != -ENOSYS)
1565− printk(KERN_WARNING "%s: failed to allocate SRQ, error %ld\n",
1567+ pr_warn("%s: failed to allocate SRQ, error %ld\n",
15661568 priv->ca->name, PTR_ERR(priv->cm.srq));
15671569 priv->cm.srq = NULL;
15681570 return;
drivers/infiniband/ulp/ipoib/ipoib_ib.c+2−3
@@ -644,7 +644,7 @@ int ipoib_send(struct net_device *dev, struct sk_buff *skb, · 643 unmodified lines
644644
645645 if (netif_queue_stopped(dev))
646646 if (ib_req_notify_cq(priv->send_cq, IB_CQ_NEXT_COMP |
647− IB_CQ_REPORT_MISSED_EVENTS))
647+ IB_CQ_REPORT_MISSED_EVENTS) < 0)
648648 ipoib_warn(priv, "request notify on send CQ failed\n");
649649
650650 rc = post_send(priv, priv->tx_head & (ipoib_sendq_size - 1),
@@ -1085,8 +1085,7 @@ static bool ipoib_dev_addr_changed_valid(struct ipoib_dev_priv *priv) · 434 unmodified lines
10851085
10861086 netif_addr_unlock_bh(priv->dev);
10871087
1088− err = ib_find_gid(priv->ca, &search_gid, IB_GID_TYPE_IB,
1089− priv->dev, &port, &index);
1088+ err = ib_find_gid(priv->ca, &search_gid, priv->dev, &port, &index);
10901089
10911090 netif_addr_lock_bh(priv->dev);
10921091
drivers/infiniband/ulp/ipoib/ipoib_main.c+67−31
@@ -768,13 +768,30 @@ static void path_rec_completion(int status, · 767 unmodified lines
768768 if (!status) {
769769 struct rdma_ah_attr av;
770770
771− if (!ib_init_ah_from_path(priv->ca, priv->port, pathrec, &av))
771+ if (!ib_init_ah_attr_from_path(priv->ca, priv->port,
772+ pathrec, &av))
772773 ah = ipoib_create_ah(dev, priv->pd, &av);
773774 }
774775
775776 spin_lock_irqsave(&priv->lock, flags);
776777
777778 if (!IS_ERR_OR_NULL(ah)) {
779+ /*
780+ * pathrec.dgid is used as the database key from the LLADDR,
781+ * it must remain unchanged even if the SA returns a different
782+ * GID to use in the AH.
783+ */
784+ if (memcmp(pathrec->dgid.raw, path->pathrec.dgid.raw,
785+ sizeof(union ib_gid))) {
786+ ipoib_dbg(
787+ priv,
788+ "%s got PathRec for gid %pI6 while asked for %pI6\n",
789+ dev->name, pathrec->dgid.raw,
790+ path->pathrec.dgid.raw);
791+ memcpy(pathrec->dgid.raw, path->pathrec.dgid.raw,
792+ sizeof(union ib_gid));
793+ }
794+
778795 path->pathrec = *pathrec;
779796
780797 old_ah = path->ah;
@@ -840,6 +857,23 @@ static void path_rec_completion(int status, · 59 unmodified lines
840857 }
841858 }
842859
860+static void init_path_rec(struct ipoib_dev_priv *priv, struct ipoib_path *path,
861+ void *gid)
862+{
863+ path->dev = priv->dev;
864+
865+ if (rdma_cap_opa_ah(priv->ca, priv->port))
866+ path->pathrec.rec_type = SA_PATH_REC_TYPE_OPA;
867+ else
868+ path->pathrec.rec_type = SA_PATH_REC_TYPE_IB;
869+
870+ memcpy(path->pathrec.dgid.raw, gid, sizeof(union ib_gid));
871+ path->pathrec.sgid = priv->local_gid;
872+ path->pathrec.pkey = cpu_to_be16(priv->pkey);
873+ path->pathrec.numb_path = 1;
874+ path->pathrec.traffic_class = priv->broadcast->mcmember.traffic_class;
875+}
876+
843877 static struct ipoib_path *path_rec_create(struct net_device *dev, void *gid)
844878 {
845879 struct ipoib_dev_priv *priv = ipoib_priv(dev);
@@ -852,21 +886,11 @@ static struct ipoib_path *path_rec_create(struct net_device *dev, void *gid) · 6 unmodified lines
852886 if (!path)
853887 return NULL;
854888
855− path->dev = dev;
856−
857889 skb_queue_head_init(&path->queue);
858890
859891 INIT_LIST_HEAD(&path->neigh_list);
860892
861− if (rdma_cap_opa_ah(priv->ca, priv->port))
862− path->pathrec.rec_type = SA_PATH_REC_TYPE_OPA;
863− else
864− path->pathrec.rec_type = SA_PATH_REC_TYPE_IB;
865− memcpy(path->pathrec.dgid.raw, gid, sizeof (union ib_gid));
866− path->pathrec.sgid = priv->local_gid;
867− path->pathrec.pkey = cpu_to_be16(priv->pkey);
868− path->pathrec.numb_path = 1;
869− path->pathrec.traffic_class = priv->broadcast->mcmember.traffic_class;
893+ init_path_rec(priv, path, gid);
870894
871895 return path;
872896 }
@@ -1005,6 +1029,10 @@ static void unicast_arp_send(struct sk_buff *skb, struct net_device *dev, · 132 unmodified lines
10051029
10061030 spin_lock_irqsave(&priv->lock, flags);
10071031
1032+ /* no broadcast means that all paths are (going to be) not valid */
1033+ if (!priv->broadcast)
1034+ goto drop_and_unlock;
1035+
10081036 path = __path_find(dev, phdr->hwaddr + 4);
10091037 if (!path || !path->valid) {
10101038 int new_path = 0;
@@ -1014,6 +1042,10 @@ static void unicast_arp_send(struct sk_buff *skb, struct net_device *dev, · 3 unmodified lines
10141042 new_path = 1;
10151043 }
10161044 if (path) {
1045+ if (!new_path)
1046+ /* make sure there is no changes in the existing path record */
1047+ init_path_rec(priv, path, phdr->hwaddr + 4);
1048+
10171049 if (skb_queue_len(&path->queue) < IPOIB_MAX_PATH_REC_QUEUE) {
10181050 push_pseudo_header(skb, phdr->hwaddr);
10191051 __skb_queue_tail(&path->queue, skb);
@@ -1030,8 +1062,7 @@ static void unicast_arp_send(struct sk_buff *skb, struct net_device *dev, · 10 unmodified lines
10301062 } else
10311063 __path_add(dev, path);
10321064 } else {
1033− ++dev->stats.tx_dropped;
1034− dev_kfree_skb_any(skb);
1065+ goto drop_and_unlock;
10351066 }
10361067
10371068 spin_unlock_irqrestore(&priv->lock, flags);
@@ -1051,11 +1082,16 @@ static void unicast_arp_send(struct sk_buff *skb, struct net_device *dev, · 13 unmodified lines
10511082 push_pseudo_header(skb, phdr->hwaddr);
10521083 __skb_queue_tail(&path->queue, skb);
10531084 } else {
1054− ++dev->stats.tx_dropped;
1055− dev_kfree_skb_any(skb);
1085+ goto drop_and_unlock;
10561086 }
10571087
10581088 spin_unlock_irqrestore(&priv->lock, flags);
1089+ return;
1090+
1091+drop_and_unlock:
1092+ ++dev->stats.tx_dropped;
1093+ dev_kfree_skb_any(skb);
1094+ spin_unlock_irqrestore(&priv->lock, flags);
10591095 }
10601096
10611097 static int ipoib_start_xmit(struct sk_buff *skb, struct net_device *dev)
@@ -1674,8 +1710,8 @@ static int ipoib_dev_init_default(struct net_device *dev) · 612 unmodified lines
16741710
16751711 priv->tx_ring = vzalloc(ipoib_sendq_size * sizeof *priv->tx_ring);
16761712 if (!priv->tx_ring) {
1677− printk(KERN_WARNING "%s: failed to allocate TX ring (%d entries)\n",
1678− priv->ca->name, ipoib_sendq_size);
1713+ pr_warn("%s: failed to allocate TX ring (%d entries)\n",
1714+ priv->ca->name, ipoib_sendq_size);
16791715 goto out_rx_ring_cleanup;
16801716 }
16811717
@@ -2207,16 +2243,17 @@ static struct net_device *ipoib_add_port(const char *format, · 525 unmodified lines
22072243 int result = -ENOMEM;
22082244
22092245 priv = ipoib_intf_alloc(hca, port, format);
2210− if (!priv)
2246+ if (!priv) {
2247+ pr_warn("%s, %d: ipoib_intf_alloc failed\n", hca->name, port);
22112248 goto alloc_mem_failed;
2249+ }
22122250
22132251 SET_NETDEV_DEV(priv->dev, hca->dev.parent);
22142252 priv->dev->dev_id = port - 1;
22152253
22162254 result = ib_query_port(hca, port, &attr);
22172255 if (result) {
2218− printk(KERN_WARNING "%s: ib_query_port %d failed\n",
2219− hca->name, port);
2256+ pr_warn("%s: ib_query_port %d failed\n", hca->name, port);
22202257 goto device_init_failed;
22212258 }
22222259
@@ -2231,8 +2268,8 @@ static struct net_device *ipoib_add_port(const char *format, · 8 unmodified lines
22312268
22322269 result = ib_query_pkey(hca, port, 0, &priv->pkey);
22332270 if (result) {
2234− printk(KERN_WARNING "%s: ib_query_pkey port %d failed (ret = %d)\n",
2235− hca->name, port, result);
2271+ pr_warn("%s: ib_query_pkey port %d failed (ret = %d)\n",
2272+ hca->name, port, result);
22362273 goto device_init_failed;
22372274 }
22382275
@@ -2249,8 +2286,8 @@ static struct net_device *ipoib_add_port(const char *format, · 10 unmodified lines
22492286
22502287 result = ib_query_gid(hca, port, 0, &priv->local_gid, NULL);
22512288 if (result) {
2252− printk(KERN_WARNING "%s: ib_query_gid port %d failed (ret = %d)\n",
2253− hca->name, port, result);
2289+ pr_warn("%s: ib_query_gid port %d failed (ret = %d)\n",
2290+ hca->name, port, result);
22542291 goto device_init_failed;
22552292 }
22562293
@@ -2260,8 +2297,8 @@ static struct net_device *ipoib_add_port(const char *format, · 3 unmodified lines
22602297
22612298 result = ipoib_dev_init(priv->dev, hca, port);
22622299 if (result) {
2263− printk(KERN_WARNING "%s: failed to initialize port %d (ret = %d)\n",
2264− hca->name, port, result);
2300+ pr_warn("%s: failed to initialize port %d (ret = %d)\n",
2301+ hca->name, port, result);
22652302 goto device_init_failed;
22662303 }
22672304
@@ -2271,8 +2308,8 @@ static struct net_device *ipoib_add_port(const char *format, · 3 unmodified lines
22712308
22722309 result = register_netdev(priv->dev);
22732310 if (result) {
2274− printk(KERN_WARNING "%s: couldn't register ipoib port %d; error %d\n",
2275− hca->name, port, result);
2311+ pr_warn("%s: couldn't register ipoib port %d; error %d\n",
2312+ hca->name, port, result);
22762313 goto register_failed;
22772314 }
22782315
@@ -2337,8 +2374,7 @@ static void ipoib_add_one(struct ib_device *device) · 58 unmodified lines
23372374 }
23382375
23392376 if (!count) {
2340− pr_err("Failed to init port, removing it\n");
2341− ipoib_remove_one(device, dev_list);
2377+ kfree(dev_list);
23422378 return;
23432379 }
23442380
drivers/infiniband/ulp/ipoib/ipoib_verbs.c+3−3
@@ -178,7 +178,7 @@ int ipoib_transport_dev_init(struct net_device *dev, struct ib_device *ca) · 177 unmodified lines
178178 priv->recv_cq = ib_create_cq(priv->ca, ipoib_ib_rx_completion, NULL,
179179 priv, &cq_attr);
180180 if (IS_ERR(priv->recv_cq)) {
181− printk(KERN_WARNING "%s: failed to create receive CQ\n", ca->name);
181+ pr_warn("%s: failed to create receive CQ\n", ca->name);
182182 goto out_cm_dev_cleanup;
183183 }
184184
@@ -187,7 +187,7 @@ int ipoib_transport_dev_init(struct net_device *dev, struct ib_device *ca) · 2 unmodified lines
187187 priv->send_cq = ib_create_cq(priv->ca, ipoib_ib_tx_completion, NULL,
188188 priv, &cq_attr);
189189 if (IS_ERR(priv->send_cq)) {
190− printk(KERN_WARNING "%s: failed to create send CQ\n", ca->name);
190+ pr_warn("%s: failed to create send CQ\n", ca->name);
191191 goto out_free_recv_cq;
192192 }
193193
@@ -208,7 +208,7 @@ int ipoib_transport_dev_init(struct net_device *dev, struct ib_device *ca) · 14 unmodified lines
208208
209209 priv->qp = ib_create_qp(priv->pd, &init_attr);
210210 if (IS_ERR(priv->qp)) {
211− printk(KERN_WARNING "%s: failed to create QP\n", ca->name);
211+ pr_warn("%s: failed to create QP\n", ca->name);
212212 goto out_free_send_cq;
213213 }
214214
drivers/infiniband/ulp/iser/iser_initiator.c+6−10
@@ -142,8 +142,7 @@ iser_prepare_write_cmd(struct iscsi_task *task, · 141 unmodified lines
142142 hdr->write_va = cpu_to_be64(mem_reg->sge.addr + unsol_sz);
143143 }
144144
145− iser_dbg("Cmd itt:%d, WRITE tags, RKEY:%#.4X "
146− "VA:%#llX + unsol:%d\n",
145+ iser_dbg("Cmd itt:%d, WRITE tags, RKEY:%#.4X VA:%#llX + unsol:%d\n",
147146 task->itt, mem_reg->rkey,
148147 (unsigned long long)mem_reg->sge.addr, unsol_sz);
149148 }
@@ -436,7 +435,7 @@ int iser_send_data_out(struct iscsi_conn *conn, · 286 unmodified lines
436435 {
437436 struct iser_conn *iser_conn = conn->dd_data;
438437 struct iscsi_iser_task *iser_task = task->dd_data;
439− struct iser_tx_desc *tx_desc = NULL;
438+ struct iser_tx_desc *tx_desc;
440439 struct iser_mem_reg *mem_reg;
441440 unsigned long buf_offset;
442441 unsigned long data_seg_len;
@@ -452,10 +451,8 @@ int iser_send_data_out(struct iscsi_conn *conn, · 9 unmodified lines
452451 __func__,(int)itt,(int)data_seg_len,(int)buf_offset);
453452
454453 tx_desc = kmem_cache_zalloc(ig.desc_cache, GFP_ATOMIC);
455− if (tx_desc == NULL) {
456− iser_err("Failed to alloc desc for post dataout\n");
454+ if (!tx_desc)
457455 return -ENOMEM;
458− }
459456
460457 tx_desc->type = ISCSI_TX_DATAOUT;
461458 tx_desc->cqe.done = iser_dataout_comp;
@@ -475,8 +472,7 @@ int iser_send_data_out(struct iscsi_conn *conn, · 13 unmodified lines
475472 tx_desc->num_sge = 2;
476473
477474 if (buf_offset + data_seg_len > iser_task->data[ISER_DIR_OUT].data_len) {
478− iser_err("Offset:%ld & DSL:%ld in Data-Out "
479− "inconsistent with total len:%ld, itt:%d\n",
475+ iser_err("Offset:%ld & DSL:%ld in Data-Out inconsistent with total len:%ld, itt:%d\n",
480476 buf_offset, data_seg_len,
481477 iser_task->data[ISER_DIR_OUT].data_len, itt);
482478 err = -EINVAL;
@@ -614,8 +610,8 @@ iser_check_remote_inv(struct iser_conn *iser_conn, · 131 unmodified lines
614610 iser_conn, rkey);
615611
616612 if (unlikely(!iser_conn->snd_w_inv)) {
617− iser_err("conn %p: unexpected remote invalidation, "
618− "terminating connection\n", iser_conn);
613+ iser_err("conn %p: unexpected remote invalidation, terminating connection\n",
614+ iser_conn);
619615 return -EPROTO;
620616 }
621617
drivers/infiniband/ulp/isert/ib_isert.c+7−0
@@ -2124,6 +2124,9 @@ isert_rdma_rw_ctx_post(struct isert_cmd *cmd, struct isert_conn *conn, · 2123 unmodified lines
21242124 u32 rkey, offset;
21252125 int ret;
21262126
2127+ if (cmd->ctx_init_done)
2128+ goto rdma_ctx_post;
2129+
21272130 if (dir == DMA_FROM_DEVICE) {
21282131 addr = cmd->write_va;
21292132 rkey = cmd->write_stag;
@@ -2151,11 +2154,15 @@ isert_rdma_rw_ctx_post(struct isert_cmd *cmd, struct isert_conn *conn, · 21 unmodified lines
21512154 se_cmd->t_data_sg, se_cmd->t_data_nents,
21522155 offset, addr, rkey, dir);
21532156 }
2157+
21542158 if (ret < 0) {
21552159 isert_err("Cmd: %p failed to prepare RDMA res\n", cmd);
21562160 return ret;
21572161 }
21582162
2163+ cmd->ctx_init_done = true;
2164+
2165+rdma_ctx_post:
21592166 ret = rdma_rw_ctx_post(&cmd->rw, conn->qp, port_num, cqe, chain_wr);
21602167 if (ret < 0)
21612168 isert_err("Cmd: %p failed to post RDMA res\n", cmd);
drivers/infiniband/ulp/isert/ib_isert.h+1−0
@@ -126,6 +126,7 @@ struct isert_cmd { · 125 unmodified lines
126126 struct rdma_rw_ctx rw;
127127 struct work_struct comp_work;
128128 struct scatterlist sg;
129+ bool ctx_init_done;
129130 };
130131
131132 static inline struct isert_cmd *tx_desc_to_cmd(struct iser_tx_desc *desc)
drivers/infiniband/ulp/opa_vnic/opa_vnic_vema.c+1−1
@@ -710,7 +710,7 @@ vema_get_port(struct opa_vnic_ctrl_port *cport, u8 port_num) · 709 unmodified lines
710710
711711 /**
712712 * opa_vnic_vema_send_trap -- This function sends a trap to the EM
713− * @cport: pointer to vnic control port
713+ * @adapter: pointer to vnic adapter
714714 * @data: pointer to trap data filled by calling function
715715 * @lid: issuers lid (encap_slid from vesw_port_info)
716716 *
drivers/infiniband/ulp/srp/ib_srp.c+624−171
@@ -41,6 +41,7 @@ · 40 unmodified lines
4141 #include <linux/random.h>
4242 #include <linux/jiffies.h>
4343 #include <linux/lockdep.h>
44+#include <linux/inet.h>
4445 #include <rdma/ib_cache.h>
4546
4647 #include <linux/atomic.h>
@@ -144,7 +145,9 @@ static void srp_remove_one(struct ib_device *device, void *client_data); · 97 unmodified lines
144145 static void srp_recv_done(struct ib_cq *cq, struct ib_wc *wc);
145146 static void srp_handle_qp_err(struct ib_cq *cq, struct ib_wc *wc,
146147 const char *opname);
147−static int srp_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event);
148+static int srp_ib_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event);
149+static int srp_rdma_cm_handler(struct rdma_cm_id *cm_id,
150+ struct rdma_cm_event *event);
148151
149152 static struct scsi_transport_template *ib_srp_transport_template;
150153 static struct workqueue_struct *srp_remove_wq;
@@ -265,8 +268,8 @@ static void srp_qp_event(struct ib_event *event, void *context) · 114 unmodified lines
265268 ib_event_msg(event->event), event->event);
266269 }
267270
268−static int srp_init_qp(struct srp_target_port *target,
269− struct ib_qp *qp)
271+static int srp_init_ib_qp(struct srp_target_port *target,
272+ struct ib_qp *qp)
270273 {
271274 struct ib_qp_attr *attr;
272275 int ret;
@@ -277,7 +280,7 @@ static int srp_init_qp(struct srp_target_port *target, · 4 unmodified lines
277280
278281 ret = ib_find_cached_pkey(target->srp_host->srp_dev->dev,
279282 target->srp_host->port,
280− be16_to_cpu(target->pkey),
283+ be16_to_cpu(target->ib_cm.pkey),
281284 &attr->pkey_index);
282285 if (ret)
283286 goto out;
@@ -298,32 +301,110 @@ out: · 14 unmodified lines
298301 return ret;
299302 }
300303
301−static int srp_new_cm_id(struct srp_rdma_ch *ch)
304+static int srp_new_ib_cm_id(struct srp_rdma_ch *ch)
302305 {
303306 struct srp_target_port *target = ch->target;
304307 struct ib_cm_id *new_cm_id;
305308
306309 new_cm_id = ib_create_cm_id(target->srp_host->srp_dev->dev,
307− srp_cm_handler, ch);
310+ srp_ib_cm_handler, ch);
308311 if (IS_ERR(new_cm_id))
309312 return PTR_ERR(new_cm_id);
310313
311− if (ch->cm_id)
312− ib_destroy_cm_id(ch->cm_id);
313− ch->cm_id = new_cm_id;
314+ if (ch->ib_cm.cm_id)
315+ ib_destroy_cm_id(ch->ib_cm.cm_id);
316+ ch->ib_cm.cm_id = new_cm_id;
314317 if (rdma_cap_opa_ah(target->srp_host->srp_dev->dev,
315318 target->srp_host->port))
316− ch->path.rec_type = SA_PATH_REC_TYPE_OPA;
319+ ch->ib_cm.path.rec_type = SA_PATH_REC_TYPE_OPA;
317320 else
318− ch->path.rec_type = SA_PATH_REC_TYPE_IB;
319− ch->path.sgid = target->sgid;
320− ch->path.dgid = target->orig_dgid;
321− ch->path.pkey = target->pkey;
322− ch->path.service_id = target->service_id;
321+ ch->ib_cm.path.rec_type = SA_PATH_REC_TYPE_IB;
322+ ch->ib_cm.path.sgid = target->sgid;
323+ ch->ib_cm.path.dgid = target->ib_cm.orig_dgid;
324+ ch->ib_cm.path.pkey = target->ib_cm.pkey;
325+ ch->ib_cm.path.service_id = target->ib_cm.service_id;
323326
324327 return 0;
325328 }
326329
330+static const char *inet_ntop(const void *sa, char *dst, unsigned int size)
331+{
332+ switch (((struct sockaddr *)sa)->sa_family) {
333+ case AF_INET:
334+ snprintf(dst, size, "%pI4",
335+ &((struct sockaddr_in *)sa)->sin_addr);
336+ break;
337+ case AF_INET6:
338+ snprintf(dst, size, "%pI6",
339+ &((struct sockaddr_in6 *)sa)->sin6_addr);
340+ break;
341+ default:
342+ snprintf(dst, size, "???");
343+ break;
344+ }
345+ return dst;
346+}
347+
348+static int srp_new_rdma_cm_id(struct srp_rdma_ch *ch)
349+{
350+ struct srp_target_port *target = ch->target;
351+ struct rdma_cm_id *new_cm_id;
352+ char src_addr[64], dst_addr[64];
353+ int ret;
354+
355+ new_cm_id = rdma_create_id(target->net, srp_rdma_cm_handler, ch,
356+ RDMA_PS_TCP, IB_QPT_RC);
357+ if (IS_ERR(new_cm_id)) {
358+ ret = PTR_ERR(new_cm_id);
359+ new_cm_id = NULL;
360+ goto out;
361+ }
362+
363+ init_completion(&ch->done);
364+ ret = rdma_resolve_addr(new_cm_id, target->rdma_cm.src_specified ?
365+ (struct sockaddr *)&target->rdma_cm.src : NULL,
366+ (struct sockaddr *)&target->rdma_cm.dst,
367+ SRP_PATH_REC_TIMEOUT_MS);
368+ if (ret) {
369+ pr_err("No route available from %s to %s (%d)\n",
370+ target->rdma_cm.src_specified ?
371+ inet_ntop(&target->rdma_cm.src, src_addr,
372+ sizeof(src_addr)) : "(any)",
373+ inet_ntop(&target->rdma_cm.dst, dst_addr,
374+ sizeof(dst_addr)),
375+ ret);
376+ goto out;
377+ }
378+ ret = wait_for_completion_interruptible(&ch->done);
379+ if (ret < 0)
380+ goto out;
381+
382+ ret = ch->status;
383+ if (ret) {
384+ pr_err("Resolving address %s failed (%d)\n",
385+ inet_ntop(&target->rdma_cm.dst, dst_addr,
386+ sizeof(dst_addr)),
387+ ret);
388+ goto out;
389+ }
390+
391+ swap(ch->rdma_cm.cm_id, new_cm_id);
392+
393+out:
394+ if (new_cm_id)
395+ rdma_destroy_id(new_cm_id);
396+
397+ return ret;
398+}
399+
400+static int srp_new_cm_id(struct srp_rdma_ch *ch)
401+{
402+ struct srp_target_port *target = ch->target;
403+
404+ return target->using_rdma_cm ? srp_new_rdma_cm_id(ch) :
405+ srp_new_ib_cm_id(ch);
406+}
407+
327408 static struct ib_fmr_pool *srp_alloc_fmr_pool(struct srp_target_port *target)
328409 {
329410 struct srp_device *dev = target->srp_host->srp_dev;
@@ -521,16 +602,25 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch) · 191 unmodified lines
521602 init_attr->send_cq = send_cq;
522603 init_attr->recv_cq = recv_cq;
523604
524− qp = ib_create_qp(dev->pd, init_attr);
525− if (IS_ERR(qp)) {
526− ret = PTR_ERR(qp);
605+ if (target->using_rdma_cm) {
606+ ret = rdma_create_qp(ch->rdma_cm.cm_id, dev->pd, init_attr);
607+ qp = ch->rdma_cm.cm_id->qp;
608+ } else {
609+ qp = ib_create_qp(dev->pd, init_attr);
610+ if (!IS_ERR(qp)) {
611+ ret = srp_init_ib_qp(target, qp);
612+ if (ret)
613+ ib_destroy_qp(qp);
614+ } else {
615+ ret = PTR_ERR(qp);
616+ }
617+ }
618+ if (ret) {
619+ pr_err("QP creation failed for dev %s: %d\n",
620+ dev_name(&dev->dev->dev), ret);
527621 goto err_send_cq;
528622 }
529623
530− ret = srp_init_qp(target, qp);
531− if (ret)
532− goto err_qp;
533−
534624 if (dev->use_fast_reg) {
535625 fr_pool = srp_alloc_fr_pool(target);
536626 if (IS_ERR(fr_pool)) {
@@ -574,7 +664,10 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch) · 37 unmodified lines
574664 return 0;
575665
576666 err_qp:
577− ib_destroy_qp(qp);
667+ if (target->using_rdma_cm)
668+ rdma_destroy_qp(ch->rdma_cm.cm_id);
669+ else
670+ ib_destroy_qp(qp);
578671
579672 err_send_cq:
580673 ib_free_cq(send_cq);
@@ -600,9 +693,16 @@ static void srp_free_ch_ib(struct srp_target_port *target, · 19 unmodified lines
600693 if (!ch->target)
601694 return;
602695
603− if (ch->cm_id) {
604− ib_destroy_cm_id(ch->cm_id);
605− ch->cm_id = NULL;
696+ if (target->using_rdma_cm) {
697+ if (ch->rdma_cm.cm_id) {
698+ rdma_destroy_id(ch->rdma_cm.cm_id);
699+ ch->rdma_cm.cm_id = NULL;
700+ }
701+ } else {
702+ if (ch->ib_cm.cm_id) {
703+ ib_destroy_cm_id(ch->ib_cm.cm_id);
704+ ch->ib_cm.cm_id = NULL;
705+ }
606706 }
607707
608708 /* If srp_new_cm_id() succeeded but srp_create_ch_ib() not, return. */
@@ -658,16 +758,16 @@ static void srp_path_rec_completion(int status, · 49 unmodified lines
658758 shost_printk(KERN_ERR, target->scsi_host,
659759 PFX "Got failed path rec status %d\n", status);
660760 else
661− ch->path = *pathrec;
761+ ch->ib_cm.path = *pathrec;
662762 complete(&ch->done);
663763 }
664764
665−static int srp_lookup_path(struct srp_rdma_ch *ch)
765+static int srp_ib_lookup_path(struct srp_rdma_ch *ch)
666766 {
667767 struct srp_target_port *target = ch->target;
668768 int ret = -ENODEV;
669769
670− ch->path.numb_path = 1;
770+ ch->ib_cm.path.numb_path = 1;
671771
672772 init_completion(&ch->done);
673773
@@ -678,10 +778,10 @@ static int srp_lookup_path(struct srp_rdma_ch *ch) · 4 unmodified lines
678778 if (!scsi_host_get(target->scsi_host))
679779 goto out;
680780
681− ch->path_query_id = ib_sa_path_rec_get(&srp_sa_client,
781+ ch->ib_cm.path_query_id = ib_sa_path_rec_get(&srp_sa_client,
682782 target->srp_host->srp_dev->dev,
683783 target->srp_host->port,
684− &ch->path,
784+ &ch->ib_cm.path,
685785 IB_SA_PATH_REC_SERVICE_ID |
686786 IB_SA_PATH_REC_DGID |
687787 IB_SA_PATH_REC_SGID |
@@ -690,8 +790,8 @@ static int srp_lookup_path(struct srp_rdma_ch *ch) · 2 unmodified lines
690790 SRP_PATH_REC_TIMEOUT_MS,
691791 GFP_KERNEL,
692792 srp_path_rec_completion,
693− ch, &ch->path_query);
694− ret = ch->path_query_id;
793+ ch, &ch->ib_cm.path_query);
794+ ret = ch->ib_cm.path_query_id;
695795 if (ret < 0)
696796 goto put;
697797
@@ -702,7 +802,10 @@ static int srp_lookup_path(struct srp_rdma_ch *ch) · 4 unmodified lines
702802 ret = ch->status;
703803 if (ret < 0)
704804 shost_printk(KERN_WARNING, target->scsi_host,
705− PFX "Path record query failed\n");
805+ PFX "Path record query failed: sgid %pI6, dgid %pI6, pkey %#04x, service_id %#16llx\n",
806+ ch->ib_cm.path.sgid.raw, ch->ib_cm.path.dgid.raw,
807+ be16_to_cpu(target->ib_cm.pkey),
808+ be64_to_cpu(target->ib_cm.service_id));
706809
707810 put:
708811 scsi_host_put(target->scsi_host);
@@ -711,6 +814,34 @@ out: · 2 unmodified lines
711814 return ret;
712815 }
713816
817+static int srp_rdma_lookup_path(struct srp_rdma_ch *ch)
818+{
819+ struct srp_target_port *target = ch->target;
820+ int ret;
821+
822+ init_completion(&ch->done);
823+
824+ ret = rdma_resolve_route(ch->rdma_cm.cm_id, SRP_PATH_REC_TIMEOUT_MS);
825+ if (ret)
826+ return ret;
827+
828+ wait_for_completion_interruptible(&ch->done);
829+
830+ if (ch->status != 0)
831+ shost_printk(KERN_WARNING, target->scsi_host,
832+ PFX "Path resolution failed\n");
833+
834+ return ch->status;
835+}
836+
837+static int srp_lookup_path(struct srp_rdma_ch *ch)
838+{
839+ struct srp_target_port *target = ch->target;
840+
841+ return target->using_rdma_cm ? srp_rdma_lookup_path(ch) :
842+ srp_ib_lookup_path(ch);
843+}
844+
714845 static u8 srp_get_subnet_timeout(struct srp_host *host)
715846 {
716847 struct ib_port_attr attr;
@@ -732,48 +863,76 @@ static int srp_send_req(struct srp_rdma_ch *ch, bool multich) · 15 unmodified lines
732863 {
733864 struct srp_target_port *target = ch->target;
734865 struct {
735− struct ib_cm_req_param param;
736− struct srp_login_req priv;
866+ struct rdma_conn_param rdma_param;
867+ struct srp_login_req_rdma rdma_req;
868+ struct ib_cm_req_param ib_param;
869+ struct srp_login_req ib_req;
737870 } *req = NULL;
871+ char *ipi, *tpi;
738872 int status;
739− u8 subnet_timeout;
740−
741− subnet_timeout = srp_get_subnet_timeout(target->srp_host);
742873
743874 req = kzalloc(sizeof *req, GFP_KERNEL);
744875 if (!req)
745876 return -ENOMEM;
746877
747− req->param.primary_path = &ch->path;
748− req->param.alternate_path = NULL;
749− req->param.service_id = target->service_id;
750− req->param.qp_num = ch->qp->qp_num;
751− req->param.qp_type = ch->qp->qp_type;
752− req->param.private_data = &req->priv;
753− req->param.private_data_len = sizeof req->priv;
754− req->param.flow_control = 1;
755−
756− get_random_bytes(&req->param.starting_psn, 4);
757− req->param.starting_psn &= 0xffffff;
878+ req->ib_param.flow_control = 1;
879+ req->ib_param.retry_count = target->tl_retry_count;
758880
759881 /*
760882 * Pick some arbitrary defaults here; we could make these
761883 * module parameters if anyone cared about setting them.
762884 */
763− req->param.responder_resources = 4;
764− req->param.remote_cm_response_timeout = subnet_timeout + 2;
765− req->param.local_cm_response_timeout = subnet_timeout + 2;
766− req->param.retry_count = target->tl_retry_count;
767− req->param.rnr_retry_count = 7;
768− req->param.max_cm_retries = 15;
769−
770− req->priv.opcode = SRP_LOGIN_REQ;
771− req->priv.tag = 0;
772− req->priv.req_it_iu_len = cpu_to_be32(target->max_iu_len);
773− req->priv.req_buf_fmt = cpu_to_be16(SRP_BUF_FORMAT_DIRECT |
885+ req->ib_param.responder_resources = 4;
886+ req->ib_param.rnr_retry_count = 7;
887+ req->ib_param.max_cm_retries = 15;
888+
889+ req->ib_req.opcode = SRP_LOGIN_REQ;
890+ req->ib_req.tag = 0;
891+ req->ib_req.req_it_iu_len = cpu_to_be32(target->max_iu_len);
892+ req->ib_req.req_buf_fmt = cpu_to_be16(SRP_BUF_FORMAT_DIRECT |
774893 SRP_BUF_FORMAT_INDIRECT);
775− req->priv.req_flags = (multich ? SRP_MULTICHAN_MULTI :
776− SRP_MULTICHAN_SINGLE);
894+ req->ib_req.req_flags = (multich ? SRP_MULTICHAN_MULTI :
895+ SRP_MULTICHAN_SINGLE);
896+
897+ if (target->using_rdma_cm) {
898+ req->rdma_param.flow_control = req->ib_param.flow_control;
899+ req->rdma_param.responder_resources =
900+ req->ib_param.responder_resources;
901+ req->rdma_param.initiator_depth = req->ib_param.initiator_depth;
902+ req->rdma_param.retry_count = req->ib_param.retry_count;
903+ req->rdma_param.rnr_retry_count = req->ib_param.rnr_retry_count;
904+ req->rdma_param.private_data = &req->rdma_req;
905+ req->rdma_param.private_data_len = sizeof(req->rdma_req);
906+
907+ req->rdma_req.opcode = req->ib_req.opcode;
908+ req->rdma_req.tag = req->ib_req.tag;
909+ req->rdma_req.req_it_iu_len = req->ib_req.req_it_iu_len;
910+ req->rdma_req.req_buf_fmt = req->ib_req.req_buf_fmt;
911+ req->rdma_req.req_flags = req->ib_req.req_flags;
912+
913+ ipi = req->rdma_req.initiator_port_id;
914+ tpi = req->rdma_req.target_port_id;
915+ } else {
916+ u8 subnet_timeout;
917+
918+ subnet_timeout = srp_get_subnet_timeout(target->srp_host);
919+
920+ req->ib_param.primary_path = &ch->ib_cm.path;
921+ req->ib_param.alternate_path = NULL;
922+ req->ib_param.service_id = target->ib_cm.service_id;
923+ get_random_bytes(&req->ib_param.starting_psn, 4);
924+ req->ib_param.starting_psn &= 0xffffff;
925+ req->ib_param.qp_num = ch->qp->qp_num;
926+ req->ib_param.qp_type = ch->qp->qp_type;
927+ req->ib_param.local_cm_response_timeout = subnet_timeout + 2;
928+ req->ib_param.remote_cm_response_timeout = subnet_timeout + 2;
929+ req->ib_param.private_data = &req->ib_req;
930+ req->ib_param.private_data_len = sizeof(req->ib_req);
931+
932+ ipi = req->ib_req.initiator_port_id;
933+ tpi = req->ib_req.target_port_id;
934+ }
935+
777936 /*
778937 * In the published SRP specification (draft rev. 16a), the
779938 * port identifier format is 8 bytes of ID extension followed
@@ -784,19 +943,15 @@ static int srp_send_req(struct srp_rdma_ch *ch, bool multich) · 4 unmodified lines
784943 * recognized by the I/O Class they report.
785944 */
786945 if (target->io_class == SRP_REV10_IB_IO_CLASS) {
787− memcpy(req->priv.initiator_port_id,
788− &target->sgid.global.interface_id, 8);
789− memcpy(req->priv.initiator_port_id + 8,
790− &target->initiator_ext, 8);
791− memcpy(req->priv.target_port_id, &target->ioc_guid, 8);
792− memcpy(req->priv.target_port_id + 8, &target->id_ext, 8);
946+ memcpy(ipi, &target->sgid.global.interface_id, 8);
947+ memcpy(ipi + 8, &target->initiator_ext, 8);
948+ memcpy(tpi, &target->ioc_guid, 8);
949+ memcpy(tpi + 8, &target->id_ext, 8);
793950 } else {
794− memcpy(req->priv.initiator_port_id,
795− &target->initiator_ext, 8);
796− memcpy(req->priv.initiator_port_id + 8,
797− &target->sgid.global.interface_id, 8);
798− memcpy(req->priv.target_port_id, &target->id_ext, 8);
799− memcpy(req->priv.target_port_id + 8, &target->ioc_guid, 8);
951+ memcpy(ipi, &target->initiator_ext, 8);
952+ memcpy(ipi + 8, &target->sgid.global.interface_id, 8);
953+ memcpy(tpi, &target->id_ext, 8);
954+ memcpy(tpi + 8, &target->ioc_guid, 8);
800955 }
801956
802957 /*
@@ -809,12 +964,14 @@ static int srp_send_req(struct srp_rdma_ch *ch, bool multich) · 6 unmodified lines
809964 PFX "Topspin/Cisco initiator port ID workaround "
810965 "activated for target GUID %016llx\n",
811966 be64_to_cpu(target->ioc_guid));
812− memset(req->priv.initiator_port_id, 0, 8);
813− memcpy(req->priv.initiator_port_id + 8,
814− &target->srp_host->srp_dev->dev->node_guid, 8);
967+ memset(ipi, 0, 8);
968+ memcpy(ipi + 8, &target->srp_host->srp_dev->dev->node_guid, 8);
815969 }
816970
817− status = ib_send_cm_req(ch->cm_id, &req->param);
971+ if (target->using_rdma_cm)
972+ status = rdma_connect(ch->rdma_cm.cm_id, &req->rdma_param);
973+ else
974+ status = ib_send_cm_req(ch->ib_cm.cm_id, &req->ib_param);
818975
819976 kfree(req);
820977
@@ -841,14 +998,23 @@ static bool srp_queue_remove_work(struct srp_target_port *target) · 20 unmodified lines
841998 static void srp_disconnect_target(struct srp_target_port *target)
842999 {
8431000 struct srp_rdma_ch *ch;
844− int i;
1001+ int i, ret;
8451002
8461003 /* XXX should send SRP_I_LOGOUT request */
8471004
8481005 for (i = 0; i < target->ch_count; i++) {
8491006 ch = &target->ch[i];
8501007 ch->connected = false;
851− if (ch->cm_id && ib_send_cm_dreq(ch->cm_id, NULL, 0)) {
1008+ ret = 0;
1009+ if (target->using_rdma_cm) {
1010+ if (ch->rdma_cm.cm_id)
1011+ rdma_disconnect(ch->rdma_cm.cm_id);
1012+ } else {
1013+ if (ch->ib_cm.cm_id)
1014+ ret = ib_send_cm_dreq(ch->ib_cm.cm_id,
1015+ NULL, 0);
1016+ }
1017+ if (ret < 0) {
8521018 shost_printk(KERN_DEBUG, target->scsi_host,
8531019 PFX "Sending CM DREQ failed\n");
8541020 }
@@ -962,6 +1128,7 @@ static void srp_remove_target(struct srp_target_port *target) · 107 unmodified lines
9621128 scsi_remove_host(target->scsi_host);
9631129 srp_stop_rport_timers(target->rport);
9641130 srp_disconnect_target(target);
1131+ kobj_ns_drop(KOBJ_NS_TYPE_NET, target->net);
9651132 for (i = 0; i < target->ch_count; i++) {
9661133 ch = &target->ch[i];
9671134 srp_free_ch_ib(target, ch);
@@ -2349,7 +2516,7 @@ static void srp_cm_rep_handler(struct ib_cm_id *cm_id, · 1381 unmodified lines
23492516 struct srp_target_port *target = ch->target;
23502517 struct ib_qp_attr *qp_attr = NULL;
23512518 int attr_mask = 0;
2352− int ret;
2519+ int ret = 0;
23532520 int i;
23542521
23552522 if (lrsp->opcode == SRP_LOGIN_RSP) {
@@ -2379,40 +2546,42 @@ static void srp_cm_rep_handler(struct ib_cm_id *cm_id, · 23 unmodified lines
23792546 goto error;
23802547 }
23812548
2382− ret = -ENOMEM;
2383− qp_attr = kmalloc(sizeof *qp_attr, GFP_KERNEL);
2384− if (!qp_attr)
2385− goto error;
2386−
2387− qp_attr->qp_state = IB_QPS_RTR;
2388− ret = ib_cm_init_qp_attr(cm_id, qp_attr, &attr_mask);
2389− if (ret)
2390− goto error_free;
2391−
2392− ret = ib_modify_qp(ch->qp, qp_attr, attr_mask);
2393− if (ret)
2394− goto error_free;
2395−
23962549 for (i = 0; i < target->queue_size; i++) {
23972550 struct srp_iu *iu = ch->rx_ring[i];
23982551
23992552 ret = srp_post_recv(ch, iu);
24002553 if (ret)
2401− goto error_free;
2554+ goto error;
24022555 }
24032556
2404− qp_attr->qp_state = IB_QPS_RTS;
2405− ret = ib_cm_init_qp_attr(cm_id, qp_attr, &attr_mask);
2406− if (ret)
2407− goto error_free;
2557+ if (!target->using_rdma_cm) {
2558+ ret = -ENOMEM;
2559+ qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL);
2560+ if (!qp_attr)
2561+ goto error;
24082562
2409− target->rq_tmo_jiffies = srp_compute_rq_tmo(qp_attr, attr_mask);
2563+ qp_attr->qp_state = IB_QPS_RTR;
2564+ ret = ib_cm_init_qp_attr(cm_id, qp_attr, &attr_mask);
2565+ if (ret)
2566+ goto error_free;
24102567
2411− ret = ib_modify_qp(ch->qp, qp_attr, attr_mask);
2412− if (ret)
2413− goto error_free;
2568+ ret = ib_modify_qp(ch->qp, qp_attr, attr_mask);
2569+ if (ret)
2570+ goto error_free;
24142571
2415− ret = ib_send_cm_rtu(cm_id, NULL, 0);
2572+ qp_attr->qp_state = IB_QPS_RTS;
2573+ ret = ib_cm_init_qp_attr(cm_id, qp_attr, &attr_mask);
2574+ if (ret)
2575+ goto error_free;
2576+
2577+ target->rq_tmo_jiffies = srp_compute_rq_tmo(qp_attr, attr_mask);
2578+
2579+ ret = ib_modify_qp(ch->qp, qp_attr, attr_mask);
2580+ if (ret)
2581+ goto error_free;
2582+
2583+ ret = ib_send_cm_rtu(cm_id, NULL, 0);
2584+ }
24162585
24172586 error_free:
24182587 kfree(qp_attr);
@@ -2421,41 +2590,43 @@ error: · 2 unmodified lines
24212590 ch->status = ret;
24222591 }
24232592
2424−static void srp_cm_rej_handler(struct ib_cm_id *cm_id,
2425− struct ib_cm_event *event,
2426− struct srp_rdma_ch *ch)
2593+static void srp_ib_cm_rej_handler(struct ib_cm_id *cm_id,
2594+ struct ib_cm_event *event,
2595+ struct srp_rdma_ch *ch)
24272596 {
24282597 struct srp_target_port *target = ch->target;
24292598 struct Scsi_Host *shost = target->scsi_host;
24302599 struct ib_class_port_info *cpi;
24312600 int opcode;
2601+ u16 dlid;
24322602
24332603 switch (event->param.rej_rcvd.reason) {
24342604 case IB_CM_REJ_PORT_CM_REDIRECT:
24352605 cpi = event->param.rej_rcvd.ari;
2436− sa_path_set_dlid(&ch->path, ntohs(cpi->redirect_lid));
2437− ch->path.pkey = cpi->redirect_pkey;
2606+ dlid = be16_to_cpu(cpi->redirect_lid);
2607+ sa_path_set_dlid(&ch->ib_cm.path, dlid);
2608+ ch->ib_cm.path.pkey = cpi->redirect_pkey;
24382609 cm_id->remote_cm_qpn = be32_to_cpu(cpi->redirect_qp) & 0x00ffffff;
2439− memcpy(ch->path.dgid.raw, cpi->redirect_gid, 16);
2610+ memcpy(ch->ib_cm.path.dgid.raw, cpi->redirect_gid, 16);
24402611
2441− ch->status = sa_path_get_dlid(&ch->path) ?
2442− SRP_DLID_REDIRECT : SRP_PORT_REDIRECT;
2612+ ch->status = dlid ? SRP_DLID_REDIRECT : SRP_PORT_REDIRECT;
24432613 break;
24442614
24452615 case IB_CM_REJ_PORT_REDIRECT:
24462616 if (srp_target_is_topspin(target)) {
2617+ union ib_gid *dgid = &ch->ib_cm.path.dgid;
2618+
24472619 /*
24482620 * Topspin/Cisco SRP gateways incorrectly send
24492621 * reject reason code 25 when they mean 24
24502622 * (port redirect).
24512623 */
2452− memcpy(ch->path.dgid.raw,
2453− event->param.rej_rcvd.ari, 16);
2624+ memcpy(dgid->raw, event->param.rej_rcvd.ari, 16);
24542625
24552626 shost_printk(KERN_DEBUG, shost,
24562627 PFX "Topspin/Cisco redirect to target port GID %016llx%016llx\n",
2457− be64_to_cpu(ch->path.dgid.global.subnet_prefix),
2458− be64_to_cpu(ch->path.dgid.global.interface_id));
2628+ be64_to_cpu(dgid->global.subnet_prefix),
2629+ be64_to_cpu(dgid->global.interface_id));
24592630
24602631 ch->status = SRP_PORT_REDIRECT;
24612632 } else {
@@ -2484,7 +2655,8 @@ static void srp_cm_rej_handler(struct ib_cm_id *cm_id, · 22 unmodified lines
24842655 shost_printk(KERN_WARNING, shost, PFX
24852656 "SRP LOGIN from %pI6 to %pI6 REJECTED, reason 0x%08x\n",
24862657 target->sgid.raw,
2487− target->orig_dgid.raw, reason);
2658+ target->ib_cm.orig_dgid.raw,
2659+ reason);
24882660 } else
24892661 shost_printk(KERN_WARNING, shost,
24902662 " REJ reason: IB_CM_REJ_CONSUMER_DEFINED,"
@@ -2504,7 +2676,7 @@ static void srp_cm_rej_handler(struct ib_cm_id *cm_id, · 13 unmodified lines
25042676 }
25052677 }
25062678
2507−static int srp_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
2679+static int srp_ib_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
25082680 {
25092681 struct srp_rdma_ch *ch = cm_id->context;
25102682 struct srp_target_port *target = ch->target;
@@ -2527,7 +2699,7 @@ static int srp_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) · 16 unmodified lines
25272699 shost_printk(KERN_DEBUG, target->scsi_host, PFX "REJ received\n");
25282700 comp = 1;
25292701
2530− srp_cm_rej_handler(cm_id, event, ch);
2702+ srp_ib_cm_rej_handler(cm_id, event, ch);
25312703 break;
25322704
25332705 case IB_CM_DREQ_RECEIVED:
@@ -2565,6 +2737,135 @@ static int srp_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) · 31 unmodified lines
25652737 return 0;
25662738 }
25672739
2740+static void srp_rdma_cm_rej_handler(struct srp_rdma_ch *ch,
2741+ struct rdma_cm_event *event)
2742+{
2743+ struct srp_target_port *target = ch->target;
2744+ struct Scsi_Host *shost = target->scsi_host;
2745+ int opcode;
2746+
2747+ switch (event->status) {
2748+ case IB_CM_REJ_DUPLICATE_LOCAL_COMM_ID:
2749+ shost_printk(KERN_WARNING, shost,
2750+ " REJ reason: IB_CM_REJ_DUPLICATE_LOCAL_COMM_ID\n");
2751+ ch->status = -ECONNRESET;
2752+ break;
2753+
2754+ case IB_CM_REJ_CONSUMER_DEFINED:
2755+ opcode = *(u8 *) event->param.conn.private_data;
2756+ if (opcode == SRP_LOGIN_REJ) {
2757+ struct srp_login_rej *rej =
2758+ (struct srp_login_rej *)
2759+ event->param.conn.private_data;
2760+ u32 reason = be32_to_cpu(rej->reason);
2761+
2762+ if (reason == SRP_LOGIN_REJ_REQ_IT_IU_LENGTH_TOO_LARGE)
2763+ shost_printk(KERN_WARNING, shost,
2764+ PFX "SRP_LOGIN_REJ: requested max_it_iu_len too large\n");
2765+ else
2766+ shost_printk(KERN_WARNING, shost,
2767+ PFX "SRP LOGIN REJECTED, reason 0x%08x\n", reason);
2768+ } else {
2769+ shost_printk(KERN_WARNING, shost,
2770+ " REJ reason: IB_CM_REJ_CONSUMER_DEFINED, opcode 0x%02x\n",
2771+ opcode);
2772+ }
2773+ ch->status = -ECONNRESET;
2774+ break;
2775+
2776+ case IB_CM_REJ_STALE_CONN:
2777+ shost_printk(KERN_WARNING, shost,
2778+ " REJ reason: stale connection\n");
2779+ ch->status = SRP_STALE_CONN;
2780+ break;
2781+
2782+ default:
2783+ shost_printk(KERN_WARNING, shost, " REJ reason 0x%x\n",
2784+ event->status);
2785+ ch->status = -ECONNRESET;
2786+ break;
2787+ }
2788+}
2789+
2790+static int srp_rdma_cm_handler(struct rdma_cm_id *cm_id,
2791+ struct rdma_cm_event *event)
2792+{
2793+ struct srp_rdma_ch *ch = cm_id->context;
2794+ struct srp_target_port *target = ch->target;
2795+ int comp = 0;
2796+
2797+ switch (event->event) {
2798+ case RDMA_CM_EVENT_ADDR_RESOLVED:
2799+ ch->status = 0;
2800+ comp = 1;
2801+ break;
2802+
2803+ case RDMA_CM_EVENT_ADDR_ERROR:
2804+ ch->status = -ENXIO;
2805+ comp = 1;
2806+ break;
2807+
2808+ case RDMA_CM_EVENT_ROUTE_RESOLVED:
2809+ ch->status = 0;
2810+ comp = 1;
2811+ break;
2812+
2813+ case RDMA_CM_EVENT_ROUTE_ERROR:
2814+ case RDMA_CM_EVENT_UNREACHABLE:
2815+ ch->status = -EHOSTUNREACH;
2816+ comp = 1;
2817+ break;
2818+
2819+ case RDMA_CM_EVENT_CONNECT_ERROR:
2820+ shost_printk(KERN_DEBUG, target->scsi_host,
2821+ PFX "Sending CM REQ failed\n");
2822+ comp = 1;
2823+ ch->status = -ECONNRESET;
2824+ break;
2825+
2826+ case RDMA_CM_EVENT_ESTABLISHED:
2827+ comp = 1;
2828+ srp_cm_rep_handler(NULL, event->param.conn.private_data, ch);
2829+ break;
2830+
2831+ case RDMA_CM_EVENT_REJECTED:
2832+ shost_printk(KERN_DEBUG, target->scsi_host, PFX "REJ received\n");
2833+ comp = 1;
2834+
2835+ srp_rdma_cm_rej_handler(ch, event);
2836+ break;
2837+
2838+ case RDMA_CM_EVENT_DISCONNECTED:
2839+ if (ch->connected) {
2840+ shost_printk(KERN_WARNING, target->scsi_host,
2841+ PFX "received DREQ\n");
2842+ rdma_disconnect(ch->rdma_cm.cm_id);
2843+ comp = 1;
2844+ ch->status = 0;
2845+ queue_work(system_long_wq, &target->tl_err_work);
2846+ }
2847+ break;
2848+
2849+ case RDMA_CM_EVENT_TIMEWAIT_EXIT:
2850+ shost_printk(KERN_ERR, target->scsi_host,
2851+ PFX "connection closed\n");
2852+
2853+ comp = 1;
2854+ ch->status = 0;
2855+ break;
2856+
2857+ default:
2858+ shost_printk(KERN_WARNING, target->scsi_host,
2859+ PFX "Unhandled CM event %d\n", event->event);
2860+ break;
2861+ }
2862+
2863+ if (comp)
2864+ complete(&ch->done);
2865+
2866+ return 0;
2867+}
2868+
25682869 /**
25692870 * srp_change_queue_depth - setting device queue depth
25702871 * @sdev: scsi device struct
@@ -2717,6 +3018,16 @@ static int srp_reset_host(struct scsi_cmnd *scmnd) · 146 unmodified lines
27173018 return srp_reconnect_rport(target->rport) == 0 ? SUCCESS : FAILED;
27183019 }
27193020
3021+static int srp_target_alloc(struct scsi_target *starget)
3022+{
3023+ struct Scsi_Host *shost = dev_to_shost(starget->dev.parent);
3024+ struct srp_target_port *target = host_to_target(shost);
3025+
3026+ if (target->target_can_queue)
3027+ starget->can_queue = target->target_can_queue;
3028+ return 0;
3029+}
3030+
27203031 static int srp_slave_alloc(struct scsi_device *sdev)
27213032 {
27223033 struct Scsi_Host *shost = sdev->host;
@@ -2766,7 +3077,10 @@ static ssize_t show_service_id(struct device *dev, · 43 unmodified lines
27663077 {
27673078 struct srp_target_port *target = host_to_target(class_to_shost(dev));
27683079
2769− return sprintf(buf, "0x%016llx\n", be64_to_cpu(target->service_id));
3080+ if (target->using_rdma_cm)
3081+ return -ENOENT;
3082+ return sprintf(buf, "0x%016llx\n",
3083+ be64_to_cpu(target->ib_cm.service_id));
27703084 }
27713085
27723086 static ssize_t show_pkey(struct device *dev, struct device_attribute *attr,
@@ -2774,7 +3088,9 @@ static ssize_t show_pkey(struct device *dev, struct device_attribute *attr, · 1 unmodified line
27743088 {
27753089 struct srp_target_port *target = host_to_target(class_to_shost(dev));
27763090
2777− return sprintf(buf, "0x%04x\n", be16_to_cpu(target->pkey));
3091+ if (target->using_rdma_cm)
3092+ return -ENOENT;
3093+ return sprintf(buf, "0x%04x\n", be16_to_cpu(target->ib_cm.pkey));
27783094 }
27793095
27803096 static ssize_t show_sgid(struct device *dev, struct device_attribute *attr,
@@ -2791,7 +3107,9 @@ static ssize_t show_dgid(struct device *dev, struct device_attribute *attr, · 10 unmodified lines
27913107 struct srp_target_port *target = host_to_target(class_to_shost(dev));
27923108 struct srp_rdma_ch *ch = &target->ch[0];
27933109
2794− return sprintf(buf, "%pI6\n", ch->path.dgid.raw);
3110+ if (target->using_rdma_cm)
3111+ return -ENOENT;
3112+ return sprintf(buf, "%pI6\n", ch->ib_cm.path.dgid.raw);
27953113 }
27963114
27973115 static ssize_t show_orig_dgid(struct device *dev,
@@ -2799,7 +3117,9 @@ static ssize_t show_orig_dgid(struct device *dev, · 1 unmodified line
27993117 {
28003118 struct srp_target_port *target = host_to_target(class_to_shost(dev));
28013119
2802− return sprintf(buf, "%pI6\n", target->orig_dgid.raw);
3120+ if (target->using_rdma_cm)
3121+ return -ENOENT;
3122+ return sprintf(buf, "%pI6\n", target->ib_cm.orig_dgid.raw);
28033123 }
28043124
28053125 static ssize_t show_req_lim(struct device *dev,
@@ -2921,6 +3241,7 @@ static struct scsi_host_template srp_template = { · 115 unmodified lines
29213241 .module = THIS_MODULE,
29223242 .name = "InfiniBand SRP initiator",
29233243 .proc_name = DRV_NAME,
3244+ .target_alloc = srp_target_alloc,
29243245 .slave_alloc = srp_slave_alloc,
29253246 .slave_configure = srp_slave_configure,
29263247 .info = srp_target_info,
@@ -3044,6 +3365,9 @@ static bool srp_conn_unique(struct srp_host *host, · 117 unmodified lines
30443365 if (t != target &&
30453366 target->id_ext == t->id_ext &&
30463367 target->ioc_guid == t->ioc_guid &&
3368+ (!target->using_rdma_cm ||
3369+ memcmp(&target->rdma_cm.dst, &t->rdma_cm.dst,
3370+ sizeof(target->rdma_cm.dst)) == 0) &&
30473371 target->initiator_ext == t->initiator_ext) {
30483372 ret = false;
30493373 break;
@@ -3060,6 +3384,9 @@ out: · 10 unmodified lines
30603384 *
30613385 * id_ext=<SRP ID ext>,ioc_guid=<SRP IOC GUID>,dgid=<dest GID>,
30623386 * pkey=<P_Key>,service_id=<service ID>
3387+ * or
3388+ * id_ext=<SRP ID ext>,ioc_guid=<SRP IOC GUID>,
3389+ * [src=<IPv4 address>,]dest=<IPv4 address>:<port number>
30633390 *
30643391 * to the add_target sysfs attribute.
30653392 */
@@ -3080,11 +3407,20 @@ enum { · 14 unmodified lines
30803407 SRP_OPT_COMP_VECTOR = 1 << 12,
30813408 SRP_OPT_TL_RETRY_COUNT = 1 << 13,
30823409 SRP_OPT_QUEUE_SIZE = 1 << 14,
3083− SRP_OPT_ALL = (SRP_OPT_ID_EXT |
3084− SRP_OPT_IOC_GUID |
3085− SRP_OPT_DGID |
3086− SRP_OPT_PKEY |
3087− SRP_OPT_SERVICE_ID),
3410+ SRP_OPT_IP_SRC = 1 << 15,
3411+ SRP_OPT_IP_DEST = 1 << 16,
3412+ SRP_OPT_TARGET_CAN_QUEUE= 1 << 17,
3413+};
3414+
3415+static unsigned int srp_opt_mandatory[] = {
3416+ SRP_OPT_ID_EXT |
3417+ SRP_OPT_IOC_GUID |
3418+ SRP_OPT_DGID |
3419+ SRP_OPT_PKEY |
3420+ SRP_OPT_SERVICE_ID,
3421+ SRP_OPT_ID_EXT |
3422+ SRP_OPT_IOC_GUID |
3423+ SRP_OPT_IP_DEST,
30883424 };
30893425
30903426 static const match_table_t srp_opt_tokens = {
@@ -3095,6 +3431,7 @@ static const match_table_t srp_opt_tokens = { · 4 unmodified lines
30953431 { SRP_OPT_SERVICE_ID, "service_id=%s" },
30963432 { SRP_OPT_MAX_SECT, "max_sect=%d" },
30973433 { SRP_OPT_MAX_CMD_PER_LUN, "max_cmd_per_lun=%d" },
3434+ { SRP_OPT_TARGET_CAN_QUEUE, "target_can_queue=%d" },
30983435 { SRP_OPT_IO_CLASS, "io_class=%x" },
30993436 { SRP_OPT_INITIATOR_EXT, "initiator_ext=%s" },
31003437 { SRP_OPT_CMD_SG_ENTRIES, "cmd_sg_entries=%u" },
@@ -3103,15 +3440,33 @@ static const match_table_t srp_opt_tokens = { · 2 unmodified lines
31033440 { SRP_OPT_COMP_VECTOR, "comp_vector=%u" },
31043441 { SRP_OPT_TL_RETRY_COUNT, "tl_retry_count=%u" },
31053442 { SRP_OPT_QUEUE_SIZE, "queue_size=%d" },
3443+ { SRP_OPT_IP_SRC, "src=%s" },
3444+ { SRP_OPT_IP_DEST, "dest=%s" },
31063445 { SRP_OPT_ERR, NULL }
31073446 };
31083447
3109−static int srp_parse_options(const char *buf, struct srp_target_port *target)
3448+static int srp_parse_in(struct net *net, struct sockaddr_storage *sa,
3449+ const char *addr_port_str)
3450+{
3451+ char *addr = kstrdup(addr_port_str, GFP_KERNEL);
3452+ char *port_str = addr;
3453+ int ret;
3454+
3455+ if (!addr)
3456+ return -ENOMEM;
3457+ strsep(&port_str, ":");
3458+ ret = inet_pton_with_scope(net, AF_UNSPEC, addr, port_str, sa);
3459+ kfree(addr);
3460+ return ret;
3461+}
3462+
3463+static int srp_parse_options(struct net *net, const char *buf,
3464+ struct srp_target_port *target)
31103465 {
31113466 char *options, *sep_opt;
31123467 char *p;
3113− char dgid[3];
31143468 substring_t args[MAX_OPT_ARGS];
3469+ unsigned long long ull;
31153470 int opt_mask = 0;
31163471 int token;
31173472 int ret = -EINVAL;
@@ -3136,7 +3491,13 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target) · 18 unmodified lines
31363491 ret = -ENOMEM;
31373492 goto out;
31383493 }
3139− target->id_ext = cpu_to_be64(simple_strtoull(p, NULL, 16));
3494+ ret = kstrtoull(p, 16, &ull);
3495+ if (ret) {
3496+ pr_warn("invalid id_ext parameter '%s'\n", p);
3497+ kfree(p);
3498+ goto out;
3499+ }
3500+ target->id_ext = cpu_to_be64(ull);
31403501 kfree(p);
31413502 break;
31423503
@@ -3146,7 +3507,13 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target) · 3 unmodified lines
31463507 ret = -ENOMEM;
31473508 goto out;
31483509 }
3149− target->ioc_guid = cpu_to_be64(simple_strtoull(p, NULL, 16));
3510+ ret = kstrtoull(p, 16, &ull);
3511+ if (ret) {
3512+ pr_warn("invalid ioc_guid parameter '%s'\n", p);
3513+ kfree(p);
3514+ goto out;
3515+ }
3516+ target->ioc_guid = cpu_to_be64(ull);
31503517 kfree(p);
31513518 break;
31523519
@@ -3162,16 +3529,10 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target) · 9 unmodified lines
31623529 goto out;
31633530 }
31643531
3165− for (i = 0; i < 16; ++i) {
3166− strlcpy(dgid, p + i * 2, sizeof(dgid));
3167− if (sscanf(dgid, "%hhx",
3168− &target->orig_dgid.raw[i]) < 1) {
3169− ret = -EINVAL;
3170− kfree(p);
3171− goto out;
3172− }
3173− }
3532+ ret = hex2bin(target->ib_cm.orig_dgid.raw, p, 16);
31743533 kfree(p);
3534+ if (ret < 0)
3535+ goto out;
31753536 break;
31763537
31773538 case SRP_OPT_PKEY:
@@ -3179,7 +3540,7 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target) · 1 unmodified line
31793540 pr_warn("bad P_Key parameter '%s'\n", p);
31803541 goto out;
31813542 }
3182− target->pkey = cpu_to_be16(token);
3543+ target->ib_cm.pkey = cpu_to_be16(token);
31833544 break;
31843545
31853546 case SRP_OPT_SERVICE_ID:
@@ -3188,7 +3549,45 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target) · 2 unmodified lines
31883549 ret = -ENOMEM;
31893550 goto out;
31903551 }
3191− target->service_id = cpu_to_be64(simple_strtoull(p, NULL, 16));
3552+ ret = kstrtoull(p, 16, &ull);
3553+ if (ret) {
3554+ pr_warn("bad service_id parameter '%s'\n", p);
3555+ kfree(p);
3556+ goto out;
3557+ }
3558+ target->ib_cm.service_id = cpu_to_be64(ull);
3559+ kfree(p);
3560+ break;
3561+
3562+ case SRP_OPT_IP_SRC:
3563+ p = match_strdup(args);
3564+ if (!p) {
3565+ ret = -ENOMEM;
3566+ goto out;
3567+ }
3568+ ret = srp_parse_in(net, &target->rdma_cm.src.ss, p);
3569+ if (ret < 0) {
3570+ pr_warn("bad source parameter '%s'\n", p);
3571+ kfree(p);
3572+ goto out;
3573+ }
3574+ target->rdma_cm.src_specified = true;
3575+ kfree(p);
3576+ break;
3577+
3578+ case SRP_OPT_IP_DEST:
3579+ p = match_strdup(args);
3580+ if (!p) {
3581+ ret = -ENOMEM;
3582+ goto out;
3583+ }
3584+ ret = srp_parse_in(net, &target->rdma_cm.dst.ss, p);
3585+ if (ret < 0) {
3586+ pr_warn("bad dest parameter '%s'\n", p);
3587+ kfree(p);
3588+ goto out;
3589+ }
3590+ target->using_rdma_cm = true;
31923591 kfree(p);
31933592 break;
31943593
@@ -3221,6 +3620,15 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target) · 26 unmodified lines
32213620 target->scsi_host->cmd_per_lun = token;
32223621 break;
32233622
3623+ case SRP_OPT_TARGET_CAN_QUEUE:
3624+ if (match_int(args, &token) || token < 1) {
3625+ pr_warn("bad max target_can_queue parameter '%s'\n",
3626+ p);
3627+ goto out;
3628+ }
3629+ target->target_can_queue = token;
3630+ break;
3631+
32243632 case SRP_OPT_IO_CLASS:
32253633 if (match_hex(args, &token)) {
32263634 pr_warn("bad IO class parameter '%s'\n", p);
@@ -3242,7 +3650,13 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target) · 15 unmodified lines
32423650 ret = -ENOMEM;
32433651 goto out;
32443652 }
3245− target->initiator_ext = cpu_to_be64(simple_strtoull(p, NULL, 16));
3653+ ret = kstrtoull(p, 16, &ull);
3654+ if (ret) {
3655+ pr_warn("bad initiator_ext value '%s'\n", p);
3656+ kfree(p);
3657+ goto out;
3658+ }
3659+ target->initiator_ext = cpu_to_be64(ull);
32463660 kfree(p);
32473661 break;
32483662
@@ -3297,14 +3711,14 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target) · 48 unmodified lines
32973711 }
32983712 }
32993713
3300− if ((opt_mask & SRP_OPT_ALL) == SRP_OPT_ALL)
3301− ret = 0;
3302− else
3303− for (i = 0; i < ARRAY_SIZE(srp_opt_tokens); ++i)
3304− if ((srp_opt_tokens[i].token & SRP_OPT_ALL) &&
3305− !(srp_opt_tokens[i].token & opt_mask))
3306− pr_warn("target creation request is missing parameter '%s'\n",
3307− srp_opt_tokens[i].pattern);
3714+ for (i = 0; i < ARRAY_SIZE(srp_opt_mandatory); i++) {
3715+ if ((opt_mask & srp_opt_mandatory[i]) == srp_opt_mandatory[i]) {
3716+ ret = 0;
3717+ break;
3718+ }
3719+ }
3720+ if (ret)
3721+ pr_warn("target creation request is missing one or more parameters\n");
33083722
33093723 if (target->scsi_host->cmd_per_lun > target->scsi_host->can_queue
33103724 && (opt_mask & SRP_OPT_MAX_CMD_PER_LUN))
@@ -3345,6 +3759,7 @@ static ssize_t srp_create_target(struct device *dev, · 34 unmodified lines
33453759
33463760 target = host_to_target(target_host);
33473761
3762+ target->net = kobj_ns_grab_current(KOBJ_NS_TYPE_NET);
33483763 target->io_class = SRP_REV16A_IB_IO_CLASS;
33493764 target->scsi_host = target_host;
33503765 target->srp_host = host;
@@ -3366,18 +3781,29 @@ static ssize_t srp_create_target(struct device *dev, · 15 unmodified lines
33663781 if (ret < 0)
33673782 goto put;
33683783
3369− ret = srp_parse_options(buf, target);
3784+ ret = srp_parse_options(target->net, buf, target);
33703785 if (ret)
33713786 goto out;
33723787
33733788 target->req_ring_size = target->queue_size - SRP_TSK_MGMT_SQ_SIZE;
33743789
33753790 if (!srp_conn_unique(target->srp_host, target)) {
3376− shost_printk(KERN_INFO, target->scsi_host,
3377− PFX "Already connected to target port with id_ext=%016llx;ioc_guid=%016llx;initiator_ext=%016llx\n",
3378− be64_to_cpu(target->id_ext),
3379− be64_to_cpu(target->ioc_guid),
3380− be64_to_cpu(target->initiator_ext));
3791+ if (target->using_rdma_cm) {
3792+ char dst_addr[64];
3793+
3794+ shost_printk(KERN_INFO, target->scsi_host,
3795+ PFX "Already connected to target port with id_ext=%016llx;ioc_guid=%016llx;dest=%s\n",
3796+ be64_to_cpu(target->id_ext),
3797+ be64_to_cpu(target->ioc_guid),
3798+ inet_ntop(&target->rdma_cm.dst, dst_addr,
3799+ sizeof(dst_addr)));
3800+ } else {
3801+ shost_printk(KERN_INFO, target->scsi_host,
3802+ PFX "Already connected to target port with id_ext=%016llx;ioc_guid=%016llx;initiator_ext=%016llx\n",
3803+ be64_to_cpu(target->id_ext),
3804+ be64_to_cpu(target->ioc_guid),
3805+ be64_to_cpu(target->initiator_ext));
3806+ }
33813807 ret = -EEXIST;
33823808 goto out;
33833809 }
@@ -3478,11 +3904,18 @@ static ssize_t srp_create_target(struct device *dev, · 94 unmodified lines
34783904
34793905 ret = srp_connect_ch(ch, multich);
34803906 if (ret) {
3907+ char dst[64];
3908+
3909+ if (target->using_rdma_cm)
3910+ inet_ntop(&target->rdma_cm.dst, dst,
3911+ sizeof(dst));
3912+ else
3913+ snprintf(dst, sizeof(dst), "%pI6",
3914+ target->ib_cm.orig_dgid.raw);
34813915 shost_printk(KERN_ERR, target->scsi_host,
3482− PFX "Connection %d/%d to %pI6 failed\n",
3916+ PFX "Connection %d/%d to %s failed\n",
34833917 ch_start + cpu_idx,
3484− target->ch_count,
3485− ch->target->orig_dgid.raw);
3918+ target->ch_count, dst);
34863919 if (node_idx == 0 && cpu_idx == 0) {
34873920 goto free_ch;
34883921 } else {
@@ -3507,13 +3940,25 @@ connected: · 18 unmodified lines
35073940 goto err_disconnect;
35083941
35093942 if (target->state != SRP_TARGET_REMOVED) {
3510− shost_printk(KERN_DEBUG, target->scsi_host, PFX
3511− "new target: id_ext %016llx ioc_guid %016llx pkey %04x service_id %016llx sgid %pI6 dgid %pI6\n",
3512− be64_to_cpu(target->id_ext),
3513− be64_to_cpu(target->ioc_guid),
3514− be16_to_cpu(target->pkey),
3515− be64_to_cpu(target->service_id),
3516− target->sgid.raw, target->orig_dgid.raw);
3943+ if (target->using_rdma_cm) {
3944+ char dst[64];
3945+
3946+ inet_ntop(&target->rdma_cm.dst, dst, sizeof(dst));
3947+ shost_printk(KERN_DEBUG, target->scsi_host, PFX
3948+ "new target: id_ext %016llx ioc_guid %016llx sgid %pI6 dest %s\n",
3949+ be64_to_cpu(target->id_ext),
3950+ be64_to_cpu(target->ioc_guid),
3951+ target->sgid.raw, dst);
3952+ } else {
3953+ shost_printk(KERN_DEBUG, target->scsi_host, PFX
3954+ "new target: id_ext %016llx ioc_guid %016llx pkey %04x service_id %016llx sgid %pI6 dgid %pI6\n",
3955+ be64_to_cpu(target->id_ext),
3956+ be64_to_cpu(target->ioc_guid),
3957+ be16_to_cpu(target->ib_cm.pkey),
3958+ be64_to_cpu(target->ib_cm.service_id),
3959+ target->sgid.raw,
3960+ target->ib_cm.orig_dgid.raw);
3961+ }
35173962 }
35183963
35193964 ret = count;
@@ -3523,8 +3968,16 @@ out: · 3 unmodified lines
35233968
35243969 put:
35253970 scsi_host_put(target->scsi_host);
3526− if (ret < 0)
3971+ if (ret < 0) {
3972+ /*
3973+ * If a call to srp_remove_target() has not been scheduled,
3974+ * drop the network namespace reference now that was obtained
3975+ * earlier in this function.
3976+ */
3977+ if (target->state != SRP_TARGET_REMOVED)
3978+ kobj_ns_drop(KOBJ_NS_TYPE_NET, target->net);
35273979 scsi_host_put(target->scsi_host);
3980+ }
35283981
35293982 return ret;
35303983
drivers/infiniband/ulp/srp/ib_srp.h+36−7
@@ -45,6 +45,7 @@ · 44 unmodified lines
4545 #include <rdma/ib_sa.h>
4646 #include <rdma/ib_cm.h>
4747 #include <rdma/ib_fmr_pool.h>
48+#include <rdma/rdma_cm.h>
4849
4950 enum {
5051 SRP_PATH_REC_TIMEOUT_MS = 1000,
@@ -153,11 +154,18 @@ struct srp_rdma_ch { · 102 unmodified lines
153154 struct completion done;
154155 int status;
155156
156− struct sa_path_rec path;
157− struct ib_sa_query *path_query;
158− int path_query_id;
157+ union {
158+ struct ib_cm {
159+ struct sa_path_rec path;
160+ struct ib_sa_query *path_query;
161+ int path_query_id;
162+ struct ib_cm_id *cm_id;
163+ } ib_cm;
164+ struct rdma_cm {
165+ struct rdma_cm_id *cm_id;
166+ } rdma_cm;
167+ };
159168
160− struct ib_cm_id *cm_id;
161169 struct srp_iu **tx_ring;
162170 struct srp_iu **rx_ring;
163171 struct srp_request *req_ring;
@@ -182,6 +190,7 @@ struct srp_target_port { · 18 unmodified lines
182190 /* read only in the hot path */
183191 u32 global_rkey;
184192 struct srp_rdma_ch *ch;
193+ struct net *net;
185194 u32 ch_count;
186195 u32 lkey;
187196 enum srp_target_state state;
@@ -194,7 +203,6 @@ struct srp_target_port { · 6 unmodified lines
194203 union ib_gid sgid;
195204 __be64 id_ext;
196205 __be64 ioc_guid;
197− __be64 service_id;
198206 __be64 initiator_ext;
199207 u16 io_class;
200208 struct srp_host *srp_host;
@@ -203,6 +211,7 @@ struct srp_target_port { · 2 unmodified lines
203211 char target_name[32];
204212 unsigned int scsi_id;
205213 unsigned int sg_tablesize;
214+ unsigned int target_can_queue;
206215 int mr_pool_size;
207216 int mr_per_cmd;
208217 int queue_size;
@@ -210,8 +219,28 @@ struct srp_target_port { · 1 unmodified line
210219 int comp_vector;
211220 int tl_retry_count;
212221
213− union ib_gid orig_dgid;
214− __be16 pkey;
222+ bool using_rdma_cm;
223+
224+ union {
225+ struct {
226+ __be64 service_id;
227+ union ib_gid orig_dgid;
228+ __be16 pkey;
229+ } ib_cm;
230+ struct {
231+ union {
232+ struct sockaddr_in ip4;
233+ struct sockaddr_in6 ip6;
234+ struct sockaddr_storage ss;
235+ } src;
236+ union {
237+ struct sockaddr_in ip4;
238+ struct sockaddr_in6 ip6;
239+ struct sockaddr_storage ss;
240+ } dst;
241+ bool src_specified;
242+ } rdma_cm;
243+ };
215244
216245 u32 rq_tmo_jiffies;
217246
drivers/infiniband/ulp/srpt/ib_srpt.c+585−377
@@ -41,6 +41,7 @@ · 40 unmodified lines
4141 #include <linux/string.h>
4242 #include <linux/delay.h>
4343 #include <linux/atomic.h>
44+#include <rdma/ib_cache.h>
4445 #include <scsi/scsi_proto.h>
4546 #include <scsi/scsi_tcq.h>
4647 #include <target/target_core_base.h>
@@ -120,7 +121,9 @@ static bool srpt_set_ch_state(struct srpt_rdma_ch *ch, enum rdma_ch_state new) · 73 unmodified lines
120121 }
121122
122123 /**
123− * srpt_event_handler() - Asynchronous IB event callback function.
124+ * srpt_event_handler - asynchronous IB event callback function
125+ * @handler: IB event handler registered by ib_register_event_handler().
126+ * @event: Description of the event that occurred.
124127 *
125128 * Callback function called by the InfiniBand core when an asynchronous IB
126129 * event occurs. This callback may occur in interrupt context. See also
@@ -132,6 +135,7 @@ static void srpt_event_handler(struct ib_event_handler *handler, · 5 unmodified lines
132135 {
133136 struct srpt_device *sdev;
134137 struct srpt_port *sport;
138+ u8 port_num;
135139
136140 sdev = ib_get_client_data(event->device, &srpt_client);
137141 if (!sdev || sdev->device != event->device)
@@ -142,10 +146,15 @@ static void srpt_event_handler(struct ib_event_handler *handler, · 4 unmodified lines
142146
143147 switch (event->event) {
144148 case IB_EVENT_PORT_ERR:
145− if (event->element.port_num <= sdev->device->phys_port_cnt) {
146− sport = &sdev->port[event->element.port_num - 1];
149+ port_num = event->element.port_num - 1;
150+ if (port_num < sdev->device->phys_port_cnt) {
151+ sport = &sdev->port[port_num];
147152 sport->lid = 0;
148153 sport->sm_lid = 0;
154+ } else {
155+ WARN(true, "event %d: port_num %d out of range 1..%d\n",
156+ event->event, port_num + 1,
157+ sdev->device->phys_port_cnt);
149158 }
150159 break;
151160 case IB_EVENT_PORT_ACTIVE:
@@ -155,25 +164,31 @@ static void srpt_event_handler(struct ib_event_handler *handler, · 3 unmodified lines
155164 case IB_EVENT_CLIENT_REREGISTER:
156165 case IB_EVENT_GID_CHANGE:
157166 /* Refresh port data asynchronously. */
158− if (event->element.port_num <= sdev->device->phys_port_cnt) {
159− sport = &sdev->port[event->element.port_num - 1];
167+ port_num = event->element.port_num - 1;
168+ if (port_num < sdev->device->phys_port_cnt) {
169+ sport = &sdev->port[port_num];
160170 if (!sport->lid && !sport->sm_lid)
161171 schedule_work(&sport->work);
172+ } else {
173+ WARN(true, "event %d: port_num %d out of range 1..%d\n",
174+ event->event, port_num + 1,
175+ sdev->device->phys_port_cnt);
162176 }
163177 break;
164178 default:
165− pr_err("received unrecognized IB event %d\n",
166− event->event);
179+ pr_err("received unrecognized IB event %d\n", event->event);
167180 break;
168181 }
169182 }
170183
171184 /**
172− * srpt_srq_event() - SRQ event callback function.
185+ * srpt_srq_event - SRQ event callback function
186+ * @event: Description of the event that occurred.
187+ * @ctx: Context pointer specified at SRQ creation time.
173188 */
174189 static void srpt_srq_event(struct ib_event *event, void *ctx)
175190 {
176− pr_info("SRQ event %d\n", event->event);
191+ pr_debug("SRQ event %d\n", event->event);
177192 }
178193
179194 static const char *get_ch_state_name(enum rdma_ch_state s)
@@ -194,16 +209,18 @@ static const char *get_ch_state_name(enum rdma_ch_state s) · 14 unmodified lines
194209 }
195210
196211 /**
197− * srpt_qp_event() - QP event callback function.
212+ * srpt_qp_event - QP event callback function
213+ * @event: Description of the event that occurred.
214+ * @ch: SRPT RDMA channel.
198215 */
199216 static void srpt_qp_event(struct ib_event *event, struct srpt_rdma_ch *ch)
200217 {
201− pr_debug("QP event %d on cm_id=%p sess_name=%s state=%d\n",
202− event->event, ch->cm_id, ch->sess_name, ch->state);
218+ pr_debug("QP event %d on ch=%p sess_name=%s state=%d\n",
219+ event->event, ch, ch->sess_name, ch->state);
203220
204221 switch (event->event) {
205222 case IB_EVENT_COMM_EST:
206− ib_cm_notify(ch->cm_id, event->event);
223+ ib_cm_notify(ch->ib_cm.cm_id, event->event);
207224 break;
208225 case IB_EVENT_QP_LAST_WQE_REACHED:
209226 pr_debug("%s-%d, state %s: received Last WQE event.\n",
@@ -217,8 +234,8 @@ static void srpt_qp_event(struct ib_event *event, struct srpt_rdma_ch *ch) · 7 unmodified lines
217234 }
218235
219236 /**
220− * srpt_set_ioc() - Helper function for initializing an IOUnitInfo structure.
221− *
237+ * srpt_set_ioc - initialize a IOUnitInfo structure
238+ * @c_list: controller list.
222239 * @slot: one-based slot number.
223240 * @value: four-bit value.
224241 *
@@ -241,7 +258,8 @@ static void srpt_set_ioc(u8 *c_list, u32 slot, u8 value) · 16 unmodified lines
241258 }
242259
243260 /**
244− * srpt_get_class_port_info() - Copy ClassPortInfo to a management datagram.
261+ * srpt_get_class_port_info - copy ClassPortInfo to a management datagram
262+ * @mad: Datagram that will be sent as response to DM_ATTR_CLASS_PORT_INFO.
245263 *
246264 * See also section 16.3.3.1 ClassPortInfo in the InfiniBand Architecture
247265 * Specification.
@@ -260,7 +278,8 @@ static void srpt_get_class_port_info(struct ib_dm_mad *mad) · 12 unmodified lines
260278 }
261279
262280 /**
263− * srpt_get_iou() - Write IOUnitInfo to a management datagram.
281+ * srpt_get_iou - write IOUnitInfo to a management datagram
282+ * @mad: Datagram that will be sent as response to DM_ATTR_IOU_INFO.
264283 *
265284 * See also section 16.3.3.3 IOUnitInfo in the InfiniBand Architecture
266285 * Specification. See also section B.7, table B.6 in the SRP r16a document.
@@ -284,7 +303,10 @@ static void srpt_get_iou(struct ib_dm_mad *mad) · 17 unmodified lines
284303 }
285304
286305 /**
287− * srpt_get_ioc() - Write IOControllerprofile to a management datagram.
306+ * srpt_get_ioc - write IOControllerprofile to a management datagram
307+ * @sport: HCA port through which the MAD has been received.
308+ * @slot: Slot number specified in DM_ATTR_IOC_PROFILE query.
309+ * @mad: Datagram that will be sent as response to DM_ATTR_IOC_PROFILE.
288310 *
289311 * See also section 16.3.3.4 IOControllerProfile in the InfiniBand
290312 * Architecture Specification. See also section B.7, table B.7 in the SRP
@@ -314,7 +336,7 @@ static void srpt_get_ioc(struct srpt_port *sport, u32 slot, · 23 unmodified lines
314336 if (sdev->use_srq)
315337 send_queue_depth = sdev->srq_size;
316338 else
317− send_queue_depth = min(SRPT_RQ_SIZE,
339+ send_queue_depth = min(MAX_SRPT_RQ_SIZE,
318340 sdev->device->attrs.max_qp_wr);
319341
320342 memset(iocp, 0, sizeof(*iocp));
@@ -342,7 +364,12 @@ static void srpt_get_ioc(struct srpt_port *sport, u32 slot, · 21 unmodified lines
342364 }
343365
344366 /**
345− * srpt_get_svc_entries() - Write ServiceEntries to a management datagram.
367+ * srpt_get_svc_entries - write ServiceEntries to a management datagram
368+ * @ioc_guid: I/O controller GUID to use in reply.
369+ * @slot: I/O controller number.
370+ * @hi: End of the range of service entries to be specified in the reply.
371+ * @lo: Start of the range of service entries to be specified in the reply..
372+ * @mad: Datagram that will be sent as response to DM_ATTR_SVC_ENTRIES.
346373 *
347374 * See also section 16.3.3.5 ServiceEntries in the InfiniBand Architecture
348375 * Specification. See also section B.7, table B.8 in the SRP r16a document.
@@ -379,8 +406,8 @@ static void srpt_get_svc_entries(u64 ioc_guid, · 30 unmodified lines
379406 }
380407
381408 /**
382− * srpt_mgmt_method_get() - Process a received management datagram.
383− * @sp: source port through which the MAD has been received.
409+ * srpt_mgmt_method_get - process a received management datagram
410+ * @sp: HCA port through which the MAD has been received.
384411 * @rq_mad: received MAD.
385412 * @rsp_mad: response MAD.
386413 */
@@ -419,7 +446,9 @@ static void srpt_mgmt_method_get(struct srpt_port *sp, struct ib_mad *rq_mad, · 32 unmodified lines
419446 }
420447
421448 /**
422− * srpt_mad_send_handler() - Post MAD-send callback function.
449+ * srpt_mad_send_handler - MAD send completion callback
450+ * @mad_agent: Return value of ib_register_mad_agent().
451+ * @mad_wc: Work completion reporting that the MAD has been sent.
423452 */
424453 static void srpt_mad_send_handler(struct ib_mad_agent *mad_agent,
425454 struct ib_mad_send_wc *mad_wc)
@@ -429,7 +458,10 @@ static void srpt_mad_send_handler(struct ib_mad_agent *mad_agent, · 3 unmodified lines
429458 }
430459
431460 /**
432− * srpt_mad_recv_handler() - MAD reception callback function.
461+ * srpt_mad_recv_handler - MAD reception callback function
462+ * @mad_agent: Return value of ib_register_mad_agent().
463+ * @send_buf: Not used.
464+ * @mad_wc: Work completion reporting that a MAD has been received.
433465 */
434466 static void srpt_mad_recv_handler(struct ib_mad_agent *mad_agent,
435467 struct ib_mad_send_buf *send_buf,
@@ -493,8 +525,18 @@ err: · 57 unmodified lines
493525 ib_free_recv_mad(mad_wc);
494526 }
495527
528+static int srpt_format_guid(char *buf, unsigned int size, const __be64 *guid)
529+{
530+ const __be16 *g = (const __be16 *)guid;
531+
532+ return snprintf(buf, size, "%04x:%04x:%04x:%04x",
533+ be16_to_cpu(g[0]), be16_to_cpu(g[1]),
534+ be16_to_cpu(g[2]), be16_to_cpu(g[3]));
535+}
536+
496537 /**
497− * srpt_refresh_port() - Configure a HCA port.
538+ * srpt_refresh_port - configure a HCA port
539+ * @sport: SRPT HCA port.
498540 *
499541 * Enable InfiniBand management datagram processing, update the cached sm_lid,
500542 * lid and gid values, and register a callback function for processing MADs
@@ -507,7 +549,6 @@ static int srpt_refresh_port(struct srpt_port *sport) · 6 unmodified lines
507549 struct ib_mad_reg_req reg_req;
508550 struct ib_port_modify port_modify;
509551 struct ib_port_attr port_attr;
510− __be16 *guid;
511552 int ret;
512553
513554 memset(&port_modify, 0, sizeof(port_modify));
@@ -531,11 +572,8 @@ static int srpt_refresh_port(struct srpt_port *sport) · 17 unmodified lines
531572 goto err_query_port;
532573
533574 sport->port_guid_wwn.priv = sport;
534− guid = (__be16 *)&sport->gid.global.interface_id;
535− snprintf(sport->port_guid, sizeof(sport->port_guid),
536− "%04x:%04x:%04x:%04x",
537− be16_to_cpu(guid[0]), be16_to_cpu(guid[1]),
538− be16_to_cpu(guid[2]), be16_to_cpu(guid[3]));
575+ srpt_format_guid(sport->port_guid, sizeof(sport->port_guid),
576+ &sport->gid.global.interface_id);
539577 sport->port_gid_wwn.priv = sport;
540578 snprintf(sport->port_gid, sizeof(sport->port_gid),
541579 "0x%016llx%016llx",
@@ -577,7 +615,8 @@ err_mod_port: · 35 unmodified lines
577615 }
578616
579617 /**
580− * srpt_unregister_mad_agent() - Unregister MAD callback functions.
618+ * srpt_unregister_mad_agent - unregister MAD callback functions
619+ * @sdev: SRPT HCA pointer.
581620 *
582621 * Note: It is safe to call this function more than once for the same device.
583622 */
@@ -602,7 +641,11 @@ static void srpt_unregister_mad_agent(struct srpt_device *sdev) · 18 unmodified lines
602641 }
603642
604643 /**
605− * srpt_alloc_ioctx() - Allocate an SRPT I/O context structure.
644+ * srpt_alloc_ioctx - allocate a SRPT I/O context structure
645+ * @sdev: SRPT HCA pointer.
646+ * @ioctx_size: I/O context size.
647+ * @dma_size: Size of I/O context DMA buffer.
648+ * @dir: DMA data direction.
606649 */
607650 static struct srpt_ioctx *srpt_alloc_ioctx(struct srpt_device *sdev,
608651 int ioctx_size, int dma_size,
@@ -633,7 +676,11 @@ err: · 24 unmodified lines
633676 }
634677
635678 /**
636− * srpt_free_ioctx() - Free an SRPT I/O context structure.
679+ * srpt_free_ioctx - free a SRPT I/O context structure
680+ * @sdev: SRPT HCA pointer.
681+ * @ioctx: I/O context pointer.
682+ * @dma_size: Size of I/O context DMA buffer.
683+ * @dir: DMA data direction.
637684 */
638685 static void srpt_free_ioctx(struct srpt_device *sdev, struct srpt_ioctx *ioctx,
639686 int dma_size, enum dma_data_direction dir)
@@ -647,7 +694,7 @@ static void srpt_free_ioctx(struct srpt_device *sdev, struct srpt_ioctx *ioctx, · 7 unmodified lines
647694 }
648695
649696 /**
650− * srpt_alloc_ioctx_ring() - Allocate a ring of SRPT I/O context structures.
697+ * srpt_alloc_ioctx_ring - allocate a ring of SRPT I/O context structures
651698 * @sdev: Device to allocate the I/O context ring for.
652699 * @ring_size: Number of elements in the I/O context ring.
653700 * @ioctx_size: I/O context size.
@@ -685,7 +732,12 @@ out: · 31 unmodified lines
685732 }
686733
687734 /**
688− * srpt_free_ioctx_ring() - Free the ring of SRPT I/O context structures.
735+ * srpt_free_ioctx_ring - free the ring of SRPT I/O context structures
736+ * @ioctx_ring: I/O context ring to be freed.
737+ * @sdev: SRPT HCA pointer.
738+ * @ring_size: Number of ring elements.
739+ * @dma_size: Size of I/O context DMA buffer.
740+ * @dir: DMA data direction.
689741 */
690742 static void srpt_free_ioctx_ring(struct srpt_ioctx **ioctx_ring,
691743 struct srpt_device *sdev, int ring_size,
@@ -702,23 +754,9 @@ static void srpt_free_ioctx_ring(struct srpt_ioctx **ioctx_ring, · 10 unmodified lines
702754 }
703755
704756 /**
705− * srpt_get_cmd_state() - Get the state of a SCSI command.
706− */
707−static enum srpt_command_state srpt_get_cmd_state(struct srpt_send_ioctx *ioctx)
708−{
709− enum srpt_command_state state;
710− unsigned long flags;
711−
712− BUG_ON(!ioctx);
713−
714− spin_lock_irqsave(&ioctx->spinlock, flags);
715− state = ioctx->state;
716− spin_unlock_irqrestore(&ioctx->spinlock, flags);
717− return state;
718−}
719−
720−/**
721− * srpt_set_cmd_state() - Set the state of a SCSI command.
757+ * srpt_set_cmd_state - set the state of a SCSI command
758+ * @ioctx: Send I/O context.
759+ * @new: New I/O context state.
722760 *
723761 * Does not modify the state of aborted commands. Returns the previous command
724762 * state.
@@ -727,21 +765,19 @@ static enum srpt_command_state srpt_set_cmd_state(struct srpt_send_ioctx *ioctx, · 2 unmodified lines
727765 enum srpt_command_state new)
728766 {
729767 enum srpt_command_state previous;
730− unsigned long flags;
731768
732− BUG_ON(!ioctx);
733−
734− spin_lock_irqsave(&ioctx->spinlock, flags);
735769 previous = ioctx->state;
736770 if (previous != SRPT_STATE_DONE)
737771 ioctx->state = new;
738− spin_unlock_irqrestore(&ioctx->spinlock, flags);
739772
740773 return previous;
741774 }
742775
743776 /**
744− * srpt_test_and_set_cmd_state() - Test and set the state of a command.
777+ * srpt_test_and_set_cmd_state - test and set the state of a command
778+ * @ioctx: Send I/O context.
779+ * @old: Current I/O context state.
780+ * @new: New I/O context state.
745781 *
746782 * Returns true if and only if the previous command state was equal to 'old'.
747783 */
@@ -750,22 +786,23 @@ static bool srpt_test_and_set_cmd_state(struct srpt_send_ioctx *ioctx, · 2 unmodified lines
750786 enum srpt_command_state new)
751787 {
752788 enum srpt_command_state previous;
753− unsigned long flags;
754789
755790 WARN_ON(!ioctx);
756791 WARN_ON(old == SRPT_STATE_DONE);
757792 WARN_ON(new == SRPT_STATE_NEW);
758793
759− spin_lock_irqsave(&ioctx->spinlock, flags);
760794 previous = ioctx->state;
761795 if (previous == old)
762796 ioctx->state = new;
763− spin_unlock_irqrestore(&ioctx->spinlock, flags);
797+
764798 return previous == old;
765799 }
766800
767801 /**
768− * srpt_post_recv() - Post an IB receive request.
802+ * srpt_post_recv - post an IB receive request
803+ * @sdev: SRPT HCA pointer.
804+ * @ch: SRPT RDMA channel.
805+ * @ioctx: Receive I/O context pointer.
769806 */
770807 static int srpt_post_recv(struct srpt_device *sdev, struct srpt_rdma_ch *ch,
771808 struct srpt_recv_ioctx *ioctx)
@@ -791,7 +828,8 @@ static int srpt_post_recv(struct srpt_device *sdev, struct srpt_rdma_ch *ch, · 19 unmodified lines
791828 }
792829
793830 /**
794− * srpt_zerolength_write() - Perform a zero-length RDMA write.
831+ * srpt_zerolength_write - perform a zero-length RDMA write
832+ * @ch: SRPT RDMA channel.
795833 *
796834 * A quote from the InfiniBand specification: C9-88: For an HCA responder
797835 * using Reliable Connection service, for each zero-length RDMA READ or WRITE
@@ -802,6 +840,9 @@ static int srpt_zerolength_write(struct srpt_rdma_ch *ch) · 4 unmodified lines
802840 {
803841 struct ib_send_wr wr, *bad_wr;
804842
843+ pr_debug("%s-%d: queued zerolength write\n", ch->sess_name,
844+ ch->qp->qp_num);
845+
805846 memset(&wr, 0, sizeof(wr));
806847 wr.opcode = IB_WR_RDMA_WRITE;
807848 wr.wr_cqe = &ch->zw_cqe;
@@ -813,13 +854,17 @@ static void srpt_zerolength_write_done(struct ib_cq *cq, struct ib_wc *wc) · 5 unmodified lines
813854 {
814855 struct srpt_rdma_ch *ch = cq->cq_context;
815856
857+ pr_debug("%s-%d wc->status %d\n", ch->sess_name, ch->qp->qp_num,
858+ wc->status);
859+
816860 if (wc->status == IB_WC_SUCCESS) {
817861 srpt_process_wait_list(ch);
818862 } else {
819863 if (srpt_set_ch_state(ch, CH_DISCONNECTED))
820864 schedule_work(&ch->release_work);
821865 else
822− WARN_ONCE(1, "%s-%d\n", ch->sess_name, ch->qp->qp_num);
866+ pr_debug("%s-%d: already disconnected.\n",
867+ ch->sess_name, ch->qp->qp_num);
823868 }
824869 }
825870
@@ -928,11 +973,13 @@ static inline void *srpt_get_desc_buf(struct srp_cmd *srp_cmd) · 102 unmodified lines
928973 }
929974
930975 /**
931− * srpt_get_desc_tbl() - Parse the data descriptors of an SRP_CMD request.
976+ * srpt_get_desc_tbl - parse the data descriptors of a SRP_CMD request
932977 * @ioctx: Pointer to the I/O context associated with the request.
933978 * @srp_cmd: Pointer to the SRP_CMD request data.
934979 * @dir: Pointer to the variable to which the transfer direction will be
935980 * written.
981+ * @sg: [out] scatterlist allocated for the parsed SRP_CMD.
982+ * @sg_cnt: [out] length of @sg.
936983 * @data_len: Pointer to the variable to which the total data length of all
937984 * descriptors in the SRP_CMD request will be written.
938985 *
@@ -998,7 +1045,9 @@ static int srpt_get_desc_tbl(struct srpt_send_ioctx *ioctx, · 59 unmodified lines
9981045 }
9991046
10001047 /**
1001− * srpt_init_ch_qp() - Initialize queue pair attributes.
1048+ * srpt_init_ch_qp - initialize queue pair attributes
1049+ * @ch: SRPT RDMA channel.
1050+ * @qp: Queue pair pointer.
10021051 *
10031052 * Initialized the attributes of queue pair 'qp' by allowing local write,
10041053 * remote read and remote write. Also transitions 'qp' to state IB_QPS_INIT.
@@ -1015,7 +1064,12 @@ static int srpt_init_ch_qp(struct srpt_rdma_ch *ch, struct ib_qp *qp) · 10 unmodified lines
10151064 attr->qp_state = IB_QPS_INIT;
10161065 attr->qp_access_flags = IB_ACCESS_LOCAL_WRITE;
10171066 attr->port_num = ch->sport->port;
1018− attr->pkey_index = 0;
1067+
1068+ ret = ib_find_cached_pkey(ch->sport->sdev->device, ch->sport->port,
1069+ ch->pkey, &attr->pkey_index);
1070+ if (ret < 0)
1071+ pr_err("Translating pkey %#x failed (%d) - using index 0\n",
1072+ ch->pkey, ret);
10191073
10201074 ret = ib_modify_qp(qp, attr,
10211075 IB_QP_STATE | IB_QP_ACCESS_FLAGS | IB_QP_PORT |
@@ -1026,7 +1080,7 @@ static int srpt_init_ch_qp(struct srpt_rdma_ch *ch, struct ib_qp *qp) · 4 unmodified lines
10261080 }
10271081
10281082 /**
1029− * srpt_ch_qp_rtr() - Change the state of a channel to 'ready to receive' (RTR).
1083+ * srpt_ch_qp_rtr - change the state of a channel to 'ready to receive' (RTR)
10301084 * @ch: channel of the queue pair.
10311085 * @qp: queue pair to change the state of.
10321086 *
@@ -1043,7 +1097,7 @@ static int srpt_ch_qp_rtr(struct srpt_rdma_ch *ch, struct ib_qp *qp) · 10 unmodified lines
10431097 int ret;
10441098
10451099 qp_attr.qp_state = IB_QPS_RTR;
1046− ret = ib_cm_init_qp_attr(ch->cm_id, &qp_attr, &attr_mask);
1100+ ret = ib_cm_init_qp_attr(ch->ib_cm.cm_id, &qp_attr, &attr_mask);
10471101 if (ret)
10481102 goto out;
10491103
@@ -1056,7 +1110,7 @@ out: · 6 unmodified lines
10561110 }
10571111
10581112 /**
1059− * srpt_ch_qp_rts() - Change the state of a channel to 'ready to send' (RTS).
1113+ * srpt_ch_qp_rts - change the state of a channel to 'ready to send' (RTS)
10601114 * @ch: channel of the queue pair.
10611115 * @qp: queue pair to change the state of.
10621116 *
@@ -1073,7 +1127,7 @@ static int srpt_ch_qp_rts(struct srpt_rdma_ch *ch, struct ib_qp *qp) · 10 unmodified lines
10731127 int ret;
10741128
10751129 qp_attr.qp_state = IB_QPS_RTS;
1076− ret = ib_cm_init_qp_attr(ch->cm_id, &qp_attr, &attr_mask);
1130+ ret = ib_cm_init_qp_attr(ch->ib_cm.cm_id, &qp_attr, &attr_mask);
10771131 if (ret)
10781132 goto out;
10791133
@@ -1086,7 +1140,8 @@ out: · 6 unmodified lines
10861140 }
10871141
10881142 /**
1089− * srpt_ch_qp_err() - Set the channel queue pair state to 'error'.
1143+ * srpt_ch_qp_err - set the channel queue pair state to 'error'
1144+ * @ch: SRPT RDMA channel.
10901145 */
10911146 static int srpt_ch_qp_err(struct srpt_rdma_ch *ch)
10921147 {
@@ -1097,7 +1152,8 @@ static int srpt_ch_qp_err(struct srpt_rdma_ch *ch) · 4 unmodified lines
10971152 }
10981153
10991154 /**
1100− * srpt_get_send_ioctx() - Obtain an I/O context for sending to the initiator.
1155+ * srpt_get_send_ioctx - obtain an I/O context for sending to the initiator
1156+ * @ch: SRPT RDMA channel.
11011157 */
11021158 static struct srpt_send_ioctx *srpt_get_send_ioctx(struct srpt_rdma_ch *ch)
11031159 {
@@ -1119,11 +1175,9 @@ static struct srpt_send_ioctx *srpt_get_send_ioctx(struct srpt_rdma_ch *ch) · 15 unmodified lines
11191175 return ioctx;
11201176
11211177 BUG_ON(ioctx->ch != ch);
1122− spin_lock_init(&ioctx->spinlock);
11231178 ioctx->state = SRPT_STATE_NEW;
11241179 ioctx->n_rdma = 0;
11251180 ioctx->n_rw_ctx = 0;
1126− init_completion(&ioctx->tx_done);
11271181 ioctx->queue_status_only = false;
11281182 /*
11291183 * transport_init_se_cmd() does not initialize all fields, so do it
@@ -1136,14 +1190,12 @@ static struct srpt_send_ioctx *srpt_get_send_ioctx(struct srpt_rdma_ch *ch) · 6 unmodified lines
11361190 }
11371191
11381192 /**
1139− * srpt_abort_cmd() - Abort a SCSI command.
1193+ * srpt_abort_cmd - abort a SCSI command
11401194 * @ioctx: I/O context associated with the SCSI command.
1141− * @context: Preferred execution context.
11421195 */
11431196 static int srpt_abort_cmd(struct srpt_send_ioctx *ioctx)
11441197 {
11451198 enum srpt_command_state state;
1146− unsigned long flags;
11471199
11481200 BUG_ON(!ioctx);
11491201
@@ -1152,7 +1204,6 @@ static int srpt_abort_cmd(struct srpt_send_ioctx *ioctx) · 2 unmodified lines
11521204 * the ib_srpt driver, change the state to the next state.
11531205 */
11541206
1155− spin_lock_irqsave(&ioctx->spinlock, flags);
11561207 state = ioctx->state;
11571208 switch (state) {
11581209 case SRPT_STATE_NEED_DATA:
@@ -1167,7 +1218,6 @@ static int srpt_abort_cmd(struct srpt_send_ioctx *ioctx) · 8 unmodified lines
11671218 __func__, state);
11681219 break;
11691220 }
1170− spin_unlock_irqrestore(&ioctx->spinlock, flags);
11711221
11721222 pr_debug("Aborting cmd with state %d -> %d and tag %lld\n", state,
11731223 ioctx->state, ioctx->cmd.tag);
@@ -1206,6 +1256,10 @@ static int srpt_abort_cmd(struct srpt_send_ioctx *ioctx) · 32 unmodified lines
12061256 }
12071257
12081258 /**
1259+ * srpt_rdma_read_done - RDMA read completion callback
1260+ * @cq: Completion queue.
1261+ * @wc: Work completion.
1262+ *
12091263 * XXX: what is now target_execute_cmd used to be asynchronous, and unmapping
12101264 * the data that has been transferred via IB RDMA had to be postponed until the
12111265 * check_stop_free() callback. None of this is necessary anymore and needs to
@@ -1233,11 +1287,11 @@ static void srpt_rdma_read_done(struct ib_cq *cq, struct ib_wc *wc) · 21 unmodified lines
12331287 target_execute_cmd(&ioctx->cmd);
12341288 else
12351289 pr_err("%s[%d]: wrong state = %d\n", __func__,
1236− __LINE__, srpt_get_cmd_state(ioctx));
1290+ __LINE__, ioctx->state);
12371291 }
12381292
12391293 /**
1240− * srpt_build_cmd_rsp() - Build an SRP_RSP response.
1294+ * srpt_build_cmd_rsp - build a SRP_RSP response
12411295 * @ch: RDMA channel through which the request has been received.
12421296 * @ioctx: I/O context associated with the SRP_CMD request. The response will
12431297 * be built in the buffer ioctx->buf points at and hence this function will
@@ -1297,7 +1351,7 @@ static int srpt_build_cmd_rsp(struct srpt_rdma_ch *ch, · 53 unmodified lines
12971351 }
12981352
12991353 /**
1300− * srpt_build_tskmgmt_rsp() - Build a task management response.
1354+ * srpt_build_tskmgmt_rsp - build a task management response
13011355 * @ch: RDMA channel through which the request has been received.
13021356 * @ioctx: I/O context in which the SRP_RSP response will be built.
13031357 * @rsp_code: RSP_CODE that will be stored in the response.
@@ -1345,7 +1399,10 @@ static int srpt_check_stop_free(struct se_cmd *cmd) · 41 unmodified lines
13451399 }
13461400
13471401 /**
1348− * srpt_handle_cmd() - Process SRP_CMD.
1402+ * srpt_handle_cmd - process a SRP_CMD information unit
1403+ * @ch: SRPT RDMA channel.
1404+ * @recv_ioctx: Receive I/O context.
1405+ * @send_ioctx: Send I/O context.
13491406 */
13501407 static void srpt_handle_cmd(struct srpt_rdma_ch *ch,
13511408 struct srpt_recv_ioctx *recv_ioctx,
@@ -1427,7 +1484,10 @@ static int srp_tmr_to_tcm(int fn) · 75 unmodified lines
14271484 }
14281485
14291486 /**
1430− * srpt_handle_tsk_mgmt() - Process an SRP_TSK_MGMT information unit.
1487+ * srpt_handle_tsk_mgmt - process a SRP_TSK_MGMT information unit
1488+ * @ch: SRPT RDMA channel.
1489+ * @recv_ioctx: Receive I/O context.
1490+ * @send_ioctx: Send I/O context.
14311491 *
14321492 * Returns 0 if and only if the request will be processed by the target core.
14331493 *
@@ -1449,9 +1509,9 @@ static void srpt_handle_tsk_mgmt(struct srpt_rdma_ch *ch, · 15 unmodified lines
14491509 srp_tsk = recv_ioctx->ioctx.buf;
14501510 cmd = &send_ioctx->cmd;
14511511
1452− pr_debug("recv tsk_mgmt fn %d for task_tag %lld and cmd tag %lld"
1453− " cm_id %p sess %p\n", srp_tsk->tsk_mgmt_func,
1454− srp_tsk->task_tag, srp_tsk->tag, ch->cm_id, ch->sess);
1512+ pr_debug("recv tsk_mgmt fn %d for task_tag %lld and cmd tag %lld ch %p sess %p\n",
1513+ srp_tsk->tsk_mgmt_func, srp_tsk->task_tag, srp_tsk->tag, ch,
1514+ ch->sess);
14551515
14561516 srpt_set_cmd_state(send_ioctx, SRPT_STATE_MGMT);
14571517 send_ioctx->cmd.tag = srp_tsk->tag;
@@ -1470,41 +1530,42 @@ fail: · 12 unmodified lines
14701530 }
14711531
14721532 /**
1473− * srpt_handle_new_iu() - Process a newly received information unit.
1533+ * srpt_handle_new_iu - process a newly received information unit
14741534 * @ch: RDMA channel through which the information unit has been received.
1475− * @ioctx: SRPT I/O context associated with the information unit.
1535+ * @recv_ioctx: Receive I/O context associated with the information unit.
14761536 */
1477−static void srpt_handle_new_iu(struct srpt_rdma_ch *ch,
1478− struct srpt_recv_ioctx *recv_ioctx,
1479− struct srpt_send_ioctx *send_ioctx)
1537+static bool
1538+srpt_handle_new_iu(struct srpt_rdma_ch *ch, struct srpt_recv_ioctx *recv_ioctx)
14801539 {
1540+ struct srpt_send_ioctx *send_ioctx = NULL;
14811541 struct srp_cmd *srp_cmd;
1542+ bool res = false;
1543+ u8 opcode;
14821544
14831545 BUG_ON(!ch);
14841546 BUG_ON(!recv_ioctx);
14851547
1548+ if (unlikely(ch->state == CH_CONNECTING))
1549+ goto push;
1550+
14861551 ib_dma_sync_single_for_cpu(ch->sport->sdev->device,
14871552 recv_ioctx->ioctx.dma, srp_max_req_size,
14881553 DMA_FROM_DEVICE);
14891554
1490− if (unlikely(ch->state == CH_CONNECTING))
1491− goto out_wait;
1492−
1493− if (unlikely(ch->state != CH_LIVE))
1494− return;
1495−
14961555 srp_cmd = recv_ioctx->ioctx.buf;
1497− if (srp_cmd->opcode == SRP_CMD || srp_cmd->opcode == SRP_TSK_MGMT) {
1498− if (!send_ioctx) {
1499− if (!list_empty(&ch->cmd_wait_list))
1500− goto out_wait;
1501− send_ioctx = srpt_get_send_ioctx(ch);
1502− }
1556+ opcode = srp_cmd->opcode;
1557+ if (opcode == SRP_CMD || opcode == SRP_TSK_MGMT) {
1558+ send_ioctx = srpt_get_send_ioctx(ch);
15031559 if (unlikely(!send_ioctx))
1504− goto out_wait;
1560+ goto push;
15051561 }
15061562
1507− switch (srp_cmd->opcode) {
1563+ if (!list_empty(&recv_ioctx->wait_list)) {
1564+ WARN_ON_ONCE(!ch->processing_wait_list);
1565+ list_del_init(&recv_ioctx->wait_list);
1566+ }
1567+
1568+ switch (opcode) {
15081569 case SRP_CMD:
15091570 srpt_handle_cmd(ch, recv_ioctx, send_ioctx);
15101571 break;
@@ -1524,16 +1585,22 @@ static void srpt_handle_new_iu(struct srpt_rdma_ch *ch, · 13 unmodified lines
15241585 pr_err("Received SRP_RSP\n");
15251586 break;
15261587 default:
1527− pr_err("received IU with unknown opcode 0x%x\n",
1528− srp_cmd->opcode);
1588+ pr_err("received IU with unknown opcode 0x%x\n", opcode);
15291589 break;
15301590 }
15311591
15321592 srpt_post_recv(ch->sport->sdev, ch, recv_ioctx);
1533− return;
1593+ res = true;
1594+
1595+out:
1596+ return res;
15341597
1535−out_wait:
1536− list_add_tail(&recv_ioctx->wait_list, &ch->cmd_wait_list);
1598+push:
1599+ if (list_empty(&recv_ioctx->wait_list)) {
1600+ WARN_ON_ONCE(ch->processing_wait_list);
1601+ list_add_tail(&recv_ioctx->wait_list, &ch->cmd_wait_list);
1602+ }
1603+ goto out;
15371604 }
15381605
15391606 static void srpt_recv_done(struct ib_cq *cq, struct ib_wc *wc)
@@ -1548,10 +1615,10 @@ static void srpt_recv_done(struct ib_cq *cq, struct ib_wc *wc) · 8 unmodified lines
15481615 req_lim = atomic_dec_return(&ch->req_lim);
15491616 if (unlikely(req_lim < 0))
15501617 pr_err("req_lim = %d < 0\n", req_lim);
1551− srpt_handle_new_iu(ch, ioctx, NULL);
1618+ srpt_handle_new_iu(ch, ioctx);
15521619 } else {
1553− pr_info("receiving failed for ioctx %p with status %d\n",
1554− ioctx, wc->status);
1620+ pr_info_ratelimited("receiving failed for ioctx %p with status %d\n",
1621+ ioctx, wc->status);
15551622 }
15561623 }
15571624
@@ -1562,22 +1629,28 @@ static void srpt_recv_done(struct ib_cq *cq, struct ib_wc *wc) · 4 unmodified lines
15621629 */
15631630 static void srpt_process_wait_list(struct srpt_rdma_ch *ch)
15641631 {
1565− struct srpt_send_ioctx *ioctx;
1632+ struct srpt_recv_ioctx *recv_ioctx, *tmp;
1633+
1634+ WARN_ON_ONCE(ch->state == CH_CONNECTING);
15661635
1567− while (!list_empty(&ch->cmd_wait_list) &&
1568− ch->state >= CH_LIVE &&
1569− (ioctx = srpt_get_send_ioctx(ch)) != NULL) {
1570− struct srpt_recv_ioctx *recv_ioctx;
1636+ if (list_empty(&ch->cmd_wait_list))
1637+ return;
15711638
1572− recv_ioctx = list_first_entry(&ch->cmd_wait_list,
1573− struct srpt_recv_ioctx,
1574− wait_list);
1575− list_del(&recv_ioctx->wait_list);
1576− srpt_handle_new_iu(ch, recv_ioctx, ioctx);
1639+ WARN_ON_ONCE(ch->processing_wait_list);
1640+ ch->processing_wait_list = true;
1641+ list_for_each_entry_safe(recv_ioctx, tmp, &ch->cmd_wait_list,
1642+ wait_list) {
1643+ if (!srpt_handle_new_iu(ch, recv_ioctx))
1644+ break;
15771645 }
1646+ ch->processing_wait_list = false;
15781647 }
15791648
15801649 /**
1650+ * srpt_send_done - send completion callback
1651+ * @cq: Completion queue.
1652+ * @wc: Work completion.
1653+ *
15811654 * Note: Although this has not yet been observed during tests, at least in
15821655 * theory it is possible that the srpt_get_send_ioctx() call invoked by
15831656 * srpt_handle_new_iu() fails. This is possible because the req_lim_delta
@@ -1619,7 +1692,8 @@ static void srpt_send_done(struct ib_cq *cq, struct ib_wc *wc) · 35 unmodified lines
16191692 }
16201693
16211694 /**
1622− * srpt_create_ch_ib() - Create receive and send completion queues.
1695+ * srpt_create_ch_ib - create receive and send completion queues
1696+ * @ch: SRPT RDMA channel.
16231697 */
16241698 static int srpt_create_ch_ib(struct srpt_rdma_ch *ch)
16251699 {
@@ -1627,7 +1701,7 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch) · 1 unmodified line
16271701 struct srpt_port *sport = ch->sport;
16281702 struct srpt_device *sdev = sport->sdev;
16291703 const struct ib_device_attr *attrs = &sdev->device->attrs;
1630− u32 srp_sq_size = sport->port_attrib.srp_sq_size;
1704+ int sq_size = sport->port_attrib.srp_sq_size;
16311705 int i, ret;
16321706
16331707 WARN_ON(ch->rq_size < 1);
@@ -1638,12 +1712,12 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch) · 4 unmodified lines
16381712 goto out;
16391713
16401714 retry:
1641− ch->cq = ib_alloc_cq(sdev->device, ch, ch->rq_size + srp_sq_size,
1715+ ch->cq = ib_alloc_cq(sdev->device, ch, ch->rq_size + sq_size,
16421716 0 /* XXX: spread CQs */, IB_POLL_WORKQUEUE);
16431717 if (IS_ERR(ch->cq)) {
16441718 ret = PTR_ERR(ch->cq);
16451719 pr_err("failed to create CQ cqe= %d ret= %d\n",
1646− ch->rq_size + srp_sq_size, ret);
1720+ ch->rq_size + sq_size, ret);
16471721 goto out;
16481722 }
16491723
@@ -1661,8 +1735,8 @@ retry: · 11 unmodified lines
16611735 * both both, as RDMA contexts will also post completions for the
16621736 * RDMA READ case.
16631737 */
1664− qp_init->cap.max_send_wr = min(srp_sq_size / 2, attrs->max_qp_wr + 0U);
1665− qp_init->cap.max_rdma_ctxs = srp_sq_size / 2;
1738+ qp_init->cap.max_send_wr = min(sq_size / 2, attrs->max_qp_wr);
1739+ qp_init->cap.max_rdma_ctxs = sq_size / 2;
16661740 qp_init->cap.max_send_sge = min(attrs->max_sge, SRPT_MAX_SG_PER_WQE);
16671741 qp_init->port_num = ch->sport->port;
16681742 if (sdev->use_srq) {
@@ -1676,8 +1750,8 @@ retry: · 7 unmodified lines
16761750 if (IS_ERR(ch->qp)) {
16771751 ret = PTR_ERR(ch->qp);
16781752 if (ret == -ENOMEM) {
1679− srp_sq_size /= 2;
1680− if (srp_sq_size >= MIN_SRPT_SQ_SIZE) {
1753+ sq_size /= 2;
1754+ if (sq_size >= MIN_SRPT_SQ_SIZE) {
16811755 ib_destroy_cq(ch->cq);
16821756 goto retry;
16831757 }
@@ -1688,9 +1762,9 @@ retry: · 4 unmodified lines
16881762
16891763 atomic_set(&ch->sq_wr_avail, qp_init->cap.max_send_wr);
16901764
1691− pr_debug("%s: max_cqe= %d max_sge= %d sq_size = %d cm_id= %p\n",
1765+ pr_debug("%s: max_cqe= %d max_sge= %d sq_size = %d ch= %p\n",
16921766 __func__, ch->cq->cqe, qp_init->cap.max_send_sge,
1693− qp_init->cap.max_send_wr, ch->cm_id);
1767+ qp_init->cap.max_send_wr, ch);
16941768
16951769 ret = srpt_init_ch_qp(ch, ch->qp);
16961770 if (ret)
@@ -1718,7 +1792,8 @@ static void srpt_destroy_ch_ib(struct srpt_rdma_ch *ch) · 21 unmodified lines
17181792 }
17191793
17201794 /**
1721− * srpt_close_ch() - Close an RDMA channel.
1795+ * srpt_close_ch - close a RDMA channel
1796+ * @ch: SRPT RDMA channel.
17221797 *
17231798 * Make sure all resources associated with the channel will be deallocated at
17241799 * an appropriate time.
@@ -1743,8 +1818,6 @@ static bool srpt_close_ch(struct srpt_rdma_ch *ch) · 18 unmodified lines
17431818 pr_err("%s-%d: changing queue pair into error state failed: %d\n",
17441819 ch->sess_name, ch->qp->qp_num, ret);
17451820
1746− pr_debug("%s-%d: queued zerolength write\n", ch->sess_name,
1747− ch->qp->qp_num);
17481821 ret = srpt_zerolength_write(ch);
17491822 if (ret < 0) {
17501823 pr_err("%s-%d: queuing zero-length write failed: %d\n",
@@ -1776,9 +1849,9 @@ static int srpt_disconnect_ch(struct srpt_rdma_ch *ch) · 25 unmodified lines
17761849 if (!srpt_set_ch_state(ch, CH_DISCONNECTING))
17771850 return -ENOTCONN;
17781851
1779− ret = ib_send_cm_dreq(ch->cm_id, NULL, 0);
1852+ ret = ib_send_cm_dreq(ch->ib_cm.cm_id, NULL, 0);
17801853 if (ret < 0)
1781− ret = ib_send_cm_drep(ch->cm_id, NULL, 0);
1854+ ret = ib_send_cm_drep(ch->ib_cm.cm_id, NULL, 0);
17821855
17831856 if (ret < 0 && srpt_close_ch(ch))
17841857 ret = 0;
@@ -1786,83 +1859,135 @@ static int srpt_disconnect_ch(struct srpt_rdma_ch *ch) · 1 unmodified line
17861859 return ret;
17871860 }
17881861
1789−/*
1790− * Send DREQ and wait for DREP. Return true if and only if this function
1791− * changed the state of @ch.
1792− */
1793−static bool srpt_disconnect_ch_sync(struct srpt_rdma_ch *ch)
1794− __must_hold(&sdev->mutex)
1862+static bool srpt_ch_closed(struct srpt_port *sport, struct srpt_rdma_ch *ch)
17951863 {
1796− DECLARE_COMPLETION_ONSTACK(release_done);
1797− struct srpt_device *sdev = ch->sport->sdev;
1798− bool wait;
1864+ struct srpt_nexus *nexus;
1865+ struct srpt_rdma_ch *ch2;
1866+ bool res = true;
1867+
1868+ rcu_read_lock();
1869+ list_for_each_entry(nexus, &sport->nexus_list, entry) {
1870+ list_for_each_entry(ch2, &nexus->ch_list, list) {
1871+ if (ch2 == ch) {
1872+ res = false;
1873+ goto done;
1874+ }
1875+ }
1876+ }
1877+done:
1878+ rcu_read_unlock();
17991879
1800− lockdep_assert_held(&sdev->mutex);
1880+ return res;
1881+}
1882+
1883+/* Send DREQ and wait for DREP. */
1884+static void srpt_disconnect_ch_sync(struct srpt_rdma_ch *ch)
1885+{
1886+ struct srpt_port *sport = ch->sport;
18011887
18021888 pr_debug("ch %s-%d state %d\n", ch->sess_name, ch->qp->qp_num,
18031889 ch->state);
18041890
1805− WARN_ON(ch->release_done);
1806− ch->release_done = &release_done;
1807− wait = !list_empty(&ch->list);
1891+ mutex_lock(&sport->mutex);
18081892 srpt_disconnect_ch(ch);
1809− mutex_unlock(&sdev->mutex);
1893+ mutex_unlock(&sport->mutex);
18101894
1811− if (!wait)
1812− goto out;
1813−
1814− while (wait_for_completion_timeout(&release_done, 180 * HZ) == 0)
1895+ while (wait_event_timeout(sport->ch_releaseQ, srpt_ch_closed(sport, ch),
1896+ 5 * HZ) == 0)
18151897 pr_info("%s(%s-%d state %d): still waiting ...\n", __func__,
18161898 ch->sess_name, ch->qp->qp_num, ch->state);
18171899
1818−out:
1819− mutex_lock(&sdev->mutex);
1820− return wait;
18211900 }
18221901
1823−static void srpt_set_enabled(struct srpt_port *sport, bool enabled)
1824− __must_hold(&sdev->mutex)
1902+static void __srpt_close_all_ch(struct srpt_port *sport)
18251903 {
1826− struct srpt_device *sdev = sport->sdev;
1904+ struct srpt_nexus *nexus;
18271905 struct srpt_rdma_ch *ch;
18281906
1829− lockdep_assert_held(&sdev->mutex);
1907+ lockdep_assert_held(&sport->mutex);
18301908
1831− if (sport->enabled == enabled)
1832− return;
1833− sport->enabled = enabled;
1834− if (sport->enabled)
1835− return;
1909+ list_for_each_entry(nexus, &sport->nexus_list, entry) {
1910+ list_for_each_entry(ch, &nexus->ch_list, list) {
1911+ if (srpt_disconnect_ch(ch) >= 0)
1912+ pr_info("Closing channel %s-%d because target %s_%d has been disabled\n",
1913+ ch->sess_name, ch->qp->qp_num,
1914+ sport->sdev->device->name, sport->port);
1915+ srpt_close_ch(ch);
1916+ }
1917+ }
1918+}
18361919
1837−again:
1838− list_for_each_entry(ch, &sdev->rch_list, list) {
1839− if (ch->sport == sport) {
1840− pr_info("%s: closing channel %s-%d\n",
1841− sdev->device->name, ch->sess_name,
1842− ch->qp->qp_num);
1843− if (srpt_disconnect_ch_sync(ch))
1844− goto again;
1920+/*
1921+ * Look up (i_port_id, t_port_id) in sport->nexus_list. Create an entry if
1922+ * it does not yet exist.
1923+ */
1924+static struct srpt_nexus *srpt_get_nexus(struct srpt_port *sport,
1925+ const u8 i_port_id[16],
1926+ const u8 t_port_id[16])
1927+{
1928+ struct srpt_nexus *nexus = NULL, *tmp_nexus = NULL, *n;
1929+
1930+ for (;;) {
1931+ mutex_lock(&sport->mutex);
1932+ list_for_each_entry(n, &sport->nexus_list, entry) {
1933+ if (memcmp(n->i_port_id, i_port_id, 16) == 0 &&
1934+ memcmp(n->t_port_id, t_port_id, 16) == 0) {
1935+ nexus = n;
1936+ break;
1937+ }
18451938 }
1939+ if (!nexus && tmp_nexus) {
1940+ list_add_tail_rcu(&tmp_nexus->entry,
1941+ &sport->nexus_list);
1942+ swap(nexus, tmp_nexus);
1943+ }
1944+ mutex_unlock(&sport->mutex);
1945+
1946+ if (nexus)
1947+ break;
1948+ tmp_nexus = kzalloc(sizeof(*nexus), GFP_KERNEL);
1949+ if (!tmp_nexus) {
1950+ nexus = ERR_PTR(-ENOMEM);
1951+ break;
1952+ }
1953+ INIT_LIST_HEAD(&tmp_nexus->ch_list);
1954+ memcpy(tmp_nexus->i_port_id, i_port_id, 16);
1955+ memcpy(tmp_nexus->t_port_id, t_port_id, 16);
18461956 }
18471957
1958+ kfree(tmp_nexus);
1959+
1960+ return nexus;
1961+}
1962+
1963+static void srpt_set_enabled(struct srpt_port *sport, bool enabled)
1964+ __must_hold(&sport->mutex)
1965+{
1966+ lockdep_assert_held(&sport->mutex);
1967+
1968+ if (sport->enabled == enabled)
1969+ return;
1970+ sport->enabled = enabled;
1971+ if (!enabled)
1972+ __srpt_close_all_ch(sport);
18481973 }
18491974
18501975 static void srpt_free_ch(struct kref *kref)
18511976 {
18521977 struct srpt_rdma_ch *ch = container_of(kref, struct srpt_rdma_ch, kref);
18531978
1854− kfree(ch);
1979+ kfree_rcu(ch, rcu);
18551980 }
18561981
18571982 static void srpt_release_channel_work(struct work_struct *w)
18581983 {
18591984 struct srpt_rdma_ch *ch;
18601985 struct srpt_device *sdev;
1986+ struct srpt_port *sport;
18611987 struct se_session *se_sess;
18621988
18631989 ch = container_of(w, struct srpt_rdma_ch, release_work);
1864− pr_debug("%s: %s-%d; release_done = %p\n", __func__, ch->sess_name,
1865− ch->qp->qp_num, ch->release_done);
1990+ pr_debug("%s-%d\n", ch->sess_name, ch->qp->qp_num);
18661991
18671992 sdev = ch->sport->sdev;
18681993 BUG_ON(!sdev);
@@ -1877,169 +2002,141 @@ static void srpt_release_channel_work(struct work_struct *w) · 8 unmodified lines
18772002 transport_deregister_session(se_sess);
18782003 ch->sess = NULL;
18792004
1880− ib_destroy_cm_id(ch->cm_id);
2005+ ib_destroy_cm_id(ch->ib_cm.cm_id);
18812006
18822007 srpt_destroy_ch_ib(ch);
18832008
18842009 srpt_free_ioctx_ring((struct srpt_ioctx **)ch->ioctx_ring,
18852010 ch->sport->sdev, ch->rq_size,
1886− ch->rsp_size, DMA_TO_DEVICE);
2011+ ch->max_rsp_size, DMA_TO_DEVICE);
18872012
18882013 srpt_free_ioctx_ring((struct srpt_ioctx **)ch->ioctx_recv_ring,
18892014 sdev, ch->rq_size,
18902015 srp_max_req_size, DMA_FROM_DEVICE);
18912016
1892− mutex_lock(&sdev->mutex);
1893− list_del_init(&ch->list);
1894− if (ch->release_done)
1895− complete(ch->release_done);
1896− mutex_unlock(&sdev->mutex);
2017+ sport = ch->sport;
2018+ mutex_lock(&sport->mutex);
2019+ list_del_rcu(&ch->list);
2020+ mutex_unlock(&sport->mutex);
18972021
1898− wake_up(&sdev->ch_releaseQ);
2022+ wake_up(&sport->ch_releaseQ);
18992023
19002024 kref_put(&ch->kref, srpt_free_ch);
19012025 }
19022026
19032027 /**
1904− * srpt_cm_req_recv() - Process the event IB_CM_REQ_RECEIVED.
2028+ * srpt_cm_req_recv - process the event IB_CM_REQ_RECEIVED
2029+ * @cm_id: IB/CM connection identifier.
2030+ * @port_num: Port through which the IB/CM REQ message was received.
2031+ * @pkey: P_Key of the incoming connection.
2032+ * @req: SRP login request.
2033+ * @src_addr: GID of the port that submitted the login request.
19052034 *
19062035 * Ownership of the cm_id is transferred to the target session if this
19072036 * functions returns zero. Otherwise the caller remains the owner of cm_id.
19082037 */
19092038 static int srpt_cm_req_recv(struct ib_cm_id *cm_id,

798 more lines aren’t shown here.

drivers/infiniband/ulp/srpt/ib_srpt.h+66−34

This diff is large: its 253 lines aren’t shown here.

drivers/net/ethernet/mellanox/mlx4/qp.c+3−0

This diff is large: its 10 lines aren’t shown here.

drivers/net/ethernet/mellanox/mlx5/core/eq.c+8−1

This diff is large: its 23 lines aren’t shown here.

drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c+6−5

This diff is large: its 46 lines aren’t shown here.

drivers/net/ethernet/mellanox/mlx5/core/fw.c+9−1

This diff is large: its 22 lines aren’t shown here.

drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c+55−0

This diff is large: its 117 lines aren’t shown here.

drivers/net/ethernet/mellanox/mlx5/core/lib/gid.c+4−1

This diff is large: its 19 lines aren’t shown here.

drivers/net/ethernet/mellanox/mlx5/core/main.c+14−1

This diff is large: its 43 lines aren’t shown here.

drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/net/ethernet/mellanox/mlx5/core/qp.c+113−12

This diff is large: its 230 lines aren’t shown here.

drivers/net/ethernet/mellanox/mlx5/core/vport.c+85−6

This diff is large: its 114 lines aren’t shown here.

include/linux/mlx5/device.h+16−0

This diff is large: its 58 lines aren’t shown here.

include/linux/mlx5/driver.h+41−2

This diff is large: its 99 lines aren’t shown here.

include/linux/mlx5/mlx5_ifc.h+25−7

This diff is large: its 88 lines aren’t shown here.

include/linux/mlx5/qp.h+12−0

This diff is large: its 35 lines aren’t shown here.

include/linux/mlx5/vport.h+4−0

This diff is large: its 9 lines aren’t shown here.

include/rdma/ib_addr.h+9−29

This diff is large: its 63 lines aren’t shown here.

include/rdma/ib_sa.h+5−5

This diff is large: its 18 lines aren’t shown here.

include/rdma/ib_verbs.h+46−18

This diff is large: its 184 lines aren’t shown here.

include/rdma/opa_addr.h+16−0

This diff is large: its 21 lines aren’t shown here.

include/rdma/rdma_cm.h+19−0

This diff is large: its 24 lines aren’t shown here.

include/rdma/rdma_cm_ib.h+4−4

This diff is large: its 22 lines aren’t shown here.

include/rdma/rdma_vt.h+24−7

This diff is large: its 45 lines aren’t shown here.

include/rdma/restrack.h+157−0

This diff is large: its 158 lines aren’t shown here.

include/scsi/srp.h+17−0

This diff is large: its 24 lines aren’t shown here.

include/uapi/rdma/bnxt_re-abi.h+9−0

This diff is large: its 16 lines aren’t shown here.

include/uapi/rdma/ib_user_verbs.h+8−3

This diff is large: its 38 lines aren’t shown here.

include/uapi/rdma/mlx4-abi.h+4−3

This diff is large: its 21 lines aren’t shown here.

include/uapi/rdma/mlx5-abi.h+49−4

This diff is large: its 96 lines aren’t shown here.

include/uapi/rdma/rdma_netlink.h+49−0

This diff is large: its 63 lines aren’t shown here.

include/uapi/rdma/vmw_pvrdma-abi.h+7−5

This diff is large: its 20 lines aren’t shown here.

lib/kobject.c+2−0

This diff is large: its 13 lines aren’t shown here.

net/rds/ib.c+2−4

This diff is large: its 16 lines aren’t shown here.