History

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma

Pull rdma updates from Doug Ledford: "This is the complete update for the rdma stack for this release cycle. Most of it is typical driver and core updates, but there is the entirely new VMWare pvrdma driver. You may have noticed that there were changes in DaveM's pull request to the bnxt Ethernet driver to support a RoCE RDMA driver. The bnxt_re driver was tentatively set to be pulled in this release cycle, but it simply wasn't ready in time and was dropped (a few review comments still to address, and some multi-arch build issues like prefetch() not working across all arches). Summary: - shared mlx5 updates with net stack (will drop out on merge if Dave's tree has already been merged) - driver updates: cxgb4, hfi1, hns-roce, i40iw, mlx4, mlx5, qedr, rxe - debug cleanups - new connection rejection helpers - SRP updates - various misc fixes - new paravirt driver from vmware" * tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma: (210 commits) IB: Add vmw_pvrdma driver IB/mlx4: fix improper return value IB/ocrdma: fix bad initialization infiniband: nes: return value of skb_linearize should be handled MAINTAINERS: Update Intel RDMA RNIC driver maintainers MAINTAINERS: Remove Mitesh Ahuja from emulex maintainers IB/core: fix unmap_sg argument qede: fix general protection fault may occur on probe IB/mthca: Replace pci_pool_alloc by pci_pool_zalloc mlx5, calc_sq_size(): Make a debug message more informative mlx5: Remove a set-but-not-used variable mlx5: Use { } instead of { 0 } to init struct IB/srp: Make writing the add_target sysfs attr interruptible IB/srp: Make mapping failures easier to debug IB/srp: Make login failures easier to debug IB/srp: Introduce a local variable in srp_add_one() IB/srp: Fix CONFIG_DYNAMIC_DEBUG=n build IB/multicast: Check ib_find_pkey() return value IPoIB: Avoid reading an uninitialized member variable IB/mad: Fix an array index check ...

Linus Torvalds committed 9 years ago

Commit 4d5b57e2 parents 6df8b74 6f94ba2203 files · +12276 −2656Browse files

Unified · Split

203 files
  1. MAINTAINERS +8 −5
  2. drivers/infiniband/Kconfig +1 −0
  3. drivers/infiniband/core/agent.c +0 −1
  4. drivers/infiniband/core/cache.c +7 −9
  5. drivers/infiniband/core/cm.c +71 −1
  6. drivers/infiniband/core/cma.c +43 −0
  7. drivers/infiniband/core/core_priv.h +0 −3
  8. drivers/infiniband/core/device.c +1 −4
  9. drivers/infiniband/core/fmr_pool.c +0 −1
  10. drivers/infiniband/core/iwcm.c +21 −0
  11. drivers/infiniband/core/iwpm_msg.c +0 −1
  12. drivers/infiniband/core/iwpm_util.c +4 −8
  13. drivers/infiniband/core/mad.c +9 −37
  14. drivers/infiniband/core/multicast.c +5 −2
  15. drivers/infiniband/core/roce_gid_mgmt.c +6 −15
  16. drivers/infiniband/core/ucm.c +4 −1
  17. drivers/infiniband/core/ucma.c +4 −1
  18. drivers/infiniband/core/umem.c +1 −1
  19. drivers/infiniband/core/uverbs.h +1 −0
  20. drivers/infiniband/core/uverbs_cmd.c +155 −74
  21. drivers/infiniband/core/uverbs_main.c +5 −1
  22. drivers/infiniband/core/verbs.c +57 −51
  23. drivers/infiniband/hw/Makefile +1 −0
  24. drivers/infiniband/hw/cxgb3/cxio_dbg.c +8 −12
  25. drivers/infiniband/hw/cxgb3/iwch_provider.c +2 −1
  26. drivers/infiniband/hw/cxgb4/device.c +3 −5
  27. drivers/infiniband/hw/cxgb4/provider.c +3 −1
  28. drivers/infiniband/hw/hfi1/affinity.c +2 −1
  29. drivers/infiniband/hw/hfi1/affinity.h +1 −8
  30. drivers/infiniband/hw/hfi1/chip.c +6 −3
  31. drivers/infiniband/hw/hfi1/chip_registers.h +3 −0
  32. drivers/infiniband/hw/hfi1/debugfs.c +110 −0
  33. drivers/infiniband/hw/hfi1/driver.c +1 −2
  34. drivers/infiniband/hw/hfi1/eprom.c +203 −8
  35. drivers/infiniband/hw/hfi1/firmware.c +133 −23
  36. drivers/infiniband/hw/hfi1/hfi.h +87 −57
  37. drivers/infiniband/hw/hfi1/iowait.h +8 −0
  38. drivers/infiniband/hw/hfi1/mad.c +16 −15
  39. drivers/infiniband/hw/hfi1/mmu_rb.c +1 −1
  40. drivers/infiniband/hw/hfi1/pio.c +13 −27
  41. drivers/infiniband/hw/hfi1/pio.h +20 −18
  42. drivers/infiniband/hw/hfi1/pio_copy.c +11 −11
  43. drivers/infiniband/hw/hfi1/platform.c +167 −26
  44. drivers/infiniband/hw/hfi1/platform.h +117 −10
  45. drivers/infiniband/hw/hfi1/qp.c +8 −3
  46. drivers/infiniband/hw/hfi1/rc.c +13 −47
  47. drivers/infiniband/hw/hfi1/ruc.c +7 −37
  48. drivers/infiniband/hw/hfi1/sdma.c +9 −9
  49. drivers/infiniband/hw/hfi1/sdma.h +9 −3
  50. drivers/infiniband/hw/hfi1/uc.c +2 −2
  51. drivers/infiniband/hw/hfi1/ud.c +2 −2
  52. drivers/infiniband/hw/hfi1/user_sdma.c +43 −17
  53. drivers/infiniband/hw/hfi1/verbs.c +174 −35
  54. drivers/infiniband/hw/hfi1/verbs.h +7 −9
  55. drivers/infiniband/hw/hfi1/verbs_txreq.c +7 −6
  56. drivers/infiniband/hw/hfi1/verbs_txreq.h +1 −0
  57. drivers/infiniband/hw/hns/hns_roce_ah.c +2 −1
  58. drivers/infiniband/hw/hns/hns_roce_alloc.c +7 −4
  59. drivers/infiniband/hw/hns/hns_roce_cmd.c +4 −4
  60. drivers/infiniband/hw/hns/hns_roce_cmd.h +1 −11
  61. drivers/infiniband/hw/hns/hns_roce_common.h +42 −2
  62. drivers/infiniband/hw/hns/hns_roce_cq.c +30 −16
  63. drivers/infiniband/hw/hns/hns_roce_device.h +46 −20
  64. drivers/infiniband/hw/hns/hns_roce_eq.c +3 −3
  65. drivers/infiniband/hw/hns/hns_roce_hem.c +3 −3
  66. drivers/infiniband/hw/hns/hns_roce_hw_v1.c +1071 −151
  67. drivers/infiniband/hw/hns/hns_roce_hw_v1.h +73 −1
  68. drivers/infiniband/hw/hns/hns_roce_main.c +73 −256
  69. drivers/infiniband/hw/hns/hns_roce_mr.c +26 −17
  70. drivers/infiniband/hw/hns/hns_roce_pd.c +3 −2
  71. drivers/infiniband/hw/hns/hns_roce_qp.c +2 −2
  72. drivers/infiniband/hw/i40iw/i40iw.h +30 −7
  73. drivers/infiniband/hw/i40iw/i40iw_cm.c +290 −87
  74. drivers/infiniband/hw/i40iw/i40iw_cm.h +8 −3
  75. drivers/infiniband/hw/i40iw/i40iw_ctrl.c +424 −222
  76. drivers/infiniband/hw/i40iw/i40iw_d.h +17 −8
  77. drivers/infiniband/hw/i40iw/i40iw_hw.c +45 −16
  78. drivers/infiniband/hw/i40iw/i40iw_main.c +129 −37
  79. drivers/infiniband/hw/i40iw/i40iw_osdep.h +5 −3
  80. drivers/infiniband/hw/i40iw/i40iw_p.h +18 −3
  81. drivers/infiniband/hw/i40iw/i40iw_pble.c +0 −4
  82. drivers/infiniband/hw/i40iw/i40iw_puda.c +162 −109
  83. drivers/infiniband/hw/i40iw/i40iw_puda.h +13 −7
  84. drivers/infiniband/hw/i40iw/i40iw_type.h +65 −33
  85. drivers/infiniband/hw/i40iw/i40iw_uk.c +15 −25
  86. drivers/infiniband/hw/i40iw/i40iw_user.h +4 −10
  87. drivers/infiniband/hw/i40iw/i40iw_utils.c +229 −60
  88. drivers/infiniband/hw/i40iw/i40iw_verbs.c +192 −46
  89. drivers/infiniband/hw/i40iw/i40iw_verbs.h +2 −0
  90. drivers/infiniband/hw/i40iw/i40iw_virtchnl.c +18 −15
  91. drivers/infiniband/hw/mlx4/ah.c +7 −3
  92. drivers/infiniband/hw/mlx4/alias_GUID.c +1 −3
  93. drivers/infiniband/hw/mlx4/cm.c +1 −3
  94. drivers/infiniband/hw/mlx4/mad.c +44 −14
  95. drivers/infiniband/hw/mlx4/main.c +24 −25
  96. drivers/infiniband/hw/mlx4/mcg.c +1 −4
  97. drivers/infiniband/hw/mlx4/mlx4_ib.h +2 −1
  98. drivers/infiniband/hw/mlx4/qp.c +8 −5
  99. drivers/infiniband/hw/mlx5/ah.c +24 −1
  100. drivers/infiniband/hw/mlx5/cq.c +31 −3
  101. drivers/infiniband/hw/mlx5/main.c +180 −88
  102. drivers/infiniband/hw/mlx5/mem.c +6 −1
  103. drivers/infiniband/hw/mlx5/mlx5_ib.h +9 −3
  104. drivers/infiniband/hw/mlx5/mr.c +52 −19
  105. drivers/infiniband/hw/mlx5/qp.c +106 −25
  106. drivers/infiniband/hw/mlx5/srq.c +2 −4
  107. drivers/infiniband/hw/mthca/mthca_av.c +2 −4
  108. drivers/infiniband/hw/mthca/mthca_provider.c +3 −1
  109. drivers/infiniband/hw/mthca/mthca_reset.c +0 −4
  110. drivers/infiniband/hw/nes/nes.c +0 −1
  111. drivers/infiniband/hw/nes/nes_cm.c +1 −3
  112. drivers/infiniband/hw/nes/nes_hw.c +2 −4
  113. drivers/infiniband/hw/nes/nes_mgt.c +3 −7
  114. drivers/infiniband/hw/nes/nes_nic.c +48 −36
  115. drivers/infiniband/hw/nes/nes_verbs.c +2 −5
  116. drivers/infiniband/hw/ocrdma/ocrdma_ah.c +2 −1
  117. drivers/infiniband/hw/ocrdma/ocrdma_ah.h +3 −1
  118. drivers/infiniband/hw/ocrdma/ocrdma_hw.c +3 −4
  119. drivers/infiniband/hw/ocrdma/ocrdma_stats.c +1 −3
  120. drivers/infiniband/hw/qedr/verbs.c +14 −13
  121. drivers/infiniband/hw/qedr/verbs.h +2 −1
  122. drivers/infiniband/hw/qib/qib_diag.c +1 −5
  123. drivers/infiniband/hw/qib/qib_driver.c +1 −2
  124. drivers/infiniband/hw/qib/qib_eeprom.c +1 −5
  125. drivers/infiniband/hw/qib/qib_file_ops.c +4 −1
  126. drivers/infiniband/hw/qib/qib_iba6120.c +1 −7
  127. drivers/infiniband/hw/qib/qib_iba7220.c +1 −7
  128. drivers/infiniband/hw/qib/qib_iba7322.c +5 −17
  129. drivers/infiniband/hw/qib/qib_init.c +8 −39
  130. drivers/infiniband/hw/qib/qib_rc.c +5 −39
  131. drivers/infiniband/hw/qib/qib_ruc.c +2 −22
  132. drivers/infiniband/hw/qib/qib_verbs.c +8 −25
  133. drivers/infiniband/hw/usnic/usnic_ib_qp_grp.c +7 −15
  134. drivers/infiniband/hw/usnic/usnic_ib_verbs.c +9 −7
  135. drivers/infiniband/hw/usnic/usnic_ib_verbs.h +3 −1
  136. drivers/infiniband/hw/usnic/usnic_vnic.c +6 −16
  137. drivers/infiniband/hw/vmw_pvrdma/Kconfig +7 −0
  138. drivers/infiniband/hw/vmw_pvrdma/Makefile +3 −0
  139. drivers/infiniband/hw/vmw_pvrdma/pvrdma.h +474 −0
  140. drivers/infiniband/hw/vmw_pvrdma/pvrdma_cmd.c +119 −0
  141. drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c +425 −0
  142. drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h +586 −0
  143. drivers/infiniband/hw/vmw_pvrdma/pvrdma_doorbell.c +127 −0
  144. drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c +1211 −0
  145. drivers/infiniband/hw/vmw_pvrdma/pvrdma_misc.c +304 −0
  146. drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c +334 −0
  147. drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c +972 −0
  148. drivers/infiniband/hw/vmw_pvrdma/pvrdma_ring.h +131 −0
  149. drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c +579 −0
  150. drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h +436 −0
  151. drivers/infiniband/sw/rdmavt/cq.c +27 −37
  152. drivers/infiniband/sw/rdmavt/mcast.c +2 −3
  153. drivers/infiniband/sw/rdmavt/mr.c +13 −9
  154. drivers/infiniband/sw/rdmavt/qp.c +19 −1
  155. drivers/infiniband/sw/rdmavt/trace.h +4 −137
  156. drivers/infiniband/sw/rdmavt/trace_mr.h +112 −0
  157. drivers/infiniband/sw/rdmavt/trace_qp.h +96 −0
  158. drivers/infiniband/sw/rdmavt/trace_rvt.h +81 −0
  159. drivers/infiniband/sw/rdmavt/trace_tx.h +132 −0
  160. drivers/infiniband/sw/rxe/rxe_comp.c +7 −2
  161. drivers/infiniband/sw/rxe/rxe_loc.h +0 −2
  162. drivers/infiniband/sw/rxe/rxe_mr.c +3 −0
  163. drivers/infiniband/sw/rxe/rxe_net.c +3 −5
  164. drivers/infiniband/sw/rxe/rxe_param.h +1 −1
  165. drivers/infiniband/sw/rxe/rxe_pool.c +0 −1
  166. drivers/infiniband/sw/rxe/rxe_recv.c +5 −6
  167. drivers/infiniband/sw/rxe/rxe_req.c +11 −9
  168. drivers/infiniband/sw/rxe/rxe_resp.c +21 −4
  169. drivers/infiniband/sw/rxe/rxe_srq.c +1 −1
  170. drivers/infiniband/sw/rxe/rxe_task.c +19 −0
  171. drivers/infiniband/sw/rxe/rxe_task.h +1 −0
  172. drivers/infiniband/sw/rxe/rxe_verbs.c +16 −5
  173. drivers/infiniband/ulp/ipoib/ipoib_cm.c +1 −9
  174. drivers/infiniband/ulp/ipoib/ipoib_ib.c +1 −4
  175. drivers/infiniband/ulp/ipoib/ipoib_main.c +1 −4
  176. drivers/infiniband/ulp/ipoib/ipoib_multicast.c +5 −2
  177. drivers/infiniband/ulp/iser/iser_verbs.c +4 −1
  178. drivers/infiniband/ulp/isert/ib_isert.c +14 −17
  179. drivers/infiniband/ulp/srp/ib_srp.c +32 −16
  180. drivers/infiniband/ulp/srpt/ib_srpt.c +9 −13
  181. drivers/net/ethernet/qlogic/qede/qede_roce.c +2 −2
  182. drivers/net/vmxnet3/vmxnet3_int.h +1 −2
  183. drivers/nvme/host/rdma.c +36 −6
  184. drivers/nvme/target/rdma.c +3 −0
  185. include/linux/mlx5/mlx5_ifc.h +1 −1
  186. include/linux/pci_ids.h +1 −0
  187. include/rdma/ib_cm.h +6 −0
  188. include/rdma/ib_mad.h +1 −1
  189. include/rdma/ib_verbs.h +56 −14
  190. include/rdma/iw_cm.h +6 −0
  191. include/rdma/opa_smi.h +0 −2
  192. include/rdma/rdma_cm.h +25 −0
  193. include/rdma/rdma_vt.h +24 −22
  194. include/rdma/rdmavt_mr.h +7 −3
  195. include/rdma/rdmavt_qp.h +77 −0
  196. include/uapi/rdma/Kbuild +2 −0
  197. include/uapi/rdma/hfi/hfi1_user.h +1 −1
  198. include/uapi/rdma/hns-abi.h +5 −4
  199. include/uapi/rdma/ib_user_verbs.h +38 −0
  200. include/uapi/rdma/mlx5-abi.h +37 −1
  201. include/uapi/rdma/rdma_user_cm.h +6 −6
  202. include/uapi/rdma/vmw_pvrdma-abi.h +289 −0
  203. net/rds/rdma_transport.c +4 −1
MAINTAINERS+8−5
@@ -6515,10 +6515,7 @@ F: drivers/net/ethernet/intel/*/ · 6514 unmodified lines
65156515
65166516 INTEL RDMA RNIC DRIVER
65176517 M: Faisal Latif <faisal.latif@intel.com>
6518−R: Chien Tin Tung <chien.tin.tung@intel.com>
6519−R: Mustafa Ismail <mustafa.ismail@intel.com>
6520−R: Shiraz Saleem <shiraz.saleem@intel.com>
6521−R: Tatyana Nikolova <tatyana.e.nikolova@intel.com>
6518+M: Shiraz Saleem <shiraz.saleem@intel.com>
65226519 L: linux-rdma@vger.kernel.org
65236520 S: Supported
65246521 F: drivers/infiniband/hw/i40iw/
@@ -11078,7 +11075,6 @@ F: drivers/net/ethernet/emulex/benet/ · 4553 unmodified lines
1107811075 EMULEX ONECONNECT ROCE DRIVER
1107911076 M: Selvin Xavier <selvin.xavier@avagotech.com>
1108011077 M: Devesh Sharma <devesh.sharma@avagotech.com>
11081−M: Mitesh Ahuja <mitesh.ahuja@avagotech.com>
1108211078 L: linux-rdma@vger.kernel.org
1108311079 W: http://www.emulex.com
1108411080 S: Supported
@@ -13129,6 +13125,13 @@ S: Maintained · 2044 unmodified lines
1312913125 F: drivers/scsi/vmw_pvscsi.c
1313013126 F: drivers/scsi/vmw_pvscsi.h
1313113127
13128+VMWARE PVRDMA DRIVER
13129+M: Adit Ranadive <aditr@vmware.com>
13130+M: VMware PV-Drivers <pv-drivers@vmware.com>
13131+L: linux-rdma@vger.kernel.org
13132+S: Maintained
13133+F: drivers/infiniband/hw/vmw_pvrdma/
13134+
1313213135 VOLTAGE AND CURRENT REGULATOR FRAMEWORK
1313313136 M: Liam Girdwood <lgirdwood@gmail.com>
1313413137 M: Mark Brown <broonie@kernel.org>
drivers/infiniband/Kconfig+1−0
@@ -73,6 +73,7 @@ source "drivers/infiniband/hw/mlx4/Kconfig" · 72 unmodified lines
7373 source "drivers/infiniband/hw/mlx5/Kconfig"
7474 source "drivers/infiniband/hw/nes/Kconfig"
7575 source "drivers/infiniband/hw/ocrdma/Kconfig"
76+source "drivers/infiniband/hw/vmw_pvrdma/Kconfig"
7677 source "drivers/infiniband/hw/usnic/Kconfig"
7778 source "drivers/infiniband/hw/hns/Kconfig"
7879
drivers/infiniband/core/agent.c+0−1
@@ -156,7 +156,6 @@ int ib_agent_port_open(struct ib_device *device, int port_num) · 155 unmodified lines
156156 /* Create new device info */
157157 port_priv = kzalloc(sizeof *port_priv, GFP_KERNEL);
158158 if (!port_priv) {
159− dev_err(&device->dev, "No memory for ib_agent_port_private\n");
160159 ret = -ENOMEM;
161160 goto error1;
162161 }
drivers/infiniband/core/cache.c+7−9
@@ -770,12 +770,8 @@ static int _gid_table_setup_one(struct ib_device *ib_dev) · 769 unmodified lines
770770 int err = 0;
771771
772772 table = kcalloc(ib_dev->phys_port_cnt, sizeof(*table), GFP_KERNEL);
773−
774− if (!table) {
775− pr_warn("failed to allocate ib gid cache for %s\n",
776− ib_dev->name);
773+ if (!table)
777774 return -ENOMEM;
778− }
779775
780776 for (port = 0; port < ib_dev->phys_port_cnt; port++) {
781777 u8 rdma_port = port + rdma_start_port(ib_dev);
@@ -1170,14 +1166,13 @@ int ib_cache_setup_one(struct ib_device *device) · 388 unmodified lines
11701166 GFP_KERNEL);
11711167 if (!device->cache.pkey_cache ||
11721168 !device->cache.lmc_cache) {
1173− pr_warn("Couldn't allocate cache for %s\n", device->name);
1174− return -ENOMEM;
1169+ err = -ENOMEM;
1170+ goto free;
11751171 }
11761172
11771173 err = gid_table_setup_one(device);
11781174 if (err)
1179− /* Allocated memory will be cleaned in the release function */
1180− return err;
1175+ goto free;
11811176
11821177 for (p = 0; p <= rdma_end_port(device) - rdma_start_port(device); ++p)
11831178 ib_cache_update(device, p + rdma_start_port(device));
@@ -1192,6 +1187,9 @@ int ib_cache_setup_one(struct ib_device *device) · 8 unmodified lines
11921187
11931188 err:
11941189 gid_table_cleanup_one(device);
1190+free:
1191+ kfree(device->cache.pkey_cache);
1192+ kfree(device->cache.lmc_cache);
11951193 return err;
11961194 }
11971195
drivers/infiniband/core/cm.c+71−1
@@ -57,6 +57,54 @@ MODULE_AUTHOR("Sean Hefty"); · 56 unmodified lines
5757 MODULE_DESCRIPTION("InfiniBand CM");
5858 MODULE_LICENSE("Dual BSD/GPL");
5959
60+static const char * const ibcm_rej_reason_strs[] = {
61+ [IB_CM_REJ_NO_QP] = "no QP",
62+ [IB_CM_REJ_NO_EEC] = "no EEC",
63+ [IB_CM_REJ_NO_RESOURCES] = "no resources",
64+ [IB_CM_REJ_TIMEOUT] = "timeout",
65+ [IB_CM_REJ_UNSUPPORTED] = "unsupported",
66+ [IB_CM_REJ_INVALID_COMM_ID] = "invalid comm ID",
67+ [IB_CM_REJ_INVALID_COMM_INSTANCE] = "invalid comm instance",
68+ [IB_CM_REJ_INVALID_SERVICE_ID] = "invalid service ID",
69+ [IB_CM_REJ_INVALID_TRANSPORT_TYPE] = "invalid transport type",
70+ [IB_CM_REJ_STALE_CONN] = "stale conn",
71+ [IB_CM_REJ_RDC_NOT_EXIST] = "RDC not exist",
72+ [IB_CM_REJ_INVALID_GID] = "invalid GID",
73+ [IB_CM_REJ_INVALID_LID] = "invalid LID",
74+ [IB_CM_REJ_INVALID_SL] = "invalid SL",
75+ [IB_CM_REJ_INVALID_TRAFFIC_CLASS] = "invalid traffic class",
76+ [IB_CM_REJ_INVALID_HOP_LIMIT] = "invalid hop limit",
77+ [IB_CM_REJ_INVALID_PACKET_RATE] = "invalid packet rate",
78+ [IB_CM_REJ_INVALID_ALT_GID] = "invalid alt GID",
79+ [IB_CM_REJ_INVALID_ALT_LID] = "invalid alt LID",
80+ [IB_CM_REJ_INVALID_ALT_SL] = "invalid alt SL",
81+ [IB_CM_REJ_INVALID_ALT_TRAFFIC_CLASS] = "invalid alt traffic class",
82+ [IB_CM_REJ_INVALID_ALT_HOP_LIMIT] = "invalid alt hop limit",
83+ [IB_CM_REJ_INVALID_ALT_PACKET_RATE] = "invalid alt packet rate",
84+ [IB_CM_REJ_PORT_CM_REDIRECT] = "port CM redirect",
85+ [IB_CM_REJ_PORT_REDIRECT] = "port redirect",
86+ [IB_CM_REJ_INVALID_MTU] = "invalid MTU",
87+ [IB_CM_REJ_INSUFFICIENT_RESP_RESOURCES] = "insufficient resp resources",
88+ [IB_CM_REJ_CONSUMER_DEFINED] = "consumer defined",
89+ [IB_CM_REJ_INVALID_RNR_RETRY] = "invalid RNR retry",
90+ [IB_CM_REJ_DUPLICATE_LOCAL_COMM_ID] = "duplicate local comm ID",
91+ [IB_CM_REJ_INVALID_CLASS_VERSION] = "invalid class version",
92+ [IB_CM_REJ_INVALID_FLOW_LABEL] = "invalid flow label",
93+ [IB_CM_REJ_INVALID_ALT_FLOW_LABEL] = "invalid alt flow label",
94+};
95+
96+const char *__attribute_const__ ibcm_reject_msg(int reason)
97+{
98+ size_t index = reason;
99+
100+ if (index < ARRAY_SIZE(ibcm_rej_reason_strs) &&
101+ ibcm_rej_reason_strs[index])
102+ return ibcm_rej_reason_strs[index];
103+ else
104+ return "unrecognized reason";
105+}
106+EXPORT_SYMBOL(ibcm_reject_msg);
107+
60108 static void cm_add_one(struct ib_device *device);
61109 static void cm_remove_one(struct ib_device *device, void *client_data);
62110
@@ -1582,6 +1630,7 @@ static struct cm_id_private * cm_match_req(struct cm_work *work, · 1519 unmodified lines
15821630 struct cm_id_private *listen_cm_id_priv, *cur_cm_id_priv;
15831631 struct cm_timewait_info *timewait_info;
15841632 struct cm_req_msg *req_msg;
1633+ struct ib_cm_id *cm_id;
15851634
15861635 req_msg = (struct cm_req_msg *)work->mad_recv_wc->recv_buf.mad;
15871636
@@ -1603,10 +1652,18 @@ static struct cm_id_private * cm_match_req(struct cm_work *work, · 15 unmodified lines
16031652 timewait_info = cm_insert_remote_qpn(cm_id_priv->timewait_info);
16041653 if (timewait_info) {
16051654 cm_cleanup_timewait(cm_id_priv->timewait_info);
1655+ cur_cm_id_priv = cm_get_id(timewait_info->work.local_id,
1656+ timewait_info->work.remote_id);
1657+
16061658 spin_unlock_irq(&cm.lock);
16071659 cm_issue_rej(work->port, work->mad_recv_wc,
16081660 IB_CM_REJ_STALE_CONN, CM_MSG_RESPONSE_REQ,
16091661 NULL, 0);
1662+ if (cur_cm_id_priv) {
1663+ cm_id = &cur_cm_id_priv->id;
1664+ ib_send_cm_dreq(cm_id, NULL, 0);
1665+ cm_deref_id(cur_cm_id_priv);
1666+ }
16101667 return NULL;
16111668 }
16121669
@@ -1984,6 +2041,9 @@ static int cm_rep_handler(struct cm_work *work) · 371 unmodified lines
19842041 struct cm_id_private *cm_id_priv;
19852042 struct cm_rep_msg *rep_msg;
19862043 int ret;
2044+ struct cm_id_private *cur_cm_id_priv;
2045+ struct ib_cm_id *cm_id;
2046+ struct cm_timewait_info *timewait_info;
19872047
19882048 rep_msg = (struct cm_rep_msg *)work->mad_recv_wc->recv_buf.mad;
19892049 cm_id_priv = cm_acquire_id(rep_msg->remote_comm_id, 0);
@@ -2018,16 +2078,26 @@ static int cm_rep_handler(struct cm_work *work) · 28 unmodified lines
20182078 goto error;
20192079 }
20202080 /* Check for a stale connection. */
2021− if (cm_insert_remote_qpn(cm_id_priv->timewait_info)) {
2081+ timewait_info = cm_insert_remote_qpn(cm_id_priv->timewait_info);
2082+ if (timewait_info) {
20222083 rb_erase(&cm_id_priv->timewait_info->remote_id_node,
20232084 &cm.remote_id_table);
20242085 cm_id_priv->timewait_info->inserted_remote_id = 0;
2086+ cur_cm_id_priv = cm_get_id(timewait_info->work.local_id,
2087+ timewait_info->work.remote_id);
2088+
20252089 spin_unlock(&cm.lock);
20262090 spin_unlock_irq(&cm_id_priv->lock);
20272091 cm_issue_rej(work->port, work->mad_recv_wc,
20282092 IB_CM_REJ_STALE_CONN, CM_MSG_RESPONSE_REP,
20292093 NULL, 0);
20302094 ret = -EINVAL;
2095+ if (cur_cm_id_priv) {
2096+ cm_id = &cur_cm_id_priv->id;
2097+ ib_send_cm_dreq(cm_id, NULL, 0);
2098+ cm_deref_id(cur_cm_id_priv);
2099+ }
2100+
20312101 goto error;
20322102 }
20332103 spin_unlock(&cm.lock);
drivers/infiniband/core/cma.c+43−0
@@ -101,6 +101,49 @@ const char *__attribute_const__ rdma_event_msg(enum rdma_cm_event_type event) · 100 unmodified lines
101101 }
102102 EXPORT_SYMBOL(rdma_event_msg);
103103
104+const char *__attribute_const__ rdma_reject_msg(struct rdma_cm_id *id,
105+ int reason)
106+{
107+ if (rdma_ib_or_roce(id->device, id->port_num))
108+ return ibcm_reject_msg(reason);
109+
110+ if (rdma_protocol_iwarp(id->device, id->port_num))
111+ return iwcm_reject_msg(reason);
112+
113+ WARN_ON_ONCE(1);
114+ return "unrecognized transport";
115+}
116+EXPORT_SYMBOL(rdma_reject_msg);
117+
118+bool rdma_is_consumer_reject(struct rdma_cm_id *id, int reason)
119+{
120+ if (rdma_ib_or_roce(id->device, id->port_num))
121+ return reason == IB_CM_REJ_CONSUMER_DEFINED;
122+
123+ if (rdma_protocol_iwarp(id->device, id->port_num))
124+ return reason == -ECONNREFUSED;
125+
126+ WARN_ON_ONCE(1);
127+ return false;
128+}
129+EXPORT_SYMBOL(rdma_is_consumer_reject);
130+
131+const void *rdma_consumer_reject_data(struct rdma_cm_id *id,
132+ struct rdma_cm_event *ev, u8 *data_len)
133+{
134+ const void *p;
135+
136+ if (rdma_is_consumer_reject(id, ev->status)) {
137+ *data_len = ev->param.conn.private_data_len;
138+ p = ev->param.conn.private_data;
139+ } else {
140+ *data_len = 0;
141+ p = NULL;
142+ }
143+ return p;
144+}
145+EXPORT_SYMBOL(rdma_consumer_reject_data);
146+
104147 static void cma_add_one(struct ib_device *device);
105148 static void cma_remove_one(struct ib_device *device, void *client_data);
106149
drivers/infiniband/core/core_priv.h+0−3
@@ -72,9 +72,6 @@ void ib_device_unregister_sysfs(struct ib_device *device); · 71 unmodified lines
7272 void ib_cache_setup(void);
7373 void ib_cache_cleanup(void);
7474
75−int ib_resolve_eth_dmac(struct ib_qp *qp,
76− struct ib_qp_attr *qp_attr, int *qp_attr_mask);
77−
7875 typedef void (*roce_netdev_callback)(struct ib_device *device, u8 port,
7976 struct net_device *idev, void *cookie);
8077
drivers/infiniband/core/device.c+1−4
@@ -254,11 +254,8 @@ static int add_client_context(struct ib_device *device, struct ib_client *client · 253 unmodified lines
254254 unsigned long flags;
255255
256256 context = kmalloc(sizeof *context, GFP_KERNEL);
257− if (!context) {
258− pr_warn("Couldn't allocate client context for %s/%s\n",
259− device->name, client->name);
257+ if (!context)
260258 return -ENOMEM;
261− }
262259
263260 context->client = client;
264261 context->data = NULL;
drivers/infiniband/core/fmr_pool.c+0−1
@@ -247,7 +247,6 @@ struct ib_fmr_pool *ib_create_fmr_pool(struct ib_pd *pd, · 246 unmodified lines
247247 kmalloc(IB_FMR_HASH_SIZE * sizeof *pool->cache_bucket,
248248 GFP_KERNEL);
249249 if (!pool->cache_bucket) {
250− pr_warn(PFX "Failed to allocate cache in pool\n");
251250 ret = -ENOMEM;
252251 goto out_free_pool;
253252 }
drivers/infiniband/core/iwcm.c+21−0
@@ -59,6 +59,27 @@ MODULE_AUTHOR("Tom Tucker"); · 58 unmodified lines
5959 MODULE_DESCRIPTION("iWARP CM");
6060 MODULE_LICENSE("Dual BSD/GPL");
6161
62+static const char * const iwcm_rej_reason_strs[] = {
63+ [ECONNRESET] = "reset by remote host",
64+ [ECONNREFUSED] = "refused by remote application",
65+ [ETIMEDOUT] = "setup timeout",
66+};
67+
68+const char *__attribute_const__ iwcm_reject_msg(int reason)
69+{
70+ size_t index;
71+
72+ /* iWARP uses negative errnos */
73+ index = -reason;
74+
75+ if (index < ARRAY_SIZE(iwcm_rej_reason_strs) &&
76+ iwcm_rej_reason_strs[index])
77+ return iwcm_rej_reason_strs[index];
78+ else
79+ return "unrecognized reason";
80+}
81+EXPORT_SYMBOL(iwcm_reject_msg);
82+
6283 static struct ibnl_client_cbs iwcm_nl_cb_table[] = {
6384 [RDMA_NL_IWPM_REG_PID] = {.dump = iwpm_register_pid_cb},
6485 [RDMA_NL_IWPM_ADD_MAPPING] = {.dump = iwpm_add_mapping_cb},
drivers/infiniband/core/iwpm_msg.c+0−1
@@ -604,7 +604,6 @@ int iwpm_remote_info_cb(struct sk_buff *skb, struct netlink_callback *cb) · 603 unmodified lines
604604 }
605605 rem_info = kzalloc(sizeof(struct iwpm_remote_info), GFP_ATOMIC);
606606 if (!rem_info) {
607− pr_err("%s: Unable to allocate a remote info\n", __func__);
608607 ret = -ENOMEM;
609608 return ret;
610609 }
drivers/infiniband/core/iwpm_util.c+4−8
@@ -62,7 +62,6 @@ int iwpm_init(u8 nl_client) · 61 unmodified lines
6262 sizeof(struct hlist_head), GFP_KERNEL);
6363 if (!iwpm_hash_bucket) {
6464 ret = -ENOMEM;
65− pr_err("%s Unable to create mapinfo hash table\n", __func__);
6665 goto init_exit;
6766 }
6867 iwpm_reminfo_bucket = kzalloc(IWPM_REMINFO_HASH_SIZE *
@@ -70,7 +69,6 @@ int iwpm_init(u8 nl_client) · 1 unmodified line
7069 if (!iwpm_reminfo_bucket) {
7170 kfree(iwpm_hash_bucket);
7271 ret = -ENOMEM;
73− pr_err("%s Unable to create reminfo hash table\n", __func__);
7472 goto init_exit;
7573 }
7674 }
@@ -128,10 +126,9 @@ int iwpm_create_mapinfo(struct sockaddr_storage *local_sockaddr, · 51 unmodified lines
128126 if (!iwpm_valid_client(nl_client))
129127 return ret;
130128 map_info = kzalloc(sizeof(struct iwpm_mapping_info), GFP_KERNEL);
131− if (!map_info) {
132− pr_err("%s: Unable to allocate a mapping info\n", __func__);
129+ if (!map_info)
133130 return -ENOMEM;
134− }
131+
135132 memcpy(&map_info->local_sockaddr, local_sockaddr,
136133 sizeof(struct sockaddr_storage));
137134 memcpy(&map_info->mapped_sockaddr, mapped_sockaddr,
@@ -309,10 +306,9 @@ struct iwpm_nlmsg_request *iwpm_get_nlmsg_request(__u32 nlmsg_seq, · 171 unmodified lines
309306 unsigned long flags;
310307
311308 nlmsg_request = kzalloc(sizeof(struct iwpm_nlmsg_request), gfp);
312− if (!nlmsg_request) {
313− pr_err("%s Unable to allocate a nlmsg_request\n", __func__);
309+ if (!nlmsg_request)
314310 return NULL;
315− }
311+
316312 spin_lock_irqsave(&iwpm_nlmsg_req_lock, flags);
317313 list_add_tail(&nlmsg_request->inprocess_list, &iwpm_nlmsg_req_list);
318314 spin_unlock_irqrestore(&iwpm_nlmsg_req_lock, flags);
drivers/infiniband/core/mad.c+9−37
@@ -769,7 +769,7 @@ static int handle_outgoing_dr_smp(struct ib_mad_agent_private *mad_agent_priv, · 768 unmodified lines
769769 * If we are at the start of the LID routed part, don't update the
770770 * hop_ptr or hop_cnt. See section 14.2.2, Vol 1 IB spec.
771771 */
772− if (opa && smp->class_version == OPA_SMP_CLASS_VERSION) {
772+ if (opa && smp->class_version == OPA_SM_CLASS_VERSION) {
773773 u32 opa_drslid;
774774
775775 if ((opa_get_smp_direction(opa_smp)
@@ -816,7 +816,6 @@ static int handle_outgoing_dr_smp(struct ib_mad_agent_private *mad_agent_priv, · 40 unmodified lines
816816 local = kmalloc(sizeof *local, GFP_ATOMIC);
817817 if (!local) {
818818 ret = -ENOMEM;
819− dev_err(&device->dev, "No memory for ib_mad_local_private\n");
820819 goto out;
821820 }
822821 local->mad_priv = NULL;
@@ -824,7 +823,6 @@ static int handle_outgoing_dr_smp(struct ib_mad_agent_private *mad_agent_priv, · 1 unmodified line
824823 mad_priv = alloc_mad_private(mad_size, GFP_ATOMIC);
825824 if (!mad_priv) {
826825 ret = -ENOMEM;
827− dev_err(&device->dev, "No memory for local response MAD\n");
828826 kfree(local);
829827 goto out;
830828 }
@@ -947,9 +945,6 @@ static int alloc_send_rmpp_list(struct ib_mad_send_wr_private *send_wr, · 116 unmodified lines
947945 for (left = send_buf->data_len + pad; left > 0; left -= seg_size) {
948946 seg = kmalloc(sizeof (*seg) + seg_size, gfp_mask);
949947 if (!seg) {
950− dev_err(&send_buf->mad_agent->device->dev,
951− "alloc_send_rmpp_segs: RMPP mem alloc failed for len %zd, gfp %#x\n",
952− sizeof (*seg) + seg_size, gfp_mask);
953948 free_send_rmpp_list(send_wr);
954949 return -ENOMEM;
955950 }
@@ -1362,12 +1357,7 @@ static int allocate_method_table(struct ib_mad_mgmt_method_table **method) · 406 unmodified lines
13621357 {
13631358 /* Allocate management method table */
13641359 *method = kzalloc(sizeof **method, GFP_ATOMIC);
1365− if (!*method) {
1366− pr_err("No memory for ib_mad_mgmt_method_table\n");
1367− return -ENOMEM;
1368− }
1369−
1370− return 0;
1360+ return (*method) ? 0 : (-ENOMEM);
13711361 }
13721362
13731363 /*
@@ -1458,8 +1448,6 @@ static int add_nonoui_reg_req(struct ib_mad_reg_req *mad_reg_req, · 84 unmodified lines
14581448 /* Allocate management class table for "new" class version */
14591449 *class = kzalloc(sizeof **class, GFP_ATOMIC);
14601450 if (!*class) {
1461− dev_err(&agent_priv->agent.device->dev,
1462− "No memory for ib_mad_mgmt_class_table\n");
14631451 ret = -ENOMEM;
14641452 goto error1;
14651453 }
@@ -1524,22 +1512,16 @@ static int add_oui_reg_req(struct ib_mad_reg_req *mad_reg_req, · 58 unmodified lines
15241512 if (!*vendor_table) {
15251513 /* Allocate mgmt vendor class table for "new" class version */
15261514 vendor = kzalloc(sizeof *vendor, GFP_ATOMIC);
1527− if (!vendor) {
1528− dev_err(&agent_priv->agent.device->dev,
1529− "No memory for ib_mad_mgmt_vendor_class_table\n");
1515+ if (!vendor)
15301516 goto error1;
1531− }
15321517
15331518 *vendor_table = vendor;
15341519 }
15351520 if (!(*vendor_table)->vendor_class[vclass]) {
15361521 /* Allocate table for this management vendor class */
15371522 vendor_class = kzalloc(sizeof *vendor_class, GFP_ATOMIC);
1538− if (!vendor_class) {
1539− dev_err(&agent_priv->agent.device->dev,
1540− "No memory for ib_mad_mgmt_vendor_class\n");
1523+ if (!vendor_class)
15411524 goto error2;
1542− }
15431525
15441526 (*vendor_table)->vendor_class[vclass] = vendor_class;
15451527 }
@@ -1746,7 +1728,7 @@ find_mad_agent(struct ib_mad_port_private *port_priv, · 200 unmodified lines
17461728 if (!class)
17471729 goto out;
17481730 if (convert_mgmt_class(mad_hdr->mgmt_class) >=
1749− IB_MGMT_MAX_METHODS)
1731+ ARRAY_SIZE(class->method_table))
17501732 goto out;
17511733 method = class->method_table[convert_mgmt_class(
17521734 mad_hdr->mgmt_class)];
@@ -2167,7 +2149,7 @@ handle_smi(struct ib_mad_port_private *port_priv, · 414 unmodified lines
21672149 struct ib_mad_hdr *mad_hdr = (struct ib_mad_hdr *)recv->mad;
21682150
21692151 if (opa && mad_hdr->base_version == OPA_MGMT_BASE_VERSION &&
2170− mad_hdr->class_version == OPA_SMI_CLASS_VERSION)
2152+ mad_hdr->class_version == OPA_SM_CLASS_VERSION)
21712153 return handle_opa_smi(port_priv, qp_info, wc, port_num, recv,
21722154 response);
21732155
@@ -2238,11 +2220,8 @@ static void ib_mad_recv_done(struct ib_cq *cq, struct ib_wc *wc) · 64 unmodified lines
22382220
22392221 mad_size = recv->mad_size;
22402222 response = alloc_mad_private(mad_size, GFP_KERNEL);
2241− if (!response) {
2242− dev_err(&port_priv->device->dev,
2243− "%s: no memory for response buffer\n", __func__);
2223+ if (!response)
22442224 goto out;
2245− }
22462225
22472226 if (rdma_cap_ib_switch(port_priv->device))
22482227 port_num = wc->port_num;
@@ -2869,8 +2848,6 @@ static int ib_mad_post_receive_mads(struct ib_mad_qp_info *qp_info, · 620 unmodified lines
28692848 mad_priv = alloc_mad_private(port_mad_size(qp_info->port_priv),
28702849 GFP_ATOMIC);
28712850 if (!mad_priv) {
2872− dev_err(&qp_info->port_priv->device->dev,
2873− "No memory for receive buffer\n");
28742851 ret = -ENOMEM;
28752852 break;
28762853 }
@@ -2961,11 +2938,8 @@ static int ib_mad_port_start(struct ib_mad_port_private *port_priv) · 84 unmodified lines
29612938 u16 pkey_index;
29622939
29632940 attr = kmalloc(sizeof *attr, GFP_KERNEL);
2964− if (!attr) {
2965− dev_err(&port_priv->device->dev,
2966− "Couldn't kmalloc ib_qp_attr\n");
2941+ if (!attr)
29672942 return -ENOMEM;
2968− }
29692943
29702944 ret = ib_find_pkey(port_priv->device, port_priv->port_num,
29712945 IB_DEFAULT_PKEY_FULL, &pkey_index);
@@ -3135,10 +3109,8 @@ static int ib_mad_port_open(struct ib_device *device, · 163 unmodified lines
31353109
31363110 /* Create new device info */
31373111 port_priv = kzalloc(sizeof *port_priv, GFP_KERNEL);
3138− if (!port_priv) {
3139− dev_err(&device->dev, "No memory for ib_mad_port_private\n");
3112+ if (!port_priv)
31403113 return -ENOMEM;
3141− }
31423114
31433115 port_priv->device = device;
31443116 port_priv->port_num = port_num;
drivers/infiniband/core/multicast.c+5−2
@@ -518,8 +518,11 @@ static void join_handler(int status, struct ib_sa_mcmember_rec *rec, · 517 unmodified lines
518518 process_join_error(group, status);
519519 else {
520520 int mgids_changed, is_mgid0;
521− ib_find_pkey(group->port->dev->device, group->port->port_num,
522− be16_to_cpu(rec->pkey), &pkey_index);
521+
522+ if (ib_find_pkey(group->port->dev->device,
523+ group->port->port_num, be16_to_cpu(rec->pkey),
524+ &pkey_index))
525+ pkey_index = MCAST_INVALID_PKEY_INDEX;
523526
524527 spin_lock_irq(&group->port->lock);
525528 if (group->state == MCAST_BUSY &&
drivers/infiniband/core/roce_gid_mgmt.c+6−15
@@ -304,10 +304,9 @@ static void enum_netdev_ipv4_ips(struct ib_device *ib_dev, · 303 unmodified lines
304304 for_ifa(in_dev) {
305305 struct sin_list *entry = kzalloc(sizeof(*entry), GFP_ATOMIC);
306306
307− if (!entry) {
308− pr_warn("roce_gid_mgmt: couldn't allocate entry for IPv4 update\n");
307+ if (!entry)
309308 continue;
310− }
309+
311310 entry->ip.sin_family = AF_INET;
312311 entry->ip.sin_addr.s_addr = ifa->ifa_address;
313312 list_add_tail(&entry->list, &sin_list);
@@ -348,10 +347,8 @@ static void enum_netdev_ipv6_ips(struct ib_device *ib_dev, · 34 unmodified lines
348347 list_for_each_entry(ifp, &in6_dev->addr_list, if_list) {
349348 struct sin6_list *entry = kzalloc(sizeof(*entry), GFP_ATOMIC);
350349
351− if (!entry) {
352− pr_warn("roce_gid_mgmt: couldn't allocate entry for IPv6 update\n");
350+ if (!entry)
353351 continue;
354− }
355352
356353 entry->sin6.sin6_family = AF_INET6;
357354 entry->sin6.sin6_addr = ifp->addr;
@@ -447,10 +444,8 @@ static int netdev_upper_walk(struct net_device *upper, void *data) · 89 unmodified lines
447444 struct upper_list *entry = kmalloc(sizeof(*entry), GFP_ATOMIC);
448445 struct list_head *upper_list = data;
449446
450− if (!entry) {
451− pr_info("roce_gid_mgmt: couldn't allocate entry to delete ndev\n");
447+ if (!entry)
452448 return 0;
453− }
454449
455450 list_add_tail(&entry->list, upper_list);
456451 dev_hold(upper);
@@ -559,10 +554,8 @@ static int netdevice_queue_work(struct netdev_event_work_cmd *cmds, · 102 unmodified lines
559554 struct netdev_event_work *ndev_work =
560555 kmalloc(sizeof(*ndev_work), GFP_KERNEL);
561556
562− if (!ndev_work) {
563− pr_warn("roce_gid_mgmt: can't allocate work for netdevice_event\n");
557+ if (!ndev_work)
564558 return NOTIFY_DONE;
565− }
566559
567560 memcpy(ndev_work->cmds, cmds, sizeof(ndev_work->cmds));
568561 for (i = 0; i < ARRAY_SIZE(ndev_work->cmds) && ndev_work->cmds[i].cb; i++) {
@@ -696,10 +689,8 @@ static int addr_event(struct notifier_block *this, unsigned long event, · 127 unmodified lines
696689 }
697690
698691 work = kmalloc(sizeof(*work), GFP_ATOMIC);
699− if (!work) {
700− pr_warn("roce_gid_mgmt: Couldn't allocate work for addr_event\n");
692+ if (!work)
701693 return NOTIFY_DONE;
702− }
703694
704695 INIT_WORK(&work->work, update_gid_event_work_handler);
705696
drivers/infiniband/core/ucm.c+4−1
@@ -1104,8 +1104,11 @@ static ssize_t ib_ucm_write(struct file *filp, const char __user *buf, · 1103 unmodified lines
11041104 struct ib_ucm_cmd_hdr hdr;
11051105 ssize_t result;
11061106
1107− if (WARN_ON_ONCE(!ib_safe_file_access(filp)))
1107+ if (!ib_safe_file_access(filp)) {
1108+ pr_err_once("ucm_write: process %d (%s) changed security contexts after opening file descriptor, this is not allowed.\n",
1109+ task_tgid_vnr(current), current->comm);
11081110 return -EACCES;
1111+ }
11091112
11101113 if (len < sizeof(hdr))
11111114 return -EINVAL;
drivers/infiniband/core/ucma.c+4−1
@@ -1584,8 +1584,11 @@ static ssize_t ucma_write(struct file *filp, const char __user *buf, · 1583 unmodified lines
15841584 struct rdma_ucm_cmd_hdr hdr;
15851585 ssize_t ret;
15861586
1587− if (WARN_ON_ONCE(!ib_safe_file_access(filp)))
1587+ if (!ib_safe_file_access(filp)) {
1588+ pr_err_once("ucma_write: process %d (%s) changed security contexts after opening file descriptor, this is not allowed.\n",
1589+ task_tgid_vnr(current), current->comm);
15881590 return -EACCES;
1591+ }
15891592
15901593 if (len < sizeof(hdr))
15911594 return -EINVAL;
drivers/infiniband/core/umem.c+1−1
@@ -51,7 +51,7 @@ static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int d · 50 unmodified lines
5151
5252 if (umem->nmap > 0)
5353 ib_dma_unmap_sg(dev, umem->sg_head.sgl,
54− umem->nmap,
54+ umem->npages,
5555 DMA_BIDIRECTIONAL);
5656
5757 for_each_sg(umem->sg_head.sgl, sg, umem->npages, i) {
drivers/infiniband/core/uverbs.h+1−0
@@ -289,5 +289,6 @@ IB_UVERBS_DECLARE_EX_CMD(modify_wq); · 288 unmodified lines
289289 IB_UVERBS_DECLARE_EX_CMD(destroy_wq);
290290 IB_UVERBS_DECLARE_EX_CMD(create_rwq_ind_table);
291291 IB_UVERBS_DECLARE_EX_CMD(destroy_rwq_ind_table);
292+IB_UVERBS_DECLARE_EX_CMD(modify_qp);
292293
293294 #endif /* UVERBS_H */
drivers/infiniband/core/uverbs_cmd.c+155−74
@@ -2328,94 +2328,88 @@ static int modify_qp_mask(enum ib_qp_type qp_type, int mask) · 2327 unmodified lines
23282328 }
23292329 }
23302330
2331−ssize_t ib_uverbs_modify_qp(struct ib_uverbs_file *file,
2332− struct ib_device *ib_dev,
2333− const char __user *buf, int in_len,
2334− int out_len)
2331+static int modify_qp(struct ib_uverbs_file *file,
2332+ struct ib_uverbs_ex_modify_qp *cmd, struct ib_udata *udata)
23352333 {
2336− struct ib_uverbs_modify_qp cmd;
2337− struct ib_udata udata;
2338− struct ib_qp *qp;
2339− struct ib_qp_attr *attr;
2340− int ret;
2341−
2342− if (copy_from_user(&cmd, buf, sizeof cmd))
2343− return -EFAULT;
2344−
2345− INIT_UDATA(&udata, buf + sizeof cmd, NULL, in_len - sizeof cmd,
2346− out_len);
2334+ struct ib_qp_attr *attr;
2335+ struct ib_qp *qp;
2336+ int ret;
23472337
23482338 attr = kmalloc(sizeof *attr, GFP_KERNEL);
23492339 if (!attr)
23502340 return -ENOMEM;
23512341
2352− qp = idr_read_qp(cmd.qp_handle, file->ucontext);
2342+ qp = idr_read_qp(cmd->base.qp_handle, file->ucontext);
23532343 if (!qp) {
23542344 ret = -EINVAL;
23552345 goto out;
23562346 }
23572347
2358− attr->qp_state = cmd.qp_state;
2359− attr->cur_qp_state = cmd.cur_qp_state;
2360− attr->path_mtu = cmd.path_mtu;
2361− attr->path_mig_state = cmd.path_mig_state;
2362− attr->qkey = cmd.qkey;
2363− attr->rq_psn = cmd.rq_psn;
2364− attr->sq_psn = cmd.sq_psn;
2365− attr->dest_qp_num = cmd.dest_qp_num;
2366− attr->qp_access_flags = cmd.qp_access_flags;
2367− attr->pkey_index = cmd.pkey_index;
2368− attr->alt_pkey_index = cmd.alt_pkey_index;
2369− attr->en_sqd_async_notify = cmd.en_sqd_async_notify;
2370− attr->max_rd_atomic = cmd.max_rd_atomic;
2371− attr->max_dest_rd_atomic = cmd.max_dest_rd_atomic;
2372− attr->min_rnr_timer = cmd.min_rnr_timer;
2373− attr->port_num = cmd.port_num;
2374− attr->timeout = cmd.timeout;
2375− attr->retry_cnt = cmd.retry_cnt;
2376− attr->rnr_retry = cmd.rnr_retry;
2377− attr->alt_port_num = cmd.alt_port_num;
2378− attr->alt_timeout = cmd.alt_timeout;
2379−
2380− memcpy(attr->ah_attr.grh.dgid.raw, cmd.dest.dgid, 16);
2381− attr->ah_attr.grh.flow_label = cmd.dest.flow_label;
2382− attr->ah_attr.grh.sgid_index = cmd.dest.sgid_index;
2383− attr->ah_attr.grh.hop_limit = cmd.dest.hop_limit;
2384− attr->ah_attr.grh.traffic_class = cmd.dest.traffic_class;
2385− attr->ah_attr.dlid = cmd.dest.dlid;
2386− attr->ah_attr.sl = cmd.dest.sl;
2387− attr->ah_attr.src_path_bits = cmd.dest.src_path_bits;
2388− attr->ah_attr.static_rate = cmd.dest.static_rate;
2389− attr->ah_attr.ah_flags = cmd.dest.is_global ? IB_AH_GRH : 0;
2390− attr->ah_attr.port_num = cmd.dest.port_num;
2391−
2392− memcpy(attr->alt_ah_attr.grh.dgid.raw, cmd.alt_dest.dgid, 16);
2393− attr->alt_ah_attr.grh.flow_label = cmd.alt_dest.flow_label;
2394− attr->alt_ah_attr.grh.sgid_index = cmd.alt_dest.sgid_index;
2395− attr->alt_ah_attr.grh.hop_limit = cmd.alt_dest.hop_limit;
2396− attr->alt_ah_attr.grh.traffic_class = cmd.alt_dest.traffic_class;
2397− attr->alt_ah_attr.dlid = cmd.alt_dest.dlid;
2398− attr->alt_ah_attr.sl = cmd.alt_dest.sl;
2399− attr->alt_ah_attr.src_path_bits = cmd.alt_dest.src_path_bits;
2400− attr->alt_ah_attr.static_rate = cmd.alt_dest.static_rate;
2401− attr->alt_ah_attr.ah_flags = cmd.alt_dest.is_global ? IB_AH_GRH : 0;
2402− attr->alt_ah_attr.port_num = cmd.alt_dest.port_num;
2348+ attr->qp_state = cmd->base.qp_state;
2349+ attr->cur_qp_state = cmd->base.cur_qp_state;
2350+ attr->path_mtu = cmd->base.path_mtu;
2351+ attr->path_mig_state = cmd->base.path_mig_state;
2352+ attr->qkey = cmd->base.qkey;
2353+ attr->rq_psn = cmd->base.rq_psn;
2354+ attr->sq_psn = cmd->base.sq_psn;
2355+ attr->dest_qp_num = cmd->base.dest_qp_num;
2356+ attr->qp_access_flags = cmd->base.qp_access_flags;
2357+ attr->pkey_index = cmd->base.pkey_index;
2358+ attr->alt_pkey_index = cmd->base.alt_pkey_index;
2359+ attr->en_sqd_async_notify = cmd->base.en_sqd_async_notify;
2360+ attr->max_rd_atomic = cmd->base.max_rd_atomic;
2361+ attr->max_dest_rd_atomic = cmd->base.max_dest_rd_atomic;
2362+ attr->min_rnr_timer = cmd->base.min_rnr_timer;
2363+ attr->port_num = cmd->base.port_num;
2364+ attr->timeout = cmd->base.timeout;
2365+ attr->retry_cnt = cmd->base.retry_cnt;
2366+ attr->rnr_retry = cmd->base.rnr_retry;
2367+ attr->alt_port_num = cmd->base.alt_port_num;
2368+ attr->alt_timeout = cmd->base.alt_timeout;
2369+ attr->rate_limit = cmd->rate_limit;
2370+
2371+ memcpy(attr->ah_attr.grh.dgid.raw, cmd->base.dest.dgid, 16);
2372+ attr->ah_attr.grh.flow_label = cmd->base.dest.flow_label;
2373+ attr->ah_attr.grh.sgid_index = cmd->base.dest.sgid_index;
2374+ attr->ah_attr.grh.hop_limit = cmd->base.dest.hop_limit;
2375+ attr->ah_attr.grh.traffic_class = cmd->base.dest.traffic_class;
2376+ attr->ah_attr.dlid = cmd->base.dest.dlid;
2377+ attr->ah_attr.sl = cmd->base.dest.sl;
2378+ attr->ah_attr.src_path_bits = cmd->base.dest.src_path_bits;
2379+ attr->ah_attr.static_rate = cmd->base.dest.static_rate;
2380+ attr->ah_attr.ah_flags = cmd->base.dest.is_global ?
2381+ IB_AH_GRH : 0;
2382+ attr->ah_attr.port_num = cmd->base.dest.port_num;
2383+
2384+ memcpy(attr->alt_ah_attr.grh.dgid.raw, cmd->base.alt_dest.dgid, 16);
2385+ attr->alt_ah_attr.grh.flow_label = cmd->base.alt_dest.flow_label;
2386+ attr->alt_ah_attr.grh.sgid_index = cmd->base.alt_dest.sgid_index;
2387+ attr->alt_ah_attr.grh.hop_limit = cmd->base.alt_dest.hop_limit;
2388+ attr->alt_ah_attr.grh.traffic_class = cmd->base.alt_dest.traffic_class;
2389+ attr->alt_ah_attr.dlid = cmd->base.alt_dest.dlid;
2390+ attr->alt_ah_attr.sl = cmd->base.alt_dest.sl;
2391+ attr->alt_ah_attr.src_path_bits = cmd->base.alt_dest.src_path_bits;
2392+ attr->alt_ah_attr.static_rate = cmd->base.alt_dest.static_rate;
2393+ attr->alt_ah_attr.ah_flags = cmd->base.alt_dest.is_global ?
2394+ IB_AH_GRH : 0;
2395+ attr->alt_ah_attr.port_num = cmd->base.alt_dest.port_num;
24032396
24042397 if (qp->real_qp == qp) {
2405− ret = ib_resolve_eth_dmac(qp, attr, &cmd.attr_mask);
2406− if (ret)
2407− goto release_qp;
2398+ if (cmd->base.attr_mask & IB_QP_AV) {
2399+ ret = ib_resolve_eth_dmac(qp->device, &attr->ah_attr);
2400+ if (ret)
2401+ goto release_qp;
2402+ }
24082403 ret = qp->device->modify_qp(qp, attr,
2409− modify_qp_mask(qp->qp_type, cmd.attr_mask), &udata);
2404+ modify_qp_mask(qp->qp_type,
2405+ cmd->base.attr_mask),
2406+ udata);
24102407 } else {
2411− ret = ib_modify_qp(qp, attr, modify_qp_mask(qp->qp_type, cmd.attr_mask));
2408+ ret = ib_modify_qp(qp, attr,
2409+ modify_qp_mask(qp->qp_type,
2410+ cmd->base.attr_mask));
24122411 }
24132412
2414− if (ret)
2415− goto release_qp;
2416−
2417− ret = in_len;
2418−
24192413 release_qp:
24202414 put_qp_read(qp);
24212415
@@ -2425,6 +2419,68 @@ out: · 3 unmodified lines
24252419 return ret;
24262420 }
24272421
2422+ssize_t ib_uverbs_modify_qp(struct ib_uverbs_file *file,
2423+ struct ib_device *ib_dev,
2424+ const char __user *buf, int in_len,
2425+ int out_len)
2426+{
2427+ struct ib_uverbs_ex_modify_qp cmd = {};
2428+ struct ib_udata udata;
2429+ int ret;
2430+
2431+ if (copy_from_user(&cmd.base, buf, sizeof(cmd.base)))
2432+ return -EFAULT;
2433+
2434+ if (cmd.base.attr_mask &
2435+ ~((IB_USER_LEGACY_LAST_QP_ATTR_MASK << 1) - 1))
2436+ return -EOPNOTSUPP;
2437+
2438+ INIT_UDATA(&udata, buf + sizeof(cmd.base), NULL,
2439+ in_len - sizeof(cmd.base), out_len);
2440+
2441+ ret = modify_qp(file, &cmd, &udata);
2442+ if (ret)
2443+ return ret;
2444+
2445+ return in_len;
2446+}
2447+
2448+int ib_uverbs_ex_modify_qp(struct ib_uverbs_file *file,
2449+ struct ib_device *ib_dev,
2450+ struct ib_udata *ucore,
2451+ struct ib_udata *uhw)
2452+{
2453+ struct ib_uverbs_ex_modify_qp cmd = {};
2454+ int ret;
2455+
2456+ /*
2457+ * Last bit is reserved for extending the attr_mask by
2458+ * using another field.
2459+ */
2460+ BUILD_BUG_ON(IB_USER_LAST_QP_ATTR_MASK == (1 << 31));
2461+
2462+ if (ucore->inlen < sizeof(cmd.base))
2463+ return -EINVAL;
2464+
2465+ ret = ib_copy_from_udata(&cmd, ucore, min(sizeof(cmd), ucore->inlen));
2466+ if (ret)
2467+ return ret;
2468+
2469+ if (cmd.base.attr_mask &
2470+ ~((IB_USER_LAST_QP_ATTR_MASK << 1) - 1))
2471+ return -EOPNOTSUPP;
2472+
2473+ if (ucore->inlen > sizeof(cmd)) {
2474+ if (ib_is_udata_cleared(ucore, sizeof(cmd),
2475+ ucore->inlen - sizeof(cmd)))
2476+ return -EOPNOTSUPP;
2477+ }
2478+
2479+ ret = modify_qp(file, &cmd, uhw);
2480+
2481+ return ret;
2482+}
2483+
24282484 ssize_t ib_uverbs_destroy_qp(struct ib_uverbs_file *file,
24292485 struct ib_device *ib_dev,
24302486 const char __user *buf, int in_len,
@@ -2875,6 +2931,7 @@ ssize_t ib_uverbs_create_ah(struct ib_uverbs_file *file, · 444 unmodified lines
28752931 struct ib_ah *ah;
28762932 struct ib_ah_attr attr;
28772933 int ret;
2934+ struct ib_udata udata;
28782935
28792936 if (out_len < sizeof resp)
28802937 return -ENOSPC;
@@ -2882,6 +2939,10 @@ ssize_t ib_uverbs_create_ah(struct ib_uverbs_file *file, · 1 unmodified line
28822939 if (copy_from_user(&cmd, buf, sizeof cmd))
28832940 return -EFAULT;
28842941
2942+ INIT_UDATA(&udata, buf + sizeof(cmd),
2943+ (unsigned long)cmd.response + sizeof(resp),
2944+ in_len - sizeof(cmd), out_len - sizeof(resp));
2945+
28852946 uobj = kmalloc(sizeof *uobj, GFP_KERNEL);
28862947 if (!uobj)
28872948 return -ENOMEM;
@@ -2908,12 +2969,16 @@ ssize_t ib_uverbs_create_ah(struct ib_uverbs_file *file, · 20 unmodified lines
29082969 memset(&attr.dmac, 0, sizeof(attr.dmac));
29092970 memcpy(attr.grh.dgid.raw, cmd.attr.grh.dgid, 16);
29102971
2911− ah = ib_create_ah(pd, &attr);
2972+ ah = pd->device->create_ah(pd, &attr, &udata);
2973+
29122974 if (IS_ERR(ah)) {
29132975 ret = PTR_ERR(ah);
29142976 goto err_put;
29152977 }
29162978
2979+ ah->device = pd->device;
2980+ ah->pd = pd;
2981+ atomic_inc(&pd->usecnt);
29172982 ah->uobject = uobj;
29182983 uobj->object = ah;
29192984
@@ -3124,8 +3189,10 @@ static int kern_spec_to_ib_spec(struct ib_uverbs_flow_spec *kern_spec, · 204 unmodified lines
31243189 kern_spec_val = (void *)kern_spec +
31253190 sizeof(struct ib_uverbs_flow_spec_hdr);
31263191 kern_spec_mask = kern_spec_val + kern_filter_sz;
3192+ if (ib_spec->type == (IB_FLOW_SPEC_INNER | IB_FLOW_SPEC_VXLAN_TUNNEL))
3193+ return -EINVAL;
31273194
3128− switch (ib_spec->type) {
3195+ switch (ib_spec->type & ~IB_FLOW_SPEC_INNER) {
31293196 case IB_FLOW_SPEC_ETH:
31303197 ib_filter_sz = offsetof(struct ib_flow_eth_filter, real_sz);
31313198 actual_filter_sz = spec_filter_size(kern_spec_mask,
@@ -3175,6 +3242,21 @@ static int kern_spec_to_ib_spec(struct ib_uverbs_flow_spec *kern_spec, · 43 unmodified lines
31753242 memcpy(&ib_spec->tcp_udp.val, kern_spec_val, actual_filter_sz);
31763243 memcpy(&ib_spec->tcp_udp.mask, kern_spec_mask, actual_filter_sz);
31773244 break;
3245+ case IB_FLOW_SPEC_VXLAN_TUNNEL:
3246+ ib_filter_sz = offsetof(struct ib_flow_tunnel_filter, real_sz);
3247+ actual_filter_sz = spec_filter_size(kern_spec_mask,
3248+ kern_filter_sz,
3249+ ib_filter_sz);
3250+ if (actual_filter_sz <= 0)
3251+ return -EINVAL;
3252+ ib_spec->tunnel.size = sizeof(struct ib_flow_spec_tunnel);
3253+ memcpy(&ib_spec->tunnel.val, kern_spec_val, actual_filter_sz);
3254+ memcpy(&ib_spec->tunnel.mask, kern_spec_mask, actual_filter_sz);
3255+
3256+ if ((ntohl(ib_spec->tunnel.mask.tunnel_id)) >= BIT(24) ||
3257+ (ntohl(ib_spec->tunnel.val.tunnel_id)) >= BIT(24))
3258+ return -EINVAL;
3259+ break;
31783260 default:
31793261 return -EINVAL;
31803262 }
@@ -3745,7 +3827,6 @@ int ib_uverbs_ex_create_flow(struct ib_uverbs_file *file, · 564 unmodified lines
37453827 err = PTR_ERR(flow_id);
37463828 goto err_free;
37473829 }
3748− flow_id->qp = qp;
37493830 flow_id->uobject = uobj;
37503831 uobj->object = flow_id;
37513832
drivers/infiniband/core/uverbs_main.c+5−1
@@ -137,6 +137,7 @@ static int (*uverbs_ex_cmd_table[])(struct ib_uverbs_file *file, · 136 unmodified lines
137137 [IB_USER_VERBS_EX_CMD_DESTROY_WQ] = ib_uverbs_ex_destroy_wq,
138138 [IB_USER_VERBS_EX_CMD_CREATE_RWQ_IND_TBL] = ib_uverbs_ex_create_rwq_ind_table,
139139 [IB_USER_VERBS_EX_CMD_DESTROY_RWQ_IND_TBL] = ib_uverbs_ex_destroy_rwq_ind_table,
140+ [IB_USER_VERBS_EX_CMD_MODIFY_QP] = ib_uverbs_ex_modify_qp,
140141 };
141142
142143 static void ib_uverbs_add_one(struct ib_device *device);
@@ -746,8 +747,11 @@ static ssize_t ib_uverbs_write(struct file *filp, const char __user *buf, · 603 unmodified lines
746747 int srcu_key;
747748 ssize_t ret;
748749
749− if (WARN_ON_ONCE(!ib_safe_file_access(filp)))
750+ if (!ib_safe_file_access(filp)) {
751+ pr_err_once("uverbs_write: process %d (%s) changed security contexts after opening file descriptor, this is not allowed.\n",
752+ task_tgid_vnr(current), current->comm);
750753 return -EACCES;
754+ }
751755
752756 if (count < sizeof hdr)
753757 return -EINVAL;
drivers/infiniband/core/verbs.c+57−51
@@ -315,7 +315,7 @@ struct ib_ah *ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr) · 314 unmodified lines
315315 {
316316 struct ib_ah *ah;
317317
318− ah = pd->device->create_ah(pd, ah_attr);
318+ ah = pd->device->create_ah(pd, ah_attr, NULL);
319319
320320 if (!IS_ERR(ah)) {
321321 ah->device = pd->device;
@@ -328,7 +328,7 @@ struct ib_ah *ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr) · 6 unmodified lines
328328 }
329329 EXPORT_SYMBOL(ib_create_ah);
330330
331−static int ib_get_header_version(const union rdma_network_hdr *hdr)
331+int ib_get_rdma_header_version(const union rdma_network_hdr *hdr)
332332 {
333333 const struct iphdr *ip4h = (struct iphdr *)&hdr->roce4grh;
334334 struct iphdr ip4h_checked;
@@ -359,6 +359,7 @@ static int ib_get_header_version(const union rdma_network_hdr *hdr) · 24 unmodified lines
359359 return 4;
360360 return 6;
361361 }
362+EXPORT_SYMBOL(ib_get_rdma_header_version);
362363
363364 static enum rdma_network_type ib_get_net_type_by_grh(struct ib_device *device,
364365 u8 port_num,
@@ -369,7 +370,7 @@ static enum rdma_network_type ib_get_net_type_by_grh(struct ib_device *device, · 4 unmodified lines
369370 if (rdma_protocol_ib(device, port_num))
370371 return RDMA_NETWORK_IB;
371372
372− grh_version = ib_get_header_version((union rdma_network_hdr *)grh);
373+ grh_version = ib_get_rdma_header_version((union rdma_network_hdr *)grh);
373374
374375 if (grh_version == 4)
375376 return RDMA_NETWORK_IPV4;
@@ -415,9 +416,9 @@ static int get_sgid_index_from_eth(struct ib_device *device, u8 port_num, · 39 unmodified lines
415416 &context, gid_index);
416417 }
417418
418−static int get_gids_from_rdma_hdr(union rdma_network_hdr *hdr,
419− enum rdma_network_type net_type,
420− union ib_gid *sgid, union ib_gid *dgid)
419+int ib_get_gids_from_rdma_hdr(const union rdma_network_hdr *hdr,
420+ enum rdma_network_type net_type,
421+ union ib_gid *sgid, union ib_gid *dgid)
421422 {
422423 struct sockaddr_in src_in;
423424 struct sockaddr_in dst_in;
@@ -447,6 +448,7 @@ static int get_gids_from_rdma_hdr(union rdma_network_hdr *hdr, · 23 unmodified lines
447448 return -EINVAL;
448449 }
449450 }
451+EXPORT_SYMBOL(ib_get_gids_from_rdma_hdr);
450452
451453 int ib_init_ah_from_wc(struct ib_device *device, u8 port_num,
452454 const struct ib_wc *wc, const struct ib_grh *grh,
@@ -469,8 +471,8 @@ int ib_init_ah_from_wc(struct ib_device *device, u8 port_num, · 16 unmodified lines
469471 net_type = ib_get_net_type_by_grh(device, port_num, grh);
470472 gid_type = ib_network_to_gid_type(net_type);
471473 }
472− ret = get_gids_from_rdma_hdr((union rdma_network_hdr *)grh, net_type,
473− &sgid, &dgid);
474+ ret = ib_get_gids_from_rdma_hdr((union rdma_network_hdr *)grh, net_type,
475+ &sgid, &dgid);
474476 if (ret)
475477 return ret;
476478
@@ -1014,6 +1016,7 @@ static const struct { · 537 unmodified lines
10141016 IB_QP_QKEY),
10151017 [IB_QPT_GSI] = (IB_QP_CUR_STATE |
10161018 IB_QP_QKEY),
1019+ [IB_QPT_RAW_PACKET] = IB_QP_RATE_LIMIT,
10171020 }
10181021 }
10191022 },
@@ -1047,6 +1050,7 @@ static const struct { · 27 unmodified lines
10471050 IB_QP_QKEY),
10481051 [IB_QPT_GSI] = (IB_QP_CUR_STATE |
10491052 IB_QP_QKEY),
1053+ [IB_QPT_RAW_PACKET] = IB_QP_RATE_LIMIT,
10501054 }
10511055 },
10521056 [IB_QPS_SQD] = {
@@ -1196,66 +1200,66 @@ int ib_modify_qp_is_ok(enum ib_qp_state cur_state, enum ib_qp_state next_state, · 143 unmodified lines
11961200 }
11971201 EXPORT_SYMBOL(ib_modify_qp_is_ok);
11981202
1199−int ib_resolve_eth_dmac(struct ib_qp *qp,
1200− struct ib_qp_attr *qp_attr, int *qp_attr_mask)
1203+int ib_resolve_eth_dmac(struct ib_device *device,
1204+ struct ib_ah_attr *ah_attr)
12011205 {
12021206 int ret = 0;
12031207
1204− if (*qp_attr_mask & IB_QP_AV) {
1205− if (qp_attr->ah_attr.port_num < rdma_start_port(qp->device) ||
1206− qp_attr->ah_attr.port_num > rdma_end_port(qp->device))
1207− return -EINVAL;
1208−
1209− if (!rdma_cap_eth_ah(qp->device, qp_attr->ah_attr.port_num))
1210− return 0;
1211−
1212− if (rdma_link_local_addr((struct in6_addr *)qp_attr->ah_attr.grh.dgid.raw)) {
1213− rdma_get_ll_mac((struct in6_addr *)qp_attr->ah_attr.grh.dgid.raw,
1214− qp_attr->ah_attr.dmac);
1215− } else {
1216− union ib_gid sgid;
1217− struct ib_gid_attr sgid_attr;
1218− int ifindex;
1219− int hop_limit;
1220−
1221− ret = ib_query_gid(qp->device,
1222− qp_attr->ah_attr.port_num,
1223− qp_attr->ah_attr.grh.sgid_index,
1224− &sgid, &sgid_attr);
1225−
1226− if (ret || !sgid_attr.ndev) {
1227− if (!ret)
1228− ret = -ENXIO;
1229− goto out;
1230− }
1208+ if (ah_attr->port_num < rdma_start_port(device) ||
1209+ ah_attr->port_num > rdma_end_port(device))
1210+ return -EINVAL;
12311211
1232− ifindex = sgid_attr.ndev->ifindex;
1212+ if (!rdma_cap_eth_ah(device, ah_attr->port_num))
1213+ return 0;
12331214
1234− ret = rdma_addr_find_l2_eth_by_grh(&sgid,
1235− &qp_attr->ah_attr.grh.dgid,
1236− qp_attr->ah_attr.dmac,
1237− NULL, &ifindex, &hop_limit);
1215+ if (rdma_link_local_addr((struct in6_addr *)ah_attr->grh.dgid.raw)) {
1216+ rdma_get_ll_mac((struct in6_addr *)ah_attr->grh.dgid.raw,
1217+ ah_attr->dmac);
1218+ } else {
1219+ union ib_gid sgid;
1220+ struct ib_gid_attr sgid_attr;
1221+ int ifindex;
1222+ int hop_limit;
1223+
1224+ ret = ib_query_gid(device,
1225+ ah_attr->port_num,
1226+ ah_attr->grh.sgid_index,
1227+ &sgid, &sgid_attr);
1228+
1229+ if (ret || !sgid_attr.ndev) {
1230+ if (!ret)
1231+ ret = -ENXIO;
1232+ goto out;
1233+ }
12381234
1239− dev_put(sgid_attr.ndev);
1235+ ifindex = sgid_attr.ndev->ifindex;
12401236
1241− qp_attr->ah_attr.grh.hop_limit = hop_limit;
1242− }
1237+ ret = rdma_addr_find_l2_eth_by_grh(&sgid,
1238+ &ah_attr->grh.dgid,
1239+ ah_attr->dmac,
1240+ NULL, &ifindex, &hop_limit);
1241+
1242+ dev_put(sgid_attr.ndev);
1243+
1244+ ah_attr->grh.hop_limit = hop_limit;
12431245 }
12441246 out:
12451247 return ret;
12461248 }
12471249 EXPORT_SYMBOL(ib_resolve_eth_dmac);
12481250
1249−
12501251 int ib_modify_qp(struct ib_qp *qp,
12511252 struct ib_qp_attr *qp_attr,
12521253 int qp_attr_mask)
12531254 {
1254− int ret;
12551255
1256− ret = ib_resolve_eth_dmac(qp, qp_attr, &qp_attr_mask);
1257− if (ret)
1258− return ret;
1256+ if (qp_attr_mask & IB_QP_AV) {
1257+ int ret;
1258+
1259+ ret = ib_resolve_eth_dmac(qp->device, &qp_attr->ah_attr);
1260+ if (ret)
1261+ return ret;
1262+ }
12591263
12601264 return qp->device->modify_qp(qp->real_qp, qp_attr, qp_attr_mask, NULL);
12611265 }
@@ -1734,8 +1738,10 @@ struct ib_flow *ib_create_flow(struct ib_qp *qp, · 472 unmodified lines
17341738 return ERR_PTR(-ENOSYS);
17351739
17361740 flow_id = qp->device->create_flow(qp, flow_attr, domain);
1737− if (!IS_ERR(flow_id))
1741+ if (!IS_ERR(flow_id)) {
17381742 atomic_inc(&qp->usecnt);
1743+ flow_id->qp = qp;
1744+ }
17391745 return flow_id;
17401746 }
17411747 EXPORT_SYMBOL(ib_create_flow);
drivers/infiniband/hw/Makefile+1−0
@@ -7,6 +7,7 @@ obj-$(CONFIG_MLX4_INFINIBAND) += mlx4/ · 6 unmodified lines
77 obj-$(CONFIG_MLX5_INFINIBAND) += mlx5/
88 obj-$(CONFIG_INFINIBAND_NES) += nes/
99 obj-$(CONFIG_INFINIBAND_OCRDMA) += ocrdma/
10+obj-$(CONFIG_INFINIBAND_VMWARE_PVRDMA) += vmw_pvrdma/
1011 obj-$(CONFIG_INFINIBAND_USNIC) += usnic/
1112 obj-$(CONFIG_INFINIBAND_HFI1) += hfi1/
1213 obj-$(CONFIG_INFINIBAND_HNS) += hns/
drivers/infiniband/hw/cxgb3/cxio_dbg.c+8−12
@@ -45,10 +45,9 @@ void cxio_dump_tpt(struct cxio_rdev *rdev, u32 stag) · 44 unmodified lines
4545 int size = 32;
4646
4747 m = kmalloc(sizeof(*m) + size, GFP_ATOMIC);
48− if (!m) {
49− PDBG("%s couldn't allocate memory.\n", __func__);
48+ if (!m)
5049 return;
51− }
50+
5251 m->mem_id = MEM_PMRX;
5352 m->addr = (stag>>8) * 32 + rdev->rnic_info.tpt_base;
5453 m->len = size;
@@ -82,10 +81,9 @@ void cxio_dump_pbl(struct cxio_rdev *rdev, u32 pbl_addr, uint len, u8 shift) · 27 unmodified lines
8281 size = npages * sizeof(u64);
8382
8483 m = kmalloc(sizeof(*m) + size, GFP_ATOMIC);
85− if (!m) {
86− PDBG("%s couldn't allocate memory.\n", __func__);
84+ if (!m)
8785 return;
88− }
86+
8987 m->mem_id = MEM_PMRX;
9088 m->addr = pbl_addr;
9189 m->len = size;
@@ -144,10 +142,9 @@ void cxio_dump_rqt(struct cxio_rdev *rdev, u32 hwtid, int nents) · 52 unmodified lines
144142 int rc;
145143
146144 m = kmalloc(sizeof(*m) + size, GFP_ATOMIC);
147− if (!m) {
148− PDBG("%s couldn't allocate memory.\n", __func__);
145+ if (!m)
149146 return;
150− }
147+
151148 m->mem_id = MEM_PMRX;
152149 m->addr = ((hwtid)<<10) + rdev->rnic_info.rqt_base;
153150 m->len = size;
@@ -177,10 +174,9 @@ void cxio_dump_tcb(struct cxio_rdev *rdev, u32 hwtid) · 23 unmodified lines
177174 int rc;
178175
179176 m = kmalloc(sizeof(*m) + size, GFP_ATOMIC);
180− if (!m) {
181− PDBG("%s couldn't allocate memory.\n", __func__);
177+ if (!m)
182178 return;
183− }
179+
184180 m->mem_id = MEM_CM;
185181 m->addr = hwtid * size;
186182 m->len = size;
drivers/infiniband/hw/cxgb3/iwch_provider.c+2−1
@@ -62,7 +62,8 @@ · 61 unmodified lines
6262 #include "common.h"
6363
6464 static struct ib_ah *iwch_ah_create(struct ib_pd *pd,
65− struct ib_ah_attr *ah_attr)
65+ struct ib_ah_attr *ah_attr,
66+ struct ib_udata *udata)
6667 {
6768 return ERR_PTR(-ENOSYS);
6869 }
drivers/infiniband/hw/cxgb4/device.c+3−5
@@ -828,8 +828,10 @@ static int c4iw_rdev_open(struct c4iw_rdev *rdev) · 827 unmodified lines
828828 }
829829 rdev->status_page = (struct t4_dev_status_page *)
830830 __get_free_page(GFP_KERNEL);
831− if (!rdev->status_page)
831+ if (!rdev->status_page) {
832+ err = -ENOMEM;
832833 goto destroy_ocqp_pool;
834+ }
833835 rdev->status_page->qp_start = rdev->lldi.vr->qp.start;
834836 rdev->status_page->qp_size = rdev->lldi.vr->qp.size;
835837 rdev->status_page->cq_start = rdev->lldi.vr->cq.start;
@@ -841,8 +843,6 @@ static int c4iw_rdev_open(struct c4iw_rdev *rdev) · 5 unmodified lines
841843 if (rdev->wr_log) {
842844 rdev->wr_log_size = 1 << c4iw_wr_log_size_order;
843845 atomic_set(&rdev->wr_log_idx, 0);
844− } else {
845− pr_err(MOD "error allocating wr_log. Logging disabled\n");
846846 }
847847 }
848848
@@ -1424,8 +1424,6 @@ static void recover_queues(struct uld_ctx *ctx) · 575 unmodified lines
14241424
14251425 qp_list.qps = kzalloc(count * sizeof *qp_list.qps, GFP_ATOMIC);
14261426 if (!qp_list.qps) {
1427− printk(KERN_ERR MOD "%s: Fatal error - DB overflow recovery failed\n",
1428− pci_name(ctx->lldi.pdev));
14291427 spin_unlock_irq(&ctx->dev->lock);
14301428 return;
14311429 }
drivers/infiniband/hw/cxgb4/provider.c+3−1
@@ -59,7 +59,9 @@ module_param(fastreg_support, int, 0644); · 58 unmodified lines
5959 MODULE_PARM_DESC(fastreg_support, "Advertise fastreg support (default=1)");
6060
6161 static struct ib_ah *c4iw_ah_create(struct ib_pd *pd,
62− struct ib_ah_attr *ah_attr)
62+ struct ib_ah_attr *ah_attr,
63+ struct ib_udata *udata)
64+
6365 {
6466 return ERR_PTR(-ENOSYS);
6567 }
drivers/infiniband/hw/hfi1/affinity.c+2−1
@@ -125,6 +125,7 @@ int node_affinity_init(void) · 124 unmodified lines
125125 cpumask_weight(topology_sibling_cpumask(
126126 cpumask_first(&node_affinity.proc.mask)
127127 ));
128+ node_affinity.num_possible_nodes = num_possible_nodes();
128129 node_affinity.num_online_nodes = num_online_nodes();
129130 node_affinity.num_online_cpus = num_online_cpus();
130131
@@ -135,7 +136,7 @@ int node_affinity_init(void) · 4 unmodified lines
135136 */
136137 init_real_cpu_mask();
137138
138− hfi1_per_node_cntr = kcalloc(num_possible_nodes(),
139+ hfi1_per_node_cntr = kcalloc(node_affinity.num_possible_nodes,
139140 sizeof(*hfi1_per_node_cntr), GFP_KERNEL);
140141 if (!hfi1_per_node_cntr)
141142 return -ENOMEM;
drivers/infiniband/hw/hfi1/affinity.h+1−8
@@ -70,14 +70,6 @@ struct cpu_mask_set { · 69 unmodified lines
7070 uint gen;
7171 };
7272
73−struct hfi1_affinity {
74− struct cpu_mask_set def_intr;
75− struct cpu_mask_set rcv_intr;
76− struct cpumask real_cpu_mask;
77− /* spin lock to protect affinity struct */
78− spinlock_t lock;
79−};
80−
8173 struct hfi1_msix_entry;
8274
8375 /* Initialize non-HT cpu cores mask */
@@ -115,6 +107,7 @@ struct hfi1_affinity_node_list { · 31 unmodified lines
115107 struct cpumask real_cpu_mask;
116108 struct cpu_mask_set proc;
117109 int num_core_siblings;
110+ int num_possible_nodes;
118111 int num_online_nodes;
119112 int num_online_cpus;
120113 struct mutex lock; /* protects affinity nodes */
drivers/infiniband/hw/hfi1/chip.c+6−3
@@ -8477,7 +8477,10 @@ static int do_8051_command( · 8476 unmodified lines
84778477 */
84788478 if (type == HCMD_WRITE_LCB_CSR) {
84798479 in_data |= ((*out_data) & 0xffffffffffull) << 8;
8480− reg = ((((*out_data) >> 40) & 0xff) <<
8480+ /* must preserve COMPLETED - it is tied to hardware */
8481+ reg = read_csr(dd, DC_DC8051_CFG_EXT_DEV_0);
8482+ reg &= DC_DC8051_CFG_EXT_DEV_0_COMPLETED_SMASK;
8483+ reg |= ((((*out_data) >> 40) & 0xff) <<
84818484 DC_DC8051_CFG_EXT_DEV_0_RETURN_CODE_SHIFT)
84828485 | ((((*out_data) >> 48) & 0xffff) <<
84838486 DC_DC8051_CFG_EXT_DEV_0_RSP_DATA_SHIFT);
@@ -9556,11 +9559,11 @@ int bringup_serdes(struct hfi1_pportdata *ppd) · 1072 unmodified lines
95569559 if (HFI1_CAP_IS_KSET(EXTENDED_PSN))
95579560 add_rcvctrl(dd, RCV_CTRL_RCV_EXTENDED_PSN_ENABLE_SMASK);
95589561
9559− guid = ppd->guid;
9562+ guid = ppd->guids[HFI1_PORT_GUID_INDEX];
95609563 if (!guid) {
95619564 if (dd->base_guid)
95629565 guid = dd->base_guid + ppd->port - 1;
9563− ppd->guid = guid;
9566+ ppd->guids[HFI1_PORT_GUID_INDEX] = guid;
95649567 }
95659568
95669569 /* Set linkinit_reason on power up per OPA spec */
drivers/infiniband/hw/hfi1/chip_registers.h+3−0
@@ -415,6 +415,9 @@ · 414 unmodified lines
415415 #define ASIC_CFG_SBUS_REQUEST_DATA_IN_SHIFT 32
416416 #define ASIC_CFG_SBUS_REQUEST_RECEIVER_ADDR_SHIFT 0
417417 #define ASIC_CFG_SCRATCH (ASIC + 0x000000000020)
418+#define ASIC_CFG_SCRATCH_1 (ASIC_CFG_SCRATCH + 0x08)
419+#define ASIC_CFG_SCRATCH_2 (ASIC_CFG_SCRATCH + 0x10)
420+#define ASIC_CFG_SCRATCH_3 (ASIC_CFG_SCRATCH + 0x18)
418421 #define ASIC_CFG_THERM_POLL_EN (ASIC + 0x000000000050)
419422 #define ASIC_EEP_ADDR_CMD (ASIC + 0x000000000308)
420423 #define ASIC_EEP_ADDR_CMD_EP_ADDR_MASK 0xFFFFFFull
drivers/infiniband/hw/hfi1/debugfs.c+110−0
@@ -541,6 +541,114 @@ static ssize_t asic_flags_write(struct file *file, const char __user *buf, · 540 unmodified lines
541541 return ret;
542542 }
543543
544+/* read the dc8051 memory */
545+static ssize_t dc8051_memory_read(struct file *file, char __user *buf,
546+ size_t count, loff_t *ppos)
547+{
548+ struct hfi1_pportdata *ppd = private2ppd(file);
549+ ssize_t rval;
550+ void *tmp;
551+ loff_t start, end;
552+
553+ /* the checks below expect the position to be positive */
554+ if (*ppos < 0)
555+ return -EINVAL;
556+
557+ tmp = kzalloc(DC8051_DATA_MEM_SIZE, GFP_KERNEL);
558+ if (!tmp)
559+ return -ENOMEM;
560+
561+ /*
562+ * Fill in the requested portion of the temporary buffer from the
563+ * 8051 memory. The 8051 memory read is done in terms of 8 bytes.
564+ * Adjust start and end to fit. Skip reading anything if out of
565+ * range.
566+ */
567+ start = *ppos & ~0x7; /* round down */
568+ if (start < DC8051_DATA_MEM_SIZE) {
569+ end = (*ppos + count + 7) & ~0x7; /* round up */
570+ if (end > DC8051_DATA_MEM_SIZE)
571+ end = DC8051_DATA_MEM_SIZE;
572+ rval = read_8051_data(ppd->dd, start, end - start,
573+ (u64 *)(tmp + start));
574+ if (rval)
575+ goto done;
576+ }
577+
578+ rval = simple_read_from_buffer(buf, count, ppos, tmp,
579+ DC8051_DATA_MEM_SIZE);
580+done:
581+ kfree(tmp);
582+ return rval;
583+}
584+
585+static ssize_t debugfs_lcb_read(struct file *file, char __user *buf,
586+ size_t count, loff_t *ppos)
587+{
588+ struct hfi1_pportdata *ppd = private2ppd(file);
589+ struct hfi1_devdata *dd = ppd->dd;
590+ unsigned long total, csr_off;
591+ u64 data;
592+
593+ if (*ppos < 0)
594+ return -EINVAL;
595+ /* only read 8 byte quantities */
596+ if ((count % 8) != 0)
597+ return -EINVAL;
598+ /* offset must be 8-byte aligned */
599+ if ((*ppos % 8) != 0)
600+ return -EINVAL;
601+ /* do nothing if out of range or zero count */
602+ if (*ppos >= (LCB_END - LCB_START) || !count)
603+ return 0;
604+ /* reduce count if needed */
605+ if (*ppos + count > LCB_END - LCB_START)
606+ count = (LCB_END - LCB_START) - *ppos;
607+
608+ csr_off = LCB_START + *ppos;
609+ for (total = 0; total < count; total += 8, csr_off += 8) {
610+ if (read_lcb_csr(dd, csr_off, (u64 *)&data))
611+ break; /* failed */
612+ if (put_user(data, (unsigned long __user *)(buf + total)))
613+ break;
614+ }
615+ *ppos += total;
616+ return total;
617+}
618+
619+static ssize_t debugfs_lcb_write(struct file *file, const char __user *buf,
620+ size_t count, loff_t *ppos)
621+{
622+ struct hfi1_pportdata *ppd = private2ppd(file);
623+ struct hfi1_devdata *dd = ppd->dd;
624+ unsigned long total, csr_off, data;
625+
626+ if (*ppos < 0)
627+ return -EINVAL;
628+ /* only write 8 byte quantities */
629+ if ((count % 8) != 0)
630+ return -EINVAL;
631+ /* offset must be 8-byte aligned */
632+ if ((*ppos % 8) != 0)
633+ return -EINVAL;
634+ /* do nothing if out of range or zero count */
635+ if (*ppos >= (LCB_END - LCB_START) || !count)
636+ return 0;
637+ /* reduce count if needed */
638+ if (*ppos + count > LCB_END - LCB_START)
639+ count = (LCB_END - LCB_START) - *ppos;
640+
641+ csr_off = LCB_START + *ppos;
642+ for (total = 0; total < count; total += 8, csr_off += 8) {
643+ if (get_user(data, (unsigned long __user *)(buf + total)))
644+ break;
645+ if (write_lcb_csr(dd, csr_off, data))
646+ break; /* failed */
647+ }
648+ *ppos += total;
649+ return total;
650+}
651+
544652 /*
545653 * read the per-port QSFP data for ppd
546654 */
@@ -931,6 +1039,8 @@ static const struct counter_info port_cntr_ops[] = { · 384 unmodified lines
9311039 DEBUGFS_XOPS("qsfp2", qsfp2_debugfs_read, qsfp2_debugfs_write,
9321040 qsfp2_debugfs_open, qsfp2_debugfs_release),
9331041 DEBUGFS_OPS("asic_flags", asic_flags_read, asic_flags_write),
1042+ DEBUGFS_OPS("dc8051_memory", dc8051_memory_read, NULL),
1043+ DEBUGFS_OPS("lcb", debugfs_lcb_read, debugfs_lcb_write),
9341044 };
9351045
9361046 static void *_sdma_cpu_list_seq_start(struct seq_file *s, loff_t *pos)
drivers/infiniband/hw/hfi1/driver.c+1−2
@@ -795,8 +795,7 @@ static inline void process_rcv_qp_work(struct hfi1_packet *packet) · 794 unmodified lines
795795 hfi1_schedule_send(qp);
796796 spin_unlock_irqrestore(&qp->s_lock, flags);
797797 }
798− if (atomic_dec_and_test(&qp->refcount))
799− wake_up(&qp->wait);
798+ rvt_put_qp(qp);
800799 }
801800 }
802801
drivers/infiniband/hw/hfi1/eprom.c+203−8
@@ -207,6 +207,40 @@ done_asic: · 206 unmodified lines
207207 /* magic character sequence that trails an image */
208208 #define IMAGE_TRAIL_MAGIC "egamiAPO"
209209
210+/* EPROM file types */
211+#define HFI1_EFT_PLATFORM_CONFIG 2
212+
213+/* segment size - 128 KiB */
214+#define SEG_SIZE (128 * 1024)
215+
216+struct hfi1_eprom_footer {
217+ u32 oprom_size; /* size of the oprom, in bytes */
218+ u16 num_table_entries;
219+ u16 version; /* version of this footer */
220+ u32 magic; /* must be last */
221+};
222+
223+struct hfi1_eprom_table_entry {
224+ u32 type; /* file type */
225+ u32 offset; /* file offset from start of EPROM */
226+ u32 size; /* file size, in bytes */
227+};
228+
229+/*
230+ * Calculate the max number of table entries that will fit within a directory
231+ * buffer of size 'dir_size'.
232+ */
233+#define MAX_TABLE_ENTRIES(dir_size) \
234+ (((dir_size) - sizeof(struct hfi1_eprom_footer)) / \
235+ sizeof(struct hfi1_eprom_table_entry))
236+
237+#define DIRECTORY_SIZE(n) (sizeof(struct hfi1_eprom_footer) + \
238+ (sizeof(struct hfi1_eprom_table_entry) * (n)))
239+
240+#define MAGIC4(a, b, c, d) ((d) << 24 | (c) << 16 | (b) << 8 | (a))
241+#define FOOTER_MAGIC MAGIC4('e', 'p', 'r', 'm')
242+#define FOOTER_VERSION 1
243+
210244 /*
211245 * Read all of partition 1. The actual file is at the front. Adjust
212246 * the returned size if a trailing image magic is found.
@@ -241,6 +275,167 @@ static int read_partition_platform_config(struct hfi1_devdata *dd, void **data, · 28 unmodified lines
241275 return 0;
242276 }
243277
278+/*
279+ * The segment magic has been checked. There is a footer and table of
280+ * contents present.
281+ *
282+ * directory is a u32 aligned buffer of size EP_PAGE_SIZE.
283+ */
284+static int read_segment_platform_config(struct hfi1_devdata *dd,
285+ void *directory, void **data, u32 *size)
286+{
287+ struct hfi1_eprom_footer *footer;
288+ struct hfi1_eprom_table_entry *table;
289+ struct hfi1_eprom_table_entry *entry;
290+ void *buffer = NULL;
291+ void *table_buffer = NULL;
292+ int ret, i;
293+ u32 directory_size;
294+ u32 seg_base, seg_offset;
295+ u32 bytes_available, ncopied, to_copy;
296+
297+ /* the footer is at the end of the directory */
298+ footer = (struct hfi1_eprom_footer *)
299+ (directory + EP_PAGE_SIZE - sizeof(*footer));
300+
301+ /* make sure the structure version is supported */
302+ if (footer->version != FOOTER_VERSION)
303+ return -EINVAL;
304+
305+ /* oprom size cannot be larger than a segment */
306+ if (footer->oprom_size >= SEG_SIZE)
307+ return -EINVAL;
308+
309+ /* the file table must fit in a segment with the oprom */
310+ if (footer->num_table_entries >
311+ MAX_TABLE_ENTRIES(SEG_SIZE - footer->oprom_size))
312+ return -EINVAL;
313+
314+ /* find the file table start, which precedes the footer */
315+ directory_size = DIRECTORY_SIZE(footer->num_table_entries);
316+ if (directory_size <= EP_PAGE_SIZE) {
317+ /* the file table fits into the directory buffer handed in */
318+ table = (struct hfi1_eprom_table_entry *)
319+ (directory + EP_PAGE_SIZE - directory_size);
320+ } else {
321+ /* need to allocate and read more */
322+ table_buffer = kmalloc(directory_size, GFP_KERNEL);
323+ if (!table_buffer)
324+ return -ENOMEM;
325+ ret = read_length(dd, SEG_SIZE - directory_size,
326+ directory_size, table_buffer);
327+ if (ret)
328+ goto done;
329+ table = table_buffer;
330+ }
331+
332+ /* look for the platform configuration file in the table */
333+ for (entry = NULL, i = 0; i < footer->num_table_entries; i++) {
334+ if (table[i].type == HFI1_EFT_PLATFORM_CONFIG) {
335+ entry = &table[i];
336+ break;
337+ }
338+ }
339+ if (!entry) {
340+ ret = -ENOENT;
341+ goto done;
342+ }
343+
344+ /*
345+ * Sanity check on the configuration file size - it should never
346+ * be larger than 4 KiB.
347+ */
348+ if (entry->size > (4 * 1024)) {
349+ dd_dev_err(dd, "Bad configuration file size 0x%x\n",
350+ entry->size);
351+ ret = -EINVAL;
352+ goto done;
353+ }
354+
355+ /* check for bogus offset and size that wrap when added together */
356+ if (entry->offset + entry->size < entry->offset) {
357+ dd_dev_err(dd,
358+ "Bad configuration file start + size 0x%x+0x%x\n",
359+ entry->offset, entry->size);
360+ ret = -EINVAL;
361+ goto done;
362+ }
363+
364+ /* allocate the buffer to return */
365+ buffer = kmalloc(entry->size, GFP_KERNEL);
366+ if (!buffer) {
367+ ret = -ENOMEM;
368+ goto done;
369+ }
370+
371+ /*
372+ * Extract the file by looping over segments until it is fully read.
373+ */
374+ seg_offset = entry->offset % SEG_SIZE;
375+ seg_base = entry->offset - seg_offset;
376+ ncopied = 0;
377+ while (ncopied < entry->size) {
378+ /* calculate data bytes available in this segment */
379+
380+ /* start with the bytes from the current offset to the end */
381+ bytes_available = SEG_SIZE - seg_offset;
382+ /* subtract off footer and table from segment 0 */
383+ if (seg_base == 0) {
384+ /*
385+ * Sanity check: should not have a starting point
386+ * at or within the directory.
387+ */
388+ if (bytes_available <= directory_size) {
389+ dd_dev_err(dd,
390+ "Bad configuration file - offset 0x%x within footer+table\n",
391+ entry->offset);
392+ ret = -EINVAL;
393+ goto done;
394+ }
395+ bytes_available -= directory_size;
396+ }
397+
398+ /* calculate bytes wanted */
399+ to_copy = entry->size - ncopied;
400+
401+ /* max out at the available bytes in this segment */
402+ if (to_copy > bytes_available)
403+ to_copy = bytes_available;
404+
405+ /*
406+ * Read from the EPROM.
407+ *
408+ * The sanity check for entry->offset is done in read_length().
409+ * The EPROM offset is validated against what the hardware
410+ * addressing supports. In addition, if the offset is larger
411+ * than the actual EPROM, it silently wraps. It will work
412+ * fine, though the reader may not get what they expected
413+ * from the EPROM.
414+ */
415+ ret = read_length(dd, seg_base + seg_offset, to_copy,
416+ buffer + ncopied);
417+ if (ret)
418+ goto done;
419+
420+ ncopied += to_copy;
421+
422+ /* set up for next segment */
423+ seg_offset = footer->oprom_size;
424+ seg_base += SEG_SIZE;
425+ }
426+
427+ /* success */
428+ ret = 0;
429+ *data = buffer;
430+ *size = entry->size;
431+
432+done:
433+ kfree(table_buffer);
434+ if (ret)
435+ kfree(buffer);
436+ return ret;
437+}
438+
244439 /*
245440 * Read the platform configuration file from the EPROM.
246441 *
@@ -253,6 +448,7 @@ static int read_partition_platform_config(struct hfi1_devdata *dd, void **data, · 6 unmodified lines
253448 * -EBUSY - not able to acquire access to the EPROM
254449 * -ENOENT - no recognizable file written
255450 * -ENOMEM - buffer could not be allocated
451+ * -EINVAL - invalid EPROM contentents found
256452 */
257453 int eprom_read_platform_config(struct hfi1_devdata *dd, void **data, u32 *size)
258454 {
@@ -266,21 +462,20 @@ int eprom_read_platform_config(struct hfi1_devdata *dd, void **data, u32 *size) · 7 unmodified lines
266462 if (ret)
267463 return -EBUSY;
268464
269− /* read the last page of P0 for the EPROM format magic */
270− ret = read_length(dd, P1_START - EP_PAGE_SIZE, EP_PAGE_SIZE, directory);
465+ /* read the last page of the segment for the EPROM format magic */
466+ ret = read_length(dd, SEG_SIZE - EP_PAGE_SIZE, EP_PAGE_SIZE, directory);
271467 if (ret)
272468 goto done;
273469
274− /* last dword of P0 contains a magic indicator */
275− if (directory[EP_PAGE_DWORDS - 1] == 0) {
470+ /* last dword of the segment contains a magic value */
471+ if (directory[EP_PAGE_DWORDS - 1] == FOOTER_MAGIC) {
472+ /* segment format */
473+ ret = read_segment_platform_config(dd, directory, data, size);
474+ } else {
276475 /* partition format */
277476 ret = read_partition_platform_config(dd, data, size);
278− goto done;
279477 }
280478
281− /* nothing recognized */
282− ret = -ENOENT;
283−
284479 done:
285480 release_chip_resource(dd, CR_EPROM);
286481 return ret;
drivers/infiniband/hw/hfi1/firmware.c+133−23
@@ -239,6 +239,16 @@ static const u8 all_fabric_serdes_broadcast = 0xe1; · 238 unmodified lines
239239 const u8 pcie_serdes_broadcast[2] = { 0xe2, 0xe3 };
240240 static const u8 all_pcie_serdes_broadcast = 0xe0;
241241
242+static const u32 platform_config_table_limits[PLATFORM_CONFIG_TABLE_MAX] = {
243+ 0,
244+ SYSTEM_TABLE_MAX,
245+ PORT_TABLE_MAX,
246+ RX_PRESET_TABLE_MAX,
247+ TX_PRESET_TABLE_MAX,
248+ QSFP_ATTEN_TABLE_MAX,
249+ VARIABLE_SETTINGS_TABLE_MAX
250+};
251+
242252 /* forwards */
243253 static void dispose_one_firmware(struct firmware_details *fdet);
244254 static int load_fabric_serdes_firmware(struct hfi1_devdata *dd,
@@ -263,11 +273,13 @@ static int __read_8051_data(struct hfi1_devdata *dd, u32 addr, u64 *result) · 18 unmodified lines
263273 u64 reg;
264274 int count;
265275
266− /* start the read at the given address */
267− reg = ((addr & DC_DC8051_CFG_RAM_ACCESS_CTRL_ADDRESS_MASK)
268− << DC_DC8051_CFG_RAM_ACCESS_CTRL_ADDRESS_SHIFT)
269− | DC_DC8051_CFG_RAM_ACCESS_CTRL_READ_ENA_SMASK;
276+ /* step 1: set the address, clear enable */
277+ reg = (addr & DC_DC8051_CFG_RAM_ACCESS_CTRL_ADDRESS_MASK)
278+ << DC_DC8051_CFG_RAM_ACCESS_CTRL_ADDRESS_SHIFT;
270279 write_csr(dd, DC_DC8051_CFG_RAM_ACCESS_CTRL, reg);
280+ /* step 2: enable */
281+ write_csr(dd, DC_DC8051_CFG_RAM_ACCESS_CTRL,
282+ reg | DC_DC8051_CFG_RAM_ACCESS_CTRL_READ_ENA_SMASK);
271283
272284 /* wait until ACCESS_COMPLETED is set */
273285 count = 0;
@@ -707,6 +719,9 @@ static int obtain_firmware(struct hfi1_devdata *dd) · 433 unmodified lines
707719 &dd->pcidev->dev);
708720 if (err) {
709721 platform_config = NULL;
722+ dd_dev_err(dd,
723+ "%s: No default platform config file found\n",
724+ __func__);
710725 goto done;
711726 }
712727 dd->platform_config.data = platform_config->data;
@@ -1761,8 +1776,17 @@ int parse_platform_config(struct hfi1_devdata *dd) · 1048 unmodified lines
17611776 u32 record_idx = 0, table_type = 0, table_length_dwords = 0;
17621777 int ret = -EINVAL; /* assume failure */
17631778
1779+ /*
1780+ * For integrated devices that did not fall back to the default file,
1781+ * the SI tuning information for active channels is acquired from the
1782+ * scratch register bitmap, thus there is no platform config to parse.
1783+ * Skip parsing in these situations.
1784+ */
1785+ if (is_integrated(dd) && !platform_config_load)
1786+ return 0;
1787+
17641788 if (!dd->platform_config.data) {
1765− dd_dev_info(dd, "%s: Missing config file\n", __func__);
1789+ dd_dev_err(dd, "%s: Missing config file\n", __func__);
17661790 goto bail;
17671791 }
17681792 ptr = (u32 *)dd->platform_config.data;
@@ -1770,7 +1794,7 @@ int parse_platform_config(struct hfi1_devdata *dd) · 1 unmodified line
17701794 magic_num = *ptr;
17711795 ptr++;
17721796 if (magic_num != PLATFORM_CONFIG_MAGIC_NUM) {
1773− dd_dev_info(dd, "%s: Bad config file\n", __func__);
1797+ dd_dev_err(dd, "%s: Bad config file\n", __func__);
17741798 goto bail;
17751799 }
17761800
@@ -1797,9 +1821,9 @@ int parse_platform_config(struct hfi1_devdata *dd) · 20 unmodified lines
17971821 header1 = *ptr;
17981822 header2 = *(ptr + 1);
17991823 if (header1 != ~header2) {
1800− dd_dev_info(dd, "%s: Failed validation at offset %ld\n",
1801− __func__, (ptr - (u32 *)
1802− dd->platform_config.data));
1824+ dd_dev_err(dd, "%s: Failed validation at offset %ld\n",
1825+ __func__, (ptr - (u32 *)
1826+ dd->platform_config.data));
18031827 goto bail;
18041828 }
18051829
@@ -1841,11 +1865,11 @@ int parse_platform_config(struct hfi1_devdata *dd) · 35 unmodified lines
18411865 table_length_dwords;
18421866 break;
18431867 default:
1844− dd_dev_info(dd,
1845− "%s: Unknown data table %d, offset %ld\n",
1846− __func__, table_type,
1847− (ptr - (u32 *)
1848− dd->platform_config.data));
1868+ dd_dev_err(dd,
1869+ "%s: Unknown data table %d, offset %ld\n",
1870+ __func__, table_type,
1871+ (ptr - (u32 *)
1872+ dd->platform_config.data));
18491873 goto bail; /* We don't trust this file now */
18501874 }
18511875 pcfgcache->config_tables[table_type].table = ptr;
@@ -1865,11 +1889,11 @@ int parse_platform_config(struct hfi1_devdata *dd) · 13 unmodified lines
18651889 case PLATFORM_CONFIG_VARIABLE_SETTINGS_TABLE:
18661890 break;
18671891 default:
1868− dd_dev_info(dd,
1869− "%s: Unknown meta table %d, offset %ld\n",
1870− __func__, table_type,
1871− (ptr -
1872− (u32 *)dd->platform_config.data));
1892+ dd_dev_err(dd,
1893+ "%s: Unknown meta table %d, offset %ld\n",
1894+ __func__, table_type,
1895+ (ptr -
1896+ (u32 *)dd->platform_config.data));
18731897 goto bail; /* We don't trust this file now */
18741898 }
18751899 pcfgcache->config_tables[table_type].table_metadata =
@@ -1884,10 +1908,9 @@ int parse_platform_config(struct hfi1_devdata *dd) · 8 unmodified lines
18841908 /* Jump the table */
18851909 ptr += table_length_dwords;
18861910 if (crc != *ptr) {
1887− dd_dev_info(dd, "%s: Failed CRC check at offset %ld\n",
1888− __func__, (ptr -
1889− (u32 *)
1890− dd->platform_config.data));
1911+ dd_dev_err(dd, "%s: Failed CRC check at offset %ld\n",
1912+ __func__, (ptr -
1913+ (u32 *)dd->platform_config.data));
18911914 goto bail;
18921915 }
18931916 /* Jump the CRC DWORD */
@@ -1901,6 +1924,84 @@ bail: · 7 unmodified lines
19011924 return ret;
19021925 }
19031926
1927+static void get_integrated_platform_config_field(
1928+ struct hfi1_devdata *dd,
1929+ enum platform_config_table_type_encoding table_type,
1930+ int field_index, u32 *data)
1931+{
1932+ struct hfi1_pportdata *ppd = dd->pport;
1933+ u8 *cache = ppd->qsfp_info.cache;
1934+ u32 tx_preset = 0;
1935+
1936+ switch (table_type) {
1937+ case PLATFORM_CONFIG_SYSTEM_TABLE:
1938+ if (field_index == SYSTEM_TABLE_QSFP_POWER_CLASS_MAX)
1939+ *data = ppd->max_power_class;
1940+ else if (field_index == SYSTEM_TABLE_QSFP_ATTENUATION_DEFAULT_25G)
1941+ *data = ppd->default_atten;
1942+ break;
1943+ case PLATFORM_CONFIG_PORT_TABLE:
1944+ if (field_index == PORT_TABLE_PORT_TYPE)
1945+ *data = ppd->port_type;
1946+ else if (field_index == PORT_TABLE_LOCAL_ATTEN_25G)
1947+ *data = ppd->local_atten;
1948+ else if (field_index == PORT_TABLE_REMOTE_ATTEN_25G)
1949+ *data = ppd->remote_atten;
1950+ break;
1951+ case PLATFORM_CONFIG_RX_PRESET_TABLE:
1952+ if (field_index == RX_PRESET_TABLE_QSFP_RX_CDR_APPLY)
1953+ *data = (ppd->rx_preset & QSFP_RX_CDR_APPLY_SMASK) >>
1954+ QSFP_RX_CDR_APPLY_SHIFT;
1955+ else if (field_index == RX_PRESET_TABLE_QSFP_RX_EMP_APPLY)
1956+ *data = (ppd->rx_preset & QSFP_RX_EMP_APPLY_SMASK) >>
1957+ QSFP_RX_EMP_APPLY_SHIFT;
1958+ else if (field_index == RX_PRESET_TABLE_QSFP_RX_AMP_APPLY)
1959+ *data = (ppd->rx_preset & QSFP_RX_AMP_APPLY_SMASK) >>
1960+ QSFP_RX_AMP_APPLY_SHIFT;
1961+ else if (field_index == RX_PRESET_TABLE_QSFP_RX_CDR)
1962+ *data = (ppd->rx_preset & QSFP_RX_CDR_SMASK) >>
1963+ QSFP_RX_CDR_SHIFT;
1964+ else if (field_index == RX_PRESET_TABLE_QSFP_RX_EMP)
1965+ *data = (ppd->rx_preset & QSFP_RX_EMP_SMASK) >>
1966+ QSFP_RX_EMP_SHIFT;
1967+ else if (field_index == RX_PRESET_TABLE_QSFP_RX_AMP)
1968+ *data = (ppd->rx_preset & QSFP_RX_AMP_SMASK) >>
1969+ QSFP_RX_AMP_SHIFT;
1970+ break;
1971+ case PLATFORM_CONFIG_TX_PRESET_TABLE:
1972+ if (cache[QSFP_EQ_INFO_OFFS] & 0x4)
1973+ tx_preset = ppd->tx_preset_eq;
1974+ else
1975+ tx_preset = ppd->tx_preset_noeq;
1976+ if (field_index == TX_PRESET_TABLE_PRECUR)
1977+ *data = (tx_preset & TX_PRECUR_SMASK) >>
1978+ TX_PRECUR_SHIFT;
1979+ else if (field_index == TX_PRESET_TABLE_ATTN)
1980+ *data = (tx_preset & TX_ATTN_SMASK) >>
1981+ TX_ATTN_SHIFT;
1982+ else if (field_index == TX_PRESET_TABLE_POSTCUR)
1983+ *data = (tx_preset & TX_POSTCUR_SMASK) >>
1984+ TX_POSTCUR_SHIFT;
1985+ else if (field_index == TX_PRESET_TABLE_QSFP_TX_CDR_APPLY)
1986+ *data = (tx_preset & QSFP_TX_CDR_APPLY_SMASK) >>
1987+ QSFP_TX_CDR_APPLY_SHIFT;
1988+ else if (field_index == TX_PRESET_TABLE_QSFP_TX_EQ_APPLY)
1989+ *data = (tx_preset & QSFP_TX_EQ_APPLY_SMASK) >>
1990+ QSFP_TX_EQ_APPLY_SHIFT;
1991+ else if (field_index == TX_PRESET_TABLE_QSFP_TX_CDR)
1992+ *data = (tx_preset & QSFP_TX_CDR_SMASK) >>
1993+ QSFP_TX_CDR_SHIFT;
1994+ else if (field_index == TX_PRESET_TABLE_QSFP_TX_EQ)
1995+ *data = (tx_preset & QSFP_TX_EQ_SMASK) >>
1996+ QSFP_TX_EQ_SHIFT;
1997+ break;
1998+ case PLATFORM_CONFIG_QSFP_ATTEN_TABLE:
1999+ case PLATFORM_CONFIG_VARIABLE_SETTINGS_TABLE:
2000+ default:
2001+ break;
2002+ }
2003+}
2004+
19042005 static int get_platform_fw_field_metadata(struct hfi1_devdata *dd, int table,
19052006 int field, u32 *field_len_bits,
19062007 u32 *field_start_bits)
@@ -1976,6 +2077,15 @@ int get_platform_config_field(struct hfi1_devdata *dd, · 69 unmodified lines
19762077 else
19772078 return -EINVAL;
19782079
2080+ if (is_integrated(dd) && !platform_config_load) {
2081+ /*
2082+ * Use saved configuration from ppd for integrated platforms
2083+ */
2084+ get_integrated_platform_config_field(dd, table_type,
2085+ field_index, data);
2086+ return 0;
2087+ }
2088+
19792089 ret = get_platform_fw_field_metadata(dd, table_type, field_index,
19802090 &field_len_bits,
19812091 &field_start_bits);
drivers/infiniband/hw/hfi1/hfi.h+87−57
@@ -492,6 +492,9 @@ struct rvt_sge_state; · 491 unmodified lines
492492 #define HFI1_MIN_VLS_SUPPORTED 1
493493 #define HFI1_MAX_VLS_SUPPORTED 8
494494
495+#define HFI1_GUIDS_PER_PORT 5
496+#define HFI1_PORT_GUID_INDEX 0
497+
495498 static inline void incr_cntr64(u64 *cntr)
496499 {
497500 if (*cntr < (u64)-1LL)
@@ -559,11 +562,20 @@ struct hfi1_pportdata { · 61 unmodified lines
559562 struct kobject vl2mtu_kobj;
560563
561564 /* PHY support */
562− u32 port_type;
563565 struct qsfp_data qsfp_info;
566+ /* Values for SI tuning of SerDes */
567+ u32 port_type;
568+ u32 tx_preset_eq;
569+ u32 tx_preset_noeq;
570+ u32 rx_preset;
571+ u8 local_atten;
572+ u8 remote_atten;
573+ u8 default_atten;
574+ u8 max_power_class;
575+
576+ /* GUIDs for this interface, in host order, guids[0] is a port guid */
577+ u64 guids[HFI1_GUIDS_PER_PORT];
564578
565− /* GUID for this interface, in host order */
566− u64 guid;
567579 /* GUID for peer interface, in host order */
568580 u64 neighbor_guid;
569581
@@ -826,32 +838,29 @@ struct hfi1_devdata { · 256 unmodified lines
826838 u8 __iomem *kregend;
827839 /* physical address of chip for io_remap, etc. */
828840 resource_size_t physaddr;
829− /* receive context data */
830− struct hfi1_ctxtdata **rcd;
841+ /* Per VL data. Enough for all VLs but not all elements are set/used. */
842+ struct per_vl_data vld[PER_VL_SEND_CONTEXTS];
831843 /* send context data */
832844 struct send_context_info *send_contexts;
833845 /* map hardware send contexts to software index */
834846 u8 *hw_to_sw;
835847 /* spinlock for allocating and releasing send context resources */
836848 spinlock_t sc_lock;
837− /* Per VL data. Enough for all VLs but not all elements are set/used. */
838− struct per_vl_data vld[PER_VL_SEND_CONTEXTS];
839849 /* lock for pio_map */
840850 spinlock_t pio_map_lock;
851+ /* Send Context initialization lock. */
852+ spinlock_t sc_init_lock;
853+ /* lock for sdma_map */
854+ spinlock_t sde_map_lock;
841855 /* array of kernel send contexts */
842856 struct send_context **kernel_send_context;
843857 /* array of vl maps */
844858 struct pio_vl_map __rcu *pio_map;
845− /* seqlock for sc2vl */
846− seqlock_t sc2vl_lock;
847− u64 sc2vl[4];
848− /* Send Context initialization lock. */
849− spinlock_t sc_init_lock;
859+ /* default flags to last descriptor */
860+ u64 default_desc1;
850861
851862 /* fields common to all SDMA engines */
852863
853− /* default flags to last descriptor */
854− u64 default_desc1;
855864 volatile __le64 *sdma_heads_dma; /* DMA'ed by chip */
856865 dma_addr_t sdma_heads_phys;
857866 void *sdma_pad_dma; /* DMA'ed by chip */
@@ -862,8 +871,6 @@ struct hfi1_devdata { · 4 unmodified lines
862871 u32 chip_sdma_engines;
863872 /* num used */
864873 u32 num_sdma;
865− /* lock for sdma_map */
866− spinlock_t sde_map_lock;
867874 /* array of engines sized by num_sdma */
868875 struct sdma_engine *per_sdma;
869876 /* array of vl maps */
@@ -872,14 +879,11 @@ struct hfi1_devdata { · 2 unmodified lines
872879 wait_queue_head_t sdma_unfreeze_wq;
873880 atomic_t sdma_unfreeze_count;
874881
882+ u32 lcb_access_count; /* count of LCB users */
883+
875884 /* common data between shared ASIC HFIs in this OS */
876885 struct hfi1_asic_data *asic_data;
877886
878− /* hfi1_pportdata, points to array of (physical) port-specific
879− * data structs, indexed by pidx (0..n-1)
880− */
881− struct hfi1_pportdata *pport;
882−
883887 /* mem-mapped pointer to base of PIO buffers */
884888 void __iomem *piobase;
885889 /*
@@ -896,20 +900,13 @@ struct hfi1_devdata { · 10 unmodified lines
896900 /* send context numbers and sizes for each type */
897901 struct sc_config_sizes sc_sizes[SC_MAX];
898902
899− u32 lcb_access_count; /* count of LCB users */
900−
901903 char *boardname; /* human readable board info */
902904
903− /* device (not port) flags, basically device capabilities */
904− u32 flags;
905−
906905 /* reset value */
907906 u64 z_int_counter;
908907 u64 z_rcv_limit;
909908 u64 z_send_schedule;
910− /* percpu int_counter */
911− u64 __percpu *int_counter;
912− u64 __percpu *rcv_limit;
909+
913910 u64 __percpu *send_schedule;
914911 /* number of receive contexts in use by the driver */
915912 u32 num_rcv_contexts;
@@ -924,6 +921,7 @@ struct hfi1_devdata { · 8 unmodified lines
924921 /* base receive interrupt timeout, in CSR units */
925922 u32 rcv_intr_timeout_csr;
926923
924+ u32 freezelen; /* max length of freezemsg */
927925 u64 __iomem *egrtidbase;
928926 spinlock_t sendctrl_lock; /* protect changes to SendCtrl */
929927 spinlock_t rcvctrl_lock; /* protect changes to RcvCtrl */
@@ -945,7 +943,6 @@ struct hfi1_devdata { · 15 unmodified lines
945943 * IB link status cheaply
946944 */
947945 struct hfi1_status *status;
948− u32 freezelen; /* max length of freezemsg */
949946
950947 /* revision register shadow */
951948 u64 revision;
@@ -973,6 +970,8 @@ struct hfi1_devdata { · 21 unmodified lines
973970 u16 rcvegrbufsize_shift;
974971 /* both sides of the PCIe link are gen3 capable */
975972 u8 link_gen3_capable;
973+ /* default link down value (poll/sleep) */
974+ u8 link_default;
976975 /* localbus width (1, 2,4,8,16,32) from config space */
977976 u32 lbus_width;
978977 /* localbus speed in MHz */
@@ -1008,8 +1007,6 @@ struct hfi1_devdata { · 29 unmodified lines
10081007 u8 hfi1_id;
10091008 /* implementation code */
10101009 u8 icode;
1011− /* default link down value (poll/sleep) */
1012− u8 link_default;
10131010 /* vAU of this device */
10141011 u8 vau;
10151012 /* vCU of this device */
@@ -1020,27 +1017,17 @@ struct hfi1_devdata { · 4 unmodified lines
10201017 u16 vl15_init;
10211018
10221019 /* Misc small ints */
1023− /* Number of physical ports available */
1024− u8 num_pports;
1025− /* Lowest context number which can be used by user processes */
1026− u8 first_user_ctxt;
10271020 u8 n_krcv_queues;
10281021 u8 qos_shift;
1029− u8 qpn_mask;
10301022
1031− u16 rhf_offset; /* offset of RHF within receive header entry */
10321023 u16 irev; /* implementation revision */
10331024 u16 dc8051_ver; /* 8051 firmware version */
10341025
1026+ spinlock_t hfi1_diag_trans_lock; /* protect diag observer ops */
10351027 struct platform_config platform_config;
10361028 struct platform_config_cache pcfg_cache;
10371029
10381030 struct diag_client *diag_client;
1039− spinlock_t hfi1_diag_trans_lock; /* protect diag observer ops */
1040−
1041− u8 psxmitwait_supported;
1042− /* cycle length of PS* counters in HW (in picoseconds) */
1043− u16 psxmitwait_check_rate;
10441031
10451032 /* MSI-X information */
10461033 struct hfi1_msix_entry *msix_entries;
@@ -1055,6 +1042,9 @@ struct hfi1_devdata { · 8 unmodified lines
10551042
10561043 struct rcv_array_data rcv_entries;
10571044
1045+ /* cycle length of PS* counters in HW (in picoseconds) */
1046+ u16 psxmitwait_check_rate;
1047+
10581048 /*
10591049 * 64 bit synthetic counters
10601050 */
@@ -1085,11 +1075,11 @@ struct hfi1_devdata { · 24 unmodified lines
10851075 struct err_info_rcvport err_info_rcvport;
10861076 struct err_info_constraint err_info_rcv_constraint;
10871077 struct err_info_constraint err_info_xmit_constraint;
1088− u8 err_info_uncorrectable;
1089− u8 err_info_fmconfig;
10901078
10911079 atomic_t drop_packet;
10921080 u8 do_drop;
1081+ u8 err_info_uncorrectable;
1082+ u8 err_info_fmconfig;
10931083
10941084 /*
10951085 * Software counters for the status bits defined by the
@@ -1112,40 +1102,60 @@ struct hfi1_devdata { · 16 unmodified lines
11121102 u64 sw_cce_err_status_aggregate;
11131103 /* Software counter that aggregates all bypass packet rcv errors */
11141104 u64 sw_rcv_bypass_packet_errors;
1115− /* receive interrupt functions */
1116− rhf_rcv_function_ptr *rhf_rcv_function_map;
1105+ /* receive interrupt function */
11171106 rhf_rcv_function_ptr normal_rhf_rcv_functions[8];
11181107
1108+ /* Save the enabled LCB error bits */
1109+ u64 lcb_err_en;
1110+
11191111 /*
11201112 * Capability to have different send engines simply by changing a
11211113 * pointer value.
11221114 */
1123− send_routine process_pio_send;
1115+ send_routine process_pio_send ____cacheline_aligned_in_smp;
11241116 send_routine process_dma_send;
11251117 void (*pio_inline_send)(struct hfi1_devdata *dd, struct pio_buf *pbuf,
11261118 u64 pbc, const void *from, size_t count);
1119+ /* hfi1_pportdata, points to array of (physical) port-specific
1120+ * data structs, indexed by pidx (0..n-1)
1121+ */
1122+ struct hfi1_pportdata *pport;
1123+ /* receive context data */
1124+ struct hfi1_ctxtdata **rcd;
1125+ u64 __percpu *int_counter;
1126+ /* device (not port) flags, basically device capabilities */
1127+ u16 flags;
1128+ /* Number of physical ports available */
1129+ u8 num_pports;
1130+ /* Lowest context number which can be used by user processes */
1131+ u8 first_user_ctxt;
1132+ /* adding a new field here would make it part of this cacheline */
1133+
1134+ /* seqlock for sc2vl */
1135+ seqlock_t sc2vl_lock ____cacheline_aligned_in_smp;
1136+ u64 sc2vl[4];
1137+ /* receive interrupt functions */
1138+ rhf_rcv_function_ptr *rhf_rcv_function_map;
1139+ u64 __percpu *rcv_limit;
1140+ u16 rhf_offset; /* offset of RHF within receive header entry */
1141+ /* adding a new field here would make it part of this cacheline */
11271142
11281143 /* OUI comes from the HW. Used everywhere as 3 separate bytes. */
11291144 u8 oui1;
11301145 u8 oui2;
11311146 u8 oui3;
1147+ u8 dc_shutdown;
1148+
11321149 /* Timer and counter used to detect RcvBufOvflCnt changes */
11331150 struct timer_list rcverr_timer;
1134− u32 rcv_ovfl_cnt;
11351151
11361152 wait_queue_head_t event_queue;
11371153
1138− /* Save the enabled LCB error bits */
1139− u64 lcb_err_en;
1140− u8 dc_shutdown;
1141−
11421154 /* receive context tail dummy address */
11431155 __le64 *rcvhdrtail_dummy_kvaddr;
11441156 dma_addr_t rcvhdrtail_dummy_dma;
11451157
1146− bool eprom_available; /* true if EPROM is available for this device */
1147− bool aspm_supported; /* Does HW support ASPM */
1148− bool aspm_enabled; /* ASPM state: enabled/disabled */
1158+ u32 rcv_ovfl_cnt;
11491159 /* Serialize ASPM enable/disable between multiple verbs contexts */
11501160 spinlock_t aspm_lock;
11511161 /* Number of verbs contexts which have disabled ASPM */
@@ -1155,8 +1165,11 @@ struct hfi1_devdata { · 3 unmodified lines
11551165 /* Used to wait for outstanding user space clients before dev removal */
11561166 struct completion user_comp;
11571167
1158− struct hfi1_affinity *affinity;
1168+ bool eprom_available; /* true if EPROM is available for this device */
1169+ bool aspm_supported; /* Does HW support ASPM */
1170+ bool aspm_enabled; /* ASPM state: enabled/disabled */
11591171 struct rhashtable sdma_rht;
1172+
11601173 struct kobject kobj;
11611174 };
11621175
@@ -1603,6 +1616,17 @@ static inline u16 hfi1_get_pkey(struct hfi1_ibport *ibp, unsigned index) · 440 unmodified lines
16031616 return ret;
16041617 }
16051618
1619+/*
1620+ * Return the indexed GUID from the port GUIDs table.
1621+ */
1622+static inline __be64 get_sguid(struct hfi1_ibport *ibp, unsigned int index)
1623+{
1624+ struct hfi1_pportdata *ppd = ppd_from_ibp(ibp);
1625+
1626+ WARN_ON(index >= HFI1_GUIDS_PER_PORT);
1627+ return cpu_to_be64(ppd->guids[index]);
1628+}
1629+
16061630 /*
16071631 * Called by readers of cc_state only, must call under rcu_read_lock().
16081632 */
@@ -1982,6 +2006,12 @@ static inline u32 qsfp_resource(struct hfi1_devdata *dd) · 373 unmodified lines
19822006 return i2c_target(dd->hfi1_id);
19832007 }
19842008
2009+/* Is this device integrated or discrete? */
2010+static inline bool is_integrated(struct hfi1_devdata *dd)
2011+{
2012+ return dd->pcidev->device == PCI_DEVICE_ID_INTEL1;
2013+}
2014+
19852015 int hfi1_tempsense_rd(struct hfi1_devdata *dd, struct hfi1_temp *temp);
19862016
19872017 #define DD_DEV_ENTRY(dd) __string(dev, dev_name(&(dd)->pcidev->dev))
drivers/infiniband/hw/hfi1/iowait.h+8−0
@@ -64,6 +64,7 @@ struct sdma_engine; · 63 unmodified lines
6464 /**
6565 * struct iowait - linkage for delayed progress/waiting
6666 * @list: used to add/insert into QP/PQ wait lists
67+ * @lock: uses to record the list head lock
6768 * @tx_head: overflow list of sdma_txreq's
6869 * @sleep: no space callback
6970 * @wakeup: space callback wakeup
@@ -91,6 +92,11 @@ struct sdma_engine; · 21 unmodified lines
9192 * so sleeping is not allowed.
9293 *
9394 * The wait_dma member along with the iow
95+ *
96+ * The lock field is used by waiters to record
97+ * the seqlock_t that guards the list head.
98+ * Waiters explicity know that, but the destroy
99+ * code that unwaits QPs does not.
94100 */
95101
96102 struct iowait {
@@ -103,6 +109,7 @@ struct iowait { · 6 unmodified lines
103109 unsigned seq);
104110 void (*wakeup)(struct iowait *wait, int reason);
105111 void (*sdma_drained)(struct iowait *wait);
112+ seqlock_t *lock;
106113 struct work_struct iowork;
107114 wait_queue_head_t wait_dma;
108115 wait_queue_head_t wait_pio;
@@ -141,6 +148,7 @@ static inline void iowait_init( · 32 unmodified lines
141148 void (*sdma_drained)(struct iowait *wait))
142149 {
143150 wait->count = 0;
151+ wait->lock = NULL;
144152 INIT_LIST_HEAD(&wait->list);
145153 INIT_LIST_HEAD(&wait->tx_head);
146154 INIT_WORK(&wait->iowork, func);
drivers/infiniband/hw/hfi1/mad.c+16−15
@@ -128,7 +128,7 @@ static void send_trap(struct hfi1_ibport *ibp, void *data, unsigned len) · 127 unmodified lines
128128 smp = send_buf->mad;
129129 smp->base_version = OPA_MGMT_BASE_VERSION;
130130 smp->mgmt_class = IB_MGMT_CLASS_SUBN_LID_ROUTED;
131− smp->class_version = OPA_SMI_CLASS_VERSION;
131+ smp->class_version = OPA_SM_CLASS_VERSION;
132132 smp->method = IB_MGMT_METHOD_TRAP;
133133 ibp->rvp.tid++;
134134 smp->tid = cpu_to_be64(ibp->rvp.tid);
@@ -336,20 +336,20 @@ static int __subn_get_opa_nodeinfo(struct opa_smp *smp, u32 am, u8 *data, · 201 unmodified lines
336336 ni = (struct opa_node_info *)data;
337337
338338 /* GUID 0 is illegal */
339− if (am || pidx >= dd->num_pports || dd->pport[pidx].guid == 0) {
339+ if (am || pidx >= dd->num_pports || ibdev->node_guid == 0 ||
340+ get_sguid(to_iport(ibdev, port), HFI1_PORT_GUID_INDEX) == 0) {
340341 smp->status |= IB_SMP_INVALID_FIELD;
341342 return reply((struct ib_mad_hdr *)smp);
342343 }
343344
344− ni->port_guid = cpu_to_be64(dd->pport[pidx].guid);
345+ ni->port_guid = get_sguid(to_iport(ibdev, port), HFI1_PORT_GUID_INDEX);
345346 ni->base_version = OPA_MGMT_BASE_VERSION;
346− ni->class_version = OPA_SMI_CLASS_VERSION;
347+ ni->class_version = OPA_SM_CLASS_VERSION;
347348 ni->node_type = 1; /* channel adapter */
348349 ni->num_ports = ibdev->phys_port_cnt;
349350 /* This is already in network order */
350351 ni->system_image_guid = ib_hfi1_sys_image_guid;
351− /* Use first-port GUID as node */
352− ni->node_guid = cpu_to_be64(dd->pport->guid);
352+ ni->node_guid = ibdev->node_guid;
353353 ni->partition_cap = cpu_to_be16(hfi1_get_npkeys(dd));
354354 ni->device_id = cpu_to_be16(dd->pcidev->device);
355355 ni->revision = cpu_to_be32(dd->minrev);
@@ -373,19 +373,20 @@ static int subn_get_nodeinfo(struct ib_smp *smp, struct ib_device *ibdev, · 17 unmodified lines
373373
374374 /* GUID 0 is illegal */
375375 if (smp->attr_mod || pidx >= dd->num_pports ||
376− dd->pport[pidx].guid == 0)
376+ ibdev->node_guid == 0 ||
377+ get_sguid(to_iport(ibdev, port), HFI1_PORT_GUID_INDEX) == 0) {
377378 smp->status |= IB_SMP_INVALID_FIELD;
378− else
379− nip->port_guid = cpu_to_be64(dd->pport[pidx].guid);
379+ return reply((struct ib_mad_hdr *)smp);
380+ }
380381
382+ nip->port_guid = get_sguid(to_iport(ibdev, port), HFI1_PORT_GUID_INDEX);
381383 nip->base_version = OPA_MGMT_BASE_VERSION;
382− nip->class_version = OPA_SMI_CLASS_VERSION;
384+ nip->class_version = OPA_SM_CLASS_VERSION;
383385 nip->node_type = 1; /* channel adapter */
384386 nip->num_ports = ibdev->phys_port_cnt;
385387 /* This is already in network order */
386388 nip->sys_guid = ib_hfi1_sys_image_guid;
387− /* Use first-port GUID as node */
388− nip->node_guid = cpu_to_be64(dd->pport->guid);
389+ nip->node_guid = ibdev->node_guid;
389390 nip->partition_cap = cpu_to_be16(hfi1_get_npkeys(dd));
390391 nip->device_id = cpu_to_be16(dd->pcidev->device);
391392 nip->revision = cpu_to_be32(dd->minrev);
@@ -2302,7 +2303,7 @@ static int pma_get_opa_classportinfo(struct opa_pma_mad *pmp, · 1910 unmodified lines
23022303 pmp->mad_hdr.status |= IB_SMP_INVALID_FIELD;
23032304
23042305 p->base_version = OPA_MGMT_BASE_VERSION;
2305− p->class_version = OPA_SMI_CLASS_VERSION;
2306+ p->class_version = OPA_SM_CLASS_VERSION;
23062307 /*
23072308 * Expected response time is 4.096 usec. * 2^18 == 1.073741824 sec.
23082309 */
@@ -4022,7 +4023,7 @@ static int process_subn_opa(struct ib_device *ibdev, int mad_flags, · 1713 unmodified lines
40224023
40234024 am = be32_to_cpu(smp->attr_mod);
40244025 attr_id = smp->attr_id;
4025− if (smp->class_version != OPA_SMI_CLASS_VERSION) {
4026+ if (smp->class_version != OPA_SM_CLASS_VERSION) {
40264027 smp->status |= IB_SMP_UNSUP_VERSION;
40274028 ret = reply((struct ib_mad_hdr *)smp);
40284029 return ret;
@@ -4232,7 +4233,7 @@ static int process_perf_opa(struct ib_device *ibdev, u8 port, · 203 unmodified lines
42324233
42334234 *out_mad = *in_mad;
42344235
4235− if (pmp->mad_hdr.class_version != OPA_SMI_CLASS_VERSION) {
4236+ if (pmp->mad_hdr.class_version != OPA_SM_CLASS_VERSION) {
42364237 pmp->mad_hdr.status |= IB_SMP_UNSUP_VERSION;
42374238 return reply((struct ib_mad_hdr *)pmp);
42384239 }
drivers/infiniband/hw/hfi1/mmu_rb.c+1−1
@@ -81,7 +81,7 @@ static void do_remove(struct mmu_rb_handler *handler, · 80 unmodified lines
8181 struct list_head *del_list);
8282 static void handle_remove(struct work_struct *work);
8383
84−static struct mmu_notifier_ops mn_opts = {
84+static const struct mmu_notifier_ops mn_opts = {
8585 .invalidate_page = mmu_notifier_page,
8686 .invalidate_range_start = mmu_notifier_range_start,
8787 };
drivers/infiniband/hw/hfi1/pio.c+13−27
@@ -758,6 +758,7 @@ struct send_context *sc_alloc(struct hfi1_devdata *dd, int type, · 757 unmodified lines
758758 sc->hw_context = hw_context;
759759 cr_group_addresses(sc, &dma);
760760 sc->credits = sci->credits;
761+ sc->size = sc->credits * PIO_BLOCK_SIZE;
761762
762763 /* PIO Send Memory Address details */
763764 #define PIO_ADDR_CONTEXT_MASK 0xfful
@@ -1242,6 +1243,7 @@ int sc_enable(struct send_context *sc) · 478 unmodified lines
12421243 sc->free = 0;
12431244 sc->alloc_free = 0;
12441245 sc->fill = 0;
1246+ sc->fill_wrap = 0;
12451247 sc->sr_head = 0;
12461248 sc->sr_tail = 0;
12471249 sc->flags = 0;
@@ -1385,7 +1387,7 @@ struct pio_buf *sc_buffer_alloc(struct send_context *sc, u32 dw_len, · 137 unmodified lines
13851387 unsigned long flags;
13861388 unsigned long avail;
13871389 unsigned long blocks = dwords_to_blocks(dw_len);
1388− unsigned long start_fill;
1390+ u32 fill_wrap;
13891391 int trycount = 0;
13901392 u32 head, next;
13911393
@@ -1410,9 +1412,7 @@ retry: · 18 unmodified lines
14101412 (sc->fill - sc->alloc_free);
14111413 if (blocks > avail) {
14121414 /* still no room, actively update */
1413− spin_unlock_irqrestore(&sc->alloc_lock, flags);
14141415 sc_release_update(sc);
1415− spin_lock_irqsave(&sc->alloc_lock, flags);
14161416 sc->alloc_free = ACCESS_ONCE(sc->free);
14171417 trycount++;
14181418 goto retry;
@@ -1428,8 +1428,11 @@ retry: · 9 unmodified lines
14281428 head = sc->sr_head;
14291429
14301430 /* "allocate" the buffer */
1431− start_fill = sc->fill;
14321431 sc->fill += blocks;
1432+ fill_wrap = sc->fill_wrap;
1433+ sc->fill_wrap += blocks;
1434+ if (sc->fill_wrap >= sc->credits)
1435+ sc->fill_wrap = sc->fill_wrap - sc->credits;
14331436
14341437 /*
14351438 * Fill the parts that the releaser looks at before moving the head.
@@ -1458,11 +1461,8 @@ retry: · 22 unmodified lines
14581461 spin_unlock_irqrestore(&sc->alloc_lock, flags);
14591462
14601463 /* finish filling in the buffer outside the lock */
1461− pbuf->start = sc->base_addr + ((start_fill % sc->credits)
1462− * PIO_BLOCK_SIZE);
1463− pbuf->size = sc->credits * PIO_BLOCK_SIZE;
1464− pbuf->end = sc->base_addr + pbuf->size;
1465− pbuf->block_count = blocks;
1464+ pbuf->start = sc->base_addr + fill_wrap * PIO_BLOCK_SIZE;
1465+ pbuf->end = sc->base_addr + sc->size;
14661466 pbuf->qw_written = 0;
14671467 pbuf->carry_bytes = 0;
14681468 pbuf->carry.val64 = 0;
@@ -1573,6 +1573,7 @@ static void sc_piobufavail(struct send_context *sc) · 104 unmodified lines
15731573 qp = iowait_to_qp(wait);
15741574 priv = qp->priv;
15751575 list_del_init(&priv->s_iowait.list);
1576+ priv->s_iowait.lock = NULL;
15761577 /* refcount held until actual wake up */
15771578 qps[n++] = qp;
15781579 }
@@ -2028,29 +2029,17 @@ freesc15: · 449 unmodified lines
20282029 int init_credit_return(struct hfi1_devdata *dd)
20292030 {
20302031 int ret;
2031− int num_numa;
20322032 int i;
20332033
2034− num_numa = num_online_nodes();
2035− /* enforce the expectation that the numas are compact */
2036− for (i = 0; i < num_numa; i++) {
2037− if (!node_online(i)) {
2038− dd_dev_err(dd, "NUMA nodes are not compact\n");
2039− ret = -EINVAL;
2040− goto done;
2041− }
2042− }
2043−
20442034 dd->cr_base = kcalloc(
2045− num_numa,
2035+ node_affinity.num_possible_nodes,
20462036 sizeof(struct credit_return_base),
20472037 GFP_KERNEL);
20482038 if (!dd->cr_base) {
2049− dd_dev_err(dd, "Unable to allocate credit return base\n");
20502039 ret = -ENOMEM;
20512040 goto done;
20522041 }
2053− for (i = 0; i < num_numa; i++) {
2042+ for_each_node_with_cpus(i) {
20542043 int bytes = TXE_NUM_CONTEXTS * sizeof(struct credit_return);
20552044
20562045 set_dev_node(&dd->pcidev->dev, i);
@@ -2077,14 +2066,11 @@ done: · 20 unmodified lines
20772066
20782067 void free_credit_return(struct hfi1_devdata *dd)
20792068 {
2080− int num_numa;
20812069 int i;
20822070
20832071 if (!dd->cr_base)
20842072 return;
2085−
2086− num_numa = num_online_nodes();
2087− for (i = 0; i < num_numa; i++) {
2073+ for (i = 0; i < node_affinity.num_possible_nodes; i++) {
20882074 if (dd->cr_base[i].va) {
20892075 dma_free_coherent(&dd->pcidev->dev,
20902076 TXE_NUM_CONTEXTS *
drivers/infiniband/hw/hfi1/pio.h+20−18
@@ -83,53 +83,55 @@ struct pio_buf { · 82 unmodified lines
8383 void *arg; /* argument for cb */
8484 void __iomem *start; /* buffer start address */
8585 void __iomem *end; /* context end address */
86− unsigned long size; /* context size, in bytes */
8786 unsigned long sent_at; /* buffer is sent when <= free */
88− u32 block_count; /* size of buffer, in blocks */
89− u32 qw_written; /* QW written so far */
90− u32 carry_bytes; /* number of valid bytes in carry */
9187 union mix carry; /* pending unwritten bytes */
88+ u16 qw_written; /* QW written so far */
89+ u8 carry_bytes; /* number of valid bytes in carry */
9290 };
9391
9492 /* cache line aligned pio buffer array */
9593 union pio_shadow_ring {
9694 struct pio_buf pbuf;
97− u64 unused[16]; /* cache line spacer */
9895 } ____cacheline_aligned;
9996
10097 /* per-NUMA send context */
10198 struct send_context {
10299 /* read-only after init */
103100 struct hfi1_devdata *dd; /* device */
104− void __iomem *base_addr; /* start of PIO memory */
105101 union pio_shadow_ring *sr; /* shadow ring */
102+ void __iomem *base_addr; /* start of PIO memory */
103+ u32 __percpu *buffers_allocated;/* count of buffers allocated */
104+ u32 size; /* context size, in bytes */
106105
107− volatile __le64 *hw_free; /* HW free counter */
108− struct work_struct halt_work; /* halted context work queue entry */
109− unsigned long flags; /* flags */
110106 int node; /* context home node */
111− int type; /* context type */
112− u32 sw_index; /* software index number */
113− u32 hw_context; /* hardware context number */
114− u32 credits; /* number of blocks in context */
115107 u32 sr_size; /* size of the shadow ring */
116− u32 group; /* credit return group */
108+ u16 flags; /* flags */
109+ u8 type; /* context type */
110+ u8 sw_index; /* software index number */
111+ u8 hw_context; /* hardware context number */
112+ u8 group; /* credit return group */
113+
117114 /* allocator fields */
118115 spinlock_t alloc_lock ____cacheline_aligned_in_smp;
116+ u32 sr_head; /* shadow ring head */
119117 unsigned long fill; /* official alloc count */
120118 unsigned long alloc_free; /* copy of free (less cache thrash) */
121− u32 sr_head; /* shadow ring head */
119+ u32 fill_wrap; /* tracks fill within ring */
120+ u32 credits; /* number of blocks in context */
121+ /* adding a new field here would make it part of this cacheline */
122+
122123 /* releaser fields */
123124 spinlock_t release_lock ____cacheline_aligned_in_smp;
124− unsigned long free; /* official free count */
125125 u32 sr_tail; /* shadow ring tail */
126+ unsigned long free; /* official free count */
127+ volatile __le64 *hw_free; /* HW free counter */
126128 /* list for PIO waiters */
127129 struct list_head piowait ____cacheline_aligned_in_smp;
128130 spinlock_t credit_ctrl_lock ____cacheline_aligned_in_smp;
129− u64 credit_ctrl; /* cache for credit control */
130131 u32 credit_intr_count; /* count of credit intr users */
131− u32 __percpu *buffers_allocated;/* count of buffers allocated */
132+ u64 credit_ctrl; /* cache for credit control */
132133 wait_queue_head_t halt_wait; /* wait until kernel sees interrupt */
134+ struct work_struct halt_work; /* halted context work queue entry */
133135 };
134136
135137 /* send context flags */
drivers/infiniband/hw/hfi1/pio_copy.c+11−11
@@ -129,8 +129,8 @@ void pio_copy(struct hfi1_devdata *dd, struct pio_buf *pbuf, u64 pbc, · 128 unmodified lines
129129 dest += sizeof(u64);
130130 }
131131
132− dest -= pbuf->size;
133− dend -= pbuf->size;
132+ dest -= pbuf->sc->size;
133+ dend -= pbuf->sc->size;
134134 }
135135
136136 /* write 8-byte non-SOP, non-wrap chunk data */
@@ -361,8 +361,8 @@ void seg_pio_copy_start(struct pio_buf *pbuf, u64 pbc, · 224 unmodified lines
361361 dest += sizeof(u64);
362362 }
363363
364− dest -= pbuf->size;
365− dend -= pbuf->size;
364+ dest -= pbuf->sc->size;
365+ dend -= pbuf->sc->size;
366366 }
367367
368368 /* write 8-byte non-SOP, non-wrap chunk data */
@@ -458,8 +458,8 @@ static void mid_copy_mix(struct pio_buf *pbuf, const void *from, size_t nbytes) · 89 unmodified lines
458458 dest += sizeof(u64);
459459 }
460460
461− dest -= pbuf->size;
462− dend -= pbuf->size;
461+ dest -= pbuf->sc->size;
462+ dend -= pbuf->sc->size;
463463 }
464464
465465 /* write 8-byte non-SOP, non-wrap chunk data */
@@ -492,7 +492,7 @@ static void mid_copy_mix(struct pio_buf *pbuf, const void *from, size_t nbytes) · 26 unmodified lines
492492 */
493493 /* adjust if we have wrapped */
494494 if (dest >= pbuf->end)
495− dest -= pbuf->size;
495+ dest -= pbuf->sc->size;
496496 /* jump to the SOP range if within the first block */
497497 else if (pbuf->qw_written < PIO_BLOCK_QWS)
498498 dest += SOP_DISTANCE;
@@ -584,8 +584,8 @@ static void mid_copy_straight(struct pio_buf *pbuf, · 85 unmodified lines
584584 dest += sizeof(u64);
585585 }
586586
587− dest -= pbuf->size;
588− dend -= pbuf->size;
587+ dest -= pbuf->sc->size;
588+ dend -= pbuf->sc->size;
589589 }
590590
591591 /* write 8-byte non-SOP, non-wrap chunk data */
@@ -666,7 +666,7 @@ void seg_pio_copy_mid(struct pio_buf *pbuf, const void *from, size_t nbytes) · 74 unmodified lines
666666 */
667667 /* adjust if we've wrapped */
668668 if (dest >= pbuf->end)
669− dest -= pbuf->size;
669+ dest -= pbuf->sc->size;
670670 /* jump to SOP range if within the first block */
671671 else if (pbuf->qw_written < PIO_BLOCK_QWS)
672672 dest += SOP_DISTANCE;
@@ -719,7 +719,7 @@ void seg_pio_copy_end(struct pio_buf *pbuf) · 46 unmodified lines
719719 */
720720 /* adjust if we have wrapped */
721721 if (dest >= pbuf->end)
722− dest -= pbuf->size;
722+ dest -= pbuf->sc->size;
723723 /* jump to the SOP range if within the first block */
724724 else if (pbuf->qw_written < PIO_BLOCK_QWS)
725725 dest += SOP_DISTANCE;
drivers/infiniband/hw/hfi1/platform.c+167−26
@@ -49,6 +49,90 @@ · 48 unmodified lines
4949 #include "efivar.h"
5050 #include "eprom.h"
5151
52+static int validate_scratch_checksum(struct hfi1_devdata *dd)
53+{
54+ u64 checksum = 0, temp_scratch = 0;
55+ int i, j, version;
56+
57+ temp_scratch = read_csr(dd, ASIC_CFG_SCRATCH);
58+ version = (temp_scratch & BITMAP_VERSION_SMASK) >> BITMAP_VERSION_SHIFT;
59+
60+ /* Prevent power on default of all zeroes from passing checksum */
61+ if (!version)
62+ return 0;
63+
64+ /*
65+ * ASIC scratch 0 only contains the checksum and bitmap version as
66+ * fields of interest, both of which are handled separately from the
67+ * loop below, so skip it
68+ */
69+ checksum += version;
70+ for (i = 1; i < ASIC_NUM_SCRATCH; i++) {
71+ temp_scratch = read_csr(dd, ASIC_CFG_SCRATCH + (8 * i));
72+ for (j = sizeof(u64); j != 0; j -= 2) {
73+ checksum += (temp_scratch & 0xFFFF);
74+ temp_scratch >>= 16;
75+ }
76+ }
77+
78+ while (checksum >> 16)
79+ checksum = (checksum & CHECKSUM_MASK) + (checksum >> 16);
80+
81+ temp_scratch = read_csr(dd, ASIC_CFG_SCRATCH);
82+ temp_scratch &= CHECKSUM_SMASK;
83+ temp_scratch >>= CHECKSUM_SHIFT;
84+
85+ if (checksum + temp_scratch == 0xFFFF)
86+ return 1;
87+ return 0;
88+}
89+
90+static void save_platform_config_fields(struct hfi1_devdata *dd)
91+{
92+ struct hfi1_pportdata *ppd = dd->pport;
93+ u64 temp_scratch = 0, temp_dest = 0;
94+
95+ temp_scratch = read_csr(dd, ASIC_CFG_SCRATCH_1);
96+
97+ temp_dest = temp_scratch &
98+ (dd->hfi1_id ? PORT1_PORT_TYPE_SMASK :
99+ PORT0_PORT_TYPE_SMASK);
100+ ppd->port_type = temp_dest >>
101+ (dd->hfi1_id ? PORT1_PORT_TYPE_SHIFT :
102+ PORT0_PORT_TYPE_SHIFT);
103+
104+ temp_dest = temp_scratch &
105+ (dd->hfi1_id ? PORT1_LOCAL_ATTEN_SMASK :
106+ PORT0_LOCAL_ATTEN_SMASK);
107+ ppd->local_atten = temp_dest >>
108+ (dd->hfi1_id ? PORT1_LOCAL_ATTEN_SHIFT :
109+ PORT0_LOCAL_ATTEN_SHIFT);
110+
111+ temp_dest = temp_scratch &
112+ (dd->hfi1_id ? PORT1_REMOTE_ATTEN_SMASK :
113+ PORT0_REMOTE_ATTEN_SMASK);
114+ ppd->remote_atten = temp_dest >>
115+ (dd->hfi1_id ? PORT1_REMOTE_ATTEN_SHIFT :
116+ PORT0_REMOTE_ATTEN_SHIFT);
117+
118+ temp_dest = temp_scratch &
119+ (dd->hfi1_id ? PORT1_DEFAULT_ATTEN_SMASK :
120+ PORT0_DEFAULT_ATTEN_SMASK);
121+ ppd->default_atten = temp_dest >>
122+ (dd->hfi1_id ? PORT1_DEFAULT_ATTEN_SHIFT :
123+ PORT0_DEFAULT_ATTEN_SHIFT);
124+
125+ temp_scratch = read_csr(dd, dd->hfi1_id ? ASIC_CFG_SCRATCH_3 :
126+ ASIC_CFG_SCRATCH_2);
127+
128+ ppd->tx_preset_eq = (temp_scratch & TX_EQ_SMASK) >> TX_EQ_SHIFT;
129+ ppd->tx_preset_noeq = (temp_scratch & TX_NO_EQ_SMASK) >> TX_NO_EQ_SHIFT;
130+ ppd->rx_preset = (temp_scratch & RX_SMASK) >> RX_SHIFT;
131+
132+ ppd->max_power_class = (temp_scratch & QSFP_MAX_POWER_SMASK) >>
133+ QSFP_MAX_POWER_SHIFT;
134+}
135+
52136 void get_platform_config(struct hfi1_devdata *dd)
53137 {
54138 int ret = 0;
@@ -56,38 +140,49 @@ void get_platform_config(struct hfi1_devdata *dd) · 1 unmodified line
56140 u8 *temp_platform_config = NULL;
57141 u32 esize;
58142
59− ret = eprom_read_platform_config(dd, (void **)&temp_platform_config,
60− &esize);
61− if (!ret) {
62− /* success */
63− size = esize;
64− goto success;
143+ if (is_integrated(dd)) {
144+ if (validate_scratch_checksum(dd)) {
145+ save_platform_config_fields(dd);
146+ return;
147+ }
148+ dd_dev_err(dd, "%s: Config bitmap corrupted/uninitialized\n",
149+ __func__);
150+ dd_dev_err(dd,
151+ "%s: Please update your BIOS to support active channels\n",
152+ __func__);
153+ } else {
154+ ret = eprom_read_platform_config(dd,
155+ (void **)&temp_platform_config,
156+ &esize);
157+ if (!ret) {
158+ /* success */
159+ dd->platform_config.data = temp_platform_config;
160+ dd->platform_config.size = esize;
161+ return;
162+ }
163+ /* fail, try EFI variable */
164+
165+ ret = read_hfi1_efi_var(dd, "configuration", &size,
166+ (void **)&temp_platform_config);
167+ if (!ret) {
168+ dd->platform_config.data = temp_platform_config;
169+ dd->platform_config.size = size;
170+ return;
171+ }
65172 }
66− /* fail, try EFI variable */
67−
68− ret = read_hfi1_efi_var(dd, "configuration", &size,
69− (void **)&temp_platform_config);
70− if (!ret)
71− goto success;
72−
73− dd_dev_info(dd,
74− "%s: Failed to get platform config from UEFI, falling back to request firmware\n",
75− __func__);
173+ dd_dev_err(dd,
174+ "%s: Failed to get platform config, falling back to sub-optimal default file\n",
175+ __func__);
76176 /* fall back to request firmware */
77177 platform_config_load = 1;
78− return;
79−
80−success:
81− dd->platform_config.data = temp_platform_config;
82− dd->platform_config.size = size;
83178 }
84179
85180 void free_platform_config(struct hfi1_devdata *dd)
86181 {
87182 if (!platform_config_load) {
88183 /*
89− * was loaded from EFI, release memory
90− * allocated by read_efi_var
184+ * was loaded from EFI or the EPROM, release memory
185+ * allocated by read_efi_var/eprom_read_platform_config
91186 */
92187 kfree(dd->platform_config.data);
93188 }
@@ -100,12 +195,16 @@ void free_platform_config(struct hfi1_devdata *dd) · 6 unmodified lines
100195 void get_port_type(struct hfi1_pportdata *ppd)
101196 {
102197 int ret;
198+ u32 temp;
103199
104200 ret = get_platform_config_field(ppd->dd, PLATFORM_CONFIG_PORT_TABLE, 0,
105− PORT_TABLE_PORT_TYPE, &ppd->port_type,
201+ PORT_TABLE_PORT_TYPE, &temp,
106202 4);
107− if (ret)
203+ if (ret) {
108204 ppd->port_type = PORT_TYPE_UNKNOWN;
205+ return;
206+ }
207+ ppd->port_type = temp;
109208 }
110209
111210 int set_qsfp_tx(struct hfi1_pportdata *ppd, int on)
@@ -538,6 +637,38 @@ static void apply_tx_lanes(struct hfi1_pportdata *ppd, u8 field_id, · 426 unmodified lines
538637 }
539638 }
540639
640+/*
641+ * Return a special SerDes setting for low power AOC cables. The power class
642+ * threshold and setting being used were all found by empirical testing.
643+ *
644+ * Summary of the logic:
645+ *
646+ * if (QSFP and QSFP_TYPE == AOC and QSFP_POWER_CLASS < 4)
647+ * return 0xe
648+ * return 0; // leave at default
649+ */
650+static u8 aoc_low_power_setting(struct hfi1_pportdata *ppd)
651+{
652+ u8 *cache = ppd->qsfp_info.cache;
653+ int power_class;
654+
655+ /* QSFP only */
656+ if (ppd->port_type != PORT_TYPE_QSFP)
657+ return 0; /* leave at default */
658+
659+ /* active optical cables only */
660+ switch ((cache[QSFP_MOD_TECH_OFFS] & 0xF0) >> 4) {
661+ case 0x0 ... 0x9: /* fallthrough */
662+ case 0xC: /* fallthrough */
663+ case 0xE:
664+ /* active AOC */
665+ power_class = get_qsfp_power_class(cache[QSFP_MOD_PWR_OFFS]);
666+ if (power_class < QSFP_POWER_CLASS_4)
667+ return 0xe;
668+ }
669+ return 0; /* leave at default */
670+}
671+
541672 static void apply_tunings(
542673 struct hfi1_pportdata *ppd, u32 tx_preset_index,
543674 u8 tuning_method, u32 total_atten, u8 limiting_active)
@@ -606,7 +737,17 @@ static void apply_tunings( · 62 unmodified lines
606737 tx_preset_index, TX_PRESET_TABLE_POSTCUR, &tx_preset, 4);
607738 postcur = tx_preset;
608739
609− config_data = precur | (attn << 8) | (postcur << 16);
740+ /*
741+ * NOTES:
742+ * o The aoc_low_power_setting is applied to all lanes even
743+ * though only lane 0's value is examined by the firmware.
744+ * o A lingering low power setting after a cable swap does
745+ * not occur. On cable unplug the 8051 is reset and
746+ * restarted on cable insert. This resets all settings to
747+ * their default, erasing any previous low power setting.
748+ */
749+ config_data = precur | (attn << 8) | (postcur << 16) |
750+ (aoc_low_power_setting(ppd) << 24);
610751
611752 apply_tx_lanes(ppd, TX_EQ_SETTINGS, config_data,
612753 "Applying TX settings");
drivers/infiniband/hw/hfi1/platform.h+117−10
@@ -168,16 +168,6 @@ struct platform_config_cache { · 167 unmodified lines
168168 struct platform_config_data config_tables[PLATFORM_CONFIG_TABLE_MAX];
169169 };
170170
171−static const u32 platform_config_table_limits[PLATFORM_CONFIG_TABLE_MAX] = {
172− 0,
173− SYSTEM_TABLE_MAX,
174− PORT_TABLE_MAX,
175− RX_PRESET_TABLE_MAX,
176− TX_PRESET_TABLE_MAX,
177− QSFP_ATTEN_TABLE_MAX,
178− VARIABLE_SETTINGS_TABLE_MAX
179−};
180−
181171 /* This section defines default values and encodings for the
182172 * fields defined for each table above
183173 */
@@ -295,6 +285,123 @@ enum link_tuning_encoding { · 111 unmodified lines
295285 OPA_UNKNOWN_TUNING
296286 };
297287
288+/*
289+ * Shifts and masks for the link SI tuning values stuffed into the ASIC scratch
290+ * registers for integrated platforms
291+ */
292+#define PORT0_PORT_TYPE_SHIFT 0
293+#define PORT0_LOCAL_ATTEN_SHIFT 4
294+#define PORT0_REMOTE_ATTEN_SHIFT 10
295+#define PORT0_DEFAULT_ATTEN_SHIFT 32
296+
297+#define PORT1_PORT_TYPE_SHIFT 16
298+#define PORT1_LOCAL_ATTEN_SHIFT 20
299+#define PORT1_REMOTE_ATTEN_SHIFT 26
300+#define PORT1_DEFAULT_ATTEN_SHIFT 40
301+
302+#define PORT0_PORT_TYPE_MASK 0xFUL
303+#define PORT0_LOCAL_ATTEN_MASK 0x3FUL
304+#define PORT0_REMOTE_ATTEN_MASK 0x3FUL
305+#define PORT0_DEFAULT_ATTEN_MASK 0xFFUL
306+
307+#define PORT1_PORT_TYPE_MASK 0xFUL
308+#define PORT1_LOCAL_ATTEN_MASK 0x3FUL
309+#define PORT1_REMOTE_ATTEN_MASK 0x3FUL
310+#define PORT1_DEFAULT_ATTEN_MASK 0xFFUL
311+
312+#define PORT0_PORT_TYPE_SMASK (PORT0_PORT_TYPE_MASK << \
313+ PORT0_PORT_TYPE_SHIFT)
314+#define PORT0_LOCAL_ATTEN_SMASK (PORT0_LOCAL_ATTEN_MASK << \
315+ PORT0_LOCAL_ATTEN_SHIFT)
316+#define PORT0_REMOTE_ATTEN_SMASK (PORT0_REMOTE_ATTEN_MASK << \
317+ PORT0_REMOTE_ATTEN_SHIFT)
318+#define PORT0_DEFAULT_ATTEN_SMASK (PORT0_DEFAULT_ATTEN_MASK << \
319+ PORT0_DEFAULT_ATTEN_SHIFT)
320+
321+#define PORT1_PORT_TYPE_SMASK (PORT1_PORT_TYPE_MASK << \
322+ PORT1_PORT_TYPE_SHIFT)
323+#define PORT1_LOCAL_ATTEN_SMASK (PORT1_LOCAL_ATTEN_MASK << \
324+ PORT1_LOCAL_ATTEN_SHIFT)
325+#define PORT1_REMOTE_ATTEN_SMASK (PORT1_REMOTE_ATTEN_MASK << \
326+ PORT1_REMOTE_ATTEN_SHIFT)
327+#define PORT1_DEFAULT_ATTEN_SMASK (PORT1_DEFAULT_ATTEN_MASK << \
328+ PORT1_DEFAULT_ATTEN_SHIFT)
329+
330+#define QSFP_MAX_POWER_SHIFT 0
331+#define TX_NO_EQ_SHIFT 4
332+#define TX_EQ_SHIFT 25
333+#define RX_SHIFT 46
334+
335+#define QSFP_MAX_POWER_MASK 0xFUL
336+#define TX_NO_EQ_MASK 0x1FFFFFUL
337+#define TX_EQ_MASK 0x1FFFFFUL
338+#define RX_MASK 0xFFFFUL
339+
340+#define QSFP_MAX_POWER_SMASK (QSFP_MAX_POWER_MASK << \
341+ QSFP_MAX_POWER_SHIFT)
342+#define TX_NO_EQ_SMASK (TX_NO_EQ_MASK << TX_NO_EQ_SHIFT)
343+#define TX_EQ_SMASK (TX_EQ_MASK << TX_EQ_SHIFT)
344+#define RX_SMASK (RX_MASK << RX_SHIFT)
345+
346+#define TX_PRECUR_SHIFT 0
347+#define TX_ATTN_SHIFT 4
348+#define QSFP_TX_CDR_APPLY_SHIFT 9
349+#define QSFP_TX_EQ_APPLY_SHIFT 10
350+#define QSFP_TX_CDR_SHIFT 11
351+#define QSFP_TX_EQ_SHIFT 12
352+#define TX_POSTCUR_SHIFT 16
353+
354+#define TX_PRECUR_MASK 0xFUL
355+#define TX_ATTN_MASK 0x1FUL
356+#define QSFP_TX_CDR_APPLY_MASK 0x1UL
357+#define QSFP_TX_EQ_APPLY_MASK 0x1UL
358+#define QSFP_TX_CDR_MASK 0x1UL
359+#define QSFP_TX_EQ_MASK 0xFUL
360+#define TX_POSTCUR_MASK 0x1FUL
361+
362+#define TX_PRECUR_SMASK (TX_PRECUR_MASK << TX_PRECUR_SHIFT)
363+#define TX_ATTN_SMASK (TX_ATTN_MASK << TX_ATTN_SHIFT)
364+#define QSFP_TX_CDR_APPLY_SMASK (QSFP_TX_CDR_APPLY_MASK << \
365+ QSFP_TX_CDR_APPLY_SHIFT)
366+#define QSFP_TX_EQ_APPLY_SMASK (QSFP_TX_EQ_APPLY_MASK << \
367+ QSFP_TX_EQ_APPLY_SHIFT)
368+#define QSFP_TX_CDR_SMASK (QSFP_TX_CDR_MASK << QSFP_TX_CDR_SHIFT)
369+#define QSFP_TX_EQ_SMASK (QSFP_TX_EQ_MASK << QSFP_TX_EQ_SHIFT)
370+#define TX_POSTCUR_SMASK (TX_POSTCUR_MASK << TX_POSTCUR_SHIFT)
371+
372+#define QSFP_RX_CDR_APPLY_SHIFT 0
373+#define QSFP_RX_EMP_APPLY_SHIFT 1
374+#define QSFP_RX_AMP_APPLY_SHIFT 2
375+#define QSFP_RX_CDR_SHIFT 3
376+#define QSFP_RX_EMP_SHIFT 4
377+#define QSFP_RX_AMP_SHIFT 8
378+
379+#define QSFP_RX_CDR_APPLY_MASK 0x1UL
380+#define QSFP_RX_EMP_APPLY_MASK 0x1UL
381+#define QSFP_RX_AMP_APPLY_MASK 0x1UL
382+#define QSFP_RX_CDR_MASK 0x1UL
383+#define QSFP_RX_EMP_MASK 0xFUL
384+#define QSFP_RX_AMP_MASK 0x3UL
385+
386+#define QSFP_RX_CDR_APPLY_SMASK (QSFP_RX_CDR_APPLY_MASK << \
387+ QSFP_RX_CDR_APPLY_SHIFT)
388+#define QSFP_RX_EMP_APPLY_SMASK (QSFP_RX_EMP_APPLY_MASK << \
389+ QSFP_RX_EMP_APPLY_SHIFT)
390+#define QSFP_RX_AMP_APPLY_SMASK (QSFP_RX_AMP_APPLY_MASK << \
391+ QSFP_RX_AMP_APPLY_SHIFT)
392+#define QSFP_RX_CDR_SMASK (QSFP_RX_CDR_MASK << QSFP_RX_CDR_SHIFT)
393+#define QSFP_RX_EMP_SMASK (QSFP_RX_EMP_MASK << QSFP_RX_EMP_SHIFT)
394+#define QSFP_RX_AMP_SMASK (QSFP_RX_AMP_MASK << QSFP_RX_AMP_SHIFT)
395+
396+#define BITMAP_VERSION 1
397+#define BITMAP_VERSION_SHIFT 44
398+#define BITMAP_VERSION_MASK 0xFUL
399+#define BITMAP_VERSION_SMASK (BITMAP_VERSION_MASK << \
400+ BITMAP_VERSION_SHIFT)
401+#define CHECKSUM_SHIFT 48
402+#define CHECKSUM_MASK 0xFFFFUL
403+#define CHECKSUM_SMASK (CHECKSUM_MASK << CHECKSUM_SHIFT)
404+
298405 /* platform.c */
299406 void get_platform_config(struct hfi1_devdata *dd);
300407 void free_platform_config(struct hfi1_devdata *dd);
drivers/infiniband/hw/hfi1/qp.c+8−3
@@ -196,15 +196,18 @@ static void flush_tx_list(struct rvt_qp *qp) · 195 unmodified lines
196196 static void flush_iowait(struct rvt_qp *qp)
197197 {
198198 struct hfi1_qp_priv *priv = qp->priv;
199− struct hfi1_ibdev *dev = to_idev(qp->ibqp.device);
200199 unsigned long flags;
200+ seqlock_t *lock = priv->s_iowait.lock;
201201
202− write_seqlock_irqsave(&dev->iowait_lock, flags);
202+ if (!lock)
203+ return;
204+ write_seqlock_irqsave(lock, flags);
203205 if (!list_empty(&priv->s_iowait.list)) {
204206 list_del_init(&priv->s_iowait.list);
207+ priv->s_iowait.lock = NULL;
205208 rvt_put_qp(qp);
206209 }
207− write_sequnlock_irqrestore(&dev->iowait_lock, flags);
210+ write_sequnlock_irqrestore(lock, flags);
208211 }
209212
210213 static inline int opa_mtu_enum_to_int(int mtu)
@@ -543,6 +546,7 @@ static int iowait_sleep( · 332 unmodified lines
543546 ibp->rvp.n_dmawait++;
544547 qp->s_flags |= RVT_S_WAIT_DMA_DESC;
545548 list_add_tail(&priv->s_iowait.list, &sde->dmawait);
549+ priv->s_iowait.lock = &dev->iowait_lock;
546550 trace_hfi1_qpsleep(qp, RVT_S_WAIT_DMA_DESC);
547551 rvt_get_qp(qp);
548552 }
@@ -964,6 +968,7 @@ void notify_error_qp(struct rvt_qp *qp) · 415 unmodified lines
964968 if (!list_empty(&priv->s_iowait.list) && !(qp->s_flags & RVT_S_BUSY)) {
965969 qp->s_flags &= ~RVT_S_ANY_WAIT_IO;
966970 list_del_init(&priv->s_iowait.list);
971+ priv->s_iowait.lock = NULL;
967972 rvt_put_qp(qp);
968973 }
969974 write_sequnlock(&dev->iowait_lock);
drivers/infiniband/hw/hfi1/rc.c+13−47
@@ -276,7 +276,7 @@ static int make_rc_ack(struct hfi1_ibdev *dev, struct rvt_qp *qp, · 275 unmodified lines
276276 rvt_get_mr(ps->s_txreq->mr);
277277 qp->s_ack_rdma_sge.sge = e->rdma_sge;
278278 qp->s_ack_rdma_sge.num_sge = 1;
279− qp->s_cur_sge = &qp->s_ack_rdma_sge;
279+ ps->s_txreq->ss = &qp->s_ack_rdma_sge;
280280 if (len > pmtu) {
281281 len = pmtu;
282282 qp->s_ack_state = OP(RDMA_READ_RESPONSE_FIRST);
@@ -290,7 +290,7 @@ static int make_rc_ack(struct hfi1_ibdev *dev, struct rvt_qp *qp, · 7 unmodified lines
290290 bth2 = mask_psn(qp->s_ack_rdma_psn++);
291291 } else {
292292 /* COMPARE_SWAP or FETCH_ADD */
293− qp->s_cur_sge = NULL;
293+ ps->s_txreq->ss = NULL;
294294 len = 0;
295295 qp->s_ack_state = OP(ATOMIC_ACKNOWLEDGE);
296296 ohdr->u.at.aeth = hfi1_compute_aeth(qp);
@@ -306,7 +306,7 @@ static int make_rc_ack(struct hfi1_ibdev *dev, struct rvt_qp *qp, · 9 unmodified lines
306306 qp->s_ack_state = OP(RDMA_READ_RESPONSE_MIDDLE);
307307 /* FALLTHROUGH */
308308 case OP(RDMA_READ_RESPONSE_MIDDLE):
309− qp->s_cur_sge = &qp->s_ack_rdma_sge;
309+ ps->s_txreq->ss = &qp->s_ack_rdma_sge;
310310 ps->s_txreq->mr = qp->s_ack_rdma_sge.sge.mr;
311311 if (ps->s_txreq->mr)
312312 rvt_get_mr(ps->s_txreq->mr);
@@ -335,7 +335,7 @@ normal: · 22 unmodified lines
335335 */
336336 qp->s_ack_state = OP(SEND_ONLY);
337337 qp->s_flags &= ~RVT_S_ACK_PENDING;
338− qp->s_cur_sge = NULL;
338+ ps->s_txreq->ss = NULL;
339339 if (qp->s_nak_state)
340340 ohdr->u.aeth =
341341 cpu_to_be32((qp->r_msn & HFI1_MSN_MASK) |
@@ -351,7 +351,7 @@ normal: · 9 unmodified lines
351351 qp->s_rdma_ack_cnt++;
352352 qp->s_hdrwords = hwords;
353353 ps->s_txreq->sde = priv->s_sde;
354− qp->s_cur_size = len;
354+ ps->s_txreq->s_cur_size = len;
355355 hfi1_make_ruc_header(qp, ohdr, bth0, bth2, middle, ps);
356356 /* pbc */
357357 ps->s_txreq->hdr_dwords = qp->s_hdrwords + 2;
@@ -801,8 +801,8 @@ int hfi1_make_rc_req(struct rvt_qp *qp, struct hfi1_pkt_state *ps) · 443 unmodified lines
801801 qp->s_len -= len;
802802 qp->s_hdrwords = hwords;
803803 ps->s_txreq->sde = priv->s_sde;
804− qp->s_cur_sge = ss;
805− qp->s_cur_size = len;
804+ ps->s_txreq->ss = ss;
805+ ps->s_txreq->s_cur_size = len;
806806 hfi1_make_ruc_header(
807807 qp,
808808 ohdr,
@@ -1146,8 +1146,6 @@ void hfi1_rc_send_complete(struct rvt_qp *qp, struct ib_header *hdr) · 337 unmodified lines
11461146 {
11471147 struct ib_other_headers *ohdr;
11481148 struct rvt_swqe *wqe;
1149− struct ib_wc wc;
1150− unsigned i;
11511149 u32 opcode;
11521150 u32 psn;
11531151
@@ -1195,22 +1193,8 @@ void hfi1_rc_send_complete(struct rvt_qp *qp, struct ib_header *hdr) · 41 unmodified lines
11951193 qp->s_last = s_last;
11961194 /* see post_send() */
11971195 barrier();
1198− for (i = 0; i < wqe->wr.num_sge; i++) {
1199− struct rvt_sge *sge = &wqe->sg_list[i];
1200−
1201− rvt_put_mr(sge->mr);
1202− }
1203− /* Post a send completion queue entry if requested. */
1204− if (!(qp->s_flags & RVT_S_SIGNAL_REQ_WR) ||
1205− (wqe->wr.send_flags & IB_SEND_SIGNALED)) {
1206− memset(&wc, 0, sizeof(wc));
1207− wc.wr_id = wqe->wr.wr_id;
1208− wc.status = IB_WC_SUCCESS;
1209− wc.opcode = ib_hfi1_wc_opcode[wqe->wr.opcode];
1210− wc.byte_len = wqe->length;
1211− wc.qp = &qp->ibqp;
1212− rvt_cq_enter(ibcq_to_rvtcq(qp->ibqp.send_cq), &wc, 0);
1213− }
1196+ rvt_put_swqe(wqe);
1197+ rvt_qp_swqe_complete(qp, wqe, IB_WC_SUCCESS);
12141198 }
12151199 /*
12161200 * If we were waiting for sends to complete before re-sending,
@@ -1240,9 +1224,6 @@ static struct rvt_swqe *do_rc_completion(struct rvt_qp *qp, · 23 unmodified lines
12401224 struct rvt_swqe *wqe,
12411225 struct hfi1_ibport *ibp)
12421226 {
1243− struct ib_wc wc;
1244− unsigned i;
1245−
12461227 lockdep_assert_held(&qp->s_lock);
12471228 /*
12481229 * Don't decrement refcount and don't generate a
@@ -1253,28 +1234,14 @@ static struct rvt_swqe *do_rc_completion(struct rvt_qp *qp, · 4 unmodified lines
12531234 cmp_psn(qp->s_sending_psn, qp->s_sending_hpsn) > 0) {
12541235 u32 s_last;
12551236
1256− for (i = 0; i < wqe->wr.num_sge; i++) {
1257− struct rvt_sge *sge = &wqe->sg_list[i];
1258−
1259− rvt_put_mr(sge->mr);
1260− }
1237+ rvt_put_swqe(wqe);
12611238 s_last = qp->s_last;
12621239 if (++s_last >= qp->s_size)
12631240 s_last = 0;
12641241 qp->s_last = s_last;
12651242 /* see post_send() */
12661243 barrier();
1267− /* Post a send completion queue entry if requested. */
1268− if (!(qp->s_flags & RVT_S_SIGNAL_REQ_WR) ||
1269− (wqe->wr.send_flags & IB_SEND_SIGNALED)) {
1270− memset(&wc, 0, sizeof(wc));
1271− wc.wr_id = wqe->wr.wr_id;
1272− wc.status = IB_WC_SUCCESS;
1273− wc.opcode = ib_hfi1_wc_opcode[wqe->wr.opcode];
1274− wc.byte_len = wqe->length;
1275− wc.qp = &qp->ibqp;
1276− rvt_cq_enter(ibcq_to_rvtcq(qp->ibqp.send_cq), &wc, 0);
1277− }
1244+ rvt_qp_swqe_complete(qp, wqe, IB_WC_SUCCESS);
12781245 } else {
12791246 struct hfi1_pportdata *ppd = ppd_from_ibp(ibp);
12801247
@@ -2295,7 +2262,7 @@ send_last: · 1014 unmodified lines
22952262 hfi1_copy_sge(&qp->r_sge, data, tlen, 1, copy_last);
22962263 rvt_put_ss(&qp->r_sge);
22972264 qp->r_msn++;
2298− if (!test_and_clear_bit(RVT_R_WRID_VALID, &qp->r_aflags))
2265+ if (!__test_and_clear_bit(RVT_R_WRID_VALID, &qp->r_aflags))
22992266 break;
23002267 wc.wr_id = qp->r_wr_id;
23012268 wc.status = IB_WC_SUCCESS;
@@ -2410,8 +2377,7 @@ send_last: · 108 unmodified lines
24102377 * Update the next expected PSN. We add 1 later
24112378 * below, so only add the remainder here.
24122379 */
2413− if (len > pmtu)
2414− qp->r_psn += (len - 1) / pmtu;
2380+ qp->r_psn += rvt_div_mtu(qp, len - 1);
24152381 } else {
24162382 e->rdma_sge.mr = NULL;
24172383 e->rdma_sge.vaddr = NULL;
drivers/infiniband/hw/hfi1/ruc.c+7−37
@@ -239,16 +239,6 @@ bail: · 238 unmodified lines
239239 return ret;
240240 }
241241
242−static __be64 get_sguid(struct hfi1_ibport *ibp, unsigned index)
243−{
244− if (!index) {
245− struct hfi1_pportdata *ppd = ppd_from_ibp(ibp);
246−
247− return cpu_to_be64(ppd->guid);
248− }
249− return ibp->guids[index - 1];
250−}
251−
252242 static int gid_ok(union ib_gid *gid, __be64 gid_prefix, __be64 id)
253243 {
254244 return (gid->global.interface_id == id &&
@@ -699,9 +689,9 @@ u32 hfi1_make_grh(struct hfi1_ibport *ibp, struct ib_grh *hdr, · 444 unmodified lines
699689 /* The SGID is 32-bit aligned. */
700690 hdr->sgid.global.subnet_prefix = ibp->rvp.gid_prefix;
701691 hdr->sgid.global.interface_id =
702− grh->sgid_index && grh->sgid_index < ARRAY_SIZE(ibp->guids) ?
703− ibp->guids[grh->sgid_index - 1] :
704− cpu_to_be64(ppd_from_ibp(ibp)->guid);
692+ grh->sgid_index < HFI1_GUIDS_PER_PORT ?
693+ get_sguid(ibp, grh->sgid_index) :
694+ get_sguid(ibp, HFI1_PORT_GUID_INDEX);
705695 hdr->dgid = grh->dgid;
706696
707697 /* GRH header size in 32-bit words. */
@@ -777,8 +767,8 @@ void hfi1_make_ruc_header(struct rvt_qp *qp, struct ib_other_headers *ohdr, · 69 unmodified lines
777767 u32 bth1;
778768
779769 /* Construct the header. */
780− extra_bytes = -qp->s_cur_size & 3;
781− nwords = (qp->s_cur_size + extra_bytes) >> 2;
770+ extra_bytes = -ps->s_txreq->s_cur_size & 3;
771+ nwords = (ps->s_txreq->s_cur_size + extra_bytes) >> 2;
782772 lrh0 = HFI1_LRH_BTH;
783773 if (unlikely(qp->remote_ah_attr.ah_flags & IB_AH_GRH)) {
784774 qp->s_hdrwords += hfi1_make_grh(ibp,
@@ -952,7 +942,6 @@ void hfi1_send_complete(struct rvt_qp *qp, struct rvt_swqe *wqe, · 167 unmodified lines
952942 enum ib_wc_status status)
953943 {
954944 u32 old_last, last;
955− unsigned i;
956945
957946 if (!(ib_rvt_state_ops[qp->state] & RVT_PROCESS_OR_FLUSH_SEND))
958947 return;
@@ -964,32 +953,13 @@ void hfi1_send_complete(struct rvt_qp *qp, struct rvt_swqe *wqe, · 5 unmodified lines
964953 qp->s_last = last;
965954 /* See post_send() */
966955 barrier();
967− for (i = 0; i < wqe->wr.num_sge; i++) {
968− struct rvt_sge *sge = &wqe->sg_list[i];
969−
970− rvt_put_mr(sge->mr);
971− }
956+ rvt_put_swqe(wqe);
972957 if (qp->ibqp.qp_type == IB_QPT_UD ||
973958 qp->ibqp.qp_type == IB_QPT_SMI ||
974959 qp->ibqp.qp_type == IB_QPT_GSI)
975960 atomic_dec(&ibah_to_rvtah(wqe->ud_wr.ah)->refcount);
976961
977− /* See ch. 11.2.4.1 and 10.7.3.1 */
978− if (!(qp->s_flags & RVT_S_SIGNAL_REQ_WR) ||
979− (wqe->wr.send_flags & IB_SEND_SIGNALED) ||
980− status != IB_WC_SUCCESS) {
981− struct ib_wc wc;
982−
983− memset(&wc, 0, sizeof(wc));
984− wc.wr_id = wqe->wr.wr_id;
985− wc.status = status;
986− wc.opcode = ib_hfi1_wc_opcode[wqe->wr.opcode];
987− wc.qp = &qp->ibqp;
988− if (status == IB_WC_SUCCESS)
989− wc.byte_len = wqe->length;
990− rvt_cq_enter(ibcq_to_rvtcq(qp->ibqp.send_cq), &wc,
991− status != IB_WC_SUCCESS);
992− }
962+ rvt_qp_swqe_complete(qp, wqe, status);
993963
994964 if (qp->s_acked == old_last)
995965 qp->s_acked = last;
drivers/infiniband/hw/hfi1/sdma.c+9−9
@@ -375,7 +375,7 @@ static inline void complete_tx(struct sdma_engine *sde, · 374 unmodified lines
375375 sde->head_sn, tx->sn);
376376 sde->head_sn++;
377377 #endif
378− sdma_txclean(sde->dd, tx);
378+ __sdma_txclean(sde->dd, tx);
379379 if (complete)
380380 (*complete)(tx, res);
381381 if (wait && iowait_sdma_dec(wait))
@@ -1643,7 +1643,7 @@ static inline u8 ahg_mode(struct sdma_txreq *tx) · 1261 unmodified lines
16431643 }
16441644
16451645 /**
1646− * sdma_txclean() - clean tx of mappings, descp *kmalloc's
1646+ * __sdma_txclean() - clean tx of mappings, descp *kmalloc's
16471647 * @dd: hfi1_devdata for unmapping
16481648 * @tx: tx request to clean
16491649 *
@@ -1653,7 +1653,7 @@ static inline u8 ahg_mode(struct sdma_txreq *tx) · 3 unmodified lines
16531653 * The code can be called multiple times without issue.
16541654 *
16551655 */
1656−void sdma_txclean(
1656+void __sdma_txclean(
16571657 struct hfi1_devdata *dd,
16581658 struct sdma_txreq *tx)
16591659 {
@@ -3065,7 +3065,7 @@ static int _extend_sdma_tx_descs(struct hfi1_devdata *dd, struct sdma_txreq *tx) · 1405 unmodified lines
30653065 tx->descp[i] = tx->descs[i];
30663066 return 0;
30673067 enomem:
3068− sdma_txclean(dd, tx);
3068+ __sdma_txclean(dd, tx);
30693069 return -ENOMEM;
30703070 }
30713071
@@ -3094,14 +3094,14 @@ int ext_coal_sdma_tx_descs(struct hfi1_devdata *dd, struct sdma_txreq *tx, · 22 unmodified lines
30943094
30953095 rval = _extend_sdma_tx_descs(dd, tx);
30963096 if (rval) {
3097− sdma_txclean(dd, tx);
3097+ __sdma_txclean(dd, tx);
30983098 return rval;
30993099 }
31003100
31013101 /* If coalesce buffer is allocated, copy data into it */
31023102 if (tx->coalesce_buf) {
31033103 if (type == SDMA_MAP_NONE) {
3104− sdma_txclean(dd, tx);
3104+ __sdma_txclean(dd, tx);
31053105 return -EINVAL;
31063106 }
31073107
@@ -3109,7 +3109,7 @@ int ext_coal_sdma_tx_descs(struct hfi1_devdata *dd, struct sdma_txreq *tx, · 1 unmodified line
31093109 kvaddr = kmap(page);
31103110 kvaddr += offset;
31113111 } else if (WARN_ON(!kvaddr)) {
3112− sdma_txclean(dd, tx);
3112+ __sdma_txclean(dd, tx);
31133113 return -EINVAL;
31143114 }
31153115
@@ -3139,7 +3139,7 @@ int ext_coal_sdma_tx_descs(struct hfi1_devdata *dd, struct sdma_txreq *tx, · 23 unmodified lines
31393139 DMA_TO_DEVICE);
31403140
31413141 if (unlikely(dma_mapping_error(&dd->pcidev->dev, addr))) {
3142− sdma_txclean(dd, tx);
3142+ __sdma_txclean(dd, tx);
31433143 return -ENOSPC;
31443144 }
31453145
@@ -3181,7 +3181,7 @@ int _pad_sdma_tx_descs(struct hfi1_devdata *dd, struct sdma_txreq *tx) · 35 unmodified lines
31813181 if ((unlikely(tx->num_desc == tx->desc_limit))) {
31823182 rval = _extend_sdma_tx_descs(dd, tx);
31833183 if (rval) {
3184− sdma_txclean(dd, tx);
3184+ __sdma_txclean(dd, tx);
31853185 return rval;
31863186 }
31873187 }
drivers/infiniband/hw/hfi1/sdma.h+9−3
@@ -667,7 +667,13 @@ int ext_coal_sdma_tx_descs(struct hfi1_devdata *dd, struct sdma_txreq *tx, · 666 unmodified lines
667667 int type, void *kvaddr, struct page *page,
668668 unsigned long offset, u16 len);
669669 int _pad_sdma_tx_descs(struct hfi1_devdata *, struct sdma_txreq *);
670−void sdma_txclean(struct hfi1_devdata *, struct sdma_txreq *);
670+void __sdma_txclean(struct hfi1_devdata *, struct sdma_txreq *);
671+
672+static inline void sdma_txclean(struct hfi1_devdata *dd, struct sdma_txreq *tx)
673+{
674+ if (tx->num_desc)
675+ __sdma_txclean(dd, tx);
676+}
671677
672678 /* helpers used by public routines */
673679 static inline void _sdma_close_tx(struct hfi1_devdata *dd,
@@ -753,7 +759,7 @@ static inline int sdma_txadd_page( · 79 unmodified lines
753759 DMA_TO_DEVICE);
754760
755761 if (unlikely(dma_mapping_error(&dd->pcidev->dev, addr))) {
756− sdma_txclean(dd, tx);
762+ __sdma_txclean(dd, tx);
757763 return -ENOSPC;
758764 }
759765
@@ -834,7 +840,7 @@ static inline int sdma_txadd_kvaddr( · 74 unmodified lines
834840 DMA_TO_DEVICE);
835841
836842 if (unlikely(dma_mapping_error(&dd->pcidev->dev, addr))) {
837− sdma_txclean(dd, tx);
843+ __sdma_txclean(dd, tx);
838844 return -ENOSPC;
839845 }
840846
drivers/infiniband/hw/hfi1/uc.c+2−2
@@ -258,8 +258,8 @@ int hfi1_make_uc_req(struct rvt_qp *qp, struct hfi1_pkt_state *ps) · 257 unmodified lines
258258 qp->s_len -= len;
259259 qp->s_hdrwords = hwords;
260260 ps->s_txreq->sde = priv->s_sde;
261− qp->s_cur_sge = &qp->s_sge;
262− qp->s_cur_size = len;
261+ ps->s_txreq->ss = &qp->s_sge;
262+ ps->s_txreq->s_cur_size = len;
263263 hfi1_make_ruc_header(qp, ohdr, bth0 | (qp->s_state << 24),
264264 mask_psn(qp->s_psn++), middle, ps);
265265 /* pbc */
drivers/infiniband/hw/hfi1/ud.c+2−2
@@ -354,8 +354,8 @@ int hfi1_make_ud_req(struct rvt_qp *qp, struct hfi1_pkt_state *ps) · 353 unmodified lines
354354
355355 /* header size in 32-bit words LRH+BTH+DETH = (8+12+8)/4. */
356356 qp->s_hdrwords = 7;
357− qp->s_cur_size = wqe->length;
358− qp->s_cur_sge = &qp->s_sge;
357+ ps->s_txreq->s_cur_size = wqe->length;
358+ ps->s_txreq->ss = &qp->s_sge;
359359 qp->s_srate = ah_attr->static_rate;
360360 qp->srate_mbps = ib_rate_to_mbps(qp->s_srate);
361361 qp->s_wqe = wqe;
drivers/infiniband/hw/hfi1/user_sdma.c+43−17
@@ -115,6 +115,7 @@ MODULE_PARM_DESC(sdma_comp_size, "Size of User SDMA completion ring. Default: 12 · 114 unmodified lines
115115 #define KDETH_HCRC_LOWER_MASK 0xff
116116
117117 #define AHG_KDETH_INTR_SHIFT 12
118+#define AHG_KDETH_SH_SHIFT 13
118119
119120 #define PBC2LRH(x) ((((x) & 0xfff) << 2) - 4)
120121 #define LRH2PBC(x) ((((x) >> 2) + 1) & 0xfff)
@@ -144,8 +145,9 @@ MODULE_PARM_DESC(sdma_comp_size, "Size of User SDMA completion ring. Default: 12 · 23 unmodified lines
144145 #define KDETH_OM_LARGE 64
145146 #define KDETH_OM_MAX_SIZE (1 << ((KDETH_OM_LARGE / KDETH_OM_SMALL) + 1))
146147
147−/* Last packet in the request */
148−#define TXREQ_FLAGS_REQ_LAST_PKT BIT(0)
148+/* Tx request flag bits */
149+#define TXREQ_FLAGS_REQ_ACK BIT(0) /* Set the ACK bit in the header */
150+#define TXREQ_FLAGS_REQ_DISABLE_SH BIT(1) /* Disable header suppression */
149151
150152 /* SDMA request flag bits */
151153 #define SDMA_REQ_FOR_THREAD 1
@@ -943,8 +945,13 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts) · 791 unmodified lines
943945 tx->busycount = 0;
944946 INIT_LIST_HEAD(&tx->list);
945947
948+ /*
949+ * For the last packet set the ACK request
950+ * and disable header suppression.
951+ */
946952 if (req->seqnum == req->info.npkts - 1)
947− tx->flags |= TXREQ_FLAGS_REQ_LAST_PKT;
953+ tx->flags |= (TXREQ_FLAGS_REQ_ACK |
954+ TXREQ_FLAGS_REQ_DISABLE_SH);
948955
949956 /*
950957 * Calculate the payload size - this is min of the fragment
@@ -963,11 +970,22 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts) · 12 unmodified lines
963970 }
964971
965972 datalen = compute_data_length(req, tx);
973+
974+ /*
975+ * Disable header suppression for the payload <= 8DWS.
976+ * If there is an uncorrectable error in the receive
977+ * data FIFO when the received payload size is less than
978+ * or equal to 8DWS then the RxDmaDataFifoRdUncErr is
979+ * not reported.There is set RHF.EccErr if the header
980+ * is not suppressed.
981+ */
966982 if (!datalen) {
967983 SDMA_DBG(req,
968984 "Request has data but pkt len is 0");
969985 ret = -EFAULT;
970986 goto free_tx;
987+ } else if (datalen <= 32) {
988+ tx->flags |= TXREQ_FLAGS_REQ_DISABLE_SH;
971989 }
972990 }
973991
@@ -990,6 +1008,10 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts) · 16 unmodified lines
9901008 LRH2PBC(lrhlen);
9911009 tx->hdr.pbc[0] = cpu_to_le16(pbclen);
9921010 }
1011+ ret = check_header_template(req, &tx->hdr,
1012+ lrhlen, datalen);
1013+ if (ret)
1014+ goto free_tx;
9931015 ret = sdma_txinit_ahg(&tx->txreq,
9941016 SDMA_TXREQ_F_AHG_COPY,
9951017 sizeof(tx->hdr) + datalen,
@@ -1351,7 +1373,7 @@ static int set_txreq_header(struct user_sdma_request *req, · 355 unmodified lines
13511373 req->seqnum));
13521374
13531375 /* Set ACK request on last packet */
1354− if (unlikely(tx->flags & TXREQ_FLAGS_REQ_LAST_PKT))
1376+ if (unlikely(tx->flags & TXREQ_FLAGS_REQ_ACK))
13551377 hdr->bth[2] |= cpu_to_be32(1UL << 31);
13561378
13571379 /* Set the new offset */
@@ -1384,8 +1406,8 @@ static int set_txreq_header(struct user_sdma_request *req, · 26 unmodified lines
13841406 /* Set KDETH.TID based on value for this TID */
13851407 KDETH_SET(hdr->kdeth.ver_tid_offset, TID,
13861408 EXP_TID_GET(tidval, IDX));
1387− /* Clear KDETH.SH only on the last packet */
1388− if (unlikely(tx->flags & TXREQ_FLAGS_REQ_LAST_PKT))
1409+ /* Clear KDETH.SH when DISABLE_SH flag is set */
1410+ if (unlikely(tx->flags & TXREQ_FLAGS_REQ_DISABLE_SH))
13891411 KDETH_SET(hdr->kdeth.ver_tid_offset, SH, 0);
13901412 /*
13911413 * Set the KDETH.OFFSET and KDETH.OM based on size of
@@ -1429,7 +1451,7 @@ static int set_txreq_header_ahg(struct user_sdma_request *req, · 37 unmodified lines
14291451 /* BTH.PSN and BTH.A */
14301452 val32 = (be32_to_cpu(hdr->bth[2]) + req->seqnum) &
14311453 (HFI1_CAP_IS_KSET(EXTENDED_PSN) ? 0x7fffffff : 0xffffff);
1432− if (unlikely(tx->flags & TXREQ_FLAGS_REQ_LAST_PKT))
1454+ if (unlikely(tx->flags & TXREQ_FLAGS_REQ_ACK))
14331455 val32 |= 1UL << 31;
14341456 AHG_HEADER_SET(req->ahg, diff, 6, 0, 16, cpu_to_be16(val32 >> 16));
14351457 AHG_HEADER_SET(req->ahg, diff, 6, 16, 16, cpu_to_be16(val32 & 0xffff));
@@ -1468,19 +1490,23 @@ static int set_txreq_header_ahg(struct user_sdma_request *req, · 32 unmodified lines
14681490 AHG_HEADER_SET(req->ahg, diff, 7, 0, 16,
14691491 ((!!(req->omfactor - KDETH_OM_SMALL)) << 15 |
14701492 ((req->tidoffset / req->omfactor) & 0x7fff)));
1471− /* KDETH.TIDCtrl, KDETH.TID */
1493+ /* KDETH.TIDCtrl, KDETH.TID, KDETH.Intr, KDETH.SH */
14721494 val = cpu_to_le16(((EXP_TID_GET(tidval, CTRL) & 0x3) << 10) |
1473− (EXP_TID_GET(tidval, IDX) & 0x3ff));
1474− /* Clear KDETH.SH on last packet */
1475− if (unlikely(tx->flags & TXREQ_FLAGS_REQ_LAST_PKT)) {
1476− val |= cpu_to_le16(KDETH_GET(hdr->kdeth.ver_tid_offset,
1477− INTR) <<
1478− AHG_KDETH_INTR_SHIFT);
1479− val &= cpu_to_le16(~(1U << 13));
1480− AHG_HEADER_SET(req->ahg, diff, 7, 16, 14, val);
1495+ (EXP_TID_GET(tidval, IDX) & 0x3ff));
1496+
1497+ if (unlikely(tx->flags & TXREQ_FLAGS_REQ_DISABLE_SH)) {
1498+ val |= cpu_to_le16((KDETH_GET(hdr->kdeth.ver_tid_offset,
1499+ INTR) <<
1500+ AHG_KDETH_INTR_SHIFT));
14811501 } else {
1482− AHG_HEADER_SET(req->ahg, diff, 7, 16, 12, val);
1502+ val |= KDETH_GET(hdr->kdeth.ver_tid_offset, SH) ?
1503+ cpu_to_le16(0x1 << AHG_KDETH_SH_SHIFT) :
1504+ cpu_to_le16((KDETH_GET(hdr->kdeth.ver_tid_offset,
1505+ INTR) <<
1506+ AHG_KDETH_INTR_SHIFT));
14831507 }
1508+
1509+ AHG_HEADER_SET(req->ahg, diff, 7, 16, 14, val);
14841510 }
14851511
14861512 trace_hfi1_sdma_user_header_ahg(pq->dd, pq->ctxt, pq->subctxt,
drivers/infiniband/hw/hfi1/verbs.c+174−35
@@ -296,22 +296,6 @@ static inline int wss_exceeds_threshold(void) · 295 unmodified lines
296296 return atomic_read(&wss.total_count) >= wss.threshold;
297297 }
298298
299−/*
300− * Translate ib_wr_opcode into ib_wc_opcode.
301− */
302−const enum ib_wc_opcode ib_hfi1_wc_opcode[] = {
303− [IB_WR_RDMA_WRITE] = IB_WC_RDMA_WRITE,
304− [IB_WR_RDMA_WRITE_WITH_IMM] = IB_WC_RDMA_WRITE,
305− [IB_WR_SEND] = IB_WC_SEND,
306− [IB_WR_SEND_WITH_IMM] = IB_WC_SEND,
307− [IB_WR_RDMA_READ] = IB_WC_RDMA_READ,
308− [IB_WR_ATOMIC_CMP_AND_SWP] = IB_WC_COMP_SWAP,
309− [IB_WR_ATOMIC_FETCH_AND_ADD] = IB_WC_FETCH_ADD,
310− [IB_WR_SEND_WITH_INV] = IB_WC_SEND,
311− [IB_WR_LOCAL_INV] = IB_WC_LOCAL_INV,
312− [IB_WR_REG_MR] = IB_WC_REG_MR
313−};
314−
315299 /*
316300 * Length of header by opcode, 0 --> not supported
317301 */
@@ -694,6 +678,7 @@ static void mem_timer(unsigned long data) · 376 unmodified lines
694678 qp = iowait_to_qp(wait);
695679 priv = qp->priv;
696680 list_del_init(&priv->s_iowait.list);
681+ priv->s_iowait.lock = NULL;
697682 /* refcount held until actual wake up */
698683 if (!list_empty(list))
699684 mod_timer(&dev->mem_timer, jiffies + 1);
@@ -769,6 +754,7 @@ static int wait_kmem(struct hfi1_ibdev *dev, · 69 unmodified lines
769754 mod_timer(&dev->mem_timer, jiffies + 1);
770755 qp->s_flags |= RVT_S_WAIT_KMEM;
771756 list_add_tail(&priv->s_iowait.list, &dev->memwait);
757+ priv->s_iowait.lock = &dev->iowait_lock;
772758 trace_hfi1_qpsleep(qp, RVT_S_WAIT_KMEM);
773759 rvt_get_qp(qp);
774760 }
@@ -788,10 +774,10 @@ static int wait_kmem(struct hfi1_ibdev *dev, · 13 unmodified lines
788774 */
789775 static noinline int build_verbs_ulp_payload(
790776 struct sdma_engine *sde,
791− struct rvt_sge_state *ss,
792777 u32 length,
793778 struct verbs_txreq *tx)
794779 {
780+ struct rvt_sge_state *ss = tx->ss;
795781 struct rvt_sge *sg_list = ss->sg_list;
796782 struct rvt_sge sge = ss->sge;
797783 u8 num_sge = ss->num_sge;
@@ -835,7 +821,6 @@ bail_txadd: · 37 unmodified lines
835821 /* New API */
836822 static int build_verbs_tx_desc(
837823 struct sdma_engine *sde,
838− struct rvt_sge_state *ss,
839824 u32 length,
840825 struct verbs_txreq *tx,
841826 struct hfi1_ahg_info *ahg_info,
@@ -879,9 +864,9 @@ static int build_verbs_tx_desc( · 37 unmodified lines
879864 goto bail_txadd;
880865 }
881866
882− /* add the ulp payload - if any. ss can be NULL for acks */
883− if (ss)
884− ret = build_verbs_ulp_payload(sde, ss, length, tx);
867+ /* add the ulp payload - if any. tx->ss can be NULL for acks */
868+ if (tx->ss)
869+ ret = build_verbs_ulp_payload(sde, length, tx);
885870 bail_txadd:
886871 return ret;
887872 }
@@ -892,8 +877,7 @@ int hfi1_verbs_send_dma(struct rvt_qp *qp, struct hfi1_pkt_state *ps, · 4 unmodified lines
892877 struct hfi1_qp_priv *priv = qp->priv;
893878 struct hfi1_ahg_info *ahg_info = priv->s_ahg;
894879 u32 hdrwords = qp->s_hdrwords;
895− struct rvt_sge_state *ss = qp->s_cur_sge;
896− u32 len = qp->s_cur_size;
880+ u32 len = ps->s_txreq->s_cur_size;
897881 u32 plen = hdrwords + ((len + 3) >> 2) + 2; /* includes pbc */
898882 struct hfi1_ibdev *dev = ps->dev;
899883 struct hfi1_pportdata *ppd = ps->ppd;
@@ -918,7 +902,7 @@ int hfi1_verbs_send_dma(struct rvt_qp *qp, struct hfi1_pkt_state *ps, · 18 unmodified lines
918902 plen);
919903 }
920904 tx->wqe = qp->s_wqe;
921− ret = build_verbs_tx_desc(tx->sde, ss, len, tx, ahg_info, pbc);
905+ ret = build_verbs_tx_desc(tx->sde, len, tx, ahg_info, pbc);
922906 if (unlikely(ret))
923907 goto bail_build;
924908 }
@@ -980,6 +964,7 @@ static int pio_wait(struct rvt_qp *qp, · 55 unmodified lines
980964 qp->s_flags |= flag;
981965 was_empty = list_empty(&sc->piowait);
982966 list_add_tail(&priv->s_iowait.list, &sc->piowait);
967+ priv->s_iowait.lock = &dev->iowait_lock;
983968 trace_hfi1_qpsleep(qp, RVT_S_WAIT_PIO);
984969 rvt_get_qp(qp);
985970 /* counting: only call wantpiobuf_intr if first user */
@@ -1008,8 +993,8 @@ int hfi1_verbs_send_pio(struct rvt_qp *qp, struct hfi1_pkt_state *ps, · 22 unmodified lines
1008993 {
1009994 struct hfi1_qp_priv *priv = qp->priv;
1010995 u32 hdrwords = qp->s_hdrwords;
1011− struct rvt_sge_state *ss = qp->s_cur_sge;
1012− u32 len = qp->s_cur_size;
996+ struct rvt_sge_state *ss = ps->s_txreq->ss;
997+ u32 len = ps->s_txreq->s_cur_size;
1013998 u32 dwords = (len + 3) >> 2;
1014999 u32 plen = hdrwords + dwords + 2; /* includes pbc */
10151000 struct hfi1_pportdata *ppd = ps->ppd;
@@ -1237,7 +1222,7 @@ static inline send_routine get_send_routine(struct rvt_qp *qp, · 221 unmodified lines
12371222 u8 op = get_opcode(h);
12381223
12391224 if (piothreshold &&
1240− qp->s_cur_size <= min(piothreshold, qp->pmtu) &&
1225+ tx->s_cur_size <= min(piothreshold, qp->pmtu) &&
12411226 (BIT(op & OPMASK) & pio_opmask[op >> 5]) &&
12421227 iowait_sdma_pending(&priv->s_iowait) == 0 &&
12431228 !sdma_txreq_built(&tx->txreq))
@@ -1483,15 +1468,11 @@ static int hfi1_get_guid_be(struct rvt_dev_info *rdi, struct rvt_ibport *rvp, · 239 unmodified lines
14831468 int guid_index, __be64 *guid)
14841469 {
14851470 struct hfi1_ibport *ibp = container_of(rvp, struct hfi1_ibport, rvp);
1486− struct hfi1_pportdata *ppd = ppd_from_ibp(ibp);
14871471
1488− if (guid_index == 0)
1489− *guid = cpu_to_be64(ppd->guid);
1490− else if (guid_index < HFI1_GUIDS_PER_PORT)
1491− *guid = ibp->guids[guid_index - 1];
1492− else
1472+ if (guid_index >= HFI1_GUIDS_PER_PORT)
14931473 return -EINVAL;
14941474
1475+ *guid = get_sguid(ibp, guid_index);
14951476 return 0;
14961477 }
14971478
@@ -1610,6 +1591,154 @@ static void hfi1_get_dev_fw_str(struct ib_device *ibdev, char *str, · 112 unmodified lines
16101591 dc8051_ver_min(ver));
16111592 }
16121593
1594+static const char * const driver_cntr_names[] = {
1595+ /* must be element 0*/
1596+ "DRIVER_KernIntr",
1597+ "DRIVER_ErrorIntr",
1598+ "DRIVER_Tx_Errs",
1599+ "DRIVER_Rcv_Errs",
1600+ "DRIVER_HW_Errs",
1601+ "DRIVER_NoPIOBufs",
1602+ "DRIVER_CtxtsOpen",
1603+ "DRIVER_RcvLen_Errs",
1604+ "DRIVER_EgrBufFull",
1605+ "DRIVER_EgrHdrFull"
1606+};
1607+
1608+static const char **dev_cntr_names;
1609+static const char **port_cntr_names;
1610+static int num_driver_cntrs = ARRAY_SIZE(driver_cntr_names);
1611+static int num_dev_cntrs;
1612+static int num_port_cntrs;
1613+static int cntr_names_initialized;
1614+
1615+/*
1616+ * Convert a list of names separated by '\n' into an array of NULL terminated
1617+ * strings. Optionally some entries can be reserved in the array to hold extra
1618+ * external strings.
1619+ */
1620+static int init_cntr_names(const char *names_in,
1621+ const int names_len,
1622+ int num_extra_names,
1623+ int *num_cntrs,
1624+ const char ***cntr_names)
1625+{
1626+ char *names_out, *p, **q;
1627+ int i, n;
1628+
1629+ n = 0;
1630+ for (i = 0; i < names_len; i++)
1631+ if (names_in[i] == '\n')
1632+ n++;
1633+
1634+ names_out = kmalloc((n + num_extra_names) * sizeof(char *) + names_len,
1635+ GFP_KERNEL);
1636+ if (!names_out) {
1637+ *num_cntrs = 0;
1638+ *cntr_names = NULL;
1639+ return -ENOMEM;
1640+ }
1641+
1642+ p = names_out + (n + num_extra_names) * sizeof(char *);
1643+ memcpy(p, names_in, names_len);
1644+
1645+ q = (char **)names_out;
1646+ for (i = 0; i < n; i++) {
1647+ q[i] = p;
1648+ p = strchr(p, '\n');
1649+ *p++ = '\0';
1650+ }
1651+
1652+ *num_cntrs = n;
1653+ *cntr_names = (const char **)names_out;
1654+ return 0;
1655+}
1656+
1657+static struct rdma_hw_stats *alloc_hw_stats(struct ib_device *ibdev,
1658+ u8 port_num)
1659+{
1660+ int i, err;
1661+
1662+ if (!cntr_names_initialized) {
1663+ struct hfi1_devdata *dd = dd_from_ibdev(ibdev);
1664+
1665+ err = init_cntr_names(dd->cntrnames,
1666+ dd->cntrnameslen,
1667+ num_driver_cntrs,
1668+ &num_dev_cntrs,
1669+ &dev_cntr_names);
1670+ if (err)
1671+ return NULL;
1672+
1673+ for (i = 0; i < num_driver_cntrs; i++)
1674+ dev_cntr_names[num_dev_cntrs + i] =
1675+ driver_cntr_names[i];
1676+
1677+ err = init_cntr_names(dd->portcntrnames,
1678+ dd->portcntrnameslen,
1679+ 0,
1680+ &num_port_cntrs,
1681+ &port_cntr_names);
1682+ if (err) {
1683+ kfree(dev_cntr_names);
1684+ dev_cntr_names = NULL;
1685+ return NULL;
1686+ }
1687+ cntr_names_initialized = 1;
1688+ }
1689+
1690+ if (!port_num)
1691+ return rdma_alloc_hw_stats_struct(
1692+ dev_cntr_names,
1693+ num_dev_cntrs + num_driver_cntrs,
1694+ RDMA_HW_STATS_DEFAULT_LIFESPAN);
1695+ else
1696+ return rdma_alloc_hw_stats_struct(
1697+ port_cntr_names,
1698+ num_port_cntrs,
1699+ RDMA_HW_STATS_DEFAULT_LIFESPAN);
1700+}
1701+
1702+static u64 hfi1_sps_ints(void)
1703+{
1704+ unsigned long flags;
1705+ struct hfi1_devdata *dd;
1706+ u64 sps_ints = 0;
1707+
1708+ spin_lock_irqsave(&hfi1_devs_lock, flags);
1709+ list_for_each_entry(dd, &hfi1_dev_list, list) {
1710+ sps_ints += get_all_cpu_total(dd->int_counter);
1711+ }
1712+ spin_unlock_irqrestore(&hfi1_devs_lock, flags);
1713+ return sps_ints;
1714+}
1715+
1716+static int get_hw_stats(struct ib_device *ibdev, struct rdma_hw_stats *stats,
1717+ u8 port, int index)
1718+{
1719+ u64 *values;
1720+ int count;
1721+
1722+ if (!port) {
1723+ u64 *stats = (u64 *)&hfi1_stats;
1724+ int i;
1725+
1726+ hfi1_read_cntrs(dd_from_ibdev(ibdev), NULL, &values);
1727+ values[num_dev_cntrs] = hfi1_sps_ints();
1728+ for (i = 1; i < num_driver_cntrs; i++)
1729+ values[num_dev_cntrs + i] = stats[i];
1730+ count = num_dev_cntrs + num_driver_cntrs;
1731+ } else {
1732+ struct hfi1_ibport *ibp = to_iport(ibdev, port);
1733+
1734+ hfi1_read_portcntrs(ppd_from_ibp(ibp), NULL, &values);
1735+ count = num_port_cntrs;
1736+ }
1737+
1738+ memcpy(stats->value, values, count * sizeof(u64));
1739+ return count;
1740+}
1741+
16131742 /**
16141743 * hfi1_register_ib_device - register our device with the infiniband core
16151744 * @dd: the device data structure
@@ -1620,6 +1749,7 @@ int hfi1_register_ib_device(struct hfi1_devdata *dd) · 4 unmodified lines
16201749 struct hfi1_ibdev *dev = &dd->verbs_dev;
16211750 struct ib_device *ibdev = &dev->rdi.ibdev;
16221751 struct hfi1_pportdata *ppd = dd->pport;
1752+ struct hfi1_ibport *ibp = &ppd->ibport_data;
16231753 unsigned i;
16241754 int ret;
16251755 size_t lcpysz = IB_DEVICE_NAME_MAX;
@@ -1632,6 +1762,7 @@ int hfi1_register_ib_device(struct hfi1_devdata *dd) · 6 unmodified lines
16321762 setup_timer(&dev->mem_timer, mem_timer, (unsigned long)dev);
16331763
16341764 seqlock_init(&dev->iowait_lock);
1765+ seqlock_init(&dev->txwait_lock);
16351766 INIT_LIST_HEAD(&dev->txwait);
16361767 INIT_LIST_HEAD(&dev->memwait);
16371768
@@ -1639,20 +1770,24 @@ int hfi1_register_ib_device(struct hfi1_devdata *dd) · 1 unmodified line
16391770 if (ret)
16401771 goto err_verbs_txreq;
16411772
1773+ /* Use first-port GUID as node guid */
1774+ ibdev->node_guid = get_sguid(ibp, HFI1_PORT_GUID_INDEX);
1775+
16421776 /*
16431777 * The system image GUID is supposed to be the same for all
16441778 * HFIs in a single system but since there can be other
16451779 * device types in the system, we can't be sure this is unique.
16461780 */
16471781 if (!ib_hfi1_sys_image_guid)
1648− ib_hfi1_sys_image_guid = cpu_to_be64(ppd->guid);
1782+ ib_hfi1_sys_image_guid = ibdev->node_guid;
16491783 lcpysz = strlcpy(ibdev->name, class_name(), lcpysz);
16501784 strlcpy(ibdev->name + lcpysz, "_%d", IB_DEVICE_NAME_MAX - lcpysz);
16511785 ibdev->owner = THIS_MODULE;
1652− ibdev->node_guid = cpu_to_be64(ppd->guid);
16531786 ibdev->phys_port_cnt = dd->num_pports;
16541787 ibdev->dma_device = &dd->pcidev->dev;
16551788 ibdev->modify_device = modify_device;
1789+ ibdev->alloc_hw_stats = alloc_hw_stats;
1790+ ibdev->get_hw_stats = get_hw_stats;
16561791
16571792 /* keep process mad in the driver */
16581793 ibdev->process_mad = hfi1_process_mad;
@@ -1767,6 +1902,10 @@ void hfi1_unregister_ib_device(struct hfi1_devdata *dd) · 108 unmodified lines
17671902
17681903 del_timer_sync(&dev->mem_timer);
17691904 verbs_txreq_exit(dev);
1905+
1906+ kfree(dev_cntr_names);
1907+ kfree(port_cntr_names);
1908+ cntr_names_initialized = 0;
17701909 }
17711910
17721911 void hfi1_cnp_rcv(struct hfi1_packet *packet)
drivers/infiniband/hw/hfi1/verbs.h+7−9
@@ -73,7 +73,6 @@ struct hfi1_packet; · 72 unmodified lines
7373 #include "iowait.h"
7474
7575 #define HFI1_MAX_RDMA_ATOMIC 16
76−#define HFI1_GUIDS_PER_PORT 5
7776
7877 /*
7978 * Increment this value if any changes that break userspace ABI
@@ -169,8 +168,6 @@ struct hfi1_ibport { · 89 unmodified lines
169168 struct rvt_qp __rcu *qp[2];
170169 struct rvt_ibport rvp;
171170
172− __be64 guids[HFI1_GUIDS_PER_PORT - 1]; /* writable GUIDs */
173−
174171 /* the first 16 entries are sl_to_vl for !OPA */
175172 u8 sl_to_sc[32];
176173 u8 sc_to_sl[32];
@@ -180,18 +177,19 @@ struct hfi1_ibdev { · 3 unmodified lines
180177 struct rvt_dev_info rdi; /* Must be first */
181178
182179 /* QP numbers are shared by all IB ports */
183− /* protect wait lists */
184− seqlock_t iowait_lock;
180+ /* protect txwait list */
181+ seqlock_t txwait_lock ____cacheline_aligned_in_smp;
185182 struct list_head txwait; /* list for wait verbs_txreq */
186183 struct list_head memwait; /* list for wait kernel memory */
187− struct list_head txreq_free;
188184 struct kmem_cache *verbs_txreq_cache;
189− struct timer_list mem_timer;
185+ u64 n_txwait;
186+ u64 n_kmem_wait;
190187
188+ /* protect iowait lists */
189+ seqlock_t iowait_lock ____cacheline_aligned_in_smp;
191190 u64 n_piowait;
192191 u64 n_piodrain;
193− u64 n_txwait;
194− u64 n_kmem_wait;
192+ struct timer_list mem_timer;
195193
196194 #ifdef CONFIG_DEBUG_FS
197195 /* per HFI debugfs */
drivers/infiniband/hw/hfi1/verbs_txreq.c+7−6
@@ -72,22 +72,22 @@ void hfi1_put_txreq(struct verbs_txreq *tx) · 71 unmodified lines
7272 kmem_cache_free(dev->verbs_txreq_cache, tx);
7373
7474 do {
75− seq = read_seqbegin(&dev->iowait_lock);
75+ seq = read_seqbegin(&dev->txwait_lock);
7676 if (!list_empty(&dev->txwait)) {
7777 struct iowait *wait;
7878
79− write_seqlock_irqsave(&dev->iowait_lock, flags);
79+ write_seqlock_irqsave(&dev->txwait_lock, flags);
8080 wait = list_first_entry(&dev->txwait, struct iowait,
8181 list);
8282 qp = iowait_to_qp(wait);
8383 priv = qp->priv;
8484 list_del_init(&priv->s_iowait.list);
8585 /* refcount held until actual wake up */
86− write_sequnlock_irqrestore(&dev->iowait_lock, flags);
86+ write_sequnlock_irqrestore(&dev->txwait_lock, flags);
8787 hfi1_qp_wakeup(qp, RVT_S_WAIT_TX);
8888 break;
8989 }
90− } while (read_seqretry(&dev->iowait_lock, seq));
90+ } while (read_seqretry(&dev->txwait_lock, seq));
9191 }
9292
9393 struct verbs_txreq *__get_txreq(struct hfi1_ibdev *dev,
@@ -96,7 +96,7 @@ struct verbs_txreq *__get_txreq(struct hfi1_ibdev *dev, · 2 unmodified lines
9696 {
9797 struct verbs_txreq *tx = ERR_PTR(-EBUSY);
9898
99− write_seqlock(&dev->iowait_lock);
99+ write_seqlock(&dev->txwait_lock);
100100 if (ib_rvt_state_ops[qp->state] & RVT_PROCESS_RECV_OK) {
101101 struct hfi1_qp_priv *priv;
102102
@@ -108,13 +108,14 @@ struct verbs_txreq *__get_txreq(struct hfi1_ibdev *dev, · 5 unmodified lines
108108 dev->n_txwait++;
109109 qp->s_flags |= RVT_S_WAIT_TX;
110110 list_add_tail(&priv->s_iowait.list, &dev->txwait);
111+ priv->s_iowait.lock = &dev->txwait_lock;
111112 trace_hfi1_qpsleep(qp, RVT_S_WAIT_TX);
112113 rvt_get_qp(qp);
113114 }
114115 qp->s_flags &= ~RVT_S_BUSY;
115116 }
116117 out:
117− write_sequnlock(&dev->iowait_lock);
118+ write_sequnlock(&dev->txwait_lock);
118119 return tx;
119120 }
120121
drivers/infiniband/hw/hfi1/verbs_txreq.h+1−0
@@ -65,6 +65,7 @@ struct verbs_txreq { · 64 unmodified lines
6565 struct sdma_engine *sde;
6666 struct send_context *psc;
6767 u16 hdr_dwords;
68+ u16 s_cur_size;
6869 };
6970
7071 struct hfi1_ibdev;
drivers/infiniband/hw/hns/hns_roce_ah.c+2−1
@@ -39,7 +39,8 @@ · 38 unmodified lines
3939 #define HNS_ROCE_VLAN_SL_BIT_MASK 7
4040 #define HNS_ROCE_VLAN_SL_SHIFT 13
4141
42−struct ib_ah *hns_roce_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *ah_attr)
42+struct ib_ah *hns_roce_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *ah_attr,
43+ struct ib_udata *udata)
4344 {
4445 struct hns_roce_dev *hr_dev = to_hr_dev(ibpd->device);
4546 struct device *dev = &hr_dev->pdev->dev;
drivers/infiniband/hw/hns/hns_roce_alloc.c+7−4
@@ -61,9 +61,10 @@ int hns_roce_bitmap_alloc(struct hns_roce_bitmap *bitmap, unsigned long *obj) · 60 unmodified lines
6161 return ret;
6262 }
6363
64−void hns_roce_bitmap_free(struct hns_roce_bitmap *bitmap, unsigned long obj)
64+void hns_roce_bitmap_free(struct hns_roce_bitmap *bitmap, unsigned long obj,
65+ int rr)
6566 {
66− hns_roce_bitmap_free_range(bitmap, obj, 1);
67+ hns_roce_bitmap_free_range(bitmap, obj, 1, rr);
6768 }
6869
6970 int hns_roce_bitmap_alloc_range(struct hns_roce_bitmap *bitmap, int cnt,
@@ -106,7 +107,8 @@ int hns_roce_bitmap_alloc_range(struct hns_roce_bitmap *bitmap, int cnt, · 36 unmodified lines
106107 }
107108
108109 void hns_roce_bitmap_free_range(struct hns_roce_bitmap *bitmap,
109− unsigned long obj, int cnt)
110+ unsigned long obj, int cnt,
111+ int rr)
110112 {
111113 int i;
112114
@@ -116,7 +118,8 @@ void hns_roce_bitmap_free_range(struct hns_roce_bitmap *bitmap, · 3 unmodified lines
116118 for (i = 0; i < cnt; i++)
117119 clear_bit(obj + i, bitmap->table);
118120
119− bitmap->last = min(bitmap->last, obj);
121+ if (!rr)
122+ bitmap->last = min(bitmap->last, obj);
120123 bitmap->top = (bitmap->top + bitmap->max + bitmap->reserved_top)
121124 & bitmap->mask;
122125 spin_unlock(&bitmap->lock);
drivers/infiniband/hw/hns/hns_roce_cmd.c+4−4
@@ -216,10 +216,10 @@ static int __hns_roce_cmd_mbox_wait(struct hns_roce_dev *hr_dev, u64 in_param, · 215 unmodified lines
216216 goto out;
217217
218218 /*
219− * It is timeout when wait_for_completion_timeout return 0
220− * The return value is the time limit set in advance
221− * how many seconds showing
222− */
219+ * It is timeout when wait_for_completion_timeout return 0
220+ * The return value is the time limit set in advance
221+ * how many seconds showing
222+ */
223223 if (!wait_for_completion_timeout(&context->done,
224224 msecs_to_jiffies(timeout))) {
225225 dev_err(dev, "[cmd]wait_for_completion_timeout timeout\n");
drivers/infiniband/hw/hns/hns_roce_cmd.h+1−11
@@ -34,6 +34,7 @@ · 33 unmodified lines
3434 #define _HNS_ROCE_CMD_H
3535
3636 #define HNS_ROCE_MAILBOX_SIZE 4096
37+#define HNS_ROCE_CMD_TIMEOUT_MSECS 10000
3738
3839 enum {
3940 /* TPT commands */
@@ -57,17 +58,6 @@ enum { · 17 unmodified lines
5758 HNS_ROCE_CMD_QUERY_QP = 0x22,
5859 };
5960
60−enum {
61− HNS_ROCE_CMD_TIME_CLASS_A = 10000,
62− HNS_ROCE_CMD_TIME_CLASS_B = 10000,
63− HNS_ROCE_CMD_TIME_CLASS_C = 10000,
64−};
65−
66−struct hns_roce_cmd_mailbox {
67− void *buf;
68− dma_addr_t dma;
69−};
70−
7161 int hns_roce_cmd_mbox(struct hns_roce_dev *hr_dev, u64 in_param, u64 out_param,
7262 unsigned long in_modifier, u8 op_modifier, u16 op,
7363 unsigned long timeout);
drivers/infiniband/hw/hns/hns_roce_common.h+42−2
@@ -57,6 +57,32 @@ · 56 unmodified lines
5757 #define roce_set_bit(origin, shift, val) \
5858 roce_set_field((origin), (1ul << (shift)), (shift), (val))
5959
60+/*
61+ * roce_hw_index_cmp_lt - Compare two hardware index values in hisilicon
62+ * SOC, check if a is less than b.
63+ * @a: hardware index value
64+ * @b: hardware index value
65+ * @bits: the number of bits of a and b, range: 0~31.
66+ *
67+ * Hardware index increases continuously till max value, and then restart
68+ * from zero, again and again. Because the bits of reg field is often
69+ * limited, the reg field can only hold the low bits of the hardware index
70+ * in hisilicon SOC.
71+ * In some scenes we need to compare two values(a,b) getted from two reg
72+ * fields in this driver, for example:
73+ * If a equals 0xfffe, b equals 0x1 and bits equals 16, we think b has
74+ * incresed from 0xffff to 0x1 and a is less than b.
75+ * If a equals 0xfffe, b equals 0x0xf001 and bits equals 16, we think a
76+ * is bigger than b.
77+ *
78+ * Return true on a less than b, otherwise false.
79+ */
80+#define roce_hw_index_mask(bits) ((1ul << (bits)) - 1)
81+#define roce_hw_index_shift(bits) (32 - (bits))
82+#define roce_hw_index_cmp_lt(a, b, bits) \
83+ ((int)((((a) - (b)) & roce_hw_index_mask(bits)) << \
84+ roce_hw_index_shift(bits)) < 0)
85+
6086 #define ROCEE_GLB_CFG_ROCEE_DB_SQ_MODE_S 3
6187 #define ROCEE_GLB_CFG_ROCEE_DB_OTH_MODE_S 4
6288
@@ -245,16 +271,26 @@ · 182 unmodified lines
245271 #define ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M \
246272 (((1UL << 28) - 1) << ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S)
247273
274+#define ROCEE_SDB_PTR_CMP_BITS 28
275+
248276 #define ROCEE_SDB_INV_CNT_SDB_INV_CNT_S 0
249277 #define ROCEE_SDB_INV_CNT_SDB_INV_CNT_M \
250278 (((1UL << 16) - 1) << ROCEE_SDB_INV_CNT_SDB_INV_CNT_S)
251279
280+#define ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_S 0
281+#define ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_M \
282+ (((1UL << 16) - 1) << ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_S)
283+
284+#define ROCEE_SDB_CNT_CMP_BITS 16
285+
286+#define ROCEE_TSP_BP_ST_QH_FIFO_ENTRY_S 20
287+
288+#define ROCEE_CNT_CLR_CE_CNT_CLR_CE_S 0
289+
252290 /*************ROCEE_REG DEFINITION****************/
253291 #define ROCEE_VENDOR_ID_REG 0x0
254292 #define ROCEE_VENDOR_PART_ID_REG 0x4
255293
256−#define ROCEE_HW_VERSION_REG 0x8
257−
258294 #define ROCEE_SYS_IMAGE_GUID_L_REG 0xC
259295 #define ROCEE_SYS_IMAGE_GUID_H_REG 0x10
260296
@@ -318,7 +354,11 @@ · 57 unmodified lines
318354
319355 #define ROCEE_SDB_ISSUE_PTR_REG 0x758
320356 #define ROCEE_SDB_SEND_PTR_REG 0x75C
357+#define ROCEE_CAEP_CQE_WCMD_EMPTY 0x850
358+#define ROCEE_SCAEP_WR_CQE_CNT 0x8D0
321359 #define ROCEE_SDB_INV_CNT_REG 0x9A4
360+#define ROCEE_SDB_RETRY_CNT_REG 0x9AC
361+#define ROCEE_TSP_BP_ST_REG 0x9EC
322362 #define ROCEE_ECC_UCERR_ALM0_REG 0xB34
323363 #define ROCEE_ECC_CERR_ALM0_REG 0xB40
324364
drivers/infiniband/hw/hns/hns_roce_cq.c+30−16
@@ -35,7 +35,7 @@ · 34 unmodified lines
3535 #include "hns_roce_device.h"
3636 #include "hns_roce_cmd.h"
3737 #include "hns_roce_hem.h"
38−#include "hns_roce_user.h"
38+#include <rdma/hns-abi.h>
3939 #include "hns_roce_common.h"
4040
4141 static void hns_roce_ib_cq_comp(struct hns_roce_cq *hr_cq)
@@ -77,7 +77,7 @@ static int hns_roce_sw2hw_cq(struct hns_roce_dev *dev, · 35 unmodified lines
7777 unsigned long cq_num)
7878 {
7979 return hns_roce_cmd_mbox(dev, mailbox->dma, 0, cq_num, 0,
80− HNS_ROCE_CMD_SW2HW_CQ, HNS_ROCE_CMD_TIME_CLASS_A);
80+ HNS_ROCE_CMD_SW2HW_CQ, HNS_ROCE_CMD_TIMEOUT_MSECS);
8181 }
8282
8383 static int hns_roce_cq_alloc(struct hns_roce_dev *hr_dev, int nent,
@@ -166,7 +166,7 @@ err_put: · 82 unmodified lines
166166 hns_roce_table_put(hr_dev, &cq_table->table, hr_cq->cqn);
167167
168168 err_out:
169− hns_roce_bitmap_free(&cq_table->bitmap, hr_cq->cqn);
169+ hns_roce_bitmap_free(&cq_table->bitmap, hr_cq->cqn, BITMAP_NO_RR);
170170 return ret;
171171 }
172172
@@ -176,11 +176,10 @@ static int hns_roce_hw2sw_cq(struct hns_roce_dev *dev, · 3 unmodified lines
176176 {
177177 return hns_roce_cmd_mbox(dev, 0, mailbox ? mailbox->dma : 0, cq_num,
178178 mailbox ? 0 : 1, HNS_ROCE_CMD_HW2SW_CQ,
179− HNS_ROCE_CMD_TIME_CLASS_A);
179+ HNS_ROCE_CMD_TIMEOUT_MSECS);
180180 }
181181
182−static void hns_roce_free_cq(struct hns_roce_dev *hr_dev,
183− struct hns_roce_cq *hr_cq)
182+void hns_roce_free_cq(struct hns_roce_dev *hr_dev, struct hns_roce_cq *hr_cq)
184183 {
185184 struct hns_roce_cq_table *cq_table = &hr_dev->cq_table;
186185 struct device *dev = &hr_dev->pdev->dev;
@@ -204,7 +203,7 @@ static void hns_roce_free_cq(struct hns_roce_dev *hr_dev, · 17 unmodified lines
204203 spin_unlock_irq(&cq_table->lock);
205204
206205 hns_roce_table_put(hr_dev, &cq_table->table, hr_cq->cqn);
207− hns_roce_bitmap_free(&cq_table->bitmap, hr_cq->cqn);
206+ hns_roce_bitmap_free(&cq_table->bitmap, hr_cq->cqn, BITMAP_NO_RR);
208207 }
209208
210209 static int hns_roce_ib_get_cq_umem(struct hns_roce_dev *hr_dev,
@@ -349,6 +348,15 @@ struct ib_cq *hns_roce_ib_create_cq(struct ib_device *ib_dev, · 138 unmodified lines
349348 goto err_mtt;
350349 }
351350
351+ /*
352+ * For the QP created by kernel space, tptr value should be initialized
353+ * to zero; For the QP created by user space, it will cause synchronous
354+ * problems if tptr is set to zero here, so we initialze it in user
355+ * space.
356+ */
357+ if (!context)
358+ *hr_cq->tptr_addr = 0;
359+
352360 /* Get created cq handler and carry out event */
353361 hr_cq->comp = hns_roce_ib_cq_comp;
354362 hr_cq->event = hns_roce_ib_cq_event;
@@ -383,19 +391,25 @@ int hns_roce_ib_destroy_cq(struct ib_cq *ib_cq) · 28 unmodified lines
383391 {
384392 struct hns_roce_dev *hr_dev = to_hr_dev(ib_cq->device);
385393 struct hns_roce_cq *hr_cq = to_hr_cq(ib_cq);
394+ int ret = 0;
386395
387− hns_roce_free_cq(hr_dev, hr_cq);
388− hns_roce_mtt_cleanup(hr_dev, &hr_cq->hr_buf.hr_mtt);
396+ if (hr_dev->hw->destroy_cq) {
397+ ret = hr_dev->hw->destroy_cq(ib_cq);
398+ } else {
399+ hns_roce_free_cq(hr_dev, hr_cq);
400+ hns_roce_mtt_cleanup(hr_dev, &hr_cq->hr_buf.hr_mtt);
389401
390− if (ib_cq->uobject)
391− ib_umem_release(hr_cq->umem);
392− else
393− /* Free the buff of stored cq */
394− hns_roce_ib_free_cq_buf(hr_dev, &hr_cq->hr_buf, ib_cq->cqe);
402+ if (ib_cq->uobject)
403+ ib_umem_release(hr_cq->umem);
404+ else
405+ /* Free the buff of stored cq */
406+ hns_roce_ib_free_cq_buf(hr_dev, &hr_cq->hr_buf,
407+ ib_cq->cqe);
395408
396− kfree(hr_cq);
409+ kfree(hr_cq);
410+ }
397411
398− return 0;
412+ return ret;
399413 }
400414
401415 void hns_roce_cq_completion(struct hns_roce_dev *hr_dev, u32 cqn)
drivers/infiniband/hw/hns/hns_roce_device.h+46−20
@@ -37,6 +37,8 @@ · 36 unmodified lines
3737
3838 #define DRV_NAME "hns_roce"
3939
40+#define HNS_ROCE_HW_VER1 ('h' << 24 | 'i' << 16 | '0' << 8 | '6')
41+
4042 #define MAC_ADDR_OCTET_NUM 6
4143 #define HNS_ROCE_MAX_MSG_LEN 0x80000000
4244
@@ -54,6 +56,12 @@ · 11 unmodified lines
5456 #define HNS_ROCE_MAX_INNER_MTPT_NUM 0x7
5557 #define HNS_ROCE_MAX_MTPT_PBL_NUM 0x100000
5658
59+#define HNS_ROCE_EACH_FREE_CQ_WAIT_MSECS 20
60+#define HNS_ROCE_MAX_FREE_CQ_WAIT_CNT \
61+ (5000 / HNS_ROCE_EACH_FREE_CQ_WAIT_MSECS)
62+#define HNS_ROCE_CQE_WCMD_EMPTY_BIT 0x2
63+#define HNS_ROCE_MIN_CQE_CNT 16
64+
5765 #define HNS_ROCE_MAX_IRQ_NUM 34
5866
5967 #define HNS_ROCE_COMP_VEC_NUM 32
@@ -70,6 +78,9 @@ · 10 unmodified lines
7078 #define HNS_ROCE_MAX_GID_NUM 16
7179 #define HNS_ROCE_GID_SIZE 16
7280
81+#define BITMAP_NO_RR 0
82+#define BITMAP_RR 1
83+
7384 #define MR_TYPE_MR 0x00
7485 #define MR_TYPE_DMA 0x03
7586
@@ -196,9 +207,9 @@ struct hns_roce_bitmap { · 120 unmodified lines
196207 /* Order = 0: bitmap is biggest, order = max bitmap is least (only a bit) */
197208 /* Every bit repesent to a partner free/used status in bitmap */
198209 /*
199−* Initial, bits of other bitmap are all 0 except that a bit of max_order is 1
200−* Bit = 1 represent to idle and available; bit = 0: not available
201−*/
210+ * Initial, bits of other bitmap are all 0 except that a bit of max_order is 1
211+ * Bit = 1 represent to idle and available; bit = 0: not available
212+ */
202213 struct hns_roce_buddy {
203214 /* Members point to every order level bitmap */
204215 unsigned long **bits;
@@ -296,7 +307,7 @@ struct hns_roce_cq { · 91 unmodified lines
296307 u32 cq_depth;
297308 u32 cons_index;
298309 void __iomem *cq_db_l;
299− void __iomem *tptr_addr;
310+ u16 *tptr_addr;
300311 unsigned long cqn;
301312 u32 vector;
302313 atomic_t refcount;
@@ -360,29 +371,34 @@ struct hns_roce_cmdq { · 57 unmodified lines
360371 struct mutex hcr_mutex;
361372 struct semaphore poll_sem;
362373 /*
363− * Event mode: cmd register mutex protection,
364− * ensure to not exceed max_cmds and user use limit region
365− */
374+ * Event mode: cmd register mutex protection,
375+ * ensure to not exceed max_cmds and user use limit region
376+ */
366377 struct semaphore event_sem;
367378 int max_cmds;
368379 spinlock_t context_lock;
369380 int free_head;
370381 struct hns_roce_cmd_context *context;
371382 /*
372− * Result of get integer part
373− * which max_comds compute according a power of 2
374− */
383+ * Result of get integer part
384+ * which max_comds compute according a power of 2
385+ */
375386 u16 token_mask;
376387 /*
377− * Process whether use event mode, init default non-zero
378− * After the event queue of cmd event ready,
379− * can switch into event mode
380− * close device, switch into poll mode(non event mode)
381− */
388+ * Process whether use event mode, init default non-zero
389+ * After the event queue of cmd event ready,
390+ * can switch into event mode
391+ * close device, switch into poll mode(non event mode)
392+ */
382393 u8 use_events;
383394 u8 toggle;
384395 };
385396
397+struct hns_roce_cmd_mailbox {
398+ void *buf;
399+ dma_addr_t dma;
400+};
401+
386402 struct hns_roce_dev;
387403
388404 struct hns_roce_qp {
@@ -424,8 +440,6 @@ struct hns_roce_ib_iboe { · 35 unmodified lines
424440 struct net_device *netdevs[HNS_ROCE_MAX_PORTS];
425441 struct notifier_block nb;
426442 struct notifier_block nb_inet;
427− /* 16 GID is shared by 6 port in v1 engine. */
428− union ib_gid gid_table[HNS_ROCE_MAX_GID_NUM];
429443 u8 phy_port[HNS_ROCE_MAX_PORTS];
430444 };
431445
@@ -519,6 +533,8 @@ struct hns_roce_hw { · 87 unmodified lines
519533 struct ib_recv_wr **bad_recv_wr);
520534 int (*req_notify_cq)(struct ib_cq *ibcq, enum ib_cq_notify_flags flags);
521535 int (*poll_cq)(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc);
536+ int (*dereg_mr)(struct hns_roce_dev *hr_dev, struct hns_roce_mr *mr);
537+ int (*destroy_cq)(struct ib_cq *ibcq);
522538 void *priv;
523539 };
524540
@@ -553,6 +569,8 @@ struct hns_roce_dev { · 28 unmodified lines
553569
554570 int cmd_mod;
555571 int loop_idc;
572+ dma_addr_t tptr_dma_addr; /*only for hw v1*/
573+ u32 tptr_size; /*only for hw v1*/
556574 struct hns_roce_hw *hw;
557575 };
558576
@@ -657,7 +675,8 @@ void hns_roce_cleanup_cq_table(struct hns_roce_dev *hr_dev); · 98 unmodified lines
657675 void hns_roce_cleanup_qp_table(struct hns_roce_dev *hr_dev);
658676
659677 int hns_roce_bitmap_alloc(struct hns_roce_bitmap *bitmap, unsigned long *obj);
660−void hns_roce_bitmap_free(struct hns_roce_bitmap *bitmap, unsigned long obj);
678+void hns_roce_bitmap_free(struct hns_roce_bitmap *bitmap, unsigned long obj,
679+ int rr);
661680 int hns_roce_bitmap_init(struct hns_roce_bitmap *bitmap, u32 num, u32 mask,
662681 u32 reserved_bot, u32 resetrved_top);
663682 void hns_roce_bitmap_cleanup(struct hns_roce_bitmap *bitmap);
@@ -665,9 +684,11 @@ void hns_roce_cleanup_bitmap(struct hns_roce_dev *hr_dev); · 1 unmodified line
665684 int hns_roce_bitmap_alloc_range(struct hns_roce_bitmap *bitmap, int cnt,
666685 int align, unsigned long *obj);
667686 void hns_roce_bitmap_free_range(struct hns_roce_bitmap *bitmap,
668− unsigned long obj, int cnt);
687+ unsigned long obj, int cnt,
688+ int rr);
669689
670−struct ib_ah *hns_roce_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr);
690+struct ib_ah *hns_roce_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr,
691+ struct ib_udata *udata);
671692 int hns_roce_query_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr);
672693 int hns_roce_destroy_ah(struct ib_ah *ah);
673694
@@ -681,6 +702,10 @@ struct ib_mr *hns_roce_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, · 7 unmodified lines
681702 u64 virt_addr, int access_flags,
682703 struct ib_udata *udata);
683704 int hns_roce_dereg_mr(struct ib_mr *ibmr);
705+int hns_roce_hw2sw_mpt(struct hns_roce_dev *hr_dev,
706+ struct hns_roce_cmd_mailbox *mailbox,
707+ unsigned long mpt_index);
708+unsigned long key_to_hw_index(u32 key);
684709
685710 void hns_roce_buf_free(struct hns_roce_dev *hr_dev, u32 size,
686711 struct hns_roce_buf *buf);
@@ -717,6 +742,7 @@ struct ib_cq *hns_roce_ib_create_cq(struct ib_device *ib_dev, · 30 unmodified lines
717742 struct ib_udata *udata);
718743
719744 int hns_roce_ib_destroy_cq(struct ib_cq *ib_cq);
745+void hns_roce_free_cq(struct hns_roce_dev *hr_dev, struct hns_roce_cq *hr_cq);
720746
721747 void hns_roce_cq_completion(struct hns_roce_dev *hr_dev, u32 cqn);
722748 void hns_roce_cq_event(struct hns_roce_dev *hr_dev, u32 cqn, int event_type);
drivers/infiniband/hw/hns/hns_roce_eq.c+3−3
@@ -371,9 +371,9 @@ static int hns_roce_aeq_ovf_int(struct hns_roce_dev *hr_dev, · 370 unmodified lines
371371 int i = 0;
372372
373373 /**
374− * AEQ overflow ECC mult bit err CEQ overflow alarm
375− * must clear interrupt, mask irq, clear irq, cancel mask operation
376− */
374+ * AEQ overflow ECC mult bit err CEQ overflow alarm
375+ * must clear interrupt, mask irq, clear irq, cancel mask operation
376+ */
377377 aeshift_val = roce_read(hr_dev, ROCEE_CAEP_AEQC_AEQE_SHIFT_REG);
378378
379379 if (roce_get_bit(aeshift_val,
drivers/infiniband/hw/hns/hns_roce_hem.c+3−3
@@ -80,9 +80,9 @@ struct hns_roce_hem *hns_roce_alloc_hem(struct hns_roce_dev *hr_dev, int npages, · 79 unmodified lines
8080 --order;
8181
8282 /*
83− * Alloc memory one time. If failed, don't alloc small block
84− * memory, directly return fail.
85− */
83+ * Alloc memory one time. If failed, don't alloc small block
84+ * memory, directly return fail.
85+ */
8686 mem = &chunk->mem[chunk->npages];
8787 buf = dma_alloc_coherent(&hr_dev->pdev->dev, PAGE_SIZE << order,
8888 &sg_dma_address(mem), gfp_mask);
drivers/infiniband/hw/hns/hns_roce_hw_v1.c+1071−151
@@ -32,6 +32,7 @@ · 31 unmodified lines
3232
3333 #include <linux/platform_device.h>
3434 #include <linux/acpi.h>
35+#include <linux/etherdevice.h>
3536 #include <rdma/ib_umem.h>
3637 #include "hns_roce_common.h"
3738 #include "hns_roce_device.h"
@@ -72,6 +73,8 @@ int hns_roce_v1_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, · 34 unmodified lines
7273 int nreq = 0;
7374 u32 ind = 0;
7475 int ret = 0;
76+ u8 *smac;
77+ int loopback;
7578
7679 if (unlikely(ibqp->qp_type != IB_QPT_GSI &&
7780 ibqp->qp_type != IB_QPT_RC)) {
@@ -129,6 +132,14 @@ int hns_roce_v1_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, · 51 unmodified lines
129132 UD_SEND_WQE_U32_8_DMAC_5_M,
130133 UD_SEND_WQE_U32_8_DMAC_5_S,
131134 ah->av.mac[5]);
135+
136+ smac = (u8 *)hr_dev->dev_addr[qp->port];
137+ loopback = ether_addr_equal_unaligned(ah->av.mac,
138+ smac) ? 1 : 0;
139+ roce_set_bit(ud_sq_wqe->u32_8,
140+ UD_SEND_WQE_U32_8_LOOPBACK_INDICATOR_S,
141+ loopback);
142+
132143 roce_set_field(ud_sq_wqe->u32_8,
133144 UD_SEND_WQE_U32_8_OPERATION_TYPE_M,
134145 UD_SEND_WQE_U32_8_OPERATION_TYPE_S,
@@ -284,6 +295,8 @@ out: · 149 unmodified lines
284295 roce_set_field(sq_db.u32_4, SQ_DOORBELL_U32_4_SQ_HEAD_M,
285296 SQ_DOORBELL_U32_4_SQ_HEAD_S,
286297 (qp->sq.head & ((qp->sq.wqe_cnt << 1) - 1)));
298+ roce_set_field(sq_db.u32_4, SQ_DOORBELL_U32_4_SL_M,
299+ SQ_DOORBELL_U32_4_SL_S, qp->sl);
287300 roce_set_field(sq_db.u32_4, SQ_DOORBELL_U32_4_PORT_M,
288301 SQ_DOORBELL_U32_4_PORT_S, qp->phy_port);
289302 roce_set_field(sq_db.u32_8, SQ_DOORBELL_U32_8_QPN_M,
@@ -611,6 +624,213 @@ ext_sdb_buf_fail_out: · 321 unmodified lines
611624 return ret;
612625 }
613626
627+static struct hns_roce_qp *hns_roce_v1_create_lp_qp(struct hns_roce_dev *hr_dev,
628+ struct ib_pd *pd)
629+{
630+ struct device *dev = &hr_dev->pdev->dev;
631+ struct ib_qp_init_attr init_attr;
632+ struct ib_qp *qp;
633+
634+ memset(&init_attr, 0, sizeof(struct ib_qp_init_attr));
635+ init_attr.qp_type = IB_QPT_RC;
636+ init_attr.sq_sig_type = IB_SIGNAL_ALL_WR;
637+ init_attr.cap.max_recv_wr = HNS_ROCE_MIN_WQE_NUM;
638+ init_attr.cap.max_send_wr = HNS_ROCE_MIN_WQE_NUM;
639+
640+ qp = hns_roce_create_qp(pd, &init_attr, NULL);
641+ if (IS_ERR(qp)) {
642+ dev_err(dev, "Create loop qp for mr free failed!");
643+ return NULL;
644+ }
645+
646+ return to_hr_qp(qp);
647+}
648+
649+static int hns_roce_v1_rsv_lp_qp(struct hns_roce_dev *hr_dev)
650+{
651+ struct hns_roce_caps *caps = &hr_dev->caps;
652+ struct device *dev = &hr_dev->pdev->dev;
653+ struct ib_cq_init_attr cq_init_attr;
654+ struct hns_roce_free_mr *free_mr;
655+ struct ib_qp_attr attr = { 0 };
656+ struct hns_roce_v1_priv *priv;
657+ struct hns_roce_qp *hr_qp;
658+ struct ib_cq *cq;
659+ struct ib_pd *pd;
660+ u64 subnet_prefix;
661+ int attr_mask = 0;
662+ int i;
663+ int ret;
664+ u8 phy_port;
665+ u8 sl;
666+
667+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
668+ free_mr = &priv->free_mr;
669+
670+ /* Reserved cq for loop qp */
671+ cq_init_attr.cqe = HNS_ROCE_MIN_WQE_NUM * 2;
672+ cq_init_attr.comp_vector = 0;
673+ cq = hns_roce_ib_create_cq(&hr_dev->ib_dev, &cq_init_attr, NULL, NULL);
674+ if (IS_ERR(cq)) {
675+ dev_err(dev, "Create cq for reseved loop qp failed!");
676+ return -ENOMEM;
677+ }
678+ free_mr->mr_free_cq = to_hr_cq(cq);
679+ free_mr->mr_free_cq->ib_cq.device = &hr_dev->ib_dev;
680+ free_mr->mr_free_cq->ib_cq.uobject = NULL;
681+ free_mr->mr_free_cq->ib_cq.comp_handler = NULL;
682+ free_mr->mr_free_cq->ib_cq.event_handler = NULL;
683+ free_mr->mr_free_cq->ib_cq.cq_context = NULL;
684+ atomic_set(&free_mr->mr_free_cq->ib_cq.usecnt, 0);
685+
686+ pd = hns_roce_alloc_pd(&hr_dev->ib_dev, NULL, NULL);
687+ if (IS_ERR(pd)) {
688+ dev_err(dev, "Create pd for reseved loop qp failed!");
689+ ret = -ENOMEM;
690+ goto alloc_pd_failed;
691+ }
692+ free_mr->mr_free_pd = to_hr_pd(pd);
693+ free_mr->mr_free_pd->ibpd.device = &hr_dev->ib_dev;
694+ free_mr->mr_free_pd->ibpd.uobject = NULL;
695+ atomic_set(&free_mr->mr_free_pd->ibpd.usecnt, 0);
696+
697+ attr.qp_access_flags = IB_ACCESS_REMOTE_WRITE;
698+ attr.pkey_index = 0;
699+ attr.min_rnr_timer = 0;
700+ /* Disable read ability */
701+ attr.max_dest_rd_atomic = 0;
702+ attr.max_rd_atomic = 0;
703+ /* Use arbitrary values as rq_psn and sq_psn */
704+ attr.rq_psn = 0x0808;
705+ attr.sq_psn = 0x0808;
706+ attr.retry_cnt = 7;
707+ attr.rnr_retry = 7;
708+ attr.timeout = 0x12;
709+ attr.path_mtu = IB_MTU_256;
710+ attr.ah_attr.ah_flags = 1;
711+ attr.ah_attr.static_rate = 3;
712+ attr.ah_attr.grh.sgid_index = 0;
713+ attr.ah_attr.grh.hop_limit = 1;
714+ attr.ah_attr.grh.flow_label = 0;
715+ attr.ah_attr.grh.traffic_class = 0;
716+
717+ subnet_prefix = cpu_to_be64(0xfe80000000000000LL);
718+ for (i = 0; i < HNS_ROCE_V1_RESV_QP; i++) {
719+ free_mr->mr_free_qp[i] = hns_roce_v1_create_lp_qp(hr_dev, pd);
720+ if (IS_ERR(free_mr->mr_free_qp[i])) {
721+ dev_err(dev, "Create loop qp failed!\n");
722+ goto create_lp_qp_failed;
723+ }
724+ hr_qp = free_mr->mr_free_qp[i];
725+
726+ sl = i / caps->num_ports;
727+
728+ if (caps->num_ports == HNS_ROCE_MAX_PORTS)
729+ phy_port = (i >= HNS_ROCE_MAX_PORTS) ? (i - 2) :
730+ (i % caps->num_ports);
731+ else
732+ phy_port = i % caps->num_ports;
733+
734+ hr_qp->port = phy_port + 1;
735+ hr_qp->phy_port = phy_port;
736+ hr_qp->ibqp.qp_type = IB_QPT_RC;
737+ hr_qp->ibqp.device = &hr_dev->ib_dev;
738+ hr_qp->ibqp.uobject = NULL;
739+ atomic_set(&hr_qp->ibqp.usecnt, 0);
740+ hr_qp->ibqp.pd = pd;
741+ hr_qp->ibqp.recv_cq = cq;
742+ hr_qp->ibqp.send_cq = cq;
743+
744+ attr.ah_attr.port_num = phy_port + 1;
745+ attr.ah_attr.sl = sl;
746+ attr.port_num = phy_port + 1;
747+
748+ attr.dest_qp_num = hr_qp->qpn;
749+ memcpy(attr.ah_attr.dmac, hr_dev->dev_addr[phy_port],
750+ MAC_ADDR_OCTET_NUM);
751+
752+ memcpy(attr.ah_attr.grh.dgid.raw,
753+ &subnet_prefix, sizeof(u64));
754+ memcpy(&attr.ah_attr.grh.dgid.raw[8],
755+ hr_dev->dev_addr[phy_port], 3);
756+ memcpy(&attr.ah_attr.grh.dgid.raw[13],
757+ hr_dev->dev_addr[phy_port] + 3, 3);
758+ attr.ah_attr.grh.dgid.raw[11] = 0xff;
759+ attr.ah_attr.grh.dgid.raw[12] = 0xfe;
760+ attr.ah_attr.grh.dgid.raw[8] ^= 2;
761+
762+ attr_mask |= IB_QP_PORT;
763+
764+ ret = hr_dev->hw->modify_qp(&hr_qp->ibqp, &attr, attr_mask,
765+ IB_QPS_RESET, IB_QPS_INIT);
766+ if (ret) {
767+ dev_err(dev, "modify qp failed(%d)!\n", ret);
768+ goto create_lp_qp_failed;
769+ }
770+
771+ ret = hr_dev->hw->modify_qp(&hr_qp->ibqp, &attr, attr_mask,
772+ IB_QPS_INIT, IB_QPS_RTR);
773+ if (ret) {
774+ dev_err(dev, "modify qp failed(%d)!\n", ret);
775+ goto create_lp_qp_failed;
776+ }
777+
778+ ret = hr_dev->hw->modify_qp(&hr_qp->ibqp, &attr, attr_mask,
779+ IB_QPS_RTR, IB_QPS_RTS);
780+ if (ret) {
781+ dev_err(dev, "modify qp failed(%d)!\n", ret);
782+ goto create_lp_qp_failed;
783+ }
784+ }
785+
786+ return 0;
787+
788+create_lp_qp_failed:
789+ for (i -= 1; i >= 0; i--) {
790+ hr_qp = free_mr->mr_free_qp[i];
791+ if (hns_roce_v1_destroy_qp(&hr_qp->ibqp))
792+ dev_err(dev, "Destroy qp %d for mr free failed!\n", i);
793+ }
794+
795+ if (hns_roce_dealloc_pd(pd))
796+ dev_err(dev, "Destroy pd for create_lp_qp failed!\n");
797+
798+alloc_pd_failed:
799+ if (hns_roce_ib_destroy_cq(cq))
800+ dev_err(dev, "Destroy cq for create_lp_qp failed!\n");
801+
802+ return -EINVAL;
803+}
804+
805+static void hns_roce_v1_release_lp_qp(struct hns_roce_dev *hr_dev)
806+{
807+ struct device *dev = &hr_dev->pdev->dev;
808+ struct hns_roce_free_mr *free_mr;
809+ struct hns_roce_v1_priv *priv;
810+ struct hns_roce_qp *hr_qp;
811+ int ret;
812+ int i;
813+
814+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
815+ free_mr = &priv->free_mr;
816+
817+ for (i = 0; i < HNS_ROCE_V1_RESV_QP; i++) {
818+ hr_qp = free_mr->mr_free_qp[i];
819+ ret = hns_roce_v1_destroy_qp(&hr_qp->ibqp);
820+ if (ret)
821+ dev_err(dev, "Destroy qp %d for mr free failed(%d)!\n",
822+ i, ret);
823+ }
824+
825+ ret = hns_roce_ib_destroy_cq(&free_mr->mr_free_cq->ib_cq);
826+ if (ret)
827+ dev_err(dev, "Destroy cq for mr_free failed(%d)!\n", ret);
828+
829+ ret = hns_roce_dealloc_pd(&free_mr->mr_free_pd->ibpd);
830+ if (ret)
831+ dev_err(dev, "Destroy pd for mr_free failed(%d)!\n", ret);
832+}
833+
614834 static int hns_roce_db_init(struct hns_roce_dev *hr_dev)
615835 {
616836 struct device *dev = &hr_dev->pdev->dev;
@@ -648,6 +868,223 @@ static int hns_roce_db_init(struct hns_roce_dev *hr_dev) · 31 unmodified lines
648868 return 0;
649869 }
650870
871+void hns_roce_v1_recreate_lp_qp_work_fn(struct work_struct *work)
872+{
873+ struct hns_roce_recreate_lp_qp_work *lp_qp_work;
874+ struct hns_roce_dev *hr_dev;
875+
876+ lp_qp_work = container_of(work, struct hns_roce_recreate_lp_qp_work,
877+ work);
878+ hr_dev = to_hr_dev(lp_qp_work->ib_dev);
879+
880+ hns_roce_v1_release_lp_qp(hr_dev);
881+
882+ if (hns_roce_v1_rsv_lp_qp(hr_dev))
883+ dev_err(&hr_dev->pdev->dev, "create reserver qp failed\n");
884+
885+ if (lp_qp_work->comp_flag)
886+ complete(lp_qp_work->comp);
887+
888+ kfree(lp_qp_work);
889+}
890+
891+static int hns_roce_v1_recreate_lp_qp(struct hns_roce_dev *hr_dev)
892+{
893+ struct device *dev = &hr_dev->pdev->dev;
894+ struct hns_roce_recreate_lp_qp_work *lp_qp_work;
895+ struct hns_roce_free_mr *free_mr;
896+ struct hns_roce_v1_priv *priv;
897+ struct completion comp;
898+ unsigned long end =
899+ msecs_to_jiffies(HNS_ROCE_V1_RECREATE_LP_QP_TIMEOUT_MSECS) + jiffies;
900+
901+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
902+ free_mr = &priv->free_mr;
903+
904+ lp_qp_work = kzalloc(sizeof(struct hns_roce_recreate_lp_qp_work),
905+ GFP_KERNEL);
906+
907+ INIT_WORK(&(lp_qp_work->work), hns_roce_v1_recreate_lp_qp_work_fn);
908+
909+ lp_qp_work->ib_dev = &(hr_dev->ib_dev);
910+ lp_qp_work->comp = &comp;
911+ lp_qp_work->comp_flag = 1;
912+
913+ init_completion(lp_qp_work->comp);
914+
915+ queue_work(free_mr->free_mr_wq, &(lp_qp_work->work));
916+
917+ while (time_before_eq(jiffies, end)) {
918+ if (try_wait_for_completion(&comp))
919+ return 0;
920+ msleep(HNS_ROCE_V1_RECREATE_LP_QP_WAIT_VALUE);
921+ }
922+
923+ lp_qp_work->comp_flag = 0;
924+ if (try_wait_for_completion(&comp))
925+ return 0;
926+
927+ dev_warn(dev, "recreate lp qp failed 20s timeout and return failed!\n");
928+ return -ETIMEDOUT;
929+}
930+
931+static int hns_roce_v1_send_lp_wqe(struct hns_roce_qp *hr_qp)
932+{
933+ struct hns_roce_dev *hr_dev = to_hr_dev(hr_qp->ibqp.device);
934+ struct device *dev = &hr_dev->pdev->dev;
935+ struct ib_send_wr send_wr, *bad_wr;
936+ int ret;
937+
938+ memset(&send_wr, 0, sizeof(send_wr));
939+ send_wr.next = NULL;
940+ send_wr.num_sge = 0;
941+ send_wr.send_flags = 0;
942+ send_wr.sg_list = NULL;
943+ send_wr.wr_id = (unsigned long long)&send_wr;
944+ send_wr.opcode = IB_WR_RDMA_WRITE;
945+
946+ ret = hns_roce_v1_post_send(&hr_qp->ibqp, &send_wr, &bad_wr);
947+ if (ret) {
948+ dev_err(dev, "Post write wqe for mr free failed(%d)!", ret);
949+ return ret;
950+ }
951+
952+ return 0;
953+}
954+
955+static void hns_roce_v1_mr_free_work_fn(struct work_struct *work)
956+{
957+ struct hns_roce_mr_free_work *mr_work;
958+ struct ib_wc wc[HNS_ROCE_V1_RESV_QP];
959+ struct hns_roce_free_mr *free_mr;
960+ struct hns_roce_cq *mr_free_cq;
961+ struct hns_roce_v1_priv *priv;
962+ struct hns_roce_dev *hr_dev;
963+ struct hns_roce_mr *hr_mr;
964+ struct hns_roce_qp *hr_qp;
965+ struct device *dev;
966+ unsigned long end =
967+ msecs_to_jiffies(HNS_ROCE_V1_FREE_MR_TIMEOUT_MSECS) + jiffies;
968+ int i;
969+ int ret;
970+ int ne;
971+
972+ mr_work = container_of(work, struct hns_roce_mr_free_work, work);
973+ hr_mr = (struct hns_roce_mr *)mr_work->mr;
974+ hr_dev = to_hr_dev(mr_work->ib_dev);
975+ dev = &hr_dev->pdev->dev;
976+
977+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
978+ free_mr = &priv->free_mr;
979+ mr_free_cq = free_mr->mr_free_cq;
980+
981+ for (i = 0; i < HNS_ROCE_V1_RESV_QP; i++) {
982+ hr_qp = free_mr->mr_free_qp[i];
983+ ret = hns_roce_v1_send_lp_wqe(hr_qp);
984+ if (ret) {
985+ dev_err(dev,
986+ "Send wqe (qp:0x%lx) for mr free failed(%d)!\n",
987+ hr_qp->qpn, ret);
988+ goto free_work;
989+ }
990+ }
991+
992+ ne = HNS_ROCE_V1_RESV_QP;
993+ do {
994+ ret = hns_roce_v1_poll_cq(&mr_free_cq->ib_cq, ne, wc);
995+ if (ret < 0) {
996+ dev_err(dev,
997+ "(qp:0x%lx) starts, Poll cqe failed(%d) for mr 0x%x free! Remain %d cqe\n",
998+ hr_qp->qpn, ret, hr_mr->key, ne);
999+ goto free_work;
1000+ }
1001+ ne -= ret;
1002+ msleep(HNS_ROCE_V1_FREE_MR_WAIT_VALUE);
1003+ } while (ne && time_before_eq(jiffies, end));
1004+
1005+ if (ne != 0)
1006+ dev_err(dev,
1007+ "Poll cqe for mr 0x%x free timeout! Remain %d cqe\n",
1008+ hr_mr->key, ne);
1009+
1010+free_work:
1011+ if (mr_work->comp_flag)
1012+ complete(mr_work->comp);
1013+ kfree(mr_work);
1014+}
1015+
1016+int hns_roce_v1_dereg_mr(struct hns_roce_dev *hr_dev, struct hns_roce_mr *mr)
1017+{
1018+ struct device *dev = &hr_dev->pdev->dev;
1019+ struct hns_roce_mr_free_work *mr_work;
1020+ struct hns_roce_free_mr *free_mr;
1021+ struct hns_roce_v1_priv *priv;
1022+ struct completion comp;
1023+ unsigned long end =
1024+ msecs_to_jiffies(HNS_ROCE_V1_FREE_MR_TIMEOUT_MSECS) + jiffies;
1025+ unsigned long start = jiffies;
1026+ int npages;
1027+ int ret = 0;
1028+
1029+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
1030+ free_mr = &priv->free_mr;
1031+
1032+ if (mr->enabled) {
1033+ if (hns_roce_hw2sw_mpt(hr_dev, NULL, key_to_hw_index(mr->key)
1034+ & (hr_dev->caps.num_mtpts - 1)))
1035+ dev_warn(dev, "HW2SW_MPT failed!\n");
1036+ }
1037+
1038+ mr_work = kzalloc(sizeof(*mr_work), GFP_KERNEL);
1039+ if (!mr_work) {
1040+ ret = -ENOMEM;
1041+ goto free_mr;
1042+ }
1043+
1044+ INIT_WORK(&(mr_work->work), hns_roce_v1_mr_free_work_fn);
1045+
1046+ mr_work->ib_dev = &(hr_dev->ib_dev);
1047+ mr_work->comp = &comp;
1048+ mr_work->comp_flag = 1;
1049+ mr_work->mr = (void *)mr;
1050+ init_completion(mr_work->comp);
1051+
1052+ queue_work(free_mr->free_mr_wq, &(mr_work->work));
1053+
1054+ while (time_before_eq(jiffies, end)) {
1055+ if (try_wait_for_completion(&comp))
1056+ goto free_mr;
1057+ msleep(HNS_ROCE_V1_FREE_MR_WAIT_VALUE);
1058+ }
1059+
1060+ mr_work->comp_flag = 0;
1061+ if (try_wait_for_completion(&comp))
1062+ goto free_mr;
1063+
1064+ dev_warn(dev, "Free mr work 0x%x over 50s and failed!\n", mr->key);
1065+ ret = -ETIMEDOUT;
1066+
1067+free_mr:
1068+ dev_dbg(dev, "Free mr 0x%x use 0x%x us.\n",
1069+ mr->key, jiffies_to_usecs(jiffies) - jiffies_to_usecs(start));
1070+
1071+ if (mr->size != ~0ULL) {
1072+ npages = ib_umem_page_count(mr->umem);
1073+ dma_free_coherent(dev, npages * 8, mr->pbl_buf,
1074+ mr->pbl_dma_addr);
1075+ }
1076+
1077+ hns_roce_bitmap_free(&hr_dev->mr_table.mtpt_bitmap,
1078+ key_to_hw_index(mr->key), 0);
1079+
1080+ if (mr->umem)
1081+ ib_umem_release(mr->umem);
1082+
1083+ kfree(mr);
1084+
1085+ return ret;
1086+}
1087+
6511088 static void hns_roce_db_free(struct hns_roce_dev *hr_dev)
6521089 {
6531090 struct device *dev = &hr_dev->pdev->dev;
@@ -849,6 +1286,85 @@ static void hns_roce_bt_free(struct hns_roce_dev *hr_dev) · 195 unmodified lines
8491286 priv->bt_table.qpc_buf.buf, priv->bt_table.qpc_buf.map);
8501287 }
8511288
1289+static int hns_roce_tptr_init(struct hns_roce_dev *hr_dev)
1290+{
1291+ struct device *dev = &hr_dev->pdev->dev;
1292+ struct hns_roce_buf_list *tptr_buf;
1293+ struct hns_roce_v1_priv *priv;
1294+
1295+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
1296+ tptr_buf = &priv->tptr_table.tptr_buf;
1297+
1298+ /*
1299+ * This buffer will be used for CQ's tptr(tail pointer), also
1300+ * named ci(customer index). Every CQ will use 2 bytes to save
1301+ * cqe ci in hip06. Hardware will read this area to get new ci
1302+ * when the queue is almost full.
1303+ */
1304+ tptr_buf->buf = dma_alloc_coherent(dev, HNS_ROCE_V1_TPTR_BUF_SIZE,
1305+ &tptr_buf->map, GFP_KERNEL);
1306+ if (!tptr_buf->buf)
1307+ return -ENOMEM;
1308+
1309+ hr_dev->tptr_dma_addr = tptr_buf->map;
1310+ hr_dev->tptr_size = HNS_ROCE_V1_TPTR_BUF_SIZE;
1311+
1312+ return 0;
1313+}
1314+
1315+static void hns_roce_tptr_free(struct hns_roce_dev *hr_dev)
1316+{
1317+ struct device *dev = &hr_dev->pdev->dev;
1318+ struct hns_roce_buf_list *tptr_buf;
1319+ struct hns_roce_v1_priv *priv;
1320+
1321+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
1322+ tptr_buf = &priv->tptr_table.tptr_buf;
1323+
1324+ dma_free_coherent(dev, HNS_ROCE_V1_TPTR_BUF_SIZE,
1325+ tptr_buf->buf, tptr_buf->map);
1326+}
1327+
1328+static int hns_roce_free_mr_init(struct hns_roce_dev *hr_dev)
1329+{
1330+ struct device *dev = &hr_dev->pdev->dev;
1331+ struct hns_roce_free_mr *free_mr;
1332+ struct hns_roce_v1_priv *priv;
1333+ int ret = 0;
1334+
1335+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
1336+ free_mr = &priv->free_mr;
1337+
1338+ free_mr->free_mr_wq = create_singlethread_workqueue("hns_roce_free_mr");
1339+ if (!free_mr->free_mr_wq) {
1340+ dev_err(dev, "Create free mr workqueue failed!\n");
1341+ return -ENOMEM;
1342+ }
1343+
1344+ ret = hns_roce_v1_rsv_lp_qp(hr_dev);
1345+ if (ret) {
1346+ dev_err(dev, "Reserved loop qp failed(%d)!\n", ret);
1347+ flush_workqueue(free_mr->free_mr_wq);
1348+ destroy_workqueue(free_mr->free_mr_wq);
1349+ }
1350+
1351+ return ret;
1352+}
1353+
1354+static void hns_roce_free_mr_free(struct hns_roce_dev *hr_dev)
1355+{
1356+ struct hns_roce_free_mr *free_mr;
1357+ struct hns_roce_v1_priv *priv;
1358+
1359+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
1360+ free_mr = &priv->free_mr;
1361+
1362+ flush_workqueue(free_mr->free_mr_wq);
1363+ destroy_workqueue(free_mr->free_mr_wq);
1364+
1365+ hns_roce_v1_release_lp_qp(hr_dev);
1366+}
1367+
8521368 /**
8531369 * hns_roce_v1_reset - reset RoCE
8541370 * @hr_dev: RoCE device struct pointer
@@ -898,6 +1414,38 @@ int hns_roce_v1_reset(struct hns_roce_dev *hr_dev, bool dereset) · 43 unmodified lines
8981414 return ret;
8991415 }
9001416
1417+static int hns_roce_des_qp_init(struct hns_roce_dev *hr_dev)
1418+{
1419+ struct device *dev = &hr_dev->pdev->dev;
1420+ struct hns_roce_v1_priv *priv;
1421+ struct hns_roce_des_qp *des_qp;
1422+
1423+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
1424+ des_qp = &priv->des_qp;
1425+
1426+ des_qp->requeue_flag = 1;
1427+ des_qp->qp_wq = create_singlethread_workqueue("hns_roce_destroy_qp");
1428+ if (!des_qp->qp_wq) {
1429+ dev_err(dev, "Create destroy qp workqueue failed!\n");
1430+ return -ENOMEM;
1431+ }
1432+
1433+ return 0;
1434+}
1435+
1436+static void hns_roce_des_qp_free(struct hns_roce_dev *hr_dev)
1437+{
1438+ struct hns_roce_v1_priv *priv;
1439+ struct hns_roce_des_qp *des_qp;
1440+
1441+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
1442+ des_qp = &priv->des_qp;
1443+
1444+ des_qp->requeue_flag = 0;
1445+ flush_workqueue(des_qp->qp_wq);
1446+ destroy_workqueue(des_qp->qp_wq);
1447+}
1448+
9011449 void hns_roce_v1_profile(struct hns_roce_dev *hr_dev)
9021450 {
9031451 int i = 0;
@@ -906,12 +1454,11 @@ void hns_roce_v1_profile(struct hns_roce_dev *hr_dev) · 2 unmodified lines
9061454 hr_dev->vendor_id = le32_to_cpu(roce_read(hr_dev, ROCEE_VENDOR_ID_REG));
9071455 hr_dev->vendor_part_id = le32_to_cpu(roce_read(hr_dev,
9081456 ROCEE_VENDOR_PART_ID_REG));
909− hr_dev->hw_rev = le32_to_cpu(roce_read(hr_dev, ROCEE_HW_VERSION_REG));
910−
9111457 hr_dev->sys_image_guid = le32_to_cpu(roce_read(hr_dev,
9121458 ROCEE_SYS_IMAGE_GUID_L_REG)) |
9131459 ((u64)le32_to_cpu(roce_read(hr_dev,
9141460 ROCEE_SYS_IMAGE_GUID_H_REG)) << 32);
1461+ hr_dev->hw_rev = HNS_ROCE_HW_VER1;
9151462
9161463 caps->num_qps = HNS_ROCE_V1_MAX_QP_NUM;
9171464 caps->max_wqes = HNS_ROCE_V1_MAX_WQE_NUM;
@@ -1001,18 +1548,44 @@ int hns_roce_v1_init(struct hns_roce_dev *hr_dev) · 83 unmodified lines
10011548 goto error_failed_raq_init;
10021549 }
10031550
1004− hns_roce_port_enable(hr_dev, HNS_ROCE_PORT_UP);
1005−
10061551 ret = hns_roce_bt_init(hr_dev);
10071552 if (ret) {
10081553 dev_err(dev, "bt init failed!\n");
10091554 goto error_failed_bt_init;
10101555 }
10111556
1557+ ret = hns_roce_tptr_init(hr_dev);
1558+ if (ret) {
1559+ dev_err(dev, "tptr init failed!\n");
1560+ goto error_failed_tptr_init;
1561+ }
1562+
1563+ ret = hns_roce_des_qp_init(hr_dev);
1564+ if (ret) {
1565+ dev_err(dev, "des qp init failed!\n");
1566+ goto error_failed_des_qp_init;
1567+ }
1568+
1569+ ret = hns_roce_free_mr_init(hr_dev);
1570+ if (ret) {
1571+ dev_err(dev, "free mr init failed!\n");
1572+ goto error_failed_free_mr_init;
1573+ }
1574+
1575+ hns_roce_port_enable(hr_dev, HNS_ROCE_PORT_UP);
1576+
10121577 return 0;
10131578
1579+error_failed_free_mr_init:
1580+ hns_roce_des_qp_free(hr_dev);
1581+
1582+error_failed_des_qp_init:
1583+ hns_roce_tptr_free(hr_dev);
1584+
1585+error_failed_tptr_init:
1586+ hns_roce_bt_free(hr_dev);
1587+
10141588 error_failed_bt_init:
1015− hns_roce_port_enable(hr_dev, HNS_ROCE_PORT_DOWN);
10161589 hns_roce_raq_free(hr_dev);
10171590
10181591 error_failed_raq_init:
@@ -1022,8 +1595,11 @@ error_failed_raq_init: · 3 unmodified lines
10221595
10231596 void hns_roce_v1_exit(struct hns_roce_dev *hr_dev)
10241597 {
1025− hns_roce_bt_free(hr_dev);
10261598 hns_roce_port_enable(hr_dev, HNS_ROCE_PORT_DOWN);
1599+ hns_roce_free_mr_free(hr_dev);
1600+ hns_roce_des_qp_free(hr_dev);
1601+ hns_roce_tptr_free(hr_dev);
1602+ hns_roce_bt_free(hr_dev);
10271603 hns_roce_raq_free(hr_dev);
10281604 hns_roce_db_free(hr_dev);
10291605 }
@@ -1061,6 +1637,14 @@ void hns_roce_v1_set_mac(struct hns_roce_dev *hr_dev, u8 phy_port, u8 *addr) · 31 unmodified lines
10611637 u32 *p;
10621638 u32 val;
10631639
1640+ /*
1641+ * When mac changed, loopback may fail
1642+ * because of smac not equal to dmac.
1643+ * We Need to release and create reserved qp again.
1644+ */
1645+ if (hr_dev->hw->dereg_mr && hns_roce_v1_recreate_lp_qp(hr_dev))
1646+ dev_warn(&hr_dev->pdev->dev, "recreate lp qp timeout!\n");
1647+
10641648 p = (u32 *)(&addr[0]);
10651649 reg_smac_l = *p;
10661650 roce_raw_write(reg_smac_l, hr_dev->reg_base + ROCEE_SMAC_L_0_REG +
@@ -1293,9 +1877,9 @@ static void __hns_roce_v1_cq_clean(struct hns_roce_cq *hr_cq, u32 qpn, · 226 unmodified lines
12931877 }
12941878
12951879 /*
1296− * Now backwards through the CQ, removing CQ entries
1297− * that match our QP by overwriting them with next entries.
1298− */
1880+ * Now backwards through the CQ, removing CQ entries
1881+ * that match our QP by overwriting them with next entries.
1882+ */
12991883 while ((int) --prod_index - (int) hr_cq->cons_index >= 0) {
13001884 cqe = get_cqe(hr_cq, prod_index & hr_cq->ib_cq.cqe);
13011885 if ((roce_get_field(cqe->cqe_byte_16, CQE_BYTE_16_LOCAL_QPN_M,
@@ -1317,9 +1901,9 @@ static void __hns_roce_v1_cq_clean(struct hns_roce_cq *hr_cq, u32 qpn, · 15 unmodified lines
13171901 if (nfreed) {
13181902 hr_cq->cons_index += nfreed;
13191903 /*
1320− * Make sure update of buffer contents is done before
1321− * updating consumer index.
1322− */
1904+ * Make sure update of buffer contents is done before
1905+ * updating consumer index.
1906+ */
13231907 wmb();
13241908
13251909 hns_roce_v1_cq_set_ci(hr_cq, hr_cq->cons_index);
@@ -1339,14 +1923,21 @@ void hns_roce_v1_write_cqc(struct hns_roce_dev *hr_dev, · 13 unmodified lines
13391923 dma_addr_t dma_handle, int nent, u32 vector)
13401924 {
13411925 struct hns_roce_cq_context *cq_context = NULL;
1342− void __iomem *tptr_addr;
1926+ struct hns_roce_buf_list *tptr_buf;
1927+ struct hns_roce_v1_priv *priv;
1928+ dma_addr_t tptr_dma_addr;
1929+ int offset;
1930+
1931+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
1932+ tptr_buf = &priv->tptr_table.tptr_buf;
13431933
13441934 cq_context = mb_buf;
13451935 memset(cq_context, 0, sizeof(*cq_context));
13461936
1347− tptr_addr = 0;
1348− hr_dev->priv_addr = tptr_addr;
1349− hr_cq->tptr_addr = tptr_addr;
1937+ /* Get the tptr for this CQ. */
1938+ offset = hr_cq->cqn * HNS_ROCE_V1_TPTR_ENTRY_SIZE;
1939+ tptr_dma_addr = tptr_buf->map + offset;
1940+ hr_cq->tptr_addr = (u16 *)(tptr_buf->buf + offset);
13501941
13511942 /* Register cq_context members */
13521943 roce_set_field(cq_context->cqc_byte_4,
@@ -1390,10 +1981,10 @@ void hns_roce_v1_write_cqc(struct hns_roce_dev *hr_dev, · 37 unmodified lines
13901981 roce_set_field(cq_context->cqc_byte_20,
13911982 CQ_CONTEXT_CQC_BYTE_20_CQE_TPTR_ADDR_H_M,
13921983 CQ_CONTEXT_CQC_BYTE_20_CQE_TPTR_ADDR_H_S,
1393− (u64)tptr_addr >> 44);
1984+ tptr_dma_addr >> 44);
13941985 cq_context->cqc_byte_20 = cpu_to_le32(cq_context->cqc_byte_20);
13951986
1396− cq_context->cqe_tptr_addr_l = (u32)((u64)tptr_addr >> 12);
1987+ cq_context->cqe_tptr_addr_l = (u32)(tptr_dma_addr >> 12);
13971988
13981989 roce_set_field(cq_context->cqc_byte_32,
13991990 CQ_CONTEXT_CQC_BYTE_32_CUR_CQE_BA1_H_M,
@@ -1407,7 +1998,7 @@ void hns_roce_v1_write_cqc(struct hns_roce_dev *hr_dev, · 7 unmodified lines
14071998 roce_set_bit(cq_context->cqc_byte_32,
14081999 CQ_CQNTEXT_CQC_BYTE_32_TYPE_OF_COMPLETION_NOTIFICATION_S,
14092000 0);
1410− /*The initial value of cq's ci is 0 */
2001+ /* The initial value of cq's ci is 0 */
14112002 roce_set_field(cq_context->cqc_byte_32,
14122003 CQ_CONTEXT_CQC_BYTE_32_CQ_CONS_IDX_M,
14132004 CQ_CONTEXT_CQC_BYTE_32_CQ_CONS_IDX_S, 0);
@@ -1424,9 +2015,9 @@ int hns_roce_v1_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags) · 10 unmodified lines
14242015 notification_flag = (flags & IB_CQ_SOLICITED_MASK) ==
14252016 IB_CQ_SOLICITED ? CQ_DB_REQ_NOT : CQ_DB_REQ_NOT_SOL;
14262017 /*
1427− * flags = 0; Notification Flag = 1, next
1428− * flags = 1; Notification Flag = 0, solocited
1429− */
2018+ * flags = 0; Notification Flag = 1, next
2019+ * flags = 1; Notification Flag = 0, solocited
2020+ */
14302021 doorbell[0] = hr_cq->cons_index & ((hr_cq->cq_depth << 1) - 1);
14312022 roce_set_bit(doorbell[1], ROCEE_DB_OTHERS_H_ROCEE_DB_OTH_HW_SYNS_S, 1);
14322023 roce_set_field(doorbell[1], ROCEE_DB_OTHERS_H_ROCEE_DB_OTH_CMD_M,
@@ -1581,10 +2172,10 @@ static int hns_roce_v1_poll_one(struct hns_roce_cq *hr_cq, · 148 unmodified lines
15812172 wq = &(*cur_qp)->sq;
15822173 if ((*cur_qp)->sq_signal_bits) {
15832174 /*
1584− * If sg_signal_bit is 1,
1585− * firstly tail pointer updated to wqe
1586− * which current cqe correspond to
1587− */
2175+ * If sg_signal_bit is 1,
2176+ * firstly tail pointer updated to wqe
2177+ * which current cqe correspond to
2178+ */
15882179 wqe_ctr = (u16)roce_get_field(cqe->cqe_byte_4,
15892180 CQE_BYTE_4_WQE_INDEX_M,
15902181 CQE_BYTE_4_WQE_INDEX_S);
@@ -1659,8 +2250,14 @@ int hns_roce_v1_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc) · 68 unmodified lines
16592250 break;
16602251 }
16612252
1662− if (npolled)
2253+ if (npolled) {
2254+ *hr_cq->tptr_addr = hr_cq->cons_index &
2255+ ((hr_cq->cq_depth << 1) - 1);
2256+
2257+ /* Memroy barrier */
2258+ wmb();
16632259 hns_roce_v1_cq_set_ci(hr_cq, hr_cq->cons_index);
2260+ }
16642261
16652262 spin_unlock_irqrestore(&hr_cq->lock, flags);
16662263
@@ -1799,12 +2396,12 @@ static int hns_roce_v1_qp_modify(struct hns_roce_dev *hr_dev, · 132 unmodified lines
17992396 if (op[cur_state][new_state] == HNS_ROCE_CMD_2RST_QP)
18002397 return hns_roce_cmd_mbox(hr_dev, 0, 0, hr_qp->qpn, 2,
18012398 HNS_ROCE_CMD_2RST_QP,
1802− HNS_ROCE_CMD_TIME_CLASS_A);
2399+ HNS_ROCE_CMD_TIMEOUT_MSECS);
18032400
18042401 if (op[cur_state][new_state] == HNS_ROCE_CMD_2ERR_QP)
18052402 return hns_roce_cmd_mbox(hr_dev, 0, 0, hr_qp->qpn, 2,
18062403 HNS_ROCE_CMD_2ERR_QP,
1807− HNS_ROCE_CMD_TIME_CLASS_A);
2404+ HNS_ROCE_CMD_TIMEOUT_MSECS);
18082405
18092406 mailbox = hns_roce_alloc_cmd_mailbox(hr_dev);
18102407 if (IS_ERR(mailbox))
@@ -1814,7 +2411,7 @@ static int hns_roce_v1_qp_modify(struct hns_roce_dev *hr_dev, · 3 unmodified lines
18142411
18152412 ret = hns_roce_cmd_mbox(hr_dev, mailbox->dma, 0, hr_qp->qpn, 0,
18162413 op[cur_state][new_state],
1817− HNS_ROCE_CMD_TIME_CLASS_C);
2414+ HNS_ROCE_CMD_TIMEOUT_MSECS);
18182415
18192416 hns_roce_free_cmd_mailbox(hr_dev, mailbox);
18202417 return ret;
@@ -2000,11 +2597,11 @@ static int hns_roce_v1_m_qp(struct ib_qp *ibqp, const struct ib_qp_attr *attr, · 179 unmodified lines
20002597 }
20012598
20022599 /*
2003− *Reset to init
2004− * Mandatory param:
2005− * IB_QP_STATE | IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_ACCESS_FLAGS
2006− * Optional param: NA
2007− */
2600+ * Reset to init
2601+ * Mandatory param:
2602+ * IB_QP_STATE | IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_ACCESS_FLAGS
2603+ * Optional param: NA
2604+ */
20082605 if (cur_state == IB_QPS_RESET && new_state == IB_QPS_INIT) {
20092606 roce_set_field(context->qpc_bytes_4,
20102607 QP_CONTEXT_QPC_BYTES_4_TRANSPORT_SERVICE_TYPE_M,
@@ -2172,24 +2769,14 @@ static int hns_roce_v1_m_qp(struct ib_qp *ibqp, const struct ib_qp_attr *attr, · 161 unmodified lines
21722769 QP_CONTEXT_QPC_BYTE_32_SIGNALING_TYPE_S,
21732770 hr_qp->sq_signal_bits);
21742771
2175− for (port = 0; port < hr_dev->caps.num_ports; port++) {
2176− smac = (u8 *)hr_dev->dev_addr[port];
2177− dev_dbg(dev, "smac: %2x: %2x: %2x: %2x: %2x: %2x\n",
2178− smac[0], smac[1], smac[2], smac[3], smac[4],
2179− smac[5]);
2180− if ((dmac[0] == smac[0]) && (dmac[1] == smac[1]) &&
2181− (dmac[2] == smac[2]) && (dmac[3] == smac[3]) &&
2182− (dmac[4] == smac[4]) && (dmac[5] == smac[5])) {
2183− roce_set_bit(context->qpc_bytes_32,
2184− QP_CONTEXT_QPC_BYTE_32_LOOPBACK_INDICATOR_S,
2185− 1);
2186− break;
2187− }
2188− }
2189−
2190− if (hr_dev->loop_idc == 0x1)
2772+ port = (attr_mask & IB_QP_PORT) ? (attr->port_num - 1) :
2773+ hr_qp->port;
2774+ smac = (u8 *)hr_dev->dev_addr[port];
2775+ /* when dmac equals smac or loop_idc is 1, it should loopback */
2776+ if (ether_addr_equal_unaligned(dmac, smac) ||
2777+ hr_dev->loop_idc == 0x1)
21912778 roce_set_bit(context->qpc_bytes_32,
2192− QP_CONTEXT_QPC_BYTE_32_LOOPBACK_INDICATOR_S, 1);
2779+ QP_CONTEXT_QPC_BYTE_32_LOOPBACK_INDICATOR_S, 1);
21932780
21942781 roce_set_bit(context->qpc_bytes_32,
21952782 QP_CONTEXT_QPC_BYTE_32_GLOBAL_HEADER_S,
@@ -2509,7 +3096,7 @@ static int hns_roce_v1_m_qp(struct ib_qp *ibqp, const struct ib_qp_attr *attr, · 313 unmodified lines
25093096 /* Every status migrate must change state */
25103097 roce_set_field(context->qpc_bytes_144,
25113098 QP_CONTEXT_QPC_BYTES_144_QP_STATE_M,
2512− QP_CONTEXT_QPC_BYTES_144_QP_STATE_S, attr->qp_state);
3099+ QP_CONTEXT_QPC_BYTES_144_QP_STATE_S, new_state);
25133100
25143101 /* SW pass context to HW */
25153102 ret = hns_roce_v1_qp_modify(hr_dev, &hr_qp->mtt,
@@ -2522,9 +3109,9 @@ static int hns_roce_v1_m_qp(struct ib_qp *ibqp, const struct ib_qp_attr *attr, · 6 unmodified lines
25223109 }
25233110
25243111 /*
2525− * Use rst2init to instead of init2init with drv,
2526− * need to hw to flash RQ HEAD by DB again
2527− */
3112+ * Use rst2init to instead of init2init with drv,
3113+ * need to hw to flash RQ HEAD by DB again
3114+ */
25283115 if (cur_state == IB_QPS_INIT && new_state == IB_QPS_INIT) {
25293116 /* Memory barrier */
25303117 wmb();
@@ -2619,7 +3206,7 @@ static int hns_roce_v1_query_qpc(struct hns_roce_dev *hr_dev, · 88 unmodified lines
26193206
26203207 ret = hns_roce_cmd_mbox(hr_dev, 0, mailbox->dma, hr_qp->qpn, 0,
26213208 HNS_ROCE_CMD_QUERY_QP,
2622− HNS_ROCE_CMD_TIME_CLASS_A);
3209+ HNS_ROCE_CMD_TIMEOUT_MSECS);
26233210 if (!ret)
26243211 memcpy(hr_context, mailbox->buf, sizeof(*hr_context));
26253212 else
@@ -2630,8 +3217,78 @@ static int hns_roce_v1_query_qpc(struct hns_roce_dev *hr_dev, · 4 unmodified lines
26303217 return ret;
26313218 }
26323219
2633−int hns_roce_v1_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
2634− int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr)
3220+static int hns_roce_v1_q_sqp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
3221+ int qp_attr_mask,
3222+ struct ib_qp_init_attr *qp_init_attr)
3223+{
3224+ struct hns_roce_dev *hr_dev = to_hr_dev(ibqp->device);
3225+ struct hns_roce_qp *hr_qp = to_hr_qp(ibqp);
3226+ struct hns_roce_sqp_context context;
3227+ u32 addr;
3228+
3229+ mutex_lock(&hr_qp->mutex);
3230+
3231+ if (hr_qp->state == IB_QPS_RESET) {
3232+ qp_attr->qp_state = IB_QPS_RESET;
3233+ goto done;
3234+ }
3235+
3236+ addr = ROCEE_QP1C_CFG0_0_REG +
3237+ hr_qp->port * sizeof(struct hns_roce_sqp_context);
3238+ context.qp1c_bytes_4 = roce_read(hr_dev, addr);
3239+ context.sq_rq_bt_l = roce_read(hr_dev, addr + 1);
3240+ context.qp1c_bytes_12 = roce_read(hr_dev, addr + 2);
3241+ context.qp1c_bytes_16 = roce_read(hr_dev, addr + 3);
3242+ context.qp1c_bytes_20 = roce_read(hr_dev, addr + 4);
3243+ context.cur_rq_wqe_ba_l = roce_read(hr_dev, addr + 5);
3244+ context.qp1c_bytes_28 = roce_read(hr_dev, addr + 6);
3245+ context.qp1c_bytes_32 = roce_read(hr_dev, addr + 7);
3246+ context.cur_sq_wqe_ba_l = roce_read(hr_dev, addr + 8);
3247+ context.qp1c_bytes_40 = roce_read(hr_dev, addr + 9);
3248+
3249+ hr_qp->state = roce_get_field(context.qp1c_bytes_4,
3250+ QP1C_BYTES_4_QP_STATE_M,
3251+ QP1C_BYTES_4_QP_STATE_S);
3252+ qp_attr->qp_state = hr_qp->state;
3253+ qp_attr->path_mtu = IB_MTU_256;
3254+ qp_attr->path_mig_state = IB_MIG_ARMED;
3255+ qp_attr->qkey = QKEY_VAL;
3256+ qp_attr->rq_psn = 0;
3257+ qp_attr->sq_psn = 0;
3258+ qp_attr->dest_qp_num = 1;
3259+ qp_attr->qp_access_flags = 6;
3260+
3261+ qp_attr->pkey_index = roce_get_field(context.qp1c_bytes_20,
3262+ QP1C_BYTES_20_PKEY_IDX_M,
3263+ QP1C_BYTES_20_PKEY_IDX_S);
3264+ qp_attr->port_num = hr_qp->port + 1;
3265+ qp_attr->sq_draining = 0;
3266+ qp_attr->max_rd_atomic = 0;
3267+ qp_attr->max_dest_rd_atomic = 0;
3268+ qp_attr->min_rnr_timer = 0;
3269+ qp_attr->timeout = 0;
3270+ qp_attr->retry_cnt = 0;
3271+ qp_attr->rnr_retry = 0;
3272+ qp_attr->alt_timeout = 0;
3273+
3274+done:
3275+ qp_attr->cur_qp_state = qp_attr->qp_state;
3276+ qp_attr->cap.max_recv_wr = hr_qp->rq.wqe_cnt;
3277+ qp_attr->cap.max_recv_sge = hr_qp->rq.max_gs;
3278+ qp_attr->cap.max_send_wr = hr_qp->sq.wqe_cnt;
3279+ qp_attr->cap.max_send_sge = hr_qp->sq.max_gs;
3280+ qp_attr->cap.max_inline_data = 0;
3281+ qp_init_attr->cap = qp_attr->cap;
3282+ qp_init_attr->create_flags = 0;
3283+
3284+ mutex_unlock(&hr_qp->mutex);
3285+
3286+ return 0;
3287+}
3288+
3289+static int hns_roce_v1_q_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
3290+ int qp_attr_mask,
3291+ struct ib_qp_init_attr *qp_init_attr)
26353292 {
26363293 struct hns_roce_dev *hr_dev = to_hr_dev(ibqp->device);
26373294 struct hns_roce_qp *hr_qp = to_hr_qp(ibqp);
@@ -2725,9 +3382,7 @@ int hns_roce_v1_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr, · 87 unmodified lines
27253382 qp_attr->pkey_index = roce_get_field(context->qpc_bytes_12,
27263383 QP_CONTEXT_QPC_BYTES_12_P_KEY_INDEX_M,
27273384 QP_CONTEXT_QPC_BYTES_12_P_KEY_INDEX_S);
2728− qp_attr->port_num = (u8)roce_get_field(context->qpc_bytes_156,
2729− QP_CONTEXT_QPC_BYTES_156_PORT_NUM_M,
2730− QP_CONTEXT_QPC_BYTES_156_PORT_NUM_S) + 1;
3385+ qp_attr->port_num = hr_qp->port + 1;
27313386 qp_attr->sq_draining = 0;
27323387 qp_attr->max_rd_atomic = roce_get_field(context->qpc_bytes_156,
27333388 QP_CONTEXT_QPC_BYTES_156_INITIATOR_DEPTH_M,
@@ -2767,134 +3422,397 @@ out: · 33 unmodified lines
27673422 return ret;
27683423 }
27693424
2770−static void hns_roce_v1_destroy_qp_common(struct hns_roce_dev *hr_dev,
2771− struct hns_roce_qp *hr_qp,
2772− int is_user)
3425+int hns_roce_v1_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
3426+ int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr)
3427+{
3428+ struct hns_roce_qp *hr_qp = to_hr_qp(ibqp);
3429+
3430+ return hr_qp->doorbell_qpn <= 1 ?
3431+ hns_roce_v1_q_sqp(ibqp, qp_attr, qp_attr_mask, qp_init_attr) :
3432+ hns_roce_v1_q_qp(ibqp, qp_attr, qp_attr_mask, qp_init_attr);
3433+}
3434+
3435+static int check_qp_db_process_status(struct hns_roce_dev *hr_dev,
3436+ struct hns_roce_qp *hr_qp,
3437+ u32 sdb_issue_ptr,
3438+ u32 *sdb_inv_cnt,
3439+ u32 *wait_stage)
27733440 {
2774− u32 sdbinvcnt;
2775− unsigned long end = 0;
2776− u32 sdbinvcnt_val;
2777− u32 sdbsendptr_val;
2778− u32 sdbisusepr_val;
2779− struct hns_roce_cq *send_cq, *recv_cq;
27803441 struct device *dev = &hr_dev->pdev->dev;
3442+ u32 sdb_retry_cnt, old_retry;
3443+ u32 sdb_send_ptr, old_send;
3444+ u32 success_flags = 0;
3445+ u32 cur_cnt, old_cnt;
3446+ unsigned long end;
3447+ u32 send_ptr;
3448+ u32 inv_cnt;
3449+ u32 tsp_st;
3450+
3451+ if (*wait_stage > HNS_ROCE_V1_DB_STAGE2 ||
3452+ *wait_stage < HNS_ROCE_V1_DB_STAGE1) {
3453+ dev_err(dev, "QP(0x%lx) db status wait stage(%d) error!\n",
3454+ hr_qp->qpn, *wait_stage);
3455+ return -EINVAL;
3456+ }
27813457
2782− if (hr_qp->ibqp.qp_type == IB_QPT_RC) {
2783− if (hr_qp->state != IB_QPS_RESET) {
2784− /*
2785− * Set qp to ERR,
2786− * waiting for hw complete processing all dbs
2787− */
2788− if (hns_roce_v1_qp_modify(hr_dev, NULL,
2789− to_hns_roce_state(
2790− (enum ib_qp_state)hr_qp->state),
2791− HNS_ROCE_QP_STATE_ERR, NULL,
2792− hr_qp))
2793− dev_err(dev, "modify QP %06lx to ERR failed.\n",
2794− hr_qp->qpn);
2795−
2796− /* Record issued doorbell */
2797− sdbisusepr_val = roce_read(hr_dev,
2798− ROCEE_SDB_ISSUE_PTR_REG);
2799− /*
2800− * Query db process status,
2801− * until hw process completely
2802− */
2803− end = msecs_to_jiffies(
2804− HNS_ROCE_QP_DESTROY_TIMEOUT_MSECS) + jiffies;
2805− do {
2806− sdbsendptr_val = roce_read(hr_dev,
3458+ /* Calculate the total timeout for the entire verification process */
3459+ end = msecs_to_jiffies(HNS_ROCE_V1_CHECK_DB_TIMEOUT_MSECS) + jiffies;
3460+
3461+ if (*wait_stage == HNS_ROCE_V1_DB_STAGE1) {
3462+ /* Query db process status, until hw process completely */
3463+ sdb_send_ptr = roce_read(hr_dev, ROCEE_SDB_SEND_PTR_REG);
3464+ while (roce_hw_index_cmp_lt(sdb_send_ptr, sdb_issue_ptr,
3465+ ROCEE_SDB_PTR_CMP_BITS)) {
3466+ if (!time_before(jiffies, end)) {
3467+ dev_dbg(dev, "QP(0x%lx) db process stage1 timeout. issue 0x%x send 0x%x.\n",
3468+ hr_qp->qpn, sdb_issue_ptr,
3469+ sdb_send_ptr);
3470+ return 0;
3471+ }
3472+
3473+ msleep(HNS_ROCE_V1_CHECK_DB_SLEEP_MSECS);
3474+ sdb_send_ptr = roce_read(hr_dev,
28073475 ROCEE_SDB_SEND_PTR_REG);
2808− if (!time_before(jiffies, end)) {
2809− dev_err(dev, "destroy qp(0x%lx) timeout!!!",
2810− hr_qp->qpn);
2811− break;
2812− }
2813− } while ((short)(roce_get_field(sdbsendptr_val,
2814− ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
2815− ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S) -
2816− roce_get_field(sdbisusepr_val,
2817− ROCEE_SDB_ISSUE_PTR_SDB_ISSUE_PTR_M,
2818− ROCEE_SDB_ISSUE_PTR_SDB_ISSUE_PTR_S)
2819− ) < 0);
3476+ }
28203477
2821− /* Get list pointer */
2822− sdbinvcnt = roce_read(hr_dev, ROCEE_SDB_INV_CNT_REG);
3478+ if (roce_get_field(sdb_issue_ptr,
3479+ ROCEE_SDB_ISSUE_PTR_SDB_ISSUE_PTR_M,
3480+ ROCEE_SDB_ISSUE_PTR_SDB_ISSUE_PTR_S) ==
3481+ roce_get_field(sdb_send_ptr,
3482+ ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
3483+ ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S)) {
3484+ old_send = roce_read(hr_dev, ROCEE_SDB_SEND_PTR_REG);
3485+ old_retry = roce_read(hr_dev, ROCEE_SDB_RETRY_CNT_REG);
28233486
2824− /* Query db's list status, until hw reversal */
28253487 do {
2826− sdbinvcnt_val = roce_read(hr_dev,
2827− ROCEE_SDB_INV_CNT_REG);
3488+ tsp_st = roce_read(hr_dev, ROCEE_TSP_BP_ST_REG);
3489+ if (roce_get_bit(tsp_st,
3490+ ROCEE_TSP_BP_ST_QH_FIFO_ENTRY_S) == 1) {
3491+ *wait_stage = HNS_ROCE_V1_DB_WAIT_OK;
3492+ return 0;
3493+ }
3494+
28283495 if (!time_before(jiffies, end)) {
2829− dev_err(dev, "destroy qp(0x%lx) timeout!!!",
2830− hr_qp->qpn);
2831− dev_err(dev, "SdbInvCnt = 0x%x\n",
2832− sdbinvcnt_val);
2833− break;
3496+ dev_dbg(dev, "QP(0x%lx) db process stage1 timeout when send ptr equals issue ptr.\n"
3497+ "issue 0x%x send 0x%x.\n",
3498+ hr_qp->qpn, sdb_issue_ptr,
3499+ sdb_send_ptr);
3500+ return 0;
3501+ }
3502+
3503+ msleep(HNS_ROCE_V1_CHECK_DB_SLEEP_MSECS);
3504+
3505+ sdb_send_ptr = roce_read(hr_dev,
3506+ ROCEE_SDB_SEND_PTR_REG);
3507+ sdb_retry_cnt = roce_read(hr_dev,
3508+ ROCEE_SDB_RETRY_CNT_REG);
3509+ cur_cnt = roce_get_field(sdb_send_ptr,
3510+ ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
3511+ ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S) +
3512+ roce_get_field(sdb_retry_cnt,
3513+ ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_M,
3514+ ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_S);
3515+ if (!roce_get_bit(tsp_st,
3516+ ROCEE_CNT_CLR_CE_CNT_CLR_CE_S)) {
3517+ old_cnt = roce_get_field(old_send,
3518+ ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
3519+ ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S) +
3520+ roce_get_field(old_retry,
3521+ ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_M,
3522+ ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_S);
3523+ if (cur_cnt - old_cnt > SDB_ST_CMP_VAL)
3524+ success_flags = 1;
3525+ } else {
3526+ old_cnt = roce_get_field(old_send,
3527+ ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
3528+ ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S);
3529+ if (cur_cnt - old_cnt > SDB_ST_CMP_VAL)
3530+ success_flags = 1;
3531+ else {
3532+ send_ptr = roce_get_field(old_send,
3533+ ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
3534+ ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S) +
3535+ roce_get_field(sdb_retry_cnt,
3536+ ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_M,
3537+ ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_S);
3538+ roce_set_field(old_send,
3539+ ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
3540+ ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S,
3541+ send_ptr);
3542+ }
28343543 }
2835− } while ((short)(roce_get_field(sdbinvcnt_val,
2836− ROCEE_SDB_INV_CNT_SDB_INV_CNT_M,
2837− ROCEE_SDB_INV_CNT_SDB_INV_CNT_S) -
2838− (sdbinvcnt + SDB_INV_CNT_OFFSET)) < 0);
2839−
2840− /* Modify qp to reset before destroying qp */
2841− if (hns_roce_v1_qp_modify(hr_dev, NULL,
2842− to_hns_roce_state(
2843− (enum ib_qp_state)hr_qp->state),
2844− HNS_ROCE_QP_STATE_RST, NULL, hr_qp))
2845− dev_err(dev, "modify QP %06lx to RESET failed.\n",
2846− hr_qp->qpn);
3544+ } while (!success_flags);
28473545 }
3546+
3547+ *wait_stage = HNS_ROCE_V1_DB_STAGE2;
3548+
3549+ /* Get list pointer */
3550+ *sdb_inv_cnt = roce_read(hr_dev, ROCEE_SDB_INV_CNT_REG);
3551+ dev_dbg(dev, "QP(0x%lx) db process stage2. inv cnt = 0x%x.\n",
3552+ hr_qp->qpn, *sdb_inv_cnt);
3553+ }
3554+
3555+ if (*wait_stage == HNS_ROCE_V1_DB_STAGE2) {
3556+ /* Query db's list status, until hw reversal */
3557+ inv_cnt = roce_read(hr_dev, ROCEE_SDB_INV_CNT_REG);
3558+ while (roce_hw_index_cmp_lt(inv_cnt,
3559+ *sdb_inv_cnt + SDB_INV_CNT_OFFSET,
3560+ ROCEE_SDB_CNT_CMP_BITS)) {
3561+ if (!time_before(jiffies, end)) {
3562+ dev_dbg(dev, "QP(0x%lx) db process stage2 timeout. inv cnt 0x%x.\n",
3563+ hr_qp->qpn, inv_cnt);
3564+ return 0;
3565+ }
3566+
3567+ msleep(HNS_ROCE_V1_CHECK_DB_SLEEP_MSECS);
3568+ inv_cnt = roce_read(hr_dev, ROCEE_SDB_INV_CNT_REG);
3569+ }
3570+
3571+ *wait_stage = HNS_ROCE_V1_DB_WAIT_OK;
3572+ }
3573+
3574+ return 0;
3575+}
3576+
3577+static int check_qp_reset_state(struct hns_roce_dev *hr_dev,
3578+ struct hns_roce_qp *hr_qp,
3579+ struct hns_roce_qp_work *qp_work_entry,
3580+ int *is_timeout)
3581+{
3582+ struct device *dev = &hr_dev->pdev->dev;
3583+ u32 sdb_issue_ptr;
3584+ int ret;
3585+
3586+ if (hr_qp->state != IB_QPS_RESET) {
3587+ /* Set qp to ERR, waiting for hw complete processing all dbs */
3588+ ret = hns_roce_v1_modify_qp(&hr_qp->ibqp, NULL, 0, hr_qp->state,
3589+ IB_QPS_ERR);
3590+ if (ret) {
3591+ dev_err(dev, "Modify QP(0x%lx) to ERR failed!\n",
3592+ hr_qp->qpn);
3593+ return ret;
3594+ }
3595+
3596+ /* Record issued doorbell */
3597+ sdb_issue_ptr = roce_read(hr_dev, ROCEE_SDB_ISSUE_PTR_REG);
3598+ qp_work_entry->sdb_issue_ptr = sdb_issue_ptr;
3599+ qp_work_entry->db_wait_stage = HNS_ROCE_V1_DB_STAGE1;
3600+
3601+ /* Query db process status, until hw process completely */
3602+ ret = check_qp_db_process_status(hr_dev, hr_qp, sdb_issue_ptr,
3603+ &qp_work_entry->sdb_inv_cnt,
3604+ &qp_work_entry->db_wait_stage);
3605+ if (ret) {
3606+ dev_err(dev, "Check QP(0x%lx) db process status failed!\n",
3607+ hr_qp->qpn);
3608+ return ret;
3609+ }
3610+
3611+ if (qp_work_entry->db_wait_stage != HNS_ROCE_V1_DB_WAIT_OK) {
3612+ qp_work_entry->sche_cnt = 0;
3613+ *is_timeout = 1;
3614+ return 0;
3615+ }
3616+
3617+ /* Modify qp to reset before destroying qp */
3618+ ret = hns_roce_v1_modify_qp(&hr_qp->ibqp, NULL, 0, hr_qp->state,
3619+ IB_QPS_RESET);
3620+ if (ret) {
3621+ dev_err(dev, "Modify QP(0x%lx) to RST failed!\n",
3622+ hr_qp->qpn);
3623+ return ret;
3624+ }
3625+ }
3626+
3627+ return 0;
3628+}
3629+
3630+static void hns_roce_v1_destroy_qp_work_fn(struct work_struct *work)
3631+{
3632+ struct hns_roce_qp_work *qp_work_entry;
3633+ struct hns_roce_v1_priv *priv;
3634+ struct hns_roce_dev *hr_dev;
3635+ struct hns_roce_qp *hr_qp;
3636+ struct device *dev;
3637+ int ret;
3638+
3639+ qp_work_entry = container_of(work, struct hns_roce_qp_work, work);
3640+ hr_dev = to_hr_dev(qp_work_entry->ib_dev);
3641+ dev = &hr_dev->pdev->dev;
3642+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
3643+ hr_qp = qp_work_entry->qp;
3644+
3645+ dev_dbg(dev, "Schedule destroy QP(0x%lx) work.\n", hr_qp->qpn);
3646+
3647+ qp_work_entry->sche_cnt++;
3648+
3649+ /* Query db process status, until hw process completely */
3650+ ret = check_qp_db_process_status(hr_dev, hr_qp,
3651+ qp_work_entry->sdb_issue_ptr,
3652+ &qp_work_entry->sdb_inv_cnt,
3653+ &qp_work_entry->db_wait_stage);
3654+ if (ret) {
3655+ dev_err(dev, "Check QP(0x%lx) db process status failed!\n",
3656+ hr_qp->qpn);
3657+ return;
3658+ }
3659+
3660+ if (qp_work_entry->db_wait_stage != HNS_ROCE_V1_DB_WAIT_OK &&
3661+ priv->des_qp.requeue_flag) {
3662+ queue_work(priv->des_qp.qp_wq, work);
3663+ return;
3664+ }
3665+
3666+ /* Modify qp to reset before destroying qp */
3667+ ret = hns_roce_v1_modify_qp(&hr_qp->ibqp, NULL, 0, hr_qp->state,
3668+ IB_QPS_RESET);
3669+ if (ret) {
3670+ dev_err(dev, "Modify QP(0x%lx) to RST failed!\n", hr_qp->qpn);
3671+ return;
3672+ }
3673+
3674+ hns_roce_qp_remove(hr_dev, hr_qp);
3675+ hns_roce_qp_free(hr_dev, hr_qp);
3676+
3677+ if (hr_qp->ibqp.qp_type == IB_QPT_RC) {
3678+ /* RC QP, release QPN */
3679+ hns_roce_release_range_qp(hr_dev, hr_qp->qpn, 1);
3680+ kfree(hr_qp);
3681+ } else
3682+ kfree(hr_to_hr_sqp(hr_qp));
3683+
3684+ kfree(qp_work_entry);
3685+
3686+ dev_dbg(dev, "Accomplished destroy QP(0x%lx) work.\n", hr_qp->qpn);
3687+}
3688+
3689+int hns_roce_v1_destroy_qp(struct ib_qp *ibqp)
3690+{
3691+ struct hns_roce_dev *hr_dev = to_hr_dev(ibqp->device);
3692+ struct hns_roce_qp *hr_qp = to_hr_qp(ibqp);
3693+ struct device *dev = &hr_dev->pdev->dev;
3694+ struct hns_roce_qp_work qp_work_entry;
3695+ struct hns_roce_qp_work *qp_work;
3696+ struct hns_roce_v1_priv *priv;
3697+ struct hns_roce_cq *send_cq, *recv_cq;
3698+ int is_user = !!ibqp->pd->uobject;
3699+ int is_timeout = 0;
3700+ int ret;
3701+
3702+ ret = check_qp_reset_state(hr_dev, hr_qp, &qp_work_entry, &is_timeout);
3703+ if (ret) {
3704+ dev_err(dev, "QP reset state check failed(%d)!\n", ret);
3705+ return ret;
28483706 }
28493707
28503708 send_cq = to_hr_cq(hr_qp->ibqp.send_cq);
28513709 recv_cq = to_hr_cq(hr_qp->ibqp.recv_cq);
28523710
28533711 hns_roce_lock_cqs(send_cq, recv_cq);
2854−
28553712 if (!is_user) {
28563713 __hns_roce_v1_cq_clean(recv_cq, hr_qp->qpn, hr_qp->ibqp.srq ?
28573714 to_hr_srq(hr_qp->ibqp.srq) : NULL);
28583715 if (send_cq != recv_cq)
28593716 __hns_roce_v1_cq_clean(send_cq, hr_qp->qpn, NULL);
28603717 }
2861−
2862− hns_roce_qp_remove(hr_dev, hr_qp);
2863−
28643718 hns_roce_unlock_cqs(send_cq, recv_cq);
28653719
2866− hns_roce_qp_free(hr_dev, hr_qp);
3720+ if (!is_timeout) {
3721+ hns_roce_qp_remove(hr_dev, hr_qp);
3722+ hns_roce_qp_free(hr_dev, hr_qp);
28673723
2868− /* Not special_QP, free their QPN */
2869− if ((hr_qp->ibqp.qp_type == IB_QPT_RC) ||
2870− (hr_qp->ibqp.qp_type == IB_QPT_UC) ||
2871− (hr_qp->ibqp.qp_type == IB_QPT_UD))
2872− hns_roce_release_range_qp(hr_dev, hr_qp->qpn, 1);
3724+ /* RC QP, release QPN */
3725+ if (hr_qp->ibqp.qp_type == IB_QPT_RC)
3726+ hns_roce_release_range_qp(hr_dev, hr_qp->qpn, 1);
3727+ }
28733728
28743729 hns_roce_mtt_cleanup(hr_dev, &hr_qp->mtt);
28753730
2876− if (is_user) {
3731+ if (is_user)
28773732 ib_umem_release(hr_qp->umem);
2878− } else {
3733+ else {
28793734 kfree(hr_qp->sq.wrid);
28803735 kfree(hr_qp->rq.wrid);
3736+
28813737 hns_roce_buf_free(hr_dev, hr_qp->buff_size, &hr_qp->hr_buf);
28823738 }
3739+
3740+ if (!is_timeout) {
3741+ if (hr_qp->ibqp.qp_type == IB_QPT_RC)
3742+ kfree(hr_qp);
3743+ else
3744+ kfree(hr_to_hr_sqp(hr_qp));
3745+ } else {
3746+ qp_work = kzalloc(sizeof(*qp_work), GFP_KERNEL);
3747+ if (!qp_work)
3748+ return -ENOMEM;
3749+
3750+ INIT_WORK(&qp_work->work, hns_roce_v1_destroy_qp_work_fn);
3751+ qp_work->ib_dev = &hr_dev->ib_dev;
3752+ qp_work->qp = hr_qp;
3753+ qp_work->db_wait_stage = qp_work_entry.db_wait_stage;
3754+ qp_work->sdb_issue_ptr = qp_work_entry.sdb_issue_ptr;
3755+ qp_work->sdb_inv_cnt = qp_work_entry.sdb_inv_cnt;
3756+ qp_work->sche_cnt = qp_work_entry.sche_cnt;
3757+
3758+ priv = (struct hns_roce_v1_priv *)hr_dev->hw->priv;
3759+ queue_work(priv->des_qp.qp_wq, &qp_work->work);
3760+ dev_dbg(dev, "Begin destroy QP(0x%lx) work.\n", hr_qp->qpn);
3761+ }
3762+
3763+ return 0;
28833764 }
28843765
2885−int hns_roce_v1_destroy_qp(struct ib_qp *ibqp)
3766+int hns_roce_v1_destroy_cq(struct ib_cq *ibcq)
28863767 {
2887− struct hns_roce_dev *hr_dev = to_hr_dev(ibqp->device);
2888− struct hns_roce_qp *hr_qp = to_hr_qp(ibqp);
3768+ struct hns_roce_dev *hr_dev = to_hr_dev(ibcq->device);
3769+ struct hns_roce_cq *hr_cq = to_hr_cq(ibcq);
3770+ struct device *dev = &hr_dev->pdev->dev;
3771+ u32 cqe_cnt_ori;
3772+ u32 cqe_cnt_cur;
3773+ u32 cq_buf_size;
3774+ int wait_time = 0;
3775+ int ret = 0;
28893776
2890− hns_roce_v1_destroy_qp_common(hr_dev, hr_qp, !!ibqp->pd->uobject);
3777+ hns_roce_free_cq(hr_dev, hr_cq);
28913778
2892− if (hr_qp->ibqp.qp_type == IB_QPT_GSI)
2893− kfree(hr_to_hr_sqp(hr_qp));
2894− else
2895− kfree(hr_qp);
3779+ /*
3780+ * Before freeing cq buffer, we need to ensure that the outstanding CQE
3781+ * have been written by checking the CQE counter.
3782+ */
3783+ cqe_cnt_ori = roce_read(hr_dev, ROCEE_SCAEP_WR_CQE_CNT);
3784+ while (1) {
3785+ if (roce_read(hr_dev, ROCEE_CAEP_CQE_WCMD_EMPTY) &
3786+ HNS_ROCE_CQE_WCMD_EMPTY_BIT)
3787+ break;
28963788
2897− return 0;
3789+ cqe_cnt_cur = roce_read(hr_dev, ROCEE_SCAEP_WR_CQE_CNT);
3790+ if ((cqe_cnt_cur - cqe_cnt_ori) >= HNS_ROCE_MIN_CQE_CNT)
3791+ break;
3792+
3793+ msleep(HNS_ROCE_EACH_FREE_CQ_WAIT_MSECS);
3794+ if (wait_time > HNS_ROCE_MAX_FREE_CQ_WAIT_CNT) {
3795+ dev_warn(dev, "Destroy cq 0x%lx timeout!\n",
3796+ hr_cq->cqn);
3797+ ret = -ETIMEDOUT;
3798+ break;
3799+ }
3800+ wait_time++;
3801+ }
3802+
3803+ hns_roce_mtt_cleanup(hr_dev, &hr_cq->hr_buf.hr_mtt);
3804+
3805+ if (ibcq->uobject)
3806+ ib_umem_release(hr_cq->umem);
3807+ else {
3808+ /* Free the buff of stored cq */
3809+ cq_buf_size = (ibcq->cqe + 1) * hr_dev->caps.cq_entry_sz;
3810+ hns_roce_buf_free(hr_dev, cq_buf_size, &hr_cq->hr_buf.hr_buf);
3811+ }
3812+
3813+ kfree(hr_cq);
3814+
3815+ return ret;
28983816 }
28993817
29003818 struct hns_roce_v1_priv hr_v1_priv;
@@ -2917,5 +3835,7 @@ struct hns_roce_hw hns_roce_hw_v1 = { · 16 unmodified lines
29173835 .post_recv = hns_roce_v1_post_recv,
29183836 .req_notify_cq = hns_roce_v1_req_notify_cq,
29193837 .poll_cq = hns_roce_v1_poll_cq,
3838+ .dereg_mr = hns_roce_v1_dereg_mr,
3839+ .destroy_cq = hns_roce_v1_destroy_cq,
29203840 .priv = &hr_v1_priv,
29213841 };
drivers/infiniband/hw/hns/hns_roce_hw_v1.h+73−1
@@ -58,6 +58,7 @@ · 57 unmodified lines
5858 #define HNS_ROCE_V1_PHY_UAR_NUM 8
5959
6060 #define HNS_ROCE_V1_GID_NUM 16
61+#define HNS_ROCE_V1_RESV_QP 8
6162
6263 #define HNS_ROCE_V1_NUM_COMP_EQE 0x8000
6364 #define HNS_ROCE_V1_NUM_ASYNC_EQE 0x400
@@ -102,8 +103,22 @@ · 38 unmodified lines
102103 #define HNS_ROCE_V1_EXT_ODB_ALFUL \
103104 (HNS_ROCE_V1_EXT_ODB_DEPTH - HNS_ROCE_V1_DB_RSVD)
104105
106+#define HNS_ROCE_V1_DB_WAIT_OK 0
107+#define HNS_ROCE_V1_DB_STAGE1 1
108+#define HNS_ROCE_V1_DB_STAGE2 2
109+#define HNS_ROCE_V1_CHECK_DB_TIMEOUT_MSECS 10000
110+#define HNS_ROCE_V1_CHECK_DB_SLEEP_MSECS 20
111+#define HNS_ROCE_V1_FREE_MR_TIMEOUT_MSECS 50000
112+#define HNS_ROCE_V1_RECREATE_LP_QP_TIMEOUT_MSECS 10000
113+#define HNS_ROCE_V1_FREE_MR_WAIT_VALUE 5
114+#define HNS_ROCE_V1_RECREATE_LP_QP_WAIT_VALUE 20
115+
105116 #define HNS_ROCE_BT_RSV_BUF_SIZE (1 << 17)
106117
118+#define HNS_ROCE_V1_TPTR_ENTRY_SIZE 2
119+#define HNS_ROCE_V1_TPTR_BUF_SIZE \
120+ (HNS_ROCE_V1_TPTR_ENTRY_SIZE * HNS_ROCE_V1_MAX_CQ_NUM)
121+
107122 #define HNS_ROCE_ODB_POLL_MODE 0
108123
109124 #define HNS_ROCE_SDB_NORMAL_MODE 0
@@ -140,6 +155,7 @@ · 30 unmodified lines
140155 #define SQ_PSN_SHIFT 8
141156 #define QKEY_VAL 0x80010000
142157 #define SDB_INV_CNT_OFFSET 8
158+#define SDB_ST_CMP_VAL 8
143159
144160 struct hns_roce_cq_context {
145161 u32 cqc_byte_4;
@@ -436,6 +452,8 @@ struct hns_roce_ud_send_wqe { · 290 unmodified lines
436452 #define UD_SEND_WQE_U32_8_DMAC_5_M \
437453 (((1UL << 8) - 1) << UD_SEND_WQE_U32_8_DMAC_5_S)
438454
455+#define UD_SEND_WQE_U32_8_LOOPBACK_INDICATOR_S 22
456+
439457 #define UD_SEND_WQE_U32_8_OPERATION_TYPE_S 16
440458 #define UD_SEND_WQE_U32_8_OPERATION_TYPE_M \
441459 (((1UL << 4) - 1) << UD_SEND_WQE_U32_8_OPERATION_TYPE_S)
@@ -480,13 +498,17 @@ struct hns_roce_sqp_context { · 38 unmodified lines
480498 u32 qp1c_bytes_12;
481499 u32 qp1c_bytes_16;
482500 u32 qp1c_bytes_20;
483− u32 qp1c_bytes_28;
484501 u32 cur_rq_wqe_ba_l;
502+ u32 qp1c_bytes_28;
485503 u32 qp1c_bytes_32;
486504 u32 cur_sq_wqe_ba_l;
487505 u32 qp1c_bytes_40;
488506 };
489507
508+#define QP1C_BYTES_4_QP_STATE_S 0
509+#define QP1C_BYTES_4_QP_STATE_M \
510+ (((1UL << 3) - 1) << QP1C_BYTES_4_QP_STATE_S)
511+
490512 #define QP1C_BYTES_4_SQ_WQE_SHIFT_S 8
491513 #define QP1C_BYTES_4_SQ_WQE_SHIFT_M \
492514 (((1UL << 4) - 1) << QP1C_BYTES_4_SQ_WQE_SHIFT_S)
@@ -952,6 +974,10 @@ struct hns_roce_sq_db { · 459 unmodified lines
952974 #define SQ_DOORBELL_U32_4_SQ_HEAD_M \
953975 (((1UL << 15) - 1) << SQ_DOORBELL_U32_4_SQ_HEAD_S)
954976
977+#define SQ_DOORBELL_U32_4_SL_S 16
978+#define SQ_DOORBELL_U32_4_SL_M \
979+ (((1UL << 2) - 1) << SQ_DOORBELL_U32_4_SL_S)
980+
955981 #define SQ_DOORBELL_U32_4_PORT_S 18
956982 #define SQ_DOORBELL_U32_4_PORT_M (((1UL << 3) - 1) << SQ_DOORBELL_U32_4_PORT_S)
957983
@@ -979,12 +1005,58 @@ struct hns_roce_bt_table { · 21 unmodified lines
9791005 struct hns_roce_buf_list cqc_buf;
9801006 };
9811007
1008+struct hns_roce_tptr_table {
1009+ struct hns_roce_buf_list tptr_buf;
1010+};
1011+
1012+struct hns_roce_qp_work {
1013+ struct work_struct work;
1014+ struct ib_device *ib_dev;
1015+ struct hns_roce_qp *qp;
1016+ u32 db_wait_stage;
1017+ u32 sdb_issue_ptr;
1018+ u32 sdb_inv_cnt;
1019+ u32 sche_cnt;
1020+};
1021+
1022+struct hns_roce_des_qp {
1023+ struct workqueue_struct *qp_wq;
1024+ int requeue_flag;
1025+};
1026+
1027+struct hns_roce_mr_free_work {
1028+ struct work_struct work;
1029+ struct ib_device *ib_dev;
1030+ struct completion *comp;
1031+ int comp_flag;
1032+ void *mr;
1033+};
1034+
1035+struct hns_roce_recreate_lp_qp_work {
1036+ struct work_struct work;
1037+ struct ib_device *ib_dev;
1038+ struct completion *comp;
1039+ int comp_flag;
1040+};
1041+
1042+struct hns_roce_free_mr {
1043+ struct workqueue_struct *free_mr_wq;
1044+ struct hns_roce_qp *mr_free_qp[HNS_ROCE_V1_RESV_QP];
1045+ struct hns_roce_cq *mr_free_cq;
1046+ struct hns_roce_pd *mr_free_pd;
1047+};
1048+
9821049 struct hns_roce_v1_priv {
9831050 struct hns_roce_db_table db_table;
9841051 struct hns_roce_raq_table raq_table;
9851052 struct hns_roce_bt_table bt_table;
1053+ struct hns_roce_tptr_table tptr_table;
1054+ struct hns_roce_des_qp des_qp;
1055+ struct hns_roce_free_mr free_mr;
9861056 };
9871057
9881058 int hns_dsaf_roce_reset(struct fwnode_handle *dsaf_fwnode, bool dereset);
1059+int hns_roce_v1_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc);
1060+int hns_roce_v1_destroy_qp(struct ib_qp *ibqp);
9891061
9901062 #endif
drivers/infiniband/hw/hns/hns_roce_main.c+73−256
@@ -35,51 +35,12 @@ · 34 unmodified lines
3535 #include <rdma/ib_addr.h>
3636 #include <rdma/ib_smi.h>
3737 #include <rdma/ib_user_verbs.h>
38+#include <rdma/ib_cache.h>
3839 #include "hns_roce_common.h"
3940 #include "hns_roce_device.h"
40−#include "hns_roce_user.h"
41+#include <rdma/hns-abi.h>
4142 #include "hns_roce_hem.h"
4243
43−/**
44− * hns_roce_addrconf_ifid_eui48 - Get default gid.
45− * @eui: eui.
46− * @vlan_id: gid
47− * @dev: net device
48− * Description:
49− * MAC convert to GID
50− * gid[0..7] = fe80 0000 0000 0000
51− * gid[8] = mac[0] ^ 2
52− * gid[9] = mac[1]
53− * gid[10] = mac[2]
54− * gid[11] = ff (VLAN ID high byte (4 MS bits))
55− * gid[12] = fe (VLAN ID low byte)
56− * gid[13] = mac[3]
57− * gid[14] = mac[4]
58− * gid[15] = mac[5]
59− */
60−static void hns_roce_addrconf_ifid_eui48(u8 *eui, u16 vlan_id,
61− struct net_device *dev)
62−{
63− memcpy(eui, dev->dev_addr, 3);
64− memcpy(eui + 5, dev->dev_addr + 3, 3);
65− if (vlan_id < 0x1000) {
66− eui[3] = vlan_id >> 8;
67− eui[4] = vlan_id & 0xff;
68− } else {
69− eui[3] = 0xff;
70− eui[4] = 0xfe;
71− }
72− eui[0] ^= 2;
73−}
74−
75−static void hns_roce_make_default_gid(struct net_device *dev, union ib_gid *gid)
76−{
77− memset(gid, 0, sizeof(*gid));
78− gid->raw[0] = 0xFE;
79− gid->raw[1] = 0x80;
80− hns_roce_addrconf_ifid_eui48(&gid->raw[8], 0xffff, dev);
81−}
82−
8344 /**
8445 * hns_get_gid_index - Get gid index.
8546 * @hr_dev: pointer to structure hns_roce_dev.
@@ -96,30 +57,6 @@ int hns_get_gid_index(struct hns_roce_dev *hr_dev, u8 port, int gid_index) · 10 unmodified lines
9657 return gid_index * hr_dev->caps.num_ports + port;
9758 }
9859
99−static int hns_roce_set_gid(struct hns_roce_dev *hr_dev, u8 port, int gid_index,
100− union ib_gid *gid)
101−{
102− struct device *dev = &hr_dev->pdev->dev;
103− u8 gid_idx = 0;
104−
105− if (gid_index >= hr_dev->caps.gid_table_len[port]) {
106− dev_err(dev, "gid_index %d illegal, port %d gid range: 0~%d\n",
107− gid_index, port, hr_dev->caps.gid_table_len[port] - 1);
108− return -EINVAL;
109− }
110−
111− gid_idx = hns_get_gid_index(hr_dev, port, gid_index);
112−
113− if (!memcmp(gid, &hr_dev->iboe.gid_table[gid_idx], sizeof(*gid)))
114− return -EINVAL;
115−
116− memcpy(&hr_dev->iboe.gid_table[gid_idx], gid, sizeof(*gid));
117−
118− hr_dev->hw->set_gid(hr_dev, port, gid_index, gid);
119−
120− return 0;
121−}
122−
12360 static void hns_roce_set_mac(struct hns_roce_dev *hr_dev, u8 port, u8 *addr)
12461 {
12562 u8 phy_port;
@@ -135,27 +72,44 @@ static void hns_roce_set_mac(struct hns_roce_dev *hr_dev, u8 port, u8 *addr) · 9 unmodified lines
13572 hr_dev->hw->set_mac(hr_dev, phy_port, addr);
13673 }
13774
138−static void hns_roce_set_mtu(struct hns_roce_dev *hr_dev, u8 port, int mtu)
75+static int hns_roce_add_gid(struct ib_device *device, u8 port_num,
76+ unsigned int index, const union ib_gid *gid,
77+ const struct ib_gid_attr *attr, void **context)
13978 {
140− u8 phy_port = hr_dev->iboe.phy_port[port];
141− enum ib_mtu tmp;
79+ struct hns_roce_dev *hr_dev = to_hr_dev(device);
80+ u8 port = port_num - 1;
81+ unsigned long flags;
82+
83+ if (port >= hr_dev->caps.num_ports)
84+ return -EINVAL;
85+
86+ spin_lock_irqsave(&hr_dev->iboe.lock, flags);
14287
143− tmp = iboe_get_mtu(mtu);
144− if (!tmp)
145− tmp = IB_MTU_256;
88+ hr_dev->hw->set_gid(hr_dev, port, index, (union ib_gid *)gid);
14689
147− hr_dev->hw->set_mtu(hr_dev, phy_port, tmp);
90+ spin_unlock_irqrestore(&hr_dev->iboe.lock, flags);
91+
92+ return 0;
14893 }
14994
150−static void hns_roce_update_gids(struct hns_roce_dev *hr_dev, int port)
95+static int hns_roce_del_gid(struct ib_device *device, u8 port_num,
96+ unsigned int index, void **context)
15197 {
152− struct ib_event event;
98+ struct hns_roce_dev *hr_dev = to_hr_dev(device);
99+ union ib_gid zgid = { {0} };
100+ u8 port = port_num - 1;
101+ unsigned long flags;
102+
103+ if (port >= hr_dev->caps.num_ports)
104+ return -EINVAL;
105+
106+ spin_lock_irqsave(&hr_dev->iboe.lock, flags);
153107
154− /* Refresh gid in ib_cache */
155− event.device = &hr_dev->ib_dev;
156− event.element.port_num = port + 1;
157− event.event = IB_EVENT_GID_CHANGE;
158− ib_dispatch_event(&event);
108+ hr_dev->hw->set_gid(hr_dev, port, index, &zgid);
109+
110+ spin_unlock_irqrestore(&hr_dev->iboe.lock, flags);
111+
112+ return 0;
159113 }
160114
161115 static int handle_en_event(struct hns_roce_dev *hr_dev, u8 port,
@@ -163,9 +117,6 @@ static int handle_en_event(struct hns_roce_dev *hr_dev, u8 port, · 1 unmodified line
163117 {
164118 struct device *dev = &hr_dev->pdev->dev;
165119 struct net_device *netdev;
166− unsigned long flags;
167− union ib_gid gid;
168− int ret = 0;
169120
170121 netdev = hr_dev->iboe.netdevs[port];
171122 if (!netdev) {
@@ -173,7 +124,7 @@ static int handle_en_event(struct hns_roce_dev *hr_dev, u8 port, · 1 unmodified line
173124 return -ENODEV;
174125 }
175126
176− spin_lock_irqsave(&hr_dev->iboe.lock, flags);
127+ spin_lock_bh(&hr_dev->iboe.lock);
177128
178129 switch (event) {
179130 case NETDEV_UP:
@@ -181,23 +132,19 @@ static int handle_en_event(struct hns_roce_dev *hr_dev, u8 port, · 1 unmodified line
181132 case NETDEV_REGISTER:
182133 case NETDEV_CHANGEADDR:
183134 hns_roce_set_mac(hr_dev, port, netdev->dev_addr);
184− hns_roce_make_default_gid(netdev, &gid);
185− ret = hns_roce_set_gid(hr_dev, port, 0, &gid);
186− if (!ret)
187− hns_roce_update_gids(hr_dev, port);
188135 break;
189136 case NETDEV_DOWN:
190137 /*
191− * In v1 engine, only support all ports closed together.
192− */
138+ * In v1 engine, only support all ports closed together.
139+ */
193140 break;
194141 default:
195142 dev_dbg(dev, "NETDEV event = 0x%x!\n", (u32)(event));
196143 break;
197144 }
198145
199− spin_unlock_irqrestore(&hr_dev->iboe.lock, flags);
200− return ret;
146+ spin_unlock_bh(&hr_dev->iboe.lock);
147+ return 0;
201148 }
202149
203150 static int hns_roce_netdev_event(struct notifier_block *self,
@@ -224,118 +171,17 @@ static int hns_roce_netdev_event(struct notifier_block *self, · 20 unmodified lines
224171 return NOTIFY_DONE;
225172 }
226173
227−static void hns_roce_addr_event(int event, struct net_device *event_netdev,
228− struct hns_roce_dev *hr_dev, union ib_gid *gid)
174+static int hns_roce_setup_mtu_mac(struct hns_roce_dev *hr_dev)
229175 {
230− struct hns_roce_ib_iboe *iboe = NULL;
231− int gid_table_len = 0;
232− unsigned long flags;
233− union ib_gid zgid;
234− u8 gid_idx = 0;
235− u8 port = 0;
236− int i = 0;
237− int free;
238− struct net_device *real_dev = rdma_vlan_dev_real_dev(event_netdev) ?
239− rdma_vlan_dev_real_dev(event_netdev) :
240− event_netdev;
241−
242− if (event != NETDEV_UP && event != NETDEV_DOWN)
243− return;
244−
245− iboe = &hr_dev->iboe;
246− while (port < hr_dev->caps.num_ports) {
247− if (real_dev == iboe->netdevs[port])
248− break;
249− port++;
250− }
251−
252− if (port >= hr_dev->caps.num_ports) {
253− dev_dbg(&hr_dev->pdev->dev, "can't find netdev\n");
254− return;
255− }
256−
257− memset(zgid.raw, 0, sizeof(zgid.raw));
258− free = -1;
259− gid_table_len = hr_dev->caps.gid_table_len[port];
260−
261− spin_lock_irqsave(&hr_dev->iboe.lock, flags);
262−
263− for (i = 0; i < gid_table_len; i++) {
264− gid_idx = hns_get_gid_index(hr_dev, port, i);
265− if (!memcmp(gid->raw, iboe->gid_table[gid_idx].raw,
266− sizeof(gid->raw)))
267− break;
268− if (free < 0 && !memcmp(zgid.raw,
269− iboe->gid_table[gid_idx].raw, sizeof(zgid.raw)))
270− free = i;
271− }
272−
273− if (i >= gid_table_len) {
274− if (free < 0) {
275− spin_unlock_irqrestore(&hr_dev->iboe.lock, flags);
276− dev_dbg(&hr_dev->pdev->dev,
277− "gid_index overflow, port(%d)\n", port);
278− return;
279− }
280− if (!hns_roce_set_gid(hr_dev, port, free, gid))
281− hns_roce_update_gids(hr_dev, port);
282− } else if (event == NETDEV_DOWN) {
283− if (!hns_roce_set_gid(hr_dev, port, i, &zgid))
284− hns_roce_update_gids(hr_dev, port);
285− }
286−
287− spin_unlock_irqrestore(&hr_dev->iboe.lock, flags);
288−}
289−
290−static int hns_roce_inet_event(struct notifier_block *self, unsigned long event,
291− void *ptr)
292−{
293− struct in_ifaddr *ifa = ptr;
294− struct hns_roce_dev *hr_dev;
295− struct net_device *dev = ifa->ifa_dev->dev;
296− union ib_gid gid;
297−
298− ipv6_addr_set_v4mapped(ifa->ifa_address, (struct in6_addr *)&gid);
299−
300− hr_dev = container_of(self, struct hns_roce_dev, iboe.nb_inet);
301−
302− hns_roce_addr_event(event, dev, hr_dev, &gid);
303−
304− return NOTIFY_DONE;
305−}
306−
307−static int hns_roce_setup_mtu_gids(struct hns_roce_dev *hr_dev)
308−{
309− struct in_ifaddr *ifa_list = NULL;
310− union ib_gid gid = {{0} };
311− u32 ipaddr = 0;
312− int index = 0;
313− int ret = 0;
314− u8 i = 0;
176+ u8 i;
315177
316178 for (i = 0; i < hr_dev->caps.num_ports; i++) {
317− hns_roce_set_mtu(hr_dev, i,
318− ib_mtu_enum_to_int(hr_dev->caps.max_mtu));
179+ hr_dev->hw->set_mtu(hr_dev, hr_dev->iboe.phy_port[i],
180+ hr_dev->caps.max_mtu);
319181 hns_roce_set_mac(hr_dev, i, hr_dev->iboe.netdevs[i]->dev_addr);
320−
321− if (hr_dev->iboe.netdevs[i]->ip_ptr) {
322− ifa_list = hr_dev->iboe.netdevs[i]->ip_ptr->ifa_list;
323− index = 1;
324− while (ifa_list) {
325− ipaddr = ifa_list->ifa_address;
326− ipv6_addr_set_v4mapped(ipaddr,
327− (struct in6_addr *)&gid);
328− ret = hns_roce_set_gid(hr_dev, i, index, &gid);
329− if (ret)
330− break;
331− index++;
332− ifa_list = ifa_list->ifa_next;
333− }
334− hns_roce_update_gids(hr_dev, i);
335− }
336182 }
337183
338− return ret;
184+ return 0;
339185 }
340186
341187 static int hns_roce_query_device(struct ib_device *ib_dev,
@@ -444,31 +290,6 @@ static enum rdma_link_layer hns_roce_get_link_layer(struct ib_device *device, · 102 unmodified lines
444290 static int hns_roce_query_gid(struct ib_device *ib_dev, u8 port_num, int index,
445291 union ib_gid *gid)
446292 {
447− struct hns_roce_dev *hr_dev = to_hr_dev(ib_dev);
448− struct device *dev = &hr_dev->pdev->dev;
449− u8 gid_idx = 0;
450− u8 port;
451−
452− if (port_num < 1 || port_num > hr_dev->caps.num_ports ||
453− index >= hr_dev->caps.gid_table_len[port_num - 1]) {
454− dev_err(dev,
455− "port_num %d index %d illegal! correct range: port_num 1~%d index 0~%d!\n",
456− port_num, index, hr_dev->caps.num_ports,
457− hr_dev->caps.gid_table_len[port_num - 1] - 1);
458− return -EINVAL;
459− }
460−
461− port = port_num - 1;
462− gid_idx = hns_get_gid_index(hr_dev, port, index);
463− if (gid_idx >= HNS_ROCE_MAX_GID_NUM) {
464− dev_err(dev, "port_num %d index %d illegal! total gid num %d!\n",
465− port_num, index, HNS_ROCE_MAX_GID_NUM);
466− return -EINVAL;
467− }
468−
469− memcpy(gid->raw, hr_dev->iboe.gid_table[gid_idx].raw,
470− HNS_ROCE_GID_SIZE);
471−
472293 return 0;
473294 }
474295
@@ -549,6 +370,8 @@ static int hns_roce_dealloc_ucontext(struct ib_ucontext *ibcontext) · 74 unmodified lines
549370 static int hns_roce_mmap(struct ib_ucontext *context,
550371 struct vm_area_struct *vma)
551372 {
373+ struct hns_roce_dev *hr_dev = to_hr_dev(context->device);
374+
552375 if (((vma->vm_end - vma->vm_start) % PAGE_SIZE) != 0)
553376 return -EINVAL;
554377
@@ -558,10 +381,15 @@ static int hns_roce_mmap(struct ib_ucontext *context, · 3 unmodified lines
558381 to_hr_ucontext(context)->uar.pfn,
559382 PAGE_SIZE, vma->vm_page_prot))
560383 return -EAGAIN;
561−
562− } else {
384+ } else if (vma->vm_pgoff == 1 && hr_dev->hw_rev == HNS_ROCE_HW_VER1) {
385+ /* vm_pgoff: 1 -- TPTR */
386+ if (io_remap_pfn_range(vma, vma->vm_start,
387+ hr_dev->tptr_dma_addr >> PAGE_SHIFT,
388+ hr_dev->tptr_size,
389+ vma->vm_page_prot))
390+ return -EAGAIN;
391+ } else
563392 return -EINVAL;
564− }
565393
566394 return 0;
567395 }
@@ -605,7 +433,7 @@ static int hns_roce_register_device(struct hns_roce_dev *hr_dev) · 37 unmodified lines
605433 spin_lock_init(&iboe->lock);
606434
607435 ib_dev = &hr_dev->ib_dev;
608− strlcpy(ib_dev->name, "hisi_%d", IB_DEVICE_NAME_MAX);
436+ strlcpy(ib_dev->name, "hns_%d", IB_DEVICE_NAME_MAX);
609437
610438 ib_dev->owner = THIS_MODULE;
611439 ib_dev->node_type = RDMA_NODE_IB_CA;
@@ -639,6 +467,8 @@ static int hns_roce_register_device(struct hns_roce_dev *hr_dev) · 27 unmodified lines
639467 ib_dev->get_link_layer = hns_roce_get_link_layer;
640468 ib_dev->get_netdev = hns_roce_get_netdev;
641469 ib_dev->query_gid = hns_roce_query_gid;
470+ ib_dev->add_gid = hns_roce_add_gid;
471+ ib_dev->del_gid = hns_roce_del_gid;
642472 ib_dev->query_pkey = hns_roce_query_pkey;
643473 ib_dev->alloc_ucontext = hns_roce_alloc_ucontext;
644474 ib_dev->dealloc_ucontext = hns_roce_dealloc_ucontext;
@@ -681,32 +511,22 @@ static int hns_roce_register_device(struct hns_roce_dev *hr_dev) · 36 unmodified lines
681511 return ret;
682512 }
683513
684− ret = hns_roce_setup_mtu_gids(hr_dev);
514+ ret = hns_roce_setup_mtu_mac(hr_dev);
685515 if (ret) {
686− dev_err(dev, "roce_setup_mtu_gids failed!\n");
687− goto error_failed_setup_mtu_gids;
516+ dev_err(dev, "setup_mtu_mac failed!\n");
517+ goto error_failed_setup_mtu_mac;
688518 }
689519
690520 iboe->nb.notifier_call = hns_roce_netdev_event;
691521 ret = register_netdevice_notifier(&iboe->nb);
692522 if (ret) {
693523 dev_err(dev, "register_netdevice_notifier failed!\n");
694− goto error_failed_setup_mtu_gids;
695− }
696−
697− iboe->nb_inet.notifier_call = hns_roce_inet_event;
698− ret = register_inetaddr_notifier(&iboe->nb_inet);
699− if (ret) {
700− dev_err(dev, "register inet addr notifier failed!\n");
701− goto error_failed_register_inetaddr_notifier;
524+ goto error_failed_setup_mtu_mac;
702525 }
703526
704527 return 0;
705528
706−error_failed_register_inetaddr_notifier:
707− unregister_netdevice_notifier(&iboe->nb);
708−
709−error_failed_setup_mtu_gids:
529+error_failed_setup_mtu_mac:
710530 ib_unregister_device(ib_dev);
711531
712532 return ret;
@@ -940,10 +760,10 @@ err_unmap_mtt: · 227 unmodified lines
940760 }
941761
942762 /**
943−* hns_roce_setup_hca - setup host channel adapter
944−* @hr_dev: pointer to hns roce device
945−* Return : int
946−*/
763+ * hns_roce_setup_hca - setup host channel adapter
764+ * @hr_dev: pointer to hns roce device
765+ * Return : int
766+ */
947767 static int hns_roce_setup_hca(struct hns_roce_dev *hr_dev)
948768 {
949769 int ret;
@@ -1008,11 +828,11 @@ err_uar_table_free: · 58 unmodified lines
1008828 }
1009829
1010830 /**
1011−* hns_roce_probe - RoCE driver entrance
1012−* @pdev: pointer to platform device
1013−* Return : int
1014−*
1015−*/
831+ * hns_roce_probe - RoCE driver entrance
832+ * @pdev: pointer to platform device
833+ * Return : int
834+ *
835+ */
1016836 static int hns_roce_probe(struct platform_device *pdev)
1017837 {
1018838 int ret;
@@ -1023,9 +843,6 @@ static int hns_roce_probe(struct platform_device *pdev) · 4 unmodified lines
1023843 if (!hr_dev)
1024844 return -ENOMEM;
1025845
1026− memset((u8 *)hr_dev + sizeof(struct ib_device), 0,
1027− sizeof(struct hns_roce_dev) - sizeof(struct ib_device));
1028−
1029846 hr_dev->pdev = pdev;
1030847 platform_set_drvdata(pdev, hr_dev);
1031848
@@ -1125,9 +942,9 @@ error_failed_get_cfg: · 93 unmodified lines
1125942 }
1126943
1127944 /**
1128−* hns_roce_remove - remove RoCE device
1129−* @pdev: pointer to platform device
1130−*/
945+ * hns_roce_remove - remove RoCE device
946+ * @pdev: pointer to platform device
947+ */
1131948 static int hns_roce_remove(struct platform_device *pdev)
1132949 {
1133950 struct hns_roce_dev *hr_dev = platform_get_drvdata(pdev);
drivers/infiniband/hw/hns/hns_roce_mr.c+26−17
@@ -42,7 +42,7 @@ static u32 hw_index_to_key(unsigned long ind) · 41 unmodified lines
4242 return (u32)(ind >> 24) | (ind << 8);
4343 }
4444
45−static unsigned long key_to_hw_index(u32 key)
45+unsigned long key_to_hw_index(u32 key)
4646 {
4747 return (key << 24) | (key >> 8);
4848 }
@@ -53,16 +53,16 @@ static int hns_roce_sw2hw_mpt(struct hns_roce_dev *hr_dev, · 4 unmodified lines
5353 {
5454 return hns_roce_cmd_mbox(hr_dev, mailbox->dma, 0, mpt_index, 0,
5555 HNS_ROCE_CMD_SW2HW_MPT,
56− HNS_ROCE_CMD_TIME_CLASS_B);
56+ HNS_ROCE_CMD_TIMEOUT_MSECS);
5757 }
5858
59−static int hns_roce_hw2sw_mpt(struct hns_roce_dev *hr_dev,
59+int hns_roce_hw2sw_mpt(struct hns_roce_dev *hr_dev,
6060 struct hns_roce_cmd_mailbox *mailbox,
6161 unsigned long mpt_index)
6262 {
6363 return hns_roce_cmd_mbox(hr_dev, 0, mailbox ? mailbox->dma : 0,
6464 mpt_index, !mailbox, HNS_ROCE_CMD_HW2SW_MPT,
65− HNS_ROCE_CMD_TIME_CLASS_B);
65+ HNS_ROCE_CMD_TIMEOUT_MSECS);
6666 }
6767
6868 static int hns_roce_buddy_alloc(struct hns_roce_buddy *buddy, int order,
@@ -137,11 +137,13 @@ static int hns_roce_buddy_init(struct hns_roce_buddy *buddy, int max_order) · 68 unmodified lines
137137
138138 for (i = 0; i <= buddy->max_order; ++i) {
139139 s = BITS_TO_LONGS(1 << (buddy->max_order - i));
140− buddy->bits[i] = kmalloc_array(s, sizeof(long), GFP_KERNEL);
141− if (!buddy->bits[i])
142− goto err_out_free;
143−
144− bitmap_zero(buddy->bits[i], 1 << (buddy->max_order - i));
140+ buddy->bits[i] = kcalloc(s, sizeof(long), GFP_KERNEL |
141+ __GFP_NOWARN);
142+ if (!buddy->bits[i]) {
143+ buddy->bits[i] = vzalloc(s * sizeof(long));
144+ if (!buddy->bits[i])
145+ goto err_out_free;
146+ }
145147 }
146148
147149 set_bit(0, buddy->bits[buddy->max_order]);
@@ -151,7 +153,7 @@ static int hns_roce_buddy_init(struct hns_roce_buddy *buddy, int max_order) · 3 unmodified lines
151153
152154 err_out_free:
153155 for (i = 0; i <= buddy->max_order; ++i)
154− kfree(buddy->bits[i]);
156+ kvfree(buddy->bits[i]);
155157
156158 err_out:
157159 kfree(buddy->bits);
@@ -164,7 +166,7 @@ static void hns_roce_buddy_cleanup(struct hns_roce_buddy *buddy) · 6 unmodified lines
164166 int i;
165167
166168 for (i = 0; i <= buddy->max_order; ++i)
167− kfree(buddy->bits[i]);
169+ kvfree(buddy->bits[i]);
168170
169171 kfree(buddy->bits);
170172 kfree(buddy->num_free);
@@ -287,7 +289,7 @@ static void hns_roce_mr_free(struct hns_roce_dev *hr_dev, · 116 unmodified lines
287289 }
288290
289291 hns_roce_bitmap_free(&hr_dev->mr_table.mtpt_bitmap,
290− key_to_hw_index(mr->key));
292+ key_to_hw_index(mr->key), BITMAP_NO_RR);
291293 }
292294
293295 static int hns_roce_mr_enable(struct hns_roce_dev *hr_dev,
@@ -605,13 +607,20 @@ err_free: · 311 unmodified lines
605607
606608 int hns_roce_dereg_mr(struct ib_mr *ibmr)
607609 {
610+ struct hns_roce_dev *hr_dev = to_hr_dev(ibmr->device);
608611 struct hns_roce_mr *mr = to_hr_mr(ibmr);
612+ int ret = 0;
609613
610− hns_roce_mr_free(to_hr_dev(ibmr->device), mr);
611− if (mr->umem)
612− ib_umem_release(mr->umem);
614+ if (hr_dev->hw->dereg_mr) {
615+ ret = hr_dev->hw->dereg_mr(hr_dev, mr);
616+ } else {
617+ hns_roce_mr_free(hr_dev, mr);
613618
614− kfree(mr);
619+ if (mr->umem)
620+ ib_umem_release(mr->umem);
615621
616− return 0;
622+ kfree(mr);
623+ }
624+
625+ return ret;
617626 }
drivers/infiniband/hw/hns/hns_roce_pd.c+3−2
@@ -40,7 +40,7 @@ static int hns_roce_pd_alloc(struct hns_roce_dev *hr_dev, unsigned long *pdn) · 39 unmodified lines
4040
4141 static void hns_roce_pd_free(struct hns_roce_dev *hr_dev, unsigned long pdn)
4242 {
43− hns_roce_bitmap_free(&hr_dev->pd_bitmap, pdn);
43+ hns_roce_bitmap_free(&hr_dev->pd_bitmap, pdn, BITMAP_NO_RR);
4444 }
4545
4646 int hns_roce_init_pd_table(struct hns_roce_dev *hr_dev)
@@ -121,7 +121,8 @@ int hns_roce_uar_alloc(struct hns_roce_dev *hr_dev, struct hns_roce_uar *uar) · 74 unmodified lines
121121
122122 void hns_roce_uar_free(struct hns_roce_dev *hr_dev, struct hns_roce_uar *uar)
123123 {
124− hns_roce_bitmap_free(&hr_dev->uar_table.bitmap, uar->index);
124+ hns_roce_bitmap_free(&hr_dev->uar_table.bitmap, uar->index,
125+ BITMAP_NO_RR);
125126 }
126127
127128 int hns_roce_init_uar_table(struct hns_roce_dev *hr_dev)
drivers/infiniband/hw/hns/hns_roce_qp.c+2−2
@@ -37,7 +37,7 @@ · 36 unmodified lines
3737 #include "hns_roce_common.h"
3838 #include "hns_roce_device.h"
3939 #include "hns_roce_hem.h"
40−#include "hns_roce_user.h"
40+#include <rdma/hns-abi.h>
4141
4242 #define SQP_NUM (2 * HNS_ROCE_MAX_PORTS)
4343
@@ -250,7 +250,7 @@ void hns_roce_release_range_qp(struct hns_roce_dev *hr_dev, int base_qpn, · 206 unmodified lines
250250 if (base_qpn < SQP_NUM)
251251 return;
252252
253− hns_roce_bitmap_free_range(&qp_table->bitmap, base_qpn, cnt);
253+ hns_roce_bitmap_free_range(&qp_table->bitmap, base_qpn, cnt, BITMAP_RR);
254254 }
255255
256256 static int hns_roce_set_rq_size(struct hns_roce_dev *hr_dev,
drivers/infiniband/hw/i40iw/i40iw.h+30−7
@@ -112,9 +112,12 @@ · 111 unmodified lines
112112 #define I40IW_DRV_OPT_MCAST_LOGPORT_MAP 0x00000800
113113
114114 #define IW_HMC_OBJ_TYPE_NUM ARRAY_SIZE(iw_hmc_obj_types)
115−#define IW_CFG_FPM_QP_COUNT 32768
116−#define I40IW_MAX_PAGES_PER_FMR 512
117−#define I40IW_MIN_PAGES_PER_FMR 1
115+#define IW_CFG_FPM_QP_COUNT 32768
116+#define I40IW_MAX_PAGES_PER_FMR 512
117+#define I40IW_MIN_PAGES_PER_FMR 1
118+#define I40IW_CQP_COMPL_RQ_WQE_FLUSHED 2
119+#define I40IW_CQP_COMPL_SQ_WQE_FLUSHED 3
120+#define I40IW_CQP_COMPL_RQ_SQ_WQE_FLUSHED 4
118121
119122 #define I40IW_MTU_TO_MSS 40
120123 #define I40IW_DEFAULT_MSS 1460
@@ -210,6 +213,12 @@ struct i40iw_msix_vector { · 89 unmodified lines
210213 u32 ceq_id;
211214 };
212215
216+struct l2params_work {
217+ struct work_struct work;
218+ struct i40iw_device *iwdev;
219+ struct i40iw_l2params l2params;
220+};
221+
213222 #define I40IW_MSIX_TABLE_SIZE 65
214223
215224 struct virtchnl_work {
@@ -227,6 +236,7 @@ struct i40iw_device { · 11 unmodified lines
227236 struct net_device *netdev;
228237 wait_queue_head_t vchnl_waitq;
229238 struct i40iw_sc_dev sc_dev;
239+ struct i40iw_sc_vsi vsi;
230240 struct i40iw_handler *hdl;
231241 struct i40e_info *ldev;
232242 struct i40e_client *client;
@@ -280,7 +290,6 @@ struct i40iw_device { · 47 unmodified lines
280290 u32 sd_type;
281291 struct workqueue_struct *param_wq;
282292 atomic_t params_busy;
283− u32 mss;
284293 enum init_completion_state init_state;
285294 u16 mac_ip_table_idx;
286295 atomic_t vchnl_msgs;
@@ -297,6 +306,14 @@ struct i40iw_device { · 10 unmodified lines
297306 u32 mr_stagmask;
298307 u32 mpa_version;
299308 bool dcb;
309+ bool closing;
310+ bool reset;
311+ u32 used_pds;
312+ u32 used_cqs;
313+ u32 used_mrs;
314+ u32 used_qps;
315+ wait_queue_head_t close_wq;
316+ atomic64_t use_count;
300317 };
301318
302319 struct i40iw_ib_device {
@@ -498,7 +515,7 @@ u32 i40iw_initialize_hw_resources(struct i40iw_device *iwdev); · 195 unmodified lines
498515
499516 int i40iw_register_rdma_device(struct i40iw_device *iwdev);
500517 void i40iw_port_ibevent(struct i40iw_device *iwdev);
501−int i40iw_cm_disconn(struct i40iw_qp *);
518+void i40iw_cm_disconn(struct i40iw_qp *iwqp);
502519 void i40iw_cm_disconn_worker(void *);
503520 int mini_cm_recv_pkt(struct i40iw_cm_core *, struct i40iw_device *,
504521 struct sk_buff *);
@@ -508,20 +525,26 @@ enum i40iw_status_code i40iw_handle_cqp_op(struct i40iw_device *iwdev, · 3 unmodified lines
508525 enum i40iw_status_code i40iw_add_mac_addr(struct i40iw_device *iwdev,
509526 u8 *mac_addr, u8 *mac_index);
510527 int i40iw_modify_qp(struct ib_qp *, struct ib_qp_attr *, int, struct ib_udata *);
528+void i40iw_cq_wq_destroy(struct i40iw_device *iwdev, struct i40iw_sc_cq *cq);
511529
512530 void i40iw_rem_pdusecount(struct i40iw_pd *iwpd, struct i40iw_device *iwdev);
513531 void i40iw_add_pdusecount(struct i40iw_pd *iwpd);
532+void i40iw_rem_devusecount(struct i40iw_device *iwdev);
533+void i40iw_add_devusecount(struct i40iw_device *iwdev);
514534 void i40iw_hw_modify_qp(struct i40iw_device *iwdev, struct i40iw_qp *iwqp,
515535 struct i40iw_modify_qp_info *info, bool wait);
516536
537+void i40iw_qp_suspend_resume(struct i40iw_sc_dev *dev,
538+ struct i40iw_sc_qp *qp,
539+ bool suspend);
517540 enum i40iw_status_code i40iw_manage_qhash(struct i40iw_device *iwdev,
518541 struct i40iw_cm_info *cminfo,
519542 enum i40iw_quad_entry_type etype,
520543 enum i40iw_quad_hash_manage_type mtype,
521544 void *cmnode,
522545 bool wait);
523−void i40iw_receive_ilq(struct i40iw_sc_dev *dev, struct i40iw_puda_buf *rbuf);
524−void i40iw_free_sqbuf(struct i40iw_sc_dev *dev, void *bufp);
546+void i40iw_receive_ilq(struct i40iw_sc_vsi *vsi, struct i40iw_puda_buf *rbuf);
547+void i40iw_free_sqbuf(struct i40iw_sc_vsi *vsi, void *bufp);
525548 void i40iw_free_qp_resources(struct i40iw_device *iwdev,
526549 struct i40iw_qp *iwqp,
527550 u32 qp_num);
drivers/infiniband/hw/i40iw/i40iw_cm.c+290−87
@@ -68,13 +68,13 @@ static void i40iw_disconnect_worker(struct work_struct *work); · 67 unmodified lines
6868
6969 /**
7070 * i40iw_free_sqbuf - put back puda buffer if refcount = 0
71− * @dev: FPK device
71+ * @vsi: pointer to vsi structure
7272 * @buf: puda buffer to free
7373 */
74−void i40iw_free_sqbuf(struct i40iw_sc_dev *dev, void *bufp)
74+void i40iw_free_sqbuf(struct i40iw_sc_vsi *vsi, void *bufp)
7575 {
7676 struct i40iw_puda_buf *buf = (struct i40iw_puda_buf *)bufp;
77− struct i40iw_puda_rsrc *ilq = dev->ilq;
77+ struct i40iw_puda_rsrc *ilq = vsi->ilq;
7878
7979 if (!atomic_dec_return(&buf->refcount))
8080 i40iw_puda_ret_bufpool(ilq, buf);
@@ -221,6 +221,7 @@ static void i40iw_get_addr_info(struct i40iw_cm_node *cm_node, · 140 unmodified lines
221221 memcpy(cm_info->rem_addr, cm_node->rem_addr, sizeof(cm_info->rem_addr));
222222 cm_info->loc_port = cm_node->loc_port;
223223 cm_info->rem_port = cm_node->rem_port;
224+ cm_info->user_pri = cm_node->user_pri;
224225 }
225226
226227 /**
@@ -271,6 +272,7 @@ static int i40iw_send_cm_event(struct i40iw_cm_node *cm_node, · 44 unmodified lines
271272 event.provider_data = (void *)cm_node;
272273 event.private_data = (void *)cm_node->pdata_buf;
273274 event.private_data_len = (u8)cm_node->pdata.size;
275+ event.ird = cm_node->ird_size;
274276 break;
275277 case IW_CM_EVENT_CONNECT_REPLY:
276278 i40iw_get_cmevent_info(cm_node, cm_id, &event);
@@ -335,13 +337,13 @@ static struct i40iw_cm_event *i40iw_create_event(struct i40iw_cm_node *cm_node, · 58 unmodified lines
335337 */
336338 static void i40iw_free_retrans_entry(struct i40iw_cm_node *cm_node)
337339 {
338− struct i40iw_sc_dev *dev = cm_node->dev;
340+ struct i40iw_device *iwdev = cm_node->iwdev;
339341 struct i40iw_timer_entry *send_entry;
340342
341343 send_entry = cm_node->send_entry;
342344 if (send_entry) {
343345 cm_node->send_entry = NULL;
344− i40iw_free_sqbuf(dev, (void *)send_entry->sqbuf);
346+ i40iw_free_sqbuf(&iwdev->vsi, (void *)send_entry->sqbuf);
345347 kfree(send_entry);
346348 atomic_dec(&cm_node->ref_count);
347349 }
@@ -360,15 +362,6 @@ static void i40iw_cleanup_retrans_entry(struct i40iw_cm_node *cm_node) · 12 unmodified lines
360362 spin_unlock_irqrestore(&cm_node->retrans_list_lock, flags);
361363 }
362364
363−static bool is_remote_ne020_or_chelsio(struct i40iw_cm_node *cm_node)
364−{
365− if ((cm_node->rem_mac[0] == 0x0) &&
366− (((cm_node->rem_mac[1] == 0x12) && (cm_node->rem_mac[2] == 0x55)) ||
367− ((cm_node->rem_mac[1] == 0x07 && (cm_node->rem_mac[2] == 0x43)))))
368− return true;
369− return false;
370−}
371−
372365 /**
373366 * i40iw_form_cm_frame - get a free packet and build frame
374367 * @cm_node: connection's node ionfo to use in frame
@@ -384,7 +377,7 @@ static struct i40iw_puda_buf *i40iw_form_cm_frame(struct i40iw_cm_node *cm_node, · 9 unmodified lines
384377 u8 flags)
385378 {
386379 struct i40iw_puda_buf *sqbuf;
387− struct i40iw_sc_dev *dev = cm_node->dev;
380+ struct i40iw_sc_vsi *vsi = &cm_node->iwdev->vsi;
388381 u8 *buf;
389382
390383 struct tcphdr *tcph;
@@ -396,8 +389,9 @@ static struct i40iw_puda_buf *i40iw_form_cm_frame(struct i40iw_cm_node *cm_node, · 5 unmodified lines
396389 u32 opts_len = 0;
397390 u32 pd_len = 0;
398391 u32 hdr_len = 0;
392+ u16 vtag;
399393
400− sqbuf = i40iw_puda_get_bufpool(dev->ilq);
394+ sqbuf = i40iw_puda_get_bufpool(vsi->ilq);
401395 if (!sqbuf)
402396 return NULL;
403397 buf = sqbuf->mem.va;
@@ -408,11 +402,8 @@ static struct i40iw_puda_buf *i40iw_form_cm_frame(struct i40iw_cm_node *cm_node, · 4 unmodified lines
408402 if (hdr)
409403 hdr_len = hdr->size;
410404
411− if (pdata) {
405+ if (pdata)
412406 pd_len = pdata->size;
413− if (!is_remote_ne020_or_chelsio(cm_node))
414− pd_len += MPA_ZERO_PAD_LEN;
415− }
416407
417408 if (cm_node->vlan_id < VLAN_TAG_PRESENT)
418409 eth_hlen += 4;
@@ -445,7 +436,8 @@ static struct i40iw_puda_buf *i40iw_form_cm_frame(struct i40iw_cm_node *cm_node, · 26 unmodified lines
445436 ether_addr_copy(ethh->h_source, cm_node->loc_mac);
446437 if (cm_node->vlan_id < VLAN_TAG_PRESENT) {
447438 ((struct vlan_ethhdr *)ethh)->h_vlan_proto = htons(ETH_P_8021Q);
448− ((struct vlan_ethhdr *)ethh)->h_vlan_TCI = htons(cm_node->vlan_id);
439+ vtag = (cm_node->user_pri << VLAN_PRIO_SHIFT) | cm_node->vlan_id;
440+ ((struct vlan_ethhdr *)ethh)->h_vlan_TCI = htons(vtag);
449441
450442 ((struct vlan_ethhdr *)ethh)->h_vlan_encapsulated_proto = htons(ETH_P_IP);
451443 } else {
@@ -454,7 +446,7 @@ static struct i40iw_puda_buf *i40iw_form_cm_frame(struct i40iw_cm_node *cm_node, · 2 unmodified lines
454446
455447 iph->version = IPVERSION;
456448 iph->ihl = 5; /* 5 * 4Byte words, IP headr len */
457− iph->tos = 0;
449+ iph->tos = cm_node->tos;
458450 iph->tot_len = htons(packetsize);
459451 iph->id = htons(++cm_node->tcp_cntxt.loc_id);
460452
@@ -474,13 +466,15 @@ static struct i40iw_puda_buf *i40iw_form_cm_frame(struct i40iw_cm_node *cm_node, · 13 unmodified lines
474466 ether_addr_copy(ethh->h_source, cm_node->loc_mac);
475467 if (cm_node->vlan_id < VLAN_TAG_PRESENT) {
476468 ((struct vlan_ethhdr *)ethh)->h_vlan_proto = htons(ETH_P_8021Q);
477− ((struct vlan_ethhdr *)ethh)->h_vlan_TCI = htons(cm_node->vlan_id);
469+ vtag = (cm_node->user_pri << VLAN_PRIO_SHIFT) | cm_node->vlan_id;
470+ ((struct vlan_ethhdr *)ethh)->h_vlan_TCI = htons(vtag);
478471 ((struct vlan_ethhdr *)ethh)->h_vlan_encapsulated_proto = htons(ETH_P_IPV6);
479472 } else {
480473 ethh->h_proto = htons(ETH_P_IPV6);
481474 }
482475 ip6h->version = 6;
483− ip6h->flow_lbl[0] = 0;
476+ ip6h->priority = cm_node->tos >> 4;
477+ ip6h->flow_lbl[0] = cm_node->tos << 4;
484478 ip6h->flow_lbl[1] = 0;
485479 ip6h->flow_lbl[2] = 0;
486480 ip6h->payload_len = htons(packetsize - sizeof(*ip6h));
@@ -1065,7 +1059,7 @@ int i40iw_schedule_cm_timer(struct i40iw_cm_node *cm_node, · 578 unmodified lines
10651059 int send_retrans,
10661060 int close_when_complete)
10671061 {
1068− struct i40iw_sc_dev *dev = cm_node->dev;
1062+ struct i40iw_sc_vsi *vsi = &cm_node->iwdev->vsi;
10691063 struct i40iw_cm_core *cm_core = cm_node->cm_core;
10701064 struct i40iw_timer_entry *new_send;
10711065 int ret = 0;
@@ -1074,7 +1068,7 @@ int i40iw_schedule_cm_timer(struct i40iw_cm_node *cm_node, · 2 unmodified lines
10741068
10751069 new_send = kzalloc(sizeof(*new_send), GFP_ATOMIC);
10761070 if (!new_send) {
1077− i40iw_free_sqbuf(cm_node->dev, (void *)sqbuf);
1071+ i40iw_free_sqbuf(vsi, (void *)sqbuf);
10781072 return -ENOMEM;
10791073 }
10801074 new_send->retrycount = I40IW_DEFAULT_RETRYS;
@@ -1089,7 +1083,7 @@ int i40iw_schedule_cm_timer(struct i40iw_cm_node *cm_node, · 8 unmodified lines
10891083 new_send->timetosend += (HZ / 10);
10901084 if (cm_node->close_entry) {
10911085 kfree(new_send);
1092− i40iw_free_sqbuf(cm_node->dev, (void *)sqbuf);
1086+ i40iw_free_sqbuf(vsi, (void *)sqbuf);
10931087 i40iw_pr_err("already close entry\n");
10941088 return -EINVAL;
10951089 }
@@ -1104,7 +1098,7 @@ int i40iw_schedule_cm_timer(struct i40iw_cm_node *cm_node, · 8 unmodified lines
11041098 new_send->timetosend = jiffies + I40IW_RETRY_TIMEOUT;
11051099
11061100 atomic_inc(&sqbuf->refcount);
1107− i40iw_puda_send_buf(dev->ilq, sqbuf);
1101+ i40iw_puda_send_buf(vsi->ilq, sqbuf);
11081102 if (!send_retrans) {
11091103 i40iw_cleanup_retrans_entry(cm_node);
11101104 if (close_when_complete)
@@ -1201,6 +1195,7 @@ static void i40iw_cm_timer_tick(unsigned long pass) · 90 unmodified lines
12011195 struct i40iw_cm_node *cm_node;
12021196 struct i40iw_timer_entry *send_entry, *close_entry;
12031197 struct list_head *list_core_temp;
1198+ struct i40iw_sc_vsi *vsi;
12041199 struct list_head *list_node;
12051200 struct i40iw_cm_core *cm_core = (struct i40iw_cm_core *)pass;
12061201 u32 settimer = 0;
@@ -1276,9 +1271,10 @@ static void i40iw_cm_timer_tick(unsigned long pass) · 69 unmodified lines
12761271 cm_node->cm_core->stats_pkt_retrans++;
12771272 spin_unlock_irqrestore(&cm_node->retrans_list_lock, flags);
12781273
1274+ vsi = &cm_node->iwdev->vsi;
12791275 dev = cm_node->dev;
12801276 atomic_inc(&send_entry->sqbuf->refcount);
1281− i40iw_puda_send_buf(dev->ilq, send_entry->sqbuf);
1277+ i40iw_puda_send_buf(vsi->ilq, send_entry->sqbuf);
12821278 spin_lock_irqsave(&cm_node->retrans_list_lock, flags);
12831279 if (send_entry->send_retrans) {
12841280 send_entry->retranscount--;
@@ -1379,10 +1375,11 @@ int i40iw_send_syn(struct i40iw_cm_node *cm_node, u32 sendack) · 94 unmodified lines
13791375 static void i40iw_send_ack(struct i40iw_cm_node *cm_node)
13801376 {
13811377 struct i40iw_puda_buf *sqbuf;
1378+ struct i40iw_sc_vsi *vsi = &cm_node->iwdev->vsi;
13821379
13831380 sqbuf = i40iw_form_cm_frame(cm_node, NULL, NULL, NULL, SET_ACK);
13841381 if (sqbuf)
1385− i40iw_puda_send_buf(cm_node->dev->ilq, sqbuf);
1382+ i40iw_puda_send_buf(vsi->ilq, sqbuf);
13861383 else
13871384 i40iw_pr_err("no sqbuf\n");
13881385 }
@@ -1564,9 +1561,15 @@ static enum i40iw_status_code i40iw_del_multiple_qhash( · 175 unmodified lines
15641561 memcpy(cm_info->loc_addr, child_listen_node->loc_addr,
15651562 sizeof(cm_info->loc_addr));
15661563 cm_info->vlan_id = child_listen_node->vlan_id;
1567− ret = i40iw_manage_qhash(iwdev, cm_info,
1568− I40IW_QHASH_TYPE_TCP_SYN,
1569− I40IW_QHASH_MANAGE_TYPE_DELETE, NULL, false);
1564+ if (child_listen_node->qhash_set) {
1565+ ret = i40iw_manage_qhash(iwdev, cm_info,
1566+ I40IW_QHASH_TYPE_TCP_SYN,
1567+ I40IW_QHASH_MANAGE_TYPE_DELETE,
1568+ NULL, false);
1569+ child_listen_node->qhash_set = false;
1570+ } else {
1571+ ret = I40IW_SUCCESS;
1572+ }
15701573 i40iw_debug(&iwdev->sc_dev,
15711574 I40IW_DEBUG_CM,
15721575 "freed pointer = %p\n",
@@ -1591,9 +1594,10 @@ static enum i40iw_status_code i40iw_del_multiple_qhash( · 18 unmodified lines
15911594 static struct net_device *i40iw_netdev_vlan_ipv6(u32 *addr, u16 *vlan_id, u8 *mac)
15921595 {
15931596 struct net_device *ip_dev = NULL;
1594−#if IS_ENABLED(CONFIG_IPV6)
15951597 struct in6_addr laddr6;
15961598
1599+ if (!IS_ENABLED(CONFIG_IPV6))
1600+ return NULL;
15971601 i40iw_copy_ip_htonl(laddr6.in6_u.u6_addr32, addr);
15981602 if (vlan_id)
15991603 *vlan_id = I40IW_NO_VLAN;
@@ -1610,7 +1614,6 @@ static struct net_device *i40iw_netdev_vlan_ipv6(u32 *addr, u16 *vlan_id, u8 *ma · 10 unmodified lines
16101614 }
16111615 }
16121616 rcu_read_unlock();
1613−#endif
16141617 return ip_dev;
16151618 }
16161619
@@ -1646,7 +1649,7 @@ static enum i40iw_status_code i40iw_add_mqh_6(struct i40iw_device *iwdev, · 29 unmodified lines
16461649 {
16471650 struct net_device *ip_dev;
16481651 struct inet6_dev *idev;
1649− struct inet6_ifaddr *ifp;
1652+ struct inet6_ifaddr *ifp, *tmp;
16501653 enum i40iw_status_code ret = 0;
16511654 struct i40iw_cm_listener *child_listen_node;
16521655 unsigned long flags;
@@ -1661,7 +1664,7 @@ static enum i40iw_status_code i40iw_add_mqh_6(struct i40iw_device *iwdev, · 8 unmodified lines
16611664 i40iw_pr_err("idev == NULL\n");
16621665 break;
16631666 }
1664− list_for_each_entry(ifp, &idev->addr_list, if_list) {
1667+ list_for_each_entry_safe(ifp, tmp, &idev->addr_list, if_list) {
16651668 i40iw_debug(&iwdev->sc_dev,
16661669 I40IW_DEBUG_CM,
16671670 "IP=%pI6, vlan_id=%d, MAC=%pM\n",
@@ -1675,7 +1678,6 @@ static enum i40iw_status_code i40iw_add_mqh_6(struct i40iw_device *iwdev, · 7 unmodified lines
16751678 "Allocating child listener %p\n",
16761679 child_listen_node);
16771680 if (!child_listen_node) {
1678− i40iw_pr_err("listener memory allocation\n");
16791681 ret = I40IW_ERR_NO_MEMORY;
16801682 goto exit;
16811683 }
@@ -1695,6 +1697,7 @@ static enum i40iw_status_code i40iw_add_mqh_6(struct i40iw_device *iwdev, · 13 unmodified lines
16951697 I40IW_QHASH_MANAGE_TYPE_ADD,
16961698 NULL, true);
16971699 if (!ret) {
1700+ child_listen_node->qhash_set = true;
16981701 spin_lock_irqsave(&iwdev->cm_core.listen_list_lock, flags);
16991702 list_add(&child_listen_node->child_listen_list,
17001703 &cm_parent_listen_node->child_listen_list);
@@ -1751,7 +1754,6 @@ static enum i40iw_status_code i40iw_add_mqh_4( · 50 unmodified lines
17511754 "Allocating child listener %p\n",
17521755 child_listen_node);
17531756 if (!child_listen_node) {
1754− i40iw_pr_err("listener memory allocation\n");
17551757 in_dev_put(idev);
17561758 ret = I40IW_ERR_NO_MEMORY;
17571759 goto exit;
@@ -1773,6 +1775,7 @@ static enum i40iw_status_code i40iw_add_mqh_4( · 15 unmodified lines
17731775 NULL,
17741776 true);
17751777 if (!ret) {
1778+ child_listen_node->qhash_set = true;
17761779 spin_lock_irqsave(&iwdev->cm_core.listen_list_lock, flags);
17771780 list_add(&child_listen_node->child_listen_list,
17781781 &cm_parent_listen_node->child_listen_list);
@@ -1880,6 +1883,7 @@ static int i40iw_dec_refcnt_listen(struct i40iw_cm_core *cm_core, · 101 unmodified lines
18801883 nfo.loc_port = listener->loc_port;
18811884 nfo.ipv4 = listener->ipv4;
18821885 nfo.vlan_id = listener->vlan_id;
1886+ nfo.user_pri = listener->user_pri;
18831887
18841888 if (!list_empty(&listener->child_listen_list)) {
18851889 i40iw_del_multiple_qhash(listener->iwdev, &nfo, listener);
@@ -2138,6 +2142,20 @@ static struct i40iw_cm_node *i40iw_make_cm_node( · 252 unmodified lines
21382142 /* set our node specific transport info */
21392143 cm_node->ipv4 = cm_info->ipv4;
21402144 cm_node->vlan_id = cm_info->vlan_id;
2145+ if ((cm_node->vlan_id == I40IW_NO_VLAN) && iwdev->dcb)
2146+ cm_node->vlan_id = 0;
2147+ cm_node->tos = cm_info->tos;
2148+ cm_node->user_pri = cm_info->user_pri;
2149+ if (listener) {
2150+ if (listener->tos != cm_info->tos)
2151+ i40iw_debug(&iwdev->sc_dev, I40IW_DEBUG_DCB,
2152+ "application TOS[%d] and remote client TOS[%d] mismatch\n",
2153+ listener->tos, cm_info->tos);
2154+ cm_node->tos = max(listener->tos, cm_info->tos);
2155+ cm_node->user_pri = rt_tos2priority(cm_node->tos);
2156+ i40iw_debug(&iwdev->sc_dev, I40IW_DEBUG_DCB, "listener: TOS:[%d] UP:[%d]\n",
2157+ cm_node->tos, cm_node->user_pri);
2158+ }
21412159 memcpy(cm_node->loc_addr, cm_info->loc_addr, sizeof(cm_node->loc_addr));
21422160 memcpy(cm_node->rem_addr, cm_info->rem_addr, sizeof(cm_node->rem_addr));
21432161 cm_node->loc_port = cm_info->loc_port;
@@ -2162,7 +2180,7 @@ static struct i40iw_cm_node *i40iw_make_cm_node( · 18 unmodified lines
21622180 I40IW_CM_DEFAULT_RCV_WND_SCALED >> I40IW_CM_DEFAULT_RCV_WND_SCALE;
21632181 ts = current_kernel_time();
21642182 cm_node->tcp_cntxt.loc_seq_num = ts.tv_nsec;
2165− cm_node->tcp_cntxt.mss = iwdev->mss;
2183+ cm_node->tcp_cntxt.mss = iwdev->vsi.mss;
21662184
21672185 cm_node->iwdev = iwdev;
21682186 cm_node->dev = &iwdev->sc_dev;
@@ -2236,7 +2254,7 @@ static void i40iw_rem_ref_cm_node(struct i40iw_cm_node *cm_node) · 67 unmodified lines
22362254 i40iw_dec_refcnt_listen(cm_core, cm_node->listener, 0, true);
22372255 } else {
22382256 if (!i40iw_listen_port_in_use(cm_core, cm_node->loc_port) &&
2239− cm_node->apbvt_set && cm_node->iwdev) {
2257+ cm_node->apbvt_set) {
22402258 i40iw_manage_apbvt(cm_node->iwdev,
22412259 cm_node->loc_port,
22422260 I40IW_MANAGE_APBVT_DEL);
@@ -2861,7 +2879,7 @@ static struct i40iw_cm_node *i40iw_create_cm_node( · 618 unmodified lines
28612879 /* create a CM connection node */
28622880 cm_node = i40iw_make_cm_node(cm_core, iwdev, cm_info, NULL);
28632881 if (!cm_node)
2864− return NULL;
2882+ return ERR_PTR(-ENOMEM);
28652883 /* set our node side to client (active) side */
28662884 cm_node->tcp_cntxt.client = 1;
28672885 cm_node->tcp_cntxt.rcv_wscale = I40IW_CM_DEFAULT_RCV_WND_SCALE;
@@ -2874,7 +2892,8 @@ static struct i40iw_cm_node *i40iw_create_cm_node( · 6 unmodified lines
28742892 cm_node->vlan_id,
28752893 I40IW_CM_LISTENER_ACTIVE_STATE);
28762894 if (!loopback_remotelistener) {
2877− i40iw_create_event(cm_node, I40IW_CM_EVENT_ABORTED);
2895+ i40iw_rem_ref_cm_node(cm_node);
2896+ return ERR_PTR(-ECONNREFUSED);
28782897 } else {
28792898 loopback_cm_info = *cm_info;
28802899 loopback_cm_info.loc_port = cm_info->rem_port;
@@ -2887,7 +2906,7 @@ static struct i40iw_cm_node *i40iw_create_cm_node( · 6 unmodified lines
28872906 loopback_remotelistener);
28882907 if (!loopback_remotenode) {
28892908 i40iw_rem_ref_cm_node(cm_node);
2890− return NULL;
2909+ return ERR_PTR(-ENOMEM);
28912910 }
28922911 cm_core->stats_loopbacks++;
28932912 loopback_remotenode->loopbackpartner = cm_node;
@@ -3041,10 +3060,10 @@ static int i40iw_cm_close(struct i40iw_cm_node *cm_node) · 147 unmodified lines
30413060 /**
30423061 * i40iw_receive_ilq - recv an ETHERNET packet, and process it
30433062 * through CM
3044− * @dev: FPK dev struct
3063+ * @vsi: pointer to the vsi structure
30453064 * @rbuf: receive buffer
30463065 */
3047−void i40iw_receive_ilq(struct i40iw_sc_dev *dev, struct i40iw_puda_buf *rbuf)
3066+void i40iw_receive_ilq(struct i40iw_sc_vsi *vsi, struct i40iw_puda_buf *rbuf)
30483067 {
30493068 struct i40iw_cm_node *cm_node;
30503069 struct i40iw_cm_listener *listener;
@@ -3052,9 +3071,11 @@ void i40iw_receive_ilq(struct i40iw_sc_dev *dev, struct i40iw_puda_buf *rbuf) · 1 unmodified line
30523071 struct ipv6hdr *ip6h;
30533072 struct tcphdr *tcph;
30543073 struct i40iw_cm_info cm_info;
3074+ struct i40iw_sc_dev *dev = vsi->dev;
30553075 struct i40iw_device *iwdev = (struct i40iw_device *)dev->back_dev;
30563076 struct i40iw_cm_core *cm_core = &iwdev->cm_core;
30573077 struct vlan_ethhdr *ethh;
3078+ u16 vtag;
30583079
30593080 /* if vlan, then maclen = 18 else 14 */
30603081 iph = (struct iphdr *)rbuf->iph;
@@ -3068,7 +3089,9 @@ void i40iw_receive_ilq(struct i40iw_sc_dev *dev, struct i40iw_puda_buf *rbuf) · 7 unmodified lines
30683089 ethh = (struct vlan_ethhdr *)rbuf->mem.va;
30693090
30703091 if (ethh->h_vlan_proto == htons(ETH_P_8021Q)) {
3071− cm_info.vlan_id = ntohs(ethh->h_vlan_TCI) & VLAN_VID_MASK;
3092+ vtag = ntohs(ethh->h_vlan_TCI);
3093+ cm_info.user_pri = (vtag & VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT;
3094+ cm_info.vlan_id = vtag & VLAN_VID_MASK;
30723095 i40iw_debug(cm_core->dev,
30733096 I40IW_DEBUG_CM,
30743097 "%s vlan_id=%d\n",
@@ -3083,6 +3106,7 @@ void i40iw_receive_ilq(struct i40iw_sc_dev *dev, struct i40iw_puda_buf *rbuf) · 8 unmodified lines
30833106 cm_info.loc_addr[0] = ntohl(iph->daddr);
30843107 cm_info.rem_addr[0] = ntohl(iph->saddr);
30853108 cm_info.ipv4 = true;
3109+ cm_info.tos = iph->tos;
30863110 } else {
30873111 ip6h = (struct ipv6hdr *)rbuf->iph;
30883112 i40iw_copy_ip_ntohl(cm_info.loc_addr,
@@ -3090,6 +3114,7 @@ void i40iw_receive_ilq(struct i40iw_sc_dev *dev, struct i40iw_puda_buf *rbuf) · 1 unmodified line
30903114 i40iw_copy_ip_ntohl(cm_info.rem_addr,
30913115 ip6h->saddr.in6_u.u6_addr32);
30923116 cm_info.ipv4 = false;
3117+ cm_info.tos = (ip6h->priority << 4) | (ip6h->flow_lbl[0] >> 4);
30933118 }
30943119 cm_info.loc_port = ntohs(tcph->dest);
30953120 cm_info.rem_port = ntohs(tcph->source);
@@ -3309,6 +3334,8 @@ static void i40iw_cm_init_tsa_conn(struct i40iw_qp *iwqp, · 213 unmodified lines
33093334
33103335 ctx_info->tcp_info_valid = true;
33113336 ctx_info->iwarp_info_valid = true;
3337+ ctx_info->add_to_qoslist = true;
3338+ ctx_info->user_pri = cm_node->user_pri;
33123339
33133340 i40iw_init_tcp_ctx(cm_node, &tcp_info, iwqp);
33143341 if (cm_node->snd_mark_en) {
@@ -3320,33 +3347,47 @@ static void i40iw_cm_init_tsa_conn(struct i40iw_qp *iwqp, · 5 unmodified lines
33203347 cm_node->state = I40IW_CM_STATE_OFFLOADED;
33213348 tcp_info.tcp_state = I40IW_TCP_STATE_ESTABLISHED;
33223349 tcp_info.src_mac_addr_idx = iwdev->mac_ip_table_idx;
3350+ tcp_info.tos = cm_node->tos;
33233351
33243352 dev->iw_priv_qp_ops->qp_setctx(&iwqp->sc_qp, (u64 *)(iwqp->host_ctx.va), ctx_info);
33253353
33263354 /* once tcp_info is set, no need to do it again */
33273355 ctx_info->tcp_info_valid = false;
33283356 ctx_info->iwarp_info_valid = false;
3357+ ctx_info->add_to_qoslist = false;
33293358 }
33303359
33313360 /**
33323361 * i40iw_cm_disconn - when a connection is being closed
33333362 * @iwqp: associate qp for the connection
33343363 */
3335−int i40iw_cm_disconn(struct i40iw_qp *iwqp)
3364+void i40iw_cm_disconn(struct i40iw_qp *iwqp)
33363365 {
33373366 struct disconn_work *work;
33383367 struct i40iw_device *iwdev = iwqp->iwdev;
33393368 struct i40iw_cm_core *cm_core = &iwdev->cm_core;
3369+ unsigned long flags;
33403370
33413371 work = kzalloc(sizeof(*work), GFP_ATOMIC);
33423372 if (!work)
3343− return -ENOMEM; /* Timer will clean up */
3344−
3373+ return; /* Timer will clean up */
3374+
3375+ spin_lock_irqsave(&iwdev->qptable_lock, flags);
3376+ if (!iwdev->qp_table[iwqp->ibqp.qp_num]) {
3377+ spin_unlock_irqrestore(&iwdev->qptable_lock, flags);
3378+ i40iw_debug(&iwdev->sc_dev, I40IW_DEBUG_CM,
3379+ "%s qp_id %d is already freed\n",
3380+ __func__, iwqp->ibqp.qp_num);
3381+ kfree(work);
3382+ return;
3383+ }
33453384 i40iw_add_ref(&iwqp->ibqp);
3385+ spin_unlock_irqrestore(&iwdev->qptable_lock, flags);
3386+
33463387 work->iwqp = iwqp;
33473388 INIT_WORK(&work->work, i40iw_disconnect_worker);
33483389 queue_work(cm_core->disconn_wq, &work->work);
3349− return 0;
3390+ return;
33503391 }
33513392
33523393 /**
@@ -3432,7 +3473,7 @@ static void i40iw_cm_disconn_true(struct i40iw_qp *iwqp) · 79 unmodified lines
34323473 *terminate-handler to issue cm_disconn which can re-free
34333474 *a QP even after its refcnt=0.
34343475 */
3435− del_timer(&iwqp->terminate_timer);
3476+ i40iw_terminate_del_timer(qp);
34363477 if (!iwqp->flush_issued) {
34373478 iwqp->flush_issued = 1;
34383479 issue_flush = 1;
@@ -3462,7 +3503,7 @@ static void i40iw_cm_disconn_true(struct i40iw_qp *iwqp) · 23 unmodified lines
34623503 /* Flush the queues */
34633504 i40iw_flush_wqes(iwdev, iwqp);
34643505
3465− if (qp->term_flags) {
3506+ if (qp->term_flags && iwqp->ibqp.event_handler) {
34663507 ibevent.device = iwqp->ibqp.device;
34673508 ibevent.event = (qp->eventtype == TERM_EVENT_QP_FATAL) ?
34683509 IB_EVENT_QP_FATAL : IB_EVENT_QP_ACCESS_ERR;
@@ -3571,7 +3612,7 @@ int i40iw_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) · 102 unmodified lines
35713612 iwqp->cm_node = (void *)cm_node;
35723613 cm_node->iwqp = iwqp;
35733614
3574− buf_len = conn_param->private_data_len + I40IW_MAX_IETF_SIZE + MPA_ZERO_PAD_LEN;
3615+ buf_len = conn_param->private_data_len + I40IW_MAX_IETF_SIZE;
35753616
35763617 status = i40iw_allocate_dma_mem(dev->hw, &iwqp->ietf_mem, buf_len, 1);
35773618
@@ -3605,18 +3646,10 @@ int i40iw_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) · 27 unmodified lines
36053646 iwqp->lsmm_mr = ibmr;
36063647 if (iwqp->page)
36073648 iwqp->sc_qp.qp_uk.sq_base = kmap(iwqp->page);
3608− if (is_remote_ne020_or_chelsio(cm_node))
3609− dev->iw_priv_qp_ops->qp_send_lsmm(
3610− &iwqp->sc_qp,
3649+ dev->iw_priv_qp_ops->qp_send_lsmm(&iwqp->sc_qp,
36113650 iwqp->ietf_mem.va,
36123651 (accept.size + conn_param->private_data_len),
36133652 ibmr->lkey);
3614− else
3615− dev->iw_priv_qp_ops->qp_send_lsmm(
3616− &iwqp->sc_qp,
3617− iwqp->ietf_mem.va,
3618− (accept.size + conn_param->private_data_len + MPA_ZERO_PAD_LEN),
3619− ibmr->lkey);
36203653
36213654 } else {
36223655 if (iwqp->page)
@@ -3714,6 +3747,7 @@ int i40iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) · 91 unmodified lines
37143747 struct sockaddr_in6 *raddr6;
37153748 bool qhash_set = false;
37163749 int apbvt_set = 0;
3750+ int err = 0;
37173751 enum i40iw_status_code status;
37183752
37193753 ibqp = i40iw_get_qp(cm_id->device, conn_param->qpn);
@@ -3759,6 +3793,10 @@ int i40iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) · 39 unmodified lines
37593793 i40iw_netdev_vlan_ipv6(cm_info.loc_addr, &cm_info.vlan_id, NULL);
37603794 }
37613795 cm_info.cm_id = cm_id;
3796+ cm_info.tos = cm_id->tos;
3797+ cm_info.user_pri = rt_tos2priority(cm_id->tos);
3798+ i40iw_debug(&iwdev->sc_dev, I40IW_DEBUG_DCB, "%s TOS:[%d] UP:[%d]\n",
3799+ __func__, cm_id->tos, cm_info.user_pri);
37623800 if ((cm_info.ipv4 && (laddr->sin_addr.s_addr != raddr->sin_addr.s_addr)) ||
37633801 (!cm_info.ipv4 && memcmp(laddr6->sin6_addr.in6_u.u6_addr32,
37643802 raddr6->sin6_addr.in6_u.u6_addr32,
@@ -3790,8 +3828,11 @@ int i40iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) · 25 unmodified lines
37903828 conn_param->private_data_len,
37913829 (void *)conn_param->private_data,
37923830 &cm_info);
3793− if (!cm_node)
3794− goto err;
3831+
3832+ if (IS_ERR(cm_node)) {
3833+ err = PTR_ERR(cm_node);
3834+ goto err_out;
3835+ }
37953836
37963837 i40iw_record_ird_ord(cm_node, (u16)conn_param->ird, (u16)conn_param->ord);
37973838 if (cm_node->send_rdma0_op == SEND_RDMA_READ_ZERO &&
@@ -3805,10 +3846,12 @@ int i40iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) · 7 unmodified lines
38053846 iwqp->cm_id = cm_id;
38063847 i40iw_add_ref(&iwqp->ibqp);
38073848
3808− if (cm_node->state == I40IW_CM_STATE_SYN_SENT) {
3809− if (i40iw_send_syn(cm_node, 0)) {
3849+ if (cm_node->state != I40IW_CM_STATE_OFFLOADED) {
3850+ cm_node->state = I40IW_CM_STATE_SYN_SENT;
3851+ err = i40iw_send_syn(cm_node, 0);
3852+ if (err) {
38103853 i40iw_rem_ref_cm_node(cm_node);
3811− goto err;
3854+ goto err_out;
38123855 }
38133856 }
38143857
@@ -3820,24 +3863,25 @@ int i40iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) · 5 unmodified lines
38203863 cm_node->cm_id);
38213864 return 0;
38223865
3823−err:
3824− if (cm_node) {
3825− if (cm_node->ipv4)
3826− i40iw_debug(cm_node->dev,
3827− I40IW_DEBUG_CM,
3828− "Api - connect() FAILED: dest addr=%pI4",
3829− cm_node->rem_addr);
3830− else
3831− i40iw_debug(cm_node->dev, I40IW_DEBUG_CM,
3832− "Api - connect() FAILED: dest addr=%pI6",
3833− cm_node->rem_addr);
3834− }
3835− i40iw_manage_qhash(iwdev,
3836− &cm_info,
3837− I40IW_QHASH_TYPE_TCP_ESTABLISHED,
3838− I40IW_QHASH_MANAGE_TYPE_DELETE,
3839− NULL,
3840− false);
3866+err_out:
3867+ if (cm_info.ipv4)
3868+ i40iw_debug(&iwdev->sc_dev,
3869+ I40IW_DEBUG_CM,
3870+ "Api - connect() FAILED: dest addr=%pI4",
3871+ cm_info.rem_addr);
3872+ else
3873+ i40iw_debug(&iwdev->sc_dev,
3874+ I40IW_DEBUG_CM,
3875+ "Api - connect() FAILED: dest addr=%pI6",
3876+ cm_info.rem_addr);
3877+
3878+ if (qhash_set)
3879+ i40iw_manage_qhash(iwdev,
3880+ &cm_info,
3881+ I40IW_QHASH_TYPE_TCP_ESTABLISHED,
3882+ I40IW_QHASH_MANAGE_TYPE_DELETE,
3883+ NULL,
3884+ false);
38413885
38423886 if (apbvt_set && !i40iw_listen_port_in_use(&iwdev->cm_core,
38433887 cm_info.loc_port))
@@ -3846,7 +3890,7 @@ err: · 2 unmodified lines
38463890 I40IW_MANAGE_APBVT_DEL);
38473891 cm_id->rem_ref(cm_id);
38483892 iwdev->cm_core.stats_connect_errs++;
3849− return -ENOMEM;
3893+ return err;
38503894 }
38513895
38523896 /**
@@ -3904,6 +3948,10 @@ int i40iw_create_listen(struct iw_cm_id *cm_id, int backlog) · 51 unmodified lines
39043948
39053949 cm_id->provider_data = cm_listen_node;
39063950
3951+ cm_listen_node->tos = cm_id->tos;
3952+ cm_listen_node->user_pri = rt_tos2priority(cm_id->tos);
3953+ cm_info.user_pri = cm_listen_node->user_pri;
3954+
39073955 if (!cm_listen_node->reused_node) {
39083956 if (wildcard) {
39093957 if (cm_info.ipv4)
@@ -4124,3 +4172,158 @@ static void i40iw_cm_post_event(struct i40iw_cm_event *event) · 214 unmodified lines
41244172
41254173 queue_work(event->cm_node->cm_core->event_wq, &event->event_work);
41264174 }
4175+
4176+/**
4177+ * i40iw_qhash_ctrl - enable/disable qhash for list
4178+ * @iwdev: device pointer
4179+ * @parent_listen_node: parent listen node
4180+ * @nfo: cm info node
4181+ * @ipaddr: Pointer to IPv4 or IPv6 address
4182+ * @ipv4: flag indicating IPv4 when true
4183+ * @ifup: flag indicating interface up when true
4184+ *
4185+ * Enables or disables the qhash for the node in the child
4186+ * listen list that matches ipaddr. If no matching IP was found
4187+ * it will allocate and add a new child listen node to the
4188+ * parent listen node. The listen_list_lock is assumed to be
4189+ * held when called.
4190+ */
4191+static void i40iw_qhash_ctrl(struct i40iw_device *iwdev,
4192+ struct i40iw_cm_listener *parent_listen_node,
4193+ struct i40iw_cm_info *nfo,
4194+ u32 *ipaddr, bool ipv4, bool ifup)
4195+{
4196+ struct list_head *child_listen_list = &parent_listen_node->child_listen_list;
4197+ struct i40iw_cm_listener *child_listen_node;
4198+ struct list_head *pos, *tpos;
4199+ enum i40iw_status_code ret;
4200+ bool node_allocated = false;
4201+ enum i40iw_quad_hash_manage_type op =
4202+ ifup ? I40IW_QHASH_MANAGE_TYPE_ADD : I40IW_QHASH_MANAGE_TYPE_DELETE;
4203+
4204+ list_for_each_safe(pos, tpos, child_listen_list) {
4205+ child_listen_node =
4206+ list_entry(pos,
4207+ struct i40iw_cm_listener,
4208+ child_listen_list);
4209+ if (!memcmp(child_listen_node->loc_addr, ipaddr, ipv4 ? 4 : 16))
4210+ goto set_qhash;
4211+ }
4212+
4213+ /* if not found then add a child listener if interface is going up */
4214+ if (!ifup)
4215+ return;
4216+ child_listen_node = kzalloc(sizeof(*child_listen_node), GFP_ATOMIC);
4217+ if (!child_listen_node)
4218+ return;
4219+ node_allocated = true;
4220+ memcpy(child_listen_node, parent_listen_node, sizeof(*child_listen_node));
4221+
4222+ memcpy(child_listen_node->loc_addr, ipaddr, ipv4 ? 4 : 16);
4223+
4224+set_qhash:
4225+ memcpy(nfo->loc_addr,
4226+ child_listen_node->loc_addr,
4227+ sizeof(nfo->loc_addr));
4228+ nfo->vlan_id = child_listen_node->vlan_id;
4229+ ret = i40iw_manage_qhash(iwdev, nfo,
4230+ I40IW_QHASH_TYPE_TCP_SYN,
4231+ op,
4232+ NULL, false);
4233+ if (!ret) {
4234+ child_listen_node->qhash_set = ifup;
4235+ if (node_allocated)
4236+ list_add(&child_listen_node->child_listen_list,
4237+ &parent_listen_node->child_listen_list);
4238+ } else if (node_allocated) {
4239+ kfree(child_listen_node);
4240+ }
4241+}
4242+
4243+/**
4244+ * i40iw_cm_disconnect_all - disconnect all connected qp's
4245+ * @iwdev: device pointer
4246+ */
4247+void i40iw_cm_disconnect_all(struct i40iw_device *iwdev)
4248+{
4249+ struct i40iw_cm_core *cm_core = &iwdev->cm_core;
4250+ struct list_head *list_core_temp;
4251+ struct list_head *list_node;
4252+ struct i40iw_cm_node *cm_node;
4253+ unsigned long flags;
4254+ struct list_head connected_list;
4255+ struct ib_qp_attr attr;
4256+
4257+ INIT_LIST_HEAD(&connected_list);
4258+ spin_lock_irqsave(&cm_core->ht_lock, flags);
4259+ list_for_each_safe(list_node, list_core_temp, &cm_core->connected_nodes) {
4260+ cm_node = container_of(list_node, struct i40iw_cm_node, list);
4261+ atomic_inc(&cm_node->ref_count);
4262+ list_add(&cm_node->connected_entry, &connected_list);
4263+ }
4264+ spin_unlock_irqrestore(&cm_core->ht_lock, flags);
4265+
4266+ list_for_each_safe(list_node, list_core_temp, &connected_list) {
4267+ cm_node = container_of(list_node, struct i40iw_cm_node, connected_entry);
4268+ attr.qp_state = IB_QPS_ERR;
4269+ i40iw_modify_qp(&cm_node->iwqp->ibqp, &attr, IB_QP_STATE, NULL);
4270+ i40iw_rem_ref_cm_node(cm_node);
4271+ }
4272+}
4273+
4274+/**
4275+ * i40iw_ifdown_notify - process an ifdown on an interface
4276+ * @iwdev: device pointer
4277+ * @ipaddr: Pointer to IPv4 or IPv6 address
4278+ * @ipv4: flag indicating IPv4 when true
4279+ * @ifup: flag indicating interface up when true
4280+ */
4281+void i40iw_if_notify(struct i40iw_device *iwdev, struct net_device *netdev,
4282+ u32 *ipaddr, bool ipv4, bool ifup)
4283+{
4284+ struct i40iw_cm_core *cm_core = &iwdev->cm_core;
4285+ unsigned long flags;
4286+ struct i40iw_cm_listener *listen_node;
4287+ static const u32 ip_zero[4] = { 0, 0, 0, 0 };
4288+ struct i40iw_cm_info nfo;
4289+ u16 vlan_id = rdma_vlan_dev_vlan_id(netdev);
4290+ enum i40iw_status_code ret;
4291+ enum i40iw_quad_hash_manage_type op =
4292+ ifup ? I40IW_QHASH_MANAGE_TYPE_ADD : I40IW_QHASH_MANAGE_TYPE_DELETE;
4293+
4294+ /* Disable or enable qhash for listeners */
4295+ spin_lock_irqsave(&cm_core->listen_list_lock, flags);
4296+ list_for_each_entry(listen_node, &cm_core->listen_nodes, list) {
4297+ if (vlan_id == listen_node->vlan_id &&
4298+ (!memcmp(listen_node->loc_addr, ipaddr, ipv4 ? 4 : 16) ||
4299+ !memcmp(listen_node->loc_addr, ip_zero, ipv4 ? 4 : 16))) {
4300+ memcpy(nfo.loc_addr, listen_node->loc_addr,
4301+ sizeof(nfo.loc_addr));
4302+ nfo.loc_port = listen_node->loc_port;
4303+ nfo.ipv4 = listen_node->ipv4;
4304+ nfo.vlan_id = listen_node->vlan_id;
4305+ nfo.user_pri = listen_node->user_pri;
4306+ if (!list_empty(&listen_node->child_listen_list)) {
4307+ i40iw_qhash_ctrl(iwdev,
4308+ listen_node,
4309+ &nfo,
4310+ ipaddr, ipv4, ifup);
4311+ } else if (memcmp(listen_node->loc_addr, ip_zero,
4312+ ipv4 ? 4 : 16)) {
4313+ ret = i40iw_manage_qhash(iwdev,
4314+ &nfo,
4315+ I40IW_QHASH_TYPE_TCP_SYN,
4316+ op,
4317+ NULL,
4318+ false);
4319+ if (!ret)
4320+ listen_node->qhash_set = ifup;
4321+ }
4322+ }
4323+ }
4324+ spin_unlock_irqrestore(&cm_core->listen_list_lock, flags);
4325+
4326+ /* disconnect any connected qp's on ifdown */
4327+ if (!ifup)
4328+ i40iw_cm_disconnect_all(iwdev);
4329+}
drivers/infiniband/hw/i40iw/i40iw_cm.h+8−3
@@ -56,8 +56,6 @@ · 55 unmodified lines
5656
5757 #define I40IW_MAX_IETF_SIZE 32
5858
59−#define MPA_ZERO_PAD_LEN 4
60−
6159 /* IETF RTR MSG Fields */
6260 #define IETF_PEER_TO_PEER 0x8000
6361 #define IETF_FLPDU_ZERO_LEN 0x4000
@@ -299,6 +297,7 @@ struct i40iw_cm_listener { · 235 unmodified lines
299297 enum i40iw_cm_listener_state listener_state;
300298 u32 reused_node;
301299 u8 user_pri;
300+ u8 tos;
302301 u16 vlan_id;
303302 bool qhash_set;
304303 bool ipv4;
@@ -341,9 +340,11 @@ struct i40iw_cm_node { · 36 unmodified lines
341340 int accept_pend;
342341 struct list_head timer_entry;
343342 struct list_head reset_entry;
343+ struct list_head connected_entry;
344344 atomic_t passive_state;
345345 bool qhash_set;
346346 u8 user_pri;
347+ u8 tos;
347348 bool ipv4;
348349 bool snd_mark_en;
349350 u16 lsmm_size;
@@ -368,7 +369,8 @@ struct i40iw_cm_info { · 18 unmodified lines
368369 u32 rem_addr[4];
369370 u16 vlan_id;
370371 int backlog;
371− u16 user_pri;
372+ u8 user_pri;
373+ u8 tos;
372374 bool ipv4;
373375 };
374376
@@ -445,4 +447,7 @@ int i40iw_arp_table(struct i40iw_device *iwdev, · 70 unmodified lines
445447 u8 *mac_addr,
446448 u32 action);
447449
450+void i40iw_if_notify(struct i40iw_device *iwdev, struct net_device *netdev,
451+ u32 *ipaddr, bool ipv4, bool ifup);
452+void i40iw_cm_disconnect_all(struct i40iw_device *iwdev);
448453 #endif /* I40IW_CM_H */
drivers/infiniband/hw/i40iw/i40iw_ctrl.c+424−222
@@ -103,6 +103,7 @@ static enum i40iw_status_code i40iw_cqp_poll_registers( · 102 unmodified lines
103103 if (newtail != tail) {
104104 /* SUCCESS */
105105 I40IW_RING_MOVE_TAIL(cqp->sq_ring);
106+ cqp->dev->cqp_cmd_stats[OP_COMPLETED_COMMANDS]++;
106107 return 0;
107108 }
108109 udelay(I40IW_SLEEP_COUNT);
@@ -222,6 +223,136 @@ static enum i40iw_status_code i40iw_sc_parse_fpm_query_buf( · 113 unmodified lines
222223 return 0;
223224 }
224225
226+/**
227+ * i40iw_fill_qos_list - Change all unknown qs handles to available ones
228+ * @qs_list: list of qs_handles to be fixed with valid qs_handles
229+ */
230+static void i40iw_fill_qos_list(u16 *qs_list)
231+{
232+ u16 qshandle = qs_list[0];
233+ int i;
234+
235+ for (i = 0; i < I40IW_MAX_USER_PRIORITY; i++) {
236+ if (qs_list[i] == QS_HANDLE_UNKNOWN)
237+ qs_list[i] = qshandle;
238+ else
239+ qshandle = qs_list[i];
240+ }
241+}
242+
243+/**
244+ * i40iw_qp_from_entry - Given entry, get to the qp structure
245+ * @entry: Points to list of qp structure
246+ */
247+static struct i40iw_sc_qp *i40iw_qp_from_entry(struct list_head *entry)
248+{
249+ if (!entry)
250+ return NULL;
251+
252+ return (struct i40iw_sc_qp *)((char *)entry - offsetof(struct i40iw_sc_qp, list));
253+}
254+
255+/**
256+ * i40iw_get_qp - get the next qp from the list given current qp
257+ * @head: Listhead of qp's
258+ * @qp: current qp
259+ */
260+static struct i40iw_sc_qp *i40iw_get_qp(struct list_head *head, struct i40iw_sc_qp *qp)
261+{
262+ struct list_head *entry = NULL;
263+ struct list_head *lastentry;
264+
265+ if (list_empty(head))
266+ return NULL;
267+
268+ if (!qp) {
269+ entry = head->next;
270+ } else {
271+ lastentry = &qp->list;
272+ entry = (lastentry != head) ? lastentry->next : NULL;
273+ }
274+
275+ return i40iw_qp_from_entry(entry);
276+}
277+
278+/**
279+ * i40iw_change_l2params - given the new l2 parameters, change all qp
280+ * @vsi: pointer to the vsi structure
281+ * @l2params: New paramaters from l2
282+ */
283+void i40iw_change_l2params(struct i40iw_sc_vsi *vsi, struct i40iw_l2params *l2params)
284+{
285+ struct i40iw_sc_dev *dev = vsi->dev;
286+ struct i40iw_sc_qp *qp = NULL;
287+ bool qs_handle_change = false;
288+ bool mss_change = false;
289+ unsigned long flags;
290+ u16 qs_handle;
291+ int i;
292+
293+ if (vsi->mss != l2params->mss) {
294+ mss_change = true;
295+ vsi->mss = l2params->mss;
296+ }
297+
298+ i40iw_fill_qos_list(l2params->qs_handle_list);
299+ for (i = 0; i < I40IW_MAX_USER_PRIORITY; i++) {
300+ qs_handle = l2params->qs_handle_list[i];
301+ if (vsi->qos[i].qs_handle != qs_handle)
302+ qs_handle_change = true;
303+ else if (!mss_change)
304+ continue; /* no MSS nor qs handle change */
305+ spin_lock_irqsave(&vsi->qos[i].lock, flags);
306+ qp = i40iw_get_qp(&vsi->qos[i].qplist, qp);
307+ while (qp) {
308+ if (mss_change)
309+ i40iw_qp_mss_modify(dev, qp);
310+ if (qs_handle_change) {
311+ qp->qs_handle = qs_handle;
312+ /* issue cqp suspend command */
313+ i40iw_qp_suspend_resume(dev, qp, true);
314+ }
315+ qp = i40iw_get_qp(&vsi->qos[i].qplist, qp);
316+ }
317+ spin_unlock_irqrestore(&vsi->qos[i].lock, flags);
318+ vsi->qos[i].qs_handle = qs_handle;
319+ }
320+}
321+
322+/**
323+ * i40iw_qp_rem_qos - remove qp from qos lists during destroy qp
324+ * @qp: qp to be removed from qos
325+ */
326+static void i40iw_qp_rem_qos(struct i40iw_sc_qp *qp)
327+{
328+ struct i40iw_sc_vsi *vsi = qp->vsi;
329+ unsigned long flags;
330+
331+ if (!qp->on_qoslist)
332+ return;
333+ spin_lock_irqsave(&vsi->qos[qp->user_pri].lock, flags);
334+ list_del(&qp->list);
335+ spin_unlock_irqrestore(&vsi->qos[qp->user_pri].lock, flags);
336+}
337+
338+/**
339+ * i40iw_qp_add_qos - called during setctx fot qp to be added to qos
340+ * @qp: qp to be added to qos
341+ */
342+void i40iw_qp_add_qos(struct i40iw_sc_qp *qp)
343+{
344+ struct i40iw_sc_vsi *vsi = qp->vsi;
345+ unsigned long flags;
346+
347+ if (qp->on_qoslist)
348+ return;
349+ spin_lock_irqsave(&vsi->qos[qp->user_pri].lock, flags);
350+ qp->qs_handle = vsi->qos[qp->user_pri].qs_handle;
351+ list_add(&qp->list, &vsi->qos[qp->user_pri].qplist);
352+ qp->on_qoslist = true;
353+ spin_unlock_irqrestore(&vsi->qos[qp->user_pri].lock, flags);
354+}
355+
225356 /**
226357 * i40iw_sc_pd_init - initialize sc pd struct
227358 * @dev: sc device struct
@@ -292,6 +423,9 @@ static enum i40iw_status_code i40iw_sc_cqp_init(struct i40iw_sc_cqp *cqp, · 64 unmodified lines
292423 info->dev->cqp = cqp;
293424
294425 I40IW_RING_INIT(cqp->sq_ring, cqp->sq_size);
426+ cqp->dev->cqp_cmd_stats[OP_REQUESTED_COMMANDS] = 0;
427+ cqp->dev->cqp_cmd_stats[OP_COMPLETED_COMMANDS] = 0;
428+
295429 i40iw_debug(cqp->dev, I40IW_DEBUG_WQE,
296430 "%s: sq_size[%04d] hw_sq_size[%04d] sq_base[%p] sq_pa[%llxh] cqp[%p] polarity[x%04X]\n",
297431 __func__, cqp->sq_size, cqp->hw_sq_size,
@@ -302,12 +436,10 @@ static enum i40iw_status_code i40iw_sc_cqp_init(struct i40iw_sc_cqp *cqp, · 4 unmodified lines
302436 /**
303437 * i40iw_sc_cqp_create - create cqp during bringup
304438 * @cqp: struct for cqp hw
305− * @disable_pfpdus: if pfpdu to be disabled
306439 * @maj_err: If error, major err number
307440 * @min_err: If error, minor err number
308441 */
309442 static enum i40iw_status_code i40iw_sc_cqp_create(struct i40iw_sc_cqp *cqp,
310− bool disable_pfpdus,
311443 u16 *maj_err,
312444 u16 *min_err)
313445 {
@@ -326,9 +458,6 @@ static enum i40iw_status_code i40iw_sc_cqp_create(struct i40iw_sc_cqp *cqp, · 12 unmodified lines
326458 temp = LS_64(cqp->hw_sq_size, I40IW_CQPHC_SQSIZE) |
327459 LS_64(cqp->struct_ver, I40IW_CQPHC_SVER);
328460
329− if (disable_pfpdus)
330− temp |= LS_64(1, I40IW_CQPHC_DISABLE_PFPDUS);
331−
332461 set_64bit_val(cqp->host_ctx, 0, temp);
333462 set_64bit_val(cqp->host_ctx, 8, cqp->sq_pa);
334463 temp = LS_64(cqp->enabled_vf_count, I40IW_CQPHC_ENABLED_VFS) |
@@ -424,6 +553,7 @@ u64 *i40iw_sc_cqp_get_next_send_wqe(struct i40iw_sc_cqp *cqp, u64 scratch) · 89 unmodified lines
424553 return NULL;
425554 }
426555 I40IW_ATOMIC_RING_MOVE_HEAD(cqp->sq_ring, wqe_idx, ret_code);
556+ cqp->dev->cqp_cmd_stats[OP_REQUESTED_COMMANDS]++;
427557 if (ret_code)
428558 return NULL;
429559 if (!wqe_idx)
@@ -559,6 +689,8 @@ static enum i40iw_status_code i40iw_sc_ccq_get_cqe_info( · 129 unmodified lines
559689 I40IW_RING_GETCURRENT_HEAD(ccq->cq_uk.cq_ring));
560690 wmb(); /* write shadow area before tail */
561691 I40IW_RING_MOVE_TAIL(cqp->sq_ring);
692+ ccq->dev->cqp_cmd_stats[OP_COMPLETED_COMMANDS]++;
693+
562694 return ret_code;
563695 }
564696
@@ -1051,6 +1183,7 @@ static enum i40iw_status_code i40iw_sc_manage_qhash_table_entry( · 486 unmodified lines
10511183 u64 qw1 = 0;
10521184 u64 qw2 = 0;
10531185 u64 temp;
1186+ struct i40iw_sc_vsi *vsi = info->vsi;
10541187
10551188 wqe = i40iw_sc_cqp_get_next_send_wqe(cqp, scratch);
10561189 if (!wqe)
@@ -1082,7 +1215,7 @@ static enum i40iw_status_code i40iw_sc_manage_qhash_table_entry( · 25 unmodified lines
10821215 LS_64(info->dest_ip[2], I40IW_CQPSQ_QHASH_ADDR2) |
10831216 LS_64(info->dest_ip[3], I40IW_CQPSQ_QHASH_ADDR3));
10841217 }
1085− qw2 = LS_64(cqp->dev->qs_handle, I40IW_CQPSQ_QHASH_QS_HANDLE);
1218+ qw2 = LS_64(vsi->qos[info->user_pri].qs_handle, I40IW_CQPSQ_QHASH_QS_HANDLE);
10861219 if (info->vlan_valid)
10871220 qw2 |= LS_64(info->vlan_id, I40IW_CQPSQ_QHASH_VLANID);
10881221 set_64bit_val(wqe, 16, qw2);
@@ -2103,6 +2236,7 @@ static enum i40iw_status_code i40iw_sc_qp_init(struct i40iw_sc_qp *qp, · 1014 unmodified lines
21032236 u32 offset;
21042237
21052238 qp->dev = info->pd->dev;
2239+ qp->vsi = info->vsi;
21062240 qp->sq_pa = info->sq_pa;
21072241 qp->rq_pa = info->rq_pa;
21082242 qp->hw_host_ctx_pa = info->host_ctx_pa;
@@ -2151,7 +2285,7 @@ static enum i40iw_status_code i40iw_sc_qp_init(struct i40iw_sc_qp *qp, · 42 unmodified lines
21512285 qp->rq_tph_en = info->rq_tph_en;
21522286 qp->rcv_tph_en = info->rcv_tph_en;
21532287 qp->xmit_tph_en = info->xmit_tph_en;
2154− qp->qs_handle = qp->pd->dev->qs_handle;
2288+ qp->qs_handle = qp->vsi->qos[qp->user_pri].qs_handle;
21552289 qp->exception_lan_queue = qp->pd->dev->exception_lan_queue;
21562290
21572291 return 0;
@@ -2296,6 +2430,7 @@ static enum i40iw_status_code i40iw_sc_qp_destroy( · 138 unmodified lines
22962430 struct i40iw_sc_cqp *cqp;
22972431 u64 header;
22982432
2433+ i40iw_qp_rem_qos(qp);
22992434 cqp = qp->pd->dev->cqp;
23002435 wqe = i40iw_sc_cqp_get_next_send_wqe(cqp, scratch);
23012436 if (!wqe)
@@ -2443,10 +2578,20 @@ static enum i40iw_status_code i40iw_sc_qp_setctx( · 141 unmodified lines
24432578 {
24442579 struct i40iwarp_offload_info *iw;
24452580 struct i40iw_tcp_offload_info *tcp;
2581+ struct i40iw_sc_vsi *vsi;
2582+ struct i40iw_sc_dev *dev;
24462583 u64 qw0, qw3, qw7 = 0;
24472584
24482585 iw = info->iwarp_info;
24492586 tcp = info->tcp_info;
2587+ vsi = qp->vsi;
2588+ dev = qp->dev;
2589+ if (info->add_to_qoslist) {
2590+ qp->user_pri = info->user_pri;
2591+ i40iw_qp_add_qos(qp);
2592+ i40iw_debug(qp->dev, I40IW_DEBUG_DCB, "%s qp[%d] UP[%d] qset[%d]\n",
2593+ __func__, qp->qp_uk.qp_id, qp->user_pri, qp->qs_handle);
2594+ }
24502595 qw0 = LS_64(qp->qp_uk.rq_wqe_size, I40IWQPC_RQWQESIZE) |
24512596 LS_64(info->err_rq_idx_valid, I40IWQPC_ERR_RQ_IDX_VALID) |
24522597 LS_64(qp->rcv_tph_en, I40IWQPC_RCVTPHEN) |
@@ -2487,16 +2632,14 @@ static enum i40iw_status_code i40iw_sc_qp_setctx( · 34 unmodified lines
24872632 LS_64(iw->rdmap_ver, I40IWQPC_RDMAP_VER);
24882633
24892634 qw7 |= LS_64(iw->pd_id, I40IWQPC_PDIDX);
2490− set_64bit_val(qp_ctx, 144, qp->q2_pa);
2635+ set_64bit_val(qp_ctx,
2636+ 144,
2637+ LS_64(qp->q2_pa, I40IWQPC_Q2ADDR) |
2638+ LS_64(vsi->fcn_id, I40IWQPC_STAT_INDEX));
24912639 set_64bit_val(qp_ctx,
24922640 152,
24932641 LS_64(iw->last_byte_sent, I40IWQPC_LASTBYTESENT));
24942642
2495− /*
2496− * Hard-code IRD_SIZE to hw-limit, 128, in qpctx, i.e matching an
2497− *advertisable IRD of 64
2498− */
2499− iw->ird_size = I40IW_QPCTX_ENCD_MAXIRD;
25002643 set_64bit_val(qp_ctx,
25012644 160,
25022645 LS_64(iw->ord_size, I40IWQPC_ORDSIZE) |
@@ -2507,6 +2650,9 @@ static enum i40iw_status_code i40iw_sc_qp_setctx( · 4 unmodified lines
25072650 LS_64(iw->bind_en, I40IWQPC_BINDEN) |
25082651 LS_64(iw->fast_reg_en, I40IWQPC_FASTREGEN) |
25092652 LS_64(iw->priv_mode_en, I40IWQPC_PRIVEN) |
2653+ LS_64((((vsi->stats_fcn_id_alloc) &&
2654+ (dev->is_pf) && (vsi->fcn_id >= I40IW_FIRST_NON_PF_STAT)) ? 1 : 0),
2655+ I40IWQPC_USESTATSINSTANCE) |
25102656 LS_64(1, I40IWQPC_IWARPMODE) |
25112657 LS_64(iw->rcv_mark_en, I40IWQPC_RCVMARKERS) |
25122658 LS_64(iw->align_hdrs, I40IWQPC_ALIGNHDRS) |
@@ -2623,7 +2769,9 @@ static enum i40iw_status_code i40iw_sc_alloc_stag( · 110 unmodified lines
26232769 u64 *wqe;
26242770 struct i40iw_sc_cqp *cqp;
26252771 u64 header;
2772+ enum i40iw_page_size page_size;
26262773
2774+ page_size = (info->page_size == 0x200000) ? I40IW_PAGE_SIZE_2M : I40IW_PAGE_SIZE_4K;
26272775 cqp = dev->cqp;
26282776 wqe = i40iw_sc_cqp_get_next_send_wqe(cqp, scratch);
26292777 if (!wqe)
@@ -2643,7 +2791,7 @@ static enum i40iw_status_code i40iw_sc_alloc_stag( · 13 unmodified lines
26432791 LS_64(1, I40IW_CQPSQ_STAG_MR) |
26442792 LS_64(info->access_rights, I40IW_CQPSQ_STAG_ARIGHTS) |
26452793 LS_64(info->chunk_size, I40IW_CQPSQ_STAG_LPBLSIZE) |
2646− LS_64(info->page_size, I40IW_CQPSQ_STAG_HPAGESIZE) |
2794+ LS_64(page_size, I40IW_CQPSQ_STAG_HPAGESIZE) |
26472795 LS_64(info->remote_access, I40IW_CQPSQ_STAG_REMACCENABLED) |
26482796 LS_64(info->use_hmc_fcn_index, I40IW_CQPSQ_STAG_USEHMCFNIDX) |
26492797 LS_64(info->use_pf_rid, I40IW_CQPSQ_STAG_USEPFRID) |
@@ -2679,7 +2827,9 @@ static enum i40iw_status_code i40iw_sc_mr_reg_non_shared( · 29 unmodified lines
26792827 u32 pble_obj_cnt;
26802828 bool remote_access;
26812829 u8 addr_type;
2830+ enum i40iw_page_size page_size;
26822831
2832+ page_size = (info->page_size == 0x200000) ? I40IW_PAGE_SIZE_2M : I40IW_PAGE_SIZE_4K;
26832833 if (info->access_rights & (I40IW_ACCESS_FLAGS_REMOTEREAD_ONLY |
26842834 I40IW_ACCESS_FLAGS_REMOTEWRITE_ONLY))
26852835 remote_access = true;
@@ -2722,7 +2872,7 @@ static enum i40iw_status_code i40iw_sc_mr_reg_non_shared( · 36 unmodified lines
27222872 header = LS_64(I40IW_CQP_OP_REG_MR, I40IW_CQPSQ_OPCODE) |
27232873 LS_64(1, I40IW_CQPSQ_STAG_MR) |
27242874 LS_64(info->chunk_size, I40IW_CQPSQ_STAG_LPBLSIZE) |
2725− LS_64(info->page_size, I40IW_CQPSQ_STAG_HPAGESIZE) |
2875+ LS_64(page_size, I40IW_CQPSQ_STAG_HPAGESIZE) |
27262876 LS_64(info->access_rights, I40IW_CQPSQ_STAG_ARIGHTS) |
27272877 LS_64(remote_access, I40IW_CQPSQ_STAG_REMACCENABLED) |
27282878 LS_64(addr_type, I40IW_CQPSQ_STAG_VABASEDTO) |
@@ -2937,7 +3087,9 @@ enum i40iw_status_code i40iw_sc_mr_fast_register( · 208 unmodified lines
29373087 u64 temp, header;
29383088 u64 *wqe;
29393089 u32 wqe_idx;
3090+ enum i40iw_page_size page_size;
29403091
3092+ page_size = (info->page_size == 0x200000) ? I40IW_PAGE_SIZE_2M : I40IW_PAGE_SIZE_4K;
29413093 wqe = i40iw_qp_get_next_send_wqe(&qp->qp_uk, &wqe_idx, I40IW_QP_WQE_MIN_SIZE,
29423094 0, info->wr_id);
29433095 if (!wqe)
@@ -2964,7 +3116,7 @@ enum i40iw_status_code i40iw_sc_mr_fast_register( · 20 unmodified lines
29643116 LS_64(info->stag_idx, I40IWQPSQ_STAGINDEX) |
29653117 LS_64(I40IWQP_OP_FAST_REGISTER, I40IWQPSQ_OPCODE) |
29663118 LS_64(info->chunk_size, I40IWQPSQ_LPBLSIZE) |
2967− LS_64(info->page_size, I40IWQPSQ_HPAGESIZE) |
3119+ LS_64(page_size, I40IWQPSQ_HPAGESIZE) |
29683120 LS_64(info->access_rights, I40IWQPSQ_STAGRIGHTS) |
29693121 LS_64(info->addr_type, I40IWQPSQ_VABASEDTO) |
29703122 LS_64(info->read_fence, I40IWQPSQ_READFENCE) |
@@ -3959,7 +4111,7 @@ enum i40iw_status_code i40iw_process_cqp_cmd(struct i40iw_sc_dev *dev, · 988 unmodified lines
39594111 struct cqp_commands_info *pcmdinfo)
39604112 {
39614113 enum i40iw_status_code status = 0;
3962− unsigned long flags;
4114+ unsigned long flags;
39634115
39644116 spin_lock_irqsave(&dev->cqp_lock, flags);
39654117 if (list_empty(&dev->cqp_cmd_head) && !i40iw_ring_full(dev->cqp))
@@ -3978,7 +4130,7 @@ enum i40iw_status_code i40iw_process_bh(struct i40iw_sc_dev *dev) · 12 unmodified lines
39784130 {
39794131 enum i40iw_status_code status = 0;
39804132 struct cqp_commands_info *pcmdinfo;
3981− unsigned long flags;
4133+ unsigned long flags;
39824134
39834135 spin_lock_irqsave(&dev->cqp_lock, flags);
39844136 while (!list_empty(&dev->cqp_cmd_head) && !i40iw_ring_full(dev->cqp)) {
@@ -4055,7 +4207,6 @@ static int i40iw_bld_terminate_hdr(struct i40iw_sc_qp *qp, · 70 unmodified lines
40554207 u16 ddp_seg_len;
40564208 int copy_len = 0;
40574209 u8 is_tagged = 0;
4058− enum i40iw_flush_opcode flush_code = FLUSH_INVALID;
40594210 u32 opcode;
40604211 struct i40iw_terminate_hdr *termhdr;
40614212
@@ -4228,9 +4379,6 @@ static int i40iw_bld_terminate_hdr(struct i40iw_sc_qp *qp, · 166 unmodified lines
42284379 if (copy_len)
42294380 memcpy(termhdr + 1, pkt, copy_len);
42304381
4231− if (flush_code && !info->in_rdrsp_wr)
4232− qp->sq_flush = (info->sq) ? true : false;
4233−
42344382 return sizeof(struct i40iw_terminate_hdr) + copy_len;
42354383 }
42364384
@@ -4321,286 +4469,370 @@ void i40iw_terminate_received(struct i40iw_sc_qp *qp, struct i40iw_aeqe_info *in · 84 unmodified lines
43214469 }
43224470
43234471 /**
4324− * i40iw_hw_stat_init - Initiliaze HW stats table
4325− * @devstat: pestat struct
4472+ * i40iw_sc_vsi_init - Initialize virtual device
4473+ * @vsi: pointer to the vsi structure
4474+ * @info: parameters to initialize vsi
4475+ **/
4476+void i40iw_sc_vsi_init(struct i40iw_sc_vsi *vsi, struct i40iw_vsi_init_info *info)
4477+{
4478+ int i;
4479+
4480+ vsi->dev = info->dev;
4481+ vsi->back_vsi = info->back_vsi;
4482+ vsi->mss = info->params->mss;
4483+ i40iw_fill_qos_list(info->params->qs_handle_list);
4484+
4485+ for (i = 0; i < I40IW_MAX_USER_PRIORITY; i++) {
4486+ vsi->qos[i].qs_handle =
4487+ info->params->qs_handle_list[i];
4488+ i40iw_debug(vsi->dev, I40IW_DEBUG_DCB, "qset[%d]: %d\n", i, vsi->qos[i].qs_handle);
4489+ spin_lock_init(&vsi->qos[i].lock);
4490+ INIT_LIST_HEAD(&vsi->qos[i].qplist);
4491+ }
4492+}
4493+
4494+/**
4495+ * i40iw_hw_stats_init - Initiliaze HW stats table
4496+ * @stats: pestat struct
43264497 * @fcn_idx: PCI fn id
4327− * @hw: PF i40iw_hw structure.
43284498 * @is_pf: Is it a PF?
43294499 *
4330− * Populate the HW stat table with register offset addr for each
4331− * stat. And start the perioidic stats timer.
4500+ * Populate the HW stats table with register offset addr for each
4501+ * stats. And start the perioidic stats timer.
43324502 */
4333−static void i40iw_hw_stat_init(struct i40iw_dev_pestat *devstat,
4334− u8 fcn_idx,
4335− struct i40iw_hw *hw, bool is_pf)
4503+void i40iw_hw_stats_init(struct i40iw_vsi_pestat *stats, u8 fcn_idx, bool is_pf)
43364504 {
4337− u32 stat_reg_offset;
4338− u32 stat_index;
4339− struct i40iw_dev_hw_stat_offsets *stat_table =
4340− &devstat->hw_stat_offsets;
4341− struct i40iw_dev_hw_stats *last_rd_stats = &devstat->last_read_hw_stats;
4342−
4343− devstat->hw = hw;
4505+ u32 stats_reg_offset;
4506+ u32 stats_index;
4507+ struct i40iw_dev_hw_stats_offsets *stats_table =
4508+ &stats->hw_stats_offsets;
4509+ struct i40iw_dev_hw_stats *last_rd_stats = &stats->last_read_hw_stats;
43444510
43454511 if (is_pf) {
4346− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_IP4RXDISCARD] =
4512+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_IP4RXDISCARD] =
43474513 I40E_GLPES_PFIP4RXDISCARD(fcn_idx);
4348− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_IP4RXTRUNC] =
4514+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_IP4RXTRUNC] =
43494515 I40E_GLPES_PFIP4RXTRUNC(fcn_idx);
4350− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_IP4TXNOROUTE] =
4516+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_IP4TXNOROUTE] =
43514517 I40E_GLPES_PFIP4TXNOROUTE(fcn_idx);
4352− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_IP6RXDISCARD] =
4518+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_IP6RXDISCARD] =
43534519 I40E_GLPES_PFIP6RXDISCARD(fcn_idx);
4354− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_IP6RXTRUNC] =
4520+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_IP6RXTRUNC] =
43554521 I40E_GLPES_PFIP6RXTRUNC(fcn_idx);
4356− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_IP6TXNOROUTE] =
4522+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_IP6TXNOROUTE] =
43574523 I40E_GLPES_PFIP6TXNOROUTE(fcn_idx);
4358− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_TCPRTXSEG] =
4524+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_TCPRTXSEG] =
43594525 I40E_GLPES_PFTCPRTXSEG(fcn_idx);
4360− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_TCPRXOPTERR] =
4526+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_TCPRXOPTERR] =
43614527 I40E_GLPES_PFTCPRXOPTERR(fcn_idx);
4362− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_TCPRXPROTOERR] =
4528+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_TCPRXPROTOERR] =
43634529 I40E_GLPES_PFTCPRXPROTOERR(fcn_idx);
43644530
4365− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4RXOCTS] =
4531+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4RXOCTS] =
43664532 I40E_GLPES_PFIP4RXOCTSLO(fcn_idx);
4367− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4RXPKTS] =
4533+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4RXPKTS] =
43684534 I40E_GLPES_PFIP4RXPKTSLO(fcn_idx);
4369− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4RXFRAGS] =
4535+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4RXFRAGS] =
43704536 I40E_GLPES_PFIP4RXFRAGSLO(fcn_idx);
4371− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4RXMCPKTS] =
4537+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4RXMCPKTS] =
43724538 I40E_GLPES_PFIP4RXMCPKTSLO(fcn_idx);
4373− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4TXOCTS] =
4539+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4TXOCTS] =
43744540 I40E_GLPES_PFIP4TXOCTSLO(fcn_idx);
4375− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4TXPKTS] =
4541+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4TXPKTS] =
43764542 I40E_GLPES_PFIP4TXPKTSLO(fcn_idx);
4377− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4TXFRAGS] =
4543+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4TXFRAGS] =
43784544 I40E_GLPES_PFIP4TXFRAGSLO(fcn_idx);
4379− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4TXMCPKTS] =
4545+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4TXMCPKTS] =
43804546 I40E_GLPES_PFIP4TXMCPKTSLO(fcn_idx);
4381− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6RXOCTS] =
4547+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6RXOCTS] =
43824548 I40E_GLPES_PFIP6RXOCTSLO(fcn_idx);
4383− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6RXPKTS] =
4549+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6RXPKTS] =
43844550 I40E_GLPES_PFIP6RXPKTSLO(fcn_idx);
4385− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6RXFRAGS] =
4551+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6RXFRAGS] =
43864552 I40E_GLPES_PFIP6RXFRAGSLO(fcn_idx);
4387− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6RXMCPKTS] =
4553+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6RXMCPKTS] =
43884554 I40E_GLPES_PFIP6RXMCPKTSLO(fcn_idx);
4389− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6TXOCTS] =
4555+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6TXOCTS] =
43904556 I40E_GLPES_PFIP6TXOCTSLO(fcn_idx);
4391− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6TXPKTS] =
4557+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6TXPKTS] =
43924558 I40E_GLPES_PFIP6TXPKTSLO(fcn_idx);
4393− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6TXPKTS] =
4559+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6TXPKTS] =
43944560 I40E_GLPES_PFIP6TXPKTSLO(fcn_idx);
4395− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6TXFRAGS] =
4561+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6TXFRAGS] =
43964562 I40E_GLPES_PFIP6TXFRAGSLO(fcn_idx);
4397− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_TCPRXSEGS] =
4563+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_TCPRXSEGS] =
43984564 I40E_GLPES_PFTCPRXSEGSLO(fcn_idx);
4399− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_TCPTXSEG] =
4565+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_TCPTXSEG] =
44004566 I40E_GLPES_PFTCPTXSEGLO(fcn_idx);
4401− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMARXRDS] =
4567+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMARXRDS] =
44024568 I40E_GLPES_PFRDMARXRDSLO(fcn_idx);
4403− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMARXSNDS] =
4569+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMARXSNDS] =
44044570 I40E_GLPES_PFRDMARXSNDSLO(fcn_idx);
4405− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMARXWRS] =
4571+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMARXWRS] =
44064572 I40E_GLPES_PFRDMARXWRSLO(fcn_idx);
4407− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMATXRDS] =
4573+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMATXRDS] =
44084574 I40E_GLPES_PFRDMATXRDSLO(fcn_idx);
4409− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMATXSNDS] =
4575+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMATXSNDS] =
44104576 I40E_GLPES_PFRDMATXSNDSLO(fcn_idx);
4411− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMATXWRS] =
4577+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMATXWRS] =
44124578 I40E_GLPES_PFRDMATXWRSLO(fcn_idx);
4413− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMAVBND] =
4579+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMAVBND] =
44144580 I40E_GLPES_PFRDMAVBNDLO(fcn_idx);
4415− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMAVINV] =
4581+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMAVINV] =
44164582 I40E_GLPES_PFRDMAVINVLO(fcn_idx);
44174583 } else {
4418− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_IP4RXDISCARD] =
4584+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_IP4RXDISCARD] =
44194585 I40E_GLPES_VFIP4RXDISCARD(fcn_idx);
4420− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_IP4RXTRUNC] =
4586+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_IP4RXTRUNC] =
44214587 I40E_GLPES_VFIP4RXTRUNC(fcn_idx);
4422− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_IP4TXNOROUTE] =
4588+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_IP4TXNOROUTE] =
44234589 I40E_GLPES_VFIP4TXNOROUTE(fcn_idx);
4424− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_IP6RXDISCARD] =
4590+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_IP6RXDISCARD] =
44254591 I40E_GLPES_VFIP6RXDISCARD(fcn_idx);
4426− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_IP6RXTRUNC] =
4592+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_IP6RXTRUNC] =
44274593 I40E_GLPES_VFIP6RXTRUNC(fcn_idx);
4428− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_IP6TXNOROUTE] =
4594+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_IP6TXNOROUTE] =
44294595 I40E_GLPES_VFIP6TXNOROUTE(fcn_idx);
4430− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_TCPRTXSEG] =
4596+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_TCPRTXSEG] =
44314597 I40E_GLPES_VFTCPRTXSEG(fcn_idx);
4432− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_TCPRXOPTERR] =
4598+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_TCPRXOPTERR] =
44334599 I40E_GLPES_VFTCPRXOPTERR(fcn_idx);
4434− stat_table->stat_offset_32[I40IW_HW_STAT_INDEX_TCPRXPROTOERR] =
4600+ stats_table->stats_offset_32[I40IW_HW_STAT_INDEX_TCPRXPROTOERR] =
44354601 I40E_GLPES_VFTCPRXPROTOERR(fcn_idx);
44364602
4437− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4RXOCTS] =
4603+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4RXOCTS] =
44384604 I40E_GLPES_VFIP4RXOCTSLO(fcn_idx);
4439− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4RXPKTS] =
4605+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4RXPKTS] =
44404606 I40E_GLPES_VFIP4RXPKTSLO(fcn_idx);
4441− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4RXFRAGS] =
4607+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4RXFRAGS] =
44424608 I40E_GLPES_VFIP4RXFRAGSLO(fcn_idx);
4443− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4RXMCPKTS] =
4609+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4RXMCPKTS] =
44444610 I40E_GLPES_VFIP4RXMCPKTSLO(fcn_idx);
4445− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4TXOCTS] =
4611+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4TXOCTS] =
44464612 I40E_GLPES_VFIP4TXOCTSLO(fcn_idx);
4447− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4TXPKTS] =
4613+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4TXPKTS] =
44484614 I40E_GLPES_VFIP4TXPKTSLO(fcn_idx);
4449− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4TXFRAGS] =
4615+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4TXFRAGS] =
44504616 I40E_GLPES_VFIP4TXFRAGSLO(fcn_idx);
4451− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP4TXMCPKTS] =
4617+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP4TXMCPKTS] =
44524618 I40E_GLPES_VFIP4TXMCPKTSLO(fcn_idx);
4453− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6RXOCTS] =
4619+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6RXOCTS] =
44544620 I40E_GLPES_VFIP6RXOCTSLO(fcn_idx);
4455− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6RXPKTS] =
4621+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6RXPKTS] =
44564622 I40E_GLPES_VFIP6RXPKTSLO(fcn_idx);
4457− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6RXFRAGS] =
4623+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6RXFRAGS] =
44584624 I40E_GLPES_VFIP6RXFRAGSLO(fcn_idx);
4459− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6RXMCPKTS] =
4625+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6RXMCPKTS] =
44604626 I40E_GLPES_VFIP6RXMCPKTSLO(fcn_idx);
4461− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6TXOCTS] =
4627+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6TXOCTS] =
44624628 I40E_GLPES_VFIP6TXOCTSLO(fcn_idx);
4463− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6TXPKTS] =
4629+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6TXPKTS] =
44644630 I40E_GLPES_VFIP6TXPKTSLO(fcn_idx);
4465− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6TXPKTS] =
4631+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6TXPKTS] =
44664632 I40E_GLPES_VFIP6TXPKTSLO(fcn_idx);
4467− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_IP6TXFRAGS] =
4633+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_IP6TXFRAGS] =
44684634 I40E_GLPES_VFIP6TXFRAGSLO(fcn_idx);
4469− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_TCPRXSEGS] =
4635+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_TCPRXSEGS] =
44704636 I40E_GLPES_VFTCPRXSEGSLO(fcn_idx);
4471− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_TCPTXSEG] =
4637+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_TCPTXSEG] =
44724638 I40E_GLPES_VFTCPTXSEGLO(fcn_idx);
4473− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMARXRDS] =
4639+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMARXRDS] =
44744640 I40E_GLPES_VFRDMARXRDSLO(fcn_idx);
4475− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMARXSNDS] =
4641+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMARXSNDS] =
44764642 I40E_GLPES_VFRDMARXSNDSLO(fcn_idx);
4477− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMARXWRS] =
4643+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMARXWRS] =
44784644 I40E_GLPES_VFRDMARXWRSLO(fcn_idx);
4479− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMATXRDS] =
4645+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMATXRDS] =
44804646 I40E_GLPES_VFRDMATXRDSLO(fcn_idx);
4481− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMATXSNDS] =
4647+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMATXSNDS] =
44824648 I40E_GLPES_VFRDMATXSNDSLO(fcn_idx);
4483− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMATXWRS] =
4649+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMATXWRS] =
44844650 I40E_GLPES_VFRDMATXWRSLO(fcn_idx);
4485− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMAVBND] =
4651+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMAVBND] =
44864652 I40E_GLPES_VFRDMAVBNDLO(fcn_idx);
4487− stat_table->stat_offset_64[I40IW_HW_STAT_INDEX_RDMAVINV] =
4653+ stats_table->stats_offset_64[I40IW_HW_STAT_INDEX_RDMAVINV] =
44884654 I40E_GLPES_VFRDMAVINVLO(fcn_idx);
44894655 }
44904656
4491− for (stat_index = 0; stat_index < I40IW_HW_STAT_INDEX_MAX_64;
4492− stat_index++) {
4493− stat_reg_offset = stat_table->stat_offset_64[stat_index];
4494− last_rd_stats->stat_value_64[stat_index] =
4495− readq(devstat->hw->hw_addr + stat_reg_offset);
4657+ for (stats_index = 0; stats_index < I40IW_HW_STAT_INDEX_MAX_64;
4658+ stats_index++) {
4659+ stats_reg_offset = stats_table->stats_offset_64[stats_index];
4660+ last_rd_stats->stats_value_64[stats_index] =
4661+ readq(stats->hw->hw_addr + stats_reg_offset);
44964662 }
44974663
4498− for (stat_index = 0; stat_index < I40IW_HW_STAT_INDEX_MAX_32;
4499− stat_index++) {
4500− stat_reg_offset = stat_table->stat_offset_32[stat_index];
4501− last_rd_stats->stat_value_32[stat_index] =
4502− i40iw_rd32(devstat->hw, stat_reg_offset);
4664+ for (stats_index = 0; stats_index < I40IW_HW_STAT_INDEX_MAX_32;
4665+ stats_index++) {
4666+ stats_reg_offset = stats_table->stats_offset_32[stats_index];
4667+ last_rd_stats->stats_value_32[stats_index] =
4668+ i40iw_rd32(stats->hw, stats_reg_offset);
45034669 }
45044670 }
45054671
45064672 /**
4507− * i40iw_hw_stat_read_32 - Read 32-bit HW stat counters and accommodates for roll-overs.
4508− * @devstat: pestat struct
4509− * @index: index in HW stat table which contains offset reg-addr
4510− * @value: hw stat value
4673+ * i40iw_hw_stats_read_32 - Read 32-bit HW stats counters and accommodates for roll-overs.
4674+ * @stat: pestat struct
4675+ * @index: index in HW stats table which contains offset reg-addr
4676+ * @value: hw stats value
45114677 */
4512−static void i40iw_hw_stat_read_32(struct i40iw_dev_pestat *devstat,
4513− enum i40iw_hw_stat_index_32b index,
4514− u64 *value)
4678+void i40iw_hw_stats_read_32(struct i40iw_vsi_pestat *stats,
4679+ enum i40iw_hw_stats_index_32b index,
4680+ u64 *value)
45154681 {
4516− struct i40iw_dev_hw_stat_offsets *stat_table =
4517− &devstat->hw_stat_offsets;
4518− struct i40iw_dev_hw_stats *last_rd_stats = &devstat->last_read_hw_stats;
4519− struct i40iw_dev_hw_stats *hw_stats = &devstat->hw_stats;
4520− u64 new_stat_value = 0;
4521− u32 stat_reg_offset = stat_table->stat_offset_32[index];
4522−
4523− new_stat_value = i40iw_rd32(devstat->hw, stat_reg_offset);
4682+ struct i40iw_dev_hw_stats_offsets *stats_table =
4683+ &stats->hw_stats_offsets;
4684+ struct i40iw_dev_hw_stats *last_rd_stats = &stats->last_read_hw_stats;
4685+ struct i40iw_dev_hw_stats *hw_stats = &stats->hw_stats;
4686+ u64 new_stats_value = 0;
4687+ u32 stats_reg_offset = stats_table->stats_offset_32[index];
4688+
4689+ new_stats_value = i40iw_rd32(stats->hw, stats_reg_offset);
45244690 /*roll-over case */
4525− if (new_stat_value < last_rd_stats->stat_value_32[index])
4526− hw_stats->stat_value_32[index] += new_stat_value;
4691+ if (new_stats_value < last_rd_stats->stats_value_32[index])
4692+ hw_stats->stats_value_32[index] += new_stats_value;
45274693 else
4528− hw_stats->stat_value_32[index] +=
4529− new_stat_value - last_rd_stats->stat_value_32[index];
4530− last_rd_stats->stat_value_32[index] = new_stat_value;
4531− *value = hw_stats->stat_value_32[index];
4694+ hw_stats->stats_value_32[index] +=
4695+ new_stats_value - last_rd_stats->stats_value_32[index];
4696+ last_rd_stats->stats_value_32[index] = new_stats_value;
4697+ *value = hw_stats->stats_value_32[index];
45324698 }
45334699
45344700 /**
4535− * i40iw_hw_stat_read_64 - Read HW stat counters (greater than 32-bit) and accommodates for roll-overs.
4536− * @devstat: pestat struct
4537− * @index: index in HW stat table which contains offset reg-addr
4538− * @value: hw stat value
4701+ * i40iw_hw_stats_read_64 - Read HW stats counters (greater than 32-bit) and accommodates for roll-overs.
4702+ * @stats: pestat struct
4703+ * @index: index in HW stats table which contains offset reg-addr
4704+ * @value: hw stats value
45394705 */
4540−static void i40iw_hw_stat_read_64(struct i40iw_dev_pestat *devstat,
4541− enum i40iw_hw_stat_index_64b index,
4542− u64 *value)
4706+void i40iw_hw_stats_read_64(struct i40iw_vsi_pestat *stats,
4707+ enum i40iw_hw_stats_index_64b index,
4708+ u64 *value)
45434709 {
4544− struct i40iw_dev_hw_stat_offsets *stat_table =
4545− &devstat->hw_stat_offsets;
4546− struct i40iw_dev_hw_stats *last_rd_stats = &devstat->last_read_hw_stats;
4547− struct i40iw_dev_hw_stats *hw_stats = &devstat->hw_stats;
4548− u64 new_stat_value = 0;
4549− u32 stat_reg_offset = stat_table->stat_offset_64[index];
4550−
4551− new_stat_value = readq(devstat->hw->hw_addr + stat_reg_offset);
4710+ struct i40iw_dev_hw_stats_offsets *stats_table =
4711+ &stats->hw_stats_offsets;
4712+ struct i40iw_dev_hw_stats *last_rd_stats = &stats->last_read_hw_stats;
4713+ struct i40iw_dev_hw_stats *hw_stats = &stats->hw_stats;
4714+ u64 new_stats_value = 0;
4715+ u32 stats_reg_offset = stats_table->stats_offset_64[index];
4716+
4717+ new_stats_value = readq(stats->hw->hw_addr + stats_reg_offset);
45524718 /*roll-over case */
4553− if (new_stat_value < last_rd_stats->stat_value_64[index])
4554− hw_stats->stat_value_64[index] += new_stat_value;
4719+ if (new_stats_value < last_rd_stats->stats_value_64[index])
4720+ hw_stats->stats_value_64[index] += new_stats_value;
45554721 else
4556− hw_stats->stat_value_64[index] +=
4557− new_stat_value - last_rd_stats->stat_value_64[index];
4558− last_rd_stats->stat_value_64[index] = new_stat_value;
4559− *value = hw_stats->stat_value_64[index];
4722+ hw_stats->stats_value_64[index] +=
4723+ new_stats_value - last_rd_stats->stats_value_64[index];
4724+ last_rd_stats->stats_value_64[index] = new_stats_value;
4725+ *value = hw_stats->stats_value_64[index];
45604726 }
45614727
45624728 /**
4563− * i40iw_hw_stat_read_all - read all HW stat counters
4564− * @devstat: pestat struct
4565− * @stat_values: hw stats structure
4729+ * i40iw_hw_stats_read_all - read all HW stat counters
4730+ * @stats: pestat struct
4731+ * @stats_values: hw stats structure
45664732 *
45674733 * Read all the HW stat counters and populates hw_stats structure
4568− * of passed-in dev's pestat as well as copy created in stat_values.
4734+ * of passed-in vsi's pestat as well as copy created in stat_values.
45694735 */
4570−static void i40iw_hw_stat_read_all(struct i40iw_dev_pestat *devstat,
4571− struct i40iw_dev_hw_stats *stat_values)
4736+void i40iw_hw_stats_read_all(struct i40iw_vsi_pestat *stats,
4737+ struct i40iw_dev_hw_stats *stats_values)
45724738 {
4573− u32 stat_index;
4574−
4575− for (stat_index = 0; stat_index < I40IW_HW_STAT_INDEX_MAX_32;
4576− stat_index++)
4577− i40iw_hw_stat_read_32(devstat, stat_index,
4578− &stat_values->stat_value_32[stat_index]);
4579− for (stat_index = 0; stat_index < I40IW_HW_STAT_INDEX_MAX_64;
4580− stat_index++)
4581− i40iw_hw_stat_read_64(devstat, stat_index,
4582− &stat_values->stat_value_64[stat_index]);
4739+ u32 stats_index;
4740+ unsigned long flags;
4741+
4742+ spin_lock_irqsave(&stats->lock, flags);
4743+
4744+ for (stats_index = 0; stats_index < I40IW_HW_STAT_INDEX_MAX_32;
4745+ stats_index++)
4746+ i40iw_hw_stats_read_32(stats, stats_index,
4747+ &stats_values->stats_value_32[stats_index]);
4748+ for (stats_index = 0; stats_index < I40IW_HW_STAT_INDEX_MAX_64;
4749+ stats_index++)
4750+ i40iw_hw_stats_read_64(stats, stats_index,
4751+ &stats_values->stats_value_64[stats_index]);
4752+ spin_unlock_irqrestore(&stats->lock, flags);
45834753 }
45844754
45854755 /**
4586− * i40iw_hw_stat_refresh_all - Update all HW stat structs
4587− * @devstat: pestat struct
4588− * @stat_values: hw stats structure
4756+ * i40iw_hw_stats_refresh_all - Update all HW stats structs
4757+ * @stats: pestat struct
45894758 *
4590− * Read all the HW stat counters to refresh values in hw_stats structure
4759+ * Read all the HW stats counters to refresh values in hw_stats structure
45914760 * of passed-in dev's pestat
45924761 */
4593−static void i40iw_hw_stat_refresh_all(struct i40iw_dev_pestat *devstat)
4762+void i40iw_hw_stats_refresh_all(struct i40iw_vsi_pestat *stats)
4763+{
4764+ u64 stats_value;
4765+ u32 stats_index;
4766+ unsigned long flags;
4767+
4768+ spin_lock_irqsave(&stats->lock, flags);
4769+
4770+ for (stats_index = 0; stats_index < I40IW_HW_STAT_INDEX_MAX_32;
4771+ stats_index++)
4772+ i40iw_hw_stats_read_32(stats, stats_index, &stats_value);
4773+ for (stats_index = 0; stats_index < I40IW_HW_STAT_INDEX_MAX_64;
4774+ stats_index++)
4775+ i40iw_hw_stats_read_64(stats, stats_index, &stats_value);
4776+ spin_unlock_irqrestore(&stats->lock, flags);
4777+}
4778+
4779+/**
4780+ * i40iw_get_fcn_id - Return the function id
4781+ * @dev: pointer to the device
4782+ */
4783+static u8 i40iw_get_fcn_id(struct i40iw_sc_dev *dev)
4784+{
4785+ u8 fcn_id = I40IW_INVALID_FCN_ID;
4786+ u8 i;
4787+
4788+ for (i = I40IW_FIRST_NON_PF_STAT; i < I40IW_MAX_STATS_COUNT; i++)
4789+ if (!dev->fcn_id_array[i]) {
4790+ fcn_id = i;
4791+ dev->fcn_id_array[i] = true;
4792+ break;
4793+ }
4794+ return fcn_id;
4795+}
4796+
4797+/**
4798+ * i40iw_vsi_stats_init - Initialize the vsi statistics
4799+ * @vsi: pointer to the vsi structure
4800+ * @info: The info structure used for initialization
4801+ */
4802+enum i40iw_status_code i40iw_vsi_stats_init(struct i40iw_sc_vsi *vsi, struct i40iw_vsi_stats_info *info)
45944803 {
4595− u64 stat_value;
4596− u32 stat_index;
4597−
4598− for (stat_index = 0; stat_index < I40IW_HW_STAT_INDEX_MAX_32;
4599− stat_index++)
4600− i40iw_hw_stat_read_32(devstat, stat_index, &stat_value);
4601− for (stat_index = 0; stat_index < I40IW_HW_STAT_INDEX_MAX_64;
4602− stat_index++)
4603− i40iw_hw_stat_read_64(devstat, stat_index, &stat_value);
4804+ u8 fcn_id = info->fcn_id;
4805+
4806+ if (info->alloc_fcn_id)
4807+ fcn_id = i40iw_get_fcn_id(vsi->dev);
4808+
4809+ if (fcn_id == I40IW_INVALID_FCN_ID)
4810+ return I40IW_ERR_NOT_READY;
4811+
4812+ vsi->pestat = info->pestat;
4813+ vsi->pestat->hw = vsi->dev->hw;
4814+
4815+ if (info->stats_initialize) {
4816+ i40iw_hw_stats_init(vsi->pestat, fcn_id, true);
4817+ spin_lock_init(&vsi->pestat->lock);
4818+ i40iw_hw_stats_start_timer(vsi);
4819+ }
4820+ vsi->stats_fcn_id_alloc = info->alloc_fcn_id;
4821+ vsi->fcn_id = fcn_id;
4822+ return I40IW_SUCCESS;
4823+}
4824+
4825+/**
4826+ * i40iw_vsi_stats_free - Free the vsi stats
4827+ * @vsi: pointer to the vsi structure
4828+ */
4829+void i40iw_vsi_stats_free(struct i40iw_sc_vsi *vsi)
4830+{
4831+ u8 fcn_id = vsi->fcn_id;
4832+
4833+ if ((vsi->stats_fcn_id_alloc) && (fcn_id != I40IW_INVALID_FCN_ID))
4834+ vsi->dev->fcn_id_array[fcn_id] = false;
4835+ i40iw_hw_stats_stop_timer(vsi);
46044836 }
46054837
46064838 static struct i40iw_cqp_ops iw_cqp_ops = {
@@ -4711,24 +4943,6 @@ static struct i40iw_hmc_ops iw_hmc_ops = { · 104 unmodified lines
47114943 NULL
47124944 };
47134945
4714−static const struct i40iw_device_pestat_ops iw_device_pestat_ops = {
4715− i40iw_hw_stat_init,
4716− i40iw_hw_stat_read_32,
4717− i40iw_hw_stat_read_64,
4718− i40iw_hw_stat_read_all,
4719− i40iw_hw_stat_refresh_all
4720−};
4721−
4722−/**
4723− * i40iw_device_init_pestat - Initialize the pestat structure
4724− * @dev: pestat struct
4725− */
4726−enum i40iw_status_code i40iw_device_init_pestat(struct i40iw_dev_pestat *devstat)
4727−{
4728− devstat->ops = iw_device_pestat_ops;
4729− return 0;
4730−}
4731−
47324946 /**
47334947 * i40iw_device_init - Initialize IWARP device
47344948 * @dev: IWARP device pointer
@@ -4750,14 +4964,7 @@ enum i40iw_status_code i40iw_device_init(struct i40iw_sc_dev *dev, · 15 unmodified lines
47504964
47514965 dev->debug_mask = info->debug_mask;
47524966
4753− ret_code = i40iw_device_init_pestat(&dev->dev_pestat);
4754− if (ret_code) {
4755− i40iw_debug(dev, I40IW_DEBUG_DEV,
4756− "%s: i40iw_device_init_pestat failed\n", __func__);
4757− return ret_code;
4758− }
47594967 dev->hmc_fn_id = info->hmc_fn_id;
4760− dev->qs_handle = info->qs_handle;
47614968 dev->exception_lan_queue = info->exception_lan_queue;
47624969 dev->is_pf = info->is_pf;
47634970
@@ -4770,15 +4977,10 @@ enum i40iw_status_code i40iw_device_init(struct i40iw_sc_dev *dev, · 6 unmodified lines
47704977 dev->hw = info->hw;
47714978 dev->hw->hw_addr = info->bar0;
47724979
4773− val = i40iw_rd32(dev->hw, I40E_GLPCI_DREVID);
4774− dev->hw_rev = (u8)RS_32(val, I40E_GLPCI_DREVID_DEFAULT_REVID);
4775−
47764980 if (dev->is_pf) {
4777− dev->dev_pestat.ops.iw_hw_stat_init(&dev->dev_pestat,
4778− dev->hmc_fn_id, dev->hw, true);
4779− spin_lock_init(&dev->dev_pestat.stats_lock);
4780− /*start the periodic stats_timer */
4781− i40iw_hw_stats_start_timer(dev);
4981+ val = i40iw_rd32(dev->hw, I40E_GLPCI_DREVID);
4982+ dev->hw_rev = (u8)RS_32(val, I40E_GLPCI_DREVID_DEFAULT_REVID);
4983+
47824984 val = i40iw_rd32(dev->hw, I40E_GLPCI_LBARCTRL);
47834985 db_size = (u8)RS_32(val, I40E_GLPCI_LBARCTRL_PE_DB_SIZE);
47844986 if ((db_size != I40IW_PE_DB_SIZE_4M) &&
drivers/infiniband/hw/i40iw/i40iw_d.h+17−8
@@ -35,6 +35,8 @@ · 34 unmodified lines
3535 #ifndef I40IW_D_H
3636 #define I40IW_D_H
3737
38+#define I40IW_FIRST_USER_QP_ID 2
39+
3840 #define I40IW_DB_ADDR_OFFSET (4 * 1024 * 1024 - 64 * 1024)
3941 #define I40IW_VF_DB_ADDR_OFFSET (64 * 1024)
4042
@@ -67,6 +69,9 @@ · 26 unmodified lines
6769 #define I40IW_STAG_TYPE_NONSHARED 1
6870
6971 #define I40IW_MAX_USER_PRIORITY 8
72+#define I40IW_MAX_STATS_COUNT 16
73+#define I40IW_FIRST_NON_PF_STAT 4
74+
7075
7176 #define LS_64_1(val, bits) ((u64)(uintptr_t)val << bits)
7277 #define RS_64_1(val, bits) ((u64)(uintptr_t)val >> bits)
@@ -74,6 +79,8 @@ · 1 unmodified line
7479 #define RS_32_1(val, bits) (u32)(val >> bits)
7580 #define I40E_HI_DWORD(x) ((u32)((((x) >> 16) >> 16) & 0xFFFFFFFF))
7681
82+#define QS_HANDLE_UNKNOWN 0xffff
83+
7784 #define LS_64(val, field) (((u64)val << field ## _SHIFT) & (field ## _MASK))
7885
7986 #define RS_64(val, field) ((u64)(val & field ## _MASK) >> field ## _SHIFT)
@@ -1199,8 +1206,11 @@ · 1119 unmodified lines
11991206 #define I40IWQPC_RXCQNUM_SHIFT 32
12001207 #define I40IWQPC_RXCQNUM_MASK (0x1ffffULL << I40IWQPC_RXCQNUM_SHIFT)
12011208
1202−#define I40IWQPC_Q2ADDR_SHIFT I40IW_CQPHC_QPCTX_SHIFT
1203−#define I40IWQPC_Q2ADDR_MASK I40IW_CQPHC_QPCTX_MASK
1209+#define I40IWQPC_STAT_INDEX_SHIFT 0
1210+#define I40IWQPC_STAT_INDEX_MASK (0x1fULL << I40IWQPC_STAT_INDEX_SHIFT)
1211+
1212+#define I40IWQPC_Q2ADDR_SHIFT 0
1213+#define I40IWQPC_Q2ADDR_MASK (0xffffffffffffff00ULL << I40IWQPC_Q2ADDR_SHIFT)
12041214
12051215 #define I40IWQPC_LASTBYTESENT_SHIFT 0
12061216 #define I40IWQPC_LASTBYTESENT_MASK (0xffUL << I40IWQPC_LASTBYTESENT_SHIFT)
@@ -1232,11 +1242,8 @@ · 25 unmodified lines
12321242 #define I40IWQPC_PRIVEN_SHIFT 25
12331243 #define I40IWQPC_PRIVEN_MASK (1UL << I40IWQPC_PRIVEN_SHIFT)
12341244
1235−#define I40IWQPC_LSMMPRESENT_SHIFT 26
1236−#define I40IWQPC_LSMMPRESENT_MASK (1UL << I40IWQPC_LSMMPRESENT_SHIFT)
1237−
1238−#define I40IWQPC_ADJUSTFORLSMM_SHIFT 27
1239−#define I40IWQPC_ADJUSTFORLSMM_MASK (1UL << I40IWQPC_ADJUSTFORLSMM_SHIFT)
1245+#define I40IWQPC_USESTATSINSTANCE_SHIFT 26
1246+#define I40IWQPC_USESTATSINSTANCE_MASK (1UL << I40IWQPC_USESTATSINSTANCE_SHIFT)
12401247
12411248 #define I40IWQPC_IWARPMODE_SHIFT 28
12421249 #define I40IWQPC_IWARPMODE_MASK (1UL << I40IWQPC_IWARPMODE_SHIFT)
@@ -1713,6 +1720,8 @@ enum i40iw_alignment { · 470 unmodified lines
17131720 #define OP_MANAGE_VF_PBLE_BP 28
17141721 #define OP_QUERY_FPM_VALUES 29
17151722 #define OP_COMMIT_FPM_VALUES 30
1716−#define OP_SIZE_CQP_STAT_ARRAY 31
1723+#define OP_REQUESTED_COMMANDS 31
1724+#define OP_COMPLETED_COMMANDS 32
1725+#define OP_SIZE_CQP_STAT_ARRAY 33
17171726
17181727 #endif
drivers/infiniband/hw/i40iw/i40iw_hw.c+45−16
@@ -62,7 +62,7 @@ u32 i40iw_initialize_hw_resources(struct i40iw_device *iwdev) · 61 unmodified lines
6262 max_mr = iwdev->sc_dev.hmc_info->hmc_obj[I40IW_HMC_IW_MR].cnt;
6363 arp_table_size = iwdev->sc_dev.hmc_info->hmc_obj[I40IW_HMC_IW_ARP].cnt;
6464 iwdev->max_cqe = 0xFFFFF;
65− num_pds = max_qp * 4;
65+ num_pds = I40IW_MAX_PDS;
6666 resources_size = sizeof(struct i40iw_arp_entry) * arp_table_size;
6767 resources_size += sizeof(unsigned long) * BITS_TO_LONGS(max_qp);
6868 resources_size += sizeof(unsigned long) * BITS_TO_LONGS(max_mr);
@@ -308,7 +308,9 @@ void i40iw_process_aeq(struct i40iw_device *iwdev) · 239 unmodified lines
308308 iwqp = iwdev->qp_table[info->qp_cq_id];
309309 if (!iwqp) {
310310 spin_unlock_irqrestore(&iwdev->qptable_lock, flags);
311− i40iw_pr_err("qp_id %d is already freed\n", info->qp_cq_id);
311+ i40iw_debug(dev, I40IW_DEBUG_AEQ,
312+ "%s qp_id %d is already freed\n",
313+ __func__, info->qp_cq_id);
312314 continue;
313315 }
314316 i40iw_add_ref(&iwqp->ibqp);
@@ -359,6 +361,9 @@ void i40iw_process_aeq(struct i40iw_device *iwdev) · 44 unmodified lines
359361 continue;
360362 i40iw_cm_disconn(iwqp);
361363 break;
364+ case I40IW_AE_QP_SUSPEND_COMPLETE:
365+ i40iw_qp_suspend_resume(dev, &iwqp->sc_qp, false);
366+ break;
362367 case I40IW_AE_TERMINATE_SENT:
363368 i40iw_terminate_send_fin(qp);
364369 break;
@@ -404,19 +409,18 @@ void i40iw_process_aeq(struct i40iw_device *iwdev) · 39 unmodified lines
404409 case I40IW_AE_LCE_CQ_CATASTROPHIC:
405410 case I40IW_AE_UDA_XMIT_DGRAM_TOO_LONG:
406411 case I40IW_AE_UDA_XMIT_IPADDR_MISMATCH:
407− case I40IW_AE_QP_SUSPEND_COMPLETE:
408412 ctx_info->err_rq_idx_valid = false;
409413 default:
410− if (!info->sq && ctx_info->err_rq_idx_valid) {
411− ctx_info->err_rq_idx = info->wqe_idx;
412− ctx_info->tcp_info_valid = false;
413− ctx_info->iwarp_info_valid = false;
414− ret = dev->iw_priv_qp_ops->qp_setctx(&iwqp->sc_qp,
415− iwqp->host_ctx.va,
416− ctx_info);
417− }
418− i40iw_terminate_connection(qp, info);
419− break;
414+ if (!info->sq && ctx_info->err_rq_idx_valid) {
415+ ctx_info->err_rq_idx = info->wqe_idx;
416+ ctx_info->tcp_info_valid = false;
417+ ctx_info->iwarp_info_valid = false;
418+ ret = dev->iw_priv_qp_ops->qp_setctx(&iwqp->sc_qp,
419+ iwqp->host_ctx.va,
420+ ctx_info);
421+ }
422+ i40iw_terminate_connection(qp, info);
423+ break;
420424 }
421425 if (info->qp)
422426 i40iw_rem_ref(&iwqp->ibqp);
@@ -538,6 +542,7 @@ enum i40iw_status_code i40iw_manage_qhash(struct i40iw_device *iwdev, · 115 unmodified lines
538542 {
539543 struct i40iw_qhash_table_info *info;
540544 struct i40iw_sc_dev *dev = &iwdev->sc_dev;
545+ struct i40iw_sc_vsi *vsi = &iwdev->vsi;
541546 enum i40iw_status_code status;
542547 struct i40iw_cqp *iwcqp = &iwdev->cqp;
543548 struct i40iw_cqp_request *cqp_request;
@@ -550,6 +555,7 @@ enum i40iw_status_code i40iw_manage_qhash(struct i40iw_device *iwdev, · 6 unmodified lines
550555 info = &cqp_info->in.u.manage_qhash_table_entry.info;
551556 memset(info, 0, sizeof(*info));
552557
558+ info->vsi = &iwdev->vsi;
553559 info->manage = mtype;
554560 info->entry_type = etype;
555561 if (cminfo->vlan_id != 0xFFFF) {
@@ -560,8 +566,9 @@ enum i40iw_status_code i40iw_manage_qhash(struct i40iw_device *iwdev, · 4 unmodified lines
560566 }
561567
562568 info->ipv4_valid = cminfo->ipv4;
569+ info->user_pri = cminfo->user_pri;
563570 ether_addr_copy(info->mac_addr, iwdev->netdev->dev_addr);
564− info->qp_num = cpu_to_le32(dev->ilq->qp_id);
571+ info->qp_num = cpu_to_le32(vsi->ilq->qp_id);
565572 info->dest_port = cpu_to_le16(cminfo->loc_port);
566573 info->dest_ip[0] = cpu_to_le32(cminfo->loc_addr[0]);
567574 info->dest_ip[1] = cpu_to_le32(cminfo->loc_addr[1]);
@@ -617,6 +624,7 @@ enum i40iw_status_code i40iw_hw_flush_wqes(struct i40iw_device *iwdev, · 49 unmodified lines
617624 struct i40iw_qp_flush_info *hw_info;
618625 struct i40iw_cqp_request *cqp_request;
619626 struct cqp_commands_info *cqp_info;
627+ struct i40iw_qp *iwqp = (struct i40iw_qp *)qp->back_qp;
620628
621629 cqp_request = i40iw_get_cqp_request(&iwdev->cqp, wait);
622630 if (!cqp_request)
@@ -631,9 +639,30 @@ enum i40iw_status_code i40iw_hw_flush_wqes(struct i40iw_device *iwdev, · 8 unmodified lines
631639 cqp_info->in.u.qp_flush_wqes.qp = qp;
632640 cqp_info->in.u.qp_flush_wqes.scratch = (uintptr_t)cqp_request;
633641 status = i40iw_handle_cqp_op(iwdev, cqp_request);
634− if (status)
642+ if (status) {
635643 i40iw_pr_err("CQP-OP Flush WQE's fail");
636− return status;
644+ complete(&iwqp->sq_drained);
645+ complete(&iwqp->rq_drained);
646+ return status;
647+ }
648+ if (!cqp_request->compl_info.maj_err_code) {
649+ switch (cqp_request->compl_info.min_err_code) {
650+ case I40IW_CQP_COMPL_RQ_WQE_FLUSHED:
651+ complete(&iwqp->sq_drained);
652+ break;
653+ case I40IW_CQP_COMPL_SQ_WQE_FLUSHED:
654+ complete(&iwqp->rq_drained);
655+ break;
656+ case I40IW_CQP_COMPL_RQ_SQ_WQE_FLUSHED:
657+ break;
658+ default:
659+ complete(&iwqp->sq_drained);
660+ complete(&iwqp->rq_drained);
661+ break;
662+ }
663+ }
664+
665+ return 0;
637666 }
638667
639668 /**
drivers/infiniband/hw/i40iw/i40iw_main.c+129−37
@@ -237,14 +237,11 @@ static irqreturn_t i40iw_irq_handler(int irq, void *data) · 236 unmodified lines
237237 */
238238 static void i40iw_destroy_cqp(struct i40iw_device *iwdev, bool free_hwcqp)
239239 {
240− enum i40iw_status_code status = 0;
241240 struct i40iw_sc_dev *dev = &iwdev->sc_dev;
242241 struct i40iw_cqp *cqp = &iwdev->cqp;
243242
244− if (free_hwcqp && dev->cqp_ops->cqp_destroy)
245− status = dev->cqp_ops->cqp_destroy(dev->cqp);
246− if (status)
247− i40iw_pr_err("destroy cqp failed");
243+ if (free_hwcqp)
244+ dev->cqp_ops->cqp_destroy(dev->cqp);
248245
249246 i40iw_free_dma_mem(dev->hw, &cqp->sq);
250247 kfree(cqp->scratch_array);
@@ -270,6 +267,7 @@ static void i40iw_disable_irq(struct i40iw_sc_dev *dev, · 19 unmodified lines
270267 i40iw_wr32(dev->hw, I40E_PFINT_DYN_CTLN(msix_vec->idx - 1), 0);
271268 else
272269 i40iw_wr32(dev->hw, I40E_VFINT_DYN_CTLN1(msix_vec->idx - 1), 0);
270+ irq_set_affinity_hint(msix_vec->irq, NULL);
273271 free_irq(msix_vec->irq, dev_id);
274272 }
275273
@@ -603,7 +601,7 @@ static enum i40iw_status_code i40iw_create_cqp(struct i40iw_device *iwdev) · 327 unmodified lines
603601 i40iw_pr_err("cqp init status %d\n", status);
604602 goto exit;
605603 }
606− status = dev->cqp_ops->cqp_create(dev->cqp, true, &maj_err, &min_err);
604+ status = dev->cqp_ops->cqp_create(dev->cqp, &maj_err, &min_err);
607605 if (status) {
608606 i40iw_pr_err("cqp create status %d maj_err %d min_err %d\n",
609607 status, maj_err, min_err);
@@ -688,6 +686,7 @@ static enum i40iw_status_code i40iw_configure_ceq_vector(struct i40iw_device *iw · 78 unmodified lines
688686 struct i40iw_msix_vector *msix_vec)
689687 {
690688 enum i40iw_status_code status;
689+ cpumask_t mask;
691690
692691 if (iwdev->msix_shared && !ceq_id) {
693692 tasklet_init(&iwdev->dpc_tasklet, i40iw_dpc, (unsigned long)iwdev);
@@ -697,12 +696,15 @@ static enum i40iw_status_code i40iw_configure_ceq_vector(struct i40iw_device *iw · 3 unmodified lines
697696 status = request_irq(msix_vec->irq, i40iw_ceq_handler, 0, "CEQ", iwceq);
698697 }
699698
699+ cpumask_clear(&mask);
700+ cpumask_set_cpu(msix_vec->cpu_affinity, &mask);
701+ irq_set_affinity_hint(msix_vec->irq, &mask);
702+
700703 if (status) {
701704 i40iw_pr_err("ceq irq config fail\n");
702705 return I40IW_ERR_CONFIG;
703706 }
704707 msix_vec->ceq_id = ceq_id;
705− msix_vec->cpu_affinity = 0;
706708
707709 return 0;
708710 }
@@ -930,6 +932,7 @@ static enum i40iw_status_code i40iw_initialize_ilq(struct i40iw_device *iwdev) · 221 unmodified lines
930932 struct i40iw_puda_rsrc_info info;
931933 enum i40iw_status_code status;
932934
935+ memset(&info, 0, sizeof(info));
933936 info.type = I40IW_PUDA_RSRC_TYPE_ILQ;
934937 info.cq_id = 1;
935938 info.qp_id = 0;
@@ -939,10 +942,9 @@ static enum i40iw_status_code i40iw_initialize_ilq(struct i40iw_device *iwdev) · 3 unmodified lines
939942 info.rq_size = 8192;
940943 info.buf_size = 1024;
941944 info.tx_buf_cnt = 16384;
942− info.mss = iwdev->mss;
943945 info.receive = i40iw_receive_ilq;
944946 info.xmit_complete = i40iw_free_sqbuf;
945− status = i40iw_puda_create_rsrc(&iwdev->sc_dev, &info);
947+ status = i40iw_puda_create_rsrc(&iwdev->vsi, &info);
946948 if (status)
947949 i40iw_pr_err("ilq create fail\n");
948950 return status;
@@ -959,6 +961,7 @@ static enum i40iw_status_code i40iw_initialize_ieq(struct i40iw_device *iwdev) · 10 unmodified lines
959961 struct i40iw_puda_rsrc_info info;
960962 enum i40iw_status_code status;
961963
964+ memset(&info, 0, sizeof(info));
962965 info.type = I40IW_PUDA_RSRC_TYPE_IEQ;
963966 info.cq_id = 2;
964967 info.qp_id = iwdev->sc_dev.exception_lan_queue;
@@ -967,9 +970,8 @@ static enum i40iw_status_code i40iw_initialize_ieq(struct i40iw_device *iwdev) · 2 unmodified lines
967970 info.sq_size = 8192;
968971 info.rq_size = 8192;
969972 info.buf_size = 2048;
970− info.mss = iwdev->mss;
971973 info.tx_buf_cnt = 16384;
972− status = i40iw_puda_create_rsrc(&iwdev->sc_dev, &info);
974+ status = i40iw_puda_create_rsrc(&iwdev->vsi, &info);
973975 if (status)
974976 i40iw_pr_err("ieq create fail\n");
975977 return status;
@@ -1159,7 +1161,7 @@ static void i40iw_add_ipv6_addr(struct i40iw_device *iwdev) · 183 unmodified lines
11591161 {
11601162 struct net_device *ip_dev;
11611163 struct inet6_dev *idev;
1162− struct inet6_ifaddr *ifp;
1164+ struct inet6_ifaddr *ifp, *tmp;
11631165 u32 local_ipaddr6[4];
11641166
11651167 rcu_read_lock();
@@ -1172,7 +1174,7 @@ static void i40iw_add_ipv6_addr(struct i40iw_device *iwdev) · 6 unmodified lines
11721174 i40iw_pr_err("ipv6 inet device not found\n");
11731175 break;
11741176 }
1175− list_for_each_entry(ifp, &idev->addr_list, if_list) {
1177+ list_for_each_entry_safe(ifp, tmp, &idev->addr_list, if_list) {
11761178 i40iw_pr_info("IP=%pI6, vlan_id=%d, MAC=%pM\n", &ifp->addr,
11771179 rdma_vlan_dev_vlan_id(ip_dev), ip_dev->dev_addr);
11781180 i40iw_copy_ip_ntohl(local_ipaddr6,
@@ -1294,17 +1296,23 @@ static enum i40iw_status_code i40iw_initialize_dev(struct i40iw_device *iwdev, · 115 unmodified lines
12941296 enum i40iw_status_code status;
12951297 struct i40iw_sc_dev *dev = &iwdev->sc_dev;
12961298 struct i40iw_device_init_info info;
1299+ struct i40iw_vsi_init_info vsi_info;
12971300 struct i40iw_dma_mem mem;
1301+ struct i40iw_l2params l2params;
12981302 u32 size;
1303+ struct i40iw_vsi_stats_info stats_info;
1304+ u16 last_qset = I40IW_NO_QSET;
1305+ u16 qset;
1306+ u32 i;
12991307
1308+ memset(&l2params, 0, sizeof(l2params));
13001309 memset(&info, 0, sizeof(info));
13011310 size = sizeof(struct i40iw_hmc_pble_rsrc) + sizeof(struct i40iw_hmc_info) +
13021311 (sizeof(struct i40iw_hmc_obj_info) * I40IW_HMC_IW_MAX);
13031312 iwdev->hmc_info_mem = kzalloc(size, GFP_KERNEL);
1304− if (!iwdev->hmc_info_mem) {
1305− i40iw_pr_err("memory alloc fail\n");
1313+ if (!iwdev->hmc_info_mem)
13061314 return I40IW_ERR_NO_MEMORY;
1307− }
1315+
13081316 iwdev->pble_rsrc = (struct i40iw_hmc_pble_rsrc *)iwdev->hmc_info_mem;
13091317 dev->hmc_info = &iwdev->hw.hmc;
13101318 dev->hmc_info->hmc_obj = (struct i40iw_hmc_obj_info *)(iwdev->pble_rsrc + 1);
@@ -1325,7 +1333,17 @@ static enum i40iw_status_code i40iw_initialize_dev(struct i40iw_device *iwdev, · 14 unmodified lines
13251333 info.bar0 = ldev->hw_addr;
13261334 info.hw = &iwdev->hw;
13271335 info.debug_mask = debug;
1328− info.qs_handle = ldev->params.qos.prio_qos[0].qs_handle;
1336+ l2params.mss =
1337+ (ldev->params.mtu) ? ldev->params.mtu - I40IW_MTU_TO_MSS : I40IW_DEFAULT_MSS;
1338+ for (i = 0; i < I40E_CLIENT_MAX_USER_PRIORITY; i++) {
1339+ qset = ldev->params.qos.prio_qos[i].qs_handle;
1340+ l2params.qs_handle_list[i] = qset;
1341+ if (last_qset == I40IW_NO_QSET)
1342+ last_qset = qset;
1343+ else if ((qset != last_qset) && (qset != I40IW_NO_QSET))
1344+ iwdev->dcb = true;
1345+ }
1346+ i40iw_pr_info("DCB is set/clear = %d\n", iwdev->dcb);
13291347 info.exception_lan_queue = 1;
13301348 info.vchnl_send = i40iw_virtchnl_send;
13311349 status = i40iw_device_init(&iwdev->sc_dev, &info);
@@ -1334,6 +1352,20 @@ exit: · 2 unmodified lines
13341352 kfree(iwdev->hmc_info_mem);
13351353 iwdev->hmc_info_mem = NULL;
13361354 }
1355+ memset(&vsi_info, 0, sizeof(vsi_info));
1356+ vsi_info.dev = &iwdev->sc_dev;
1357+ vsi_info.back_vsi = (void *)iwdev;
1358+ vsi_info.params = &l2params;
1359+ i40iw_sc_vsi_init(&iwdev->vsi, &vsi_info);
1360+
1361+ if (dev->is_pf) {
1362+ memset(&stats_info, 0, sizeof(stats_info));
1363+ stats_info.fcn_id = ldev->fid;
1364+ stats_info.pestat = kzalloc(sizeof(*stats_info.pestat), GFP_KERNEL);
1365+ stats_info.stats_initialize = true;
1366+ if (stats_info.pestat)
1367+ i40iw_vsi_stats_init(&iwdev->vsi, &stats_info);
1368+ }
13371369 return status;
13381370 }
13391371
@@ -1384,6 +1416,7 @@ static enum i40iw_status_code i40iw_save_msix_info(struct i40iw_device *iwdev, · 44 unmodified lines
13841416 for (i = 0, ceq_idx = 0; i < iwdev->msix_count; i++, iw_qvinfo++) {
13851417 iwdev->iw_msixtbl[i].idx = ldev->msix_entries[i].entry;
13861418 iwdev->iw_msixtbl[i].irq = ldev->msix_entries[i].vector;
1419+ iwdev->iw_msixtbl[i].cpu_affinity = ceq_idx;
13871420 if (i == 0) {
13881421 iw_qvinfo->aeq_idx = 0;
13891422 if (iwdev->msix_shared)
@@ -1404,18 +1437,19 @@ static enum i40iw_status_code i40iw_save_msix_info(struct i40iw_device *iwdev, · 14 unmodified lines
14041437 * i40iw_deinit_device - clean up the device resources
14051438 * @iwdev: iwarp device
14061439 * @reset: true if called before reset
1407− * @del_hdl: true if delete hdl entry
14081440 *
14091441 * Destroy the ib device interface, remove the mac ip entry and ipv4/ipv6 addresses,
14101442 * destroy the device queues and free the pble and the hmc objects
14111443 */
1412−static void i40iw_deinit_device(struct i40iw_device *iwdev, bool reset, bool del_hdl)
1444+static void i40iw_deinit_device(struct i40iw_device *iwdev, bool reset)
14131445 {
14141446 struct i40e_info *ldev = iwdev->ldev;
14151447
14161448 struct i40iw_sc_dev *dev = &iwdev->sc_dev;
14171449
14181450 i40iw_pr_info("state = %d\n", iwdev->init_state);
1451+ if (iwdev->param_wq)
1452+ destroy_workqueue(iwdev->param_wq);
14191453
14201454 switch (iwdev->init_state) {
14211455 case RDMA_DEV_REGISTERED:
@@ -1441,10 +1475,10 @@ static void i40iw_deinit_device(struct i40iw_device *iwdev, bool reset, bool del · 19 unmodified lines
14411475 i40iw_destroy_aeq(iwdev, reset);
14421476 /* fallthrough */
14431477 case IEQ_CREATED:
1444− i40iw_puda_dele_resources(dev, I40IW_PUDA_RSRC_TYPE_IEQ, reset);
1478+ i40iw_puda_dele_resources(&iwdev->vsi, I40IW_PUDA_RSRC_TYPE_IEQ, reset);
14451479 /* fallthrough */
14461480 case ILQ_CREATED:
1447− i40iw_puda_dele_resources(dev, I40IW_PUDA_RSRC_TYPE_ILQ, reset);
1481+ i40iw_puda_dele_resources(&iwdev->vsi, I40IW_PUDA_RSRC_TYPE_ILQ, reset);
14481482 /* fallthrough */
14491483 case CCQ_CREATED:
14501484 i40iw_destroy_ccq(iwdev, reset);
@@ -1456,13 +1490,14 @@ static void i40iw_deinit_device(struct i40iw_device *iwdev, bool reset, bool del · 5 unmodified lines
14561490 i40iw_del_hmc_objects(dev, dev->hmc_info, true, reset);
14571491 /* fallthrough */
14581492 case CQP_CREATED:
1459− i40iw_destroy_cqp(iwdev, !reset);
1493+ i40iw_destroy_cqp(iwdev, true);
14601494 /* fallthrough */
14611495 case INITIAL_STATE:
14621496 i40iw_cleanup_cm_core(&iwdev->cm_core);
1463− if (dev->is_pf)
1464− i40iw_hw_stats_del_timer(dev);
1465−
1497+ if (iwdev->vsi.pestat) {
1498+ i40iw_vsi_stats_free(&iwdev->vsi);
1499+ kfree(iwdev->vsi.pestat);
1500+ }
14661501 i40iw_del_init_mem(iwdev);
14671502 break;
14681503 case INVALID_STATE:
@@ -1472,8 +1507,7 @@ static void i40iw_deinit_device(struct i40iw_device *iwdev, bool reset, bool del · 3 unmodified lines
14721507 break;
14731508 }
14741509
1475− if (del_hdl)
1476− i40iw_del_handler(i40iw_find_i40e_handler(ldev));
1510+ i40iw_del_handler(i40iw_find_i40e_handler(ldev));
14771511 kfree(iwdev->hdl);
14781512 }
14791513
@@ -1508,7 +1542,6 @@ static enum i40iw_status_code i40iw_setup_init_state(struct i40iw_handler *hdl, · 28 unmodified lines
15081542 iwdev->max_enabled_vfs = iwdev->max_rdma_vfs;
15091543 iwdev->netdev = ldev->netdev;
15101544 hdl->client = client;
1511− iwdev->mss = (!ldev->params.mtu) ? I40IW_DEFAULT_MSS : ldev->params.mtu - I40IW_MTU_TO_MSS;
15121545 if (!ldev->ftype)
15131546 iwdev->db_start = pci_resource_start(ldev->pcidev, 0) + I40IW_DB_ADDR_OFFSET;
15141547 else
@@ -1528,6 +1561,7 @@ static enum i40iw_status_code i40iw_setup_init_state(struct i40iw_handler *hdl, · 13 unmodified lines
15281561
15291562 init_waitqueue_head(&iwdev->vchnl_waitq);
15301563 init_waitqueue_head(&dev->vf_reqs);
1564+ init_waitqueue_head(&iwdev->close_wq);
15311565
15321566 status = i40iw_initialize_dev(iwdev, ldev);
15331567 exit:
@@ -1539,6 +1573,20 @@ exit: · 5 unmodified lines
15391573 return status;
15401574 }
15411575
1576+/**
1577+ * i40iw_get_used_rsrc - determine resources used internally
1578+ * @iwdev: iwarp device
1579+ *
1580+ * Called after internal allocations
1581+ */
1582+static void i40iw_get_used_rsrc(struct i40iw_device *iwdev)
1583+{
1584+ iwdev->used_pds = find_next_zero_bit(iwdev->allocated_pds, iwdev->max_pd, 0);
1585+ iwdev->used_qps = find_next_zero_bit(iwdev->allocated_qps, iwdev->max_qp, 0);
1586+ iwdev->used_cqs = find_next_zero_bit(iwdev->allocated_cqs, iwdev->max_cq, 0);
1587+ iwdev->used_mrs = find_next_zero_bit(iwdev->allocated_mrs, iwdev->max_mr, 0);
1588+}
1589+
15421590 /**
15431591 * i40iw_open - client interface operation open for iwarp/uda device
15441592 * @ldev: lan device information
@@ -1611,6 +1659,7 @@ static int i40iw_open(struct i40e_info *ldev, struct i40e_client *client) · 66 unmodified lines
16111659 status = i40iw_initialize_hw_resources(iwdev);
16121660 if (status)
16131661 break;
1662+ i40iw_get_used_rsrc(iwdev);
16141663 dev->ccq_ops->ccq_arm(dev->ccq);
16151664 status = i40iw_hmc_init_pble(&iwdev->sc_dev, iwdev->pble_rsrc);
16161665 if (status)
@@ -1630,35 +1679,73 @@ static int i40iw_open(struct i40e_info *ldev, struct i40e_client *client) · 13 unmodified lines
16301679 iwdev->init_state = RDMA_DEV_REGISTERED;
16311680 iwdev->iw_status = 1;
16321681 i40iw_port_ibevent(iwdev);
1682+ iwdev->param_wq = alloc_ordered_workqueue("l2params", WQ_MEM_RECLAIM);
1683+ if(iwdev->param_wq == NULL)
1684+ break;
16331685 i40iw_pr_info("i40iw_open completed\n");
16341686 return 0;
16351687 } while (0);
16361688
16371689 i40iw_pr_err("status = %d last completion = %d\n", status, iwdev->init_state);
1638− i40iw_deinit_device(iwdev, false, false);
1690+ i40iw_deinit_device(iwdev, false);
16391691 return -ERESTART;
16401692 }
16411693
16421694 /**
1643− * i40iw_l2param_change : handle qs handles for qos and mss change
1695+ * i40iw_l2params_worker - worker for l2 params change
1696+ * @work: work pointer for l2 params
1697+ */
1698+static void i40iw_l2params_worker(struct work_struct *work)
1699+{
1700+ struct l2params_work *dwork =
1701+ container_of(work, struct l2params_work, work);
1702+ struct i40iw_device *iwdev = dwork->iwdev;
1703+
1704+ i40iw_change_l2params(&iwdev->vsi, &dwork->l2params);
1705+ atomic_dec(&iwdev->params_busy);
1706+ kfree(work);
1707+}
1708+
1709+/**
1710+ * i40iw_l2param_change - handle qs handles for qos and mss change
16441711 * @ldev: lan device information
16451712 * @client: client for paramater change
16461713 * @params: new parameters from L2
16471714 */
1648−static void i40iw_l2param_change(struct i40e_info *ldev,
1649− struct i40e_client *client,
1715+static void i40iw_l2param_change(struct i40e_info *ldev, struct i40e_client *client,
16501716 struct i40e_params *params)
16511717 {
16521718 struct i40iw_handler *hdl;
1719+ struct i40iw_l2params *l2params;
1720+ struct l2params_work *work;
16531721 struct i40iw_device *iwdev;
1722+ int i;
16541723
16551724 hdl = i40iw_find_i40e_handler(ldev);
16561725 if (!hdl)
16571726 return;
16581727
16591728 iwdev = &hdl->device;
1660− if (params->mtu)
1661− iwdev->mss = params->mtu - I40IW_MTU_TO_MSS;
1729+
1730+ if (atomic_read(&iwdev->params_busy))
1731+ return;
1732+
1733+
1734+ work = kzalloc(sizeof(*work), GFP_ATOMIC);
1735+ if (!work)
1736+ return;
1737+
1738+ atomic_inc(&iwdev->params_busy);
1739+
1740+ work->iwdev = iwdev;
1741+ l2params = &work->l2params;
1742+ for (i = 0; i < I40E_CLIENT_MAX_USER_PRIORITY; i++)
1743+ l2params->qs_handle_list[i] = params->qos.prio_qos[i].qs_handle;
1744+
1745+ l2params->mss = (params->mtu) ? params->mtu - I40IW_MTU_TO_MSS : iwdev->vsi.mss;
1746+
1747+ INIT_WORK(&work->work, i40iw_l2params_worker);
1748+ queue_work(iwdev->param_wq, &work->work);
16621749 }
16631750
16641751 /**
@@ -1679,8 +1766,11 @@ static void i40iw_close(struct i40e_info *ldev, struct i40e_client *client, bool · 14 unmodified lines
16791766 return;
16801767
16811768 iwdev = &hdl->device;
1769+ iwdev->closing = true;
1770+
1771+ i40iw_cm_disconnect_all(iwdev);
16821772 destroy_workqueue(iwdev->virtchnl_wq);
1683− i40iw_deinit_device(iwdev, reset, true);
1773+ i40iw_deinit_device(iwdev, reset);
16841774 }
16851775
16861776 /**
@@ -1701,21 +1791,23 @@ static void i40iw_vf_reset(struct i40e_info *ldev, struct i40e_client *client, u · 14 unmodified lines
17011791 struct i40iw_vfdev *tmp_vfdev;
17021792 unsigned int i;
17031793 unsigned long flags;
1794+ struct i40iw_device *iwdev;
17041795
17051796 hdl = i40iw_find_i40e_handler(ldev);
17061797 if (!hdl)
17071798 return;
17081799
17091800 dev = &hdl->device.sc_dev;
1801+ iwdev = (struct i40iw_device *)dev->back_dev;
17101802
17111803 for (i = 0; i < I40IW_MAX_PE_ENABLED_VF_COUNT; i++) {
17121804 if (!dev->vf_dev[i] || (dev->vf_dev[i]->vf_id != vf_id))
17131805 continue;
17141806 /* free all resources allocated on behalf of vf */
17151807 tmp_vfdev = dev->vf_dev[i];
1716− spin_lock_irqsave(&dev->dev_pestat.stats_lock, flags);
1808+ spin_lock_irqsave(&iwdev->vsi.pestat->lock, flags);
17171809 dev->vf_dev[i] = NULL;
1718− spin_unlock_irqrestore(&dev->dev_pestat.stats_lock, flags);
1810+ spin_unlock_irqrestore(&iwdev->vsi.pestat->lock, flags);
17191811 i40iw_del_hmc_objects(dev, &tmp_vfdev->hmc_info, false, false);
17201812 /* remove vf hmc function */
17211813 memset(&hmc_fcn_info, 0, sizeof(hmc_fcn_info));
drivers/infiniband/hw/i40iw/i40iw_osdep.h+5−3
@@ -198,6 +198,8 @@ enum i40iw_status_code i40iw_cqp_manage_vf_pble_bp(struct i40iw_sc_dev *dev, · 197 unmodified lines
198198 void i40iw_cqp_spawn_worker(struct i40iw_sc_dev *dev,
199199 struct i40iw_virtchnl_work_info *work_info, u32 iw_vf_idx);
200200 void *i40iw_remove_head(struct list_head *list);
201+void i40iw_qp_suspend_resume(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp, bool suspend);
202+void i40iw_qp_mss_modify(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp);
201203
202204 void i40iw_term_modify_qp(struct i40iw_sc_qp *qp, u8 next_state, u8 term, u8 term_len);
203205 void i40iw_terminate_done(struct i40iw_sc_qp *qp, int timeout_occurred);
@@ -207,9 +209,9 @@ void i40iw_terminate_del_timer(struct i40iw_sc_qp *qp); · 3 unmodified lines
207209 enum i40iw_status_code i40iw_hw_manage_vf_pble_bp(struct i40iw_device *iwdev,
208210 struct i40iw_manage_vf_pble_info *info,
209211 bool wait);
210−struct i40iw_dev_pestat;
211−void i40iw_hw_stats_start_timer(struct i40iw_sc_dev *);
212−void i40iw_hw_stats_del_timer(struct i40iw_sc_dev *);
212+struct i40iw_sc_vsi;
213+void i40iw_hw_stats_start_timer(struct i40iw_sc_vsi *vsi);
214+void i40iw_hw_stats_stop_timer(struct i40iw_sc_vsi *vsi);
213215 #define i40iw_mmiowb() mmiowb()
214216 void i40iw_wr32(struct i40iw_hw *hw, u32 reg, u32 value);
215217 u32 i40iw_rd32(struct i40iw_hw *hw, u32 reg);
drivers/infiniband/hw/i40iw/i40iw_p.h+18−3
@@ -47,8 +47,6 @@ void i40iw_debug_buf(struct i40iw_sc_dev *dev, enum i40iw_debug_flag mask, · 46 unmodified lines
4747 enum i40iw_status_code i40iw_device_init(struct i40iw_sc_dev *dev,
4848 struct i40iw_device_init_info *info);
4949
50−enum i40iw_status_code i40iw_device_init_pestat(struct i40iw_dev_pestat *);
51−
5250 void i40iw_sc_cqp_post_sq(struct i40iw_sc_cqp *cqp);
5351
5452 u64 *i40iw_sc_cqp_get_next_send_wqe(struct i40iw_sc_cqp *cqp, u64 scratch);
@@ -64,7 +62,24 @@ enum i40iw_status_code i40iw_sc_init_iw_hmc(struct i40iw_sc_dev *dev, · 9 unmodified lines
6462 enum i40iw_status_code i40iw_pf_init_vfhmc(struct i40iw_sc_dev *dev, u8 vf_hmc_fn_id,
6563 u32 *vf_cnt_array);
6664
67−/* cqp misc functions */
65+/* stats functions */
66+void i40iw_hw_stats_refresh_all(struct i40iw_vsi_pestat *stats);
67+void i40iw_hw_stats_read_all(struct i40iw_vsi_pestat *stats, struct i40iw_dev_hw_stats *stats_values);
68+void i40iw_hw_stats_read_32(struct i40iw_vsi_pestat *stats,
69+ enum i40iw_hw_stats_index_32b index,
70+ u64 *value);
71+void i40iw_hw_stats_read_64(struct i40iw_vsi_pestat *stats,
72+ enum i40iw_hw_stats_index_64b index,
73+ u64 *value);
74+void i40iw_hw_stats_init(struct i40iw_vsi_pestat *stats, u8 index, bool is_pf);
75+
76+/* vsi misc functions */
77+enum i40iw_status_code i40iw_vsi_stats_init(struct i40iw_sc_vsi *vsi, struct i40iw_vsi_stats_info *info);
78+void i40iw_vsi_stats_free(struct i40iw_sc_vsi *vsi);
79+void i40iw_sc_vsi_init(struct i40iw_sc_vsi *vsi, struct i40iw_vsi_init_info *info);
80+
81+void i40iw_change_l2params(struct i40iw_sc_vsi *vsi, struct i40iw_l2params *l2params);
82+void i40iw_qp_add_qos(struct i40iw_sc_qp *qp);
6883
6984 void i40iw_terminate_send_fin(struct i40iw_sc_qp *qp);
7085
drivers/infiniband/hw/i40iw/i40iw_pble.c+0−4
@@ -353,10 +353,6 @@ static enum i40iw_status_code add_pble_pool(struct i40iw_sc_dev *dev, · 352 unmodified lines
353353 pages = (idx->rel_pd_idx) ? (I40IW_HMC_PD_CNT_IN_SD -
354354 idx->rel_pd_idx) : I40IW_HMC_PD_CNT_IN_SD;
355355 pages = min(pages, pble_rsrc->unallocated_pble >> PBLE_512_SHIFT);
356− if (!pages) {
357− ret_code = I40IW_ERR_NO_PBLCHUNKS_AVAILABLE;
358− goto error;
359− }
360356 info.chunk = chunk;
361357 info.hmc_info = hmc_info;
362358 info.pages = pages;
drivers/infiniband/hw/i40iw/i40iw_puda.c+162−109
@@ -42,12 +42,13 @@ · 41 unmodified lines
4242 #include "i40iw_p.h"
4343 #include "i40iw_puda.h"
4444
45−static void i40iw_ieq_receive(struct i40iw_sc_dev *dev,
45+static void i40iw_ieq_receive(struct i40iw_sc_vsi *vsi,
4646 struct i40iw_puda_buf *buf);
47−static void i40iw_ieq_tx_compl(struct i40iw_sc_dev *dev, void *sqwrid);
47+static void i40iw_ieq_tx_compl(struct i40iw_sc_vsi *vsi, void *sqwrid);
4848 static void i40iw_ilq_putback_rcvbuf(struct i40iw_sc_qp *qp, u32 wqe_idx);
4949 static enum i40iw_status_code i40iw_puda_replenish_rq(struct i40iw_puda_rsrc
5050 *rsrc, bool initial);
51+static void i40iw_ieq_cleanup_qp(struct i40iw_puda_rsrc *ieq, struct i40iw_sc_qp *qp);
5152 /**
5253 * i40iw_puda_get_listbuf - get buffer from puda list
5354 * @list: list to use for buffers (ILQ or IEQ)
@@ -292,7 +293,7 @@ enum i40iw_status_code i40iw_puda_poll_completion(struct i40iw_sc_dev *dev, · 238 unmodified lines
292293 unsigned long flags;
293294
294295 if ((cq_type == I40IW_CQ_TYPE_ILQ) || (cq_type == I40IW_CQ_TYPE_IEQ)) {
295− rsrc = (cq_type == I40IW_CQ_TYPE_ILQ) ? dev->ilq : dev->ieq;
296+ rsrc = (cq_type == I40IW_CQ_TYPE_ILQ) ? cq->vsi->ilq : cq->vsi->ieq;
296297 } else {
297298 i40iw_debug(dev, I40IW_DEBUG_PUDA, "%s qp_type error\n", __func__);
298299 return I40IW_ERR_BAD_PTR;
@@ -335,7 +336,7 @@ enum i40iw_status_code i40iw_puda_poll_completion(struct i40iw_sc_dev *dev, · 36 unmodified lines
335336 rsrc->stats_pkt_rcvd++;
336337 rsrc->compl_rxwqe_idx = info.wqe_idx;
337338 i40iw_debug(dev, I40IW_DEBUG_PUDA, "%s RQ completion\n", __func__);
338− rsrc->receive(rsrc->dev, buf);
339+ rsrc->receive(rsrc->vsi, buf);
339340 if (cq_type == I40IW_CQ_TYPE_ILQ)
340341 i40iw_ilq_putback_rcvbuf(&rsrc->qp, info.wqe_idx);
341342 else
@@ -345,12 +346,12 @@ enum i40iw_status_code i40iw_puda_poll_completion(struct i40iw_sc_dev *dev, · 3 unmodified lines
345346 i40iw_debug(dev, I40IW_DEBUG_PUDA, "%s SQ completion\n", __func__);
346347 sqwrid = (void *)(uintptr_t)qp->sq_wrtrk_array[info.wqe_idx].wrid;
347348 I40IW_RING_SET_TAIL(qp->sq_ring, info.wqe_idx);
348− rsrc->xmit_complete(rsrc->dev, sqwrid);
349+ rsrc->xmit_complete(rsrc->vsi, sqwrid);
349350 spin_lock_irqsave(&rsrc->bufpool_lock, flags);
350351 rsrc->tx_wqe_avail_cnt++;
351352 spin_unlock_irqrestore(&rsrc->bufpool_lock, flags);
352− if (!list_empty(&dev->ilq->txpend))
353− i40iw_puda_send_buf(dev->ilq, NULL);
353+ if (!list_empty(&rsrc->vsi->ilq->txpend))
354+ i40iw_puda_send_buf(rsrc->vsi->ilq, NULL);
354355 }
355356
356357 done:
@@ -513,10 +514,8 @@ static void i40iw_puda_qp_setctx(struct i40iw_puda_rsrc *rsrc) · 156 unmodified lines
513514 * i40iw_puda_qp_wqe - setup wqe for qp create
514515 * @rsrc: resource for qp
515516 */
516−static enum i40iw_status_code i40iw_puda_qp_wqe(struct i40iw_puda_rsrc *rsrc)
517+static enum i40iw_status_code i40iw_puda_qp_wqe(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp)
517518 {
518− struct i40iw_sc_qp *qp = &rsrc->qp;
519− struct i40iw_sc_dev *dev = rsrc->dev;
520519 struct i40iw_sc_cqp *cqp;
521520 u64 *wqe;
522521 u64 header;
@@ -582,6 +581,7 @@ static enum i40iw_status_code i40iw_puda_qp_create(struct i40iw_puda_rsrc *rsrc) · 59 unmodified lines
582581 qp->back_qp = (void *)rsrc;
583582 qp->sq_pa = mem->pa;
584583 qp->rq_pa = qp->sq_pa + sq_size;
584+ qp->vsi = rsrc->vsi;
585585 ukqp->sq_base = mem->va;
586586 ukqp->rq_base = &ukqp->sq_base[rsrc->sq_size];
587587 ukqp->shadow_area = ukqp->rq_base[rsrc->rq_size].elem;
@@ -608,14 +608,62 @@ static enum i40iw_status_code i40iw_puda_qp_create(struct i40iw_puda_rsrc *rsrc) · 20 unmodified lines
608608 ukqp->wqe_alloc_reg = (u32 __iomem *)(i40iw_get_hw_addr(qp->pd->dev) +
609609 I40E_VFPE_WQEALLOC1);
610610
611− qp->qs_handle = qp->dev->qs_handle;
611+ qp->user_pri = 0;
612+ i40iw_qp_add_qos(qp);
612613 i40iw_puda_qp_setctx(rsrc);
613− ret = i40iw_puda_qp_wqe(rsrc);
614+ if (rsrc->ceq_valid)
615+ ret = i40iw_cqp_qp_create_cmd(rsrc->dev, qp);
616+ else
617+ ret = i40iw_puda_qp_wqe(rsrc->dev, qp);
614618 if (ret)
615619 i40iw_free_dma_mem(rsrc->dev->hw, &rsrc->qpmem);
616620 return ret;
617621 }
618622
623+/**
624+ * i40iw_puda_cq_wqe - setup wqe for cq create
625+ * @rsrc: resource for cq
626+ */
627+static enum i40iw_status_code i40iw_puda_cq_wqe(struct i40iw_sc_dev *dev, struct i40iw_sc_cq *cq)
628+{
629+ u64 *wqe;
630+ struct i40iw_sc_cqp *cqp;
631+ u64 header;
632+ struct i40iw_ccq_cqe_info compl_info;
633+ enum i40iw_status_code status = 0;
634+
635+ cqp = dev->cqp;
636+ wqe = i40iw_sc_cqp_get_next_send_wqe(cqp, 0);
637+ if (!wqe)
638+ return I40IW_ERR_RING_FULL;
639+
640+ set_64bit_val(wqe, 0, cq->cq_uk.cq_size);
641+ set_64bit_val(wqe, 8, RS_64_1(cq, 1));
642+ set_64bit_val(wqe, 16,
643+ LS_64(cq->shadow_read_threshold,
644+ I40IW_CQPSQ_CQ_SHADOW_READ_THRESHOLD));
645+ set_64bit_val(wqe, 32, cq->cq_pa);
646+
647+ set_64bit_val(wqe, 40, cq->shadow_area_pa);
648+
649+ header = cq->cq_uk.cq_id |
650+ LS_64(I40IW_CQP_OP_CREATE_CQ, I40IW_CQPSQ_OPCODE) |
651+ LS_64(1, I40IW_CQPSQ_CQ_CHKOVERFLOW) |
652+ LS_64(1, I40IW_CQPSQ_CQ_ENCEQEMASK) |
653+ LS_64(1, I40IW_CQPSQ_CQ_CEQIDVALID) |
654+ LS_64(cqp->polarity, I40IW_CQPSQ_WQEVALID);
655+ set_64bit_val(wqe, 24, header);
656+
657+ i40iw_debug_buf(dev, I40IW_DEBUG_PUDA, "PUDA CQE",
658+ wqe, I40IW_CQP_WQE_SIZE * 8);
659+
660+ i40iw_sc_cqp_post_sq(dev->cqp);
661+ status = dev->cqp_ops->poll_for_cqp_op_done(dev->cqp,
662+ I40IW_CQP_OP_CREATE_CQ,
663+ &compl_info);
664+ return status;
665+}
666+
619667 /**
620668 * i40iw_puda_cq_create - create cq for resource
621669 * @rsrc: resource for which cq to create
@@ -624,18 +672,13 @@ static enum i40iw_status_code i40iw_puda_cq_create(struct i40iw_puda_rsrc *rsrc) · 2 unmodified lines
624672 {
625673 struct i40iw_sc_dev *dev = rsrc->dev;
626674 struct i40iw_sc_cq *cq = &rsrc->cq;
627− u64 *wqe;
628− struct i40iw_sc_cqp *cqp;
629− u64 header;
630675 enum i40iw_status_code ret = 0;
631676 u32 tsize, cqsize;
632− u32 shadow_read_threshold = 128;
633677 struct i40iw_dma_mem *mem;
634− struct i40iw_ccq_cqe_info compl_info;
635678 struct i40iw_cq_init_info info;
636679 struct i40iw_cq_uk_init_info *init_info = &info.cq_uk_init_info;
637680
638− cq->back_cq = (void *)rsrc;
681+ cq->vsi = rsrc->vsi;
639682 cqsize = rsrc->cq_size * (sizeof(struct i40iw_cqe));
640683 tsize = cqsize + sizeof(struct i40iw_cq_shadow_area);
641684 ret = i40iw_allocate_dma_mem(dev->hw, &rsrc->cqmem, tsize,
@@ -656,43 +699,84 @@ static enum i40iw_status_code i40iw_puda_cq_create(struct i40iw_puda_rsrc *rsrc) · 14 unmodified lines
656699 init_info->shadow_area = (u64 *)((u8 *)mem->va + cqsize);
657700 init_info->cq_size = rsrc->cq_size;
658701 init_info->cq_id = rsrc->cq_id;
702+ info.ceqe_mask = true;
703+ info.ceq_id_valid = true;
659704 ret = dev->iw_priv_cq_ops->cq_init(cq, &info);
660705 if (ret)
661706 goto error;
662− cqp = dev->cqp;
663− wqe = i40iw_sc_cqp_get_next_send_wqe(cqp, 0);
664− if (!wqe) {
665− ret = I40IW_ERR_RING_FULL;
666− goto error;
667− }
707+ if (rsrc->ceq_valid)
708+ ret = i40iw_cqp_cq_create_cmd(dev, cq);
709+ else
710+ ret = i40iw_puda_cq_wqe(dev, cq);
711+error:
712+ if (ret)
713+ i40iw_free_dma_mem(dev->hw, &rsrc->cqmem);
714+ return ret;
715+}
668716
669− set_64bit_val(wqe, 0, rsrc->cq_size);
670− set_64bit_val(wqe, 8, RS_64_1(cq, 1));
671− set_64bit_val(wqe, 16, LS_64(shadow_read_threshold, I40IW_CQPSQ_CQ_SHADOW_READ_THRESHOLD));
672− set_64bit_val(wqe, 32, cq->cq_pa);
717+/**
718+ * i40iw_puda_free_qp - free qp for resource
719+ * @rsrc: resource for which qp to free
720+ */
721+static void i40iw_puda_free_qp(struct i40iw_puda_rsrc *rsrc)
722+{
723+ enum i40iw_status_code ret;
724+ struct i40iw_ccq_cqe_info compl_info;
725+ struct i40iw_sc_dev *dev = rsrc->dev;
673726
674− set_64bit_val(wqe, 40, cq->shadow_area_pa);
727+ if (rsrc->ceq_valid) {
728+ i40iw_cqp_qp_destroy_cmd(dev, &rsrc->qp);
729+ return;
730+ }
675731
676− header = rsrc->cq_id |
677− LS_64(I40IW_CQP_OP_CREATE_CQ, I40IW_CQPSQ_OPCODE) |
678− LS_64(1, I40IW_CQPSQ_CQ_CHKOVERFLOW) |
679− LS_64(1, I40IW_CQPSQ_CQ_ENCEQEMASK) |
680− LS_64(1, I40IW_CQPSQ_CQ_CEQIDVALID) |
681− LS_64(cqp->polarity, I40IW_CQPSQ_WQEVALID);
682− set_64bit_val(wqe, 24, header);
732+ ret = dev->iw_priv_qp_ops->qp_destroy(&rsrc->qp,
733+ 0, false, true, true);
734+ if (ret)
735+ i40iw_debug(dev, I40IW_DEBUG_PUDA,
736+ "%s error puda qp destroy wqe\n",
737+ __func__);
683738
684− i40iw_debug_buf(dev, I40IW_DEBUG_PUDA, "PUDA CQE",
685− wqe, I40IW_CQP_WQE_SIZE * 8);
739+ if (!ret) {
740+ ret = dev->cqp_ops->poll_for_cqp_op_done(dev->cqp,
741+ I40IW_CQP_OP_DESTROY_QP,
742+ &compl_info);
743+ if (ret)
744+ i40iw_debug(dev, I40IW_DEBUG_PUDA,
745+ "%s error puda qp destroy failed\n",
746+ __func__);
747+ }
748+}
686749
687− i40iw_sc_cqp_post_sq(dev->cqp);
688− ret = dev->cqp_ops->poll_for_cqp_op_done(dev->cqp,
689− I40IW_CQP_OP_CREATE_CQ,
690− &compl_info);
750+/**
751+ * i40iw_puda_free_cq - free cq for resource
752+ * @rsrc: resource for which cq to free
753+ */
754+static void i40iw_puda_free_cq(struct i40iw_puda_rsrc *rsrc)
755+{
756+ enum i40iw_status_code ret;
757+ struct i40iw_ccq_cqe_info compl_info;
758+ struct i40iw_sc_dev *dev = rsrc->dev;
759+
760+ if (rsrc->ceq_valid) {
761+ i40iw_cqp_cq_destroy_cmd(dev, &rsrc->cq);
762+ return;
763+ }
764+ ret = dev->iw_priv_cq_ops->cq_destroy(&rsrc->cq, 0, true);
691765
692−error:
693766 if (ret)
694− i40iw_free_dma_mem(dev->hw, &rsrc->cqmem);
695− return ret;
767+ i40iw_debug(dev, I40IW_DEBUG_PUDA,
768+ "%s error ieq cq destroy\n",
769+ __func__);
770+
771+ if (!ret) {
772+ ret = dev->cqp_ops->poll_for_cqp_op_done(dev->cqp,
773+ I40IW_CQP_OP_DESTROY_CQ,
774+ &compl_info);
775+ if (ret)
776+ i40iw_debug(dev, I40IW_DEBUG_PUDA,
777+ "%s error ieq qp destroy done\n",
778+ __func__);
779+ }
696780 }
697781
698782 /**
@@ -701,25 +785,24 @@ error: · 2 unmodified lines
701785 * @type: type of resource to dele
702786 * @reset: true if reset chip
703787 */
704−void i40iw_puda_dele_resources(struct i40iw_sc_dev *dev,
788+void i40iw_puda_dele_resources(struct i40iw_sc_vsi *vsi,
705789 enum puda_resource_type type,
706790 bool reset)
707791 {
708− struct i40iw_ccq_cqe_info compl_info;
792+ struct i40iw_sc_dev *dev = vsi->dev;
709793 struct i40iw_puda_rsrc *rsrc;
710794 struct i40iw_puda_buf *buf = NULL;
711795 struct i40iw_puda_buf *nextbuf = NULL;
712796 struct i40iw_virt_mem *vmem;
713− enum i40iw_status_code ret;
714797
715798 switch (type) {
716799 case I40IW_PUDA_RSRC_TYPE_ILQ:
717− rsrc = dev->ilq;
718− vmem = &dev->ilq_mem;
800+ rsrc = vsi->ilq;
801+ vmem = &vsi->ilq_mem;
719802 break;
720803 case I40IW_PUDA_RSRC_TYPE_IEQ:
721− rsrc = dev->ieq;
722− vmem = &dev->ieq_mem;
804+ rsrc = vsi->ieq;
805+ vmem = &vsi->ieq_mem;
723806 break;
724807 default:
725808 i40iw_debug(dev, I40IW_DEBUG_PUDA, "%s: error resource type = 0x%x\n",
@@ -731,45 +814,14 @@ void i40iw_puda_dele_resources(struct i40iw_sc_dev *dev, · 5 unmodified lines
731814 case PUDA_HASH_CRC_COMPLETE:
732815 i40iw_free_hash_desc(rsrc->hash_desc);
733816 case PUDA_QP_CREATED:
734− do {
735− if (reset)
736− break;
737− ret = dev->iw_priv_qp_ops->qp_destroy(&rsrc->qp,
738− 0, false, true, true);
739− if (ret)
740− i40iw_debug(rsrc->dev, I40IW_DEBUG_PUDA,
741− "%s error ieq qp destroy\n",
742− __func__);
743−
744− ret = dev->cqp_ops->poll_for_cqp_op_done(dev->cqp,
745− I40IW_CQP_OP_DESTROY_QP,
746− &compl_info);
747− if (ret)
748− i40iw_debug(rsrc->dev, I40IW_DEBUG_PUDA,
749− "%s error ieq qp destroy done\n",
750− __func__);
751− } while (0);
817+ if (!reset)
818+ i40iw_puda_free_qp(rsrc);
752819
753820 i40iw_free_dma_mem(dev->hw, &rsrc->qpmem);
754821 /* fallthrough */
755822 case PUDA_CQ_CREATED:
756− do {
757− if (reset)
758− break;
759− ret = dev->iw_priv_cq_ops->cq_destroy(&rsrc->cq, 0, true);
760− if (ret)
761− i40iw_debug(rsrc->dev, I40IW_DEBUG_PUDA,
762− "%s error ieq cq destroy\n",
763− __func__);
764−
765− ret = dev->cqp_ops->poll_for_cqp_op_done(dev->cqp,
766− I40IW_CQP_OP_DESTROY_CQ,
767− &compl_info);
768− if (ret)
769− i40iw_debug(rsrc->dev, I40IW_DEBUG_PUDA,
770− "%s error ieq qp destroy done\n",
771− __func__);
772− } while (0);
823+ if (!reset)
824+ i40iw_puda_free_cq(rsrc);
773825
774826 i40iw_free_dma_mem(dev->hw, &rsrc->cqmem);
775827 break;
@@ -825,9 +877,10 @@ static enum i40iw_status_code i40iw_puda_allocbufs(struct i40iw_puda_rsrc *rsrc, · 49 unmodified lines
825877 * @dev: iwarp device
826878 * @info: resource information
827879 */
828−enum i40iw_status_code i40iw_puda_create_rsrc(struct i40iw_sc_dev *dev,
880+enum i40iw_status_code i40iw_puda_create_rsrc(struct i40iw_sc_vsi *vsi,
829881 struct i40iw_puda_rsrc_info *info)
830882 {
883+ struct i40iw_sc_dev *dev = vsi->dev;
831884 enum i40iw_status_code ret = 0;
832885 struct i40iw_puda_rsrc *rsrc;
833886 u32 pudasize;
@@ -840,10 +893,10 @@ enum i40iw_status_code i40iw_puda_create_rsrc(struct i40iw_sc_dev *dev, · 6 unmodified lines
840893 rqwridsize = info->rq_size * 8;
841894 switch (info->type) {
842895 case I40IW_PUDA_RSRC_TYPE_ILQ:
843− vmem = &dev->ilq_mem;
896+ vmem = &vsi->ilq_mem;
844897 break;
845898 case I40IW_PUDA_RSRC_TYPE_IEQ:
846− vmem = &dev->ieq_mem;
899+ vmem = &vsi->ieq_mem;
847900 break;
848901 default:
849902 return I40IW_NOT_SUPPORTED;
@@ -856,22 +909,22 @@ enum i40iw_status_code i40iw_puda_create_rsrc(struct i40iw_sc_dev *dev, · 6 unmodified lines
856909 rsrc = (struct i40iw_puda_rsrc *)vmem->va;
857910 spin_lock_init(&rsrc->bufpool_lock);
858911 if (info->type == I40IW_PUDA_RSRC_TYPE_ILQ) {
859− dev->ilq = (struct i40iw_puda_rsrc *)vmem->va;
860− dev->ilq_count = info->count;
912+ vsi->ilq = (struct i40iw_puda_rsrc *)vmem->va;
913+ vsi->ilq_count = info->count;
861914 rsrc->receive = info->receive;
862915 rsrc->xmit_complete = info->xmit_complete;
863916 } else {
864− vmem = &dev->ieq_mem;
865− dev->ieq_count = info->count;
866− dev->ieq = (struct i40iw_puda_rsrc *)vmem->va;
917+ vmem = &vsi->ieq_mem;
918+ vsi->ieq_count = info->count;
919+ vsi->ieq = (struct i40iw_puda_rsrc *)vmem->va;
867920 rsrc->receive = i40iw_ieq_receive;
868921 rsrc->xmit_complete = i40iw_ieq_tx_compl;
869922 }
870923
924+ rsrc->ceq_valid = info->ceq_valid;
871925 rsrc->type = info->type;
872926 rsrc->sq_wrtrk_array = (struct i40iw_sq_uk_wr_trk_info *)((u8 *)vmem->va + pudasize);
873927 rsrc->rq_wrid_array = (u64 *)((u8 *)vmem->va + pudasize + sqwridsize);
874− rsrc->mss = info->mss;
875928 /* Initialize all ieq lists */
876929 INIT_LIST_HEAD(&rsrc->bufpool);
877930 INIT_LIST_HEAD(&rsrc->txpend);
@@ -885,6 +938,7 @@ enum i40iw_status_code i40iw_puda_create_rsrc(struct i40iw_sc_dev *dev, · 7 unmodified lines
885938 rsrc->cq_size = info->rq_size + info->sq_size;
886939 rsrc->buf_size = info->buf_size;
887940 rsrc->dev = dev;
941+ rsrc->vsi = vsi;
888942
889943 ret = i40iw_puda_cq_create(rsrc);
890944 if (!ret) {
@@ -919,7 +973,7 @@ enum i40iw_status_code i40iw_puda_create_rsrc(struct i40iw_sc_dev *dev, · 28 unmodified lines
919973 dev->ccq_ops->ccq_arm(&rsrc->cq);
920974 return ret;
921975 error:
922− i40iw_puda_dele_resources(dev, info->type, false);
976+ i40iw_puda_dele_resources(vsi, info->type, false);
923977
924978 return ret;
925979 }
@@ -1131,7 +1185,7 @@ static enum i40iw_status_code i40iw_ieq_handle_partial(struct i40iw_puda_rsrc *i · 205 unmodified lines
11311185 list_add(&buf->list, &pbufl);
11321186
11331187 status = i40iw_ieq_create_pbufl(pfpdu, rxlist, &pbufl, buf, fpdu_len);
1134− if (!status)
1188+ if (status)
11351189 goto error;
11361190
11371191 txbuf = i40iw_puda_get_bufpool(ieq);
@@ -1332,7 +1386,7 @@ static void i40iw_ieq_handle_exception(struct i40iw_puda_rsrc *ieq, · 194 unmodified lines
13321386 }
13331387 if (pfpdu->mode && (fps != pfpdu->fps)) {
13341388 /* clean up qp as it is new partial sequence */
1335− i40iw_ieq_cleanup_qp(ieq->dev, qp);
1389+ i40iw_ieq_cleanup_qp(ieq, qp);
13361390 i40iw_debug(ieq->dev, I40IW_DEBUG_IEQ,
13371391 "%s: restarting new partial\n", __func__);
13381392 pfpdu->mode = false;
@@ -1344,7 +1398,7 @@ static void i40iw_ieq_handle_exception(struct i40iw_puda_rsrc *ieq, · 5 unmodified lines
13441398 pfpdu->rcv_nxt = fps;
13451399 pfpdu->fps = fps;
13461400 pfpdu->mode = true;
1347− pfpdu->max_fpdu_data = ieq->mss;
1401+ pfpdu->max_fpdu_data = ieq->vsi->mss;
13481402 pfpdu->pmode_count++;
13491403 INIT_LIST_HEAD(rxlist);
13501404 i40iw_ieq_check_first_buf(buf, fps);
@@ -1379,14 +1433,14 @@ static void i40iw_ieq_handle_exception(struct i40iw_puda_rsrc *ieq, · 28 unmodified lines
13791433 * @dev: iwarp device
13801434 * @buf: exception buffer received
13811435 */
1382−static void i40iw_ieq_receive(struct i40iw_sc_dev *dev,
1436+static void i40iw_ieq_receive(struct i40iw_sc_vsi *vsi,
13831437 struct i40iw_puda_buf *buf)
13841438 {
1385− struct i40iw_puda_rsrc *ieq = dev->ieq;
1439+ struct i40iw_puda_rsrc *ieq = vsi->ieq;
13861440 struct i40iw_sc_qp *qp = NULL;
13871441 u32 wqe_idx = ieq->compl_rxwqe_idx;
13881442
1389− qp = i40iw_ieq_get_qp(dev, buf);
1443+ qp = i40iw_ieq_get_qp(vsi->dev, buf);
13901444 if (!qp) {
13911445 ieq->stats_bad_qp_id++;
13921446 i40iw_puda_ret_bufpool(ieq, buf);
@@ -1404,12 +1458,12 @@ static void i40iw_ieq_receive(struct i40iw_sc_dev *dev, · 11 unmodified lines
14041458
14051459 /**
14061460 * i40iw_ieq_tx_compl - put back after sending completed exception buffer
1407− * @dev: iwarp device
1461+ * @vsi: pointer to the vsi structure
14081462 * @sqwrid: pointer to puda buffer
14091463 */
1410−static void i40iw_ieq_tx_compl(struct i40iw_sc_dev *dev, void *sqwrid)
1464+static void i40iw_ieq_tx_compl(struct i40iw_sc_vsi *vsi, void *sqwrid)
14111465 {
1412− struct i40iw_puda_rsrc *ieq = dev->ieq;
1466+ struct i40iw_puda_rsrc *ieq = vsi->ieq;
14131467 struct i40iw_puda_buf *buf = (struct i40iw_puda_buf *)sqwrid;
14141468
14151469 i40iw_puda_ret_bufpool(ieq, buf);
@@ -1421,15 +1475,14 @@ static void i40iw_ieq_tx_compl(struct i40iw_sc_dev *dev, void *sqwrid) · 5 unmodified lines
14211475
14221476 /**
14231477 * i40iw_ieq_cleanup_qp - qp is being destroyed
1424− * @dev: iwarp device
1478+ * @ieq: ieq resource
14251479 * @qp: all pending fpdu buffers
14261480 */
1427−void i40iw_ieq_cleanup_qp(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp)
1481+static void i40iw_ieq_cleanup_qp(struct i40iw_puda_rsrc *ieq, struct i40iw_sc_qp *qp)
14281482 {
14291483 struct i40iw_puda_buf *buf;
14301484 struct i40iw_pfpdu *pfpdu = &qp->pfpdu;
14311485 struct list_head *rxlist = &pfpdu->rxlist;
1432− struct i40iw_puda_rsrc *ieq = dev->ieq;
14331486
14341487 if (!pfpdu->mode)
14351488 return;
drivers/infiniband/hw/i40iw/i40iw_puda.h+13−7
@@ -100,6 +100,7 @@ struct i40iw_puda_rsrc_info { · 99 unmodified lines
100100 enum puda_resource_type type; /* ILQ or IEQ */
101101 u32 count;
102102 u16 pd_id;
103+ bool ceq_valid;
103104 u32 cq_id;
104105 u32 qp_id;
105106 u32 sq_size;
@@ -107,8 +108,8 @@ struct i40iw_puda_rsrc_info { · 1 unmodified line
107108 u16 buf_size;
108109 u16 mss;
109110 u32 tx_buf_cnt; /* total bufs allocated will be rq_size + tx_buf_cnt */
110− void (*receive)(struct i40iw_sc_dev *, struct i40iw_puda_buf *);
111− void (*xmit_complete)(struct i40iw_sc_dev *, void *);
111+ void (*receive)(struct i40iw_sc_vsi *, struct i40iw_puda_buf *);
112+ void (*xmit_complete)(struct i40iw_sc_vsi *, void *);
112113 };
113114
114115 struct i40iw_puda_rsrc {
@@ -116,6 +117,7 @@ struct i40iw_puda_rsrc { · 1 unmodified line
116117 struct i40iw_sc_qp qp;
117118 struct i40iw_sc_pd sc_pd;
118119 struct i40iw_sc_dev *dev;
120+ struct i40iw_sc_vsi *vsi;
119121 struct i40iw_dma_mem cqmem;
120122 struct i40iw_dma_mem qpmem;
121123 struct i40iw_virt_mem ilq_mem;
@@ -123,6 +125,7 @@ struct i40iw_puda_rsrc { · 1 unmodified line
123125 enum puda_resource_type type;
124126 u16 buf_size; /*buffer must be max datalen + tcpip hdr + mac */
125127 u16 mss;
128+ bool ceq_valid;
126129 u32 cq_id;
127130 u32 qp_id;
128131 u32 sq_size;
@@ -142,8 +145,8 @@ struct i40iw_puda_rsrc { · 13 unmodified lines
142145 u32 avail_buf_count; /* snapshot of currently available buffers */
143146 spinlock_t bufpool_lock;
144147 struct i40iw_puda_buf *alloclist;
145− void (*receive)(struct i40iw_sc_dev *, struct i40iw_puda_buf *);
146− void (*xmit_complete)(struct i40iw_sc_dev *, void *);
148+ void (*receive)(struct i40iw_sc_vsi *, struct i40iw_puda_buf *);
149+ void (*xmit_complete)(struct i40iw_sc_vsi *, void *);
147150 /* puda stats */
148151 u64 stats_buf_alloc_fail;
149152 u64 stats_pkt_rcvd;
@@ -160,14 +163,13 @@ void i40iw_puda_send_buf(struct i40iw_puda_rsrc *rsrc, · 10 unmodified lines
160163 struct i40iw_puda_buf *buf);
161164 enum i40iw_status_code i40iw_puda_send(struct i40iw_sc_qp *qp,
162165 struct i40iw_puda_send_info *info);
163−enum i40iw_status_code i40iw_puda_create_rsrc(struct i40iw_sc_dev *dev,
166+enum i40iw_status_code i40iw_puda_create_rsrc(struct i40iw_sc_vsi *vsi,
164167 struct i40iw_puda_rsrc_info *info);
165−void i40iw_puda_dele_resources(struct i40iw_sc_dev *dev,
168+void i40iw_puda_dele_resources(struct i40iw_sc_vsi *vsi,
166169 enum puda_resource_type type,
167170 bool reset);
168171 enum i40iw_status_code i40iw_puda_poll_completion(struct i40iw_sc_dev *dev,
169172 struct i40iw_sc_cq *cq, u32 *compl_err);
170−void i40iw_ieq_cleanup_qp(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp);
171173
172174 struct i40iw_sc_qp *i40iw_ieq_get_qp(struct i40iw_sc_dev *dev,
173175 struct i40iw_puda_buf *buf);
@@ -180,4 +182,8 @@ void i40iw_ieq_mpa_crc_ae(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp); · 6 unmodified lines
180182 void i40iw_free_hash_desc(struct shash_desc *desc);
181183 void i40iw_ieq_update_tcpip_info(struct i40iw_puda_buf *buf, u16 length,
182184 u32 seqnum);
185+enum i40iw_status_code i40iw_cqp_qp_create_cmd(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp);
186+enum i40iw_status_code i40iw_cqp_cq_create_cmd(struct i40iw_sc_dev *dev, struct i40iw_sc_cq *cq);
187+void i40iw_cqp_qp_destroy_cmd(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp);
188+void i40iw_cqp_cq_destroy_cmd(struct i40iw_sc_dev *dev, struct i40iw_sc_cq *cq);
183189 #endif
drivers/infiniband/hw/i40iw/i40iw_type.h+65−33
@@ -61,7 +61,7 @@ struct i40iw_cq_shadow_area { · 60 unmodified lines
6161
6262 struct i40iw_sc_dev;
6363 struct i40iw_hmc_info;
64−struct i40iw_dev_pestat;
64+struct i40iw_vsi_pestat;
6565
6666 struct i40iw_cqp_ops;
6767 struct i40iw_ccq_ops;
@@ -74,6 +74,11 @@ struct i40iw_priv_qp_ops; · 6 unmodified lines
7474 struct i40iw_priv_cq_ops;
7575 struct i40iw_hmc_ops;
7676
77+enum i40iw_page_size {
78+ I40IW_PAGE_SIZE_4K,
79+ I40IW_PAGE_SIZE_2M
80+};
81+
7782 enum i40iw_resource_indicator_type {
7883 I40IW_RSRC_INDICATOR_TYPE_ADAPTER = 0,
7984 I40IW_RSRC_INDICATOR_TYPE_CQ,
@@ -186,7 +191,7 @@ enum i40iw_debug_flag { · 106 unmodified lines
186191 I40IW_DEBUG_ALL = 0xFFFFFFFF
187192 };
188193
189−enum i40iw_hw_stat_index_32b {
194+enum i40iw_hw_stats_index_32b {
190195 I40IW_HW_STAT_INDEX_IP4RXDISCARD = 0,
191196 I40IW_HW_STAT_INDEX_IP4RXTRUNC,
192197 I40IW_HW_STAT_INDEX_IP4TXNOROUTE,
@@ -199,7 +204,7 @@ enum i40iw_hw_stat_index_32b { · 6 unmodified lines
199204 I40IW_HW_STAT_INDEX_MAX_32
200205 };
201206
202−enum i40iw_hw_stat_index_64b {
207+enum i40iw_hw_stats_index_64b {
203208 I40IW_HW_STAT_INDEX_IP4RXOCTS = 0,
204209 I40IW_HW_STAT_INDEX_IP4RXPKTS,
205210 I40IW_HW_STAT_INDEX_IP4RXFRAGS,
@@ -229,32 +234,23 @@ enum i40iw_hw_stat_index_64b { · 23 unmodified lines
229234 I40IW_HW_STAT_INDEX_MAX_64
230235 };
231236
232−struct i40iw_dev_hw_stat_offsets {
233− u32 stat_offset_32[I40IW_HW_STAT_INDEX_MAX_32];
234− u32 stat_offset_64[I40IW_HW_STAT_INDEX_MAX_64];
237+struct i40iw_dev_hw_stats_offsets {
238+ u32 stats_offset_32[I40IW_HW_STAT_INDEX_MAX_32];
239+ u32 stats_offset_64[I40IW_HW_STAT_INDEX_MAX_64];
235240 };
236241
237242 struct i40iw_dev_hw_stats {
238− u64 stat_value_32[I40IW_HW_STAT_INDEX_MAX_32];
239− u64 stat_value_64[I40IW_HW_STAT_INDEX_MAX_64];
240−};
241−
242−struct i40iw_device_pestat_ops {
243− void (*iw_hw_stat_init)(struct i40iw_dev_pestat *, u8, struct i40iw_hw *, bool);
244− void (*iw_hw_stat_read_32)(struct i40iw_dev_pestat *, enum i40iw_hw_stat_index_32b, u64 *);
245− void (*iw_hw_stat_read_64)(struct i40iw_dev_pestat *, enum i40iw_hw_stat_index_64b, u64 *);
246− void (*iw_hw_stat_read_all)(struct i40iw_dev_pestat *, struct i40iw_dev_hw_stats *);
247− void (*iw_hw_stat_refresh_all)(struct i40iw_dev_pestat *);
243+ u64 stats_value_32[I40IW_HW_STAT_INDEX_MAX_32];
244+ u64 stats_value_64[I40IW_HW_STAT_INDEX_MAX_64];
248245 };
249246
250−struct i40iw_dev_pestat {
247+struct i40iw_vsi_pestat {
251248 struct i40iw_hw *hw;
252− struct i40iw_device_pestat_ops ops;
253249 struct i40iw_dev_hw_stats hw_stats;
254250 struct i40iw_dev_hw_stats last_read_hw_stats;
255− struct i40iw_dev_hw_stat_offsets hw_stat_offsets;
251+ struct i40iw_dev_hw_stats_offsets hw_stats_offsets;
256252 struct timer_list stats_timer;
257− spinlock_t stats_lock; /* rdma stats lock */
253+ spinlock_t lock; /* rdma stats lock */
258254 };
259255
260256 struct i40iw_hw {
@@ -350,6 +346,7 @@ struct i40iw_sc_cq { · 89 unmodified lines
350346 u64 cq_pa;
351347 u64 shadow_area_pa;
352348 struct i40iw_sc_dev *dev;
349+ struct i40iw_sc_vsi *vsi;
353350 void *pbl_list;
354351 void *back_cq;
355352 u32 ceq_id;
@@ -373,6 +370,7 @@ struct i40iw_sc_qp { · 17 unmodified lines
373370 u64 shadow_area_pa;
374371 u64 q2_pa;
375372 struct i40iw_sc_dev *dev;
373+ struct i40iw_sc_vsi *vsi;
376374 struct i40iw_sc_pd *pd;
377375 u64 *hw_host_ctx;
378376 void *llp_stream_handle;
@@ -397,6 +395,9 @@ struct i40iw_sc_qp { · 18 unmodified lines
397395 bool virtual_map;
398396 bool flush_sq;
399397 bool flush_rq;
398+ u8 user_pri;
399+ struct list_head list;
400+ bool on_qoslist;
400401 bool sq_flush;
401402 enum i40iw_flush_opcode flush_code;
402403 enum i40iw_term_eventtypes eventtype;
@@ -424,10 +425,16 @@ struct i40iw_vchnl_vf_msg_buffer { · 21 unmodified lines
424425 char parm_buffer[I40IW_VCHNL_MAX_VF_MSG_SIZE - 1];
425426 };
426427
428+struct i40iw_qos {
429+ struct list_head qplist;
430+ spinlock_t lock; /* qos list */
431+ u16 qs_handle;
432+};
433+
427434 struct i40iw_vfdev {
428435 struct i40iw_sc_dev *pf_dev;
429436 u8 *hmc_info_mem;
430− struct i40iw_dev_pestat dev_pestat;
437+ struct i40iw_vsi_pestat pestat;
431438 struct i40iw_hmc_pble_info *pble_info;
432439 struct i40iw_hmc_info hmc_info;
433440 struct i40iw_vchnl_vf_msg_buffer vf_msg_buffer;
@@ -441,11 +448,28 @@ struct i40iw_vfdev { · 7 unmodified lines
441448 bool stats_initialized;
442449 };
443450
451+#define I40IW_INVALID_FCN_ID 0xff
452+struct i40iw_sc_vsi {
453+ struct i40iw_sc_dev *dev;
454+ void *back_vsi; /* Owned by OS */
455+ u32 ilq_count;
456+ struct i40iw_virt_mem ilq_mem;
457+ struct i40iw_puda_rsrc *ilq;
458+ u32 ieq_count;
459+ struct i40iw_virt_mem ieq_mem;
460+ struct i40iw_puda_rsrc *ieq;
461+ u16 mss;
462+ u8 fcn_id;
463+ bool stats_fcn_id_alloc;
464+ struct i40iw_qos qos[I40IW_MAX_USER_PRIORITY];
465+ struct i40iw_vsi_pestat *pestat;
466+};
467+
444468 struct i40iw_sc_dev {
445469 struct list_head cqp_cmd_head; /* head of the CQP command list */
446470 spinlock_t cqp_lock; /* cqp list sync */
447471 struct i40iw_dev_uk dev_uk;
448− struct i40iw_dev_pestat dev_pestat;
472+ bool fcn_id_array[I40IW_MAX_STATS_COUNT];
449473 struct i40iw_dma_mem vf_fpm_query_buf[I40IW_MAX_PE_ENABLED_VF_COUNT];
450474 u64 fpm_query_buf_pa;
451475 u64 fpm_commit_buf_pa;
@@ -472,17 +496,9 @@ struct i40iw_sc_dev { · 20 unmodified lines
472496 struct i40iw_cqp_misc_ops *cqp_misc_ops;
473497 struct i40iw_hmc_ops *hmc_ops;
474498 struct i40iw_vchnl_if vchnl_if;
475− u32 ilq_count;
476− struct i40iw_virt_mem ilq_mem;
477− struct i40iw_puda_rsrc *ilq;
478− u32 ieq_count;
479− struct i40iw_virt_mem ieq_mem;
480− struct i40iw_puda_rsrc *ieq;
481−
482499 const struct i40iw_vf_cqp_ops *iw_vf_cqp_ops;
483500
484501 struct i40iw_hmc_fpm_misc hmc_fpm_misc;
485− u16 qs_handle;
486502 u32 debug_mask;
487503 u16 exception_lan_queue;
488504 u8 hmc_fn_id;
@@ -556,6 +572,19 @@ struct i40iw_l2params { · 67 unmodified lines
556572 u16 mss;
557573 };
558574
575+struct i40iw_vsi_init_info {
576+ struct i40iw_sc_dev *dev;
577+ void *back_vsi;
578+ struct i40iw_l2params *params;
579+};
580+
581+struct i40iw_vsi_stats_info {
582+ struct i40iw_vsi_pestat *pestat;
583+ u8 fcn_id;
584+ bool alloc_fcn_id;
585+ bool stats_initialize;
586+};
587+
559588 struct i40iw_device_init_info {
560589 u64 fpm_query_buf_pa;
561590 u64 fpm_commit_buf_pa;
@@ -564,7 +593,6 @@ struct i40iw_device_init_info { · 2 unmodified lines
564593 struct i40iw_hw *hw;
565594 void __iomem *bar0;
566595 enum i40iw_status_code (*vchnl_send)(struct i40iw_sc_dev *, u32, u8 *, u16);
567− u16 qs_handle;
568596 u16 exception_lan_queue;
569597 u8 hmc_fn_id;
570598 bool is_pf;
@@ -722,6 +750,8 @@ struct i40iw_qp_host_ctx_info { · 151 unmodified lines
722750 bool iwarp_info_valid;
723751 bool err_rq_idx_valid;
724752 u16 err_rq_idx;
753+ bool add_to_qoslist;
754+ u8 user_pri;
725755 };
726756
727757 struct i40iw_aeqe_info {
@@ -814,6 +844,7 @@ struct i40iw_register_shared_stag { · 86 unmodified lines
814844 struct i40iw_qp_init_info {
815845 struct i40iw_qp_uk_init_info qp_uk_init_info;
816846 struct i40iw_sc_pd *pd;
847+ struct i40iw_sc_vsi *vsi;
817848 u64 *host_ctx;
818849 u8 *q2;
819850 u64 sq_pa;
@@ -880,13 +911,14 @@ enum i40iw_quad_hash_manage_type { · 60 unmodified lines
880911 };
881912
882913 struct i40iw_qhash_table_info {
914+ struct i40iw_sc_vsi *vsi;
883915 enum i40iw_quad_hash_manage_type manage;
884916 enum i40iw_quad_entry_type entry_type;
885917 bool vlan_valid;
886918 bool ipv4_valid;
887919 u8 mac_addr[6];
888920 u16 vlan_id;
889− u16 qs_handle;
921+ u8 user_pri;
890922 u32 qp_num;
891923 u32 dest_ip[4];
892924 u32 src_ip[4];
@@ -976,7 +1008,7 @@ struct i40iw_cqp_query_fpm_values { · 83 unmodified lines
9761008 struct i40iw_cqp_ops {
9771009 enum i40iw_status_code (*cqp_init)(struct i40iw_sc_cqp *,
9781010 struct i40iw_cqp_init_info *);
979− enum i40iw_status_code (*cqp_create)(struct i40iw_sc_cqp *, bool, u16 *, u16 *);
1011+ enum i40iw_status_code (*cqp_create)(struct i40iw_sc_cqp *, u16 *, u16 *);
9801012 void (*cqp_post_sq)(struct i40iw_sc_cqp *);
9811013 u64 *(*cqp_get_next_send_wqe)(struct i40iw_sc_cqp *, u64 scratch);
9821014 enum i40iw_status_code (*cqp_destroy)(struct i40iw_sc_cqp *);
drivers/infiniband/hw/i40iw/i40iw_uk.c+15−25
@@ -175,12 +175,10 @@ u64 *i40iw_qp_get_next_send_wqe(struct i40iw_qp_uk *qp, · 174 unmodified lines
175175 if (!*wqe_idx)
176176 qp->swqe_polarity = !qp->swqe_polarity;
177177 }
178−
179− for (i = 0; i < wqe_size / I40IW_QP_WQE_MIN_SIZE; i++) {
180− I40IW_RING_MOVE_HEAD(qp->sq_ring, ret_code);
181− if (ret_code)
182− return NULL;
183− }
178+ I40IW_RING_MOVE_HEAD_BY_COUNT(qp->sq_ring,
179+ wqe_size / I40IW_QP_WQE_MIN_SIZE, ret_code);
180+ if (ret_code)
181+ return NULL;
184182
185183 wqe = qp->sq_base[*wqe_idx].elem;
186184
@@ -430,7 +428,7 @@ static enum i40iw_status_code i40iw_inline_rdma_write(struct i40iw_qp_uk *qp, · 243 unmodified lines
430428 struct i40iw_inline_rdma_write *op_info;
431429 u64 *push;
432430 u64 header = 0;
433− u32 i, wqe_idx;
431+ u32 wqe_idx;
434432 enum i40iw_status_code ret_code;
435433 bool read_fence = false;
436434 u8 wqe_size;
@@ -465,14 +463,12 @@ static enum i40iw_status_code i40iw_inline_rdma_write(struct i40iw_qp_uk *qp, · 28 unmodified lines
465463 src = (u8 *)(op_info->data);
466464
467465 if (op_info->len <= 16) {
468− for (i = 0; i < op_info->len; i++, src++, dest++)
469− *dest = *src;
466+ memcpy(dest, src, op_info->len);
470467 } else {
471− for (i = 0; i < 16; i++, src++, dest++)
472− *dest = *src;
468+ memcpy(dest, src, 16);
469+ src += 16;
473470 dest = (u8 *)wqe + 32;
474− for (; i < op_info->len; i++, src++, dest++)
475− *dest = *src;
471+ memcpy(dest, src, op_info->len - 16);
476472 }
477473
478474 wmb(); /* make sure WQE is populated before valid bit is set */
@@ -507,7 +503,7 @@ static enum i40iw_status_code i40iw_inline_send(struct i40iw_qp_uk *qp, · 28 unmodified lines
507503 u8 *dest, *src;
508504 struct i40iw_post_inline_send *op_info;
509505 u64 header;
510− u32 wqe_idx, i;
506+ u32 wqe_idx;
511507 enum i40iw_status_code ret_code;
512508 bool read_fence = false;
513509 u8 wqe_size;
@@ -540,14 +536,12 @@ static enum i40iw_status_code i40iw_inline_send(struct i40iw_qp_uk *qp, · 26 unmodified lines
540536 src = (u8 *)(op_info->data);
541537
542538 if (op_info->len <= 16) {
543− for (i = 0; i < op_info->len; i++, src++, dest++)
544− *dest = *src;
539+ memcpy(dest, src, op_info->len);
545540 } else {
546− for (i = 0; i < 16; i++, src++, dest++)
547− *dest = *src;
541+ memcpy(dest, src, 16);
542+ src += 16;
548543 dest = (u8 *)wqe + 32;
549− for (; i < op_info->len; i++, src++, dest++)
550− *dest = *src;
544+ memcpy(dest, src, op_info->len - 16);
551545 }
552546
553547 wmb(); /* make sure WQE is populated before valid bit is set */
@@ -1190,12 +1184,8 @@ enum i40iw_status_code i40iw_inline_data_size_to_wqesize(u32 data_size, · 636 unmodified lines
11901184
11911185 if (data_size <= 16)
11921186 *wqe_size = I40IW_QP_WQE_MIN_SIZE;
1193− else if (data_size <= 48)
1194− *wqe_size = 64;
1195− else if (data_size <= 80)
1196− *wqe_size = 96;
11971187 else
1198− *wqe_size = 128;
1188+ *wqe_size = 64;
11991189
12001190 return 0;
12011191 }
drivers/infiniband/hw/i40iw/i40iw_user.h+4−10
@@ -72,12 +72,12 @@ enum i40iw_device_capabilities_const { · 71 unmodified lines
7272 I40IW_MAX_SQ_PAYLOAD_SIZE = 2145386496,
7373 I40IW_MAX_INLINE_DATA_SIZE = 48,
7474 I40IW_MAX_PUSHMODE_INLINE_DATA_SIZE = 48,
75− I40IW_MAX_IRD_SIZE = 32,
76− I40IW_QPCTX_ENCD_MAXIRD = 3,
75+ I40IW_MAX_IRD_SIZE = 63,
76+ I40IW_MAX_ORD_SIZE = 127,
7777 I40IW_MAX_WQ_ENTRIES = 2048,
78− I40IW_MAX_ORD_SIZE = 32,
7978 I40IW_Q2_BUFFER_SIZE = (248 + 100),
80− I40IW_QP_CTX_SIZE = 248
79+ I40IW_QP_CTX_SIZE = 248,
80+ I40IW_MAX_PDS = 32768
8181 };
8282
8383 #define i40iw_handle void *
@@ -96,12 +96,6 @@ enum i40iw_device_capabilities_const { · 12 unmodified lines
9696 #define i40iw_physical_fragment u64
9797 #define i40iw_address_list u64 *
9898
99−#define I40IW_CREATE_STAG(index, key) (((index) << 8) + (key))
100−
101−#define I40IW_STAG_KEY_FROM_STAG(stag) ((stag) && 0x000000FF)
102−
103−#define I40IW_STAG_INDEX_FROM_STAG(stag) (((stag) && 0xFFFFFF00) >> 8)
104−
10599 #define I40IW_MAX_MR_SIZE 0x10000000000L
106100
107101 struct i40iw_qp_uk;
drivers/infiniband/hw/i40iw/i40iw_utils.c+229−60
@@ -153,6 +153,7 @@ int i40iw_inetaddr_event(struct notifier_block *notifier, · 152 unmodified lines
153153 struct i40iw_device *iwdev;
154154 struct i40iw_handler *hdl;
155155 u32 local_ipaddr;
156+ u32 action = I40IW_ARP_ADD;
156157
157158 hdl = i40iw_find_netdev(event_netdev);
158159 if (!hdl)
@@ -164,44 +165,25 @@ int i40iw_inetaddr_event(struct notifier_block *notifier, · 5 unmodified lines
164165 if (netdev != event_netdev)
165166 return NOTIFY_DONE;
166167
168+ if (upper_dev)
169+ local_ipaddr = ntohl(
170+ ((struct in_device *)upper_dev->ip_ptr)->ifa_list->ifa_address);
171+ else
172+ local_ipaddr = ntohl(ifa->ifa_address);
167173 switch (event) {
168174 case NETDEV_DOWN:
169− if (upper_dev)
170− local_ipaddr = ntohl(
171− ((struct in_device *)upper_dev->ip_ptr)->ifa_list->ifa_address);
172− else
173− local_ipaddr = ntohl(ifa->ifa_address);
174− i40iw_manage_arp_cache(iwdev,
175− netdev->dev_addr,
176− &local_ipaddr,
177− true,
178− I40IW_ARP_DELETE);
179− return NOTIFY_OK;
175+ action = I40IW_ARP_DELETE;
176+ /* Fall through */
180177 case NETDEV_UP:
181− if (upper_dev)
182− local_ipaddr = ntohl(
183− ((struct in_device *)upper_dev->ip_ptr)->ifa_list->ifa_address);
184− else
185− local_ipaddr = ntohl(ifa->ifa_address);
186− i40iw_manage_arp_cache(iwdev,
187− netdev->dev_addr,
188− &local_ipaddr,
189− true,
190− I40IW_ARP_ADD);
191− break;
178+ /* Fall through */
192179 case NETDEV_CHANGEADDR:
193− /* Add the address to the IP table */
194− if (upper_dev)
195− local_ipaddr = ntohl(
196− ((struct in_device *)upper_dev->ip_ptr)->ifa_list->ifa_address);
197− else
198− local_ipaddr = ntohl(ifa->ifa_address);
199−
200180 i40iw_manage_arp_cache(iwdev,
201181 netdev->dev_addr,
202182 &local_ipaddr,
203183 true,
204− I40IW_ARP_ADD);
184+ action);
185+ i40iw_if_notify(iwdev, netdev, &local_ipaddr, true,
186+ (action == I40IW_ARP_ADD) ? true : false);
205187 break;
206188 default:
207189 break;
@@ -225,6 +207,7 @@ int i40iw_inet6addr_event(struct notifier_block *notifier, · 17 unmodified lines
225207 struct i40iw_device *iwdev;
226208 struct i40iw_handler *hdl;
227209 u32 local_ipaddr6[4];
210+ u32 action = I40IW_ARP_ADD;
228211
229212 hdl = i40iw_find_netdev(event_netdev);
230213 if (!hdl)
@@ -235,24 +218,21 @@ int i40iw_inet6addr_event(struct notifier_block *notifier, · 4 unmodified lines
235218 if (netdev != event_netdev)
236219 return NOTIFY_DONE;
237220
221+ i40iw_copy_ip_ntohl(local_ipaddr6, ifa->addr.in6_u.u6_addr32);
238222 switch (event) {
239223 case NETDEV_DOWN:
240− i40iw_copy_ip_ntohl(local_ipaddr6, ifa->addr.in6_u.u6_addr32);
241− i40iw_manage_arp_cache(iwdev,
242− netdev->dev_addr,
243− local_ipaddr6,
244− false,
245− I40IW_ARP_DELETE);
246− return NOTIFY_OK;
224+ action = I40IW_ARP_DELETE;
225+ /* Fall through */
247226 case NETDEV_UP:
248227 /* Fall through */
249228 case NETDEV_CHANGEADDR:
250− i40iw_copy_ip_ntohl(local_ipaddr6, ifa->addr.in6_u.u6_addr32);
251229 i40iw_manage_arp_cache(iwdev,
252230 netdev->dev_addr,
253231 local_ipaddr6,
254232 false,
255− I40IW_ARP_ADD);
233+ action);
234+ i40iw_if_notify(iwdev, netdev, local_ipaddr6, false,
235+ (action == I40IW_ARP_ADD) ? true : false);
256236 break;
257237 default:
258238 break;
@@ -392,6 +372,7 @@ static void i40iw_free_qp(struct i40iw_cqp_request *cqp_request, u32 num) · 133 unmodified lines
392372
393373 i40iw_rem_pdusecount(iwqp->iwpd, iwdev);
394374 i40iw_free_qp_resources(iwdev, iwqp, qp_num);
375+ i40iw_rem_devusecount(iwdev);
395376 }
396377
397378 /**
@@ -415,7 +396,10 @@ static int i40iw_wait_event(struct i40iw_device *iwdev, · 17 unmodified lines
415396 i40iw_pr_err("error cqp command 0x%x timed out ret = %d\n",
416397 info->cqp_cmd, timeout_ret);
417398 err_code = -ETIME;
418− i40iw_request_reset(iwdev);
399+ if (!iwdev->reset) {
400+ iwdev->reset = true;
401+ i40iw_request_reset(iwdev);
402+ }
419403 goto done;
420404 }
421405 cqp_error = cqp_request->compl_info.error;
@@ -445,6 +429,11 @@ enum i40iw_status_code i40iw_handle_cqp_op(struct i40iw_device *iwdev, · 23 unmodified lines
445429 struct cqp_commands_info *info = &cqp_request->info;
446430 int err_code = 0;
447431
432+ if (iwdev->reset) {
433+ i40iw_free_cqp_request(&iwdev->cqp, cqp_request);
434+ return I40IW_ERR_CQP_COMPL_ERROR;
435+ }
436+
448437 status = i40iw_process_cqp_cmd(dev, info);
449438 if (status) {
450439 i40iw_pr_err("error cqp command 0x%x failed\n", info->cqp_cmd);
@@ -458,6 +447,26 @@ enum i40iw_status_code i40iw_handle_cqp_op(struct i40iw_device *iwdev, · 7 unmodified lines
458447 return status;
459448 }
460449
450+/**
451+ * i40iw_add_devusecount - add dev refcount
452+ * @iwdev: dev for refcount
453+ */
454+void i40iw_add_devusecount(struct i40iw_device *iwdev)
455+{
456+ atomic64_inc(&iwdev->use_count);
457+}
458+
459+/**
460+ * i40iw_rem_devusecount - decrement refcount for dev
461+ * @iwdev: device
462+ */
463+void i40iw_rem_devusecount(struct i40iw_device *iwdev)
464+{
465+ if (!atomic64_dec_and_test(&iwdev->use_count))
466+ return;
467+ wake_up(&iwdev->close_wq);
468+}
469+
461470 /**
462471 * i40iw_add_pdusecount - add pd refcount
463472 * @iwpd: pd for refcount
@@ -711,6 +720,51 @@ enum i40iw_status_code i40iw_cqp_sds_cmd(struct i40iw_sc_dev *dev, · 247 unmodified lines
711720 return status;
712721 }
713722
723+/**
724+ * i40iw_qp_suspend_resume - cqp command for suspend/resume
725+ * @dev: hardware control device structure
726+ * @qp: hardware control qp
727+ * @suspend: flag if suspend or resume
728+ */
729+void i40iw_qp_suspend_resume(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp, bool suspend)
730+{
731+ struct i40iw_device *iwdev = (struct i40iw_device *)dev->back_dev;
732+ struct i40iw_cqp_request *cqp_request;
733+ struct i40iw_sc_cqp *cqp = dev->cqp;
734+ struct cqp_commands_info *cqp_info;
735+ enum i40iw_status_code status;
736+
737+ cqp_request = i40iw_get_cqp_request(&iwdev->cqp, false);
738+ if (!cqp_request)
739+ return;
740+
741+ cqp_info = &cqp_request->info;
742+ cqp_info->cqp_cmd = (suspend) ? OP_SUSPEND : OP_RESUME;
743+ cqp_info->in.u.suspend_resume.cqp = cqp;
744+ cqp_info->in.u.suspend_resume.qp = qp;
745+ cqp_info->in.u.suspend_resume.scratch = (uintptr_t)cqp_request;
746+ status = i40iw_handle_cqp_op(iwdev, cqp_request);
747+ if (status)
748+ i40iw_pr_err("CQP-OP QP Suspend/Resume fail");
749+}
750+
751+/**
752+ * i40iw_qp_mss_modify - modify mss for qp
753+ * @dev: hardware control device structure
754+ * @qp: hardware control qp
755+ */
756+void i40iw_qp_mss_modify(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp)
757+{
758+ struct i40iw_device *iwdev = (struct i40iw_device *)dev->back_dev;
759+ struct i40iw_qp *iwqp = (struct i40iw_qp *)qp->back_qp;
760+ struct i40iw_modify_qp_info info;
761+
762+ memset(&info, 0, sizeof(info));
763+ info.mss_change = true;
764+ info.new_mss = qp->vsi->mss;
765+ i40iw_hw_modify_qp(iwdev, iwqp, &info, false);
766+}
767+
714768 /**
715769 * i40iw_term_modify_qp - modify qp for term message
716770 * @qp: hardware control qp
@@ -769,6 +823,7 @@ static void i40iw_terminate_timeout(unsigned long context) · 52 unmodified lines
769823 struct i40iw_sc_qp *qp = (struct i40iw_sc_qp *)&iwqp->sc_qp;
770824
771825 i40iw_terminate_done(qp, 1);
826+ i40iw_rem_ref(&iwqp->ibqp);
772827 }
773828
774829 /**
@@ -780,6 +835,7 @@ void i40iw_terminate_start_timer(struct i40iw_sc_qp *qp) · 5 unmodified lines
780835 struct i40iw_qp *iwqp;
781836
782837 iwqp = (struct i40iw_qp *)qp->back_qp;
838+ i40iw_add_ref(&iwqp->ibqp);
783839 init_timer(&iwqp->terminate_timer);
784840 iwqp->terminate_timer.function = i40iw_terminate_timeout;
785841 iwqp->terminate_timer.expires = jiffies + HZ;
@@ -796,7 +852,8 @@ void i40iw_terminate_del_timer(struct i40iw_sc_qp *qp) · 10 unmodified lines
796852 struct i40iw_qp *iwqp;
797853
798854 iwqp = (struct i40iw_qp *)qp->back_qp;
799− del_timer(&iwqp->terminate_timer);
855+ if (del_timer(&iwqp->terminate_timer))
856+ i40iw_rem_ref(&iwqp->ibqp);
800857 }
801858
802859 /**
@@ -1010,6 +1067,116 @@ enum i40iw_status_code i40iw_vf_wait_vchnl_resp(struct i40iw_sc_dev *dev) · 207 unmodified lines
10101067 return 0;
10111068 }
10121069
1070+/**
1071+ * i40iw_cqp_cq_create_cmd - create a cq for the cqp
1072+ * @dev: device pointer
1073+ * @cq: pointer to created cq
1074+ */
1075+enum i40iw_status_code i40iw_cqp_cq_create_cmd(struct i40iw_sc_dev *dev,
1076+ struct i40iw_sc_cq *cq)
1077+{
1078+ struct i40iw_device *iwdev = (struct i40iw_device *)dev->back_dev;
1079+ struct i40iw_cqp *iwcqp = &iwdev->cqp;
1080+ struct i40iw_cqp_request *cqp_request;
1081+ struct cqp_commands_info *cqp_info;
1082+ enum i40iw_status_code status;
1083+
1084+ cqp_request = i40iw_get_cqp_request(iwcqp, true);
1085+ if (!cqp_request)
1086+ return I40IW_ERR_NO_MEMORY;
1087+
1088+ cqp_info = &cqp_request->info;
1089+ cqp_info->cqp_cmd = OP_CQ_CREATE;
1090+ cqp_info->post_sq = 1;
1091+ cqp_info->in.u.cq_create.cq = cq;
1092+ cqp_info->in.u.cq_create.scratch = (uintptr_t)cqp_request;
1093+ status = i40iw_handle_cqp_op(iwdev, cqp_request);
1094+ if (status)
1095+ i40iw_pr_err("CQP-OP Create QP fail");
1096+
1097+ return status;
1098+}
1099+
1100+/**
1101+ * i40iw_cqp_qp_create_cmd - create a qp for the cqp
1102+ * @dev: device pointer
1103+ * @qp: pointer to created qp
1104+ */
1105+enum i40iw_status_code i40iw_cqp_qp_create_cmd(struct i40iw_sc_dev *dev,
1106+ struct i40iw_sc_qp *qp)
1107+{
1108+ struct i40iw_device *iwdev = (struct i40iw_device *)dev->back_dev;
1109+ struct i40iw_cqp *iwcqp = &iwdev->cqp;
1110+ struct i40iw_cqp_request *cqp_request;
1111+ struct cqp_commands_info *cqp_info;
1112+ struct i40iw_create_qp_info *qp_info;
1113+ enum i40iw_status_code status;
1114+
1115+ cqp_request = i40iw_get_cqp_request(iwcqp, true);
1116+ if (!cqp_request)
1117+ return I40IW_ERR_NO_MEMORY;
1118+
1119+ cqp_info = &cqp_request->info;
1120+ qp_info = &cqp_request->info.in.u.qp_create.info;
1121+
1122+ memset(qp_info, 0, sizeof(*qp_info));
1123+
1124+ qp_info->cq_num_valid = true;
1125+ qp_info->next_iwarp_state = I40IW_QP_STATE_RTS;
1126+
1127+ cqp_info->cqp_cmd = OP_QP_CREATE;
1128+ cqp_info->post_sq = 1;
1129+ cqp_info->in.u.qp_create.qp = qp;
1130+ cqp_info->in.u.qp_create.scratch = (uintptr_t)cqp_request;
1131+ status = i40iw_handle_cqp_op(iwdev, cqp_request);
1132+ if (status)
1133+ i40iw_pr_err("CQP-OP QP create fail");
1134+ return status;
1135+}
1136+
1137+/**
1138+ * i40iw_cqp_cq_destroy_cmd - destroy the cqp cq
1139+ * @dev: device pointer
1140+ * @cq: pointer to cq
1141+ */
1142+void i40iw_cqp_cq_destroy_cmd(struct i40iw_sc_dev *dev, struct i40iw_sc_cq *cq)
1143+{
1144+ struct i40iw_device *iwdev = (struct i40iw_device *)dev->back_dev;
1145+
1146+ i40iw_cq_wq_destroy(iwdev, cq);
1147+}
1148+
1149+/**
1150+ * i40iw_cqp_qp_destroy_cmd - destroy the cqp
1151+ * @dev: device pointer
1152+ * @qp: pointer to qp
1153+ */
1154+void i40iw_cqp_qp_destroy_cmd(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp)
1155+{
1156+ struct i40iw_device *iwdev = (struct i40iw_device *)dev->back_dev;
1157+ struct i40iw_cqp *iwcqp = &iwdev->cqp;
1158+ struct i40iw_cqp_request *cqp_request;
1159+ struct cqp_commands_info *cqp_info;
1160+ enum i40iw_status_code status;
1161+
1162+ cqp_request = i40iw_get_cqp_request(iwcqp, true);
1163+ if (!cqp_request)
1164+ return;
1165+
1166+ cqp_info = &cqp_request->info;
1167+ memset(cqp_info, 0, sizeof(*cqp_info));
1168+
1169+ cqp_info->cqp_cmd = OP_QP_DESTROY;
1170+ cqp_info->post_sq = 1;
1171+ cqp_info->in.u.qp_destroy.qp = qp;
1172+ cqp_info->in.u.qp_destroy.scratch = (uintptr_t)cqp_request;
1173+ cqp_info->in.u.qp_destroy.remove_hash_idx = true;
1174+ status = i40iw_handle_cqp_op(iwdev, cqp_request);
1175+ if (status)
1176+ i40iw_pr_err("CQP QP_DESTROY fail");
1177+}
1178+
1179+
10131180 /**
10141181 * i40iw_ieq_mpa_crc_ae - generate AE for crc error
10151182 * @dev: hardware control device structure
@@ -1208,7 +1375,7 @@ enum i40iw_status_code i40iw_puda_get_tcpip_info(struct i40iw_puda_completion_in · 192 unmodified lines
12081375
12091376 buf->totallen = pkt_len + buf->maclen;
12101377
1211− if (info->payload_len < buf->totallen - 4) {
1378+ if (info->payload_len < buf->totallen) {
12121379 i40iw_pr_err("payload_len = 0x%x totallen expected0x%x\n",
12131380 info->payload_len, buf->totallen);
12141381 return I40IW_ERR_INVALID_SIZE;
@@ -1224,27 +1391,29 @@ enum i40iw_status_code i40iw_puda_get_tcpip_info(struct i40iw_puda_completion_in · 9 unmodified lines
12241391
12251392 /**
12261393 * i40iw_hw_stats_timeout - Stats timer-handler which updates all HW stats
1227− * @dev: hardware control device structure
1394+ * @vsi: pointer to the vsi structure
12281395 */
1229−static void i40iw_hw_stats_timeout(unsigned long dev)
1396+static void i40iw_hw_stats_timeout(unsigned long vsi)
12301397 {
1231− struct i40iw_sc_dev *pf_dev = (struct i40iw_sc_dev *)dev;
1232− struct i40iw_dev_pestat *pf_devstat = &pf_dev->dev_pestat;
1233− struct i40iw_dev_pestat *vf_devstat = NULL;
1398+ struct i40iw_sc_vsi *sc_vsi = (struct i40iw_sc_vsi *)vsi;
1399+ struct i40iw_sc_dev *pf_dev = sc_vsi->dev;
1400+ struct i40iw_vsi_pestat *pf_devstat = sc_vsi->pestat;
1401+ struct i40iw_vsi_pestat *vf_devstat = NULL;
12341402 u16 iw_vf_idx;
12351403 unsigned long flags;
12361404
12371405 /*PF*/
1238− pf_devstat->ops.iw_hw_stat_read_all(pf_devstat, &pf_devstat->hw_stats);
1406+ i40iw_hw_stats_read_all(pf_devstat, &pf_devstat->hw_stats);
1407+
12391408 for (iw_vf_idx = 0; iw_vf_idx < I40IW_MAX_PE_ENABLED_VF_COUNT; iw_vf_idx++) {
1240− spin_lock_irqsave(&pf_devstat->stats_lock, flags);
1409+ spin_lock_irqsave(&pf_devstat->lock, flags);
12411410 if (pf_dev->vf_dev[iw_vf_idx]) {
12421411 if (pf_dev->vf_dev[iw_vf_idx]->stats_initialized) {
1243− vf_devstat = &pf_dev->vf_dev[iw_vf_idx]->dev_pestat;
1244− vf_devstat->ops.iw_hw_stat_read_all(vf_devstat, &vf_devstat->hw_stats);
1412+ vf_devstat = &pf_dev->vf_dev[iw_vf_idx]->pestat;
1413+ i40iw_hw_stats_read_all(vf_devstat, &vf_devstat->hw_stats);
12451414 }
12461415 }
1247− spin_unlock_irqrestore(&pf_devstat->stats_lock, flags);
1416+ spin_unlock_irqrestore(&pf_devstat->lock, flags);
12481417 }
12491418
12501419 mod_timer(&pf_devstat->stats_timer,
@@ -1253,26 +1422,26 @@ static void i40iw_hw_stats_timeout(unsigned long dev) · 2 unmodified lines
12531422
12541423 /**
12551424 * i40iw_hw_stats_start_timer - Start periodic stats timer
1256− * @dev: hardware control device structure
1425+ * @vsi: pointer to the vsi structure
12571426 */
1258−void i40iw_hw_stats_start_timer(struct i40iw_sc_dev *dev)
1427+void i40iw_hw_stats_start_timer(struct i40iw_sc_vsi *vsi)
12591428 {
1260− struct i40iw_dev_pestat *devstat = &dev->dev_pestat;
1429+ struct i40iw_vsi_pestat *devstat = vsi->pestat;
12611430
12621431 init_timer(&devstat->stats_timer);
12631432 devstat->stats_timer.function = i40iw_hw_stats_timeout;
1264− devstat->stats_timer.data = (unsigned long)dev;
1433+ devstat->stats_timer.data = (unsigned long)vsi;
12651434 mod_timer(&devstat->stats_timer,
12661435 jiffies + msecs_to_jiffies(STATS_TIMER_DELAY));
12671436 }
12681437
12691438 /**
1270− * i40iw_hw_stats_del_timer - Delete periodic stats timer
1271− * @dev: hardware control device structure
1439+ * i40iw_hw_stats_stop_timer - Delete periodic stats timer
1440+ * @vsi: pointer to the vsi structure
12721441 */
1273−void i40iw_hw_stats_del_timer(struct i40iw_sc_dev *dev)
1442+void i40iw_hw_stats_stop_timer(struct i40iw_sc_vsi *vsi)
12741443 {
1275− struct i40iw_dev_pestat *devstat = &dev->dev_pestat;
1444+ struct i40iw_vsi_pestat *devstat = vsi->pestat;
12761445
12771446 del_timer_sync(&devstat->stats_timer);
12781447 }
drivers/infiniband/hw/i40iw/i40iw_verbs.c+192−46
@@ -37,6 +37,7 @@ · 36 unmodified lines
3737 #include <linux/random.h>
3838 #include <linux/highmem.h>
3939 #include <linux/time.h>
40+#include <linux/hugetlb.h>
4041 #include <asm/byteorder.h>
4142 #include <net/ip.h>
4243 #include <rdma/ib_verbs.h>
@@ -67,13 +68,13 @@ static int i40iw_query_device(struct ib_device *ibdev, · 24 unmodified lines
6768 props->vendor_part_id = iwdev->ldev->pcidev->device;
6869 props->hw_ver = (u32)iwdev->sc_dev.hw_rev;
6970 props->max_mr_size = I40IW_MAX_OUTBOUND_MESSAGE_SIZE;
70− props->max_qp = iwdev->max_qp;
71+ props->max_qp = iwdev->max_qp - iwdev->used_qps;
7172 props->max_qp_wr = (I40IW_MAX_WQ_ENTRIES >> 2) - 1;
7273 props->max_sge = I40IW_MAX_WQ_FRAGMENT_COUNT;
73− props->max_cq = iwdev->max_cq;
74+ props->max_cq = iwdev->max_cq - iwdev->used_cqs;
7475 props->max_cqe = iwdev->max_cqe;
75− props->max_mr = iwdev->max_mr;
76− props->max_pd = iwdev->max_pd;
76+ props->max_mr = iwdev->max_mr - iwdev->used_mrs;
77+ props->max_pd = iwdev->max_pd - iwdev->used_pds;
7778 props->max_sge_rd = I40IW_MAX_SGE_RD;
7879 props->max_qp_rd_atom = I40IW_MAX_IRD_SIZE;
7980 props->max_qp_init_rd_atom = props->max_qp_rd_atom;
@@ -254,7 +255,6 @@ static void i40iw_alloc_push_page(struct i40iw_device *iwdev, struct i40iw_sc_qp · 174 unmodified lines
254255 {
255256 struct i40iw_cqp_request *cqp_request;
256257 struct cqp_commands_info *cqp_info;
257− struct i40iw_sc_dev *dev = &iwdev->sc_dev;
258258 enum i40iw_status_code status;
259259
260260 if (qp->push_idx != I40IW_INVALID_PUSH_PAGE_INDEX)
@@ -270,7 +270,7 @@ static void i40iw_alloc_push_page(struct i40iw_device *iwdev, struct i40iw_sc_qp · 9 unmodified lines
270270 cqp_info->cqp_cmd = OP_MANAGE_PUSH_PAGE;
271271 cqp_info->post_sq = 1;
272272
273− cqp_info->in.u.manage_push_page.info.qs_handle = dev->qs_handle;
273+ cqp_info->in.u.manage_push_page.info.qs_handle = qp->qs_handle;
274274 cqp_info->in.u.manage_push_page.info.free_page = 0;
275275 cqp_info->in.u.manage_push_page.cqp = &iwdev->cqp.sc_cqp;
276276 cqp_info->in.u.manage_push_page.scratch = (uintptr_t)cqp_request;
@@ -292,7 +292,6 @@ static void i40iw_dealloc_push_page(struct i40iw_device *iwdev, struct i40iw_sc_ · 15 unmodified lines
292292 {
293293 struct i40iw_cqp_request *cqp_request;
294294 struct cqp_commands_info *cqp_info;
295− struct i40iw_sc_dev *dev = &iwdev->sc_dev;
296295 enum i40iw_status_code status;
297296
298297 if (qp->push_idx == I40IW_INVALID_PUSH_PAGE_INDEX)
@@ -307,7 +306,7 @@ static void i40iw_dealloc_push_page(struct i40iw_device *iwdev, struct i40iw_sc_ · 8 unmodified lines
307306 cqp_info->post_sq = 1;
308307
309308 cqp_info->in.u.manage_push_page.info.push_idx = qp->push_idx;
310− cqp_info->in.u.manage_push_page.info.qs_handle = dev->qs_handle;
309+ cqp_info->in.u.manage_push_page.info.qs_handle = qp->qs_handle;
311310 cqp_info->in.u.manage_push_page.info.free_page = 1;
312311 cqp_info->in.u.manage_push_page.cqp = &iwdev->cqp.sc_cqp;
313312 cqp_info->in.u.manage_push_page.scratch = (uintptr_t)cqp_request;
@@ -337,6 +336,9 @@ static struct ib_pd *i40iw_alloc_pd(struct ib_device *ibdev, · 23 unmodified lines
337336 u32 pd_id = 0;
338337 int err;
339338
339+ if (iwdev->closing)
340+ return ERR_PTR(-ENODEV);
341+
340342 err = i40iw_alloc_resource(iwdev, iwdev->allocated_pds,
341343 iwdev->max_pd, &pd_id, &iwdev->next_pd);
342344 if (err) {
@@ -602,6 +604,9 @@ static struct ib_qp *i40iw_create_qp(struct ib_pd *ibpd, · 259 unmodified lines
602604 struct i40iwarp_offload_info *iwarp_info;
603605 unsigned long flags;
604606
607+ if (iwdev->closing)
608+ return ERR_PTR(-ENODEV);
609+
605610 if (init_attr->create_flags)
606611 return ERR_PTR(-EINVAL);
607612 if (init_attr->cap.max_inline_data > I40IW_MAX_INLINE_DATA_SIZE)
@@ -610,11 +615,15 @@ static struct ib_qp *i40iw_create_qp(struct ib_pd *ibpd, · 2 unmodified lines
610615 if (init_attr->cap.max_send_sge > I40IW_MAX_WQ_FRAGMENT_COUNT)
611616 init_attr->cap.max_send_sge = I40IW_MAX_WQ_FRAGMENT_COUNT;
612617
618+ if (init_attr->cap.max_recv_sge > I40IW_MAX_WQ_FRAGMENT_COUNT)
619+ init_attr->cap.max_recv_sge = I40IW_MAX_WQ_FRAGMENT_COUNT;
620+
613621 memset(&init_info, 0, sizeof(init_info));
614622
615623 sq_size = init_attr->cap.max_send_wr;
616624 rq_size = init_attr->cap.max_recv_wr;
617625
626+ init_info.vsi = &iwdev->vsi;
618627 init_info.qp_uk_init_info.sq_size = sq_size;
619628 init_info.qp_uk_init_info.rq_size = rq_size;
620629 init_info.qp_uk_init_info.max_sq_frag_cnt = init_attr->cap.max_send_sge;
@@ -774,6 +783,7 @@ static struct ib_qp *i40iw_create_qp(struct ib_pd *ibpd, · 153 unmodified lines
774783 iwqp->sig_all = (init_attr->sq_sig_type == IB_SIGNAL_ALL_WR) ? 1 : 0;
775784 iwdev->qp_table[qp_num] = iwqp;
776785 i40iw_add_pdusecount(iwqp->iwpd);
786+ i40iw_add_devusecount(iwdev);
777787 if (ibpd->uobject && udata) {
778788 memset(&uresp, 0, sizeof(uresp));
779789 uresp.actual_sq_size = sq_size;
@@ -815,8 +825,9 @@ static int i40iw_query_qp(struct ib_qp *ibqp, · 35 unmodified lines
815825 attr->qp_access_flags = 0;
816826 attr->cap.max_send_wr = qp->qp_uk.sq_size;
817827 attr->cap.max_recv_wr = qp->qp_uk.rq_size;
818− attr->cap.max_recv_sge = 1;
819828 attr->cap.max_inline_data = I40IW_MAX_INLINE_DATA_SIZE;
829+ attr->cap.max_send_sge = I40IW_MAX_WQ_FRAGMENT_COUNT;
830+ attr->cap.max_recv_sge = I40IW_MAX_WQ_FRAGMENT_COUNT;
820831 init_attr->event_handler = iwqp->ibqp.event_handler;
821832 init_attr->qp_context = iwqp->ibqp.qp_context;
822833 init_attr->send_cq = iwqp->ibqp.send_cq;
@@ -884,6 +895,11 @@ int i40iw_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 61 unmodified lines
884895 spin_lock_irqsave(&iwqp->lock, flags);
885896
886897 if (attr_mask & IB_QP_STATE) {
898+ if (iwdev->closing && attr->qp_state != IB_QPS_ERR) {
899+ err = -EINVAL;
900+ goto exit;
901+ }
902+
887903 switch (attr->qp_state) {
888904 case IB_QPS_INIT:
889905 case IB_QPS_RTR:
@@ -944,7 +960,7 @@ int i40iw_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 54 unmodified lines
944960 goto exit;
945961 }
946962 if (iwqp->sc_qp.term_flags)
947− del_timer(&iwqp->terminate_timer);
963+ i40iw_terminate_del_timer(&iwqp->sc_qp);
948964 info.next_iwarp_state = I40IW_QP_STATE_ERROR;
949965 if ((iwqp->hw_tcp_state > I40IW_TCP_STATE_CLOSED) &&
950966 iwdev->iw_status &&
@@ -1037,11 +1053,11 @@ static void cq_free_resources(struct i40iw_device *iwdev, struct i40iw_cq *iwcq) · 86 unmodified lines
10371053 }
10381054
10391055 /**
1040− * cq_wq_destroy - send cq destroy cqp
1056+ * i40iw_cq_wq_destroy - send cq destroy cqp
10411057 * @iwdev: iwarp device
10421058 * @cq: hardware control cq
10431059 */
1044−static void cq_wq_destroy(struct i40iw_device *iwdev, struct i40iw_sc_cq *cq)
1060+void i40iw_cq_wq_destroy(struct i40iw_device *iwdev, struct i40iw_sc_cq *cq)
10451061 {
10461062 enum i40iw_status_code status;
10471063 struct i40iw_cqp_request *cqp_request;
@@ -1080,9 +1096,10 @@ static int i40iw_destroy_cq(struct ib_cq *ib_cq) · 32 unmodified lines
10801096 iwcq = to_iwcq(ib_cq);
10811097 iwdev = to_iwdev(ib_cq->device);
10821098 cq = &iwcq->sc_cq;
1083− cq_wq_destroy(iwdev, cq);
1099+ i40iw_cq_wq_destroy(iwdev, cq);
10841100 cq_free_resources(iwdev, iwcq);
10851101 kfree(iwcq);
1102+ i40iw_rem_devusecount(iwdev);
10861103 return 0;
10871104 }
10881105
@@ -1113,6 +1130,9 @@ static struct ib_cq *i40iw_create_cq(struct ib_device *ibdev, · 24 unmodified lines
11131130 int err_code;
11141131 int entries = attr->cqe;
11151132
1133+ if (iwdev->closing)
1134+ return ERR_PTR(-ENODEV);
1135+
11161136 if (entries > iwdev->max_cqe)
11171137 return ERR_PTR(-EINVAL);
11181138
@@ -1137,7 +1157,8 @@ static struct ib_cq *i40iw_create_cq(struct ib_device *ibdev, · 18 unmodified lines
11371157 ukinfo->cq_id = cq_num;
11381158 iwcq->ibcq.cqe = info.cq_uk_init_info.cq_size;
11391159 info.ceqe_mask = 0;
1140− info.ceq_id = 0;
1160+ if (attr->comp_vector < iwdev->ceqs_count)
1161+ info.ceq_id = attr->comp_vector;
11411162 info.ceq_id_valid = true;
11421163 info.ceqe_mask = 1;
11431164 info.type = I40IW_CQ_TYPE_IWARP;
@@ -1229,10 +1250,11 @@ static struct ib_cq *i40iw_create_cq(struct ib_device *ibdev, · 85 unmodified lines
12291250 }
12301251 }
12311252
1253+ i40iw_add_devusecount(iwdev);
12321254 return (struct ib_cq *)iwcq;
12331255
12341256 cq_destroy:
1235− cq_wq_destroy(iwdev, cq);
1257+ i40iw_cq_wq_destroy(iwdev, cq);
12361258 cq_free_resources:
12371259 cq_free_resources(iwdev, iwcq);
12381260 error:
@@ -1266,6 +1288,7 @@ static void i40iw_free_stag(struct i40iw_device *iwdev, u32 stag) · 27 unmodified lines
12661288
12671289 stag_idx = (stag & iwdev->mr_stagmask) >> I40IW_CQPSQ_STAG_IDX_SHIFT;
12681290 i40iw_free_resource(iwdev, iwdev->allocated_mrs, stag_idx);
1291+ i40iw_rem_devusecount(iwdev);
12691292 }
12701293
12711294 /**
@@ -1296,19 +1319,18 @@ static u32 i40iw_create_stag(struct i40iw_device *iwdev) · 24 unmodified lines
12961319 stag = stag_index << I40IW_CQPSQ_STAG_IDX_SHIFT;
12971320 stag |= driver_key;
12981321 stag += (u32)consumer_key;
1322+ i40iw_add_devusecount(iwdev);
12991323 }
13001324 return stag;
13011325 }
13021326
13031327 /**
13041328 * i40iw_next_pbl_addr - Get next pbl address
1305− * @palloc: Poiner to allocated pbles
13061329 * @pbl: pointer to a pble
13071330 * @pinfo: info pointer
13081331 * @idx: index
13091332 */
1310−static inline u64 *i40iw_next_pbl_addr(struct i40iw_pble_alloc *palloc,
1311− u64 *pbl,
1333+static inline u64 *i40iw_next_pbl_addr(u64 *pbl,
13121334 struct i40iw_pble_info **pinfo,
13131335 u32 *idx)
13141336 {
@@ -1336,9 +1358,11 @@ static void i40iw_copy_user_pgaddrs(struct i40iw_mr *iwmr, · 21 unmodified lines
13361358 struct i40iw_pble_alloc *palloc = &iwpbl->pble_alloc;
13371359 struct i40iw_pble_info *pinfo;
13381360 struct scatterlist *sg;
1361+ u64 pg_addr = 0;
13391362 u32 idx = 0;
13401363
13411364 pinfo = (level == I40IW_LEVEL_1) ? NULL : palloc->level2.leaf;
1365+
13421366 pg_shift = ffs(region->page_size) - 1;
13431367 for_each_sg(region->sg_head.sgl, sg, region->nmap, entry) {
13441368 chunk_pages = sg_dma_len(sg) >> pg_shift;
@@ -1346,17 +1370,96 @@ static void i40iw_copy_user_pgaddrs(struct i40iw_mr *iwmr, · 1 unmodified line
13461370 !iwpbl->qp_mr.sq_page)
13471371 iwpbl->qp_mr.sq_page = sg_page(sg);
13481372 for (i = 0; i < chunk_pages; i++) {
1349− *pbl = cpu_to_le64(sg_dma_address(sg) + region->page_size * i);
1350− pbl = i40iw_next_pbl_addr(palloc, pbl, &pinfo, &idx);
1373+ pg_addr = sg_dma_address(sg) + region->page_size * i;
1374+
1375+ if ((entry + i) == 0)
1376+ *pbl = cpu_to_le64(pg_addr & iwmr->page_msk);
1377+ else if (!(pg_addr & ~iwmr->page_msk))
1378+ *pbl = cpu_to_le64(pg_addr);
1379+ else
1380+ continue;
1381+ pbl = i40iw_next_pbl_addr(pbl, &pinfo, &idx);
1382+ }
1383+ }
1384+}
1385+
1386+/**
1387+ * i40iw_set_hugetlb_params - set MR pg size and mask to huge pg values.
1388+ * @addr: virtual address
1389+ * @iwmr: mr pointer for this memory registration
1390+ */
1391+static void i40iw_set_hugetlb_values(u64 addr, struct i40iw_mr *iwmr)
1392+{
1393+ struct vm_area_struct *vma;
1394+ struct hstate *h;
1395+
1396+ vma = find_vma(current->mm, addr);
1397+ if (vma && is_vm_hugetlb_page(vma)) {
1398+ h = hstate_vma(vma);
1399+ if (huge_page_size(h) == 0x200000) {
1400+ iwmr->page_size = huge_page_size(h);
1401+ iwmr->page_msk = huge_page_mask(h);
13511402 }
13521403 }
13531404 }
13541405
1406+/**
1407+ * i40iw_check_mem_contiguous - check if pbls stored in arr are contiguous
1408+ * @arr: lvl1 pbl array
1409+ * @npages: page count
1410+ * pg_size: page size
1411+ *
1412+ */
1413+static bool i40iw_check_mem_contiguous(u64 *arr, u32 npages, u32 pg_size)
1414+{
1415+ u32 pg_idx;
1416+
1417+ for (pg_idx = 0; pg_idx < npages; pg_idx++) {
1418+ if ((*arr + (pg_size * pg_idx)) != arr[pg_idx])
1419+ return false;
1420+ }
1421+ return true;
1422+}
1423+
1424+/**
1425+ * i40iw_check_mr_contiguous - check if MR is physically contiguous
1426+ * @palloc: pbl allocation struct
1427+ * pg_size: page size
1428+ */
1429+static bool i40iw_check_mr_contiguous(struct i40iw_pble_alloc *palloc, u32 pg_size)
1430+{
1431+ struct i40iw_pble_level2 *lvl2 = &palloc->level2;
1432+ struct i40iw_pble_info *leaf = lvl2->leaf;
1433+ u64 *arr = NULL;
1434+ u64 *start_addr = NULL;
1435+ int i;
1436+ bool ret;
1437+
1438+ if (palloc->level == I40IW_LEVEL_1) {
1439+ arr = (u64 *)palloc->level1.addr;
1440+ ret = i40iw_check_mem_contiguous(arr, palloc->total_cnt, pg_size);
1441+ return ret;
1442+ }
1443+
1444+ start_addr = (u64 *)leaf->addr;
1445+
1446+ for (i = 0; i < lvl2->leaf_cnt; i++, leaf++) {
1447+ arr = (u64 *)leaf->addr;
1448+ if ((*start_addr + (i * pg_size * PBLE_PER_PAGE)) != *arr)
1449+ return false;
1450+ ret = i40iw_check_mem_contiguous(arr, leaf->cnt, pg_size);
1451+ if (!ret)
1452+ return false;
1453+ }
1454+
1455+ return true;
1456+}
1457+
13551458 /**
13561459 * i40iw_setup_pbles - copy user pg address to pble's
13571460 * @iwdev: iwarp device
13581461 * @iwmr: mr pointer for this memory registration
1359− * @use_pbles: flag if to use pble's or memory (level 0)
1462+ * @use_pbles: flag if to use pble's
13601463 */
13611464 static int i40iw_setup_pbles(struct i40iw_device *iwdev,
13621465 struct i40iw_mr *iwmr,
@@ -1369,9 +1472,6 @@ static int i40iw_setup_pbles(struct i40iw_device *iwdev, · 6 unmodified lines
13691472 enum i40iw_status_code status;
13701473 enum i40iw_pble_level level = I40IW_LEVEL_1;
13711474
1372− if (!use_pbles && (iwmr->page_cnt > MAX_SAVE_PAGE_ADDRS))
1373− return -ENOMEM;
1374−
13751475 if (use_pbles) {
13761476 mutex_lock(&iwdev->pbl_mutex);
13771477 status = i40iw_get_pble(&iwdev->sc_dev, iwdev->pble_rsrc, palloc, iwmr->page_cnt);
@@ -1388,6 +1488,10 @@ static int i40iw_setup_pbles(struct i40iw_device *iwdev, · 10 unmodified lines
13881488 }
13891489
13901490 i40iw_copy_user_pgaddrs(iwmr, pbl, level);
1491+
1492+ if (use_pbles)
1493+ iwmr->pgaddrmem[0] = *pbl;
1494+
13911495 return 0;
13921496 }
13931497
@@ -1409,14 +1513,18 @@ static int i40iw_handle_q_mem(struct i40iw_device *iwdev, · 15 unmodified lines
14091513 struct i40iw_cq_mr *cqmr = &iwpbl->cq_mr;
14101514 struct i40iw_hmc_pble *hmc_p;
14111515 u64 *arr = iwmr->pgaddrmem;
1516+ u32 pg_size;
14121517 int err;
14131518 int total;
1519+ bool ret = true;
14141520
14151521 total = req->sq_pages + req->rq_pages + req->cq_pages;
1522+ pg_size = iwmr->page_size;
14161523
14171524 err = i40iw_setup_pbles(iwdev, iwmr, use_pbles);
14181525 if (err)
14191526 return err;
1527+
14201528 if (use_pbles && (palloc->level != I40IW_LEVEL_1)) {
14211529 i40iw_free_pble(iwdev->pble_rsrc, palloc);
14221530 iwpbl->pbl_allocated = false;
@@ -1425,26 +1533,44 @@ static int i40iw_handle_q_mem(struct i40iw_device *iwdev, · 2 unmodified lines
14251533
14261534 if (use_pbles)
14271535 arr = (u64 *)palloc->level1.addr;
1428− if (req->reg_type == IW_MEMREG_TYPE_QP) {
1536+
1537+ if (iwmr->type == IW_MEMREG_TYPE_QP) {
14291538 hmc_p = &qpmr->sq_pbl;
14301539 qpmr->shadow = (dma_addr_t)arr[total];
1540+
14311541 if (use_pbles) {
1542+ ret = i40iw_check_mem_contiguous(arr, req->sq_pages, pg_size);
1543+ if (ret)
1544+ ret = i40iw_check_mem_contiguous(&arr[req->sq_pages], req->rq_pages, pg_size);
1545+ }
1546+
1547+ if (!ret) {
14321548 hmc_p->idx = palloc->level1.idx;
14331549 hmc_p = &qpmr->rq_pbl;
14341550 hmc_p->idx = palloc->level1.idx + req->sq_pages;
14351551 } else {
14361552 hmc_p->addr = arr[0];
14371553 hmc_p = &qpmr->rq_pbl;
1438− hmc_p->addr = arr[1];
1554+ hmc_p->addr = arr[req->sq_pages];
14391555 }
14401556 } else { /* CQ */
14411557 hmc_p = &cqmr->cq_pbl;
14421558 cqmr->shadow = (dma_addr_t)arr[total];
1559+
14431560 if (use_pbles)
1561+ ret = i40iw_check_mem_contiguous(arr, req->cq_pages, pg_size);
1562+
1563+ if (!ret)
14441564 hmc_p->idx = palloc->level1.idx;
14451565 else
14461566 hmc_p->addr = arr[0];
14471567 }
1568+
1569+ if (use_pbles && ret) {
1570+ i40iw_free_pble(iwdev->pble_rsrc, palloc);
1571+ iwpbl->pbl_allocated = false;
1572+ }
1573+
14481574 return err;
14491575 }
14501576
@@ -1642,8 +1768,9 @@ static int i40iw_hwreg_mr(struct i40iw_device *iwdev, · 191 unmodified lines
16421768 stag_info->access_rights = access;
16431769 stag_info->pd_id = iwpd->sc_pd.pd_id;
16441770 stag_info->addr_type = I40IW_ADDR_TYPE_VA_BASED;
1771+ stag_info->page_size = iwmr->page_size;
16451772
1646− if (iwmr->page_cnt > 1) {
1773+ if (iwpbl->pbl_allocated) {
16471774 if (palloc->level == I40IW_LEVEL_1) {
16481775 stag_info->first_pm_pbl_index = palloc->level1.idx;
16491776 stag_info->chunk_size = 1;
@@ -1699,6 +1826,11 @@ static struct ib_mr *i40iw_reg_user_mr(struct ib_pd *pd, · 49 unmodified lines
16991826 bool use_pbles = false;
17001827 unsigned long flags;
17011828 int err = -ENOSYS;
1829+ int ret;
1830+ int pg_shift;
1831+
1832+ if (iwdev->closing)
1833+ return ERR_PTR(-ENODEV);
17021834
17031835 if (length > I40IW_MAX_MR_SIZE)
17041836 return ERR_PTR(-EINVAL);
@@ -1723,9 +1855,17 @@ static struct ib_mr *i40iw_reg_user_mr(struct ib_pd *pd, · 18 unmodified lines
17231855 iwmr->ibmr.pd = pd;
17241856 iwmr->ibmr.device = pd->device;
17251857 ucontext = to_ucontext(pd->uobject->context);
1726− region_length = region->length + (start & 0xfff);
1727− pbl_depth = region_length >> 12;
1728− pbl_depth += (region_length & (4096 - 1)) ? 1 : 0;
1858+
1859+ iwmr->page_size = region->page_size;
1860+ iwmr->page_msk = PAGE_MASK;
1861+
1862+ if (region->hugetlb && (req.reg_type == IW_MEMREG_TYPE_MEM))
1863+ i40iw_set_hugetlb_values(start, iwmr);
1864+
1865+ region_length = region->length + (start & (iwmr->page_size - 1));
1866+ pg_shift = ffs(iwmr->page_size) - 1;
1867+ pbl_depth = region_length >> pg_shift;
1868+ pbl_depth += (region_length & (iwmr->page_size - 1)) ? 1 : 0;
17291869 iwmr->length = region->length;
17301870
17311871 iwpbl->user_base = virt;
@@ -1755,13 +1895,21 @@ static struct ib_mr *i40iw_reg_user_mr(struct ib_pd *pd, · 23 unmodified lines
17551895 spin_unlock_irqrestore(&ucontext->cq_reg_mem_list_lock, flags);
17561896 break;
17571897 case IW_MEMREG_TYPE_MEM:
1898+ use_pbles = (iwmr->page_cnt != 1);
17581899 access = I40IW_ACCESS_FLAGS_LOCALREAD;
17591900
1760− use_pbles = (iwmr->page_cnt != 1);
17611901 err = i40iw_setup_pbles(iwdev, iwmr, use_pbles);
17621902 if (err)
17631903 goto error;
17641904
1905+ if (use_pbles) {
1906+ ret = i40iw_check_mr_contiguous(palloc, iwmr->page_size);
1907+ if (ret) {
1908+ i40iw_free_pble(iwdev->pble_rsrc, palloc);
1909+ iwpbl->pbl_allocated = false;
1910+ }
1911+ }
1912+
17651913 access |= i40iw_get_user_access(acc);
17661914 stag = i40iw_create_stag(iwdev);
17671915 if (!stag) {
@@ -1778,6 +1926,7 @@ static struct ib_mr *i40iw_reg_user_mr(struct ib_pd *pd, · 10 unmodified lines
17781926 i40iw_free_stag(iwdev, stag);
17791927 goto error;
17801928 }
1929+
17811930 break;
17821931 default:
17831932 goto error;
@@ -1789,7 +1938,7 @@ static struct ib_mr *i40iw_reg_user_mr(struct ib_pd *pd, · 5 unmodified lines
17891938 return &iwmr->ibmr;
17901939
17911940 error:
1792− if (palloc->level != I40IW_LEVEL_0)
1941+ if (palloc->level != I40IW_LEVEL_0 && iwpbl->pbl_allocated)
17931942 i40iw_free_pble(iwdev->pble_rsrc, palloc);
17941943 ib_umem_release(region);
17951944 kfree(iwmr);
@@ -2142,7 +2291,6 @@ static int i40iw_post_send(struct ib_qp *ibqp, · 346 unmodified lines
21422291 case IB_WR_REG_MR:
21432292 {
21442293 struct i40iw_mr *iwmr = to_iwmr(reg_wr(ib_wr)->mr);
2145− int page_shift = ilog2(reg_wr(ib_wr)->mr->page_size);
21462294 int flags = reg_wr(ib_wr)->access;
21472295 struct i40iw_pble_alloc *palloc = &iwmr->iwpbl.pble_alloc;
21482296 struct i40iw_sc_dev *dev = &iwqp->iwdev->sc_dev;
@@ -2153,6 +2301,7 @@ static int i40iw_post_send(struct ib_qp *ibqp, · 4 unmodified lines
21532301 info.access_rights |= i40iw_get_user_access(flags);
21542302 info.stag_key = reg_wr(ib_wr)->key & 0xff;
21552303 info.stag_idx = reg_wr(ib_wr)->key >> 8;
2304+ info.page_size = reg_wr(ib_wr)->mr->page_size;
21562305 info.wr_id = ib_wr->wr_id;
21572306
21582307 info.addr_type = I40IW_ADDR_TYPE_VA_BASED;
@@ -2166,9 +2315,6 @@ static int i40iw_post_send(struct ib_qp *ibqp, · 7 unmodified lines
21662315 if (iwmr->npages > I40IW_MIN_PAGES_PER_FMR)
21672316 info.chunk_size = 1;
21682317
2169− if (page_shift == 21)
2170− info.page_size = 1; /* 2M page */
2171−
21722318 ret = dev->iw_priv_qp_ops->iw_mr_fast_register(&iwqp->sc_qp, &info, true);
21732319 if (ret)
21742320 err = -ENOMEM;
@@ -2487,21 +2633,17 @@ static int i40iw_get_hw_stats(struct ib_device *ibdev, · 312 unmodified lines
24872633 {
24882634 struct i40iw_device *iwdev = to_iwdev(ibdev);
24892635 struct i40iw_sc_dev *dev = &iwdev->sc_dev;
2490− struct i40iw_dev_pestat *devstat = &dev->dev_pestat;
2636+ struct i40iw_vsi_pestat *devstat = iwdev->vsi.pestat;
24912637 struct i40iw_dev_hw_stats *hw_stats = &devstat->hw_stats;
2492− unsigned long flags;
24932638
24942639 if (dev->is_pf) {
2495− spin_lock_irqsave(&devstat->stats_lock, flags);
2496− devstat->ops.iw_hw_stat_read_all(devstat,
2497− &devstat->hw_stats);
2498− spin_unlock_irqrestore(&devstat->stats_lock, flags);
2640+ i40iw_hw_stats_read_all(devstat, &devstat->hw_stats);
24992641 } else {
25002642 if (i40iw_vchnl_vf_get_pe_stats(dev, &devstat->hw_stats))
25012643 return -ENOSYS;
25022644 }
25032645
2504− memcpy(&stats->value[0], &hw_stats, sizeof(*hw_stats));
2646+ memcpy(&stats->value[0], hw_stats, sizeof(*hw_stats));
25052647
25062648 return stats->num_counters;
25072649 }
@@ -2562,7 +2704,9 @@ static int i40iw_query_pkey(struct ib_device *ibdev, · 54 unmodified lines
25622704 * @ah_attr: address handle attributes
25632705 */
25642706 static struct ib_ah *i40iw_create_ah(struct ib_pd *ibpd,
2565− struct ib_ah_attr *attr)
2707+ struct ib_ah_attr *attr,
2708+ struct ib_udata *udata)
2709+
25662710 {
25672711 return ERR_PTR(-ENOSYS);
25682712 }
@@ -2621,7 +2765,7 @@ static struct i40iw_ib_device *i40iw_init_rdma_device(struct i40iw_device *iwdev · 52 unmodified lines
26212765 (1ull << IB_USER_VERBS_CMD_POST_RECV) |
26222766 (1ull << IB_USER_VERBS_CMD_POST_SEND);
26232767 iwibdev->ibdev.phys_port_cnt = 1;
2624− iwibdev->ibdev.num_comp_vectors = 1;
2768+ iwibdev->ibdev.num_comp_vectors = iwdev->ceqs_count;
26252769 iwibdev->ibdev.dma_device = &pcidev->dev;
26262770 iwibdev->ibdev.dev.parent = &pcidev->dev;
26272771 iwibdev->ibdev.query_port = i40iw_query_port;
@@ -2654,7 +2798,6 @@ static struct i40iw_ib_device *i40iw_init_rdma_device(struct i40iw_device *iwdev · 26 unmodified lines
26542798 iwibdev->ibdev.iwcm = kzalloc(sizeof(*iwibdev->ibdev.iwcm), GFP_KERNEL);
26552799 if (!iwibdev->ibdev.iwcm) {
26562800 ib_dealloc_device(&iwibdev->ibdev);
2657− i40iw_pr_err("iwcm == NULL\n");
26582801 return NULL;
26592802 }
26602803
@@ -2719,6 +2862,9 @@ void i40iw_destroy_rdma_device(struct i40iw_ib_device *iwibdev) · 58 unmodified lines
27192862 i40iw_unregister_rdma_device(iwibdev);
27202863 kfree(iwibdev->ibdev.iwcm);
27212864 iwibdev->ibdev.iwcm = NULL;
2865+ wait_event_timeout(iwibdev->iwdev->close_wq,
2866+ !atomic64_read(&iwibdev->iwdev->use_count),
2867+ I40IW_EVENT_TIMEOUT);
27222868 ib_dealloc_device(&iwibdev->ibdev);
27232869 }
27242870
drivers/infiniband/hw/i40iw/i40iw_verbs.h+2−0
@@ -92,6 +92,8 @@ struct i40iw_mr { · 91 unmodified lines
9292 struct ib_umem *region;
9393 u16 type;
9494 u32 page_cnt;
95+ u32 page_size;
96+ u64 page_msk;
9597 u32 npages;
9698 u32 stag;
9799 u64 length;
drivers/infiniband/hw/i40iw/i40iw_virtchnl.c+18−15
@@ -402,6 +402,19 @@ del_out: · 401 unmodified lines
402402 vchnl_pf_send_error_resp(vf_dev->pf_dev, vf_dev->vf_id, vchnl_msg, (u16)ret_code);
403403 }
404404
405+/**
406+ * i40iw_vf_init_pestat - Initialize stats for VF
407+ * @devL pointer to the VF Device
408+ * @stats: Statistics structure pointer
409+ * @index: Stats index
410+ */
411+static void i40iw_vf_init_pestat(struct i40iw_sc_dev *dev, struct i40iw_vsi_pestat *stats, u16 index)
412+{
413+ stats->hw = dev->hw;
414+ i40iw_hw_stats_init(stats, (u8)index, false);
415+ spin_lock_init(&stats->lock);
416+}
417+
405418 /**
406419 * i40iw_vchnl_recv_pf - Receive PF virtual channel messages
407420 * @dev: IWARP device pointer
@@ -421,9 +434,8 @@ enum i40iw_status_code i40iw_vchnl_recv_pf(struct i40iw_sc_dev *dev, · 13 unmodified lines
421434 u16 first_avail_iw_vf = I40IW_MAX_PE_ENABLED_VF_COUNT;
422435 struct i40iw_virt_mem vf_dev_mem;
423436 struct i40iw_virtchnl_work_info work_info;
424− struct i40iw_dev_pestat *devstat;
437+ struct i40iw_vsi_pestat *stats;
425438 enum i40iw_status_code ret_code;
426− unsigned long flags;
427439
428440 if (!dev || !msg || !len)
429441 return I40IW_ERR_PARAM;
@@ -496,14 +508,7 @@ enum i40iw_status_code i40iw_vchnl_recv_pf(struct i40iw_sc_dev *dev, · 66 unmodified lines
496508 i40iw_debug(dev, I40IW_DEBUG_VIRT,
497509 "VF%u error CQP HMC Function operation.\n",
498510 vf_id);
499− ret_code = i40iw_device_init_pestat(&vf_dev->dev_pestat);
500− if (ret_code)
501− i40iw_debug(dev, I40IW_DEBUG_VIRT,
502− "VF%u - i40iw_device_init_pestat failed\n",
503− vf_id);
504− vf_dev->dev_pestat.ops.iw_hw_stat_init(&vf_dev->dev_pestat,
505− (u8)vf_dev->pmf_index,
506− dev->hw, false);
511+ i40iw_vf_init_pestat(dev, &vf_dev->pestat, vf_dev->pmf_index);
507512 vf_dev->stats_initialized = true;
508513 } else {
509514 if (vf_dev) {
@@ -534,12 +539,10 @@ enum i40iw_status_code i40iw_vchnl_recv_pf(struct i40iw_sc_dev *dev, · 24 unmodified lines
534539 case I40IW_VCHNL_OP_GET_STATS:
535540 if (!vf_dev)
536541 return I40IW_ERR_BAD_PTR;
537− devstat = &vf_dev->dev_pestat;
538− spin_lock_irqsave(&dev->dev_pestat.stats_lock, flags);
539− devstat->ops.iw_hw_stat_read_all(devstat, &devstat->hw_stats);
540− spin_unlock_irqrestore(&dev->dev_pestat.stats_lock, flags);
542+ stats = &vf_dev->pestat;
543+ i40iw_hw_stats_read_all(stats, &stats->hw_stats);
541544 vf_dev->msg_count--;
542− vchnl_pf_send_get_pe_stats_resp(dev, vf_id, vchnl_msg, &devstat->hw_stats);
545+ vchnl_pf_send_get_pe_stats_resp(dev, vf_id, vchnl_msg, &stats->hw_stats);
543546 break;
544547 default:
545548 i40iw_debug(dev, I40IW_DEBUG_VIRT,
drivers/infiniband/hw/mlx4/ah.c+7−3
@@ -114,7 +114,9 @@ static struct ib_ah *create_iboe_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr · 113 unmodified lines
114114 !(1 << ah->av.eth.stat_rate & dev->caps.stat_rate_support))
115115 --ah->av.eth.stat_rate;
116116 }
117−
117+ ah->av.eth.sl_tclass_flowlabel |=
118+ cpu_to_be32((ah_attr->grh.traffic_class << 20) |
119+ ah_attr->grh.flow_label);
118120 /*
119121 * HW requires multicast LID so we just choose one.
120122 */
@@ -122,12 +124,14 @@ static struct ib_ah *create_iboe_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr · 1 unmodified line
122124 ah->av.ib.dlid = cpu_to_be16(0xc000);
123125
124126 memcpy(ah->av.eth.dgid, ah_attr->grh.dgid.raw, 16);
125− ah->av.eth.sl_tclass_flowlabel = cpu_to_be32(ah_attr->sl << 29);
127+ ah->av.eth.sl_tclass_flowlabel |= cpu_to_be32(ah_attr->sl << 29);
126128
127129 return &ah->ibah;
128130 }
129131
130−struct ib_ah *mlx4_ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr)
132+struct ib_ah *mlx4_ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr,
133+ struct ib_udata *udata)
134+
131135 {
132136 struct mlx4_ib_ah *ah;
133137 struct ib_ah *ret;
drivers/infiniband/hw/mlx4/alias_GUID.c+1−3
@@ -755,10 +755,8 @@ static void alias_guid_work(struct work_struct *work) · 754 unmodified lines
755755 struct mlx4_ib_dev *dev = container_of(ib_sriov, struct mlx4_ib_dev, sriov);
756756
757757 rec = kzalloc(sizeof *rec, GFP_KERNEL);
758− if (!rec) {
759− pr_err("alias_guid_work: No Memory\n");
758+ if (!rec)
760759 return;
761− }
762760
763761 pr_debug("starting [port: %d]...\n", sriov_alias_port->port + 1);
764762 ret = get_next_record_to_update(dev, sriov_alias_port->port, rec);
drivers/infiniband/hw/mlx4/cm.c+1−3
@@ -247,10 +247,8 @@ id_map_alloc(struct ib_device *ibdev, int slave_id, u32 sl_cm_id) · 246 unmodified lines
247247 struct mlx4_ib_sriov *sriov = &to_mdev(ibdev)->sriov;
248248
249249 ent = kmalloc(sizeof (struct id_map_entry), GFP_KERNEL);
250− if (!ent) {
251− mlx4_ib_warn(ibdev, "Couldn't allocate id cache entry - out of memory\n");
250+ if (!ent)
252251 return ERR_PTR(-ENOMEM);
253− }
254252
255253 ent->sl_cm_id = sl_cm_id;
256254 ent->slave_id = slave_id;
drivers/infiniband/hw/mlx4/mad.c+44−14
@@ -39,6 +39,8 @@ · 38 unmodified lines
3939 #include <linux/mlx4/cmd.h>
4040 #include <linux/gfp.h>
4141 #include <rdma/ib_pma.h>
42+#include <linux/ip.h>
43+#include <net/ipv6.h>
4244
4345 #include <linux/mlx4/driver.h>
4446 #include "mlx4_ib.h"
@@ -480,6 +482,23 @@ static int find_slave_port_pkey_ix(struct mlx4_ib_dev *dev, int slave, · 435 unmodified lines
480482 return -EINVAL;
481483 }
482484
485+static int get_gids_from_l3_hdr(struct ib_grh *grh, union ib_gid *sgid,
486+ union ib_gid *dgid)
487+{
488+ int version = ib_get_rdma_header_version((const union rdma_network_hdr *)grh);
489+ enum rdma_network_type net_type;
490+
491+ if (version == 4)
492+ net_type = RDMA_NETWORK_IPV4;
493+ else if (version == 6)
494+ net_type = RDMA_NETWORK_IPV6;
495+ else
496+ return -EINVAL;
497+
498+ return ib_get_gids_from_rdma_hdr((union rdma_network_hdr *)grh, net_type,
499+ sgid, dgid);
500+}
501+
483502 int mlx4_ib_send_to_slave(struct mlx4_ib_dev *dev, int slave, u8 port,
484503 enum ib_qp_type dest_qpt, struct ib_wc *wc,
485504 struct ib_grh *grh, struct ib_mad *mad)
@@ -538,7 +557,10 @@ int mlx4_ib_send_to_slave(struct mlx4_ib_dev *dev, int slave, u8 port, · 52 unmodified lines
538557 memset(&attr, 0, sizeof attr);
539558 attr.port_num = port;
540559 if (is_eth) {
541− memcpy(&attr.grh.dgid.raw[0], &grh->dgid.raw[0], 16);
560+ union ib_gid sgid;
561+
562+ if (get_gids_from_l3_hdr(grh, &sgid, &attr.grh.dgid))
563+ return -EINVAL;
542564 attr.ah_flags = IB_AH_GRH;
543565 }
544566 ah = ib_create_ah(tun_ctx->pd, &attr);
@@ -651,6 +673,11 @@ static int mlx4_ib_demux_mad(struct ib_device *ibdev, u8 port, · 106 unmodified lines
651673 is_eth = 1;
652674
653675 if (is_eth) {
676+ union ib_gid dgid;
677+ union ib_gid sgid;
678+
679+ if (get_gids_from_l3_hdr(grh, &sgid, &dgid))
680+ return -EINVAL;
654681 if (!(wc->wc_flags & IB_WC_GRH)) {
655682 mlx4_ib_warn(ibdev, "RoCE grh not present.\n");
656683 return -EINVAL;
@@ -659,10 +686,10 @@ static int mlx4_ib_demux_mad(struct ib_device *ibdev, u8 port, · 2 unmodified lines
659686 mlx4_ib_warn(ibdev, "RoCE mgmt class is not CM\n");
660687 return -EINVAL;
661688 }
662− err = mlx4_get_slave_from_roce_gid(dev->dev, port, grh->dgid.raw, &slave);
689+ err = mlx4_get_slave_from_roce_gid(dev->dev, port, dgid.raw, &slave);
663690 if (err && mlx4_is_mf_bonded(dev->dev)) {
664691 other_port = (port == 1) ? 2 : 1;
665− err = mlx4_get_slave_from_roce_gid(dev->dev, other_port, grh->dgid.raw, &slave);
692+ err = mlx4_get_slave_from_roce_gid(dev->dev, other_port, dgid.raw, &slave);
666693 if (!err) {
667694 port = other_port;
668695 pr_debug("resolved slave %d from gid %pI6 wire port %d other %d\n",
@@ -702,10 +729,18 @@ static int mlx4_ib_demux_mad(struct ib_device *ibdev, u8 port, · 33 unmodified lines
702729
703730 /* If a grh is present, we demux according to it */
704731 if (wc->wc_flags & IB_WC_GRH) {
705− slave = mlx4_ib_find_real_gid(ibdev, port, grh->dgid.global.interface_id);
706− if (slave < 0) {
707− mlx4_ib_warn(ibdev, "failed matching grh\n");
708− return -ENOENT;
732+ if (grh->dgid.global.interface_id ==
733+ cpu_to_be64(IB_SA_WELL_KNOWN_GUID) &&
734+ grh->dgid.global.subnet_prefix == cpu_to_be64(
735+ atomic64_read(&dev->sriov.demux[port - 1].subnet_prefix))) {
736+ slave = 0;
737+ } else {
738+ slave = mlx4_ib_find_real_gid(ibdev, port,
739+ grh->dgid.global.interface_id);
740+ if (slave < 0) {
741+ mlx4_ib_warn(ibdev, "failed matching grh\n");
742+ return -ENOENT;
743+ }
709744 }
710745 }
711746 /* Class-specific handling */
@@ -1102,10 +1137,8 @@ static void handle_slaves_guid_change(struct mlx4_ib_dev *dev, u8 port_num, · 390 unmodified lines
11021137
11031138 in_mad = kmalloc(sizeof *in_mad, GFP_KERNEL);
11041139 out_mad = kmalloc(sizeof *out_mad, GFP_KERNEL);
1105− if (!in_mad || !out_mad) {
1106− mlx4_ib_warn(&dev->ib_dev, "failed to allocate memory for guid info mads\n");
1140+ if (!in_mad || !out_mad)
11071141 goto out;
1108− }
11091142
11101143 guid_tbl_blk_num *= 4;
11111144
@@ -1916,11 +1949,8 @@ static int alloc_pv_object(struct mlx4_ib_dev *dev, int slave, int port, · 804 unmodified lines
19161949
19171950 *ret_ctx = NULL;
19181951 ctx = kzalloc(sizeof (struct mlx4_ib_demux_pv_ctx), GFP_KERNEL);
1919− if (!ctx) {
1920− pr_err("failed allocating pv resource context "
1921− "for port %d, slave %d\n", port, slave);
1952+ if (!ctx)
19221953 return -ENOMEM;
1923− }
19241954
19251955 ctx->ib_dev = &dev->ib_dev;
19261956 ctx->port = port;
drivers/infiniband/hw/mlx4/main.c+24−25
@@ -430,7 +430,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev, · 429 unmodified lines
430430 struct mlx4_ib_dev *dev = to_mdev(ibdev);
431431 struct ib_smp *in_mad = NULL;
432432 struct ib_smp *out_mad = NULL;
433− int err = -ENOMEM;
433+ int err;
434434 int have_ib_ports;
435435 struct mlx4_uverbs_ex_query_device cmd;
436436 struct mlx4_uverbs_ex_query_device_resp resp = {.comp_mask = 0};
@@ -455,6 +455,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev, · 18 unmodified lines
455455 sizeof(resp.response_length);
456456 in_mad = kzalloc(sizeof *in_mad, GFP_KERNEL);
457457 out_mad = kmalloc(sizeof *out_mad, GFP_KERNEL);
458+ err = -ENOMEM;
458459 if (!in_mad || !out_mad)
459460 goto out;
460461
@@ -547,6 +548,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev, · 86 unmodified lines
547548 props->max_map_per_fmr = dev->dev->caps.max_fmr_maps;
548549 props->hca_core_clock = dev->dev->caps.hca_core_clock * 1000UL;
549550 props->timestamp_mask = 0xFFFFFFFFFFFFULL;
551+ props->max_ah = INT_MAX;
550552
551553 if (!mlx4_is_slave(dev->dev))
552554 err = mlx4_get_internal_clock_params(dev->dev, &clock_params);
@@ -697,9 +699,11 @@ static int eth_link_query_port(struct ib_device *ibdev, u8 port, · 144 unmodified lines
697699 if (err)
698700 goto out;
699701
700− props->active_width = (((u8 *)mailbox->buf)[5] == 0x40) ?
701− IB_WIDTH_4X : IB_WIDTH_1X;
702− props->active_speed = IB_SPEED_QDR;
702+ props->active_width = (((u8 *)mailbox->buf)[5] == 0x40) ||
703+ (((u8 *)mailbox->buf)[5] == 0x20 /*56Gb*/) ?
704+ IB_WIDTH_4X : IB_WIDTH_1X;
705+ props->active_speed = (((u8 *)mailbox->buf)[5] == 0x20 /*56Gb*/) ?
706+ IB_SPEED_FDR : IB_SPEED_QDR;
703707 props->port_cap_flags = IB_PORT_CM_SUP | IB_PORT_IP_BASED_GIDS;
704708 props->gid_tbl_len = mdev->dev->caps.gid_table_len[port];
705709 props->max_msg_sz = mdev->dev->caps.max_msg_sz;
@@ -2814,20 +2818,22 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) · 2108 unmodified lines
28142818 kmalloc(BITS_TO_LONGS(ibdev->steer_qpn_count) *
28152819 sizeof(long),
28162820 GFP_KERNEL);
2817− if (!ibdev->ib_uc_qpns_bitmap) {
2818− dev_err(&dev->persist->pdev->dev,
2819− "bit map alloc failed\n");
2821+ if (!ibdev->ib_uc_qpns_bitmap)
28202822 goto err_steer_qp_release;
2821− }
28222823
2823− bitmap_zero(ibdev->ib_uc_qpns_bitmap, ibdev->steer_qpn_count);
2824−
2825− err = mlx4_FLOW_STEERING_IB_UC_QP_RANGE(
2826− dev, ibdev->steer_qpn_base,
2827− ibdev->steer_qpn_base +
2828− ibdev->steer_qpn_count - 1);
2829− if (err)
2830− goto err_steer_free_bitmap;
2824+ if (dev->caps.flags2 & MLX4_DEV_CAP_FLAG2_DMFS_IPOIB) {
2825+ bitmap_zero(ibdev->ib_uc_qpns_bitmap,
2826+ ibdev->steer_qpn_count);
2827+ err = mlx4_FLOW_STEERING_IB_UC_QP_RANGE(
2828+ dev, ibdev->steer_qpn_base,
2829+ ibdev->steer_qpn_base +
2830+ ibdev->steer_qpn_count - 1);
2831+ if (err)
2832+ goto err_steer_free_bitmap;
2833+ } else {
2834+ bitmap_fill(ibdev->ib_uc_qpns_bitmap,
2835+ ibdev->steer_qpn_count);
2836+ }
28312837 }
28322838
28332839 for (j = 1; j <= ibdev->dev->caps.num_ports; j++)
@@ -3055,15 +3061,12 @@ static void do_slave_init(struct mlx4_ib_dev *ibdev, int slave, int do_init) · 221 unmodified lines
30553061 first_port = find_first_bit(actv_ports.ports, dev->caps.num_ports);
30563062
30573063 dm = kcalloc(ports, sizeof(*dm), GFP_ATOMIC);
3058− if (!dm) {
3059− pr_err("failed to allocate memory for tunneling qp update\n");
3064+ if (!dm)
30603065 return;
3061− }
30623066
30633067 for (i = 0; i < ports; i++) {
30643068 dm[i] = kmalloc(sizeof (struct mlx4_ib_demux_work), GFP_ATOMIC);
30653069 if (!dm[i]) {
3066− pr_err("failed to allocate memory for tunneling qp update work struct\n");
30673070 while (--i >= 0)
30683071 kfree(dm[i]);
30693072 goto out;
@@ -3223,8 +3226,6 @@ void mlx4_sched_ib_sl2vl_update_work(struct mlx4_ib_dev *ibdev, · 153 unmodified lines
32233226 ew->port = port;
32243227 ew->ib_dev = ibdev;
32253228 queue_work(wq, &ew->work);
3226− } else {
3227− pr_err("failed to allocate memory for sl2vl update work\n");
32283229 }
32293230 }
32303231
@@ -3284,10 +3285,8 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, · 53 unmodified lines
32843285
32853286 case MLX4_DEV_EVENT_PORT_MGMT_CHANGE:
32863287 ew = kmalloc(sizeof *ew, GFP_ATOMIC);
3287− if (!ew) {
3288− pr_err("failed to allocate memory for events work\n");
3288+ if (!ew)
32893289 break;
3290− }
32913290
32923291 INIT_WORK(&ew->work, handle_port_mgmt_change_event);
32933292 memcpy(&ew->ib_eqe, eqe, sizeof *eqe);
drivers/infiniband/hw/mlx4/mcg.c+1−4
@@ -1142,7 +1142,6 @@ void mlx4_ib_mcg_port_cleanup(struct mlx4_ib_demux_ctx *ctx, int destroy_wq) · 1141 unmodified lines
11421142 work = kmalloc(sizeof *work, GFP_KERNEL);
11431143 if (!work) {
11441144 ctx->flushing = 0;
1145− mcg_warn("failed allocating work for cleanup\n");
11461145 return;
11471146 }
11481147
@@ -1202,10 +1201,8 @@ static int push_deleteing_req(struct mcast_group *group, int slave) · 53 unmodified lines
12021201 return 0;
12031202
12041203 req = kzalloc(sizeof *req, GFP_KERNEL);
1205− if (!req) {
1206− mcg_warn_group(group, "failed allocation - may leave stall groups\n");
1204+ if (!req)
12071205 return -ENOMEM;
1208− }
12091206
12101207 if (!list_empty(&group->func[slave].pending)) {
12111208 pend_req = list_entry(group->func[slave].pending.prev, struct mcast_req, group_list);
drivers/infiniband/hw/mlx4/mlx4_ib.h+2−1
@@ -742,7 +742,8 @@ int mlx4_ib_arm_cq(struct ib_cq *cq, enum ib_cq_notify_flags flags); · 741 unmodified lines
742742 void __mlx4_ib_cq_clean(struct mlx4_ib_cq *cq, u32 qpn, struct mlx4_ib_srq *srq);
743743 void mlx4_ib_cq_clean(struct mlx4_ib_cq *cq, u32 qpn, struct mlx4_ib_srq *srq);
744744
745−struct ib_ah *mlx4_ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr);
745+struct ib_ah *mlx4_ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr,
746+ struct ib_udata *udata);
746747 int mlx4_ib_query_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr);
747748 int mlx4_ib_destroy_ah(struct ib_ah *ah);
748749
drivers/infiniband/hw/mlx4/qp.c+8−5
@@ -644,7 +644,7 @@ static int create_qp_common(struct mlx4_ib_dev *dev, struct ib_pd *pd, · 643 unmodified lines
644644 int qpn;
645645 int err;
646646 struct ib_qp_cap backup_cap;
647− struct mlx4_ib_sqp *sqp;
647+ struct mlx4_ib_sqp *sqp = NULL;
648648 struct mlx4_ib_qp *qp;
649649 enum mlx4_ib_qp_type qp_type = (enum mlx4_ib_qp_type) init_attr->qp_type;
650650 struct mlx4_ib_cq *mcq;
@@ -933,7 +933,9 @@ err_db: · 282 unmodified lines
933933 mlx4_db_free(dev->dev, &qp->db);
934934
935935 err:
936− if (!*caller_qp)
936+ if (sqp)
937+ kfree(sqp);
938+ else if (!*caller_qp)
937939 kfree(qp);
938940 return err;
939941 }
@@ -1280,7 +1282,8 @@ static int _mlx4_ib_destroy_qp(struct ib_qp *qp) · 340 unmodified lines
12801282 if (is_qp0(dev, mqp))
12811283 mlx4_CLOSE_PORT(dev->dev, mqp->port);
12821284
1283− if (dev->qp1_proxy[mqp->port - 1] == mqp) {
1285+ if (mqp->mlx4_ib_qp_type == MLX4_IB_QPT_PROXY_GSI &&
1286+ dev->qp1_proxy[mqp->port - 1] == mqp) {
12841287 mutex_lock(&dev->qp1_proxy_lock[mqp->port - 1]);
12851288 dev->qp1_proxy[mqp->port - 1] = NULL;
12861289 mutex_unlock(&dev->qp1_proxy_lock[mqp->port - 1]);
@@ -1764,14 +1767,14 @@ static int __mlx4_ib_modify_qp(struct ib_qp *ibqp, · 477 unmodified lines
17641767 u8 port_num = mlx4_is_bonded(to_mdev(ibqp->device)->dev) ? 1 :
17651768 attr_mask & IB_QP_PORT ? attr->port_num : qp->port;
17661769 union ib_gid gid;
1767− struct ib_gid_attr gid_attr;
1770+ struct ib_gid_attr gid_attr = {.gid_type = IB_GID_TYPE_IB};
17681771 u16 vlan = 0xffff;
17691772 u8 smac[ETH_ALEN];
17701773 int status = 0;
17711774 int is_eth = rdma_cap_eth_ah(&dev->ib_dev, port_num) &&
17721775 attr->ah_attr.ah_flags & IB_AH_GRH;
17731776
1774− if (is_eth) {
1777+ if (is_eth && attr->ah_attr.ah_flags & IB_AH_GRH) {
17751778 int index = attr->ah_attr.grh.sgid_index;
17761779
17771780 status = ib_get_cached_gid(ibqp->device, port_num,
drivers/infiniband/hw/mlx5/ah.c+24−1
@@ -64,7 +64,9 @@ static struct ib_ah *create_ib_ah(struct mlx5_ib_dev *dev, · 63 unmodified lines
6464 return &ah->ibah;
6565 }
6666
67−struct ib_ah *mlx5_ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr)
67+struct ib_ah *mlx5_ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr,
68+ struct ib_udata *udata)
69+
6870 {
6971 struct mlx5_ib_ah *ah;
7072 struct mlx5_ib_dev *dev = to_mdev(pd->device);
@@ -75,6 +77,27 @@ struct ib_ah *mlx5_ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr) · 4 unmodified lines
7577 if (ll == IB_LINK_LAYER_ETHERNET && !(ah_attr->ah_flags & IB_AH_GRH))
7678 return ERR_PTR(-EINVAL);
7779
80+ if (ll == IB_LINK_LAYER_ETHERNET && udata) {
81+ int err;
82+ struct mlx5_ib_create_ah_resp resp = {};
83+ u32 min_resp_len = offsetof(typeof(resp), dmac) +
84+ sizeof(resp.dmac);
85+
86+ if (udata->outlen < min_resp_len)
87+ return ERR_PTR(-EINVAL);
88+
89+ resp.response_length = min_resp_len;
90+
91+ err = ib_resolve_eth_dmac(pd->device, ah_attr);
92+ if (err)
93+ return ERR_PTR(err);
94+
95+ memcpy(resp.dmac, ah_attr->dmac, ETH_ALEN);
96+ err = ib_copy_to_udata(udata, &resp, resp.response_length);
97+ if (err)
98+ return ERR_PTR(err);
99+ }
100+
78101 ah = kzalloc(sizeof(*ah), GFP_ATOMIC);
79102 if (!ah)
80103 return ERR_PTR(-ENOMEM);
drivers/infiniband/hw/mlx5/cq.c+31−3
@@ -731,7 +731,7 @@ static int create_cq_user(struct mlx5_ib_dev *dev, struct ib_udata *udata, · 730 unmodified lines
731731 int entries, u32 **cqb,
732732 int *cqe_size, int *index, int *inlen)
733733 {
734− struct mlx5_ib_create_cq ucmd;
734+ struct mlx5_ib_create_cq ucmd = {};
735735 size_t ucmdlen;
736736 int page_shift;
737737 __be64 *pas;
@@ -770,7 +770,7 @@ static int create_cq_user(struct mlx5_ib_dev *dev, struct ib_udata *udata, · 32 unmodified lines
770770 if (err)
771771 goto err_umem;
772772
773− mlx5_ib_cont_pages(cq->buf.umem, ucmd.buf_addr, &npages, &page_shift,
773+ mlx5_ib_cont_pages(cq->buf.umem, ucmd.buf_addr, 0, &npages, &page_shift,
774774 &ncont, NULL);
775775 mlx5_ib_dbg(dev, "addr 0x%llx, size %u, npages %d, page_shift %d, ncont %d\n",
776776 ucmd.buf_addr, entries * ucmd.cqe_size, npages, page_shift, ncont);
@@ -792,8 +792,36 @@ static int create_cq_user(struct mlx5_ib_dev *dev, struct ib_udata *udata, · 15 unmodified lines
792792
793793 *index = to_mucontext(context)->uuari.uars[0].index;
794794
795+ if (ucmd.cqe_comp_en == 1) {
796+ if (unlikely((*cqe_size != 64) ||
797+ !MLX5_CAP_GEN(dev->mdev, cqe_compression))) {
798+ err = -EOPNOTSUPP;
799+ mlx5_ib_warn(dev, "CQE compression is not supported for size %d!\n",
800+ *cqe_size);
801+ goto err_cqb;
802+ }
803+
804+ if (unlikely(!ucmd.cqe_comp_res_format ||
805+ !(ucmd.cqe_comp_res_format <
806+ MLX5_IB_CQE_RES_RESERVED) ||
807+ (ucmd.cqe_comp_res_format &
808+ (ucmd.cqe_comp_res_format - 1)))) {
809+ err = -EOPNOTSUPP;
810+ mlx5_ib_warn(dev, "CQE compression res format %d is not supported!\n",
811+ ucmd.cqe_comp_res_format);
812+ goto err_cqb;
813+ }
814+
815+ MLX5_SET(cqc, cqc, cqe_comp_en, 1);
816+ MLX5_SET(cqc, cqc, mini_cqe_res_format,
817+ ilog2(ucmd.cqe_comp_res_format));
818+ }
819+
795820 return 0;
796821
822+err_cqb:
823+ kfree(cqb);
824+
797825 err_db:
798826 mlx5_ib_db_unmap_user(to_mucontext(context), &cq->db);
799827
@@ -1124,7 +1152,7 @@ static int resize_user(struct mlx5_ib_dev *dev, struct mlx5_ib_cq *cq, · 324 unmodified lines
11241152 return err;
11251153 }
11261154
1127− mlx5_ib_cont_pages(umem, ucmd.buf_addr, &npages, page_shift,
1155+ mlx5_ib_cont_pages(umem, ucmd.buf_addr, 0, &npages, page_shift,
11281156 npas, NULL);
11291157
11301158 cq->resize_umem = umem;
drivers/infiniband/hw/mlx5/main.c+180−88
@@ -127,7 +127,7 @@ static int mlx5_netdev_event(struct notifier_block *this, · 126 unmodified lines
127127
128128 if ((upper == ndev || (!upper && ndev == ibdev->roce.netdev))
129129 && ibdev->ib_active) {
130− struct ib_event ibev = {0};
130+ struct ib_event ibev = { };
131131
132132 ibev.device = &ibdev->ib_dev;
133133 ibev.event = (event == NETDEV_UP) ?
@@ -496,6 +496,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 362 unmodified lines
496496 struct mlx5_ib_dev *dev = to_mdev(ibdev);
497497 struct mlx5_core_dev *mdev = dev->mdev;
498498 int err = -ENOMEM;
499+ int max_sq_desc;
499500 int max_rq_sg;
500501 int max_sq_sg;
501502 u64 min_page_size = 1ull << MLX5_CAP_GEN(mdev, log_pg_sz);
@@ -618,9 +619,10 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 116 unmodified lines
618619 props->max_qp_wr = 1 << MLX5_CAP_GEN(mdev, log_max_qp_sz);
619620 max_rq_sg = MLX5_CAP_GEN(mdev, max_wqe_sz_rq) /
620621 sizeof(struct mlx5_wqe_data_seg);
621− max_sq_sg = (MLX5_CAP_GEN(mdev, max_wqe_sz_sq) -
622− sizeof(struct mlx5_wqe_ctrl_seg)) /
623− sizeof(struct mlx5_wqe_data_seg);
622+ max_sq_desc = min_t(int, MLX5_CAP_GEN(mdev, max_wqe_sz_sq), 512);
623+ max_sq_sg = (max_sq_desc - sizeof(struct mlx5_wqe_ctrl_seg) -
624+ sizeof(struct mlx5_wqe_raddr_seg)) /
625+ sizeof(struct mlx5_wqe_data_seg);
624626 props->max_sge = min(max_rq_sg, max_sq_sg);
625627 props->max_sge_rd = MLX5_MAX_SGE_RD;
626628 props->max_cq = 1 << MLX5_CAP_GEN(mdev, log_max_cq);
@@ -643,6 +645,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 16 unmodified lines
643645 props->max_total_mcast_qp_attach = props->max_mcast_qp_attach *
644646 props->max_mcast_grp;
645647 props->max_map_per_fmr = INT_MAX; /* no limit in ConnectIB */
648+ props->max_ah = INT_MAX;
646649 props->hca_core_clock = MLX5_CAP_GEN(mdev, device_frequency_khz);
647650 props->timestamp_mask = 0x7FFFFFFFFFFFFFFFULL;
648651
@@ -669,6 +672,40 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, · 20 unmodified lines
669672 1 << MLX5_CAP_GEN(dev->mdev, log_max_rq);
670673 }
671674
675+ if (field_avail(typeof(resp), mlx5_ib_support_multi_pkt_send_wqes,
676+ uhw->outlen)) {
677+ resp.mlx5_ib_support_multi_pkt_send_wqes =
678+ MLX5_CAP_ETH(mdev, multi_pkt_send_wqe);
679+ resp.response_length +=
680+ sizeof(resp.mlx5_ib_support_multi_pkt_send_wqes);
681+ }
682+
683+ if (field_avail(typeof(resp), reserved, uhw->outlen))
684+ resp.response_length += sizeof(resp.reserved);
685+
686+ if (field_avail(typeof(resp), cqe_comp_caps, uhw->outlen)) {
687+ resp.cqe_comp_caps.max_num =
688+ MLX5_CAP_GEN(dev->mdev, cqe_compression) ?
689+ MLX5_CAP_GEN(dev->mdev, cqe_compression_max_num) : 0;
690+ resp.cqe_comp_caps.supported_format =
691+ MLX5_IB_CQE_RES_FORMAT_HASH |
692+ MLX5_IB_CQE_RES_FORMAT_CSUM;
693+ resp.response_length += sizeof(resp.cqe_comp_caps);
694+ }
695+
696+ if (field_avail(typeof(resp), packet_pacing_caps, uhw->outlen)) {
697+ if (MLX5_CAP_QOS(mdev, packet_pacing) &&
698+ MLX5_CAP_GEN(mdev, qos)) {
699+ resp.packet_pacing_caps.qp_rate_limit_max =
700+ MLX5_CAP_QOS(mdev, packet_pacing_max_rate);
701+ resp.packet_pacing_caps.qp_rate_limit_min =
702+ MLX5_CAP_QOS(mdev, packet_pacing_min_rate);
703+ resp.packet_pacing_caps.supported_qpts |=
704+ 1 << IB_QPT_RAW_PACKET;
705+ }
706+ resp.response_length += sizeof(resp.packet_pacing_caps);
707+ }
708+
672709 if (uhw->outlen) {
673710 err = ib_copy_to_udata(uhw, &resp, resp.response_length);
674711
@@ -1093,7 +1130,8 @@ static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev, · 418 unmodified lines
10931130 resp.response_length += sizeof(resp.cqe_version);
10941131
10951132 if (field_avail(typeof(resp), cmds_supp_uhw, udata->outlen)) {
1096− resp.cmds_supp_uhw |= MLX5_USER_CMDS_SUPP_UHW_QUERY_DEVICE;
1133+ resp.cmds_supp_uhw |= MLX5_USER_CMDS_SUPP_UHW_QUERY_DEVICE |
1134+ MLX5_USER_CMDS_SUPP_UHW_CREATE_AH;
10971135 resp.response_length += sizeof(resp.cmds_supp_uhw);
10981136 }
10991137
@@ -1502,6 +1540,22 @@ static void set_proto(void *outer_c, void *outer_v, u8 mask, u8 val) · 402 unmodified lines
15021540 MLX5_SET(fte_match_set_lyr_2_4, outer_v, ip_protocol, val);
15031541 }
15041542
1543+static void set_flow_label(void *misc_c, void *misc_v, u8 mask, u8 val,
1544+ bool inner)
1545+{
1546+ if (inner) {
1547+ MLX5_SET(fte_match_set_misc,
1548+ misc_c, inner_ipv6_flow_label, mask);
1549+ MLX5_SET(fte_match_set_misc,
1550+ misc_v, inner_ipv6_flow_label, val);
1551+ } else {
1552+ MLX5_SET(fte_match_set_misc,
1553+ misc_c, outer_ipv6_flow_label, mask);
1554+ MLX5_SET(fte_match_set_misc,
1555+ misc_v, outer_ipv6_flow_label, val);
1556+ }
1557+}
1558+
15051559 static void set_tos(void *outer_c, void *outer_v, u8 mask, u8 val)
15061560 {
15071561 MLX5_SET(fte_match_set_lyr_2_4, outer_c, ip_ecn, mask);
@@ -1515,6 +1569,7 @@ static void set_tos(void *outer_c, void *outer_v, u8 mask, u8 val) · 7 unmodified lines
15151569 #define LAST_IPV4_FIELD tos
15161570 #define LAST_IPV6_FIELD traffic_class
15171571 #define LAST_TCP_UDP_FIELD src_port
1572+#define LAST_TUNNEL_FIELD tunnel_id
15181573
15191574 /* Field is the last supported field */
15201575 #define FIELDS_NOT_SUPPORTED(filter, field)\
@@ -1527,155 +1582,164 @@ static void set_tos(void *outer_c, void *outer_v, u8 mask, u8 val) · 6 unmodified lines
15271582 static int parse_flow_attr(u32 *match_c, u32 *match_v,
15281583 const union ib_flow_spec *ib_spec)
15291584 {
1530− void *outer_headers_c = MLX5_ADDR_OF(fte_match_param, match_c,
1531− outer_headers);
1532− void *outer_headers_v = MLX5_ADDR_OF(fte_match_param, match_v,
1533− outer_headers);
15341585 void *misc_params_c = MLX5_ADDR_OF(fte_match_param, match_c,
15351586 misc_parameters);
15361587 void *misc_params_v = MLX5_ADDR_OF(fte_match_param, match_v,
15371588 misc_parameters);
1589+ void *headers_c;
1590+ void *headers_v;
1591+
1592+ if (ib_spec->type & IB_FLOW_SPEC_INNER) {
1593+ headers_c = MLX5_ADDR_OF(fte_match_param, match_c,
1594+ inner_headers);
1595+ headers_v = MLX5_ADDR_OF(fte_match_param, match_v,
1596+ inner_headers);
1597+ } else {
1598+ headers_c = MLX5_ADDR_OF(fte_match_param, match_c,
1599+ outer_headers);
1600+ headers_v = MLX5_ADDR_OF(fte_match_param, match_v,
1601+ outer_headers);
1602+ }
15381603
1539− switch (ib_spec->type) {
1604+ switch (ib_spec->type & ~IB_FLOW_SPEC_INNER) {
15401605 case IB_FLOW_SPEC_ETH:
15411606 if (FIELDS_NOT_SUPPORTED(ib_spec->eth.mask, LAST_ETH_FIELD))
15421607 return -ENOTSUPP;
15431608
1544− ether_addr_copy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_c,
1609+ ether_addr_copy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, headers_c,
15451610 dmac_47_16),
15461611 ib_spec->eth.mask.dst_mac);
1547− ether_addr_copy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_v,
1612+ ether_addr_copy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, headers_v,
15481613 dmac_47_16),
15491614 ib_spec->eth.val.dst_mac);
15501615
1551− ether_addr_copy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_c,
1616+ ether_addr_copy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, headers_c,
15521617 smac_47_16),
15531618 ib_spec->eth.mask.src_mac);
1554− ether_addr_copy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_v,
1619+ ether_addr_copy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, headers_v,
15551620 smac_47_16),
15561621 ib_spec->eth.val.src_mac);
15571622
15581623 if (ib_spec->eth.mask.vlan_tag) {
1559− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c,
1624+ MLX5_SET(fte_match_set_lyr_2_4, headers_c,
15601625 vlan_tag, 1);
1561− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v,
1626+ MLX5_SET(fte_match_set_lyr_2_4, headers_v,
15621627 vlan_tag, 1);
15631628
1564− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c,
1629+ MLX5_SET(fte_match_set_lyr_2_4, headers_c,
15651630 first_vid, ntohs(ib_spec->eth.mask.vlan_tag));
1566− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v,
1631+ MLX5_SET(fte_match_set_lyr_2_4, headers_v,
15671632 first_vid, ntohs(ib_spec->eth.val.vlan_tag));
15681633
1569− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c,
1634+ MLX5_SET(fte_match_set_lyr_2_4, headers_c,
15701635 first_cfi,
15711636 ntohs(ib_spec->eth.mask.vlan_tag) >> 12);
1572− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v,
1637+ MLX5_SET(fte_match_set_lyr_2_4, headers_v,
15731638 first_cfi,
15741639 ntohs(ib_spec->eth.val.vlan_tag) >> 12);
15751640
1576− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c,
1641+ MLX5_SET(fte_match_set_lyr_2_4, headers_c,
15771642 first_prio,
15781643 ntohs(ib_spec->eth.mask.vlan_tag) >> 13);
1579− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v,
1644+ MLX5_SET(fte_match_set_lyr_2_4, headers_v,
15801645 first_prio,
15811646 ntohs(ib_spec->eth.val.vlan_tag) >> 13);
15821647 }
1583− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c,
1648+ MLX5_SET(fte_match_set_lyr_2_4, headers_c,
15841649 ethertype, ntohs(ib_spec->eth.mask.ether_type));
1585− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v,
1650+ MLX5_SET(fte_match_set_lyr_2_4, headers_v,
15861651 ethertype, ntohs(ib_spec->eth.val.ether_type));
15871652 break;
15881653 case IB_FLOW_SPEC_IPV4:
15891654 if (FIELDS_NOT_SUPPORTED(ib_spec->ipv4.mask, LAST_IPV4_FIELD))
15901655 return -ENOTSUPP;
15911656
1592− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c,
1657+ MLX5_SET(fte_match_set_lyr_2_4, headers_c,
15931658 ethertype, 0xffff);
1594− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v,
1659+ MLX5_SET(fte_match_set_lyr_2_4, headers_v,
15951660 ethertype, ETH_P_IP);
15961661
1597− memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_c,
1662+ memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, headers_c,
15981663 src_ipv4_src_ipv6.ipv4_layout.ipv4),
15991664 &ib_spec->ipv4.mask.src_ip,
16001665 sizeof(ib_spec->ipv4.mask.src_ip));
1601− memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_v,
1666+ memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, headers_v,
16021667 src_ipv4_src_ipv6.ipv4_layout.ipv4),
16031668 &ib_spec->ipv4.val.src_ip,
16041669 sizeof(ib_spec->ipv4.val.src_ip));
1605− memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_c,
1670+ memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, headers_c,
16061671 dst_ipv4_dst_ipv6.ipv4_layout.ipv4),
16071672 &ib_spec->ipv4.mask.dst_ip,
16081673 sizeof(ib_spec->ipv4.mask.dst_ip));
1609− memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_v,
1674+ memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, headers_v,
16101675 dst_ipv4_dst_ipv6.ipv4_layout.ipv4),
16111676 &ib_spec->ipv4.val.dst_ip,
16121677 sizeof(ib_spec->ipv4.val.dst_ip));
16131678
1614− set_tos(outer_headers_c, outer_headers_v,
1679+ set_tos(headers_c, headers_v,
16151680 ib_spec->ipv4.mask.tos, ib_spec->ipv4.val.tos);
16161681
1617− set_proto(outer_headers_c, outer_headers_v,
1682+ set_proto(headers_c, headers_v,
16181683 ib_spec->ipv4.mask.proto, ib_spec->ipv4.val.proto);
16191684 break;
16201685 case IB_FLOW_SPEC_IPV6:
16211686 if (FIELDS_NOT_SUPPORTED(ib_spec->ipv6.mask, LAST_IPV6_FIELD))
16221687 return -ENOTSUPP;
16231688
1624− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c,
1689+ MLX5_SET(fte_match_set_lyr_2_4, headers_c,
16251690 ethertype, 0xffff);
1626− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v,
1691+ MLX5_SET(fte_match_set_lyr_2_4, headers_v,
16271692 ethertype, ETH_P_IPV6);
16281693
1629− memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_c,
1694+ memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, headers_c,
16301695 src_ipv4_src_ipv6.ipv6_layout.ipv6),
16311696 &ib_spec->ipv6.mask.src_ip,
16321697 sizeof(ib_spec->ipv6.mask.src_ip));
1633− memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_v,
1698+ memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, headers_v,
16341699 src_ipv4_src_ipv6.ipv6_layout.ipv6),
16351700 &ib_spec->ipv6.val.src_ip,
16361701 sizeof(ib_spec->ipv6.val.src_ip));
1637− memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_c,
1702+ memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, headers_c,
16381703 dst_ipv4_dst_ipv6.ipv6_layout.ipv6),
16391704 &ib_spec->ipv6.mask.dst_ip,
16401705 sizeof(ib_spec->ipv6.mask.dst_ip));
1641− memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_v,
1706+ memcpy(MLX5_ADDR_OF(fte_match_set_lyr_2_4, headers_v,
16421707 dst_ipv4_dst_ipv6.ipv6_layout.ipv6),
16431708 &ib_spec->ipv6.val.dst_ip,
16441709 sizeof(ib_spec->ipv6.val.dst_ip));
16451710
1646− set_tos(outer_headers_c, outer_headers_v,
1711+ set_tos(headers_c, headers_v,
16471712 ib_spec->ipv6.mask.traffic_class,
16481713 ib_spec->ipv6.val.traffic_class);
16491714
1650− set_proto(outer_headers_c, outer_headers_v,
1715+ set_proto(headers_c, headers_v,
16511716 ib_spec->ipv6.mask.next_hdr,
16521717 ib_spec->ipv6.val.next_hdr);
16531718
1654− MLX5_SET(fte_match_set_misc, misc_params_c,
1655− outer_ipv6_flow_label,
1656− ntohl(ib_spec->ipv6.mask.flow_label));
1657− MLX5_SET(fte_match_set_misc, misc_params_v,
1658− outer_ipv6_flow_label,
1659− ntohl(ib_spec->ipv6.val.flow_label));
1719+ set_flow_label(misc_params_c, misc_params_v,
1720+ ntohl(ib_spec->ipv6.mask.flow_label),
1721+ ntohl(ib_spec->ipv6.val.flow_label),
1722+ ib_spec->type & IB_FLOW_SPEC_INNER);
1723+
16601724 break;
16611725 case IB_FLOW_SPEC_TCP:
16621726 if (FIELDS_NOT_SUPPORTED(ib_spec->tcp_udp.mask,
16631727 LAST_TCP_UDP_FIELD))
16641728 return -ENOTSUPP;
16651729
1666− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c, ip_protocol,
1730+ MLX5_SET(fte_match_set_lyr_2_4, headers_c, ip_protocol,
16671731 0xff);
1668− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v, ip_protocol,
1732+ MLX5_SET(fte_match_set_lyr_2_4, headers_v, ip_protocol,
16691733 IPPROTO_TCP);
16701734
1671− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c, tcp_sport,
1735+ MLX5_SET(fte_match_set_lyr_2_4, headers_c, tcp_sport,
16721736 ntohs(ib_spec->tcp_udp.mask.src_port));
1673− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v, tcp_sport,
1737+ MLX5_SET(fte_match_set_lyr_2_4, headers_v, tcp_sport,
16741738 ntohs(ib_spec->tcp_udp.val.src_port));
16751739
1676− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c, tcp_dport,
1740+ MLX5_SET(fte_match_set_lyr_2_4, headers_c, tcp_dport,
16771741 ntohs(ib_spec->tcp_udp.mask.dst_port));
1678− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v, tcp_dport,
1742+ MLX5_SET(fte_match_set_lyr_2_4, headers_v, tcp_dport,
16791743 ntohs(ib_spec->tcp_udp.val.dst_port));
16801744 break;
16811745 case IB_FLOW_SPEC_UDP:
@@ -1683,21 +1747,31 @@ static int parse_flow_attr(u32 *match_c, u32 *match_v, · 1 unmodified line
16831747 LAST_TCP_UDP_FIELD))
16841748 return -ENOTSUPP;
16851749
1686− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c, ip_protocol,
1750+ MLX5_SET(fte_match_set_lyr_2_4, headers_c, ip_protocol,
16871751 0xff);
1688− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v, ip_protocol,
1752+ MLX5_SET(fte_match_set_lyr_2_4, headers_v, ip_protocol,
16891753 IPPROTO_UDP);
16901754
1691− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c, udp_sport,
1755+ MLX5_SET(fte_match_set_lyr_2_4, headers_c, udp_sport,
16921756 ntohs(ib_spec->tcp_udp.mask.src_port));
1693− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v, udp_sport,
1757+ MLX5_SET(fte_match_set_lyr_2_4, headers_v, udp_sport,
16941758 ntohs(ib_spec->tcp_udp.val.src_port));
16951759
1696− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_c, udp_dport,
1760+ MLX5_SET(fte_match_set_lyr_2_4, headers_c, udp_dport,
16971761 ntohs(ib_spec->tcp_udp.mask.dst_port));
1698− MLX5_SET(fte_match_set_lyr_2_4, outer_headers_v, udp_dport,
1762+ MLX5_SET(fte_match_set_lyr_2_4, headers_v, udp_dport,
16991763 ntohs(ib_spec->tcp_udp.val.dst_port));
17001764 break;
1765+ case IB_FLOW_SPEC_VXLAN_TUNNEL:
1766+ if (FIELDS_NOT_SUPPORTED(ib_spec->tunnel.mask,
1767+ LAST_TUNNEL_FIELD))
1768+ return -ENOTSUPP;
1769+
1770+ MLX5_SET(fte_match_set_misc, misc_params_c, vxlan_vni,
1771+ ntohl(ib_spec->tunnel.mask.tunnel_id));
1772+ MLX5_SET(fte_match_set_misc, misc_params_v, vxlan_vni,
1773+ ntohl(ib_spec->tunnel.val.tunnel_id));
1774+ break;
17011775 default:
17021776 return -EINVAL;
17031777 }
@@ -2721,6 +2795,8 @@ static int mlx5_port_immutable(struct ib_device *ibdev, u8 port_num, · 1017 unmodified lines
27212795 struct ib_port_immutable *immutable)
27222796 {
27232797 struct ib_port_attr attr;
2798+ struct mlx5_ib_dev *dev = to_mdev(ibdev);
2799+ enum rdma_link_layer ll = mlx5_ib_port_link_layer(ibdev, port_num);
27242800 int err;
27252801
27262802 err = mlx5_ib_query_port(ibdev, port_num, &attr);
@@ -2730,7 +2806,8 @@ static int mlx5_port_immutable(struct ib_device *ibdev, u8 port_num, · 3 unmodified lines
27302806 immutable->pkey_tbl_len = attr.pkey_tbl_len;
27312807 immutable->gid_tbl_len = attr.gid_tbl_len;
27322808 immutable->core_cap_flags = get_core_cap_flags(ibdev);
2733− immutable->max_mad_size = IB_MGMT_MAD_SIZE;
2809+ if ((ll == IB_LINK_LAYER_INFINIBAND) || MLX5_CAP_GEN(dev->mdev, roce))
2810+ immutable->max_mad_size = IB_MGMT_MAD_SIZE;
27342811
27352812 return 0;
27362813 }
@@ -2744,7 +2821,7 @@ static void get_dev_fw_str(struct ib_device *ibdev, char *str, · 7 unmodified lines
27442821 fw_rev_min(dev->mdev), fw_rev_sub(dev->mdev));
27452822 }
27462823
2747−static int mlx5_roce_lag_init(struct mlx5_ib_dev *dev)
2824+static int mlx5_eth_lag_init(struct mlx5_ib_dev *dev)
27482825 {
27492826 struct mlx5_core_dev *mdev = dev->mdev;
27502827 struct mlx5_flow_namespace *ns = mlx5_get_flow_namespace(mdev,
@@ -2773,7 +2850,7 @@ err_destroy_vport_lag: · 22 unmodified lines
27732850 return err;
27742851 }
27752852
2776−static void mlx5_roce_lag_cleanup(struct mlx5_ib_dev *dev)
2853+static void mlx5_eth_lag_cleanup(struct mlx5_ib_dev *dev)
27772854 {
27782855 struct mlx5_core_dev *mdev = dev->mdev;
27792856
@@ -2785,7 +2862,21 @@ static void mlx5_roce_lag_cleanup(struct mlx5_ib_dev *dev) · 5 unmodified lines
27852862 }
27862863 }
27872864
2788−static void mlx5_remove_roce_notifier(struct mlx5_ib_dev *dev)
2865+static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev)
2866+{
2867+ int err;
2868+
2869+ dev->roce.nb.notifier_call = mlx5_netdev_event;
2870+ err = register_netdevice_notifier(&dev->roce.nb);
2871+ if (err) {
2872+ dev->roce.nb.notifier_call = NULL;
2873+ return err;
2874+ }
2875+
2876+ return 0;
2877+}
2878+
2879+static void mlx5_remove_netdev_notifier(struct mlx5_ib_dev *dev)
27892880 {
27902881 if (dev->roce.nb.notifier_call) {
27912882 unregister_netdevice_notifier(&dev->roce.nb);
@@ -2793,39 +2884,40 @@ static void mlx5_remove_roce_notifier(struct mlx5_ib_dev *dev) · 1 unmodified line
27932884 }
27942885 }
27952886
2796−static int mlx5_enable_roce(struct mlx5_ib_dev *dev)
2887+static int mlx5_enable_eth(struct mlx5_ib_dev *dev)
27972888 {
27982889 int err;
27992890
2800− dev->roce.nb.notifier_call = mlx5_netdev_event;
2801− err = register_netdevice_notifier(&dev->roce.nb);
2802− if (err) {
2803− dev->roce.nb.notifier_call = NULL;
2891+ err = mlx5_add_netdev_notifier(dev);
2892+ if (err)
28042893 return err;
2805− }
28062894
2807− err = mlx5_nic_vport_enable_roce(dev->mdev);
2808− if (err)
2809− goto err_unregister_netdevice_notifier;
2895+ if (MLX5_CAP_GEN(dev->mdev, roce)) {
2896+ err = mlx5_nic_vport_enable_roce(dev->mdev);
2897+ if (err)
2898+ goto err_unregister_netdevice_notifier;
2899+ }
28102900
2811− err = mlx5_roce_lag_init(dev);
2901+ err = mlx5_eth_lag_init(dev);
28122902 if (err)
28132903 goto err_disable_roce;
28142904
28152905 return 0;
28162906
28172907 err_disable_roce:
2818− mlx5_nic_vport_disable_roce(dev->mdev);
2908+ if (MLX5_CAP_GEN(dev->mdev, roce))
2909+ mlx5_nic_vport_disable_roce(dev->mdev);
28192910
28202911 err_unregister_netdevice_notifier:
2821− mlx5_remove_roce_notifier(dev);
2912+ mlx5_remove_netdev_notifier(dev);
28222913 return err;
28232914 }
28242915
2825−static void mlx5_disable_roce(struct mlx5_ib_dev *dev)
2916+static void mlx5_disable_eth(struct mlx5_ib_dev *dev)
28262917 {
2827− mlx5_roce_lag_cleanup(dev);
2828− mlx5_nic_vport_disable_roce(dev->mdev);
2918+ mlx5_eth_lag_cleanup(dev);
2919+ if (MLX5_CAP_GEN(dev->mdev, roce))
2920+ mlx5_nic_vport_disable_roce(dev->mdev);
28292921 }
28302922
28312923 static void mlx5_ib_dealloc_q_counters(struct mlx5_ib_dev *dev)
@@ -2947,9 +3039,6 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev) · 115 unmodified lines
29473039 port_type_cap = MLX5_CAP_GEN(mdev, port_type);
29483040 ll = mlx5_port_type_cap_to_rdma_ll(port_type_cap);
29493041
2950− if ((ll == IB_LINK_LAYER_ETHERNET) && !MLX5_CAP_GEN(mdev, roce))
2951− return NULL;
2952−
29533042 printk_once(KERN_INFO "%s", mlx5_version);
29543043
29553044 dev = (struct mlx5_ib_dev *)ib_alloc_device(sizeof(*dev));
@@ -2995,6 +3084,8 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev) · 39 unmodified lines
29953084 (1ull << IB_USER_VERBS_CMD_QUERY_PORT) |
29963085 (1ull << IB_USER_VERBS_CMD_ALLOC_PD) |
29973086 (1ull << IB_USER_VERBS_CMD_DEALLOC_PD) |
3087+ (1ull << IB_USER_VERBS_CMD_CREATE_AH) |
3088+ (1ull << IB_USER_VERBS_CMD_DESTROY_AH) |
29983089 (1ull << IB_USER_VERBS_CMD_REG_MR) |
29993090 (1ull << IB_USER_VERBS_CMD_REREG_MR) |
30003091 (1ull << IB_USER_VERBS_CMD_DEREG_MR) |
@@ -3017,7 +3108,8 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev) · 16 unmodified lines
30173108 dev->ib_dev.uverbs_ex_cmd_mask =
30183109 (1ull << IB_USER_VERBS_EX_CMD_QUERY_DEVICE) |
30193110 (1ull << IB_USER_VERBS_EX_CMD_CREATE_CQ) |
3020− (1ull << IB_USER_VERBS_EX_CMD_CREATE_QP);
3111+ (1ull << IB_USER_VERBS_EX_CMD_CREATE_QP) |
3112+ (1ull << IB_USER_VERBS_EX_CMD_MODIFY_QP);
30213113
30223114 dev->ib_dev.query_device = mlx5_ib_query_device;
30233115 dev->ib_dev.query_port = mlx5_ib_query_port;
@@ -3128,14 +3220,14 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev) · 104 unmodified lines
31283220 spin_lock_init(&dev->reset_flow_resource_lock);
31293221
31303222 if (ll == IB_LINK_LAYER_ETHERNET) {
3131− err = mlx5_enable_roce(dev);
3223+ err = mlx5_enable_eth(dev);
31323224 if (err)
31333225 goto err_free_port;
31343226 }
31353227
31363228 err = create_dev_resources(&dev->devr);
31373229 if (err)
3138− goto err_disable_roce;
3230+ goto err_disable_eth;
31393231
31403232 err = mlx5_ib_odp_init_one(dev);
31413233 if (err)
@@ -3179,10 +3271,10 @@ err_odp: · 37 unmodified lines
31793271 err_rsrc:
31803272 destroy_dev_resources(&dev->devr);
31813273
3182−err_disable_roce:
3274+err_disable_eth:
31833275 if (ll == IB_LINK_LAYER_ETHERNET) {
3184− mlx5_disable_roce(dev);
3185− mlx5_remove_roce_notifier(dev);
3276+ mlx5_disable_eth(dev);
3277+ mlx5_remove_netdev_notifier(dev);
31863278 }
31873279
31883280 err_free_port:
@@ -3199,14 +3291,14 @@ static void mlx5_ib_remove(struct mlx5_core_dev *mdev, void *context) · 10 unmodified lines
31993291 struct mlx5_ib_dev *dev = context;
32003292 enum rdma_link_layer ll = mlx5_ib_port_link_layer(&dev->ib_dev, 1);
32013293
3202− mlx5_remove_roce_notifier(dev);
3294+ mlx5_remove_netdev_notifier(dev);
32033295 ib_unregister_device(&dev->ib_dev);
32043296 mlx5_ib_dealloc_q_counters(dev);
32053297 destroy_umrc_res(dev);
32063298 mlx5_ib_odp_remove_one(dev);
32073299 destroy_dev_resources(&dev->devr);
32083300 if (ll == IB_LINK_LAYER_ETHERNET)
3209− mlx5_disable_roce(dev);
3301+ mlx5_disable_eth(dev);
32103302 kfree(dev->port);
32113303 ib_dealloc_device(&dev->ib_dev);
32123304 }
drivers/infiniband/hw/mlx5/mem.c+6−1
@@ -37,12 +37,15 @@ · 36 unmodified lines
3737
3838 /* @umem: umem object to scan
3939 * @addr: ib virtual address requested by the user
40+ * @max_page_shift: high limit for page_shift - 0 means no limit
4041 * @count: number of PAGE_SIZE pages covered by umem
4142 * @shift: page shift for the compound pages found in the region
4243 * @ncont: number of compund pages
4344 * @order: log2 of the number of compound pages
4445 */
45−void mlx5_ib_cont_pages(struct ib_umem *umem, u64 addr, int *count, int *shift,
46+void mlx5_ib_cont_pages(struct ib_umem *umem, u64 addr,
47+ unsigned long max_page_shift,
48+ int *count, int *shift,
4649 int *ncont, int *order)
4750 {
4851 unsigned long tmp;
@@ -72,6 +75,8 @@ void mlx5_ib_cont_pages(struct ib_umem *umem, u64 addr, int *count, int *shift, · 23 unmodified lines
7275 addr = addr >> page_shift;
7376 tmp = (unsigned long)addr;
7477 m = find_first_bit(&tmp, BITS_PER_LONG);
78+ if (max_page_shift)
79+ m = min_t(unsigned long, max_page_shift - page_shift, m);
7580 skip = 1 << m;
7681 mask = skip - 1;
7782 i = 0;
drivers/infiniband/hw/mlx5/mlx5_ib.h+9−3
@@ -63,6 +63,8 @@ pr_warn("%s:%s:%d:(pid %d): " format, (dev)->ib_dev.name, __func__, \ · 62 unmodified lines
6363 #define MLX5_IB_DEFAULT_UIDX 0xffffff
6464 #define MLX5_USER_ASSIGNED_UIDX_MASK __mlx5_mask(qpc, user_index)
6565
66+#define MLX5_MKEY_PAGE_SHIFT_MASK __mlx5_mask(mkc, log_page_size)
67+
6668 enum {
6769 MLX5_IB_MMAP_CMD_SHIFT = 8,
6870 MLX5_IB_MMAP_CMD_MASK = 0xff,
@@ -387,6 +389,7 @@ struct mlx5_ib_qp { · 318 unmodified lines
387389 struct list_head qps_list;
388390 struct list_head cq_recv_list;
389391 struct list_head cq_send_list;
392+ u32 rate_limit;
390393 };
391394
392395 struct mlx5_ib_cq_buf {
@@ -418,7 +421,7 @@ struct mlx5_umr_wr { · 25 unmodified lines
418421 struct ib_pd *pd;
419422 unsigned int page_shift;
420423 unsigned int npages;
421− u32 length;
424+ u64 length;
422425 int access_flags;
423426 u32 mkey;
424427 };
@@ -739,7 +742,8 @@ void mlx5_ib_free_srq_wqe(struct mlx5_ib_srq *srq, int wqe_index); · 314 unmodified lines
739742 int mlx5_MAD_IFC(struct mlx5_ib_dev *dev, int ignore_mkey, int ignore_bkey,
740743 u8 port, const struct ib_wc *in_wc, const struct ib_grh *in_grh,
741744 const void *in_mad, void *response_mad);
742−struct ib_ah *mlx5_ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr);
745+struct ib_ah *mlx5_ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr,
746+ struct ib_udata *udata);
743747 int mlx5_ib_query_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr);
744748 int mlx5_ib_destroy_ah(struct ib_ah *ah);
745749 struct ib_srq *mlx5_ib_create_srq(struct ib_pd *pd,
@@ -825,7 +829,9 @@ int mlx5_ib_query_port(struct ib_device *ibdev, u8 port, · 79 unmodified lines
825829 struct ib_port_attr *props);
826830 int mlx5_ib_init_fmr(struct mlx5_ib_dev *dev);
827831 void mlx5_ib_cleanup_fmr(struct mlx5_ib_dev *dev);
828−void mlx5_ib_cont_pages(struct ib_umem *umem, u64 addr, int *count, int *shift,
832+void mlx5_ib_cont_pages(struct ib_umem *umem, u64 addr,
833+ unsigned long max_page_shift,
834+ int *count, int *shift,
829835 int *ncont, int *order);
830836 void __mlx5_ib_populate_pas(struct mlx5_ib_dev *dev, struct ib_umem *umem,
831837 int page_shift, size_t offset, size_t num_pages,
drivers/infiniband/hw/mlx5/mr.c+52−19
@@ -628,7 +628,8 @@ int mlx5_mr_cache_init(struct mlx5_ib_dev *dev) · 627 unmodified lines
628628 ent->order = i + 2;
629629 ent->dev = dev;
630630
631− if (dev->mdev->profile->mask & MLX5_PROF_MASK_MR_CACHE)
631+ if ((dev->mdev->profile->mask & MLX5_PROF_MASK_MR_CACHE) &&
632+ (mlx5_core_is_pf(dev->mdev)))
632633 limit = dev->mdev->profile->mr_cache[i].limit;
633634 else
634635 limit = 0;
@@ -646,6 +647,33 @@ int mlx5_mr_cache_init(struct mlx5_ib_dev *dev) · 11 unmodified lines
646647 return 0;
647648 }
648649
650+static void wait_for_async_commands(struct mlx5_ib_dev *dev)
651+{
652+ struct mlx5_mr_cache *cache = &dev->cache;
653+ struct mlx5_cache_ent *ent;
654+ int total = 0;
655+ int i;
656+ int j;
657+
658+ for (i = 0; i < MAX_MR_CACHE_ENTRIES; i++) {
659+ ent = &cache->ent[i];
660+ for (j = 0 ; j < 1000; j++) {
661+ if (!ent->pending)
662+ break;
663+ msleep(50);
664+ }
665+ }
666+ for (i = 0; i < MAX_MR_CACHE_ENTRIES; i++) {
667+ ent = &cache->ent[i];
668+ total += ent->pending;
669+ }
670+
671+ if (total)
672+ mlx5_ib_warn(dev, "aborted while there are %d pending mr requests\n", total);
673+ else
674+ mlx5_ib_warn(dev, "done with all pending requests\n");
675+}
676+
649677 int mlx5_mr_cache_cleanup(struct mlx5_ib_dev *dev)
650678 {
651679 int i;
@@ -659,6 +687,7 @@ int mlx5_mr_cache_cleanup(struct mlx5_ib_dev *dev) · 7 unmodified lines
659687 clean_keys(dev, i);
660688
661689 destroy_workqueue(dev->cache.wq);
690+ wait_for_async_commands(dev);
662691 del_timer_sync(&dev->delay_timer);
663692
664693 return 0;
@@ -816,29 +845,34 @@ static void prep_umr_unreg_wqe(struct mlx5_ib_dev *dev, · 151 unmodified lines
816845 umrwr->mkey = key;
817846 }
818847
819−static struct ib_umem *mr_umem_get(struct ib_pd *pd, u64 start, u64 length,
820− int access_flags, int *npages,
821− int *page_shift, int *ncont, int *order)
848+static int mr_umem_get(struct ib_pd *pd, u64 start, u64 length,
849+ int access_flags, struct ib_umem **umem,
850+ int *npages, int *page_shift, int *ncont,
851+ int *order)
822852 {
823853 struct mlx5_ib_dev *dev = to_mdev(pd->device);
824− struct ib_umem *umem = ib_umem_get(pd->uobject->context, start, length,
825− access_flags, 0);
826− if (IS_ERR(umem)) {
854+ int err;
855+
856+ *umem = ib_umem_get(pd->uobject->context, start, length,
857+ access_flags, 0);
858+ err = PTR_ERR_OR_ZERO(*umem);
859+ if (err < 0) {
827860 mlx5_ib_err(dev, "umem get failed (%ld)\n", PTR_ERR(umem));
828− return (void *)umem;
861+ return err;
829862 }
830863
831− mlx5_ib_cont_pages(umem, start, npages, page_shift, ncont, order);
864+ mlx5_ib_cont_pages(*umem, start, MLX5_MKEY_PAGE_SHIFT_MASK, npages,
865+ page_shift, ncont, order);
832866 if (!*npages) {
833867 mlx5_ib_warn(dev, "avoid zero region\n");
834− ib_umem_release(umem);
835− return ERR_PTR(-EINVAL);
868+ ib_umem_release(*umem);
869+ return -EINVAL;
836870 }
837871
838872 mlx5_ib_dbg(dev, "npages %d, ncont %d, order %d, page_shift %d\n",
839873 *npages, *ncont, *order, *page_shift);
840874
841− return umem;
875+ return 0;
842876 }
843877
844878 static void mlx5_ib_umr_done(struct ib_cq *cq, struct ib_wc *wc)
@@ -1164,11 +1198,11 @@ struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, · 319 unmodified lines
11641198
11651199 mlx5_ib_dbg(dev, "start 0x%llx, virt_addr 0x%llx, length 0x%llx, access_flags 0x%x\n",
11661200 start, virt_addr, length, access_flags);
1167− umem = mr_umem_get(pd, start, length, access_flags, &npages,
1201+ err = mr_umem_get(pd, start, length, access_flags, &umem, &npages,
11681202 &page_shift, &ncont, &order);
11691203
1170− if (IS_ERR(umem))
1171− return (void *)umem;
1204+ if (err < 0)
1205+ return ERR_PTR(err);
11721206
11731207 if (use_umr(order)) {
11741208 mr = reg_umr(pd, umem, virt_addr, length, ncont, page_shift,
@@ -1345,10 +1379,9 @@ int mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start, · 170 unmodified lines
13451379 */
13461380 flags |= IB_MR_REREG_TRANS;
13471381 ib_umem_release(mr->umem);
1348− mr->umem = mr_umem_get(pd, addr, len, access_flags, &npages,
1349− &page_shift, &ncont, &order);
1350− if (IS_ERR(mr->umem)) {
1351− err = PTR_ERR(mr->umem);
1382+ err = mr_umem_get(pd, addr, len, access_flags, &mr->umem,
1383+ &npages, &page_shift, &ncont, &order);
1384+ if (err < 0) {
13521385 mr->umem = NULL;
13531386 return err;
13541387 }
drivers/infiniband/hw/mlx5/qp.c+106−25
@@ -77,12 +77,14 @@ struct mlx5_wqe_eth_pad { · 76 unmodified lines
7777
7878 enum raw_qp_set_mask_map {
7979 MLX5_RAW_QP_MOD_SET_RQ_Q_CTR_ID = 1UL << 0,
80+ MLX5_RAW_QP_RATE_LIMIT = 1UL << 1,
8081 };
8182
8283 struct mlx5_modify_raw_qp_param {
8384 u16 operation;
8485
8586 u32 set_mask; /* raw_qp_set_mask_map */
87+ u32 rate_limit;
8688 u8 rq_q_ctr_id;
8789 };
8890
@@ -351,6 +353,29 @@ static int calc_send_wqe(struct ib_qp_init_attr *attr) · 262 unmodified lines
351353 return ALIGN(max_t(int, inl_size, size), MLX5_SEND_WQE_BB);
352354 }
353355
356+static int get_send_sge(struct ib_qp_init_attr *attr, int wqe_size)
357+{
358+ int max_sge;
359+
360+ if (attr->qp_type == IB_QPT_RC)
361+ max_sge = (min_t(int, wqe_size, 512) -
362+ sizeof(struct mlx5_wqe_ctrl_seg) -
363+ sizeof(struct mlx5_wqe_raddr_seg)) /
364+ sizeof(struct mlx5_wqe_data_seg);
365+ else if (attr->qp_type == IB_QPT_XRC_INI)
366+ max_sge = (min_t(int, wqe_size, 512) -
367+ sizeof(struct mlx5_wqe_ctrl_seg) -
368+ sizeof(struct mlx5_wqe_xrc_seg) -
369+ sizeof(struct mlx5_wqe_raddr_seg)) /
370+ sizeof(struct mlx5_wqe_data_seg);
371+ else
372+ max_sge = (wqe_size - sq_overhead(attr)) /
373+ sizeof(struct mlx5_wqe_data_seg);
374+
375+ return min_t(int, max_sge, wqe_size - sq_overhead(attr) /
376+ sizeof(struct mlx5_wqe_data_seg));
377+}
378+
354379 static int calc_sq_size(struct mlx5_ib_dev *dev, struct ib_qp_init_attr *attr,
355380 struct mlx5_ib_qp *qp)
356381 {
@@ -381,13 +406,18 @@ static int calc_sq_size(struct mlx5_ib_dev *dev, struct ib_qp_init_attr *attr, · 24 unmodified lines
381406 wq_size = roundup_pow_of_two(attr->cap.max_send_wr * wqe_size);
382407 qp->sq.wqe_cnt = wq_size / MLX5_SEND_WQE_BB;
383408 if (qp->sq.wqe_cnt > (1 << MLX5_CAP_GEN(dev->mdev, log_max_qp_sz))) {
384− mlx5_ib_dbg(dev, "wqe count(%d) exceeds limits(%d)\n",
409+ mlx5_ib_dbg(dev, "send queue size (%d * %d / %d -> %d) exceeds limits(%d)\n",
410+ attr->cap.max_send_wr, wqe_size, MLX5_SEND_WQE_BB,
385411 qp->sq.wqe_cnt,
386412 1 << MLX5_CAP_GEN(dev->mdev, log_max_qp_sz));
387413 return -ENOMEM;
388414 }
389415 qp->sq.wqe_shift = ilog2(MLX5_SEND_WQE_BB);
390− qp->sq.max_gs = attr->cap.max_send_sge;
416+ qp->sq.max_gs = get_send_sge(attr, wqe_size);
417+ if (qp->sq.max_gs < attr->cap.max_send_sge)
418+ return -ENOMEM;
419+
420+ attr->cap.max_send_sge = qp->sq.max_gs;
391421 qp->sq.max_post = wq_size / wqe_size;
392422 attr->cap.max_send_wr = qp->sq.max_post;
393423
@@ -647,7 +677,7 @@ static int mlx5_ib_umem_get(struct mlx5_ib_dev *dev, · 253 unmodified lines
647677 return PTR_ERR(*umem);
648678 }
649679
650− mlx5_ib_cont_pages(*umem, addr, npages, page_shift, ncont, NULL);
680+ mlx5_ib_cont_pages(*umem, addr, 0, npages, page_shift, ncont, NULL);
651681
652682 err = mlx5_ib_get_buf_offset(addr, *page_shift, offset);
653683 if (err) {
@@ -700,7 +730,7 @@ static int create_user_rq(struct mlx5_ib_dev *dev, struct ib_pd *pd, · 46 unmodified lines
700730 return err;
701731 }
702732
703− mlx5_ib_cont_pages(rwq->umem, ucmd->buf_addr, &npages, &page_shift,
733+ mlx5_ib_cont_pages(rwq->umem, ucmd->buf_addr, 0, &npages, &page_shift,
704734 &ncont, NULL);
705735 err = mlx5_ib_get_buf_offset(ucmd->buf_addr, page_shift,
706736 &rwq->rq_page_offset);
@@ -2442,8 +2472,14 @@ out: · 1735 unmodified lines
24422472 }
24432473
24442474 static int modify_raw_packet_qp_sq(struct mlx5_core_dev *dev,
2445− struct mlx5_ib_sq *sq, int new_state)
2475+ struct mlx5_ib_sq *sq,
2476+ int new_state,
2477+ const struct mlx5_modify_raw_qp_param *raw_qp_param)
24462478 {
2479+ struct mlx5_ib_qp *ibqp = sq->base.container_mibqp;
2480+ u32 old_rate = ibqp->rate_limit;
2481+ u32 new_rate = old_rate;
2482+ u16 rl_index = 0;
24472483 void *in;
24482484 void *sqc;
24492485 int inlen;
@@ -2459,10 +2495,44 @@ static int modify_raw_packet_qp_sq(struct mlx5_core_dev *dev, · 9 unmodified lines
24592495 sqc = MLX5_ADDR_OF(modify_sq_in, in, ctx);
24602496 MLX5_SET(sqc, sqc, state, new_state);
24612497
2498+ if (raw_qp_param->set_mask & MLX5_RAW_QP_RATE_LIMIT) {
2499+ if (new_state != MLX5_SQC_STATE_RDY)
2500+ pr_warn("%s: Rate limit can only be changed when SQ is moving to RDY\n",
2501+ __func__);
2502+ else
2503+ new_rate = raw_qp_param->rate_limit;
2504+ }
2505+
2506+ if (old_rate != new_rate) {
2507+ if (new_rate) {
2508+ err = mlx5_rl_add_rate(dev, new_rate, &rl_index);
2509+ if (err) {
2510+ pr_err("Failed configuring rate %u: %d\n",
2511+ new_rate, err);
2512+ goto out;
2513+ }
2514+ }
2515+
2516+ MLX5_SET64(modify_sq_in, in, modify_bitmask, 1);
2517+ MLX5_SET(sqc, sqc, packet_pacing_rate_limit_index, rl_index);
2518+ }
2519+
24622520 err = mlx5_core_modify_sq(dev, sq->base.mqp.qpn, in, inlen);
2463− if (err)
2521+ if (err) {
2522+ /* Remove new rate from table if failed */
2523+ if (new_rate &&
2524+ old_rate != new_rate)
2525+ mlx5_rl_remove_rate(dev, new_rate);
24642526 goto out;
2527+ }
2528+
2529+ /* Only remove the old rate after new rate was set */
2530+ if ((old_rate &&
2531+ (old_rate != new_rate)) ||
2532+ (new_state != MLX5_SQC_STATE_RDY))
2533+ mlx5_rl_remove_rate(dev, old_rate);
24652534
2535+ ibqp->rate_limit = new_rate;
24662536 sq->state = new_state;
24672537
24682538 out:
@@ -2477,6 +2547,8 @@ static int modify_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, · 8 unmodified lines
24772547 struct mlx5_ib_raw_packet_qp *raw_packet_qp = &qp->raw_packet_qp;
24782548 struct mlx5_ib_rq *rq = &raw_packet_qp->rq;
24792549 struct mlx5_ib_sq *sq = &raw_packet_qp->sq;
2550+ int modify_rq = !!qp->rq.wqe_cnt;
2551+ int modify_sq = !!qp->sq.wqe_cnt;
24802552 int rq_state;
24812553 int sq_state;
24822554 int err;
@@ -2494,10 +2566,18 @@ static int modify_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, · 11 unmodified lines
24942566 rq_state = MLX5_RQC_STATE_RST;
24952567 sq_state = MLX5_SQC_STATE_RST;
24962568 break;
2497− case MLX5_CMD_OP_INIT2INIT_QP:
2498− case MLX5_CMD_OP_INIT2RTR_QP:
24992569 case MLX5_CMD_OP_RTR2RTS_QP:
25002570 case MLX5_CMD_OP_RTS2RTS_QP:
2571+ if (raw_qp_param->set_mask ==
2572+ MLX5_RAW_QP_RATE_LIMIT) {
2573+ modify_rq = 0;
2574+ sq_state = sq->state;
2575+ } else {
2576+ return raw_qp_param->set_mask ? -EINVAL : 0;
2577+ }
2578+ break;
2579+ case MLX5_CMD_OP_INIT2INIT_QP:
2580+ case MLX5_CMD_OP_INIT2RTR_QP:
25012581 if (raw_qp_param->set_mask)
25022582 return -EINVAL;
25032583 else
@@ -2507,13 +2587,13 @@ static int modify_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, · 3 unmodified lines
25072587 return -EINVAL;
25082588 }
25092589
2510− if (qp->rq.wqe_cnt) {
2511− err = modify_raw_packet_qp_rq(dev, rq, rq_state, raw_qp_param);
2590+ if (modify_rq) {
2591+ err = modify_raw_packet_qp_rq(dev, rq, rq_state, raw_qp_param);
25122592 if (err)
25132593 return err;
25142594 }
25152595
2516− if (qp->sq.wqe_cnt) {
2596+ if (modify_sq) {
25172597 if (tx_affinity) {
25182598 err = modify_raw_packet_tx_affinity(dev->mdev, sq,
25192599 tx_affinity);
@@ -2521,7 +2601,7 @@ static int modify_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, · 1 unmodified line
25212601 return err;
25222602 }
25232603
2524− return modify_raw_packet_qp_sq(dev->mdev, sq, sq_state);
2604+ return modify_raw_packet_qp_sq(dev->mdev, sq, sq_state, raw_qp_param);
25252605 }
25262606
25272607 return 0;
@@ -2577,7 +2657,6 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, · 49 unmodified lines
25772657 struct mlx5_ib_port *mibport = NULL;
25782658 enum mlx5_qp_state mlx5_cur, mlx5_new;
25792659 enum mlx5_qp_optpar optpar;
2580− int sqd_event;
25812660 int mlx5_st;
25822661 int err;
25832662 u16 op;
@@ -2724,12 +2803,6 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, · 140 unmodified lines
27242803 if (qp->rq.wqe_cnt && cur_state == IB_QPS_RESET && new_state == IB_QPS_INIT)
27252804 context->db_rec_addr = cpu_to_be64(qp->db.dma);
27262805
2727− if (cur_state == IB_QPS_RTS && new_state == IB_QPS_SQD &&
2728− attr_mask & IB_QP_EN_SQD_ASYNC_NOTIFY && attr->en_sqd_async_notify)
2729− sqd_event = 1;
2730− else
2731− sqd_event = 0;
2732−
27332806 if (cur_state == IB_QPS_RESET && new_state == IB_QPS_INIT) {
27342807 u8 port_num = (attr_mask & IB_QP_PORT ? attr->port_num :
27352808 qp->port) - 1;
@@ -2776,6 +2849,12 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, · 40 unmodified lines
27762849 raw_qp_param.rq_q_ctr_id = mibport->q_cnt_id;
27772850 raw_qp_param.set_mask |= MLX5_RAW_QP_MOD_SET_RQ_Q_CTR_ID;
27782851 }
2852+
2853+ if (attr_mask & IB_QP_RATE_LIMIT) {
2854+ raw_qp_param.rate_limit = attr->rate_limit;
2855+ raw_qp_param.set_mask |= MLX5_RAW_QP_RATE_LIMIT;
2856+ }
2857+
27792858 err = modify_raw_packet_qp(dev, qp, &raw_qp_param, tx_affinity);
27802859 } else {
27812860 err = mlx5_core_qp_modify(dev->mdev, op, optpar, context,
@@ -3067,10 +3146,10 @@ static void set_linv_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr) · 285 unmodified lines
30673146 {
30683147 memset(umr, 0, sizeof(*umr));
30693148 umr->mkey_mask = cpu_to_be64(MLX5_MKEY_MASK_FREE);
3070− umr->flags = 1 << 7;
3149+ umr->flags = MLX5_UMR_INLINE;
30713150 }
30723151
3073−static __be64 get_umr_reg_mr_mask(void)
3152+static __be64 get_umr_reg_mr_mask(int atomic)
30743153 {
30753154 u64 result;
30763155
@@ -3083,9 +3162,11 @@ static __be64 get_umr_reg_mr_mask(void) · 6 unmodified lines
30833162 MLX5_MKEY_MASK_KEY |
30843163 MLX5_MKEY_MASK_RR |
30853164 MLX5_MKEY_MASK_RW |
3086− MLX5_MKEY_MASK_A |
30873165 MLX5_MKEY_MASK_FREE;
30883166
3167+ if (atomic)
3168+ result |= MLX5_MKEY_MASK_A;
3169+
30893170 return cpu_to_be64(result);
30903171 }
30913172
@@ -3146,7 +3227,7 @@ static __be64 get_umr_update_pd_mask(void) · 54 unmodified lines
31463227 }
31473228
31483229 static void set_reg_umr_segment(struct mlx5_wqe_umr_ctrl_seg *umr,
3149− struct ib_send_wr *wr)
3230+ struct ib_send_wr *wr, int atomic)
31503231 {
31513232 struct mlx5_umr_wr *umrwr = umr_wr(wr);
31523233
@@ -3171,7 +3252,7 @@ static void set_reg_umr_segment(struct mlx5_wqe_umr_ctrl_seg *umr, · 18 unmodified lines
31713252 if (wr->send_flags & MLX5_IB_SEND_UMR_UPDATE_PD)
31723253 umr->mkey_mask |= get_umr_update_pd_mask();
31733254 if (!umr->mkey_mask)
3174− umr->mkey_mask = get_umr_reg_mr_mask();
3255+ umr->mkey_mask = get_umr_reg_mr_mask(atomic);
31753256 } else {
31763257 umr->mkey_mask = get_umr_unreg_mr_mask();
31773258 }
@@ -4024,7 +4105,7 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, · 846 unmodified lines
40244105 }
40254106 qp->sq.wr_data[idx] = MLX5_IB_WR_UMR;
40264107 ctrl->imm = cpu_to_be32(umr_wr(wr)->mkey);
4027− set_reg_umr_segment(seg, wr);
4108+ set_reg_umr_segment(seg, wr, !!(MLX5_CAP_GEN(mdev, atomic)));
40284109 seg += sizeof(struct mlx5_wqe_umr_ctrl_seg);
40294110 size += sizeof(struct mlx5_wqe_umr_ctrl_seg) / 16;
40304111 if (unlikely((seg == qend)))
drivers/infiniband/hw/mlx5/srq.c+2−4
@@ -118,7 +118,7 @@ static int create_srq_user(struct ib_pd *pd, struct mlx5_ib_srq *srq, · 117 unmodified lines
118118 return err;
119119 }
120120
121− mlx5_ib_cont_pages(srq->umem, ucmd.buf_addr, &npages,
121+ mlx5_ib_cont_pages(srq->umem, ucmd.buf_addr, 0, &npages,
122122 &page_shift, &ncont, NULL);
123123 err = mlx5_ib_get_buf_offset(ucmd.buf_addr, page_shift,
124124 &offset);
@@ -203,8 +203,6 @@ static int create_srq_kernel(struct mlx5_ib_dev *dev, struct mlx5_ib_srq *srq, · 78 unmodified lines
203203
204204 srq->wrid = kmalloc(srq->msrq.max * sizeof(u64), GFP_KERNEL);
205205 if (!srq->wrid) {
206− mlx5_ib_dbg(dev, "kmalloc failed %lu\n",
207− (unsigned long)(srq->msrq.max * sizeof(u64)));
208206 err = -ENOMEM;
209207 goto err_in;
210208 }
@@ -282,6 +280,7 @@ struct ib_srq *mlx5_ib_create_srq(struct ib_pd *pd, · 71 unmodified lines
282280 mlx5_ib_dbg(dev, "desc_size 0x%x, req wr 0x%x, srq size 0x%x, max_gs 0x%x, max_avail_gather 0x%x\n",
283281 desc_size, init_attr->attr.max_wr, srq->msrq.max, srq->msrq.max_gs,
284282 srq->msrq.max_avail_gather);
283+ in.type = init_attr->srq_type;
285284
286285 if (pd->uobject)
287286 err = create_srq_user(pd, srq, &in, udata, buf_size);
@@ -294,7 +293,6 @@ struct ib_srq *mlx5_ib_create_srq(struct ib_pd *pd, · 6 unmodified lines
294293 goto err_srq;
295294 }
296295
297− in.type = init_attr->srq_type;
298296 in.log_size = ilog2(srq->msrq.max);
299297 in.wqe_shift = srq->msrq.wqe_shift - 4;
300298 if (srq->wq_sig)
drivers/infiniband/hw/mthca/mthca_av.c+2−4
@@ -186,8 +186,8 @@ int mthca_create_ah(struct mthca_dev *dev, · 185 unmodified lines
186186
187187 on_hca_fail:
188188 if (ah->type == MTHCA_AH_PCI_POOL) {
189− ah->av = pci_pool_alloc(dev->av_table.pool,
190− GFP_ATOMIC, &ah->avdma);
189+ ah->av = pci_pool_zalloc(dev->av_table.pool,
190+ GFP_ATOMIC, &ah->avdma);
191191 if (!ah->av)
192192 return -ENOMEM;
193193
@@ -196,8 +196,6 @@ on_hca_fail: · 2 unmodified lines
196196
197197 ah->key = pd->ntmr.ibmr.lkey;
198198
199− memset(av, 0, MTHCA_AV_SIZE);
200−
201199 av->port_pd = cpu_to_be32(pd->pd_num | (ah_attr->port_num << 24));
202200 av->g_slid = ah_attr->src_path_bits;
203201 av->dlid = cpu_to_be16(ah_attr->dlid);
drivers/infiniband/hw/mthca/mthca_provider.c+3−1
@@ -410,7 +410,9 @@ static int mthca_dealloc_pd(struct ib_pd *pd) · 409 unmodified lines
410410 }
411411
412412 static struct ib_ah *mthca_ah_create(struct ib_pd *pd,
413− struct ib_ah_attr *ah_attr)
413+ struct ib_ah_attr *ah_attr,
414+ struct ib_udata *udata)
415+
414416 {
415417 int err;
416418 struct mthca_ah *ah;
drivers/infiniband/hw/mthca/mthca_reset.c+0−4
@@ -96,8 +96,6 @@ int mthca_reset(struct mthca_dev *mdev) · 95 unmodified lines
9696 hca_header = kmalloc(256, GFP_KERNEL);
9797 if (!hca_header) {
9898 err = -ENOMEM;
99− mthca_err(mdev, "Couldn't allocate memory to save HCA "
100− "PCI header, aborting.\n");
10199 goto put_dev;
102100 }
103101
@@ -119,8 +117,6 @@ int mthca_reset(struct mthca_dev *mdev) · 15 unmodified lines
119117 bridge_header = kmalloc(256, GFP_KERNEL);
120118 if (!bridge_header) {
121119 err = -ENOMEM;
122− mthca_err(mdev, "Couldn't allocate memory to save HCA "
123− "bridge PCI header, aborting.\n");
124120 goto free_hca;
125121 }
126122
drivers/infiniband/hw/nes/nes.c+0−1
@@ -515,7 +515,6 @@ static int nes_probe(struct pci_dev *pcidev, const struct pci_device_id *ent) · 514 unmodified lines
515515 /* Allocate hardware structure */
516516 nesdev = kzalloc(sizeof(struct nes_device), GFP_KERNEL);
517517 if (!nesdev) {
518− printk(KERN_ERR PFX "%s: Unable to alloc hardware struct\n", pci_name(pcidev));
519518 ret = -ENOMEM;
520519 goto bail2;
521520 }
drivers/infiniband/hw/nes/nes_cm.c+1−3
@@ -2282,10 +2282,8 @@ static struct nes_cm_listener *mini_cm_listen(struct nes_cm_core *cm_core, · 2281 unmodified lines
22822282 if (!listener) {
22832283 /* create a CM listen node (1/2 node to compare incoming traffic to) */
22842284 listener = kzalloc(sizeof(*listener), GFP_ATOMIC);
2285− if (!listener) {
2286− nes_debug(NES_DBG_CM, "Not creating listener memory allocation failed\n");
2285+ if (!listener)
22872286 return NULL;
2288− }
22892287
22902288 listener->loc_addr = cm_info->loc_addr;
22912289 listener->loc_port = cm_info->loc_port;
drivers/infiniband/hw/nes/nes_hw.c+2−4
@@ -351,9 +351,8 @@ struct nes_adapter *nes_init_adapter(struct nes_device *nesdev, u8 hw_rev) { · 350 unmodified lines
351351
352352 /* allocate a new adapter struct */
353353 nesadapter = kzalloc(adapter_size, GFP_KERNEL);
354− if (nesadapter == NULL) {
354+ if (!nesadapter)
355355 return NULL;
356− }
357356
358357 nes_debug(NES_DBG_INIT, "Allocating new nesadapter @ %p, size = %u (actual size = %u).\n",
359358 nesadapter, (u32)sizeof(struct nes_adapter), adapter_size);
@@ -1007,8 +1006,7 @@ int nes_init_cqp(struct nes_device *nesdev) · 647 unmodified lines
10071006 /* Allocate a twice the number of CQP requests as the SQ size */
10081007 nesdev->nes_cqp_requests = kzalloc(sizeof(struct nes_cqp_request) *
10091008 2 * NES_CQP_SQ_SIZE, GFP_KERNEL);
1010− if (nesdev->nes_cqp_requests == NULL) {
1011− nes_debug(NES_DBG_INIT, "Unable to allocate memory CQP request entries.\n");
1009+ if (!nesdev->nes_cqp_requests) {
10121010 pci_free_consistent(nesdev->pcidev, nesdev->cqp_mem_size, nesdev->cqp.sq_vbase,
10131011 nesdev->cqp.sq_pbase);
10141012 return -ENOMEM;
drivers/infiniband/hw/nes/nes_mgt.c+3−7
@@ -320,8 +320,7 @@ static int get_fpdu_info(struct nes_device *nesdev, struct nes_qp *nesqp, · 319 unmodified lines
320320
321321 /* Found one */
322322 fpdu_info = kzalloc(sizeof(*fpdu_info), GFP_ATOMIC);
323− if (fpdu_info == NULL) {
324− nes_debug(NES_DBG_PAU, "Failed to alloc a fpdu_info.\n");
323+ if (!fpdu_info) {
325324 rc = -ENOMEM;
326325 goto out;
327326 }
@@ -729,8 +728,7 @@ static int nes_change_quad_hash(struct nes_device *nesdev, · 401 unmodified lines
729728 }
730729
731730 qh_chg = kmalloc(sizeof *qh_chg, GFP_ATOMIC);
732− if (qh_chg == NULL) {
733− nes_debug(NES_DBG_PAU, "Failed to get a cqp_request.\n");
731+ if (!qh_chg) {
734732 ret = -ENOMEM;
735733 goto chg_qh_err;
736734 }
@@ -880,10 +878,8 @@ int nes_init_mgt_qp(struct nes_device *nesdev, struct net_device *netdev, struct · 143 unmodified lines
880878
881879 /* Allocate space the all mgt QPs once */
882880 mgtvnic = kzalloc(NES_MGT_QP_COUNT * sizeof(struct nes_vnic_mgt), GFP_KERNEL);
883− if (mgtvnic == NULL) {
884− nes_debug(NES_DBG_INIT, "Unable to allocate memory for mgt structure\n");
881+ if (!mgtvnic)
885882 return -ENOMEM;
886− }
887883
888884 /* Allocate fragment, RQ, and CQ; Reuse CEQ based on the PCI function */
889885 /* We are not sending from this NIC so sq is not allocated */
drivers/infiniband/hw/nes/nes_nic.c+48−36
@@ -662,10 +662,14 @@ tso_sq_no_longer_full: · 661 unmodified lines
662662 nesnic->sq_head &= nesnic->sq_size-1;
663663 }
664664 } else {
665− nesvnic->linearized_skbs++;
666665 hoffset = skb_transport_header(skb) - skb->data;
667666 nhoffset = skb_network_header(skb) - skb->data;
668− skb_linearize(skb);
667+ if (skb_linearize(skb)) {
668+ nesvnic->tx_sw_dropped++;
669+ kfree_skb(skb);
670+ return NETDEV_TX_OK;
671+ }
672+ nesvnic->linearized_skbs++;
669673 skb_set_transport_header(skb, hoffset);
670674 skb_set_network_header(skb, nhoffset);
671675 if (!nes_nic_send(skb, netdev))
@@ -1461,7 +1465,8 @@ static int nes_netdev_set_pauseparam(struct net_device *netdev, · 789 unmodified lines
14611465 /**
14621466 * nes_netdev_get_settings
14631467 */
1464−static int nes_netdev_get_settings(struct net_device *netdev, struct ethtool_cmd *et_cmd)
1468+static int nes_netdev_get_link_ksettings(struct net_device *netdev,
1469+ struct ethtool_link_ksettings *cmd)
14651470 {
14661471 struct nes_vnic *nesvnic = netdev_priv(netdev);
14671472 struct nes_device *nesdev = nesvnic->nesdev;
@@ -1470,54 +1475,59 @@ static int nes_netdev_get_settings(struct net_device *netdev, struct ethtool_cmd · 2 unmodified lines
14701475 u8 phy_type = nesadapter->phy_type[mac_index];
14711476 u8 phy_index = nesadapter->phy_index[mac_index];
14721477 u16 phy_data;
1478+ u32 supported, advertising;
14731479
1474− et_cmd->duplex = DUPLEX_FULL;
1475− et_cmd->port = PORT_MII;
1476− et_cmd->maxtxpkt = 511;
1477− et_cmd->maxrxpkt = 511;
1480+ cmd->base.duplex = DUPLEX_FULL;
1481+ cmd->base.port = PORT_MII;
14781482
14791483 if (nesadapter->OneG_Mode) {
1480− ethtool_cmd_speed_set(et_cmd, SPEED_1000);
1484+ cmd->base.speed = SPEED_1000;
14811485 if (phy_type == NES_PHY_TYPE_PUMA_1G) {
1482− et_cmd->supported = SUPPORTED_1000baseT_Full;
1483− et_cmd->advertising = ADVERTISED_1000baseT_Full;
1484− et_cmd->autoneg = AUTONEG_DISABLE;
1485− et_cmd->transceiver = XCVR_INTERNAL;
1486− et_cmd->phy_address = mac_index;
1486+ supported = SUPPORTED_1000baseT_Full;
1487+ advertising = ADVERTISED_1000baseT_Full;
1488+ cmd->base.autoneg = AUTONEG_DISABLE;
1489+ cmd->base.phy_address = mac_index;
14871490 } else {
14881491 unsigned long flags;
1489− et_cmd->supported = SUPPORTED_1000baseT_Full
1490− | SUPPORTED_Autoneg;
1491− et_cmd->advertising = ADVERTISED_1000baseT_Full
1492− | ADVERTISED_Autoneg;
1492+
1493+ supported = SUPPORTED_1000baseT_Full
1494+ | SUPPORTED_Autoneg;
1495+ advertising = ADVERTISED_1000baseT_Full
1496+ | ADVERTISED_Autoneg;
14931497 spin_lock_irqsave(&nesadapter->phy_lock, flags);
14941498 nes_read_1G_phy_reg(nesdev, 0, phy_index, &phy_data);
14951499 spin_unlock_irqrestore(&nesadapter->phy_lock, flags);
14961500 if (phy_data & 0x1000)
1497− et_cmd->autoneg = AUTONEG_ENABLE;
1501+ cmd->base.autoneg = AUTONEG_ENABLE;
14981502 else
1499− et_cmd->autoneg = AUTONEG_DISABLE;
1500− et_cmd->transceiver = XCVR_EXTERNAL;
1501− et_cmd->phy_address = phy_index;
1503+ cmd->base.autoneg = AUTONEG_DISABLE;
1504+ cmd->base.phy_address = phy_index;
15021505 }
1506+ ethtool_convert_legacy_u32_to_link_mode(
1507+ cmd->link_modes.supported, supported);
1508+ ethtool_convert_legacy_u32_to_link_mode(
1509+ cmd->link_modes.advertising, advertising);
15031510 return 0;
15041511 }
15051512 if ((phy_type == NES_PHY_TYPE_ARGUS) ||
15061513 (phy_type == NES_PHY_TYPE_SFP_D) ||
15071514 (phy_type == NES_PHY_TYPE_KR)) {
1508− et_cmd->transceiver = XCVR_EXTERNAL;
1509− et_cmd->port = PORT_FIBRE;
1510− et_cmd->supported = SUPPORTED_FIBRE;
1511− et_cmd->advertising = ADVERTISED_FIBRE;
1512− et_cmd->phy_address = phy_index;
1515+ cmd->base.port = PORT_FIBRE;
1516+ supported = SUPPORTED_FIBRE;
1517+ advertising = ADVERTISED_FIBRE;
1518+ cmd->base.phy_address = phy_index;
15131519 } else {
1514− et_cmd->transceiver = XCVR_INTERNAL;
1515− et_cmd->supported = SUPPORTED_10000baseT_Full;
1516− et_cmd->advertising = ADVERTISED_10000baseT_Full;
1517− et_cmd->phy_address = mac_index;
1520+ supported = SUPPORTED_10000baseT_Full;
1521+ advertising = ADVERTISED_10000baseT_Full;
1522+ cmd->base.phy_address = mac_index;
15181523 }
1519− ethtool_cmd_speed_set(et_cmd, SPEED_10000);
1520− et_cmd->autoneg = AUTONEG_DISABLE;
1524+ cmd->base.speed = SPEED_10000;
1525+ cmd->base.autoneg = AUTONEG_DISABLE;
1526+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
1527+ supported);
1528+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
1529+ advertising);
1530+
15211531 return 0;
15221532 }
15231533
@@ -1525,7 +1535,9 @@ static int nes_netdev_get_settings(struct net_device *netdev, struct ethtool_cmd · 1 unmodified line
15251535 /**
15261536 * nes_netdev_set_settings
15271537 */
1528−static int nes_netdev_set_settings(struct net_device *netdev, struct ethtool_cmd *et_cmd)
1538+static int
1539+nes_netdev_set_link_ksettings(struct net_device *netdev,
1540+ const struct ethtool_link_ksettings *cmd)
15291541 {
15301542 struct nes_vnic *nesvnic = netdev_priv(netdev);
15311543 struct nes_device *nesdev = nesvnic->nesdev;
@@ -1539,7 +1551,7 @@ static int nes_netdev_set_settings(struct net_device *netdev, struct ethtool_cmd · 7 unmodified lines
15391551
15401552 spin_lock_irqsave(&nesadapter->phy_lock, flags);
15411553 nes_read_1G_phy_reg(nesdev, 0, phy_index, &phy_data);
1542− if (et_cmd->autoneg) {
1554+ if (cmd->base.autoneg) {
15431555 /* Turn on Full duplex, Autoneg, and restart autonegotiation */
15441556 phy_data |= 0x1300;
15451557 } else {
@@ -1556,8 +1568,6 @@ static int nes_netdev_set_settings(struct net_device *netdev, struct ethtool_cmd · 10 unmodified lines
15561568
15571569 static const struct ethtool_ops nes_ethtool_ops = {
15581570 .get_link = ethtool_op_get_link,
1559− .get_settings = nes_netdev_get_settings,
1560− .set_settings = nes_netdev_set_settings,
15611571 .get_strings = nes_netdev_get_strings,
15621572 .get_sset_count = nes_netdev_get_sset_count,
15631573 .get_ethtool_stats = nes_netdev_get_ethtool_stats,
@@ -1566,6 +1576,8 @@ static const struct ethtool_ops nes_ethtool_ops = { · 2 unmodified lines
15661576 .set_coalesce = nes_netdev_set_coalesce,
15671577 .get_pauseparam = nes_netdev_get_pauseparam,
15681578 .set_pauseparam = nes_netdev_set_pauseparam,
1579+ .get_link_ksettings = nes_netdev_get_link_ksettings,
1580+ .set_link_ksettings = nes_netdev_set_link_ksettings,
15691581 };
15701582
15711583 static void nes_vlan_mode(struct net_device *netdev, struct nes_device *nesdev, netdev_features_t features)
drivers/infiniband/hw/nes/nes_verbs.c+2−5
@@ -771,7 +771,8 @@ static int nes_dealloc_pd(struct ib_pd *ibpd) · 770 unmodified lines
771771 /**
772772 * nes_create_ah
773773 */
774−static struct ib_ah *nes_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr)
774+static struct ib_ah *nes_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr,
775+ struct ib_udata *udata)
775776 {
776777 return ERR_PTR(-ENOSYS);
777778 }
@@ -1075,7 +1076,6 @@ static struct ib_qp *nes_create_qp(struct ib_pd *ibpd, · 297 unmodified lines
10751076 mem = kzalloc(sizeof(*nesqp)+NES_SW_CONTEXT_ALIGN-1, GFP_KERNEL);
10761077 if (!mem) {
10771078 nes_free_resource(nesadapter, nesadapter->allocated_qps, qp_num);
1078− nes_debug(NES_DBG_QP, "Unable to allocate QP\n");
10791079 return ERR_PTR(-ENOMEM);
10801080 }
10811081 u64nesqp = (unsigned long)mem;
@@ -1475,7 +1475,6 @@ static struct ib_cq *nes_create_cq(struct ib_device *ibdev, · 393 unmodified lines
14751475 nescq = kzalloc(sizeof(struct nes_cq), GFP_KERNEL);
14761476 if (!nescq) {
14771477 nes_free_resource(nesadapter, nesadapter->allocated_cqs, cq_num);
1478− nes_debug(NES_DBG_CQ, "Unable to allocate nes_cq struct\n");
14791478 return ERR_PTR(-ENOMEM);
14801479 }
14811480
@@ -2408,7 +2407,6 @@ static struct ib_mr *nes_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, · 926 unmodified lines
24082407 }
24092408 nespbl = kzalloc(sizeof(*nespbl), GFP_KERNEL);
24102409 if (!nespbl) {
2411− nes_debug(NES_DBG_MR, "Unable to allocate PBL\n");
24122410 ib_umem_release(region);
24132411 return ERR_PTR(-ENOMEM);
24142412 }
@@ -2416,7 +2414,6 @@ static struct ib_mr *nes_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, · 1 unmodified line
24162414 if (!nesmr) {
24172415 ib_umem_release(region);
24182416 kfree(nespbl);
2419− nes_debug(NES_DBG_MR, "Unable to allocate nesmr\n");
24202417 return ERR_PTR(-ENOMEM);
24212418 }
24222419 nesmr->region = region;
drivers/infiniband/hw/ocrdma/ocrdma_ah.c+2−1
@@ -154,7 +154,8 @@ static inline int set_av_attr(struct ocrdma_dev *dev, struct ocrdma_ah *ah, · 153 unmodified lines
154154 return status;
155155 }
156156
157−struct ib_ah *ocrdma_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *attr)
157+struct ib_ah *ocrdma_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *attr,
158+ struct ib_udata *udata)
158159 {
159160 u32 *ahid_addr;
160161 int status;
drivers/infiniband/hw/ocrdma/ocrdma_ah.h+3−1
@@ -50,7 +50,9 @@ enum { · 49 unmodified lines
5050 OCRDMA_AH_L3_TYPE_MASK = 0x03,
5151 OCRDMA_AH_L3_TYPE_SHIFT = 0x1D /* 29 bits */
5252 };
53−struct ib_ah *ocrdma_create_ah(struct ib_pd *, struct ib_ah_attr *);
53+
54+struct ib_ah *ocrdma_create_ah(struct ib_pd *, struct ib_ah_attr *,
55+ struct ib_udata *);
5456 int ocrdma_destroy_ah(struct ib_ah *);
5557 int ocrdma_query_ah(struct ib_ah *, struct ib_ah_attr *);
5658 int ocrdma_modify_ah(struct ib_ah *, struct ib_ah_attr *);
drivers/infiniband/hw/ocrdma/ocrdma_hw.c+3−4
@@ -1596,10 +1596,9 @@ void ocrdma_alloc_pd_pool(struct ocrdma_dev *dev) · 1595 unmodified lines
15961596
15971597 dev->pd_mgr = kzalloc(sizeof(struct ocrdma_pd_resource_mgr),
15981598 GFP_KERNEL);
1599− if (!dev->pd_mgr) {
1600− pr_err("%s(%d)Memory allocation failure.\n", __func__, dev->id);
1599+ if (!dev->pd_mgr)
16011600 return;
1602− }
1601+
16031602 status = ocrdma_mbx_alloc_pd_range(dev);
16041603 if (status) {
16051604 pr_err("%s(%d) Unable to initialize PD pool, using default.\n",
@@ -1642,7 +1641,7 @@ static int ocrdma_build_q_conf(u32 *num_entries, int entry_size, · 36 unmodified lines
16421641 static int ocrdma_mbx_create_ah_tbl(struct ocrdma_dev *dev)
16431642 {
16441643 int i;
1645− int status = 0;
1644+ int status = -ENOMEM;
16461645 int max_ah;
16471646 struct ocrdma_create_ah_tbl *cmd;
16481647 struct ocrdma_create_ah_tbl_rsp *rsp;
drivers/infiniband/hw/ocrdma/ocrdma_stats.c+1−3
@@ -84,10 +84,8 @@ bool ocrdma_alloc_stats_resources(struct ocrdma_dev *dev) · 83 unmodified lines
8484
8585 /* Alloc debugfs mem */
8686 mem->debugfs_mem = kzalloc(OCRDMA_MAX_DBGFS_MEM, GFP_KERNEL);
87− if (!mem->debugfs_mem) {
88− pr_err("%s: stats debugfs mem allocation failed\n", __func__);
87+ if (!mem->debugfs_mem)
8988 return false;
90− }
9189
9290 return true;
9391 }
drivers/infiniband/hw/qedr/verbs.c+14−13
@@ -511,8 +511,10 @@ int qedr_dealloc_pd(struct ib_pd *ibpd) · 510 unmodified lines
511511 struct qedr_dev *dev = get_qedr_dev(ibpd->device);
512512 struct qedr_pd *pd = get_qedr_pd(ibpd);
513513
514− if (!pd)
514+ if (!pd) {
515515 pr_err("Invalid PD received in dealloc_pd\n");
516+ return -EINVAL;
517+ }
516518
517519 DP_DEBUG(dev, QEDR_MSG_INIT, "Deallocating PD %d\n", pd->pd_id);
518520 dev->ops->rdma_dealloc_pd(dev->rdma_ctx, pd->pd_id);
@@ -1477,6 +1479,7 @@ struct ib_qp *qedr_create_qp(struct ib_pd *ibpd, · 958 unmodified lines
14771479 struct qedr_ucontext *ctx = NULL;
14781480 struct qedr_create_qp_ureq ureq;
14791481 struct qedr_qp *qp;
1482+ struct ib_qp *ibqp;
14801483 int rc = 0;
14811484
14821485 DP_DEBUG(dev, QEDR_MSG_QP, "create qp: called from %s, pd=%p\n",
@@ -1486,13 +1489,13 @@ struct ib_qp *qedr_create_qp(struct ib_pd *ibpd, · 3 unmodified lines
14861489 if (rc)
14871490 return ERR_PTR(rc);
14881491
1492+ if (attrs->srq)
1493+ return ERR_PTR(-EINVAL);
1494+
14891495 qp = kzalloc(sizeof(*qp), GFP_KERNEL);
14901496 if (!qp)
14911497 return ERR_PTR(-ENOMEM);
14921498
1493− if (attrs->srq)
1494− return ERR_PTR(-EINVAL);
1495−
14961499 DP_DEBUG(dev, QEDR_MSG_QP,
14971500 "create qp: sq_cq=%p, sq_icid=%d, rq_cq=%p, rq_icid=%d\n",
14981501 get_qedr_cq(attrs->send_cq),
@@ -1508,7 +1511,10 @@ struct ib_qp *qedr_create_qp(struct ib_pd *ibpd, · 9 unmodified lines
15081511 "create qp: unexpected udata when creating GSI QP\n");
15091512 goto err0;
15101513 }
1511− return qedr_create_gsi_qp(dev, attrs, qp);
1514+ ibqp = qedr_create_gsi_qp(dev, attrs, qp);
1515+ if (IS_ERR(ibqp))
1516+ kfree(qp);
1517+ return ibqp;
15121518 }
15131519
15141520 memset(&in_params, 0, sizeof(in_params));
@@ -2094,7 +2100,8 @@ int qedr_destroy_qp(struct ib_qp *ibqp) · 579 unmodified lines
20942100 return rc;
20952101 }
20962102
2097−struct ib_ah *qedr_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *attr)
2103+struct ib_ah *qedr_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *attr,
2104+ struct ib_udata *udata)
20982105 {
20992106 struct qedr_ah *ah;
21002107
@@ -2413,8 +2420,7 @@ static void handle_completed_mrs(struct qedr_dev *dev, struct mr_info *info) · 312 unmodified lines
24132420 */
24142421 pbl = list_first_entry(&info->inuse_pbl_list,
24152422 struct qedr_pbl, list_entry);
2416− list_del(&pbl->list_entry);
2417− list_add_tail(&pbl->list_entry, &info->free_pbl_list);
2423+ list_move_tail(&pbl->list_entry, &info->free_pbl_list);
24182424 info->completed_handled++;
24192425 }
24202426 }
@@ -2981,11 +2987,6 @@ int qedr_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, · 560 unmodified lines
29812987 return -EINVAL;
29822988 }
29832989
2984− if (!wr) {
2985− DP_ERR(dev, "Got an empty post send.\n");
2986− return -EINVAL;
2987− }
2988−
29892990 while (wr) {
29902991 rc = __qedr_post_send(ibqp, wr, bad_wr);
29912992 if (rc)
drivers/infiniband/hw/qedr/verbs.h+2−1
@@ -70,7 +70,8 @@ int qedr_query_qp(struct ib_qp *, struct ib_qp_attr *qp_attr, · 69 unmodified lines
7070 int qp_attr_mask, struct ib_qp_init_attr *);
7171 int qedr_destroy_qp(struct ib_qp *ibqp);
7272
73−struct ib_ah *qedr_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *attr);
73+struct ib_ah *qedr_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *attr,
74+ struct ib_udata *udata);
7475 int qedr_destroy_ah(struct ib_ah *ibah);
7576
7677 int qedr_dereg_mr(struct ib_mr *);
drivers/infiniband/hw/qib/qib_diag.c+1−5
@@ -609,8 +609,6 @@ static ssize_t qib_diagpkt_write(struct file *fp, · 608 unmodified lines
609609
610610 tmpbuf = vmalloc(plen);
611611 if (!tmpbuf) {
612− qib_devinfo(dd->pcidev,
613− "Unable to allocate tmp buffer, failing\n");
614612 ret = -ENOMEM;
615613 goto bail;
616614 }
@@ -702,10 +700,8 @@ int qib_register_observer(struct qib_devdata *dd, · 85 unmodified lines
702700 if (!dd || !op)
703701 return -EINVAL;
704702 olp = vmalloc(sizeof(*olp));
705− if (!olp) {
706− pr_err("vmalloc for observer failed\n");
703+ if (!olp)
707704 return -ENOMEM;
708− }
709705
710706 spin_lock_irqsave(&dd->qib_diag_trans_lock, flags);
711707 olp->op = op;
drivers/infiniband/hw/qib/qib_driver.c+1−2
@@ -420,8 +420,7 @@ static u32 qib_rcv_hdrerr(struct qib_ctxtdata *rcd, struct qib_pportdata *ppd, · 419 unmodified lines
420420 if (list_empty(&qp->rspwait)) {
421421 qp->r_flags |=
422422 RVT_R_RSP_NAK;
423− atomic_inc(
424− &qp->refcount);
423+ rvt_get_qp(qp);
425424 list_add_tail(
426425 &qp->rspwait,
427426 &rcd->qp_wait_list);
drivers/infiniband/hw/qib/qib_eeprom.c+1−5
@@ -182,12 +182,8 @@ void qib_get_eeprom_info(struct qib_devdata *dd) · 181 unmodified lines
182182 * */
183183 len = sizeof(struct qib_flash);
184184 buf = vmalloc(len);
185− if (!buf) {
186− qib_dev_err(dd,
187− "Couldn't allocate memory to read %u bytes from eeprom for GUID\n",
188− len);
185+ if (!buf)
189186 goto bail;
190− }
191187
192188 /*
193189 * Use "public" eeprom read function, which does locking and
drivers/infiniband/hw/qib/qib_file_ops.c+4−1
@@ -2066,8 +2066,11 @@ static ssize_t qib_write(struct file *fp, const char __user *data, · 2065 unmodified lines
20662066 ssize_t ret = 0;
20672067 void *dest;
20682068
2069− if (WARN_ON_ONCE(!ib_safe_file_access(fp)))
2069+ if (!ib_safe_file_access(fp)) {
2070+ pr_err_once("qib_write: process %d (%s) changed security contexts after opening file descriptor, this is not allowed.\n",
2071+ task_tgid_vnr(current), current->comm);
20702072 return -EACCES;
2073+ }
20712074
20722075 if (count < sizeof(cmd.type)) {
20732076 ret = -EINVAL;
drivers/infiniband/hw/qib/qib_iba6120.c+1−7
@@ -1759,9 +1759,7 @@ static void pe_boardname(struct qib_devdata *dd) · 1758 unmodified lines
17591759 }
17601760 namelen = strlen(n) + 1;
17611761 dd->boardname = kmalloc(namelen, GFP_KERNEL);
1762− if (!dd->boardname)
1763− qib_dev_err(dd, "Failed allocation for board name: %s\n", n);
1764− else
1762+ if (dd->boardname)
17651763 snprintf(dd->boardname, namelen, "%s", n);
17661764
17671765 if (dd->majrev != 4 || !dd->minrev || dd->minrev > 2)
@@ -2533,8 +2531,6 @@ static void init_6120_cntrnames(struct qib_devdata *dd) · 765 unmodified lines
25332531 dd->cspec->cntrnamelen = 1 + s - cntr6120names;
25342532 dd->cspec->cntrs = kmalloc(dd->cspec->ncntrs
25352533 * sizeof(u64), GFP_KERNEL);
2536− if (!dd->cspec->cntrs)
2537− qib_dev_err(dd, "Failed allocation for counters\n");
25382534
25392535 for (i = 0, s = (char *)portcntr6120names; s; i++)
25402536 s = strchr(s + 1, '\n');
@@ -2542,8 +2538,6 @@ static void init_6120_cntrnames(struct qib_devdata *dd) · 1 unmodified line
25422538 dd->cspec->portcntrnamelen = sizeof(portcntr6120names) - 1;
25432539 dd->cspec->portcntrs = kmalloc(dd->cspec->nportcntrs
25442540 * sizeof(u64), GFP_KERNEL);
2545− if (!dd->cspec->portcntrs)
2546− qib_dev_err(dd, "Failed allocation for portcounters\n");
25472541 }
25482542
25492543 static u32 qib_read_6120cntrs(struct qib_devdata *dd, loff_t pos, char **namep,
drivers/infiniband/hw/qib/qib_iba7220.c+1−7
@@ -2070,9 +2070,7 @@ static void qib_7220_boardname(struct qib_devdata *dd) · 2069 unmodified lines
20702070
20712071 namelen = strlen(n) + 1;
20722072 dd->boardname = kmalloc(namelen, GFP_KERNEL);
2073− if (!dd->boardname)
2074− qib_dev_err(dd, "Failed allocation for board name: %s\n", n);
2075− else
2073+ if (dd->boardname)
20762074 snprintf(dd->boardname, namelen, "%s", n);
20772075
20782076 if (dd->majrev != 5 || !dd->minrev || dd->minrev > 2)
@@ -3179,8 +3177,6 @@ static void init_7220_cntrnames(struct qib_devdata *dd) · 1100 unmodified lines
31793177 dd->cspec->cntrnamelen = 1 + s - cntr7220names;
31803178 dd->cspec->cntrs = kmalloc(dd->cspec->ncntrs
31813179 * sizeof(u64), GFP_KERNEL);
3182− if (!dd->cspec->cntrs)
3183− qib_dev_err(dd, "Failed allocation for counters\n");
31843180
31853181 for (i = 0, s = (char *)portcntr7220names; s; i++)
31863182 s = strchr(s + 1, '\n');
@@ -3188,8 +3184,6 @@ static void init_7220_cntrnames(struct qib_devdata *dd) · 1 unmodified line
31883184 dd->cspec->portcntrnamelen = sizeof(portcntr7220names) - 1;
31893185 dd->cspec->portcntrs = kmalloc(dd->cspec->nportcntrs
31903186 * sizeof(u64), GFP_KERNEL);
3191− if (!dd->cspec->portcntrs)
3192− qib_dev_err(dd, "Failed allocation for portcounters\n");
31933187 }
31943188
31953189 static u32 qib_read_7220cntrs(struct qib_devdata *dd, loff_t pos, char **namep,
drivers/infiniband/hw/qib/qib_iba7322.c+5−17
@@ -3627,9 +3627,7 @@ static unsigned qib_7322_boardname(struct qib_devdata *dd) · 3626 unmodified lines
36273627
36283628 namelen = strlen(n) + 1;
36293629 dd->boardname = kmalloc(namelen, GFP_KERNEL);
3630− if (!dd->boardname)
3631− qib_dev_err(dd, "Failed allocation for board name: %s\n", n);
3632− else
3630+ if (dd->boardname)
36333631 snprintf(dd->boardname, namelen, "%s", n);
36343632
36353633 snprintf(dd->boardversion, sizeof(dd->boardversion),
@@ -3656,7 +3654,7 @@ static unsigned qib_7322_boardname(struct qib_devdata *dd) · 20 unmodified lines
36563654 static int qib_do_7322_reset(struct qib_devdata *dd)
36573655 {
36583656 u64 val;
3659− u64 *msix_vecsave;
3657+ u64 *msix_vecsave = NULL;
36603658 int i, msix_entries, ret = 1;
36613659 u16 cmdval;
36623660 u8 int_line, clinesz;
@@ -3677,10 +3675,7 @@ static int qib_do_7322_reset(struct qib_devdata *dd) · 14 unmodified lines
36773675 /* can be up to 512 bytes, too big for stack */
36783676 msix_vecsave = kmalloc(2 * dd->cspec->num_msix_entries *
36793677 sizeof(u64), GFP_KERNEL);
3680− if (!msix_vecsave)
3681− qib_dev_err(dd, "No mem to save MSIx data\n");
3682− } else
3683− msix_vecsave = NULL;
3678+ }
36843679
36853680 /*
36863681 * Core PCI (as of 2.6.18) doesn't save or rewrite the full vector
@@ -5043,8 +5038,6 @@ static void init_7322_cntrnames(struct qib_devdata *dd) · 1356 unmodified lines
50435038 dd->cspec->cntrnamelen = 1 + s - cntr7322names;
50445039 dd->cspec->cntrs = kmalloc(dd->cspec->ncntrs
50455040 * sizeof(u64), GFP_KERNEL);
5046− if (!dd->cspec->cntrs)
5047− qib_dev_err(dd, "Failed allocation for counters\n");
50485041
50495042 for (i = 0, s = (char *)portcntr7322names; s; i++)
50505043 s = strchr(s + 1, '\n');
@@ -5053,9 +5046,6 @@ static void init_7322_cntrnames(struct qib_devdata *dd) · 2 unmodified lines
50535046 for (i = 0; i < dd->num_pports; ++i) {
50545047 dd->pport[i].cpspec->portcntrs = kmalloc(dd->cspec->nportcntrs
50555048 * sizeof(u64), GFP_KERNEL);
5056− if (!dd->pport[i].cpspec->portcntrs)
5057− qib_dev_err(dd,
5058− "Failed allocation for portcounters\n");
50595049 }
50605050 }
50615051
@@ -6461,7 +6451,6 @@ static int qib_init_7322_variables(struct qib_devdata *dd) · 1399 unmodified lines
64616451 sizeof(*dd->cspec->sendibchk), GFP_KERNEL);
64626452 if (!dd->cspec->sendchkenable || !dd->cspec->sendgrhchk ||
64636453 !dd->cspec->sendibchk) {
6464− qib_dev_err(dd, "Failed allocation for hdrchk bitmaps\n");
64656454 ret = -ENOMEM;
64666455 goto bail;
64676456 }
@@ -7338,10 +7327,9 @@ struct qib_devdata *qib_init_iba7322_funcs(struct pci_dev *pdev, · 870 unmodified lines
73387327 tabsize = actual_cnt;
73397328 dd->cspec->msix_entries = kzalloc(tabsize *
73407329 sizeof(struct qib_msix_entry), GFP_KERNEL);
7341− if (!dd->cspec->msix_entries) {
7342− qib_dev_err(dd, "No memory for MSIx table\n");
7330+ if (!dd->cspec->msix_entries)
73437331 tabsize = 0;
7344− }
7332+
73457333 for (i = 0; i < tabsize; i++)
73467334 dd->cspec->msix_entries[i].msix.entry = i;
73477335
drivers/infiniband/hw/qib/qib_init.c+8−39
@@ -133,11 +133,8 @@ int qib_create_ctxts(struct qib_devdata *dd) · 132 unmodified lines
133133 * cleanup iterates across all possible ctxts.
134134 */
135135 dd->rcd = kcalloc(dd->ctxtcnt, sizeof(*dd->rcd), GFP_KERNEL);
136− if (!dd->rcd) {
137− qib_dev_err(dd,
138− "Unable to allocate ctxtdata array, failing\n");
136+ if (!dd->rcd)
139137 return -ENOMEM;
140− }
141138
142139 /* create (one or more) kctxt */
143140 for (i = 0; i < dd->first_user_ctxt; ++i) {
@@ -265,39 +262,23 @@ int qib_init_pportdata(struct qib_pportdata *ppd, struct qib_devdata *dd, · 121 unmodified lines
265262 size = IB_CC_TABLE_CAP_DEFAULT * sizeof(struct ib_cc_table_entry)
266263 * IB_CCT_ENTRIES;
267264 ppd->ccti_entries = kzalloc(size, GFP_KERNEL);
268− if (!ppd->ccti_entries) {
269− qib_dev_err(dd,
270− "failed to allocate congestion control table for port %d!\n",
271− port);
265+ if (!ppd->ccti_entries)
272266 goto bail;
273− }
274267
275268 size = IB_CC_CCS_ENTRIES * sizeof(struct ib_cc_congestion_entry);
276269 ppd->congestion_entries = kzalloc(size, GFP_KERNEL);
277− if (!ppd->congestion_entries) {
278− qib_dev_err(dd,
279− "failed to allocate congestion setting list for port %d!\n",
280− port);
270+ if (!ppd->congestion_entries)
281271 goto bail_1;
282− }
283272
284273 size = sizeof(struct cc_table_shadow);
285274 ppd->ccti_entries_shadow = kzalloc(size, GFP_KERNEL);
286− if (!ppd->ccti_entries_shadow) {
287− qib_dev_err(dd,
288− "failed to allocate shadow ccti list for port %d!\n",
289− port);
275+ if (!ppd->ccti_entries_shadow)
290276 goto bail_2;
291− }
292277
293278 size = sizeof(struct ib_cc_congestion_setting_attr);
294279 ppd->congestion_entries_shadow = kzalloc(size, GFP_KERNEL);
295− if (!ppd->congestion_entries_shadow) {
296− qib_dev_err(dd,
297− "failed to allocate shadow congestion setting list for port %d!\n",
298− port);
280+ if (!ppd->congestion_entries_shadow)
299281 goto bail_3;
300− }
301282
302283 return 0;
303284
@@ -391,18 +372,12 @@ static void init_shadow_tids(struct qib_devdata *dd) · 87 unmodified lines
391372 dma_addr_t *addrs;
392373
393374 pages = vzalloc(dd->cfgctxts * dd->rcvtidcnt * sizeof(struct page *));
394− if (!pages) {
395− qib_dev_err(dd,
396− "failed to allocate shadow page * array, no expected sends!\n");
375+ if (!pages)
397376 goto bail;
398− }
399377
400378 addrs = vzalloc(dd->cfgctxts * dd->rcvtidcnt * sizeof(dma_addr_t));
401− if (!addrs) {
402− qib_dev_err(dd,
403− "failed to allocate shadow dma handle array, no expected sends!\n");
379+ if (!addrs)
404380 goto bail_free;
405− }
406381
407382 dd->pageshadow = pages;
408383 dd->physshadow = addrs;
@@ -1026,11 +1001,8 @@ static void qib_verify_pioperf(struct qib_devdata *dd) · 617 unmodified lines
10261001 cnt = 1024;
10271002
10281003 addr = vmalloc(cnt);
1029− if (!addr) {
1030− qib_devinfo(dd->pcidev,
1031− "Couldn't get memory for checking PIO perf, skipping\n");
1004+ if (!addr)
10321005 goto done;
1033− }
10341006
10351007 preempt_disable(); /* we want reasonably accurate elapsed time */
10361008 msecs = 1 + jiffies_to_msecs(jiffies);
@@ -1172,9 +1144,6 @@ struct qib_devdata *qib_alloc_devdata(struct pci_dev *pdev, size_t extra) · 135 unmodified lines
11721144 sizeof(long), GFP_KERNEL);
11731145 if (qib_cpulist)
11741146 qib_cpulist_count = count;
1175− else
1176− qib_early_err(&pdev->dev,
1177− "Could not alloc cpulist info, cpu affinity might be wrong\n");
11781147 }
11791148 #ifdef CONFIG_DEBUG_FS
11801149 qib_dbg_ibdev_init(&dd->verbs_dev);
drivers/infiniband/hw/qib/qib_rc.c+5−39
@@ -941,8 +941,6 @@ void qib_rc_send_complete(struct rvt_qp *qp, struct ib_header *hdr) · 940 unmodified lines
941941 {
942942 struct ib_other_headers *ohdr;
943943 struct rvt_swqe *wqe;
944− struct ib_wc wc;
945− unsigned i;
946944 u32 opcode;
947945 u32 psn;
948946
@@ -988,22 +986,8 @@ void qib_rc_send_complete(struct rvt_qp *qp, struct ib_header *hdr) · 39 unmodified lines
988986 qp->s_last = s_last;
989987 /* see post_send() */
990988 barrier();
991− for (i = 0; i < wqe->wr.num_sge; i++) {
992− struct rvt_sge *sge = &wqe->sg_list[i];
993−
994− rvt_put_mr(sge->mr);
995− }
996− /* Post a send completion queue entry if requested. */
997− if (!(qp->s_flags & RVT_S_SIGNAL_REQ_WR) ||
998− (wqe->wr.send_flags & IB_SEND_SIGNALED)) {
999− memset(&wc, 0, sizeof(wc));
1000− wc.wr_id = wqe->wr.wr_id;
1001− wc.status = IB_WC_SUCCESS;
1002− wc.opcode = ib_qib_wc_opcode[wqe->wr.opcode];
1003− wc.byte_len = wqe->length;
1004− wc.qp = &qp->ibqp;
1005− rvt_cq_enter(ibcq_to_rvtcq(qp->ibqp.send_cq), &wc, 0);
1006− }
989+ rvt_put_swqe(wqe);
990+ rvt_qp_swqe_complete(qp, wqe, IB_WC_SUCCESS);
1007991 }
1008992 /*
1009993 * If we were waiting for sends to complete before resending,
@@ -1032,9 +1016,6 @@ static struct rvt_swqe *do_rc_completion(struct rvt_qp *qp, · 22 unmodified lines
10321016 struct rvt_swqe *wqe,
10331017 struct qib_ibport *ibp)
10341018 {
1035− struct ib_wc wc;
1036− unsigned i;
1037−
10381019 /*
10391020 * Don't decrement refcount and don't generate a
10401021 * completion if the SWQE is being resent until the send
@@ -1044,28 +1025,14 @@ static struct rvt_swqe *do_rc_completion(struct rvt_qp *qp, · 3 unmodified lines
10441025 qib_cmp24(qp->s_sending_psn, qp->s_sending_hpsn) > 0) {
10451026 u32 s_last;
10461027
1047− for (i = 0; i < wqe->wr.num_sge; i++) {
1048− struct rvt_sge *sge = &wqe->sg_list[i];
1049−
1050− rvt_put_mr(sge->mr);
1051− }
1028+ rvt_put_swqe(wqe);
10521029 s_last = qp->s_last;
10531030 if (++s_last >= qp->s_size)
10541031 s_last = 0;
10551032 qp->s_last = s_last;
10561033 /* see post_send() */
10571034 barrier();
1058− /* Post a send completion queue entry if requested. */
1059− if (!(qp->s_flags & RVT_S_SIGNAL_REQ_WR) ||
1060− (wqe->wr.send_flags & IB_SEND_SIGNALED)) {
1061− memset(&wc, 0, sizeof(wc));
1062− wc.wr_id = wqe->wr.wr_id;
1063− wc.status = IB_WC_SUCCESS;
1064− wc.opcode = ib_qib_wc_opcode[wqe->wr.opcode];
1065− wc.byte_len = wqe->length;
1066− wc.qp = &qp->ibqp;
1067− rvt_cq_enter(ibcq_to_rvtcq(qp->ibqp.send_cq), &wc, 0);
1068− }
1035+ rvt_qp_swqe_complete(qp, wqe, IB_WC_SUCCESS);
10691036 } else
10701037 this_cpu_inc(*ibp->rvp.rc_delayed_comp);
10711038
@@ -2112,8 +2079,7 @@ send_last: · 1040 unmodified lines
21122079 * Update the next expected PSN. We add 1 later
21132080 * below, so only add the remainder here.
21142081 */
2115− if (len > pmtu)
2116− qp->r_psn += (len - 1) / pmtu;
2082+ qp->r_psn += rvt_div_mtu(qp, len - 1);
21172083 } else {
21182084 e->rdma_sge.mr = NULL;
21192085 e->rdma_sge.vaddr = NULL;
drivers/infiniband/hw/qib/qib_ruc.c+2−22
@@ -793,7 +793,6 @@ void qib_send_complete(struct rvt_qp *qp, struct rvt_swqe *wqe, · 792 unmodified lines
793793 enum ib_wc_status status)
794794 {
795795 u32 old_last, last;
796− unsigned i;
797796
798797 if (!(ib_rvt_state_ops[qp->state] & RVT_PROCESS_OR_FLUSH_SEND))
799798 return;
@@ -805,32 +804,13 @@ void qib_send_complete(struct rvt_qp *qp, struct rvt_swqe *wqe, · 5 unmodified lines
805804 qp->s_last = last;
806805 /* See post_send() */
807806 barrier();
808− for (i = 0; i < wqe->wr.num_sge; i++) {
809− struct rvt_sge *sge = &wqe->sg_list[i];
810−
811− rvt_put_mr(sge->mr);
812− }
807+ rvt_put_swqe(wqe);
813808 if (qp->ibqp.qp_type == IB_QPT_UD ||
814809 qp->ibqp.qp_type == IB_QPT_SMI ||
815810 qp->ibqp.qp_type == IB_QPT_GSI)
816811 atomic_dec(&ibah_to_rvtah(wqe->ud_wr.ah)->refcount);
817812
818− /* See ch. 11.2.4.1 and 10.7.3.1 */
819− if (!(qp->s_flags & RVT_S_SIGNAL_REQ_WR) ||
820− (wqe->wr.send_flags & IB_SEND_SIGNALED) ||
821− status != IB_WC_SUCCESS) {
822− struct ib_wc wc;
823−
824− memset(&wc, 0, sizeof(wc));
825− wc.wr_id = wqe->wr.wr_id;
826− wc.status = status;
827− wc.opcode = ib_qib_wc_opcode[wqe->wr.opcode];
828− wc.qp = &qp->ibqp;
829− if (status == IB_WC_SUCCESS)
830− wc.byte_len = wqe->length;
831− rvt_cq_enter(ibcq_to_rvtcq(qp->ibqp.send_cq), &wc,
832− status != IB_WC_SUCCESS);
833− }
813+ rvt_qp_swqe_complete(qp, wqe, status);
834814
835815 if (qp->s_acked == old_last)
836816 qp->s_acked = last;
drivers/infiniband/hw/qib/qib_verbs.c+8−25
@@ -113,19 +113,6 @@ static unsigned int ib_qib_disable_sma; · 112 unmodified lines
113113 module_param_named(disable_sma, ib_qib_disable_sma, uint, S_IWUSR | S_IRUGO);
114114 MODULE_PARM_DESC(disable_sma, "Disable the SMA");
115115
116−/*
117− * Translate ib_wr_opcode into ib_wc_opcode.
118− */
119−const enum ib_wc_opcode ib_qib_wc_opcode[] = {
120− [IB_WR_RDMA_WRITE] = IB_WC_RDMA_WRITE,
121− [IB_WR_RDMA_WRITE_WITH_IMM] = IB_WC_RDMA_WRITE,
122− [IB_WR_SEND] = IB_WC_SEND,
123− [IB_WR_SEND_WITH_IMM] = IB_WC_SEND,
124− [IB_WR_RDMA_READ] = IB_WC_RDMA_READ,
125− [IB_WR_ATOMIC_CMP_AND_SWP] = IB_WC_COMP_SWAP,
126− [IB_WR_ATOMIC_FETCH_AND_ADD] = IB_WC_FETCH_ADD
127−};
128−
129116 /*
130117 * System image GUID.
131118 */
@@ -464,7 +451,7 @@ static void mem_timer(unsigned long data) · 332 unmodified lines
464451 priv = list_entry(list->next, struct qib_qp_priv, iowait);
465452 qp = priv->owner;
466453 list_del_init(&priv->iowait);
467− atomic_inc(&qp->refcount);
454+ rvt_get_qp(qp);
468455 if (!list_empty(list))
469456 mod_timer(&dev->mem_timer, jiffies + 1);
470457 }
@@ -477,8 +464,7 @@ static void mem_timer(unsigned long data) · 6 unmodified lines
477464 qib_schedule_send(qp);
478465 }
479466 spin_unlock_irqrestore(&qp->s_lock, flags);
480− if (atomic_dec_and_test(&qp->refcount))
481− wake_up(&qp->wait);
467+ rvt_put_qp(qp);
482468 }
483469 }
484470
@@ -762,7 +748,7 @@ void qib_put_txreq(struct qib_verbs_txreq *tx) · 277 unmodified lines
762748 iowait);
763749 qp = priv->owner;
764750 list_del_init(&priv->iowait);
765− atomic_inc(&qp->refcount);
751+ rvt_get_qp(qp);
766752 spin_unlock_irqrestore(&dev->rdi.pending_lock, flags);
767753
768754 spin_lock_irqsave(&qp->s_lock, flags);
@@ -772,8 +758,7 @@ void qib_put_txreq(struct qib_verbs_txreq *tx) · 3 unmodified lines
772758 }
773759 spin_unlock_irqrestore(&qp->s_lock, flags);
774760
775− if (atomic_dec_and_test(&qp->refcount))
776− wake_up(&qp->wait);
761+ rvt_put_qp(qp);
777762 } else
778763 spin_unlock_irqrestore(&dev->rdi.pending_lock, flags);
779764 }
@@ -808,7 +793,7 @@ void qib_verbs_sdma_desc_avail(struct qib_pportdata *ppd, unsigned avail) · 28 unmodified lines
808793 break;
809794 avail -= qpp->s_tx->txreq.sg_count;
810795 list_del_init(&qpp->iowait);
811− atomic_inc(&qp->refcount);
796+ rvt_get_qp(qp);
812797 qps[n++] = qp;
813798 }
814799
@@ -822,8 +807,7 @@ void qib_verbs_sdma_desc_avail(struct qib_pportdata *ppd, unsigned avail) · 7 unmodified lines
822807 qib_schedule_send(qp);
823808 }
824809 spin_unlock(&qp->s_lock);
825− if (atomic_dec_and_test(&qp->refcount))
826− wake_up(&qp->wait);
810+ rvt_put_qp(qp);
827811 }
828812 }
829813
@@ -1288,7 +1272,7 @@ void qib_ib_piobufavail(struct qib_devdata *dd) · 458 unmodified lines
12881272 priv = list_entry(list->next, struct qib_qp_priv, iowait);
12891273 qp = priv->owner;
12901274 list_del_init(&priv->iowait);
1291− atomic_inc(&qp->refcount);
1275+ rvt_get_qp(qp);
12921276 qps[n++] = qp;
12931277 }
12941278 dd->f_wantpiobuf_intr(dd, 0);
@@ -1306,8 +1290,7 @@ full: · 11 unmodified lines
13061290 spin_unlock_irqrestore(&qp->s_lock, flags);
13071291
13081292 /* Notify qib_destroy_qp() if it is waiting. */
1309− if (atomic_dec_and_test(&qp->refcount))
1310− wake_up(&qp->wait);
1293+ rvt_put_qp(qp);
13111294 }
13121295 }
13131296
drivers/infiniband/hw/usnic/usnic_ib_qp_grp.c+7−15
@@ -117,10 +117,10 @@ static int enable_qp_grp(struct usnic_ib_qp_grp *qp_grp) · 116 unmodified lines
117117 vnic_idx = usnic_vnic_get_index(qp_grp->vf->vnic);
118118
119119 res_chunk = get_qp_res_chunk(qp_grp);
120− if (IS_ERR_OR_NULL(res_chunk)) {
120+ if (IS_ERR(res_chunk)) {
121121 usnic_err("Unable to get qp res with err %ld\n",
122122 PTR_ERR(res_chunk));
123− return res_chunk ? PTR_ERR(res_chunk) : -ENOMEM;
123+ return PTR_ERR(res_chunk);
124124 }
125125
126126 for (i = 0; i < res_chunk->cnt; i++) {
@@ -158,10 +158,10 @@ static int disable_qp_grp(struct usnic_ib_qp_grp *qp_grp) · 31 unmodified lines
158158 vnic_idx = usnic_vnic_get_index(qp_grp->vf->vnic);
159159
160160 res_chunk = get_qp_res_chunk(qp_grp);
161− if (IS_ERR_OR_NULL(res_chunk)) {
161+ if (IS_ERR(res_chunk)) {
162162 usnic_err("Unable to get qp res with err %ld\n",
163163 PTR_ERR(res_chunk));
164− return res_chunk ? PTR_ERR(res_chunk) : -ENOMEM;
164+ return PTR_ERR(res_chunk);
165165 }
166166
167167 for (i = 0; i < res_chunk->cnt; i++) {
@@ -186,11 +186,11 @@ static int init_filter_action(struct usnic_ib_qp_grp *qp_grp, · 18 unmodified lines
186186 struct usnic_vnic_res_chunk *res_chunk;
187187
188188 res_chunk = usnic_ib_qp_grp_get_chunk(qp_grp, USNIC_VNIC_RES_TYPE_RQ);
189− if (IS_ERR_OR_NULL(res_chunk)) {
189+ if (IS_ERR(res_chunk)) {
190190 usnic_err("Unable to get %s with err %ld\n",
191191 usnic_vnic_res_type_to_str(USNIC_VNIC_RES_TYPE_RQ),
192192 PTR_ERR(res_chunk));
193− return res_chunk ? PTR_ERR(res_chunk) : -ENOMEM;
193+ return PTR_ERR(res_chunk);
194194 }
195195
196196 uaction->vnic_idx = usnic_vnic_get_index(qp_grp->vf->vnic);
@@ -228,8 +228,6 @@ create_roce_custom_flow(struct usnic_ib_qp_grp *qp_grp, · 31 unmodified lines
228228
229229 flow = usnic_fwd_alloc_flow(qp_grp->ufdev, &filter, &uaction);
230230 if (IS_ERR_OR_NULL(flow)) {
231− usnic_err("Unable to alloc flow failed with err %ld\n",
232− PTR_ERR(flow));
233231 err = flow ? PTR_ERR(flow) : -EFAULT;
234232 goto out_unreserve_port;
235233 }
@@ -303,8 +301,6 @@ create_udp_flow(struct usnic_ib_qp_grp *qp_grp, · 67 unmodified lines
303301
304302 flow = usnic_fwd_alloc_flow(qp_grp->ufdev, &filter, &uaction);
305303 if (IS_ERR_OR_NULL(flow)) {
306− usnic_err("Unable to alloc flow failed with err %ld\n",
307− PTR_ERR(flow));
308304 err = flow ? PTR_ERR(flow) : -EFAULT;
309305 goto out_put_sock;
310306 }
@@ -694,18 +690,14 @@ usnic_ib_qp_grp_create(struct usnic_fwd_dev *ufdev, struct usnic_ib_vf *vf, · 383 unmodified lines
694690 }
695691
696692 qp_grp = kzalloc(sizeof(*qp_grp), GFP_ATOMIC);
697− if (!qp_grp) {
698− usnic_err("Unable to alloc qp_grp - Out of memory\n");
693+ if (!qp_grp)
699694 return NULL;
700− }
701695
702696 qp_grp->res_chunk_list = alloc_res_chunk_list(vf->vnic, res_spec,
703697 qp_grp);
704698 if (IS_ERR_OR_NULL(qp_grp->res_chunk_list)) {
705699 err = qp_grp->res_chunk_list ?
706700 PTR_ERR(qp_grp->res_chunk_list) : -ENOMEM;
707− usnic_err("Unable to alloc res for %d with err %d\n",
708− qp_grp->grp_id, err);
709701 goto out_free_qp_grp;
710702 }
711703
drivers/infiniband/hw/usnic/usnic_ib_verbs.c+9−7
@@ -87,12 +87,12 @@ static int usnic_ib_fill_create_qp_resp(struct usnic_ib_qp_grp *qp_grp, · 86 unmodified lines
8787 resp.bar_len = bar->len;
8888
8989 chunk = usnic_ib_qp_grp_get_chunk(qp_grp, USNIC_VNIC_RES_TYPE_RQ);
90− if (IS_ERR_OR_NULL(chunk)) {
90+ if (IS_ERR(chunk)) {
9191 usnic_err("Failed to get chunk %s for qp_grp %d with err %ld\n",
9292 usnic_vnic_res_type_to_str(USNIC_VNIC_RES_TYPE_RQ),
9393 qp_grp->grp_id,
9494 PTR_ERR(chunk));
95− return chunk ? PTR_ERR(chunk) : -ENOMEM;
95+ return PTR_ERR(chunk);
9696 }
9797
9898 WARN_ON(chunk->type != USNIC_VNIC_RES_TYPE_RQ);
@@ -101,12 +101,12 @@ static int usnic_ib_fill_create_qp_resp(struct usnic_ib_qp_grp *qp_grp, · 2 unmodified lines
101101 resp.rq_idx[i] = chunk->res[i]->vnic_idx;
102102
103103 chunk = usnic_ib_qp_grp_get_chunk(qp_grp, USNIC_VNIC_RES_TYPE_WQ);
104− if (IS_ERR_OR_NULL(chunk)) {
104+ if (IS_ERR(chunk)) {
105105 usnic_err("Failed to get chunk %s for qp_grp %d with err %ld\n",
106106 usnic_vnic_res_type_to_str(USNIC_VNIC_RES_TYPE_WQ),
107107 qp_grp->grp_id,
108108 PTR_ERR(chunk));
109− return chunk ? PTR_ERR(chunk) : -ENOMEM;
109+ return PTR_ERR(chunk);
110110 }
111111
112112 WARN_ON(chunk->type != USNIC_VNIC_RES_TYPE_WQ);
@@ -115,12 +115,12 @@ static int usnic_ib_fill_create_qp_resp(struct usnic_ib_qp_grp *qp_grp, · 2 unmodified lines
115115 resp.wq_idx[i] = chunk->res[i]->vnic_idx;
116116
117117 chunk = usnic_ib_qp_grp_get_chunk(qp_grp, USNIC_VNIC_RES_TYPE_CQ);
118− if (IS_ERR_OR_NULL(chunk)) {
118+ if (IS_ERR(chunk)) {
119119 usnic_err("Failed to get chunk %s for qp_grp %d with err %ld\n",
120120 usnic_vnic_res_type_to_str(USNIC_VNIC_RES_TYPE_CQ),
121121 qp_grp->grp_id,
122122 PTR_ERR(chunk));
123− return chunk ? PTR_ERR(chunk) : -ENOMEM;
123+ return PTR_ERR(chunk);
124124 }
125125
126126 WARN_ON(chunk->type != USNIC_VNIC_RES_TYPE_CQ);
@@ -738,7 +738,9 @@ int usnic_ib_mmap(struct ib_ucontext *context, · 611 unmodified lines
738738
739739 /* In ib callbacks section - Start of stub funcs */
740740 struct ib_ah *usnic_ib_create_ah(struct ib_pd *pd,
741− struct ib_ah_attr *ah_attr)
741+ struct ib_ah_attr *ah_attr,
742+ struct ib_udata *udata)
743+
742744 {
743745 usnic_dbg("\n");
744746 return ERR_PTR(-EPERM);
drivers/infiniband/hw/usnic/usnic_ib_verbs.h+3−1
@@ -75,7 +75,9 @@ int usnic_ib_dealloc_ucontext(struct ib_ucontext *ibcontext); · 74 unmodified lines
7575 int usnic_ib_mmap(struct ib_ucontext *context,
7676 struct vm_area_struct *vma);
7777 struct ib_ah *usnic_ib_create_ah(struct ib_pd *pd,
78− struct ib_ah_attr *ah_attr);
78+ struct ib_ah_attr *ah_attr,
79+ struct ib_udata *udata);
80+
7981 int usnic_ib_destroy_ah(struct ib_ah *ah);
8082 int usnic_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
8183 struct ib_send_wr **bad_wr);
drivers/infiniband/hw/usnic/usnic_vnic.c+6−16
@@ -241,17 +241,12 @@ usnic_vnic_get_resources(struct usnic_vnic *vnic, enum usnic_vnic_res_type type, · 240 unmodified lines
241241 return ERR_PTR(-EINVAL);
242242
243243 ret = kzalloc(sizeof(*ret), GFP_ATOMIC);
244− if (!ret) {
245− usnic_err("Failed to allocate chunk for %s - Out of memory\n",
246− usnic_vnic_pci_name(vnic));
244+ if (!ret)
247245 return ERR_PTR(-ENOMEM);
248− }
249246
250247 if (cnt > 0) {
251248 ret->res = kcalloc(cnt, sizeof(*(ret->res)), GFP_ATOMIC);
252249 if (!ret->res) {
253− usnic_err("Failed to allocate resources for %s. Out of memory\n",
254− usnic_vnic_pci_name(vnic));
255250 kfree(ret);
256251 return ERR_PTR(-ENOMEM);
257252 }
@@ -311,8 +306,10 @@ static int usnic_vnic_alloc_res_chunk(struct usnic_vnic *vnic, · 53 unmodified lines
311306 struct usnic_vnic_res *res;
312307
313308 cnt = vnic_dev_get_res_count(vnic->vdev, _to_vnic_res_type(type));
314− if (cnt < 1)
309+ if (cnt < 1) {
310+ usnic_err("Wrong res count with cnt %d\n", cnt);
315311 return -EINVAL;
312+ }
316313
317314 chunk->cnt = chunk->free_cnt = cnt;
318315 chunk->res = kzalloc(sizeof(*(chunk->res))*cnt, GFP_KERNEL);
@@ -384,12 +381,8 @@ static int usnic_vnic_discover_resources(struct pci_dev *pdev, · 65 unmodified lines
384381 res_type < USNIC_VNIC_RES_TYPE_MAX; res_type++) {
385382 err = usnic_vnic_alloc_res_chunk(vnic, res_type,
386383 &vnic->chunks[res_type]);
387− if (err) {
388− usnic_err("Failed to alloc res %s with err %d\n",
389− usnic_vnic_res_type_to_str(res_type),
390− err);
384+ if (err)
391385 goto out_clean_chunks;
392− }
393386 }
394387
395388 return 0;
@@ -454,11 +447,8 @@ struct usnic_vnic *usnic_vnic_alloc(struct pci_dev *pdev) · 58 unmodified lines
454447 }
455448
456449 vnic = kzalloc(sizeof(*vnic), GFP_KERNEL);
457− if (!vnic) {
458− usnic_err("Failed to alloc vnic for %s - out of memory\n",
459− pci_name(pdev));
450+ if (!vnic)
460451 return ERR_PTR(-ENOMEM);
461− }
462452
463453 spin_lock_init(&vnic->res_lock);
464454
drivers/infiniband/hw/vmw_pvrdma/Kconfig+7−0
@@ -0,0 +1,7 @@
1+config INFINIBAND_VMWARE_PVRDMA
2+ tristate "VMware Paravirtualized RDMA Driver"
3+ depends on NETDEVICES && ETHERNET && PCI && INET && VMXNET3
4+ ---help---
5+ This driver provides low-level support for VMware Paravirtual
6+ RDMA adapter. It interacts with the VMXNet3 driver to provide
7+ Ethernet capabilities.
drivers/infiniband/hw/vmw_pvrdma/Makefile+3−0
@@ -0,0 +1,3 @@
1+obj-$(CONFIG_INFINIBAND_VMWARE_PVRDMA) += vmw_pvrdma.o
2+
3+vmw_pvrdma-y := pvrdma_cmd.o pvrdma_cq.o pvrdma_doorbell.o pvrdma_main.o pvrdma_misc.o pvrdma_mr.o pvrdma_qp.o pvrdma_verbs.o
drivers/infiniband/hw/vmw_pvrdma/pvrdma.h+474−0
@@ -0,0 +1,474 @@
1+/*
2+ * Copyright (c) 2012-2016 VMware, Inc. All rights reserved.
3+ *
4+ * This program is free software; you can redistribute it and/or
5+ * modify it under the terms of EITHER the GNU General Public License
6+ * version 2 as published by the Free Software Foundation or the BSD
7+ * 2-Clause License. This program is distributed in the hope that it
8+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
9+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
10+ * See the GNU General Public License version 2 for more details at
11+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
12+ *
13+ * You should have received a copy of the GNU General Public License
14+ * along with this program available in the file COPYING in the main
15+ * directory of this source tree.
16+ *
17+ * The BSD 2-Clause License
18+ *
19+ * Redistribution and use in source and binary forms, with or
20+ * without modification, are permitted provided that the following
21+ * conditions are met:
22+ *
23+ * - Redistributions of source code must retain the above
24+ * copyright notice, this list of conditions and the following
25+ * disclaimer.
26+ *
27+ * - Redistributions in binary form must reproduce the above
28+ * copyright notice, this list of conditions and the following
29+ * disclaimer in the documentation and/or other materials
30+ * provided with the distribution.
31+ *
32+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
33+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
34+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
35+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
36+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
37+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
38+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
39+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
40+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
41+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
42+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
43+ * OF THE POSSIBILITY OF SUCH DAMAGE.
44+ */
45+
46+#ifndef __PVRDMA_H__
47+#define __PVRDMA_H__
48+
49+#include <linux/compiler.h>
50+#include <linux/interrupt.h>
51+#include <linux/list.h>
52+#include <linux/mutex.h>
53+#include <linux/pci.h>
54+#include <linux/semaphore.h>
55+#include <linux/workqueue.h>
56+#include <rdma/ib_umem.h>
57+#include <rdma/ib_verbs.h>
58+#include <rdma/vmw_pvrdma-abi.h>
59+
60+#include "pvrdma_ring.h"
61+#include "pvrdma_dev_api.h"
62+#include "pvrdma_verbs.h"
63+
64+/* NOT the same as BIT_MASK(). */
65+#define PVRDMA_MASK(n) ((n << 1) - 1)
66+
67+/*
68+ * VMware PVRDMA PCI device id.
69+ */
70+#define PCI_DEVICE_ID_VMWARE_PVRDMA 0x0820
71+
72+struct pvrdma_dev;
73+
74+struct pvrdma_page_dir {
75+ dma_addr_t dir_dma;
76+ u64 *dir;
77+ int ntables;
78+ u64 **tables;
79+ u64 npages;
80+ void **pages;
81+};
82+
83+struct pvrdma_cq {
84+ struct ib_cq ibcq;
85+ int offset;
86+ spinlock_t cq_lock; /* Poll lock. */
87+ struct pvrdma_uar_map *uar;
88+ struct ib_umem *umem;
89+ struct pvrdma_ring_state *ring_state;
90+ struct pvrdma_page_dir pdir;
91+ u32 cq_handle;
92+ bool is_kernel;
93+ atomic_t refcnt;
94+ wait_queue_head_t wait;
95+};
96+
97+struct pvrdma_id_table {
98+ u32 last;
99+ u32 top;
100+ u32 max;
101+ u32 mask;
102+ spinlock_t lock; /* Table lock. */
103+ unsigned long *table;
104+};
105+
106+struct pvrdma_uar_map {
107+ unsigned long pfn;
108+ void __iomem *map;
109+ int index;
110+};
111+
112+struct pvrdma_uar_table {
113+ struct pvrdma_id_table tbl;
114+ int size;
115+};
116+
117+struct pvrdma_ucontext {
118+ struct ib_ucontext ibucontext;
119+ struct pvrdma_dev *dev;
120+ struct pvrdma_uar_map uar;
121+ u64 ctx_handle;
122+};
123+
124+struct pvrdma_pd {
125+ struct ib_pd ibpd;
126+ u32 pdn;
127+ u32 pd_handle;
128+ int privileged;
129+};
130+
131+struct pvrdma_mr {
132+ u32 mr_handle;
133+ u64 iova;
134+ u64 size;
135+};
136+
137+struct pvrdma_user_mr {
138+ struct ib_mr ibmr;
139+ struct ib_umem *umem;
140+ struct pvrdma_mr mmr;
141+ struct pvrdma_page_dir pdir;
142+ u64 *pages;
143+ u32 npages;
144+ u32 max_pages;
145+ u32 page_shift;
146+};
147+
148+struct pvrdma_wq {
149+ struct pvrdma_ring *ring;
150+ spinlock_t lock; /* Work queue lock. */
151+ int wqe_cnt;
152+ int wqe_size;
153+ int max_sg;
154+ int offset;
155+};
156+
157+struct pvrdma_ah {
158+ struct ib_ah ibah;
159+ struct pvrdma_av av;
160+};
161+
162+struct pvrdma_qp {
163+ struct ib_qp ibqp;
164+ u32 qp_handle;
165+ u32 qkey;
166+ struct pvrdma_wq sq;
167+ struct pvrdma_wq rq;
168+ struct ib_umem *rumem;
169+ struct ib_umem *sumem;
170+ struct pvrdma_page_dir pdir;
171+ int npages;
172+ int npages_send;
173+ int npages_recv;
174+ u32 flags;
175+ u8 port;
176+ u8 state;
177+ bool is_kernel;
178+ struct mutex mutex; /* QP state mutex. */
179+ atomic_t refcnt;
180+ wait_queue_head_t wait;
181+};
182+
183+struct pvrdma_dev {
184+ /* PCI device-related information. */
185+ struct ib_device ib_dev;
186+ struct pci_dev *pdev;
187+ void __iomem *regs;
188+ struct pvrdma_device_shared_region *dsr; /* Shared region pointer */
189+ dma_addr_t dsrbase; /* Shared region base address */
190+ void *cmd_slot;
191+ void *resp_slot;
192+ unsigned long flags;
193+ struct list_head device_link;
194+
195+ /* Locking and interrupt information. */
196+ spinlock_t cmd_lock; /* Command lock. */
197+ struct semaphore cmd_sema;
198+ struct completion cmd_done;
199+ struct {
200+ enum pvrdma_intr_type type; /* Intr type */
201+ struct msix_entry msix_entry[PVRDMA_MAX_INTERRUPTS];
202+ irq_handler_t handler[PVRDMA_MAX_INTERRUPTS];
203+ u8 enabled[PVRDMA_MAX_INTERRUPTS];
204+ u8 size;
205+ } intr;
206+
207+ /* RDMA-related device information. */
208+ union ib_gid *sgid_tbl;
209+ struct pvrdma_ring_state *async_ring_state;
210+ struct pvrdma_page_dir async_pdir;
211+ struct pvrdma_ring_state *cq_ring_state;
212+ struct pvrdma_page_dir cq_pdir;
213+ struct pvrdma_cq **cq_tbl;
214+ spinlock_t cq_tbl_lock;
215+ struct pvrdma_qp **qp_tbl;
216+ spinlock_t qp_tbl_lock;
217+ struct pvrdma_uar_table uar_table;
218+ struct pvrdma_uar_map driver_uar;
219+ __be64 sys_image_guid;
220+ spinlock_t desc_lock; /* Device modification lock. */
221+ u32 port_cap_mask;
222+ struct mutex port_mutex; /* Port modification mutex. */
223+ bool ib_active;
224+ atomic_t num_qps;
225+ atomic_t num_cqs;
226+ atomic_t num_pds;
227+ atomic_t num_ahs;
228+
229+ /* Network device information. */
230+ struct net_device *netdev;
231+ struct notifier_block nb_netdev;
232+};
233+
234+struct pvrdma_netdevice_work {
235+ struct work_struct work;
236+ struct net_device *event_netdev;
237+ unsigned long event;
238+};
239+
240+static inline struct pvrdma_dev *to_vdev(struct ib_device *ibdev)
241+{
242+ return container_of(ibdev, struct pvrdma_dev, ib_dev);
243+}
244+
245+static inline struct
246+pvrdma_ucontext *to_vucontext(struct ib_ucontext *ibucontext)
247+{
248+ return container_of(ibucontext, struct pvrdma_ucontext, ibucontext);
249+}
250+
251+static inline struct pvrdma_pd *to_vpd(struct ib_pd *ibpd)
252+{
253+ return container_of(ibpd, struct pvrdma_pd, ibpd);
254+}
255+
256+static inline struct pvrdma_cq *to_vcq(struct ib_cq *ibcq)
257+{
258+ return container_of(ibcq, struct pvrdma_cq, ibcq);
259+}
260+
261+static inline struct pvrdma_user_mr *to_vmr(struct ib_mr *ibmr)
262+{
263+ return container_of(ibmr, struct pvrdma_user_mr, ibmr);
264+}
265+
266+static inline struct pvrdma_qp *to_vqp(struct ib_qp *ibqp)
267+{
268+ return container_of(ibqp, struct pvrdma_qp, ibqp);
269+}
270+
271+static inline struct pvrdma_ah *to_vah(struct ib_ah *ibah)
272+{
273+ return container_of(ibah, struct pvrdma_ah, ibah);
274+}
275+
276+static inline void pvrdma_write_reg(struct pvrdma_dev *dev, u32 reg, u32 val)
277+{
278+ writel(cpu_to_le32(val), dev->regs + reg);
279+}
280+
281+static inline u32 pvrdma_read_reg(struct pvrdma_dev *dev, u32 reg)
282+{
283+ return le32_to_cpu(readl(dev->regs + reg));
284+}
285+
286+static inline void pvrdma_write_uar_cq(struct pvrdma_dev *dev, u32 val)
287+{
288+ writel(cpu_to_le32(val), dev->driver_uar.map + PVRDMA_UAR_CQ_OFFSET);
289+}
290+
291+static inline void pvrdma_write_uar_qp(struct pvrdma_dev *dev, u32 val)
292+{
293+ writel(cpu_to_le32(val), dev->driver_uar.map + PVRDMA_UAR_QP_OFFSET);
294+}
295+
296+static inline void *pvrdma_page_dir_get_ptr(struct pvrdma_page_dir *pdir,
297+ u64 offset)
298+{
299+ return pdir->pages[offset / PAGE_SIZE] + (offset % PAGE_SIZE);
300+}
301+
302+static inline enum pvrdma_mtu ib_mtu_to_pvrdma(enum ib_mtu mtu)
303+{
304+ return (enum pvrdma_mtu)mtu;
305+}
306+
307+static inline enum ib_mtu pvrdma_mtu_to_ib(enum pvrdma_mtu mtu)
308+{
309+ return (enum ib_mtu)mtu;
310+}
311+
312+static inline enum pvrdma_port_state ib_port_state_to_pvrdma(
313+ enum ib_port_state state)
314+{
315+ return (enum pvrdma_port_state)state;
316+}
317+
318+static inline enum ib_port_state pvrdma_port_state_to_ib(
319+ enum pvrdma_port_state state)
320+{
321+ return (enum ib_port_state)state;
322+}
323+
324+static inline int ib_port_cap_flags_to_pvrdma(int flags)
325+{
326+ return flags & PVRDMA_MASK(PVRDMA_PORT_CAP_FLAGS_MAX);
327+}
328+
329+static inline int pvrdma_port_cap_flags_to_ib(int flags)
330+{
331+ return flags;
332+}
333+
334+static inline enum pvrdma_port_width ib_port_width_to_pvrdma(
335+ enum ib_port_width width)
336+{
337+ return (enum pvrdma_port_width)width;
338+}
339+
340+static inline enum ib_port_width pvrdma_port_width_to_ib(
341+ enum pvrdma_port_width width)
342+{
343+ return (enum ib_port_width)width;
344+}
345+
346+static inline enum pvrdma_port_speed ib_port_speed_to_pvrdma(
347+ enum ib_port_speed speed)
348+{
349+ return (enum pvrdma_port_speed)speed;
350+}
351+
352+static inline enum ib_port_speed pvrdma_port_speed_to_ib(
353+ enum pvrdma_port_speed speed)
354+{
355+ return (enum ib_port_speed)speed;
356+}
357+
358+static inline int pvrdma_qp_attr_mask_to_ib(int attr_mask)
359+{
360+ return attr_mask;
361+}
362+
363+static inline int ib_qp_attr_mask_to_pvrdma(int attr_mask)
364+{
365+ return attr_mask & PVRDMA_MASK(PVRDMA_QP_ATTR_MASK_MAX);
366+}
367+
368+static inline enum pvrdma_mig_state ib_mig_state_to_pvrdma(
369+ enum ib_mig_state state)
370+{
371+ return (enum pvrdma_mig_state)state;
372+}
373+
374+static inline enum ib_mig_state pvrdma_mig_state_to_ib(
375+ enum pvrdma_mig_state state)
376+{
377+ return (enum ib_mig_state)state;
378+}
379+
380+static inline int ib_access_flags_to_pvrdma(int flags)
381+{
382+ return flags;
383+}
384+
385+static inline int pvrdma_access_flags_to_ib(int flags)
386+{
387+ return flags & PVRDMA_MASK(PVRDMA_ACCESS_FLAGS_MAX);
388+}
389+
390+static inline enum pvrdma_qp_type ib_qp_type_to_pvrdma(enum ib_qp_type type)
391+{
392+ return (enum pvrdma_qp_type)type;
393+}
394+
395+static inline enum ib_qp_type pvrdma_qp_type_to_ib(enum pvrdma_qp_type type)
396+{
397+ return (enum ib_qp_type)type;
398+}
399+
400+static inline enum pvrdma_qp_state ib_qp_state_to_pvrdma(enum ib_qp_state state)
401+{
402+ return (enum pvrdma_qp_state)state;
403+}
404+
405+static inline enum ib_qp_state pvrdma_qp_state_to_ib(enum pvrdma_qp_state state)
406+{
407+ return (enum ib_qp_state)state;
408+}
409+
410+static inline enum pvrdma_wr_opcode ib_wr_opcode_to_pvrdma(enum ib_wr_opcode op)
411+{
412+ return (enum pvrdma_wr_opcode)op;
413+}
414+
415+static inline enum ib_wc_status pvrdma_wc_status_to_ib(
416+ enum pvrdma_wc_status status)
417+{
418+ return (enum ib_wc_status)status;
419+}
420+
421+static inline int pvrdma_wc_opcode_to_ib(int opcode)
422+{
423+ return opcode;
424+}
425+
426+static inline int pvrdma_wc_flags_to_ib(int flags)
427+{
428+ return flags;
429+}
430+
431+static inline int ib_send_flags_to_pvrdma(int flags)
432+{
433+ return flags & PVRDMA_MASK(PVRDMA_SEND_FLAGS_MAX);
434+}
435+
436+void pvrdma_qp_cap_to_ib(struct ib_qp_cap *dst,
437+ const struct pvrdma_qp_cap *src);
438+void ib_qp_cap_to_pvrdma(struct pvrdma_qp_cap *dst,
439+ const struct ib_qp_cap *src);
440+void pvrdma_gid_to_ib(union ib_gid *dst, const union pvrdma_gid *src);
441+void ib_gid_to_pvrdma(union pvrdma_gid *dst, const union ib_gid *src);
442+void pvrdma_global_route_to_ib(struct ib_global_route *dst,
443+ const struct pvrdma_global_route *src);
444+void ib_global_route_to_pvrdma(struct pvrdma_global_route *dst,
445+ const struct ib_global_route *src);
446+void pvrdma_ah_attr_to_ib(struct ib_ah_attr *dst,
447+ const struct pvrdma_ah_attr *src);
448+void ib_ah_attr_to_pvrdma(struct pvrdma_ah_attr *dst,
449+ const struct ib_ah_attr *src);
450+
451+int pvrdma_uar_table_init(struct pvrdma_dev *dev);
452+void pvrdma_uar_table_cleanup(struct pvrdma_dev *dev);
453+
454+int pvrdma_uar_alloc(struct pvrdma_dev *dev, struct pvrdma_uar_map *uar);
455+void pvrdma_uar_free(struct pvrdma_dev *dev, struct pvrdma_uar_map *uar);
456+
457+void _pvrdma_flush_cqe(struct pvrdma_qp *qp, struct pvrdma_cq *cq);
458+
459+int pvrdma_page_dir_init(struct pvrdma_dev *dev, struct pvrdma_page_dir *pdir,
460+ u64 npages, bool alloc_pages);
461+void pvrdma_page_dir_cleanup(struct pvrdma_dev *dev,
462+ struct pvrdma_page_dir *pdir);
463+int pvrdma_page_dir_insert_dma(struct pvrdma_page_dir *pdir, u64 idx,
464+ dma_addr_t daddr);
465+int pvrdma_page_dir_insert_umem(struct pvrdma_page_dir *pdir,
466+ struct ib_umem *umem, u64 offset);
467+dma_addr_t pvrdma_page_dir_get_dma(struct pvrdma_page_dir *pdir, u64 idx);
468+int pvrdma_page_dir_insert_page_list(struct pvrdma_page_dir *pdir,
469+ u64 *page_list, int num_pages);
470+
471+int pvrdma_cmd_post(struct pvrdma_dev *dev, union pvrdma_cmd_req *req,
472+ union pvrdma_cmd_resp *rsp, unsigned resp_code);
473+
474+#endif /* __PVRDMA_H__ */
drivers/infiniband/hw/vmw_pvrdma/pvrdma_cmd.c+119−0
@@ -0,0 +1,119 @@
1+/*
2+ * Copyright (c) 2012-2016 VMware, Inc. All rights reserved.
3+ *
4+ * This program is free software; you can redistribute it and/or
5+ * modify it under the terms of EITHER the GNU General Public License
6+ * version 2 as published by the Free Software Foundation or the BSD
7+ * 2-Clause License. This program is distributed in the hope that it
8+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
9+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
10+ * See the GNU General Public License version 2 for more details at
11+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
12+ *
13+ * You should have received a copy of the GNU General Public License
14+ * along with this program available in the file COPYING in the main
15+ * directory of this source tree.
16+ *
17+ * The BSD 2-Clause License
18+ *
19+ * Redistribution and use in source and binary forms, with or
20+ * without modification, are permitted provided that the following
21+ * conditions are met:
22+ *
23+ * - Redistributions of source code must retain the above
24+ * copyright notice, this list of conditions and the following
25+ * disclaimer.
26+ *
27+ * - Redistributions in binary form must reproduce the above
28+ * copyright notice, this list of conditions and the following
29+ * disclaimer in the documentation and/or other materials
30+ * provided with the distribution.
31+ *
32+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
33+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
34+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
35+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
36+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
37+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
38+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
39+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
40+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
41+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
42+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
43+ * OF THE POSSIBILITY OF SUCH DAMAGE.
44+ */
45+
46+#include <linux/list.h>
47+
48+#include "pvrdma.h"
49+
50+#define PVRDMA_CMD_TIMEOUT 10000 /* ms */
51+
52+static inline int pvrdma_cmd_recv(struct pvrdma_dev *dev,
53+ union pvrdma_cmd_resp *resp,
54+ unsigned resp_code)
55+{
56+ int err;
57+
58+ dev_dbg(&dev->pdev->dev, "receive response from device\n");
59+
60+ err = wait_for_completion_interruptible_timeout(&dev->cmd_done,
61+ msecs_to_jiffies(PVRDMA_CMD_TIMEOUT));
62+ if (err == 0 || err == -ERESTARTSYS) {
63+ dev_warn(&dev->pdev->dev,
64+ "completion timeout or interrupted\n");
65+ return -ETIMEDOUT;
66+ }
67+
68+ spin_lock(&dev->cmd_lock);
69+ memcpy(resp, dev->resp_slot, sizeof(*resp));
70+ spin_unlock(&dev->cmd_lock);
71+
72+ if (resp->hdr.ack != resp_code) {
73+ dev_warn(&dev->pdev->dev,
74+ "unknown response %#x expected %#x\n",
75+ resp->hdr.ack, resp_code);
76+ return -EFAULT;
77+ }
78+
79+ return 0;
80+}
81+
82+int
83+pvrdma_cmd_post(struct pvrdma_dev *dev, union pvrdma_cmd_req *req,
84+ union pvrdma_cmd_resp *resp, unsigned resp_code)
85+{
86+ int err;
87+
88+ dev_dbg(&dev->pdev->dev, "post request to device\n");
89+
90+ /* Serializiation */
91+ down(&dev->cmd_sema);
92+
93+ BUILD_BUG_ON(sizeof(union pvrdma_cmd_req) !=
94+ sizeof(struct pvrdma_cmd_modify_qp));
95+
96+ spin_lock(&dev->cmd_lock);
97+ memcpy(dev->cmd_slot, req, sizeof(*req));
98+ spin_unlock(&dev->cmd_lock);
99+
100+ init_completion(&dev->cmd_done);
101+ pvrdma_write_reg(dev, PVRDMA_REG_REQUEST, 0);
102+
103+ /* Make sure the request is written before reading status. */
104+ mb();
105+
106+ err = pvrdma_read_reg(dev, PVRDMA_REG_ERR);
107+ if (err == 0) {
108+ if (resp != NULL)
109+ err = pvrdma_cmd_recv(dev, resp, resp_code);
110+ } else {
111+ dev_warn(&dev->pdev->dev,
112+ "failed to write request error reg: %d\n", err);
113+ err = -EFAULT;
114+ }
115+
116+ up(&dev->cmd_sema);
117+
118+ return err;
119+}
drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c+425−0
@@ -0,0 +1,425 @@
1+/*
2+ * Copyright (c) 2012-2016 VMware, Inc. All rights reserved.
3+ *
4+ * This program is free software; you can redistribute it and/or
5+ * modify it under the terms of EITHER the GNU General Public License
6+ * version 2 as published by the Free Software Foundation or the BSD
7+ * 2-Clause License. This program is distributed in the hope that it
8+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
9+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
10+ * See the GNU General Public License version 2 for more details at
11+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
12+ *
13+ * You should have received a copy of the GNU General Public License
14+ * along with this program available in the file COPYING in the main
15+ * directory of this source tree.
16+ *
17+ * The BSD 2-Clause License
18+ *
19+ * Redistribution and use in source and binary forms, with or
20+ * without modification, are permitted provided that the following
21+ * conditions are met:
22+ *
23+ * - Redistributions of source code must retain the above
24+ * copyright notice, this list of conditions and the following
25+ * disclaimer.
26+ *
27+ * - Redistributions in binary form must reproduce the above
28+ * copyright notice, this list of conditions and the following
29+ * disclaimer in the documentation and/or other materials
30+ * provided with the distribution.
31+ *
32+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
33+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
34+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
35+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
36+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
37+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
38+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
39+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
40+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
41+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
42+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
43+ * OF THE POSSIBILITY OF SUCH DAMAGE.
44+ */
45+
46+#include <asm/page.h>
47+#include <linux/io.h>
48+#include <linux/wait.h>
49+#include <rdma/ib_addr.h>
50+#include <rdma/ib_smi.h>
51+#include <rdma/ib_user_verbs.h>
52+
53+#include "pvrdma.h"
54+
55+/**
56+ * pvrdma_req_notify_cq - request notification for a completion queue
57+ * @ibcq: the completion queue
58+ * @notify_flags: notification flags
59+ *
60+ * @return: 0 for success.
61+ */
62+int pvrdma_req_notify_cq(struct ib_cq *ibcq,
63+ enum ib_cq_notify_flags notify_flags)
64+{
65+ struct pvrdma_dev *dev = to_vdev(ibcq->device);
66+ struct pvrdma_cq *cq = to_vcq(ibcq);
67+ u32 val = cq->cq_handle;
68+
69+ val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ?
70+ PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM;
71+
72+ pvrdma_write_uar_cq(dev, val);
73+
74+ return 0;
75+}
76+
77+/**
78+ * pvrdma_create_cq - create completion queue
79+ * @ibdev: the device
80+ * @attr: completion queue attributes
81+ * @context: user context
82+ * @udata: user data
83+ *
84+ * @return: ib_cq completion queue pointer on success,
85+ * otherwise returns negative errno.
86+ */
87+struct ib_cq *pvrdma_create_cq(struct ib_device *ibdev,
88+ const struct ib_cq_init_attr *attr,
89+ struct ib_ucontext *context,
90+ struct ib_udata *udata)
91+{
92+ int entries = attr->cqe;
93+ struct pvrdma_dev *dev = to_vdev(ibdev);
94+ struct pvrdma_cq *cq;
95+ int ret;
96+ int npages;
97+ unsigned long flags;
98+ union pvrdma_cmd_req req;
99+ union pvrdma_cmd_resp rsp;
100+ struct pvrdma_cmd_create_cq *cmd = &req.create_cq;
101+ struct pvrdma_cmd_create_cq_resp *resp = &rsp.create_cq_resp;
102+ struct pvrdma_create_cq ucmd;
103+
104+ BUILD_BUG_ON(sizeof(struct pvrdma_cqe) != 64);
105+
106+ entries = roundup_pow_of_two(entries);
107+ if (entries < 1 || entries > dev->dsr->caps.max_cqe)
108+ return ERR_PTR(-EINVAL);
109+
110+ if (!atomic_add_unless(&dev->num_cqs, 1, dev->dsr->caps.max_cq))
111+ return ERR_PTR(-ENOMEM);
112+
113+ cq = kzalloc(sizeof(*cq), GFP_KERNEL);
114+ if (!cq) {
115+ atomic_dec(&dev->num_cqs);
116+ return ERR_PTR(-ENOMEM);
117+ }
118+
119+ cq->ibcq.cqe = entries;
120+
121+ if (context) {
122+ if (ib_copy_from_udata(&ucmd, udata, sizeof(ucmd))) {
123+ ret = -EFAULT;
124+ goto err_cq;
125+ }
126+
127+ cq->umem = ib_umem_get(context, ucmd.buf_addr, ucmd.buf_size,
128+ IB_ACCESS_LOCAL_WRITE, 1);
129+ if (IS_ERR(cq->umem)) {
130+ ret = PTR_ERR(cq->umem);
131+ goto err_cq;
132+ }
133+
134+ npages = ib_umem_page_count(cq->umem);
135+ } else {
136+ cq->is_kernel = true;
137+
138+ /* One extra page for shared ring state */
139+ npages = 1 + (entries * sizeof(struct pvrdma_cqe) +
140+ PAGE_SIZE - 1) / PAGE_SIZE;
141+
142+ /* Skip header page. */
143+ cq->offset = PAGE_SIZE;
144+ }
145+
146+ if (npages < 0 || npages > PVRDMA_PAGE_DIR_MAX_PAGES) {
147+ dev_warn(&dev->pdev->dev,
148+ "overflow pages in completion queue\n");
149+ ret = -EINVAL;
150+ goto err_umem;
151+ }
152+
153+ ret = pvrdma_page_dir_init(dev, &cq->pdir, npages, cq->is_kernel);
154+ if (ret) {
155+ dev_warn(&dev->pdev->dev,
156+ "could not allocate page directory\n");
157+ goto err_umem;
158+ }
159+
160+ /* Ring state is always the first page. Set in library for user cq. */
161+ if (cq->is_kernel)
162+ cq->ring_state = cq->pdir.pages[0];
163+ else
164+ pvrdma_page_dir_insert_umem(&cq->pdir, cq->umem, 0);
165+
166+ atomic_set(&cq->refcnt, 1);
167+ init_waitqueue_head(&cq->wait);
168+ spin_lock_init(&cq->cq_lock);
169+
170+ memset(cmd, 0, sizeof(*cmd));
171+ cmd->hdr.cmd = PVRDMA_CMD_CREATE_CQ;
172+ cmd->nchunks = npages;
173+ cmd->ctx_handle = (context) ?
174+ (u64)to_vucontext(context)->ctx_handle : 0;
175+ cmd->cqe = entries;
176+ cmd->pdir_dma = cq->pdir.dir_dma;
177+ ret = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_CREATE_CQ_RESP);
178+ if (ret < 0) {
179+ dev_warn(&dev->pdev->dev,
180+ "could not create completion queue, error: %d\n", ret);
181+ goto err_page_dir;
182+ }
183+
184+ cq->ibcq.cqe = resp->cqe;
185+ cq->cq_handle = resp->cq_handle;
186+ spin_lock_irqsave(&dev->cq_tbl_lock, flags);
187+ dev->cq_tbl[cq->cq_handle % dev->dsr->caps.max_cq] = cq;
188+ spin_unlock_irqrestore(&dev->cq_tbl_lock, flags);
189+
190+ if (context) {
191+ cq->uar = &(to_vucontext(context)->uar);
192+
193+ /* Copy udata back. */
194+ if (ib_copy_to_udata(udata, &cq->cq_handle, sizeof(__u32))) {
195+ dev_warn(&dev->pdev->dev,
196+ "failed to copy back udata\n");
197+ pvrdma_destroy_cq(&cq->ibcq);
198+ return ERR_PTR(-EINVAL);
199+ }
200+ }
201+
202+ return &cq->ibcq;
203+
204+err_page_dir:
205+ pvrdma_page_dir_cleanup(dev, &cq->pdir);
206+err_umem:
207+ if (context)
208+ ib_umem_release(cq->umem);
209+err_cq:
210+ atomic_dec(&dev->num_cqs);
211+ kfree(cq);
212+
213+ return ERR_PTR(ret);
214+}
215+
216+static void pvrdma_free_cq(struct pvrdma_dev *dev, struct pvrdma_cq *cq)
217+{
218+ atomic_dec(&cq->refcnt);
219+ wait_event(cq->wait, !atomic_read(&cq->refcnt));
220+
221+ if (!cq->is_kernel)
222+ ib_umem_release(cq->umem);
223+
224+ pvrdma_page_dir_cleanup(dev, &cq->pdir);
225+ kfree(cq);
226+}
227+
228+/**
229+ * pvrdma_destroy_cq - destroy completion queue
230+ * @cq: the completion queue to destroy.
231+ *
232+ * @return: 0 for success.
233+ */
234+int pvrdma_destroy_cq(struct ib_cq *cq)
235+{
236+ struct pvrdma_cq *vcq = to_vcq(cq);
237+ union pvrdma_cmd_req req;
238+ struct pvrdma_cmd_destroy_cq *cmd = &req.destroy_cq;
239+ struct pvrdma_dev *dev = to_vdev(cq->device);
240+ unsigned long flags;
241+ int ret;
242+
243+ memset(cmd, 0, sizeof(*cmd));
244+ cmd->hdr.cmd = PVRDMA_CMD_DESTROY_CQ;
245+ cmd->cq_handle = vcq->cq_handle;
246+
247+ ret = pvrdma_cmd_post(dev, &req, NULL, 0);
248+ if (ret < 0)
249+ dev_warn(&dev->pdev->dev,
250+ "could not destroy completion queue, error: %d\n",
251+ ret);
252+
253+ /* free cq's resources */
254+ spin_lock_irqsave(&dev->cq_tbl_lock, flags);
255+ dev->cq_tbl[vcq->cq_handle] = NULL;
256+ spin_unlock_irqrestore(&dev->cq_tbl_lock, flags);
257+
258+ pvrdma_free_cq(dev, vcq);
259+ atomic_dec(&dev->num_cqs);
260+
261+ return ret;
262+}
263+
264+/**
265+ * pvrdma_modify_cq - modify the CQ moderation parameters
266+ * @ibcq: the CQ to modify
267+ * @cq_count: number of CQEs that will trigger an event
268+ * @cq_period: max period of time in usec before triggering an event
269+ *
270+ * @return: -EOPNOTSUPP as CQ resize is not supported.
271+ */
272+int pvrdma_modify_cq(struct ib_cq *cq, u16 cq_count, u16 cq_period)
273+{
274+ return -EOPNOTSUPP;
275+}
276+
277+static inline struct pvrdma_cqe *get_cqe(struct pvrdma_cq *cq, int i)
278+{
279+ return (struct pvrdma_cqe *)pvrdma_page_dir_get_ptr(
280+ &cq->pdir,
281+ cq->offset +
282+ sizeof(struct pvrdma_cqe) * i);
283+}
284+
285+void _pvrdma_flush_cqe(struct pvrdma_qp *qp, struct pvrdma_cq *cq)
286+{
287+ int head;
288+ int has_data;
289+
290+ if (!cq->is_kernel)
291+ return;
292+
293+ /* Lock held */
294+ has_data = pvrdma_idx_ring_has_data(&cq->ring_state->rx,
295+ cq->ibcq.cqe, &head);
296+ if (unlikely(has_data > 0)) {
297+ int items;
298+ int curr;
299+ int tail = pvrdma_idx(&cq->ring_state->rx.prod_tail,
300+ cq->ibcq.cqe);
301+ struct pvrdma_cqe *cqe;
302+ struct pvrdma_cqe *curr_cqe;
303+
304+ items = (tail > head) ? (tail - head) :
305+ (cq->ibcq.cqe - head + tail);
306+ curr = --tail;
307+ while (items-- > 0) {
308+ if (curr < 0)
309+ curr = cq->ibcq.cqe - 1;
310+ if (tail < 0)
311+ tail = cq->ibcq.cqe - 1;
312+ curr_cqe = get_cqe(cq, curr);
313+ if ((curr_cqe->qp & 0xFFFF) != qp->qp_handle) {
314+ if (curr != tail) {
315+ cqe = get_cqe(cq, tail);
316+ *cqe = *curr_cqe;
317+ }
318+ tail--;
319+ } else {
320+ pvrdma_idx_ring_inc(
321+ &cq->ring_state->rx.cons_head,
322+ cq->ibcq.cqe);
323+ }
324+ curr--;
325+ }
326+ }
327+}
328+
329+static int pvrdma_poll_one(struct pvrdma_cq *cq, struct pvrdma_qp **cur_qp,
330+ struct ib_wc *wc)
331+{
332+ struct pvrdma_dev *dev = to_vdev(cq->ibcq.device);
333+ int has_data;
334+ unsigned int head;
335+ bool tried = false;
336+ struct pvrdma_cqe *cqe;
337+
338+retry:
339+ has_data = pvrdma_idx_ring_has_data(&cq->ring_state->rx,
340+ cq->ibcq.cqe, &head);
341+ if (has_data == 0) {
342+ if (tried)
343+ return -EAGAIN;
344+
345+ pvrdma_write_uar_cq(dev, cq->cq_handle | PVRDMA_UAR_CQ_POLL);
346+
347+ tried = true;
348+ goto retry;
349+ } else if (has_data == PVRDMA_INVALID_IDX) {
350+ dev_err(&dev->pdev->dev, "CQ ring state invalid\n");
351+ return -EAGAIN;
352+ }
353+
354+ cqe = get_cqe(cq, head);
355+
356+ /* Ensure cqe is valid. */
357+ rmb();
358+ if (dev->qp_tbl[cqe->qp & 0xffff])
359+ *cur_qp = (struct pvrdma_qp *)dev->qp_tbl[cqe->qp & 0xffff];
360+ else
361+ return -EAGAIN;
362+
363+ wc->opcode = pvrdma_wc_opcode_to_ib(cqe->opcode);
364+ wc->status = pvrdma_wc_status_to_ib(cqe->status);
365+ wc->wr_id = cqe->wr_id;
366+ wc->qp = &(*cur_qp)->ibqp;
367+ wc->byte_len = cqe->byte_len;
368+ wc->ex.imm_data = cqe->imm_data;
369+ wc->src_qp = cqe->src_qp;
370+ wc->wc_flags = pvrdma_wc_flags_to_ib(cqe->wc_flags);
371+ wc->pkey_index = cqe->pkey_index;
372+ wc->slid = cqe->slid;
373+ wc->sl = cqe->sl;
374+ wc->dlid_path_bits = cqe->dlid_path_bits;
375+ wc->port_num = cqe->port_num;
376+ wc->vendor_err = 0;
377+
378+ /* Update shared ring state */
379+ pvrdma_idx_ring_inc(&cq->ring_state->rx.cons_head, cq->ibcq.cqe);
380+
381+ return 0;
382+}
383+
384+/**
385+ * pvrdma_poll_cq - poll for work completion queue entries
386+ * @ibcq: completion queue
387+ * @num_entries: the maximum number of entries
388+ * @entry: pointer to work completion array
389+ *
390+ * @return: number of polled completion entries
391+ */
392+int pvrdma_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
393+{
394+ struct pvrdma_cq *cq = to_vcq(ibcq);
395+ struct pvrdma_qp *cur_qp = NULL;
396+ unsigned long flags;
397+ int npolled;
398+
399+ if (num_entries < 1 || wc == NULL)
400+ return 0;
401+
402+ spin_lock_irqsave(&cq->cq_lock, flags);
403+ for (npolled = 0; npolled < num_entries; ++npolled) {
404+ if (pvrdma_poll_one(cq, &cur_qp, wc + npolled))
405+ break;
406+ }
407+
408+ spin_unlock_irqrestore(&cq->cq_lock, flags);
409+
410+ /* Ensure we do not return errors from poll_cq */
411+ return npolled;
412+}
413+
414+/**
415+ * pvrdma_resize_cq - resize CQ
416+ * @ibcq: the completion queue
417+ * @entries: CQ entries
418+ * @udata: user data
419+ *
420+ * @return: -EOPNOTSUPP as CQ resize is not supported.
421+ */
422+int pvrdma_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata)
423+{
424+ return -EOPNOTSUPP;
425+}
drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h+586−0
@@ -0,0 +1,586 @@
1+/*
2+ * Copyright (c) 2012-2016 VMware, Inc. All rights reserved.
3+ *
4+ * This program is free software; you can redistribute it and/or
5+ * modify it under the terms of EITHER the GNU General Public License
6+ * version 2 as published by the Free Software Foundation or the BSD
7+ * 2-Clause License. This program is distributed in the hope that it
8+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
9+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
10+ * See the GNU General Public License version 2 for more details at
11+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
12+ *
13+ * You should have received a copy of the GNU General Public License
14+ * along with this program available in the file COPYING in the main
15+ * directory of this source tree.
16+ *
17+ * The BSD 2-Clause License
18+ *
19+ * Redistribution and use in source and binary forms, with or
20+ * without modification, are permitted provided that the following
21+ * conditions are met:
22+ *
23+ * - Redistributions of source code must retain the above
24+ * copyright notice, this list of conditions and the following
25+ * disclaimer.
26+ *
27+ * - Redistributions in binary form must reproduce the above
28+ * copyright notice, this list of conditions and the following
29+ * disclaimer in the documentation and/or other materials
30+ * provided with the distribution.
31+ *
32+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
33+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
34+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
35+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
36+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
37+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
38+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
39+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
40+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
41+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
42+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
43+ * OF THE POSSIBILITY OF SUCH DAMAGE.
44+ */
45+
46+#ifndef __PVRDMA_DEV_API_H__
47+#define __PVRDMA_DEV_API_H__
48+
49+#include <linux/types.h>
50+
51+#include "pvrdma_verbs.h"
52+
53+#define PVRDMA_VERSION 17
54+#define PVRDMA_BOARD_ID 1
55+#define PVRDMA_REV_ID 1
56+
57+/*
58+ * Masks and accessors for page directory, which is a two-level lookup:
59+ * page directory -> page table -> page. Only one directory for now, but we
60+ * could expand that easily. 9 bits for tables, 9 bits for pages, gives one
61+ * gigabyte for memory regions and so forth.
62+ */
63+
64+#define PVRDMA_PDIR_SHIFT 18
65+#define PVRDMA_PTABLE_SHIFT 9
66+#define PVRDMA_PAGE_DIR_DIR(x) (((x) >> PVRDMA_PDIR_SHIFT) & 0x1)
67+#define PVRDMA_PAGE_DIR_TABLE(x) (((x) >> PVRDMA_PTABLE_SHIFT) & 0x1ff)
68+#define PVRDMA_PAGE_DIR_PAGE(x) ((x) & 0x1ff)
69+#define PVRDMA_PAGE_DIR_MAX_PAGES (1 * 512 * 512)
70+#define PVRDMA_MAX_FAST_REG_PAGES 128
71+
72+/*
73+ * Max MSI-X vectors.
74+ */
75+
76+#define PVRDMA_MAX_INTERRUPTS 3
77+
78+/* Register offsets within PCI resource on BAR1. */
79+#define PVRDMA_REG_VERSION 0x00 /* R: Version of device. */
80+#define PVRDMA_REG_DSRLOW 0x04 /* W: Device shared region low PA. */
81+#define PVRDMA_REG_DSRHIGH 0x08 /* W: Device shared region high PA. */
82+#define PVRDMA_REG_CTL 0x0c /* W: PVRDMA_DEVICE_CTL */
83+#define PVRDMA_REG_REQUEST 0x10 /* W: Indicate device request. */
84+#define PVRDMA_REG_ERR 0x14 /* R: Device error. */
85+#define PVRDMA_REG_ICR 0x18 /* R: Interrupt cause. */
86+#define PVRDMA_REG_IMR 0x1c /* R/W: Interrupt mask. */
87+#define PVRDMA_REG_MACL 0x20 /* R/W: MAC address low. */
88+#define PVRDMA_REG_MACH 0x24 /* R/W: MAC address high. */
89+
90+/* Object flags. */
91+#define PVRDMA_CQ_FLAG_ARMED_SOL BIT(0) /* Armed for solicited-only. */
92+#define PVRDMA_CQ_FLAG_ARMED BIT(1) /* Armed. */
93+#define PVRDMA_MR_FLAG_DMA BIT(0) /* DMA region. */
94+#define PVRDMA_MR_FLAG_FRMR BIT(1) /* Fast reg memory region. */
95+
96+/*
97+ * Atomic operation capability (masked versions are extended atomic
98+ * operations.
99+ */
100+
101+#define PVRDMA_ATOMIC_OP_COMP_SWAP BIT(0) /* Compare and swap. */
102+#define PVRDMA_ATOMIC_OP_FETCH_ADD BIT(1) /* Fetch and add. */
103+#define PVRDMA_ATOMIC_OP_MASK_COMP_SWAP BIT(2) /* Masked compare and swap. */
104+#define PVRDMA_ATOMIC_OP_MASK_FETCH_ADD BIT(3) /* Masked fetch and add. */
105+
106+/*
107+ * Base Memory Management Extension flags to support Fast Reg Memory Regions
108+ * and Fast Reg Work Requests. Each flag represents a verb operation and we
109+ * must support all of them to qualify for the BMME device cap.
110+ */
111+
112+#define PVRDMA_BMME_FLAG_LOCAL_INV BIT(0) /* Local Invalidate. */
113+#define PVRDMA_BMME_FLAG_REMOTE_INV BIT(1) /* Remote Invalidate. */
114+#define PVRDMA_BMME_FLAG_FAST_REG_WR BIT(2) /* Fast Reg Work Request. */
115+
116+/*
117+ * GID types. The interpretation of the gid_types bit field in the device
118+ * capabilities will depend on the device mode. For now, the device only
119+ * supports RoCE as mode, so only the different GID types for RoCE are
120+ * defined.
121+ */
122+
123+#define PVRDMA_GID_TYPE_FLAG_ROCE_V1 BIT(0)
124+#define PVRDMA_GID_TYPE_FLAG_ROCE_V2 BIT(1)
125+
126+enum pvrdma_pci_resource {
127+ PVRDMA_PCI_RESOURCE_MSIX, /* BAR0: MSI-X, MMIO. */
128+ PVRDMA_PCI_RESOURCE_REG, /* BAR1: Registers, MMIO. */
129+ PVRDMA_PCI_RESOURCE_UAR, /* BAR2: UAR pages, MMIO, 64-bit. */
130+ PVRDMA_PCI_RESOURCE_LAST, /* Last. */
131+};
132+
133+enum pvrdma_device_ctl {
134+ PVRDMA_DEVICE_CTL_ACTIVATE, /* Activate device. */
135+ PVRDMA_DEVICE_CTL_QUIESCE, /* Quiesce device. */
136+ PVRDMA_DEVICE_CTL_RESET, /* Reset device. */
137+};
138+
139+enum pvrdma_intr_vector {
140+ PVRDMA_INTR_VECTOR_RESPONSE, /* Command response. */
141+ PVRDMA_INTR_VECTOR_ASYNC, /* Async events. */
142+ PVRDMA_INTR_VECTOR_CQ, /* CQ notification. */
143+ /* Additional CQ notification vectors. */
144+};
145+
146+enum pvrdma_intr_cause {
147+ PVRDMA_INTR_CAUSE_RESPONSE = (1 << PVRDMA_INTR_VECTOR_RESPONSE),
148+ PVRDMA_INTR_CAUSE_ASYNC = (1 << PVRDMA_INTR_VECTOR_ASYNC),
149+ PVRDMA_INTR_CAUSE_CQ = (1 << PVRDMA_INTR_VECTOR_CQ),
150+};
151+
152+enum pvrdma_intr_type {
153+ PVRDMA_INTR_TYPE_INTX, /* Legacy. */
154+ PVRDMA_INTR_TYPE_MSI, /* MSI. */
155+ PVRDMA_INTR_TYPE_MSIX, /* MSI-X. */
156+};
157+
158+enum pvrdma_gos_bits {
159+ PVRDMA_GOS_BITS_UNK, /* Unknown. */
160+ PVRDMA_GOS_BITS_32, /* 32-bit. */
161+ PVRDMA_GOS_BITS_64, /* 64-bit. */
162+};
163+
164+enum pvrdma_gos_type {
165+ PVRDMA_GOS_TYPE_UNK, /* Unknown. */
166+ PVRDMA_GOS_TYPE_LINUX, /* Linux. */
167+};
168+
169+enum pvrdma_device_mode {
170+ PVRDMA_DEVICE_MODE_ROCE, /* RoCE. */
171+ PVRDMA_DEVICE_MODE_IWARP, /* iWarp. */
172+ PVRDMA_DEVICE_MODE_IB, /* InfiniBand. */
173+};
174+
175+struct pvrdma_gos_info {
176+ u32 gos_bits:2; /* W: PVRDMA_GOS_BITS_ */
177+ u32 gos_type:4; /* W: PVRDMA_GOS_TYPE_ */
178+ u32 gos_ver:16; /* W: Guest OS version. */
179+ u32 gos_misc:10; /* W: Other. */
180+ u32 pad; /* Pad to 8-byte alignment. */
181+};
182+
183+struct pvrdma_device_caps {
184+ u64 fw_ver; /* R: Query device. */
185+ __be64 node_guid;
186+ __be64 sys_image_guid;
187+ u64 max_mr_size;
188+ u64 page_size_cap;
189+ u64 atomic_arg_sizes; /* EX verbs. */
190+ u32 ex_comp_mask; /* EX verbs. */
191+ u32 device_cap_flags2; /* EX verbs. */
192+ u32 max_fa_bit_boundary; /* EX verbs. */
193+ u32 log_max_atomic_inline_arg; /* EX verbs. */
194+ u32 vendor_id;
195+ u32 vendor_part_id;
196+ u32 hw_ver;
197+ u32 max_qp;
198+ u32 max_qp_wr;
199+ u32 device_cap_flags;
200+ u32 max_sge;
201+ u32 max_sge_rd;
202+ u32 max_cq;
203+ u32 max_cqe;
204+ u32 max_mr;
205+ u32 max_pd;
206+ u32 max_qp_rd_atom;
207+ u32 max_ee_rd_atom;
208+ u32 max_res_rd_atom;
209+ u32 max_qp_init_rd_atom;
210+ u32 max_ee_init_rd_atom;
211+ u32 max_ee;
212+ u32 max_rdd;
213+ u32 max_mw;
214+ u32 max_raw_ipv6_qp;
215+ u32 max_raw_ethy_qp;
216+ u32 max_mcast_grp;
217+ u32 max_mcast_qp_attach;
218+ u32 max_total_mcast_qp_attach;
219+ u32 max_ah;
220+ u32 max_fmr;
221+ u32 max_map_per_fmr;
222+ u32 max_srq;
223+ u32 max_srq_wr;
224+ u32 max_srq_sge;
225+ u32 max_uar;
226+ u32 gid_tbl_len;
227+ u16 max_pkeys;
228+ u8 local_ca_ack_delay;
229+ u8 phys_port_cnt;
230+ u8 mode; /* PVRDMA_DEVICE_MODE_ */
231+ u8 atomic_ops; /* PVRDMA_ATOMIC_OP_* bits */
232+ u8 bmme_flags; /* FRWR Mem Mgmt Extensions */
233+ u8 gid_types; /* PVRDMA_GID_TYPE_FLAG_ */
234+ u8 reserved[4];
235+};
236+
237+struct pvrdma_ring_page_info {
238+ u32 num_pages; /* Num pages incl. header. */
239+ u32 reserved; /* Reserved. */
240+ u64 pdir_dma; /* Page directory PA. */
241+};
242+
243+#pragma pack(push, 1)
244+
245+struct pvrdma_device_shared_region {
246+ u32 driver_version; /* W: Driver version. */
247+ u32 pad; /* Pad to 8-byte align. */
248+ struct pvrdma_gos_info gos_info; /* W: Guest OS information. */
249+ u64 cmd_slot_dma; /* W: Command slot address. */
250+ u64 resp_slot_dma; /* W: Response slot address. */
251+ struct pvrdma_ring_page_info async_ring_pages;
252+ /* W: Async ring page info. */
253+ struct pvrdma_ring_page_info cq_ring_pages;
254+ /* W: CQ ring page info. */
255+ u32 uar_pfn; /* W: UAR pageframe. */
256+ u32 pad2; /* Pad to 8-byte align. */
257+ struct pvrdma_device_caps caps; /* R: Device capabilities. */
258+};
259+
260+#pragma pack(pop)
261+
262+/* Event types. Currently a 1:1 mapping with enum ib_event. */
263+enum pvrdma_eqe_type {
264+ PVRDMA_EVENT_CQ_ERR,
265+ PVRDMA_EVENT_QP_FATAL,
266+ PVRDMA_EVENT_QP_REQ_ERR,
267+ PVRDMA_EVENT_QP_ACCESS_ERR,
268+ PVRDMA_EVENT_COMM_EST,
269+ PVRDMA_EVENT_SQ_DRAINED,
270+ PVRDMA_EVENT_PATH_MIG,
271+ PVRDMA_EVENT_PATH_MIG_ERR,
272+ PVRDMA_EVENT_DEVICE_FATAL,
273+ PVRDMA_EVENT_PORT_ACTIVE,
274+ PVRDMA_EVENT_PORT_ERR,
275+ PVRDMA_EVENT_LID_CHANGE,
276+ PVRDMA_EVENT_PKEY_CHANGE,
277+ PVRDMA_EVENT_SM_CHANGE,
278+ PVRDMA_EVENT_SRQ_ERR,
279+ PVRDMA_EVENT_SRQ_LIMIT_REACHED,
280+ PVRDMA_EVENT_QP_LAST_WQE_REACHED,
281+ PVRDMA_EVENT_CLIENT_REREGISTER,
282+ PVRDMA_EVENT_GID_CHANGE,
283+};
284+
285+/* Event queue element. */
286+struct pvrdma_eqe {
287+ u32 type; /* Event type. */
288+ u32 info; /* Handle, other. */
289+};
290+
291+/* CQ notification queue element. */
292+struct pvrdma_cqne {
293+ u32 info; /* Handle */
294+};
295+
296+enum {
297+ PVRDMA_CMD_FIRST,
298+ PVRDMA_CMD_QUERY_PORT = PVRDMA_CMD_FIRST,
299+ PVRDMA_CMD_QUERY_PKEY,
300+ PVRDMA_CMD_CREATE_PD,
301+ PVRDMA_CMD_DESTROY_PD,
302+ PVRDMA_CMD_CREATE_MR,
303+ PVRDMA_CMD_DESTROY_MR,
304+ PVRDMA_CMD_CREATE_CQ,
305+ PVRDMA_CMD_RESIZE_CQ,
306+ PVRDMA_CMD_DESTROY_CQ,
307+ PVRDMA_CMD_CREATE_QP,
308+ PVRDMA_CMD_MODIFY_QP,
309+ PVRDMA_CMD_QUERY_QP,
310+ PVRDMA_CMD_DESTROY_QP,
311+ PVRDMA_CMD_CREATE_UC,
312+ PVRDMA_CMD_DESTROY_UC,
313+ PVRDMA_CMD_CREATE_BIND,
314+ PVRDMA_CMD_DESTROY_BIND,
315+ PVRDMA_CMD_MAX,
316+};
317+
318+enum {
319+ PVRDMA_CMD_FIRST_RESP = (1 << 31),
320+ PVRDMA_CMD_QUERY_PORT_RESP = PVRDMA_CMD_FIRST_RESP,
321+ PVRDMA_CMD_QUERY_PKEY_RESP,
322+ PVRDMA_CMD_CREATE_PD_RESP,
323+ PVRDMA_CMD_DESTROY_PD_RESP_NOOP,
324+ PVRDMA_CMD_CREATE_MR_RESP,
325+ PVRDMA_CMD_DESTROY_MR_RESP_NOOP,
326+ PVRDMA_CMD_CREATE_CQ_RESP,
327+ PVRDMA_CMD_RESIZE_CQ_RESP,
328+ PVRDMA_CMD_DESTROY_CQ_RESP_NOOP,
329+ PVRDMA_CMD_CREATE_QP_RESP,
330+ PVRDMA_CMD_MODIFY_QP_RESP,
331+ PVRDMA_CMD_QUERY_QP_RESP,
332+ PVRDMA_CMD_DESTROY_QP_RESP,
333+ PVRDMA_CMD_CREATE_UC_RESP,
334+ PVRDMA_CMD_DESTROY_UC_RESP_NOOP,
335+ PVRDMA_CMD_CREATE_BIND_RESP_NOOP,
336+ PVRDMA_CMD_DESTROY_BIND_RESP_NOOP,
337+ PVRDMA_CMD_MAX_RESP,
338+};
339+
340+struct pvrdma_cmd_hdr {
341+ u64 response; /* Key for response lookup. */
342+ u32 cmd; /* PVRDMA_CMD_ */
343+ u32 reserved; /* Reserved. */
344+};
345+
346+struct pvrdma_cmd_resp_hdr {
347+ u64 response; /* From cmd hdr. */
348+ u32 ack; /* PVRDMA_CMD_XXX_RESP */
349+ u8 err; /* Error. */
350+ u8 reserved[3]; /* Reserved. */
351+};
352+
353+struct pvrdma_cmd_query_port {
354+ struct pvrdma_cmd_hdr hdr;
355+ u8 port_num;
356+ u8 reserved[7];
357+};
358+
359+struct pvrdma_cmd_query_port_resp {
360+ struct pvrdma_cmd_resp_hdr hdr;
361+ struct pvrdma_port_attr attrs;
362+};
363+
364+struct pvrdma_cmd_query_pkey {
365+ struct pvrdma_cmd_hdr hdr;
366+ u8 port_num;
367+ u8 index;
368+ u8 reserved[6];
369+};
370+
371+struct pvrdma_cmd_query_pkey_resp {
372+ struct pvrdma_cmd_resp_hdr hdr;
373+ u16 pkey;
374+ u8 reserved[6];
375+};
376+
377+struct pvrdma_cmd_create_uc {
378+ struct pvrdma_cmd_hdr hdr;
379+ u32 pfn; /* UAR page frame number */
380+ u8 reserved[4];
381+};
382+
383+struct pvrdma_cmd_create_uc_resp {
384+ struct pvrdma_cmd_resp_hdr hdr;
385+ u32 ctx_handle;
386+ u8 reserved[4];
387+};
388+
389+struct pvrdma_cmd_destroy_uc {
390+ struct pvrdma_cmd_hdr hdr;
391+ u32 ctx_handle;
392+ u8 reserved[4];
393+};
394+
395+struct pvrdma_cmd_create_pd {
396+ struct pvrdma_cmd_hdr hdr;
397+ u32 ctx_handle;
398+ u8 reserved[4];
399+};
400+
401+struct pvrdma_cmd_create_pd_resp {
402+ struct pvrdma_cmd_resp_hdr hdr;
403+ u32 pd_handle;
404+ u8 reserved[4];
405+};
406+
407+struct pvrdma_cmd_destroy_pd {
408+ struct pvrdma_cmd_hdr hdr;
409+ u32 pd_handle;
410+ u8 reserved[4];
411+};
412+
413+struct pvrdma_cmd_create_mr {
414+ struct pvrdma_cmd_hdr hdr;
415+ u64 start;
416+ u64 length;
417+ u64 pdir_dma;
418+ u32 pd_handle;
419+ u32 access_flags;
420+ u32 flags;
421+ u32 nchunks;
422+};
423+
424+struct pvrdma_cmd_create_mr_resp {
425+ struct pvrdma_cmd_resp_hdr hdr;
426+ u32 mr_handle;
427+ u32 lkey;
428+ u32 rkey;
429+ u8 reserved[4];
430+};
431+
432+struct pvrdma_cmd_destroy_mr {
433+ struct pvrdma_cmd_hdr hdr;
434+ u32 mr_handle;
435+ u8 reserved[4];
436+};
437+
438+struct pvrdma_cmd_create_cq {
439+ struct pvrdma_cmd_hdr hdr;
440+ u64 pdir_dma;
441+ u32 ctx_handle;
442+ u32 cqe;
443+ u32 nchunks;
444+ u8 reserved[4];
445+};
446+
447+struct pvrdma_cmd_create_cq_resp {
448+ struct pvrdma_cmd_resp_hdr hdr;
449+ u32 cq_handle;
450+ u32 cqe;
451+};
452+
453+struct pvrdma_cmd_resize_cq {
454+ struct pvrdma_cmd_hdr hdr;
455+ u32 cq_handle;
456+ u32 cqe;
457+};
458+
459+struct pvrdma_cmd_resize_cq_resp {
460+ struct pvrdma_cmd_resp_hdr hdr;
461+ u32 cqe;
462+ u8 reserved[4];
463+};
464+
465+struct pvrdma_cmd_destroy_cq {
466+ struct pvrdma_cmd_hdr hdr;
467+ u32 cq_handle;
468+ u8 reserved[4];
469+};
470+
471+struct pvrdma_cmd_create_qp {
472+ struct pvrdma_cmd_hdr hdr;
473+ u64 pdir_dma;
474+ u32 pd_handle;
475+ u32 send_cq_handle;
476+ u32 recv_cq_handle;
477+ u32 srq_handle;
478+ u32 max_send_wr;
479+ u32 max_recv_wr;
480+ u32 max_send_sge;
481+ u32 max_recv_sge;
482+ u32 max_inline_data;
483+ u32 lkey;
484+ u32 access_flags;
485+ u16 total_chunks;
486+ u16 send_chunks;
487+ u16 max_atomic_arg;
488+ u8 sq_sig_all;
489+ u8 qp_type;
490+ u8 is_srq;
491+ u8 reserved[3];
492+};
493+
494+struct pvrdma_cmd_create_qp_resp {
495+ struct pvrdma_cmd_resp_hdr hdr;
496+ u32 qpn;
497+ u32 max_send_wr;
498+ u32 max_recv_wr;
499+ u32 max_send_sge;
500+ u32 max_recv_sge;
501+ u32 max_inline_data;
502+};
503+
504+struct pvrdma_cmd_modify_qp {
505+ struct pvrdma_cmd_hdr hdr;
506+ u32 qp_handle;
507+ u32 attr_mask;
508+ struct pvrdma_qp_attr attrs;
509+};
510+
511+struct pvrdma_cmd_query_qp {
512+ struct pvrdma_cmd_hdr hdr;
513+ u32 qp_handle;
514+ u32 attr_mask;
515+};
516+
517+struct pvrdma_cmd_query_qp_resp {
518+ struct pvrdma_cmd_resp_hdr hdr;
519+ struct pvrdma_qp_attr attrs;
520+};
521+
522+struct pvrdma_cmd_destroy_qp {
523+ struct pvrdma_cmd_hdr hdr;
524+ u32 qp_handle;
525+ u8 reserved[4];
526+};
527+
528+struct pvrdma_cmd_destroy_qp_resp {
529+ struct pvrdma_cmd_resp_hdr hdr;
530+ u32 events_reported;
531+ u8 reserved[4];
532+};
533+
534+struct pvrdma_cmd_create_bind {
535+ struct pvrdma_cmd_hdr hdr;
536+ u32 mtu;
537+ u32 vlan;
538+ u32 index;
539+ u8 new_gid[16];
540+ u8 gid_type;
541+ u8 reserved[3];
542+};
543+
544+struct pvrdma_cmd_destroy_bind {
545+ struct pvrdma_cmd_hdr hdr;
546+ u32 index;
547+ u8 dest_gid[16];
548+ u8 reserved[4];
549+};
550+
551+union pvrdma_cmd_req {
552+ struct pvrdma_cmd_hdr hdr;
553+ struct pvrdma_cmd_query_port query_port;
554+ struct pvrdma_cmd_query_pkey query_pkey;
555+ struct pvrdma_cmd_create_uc create_uc;
556+ struct pvrdma_cmd_destroy_uc destroy_uc;
557+ struct pvrdma_cmd_create_pd create_pd;
558+ struct pvrdma_cmd_destroy_pd destroy_pd;
559+ struct pvrdma_cmd_create_mr create_mr;
560+ struct pvrdma_cmd_destroy_mr destroy_mr;
561+ struct pvrdma_cmd_create_cq create_cq;
562+ struct pvrdma_cmd_resize_cq resize_cq;
563+ struct pvrdma_cmd_destroy_cq destroy_cq;
564+ struct pvrdma_cmd_create_qp create_qp;
565+ struct pvrdma_cmd_modify_qp modify_qp;
566+ struct pvrdma_cmd_query_qp query_qp;
567+ struct pvrdma_cmd_destroy_qp destroy_qp;
568+ struct pvrdma_cmd_create_bind create_bind;
569+ struct pvrdma_cmd_destroy_bind destroy_bind;
570+};
571+
572+union pvrdma_cmd_resp {
573+ struct pvrdma_cmd_resp_hdr hdr;
574+ struct pvrdma_cmd_query_port_resp query_port_resp;
575+ struct pvrdma_cmd_query_pkey_resp query_pkey_resp;
576+ struct pvrdma_cmd_create_uc_resp create_uc_resp;
577+ struct pvrdma_cmd_create_pd_resp create_pd_resp;
578+ struct pvrdma_cmd_create_mr_resp create_mr_resp;
579+ struct pvrdma_cmd_create_cq_resp create_cq_resp;
580+ struct pvrdma_cmd_resize_cq_resp resize_cq_resp;
581+ struct pvrdma_cmd_create_qp_resp create_qp_resp;
582+ struct pvrdma_cmd_query_qp_resp query_qp_resp;
583+ struct pvrdma_cmd_destroy_qp_resp destroy_qp_resp;
584+};
585+
586+#endif /* __PVRDMA_DEV_API_H__ */
drivers/infiniband/hw/vmw_pvrdma/pvrdma_doorbell.c+127−0
@@ -0,0 +1,127 @@
1+/*
2+ * Copyright (c) 2012-2016 VMware, Inc. All rights reserved.
3+ *
4+ * This program is free software; you can redistribute it and/or
5+ * modify it under the terms of EITHER the GNU General Public License
6+ * version 2 as published by the Free Software Foundation or the BSD
7+ * 2-Clause License. This program is distributed in the hope that it
8+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
9+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
10+ * See the GNU General Public License version 2 for more details at
11+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
12+ *
13+ * You should have received a copy of the GNU General Public License
14+ * along with this program available in the file COPYING in the main
15+ * directory of this source tree.
16+ *
17+ * The BSD 2-Clause License
18+ *
19+ * Redistribution and use in source and binary forms, with or
20+ * without modification, are permitted provided that the following
21+ * conditions are met:
22+ *
23+ * - Redistributions of source code must retain the above
24+ * copyright notice, this list of conditions and the following
25+ * disclaimer.
26+ *
27+ * - Redistributions in binary form must reproduce the above
28+ * copyright notice, this list of conditions and the following
29+ * disclaimer in the documentation and/or other materials
30+ * provided with the distribution.
31+ *
32+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
33+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
34+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
35+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
36+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
37+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
38+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
39+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
40+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
41+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
42+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
43+ * OF THE POSSIBILITY OF SUCH DAMAGE.
44+ */
45+
46+#include <linux/bitmap.h>
47+#include <linux/errno.h>
48+#include <linux/slab.h>
49+
50+#include "pvrdma.h"
51+
52+int pvrdma_uar_table_init(struct pvrdma_dev *dev)
53+{
54+ u32 num = dev->dsr->caps.max_uar;
55+ u32 mask = num - 1;
56+ struct pvrdma_id_table *tbl = &dev->uar_table.tbl;
57+
58+ if (!is_power_of_2(num))
59+ return -EINVAL;
60+
61+ tbl->last = 0;
62+ tbl->top = 0;
63+ tbl->max = num;
64+ tbl->mask = mask;
65+ spin_lock_init(&tbl->lock);
66+ tbl->table = kcalloc(BITS_TO_LONGS(num), sizeof(long), GFP_KERNEL);
67+ if (!tbl->table)
68+ return -ENOMEM;
69+
70+ /* 0th UAR is taken by the device. */
71+ set_bit(0, tbl->table);
72+
73+ return 0;
74+}
75+
76+void pvrdma_uar_table_cleanup(struct pvrdma_dev *dev)
77+{
78+ struct pvrdma_id_table *tbl = &dev->uar_table.tbl;
79+
80+ kfree(tbl->table);
81+}
82+
83+int pvrdma_uar_alloc(struct pvrdma_dev *dev, struct pvrdma_uar_map *uar)
84+{
85+ struct pvrdma_id_table *tbl;
86+ unsigned long flags;
87+ u32 obj;
88+
89+ tbl = &dev->uar_table.tbl;
90+
91+ spin_lock_irqsave(&tbl->lock, flags);
92+ obj = find_next_zero_bit(tbl->table, tbl->max, tbl->last);
93+ if (obj >= tbl->max) {
94+ tbl->top = (tbl->top + tbl->max) & tbl->mask;
95+ obj = find_first_zero_bit(tbl->table, tbl->max);
96+ }
97+
98+ if (obj >= tbl->max) {
99+ spin_unlock_irqrestore(&tbl->lock, flags);
100+ return -ENOMEM;
101+ }
102+
103+ set_bit(obj, tbl->table);
104+ obj |= tbl->top;
105+
106+ spin_unlock_irqrestore(&tbl->lock, flags);
107+
108+ uar->index = obj;
109+ uar->pfn = (pci_resource_start(dev->pdev, PVRDMA_PCI_RESOURCE_UAR) >>
110+ PAGE_SHIFT) + uar->index;
111+
112+ return 0;
113+}
114+
115+void pvrdma_uar_free(struct pvrdma_dev *dev, struct pvrdma_uar_map *uar)
116+{
117+ struct pvrdma_id_table *tbl = &dev->uar_table.tbl;
118+ unsigned long flags;
119+ u32 obj;
120+
121+ obj = uar->index & (tbl->max - 1);
122+ spin_lock_irqsave(&tbl->lock, flags);
123+ clear_bit(obj, tbl->table);
124+ tbl->last = min(tbl->last, obj);
125+ tbl->top = (tbl->top + tbl->max) & tbl->mask;
126+ spin_unlock_irqrestore(&tbl->lock, flags);
127+}
drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c+1211−0
@@ -0,0 +1,1211 @@
1+/*
2+ * Copyright (c) 2012-2016 VMware, Inc. All rights reserved.
3+ *
4+ * This program is free software; you can redistribute it and/or
5+ * modify it under the terms of EITHER the GNU General Public License
6+ * version 2 as published by the Free Software Foundation or the BSD
7+ * 2-Clause License. This program is distributed in the hope that it
8+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
9+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
10+ * See the GNU General Public License version 2 for more details at
11+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
12+ *
13+ * You should have received a copy of the GNU General Public License
14+ * along with this program available in the file COPYING in the main
15+ * directory of this source tree.
16+ *
17+ * The BSD 2-Clause License
18+ *
19+ * Redistribution and use in source and binary forms, with or
20+ * without modification, are permitted provided that the following
21+ * conditions are met:
22+ *
23+ * - Redistributions of source code must retain the above
24+ * copyright notice, this list of conditions and the following
25+ * disclaimer.
26+ *
27+ * - Redistributions in binary form must reproduce the above
28+ * copyright notice, this list of conditions and the following
29+ * disclaimer in the documentation and/or other materials
30+ * provided with the distribution.
31+ *
32+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
33+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
34+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
35+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
36+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
37+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
38+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
39+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
40+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
41+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
42+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
43+ * OF THE POSSIBILITY OF SUCH DAMAGE.
44+ */
45+
46+#include <linux/errno.h>
47+#include <linux/inetdevice.h>
48+#include <linux/init.h>
49+#include <linux/module.h>
50+#include <linux/slab.h>
51+#include <rdma/ib_addr.h>
52+#include <rdma/ib_smi.h>
53+#include <rdma/ib_user_verbs.h>
54+#include <net/addrconf.h>
55+
56+#include "pvrdma.h"
57+
58+#define DRV_NAME "vmw_pvrdma"
59+#define DRV_VERSION "1.0.0.0-k"
60+
61+static DEFINE_MUTEX(pvrdma_device_list_lock);
62+static LIST_HEAD(pvrdma_device_list);
63+static struct workqueue_struct *event_wq;
64+
65+static int pvrdma_add_gid(struct ib_device *ibdev,
66+ u8 port_num,
67+ unsigned int index,
68+ const union ib_gid *gid,
69+ const struct ib_gid_attr *attr,
70+ void **context);
71+static int pvrdma_del_gid(struct ib_device *ibdev,
72+ u8 port_num,
73+ unsigned int index,
74+ void **context);
75+
76+
77+static ssize_t show_hca(struct device *device, struct device_attribute *attr,
78+ char *buf)
79+{
80+ return sprintf(buf, "VMW_PVRDMA-%s\n", DRV_VERSION);
81+}
82+
83+static ssize_t show_rev(struct device *device, struct device_attribute *attr,
84+ char *buf)
85+{
86+ return sprintf(buf, "%d\n", PVRDMA_REV_ID);
87+}
88+
89+static ssize_t show_board(struct device *device, struct device_attribute *attr,
90+ char *buf)
91+{
92+ return sprintf(buf, "%d\n", PVRDMA_BOARD_ID);
93+}
94+
95+static DEVICE_ATTR(hw_rev, S_IRUGO, show_rev, NULL);
96+static DEVICE_ATTR(hca_type, S_IRUGO, show_hca, NULL);
97+static DEVICE_ATTR(board_id, S_IRUGO, show_board, NULL);
98+
99+static struct device_attribute *pvrdma_class_attributes[] = {
100+ &dev_attr_hw_rev,
101+ &dev_attr_hca_type,
102+ &dev_attr_board_id
103+};
104+
105+static void pvrdma_get_fw_ver_str(struct ib_device *device, char *str,
106+ size_t str_len)
107+{
108+ struct pvrdma_dev *dev =
109+ container_of(device, struct pvrdma_dev, ib_dev);
110+ snprintf(str, str_len, "%d.%d.%d\n",
111+ (int) (dev->dsr->caps.fw_ver >> 32),
112+ (int) (dev->dsr->caps.fw_ver >> 16) & 0xffff,
113+ (int) dev->dsr->caps.fw_ver & 0xffff);
114+}
115+
116+static int pvrdma_init_device(struct pvrdma_dev *dev)
117+{
118+ /* Initialize some device related stuff */
119+ spin_lock_init(&dev->cmd_lock);
120+ sema_init(&dev->cmd_sema, 1);
121+ atomic_set(&dev->num_qps, 0);
122+ atomic_set(&dev->num_cqs, 0);
123+ atomic_set(&dev->num_pds, 0);
124+ atomic_set(&dev->num_ahs, 0);
125+
126+ return 0;
127+}
128+
129+static int pvrdma_port_immutable(struct ib_device *ibdev, u8 port_num,
130+ struct ib_port_immutable *immutable)
131+{
132+ struct ib_port_attr attr;
133+ int err;
134+
135+ err = pvrdma_query_port(ibdev, port_num, &attr);
136+ if (err)
137+ return err;
138+
139+ immutable->pkey_tbl_len = attr.pkey_tbl_len;
140+ immutable->gid_tbl_len = attr.gid_tbl_len;
141+ immutable->core_cap_flags = RDMA_CORE_PORT_IBA_ROCE;
142+ immutable->max_mad_size = IB_MGMT_MAD_SIZE;
143+ return 0;
144+}
145+
146+static struct net_device *pvrdma_get_netdev(struct ib_device *ibdev,
147+ u8 port_num)
148+{
149+ struct net_device *netdev;
150+ struct pvrdma_dev *dev = to_vdev(ibdev);
151+
152+ if (port_num != 1)
153+ return NULL;
154+
155+ rcu_read_lock();
156+ netdev = dev->netdev;
157+ if (netdev)
158+ dev_hold(netdev);
159+ rcu_read_unlock();
160+
161+ return netdev;
162+}
163+
164+static int pvrdma_register_device(struct pvrdma_dev *dev)
165+{
166+ int ret = -1;
167+ int i = 0;
168+
169+ strlcpy(dev->ib_dev.name, "vmw_pvrdma%d", IB_DEVICE_NAME_MAX);
170+ dev->ib_dev.node_guid = dev->dsr->caps.node_guid;
171+ dev->sys_image_guid = dev->dsr->caps.sys_image_guid;
172+ dev->flags = 0;
173+ dev->ib_dev.owner = THIS_MODULE;
174+ dev->ib_dev.num_comp_vectors = 1;
175+ dev->ib_dev.dma_device = &dev->pdev->dev;
176+ dev->ib_dev.uverbs_abi_ver = PVRDMA_UVERBS_ABI_VERSION;
177+ dev->ib_dev.uverbs_cmd_mask =
178+ (1ull << IB_USER_VERBS_CMD_GET_CONTEXT) |
179+ (1ull << IB_USER_VERBS_CMD_QUERY_DEVICE) |
180+ (1ull << IB_USER_VERBS_CMD_QUERY_PORT) |
181+ (1ull << IB_USER_VERBS_CMD_ALLOC_PD) |
182+ (1ull << IB_USER_VERBS_CMD_DEALLOC_PD) |
183+ (1ull << IB_USER_VERBS_CMD_REG_MR) |
184+ (1ull << IB_USER_VERBS_CMD_DEREG_MR) |
185+ (1ull << IB_USER_VERBS_CMD_CREATE_COMP_CHANNEL) |
186+ (1ull << IB_USER_VERBS_CMD_CREATE_CQ) |
187+ (1ull << IB_USER_VERBS_CMD_POLL_CQ) |
188+ (1ull << IB_USER_VERBS_CMD_REQ_NOTIFY_CQ) |
189+ (1ull << IB_USER_VERBS_CMD_DESTROY_CQ) |
190+ (1ull << IB_USER_VERBS_CMD_CREATE_QP) |
191+ (1ull << IB_USER_VERBS_CMD_MODIFY_QP) |
192+ (1ull << IB_USER_VERBS_CMD_QUERY_QP) |
193+ (1ull << IB_USER_VERBS_CMD_DESTROY_QP) |
194+ (1ull << IB_USER_VERBS_CMD_POST_SEND) |
195+ (1ull << IB_USER_VERBS_CMD_POST_RECV) |
196+ (1ull << IB_USER_VERBS_CMD_CREATE_AH) |
197+ (1ull << IB_USER_VERBS_CMD_DESTROY_AH);
198+
199+ dev->ib_dev.node_type = RDMA_NODE_IB_CA;
200+ dev->ib_dev.phys_port_cnt = dev->dsr->caps.phys_port_cnt;
201+
202+ dev->ib_dev.query_device = pvrdma_query_device;
203+ dev->ib_dev.query_port = pvrdma_query_port;
204+ dev->ib_dev.query_gid = pvrdma_query_gid;
205+ dev->ib_dev.query_pkey = pvrdma_query_pkey;
206+ dev->ib_dev.modify_port = pvrdma_modify_port;
207+ dev->ib_dev.alloc_ucontext = pvrdma_alloc_ucontext;
208+ dev->ib_dev.dealloc_ucontext = pvrdma_dealloc_ucontext;
209+ dev->ib_dev.mmap = pvrdma_mmap;
210+ dev->ib_dev.alloc_pd = pvrdma_alloc_pd;
211+ dev->ib_dev.dealloc_pd = pvrdma_dealloc_pd;
212+ dev->ib_dev.create_ah = pvrdma_create_ah;
213+ dev->ib_dev.destroy_ah = pvrdma_destroy_ah;
214+ dev->ib_dev.create_qp = pvrdma_create_qp;
215+ dev->ib_dev.modify_qp = pvrdma_modify_qp;
216+ dev->ib_dev.query_qp = pvrdma_query_qp;
217+ dev->ib_dev.destroy_qp = pvrdma_destroy_qp;
218+ dev->ib_dev.post_send = pvrdma_post_send;
219+ dev->ib_dev.post_recv = pvrdma_post_recv;
220+ dev->ib_dev.create_cq = pvrdma_create_cq;
221+ dev->ib_dev.modify_cq = pvrdma_modify_cq;
222+ dev->ib_dev.resize_cq = pvrdma_resize_cq;
223+ dev->ib_dev.destroy_cq = pvrdma_destroy_cq;
224+ dev->ib_dev.poll_cq = pvrdma_poll_cq;
225+ dev->ib_dev.req_notify_cq = pvrdma_req_notify_cq;
226+ dev->ib_dev.get_dma_mr = pvrdma_get_dma_mr;
227+ dev->ib_dev.reg_user_mr = pvrdma_reg_user_mr;
228+ dev->ib_dev.dereg_mr = pvrdma_dereg_mr;
229+ dev->ib_dev.alloc_mr = pvrdma_alloc_mr;
230+ dev->ib_dev.map_mr_sg = pvrdma_map_mr_sg;
231+ dev->ib_dev.add_gid = pvrdma_add_gid;
232+ dev->ib_dev.del_gid = pvrdma_del_gid;
233+ dev->ib_dev.get_netdev = pvrdma_get_netdev;
234+ dev->ib_dev.get_port_immutable = pvrdma_port_immutable;
235+ dev->ib_dev.get_link_layer = pvrdma_port_link_layer;
236+ dev->ib_dev.get_dev_fw_str = pvrdma_get_fw_ver_str;
237+
238+ mutex_init(&dev->port_mutex);
239+ spin_lock_init(&dev->desc_lock);
240+
241+ dev->cq_tbl = kcalloc(dev->dsr->caps.max_cq, sizeof(void *),
242+ GFP_KERNEL);
243+ if (!dev->cq_tbl)
244+ return ret;
245+ spin_lock_init(&dev->cq_tbl_lock);
246+
247+ dev->qp_tbl = kcalloc(dev->dsr->caps.max_qp, sizeof(void *),
248+ GFP_KERNEL);
249+ if (!dev->qp_tbl)
250+ goto err_cq_free;
251+ spin_lock_init(&dev->qp_tbl_lock);
252+
253+ ret = ib_register_device(&dev->ib_dev, NULL);
254+ if (ret)
255+ goto err_qp_free;
256+
257+ for (i = 0; i < ARRAY_SIZE(pvrdma_class_attributes); ++i) {
258+ ret = device_create_file(&dev->ib_dev.dev,
259+ pvrdma_class_attributes[i]);
260+ if (ret)
261+ goto err_class;
262+ }
263+
264+ dev->ib_active = true;
265+
266+ return 0;
267+
268+err_class:
269+ ib_unregister_device(&dev->ib_dev);
270+err_qp_free:
271+ kfree(dev->qp_tbl);
272+err_cq_free:
273+ kfree(dev->cq_tbl);
274+
275+ return ret;
276+}
277+
278+static irqreturn_t pvrdma_intr0_handler(int irq, void *dev_id)
279+{
280+ u32 icr = PVRDMA_INTR_CAUSE_RESPONSE;
281+ struct pvrdma_dev *dev = dev_id;
282+
283+ dev_dbg(&dev->pdev->dev, "interrupt 0 (response) handler\n");
284+
285+ if (dev->intr.type != PVRDMA_INTR_TYPE_MSIX) {
286+ /* Legacy intr */
287+ icr = pvrdma_read_reg(dev, PVRDMA_REG_ICR);
288+ if (icr == 0)
289+ return IRQ_NONE;
290+ }
291+
292+ if (icr == PVRDMA_INTR_CAUSE_RESPONSE)
293+ complete(&dev->cmd_done);
294+
295+ return IRQ_HANDLED;
296+}
297+
298+static void pvrdma_qp_event(struct pvrdma_dev *dev, u32 qpn, int type)
299+{
300+ struct pvrdma_qp *qp;
301+ unsigned long flags;
302+
303+ spin_lock_irqsave(&dev->qp_tbl_lock, flags);
304+ qp = dev->qp_tbl[qpn % dev->dsr->caps.max_qp];
305+ if (qp)
306+ atomic_inc(&qp->refcnt);
307+ spin_unlock_irqrestore(&dev->qp_tbl_lock, flags);
308+
309+ if (qp && qp->ibqp.event_handler) {
310+ struct ib_qp *ibqp = &qp->ibqp;
311+ struct ib_event e;
312+
313+ e.device = ibqp->device;
314+ e.element.qp = ibqp;
315+ e.event = type; /* 1:1 mapping for now. */
316+ ibqp->event_handler(&e, ibqp->qp_context);
317+ }
318+ if (qp) {
319+ atomic_dec(&qp->refcnt);
320+ if (atomic_read(&qp->refcnt) == 0)
321+ wake_up(&qp->wait);
322+ }
323+}
324+
325+static void pvrdma_cq_event(struct pvrdma_dev *dev, u32 cqn, int type)
326+{
327+ struct pvrdma_cq *cq;
328+ unsigned long flags;
329+
330+ spin_lock_irqsave(&dev->cq_tbl_lock, flags);
331+ cq = dev->cq_tbl[cqn % dev->dsr->caps.max_cq];
332+ if (cq)
333+ atomic_inc(&cq->refcnt);
334+ spin_unlock_irqrestore(&dev->cq_tbl_lock, flags);
335+
336+ if (cq && cq->ibcq.event_handler) {
337+ struct ib_cq *ibcq = &cq->ibcq;
338+ struct ib_event e;
339+
340+ e.device = ibcq->device;
341+ e.element.cq = ibcq;
342+ e.event = type; /* 1:1 mapping for now. */
343+ ibcq->event_handler(&e, ibcq->cq_context);
344+ }
345+ if (cq) {
346+ atomic_dec(&cq->refcnt);
347+ if (atomic_read(&cq->refcnt) == 0)
348+ wake_up(&cq->wait);
349+ }
350+}
351+
352+static void pvrdma_dispatch_event(struct pvrdma_dev *dev, int port,
353+ enum ib_event_type event)
354+{
355+ struct ib_event ib_event;
356+
357+ memset(&ib_event, 0, sizeof(ib_event));
358+ ib_event.device = &dev->ib_dev;
359+ ib_event.element.port_num = port;
360+ ib_event.event = event;
361+ ib_dispatch_event(&ib_event);
362+}
363+
364+static void pvrdma_dev_event(struct pvrdma_dev *dev, u8 port, int type)
365+{
366+ if (port < 1 || port > dev->dsr->caps.phys_port_cnt) {
367+ dev_warn(&dev->pdev->dev, "event on port %d\n", port);
368+ return;
369+ }
370+
371+ pvrdma_dispatch_event(dev, port, type);
372+}
373+
374+static inline struct pvrdma_eqe *get_eqe(struct pvrdma_dev *dev, unsigned int i)
375+{
376+ return (struct pvrdma_eqe *)pvrdma_page_dir_get_ptr(
377+ &dev->async_pdir,
378+ PAGE_SIZE +
379+ sizeof(struct pvrdma_eqe) * i);
380+}
381+
382+static irqreturn_t pvrdma_intr1_handler(int irq, void *dev_id)
383+{
384+ struct pvrdma_dev *dev = dev_id;
385+ struct pvrdma_ring *ring = &dev->async_ring_state->rx;
386+ int ring_slots = (dev->dsr->async_ring_pages.num_pages - 1) *
387+ PAGE_SIZE / sizeof(struct pvrdma_eqe);
388+ unsigned int head;
389+
390+ dev_dbg(&dev->pdev->dev, "interrupt 1 (async event) handler\n");
391+
392+ /*
393+ * Don't process events until the IB device is registered. Otherwise
394+ * we'll try to ib_dispatch_event() on an invalid device.
395+ */
396+ if (!dev->ib_active)
397+ return IRQ_HANDLED;
398+
399+ while (pvrdma_idx_ring_has_data(ring, ring_slots, &head) > 0) {
400+ struct pvrdma_eqe *eqe;
401+
402+ eqe = get_eqe(dev, head);
403+
404+ switch (eqe->type) {
405+ case PVRDMA_EVENT_QP_FATAL:
406+ case PVRDMA_EVENT_QP_REQ_ERR:
407+ case PVRDMA_EVENT_QP_ACCESS_ERR:
408+ case PVRDMA_EVENT_COMM_EST:
409+ case PVRDMA_EVENT_SQ_DRAINED:
410+ case PVRDMA_EVENT_PATH_MIG:
411+ case PVRDMA_EVENT_PATH_MIG_ERR:
412+ case PVRDMA_EVENT_QP_LAST_WQE_REACHED:
413+ pvrdma_qp_event(dev, eqe->info, eqe->type);
414+ break;
415+
416+ case PVRDMA_EVENT_CQ_ERR:
417+ pvrdma_cq_event(dev, eqe->info, eqe->type);
418+ break;
419+
420+ case PVRDMA_EVENT_SRQ_ERR:
421+ case PVRDMA_EVENT_SRQ_LIMIT_REACHED:
422+ break;
423+
424+ case PVRDMA_EVENT_PORT_ACTIVE:
425+ case PVRDMA_EVENT_PORT_ERR:
426+ case PVRDMA_EVENT_LID_CHANGE:
427+ case PVRDMA_EVENT_PKEY_CHANGE:
428+ case PVRDMA_EVENT_SM_CHANGE:
429+ case PVRDMA_EVENT_CLIENT_REREGISTER:
430+ case PVRDMA_EVENT_GID_CHANGE:
431+ pvrdma_dev_event(dev, eqe->info, eqe->type);
432+ break;
433+
434+ case PVRDMA_EVENT_DEVICE_FATAL:
435+ pvrdma_dev_event(dev, 1, eqe->type);
436+ break;
437+
438+ default:
439+ break;
440+ }
441+
442+ pvrdma_idx_ring_inc(&ring->cons_head, ring_slots);
443+ }
444+
445+ return IRQ_HANDLED;
446+}
447+
448+static inline struct pvrdma_cqne *get_cqne(struct pvrdma_dev *dev,
449+ unsigned int i)
450+{
451+ return (struct pvrdma_cqne *)pvrdma_page_dir_get_ptr(
452+ &dev->cq_pdir,
453+ PAGE_SIZE +
454+ sizeof(struct pvrdma_cqne) * i);
455+}
456+
457+static irqreturn_t pvrdma_intrx_handler(int irq, void *dev_id)
458+{
459+ struct pvrdma_dev *dev = dev_id;
460+ struct pvrdma_ring *ring = &dev->cq_ring_state->rx;
461+ int ring_slots = (dev->dsr->cq_ring_pages.num_pages - 1) * PAGE_SIZE /
462+ sizeof(struct pvrdma_cqne);
463+ unsigned int head;
464+ unsigned long flags;
465+
466+ dev_dbg(&dev->pdev->dev, "interrupt x (completion) handler\n");
467+
468+ while (pvrdma_idx_ring_has_data(ring, ring_slots, &head) > 0) {
469+ struct pvrdma_cqne *cqne;
470+ struct pvrdma_cq *cq;
471+
472+ cqne = get_cqne(dev, head);
473+ spin_lock_irqsave(&dev->cq_tbl_lock, flags);
474+ cq = dev->cq_tbl[cqne->info % dev->dsr->caps.max_cq];
475+ if (cq)
476+ atomic_inc(&cq->refcnt);
477+ spin_unlock_irqrestore(&dev->cq_tbl_lock, flags);
478+
479+ if (cq && cq->ibcq.comp_handler)
480+ cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context);
481+ if (cq) {
482+ atomic_dec(&cq->refcnt);
483+ if (atomic_read(&cq->refcnt))
484+ wake_up(&cq->wait);
485+ }
486+ pvrdma_idx_ring_inc(&ring->cons_head, ring_slots);
487+ }
488+
489+ return IRQ_HANDLED;
490+}
491+
492+static void pvrdma_disable_msi_all(struct pvrdma_dev *dev)
493+{
494+ if (dev->intr.type == PVRDMA_INTR_TYPE_MSIX)
495+ pci_disable_msix(dev->pdev);
496+ else if (dev->intr.type == PVRDMA_INTR_TYPE_MSI)
497+ pci_disable_msi(dev->pdev);
498+}
499+
500+static void pvrdma_free_irq(struct pvrdma_dev *dev)
501+{
502+ int i;
503+
504+ dev_dbg(&dev->pdev->dev, "freeing interrupts\n");
505+
506+ if (dev->intr.type == PVRDMA_INTR_TYPE_MSIX) {
507+ for (i = 0; i < dev->intr.size; i++) {
508+ if (dev->intr.enabled[i]) {
509+ free_irq(dev->intr.msix_entry[i].vector, dev);
510+ dev->intr.enabled[i] = 0;
511+ }
512+ }
513+ } else if (dev->intr.type == PVRDMA_INTR_TYPE_INTX ||
514+ dev->intr.type == PVRDMA_INTR_TYPE_MSI) {
515+ free_irq(dev->pdev->irq, dev);
516+ }
517+}
518+
519+static void pvrdma_enable_intrs(struct pvrdma_dev *dev)
520+{
521+ dev_dbg(&dev->pdev->dev, "enable interrupts\n");
522+ pvrdma_write_reg(dev, PVRDMA_REG_IMR, 0);
523+}
524+
525+static void pvrdma_disable_intrs(struct pvrdma_dev *dev)
526+{
527+ dev_dbg(&dev->pdev->dev, "disable interrupts\n");
528+ pvrdma_write_reg(dev, PVRDMA_REG_IMR, ~0);
529+}
530+
531+static int pvrdma_enable_msix(struct pci_dev *pdev, struct pvrdma_dev *dev)
532+{
533+ int i;
534+ int ret;
535+
536+ for (i = 0; i < PVRDMA_MAX_INTERRUPTS; i++) {
537+ dev->intr.msix_entry[i].entry = i;
538+ dev->intr.msix_entry[i].vector = i;
539+
540+ switch (i) {
541+ case 0:
542+ /* CMD ring handler */
543+ dev->intr.handler[i] = pvrdma_intr0_handler;
544+ break;
545+ case 1:
546+ /* Async event ring handler */
547+ dev->intr.handler[i] = pvrdma_intr1_handler;
548+ break;
549+ default:
550+ /* Completion queue handler */
551+ dev->intr.handler[i] = pvrdma_intrx_handler;
552+ break;
553+ }
554+ }
555+
556+ ret = pci_enable_msix(pdev, dev->intr.msix_entry,
557+ PVRDMA_MAX_INTERRUPTS);
558+ if (!ret) {
559+ dev->intr.type = PVRDMA_INTR_TYPE_MSIX;
560+ dev->intr.size = PVRDMA_MAX_INTERRUPTS;
561+ } else if (ret > 0) {
562+ ret = pci_enable_msix(pdev, dev->intr.msix_entry, ret);
563+ if (!ret) {
564+ dev->intr.type = PVRDMA_INTR_TYPE_MSIX;
565+ dev->intr.size = ret;
566+ } else {
567+ dev->intr.size = 0;
568+ }
569+ }
570+
571+ dev_dbg(&pdev->dev, "using interrupt type %d, size %d\n",
572+ dev->intr.type, dev->intr.size);
573+
574+ return ret;
575+}
576+
577+static int pvrdma_alloc_intrs(struct pvrdma_dev *dev)
578+{
579+ int ret = 0;
580+ int i;
581+
582+ if (pci_find_capability(dev->pdev, PCI_CAP_ID_MSIX) &&
583+ pvrdma_enable_msix(dev->pdev, dev)) {
584+ /* Try MSI */
585+ ret = pci_enable_msi(dev->pdev);
586+ if (!ret) {
587+ dev->intr.type = PVRDMA_INTR_TYPE_MSI;
588+ } else {
589+ /* Legacy INTR */
590+ dev->intr.type = PVRDMA_INTR_TYPE_INTX;
591+ }
592+ }
593+
594+ /* Request First IRQ */
595+ switch (dev->intr.type) {
596+ case PVRDMA_INTR_TYPE_INTX:
597+ case PVRDMA_INTR_TYPE_MSI:
598+ ret = request_irq(dev->pdev->irq, pvrdma_intr0_handler,
599+ IRQF_SHARED, DRV_NAME, dev);
600+ if (ret) {
601+ dev_err(&dev->pdev->dev,
602+ "failed to request interrupt\n");
603+ goto disable_msi;
604+ }
605+ break;
606+ case PVRDMA_INTR_TYPE_MSIX:
607+ ret = request_irq(dev->intr.msix_entry[0].vector,
608+ pvrdma_intr0_handler, 0, DRV_NAME, dev);
609+ if (ret) {
610+ dev_err(&dev->pdev->dev,
611+ "failed to request interrupt 0\n");
612+ goto disable_msi;
613+ }
614+ dev->intr.enabled[0] = 1;
615+ break;
616+ default:
617+ /* Not reached */
618+ break;
619+ }
620+
621+ /* For MSIX: request intr for each vector */
622+ if (dev->intr.size > 1) {
623+ ret = request_irq(dev->intr.msix_entry[1].vector,
624+ pvrdma_intr1_handler, 0, DRV_NAME, dev);
625+ if (ret) {
626+ dev_err(&dev->pdev->dev,
627+ "failed to request interrupt 1\n");
628+ goto free_irq;
629+ }
630+ dev->intr.enabled[1] = 1;
631+
632+ for (i = 2; i < dev->intr.size; i++) {
633+ ret = request_irq(dev->intr.msix_entry[i].vector,
634+ pvrdma_intrx_handler, 0,
635+ DRV_NAME, dev);
636+ if (ret) {
637+ dev_err(&dev->pdev->dev,
638+ "failed to request interrupt %d\n", i);
639+ goto free_irq;
640+ }
641+ dev->intr.enabled[i] = 1;
642+ }
643+ }
644+
645+ return 0;
646+
647+free_irq:
648+ pvrdma_free_irq(dev);
649+disable_msi:
650+ pvrdma_disable_msi_all(dev);
651+ return ret;
652+}
653+
654+static void pvrdma_free_slots(struct pvrdma_dev *dev)
655+{
656+ struct pci_dev *pdev = dev->pdev;
657+
658+ if (dev->resp_slot)
659+ dma_free_coherent(&pdev->dev, PAGE_SIZE, dev->resp_slot,
660+ dev->dsr->resp_slot_dma);
661+ if (dev->cmd_slot)
662+ dma_free_coherent(&pdev->dev, PAGE_SIZE, dev->cmd_slot,
663+ dev->dsr->cmd_slot_dma);
664+}
665+
666+static int pvrdma_add_gid_at_index(struct pvrdma_dev *dev,
667+ const union ib_gid *gid,
668+ int index)
669+{
670+ int ret;
671+ union pvrdma_cmd_req req;
672+ struct pvrdma_cmd_create_bind *cmd_bind = &req.create_bind;
673+
674+ if (!dev->sgid_tbl) {
675+ dev_warn(&dev->pdev->dev, "sgid table not initialized\n");
676+ return -EINVAL;
677+ }
678+
679+ memset(cmd_bind, 0, sizeof(*cmd_bind));
680+ cmd_bind->hdr.cmd = PVRDMA_CMD_CREATE_BIND;
681+ memcpy(cmd_bind->new_gid, gid->raw, 16);
682+ cmd_bind->mtu = ib_mtu_enum_to_int(IB_MTU_1024);
683+ cmd_bind->vlan = 0xfff;
684+ cmd_bind->index = index;
685+ cmd_bind->gid_type = PVRDMA_GID_TYPE_FLAG_ROCE_V1;
686+
687+ ret = pvrdma_cmd_post(dev, &req, NULL, 0);
688+ if (ret < 0) {
689+ dev_warn(&dev->pdev->dev,
690+ "could not create binding, error: %d\n", ret);
691+ return -EFAULT;
692+ }
693+ memcpy(&dev->sgid_tbl[index], gid, sizeof(*gid));
694+ return 0;
695+}
696+
697+static int pvrdma_add_gid(struct ib_device *ibdev,
698+ u8 port_num,
699+ unsigned int index,
700+ const union ib_gid *gid,
701+ const struct ib_gid_attr *attr,
702+ void **context)
703+{
704+ struct pvrdma_dev *dev = to_vdev(ibdev);
705+
706+ return pvrdma_add_gid_at_index(dev, gid, index);
707+}
708+
709+static int pvrdma_del_gid_at_index(struct pvrdma_dev *dev, int index)
710+{
711+ int ret;
712+ union pvrdma_cmd_req req;
713+ struct pvrdma_cmd_destroy_bind *cmd_dest = &req.destroy_bind;
714+
715+ /* Update sgid table. */
716+ if (!dev->sgid_tbl) {
717+ dev_warn(&dev->pdev->dev, "sgid table not initialized\n");
718+ return -EINVAL;
719+ }
720+
721+ memset(cmd_dest, 0, sizeof(*cmd_dest));
722+ cmd_dest->hdr.cmd = PVRDMA_CMD_DESTROY_BIND;
723+ memcpy(cmd_dest->dest_gid, &dev->sgid_tbl[index], 16);
724+ cmd_dest->index = index;
725+
726+ ret = pvrdma_cmd_post(dev, &req, NULL, 0);
727+ if (ret < 0) {
728+ dev_warn(&dev->pdev->dev,
729+ "could not destroy binding, error: %d\n", ret);
730+ return ret;
731+ }
732+ memset(&dev->sgid_tbl[index], 0, 16);
733+ return 0;
734+}
735+
736+static int pvrdma_del_gid(struct ib_device *ibdev,
737+ u8 port_num,
738+ unsigned int index,
739+ void **context)
740+{
741+ struct pvrdma_dev *dev = to_vdev(ibdev);
742+
743+ dev_dbg(&dev->pdev->dev, "removing gid at index %u from %s",
744+ index, dev->netdev->name);
745+
746+ return pvrdma_del_gid_at_index(dev, index);
747+}
748+
749+static void pvrdma_netdevice_event_handle(struct pvrdma_dev *dev,
750+ unsigned long event)
751+{
752+ switch (event) {
753+ case NETDEV_REBOOT:
754+ case NETDEV_DOWN:
755+ pvrdma_dispatch_event(dev, 1, IB_EVENT_PORT_ERR);
756+ break;
757+ case NETDEV_UP:
758+ pvrdma_dispatch_event(dev, 1, IB_EVENT_PORT_ACTIVE);
759+ break;
760+ default:
761+ dev_dbg(&dev->pdev->dev, "ignore netdevice event %ld on %s\n",
762+ event, dev->ib_dev.name);
763+ break;
764+ }
765+}
766+
767+static void pvrdma_netdevice_event_work(struct work_struct *work)
768+{
769+ struct pvrdma_netdevice_work *netdev_work;
770+ struct pvrdma_dev *dev;
771+
772+ netdev_work = container_of(work, struct pvrdma_netdevice_work, work);
773+
774+ mutex_lock(&pvrdma_device_list_lock);
775+ list_for_each_entry(dev, &pvrdma_device_list, device_link) {
776+ if (dev->netdev == netdev_work->event_netdev) {
777+ pvrdma_netdevice_event_handle(dev, netdev_work->event);
778+ break;
779+ }
780+ }
781+ mutex_unlock(&pvrdma_device_list_lock);
782+
783+ kfree(netdev_work);
784+}
785+
786+static int pvrdma_netdevice_event(struct notifier_block *this,
787+ unsigned long event, void *ptr)
788+{
789+ struct net_device *event_netdev = netdev_notifier_info_to_dev(ptr);
790+ struct pvrdma_netdevice_work *netdev_work;
791+
792+ netdev_work = kmalloc(sizeof(*netdev_work), GFP_ATOMIC);
793+ if (!netdev_work)
794+ return NOTIFY_BAD;
795+
796+ INIT_WORK(&netdev_work->work, pvrdma_netdevice_event_work);
797+ netdev_work->event_netdev = event_netdev;
798+ netdev_work->event = event;
799+ queue_work(event_wq, &netdev_work->work);
800+
801+ return NOTIFY_DONE;
802+}
803+
804+static int pvrdma_pci_probe(struct pci_dev *pdev,
805+ const struct pci_device_id *id)
806+{
807+ struct pci_dev *pdev_net;
808+ struct pvrdma_dev *dev;
809+ int ret;
810+ unsigned long start;
811+ unsigned long len;
812+ unsigned int version;
813+ dma_addr_t slot_dma = 0;
814+
815+ dev_dbg(&pdev->dev, "initializing driver %s\n", pci_name(pdev));
816+
817+ /* Allocate zero-out device */
818+ dev = (struct pvrdma_dev *)ib_alloc_device(sizeof(*dev));
819+ if (!dev) {
820+ dev_err(&pdev->dev, "failed to allocate IB device\n");
821+ return -ENOMEM;
822+ }
823+
824+ mutex_lock(&pvrdma_device_list_lock);
825+ list_add(&dev->device_link, &pvrdma_device_list);
826+ mutex_unlock(&pvrdma_device_list_lock);
827+
828+ ret = pvrdma_init_device(dev);
829+ if (ret)
830+ goto err_free_device;
831+
832+ dev->pdev = pdev;
833+ pci_set_drvdata(pdev, dev);
834+
835+ ret = pci_enable_device(pdev);
836+ if (ret) {
837+ dev_err(&pdev->dev, "cannot enable PCI device\n");
838+ goto err_free_device;
839+ }
840+
841+ dev_dbg(&pdev->dev, "PCI resource flags BAR0 %#lx\n",
842+ pci_resource_flags(pdev, 0));
843+ dev_dbg(&pdev->dev, "PCI resource len %#llx\n",
844+ (unsigned long long)pci_resource_len(pdev, 0));
845+ dev_dbg(&pdev->dev, "PCI resource start %#llx\n",
846+ (unsigned long long)pci_resource_start(pdev, 0));
847+ dev_dbg(&pdev->dev, "PCI resource flags BAR1 %#lx\n",
848+ pci_resource_flags(pdev, 1));
849+ dev_dbg(&pdev->dev, "PCI resource len %#llx\n",
850+ (unsigned long long)pci_resource_len(pdev, 1));
851+ dev_dbg(&pdev->dev, "PCI resource start %#llx\n",
852+ (unsigned long long)pci_resource_start(pdev, 1));
853+
854+ if (!(pci_resource_flags(pdev, 0) & IORESOURCE_MEM) ||
855+ !(pci_resource_flags(pdev, 1) & IORESOURCE_MEM)) {
856+ dev_err(&pdev->dev, "PCI BAR region not MMIO\n");
857+ ret = -ENOMEM;
858+ goto err_free_device;
859+ }
860+
861+ ret = pci_request_regions(pdev, DRV_NAME);
862+ if (ret) {
863+ dev_err(&pdev->dev, "cannot request PCI resources\n");
864+ goto err_disable_pdev;
865+ }
866+
867+ /* Enable 64-Bit DMA */
868+ if (pci_set_dma_mask(pdev, DMA_BIT_MASK(64)) == 0) {
869+ ret = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
870+ if (ret != 0) {
871+ dev_err(&pdev->dev,
872+ "pci_set_consistent_dma_mask failed\n");
873+ goto err_free_resource;
874+ }
875+ } else {
876+ ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
877+ if (ret != 0) {
878+ dev_err(&pdev->dev,
879+ "pci_set_dma_mask failed\n");
880+ goto err_free_resource;
881+ }
882+ }
883+
884+ pci_set_master(pdev);
885+
886+ /* Map register space */
887+ start = pci_resource_start(dev->pdev, PVRDMA_PCI_RESOURCE_REG);
888+ len = pci_resource_len(dev->pdev, PVRDMA_PCI_RESOURCE_REG);
889+ dev->regs = ioremap(start, len);
890+ if (!dev->regs) {
891+ dev_err(&pdev->dev, "register mapping failed\n");
892+ ret = -ENOMEM;
893+ goto err_free_resource;
894+ }
895+
896+ /* Setup per-device UAR. */
897+ dev->driver_uar.index = 0;
898+ dev->driver_uar.pfn =
899+ pci_resource_start(dev->pdev, PVRDMA_PCI_RESOURCE_UAR) >>
900+ PAGE_SHIFT;
901+ dev->driver_uar.map =
902+ ioremap(dev->driver_uar.pfn << PAGE_SHIFT, PAGE_SIZE);
903+ if (!dev->driver_uar.map) {
904+ dev_err(&pdev->dev, "failed to remap UAR pages\n");
905+ ret = -ENOMEM;
906+ goto err_unmap_regs;
907+ }
908+
909+ version = pvrdma_read_reg(dev, PVRDMA_REG_VERSION);
910+ dev_info(&pdev->dev, "device version %d, driver version %d\n",
911+ version, PVRDMA_VERSION);
912+ if (version < PVRDMA_VERSION) {
913+ dev_err(&pdev->dev, "incompatible device version\n");
914+ goto err_uar_unmap;
915+ }
916+
917+ dev->dsr = dma_alloc_coherent(&pdev->dev, sizeof(*dev->dsr),
918+ &dev->dsrbase, GFP_KERNEL);
919+ if (!dev->dsr) {
920+ dev_err(&pdev->dev, "failed to allocate shared region\n");
921+ ret = -ENOMEM;
922+ goto err_uar_unmap;
923+ }
924+
925+ /* Setup the shared region */
926+ memset(dev->dsr, 0, sizeof(*dev->dsr));
927+ dev->dsr->driver_version = PVRDMA_VERSION;
928+ dev->dsr->gos_info.gos_bits = sizeof(void *) == 4 ?
929+ PVRDMA_GOS_BITS_32 :
930+ PVRDMA_GOS_BITS_64;
931+ dev->dsr->gos_info.gos_type = PVRDMA_GOS_TYPE_LINUX;
932+ dev->dsr->gos_info.gos_ver = 1;
933+ dev->dsr->uar_pfn = dev->driver_uar.pfn;
934+
935+ /* Command slot. */
936+ dev->cmd_slot = dma_alloc_coherent(&pdev->dev, PAGE_SIZE,
937+ &slot_dma, GFP_KERNEL);
938+ if (!dev->cmd_slot) {
939+ ret = -ENOMEM;
940+ goto err_free_dsr;
941+ }
942+
943+ dev->dsr->cmd_slot_dma = (u64)slot_dma;
944+
945+ /* Response slot. */
946+ dev->resp_slot = dma_alloc_coherent(&pdev->dev, PAGE_SIZE,
947+ &slot_dma, GFP_KERNEL);
948+ if (!dev->resp_slot) {
949+ ret = -ENOMEM;
950+ goto err_free_slots;
951+ }
952+
953+ dev->dsr->resp_slot_dma = (u64)slot_dma;
954+
955+ /* Async event ring */
956+ dev->dsr->async_ring_pages.num_pages = 4;
957+ ret = pvrdma_page_dir_init(dev, &dev->async_pdir,
958+ dev->dsr->async_ring_pages.num_pages, true);
959+ if (ret)
960+ goto err_free_slots;
961+ dev->async_ring_state = dev->async_pdir.pages[0];
962+ dev->dsr->async_ring_pages.pdir_dma = dev->async_pdir.dir_dma;
963+
964+ /* CQ notification ring */
965+ dev->dsr->cq_ring_pages.num_pages = 4;
966+ ret = pvrdma_page_dir_init(dev, &dev->cq_pdir,
967+ dev->dsr->cq_ring_pages.num_pages, true);
968+ if (ret)
969+ goto err_free_async_ring;
970+ dev->cq_ring_state = dev->cq_pdir.pages[0];
971+ dev->dsr->cq_ring_pages.pdir_dma = dev->cq_pdir.dir_dma;
972+
973+ /*
974+ * Write the PA of the shared region to the device. The writes must be
975+ * ordered such that the high bits are written last. When the writes
976+ * complete, the device will have filled out the capabilities.
977+ */
978+
979+ pvrdma_write_reg(dev, PVRDMA_REG_DSRLOW, (u32)dev->dsrbase);
980+ pvrdma_write_reg(dev, PVRDMA_REG_DSRHIGH,
981+ (u32)((u64)(dev->dsrbase) >> 32));
982+
983+ /* Make sure the write is complete before reading status. */
984+ mb();
985+
986+ /* Currently, the driver only supports RoCE mode. */
987+ if (dev->dsr->caps.mode != PVRDMA_DEVICE_MODE_ROCE) {
988+ dev_err(&pdev->dev, "unsupported transport %d\n",
989+ dev->dsr->caps.mode);
990+ ret = -EFAULT;
991+ goto err_free_cq_ring;
992+ }
993+
994+ /* Currently, the driver only supports RoCE V1. */
995+ if (!(dev->dsr->caps.gid_types & PVRDMA_GID_TYPE_FLAG_ROCE_V1)) {
996+ dev_err(&pdev->dev, "driver needs RoCE v1 support\n");
997+ ret = -EFAULT;
998+ goto err_free_cq_ring;
999+ }
1000+
1001+ /* Paired vmxnet3 will have same bus, slot. But func will be 0 */
1002+ pdev_net = pci_get_slot(pdev->bus, PCI_DEVFN(PCI_SLOT(pdev->devfn), 0));
1003+ if (!pdev_net) {
1004+ dev_err(&pdev->dev, "failed to find paired net device\n");
1005+ ret = -ENODEV;
1006+ goto err_free_cq_ring;
1007+ }
1008+
1009+ if (pdev_net->vendor != PCI_VENDOR_ID_VMWARE ||
1010+ pdev_net->device != PCI_DEVICE_ID_VMWARE_VMXNET3) {
1011+ dev_err(&pdev->dev, "failed to find paired vmxnet3 device\n");
1012+ pci_dev_put(pdev_net);
1013+ ret = -ENODEV;
1014+ goto err_free_cq_ring;
1015+ }
1016+
1017+ dev->netdev = pci_get_drvdata(pdev_net);
1018+ pci_dev_put(pdev_net);
1019+ if (!dev->netdev) {
1020+ dev_err(&pdev->dev, "failed to get vmxnet3 device\n");
1021+ ret = -ENODEV;
1022+ goto err_free_cq_ring;
1023+ }
1024+
1025+ dev_info(&pdev->dev, "paired device to %s\n", dev->netdev->name);
1026+
1027+ /* Interrupt setup */
1028+ ret = pvrdma_alloc_intrs(dev);
1029+ if (ret) {
1030+ dev_err(&pdev->dev, "failed to allocate interrupts\n");
1031+ ret = -ENOMEM;
1032+ goto err_netdevice;
1033+ }
1034+
1035+ /* Allocate UAR table. */
1036+ ret = pvrdma_uar_table_init(dev);
1037+ if (ret) {
1038+ dev_err(&pdev->dev, "failed to allocate UAR table\n");
1039+ ret = -ENOMEM;
1040+ goto err_free_intrs;
1041+ }
1042+
1043+ /* Allocate GID table */
1044+ dev->sgid_tbl = kcalloc(dev->dsr->caps.gid_tbl_len,
1045+ sizeof(union ib_gid), GFP_KERNEL);
1046+ if (!dev->sgid_tbl) {
1047+ ret = -ENOMEM;
1048+ goto err_free_uar_table;
1049+ }
1050+ dev_dbg(&pdev->dev, "gid table len %d\n", dev->dsr->caps.gid_tbl_len);
1051+
1052+ pvrdma_enable_intrs(dev);
1053+
1054+ /* Activate pvrdma device */
1055+ pvrdma_write_reg(dev, PVRDMA_REG_CTL, PVRDMA_DEVICE_CTL_ACTIVATE);
1056+
1057+ /* Make sure the write is complete before reading status. */
1058+ mb();
1059+
1060+ /* Check if device was successfully activated */
1061+ ret = pvrdma_read_reg(dev, PVRDMA_REG_ERR);
1062+ if (ret != 0) {
1063+ dev_err(&pdev->dev, "failed to activate device\n");
1064+ ret = -EFAULT;
1065+ goto err_disable_intr;
1066+ }
1067+
1068+ /* Register IB device */
1069+ ret = pvrdma_register_device(dev);
1070+ if (ret) {
1071+ dev_err(&pdev->dev, "failed to register IB device\n");
1072+ goto err_disable_intr;
1073+ }
1074+
1075+ dev->nb_netdev.notifier_call = pvrdma_netdevice_event;
1076+ ret = register_netdevice_notifier(&dev->nb_netdev);
1077+ if (ret) {
1078+ dev_err(&pdev->dev, "failed to register netdevice events\n");
1079+ goto err_unreg_ibdev;
1080+ }
1081+
1082+ dev_info(&pdev->dev, "attached to device\n");
1083+ return 0;
1084+
1085+err_unreg_ibdev:
1086+ ib_unregister_device(&dev->ib_dev);
1087+err_disable_intr:
1088+ pvrdma_disable_intrs(dev);
1089+ kfree(dev->sgid_tbl);
1090+err_free_uar_table:
1091+ pvrdma_uar_table_cleanup(dev);
1092+err_free_intrs:
1093+ pvrdma_free_irq(dev);
1094+ pvrdma_disable_msi_all(dev);
1095+err_netdevice:
1096+ unregister_netdevice_notifier(&dev->nb_netdev);
1097+err_free_cq_ring:
1098+ pvrdma_page_dir_cleanup(dev, &dev->cq_pdir);
1099+err_free_async_ring:
1100+ pvrdma_page_dir_cleanup(dev, &dev->async_pdir);
1101+err_free_slots:
1102+ pvrdma_free_slots(dev);
1103+err_free_dsr:
1104+ dma_free_coherent(&pdev->dev, sizeof(*dev->dsr), dev->dsr,
1105+ dev->dsrbase);
1106+err_uar_unmap:
1107+ iounmap(dev->driver_uar.map);
1108+err_unmap_regs:
1109+ iounmap(dev->regs);
1110+err_free_resource:
1111+ pci_release_regions(pdev);
1112+err_disable_pdev:
1113+ pci_disable_device(pdev);
1114+ pci_set_drvdata(pdev, NULL);
1115+err_free_device:
1116+ mutex_lock(&pvrdma_device_list_lock);
1117+ list_del(&dev->device_link);
1118+ mutex_unlock(&pvrdma_device_list_lock);
1119+ ib_dealloc_device(&dev->ib_dev);
1120+ return ret;
1121+}
1122+
1123+static void pvrdma_pci_remove(struct pci_dev *pdev)
1124+{
1125+ struct pvrdma_dev *dev = pci_get_drvdata(pdev);
1126+
1127+ if (!dev)
1128+ return;
1129+
1130+ dev_info(&pdev->dev, "detaching from device\n");
1131+
1132+ unregister_netdevice_notifier(&dev->nb_netdev);
1133+ dev->nb_netdev.notifier_call = NULL;
1134+
1135+ flush_workqueue(event_wq);
1136+
1137+ /* Unregister ib device */
1138+ ib_unregister_device(&dev->ib_dev);
1139+
1140+ mutex_lock(&pvrdma_device_list_lock);
1141+ list_del(&dev->device_link);
1142+ mutex_unlock(&pvrdma_device_list_lock);
1143+
1144+ pvrdma_disable_intrs(dev);
1145+ pvrdma_free_irq(dev);
1146+ pvrdma_disable_msi_all(dev);
1147+
1148+ /* Deactivate pvrdma device */
1149+ pvrdma_write_reg(dev, PVRDMA_REG_CTL, PVRDMA_DEVICE_CTL_RESET);
1150+ pvrdma_page_dir_cleanup(dev, &dev->cq_pdir);
1151+ pvrdma_page_dir_cleanup(dev, &dev->async_pdir);
1152+ pvrdma_free_slots(dev);
1153+
1154+ iounmap(dev->regs);
1155+ kfree(dev->sgid_tbl);
1156+ kfree(dev->cq_tbl);
1157+ kfree(dev->qp_tbl);
1158+ pvrdma_uar_table_cleanup(dev);
1159+ iounmap(dev->driver_uar.map);
1160+
1161+ ib_dealloc_device(&dev->ib_dev);
1162+
1163+ /* Free pci resources */
1164+ pci_release_regions(pdev);
1165+ pci_disable_device(pdev);
1166+ pci_set_drvdata(pdev, NULL);
1167+}
1168+
1169+static struct pci_device_id pvrdma_pci_table[] = {
1170+ { PCI_DEVICE(PCI_VENDOR_ID_VMWARE, PCI_DEVICE_ID_VMWARE_PVRDMA), },
1171+ { 0 },
1172+};
1173+
1174+MODULE_DEVICE_TABLE(pci, pvrdma_pci_table);
1175+
1176+static struct pci_driver pvrdma_driver = {
1177+ .name = DRV_NAME,
1178+ .id_table = pvrdma_pci_table,
1179+ .probe = pvrdma_pci_probe,
1180+ .remove = pvrdma_pci_remove,
1181+};
1182+
1183+static int __init pvrdma_init(void)
1184+{
1185+ int err;
1186+
1187+ event_wq = alloc_ordered_workqueue("pvrdma_event_wq", WQ_MEM_RECLAIM);
1188+ if (!event_wq)
1189+ return -ENOMEM;
1190+
1191+ err = pci_register_driver(&pvrdma_driver);
1192+ if (err)
1193+ destroy_workqueue(event_wq);
1194+
1195+ return err;
1196+}
1197+
1198+static void __exit pvrdma_cleanup(void)
1199+{
1200+ pci_unregister_driver(&pvrdma_driver);
1201+
1202+ destroy_workqueue(event_wq);
1203+}
1204+
1205+module_init(pvrdma_init);
1206+module_exit(pvrdma_cleanup);
1207+
1208+MODULE_AUTHOR("VMware, Inc");
1209+MODULE_DESCRIPTION("VMware Paravirtual RDMA driver");
1210+MODULE_VERSION(DRV_VERSION);
1211+MODULE_LICENSE("Dual BSD/GPL");
drivers/infiniband/hw/vmw_pvrdma/pvrdma_misc.c+304−0
@@ -0,0 +1,304 @@
1+/*
2+ * Copyright (c) 2012-2016 VMware, Inc. All rights reserved.
3+ *
4+ * This program is free software; you can redistribute it and/or
5+ * modify it under the terms of EITHER the GNU General Public License
6+ * version 2 as published by the Free Software Foundation or the BSD
7+ * 2-Clause License. This program is distributed in the hope that it
8+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
9+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
10+ * See the GNU General Public License version 2 for more details at
11+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
12+ *
13+ * You should have received a copy of the GNU General Public License
14+ * along with this program available in the file COPYING in the main
15+ * directory of this source tree.
16+ *
17+ * The BSD 2-Clause License
18+ *
19+ * Redistribution and use in source and binary forms, with or
20+ * without modification, are permitted provided that the following
21+ * conditions are met:
22+ *
23+ * - Redistributions of source code must retain the above
24+ * copyright notice, this list of conditions and the following
25+ * disclaimer.
26+ *
27+ * - Redistributions in binary form must reproduce the above
28+ * copyright notice, this list of conditions and the following
29+ * disclaimer in the documentation and/or other materials
30+ * provided with the distribution.
31+ *
32+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
33+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
34+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
35+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
36+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
37+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
38+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
39+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
40+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
41+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
42+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
43+ * OF THE POSSIBILITY OF SUCH DAMAGE.
44+ */
45+
46+#include <linux/errno.h>
47+#include <linux/slab.h>
48+#include <linux/bitmap.h>
49+
50+#include "pvrdma.h"
51+
52+int pvrdma_page_dir_init(struct pvrdma_dev *dev, struct pvrdma_page_dir *pdir,
53+ u64 npages, bool alloc_pages)
54+{
55+ u64 i;
56+
57+ if (npages > PVRDMA_PAGE_DIR_MAX_PAGES)
58+ return -EINVAL;
59+
60+ memset(pdir, 0, sizeof(*pdir));
61+
62+ pdir->dir = dma_alloc_coherent(&dev->pdev->dev, PAGE_SIZE,
63+ &pdir->dir_dma, GFP_KERNEL);
64+ if (!pdir->dir)
65+ goto err;
66+
67+ pdir->ntables = PVRDMA_PAGE_DIR_TABLE(npages - 1) + 1;
68+ pdir->tables = kcalloc(pdir->ntables, sizeof(*pdir->tables),
69+ GFP_KERNEL);
70+ if (!pdir->tables)
71+ goto err;
72+
73+ for (i = 0; i < pdir->ntables; i++) {
74+ pdir->tables[i] = dma_alloc_coherent(&dev->pdev->dev, PAGE_SIZE,
75+ (dma_addr_t *)&pdir->dir[i],
76+ GFP_KERNEL);
77+ if (!pdir->tables[i])
78+ goto err;
79+ }
80+
81+ pdir->npages = npages;
82+
83+ if (alloc_pages) {
84+ pdir->pages = kcalloc(npages, sizeof(*pdir->pages),
85+ GFP_KERNEL);
86+ if (!pdir->pages)
87+ goto err;
88+
89+ for (i = 0; i < pdir->npages; i++) {
90+ dma_addr_t page_dma;
91+
92+ pdir->pages[i] = dma_alloc_coherent(&dev->pdev->dev,
93+ PAGE_SIZE,
94+ &page_dma,
95+ GFP_KERNEL);
96+ if (!pdir->pages[i])
97+ goto err;
98+
99+ pvrdma_page_dir_insert_dma(pdir, i, page_dma);
100+ }
101+ }
102+
103+ return 0;
104+
105+err:
106+ pvrdma_page_dir_cleanup(dev, pdir);
107+
108+ return -ENOMEM;
109+}
110+
111+static u64 *pvrdma_page_dir_table(struct pvrdma_page_dir *pdir, u64 idx)
112+{
113+ return pdir->tables[PVRDMA_PAGE_DIR_TABLE(idx)];
114+}
115+
116+dma_addr_t pvrdma_page_dir_get_dma(struct pvrdma_page_dir *pdir, u64 idx)
117+{
118+ return pvrdma_page_dir_table(pdir, idx)[PVRDMA_PAGE_DIR_PAGE(idx)];
119+}
120+
121+static void pvrdma_page_dir_cleanup_pages(struct pvrdma_dev *dev,
122+ struct pvrdma_page_dir *pdir)
123+{
124+ if (pdir->pages) {
125+ u64 i;
126+
127+ for (i = 0; i < pdir->npages && pdir->pages[i]; i++) {
128+ dma_addr_t page_dma = pvrdma_page_dir_get_dma(pdir, i);
129+
130+ dma_free_coherent(&dev->pdev->dev, PAGE_SIZE,
131+ pdir->pages[i], page_dma);
132+ }
133+
134+ kfree(pdir->pages);
135+ }
136+}
137+
138+static void pvrdma_page_dir_cleanup_tables(struct pvrdma_dev *dev,
139+ struct pvrdma_page_dir *pdir)
140+{
141+ if (pdir->tables) {
142+ int i;
143+
144+ pvrdma_page_dir_cleanup_pages(dev, pdir);
145+
146+ for (i = 0; i < pdir->ntables; i++) {
147+ u64 *table = pdir->tables[i];
148+
149+ if (table)
150+ dma_free_coherent(&dev->pdev->dev, PAGE_SIZE,
151+ table, pdir->dir[i]);
152+ }
153+
154+ kfree(pdir->tables);
155+ }
156+}
157+
158+void pvrdma_page_dir_cleanup(struct pvrdma_dev *dev,
159+ struct pvrdma_page_dir *pdir)
160+{
161+ if (pdir->dir) {
162+ pvrdma_page_dir_cleanup_tables(dev, pdir);
163+ dma_free_coherent(&dev->pdev->dev, PAGE_SIZE,
164+ pdir->dir, pdir->dir_dma);
165+ }
166+}
167+
168+int pvrdma_page_dir_insert_dma(struct pvrdma_page_dir *pdir, u64 idx,
169+ dma_addr_t daddr)
170+{
171+ u64 *table;
172+
173+ if (idx >= pdir->npages)
174+ return -EINVAL;
175+
176+ table = pvrdma_page_dir_table(pdir, idx);
177+ table[PVRDMA_PAGE_DIR_PAGE(idx)] = daddr;
178+
179+ return 0;
180+}
181+
182+int pvrdma_page_dir_insert_umem(struct pvrdma_page_dir *pdir,
183+ struct ib_umem *umem, u64 offset)
184+{
185+ u64 i = offset;
186+ int j, entry;
187+ int ret = 0, len = 0;
188+ struct scatterlist *sg;
189+
190+ if (offset >= pdir->npages)
191+ return -EINVAL;
192+
193+ for_each_sg(umem->sg_head.sgl, sg, umem->nmap, entry) {
194+ len = sg_dma_len(sg) >> PAGE_SHIFT;
195+ for (j = 0; j < len; j++) {
196+ dma_addr_t addr = sg_dma_address(sg) +
197+ umem->page_size * j;
198+
199+ ret = pvrdma_page_dir_insert_dma(pdir, i, addr);
200+ if (ret)
201+ goto exit;
202+
203+ i++;
204+ }
205+ }
206+
207+exit:
208+ return ret;
209+}
210+
211+int pvrdma_page_dir_insert_page_list(struct pvrdma_page_dir *pdir,
212+ u64 *page_list,
213+ int num_pages)
214+{
215+ int i;
216+ int ret;
217+
218+ if (num_pages > pdir->npages)
219+ return -EINVAL;
220+
221+ for (i = 0; i < num_pages; i++) {
222+ ret = pvrdma_page_dir_insert_dma(pdir, i, page_list[i]);
223+ if (ret)
224+ return ret;
225+ }
226+
227+ return 0;
228+}
229+
230+void pvrdma_qp_cap_to_ib(struct ib_qp_cap *dst, const struct pvrdma_qp_cap *src)
231+{
232+ dst->max_send_wr = src->max_send_wr;
233+ dst->max_recv_wr = src->max_recv_wr;
234+ dst->max_send_sge = src->max_send_sge;
235+ dst->max_recv_sge = src->max_recv_sge;
236+ dst->max_inline_data = src->max_inline_data;
237+}
238+
239+void ib_qp_cap_to_pvrdma(struct pvrdma_qp_cap *dst, const struct ib_qp_cap *src)
240+{
241+ dst->max_send_wr = src->max_send_wr;
242+ dst->max_recv_wr = src->max_recv_wr;
243+ dst->max_send_sge = src->max_send_sge;
244+ dst->max_recv_sge = src->max_recv_sge;
245+ dst->max_inline_data = src->max_inline_data;
246+}
247+
248+void pvrdma_gid_to_ib(union ib_gid *dst, const union pvrdma_gid *src)
249+{
250+ BUILD_BUG_ON(sizeof(union pvrdma_gid) != sizeof(union ib_gid));
251+ memcpy(dst, src, sizeof(*src));
252+}
253+
254+void ib_gid_to_pvrdma(union pvrdma_gid *dst, const union ib_gid *src)
255+{
256+ BUILD_BUG_ON(sizeof(union pvrdma_gid) != sizeof(union ib_gid));
257+ memcpy(dst, src, sizeof(*src));
258+}
259+
260+void pvrdma_global_route_to_ib(struct ib_global_route *dst,
261+ const struct pvrdma_global_route *src)
262+{
263+ pvrdma_gid_to_ib(&dst->dgid, &src->dgid);
264+ dst->flow_label = src->flow_label;
265+ dst->sgid_index = src->sgid_index;
266+ dst->hop_limit = src->hop_limit;
267+ dst->traffic_class = src->traffic_class;
268+}
269+
270+void ib_global_route_to_pvrdma(struct pvrdma_global_route *dst,
271+ const struct ib_global_route *src)
272+{
273+ ib_gid_to_pvrdma(&dst->dgid, &src->dgid);
274+ dst->flow_label = src->flow_label;
275+ dst->sgid_index = src->sgid_index;
276+ dst->hop_limit = src->hop_limit;
277+ dst->traffic_class = src->traffic_class;
278+}
279+
280+void pvrdma_ah_attr_to_ib(struct ib_ah_attr *dst,
281+ const struct pvrdma_ah_attr *src)
282+{
283+ pvrdma_global_route_to_ib(&dst->grh, &src->grh);
284+ dst->dlid = src->dlid;
285+ dst->sl = src->sl;
286+ dst->src_path_bits = src->src_path_bits;
287+ dst->static_rate = src->static_rate;
288+ dst->ah_flags = src->ah_flags;
289+ dst->port_num = src->port_num;
290+ memcpy(&dst->dmac, &src->dmac, sizeof(dst->dmac));
291+}
292+
293+void ib_ah_attr_to_pvrdma(struct pvrdma_ah_attr *dst,
294+ const struct ib_ah_attr *src)
295+{
296+ ib_global_route_to_pvrdma(&dst->grh, &src->grh);
297+ dst->dlid = src->dlid;
298+ dst->sl = src->sl;
299+ dst->src_path_bits = src->src_path_bits;
300+ dst->static_rate = src->static_rate;
301+ dst->ah_flags = src->ah_flags;
302+ dst->port_num = src->port_num;
303+ memcpy(&dst->dmac, &src->dmac, sizeof(dst->dmac));
304+}
drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c+334−0
@@ -0,0 +1,334 @@
1+/*
2+ * Copyright (c) 2012-2016 VMware, Inc. All rights reserved.
3+ *
4+ * This program is free software; you can redistribute it and/or
5+ * modify it under the terms of EITHER the GNU General Public License
6+ * version 2 as published by the Free Software Foundation or the BSD
7+ * 2-Clause License. This program is distributed in the hope that it
8+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
9+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
10+ * See the GNU General Public License version 2 for more details at
11+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
12+ *
13+ * You should have received a copy of the GNU General Public License
14+ * along with this program available in the file COPYING in the main
15+ * directory of this source tree.
16+ *
17+ * The BSD 2-Clause License
18+ *
19+ * Redistribution and use in source and binary forms, with or
20+ * without modification, are permitted provided that the following
21+ * conditions are met:
22+ *
23+ * - Redistributions of source code must retain the above
24+ * copyright notice, this list of conditions and the following
25+ * disclaimer.
26+ *
27+ * - Redistributions in binary form must reproduce the above
28+ * copyright notice, this list of conditions and the following
29+ * disclaimer in the documentation and/or other materials
30+ * provided with the distribution.
31+ *
32+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
33+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
34+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
35+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
36+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
37+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
38+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
39+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
40+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
41+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
42+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
43+ * OF THE POSSIBILITY OF SUCH DAMAGE.
44+ */
45+
46+#include <linux/list.h>
47+#include <linux/slab.h>
48+
49+#include "pvrdma.h"
50+
51+/**
52+ * pvrdma_get_dma_mr - get a DMA memory region
53+ * @pd: protection domain
54+ * @acc: access flags
55+ *
56+ * @return: ib_mr pointer on success, otherwise returns an errno.
57+ */
58+struct ib_mr *pvrdma_get_dma_mr(struct ib_pd *pd, int acc)
59+{
60+ struct pvrdma_dev *dev = to_vdev(pd->device);
61+ struct pvrdma_user_mr *mr;
62+ union pvrdma_cmd_req req;
63+ union pvrdma_cmd_resp rsp;
64+ struct pvrdma_cmd_create_mr *cmd = &req.create_mr;
65+ struct pvrdma_cmd_create_mr_resp *resp = &rsp.create_mr_resp;
66+ int ret;
67+
68+ /* Support only LOCAL_WRITE flag for DMA MRs */
69+ if (acc & ~IB_ACCESS_LOCAL_WRITE) {
70+ dev_warn(&dev->pdev->dev,
71+ "unsupported dma mr access flags %#x\n", acc);
72+ return ERR_PTR(-EOPNOTSUPP);
73+ }
74+
75+ mr = kzalloc(sizeof(*mr), GFP_KERNEL);
76+ if (!mr)
77+ return ERR_PTR(-ENOMEM);
78+
79+ memset(cmd, 0, sizeof(*cmd));
80+ cmd->hdr.cmd = PVRDMA_CMD_CREATE_MR;
81+ cmd->pd_handle = to_vpd(pd)->pd_handle;
82+ cmd->access_flags = acc;
83+ cmd->flags = PVRDMA_MR_FLAG_DMA;
84+
85+ ret = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_CREATE_MR_RESP);
86+ if (ret < 0) {
87+ dev_warn(&dev->pdev->dev,
88+ "could not get DMA mem region, error: %d\n", ret);
89+ kfree(mr);
90+ return ERR_PTR(ret);
91+ }
92+
93+ mr->mmr.mr_handle = resp->mr_handle;
94+ mr->ibmr.lkey = resp->lkey;
95+ mr->ibmr.rkey = resp->rkey;
96+
97+ return &mr->ibmr;
98+}
99+
100+/**
101+ * pvrdma_reg_user_mr - register a userspace memory region
102+ * @pd: protection domain
103+ * @start: starting address
104+ * @length: length of region
105+ * @virt_addr: I/O virtual address
106+ * @access_flags: access flags for memory region
107+ * @udata: user data
108+ *
109+ * @return: ib_mr pointer on success, otherwise returns an errno.
110+ */
111+struct ib_mr *pvrdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
112+ u64 virt_addr, int access_flags,
113+ struct ib_udata *udata)
114+{
115+ struct pvrdma_dev *dev = to_vdev(pd->device);
116+ struct pvrdma_user_mr *mr = NULL;
117+ struct ib_umem *umem;
118+ union pvrdma_cmd_req req;
119+ union pvrdma_cmd_resp rsp;
120+ struct pvrdma_cmd_create_mr *cmd = &req.create_mr;
121+ struct pvrdma_cmd_create_mr_resp *resp = &rsp.create_mr_resp;
122+ int nchunks;
123+ int ret;
124+ int entry;
125+ struct scatterlist *sg;
126+
127+ if (length == 0 || length > dev->dsr->caps.max_mr_size) {
128+ dev_warn(&dev->pdev->dev, "invalid mem region length\n");
129+ return ERR_PTR(-EINVAL);
130+ }
131+
132+ umem = ib_umem_get(pd->uobject->context, start,
133+ length, access_flags, 0);
134+ if (IS_ERR(umem)) {
135+ dev_warn(&dev->pdev->dev,
136+ "could not get umem for mem region\n");
137+ return ERR_CAST(umem);
138+ }
139+
140+ nchunks = 0;
141+ for_each_sg(umem->sg_head.sgl, sg, umem->nmap, entry)
142+ nchunks += sg_dma_len(sg) >> PAGE_SHIFT;
143+
144+ if (nchunks < 0 || nchunks > PVRDMA_PAGE_DIR_MAX_PAGES) {
145+ dev_warn(&dev->pdev->dev, "overflow %d pages in mem region\n",
146+ nchunks);
147+ ret = -EINVAL;
148+ goto err_umem;
149+ }
150+
151+ mr = kzalloc(sizeof(*mr), GFP_KERNEL);
152+ if (!mr) {
153+ ret = -ENOMEM;
154+ goto err_umem;
155+ }
156+
157+ mr->mmr.iova = virt_addr;
158+ mr->mmr.size = length;
159+ mr->umem = umem;
160+
161+ ret = pvrdma_page_dir_init(dev, &mr->pdir, nchunks, false);
162+ if (ret) {
163+ dev_warn(&dev->pdev->dev,
164+ "could not allocate page directory\n");
165+ goto err_umem;
166+ }
167+
168+ ret = pvrdma_page_dir_insert_umem(&mr->pdir, mr->umem, 0);
169+ if (ret)
170+ goto err_pdir;
171+
172+ memset(cmd, 0, sizeof(*cmd));
173+ cmd->hdr.cmd = PVRDMA_CMD_CREATE_MR;
174+ cmd->start = start;
175+ cmd->length = length;
176+ cmd->pd_handle = to_vpd(pd)->pd_handle;
177+ cmd->access_flags = access_flags;
178+ cmd->nchunks = nchunks;
179+ cmd->pdir_dma = mr->pdir.dir_dma;
180+
181+ ret = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_CREATE_MR_RESP);
182+ if (ret < 0) {
183+ dev_warn(&dev->pdev->dev,
184+ "could not register mem region, error: %d\n", ret);
185+ goto err_pdir;
186+ }
187+
188+ mr->mmr.mr_handle = resp->mr_handle;
189+ mr->ibmr.lkey = resp->lkey;
190+ mr->ibmr.rkey = resp->rkey;
191+
192+ return &mr->ibmr;
193+
194+err_pdir:
195+ pvrdma_page_dir_cleanup(dev, &mr->pdir);
196+err_umem:
197+ ib_umem_release(umem);
198+ kfree(mr);
199+
200+ return ERR_PTR(ret);
201+}
202+
203+/**
204+ * pvrdma_alloc_mr - allocate a memory region
205+ * @pd: protection domain
206+ * @mr_type: type of memory region
207+ * @max_num_sg: maximum number of pages
208+ *
209+ * @return: ib_mr pointer on success, otherwise returns an errno.
210+ */
211+struct ib_mr *pvrdma_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
212+ u32 max_num_sg)
213+{
214+ struct pvrdma_dev *dev = to_vdev(pd->device);
215+ struct pvrdma_user_mr *mr;
216+ union pvrdma_cmd_req req;
217+ union pvrdma_cmd_resp rsp;
218+ struct pvrdma_cmd_create_mr *cmd = &req.create_mr;
219+ struct pvrdma_cmd_create_mr_resp *resp = &rsp.create_mr_resp;
220+ int size = max_num_sg * sizeof(u64);
221+ int ret;
222+
223+ if (mr_type != IB_MR_TYPE_MEM_REG ||
224+ max_num_sg > PVRDMA_MAX_FAST_REG_PAGES)
225+ return ERR_PTR(-EINVAL);
226+
227+ mr = kzalloc(sizeof(*mr), GFP_KERNEL);
228+ if (!mr)
229+ return ERR_PTR(-ENOMEM);
230+
231+ mr->pages = kzalloc(size, GFP_KERNEL);
232+ if (!mr->pages) {
233+ ret = -ENOMEM;
234+ goto freemr;
235+ }
236+
237+ ret = pvrdma_page_dir_init(dev, &mr->pdir, max_num_sg, false);
238+ if (ret) {
239+ dev_warn(&dev->pdev->dev,
240+ "failed to allocate page dir for mr\n");
241+ ret = -ENOMEM;
242+ goto freepages;
243+ }
244+
245+ memset(cmd, 0, sizeof(*cmd));
246+ cmd->hdr.cmd = PVRDMA_CMD_CREATE_MR;
247+ cmd->pd_handle = to_vpd(pd)->pd_handle;
248+ cmd->access_flags = 0;
249+ cmd->flags = PVRDMA_MR_FLAG_FRMR;
250+ cmd->nchunks = max_num_sg;
251+
252+ ret = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_CREATE_MR_RESP);
253+ if (ret < 0) {
254+ dev_warn(&dev->pdev->dev,
255+ "could not create FR mem region, error: %d\n", ret);
256+ goto freepdir;
257+ }
258+
259+ mr->max_pages = max_num_sg;
260+ mr->mmr.mr_handle = resp->mr_handle;
261+ mr->ibmr.lkey = resp->lkey;
262+ mr->ibmr.rkey = resp->rkey;
263+ mr->page_shift = PAGE_SHIFT;
264+ mr->umem = NULL;
265+
266+ return &mr->ibmr;
267+
268+freepdir:
269+ pvrdma_page_dir_cleanup(dev, &mr->pdir);
270+freepages:
271+ kfree(mr->pages);
272+freemr:
273+ kfree(mr);
274+ return ERR_PTR(ret);
275+}
276+
277+/**
278+ * pvrdma_dereg_mr - deregister a memory region
279+ * @ibmr: memory region
280+ *
281+ * @return: 0 on success.
282+ */
283+int pvrdma_dereg_mr(struct ib_mr *ibmr)
284+{
285+ struct pvrdma_user_mr *mr = to_vmr(ibmr);
286+ struct pvrdma_dev *dev = to_vdev(ibmr->device);
287+ union pvrdma_cmd_req req;
288+ struct pvrdma_cmd_destroy_mr *cmd = &req.destroy_mr;
289+ int ret;
290+
291+ memset(cmd, 0, sizeof(*cmd));
292+ cmd->hdr.cmd = PVRDMA_CMD_DESTROY_MR;
293+ cmd->mr_handle = mr->mmr.mr_handle;
294+ ret = pvrdma_cmd_post(dev, &req, NULL, 0);
295+ if (ret < 0)
296+ dev_warn(&dev->pdev->dev,
297+ "could not deregister mem region, error: %d\n", ret);
298+
299+ pvrdma_page_dir_cleanup(dev, &mr->pdir);
300+ if (mr->umem)
301+ ib_umem_release(mr->umem);
302+
303+ kfree(mr->pages);
304+ kfree(mr);
305+
306+ return 0;
307+}
308+
309+static int pvrdma_set_page(struct ib_mr *ibmr, u64 addr)
310+{
311+ struct pvrdma_user_mr *mr = to_vmr(ibmr);
312+
313+ if (mr->npages == mr->max_pages)
314+ return -ENOMEM;
315+
316+ mr->pages[mr->npages++] = addr;
317+ return 0;
318+}
319+
320+int pvrdma_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
321+ unsigned int *sg_offset)
322+{
323+ struct pvrdma_user_mr *mr = to_vmr(ibmr);
324+ struct pvrdma_dev *dev = to_vdev(ibmr->device);
325+ int ret;
326+
327+ mr->npages = 0;
328+
329+ ret = ib_sg_to_pages(ibmr, sg, sg_nents, sg_offset, pvrdma_set_page);
330+ if (ret < 0)
331+ dev_warn(&dev->pdev->dev, "could not map sg to pages\n");
332+
333+ return ret;
334+}
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c+972−0
@@ -0,0 +1,972 @@
1+/*
2+ * Copyright (c) 2012-2016 VMware, Inc. All rights reserved.
3+ *
4+ * This program is free software; you can redistribute it and/or
5+ * modify it under the terms of EITHER the GNU General Public License
6+ * version 2 as published by the Free Software Foundation or the BSD
7+ * 2-Clause License. This program is distributed in the hope that it
8+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
9+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
10+ * See the GNU General Public License version 2 for more details at
11+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
12+ *
13+ * You should have received a copy of the GNU General Public License
14+ * along with this program available in the file COPYING in the main
15+ * directory of this source tree.
16+ *
17+ * The BSD 2-Clause License
18+ *
19+ * Redistribution and use in source and binary forms, with or
20+ * without modification, are permitted provided that the following
21+ * conditions are met:
22+ *
23+ * - Redistributions of source code must retain the above
24+ * copyright notice, this list of conditions and the following
25+ * disclaimer.
26+ *
27+ * - Redistributions in binary form must reproduce the above
28+ * copyright notice, this list of conditions and the following
29+ * disclaimer in the documentation and/or other materials
30+ * provided with the distribution.
31+ *
32+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
33+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
34+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
35+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
36+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
37+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
38+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
39+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
40+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
41+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
42+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
43+ * OF THE POSSIBILITY OF SUCH DAMAGE.
44+ */
45+
46+#include <asm/page.h>
47+#include <linux/io.h>
48+#include <linux/wait.h>
49+#include <rdma/ib_addr.h>
50+#include <rdma/ib_smi.h>
51+#include <rdma/ib_user_verbs.h>
52+
53+#include "pvrdma.h"
54+
55+static inline void get_cqs(struct pvrdma_qp *qp, struct pvrdma_cq **send_cq,
56+ struct pvrdma_cq **recv_cq)
57+{
58+ *send_cq = to_vcq(qp->ibqp.send_cq);
59+ *recv_cq = to_vcq(qp->ibqp.recv_cq);
60+}
61+
62+static void pvrdma_lock_cqs(struct pvrdma_cq *scq, struct pvrdma_cq *rcq,
63+ unsigned long *scq_flags,
64+ unsigned long *rcq_flags)
65+ __acquires(scq->cq_lock) __acquires(rcq->cq_lock)
66+{
67+ if (scq == rcq) {
68+ spin_lock_irqsave(&scq->cq_lock, *scq_flags);
69+ __acquire(rcq->cq_lock);
70+ } else if (scq->cq_handle < rcq->cq_handle) {
71+ spin_lock_irqsave(&scq->cq_lock, *scq_flags);
72+ spin_lock_irqsave_nested(&rcq->cq_lock, *rcq_flags,
73+ SINGLE_DEPTH_NESTING);
74+ } else {
75+ spin_lock_irqsave(&rcq->cq_lock, *rcq_flags);
76+ spin_lock_irqsave_nested(&scq->cq_lock, *scq_flags,
77+ SINGLE_DEPTH_NESTING);
78+ }
79+}
80+
81+static void pvrdma_unlock_cqs(struct pvrdma_cq *scq, struct pvrdma_cq *rcq,
82+ unsigned long *scq_flags,
83+ unsigned long *rcq_flags)
84+ __releases(scq->cq_lock) __releases(rcq->cq_lock)
85+{
86+ if (scq == rcq) {
87+ __release(rcq->cq_lock);
88+ spin_unlock_irqrestore(&scq->cq_lock, *scq_flags);
89+ } else if (scq->cq_handle < rcq->cq_handle) {
90+ spin_unlock_irqrestore(&rcq->cq_lock, *rcq_flags);
91+ spin_unlock_irqrestore(&scq->cq_lock, *scq_flags);
92+ } else {
93+ spin_unlock_irqrestore(&scq->cq_lock, *scq_flags);
94+ spin_unlock_irqrestore(&rcq->cq_lock, *rcq_flags);
95+ }
96+}
97+
98+static void pvrdma_reset_qp(struct pvrdma_qp *qp)
99+{
100+ struct pvrdma_cq *scq, *rcq;
101+ unsigned long scq_flags, rcq_flags;
102+
103+ /* Clean up cqes */
104+ get_cqs(qp, &scq, &rcq);
105+ pvrdma_lock_cqs(scq, rcq, &scq_flags, &rcq_flags);
106+
107+ _pvrdma_flush_cqe(qp, scq);
108+ if (scq != rcq)
109+ _pvrdma_flush_cqe(qp, rcq);
110+
111+ pvrdma_unlock_cqs(scq, rcq, &scq_flags, &rcq_flags);
112+
113+ /*
114+ * Reset queuepair. The checks are because usermode queuepairs won't
115+ * have kernel ringstates.
116+ */
117+ if (qp->rq.ring) {
118+ atomic_set(&qp->rq.ring->cons_head, 0);
119+ atomic_set(&qp->rq.ring->prod_tail, 0);
120+ }
121+ if (qp->sq.ring) {
122+ atomic_set(&qp->sq.ring->cons_head, 0);
123+ atomic_set(&qp->sq.ring->prod_tail, 0);
124+ }
125+}
126+
127+static int pvrdma_set_rq_size(struct pvrdma_dev *dev,
128+ struct ib_qp_cap *req_cap,
129+ struct pvrdma_qp *qp)
130+{
131+ if (req_cap->max_recv_wr > dev->dsr->caps.max_qp_wr ||
132+ req_cap->max_recv_sge > dev->dsr->caps.max_sge) {
133+ dev_warn(&dev->pdev->dev, "recv queue size invalid\n");
134+ return -EINVAL;
135+ }
136+
137+ qp->rq.wqe_cnt = roundup_pow_of_two(max(1U, req_cap->max_recv_wr));
138+ qp->rq.max_sg = roundup_pow_of_two(max(1U, req_cap->max_recv_sge));
139+
140+ /* Write back */
141+ req_cap->max_recv_wr = qp->rq.wqe_cnt;
142+ req_cap->max_recv_sge = qp->rq.max_sg;
143+
144+ qp->rq.wqe_size = roundup_pow_of_two(sizeof(struct pvrdma_rq_wqe_hdr) +
145+ sizeof(struct pvrdma_sge) *
146+ qp->rq.max_sg);
147+ qp->npages_recv = (qp->rq.wqe_cnt * qp->rq.wqe_size + PAGE_SIZE - 1) /
148+ PAGE_SIZE;
149+
150+ return 0;
151+}
152+
153+static int pvrdma_set_sq_size(struct pvrdma_dev *dev, struct ib_qp_cap *req_cap,
154+ enum ib_qp_type type, struct pvrdma_qp *qp)
155+{
156+ if (req_cap->max_send_wr > dev->dsr->caps.max_qp_wr ||
157+ req_cap->max_send_sge > dev->dsr->caps.max_sge) {
158+ dev_warn(&dev->pdev->dev, "send queue size invalid\n");
159+ return -EINVAL;
160+ }
161+
162+ qp->sq.wqe_cnt = roundup_pow_of_two(max(1U, req_cap->max_send_wr));
163+ qp->sq.max_sg = roundup_pow_of_two(max(1U, req_cap->max_send_sge));
164+
165+ /* Write back */
166+ req_cap->max_send_wr = qp->sq.wqe_cnt;
167+ req_cap->max_send_sge = qp->sq.max_sg;
168+
169+ qp->sq.wqe_size = roundup_pow_of_two(sizeof(struct pvrdma_sq_wqe_hdr) +
170+ sizeof(struct pvrdma_sge) *
171+ qp->sq.max_sg);
172+ /* Note: one extra page for the header. */
173+ qp->npages_send = 1 + (qp->sq.wqe_cnt * qp->sq.wqe_size +
174+ PAGE_SIZE - 1) / PAGE_SIZE;
175+
176+ return 0;
177+}
178+
179+/**
180+ * pvrdma_create_qp - create queue pair
181+ * @pd: protection domain
182+ * @init_attr: queue pair attributes
183+ * @udata: user data
184+ *
185+ * @return: the ib_qp pointer on success, otherwise returns an errno.
186+ */
187+struct ib_qp *pvrdma_create_qp(struct ib_pd *pd,
188+ struct ib_qp_init_attr *init_attr,
189+ struct ib_udata *udata)
190+{
191+ struct pvrdma_qp *qp = NULL;
192+ struct pvrdma_dev *dev = to_vdev(pd->device);
193+ union pvrdma_cmd_req req;
194+ union pvrdma_cmd_resp rsp;
195+ struct pvrdma_cmd_create_qp *cmd = &req.create_qp;
196+ struct pvrdma_cmd_create_qp_resp *resp = &rsp.create_qp_resp;
197+ struct pvrdma_create_qp ucmd;
198+ unsigned long flags;
199+ int ret;
200+
201+ if (init_attr->create_flags) {
202+ dev_warn(&dev->pdev->dev,
203+ "invalid create queuepair flags %#x\n",
204+ init_attr->create_flags);
205+ return ERR_PTR(-EINVAL);
206+ }
207+
208+ if (init_attr->qp_type != IB_QPT_RC &&
209+ init_attr->qp_type != IB_QPT_UD &&
210+ init_attr->qp_type != IB_QPT_GSI) {
211+ dev_warn(&dev->pdev->dev, "queuepair type %d not supported\n",
212+ init_attr->qp_type);
213+ return ERR_PTR(-EINVAL);
214+ }
215+
216+ if (!atomic_add_unless(&dev->num_qps, 1, dev->dsr->caps.max_qp))
217+ return ERR_PTR(-ENOMEM);
218+
219+ switch (init_attr->qp_type) {
220+ case IB_QPT_GSI:
221+ if (init_attr->port_num == 0 ||
222+ init_attr->port_num > pd->device->phys_port_cnt ||
223+ udata) {
224+ dev_warn(&dev->pdev->dev, "invalid queuepair attrs\n");
225+ ret = -EINVAL;
226+ goto err_qp;
227+ }
228+ /* fall through */
229+ case IB_QPT_RC:
230+ case IB_QPT_UD:
231+ qp = kzalloc(sizeof(*qp), GFP_KERNEL);
232+ if (!qp) {
233+ ret = -ENOMEM;
234+ goto err_qp;
235+ }
236+
237+ spin_lock_init(&qp->sq.lock);
238+ spin_lock_init(&qp->rq.lock);
239+ mutex_init(&qp->mutex);
240+ atomic_set(&qp->refcnt, 1);
241+ init_waitqueue_head(&qp->wait);
242+
243+ qp->state = IB_QPS_RESET;
244+
245+ if (pd->uobject && udata) {
246+ dev_dbg(&dev->pdev->dev,
247+ "create queuepair from user space\n");
248+
249+ if (ib_copy_from_udata(&ucmd, udata, sizeof(ucmd))) {
250+ ret = -EFAULT;
251+ goto err_qp;
252+ }
253+
254+ /* set qp->sq.wqe_cnt, shift, buf_size.. */
255+ qp->rumem = ib_umem_get(pd->uobject->context,
256+ ucmd.rbuf_addr,
257+ ucmd.rbuf_size, 0, 0);
258+ if (IS_ERR(qp->rumem)) {
259+ ret = PTR_ERR(qp->rumem);
260+ goto err_qp;
261+ }
262+
263+ qp->sumem = ib_umem_get(pd->uobject->context,
264+ ucmd.sbuf_addr,
265+ ucmd.sbuf_size, 0, 0);
266+ if (IS_ERR(qp->sumem)) {
267+ ib_umem_release(qp->rumem);
268+ ret = PTR_ERR(qp->sumem);
269+ goto err_qp;
270+ }
271+
272+ qp->npages_send = ib_umem_page_count(qp->sumem);
273+ qp->npages_recv = ib_umem_page_count(qp->rumem);
274+ qp->npages = qp->npages_send + qp->npages_recv;
275+ } else {
276+ qp->is_kernel = true;
277+
278+ ret = pvrdma_set_sq_size(to_vdev(pd->device),
279+ &init_attr->cap,
280+ init_attr->qp_type, qp);
281+ if (ret)
282+ goto err_qp;
283+
284+ ret = pvrdma_set_rq_size(to_vdev(pd->device),
285+ &init_attr->cap, qp);
286+ if (ret)
287+ goto err_qp;
288+
289+ qp->npages = qp->npages_send + qp->npages_recv;
290+
291+ /* Skip header page. */
292+ qp->sq.offset = PAGE_SIZE;
293+
294+ /* Recv queue pages are after send pages. */
295+ qp->rq.offset = qp->npages_send * PAGE_SIZE;
296+ }
297+
298+ if (qp->npages < 0 || qp->npages > PVRDMA_PAGE_DIR_MAX_PAGES) {
299+ dev_warn(&dev->pdev->dev,
300+ "overflow pages in queuepair\n");
301+ ret = -EINVAL;
302+ goto err_umem;
303+ }
304+
305+ ret = pvrdma_page_dir_init(dev, &qp->pdir, qp->npages,
306+ qp->is_kernel);
307+ if (ret) {
308+ dev_warn(&dev->pdev->dev,
309+ "could not allocate page directory\n");
310+ goto err_umem;
311+ }
312+
313+ if (!qp->is_kernel) {
314+ pvrdma_page_dir_insert_umem(&qp->pdir, qp->sumem, 0);
315+ pvrdma_page_dir_insert_umem(&qp->pdir, qp->rumem,
316+ qp->npages_send);
317+ } else {
318+ /* Ring state is always the first page. */
319+ qp->sq.ring = qp->pdir.pages[0];
320+ qp->rq.ring = &qp->sq.ring[1];
321+ }
322+ break;
323+ default:
324+ ret = -EINVAL;
325+ goto err_qp;
326+ }
327+
328+ /* Not supported */
329+ init_attr->cap.max_inline_data = 0;
330+
331+ memset(cmd, 0, sizeof(*cmd));
332+ cmd->hdr.cmd = PVRDMA_CMD_CREATE_QP;
333+ cmd->pd_handle = to_vpd(pd)->pd_handle;
334+ cmd->send_cq_handle = to_vcq(init_attr->send_cq)->cq_handle;
335+ cmd->recv_cq_handle = to_vcq(init_attr->recv_cq)->cq_handle;
336+ cmd->max_send_wr = init_attr->cap.max_send_wr;
337+ cmd->max_recv_wr = init_attr->cap.max_recv_wr;
338+ cmd->max_send_sge = init_attr->cap.max_send_sge;
339+ cmd->max_recv_sge = init_attr->cap.max_recv_sge;
340+ cmd->max_inline_data = init_attr->cap.max_inline_data;
341+ cmd->sq_sig_all = (init_attr->sq_sig_type == IB_SIGNAL_ALL_WR) ? 1 : 0;
342+ cmd->qp_type = ib_qp_type_to_pvrdma(init_attr->qp_type);
343+ cmd->access_flags = IB_ACCESS_LOCAL_WRITE;
344+ cmd->total_chunks = qp->npages;
345+ cmd->send_chunks = qp->npages_send - 1;
346+ cmd->pdir_dma = qp->pdir.dir_dma;
347+
348+ dev_dbg(&dev->pdev->dev, "create queuepair with %d, %d, %d, %d\n",
349+ cmd->max_send_wr, cmd->max_recv_wr, cmd->max_send_sge,
350+ cmd->max_recv_sge);
351+
352+ ret = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_CREATE_QP_RESP);
353+ if (ret < 0) {
354+ dev_warn(&dev->pdev->dev,
355+ "could not create queuepair, error: %d\n", ret);
356+ goto err_pdir;
357+ }
358+
359+ /* max_send_wr/_recv_wr/_send_sge/_recv_sge/_inline_data */
360+ qp->qp_handle = resp->qpn;
361+ qp->port = init_attr->port_num;
362+ qp->ibqp.qp_num = resp->qpn;
363+ spin_lock_irqsave(&dev->qp_tbl_lock, flags);
364+ dev->qp_tbl[qp->qp_handle % dev->dsr->caps.max_qp] = qp;
365+ spin_unlock_irqrestore(&dev->qp_tbl_lock, flags);
366+
367+ return &qp->ibqp;
368+
369+err_pdir:
370+ pvrdma_page_dir_cleanup(dev, &qp->pdir);
371+err_umem:
372+ if (pd->uobject && udata) {
373+ if (qp->rumem)
374+ ib_umem_release(qp->rumem);
375+ if (qp->sumem)
376+ ib_umem_release(qp->sumem);
377+ }
378+err_qp:
379+ kfree(qp);
380+ atomic_dec(&dev->num_qps);
381+
382+ return ERR_PTR(ret);
383+}
384+
385+static void pvrdma_free_qp(struct pvrdma_qp *qp)
386+{
387+ struct pvrdma_dev *dev = to_vdev(qp->ibqp.device);
388+ struct pvrdma_cq *scq;
389+ struct pvrdma_cq *rcq;
390+ unsigned long flags, scq_flags, rcq_flags;
391+
392+ /* In case cq is polling */
393+ get_cqs(qp, &scq, &rcq);
394+ pvrdma_lock_cqs(scq, rcq, &scq_flags, &rcq_flags);
395+
396+ _pvrdma_flush_cqe(qp, scq);
397+ if (scq != rcq)
398+ _pvrdma_flush_cqe(qp, rcq);
399+
400+ spin_lock_irqsave(&dev->qp_tbl_lock, flags);
401+ dev->qp_tbl[qp->qp_handle] = NULL;
402+ spin_unlock_irqrestore(&dev->qp_tbl_lock, flags);
403+
404+ pvrdma_unlock_cqs(scq, rcq, &scq_flags, &rcq_flags);
405+
406+ atomic_dec(&qp->refcnt);
407+ wait_event(qp->wait, !atomic_read(&qp->refcnt));
408+
409+ pvrdma_page_dir_cleanup(dev, &qp->pdir);
410+
411+ kfree(qp);
412+
413+ atomic_dec(&dev->num_qps);
414+}
415+
416+/**
417+ * pvrdma_destroy_qp - destroy a queue pair
418+ * @qp: the queue pair to destroy
419+ *
420+ * @return: 0 on success.
421+ */
422+int pvrdma_destroy_qp(struct ib_qp *qp)
423+{
424+ struct pvrdma_qp *vqp = to_vqp(qp);
425+ union pvrdma_cmd_req req;
426+ struct pvrdma_cmd_destroy_qp *cmd = &req.destroy_qp;
427+ int ret;
428+
429+ memset(cmd, 0, sizeof(*cmd));
430+ cmd->hdr.cmd = PVRDMA_CMD_DESTROY_QP;
431+ cmd->qp_handle = vqp->qp_handle;
432+
433+ ret = pvrdma_cmd_post(to_vdev(qp->device), &req, NULL, 0);
434+ if (ret < 0)
435+ dev_warn(&to_vdev(qp->device)->pdev->dev,
436+ "destroy queuepair failed, error: %d\n", ret);
437+
438+ pvrdma_free_qp(vqp);
439+
440+ return 0;
441+}
442+
443+/**
444+ * pvrdma_modify_qp - modify queue pair attributes
445+ * @ibqp: the queue pair
446+ * @attr: the new queue pair's attributes
447+ * @attr_mask: attributes mask
448+ * @udata: user data
449+ *
450+ * @returns 0 on success, otherwise returns an errno.
451+ */
452+int pvrdma_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
453+ int attr_mask, struct ib_udata *udata)
454+{
455+ struct pvrdma_dev *dev = to_vdev(ibqp->device);
456+ struct pvrdma_qp *qp = to_vqp(ibqp);
457+ union pvrdma_cmd_req req;
458+ union pvrdma_cmd_resp rsp;
459+ struct pvrdma_cmd_modify_qp *cmd = &req.modify_qp;
460+ int cur_state, next_state;
461+ int ret;
462+
463+ /* Sanity checking. Should need lock here */
464+ mutex_lock(&qp->mutex);
465+ cur_state = (attr_mask & IB_QP_CUR_STATE) ? attr->cur_qp_state :
466+ qp->state;
467+ next_state = (attr_mask & IB_QP_STATE) ? attr->qp_state : cur_state;
468+
469+ if (!ib_modify_qp_is_ok(cur_state, next_state, ibqp->qp_type,
470+ attr_mask, IB_LINK_LAYER_ETHERNET)) {
471+ ret = -EINVAL;
472+ goto out;
473+ }
474+
475+ if (attr_mask & IB_QP_PORT) {
476+ if (attr->port_num == 0 ||
477+ attr->port_num > ibqp->device->phys_port_cnt) {
478+ ret = -EINVAL;
479+ goto out;
480+ }
481+ }
482+
483+ if (attr_mask & IB_QP_MIN_RNR_TIMER) {
484+ if (attr->min_rnr_timer > 31) {
485+ ret = -EINVAL;
486+ goto out;
487+ }
488+ }
489+
490+ if (attr_mask & IB_QP_PKEY_INDEX) {
491+ if (attr->pkey_index >= dev->dsr->caps.max_pkeys) {
492+ ret = -EINVAL;
493+ goto out;
494+ }
495+ }
496+
497+ if (attr_mask & IB_QP_QKEY)
498+ qp->qkey = attr->qkey;
499+
500+ if (cur_state == next_state && cur_state == IB_QPS_RESET) {
501+ ret = 0;
502+ goto out;
503+ }
504+
505+ qp->state = next_state;
506+ memset(cmd, 0, sizeof(*cmd));
507+ cmd->hdr.cmd = PVRDMA_CMD_MODIFY_QP;
508+ cmd->qp_handle = qp->qp_handle;
509+ cmd->attr_mask = ib_qp_attr_mask_to_pvrdma(attr_mask);
510+ cmd->attrs.qp_state = ib_qp_state_to_pvrdma(attr->qp_state);
511+ cmd->attrs.cur_qp_state =
512+ ib_qp_state_to_pvrdma(attr->cur_qp_state);
513+ cmd->attrs.path_mtu = ib_mtu_to_pvrdma(attr->path_mtu);
514+ cmd->attrs.path_mig_state =
515+ ib_mig_state_to_pvrdma(attr->path_mig_state);
516+ cmd->attrs.qkey = attr->qkey;
517+ cmd->attrs.rq_psn = attr->rq_psn;
518+ cmd->attrs.sq_psn = attr->sq_psn;
519+ cmd->attrs.dest_qp_num = attr->dest_qp_num;
520+ cmd->attrs.qp_access_flags =
521+ ib_access_flags_to_pvrdma(attr->qp_access_flags);
522+ cmd->attrs.pkey_index = attr->pkey_index;
523+ cmd->attrs.alt_pkey_index = attr->alt_pkey_index;
524+ cmd->attrs.en_sqd_async_notify = attr->en_sqd_async_notify;
525+ cmd->attrs.sq_draining = attr->sq_draining;
526+ cmd->attrs.max_rd_atomic = attr->max_rd_atomic;
527+ cmd->attrs.max_dest_rd_atomic = attr->max_dest_rd_atomic;
528+ cmd->attrs.min_rnr_timer = attr->min_rnr_timer;
529+ cmd->attrs.port_num = attr->port_num;
530+ cmd->attrs.timeout = attr->timeout;
531+ cmd->attrs.retry_cnt = attr->retry_cnt;
532+ cmd->attrs.rnr_retry = attr->rnr_retry;
533+ cmd->attrs.alt_port_num = attr->alt_port_num;
534+ cmd->attrs.alt_timeout = attr->alt_timeout;
535+ ib_qp_cap_to_pvrdma(&cmd->attrs.cap, &attr->cap);
536+ ib_ah_attr_to_pvrdma(&cmd->attrs.ah_attr, &attr->ah_attr);
537+ ib_ah_attr_to_pvrdma(&cmd->attrs.alt_ah_attr, &attr->alt_ah_attr);
538+
539+ ret = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_MODIFY_QP_RESP);
540+ if (ret < 0) {
541+ dev_warn(&dev->pdev->dev,
542+ "could not modify queuepair, error: %d\n", ret);
543+ } else if (rsp.hdr.err > 0) {
544+ dev_warn(&dev->pdev->dev,
545+ "cannot modify queuepair, error: %d\n", rsp.hdr.err);
546+ ret = -EINVAL;
547+ }
548+
549+ if (ret == 0 && next_state == IB_QPS_RESET)
550+ pvrdma_reset_qp(qp);
551+
552+out:
553+ mutex_unlock(&qp->mutex);
554+
555+ return ret;
556+}
557+
558+static inline void *get_sq_wqe(struct pvrdma_qp *qp, int n)
559+{
560+ return pvrdma_page_dir_get_ptr(&qp->pdir,
561+ qp->sq.offset + n * qp->sq.wqe_size);
562+}
563+
564+static inline void *get_rq_wqe(struct pvrdma_qp *qp, int n)
565+{
566+ return pvrdma_page_dir_get_ptr(&qp->pdir,
567+ qp->rq.offset + n * qp->rq.wqe_size);
568+}
569+
570+static int set_reg_seg(struct pvrdma_sq_wqe_hdr *wqe_hdr, struct ib_reg_wr *wr)
571+{
572+ struct pvrdma_user_mr *mr = to_vmr(wr->mr);
573+
574+ wqe_hdr->wr.fast_reg.iova_start = mr->ibmr.iova;
575+ wqe_hdr->wr.fast_reg.pl_pdir_dma = mr->pdir.dir_dma;
576+ wqe_hdr->wr.fast_reg.page_shift = mr->page_shift;
577+ wqe_hdr->wr.fast_reg.page_list_len = mr->npages;
578+ wqe_hdr->wr.fast_reg.length = mr->ibmr.length;
579+ wqe_hdr->wr.fast_reg.access_flags = wr->access;
580+ wqe_hdr->wr.fast_reg.rkey = wr->key;
581+
582+ return pvrdma_page_dir_insert_page_list(&mr->pdir, mr->pages,
583+ mr->npages);
584+}
585+
586+/**
587+ * pvrdma_post_send - post send work request entries on a QP
588+ * @ibqp: the QP
589+ * @wr: work request list to post
590+ * @bad_wr: the first bad WR returned
591+ *
592+ * @return: 0 on success, otherwise errno returned.
593+ */
594+int pvrdma_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
595+ struct ib_send_wr **bad_wr)
596+{
597+ struct pvrdma_qp *qp = to_vqp(ibqp);
598+ struct pvrdma_dev *dev = to_vdev(ibqp->device);
599+ unsigned long flags;
600+ struct pvrdma_sq_wqe_hdr *wqe_hdr;
601+ struct pvrdma_sge *sge;
602+ int i, index;
603+ int nreq;
604+ int ret;
605+
606+ /*
607+ * In states lower than RTS, we can fail immediately. In other states,
608+ * just post and let the device figure it out.
609+ */
610+ if (qp->state < IB_QPS_RTS) {
611+ *bad_wr = wr;
612+ return -EINVAL;
613+ }
614+
615+ spin_lock_irqsave(&qp->sq.lock, flags);
616+
617+ index = pvrdma_idx(&qp->sq.ring->prod_tail, qp->sq.wqe_cnt);
618+ for (nreq = 0; wr; nreq++, wr = wr->next) {
619+ unsigned int tail;
620+
621+ if (unlikely(!pvrdma_idx_ring_has_space(
622+ qp->sq.ring, qp->sq.wqe_cnt, &tail))) {
623+ dev_warn_ratelimited(&dev->pdev->dev,
624+ "send queue is full\n");
625+ *bad_wr = wr;
626+ ret = -ENOMEM;
627+ goto out;
628+ }
629+
630+ if (unlikely(wr->num_sge > qp->sq.max_sg || wr->num_sge < 0)) {
631+ dev_warn_ratelimited(&dev->pdev->dev,
632+ "send SGE overflow\n");
633+ *bad_wr = wr;
634+ ret = -EINVAL;
635+ goto out;
636+ }
637+
638+ if (unlikely(wr->opcode < 0)) {
639+ dev_warn_ratelimited(&dev->pdev->dev,
640+ "invalid send opcode\n");
641+ *bad_wr = wr;
642+ ret = -EINVAL;
643+ goto out;
644+ }
645+
646+ /*
647+ * Only support UD, RC.
648+ * Need to check opcode table for thorough checking.
649+ * opcode _UD _UC _RC
650+ * _SEND x x x
651+ * _SEND_WITH_IMM x x x
652+ * _RDMA_WRITE x x
653+ * _RDMA_WRITE_WITH_IMM x x
654+ * _LOCAL_INV x x
655+ * _SEND_WITH_INV x x
656+ * _RDMA_READ x
657+ * _ATOMIC_CMP_AND_SWP x
658+ * _ATOMIC_FETCH_AND_ADD x
659+ * _MASK_ATOMIC_CMP_AND_SWP x
660+ * _MASK_ATOMIC_FETCH_AND_ADD x
661+ * _REG_MR x
662+ *
663+ */
664+ if (qp->ibqp.qp_type != IB_QPT_UD &&
665+ qp->ibqp.qp_type != IB_QPT_RC &&
666+ wr->opcode != IB_WR_SEND) {
667+ dev_warn_ratelimited(&dev->pdev->dev,
668+ "unsupported queuepair type\n");
669+ *bad_wr = wr;
670+ ret = -EINVAL;
671+ goto out;
672+ } else if (qp->ibqp.qp_type == IB_QPT_UD ||
673+ qp->ibqp.qp_type == IB_QPT_GSI) {
674+ if (wr->opcode != IB_WR_SEND &&
675+ wr->opcode != IB_WR_SEND_WITH_IMM) {
676+ dev_warn_ratelimited(&dev->pdev->dev,
677+ "invalid send opcode\n");
678+ *bad_wr = wr;
679+ ret = -EINVAL;
680+ goto out;
681+ }
682+ }
683+
684+ wqe_hdr = (struct pvrdma_sq_wqe_hdr *)get_sq_wqe(qp, index);
685+ memset(wqe_hdr, 0, sizeof(*wqe_hdr));
686+ wqe_hdr->wr_id = wr->wr_id;
687+ wqe_hdr->num_sge = wr->num_sge;
688+ wqe_hdr->opcode = ib_wr_opcode_to_pvrdma(wr->opcode);
689+ wqe_hdr->send_flags = ib_send_flags_to_pvrdma(wr->send_flags);
690+ if (wr->opcode == IB_WR_SEND_WITH_IMM ||
691+ wr->opcode == IB_WR_RDMA_WRITE_WITH_IMM)
692+ wqe_hdr->ex.imm_data = wr->ex.imm_data;
693+
694+ switch (qp->ibqp.qp_type) {
695+ case IB_QPT_GSI:
696+ case IB_QPT_UD:
697+ if (unlikely(!ud_wr(wr)->ah)) {
698+ dev_warn_ratelimited(&dev->pdev->dev,
699+ "invalid address handle\n");
700+ *bad_wr = wr;
701+ ret = -EINVAL;
702+ goto out;
703+ }
704+
705+ /*
706+ * Use qkey from qp context if high order bit set,
707+ * otherwise from work request.
708+ */
709+ wqe_hdr->wr.ud.remote_qpn = ud_wr(wr)->remote_qpn;
710+ wqe_hdr->wr.ud.remote_qkey =
711+ ud_wr(wr)->remote_qkey & 0x80000000 ?
712+ qp->qkey : ud_wr(wr)->remote_qkey;
713+ wqe_hdr->wr.ud.av = to_vah(ud_wr(wr)->ah)->av;
714+
715+ break;
716+ case IB_QPT_RC:
717+ switch (wr->opcode) {
718+ case IB_WR_RDMA_READ:
719+ case IB_WR_RDMA_WRITE:
720+ case IB_WR_RDMA_WRITE_WITH_IMM:
721+ wqe_hdr->wr.rdma.remote_addr =
722+ rdma_wr(wr)->remote_addr;
723+ wqe_hdr->wr.rdma.rkey = rdma_wr(wr)->rkey;
724+ break;
725+ case IB_WR_LOCAL_INV:
726+ case IB_WR_SEND_WITH_INV:
727+ wqe_hdr->ex.invalidate_rkey =
728+ wr->ex.invalidate_rkey;
729+ break;
730+ case IB_WR_ATOMIC_CMP_AND_SWP:
731+ case IB_WR_ATOMIC_FETCH_AND_ADD:
732+ wqe_hdr->wr.atomic.remote_addr =
733+ atomic_wr(wr)->remote_addr;
734+ wqe_hdr->wr.atomic.rkey = atomic_wr(wr)->rkey;
735+ wqe_hdr->wr.atomic.compare_add =
736+ atomic_wr(wr)->compare_add;
737+ if (wr->opcode == IB_WR_ATOMIC_CMP_AND_SWP)
738+ wqe_hdr->wr.atomic.swap =
739+ atomic_wr(wr)->swap;
740+ break;
741+ case IB_WR_REG_MR:
742+ ret = set_reg_seg(wqe_hdr, reg_wr(wr));
743+ if (ret < 0) {
744+ dev_warn_ratelimited(&dev->pdev->dev,
745+ "Failed to set fast register work request\n");
746+ *bad_wr = wr;
747+ goto out;
748+ }
749+ break;
750+ default:
751+ break;
752+ }
753+
754+ break;
755+ default:
756+ dev_warn_ratelimited(&dev->pdev->dev,
757+ "invalid queuepair type\n");
758+ ret = -EINVAL;
759+ *bad_wr = wr;
760+ goto out;
761+ }
762+
763+ sge = (struct pvrdma_sge *)(wqe_hdr + 1);
764+ for (i = 0; i < wr->num_sge; i++) {
765+ /* Need to check wqe_size 0 or max size */
766+ sge->addr = wr->sg_list[i].addr;
767+ sge->length = wr->sg_list[i].length;
768+ sge->lkey = wr->sg_list[i].lkey;
769+ sge++;
770+ }
771+
772+ /* Make sure wqe is written before index update */
773+ smp_wmb();
774+
775+ index++;
776+ if (unlikely(index >= qp->sq.wqe_cnt))
777+ index = 0;
778+ /* Update shared sq ring */
779+ pvrdma_idx_ring_inc(&qp->sq.ring->prod_tail,
780+ qp->sq.wqe_cnt);
781+ }
782+
783+ ret = 0;
784+
785+out:
786+ spin_unlock_irqrestore(&qp->sq.lock, flags);
787+
788+ if (!ret)
789+ pvrdma_write_uar_qp(dev, PVRDMA_UAR_QP_SEND | qp->qp_handle);
790+
791+ return ret;
792+}
793+
794+/**
795+ * pvrdma_post_receive - post receive work request entries on a QP
796+ * @ibqp: the QP
797+ * @wr: the work request list to post
798+ * @bad_wr: the first bad WR returned
799+ *
800+ * @return: 0 on success, otherwise errno returned.
801+ */
802+int pvrdma_post_recv(struct ib_qp *ibqp, struct ib_recv_wr *wr,
803+ struct ib_recv_wr **bad_wr)
804+{
805+ struct pvrdma_dev *dev = to_vdev(ibqp->device);
806+ unsigned long flags;
807+ struct pvrdma_qp *qp = to_vqp(ibqp);
808+ struct pvrdma_rq_wqe_hdr *wqe_hdr;
809+ struct pvrdma_sge *sge;
810+ int index, nreq;
811+ int ret = 0;
812+ int i;
813+
814+ /*
815+ * In the RESET state, we can fail immediately. For other states,
816+ * just post and let the device figure it out.
817+ */
818+ if (qp->state == IB_QPS_RESET) {
819+ *bad_wr = wr;
820+ return -EINVAL;
821+ }
822+
823+ spin_lock_irqsave(&qp->rq.lock, flags);
824+
825+ index = pvrdma_idx(&qp->rq.ring->prod_tail, qp->rq.wqe_cnt);
826+ for (nreq = 0; wr; nreq++, wr = wr->next) {
827+ unsigned int tail;
828+
829+ if (unlikely(wr->num_sge > qp->rq.max_sg ||
830+ wr->num_sge < 0)) {
831+ ret = -EINVAL;
832+ *bad_wr = wr;
833+ dev_warn_ratelimited(&dev->pdev->dev,
834+ "recv SGE overflow\n");
835+ goto out;
836+ }
837+
838+ if (unlikely(!pvrdma_idx_ring_has_space(
839+ qp->rq.ring, qp->rq.wqe_cnt, &tail))) {
840+ ret = -ENOMEM;
841+ *bad_wr = wr;
842+ dev_warn_ratelimited(&dev->pdev->dev,
843+ "recv queue full\n");
844+ goto out;
845+ }
846+
847+ wqe_hdr = (struct pvrdma_rq_wqe_hdr *)get_rq_wqe(qp, index);
848+ wqe_hdr->wr_id = wr->wr_id;
849+ wqe_hdr->num_sge = wr->num_sge;
850+ wqe_hdr->total_len = 0;
851+
852+ sge = (struct pvrdma_sge *)(wqe_hdr + 1);
853+ for (i = 0; i < wr->num_sge; i++) {
854+ sge->addr = wr->sg_list[i].addr;
855+ sge->length = wr->sg_list[i].length;
856+ sge->lkey = wr->sg_list[i].lkey;
857+ sge++;
858+ }
859+
860+ /* Make sure wqe is written before index update */
861+ smp_wmb();
862+
863+ index++;
864+ if (unlikely(index >= qp->rq.wqe_cnt))
865+ index = 0;
866+ /* Update shared rq ring */
867+ pvrdma_idx_ring_inc(&qp->rq.ring->prod_tail,
868+ qp->rq.wqe_cnt);
869+ }
870+
871+ spin_unlock_irqrestore(&qp->rq.lock, flags);
872+
873+ pvrdma_write_uar_qp(dev, PVRDMA_UAR_QP_RECV | qp->qp_handle);
874+
875+ return ret;
876+
877+out:
878+ spin_unlock_irqrestore(&qp->rq.lock, flags);
879+
880+ return ret;
881+}
882+
883+/**
884+ * pvrdma_query_qp - query a queue pair's attributes
885+ * @ibqp: the queue pair to query
886+ * @attr: the queue pair's attributes
887+ * @attr_mask: attributes mask
888+ * @init_attr: initial queue pair attributes
889+ *
890+ * @returns 0 on success, otherwise returns an errno.
891+ */
892+int pvrdma_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
893+ int attr_mask, struct ib_qp_init_attr *init_attr)
894+{
895+ struct pvrdma_dev *dev = to_vdev(ibqp->device);
896+ struct pvrdma_qp *qp = to_vqp(ibqp);
897+ union pvrdma_cmd_req req;
898+ union pvrdma_cmd_resp rsp;
899+ struct pvrdma_cmd_query_qp *cmd = &req.query_qp;
900+ struct pvrdma_cmd_query_qp_resp *resp = &rsp.query_qp_resp;
901+ int ret = 0;
902+
903+ mutex_lock(&qp->mutex);
904+
905+ if (qp->state == IB_QPS_RESET) {
906+ attr->qp_state = IB_QPS_RESET;
907+ goto out;
908+ }
909+
910+ memset(cmd, 0, sizeof(*cmd));
911+ cmd->hdr.cmd = PVRDMA_CMD_QUERY_QP;
912+ cmd->qp_handle = qp->qp_handle;
913+ cmd->attr_mask = ib_qp_attr_mask_to_pvrdma(attr_mask);
914+
915+ ret = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_QUERY_QP_RESP);
916+ if (ret < 0) {
917+ dev_warn(&dev->pdev->dev,
918+ "could not query queuepair, error: %d\n", ret);
919+ goto out;
920+ }
921+
922+ attr->qp_state = pvrdma_qp_state_to_ib(resp->attrs.qp_state);
923+ attr->cur_qp_state =
924+ pvrdma_qp_state_to_ib(resp->attrs.cur_qp_state);
925+ attr->path_mtu = pvrdma_mtu_to_ib(resp->attrs.path_mtu);
926+ attr->path_mig_state =
927+ pvrdma_mig_state_to_ib(resp->attrs.path_mig_state);
928+ attr->qkey = resp->attrs.qkey;
929+ attr->rq_psn = resp->attrs.rq_psn;
930+ attr->sq_psn = resp->attrs.sq_psn;
931+ attr->dest_qp_num = resp->attrs.dest_qp_num;
932+ attr->qp_access_flags =
933+ pvrdma_access_flags_to_ib(resp->attrs.qp_access_flags);
934+ attr->pkey_index = resp->attrs.pkey_index;
935+ attr->alt_pkey_index = resp->attrs.alt_pkey_index;
936+ attr->en_sqd_async_notify = resp->attrs.en_sqd_async_notify;
937+ attr->sq_draining = resp->attrs.sq_draining;
938+ attr->max_rd_atomic = resp->attrs.max_rd_atomic;
939+ attr->max_dest_rd_atomic = resp->attrs.max_dest_rd_atomic;
940+ attr->min_rnr_timer = resp->attrs.min_rnr_timer;
941+ attr->port_num = resp->attrs.port_num;
942+ attr->timeout = resp->attrs.timeout;
943+ attr->retry_cnt = resp->attrs.retry_cnt;
944+ attr->rnr_retry = resp->attrs.rnr_retry;
945+ attr->alt_port_num = resp->attrs.alt_port_num;
946+ attr->alt_timeout = resp->attrs.alt_timeout;
947+ pvrdma_qp_cap_to_ib(&attr->cap, &resp->attrs.cap);
948+ pvrdma_ah_attr_to_ib(&attr->ah_attr, &resp->attrs.ah_attr);
949+ pvrdma_ah_attr_to_ib(&attr->alt_ah_attr, &resp->attrs.alt_ah_attr);
950+
951+ qp->state = attr->qp_state;
952+
953+ ret = 0;
954+
955+out:
956+ attr->cur_qp_state = attr->qp_state;
957+
958+ init_attr->event_handler = qp->ibqp.event_handler;
959+ init_attr->qp_context = qp->ibqp.qp_context;
960+ init_attr->send_cq = qp->ibqp.send_cq;
961+ init_attr->recv_cq = qp->ibqp.recv_cq;
962+ init_attr->srq = qp->ibqp.srq;
963+ init_attr->xrcd = NULL;
964+ init_attr->cap = attr->cap;
965+ init_attr->sq_sig_type = 0;
966+ init_attr->qp_type = qp->ibqp.qp_type;
967+ init_attr->create_flags = 0;
968+ init_attr->port_num = qp->port;
969+
970+ mutex_unlock(&qp->mutex);
971+ return ret;
972+}
drivers/infiniband/hw/vmw_pvrdma/pvrdma_ring.h+131−0
@@ -0,0 +1,131 @@
1+/*
2+ * Copyright (c) 2012-2016 VMware, Inc. All rights reserved.
3+ *
4+ * This program is free software; you can redistribute it and/or
5+ * modify it under the terms of EITHER the GNU General Public License
6+ * version 2 as published by the Free Software Foundation or the BSD
7+ * 2-Clause License. This program is distributed in the hope that it
8+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
9+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
10+ * See the GNU General Public License version 2 for more details at
11+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
12+ *
13+ * You should have received a copy of the GNU General Public License
14+ * along with this program available in the file COPYING in the main
15+ * directory of this source tree.
16+ *
17+ * The BSD 2-Clause License
18+ *
19+ * Redistribution and use in source and binary forms, with or
20+ * without modification, are permitted provided that the following
21+ * conditions are met:
22+ *
23+ * - Redistributions of source code must retain the above
24+ * copyright notice, this list of conditions and the following
25+ * disclaimer.
26+ *
27+ * - Redistributions in binary form must reproduce the above
28+ * copyright notice, this list of conditions and the following
29+ * disclaimer in the documentation and/or other materials
30+ * provided with the distribution.
31+ *
32+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
33+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
34+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
35+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
36+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
37+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
38+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
39+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
40+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
41+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
42+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
43+ * OF THE POSSIBILITY OF SUCH DAMAGE.
44+ */
45+
46+#ifndef __PVRDMA_RING_H__
47+#define __PVRDMA_RING_H__
48+
49+#include <linux/types.h>
50+
51+#define PVRDMA_INVALID_IDX -1 /* Invalid index. */
52+
53+struct pvrdma_ring {
54+ atomic_t prod_tail; /* Producer tail. */
55+ atomic_t cons_head; /* Consumer head. */
56+};
57+
58+struct pvrdma_ring_state {
59+ struct pvrdma_ring tx; /* Tx ring. */
60+ struct pvrdma_ring rx; /* Rx ring. */
61+};
62+
63+static inline int pvrdma_idx_valid(__u32 idx, __u32 max_elems)
64+{
65+ /* Generates fewer instructions than a less-than. */
66+ return (idx & ~((max_elems << 1) - 1)) == 0;
67+}
68+
69+static inline __s32 pvrdma_idx(atomic_t *var, __u32 max_elems)
70+{
71+ const unsigned int idx = atomic_read(var);
72+
73+ if (pvrdma_idx_valid(idx, max_elems))
74+ return idx & (max_elems - 1);
75+ return PVRDMA_INVALID_IDX;
76+}
77+
78+static inline void pvrdma_idx_ring_inc(atomic_t *var, __u32 max_elems)
79+{
80+ __u32 idx = atomic_read(var) + 1; /* Increment. */
81+
82+ idx &= (max_elems << 1) - 1; /* Modulo size, flip gen. */
83+ atomic_set(var, idx);
84+}
85+
86+static inline __s32 pvrdma_idx_ring_has_space(const struct pvrdma_ring *r,
87+ __u32 max_elems, __u32 *out_tail)
88+{
89+ const __u32 tail = atomic_read(&r->prod_tail);
90+ const __u32 head = atomic_read(&r->cons_head);
91+
92+ if (pvrdma_idx_valid(tail, max_elems) &&
93+ pvrdma_idx_valid(head, max_elems)) {
94+ *out_tail = tail & (max_elems - 1);
95+ return tail != (head ^ max_elems);
96+ }
97+ return PVRDMA_INVALID_IDX;
98+}
99+
100+static inline __s32 pvrdma_idx_ring_has_data(const struct pvrdma_ring *r,
101+ __u32 max_elems, __u32 *out_head)
102+{
103+ const __u32 tail = atomic_read(&r->prod_tail);
104+ const __u32 head = atomic_read(&r->cons_head);
105+
106+ if (pvrdma_idx_valid(tail, max_elems) &&
107+ pvrdma_idx_valid(head, max_elems)) {
108+ *out_head = head & (max_elems - 1);
109+ return tail != head;
110+ }
111+ return PVRDMA_INVALID_IDX;
112+}
113+
114+static inline bool pvrdma_idx_ring_is_valid_idx(const struct pvrdma_ring *r,
115+ __u32 max_elems, __u32 *idx)
116+{
117+ const __u32 tail = atomic_read(&r->prod_tail);
118+ const __u32 head = atomic_read(&r->cons_head);
119+
120+ if (pvrdma_idx_valid(tail, max_elems) &&
121+ pvrdma_idx_valid(head, max_elems) &&
122+ pvrdma_idx_valid(*idx, max_elems)) {
123+ if (tail > head && (*idx < tail && *idx >= head))
124+ return true;
125+ else if (head > tail && (*idx >= head || *idx < tail))
126+ return true;
127+ }
128+ return false;
129+}
130+
131+#endif /* __PVRDMA_RING_H__ */
drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c+579−0
@@ -0,0 +1,579 @@
1+/*
2+ * Copyright (c) 2012-2016 VMware, Inc. All rights reserved.
3+ *
4+ * This program is free software; you can redistribute it and/or
5+ * modify it under the terms of EITHER the GNU General Public License
6+ * version 2 as published by the Free Software Foundation or the BSD
7+ * 2-Clause License. This program is distributed in the hope that it
8+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
9+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
10+ * See the GNU General Public License version 2 for more details at
11+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
12+ *
13+ * You should have received a copy of the GNU General Public License
14+ * along with this program available in the file COPYING in the main
15+ * directory of this source tree.
16+ *
17+ * The BSD 2-Clause License
18+ *
19+ * Redistribution and use in source and binary forms, with or
20+ * without modification, are permitted provided that the following
21+ * conditions are met:
22+ *
23+ * - Redistributions of source code must retain the above
24+ * copyright notice, this list of conditions and the following
25+ * disclaimer.
26+ *
27+ * - Redistributions in binary form must reproduce the above
28+ * copyright notice, this list of conditions and the following
29+ * disclaimer in the documentation and/or other materials
30+ * provided with the distribution.
31+ *
32+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
33+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
34+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
35+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
36+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
37+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
38+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
39+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
40+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
41+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
42+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
43+ * OF THE POSSIBILITY OF SUCH DAMAGE.
44+ */
45+
46+#include <asm/page.h>
47+#include <linux/inet.h>
48+#include <linux/io.h>
49+#include <rdma/ib_addr.h>
50+#include <rdma/ib_smi.h>
51+#include <rdma/ib_user_verbs.h>
52+#include <rdma/vmw_pvrdma-abi.h>
53+
54+#include "pvrdma.h"
55+
56+/**
57+ * pvrdma_query_device - query device
58+ * @ibdev: the device to query
59+ * @props: the device properties
60+ * @uhw: user data
61+ *
62+ * @return: 0 on success, otherwise negative errno
63+ */
64+int pvrdma_query_device(struct ib_device *ibdev,
65+ struct ib_device_attr *props,
66+ struct ib_udata *uhw)
67+{
68+ struct pvrdma_dev *dev = to_vdev(ibdev);
69+
70+ if (uhw->inlen || uhw->outlen)
71+ return -EINVAL;
72+
73+ memset(props, 0, sizeof(*props));
74+
75+ props->fw_ver = dev->dsr->caps.fw_ver;
76+ props->sys_image_guid = dev->dsr->caps.sys_image_guid;
77+ props->max_mr_size = dev->dsr->caps.max_mr_size;
78+ props->page_size_cap = dev->dsr->caps.page_size_cap;
79+ props->vendor_id = dev->dsr->caps.vendor_id;
80+ props->vendor_part_id = dev->pdev->device;
81+ props->hw_ver = dev->dsr->caps.hw_ver;
82+ props->max_qp = dev->dsr->caps.max_qp;
83+ props->max_qp_wr = dev->dsr->caps.max_qp_wr;
84+ props->device_cap_flags = dev->dsr->caps.device_cap_flags;
85+ props->max_sge = dev->dsr->caps.max_sge;
86+ props->max_cq = dev->dsr->caps.max_cq;
87+ props->max_cqe = dev->dsr->caps.max_cqe;
88+ props->max_mr = dev->dsr->caps.max_mr;
89+ props->max_pd = dev->dsr->caps.max_pd;
90+ props->max_qp_rd_atom = dev->dsr->caps.max_qp_rd_atom;
91+ props->max_qp_init_rd_atom = dev->dsr->caps.max_qp_init_rd_atom;
92+ props->atomic_cap =
93+ dev->dsr->caps.atomic_ops &
94+ (PVRDMA_ATOMIC_OP_COMP_SWAP | PVRDMA_ATOMIC_OP_FETCH_ADD) ?
95+ IB_ATOMIC_HCA : IB_ATOMIC_NONE;
96+ props->masked_atomic_cap = props->atomic_cap;
97+ props->max_ah = dev->dsr->caps.max_ah;
98+ props->max_pkeys = dev->dsr->caps.max_pkeys;
99+ props->local_ca_ack_delay = dev->dsr->caps.local_ca_ack_delay;
100+ if ((dev->dsr->caps.bmme_flags & PVRDMA_BMME_FLAG_LOCAL_INV) &&
101+ (dev->dsr->caps.bmme_flags & PVRDMA_BMME_FLAG_REMOTE_INV) &&
102+ (dev->dsr->caps.bmme_flags & PVRDMA_BMME_FLAG_FAST_REG_WR)) {
103+ props->device_cap_flags |= IB_DEVICE_MEM_MGT_EXTENSIONS;
104+ }
105+
106+ return 0;
107+}
108+
109+/**
110+ * pvrdma_query_port - query device port attributes
111+ * @ibdev: the device to query
112+ * @port: the port number
113+ * @props: the device properties
114+ *
115+ * @return: 0 on success, otherwise negative errno
116+ */
117+int pvrdma_query_port(struct ib_device *ibdev, u8 port,
118+ struct ib_port_attr *props)
119+{
120+ struct pvrdma_dev *dev = to_vdev(ibdev);
121+ union pvrdma_cmd_req req;
122+ union pvrdma_cmd_resp rsp;
123+ struct pvrdma_cmd_query_port *cmd = &req.query_port;
124+ struct pvrdma_cmd_query_port_resp *resp = &rsp.query_port_resp;
125+ int err;
126+
127+ memset(cmd, 0, sizeof(*cmd));
128+ cmd->hdr.cmd = PVRDMA_CMD_QUERY_PORT;
129+ cmd->port_num = port;
130+
131+ err = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_QUERY_PORT_RESP);
132+ if (err < 0) {
133+ dev_warn(&dev->pdev->dev,
134+ "could not query port, error: %d\n", err);
135+ return err;
136+ }
137+
138+ memset(props, 0, sizeof(*props));
139+
140+ props->state = pvrdma_port_state_to_ib(resp->attrs.state);
141+ props->max_mtu = pvrdma_mtu_to_ib(resp->attrs.max_mtu);
142+ props->active_mtu = pvrdma_mtu_to_ib(resp->attrs.active_mtu);
143+ props->gid_tbl_len = resp->attrs.gid_tbl_len;
144+ props->port_cap_flags =
145+ pvrdma_port_cap_flags_to_ib(resp->attrs.port_cap_flags);
146+ props->max_msg_sz = resp->attrs.max_msg_sz;
147+ props->bad_pkey_cntr = resp->attrs.bad_pkey_cntr;
148+ props->qkey_viol_cntr = resp->attrs.qkey_viol_cntr;
149+ props->pkey_tbl_len = resp->attrs.pkey_tbl_len;
150+ props->lid = resp->attrs.lid;
151+ props->sm_lid = resp->attrs.sm_lid;
152+ props->lmc = resp->attrs.lmc;
153+ props->max_vl_num = resp->attrs.max_vl_num;
154+ props->sm_sl = resp->attrs.sm_sl;
155+ props->subnet_timeout = resp->attrs.subnet_timeout;
156+ props->init_type_reply = resp->attrs.init_type_reply;
157+ props->active_width = pvrdma_port_width_to_ib(resp->attrs.active_width);
158+ props->active_speed = pvrdma_port_speed_to_ib(resp->attrs.active_speed);
159+ props->phys_state = resp->attrs.phys_state;
160+
161+ return 0;
162+}
163+
164+/**
165+ * pvrdma_query_gid - query device gid
166+ * @ibdev: the device to query
167+ * @port: the port number
168+ * @index: the index
169+ * @gid: the device gid value
170+ *
171+ * @return: 0 on success, otherwise negative errno
172+ */
173+int pvrdma_query_gid(struct ib_device *ibdev, u8 port, int index,
174+ union ib_gid *gid)
175+{
176+ struct pvrdma_dev *dev = to_vdev(ibdev);
177+
178+ if (index >= dev->dsr->caps.gid_tbl_len)
179+ return -EINVAL;
180+
181+ memcpy(gid, &dev->sgid_tbl[index], sizeof(union ib_gid));
182+
183+ return 0;
184+}
185+
186+/**
187+ * pvrdma_query_pkey - query device port's P_Key table
188+ * @ibdev: the device to query
189+ * @port: the port number
190+ * @index: the index
191+ * @pkey: the device P_Key value
192+ *
193+ * @return: 0 on success, otherwise negative errno
194+ */
195+int pvrdma_query_pkey(struct ib_device *ibdev, u8 port, u16 index,
196+ u16 *pkey)
197+{
198+ int err = 0;
199+ union pvrdma_cmd_req req;
200+ union pvrdma_cmd_resp rsp;
201+ struct pvrdma_cmd_query_pkey *cmd = &req.query_pkey;
202+
203+ memset(cmd, 0, sizeof(*cmd));
204+ cmd->hdr.cmd = PVRDMA_CMD_QUERY_PKEY;
205+ cmd->port_num = port;
206+ cmd->index = index;
207+
208+ err = pvrdma_cmd_post(to_vdev(ibdev), &req, &rsp,
209+ PVRDMA_CMD_QUERY_PKEY_RESP);
210+ if (err < 0) {
211+ dev_warn(&to_vdev(ibdev)->pdev->dev,
212+ "could not query pkey, error: %d\n", err);
213+ return err;
214+ }
215+
216+ *pkey = rsp.query_pkey_resp.pkey;
217+
218+ return 0;
219+}
220+
221+enum rdma_link_layer pvrdma_port_link_layer(struct ib_device *ibdev,
222+ u8 port)
223+{
224+ return IB_LINK_LAYER_ETHERNET;
225+}
226+
227+int pvrdma_modify_device(struct ib_device *ibdev, int mask,
228+ struct ib_device_modify *props)
229+{
230+ unsigned long flags;
231+
232+ if (mask & ~(IB_DEVICE_MODIFY_SYS_IMAGE_GUID |
233+ IB_DEVICE_MODIFY_NODE_DESC)) {
234+ dev_warn(&to_vdev(ibdev)->pdev->dev,
235+ "unsupported device modify mask %#x\n", mask);
236+ return -EOPNOTSUPP;
237+ }
238+
239+ if (mask & IB_DEVICE_MODIFY_NODE_DESC) {
240+ spin_lock_irqsave(&to_vdev(ibdev)->desc_lock, flags);
241+ memcpy(ibdev->node_desc, props->node_desc, 64);
242+ spin_unlock_irqrestore(&to_vdev(ibdev)->desc_lock, flags);
243+ }
244+
245+ if (mask & IB_DEVICE_MODIFY_SYS_IMAGE_GUID) {
246+ mutex_lock(&to_vdev(ibdev)->port_mutex);
247+ to_vdev(ibdev)->sys_image_guid =
248+ cpu_to_be64(props->sys_image_guid);
249+ mutex_unlock(&to_vdev(ibdev)->port_mutex);
250+ }
251+
252+ return 0;
253+}
254+
255+/**
256+ * pvrdma_modify_port - modify device port attributes
257+ * @ibdev: the device to modify
258+ * @port: the port number
259+ * @mask: attributes to modify
260+ * @props: the device properties
261+ *
262+ * @return: 0 on success, otherwise negative errno
263+ */
264+int pvrdma_modify_port(struct ib_device *ibdev, u8 port, int mask,
265+ struct ib_port_modify *props)
266+{
267+ struct ib_port_attr attr;
268+ struct pvrdma_dev *vdev = to_vdev(ibdev);
269+ int ret;
270+
271+ if (mask & ~IB_PORT_SHUTDOWN) {
272+ dev_warn(&vdev->pdev->dev,
273+ "unsupported port modify mask %#x\n", mask);
274+ return -EOPNOTSUPP;
275+ }
276+
277+ mutex_lock(&vdev->port_mutex);
278+ ret = pvrdma_query_port(ibdev, port, &attr);
279+ if (ret)
280+ goto out;
281+
282+ vdev->port_cap_mask |= props->set_port_cap_mask;
283+ vdev->port_cap_mask &= ~props->clr_port_cap_mask;
284+
285+ if (mask & IB_PORT_SHUTDOWN)
286+ vdev->ib_active = false;
287+
288+out:
289+ mutex_unlock(&vdev->port_mutex);
290+ return ret;
291+}
292+
293+/**
294+ * pvrdma_alloc_ucontext - allocate ucontext
295+ * @ibdev: the IB device
296+ * @udata: user data
297+ *
298+ * @return: the ib_ucontext pointer on success, otherwise errno.
299+ */
300+struct ib_ucontext *pvrdma_alloc_ucontext(struct ib_device *ibdev,
301+ struct ib_udata *udata)
302+{
303+ struct pvrdma_dev *vdev = to_vdev(ibdev);
304+ struct pvrdma_ucontext *context;
305+ union pvrdma_cmd_req req;
306+ union pvrdma_cmd_resp rsp;
307+ struct pvrdma_cmd_create_uc *cmd = &req.create_uc;
308+ struct pvrdma_cmd_create_uc_resp *resp = &rsp.create_uc_resp;
309+ struct pvrdma_alloc_ucontext_resp uresp;
310+ int ret;
311+ void *ptr;
312+
313+ if (!vdev->ib_active)
314+ return ERR_PTR(-EAGAIN);
315+
316+ context = kmalloc(sizeof(*context), GFP_KERNEL);
317+ if (!context)
318+ return ERR_PTR(-ENOMEM);
319+
320+ context->dev = vdev;
321+ ret = pvrdma_uar_alloc(vdev, &context->uar);
322+ if (ret) {
323+ kfree(context);
324+ return ERR_PTR(-ENOMEM);
325+ }
326+
327+ /* get ctx_handle from host */
328+ memset(cmd, 0, sizeof(*cmd));
329+ cmd->pfn = context->uar.pfn;
330+ cmd->hdr.cmd = PVRDMA_CMD_CREATE_UC;
331+ ret = pvrdma_cmd_post(vdev, &req, &rsp, PVRDMA_CMD_CREATE_UC_RESP);
332+ if (ret < 0) {
333+ dev_warn(&vdev->pdev->dev,
334+ "could not create ucontext, error: %d\n", ret);
335+ ptr = ERR_PTR(ret);
336+ goto err;
337+ }
338+
339+ context->ctx_handle = resp->ctx_handle;
340+
341+ /* copy back to user */
342+ uresp.qp_tab_size = vdev->dsr->caps.max_qp;
343+ ret = ib_copy_to_udata(udata, &uresp, sizeof(uresp));
344+ if (ret) {
345+ pvrdma_uar_free(vdev, &context->uar);
346+ context->ibucontext.device = ibdev;
347+ pvrdma_dealloc_ucontext(&context->ibucontext);
348+ return ERR_PTR(-EFAULT);
349+ }
350+
351+ return &context->ibucontext;
352+
353+err:
354+ pvrdma_uar_free(vdev, &context->uar);
355+ kfree(context);
356+ return ptr;
357+}
358+
359+/**
360+ * pvrdma_dealloc_ucontext - deallocate ucontext
361+ * @ibcontext: the ucontext
362+ *
363+ * @return: 0 on success, otherwise errno.
364+ */
365+int pvrdma_dealloc_ucontext(struct ib_ucontext *ibcontext)
366+{
367+ struct pvrdma_ucontext *context = to_vucontext(ibcontext);
368+ union pvrdma_cmd_req req;
369+ struct pvrdma_cmd_destroy_uc *cmd = &req.destroy_uc;
370+ int ret;
371+
372+ memset(cmd, 0, sizeof(*cmd));
373+ cmd->hdr.cmd = PVRDMA_CMD_DESTROY_UC;
374+ cmd->ctx_handle = context->ctx_handle;
375+
376+ ret = pvrdma_cmd_post(context->dev, &req, NULL, 0);
377+ if (ret < 0)
378+ dev_warn(&context->dev->pdev->dev,
379+ "destroy ucontext failed, error: %d\n", ret);
380+
381+ /* Free the UAR even if the device command failed */
382+ pvrdma_uar_free(to_vdev(ibcontext->device), &context->uar);
383+ kfree(context);
384+
385+ return ret;
386+}
387+
388+/**
389+ * pvrdma_mmap - create mmap region
390+ * @ibcontext: the user context
391+ * @vma: the VMA
392+ *
393+ * @return: 0 on success, otherwise errno.
394+ */
395+int pvrdma_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vma)
396+{
397+ struct pvrdma_ucontext *context = to_vucontext(ibcontext);
398+ unsigned long start = vma->vm_start;
399+ unsigned long size = vma->vm_end - vma->vm_start;
400+ unsigned long offset = vma->vm_pgoff << PAGE_SHIFT;
401+
402+ dev_dbg(&context->dev->pdev->dev, "create mmap region\n");
403+
404+ if ((size != PAGE_SIZE) || (offset & ~PAGE_MASK)) {
405+ dev_warn(&context->dev->pdev->dev,
406+ "invalid params for mmap region\n");
407+ return -EINVAL;
408+ }
409+
410+ /* Map UAR to kernel space, VM_LOCKED? */
411+ vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND;
412+ vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
413+ if (io_remap_pfn_range(vma, start, context->uar.pfn, size,
414+ vma->vm_page_prot))
415+ return -EAGAIN;
416+
417+ return 0;
418+}
419+
420+/**
421+ * pvrdma_alloc_pd - allocate protection domain
422+ * @ibdev: the IB device
423+ * @context: user context
424+ * @udata: user data
425+ *
426+ * @return: the ib_pd protection domain pointer on success, otherwise errno.
427+ */
428+struct ib_pd *pvrdma_alloc_pd(struct ib_device *ibdev,
429+ struct ib_ucontext *context,
430+ struct ib_udata *udata)
431+{
432+ struct pvrdma_pd *pd;
433+ struct pvrdma_dev *dev = to_vdev(ibdev);
434+ union pvrdma_cmd_req req;
435+ union pvrdma_cmd_resp rsp;
436+ struct pvrdma_cmd_create_pd *cmd = &req.create_pd;
437+ struct pvrdma_cmd_create_pd_resp *resp = &rsp.create_pd_resp;
438+ int ret;
439+ void *ptr;
440+
441+ /* Check allowed max pds */
442+ if (!atomic_add_unless(&dev->num_pds, 1, dev->dsr->caps.max_pd))
443+ return ERR_PTR(-ENOMEM);
444+
445+ pd = kmalloc(sizeof(*pd), GFP_KERNEL);
446+ if (!pd) {
447+ ptr = ERR_PTR(-ENOMEM);
448+ goto err;
449+ }
450+
451+ memset(cmd, 0, sizeof(*cmd));
452+ cmd->hdr.cmd = PVRDMA_CMD_CREATE_PD;
453+ cmd->ctx_handle = (context) ? to_vucontext(context)->ctx_handle : 0;
454+ ret = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_CREATE_PD_RESP);
455+ if (ret < 0) {
456+ dev_warn(&dev->pdev->dev,
457+ "failed to allocate protection domain, error: %d\n",
458+ ret);
459+ ptr = ERR_PTR(ret);
460+ goto freepd;
461+ }
462+
463+ pd->privileged = !context;
464+ pd->pd_handle = resp->pd_handle;
465+ pd->pdn = resp->pd_handle;
466+
467+ if (context) {
468+ if (ib_copy_to_udata(udata, &pd->pdn, sizeof(__u32))) {
469+ dev_warn(&dev->pdev->dev,
470+ "failed to copy back protection domain\n");
471+ pvrdma_dealloc_pd(&pd->ibpd);
472+ return ERR_PTR(-EFAULT);
473+ }
474+ }
475+
476+ /* u32 pd handle */
477+ return &pd->ibpd;
478+
479+freepd:
480+ kfree(pd);
481+err:
482+ atomic_dec(&dev->num_pds);
483+ return ptr;
484+}
485+
486+/**
487+ * pvrdma_dealloc_pd - deallocate protection domain
488+ * @pd: the protection domain to be released
489+ *
490+ * @return: 0 on success, otherwise errno.
491+ */
492+int pvrdma_dealloc_pd(struct ib_pd *pd)
493+{
494+ struct pvrdma_dev *dev = to_vdev(pd->device);
495+ union pvrdma_cmd_req req;
496+ struct pvrdma_cmd_destroy_pd *cmd = &req.destroy_pd;
497+ int ret;
498+
499+ memset(cmd, 0, sizeof(*cmd));
500+ cmd->hdr.cmd = PVRDMA_CMD_DESTROY_PD;
501+ cmd->pd_handle = to_vpd(pd)->pd_handle;
502+
503+ ret = pvrdma_cmd_post(dev, &req, NULL, 0);
504+ if (ret)
505+ dev_warn(&dev->pdev->dev,
506+ "could not dealloc protection domain, error: %d\n",
507+ ret);
508+
509+ kfree(to_vpd(pd));
510+ atomic_dec(&dev->num_pds);
511+
512+ return 0;
513+}
514+
515+/**
516+ * pvrdma_create_ah - create an address handle
517+ * @pd: the protection domain
518+ * @ah_attr: the attributes of the AH
519+ * @udata: user data blob
520+ *
521+ * @return: the ib_ah pointer on success, otherwise errno.
522+ */
523+struct ib_ah *pvrdma_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr,
524+ struct ib_udata *udata)
525+{
526+ struct pvrdma_dev *dev = to_vdev(pd->device);
527+ struct pvrdma_ah *ah;
528+ enum rdma_link_layer ll;
529+
530+ if (!(ah_attr->ah_flags & IB_AH_GRH))
531+ return ERR_PTR(-EINVAL);
532+
533+ ll = rdma_port_get_link_layer(pd->device, ah_attr->port_num);
534+
535+ if (ll != IB_LINK_LAYER_ETHERNET ||
536+ rdma_is_multicast_addr((struct in6_addr *)ah_attr->grh.dgid.raw))
537+ return ERR_PTR(-EINVAL);
538+
539+ if (!atomic_add_unless(&dev->num_ahs, 1, dev->dsr->caps.max_ah))
540+ return ERR_PTR(-ENOMEM);
541+
542+ ah = kzalloc(sizeof(*ah), GFP_KERNEL);
543+ if (!ah) {
544+ atomic_dec(&dev->num_ahs);
545+ return ERR_PTR(-ENOMEM);
546+ }
547+
548+ ah->av.port_pd = to_vpd(pd)->pd_handle | (ah_attr->port_num << 24);
549+ ah->av.src_path_bits = ah_attr->src_path_bits;
550+ ah->av.src_path_bits |= 0x80;
551+ ah->av.gid_index = ah_attr->grh.sgid_index;
552+ ah->av.hop_limit = ah_attr->grh.hop_limit;
553+ ah->av.sl_tclass_flowlabel = (ah_attr->grh.traffic_class << 20) |
554+ ah_attr->grh.flow_label;
555+ memcpy(ah->av.dgid, ah_attr->grh.dgid.raw, 16);
556+ memcpy(ah->av.dmac, ah_attr->dmac, 6);
557+
558+ ah->ibah.device = pd->device;
559+ ah->ibah.pd = pd;
560+ ah->ibah.uobject = NULL;
561+
562+ return &ah->ibah;
563+}
564+
565+/**
566+ * pvrdma_destroy_ah - destroy an address handle
567+ * @ah: the address handle to destroyed
568+ *
569+ * @return: 0 on success.
570+ */
571+int pvrdma_destroy_ah(struct ib_ah *ah)
572+{
573+ struct pvrdma_dev *dev = to_vdev(ah->device);
574+
575+ kfree(to_vah(ah));
576+ atomic_dec(&dev->num_ahs);
577+
578+ return 0;
579+}
drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h+436−0
@@ -0,0 +1,436 @@
1+/*
2+ * Copyright (c) 2012-2016 VMware, Inc. All rights reserved.
3+ *
4+ * This program is free software; you can redistribute it and/or
5+ * modify it under the terms of EITHER the GNU General Public License
6+ * version 2 as published by the Free Software Foundation or the BSD
7+ * 2-Clause License. This program is distributed in the hope that it
8+ * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED
9+ * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
10+ * See the GNU General Public License version 2 for more details at
11+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html.
12+ *
13+ * You should have received a copy of the GNU General Public License
14+ * along with this program available in the file COPYING in the main
15+ * directory of this source tree.
16+ *
17+ * The BSD 2-Clause License
18+ *
19+ * Redistribution and use in source and binary forms, with or
20+ * without modification, are permitted provided that the following
21+ * conditions are met:
22+ *
23+ * - Redistributions of source code must retain the above
24+ * copyright notice, this list of conditions and the following
25+ * disclaimer.
26+ *
27+ * - Redistributions in binary form must reproduce the above
28+ * copyright notice, this list of conditions and the following
29+ * disclaimer in the documentation and/or other materials
30+ * provided with the distribution.
31+ *
32+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
33+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
34+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
35+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
36+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
37+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
38+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
39+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
40+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
41+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
42+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
43+ * OF THE POSSIBILITY OF SUCH DAMAGE.
44+ */
45+
46+#ifndef __PVRDMA_VERBS_H__
47+#define __PVRDMA_VERBS_H__
48+
49+#include <linux/types.h>
50+
51+union pvrdma_gid {
52+ u8 raw[16];
53+ struct {
54+ __be64 subnet_prefix;
55+ __be64 interface_id;
56+ } global;
57+};
58+
59+enum pvrdma_link_layer {
60+ PVRDMA_LINK_LAYER_UNSPECIFIED,
61+ PVRDMA_LINK_LAYER_INFINIBAND,
62+ PVRDMA_LINK_LAYER_ETHERNET,
63+};
64+
65+enum pvrdma_mtu {
66+ PVRDMA_MTU_256 = 1,
67+ PVRDMA_MTU_512 = 2,
68+ PVRDMA_MTU_1024 = 3,
69+ PVRDMA_MTU_2048 = 4,
70+ PVRDMA_MTU_4096 = 5,
71+};
72+
73+static inline int pvrdma_mtu_enum_to_int(enum pvrdma_mtu mtu)
74+{
75+ switch (mtu) {
76+ case PVRDMA_MTU_256: return 256;
77+ case PVRDMA_MTU_512: return 512;
78+ case PVRDMA_MTU_1024: return 1024;
79+ case PVRDMA_MTU_2048: return 2048;
80+ case PVRDMA_MTU_4096: return 4096;
81+ default: return -1;
82+ }
83+}
84+
85+static inline enum pvrdma_mtu pvrdma_mtu_int_to_enum(int mtu)
86+{
87+ switch (mtu) {
88+ case 256: return PVRDMA_MTU_256;
89+ case 512: return PVRDMA_MTU_512;
90+ case 1024: return PVRDMA_MTU_1024;
91+ case 2048: return PVRDMA_MTU_2048;
92+ case 4096:
93+ default: return PVRDMA_MTU_4096;
94+ }
95+}
96+
97+enum pvrdma_port_state {
98+ PVRDMA_PORT_NOP = 0,
99+ PVRDMA_PORT_DOWN = 1,
100+ PVRDMA_PORT_INIT = 2,
101+ PVRDMA_PORT_ARMED = 3,
102+ PVRDMA_PORT_ACTIVE = 4,
103+ PVRDMA_PORT_ACTIVE_DEFER = 5,
104+};
105+
106+enum pvrdma_port_cap_flags {
107+ PVRDMA_PORT_SM = 1 << 1,
108+ PVRDMA_PORT_NOTICE_SUP = 1 << 2,
109+ PVRDMA_PORT_TRAP_SUP = 1 << 3,
110+ PVRDMA_PORT_OPT_IPD_SUP = 1 << 4,
111+ PVRDMA_PORT_AUTO_MIGR_SUP = 1 << 5,
112+ PVRDMA_PORT_SL_MAP_SUP = 1 << 6,
113+ PVRDMA_PORT_MKEY_NVRAM = 1 << 7,
114+ PVRDMA_PORT_PKEY_NVRAM = 1 << 8,
115+ PVRDMA_PORT_LED_INFO_SUP = 1 << 9,
116+ PVRDMA_PORT_SM_DISABLED = 1 << 10,
117+ PVRDMA_PORT_SYS_IMAGE_GUID_SUP = 1 << 11,
118+ PVRDMA_PORT_PKEY_SW_EXT_PORT_TRAP_SUP = 1 << 12,
119+ PVRDMA_PORT_EXTENDED_SPEEDS_SUP = 1 << 14,
120+ PVRDMA_PORT_CM_SUP = 1 << 16,
121+ PVRDMA_PORT_SNMP_TUNNEL_SUP = 1 << 17,
122+ PVRDMA_PORT_REINIT_SUP = 1 << 18,
123+ PVRDMA_PORT_DEVICE_MGMT_SUP = 1 << 19,
124+ PVRDMA_PORT_VENDOR_CLASS_SUP = 1 << 20,
125+ PVRDMA_PORT_DR_NOTICE_SUP = 1 << 21,
126+ PVRDMA_PORT_CAP_MASK_NOTICE_SUP = 1 << 22,
127+ PVRDMA_PORT_BOOT_MGMT_SUP = 1 << 23,
128+ PVRDMA_PORT_LINK_LATENCY_SUP = 1 << 24,
129+ PVRDMA_PORT_CLIENT_REG_SUP = 1 << 25,
130+ PVRDMA_PORT_IP_BASED_GIDS = 1 << 26,
131+ PVRDMA_PORT_CAP_FLAGS_MAX = PVRDMA_PORT_IP_BASED_GIDS,
132+};
133+
134+enum pvrdma_port_width {
135+ PVRDMA_WIDTH_1X = 1,
136+ PVRDMA_WIDTH_4X = 2,
137+ PVRDMA_WIDTH_8X = 4,
138+ PVRDMA_WIDTH_12X = 8,
139+};
140+
141+static inline int pvrdma_width_enum_to_int(enum pvrdma_port_width width)
142+{
143+ switch (width) {
144+ case PVRDMA_WIDTH_1X: return 1;
145+ case PVRDMA_WIDTH_4X: return 4;
146+ case PVRDMA_WIDTH_8X: return 8;
147+ case PVRDMA_WIDTH_12X: return 12;
148+ default: return -1;
149+ }
150+}
151+
152+enum pvrdma_port_speed {
153+ PVRDMA_SPEED_SDR = 1,
154+ PVRDMA_SPEED_DDR = 2,
155+ PVRDMA_SPEED_QDR = 4,
156+ PVRDMA_SPEED_FDR10 = 8,
157+ PVRDMA_SPEED_FDR = 16,
158+ PVRDMA_SPEED_EDR = 32,
159+};
160+
161+struct pvrdma_port_attr {
162+ enum pvrdma_port_state state;
163+ enum pvrdma_mtu max_mtu;
164+ enum pvrdma_mtu active_mtu;
165+ u32 gid_tbl_len;
166+ u32 port_cap_flags;
167+ u32 max_msg_sz;
168+ u32 bad_pkey_cntr;
169+ u32 qkey_viol_cntr;
170+ u16 pkey_tbl_len;
171+ u16 lid;
172+ u16 sm_lid;
173+ u8 lmc;
174+ u8 max_vl_num;
175+ u8 sm_sl;
176+ u8 subnet_timeout;
177+ u8 init_type_reply;
178+ u8 active_width;
179+ u8 active_speed;
180+ u8 phys_state;
181+ u8 reserved[2];
182+};
183+
184+struct pvrdma_global_route {
185+ union pvrdma_gid dgid;
186+ u32 flow_label;
187+ u8 sgid_index;
188+ u8 hop_limit;
189+ u8 traffic_class;
190+ u8 reserved;
191+};
192+
193+struct pvrdma_grh {
194+ __be32 version_tclass_flow;
195+ __be16 paylen;
196+ u8 next_hdr;
197+ u8 hop_limit;
198+ union pvrdma_gid sgid;
199+ union pvrdma_gid dgid;
200+};
201+
202+enum pvrdma_ah_flags {
203+ PVRDMA_AH_GRH = 1,
204+};
205+
206+enum pvrdma_rate {
207+ PVRDMA_RATE_PORT_CURRENT = 0,
208+ PVRDMA_RATE_2_5_GBPS = 2,
209+ PVRDMA_RATE_5_GBPS = 5,
210+ PVRDMA_RATE_10_GBPS = 3,
211+ PVRDMA_RATE_20_GBPS = 6,
212+ PVRDMA_RATE_30_GBPS = 4,
213+ PVRDMA_RATE_40_GBPS = 7,
214+ PVRDMA_RATE_60_GBPS = 8,
215+ PVRDMA_RATE_80_GBPS = 9,
216+ PVRDMA_RATE_120_GBPS = 10,
217+ PVRDMA_RATE_14_GBPS = 11,
218+ PVRDMA_RATE_56_GBPS = 12,
219+ PVRDMA_RATE_112_GBPS = 13,
220+ PVRDMA_RATE_168_GBPS = 14,
221+ PVRDMA_RATE_25_GBPS = 15,
222+ PVRDMA_RATE_100_GBPS = 16,
223+ PVRDMA_RATE_200_GBPS = 17,
224+ PVRDMA_RATE_300_GBPS = 18,
225+};
226+
227+struct pvrdma_ah_attr {
228+ struct pvrdma_global_route grh;
229+ u16 dlid;
230+ u16 vlan_id;
231+ u8 sl;
232+ u8 src_path_bits;
233+ u8 static_rate;
234+ u8 ah_flags;
235+ u8 port_num;
236+ u8 dmac[6];
237+ u8 reserved;
238+};
239+
240+enum pvrdma_cq_notify_flags {
241+ PVRDMA_CQ_SOLICITED = 1 << 0,
242+ PVRDMA_CQ_NEXT_COMP = 1 << 1,
243+ PVRDMA_CQ_SOLICITED_MASK = PVRDMA_CQ_SOLICITED |
244+ PVRDMA_CQ_NEXT_COMP,
245+ PVRDMA_CQ_REPORT_MISSED_EVENTS = 1 << 2,
246+};
247+
248+struct pvrdma_qp_cap {
249+ u32 max_send_wr;
250+ u32 max_recv_wr;
251+ u32 max_send_sge;
252+ u32 max_recv_sge;
253+ u32 max_inline_data;
254+ u32 reserved;
255+};
256+
257+enum pvrdma_sig_type {
258+ PVRDMA_SIGNAL_ALL_WR,
259+ PVRDMA_SIGNAL_REQ_WR,
260+};
261+
262+enum pvrdma_qp_type {
263+ PVRDMA_QPT_SMI,
264+ PVRDMA_QPT_GSI,
265+ PVRDMA_QPT_RC,
266+ PVRDMA_QPT_UC,
267+ PVRDMA_QPT_UD,
268+ PVRDMA_QPT_RAW_IPV6,
269+ PVRDMA_QPT_RAW_ETHERTYPE,
270+ PVRDMA_QPT_RAW_PACKET = 8,
271+ PVRDMA_QPT_XRC_INI = 9,
272+ PVRDMA_QPT_XRC_TGT,
273+ PVRDMA_QPT_MAX,
274+};
275+
276+enum pvrdma_qp_create_flags {
277+ PVRDMA_QP_CREATE_IPOPVRDMA_UD_LSO = 1 << 0,
278+ PVRDMA_QP_CREATE_BLOCK_MULTICAST_LOOPBACK = 1 << 1,
279+};
280+
281+enum pvrdma_qp_attr_mask {
282+ PVRDMA_QP_STATE = 1 << 0,
283+ PVRDMA_QP_CUR_STATE = 1 << 1,
284+ PVRDMA_QP_EN_SQD_ASYNC_NOTIFY = 1 << 2,
285+ PVRDMA_QP_ACCESS_FLAGS = 1 << 3,
286+ PVRDMA_QP_PKEY_INDEX = 1 << 4,
287+ PVRDMA_QP_PORT = 1 << 5,
288+ PVRDMA_QP_QKEY = 1 << 6,
289+ PVRDMA_QP_AV = 1 << 7,
290+ PVRDMA_QP_PATH_MTU = 1 << 8,
291+ PVRDMA_QP_TIMEOUT = 1 << 9,
292+ PVRDMA_QP_RETRY_CNT = 1 << 10,
293+ PVRDMA_QP_RNR_RETRY = 1 << 11,
294+ PVRDMA_QP_RQ_PSN = 1 << 12,
295+ PVRDMA_QP_MAX_QP_RD_ATOMIC = 1 << 13,
296+ PVRDMA_QP_ALT_PATH = 1 << 14,
297+ PVRDMA_QP_MIN_RNR_TIMER = 1 << 15,
298+ PVRDMA_QP_SQ_PSN = 1 << 16,
299+ PVRDMA_QP_MAX_DEST_RD_ATOMIC = 1 << 17,
300+ PVRDMA_QP_PATH_MIG_STATE = 1 << 18,
301+ PVRDMA_QP_CAP = 1 << 19,
302+ PVRDMA_QP_DEST_QPN = 1 << 20,
303+ PVRDMA_QP_ATTR_MASK_MAX = PVRDMA_QP_DEST_QPN,
304+};
305+
306+enum pvrdma_qp_state {
307+ PVRDMA_QPS_RESET,
308+ PVRDMA_QPS_INIT,
309+ PVRDMA_QPS_RTR,
310+ PVRDMA_QPS_RTS,
311+ PVRDMA_QPS_SQD,
312+ PVRDMA_QPS_SQE,
313+ PVRDMA_QPS_ERR,
314+};
315+
316+enum pvrdma_mig_state {
317+ PVRDMA_MIG_MIGRATED,
318+ PVRDMA_MIG_REARM,
319+ PVRDMA_MIG_ARMED,
320+};
321+
322+enum pvrdma_mw_type {
323+ PVRDMA_MW_TYPE_1 = 1,
324+ PVRDMA_MW_TYPE_2 = 2,
325+};
326+
327+struct pvrdma_qp_attr {
328+ enum pvrdma_qp_state qp_state;
329+ enum pvrdma_qp_state cur_qp_state;
330+ enum pvrdma_mtu path_mtu;
331+ enum pvrdma_mig_state path_mig_state;
332+ u32 qkey;
333+ u32 rq_psn;
334+ u32 sq_psn;
335+ u32 dest_qp_num;
336+ u32 qp_access_flags;
337+ u16 pkey_index;
338+ u16 alt_pkey_index;
339+ u8 en_sqd_async_notify;
340+ u8 sq_draining;
341+ u8 max_rd_atomic;
342+ u8 max_dest_rd_atomic;
343+ u8 min_rnr_timer;
344+ u8 port_num;
345+ u8 timeout;
346+ u8 retry_cnt;
347+ u8 rnr_retry;
348+ u8 alt_port_num;
349+ u8 alt_timeout;
350+ u8 reserved[5];
351+ struct pvrdma_qp_cap cap;
352+ struct pvrdma_ah_attr ah_attr;
353+ struct pvrdma_ah_attr alt_ah_attr;
354+};
355+
356+enum pvrdma_send_flags {
357+ PVRDMA_SEND_FENCE = 1 << 0,
358+ PVRDMA_SEND_SIGNALED = 1 << 1,
359+ PVRDMA_SEND_SOLICITED = 1 << 2,
360+ PVRDMA_SEND_INLINE = 1 << 3,
361+ PVRDMA_SEND_IP_CSUM = 1 << 4,
362+ PVRDMA_SEND_FLAGS_MAX = PVRDMA_SEND_IP_CSUM,
363+};
364+
365+enum pvrdma_access_flags {
366+ PVRDMA_ACCESS_LOCAL_WRITE = 1 << 0,
367+ PVRDMA_ACCESS_REMOTE_WRITE = 1 << 1,
368+ PVRDMA_ACCESS_REMOTE_READ = 1 << 2,
369+ PVRDMA_ACCESS_REMOTE_ATOMIC = 1 << 3,
370+ PVRDMA_ACCESS_MW_BIND = 1 << 4,
371+ PVRDMA_ZERO_BASED = 1 << 5,
372+ PVRDMA_ACCESS_ON_DEMAND = 1 << 6,
373+ PVRDMA_ACCESS_FLAGS_MAX = PVRDMA_ACCESS_ON_DEMAND,
374+};
375+
376+int pvrdma_query_device(struct ib_device *ibdev,
377+ struct ib_device_attr *props,
378+ struct ib_udata *udata);
379+int pvrdma_query_port(struct ib_device *ibdev, u8 port,
380+ struct ib_port_attr *props);
381+int pvrdma_query_gid(struct ib_device *ibdev, u8 port,
382+ int index, union ib_gid *gid);
383+int pvrdma_query_pkey(struct ib_device *ibdev, u8 port,
384+ u16 index, u16 *pkey);
385+enum rdma_link_layer pvrdma_port_link_layer(struct ib_device *ibdev,
386+ u8 port);
387+int pvrdma_modify_device(struct ib_device *ibdev, int mask,
388+ struct ib_device_modify *props);
389+int pvrdma_modify_port(struct ib_device *ibdev, u8 port,
390+ int mask, struct ib_port_modify *props);
391+int pvrdma_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
392+struct ib_ucontext *pvrdma_alloc_ucontext(struct ib_device *ibdev,
393+ struct ib_udata *udata);
394+int pvrdma_dealloc_ucontext(struct ib_ucontext *context);
395+struct ib_pd *pvrdma_alloc_pd(struct ib_device *ibdev,
396+ struct ib_ucontext *context,
397+ struct ib_udata *udata);
398+int pvrdma_dealloc_pd(struct ib_pd *ibpd);
399+struct ib_mr *pvrdma_get_dma_mr(struct ib_pd *pd, int acc);
400+struct ib_mr *pvrdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
401+ u64 virt_addr, int access_flags,
402+ struct ib_udata *udata);
403+int pvrdma_dereg_mr(struct ib_mr *mr);
404+struct ib_mr *pvrdma_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
405+ u32 max_num_sg);
406+int pvrdma_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg,
407+ int sg_nents, unsigned int *sg_offset);
408+int pvrdma_modify_cq(struct ib_cq *cq, u16 cq_count, u16 cq_period);
409+int pvrdma_resize_cq(struct ib_cq *ibcq, int entries,
410+ struct ib_udata *udata);
411+struct ib_cq *pvrdma_create_cq(struct ib_device *ibdev,
412+ const struct ib_cq_init_attr *attr,
413+ struct ib_ucontext *context,
414+ struct ib_udata *udata);
415+int pvrdma_resize_cq(struct ib_cq *ibcq, int entries,
416+ struct ib_udata *udata);
417+int pvrdma_destroy_cq(struct ib_cq *cq);
418+int pvrdma_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc);
419+int pvrdma_req_notify_cq(struct ib_cq *cq, enum ib_cq_notify_flags flags);
420+struct ib_ah *pvrdma_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr,
421+ struct ib_udata *udata);
422+int pvrdma_destroy_ah(struct ib_ah *ah);
423+struct ib_qp *pvrdma_create_qp(struct ib_pd *pd,
424+ struct ib_qp_init_attr *init_attr,
425+ struct ib_udata *udata);
426+int pvrdma_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
427+ int attr_mask, struct ib_udata *udata);
428+int pvrdma_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
429+ int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr);
430+int pvrdma_destroy_qp(struct ib_qp *qp);
431+int pvrdma_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
432+ struct ib_send_wr **bad_wr);
433+int pvrdma_post_recv(struct ib_qp *ibqp, struct ib_recv_wr *wr,
434+ struct ib_recv_wr **bad_wr);
435+
436+#endif /* __PVRDMA_VERBS_H__ */
drivers/infiniband/sw/rdmavt/cq.c+27−37
@@ -119,18 +119,17 @@ void rvt_cq_enter(struct rvt_cq *cq, struct ib_wc *entry, bool solicited) · 118 unmodified lines
119119 if (cq->notify == IB_CQ_NEXT_COMP ||
120120 (cq->notify == IB_CQ_SOLICITED &&
121121 (solicited || entry->status != IB_WC_SUCCESS))) {
122− struct kthread_worker *worker;
123122 /*
124123 * This will cause send_complete() to be called in
125124 * another thread.
126125 */
127− smp_read_barrier_depends(); /* see rvt_cq_exit */
128− worker = cq->rdi->worker;
129− if (likely(worker)) {
126+ spin_lock(&cq->rdi->n_cqs_lock);
127+ if (likely(cq->rdi->worker)) {
130128 cq->notify = RVT_CQ_NONE;
131129 cq->triggered++;
132− kthread_queue_work(worker, &cq->comptask);
130+ kthread_queue_work(cq->rdi->worker, &cq->comptask);
133131 }
132+ spin_unlock(&cq->rdi->n_cqs_lock);
134133 }
135134
136135 spin_unlock_irqrestore(&cq->lock, flags);
@@ -240,15 +239,15 @@ struct ib_cq *rvt_create_cq(struct ib_device *ibdev, · 103 unmodified lines
240239 }
241240 }
242241
243− spin_lock(&rdi->n_cqs_lock);
242+ spin_lock_irq(&rdi->n_cqs_lock);
244243 if (rdi->n_cqs_allocated == rdi->dparms.props.max_cq) {
245− spin_unlock(&rdi->n_cqs_lock);
244+ spin_unlock_irq(&rdi->n_cqs_lock);
246245 ret = ERR_PTR(-ENOMEM);
247246 goto bail_ip;
248247 }
249248
250249 rdi->n_cqs_allocated++;
251− spin_unlock(&rdi->n_cqs_lock);
250+ spin_unlock_irq(&rdi->n_cqs_lock);
252251
253252 if (cq->ip) {
254253 spin_lock_irq(&rdi->pending_lock);
@@ -296,9 +295,9 @@ int rvt_destroy_cq(struct ib_cq *ibcq) · 41 unmodified lines
296295 struct rvt_dev_info *rdi = cq->rdi;
297296
298297 kthread_flush_work(&cq->comptask);
299− spin_lock(&rdi->n_cqs_lock);
298+ spin_lock_irq(&rdi->n_cqs_lock);
300299 rdi->n_cqs_allocated--;
301− spin_unlock(&rdi->n_cqs_lock);
300+ spin_unlock_irq(&rdi->n_cqs_lock);
302301 if (cq->ip)
303302 kref_put(&cq->ip->ref, rvt_release_mmap_info);
304303 else
@@ -504,33 +503,23 @@ int rvt_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *entry) · 199 unmodified lines
504503 */
505504 int rvt_driver_cq_init(struct rvt_dev_info *rdi)
506505 {
507− int ret = 0;
508506 int cpu;
509− struct task_struct *task;
507+ struct kthread_worker *worker;
510508
511509 if (rdi->worker)
512510 return 0;
511+
513512 spin_lock_init(&rdi->n_cqs_lock);
514− rdi->worker = kzalloc(sizeof(*rdi->worker), GFP_KERNEL);
515− if (!rdi->worker)
516− return -ENOMEM;
517− kthread_init_worker(rdi->worker);
518− task = kthread_create_on_node(
519− kthread_worker_fn,
520− rdi->worker,
521− rdi->dparms.node,
522− "%s", rdi->dparms.cq_name);
523− if (IS_ERR(task)) {
524− kfree(rdi->worker);
525− rdi->worker = NULL;
526− return PTR_ERR(task);
527− }
528513
529− set_user_nice(task, MIN_NICE);
530514 cpu = cpumask_first(cpumask_of_node(rdi->dparms.node));
531− kthread_bind(task, cpu);
532− wake_up_process(task);
533− return ret;
515+ worker = kthread_create_worker_on_cpu(cpu, 0,
516+ "%s", rdi->dparms.cq_name);
517+ if (IS_ERR(worker))
518+ return PTR_ERR(worker);
519+
520+ set_user_nice(worker->task, MIN_NICE);
521+ rdi->worker = worker;
522+ return 0;
534523 }
535524
536525 /**
@@ -541,13 +530,14 @@ void rvt_cq_exit(struct rvt_dev_info *rdi) · 4 unmodified lines
541530 {
542531 struct kthread_worker *worker;
543532
544− worker = rdi->worker;
545− if (!worker)
533+ /* block future queuing from send_complete() */
534+ spin_lock_irq(&rdi->n_cqs_lock);
535+ if (!rdi->worker) {
536+ spin_unlock_irq(&rdi->n_cqs_lock);
546537 return;
547− /* blocks future queuing from send_complete() */
538+ }
548539 rdi->worker = NULL;
549− smp_wmb(); /* See rdi_cq_enter */
550− kthread_flush_worker(worker);
551− kthread_stop(worker->task);
552− kfree(worker);
540+ spin_unlock_irq(&rdi->n_cqs_lock);
541+
542+ kthread_destroy_worker(worker);
553543 }
drivers/infiniband/sw/rdmavt/mcast.c+2−3

This diff is large: its 19 lines aren’t shown here.

drivers/infiniband/sw/rdmavt/mr.c+13−9

This diff is large: its 111 lines aren’t shown here.

drivers/infiniband/sw/rdmavt/qp.c+19−1

This diff is large: its 34 lines aren’t shown here.

drivers/infiniband/sw/rdmavt/trace.h+4−137

This diff is large: its 148 lines aren’t shown here.

drivers/infiniband/sw/rdmavt/trace_mr.h+112−0

This diff is large: its 113 lines aren’t shown here.

drivers/infiniband/sw/rdmavt/trace_qp.h+96−0

This diff is large: its 97 lines aren’t shown here.

drivers/infiniband/sw/rdmavt/trace_rvt.h+81−0

This diff is large: its 82 lines aren’t shown here.

drivers/infiniband/sw/rdmavt/trace_tx.h+132−0

This diff is large: its 133 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_comp.c+7−2

This diff is large: its 38 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_loc.h+0−2

This diff is large: its 9 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_mr.c+3−0

This diff is large: its 10 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_net.c+3−5

This diff is large: its 36 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_param.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_pool.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_recv.c+5−6

This diff is large: its 22 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_req.c+11−9

This diff is large: its 76 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_resp.c+21−4

This diff is large: its 66 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_srq.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_task.c+19−0

This diff is large: its 38 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_task.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/infiniband/sw/rxe/rxe_verbs.c+16−5

This diff is large: its 57 lines aren’t shown here.

drivers/infiniband/ulp/ipoib/ipoib_cm.c+1−9

This diff is large: its 39 lines aren’t shown here.

drivers/infiniband/ulp/ipoib/ipoib_ib.c+1−4

This diff is large: its 13 lines aren’t shown here.

drivers/infiniband/ulp/ipoib/ipoib_main.c+1−4

This diff is large: its 13 lines aren’t shown here.

drivers/infiniband/ulp/ipoib/ipoib_multicast.c+5−2

This diff is large: its 14 lines aren’t shown here.

drivers/infiniband/ulp/iser/iser_verbs.c+4−1

This diff is large: its 16 lines aren’t shown here.

drivers/infiniband/ulp/isert/ib_isert.c+14−17

This diff is large: its 113 lines aren’t shown here.

drivers/infiniband/ulp/srp/ib_srp.c+32−16

This diff is large: its 157 lines aren’t shown here.

drivers/infiniband/ulp/srpt/ib_srpt.c+9−13

This diff is large: its 39 lines aren’t shown here.

drivers/net/ethernet/qlogic/qede/qede_roce.c+2−2

This diff is large: its 11 lines aren’t shown here.

drivers/net/vmxnet3/vmxnet3_int.h+1−2

This diff is large: its 11 lines aren’t shown here.

drivers/nvme/host/rdma.c+36−6

This diff is large: its 60 lines aren’t shown here.

drivers/nvme/target/rdma.c+3−0

This diff is large: its 10 lines aren’t shown here.

include/linux/mlx5/mlx5_ifc.h+1−1

This diff is large: its 9 lines aren’t shown here.

include/linux/pci_ids.h+1−0

This diff is large: its 8 lines aren’t shown here.

include/rdma/ib_cm.h+6−0

This diff is large: its 11 lines aren’t shown here.

include/rdma/ib_mad.h+1−1

This diff is large: its 9 lines aren’t shown here.

include/rdma/ib_verbs.h+56−14

This diff is large: its 163 lines aren’t shown here.

include/rdma/iw_cm.h+6−0

This diff is large: its 11 lines aren’t shown here.

include/rdma/opa_smi.h+0−2

This diff is large: its 9 lines aren’t shown here.

include/rdma/rdma_cm.h+25−0

This diff is large: its 30 lines aren’t shown here.

include/rdma/rdma_vt.h+24−22

This diff is large: its 77 lines aren’t shown here.

include/rdma/rdmavt_mr.h+7−3

This diff is large: its 19 lines aren’t shown here.

include/rdma/rdmavt_qp.h+77−0

This diff is large: its 98 lines aren’t shown here.

include/uapi/rdma/Kbuild+2−0

This diff is large: its 6 lines aren’t shown here.

include/uapi/rdma/hfi/hfi1_user.h+1−1

This diff is large: its 9 lines aren’t shown here.

include/uapi/rdma/hns-abi.hrenamed from drivers/infiniband/hw/hns/hns_roce_user.h+5−4

This diff is large: its 20 lines aren’t shown here.

include/uapi/rdma/ib_user_verbs.h+38−0

This diff is large: its 76 lines aren’t shown here.

include/uapi/rdma/mlx5-abi.h+37−1

This diff is large: its 70 lines aren’t shown here.

include/uapi/rdma/rdma_user_cm.h+6−6

This diff is large: its 40 lines aren’t shown here.

include/uapi/rdma/vmw_pvrdma-abi.h+289−0

This diff is large: its 290 lines aren’t shown here.

net/rds/rdma_transport.c+4−1

This diff is large: its 16 lines aren’t shown here.

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t