Pull rdma updates from Jason Gunthorpe: "A new Pensando ionic driver, a new Gen 3 HW support for Intel irdma, and lots of small bnxt_re improvements. - Small bug fixes and improves to hfi1, efa, mlx5, erdma, rdmarvt, siw - Allow userspace access to IB service records through the rdmacm - Optimize dma mapping for erdma - Fix shutdown of the GSI QP in mana - Support relaxed ordering MR and fix a corruption bug with mlx5 DMA Data Direct - Many improvement to bnxt_re: - Debugging features and counters - Improve performance of some commands - Change flow_label reporting in completions - Mirror vnic - RDMA flow support - New RDMA driver for Pensando Ethernet devices: ionic - Gen 3 hardware support for the Intel irdma driver - Fix rdma routing resolution with VRFs" * tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma: (85 commits) RDMA/ionic: Fix memory leak of admin q_wr RDMA/siw: Always report immediate post SQ errors RDMA/bnxt_re: improve clarity in ALLOC_PAGE handler RDMA/irdma: Remove unused struct irdma_cq fields RDMA/irdma: Fix positive vs negative error codes in irdma_post_send() RDMA/bnxt_re: Remove non-statistics counters from hw_counters RDMA/bnxt_re: Add debugfs info entry for device and resource information RDMA/bnxt_re: Fix incorrect errno used in function comments RDMA: Use %pe format specifier for error pointers RDMA/ionic: Use ether_addr_copy instead of memcpy RDMA/ionic: Fix build failure on SPARC due to xchg() operand size RDMA/rxe: Fix race in do_task() when draining IB/sa: Fix sa_local_svc_timeout_ms read race IB/ipoib: Ignore L3 master device RDMA/core: Use route entry flag to decide on loopback traffic RDMA/core: Resolve MAC of next-hop device without ARP support RDMA/core: Squash a single user static function RDMA/irdma: Update Kconfig RDMA/irdma: Extend CQE Error and Flush Handling for GEN3 Devices RDMA/irdma: Add Atomic Operations support ...
Unified · Split
| @@ -50,6 +50,7 @@ Contents: · 49 unmodified lines | |||
| 50 | neterion/s2io | 50 | neterion/s2io |
| 51 | netronome/nfp | 51 | netronome/nfp |
| 52 | pensando/ionic | 52 | pensando/ionic |
| 53 | pensando/ionic_rdma | ||
| 53 | qualcomm/ppe/ppe | 54 | qualcomm/ppe/ppe |
| 54 | smsc/smc9 | 55 | smsc/smc9 |
| 55 | stmicro/stmmac | 56 | stmicro/stmmac |
| @@ -13,6 +13,7 @@ Contents · 12 unmodified lines | |||
| 13 | - Identifying the Adapter | 13 | - Identifying the Adapter |
| 14 | - Enabling the driver | 14 | - Enabling the driver |
| 15 | - Configuring the driver | 15 | - Configuring the driver |
| 16 | - RDMA Support via Auxiliary Device | ||
| 16 | - Statistics | 17 | - Statistics |
| 17 | - Support | 18 | - Support |
| 18 | 19 | ||
| @@ -105,6 +106,15 @@ XDP · 86 unmodified lines | |||
| 105 | Support for XDP includes the basics, plus Jumbo frames, Redirect and | 106 | Support for XDP includes the basics, plus Jumbo frames, Redirect and |
| 106 | ndo_xmit. There is no current support for zero-copy sockets or HW offload. | 107 | ndo_xmit. There is no current support for zero-copy sockets or HW offload. |
| 107 | 108 | ||
| 109 | RDMA Support via Auxiliary Device | ||
| 110 | ================================= | ||
| 111 | |||
| 112 | The ionic driver supports RDMA (Remote Direct Memory Access) functionality | ||
| 113 | through the Linux auxiliary device framework when advertised by the firmware. | ||
| 114 | RDMA capability is detected during device initialization, and if supported, | ||
| 115 | the ethernet driver will create an auxiliary device that allows the RDMA | ||
| 116 | driver to bind and provide InfiniBand/RoCE functionality. | ||
| 117 | |||
| 108 | Statistics | 118 | Statistics |
| 109 | ========== | 119 | ========== |
| 110 | 120 | ||
| @@ -0,0 +1,52 @@ | |||
| 1 | .. SPDX-License-Identifier: GPL-2.0+ | ||
| 2 | |||
| 3 | =========================================================== | ||
| 4 | RDMA Driver for the AMD Pensando(R) Ethernet adapter family | ||
| 5 | =========================================================== | ||
| 6 | |||
| 7 | AMD Pensando RDMA driver. | ||
| 8 | Copyright (C) 2018-2025, Advanced Micro Devices, Inc. | ||
| 9 | |||
| 10 | Overview | ||
| 11 | ======== | ||
| 12 | |||
| 13 | The ionic_rdma driver provides Remote Direct Memory Access functionality | ||
| 14 | for AMD Pensando DSC (Distributed Services Card) devices. This driver | ||
| 15 | implements RDMA capabilities as an auxiliary driver that operates in | ||
| 16 | conjunction with the ionic ethernet driver. | ||
| 17 | |||
| 18 | The ionic ethernet driver detects RDMA capability during device | ||
| 19 | initialization and creates auxiliary devices that the ionic_rdma driver | ||
| 20 | binds to, establishing the RDMA data path and control interfaces. | ||
| 21 | |||
| 22 | Identifying the Adapter | ||
| 23 | ======================= | ||
| 24 | |||
| 25 | See Documentation/networking/device_drivers/ethernet/pensando/ionic.rst | ||
| 26 | for more information on identifying the adapter. | ||
| 27 | |||
| 28 | Enabling the driver | ||
| 29 | =================== | ||
| 30 | |||
| 31 | The ionic_rdma driver depends on the ionic ethernet driver. | ||
| 32 | See Documentation/networking/device_drivers/ethernet/pensando/ionic.rst | ||
| 33 | for detailed information on enabling and configuring the ionic driver. | ||
| 34 | |||
| 35 | The ionic_rdma driver is enabled via the standard kernel configuration system, | ||
| 36 | using the make command:: | ||
| 37 | |||
| 38 | make oldconfig/menuconfig/etc. | ||
| 39 | |||
| 40 | The driver is located in the menu structure at: | ||
| 41 | |||
| 42 | -> Device Drivers | ||
| 43 | -> InfiniBand support | ||
| 44 | -> AMD Pensando DSC RDMA/RoCE Support | ||
| 45 | |||
| 46 | Support | ||
| 47 | ======= | ||
| 48 | |||
| 49 | For general Linux RDMA support, please use the RDMA mailing | ||
| 50 | list, which is monitored by AMD Pensando personnel:: | ||
| 51 | |||
| 52 | linux-rdma@vger.kernel.org | ||
| @@ -1177,6 +1177,15 @@ F: Documentation/networking/device_drivers/ethernet/amd/pds_core.rst · 1176 unmodified lines | |||
| 1177 | F: drivers/net/ethernet/amd/pds_core/ | 1177 | F: drivers/net/ethernet/amd/pds_core/ |
| 1178 | F: include/linux/pds/ | 1178 | F: include/linux/pds/ |
| 1179 | 1179 | ||
| 1180 | AMD PENSANDO RDMA DRIVER | ||
| 1181 | M: Abhijit Gangurde <abhijit.gangurde@amd.com> | ||
| 1182 | M: Allen Hubbe <allen.hubbe@amd.com> | ||
| 1183 | L: linux-rdma@vger.kernel.org | ||
| 1184 | S: Maintained | ||
| 1185 | F: Documentation/networking/device_drivers/ethernet/pensando/ionic_rdma.rst | ||
| 1186 | F: drivers/infiniband/hw/ionic/ | ||
| 1187 | F: include/uapi/rdma/ionic-abi.h | ||
| 1188 | |||
| 1180 | AMD PMC DRIVER | 1189 | AMD PMC DRIVER |
| 1181 | M: Shyam Sundar S K <Shyam-sundar.S-k@amd.com> | 1190 | M: Shyam Sundar S K <Shyam-sundar.S-k@amd.com> |
| 1182 | L: platform-driver-x86@vger.kernel.org | 1191 | L: platform-driver-x86@vger.kernel.org |
| @@ -85,6 +85,7 @@ source "drivers/infiniband/hw/efa/Kconfig" · 84 unmodified lines | |||
| 85 | source "drivers/infiniband/hw/erdma/Kconfig" | 85 | source "drivers/infiniband/hw/erdma/Kconfig" |
| 86 | source "drivers/infiniband/hw/hfi1/Kconfig" | 86 | source "drivers/infiniband/hw/hfi1/Kconfig" |
| 87 | source "drivers/infiniband/hw/hns/Kconfig" | 87 | source "drivers/infiniband/hw/hns/Kconfig" |
| 88 | source "drivers/infiniband/hw/ionic/Kconfig" | ||
| 88 | source "drivers/infiniband/hw/irdma/Kconfig" | 89 | source "drivers/infiniband/hw/irdma/Kconfig" |
| 89 | source "drivers/infiniband/hw/mana/Kconfig" | 90 | source "drivers/infiniband/hw/mana/Kconfig" |
| 90 | source "drivers/infiniband/hw/mlx4/Kconfig" | 91 | source "drivers/infiniband/hw/mlx4/Kconfig" |
| @@ -446,63 +446,41 @@ static int addr6_resolve(struct sockaddr *src_sock, · 445 unmodified lines | |||
| 446 | } | 446 | } |
| 447 | #endif | 447 | #endif |
| 448 | 448 | ||
| 449 | static bool is_dst_local(const struct dst_entry *dst) | ||
| 450 | { | ||
| 451 | if (dst->ops->family == AF_INET) | ||
| 452 | return !!(dst_rtable(dst)->rt_type & RTN_LOCAL); | ||
| 453 | else if (dst->ops->family == AF_INET6) | ||
| 454 | return !!(dst_rt6_info(dst)->rt6i_flags & RTF_LOCAL); | ||
| 455 | else | ||
| 456 | return false; | ||
| 457 | } | ||
| 458 | |||
| 449 | static int addr_resolve_neigh(const struct dst_entry *dst, | 459 | static int addr_resolve_neigh(const struct dst_entry *dst, |
| 450 | const struct sockaddr *dst_in, | 460 | const struct sockaddr *dst_in, |
| 451 | struct rdma_dev_addr *addr, | 461 | struct rdma_dev_addr *addr, |
| 452 | unsigned int ndev_flags, | ||
| 453 | u32 seq) | 462 | u32 seq) |
| 454 | { | 463 | { |
| 455 | int ret = 0; | 464 | if (is_dst_local(dst)) { |
| 456 | 465 | /* When the destination is local entry, source and destination | |
| 457 | if (ndev_flags & IFF_LOOPBACK) { | 466 | * are same. Skip the neighbour lookup. |
| 467 | */ | ||
| 458 | memcpy(addr->dst_dev_addr, addr->src_dev_addr, MAX_ADDR_LEN); | 468 | memcpy(addr->dst_dev_addr, addr->src_dev_addr, MAX_ADDR_LEN); |
| 459 | } else { | 469 | return 0; |
| 460 | if (!(ndev_flags & IFF_NOARP)) { | ||
| 461 | /* If the device doesn't do ARP internally */ | ||
| 462 | ret = fetch_ha(dst, addr, dst_in, seq); | ||
| 463 | } | ||
| 464 | } | 470 | } |
| 465 | return ret; | ||
| 466 | } | ||
| 467 | |||
| 468 | static int copy_src_l2_addr(struct rdma_dev_addr *dev_addr, | ||
| 469 | const struct sockaddr *dst_in, | ||
| 470 | const struct dst_entry *dst, | ||
| 471 | const struct net_device *ndev) | ||
| 472 | { | ||
| 473 | int ret = 0; | ||
| 474 | |||
| 475 | if (dst->dev->flags & IFF_LOOPBACK) | ||
| 476 | ret = rdma_translate_ip(dst_in, dev_addr); | ||
| 477 | else | ||
| 478 | rdma_copy_src_l2_addr(dev_addr, dst->dev); | ||
| 479 | |||
| 480 | /* | ||
| 481 | * If there's a gateway and type of device not ARPHRD_INFINIBAND, | ||
| 482 | * we're definitely in RoCE v2 (as RoCE v1 isn't routable) set the | ||
| 483 | * network type accordingly. | ||
| 484 | */ | ||
| 485 | if (has_gateway(dst, dst_in->sa_family) && | ||
| 486 | ndev->type != ARPHRD_INFINIBAND) | ||
| 487 | dev_addr->network = dst_in->sa_family == AF_INET ? | ||
| 488 | RDMA_NETWORK_IPV4 : | ||
| 489 | RDMA_NETWORK_IPV6; | ||
| 490 | else | ||
| 491 | dev_addr->network = RDMA_NETWORK_IB; | ||
| 492 | 471 | ||
| 493 | return ret; | 472 | return fetch_ha(dst, addr, dst_in, seq); |
| 494 | } | 473 | } |
| 495 | 474 | ||
| 496 | static int rdma_set_src_addr_rcu(struct rdma_dev_addr *dev_addr, | 475 | static int rdma_set_src_addr_rcu(struct rdma_dev_addr *dev_addr, |
| 497 | unsigned int *ndev_flags, | ||
| 498 | const struct sockaddr *dst_in, | 476 | const struct sockaddr *dst_in, |
| 499 | const struct dst_entry *dst) | 477 | const struct dst_entry *dst) |
| 500 | { | 478 | { |
| 501 | struct net_device *ndev = READ_ONCE(dst->dev); | 479 | struct net_device *ndev = READ_ONCE(dst->dev); |
| 502 | 480 | ||
| 503 | *ndev_flags = ndev->flags; | ||
| 504 | /* A physical device must be the RDMA device to use */ | 481 | /* A physical device must be the RDMA device to use */ |
| 505 | if (ndev->flags & IFF_LOOPBACK) { | 482 | if (is_dst_local(dst)) { |
| 483 | int ret; | ||
| 506 | /* | 484 | /* |
| 507 | * RDMA (IB/RoCE, iWarp) doesn't run on lo interface or | 485 | * RDMA (IB/RoCE, iWarp) doesn't run on lo interface or |
| 508 | * loopback IP address. So if route is resolved to loopback | 486 | * loopback IP address. So if route is resolved to loopback |
| @@ -512,9 +490,27 @@ static int rdma_set_src_addr_rcu(struct rdma_dev_addr *dev_addr, · 3 unmodified lines | |||
| 512 | ndev = rdma_find_ndev_for_src_ip_rcu(dev_net(ndev), dst_in); | 490 | ndev = rdma_find_ndev_for_src_ip_rcu(dev_net(ndev), dst_in); |
| 513 | if (IS_ERR(ndev)) | 491 | if (IS_ERR(ndev)) |
| 514 | return -ENODEV; | 492 | return -ENODEV; |
| 493 | ret = rdma_translate_ip(dst_in, dev_addr); | ||
| 494 | if (ret) | ||
| 495 | return ret; | ||
| 496 | } else { | ||
| 497 | rdma_copy_src_l2_addr(dev_addr, dst->dev); | ||
| 515 | } | 498 | } |
| 516 | 499 | ||
| 517 | return copy_src_l2_addr(dev_addr, dst_in, dst, ndev); | 500 | /* |
| 501 | * If there's a gateway and type of device not ARPHRD_INFINIBAND, | ||
| 502 | * we're definitely in RoCE v2 (as RoCE v1 isn't routable) set the | ||
| 503 | * network type accordingly. | ||
| 504 | */ | ||
| 505 | if (has_gateway(dst, dst_in->sa_family) && | ||
| 506 | ndev->type != ARPHRD_INFINIBAND) | ||
| 507 | dev_addr->network = dst_in->sa_family == AF_INET ? | ||
| 508 | RDMA_NETWORK_IPV4 : | ||
| 509 | RDMA_NETWORK_IPV6; | ||
| 510 | else | ||
| 511 | dev_addr->network = RDMA_NETWORK_IB; | ||
| 512 | |||
| 513 | return 0; | ||
| 518 | } | 514 | } |
| 519 | 515 | ||
| 520 | static int set_addr_netns_by_gid_rcu(struct rdma_dev_addr *addr) | 516 | static int set_addr_netns_by_gid_rcu(struct rdma_dev_addr *addr) |
| @@ -551,7 +547,6 @@ static int addr_resolve(struct sockaddr *src_in, · 30 unmodified lines | |||
| 551 | u32 seq) | 547 | u32 seq) |
| 552 | { | 548 | { |
| 553 | struct dst_entry *dst = NULL; | 549 | struct dst_entry *dst = NULL; |
| 554 | unsigned int ndev_flags = 0; | ||
| 555 | struct rtable *rt = NULL; | 550 | struct rtable *rt = NULL; |
| 556 | int ret; | 551 | int ret; |
| 557 | 552 | ||
| @@ -588,7 +583,7 @@ static int addr_resolve(struct sockaddr *src_in, · 30 unmodified lines | |||
| 588 | rcu_read_unlock(); | 583 | rcu_read_unlock(); |
| 589 | goto done; | 584 | goto done; |
| 590 | } | 585 | } |
| 591 | ret = rdma_set_src_addr_rcu(addr, &ndev_flags, dst_in, dst); | 586 | ret = rdma_set_src_addr_rcu(addr, dst_in, dst); |
| 592 | rcu_read_unlock(); | 587 | rcu_read_unlock(); |
| 593 | 588 | ||
| 594 | /* | 589 | /* |
| @@ -596,7 +591,7 @@ static int addr_resolve(struct sockaddr *src_in, · 1 unmodified line | |||
| 596 | * only if src addr translation didn't fail. | 591 | * only if src addr translation didn't fail. |
| 597 | */ | 592 | */ |
| 598 | if (!ret && resolve_neigh) | 593 | if (!ret && resolve_neigh) |
| 599 | ret = addr_resolve_neigh(dst, dst_in, addr, ndev_flags, seq); | 594 | ret = addr_resolve_neigh(dst, dst_in, addr, seq); |
| 600 | 595 | ||
| 601 | if (src_in->sa_family == AF_INET) | 596 | if (src_in->sa_family == AF_INET) |
| 602 | ip_rt_put(rt); | 597 | ip_rt_put(rt); |
| @@ -110,8 +110,7 @@ void agent_send_response(const struct ib_mad_hdr *mad_hdr, const struct ib_grh * · 109 unmodified lines | |||
| 110 | agent = port_priv->agent[qpn]; | 110 | agent = port_priv->agent[qpn]; |
| 111 | ah = ib_create_ah_from_wc(agent->qp->pd, wc, grh, port_num); | 111 | ah = ib_create_ah_from_wc(agent->qp->pd, wc, grh, port_num); |
| 112 | if (IS_ERR(ah)) { | 112 | if (IS_ERR(ah)) { |
| 113 | dev_err(&device->dev, "ib_create_ah_from_wc error %ld\n", | 113 | dev_err(&device->dev, "ib_create_ah_from_wc error %pe\n", ah); |
| 114 | PTR_ERR(ah)); | ||
| 115 | return; | 114 | return; |
| 116 | } | 115 | } |
| 117 | 116 | ||
| @@ -1049,8 +1049,8 @@ static noinline void cm_destroy_id_wait_timeout(struct ib_cm_id *cm_id, · 1048 unmodified lines | |||
| 1049 | struct cm_id_private *cm_id_priv; | 1049 | struct cm_id_private *cm_id_priv; |
| 1050 | 1050 | ||
| 1051 | cm_id_priv = container_of(cm_id, struct cm_id_private, id); | 1051 | cm_id_priv = container_of(cm_id, struct cm_id_private, id); |
| 1052 | pr_err("%s: cm_id=%p timed out. state %d -> %d, refcnt=%d\n", __func__, | 1052 | pr_err_ratelimited("%s: cm_id=%p timed out. state %d -> %d, refcnt=%d\n", __func__, |
| 1053 | cm_id, old_state, cm_id->state, refcount_read(&cm_id_priv->refcount)); | 1053 | cm_id, old_state, cm_id->state, refcount_read(&cm_id_priv->refcount)); |
| 1054 | } | 1054 | } |
| 1055 | 1055 | ||
| 1056 | static void cm_destroy_id(struct ib_cm_id *cm_id, int err) | 1056 | static void cm_destroy_id(struct ib_cm_id *cm_id, int err) |
| @@ -2076,6 +2076,7 @@ static void _destroy_id(struct rdma_id_private *id_priv, · 2075 unmodified lines | |||
| 2076 | kfree(id_priv->id.route.path_rec); | 2076 | kfree(id_priv->id.route.path_rec); |
| 2077 | kfree(id_priv->id.route.path_rec_inbound); | 2077 | kfree(id_priv->id.route.path_rec_inbound); |
| 2078 | kfree(id_priv->id.route.path_rec_outbound); | 2078 | kfree(id_priv->id.route.path_rec_outbound); |
| 2079 | kfree(id_priv->id.route.service_recs); | ||
| 2079 | 2080 | ||
| 2080 | put_net(id_priv->id.route.addr.dev_addr.net); | 2081 | put_net(id_priv->id.route.addr.dev_addr.net); |
| 2081 | kfree(id_priv); | 2082 | kfree(id_priv); |
| @@ -3382,13 +3383,18 @@ err1: · 1300 unmodified lines | |||
| 3382 | int rdma_resolve_route(struct rdma_cm_id *id, unsigned long timeout_ms) | 3383 | int rdma_resolve_route(struct rdma_cm_id *id, unsigned long timeout_ms) |
| 3383 | { | 3384 | { |
| 3384 | struct rdma_id_private *id_priv; | 3385 | struct rdma_id_private *id_priv; |
| 3386 | enum rdma_cm_state state; | ||
| 3385 | int ret; | 3387 | int ret; |
| 3386 | 3388 | ||
| 3387 | if (!timeout_ms) | 3389 | if (!timeout_ms) |
| 3388 | return -EINVAL; | 3390 | return -EINVAL; |
| 3389 | 3391 | ||
| 3390 | id_priv = container_of(id, struct rdma_id_private, id); | 3392 | id_priv = container_of(id, struct rdma_id_private, id); |
| 3391 | if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_RESOLVED, RDMA_CM_ROUTE_QUERY)) | 3393 | state = id_priv->state; |
| 3394 | if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_RESOLVED, | ||
| 3395 | RDMA_CM_ROUTE_QUERY) && | ||
| 3396 | !cma_comp_exch(id_priv, RDMA_CM_ADDRINFO_RESOLVED, | ||
| 3397 | RDMA_CM_ROUTE_QUERY)) | ||
| 3392 | return -EINVAL; | 3398 | return -EINVAL; |
| 3393 | 3399 | ||
| 3394 | cma_id_get(id_priv); | 3400 | cma_id_get(id_priv); |
| @@ -3409,7 +3415,7 @@ int rdma_resolve_route(struct rdma_cm_id *id, unsigned long timeout_ms) · 14 unmodified lines | |||
| 3409 | 3415 | ||
| 3410 | return 0; | 3416 | return 0; |
| 3411 | err: | 3417 | err: |
| 3412 | cma_comp_exch(id_priv, RDMA_CM_ROUTE_QUERY, RDMA_CM_ADDR_RESOLVED); | 3418 | cma_comp_exch(id_priv, RDMA_CM_ROUTE_QUERY, state); |
| 3413 | cma_id_put(id_priv); | 3419 | cma_id_put(id_priv); |
| 3414 | return ret; | 3420 | return ret; |
| 3415 | } | 3421 | } |
| @@ -5506,3 +5512,129 @@ static void __exit cma_cleanup(void) · 2090 unmodified lines | |||
| 5506 | 5512 | ||
| 5507 | module_init(cma_init); | 5513 | module_init(cma_init); |
| 5508 | module_exit(cma_cleanup); | 5514 | module_exit(cma_cleanup); |
| 5515 | |||
| 5516 | static void cma_query_ib_service_handler(int status, | ||
| 5517 | struct sa_service_rec *recs, | ||
| 5518 | unsigned int num_recs, void *context) | ||
| 5519 | { | ||
| 5520 | struct cma_work *work = context; | ||
| 5521 | struct rdma_id_private *id_priv = work->id; | ||
| 5522 | struct sockaddr_ib *addr; | ||
| 5523 | |||
| 5524 | if (status) | ||
| 5525 | goto fail; | ||
| 5526 | |||
| 5527 | if (!num_recs) { | ||
| 5528 | status = -ENOENT; | ||
| 5529 | goto fail; | ||
| 5530 | } | ||
| 5531 | |||
| 5532 | if (id_priv->id.route.service_recs) { | ||
| 5533 | status = -EALREADY; | ||
| 5534 | goto fail; | ||
| 5535 | } | ||
| 5536 | |||
| 5537 | id_priv->id.route.service_recs = | ||
| 5538 | kmalloc_array(num_recs, sizeof(*recs), GFP_KERNEL); | ||
| 5539 | if (!id_priv->id.route.service_recs) { | ||
| 5540 | status = -ENOMEM; | ||
| 5541 | goto fail; | ||
| 5542 | } | ||
| 5543 | |||
| 5544 | id_priv->id.route.num_service_recs = num_recs; | ||
| 5545 | memcpy(id_priv->id.route.service_recs, recs, sizeof(*recs) * num_recs); | ||
| 5546 | |||
| 5547 | addr = (struct sockaddr_ib *)&id_priv->id.route.addr.dst_addr; | ||
| 5548 | addr->sib_family = AF_IB; | ||
| 5549 | addr->sib_addr = *(struct ib_addr *)&recs->gid; | ||
| 5550 | addr->sib_pkey = recs->pkey; | ||
| 5551 | addr->sib_sid = recs->id; | ||
| 5552 | rdma_addr_set_dgid(&id_priv->id.route.addr.dev_addr, | ||
| 5553 | (union ib_gid *)&addr->sib_addr); | ||
| 5554 | ib_addr_set_pkey(&id_priv->id.route.addr.dev_addr, | ||
| 5555 | ntohs(addr->sib_pkey)); | ||
| 5556 | |||
| 5557 | queue_work(cma_wq, &work->work); | ||
| 5558 | return; | ||
| 5559 | |||
| 5560 | fail: | ||
| 5561 | work->old_state = RDMA_CM_ADDRINFO_QUERY; | ||
| 5562 | work->new_state = RDMA_CM_ADDR_BOUND; | ||
| 5563 | work->event.event = RDMA_CM_EVENT_ADDRINFO_ERROR; | ||
| 5564 | work->event.status = status; | ||
| 5565 | pr_debug_ratelimited( | ||
| 5566 | "RDMA CM: SERVICE_ERROR: failed to query service record. status %d\n", | ||
| 5567 | status); | ||
| 5568 | queue_work(cma_wq, &work->work); | ||
| 5569 | } | ||
| 5570 | |||
| 5571 | static int cma_resolve_ib_service(struct rdma_id_private *id_priv, | ||
| 5572 | struct rdma_ucm_ib_service *ibs) | ||
| 5573 | { | ||
| 5574 | struct sa_service_rec sr = {}; | ||
| 5575 | ib_sa_comp_mask mask = 0; | ||
| 5576 | struct cma_work *work; | ||
| 5577 | |||
| 5578 | work = kzalloc(sizeof(*work), GFP_KERNEL); | ||
| 5579 | if (!work) | ||
| 5580 | return -ENOMEM; | ||
| 5581 | |||
| 5582 | cma_id_get(id_priv); | ||
| 5583 | |||
| 5584 | work->id = id_priv; | ||
| 5585 | INIT_WORK(&work->work, cma_work_handler); | ||
| 5586 | work->old_state = RDMA_CM_ADDRINFO_QUERY; | ||
| 5587 | work->new_state = RDMA_CM_ADDRINFO_RESOLVED; | ||
| 5588 | work->event.event = RDMA_CM_EVENT_ADDRINFO_RESOLVED; | ||
| 5589 | |||
| 5590 | if (ibs->flags & RDMA_USER_CM_IB_SERVICE_FLAG_ID) { | ||
| 5591 | sr.id = cpu_to_be64(ibs->service_id); | ||
| 5592 | mask |= IB_SA_SERVICE_REC_SERVICE_ID; | ||
| 5593 | } | ||
| 5594 | if (ibs->flags & RDMA_USER_CM_IB_SERVICE_FLAG_NAME) { | ||
| 5595 | strscpy(sr.name, ibs->service_name, sizeof(sr.name)); | ||
| 5596 | mask |= IB_SA_SERVICE_REC_SERVICE_NAME; | ||
| 5597 | } | ||
| 5598 | |||
| 5599 | id_priv->query_id = ib_sa_service_rec_get(&sa_client, | ||
| 5600 | id_priv->id.device, | ||
| 5601 | id_priv->id.port_num, | ||
| 5602 | &sr, mask, | ||
| 5603 | 2000, GFP_KERNEL, | ||
| 5604 | cma_query_ib_service_handler, | ||
| 5605 | work, &id_priv->query); | ||
| 5606 | |||
| 5607 | if (id_priv->query_id < 0) { | ||
| 5608 | cma_id_put(id_priv); | ||
| 5609 | kfree(work); | ||
| 5610 | return id_priv->query_id; | ||
| 5611 | } | ||
| 5612 | |||
| 5613 | return 0; | ||
| 5614 | } | ||
| 5615 | |||
| 5616 | int rdma_resolve_ib_service(struct rdma_cm_id *id, | ||
| 5617 | struct rdma_ucm_ib_service *ibs) | ||
| 5618 | { | ||
| 5619 | struct rdma_id_private *id_priv; | ||
| 5620 | int ret; | ||
| 5621 | |||
| 5622 | id_priv = container_of(id, struct rdma_id_private, id); | ||
| 5623 | if (!id_priv->cma_dev || | ||
| 5624 | !cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_ADDRINFO_QUERY)) | ||
| 5625 | return -EINVAL; | ||
| 5626 | |||
| 5627 | if (rdma_cap_ib_sa(id->device, id->port_num)) | ||
| 5628 | ret = cma_resolve_ib_service(id_priv, ibs); | ||
| 5629 | else | ||
| 5630 | ret = -EOPNOTSUPP; | ||
| 5631 | |||
| 5632 | if (ret) | ||
| 5633 | goto err; | ||
| 5634 | |||
| 5635 | return 0; | ||
| 5636 | err: | ||
| 5637 | cma_comp_exch(id_priv, RDMA_CM_ADDRINFO_QUERY, RDMA_CM_ADDR_BOUND); | ||
| 5638 | return ret; | ||
| 5639 | } | ||
| 5640 | EXPORT_SYMBOL(rdma_resolve_ib_service); | ||
| @@ -47,7 +47,9 @@ enum rdma_cm_state { · 46 unmodified lines | |||
| 47 | RDMA_CM_ADDR_BOUND, | 47 | RDMA_CM_ADDR_BOUND, |
| 48 | RDMA_CM_LISTEN, | 48 | RDMA_CM_LISTEN, |
| 49 | RDMA_CM_DEVICE_REMOVAL, | 49 | RDMA_CM_DEVICE_REMOVAL, |
| 50 | RDMA_CM_DESTROYING | 50 | RDMA_CM_DESTROYING, |
| 51 | RDMA_CM_ADDRINFO_QUERY, | ||
| 52 | RDMA_CM_ADDRINFO_RESOLVED | ||
| 51 | }; | 53 | }; |
| 52 | 54 | ||
| 53 | struct rdma_id_private { | 55 | struct rdma_id_private { |
| @@ -1543,7 +1543,7 @@ static void __ib_unregister_device(struct ib_device *ib_dev) · 1542 unmodified lines | |||
| 1543 | 1543 | ||
| 1544 | /* | 1544 | /* |
| 1545 | * We have a registration lock so that all the calls to unregister are | 1545 | * We have a registration lock so that all the calls to unregister are |
| 1546 | * fully fenced, once any unregister returns the device is truely | 1546 | * fully fenced, once any unregister returns the device is truly |
| 1547 | * unregistered even if multiple callers are unregistering it at the | 1547 | * unregistered even if multiple callers are unregistering it at the |
| 1548 | * same time. This also interacts with the registration flow and | 1548 | * same time. This also interacts with the registration flow and |
| 1549 | * provides sane semantics if register and unregister are racing. | 1549 | * provides sane semantics if register and unregister are racing. |
| @@ -107,6 +107,8 @@ struct ib_sa_device { · 106 unmodified lines | |||
| 107 | struct ib_sa_query { | 107 | struct ib_sa_query { |
| 108 | void (*callback)(struct ib_sa_query *sa_query, int status, | 108 | void (*callback)(struct ib_sa_query *sa_query, int status, |
| 109 | struct ib_sa_mad *mad); | 109 | struct ib_sa_mad *mad); |
| 110 | void (*rmpp_callback)(struct ib_sa_query *sa_query, int status, | ||
| 111 | struct ib_mad_recv_wc *mad); | ||
| 110 | void (*release)(struct ib_sa_query *); | 112 | void (*release)(struct ib_sa_query *); |
| 111 | struct ib_sa_client *client; | 113 | struct ib_sa_client *client; |
| 112 | struct ib_sa_port *port; | 114 | struct ib_sa_port *port; |
| @@ -150,6 +152,13 @@ struct ib_sa_mcmember_query { · 37 unmodified lines | |||
| 150 | struct ib_sa_query sa_query; | 152 | struct ib_sa_query sa_query; |
| 151 | }; | 153 | }; |
| 152 | 154 | ||
| 155 | struct ib_sa_service_query { | ||
| 156 | void (*callback)(int status, struct sa_service_rec *rec, | ||
| 157 | unsigned int num_services, void *context); | ||
| 158 | void *context; | ||
| 159 | struct ib_sa_query sa_query; | ||
| 160 | }; | ||
| 161 | |||
| 153 | static LIST_HEAD(ib_nl_request_list); | 162 | static LIST_HEAD(ib_nl_request_list); |
| 154 | static DEFINE_SPINLOCK(ib_nl_request_lock); | 163 | static DEFINE_SPINLOCK(ib_nl_request_lock); |
| 155 | static atomic_t ib_nl_sa_request_seq; | 164 | static atomic_t ib_nl_sa_request_seq; |
| @@ -684,6 +693,58 @@ static const struct ib_field guidinfo_rec_table[] = { · 528 unmodified lines | |||
| 684 | .size_bits = 512 }, | 693 | .size_bits = 512 }, |
| 685 | }; | 694 | }; |
| 686 | 695 | ||
| 696 | #define SERVICE_REC_FIELD(field) \ | ||
| 697 | .struct_offset_bytes = offsetof(struct sa_service_rec, field), \ | ||
| 698 | .struct_size_bytes = sizeof_field(struct sa_service_rec, field), \ | ||
| 699 | .field_name = "sa_service_rec:" #field | ||
| 700 | |||
| 701 | static const struct ib_field service_rec_table[] = { | ||
| 702 | { SERVICE_REC_FIELD(id), | ||
| 703 | .offset_words = 0, | ||
| 704 | .offset_bits = 0, | ||
| 705 | .size_bits = 64 }, | ||
| 706 | { SERVICE_REC_FIELD(gid), | ||
| 707 | .offset_words = 2, | ||
| 708 | .offset_bits = 0, | ||
| 709 | .size_bits = 128 }, | ||
| 710 | { SERVICE_REC_FIELD(pkey), | ||
| 711 | .offset_words = 6, | ||
| 712 | .offset_bits = 0, | ||
| 713 | .size_bits = 16 }, | ||
| 714 | { RESERVED, | ||
| 715 | .offset_words = 6, | ||
| 716 | .offset_bits = 16, | ||
| 717 | .size_bits = 16 }, | ||
| 718 | { SERVICE_REC_FIELD(lease), | ||
| 719 | .offset_words = 7, | ||
| 720 | .offset_bits = 0, | ||
| 721 | .size_bits = 32 }, | ||
| 722 | { SERVICE_REC_FIELD(key), | ||
| 723 | .offset_words = 8, | ||
| 724 | .offset_bits = 0, | ||
| 725 | .size_bits = 128 }, | ||
| 726 | { SERVICE_REC_FIELD(name), | ||
| 727 | .offset_words = 12, | ||
| 728 | .offset_bits = 0, | ||
| 729 | .size_bits = 512 }, | ||
| 730 | { SERVICE_REC_FIELD(data_8), | ||
| 731 | .offset_words = 28, | ||
| 732 | .offset_bits = 0, | ||
| 733 | .size_bits = 128 }, | ||
| 734 | { SERVICE_REC_FIELD(data_16), | ||
| 735 | .offset_words = 32, | ||
| 736 | .offset_bits = 0, | ||
| 737 | .size_bits = 128 }, | ||
| 738 | { SERVICE_REC_FIELD(data_32), | ||
| 739 | .offset_words = 36, | ||
| 740 | .offset_bits = 0, | ||
| 741 | .size_bits = 128 }, | ||
| 742 | { SERVICE_REC_FIELD(data_64), | ||
| 743 | .offset_words = 40, | ||
| 744 | .offset_bits = 0, | ||
| 745 | .size_bits = 128 }, | ||
| 746 | }; | ||
| 747 | |||
| 687 | #define RDMA_PRIMARY_PATH_MAX_REC_NUM 3 | 748 | #define RDMA_PRIMARY_PATH_MAX_REC_NUM 3 |
| 688 | 749 | ||
| 689 | static inline void ib_sa_disable_local_svc(struct ib_sa_query *query) | 750 | static inline void ib_sa_disable_local_svc(struct ib_sa_query *query) |
| @@ -1013,6 +1074,8 @@ int ib_nl_handle_set_timeout(struct sk_buff *skb, · 323 unmodified lines | |||
| 1013 | if (timeout > IB_SA_LOCAL_SVC_TIMEOUT_MAX) | 1074 | if (timeout > IB_SA_LOCAL_SVC_TIMEOUT_MAX) |
| 1014 | timeout = IB_SA_LOCAL_SVC_TIMEOUT_MAX; | 1075 | timeout = IB_SA_LOCAL_SVC_TIMEOUT_MAX; |
| 1015 | 1076 | ||
| 1077 | spin_lock_irqsave(&ib_nl_request_lock, flags); | ||
| 1078 | |||
| 1016 | delta = timeout - sa_local_svc_timeout_ms; | 1079 | delta = timeout - sa_local_svc_timeout_ms; |
| 1017 | if (delta < 0) | 1080 | if (delta < 0) |
| 1018 | abs_delta = -delta; | 1081 | abs_delta = -delta; |
| @@ -1020,7 +1083,6 @@ int ib_nl_handle_set_timeout(struct sk_buff *skb, · 1 unmodified line | |||
| 1020 | abs_delta = delta; | 1083 | abs_delta = delta; |
| 1021 | 1084 | ||
| 1022 | if (delta != 0) { | 1085 | if (delta != 0) { |
| 1023 | spin_lock_irqsave(&ib_nl_request_lock, flags); | ||
| 1024 | sa_local_svc_timeout_ms = timeout; | 1086 | sa_local_svc_timeout_ms = timeout; |
| 1025 | list_for_each_entry(query, &ib_nl_request_list, list) { | 1087 | list_for_each_entry(query, &ib_nl_request_list, list) { |
| 1026 | if (delta < 0 && abs_delta > query->timeout) | 1088 | if (delta < 0 && abs_delta > query->timeout) |
| @@ -1038,9 +1100,10 @@ int ib_nl_handle_set_timeout(struct sk_buff *skb, · 11 unmodified lines | |||
| 1038 | if (delay) | 1100 | if (delay) |
| 1039 | mod_delayed_work(ib_nl_wq, &ib_nl_timed_work, | 1101 | mod_delayed_work(ib_nl_wq, &ib_nl_timed_work, |
| 1040 | (unsigned long)delay); | 1102 | (unsigned long)delay); |
| 1041 | spin_unlock_irqrestore(&ib_nl_request_lock, flags); | ||
| 1042 | } | 1103 | } |
| 1043 | 1104 | ||
| 1105 | spin_unlock_irqrestore(&ib_nl_request_lock, flags); | ||
| 1106 | |||
| 1044 | settimeout_out: | 1107 | settimeout_out: |
| 1045 | return 0; | 1108 | return 0; |
| 1046 | } | 1109 | } |
| @@ -1390,6 +1453,20 @@ void ib_sa_pack_path(struct sa_path_rec *rec, void *attribute) · 343 unmodified lines | |||
| 1390 | } | 1453 | } |
| 1391 | EXPORT_SYMBOL(ib_sa_pack_path); | 1454 | EXPORT_SYMBOL(ib_sa_pack_path); |
| 1392 | 1455 | ||
| 1456 | void ib_sa_pack_service(struct sa_service_rec *rec, void *attribute) | ||
| 1457 | { | ||
| 1458 | ib_pack(service_rec_table, ARRAY_SIZE(service_rec_table), rec, | ||
| 1459 | attribute); | ||
| 1460 | } | ||
| 1461 | EXPORT_SYMBOL(ib_sa_pack_service); | ||
| 1462 | |||
| 1463 | void ib_sa_unpack_service(void *attribute, struct sa_service_rec *rec) | ||
| 1464 | { | ||
| 1465 | ib_unpack(service_rec_table, ARRAY_SIZE(service_rec_table), attribute, | ||
| 1466 | rec); | ||
| 1467 | } | ||
| 1468 | EXPORT_SYMBOL(ib_sa_unpack_service); | ||
| 1469 | |||
| 1393 | static bool ib_sa_opa_pathrecord_support(struct ib_sa_client *client, | 1470 | static bool ib_sa_opa_pathrecord_support(struct ib_sa_client *client, |
| 1394 | struct ib_sa_device *sa_dev, | 1471 | struct ib_sa_device *sa_dev, |
| 1395 | u32 port_num) | 1472 | u32 port_num) |
| @@ -1479,6 +1556,68 @@ static void ib_sa_path_rec_callback(struct ib_sa_query *sa_query, · 83 unmodified lines | |||
| 1479 | } | 1556 | } |
| 1480 | } | 1557 | } |
| 1481 | 1558 | ||
| 1559 | #define IB_SA_DATA_OFFS 56 | ||
| 1560 | #define IB_SERVICE_REC_SZ 176 | ||
| 1561 | |||
| 1562 | static void ib_unpack_service_rmpp(struct sa_service_rec *rec, | ||
| 1563 | struct ib_mad_recv_wc *mad_wc, | ||
| 1564 | int num_services) | ||
| 1565 | { | ||
| 1566 | unsigned int cp_sz, data_i, data_size, rec_i = 0, buf_i = 0; | ||
| 1567 | struct ib_mad_recv_buf *mad_buf; | ||
| 1568 | u8 buf[IB_SERVICE_REC_SZ]; | ||
| 1569 | u8 *data; | ||
| 1570 | |||
| 1571 | data_size = sizeof(((struct ib_sa_mad *) mad_buf->mad)->data); | ||
| 1572 | |||
| 1573 | list_for_each_entry(mad_buf, &mad_wc->rmpp_list, list) { | ||
| 1574 | data = ((struct ib_sa_mad *) mad_buf->mad)->data; | ||
| 1575 | data_i = 0; | ||
| 1576 | while (data_i < data_size && rec_i < num_services) { | ||
| 1577 | cp_sz = min(IB_SERVICE_REC_SZ - buf_i, | ||
| 1578 | data_size - data_i); | ||
| 1579 | memcpy(buf + buf_i, data + data_i, cp_sz); | ||
| 1580 | data_i += cp_sz; | ||
| 1581 | buf_i += cp_sz; | ||
| 1582 | if (buf_i == IB_SERVICE_REC_SZ) { | ||
| 1583 | ib_sa_unpack_service(buf, rec + rec_i); | ||
| 1584 | buf_i = 0; | ||
| 1585 | rec_i++; | ||
| 1586 | } | ||
| 1587 | } | ||
| 1588 | } | ||
| 1589 | } | ||
| 1590 | |||
| 1591 | static void ib_sa_service_rec_callback(struct ib_sa_query *sa_query, int status, | ||
| 1592 | struct ib_mad_recv_wc *mad_wc) | ||
| 1593 | { | ||
| 1594 | struct ib_sa_service_query *query = | ||
| 1595 | container_of(sa_query, struct ib_sa_service_query, sa_query); | ||
| 1596 | struct sa_service_rec *rec; | ||
| 1597 | int num_services; | ||
| 1598 | |||
| 1599 | if (!mad_wc || !mad_wc->recv_buf.mad) { | ||
| 1600 | query->callback(status, NULL, 0, query->context); | ||
| 1601 | return; | ||
| 1602 | } | ||
| 1603 | |||
| 1604 | num_services = (mad_wc->mad_len - IB_SA_DATA_OFFS) / IB_SERVICE_REC_SZ; | ||
| 1605 | if (!num_services) { | ||
| 1606 | query->callback(-ENODATA, NULL, 0, query->context); | ||
| 1607 | return; | ||
| 1608 | } | ||
| 1609 | |||
| 1610 | rec = kmalloc_array(num_services, sizeof(*rec), GFP_KERNEL); | ||
| 1611 | if (!rec) { | ||
| 1612 | query->callback(-ENOMEM, NULL, 0, query->context); | ||
| 1613 | return; | ||
| 1614 | } | ||
| 1615 | |||
| 1616 | ib_unpack_service_rmpp(rec, mad_wc, num_services); | ||
| 1617 | query->callback(status, rec, num_services, query->context); | ||
| 1618 | kfree(rec); | ||
| 1619 | } | ||
| 1620 | |||
| 1482 | static void ib_sa_path_rec_release(struct ib_sa_query *sa_query) | 1621 | static void ib_sa_path_rec_release(struct ib_sa_query *sa_query) |
| 1483 | { | 1622 | { |
| 1484 | struct ib_sa_path_query *query = | 1623 | struct ib_sa_path_query *query = |
| @@ -1488,6 +1627,14 @@ static void ib_sa_path_rec_release(struct ib_sa_query *sa_query) · 3 unmodified lines | |||
| 1488 | kfree(query); | 1627 | kfree(query); |
| 1489 | } | 1628 | } |
| 1490 | 1629 | ||
| 1630 | static void ib_sa_service_rec_release(struct ib_sa_query *sa_query) | ||
| 1631 | { | ||
| 1632 | struct ib_sa_service_query *query = | ||
| 1633 | container_of(sa_query, struct ib_sa_service_query, sa_query); | ||
| 1634 | |||
| 1635 | kfree(query); | ||
| 1636 | } | ||
| 1637 | |||
| 1491 | /** | 1638 | /** |
| 1492 | * ib_sa_path_rec_get - Start a Path get query | 1639 | * ib_sa_path_rec_get - Start a Path get query |
| 1493 | * @client:SA client | 1640 | * @client:SA client |
| @@ -1618,6 +1765,101 @@ err1: · 124 unmodified lines | |||
| 1618 | } | 1765 | } |
| 1619 | EXPORT_SYMBOL(ib_sa_path_rec_get); | 1766 | EXPORT_SYMBOL(ib_sa_path_rec_get); |
| 1620 | 1767 | ||
| 1768 | /** | ||
| 1769 | * ib_sa_service_rec_get - Start a Service get query | ||
| 1770 | * @client: SA client | ||
| 1771 | * @device: device to send query on | ||
| 1772 | * @port_num: port number to send query on | ||
| 1773 | * @rec: Service Record to send in query | ||
| 1774 | * @comp_mask: component mask to send in query | ||
| 1775 | * @timeout_ms: time to wait for response | ||
| 1776 | * @gfp_mask: GFP mask to use for internal allocations | ||
| 1777 | * @callback: function called when query completes, times out or is | ||
| 1778 | * canceled | ||
| 1779 | * @context: opaque user context passed to callback | ||
| 1780 | * @sa_query: query context, used to cancel query | ||
| 1781 | * | ||
| 1782 | * Send a Service Record Get query to the SA to look up a path. The | ||
| 1783 | * callback function will be called when the query completes (or | ||
| 1784 | * fails); status is 0 for a successful response, -EINTR if the query | ||
| 1785 | * is canceled, -ETIMEDOUT is the query timed out, or -EIO if an error | ||
| 1786 | * occurred sending the query. The resp parameter of the callback is | ||
| 1787 | * only valid if status is 0. | ||
| 1788 | * | ||
| 1789 | * If the return value of ib_sa_service_rec_get() is negative, it is an | ||
| 1790 | * error code. Otherwise it is a query ID that can be used to cancel | ||
| 1791 | * the query. | ||
| 1792 | */ | ||
| 1793 | int ib_sa_service_rec_get(struct ib_sa_client *client, | ||
| 1794 | struct ib_device *device, u32 port_num, | ||
| 1795 | struct sa_service_rec *rec, | ||
| 1796 | ib_sa_comp_mask comp_mask, | ||
| 1797 | unsigned long timeout_ms, gfp_t gfp_mask, | ||
| 1798 | void (*callback)(int status, | ||
| 1799 | struct sa_service_rec *resp, | ||
| 1800 | unsigned int num_services, | ||
| 1801 | void *context), | ||
| 1802 | void *context, struct ib_sa_query **sa_query) | ||
| 1803 | { | ||
| 1804 | struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client); | ||
| 1805 | struct ib_sa_service_query *query; | ||
| 1806 | struct ib_mad_agent *agent; | ||
| 1807 | struct ib_sa_port *port; | ||
| 1808 | struct ib_sa_mad *mad; | ||
| 1809 | int ret; | ||
| 1810 | |||
| 1811 | if (!sa_dev) | ||
| 1812 | return -ENODEV; | ||
| 1813 | |||
| 1814 | port = &sa_dev->port[port_num - sa_dev->start_port]; | ||
| 1815 | agent = port->agent; | ||
| 1816 | |||
| 1817 | query = kzalloc(sizeof(*query), gfp_mask); | ||
| 1818 | if (!query) | ||
| 1819 | return -ENOMEM; | ||
| 1820 | |||
| 1821 | query->sa_query.port = port; | ||
| 1822 | |||
| 1823 | ret = alloc_mad(&query->sa_query, gfp_mask); | ||
| 1824 | if (ret) | ||
| 1825 | goto err1; | ||
| 1826 | |||
| 1827 | ib_sa_client_get(client); | ||
| 1828 | query->sa_query.client = client; | ||
| 1829 | query->callback = callback; | ||
| 1830 | query->context = context; | ||
| 1831 | |||
| 1832 | mad = query->sa_query.mad_buf->mad; | ||
| 1833 | init_mad(&query->sa_query, agent); | ||
| 1834 | |||
| 1835 | query->sa_query.rmpp_callback = callback ? ib_sa_service_rec_callback : | ||
| 1836 | NULL; | ||
| 1837 | query->sa_query.release = ib_sa_service_rec_release; | ||
| 1838 | mad->mad_hdr.method = IB_MGMT_METHOD_GET_TABLE; | ||
| 1839 | mad->mad_hdr.attr_id = cpu_to_be16(IB_SA_ATTR_SERVICE_REC); | ||
| 1840 | mad->sa_hdr.comp_mask = comp_mask; | ||
| 1841 | |||
| 1842 | ib_sa_pack_service(rec, mad->data); | ||
| 1843 | |||
| 1844 | *sa_query = &query->sa_query; | ||
| 1845 | query->sa_query.mad_buf->context[1] = rec; | ||
| 1846 | |||
| 1847 | ret = send_mad(&query->sa_query, timeout_ms, gfp_mask); | ||
| 1848 | if (ret < 0) | ||
| 1849 | goto err2; | ||
| 1850 | |||
| 1851 | return ret; | ||
| 1852 | |||
| 1853 | err2: | ||
| 1854 | *sa_query = NULL; | ||
| 1855 | ib_sa_client_put(query->sa_query.client); | ||
| 1856 | free_mad(&query->sa_query); | ||
| 1857 | err1: | ||
| 1858 | kfree(query); | ||
| 1859 | return ret; | ||
| 1860 | } | ||
| 1861 | EXPORT_SYMBOL(ib_sa_service_rec_get); | ||
| 1862 | |||
| 1621 | static void ib_sa_mcmember_rec_callback(struct ib_sa_query *sa_query, | 1863 | static void ib_sa_mcmember_rec_callback(struct ib_sa_query *sa_query, |
| 1622 | int status, struct ib_sa_mad *mad) | 1864 | int status, struct ib_sa_mad *mad) |
| 1623 | { | 1865 | { |
| @@ -1987,23 +2229,29 @@ static void send_handler(struct ib_mad_agent *agent, · 363 unmodified lines | |||
| 1987 | { | 2229 | { |
| 1988 | struct ib_sa_query *query = mad_send_wc->send_buf->context[0]; | 2230 | struct ib_sa_query *query = mad_send_wc->send_buf->context[0]; |
| 1989 | unsigned long flags; | 2231 | unsigned long flags; |
| 2232 | int status = 0; | ||
| 1990 | 2233 | ||
| 1991 | if (query->callback) | 2234 | if (query->callback || query->rmpp_callback) { |
| 1992 | switch (mad_send_wc->status) { | 2235 | switch (mad_send_wc->status) { |
| 1993 | case IB_WC_SUCCESS: | 2236 | case IB_WC_SUCCESS: |
| 1994 | /* No callback -- already got recv */ | 2237 | /* No callback -- already got recv */ |
| 1995 | break; | 2238 | break; |
| 1996 | case IB_WC_RESP_TIMEOUT_ERR: | 2239 | case IB_WC_RESP_TIMEOUT_ERR: |
| 1997 | query->callback(query, -ETIMEDOUT, NULL); | 2240 | status = -ETIMEDOUT; |
| 1998 | break; | 2241 | break; |
| 1999 | case IB_WC_WR_FLUSH_ERR: | 2242 | case IB_WC_WR_FLUSH_ERR: |
| 2000 | query->callback(query, -EINTR, NULL); | 2243 | status = -EINTR; |
| 2001 | break; | 2244 | break; |
| 2002 | default: | 2245 | default: |
| 2003 | query->callback(query, -EIO, NULL); | 2246 | status = -EIO; |
| 2004 | break; | 2247 | break; |
| 2005 | } | 2248 | } |
| 2006 | 2249 | ||
| 2250 | if (status) | ||
| 2251 | query->callback ? query->callback(query, status, NULL) : | ||
| 2252 | query->rmpp_callback(query, status, NULL); | ||
| 2253 | } | ||
| 2254 | |||
| 2007 | xa_lock_irqsave(&queries, flags); | 2255 | xa_lock_irqsave(&queries, flags); |
| 2008 | __xa_erase(&queries, query->id); | 2256 | __xa_erase(&queries, query->id); |
| 2009 | xa_unlock_irqrestore(&queries, flags); | 2257 | xa_unlock_irqrestore(&queries, flags); |
| @@ -2019,17 +2267,25 @@ static void recv_handler(struct ib_mad_agent *mad_agent, · 9 unmodified lines | |||
| 2019 | struct ib_mad_recv_wc *mad_recv_wc) | 2267 | struct ib_mad_recv_wc *mad_recv_wc) |
| 2020 | { | 2268 | { |
| 2021 | struct ib_sa_query *query; | 2269 | struct ib_sa_query *query; |
| 2270 | struct ib_mad *mad; | ||
| 2271 | |||
| 2022 | 2272 | ||
| 2023 | if (!send_buf) | 2273 | if (!send_buf) |
| 2024 | return; | 2274 | return; |
| 2025 | 2275 | ||
| 2026 | query = send_buf->context[0]; | 2276 | query = send_buf->context[0]; |
| 2027 | if (query->callback) { | 2277 | mad = mad_recv_wc->recv_buf.mad; |
| 2278 | |||
| 2279 | if (query->rmpp_callback) { | ||
| 2280 | if (mad_recv_wc->wc->status == IB_WC_SUCCESS) | ||
| 2281 | query->rmpp_callback(query, mad->mad_hdr.status ? | ||
| 2282 | -EINVAL : 0, mad_recv_wc); | ||
| 2283 | else | ||
| 2284 | query->rmpp_callback(query, -EIO, NULL); | ||
| 2285 | } else if (query->callback) { | ||
| 2028 | if (mad_recv_wc->wc->status == IB_WC_SUCCESS) | 2286 | if (mad_recv_wc->wc->status == IB_WC_SUCCESS) |
| 2029 | query->callback(query, | 2287 | query->callback(query, mad->mad_hdr.status ? |
| 2030 | mad_recv_wc->recv_buf.mad->mad_hdr.status ? | 2288 | -EINVAL : 0, (struct ib_sa_mad *)mad); |
| 2031 | -EINVAL : 0, | ||
| 2032 | (struct ib_sa_mad *) mad_recv_wc->recv_buf.mad); | ||
| 2033 | else | 2289 | else |
| 2034 | query->callback(query, -EIO, NULL); | 2290 | query->callback(query, -EIO, NULL); |
| 2035 | } | 2291 | } |
| @@ -2181,8 +2437,9 @@ static int ib_sa_add_one(struct ib_device *device) · 145 unmodified lines | |||
| 2181 | 2437 | ||
| 2182 | sa_dev->port[i].agent = | 2438 | sa_dev->port[i].agent = |
| 2183 | ib_register_mad_agent(device, i + s, IB_QPT_GSI, | 2439 | ib_register_mad_agent(device, i + s, IB_QPT_GSI, |
| 2184 | NULL, 0, send_handler, | 2440 | NULL, IB_MGMT_RMPP_VERSION, |
| 2185 | recv_handler, sa_dev, 0); | 2441 | send_handler, recv_handler, |
| 2442 | sa_dev, 0); | ||
| 2186 | if (IS_ERR(sa_dev->port[i].agent)) { | 2443 | if (IS_ERR(sa_dev->port[i].agent)) { |
| 2187 | ret = PTR_ERR(sa_dev->port[i].agent); | 2444 | ret = PTR_ERR(sa_dev->port[i].agent); |
| 2188 | goto err; | 2445 | goto err; |
| @@ -282,6 +282,10 @@ static struct ucma_event *ucma_create_uevent(struct ucma_context *ctx, · 281 unmodified lines | |||
| 282 | } | 282 | } |
| 283 | uevent->resp.event = event->event; | 283 | uevent->resp.event = event->event; |
| 284 | uevent->resp.status = event->status; | 284 | uevent->resp.status = event->status; |
| 285 | |||
| 286 | if (event->event == RDMA_CM_EVENT_ADDRINFO_RESOLVED) | ||
| 287 | goto out; | ||
| 288 | |||
| 285 | if (ctx->cm_id->qp_type == IB_QPT_UD) | 289 | if (ctx->cm_id->qp_type == IB_QPT_UD) |
| 286 | ucma_copy_ud_event(ctx->cm_id->device, &uevent->resp.param.ud, | 290 | ucma_copy_ud_event(ctx->cm_id->device, &uevent->resp.param.ud, |
| 287 | &event->param.ud); | 291 | &event->param.ud); |
| @@ -289,6 +293,7 @@ static struct ucma_event *ucma_create_uevent(struct ucma_context *ctx, · 1 unmodified line | |||
| 289 | ucma_copy_conn_event(&uevent->resp.param.conn, | 293 | ucma_copy_conn_event(&uevent->resp.param.conn, |
| 290 | &event->param.conn); | 294 | &event->param.conn); |
| 291 | 295 | ||
| 296 | out: | ||
| 292 | uevent->resp.ece.vendor_id = event->ece.vendor_id; | 297 | uevent->resp.ece.vendor_id = event->ece.vendor_id; |
| 293 | uevent->resp.ece.attr_mod = event->ece.attr_mod; | 298 | uevent->resp.ece.attr_mod = event->ece.attr_mod; |
| 294 | return uevent; | 299 | return uevent; |
| @@ -728,6 +733,28 @@ static ssize_t ucma_resolve_addr(struct ucma_file *file, · 433 unmodified lines | |||
| 728 | return ret; | 733 | return ret; |
| 729 | } | 734 | } |
| 730 | 735 | ||
| 736 | static ssize_t ucma_resolve_ib_service(struct ucma_file *file, | ||
| 737 | const char __user *inbuf, int in_len, | ||
| 738 | int out_len) | ||
| 739 | { | ||
| 740 | struct rdma_ucm_resolve_ib_service cmd; | ||
| 741 | struct ucma_context *ctx; | ||
| 742 | int ret; | ||
| 743 | |||
| 744 | if (copy_from_user(&cmd, inbuf, sizeof(cmd))) | ||
| 745 | return -EFAULT; | ||
| 746 | |||
| 747 | ctx = ucma_get_ctx(file, cmd.id); | ||
| 748 | if (IS_ERR(ctx)) | ||
| 749 | return PTR_ERR(ctx); | ||
| 750 | |||
| 751 | mutex_lock(&ctx->mutex); | ||
| 752 | ret = rdma_resolve_ib_service(ctx->cm_id, &cmd.ibs); | ||
| 753 | mutex_unlock(&ctx->mutex); | ||
| 754 | ucma_put_ctx(ctx); | ||
| 755 | return ret; | ||
| 756 | } | ||
| 757 | |||
| 731 | static ssize_t ucma_resolve_route(struct ucma_file *file, | 758 | static ssize_t ucma_resolve_route(struct ucma_file *file, |
| 732 | const char __user *inbuf, | 759 | const char __user *inbuf, |
| 733 | int in_len, int out_len) | 760 | int in_len, int out_len) |
| @@ -994,6 +1021,43 @@ static ssize_t ucma_query_gid(struct ucma_context *ctx, · 260 unmodified lines | |||
| 994 | return ret; | 1021 | return ret; |
| 995 | } | 1022 | } |
| 996 | 1023 | ||
| 1024 | static ssize_t ucma_query_ib_service(struct ucma_context *ctx, | ||
| 1025 | void __user *response, int out_len) | ||
| 1026 | { | ||
| 1027 | struct rdma_ucm_query_ib_service_resp *resp; | ||
| 1028 | int n, ret = 0; | ||
| 1029 | |||
| 1030 | if (out_len < sizeof(struct rdma_ucm_query_ib_service_resp)) | ||
| 1031 | return -ENOSPC; | ||
| 1032 | |||
| 1033 | if (!ctx->cm_id->route.service_recs) | ||
| 1034 | return -ENODATA; | ||
| 1035 | |||
| 1036 | resp = kzalloc(out_len, GFP_KERNEL); | ||
| 1037 | if (!resp) | ||
| 1038 | return -ENOMEM; | ||
| 1039 | |||
| 1040 | resp->num_service_recs = ctx->cm_id->route.num_service_recs; | ||
| 1041 | |||
| 1042 | n = (out_len - sizeof(struct rdma_ucm_query_ib_service_resp)) / | ||
| 1043 | sizeof(struct ib_user_service_rec); | ||
| 1044 | |||
| 1045 | if (!n) | ||
| 1046 | goto out; | ||
| 1047 | |||
| 1048 | if (n > ctx->cm_id->route.num_service_recs) | ||
| 1049 | n = ctx->cm_id->route.num_service_recs; | ||
| 1050 | |||
| 1051 | memcpy(resp->recs, ctx->cm_id->route.service_recs, | ||
| 1052 | sizeof(*resp->recs) * n); | ||
| 1053 | if (copy_to_user(response, resp, struct_size(resp, recs, n))) | ||
| 1054 | ret = -EFAULT; | ||
| 1055 | |||
| 1056 | out: | ||
| 1057 | kfree(resp); | ||
| 1058 | return ret; | ||
| 1059 | } | ||
| 1060 | |||
| 997 | static ssize_t ucma_query(struct ucma_file *file, | 1061 | static ssize_t ucma_query(struct ucma_file *file, |
| 998 | const char __user *inbuf, | 1062 | const char __user *inbuf, |
| 999 | int in_len, int out_len) | 1063 | int in_len, int out_len) |
| @@ -1022,6 +1086,9 @@ static ssize_t ucma_query(struct ucma_file *file, · 22 unmodified lines | |||
| 1022 | case RDMA_USER_CM_QUERY_GID: | 1086 | case RDMA_USER_CM_QUERY_GID: |
| 1023 | ret = ucma_query_gid(ctx, response, out_len); | 1087 | ret = ucma_query_gid(ctx, response, out_len); |
| 1024 | break; | 1088 | break; |
| 1089 | case RDMA_USER_CM_QUERY_IB_SERVICE: | ||
| 1090 | ret = ucma_query_ib_service(ctx, response, out_len); | ||
| 1091 | break; | ||
| 1025 | default: | 1092 | default: |
| 1026 | ret = -ENOSYS; | 1093 | ret = -ENOSYS; |
| 1027 | break; | 1094 | break; |
| @@ -1678,6 +1745,55 @@ err_unlock: · 650 unmodified lines | |||
| 1678 | return ret; | 1745 | return ret; |
| 1679 | } | 1746 | } |
| 1680 | 1747 | ||
| 1748 | static ssize_t ucma_write_cm_event(struct ucma_file *file, | ||
| 1749 | const char __user *inbuf, int in_len, | ||
| 1750 | int out_len) | ||
| 1751 | { | ||
| 1752 | struct rdma_ucm_write_cm_event cmd; | ||
| 1753 | struct rdma_cm_event event = {}; | ||
| 1754 | struct ucma_event *uevent; | ||
| 1755 | struct ucma_context *ctx; | ||
| 1756 | int ret = 0; | ||
| 1757 | |||
| 1758 | if (copy_from_user(&cmd, inbuf, sizeof(cmd))) | ||
| 1759 | return -EFAULT; | ||
| 1760 | |||
| 1761 | if ((cmd.event != RDMA_CM_EVENT_USER) && | ||
| 1762 | (cmd.event != RDMA_CM_EVENT_INTERNAL)) | ||
| 1763 | return -EINVAL; | ||
| 1764 | |||
| 1765 | ctx = ucma_get_ctx(file, cmd.id); | ||
| 1766 | if (IS_ERR(ctx)) | ||
| 1767 | return PTR_ERR(ctx); | ||
| 1768 | |||
| 1769 | event.event = cmd.event; | ||
| 1770 | event.status = cmd.status; | ||
| 1771 | event.param.arg = cmd.param.arg; | ||
| 1772 | |||
| 1773 | uevent = kzalloc(sizeof(*uevent), GFP_KERNEL); | ||
| 1774 | if (!uevent) { | ||
| 1775 | ret = -ENOMEM; | ||
| 1776 | goto out; | ||
| 1777 | } | ||
| 1778 | |||
| 1779 | uevent->ctx = ctx; | ||
| 1780 | uevent->resp.uid = ctx->uid; | ||
| 1781 | uevent->resp.id = ctx->id; | ||
| 1782 | uevent->resp.event = event.event; | ||
| 1783 | uevent->resp.status = event.status; | ||
| 1784 | memcpy(uevent->resp.param.arg32, &event.param.arg, | ||
| 1785 | sizeof(event.param.arg)); | ||
| 1786 | |||
| 1787 | mutex_lock(&ctx->file->mut); | ||
| 1788 | list_add_tail(&uevent->list, &ctx->file->event_list); | ||
| 1789 | mutex_unlock(&ctx->file->mut); | ||
| 1790 | wake_up_interruptible(&ctx->file->poll_wait); | ||
| 1791 | |||
| 1792 | out: | ||
| 1793 | ucma_put_ctx(ctx); | ||
| 1794 | return ret; | ||
| 1795 | } | ||
| 1796 | |||
| 1681 | static ssize_t (*ucma_cmd_table[])(struct ucma_file *file, | 1797 | static ssize_t (*ucma_cmd_table[])(struct ucma_file *file, |
| 1682 | const char __user *inbuf, | 1798 | const char __user *inbuf, |
| 1683 | int in_len, int out_len) = { | 1799 | int in_len, int out_len) = { |
| @@ -1703,7 +1819,9 @@ static ssize_t (*ucma_cmd_table[])(struct ucma_file *file, · 19 unmodified lines | |||
| 1703 | [RDMA_USER_CM_CMD_QUERY] = ucma_query, | 1819 | [RDMA_USER_CM_CMD_QUERY] = ucma_query, |
| 1704 | [RDMA_USER_CM_CMD_BIND] = ucma_bind, | 1820 | [RDMA_USER_CM_CMD_BIND] = ucma_bind, |
| 1705 | [RDMA_USER_CM_CMD_RESOLVE_ADDR] = ucma_resolve_addr, | 1821 | [RDMA_USER_CM_CMD_RESOLVE_ADDR] = ucma_resolve_addr, |
| 1706 | [RDMA_USER_CM_CMD_JOIN_MCAST] = ucma_join_multicast | 1822 | [RDMA_USER_CM_CMD_JOIN_MCAST] = ucma_join_multicast, |
| 1823 | [RDMA_USER_CM_CMD_RESOLVE_IB_SERVICE] = ucma_resolve_ib_service, | ||
| 1824 | [RDMA_USER_CM_CMD_WRITE_CM_EVENT] = ucma_write_cm_event, | ||
| 1707 | }; | 1825 | }; |
| 1708 | 1826 | ||
| 1709 | static ssize_t ucma_write(struct file *filp, const char __user *buf, | 1827 | static ssize_t ucma_write(struct file *filp, const char __user *buf, |
| @@ -14,3 +14,4 @@ obj-$(CONFIG_INFINIBAND_HNS_HIP08) += hns/ · 13 unmodified lines | |||
| 14 | obj-$(CONFIG_INFINIBAND_QEDR) += qedr/ | 14 | obj-$(CONFIG_INFINIBAND_QEDR) += qedr/ |
| 15 | obj-$(CONFIG_INFINIBAND_BNXT_RE) += bnxt_re/ | 15 | obj-$(CONFIG_INFINIBAND_BNXT_RE) += bnxt_re/ |
| 16 | obj-$(CONFIG_INFINIBAND_ERDMA) += erdma/ | 16 | obj-$(CONFIG_INFINIBAND_ERDMA) += erdma/ |
| 17 | obj-$(CONFIG_INFINIBAND_IONIC) += ionic/ | ||
| @@ -172,9 +172,9 @@ struct bnxt_re_dev { · 171 unmodified lines | |||
| 172 | struct list_head list; | 172 | struct list_head list; |
| 173 | unsigned long flags; | 173 | unsigned long flags; |
| 174 | #define BNXT_RE_FLAG_NETDEV_REGISTERED 0 | 174 | #define BNXT_RE_FLAG_NETDEV_REGISTERED 0 |
| 175 | #define BNXT_RE_FLAG_STATS_CTX3_ALLOC 1 | ||
| 175 | #define BNXT_RE_FLAG_HAVE_L2_REF 3 | 176 | #define BNXT_RE_FLAG_HAVE_L2_REF 3 |
| 176 | #define BNXT_RE_FLAG_RCFW_CHANNEL_EN 4 | 177 | #define BNXT_RE_FLAG_RCFW_CHANNEL_EN 4 |
| 177 | #define BNXT_RE_FLAG_QOS_WORK_REG 5 | ||
| 178 | #define BNXT_RE_FLAG_RESOURCES_ALLOCATED 7 | 178 | #define BNXT_RE_FLAG_RESOURCES_ALLOCATED 7 |
| 179 | #define BNXT_RE_FLAG_RESOURCES_INITIALIZED 8 | 179 | #define BNXT_RE_FLAG_RESOURCES_INITIALIZED 8 |
| 180 | #define BNXT_RE_FLAG_ERR_DEVICE_DETACHED 17 | 180 | #define BNXT_RE_FLAG_ERR_DEVICE_DETACHED 17 |
| @@ -187,9 +187,6 @@ struct bnxt_re_dev { · 6 unmodified lines | |||
| 187 | 187 | ||
| 188 | int id; | 188 | int id; |
| 189 | 189 | ||
| 190 | struct delayed_work worker; | ||
| 191 | u8 cur_prio_map; | ||
| 192 | |||
| 193 | /* RCFW Channel */ | 190 | /* RCFW Channel */ |
| 194 | struct bnxt_qplib_rcfw rcfw; | 191 | struct bnxt_qplib_rcfw rcfw; |
| 195 | 192 | ||
| @@ -227,6 +224,13 @@ struct bnxt_re_dev { · 31 unmodified lines | |||
| 227 | struct workqueue_struct *dcb_wq; | 224 | struct workqueue_struct *dcb_wq; |
| 228 | struct dentry *cc_config; | 225 | struct dentry *cc_config; |
| 229 | struct bnxt_re_dbg_cc_config_params *cc_config_params; | 226 | struct bnxt_re_dbg_cc_config_params *cc_config_params; |
| 227 | #define BNXT_VPD_FLD_LEN 32 | ||
| 228 | char board_partno[BNXT_VPD_FLD_LEN]; | ||
| 229 | /* RoCE mirror */ | ||
| 230 | u16 mirror_vnic_id; | ||
| 231 | union ib_gid ugid; | ||
| 232 | u32 ugid_index; | ||
| 233 | u8 sniffer_flow_created : 1; | ||
| 230 | }; | 234 | }; |
| 231 | 235 | ||
| 232 | #define to_bnxt_re_dev(ptr, member) \ | 236 | #define to_bnxt_re_dev(ptr, member) \ |
| @@ -243,6 +247,10 @@ int bnxt_re_assign_pma_port_counters(struct bnxt_re_dev *rdev, struct ib_mad *ou · 10 unmodified lines | |||
| 243 | int bnxt_re_assign_pma_port_ext_counters(struct bnxt_re_dev *rdev, | 247 | int bnxt_re_assign_pma_port_ext_counters(struct bnxt_re_dev *rdev, |
| 244 | struct ib_mad *out_mad); | 248 | struct ib_mad *out_mad); |
| 245 | 249 | ||
| 250 | void bnxt_re_hwrm_free_vnic(struct bnxt_re_dev *rdev); | ||
| 251 | int bnxt_re_hwrm_alloc_vnic(struct bnxt_re_dev *rdev); | ||
| 252 | int bnxt_re_hwrm_cfg_vnic(struct bnxt_re_dev *rdev, u32 qp_id); | ||
| 253 | |||
| 246 | static inline struct device *rdev_to_dev(struct bnxt_re_dev *rdev) | 254 | static inline struct device *rdev_to_dev(struct bnxt_re_dev *rdev) |
| 247 | { | 255 | { |
| 248 | if (rdev) | 256 | if (rdev) |
| @@ -276,4 +284,7 @@ static inline int bnxt_re_read_context_allowed(struct bnxt_re_dev *rdev) · 27 unmodified lines | |||
| 276 | #define BNXT_RE_CONTEXT_TYPE_MRW_SIZE_P7 192 | 284 | #define BNXT_RE_CONTEXT_TYPE_MRW_SIZE_P7 192 |
| 277 | #define BNXT_RE_CONTEXT_TYPE_SRQ_SIZE_P7 192 | 285 | #define BNXT_RE_CONTEXT_TYPE_SRQ_SIZE_P7 192 |
| 278 | 286 | ||
| 287 | #define BNXT_RE_HWRM_CMD_TIMEOUT(rdev) \ | ||
| 288 | ((rdev)->chip_ctx->hwrm_cmd_max_timeout * 1000) | ||
| 289 | |||
| 279 | #endif | 290 | #endif |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | |||
| 8 | 8 | ||
| 9 | #include <linux/debugfs.h> | 9 | #include <linux/debugfs.h> |
| 10 | #include <linux/pci.h> | 10 | #include <linux/pci.h> |
| 11 | #include <linux/seq_file.h> | ||
| 11 | #include <rdma/ib_addr.h> | 12 | #include <rdma/ib_addr.h> |
| 12 | 13 | ||
| 13 | #include "bnxt_ulp.h" | 14 | #include "bnxt_ulp.h" |
| @@ -314,6 +315,40 @@ static const struct file_operations bnxt_re_cc_config_ops = { · 300 unmodified lines | |||
| 314 | .write = bnxt_re_cc_config_set, | 315 | .write = bnxt_re_cc_config_set, |
| 315 | }; | 316 | }; |
| 316 | 317 | ||
| 318 | static int info_show(struct seq_file *m, void *unused) | ||
| 319 | { | ||
| 320 | struct bnxt_re_dev *rdev = m->private; | ||
| 321 | struct bnxt_re_res_cntrs *res_s = &rdev->stats.res; | ||
| 322 | |||
| 323 | seq_puts(m, "Info:\n"); | ||
| 324 | seq_printf(m, "Device Name\t\t: %s\n", dev_name(&rdev->ibdev.dev)); | ||
| 325 | seq_printf(m, "PD Watermark\t\t: %llu\n", res_s->pd_watermark); | ||
| 326 | seq_printf(m, "AH Watermark\t\t: %llu\n", res_s->ah_watermark); | ||
| 327 | seq_printf(m, "QP Watermark\t\t: %llu\n", res_s->qp_watermark); | ||
| 328 | seq_printf(m, "RC QP Watermark\t\t: %llu\n", res_s->rc_qp_watermark); | ||
| 329 | seq_printf(m, "UD QP Watermark\t\t: %llu\n", res_s->ud_qp_watermark); | ||
| 330 | seq_printf(m, "SRQ Watermark\t\t: %llu\n", res_s->srq_watermark); | ||
| 331 | seq_printf(m, "CQ Watermark\t\t: %llu\n", res_s->cq_watermark); | ||
| 332 | seq_printf(m, "MR Watermark\t\t: %llu\n", res_s->mr_watermark); | ||
| 333 | seq_printf(m, "MW Watermark\t\t: %llu\n", res_s->mw_watermark); | ||
| 334 | seq_printf(m, "CQ Resize Count\t\t: %d\n", atomic_read(&res_s->resize_count)); | ||
| 335 | if (rdev->pacing.dbr_pacing) { | ||
| 336 | seq_printf(m, "DB Pacing Reschedule\t: %llu\n", rdev->stats.pacing.resched); | ||
| 337 | seq_printf(m, "DB Pacing Complete\t: %llu\n", rdev->stats.pacing.complete); | ||
| 338 | seq_printf(m, "DB Pacing Alerts\t: %llu\n", rdev->stats.pacing.alerts); | ||
| 339 | seq_printf(m, "DB FIFO Register\t: 0x%x\n", | ||
| 340 | readl(rdev->en_dev->bar0 + rdev->pacing.dbr_db_fifo_reg_off)); | ||
| 341 | } | ||
| 342 | |||
| 343 | return 0; | ||
| 344 | } | ||
| 345 | DEFINE_SHOW_ATTRIBUTE(info); | ||
| 346 | |||
| 347 | static void bnxt_re_debugfs_add_info(struct bnxt_re_dev *rdev) | ||
| 348 | { | ||
| 349 | debugfs_create_file("info", 0400, rdev->dbg_root, rdev, &info_fops); | ||
| 350 | } | ||
| 351 | |||
| 317 | void bnxt_re_debugfs_add_pdev(struct bnxt_re_dev *rdev) | 352 | void bnxt_re_debugfs_add_pdev(struct bnxt_re_dev *rdev) |
| 318 | { | 353 | { |
| 319 | struct pci_dev *pdev = rdev->en_dev->pdev; | 354 | struct pci_dev *pdev = rdev->en_dev->pdev; |
| @@ -325,6 +360,8 @@ void bnxt_re_debugfs_add_pdev(struct bnxt_re_dev *rdev) · 5 unmodified lines | |||
| 325 | rdev->qp_debugfs = debugfs_create_dir("QPs", rdev->dbg_root); | 360 | rdev->qp_debugfs = debugfs_create_dir("QPs", rdev->dbg_root); |
| 326 | rdev->cc_config = debugfs_create_dir("cc_config", rdev->dbg_root); | 361 | rdev->cc_config = debugfs_create_dir("cc_config", rdev->dbg_root); |
| 327 | 362 | ||
| 363 | bnxt_re_debugfs_add_info(rdev); | ||
| 364 | |||
| 328 | rdev->cc_config_params = kzalloc(sizeof(*cc_params), GFP_KERNEL); | 365 | rdev->cc_config_params = kzalloc(sizeof(*cc_params), GFP_KERNEL); |
| 329 | 366 | ||
| 330 | for (i = 0; i < BNXT_RE_CC_PARAM_GEN0; i++) { | 367 | for (i = 0; i < BNXT_RE_CC_PARAM_GEN0; i++) { |
| @@ -51,25 +51,6 @@ · 50 unmodified lines | |||
| 51 | #include "hw_counters.h" | 51 | #include "hw_counters.h" |
| 52 | 52 | ||
| 53 | static const struct rdma_stat_desc bnxt_re_stat_descs[] = { | 53 | static const struct rdma_stat_desc bnxt_re_stat_descs[] = { |
| 54 | [BNXT_RE_ACTIVE_PD].name = "active_pds", | ||
| 55 | [BNXT_RE_ACTIVE_AH].name = "active_ahs", | ||
| 56 | [BNXT_RE_ACTIVE_QP].name = "active_qps", | ||
| 57 | [BNXT_RE_ACTIVE_RC_QP].name = "active_rc_qps", | ||
| 58 | [BNXT_RE_ACTIVE_UD_QP].name = "active_ud_qps", | ||
| 59 | [BNXT_RE_ACTIVE_SRQ].name = "active_srqs", | ||
| 60 | [BNXT_RE_ACTIVE_CQ].name = "active_cqs", | ||
| 61 | [BNXT_RE_ACTIVE_MR].name = "active_mrs", | ||
| 62 | [BNXT_RE_ACTIVE_MW].name = "active_mws", | ||
| 63 | [BNXT_RE_WATERMARK_PD].name = "watermark_pds", | ||
| 64 | [BNXT_RE_WATERMARK_AH].name = "watermark_ahs", | ||
| 65 | [BNXT_RE_WATERMARK_QP].name = "watermark_qps", | ||
| 66 | [BNXT_RE_WATERMARK_RC_QP].name = "watermark_rc_qps", | ||
| 67 | [BNXT_RE_WATERMARK_UD_QP].name = "watermark_ud_qps", | ||
| 68 | [BNXT_RE_WATERMARK_SRQ].name = "watermark_srqs", | ||
| 69 | [BNXT_RE_WATERMARK_CQ].name = "watermark_cqs", | ||
| 70 | [BNXT_RE_WATERMARK_MR].name = "watermark_mrs", | ||
| 71 | [BNXT_RE_WATERMARK_MW].name = "watermark_mws", | ||
| 72 | [BNXT_RE_RESIZE_CQ_CNT].name = "resize_cq_cnt", | ||
| 73 | [BNXT_RE_RX_PKTS].name = "rx_pkts", | 54 | [BNXT_RE_RX_PKTS].name = "rx_pkts", |
| 74 | [BNXT_RE_RX_BYTES].name = "rx_bytes", | 55 | [BNXT_RE_RX_BYTES].name = "rx_bytes", |
| 75 | [BNXT_RE_TX_PKTS].name = "tx_pkts", | 56 | [BNXT_RE_TX_PKTS].name = "tx_pkts", |
| @@ -79,22 +60,22 @@ static const struct rdma_stat_desc bnxt_re_stat_descs[] = { · 3 unmodified lines | |||
| 79 | [BNXT_RE_TX_DISCARDS].name = "tx_roce_discards", | 60 | [BNXT_RE_TX_DISCARDS].name = "tx_roce_discards", |
| 80 | [BNXT_RE_RX_ERRORS].name = "rx_roce_errors", | 61 | [BNXT_RE_RX_ERRORS].name = "rx_roce_errors", |
| 81 | [BNXT_RE_RX_DISCARDS].name = "rx_roce_discards", | 62 | [BNXT_RE_RX_DISCARDS].name = "rx_roce_discards", |
| 82 | [BNXT_RE_TO_RETRANSMITS].name = "to_retransmits", | 63 | [BNXT_RE_TO_RETRANSMITS].name = "local_ack_timeout_err", |
| 83 | [BNXT_RE_SEQ_ERR_NAKS_RCVD].name = "seq_err_naks_rcvd", | 64 | [BNXT_RE_SEQ_ERR_NAKS_RCVD].name = "packet_seq_err", |
| 84 | [BNXT_RE_MAX_RETRY_EXCEEDED].name = "max_retry_exceeded", | 65 | [BNXT_RE_MAX_RETRY_EXCEEDED].name = "max_retry_exceeded", |
| 85 | [BNXT_RE_RNR_NAKS_RCVD].name = "rnr_naks_rcvd", | 66 | [BNXT_RE_RNR_NAKS_RCVD].name = "rnr_nak_retry_err", |
| 86 | [BNXT_RE_MISSING_RESP].name = "missing_resp", | 67 | [BNXT_RE_MISSING_RESP].name = "implied_nak_seq_err", |
| 87 | [BNXT_RE_UNRECOVERABLE_ERR].name = "unrecoverable_err", | 68 | [BNXT_RE_UNRECOVERABLE_ERR].name = "unrecoverable_err", |
| 88 | [BNXT_RE_BAD_RESP_ERR].name = "bad_resp_err", | 69 | [BNXT_RE_BAD_RESP_ERR].name = "bad_resp_err", |
| 89 | [BNXT_RE_LOCAL_QP_OP_ERR].name = "local_qp_op_err", | 70 | [BNXT_RE_LOCAL_QP_OP_ERR].name = "local_qp_op_err", |
| 90 | [BNXT_RE_LOCAL_PROTECTION_ERR].name = "local_protection_err", | 71 | [BNXT_RE_LOCAL_PROTECTION_ERR].name = "local_protection_err", |
| 91 | [BNXT_RE_MEM_MGMT_OP_ERR].name = "mem_mgmt_op_err", | 72 | [BNXT_RE_MEM_MGMT_OP_ERR].name = "mem_mgmt_op_err", |
| 92 | [BNXT_RE_REMOTE_INVALID_REQ_ERR].name = "remote_invalid_req_err", | 73 | [BNXT_RE_REMOTE_INVALID_REQ_ERR].name = "req_remote_invalid_request", |
| 93 | [BNXT_RE_REMOTE_ACCESS_ERR].name = "remote_access_err", | 74 | [BNXT_RE_REMOTE_ACCESS_ERR].name = "req_remote_access_errors", |
| 94 | [BNXT_RE_REMOTE_OP_ERR].name = "remote_op_err", | 75 | [BNXT_RE_REMOTE_OP_ERR].name = "remote_op_err", |
| 95 | [BNXT_RE_DUP_REQ].name = "dup_req", | 76 | [BNXT_RE_DUP_REQ].name = "duplicate_request", |
| 96 | [BNXT_RE_RES_EXCEED_MAX].name = "res_exceed_max", | 77 | [BNXT_RE_RES_EXCEED_MAX].name = "res_exceed_max", |
| 97 | [BNXT_RE_RES_LENGTH_MISMATCH].name = "res_length_mismatch", | 78 | [BNXT_RE_RES_LENGTH_MISMATCH].name = "resp_local_length_error", |
| 98 | [BNXT_RE_RES_EXCEEDS_WQE].name = "res_exceeds_wqe", | 79 | [BNXT_RE_RES_EXCEEDS_WQE].name = "res_exceeds_wqe", |
| 99 | [BNXT_RE_RES_OPCODE_ERR].name = "res_opcode_err", | 80 | [BNXT_RE_RES_OPCODE_ERR].name = "res_opcode_err", |
| 100 | [BNXT_RE_RES_RX_INVALID_RKEY].name = "res_rx_invalid_rkey", | 81 | [BNXT_RE_RES_RX_INVALID_RKEY].name = "res_rx_invalid_rkey", |
| @@ -118,7 +99,7 @@ static const struct rdma_stat_desc bnxt_re_stat_descs[] = { · 17 unmodified lines | |||
| 118 | [BNXT_RE_RES_SRQ_LOAD_ERR].name = "res_srq_load_err", | 99 | [BNXT_RE_RES_SRQ_LOAD_ERR].name = "res_srq_load_err", |
| 119 | [BNXT_RE_RES_TX_PCI_ERR].name = "res_tx_pci_err", | 100 | [BNXT_RE_RES_TX_PCI_ERR].name = "res_tx_pci_err", |
| 120 | [BNXT_RE_RES_RX_PCI_ERR].name = "res_rx_pci_err", | 101 | [BNXT_RE_RES_RX_PCI_ERR].name = "res_rx_pci_err", |
| 121 | [BNXT_RE_OUT_OF_SEQ_ERR].name = "oos_drop_count", | 102 | [BNXT_RE_OUT_OF_SEQ_ERR].name = "out_of_sequence", |
| 122 | [BNXT_RE_TX_ATOMIC_REQ].name = "tx_atomic_req", | 103 | [BNXT_RE_TX_ATOMIC_REQ].name = "tx_atomic_req", |
| 123 | [BNXT_RE_TX_READ_REQ].name = "tx_read_req", | 104 | [BNXT_RE_TX_READ_REQ].name = "tx_read_req", |
| 124 | [BNXT_RE_TX_READ_RES].name = "tx_read_resp", | 105 | [BNXT_RE_TX_READ_RES].name = "tx_read_resp", |
| @@ -126,23 +107,22 @@ static const struct rdma_stat_desc bnxt_re_stat_descs[] = { · 1 unmodified line | |||
| 126 | [BNXT_RE_TX_SEND_REQ].name = "tx_send_req", | 107 | [BNXT_RE_TX_SEND_REQ].name = "tx_send_req", |
| 127 | [BNXT_RE_TX_ROCE_PKTS].name = "tx_roce_only_pkts", | 108 | [BNXT_RE_TX_ROCE_PKTS].name = "tx_roce_only_pkts", |
| 128 | [BNXT_RE_TX_ROCE_BYTES].name = "tx_roce_only_bytes", | 109 | [BNXT_RE_TX_ROCE_BYTES].name = "tx_roce_only_bytes", |
| 129 | [BNXT_RE_RX_ATOMIC_REQ].name = "rx_atomic_req", | 110 | [BNXT_RE_RX_ATOMIC_REQ].name = "rx_atomic_requests", |
| 130 | [BNXT_RE_RX_READ_REQ].name = "rx_read_req", | 111 | [BNXT_RE_RX_READ_REQ].name = "rx_read_requests", |
| 131 | [BNXT_RE_RX_READ_RESP].name = "rx_read_resp", | 112 | [BNXT_RE_RX_READ_RESP].name = "rx_read_resp", |
| 132 | [BNXT_RE_RX_WRITE_REQ].name = "rx_write_req", | 113 | [BNXT_RE_RX_WRITE_REQ].name = "rx_write_requests", |
| 133 | [BNXT_RE_RX_SEND_REQ].name = "rx_send_req", | 114 | [BNXT_RE_RX_SEND_REQ].name = "rx_send_req", |
| 134 | [BNXT_RE_RX_ROCE_PKTS].name = "rx_roce_only_pkts", | 115 | [BNXT_RE_RX_ROCE_PKTS].name = "rx_roce_only_pkts", |
| 135 | [BNXT_RE_RX_ROCE_BYTES].name = "rx_roce_only_bytes", | 116 | [BNXT_RE_RX_ROCE_BYTES].name = "rx_roce_only_bytes", |
| 136 | [BNXT_RE_RX_ROCE_GOOD_PKTS].name = "rx_roce_good_pkts", | 117 | [BNXT_RE_RX_ROCE_GOOD_PKTS].name = "rx_roce_good_pkts", |
| 137 | [BNXT_RE_RX_ROCE_GOOD_BYTES].name = "rx_roce_good_bytes", | 118 | [BNXT_RE_RX_ROCE_GOOD_BYTES].name = "rx_roce_good_bytes", |
| 138 | [BNXT_RE_OOB].name = "rx_out_of_buffer", | 119 | [BNXT_RE_OOB].name = "out_of_buffer", |
| 139 | [BNXT_RE_TX_CNP].name = "tx_cnp_pkts", | 120 | [BNXT_RE_TX_CNP].name = "np_cnp_pkts", |
| 140 | [BNXT_RE_RX_CNP].name = "rx_cnp_pkts", | 121 | [BNXT_RE_RX_CNP].name = "rp_cnp_handled", |
| 141 | [BNXT_RE_RX_ECN].name = "rx_ecn_marked_pkts", | 122 | [BNXT_RE_RX_ECN].name = "np_ecn_marked_roce_packets", |
| 142 | [BNXT_RE_PACING_RESCHED].name = "pacing_reschedule", | 123 | [BNXT_RE_REQ_CQE_ERROR].name = "req_cqe_error", |
| 143 | [BNXT_RE_PACING_CMPL].name = "pacing_complete", | 124 | [BNXT_RE_RESP_CQE_ERROR].name = "resp_cqe_error", |
| 144 | [BNXT_RE_PACING_ALERT].name = "pacing_alerts", | 125 | [BNXT_RE_RESP_REMOTE_ACCESS_ERRS].name = "resp_remote_access_errors", |
| 145 | [BNXT_RE_DB_FIFO_REG].name = "db_fifo_register", | ||
| 146 | }; | 126 | }; |
| 147 | 127 | ||
| 148 | static void bnxt_re_copy_ext_stats(struct bnxt_re_dev *rdev, | 128 | static void bnxt_re_copy_ext_stats(struct bnxt_re_dev *rdev, |
| @@ -273,18 +253,20 @@ static void bnxt_re_copy_err_stats(struct bnxt_re_dev *rdev, · 124 unmodified lines | |||
| 273 | err_s->res_rx_pci_err; | 253 | err_s->res_rx_pci_err; |
| 274 | stats->value[BNXT_RE_OUT_OF_SEQ_ERR] = | 254 | stats->value[BNXT_RE_OUT_OF_SEQ_ERR] = |
| 275 | err_s->res_oos_drop_count; | 255 | err_s->res_oos_drop_count; |
| 276 | } | 256 | stats->value[BNXT_RE_REQ_CQE_ERROR] = |
| 277 | 257 | err_s->bad_resp_err + | |
| 278 | static void bnxt_re_copy_db_pacing_stats(struct bnxt_re_dev *rdev, | 258 | err_s->local_qp_op_err + |
| 279 | struct rdma_hw_stats *stats) | 259 | err_s->local_protection_err + |
| 280 | { | 260 | err_s->mem_mgmt_op_err + |
| 281 | struct bnxt_re_db_pacing_stats *pacing_s = &rdev->stats.pacing; | 261 | err_s->remote_invalid_req_err + |
| 282 | 262 | err_s->remote_access_err + | |
| 283 | stats->value[BNXT_RE_PACING_RESCHED] = pacing_s->resched; | 263 | err_s->remote_op_err; |
| 284 | stats->value[BNXT_RE_PACING_CMPL] = pacing_s->complete; | 264 | stats->value[BNXT_RE_RESP_CQE_ERROR] = |
| 285 | stats->value[BNXT_RE_PACING_ALERT] = pacing_s->alerts; | 265 | err_s->res_cmp_err + |
| 286 | stats->value[BNXT_RE_DB_FIFO_REG] = | 266 | err_s->res_cq_load_err; |
| 287 | readl(rdev->en_dev->bar0 + rdev->pacing.dbr_db_fifo_reg_off); | 267 | stats->value[BNXT_RE_RESP_REMOTE_ACCESS_ERRS] = |
| 268 | err_s->res_rx_no_perm + | ||
| 269 | err_s->res_tx_no_perm; | ||
| 288 | } | 270 | } |
| 289 | 271 | ||
| 290 | int bnxt_re_assign_pma_port_ext_counters(struct bnxt_re_dev *rdev, struct ib_mad *out_mad) | 272 | int bnxt_re_assign_pma_port_ext_counters(struct bnxt_re_dev *rdev, struct ib_mad *out_mad) |
| @@ -382,7 +364,6 @@ int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev, · 91 unmodified lines | |||
| 382 | u32 port, int index) | 364 | u32 port, int index) |
| 383 | { | 365 | { |
| 384 | struct bnxt_re_dev *rdev = to_bnxt_re_dev(ibdev, ibdev); | 366 | struct bnxt_re_dev *rdev = to_bnxt_re_dev(ibdev, ibdev); |
| 385 | struct bnxt_re_res_cntrs *res_s = &rdev->stats.res; | ||
| 386 | struct bnxt_qplib_roce_stats *err_s = NULL; | 367 | struct bnxt_qplib_roce_stats *err_s = NULL; |
| 387 | struct ctx_hw_stats *hw_stats = NULL; | 368 | struct ctx_hw_stats *hw_stats = NULL; |
| 388 | int rc = 0; | 369 | int rc = 0; |
| @@ -391,26 +372,6 @@ int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev, · 2 unmodified lines | |||
| 391 | if (!port || !stats) | 372 | if (!port || !stats) |
| 392 | return -EINVAL; | 373 | return -EINVAL; |
| 393 | 374 | ||
| 394 | stats->value[BNXT_RE_ACTIVE_QP] = atomic_read(&res_s->qp_count); | ||
| 395 | stats->value[BNXT_RE_ACTIVE_RC_QP] = atomic_read(&res_s->rc_qp_count); | ||
| 396 | stats->value[BNXT_RE_ACTIVE_UD_QP] = atomic_read(&res_s->ud_qp_count); | ||
| 397 | stats->value[BNXT_RE_ACTIVE_SRQ] = atomic_read(&res_s->srq_count); | ||
| 398 | stats->value[BNXT_RE_ACTIVE_CQ] = atomic_read(&res_s->cq_count); | ||
| 399 | stats->value[BNXT_RE_ACTIVE_MR] = atomic_read(&res_s->mr_count); | ||
| 400 | stats->value[BNXT_RE_ACTIVE_MW] = atomic_read(&res_s->mw_count); | ||
| 401 | stats->value[BNXT_RE_ACTIVE_PD] = atomic_read(&res_s->pd_count); | ||
| 402 | stats->value[BNXT_RE_ACTIVE_AH] = atomic_read(&res_s->ah_count); | ||
| 403 | stats->value[BNXT_RE_WATERMARK_QP] = res_s->qp_watermark; | ||
| 404 | stats->value[BNXT_RE_WATERMARK_RC_QP] = res_s->rc_qp_watermark; | ||
| 405 | stats->value[BNXT_RE_WATERMARK_UD_QP] = res_s->ud_qp_watermark; | ||
| 406 | stats->value[BNXT_RE_WATERMARK_SRQ] = res_s->srq_watermark; | ||
| 407 | stats->value[BNXT_RE_WATERMARK_CQ] = res_s->cq_watermark; | ||
| 408 | stats->value[BNXT_RE_WATERMARK_MR] = res_s->mr_watermark; | ||
| 409 | stats->value[BNXT_RE_WATERMARK_MW] = res_s->mw_watermark; | ||
| 410 | stats->value[BNXT_RE_WATERMARK_PD] = res_s->pd_watermark; | ||
| 411 | stats->value[BNXT_RE_WATERMARK_AH] = res_s->ah_watermark; | ||
| 412 | stats->value[BNXT_RE_RESIZE_CQ_CNT] = atomic_read(&res_s->resize_count); | ||
| 413 | |||
| 414 | if (hw_stats) { | 375 | if (hw_stats) { |
| 415 | stats->value[BNXT_RE_RECOVERABLE_ERRORS] = | 376 | stats->value[BNXT_RE_RECOVERABLE_ERRORS] = |
| 416 | le64_to_cpu(hw_stats->tx_bcast_pkts); | 377 | le64_to_cpu(hw_stats->tx_bcast_pkts); |
| @@ -449,8 +410,6 @@ int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev, · 32 unmodified lines | |||
| 449 | goto done; | 410 | goto done; |
| 450 | } | 411 | } |
| 451 | } | 412 | } |
| 452 | if (rdev->pacing.dbr_pacing && bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx)) | ||
| 453 | bnxt_re_copy_db_pacing_stats(rdev, stats); | ||
| 454 | } | 413 | } |
| 455 | 414 | ||
| 456 | done: | 415 | done: |
| @@ -41,25 +41,6 @@ · 40 unmodified lines | |||
| 41 | #define __BNXT_RE_HW_STATS_H__ | 41 | #define __BNXT_RE_HW_STATS_H__ |
| 42 | 42 | ||
| 43 | enum bnxt_re_hw_stats { | 43 | enum bnxt_re_hw_stats { |
| 44 | BNXT_RE_ACTIVE_PD, | ||
| 45 | BNXT_RE_ACTIVE_AH, | ||
| 46 | BNXT_RE_ACTIVE_QP, | ||
| 47 | BNXT_RE_ACTIVE_RC_QP, | ||
| 48 | BNXT_RE_ACTIVE_UD_QP, | ||
| 49 | BNXT_RE_ACTIVE_SRQ, | ||
| 50 | BNXT_RE_ACTIVE_CQ, | ||
| 51 | BNXT_RE_ACTIVE_MR, | ||
| 52 | BNXT_RE_ACTIVE_MW, | ||
| 53 | BNXT_RE_WATERMARK_PD, | ||
| 54 | BNXT_RE_WATERMARK_AH, | ||
| 55 | BNXT_RE_WATERMARK_QP, | ||
| 56 | BNXT_RE_WATERMARK_RC_QP, | ||
| 57 | BNXT_RE_WATERMARK_UD_QP, | ||
| 58 | BNXT_RE_WATERMARK_SRQ, | ||
| 59 | BNXT_RE_WATERMARK_CQ, | ||
| 60 | BNXT_RE_WATERMARK_MR, | ||
| 61 | BNXT_RE_WATERMARK_MW, | ||
| 62 | BNXT_RE_RESIZE_CQ_CNT, | ||
| 63 | BNXT_RE_RX_PKTS, | 44 | BNXT_RE_RX_PKTS, |
| 64 | BNXT_RE_RX_BYTES, | 45 | BNXT_RE_RX_BYTES, |
| 65 | BNXT_RE_TX_PKTS, | 46 | BNXT_RE_TX_PKTS, |
| @@ -129,10 +110,9 @@ enum bnxt_re_hw_stats { · 63 unmodified lines | |||
| 129 | BNXT_RE_TX_CNP, | 110 | BNXT_RE_TX_CNP, |
| 130 | BNXT_RE_RX_CNP, | 111 | BNXT_RE_RX_CNP, |
| 131 | BNXT_RE_RX_ECN, | 112 | BNXT_RE_RX_ECN, |
| 132 | BNXT_RE_PACING_RESCHED, | 113 | BNXT_RE_REQ_CQE_ERROR, |
| 133 | BNXT_RE_PACING_CMPL, | 114 | BNXT_RE_RESP_CQE_ERROR, |
| 134 | BNXT_RE_PACING_ALERT, | 115 | BNXT_RE_RESP_REMOTE_ACCESS_ERRS, |
| 135 | BNXT_RE_DB_FIFO_REG, | ||
| 136 | BNXT_RE_NUM_EXT_COUNTERS | 116 | BNXT_RE_NUM_EXT_COUNTERS |
| 137 | }; | 117 | }; |
| 138 | 118 | ||
| @@ -288,7 +288,9 @@ int bnxt_re_query_port(struct ib_device *ibdev, u32 port_num, · 287 unmodified lines | |||
| 288 | } | 288 | } |
| 289 | port_attr->max_mtu = IB_MTU_4096; | 289 | port_attr->max_mtu = IB_MTU_4096; |
| 290 | port_attr->active_mtu = iboe_get_mtu(rdev->netdev->mtu); | 290 | port_attr->active_mtu = iboe_get_mtu(rdev->netdev->mtu); |
| 291 | port_attr->gid_tbl_len = dev_attr->max_sgid; | 291 | /* One GID is reserved for RawEth QP. Report one less */ |
| 292 | port_attr->gid_tbl_len = (rdev->rcfw.roce_mirror ? (dev_attr->max_sgid - 1) : | ||
| 293 | dev_attr->max_sgid); | ||
| 292 | port_attr->port_cap_flags = IB_PORT_CM_SUP | IB_PORT_REINIT_SUP | | 294 | port_attr->port_cap_flags = IB_PORT_CM_SUP | IB_PORT_REINIT_SUP | |
| 293 | IB_PORT_DEVICE_MGMT_SUP | | 295 | IB_PORT_DEVICE_MGMT_SUP | |
| 294 | IB_PORT_VENDOR_CLASS_SUP; | 296 | IB_PORT_VENDOR_CLASS_SUP; |
| @@ -375,7 +377,7 @@ int bnxt_re_del_gid(const struct ib_gid_attr *attr, void **context) · 80 unmodified lines | |||
| 375 | if (!ctx) | 377 | if (!ctx) |
| 376 | return -EINVAL; | 378 | return -EINVAL; |
| 377 | 379 | ||
| 378 | if (sgid_tbl && sgid_tbl->active) { | 380 | if (sgid_tbl->active) { |
| 379 | if (ctx->idx >= sgid_tbl->max) | 381 | if (ctx->idx >= sgid_tbl->max) |
| 380 | return -EINVAL; | 382 | return -EINVAL; |
| 381 | gid_to_del = &sgid_tbl->tbl[ctx->idx].gid; | 383 | gid_to_del = &sgid_tbl->tbl[ctx->idx].gid; |
| @@ -429,7 +431,7 @@ int bnxt_re_add_gid(const struct ib_gid_attr *attr, void **context) · 47 unmodified lines | |||
| 429 | 431 | ||
| 430 | rc = bnxt_qplib_add_sgid(sgid_tbl, (struct bnxt_qplib_gid *)&attr->gid, | 432 | rc = bnxt_qplib_add_sgid(sgid_tbl, (struct bnxt_qplib_gid *)&attr->gid, |
| 431 | rdev->qplib_res.netdev->dev_addr, | 433 | rdev->qplib_res.netdev->dev_addr, |
| 432 | vlan_id, true, &tbl_idx); | 434 | vlan_id, true, &tbl_idx, false, 0); |
| 433 | if (rc == -EALREADY) { | 435 | if (rc == -EALREADY) { |
| 434 | ctx_tbl = sgid_tbl->ctx; | 436 | ctx_tbl = sgid_tbl->ctx; |
| 435 | ctx_tbl[tbl_idx]->refcnt++; | 437 | ctx_tbl[tbl_idx]->refcnt++; |
| @@ -955,6 +957,20 @@ fail: · 519 unmodified lines | |||
| 955 | return rc; | 957 | return rc; |
| 956 | } | 958 | } |
| 957 | 959 | ||
| 960 | static void bnxt_re_del_unique_gid(struct bnxt_re_dev *rdev) | ||
| 961 | { | ||
| 962 | int rc; | ||
| 963 | |||
| 964 | if (!rdev->rcfw.roce_mirror) | ||
| 965 | return; | ||
| 966 | |||
| 967 | rc = bnxt_qplib_del_sgid(&rdev->qplib_res.sgid_tbl, | ||
| 968 | (struct bnxt_qplib_gid *)&rdev->ugid, | ||
| 969 | 0xFFFF, true); | ||
| 970 | if (rc) | ||
| 971 | dev_err(rdev_to_dev(rdev), "Failed to delete unique GID, rc: %d\n", rc); | ||
| 972 | } | ||
| 973 | |||
| 958 | /* Queue Pairs */ | 974 | /* Queue Pairs */ |
| 959 | int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata) | 975 | int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata) |
| 960 | { | 976 | { |
| @@ -994,6 +1010,9 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata) · 33 unmodified lines | |||
| 994 | else if (qp->qplib_qp.type == CMDQ_CREATE_QP_TYPE_UD) | 1010 | else if (qp->qplib_qp.type == CMDQ_CREATE_QP_TYPE_UD) |
| 995 | atomic_dec(&rdev->stats.res.ud_qp_count); | 1011 | atomic_dec(&rdev->stats.res.ud_qp_count); |
| 996 | 1012 | ||
| 1013 | if (qp->qplib_qp.type == CMDQ_CREATE_QP_TYPE_RAW_ETHERTYPE) | ||
| 1014 | bnxt_re_del_unique_gid(rdev); | ||
| 1015 | |||
| 997 | ib_umem_release(qp->rumem); | 1016 | ib_umem_release(qp->rumem); |
| 998 | ib_umem_release(qp->sumem); | 1017 | ib_umem_release(qp->sumem); |
| 999 | 1018 | ||
| @@ -1018,6 +1037,8 @@ static u8 __from_ib_qp_type(enum ib_qp_type type) · 18 unmodified lines | |||
| 1018 | return CMDQ_CREATE_QP_TYPE_RC; | 1037 | return CMDQ_CREATE_QP_TYPE_RC; |
| 1019 | case IB_QPT_UD: | 1038 | case IB_QPT_UD: |
| 1020 | return CMDQ_CREATE_QP_TYPE_UD; | 1039 | return CMDQ_CREATE_QP_TYPE_UD; |
| 1040 | case IB_QPT_RAW_PACKET: | ||
| 1041 | return CMDQ_CREATE_QP_TYPE_RAW_ETHERTYPE; | ||
| 1021 | default: | 1042 | default: |
| 1022 | return IB_QPT_MAX; | 1043 | return IB_QPT_MAX; |
| 1023 | } | 1044 | } |
| @@ -1595,6 +1616,29 @@ static bool bnxt_re_test_qp_limits(struct bnxt_re_dev *rdev, · 571 unmodified lines | |||
| 1595 | return rc; | 1616 | return rc; |
| 1596 | } | 1617 | } |
| 1597 | 1618 | ||
| 1619 | static int bnxt_re_add_unique_gid(struct bnxt_re_dev *rdev) | ||
| 1620 | { | ||
| 1621 | struct bnxt_qplib_ctx *hctx = &rdev->qplib_ctx; | ||
| 1622 | struct bnxt_qplib_res *res = &rdev->qplib_res; | ||
| 1623 | int rc; | ||
| 1624 | |||
| 1625 | if (!rdev->rcfw.roce_mirror) | ||
| 1626 | return 0; | ||
| 1627 | |||
| 1628 | rdev->ugid.global.subnet_prefix = cpu_to_be64(0xfe8000000000abcdLL); | ||
| 1629 | addrconf_ifid_eui48(&rdev->ugid.raw[8], rdev->netdev); | ||
| 1630 | |||
| 1631 | rc = bnxt_qplib_add_sgid(&res->sgid_tbl, | ||
| 1632 | (struct bnxt_qplib_gid *)&rdev->ugid, | ||
| 1633 | rdev->qplib_res.netdev->dev_addr, | ||
| 1634 | 0xFFFF, true, &rdev->ugid_index, true, | ||
| 1635 | hctx->stats3.fw_id); | ||
| 1636 | if (rc) | ||
| 1637 | dev_err(rdev_to_dev(rdev), "Failed to add unique GID. rc = %d\n", rc); | ||
| 1638 | |||
| 1639 | return rc; | ||
| 1640 | } | ||
| 1641 | |||
| 1598 | int bnxt_re_create_qp(struct ib_qp *ib_qp, struct ib_qp_init_attr *qp_init_attr, | 1642 | int bnxt_re_create_qp(struct ib_qp *ib_qp, struct ib_qp_init_attr *qp_init_attr, |
| 1599 | struct ib_udata *udata) | 1643 | struct ib_udata *udata) |
| 1600 | { | 1644 | { |
| @@ -1656,6 +1700,17 @@ int bnxt_re_create_qp(struct ib_qp *ib_qp, struct ib_qp_init_attr *qp_init_attr, · 55 unmodified lines | |||
| 1656 | } | 1700 | } |
| 1657 | } | 1701 | } |
| 1658 | 1702 | ||
| 1703 | /* Support for RawEth QP is added to capture TCP pkt dump. | ||
| 1704 | * So unique SGID is used to avoid incorrect statistics on per | ||
| 1705 | * function stats_ctx | ||
| 1706 | */ | ||
| 1707 | if (qp->qplib_qp.type == CMDQ_CREATE_QP_TYPE_RAW_ETHERTYPE) { | ||
| 1708 | rc = bnxt_re_add_unique_gid(rdev); | ||
| 1709 | if (rc) | ||
| 1710 | goto qp_destroy; | ||
| 1711 | qp->qplib_qp.ugid_index = rdev->ugid_index; | ||
| 1712 | } | ||
| 1713 | |||
| 1659 | qp->ib_qp.qp_num = qp->qplib_qp.id; | 1714 | qp->ib_qp.qp_num = qp->qplib_qp.id; |
| 1660 | if (qp_init_attr->qp_type == IB_QPT_GSI) | 1715 | if (qp_init_attr->qp_type == IB_QPT_GSI) |
| 1661 | rdev->gsi_ctx.gsi_qp = qp; | 1716 | rdev->gsi_ctx.gsi_qp = qp; |
| @@ -2301,7 +2356,7 @@ int bnxt_re_query_qp(struct ib_qp *ib_qp, struct ib_qp_attr *qp_attr, · 639 unmodified lines | |||
| 2301 | qp_attr->pkey_index = qplib_qp->pkey_index; | 2356 | qp_attr->pkey_index = qplib_qp->pkey_index; |
| 2302 | qp_attr->qkey = qplib_qp->qkey; | 2357 | qp_attr->qkey = qplib_qp->qkey; |
| 2303 | qp_attr->ah_attr.type = RDMA_AH_ATTR_TYPE_ROCE; | 2358 | qp_attr->ah_attr.type = RDMA_AH_ATTR_TYPE_ROCE; |
| 2304 | rdma_ah_set_grh(&qp_attr->ah_attr, NULL, qplib_qp->ah.flow_label, | 2359 | rdma_ah_set_grh(&qp_attr->ah_attr, NULL, qplib_qp->udp_sport, |
| 2305 | qplib_qp->ah.host_sgid_index, | 2360 | qplib_qp->ah.host_sgid_index, |
| 2306 | qplib_qp->ah.hop_limit, | 2361 | qplib_qp->ah.hop_limit, |
| 2307 | qplib_qp->ah.traffic_class); | 2362 | qplib_qp->ah.traffic_class); |
| @@ -3248,9 +3303,9 @@ int bnxt_re_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata) · 940 unmodified lines | |||
| 3248 | IB_ACCESS_LOCAL_WRITE); | 3303 | IB_ACCESS_LOCAL_WRITE); |
| 3249 | if (IS_ERR(cq->resize_umem)) { | 3304 | if (IS_ERR(cq->resize_umem)) { |
| 3250 | rc = PTR_ERR(cq->resize_umem); | 3305 | rc = PTR_ERR(cq->resize_umem); |
| 3306 | ibdev_err(&rdev->ibdev, "%s: ib_umem_get failed! rc = %pe\n", | ||
| 3307 | __func__, cq->resize_umem); | ||
| 3251 | cq->resize_umem = NULL; | 3308 | cq->resize_umem = NULL; |
| 3252 | ibdev_err(&rdev->ibdev, "%s: ib_umem_get failed! rc = %d\n", | ||
| 3253 | __func__, rc); | ||
| 3254 | goto fail; | 3309 | goto fail; |
| 3255 | } | 3310 | } |
| 3256 | cq->resize_cqe = entries; | 3311 | cq->resize_cqe = entries; |
| @@ -4392,6 +4447,93 @@ void bnxt_re_dealloc_ucontext(struct ib_ucontext *ib_uctx) · 1135 unmodified lines | |||
| 4392 | } | 4447 | } |
| 4393 | } | 4448 | } |
| 4394 | 4449 | ||
| 4450 | static int bnxt_re_setup_vnic(struct bnxt_re_dev *rdev, struct bnxt_re_qp *qp) | ||
| 4451 | { | ||
| 4452 | int rc; | ||
| 4453 | |||
| 4454 | rc = bnxt_re_hwrm_alloc_vnic(rdev); | ||
| 4455 | if (rc) | ||
| 4456 | return rc; | ||
| 4457 | |||
| 4458 | rc = bnxt_re_hwrm_cfg_vnic(rdev, qp->qplib_qp.id); | ||
| 4459 | if (rc) | ||
| 4460 | goto out_free_vnic; | ||
| 4461 | |||
| 4462 | return 0; | ||
| 4463 | out_free_vnic: | ||
| 4464 | bnxt_re_hwrm_free_vnic(rdev); | ||
| 4465 | return rc; | ||
| 4466 | } | ||
| 4467 | |||
| 4468 | struct ib_flow *bnxt_re_create_flow(struct ib_qp *ib_qp, | ||
| 4469 | struct ib_flow_attr *attr, | ||
| 4470 | struct ib_udata *udata) | ||
| 4471 | { | ||
| 4472 | struct bnxt_re_qp *qp = container_of(ib_qp, struct bnxt_re_qp, ib_qp); | ||
| 4473 | struct bnxt_re_dev *rdev = qp->rdev; | ||
| 4474 | struct bnxt_re_flow *flow; | ||
| 4475 | int rc; | ||
| 4476 | |||
| 4477 | if (attr->type != IB_FLOW_ATTR_SNIFFER || | ||
| 4478 | !rdev->rcfw.roce_mirror) | ||
| 4479 | return ERR_PTR(-EOPNOTSUPP); | ||
| 4480 | |||
| 4481 | mutex_lock(&rdev->qp_lock); | ||
| 4482 | if (rdev->sniffer_flow_created) { | ||
| 4483 | ibdev_err(&rdev->ibdev, "RoCE Mirroring is already Configured\n"); | ||
| 4484 | mutex_unlock(&rdev->qp_lock); | ||
| 4485 | return ERR_PTR(-EBUSY); | ||
| 4486 | } | ||
| 4487 | |||
| 4488 | flow = kzalloc(sizeof(*flow), GFP_KERNEL); | ||
| 4489 | if (!flow) { | ||
| 4490 | mutex_unlock(&rdev->qp_lock); | ||
| 4491 | return ERR_PTR(-ENOMEM); | ||
| 4492 | } | ||
| 4493 | |||
| 4494 | flow->rdev = rdev; | ||
| 4495 | |||
| 4496 | rc = bnxt_re_setup_vnic(rdev, qp); | ||
| 4497 | if (rc) | ||
| 4498 | goto out_free_flow; | ||
| 4499 | |||
| 4500 | rc = bnxt_qplib_create_flow(&rdev->qplib_res); | ||
| 4501 | if (rc) | ||
| 4502 | goto out_free_vnic; | ||
| 4503 | |||
| 4504 | rdev->sniffer_flow_created = 1; | ||
| 4505 | mutex_unlock(&rdev->qp_lock); | ||
| 4506 | |||
| 4507 | return &flow->ib_flow; | ||
| 4508 | |||
| 4509 | out_free_vnic: | ||
| 4510 | bnxt_re_hwrm_free_vnic(rdev); | ||
| 4511 | out_free_flow: | ||
| 4512 | mutex_unlock(&rdev->qp_lock); | ||
| 4513 | kfree(flow); | ||
| 4514 | return ERR_PTR(rc); | ||
| 4515 | } | ||
| 4516 | |||
| 4517 | int bnxt_re_destroy_flow(struct ib_flow *flow_id) | ||
| 4518 | { | ||
| 4519 | struct bnxt_re_flow *flow = | ||
| 4520 | container_of(flow_id, struct bnxt_re_flow, ib_flow); | ||
| 4521 | struct bnxt_re_dev *rdev = flow->rdev; | ||
| 4522 | int rc; | ||
| 4523 | |||
| 4524 | mutex_lock(&rdev->qp_lock); | ||
| 4525 | rc = bnxt_qplib_destroy_flow(&rdev->qplib_res); | ||
| 4526 | if (rc) | ||
| 4527 | ibdev_dbg(&rdev->ibdev, "failed to destroy_flow rc = %d\n", rc); | ||
| 4528 | rdev->sniffer_flow_created = 0; | ||
| 4529 | |||
| 4530 | bnxt_re_hwrm_free_vnic(rdev); | ||
| 4531 | mutex_unlock(&rdev->qp_lock); | ||
| 4532 | kfree(flow); | ||
| 4533 | |||
| 4534 | return rc; | ||
| 4535 | } | ||
| 4536 | |||
| 4395 | static struct bnxt_re_cq *bnxt_re_search_for_cq(struct bnxt_re_dev *rdev, u32 cq_id) | 4537 | static struct bnxt_re_cq *bnxt_re_search_for_cq(struct bnxt_re_dev *rdev, u32 cq_id) |
| 4396 | { | 4538 | { |
| 4397 | struct bnxt_re_cq *cq = NULL, *tmp_cq; | 4539 | struct bnxt_re_cq *cq = NULL, *tmp_cq; |
| @@ -4604,7 +4746,7 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_ALLOC_PAGE)(struct uverbs_attr_bundle * · 206 unmodified lines | |||
| 4604 | return err; | 4746 | return err; |
| 4605 | 4747 | ||
| 4606 | err = uverbs_copy_to(attrs, BNXT_RE_ALLOC_PAGE_DPI, | 4748 | err = uverbs_copy_to(attrs, BNXT_RE_ALLOC_PAGE_DPI, |
| 4607 | &dpi, sizeof(length)); | 4749 | &dpi, sizeof(dpi)); |
| 4608 | if (err) | 4750 | if (err) |
| 4609 | return err; | 4751 | return err; |
| 4610 | 4752 | ||
| @@ -164,6 +164,11 @@ struct bnxt_re_user_mmap_entry { · 163 unmodified lines | |||
| 164 | u8 mmap_flag; | 164 | u8 mmap_flag; |
| 165 | }; | 165 | }; |
| 166 | 166 | ||
| 167 | struct bnxt_re_flow { | ||
| 168 | struct ib_flow ib_flow; | ||
| 169 | struct bnxt_re_dev *rdev; | ||
| 170 | }; | ||
| 171 | |||
| 167 | static inline u16 bnxt_re_get_swqe_size(int nsge) | 172 | static inline u16 bnxt_re_get_swqe_size(int nsge) |
| 168 | { | 173 | { |
| 169 | return sizeof(struct sq_send_hdr) + nsge * sizeof(struct sq_sge); | 174 | return sizeof(struct sq_send_hdr) + nsge * sizeof(struct sq_sge); |
| @@ -267,6 +272,11 @@ struct ib_mr *bnxt_re_reg_user_mr_dmabuf(struct ib_pd *ib_pd, u64 start, · 97 unmodified lines | |||
| 267 | struct uverbs_attr_bundle *attrs); | 272 | struct uverbs_attr_bundle *attrs); |
| 268 | int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata); | 273 | int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata); |
| 269 | void bnxt_re_dealloc_ucontext(struct ib_ucontext *context); | 274 | void bnxt_re_dealloc_ucontext(struct ib_ucontext *context); |
| 275 | struct ib_flow *bnxt_re_create_flow(struct ib_qp *ib_qp, | ||
| 276 | struct ib_flow_attr *attr, | ||
| 277 | struct ib_udata *udata); | ||
| 278 | int bnxt_re_destroy_flow(struct ib_flow *flow_id); | ||
| 279 | |||
| 270 | int bnxt_re_mmap(struct ib_ucontext *context, struct vm_area_struct *vma); | 280 | int bnxt_re_mmap(struct ib_ucontext *context, struct vm_area_struct *vma); |
| 271 | void bnxt_re_mmap_free(struct rdma_user_mmap_entry *rdma_entry); | 281 | void bnxt_re_mmap_free(struct rdma_user_mmap_entry *rdma_entry); |
| 272 | 282 | ||
| @@ -80,6 +80,7 @@ MODULE_LICENSE("Dual BSD/GPL"); · 79 unmodified lines | |||
| 80 | static DEFINE_MUTEX(bnxt_re_mutex); | 80 | static DEFINE_MUTEX(bnxt_re_mutex); |
| 81 | 81 | ||
| 82 | static int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev); | 82 | static int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev); |
| 83 | static int bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev); | ||
| 83 | 84 | ||
| 84 | static int bnxt_re_hwrm_qcfg(struct bnxt_re_dev *rdev, u32 *db_len, | 85 | static int bnxt_re_hwrm_qcfg(struct bnxt_re_dev *rdev, u32 *db_len, |
| 85 | u32 *offset); | 86 | u32 *offset); |
| @@ -188,6 +189,10 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev) · 102 unmodified lines | |||
| 188 | rdev->qplib_res.is_vf = BNXT_EN_VF(en_dev); | 189 | rdev->qplib_res.is_vf = BNXT_EN_VF(en_dev); |
| 189 | rdev->qplib_res.en_dev = en_dev; | 190 | rdev->qplib_res.en_dev = en_dev; |
| 190 | 191 | ||
| 192 | rc = bnxt_re_query_hwrm_intf_version(rdev); | ||
| 193 | if (rc) | ||
| 194 | goto free_dev_attr; | ||
| 195 | |||
| 191 | bnxt_re_set_drv_mode(rdev); | 196 | bnxt_re_set_drv_mode(rdev); |
| 192 | 197 | ||
| 193 | bnxt_re_set_db_offset(rdev); | 198 | bnxt_re_set_db_offset(rdev); |
| @@ -540,6 +545,72 @@ static void bnxt_re_fill_fw_msg(struct bnxt_fw_msg *fw_msg, void *msg, · 346 unmodified lines | |||
| 540 | fw_msg->timeout = timeout; | 545 | fw_msg->timeout = timeout; |
| 541 | } | 546 | } |
| 542 | 547 | ||
| 548 | void bnxt_re_hwrm_free_vnic(struct bnxt_re_dev *rdev) | ||
| 549 | { | ||
| 550 | struct bnxt_en_dev *en_dev = rdev->en_dev; | ||
| 551 | struct hwrm_vnic_free_input req = {}; | ||
| 552 | struct bnxt_fw_msg fw_msg = {}; | ||
| 553 | int rc; | ||
| 554 | |||
| 555 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_VNIC_FREE); | ||
| 556 | |||
| 557 | req.vnic_id = cpu_to_le32(rdev->mirror_vnic_id); | ||
| 558 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), NULL, | ||
| 559 | 0, BNXT_RE_HWRM_CMD_TIMEOUT(rdev)); | ||
| 560 | rc = bnxt_send_msg(en_dev, &fw_msg); | ||
| 561 | if (rc) | ||
| 562 | ibdev_dbg(&rdev->ibdev, | ||
| 563 | "Failed to free vnic, rc = %d\n", rc); | ||
| 564 | } | ||
| 565 | |||
| 566 | int bnxt_re_hwrm_alloc_vnic(struct bnxt_re_dev *rdev) | ||
| 567 | { | ||
| 568 | struct bnxt_en_dev *en_dev = rdev->en_dev; | ||
| 569 | struct hwrm_vnic_alloc_output resp = {}; | ||
| 570 | struct hwrm_vnic_alloc_input req = {}; | ||
| 571 | struct bnxt_fw_msg fw_msg = {}; | ||
| 572 | int rc; | ||
| 573 | |||
| 574 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_VNIC_ALLOC); | ||
| 575 | |||
| 576 | req.vnic_id = cpu_to_le16(rdev->mirror_vnic_id); | ||
| 577 | req.flags = cpu_to_le32(VNIC_ALLOC_REQ_FLAGS_VNIC_ID_VALID); | ||
| 578 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, | ||
| 579 | sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev)); | ||
| 580 | rc = bnxt_send_msg(en_dev, &fw_msg); | ||
| 581 | if (rc) | ||
| 582 | ibdev_dbg(&rdev->ibdev, | ||
| 583 | "Failed to alloc vnic, rc = %d\n", rc); | ||
| 584 | |||
| 585 | return rc; | ||
| 586 | } | ||
| 587 | |||
| 588 | int bnxt_re_hwrm_cfg_vnic(struct bnxt_re_dev *rdev, u32 qp_id) | ||
| 589 | { | ||
| 590 | struct bnxt_en_dev *en_dev = rdev->en_dev; | ||
| 591 | struct hwrm_vnic_cfg_input req = {}; | ||
| 592 | struct bnxt_fw_msg fw_msg = {}; | ||
| 593 | int rc; | ||
| 594 | |||
| 595 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_VNIC_CFG); | ||
| 596 | |||
| 597 | req.flags = cpu_to_le32(VNIC_CFG_REQ_FLAGS_ROCE_ONLY_VNIC_MODE); | ||
| 598 | req.enables = cpu_to_le32(VNIC_CFG_REQ_ENABLES_RAW_QP_ID | | ||
| 599 | VNIC_CFG_REQ_ENABLES_MRU); | ||
| 600 | req.vnic_id = cpu_to_le16(rdev->mirror_vnic_id); | ||
| 601 | req.raw_qp_id = cpu_to_le32(qp_id); | ||
| 602 | req.mru = cpu_to_le16(rdev->netdev->mtu + VLAN_ETH_HLEN); | ||
| 603 | |||
| 604 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), NULL, | ||
| 605 | 0, BNXT_RE_HWRM_CMD_TIMEOUT(rdev)); | ||
| 606 | rc = bnxt_send_msg(en_dev, &fw_msg); | ||
| 607 | if (rc) | ||
| 608 | ibdev_dbg(&rdev->ibdev, | ||
| 609 | "Failed to cfg vnic, rc = %d\n", rc); | ||
| 610 | |||
| 611 | return rc; | ||
| 612 | } | ||
| 613 | |||
| 543 | /* Query device config using common hwrm */ | 614 | /* Query device config using common hwrm */ |
| 544 | static int bnxt_re_hwrm_qcfg(struct bnxt_re_dev *rdev, u32 *db_len, | 615 | static int bnxt_re_hwrm_qcfg(struct bnxt_re_dev *rdev, u32 *db_len, |
| 545 | u32 *offset) | 616 | u32 *offset) |
| @@ -553,11 +624,12 @@ static int bnxt_re_hwrm_qcfg(struct bnxt_re_dev *rdev, u32 *db_len, · 7 unmodified lines | |||
| 553 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_FUNC_QCFG); | 624 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_FUNC_QCFG); |
| 554 | req.fid = cpu_to_le16(0xffff); | 625 | req.fid = cpu_to_le16(0xffff); |
| 555 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, | 626 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, |
| 556 | sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); | 627 | sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev)); |
| 557 | rc = bnxt_send_msg(en_dev, &fw_msg); | 628 | rc = bnxt_send_msg(en_dev, &fw_msg); |
| 558 | if (!rc) { | 629 | if (!rc) { |
| 559 | *db_len = PAGE_ALIGN(le16_to_cpu(resp.l2_doorbell_bar_size_kb) * 1024); | 630 | *db_len = PAGE_ALIGN(le16_to_cpu(resp.l2_doorbell_bar_size_kb) * 1024); |
| 560 | *offset = PAGE_ALIGN(le16_to_cpu(resp.legacy_l2_db_size_kb) * 1024); | 631 | *offset = PAGE_ALIGN(le16_to_cpu(resp.legacy_l2_db_size_kb) * 1024); |
| 632 | rdev->mirror_vnic_id = le16_to_cpu(resp.mirror_vnic_id); | ||
| 561 | } | 633 | } |
| 562 | return rc; | 634 | return rc; |
| 563 | } | 635 | } |
| @@ -577,7 +649,7 @@ int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev) · 13 unmodified lines | |||
| 577 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_FUNC_QCAPS); | 649 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_FUNC_QCAPS); |
| 578 | req.fid = cpu_to_le16(0xffff); | 650 | req.fid = cpu_to_le16(0xffff); |
| 579 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, | 651 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, |
| 580 | sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); | 652 | sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev)); |
| 581 | 653 | ||
| 582 | rc = bnxt_send_msg(en_dev, &fw_msg); | 654 | rc = bnxt_send_msg(en_dev, &fw_msg); |
| 583 | if (rc) | 655 | if (rc) |
| @@ -587,6 +659,8 @@ int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev) · 3 unmodified lines | |||
| 587 | flags_ext2 = le32_to_cpu(resp.flags_ext2); | 659 | flags_ext2 = le32_to_cpu(resp.flags_ext2); |
| 588 | cctx->modes.dbr_pacing = flags_ext2 & FUNC_QCAPS_RESP_FLAGS_EXT2_DBR_PACING_EXT_SUPPORTED || | 660 | cctx->modes.dbr_pacing = flags_ext2 & FUNC_QCAPS_RESP_FLAGS_EXT2_DBR_PACING_EXT_SUPPORTED || |
| 589 | flags_ext2 & FUNC_QCAPS_RESP_FLAGS_EXT2_DBR_PACING_V0_SUPPORTED; | 661 | flags_ext2 & FUNC_QCAPS_RESP_FLAGS_EXT2_DBR_PACING_V0_SUPPORTED; |
| 662 | cctx->modes.roce_mirror = !!(le32_to_cpu(resp.flags_ext3) & | ||
| 663 | FUNC_QCAPS_RESP_FLAGS_EXT3_MIRROR_ON_ROCE_SUPPORTED); | ||
| 590 | return 0; | 664 | return 0; |
| 591 | } | 665 | } |
| 592 | 666 | ||
| @@ -603,7 +677,7 @@ static int bnxt_re_hwrm_dbr_pacing_qcfg(struct bnxt_re_dev *rdev) · 10 unmodified lines | |||
| 603 | cctx = rdev->chip_ctx; | 677 | cctx = rdev->chip_ctx; |
| 604 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_FUNC_DBR_PACING_QCFG); | 678 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_FUNC_DBR_PACING_QCFG); |
| 605 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, | 679 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, |
| 606 | sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); | 680 | sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev)); |
| 607 | rc = bnxt_send_msg(en_dev, &fw_msg); | 681 | rc = bnxt_send_msg(en_dev, &fw_msg); |
| 608 | if (rc) | 682 | if (rc) |
| 609 | return rc; | 683 | return rc; |
| @@ -842,20 +916,12 @@ static void bnxt_re_deinitialize_dbr_pacing(struct bnxt_re_dev *rdev) · 232 unmodified lines | |||
| 842 | static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, | 916 | static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, |
| 843 | u16 fw_ring_id, int type) | 917 | u16 fw_ring_id, int type) |
| 844 | { | 918 | { |
| 845 | struct bnxt_en_dev *en_dev; | 919 | struct bnxt_en_dev *en_dev = rdev->en_dev; |
| 846 | struct hwrm_ring_free_input req = {}; | 920 | struct hwrm_ring_free_input req = {}; |
| 847 | struct hwrm_ring_free_output resp; | 921 | struct hwrm_ring_free_output resp; |
| 848 | struct bnxt_fw_msg fw_msg = {}; | 922 | struct bnxt_fw_msg fw_msg = {}; |
| 849 | int rc = -EINVAL; | 923 | int rc = -EINVAL; |
| 850 | 924 | ||
| 851 | if (!rdev) | ||
| 852 | return rc; | ||
| 853 | |||
| 854 | en_dev = rdev->en_dev; | ||
| 855 | |||
| 856 | if (!en_dev) | ||
| 857 | return rc; | ||
| 858 | |||
| 859 | if (test_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags)) | 925 | if (test_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags)) |
| 860 | return 0; | 926 | return 0; |
| 861 | 927 | ||
| @@ -863,7 +929,7 @@ static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, · 1 unmodified line | |||
| 863 | req.ring_type = type; | 929 | req.ring_type = type; |
| 864 | req.ring_id = cpu_to_le16(fw_ring_id); | 930 | req.ring_id = cpu_to_le16(fw_ring_id); |
| 865 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, | 931 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, |
| 866 | sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); | 932 | sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev)); |
| 867 | rc = bnxt_send_msg(en_dev, &fw_msg); | 933 | rc = bnxt_send_msg(en_dev, &fw_msg); |
| 868 | if (rc) | 934 | if (rc) |
| 869 | ibdev_err(&rdev->ibdev, "Failed to free HW ring:%d :%#x", | 935 | ibdev_err(&rdev->ibdev, "Failed to free HW ring:%d :%#x", |
| @@ -881,9 +947,6 @@ static int bnxt_re_net_ring_alloc(struct bnxt_re_dev *rdev, · 11 unmodified lines | |||
| 881 | struct bnxt_fw_msg fw_msg = {}; | 947 | struct bnxt_fw_msg fw_msg = {}; |
| 882 | int rc = -EINVAL; | 948 | int rc = -EINVAL; |
| 883 | 949 | ||
| 884 | if (!en_dev) | ||
| 885 | return rc; | ||
| 886 | |||
| 887 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_RING_ALLOC); | 950 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_RING_ALLOC); |
| 888 | req.enables = 0; | 951 | req.enables = 0; |
| 889 | req.page_tbl_addr = cpu_to_le64(ring_attr->dma_arr[0]); | 952 | req.page_tbl_addr = cpu_to_le64(ring_attr->dma_arr[0]); |
| @@ -899,7 +962,7 @@ static int bnxt_re_net_ring_alloc(struct bnxt_re_dev *rdev, · 9 unmodified lines | |||
| 899 | req.ring_type = ring_attr->type; | 962 | req.ring_type = ring_attr->type; |
| 900 | req.int_mode = ring_attr->mode; | 963 | req.int_mode = ring_attr->mode; |
| 901 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, | 964 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, |
| 902 | sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); | 965 | sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev)); |
| 903 | rc = bnxt_send_msg(en_dev, &fw_msg); | 966 | rc = bnxt_send_msg(en_dev, &fw_msg); |
| 904 | if (!rc) | 967 | if (!rc) |
| 905 | *fw_ring_id = le16_to_cpu(resp.ring_id); | 968 | *fw_ring_id = le16_to_cpu(resp.ring_id); |
| @@ -916,16 +979,13 @@ static int bnxt_re_net_stats_ctx_free(struct bnxt_re_dev *rdev, · 10 unmodified lines | |||
| 916 | struct bnxt_fw_msg fw_msg = {}; | 979 | struct bnxt_fw_msg fw_msg = {}; |
| 917 | int rc = -EINVAL; | 980 | int rc = -EINVAL; |
| 918 | 981 | ||
| 919 | if (!en_dev) | ||
| 920 | return rc; | ||
| 921 | |||
| 922 | if (test_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags)) | 982 | if (test_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags)) |
| 923 | return 0; | 983 | return 0; |
| 924 | 984 | ||
| 925 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_STAT_CTX_FREE); | 985 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_STAT_CTX_FREE); |
| 926 | req.stat_ctx_id = cpu_to_le32(fw_stats_ctx_id); | 986 | req.stat_ctx_id = cpu_to_le32(fw_stats_ctx_id); |
| 927 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, | 987 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, |
| 928 | sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); | 988 | sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev)); |
| 929 | rc = bnxt_send_msg(en_dev, &fw_msg); | 989 | rc = bnxt_send_msg(en_dev, &fw_msg); |
| 930 | if (rc) | 990 | if (rc) |
| 931 | ibdev_err(&rdev->ibdev, "Failed to free HW stats context %#x", | 991 | ibdev_err(&rdev->ibdev, "Failed to free HW stats context %#x", |
| @@ -935,8 +995,7 @@ static int bnxt_re_net_stats_ctx_free(struct bnxt_re_dev *rdev, · 3 unmodified lines | |||
| 935 | } | 995 | } |
| 936 | 996 | ||
| 937 | static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, | 997 | static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, |
| 938 | dma_addr_t dma_map, | 998 | struct bnxt_qplib_stats *stats) |
| 939 | u32 *fw_stats_ctx_id) | ||
| 940 | { | 999 | { |
| 941 | struct bnxt_qplib_chip_ctx *chip_ctx = rdev->chip_ctx; | 1000 | struct bnxt_qplib_chip_ctx *chip_ctx = rdev->chip_ctx; |
| 942 | struct hwrm_stat_ctx_alloc_output resp = {}; | 1001 | struct hwrm_stat_ctx_alloc_output resp = {}; |
| @@ -945,21 +1004,18 @@ static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, · 2 unmodified lines | |||
| 945 | struct bnxt_fw_msg fw_msg = {}; | 1004 | struct bnxt_fw_msg fw_msg = {}; |
| 946 | int rc = -EINVAL; | 1005 | int rc = -EINVAL; |
| 947 | 1006 | ||
| 948 | *fw_stats_ctx_id = INVALID_STATS_CTX_ID; | 1007 | stats->fw_id = INVALID_STATS_CTX_ID; |
| 949 | |||
| 950 | if (!en_dev) | ||
| 951 | return rc; | ||
| 952 | 1008 | ||
| 953 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_STAT_CTX_ALLOC); | 1009 | bnxt_re_init_hwrm_hdr((void *)&req, HWRM_STAT_CTX_ALLOC); |
| 954 | req.update_period_ms = cpu_to_le32(1000); | 1010 | req.update_period_ms = cpu_to_le32(1000); |
| 955 | req.stats_dma_addr = cpu_to_le64(dma_map); | 1011 | req.stats_dma_addr = cpu_to_le64(stats->dma_map); |
| 956 | req.stats_dma_length = cpu_to_le16(chip_ctx->hw_stats_size); | 1012 | req.stats_dma_length = cpu_to_le16(chip_ctx->hw_stats_size); |
| 957 | req.stat_ctx_flags = STAT_CTX_ALLOC_REQ_STAT_CTX_FLAGS_ROCE; | 1013 | req.stat_ctx_flags = STAT_CTX_ALLOC_REQ_STAT_CTX_FLAGS_ROCE; |
| 958 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, | 1014 | bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, |
| 959 | sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); | 1015 | sizeof(resp), BNXT_RE_HWRM_CMD_TIMEOUT(rdev)); |
| 960 | rc = bnxt_send_msg(en_dev, &fw_msg); | 1016 | rc = bnxt_send_msg(en_dev, &fw_msg); |
| 961 | if (!rc) | 1017 | if (!rc) |
| 962 | *fw_stats_ctx_id = le32_to_cpu(resp.stat_ctx_id); | 1018 | stats->fw_id = le32_to_cpu(resp.stat_ctx_id); |
| 963 | 1019 | ||
| 964 | return rc; | 1020 | return rc; |
| 965 | } | 1021 | } |
| @@ -975,7 +1031,7 @@ static ssize_t hw_rev_show(struct device *device, struct device_attribute *attr, · 9 unmodified lines | |||
| 975 | struct bnxt_re_dev *rdev = | 1031 | struct bnxt_re_dev *rdev = |
| 976 | rdma_device_to_drv_device(device, struct bnxt_re_dev, ibdev); | 1032 | rdma_device_to_drv_device(device, struct bnxt_re_dev, ibdev); |
| 977 | 1033 | ||
| 978 | return sysfs_emit(buf, "0x%x\n", rdev->en_dev->pdev->vendor); | 1034 | return sysfs_emit(buf, "0x%x\n", rdev->en_dev->pdev->revision); |
| 979 | } | 1035 | } |
| 980 | static DEVICE_ATTR_RO(hw_rev); | 1036 | static DEVICE_ATTR_RO(hw_rev); |
| 981 | 1037 | ||
| @@ -985,13 +1041,31 @@ static ssize_t hca_type_show(struct device *device, · 3 unmodified lines | |||
| 985 | struct bnxt_re_dev *rdev = | 1041 | struct bnxt_re_dev *rdev = |
| 986 | rdma_device_to_drv_device(device, struct bnxt_re_dev, ibdev); | 1042 | rdma_device_to_drv_device(device, struct bnxt_re_dev, ibdev); |
| 987 | 1043 | ||
| 988 | return sysfs_emit(buf, "%s\n", rdev->ibdev.node_desc); | 1044 | return sysfs_emit(buf, "0x%x\n", rdev->en_dev->pdev->device); |
| 989 | } | 1045 | } |
| 990 | static DEVICE_ATTR_RO(hca_type); | 1046 | static DEVICE_ATTR_RO(hca_type); |
| 991 | 1047 | ||
| 1048 | static ssize_t board_id_show(struct device *device, struct device_attribute *attr, | ||
| 1049 | char *buf) | ||
| 1050 | { | ||
| 1051 | struct bnxt_re_dev *rdev = rdma_device_to_drv_device(device, | ||
| 1052 | struct bnxt_re_dev, ibdev); | ||
| 1053 | char buffer[BNXT_VPD_FLD_LEN] = {}; | ||
| 1054 | |||
| 1055 | if (!rdev->is_virtfn) | ||
| 1056 | memcpy(buffer, rdev->board_partno, BNXT_VPD_FLD_LEN - 1); | ||
| 1057 | else | ||
| 1058 | scnprintf(buffer, BNXT_VPD_FLD_LEN, "0x%x-VF", | ||
| 1059 | rdev->en_dev->pdev->device); | ||
| 1060 | |||
| 1061 | return sysfs_emit(buf, "%s\n", buffer); | ||
| 1062 | } | ||
| 1063 | static DEVICE_ATTR_RO(board_id); | ||
| 1064 | |||
| 992 | static struct attribute *bnxt_re_attributes[] = { | 1065 | static struct attribute *bnxt_re_attributes[] = { |
| 993 | &dev_attr_hw_rev.attr, | 1066 | &dev_attr_hw_rev.attr, |
| 994 | &dev_attr_hca_type.attr, | 1067 | &dev_attr_hca_type.attr, |
| 1068 | &dev_attr_board_id.attr, | ||
| 995 | NULL | 1069 | NULL |
| 996 | }; | 1070 | }; |
| 997 | 1071 | ||
| @@ -1207,6 +1281,8 @@ static int bnxt_re_fill_res_srq_entry(struct sk_buff *msg, struct ib_srq *ib_srq · 209 unmodified lines | |||
| 1207 | goto err; | 1281 | goto err; |
| 1208 | if (rdma_nl_put_driver_u32_hex(msg, "max_sge", srq->qplib_srq.max_sge)) | 1282 | if (rdma_nl_put_driver_u32_hex(msg, "max_sge", srq->qplib_srq.max_sge)) |
| 1209 | goto err; | 1283 | goto err; |
| 1284 | if (rdma_nl_put_driver_u32_hex(msg, "srq_limit", srq->qplib_srq.threshold)) | ||
| 1285 | goto err; | ||
| 1210 | 1286 | ||
| 1211 | nla_nest_end(msg, table_attr); | 1287 | nla_nest_end(msg, table_attr); |
| 1212 | return 0; | 1288 | return 0; |
| @@ -1297,6 +1373,8 @@ static const struct ib_device_ops bnxt_re_dev_ops = { · 84 unmodified lines | |||
| 1297 | .reg_user_mr_dmabuf = bnxt_re_reg_user_mr_dmabuf, | 1373 | .reg_user_mr_dmabuf = bnxt_re_reg_user_mr_dmabuf, |
| 1298 | .req_notify_cq = bnxt_re_req_notify_cq, | 1374 | .req_notify_cq = bnxt_re_req_notify_cq, |
| 1299 | .resize_cq = bnxt_re_resize_cq, | 1375 | .resize_cq = bnxt_re_resize_cq, |
| 1376 | .create_flow = bnxt_re_create_flow, | ||
| 1377 | .destroy_flow = bnxt_re_destroy_flow, | ||
| 1300 | INIT_RDMA_OBJ_SIZE(ib_ah, bnxt_re_ah, ib_ah), | 1378 | INIT_RDMA_OBJ_SIZE(ib_ah, bnxt_re_ah, ib_ah), |
| 1301 | INIT_RDMA_OBJ_SIZE(ib_cq, bnxt_re_cq, ib_cq), | 1379 | INIT_RDMA_OBJ_SIZE(ib_cq, bnxt_re_cq, ib_cq), |
| 1302 | INIT_RDMA_OBJ_SIZE(ib_pd, bnxt_re_pd, ib_pd), | 1380 | INIT_RDMA_OBJ_SIZE(ib_pd, bnxt_re_pd, ib_pd), |
| @@ -1323,8 +1401,7 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev) · 20 unmodified lines | |||
| 1323 | 1401 | ||
| 1324 | /* ib device init */ | 1402 | /* ib device init */ |
| 1325 | ibdev->node_type = RDMA_NODE_IB_CA; | 1403 | ibdev->node_type = RDMA_NODE_IB_CA; |
| 1326 | strscpy(ibdev->node_desc, BNXT_RE_DESC " HCA", | 1404 | strscpy(ibdev->node_desc, BNXT_RE_DESC " HCA"); |
| 1327 | strlen(BNXT_RE_DESC) + 5); | ||
| 1328 | ibdev->phys_port_cnt = 1; | 1405 | ibdev->phys_port_cnt = 1; |
| 1329 | 1406 | ||
| 1330 | addrconf_addr_eui48((u8 *)&ibdev->node_guid, rdev->netdev->dev_addr); | 1407 | addrconf_addr_eui48((u8 *)&ibdev->node_guid, rdev->netdev->dev_addr); |
| @@ -1850,81 +1927,6 @@ static void bnxt_re_dev_stop(struct bnxt_re_dev *rdev) · 519 unmodified lines | |||
| 1850 | mutex_unlock(&rdev->qp_lock); | 1927 | mutex_unlock(&rdev->qp_lock); |
| 1851 | } | 1928 | } |
| 1852 | 1929 | ||
| 1853 | static int bnxt_re_update_gid(struct bnxt_re_dev *rdev) | ||
| 1854 | { | ||
| 1855 | struct bnxt_qplib_sgid_tbl *sgid_tbl = &rdev->qplib_res.sgid_tbl; | ||
| 1856 | struct bnxt_qplib_gid gid; | ||
| 1857 | u16 gid_idx, index; | ||
| 1858 | int rc = 0; | ||
| 1859 | |||
| 1860 | if (!ib_device_try_get(&rdev->ibdev)) | ||
| 1861 | return 0; | ||
| 1862 | |||
| 1863 | for (index = 0; index < sgid_tbl->active; index++) { | ||
| 1864 | gid_idx = sgid_tbl->hw_id[index]; | ||
| 1865 | |||
| 1866 | if (!memcmp(&sgid_tbl->tbl[index], &bnxt_qplib_gid_zero, | ||
| 1867 | sizeof(bnxt_qplib_gid_zero))) | ||
| 1868 | continue; | ||
| 1869 | /* need to modify the VLAN enable setting of non VLAN GID only | ||
| 1870 | * as setting is done for VLAN GID while adding GID | ||
| 1871 | */ | ||
| 1872 | if (sgid_tbl->vlan[index]) | ||
| 1873 | continue; | ||
| 1874 | |||
| 1875 | memcpy(&gid, &sgid_tbl->tbl[index], sizeof(gid)); | ||
| 1876 | |||
| 1877 | rc = bnxt_qplib_update_sgid(sgid_tbl, &gid, gid_idx, | ||
| 1878 | rdev->qplib_res.netdev->dev_addr); | ||
| 1879 | } | ||
| 1880 | |||
| 1881 | ib_device_put(&rdev->ibdev); | ||
| 1882 | return rc; | ||
| 1883 | } | ||
| 1884 | |||
| 1885 | static u32 bnxt_re_get_priority_mask(struct bnxt_re_dev *rdev) | ||
| 1886 | { | ||
| 1887 | u32 prio_map = 0, tmp_map = 0; | ||
| 1888 | struct net_device *netdev; | ||
| 1889 | struct dcb_app app = {}; | ||
| 1890 | |||
| 1891 | netdev = rdev->netdev; | ||
| 1892 | |||
| 1893 | app.selector = IEEE_8021QAZ_APP_SEL_ETHERTYPE; | ||
| 1894 | app.protocol = ETH_P_IBOE; | ||
| 1895 | tmp_map = dcb_ieee_getapp_mask(netdev, &app); | ||
| 1896 | prio_map = tmp_map; | ||
| 1897 | |||
| 1898 | app.selector = IEEE_8021QAZ_APP_SEL_DGRAM; | ||
| 1899 | app.protocol = ROCE_V2_UDP_DPORT; | ||
| 1900 | tmp_map = dcb_ieee_getapp_mask(netdev, &app); | ||
| 1901 | prio_map |= tmp_map; | ||
| 1902 | |||
| 1903 | return prio_map; | ||
| 1904 | } | ||
| 1905 | |||
| 1906 | static int bnxt_re_setup_qos(struct bnxt_re_dev *rdev) | ||
| 1907 | { | ||
| 1908 | u8 prio_map = 0; | ||
| 1909 | |||
| 1910 | /* Get priority for roce */ | ||
| 1911 | prio_map = bnxt_re_get_priority_mask(rdev); | ||
| 1912 | |||
| 1913 | if (prio_map == rdev->cur_prio_map) | ||
| 1914 | return 0; | ||
| 1915 | rdev->cur_prio_map = prio_map; | ||
| 1916 | /* Actual priorities are not programmed as they are already | ||
| 1917 | * done by L2 driver; just enable or disable priority vlan tagging | ||
| 1918 | */ | ||
| 1919 | if ((prio_map == 0 && rdev->qplib_res.prio) || | ||
| 1920 | (prio_map != 0 && !rdev->qplib_res.prio)) { | ||
| 1921 | rdev->qplib_res.prio = prio_map; | ||
| 1922 | bnxt_re_update_gid(rdev); | ||
| 1923 | } | ||
| 1924 | |||
| 1925 | return 0; | ||
| 1926 | } | ||
| 1927 | |||
| 1928 | static void bnxt_re_net_unregister_async_event(struct bnxt_re_dev *rdev) | 1930 | static void bnxt_re_net_unregister_async_event(struct bnxt_re_dev *rdev) |
| 1929 | { | 1931 | { |
| 1930 | if (rdev->is_virtfn) | 1932 | if (rdev->is_virtfn) |
| @@ -1945,7 +1947,31 @@ static void bnxt_re_net_register_async_event(struct bnxt_re_dev *rdev) · 14 unmodified lines | |||
| 1945 | ASYNC_EVENT_CMPL_EVENT_ID_DCB_CONFIG_CHANGE); | 1947 | ASYNC_EVENT_CMPL_EVENT_ID_DCB_CONFIG_CHANGE); |
| 1946 | } | 1948 | } |
| 1947 | 1949 | ||
| 1948 | static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) | 1950 | static void bnxt_re_read_vpd_info(struct bnxt_re_dev *rdev) |
| 1951 | { | ||
| 1952 | struct pci_dev *pdev = rdev->en_dev->pdev; | ||
| 1953 | unsigned int vpd_size, kw_len; | ||
| 1954 | int pos, size; | ||
| 1955 | u8 *vpd_data; | ||
| 1956 | |||
| 1957 | vpd_data = pci_vpd_alloc(pdev, &vpd_size); | ||
| 1958 | if (IS_ERR(vpd_data)) { | ||
| 1959 | pci_warn(pdev, "Unable to read VPD, err=%pe\n", vpd_data); | ||
| 1960 | return; | ||
| 1961 | } | ||
| 1962 | |||
| 1963 | pos = pci_vpd_find_ro_info_keyword(vpd_data, vpd_size, | ||
| 1964 | PCI_VPD_RO_KEYWORD_PARTNO, &kw_len); | ||
| 1965 | if (pos < 0) | ||
| 1966 | goto free; | ||
| 1967 | |||
| 1968 | size = min_t(int, kw_len, BNXT_VPD_FLD_LEN - 1); | ||
| 1969 | memcpy(rdev->board_partno, &vpd_data[pos], size); | ||
| 1970 | free: | ||
| 1971 | kfree(vpd_data); | ||
| 1972 | } | ||
| 1973 | |||
| 1974 | static int bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) | ||
| 1949 | { | 1975 | { |
| 1950 | struct bnxt_en_dev *en_dev = rdev->en_dev; | 1976 | struct bnxt_en_dev *en_dev = rdev->en_dev; |
| 1951 | struct hwrm_ver_get_output resp = {}; | 1977 | struct hwrm_ver_get_output resp = {}; |
| @@ -1964,7 +1990,7 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) · 12 unmodified lines | |||
| 1964 | if (rc) { | 1990 | if (rc) { |
| 1965 | ibdev_err(&rdev->ibdev, "Failed to query HW version, rc = 0x%x", | 1991 | ibdev_err(&rdev->ibdev, "Failed to query HW version, rc = 0x%x", |
| 1966 | rc); | 1992 | rc); |
| 1967 | return; | 1993 | return rc; |
| 1968 | } | 1994 | } |
| 1969 | 1995 | ||
| 1970 | cctx = rdev->chip_ctx; | 1996 | cctx = rdev->chip_ctx; |
| @@ -1978,6 +2004,8 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) · 7 unmodified lines | |||
| 1978 | 2004 | ||
| 1979 | if (!cctx->hwrm_cmd_max_timeout) | 2005 | if (!cctx->hwrm_cmd_max_timeout) |
| 1980 | cctx->hwrm_cmd_max_timeout = RCFW_FW_STALL_MAX_TIMEOUT; | 2006 | cctx->hwrm_cmd_max_timeout = RCFW_FW_STALL_MAX_TIMEOUT; |
| 2007 | |||
| 2008 | return 0; | ||
| 1981 | } | 2009 | } |
| 1982 | 2010 | ||
| 1983 | static int bnxt_re_ib_init(struct bnxt_re_dev *rdev) | 2011 | static int bnxt_re_ib_init(struct bnxt_re_dev *rdev) |
| @@ -2039,6 +2067,72 @@ static void bnxt_re_free_gid_ctx(struct bnxt_re_dev *rdev) · 55 unmodified lines | |||
| 2039 | } | 2067 | } |
| 2040 | } | 2068 | } |
| 2041 | 2069 | ||
| 2070 | static int bnxt_re_get_stats_ctx(struct bnxt_re_dev *rdev) | ||
| 2071 | { | ||
| 2072 | struct bnxt_qplib_ctx *hctx = &rdev->qplib_ctx; | ||
| 2073 | struct bnxt_qplib_res *res = &rdev->qplib_res; | ||
| 2074 | int rc; | ||
| 2075 | |||
| 2076 | rc = bnxt_qplib_alloc_stats_ctx(res->pdev, res->cctx, &hctx->stats); | ||
| 2077 | if (rc) | ||
| 2078 | return rc; | ||
| 2079 | |||
| 2080 | rc = bnxt_re_net_stats_ctx_alloc(rdev, &hctx->stats); | ||
| 2081 | if (rc) | ||
| 2082 | goto free_stat_mem; | ||
| 2083 | |||
| 2084 | return 0; | ||
| 2085 | free_stat_mem: | ||
| 2086 | bnxt_qplib_free_stats_ctx(res->pdev, &hctx->stats); | ||
| 2087 | |||
| 2088 | return rc; | ||
| 2089 | } | ||
| 2090 | |||
| 2091 | static int bnxt_re_get_stats3_ctx(struct bnxt_re_dev *rdev) | ||
| 2092 | { | ||
| 2093 | struct bnxt_qplib_ctx *hctx = &rdev->qplib_ctx; | ||
| 2094 | struct bnxt_qplib_res *res = &rdev->qplib_res; | ||
| 2095 | int rc; | ||
| 2096 | |||
| 2097 | if (!rdev->rcfw.roce_mirror) | ||
| 2098 | return 0; | ||
| 2099 | |||
| 2100 | rc = bnxt_qplib_alloc_stats_ctx(res->pdev, res->cctx, &hctx->stats3); | ||
| 2101 | if (rc) | ||
| 2102 | return rc; | ||
| 2103 | |||
| 2104 | rc = bnxt_re_net_stats_ctx_alloc(rdev, &hctx->stats3); | ||
| 2105 | if (rc) | ||
| 2106 | goto free_stat_mem; | ||
| 2107 | |||
| 2108 | return 0; | ||
| 2109 | free_stat_mem: | ||
| 2110 | bnxt_qplib_free_stats_ctx(res->pdev, &hctx->stats3); | ||
| 2111 | |||
| 2112 | return rc; | ||
| 2113 | } | ||
| 2114 | |||
| 2115 | static void bnxt_re_put_stats3_ctx(struct bnxt_re_dev *rdev) | ||
| 2116 | { | ||
| 2117 | struct bnxt_qplib_ctx *hctx = &rdev->qplib_ctx; | ||
| 2118 | struct bnxt_qplib_res *res = &rdev->qplib_res; | ||
| 2119 | |||
| 2120 | if (!rdev->rcfw.roce_mirror) | ||
| 2121 | return; | ||
| 2122 | |||
| 2123 | bnxt_re_net_stats_ctx_free(rdev, hctx->stats3.fw_id); | ||
| 2124 | bnxt_qplib_free_stats_ctx(res->pdev, &hctx->stats3); | ||
| 2125 | } | ||
| 2126 | |||
| 2127 | static void bnxt_re_put_stats_ctx(struct bnxt_re_dev *rdev) | ||
| 2128 | { | ||
| 2129 | struct bnxt_qplib_ctx *hctx = &rdev->qplib_ctx; | ||
| 2130 | struct bnxt_qplib_res *res = &rdev->qplib_res; | ||
| 2131 | |||
| 2132 | bnxt_re_net_stats_ctx_free(rdev, hctx->stats.fw_id); | ||
| 2133 | bnxt_qplib_free_stats_ctx(res->pdev, &hctx->stats); | ||
| 2134 | } | ||
| 2135 | |||
| 2042 | static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) | 2136 | static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) |
| 2043 | { | 2137 | { |
| 2044 | u8 type; | 2138 | u8 type; |
| @@ -2049,8 +2143,7 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · 4 unmodified lines | |||
| 2049 | bnxt_re_net_unregister_async_event(rdev); | 2143 | bnxt_re_net_unregister_async_event(rdev); |
| 2050 | bnxt_re_uninit_dcb_wq(rdev); | 2144 | bnxt_re_uninit_dcb_wq(rdev); |
| 2051 | 2145 | ||
| 2052 | if (test_and_clear_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags)) | 2146 | bnxt_re_put_stats3_ctx(rdev); |
| 2053 | cancel_delayed_work_sync(&rdev->worker); | ||
| 2054 | 2147 | ||
| 2055 | bnxt_re_free_gid_ctx(rdev); | 2148 | bnxt_re_free_gid_ctx(rdev); |
| 2056 | if (test_and_clear_bit(BNXT_RE_FLAG_RESOURCES_INITIALIZED, | 2149 | if (test_and_clear_bit(BNXT_RE_FLAG_RESOURCES_INITIALIZED, |
| @@ -2064,8 +2157,8 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · 7 unmodified lines | |||
| 2064 | if (rc) | 2157 | if (rc) |
| 2065 | ibdev_warn(&rdev->ibdev, | 2158 | ibdev_warn(&rdev->ibdev, |
| 2066 | "Failed to deinitialize RCFW: %#x", rc); | 2159 | "Failed to deinitialize RCFW: %#x", rc); |
| 2067 | bnxt_re_net_stats_ctx_free(rdev, rdev->qplib_ctx.stats.fw_id); | 2160 | bnxt_re_put_stats_ctx(rdev); |
| 2068 | bnxt_qplib_free_ctx(&rdev->qplib_res, &rdev->qplib_ctx); | 2161 | bnxt_qplib_free_hwctx(&rdev->qplib_res, &rdev->qplib_ctx); |
| 2069 | bnxt_qplib_disable_rcfw_channel(&rdev->rcfw); | 2162 | bnxt_qplib_disable_rcfw_channel(&rdev->rcfw); |
| 2070 | type = bnxt_qplib_get_ring_type(rdev->chip_ctx); | 2163 | type = bnxt_qplib_get_ring_type(rdev->chip_ctx); |
| 2071 | bnxt_re_net_ring_free(rdev, rdev->rcfw.creq.ring_id, type); | 2164 | bnxt_re_net_ring_free(rdev, rdev->rcfw.creq.ring_id, type); |
| @@ -2085,16 +2178,6 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) · 13 unmodified lines | |||
| 2085 | } | 2178 | } |
| 2086 | } | 2179 | } |
| 2087 | 2180 | ||
| 2088 | /* worker thread for polling periodic events. Now used for QoS programming*/ | ||
| 2089 | static void bnxt_re_worker(struct work_struct *work) | ||
| 2090 | { | ||
| 2091 | struct bnxt_re_dev *rdev = container_of(work, struct bnxt_re_dev, | ||
| 2092 | worker.work); | ||
| 2093 | |||
| 2094 | bnxt_re_setup_qos(rdev); | ||
| 2095 | schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); | ||
| 2096 | } | ||
| 2097 | |||
| 2098 | static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) | 2181 | static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) |
| 2099 | { | 2182 | { |
| 2100 | struct bnxt_re_ring_attr rattr = {}; | 2183 | struct bnxt_re_ring_attr rattr = {}; |
| @@ -2109,8 +2192,9 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) · 8 unmodified lines | |||
| 2109 | rc = bnxt_re_register_netdev(rdev); | 2192 | rc = bnxt_re_register_netdev(rdev); |
| 2110 | if (rc) { | 2193 | if (rc) { |
| 2111 | ibdev_err(&rdev->ibdev, | 2194 | ibdev_err(&rdev->ibdev, |
| 2112 | "Failed to register with netedev: %#x\n", rc); | 2195 | "Failed to register with Ethernet driver, rc %d\n", |
| 2113 | return -EINVAL; | 2196 | rc); |
| 2197 | return rc; | ||
| 2114 | } | 2198 | } |
| 2115 | } | 2199 | } |
| 2116 | set_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags); | 2200 | set_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags); |
| @@ -2148,8 +2232,6 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) · 31 unmodified lines | |||
| 2148 | /* Check whether VF or PF */ | 2232 | /* Check whether VF or PF */ |
| 2149 | bnxt_re_get_sriov_func_type(rdev); | 2233 | bnxt_re_get_sriov_func_type(rdev); |
| 2150 | 2234 | ||
| 2151 | bnxt_re_query_hwrm_intf_version(rdev); | ||
| 2152 | |||
| 2153 | /* Establish RCFW Communication Channel to initialize the context | 2235 | /* Establish RCFW Communication Channel to initialize the context |
| 2154 | * memory for the function and all child VFs | 2236 | * memory for the function and all child VFs |
| 2155 | */ | 2237 | */ |
| @@ -2199,18 +2281,20 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) · 43 unmodified lines | |||
| 2199 | if (rc) | 2281 | if (rc) |
| 2200 | goto disable_rcfw; | 2282 | goto disable_rcfw; |
| 2201 | 2283 | ||
| 2284 | bnxt_qplib_query_version(&rdev->rcfw); | ||
| 2202 | bnxt_re_set_resource_limits(rdev); | 2285 | bnxt_re_set_resource_limits(rdev); |
| 2203 | 2286 | ||
| 2204 | rc = bnxt_qplib_alloc_ctx(&rdev->qplib_res, &rdev->qplib_ctx, 0, | 2287 | if (!rdev->is_virtfn && |
| 2205 | bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx)); | 2288 | !bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx)) { |
| 2206 | if (rc) { | 2289 | rc = bnxt_qplib_alloc_hwctx(&rdev->qplib_res, &rdev->qplib_ctx); |
| 2207 | ibdev_err(&rdev->ibdev, | 2290 | if (rc) { |
| 2208 | "Failed to allocate QPLIB context: %#x\n", rc); | 2291 | ibdev_err(&rdev->ibdev, |
| 2209 | goto disable_rcfw; | 2292 | "Failed to allocate hw context: %#x\n", rc); |
| 2293 | goto disable_rcfw; | ||
| 2294 | } | ||
| 2210 | } | 2295 | } |
| 2211 | rc = bnxt_re_net_stats_ctx_alloc(rdev, | 2296 | |
| 2212 | rdev->qplib_ctx.stats.dma_map, | 2297 | rc = bnxt_re_get_stats_ctx(rdev); |
| 2213 | &rdev->qplib_ctx.stats.fw_id); | ||
| 2214 | if (rc) { | 2298 | if (rc) { |
| 2215 | ibdev_err(&rdev->ibdev, | 2299 | ibdev_err(&rdev->ibdev, |
| 2216 | "Failed to allocate stats context: %#x\n", rc); | 2300 | "Failed to allocate stats context: %#x\n", rc); |
| @@ -2249,15 +2333,6 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) · 32 unmodified lines | |||
| 2249 | if (rc) | 2333 | if (rc) |
| 2250 | ibdev_warn(&rdev->ibdev, "Failed to query CC defaults\n"); | 2334 | ibdev_warn(&rdev->ibdev, "Failed to query CC defaults\n"); |
| 2251 | 2335 | ||
| 2252 | rc = bnxt_re_setup_qos(rdev); | ||
| 2253 | if (rc) | ||
| 2254 | ibdev_info(&rdev->ibdev, | ||
| 2255 | "RoCE priority not yet configured\n"); | ||
| 2256 | |||
| 2257 | INIT_DELAYED_WORK(&rdev->worker, bnxt_re_worker); | ||
| 2258 | set_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags); | ||
| 2259 | schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); | ||
| 2260 | |||
| 2261 | if (!(rdev->qplib_res.en_dev->flags & BNXT_EN_FLAG_ROCE_VF_RES_MGMT)) | 2336 | if (!(rdev->qplib_res.en_dev->flags & BNXT_EN_FLAG_ROCE_VF_RES_MGMT)) |
| 2262 | bnxt_re_vf_res_config(rdev); | 2337 | bnxt_re_vf_res_config(rdev); |
| 2263 | } | 2338 | } |
| @@ -2270,11 +2345,18 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) · 6 unmodified lines | |||
| 2270 | bnxt_re_init_dcb_wq(rdev); | 2345 | bnxt_re_init_dcb_wq(rdev); |
| 2271 | bnxt_re_net_register_async_event(rdev); | 2346 | bnxt_re_net_register_async_event(rdev); |
| 2272 | 2347 | ||
| 2348 | if (!rdev->is_virtfn) | ||
| 2349 | bnxt_re_read_vpd_info(rdev); | ||
| 2350 | |||
| 2351 | rc = bnxt_re_get_stats3_ctx(rdev); | ||
| 2352 | if (rc) | ||
| 2353 | goto fail; | ||
| 2354 | |||
| 2273 | return 0; | 2355 | return 0; |
| 2274 | free_sctx: | 2356 | free_sctx: |
| 2275 | bnxt_re_net_stats_ctx_free(rdev, rdev->qplib_ctx.stats.fw_id); | 2357 | bnxt_re_net_stats_ctx_free(rdev, rdev->qplib_ctx.stats.fw_id); |
| 2276 | free_ctx: | 2358 | free_ctx: |
| 2277 | bnxt_qplib_free_ctx(&rdev->qplib_res, &rdev->qplib_ctx); | 2359 | bnxt_qplib_free_hwctx(&rdev->qplib_res, &rdev->qplib_ctx); |
| 2278 | disable_rcfw: | 2360 | disable_rcfw: |
| 2279 | bnxt_qplib_disable_rcfw_channel(&rdev->rcfw); | 2361 | bnxt_qplib_disable_rcfw_channel(&rdev->rcfw); |
| 2280 | free_ring: | 2362 | free_ring: |
| @@ -1307,6 +1307,7 @@ static bool is_optimized_state_transition(struct bnxt_qplib_qp *qp) · 1306 unmodified lines | |||
| 1307 | 1307 | ||
| 1308 | int bnxt_qplib_modify_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp) | 1308 | int bnxt_qplib_modify_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp) |
| 1309 | { | 1309 | { |
| 1310 | struct bnxt_qplib_sgid_tbl *sgid_tbl = &res->sgid_tbl; | ||
| 1310 | struct bnxt_qplib_rcfw *rcfw = res->rcfw; | 1311 | struct bnxt_qplib_rcfw *rcfw = res->rcfw; |
| 1311 | struct creq_modify_qp_resp resp = {}; | 1312 | struct creq_modify_qp_resp resp = {}; |
| 1312 | struct bnxt_qplib_cmdqmsg msg = {}; | 1313 | struct bnxt_qplib_cmdqmsg msg = {}; |
| @@ -1358,9 +1359,14 @@ int bnxt_qplib_modify_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp) · 45 unmodified lines | |||
| 1358 | if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_FLOW_LABEL) | 1359 | if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_FLOW_LABEL) |
| 1359 | req.flow_label = cpu_to_le32(qp->ah.flow_label); | 1360 | req.flow_label = cpu_to_le32(qp->ah.flow_label); |
| 1360 | 1361 | ||
| 1361 | if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_SGID_INDEX) | 1362 | if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_SGID_INDEX) { |
| 1362 | req.sgid_index = cpu_to_le16(res->sgid_tbl.hw_id | 1363 | if (qp->type == CMDQ_CREATE_QP_TYPE_RAW_ETHERTYPE) |
| 1363 | [qp->ah.sgid_index]); | 1364 | req.sgid_index = |
| 1365 | cpu_to_le16(sgid_tbl->hw_id[qp->ugid_index]); | ||
| 1366 | else | ||
| 1367 | req.sgid_index = | ||
| 1368 | cpu_to_le16(sgid_tbl->hw_id[qp->ah.sgid_index]); | ||
| 1369 | } | ||
| 1364 | 1370 | ||
| 1365 | if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_HOP_LIMIT) | 1371 | if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_HOP_LIMIT) |
| 1366 | req.hop_limit = qp->ah.hop_limit; | 1372 | req.hop_limit = qp->ah.hop_limit; |
| @@ -1464,6 +1470,7 @@ int bnxt_qplib_query_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp) · 97 unmodified lines | |||
| 1464 | qp->access = sb->access; | 1470 | qp->access = sb->access; |
| 1465 | qp->pkey_index = le16_to_cpu(sb->pkey); | 1471 | qp->pkey_index = le16_to_cpu(sb->pkey); |
| 1466 | qp->qkey = le32_to_cpu(sb->qkey); | 1472 | qp->qkey = le32_to_cpu(sb->qkey); |
| 1473 | qp->udp_sport = le16_to_cpu(sb->udp_src_port); | ||
| 1467 | 1474 | ||
| 1468 | temp32[0] = le32_to_cpu(sb->dgid[0]); | 1475 | temp32[0] = le32_to_cpu(sb->dgid[0]); |
| 1469 | temp32[1] = le32_to_cpu(sb->dgid[1]); | 1476 | temp32[1] = le32_to_cpu(sb->dgid[1]); |
| @@ -299,6 +299,7 @@ struct bnxt_qplib_qp { · 298 unmodified lines | |||
| 299 | u8 smac[6]; | 299 | u8 smac[6]; |
| 300 | u16 vlan_id; | 300 | u16 vlan_id; |
| 301 | u16 port_id; | 301 | u16 port_id; |
| 302 | u16 udp_sport; | ||
| 302 | u8 nw_type; | 303 | u8 nw_type; |
| 303 | struct bnxt_qplib_ah ah; | 304 | struct bnxt_qplib_ah ah; |
| 304 | 305 | ||
| @@ -344,6 +345,7 @@ struct bnxt_qplib_qp { · 39 unmodified lines | |||
| 344 | u32 msn_tbl_sz; | 345 | u32 msn_tbl_sz; |
| 345 | bool is_host_msn_tbl; | 346 | bool is_host_msn_tbl; |
| 346 | u8 tos_dscp; | 347 | u8 tos_dscp; |
| 348 | u32 ugid_index; | ||
| 347 | }; | 349 | }; |
| 348 | 350 | ||
| 349 | #define BNXT_RE_MAX_MSG_SIZE 0x80000000 | 351 | #define BNXT_RE_MAX_MSG_SIZE 0x80000000 |
| @@ -186,7 +186,7 @@ static int __wait_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) · 185 unmodified lines | |||
| 186 | * wait for command completion. Maximum holding interval is 8 second. | 186 | * wait for command completion. Maximum holding interval is 8 second. |
| 187 | * | 187 | * |
| 188 | * Returns: | 188 | * Returns: |
| 189 | * -ETIMEOUT if command is not completed in specific time interval. | 189 | * -ETIMEDOUT if command is not completed in specific time interval. |
| 190 | * 0 if command is completed by firmware. | 190 | * 0 if command is completed by firmware. |
| 191 | */ | 191 | */ |
| 192 | static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) | 192 | static int __block_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) |
| @@ -366,6 +366,7 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, · 173 unmodified lines | |||
| 366 | wmb(); | 366 | wmb(); |
| 367 | writel(cmdq_prod, cmdq->cmdq_mbox.prod); | 367 | writel(cmdq_prod, cmdq->cmdq_mbox.prod); |
| 368 | writel(RCFW_CMDQ_TRIG_VAL, cmdq->cmdq_mbox.db); | 368 | writel(RCFW_CMDQ_TRIG_VAL, cmdq->cmdq_mbox.db); |
| 369 | print_hex_dump_bytes("req: ", DUMP_PREFIX_OFFSET, msg->req, msg->req_sz); | ||
| 369 | spin_unlock_bh(&hwq->lock); | 370 | spin_unlock_bh(&hwq->lock); |
| 370 | /* Return the CREQ response pointer */ | 371 | /* Return the CREQ response pointer */ |
| 371 | return 0; | 372 | return 0; |
| @@ -381,7 +382,7 @@ static int __send_message(struct bnxt_qplib_rcfw *rcfw, · 9 unmodified lines | |||
| 381 | * This function can not be called from non-sleepable context. | 382 | * This function can not be called from non-sleepable context. |
| 382 | * | 383 | * |
| 383 | * Returns: | 384 | * Returns: |
| 384 | * -ETIMEOUT if command is not completed in specific time interval. | 385 | * -ETIMEDOUT if command is not completed in specific time interval. |
| 385 | * 0 if command is completed by firmware. | 386 | * 0 if command is completed by firmware. |
| 386 | */ | 387 | */ |
| 387 | static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) | 388 | static int __poll_for_resp(struct bnxt_qplib_rcfw *rcfw, u16 cookie) |
| @@ -631,6 +632,7 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, · 243 unmodified lines | |||
| 631 | int rc = 0; | 632 | int rc = 0; |
| 632 | 633 | ||
| 633 | pdev = rcfw->pdev; | 634 | pdev = rcfw->pdev; |
| 635 | print_hex_dump_bytes("event: ", DUMP_PREFIX_OFFSET, qp_event, sizeof(*qp_event)); | ||
| 634 | switch (qp_event->event) { | 636 | switch (qp_event->event) { |
| 635 | case CREQ_QP_EVENT_EVENT_QP_ERROR_NOTIFICATION: | 637 | case CREQ_QP_EVENT_EVENT_QP_ERROR_NOTIFICATION: |
| 636 | err_event = (struct creq_qp_error_notification *)qp_event; | 638 | err_event = (struct creq_qp_error_notification *)qp_event; |
| @@ -903,6 +905,10 @@ skip_ctx_setup: · 266 unmodified lines | |||
| 903 | flags |= CMDQ_INITIALIZE_FW_FLAGS_OPTIMIZE_MODIFY_QP_SUPPORTED; | 905 | flags |= CMDQ_INITIALIZE_FW_FLAGS_OPTIMIZE_MODIFY_QP_SUPPORTED; |
| 904 | if (rcfw->res->en_dev->flags & BNXT_EN_FLAG_ROCE_VF_RES_MGMT) | 906 | if (rcfw->res->en_dev->flags & BNXT_EN_FLAG_ROCE_VF_RES_MGMT) |
| 905 | flags |= CMDQ_INITIALIZE_FW_FLAGS_L2_VF_RESOURCE_MGMT; | 907 | flags |= CMDQ_INITIALIZE_FW_FLAGS_L2_VF_RESOURCE_MGMT; |
| 908 | if (bnxt_qplib_roce_mirror_supported(rcfw->res->cctx)) { | ||
| 909 | flags |= CMDQ_INITIALIZE_FW_FLAGS_MIRROR_ON_ROCE_SUPPORTED; | ||
| 910 | rcfw->roce_mirror = true; | ||
| 911 | } | ||
| 906 | req.flags |= cpu_to_le16(flags); | 912 | req.flags |= cpu_to_le16(flags); |
| 907 | req.stat_ctx_id = cpu_to_le32(ctx->stats.fw_id); | 913 | req.stat_ctx_id = cpu_to_le32(ctx->stats.fw_id); |
| 908 | bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), sizeof(resp), 0); | 914 | bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), sizeof(resp), 0); |
| @@ -236,6 +236,7 @@ struct bnxt_qplib_rcfw { · 235 unmodified lines | |||
| 236 | atomic_t timeout_send; | 236 | atomic_t timeout_send; |
| 237 | /* cached from chip cctx for quick reference in slow path */ | 237 | /* cached from chip cctx for quick reference in slow path */ |
| 238 | u16 max_timeout; | 238 | u16 max_timeout; |
| 239 | bool roce_mirror; | ||
| 239 | }; | 240 | }; |
| 240 | 241 | ||
| 241 | struct bnxt_qplib_cmdqmsg { | 242 | struct bnxt_qplib_cmdqmsg { |
| @@ -53,12 +53,6 @@ · 52 unmodified lines | |||
| 53 | #include "qplib_sp.h" | 53 | #include "qplib_sp.h" |
| 54 | #include "qplib_rcfw.h" | 54 | #include "qplib_rcfw.h" |
| 55 | 55 | ||
| 56 | static void bnxt_qplib_free_stats_ctx(struct pci_dev *pdev, | ||
| 57 | struct bnxt_qplib_stats *stats); | ||
| 58 | static int bnxt_qplib_alloc_stats_ctx(struct pci_dev *pdev, | ||
| 59 | struct bnxt_qplib_chip_ctx *cctx, | ||
| 60 | struct bnxt_qplib_stats *stats); | ||
| 61 | |||
| 62 | /* PBL */ | 56 | /* PBL */ |
| 63 | static void __free_pbl(struct bnxt_qplib_res *res, struct bnxt_qplib_pbl *pbl, | 57 | static void __free_pbl(struct bnxt_qplib_res *res, struct bnxt_qplib_pbl *pbl, |
| 64 | bool is_umem) | 58 | bool is_umem) |
| @@ -352,8 +346,8 @@ fail: · 287 unmodified lines | |||
| 352 | } | 346 | } |
| 353 | 347 | ||
| 354 | /* Context Tables */ | 348 | /* Context Tables */ |
| 355 | void bnxt_qplib_free_ctx(struct bnxt_qplib_res *res, | 349 | void bnxt_qplib_free_hwctx(struct bnxt_qplib_res *res, |
| 356 | struct bnxt_qplib_ctx *ctx) | 350 | struct bnxt_qplib_ctx *ctx) |
| 357 | { | 351 | { |
| 358 | int i; | 352 | int i; |
| 359 | 353 | ||
| @@ -367,7 +361,6 @@ void bnxt_qplib_free_ctx(struct bnxt_qplib_res *res, · 7 unmodified lines | |||
| 367 | /* restore original pde level before destroy */ | 361 | /* restore original pde level before destroy */ |
| 368 | ctx->tqm_ctx.pde.level = ctx->tqm_ctx.pde_level; | 362 | ctx->tqm_ctx.pde.level = ctx->tqm_ctx.pde_level; |
| 369 | bnxt_qplib_free_hwq(res, &ctx->tqm_ctx.pde); | 363 | bnxt_qplib_free_hwq(res, &ctx->tqm_ctx.pde); |
| 370 | bnxt_qplib_free_stats_ctx(res->pdev, &ctx->stats); | ||
| 371 | } | 364 | } |
| 372 | 365 | ||
| 373 | static int bnxt_qplib_alloc_tqm_rings(struct bnxt_qplib_res *res, | 366 | static int bnxt_qplib_alloc_tqm_rings(struct bnxt_qplib_res *res, |
| @@ -466,7 +459,7 @@ fail: · 92 unmodified lines | |||
| 466 | } | 459 | } |
| 467 | 460 | ||
| 468 | /* | 461 | /* |
| 469 | * Routine: bnxt_qplib_alloc_ctx | 462 | * Routine: bnxt_qplib_alloc_hwctx |
| 470 | * Description: | 463 | * Description: |
| 471 | * Context tables are memories which are used by the chip fw. | 464 | * Context tables are memories which are used by the chip fw. |
| 472 | * The 6 tables defined are: | 465 | * The 6 tables defined are: |
| @@ -486,17 +479,13 @@ fail: · 13 unmodified lines | |||
| 486 | * Returns: | 479 | * Returns: |
| 487 | * 0 if success, else -ERRORS | 480 | * 0 if success, else -ERRORS |
| 488 | */ | 481 | */ |
| 489 | int bnxt_qplib_alloc_ctx(struct bnxt_qplib_res *res, | 482 | int bnxt_qplib_alloc_hwctx(struct bnxt_qplib_res *res, |
| 490 | struct bnxt_qplib_ctx *ctx, | 483 | struct bnxt_qplib_ctx *ctx) |
| 491 | bool virt_fn, bool is_p5) | ||
| 492 | { | 484 | { |
| 493 | struct bnxt_qplib_hwq_attr hwq_attr = {}; | 485 | struct bnxt_qplib_hwq_attr hwq_attr = {}; |
| 494 | struct bnxt_qplib_sg_info sginfo = {}; | 486 | struct bnxt_qplib_sg_info sginfo = {}; |
| 495 | int rc; | 487 | int rc; |
| 496 | 488 | ||
| 497 | if (virt_fn || is_p5) | ||
| 498 | goto stats_alloc; | ||
| 499 | |||
| 500 | /* QPC Tables */ | 489 | /* QPC Tables */ |
| 501 | sginfo.pgsize = PAGE_SIZE; | 490 | sginfo.pgsize = PAGE_SIZE; |
| 502 | sginfo.pgshft = PAGE_SHIFT; | 491 | sginfo.pgshft = PAGE_SHIFT; |
| @@ -542,16 +531,11 @@ int bnxt_qplib_alloc_ctx(struct bnxt_qplib_res *res, · 39 unmodified lines | |||
| 542 | rc = bnxt_qplib_alloc_init_hwq(&ctx->tim_tbl, &hwq_attr); | 531 | rc = bnxt_qplib_alloc_init_hwq(&ctx->tim_tbl, &hwq_attr); |
| 543 | if (rc) | 532 | if (rc) |
| 544 | goto fail; | 533 | goto fail; |
| 545 | stats_alloc: | ||
| 546 | /* Stats */ | ||
| 547 | rc = bnxt_qplib_alloc_stats_ctx(res->pdev, res->cctx, &ctx->stats); | ||
| 548 | if (rc) | ||
| 549 | goto fail; | ||
| 550 | 534 | ||
| 551 | return 0; | 535 | return 0; |
| 552 | 536 | ||
| 553 | fail: | 537 | fail: |
| 554 | bnxt_qplib_free_ctx(res, ctx); | 538 | bnxt_qplib_free_hwctx(res, ctx); |
| 555 | return rc; | 539 | return rc; |
| 556 | } | 540 | } |
| 557 | 541 | ||
| @@ -832,8 +816,8 @@ static int bnxt_qplib_alloc_dpi_tbl(struct bnxt_qplib_res *res, · 274 unmodified lines | |||
| 832 | } | 816 | } |
| 833 | 817 | ||
| 834 | /* Stats */ | 818 | /* Stats */ |
| 835 | static void bnxt_qplib_free_stats_ctx(struct pci_dev *pdev, | 819 | void bnxt_qplib_free_stats_ctx(struct pci_dev *pdev, |
| 836 | struct bnxt_qplib_stats *stats) | 820 | struct bnxt_qplib_stats *stats) |
| 837 | { | 821 | { |
| 838 | if (stats->dma) { | 822 | if (stats->dma) { |
| 839 | dma_free_coherent(&pdev->dev, stats->size, | 823 | dma_free_coherent(&pdev->dev, stats->size, |
| @@ -843,9 +827,9 @@ static void bnxt_qplib_free_stats_ctx(struct pci_dev *pdev, · 3 unmodified lines | |||
| 843 | stats->fw_id = -1; | 827 | stats->fw_id = -1; |
| 844 | } | 828 | } |
| 845 | 829 | ||
| 846 | static int bnxt_qplib_alloc_stats_ctx(struct pci_dev *pdev, | 830 | int bnxt_qplib_alloc_stats_ctx(struct pci_dev *pdev, |
| 847 | struct bnxt_qplib_chip_ctx *cctx, | 831 | struct bnxt_qplib_chip_ctx *cctx, |
| 848 | struct bnxt_qplib_stats *stats) | 832 | struct bnxt_qplib_stats *stats) |
| 849 | { | 833 | { |
| 850 | memset(stats, 0, sizeof(*stats)); | 834 | memset(stats, 0, sizeof(*stats)); |
| 851 | stats->fw_id = -1; | 835 | stats->fw_id = -1; |
| @@ -65,6 +65,7 @@ struct bnxt_qplib_drv_modes { · 64 unmodified lines | |||
| 65 | bool db_push; | 65 | bool db_push; |
| 66 | bool dbr_pacing; | 66 | bool dbr_pacing; |
| 67 | u32 toggle_bits; | 67 | u32 toggle_bits; |
| 68 | u8 roce_mirror; | ||
| 68 | }; | 69 | }; |
| 69 | 70 | ||
| 70 | enum bnxt_re_toggle_modes { | 71 | enum bnxt_re_toggle_modes { |
| @@ -303,6 +304,7 @@ struct bnxt_qplib_ctx { · 232 unmodified lines | |||
| 303 | struct bnxt_qplib_hwq tim_tbl; | 304 | struct bnxt_qplib_hwq tim_tbl; |
| 304 | struct bnxt_qplib_tqm_ctx tqm_ctx; | 305 | struct bnxt_qplib_tqm_ctx tqm_ctx; |
| 305 | struct bnxt_qplib_stats stats; | 306 | struct bnxt_qplib_stats stats; |
| 307 | struct bnxt_qplib_stats stats3; | ||
| 306 | struct bnxt_qplib_vf_res vf_res; | 308 | struct bnxt_qplib_vf_res vf_res; |
| 307 | }; | 309 | }; |
| 308 | 310 | ||
| @@ -432,15 +434,19 @@ void bnxt_qplib_cleanup_res(struct bnxt_qplib_res *res); · 123 unmodified lines | |||
| 432 | int bnxt_qplib_init_res(struct bnxt_qplib_res *res); | 434 | int bnxt_qplib_init_res(struct bnxt_qplib_res *res); |
| 433 | void bnxt_qplib_free_res(struct bnxt_qplib_res *res); | 435 | void bnxt_qplib_free_res(struct bnxt_qplib_res *res); |
| 434 | int bnxt_qplib_alloc_res(struct bnxt_qplib_res *res, struct net_device *netdev); | 436 | int bnxt_qplib_alloc_res(struct bnxt_qplib_res *res, struct net_device *netdev); |
| 435 | void bnxt_qplib_free_ctx(struct bnxt_qplib_res *res, | 437 | void bnxt_qplib_free_hwctx(struct bnxt_qplib_res *res, |
| 436 | struct bnxt_qplib_ctx *ctx); | 438 | struct bnxt_qplib_ctx *ctx); |
| 437 | int bnxt_qplib_alloc_ctx(struct bnxt_qplib_res *res, | 439 | int bnxt_qplib_alloc_hwctx(struct bnxt_qplib_res *res, |
| 438 | struct bnxt_qplib_ctx *ctx, | 440 | struct bnxt_qplib_ctx *ctx); |
| 439 | bool virt_fn, bool is_p5); | ||
| 440 | int bnxt_qplib_map_db_bar(struct bnxt_qplib_res *res); | 441 | int bnxt_qplib_map_db_bar(struct bnxt_qplib_res *res); |
| 441 | void bnxt_qplib_unmap_db_bar(struct bnxt_qplib_res *res); | 442 | void bnxt_qplib_unmap_db_bar(struct bnxt_qplib_res *res); |
| 442 | 443 | ||
| 443 | int bnxt_qplib_determine_atomics(struct pci_dev *dev); | 444 | int bnxt_qplib_determine_atomics(struct pci_dev *dev); |
| 445 | int bnxt_qplib_alloc_stats_ctx(struct pci_dev *pdev, | ||
| 446 | struct bnxt_qplib_chip_ctx *cctx, | ||
| 447 | struct bnxt_qplib_stats *stats); | ||
| 448 | void bnxt_qplib_free_stats_ctx(struct pci_dev *pdev, | ||
| 449 | struct bnxt_qplib_stats *stats); | ||
| 444 | 450 | ||
| 445 | static inline void bnxt_qplib_hwq_incr_prod(struct bnxt_qplib_db_info *dbinfo, | 451 | static inline void bnxt_qplib_hwq_incr_prod(struct bnxt_qplib_db_info *dbinfo, |
| 446 | struct bnxt_qplib_hwq *hwq, u32 cnt) | 452 | struct bnxt_qplib_hwq *hwq, u32 cnt) |
| @@ -582,6 +588,11 @@ static inline u8 bnxt_qplib_dbr_pacing_en(struct bnxt_qplib_chip_ctx *cctx) · 135 unmodified lines | |||
| 582 | return cctx->modes.dbr_pacing; | 588 | return cctx->modes.dbr_pacing; |
| 583 | } | 589 | } |
| 584 | 590 | ||
| 591 | static inline u8 bnxt_qplib_roce_mirror_supported(struct bnxt_qplib_chip_ctx *cctx) | ||
| 592 | { | ||
| 593 | return cctx->modes.roce_mirror; | ||
| 594 | } | ||
| 595 | |||
| 585 | static inline bool _is_alloc_mr_unified(u16 dev_cap_flags) | 596 | static inline bool _is_alloc_mr_unified(u16 dev_cap_flags) |
| 586 | { | 597 | { |
| 587 | return dev_cap_flags & CREQ_QUERY_FUNC_RESP_SB_MR_REGISTER_ALLOC; | 598 | return dev_cap_flags & CREQ_QUERY_FUNC_RESP_SB_MR_REGISTER_ALLOC; |
| @@ -66,14 +66,15 @@ static bool bnxt_qplib_is_atomic_cap(struct bnxt_qplib_rcfw *rcfw) · 65 unmodified lines | |||
| 66 | return (pcie_ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ); | 66 | return (pcie_ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ); |
| 67 | } | 67 | } |
| 68 | 68 | ||
| 69 | static void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw, | 69 | void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw) |
| 70 | char *fw_ver) | ||
| 71 | { | 70 | { |
| 72 | struct creq_query_version_resp resp = {}; | 71 | struct creq_query_version_resp resp = {}; |
| 73 | struct bnxt_qplib_cmdqmsg msg = {}; | 72 | struct bnxt_qplib_cmdqmsg msg = {}; |
| 74 | struct cmdq_query_version req = {}; | 73 | struct cmdq_query_version req = {}; |
| 74 | struct bnxt_qplib_dev_attr *attr; | ||
| 75 | int rc; | 75 | int rc; |
| 76 | 76 | ||
| 77 | attr = rcfw->res->dattr; | ||
| 77 | bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req, | 78 | bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req, |
| 78 | CMDQ_BASE_OPCODE_QUERY_VERSION, | 79 | CMDQ_BASE_OPCODE_QUERY_VERSION, |
| 79 | sizeof(req)); | 80 | sizeof(req)); |
| @@ -82,10 +83,10 @@ static void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw, · 2 unmodified lines | |||
| 82 | rc = bnxt_qplib_rcfw_send_message(rcfw, &msg); | 83 | rc = bnxt_qplib_rcfw_send_message(rcfw, &msg); |
| 83 | if (rc) | 84 | if (rc) |
| 84 | return; | 85 | return; |
| 85 | fw_ver[0] = resp.fw_maj; | 86 | attr->fw_ver[0] = resp.fw_maj; |
| 86 | fw_ver[1] = resp.fw_minor; | 87 | attr->fw_ver[1] = resp.fw_minor; |
| 87 | fw_ver[2] = resp.fw_bld; | 88 | attr->fw_ver[2] = resp.fw_bld; |
| 88 | fw_ver[3] = resp.fw_rsvd; | 89 | attr->fw_ver[3] = resp.fw_rsvd; |
| 89 | } | 90 | } |
| 90 | 91 | ||
| 91 | int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw) | 92 | int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw) |
| @@ -179,8 +180,6 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw) · 87 unmodified lines | |||
| 179 | if (_is_max_srq_ext_supported(attr->dev_cap_flags2)) | 180 | if (_is_max_srq_ext_supported(attr->dev_cap_flags2)) |
| 180 | attr->max_srq += le16_to_cpu(sb->max_srq_ext); | 181 | attr->max_srq += le16_to_cpu(sb->max_srq_ext); |
| 181 | 182 | ||
| 182 | bnxt_qplib_query_version(rcfw, attr->fw_ver); | ||
| 183 | |||
| 184 | for (i = 0; i < MAX_TQM_ALLOC_REQ / 4; i++) { | 183 | for (i = 0; i < MAX_TQM_ALLOC_REQ / 4; i++) { |
| 185 | temp = le32_to_cpu(sb->tqm_alloc_reqs[i]); | 184 | temp = le32_to_cpu(sb->tqm_alloc_reqs[i]); |
| 186 | tqm_alloc = (u8 *)&temp; | 185 | tqm_alloc = (u8 *)&temp; |
| @@ -309,7 +308,8 @@ int bnxt_qplib_del_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, · 122 unmodified lines | |||
| 309 | 308 | ||
| 310 | int bnxt_qplib_add_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, | 309 | int bnxt_qplib_add_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, |
| 311 | struct bnxt_qplib_gid *gid, const u8 *smac, | 310 | struct bnxt_qplib_gid *gid, const u8 *smac, |
| 312 | u16 vlan_id, bool update, u32 *index) | 311 | u16 vlan_id, bool update, u32 *index, |
| 312 | bool is_ugid, u32 stats_ctx_id) | ||
| 313 | { | 313 | { |
| 314 | struct bnxt_qplib_res *res = to_bnxt_qplib(sgid_tbl, | 314 | struct bnxt_qplib_res *res = to_bnxt_qplib(sgid_tbl, |
| 315 | struct bnxt_qplib_res, | 315 | struct bnxt_qplib_res, |
| @@ -374,6 +374,9 @@ int bnxt_qplib_add_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, · 58 unmodified lines | |||
| 374 | req.src_mac[1] = cpu_to_be16(((u16 *)smac)[1]); | 374 | req.src_mac[1] = cpu_to_be16(((u16 *)smac)[1]); |
| 375 | req.src_mac[2] = cpu_to_be16(((u16 *)smac)[2]); | 375 | req.src_mac[2] = cpu_to_be16(((u16 *)smac)[2]); |
| 376 | 376 | ||
| 377 | req.stats_ctx = cpu_to_le16(CMDQ_ADD_GID_STATS_CTX_STATS_CTX_VALID | | ||
| 378 | (u16)stats_ctx_id); | ||
| 379 | |||
| 377 | bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), | 380 | bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), |
| 378 | sizeof(resp), 0); | 381 | sizeof(resp), 0); |
| 379 | rc = bnxt_qplib_rcfw_send_message(rcfw, &msg); | 382 | rc = bnxt_qplib_rcfw_send_message(rcfw, &msg); |
| @@ -397,46 +400,6 @@ int bnxt_qplib_add_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, · 17 unmodified lines | |||
| 397 | return 0; | 400 | return 0; |
| 398 | } | 401 | } |
| 399 | 402 | ||
| 400 | int bnxt_qplib_update_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, | ||
| 401 | struct bnxt_qplib_gid *gid, u16 gid_idx, | ||
| 402 | const u8 *smac) | ||
| 403 | { | ||
| 404 | struct bnxt_qplib_res *res = to_bnxt_qplib(sgid_tbl, | ||
| 405 | struct bnxt_qplib_res, | ||
| 406 | sgid_tbl); | ||
| 407 | struct bnxt_qplib_rcfw *rcfw = res->rcfw; | ||
| 408 | struct creq_modify_gid_resp resp = {}; | ||
| 409 | struct bnxt_qplib_cmdqmsg msg = {}; | ||
| 410 | struct cmdq_modify_gid req = {}; | ||
| 411 | int rc; | ||
| 412 | |||
| 413 | bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req, | ||
| 414 | CMDQ_BASE_OPCODE_MODIFY_GID, | ||
| 415 | sizeof(req)); | ||
| 416 | |||
| 417 | req.gid[0] = cpu_to_be32(((u32 *)gid->data)[3]); | ||
| 418 | req.gid[1] = cpu_to_be32(((u32 *)gid->data)[2]); | ||
| 419 | req.gid[2] = cpu_to_be32(((u32 *)gid->data)[1]); | ||
| 420 | req.gid[3] = cpu_to_be32(((u32 *)gid->data)[0]); | ||
| 421 | if (res->prio) { | ||
| 422 | req.vlan |= cpu_to_le16 | ||
| 423 | (CMDQ_ADD_GID_VLAN_TPID_TPID_8100 | | ||
| 424 | CMDQ_ADD_GID_VLAN_VLAN_EN); | ||
| 425 | } | ||
| 426 | |||
| 427 | /* MAC in network format */ | ||
| 428 | req.src_mac[0] = cpu_to_be16(((u16 *)smac)[0]); | ||
| 429 | req.src_mac[1] = cpu_to_be16(((u16 *)smac)[1]); | ||
| 430 | req.src_mac[2] = cpu_to_be16(((u16 *)smac)[2]); | ||
| 431 | |||
| 432 | req.gid_index = cpu_to_le16(gid_idx); | ||
| 433 | |||
| 434 | bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), | ||
| 435 | sizeof(resp), 0); | ||
| 436 | rc = bnxt_qplib_rcfw_send_message(rcfw, &msg); | ||
| 437 | return rc; | ||
| 438 | } | ||
| 439 | |||
| 440 | /* AH */ | 403 | /* AH */ |
| 441 | int bnxt_qplib_create_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah, | 404 | int bnxt_qplib_create_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah, |
| 442 | bool block) | 405 | bool block) |
| @@ -1143,3 +1106,40 @@ out: · 700 unmodified lines | |||
| 1143 | dma_free_coherent(&rcfw->pdev->dev, sbuf.size, sbuf.sb, sbuf.dma_addr); | 1106 | dma_free_coherent(&rcfw->pdev->dev, sbuf.size, sbuf.sb, sbuf.dma_addr); |
| 1144 | return rc; | 1107 | return rc; |
| 1145 | } | 1108 | } |
| 1109 | |||
| 1110 | int bnxt_qplib_create_flow(struct bnxt_qplib_res *res) | ||
| 1111 | { | ||
| 1112 | struct creq_roce_mirror_cfg_resp resp = {}; | ||
| 1113 | struct bnxt_qplib_rcfw *rcfw = res->rcfw; | ||
| 1114 | struct cmdq_roce_mirror_cfg req = {}; | ||
| 1115 | struct bnxt_qplib_cmdqmsg msg = {}; | ||
| 1116 | |||
| 1117 | bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req, | ||
| 1118 | CMDQ_BASE_OPCODE_ROCE_MIRROR_CFG, | ||
| 1119 | sizeof(req)); | ||
| 1120 | |||
| 1121 | req.mirror_flags = (u8)CMDQ_ROCE_MIRROR_CFG_MIRROR_ENABLE; | ||
| 1122 | |||
| 1123 | bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), | ||
| 1124 | sizeof(resp), 0); | ||
| 1125 | return bnxt_qplib_rcfw_send_message(rcfw, &msg); | ||
| 1126 | } | ||
| 1127 | |||
| 1128 | int bnxt_qplib_destroy_flow(struct bnxt_qplib_res *res) | ||
| 1129 | { | ||
| 1130 | struct creq_roce_mirror_cfg_resp resp = {}; | ||
| 1131 | struct bnxt_qplib_rcfw *rcfw = res->rcfw; | ||
| 1132 | struct cmdq_roce_mirror_cfg req = {}; | ||
| 1133 | struct bnxt_qplib_cmdqmsg msg = {}; | ||
| 1134 | |||
| 1135 | bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req, | ||
| 1136 | CMDQ_BASE_OPCODE_ROCE_MIRROR_CFG, | ||
| 1137 | sizeof(req)); | ||
| 1138 | |||
| 1139 | req.mirror_flags &= ~((u8)CMDQ_ROCE_MIRROR_CFG_MIRROR_ENABLE); | ||
| 1140 | |||
| 1141 | bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), | ||
| 1142 | sizeof(resp), 0); | ||
| 1143 | |||
| 1144 | return bnxt_qplib_rcfw_send_message(rcfw, &msg); | ||
| 1145 | } | ||
| @@ -323,7 +323,8 @@ int bnxt_qplib_del_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, · 322 unmodified lines | |||
| 323 | struct bnxt_qplib_gid *gid, u16 vlan_id, bool update); | 323 | struct bnxt_qplib_gid *gid, u16 vlan_id, bool update); |
| 324 | int bnxt_qplib_add_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, | 324 | int bnxt_qplib_add_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, |
| 325 | struct bnxt_qplib_gid *gid, const u8 *mac, u16 vlan_id, | 325 | struct bnxt_qplib_gid *gid, const u8 *mac, u16 vlan_id, |
| 326 | bool update, u32 *index); | 326 | bool update, u32 *index, |
| 327 | bool is_ugid, u32 stats_ctx_id); | ||
| 327 | int bnxt_qplib_update_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, | 328 | int bnxt_qplib_update_sgid(struct bnxt_qplib_sgid_tbl *sgid_tbl, |
| 328 | struct bnxt_qplib_gid *gid, u16 gid_idx, | 329 | struct bnxt_qplib_gid *gid, u16 gid_idx, |
| 329 | const u8 *smac); | 330 | const u8 *smac); |
| @@ -358,6 +359,9 @@ int bnxt_qplib_read_context(struct bnxt_qplib_rcfw *rcfw, u8 type, u32 xid, · 28 unmodified lines | |||
| 358 | u32 resp_size, void *resp_va); | 359 | u32 resp_size, void *resp_va); |
| 359 | int bnxt_qplib_query_cc_param(struct bnxt_qplib_res *res, | 360 | int bnxt_qplib_query_cc_param(struct bnxt_qplib_res *res, |
| 360 | struct bnxt_qplib_cc_param *cc_param); | 361 | struct bnxt_qplib_cc_param *cc_param); |
| 362 | void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw); | ||
| 363 | int bnxt_qplib_create_flow(struct bnxt_qplib_res *res); | ||
| 364 | int bnxt_qplib_destroy_flow(struct bnxt_qplib_res *res); | ||
| 361 | 365 | ||
| 362 | #define BNXT_VAR_MAX_WQE 4352 | 366 | #define BNXT_VAR_MAX_WQE 4352 |
| 363 | #define BNXT_VAR_MAX_SLOT_ALIGN 256 | 367 | #define BNXT_VAR_MAX_SLOT_ALIGN 256 |
| @@ -144,7 +144,8 @@ struct cmdq_base { · 143 unmodified lines | |||
| 144 | #define CMDQ_BASE_OPCODE_MODIFY_CQ 0x90UL | 144 | #define CMDQ_BASE_OPCODE_MODIFY_CQ 0x90UL |
| 145 | #define CMDQ_BASE_OPCODE_QUERY_QP_EXTEND 0x91UL | 145 | #define CMDQ_BASE_OPCODE_QUERY_QP_EXTEND 0x91UL |
| 146 | #define CMDQ_BASE_OPCODE_QUERY_ROCE_STATS_EXT 0x92UL | 146 | #define CMDQ_BASE_OPCODE_QUERY_ROCE_STATS_EXT 0x92UL |
| 147 | #define CMDQ_BASE_OPCODE_LAST CMDQ_BASE_OPCODE_QUERY_ROCE_STATS_EXT | 147 | #define CMDQ_BASE_OPCODE_ROCE_MIRROR_CFG 0x99UL |
| 148 | #define CMDQ_BASE_OPCODE_LAST CMDQ_BASE_OPCODE_ROCE_MIRROR_CFG | ||
| 148 | u8 cmd_size; | 149 | u8 cmd_size; |
| 149 | __le16 flags; | 150 | __le16 flags; |
| 150 | __le16 cookie; | 151 | __le16 cookie; |
| @@ -218,6 +219,7 @@ struct cmdq_initialize_fw { · 67 unmodified lines | |||
| 218 | #define CMDQ_INITIALIZE_FW_FLAGS_HW_REQUESTER_RETX_SUPPORTED 0x2UL | 219 | #define CMDQ_INITIALIZE_FW_FLAGS_HW_REQUESTER_RETX_SUPPORTED 0x2UL |
| 219 | #define CMDQ_INITIALIZE_FW_FLAGS_OPTIMIZE_MODIFY_QP_SUPPORTED 0x8UL | 220 | #define CMDQ_INITIALIZE_FW_FLAGS_OPTIMIZE_MODIFY_QP_SUPPORTED 0x8UL |
| 220 | #define CMDQ_INITIALIZE_FW_FLAGS_L2_VF_RESOURCE_MGMT 0x10UL | 221 | #define CMDQ_INITIALIZE_FW_FLAGS_L2_VF_RESOURCE_MGMT 0x10UL |
| 222 | #define CMDQ_INITIALIZE_FW_FLAGS_MIRROR_ON_ROCE_SUPPORTED 0x80UL | ||
| 221 | __le16 cookie; | 223 | __le16 cookie; |
| 222 | u8 resp_size; | 224 | u8 resp_size; |
| 223 | u8 reserved8; | 225 | u8 reserved8; |
| @@ -788,7 +790,8 @@ struct creq_query_qp_resp_sb { · 564 unmodified lines | |||
| 788 | #define CREQ_QUERY_QP_RESP_SB_ACCESS_REMOTE_ATOMIC 0x8UL | 790 | #define CREQ_QUERY_QP_RESP_SB_ACCESS_REMOTE_ATOMIC 0x8UL |
| 789 | __le16 pkey; | 791 | __le16 pkey; |
| 790 | __le32 qkey; | 792 | __le32 qkey; |
| 791 | __le32 reserved32; | 793 | __le16 udp_src_port; |
| 794 | __le16 reserved16; | ||
| 792 | __le32 dgid[4]; | 795 | __le32 dgid[4]; |
| 793 | __le32 flow_label; | 796 | __le32 flow_label; |
| 794 | __le16 sgid_index; | 797 | __le16 sgid_index; |
| @@ -2108,6 +2111,43 @@ struct creq_query_roce_stats_ext_resp_sb { · 1313 unmodified lines | |||
| 2108 | __le64 dup_req; | 2111 | __le64 dup_req; |
| 2109 | }; | 2112 | }; |
| 2110 | 2113 | ||
| 2114 | /* cmdq_roce_mirror_cfg (size:192b/24B) */ | ||
| 2115 | struct cmdq_roce_mirror_cfg { | ||
| 2116 | u8 opcode; | ||
| 2117 | #define CMDQ_ROCE_MIRROR_CFG_OPCODE_ROCE_MIRROR_CFG 0x99UL | ||
| 2118 | #define CMDQ_ROCE_MIRROR_CFG_OPCODE_LAST \ | ||
| 2119 | CMDQ_ROCE_MIRROR_CFG_OPCODE_ROCE_MIRROR_CFG | ||
| 2120 | u8 cmd_size; | ||
| 2121 | __le16 flags; | ||
| 2122 | __le16 cookie; | ||
| 2123 | u8 resp_size; | ||
| 2124 | u8 reserved8; | ||
| 2125 | __le64 resp_addr; | ||
| 2126 | u8 mirror_flags; | ||
| 2127 | #define CMDQ_ROCE_MIRROR_CFG_MIRROR_ENABLE 0x1UL | ||
| 2128 | u8 rsvd[7]; | ||
| 2129 | }; | ||
| 2130 | |||
| 2131 | /* creq_roce_mirror_cfg_resp (size:128b/16B) */ | ||
| 2132 | struct creq_roce_mirror_cfg_resp { | ||
| 2133 | u8 type; | ||
| 2134 | #define CREQ_ROCE_MIRROR_CFG_RESP_TYPE_MASK 0x3fUL | ||
| 2135 | #define CREQ_ROCE_MIRROR_CFG_RESP_TYPE_SFT 0 | ||
| 2136 | #define CREQ_ROCE_MIRROR_CFG_RESP_TYPE_QP_EVENT 0x38UL | ||
| 2137 | #define CREQ_ROCE_MIRROR_CFG_RESP_TYPE_LAST \ | ||
| 2138 | CREQ_ROCE_MIRROR_CFG_RESP_TYPE_QP_EVENT | ||
| 2139 | u8 status; | ||
| 2140 | __le16 cookie; | ||
| 2141 | __le32 reserved32; | ||
| 2142 | u8 v; | ||
| 2143 | #define CREQ_ROCE_MIRROR_CFG_RESP_V 0x1UL | ||
| 2144 | u8 event; | ||
| 2145 | #define CREQ_ROCE_MIRROR_CFG_RESP_EVENT_ROCE_MIRROR_CFG 0x99UL | ||
| 2146 | #define CREQ_ROCE_MIRROR_CFG_RESP_EVENT_LAST \ | ||
| 2147 | CREQ_ROCE_MIRROR_CFG_RESP_EVENT_ROCE_MIRROR_CFG | ||
| 2148 | u8 reserved48[6]; | ||
| 2149 | }; | ||
| 2150 | |||
| 2111 | /* cmdq_query_func (size:128b/16B) */ | 2151 | /* cmdq_query_func (size:128b/16B) */ |
| 2112 | struct cmdq_query_func { | 2152 | struct cmdq_query_func { |
| 2113 | u8 opcode; | 2153 | u8 opcode; |
| @@ -1228,9 +1228,8 @@ static int c4iw_uld_state_change(void *handle, enum cxgb4_state new_state) · 1227 unmodified lines | |||
| 1228 | if (!ctx->dev) { | 1228 | if (!ctx->dev) { |
| 1229 | ctx->dev = c4iw_alloc(&ctx->lldi); | 1229 | ctx->dev = c4iw_alloc(&ctx->lldi); |
| 1230 | if (IS_ERR(ctx->dev)) { | 1230 | if (IS_ERR(ctx->dev)) { |
| 1231 | pr_err("%s: initialization failed: %ld\n", | 1231 | pr_err("%s: initialization failed: %pe\n", |
| 1232 | pci_name(ctx->lldi.pdev), | 1232 | pci_name(ctx->lldi.pdev), ctx->dev); |
| 1233 | PTR_ERR(ctx->dev)); | ||
| 1234 | ctx->dev = NULL; | 1233 | ctx->dev = NULL; |
| 1235 | break; | 1234 | break; |
| 1236 | } | 1235 | } |
| @@ -1,6 +1,6 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause | 1 | // SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause |
| 2 | /* | 2 | /* |
| 3 | * Copyright 2018-2024 Amazon.com, Inc. or its affiliates. All rights reserved. | 3 | * Copyright 2018-2025 Amazon.com, Inc. or its affiliates. All rights reserved. |
| 4 | */ | 4 | */ |
| 5 | 5 | ||
| 6 | #include "efa_com.h" | 6 | #include "efa_com.h" |
| @@ -30,6 +30,7 @@ struct efa_comp_ctx { · 23 unmodified lines | |||
| 30 | struct efa_admin_acq_entry *user_cqe; | 30 | struct efa_admin_acq_entry *user_cqe; |
| 31 | u32 comp_size; | 31 | u32 comp_size; |
| 32 | enum efa_cmd_status status; | 32 | enum efa_cmd_status status; |
| 33 | u16 cmd_id; | ||
| 33 | u8 cmd_opcode; | 34 | u8 cmd_opcode; |
| 34 | u8 occupied; | 35 | u8 occupied; |
| 35 | }; | 36 | }; |
| @@ -333,6 +334,7 @@ static struct efa_comp_ctx *__efa_com_submit_admin_cmd(struct efa_com_admin_queu · 297 unmodified lines | |||
| 333 | comp_ctx->comp_size = comp_size_in_bytes; | 334 | comp_ctx->comp_size = comp_size_in_bytes; |
| 334 | comp_ctx->user_cqe = comp; | 335 | comp_ctx->user_cqe = comp; |
| 335 | comp_ctx->cmd_opcode = cmd->aq_common_descriptor.opcode; | 336 | comp_ctx->cmd_opcode = cmd->aq_common_descriptor.opcode; |
| 337 | comp_ctx->cmd_id = cmd_id; | ||
| 336 | 338 | ||
| 337 | reinit_completion(&comp_ctx->wait_event); | 339 | reinit_completion(&comp_ctx->wait_event); |
| 338 | 340 | ||
| @@ -557,17 +559,19 @@ static int efa_com_wait_and_process_admin_cq_interrupts(struct efa_comp_ctx *com · 218 unmodified lines | |||
| 557 | if (comp_ctx->status == EFA_CMD_COMPLETED) | 559 | if (comp_ctx->status == EFA_CMD_COMPLETED) |
| 558 | ibdev_err_ratelimited( | 560 | ibdev_err_ratelimited( |
| 559 | aq->efa_dev, | 561 | aq->efa_dev, |
| 560 | "The device sent a completion but the driver didn't receive any MSI-X interrupt for admin cmd %s(%d) status %d (ctx: 0x%p, sq producer: %d, sq consumer: %d, cq consumer: %d)\n", | 562 | "The device sent a completion but the driver didn't receive any MSI-X interrupt for admin cmd %s(%d) status %d (id: %d, sq producer: %d, sq consumer: %d, cq consumer: %d)\n", |
| 561 | efa_com_cmd_str(comp_ctx->cmd_opcode), | 563 | efa_com_cmd_str(comp_ctx->cmd_opcode), |
| 562 | comp_ctx->cmd_opcode, comp_ctx->status, | 564 | comp_ctx->cmd_opcode, comp_ctx->status, |
| 563 | comp_ctx, aq->sq.pc, aq->sq.cc, aq->cq.cc); | 565 | comp_ctx->cmd_id, aq->sq.pc, aq->sq.cc, |
| 566 | aq->cq.cc); | ||
| 564 | else | 567 | else |
| 565 | ibdev_err_ratelimited( | 568 | ibdev_err_ratelimited( |
| 566 | aq->efa_dev, | 569 | aq->efa_dev, |
| 567 | "The device didn't send any completion for admin cmd %s(%d) status %d (ctx 0x%p, sq producer: %d, sq consumer: %d, cq consumer: %d)\n", | 570 | "The device didn't send any completion for admin cmd %s(%d) status %d (id: %d, sq producer: %d, sq consumer: %d, cq consumer: %d)\n", |
| 568 | efa_com_cmd_str(comp_ctx->cmd_opcode), | 571 | efa_com_cmd_str(comp_ctx->cmd_opcode), |
| 569 | comp_ctx->cmd_opcode, comp_ctx->status, | 572 | comp_ctx->cmd_opcode, comp_ctx->status, |
| 570 | comp_ctx, aq->sq.pc, aq->sq.cc, aq->cq.cc); | 573 | comp_ctx->cmd_id, aq->sq.pc, aq->sq.cc, |
| 574 | aq->cq.cc); | ||
| 571 | 575 | ||
| 572 | clear_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state); | 576 | clear_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state); |
| 573 | err = -ETIME; | 577 | err = -ETIME; |
| @@ -631,9 +635,9 @@ int efa_com_cmd_exec(struct efa_com_admin_queue *aq, · 57 unmodified lines | |||
| 631 | if (IS_ERR(comp_ctx)) { | 635 | if (IS_ERR(comp_ctx)) { |
| 632 | ibdev_err_ratelimited( | 636 | ibdev_err_ratelimited( |
| 633 | aq->efa_dev, | 637 | aq->efa_dev, |
| 634 | "Failed to submit command %s (opcode %u) err %ld\n", | 638 | "Failed to submit command %s (opcode %u) err %pe\n", |
| 635 | efa_com_cmd_str(cmd->aq_common_descriptor.opcode), | 639 | efa_com_cmd_str(cmd->aq_common_descriptor.opcode), |
| 636 | cmd->aq_common_descriptor.opcode, PTR_ERR(comp_ctx)); | 640 | cmd->aq_common_descriptor.opcode, comp_ctx); |
| 637 | 641 | ||
| 638 | up(&aq->avail_cmds); | 642 | up(&aq->avail_cmds); |
| 639 | atomic64_inc(&aq->stats.cmd_err); | 643 | atomic64_inc(&aq->stats.cmd_err); |
| @@ -1788,7 +1788,8 @@ struct ib_mr *efa_reg_user_mr_dmabuf(struct ib_pd *ibpd, u64 start, · 1787 unmodified lines | |||
| 1788 | access_flags); | 1788 | access_flags); |
| 1789 | if (IS_ERR(umem_dmabuf)) { | 1789 | if (IS_ERR(umem_dmabuf)) { |
| 1790 | err = PTR_ERR(umem_dmabuf); | 1790 | err = PTR_ERR(umem_dmabuf); |
| 1791 | ibdev_dbg(&dev->ibdev, "Failed to get dmabuf umem[%d]\n", err); | 1791 | ibdev_dbg(&dev->ibdev, "Failed to get dmabuf umem[%pe]\n", |
| 1792 | umem_dmabuf); | ||
| 1792 | goto err_free; | 1793 | goto err_free; |
| 1793 | } | 1794 | } |
| 1794 | 1795 | ||
| @@ -1832,7 +1833,8 @@ struct ib_mr *efa_reg_mr(struct ib_pd *ibpd, u64 start, u64 length, · 37 unmodified lines | |||
| 1832 | if (IS_ERR(mr->umem)) { | 1833 | if (IS_ERR(mr->umem)) { |
| 1833 | err = PTR_ERR(mr->umem); | 1834 | err = PTR_ERR(mr->umem); |
| 1834 | ibdev_dbg(&dev->ibdev, | 1835 | ibdev_dbg(&dev->ibdev, |
| 1835 | "Failed to pin and map user space memory[%d]\n", err); | 1836 | "Failed to pin and map user space memory[%pe]\n", |
| 1837 | mr->umem); | ||
| 1836 | goto err_free; | 1838 | goto err_free; |
| 1837 | } | 1839 | } |
| 1838 | 1840 | ||
| @@ -149,7 +149,7 @@ static int regmr_cmd(struct erdma_dev *dev, struct erdma_mr *mr) · 148 unmodified lines | |||
| 149 | req.phy_addr[0] = mr->mem.mtt->buf_dma; | 149 | req.phy_addr[0] = mr->mem.mtt->buf_dma; |
| 150 | mtt_level = ERDMA_MR_MTT_1LEVEL; | 150 | mtt_level = ERDMA_MR_MTT_1LEVEL; |
| 151 | } else { | 151 | } else { |
| 152 | req.phy_addr[0] = sg_dma_address(mr->mem.mtt->sglist); | 152 | req.phy_addr[0] = mr->mem.mtt->dma_addrs[0]; |
| 153 | mtt_level = mr->mem.mtt->level; | 153 | mtt_level = mr->mem.mtt->level; |
| 154 | } | 154 | } |
| 155 | } else if (mr->type != ERDMA_MR_TYPE_DMA) { | 155 | } else if (mr->type != ERDMA_MR_TYPE_DMA) { |
| @@ -626,18 +626,27 @@ err_free_mtt: · 470 unmodified lines | |||
| 626 | return ERR_PTR(-ENOMEM); | 626 | return ERR_PTR(-ENOMEM); |
| 627 | } | 627 | } |
| 628 | 628 | ||
| 629 | static void erdma_destroy_mtt_buf_sg(struct erdma_dev *dev, | 629 | static void erdma_unmap_page_list(struct erdma_dev *dev, dma_addr_t *pg_dma, |
| 630 | struct erdma_mtt *mtt) | 630 | u32 npages) |
| 631 | { | 631 | { |
| 632 | dma_unmap_sg(&dev->pdev->dev, mtt->sglist, | 632 | u32 i; |
| 633 | DIV_ROUND_UP(mtt->size, PAGE_SIZE), DMA_TO_DEVICE); | 633 | |
| 634 | vfree(mtt->sglist); | 634 | for (i = 0; i < npages; i++) |
| 635 | dma_unmap_page(&dev->pdev->dev, pg_dma[i], PAGE_SIZE, | ||
| 636 | DMA_TO_DEVICE); | ||
| 637 | } | ||
| 638 | |||
| 639 | static void erdma_destroy_mtt_buf_dma_addrs(struct erdma_dev *dev, | ||
| 640 | struct erdma_mtt *mtt) | ||
| 641 | { | ||
| 642 | erdma_unmap_page_list(dev, mtt->dma_addrs, mtt->npages); | ||
| 643 | vfree(mtt->dma_addrs); | ||
| 635 | } | 644 | } |
| 636 | 645 | ||
| 637 | static void erdma_destroy_scatter_mtt(struct erdma_dev *dev, | 646 | static void erdma_destroy_scatter_mtt(struct erdma_dev *dev, |
| 638 | struct erdma_mtt *mtt) | 647 | struct erdma_mtt *mtt) |
| 639 | { | 648 | { |
| 640 | erdma_destroy_mtt_buf_sg(dev, mtt); | 649 | erdma_destroy_mtt_buf_dma_addrs(dev, mtt); |
| 641 | vfree(mtt->buf); | 650 | vfree(mtt->buf); |
| 642 | kfree(mtt); | 651 | kfree(mtt); |
| 643 | } | 652 | } |
| @@ -645,50 +654,69 @@ static void erdma_destroy_scatter_mtt(struct erdma_dev *dev, · 1 unmodified line | |||
| 645 | static void erdma_init_middle_mtt(struct erdma_mtt *mtt, | 654 | static void erdma_init_middle_mtt(struct erdma_mtt *mtt, |
| 646 | struct erdma_mtt *low_mtt) | 655 | struct erdma_mtt *low_mtt) |
| 647 | { | 656 | { |
| 648 | struct scatterlist *sg; | 657 | dma_addr_t *pg_addr = mtt->buf; |
| 649 | u32 idx = 0, i; | 658 | u32 i; |
| 650 | 659 | ||
| 651 | for_each_sg(low_mtt->sglist, sg, low_mtt->nsg, i) | 660 | for (i = 0; i < low_mtt->npages; i++) |
| 652 | mtt->buf[idx++] = sg_dma_address(sg); | 661 | pg_addr[i] = low_mtt->dma_addrs[i]; |
| 653 | } | 662 | } |
| 654 | 663 | ||
| 655 | static int erdma_create_mtt_buf_sg(struct erdma_dev *dev, struct erdma_mtt *mtt) | 664 | static u32 vmalloc_to_dma_addrs(struct erdma_dev *dev, dma_addr_t **dma_addrs, |
| 665 | void *buf, u64 len) | ||
| 656 | { | 666 | { |
| 657 | struct scatterlist *sglist; | 667 | dma_addr_t *pg_dma; |
| 658 | void *buf = mtt->buf; | ||
| 659 | u32 npages, i, nsg; | ||
| 660 | struct page *pg; | 668 | struct page *pg; |
| 669 | u32 npages, i; | ||
| 670 | void *addr; | ||
| 661 | 671 | ||
| 662 | /* Failed if buf is not page aligned */ | 672 | npages = (PAGE_ALIGN((u64)buf + len) - PAGE_ALIGN_DOWN((u64)buf)) >> |
| 663 | if ((uintptr_t)buf & ~PAGE_MASK) | 673 | PAGE_SHIFT; |
| 664 | return -EINVAL; | 674 | pg_dma = vcalloc(npages, sizeof(*pg_dma)); |
| 665 | 675 | if (!pg_dma) | |
| 666 | npages = DIV_ROUND_UP(mtt->size, PAGE_SIZE); | 676 | return 0; |
| 667 | sglist = vzalloc(npages * sizeof(*sglist)); | ||
| 668 | if (!sglist) | ||
| 669 | return -ENOMEM; | ||
| 670 | 677 | ||
| 671 | sg_init_table(sglist, npages); | 678 | addr = buf; |
| 672 | for (i = 0; i < npages; i++) { | 679 | for (i = 0; i < npages; i++) { |
| 673 | pg = vmalloc_to_page(buf); | 680 | pg = vmalloc_to_page(addr); |
| 674 | if (!pg) | 681 | if (!pg) |
| 675 | goto err; | 682 | goto err; |
| 676 | sg_set_page(&sglist[i], pg, PAGE_SIZE, 0); | 683 | |
| 677 | buf += PAGE_SIZE; | 684 | pg_dma[i] = dma_map_page(&dev->pdev->dev, pg, 0, PAGE_SIZE, |
| 685 | DMA_TO_DEVICE); | ||
| 686 | if (dma_mapping_error(&dev->pdev->dev, pg_dma[i])) | ||
| 687 | goto err; | ||
| 688 | |||
| 689 | addr += PAGE_SIZE; | ||
| 678 | } | 690 | } |
| 679 | 691 | ||
| 680 | nsg = dma_map_sg(&dev->pdev->dev, sglist, npages, DMA_TO_DEVICE); | 692 | *dma_addrs = pg_dma; |
| 681 | if (!nsg) | ||
| 682 | goto err; | ||
| 683 | 693 | ||
| 684 | mtt->sglist = sglist; | 694 | return npages; |
| 685 | mtt->nsg = nsg; | 695 | err: |
| 696 | erdma_unmap_page_list(dev, pg_dma, i); | ||
| 697 | vfree(pg_dma); | ||
| 686 | 698 | ||
| 687 | return 0; | 699 | return 0; |
| 688 | err: | 700 | } |
| 689 | vfree(sglist); | ||
| 690 | 701 | ||
| 691 | return -ENOMEM; | 702 | static int erdma_create_mtt_buf_dma_addrs(struct erdma_dev *dev, |
| 703 | struct erdma_mtt *mtt) | ||
| 704 | { | ||
| 705 | dma_addr_t *addrs; | ||
| 706 | u32 npages; | ||
| 707 | |||
| 708 | /* Failed if buf is not page aligned */ | ||
| 709 | if ((uintptr_t)mtt->buf & ~PAGE_MASK) | ||
| 710 | return -EINVAL; | ||
| 711 | |||
| 712 | npages = vmalloc_to_dma_addrs(dev, &addrs, mtt->buf, mtt->size); | ||
| 713 | if (!npages) | ||
| 714 | return -ENOMEM; | ||
| 715 | |||
| 716 | mtt->dma_addrs = addrs; | ||
| 717 | mtt->npages = npages; | ||
| 718 | |||
| 719 | return 0; | ||
| 692 | } | 720 | } |
| 693 | 721 | ||
| 694 | static struct erdma_mtt *erdma_create_scatter_mtt(struct erdma_dev *dev, | 722 | static struct erdma_mtt *erdma_create_scatter_mtt(struct erdma_dev *dev, |
| @@ -707,12 +735,12 @@ static struct erdma_mtt *erdma_create_scatter_mtt(struct erdma_dev *dev, · 12 unmodified lines | |||
| 707 | if (!mtt->buf) | 735 | if (!mtt->buf) |
| 708 | goto err_free_mtt; | 736 | goto err_free_mtt; |
| 709 | 737 | ||
| 710 | ret = erdma_create_mtt_buf_sg(dev, mtt); | 738 | ret = erdma_create_mtt_buf_dma_addrs(dev, mtt); |
| 711 | if (ret) | 739 | if (ret) |
| 712 | goto err_free_mtt_buf; | 740 | goto err_free_mtt_buf; |
| 713 | 741 | ||
| 714 | ibdev_dbg(&dev->ibdev, "create scatter mtt, size:%lu, nsg:%u\n", | 742 | ibdev_dbg(&dev->ibdev, "create scatter mtt, size:%lu, npages:%u\n", |
| 715 | mtt->size, mtt->nsg); | 743 | mtt->size, mtt->npages); |
| 716 | 744 | ||
| 717 | return mtt; | 745 | return mtt; |
| 718 | 746 | ||
| @@ -746,8 +774,8 @@ static struct erdma_mtt *erdma_create_mtt(struct erdma_dev *dev, size_t size, · 27 unmodified lines | |||
| 746 | level = 1; | 774 | level = 1; |
| 747 | 775 | ||
| 748 | /* convergence the mtt table. */ | 776 | /* convergence the mtt table. */ |
| 749 | while (mtt->nsg != 1 && level <= 3) { | 777 | while (mtt->npages != 1 && level <= 3) { |
| 750 | tmp_mtt = erdma_create_scatter_mtt(dev, MTT_SIZE(mtt->nsg)); | 778 | tmp_mtt = erdma_create_scatter_mtt(dev, MTT_SIZE(mtt->npages)); |
| 751 | if (IS_ERR(tmp_mtt)) { | 779 | if (IS_ERR(tmp_mtt)) { |
| 752 | ret = PTR_ERR(tmp_mtt); | 780 | ret = PTR_ERR(tmp_mtt); |
| 753 | goto err_free_mtt; | 781 | goto err_free_mtt; |
| @@ -765,7 +793,7 @@ static struct erdma_mtt *erdma_create_mtt(struct erdma_dev *dev, size_t size, · 11 unmodified lines | |||
| 765 | 793 | ||
| 766 | mtt->level = level; | 794 | mtt->level = level; |
| 767 | ibdev_dbg(&dev->ibdev, "top mtt: level:%d, dma_addr 0x%llx\n", | 795 | ibdev_dbg(&dev->ibdev, "top mtt: level:%d, dma_addr 0x%llx\n", |
| 768 | mtt->level, mtt->sglist[0].dma_address); | 796 | mtt->level, mtt->dma_addrs[0]); |
| 769 | 797 | ||
| 770 | return mtt; | 798 | return mtt; |
| 771 | err_free_mtt: | 799 | err_free_mtt: |
| @@ -99,8 +99,8 @@ struct erdma_mtt { · 98 unmodified lines | |||
| 99 | union { | 99 | union { |
| 100 | dma_addr_t buf_dma; | 100 | dma_addr_t buf_dma; |
| 101 | struct { | 101 | struct { |
| 102 | struct scatterlist *sglist; | 102 | dma_addr_t *dma_addrs; |
| 103 | u32 nsg; | 103 | u32 npages; |
| 104 | u32 level; | 104 | u32 level; |
| 105 | }; | 105 | }; |
| 106 | }; | 106 | }; |
| @@ -64,9 +64,9 @@ int hfi1_cdev_init(int minor, const char *name, · 63 unmodified lines | |||
| 64 | 64 | ||
| 65 | if (IS_ERR(device)) { | 65 | if (IS_ERR(device)) { |
| 66 | ret = PTR_ERR(device); | 66 | ret = PTR_ERR(device); |
| 67 | pr_err("Could not create device for minor %d, %s (err %pe)\n", | ||
| 68 | minor, name, device); | ||
| 67 | device = NULL; | 69 | device = NULL; |
| 68 | pr_err("Could not create device for minor %d, %s (err %d)\n", | ||
| 69 | minor, name, -ret); | ||
| 70 | cdev_del(cdev); | 70 | cdev_del(cdev); |
| 71 | } | 71 | } |
| 72 | done: | 72 | done: |
| @@ -990,7 +990,7 @@ ssize_t sdma_set_cpu_to_sde_map(struct sdma_engine *sde, const char *buf, · 989 unmodified lines | |||
| 990 | } | 990 | } |
| 991 | 991 | ||
| 992 | /* Clean up old mappings */ | 992 | /* Clean up old mappings */ |
| 993 | for_each_cpu(cpu, cpu_online_mask) { | 993 | for_each_online_cpu(cpu) { |
| 994 | struct sdma_rht_node *rht_node; | 994 | struct sdma_rht_node *rht_node; |
| 995 | 995 | ||
| 996 | /* Don't cleanup sdes that are set in the new mask */ | 996 | /* Don't cleanup sdes that are set in the new mask */ |
| @@ -498,8 +498,8 @@ int hfi1_user_sdma_process_request(struct hfi1_filedata *fd, · 497 unmodified lines | |||
| 498 | ntids, sizeof(*req->tids)); | 498 | ntids, sizeof(*req->tids)); |
| 499 | if (IS_ERR(tmp)) { | 499 | if (IS_ERR(tmp)) { |
| 500 | ret = PTR_ERR(tmp); | 500 | ret = PTR_ERR(tmp); |
| 501 | SDMA_DBG(req, "Failed to copy %d TIDs (%d)", | 501 | SDMA_DBG(req, "Failed to copy %d TIDs (%pe)", ntids, |
| 502 | ntids, ret); | 502 | tmp); |
| 503 | goto free_req; | 503 | goto free_req; |
| 504 | } | 504 | } |
| 505 | req->tids = tmp; | 505 | req->tids = tmp; |
| @@ -594,8 +594,8 @@ static int mtr_alloc_bufs(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr, · 593 unmodified lines | |||
| 594 | mtr->umem = ib_umem_get(ibdev, user_addr, total_size, | 594 | mtr->umem = ib_umem_get(ibdev, user_addr, total_size, |
| 595 | buf_attr->user_access); | 595 | buf_attr->user_access); |
| 596 | if (IS_ERR(mtr->umem)) { | 596 | if (IS_ERR(mtr->umem)) { |
| 597 | ibdev_err(ibdev, "failed to get umem, ret = %ld.\n", | 597 | ibdev_err(ibdev, "failed to get umem, ret = %pe.\n", |
| 598 | PTR_ERR(mtr->umem)); | 598 | mtr->umem); |
| 599 | return -ENOMEM; | 599 | return -ENOMEM; |
| 600 | } | 600 | } |
| 601 | } else { | 601 | } else { |
| @@ -605,8 +605,8 @@ static int mtr_alloc_bufs(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr, · 3 unmodified lines | |||
| 605 | !mtr_has_mtt(buf_attr) ? | 605 | !mtr_has_mtt(buf_attr) ? |
| 606 | HNS_ROCE_BUF_DIRECT : 0); | 606 | HNS_ROCE_BUF_DIRECT : 0); |
| 607 | if (IS_ERR(mtr->kmem)) { | 607 | if (IS_ERR(mtr->kmem)) { |
| 608 | ibdev_err(ibdev, "failed to alloc kmem, ret = %ld.\n", | 608 | ibdev_err(ibdev, "failed to alloc kmem, ret = %pe.\n", |
| 609 | PTR_ERR(mtr->kmem)); | 609 | mtr->kmem); |
| 610 | return PTR_ERR(mtr->kmem); | 610 | return PTR_ERR(mtr->kmem); |
| 611 | } | 611 | } |
| 612 | } | 612 | } |
| @@ -0,0 +1,15 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0 | ||
| 2 | # Copyright (C) 2018-2025, Advanced Micro Devices, Inc. | ||
| 3 | |||
| 4 | config INFINIBAND_IONIC | ||
| 5 | tristate "AMD Pensando DSC RDMA/RoCE Support" | ||
| 6 | depends on NETDEVICES && ETHERNET && PCI && INET && IONIC | ||
| 7 | help | ||
| 8 | This enables RDMA/RoCE support for the AMD Pensando family of | ||
| 9 | Distributed Services Cards (DSCs). | ||
| 10 | |||
| 11 | To learn more, visit our website at | ||
| 12 | <https://www.amd.com/en/products/accelerators/pensando.html>. | ||
| 13 | |||
| 14 | To compile this driver as a module, choose M here. The module | ||
| 15 | will be called ionic_rdma. | ||
| @@ -0,0 +1,9 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0 | ||
| 2 | |||
| 3 | ccflags-y := -I $(srctree)/drivers/net/ethernet/pensando/ionic | ||
| 4 | |||
| 5 | obj-$(CONFIG_INFINIBAND_IONIC) += ionic_rdma.o | ||
| 6 | |||
| 7 | ionic_rdma-y := \ | ||
| 8 | ionic_ibdev.o ionic_lif_cfg.o ionic_queue.o ionic_pgtbl.o ionic_admin.o \ | ||
| 9 | ionic_controlpath.o ionic_datapath.o ionic_hw_stats.o | ||
| @@ -0,0 +1,1229 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #include <linux/interrupt.h> | ||
| 5 | #include <linux/module.h> | ||
| 6 | #include <linux/printk.h> | ||
| 7 | |||
| 8 | #include "ionic_fw.h" | ||
| 9 | #include "ionic_ibdev.h" | ||
| 10 | |||
| 11 | #define IONIC_EQ_COUNT_MIN 4 | ||
| 12 | #define IONIC_AQ_COUNT_MIN 1 | ||
| 13 | |||
| 14 | /* not a valid queue position or negative error status */ | ||
| 15 | #define IONIC_ADMIN_POSTED 0x10000 | ||
| 16 | |||
| 17 | /* cpu can be held with irq disabled for COUNT * MS (for create/destroy_ah) */ | ||
| 18 | #define IONIC_ADMIN_BUSY_RETRY_COUNT 2000 | ||
| 19 | #define IONIC_ADMIN_BUSY_RETRY_MS 1 | ||
| 20 | |||
| 21 | /* admin queue will be considered failed if a command takes longer */ | ||
| 22 | #define IONIC_ADMIN_TIMEOUT (HZ * 2) | ||
| 23 | #define IONIC_ADMIN_WARN (HZ / 8) | ||
| 24 | |||
| 25 | /* will poll for admin cq to tolerate and report from missed event */ | ||
| 26 | #define IONIC_ADMIN_DELAY (HZ / 8) | ||
| 27 | |||
| 28 | /* work queue for polling the event queue and admin cq */ | ||
| 29 | struct workqueue_struct *ionic_evt_workq; | ||
| 30 | |||
| 31 | static void ionic_admin_timedout(struct ionic_aq *aq) | ||
| 32 | { | ||
| 33 | struct ionic_ibdev *dev = aq->dev; | ||
| 34 | unsigned long irqflags; | ||
| 35 | u16 pos; | ||
| 36 | |||
| 37 | spin_lock_irqsave(&aq->lock, irqflags); | ||
| 38 | if (ionic_queue_empty(&aq->q)) | ||
| 39 | goto out; | ||
| 40 | |||
| 41 | /* Reset ALL adminq if any one times out */ | ||
| 42 | if (atomic_read(&aq->admin_state) < IONIC_ADMIN_KILLED) | ||
| 43 | queue_work(ionic_evt_workq, &dev->reset_work); | ||
| 44 | |||
| 45 | ibdev_err(&dev->ibdev, "admin command timed out, aq %d after: %ums\n", | ||
| 46 | aq->aqid, (u32)jiffies_to_msecs(jiffies - aq->stamp)); | ||
| 47 | |||
| 48 | pos = (aq->q.prod - 1) & aq->q.mask; | ||
| 49 | if (pos == aq->q.cons) | ||
| 50 | goto out; | ||
| 51 | |||
| 52 | ibdev_warn(&dev->ibdev, "admin pos %u (last posted)\n", pos); | ||
| 53 | print_hex_dump(KERN_WARNING, "cmd ", DUMP_PREFIX_OFFSET, 16, 1, | ||
| 54 | ionic_queue_at(&aq->q, pos), | ||
| 55 | BIT(aq->q.stride_log2), true); | ||
| 56 | |||
| 57 | out: | ||
| 58 | spin_unlock_irqrestore(&aq->lock, irqflags); | ||
| 59 | } | ||
| 60 | |||
| 61 | static void ionic_admin_reset_dwork(struct ionic_ibdev *dev) | ||
| 62 | { | ||
| 63 | if (atomic_read(&dev->admin_state) == IONIC_ADMIN_KILLED) | ||
| 64 | return; | ||
| 65 | |||
| 66 | queue_delayed_work(ionic_evt_workq, &dev->admin_dwork, | ||
| 67 | IONIC_ADMIN_DELAY); | ||
| 68 | } | ||
| 69 | |||
| 70 | static void ionic_admin_reset_wdog(struct ionic_aq *aq) | ||
| 71 | { | ||
| 72 | if (atomic_read(&aq->admin_state) == IONIC_ADMIN_KILLED) | ||
| 73 | return; | ||
| 74 | |||
| 75 | aq->stamp = jiffies; | ||
| 76 | ionic_admin_reset_dwork(aq->dev); | ||
| 77 | } | ||
| 78 | |||
| 79 | static bool ionic_admin_next_cqe(struct ionic_ibdev *dev, struct ionic_cq *cq, | ||
| 80 | struct ionic_v1_cqe **cqe) | ||
| 81 | { | ||
| 82 | struct ionic_v1_cqe *qcqe = ionic_queue_at_prod(&cq->q); | ||
| 83 | |||
| 84 | if (unlikely(cq->color != ionic_v1_cqe_color(qcqe))) | ||
| 85 | return false; | ||
| 86 | |||
| 87 | /* Prevent out-of-order reads of the CQE */ | ||
| 88 | dma_rmb(); | ||
| 89 | *cqe = qcqe; | ||
| 90 | |||
| 91 | return true; | ||
| 92 | } | ||
| 93 | |||
| 94 | static void ionic_admin_poll_locked(struct ionic_aq *aq) | ||
| 95 | { | ||
| 96 | struct ionic_cq *cq = &aq->vcq->cq[0]; | ||
| 97 | struct ionic_admin_wr *wr, *wr_next; | ||
| 98 | struct ionic_ibdev *dev = aq->dev; | ||
| 99 | u32 wr_strides, avlbl_strides; | ||
| 100 | struct ionic_v1_cqe *cqe; | ||
| 101 | u32 qtf, qid; | ||
| 102 | u16 old_prod; | ||
| 103 | u8 type; | ||
| 104 | |||
| 105 | lockdep_assert_held(&aq->lock); | ||
| 106 | |||
| 107 | if (atomic_read(&aq->admin_state) == IONIC_ADMIN_KILLED) { | ||
| 108 | list_for_each_entry_safe(wr, wr_next, &aq->wr_prod, aq_ent) { | ||
| 109 | INIT_LIST_HEAD(&wr->aq_ent); | ||
| 110 | aq->q_wr[wr->status].wr = NULL; | ||
| 111 | wr->status = atomic_read(&aq->admin_state); | ||
| 112 | complete_all(&wr->work); | ||
| 113 | } | ||
| 114 | INIT_LIST_HEAD(&aq->wr_prod); | ||
| 115 | |||
| 116 | list_for_each_entry_safe(wr, wr_next, &aq->wr_post, aq_ent) { | ||
| 117 | INIT_LIST_HEAD(&wr->aq_ent); | ||
| 118 | wr->status = atomic_read(&aq->admin_state); | ||
| 119 | complete_all(&wr->work); | ||
| 120 | } | ||
| 121 | INIT_LIST_HEAD(&aq->wr_post); | ||
| 122 | |||
| 123 | return; | ||
| 124 | } | ||
| 125 | |||
| 126 | old_prod = cq->q.prod; | ||
| 127 | |||
| 128 | while (ionic_admin_next_cqe(dev, cq, &cqe)) { | ||
| 129 | qtf = ionic_v1_cqe_qtf(cqe); | ||
| 130 | qid = ionic_v1_cqe_qtf_qid(qtf); | ||
| 131 | type = ionic_v1_cqe_qtf_type(qtf); | ||
| 132 | |||
| 133 | if (unlikely(type != IONIC_V1_CQE_TYPE_ADMIN)) { | ||
| 134 | ibdev_warn_ratelimited(&dev->ibdev, | ||
| 135 | "bad cqe type %u\n", type); | ||
| 136 | goto cq_next; | ||
| 137 | } | ||
| 138 | |||
| 139 | if (unlikely(qid != aq->aqid)) { | ||
| 140 | ibdev_warn_ratelimited(&dev->ibdev, | ||
| 141 | "bad cqe qid %u\n", qid); | ||
| 142 | goto cq_next; | ||
| 143 | } | ||
| 144 | |||
| 145 | if (unlikely(be16_to_cpu(cqe->admin.cmd_idx) != aq->q.cons)) { | ||
| 146 | ibdev_warn_ratelimited(&dev->ibdev, | ||
| 147 | "bad idx %u cons %u qid %u\n", | ||
| 148 | be16_to_cpu(cqe->admin.cmd_idx), | ||
| 149 | aq->q.cons, qid); | ||
| 150 | goto cq_next; | ||
| 151 | } | ||
| 152 | |||
| 153 | if (unlikely(ionic_queue_empty(&aq->q))) { | ||
| 154 | ibdev_warn_ratelimited(&dev->ibdev, | ||
| 155 | "bad cqe for empty adminq\n"); | ||
| 156 | goto cq_next; | ||
| 157 | } | ||
| 158 | |||
| 159 | wr = aq->q_wr[aq->q.cons].wr; | ||
| 160 | if (wr) { | ||
| 161 | aq->q_wr[aq->q.cons].wr = NULL; | ||
| 162 | list_del_init(&wr->aq_ent); | ||
| 163 | |||
| 164 | wr->cqe = *cqe; | ||
| 165 | wr->status = atomic_read(&aq->admin_state); | ||
| 166 | complete_all(&wr->work); | ||
| 167 | } | ||
| 168 | |||
| 169 | ionic_queue_consume_entries(&aq->q, | ||
| 170 | aq->q_wr[aq->q.cons].wqe_strides); | ||
| 171 | |||
| 172 | cq_next: | ||
| 173 | ionic_queue_produce(&cq->q); | ||
| 174 | cq->color = ionic_color_wrap(cq->q.prod, cq->color); | ||
| 175 | } | ||
| 176 | |||
| 177 | if (old_prod != cq->q.prod) { | ||
| 178 | ionic_admin_reset_wdog(aq); | ||
| 179 | cq->q.cons = cq->q.prod; | ||
| 180 | ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.cq_qtype, | ||
| 181 | ionic_queue_dbell_val(&cq->q)); | ||
| 182 | queue_work(ionic_evt_workq, &aq->work); | ||
| 183 | } else if (!aq->armed) { | ||
| 184 | aq->armed = true; | ||
| 185 | cq->arm_any_prod = ionic_queue_next(&cq->q, cq->arm_any_prod); | ||
| 186 | ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.cq_qtype, | ||
| 187 | cq->q.dbell | IONIC_CQ_RING_ARM | | ||
| 188 | cq->arm_any_prod); | ||
| 189 | queue_work(ionic_evt_workq, &aq->work); | ||
| 190 | } | ||
| 191 | |||
| 192 | if (atomic_read(&aq->admin_state) != IONIC_ADMIN_ACTIVE) | ||
| 193 | return; | ||
| 194 | |||
| 195 | old_prod = aq->q.prod; | ||
| 196 | |||
| 197 | if (ionic_queue_empty(&aq->q) && !list_empty(&aq->wr_post)) | ||
| 198 | ionic_admin_reset_wdog(aq); | ||
| 199 | |||
| 200 | if (list_empty(&aq->wr_post)) | ||
| 201 | return; | ||
| 202 | |||
| 203 | do { | ||
| 204 | u8 *src; | ||
| 205 | int i, src_len; | ||
| 206 | size_t stride_len; | ||
| 207 | |||
| 208 | wr = list_first_entry(&aq->wr_post, struct ionic_admin_wr, | ||
| 209 | aq_ent); | ||
| 210 | wr_strides = (le16_to_cpu(wr->wqe.len) + ADMIN_WQE_HDR_LEN + | ||
| 211 | (ADMIN_WQE_STRIDE - 1)) >> aq->q.stride_log2; | ||
| 212 | avlbl_strides = ionic_queue_length_remaining(&aq->q); | ||
| 213 | |||
| 214 | if (wr_strides > avlbl_strides) | ||
| 215 | break; | ||
| 216 | |||
| 217 | list_move(&wr->aq_ent, &aq->wr_prod); | ||
| 218 | wr->status = aq->q.prod; | ||
| 219 | aq->q_wr[aq->q.prod].wr = wr; | ||
| 220 | aq->q_wr[aq->q.prod].wqe_strides = wr_strides; | ||
| 221 | |||
| 222 | src_len = le16_to_cpu(wr->wqe.len); | ||
| 223 | src = (uint8_t *)&wr->wqe.cmd; | ||
| 224 | |||
| 225 | /* First stride */ | ||
| 226 | memcpy(ionic_queue_at_prod(&aq->q), &wr->wqe, | ||
| 227 | ADMIN_WQE_HDR_LEN); | ||
| 228 | stride_len = ADMIN_WQE_STRIDE - ADMIN_WQE_HDR_LEN; | ||
| 229 | if (stride_len > src_len) | ||
| 230 | stride_len = src_len; | ||
| 231 | memcpy(ionic_queue_at_prod(&aq->q) + ADMIN_WQE_HDR_LEN, | ||
| 232 | src, stride_len); | ||
| 233 | ibdev_dbg(&dev->ibdev, "post admin prod %u (%u strides)\n", | ||
| 234 | aq->q.prod, wr_strides); | ||
| 235 | print_hex_dump_debug("wqe ", DUMP_PREFIX_OFFSET, 16, 1, | ||
| 236 | ionic_queue_at_prod(&aq->q), | ||
| 237 | BIT(aq->q.stride_log2), true); | ||
| 238 | ionic_queue_produce(&aq->q); | ||
| 239 | |||
| 240 | /* Remaining strides */ | ||
| 241 | for (i = stride_len; i < src_len; i += stride_len) { | ||
| 242 | stride_len = ADMIN_WQE_STRIDE; | ||
| 243 | |||
| 244 | if (i + stride_len > src_len) | ||
| 245 | stride_len = src_len - i; | ||
| 246 | |||
| 247 | memcpy(ionic_queue_at_prod(&aq->q), src + i, | ||
| 248 | stride_len); | ||
| 249 | print_hex_dump_debug("wqe ", DUMP_PREFIX_OFFSET, 16, 1, | ||
| 250 | ionic_queue_at_prod(&aq->q), | ||
| 251 | BIT(aq->q.stride_log2), true); | ||
| 252 | ionic_queue_produce(&aq->q); | ||
| 253 | } | ||
| 254 | } while (!list_empty(&aq->wr_post)); | ||
| 255 | |||
| 256 | if (old_prod != aq->q.prod) | ||
| 257 | ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.aq_qtype, | ||
| 258 | ionic_queue_dbell_val(&aq->q)); | ||
| 259 | } | ||
| 260 | |||
| 261 | static void ionic_admin_dwork(struct work_struct *ws) | ||
| 262 | { | ||
| 263 | struct ionic_ibdev *dev = | ||
| 264 | container_of(ws, struct ionic_ibdev, admin_dwork.work); | ||
| 265 | struct ionic_aq *aq, *bad_aq = NULL; | ||
| 266 | bool do_reschedule = false; | ||
| 267 | unsigned long irqflags; | ||
| 268 | bool do_reset = false; | ||
| 269 | u16 pos; | ||
| 270 | int i; | ||
| 271 | |||
| 272 | for (i = 0; i < dev->lif_cfg.aq_count; i++) { | ||
| 273 | aq = dev->aq_vec[i]; | ||
| 274 | |||
| 275 | spin_lock_irqsave(&aq->lock, irqflags); | ||
| 276 | |||
| 277 | if (ionic_queue_empty(&aq->q)) | ||
| 278 | goto next_aq; | ||
| 279 | |||
| 280 | /* Reschedule if any queue has outstanding work */ | ||
| 281 | do_reschedule = true; | ||
| 282 | |||
| 283 | if (time_is_after_eq_jiffies(aq->stamp + IONIC_ADMIN_WARN)) | ||
| 284 | /* Warning threshold not met, nothing to do */ | ||
| 285 | goto next_aq; | ||
| 286 | |||
| 287 | /* See if polling now makes some progress */ | ||
| 288 | pos = aq->q.cons; | ||
| 289 | ionic_admin_poll_locked(aq); | ||
| 290 | if (pos != aq->q.cons) { | ||
| 291 | ibdev_dbg(&dev->ibdev, | ||
| 292 | "missed event for acq %d\n", aq->cqid); | ||
| 293 | goto next_aq; | ||
| 294 | } | ||
| 295 | |||
| 296 | if (time_is_after_eq_jiffies(aq->stamp + | ||
| 297 | IONIC_ADMIN_TIMEOUT)) { | ||
| 298 | /* Timeout threshold not met */ | ||
| 299 | ibdev_dbg(&dev->ibdev, "no progress after %ums\n", | ||
| 300 | (u32)jiffies_to_msecs(jiffies - aq->stamp)); | ||
| 301 | goto next_aq; | ||
| 302 | } | ||
| 303 | |||
| 304 | /* Queue timed out */ | ||
| 305 | bad_aq = aq; | ||
| 306 | do_reset = true; | ||
| 307 | next_aq: | ||
| 308 | spin_unlock_irqrestore(&aq->lock, irqflags); | ||
| 309 | } | ||
| 310 | |||
| 311 | if (do_reset) | ||
| 312 | /* Reset RDMA lif on a timeout */ | ||
| 313 | ionic_admin_timedout(bad_aq); | ||
| 314 | else if (do_reschedule) | ||
| 315 | /* Try to poll again later */ | ||
| 316 | ionic_admin_reset_dwork(dev); | ||
| 317 | } | ||
| 318 | |||
| 319 | static void ionic_admin_work(struct work_struct *ws) | ||
| 320 | { | ||
| 321 | struct ionic_aq *aq = container_of(ws, struct ionic_aq, work); | ||
| 322 | unsigned long irqflags; | ||
| 323 | |||
| 324 | spin_lock_irqsave(&aq->lock, irqflags); | ||
| 325 | ionic_admin_poll_locked(aq); | ||
| 326 | spin_unlock_irqrestore(&aq->lock, irqflags); | ||
| 327 | } | ||
| 328 | |||
| 329 | static void ionic_admin_post_aq(struct ionic_aq *aq, struct ionic_admin_wr *wr) | ||
| 330 | { | ||
| 331 | unsigned long irqflags; | ||
| 332 | bool poll; | ||
| 333 | |||
| 334 | wr->status = IONIC_ADMIN_POSTED; | ||
| 335 | wr->aq = aq; | ||
| 336 | |||
| 337 | spin_lock_irqsave(&aq->lock, irqflags); | ||
| 338 | poll = list_empty(&aq->wr_post); | ||
| 339 | list_add(&wr->aq_ent, &aq->wr_post); | ||
| 340 | if (poll) | ||
| 341 | ionic_admin_poll_locked(aq); | ||
| 342 | spin_unlock_irqrestore(&aq->lock, irqflags); | ||
| 343 | } | ||
| 344 | |||
| 345 | void ionic_admin_post(struct ionic_ibdev *dev, struct ionic_admin_wr *wr) | ||
| 346 | { | ||
| 347 | int aq_idx; | ||
| 348 | |||
| 349 | /* Use cpu id for the adminq selection */ | ||
| 350 | aq_idx = raw_smp_processor_id() % dev->lif_cfg.aq_count; | ||
| 351 | ionic_admin_post_aq(dev->aq_vec[aq_idx], wr); | ||
| 352 | } | ||
| 353 | |||
| 354 | static void ionic_admin_cancel(struct ionic_admin_wr *wr) | ||
| 355 | { | ||
| 356 | struct ionic_aq *aq = wr->aq; | ||
| 357 | unsigned long irqflags; | ||
| 358 | |||
| 359 | spin_lock_irqsave(&aq->lock, irqflags); | ||
| 360 | |||
| 361 | if (!list_empty(&wr->aq_ent)) { | ||
| 362 | list_del(&wr->aq_ent); | ||
| 363 | if (wr->status != IONIC_ADMIN_POSTED) | ||
| 364 | aq->q_wr[wr->status].wr = NULL; | ||
| 365 | } | ||
| 366 | |||
| 367 | spin_unlock_irqrestore(&aq->lock, irqflags); | ||
| 368 | } | ||
| 369 | |||
| 370 | static int ionic_admin_busy_wait(struct ionic_admin_wr *wr) | ||
| 371 | { | ||
| 372 | struct ionic_aq *aq = wr->aq; | ||
| 373 | unsigned long irqflags; | ||
| 374 | int try_i; | ||
| 375 | |||
| 376 | for (try_i = 0; try_i < IONIC_ADMIN_BUSY_RETRY_COUNT; ++try_i) { | ||
| 377 | if (completion_done(&wr->work)) | ||
| 378 | return 0; | ||
| 379 | |||
| 380 | mdelay(IONIC_ADMIN_BUSY_RETRY_MS); | ||
| 381 | |||
| 382 | spin_lock_irqsave(&aq->lock, irqflags); | ||
| 383 | ionic_admin_poll_locked(aq); | ||
| 384 | spin_unlock_irqrestore(&aq->lock, irqflags); | ||
| 385 | } | ||
| 386 | |||
| 387 | /* | ||
| 388 | * we timed out. Initiate RDMA LIF reset and indicate | ||
| 389 | * error to caller. | ||
| 390 | */ | ||
| 391 | ionic_admin_timedout(aq); | ||
| 392 | return -ETIMEDOUT; | ||
| 393 | } | ||
| 394 | |||
| 395 | int ionic_admin_wait(struct ionic_ibdev *dev, struct ionic_admin_wr *wr, | ||
| 396 | enum ionic_admin_flags flags) | ||
| 397 | { | ||
| 398 | int rc, timo; | ||
| 399 | |||
| 400 | if (flags & IONIC_ADMIN_F_BUSYWAIT) { | ||
| 401 | /* Spin */ | ||
| 402 | rc = ionic_admin_busy_wait(wr); | ||
| 403 | } else if (flags & IONIC_ADMIN_F_INTERRUPT) { | ||
| 404 | /* | ||
| 405 | * Interruptible sleep, 1s timeout | ||
| 406 | * This is used for commands which are safe for the caller | ||
| 407 | * to clean up without killing and resetting the adminq. | ||
| 408 | */ | ||
| 409 | timo = wait_for_completion_interruptible_timeout(&wr->work, | ||
| 410 | HZ); | ||
| 411 | if (timo > 0) | ||
| 412 | rc = 0; | ||
| 413 | else if (timo == 0) | ||
| 414 | rc = -ETIMEDOUT; | ||
| 415 | else | ||
| 416 | rc = timo; | ||
| 417 | } else { | ||
| 418 | /* | ||
| 419 | * Uninterruptible sleep | ||
| 420 | * This is used for commands which are NOT safe for the | ||
| 421 | * caller to clean up. Cleanup must be handled by the | ||
| 422 | * adminq kill and reset process so that host memory is | ||
| 423 | * not corrupted by the device. | ||
| 424 | */ | ||
| 425 | wait_for_completion(&wr->work); | ||
| 426 | rc = 0; | ||
| 427 | } | ||
| 428 | |||
| 429 | if (rc) { | ||
| 430 | ibdev_warn(&dev->ibdev, "wait status %d\n", rc); | ||
| 431 | ionic_admin_cancel(wr); | ||
| 432 | } else if (wr->status == IONIC_ADMIN_KILLED) { | ||
| 433 | ibdev_dbg(&dev->ibdev, "admin killed\n"); | ||
| 434 | |||
| 435 | /* No error if admin already killed during teardown */ | ||
| 436 | rc = (flags & IONIC_ADMIN_F_TEARDOWN) ? 0 : -ENODEV; | ||
| 437 | } else if (ionic_v1_cqe_error(&wr->cqe)) { | ||
| 438 | ibdev_warn(&dev->ibdev, "opcode %u error %u\n", | ||
| 439 | wr->wqe.op, | ||
| 440 | be32_to_cpu(wr->cqe.status_length)); | ||
| 441 | rc = -EINVAL; | ||
| 442 | } | ||
| 443 | return rc; | ||
| 444 | } | ||
| 445 | |||
| 446 | static int ionic_rdma_devcmd(struct ionic_ibdev *dev, | ||
| 447 | struct ionic_admin_ctx *admin) | ||
| 448 | { | ||
| 449 | int rc; | ||
| 450 | |||
| 451 | rc = ionic_adminq_post_wait(dev->lif_cfg.lif, admin); | ||
| 452 | if (rc) | ||
| 453 | return rc; | ||
| 454 | |||
| 455 | return ionic_error_to_errno(admin->comp.comp.status); | ||
| 456 | } | ||
| 457 | |||
| 458 | int ionic_rdma_reset_devcmd(struct ionic_ibdev *dev) | ||
| 459 | { | ||
| 460 | struct ionic_admin_ctx admin = { | ||
| 461 | .work = COMPLETION_INITIALIZER_ONSTACK(admin.work), | ||
| 462 | .cmd.rdma_reset = { | ||
| 463 | .opcode = IONIC_CMD_RDMA_RESET_LIF, | ||
| 464 | .lif_index = cpu_to_le16(dev->lif_cfg.lif_index), | ||
| 465 | }, | ||
| 466 | }; | ||
| 467 | |||
| 468 | return ionic_rdma_devcmd(dev, &admin); | ||
| 469 | } | ||
| 470 | |||
| 471 | static int ionic_rdma_queue_devcmd(struct ionic_ibdev *dev, | ||
| 472 | struct ionic_queue *q, | ||
| 473 | u32 qid, u32 cid, u16 opcode) | ||
| 474 | { | ||
| 475 | struct ionic_admin_ctx admin = { | ||
| 476 | .work = COMPLETION_INITIALIZER_ONSTACK(admin.work), | ||
| 477 | .cmd.rdma_queue = { | ||
| 478 | .opcode = opcode, | ||
| 479 | .lif_index = cpu_to_le16(dev->lif_cfg.lif_index), | ||
| 480 | .qid_ver = cpu_to_le32(qid), | ||
| 481 | .cid = cpu_to_le32(cid), | ||
| 482 | .dbid = cpu_to_le16(dev->lif_cfg.dbid), | ||
| 483 | .depth_log2 = q->depth_log2, | ||
| 484 | .stride_log2 = q->stride_log2, | ||
| 485 | .dma_addr = cpu_to_le64(q->dma), | ||
| 486 | }, | ||
| 487 | }; | ||
| 488 | |||
| 489 | return ionic_rdma_devcmd(dev, &admin); | ||
| 490 | } | ||
| 491 | |||
| 492 | static void ionic_rdma_admincq_comp(struct ib_cq *ibcq, void *cq_context) | ||
| 493 | { | ||
| 494 | struct ionic_aq *aq = cq_context; | ||
| 495 | unsigned long irqflags; | ||
| 496 | |||
| 497 | spin_lock_irqsave(&aq->lock, irqflags); | ||
| 498 | aq->armed = false; | ||
| 499 | if (atomic_read(&aq->admin_state) < IONIC_ADMIN_KILLED) | ||
| 500 | queue_work(ionic_evt_workq, &aq->work); | ||
| 501 | spin_unlock_irqrestore(&aq->lock, irqflags); | ||
| 502 | } | ||
| 503 | |||
| 504 | static void ionic_rdma_admincq_event(struct ib_event *event, void *cq_context) | ||
| 505 | { | ||
| 506 | struct ionic_aq *aq = cq_context; | ||
| 507 | |||
| 508 | ibdev_err(&aq->dev->ibdev, "admincq event %d\n", event->event); | ||
| 509 | } | ||
| 510 | |||
| 511 | static struct ionic_vcq *ionic_create_rdma_admincq(struct ionic_ibdev *dev, | ||
| 512 | int comp_vector) | ||
| 513 | { | ||
| 514 | struct ib_cq_init_attr attr = { | ||
| 515 | .cqe = IONIC_AQ_DEPTH, | ||
| 516 | .comp_vector = comp_vector, | ||
| 517 | }; | ||
| 518 | struct ionic_tbl_buf buf = {}; | ||
| 519 | struct ionic_vcq *vcq; | ||
| 520 | struct ionic_cq *cq; | ||
| 521 | int rc; | ||
| 522 | |||
| 523 | vcq = kzalloc(sizeof(*vcq), GFP_KERNEL); | ||
| 524 | if (!vcq) | ||
| 525 | return ERR_PTR(-ENOMEM); | ||
| 526 | |||
| 527 | vcq->ibcq.device = &dev->ibdev; | ||
| 528 | vcq->ibcq.comp_handler = ionic_rdma_admincq_comp; | ||
| 529 | vcq->ibcq.event_handler = ionic_rdma_admincq_event; | ||
| 530 | atomic_set(&vcq->ibcq.usecnt, 0); | ||
| 531 | |||
| 532 | vcq->udma_mask = 1; | ||
| 533 | cq = &vcq->cq[0]; | ||
| 534 | |||
| 535 | rc = ionic_create_cq_common(vcq, &buf, &attr, NULL, NULL, | ||
| 536 | NULL, NULL, 0); | ||
| 537 | if (rc) | ||
| 538 | goto err_init; | ||
| 539 | |||
| 540 | rc = ionic_rdma_queue_devcmd(dev, &cq->q, cq->cqid, cq->eqid, | ||
| 541 | IONIC_CMD_RDMA_CREATE_CQ); | ||
| 542 | if (rc) | ||
| 543 | goto err_cmd; | ||
| 544 | |||
| 545 | return vcq; | ||
| 546 | |||
| 547 | err_cmd: | ||
| 548 | ionic_destroy_cq_common(dev, cq); | ||
| 549 | err_init: | ||
| 550 | kfree(vcq); | ||
| 551 | |||
| 552 | return ERR_PTR(rc); | ||
| 553 | } | ||
| 554 | |||
| 555 | static struct ionic_aq *__ionic_create_rdma_adminq(struct ionic_ibdev *dev, | ||
| 556 | u32 aqid, u32 cqid) | ||
| 557 | { | ||
| 558 | struct ionic_aq *aq; | ||
| 559 | int rc; | ||
| 560 | |||
| 561 | aq = kzalloc(sizeof(*aq), GFP_KERNEL); | ||
| 562 | if (!aq) | ||
| 563 | return ERR_PTR(-ENOMEM); | ||
| 564 | |||
| 565 | atomic_set(&aq->admin_state, IONIC_ADMIN_KILLED); | ||
| 566 | aq->dev = dev; | ||
| 567 | aq->aqid = aqid; | ||
| 568 | aq->cqid = cqid; | ||
| 569 | spin_lock_init(&aq->lock); | ||
| 570 | |||
| 571 | rc = ionic_queue_init(&aq->q, dev->lif_cfg.hwdev, IONIC_EQ_DEPTH, | ||
| 572 | ADMIN_WQE_STRIDE); | ||
| 573 | if (rc) | ||
| 574 | goto err_q; | ||
| 575 | |||
| 576 | ionic_queue_dbell_init(&aq->q, aq->aqid); | ||
| 577 | |||
| 578 | aq->q_wr = kcalloc((u32)aq->q.mask + 1, sizeof(*aq->q_wr), GFP_KERNEL); | ||
| 579 | if (!aq->q_wr) { | ||
| 580 | rc = -ENOMEM; | ||
| 581 | goto err_wr; | ||
| 582 | } | ||
| 583 | |||
| 584 | INIT_LIST_HEAD(&aq->wr_prod); | ||
| 585 | INIT_LIST_HEAD(&aq->wr_post); | ||
| 586 | |||
| 587 | INIT_WORK(&aq->work, ionic_admin_work); | ||
| 588 | aq->armed = false; | ||
| 589 | |||
| 590 | return aq; | ||
| 591 | |||
| 592 | err_wr: | ||
| 593 | ionic_queue_destroy(&aq->q, dev->lif_cfg.hwdev); | ||
| 594 | err_q: | ||
| 595 | kfree(aq); | ||
| 596 | |||
| 597 | return ERR_PTR(rc); | ||
| 598 | } | ||
| 599 | |||
| 600 | static void __ionic_destroy_rdma_adminq(struct ionic_ibdev *dev, | ||
| 601 | struct ionic_aq *aq) | ||
| 602 | { | ||
| 603 | kfree(aq->q_wr); | ||
| 604 | ionic_queue_destroy(&aq->q, dev->lif_cfg.hwdev); | ||
| 605 | kfree(aq); | ||
| 606 | } | ||
| 607 | |||
| 608 | static struct ionic_aq *ionic_create_rdma_adminq(struct ionic_ibdev *dev, | ||
| 609 | u32 aqid, u32 cqid) | ||
| 610 | { | ||
| 611 | struct ionic_aq *aq; | ||
| 612 | int rc; | ||
| 613 | |||
| 614 | aq = __ionic_create_rdma_adminq(dev, aqid, cqid); | ||
| 615 | if (IS_ERR(aq)) | ||
| 616 | return aq; | ||
| 617 | |||
| 618 | rc = ionic_rdma_queue_devcmd(dev, &aq->q, aq->aqid, aq->cqid, | ||
| 619 | IONIC_CMD_RDMA_CREATE_ADMINQ); | ||
| 620 | if (rc) | ||
| 621 | goto err_cmd; | ||
| 622 | |||
| 623 | return aq; | ||
| 624 | |||
| 625 | err_cmd: | ||
| 626 | __ionic_destroy_rdma_adminq(dev, aq); | ||
| 627 | |||
| 628 | return ERR_PTR(rc); | ||
| 629 | } | ||
| 630 | |||
| 631 | static void ionic_flush_qs(struct ionic_ibdev *dev) | ||
| 632 | { | ||
| 633 | struct ionic_qp *qp, *qp_tmp; | ||
| 634 | struct ionic_cq *cq, *cq_tmp; | ||
| 635 | LIST_HEAD(flush_list); | ||
| 636 | unsigned long index; | ||
| 637 | |||
| 638 | WARN_ON(!irqs_disabled()); | ||
| 639 | |||
| 640 | /* Flush qp send and recv */ | ||
| 641 | xa_lock(&dev->qp_tbl); | ||
| 642 | xa_for_each(&dev->qp_tbl, index, qp) { | ||
| 643 | kref_get(&qp->qp_kref); | ||
| 644 | list_add_tail(&qp->ibkill_flush_ent, &flush_list); | ||
| 645 | } | ||
| 646 | xa_unlock(&dev->qp_tbl); | ||
| 647 | |||
| 648 | list_for_each_entry_safe(qp, qp_tmp, &flush_list, ibkill_flush_ent) { | ||
| 649 | ionic_flush_qp(dev, qp); | ||
| 650 | kref_put(&qp->qp_kref, ionic_qp_complete); | ||
| 651 | list_del(&qp->ibkill_flush_ent); | ||
| 652 | } | ||
| 653 | |||
| 654 | /* Notify completions */ | ||
| 655 | xa_lock(&dev->cq_tbl); | ||
| 656 | xa_for_each(&dev->cq_tbl, index, cq) { | ||
| 657 | kref_get(&cq->cq_kref); | ||
| 658 | list_add_tail(&cq->ibkill_flush_ent, &flush_list); | ||
| 659 | } | ||
| 660 | xa_unlock(&dev->cq_tbl); | ||
| 661 | |||
| 662 | list_for_each_entry_safe(cq, cq_tmp, &flush_list, ibkill_flush_ent) { | ||
| 663 | ionic_notify_flush_cq(cq); | ||
| 664 | kref_put(&cq->cq_kref, ionic_cq_complete); | ||
| 665 | list_del(&cq->ibkill_flush_ent); | ||
| 666 | } | ||
| 667 | } | ||
| 668 | |||
| 669 | static void ionic_kill_ibdev(struct ionic_ibdev *dev, bool fatal_path) | ||
| 670 | { | ||
| 671 | unsigned long irqflags; | ||
| 672 | bool do_flush = false; | ||
| 673 | int i; | ||
| 674 | |||
| 675 | /* Mark AQs for drain and flush the QPs while irq is disabled */ | ||
| 676 | local_irq_save(irqflags); | ||
| 677 | |||
| 678 | /* Mark the admin queue, flushing at most once */ | ||
| 679 | for (i = 0; i < dev->lif_cfg.aq_count; i++) { | ||
| 680 | struct ionic_aq *aq = dev->aq_vec[i]; | ||
| 681 | |||
| 682 | spin_lock(&aq->lock); | ||
| 683 | if (atomic_read(&aq->admin_state) != IONIC_ADMIN_KILLED) { | ||
| 684 | atomic_set(&aq->admin_state, IONIC_ADMIN_KILLED); | ||
| 685 | /* Flush incomplete admin commands */ | ||
| 686 | ionic_admin_poll_locked(aq); | ||
| 687 | do_flush = true; | ||
| 688 | } | ||
| 689 | spin_unlock(&aq->lock); | ||
| 690 | } | ||
| 691 | |||
| 692 | if (do_flush) | ||
| 693 | ionic_flush_qs(dev); | ||
| 694 | |||
| 695 | local_irq_restore(irqflags); | ||
| 696 | |||
| 697 | /* Post a fatal event if requested */ | ||
| 698 | if (fatal_path) { | ||
| 699 | struct ib_event ev; | ||
| 700 | |||
| 701 | ev.device = &dev->ibdev; | ||
| 702 | ev.element.port_num = 1; | ||
| 703 | ev.event = IB_EVENT_DEVICE_FATAL; | ||
| 704 | |||
| 705 | ib_dispatch_event(&ev); | ||
| 706 | } | ||
| 707 | |||
| 708 | atomic_set(&dev->admin_state, IONIC_ADMIN_KILLED); | ||
| 709 | } | ||
| 710 | |||
| 711 | void ionic_kill_rdma_admin(struct ionic_ibdev *dev, bool fatal_path) | ||
| 712 | { | ||
| 713 | enum ionic_admin_state old_state; | ||
| 714 | unsigned long irqflags = 0; | ||
| 715 | int i, rc; | ||
| 716 | |||
| 717 | if (!dev->aq_vec) | ||
| 718 | return; | ||
| 719 | |||
| 720 | /* | ||
| 721 | * Admin queues are transitioned from active to paused to killed state. | ||
| 722 | * When in paused state, no new commands are issued to the device, | ||
| 723 | * nor are any completed locally. After resetting the lif, it will be | ||
| 724 | * safe to resume the rdma admin queues in the killed state. Commands | ||
| 725 | * will not be issued to the device, but will complete locally with status | ||
| 726 | * IONIC_ADMIN_KILLED. Handling completion will ensure that creating or | ||
| 727 | * modifying resources fails, but destroying resources succeeds. | ||
| 728 | * If there was a failure resetting the lif using this strategy, | ||
| 729 | * then the state of the device is unknown. | ||
| 730 | */ | ||
| 731 | old_state = atomic_cmpxchg(&dev->admin_state, IONIC_ADMIN_ACTIVE, | ||
| 732 | IONIC_ADMIN_PAUSED); | ||
| 733 | if (old_state != IONIC_ADMIN_ACTIVE) | ||
| 734 | return; | ||
| 735 | |||
| 736 | /* Pause all the AQs */ | ||
| 737 | local_irq_save(irqflags); | ||
| 738 | for (i = 0; i < dev->lif_cfg.aq_count; i++) { | ||
| 739 | struct ionic_aq *aq = dev->aq_vec[i]; | ||
| 740 | |||
| 741 | spin_lock(&aq->lock); | ||
| 742 | /* pause rdma admin queues to reset lif */ | ||
| 743 | if (atomic_read(&aq->admin_state) == IONIC_ADMIN_ACTIVE) | ||
| 744 | atomic_set(&aq->admin_state, IONIC_ADMIN_PAUSED); | ||
| 745 | spin_unlock(&aq->lock); | ||
| 746 | } | ||
| 747 | local_irq_restore(irqflags); | ||
| 748 | |||
| 749 | rc = ionic_rdma_reset_devcmd(dev); | ||
| 750 | if (unlikely(rc)) { | ||
| 751 | ibdev_err(&dev->ibdev, "failed to reset rdma %d\n", rc); | ||
| 752 | ionic_request_rdma_reset(dev->lif_cfg.lif); | ||
| 753 | } | ||
| 754 | |||
| 755 | ionic_kill_ibdev(dev, fatal_path); | ||
| 756 | } | ||
| 757 | |||
| 758 | static void ionic_reset_work(struct work_struct *ws) | ||
| 759 | { | ||
| 760 | struct ionic_ibdev *dev = | ||
| 761 | container_of(ws, struct ionic_ibdev, reset_work); | ||
| 762 | |||
| 763 | ionic_kill_rdma_admin(dev, true); | ||
| 764 | } | ||
| 765 | |||
| 766 | static bool ionic_next_eqe(struct ionic_eq *eq, struct ionic_v1_eqe *eqe) | ||
| 767 | { | ||
| 768 | struct ionic_v1_eqe *qeqe; | ||
| 769 | bool color; | ||
| 770 | |||
| 771 | qeqe = ionic_queue_at_prod(&eq->q); | ||
| 772 | color = ionic_v1_eqe_color(qeqe); | ||
| 773 | |||
| 774 | /* cons is color for eq */ | ||
| 775 | if (eq->q.cons != color) | ||
| 776 | return false; | ||
| 777 | |||
| 778 | /* Prevent out-of-order reads of the EQE */ | ||
| 779 | dma_rmb(); | ||
| 780 | |||
| 781 | ibdev_dbg(&eq->dev->ibdev, "poll eq prod %u\n", eq->q.prod); | ||
| 782 | print_hex_dump_debug("eqe ", DUMP_PREFIX_OFFSET, 16, 1, | ||
| 783 | qeqe, BIT(eq->q.stride_log2), true); | ||
| 784 | *eqe = *qeqe; | ||
| 785 | |||
| 786 | return true; | ||
| 787 | } | ||
| 788 | |||
| 789 | static void ionic_cq_event(struct ionic_ibdev *dev, u32 cqid, u8 code) | ||
| 790 | { | ||
| 791 | unsigned long irqflags; | ||
| 792 | struct ib_event ibev; | ||
| 793 | struct ionic_cq *cq; | ||
| 794 | |||
| 795 | xa_lock_irqsave(&dev->cq_tbl, irqflags); | ||
| 796 | cq = xa_load(&dev->cq_tbl, cqid); | ||
| 797 | if (cq) | ||
| 798 | kref_get(&cq->cq_kref); | ||
| 799 | xa_unlock_irqrestore(&dev->cq_tbl, irqflags); | ||
| 800 | |||
| 801 | if (!cq) { | ||
| 802 | ibdev_dbg(&dev->ibdev, | ||
| 803 | "missing cqid %#x code %u\n", cqid, code); | ||
| 804 | return; | ||
| 805 | } | ||
| 806 | |||
| 807 | switch (code) { | ||
| 808 | case IONIC_V1_EQE_CQ_NOTIFY: | ||
| 809 | if (cq->vcq->ibcq.comp_handler) | ||
| 810 | cq->vcq->ibcq.comp_handler(&cq->vcq->ibcq, | ||
| 811 | cq->vcq->ibcq.cq_context); | ||
| 812 | break; | ||
| 813 | |||
| 814 | case IONIC_V1_EQE_CQ_ERR: | ||
| 815 | if (cq->vcq->ibcq.event_handler) { | ||
| 816 | ibev.event = IB_EVENT_CQ_ERR; | ||
| 817 | ibev.device = &dev->ibdev; | ||
| 818 | ibev.element.cq = &cq->vcq->ibcq; | ||
| 819 | |||
| 820 | cq->vcq->ibcq.event_handler(&ibev, | ||
| 821 | cq->vcq->ibcq.cq_context); | ||
| 822 | } | ||
| 823 | break; | ||
| 824 | |||
| 825 | default: | ||
| 826 | ibdev_dbg(&dev->ibdev, | ||
| 827 | "unrecognized cqid %#x code %u\n", cqid, code); | ||
| 828 | break; | ||
| 829 | } | ||
| 830 | |||
| 831 | kref_put(&cq->cq_kref, ionic_cq_complete); | ||
| 832 | } | ||
| 833 | |||
| 834 | static void ionic_qp_event(struct ionic_ibdev *dev, u32 qpid, u8 code) | ||
| 835 | { | ||
| 836 | unsigned long irqflags; | ||
| 837 | struct ib_event ibev; | ||
| 838 | struct ionic_qp *qp; | ||
| 839 | |||
| 840 | xa_lock_irqsave(&dev->qp_tbl, irqflags); | ||
| 841 | qp = xa_load(&dev->qp_tbl, qpid); | ||
| 842 | if (qp) | ||
| 843 | kref_get(&qp->qp_kref); | ||
| 844 | xa_unlock_irqrestore(&dev->qp_tbl, irqflags); | ||
| 845 | |||
| 846 | if (!qp) { | ||
| 847 | ibdev_dbg(&dev->ibdev, | ||
| 848 | "missing qpid %#x code %u\n", qpid, code); | ||
| 849 | return; | ||
| 850 | } | ||
| 851 | |||
| 852 | ibev.device = &dev->ibdev; | ||
| 853 | ibev.element.qp = &qp->ibqp; | ||
| 854 | |||
| 855 | switch (code) { | ||
| 856 | case IONIC_V1_EQE_SQ_DRAIN: | ||
| 857 | ibev.event = IB_EVENT_SQ_DRAINED; | ||
| 858 | break; | ||
| 859 | |||
| 860 | case IONIC_V1_EQE_QP_COMM_EST: | ||
| 861 | ibev.event = IB_EVENT_COMM_EST; | ||
| 862 | break; | ||
| 863 | |||
| 864 | case IONIC_V1_EQE_QP_LAST_WQE: | ||
| 865 | ibev.event = IB_EVENT_QP_LAST_WQE_REACHED; | ||
| 866 | break; | ||
| 867 | |||
| 868 | case IONIC_V1_EQE_QP_ERR: | ||
| 869 | ibev.event = IB_EVENT_QP_FATAL; | ||
| 870 | break; | ||
| 871 | |||
| 872 | case IONIC_V1_EQE_QP_ERR_REQUEST: | ||
| 873 | ibev.event = IB_EVENT_QP_REQ_ERR; | ||
| 874 | break; | ||
| 875 | |||
| 876 | case IONIC_V1_EQE_QP_ERR_ACCESS: | ||
| 877 | ibev.event = IB_EVENT_QP_ACCESS_ERR; | ||
| 878 | break; | ||
| 879 | |||
| 880 | default: | ||
| 881 | ibdev_dbg(&dev->ibdev, | ||
| 882 | "unrecognized qpid %#x code %u\n", qpid, code); | ||
| 883 | goto out; | ||
| 884 | } | ||
| 885 | |||
| 886 | if (qp->ibqp.event_handler) | ||
| 887 | qp->ibqp.event_handler(&ibev, qp->ibqp.qp_context); | ||
| 888 | |||
| 889 | out: | ||
| 890 | kref_put(&qp->qp_kref, ionic_qp_complete); | ||
| 891 | } | ||
| 892 | |||
| 893 | static u16 ionic_poll_eq(struct ionic_eq *eq, u16 budget) | ||
| 894 | { | ||
| 895 | struct ionic_ibdev *dev = eq->dev; | ||
| 896 | struct ionic_v1_eqe eqe; | ||
| 897 | u16 npolled = 0; | ||
| 898 | u8 type, code; | ||
| 899 | u32 evt, qid; | ||
| 900 | |||
| 901 | while (npolled < budget) { | ||
| 902 | if (!ionic_next_eqe(eq, &eqe)) | ||
| 903 | break; | ||
| 904 | |||
| 905 | ionic_queue_produce(&eq->q); | ||
| 906 | |||
| 907 | /* cons is color for eq */ | ||
| 908 | eq->q.cons = ionic_color_wrap(eq->q.prod, eq->q.cons); | ||
| 909 | |||
| 910 | ++npolled; | ||
| 911 | |||
| 912 | evt = ionic_v1_eqe_evt(&eqe); | ||
| 913 | type = ionic_v1_eqe_evt_type(evt); | ||
| 914 | code = ionic_v1_eqe_evt_code(evt); | ||
| 915 | qid = ionic_v1_eqe_evt_qid(evt); | ||
| 916 | |||
| 917 | switch (type) { | ||
| 918 | case IONIC_V1_EQE_TYPE_CQ: | ||
| 919 | ionic_cq_event(dev, qid, code); | ||
| 920 | break; | ||
| 921 | |||
| 922 | case IONIC_V1_EQE_TYPE_QP: | ||
| 923 | ionic_qp_event(dev, qid, code); | ||
| 924 | break; | ||
| 925 | |||
| 926 | default: | ||
| 927 | ibdev_dbg(&dev->ibdev, | ||
| 928 | "unknown event %#x type %u\n", evt, type); | ||
| 929 | } | ||
| 930 | } | ||
| 931 | |||
| 932 | return npolled; | ||
| 933 | } | ||
| 934 | |||
| 935 | static void ionic_poll_eq_work(struct work_struct *work) | ||
| 936 | { | ||
| 937 | struct ionic_eq *eq = container_of(work, struct ionic_eq, work); | ||
| 938 | u32 npolled; | ||
| 939 | |||
| 940 | if (unlikely(!eq->enable) || WARN_ON(eq->armed)) | ||
| 941 | return; | ||
| 942 | |||
| 943 | npolled = ionic_poll_eq(eq, IONIC_EQ_WORK_BUDGET); | ||
| 944 | if (npolled == IONIC_EQ_WORK_BUDGET) { | ||
| 945 | ionic_intr_credits(eq->dev->lif_cfg.intr_ctrl, eq->intr, | ||
| 946 | npolled, 0); | ||
| 947 | queue_work(ionic_evt_workq, &eq->work); | ||
| 948 | } else { | ||
| 949 | xchg(&eq->armed, 1); | ||
| 950 | ionic_intr_credits(eq->dev->lif_cfg.intr_ctrl, eq->intr, | ||
| 951 | 0, IONIC_INTR_CRED_UNMASK); | ||
| 952 | } | ||
| 953 | } | ||
| 954 | |||
| 955 | static irqreturn_t ionic_poll_eq_isr(int irq, void *eqptr) | ||
| 956 | { | ||
| 957 | struct ionic_eq *eq = eqptr; | ||
| 958 | int was_armed; | ||
| 959 | u32 npolled; | ||
| 960 | |||
| 961 | was_armed = xchg(&eq->armed, 0); | ||
| 962 | |||
| 963 | if (unlikely(!eq->enable) || !was_armed) | ||
| 964 | return IRQ_HANDLED; | ||
| 965 | |||
| 966 | npolled = ionic_poll_eq(eq, IONIC_EQ_ISR_BUDGET); | ||
| 967 | if (npolled == IONIC_EQ_ISR_BUDGET) { | ||
| 968 | ionic_intr_credits(eq->dev->lif_cfg.intr_ctrl, eq->intr, | ||
| 969 | npolled, 0); | ||
| 970 | queue_work(ionic_evt_workq, &eq->work); | ||
| 971 | } else { | ||
| 972 | xchg(&eq->armed, 1); | ||
| 973 | ionic_intr_credits(eq->dev->lif_cfg.intr_ctrl, eq->intr, | ||
| 974 | 0, IONIC_INTR_CRED_UNMASK); | ||
| 975 | } | ||
| 976 | |||
| 977 | return IRQ_HANDLED; | ||
| 978 | } | ||
| 979 | |||
| 980 | static struct ionic_eq *ionic_create_eq(struct ionic_ibdev *dev, int eqid) | ||
| 981 | { | ||
| 982 | struct ionic_intr_info intr_obj = { }; | ||
| 983 | struct ionic_eq *eq; | ||
| 984 | int rc; | ||
| 985 | |||
| 986 | eq = kzalloc(sizeof(*eq), GFP_KERNEL); | ||
| 987 | if (!eq) | ||
| 988 | return ERR_PTR(-ENOMEM); | ||
| 989 | |||
| 990 | eq->dev = dev; | ||
| 991 | |||
| 992 | rc = ionic_queue_init(&eq->q, dev->lif_cfg.hwdev, IONIC_EQ_DEPTH, | ||
| 993 | sizeof(struct ionic_v1_eqe)); | ||
| 994 | if (rc) | ||
| 995 | goto err_q; | ||
| 996 | |||
| 997 | eq->eqid = eqid; | ||
| 998 | |||
| 999 | eq->armed = true; | ||
| 1000 | eq->enable = false; | ||
| 1001 | INIT_WORK(&eq->work, ionic_poll_eq_work); | ||
| 1002 | |||
| 1003 | rc = ionic_intr_alloc(dev->lif_cfg.lif, &intr_obj); | ||
| 1004 | if (rc < 0) | ||
| 1005 | goto err_intr; | ||
| 1006 | |||
| 1007 | eq->irq = intr_obj.vector; | ||
| 1008 | eq->intr = intr_obj.index; | ||
| 1009 | |||
| 1010 | ionic_queue_dbell_init(&eq->q, eq->eqid); | ||
| 1011 | |||
| 1012 | /* cons is color for eq */ | ||
| 1013 | eq->q.cons = true; | ||
| 1014 | |||
| 1015 | snprintf(eq->name, sizeof(eq->name), "%s-%d-%d-eq", | ||
| 1016 | "ionr", dev->lif_cfg.lif_index, eq->eqid); | ||
| 1017 | |||
| 1018 | ionic_intr_mask(dev->lif_cfg.intr_ctrl, eq->intr, IONIC_INTR_MASK_SET); | ||
| 1019 | ionic_intr_mask_assert(dev->lif_cfg.intr_ctrl, eq->intr, IONIC_INTR_MASK_SET); | ||
| 1020 | ionic_intr_coal_init(dev->lif_cfg.intr_ctrl, eq->intr, 0); | ||
| 1021 | ionic_intr_clean(dev->lif_cfg.intr_ctrl, eq->intr); | ||
| 1022 | |||
| 1023 | eq->enable = true; | ||
| 1024 | |||
| 1025 | rc = request_irq(eq->irq, ionic_poll_eq_isr, 0, eq->name, eq); | ||
| 1026 | if (rc) | ||
| 1027 | goto err_irq; | ||
| 1028 | |||
| 1029 | rc = ionic_rdma_queue_devcmd(dev, &eq->q, eq->eqid, eq->intr, | ||
| 1030 | IONIC_CMD_RDMA_CREATE_EQ); | ||
| 1031 | if (rc) | ||
| 1032 | goto err_cmd; | ||
| 1033 | |||
| 1034 | ionic_intr_mask(dev->lif_cfg.intr_ctrl, eq->intr, IONIC_INTR_MASK_CLEAR); | ||
| 1035 | |||
| 1036 | return eq; | ||
| 1037 | |||
| 1038 | err_cmd: | ||
| 1039 | eq->enable = false; | ||
| 1040 | free_irq(eq->irq, eq); | ||
| 1041 | flush_work(&eq->work); | ||
| 1042 | err_irq: | ||
| 1043 | ionic_intr_free(dev->lif_cfg.lif, eq->intr); | ||
| 1044 | err_intr: | ||
| 1045 | ionic_queue_destroy(&eq->q, dev->lif_cfg.hwdev); | ||
| 1046 | err_q: | ||
| 1047 | kfree(eq); | ||
| 1048 | |||
| 1049 | return ERR_PTR(rc); | ||
| 1050 | } | ||
| 1051 | |||
| 1052 | static void ionic_destroy_eq(struct ionic_eq *eq) | ||
| 1053 | { | ||
| 1054 | struct ionic_ibdev *dev = eq->dev; | ||
| 1055 | |||
| 1056 | eq->enable = false; | ||
| 1057 | free_irq(eq->irq, eq); | ||
| 1058 | flush_work(&eq->work); | ||
| 1059 | |||
| 1060 | ionic_intr_free(dev->lif_cfg.lif, eq->intr); | ||
| 1061 | ionic_queue_destroy(&eq->q, dev->lif_cfg.hwdev); | ||
| 1062 | kfree(eq); | ||
| 1063 | } | ||
| 1064 | |||
| 1065 | int ionic_create_rdma_admin(struct ionic_ibdev *dev) | ||
| 1066 | { | ||
| 1067 | int eq_i = 0, aq_i = 0, rc = 0; | ||
| 1068 | struct ionic_vcq *vcq; | ||
| 1069 | struct ionic_aq *aq; | ||
| 1070 | struct ionic_eq *eq; | ||
| 1071 | |||
| 1072 | dev->eq_vec = NULL; | ||
| 1073 | dev->aq_vec = NULL; | ||
| 1074 | |||
| 1075 | INIT_WORK(&dev->reset_work, ionic_reset_work); | ||
| 1076 | INIT_DELAYED_WORK(&dev->admin_dwork, ionic_admin_dwork); | ||
| 1077 | atomic_set(&dev->admin_state, IONIC_ADMIN_KILLED); | ||
| 1078 | |||
| 1079 | if (dev->lif_cfg.aq_count > IONIC_AQ_COUNT) { | ||
| 1080 | ibdev_dbg(&dev->ibdev, "limiting adminq count to %d\n", | ||
| 1081 | IONIC_AQ_COUNT); | ||
| 1082 | dev->lif_cfg.aq_count = IONIC_AQ_COUNT; | ||
| 1083 | } | ||
| 1084 | |||
| 1085 | if (dev->lif_cfg.eq_count > IONIC_EQ_COUNT) { | ||
| 1086 | dev_dbg(&dev->ibdev.dev, "limiting eventq count to %d\n", | ||
| 1087 | IONIC_EQ_COUNT); | ||
| 1088 | dev->lif_cfg.eq_count = IONIC_EQ_COUNT; | ||
| 1089 | } | ||
| 1090 | |||
| 1091 | /* need at least two eq and one aq */ | ||
| 1092 | if (dev->lif_cfg.eq_count < IONIC_EQ_COUNT_MIN || | ||
| 1093 | dev->lif_cfg.aq_count < IONIC_AQ_COUNT_MIN) { | ||
| 1094 | rc = -EINVAL; | ||
| 1095 | goto out; | ||
| 1096 | } | ||
| 1097 | |||
| 1098 | dev->eq_vec = kmalloc_array(dev->lif_cfg.eq_count, sizeof(*dev->eq_vec), | ||
| 1099 | GFP_KERNEL); | ||
| 1100 | if (!dev->eq_vec) { | ||
| 1101 | rc = -ENOMEM; | ||
| 1102 | goto out; | ||
| 1103 | } | ||
| 1104 | |||
| 1105 | for (eq_i = 0; eq_i < dev->lif_cfg.eq_count; ++eq_i) { | ||
| 1106 | eq = ionic_create_eq(dev, eq_i + dev->lif_cfg.eq_base); | ||
| 1107 | if (IS_ERR(eq)) { | ||
| 1108 | rc = PTR_ERR(eq); | ||
| 1109 | |||
| 1110 | if (eq_i < IONIC_EQ_COUNT_MIN) { | ||
| 1111 | ibdev_err(&dev->ibdev, | ||
| 1112 | "fail create eq %pe\n", eq); | ||
| 1113 | goto out; | ||
| 1114 | } | ||
| 1115 | |||
| 1116 | /* ok, just fewer eq than device supports */ | ||
| 1117 | ibdev_dbg(&dev->ibdev, "eq count %d want %d rc %pe\n", | ||
| 1118 | eq_i, dev->lif_cfg.eq_count, eq); | ||
| 1119 | |||
| 1120 | rc = 0; | ||
| 1121 | break; | ||
| 1122 | } | ||
| 1123 | |||
| 1124 | dev->eq_vec[eq_i] = eq; | ||
| 1125 | } | ||
| 1126 | |||
| 1127 | dev->lif_cfg.eq_count = eq_i; | ||
| 1128 | |||
| 1129 | dev->aq_vec = kmalloc_array(dev->lif_cfg.aq_count, sizeof(*dev->aq_vec), | ||
| 1130 | GFP_KERNEL); | ||
| 1131 | if (!dev->aq_vec) { | ||
| 1132 | rc = -ENOMEM; | ||
| 1133 | goto out; | ||
| 1134 | } | ||
| 1135 | |||
| 1136 | /* Create one CQ per AQ */ | ||
| 1137 | for (aq_i = 0; aq_i < dev->lif_cfg.aq_count; ++aq_i) { | ||
| 1138 | vcq = ionic_create_rdma_admincq(dev, aq_i % eq_i); | ||
| 1139 | if (IS_ERR(vcq)) { | ||
| 1140 | rc = PTR_ERR(vcq); | ||
| 1141 | |||
| 1142 | if (!aq_i) { | ||
| 1143 | ibdev_err(&dev->ibdev, | ||
| 1144 | "failed to create acq %pe\n", vcq); | ||
| 1145 | goto out; | ||
| 1146 | } | ||
| 1147 | |||
| 1148 | /* ok, just fewer adminq than device supports */ | ||
| 1149 | ibdev_dbg(&dev->ibdev, "acq count %d want %d rc %pe\n", | ||
| 1150 | aq_i, dev->lif_cfg.aq_count, vcq); | ||
| 1151 | break; | ||
| 1152 | } | ||
| 1153 | |||
| 1154 | aq = ionic_create_rdma_adminq(dev, aq_i + dev->lif_cfg.aq_base, | ||
| 1155 | vcq->cq[0].cqid); | ||
| 1156 | if (IS_ERR(aq)) { | ||
| 1157 | /* Clean up the dangling CQ */ | ||
| 1158 | ionic_destroy_cq_common(dev, &vcq->cq[0]); | ||
| 1159 | kfree(vcq); | ||
| 1160 | |||
| 1161 | rc = PTR_ERR(aq); | ||
| 1162 | |||
| 1163 | if (!aq_i) { | ||
| 1164 | ibdev_err(&dev->ibdev, | ||
| 1165 | "failed to create aq %pe\n", aq); | ||
| 1166 | goto out; | ||
| 1167 | } | ||
| 1168 | |||
| 1169 | /* ok, just fewer adminq than device supports */ | ||
| 1170 | ibdev_dbg(&dev->ibdev, "aq count %d want %d rc %pe\n", | ||
| 1171 | aq_i, dev->lif_cfg.aq_count, aq); | ||
| 1172 | break; | ||
| 1173 | } | ||
| 1174 | |||
| 1175 | vcq->ibcq.cq_context = aq; | ||
| 1176 | aq->vcq = vcq; | ||
| 1177 | |||
| 1178 | atomic_set(&aq->admin_state, IONIC_ADMIN_ACTIVE); | ||
| 1179 | dev->aq_vec[aq_i] = aq; | ||
| 1180 | } | ||
| 1181 | |||
| 1182 | atomic_set(&dev->admin_state, IONIC_ADMIN_ACTIVE); | ||
| 1183 | out: | ||
| 1184 | dev->lif_cfg.eq_count = eq_i; | ||
| 1185 | dev->lif_cfg.aq_count = aq_i; | ||
| 1186 | |||
| 1187 | return rc; | ||
| 1188 | } | ||
| 1189 | |||
| 1190 | void ionic_destroy_rdma_admin(struct ionic_ibdev *dev) | ||
| 1191 | { | ||
| 1192 | struct ionic_vcq *vcq; | ||
| 1193 | struct ionic_aq *aq; | ||
| 1194 | struct ionic_eq *eq; | ||
| 1195 | |||
| 1196 | /* | ||
| 1197 | * Killing the admin before destroy makes sure all admin and | ||
| 1198 | * completions are flushed. admin_state = IONIC_ADMIN_KILLED | ||
| 1199 | * stops queueing up further works. | ||
| 1200 | */ | ||
| 1201 | cancel_delayed_work_sync(&dev->admin_dwork); | ||
| 1202 | cancel_work_sync(&dev->reset_work); | ||
| 1203 | |||
| 1204 | if (dev->aq_vec) { | ||
| 1205 | while (dev->lif_cfg.aq_count > 0) { | ||
| 1206 | aq = dev->aq_vec[--dev->lif_cfg.aq_count]; | ||
| 1207 | vcq = aq->vcq; | ||
| 1208 | |||
| 1209 | cancel_work_sync(&aq->work); | ||
| 1210 | |||
| 1211 | __ionic_destroy_rdma_adminq(dev, aq); | ||
| 1212 | if (vcq) { | ||
| 1213 | ionic_destroy_cq_common(dev, &vcq->cq[0]); | ||
| 1214 | kfree(vcq); | ||
| 1215 | } | ||
| 1216 | } | ||
| 1217 | |||
| 1218 | kfree(dev->aq_vec); | ||
| 1219 | } | ||
| 1220 | |||
| 1221 | if (dev->eq_vec) { | ||
| 1222 | while (dev->lif_cfg.eq_count > 0) { | ||
| 1223 | eq = dev->eq_vec[--dev->lif_cfg.eq_count]; | ||
| 1224 | ionic_destroy_eq(eq); | ||
| 1225 | } | ||
| 1226 | |||
| 1227 | kfree(dev->eq_vec); | ||
| 1228 | } | ||
| 1229 | } | ||
| @@ -0,0 +1,2679 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #include <linux/module.h> | ||
| 5 | #include <linux/printk.h> | ||
| 6 | #include <rdma/ib_addr.h> | ||
| 7 | #include <rdma/ib_cache.h> | ||
| 8 | #include <rdma/ib_user_verbs.h> | ||
| 9 | #include <ionic_api.h> | ||
| 10 | |||
| 11 | #include "ionic_fw.h" | ||
| 12 | #include "ionic_ibdev.h" | ||
| 13 | |||
| 14 | #define ionic_set_ecn(tos) (((tos) | 2u) & ~1u) | ||
| 15 | #define ionic_clear_ecn(tos) ((tos) & ~3u) | ||
| 16 | |||
| 17 | static int ionic_validate_qdesc(struct ionic_qdesc *q) | ||
| 18 | { | ||
| 19 | if (!q->addr || !q->size || !q->mask || | ||
| 20 | !q->depth_log2 || !q->stride_log2) | ||
| 21 | return -EINVAL; | ||
| 22 | |||
| 23 | if (q->addr & (PAGE_SIZE - 1)) | ||
| 24 | return -EINVAL; | ||
| 25 | |||
| 26 | if (q->mask != BIT(q->depth_log2) - 1) | ||
| 27 | return -EINVAL; | ||
| 28 | |||
| 29 | if (q->size < BIT_ULL(q->depth_log2 + q->stride_log2)) | ||
| 30 | return -EINVAL; | ||
| 31 | |||
| 32 | return 0; | ||
| 33 | } | ||
| 34 | |||
| 35 | static u32 ionic_get_eqid(struct ionic_ibdev *dev, u32 comp_vector, u8 udma_idx) | ||
| 36 | { | ||
| 37 | /* EQ per vector per udma, and the first eqs reserved for async events. | ||
| 38 | * The rest of the vectors can be requested for completions. | ||
| 39 | */ | ||
| 40 | u32 comp_vec_count = dev->lif_cfg.eq_count / dev->lif_cfg.udma_count - 1; | ||
| 41 | |||
| 42 | return (comp_vector % comp_vec_count + 1) * dev->lif_cfg.udma_count + udma_idx; | ||
| 43 | } | ||
| 44 | |||
| 45 | static int ionic_get_cqid(struct ionic_ibdev *dev, u32 *cqid, u8 udma_idx) | ||
| 46 | { | ||
| 47 | unsigned int size, base, bound; | ||
| 48 | int rc; | ||
| 49 | |||
| 50 | size = dev->lif_cfg.cq_count / dev->lif_cfg.udma_count; | ||
| 51 | base = size * udma_idx; | ||
| 52 | bound = base + size; | ||
| 53 | |||
| 54 | rc = ionic_resid_get_shared(&dev->inuse_cqid, base, bound); | ||
| 55 | if (rc >= 0) { | ||
| 56 | /* cq_base is zero or a multiple of two queue groups */ | ||
| 57 | *cqid = dev->lif_cfg.cq_base + | ||
| 58 | ionic_bitid_to_qid(rc, dev->lif_cfg.udma_qgrp_shift, | ||
| 59 | dev->half_cqid_udma_shift); | ||
| 60 | |||
| 61 | rc = 0; | ||
| 62 | } | ||
| 63 | |||
| 64 | return rc; | ||
| 65 | } | ||
| 66 | |||
| 67 | static void ionic_put_cqid(struct ionic_ibdev *dev, u32 cqid) | ||
| 68 | { | ||
| 69 | u32 bitid = ionic_qid_to_bitid(cqid - dev->lif_cfg.cq_base, | ||
| 70 | dev->lif_cfg.udma_qgrp_shift, | ||
| 71 | dev->half_cqid_udma_shift); | ||
| 72 | |||
| 73 | ionic_resid_put(&dev->inuse_cqid, bitid); | ||
| 74 | } | ||
| 75 | |||
| 76 | int ionic_create_cq_common(struct ionic_vcq *vcq, | ||
| 77 | struct ionic_tbl_buf *buf, | ||
| 78 | const struct ib_cq_init_attr *attr, | ||
| 79 | struct ionic_ctx *ctx, | ||
| 80 | struct ib_udata *udata, | ||
| 81 | struct ionic_qdesc *req_cq, | ||
| 82 | __u32 *resp_cqid, | ||
| 83 | int udma_idx) | ||
| 84 | { | ||
| 85 | struct ionic_ibdev *dev = to_ionic_ibdev(vcq->ibcq.device); | ||
| 86 | struct ionic_cq *cq = &vcq->cq[udma_idx]; | ||
| 87 | void *entry; | ||
| 88 | int rc; | ||
| 89 | |||
| 90 | cq->vcq = vcq; | ||
| 91 | |||
| 92 | if (attr->cqe < 1 || attr->cqe + IONIC_CQ_GRACE > 0xffff) { | ||
| 93 | rc = -EINVAL; | ||
| 94 | goto err_args; | ||
| 95 | } | ||
| 96 | |||
| 97 | rc = ionic_get_cqid(dev, &cq->cqid, udma_idx); | ||
| 98 | if (rc) | ||
| 99 | goto err_args; | ||
| 100 | |||
| 101 | cq->eqid = ionic_get_eqid(dev, attr->comp_vector, udma_idx); | ||
| 102 | |||
| 103 | spin_lock_init(&cq->lock); | ||
| 104 | INIT_LIST_HEAD(&cq->poll_sq); | ||
| 105 | INIT_LIST_HEAD(&cq->flush_sq); | ||
| 106 | INIT_LIST_HEAD(&cq->flush_rq); | ||
| 107 | |||
| 108 | if (udata) { | ||
| 109 | rc = ionic_validate_qdesc(req_cq); | ||
| 110 | if (rc) | ||
| 111 | goto err_qdesc; | ||
| 112 | |||
| 113 | cq->umem = ib_umem_get(&dev->ibdev, req_cq->addr, req_cq->size, | ||
| 114 | IB_ACCESS_LOCAL_WRITE); | ||
| 115 | if (IS_ERR(cq->umem)) { | ||
| 116 | rc = PTR_ERR(cq->umem); | ||
| 117 | goto err_qdesc; | ||
| 118 | } | ||
| 119 | |||
| 120 | cq->q.ptr = NULL; | ||
| 121 | cq->q.size = req_cq->size; | ||
| 122 | cq->q.mask = req_cq->mask; | ||
| 123 | cq->q.depth_log2 = req_cq->depth_log2; | ||
| 124 | cq->q.stride_log2 = req_cq->stride_log2; | ||
| 125 | |||
| 126 | *resp_cqid = cq->cqid; | ||
| 127 | } else { | ||
| 128 | rc = ionic_queue_init(&cq->q, dev->lif_cfg.hwdev, | ||
| 129 | attr->cqe + IONIC_CQ_GRACE, | ||
| 130 | sizeof(struct ionic_v1_cqe)); | ||
| 131 | if (rc) | ||
| 132 | goto err_q_init; | ||
| 133 | |||
| 134 | ionic_queue_dbell_init(&cq->q, cq->cqid); | ||
| 135 | cq->color = true; | ||
| 136 | cq->credit = cq->q.mask; | ||
| 137 | } | ||
| 138 | |||
| 139 | rc = ionic_pgtbl_init(dev, buf, cq->umem, cq->q.dma, 1, PAGE_SIZE); | ||
| 140 | if (rc) | ||
| 141 | goto err_pgtbl_init; | ||
| 142 | |||
| 143 | init_completion(&cq->cq_rel_comp); | ||
| 144 | kref_init(&cq->cq_kref); | ||
| 145 | |||
| 146 | entry = xa_store_irq(&dev->cq_tbl, cq->cqid, cq, GFP_KERNEL); | ||
| 147 | if (entry) { | ||
| 148 | if (!xa_is_err(entry)) | ||
| 149 | rc = -EINVAL; | ||
| 150 | else | ||
| 151 | rc = xa_err(entry); | ||
| 152 | |||
| 153 | goto err_xa; | ||
| 154 | } | ||
| 155 | |||
| 156 | return 0; | ||
| 157 | |||
| 158 | err_xa: | ||
| 159 | ionic_pgtbl_unbuf(dev, buf); | ||
| 160 | err_pgtbl_init: | ||
| 161 | if (!udata) | ||
| 162 | ionic_queue_destroy(&cq->q, dev->lif_cfg.hwdev); | ||
| 163 | err_q_init: | ||
| 164 | if (cq->umem) | ||
| 165 | ib_umem_release(cq->umem); | ||
| 166 | err_qdesc: | ||
| 167 | ionic_put_cqid(dev, cq->cqid); | ||
| 168 | err_args: | ||
| 169 | cq->vcq = NULL; | ||
| 170 | |||
| 171 | return rc; | ||
| 172 | } | ||
| 173 | |||
| 174 | void ionic_destroy_cq_common(struct ionic_ibdev *dev, struct ionic_cq *cq) | ||
| 175 | { | ||
| 176 | if (!cq->vcq) | ||
| 177 | return; | ||
| 178 | |||
| 179 | xa_erase_irq(&dev->cq_tbl, cq->cqid); | ||
| 180 | |||
| 181 | kref_put(&cq->cq_kref, ionic_cq_complete); | ||
| 182 | wait_for_completion(&cq->cq_rel_comp); | ||
| 183 | |||
| 184 | if (cq->umem) | ||
| 185 | ib_umem_release(cq->umem); | ||
| 186 | else | ||
| 187 | ionic_queue_destroy(&cq->q, dev->lif_cfg.hwdev); | ||
| 188 | |||
| 189 | ionic_put_cqid(dev, cq->cqid); | ||
| 190 | |||
| 191 | cq->vcq = NULL; | ||
| 192 | } | ||
| 193 | |||
| 194 | static int ionic_validate_qdesc_zero(struct ionic_qdesc *q) | ||
| 195 | { | ||
| 196 | if (q->addr || q->size || q->mask || q->depth_log2 || q->stride_log2) | ||
| 197 | return -EINVAL; | ||
| 198 | |||
| 199 | return 0; | ||
| 200 | } | ||
| 201 | |||
| 202 | static int ionic_get_pdid(struct ionic_ibdev *dev, u32 *pdid) | ||
| 203 | { | ||
| 204 | int rc; | ||
| 205 | |||
| 206 | rc = ionic_resid_get(&dev->inuse_pdid); | ||
| 207 | if (rc < 0) | ||
| 208 | return rc; | ||
| 209 | |||
| 210 | *pdid = rc; | ||
| 211 | return 0; | ||
| 212 | } | ||
| 213 | |||
| 214 | static int ionic_get_ahid(struct ionic_ibdev *dev, u32 *ahid) | ||
| 215 | { | ||
| 216 | int rc; | ||
| 217 | |||
| 218 | rc = ionic_resid_get(&dev->inuse_ahid); | ||
| 219 | if (rc < 0) | ||
| 220 | return rc; | ||
| 221 | |||
| 222 | *ahid = rc; | ||
| 223 | return 0; | ||
| 224 | } | ||
| 225 | |||
| 226 | static int ionic_get_mrid(struct ionic_ibdev *dev, u32 *mrid) | ||
| 227 | { | ||
| 228 | int rc; | ||
| 229 | |||
| 230 | /* wrap to 1, skip reserved lkey */ | ||
| 231 | rc = ionic_resid_get_shared(&dev->inuse_mrid, 1, | ||
| 232 | dev->inuse_mrid.inuse_size); | ||
| 233 | if (rc < 0) | ||
| 234 | return rc; | ||
| 235 | |||
| 236 | *mrid = ionic_mrid(rc, dev->next_mrkey++); | ||
| 237 | return 0; | ||
| 238 | } | ||
| 239 | |||
| 240 | static int ionic_get_gsi_qpid(struct ionic_ibdev *dev, u32 *qpid) | ||
| 241 | { | ||
| 242 | int rc = 0; | ||
| 243 | |||
| 244 | rc = ionic_resid_get_shared(&dev->inuse_qpid, IB_QPT_GSI, IB_QPT_GSI + 1); | ||
| 245 | if (rc < 0) | ||
| 246 | return rc; | ||
| 247 | |||
| 248 | *qpid = IB_QPT_GSI; | ||
| 249 | return 0; | ||
| 250 | } | ||
| 251 | |||
| 252 | static int ionic_get_qpid(struct ionic_ibdev *dev, u32 *qpid, | ||
| 253 | u8 *udma_idx, u8 udma_mask) | ||
| 254 | { | ||
| 255 | unsigned int size, base, bound; | ||
| 256 | int udma_i, udma_x, udma_ix; | ||
| 257 | int rc = -EINVAL; | ||
| 258 | |||
| 259 | udma_x = dev->next_qpid_udma_idx; | ||
| 260 | |||
| 261 | dev->next_qpid_udma_idx ^= dev->lif_cfg.udma_count - 1; | ||
| 262 | |||
| 263 | for (udma_i = 0; udma_i < dev->lif_cfg.udma_count; ++udma_i) { | ||
| 264 | udma_ix = udma_i ^ udma_x; | ||
| 265 | |||
| 266 | if (!(udma_mask & BIT(udma_ix))) | ||
| 267 | continue; | ||
| 268 | |||
| 269 | size = dev->lif_cfg.qp_count / dev->lif_cfg.udma_count; | ||
| 270 | base = size * udma_ix; | ||
| 271 | bound = base + size; | ||
| 272 | |||
| 273 | /* skip reserved SMI and GSI qpids in group zero */ | ||
| 274 | if (!base) | ||
| 275 | base = 2; | ||
| 276 | |||
| 277 | rc = ionic_resid_get_shared(&dev->inuse_qpid, base, bound); | ||
| 278 | if (rc >= 0) { | ||
| 279 | *qpid = ionic_bitid_to_qid(rc, | ||
| 280 | dev->lif_cfg.udma_qgrp_shift, | ||
| 281 | dev->half_qpid_udma_shift); | ||
| 282 | *udma_idx = udma_ix; | ||
| 283 | |||
| 284 | rc = 0; | ||
| 285 | break; | ||
| 286 | } | ||
| 287 | } | ||
| 288 | |||
| 289 | return rc; | ||
| 290 | } | ||
| 291 | |||
| 292 | static int ionic_get_dbid(struct ionic_ibdev *dev, u32 *dbid, phys_addr_t *addr) | ||
| 293 | { | ||
| 294 | int rc, dbpage_num; | ||
| 295 | |||
| 296 | /* wrap to 1, skip kernel reserved */ | ||
| 297 | rc = ionic_resid_get_shared(&dev->inuse_dbid, 1, | ||
| 298 | dev->inuse_dbid.inuse_size); | ||
| 299 | if (rc < 0) | ||
| 300 | return rc; | ||
| 301 | |||
| 302 | dbpage_num = (dev->lif_cfg.lif_hw_index * dev->lif_cfg.dbid_count) + rc; | ||
| 303 | *addr = dev->lif_cfg.db_phys + ((phys_addr_t)dbpage_num << PAGE_SHIFT); | ||
| 304 | |||
| 305 | *dbid = rc; | ||
| 306 | |||
| 307 | return 0; | ||
| 308 | } | ||
| 309 | |||
| 310 | static void ionic_put_pdid(struct ionic_ibdev *dev, u32 pdid) | ||
| 311 | { | ||
| 312 | ionic_resid_put(&dev->inuse_pdid, pdid); | ||
| 313 | } | ||
| 314 | |||
| 315 | static void ionic_put_ahid(struct ionic_ibdev *dev, u32 ahid) | ||
| 316 | { | ||
| 317 | ionic_resid_put(&dev->inuse_ahid, ahid); | ||
| 318 | } | ||
| 319 | |||
| 320 | static void ionic_put_mrid(struct ionic_ibdev *dev, u32 mrid) | ||
| 321 | { | ||
| 322 | ionic_resid_put(&dev->inuse_mrid, ionic_mrid_index(mrid)); | ||
| 323 | } | ||
| 324 | |||
| 325 | static void ionic_put_qpid(struct ionic_ibdev *dev, u32 qpid) | ||
| 326 | { | ||
| 327 | u32 bitid = ionic_qid_to_bitid(qpid, | ||
| 328 | dev->lif_cfg.udma_qgrp_shift, | ||
| 329 | dev->half_qpid_udma_shift); | ||
| 330 | |||
| 331 | ionic_resid_put(&dev->inuse_qpid, bitid); | ||
| 332 | } | ||
| 333 | |||
| 334 | static void ionic_put_dbid(struct ionic_ibdev *dev, u32 dbid) | ||
| 335 | { | ||
| 336 | ionic_resid_put(&dev->inuse_dbid, dbid); | ||
| 337 | } | ||
| 338 | |||
| 339 | static struct rdma_user_mmap_entry* | ||
| 340 | ionic_mmap_entry_insert(struct ionic_ctx *ctx, unsigned long size, | ||
| 341 | unsigned long pfn, u8 mmap_flags, u64 *offset) | ||
| 342 | { | ||
| 343 | struct ionic_mmap_entry *entry; | ||
| 344 | int rc; | ||
| 345 | |||
| 346 | entry = kzalloc(sizeof(*entry), GFP_KERNEL); | ||
| 347 | if (!entry) | ||
| 348 | return NULL; | ||
| 349 | |||
| 350 | entry->size = size; | ||
| 351 | entry->pfn = pfn; | ||
| 352 | entry->mmap_flags = mmap_flags; | ||
| 353 | |||
| 354 | rc = rdma_user_mmap_entry_insert(&ctx->ibctx, &entry->rdma_entry, | ||
| 355 | entry->size); | ||
| 356 | if (rc) { | ||
| 357 | kfree(entry); | ||
| 358 | return NULL; | ||
| 359 | } | ||
| 360 | |||
| 361 | if (offset) | ||
| 362 | *offset = rdma_user_mmap_get_offset(&entry->rdma_entry); | ||
| 363 | |||
| 364 | return &entry->rdma_entry; | ||
| 365 | } | ||
| 366 | |||
| 367 | int ionic_alloc_ucontext(struct ib_ucontext *ibctx, struct ib_udata *udata) | ||
| 368 | { | ||
| 369 | struct ionic_ibdev *dev = to_ionic_ibdev(ibctx->device); | ||
| 370 | struct ionic_ctx *ctx = to_ionic_ctx(ibctx); | ||
| 371 | struct ionic_ctx_resp resp = {}; | ||
| 372 | struct ionic_ctx_req req; | ||
| 373 | phys_addr_t db_phys = 0; | ||
| 374 | int rc; | ||
| 375 | |||
| 376 | rc = ib_copy_from_udata(&req, udata, sizeof(req)); | ||
| 377 | if (rc) | ||
| 378 | return rc; | ||
| 379 | |||
| 380 | /* try to allocate dbid for user ctx */ | ||
| 381 | rc = ionic_get_dbid(dev, &ctx->dbid, &db_phys); | ||
| 382 | if (rc < 0) | ||
| 383 | return rc; | ||
| 384 | |||
| 385 | ibdev_dbg(&dev->ibdev, "user space dbid %u\n", ctx->dbid); | ||
| 386 | |||
| 387 | ctx->mmap_dbell = ionic_mmap_entry_insert(ctx, PAGE_SIZE, | ||
| 388 | PHYS_PFN(db_phys), 0, NULL); | ||
| 389 | if (!ctx->mmap_dbell) { | ||
| 390 | rc = -ENOMEM; | ||
| 391 | goto err_mmap_dbell; | ||
| 392 | } | ||
| 393 | |||
| 394 | resp.page_shift = PAGE_SHIFT; | ||
| 395 | |||
| 396 | resp.dbell_offset = db_phys & ~PAGE_MASK; | ||
| 397 | |||
| 398 | resp.version = dev->lif_cfg.rdma_version; | ||
| 399 | resp.qp_opcodes = dev->lif_cfg.qp_opcodes; | ||
| 400 | resp.admin_opcodes = dev->lif_cfg.admin_opcodes; | ||
| 401 | |||
| 402 | resp.sq_qtype = dev->lif_cfg.sq_qtype; | ||
| 403 | resp.rq_qtype = dev->lif_cfg.rq_qtype; | ||
| 404 | resp.cq_qtype = dev->lif_cfg.cq_qtype; | ||
| 405 | resp.admin_qtype = dev->lif_cfg.aq_qtype; | ||
| 406 | resp.max_stride = dev->lif_cfg.max_stride; | ||
| 407 | resp.max_spec = IONIC_SPEC_HIGH; | ||
| 408 | |||
| 409 | resp.udma_count = dev->lif_cfg.udma_count; | ||
| 410 | resp.expdb_mask = dev->lif_cfg.expdb_mask; | ||
| 411 | |||
| 412 | if (dev->lif_cfg.sq_expdb) | ||
| 413 | resp.expdb_qtypes |= IONIC_EXPDB_SQ; | ||
| 414 | if (dev->lif_cfg.rq_expdb) | ||
| 415 | resp.expdb_qtypes |= IONIC_EXPDB_RQ; | ||
| 416 | |||
| 417 | rc = ib_copy_to_udata(udata, &resp, sizeof(resp)); | ||
| 418 | if (rc) | ||
| 419 | goto err_resp; | ||
| 420 | |||
| 421 | return 0; | ||
| 422 | |||
| 423 | err_resp: | ||
| 424 | rdma_user_mmap_entry_remove(ctx->mmap_dbell); | ||
| 425 | err_mmap_dbell: | ||
| 426 | ionic_put_dbid(dev, ctx->dbid); | ||
| 427 | |||
| 428 | return rc; | ||
| 429 | } | ||
| 430 | |||
| 431 | void ionic_dealloc_ucontext(struct ib_ucontext *ibctx) | ||
| 432 | { | ||
| 433 | struct ionic_ibdev *dev = to_ionic_ibdev(ibctx->device); | ||
| 434 | struct ionic_ctx *ctx = to_ionic_ctx(ibctx); | ||
| 435 | |||
| 436 | rdma_user_mmap_entry_remove(ctx->mmap_dbell); | ||
| 437 | ionic_put_dbid(dev, ctx->dbid); | ||
| 438 | } | ||
| 439 | |||
| 440 | int ionic_mmap(struct ib_ucontext *ibctx, struct vm_area_struct *vma) | ||
| 441 | { | ||
| 442 | struct ionic_ibdev *dev = to_ionic_ibdev(ibctx->device); | ||
| 443 | struct ionic_ctx *ctx = to_ionic_ctx(ibctx); | ||
| 444 | struct rdma_user_mmap_entry *rdma_entry; | ||
| 445 | struct ionic_mmap_entry *ionic_entry; | ||
| 446 | int rc = 0; | ||
| 447 | |||
| 448 | rdma_entry = rdma_user_mmap_entry_get(&ctx->ibctx, vma); | ||
| 449 | if (!rdma_entry) { | ||
| 450 | ibdev_dbg(&dev->ibdev, "not found %#lx\n", | ||
| 451 | vma->vm_pgoff << PAGE_SHIFT); | ||
| 452 | return -EINVAL; | ||
| 453 | } | ||
| 454 | |||
| 455 | ionic_entry = container_of(rdma_entry, struct ionic_mmap_entry, | ||
| 456 | rdma_entry); | ||
| 457 | |||
| 458 | ibdev_dbg(&dev->ibdev, "writecombine? %d\n", | ||
| 459 | ionic_entry->mmap_flags & IONIC_MMAP_WC); | ||
| 460 | if (ionic_entry->mmap_flags & IONIC_MMAP_WC) | ||
| 461 | vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); | ||
| 462 | else | ||
| 463 | vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); | ||
| 464 | |||
| 465 | ibdev_dbg(&dev->ibdev, "remap st %#lx pf %#lx sz %#lx\n", | ||
| 466 | vma->vm_start, ionic_entry->pfn, ionic_entry->size); | ||
| 467 | rc = rdma_user_mmap_io(&ctx->ibctx, vma, ionic_entry->pfn, | ||
| 468 | ionic_entry->size, vma->vm_page_prot, | ||
| 469 | rdma_entry); | ||
| 470 | if (rc) | ||
| 471 | ibdev_dbg(&dev->ibdev, "remap failed %d\n", rc); | ||
| 472 | |||
| 473 | rdma_user_mmap_entry_put(rdma_entry); | ||
| 474 | return rc; | ||
| 475 | } | ||
| 476 | |||
| 477 | void ionic_mmap_free(struct rdma_user_mmap_entry *rdma_entry) | ||
| 478 | { | ||
| 479 | struct ionic_mmap_entry *ionic_entry; | ||
| 480 | |||
| 481 | ionic_entry = container_of(rdma_entry, struct ionic_mmap_entry, | ||
| 482 | rdma_entry); | ||
| 483 | kfree(ionic_entry); | ||
| 484 | } | ||
| 485 | |||
| 486 | int ionic_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) | ||
| 487 | { | ||
| 488 | struct ionic_ibdev *dev = to_ionic_ibdev(ibpd->device); | ||
| 489 | struct ionic_pd *pd = to_ionic_pd(ibpd); | ||
| 490 | |||
| 491 | return ionic_get_pdid(dev, &pd->pdid); | ||
| 492 | } | ||
| 493 | |||
| 494 | int ionic_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) | ||
| 495 | { | ||
| 496 | struct ionic_ibdev *dev = to_ionic_ibdev(ibpd->device); | ||
| 497 | struct ionic_pd *pd = to_ionic_pd(ibpd); | ||
| 498 | |||
| 499 | ionic_put_pdid(dev, pd->pdid); | ||
| 500 | |||
| 501 | return 0; | ||
| 502 | } | ||
| 503 | |||
| 504 | static int ionic_build_hdr(struct ionic_ibdev *dev, | ||
| 505 | struct ib_ud_header *hdr, | ||
| 506 | const struct rdma_ah_attr *attr, | ||
| 507 | u16 sport, bool want_ecn) | ||
| 508 | { | ||
| 509 | const struct ib_global_route *grh; | ||
| 510 | enum rdma_network_type net; | ||
| 511 | u16 vlan; | ||
| 512 | int rc; | ||
| 513 | |||
| 514 | if (attr->ah_flags != IB_AH_GRH) | ||
| 515 | return -EINVAL; | ||
| 516 | if (attr->type != RDMA_AH_ATTR_TYPE_ROCE) | ||
| 517 | return -EINVAL; | ||
| 518 | |||
| 519 | grh = rdma_ah_read_grh(attr); | ||
| 520 | |||
| 521 | rc = rdma_read_gid_l2_fields(grh->sgid_attr, &vlan, &hdr->eth.smac_h[0]); | ||
| 522 | if (rc) | ||
| 523 | return rc; | ||
| 524 | |||
| 525 | net = rdma_gid_attr_network_type(grh->sgid_attr); | ||
| 526 | |||
| 527 | rc = ib_ud_header_init(0, /* no payload */ | ||
| 528 | 0, /* no lrh */ | ||
| 529 | 1, /* yes eth */ | ||
| 530 | vlan != 0xffff, | ||
| 531 | 0, /* no grh */ | ||
| 532 | net == RDMA_NETWORK_IPV4 ? 4 : 6, | ||
| 533 | 1, /* yes udp */ | ||
| 534 | 0, /* no imm */ | ||
| 535 | hdr); | ||
| 536 | if (rc) | ||
| 537 | return rc; | ||
| 538 | |||
| 539 | ether_addr_copy(hdr->eth.dmac_h, attr->roce.dmac); | ||
| 540 | |||
| 541 | if (net == RDMA_NETWORK_IPV4) { | ||
| 542 | hdr->eth.type = cpu_to_be16(ETH_P_IP); | ||
| 543 | hdr->ip4.frag_off = cpu_to_be16(0x4000); /* don't fragment */ | ||
| 544 | hdr->ip4.ttl = grh->hop_limit; | ||
| 545 | hdr->ip4.tot_len = cpu_to_be16(0xffff); | ||
| 546 | hdr->ip4.saddr = | ||
| 547 | *(const __be32 *)(grh->sgid_attr->gid.raw + 12); | ||
| 548 | hdr->ip4.daddr = *(const __be32 *)(grh->dgid.raw + 12); | ||
| 549 | |||
| 550 | if (want_ecn) | ||
| 551 | hdr->ip4.tos = ionic_set_ecn(grh->traffic_class); | ||
| 552 | else | ||
| 553 | hdr->ip4.tos = ionic_clear_ecn(grh->traffic_class); | ||
| 554 | } else { | ||
| 555 | hdr->eth.type = cpu_to_be16(ETH_P_IPV6); | ||
| 556 | hdr->grh.flow_label = cpu_to_be32(grh->flow_label); | ||
| 557 | hdr->grh.hop_limit = grh->hop_limit; | ||
| 558 | hdr->grh.source_gid = grh->sgid_attr->gid; | ||
| 559 | hdr->grh.destination_gid = grh->dgid; | ||
| 560 | |||
| 561 | if (want_ecn) | ||
| 562 | hdr->grh.traffic_class = | ||
| 563 | ionic_set_ecn(grh->traffic_class); | ||
| 564 | else | ||
| 565 | hdr->grh.traffic_class = | ||
| 566 | ionic_clear_ecn(grh->traffic_class); | ||
| 567 | } | ||
| 568 | |||
| 569 | if (vlan != 0xffff) { | ||
| 570 | vlan |= rdma_ah_get_sl(attr) << VLAN_PRIO_SHIFT; | ||
| 571 | hdr->vlan.tag = cpu_to_be16(vlan); | ||
| 572 | hdr->vlan.type = hdr->eth.type; | ||
| 573 | hdr->eth.type = cpu_to_be16(ETH_P_8021Q); | ||
| 574 | } | ||
| 575 | |||
| 576 | hdr->udp.sport = cpu_to_be16(sport); | ||
| 577 | hdr->udp.dport = cpu_to_be16(ROCE_V2_UDP_DPORT); | ||
| 578 | |||
| 579 | return 0; | ||
| 580 | } | ||
| 581 | |||
| 582 | static void ionic_set_ah_attr(struct ionic_ibdev *dev, | ||
| 583 | struct rdma_ah_attr *ah_attr, | ||
| 584 | struct ib_ud_header *hdr, | ||
| 585 | int sgid_index) | ||
| 586 | { | ||
| 587 | u32 flow_label; | ||
| 588 | u16 vlan = 0; | ||
| 589 | u8 tos, ttl; | ||
| 590 | |||
| 591 | if (hdr->vlan_present) | ||
| 592 | vlan = be16_to_cpu(hdr->vlan.tag); | ||
| 593 | |||
| 594 | if (hdr->ipv4_present) { | ||
| 595 | flow_label = 0; | ||
| 596 | ttl = hdr->ip4.ttl; | ||
| 597 | tos = hdr->ip4.tos; | ||
| 598 | *(__be16 *)(hdr->grh.destination_gid.raw + 10) = cpu_to_be16(0xffff); | ||
| 599 | *(__be32 *)(hdr->grh.destination_gid.raw + 12) = hdr->ip4.daddr; | ||
| 600 | } else { | ||
| 601 | flow_label = be32_to_cpu(hdr->grh.flow_label); | ||
| 602 | ttl = hdr->grh.hop_limit; | ||
| 603 | tos = hdr->grh.traffic_class; | ||
| 604 | } | ||
| 605 | |||
| 606 | memset(ah_attr, 0, sizeof(*ah_attr)); | ||
| 607 | ah_attr->type = RDMA_AH_ATTR_TYPE_ROCE; | ||
| 608 | if (hdr->eth_present) | ||
| 609 | ether_addr_copy(ah_attr->roce.dmac, hdr->eth.dmac_h); | ||
| 610 | rdma_ah_set_sl(ah_attr, vlan >> VLAN_PRIO_SHIFT); | ||
| 611 | rdma_ah_set_port_num(ah_attr, 1); | ||
| 612 | rdma_ah_set_grh(ah_attr, NULL, flow_label, sgid_index, ttl, tos); | ||
| 613 | rdma_ah_set_dgid_raw(ah_attr, &hdr->grh.destination_gid); | ||
| 614 | } | ||
| 615 | |||
| 616 | static int ionic_create_ah_cmd(struct ionic_ibdev *dev, | ||
| 617 | struct ionic_ah *ah, | ||
| 618 | struct ionic_pd *pd, | ||
| 619 | struct rdma_ah_attr *attr, | ||
| 620 | u32 flags) | ||
| 621 | { | ||
| 622 | struct ionic_admin_wr wr = { | ||
| 623 | .work = COMPLETION_INITIALIZER_ONSTACK(wr.work), | ||
| 624 | .wqe = { | ||
| 625 | .op = IONIC_V1_ADMIN_CREATE_AH, | ||
| 626 | .len = cpu_to_le16(IONIC_ADMIN_CREATE_AH_IN_V1_LEN), | ||
| 627 | .cmd.create_ah = { | ||
| 628 | .pd_id = cpu_to_le32(pd->pdid), | ||
| 629 | .dbid_flags = cpu_to_le16(dev->lif_cfg.dbid), | ||
| 630 | .id_ver = cpu_to_le32(ah->ahid), | ||
| 631 | } | ||
| 632 | } | ||
| 633 | }; | ||
| 634 | enum ionic_admin_flags admin_flags = 0; | ||
| 635 | dma_addr_t hdr_dma = 0; | ||
| 636 | void *hdr_buf; | ||
| 637 | gfp_t gfp = GFP_ATOMIC; | ||
| 638 | int rc, hdr_len = 0; | ||
| 639 | |||
| 640 | if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_CREATE_AH) | ||
| 641 | return -EBADRQC; | ||
| 642 | |||
| 643 | if (flags & RDMA_CREATE_AH_SLEEPABLE) | ||
| 644 | gfp = GFP_KERNEL; | ||
| 645 | else | ||
| 646 | admin_flags |= IONIC_ADMIN_F_BUSYWAIT; | ||
| 647 | |||
| 648 | rc = ionic_build_hdr(dev, &ah->hdr, attr, IONIC_ROCE_UDP_SPORT, false); | ||
| 649 | if (rc) | ||
| 650 | return rc; | ||
| 651 | |||
| 652 | if (ah->hdr.eth.type == cpu_to_be16(ETH_P_8021Q)) { | ||
| 653 | if (ah->hdr.vlan.type == cpu_to_be16(ETH_P_IP)) | ||
| 654 | wr.wqe.cmd.create_ah.csum_profile = | ||
| 655 | IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_UDP; | ||
| 656 | else | ||
| 657 | wr.wqe.cmd.create_ah.csum_profile = | ||
| 658 | IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV6_UDP; | ||
| 659 | } else { | ||
| 660 | if (ah->hdr.eth.type == cpu_to_be16(ETH_P_IP)) | ||
| 661 | wr.wqe.cmd.create_ah.csum_profile = | ||
| 662 | IONIC_TFP_CSUM_PROF_ETH_IPV4_UDP; | ||
| 663 | else | ||
| 664 | wr.wqe.cmd.create_ah.csum_profile = | ||
| 665 | IONIC_TFP_CSUM_PROF_ETH_IPV6_UDP; | ||
| 666 | } | ||
| 667 | |||
| 668 | ah->sgid_index = rdma_ah_read_grh(attr)->sgid_index; | ||
| 669 | |||
| 670 | hdr_buf = kmalloc(PAGE_SIZE, gfp); | ||
| 671 | if (!hdr_buf) | ||
| 672 | return -ENOMEM; | ||
| 673 | |||
| 674 | hdr_len = ib_ud_header_pack(&ah->hdr, hdr_buf); | ||
| 675 | hdr_len -= IB_BTH_BYTES; | ||
| 676 | hdr_len -= IB_DETH_BYTES; | ||
| 677 | ibdev_dbg(&dev->ibdev, "roce packet header template\n"); | ||
| 678 | print_hex_dump_debug("hdr ", DUMP_PREFIX_OFFSET, 16, 1, | ||
| 679 | hdr_buf, hdr_len, true); | ||
| 680 | |||
| 681 | hdr_dma = dma_map_single(dev->lif_cfg.hwdev, hdr_buf, hdr_len, | ||
| 682 | DMA_TO_DEVICE); | ||
| 683 | |||
| 684 | rc = dma_mapping_error(dev->lif_cfg.hwdev, hdr_dma); | ||
| 685 | if (rc) | ||
| 686 | goto err_dma; | ||
| 687 | |||
| 688 | wr.wqe.cmd.create_ah.dma_addr = cpu_to_le64(hdr_dma); | ||
| 689 | wr.wqe.cmd.create_ah.length = cpu_to_le32(hdr_len); | ||
| 690 | |||
| 691 | ionic_admin_post(dev, &wr); | ||
| 692 | rc = ionic_admin_wait(dev, &wr, admin_flags); | ||
| 693 | |||
| 694 | dma_unmap_single(dev->lif_cfg.hwdev, hdr_dma, hdr_len, | ||
| 695 | DMA_TO_DEVICE); | ||
| 696 | err_dma: | ||
| 697 | kfree(hdr_buf); | ||
| 698 | |||
| 699 | return rc; | ||
| 700 | } | ||
| 701 | |||
| 702 | static int ionic_destroy_ah_cmd(struct ionic_ibdev *dev, u32 ahid, u32 flags) | ||
| 703 | { | ||
| 704 | struct ionic_admin_wr wr = { | ||
| 705 | .work = COMPLETION_INITIALIZER_ONSTACK(wr.work), | ||
| 706 | .wqe = { | ||
| 707 | .op = IONIC_V1_ADMIN_DESTROY_AH, | ||
| 708 | .len = cpu_to_le16(IONIC_ADMIN_DESTROY_AH_IN_V1_LEN), | ||
| 709 | .cmd.destroy_ah = { | ||
| 710 | .ah_id = cpu_to_le32(ahid), | ||
| 711 | }, | ||
| 712 | } | ||
| 713 | }; | ||
| 714 | enum ionic_admin_flags admin_flags = IONIC_ADMIN_F_TEARDOWN; | ||
| 715 | |||
| 716 | if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_DESTROY_AH) | ||
| 717 | return -EBADRQC; | ||
| 718 | |||
| 719 | if (!(flags & RDMA_CREATE_AH_SLEEPABLE)) | ||
| 720 | admin_flags |= IONIC_ADMIN_F_BUSYWAIT; | ||
| 721 | |||
| 722 | ionic_admin_post(dev, &wr); | ||
| 723 | ionic_admin_wait(dev, &wr, admin_flags); | ||
| 724 | |||
| 725 | /* No host-memory resource is associated with ah, so it is ok | ||
| 726 | * to "succeed" and complete this destroy ah on the host. | ||
| 727 | */ | ||
| 728 | return 0; | ||
| 729 | } | ||
| 730 | |||
| 731 | int ionic_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *init_attr, | ||
| 732 | struct ib_udata *udata) | ||
| 733 | { | ||
| 734 | struct ionic_ibdev *dev = to_ionic_ibdev(ibah->device); | ||
| 735 | struct rdma_ah_attr *attr = init_attr->ah_attr; | ||
| 736 | struct ionic_pd *pd = to_ionic_pd(ibah->pd); | ||
| 737 | struct ionic_ah *ah = to_ionic_ah(ibah); | ||
| 738 | struct ionic_ah_resp resp = {}; | ||
| 739 | u32 flags = init_attr->flags; | ||
| 740 | int rc; | ||
| 741 | |||
| 742 | rc = ionic_get_ahid(dev, &ah->ahid); | ||
| 743 | if (rc) | ||
| 744 | return rc; | ||
| 745 | |||
| 746 | rc = ionic_create_ah_cmd(dev, ah, pd, attr, flags); | ||
| 747 | if (rc) | ||
| 748 | goto err_cmd; | ||
| 749 | |||
| 750 | if (udata) { | ||
| 751 | resp.ahid = ah->ahid; | ||
| 752 | |||
| 753 | rc = ib_copy_to_udata(udata, &resp, sizeof(resp)); | ||
| 754 | if (rc) | ||
| 755 | goto err_resp; | ||
| 756 | } | ||
| 757 | |||
| 758 | return 0; | ||
| 759 | |||
| 760 | err_resp: | ||
| 761 | ionic_destroy_ah_cmd(dev, ah->ahid, flags); | ||
| 762 | err_cmd: | ||
| 763 | ionic_put_ahid(dev, ah->ahid); | ||
| 764 | return rc; | ||
| 765 | } | ||
| 766 | |||
| 767 | int ionic_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr) | ||
| 768 | { | ||
| 769 | struct ionic_ibdev *dev = to_ionic_ibdev(ibah->device); | ||
| 770 | struct ionic_ah *ah = to_ionic_ah(ibah); | ||
| 771 | |||
| 772 | ionic_set_ah_attr(dev, ah_attr, &ah->hdr, ah->sgid_index); | ||
| 773 | |||
| 774 | return 0; | ||
| 775 | } | ||
| 776 | |||
| 777 | int ionic_destroy_ah(struct ib_ah *ibah, u32 flags) | ||
| 778 | { | ||
| 779 | struct ionic_ibdev *dev = to_ionic_ibdev(ibah->device); | ||
| 780 | struct ionic_ah *ah = to_ionic_ah(ibah); | ||
| 781 | int rc; | ||
| 782 | |||
| 783 | rc = ionic_destroy_ah_cmd(dev, ah->ahid, flags); | ||
| 784 | if (rc) | ||
| 785 | return rc; | ||
| 786 | |||
| 787 | ionic_put_ahid(dev, ah->ahid); | ||
| 788 | |||
| 789 | return 0; | ||
| 790 | } | ||
| 791 | |||
| 792 | static int ionic_create_mr_cmd(struct ionic_ibdev *dev, | ||
| 793 | struct ionic_pd *pd, | ||
| 794 | struct ionic_mr *mr, | ||
| 795 | u64 addr, | ||
| 796 | u64 length) | ||
| 797 | { | ||
| 798 | struct ionic_admin_wr wr = { | ||
| 799 | .work = COMPLETION_INITIALIZER_ONSTACK(wr.work), | ||
| 800 | .wqe = { | ||
| 801 | .op = IONIC_V1_ADMIN_CREATE_MR, | ||
| 802 | .len = cpu_to_le16(IONIC_ADMIN_CREATE_MR_IN_V1_LEN), | ||
| 803 | .cmd.create_mr = { | ||
| 804 | .va = cpu_to_le64(addr), | ||
| 805 | .length = cpu_to_le64(length), | ||
| 806 | .pd_id = cpu_to_le32(pd->pdid), | ||
| 807 | .page_size_log2 = mr->buf.page_size_log2, | ||
| 808 | .tbl_index = cpu_to_le32(~0), | ||
| 809 | .map_count = cpu_to_le32(mr->buf.tbl_pages), | ||
| 810 | .dma_addr = ionic_pgtbl_dma(&mr->buf, addr), | ||
| 811 | .dbid_flags = cpu_to_le16(mr->flags), | ||
| 812 | .id_ver = cpu_to_le32(mr->mrid), | ||
| 813 | } | ||
| 814 | } | ||
| 815 | }; | ||
| 816 | int rc; | ||
| 817 | |||
| 818 | if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_CREATE_MR) | ||
| 819 | return -EBADRQC; | ||
| 820 | |||
| 821 | ionic_admin_post(dev, &wr); | ||
| 822 | rc = ionic_admin_wait(dev, &wr, 0); | ||
| 823 | if (!rc) | ||
| 824 | mr->created = true; | ||
| 825 | |||
| 826 | return rc; | ||
| 827 | } | ||
| 828 | |||
| 829 | static int ionic_destroy_mr_cmd(struct ionic_ibdev *dev, u32 mrid) | ||
| 830 | { | ||
| 831 | struct ionic_admin_wr wr = { | ||
| 832 | .work = COMPLETION_INITIALIZER_ONSTACK(wr.work), | ||
| 833 | .wqe = { | ||
| 834 | .op = IONIC_V1_ADMIN_DESTROY_MR, | ||
| 835 | .len = cpu_to_le16(IONIC_ADMIN_DESTROY_MR_IN_V1_LEN), | ||
| 836 | .cmd.destroy_mr = { | ||
| 837 | .mr_id = cpu_to_le32(mrid), | ||
| 838 | }, | ||
| 839 | } | ||
| 840 | }; | ||
| 841 | |||
| 842 | if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_DESTROY_MR) | ||
| 843 | return -EBADRQC; | ||
| 844 | |||
| 845 | ionic_admin_post(dev, &wr); | ||
| 846 | |||
| 847 | return ionic_admin_wait(dev, &wr, IONIC_ADMIN_F_TEARDOWN); | ||
| 848 | } | ||
| 849 | |||
| 850 | struct ib_mr *ionic_get_dma_mr(struct ib_pd *ibpd, int access) | ||
| 851 | { | ||
| 852 | struct ionic_pd *pd = to_ionic_pd(ibpd); | ||
| 853 | struct ionic_mr *mr; | ||
| 854 | |||
| 855 | mr = kzalloc(sizeof(*mr), GFP_KERNEL); | ||
| 856 | if (!mr) | ||
| 857 | return ERR_PTR(-ENOMEM); | ||
| 858 | |||
| 859 | mr->ibmr.lkey = IONIC_DMA_LKEY; | ||
| 860 | mr->ibmr.rkey = IONIC_DMA_RKEY; | ||
| 861 | |||
| 862 | if (pd) | ||
| 863 | pd->flags |= IONIC_QPF_PRIVILEGED; | ||
| 864 | |||
| 865 | return &mr->ibmr; | ||
| 866 | } | ||
| 867 | |||
| 868 | struct ib_mr *ionic_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length, | ||
| 869 | u64 addr, int access, struct ib_dmah *dmah, | ||
| 870 | struct ib_udata *udata) | ||
| 871 | { | ||
| 872 | struct ionic_ibdev *dev = to_ionic_ibdev(ibpd->device); | ||
| 873 | struct ionic_pd *pd = to_ionic_pd(ibpd); | ||
| 874 | struct ionic_mr *mr; | ||
| 875 | unsigned long pg_sz; | ||
| 876 | int rc; | ||
| 877 | |||
| 878 | if (dmah) | ||
| 879 | return ERR_PTR(-EOPNOTSUPP); | ||
| 880 | |||
| 881 | mr = kzalloc(sizeof(*mr), GFP_KERNEL); | ||
| 882 | if (!mr) | ||
| 883 | return ERR_PTR(-ENOMEM); | ||
| 884 | |||
| 885 | rc = ionic_get_mrid(dev, &mr->mrid); | ||
| 886 | if (rc) | ||
| 887 | goto err_mrid; | ||
| 888 | |||
| 889 | mr->ibmr.lkey = mr->mrid; | ||
| 890 | mr->ibmr.rkey = mr->mrid; | ||
| 891 | mr->ibmr.iova = addr; | ||
| 892 | mr->ibmr.length = length; | ||
| 893 | |||
| 894 | mr->flags = IONIC_MRF_USER_MR | to_ionic_mr_flags(access); | ||
| 895 | |||
| 896 | mr->umem = ib_umem_get(&dev->ibdev, start, length, access); | ||
| 897 | if (IS_ERR(mr->umem)) { | ||
| 898 | rc = PTR_ERR(mr->umem); | ||
| 899 | goto err_umem; | ||
| 900 | } | ||
| 901 | |||
| 902 | pg_sz = ib_umem_find_best_pgsz(mr->umem, | ||
| 903 | dev->lif_cfg.page_size_supported, | ||
| 904 | addr); | ||
| 905 | if (!pg_sz) { | ||
| 906 | rc = -EINVAL; | ||
| 907 | goto err_pgtbl; | ||
| 908 | } | ||
| 909 | |||
| 910 | rc = ionic_pgtbl_init(dev, &mr->buf, mr->umem, 0, 1, pg_sz); | ||
| 911 | if (rc) | ||
| 912 | goto err_pgtbl; | ||
| 913 | |||
| 914 | rc = ionic_create_mr_cmd(dev, pd, mr, addr, length); | ||
| 915 | if (rc) | ||
| 916 | goto err_cmd; | ||
| 917 | |||
| 918 | ionic_pgtbl_unbuf(dev, &mr->buf); | ||
| 919 | |||
| 920 | return &mr->ibmr; | ||
| 921 | |||
| 922 | err_cmd: | ||
| 923 | ionic_pgtbl_unbuf(dev, &mr->buf); | ||
| 924 | err_pgtbl: | ||
| 925 | ib_umem_release(mr->umem); | ||
| 926 | err_umem: | ||
| 927 | ionic_put_mrid(dev, mr->mrid); | ||
| 928 | err_mrid: | ||
| 929 | kfree(mr); | ||
| 930 | return ERR_PTR(rc); | ||
| 931 | } | ||
| 932 | |||
| 933 | struct ib_mr *ionic_reg_user_mr_dmabuf(struct ib_pd *ibpd, u64 offset, | ||
| 934 | u64 length, u64 addr, int fd, int access, | ||
| 935 | struct ib_dmah *dmah, | ||
| 936 | struct uverbs_attr_bundle *attrs) | ||
| 937 | { | ||
| 938 | struct ionic_ibdev *dev = to_ionic_ibdev(ibpd->device); | ||
| 939 | struct ionic_pd *pd = to_ionic_pd(ibpd); | ||
| 940 | struct ib_umem_dmabuf *umem_dmabuf; | ||
| 941 | struct ionic_mr *mr; | ||
| 942 | u64 pg_sz; | ||
| 943 | int rc; | ||
| 944 | |||
| 945 | if (dmah) | ||
| 946 | return ERR_PTR(-EOPNOTSUPP); | ||
| 947 | |||
| 948 | mr = kzalloc(sizeof(*mr), GFP_KERNEL); | ||
| 949 | if (!mr) | ||
| 950 | return ERR_PTR(-ENOMEM); | ||
| 951 | |||
| 952 | rc = ionic_get_mrid(dev, &mr->mrid); | ||
| 953 | if (rc) | ||
| 954 | goto err_mrid; | ||
| 955 | |||
| 956 | mr->ibmr.lkey = mr->mrid; | ||
| 957 | mr->ibmr.rkey = mr->mrid; | ||
| 958 | mr->ibmr.iova = addr; | ||
| 959 | mr->ibmr.length = length; | ||
| 960 | |||
| 961 | mr->flags = IONIC_MRF_USER_MR | to_ionic_mr_flags(access); | ||
| 962 | |||
| 963 | umem_dmabuf = ib_umem_dmabuf_get_pinned(&dev->ibdev, offset, length, | ||
| 964 | fd, access); | ||
| 965 | if (IS_ERR(umem_dmabuf)) { | ||
| 966 | rc = PTR_ERR(umem_dmabuf); | ||
| 967 | goto err_umem; | ||
| 968 | } | ||
| 969 | |||
| 970 | mr->umem = &umem_dmabuf->umem; | ||
| 971 | |||
| 972 | pg_sz = ib_umem_find_best_pgsz(mr->umem, | ||
| 973 | dev->lif_cfg.page_size_supported, | ||
| 974 | addr); | ||
| 975 | if (!pg_sz) { | ||
| 976 | rc = -EINVAL; | ||
| 977 | goto err_pgtbl; | ||
| 978 | } | ||
| 979 | |||
| 980 | rc = ionic_pgtbl_init(dev, &mr->buf, mr->umem, 0, 1, pg_sz); | ||
| 981 | if (rc) | ||
| 982 | goto err_pgtbl; | ||
| 983 | |||
| 984 | rc = ionic_create_mr_cmd(dev, pd, mr, addr, length); | ||
| 985 | if (rc) | ||
| 986 | goto err_cmd; | ||
| 987 | |||
| 988 | ionic_pgtbl_unbuf(dev, &mr->buf); | ||
| 989 | |||
| 990 | return &mr->ibmr; | ||
| 991 | |||
| 992 | err_cmd: | ||
| 993 | ionic_pgtbl_unbuf(dev, &mr->buf); | ||
| 994 | err_pgtbl: | ||
| 995 | ib_umem_release(mr->umem); | ||
| 996 | err_umem: | ||
| 997 | ionic_put_mrid(dev, mr->mrid); | ||
| 998 | err_mrid: | ||
| 999 | kfree(mr); | ||
| 1000 | return ERR_PTR(rc); | ||
| 1001 | } | ||
| 1002 | |||
| 1003 | int ionic_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) | ||
| 1004 | { | ||
| 1005 | struct ionic_ibdev *dev = to_ionic_ibdev(ibmr->device); | ||
| 1006 | struct ionic_mr *mr = to_ionic_mr(ibmr); | ||
| 1007 | int rc; | ||
| 1008 | |||
| 1009 | if (!mr->ibmr.lkey) | ||
| 1010 | goto out; | ||
| 1011 | |||
| 1012 | if (mr->created) { | ||
| 1013 | rc = ionic_destroy_mr_cmd(dev, mr->mrid); | ||
| 1014 | if (rc) | ||
| 1015 | return rc; | ||
| 1016 | } | ||
| 1017 | |||
| 1018 | ionic_pgtbl_unbuf(dev, &mr->buf); | ||
| 1019 | |||
| 1020 | if (mr->umem) | ||
| 1021 | ib_umem_release(mr->umem); | ||
| 1022 | |||
| 1023 | ionic_put_mrid(dev, mr->mrid); | ||
| 1024 | |||
| 1025 | out: | ||
| 1026 | kfree(mr); | ||
| 1027 | |||
| 1028 | return 0; | ||
| 1029 | } | ||
| 1030 | |||
| 1031 | struct ib_mr *ionic_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type type, | ||
| 1032 | u32 max_sg) | ||
| 1033 | { | ||
| 1034 | struct ionic_ibdev *dev = to_ionic_ibdev(ibpd->device); | ||
| 1035 | struct ionic_pd *pd = to_ionic_pd(ibpd); | ||
| 1036 | struct ionic_mr *mr; | ||
| 1037 | int rc; | ||
| 1038 | |||
| 1039 | if (type != IB_MR_TYPE_MEM_REG) | ||
| 1040 | return ERR_PTR(-EINVAL); | ||
| 1041 | |||
| 1042 | mr = kzalloc(sizeof(*mr), GFP_KERNEL); | ||
| 1043 | if (!mr) | ||
| 1044 | return ERR_PTR(-ENOMEM); | ||
| 1045 | |||
| 1046 | rc = ionic_get_mrid(dev, &mr->mrid); | ||
| 1047 | if (rc) | ||
| 1048 | goto err_mrid; | ||
| 1049 | |||
| 1050 | mr->ibmr.lkey = mr->mrid; | ||
| 1051 | mr->ibmr.rkey = mr->mrid; | ||
| 1052 | |||
| 1053 | mr->flags = IONIC_MRF_PHYS_MR; | ||
| 1054 | |||
| 1055 | rc = ionic_pgtbl_init(dev, &mr->buf, mr->umem, 0, max_sg, PAGE_SIZE); | ||
| 1056 | if (rc) | ||
| 1057 | goto err_pgtbl; | ||
| 1058 | |||
| 1059 | mr->buf.tbl_pages = 0; | ||
| 1060 | |||
| 1061 | rc = ionic_create_mr_cmd(dev, pd, mr, 0, 0); | ||
| 1062 | if (rc) | ||
| 1063 | goto err_cmd; | ||
| 1064 | |||
| 1065 | return &mr->ibmr; | ||
| 1066 | |||
| 1067 | err_cmd: | ||
| 1068 | ionic_pgtbl_unbuf(dev, &mr->buf); | ||
| 1069 | err_pgtbl: | ||
| 1070 | ionic_put_mrid(dev, mr->mrid); | ||
| 1071 | err_mrid: | ||
| 1072 | kfree(mr); | ||
| 1073 | return ERR_PTR(rc); | ||
| 1074 | } | ||
| 1075 | |||
| 1076 | static int ionic_map_mr_page(struct ib_mr *ibmr, u64 dma) | ||
| 1077 | { | ||
| 1078 | struct ionic_ibdev *dev = to_ionic_ibdev(ibmr->device); | ||
| 1079 | struct ionic_mr *mr = to_ionic_mr(ibmr); | ||
| 1080 | |||
| 1081 | ibdev_dbg(&dev->ibdev, "dma %p\n", (void *)dma); | ||
| 1082 | return ionic_pgtbl_page(&mr->buf, dma); | ||
| 1083 | } | ||
| 1084 | |||
| 1085 | int ionic_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents, | ||
| 1086 | unsigned int *sg_offset) | ||
| 1087 | { | ||
| 1088 | struct ionic_ibdev *dev = to_ionic_ibdev(ibmr->device); | ||
| 1089 | struct ionic_mr *mr = to_ionic_mr(ibmr); | ||
| 1090 | int rc; | ||
| 1091 | |||
| 1092 | /* mr must be allocated using ib_alloc_mr() */ | ||
| 1093 | if (unlikely(!mr->buf.tbl_limit)) | ||
| 1094 | return -EINVAL; | ||
| 1095 | |||
| 1096 | mr->buf.tbl_pages = 0; | ||
| 1097 | |||
| 1098 | if (mr->buf.tbl_buf) | ||
| 1099 | dma_sync_single_for_cpu(dev->lif_cfg.hwdev, mr->buf.tbl_dma, | ||
| 1100 | mr->buf.tbl_size, DMA_TO_DEVICE); | ||
| 1101 | |||
| 1102 | ibdev_dbg(&dev->ibdev, "sg %p nent %d\n", sg, sg_nents); | ||
| 1103 | rc = ib_sg_to_pages(ibmr, sg, sg_nents, sg_offset, ionic_map_mr_page); | ||
| 1104 | |||
| 1105 | mr->buf.page_size_log2 = order_base_2(ibmr->page_size); | ||
| 1106 | |||
| 1107 | if (mr->buf.tbl_buf) | ||
| 1108 | dma_sync_single_for_device(dev->lif_cfg.hwdev, mr->buf.tbl_dma, | ||
| 1109 | mr->buf.tbl_size, DMA_TO_DEVICE); | ||
| 1110 | |||
| 1111 | return rc; | ||
| 1112 | } | ||
| 1113 | |||
| 1114 | int ionic_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata) | ||
| 1115 | { | ||
| 1116 | struct ionic_ibdev *dev = to_ionic_ibdev(ibmw->device); | ||
| 1117 | struct ionic_pd *pd = to_ionic_pd(ibmw->pd); | ||
| 1118 | struct ionic_mr *mr = to_ionic_mw(ibmw); | ||
| 1119 | int rc; | ||
| 1120 | |||
| 1121 | rc = ionic_get_mrid(dev, &mr->mrid); | ||
| 1122 | if (rc) | ||
| 1123 | return rc; | ||
| 1124 | |||
| 1125 | mr->ibmw.rkey = mr->mrid; | ||
| 1126 | |||
| 1127 | if (mr->ibmw.type == IB_MW_TYPE_1) | ||
| 1128 | mr->flags = IONIC_MRF_MW_1; | ||
| 1129 | else | ||
| 1130 | mr->flags = IONIC_MRF_MW_2; | ||
| 1131 | |||
| 1132 | rc = ionic_create_mr_cmd(dev, pd, mr, 0, 0); | ||
| 1133 | if (rc) | ||
| 1134 | goto err_cmd; | ||
| 1135 | |||
| 1136 | return 0; | ||
| 1137 | |||
| 1138 | err_cmd: | ||
| 1139 | ionic_put_mrid(dev, mr->mrid); | ||
| 1140 | return rc; | ||
| 1141 | } | ||
| 1142 | |||
| 1143 | int ionic_dealloc_mw(struct ib_mw *ibmw) | ||
| 1144 | { | ||
| 1145 | struct ionic_ibdev *dev = to_ionic_ibdev(ibmw->device); | ||
| 1146 | struct ionic_mr *mr = to_ionic_mw(ibmw); | ||
| 1147 | int rc; | ||
| 1148 | |||
| 1149 | rc = ionic_destroy_mr_cmd(dev, mr->mrid); | ||
| 1150 | if (rc) | ||
| 1151 | return rc; | ||
| 1152 | |||
| 1153 | ionic_put_mrid(dev, mr->mrid); | ||
| 1154 | |||
| 1155 | return 0; | ||
| 1156 | } | ||
| 1157 | |||
| 1158 | static int ionic_create_cq_cmd(struct ionic_ibdev *dev, | ||
| 1159 | struct ionic_ctx *ctx, | ||
| 1160 | struct ionic_cq *cq, | ||
| 1161 | struct ionic_tbl_buf *buf) | ||
| 1162 | { | ||
| 1163 | const u16 dbid = ionic_ctx_dbid(dev, ctx); | ||
| 1164 | struct ionic_admin_wr wr = { | ||
| 1165 | .work = COMPLETION_INITIALIZER_ONSTACK(wr.work), | ||
| 1166 | .wqe = { | ||
| 1167 | .op = IONIC_V1_ADMIN_CREATE_CQ, | ||
| 1168 | .len = cpu_to_le16(IONIC_ADMIN_CREATE_CQ_IN_V1_LEN), | ||
| 1169 | .cmd.create_cq = { | ||
| 1170 | .eq_id = cpu_to_le32(cq->eqid), | ||
| 1171 | .depth_log2 = cq->q.depth_log2, | ||
| 1172 | .stride_log2 = cq->q.stride_log2, | ||
| 1173 | .page_size_log2 = buf->page_size_log2, | ||
| 1174 | .tbl_index = cpu_to_le32(~0), | ||
| 1175 | .map_count = cpu_to_le32(buf->tbl_pages), | ||
| 1176 | .dma_addr = ionic_pgtbl_dma(buf, 0), | ||
| 1177 | .dbid_flags = cpu_to_le16(dbid), | ||
| 1178 | .id_ver = cpu_to_le32(cq->cqid), | ||
| 1179 | } | ||
| 1180 | } | ||
| 1181 | }; | ||
| 1182 | |||
| 1183 | if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_CREATE_CQ) | ||
| 1184 | return -EBADRQC; | ||
| 1185 | |||
| 1186 | ionic_admin_post(dev, &wr); | ||
| 1187 | |||
| 1188 | return ionic_admin_wait(dev, &wr, 0); | ||
| 1189 | } | ||
| 1190 | |||
| 1191 | static int ionic_destroy_cq_cmd(struct ionic_ibdev *dev, u32 cqid) | ||
| 1192 | { | ||
| 1193 | struct ionic_admin_wr wr = { | ||
| 1194 | .work = COMPLETION_INITIALIZER_ONSTACK(wr.work), | ||
| 1195 | .wqe = { | ||
| 1196 | .op = IONIC_V1_ADMIN_DESTROY_CQ, | ||
| 1197 | .len = cpu_to_le16(IONIC_ADMIN_DESTROY_CQ_IN_V1_LEN), | ||
| 1198 | .cmd.destroy_cq = { | ||
| 1199 | .cq_id = cpu_to_le32(cqid), | ||
| 1200 | }, | ||
| 1201 | } | ||
| 1202 | }; | ||
| 1203 | |||
| 1204 | if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_DESTROY_CQ) | ||
| 1205 | return -EBADRQC; | ||
| 1206 | |||
| 1207 | ionic_admin_post(dev, &wr); | ||
| 1208 | |||
| 1209 | return ionic_admin_wait(dev, &wr, IONIC_ADMIN_F_TEARDOWN); | ||
| 1210 | } | ||
| 1211 | |||
| 1212 | int ionic_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, | ||
| 1213 | struct uverbs_attr_bundle *attrs) | ||
| 1214 | { | ||
| 1215 | struct ionic_ibdev *dev = to_ionic_ibdev(ibcq->device); | ||
| 1216 | struct ib_udata *udata = &attrs->driver_udata; | ||
| 1217 | struct ionic_ctx *ctx = | ||
| 1218 | rdma_udata_to_drv_context(udata, struct ionic_ctx, ibctx); | ||
| 1219 | struct ionic_vcq *vcq = to_ionic_vcq(ibcq); | ||
| 1220 | struct ionic_tbl_buf buf = {}; | ||
| 1221 | struct ionic_cq_resp resp; | ||
| 1222 | struct ionic_cq_req req; | ||
| 1223 | int udma_idx = 0, rc; | ||
| 1224 | |||
| 1225 | if (udata) { | ||
| 1226 | rc = ib_copy_from_udata(&req, udata, sizeof(req)); | ||
| 1227 | if (rc) | ||
| 1228 | return rc; | ||
| 1229 | } | ||
| 1230 | |||
| 1231 | vcq->udma_mask = BIT(dev->lif_cfg.udma_count) - 1; | ||
| 1232 | |||
| 1233 | if (udata) | ||
| 1234 | vcq->udma_mask &= req.udma_mask; | ||
| 1235 | |||
| 1236 | if (!vcq->udma_mask) { | ||
| 1237 | rc = -EINVAL; | ||
| 1238 | goto err_init; | ||
| 1239 | } | ||
| 1240 | |||
| 1241 | for (; udma_idx < dev->lif_cfg.udma_count; ++udma_idx) { | ||
| 1242 | if (!(vcq->udma_mask & BIT(udma_idx))) | ||
| 1243 | continue; | ||
| 1244 | |||
| 1245 | rc = ionic_create_cq_common(vcq, &buf, attr, ctx, udata, | ||
| 1246 | &req.cq[udma_idx], | ||
| 1247 | &resp.cqid[udma_idx], | ||
| 1248 | udma_idx); | ||
| 1249 | if (rc) | ||
| 1250 | goto err_init; | ||
| 1251 | |||
| 1252 | rc = ionic_create_cq_cmd(dev, ctx, &vcq->cq[udma_idx], &buf); | ||
| 1253 | if (rc) | ||
| 1254 | goto err_cmd; | ||
| 1255 | |||
| 1256 | ionic_pgtbl_unbuf(dev, &buf); | ||
| 1257 | } | ||
| 1258 | |||
| 1259 | vcq->ibcq.cqe = attr->cqe; | ||
| 1260 | |||
| 1261 | if (udata) { | ||
| 1262 | resp.udma_mask = vcq->udma_mask; | ||
| 1263 | |||
| 1264 | rc = ib_copy_to_udata(udata, &resp, sizeof(resp)); | ||
| 1265 | if (rc) | ||
| 1266 | goto err_resp; | ||
| 1267 | } | ||
| 1268 | |||
| 1269 | return 0; | ||
| 1270 | |||
| 1271 | err_resp: | ||
| 1272 | while (udma_idx) { | ||
| 1273 | --udma_idx; | ||
| 1274 | if (!(vcq->udma_mask & BIT(udma_idx))) | ||
| 1275 | continue; | ||
| 1276 | ionic_destroy_cq_cmd(dev, vcq->cq[udma_idx].cqid); | ||
| 1277 | err_cmd: | ||
| 1278 | ionic_pgtbl_unbuf(dev, &buf); | ||
| 1279 | ionic_destroy_cq_common(dev, &vcq->cq[udma_idx]); | ||
| 1280 | err_init: | ||
| 1281 | ; | ||
| 1282 | } | ||
| 1283 | |||
| 1284 | return rc; | ||
| 1285 | } | ||
| 1286 | |||
| 1287 | int ionic_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) | ||
| 1288 | { | ||
| 1289 | struct ionic_ibdev *dev = to_ionic_ibdev(ibcq->device); | ||
| 1290 | struct ionic_vcq *vcq = to_ionic_vcq(ibcq); | ||
| 1291 | int udma_idx, rc_tmp, rc = 0; | ||
| 1292 | |||
| 1293 | for (udma_idx = dev->lif_cfg.udma_count; udma_idx; ) { | ||
| 1294 | --udma_idx; | ||
| 1295 | |||
| 1296 | if (!(vcq->udma_mask & BIT(udma_idx))) | ||
| 1297 | continue; | ||
| 1298 | |||
| 1299 | rc_tmp = ionic_destroy_cq_cmd(dev, vcq->cq[udma_idx].cqid); | ||
| 1300 | if (rc_tmp) { | ||
| 1301 | if (!rc) | ||
| 1302 | rc = rc_tmp; | ||
| 1303 | |||
| 1304 | continue; | ||
| 1305 | } | ||
| 1306 | |||
| 1307 | ionic_destroy_cq_common(dev, &vcq->cq[udma_idx]); | ||
| 1308 | } | ||
| 1309 | |||
| 1310 | return rc; | ||
| 1311 | } | ||
| 1312 | |||
| 1313 | static bool pd_remote_privileged(struct ib_pd *pd) | ||
| 1314 | { | ||
| 1315 | return pd->flags & IB_PD_UNSAFE_GLOBAL_RKEY; | ||
| 1316 | } | ||
| 1317 | |||
| 1318 | static int ionic_create_qp_cmd(struct ionic_ibdev *dev, | ||
| 1319 | struct ionic_pd *pd, | ||
| 1320 | struct ionic_cq *send_cq, | ||
| 1321 | struct ionic_cq *recv_cq, | ||
| 1322 | struct ionic_qp *qp, | ||
| 1323 | struct ionic_tbl_buf *sq_buf, | ||
| 1324 | struct ionic_tbl_buf *rq_buf, | ||
| 1325 | struct ib_qp_init_attr *attr) | ||
| 1326 | { | ||
| 1327 | const u16 dbid = ionic_obj_dbid(dev, pd->ibpd.uobject); | ||
| 1328 | const u32 flags = to_ionic_qp_flags(0, 0, | ||
| 1329 | qp->sq_cmb & IONIC_CMB_ENABLE, | ||
| 1330 | qp->rq_cmb & IONIC_CMB_ENABLE, | ||
| 1331 | qp->sq_spec, qp->rq_spec, | ||
| 1332 | pd->flags & IONIC_QPF_PRIVILEGED, | ||
| 1333 | pd_remote_privileged(&pd->ibpd)); | ||
| 1334 | struct ionic_admin_wr wr = { | ||
| 1335 | .work = COMPLETION_INITIALIZER_ONSTACK(wr.work), | ||
| 1336 | .wqe = { | ||
| 1337 | .op = IONIC_V1_ADMIN_CREATE_QP, | ||
| 1338 | .len = cpu_to_le16(IONIC_ADMIN_CREATE_QP_IN_V1_LEN), | ||
| 1339 | .cmd.create_qp = { | ||
| 1340 | .pd_id = cpu_to_le32(pd->pdid), | ||
| 1341 | .priv_flags = cpu_to_be32(flags), | ||
| 1342 | .type_state = to_ionic_qp_type(attr->qp_type), | ||
| 1343 | .dbid_flags = cpu_to_le16(dbid), | ||
| 1344 | .id_ver = cpu_to_le32(qp->qpid), | ||
| 1345 | } | ||
| 1346 | } | ||
| 1347 | }; | ||
| 1348 | |||
| 1349 | if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_CREATE_QP) | ||
| 1350 | return -EBADRQC; | ||
| 1351 | |||
| 1352 | if (qp->has_sq) { | ||
| 1353 | wr.wqe.cmd.create_qp.sq_cq_id = cpu_to_le32(send_cq->cqid); | ||
| 1354 | wr.wqe.cmd.create_qp.sq_depth_log2 = qp->sq.depth_log2; | ||
| 1355 | wr.wqe.cmd.create_qp.sq_stride_log2 = qp->sq.stride_log2; | ||
| 1356 | wr.wqe.cmd.create_qp.sq_page_size_log2 = sq_buf->page_size_log2; | ||
| 1357 | wr.wqe.cmd.create_qp.sq_tbl_index_xrcd_id = cpu_to_le32(~0); | ||
| 1358 | wr.wqe.cmd.create_qp.sq_map_count = | ||
| 1359 | cpu_to_le32(sq_buf->tbl_pages); | ||
| 1360 | wr.wqe.cmd.create_qp.sq_dma_addr = ionic_pgtbl_dma(sq_buf, 0); | ||
| 1361 | } | ||
| 1362 | |||
| 1363 | if (qp->has_rq) { | ||
| 1364 | wr.wqe.cmd.create_qp.rq_cq_id = cpu_to_le32(recv_cq->cqid); | ||
| 1365 | wr.wqe.cmd.create_qp.rq_depth_log2 = qp->rq.depth_log2; | ||
| 1366 | wr.wqe.cmd.create_qp.rq_stride_log2 = qp->rq.stride_log2; | ||
| 1367 | wr.wqe.cmd.create_qp.rq_page_size_log2 = rq_buf->page_size_log2; | ||
| 1368 | wr.wqe.cmd.create_qp.rq_tbl_index_srq_id = cpu_to_le32(~0); | ||
| 1369 | wr.wqe.cmd.create_qp.rq_map_count = | ||
| 1370 | cpu_to_le32(rq_buf->tbl_pages); | ||
| 1371 | wr.wqe.cmd.create_qp.rq_dma_addr = ionic_pgtbl_dma(rq_buf, 0); | ||
| 1372 | } | ||
| 1373 | |||
| 1374 | ionic_admin_post(dev, &wr); | ||
| 1375 | |||
| 1376 | return ionic_admin_wait(dev, &wr, 0); | ||
| 1377 | } | ||
| 1378 | |||
| 1379 | static int ionic_modify_qp_cmd(struct ionic_ibdev *dev, | ||
| 1380 | struct ionic_pd *pd, | ||
| 1381 | struct ionic_qp *qp, | ||
| 1382 | struct ib_qp_attr *attr, | ||
| 1383 | int mask) | ||
| 1384 | { | ||
| 1385 | const u32 flags = to_ionic_qp_flags(attr->qp_access_flags, | ||
| 1386 | attr->en_sqd_async_notify, | ||
| 1387 | qp->sq_cmb & IONIC_CMB_ENABLE, | ||
| 1388 | qp->rq_cmb & IONIC_CMB_ENABLE, | ||
| 1389 | qp->sq_spec, qp->rq_spec, | ||
| 1390 | pd->flags & IONIC_QPF_PRIVILEGED, | ||
| 1391 | pd_remote_privileged(qp->ibqp.pd)); | ||
| 1392 | const u8 state = to_ionic_qp_modify_state(attr->qp_state, | ||
| 1393 | attr->cur_qp_state); | ||
| 1394 | struct ionic_admin_wr wr = { | ||
| 1395 | .work = COMPLETION_INITIALIZER_ONSTACK(wr.work), | ||
| 1396 | .wqe = { | ||
| 1397 | .op = IONIC_V1_ADMIN_MODIFY_QP, | ||
| 1398 | .len = cpu_to_le16(IONIC_ADMIN_MODIFY_QP_IN_V1_LEN), | ||
| 1399 | .cmd.mod_qp = { | ||
| 1400 | .attr_mask = cpu_to_be32(mask), | ||
| 1401 | .access_flags = cpu_to_be16(flags), | ||
| 1402 | .rq_psn = cpu_to_le32(attr->rq_psn), | ||
| 1403 | .sq_psn = cpu_to_le32(attr->sq_psn), | ||
| 1404 | .rate_limit_kbps = | ||
| 1405 | cpu_to_le32(attr->rate_limit), | ||
| 1406 | .pmtu = (attr->path_mtu + 7), | ||
| 1407 | .retry = (attr->retry_cnt | | ||
| 1408 | (attr->rnr_retry << 4)), | ||
| 1409 | .rnr_timer = attr->min_rnr_timer, | ||
| 1410 | .retry_timeout = attr->timeout, | ||
| 1411 | .type_state = state, | ||
| 1412 | .id_ver = cpu_to_le32(qp->qpid), | ||
| 1413 | } | ||
| 1414 | } | ||
| 1415 | }; | ||
| 1416 | const struct ib_global_route *grh = rdma_ah_read_grh(&attr->ah_attr); | ||
| 1417 | void *hdr_buf = NULL; | ||
| 1418 | dma_addr_t hdr_dma = 0; | ||
| 1419 | int rc, hdr_len = 0; | ||
| 1420 | u16 sport; | ||
| 1421 | |||
| 1422 | if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_MODIFY_QP) | ||
| 1423 | return -EBADRQC; | ||
| 1424 | |||
| 1425 | if ((mask & IB_QP_MAX_DEST_RD_ATOMIC) && attr->max_dest_rd_atomic) { | ||
| 1426 | /* Note, round up/down was already done for allocating | ||
| 1427 | * resources on the device. The allocation order is in cache | ||
| 1428 | * line size. We can't use the order of the resource | ||
| 1429 | * allocation to determine the order wqes here, because for | ||
| 1430 | * queue length <= one cache line it is not distinct. | ||
| 1431 | * | ||
| 1432 | * Therefore, order wqes is computed again here. | ||
| 1433 | * | ||
| 1434 | * Account for hole and round up to the next order. | ||
| 1435 | */ | ||
| 1436 | wr.wqe.cmd.mod_qp.rsq_depth = | ||
| 1437 | order_base_2(attr->max_dest_rd_atomic + 1); | ||
| 1438 | wr.wqe.cmd.mod_qp.rsq_index = cpu_to_le32(~0); | ||
| 1439 | } | ||
| 1440 | |||
| 1441 | if ((mask & IB_QP_MAX_QP_RD_ATOMIC) && attr->max_rd_atomic) { | ||
| 1442 | /* Account for hole and round down to the next order */ | ||
| 1443 | wr.wqe.cmd.mod_qp.rrq_depth = | ||
| 1444 | order_base_2(attr->max_rd_atomic + 2) - 1; | ||
| 1445 | wr.wqe.cmd.mod_qp.rrq_index = cpu_to_le32(~0); | ||
| 1446 | } | ||
| 1447 | |||
| 1448 | if (qp->ibqp.qp_type == IB_QPT_RC || qp->ibqp.qp_type == IB_QPT_UC) | ||
| 1449 | wr.wqe.cmd.mod_qp.qkey_dest_qpn = | ||
| 1450 | cpu_to_le32(attr->dest_qp_num); | ||
| 1451 | else | ||
| 1452 | wr.wqe.cmd.mod_qp.qkey_dest_qpn = cpu_to_le32(attr->qkey); | ||
| 1453 | |||
| 1454 | if (mask & IB_QP_AV) { | ||
| 1455 | if (!qp->hdr) | ||
| 1456 | return -ENOMEM; | ||
| 1457 | |||
| 1458 | sport = rdma_get_udp_sport(grh->flow_label, | ||
| 1459 | qp->qpid, | ||
| 1460 | attr->dest_qp_num); | ||
| 1461 | |||
| 1462 | rc = ionic_build_hdr(dev, qp->hdr, &attr->ah_attr, sport, true); | ||
| 1463 | if (rc) | ||
| 1464 | return rc; | ||
| 1465 | |||
| 1466 | qp->sgid_index = grh->sgid_index; | ||
| 1467 | |||
| 1468 | hdr_buf = kmalloc(PAGE_SIZE, GFP_KERNEL); | ||
| 1469 | if (!hdr_buf) | ||
| 1470 | return -ENOMEM; | ||
| 1471 | |||
| 1472 | hdr_len = ib_ud_header_pack(qp->hdr, hdr_buf); | ||
| 1473 | hdr_len -= IB_BTH_BYTES; | ||
| 1474 | hdr_len -= IB_DETH_BYTES; | ||
| 1475 | ibdev_dbg(&dev->ibdev, "roce packet header template\n"); | ||
| 1476 | print_hex_dump_debug("hdr ", DUMP_PREFIX_OFFSET, 16, 1, | ||
| 1477 | hdr_buf, hdr_len, true); | ||
| 1478 | |||
| 1479 | hdr_dma = dma_map_single(dev->lif_cfg.hwdev, hdr_buf, hdr_len, | ||
| 1480 | DMA_TO_DEVICE); | ||
| 1481 | |||
| 1482 | rc = dma_mapping_error(dev->lif_cfg.hwdev, hdr_dma); | ||
| 1483 | if (rc) | ||
| 1484 | goto err_dma; | ||
| 1485 | |||
| 1486 | if (qp->hdr->ipv4_present) { | ||
| 1487 | wr.wqe.cmd.mod_qp.tfp_csum_profile = | ||
| 1488 | qp->hdr->vlan_present ? | ||
| 1489 | IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_UDP : | ||
| 1490 | IONIC_TFP_CSUM_PROF_ETH_IPV4_UDP; | ||
| 1491 | } else { | ||
| 1492 | wr.wqe.cmd.mod_qp.tfp_csum_profile = | ||
| 1493 | qp->hdr->vlan_present ? | ||
| 1494 | IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV6_UDP : | ||
| 1495 | IONIC_TFP_CSUM_PROF_ETH_IPV6_UDP; | ||
| 1496 | } | ||
| 1497 | |||
| 1498 | wr.wqe.cmd.mod_qp.ah_id_len = | ||
| 1499 | cpu_to_le32(qp->ahid | (hdr_len << 24)); | ||
| 1500 | wr.wqe.cmd.mod_qp.dma_addr = cpu_to_le64(hdr_dma); | ||
| 1501 | |||
| 1502 | wr.wqe.cmd.mod_qp.en_pcp = attr->ah_attr.sl; | ||
| 1503 | wr.wqe.cmd.mod_qp.ip_dscp = grh->traffic_class >> 2; | ||
| 1504 | } | ||
| 1505 | |||
| 1506 | ionic_admin_post(dev, &wr); | ||
| 1507 | |||
| 1508 | rc = ionic_admin_wait(dev, &wr, 0); | ||
| 1509 | |||
| 1510 | if (mask & IB_QP_AV) | ||
| 1511 | dma_unmap_single(dev->lif_cfg.hwdev, hdr_dma, hdr_len, | ||
| 1512 | DMA_TO_DEVICE); | ||
| 1513 | err_dma: | ||
| 1514 | if (mask & IB_QP_AV) | ||
| 1515 | kfree(hdr_buf); | ||
| 1516 | |||
| 1517 | return rc; | ||
| 1518 | } | ||
| 1519 | |||
| 1520 | static int ionic_query_qp_cmd(struct ionic_ibdev *dev, | ||
| 1521 | struct ionic_qp *qp, | ||
| 1522 | struct ib_qp_attr *attr, | ||
| 1523 | int mask) | ||
| 1524 | { | ||
| 1525 | struct ionic_admin_wr wr = { | ||
| 1526 | .work = COMPLETION_INITIALIZER_ONSTACK(wr.work), | ||
| 1527 | .wqe = { | ||
| 1528 | .op = IONIC_V1_ADMIN_QUERY_QP, | ||
| 1529 | .len = cpu_to_le16(IONIC_ADMIN_QUERY_QP_IN_V1_LEN), | ||
| 1530 | .cmd.query_qp = { | ||
| 1531 | .id_ver = cpu_to_le32(qp->qpid), | ||
| 1532 | }, | ||
| 1533 | } | ||
| 1534 | }; | ||
| 1535 | struct ionic_v1_admin_query_qp_sq *query_sqbuf; | ||
| 1536 | struct ionic_v1_admin_query_qp_rq *query_rqbuf; | ||
| 1537 | dma_addr_t query_sqdma; | ||
| 1538 | dma_addr_t query_rqdma; | ||
| 1539 | dma_addr_t hdr_dma = 0; | ||
| 1540 | void *hdr_buf = NULL; | ||
| 1541 | int flags, rc; | ||
| 1542 | |||
| 1543 | if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_QUERY_QP) | ||
| 1544 | return -EBADRQC; | ||
| 1545 | |||
| 1546 | if (qp->has_sq) { | ||
| 1547 | bool expdb = !!(qp->sq_cmb & IONIC_CMB_EXPDB); | ||
| 1548 | |||
| 1549 | attr->cap.max_send_sge = | ||
| 1550 | ionic_v1_send_wqe_max_sge(qp->sq.stride_log2, | ||
| 1551 | qp->sq_spec, | ||
| 1552 | expdb); | ||
| 1553 | attr->cap.max_inline_data = | ||
| 1554 | ionic_v1_send_wqe_max_data(qp->sq.stride_log2, expdb); | ||
| 1555 | } | ||
| 1556 | |||
| 1557 | if (qp->has_rq) { | ||
| 1558 | attr->cap.max_recv_sge = | ||
| 1559 | ionic_v1_recv_wqe_max_sge(qp->rq.stride_log2, | ||
| 1560 | qp->rq_spec, | ||
| 1561 | qp->rq_cmb & IONIC_CMB_EXPDB); | ||
| 1562 | } | ||
| 1563 | |||
| 1564 | query_sqbuf = kzalloc(PAGE_SIZE, GFP_KERNEL); | ||
| 1565 | if (!query_sqbuf) | ||
| 1566 | return -ENOMEM; | ||
| 1567 | |||
| 1568 | query_rqbuf = kzalloc(PAGE_SIZE, GFP_KERNEL); | ||
| 1569 | if (!query_rqbuf) { | ||
| 1570 | rc = -ENOMEM; | ||
| 1571 | goto err_rqbuf; | ||
| 1572 | } | ||
| 1573 | |||
| 1574 | query_sqdma = dma_map_single(dev->lif_cfg.hwdev, query_sqbuf, PAGE_SIZE, | ||
| 1575 | DMA_FROM_DEVICE); | ||
| 1576 | rc = dma_mapping_error(dev->lif_cfg.hwdev, query_sqdma); | ||
| 1577 | if (rc) | ||
| 1578 | goto err_sqdma; | ||
| 1579 | |||
| 1580 | query_rqdma = dma_map_single(dev->lif_cfg.hwdev, query_rqbuf, PAGE_SIZE, | ||
| 1581 | DMA_FROM_DEVICE); | ||
| 1582 | rc = dma_mapping_error(dev->lif_cfg.hwdev, query_rqdma); | ||
| 1583 | if (rc) | ||
| 1584 | goto err_rqdma; | ||
| 1585 | |||
| 1586 | if (mask & IB_QP_AV) { | ||
| 1587 | hdr_buf = kmalloc(PAGE_SIZE, GFP_KERNEL); | ||
| 1588 | if (!hdr_buf) { | ||
| 1589 | rc = -ENOMEM; | ||
| 1590 | goto err_hdrbuf; | ||
| 1591 | } | ||
| 1592 | |||
| 1593 | hdr_dma = dma_map_single(dev->lif_cfg.hwdev, hdr_buf, | ||
| 1594 | PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 1595 | rc = dma_mapping_error(dev->lif_cfg.hwdev, hdr_dma); | ||
| 1596 | if (rc) | ||
| 1597 | goto err_hdrdma; | ||
| 1598 | } | ||
| 1599 | |||
| 1600 | wr.wqe.cmd.query_qp.sq_dma_addr = cpu_to_le64(query_sqdma); | ||
| 1601 | wr.wqe.cmd.query_qp.rq_dma_addr = cpu_to_le64(query_rqdma); | ||
| 1602 | wr.wqe.cmd.query_qp.hdr_dma_addr = cpu_to_le64(hdr_dma); | ||
| 1603 | wr.wqe.cmd.query_qp.ah_id = cpu_to_le32(qp->ahid); | ||
| 1604 | |||
| 1605 | ionic_admin_post(dev, &wr); | ||
| 1606 | |||
| 1607 | rc = ionic_admin_wait(dev, &wr, 0); | ||
| 1608 | |||
| 1609 | if (rc) | ||
| 1610 | goto err_hdrdma; | ||
| 1611 | |||
| 1612 | flags = be16_to_cpu(query_sqbuf->access_perms_flags | | ||
| 1613 | query_rqbuf->access_perms_flags); | ||
| 1614 | |||
| 1615 | print_hex_dump_debug("sqbuf ", DUMP_PREFIX_OFFSET, 16, 1, | ||
| 1616 | query_sqbuf, sizeof(*query_sqbuf), true); | ||
| 1617 | print_hex_dump_debug("rqbuf ", DUMP_PREFIX_OFFSET, 16, 1, | ||
| 1618 | query_rqbuf, sizeof(*query_rqbuf), true); | ||
| 1619 | ibdev_dbg(&dev->ibdev, "query qp %u state_pmtu %#x flags %#x", | ||
| 1620 | qp->qpid, query_rqbuf->state_pmtu, flags); | ||
| 1621 | |||
| 1622 | attr->qp_state = from_ionic_qp_state(query_rqbuf->state_pmtu >> 4); | ||
| 1623 | attr->cur_qp_state = attr->qp_state; | ||
| 1624 | attr->path_mtu = (query_rqbuf->state_pmtu & 0xf) - 7; | ||
| 1625 | attr->path_mig_state = IB_MIG_MIGRATED; | ||
| 1626 | attr->qkey = be32_to_cpu(query_sqbuf->qkey_dest_qpn); | ||
| 1627 | attr->rq_psn = be32_to_cpu(query_sqbuf->rq_psn); | ||
| 1628 | attr->sq_psn = be32_to_cpu(query_rqbuf->sq_psn); | ||
| 1629 | attr->dest_qp_num = attr->qkey; | ||
| 1630 | attr->qp_access_flags = from_ionic_qp_flags(flags); | ||
| 1631 | attr->pkey_index = 0; | ||
| 1632 | attr->alt_pkey_index = 0; | ||
| 1633 | attr->en_sqd_async_notify = !!(flags & IONIC_QPF_SQD_NOTIFY); | ||
| 1634 | attr->sq_draining = !!(flags & IONIC_QPF_SQ_DRAINING); | ||
| 1635 | attr->max_rd_atomic = BIT(query_rqbuf->rrq_depth) - 1; | ||
| 1636 | attr->max_dest_rd_atomic = BIT(query_rqbuf->rsq_depth) - 1; | ||
| 1637 | attr->min_rnr_timer = query_sqbuf->rnr_timer; | ||
| 1638 | attr->port_num = 0; | ||
| 1639 | attr->timeout = query_sqbuf->retry_timeout; | ||
| 1640 | attr->retry_cnt = query_rqbuf->retry_rnrtry & 0xf; | ||
| 1641 | attr->rnr_retry = query_rqbuf->retry_rnrtry >> 4; | ||
| 1642 | attr->alt_port_num = 0; | ||
| 1643 | attr->alt_timeout = 0; | ||
| 1644 | attr->rate_limit = be32_to_cpu(query_sqbuf->rate_limit_kbps); | ||
| 1645 | |||
| 1646 | if (mask & IB_QP_AV) | ||
| 1647 | ionic_set_ah_attr(dev, &attr->ah_attr, | ||
| 1648 | qp->hdr, qp->sgid_index); | ||
| 1649 | |||
| 1650 | err_hdrdma: | ||
| 1651 | if (mask & IB_QP_AV) { | ||
| 1652 | dma_unmap_single(dev->lif_cfg.hwdev, hdr_dma, | ||
| 1653 | PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 1654 | kfree(hdr_buf); | ||
| 1655 | } | ||
| 1656 | err_hdrbuf: | ||
| 1657 | dma_unmap_single(dev->lif_cfg.hwdev, query_rqdma, sizeof(*query_rqbuf), | ||
| 1658 | DMA_FROM_DEVICE); | ||
| 1659 | err_rqdma: | ||
| 1660 | dma_unmap_single(dev->lif_cfg.hwdev, query_sqdma, sizeof(*query_sqbuf), | ||
| 1661 | DMA_FROM_DEVICE); | ||
| 1662 | err_sqdma: | ||
| 1663 | kfree(query_rqbuf); | ||
| 1664 | err_rqbuf: | ||
| 1665 | kfree(query_sqbuf); | ||
| 1666 | |||
| 1667 | return rc; | ||
| 1668 | } | ||
| 1669 | |||
| 1670 | static int ionic_destroy_qp_cmd(struct ionic_ibdev *dev, u32 qpid) | ||
| 1671 | { | ||
| 1672 | struct ionic_admin_wr wr = { | ||
| 1673 | .work = COMPLETION_INITIALIZER_ONSTACK(wr.work), | ||
| 1674 | .wqe = { | ||
| 1675 | .op = IONIC_V1_ADMIN_DESTROY_QP, | ||
| 1676 | .len = cpu_to_le16(IONIC_ADMIN_DESTROY_QP_IN_V1_LEN), | ||
| 1677 | .cmd.destroy_qp = { | ||
| 1678 | .qp_id = cpu_to_le32(qpid), | ||
| 1679 | }, | ||
| 1680 | } | ||
| 1681 | }; | ||
| 1682 | |||
| 1683 | if (dev->lif_cfg.admin_opcodes <= IONIC_V1_ADMIN_DESTROY_QP) | ||
| 1684 | return -EBADRQC; | ||
| 1685 | |||
| 1686 | ionic_admin_post(dev, &wr); | ||
| 1687 | |||
| 1688 | return ionic_admin_wait(dev, &wr, IONIC_ADMIN_F_TEARDOWN); | ||
| 1689 | } | ||
| 1690 | |||
| 1691 | static bool ionic_expdb_wqe_size_supported(struct ionic_ibdev *dev, | ||
| 1692 | uint32_t wqe_size) | ||
| 1693 | { | ||
| 1694 | switch (wqe_size) { | ||
| 1695 | case 64: return dev->lif_cfg.expdb_mask & IONIC_EXPDB_64; | ||
| 1696 | case 128: return dev->lif_cfg.expdb_mask & IONIC_EXPDB_128; | ||
| 1697 | case 256: return dev->lif_cfg.expdb_mask & IONIC_EXPDB_256; | ||
| 1698 | case 512: return dev->lif_cfg.expdb_mask & IONIC_EXPDB_512; | ||
| 1699 | } | ||
| 1700 | |||
| 1701 | return false; | ||
| 1702 | } | ||
| 1703 | |||
| 1704 | static void ionic_qp_sq_init_cmb(struct ionic_ibdev *dev, | ||
| 1705 | struct ionic_qp *qp, | ||
| 1706 | struct ib_udata *udata, | ||
| 1707 | int max_data) | ||
| 1708 | { | ||
| 1709 | u8 expdb_stride_log2 = 0; | ||
| 1710 | bool expdb; | ||
| 1711 | int rc; | ||
| 1712 | |||
| 1713 | if (!(qp->sq_cmb & IONIC_CMB_ENABLE)) | ||
| 1714 | goto not_in_cmb; | ||
| 1715 | |||
| 1716 | if (qp->sq_cmb & ~IONIC_CMB_SUPPORTED) { | ||
| 1717 | if (qp->sq_cmb & IONIC_CMB_REQUIRE) | ||
| 1718 | goto not_in_cmb; | ||
| 1719 | |||
| 1720 | qp->sq_cmb &= IONIC_CMB_SUPPORTED; | ||
| 1721 | } | ||
| 1722 | |||
| 1723 | if ((qp->sq_cmb & IONIC_CMB_EXPDB) && !dev->lif_cfg.sq_expdb) { | ||
| 1724 | if (qp->sq_cmb & IONIC_CMB_REQUIRE) | ||
| 1725 | goto not_in_cmb; | ||
| 1726 | |||
| 1727 | qp->sq_cmb &= ~IONIC_CMB_EXPDB; | ||
| 1728 | } | ||
| 1729 | |||
| 1730 | qp->sq_cmb_order = order_base_2(qp->sq.size / PAGE_SIZE); | ||
| 1731 | |||
| 1732 | if (qp->sq_cmb_order >= IONIC_SQCMB_ORDER) | ||
| 1733 | goto not_in_cmb; | ||
| 1734 | |||
| 1735 | if (qp->sq_cmb & IONIC_CMB_EXPDB) | ||
| 1736 | expdb_stride_log2 = qp->sq.stride_log2; | ||
| 1737 | |||
| 1738 | rc = ionic_get_cmb(dev->lif_cfg.lif, &qp->sq_cmb_pgid, | ||
| 1739 | &qp->sq_cmb_addr, qp->sq_cmb_order, | ||
| 1740 | expdb_stride_log2, &expdb); | ||
| 1741 | if (rc) | ||
| 1742 | goto not_in_cmb; | ||
| 1743 | |||
| 1744 | if ((qp->sq_cmb & IONIC_CMB_EXPDB) && !expdb) { | ||
| 1745 | if (qp->sq_cmb & IONIC_CMB_REQUIRE) | ||
| 1746 | goto err_map; | ||
| 1747 | |||
| 1748 | qp->sq_cmb &= ~IONIC_CMB_EXPDB; | ||
| 1749 | } | ||
| 1750 | |||
| 1751 | return; | ||
| 1752 | |||
| 1753 | err_map: | ||
| 1754 | ionic_put_cmb(dev->lif_cfg.lif, qp->sq_cmb_pgid, qp->sq_cmb_order); | ||
| 1755 | not_in_cmb: | ||
| 1756 | if (qp->sq_cmb & IONIC_CMB_REQUIRE) | ||
| 1757 | ibdev_dbg(&dev->ibdev, "could not place sq in cmb as required\n"); | ||
| 1758 | |||
| 1759 | qp->sq_cmb = 0; | ||
| 1760 | qp->sq_cmb_order = IONIC_RES_INVALID; | ||
| 1761 | qp->sq_cmb_pgid = 0; | ||
| 1762 | qp->sq_cmb_addr = 0; | ||
| 1763 | } | ||
| 1764 | |||
| 1765 | static void ionic_qp_sq_destroy_cmb(struct ionic_ibdev *dev, | ||
| 1766 | struct ionic_ctx *ctx, | ||
| 1767 | struct ionic_qp *qp) | ||
| 1768 | { | ||
| 1769 | if (!(qp->sq_cmb & IONIC_CMB_ENABLE)) | ||
| 1770 | return; | ||
| 1771 | |||
| 1772 | if (ctx) | ||
| 1773 | rdma_user_mmap_entry_remove(qp->mmap_sq_cmb); | ||
| 1774 | |||
| 1775 | ionic_put_cmb(dev->lif_cfg.lif, qp->sq_cmb_pgid, qp->sq_cmb_order); | ||
| 1776 | } | ||
| 1777 | |||
| 1778 | static int ionic_qp_sq_init(struct ionic_ibdev *dev, struct ionic_ctx *ctx, | ||
| 1779 | struct ionic_qp *qp, struct ionic_qdesc *sq, | ||
| 1780 | struct ionic_tbl_buf *buf, int max_wr, int max_sge, | ||
| 1781 | int max_data, int sq_spec, struct ib_udata *udata) | ||
| 1782 | { | ||
| 1783 | u32 wqe_size; | ||
| 1784 | int rc = 0; | ||
| 1785 | |||
| 1786 | qp->sq_msn_prod = 0; | ||
| 1787 | qp->sq_msn_cons = 0; | ||
| 1788 | |||
| 1789 | if (!qp->has_sq) { | ||
| 1790 | if (buf) { | ||
| 1791 | buf->tbl_buf = NULL; | ||
| 1792 | buf->tbl_limit = 0; | ||
| 1793 | buf->tbl_pages = 0; | ||
| 1794 | } | ||
| 1795 | if (udata) | ||
| 1796 | rc = ionic_validate_qdesc_zero(sq); | ||
| 1797 | |||
| 1798 | return rc; | ||
| 1799 | } | ||
| 1800 | |||
| 1801 | rc = -EINVAL; | ||
| 1802 | |||
| 1803 | if (max_wr < 0 || max_wr > 0xffff) | ||
| 1804 | return rc; | ||
| 1805 | |||
| 1806 | if (max_sge < 1) | ||
| 1807 | return rc; | ||
| 1808 | |||
| 1809 | if (max_sge > min(ionic_v1_send_wqe_max_sge(dev->lif_cfg.max_stride, 0, | ||
| 1810 | qp->sq_cmb & | ||
| 1811 | IONIC_CMB_EXPDB), | ||
| 1812 | IONIC_SPEC_HIGH)) | ||
| 1813 | return rc; | ||
| 1814 | |||
| 1815 | if (max_data < 0) | ||
| 1816 | return rc; | ||
| 1817 | |||
| 1818 | if (max_data > ionic_v1_send_wqe_max_data(dev->lif_cfg.max_stride, | ||
| 1819 | qp->sq_cmb & IONIC_CMB_EXPDB)) | ||
| 1820 | return rc; | ||
| 1821 | |||
| 1822 | if (udata) { | ||
| 1823 | rc = ionic_validate_qdesc(sq); | ||
| 1824 | if (rc) | ||
| 1825 | return rc; | ||
| 1826 | |||
| 1827 | qp->sq_spec = sq_spec; | ||
| 1828 | |||
| 1829 | qp->sq.ptr = NULL; | ||
| 1830 | qp->sq.size = sq->size; | ||
| 1831 | qp->sq.mask = sq->mask; | ||
| 1832 | qp->sq.depth_log2 = sq->depth_log2; | ||
| 1833 | qp->sq.stride_log2 = sq->stride_log2; | ||
| 1834 | |||
| 1835 | qp->sq_meta = NULL; | ||
| 1836 | qp->sq_msn_idx = NULL; | ||
| 1837 | |||
| 1838 | qp->sq_umem = ib_umem_get(&dev->ibdev, sq->addr, sq->size, 0); | ||
| 1839 | if (IS_ERR(qp->sq_umem)) | ||
| 1840 | return PTR_ERR(qp->sq_umem); | ||
| 1841 | } else { | ||
| 1842 | qp->sq_umem = NULL; | ||
| 1843 | |||
| 1844 | qp->sq_spec = ionic_v1_use_spec_sge(max_sge, sq_spec); | ||
| 1845 | if (sq_spec && !qp->sq_spec) | ||
| 1846 | ibdev_dbg(&dev->ibdev, | ||
| 1847 | "init sq: max_sge %u disables spec\n", | ||
| 1848 | max_sge); | ||
| 1849 | |||
| 1850 | if (qp->sq_cmb & IONIC_CMB_EXPDB) { | ||
| 1851 | wqe_size = ionic_v1_send_wqe_min_size(max_sge, max_data, | ||
| 1852 | qp->sq_spec, | ||
| 1853 | true); | ||
| 1854 | |||
| 1855 | if (!ionic_expdb_wqe_size_supported(dev, wqe_size)) | ||
| 1856 | qp->sq_cmb &= ~IONIC_CMB_EXPDB; | ||
| 1857 | } | ||
| 1858 | |||
| 1859 | if (!(qp->sq_cmb & IONIC_CMB_EXPDB)) | ||
| 1860 | wqe_size = ionic_v1_send_wqe_min_size(max_sge, max_data, | ||
| 1861 | qp->sq_spec, | ||
| 1862 | false); | ||
| 1863 | |||
| 1864 | rc = ionic_queue_init(&qp->sq, dev->lif_cfg.hwdev, | ||
| 1865 | max_wr, wqe_size); | ||
| 1866 | if (rc) | ||
| 1867 | return rc; | ||
| 1868 | |||
| 1869 | ionic_queue_dbell_init(&qp->sq, qp->qpid); | ||
| 1870 | |||
| 1871 | qp->sq_meta = kmalloc_array((u32)qp->sq.mask + 1, | ||
| 1872 | sizeof(*qp->sq_meta), | ||
| 1873 | GFP_KERNEL); | ||
| 1874 | if (!qp->sq_meta) { | ||
| 1875 | rc = -ENOMEM; | ||
| 1876 | goto err_sq_meta; | ||
| 1877 | } | ||
| 1878 | |||
| 1879 | qp->sq_msn_idx = kmalloc_array((u32)qp->sq.mask + 1, | ||
| 1880 | sizeof(*qp->sq_msn_idx), | ||
| 1881 | GFP_KERNEL); | ||
| 1882 | if (!qp->sq_msn_idx) { | ||
| 1883 | rc = -ENOMEM; | ||
| 1884 | goto err_sq_msn; | ||
| 1885 | } | ||
| 1886 | } | ||
| 1887 | |||
| 1888 | ionic_qp_sq_init_cmb(dev, qp, udata, max_data); | ||
| 1889 | |||
| 1890 | if (qp->sq_cmb & IONIC_CMB_ENABLE) | ||
| 1891 | rc = ionic_pgtbl_init(dev, buf, NULL, | ||
| 1892 | (u64)qp->sq_cmb_pgid << PAGE_SHIFT, | ||
| 1893 | 1, PAGE_SIZE); | ||
| 1894 | else | ||
| 1895 | rc = ionic_pgtbl_init(dev, buf, | ||
| 1896 | qp->sq_umem, qp->sq.dma, 1, PAGE_SIZE); | ||
| 1897 | if (rc) | ||
| 1898 | goto err_sq_tbl; | ||
| 1899 | |||
| 1900 | return 0; | ||
| 1901 | |||
| 1902 | err_sq_tbl: | ||
| 1903 | ionic_qp_sq_destroy_cmb(dev, ctx, qp); | ||
| 1904 | kfree(qp->sq_msn_idx); | ||
| 1905 | err_sq_msn: | ||
| 1906 | kfree(qp->sq_meta); | ||
| 1907 | err_sq_meta: | ||
| 1908 | if (qp->sq_umem) | ||
| 1909 | ib_umem_release(qp->sq_umem); | ||
| 1910 | else | ||
| 1911 | ionic_queue_destroy(&qp->sq, dev->lif_cfg.hwdev); | ||
| 1912 | return rc; | ||
| 1913 | } | ||
| 1914 | |||
| 1915 | static void ionic_qp_sq_destroy(struct ionic_ibdev *dev, | ||
| 1916 | struct ionic_ctx *ctx, | ||
| 1917 | struct ionic_qp *qp) | ||
| 1918 | { | ||
| 1919 | if (!qp->has_sq) | ||
| 1920 | return; | ||
| 1921 | |||
| 1922 | ionic_qp_sq_destroy_cmb(dev, ctx, qp); | ||
| 1923 | |||
| 1924 | kfree(qp->sq_msn_idx); | ||
| 1925 | kfree(qp->sq_meta); | ||
| 1926 | |||
| 1927 | if (qp->sq_umem) | ||
| 1928 | ib_umem_release(qp->sq_umem); | ||
| 1929 | else | ||
| 1930 | ionic_queue_destroy(&qp->sq, dev->lif_cfg.hwdev); | ||
| 1931 | } | ||
| 1932 | |||
| 1933 | static void ionic_qp_rq_init_cmb(struct ionic_ibdev *dev, | ||
| 1934 | struct ionic_qp *qp, | ||
| 1935 | struct ib_udata *udata) | ||
| 1936 | { | ||
| 1937 | u8 expdb_stride_log2 = 0; | ||
| 1938 | bool expdb; | ||
| 1939 | int rc; | ||
| 1940 | |||
| 1941 | if (!(qp->rq_cmb & IONIC_CMB_ENABLE)) | ||
| 1942 | goto not_in_cmb; | ||
| 1943 | |||
| 1944 | if (qp->rq_cmb & ~IONIC_CMB_SUPPORTED) { | ||
| 1945 | if (qp->rq_cmb & IONIC_CMB_REQUIRE) | ||
| 1946 | goto not_in_cmb; | ||
| 1947 | |||
| 1948 | qp->rq_cmb &= IONIC_CMB_SUPPORTED; | ||
| 1949 | } | ||
| 1950 | |||
| 1951 | if ((qp->rq_cmb & IONIC_CMB_EXPDB) && !dev->lif_cfg.rq_expdb) { | ||
| 1952 | if (qp->rq_cmb & IONIC_CMB_REQUIRE) | ||
| 1953 | goto not_in_cmb; | ||
| 1954 | |||
| 1955 | qp->rq_cmb &= ~IONIC_CMB_EXPDB; | ||
| 1956 | } | ||
| 1957 | |||
| 1958 | qp->rq_cmb_order = order_base_2(qp->rq.size / PAGE_SIZE); | ||
| 1959 | |||
| 1960 | if (qp->rq_cmb_order >= IONIC_RQCMB_ORDER) | ||
| 1961 | goto not_in_cmb; | ||
| 1962 | |||
| 1963 | if (qp->rq_cmb & IONIC_CMB_EXPDB) | ||
| 1964 | expdb_stride_log2 = qp->rq.stride_log2; | ||
| 1965 | |||
| 1966 | rc = ionic_get_cmb(dev->lif_cfg.lif, &qp->rq_cmb_pgid, | ||
| 1967 | &qp->rq_cmb_addr, qp->rq_cmb_order, | ||
| 1968 | expdb_stride_log2, &expdb); | ||
| 1969 | if (rc) | ||
| 1970 | goto not_in_cmb; | ||
| 1971 | |||
| 1972 | if ((qp->rq_cmb & IONIC_CMB_EXPDB) && !expdb) { | ||
| 1973 | if (qp->rq_cmb & IONIC_CMB_REQUIRE) | ||
| 1974 | goto err_map; | ||
| 1975 | |||
| 1976 | qp->rq_cmb &= ~IONIC_CMB_EXPDB; | ||
| 1977 | } | ||
| 1978 | |||
| 1979 | return; | ||
| 1980 | |||
| 1981 | err_map: | ||
| 1982 | ionic_put_cmb(dev->lif_cfg.lif, qp->rq_cmb_pgid, qp->rq_cmb_order); | ||
| 1983 | not_in_cmb: | ||
| 1984 | if (qp->rq_cmb & IONIC_CMB_REQUIRE) | ||
| 1985 | ibdev_dbg(&dev->ibdev, "could not place rq in cmb as required\n"); | ||
| 1986 | |||
| 1987 | qp->rq_cmb = 0; | ||
| 1988 | qp->rq_cmb_order = IONIC_RES_INVALID; | ||
| 1989 | qp->rq_cmb_pgid = 0; | ||
| 1990 | qp->rq_cmb_addr = 0; | ||
| 1991 | } | ||
| 1992 | |||
| 1993 | static void ionic_qp_rq_destroy_cmb(struct ionic_ibdev *dev, | ||
| 1994 | struct ionic_ctx *ctx, | ||
| 1995 | struct ionic_qp *qp) | ||
| 1996 | { | ||
| 1997 | if (!(qp->rq_cmb & IONIC_CMB_ENABLE)) | ||
| 1998 | return; | ||
| 1999 | |||
| 2000 | if (ctx) | ||
| 2001 | rdma_user_mmap_entry_remove(qp->mmap_rq_cmb); | ||
| 2002 | |||
| 2003 | ionic_put_cmb(dev->lif_cfg.lif, qp->rq_cmb_pgid, qp->rq_cmb_order); | ||
| 2004 | } | ||
| 2005 | |||
| 2006 | static int ionic_qp_rq_init(struct ionic_ibdev *dev, struct ionic_ctx *ctx, | ||
| 2007 | struct ionic_qp *qp, struct ionic_qdesc *rq, | ||
| 2008 | struct ionic_tbl_buf *buf, int max_wr, int max_sge, | ||
| 2009 | int rq_spec, struct ib_udata *udata) | ||
| 2010 | { | ||
| 2011 | int rc = 0, i; | ||
| 2012 | u32 wqe_size; | ||
| 2013 | |||
| 2014 | if (!qp->has_rq) { | ||
| 2015 | if (buf) { | ||
| 2016 | buf->tbl_buf = NULL; | ||
| 2017 | buf->tbl_limit = 0; | ||
| 2018 | buf->tbl_pages = 0; | ||
| 2019 | } | ||
| 2020 | if (udata) | ||
| 2021 | rc = ionic_validate_qdesc_zero(rq); | ||
| 2022 | |||
| 2023 | return rc; | ||
| 2024 | } | ||
| 2025 | |||
| 2026 | rc = -EINVAL; | ||
| 2027 | |||
| 2028 | if (max_wr < 0 || max_wr > 0xffff) | ||
| 2029 | return rc; | ||
| 2030 | |||
| 2031 | if (max_sge < 1) | ||
| 2032 | return rc; | ||
| 2033 | |||
| 2034 | if (max_sge > min(ionic_v1_recv_wqe_max_sge(dev->lif_cfg.max_stride, 0, false), | ||
| 2035 | IONIC_SPEC_HIGH)) | ||
| 2036 | return rc; | ||
| 2037 | |||
| 2038 | if (udata) { | ||
| 2039 | rc = ionic_validate_qdesc(rq); | ||
| 2040 | if (rc) | ||
| 2041 | return rc; | ||
| 2042 | |||
| 2043 | qp->rq_spec = rq_spec; | ||
| 2044 | |||
| 2045 | qp->rq.ptr = NULL; | ||
| 2046 | qp->rq.size = rq->size; | ||
| 2047 | qp->rq.mask = rq->mask; | ||
| 2048 | qp->rq.depth_log2 = rq->depth_log2; | ||
| 2049 | qp->rq.stride_log2 = rq->stride_log2; | ||
| 2050 | |||
| 2051 | qp->rq_meta = NULL; | ||
| 2052 | |||
| 2053 | qp->rq_umem = ib_umem_get(&dev->ibdev, rq->addr, rq->size, 0); | ||
| 2054 | if (IS_ERR(qp->rq_umem)) | ||
| 2055 | return PTR_ERR(qp->rq_umem); | ||
| 2056 | } else { | ||
| 2057 | qp->rq_umem = NULL; | ||
| 2058 | |||
| 2059 | qp->rq_spec = ionic_v1_use_spec_sge(max_sge, rq_spec); | ||
| 2060 | if (rq_spec && !qp->rq_spec) | ||
| 2061 | ibdev_dbg(&dev->ibdev, | ||
| 2062 | "init rq: max_sge %u disables spec\n", | ||
| 2063 | max_sge); | ||
| 2064 | |||
| 2065 | if (qp->rq_cmb & IONIC_CMB_EXPDB) { | ||
| 2066 | wqe_size = ionic_v1_recv_wqe_min_size(max_sge, | ||
| 2067 | qp->rq_spec, | ||
| 2068 | true); | ||
| 2069 | |||
| 2070 | if (!ionic_expdb_wqe_size_supported(dev, wqe_size)) | ||
| 2071 | qp->rq_cmb &= ~IONIC_CMB_EXPDB; | ||
| 2072 | } | ||
| 2073 | |||
| 2074 | if (!(qp->rq_cmb & IONIC_CMB_EXPDB)) | ||
| 2075 | wqe_size = ionic_v1_recv_wqe_min_size(max_sge, | ||
| 2076 | qp->rq_spec, | ||
| 2077 | false); | ||
| 2078 | |||
| 2079 | rc = ionic_queue_init(&qp->rq, dev->lif_cfg.hwdev, | ||
| 2080 | max_wr, wqe_size); | ||
| 2081 | if (rc) | ||
| 2082 | return rc; | ||
| 2083 | |||
| 2084 | ionic_queue_dbell_init(&qp->rq, qp->qpid); | ||
| 2085 | |||
| 2086 | qp->rq_meta = kmalloc_array((u32)qp->rq.mask + 1, | ||
| 2087 | sizeof(*qp->rq_meta), | ||
| 2088 | GFP_KERNEL); | ||
| 2089 | if (!qp->rq_meta) { | ||
| 2090 | rc = -ENOMEM; | ||
| 2091 | goto err_rq_meta; | ||
| 2092 | } | ||
| 2093 | |||
| 2094 | for (i = 0; i < qp->rq.mask; ++i) | ||
| 2095 | qp->rq_meta[i].next = &qp->rq_meta[i + 1]; | ||
| 2096 | qp->rq_meta[i].next = IONIC_META_LAST; | ||
| 2097 | qp->rq_meta_head = &qp->rq_meta[0]; | ||
| 2098 | } | ||
| 2099 | |||
| 2100 | ionic_qp_rq_init_cmb(dev, qp, udata); | ||
| 2101 | |||
| 2102 | if (qp->rq_cmb & IONIC_CMB_ENABLE) | ||
| 2103 | rc = ionic_pgtbl_init(dev, buf, NULL, | ||
| 2104 | (u64)qp->rq_cmb_pgid << PAGE_SHIFT, | ||
| 2105 | 1, PAGE_SIZE); | ||
| 2106 | else | ||
| 2107 | rc = ionic_pgtbl_init(dev, buf, | ||
| 2108 | qp->rq_umem, qp->rq.dma, 1, PAGE_SIZE); | ||
| 2109 | if (rc) | ||
| 2110 | goto err_rq_tbl; | ||
| 2111 | |||
| 2112 | return 0; | ||
| 2113 | |||
| 2114 | err_rq_tbl: | ||
| 2115 | ionic_qp_rq_destroy_cmb(dev, ctx, qp); | ||
| 2116 | kfree(qp->rq_meta); | ||
| 2117 | err_rq_meta: | ||
| 2118 | if (qp->rq_umem) | ||
| 2119 | ib_umem_release(qp->rq_umem); | ||
| 2120 | else | ||
| 2121 | ionic_queue_destroy(&qp->rq, dev->lif_cfg.hwdev); | ||
| 2122 | return rc; | ||
| 2123 | } | ||
| 2124 | |||
| 2125 | static void ionic_qp_rq_destroy(struct ionic_ibdev *dev, | ||
| 2126 | struct ionic_ctx *ctx, | ||
| 2127 | struct ionic_qp *qp) | ||
| 2128 | { | ||
| 2129 | if (!qp->has_rq) | ||
| 2130 | return; | ||
| 2131 | |||
| 2132 | ionic_qp_rq_destroy_cmb(dev, ctx, qp); | ||
| 2133 | |||
| 2134 | kfree(qp->rq_meta); | ||
| 2135 | |||
| 2136 | if (qp->rq_umem) | ||
| 2137 | ib_umem_release(qp->rq_umem); | ||
| 2138 | else | ||
| 2139 | ionic_queue_destroy(&qp->rq, dev->lif_cfg.hwdev); | ||
| 2140 | } | ||
| 2141 | |||
| 2142 | int ionic_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *attr, | ||
| 2143 | struct ib_udata *udata) | ||
| 2144 | { | ||
| 2145 | struct ionic_ibdev *dev = to_ionic_ibdev(ibqp->device); | ||
| 2146 | struct ionic_tbl_buf sq_buf = {}, rq_buf = {}; | ||
| 2147 | struct ionic_pd *pd = to_ionic_pd(ibqp->pd); | ||
| 2148 | struct ionic_qp *qp = to_ionic_qp(ibqp); | ||
| 2149 | struct ionic_ctx *ctx = | ||
| 2150 | rdma_udata_to_drv_context(udata, struct ionic_ctx, ibctx); | ||
| 2151 | struct ionic_qp_resp resp = {}; | ||
| 2152 | struct ionic_qp_req req = {}; | ||
| 2153 | struct ionic_cq *cq; | ||
| 2154 | u8 udma_mask; | ||
| 2155 | void *entry; | ||
| 2156 | int rc; | ||
| 2157 | |||
| 2158 | if (udata) { | ||
| 2159 | rc = ib_copy_from_udata(&req, udata, sizeof(req)); | ||
| 2160 | if (rc) | ||
| 2161 | return rc; | ||
| 2162 | } else { | ||
| 2163 | req.sq_spec = IONIC_SPEC_HIGH; | ||
| 2164 | req.rq_spec = IONIC_SPEC_HIGH; | ||
| 2165 | } | ||
| 2166 | |||
| 2167 | if (attr->qp_type == IB_QPT_SMI || attr->qp_type > IB_QPT_UD) | ||
| 2168 | return -EOPNOTSUPP; | ||
| 2169 | |||
| 2170 | qp->state = IB_QPS_RESET; | ||
| 2171 | |||
| 2172 | INIT_LIST_HEAD(&qp->cq_poll_sq); | ||
| 2173 | INIT_LIST_HEAD(&qp->cq_flush_sq); | ||
| 2174 | INIT_LIST_HEAD(&qp->cq_flush_rq); | ||
| 2175 | |||
| 2176 | spin_lock_init(&qp->sq_lock); | ||
| 2177 | spin_lock_init(&qp->rq_lock); | ||
| 2178 | |||
| 2179 | qp->has_sq = 1; | ||
| 2180 | qp->has_rq = 1; | ||
| 2181 | |||
| 2182 | if (attr->qp_type == IB_QPT_GSI) { | ||
| 2183 | rc = ionic_get_gsi_qpid(dev, &qp->qpid); | ||
| 2184 | } else { | ||
| 2185 | udma_mask = BIT(dev->lif_cfg.udma_count) - 1; | ||
| 2186 | |||
| 2187 | if (qp->has_sq) | ||
| 2188 | udma_mask &= to_ionic_vcq(attr->send_cq)->udma_mask; | ||
| 2189 | |||
| 2190 | if (qp->has_rq) | ||
| 2191 | udma_mask &= to_ionic_vcq(attr->recv_cq)->udma_mask; | ||
| 2192 | |||
| 2193 | if (udata && req.udma_mask) | ||
| 2194 | udma_mask &= req.udma_mask; | ||
| 2195 | |||
| 2196 | if (!udma_mask) | ||
| 2197 | return -EINVAL; | ||
| 2198 | |||
| 2199 | rc = ionic_get_qpid(dev, &qp->qpid, &qp->udma_idx, udma_mask); | ||
| 2200 | } | ||
| 2201 | if (rc) | ||
| 2202 | return rc; | ||
| 2203 | |||
| 2204 | qp->sig_all = attr->sq_sig_type == IB_SIGNAL_ALL_WR; | ||
| 2205 | qp->has_ah = attr->qp_type == IB_QPT_RC; | ||
| 2206 | |||
| 2207 | if (qp->has_ah) { | ||
| 2208 | qp->hdr = kzalloc(sizeof(*qp->hdr), GFP_KERNEL); | ||
| 2209 | if (!qp->hdr) { | ||
| 2210 | rc = -ENOMEM; | ||
| 2211 | goto err_ah_alloc; | ||
| 2212 | } | ||
| 2213 | |||
| 2214 | rc = ionic_get_ahid(dev, &qp->ahid); | ||
| 2215 | if (rc) | ||
| 2216 | goto err_ahid; | ||
| 2217 | } | ||
| 2218 | |||
| 2219 | if (udata) { | ||
| 2220 | if (req.rq_cmb & IONIC_CMB_ENABLE) | ||
| 2221 | qp->rq_cmb = req.rq_cmb; | ||
| 2222 | |||
| 2223 | if (req.sq_cmb & IONIC_CMB_ENABLE) | ||
| 2224 | qp->sq_cmb = req.sq_cmb; | ||
| 2225 | } | ||
| 2226 | |||
| 2227 | rc = ionic_qp_sq_init(dev, ctx, qp, &req.sq, &sq_buf, | ||
| 2228 | attr->cap.max_send_wr, attr->cap.max_send_sge, | ||
| 2229 | attr->cap.max_inline_data, req.sq_spec, udata); | ||
| 2230 | if (rc) | ||
| 2231 | goto err_sq; | ||
| 2232 | |||
| 2233 | rc = ionic_qp_rq_init(dev, ctx, qp, &req.rq, &rq_buf, | ||
| 2234 | attr->cap.max_recv_wr, attr->cap.max_recv_sge, | ||
| 2235 | req.rq_spec, udata); | ||
| 2236 | if (rc) | ||
| 2237 | goto err_rq; | ||
| 2238 | |||
| 2239 | rc = ionic_create_qp_cmd(dev, pd, | ||
| 2240 | to_ionic_vcq_cq(attr->send_cq, qp->udma_idx), | ||
| 2241 | to_ionic_vcq_cq(attr->recv_cq, qp->udma_idx), | ||
| 2242 | qp, &sq_buf, &rq_buf, attr); | ||
| 2243 | if (rc) | ||
| 2244 | goto err_cmd; | ||
| 2245 | |||
| 2246 | if (udata) { | ||
| 2247 | resp.qpid = qp->qpid; | ||
| 2248 | resp.udma_idx = qp->udma_idx; | ||
| 2249 | |||
| 2250 | if (qp->sq_cmb & IONIC_CMB_ENABLE) { | ||
| 2251 | bool wc; | ||
| 2252 | |||
| 2253 | if ((qp->sq_cmb & (IONIC_CMB_WC | IONIC_CMB_UC)) == | ||
| 2254 | (IONIC_CMB_WC | IONIC_CMB_UC)) { | ||
| 2255 | ibdev_dbg(&dev->ibdev, | ||
| 2256 | "Both sq_cmb flags IONIC_CMB_WC and IONIC_CMB_UC are set, using default driver mapping\n"); | ||
| 2257 | qp->sq_cmb &= ~(IONIC_CMB_WC | IONIC_CMB_UC); | ||
| 2258 | } | ||
| 2259 | |||
| 2260 | wc = (qp->sq_cmb & (IONIC_CMB_WC | IONIC_CMB_UC)) | ||
| 2261 | != IONIC_CMB_UC; | ||
| 2262 | |||
| 2263 | /* let userspace know the mapping */ | ||
| 2264 | if (wc) | ||
| 2265 | qp->sq_cmb |= IONIC_CMB_WC; | ||
| 2266 | else | ||
| 2267 | qp->sq_cmb |= IONIC_CMB_UC; | ||
| 2268 | |||
| 2269 | qp->mmap_sq_cmb = | ||
| 2270 | ionic_mmap_entry_insert(ctx, | ||
| 2271 | qp->sq.size, | ||
| 2272 | PHYS_PFN(qp->sq_cmb_addr), | ||
| 2273 | wc ? IONIC_MMAP_WC : 0, | ||
| 2274 | &resp.sq_cmb_offset); | ||
| 2275 | if (!qp->mmap_sq_cmb) { | ||
| 2276 | rc = -ENOMEM; | ||
| 2277 | goto err_mmap_sq; | ||
| 2278 | } | ||
| 2279 | |||
| 2280 | resp.sq_cmb = qp->sq_cmb; | ||
| 2281 | } | ||
| 2282 | |||
| 2283 | if (qp->rq_cmb & IONIC_CMB_ENABLE) { | ||
| 2284 | bool wc; | ||
| 2285 | |||
| 2286 | if ((qp->rq_cmb & (IONIC_CMB_WC | IONIC_CMB_UC)) == | ||
| 2287 | (IONIC_CMB_WC | IONIC_CMB_UC)) { | ||
| 2288 | ibdev_dbg(&dev->ibdev, | ||
| 2289 | "Both rq_cmb flags IONIC_CMB_WC and IONIC_CMB_UC are set, using default driver mapping\n"); | ||
| 2290 | qp->rq_cmb &= ~(IONIC_CMB_WC | IONIC_CMB_UC); | ||
| 2291 | } | ||
| 2292 | |||
| 2293 | if (qp->rq_cmb & IONIC_CMB_EXPDB) | ||
| 2294 | wc = (qp->rq_cmb & (IONIC_CMB_WC | IONIC_CMB_UC)) | ||
| 2295 | == IONIC_CMB_WC; | ||
| 2296 | else | ||
| 2297 | wc = (qp->rq_cmb & (IONIC_CMB_WC | IONIC_CMB_UC)) | ||
| 2298 | != IONIC_CMB_UC; | ||
| 2299 | |||
| 2300 | /* let userspace know the mapping */ | ||
| 2301 | if (wc) | ||
| 2302 | qp->rq_cmb |= IONIC_CMB_WC; | ||
| 2303 | else | ||
| 2304 | qp->rq_cmb |= IONIC_CMB_UC; | ||
| 2305 | |||
| 2306 | qp->mmap_rq_cmb = | ||
| 2307 | ionic_mmap_entry_insert(ctx, | ||
| 2308 | qp->rq.size, | ||
| 2309 | PHYS_PFN(qp->rq_cmb_addr), | ||
| 2310 | wc ? IONIC_MMAP_WC : 0, | ||
| 2311 | &resp.rq_cmb_offset); | ||
| 2312 | if (!qp->mmap_rq_cmb) { | ||
| 2313 | rc = -ENOMEM; | ||
| 2314 | goto err_mmap_rq; | ||
| 2315 | } | ||
| 2316 | |||
| 2317 | resp.rq_cmb = qp->rq_cmb; | ||
| 2318 | } | ||
| 2319 | |||
| 2320 | rc = ib_copy_to_udata(udata, &resp, sizeof(resp)); | ||
| 2321 | if (rc) | ||
| 2322 | goto err_resp; | ||
| 2323 | } | ||
| 2324 | |||
| 2325 | ionic_pgtbl_unbuf(dev, &rq_buf); | ||
| 2326 | ionic_pgtbl_unbuf(dev, &sq_buf); | ||
| 2327 | |||
| 2328 | qp->ibqp.qp_num = qp->qpid; | ||
| 2329 | |||
| 2330 | init_completion(&qp->qp_rel_comp); | ||
| 2331 | kref_init(&qp->qp_kref); | ||
| 2332 | |||
| 2333 | entry = xa_store_irq(&dev->qp_tbl, qp->qpid, qp, GFP_KERNEL); | ||
| 2334 | if (entry) { | ||
| 2335 | if (!xa_is_err(entry)) | ||
| 2336 | rc = -EINVAL; | ||
| 2337 | else | ||
| 2338 | rc = xa_err(entry); | ||
| 2339 | |||
| 2340 | goto err_resp; | ||
| 2341 | } | ||
| 2342 | |||
| 2343 | if (qp->has_sq) { | ||
| 2344 | cq = to_ionic_vcq_cq(attr->send_cq, qp->udma_idx); | ||
| 2345 | |||
| 2346 | attr->cap.max_send_wr = qp->sq.mask; | ||
| 2347 | attr->cap.max_send_sge = | ||
| 2348 | ionic_v1_send_wqe_max_sge(qp->sq.stride_log2, | ||
| 2349 | qp->sq_spec, | ||
| 2350 | qp->sq_cmb & IONIC_CMB_EXPDB); | ||
| 2351 | attr->cap.max_inline_data = | ||
| 2352 | ionic_v1_send_wqe_max_data(qp->sq.stride_log2, | ||
| 2353 | qp->sq_cmb & | ||
| 2354 | IONIC_CMB_EXPDB); | ||
| 2355 | qp->sq_cqid = cq->cqid; | ||
| 2356 | } | ||
| 2357 | |||
| 2358 | if (qp->has_rq) { | ||
| 2359 | cq = to_ionic_vcq_cq(attr->recv_cq, qp->udma_idx); | ||
| 2360 | |||
| 2361 | attr->cap.max_recv_wr = qp->rq.mask; | ||
| 2362 | attr->cap.max_recv_sge = | ||
| 2363 | ionic_v1_recv_wqe_max_sge(qp->rq.stride_log2, | ||
| 2364 | qp->rq_spec, | ||
| 2365 | qp->rq_cmb & IONIC_CMB_EXPDB); | ||
| 2366 | qp->rq_cqid = cq->cqid; | ||
| 2367 | } | ||
| 2368 | |||
| 2369 | return 0; | ||
| 2370 | |||
| 2371 | err_resp: | ||
| 2372 | if (udata && (qp->rq_cmb & IONIC_CMB_ENABLE)) | ||
| 2373 | rdma_user_mmap_entry_remove(qp->mmap_rq_cmb); | ||
| 2374 | err_mmap_rq: | ||
| 2375 | if (udata && (qp->sq_cmb & IONIC_CMB_ENABLE)) | ||
| 2376 | rdma_user_mmap_entry_remove(qp->mmap_sq_cmb); | ||
| 2377 | err_mmap_sq: | ||
| 2378 | ionic_destroy_qp_cmd(dev, qp->qpid); | ||
| 2379 | err_cmd: | ||
| 2380 | ionic_pgtbl_unbuf(dev, &rq_buf); | ||
| 2381 | ionic_qp_rq_destroy(dev, ctx, qp); | ||
| 2382 | err_rq: | ||
| 2383 | ionic_pgtbl_unbuf(dev, &sq_buf); | ||
| 2384 | ionic_qp_sq_destroy(dev, ctx, qp); | ||
| 2385 | err_sq: | ||
| 2386 | if (qp->has_ah) | ||
| 2387 | ionic_put_ahid(dev, qp->ahid); | ||
| 2388 | err_ahid: | ||
| 2389 | kfree(qp->hdr); | ||
| 2390 | err_ah_alloc: | ||
| 2391 | ionic_put_qpid(dev, qp->qpid); | ||
| 2392 | return rc; | ||
| 2393 | } | ||
| 2394 | |||
| 2395 | void ionic_notify_flush_cq(struct ionic_cq *cq) | ||
| 2396 | { | ||
| 2397 | if (cq->flush && cq->vcq->ibcq.comp_handler) | ||
| 2398 | cq->vcq->ibcq.comp_handler(&cq->vcq->ibcq, | ||
| 2399 | cq->vcq->ibcq.cq_context); | ||
| 2400 | } | ||
| 2401 | |||
| 2402 | static void ionic_notify_qp_cqs(struct ionic_ibdev *dev, struct ionic_qp *qp) | ||
| 2403 | { | ||
| 2404 | if (qp->ibqp.send_cq) | ||
| 2405 | ionic_notify_flush_cq(to_ionic_vcq_cq(qp->ibqp.send_cq, | ||
| 2406 | qp->udma_idx)); | ||
| 2407 | if (qp->ibqp.recv_cq && qp->ibqp.recv_cq != qp->ibqp.send_cq) | ||
| 2408 | ionic_notify_flush_cq(to_ionic_vcq_cq(qp->ibqp.recv_cq, | ||
| 2409 | qp->udma_idx)); | ||
| 2410 | } | ||
| 2411 | |||
| 2412 | void ionic_flush_qp(struct ionic_ibdev *dev, struct ionic_qp *qp) | ||
| 2413 | { | ||
| 2414 | unsigned long irqflags; | ||
| 2415 | struct ionic_cq *cq; | ||
| 2416 | |||
| 2417 | if (qp->ibqp.send_cq) { | ||
| 2418 | cq = to_ionic_vcq_cq(qp->ibqp.send_cq, qp->udma_idx); | ||
| 2419 | |||
| 2420 | /* Hold the CQ lock and QP sq_lock to set up flush */ | ||
| 2421 | spin_lock_irqsave(&cq->lock, irqflags); | ||
| 2422 | spin_lock(&qp->sq_lock); | ||
| 2423 | qp->sq_flush = true; | ||
| 2424 | if (!ionic_queue_empty(&qp->sq)) { | ||
| 2425 | cq->flush = true; | ||
| 2426 | list_move_tail(&qp->cq_flush_sq, &cq->flush_sq); | ||
| 2427 | } | ||
| 2428 | spin_unlock(&qp->sq_lock); | ||
| 2429 | spin_unlock_irqrestore(&cq->lock, irqflags); | ||
| 2430 | } | ||
| 2431 | |||
| 2432 | if (qp->ibqp.recv_cq) { | ||
| 2433 | cq = to_ionic_vcq_cq(qp->ibqp.recv_cq, qp->udma_idx); | ||
| 2434 | |||
| 2435 | /* Hold the CQ lock and QP rq_lock to set up flush */ | ||
| 2436 | spin_lock_irqsave(&cq->lock, irqflags); | ||
| 2437 | spin_lock(&qp->rq_lock); | ||
| 2438 | qp->rq_flush = true; | ||
| 2439 | if (!ionic_queue_empty(&qp->rq)) { | ||
| 2440 | cq->flush = true; | ||
| 2441 | list_move_tail(&qp->cq_flush_rq, &cq->flush_rq); | ||
| 2442 | } | ||
| 2443 | spin_unlock(&qp->rq_lock); | ||
| 2444 | spin_unlock_irqrestore(&cq->lock, irqflags); | ||
| 2445 | } | ||
| 2446 | } | ||
| 2447 | |||
| 2448 | static void ionic_clean_cq(struct ionic_cq *cq, u32 qpid) | ||
| 2449 | { | ||
| 2450 | struct ionic_v1_cqe *qcqe; | ||
| 2451 | int prod, qtf, qid, type; | ||
| 2452 | bool color; | ||
| 2453 | |||
| 2454 | if (!cq->q.ptr) | ||
| 2455 | return; | ||
| 2456 | |||
| 2457 | color = cq->color; | ||
| 2458 | prod = cq->q.prod; | ||
| 2459 | qcqe = ionic_queue_at(&cq->q, prod); | ||
| 2460 | |||
| 2461 | while (color == ionic_v1_cqe_color(qcqe)) { | ||
| 2462 | qtf = ionic_v1_cqe_qtf(qcqe); | ||
| 2463 | qid = ionic_v1_cqe_qtf_qid(qtf); | ||
| 2464 | type = ionic_v1_cqe_qtf_type(qtf); | ||
| 2465 | |||
| 2466 | if (qid == qpid && type != IONIC_V1_CQE_TYPE_ADMIN) | ||
| 2467 | ionic_v1_cqe_clean(qcqe); | ||
| 2468 | |||
| 2469 | prod = ionic_queue_next(&cq->q, prod); | ||
| 2470 | qcqe = ionic_queue_at(&cq->q, prod); | ||
| 2471 | color = ionic_color_wrap(prod, color); | ||
| 2472 | } | ||
| 2473 | } | ||
| 2474 | |||
| 2475 | static void ionic_reset_qp(struct ionic_ibdev *dev, struct ionic_qp *qp) | ||
| 2476 | { | ||
| 2477 | unsigned long irqflags; | ||
| 2478 | struct ionic_cq *cq; | ||
| 2479 | int i; | ||
| 2480 | |||
| 2481 | local_irq_save(irqflags); | ||
| 2482 | |||
| 2483 | if (qp->ibqp.send_cq) { | ||
| 2484 | cq = to_ionic_vcq_cq(qp->ibqp.send_cq, qp->udma_idx); | ||
| 2485 | spin_lock(&cq->lock); | ||
| 2486 | ionic_clean_cq(cq, qp->qpid); | ||
| 2487 | spin_unlock(&cq->lock); | ||
| 2488 | } | ||
| 2489 | |||
| 2490 | if (qp->ibqp.recv_cq) { | ||
| 2491 | cq = to_ionic_vcq_cq(qp->ibqp.recv_cq, qp->udma_idx); | ||
| 2492 | spin_lock(&cq->lock); | ||
| 2493 | ionic_clean_cq(cq, qp->qpid); | ||
| 2494 | spin_unlock(&cq->lock); | ||
| 2495 | } | ||
| 2496 | |||
| 2497 | if (qp->has_sq) { | ||
| 2498 | spin_lock(&qp->sq_lock); | ||
| 2499 | qp->sq_flush = false; | ||
| 2500 | qp->sq_flush_rcvd = false; | ||
| 2501 | qp->sq_msn_prod = 0; | ||
| 2502 | qp->sq_msn_cons = 0; | ||
| 2503 | qp->sq.prod = 0; | ||
| 2504 | qp->sq.cons = 0; | ||
| 2505 | spin_unlock(&qp->sq_lock); | ||
| 2506 | } | ||
| 2507 | |||
| 2508 | if (qp->has_rq) { | ||
| 2509 | spin_lock(&qp->rq_lock); | ||
| 2510 | qp->rq_flush = false; | ||
| 2511 | qp->rq.prod = 0; | ||
| 2512 | qp->rq.cons = 0; | ||
| 2513 | if (qp->rq_meta) { | ||
| 2514 | for (i = 0; i < qp->rq.mask; ++i) | ||
| 2515 | qp->rq_meta[i].next = &qp->rq_meta[i + 1]; | ||
| 2516 | qp->rq_meta[i].next = IONIC_META_LAST; | ||
| 2517 | } | ||
| 2518 | qp->rq_meta_head = &qp->rq_meta[0]; | ||
| 2519 | spin_unlock(&qp->rq_lock); | ||
| 2520 | } | ||
| 2521 | |||
| 2522 | local_irq_restore(irqflags); | ||
| 2523 | } | ||
| 2524 | |||
| 2525 | static bool ionic_qp_cur_state_is_ok(enum ib_qp_state q_state, | ||
| 2526 | enum ib_qp_state attr_state) | ||
| 2527 | { | ||
| 2528 | if (q_state == attr_state) | ||
| 2529 | return true; | ||
| 2530 | |||
| 2531 | if (attr_state == IB_QPS_ERR) | ||
| 2532 | return true; | ||
| 2533 | |||
| 2534 | if (attr_state == IB_QPS_SQE) | ||
| 2535 | return q_state == IB_QPS_RTS || q_state == IB_QPS_SQD; | ||
| 2536 | |||
| 2537 | return false; | ||
| 2538 | } | ||
| 2539 | |||
| 2540 | static int ionic_check_modify_qp(struct ionic_qp *qp, struct ib_qp_attr *attr, | ||
| 2541 | int mask) | ||
| 2542 | { | ||
| 2543 | enum ib_qp_state cur_state = (mask & IB_QP_CUR_STATE) ? | ||
| 2544 | attr->cur_qp_state : qp->state; | ||
| 2545 | enum ib_qp_state next_state = (mask & IB_QP_STATE) ? | ||
| 2546 | attr->qp_state : cur_state; | ||
| 2547 | |||
| 2548 | if ((mask & IB_QP_CUR_STATE) && | ||
| 2549 | !ionic_qp_cur_state_is_ok(qp->state, attr->cur_qp_state)) | ||
| 2550 | return -EINVAL; | ||
| 2551 | |||
| 2552 | if (!ib_modify_qp_is_ok(cur_state, next_state, qp->ibqp.qp_type, mask)) | ||
| 2553 | return -EINVAL; | ||
| 2554 | |||
| 2555 | /* unprivileged qp not allowed privileged qkey */ | ||
| 2556 | if ((mask & IB_QP_QKEY) && (attr->qkey & 0x80000000) && | ||
| 2557 | qp->ibqp.uobject) | ||
| 2558 | return -EPERM; | ||
| 2559 | |||
| 2560 | return 0; | ||
| 2561 | } | ||
| 2562 | |||
| 2563 | int ionic_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int mask, | ||
| 2564 | struct ib_udata *udata) | ||
| 2565 | { | ||
| 2566 | struct ionic_ibdev *dev = to_ionic_ibdev(ibqp->device); | ||
| 2567 | struct ionic_pd *pd = to_ionic_pd(ibqp->pd); | ||
| 2568 | struct ionic_qp *qp = to_ionic_qp(ibqp); | ||
| 2569 | int rc; | ||
| 2570 | |||
| 2571 | rc = ionic_check_modify_qp(qp, attr, mask); | ||
| 2572 | if (rc) | ||
| 2573 | return rc; | ||
| 2574 | |||
| 2575 | if (mask & IB_QP_CAP) | ||
| 2576 | return -EINVAL; | ||
| 2577 | |||
| 2578 | rc = ionic_modify_qp_cmd(dev, pd, qp, attr, mask); | ||
| 2579 | if (rc) | ||
| 2580 | return rc; | ||
| 2581 | |||
| 2582 | if (mask & IB_QP_STATE) { | ||
| 2583 | qp->state = attr->qp_state; | ||
| 2584 | |||
| 2585 | if (attr->qp_state == IB_QPS_ERR) { | ||
| 2586 | ionic_flush_qp(dev, qp); | ||
| 2587 | ionic_notify_qp_cqs(dev, qp); | ||
| 2588 | } else if (attr->qp_state == IB_QPS_RESET) { | ||
| 2589 | ionic_reset_qp(dev, qp); | ||
| 2590 | } | ||
| 2591 | } | ||
| 2592 | |||
| 2593 | return 0; | ||
| 2594 | } | ||
| 2595 | |||
| 2596 | int ionic_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, | ||
| 2597 | int mask, struct ib_qp_init_attr *init_attr) | ||
| 2598 | { | ||
| 2599 | struct ionic_ibdev *dev = to_ionic_ibdev(ibqp->device); | ||
| 2600 | struct ionic_qp *qp = to_ionic_qp(ibqp); | ||
| 2601 | int rc; | ||
| 2602 | |||
| 2603 | memset(attr, 0, sizeof(*attr)); | ||
| 2604 | memset(init_attr, 0, sizeof(*init_attr)); | ||
| 2605 | |||
| 2606 | rc = ionic_query_qp_cmd(dev, qp, attr, mask); | ||
| 2607 | if (rc) | ||
| 2608 | return rc; | ||
| 2609 | |||
| 2610 | if (qp->has_sq) | ||
| 2611 | attr->cap.max_send_wr = qp->sq.mask; | ||
| 2612 | |||
| 2613 | if (qp->has_rq) | ||
| 2614 | attr->cap.max_recv_wr = qp->rq.mask; | ||
| 2615 | |||
| 2616 | init_attr->event_handler = ibqp->event_handler; | ||
| 2617 | init_attr->qp_context = ibqp->qp_context; | ||
| 2618 | init_attr->send_cq = ibqp->send_cq; | ||
| 2619 | init_attr->recv_cq = ibqp->recv_cq; | ||
| 2620 | init_attr->srq = ibqp->srq; | ||
| 2621 | init_attr->xrcd = ibqp->xrcd; | ||
| 2622 | init_attr->cap = attr->cap; | ||
| 2623 | init_attr->sq_sig_type = qp->sig_all ? | ||
| 2624 | IB_SIGNAL_ALL_WR : IB_SIGNAL_REQ_WR; | ||
| 2625 | init_attr->qp_type = ibqp->qp_type; | ||
| 2626 | init_attr->create_flags = 0; | ||
| 2627 | init_attr->port_num = 0; | ||
| 2628 | init_attr->rwq_ind_tbl = ibqp->rwq_ind_tbl; | ||
| 2629 | init_attr->source_qpn = 0; | ||
| 2630 | |||
| 2631 | return rc; | ||
| 2632 | } | ||
| 2633 | |||
| 2634 | int ionic_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) | ||
| 2635 | { | ||
| 2636 | struct ionic_ctx *ctx = | ||
| 2637 | rdma_udata_to_drv_context(udata, struct ionic_ctx, ibctx); | ||
| 2638 | struct ionic_ibdev *dev = to_ionic_ibdev(ibqp->device); | ||
| 2639 | struct ionic_qp *qp = to_ionic_qp(ibqp); | ||
| 2640 | unsigned long irqflags; | ||
| 2641 | struct ionic_cq *cq; | ||
| 2642 | int rc; | ||
| 2643 | |||
| 2644 | rc = ionic_destroy_qp_cmd(dev, qp->qpid); | ||
| 2645 | if (rc) | ||
| 2646 | return rc; | ||
| 2647 | |||
| 2648 | xa_erase_irq(&dev->qp_tbl, qp->qpid); | ||
| 2649 | |||
| 2650 | kref_put(&qp->qp_kref, ionic_qp_complete); | ||
| 2651 | wait_for_completion(&qp->qp_rel_comp); | ||
| 2652 | |||
| 2653 | if (qp->ibqp.send_cq) { | ||
| 2654 | cq = to_ionic_vcq_cq(qp->ibqp.send_cq, qp->udma_idx); | ||
| 2655 | spin_lock_irqsave(&cq->lock, irqflags); | ||
| 2656 | ionic_clean_cq(cq, qp->qpid); | ||
| 2657 | list_del(&qp->cq_poll_sq); | ||
| 2658 | list_del(&qp->cq_flush_sq); | ||
| 2659 | spin_unlock_irqrestore(&cq->lock, irqflags); | ||
| 2660 | } | ||
| 2661 | |||
| 2662 | if (qp->ibqp.recv_cq) { | ||
| 2663 | cq = to_ionic_vcq_cq(qp->ibqp.recv_cq, qp->udma_idx); | ||
| 2664 | spin_lock_irqsave(&cq->lock, irqflags); | ||
| 2665 | ionic_clean_cq(cq, qp->qpid); | ||
| 2666 | list_del(&qp->cq_flush_rq); | ||
| 2667 | spin_unlock_irqrestore(&cq->lock, irqflags); | ||
| 2668 | } | ||
| 2669 | |||
| 2670 | ionic_qp_rq_destroy(dev, ctx, qp); | ||
| 2671 | ionic_qp_sq_destroy(dev, ctx, qp); | ||
| 2672 | if (qp->has_ah) { | ||
| 2673 | ionic_put_ahid(dev, qp->ahid); | ||
| 2674 | kfree(qp->hdr); | ||
| 2675 | } | ||
| 2676 | ionic_put_qpid(dev, qp->qpid); | ||
| 2677 | |||
| 2678 | return 0; | ||
| 2679 | } | ||
| @@ -0,0 +1,1399 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #include <linux/module.h> | ||
| 5 | #include <linux/printk.h> | ||
| 6 | #include <rdma/ib_addr.h> | ||
| 7 | #include <rdma/ib_user_verbs.h> | ||
| 8 | |||
| 9 | #include "ionic_fw.h" | ||
| 10 | #include "ionic_ibdev.h" | ||
| 11 | |||
| 12 | #define IONIC_OP(version, opname) \ | ||
| 13 | ((version) < 2 ? IONIC_V1_OP_##opname : IONIC_V2_OP_##opname) | ||
| 14 | |||
| 15 | static bool ionic_next_cqe(struct ionic_ibdev *dev, struct ionic_cq *cq, | ||
| 16 | struct ionic_v1_cqe **cqe) | ||
| 17 | { | ||
| 18 | struct ionic_v1_cqe *qcqe = ionic_queue_at_prod(&cq->q); | ||
| 19 | |||
| 20 | if (unlikely(cq->color != ionic_v1_cqe_color(qcqe))) | ||
| 21 | return false; | ||
| 22 | |||
| 23 | /* Prevent out-of-order reads of the CQE */ | ||
| 24 | dma_rmb(); | ||
| 25 | |||
| 26 | *cqe = qcqe; | ||
| 27 | |||
| 28 | return true; | ||
| 29 | } | ||
| 30 | |||
| 31 | static int ionic_flush_recv(struct ionic_qp *qp, struct ib_wc *wc) | ||
| 32 | { | ||
| 33 | struct ionic_rq_meta *meta; | ||
| 34 | struct ionic_v1_wqe *wqe; | ||
| 35 | |||
| 36 | if (!qp->rq_flush) | ||
| 37 | return 0; | ||
| 38 | |||
| 39 | if (ionic_queue_empty(&qp->rq)) | ||
| 40 | return 0; | ||
| 41 | |||
| 42 | wqe = ionic_queue_at_cons(&qp->rq); | ||
| 43 | |||
| 44 | /* wqe_id must be a valid queue index */ | ||
| 45 | if (unlikely(wqe->base.wqe_id >> qp->rq.depth_log2)) { | ||
| 46 | ibdev_warn(qp->ibqp.device, | ||
| 47 | "flush qp %u recv index %llu invalid\n", | ||
| 48 | qp->qpid, (unsigned long long)wqe->base.wqe_id); | ||
| 49 | return -EIO; | ||
| 50 | } | ||
| 51 | |||
| 52 | /* wqe_id must indicate a request that is outstanding */ | ||
| 53 | meta = &qp->rq_meta[wqe->base.wqe_id]; | ||
| 54 | if (unlikely(meta->next != IONIC_META_POSTED)) { | ||
| 55 | ibdev_warn(qp->ibqp.device, | ||
| 56 | "flush qp %u recv index %llu not posted\n", | ||
| 57 | qp->qpid, (unsigned long long)wqe->base.wqe_id); | ||
| 58 | return -EIO; | ||
| 59 | } | ||
| 60 | |||
| 61 | ionic_queue_consume(&qp->rq); | ||
| 62 | |||
| 63 | memset(wc, 0, sizeof(*wc)); | ||
| 64 | |||
| 65 | wc->status = IB_WC_WR_FLUSH_ERR; | ||
| 66 | wc->wr_id = meta->wrid; | ||
| 67 | wc->qp = &qp->ibqp; | ||
| 68 | |||
| 69 | meta->next = qp->rq_meta_head; | ||
| 70 | qp->rq_meta_head = meta; | ||
| 71 | |||
| 72 | return 1; | ||
| 73 | } | ||
| 74 | |||
| 75 | static int ionic_flush_recv_many(struct ionic_qp *qp, | ||
| 76 | struct ib_wc *wc, int nwc) | ||
| 77 | { | ||
| 78 | int rc = 0, npolled = 0; | ||
| 79 | |||
| 80 | while (npolled < nwc) { | ||
| 81 | rc = ionic_flush_recv(qp, wc + npolled); | ||
| 82 | if (rc <= 0) | ||
| 83 | break; | ||
| 84 | |||
| 85 | npolled += rc; | ||
| 86 | } | ||
| 87 | |||
| 88 | return npolled ?: rc; | ||
| 89 | } | ||
| 90 | |||
| 91 | static int ionic_flush_send(struct ionic_qp *qp, struct ib_wc *wc) | ||
| 92 | { | ||
| 93 | struct ionic_sq_meta *meta; | ||
| 94 | |||
| 95 | if (!qp->sq_flush) | ||
| 96 | return 0; | ||
| 97 | |||
| 98 | if (ionic_queue_empty(&qp->sq)) | ||
| 99 | return 0; | ||
| 100 | |||
| 101 | meta = &qp->sq_meta[qp->sq.cons]; | ||
| 102 | |||
| 103 | ionic_queue_consume(&qp->sq); | ||
| 104 | |||
| 105 | memset(wc, 0, sizeof(*wc)); | ||
| 106 | |||
| 107 | wc->status = IB_WC_WR_FLUSH_ERR; | ||
| 108 | wc->wr_id = meta->wrid; | ||
| 109 | wc->qp = &qp->ibqp; | ||
| 110 | |||
| 111 | return 1; | ||
| 112 | } | ||
| 113 | |||
| 114 | static int ionic_flush_send_many(struct ionic_qp *qp, | ||
| 115 | struct ib_wc *wc, int nwc) | ||
| 116 | { | ||
| 117 | int rc = 0, npolled = 0; | ||
| 118 | |||
| 119 | while (npolled < nwc) { | ||
| 120 | rc = ionic_flush_send(qp, wc + npolled); | ||
| 121 | if (rc <= 0) | ||
| 122 | break; | ||
| 123 | |||
| 124 | npolled += rc; | ||
| 125 | } | ||
| 126 | |||
| 127 | return npolled ?: rc; | ||
| 128 | } | ||
| 129 | |||
| 130 | static int ionic_poll_recv(struct ionic_ibdev *dev, struct ionic_cq *cq, | ||
| 131 | struct ionic_qp *cqe_qp, struct ionic_v1_cqe *cqe, | ||
| 132 | struct ib_wc *wc) | ||
| 133 | { | ||
| 134 | struct ionic_qp *qp = NULL; | ||
| 135 | struct ionic_rq_meta *meta; | ||
| 136 | u32 src_qpn, st_len; | ||
| 137 | u16 vlan_tag; | ||
| 138 | u8 op; | ||
| 139 | |||
| 140 | if (cqe_qp->rq_flush) | ||
| 141 | return 0; | ||
| 142 | |||
| 143 | qp = cqe_qp; | ||
| 144 | |||
| 145 | st_len = be32_to_cpu(cqe->status_length); | ||
| 146 | |||
| 147 | /* ignore wqe_id in case of flush error */ | ||
| 148 | if (ionic_v1_cqe_error(cqe) && st_len == IONIC_STS_WQE_FLUSHED_ERR) { | ||
| 149 | cqe_qp->rq_flush = true; | ||
| 150 | cq->flush = true; | ||
| 151 | list_move_tail(&qp->cq_flush_rq, &cq->flush_rq); | ||
| 152 | |||
| 153 | /* posted recvs (if any) flushed by ionic_flush_recv */ | ||
| 154 | return 0; | ||
| 155 | } | ||
| 156 | |||
| 157 | /* there had better be something in the recv queue to complete */ | ||
| 158 | if (ionic_queue_empty(&qp->rq)) { | ||
| 159 | ibdev_warn(&dev->ibdev, "qp %u is empty\n", qp->qpid); | ||
| 160 | return -EIO; | ||
| 161 | } | ||
| 162 | |||
| 163 | /* wqe_id must be a valid queue index */ | ||
| 164 | if (unlikely(cqe->recv.wqe_id >> qp->rq.depth_log2)) { | ||
| 165 | ibdev_warn(&dev->ibdev, | ||
| 166 | "qp %u recv index %llu invalid\n", | ||
| 167 | qp->qpid, (unsigned long long)cqe->recv.wqe_id); | ||
| 168 | return -EIO; | ||
| 169 | } | ||
| 170 | |||
| 171 | /* wqe_id must indicate a request that is outstanding */ | ||
| 172 | meta = &qp->rq_meta[cqe->recv.wqe_id]; | ||
| 173 | if (unlikely(meta->next != IONIC_META_POSTED)) { | ||
| 174 | ibdev_warn(&dev->ibdev, | ||
| 175 | "qp %u recv index %llu not posted\n", | ||
| 176 | qp->qpid, (unsigned long long)cqe->recv.wqe_id); | ||
| 177 | return -EIO; | ||
| 178 | } | ||
| 179 | |||
| 180 | meta->next = qp->rq_meta_head; | ||
| 181 | qp->rq_meta_head = meta; | ||
| 182 | |||
| 183 | memset(wc, 0, sizeof(*wc)); | ||
| 184 | |||
| 185 | wc->wr_id = meta->wrid; | ||
| 186 | |||
| 187 | wc->qp = &cqe_qp->ibqp; | ||
| 188 | |||
| 189 | if (ionic_v1_cqe_error(cqe)) { | ||
| 190 | wc->vendor_err = st_len; | ||
| 191 | wc->status = ionic_to_ib_status(st_len); | ||
| 192 | |||
| 193 | cqe_qp->rq_flush = true; | ||
| 194 | cq->flush = true; | ||
| 195 | list_move_tail(&qp->cq_flush_rq, &cq->flush_rq); | ||
| 196 | |||
| 197 | ibdev_warn(&dev->ibdev, | ||
| 198 | "qp %d recv cqe with error\n", qp->qpid); | ||
| 199 | print_hex_dump(KERN_WARNING, "cqe ", DUMP_PREFIX_OFFSET, 16, 1, | ||
| 200 | cqe, BIT(cq->q.stride_log2), true); | ||
| 201 | goto out; | ||
| 202 | } | ||
| 203 | |||
| 204 | wc->vendor_err = 0; | ||
| 205 | wc->status = IB_WC_SUCCESS; | ||
| 206 | |||
| 207 | src_qpn = be32_to_cpu(cqe->recv.src_qpn_op); | ||
| 208 | op = src_qpn >> IONIC_V1_CQE_RECV_OP_SHIFT; | ||
| 209 | |||
| 210 | src_qpn &= IONIC_V1_CQE_RECV_QPN_MASK; | ||
| 211 | op &= IONIC_V1_CQE_RECV_OP_MASK; | ||
| 212 | |||
| 213 | wc->opcode = IB_WC_RECV; | ||
| 214 | switch (op) { | ||
| 215 | case IONIC_V1_CQE_RECV_OP_RDMA_IMM: | ||
| 216 | wc->opcode = IB_WC_RECV_RDMA_WITH_IMM; | ||
| 217 | wc->wc_flags |= IB_WC_WITH_IMM; | ||
| 218 | wc->ex.imm_data = cqe->recv.imm_data_rkey; /* be32 in wc */ | ||
| 219 | break; | ||
| 220 | case IONIC_V1_CQE_RECV_OP_SEND_IMM: | ||
| 221 | wc->wc_flags |= IB_WC_WITH_IMM; | ||
| 222 | wc->ex.imm_data = cqe->recv.imm_data_rkey; /* be32 in wc */ | ||
| 223 | break; | ||
| 224 | case IONIC_V1_CQE_RECV_OP_SEND_INV: | ||
| 225 | wc->wc_flags |= IB_WC_WITH_INVALIDATE; | ||
| 226 | wc->ex.invalidate_rkey = be32_to_cpu(cqe->recv.imm_data_rkey); | ||
| 227 | break; | ||
| 228 | } | ||
| 229 | |||
| 230 | wc->byte_len = st_len; | ||
| 231 | wc->src_qp = src_qpn; | ||
| 232 | |||
| 233 | if (qp->ibqp.qp_type == IB_QPT_UD || | ||
| 234 | qp->ibqp.qp_type == IB_QPT_GSI) { | ||
| 235 | wc->wc_flags |= IB_WC_GRH | IB_WC_WITH_SMAC; | ||
| 236 | ether_addr_copy(wc->smac, cqe->recv.src_mac); | ||
| 237 | |||
| 238 | wc->wc_flags |= IB_WC_WITH_NETWORK_HDR_TYPE; | ||
| 239 | if (ionic_v1_cqe_recv_is_ipv4(cqe)) | ||
| 240 | wc->network_hdr_type = RDMA_NETWORK_IPV4; | ||
| 241 | else | ||
| 242 | wc->network_hdr_type = RDMA_NETWORK_IPV6; | ||
| 243 | |||
| 244 | if (ionic_v1_cqe_recv_is_vlan(cqe)) | ||
| 245 | wc->wc_flags |= IB_WC_WITH_VLAN; | ||
| 246 | |||
| 247 | /* vlan_tag in cqe will be valid from dpath even if no vlan */ | ||
| 248 | vlan_tag = be16_to_cpu(cqe->recv.vlan_tag); | ||
| 249 | wc->vlan_id = vlan_tag & 0xfff; /* 802.1q VID */ | ||
| 250 | wc->sl = vlan_tag >> VLAN_PRIO_SHIFT; /* 802.1q PCP */ | ||
| 251 | } | ||
| 252 | |||
| 253 | wc->pkey_index = 0; | ||
| 254 | wc->port_num = 1; | ||
| 255 | |||
| 256 | out: | ||
| 257 | ionic_queue_consume(&qp->rq); | ||
| 258 | |||
| 259 | return 1; | ||
| 260 | } | ||
| 261 | |||
| 262 | static bool ionic_peek_send(struct ionic_qp *qp) | ||
| 263 | { | ||
| 264 | struct ionic_sq_meta *meta; | ||
| 265 | |||
| 266 | if (qp->sq_flush) | ||
| 267 | return false; | ||
| 268 | |||
| 269 | /* completed all send queue requests */ | ||
| 270 | if (ionic_queue_empty(&qp->sq)) | ||
| 271 | return false; | ||
| 272 | |||
| 273 | meta = &qp->sq_meta[qp->sq.cons]; | ||
| 274 | |||
| 275 | /* waiting for remote completion */ | ||
| 276 | if (meta->remote && meta->seq == qp->sq_msn_cons) | ||
| 277 | return false; | ||
| 278 | |||
| 279 | /* waiting for local completion */ | ||
| 280 | if (!meta->remote && !meta->local_comp) | ||
| 281 | return false; | ||
| 282 | |||
| 283 | return true; | ||
| 284 | } | ||
| 285 | |||
| 286 | static int ionic_poll_send(struct ionic_ibdev *dev, struct ionic_cq *cq, | ||
| 287 | struct ionic_qp *qp, struct ib_wc *wc) | ||
| 288 | { | ||
| 289 | struct ionic_sq_meta *meta; | ||
| 290 | |||
| 291 | if (qp->sq_flush) | ||
| 292 | return 0; | ||
| 293 | |||
| 294 | do { | ||
| 295 | /* completed all send queue requests */ | ||
| 296 | if (ionic_queue_empty(&qp->sq)) | ||
| 297 | goto out_empty; | ||
| 298 | |||
| 299 | meta = &qp->sq_meta[qp->sq.cons]; | ||
| 300 | |||
| 301 | /* waiting for remote completion */ | ||
| 302 | if (meta->remote && meta->seq == qp->sq_msn_cons) | ||
| 303 | goto out_empty; | ||
| 304 | |||
| 305 | /* waiting for local completion */ | ||
| 306 | if (!meta->remote && !meta->local_comp) | ||
| 307 | goto out_empty; | ||
| 308 | |||
| 309 | ionic_queue_consume(&qp->sq); | ||
| 310 | |||
| 311 | /* produce wc only if signaled or error status */ | ||
| 312 | } while (!meta->signal && meta->ibsts == IB_WC_SUCCESS); | ||
| 313 | |||
| 314 | memset(wc, 0, sizeof(*wc)); | ||
| 315 | |||
| 316 | wc->status = meta->ibsts; | ||
| 317 | wc->wr_id = meta->wrid; | ||
| 318 | wc->qp = &qp->ibqp; | ||
| 319 | |||
| 320 | if (meta->ibsts == IB_WC_SUCCESS) { | ||
| 321 | wc->byte_len = meta->len; | ||
| 322 | wc->opcode = meta->ibop; | ||
| 323 | } else { | ||
| 324 | wc->vendor_err = meta->len; | ||
| 325 | |||
| 326 | qp->sq_flush = true; | ||
| 327 | cq->flush = true; | ||
| 328 | list_move_tail(&qp->cq_flush_sq, &cq->flush_sq); | ||
| 329 | } | ||
| 330 | |||
| 331 | return 1; | ||
| 332 | |||
| 333 | out_empty: | ||
| 334 | if (qp->sq_flush_rcvd) { | ||
| 335 | qp->sq_flush = true; | ||
| 336 | cq->flush = true; | ||
| 337 | list_move_tail(&qp->cq_flush_sq, &cq->flush_sq); | ||
| 338 | } | ||
| 339 | return 0; | ||
| 340 | } | ||
| 341 | |||
| 342 | static int ionic_poll_send_many(struct ionic_ibdev *dev, struct ionic_cq *cq, | ||
| 343 | struct ionic_qp *qp, struct ib_wc *wc, int nwc) | ||
| 344 | { | ||
| 345 | int rc = 0, npolled = 0; | ||
| 346 | |||
| 347 | while (npolled < nwc) { | ||
| 348 | rc = ionic_poll_send(dev, cq, qp, wc + npolled); | ||
| 349 | if (rc <= 0) | ||
| 350 | break; | ||
| 351 | |||
| 352 | npolled += rc; | ||
| 353 | } | ||
| 354 | |||
| 355 | return npolled ?: rc; | ||
| 356 | } | ||
| 357 | |||
| 358 | static int ionic_validate_cons(u16 prod, u16 cons, | ||
| 359 | u16 comp, u16 mask) | ||
| 360 | { | ||
| 361 | if (((prod - cons) & mask) <= ((comp - cons) & mask)) | ||
| 362 | return -EIO; | ||
| 363 | |||
| 364 | return 0; | ||
| 365 | } | ||
| 366 | |||
| 367 | static int ionic_comp_msn(struct ionic_qp *qp, struct ionic_v1_cqe *cqe) | ||
| 368 | { | ||
| 369 | struct ionic_sq_meta *meta; | ||
| 370 | u16 cqe_seq, cqe_idx; | ||
| 371 | int rc; | ||
| 372 | |||
| 373 | if (qp->sq_flush) | ||
| 374 | return 0; | ||
| 375 | |||
| 376 | cqe_seq = be32_to_cpu(cqe->send.msg_msn) & qp->sq.mask; | ||
| 377 | |||
| 378 | rc = ionic_validate_cons(qp->sq_msn_prod, | ||
| 379 | qp->sq_msn_cons, | ||
| 380 | cqe_seq - 1, | ||
| 381 | qp->sq.mask); | ||
| 382 | if (rc) { | ||
| 383 | ibdev_warn(qp->ibqp.device, | ||
| 384 | "qp %u bad msn %#x seq %u for prod %u cons %u\n", | ||
| 385 | qp->qpid, be32_to_cpu(cqe->send.msg_msn), | ||
| 386 | cqe_seq, qp->sq_msn_prod, qp->sq_msn_cons); | ||
| 387 | return rc; | ||
| 388 | } | ||
| 389 | |||
| 390 | qp->sq_msn_cons = cqe_seq; | ||
| 391 | |||
| 392 | if (ionic_v1_cqe_error(cqe)) { | ||
| 393 | cqe_idx = qp->sq_msn_idx[(cqe_seq - 1) & qp->sq.mask]; | ||
| 394 | |||
| 395 | meta = &qp->sq_meta[cqe_idx]; | ||
| 396 | meta->len = be32_to_cpu(cqe->status_length); | ||
| 397 | meta->ibsts = ionic_to_ib_status(meta->len); | ||
| 398 | |||
| 399 | ibdev_warn(qp->ibqp.device, | ||
| 400 | "qp %d msn cqe with error\n", qp->qpid); | ||
| 401 | print_hex_dump(KERN_WARNING, "cqe ", DUMP_PREFIX_OFFSET, 16, 1, | ||
| 402 | cqe, sizeof(*cqe), true); | ||
| 403 | } | ||
| 404 | |||
| 405 | return 0; | ||
| 406 | } | ||
| 407 | |||
| 408 | static int ionic_comp_npg(struct ionic_qp *qp, struct ionic_v1_cqe *cqe) | ||
| 409 | { | ||
| 410 | struct ionic_sq_meta *meta; | ||
| 411 | u16 cqe_idx; | ||
| 412 | u32 st_len; | ||
| 413 | |||
| 414 | if (qp->sq_flush) | ||
| 415 | return 0; | ||
| 416 | |||
| 417 | st_len = be32_to_cpu(cqe->status_length); | ||
| 418 | |||
| 419 | if (ionic_v1_cqe_error(cqe) && st_len == IONIC_STS_WQE_FLUSHED_ERR) { | ||
| 420 | /* | ||
| 421 | * Flush cqe does not consume a wqe on the device, and maybe | ||
| 422 | * no such work request is posted. | ||
| 423 | * | ||
| 424 | * The driver should begin flushing after the last indicated | ||
| 425 | * normal or error completion. Here, only set a hint that the | ||
| 426 | * flush request was indicated. In poll_send, if nothing more | ||
| 427 | * can be polled normally, then begin flushing. | ||
| 428 | */ | ||
| 429 | qp->sq_flush_rcvd = true; | ||
| 430 | return 0; | ||
| 431 | } | ||
| 432 | |||
| 433 | cqe_idx = cqe->send.npg_wqe_id & qp->sq.mask; | ||
| 434 | meta = &qp->sq_meta[cqe_idx]; | ||
| 435 | meta->local_comp = true; | ||
| 436 | |||
| 437 | if (ionic_v1_cqe_error(cqe)) { | ||
| 438 | meta->len = st_len; | ||
| 439 | meta->ibsts = ionic_to_ib_status(st_len); | ||
| 440 | meta->remote = false; | ||
| 441 | ibdev_warn(qp->ibqp.device, | ||
| 442 | "qp %d npg cqe with error\n", qp->qpid); | ||
| 443 | print_hex_dump(KERN_WARNING, "cqe ", DUMP_PREFIX_OFFSET, 16, 1, | ||
| 444 | cqe, sizeof(*cqe), true); | ||
| 445 | } | ||
| 446 | |||
| 447 | return 0; | ||
| 448 | } | ||
| 449 | |||
| 450 | static void ionic_reserve_sync_cq(struct ionic_ibdev *dev, struct ionic_cq *cq) | ||
| 451 | { | ||
| 452 | if (!ionic_queue_empty(&cq->q)) { | ||
| 453 | cq->credit += ionic_queue_length(&cq->q); | ||
| 454 | cq->q.cons = cq->q.prod; | ||
| 455 | |||
| 456 | ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.cq_qtype, | ||
| 457 | ionic_queue_dbell_val(&cq->q)); | ||
| 458 | } | ||
| 459 | } | ||
| 460 | |||
| 461 | static void ionic_reserve_cq(struct ionic_ibdev *dev, struct ionic_cq *cq, | ||
| 462 | int spend) | ||
| 463 | { | ||
| 464 | cq->credit -= spend; | ||
| 465 | |||
| 466 | if (cq->credit <= 0) | ||
| 467 | ionic_reserve_sync_cq(dev, cq); | ||
| 468 | } | ||
| 469 | |||
| 470 | static int ionic_poll_vcq_cq(struct ionic_ibdev *dev, | ||
| 471 | struct ionic_cq *cq, | ||
| 472 | int nwc, struct ib_wc *wc) | ||
| 473 | { | ||
| 474 | struct ionic_qp *qp, *qp_next; | ||
| 475 | struct ionic_v1_cqe *cqe; | ||
| 476 | int rc = 0, npolled = 0; | ||
| 477 | unsigned long irqflags; | ||
| 478 | u32 qtf, qid; | ||
| 479 | bool peek; | ||
| 480 | u8 type; | ||
| 481 | |||
| 482 | if (nwc < 1) | ||
| 483 | return 0; | ||
| 484 | |||
| 485 | spin_lock_irqsave(&cq->lock, irqflags); | ||
| 486 | |||
| 487 | /* poll already indicated work completions for send queue */ | ||
| 488 | list_for_each_entry_safe(qp, qp_next, &cq->poll_sq, cq_poll_sq) { | ||
| 489 | if (npolled == nwc) | ||
| 490 | goto out; | ||
| 491 | |||
| 492 | spin_lock(&qp->sq_lock); | ||
| 493 | rc = ionic_poll_send_many(dev, cq, qp, wc + npolled, | ||
| 494 | nwc - npolled); | ||
| 495 | spin_unlock(&qp->sq_lock); | ||
| 496 | |||
| 497 | if (rc > 0) | ||
| 498 | npolled += rc; | ||
| 499 | |||
| 500 | if (npolled < nwc) | ||
| 501 | list_del_init(&qp->cq_poll_sq); | ||
| 502 | } | ||
| 503 | |||
| 504 | /* poll for more work completions */ | ||
| 505 | while (likely(ionic_next_cqe(dev, cq, &cqe))) { | ||
| 506 | if (npolled == nwc) | ||
| 507 | goto out; | ||
| 508 | |||
| 509 | qtf = ionic_v1_cqe_qtf(cqe); | ||
| 510 | qid = ionic_v1_cqe_qtf_qid(qtf); | ||
| 511 | type = ionic_v1_cqe_qtf_type(qtf); | ||
| 512 | |||
| 513 | /* | ||
| 514 | * Safe to access QP without additional reference here as, | ||
| 515 | * 1. We hold cq->lock throughout | ||
| 516 | * 2. ionic_destroy_qp() acquires the same cq->lock before cleanup | ||
| 517 | * 3. QP is removed from qp_tbl before any cleanup begins | ||
| 518 | * This ensures no concurrent access between polling and destruction. | ||
| 519 | */ | ||
| 520 | qp = xa_load(&dev->qp_tbl, qid); | ||
| 521 | if (unlikely(!qp)) { | ||
| 522 | ibdev_dbg(&dev->ibdev, "missing qp for qid %u\n", qid); | ||
| 523 | goto cq_next; | ||
| 524 | } | ||
| 525 | |||
| 526 | switch (type) { | ||
| 527 | case IONIC_V1_CQE_TYPE_RECV: | ||
| 528 | spin_lock(&qp->rq_lock); | ||
| 529 | rc = ionic_poll_recv(dev, cq, qp, cqe, wc + npolled); | ||
| 530 | spin_unlock(&qp->rq_lock); | ||
| 531 | |||
| 532 | if (rc < 0) | ||
| 533 | goto out; | ||
| 534 | |||
| 535 | npolled += rc; | ||
| 536 | |||
| 537 | break; | ||
| 538 | |||
| 539 | case IONIC_V1_CQE_TYPE_SEND_MSN: | ||
| 540 | spin_lock(&qp->sq_lock); | ||
| 541 | rc = ionic_comp_msn(qp, cqe); | ||
| 542 | if (!rc) { | ||
| 543 | rc = ionic_poll_send_many(dev, cq, qp, | ||
| 544 | wc + npolled, | ||
| 545 | nwc - npolled); | ||
| 546 | peek = ionic_peek_send(qp); | ||
| 547 | } | ||
| 548 | spin_unlock(&qp->sq_lock); | ||
| 549 | |||
| 550 | if (rc < 0) | ||
| 551 | goto out; | ||
| 552 | |||
| 553 | npolled += rc; | ||
| 554 | |||
| 555 | if (peek) | ||
| 556 | list_move_tail(&qp->cq_poll_sq, &cq->poll_sq); | ||
| 557 | break; | ||
| 558 | |||
| 559 | case IONIC_V1_CQE_TYPE_SEND_NPG: | ||
| 560 | spin_lock(&qp->sq_lock); | ||
| 561 | rc = ionic_comp_npg(qp, cqe); | ||
| 562 | if (!rc) { | ||
| 563 | rc = ionic_poll_send_many(dev, cq, qp, | ||
| 564 | wc + npolled, | ||
| 565 | nwc - npolled); | ||
| 566 | peek = ionic_peek_send(qp); | ||
| 567 | } | ||
| 568 | spin_unlock(&qp->sq_lock); | ||
| 569 | |||
| 570 | if (rc < 0) | ||
| 571 | goto out; | ||
| 572 | |||
| 573 | npolled += rc; | ||
| 574 | |||
| 575 | if (peek) | ||
| 576 | list_move_tail(&qp->cq_poll_sq, &cq->poll_sq); | ||
| 577 | break; | ||
| 578 | |||
| 579 | default: | ||
| 580 | ibdev_warn(&dev->ibdev, | ||
| 581 | "unexpected cqe type %u\n", type); | ||
| 582 | rc = -EIO; | ||
| 583 | goto out; | ||
| 584 | } | ||
| 585 | |||
| 586 | cq_next: | ||
| 587 | ionic_queue_produce(&cq->q); | ||
| 588 | cq->color = ionic_color_wrap(cq->q.prod, cq->color); | ||
| 589 | } | ||
| 590 | |||
| 591 | /* lastly, flush send and recv queues */ | ||
| 592 | if (likely(!cq->flush)) | ||
| 593 | goto out; | ||
| 594 | |||
| 595 | cq->flush = false; | ||
| 596 | |||
| 597 | list_for_each_entry_safe(qp, qp_next, &cq->flush_sq, cq_flush_sq) { | ||
| 598 | if (npolled == nwc) | ||
| 599 | goto out; | ||
| 600 | |||
| 601 | spin_lock(&qp->sq_lock); | ||
| 602 | rc = ionic_flush_send_many(qp, wc + npolled, nwc - npolled); | ||
| 603 | spin_unlock(&qp->sq_lock); | ||
| 604 | |||
| 605 | if (rc > 0) | ||
| 606 | npolled += rc; | ||
| 607 | |||
| 608 | if (npolled < nwc) | ||
| 609 | list_del_init(&qp->cq_flush_sq); | ||
| 610 | else | ||
| 611 | cq->flush = true; | ||
| 612 | } | ||
| 613 | |||
| 614 | list_for_each_entry_safe(qp, qp_next, &cq->flush_rq, cq_flush_rq) { | ||
| 615 | if (npolled == nwc) | ||
| 616 | goto out; | ||
| 617 | |||
| 618 | spin_lock(&qp->rq_lock); | ||
| 619 | rc = ionic_flush_recv_many(qp, wc + npolled, nwc - npolled); | ||
| 620 | spin_unlock(&qp->rq_lock); | ||
| 621 | |||
| 622 | if (rc > 0) | ||
| 623 | npolled += rc; | ||
| 624 | |||
| 625 | if (npolled < nwc) | ||
| 626 | list_del_init(&qp->cq_flush_rq); | ||
| 627 | else | ||
| 628 | cq->flush = true; | ||
| 629 | } | ||
| 630 | |||
| 631 | out: | ||
| 632 | /* in case credit was depleted (more work posted than cq depth) */ | ||
| 633 | if (cq->credit <= 0) | ||
| 634 | ionic_reserve_sync_cq(dev, cq); | ||
| 635 | |||
| 636 | spin_unlock_irqrestore(&cq->lock, irqflags); | ||
| 637 | |||
| 638 | return npolled ?: rc; | ||
| 639 | } | ||
| 640 | |||
| 641 | int ionic_poll_cq(struct ib_cq *ibcq, int nwc, struct ib_wc *wc) | ||
| 642 | { | ||
| 643 | struct ionic_ibdev *dev = to_ionic_ibdev(ibcq->device); | ||
| 644 | struct ionic_vcq *vcq = to_ionic_vcq(ibcq); | ||
| 645 | int rc_tmp, rc = 0, npolled = 0; | ||
| 646 | int cq_i, cq_x, cq_ix; | ||
| 647 | |||
| 648 | cq_x = vcq->poll_idx; | ||
| 649 | vcq->poll_idx ^= dev->lif_cfg.udma_count - 1; | ||
| 650 | |||
| 651 | for (cq_i = 0; npolled < nwc && cq_i < dev->lif_cfg.udma_count; ++cq_i) { | ||
| 652 | cq_ix = cq_i ^ cq_x; | ||
| 653 | |||
| 654 | if (!(vcq->udma_mask & BIT(cq_ix))) | ||
| 655 | continue; | ||
| 656 | |||
| 657 | rc_tmp = ionic_poll_vcq_cq(dev, &vcq->cq[cq_ix], | ||
| 658 | nwc - npolled, | ||
| 659 | wc + npolled); | ||
| 660 | |||
| 661 | if (rc_tmp >= 0) | ||
| 662 | npolled += rc_tmp; | ||
| 663 | else if (!rc) | ||
| 664 | rc = rc_tmp; | ||
| 665 | } | ||
| 666 | |||
| 667 | return npolled ?: rc; | ||
| 668 | } | ||
| 669 | |||
| 670 | static int ionic_req_notify_vcq_cq(struct ionic_ibdev *dev, struct ionic_cq *cq, | ||
| 671 | enum ib_cq_notify_flags flags) | ||
| 672 | { | ||
| 673 | u64 dbell_val = cq->q.dbell; | ||
| 674 | |||
| 675 | if (flags & IB_CQ_SOLICITED) { | ||
| 676 | cq->arm_sol_prod = ionic_queue_next(&cq->q, cq->arm_sol_prod); | ||
| 677 | dbell_val |= cq->arm_sol_prod | IONIC_CQ_RING_SOL; | ||
| 678 | } else { | ||
| 679 | cq->arm_any_prod = ionic_queue_next(&cq->q, cq->arm_any_prod); | ||
| 680 | dbell_val |= cq->arm_any_prod | IONIC_CQ_RING_ARM; | ||
| 681 | } | ||
| 682 | |||
| 683 | ionic_reserve_sync_cq(dev, cq); | ||
| 684 | |||
| 685 | ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.cq_qtype, dbell_val); | ||
| 686 | |||
| 687 | /* | ||
| 688 | * IB_CQ_REPORT_MISSED_EVENTS: | ||
| 689 | * | ||
| 690 | * The queue index in ring zero guarantees no missed events. | ||
| 691 | * | ||
| 692 | * Here, we check if the color bit in the next cqe is flipped. If it | ||
| 693 | * is flipped, then progress can be made by immediately polling the cq. | ||
| 694 | * Still, the cq will be armed, and an event will be generated. The cq | ||
| 695 | * may be empty when polled after the event, because the next poll | ||
| 696 | * after arming the cq can empty it. | ||
| 697 | */ | ||
| 698 | return (flags & IB_CQ_REPORT_MISSED_EVENTS) && | ||
| 699 | cq->color == ionic_v1_cqe_color(ionic_queue_at_prod(&cq->q)); | ||
| 700 | } | ||
| 701 | |||
| 702 | int ionic_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags) | ||
| 703 | { | ||
| 704 | struct ionic_ibdev *dev = to_ionic_ibdev(ibcq->device); | ||
| 705 | struct ionic_vcq *vcq = to_ionic_vcq(ibcq); | ||
| 706 | int rc = 0, cq_i; | ||
| 707 | |||
| 708 | for (cq_i = 0; cq_i < dev->lif_cfg.udma_count; ++cq_i) { | ||
| 709 | if (!(vcq->udma_mask & BIT(cq_i))) | ||
| 710 | continue; | ||
| 711 | |||
| 712 | if (ionic_req_notify_vcq_cq(dev, &vcq->cq[cq_i], flags)) | ||
| 713 | rc = 1; | ||
| 714 | } | ||
| 715 | |||
| 716 | return rc; | ||
| 717 | } | ||
| 718 | |||
| 719 | static s64 ionic_prep_inline(void *data, u32 max_data, | ||
| 720 | const struct ib_sge *ib_sgl, int num_sge) | ||
| 721 | { | ||
| 722 | static const s64 bit_31 = 1u << 31; | ||
| 723 | s64 len = 0, sg_len; | ||
| 724 | int sg_i; | ||
| 725 | |||
| 726 | for (sg_i = 0; sg_i < num_sge; ++sg_i) { | ||
| 727 | sg_len = ib_sgl[sg_i].length; | ||
| 728 | |||
| 729 | /* sge length zero means 2GB */ | ||
| 730 | if (unlikely(sg_len == 0)) | ||
| 731 | sg_len = bit_31; | ||
| 732 | |||
| 733 | /* greater than max inline data is invalid */ | ||
| 734 | if (unlikely(len + sg_len > max_data)) | ||
| 735 | return -EINVAL; | ||
| 736 | |||
| 737 | memcpy(data + len, (void *)ib_sgl[sg_i].addr, sg_len); | ||
| 738 | |||
| 739 | len += sg_len; | ||
| 740 | } | ||
| 741 | |||
| 742 | return len; | ||
| 743 | } | ||
| 744 | |||
| 745 | static s64 ionic_prep_pld(struct ionic_v1_wqe *wqe, | ||
| 746 | union ionic_v1_pld *pld, | ||
| 747 | int spec, u32 max_sge, | ||
| 748 | const struct ib_sge *ib_sgl, | ||
| 749 | int num_sge) | ||
| 750 | { | ||
| 751 | static const s64 bit_31 = 1l << 31; | ||
| 752 | struct ionic_sge *sgl; | ||
| 753 | __be32 *spec32 = NULL; | ||
| 754 | __be16 *spec16 = NULL; | ||
| 755 | s64 len = 0, sg_len; | ||
| 756 | int sg_i = 0; | ||
| 757 | |||
| 758 | if (unlikely(num_sge < 0 || (u32)num_sge > max_sge)) | ||
| 759 | return -EINVAL; | ||
| 760 | |||
| 761 | if (spec && num_sge > IONIC_V1_SPEC_FIRST_SGE) { | ||
| 762 | sg_i = IONIC_V1_SPEC_FIRST_SGE; | ||
| 763 | |||
| 764 | if (num_sge > 8) { | ||
| 765 | wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_SPEC16); | ||
| 766 | spec16 = pld->spec16; | ||
| 767 | } else { | ||
| 768 | wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_SPEC32); | ||
| 769 | spec32 = pld->spec32; | ||
| 770 | } | ||
| 771 | } | ||
| 772 | |||
| 773 | sgl = &pld->sgl[sg_i]; | ||
| 774 | |||
| 775 | for (sg_i = 0; sg_i < num_sge; ++sg_i) { | ||
| 776 | sg_len = ib_sgl[sg_i].length; | ||
| 777 | |||
| 778 | /* sge length zero means 2GB */ | ||
| 779 | if (unlikely(sg_len == 0)) | ||
| 780 | sg_len = bit_31; | ||
| 781 | |||
| 782 | /* greater than 2GB data is invalid */ | ||
| 783 | if (unlikely(len + sg_len > bit_31)) | ||
| 784 | return -EINVAL; | ||
| 785 | |||
| 786 | sgl[sg_i].va = cpu_to_be64(ib_sgl[sg_i].addr); | ||
| 787 | sgl[sg_i].len = cpu_to_be32(sg_len); | ||
| 788 | sgl[sg_i].lkey = cpu_to_be32(ib_sgl[sg_i].lkey); | ||
| 789 | |||
| 790 | if (spec32) { | ||
| 791 | spec32[sg_i] = sgl[sg_i].len; | ||
| 792 | } else if (spec16) { | ||
| 793 | if (unlikely(sg_len > U16_MAX)) | ||
| 794 | return -EINVAL; | ||
| 795 | spec16[sg_i] = cpu_to_be16(sg_len); | ||
| 796 | } | ||
| 797 | |||
| 798 | len += sg_len; | ||
| 799 | } | ||
| 800 | |||
| 801 | return len; | ||
| 802 | } | ||
| 803 | |||
| 804 | static void ionic_prep_base(struct ionic_qp *qp, | ||
| 805 | const struct ib_send_wr *wr, | ||
| 806 | struct ionic_sq_meta *meta, | ||
| 807 | struct ionic_v1_wqe *wqe) | ||
| 808 | { | ||
| 809 | meta->wrid = wr->wr_id; | ||
| 810 | meta->ibsts = IB_WC_SUCCESS; | ||
| 811 | meta->signal = false; | ||
| 812 | meta->local_comp = false; | ||
| 813 | |||
| 814 | wqe->base.wqe_id = qp->sq.prod; | ||
| 815 | |||
| 816 | if (wr->send_flags & IB_SEND_FENCE) | ||
| 817 | wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_FENCE); | ||
| 818 | |||
| 819 | if (wr->send_flags & IB_SEND_SOLICITED) | ||
| 820 | wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_SOL); | ||
| 821 | |||
| 822 | if (qp->sig_all || wr->send_flags & IB_SEND_SIGNALED) { | ||
| 823 | wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_SIG); | ||
| 824 | meta->signal = true; | ||
| 825 | } | ||
| 826 | |||
| 827 | meta->seq = qp->sq_msn_prod; | ||
| 828 | meta->remote = | ||
| 829 | qp->ibqp.qp_type != IB_QPT_UD && | ||
| 830 | qp->ibqp.qp_type != IB_QPT_GSI && | ||
| 831 | !ionic_ibop_is_local(wr->opcode); | ||
| 832 | |||
| 833 | if (meta->remote) { | ||
| 834 | qp->sq_msn_idx[meta->seq] = qp->sq.prod; | ||
| 835 | qp->sq_msn_prod = ionic_queue_next(&qp->sq, qp->sq_msn_prod); | ||
| 836 | } | ||
| 837 | |||
| 838 | ionic_queue_produce(&qp->sq); | ||
| 839 | } | ||
| 840 | |||
| 841 | static int ionic_prep_common(struct ionic_qp *qp, | ||
| 842 | const struct ib_send_wr *wr, | ||
| 843 | struct ionic_sq_meta *meta, | ||
| 844 | struct ionic_v1_wqe *wqe) | ||
| 845 | { | ||
| 846 | s64 signed_len; | ||
| 847 | u32 mval; | ||
| 848 | |||
| 849 | if (wr->send_flags & IB_SEND_INLINE) { | ||
| 850 | wqe->base.num_sge_key = 0; | ||
| 851 | wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_INL); | ||
| 852 | mval = ionic_v1_send_wqe_max_data(qp->sq.stride_log2, false); | ||
| 853 | signed_len = ionic_prep_inline(wqe->common.pld.data, mval, | ||
| 854 | wr->sg_list, wr->num_sge); | ||
| 855 | } else { | ||
| 856 | wqe->base.num_sge_key = wr->num_sge; | ||
| 857 | mval = ionic_v1_send_wqe_max_sge(qp->sq.stride_log2, | ||
| 858 | qp->sq_spec, | ||
| 859 | false); | ||
| 860 | signed_len = ionic_prep_pld(wqe, &wqe->common.pld, | ||
| 861 | qp->sq_spec, mval, | ||
| 862 | wr->sg_list, wr->num_sge); | ||
| 863 | } | ||
| 864 | |||
| 865 | if (unlikely(signed_len < 0)) | ||
| 866 | return signed_len; | ||
| 867 | |||
| 868 | meta->len = signed_len; | ||
| 869 | wqe->common.length = cpu_to_be32(signed_len); | ||
| 870 | |||
| 871 | ionic_prep_base(qp, wr, meta, wqe); | ||
| 872 | |||
| 873 | return 0; | ||
| 874 | } | ||
| 875 | |||
| 876 | static void ionic_prep_sq_wqe(struct ionic_qp *qp, void *wqe) | ||
| 877 | { | ||
| 878 | memset(wqe, 0, 1u << qp->sq.stride_log2); | ||
| 879 | } | ||
| 880 | |||
| 881 | static void ionic_prep_rq_wqe(struct ionic_qp *qp, void *wqe) | ||
| 882 | { | ||
| 883 | memset(wqe, 0, 1u << qp->rq.stride_log2); | ||
| 884 | } | ||
| 885 | |||
| 886 | static int ionic_prep_send(struct ionic_qp *qp, | ||
| 887 | const struct ib_send_wr *wr) | ||
| 888 | { | ||
| 889 | struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device); | ||
| 890 | struct ionic_sq_meta *meta; | ||
| 891 | struct ionic_v1_wqe *wqe; | ||
| 892 | |||
| 893 | meta = &qp->sq_meta[qp->sq.prod]; | ||
| 894 | wqe = ionic_queue_at_prod(&qp->sq); | ||
| 895 | |||
| 896 | ionic_prep_sq_wqe(qp, wqe); | ||
| 897 | |||
| 898 | meta->ibop = IB_WC_SEND; | ||
| 899 | |||
| 900 | switch (wr->opcode) { | ||
| 901 | case IB_WR_SEND: | ||
| 902 | wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, SEND); | ||
| 903 | break; | ||
| 904 | case IB_WR_SEND_WITH_IMM: | ||
| 905 | wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, SEND_IMM); | ||
| 906 | wqe->base.imm_data_key = wr->ex.imm_data; | ||
| 907 | break; | ||
| 908 | case IB_WR_SEND_WITH_INV: | ||
| 909 | wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, SEND_INV); | ||
| 910 | wqe->base.imm_data_key = | ||
| 911 | cpu_to_be32(wr->ex.invalidate_rkey); | ||
| 912 | break; | ||
| 913 | default: | ||
| 914 | return -EINVAL; | ||
| 915 | } | ||
| 916 | |||
| 917 | return ionic_prep_common(qp, wr, meta, wqe); | ||
| 918 | } | ||
| 919 | |||
| 920 | static int ionic_prep_send_ud(struct ionic_qp *qp, | ||
| 921 | const struct ib_ud_wr *wr) | ||
| 922 | { | ||
| 923 | struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device); | ||
| 924 | struct ionic_sq_meta *meta; | ||
| 925 | struct ionic_v1_wqe *wqe; | ||
| 926 | struct ionic_ah *ah; | ||
| 927 | |||
| 928 | if (unlikely(!wr->ah)) | ||
| 929 | return -EINVAL; | ||
| 930 | |||
| 931 | ah = to_ionic_ah(wr->ah); | ||
| 932 | |||
| 933 | meta = &qp->sq_meta[qp->sq.prod]; | ||
| 934 | wqe = ionic_queue_at_prod(&qp->sq); | ||
| 935 | |||
| 936 | ionic_prep_sq_wqe(qp, wqe); | ||
| 937 | |||
| 938 | wqe->common.send.ah_id = cpu_to_be32(ah->ahid); | ||
| 939 | wqe->common.send.dest_qpn = cpu_to_be32(wr->remote_qpn); | ||
| 940 | wqe->common.send.dest_qkey = cpu_to_be32(wr->remote_qkey); | ||
| 941 | |||
| 942 | meta->ibop = IB_WC_SEND; | ||
| 943 | |||
| 944 | switch (wr->wr.opcode) { | ||
| 945 | case IB_WR_SEND: | ||
| 946 | wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, SEND); | ||
| 947 | break; | ||
| 948 | case IB_WR_SEND_WITH_IMM: | ||
| 949 | wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, SEND_IMM); | ||
| 950 | wqe->base.imm_data_key = wr->wr.ex.imm_data; | ||
| 951 | break; | ||
| 952 | default: | ||
| 953 | return -EINVAL; | ||
| 954 | } | ||
| 955 | |||
| 956 | return ionic_prep_common(qp, &wr->wr, meta, wqe); | ||
| 957 | } | ||
| 958 | |||
| 959 | static int ionic_prep_rdma(struct ionic_qp *qp, | ||
| 960 | const struct ib_rdma_wr *wr) | ||
| 961 | { | ||
| 962 | struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device); | ||
| 963 | struct ionic_sq_meta *meta; | ||
| 964 | struct ionic_v1_wqe *wqe; | ||
| 965 | |||
| 966 | meta = &qp->sq_meta[qp->sq.prod]; | ||
| 967 | wqe = ionic_queue_at_prod(&qp->sq); | ||
| 968 | |||
| 969 | ionic_prep_sq_wqe(qp, wqe); | ||
| 970 | |||
| 971 | meta->ibop = IB_WC_RDMA_WRITE; | ||
| 972 | |||
| 973 | switch (wr->wr.opcode) { | ||
| 974 | case IB_WR_RDMA_READ: | ||
| 975 | if (wr->wr.send_flags & (IB_SEND_SOLICITED | IB_SEND_INLINE)) | ||
| 976 | return -EINVAL; | ||
| 977 | meta->ibop = IB_WC_RDMA_READ; | ||
| 978 | wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, RDMA_READ); | ||
| 979 | break; | ||
| 980 | case IB_WR_RDMA_WRITE: | ||
| 981 | if (wr->wr.send_flags & IB_SEND_SOLICITED) | ||
| 982 | return -EINVAL; | ||
| 983 | wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, RDMA_WRITE); | ||
| 984 | break; | ||
| 985 | case IB_WR_RDMA_WRITE_WITH_IMM: | ||
| 986 | wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, RDMA_WRITE_IMM); | ||
| 987 | wqe->base.imm_data_key = wr->wr.ex.imm_data; | ||
| 988 | break; | ||
| 989 | default: | ||
| 990 | return -EINVAL; | ||
| 991 | } | ||
| 992 | |||
| 993 | wqe->common.rdma.remote_va_high = cpu_to_be32(wr->remote_addr >> 32); | ||
| 994 | wqe->common.rdma.remote_va_low = cpu_to_be32(wr->remote_addr); | ||
| 995 | wqe->common.rdma.remote_rkey = cpu_to_be32(wr->rkey); | ||
| 996 | |||
| 997 | return ionic_prep_common(qp, &wr->wr, meta, wqe); | ||
| 998 | } | ||
| 999 | |||
| 1000 | static int ionic_prep_atomic(struct ionic_qp *qp, | ||
| 1001 | const struct ib_atomic_wr *wr) | ||
| 1002 | { | ||
| 1003 | struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device); | ||
| 1004 | struct ionic_sq_meta *meta; | ||
| 1005 | struct ionic_v1_wqe *wqe; | ||
| 1006 | |||
| 1007 | if (wr->wr.num_sge != 1 || wr->wr.sg_list[0].length != 8) | ||
| 1008 | return -EINVAL; | ||
| 1009 | |||
| 1010 | if (wr->wr.send_flags & (IB_SEND_SOLICITED | IB_SEND_INLINE)) | ||
| 1011 | return -EINVAL; | ||
| 1012 | |||
| 1013 | meta = &qp->sq_meta[qp->sq.prod]; | ||
| 1014 | wqe = ionic_queue_at_prod(&qp->sq); | ||
| 1015 | |||
| 1016 | ionic_prep_sq_wqe(qp, wqe); | ||
| 1017 | |||
| 1018 | meta->ibop = IB_WC_RDMA_WRITE; | ||
| 1019 | |||
| 1020 | switch (wr->wr.opcode) { | ||
| 1021 | case IB_WR_ATOMIC_CMP_AND_SWP: | ||
| 1022 | meta->ibop = IB_WC_COMP_SWAP; | ||
| 1023 | wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, ATOMIC_CS); | ||
| 1024 | wqe->atomic.swap_add_high = cpu_to_be32(wr->swap >> 32); | ||
| 1025 | wqe->atomic.swap_add_low = cpu_to_be32(wr->swap); | ||
| 1026 | wqe->atomic.compare_high = cpu_to_be32(wr->compare_add >> 32); | ||
| 1027 | wqe->atomic.compare_low = cpu_to_be32(wr->compare_add); | ||
| 1028 | break; | ||
| 1029 | case IB_WR_ATOMIC_FETCH_AND_ADD: | ||
| 1030 | meta->ibop = IB_WC_FETCH_ADD; | ||
| 1031 | wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, ATOMIC_FA); | ||
| 1032 | wqe->atomic.swap_add_high = cpu_to_be32(wr->compare_add >> 32); | ||
| 1033 | wqe->atomic.swap_add_low = cpu_to_be32(wr->compare_add); | ||
| 1034 | break; | ||
| 1035 | default: | ||
| 1036 | return -EINVAL; | ||
| 1037 | } | ||
| 1038 | |||
| 1039 | wqe->atomic.remote_va_high = cpu_to_be32(wr->remote_addr >> 32); | ||
| 1040 | wqe->atomic.remote_va_low = cpu_to_be32(wr->remote_addr); | ||
| 1041 | wqe->atomic.remote_rkey = cpu_to_be32(wr->rkey); | ||
| 1042 | |||
| 1043 | wqe->base.num_sge_key = 1; | ||
| 1044 | wqe->atomic.sge.va = cpu_to_be64(wr->wr.sg_list[0].addr); | ||
| 1045 | wqe->atomic.sge.len = cpu_to_be32(8); | ||
| 1046 | wqe->atomic.sge.lkey = cpu_to_be32(wr->wr.sg_list[0].lkey); | ||
| 1047 | |||
| 1048 | return ionic_prep_common(qp, &wr->wr, meta, wqe); | ||
| 1049 | } | ||
| 1050 | |||
| 1051 | static int ionic_prep_inv(struct ionic_qp *qp, | ||
| 1052 | const struct ib_send_wr *wr) | ||
| 1053 | { | ||
| 1054 | struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device); | ||
| 1055 | struct ionic_sq_meta *meta; | ||
| 1056 | struct ionic_v1_wqe *wqe; | ||
| 1057 | |||
| 1058 | if (wr->send_flags & (IB_SEND_SOLICITED | IB_SEND_INLINE)) | ||
| 1059 | return -EINVAL; | ||
| 1060 | |||
| 1061 | meta = &qp->sq_meta[qp->sq.prod]; | ||
| 1062 | wqe = ionic_queue_at_prod(&qp->sq); | ||
| 1063 | |||
| 1064 | ionic_prep_sq_wqe(qp, wqe); | ||
| 1065 | |||
| 1066 | wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, LOCAL_INV); | ||
| 1067 | wqe->base.imm_data_key = cpu_to_be32(wr->ex.invalidate_rkey); | ||
| 1068 | |||
| 1069 | meta->len = 0; | ||
| 1070 | meta->ibop = IB_WC_LOCAL_INV; | ||
| 1071 | |||
| 1072 | ionic_prep_base(qp, wr, meta, wqe); | ||
| 1073 | |||
| 1074 | return 0; | ||
| 1075 | } | ||
| 1076 | |||
| 1077 | static int ionic_prep_reg(struct ionic_qp *qp, | ||
| 1078 | const struct ib_reg_wr *wr) | ||
| 1079 | { | ||
| 1080 | struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device); | ||
| 1081 | struct ionic_mr *mr = to_ionic_mr(wr->mr); | ||
| 1082 | struct ionic_sq_meta *meta; | ||
| 1083 | struct ionic_v1_wqe *wqe; | ||
| 1084 | __le64 dma_addr; | ||
| 1085 | int flags; | ||
| 1086 | |||
| 1087 | if (wr->wr.send_flags & (IB_SEND_SOLICITED | IB_SEND_INLINE)) | ||
| 1088 | return -EINVAL; | ||
| 1089 | |||
| 1090 | /* must call ib_map_mr_sg before posting reg wr */ | ||
| 1091 | if (!mr->buf.tbl_pages) | ||
| 1092 | return -EINVAL; | ||
| 1093 | |||
| 1094 | meta = &qp->sq_meta[qp->sq.prod]; | ||
| 1095 | wqe = ionic_queue_at_prod(&qp->sq); | ||
| 1096 | |||
| 1097 | ionic_prep_sq_wqe(qp, wqe); | ||
| 1098 | |||
| 1099 | flags = to_ionic_mr_flags(wr->access); | ||
| 1100 | |||
| 1101 | wqe->base.op = IONIC_OP(dev->lif_cfg.rdma_version, REG_MR); | ||
| 1102 | wqe->base.num_sge_key = wr->key; | ||
| 1103 | wqe->base.imm_data_key = cpu_to_be32(mr->ibmr.lkey); | ||
| 1104 | wqe->reg_mr.va = cpu_to_be64(mr->ibmr.iova); | ||
| 1105 | wqe->reg_mr.length = cpu_to_be64(mr->ibmr.length); | ||
| 1106 | wqe->reg_mr.offset = ionic_pgtbl_off(&mr->buf, mr->ibmr.iova); | ||
| 1107 | dma_addr = ionic_pgtbl_dma(&mr->buf, mr->ibmr.iova); | ||
| 1108 | wqe->reg_mr.dma_addr = cpu_to_be64(le64_to_cpu(dma_addr)); | ||
| 1109 | |||
| 1110 | wqe->reg_mr.map_count = cpu_to_be32(mr->buf.tbl_pages); | ||
| 1111 | wqe->reg_mr.flags = cpu_to_be16(flags); | ||
| 1112 | wqe->reg_mr.dir_size_log2 = 0; | ||
| 1113 | wqe->reg_mr.page_size_log2 = order_base_2(mr->ibmr.page_size); | ||
| 1114 | |||
| 1115 | meta->len = 0; | ||
| 1116 | meta->ibop = IB_WC_REG_MR; | ||
| 1117 | |||
| 1118 | ionic_prep_base(qp, &wr->wr, meta, wqe); | ||
| 1119 | |||
| 1120 | return 0; | ||
| 1121 | } | ||
| 1122 | |||
| 1123 | static int ionic_prep_one_rc(struct ionic_qp *qp, | ||
| 1124 | const struct ib_send_wr *wr) | ||
| 1125 | { | ||
| 1126 | struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device); | ||
| 1127 | int rc = 0; | ||
| 1128 | |||
| 1129 | switch (wr->opcode) { | ||
| 1130 | case IB_WR_SEND: | ||
| 1131 | case IB_WR_SEND_WITH_IMM: | ||
| 1132 | case IB_WR_SEND_WITH_INV: | ||
| 1133 | rc = ionic_prep_send(qp, wr); | ||
| 1134 | break; | ||
| 1135 | case IB_WR_RDMA_READ: | ||
| 1136 | case IB_WR_RDMA_WRITE: | ||
| 1137 | case IB_WR_RDMA_WRITE_WITH_IMM: | ||
| 1138 | rc = ionic_prep_rdma(qp, rdma_wr(wr)); | ||
| 1139 | break; | ||
| 1140 | case IB_WR_ATOMIC_CMP_AND_SWP: | ||
| 1141 | case IB_WR_ATOMIC_FETCH_AND_ADD: | ||
| 1142 | rc = ionic_prep_atomic(qp, atomic_wr(wr)); | ||
| 1143 | break; | ||
| 1144 | case IB_WR_LOCAL_INV: | ||
| 1145 | rc = ionic_prep_inv(qp, wr); | ||
| 1146 | break; | ||
| 1147 | case IB_WR_REG_MR: | ||
| 1148 | rc = ionic_prep_reg(qp, reg_wr(wr)); | ||
| 1149 | break; | ||
| 1150 | default: | ||
| 1151 | ibdev_dbg(&dev->ibdev, "invalid opcode %d\n", wr->opcode); | ||
| 1152 | rc = -EINVAL; | ||
| 1153 | } | ||
| 1154 | |||
| 1155 | return rc; | ||
| 1156 | } | ||
| 1157 | |||
| 1158 | static int ionic_prep_one_ud(struct ionic_qp *qp, | ||
| 1159 | const struct ib_send_wr *wr) | ||
| 1160 | { | ||
| 1161 | struct ionic_ibdev *dev = to_ionic_ibdev(qp->ibqp.device); | ||
| 1162 | int rc = 0; | ||
| 1163 | |||
| 1164 | switch (wr->opcode) { | ||
| 1165 | case IB_WR_SEND: | ||
| 1166 | case IB_WR_SEND_WITH_IMM: | ||
| 1167 | rc = ionic_prep_send_ud(qp, ud_wr(wr)); | ||
| 1168 | break; | ||
| 1169 | default: | ||
| 1170 | ibdev_dbg(&dev->ibdev, "invalid opcode %d\n", wr->opcode); | ||
| 1171 | rc = -EINVAL; | ||
| 1172 | } | ||
| 1173 | |||
| 1174 | return rc; | ||
| 1175 | } | ||
| 1176 | |||
| 1177 | static int ionic_prep_recv(struct ionic_qp *qp, | ||
| 1178 | const struct ib_recv_wr *wr) | ||
| 1179 | { | ||
| 1180 | struct ionic_rq_meta *meta; | ||
| 1181 | struct ionic_v1_wqe *wqe; | ||
| 1182 | s64 signed_len; | ||
| 1183 | u32 mval; | ||
| 1184 | |||
| 1185 | wqe = ionic_queue_at_prod(&qp->rq); | ||
| 1186 | |||
| 1187 | /* if wqe is owned by device, caller can try posting again soon */ | ||
| 1188 | if (wqe->base.flags & cpu_to_be16(IONIC_V1_FLAG_FENCE)) | ||
| 1189 | return -EAGAIN; | ||
| 1190 | |||
| 1191 | meta = qp->rq_meta_head; | ||
| 1192 | if (unlikely(meta == IONIC_META_LAST) || | ||
| 1193 | unlikely(meta == IONIC_META_POSTED)) | ||
| 1194 | return -EIO; | ||
| 1195 | |||
| 1196 | ionic_prep_rq_wqe(qp, wqe); | ||
| 1197 | |||
| 1198 | mval = ionic_v1_recv_wqe_max_sge(qp->rq.stride_log2, qp->rq_spec, | ||
| 1199 | false); | ||
| 1200 | signed_len = ionic_prep_pld(wqe, &wqe->recv.pld, | ||
| 1201 | qp->rq_spec, mval, | ||
| 1202 | wr->sg_list, wr->num_sge); | ||
| 1203 | if (signed_len < 0) | ||
| 1204 | return signed_len; | ||
| 1205 | |||
| 1206 | meta->wrid = wr->wr_id; | ||
| 1207 | |||
| 1208 | wqe->base.wqe_id = meta - qp->rq_meta; | ||
| 1209 | wqe->base.num_sge_key = wr->num_sge; | ||
| 1210 | |||
| 1211 | /* total length for recv goes in base imm_data_key */ | ||
| 1212 | wqe->base.imm_data_key = cpu_to_be32(signed_len); | ||
| 1213 | |||
| 1214 | ionic_queue_produce(&qp->rq); | ||
| 1215 | |||
| 1216 | qp->rq_meta_head = meta->next; | ||
| 1217 | meta->next = IONIC_META_POSTED; | ||
| 1218 | |||
| 1219 | return 0; | ||
| 1220 | } | ||
| 1221 | |||
| 1222 | static int ionic_post_send_common(struct ionic_ibdev *dev, | ||
| 1223 | struct ionic_vcq *vcq, | ||
| 1224 | struct ionic_cq *cq, | ||
| 1225 | struct ionic_qp *qp, | ||
| 1226 | const struct ib_send_wr *wr, | ||
| 1227 | const struct ib_send_wr **bad) | ||
| 1228 | { | ||
| 1229 | unsigned long irqflags; | ||
| 1230 | bool notify = false; | ||
| 1231 | int spend, rc = 0; | ||
| 1232 | |||
| 1233 | if (!bad) | ||
| 1234 | return -EINVAL; | ||
| 1235 | |||
| 1236 | if (!qp->has_sq) { | ||
| 1237 | *bad = wr; | ||
| 1238 | return -EINVAL; | ||
| 1239 | } | ||
| 1240 | |||
| 1241 | if (qp->state < IB_QPS_RTS) { | ||
| 1242 | *bad = wr; | ||
| 1243 | return -EINVAL; | ||
| 1244 | } | ||
| 1245 | |||
| 1246 | spin_lock_irqsave(&qp->sq_lock, irqflags); | ||
| 1247 | |||
| 1248 | while (wr) { | ||
| 1249 | if (ionic_queue_full(&qp->sq)) { | ||
| 1250 | ibdev_dbg(&dev->ibdev, "queue full"); | ||
| 1251 | rc = -ENOMEM; | ||
| 1252 | goto out; | ||
| 1253 | } | ||
| 1254 | |||
| 1255 | if (qp->ibqp.qp_type == IB_QPT_UD || | ||
| 1256 | qp->ibqp.qp_type == IB_QPT_GSI) | ||
| 1257 | rc = ionic_prep_one_ud(qp, wr); | ||
| 1258 | else | ||
| 1259 | rc = ionic_prep_one_rc(qp, wr); | ||
| 1260 | if (rc) | ||
| 1261 | goto out; | ||
| 1262 | |||
| 1263 | wr = wr->next; | ||
| 1264 | } | ||
| 1265 | |||
| 1266 | out: | ||
| 1267 | spin_unlock_irqrestore(&qp->sq_lock, irqflags); | ||
| 1268 | |||
| 1269 | spin_lock_irqsave(&cq->lock, irqflags); | ||
| 1270 | spin_lock(&qp->sq_lock); | ||
| 1271 | |||
| 1272 | if (likely(qp->sq.prod != qp->sq_old_prod)) { | ||
| 1273 | /* ring cq doorbell just in time */ | ||
| 1274 | spend = (qp->sq.prod - qp->sq_old_prod) & qp->sq.mask; | ||
| 1275 | ionic_reserve_cq(dev, cq, spend); | ||
| 1276 | |||
| 1277 | qp->sq_old_prod = qp->sq.prod; | ||
| 1278 | |||
| 1279 | ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.sq_qtype, | ||
| 1280 | ionic_queue_dbell_val(&qp->sq)); | ||
| 1281 | } | ||
| 1282 | |||
| 1283 | if (qp->sq_flush) { | ||
| 1284 | notify = true; | ||
| 1285 | cq->flush = true; | ||
| 1286 | list_move_tail(&qp->cq_flush_sq, &cq->flush_sq); | ||
| 1287 | } | ||
| 1288 | |||
| 1289 | spin_unlock(&qp->sq_lock); | ||
| 1290 | spin_unlock_irqrestore(&cq->lock, irqflags); | ||
| 1291 | |||
| 1292 | if (notify && vcq->ibcq.comp_handler) | ||
| 1293 | vcq->ibcq.comp_handler(&vcq->ibcq, vcq->ibcq.cq_context); | ||
| 1294 | |||
| 1295 | *bad = wr; | ||
| 1296 | return rc; | ||
| 1297 | } | ||
| 1298 | |||
| 1299 | static int ionic_post_recv_common(struct ionic_ibdev *dev, | ||
| 1300 | struct ionic_vcq *vcq, | ||
| 1301 | struct ionic_cq *cq, | ||
| 1302 | struct ionic_qp *qp, | ||
| 1303 | const struct ib_recv_wr *wr, | ||
| 1304 | const struct ib_recv_wr **bad) | ||
| 1305 | { | ||
| 1306 | unsigned long irqflags; | ||
| 1307 | bool notify = false; | ||
| 1308 | int spend, rc = 0; | ||
| 1309 | |||
| 1310 | if (!bad) | ||
| 1311 | return -EINVAL; | ||
| 1312 | |||
| 1313 | if (!qp->has_rq) { | ||
| 1314 | *bad = wr; | ||
| 1315 | return -EINVAL; | ||
| 1316 | } | ||
| 1317 | |||
| 1318 | if (qp->state < IB_QPS_INIT) { | ||
| 1319 | *bad = wr; | ||
| 1320 | return -EINVAL; | ||
| 1321 | } | ||
| 1322 | |||
| 1323 | spin_lock_irqsave(&qp->rq_lock, irqflags); | ||
| 1324 | |||
| 1325 | while (wr) { | ||
| 1326 | if (ionic_queue_full(&qp->rq)) { | ||
| 1327 | ibdev_dbg(&dev->ibdev, "queue full"); | ||
| 1328 | rc = -ENOMEM; | ||
| 1329 | goto out; | ||
| 1330 | } | ||
| 1331 | |||
| 1332 | rc = ionic_prep_recv(qp, wr); | ||
| 1333 | if (rc) | ||
| 1334 | goto out; | ||
| 1335 | |||
| 1336 | wr = wr->next; | ||
| 1337 | } | ||
| 1338 | |||
| 1339 | out: | ||
| 1340 | if (!cq) { | ||
| 1341 | spin_unlock_irqrestore(&qp->rq_lock, irqflags); | ||
| 1342 | goto out_unlocked; | ||
| 1343 | } | ||
| 1344 | spin_unlock_irqrestore(&qp->rq_lock, irqflags); | ||
| 1345 | |||
| 1346 | spin_lock_irqsave(&cq->lock, irqflags); | ||
| 1347 | spin_lock(&qp->rq_lock); | ||
| 1348 | |||
| 1349 | if (likely(qp->rq.prod != qp->rq_old_prod)) { | ||
| 1350 | /* ring cq doorbell just in time */ | ||
| 1351 | spend = (qp->rq.prod - qp->rq_old_prod) & qp->rq.mask; | ||
| 1352 | ionic_reserve_cq(dev, cq, spend); | ||
| 1353 | |||
| 1354 | qp->rq_old_prod = qp->rq.prod; | ||
| 1355 | |||
| 1356 | ionic_dbell_ring(dev->lif_cfg.dbpage, dev->lif_cfg.rq_qtype, | ||
| 1357 | ionic_queue_dbell_val(&qp->rq)); | ||
| 1358 | } | ||
| 1359 | |||
| 1360 | if (qp->rq_flush) { | ||
| 1361 | notify = true; | ||
| 1362 | cq->flush = true; | ||
| 1363 | list_move_tail(&qp->cq_flush_rq, &cq->flush_rq); | ||
| 1364 | } | ||
| 1365 | |||
| 1366 | spin_unlock(&qp->rq_lock); | ||
| 1367 | spin_unlock_irqrestore(&cq->lock, irqflags); | ||
| 1368 | |||
| 1369 | if (notify && vcq->ibcq.comp_handler) | ||
| 1370 | vcq->ibcq.comp_handler(&vcq->ibcq, vcq->ibcq.cq_context); | ||
| 1371 | |||
| 1372 | out_unlocked: | ||
| 1373 | *bad = wr; | ||
| 1374 | return rc; | ||
| 1375 | } | ||
| 1376 | |||
| 1377 | int ionic_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr, | ||
| 1378 | const struct ib_send_wr **bad) | ||
| 1379 | { | ||
| 1380 | struct ionic_ibdev *dev = to_ionic_ibdev(ibqp->device); | ||
| 1381 | struct ionic_vcq *vcq = to_ionic_vcq(ibqp->send_cq); | ||
| 1382 | struct ionic_qp *qp = to_ionic_qp(ibqp); | ||
| 1383 | struct ionic_cq *cq = | ||
| 1384 | to_ionic_vcq_cq(ibqp->send_cq, qp->udma_idx); | ||
| 1385 | |||
| 1386 | return ionic_post_send_common(dev, vcq, cq, qp, wr, bad); | ||
| 1387 | } | ||
| 1388 | |||
| 1389 | int ionic_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr, | ||
| 1390 | const struct ib_recv_wr **bad) | ||
| 1391 | { | ||
| 1392 | struct ionic_ibdev *dev = to_ionic_ibdev(ibqp->device); | ||
| 1393 | struct ionic_vcq *vcq = to_ionic_vcq(ibqp->recv_cq); | ||
| 1394 | struct ionic_qp *qp = to_ionic_qp(ibqp); | ||
| 1395 | struct ionic_cq *cq = | ||
| 1396 | to_ionic_vcq_cq(ibqp->recv_cq, qp->udma_idx); | ||
| 1397 | |||
| 1398 | return ionic_post_recv_common(dev, vcq, cq, qp, wr, bad); | ||
| 1399 | } | ||
| @@ -0,0 +1,1029 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 */ | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #ifndef _IONIC_FW_H_ | ||
| 5 | #define _IONIC_FW_H_ | ||
| 6 | |||
| 7 | #include <linux/kernel.h> | ||
| 8 | #include <rdma/ib_verbs.h> | ||
| 9 | |||
| 10 | /* common for ib spec */ | ||
| 11 | |||
| 12 | #define IONIC_EXP_DBELL_SZ 8 | ||
| 13 | |||
| 14 | enum ionic_mrid_bits { | ||
| 15 | IONIC_MRID_INDEX_SHIFT = 8, | ||
| 16 | }; | ||
| 17 | |||
| 18 | static inline u32 ionic_mrid(u32 index, u8 key) | ||
| 19 | { | ||
| 20 | return (index << IONIC_MRID_INDEX_SHIFT) | key; | ||
| 21 | } | ||
| 22 | |||
| 23 | static inline u32 ionic_mrid_index(u32 lrkey) | ||
| 24 | { | ||
| 25 | return lrkey >> IONIC_MRID_INDEX_SHIFT; | ||
| 26 | } | ||
| 27 | |||
| 28 | /* common to all versions */ | ||
| 29 | |||
| 30 | /* wqe scatter gather element */ | ||
| 31 | struct ionic_sge { | ||
| 32 | __be64 va; | ||
| 33 | __be32 len; | ||
| 34 | __be32 lkey; | ||
| 35 | }; | ||
| 36 | |||
| 37 | /* admin queue mr type */ | ||
| 38 | enum ionic_mr_flags { | ||
| 39 | /* bits that determine mr access */ | ||
| 40 | IONIC_MRF_LOCAL_WRITE = BIT(0), | ||
| 41 | IONIC_MRF_REMOTE_WRITE = BIT(1), | ||
| 42 | IONIC_MRF_REMOTE_READ = BIT(2), | ||
| 43 | IONIC_MRF_REMOTE_ATOMIC = BIT(3), | ||
| 44 | IONIC_MRF_MW_BIND = BIT(4), | ||
| 45 | IONIC_MRF_ZERO_BASED = BIT(5), | ||
| 46 | IONIC_MRF_ON_DEMAND = BIT(6), | ||
| 47 | IONIC_MRF_PB = BIT(7), | ||
| 48 | IONIC_MRF_ACCESS_MASK = BIT(12) - 1, | ||
| 49 | |||
| 50 | /* bits that determine mr type */ | ||
| 51 | IONIC_MRF_UKEY_EN = BIT(13), | ||
| 52 | IONIC_MRF_IS_MW = BIT(14), | ||
| 53 | IONIC_MRF_INV_EN = BIT(15), | ||
| 54 | |||
| 55 | /* base flags combinations for mr types */ | ||
| 56 | IONIC_MRF_USER_MR = 0, | ||
| 57 | IONIC_MRF_PHYS_MR = (IONIC_MRF_UKEY_EN | | ||
| 58 | IONIC_MRF_INV_EN), | ||
| 59 | IONIC_MRF_MW_1 = (IONIC_MRF_UKEY_EN | | ||
| 60 | IONIC_MRF_IS_MW), | ||
| 61 | IONIC_MRF_MW_2 = (IONIC_MRF_UKEY_EN | | ||
| 62 | IONIC_MRF_IS_MW | | ||
| 63 | IONIC_MRF_INV_EN), | ||
| 64 | }; | ||
| 65 | |||
| 66 | static inline int to_ionic_mr_flags(int access) | ||
| 67 | { | ||
| 68 | int flags = 0; | ||
| 69 | |||
| 70 | if (access & IB_ACCESS_LOCAL_WRITE) | ||
| 71 | flags |= IONIC_MRF_LOCAL_WRITE; | ||
| 72 | |||
| 73 | if (access & IB_ACCESS_REMOTE_READ) | ||
| 74 | flags |= IONIC_MRF_REMOTE_READ; | ||
| 75 | |||
| 76 | if (access & IB_ACCESS_REMOTE_WRITE) | ||
| 77 | flags |= IONIC_MRF_REMOTE_WRITE; | ||
| 78 | |||
| 79 | if (access & IB_ACCESS_REMOTE_ATOMIC) | ||
| 80 | flags |= IONIC_MRF_REMOTE_ATOMIC; | ||
| 81 | |||
| 82 | if (access & IB_ACCESS_MW_BIND) | ||
| 83 | flags |= IONIC_MRF_MW_BIND; | ||
| 84 | |||
| 85 | if (access & IB_ZERO_BASED) | ||
| 86 | flags |= IONIC_MRF_ZERO_BASED; | ||
| 87 | |||
| 88 | return flags; | ||
| 89 | } | ||
| 90 | |||
| 91 | enum ionic_qp_flags { | ||
| 92 | /* bits that determine qp access */ | ||
| 93 | IONIC_QPF_REMOTE_WRITE = BIT(0), | ||
| 94 | IONIC_QPF_REMOTE_READ = BIT(1), | ||
| 95 | IONIC_QPF_REMOTE_ATOMIC = BIT(2), | ||
| 96 | |||
| 97 | /* bits that determine other qp behavior */ | ||
| 98 | IONIC_QPF_SQ_PB = BIT(6), | ||
| 99 | IONIC_QPF_RQ_PB = BIT(7), | ||
| 100 | IONIC_QPF_SQ_SPEC = BIT(8), | ||
| 101 | IONIC_QPF_RQ_SPEC = BIT(9), | ||
| 102 | IONIC_QPF_REMOTE_PRIVILEGED = BIT(10), | ||
| 103 | IONIC_QPF_SQ_DRAINING = BIT(11), | ||
| 104 | IONIC_QPF_SQD_NOTIFY = BIT(12), | ||
| 105 | IONIC_QPF_SQ_CMB = BIT(13), | ||
| 106 | IONIC_QPF_RQ_CMB = BIT(14), | ||
| 107 | IONIC_QPF_PRIVILEGED = BIT(15), | ||
| 108 | }; | ||
| 109 | |||
| 110 | static inline int from_ionic_qp_flags(int flags) | ||
| 111 | { | ||
| 112 | int access_flags = 0; | ||
| 113 | |||
| 114 | if (flags & IONIC_QPF_REMOTE_WRITE) | ||
| 115 | access_flags |= IB_ACCESS_REMOTE_WRITE; | ||
| 116 | |||
| 117 | if (flags & IONIC_QPF_REMOTE_READ) | ||
| 118 | access_flags |= IB_ACCESS_REMOTE_READ; | ||
| 119 | |||
| 120 | if (flags & IONIC_QPF_REMOTE_ATOMIC) | ||
| 121 | access_flags |= IB_ACCESS_REMOTE_ATOMIC; | ||
| 122 | |||
| 123 | return access_flags; | ||
| 124 | } | ||
| 125 | |||
| 126 | static inline int to_ionic_qp_flags(int access, bool sqd_notify, | ||
| 127 | bool sq_is_cmb, bool rq_is_cmb, | ||
| 128 | bool sq_spec, bool rq_spec, | ||
| 129 | bool privileged, bool remote_privileged) | ||
| 130 | { | ||
| 131 | int flags = 0; | ||
| 132 | |||
| 133 | if (access & IB_ACCESS_REMOTE_WRITE) | ||
| 134 | flags |= IONIC_QPF_REMOTE_WRITE; | ||
| 135 | |||
| 136 | if (access & IB_ACCESS_REMOTE_READ) | ||
| 137 | flags |= IONIC_QPF_REMOTE_READ; | ||
| 138 | |||
| 139 | if (access & IB_ACCESS_REMOTE_ATOMIC) | ||
| 140 | flags |= IONIC_QPF_REMOTE_ATOMIC; | ||
| 141 | |||
| 142 | if (sqd_notify) | ||
| 143 | flags |= IONIC_QPF_SQD_NOTIFY; | ||
| 144 | |||
| 145 | if (sq_is_cmb) | ||
| 146 | flags |= IONIC_QPF_SQ_CMB; | ||
| 147 | |||
| 148 | if (rq_is_cmb) | ||
| 149 | flags |= IONIC_QPF_RQ_CMB; | ||
| 150 | |||
| 151 | if (sq_spec) | ||
| 152 | flags |= IONIC_QPF_SQ_SPEC; | ||
| 153 | |||
| 154 | if (rq_spec) | ||
| 155 | flags |= IONIC_QPF_RQ_SPEC; | ||
| 156 | |||
| 157 | if (privileged) | ||
| 158 | flags |= IONIC_QPF_PRIVILEGED; | ||
| 159 | |||
| 160 | if (remote_privileged) | ||
| 161 | flags |= IONIC_QPF_REMOTE_PRIVILEGED; | ||
| 162 | |||
| 163 | return flags; | ||
| 164 | } | ||
| 165 | |||
| 166 | /* cqe non-admin status indicated in status_length field when err bit is set */ | ||
| 167 | enum ionic_status { | ||
| 168 | IONIC_STS_OK, | ||
| 169 | IONIC_STS_LOCAL_LEN_ERR, | ||
| 170 | IONIC_STS_LOCAL_QP_OPER_ERR, | ||
| 171 | IONIC_STS_LOCAL_PROT_ERR, | ||
| 172 | IONIC_STS_WQE_FLUSHED_ERR, | ||
| 173 | IONIC_STS_MEM_MGMT_OPER_ERR, | ||
| 174 | IONIC_STS_BAD_RESP_ERR, | ||
| 175 | IONIC_STS_LOCAL_ACC_ERR, | ||
| 176 | IONIC_STS_REMOTE_INV_REQ_ERR, | ||
| 177 | IONIC_STS_REMOTE_ACC_ERR, | ||
| 178 | IONIC_STS_REMOTE_OPER_ERR, | ||
| 179 | IONIC_STS_RETRY_EXCEEDED, | ||
| 180 | IONIC_STS_RNR_RETRY_EXCEEDED, | ||
| 181 | IONIC_STS_XRC_VIO_ERR, | ||
| 182 | IONIC_STS_LOCAL_SGL_INV_ERR, | ||
| 183 | }; | ||
| 184 | |||
| 185 | static inline int ionic_to_ib_status(int sts) | ||
| 186 | { | ||
| 187 | switch (sts) { | ||
| 188 | case IONIC_STS_OK: | ||
| 189 | return IB_WC_SUCCESS; | ||
| 190 | case IONIC_STS_LOCAL_LEN_ERR: | ||
| 191 | return IB_WC_LOC_LEN_ERR; | ||
| 192 | case IONIC_STS_LOCAL_QP_OPER_ERR: | ||
| 193 | case IONIC_STS_LOCAL_SGL_INV_ERR: | ||
| 194 | return IB_WC_LOC_QP_OP_ERR; | ||
| 195 | case IONIC_STS_LOCAL_PROT_ERR: | ||
| 196 | return IB_WC_LOC_PROT_ERR; | ||
| 197 | case IONIC_STS_WQE_FLUSHED_ERR: | ||
| 198 | return IB_WC_WR_FLUSH_ERR; | ||
| 199 | case IONIC_STS_MEM_MGMT_OPER_ERR: | ||
| 200 | return IB_WC_MW_BIND_ERR; | ||
| 201 | case IONIC_STS_BAD_RESP_ERR: | ||
| 202 | return IB_WC_BAD_RESP_ERR; | ||
| 203 | case IONIC_STS_LOCAL_ACC_ERR: | ||
| 204 | return IB_WC_LOC_ACCESS_ERR; | ||
| 205 | case IONIC_STS_REMOTE_INV_REQ_ERR: | ||
| 206 | return IB_WC_REM_INV_REQ_ERR; | ||
| 207 | case IONIC_STS_REMOTE_ACC_ERR: | ||
| 208 | return IB_WC_REM_ACCESS_ERR; | ||
| 209 | case IONIC_STS_REMOTE_OPER_ERR: | ||
| 210 | return IB_WC_REM_OP_ERR; | ||
| 211 | case IONIC_STS_RETRY_EXCEEDED: | ||
| 212 | return IB_WC_RETRY_EXC_ERR; | ||
| 213 | case IONIC_STS_RNR_RETRY_EXCEEDED: | ||
| 214 | return IB_WC_RNR_RETRY_EXC_ERR; | ||
| 215 | case IONIC_STS_XRC_VIO_ERR: | ||
| 216 | default: | ||
| 217 | return IB_WC_GENERAL_ERR; | ||
| 218 | } | ||
| 219 | } | ||
| 220 | |||
| 221 | /* admin queue qp type */ | ||
| 222 | enum ionic_qp_type { | ||
| 223 | IONIC_QPT_RC, | ||
| 224 | IONIC_QPT_UC, | ||
| 225 | IONIC_QPT_RD, | ||
| 226 | IONIC_QPT_UD, | ||
| 227 | IONIC_QPT_SRQ, | ||
| 228 | IONIC_QPT_XRC_INI, | ||
| 229 | IONIC_QPT_XRC_TGT, | ||
| 230 | IONIC_QPT_XRC_SRQ, | ||
| 231 | }; | ||
| 232 | |||
| 233 | static inline int to_ionic_qp_type(enum ib_qp_type type) | ||
| 234 | { | ||
| 235 | switch (type) { | ||
| 236 | case IB_QPT_GSI: | ||
| 237 | case IB_QPT_UD: | ||
| 238 | return IONIC_QPT_UD; | ||
| 239 | case IB_QPT_RC: | ||
| 240 | return IONIC_QPT_RC; | ||
| 241 | case IB_QPT_UC: | ||
| 242 | return IONIC_QPT_UC; | ||
| 243 | case IB_QPT_XRC_INI: | ||
| 244 | return IONIC_QPT_XRC_INI; | ||
| 245 | case IB_QPT_XRC_TGT: | ||
| 246 | return IONIC_QPT_XRC_TGT; | ||
| 247 | default: | ||
| 248 | return -EINVAL; | ||
| 249 | } | ||
| 250 | } | ||
| 251 | |||
| 252 | /* admin queue qp state */ | ||
| 253 | enum ionic_qp_state { | ||
| 254 | IONIC_QPS_RESET, | ||
| 255 | IONIC_QPS_INIT, | ||
| 256 | IONIC_QPS_RTR, | ||
| 257 | IONIC_QPS_RTS, | ||
| 258 | IONIC_QPS_SQD, | ||
| 259 | IONIC_QPS_SQE, | ||
| 260 | IONIC_QPS_ERR, | ||
| 261 | }; | ||
| 262 | |||
| 263 | static inline int from_ionic_qp_state(enum ionic_qp_state state) | ||
| 264 | { | ||
| 265 | switch (state) { | ||
| 266 | case IONIC_QPS_RESET: | ||
| 267 | return IB_QPS_RESET; | ||
| 268 | case IONIC_QPS_INIT: | ||
| 269 | return IB_QPS_INIT; | ||
| 270 | case IONIC_QPS_RTR: | ||
| 271 | return IB_QPS_RTR; | ||
| 272 | case IONIC_QPS_RTS: | ||
| 273 | return IB_QPS_RTS; | ||
| 274 | case IONIC_QPS_SQD: | ||
| 275 | return IB_QPS_SQD; | ||
| 276 | case IONIC_QPS_SQE: | ||
| 277 | return IB_QPS_SQE; | ||
| 278 | case IONIC_QPS_ERR: | ||
| 279 | return IB_QPS_ERR; | ||
| 280 | default: | ||
| 281 | return -EINVAL; | ||
| 282 | } | ||
| 283 | } | ||
| 284 | |||
| 285 | static inline int to_ionic_qp_state(enum ib_qp_state state) | ||
| 286 | { | ||
| 287 | switch (state) { | ||
| 288 | case IB_QPS_RESET: | ||
| 289 | return IONIC_QPS_RESET; | ||
| 290 | case IB_QPS_INIT: | ||
| 291 | return IONIC_QPS_INIT; | ||
| 292 | case IB_QPS_RTR: | ||
| 293 | return IONIC_QPS_RTR; | ||
| 294 | case IB_QPS_RTS: | ||
| 295 | return IONIC_QPS_RTS; | ||
| 296 | case IB_QPS_SQD: | ||
| 297 | return IONIC_QPS_SQD; | ||
| 298 | case IB_QPS_SQE: | ||
| 299 | return IONIC_QPS_SQE; | ||
| 300 | case IB_QPS_ERR: | ||
| 301 | return IONIC_QPS_ERR; | ||
| 302 | default: | ||
| 303 | return 0; | ||
| 304 | } | ||
| 305 | } | ||
| 306 | |||
| 307 | static inline int to_ionic_qp_modify_state(enum ib_qp_state to_state, | ||
| 308 | enum ib_qp_state from_state) | ||
| 309 | { | ||
| 310 | return to_ionic_qp_state(to_state) | | ||
| 311 | (to_ionic_qp_state(from_state) << 4); | ||
| 312 | } | ||
| 313 | |||
| 314 | /* fw abi v1 */ | ||
| 315 | |||
| 316 | /* data payload part of v1 wqe */ | ||
| 317 | union ionic_v1_pld { | ||
| 318 | struct ionic_sge sgl[2]; | ||
| 319 | __be32 spec32[8]; | ||
| 320 | __be16 spec16[16]; | ||
| 321 | __u8 data[32]; | ||
| 322 | }; | ||
| 323 | |||
| 324 | /* completion queue v1 cqe */ | ||
| 325 | struct ionic_v1_cqe { | ||
| 326 | union { | ||
| 327 | struct { | ||
| 328 | __be16 cmd_idx; | ||
| 329 | __u8 cmd_op; | ||
| 330 | __u8 rsvd[17]; | ||
| 331 | __le16 old_sq_cindex; | ||
| 332 | __le16 old_rq_cq_cindex; | ||
| 333 | } admin; | ||
| 334 | struct { | ||
| 335 | __u64 wqe_id; | ||
| 336 | __be32 src_qpn_op; | ||
| 337 | __u8 src_mac[6]; | ||
| 338 | __be16 vlan_tag; | ||
| 339 | __be32 imm_data_rkey; | ||
| 340 | } recv; | ||
| 341 | struct { | ||
| 342 | __u8 rsvd[4]; | ||
| 343 | __be32 msg_msn; | ||
| 344 | __u8 rsvd2[8]; | ||
| 345 | __u64 npg_wqe_id; | ||
| 346 | } send; | ||
| 347 | }; | ||
| 348 | __be32 status_length; | ||
| 349 | __be32 qid_type_flags; | ||
| 350 | }; | ||
| 351 | |||
| 352 | /* bits for cqe recv */ | ||
| 353 | enum ionic_v1_cqe_src_qpn_bits { | ||
| 354 | IONIC_V1_CQE_RECV_QPN_MASK = 0xffffff, | ||
| 355 | IONIC_V1_CQE_RECV_OP_SHIFT = 24, | ||
| 356 | |||
| 357 | /* MASK could be 0x3, but need 0x1f for makeshift values: | ||
| 358 | * OP_TYPE_RDMA_OPER_WITH_IMM, OP_TYPE_SEND_RCVD | ||
| 359 | */ | ||
| 360 | IONIC_V1_CQE_RECV_OP_MASK = 0x1f, | ||
| 361 | IONIC_V1_CQE_RECV_OP_SEND = 0, | ||
| 362 | IONIC_V1_CQE_RECV_OP_SEND_INV = 1, | ||
| 363 | IONIC_V1_CQE_RECV_OP_SEND_IMM = 2, | ||
| 364 | IONIC_V1_CQE_RECV_OP_RDMA_IMM = 3, | ||
| 365 | |||
| 366 | IONIC_V1_CQE_RECV_IS_IPV4 = BIT(7 + IONIC_V1_CQE_RECV_OP_SHIFT), | ||
| 367 | IONIC_V1_CQE_RECV_IS_VLAN = BIT(6 + IONIC_V1_CQE_RECV_OP_SHIFT), | ||
| 368 | }; | ||
| 369 | |||
| 370 | /* bits for cqe qid_type_flags */ | ||
| 371 | enum ionic_v1_cqe_qtf_bits { | ||
| 372 | IONIC_V1_CQE_COLOR = BIT(0), | ||
| 373 | IONIC_V1_CQE_ERROR = BIT(1), | ||
| 374 | IONIC_V1_CQE_TYPE_SHIFT = 5, | ||
| 375 | IONIC_V1_CQE_TYPE_MASK = 0x7, | ||
| 376 | IONIC_V1_CQE_QID_SHIFT = 8, | ||
| 377 | |||
| 378 | IONIC_V1_CQE_TYPE_ADMIN = 0, | ||
| 379 | IONIC_V1_CQE_TYPE_RECV = 1, | ||
| 380 | IONIC_V1_CQE_TYPE_SEND_MSN = 2, | ||
| 381 | IONIC_V1_CQE_TYPE_SEND_NPG = 3, | ||
| 382 | }; | ||
| 383 | |||
| 384 | static inline bool ionic_v1_cqe_color(struct ionic_v1_cqe *cqe) | ||
| 385 | { | ||
| 386 | return cqe->qid_type_flags & cpu_to_be32(IONIC_V1_CQE_COLOR); | ||
| 387 | } | ||
| 388 | |||
| 389 | static inline bool ionic_v1_cqe_error(struct ionic_v1_cqe *cqe) | ||
| 390 | { | ||
| 391 | return cqe->qid_type_flags & cpu_to_be32(IONIC_V1_CQE_ERROR); | ||
| 392 | } | ||
| 393 | |||
| 394 | static inline bool ionic_v1_cqe_recv_is_ipv4(struct ionic_v1_cqe *cqe) | ||
| 395 | { | ||
| 396 | return cqe->recv.src_qpn_op & cpu_to_be32(IONIC_V1_CQE_RECV_IS_IPV4); | ||
| 397 | } | ||
| 398 | |||
| 399 | static inline bool ionic_v1_cqe_recv_is_vlan(struct ionic_v1_cqe *cqe) | ||
| 400 | { | ||
| 401 | return cqe->recv.src_qpn_op & cpu_to_be32(IONIC_V1_CQE_RECV_IS_VLAN); | ||
| 402 | } | ||
| 403 | |||
| 404 | static inline void ionic_v1_cqe_clean(struct ionic_v1_cqe *cqe) | ||
| 405 | { | ||
| 406 | cqe->qid_type_flags |= cpu_to_be32(~0u << IONIC_V1_CQE_QID_SHIFT); | ||
| 407 | } | ||
| 408 | |||
| 409 | static inline u32 ionic_v1_cqe_qtf(struct ionic_v1_cqe *cqe) | ||
| 410 | { | ||
| 411 | return be32_to_cpu(cqe->qid_type_flags); | ||
| 412 | } | ||
| 413 | |||
| 414 | static inline u8 ionic_v1_cqe_qtf_type(u32 qtf) | ||
| 415 | { | ||
| 416 | return (qtf >> IONIC_V1_CQE_TYPE_SHIFT) & IONIC_V1_CQE_TYPE_MASK; | ||
| 417 | } | ||
| 418 | |||
| 419 | static inline u32 ionic_v1_cqe_qtf_qid(u32 qtf) | ||
| 420 | { | ||
| 421 | return qtf >> IONIC_V1_CQE_QID_SHIFT; | ||
| 422 | } | ||
| 423 | |||
| 424 | /* v1 base wqe header */ | ||
| 425 | struct ionic_v1_base_hdr { | ||
| 426 | __u64 wqe_id; | ||
| 427 | __u8 op; | ||
| 428 | __u8 num_sge_key; | ||
| 429 | __be16 flags; | ||
| 430 | __be32 imm_data_key; | ||
| 431 | }; | ||
| 432 | |||
| 433 | /* v1 receive wqe body */ | ||
| 434 | struct ionic_v1_recv_bdy { | ||
| 435 | __u8 rsvd[16]; | ||
| 436 | union ionic_v1_pld pld; | ||
| 437 | }; | ||
| 438 | |||
| 439 | /* v1 send/rdma wqe body (common, has sgl) */ | ||
| 440 | struct ionic_v1_common_bdy { | ||
| 441 | union { | ||
| 442 | struct { | ||
| 443 | __be32 ah_id; | ||
| 444 | __be32 dest_qpn; | ||
| 445 | __be32 dest_qkey; | ||
| 446 | } send; | ||
| 447 | struct { | ||
| 448 | __be32 remote_va_high; | ||
| 449 | __be32 remote_va_low; | ||
| 450 | __be32 remote_rkey; | ||
| 451 | } rdma; | ||
| 452 | }; | ||
| 453 | __be32 length; | ||
| 454 | union ionic_v1_pld pld; | ||
| 455 | }; | ||
| 456 | |||
| 457 | /* v1 atomic wqe body */ | ||
| 458 | struct ionic_v1_atomic_bdy { | ||
| 459 | __be32 remote_va_high; | ||
| 460 | __be32 remote_va_low; | ||
| 461 | __be32 remote_rkey; | ||
| 462 | __be32 swap_add_high; | ||
| 463 | __be32 swap_add_low; | ||
| 464 | __be32 compare_high; | ||
| 465 | __be32 compare_low; | ||
| 466 | __u8 rsvd[4]; | ||
| 467 | struct ionic_sge sge; | ||
| 468 | }; | ||
| 469 | |||
| 470 | /* v1 reg mr wqe body */ | ||
| 471 | struct ionic_v1_reg_mr_bdy { | ||
| 472 | __be64 va; | ||
| 473 | __be64 length; | ||
| 474 | __be64 offset; | ||
| 475 | __be64 dma_addr; | ||
| 476 | __be32 map_count; | ||
| 477 | __be16 flags; | ||
| 478 | __u8 dir_size_log2; | ||
| 479 | __u8 page_size_log2; | ||
| 480 | __u8 rsvd[8]; | ||
| 481 | }; | ||
| 482 | |||
| 483 | /* v1 bind mw wqe body */ | ||
| 484 | struct ionic_v1_bind_mw_bdy { | ||
| 485 | __be64 va; | ||
| 486 | __be64 length; | ||
| 487 | __be32 lkey; | ||
| 488 | __be16 flags; | ||
| 489 | __u8 rsvd[26]; | ||
| 490 | }; | ||
| 491 | |||
| 492 | /* v1 send/recv wqe */ | ||
| 493 | struct ionic_v1_wqe { | ||
| 494 | struct ionic_v1_base_hdr base; | ||
| 495 | union { | ||
| 496 | struct ionic_v1_recv_bdy recv; | ||
| 497 | struct ionic_v1_common_bdy common; | ||
| 498 | struct ionic_v1_atomic_bdy atomic; | ||
| 499 | struct ionic_v1_reg_mr_bdy reg_mr; | ||
| 500 | struct ionic_v1_bind_mw_bdy bind_mw; | ||
| 501 | }; | ||
| 502 | }; | ||
| 503 | |||
| 504 | /* queue pair v1 send opcodes */ | ||
| 505 | enum ionic_v1_op { | ||
| 506 | IONIC_V1_OP_SEND, | ||
| 507 | IONIC_V1_OP_SEND_INV, | ||
| 508 | IONIC_V1_OP_SEND_IMM, | ||
| 509 | IONIC_V1_OP_RDMA_READ, | ||
| 510 | IONIC_V1_OP_RDMA_WRITE, | ||
| 511 | IONIC_V1_OP_RDMA_WRITE_IMM, | ||
| 512 | IONIC_V1_OP_ATOMIC_CS, | ||
| 513 | IONIC_V1_OP_ATOMIC_FA, | ||
| 514 | IONIC_V1_OP_REG_MR, | ||
| 515 | IONIC_V1_OP_LOCAL_INV, | ||
| 516 | IONIC_V1_OP_BIND_MW, | ||
| 517 | |||
| 518 | /* flags */ | ||
| 519 | IONIC_V1_FLAG_FENCE = BIT(0), | ||
| 520 | IONIC_V1_FLAG_SOL = BIT(1), | ||
| 521 | IONIC_V1_FLAG_INL = BIT(2), | ||
| 522 | IONIC_V1_FLAG_SIG = BIT(3), | ||
| 523 | |||
| 524 | /* flags last four bits for sgl spec format */ | ||
| 525 | IONIC_V1_FLAG_SPEC32 = (1u << 12), | ||
| 526 | IONIC_V1_FLAG_SPEC16 = (2u << 12), | ||
| 527 | IONIC_V1_SPEC_FIRST_SGE = 2, | ||
| 528 | }; | ||
| 529 | |||
| 530 | /* queue pair v2 send opcodes */ | ||
| 531 | enum ionic_v2_op { | ||
| 532 | IONIC_V2_OPSL_OUT = 0x20, | ||
| 533 | IONIC_V2_OPSL_IMM = 0x40, | ||
| 534 | IONIC_V2_OPSL_INV = 0x80, | ||
| 535 | |||
| 536 | IONIC_V2_OP_SEND = 0x0 | IONIC_V2_OPSL_OUT, | ||
| 537 | IONIC_V2_OP_SEND_IMM = IONIC_V2_OP_SEND | IONIC_V2_OPSL_IMM, | ||
| 538 | IONIC_V2_OP_SEND_INV = IONIC_V2_OP_SEND | IONIC_V2_OPSL_INV, | ||
| 539 | |||
| 540 | IONIC_V2_OP_RDMA_WRITE = 0x1 | IONIC_V2_OPSL_OUT, | ||
| 541 | IONIC_V2_OP_RDMA_WRITE_IMM = IONIC_V2_OP_RDMA_WRITE | IONIC_V2_OPSL_IMM, | ||
| 542 | |||
| 543 | IONIC_V2_OP_RDMA_READ = 0x2, | ||
| 544 | |||
| 545 | IONIC_V2_OP_ATOMIC_CS = 0x4, | ||
| 546 | IONIC_V2_OP_ATOMIC_FA = 0x5, | ||
| 547 | IONIC_V2_OP_REG_MR = 0x6, | ||
| 548 | IONIC_V2_OP_LOCAL_INV = 0x7, | ||
| 549 | IONIC_V2_OP_BIND_MW = 0x8, | ||
| 550 | }; | ||
| 551 | |||
| 552 | static inline size_t ionic_v1_send_wqe_min_size(int min_sge, int min_data, | ||
| 553 | int spec, bool expdb) | ||
| 554 | { | ||
| 555 | size_t sz_wqe, sz_sgl, sz_data; | ||
| 556 | |||
| 557 | if (spec > IONIC_V1_SPEC_FIRST_SGE) | ||
| 558 | min_sge += IONIC_V1_SPEC_FIRST_SGE; | ||
| 559 | |||
| 560 | if (expdb) { | ||
| 561 | min_sge += 1; | ||
| 562 | min_data += IONIC_EXP_DBELL_SZ; | ||
| 563 | } | ||
| 564 | |||
| 565 | sz_wqe = sizeof(struct ionic_v1_wqe); | ||
| 566 | sz_sgl = offsetof(struct ionic_v1_wqe, common.pld.sgl[min_sge]); | ||
| 567 | sz_data = offsetof(struct ionic_v1_wqe, common.pld.data[min_data]); | ||
| 568 | |||
| 569 | if (sz_sgl > sz_wqe) | ||
| 570 | sz_wqe = sz_sgl; | ||
| 571 | |||
| 572 | if (sz_data > sz_wqe) | ||
| 573 | sz_wqe = sz_data; | ||
| 574 | |||
| 575 | return sz_wqe; | ||
| 576 | } | ||
| 577 | |||
| 578 | static inline int ionic_v1_send_wqe_max_sge(u8 stride_log2, int spec, | ||
| 579 | bool expdb) | ||
| 580 | { | ||
| 581 | struct ionic_sge *sge = (void *)(1ull << stride_log2); | ||
| 582 | struct ionic_v1_wqe *wqe = (void *)0; | ||
| 583 | int num_sge = 0; | ||
| 584 | |||
| 585 | if (expdb) | ||
| 586 | sge -= 1; | ||
| 587 | |||
| 588 | if (spec > IONIC_V1_SPEC_FIRST_SGE) | ||
| 589 | num_sge = IONIC_V1_SPEC_FIRST_SGE; | ||
| 590 | |||
| 591 | num_sge = sge - &wqe->common.pld.sgl[num_sge]; | ||
| 592 | |||
| 593 | if (spec && num_sge > spec) | ||
| 594 | num_sge = spec; | ||
| 595 | |||
| 596 | return num_sge; | ||
| 597 | } | ||
| 598 | |||
| 599 | static inline int ionic_v1_send_wqe_max_data(u8 stride_log2, bool expdb) | ||
| 600 | { | ||
| 601 | struct ionic_v1_wqe *wqe = (void *)0; | ||
| 602 | __u8 *data = (void *)(1ull << stride_log2); | ||
| 603 | |||
| 604 | if (expdb) | ||
| 605 | data -= IONIC_EXP_DBELL_SZ; | ||
| 606 | |||
| 607 | return data - wqe->common.pld.data; | ||
| 608 | } | ||
| 609 | |||
| 610 | static inline size_t ionic_v1_recv_wqe_min_size(int min_sge, int spec, | ||
| 611 | bool expdb) | ||
| 612 | { | ||
| 613 | size_t sz_wqe, sz_sgl; | ||
| 614 | |||
| 615 | if (spec > IONIC_V1_SPEC_FIRST_SGE) | ||
| 616 | min_sge += IONIC_V1_SPEC_FIRST_SGE; | ||
| 617 | |||
| 618 | if (expdb) | ||
| 619 | min_sge += 1; | ||
| 620 | |||
| 621 | sz_wqe = sizeof(struct ionic_v1_wqe); | ||
| 622 | sz_sgl = offsetof(struct ionic_v1_wqe, recv.pld.sgl[min_sge]); | ||
| 623 | |||
| 624 | if (sz_sgl > sz_wqe) | ||
| 625 | sz_wqe = sz_sgl; | ||
| 626 | |||
| 627 | return sz_wqe; | ||
| 628 | } | ||
| 629 | |||
| 630 | static inline int ionic_v1_recv_wqe_max_sge(u8 stride_log2, int spec, | ||
| 631 | bool expdb) | ||
| 632 | { | ||
| 633 | struct ionic_sge *sge = (void *)(1ull << stride_log2); | ||
| 634 | struct ionic_v1_wqe *wqe = (void *)0; | ||
| 635 | int num_sge = 0; | ||
| 636 | |||
| 637 | if (expdb) | ||
| 638 | sge -= 1; | ||
| 639 | |||
| 640 | if (spec > IONIC_V1_SPEC_FIRST_SGE) | ||
| 641 | num_sge = IONIC_V1_SPEC_FIRST_SGE; | ||
| 642 | |||
| 643 | num_sge = sge - &wqe->recv.pld.sgl[num_sge]; | ||
| 644 | |||
| 645 | if (spec && num_sge > spec) | ||
| 646 | num_sge = spec; | ||
| 647 | |||
| 648 | return num_sge; | ||
| 649 | } | ||
| 650 | |||
| 651 | static inline int ionic_v1_use_spec_sge(int min_sge, int spec) | ||
| 652 | { | ||
| 653 | if (!spec || min_sge > spec) | ||
| 654 | return 0; | ||
| 655 | |||
| 656 | if (min_sge <= IONIC_V1_SPEC_FIRST_SGE) | ||
| 657 | return IONIC_V1_SPEC_FIRST_SGE; | ||
| 658 | |||
| 659 | return spec; | ||
| 660 | } | ||
| 661 | |||
| 662 | struct ionic_admin_stats_hdr { | ||
| 663 | __le64 dma_addr; | ||
| 664 | __le32 length; | ||
| 665 | __le32 id_ver; | ||
| 666 | __u8 type_state; | ||
| 667 | } __packed; | ||
| 668 | |||
| 669 | #define IONIC_ADMIN_STATS_HDRS_IN_V1_LEN 17 | ||
| 670 | static_assert(sizeof(struct ionic_admin_stats_hdr) == | ||
| 671 | IONIC_ADMIN_STATS_HDRS_IN_V1_LEN); | ||
| 672 | |||
| 673 | struct ionic_admin_create_ah { | ||
| 674 | __le64 dma_addr; | ||
| 675 | __le32 length; | ||
| 676 | __le32 pd_id; | ||
| 677 | __le32 id_ver; | ||
| 678 | __le16 dbid_flags; | ||
| 679 | __u8 csum_profile; | ||
| 680 | __u8 crypto; | ||
| 681 | } __packed; | ||
| 682 | |||
| 683 | #define IONIC_ADMIN_CREATE_AH_IN_V1_LEN 24 | ||
| 684 | static_assert(sizeof(struct ionic_admin_create_ah) == | ||
| 685 | IONIC_ADMIN_CREATE_AH_IN_V1_LEN); | ||
| 686 | |||
| 687 | struct ionic_admin_destroy_ah { | ||
| 688 | __le32 ah_id; | ||
| 689 | } __packed; | ||
| 690 | |||
| 691 | #define IONIC_ADMIN_DESTROY_AH_IN_V1_LEN 4 | ||
| 692 | static_assert(sizeof(struct ionic_admin_destroy_ah) == | ||
| 693 | IONIC_ADMIN_DESTROY_AH_IN_V1_LEN); | ||
| 694 | |||
| 695 | struct ionic_admin_query_ah { | ||
| 696 | __le64 dma_addr; | ||
| 697 | } __packed; | ||
| 698 | |||
| 699 | #define IONIC_ADMIN_QUERY_AH_IN_V1_LEN 8 | ||
| 700 | static_assert(sizeof(struct ionic_admin_query_ah) == | ||
| 701 | IONIC_ADMIN_QUERY_AH_IN_V1_LEN); | ||
| 702 | |||
| 703 | struct ionic_admin_create_mr { | ||
| 704 | __le64 va; | ||
| 705 | __le64 length; | ||
| 706 | __le32 pd_id; | ||
| 707 | __le32 id_ver; | ||
| 708 | __le32 tbl_index; | ||
| 709 | __le32 map_count; | ||
| 710 | __le64 dma_addr; | ||
| 711 | __le16 dbid_flags; | ||
| 712 | __u8 pt_type; | ||
| 713 | __u8 dir_size_log2; | ||
| 714 | __u8 page_size_log2; | ||
| 715 | } __packed; | ||
| 716 | |||
| 717 | #define IONIC_ADMIN_CREATE_MR_IN_V1_LEN 45 | ||
| 718 | static_assert(sizeof(struct ionic_admin_create_mr) == | ||
| 719 | IONIC_ADMIN_CREATE_MR_IN_V1_LEN); | ||
| 720 | |||
| 721 | struct ionic_admin_destroy_mr { | ||
| 722 | __le32 mr_id; | ||
| 723 | } __packed; | ||
| 724 | |||
| 725 | #define IONIC_ADMIN_DESTROY_MR_IN_V1_LEN 4 | ||
| 726 | static_assert(sizeof(struct ionic_admin_destroy_mr) == | ||
| 727 | IONIC_ADMIN_DESTROY_MR_IN_V1_LEN); | ||
| 728 | |||
| 729 | struct ionic_admin_create_cq { | ||
| 730 | __le32 eq_id; | ||
| 731 | __u8 depth_log2; | ||
| 732 | __u8 stride_log2; | ||
| 733 | __u8 dir_size_log2_rsvd; | ||
| 734 | __u8 page_size_log2; | ||
| 735 | __le32 cq_flags; | ||
| 736 | __le32 id_ver; | ||
| 737 | __le32 tbl_index; | ||
| 738 | __le32 map_count; | ||
| 739 | __le64 dma_addr; | ||
| 740 | __le16 dbid_flags; | ||
| 741 | } __packed; | ||
| 742 | |||
| 743 | #define IONIC_ADMIN_CREATE_CQ_IN_V1_LEN 34 | ||
| 744 | static_assert(sizeof(struct ionic_admin_create_cq) == | ||
| 745 | IONIC_ADMIN_CREATE_CQ_IN_V1_LEN); | ||
| 746 | |||
| 747 | struct ionic_admin_destroy_cq { | ||
| 748 | __le32 cq_id; | ||
| 749 | } __packed; | ||
| 750 | |||
| 751 | #define IONIC_ADMIN_DESTROY_CQ_IN_V1_LEN 4 | ||
| 752 | static_assert(sizeof(struct ionic_admin_destroy_cq) == | ||
| 753 | IONIC_ADMIN_DESTROY_CQ_IN_V1_LEN); | ||
| 754 | |||
| 755 | struct ionic_admin_create_qp { | ||
| 756 | __le32 pd_id; | ||
| 757 | __be32 priv_flags; | ||
| 758 | __le32 sq_cq_id; | ||
| 759 | __u8 sq_depth_log2; | ||
| 760 | __u8 sq_stride_log2; | ||
| 761 | __u8 sq_dir_size_log2_rsvd; | ||
| 762 | __u8 sq_page_size_log2; | ||
| 763 | __le32 sq_tbl_index_xrcd_id; | ||
| 764 | __le32 sq_map_count; | ||
| 765 | __le64 sq_dma_addr; | ||
| 766 | __le32 rq_cq_id; | ||
| 767 | __u8 rq_depth_log2; | ||
| 768 | __u8 rq_stride_log2; | ||
| 769 | __u8 rq_dir_size_log2_rsvd; | ||
| 770 | __u8 rq_page_size_log2; | ||
| 771 | __le32 rq_tbl_index_srq_id; | ||
| 772 | __le32 rq_map_count; | ||
| 773 | __le64 rq_dma_addr; | ||
| 774 | __le32 id_ver; | ||
| 775 | __le16 dbid_flags; | ||
| 776 | __u8 type_state; | ||
| 777 | __u8 rsvd; | ||
| 778 | } __packed; | ||
| 779 | |||
| 780 | #define IONIC_ADMIN_CREATE_QP_IN_V1_LEN 64 | ||
| 781 | static_assert(sizeof(struct ionic_admin_create_qp) == | ||
| 782 | IONIC_ADMIN_CREATE_QP_IN_V1_LEN); | ||
| 783 | |||
| 784 | struct ionic_admin_destroy_qp { | ||
| 785 | __le32 qp_id; | ||
| 786 | } __packed; | ||
| 787 | |||
| 788 | #define IONIC_ADMIN_DESTROY_QP_IN_V1_LEN 4 | ||
| 789 | static_assert(sizeof(struct ionic_admin_destroy_qp) == | ||
| 790 | IONIC_ADMIN_DESTROY_QP_IN_V1_LEN); | ||
| 791 | |||
| 792 | struct ionic_admin_mod_qp { | ||
| 793 | __be32 attr_mask; | ||
| 794 | __u8 dcqcn_profile; | ||
| 795 | __u8 tfp_csum_profile; | ||
| 796 | __be16 access_flags; | ||
| 797 | __le32 rq_psn; | ||
| 798 | __le32 sq_psn; | ||
| 799 | __le32 qkey_dest_qpn; | ||
| 800 | __le32 rate_limit_kbps; | ||
| 801 | __u8 pmtu; | ||
| 802 | __u8 retry; | ||
| 803 | __u8 rnr_timer; | ||
| 804 | __u8 retry_timeout; | ||
| 805 | __u8 rsq_depth; | ||
| 806 | __u8 rrq_depth; | ||
| 807 | __le16 pkey_id; | ||
| 808 | __le32 ah_id_len; | ||
| 809 | __u8 en_pcp; | ||
| 810 | __u8 ip_dscp; | ||
| 811 | __u8 rsvd2; | ||
| 812 | __u8 type_state; | ||
| 813 | union { | ||
| 814 | struct { | ||
| 815 | __le16 rsvd1; | ||
| 816 | }; | ||
| 817 | __le32 rrq_index; | ||
| 818 | }; | ||
| 819 | __le32 rsq_index; | ||
| 820 | __le64 dma_addr; | ||
| 821 | __le32 id_ver; | ||
| 822 | } __packed; | ||
| 823 | |||
| 824 | #define IONIC_ADMIN_MODIFY_QP_IN_V1_LEN 60 | ||
| 825 | static_assert(sizeof(struct ionic_admin_mod_qp) == | ||
| 826 | IONIC_ADMIN_MODIFY_QP_IN_V1_LEN); | ||
| 827 | |||
| 828 | struct ionic_admin_query_qp { | ||
| 829 | __le64 hdr_dma_addr; | ||
| 830 | __le64 sq_dma_addr; | ||
| 831 | __le64 rq_dma_addr; | ||
| 832 | __le32 ah_id; | ||
| 833 | __le32 id_ver; | ||
| 834 | __le16 dbid_flags; | ||
| 835 | } __packed; | ||
| 836 | |||
| 837 | #define IONIC_ADMIN_QUERY_QP_IN_V1_LEN 34 | ||
| 838 | static_assert(sizeof(struct ionic_admin_query_qp) == | ||
| 839 | IONIC_ADMIN_QUERY_QP_IN_V1_LEN); | ||
| 840 | |||
| 841 | #define ADMIN_WQE_STRIDE 64 | ||
| 842 | #define ADMIN_WQE_HDR_LEN 4 | ||
| 843 | |||
| 844 | /* admin queue v1 wqe */ | ||
| 845 | struct ionic_v1_admin_wqe { | ||
| 846 | __u8 op; | ||
| 847 | __u8 rsvd; | ||
| 848 | __le16 len; | ||
| 849 | |||
| 850 | union { | ||
| 851 | struct ionic_admin_stats_hdr stats; | ||
| 852 | struct ionic_admin_create_ah create_ah; | ||
| 853 | struct ionic_admin_destroy_ah destroy_ah; | ||
| 854 | struct ionic_admin_query_ah query_ah; | ||
| 855 | struct ionic_admin_create_mr create_mr; | ||
| 856 | struct ionic_admin_destroy_mr destroy_mr; | ||
| 857 | struct ionic_admin_create_cq create_cq; | ||
| 858 | struct ionic_admin_destroy_cq destroy_cq; | ||
| 859 | struct ionic_admin_create_qp create_qp; | ||
| 860 | struct ionic_admin_destroy_qp destroy_qp; | ||
| 861 | struct ionic_admin_mod_qp mod_qp; | ||
| 862 | struct ionic_admin_query_qp query_qp; | ||
| 863 | } cmd; | ||
| 864 | }; | ||
| 865 | |||
| 866 | /* side data for query qp */ | ||
| 867 | struct ionic_v1_admin_query_qp_sq { | ||
| 868 | __u8 rnr_timer; | ||
| 869 | __u8 retry_timeout; | ||
| 870 | __be16 access_perms_flags; | ||
| 871 | __be16 rsvd; | ||
| 872 | __be16 pkey_id; | ||
| 873 | __be32 qkey_dest_qpn; | ||
| 874 | __be32 rate_limit_kbps; | ||
| 875 | __be32 rq_psn; | ||
| 876 | }; | ||
| 877 | |||
| 878 | struct ionic_v1_admin_query_qp_rq { | ||
| 879 | __u8 state_pmtu; | ||
| 880 | __u8 retry_rnrtry; | ||
| 881 | __u8 rrq_depth; | ||
| 882 | __u8 rsq_depth; | ||
| 883 | __be32 sq_psn; | ||
| 884 | __be16 access_perms_flags; | ||
| 885 | __be16 rsvd; | ||
| 886 | }; | ||
| 887 | |||
| 888 | /* admin queue v1 opcodes */ | ||
| 889 | enum ionic_v1_admin_op { | ||
| 890 | IONIC_V1_ADMIN_NOOP, | ||
| 891 | IONIC_V1_ADMIN_CREATE_CQ, | ||
| 892 | IONIC_V1_ADMIN_CREATE_QP, | ||
| 893 | IONIC_V1_ADMIN_CREATE_MR, | ||
| 894 | IONIC_V1_ADMIN_STATS_HDRS, | ||
| 895 | IONIC_V1_ADMIN_STATS_VALS, | ||
| 896 | IONIC_V1_ADMIN_DESTROY_MR, | ||
| 897 | IONIC_V1_ADMIN_RSVD_7, /* RESIZE_CQ */ | ||
| 898 | IONIC_V1_ADMIN_DESTROY_CQ, | ||
| 899 | IONIC_V1_ADMIN_MODIFY_QP, | ||
| 900 | IONIC_V1_ADMIN_QUERY_QP, | ||
| 901 | IONIC_V1_ADMIN_DESTROY_QP, | ||
| 902 | IONIC_V1_ADMIN_DEBUG, | ||
| 903 | IONIC_V1_ADMIN_CREATE_AH, | ||
| 904 | IONIC_V1_ADMIN_QUERY_AH, | ||
| 905 | IONIC_V1_ADMIN_MODIFY_DCQCN, | ||
| 906 | IONIC_V1_ADMIN_DESTROY_AH, | ||
| 907 | IONIC_V1_ADMIN_QP_STATS_HDRS, | ||
| 908 | IONIC_V1_ADMIN_QP_STATS_VALS, | ||
| 909 | IONIC_V1_ADMIN_OPCODES_MAX, | ||
| 910 | }; | ||
| 911 | |||
| 912 | /* admin queue v1 cqe status */ | ||
| 913 | enum ionic_v1_admin_status { | ||
| 914 | IONIC_V1_ASTS_OK, | ||
| 915 | IONIC_V1_ASTS_BAD_CMD, | ||
| 916 | IONIC_V1_ASTS_BAD_INDEX, | ||
| 917 | IONIC_V1_ASTS_BAD_STATE, | ||
| 918 | IONIC_V1_ASTS_BAD_TYPE, | ||
| 919 | IONIC_V1_ASTS_BAD_ATTR, | ||
| 920 | IONIC_V1_ASTS_MSG_TOO_BIG, | ||
| 921 | }; | ||
| 922 | |||
| 923 | /* event queue v1 eqe */ | ||
| 924 | struct ionic_v1_eqe { | ||
| 925 | __be32 evt; | ||
| 926 | }; | ||
| 927 | |||
| 928 | /* bits for cqe queue_type_flags */ | ||
| 929 | enum ionic_v1_eqe_evt_bits { | ||
| 930 | IONIC_V1_EQE_COLOR = BIT(0), | ||
| 931 | IONIC_V1_EQE_TYPE_SHIFT = 1, | ||
| 932 | IONIC_V1_EQE_TYPE_MASK = 0x7, | ||
| 933 | IONIC_V1_EQE_CODE_SHIFT = 4, | ||
| 934 | IONIC_V1_EQE_CODE_MASK = 0xf, | ||
| 935 | IONIC_V1_EQE_QID_SHIFT = 8, | ||
| 936 | |||
| 937 | /* cq events */ | ||
| 938 | IONIC_V1_EQE_TYPE_CQ = 0, | ||
| 939 | /* cq normal events */ | ||
| 940 | IONIC_V1_EQE_CQ_NOTIFY = 0, | ||
| 941 | /* cq error events */ | ||
| 942 | IONIC_V1_EQE_CQ_ERR = 8, | ||
| 943 | |||
| 944 | /* qp and srq events */ | ||
| 945 | IONIC_V1_EQE_TYPE_QP = 1, | ||
| 946 | /* qp normal events */ | ||
| 947 | IONIC_V1_EQE_SRQ_LEVEL = 0, | ||
| 948 | IONIC_V1_EQE_SQ_DRAIN = 1, | ||
| 949 | IONIC_V1_EQE_QP_COMM_EST = 2, | ||
| 950 | IONIC_V1_EQE_QP_LAST_WQE = 3, | ||
| 951 | /* qp error events */ | ||
| 952 | IONIC_V1_EQE_QP_ERR = 8, | ||
| 953 | IONIC_V1_EQE_QP_ERR_REQUEST = 9, | ||
| 954 | IONIC_V1_EQE_QP_ERR_ACCESS = 10, | ||
| 955 | }; | ||
| 956 | |||
| 957 | enum ionic_tfp_csum_profiles { | ||
| 958 | IONIC_TFP_CSUM_PROF_ETH_IPV4_UDP = 0, | ||
| 959 | IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_UDP = 1, | ||
| 960 | IONIC_TFP_CSUM_PROF_ETH_IPV6_UDP = 2, | ||
| 961 | IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV6_UDP = 3, | ||
| 962 | IONIC_TFP_CSUM_PROF_IPV4_UDP_VXLAN_ETH_QTAG_IPV4_UDP = 4, | ||
| 963 | IONIC_TFP_CSUM_PROF_IPV4_UDP_VXLAN_ETH_QTAG_IPV6_UDP = 5, | ||
| 964 | IONIC_TFP_CSUM_PROF_QTAG_IPV4_UDP_VXLAN_ETH_QTAG_IPV4_UDP = 6, | ||
| 965 | IONIC_TFP_CSUM_PROF_QTAG_IPV4_UDP_VXLAN_ETH_QTAG_IPV6_UDP = 7, | ||
| 966 | IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_UDP_ESP_IPV4_UDP = 8, | ||
| 967 | IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_ESP_UDP = 9, | ||
| 968 | IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_UDP_ESP_UDP = 10, | ||
| 969 | IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV6_ESP_UDP = 11, | ||
| 970 | IONIC_TFP_CSUM_PROF_ETH_QTAG_IPV4_UDP_CSUM = 12, | ||
| 971 | }; | ||
| 972 | |||
| 973 | static inline bool ionic_v1_eqe_color(struct ionic_v1_eqe *eqe) | ||
| 974 | { | ||
| 975 | return eqe->evt & cpu_to_be32(IONIC_V1_EQE_COLOR); | ||
| 976 | } | ||
| 977 | |||
| 978 | static inline u32 ionic_v1_eqe_evt(struct ionic_v1_eqe *eqe) | ||
| 979 | { | ||
| 980 | return be32_to_cpu(eqe->evt); | ||
| 981 | } | ||
| 982 | |||
| 983 | static inline u8 ionic_v1_eqe_evt_type(u32 evt) | ||
| 984 | { | ||
| 985 | return (evt >> IONIC_V1_EQE_TYPE_SHIFT) & IONIC_V1_EQE_TYPE_MASK; | ||
| 986 | } | ||
| 987 | |||
| 988 | static inline u8 ionic_v1_eqe_evt_code(u32 evt) | ||
| 989 | { | ||
| 990 | return (evt >> IONIC_V1_EQE_CODE_SHIFT) & IONIC_V1_EQE_CODE_MASK; | ||
| 991 | } | ||
| 992 | |||
| 993 | static inline u32 ionic_v1_eqe_evt_qid(u32 evt) | ||
| 994 | { | ||
| 995 | return evt >> IONIC_V1_EQE_QID_SHIFT; | ||
| 996 | } | ||
| 997 | |||
| 998 | enum ionic_v1_stat_bits { | ||
| 999 | IONIC_V1_STAT_TYPE_SHIFT = 28, | ||
| 1000 | IONIC_V1_STAT_TYPE_NONE = 0, | ||
| 1001 | IONIC_V1_STAT_TYPE_8 = 1, | ||
| 1002 | IONIC_V1_STAT_TYPE_LE16 = 2, | ||
| 1003 | IONIC_V1_STAT_TYPE_LE32 = 3, | ||
| 1004 | IONIC_V1_STAT_TYPE_LE64 = 4, | ||
| 1005 | IONIC_V1_STAT_TYPE_BE16 = 5, | ||
| 1006 | IONIC_V1_STAT_TYPE_BE32 = 6, | ||
| 1007 | IONIC_V1_STAT_TYPE_BE64 = 7, | ||
| 1008 | IONIC_V1_STAT_OFF_MASK = BIT(IONIC_V1_STAT_TYPE_SHIFT) - 1, | ||
| 1009 | }; | ||
| 1010 | |||
| 1011 | struct ionic_v1_stat { | ||
| 1012 | union { | ||
| 1013 | __be32 be_type_off; | ||
| 1014 | u32 type_off; | ||
| 1015 | }; | ||
| 1016 | char name[28]; | ||
| 1017 | }; | ||
| 1018 | |||
| 1019 | static inline int ionic_v1_stat_type(struct ionic_v1_stat *hdr) | ||
| 1020 | { | ||
| 1021 | return hdr->type_off >> IONIC_V1_STAT_TYPE_SHIFT; | ||
| 1022 | } | ||
| 1023 | |||
| 1024 | static inline unsigned int ionic_v1_stat_off(struct ionic_v1_stat *hdr) | ||
| 1025 | { | ||
| 1026 | return hdr->type_off & IONIC_V1_STAT_OFF_MASK; | ||
| 1027 | } | ||
| 1028 | |||
| 1029 | #endif /* _IONIC_FW_H_ */ | ||
| @@ -0,0 +1,484 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #include <linux/dma-mapping.h> | ||
| 5 | |||
| 6 | #include "ionic_fw.h" | ||
| 7 | #include "ionic_ibdev.h" | ||
| 8 | |||
| 9 | static int ionic_v1_stat_normalize(struct ionic_v1_stat *hw_stats, | ||
| 10 | int hw_stats_count) | ||
| 11 | { | ||
| 12 | int hw_stat_i; | ||
| 13 | |||
| 14 | for (hw_stat_i = 0; hw_stat_i < hw_stats_count; ++hw_stat_i) { | ||
| 15 | struct ionic_v1_stat *stat = &hw_stats[hw_stat_i]; | ||
| 16 | |||
| 17 | stat->type_off = be32_to_cpu(stat->be_type_off); | ||
| 18 | stat->name[sizeof(stat->name) - 1] = 0; | ||
| 19 | if (ionic_v1_stat_type(stat) == IONIC_V1_STAT_TYPE_NONE) | ||
| 20 | break; | ||
| 21 | } | ||
| 22 | |||
| 23 | return hw_stat_i; | ||
| 24 | } | ||
| 25 | |||
| 26 | static void ionic_fill_stats_desc(struct rdma_stat_desc *hw_stats_hdrs, | ||
| 27 | struct ionic_v1_stat *hw_stats, | ||
| 28 | int hw_stats_count) | ||
| 29 | { | ||
| 30 | int hw_stat_i; | ||
| 31 | |||
| 32 | for (hw_stat_i = 0; hw_stat_i < hw_stats_count; ++hw_stat_i) { | ||
| 33 | struct ionic_v1_stat *stat = &hw_stats[hw_stat_i]; | ||
| 34 | |||
| 35 | hw_stats_hdrs[hw_stat_i].name = stat->name; | ||
| 36 | } | ||
| 37 | } | ||
| 38 | |||
| 39 | static u64 ionic_v1_stat_val(struct ionic_v1_stat *stat, | ||
| 40 | void *vals_buf, size_t vals_len) | ||
| 41 | { | ||
| 42 | unsigned int off = ionic_v1_stat_off(stat); | ||
| 43 | int type = ionic_v1_stat_type(stat); | ||
| 44 | |||
| 45 | #define __ionic_v1_stat_validate(__type) \ | ||
| 46 | ((off + sizeof(__type) <= vals_len) && \ | ||
| 47 | (IS_ALIGNED(off, sizeof(__type)))) | ||
| 48 | |||
| 49 | switch (type) { | ||
| 50 | case IONIC_V1_STAT_TYPE_8: | ||
| 51 | if (__ionic_v1_stat_validate(u8)) | ||
| 52 | return *(u8 *)(vals_buf + off); | ||
| 53 | break; | ||
| 54 | case IONIC_V1_STAT_TYPE_LE16: | ||
| 55 | if (__ionic_v1_stat_validate(__le16)) | ||
| 56 | return le16_to_cpu(*(__le16 *)(vals_buf + off)); | ||
| 57 | break; | ||
| 58 | case IONIC_V1_STAT_TYPE_LE32: | ||
| 59 | if (__ionic_v1_stat_validate(__le32)) | ||
| 60 | return le32_to_cpu(*(__le32 *)(vals_buf + off)); | ||
| 61 | break; | ||
| 62 | case IONIC_V1_STAT_TYPE_LE64: | ||
| 63 | if (__ionic_v1_stat_validate(__le64)) | ||
| 64 | return le64_to_cpu(*(__le64 *)(vals_buf + off)); | ||
| 65 | break; | ||
| 66 | case IONIC_V1_STAT_TYPE_BE16: | ||
| 67 | if (__ionic_v1_stat_validate(__be16)) | ||
| 68 | return be16_to_cpu(*(__be16 *)(vals_buf + off)); | ||
| 69 | break; | ||
| 70 | case IONIC_V1_STAT_TYPE_BE32: | ||
| 71 | if (__ionic_v1_stat_validate(__be32)) | ||
| 72 | return be32_to_cpu(*(__be32 *)(vals_buf + off)); | ||
| 73 | break; | ||
| 74 | case IONIC_V1_STAT_TYPE_BE64: | ||
| 75 | if (__ionic_v1_stat_validate(__be64)) | ||
| 76 | return be64_to_cpu(*(__be64 *)(vals_buf + off)); | ||
| 77 | break; | ||
| 78 | } | ||
| 79 | |||
| 80 | return ~0ull; | ||
| 81 | #undef __ionic_v1_stat_validate | ||
| 82 | } | ||
| 83 | |||
| 84 | static int ionic_hw_stats_cmd(struct ionic_ibdev *dev, | ||
| 85 | dma_addr_t dma, size_t len, int qid, int op) | ||
| 86 | { | ||
| 87 | struct ionic_admin_wr wr = { | ||
| 88 | .work = COMPLETION_INITIALIZER_ONSTACK(wr.work), | ||
| 89 | .wqe = { | ||
| 90 | .op = op, | ||
| 91 | .len = cpu_to_le16(IONIC_ADMIN_STATS_HDRS_IN_V1_LEN), | ||
| 92 | .cmd.stats = { | ||
| 93 | .dma_addr = cpu_to_le64(dma), | ||
| 94 | .length = cpu_to_le32(len), | ||
| 95 | .id_ver = cpu_to_le32(qid), | ||
| 96 | }, | ||
| 97 | } | ||
| 98 | }; | ||
| 99 | |||
| 100 | if (dev->lif_cfg.admin_opcodes <= op) | ||
| 101 | return -EBADRQC; | ||
| 102 | |||
| 103 | ionic_admin_post(dev, &wr); | ||
| 104 | |||
| 105 | return ionic_admin_wait(dev, &wr, IONIC_ADMIN_F_INTERRUPT); | ||
| 106 | } | ||
| 107 | |||
| 108 | static int ionic_init_hw_stats(struct ionic_ibdev *dev) | ||
| 109 | { | ||
| 110 | dma_addr_t hw_stats_dma; | ||
| 111 | int rc, hw_stats_count; | ||
| 112 | |||
| 113 | if (dev->hw_stats_hdrs) | ||
| 114 | return 0; | ||
| 115 | |||
| 116 | dev->hw_stats_count = 0; | ||
| 117 | |||
| 118 | /* buffer for current values from the device */ | ||
| 119 | dev->hw_stats_buf = kzalloc(PAGE_SIZE, GFP_KERNEL); | ||
| 120 | if (!dev->hw_stats_buf) { | ||
| 121 | rc = -ENOMEM; | ||
| 122 | goto err_buf; | ||
| 123 | } | ||
| 124 | |||
| 125 | /* buffer for names, sizes, offsets of values */ | ||
| 126 | dev->hw_stats = kzalloc(PAGE_SIZE, GFP_KERNEL); | ||
| 127 | if (!dev->hw_stats) { | ||
| 128 | rc = -ENOMEM; | ||
| 129 | goto err_hw_stats; | ||
| 130 | } | ||
| 131 | |||
| 132 | /* request the names, sizes, offsets */ | ||
| 133 | hw_stats_dma = dma_map_single(dev->lif_cfg.hwdev, dev->hw_stats, | ||
| 134 | PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 135 | rc = dma_mapping_error(dev->lif_cfg.hwdev, hw_stats_dma); | ||
| 136 | if (rc) | ||
| 137 | goto err_dma; | ||
| 138 | |||
| 139 | rc = ionic_hw_stats_cmd(dev, hw_stats_dma, PAGE_SIZE, 0, | ||
| 140 | IONIC_V1_ADMIN_STATS_HDRS); | ||
| 141 | if (rc) | ||
| 142 | goto err_cmd; | ||
| 143 | |||
| 144 | dma_unmap_single(dev->lif_cfg.hwdev, hw_stats_dma, PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 145 | |||
| 146 | /* normalize and count the number of hw_stats */ | ||
| 147 | hw_stats_count = | ||
| 148 | ionic_v1_stat_normalize(dev->hw_stats, | ||
| 149 | PAGE_SIZE / sizeof(*dev->hw_stats)); | ||
| 150 | if (!hw_stats_count) { | ||
| 151 | rc = -ENODATA; | ||
| 152 | goto err_dma; | ||
| 153 | } | ||
| 154 | |||
| 155 | dev->hw_stats_count = hw_stats_count; | ||
| 156 | |||
| 157 | /* alloc and init array of names, for alloc_hw_stats */ | ||
| 158 | dev->hw_stats_hdrs = kcalloc(hw_stats_count, | ||
| 159 | sizeof(*dev->hw_stats_hdrs), | ||
| 160 | GFP_KERNEL); | ||
| 161 | if (!dev->hw_stats_hdrs) { | ||
| 162 | rc = -ENOMEM; | ||
| 163 | goto err_dma; | ||
| 164 | } | ||
| 165 | |||
| 166 | ionic_fill_stats_desc(dev->hw_stats_hdrs, dev->hw_stats, | ||
| 167 | hw_stats_count); | ||
| 168 | |||
| 169 | return 0; | ||
| 170 | |||
| 171 | err_cmd: | ||
| 172 | dma_unmap_single(dev->lif_cfg.hwdev, hw_stats_dma, PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 173 | err_dma: | ||
| 174 | kfree(dev->hw_stats); | ||
| 175 | err_hw_stats: | ||
| 176 | kfree(dev->hw_stats_buf); | ||
| 177 | err_buf: | ||
| 178 | dev->hw_stats_count = 0; | ||
| 179 | dev->hw_stats = NULL; | ||
| 180 | dev->hw_stats_buf = NULL; | ||
| 181 | dev->hw_stats_hdrs = NULL; | ||
| 182 | return rc; | ||
| 183 | } | ||
| 184 | |||
| 185 | static struct rdma_hw_stats *ionic_alloc_hw_stats(struct ib_device *ibdev, | ||
| 186 | u32 port) | ||
| 187 | { | ||
| 188 | struct ionic_ibdev *dev = to_ionic_ibdev(ibdev); | ||
| 189 | |||
| 190 | if (port != 1) | ||
| 191 | return NULL; | ||
| 192 | |||
| 193 | return rdma_alloc_hw_stats_struct(dev->hw_stats_hdrs, | ||
| 194 | dev->hw_stats_count, | ||
| 195 | RDMA_HW_STATS_DEFAULT_LIFESPAN); | ||
| 196 | } | ||
| 197 | |||
| 198 | static int ionic_get_hw_stats(struct ib_device *ibdev, | ||
| 199 | struct rdma_hw_stats *hw_stats, | ||
| 200 | u32 port, int index) | ||
| 201 | { | ||
| 202 | struct ionic_ibdev *dev = to_ionic_ibdev(ibdev); | ||
| 203 | dma_addr_t hw_stats_dma; | ||
| 204 | int rc, hw_stat_i; | ||
| 205 | |||
| 206 | if (port != 1) | ||
| 207 | return -EINVAL; | ||
| 208 | |||
| 209 | hw_stats_dma = dma_map_single(dev->lif_cfg.hwdev, dev->hw_stats_buf, | ||
| 210 | PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 211 | rc = dma_mapping_error(dev->lif_cfg.hwdev, hw_stats_dma); | ||
| 212 | if (rc) | ||
| 213 | goto err_dma; | ||
| 214 | |||
| 215 | rc = ionic_hw_stats_cmd(dev, hw_stats_dma, PAGE_SIZE, | ||
| 216 | 0, IONIC_V1_ADMIN_STATS_VALS); | ||
| 217 | if (rc) | ||
| 218 | goto err_cmd; | ||
| 219 | |||
| 220 | dma_unmap_single(dev->lif_cfg.hwdev, hw_stats_dma, | ||
| 221 | PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 222 | |||
| 223 | for (hw_stat_i = 0; hw_stat_i < dev->hw_stats_count; ++hw_stat_i) | ||
| 224 | hw_stats->value[hw_stat_i] = | ||
| 225 | ionic_v1_stat_val(&dev->hw_stats[hw_stat_i], | ||
| 226 | dev->hw_stats_buf, PAGE_SIZE); | ||
| 227 | |||
| 228 | return hw_stat_i; | ||
| 229 | |||
| 230 | err_cmd: | ||
| 231 | dma_unmap_single(dev->lif_cfg.hwdev, hw_stats_dma, | ||
| 232 | PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 233 | err_dma: | ||
| 234 | return rc; | ||
| 235 | } | ||
| 236 | |||
| 237 | static struct rdma_hw_stats * | ||
| 238 | ionic_counter_alloc_stats(struct rdma_counter *counter) | ||
| 239 | { | ||
| 240 | struct ionic_ibdev *dev = to_ionic_ibdev(counter->device); | ||
| 241 | struct ionic_counter *cntr; | ||
| 242 | int err; | ||
| 243 | |||
| 244 | cntr = kzalloc(sizeof(*cntr), GFP_KERNEL); | ||
| 245 | if (!cntr) | ||
| 246 | return NULL; | ||
| 247 | |||
| 248 | /* buffer for current values from the device */ | ||
| 249 | cntr->vals = kzalloc(PAGE_SIZE, GFP_KERNEL); | ||
| 250 | if (!cntr->vals) | ||
| 251 | goto err_vals; | ||
| 252 | |||
| 253 | err = xa_alloc(&dev->counter_stats->xa_counters, &counter->id, | ||
| 254 | cntr, | ||
| 255 | XA_LIMIT(0, IONIC_MAX_QPID), | ||
| 256 | GFP_KERNEL); | ||
| 257 | if (err) | ||
| 258 | goto err_xa; | ||
| 259 | |||
| 260 | INIT_LIST_HEAD(&cntr->qp_list); | ||
| 261 | |||
| 262 | return rdma_alloc_hw_stats_struct(dev->counter_stats->stats_hdrs, | ||
| 263 | dev->counter_stats->queue_stats_count, | ||
| 264 | RDMA_HW_STATS_DEFAULT_LIFESPAN); | ||
| 265 | err_xa: | ||
| 266 | kfree(cntr->vals); | ||
| 267 | err_vals: | ||
| 268 | kfree(cntr); | ||
| 269 | |||
| 270 | return NULL; | ||
| 271 | } | ||
| 272 | |||
| 273 | static int ionic_counter_dealloc(struct rdma_counter *counter) | ||
| 274 | { | ||
| 275 | struct ionic_ibdev *dev = to_ionic_ibdev(counter->device); | ||
| 276 | struct ionic_counter *cntr; | ||
| 277 | |||
| 278 | cntr = xa_erase(&dev->counter_stats->xa_counters, counter->id); | ||
| 279 | if (!cntr) | ||
| 280 | return -EINVAL; | ||
| 281 | |||
| 282 | kfree(cntr->vals); | ||
| 283 | kfree(cntr); | ||
| 284 | |||
| 285 | return 0; | ||
| 286 | } | ||
| 287 | |||
| 288 | static int ionic_counter_bind_qp(struct rdma_counter *counter, | ||
| 289 | struct ib_qp *ibqp, | ||
| 290 | u32 port) | ||
| 291 | { | ||
| 292 | struct ionic_ibdev *dev = to_ionic_ibdev(counter->device); | ||
| 293 | struct ionic_qp *qp = to_ionic_qp(ibqp); | ||
| 294 | struct ionic_counter *cntr; | ||
| 295 | |||
| 296 | cntr = xa_load(&dev->counter_stats->xa_counters, counter->id); | ||
| 297 | if (!cntr) | ||
| 298 | return -EINVAL; | ||
| 299 | |||
| 300 | list_add_tail(&qp->qp_list_counter, &cntr->qp_list); | ||
| 301 | ibqp->counter = counter; | ||
| 302 | |||
| 303 | return 0; | ||
| 304 | } | ||
| 305 | |||
| 306 | static int ionic_counter_unbind_qp(struct ib_qp *ibqp, u32 port) | ||
| 307 | { | ||
| 308 | struct ionic_qp *qp = to_ionic_qp(ibqp); | ||
| 309 | |||
| 310 | if (ibqp->counter) { | ||
| 311 | list_del(&qp->qp_list_counter); | ||
| 312 | ibqp->counter = NULL; | ||
| 313 | } | ||
| 314 | |||
| 315 | return 0; | ||
| 316 | } | ||
| 317 | |||
| 318 | static int ionic_get_qp_stats(struct ib_device *ibdev, | ||
| 319 | struct rdma_hw_stats *hw_stats, | ||
| 320 | u32 counter_id) | ||
| 321 | { | ||
| 322 | struct ionic_ibdev *dev = to_ionic_ibdev(ibdev); | ||
| 323 | struct ionic_counter_stats *cs; | ||
| 324 | struct ionic_counter *cntr; | ||
| 325 | dma_addr_t hw_stats_dma; | ||
| 326 | struct ionic_qp *qp; | ||
| 327 | int rc, stat_i = 0; | ||
| 328 | |||
| 329 | cs = dev->counter_stats; | ||
| 330 | cntr = xa_load(&cs->xa_counters, counter_id); | ||
| 331 | if (!cntr) | ||
| 332 | return -EINVAL; | ||
| 333 | |||
| 334 | hw_stats_dma = dma_map_single(dev->lif_cfg.hwdev, cntr->vals, | ||
| 335 | PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 336 | rc = dma_mapping_error(dev->lif_cfg.hwdev, hw_stats_dma); | ||
| 337 | if (rc) | ||
| 338 | return rc; | ||
| 339 | |||
| 340 | memset(hw_stats->value, 0, sizeof(u64) * hw_stats->num_counters); | ||
| 341 | |||
| 342 | list_for_each_entry(qp, &cntr->qp_list, qp_list_counter) { | ||
| 343 | rc = ionic_hw_stats_cmd(dev, hw_stats_dma, PAGE_SIZE, | ||
| 344 | qp->qpid, | ||
| 345 | IONIC_V1_ADMIN_QP_STATS_VALS); | ||
| 346 | if (rc) | ||
| 347 | goto err_cmd; | ||
| 348 | |||
| 349 | for (stat_i = 0; stat_i < cs->queue_stats_count; ++stat_i) | ||
| 350 | hw_stats->value[stat_i] += | ||
| 351 | ionic_v1_stat_val(&cs->hdr[stat_i], | ||
| 352 | cntr->vals, | ||
| 353 | PAGE_SIZE); | ||
| 354 | } | ||
| 355 | |||
| 356 | dma_unmap_single(dev->lif_cfg.hwdev, hw_stats_dma, PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 357 | return stat_i; | ||
| 358 | |||
| 359 | err_cmd: | ||
| 360 | dma_unmap_single(dev->lif_cfg.hwdev, hw_stats_dma, PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 361 | |||
| 362 | return rc; | ||
| 363 | } | ||
| 364 | |||
| 365 | static int ionic_counter_update_stats(struct rdma_counter *counter) | ||
| 366 | { | ||
| 367 | return ionic_get_qp_stats(counter->device, counter->stats, counter->id); | ||
| 368 | } | ||
| 369 | |||
| 370 | static int ionic_alloc_counters(struct ionic_ibdev *dev) | ||
| 371 | { | ||
| 372 | struct ionic_counter_stats *cs = dev->counter_stats; | ||
| 373 | int rc, hw_stats_count; | ||
| 374 | dma_addr_t hdr_dma; | ||
| 375 | |||
| 376 | /* buffer for names, sizes, offsets of values */ | ||
| 377 | cs->hdr = kzalloc(PAGE_SIZE, GFP_KERNEL); | ||
| 378 | if (!cs->hdr) | ||
| 379 | return -ENOMEM; | ||
| 380 | |||
| 381 | hdr_dma = dma_map_single(dev->lif_cfg.hwdev, cs->hdr, | ||
| 382 | PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 383 | rc = dma_mapping_error(dev->lif_cfg.hwdev, hdr_dma); | ||
| 384 | if (rc) | ||
| 385 | goto err_dma; | ||
| 386 | |||
| 387 | rc = ionic_hw_stats_cmd(dev, hdr_dma, PAGE_SIZE, 0, | ||
| 388 | IONIC_V1_ADMIN_QP_STATS_HDRS); | ||
| 389 | if (rc) | ||
| 390 | goto err_cmd; | ||
| 391 | |||
| 392 | dma_unmap_single(dev->lif_cfg.hwdev, hdr_dma, PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 393 | |||
| 394 | /* normalize and count the number of hw_stats */ | ||
| 395 | hw_stats_count = ionic_v1_stat_normalize(cs->hdr, | ||
| 396 | PAGE_SIZE / sizeof(*cs->hdr)); | ||
| 397 | if (!hw_stats_count) { | ||
| 398 | rc = -ENODATA; | ||
| 399 | goto err_dma; | ||
| 400 | } | ||
| 401 | |||
| 402 | cs->queue_stats_count = hw_stats_count; | ||
| 403 | |||
| 404 | /* alloc and init array of names */ | ||
| 405 | cs->stats_hdrs = kcalloc(hw_stats_count, sizeof(*cs->stats_hdrs), | ||
| 406 | GFP_KERNEL); | ||
| 407 | if (!cs->stats_hdrs) { | ||
| 408 | rc = -ENOMEM; | ||
| 409 | goto err_dma; | ||
| 410 | } | ||
| 411 | |||
| 412 | ionic_fill_stats_desc(cs->stats_hdrs, cs->hdr, hw_stats_count); | ||
| 413 | |||
| 414 | return 0; | ||
| 415 | |||
| 416 | err_cmd: | ||
| 417 | dma_unmap_single(dev->lif_cfg.hwdev, hdr_dma, PAGE_SIZE, DMA_FROM_DEVICE); | ||
| 418 | err_dma: | ||
| 419 | kfree(cs->hdr); | ||
| 420 | |||
| 421 | return rc; | ||
| 422 | } | ||
| 423 | |||
| 424 | static const struct ib_device_ops ionic_hw_stats_ops = { | ||
| 425 | .driver_id = RDMA_DRIVER_IONIC, | ||
| 426 | .alloc_hw_port_stats = ionic_alloc_hw_stats, | ||
| 427 | .get_hw_stats = ionic_get_hw_stats, | ||
| 428 | }; | ||
| 429 | |||
| 430 | static const struct ib_device_ops ionic_counter_stats_ops = { | ||
| 431 | .counter_alloc_stats = ionic_counter_alloc_stats, | ||
| 432 | .counter_dealloc = ionic_counter_dealloc, | ||
| 433 | .counter_bind_qp = ionic_counter_bind_qp, | ||
| 434 | .counter_unbind_qp = ionic_counter_unbind_qp, | ||
| 435 | .counter_update_stats = ionic_counter_update_stats, | ||
| 436 | }; | ||
| 437 | |||
| 438 | void ionic_stats_init(struct ionic_ibdev *dev) | ||
| 439 | { | ||
| 440 | u16 stats_type = dev->lif_cfg.stats_type; | ||
| 441 | int rc; | ||
| 442 | |||
| 443 | if (stats_type & IONIC_LIF_RDMA_STAT_GLOBAL) { | ||
| 444 | rc = ionic_init_hw_stats(dev); | ||
| 445 | if (rc) | ||
| 446 | ibdev_dbg(&dev->ibdev, "Failed to init hw stats\n"); | ||
| 447 | else | ||
| 448 | ib_set_device_ops(&dev->ibdev, &ionic_hw_stats_ops); | ||
| 449 | } | ||
| 450 | |||
| 451 | if (stats_type & IONIC_LIF_RDMA_STAT_QP) { | ||
| 452 | dev->counter_stats = kzalloc(sizeof(*dev->counter_stats), | ||
| 453 | GFP_KERNEL); | ||
| 454 | if (!dev->counter_stats) | ||
| 455 | return; | ||
| 456 | |||
| 457 | rc = ionic_alloc_counters(dev); | ||
| 458 | if (rc) { | ||
| 459 | ibdev_dbg(&dev->ibdev, "Failed to init counter stats\n"); | ||
| 460 | kfree(dev->counter_stats); | ||
| 461 | dev->counter_stats = NULL; | ||
| 462 | return; | ||
| 463 | } | ||
| 464 | |||
| 465 | xa_init_flags(&dev->counter_stats->xa_counters, XA_FLAGS_ALLOC); | ||
| 466 | |||
| 467 | ib_set_device_ops(&dev->ibdev, &ionic_counter_stats_ops); | ||
| 468 | } | ||
| 469 | } | ||
| 470 | |||
| 471 | void ionic_stats_cleanup(struct ionic_ibdev *dev) | ||
| 472 | { | ||
| 473 | if (dev->counter_stats) { | ||
| 474 | xa_destroy(&dev->counter_stats->xa_counters); | ||
| 475 | kfree(dev->counter_stats->hdr); | ||
| 476 | kfree(dev->counter_stats->stats_hdrs); | ||
| 477 | kfree(dev->counter_stats); | ||
| 478 | dev->counter_stats = NULL; | ||
| 479 | } | ||
| 480 | |||
| 481 | kfree(dev->hw_stats); | ||
| 482 | kfree(dev->hw_stats_buf); | ||
| 483 | kfree(dev->hw_stats_hdrs); | ||
| 484 | } | ||
| @@ -0,0 +1,440 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #include <linux/module.h> | ||
| 5 | #include <linux/printk.h> | ||
| 6 | #include <linux/pci.h> | ||
| 7 | #include <linux/irq.h> | ||
| 8 | #include <net/addrconf.h> | ||
| 9 | #include <rdma/ib_addr.h> | ||
| 10 | #include <rdma/ib_mad.h> | ||
| 11 | |||
| 12 | #include "ionic_ibdev.h" | ||
| 13 | |||
| 14 | #define DRIVER_DESCRIPTION "AMD Pensando RoCE HCA driver" | ||
| 15 | #define DEVICE_DESCRIPTION "AMD Pensando RoCE HCA" | ||
| 16 | |||
| 17 | MODULE_AUTHOR("Allen Hubbe <allen.hubbe@amd.com>"); | ||
| 18 | MODULE_DESCRIPTION(DRIVER_DESCRIPTION); | ||
| 19 | MODULE_LICENSE("GPL"); | ||
| 20 | MODULE_IMPORT_NS("NET_IONIC"); | ||
| 21 | |||
| 22 | static int ionic_query_device(struct ib_device *ibdev, | ||
| 23 | struct ib_device_attr *attr, | ||
| 24 | struct ib_udata *udata) | ||
| 25 | { | ||
| 26 | struct ionic_ibdev *dev = to_ionic_ibdev(ibdev); | ||
| 27 | struct net_device *ndev; | ||
| 28 | |||
| 29 | ndev = ib_device_get_netdev(ibdev, 1); | ||
| 30 | addrconf_ifid_eui48((u8 *)&attr->sys_image_guid, ndev); | ||
| 31 | dev_put(ndev); | ||
| 32 | attr->max_mr_size = dev->lif_cfg.npts_per_lif * PAGE_SIZE / 2; | ||
| 33 | attr->page_size_cap = dev->lif_cfg.page_size_supported; | ||
| 34 | |||
| 35 | attr->vendor_id = to_pci_dev(dev->lif_cfg.hwdev)->vendor; | ||
| 36 | attr->vendor_part_id = to_pci_dev(dev->lif_cfg.hwdev)->device; | ||
| 37 | |||
| 38 | attr->hw_ver = ionic_lif_asic_rev(dev->lif_cfg.lif); | ||
| 39 | attr->fw_ver = 0; | ||
| 40 | attr->max_qp = dev->lif_cfg.qp_count; | ||
| 41 | attr->max_qp_wr = IONIC_MAX_DEPTH; | ||
| 42 | attr->device_cap_flags = | ||
| 43 | IB_DEVICE_MEM_WINDOW | | ||
| 44 | IB_DEVICE_MEM_MGT_EXTENSIONS | | ||
| 45 | IB_DEVICE_MEM_WINDOW_TYPE_2B | | ||
| 46 | 0; | ||
| 47 | attr->max_send_sge = | ||
| 48 | min(ionic_v1_send_wqe_max_sge(dev->lif_cfg.max_stride, 0, false), | ||
| 49 | IONIC_SPEC_HIGH); | ||
| 50 | attr->max_recv_sge = | ||
| 51 | min(ionic_v1_recv_wqe_max_sge(dev->lif_cfg.max_stride, 0, false), | ||
| 52 | IONIC_SPEC_HIGH); | ||
| 53 | attr->max_sge_rd = attr->max_send_sge; | ||
| 54 | attr->max_cq = dev->lif_cfg.cq_count / dev->lif_cfg.udma_count; | ||
| 55 | attr->max_cqe = IONIC_MAX_CQ_DEPTH - IONIC_CQ_GRACE; | ||
| 56 | attr->max_mr = dev->lif_cfg.nmrs_per_lif; | ||
| 57 | attr->max_pd = IONIC_MAX_PD; | ||
| 58 | attr->max_qp_rd_atom = IONIC_MAX_RD_ATOM; | ||
| 59 | attr->max_ee_rd_atom = 0; | ||
| 60 | attr->max_res_rd_atom = IONIC_MAX_RD_ATOM; | ||
| 61 | attr->max_qp_init_rd_atom = IONIC_MAX_RD_ATOM; | ||
| 62 | attr->max_ee_init_rd_atom = 0; | ||
| 63 | attr->atomic_cap = IB_ATOMIC_GLOB; | ||
| 64 | attr->masked_atomic_cap = IB_ATOMIC_GLOB; | ||
| 65 | attr->max_mw = dev->lif_cfg.nmrs_per_lif; | ||
| 66 | attr->max_mcast_grp = 0; | ||
| 67 | attr->max_mcast_qp_attach = 0; | ||
| 68 | attr->max_ah = dev->lif_cfg.nahs_per_lif; | ||
| 69 | attr->max_fast_reg_page_list_len = dev->lif_cfg.npts_per_lif / 2; | ||
| 70 | attr->max_pkeys = IONIC_PKEY_TBL_LEN; | ||
| 71 | |||
| 72 | return 0; | ||
| 73 | } | ||
| 74 | |||
| 75 | static int ionic_query_port(struct ib_device *ibdev, u32 port, | ||
| 76 | struct ib_port_attr *attr) | ||
| 77 | { | ||
| 78 | struct net_device *ndev; | ||
| 79 | |||
| 80 | if (port != 1) | ||
| 81 | return -EINVAL; | ||
| 82 | |||
| 83 | ndev = ib_device_get_netdev(ibdev, port); | ||
| 84 | |||
| 85 | if (netif_running(ndev) && netif_carrier_ok(ndev)) { | ||
| 86 | attr->state = IB_PORT_ACTIVE; | ||
| 87 | attr->phys_state = IB_PORT_PHYS_STATE_LINK_UP; | ||
| 88 | } else if (netif_running(ndev)) { | ||
| 89 | attr->state = IB_PORT_DOWN; | ||
| 90 | attr->phys_state = IB_PORT_PHYS_STATE_POLLING; | ||
| 91 | } else { | ||
| 92 | attr->state = IB_PORT_DOWN; | ||
| 93 | attr->phys_state = IB_PORT_PHYS_STATE_DISABLED; | ||
| 94 | } | ||
| 95 | |||
| 96 | attr->max_mtu = iboe_get_mtu(ndev->max_mtu); | ||
| 97 | attr->active_mtu = min(attr->max_mtu, iboe_get_mtu(ndev->mtu)); | ||
| 98 | attr->gid_tbl_len = IONIC_GID_TBL_LEN; | ||
| 99 | attr->ip_gids = true; | ||
| 100 | attr->port_cap_flags = 0; | ||
| 101 | attr->max_msg_sz = 0x80000000; | ||
| 102 | attr->pkey_tbl_len = IONIC_PKEY_TBL_LEN; | ||
| 103 | attr->max_vl_num = 1; | ||
| 104 | attr->subnet_prefix = 0xfe80000000000000ull; | ||
| 105 | |||
| 106 | dev_put(ndev); | ||
| 107 | |||
| 108 | return ib_get_eth_speed(ibdev, port, | ||
| 109 | &attr->active_speed, | ||
| 110 | &attr->active_width); | ||
| 111 | } | ||
| 112 | |||
| 113 | static enum rdma_link_layer ionic_get_link_layer(struct ib_device *ibdev, | ||
| 114 | u32 port) | ||
| 115 | { | ||
| 116 | return IB_LINK_LAYER_ETHERNET; | ||
| 117 | } | ||
| 118 | |||
| 119 | static int ionic_query_pkey(struct ib_device *ibdev, u32 port, u16 index, | ||
| 120 | u16 *pkey) | ||
| 121 | { | ||
| 122 | if (port != 1) | ||
| 123 | return -EINVAL; | ||
| 124 | |||
| 125 | if (index != 0) | ||
| 126 | return -EINVAL; | ||
| 127 | |||
| 128 | *pkey = IB_DEFAULT_PKEY_FULL; | ||
| 129 | |||
| 130 | return 0; | ||
| 131 | } | ||
| 132 | |||
| 133 | static int ionic_modify_device(struct ib_device *ibdev, int mask, | ||
| 134 | struct ib_device_modify *attr) | ||
| 135 | { | ||
| 136 | struct ionic_ibdev *dev = to_ionic_ibdev(ibdev); | ||
| 137 | |||
| 138 | if (mask & ~IB_DEVICE_MODIFY_NODE_DESC) | ||
| 139 | return -EOPNOTSUPP; | ||
| 140 | |||
| 141 | if (mask & IB_DEVICE_MODIFY_NODE_DESC) | ||
| 142 | memcpy(dev->ibdev.node_desc, attr->node_desc, | ||
| 143 | IB_DEVICE_NODE_DESC_MAX); | ||
| 144 | |||
| 145 | return 0; | ||
| 146 | } | ||
| 147 | |||
| 148 | static int ionic_get_port_immutable(struct ib_device *ibdev, u32 port, | ||
| 149 | struct ib_port_immutable *attr) | ||
| 150 | { | ||
| 151 | if (port != 1) | ||
| 152 | return -EINVAL; | ||
| 153 | |||
| 154 | attr->core_cap_flags = RDMA_CORE_PORT_IBA_ROCE_UDP_ENCAP; | ||
| 155 | |||
| 156 | attr->pkey_tbl_len = IONIC_PKEY_TBL_LEN; | ||
| 157 | attr->gid_tbl_len = IONIC_GID_TBL_LEN; | ||
| 158 | attr->max_mad_size = IB_MGMT_MAD_SIZE; | ||
| 159 | |||
| 160 | return 0; | ||
| 161 | } | ||
| 162 | |||
| 163 | static void ionic_get_dev_fw_str(struct ib_device *ibdev, char *str) | ||
| 164 | { | ||
| 165 | struct ionic_ibdev *dev = to_ionic_ibdev(ibdev); | ||
| 166 | |||
| 167 | ionic_lif_fw_version(dev->lif_cfg.lif, str, IB_FW_VERSION_NAME_MAX); | ||
| 168 | } | ||
| 169 | |||
| 170 | static ssize_t hw_rev_show(struct device *device, struct device_attribute *attr, | ||
| 171 | char *buf) | ||
| 172 | { | ||
| 173 | struct ionic_ibdev *dev = | ||
| 174 | rdma_device_to_drv_device(device, struct ionic_ibdev, ibdev); | ||
| 175 | |||
| 176 | return sysfs_emit(buf, "0x%x\n", ionic_lif_asic_rev(dev->lif_cfg.lif)); | ||
| 177 | } | ||
| 178 | static DEVICE_ATTR_RO(hw_rev); | ||
| 179 | |||
| 180 | static ssize_t hca_type_show(struct device *device, | ||
| 181 | struct device_attribute *attr, char *buf) | ||
| 182 | { | ||
| 183 | struct ionic_ibdev *dev = | ||
| 184 | rdma_device_to_drv_device(device, struct ionic_ibdev, ibdev); | ||
| 185 | |||
| 186 | return sysfs_emit(buf, "%s\n", dev->ibdev.node_desc); | ||
| 187 | } | ||
| 188 | static DEVICE_ATTR_RO(hca_type); | ||
| 189 | |||
| 190 | static struct attribute *ionic_rdma_attributes[] = { | ||
| 191 | &dev_attr_hw_rev.attr, | ||
| 192 | &dev_attr_hca_type.attr, | ||
| 193 | NULL | ||
| 194 | }; | ||
| 195 | |||
| 196 | static const struct attribute_group ionic_rdma_attr_group = { | ||
| 197 | .attrs = ionic_rdma_attributes, | ||
| 198 | }; | ||
| 199 | |||
| 200 | static void ionic_disassociate_ucontext(struct ib_ucontext *ibcontext) | ||
| 201 | { | ||
| 202 | /* | ||
| 203 | * Dummy define disassociate_ucontext so that it does not | ||
| 204 | * wait for user context before cleaning up hw resources. | ||
| 205 | */ | ||
| 206 | } | ||
| 207 | |||
| 208 | static const struct ib_device_ops ionic_dev_ops = { | ||
| 209 | .owner = THIS_MODULE, | ||
| 210 | .driver_id = RDMA_DRIVER_IONIC, | ||
| 211 | .uverbs_abi_ver = IONIC_ABI_VERSION, | ||
| 212 | |||
| 213 | .alloc_ucontext = ionic_alloc_ucontext, | ||
| 214 | .dealloc_ucontext = ionic_dealloc_ucontext, | ||
| 215 | .mmap = ionic_mmap, | ||
| 216 | .mmap_free = ionic_mmap_free, | ||
| 217 | .alloc_pd = ionic_alloc_pd, | ||
| 218 | .dealloc_pd = ionic_dealloc_pd, | ||
| 219 | .create_ah = ionic_create_ah, | ||
| 220 | .query_ah = ionic_query_ah, | ||
| 221 | .destroy_ah = ionic_destroy_ah, | ||
| 222 | .create_user_ah = ionic_create_ah, | ||
| 223 | .get_dma_mr = ionic_get_dma_mr, | ||
| 224 | .reg_user_mr = ionic_reg_user_mr, | ||
| 225 | .reg_user_mr_dmabuf = ionic_reg_user_mr_dmabuf, | ||
| 226 | .dereg_mr = ionic_dereg_mr, | ||
| 227 | .alloc_mr = ionic_alloc_mr, | ||
| 228 | .map_mr_sg = ionic_map_mr_sg, | ||
| 229 | .alloc_mw = ionic_alloc_mw, | ||
| 230 | .dealloc_mw = ionic_dealloc_mw, | ||
| 231 | .create_cq = ionic_create_cq, | ||
| 232 | .destroy_cq = ionic_destroy_cq, | ||
| 233 | .create_qp = ionic_create_qp, | ||
| 234 | .modify_qp = ionic_modify_qp, | ||
| 235 | .query_qp = ionic_query_qp, | ||
| 236 | .destroy_qp = ionic_destroy_qp, | ||
| 237 | |||
| 238 | .post_send = ionic_post_send, | ||
| 239 | .post_recv = ionic_post_recv, | ||
| 240 | .poll_cq = ionic_poll_cq, | ||
| 241 | .req_notify_cq = ionic_req_notify_cq, | ||
| 242 | |||
| 243 | .query_device = ionic_query_device, | ||
| 244 | .query_port = ionic_query_port, | ||
| 245 | .get_link_layer = ionic_get_link_layer, | ||
| 246 | .query_pkey = ionic_query_pkey, | ||
| 247 | .modify_device = ionic_modify_device, | ||
| 248 | .get_port_immutable = ionic_get_port_immutable, | ||
| 249 | .get_dev_fw_str = ionic_get_dev_fw_str, | ||
| 250 | .device_group = &ionic_rdma_attr_group, | ||
| 251 | .disassociate_ucontext = ionic_disassociate_ucontext, | ||
| 252 | |||
| 253 | INIT_RDMA_OBJ_SIZE(ib_ucontext, ionic_ctx, ibctx), | ||
| 254 | INIT_RDMA_OBJ_SIZE(ib_pd, ionic_pd, ibpd), | ||
| 255 | INIT_RDMA_OBJ_SIZE(ib_ah, ionic_ah, ibah), | ||
| 256 | INIT_RDMA_OBJ_SIZE(ib_cq, ionic_vcq, ibcq), | ||
| 257 | INIT_RDMA_OBJ_SIZE(ib_qp, ionic_qp, ibqp), | ||
| 258 | INIT_RDMA_OBJ_SIZE(ib_mw, ionic_mr, ibmw), | ||
| 259 | }; | ||
| 260 | |||
| 261 | static void ionic_init_resids(struct ionic_ibdev *dev) | ||
| 262 | { | ||
| 263 | ionic_resid_init(&dev->inuse_cqid, dev->lif_cfg.cq_count); | ||
| 264 | dev->half_cqid_udma_shift = | ||
| 265 | order_base_2(dev->lif_cfg.cq_count / dev->lif_cfg.udma_count); | ||
| 266 | ionic_resid_init(&dev->inuse_pdid, IONIC_MAX_PD); | ||
| 267 | ionic_resid_init(&dev->inuse_ahid, dev->lif_cfg.nahs_per_lif); | ||
| 268 | ionic_resid_init(&dev->inuse_mrid, dev->lif_cfg.nmrs_per_lif); | ||
| 269 | /* skip reserved lkey */ | ||
| 270 | dev->next_mrkey = 1; | ||
| 271 | ionic_resid_init(&dev->inuse_qpid, dev->lif_cfg.qp_count); | ||
| 272 | /* skip reserved SMI and GSI qpids */ | ||
| 273 | dev->half_qpid_udma_shift = | ||
| 274 | order_base_2(dev->lif_cfg.qp_count / dev->lif_cfg.udma_count); | ||
| 275 | ionic_resid_init(&dev->inuse_dbid, dev->lif_cfg.dbid_count); | ||
| 276 | } | ||
| 277 | |||
| 278 | static void ionic_destroy_resids(struct ionic_ibdev *dev) | ||
| 279 | { | ||
| 280 | ionic_resid_destroy(&dev->inuse_cqid); | ||
| 281 | ionic_resid_destroy(&dev->inuse_pdid); | ||
| 282 | ionic_resid_destroy(&dev->inuse_ahid); | ||
| 283 | ionic_resid_destroy(&dev->inuse_mrid); | ||
| 284 | ionic_resid_destroy(&dev->inuse_qpid); | ||
| 285 | ionic_resid_destroy(&dev->inuse_dbid); | ||
| 286 | } | ||
| 287 | |||
| 288 | static void ionic_destroy_ibdev(struct ionic_ibdev *dev) | ||
| 289 | { | ||
| 290 | ionic_kill_rdma_admin(dev, false); | ||
| 291 | ib_unregister_device(&dev->ibdev); | ||
| 292 | ionic_stats_cleanup(dev); | ||
| 293 | ionic_destroy_rdma_admin(dev); | ||
| 294 | ionic_destroy_resids(dev); | ||
| 295 | WARN_ON(!xa_empty(&dev->qp_tbl)); | ||
| 296 | xa_destroy(&dev->qp_tbl); | ||
| 297 | WARN_ON(!xa_empty(&dev->cq_tbl)); | ||
| 298 | xa_destroy(&dev->cq_tbl); | ||
| 299 | ib_dealloc_device(&dev->ibdev); | ||
| 300 | } | ||
| 301 | |||
| 302 | static struct ionic_ibdev *ionic_create_ibdev(struct ionic_aux_dev *ionic_adev) | ||
| 303 | { | ||
| 304 | struct ib_device *ibdev; | ||
| 305 | struct ionic_ibdev *dev; | ||
| 306 | struct net_device *ndev; | ||
| 307 | int rc; | ||
| 308 | |||
| 309 | dev = ib_alloc_device(ionic_ibdev, ibdev); | ||
| 310 | if (!dev) | ||
| 311 | return ERR_PTR(-EINVAL); | ||
| 312 | |||
| 313 | ionic_fill_lif_cfg(ionic_adev->lif, &dev->lif_cfg); | ||
| 314 | |||
| 315 | xa_init_flags(&dev->qp_tbl, GFP_ATOMIC); | ||
| 316 | xa_init_flags(&dev->cq_tbl, GFP_ATOMIC); | ||
| 317 | |||
| 318 | ionic_init_resids(dev); | ||
| 319 | |||
| 320 | rc = ionic_rdma_reset_devcmd(dev); | ||
| 321 | if (rc) | ||
| 322 | goto err_reset; | ||
| 323 | |||
| 324 | rc = ionic_create_rdma_admin(dev); | ||
| 325 | if (rc) | ||
| 326 | goto err_admin; | ||
| 327 | |||
| 328 | ibdev = &dev->ibdev; | ||
| 329 | ibdev->dev.parent = dev->lif_cfg.hwdev; | ||
| 330 | |||
| 331 | strscpy(ibdev->name, "ionic_%d", IB_DEVICE_NAME_MAX); | ||
| 332 | strscpy(ibdev->node_desc, DEVICE_DESCRIPTION, IB_DEVICE_NODE_DESC_MAX); | ||
| 333 | |||
| 334 | ibdev->node_type = RDMA_NODE_IB_CA; | ||
| 335 | ibdev->phys_port_cnt = 1; | ||
| 336 | |||
| 337 | /* the first two eq are reserved for async events */ | ||
| 338 | ibdev->num_comp_vectors = dev->lif_cfg.eq_count - 2; | ||
| 339 | |||
| 340 | ndev = ionic_lif_netdev(ionic_adev->lif); | ||
| 341 | addrconf_ifid_eui48((u8 *)&ibdev->node_guid, ndev); | ||
| 342 | rc = ib_device_set_netdev(ibdev, ndev, 1); | ||
| 343 | /* ionic_lif_netdev() returns ndev with refcount held */ | ||
| 344 | dev_put(ndev); | ||
| 345 | if (rc) | ||
| 346 | goto err_admin; | ||
| 347 | |||
| 348 | ib_set_device_ops(&dev->ibdev, &ionic_dev_ops); | ||
| 349 | |||
| 350 | ionic_stats_init(dev); | ||
| 351 | |||
| 352 | rc = ib_register_device(ibdev, "ionic_%d", ibdev->dev.parent); | ||
| 353 | if (rc) | ||
| 354 | goto err_register; | ||
| 355 | |||
| 356 | return dev; | ||
| 357 | |||
| 358 | err_register: | ||
| 359 | ionic_stats_cleanup(dev); | ||
| 360 | err_admin: | ||
| 361 | ionic_kill_rdma_admin(dev, false); | ||
| 362 | ionic_destroy_rdma_admin(dev); | ||
| 363 | err_reset: | ||
| 364 | ionic_destroy_resids(dev); | ||
| 365 | xa_destroy(&dev->qp_tbl); | ||
| 366 | xa_destroy(&dev->cq_tbl); | ||
| 367 | ib_dealloc_device(&dev->ibdev); | ||
| 368 | |||
| 369 | return ERR_PTR(rc); | ||
| 370 | } | ||
| 371 | |||
| 372 | static int ionic_aux_probe(struct auxiliary_device *adev, | ||
| 373 | const struct auxiliary_device_id *id) | ||
| 374 | { | ||
| 375 | struct ionic_aux_dev *ionic_adev; | ||
| 376 | struct ionic_ibdev *dev; | ||
| 377 | |||
| 378 | ionic_adev = container_of(adev, struct ionic_aux_dev, adev); | ||
| 379 | dev = ionic_create_ibdev(ionic_adev); | ||
| 380 | if (IS_ERR(dev)) | ||
| 381 | return dev_err_probe(&adev->dev, PTR_ERR(dev), | ||
| 382 | "Failed to register ibdev\n"); | ||
| 383 | |||
| 384 | auxiliary_set_drvdata(adev, dev); | ||
| 385 | ibdev_dbg(&dev->ibdev, "registered\n"); | ||
| 386 | |||
| 387 | return 0; | ||
| 388 | } | ||
| 389 | |||
| 390 | static void ionic_aux_remove(struct auxiliary_device *adev) | ||
| 391 | { | ||
| 392 | struct ionic_ibdev *dev = auxiliary_get_drvdata(adev); | ||
| 393 | |||
| 394 | dev_dbg(&adev->dev, "unregister ibdev\n"); | ||
| 395 | ionic_destroy_ibdev(dev); | ||
| 396 | dev_dbg(&adev->dev, "unregistered\n"); | ||
| 397 | } | ||
| 398 | |||
| 399 | static const struct auxiliary_device_id ionic_aux_id_table[] = { | ||
| 400 | { .name = "ionic.rdma", }, | ||
| 401 | {}, | ||
| 402 | }; | ||
| 403 | |||
| 404 | MODULE_DEVICE_TABLE(auxiliary, ionic_aux_id_table); | ||
| 405 | |||
| 406 | static struct auxiliary_driver ionic_aux_r_driver = { | ||
| 407 | .name = "rdma", | ||
| 408 | .probe = ionic_aux_probe, | ||
| 409 | .remove = ionic_aux_remove, | ||
| 410 | .id_table = ionic_aux_id_table, | ||
| 411 | }; | ||
| 412 | |||
| 413 | static int __init ionic_mod_init(void) | ||
| 414 | { | ||
| 415 | int rc; | ||
| 416 | |||
| 417 | ionic_evt_workq = create_workqueue(KBUILD_MODNAME "-evt"); | ||
| 418 | if (!ionic_evt_workq) | ||
| 419 | return -ENOMEM; | ||
| 420 | |||
| 421 | rc = auxiliary_driver_register(&ionic_aux_r_driver); | ||
| 422 | if (rc) | ||
| 423 | goto err_aux; | ||
| 424 | |||
| 425 | return 0; | ||
| 426 | |||
| 427 | err_aux: | ||
| 428 | destroy_workqueue(ionic_evt_workq); | ||
| 429 | |||
| 430 | return rc; | ||
| 431 | } | ||
| 432 | |||
| 433 | static void __exit ionic_mod_exit(void) | ||
| 434 | { | ||
| 435 | auxiliary_driver_unregister(&ionic_aux_r_driver); | ||
| 436 | destroy_workqueue(ionic_evt_workq); | ||
| 437 | } | ||
| 438 | |||
| 439 | module_init(ionic_mod_init); | ||
| 440 | module_exit(ionic_mod_exit); | ||
| @@ -0,0 +1,517 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 */ | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #ifndef _IONIC_IBDEV_H_ | ||
| 5 | #define _IONIC_IBDEV_H_ | ||
| 6 | |||
| 7 | #include <rdma/ib_umem.h> | ||
| 8 | #include <rdma/ib_verbs.h> | ||
| 9 | #include <rdma/ib_pack.h> | ||
| 10 | #include <rdma/uverbs_ioctl.h> | ||
| 11 | |||
| 12 | #include <rdma/ionic-abi.h> | ||
| 13 | #include <ionic_api.h> | ||
| 14 | #include <ionic_regs.h> | ||
| 15 | |||
| 16 | #include "ionic_fw.h" | ||
| 17 | #include "ionic_queue.h" | ||
| 18 | #include "ionic_res.h" | ||
| 19 | |||
| 20 | #include "ionic_lif_cfg.h" | ||
| 21 | |||
| 22 | /* Config knobs */ | ||
| 23 | #define IONIC_EQ_DEPTH 511 | ||
| 24 | #define IONIC_EQ_COUNT 32 | ||
| 25 | #define IONIC_AQ_DEPTH 63 | ||
| 26 | #define IONIC_AQ_COUNT 4 | ||
| 27 | #define IONIC_EQ_ISR_BUDGET 10 | ||
| 28 | #define IONIC_EQ_WORK_BUDGET 1000 | ||
| 29 | #define IONIC_MAX_RD_ATOM 16 | ||
| 30 | #define IONIC_PKEY_TBL_LEN 1 | ||
| 31 | #define IONIC_GID_TBL_LEN 256 | ||
| 32 | |||
| 33 | #define IONIC_MAX_QPID 0xffffff | ||
| 34 | #define IONIC_SPEC_HIGH 8 | ||
| 35 | #define IONIC_MAX_PD 1024 | ||
| 36 | #define IONIC_SPEC_HIGH 8 | ||
| 37 | #define IONIC_SQCMB_ORDER 5 | ||
| 38 | #define IONIC_RQCMB_ORDER 0 | ||
| 39 | |||
| 40 | #define IONIC_META_LAST ((void *)1ul) | ||
| 41 | #define IONIC_META_POSTED ((void *)2ul) | ||
| 42 | |||
| 43 | #define IONIC_CQ_GRACE 100 | ||
| 44 | |||
| 45 | #define IONIC_ROCE_UDP_SPORT 28272 | ||
| 46 | #define IONIC_DMA_LKEY 0 | ||
| 47 | #define IONIC_DMA_RKEY IONIC_DMA_LKEY | ||
| 48 | |||
| 49 | #define IONIC_CMB_SUPPORTED \ | ||
| 50 | (IONIC_CMB_ENABLE | IONIC_CMB_REQUIRE | IONIC_CMB_EXPDB | \ | ||
| 51 | IONIC_CMB_WC | IONIC_CMB_UC) | ||
| 52 | |||
| 53 | /* resource is not reserved on the device, indicated in tbl_order */ | ||
| 54 | #define IONIC_RES_INVALID -1 | ||
| 55 | |||
| 56 | struct ionic_aq; | ||
| 57 | struct ionic_cq; | ||
| 58 | struct ionic_eq; | ||
| 59 | struct ionic_vcq; | ||
| 60 | |||
| 61 | enum ionic_admin_state { | ||
| 62 | IONIC_ADMIN_ACTIVE, /* submitting admin commands to queue */ | ||
| 63 | IONIC_ADMIN_PAUSED, /* not submitting, but may complete normally */ | ||
| 64 | IONIC_ADMIN_KILLED, /* not submitting, locally completed */ | ||
| 65 | }; | ||
| 66 | |||
| 67 | enum ionic_admin_flags { | ||
| 68 | IONIC_ADMIN_F_BUSYWAIT = BIT(0), /* Don't sleep */ | ||
| 69 | IONIC_ADMIN_F_TEARDOWN = BIT(1), /* In destroy path */ | ||
| 70 | IONIC_ADMIN_F_INTERRUPT = BIT(2), /* Interruptible w/timeout */ | ||
| 71 | }; | ||
| 72 | |||
| 73 | enum ionic_mmap_flag { | ||
| 74 | IONIC_MMAP_WC = BIT(0), | ||
| 75 | }; | ||
| 76 | |||
| 77 | struct ionic_mmap_entry { | ||
| 78 | struct rdma_user_mmap_entry rdma_entry; | ||
| 79 | unsigned long size; | ||
| 80 | unsigned long pfn; | ||
| 81 | u8 mmap_flags; | ||
| 82 | }; | ||
| 83 | |||
| 84 | struct ionic_ibdev { | ||
| 85 | struct ib_device ibdev; | ||
| 86 | |||
| 87 | struct ionic_lif_cfg lif_cfg; | ||
| 88 | |||
| 89 | struct xarray qp_tbl; | ||
| 90 | struct xarray cq_tbl; | ||
| 91 | |||
| 92 | struct ionic_resid_bits inuse_dbid; | ||
| 93 | struct ionic_resid_bits inuse_pdid; | ||
| 94 | struct ionic_resid_bits inuse_ahid; | ||
| 95 | struct ionic_resid_bits inuse_mrid; | ||
| 96 | struct ionic_resid_bits inuse_qpid; | ||
| 97 | struct ionic_resid_bits inuse_cqid; | ||
| 98 | |||
| 99 | u8 half_cqid_udma_shift; | ||
| 100 | u8 half_qpid_udma_shift; | ||
| 101 | u8 next_qpid_udma_idx; | ||
| 102 | u8 next_mrkey; | ||
| 103 | |||
| 104 | struct work_struct reset_work; | ||
| 105 | bool reset_posted; | ||
| 106 | u32 reset_cnt; | ||
| 107 | |||
| 108 | struct delayed_work admin_dwork; | ||
| 109 | struct ionic_aq **aq_vec; | ||
| 110 | atomic_t admin_state; | ||
| 111 | |||
| 112 | struct ionic_eq **eq_vec; | ||
| 113 | |||
| 114 | struct ionic_v1_stat *hw_stats; | ||
| 115 | void *hw_stats_buf; | ||
| 116 | struct rdma_stat_desc *hw_stats_hdrs; | ||
| 117 | struct ionic_counter_stats *counter_stats; | ||
| 118 | int hw_stats_count; | ||
| 119 | }; | ||
| 120 | |||
| 121 | struct ionic_eq { | ||
| 122 | struct ionic_ibdev *dev; | ||
| 123 | |||
| 124 | u32 eqid; | ||
| 125 | u32 intr; | ||
| 126 | |||
| 127 | struct ionic_queue q; | ||
| 128 | |||
| 129 | int armed; | ||
| 130 | bool enable; | ||
| 131 | |||
| 132 | struct work_struct work; | ||
| 133 | |||
| 134 | int irq; | ||
| 135 | char name[32]; | ||
| 136 | }; | ||
| 137 | |||
| 138 | struct ionic_admin_wr { | ||
| 139 | struct completion work; | ||
| 140 | struct list_head aq_ent; | ||
| 141 | struct ionic_v1_admin_wqe wqe; | ||
| 142 | struct ionic_v1_cqe cqe; | ||
| 143 | struct ionic_aq *aq; | ||
| 144 | int status; | ||
| 145 | }; | ||
| 146 | |||
| 147 | struct ionic_admin_wr_q { | ||
| 148 | struct ionic_admin_wr *wr; | ||
| 149 | int wqe_strides; | ||
| 150 | }; | ||
| 151 | |||
| 152 | struct ionic_aq { | ||
| 153 | struct ionic_ibdev *dev; | ||
| 154 | struct ionic_vcq *vcq; | ||
| 155 | |||
| 156 | struct work_struct work; | ||
| 157 | |||
| 158 | atomic_t admin_state; | ||
| 159 | unsigned long stamp; | ||
| 160 | bool armed; | ||
| 161 | |||
| 162 | u32 aqid; | ||
| 163 | u32 cqid; | ||
| 164 | |||
| 165 | spinlock_t lock; /* for posting */ | ||
| 166 | struct ionic_queue q; | ||
| 167 | struct ionic_admin_wr_q *q_wr; | ||
| 168 | struct list_head wr_prod; | ||
| 169 | struct list_head wr_post; | ||
| 170 | }; | ||
| 171 | |||
| 172 | struct ionic_ctx { | ||
| 173 | struct ib_ucontext ibctx; | ||
| 174 | u32 dbid; | ||
| 175 | struct rdma_user_mmap_entry *mmap_dbell; | ||
| 176 | }; | ||
| 177 | |||
| 178 | struct ionic_tbl_buf { | ||
| 179 | u32 tbl_limit; | ||
| 180 | u32 tbl_pages; | ||
| 181 | size_t tbl_size; | ||
| 182 | __le64 *tbl_buf; | ||
| 183 | dma_addr_t tbl_dma; | ||
| 184 | u8 page_size_log2; | ||
| 185 | }; | ||
| 186 | |||
| 187 | struct ionic_pd { | ||
| 188 | struct ib_pd ibpd; | ||
| 189 | |||
| 190 | u32 pdid; | ||
| 191 | u32 flags; | ||
| 192 | }; | ||
| 193 | |||
| 194 | struct ionic_cq { | ||
| 195 | struct ionic_vcq *vcq; | ||
| 196 | |||
| 197 | u32 cqid; | ||
| 198 | u32 eqid; | ||
| 199 | |||
| 200 | spinlock_t lock; /* for polling */ | ||
| 201 | struct list_head poll_sq; | ||
| 202 | bool flush; | ||
| 203 | struct list_head flush_sq; | ||
| 204 | struct list_head flush_rq; | ||
| 205 | struct list_head ibkill_flush_ent; | ||
| 206 | |||
| 207 | struct ionic_queue q; | ||
| 208 | bool color; | ||
| 209 | int credit; | ||
| 210 | u16 arm_any_prod; | ||
| 211 | u16 arm_sol_prod; | ||
| 212 | |||
| 213 | struct kref cq_kref; | ||
| 214 | struct completion cq_rel_comp; | ||
| 215 | |||
| 216 | /* infrequently accessed, keep at end */ | ||
| 217 | struct ib_umem *umem; | ||
| 218 | }; | ||
| 219 | |||
| 220 | struct ionic_vcq { | ||
| 221 | struct ib_cq ibcq; | ||
| 222 | struct ionic_cq cq[2]; | ||
| 223 | u8 udma_mask; | ||
| 224 | u8 poll_idx; | ||
| 225 | }; | ||
| 226 | |||
| 227 | struct ionic_sq_meta { | ||
| 228 | u64 wrid; | ||
| 229 | u32 len; | ||
| 230 | u16 seq; | ||
| 231 | u8 ibop; | ||
| 232 | u8 ibsts; | ||
| 233 | u8 remote:1; | ||
| 234 | u8 signal:1; | ||
| 235 | u8 local_comp:1; | ||
| 236 | }; | ||
| 237 | |||
| 238 | struct ionic_rq_meta { | ||
| 239 | struct ionic_rq_meta *next; | ||
| 240 | u64 wrid; | ||
| 241 | }; | ||
| 242 | |||
| 243 | struct ionic_qp { | ||
| 244 | struct ib_qp ibqp; | ||
| 245 | enum ib_qp_state state; | ||
| 246 | |||
| 247 | u32 qpid; | ||
| 248 | u32 ahid; | ||
| 249 | u32 sq_cqid; | ||
| 250 | u32 rq_cqid; | ||
| 251 | u8 udma_idx; | ||
| 252 | u8 has_ah:1; | ||
| 253 | u8 has_sq:1; | ||
| 254 | u8 has_rq:1; | ||
| 255 | u8 sig_all:1; | ||
| 256 | |||
| 257 | struct list_head qp_list_counter; | ||
| 258 | |||
| 259 | struct list_head cq_poll_sq; | ||
| 260 | struct list_head cq_flush_sq; | ||
| 261 | struct list_head cq_flush_rq; | ||
| 262 | struct list_head ibkill_flush_ent; | ||
| 263 | |||
| 264 | spinlock_t sq_lock; /* for posting and polling */ | ||
| 265 | struct ionic_queue sq; | ||
| 266 | struct ionic_sq_meta *sq_meta; | ||
| 267 | u16 *sq_msn_idx; | ||
| 268 | int sq_spec; | ||
| 269 | u16 sq_old_prod; | ||
| 270 | u16 sq_msn_prod; | ||
| 271 | u16 sq_msn_cons; | ||
| 272 | u8 sq_cmb; | ||
| 273 | bool sq_flush; | ||
| 274 | bool sq_flush_rcvd; | ||
| 275 | |||
| 276 | spinlock_t rq_lock; /* for posting and polling */ | ||
| 277 | struct ionic_queue rq; | ||
| 278 | struct ionic_rq_meta *rq_meta; | ||
| 279 | struct ionic_rq_meta *rq_meta_head; | ||
| 280 | int rq_spec; | ||
| 281 | u16 rq_old_prod; | ||
| 282 | u8 rq_cmb; | ||
| 283 | bool rq_flush; | ||
| 284 | |||
| 285 | struct kref qp_kref; | ||
| 286 | struct completion qp_rel_comp; | ||
| 287 | |||
| 288 | /* infrequently accessed, keep at end */ | ||
| 289 | int sgid_index; | ||
| 290 | int sq_cmb_order; | ||
| 291 | u32 sq_cmb_pgid; | ||
| 292 | phys_addr_t sq_cmb_addr; | ||
| 293 | struct rdma_user_mmap_entry *mmap_sq_cmb; | ||
| 294 | |||
| 295 | struct ib_umem *sq_umem; | ||
| 296 | |||
| 297 | int rq_cmb_order; | ||
| 298 | u32 rq_cmb_pgid; | ||
| 299 | phys_addr_t rq_cmb_addr; | ||
| 300 | struct rdma_user_mmap_entry *mmap_rq_cmb; | ||
| 301 | |||
| 302 | struct ib_umem *rq_umem; | ||
| 303 | |||
| 304 | int dcqcn_profile; | ||
| 305 | |||
| 306 | struct ib_ud_header *hdr; | ||
| 307 | }; | ||
| 308 | |||
| 309 | struct ionic_ah { | ||
| 310 | struct ib_ah ibah; | ||
| 311 | u32 ahid; | ||
| 312 | int sgid_index; | ||
| 313 | struct ib_ud_header hdr; | ||
| 314 | }; | ||
| 315 | |||
| 316 | struct ionic_mr { | ||
| 317 | union { | ||
| 318 | struct ib_mr ibmr; | ||
| 319 | struct ib_mw ibmw; | ||
| 320 | }; | ||
| 321 | |||
| 322 | u32 mrid; | ||
| 323 | int flags; | ||
| 324 | |||
| 325 | struct ib_umem *umem; | ||
| 326 | struct ionic_tbl_buf buf; | ||
| 327 | bool created; | ||
| 328 | }; | ||
| 329 | |||
| 330 | struct ionic_counter_stats { | ||
| 331 | int queue_stats_count; | ||
| 332 | struct ionic_v1_stat *hdr; | ||
| 333 | struct rdma_stat_desc *stats_hdrs; | ||
| 334 | struct xarray xa_counters; | ||
| 335 | }; | ||
| 336 | |||
| 337 | struct ionic_counter { | ||
| 338 | void *vals; | ||
| 339 | struct list_head qp_list; | ||
| 340 | }; | ||
| 341 | |||
| 342 | static inline struct ionic_ibdev *to_ionic_ibdev(struct ib_device *ibdev) | ||
| 343 | { | ||
| 344 | return container_of(ibdev, struct ionic_ibdev, ibdev); | ||
| 345 | } | ||
| 346 | |||
| 347 | static inline struct ionic_ctx *to_ionic_ctx(struct ib_ucontext *ibctx) | ||
| 348 | { | ||
| 349 | return container_of(ibctx, struct ionic_ctx, ibctx); | ||
| 350 | } | ||
| 351 | |||
| 352 | static inline struct ionic_ctx *to_ionic_ctx_uobj(struct ib_uobject *uobj) | ||
| 353 | { | ||
| 354 | if (!uobj) | ||
| 355 | return NULL; | ||
| 356 | |||
| 357 | if (!uobj->context) | ||
| 358 | return NULL; | ||
| 359 | |||
| 360 | return to_ionic_ctx(uobj->context); | ||
| 361 | } | ||
| 362 | |||
| 363 | static inline struct ionic_pd *to_ionic_pd(struct ib_pd *ibpd) | ||
| 364 | { | ||
| 365 | return container_of(ibpd, struct ionic_pd, ibpd); | ||
| 366 | } | ||
| 367 | |||
| 368 | static inline struct ionic_mr *to_ionic_mr(struct ib_mr *ibmr) | ||
| 369 | { | ||
| 370 | return container_of(ibmr, struct ionic_mr, ibmr); | ||
| 371 | } | ||
| 372 | |||
| 373 | static inline struct ionic_mr *to_ionic_mw(struct ib_mw *ibmw) | ||
| 374 | { | ||
| 375 | return container_of(ibmw, struct ionic_mr, ibmw); | ||
| 376 | } | ||
| 377 | |||
| 378 | static inline struct ionic_vcq *to_ionic_vcq(struct ib_cq *ibcq) | ||
| 379 | { | ||
| 380 | return container_of(ibcq, struct ionic_vcq, ibcq); | ||
| 381 | } | ||
| 382 | |||
| 383 | static inline struct ionic_cq *to_ionic_vcq_cq(struct ib_cq *ibcq, | ||
| 384 | uint8_t udma_idx) | ||
| 385 | { | ||
| 386 | return &to_ionic_vcq(ibcq)->cq[udma_idx]; | ||
| 387 | } | ||
| 388 | |||
| 389 | static inline struct ionic_qp *to_ionic_qp(struct ib_qp *ibqp) | ||
| 390 | { | ||
| 391 | return container_of(ibqp, struct ionic_qp, ibqp); | ||
| 392 | } | ||
| 393 | |||
| 394 | static inline struct ionic_ah *to_ionic_ah(struct ib_ah *ibah) | ||
| 395 | { | ||
| 396 | return container_of(ibah, struct ionic_ah, ibah); | ||
| 397 | } | ||
| 398 | |||
| 399 | static inline u32 ionic_ctx_dbid(struct ionic_ibdev *dev, | ||
| 400 | struct ionic_ctx *ctx) | ||
| 401 | { | ||
| 402 | if (!ctx) | ||
| 403 | return dev->lif_cfg.dbid; | ||
| 404 | |||
| 405 | return ctx->dbid; | ||
| 406 | } | ||
| 407 | |||
| 408 | static inline u32 ionic_obj_dbid(struct ionic_ibdev *dev, | ||
| 409 | struct ib_uobject *uobj) | ||
| 410 | { | ||
| 411 | return ionic_ctx_dbid(dev, to_ionic_ctx_uobj(uobj)); | ||
| 412 | } | ||
| 413 | |||
| 414 | static inline bool ionic_ibop_is_local(enum ib_wr_opcode op) | ||
| 415 | { | ||
| 416 | return op == IB_WR_LOCAL_INV || op == IB_WR_REG_MR; | ||
| 417 | } | ||
| 418 | |||
| 419 | static inline void ionic_qp_complete(struct kref *kref) | ||
| 420 | { | ||
| 421 | struct ionic_qp *qp = container_of(kref, struct ionic_qp, qp_kref); | ||
| 422 | |||
| 423 | complete(&qp->qp_rel_comp); | ||
| 424 | } | ||
| 425 | |||
| 426 | static inline void ionic_cq_complete(struct kref *kref) | ||
| 427 | { | ||
| 428 | struct ionic_cq *cq = container_of(kref, struct ionic_cq, cq_kref); | ||
| 429 | |||
| 430 | complete(&cq->cq_rel_comp); | ||
| 431 | } | ||
| 432 | |||
| 433 | /* ionic_admin.c */ | ||
| 434 | extern struct workqueue_struct *ionic_evt_workq; | ||
| 435 | void ionic_admin_post(struct ionic_ibdev *dev, struct ionic_admin_wr *wr); | ||
| 436 | int ionic_admin_wait(struct ionic_ibdev *dev, struct ionic_admin_wr *wr, | ||
| 437 | enum ionic_admin_flags); | ||
| 438 | |||
| 439 | int ionic_rdma_reset_devcmd(struct ionic_ibdev *dev); | ||
| 440 | |||
| 441 | int ionic_create_rdma_admin(struct ionic_ibdev *dev); | ||
| 442 | void ionic_destroy_rdma_admin(struct ionic_ibdev *dev); | ||
| 443 | void ionic_kill_rdma_admin(struct ionic_ibdev *dev, bool fatal_path); | ||
| 444 | |||
| 445 | /* ionic_controlpath.c */ | ||
| 446 | int ionic_create_cq_common(struct ionic_vcq *vcq, | ||
| 447 | struct ionic_tbl_buf *buf, | ||
| 448 | const struct ib_cq_init_attr *attr, | ||
| 449 | struct ionic_ctx *ctx, | ||
| 450 | struct ib_udata *udata, | ||
| 451 | struct ionic_qdesc *req_cq, | ||
| 452 | __u32 *resp_cqid, | ||
| 453 | int udma_idx); | ||
| 454 | void ionic_destroy_cq_common(struct ionic_ibdev *dev, struct ionic_cq *cq); | ||
| 455 | void ionic_flush_qp(struct ionic_ibdev *dev, struct ionic_qp *qp); | ||
| 456 | void ionic_notify_flush_cq(struct ionic_cq *cq); | ||
| 457 | |||
| 458 | int ionic_alloc_ucontext(struct ib_ucontext *ibctx, struct ib_udata *udata); | ||
| 459 | void ionic_dealloc_ucontext(struct ib_ucontext *ibctx); | ||
| 460 | int ionic_mmap(struct ib_ucontext *ibctx, struct vm_area_struct *vma); | ||
| 461 | void ionic_mmap_free(struct rdma_user_mmap_entry *rdma_entry); | ||
| 462 | int ionic_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata); | ||
| 463 | int ionic_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata); | ||
| 464 | int ionic_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *init_attr, | ||
| 465 | struct ib_udata *udata); | ||
| 466 | int ionic_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr); | ||
| 467 | int ionic_destroy_ah(struct ib_ah *ibah, u32 flags); | ||
| 468 | struct ib_mr *ionic_get_dma_mr(struct ib_pd *ibpd, int access); | ||
| 469 | struct ib_mr *ionic_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length, | ||
| 470 | u64 addr, int access, struct ib_dmah *dmah, | ||
| 471 | struct ib_udata *udata); | ||
| 472 | struct ib_mr *ionic_reg_user_mr_dmabuf(struct ib_pd *ibpd, u64 offset, | ||
| 473 | u64 length, u64 addr, int fd, int access, | ||
| 474 | struct ib_dmah *dmah, | ||
| 475 | struct uverbs_attr_bundle *attrs); | ||
| 476 | int ionic_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata); | ||
| 477 | struct ib_mr *ionic_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type type, | ||
| 478 | u32 max_sg); | ||
| 479 | int ionic_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents, | ||
| 480 | unsigned int *sg_offset); | ||
| 481 | int ionic_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata); | ||
| 482 | int ionic_dealloc_mw(struct ib_mw *ibmw); | ||
| 483 | int ionic_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, | ||
| 484 | struct uverbs_attr_bundle *attrs); | ||
| 485 | int ionic_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata); | ||
| 486 | int ionic_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *attr, | ||
| 487 | struct ib_udata *udata); | ||
| 488 | int ionic_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int mask, | ||
| 489 | struct ib_udata *udata); | ||
| 490 | int ionic_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int mask, | ||
| 491 | struct ib_qp_init_attr *init_attr); | ||
| 492 | int ionic_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata); | ||
| 493 | |||
| 494 | /* ionic_datapath.c */ | ||
| 495 | int ionic_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr, | ||
| 496 | const struct ib_send_wr **bad); | ||
| 497 | int ionic_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr, | ||
| 498 | const struct ib_recv_wr **bad); | ||
| 499 | int ionic_poll_cq(struct ib_cq *ibcq, int nwc, struct ib_wc *wc); | ||
| 500 | int ionic_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags); | ||
| 501 | |||
| 502 | /* ionic_hw_stats.c */ | ||
| 503 | void ionic_stats_init(struct ionic_ibdev *dev); | ||
| 504 | void ionic_stats_cleanup(struct ionic_ibdev *dev); | ||
| 505 | |||
| 506 | /* ionic_pgtbl.c */ | ||
| 507 | __le64 ionic_pgtbl_dma(struct ionic_tbl_buf *buf, u64 va); | ||
| 508 | __be64 ionic_pgtbl_off(struct ionic_tbl_buf *buf, u64 va); | ||
| 509 | int ionic_pgtbl_page(struct ionic_tbl_buf *buf, u64 dma); | ||
| 510 | int ionic_pgtbl_init(struct ionic_ibdev *dev, | ||
| 511 | struct ionic_tbl_buf *buf, | ||
| 512 | struct ib_umem *umem, | ||
| 513 | dma_addr_t dma, | ||
| 514 | int limit, | ||
| 515 | u64 page_size); | ||
| 516 | void ionic_pgtbl_unbuf(struct ionic_ibdev *dev, struct ionic_tbl_buf *buf); | ||
| 517 | #endif /* _IONIC_IBDEV_H_ */ | ||
| @@ -0,0 +1,111 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #include <linux/kernel.h> | ||
| 5 | |||
| 6 | #include <ionic.h> | ||
| 7 | #include <ionic_lif.h> | ||
| 8 | |||
| 9 | #include "ionic_lif_cfg.h" | ||
| 10 | |||
| 11 | #define IONIC_MIN_RDMA_VERSION 0 | ||
| 12 | #define IONIC_MAX_RDMA_VERSION 2 | ||
| 13 | |||
| 14 | static u8 ionic_get_expdb(struct ionic_lif *lif) | ||
| 15 | { | ||
| 16 | u8 expdb_support = 0; | ||
| 17 | |||
| 18 | if (lif->ionic->idev.phy_cmb_expdb64_pages) | ||
| 19 | expdb_support |= IONIC_EXPDB_64B_WQE; | ||
| 20 | if (lif->ionic->idev.phy_cmb_expdb128_pages) | ||
| 21 | expdb_support |= IONIC_EXPDB_128B_WQE; | ||
| 22 | if (lif->ionic->idev.phy_cmb_expdb256_pages) | ||
| 23 | expdb_support |= IONIC_EXPDB_256B_WQE; | ||
| 24 | if (lif->ionic->idev.phy_cmb_expdb512_pages) | ||
| 25 | expdb_support |= IONIC_EXPDB_512B_WQE; | ||
| 26 | |||
| 27 | return expdb_support; | ||
| 28 | } | ||
| 29 | |||
| 30 | void ionic_fill_lif_cfg(struct ionic_lif *lif, struct ionic_lif_cfg *cfg) | ||
| 31 | { | ||
| 32 | union ionic_lif_identity *ident = &lif->ionic->ident.lif; | ||
| 33 | |||
| 34 | cfg->lif = lif; | ||
| 35 | cfg->hwdev = &lif->ionic->pdev->dev; | ||
| 36 | cfg->lif_index = lif->index; | ||
| 37 | cfg->lif_hw_index = lif->hw_index; | ||
| 38 | |||
| 39 | cfg->dbid = lif->kern_pid; | ||
| 40 | cfg->dbid_count = le32_to_cpu(lif->ionic->ident.dev.ndbpgs_per_lif); | ||
| 41 | cfg->dbpage = lif->kern_dbpage; | ||
| 42 | cfg->intr_ctrl = lif->ionic->idev.intr_ctrl; | ||
| 43 | |||
| 44 | cfg->db_phys = lif->ionic->bars[IONIC_PCI_BAR_DBELL].bus_addr; | ||
| 45 | |||
| 46 | if (IONIC_VERSION(ident->rdma.version, ident->rdma.minor_version) >= | ||
| 47 | IONIC_VERSION(2, 1)) | ||
| 48 | cfg->page_size_supported = | ||
| 49 | le64_to_cpu(ident->rdma.page_size_cap); | ||
| 50 | else | ||
| 51 | cfg->page_size_supported = IONIC_PAGE_SIZE_SUPPORTED; | ||
| 52 | |||
| 53 | cfg->rdma_version = ident->rdma.version; | ||
| 54 | cfg->qp_opcodes = ident->rdma.qp_opcodes; | ||
| 55 | cfg->admin_opcodes = ident->rdma.admin_opcodes; | ||
| 56 | |||
| 57 | cfg->stats_type = le16_to_cpu(ident->rdma.stats_type); | ||
| 58 | cfg->npts_per_lif = le32_to_cpu(ident->rdma.npts_per_lif); | ||
| 59 | cfg->nmrs_per_lif = le32_to_cpu(ident->rdma.nmrs_per_lif); | ||
| 60 | cfg->nahs_per_lif = le32_to_cpu(ident->rdma.nahs_per_lif); | ||
| 61 | |||
| 62 | cfg->aq_base = le32_to_cpu(ident->rdma.aq_qtype.qid_base); | ||
| 63 | cfg->cq_base = le32_to_cpu(ident->rdma.cq_qtype.qid_base); | ||
| 64 | cfg->eq_base = le32_to_cpu(ident->rdma.eq_qtype.qid_base); | ||
| 65 | |||
| 66 | /* | ||
| 67 | * ionic_create_rdma_admin() may reduce aq_count or eq_count if | ||
| 68 | * it is unable to allocate all that were requested. | ||
| 69 | * aq_count is tunable; see ionic_aq_count | ||
| 70 | * eq_count is tunable; see ionic_eq_count | ||
| 71 | */ | ||
| 72 | cfg->aq_count = le32_to_cpu(ident->rdma.aq_qtype.qid_count); | ||
| 73 | cfg->eq_count = le32_to_cpu(ident->rdma.eq_qtype.qid_count); | ||
| 74 | cfg->cq_count = le32_to_cpu(ident->rdma.cq_qtype.qid_count); | ||
| 75 | cfg->qp_count = le32_to_cpu(ident->rdma.sq_qtype.qid_count); | ||
| 76 | cfg->dbid_count = le32_to_cpu(lif->ionic->ident.dev.ndbpgs_per_lif); | ||
| 77 | |||
| 78 | cfg->aq_qtype = ident->rdma.aq_qtype.qtype; | ||
| 79 | cfg->sq_qtype = ident->rdma.sq_qtype.qtype; | ||
| 80 | cfg->rq_qtype = ident->rdma.rq_qtype.qtype; | ||
| 81 | cfg->cq_qtype = ident->rdma.cq_qtype.qtype; | ||
| 82 | cfg->eq_qtype = ident->rdma.eq_qtype.qtype; | ||
| 83 | cfg->udma_qgrp_shift = ident->rdma.udma_shift; | ||
| 84 | cfg->udma_count = 2; | ||
| 85 | |||
| 86 | cfg->max_stride = ident->rdma.max_stride; | ||
| 87 | cfg->expdb_mask = ionic_get_expdb(lif); | ||
| 88 | |||
| 89 | cfg->sq_expdb = | ||
| 90 | !!(lif->qtype_info[IONIC_QTYPE_TXQ].features & IONIC_QIDENT_F_EXPDB); | ||
| 91 | cfg->rq_expdb = | ||
| 92 | !!(lif->qtype_info[IONIC_QTYPE_RXQ].features & IONIC_QIDENT_F_EXPDB); | ||
| 93 | } | ||
| 94 | |||
| 95 | struct net_device *ionic_lif_netdev(struct ionic_lif *lif) | ||
| 96 | { | ||
| 97 | struct net_device *netdev = lif->netdev; | ||
| 98 | |||
| 99 | dev_hold(netdev); | ||
| 100 | return netdev; | ||
| 101 | } | ||
| 102 | |||
| 103 | void ionic_lif_fw_version(struct ionic_lif *lif, char *str, size_t len) | ||
| 104 | { | ||
| 105 | strscpy(str, lif->ionic->idev.dev_info.fw_version, len); | ||
| 106 | } | ||
| 107 | |||
| 108 | u8 ionic_lif_asic_rev(struct ionic_lif *lif) | ||
| 109 | { | ||
| 110 | return lif->ionic->idev.dev_info.asic_rev; | ||
| 111 | } | ||
| @@ -0,0 +1,66 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 */ | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #ifndef _IONIC_LIF_CFG_H_ | ||
| 5 | |||
| 6 | #define IONIC_VERSION(a, b) (((a) << 16) + ((b) << 8)) | ||
| 7 | #define IONIC_PAGE_SIZE_SUPPORTED 0x40201000 /* 4kb, 2Mb, 1Gb */ | ||
| 8 | |||
| 9 | #define IONIC_EXPDB_64B_WQE BIT(0) | ||
| 10 | #define IONIC_EXPDB_128B_WQE BIT(1) | ||
| 11 | #define IONIC_EXPDB_256B_WQE BIT(2) | ||
| 12 | #define IONIC_EXPDB_512B_WQE BIT(3) | ||
| 13 | |||
| 14 | struct ionic_lif_cfg { | ||
| 15 | struct device *hwdev; | ||
| 16 | struct ionic_lif *lif; | ||
| 17 | |||
| 18 | int lif_index; | ||
| 19 | int lif_hw_index; | ||
| 20 | |||
| 21 | u32 dbid; | ||
| 22 | int dbid_count; | ||
| 23 | u64 __iomem *dbpage; | ||
| 24 | struct ionic_intr __iomem *intr_ctrl; | ||
| 25 | phys_addr_t db_phys; | ||
| 26 | |||
| 27 | u64 page_size_supported; | ||
| 28 | u32 npts_per_lif; | ||
| 29 | u32 nmrs_per_lif; | ||
| 30 | u32 nahs_per_lif; | ||
| 31 | |||
| 32 | u32 aq_base; | ||
| 33 | u32 cq_base; | ||
| 34 | u32 eq_base; | ||
| 35 | |||
| 36 | int aq_count; | ||
| 37 | int eq_count; | ||
| 38 | int cq_count; | ||
| 39 | int qp_count; | ||
| 40 | |||
| 41 | u16 stats_type; | ||
| 42 | u8 aq_qtype; | ||
| 43 | u8 sq_qtype; | ||
| 44 | u8 rq_qtype; | ||
| 45 | u8 cq_qtype; | ||
| 46 | u8 eq_qtype; | ||
| 47 | |||
| 48 | u8 udma_count; | ||
| 49 | u8 udma_qgrp_shift; | ||
| 50 | |||
| 51 | u8 rdma_version; | ||
| 52 | u8 qp_opcodes; | ||
| 53 | u8 admin_opcodes; | ||
| 54 | |||
| 55 | u8 max_stride; | ||
| 56 | bool sq_expdb; | ||
| 57 | bool rq_expdb; | ||
| 58 | u8 expdb_mask; | ||
| 59 | }; | ||
| 60 | |||
| 61 | void ionic_fill_lif_cfg(struct ionic_lif *lif, struct ionic_lif_cfg *cfg); | ||
| 62 | struct net_device *ionic_lif_netdev(struct ionic_lif *lif); | ||
| 63 | void ionic_lif_fw_version(struct ionic_lif *lif, char *str, size_t len); | ||
| 64 | u8 ionic_lif_asic_rev(struct ionic_lif *lif); | ||
| 65 | |||
| 66 | #endif /* _IONIC_LIF_CFG_H_ */ | ||
| @@ -0,0 +1,143 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #include <linux/mman.h> | ||
| 5 | #include <linux/dma-mapping.h> | ||
| 6 | |||
| 7 | #include "ionic_fw.h" | ||
| 8 | #include "ionic_ibdev.h" | ||
| 9 | |||
| 10 | __le64 ionic_pgtbl_dma(struct ionic_tbl_buf *buf, u64 va) | ||
| 11 | { | ||
| 12 | u64 pg_mask = BIT_ULL(buf->page_size_log2) - 1; | ||
| 13 | u64 dma; | ||
| 14 | |||
| 15 | if (!buf->tbl_pages) | ||
| 16 | return cpu_to_le64(0); | ||
| 17 | |||
| 18 | if (buf->tbl_pages > 1) | ||
| 19 | return cpu_to_le64(buf->tbl_dma); | ||
| 20 | |||
| 21 | if (buf->tbl_buf) | ||
| 22 | dma = le64_to_cpu(buf->tbl_buf[0]); | ||
| 23 | else | ||
| 24 | dma = buf->tbl_dma; | ||
| 25 | |||
| 26 | return cpu_to_le64(dma + (va & pg_mask)); | ||
| 27 | } | ||
| 28 | |||
| 29 | __be64 ionic_pgtbl_off(struct ionic_tbl_buf *buf, u64 va) | ||
| 30 | { | ||
| 31 | if (buf->tbl_pages > 1) { | ||
| 32 | u64 pg_mask = BIT_ULL(buf->page_size_log2) - 1; | ||
| 33 | |||
| 34 | return cpu_to_be64(va & pg_mask); | ||
| 35 | } | ||
| 36 | |||
| 37 | return 0; | ||
| 38 | } | ||
| 39 | |||
| 40 | int ionic_pgtbl_page(struct ionic_tbl_buf *buf, u64 dma) | ||
| 41 | { | ||
| 42 | if (unlikely(buf->tbl_pages == buf->tbl_limit)) | ||
| 43 | return -ENOMEM; | ||
| 44 | |||
| 45 | if (buf->tbl_buf) | ||
| 46 | buf->tbl_buf[buf->tbl_pages] = cpu_to_le64(dma); | ||
| 47 | else | ||
| 48 | buf->tbl_dma = dma; | ||
| 49 | |||
| 50 | ++buf->tbl_pages; | ||
| 51 | |||
| 52 | return 0; | ||
| 53 | } | ||
| 54 | |||
| 55 | static int ionic_tbl_buf_alloc(struct ionic_ibdev *dev, | ||
| 56 | struct ionic_tbl_buf *buf) | ||
| 57 | { | ||
| 58 | int rc; | ||
| 59 | |||
| 60 | buf->tbl_size = buf->tbl_limit * sizeof(*buf->tbl_buf); | ||
| 61 | buf->tbl_buf = kmalloc(buf->tbl_size, GFP_KERNEL); | ||
| 62 | if (!buf->tbl_buf) | ||
| 63 | return -ENOMEM; | ||
| 64 | |||
| 65 | buf->tbl_dma = dma_map_single(dev->lif_cfg.hwdev, buf->tbl_buf, | ||
| 66 | buf->tbl_size, DMA_TO_DEVICE); | ||
| 67 | rc = dma_mapping_error(dev->lif_cfg.hwdev, buf->tbl_dma); | ||
| 68 | if (rc) { | ||
| 69 | kfree(buf->tbl_buf); | ||
| 70 | return rc; | ||
| 71 | } | ||
| 72 | |||
| 73 | return 0; | ||
| 74 | } | ||
| 75 | |||
| 76 | static int ionic_pgtbl_umem(struct ionic_tbl_buf *buf, struct ib_umem *umem) | ||
| 77 | { | ||
| 78 | struct ib_block_iter biter; | ||
| 79 | u64 page_dma; | ||
| 80 | int rc; | ||
| 81 | |||
| 82 | rdma_umem_for_each_dma_block(umem, &biter, BIT_ULL(buf->page_size_log2)) { | ||
| 83 | page_dma = rdma_block_iter_dma_address(&biter); | ||
| 84 | rc = ionic_pgtbl_page(buf, page_dma); | ||
| 85 | if (rc) | ||
| 86 | return rc; | ||
| 87 | } | ||
| 88 | |||
| 89 | return 0; | ||
| 90 | } | ||
| 91 | |||
| 92 | void ionic_pgtbl_unbuf(struct ionic_ibdev *dev, struct ionic_tbl_buf *buf) | ||
| 93 | { | ||
| 94 | if (buf->tbl_buf) | ||
| 95 | dma_unmap_single(dev->lif_cfg.hwdev, buf->tbl_dma, | ||
| 96 | buf->tbl_size, DMA_TO_DEVICE); | ||
| 97 | |||
| 98 | kfree(buf->tbl_buf); | ||
| 99 | memset(buf, 0, sizeof(*buf)); | ||
| 100 | } | ||
| 101 | |||
| 102 | int ionic_pgtbl_init(struct ionic_ibdev *dev, | ||
| 103 | struct ionic_tbl_buf *buf, | ||
| 104 | struct ib_umem *umem, | ||
| 105 | dma_addr_t dma, | ||
| 106 | int limit, | ||
| 107 | u64 page_size) | ||
| 108 | { | ||
| 109 | int rc; | ||
| 110 | |||
| 111 | memset(buf, 0, sizeof(*buf)); | ||
| 112 | |||
| 113 | if (umem) { | ||
| 114 | limit = ib_umem_num_dma_blocks(umem, page_size); | ||
| 115 | buf->page_size_log2 = order_base_2(page_size); | ||
| 116 | } | ||
| 117 | |||
| 118 | if (limit < 1) | ||
| 119 | return -EINVAL; | ||
| 120 | |||
| 121 | buf->tbl_limit = limit; | ||
| 122 | |||
| 123 | /* skip pgtbl if contiguous / direct translation */ | ||
| 124 | if (limit > 1) { | ||
| 125 | rc = ionic_tbl_buf_alloc(dev, buf); | ||
| 126 | if (rc) | ||
| 127 | return rc; | ||
| 128 | } | ||
| 129 | |||
| 130 | if (umem) | ||
| 131 | rc = ionic_pgtbl_umem(buf, umem); | ||
| 132 | else | ||
| 133 | rc = ionic_pgtbl_page(buf, dma); | ||
| 134 | |||
| 135 | if (rc) | ||
| 136 | goto err_unbuf; | ||
| 137 | |||
| 138 | return 0; | ||
| 139 | |||
| 140 | err_unbuf: | ||
| 141 | ionic_pgtbl_unbuf(dev, buf); | ||
| 142 | return rc; | ||
| 143 | } | ||
| @@ -0,0 +1,52 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #include <linux/dma-mapping.h> | ||
| 5 | |||
| 6 | #include "ionic_queue.h" | ||
| 7 | |||
| 8 | int ionic_queue_init(struct ionic_queue *q, struct device *dma_dev, | ||
| 9 | int depth, size_t stride) | ||
| 10 | { | ||
| 11 | if (depth < 0 || depth > 0xffff) | ||
| 12 | return -EINVAL; | ||
| 13 | |||
| 14 | if (stride == 0 || stride > 0x10000) | ||
| 15 | return -EINVAL; | ||
| 16 | |||
| 17 | if (depth == 0) | ||
| 18 | depth = 1; | ||
| 19 | |||
| 20 | q->depth_log2 = order_base_2(depth + 1); | ||
| 21 | q->stride_log2 = order_base_2(stride); | ||
| 22 | |||
| 23 | if (q->depth_log2 + q->stride_log2 < PAGE_SHIFT) | ||
| 24 | q->depth_log2 = PAGE_SHIFT - q->stride_log2; | ||
| 25 | |||
| 26 | if (q->depth_log2 > 16 || q->stride_log2 > 16) | ||
| 27 | return -EINVAL; | ||
| 28 | |||
| 29 | q->size = BIT_ULL(q->depth_log2 + q->stride_log2); | ||
| 30 | q->mask = BIT(q->depth_log2) - 1; | ||
| 31 | |||
| 32 | q->ptr = dma_alloc_coherent(dma_dev, q->size, &q->dma, GFP_KERNEL); | ||
| 33 | if (!q->ptr) | ||
| 34 | return -ENOMEM; | ||
| 35 | |||
| 36 | /* it will always be page aligned, but just to be sure... */ | ||
| 37 | if (!PAGE_ALIGNED(q->ptr)) { | ||
| 38 | dma_free_coherent(dma_dev, q->size, q->ptr, q->dma); | ||
| 39 | return -ENOMEM; | ||
| 40 | } | ||
| 41 | |||
| 42 | q->prod = 0; | ||
| 43 | q->cons = 0; | ||
| 44 | q->dbell = 0; | ||
| 45 | |||
| 46 | return 0; | ||
| 47 | } | ||
| 48 | |||
| 49 | void ionic_queue_destroy(struct ionic_queue *q, struct device *dma_dev) | ||
| 50 | { | ||
| 51 | dma_free_coherent(dma_dev, q->size, q->ptr, q->dma); | ||
| 52 | } | ||
| @@ -0,0 +1,234 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 */ | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #ifndef _IONIC_QUEUE_H_ | ||
| 5 | #define _IONIC_QUEUE_H_ | ||
| 6 | |||
| 7 | #include <linux/io.h> | ||
| 8 | #include <ionic_regs.h> | ||
| 9 | |||
| 10 | #define IONIC_MAX_DEPTH 0xffff | ||
| 11 | #define IONIC_MAX_CQ_DEPTH 0xffff | ||
| 12 | #define IONIC_CQ_RING_ARM IONIC_DBELL_RING_1 | ||
| 13 | #define IONIC_CQ_RING_SOL IONIC_DBELL_RING_2 | ||
| 14 | |||
| 15 | /** | ||
| 16 | * struct ionic_queue - Ring buffer used between device and driver | ||
| 17 | * @size: Size of the buffer, in bytes | ||
| 18 | * @dma: Dma address of the buffer | ||
| 19 | * @ptr: Buffer virtual address | ||
| 20 | * @prod: Driver position in the queue | ||
| 21 | * @cons: Device position in the queue | ||
| 22 | * @mask: Capacity of the queue, subtracting the hole | ||
| 23 | * This value is equal to ((1 << depth_log2) - 1) | ||
| 24 | * @depth_log2: Log base two size depth of the queue | ||
| 25 | * @stride_log2: Log base two size of an element in the queue | ||
| 26 | * @dbell: Doorbell identifying bits | ||
| 27 | */ | ||
| 28 | struct ionic_queue { | ||
| 29 | size_t size; | ||
| 30 | dma_addr_t dma; | ||
| 31 | void *ptr; | ||
| 32 | u16 prod; | ||
| 33 | u16 cons; | ||
| 34 | u16 mask; | ||
| 35 | u8 depth_log2; | ||
| 36 | u8 stride_log2; | ||
| 37 | u64 dbell; | ||
| 38 | }; | ||
| 39 | |||
| 40 | /** | ||
| 41 | * ionic_queue_init() - Initialize user space queue | ||
| 42 | * @q: Uninitialized queue structure | ||
| 43 | * @dma_dev: DMA device for mapping | ||
| 44 | * @depth: Depth of the queue | ||
| 45 | * @stride: Size of each element of the queue | ||
| 46 | * | ||
| 47 | * Return: status code | ||
| 48 | */ | ||
| 49 | int ionic_queue_init(struct ionic_queue *q, struct device *dma_dev, | ||
| 50 | int depth, size_t stride); | ||
| 51 | |||
| 52 | /** | ||
| 53 | * ionic_queue_destroy() - Destroy user space queue | ||
| 54 | * @q: Queue structure | ||
| 55 | * @dma_dev: DMA device for mapping | ||
| 56 | * | ||
| 57 | * Return: status code | ||
| 58 | */ | ||
| 59 | void ionic_queue_destroy(struct ionic_queue *q, struct device *dma_dev); | ||
| 60 | |||
| 61 | /** | ||
| 62 | * ionic_queue_empty() - Test if queue is empty | ||
| 63 | * @q: Queue structure | ||
| 64 | * | ||
| 65 | * This is only valid for to-device queues. | ||
| 66 | * | ||
| 67 | * Return: is empty | ||
| 68 | */ | ||
| 69 | static inline bool ionic_queue_empty(struct ionic_queue *q) | ||
| 70 | { | ||
| 71 | return q->prod == q->cons; | ||
| 72 | } | ||
| 73 | |||
| 74 | /** | ||
| 75 | * ionic_queue_length() - Get the current length of the queue | ||
| 76 | * @q: Queue structure | ||
| 77 | * | ||
| 78 | * This is only valid for to-device queues. | ||
| 79 | * | ||
| 80 | * Return: length | ||
| 81 | */ | ||
| 82 | static inline u16 ionic_queue_length(struct ionic_queue *q) | ||
| 83 | { | ||
| 84 | return (q->prod - q->cons) & q->mask; | ||
| 85 | } | ||
| 86 | |||
| 87 | /** | ||
| 88 | * ionic_queue_length_remaining() - Get the remaining length of the queue | ||
| 89 | * @q: Queue structure | ||
| 90 | * | ||
| 91 | * This is only valid for to-device queues. | ||
| 92 | * | ||
| 93 | * Return: length remaining | ||
| 94 | */ | ||
| 95 | static inline u16 ionic_queue_length_remaining(struct ionic_queue *q) | ||
| 96 | { | ||
| 97 | return q->mask - ionic_queue_length(q); | ||
| 98 | } | ||
| 99 | |||
| 100 | /** | ||
| 101 | * ionic_queue_full() - Test if queue is full | ||
| 102 | * @q: Queue structure | ||
| 103 | * | ||
| 104 | * This is only valid for to-device queues. | ||
| 105 | * | ||
| 106 | * Return: is full | ||
| 107 | */ | ||
| 108 | static inline bool ionic_queue_full(struct ionic_queue *q) | ||
| 109 | { | ||
| 110 | return q->mask == ionic_queue_length(q); | ||
| 111 | } | ||
| 112 | |||
| 113 | /** | ||
| 114 | * ionic_color_wrap() - Flip the color if prod is wrapped | ||
| 115 | * @prod: Queue index just after advancing | ||
| 116 | * @color: Queue color just prior to advancing the index | ||
| 117 | * | ||
| 118 | * Return: color after advancing the index | ||
| 119 | */ | ||
| 120 | static inline bool ionic_color_wrap(u16 prod, bool color) | ||
| 121 | { | ||
| 122 | /* logical xor color with (prod == 0) */ | ||
| 123 | return color != (prod == 0); | ||
| 124 | } | ||
| 125 | |||
| 126 | /** | ||
| 127 | * ionic_queue_at() - Get the element at the given index | ||
| 128 | * @q: Queue structure | ||
| 129 | * @idx: Index in the queue | ||
| 130 | * | ||
| 131 | * The index must be within the bounds of the queue. It is not checked here. | ||
| 132 | * | ||
| 133 | * Return: pointer to element at index | ||
| 134 | */ | ||
| 135 | static inline void *ionic_queue_at(struct ionic_queue *q, u16 idx) | ||
| 136 | { | ||
| 137 | return q->ptr + ((unsigned long)idx << q->stride_log2); | ||
| 138 | } | ||
| 139 | |||
| 140 | /** | ||
| 141 | * ionic_queue_at_prod() - Get the element at the producer index | ||
| 142 | * @q: Queue structure | ||
| 143 | * | ||
| 144 | * Return: pointer to element at producer index | ||
| 145 | */ | ||
| 146 | static inline void *ionic_queue_at_prod(struct ionic_queue *q) | ||
| 147 | { | ||
| 148 | return ionic_queue_at(q, q->prod); | ||
| 149 | } | ||
| 150 | |||
| 151 | /** | ||
| 152 | * ionic_queue_at_cons() - Get the element at the consumer index | ||
| 153 | * @q: Queue structure | ||
| 154 | * | ||
| 155 | * Return: pointer to element at consumer index | ||
| 156 | */ | ||
| 157 | static inline void *ionic_queue_at_cons(struct ionic_queue *q) | ||
| 158 | { | ||
| 159 | return ionic_queue_at(q, q->cons); | ||
| 160 | } | ||
| 161 | |||
| 162 | /** | ||
| 163 | * ionic_queue_next() - Compute the next index | ||
| 164 | * @q: Queue structure | ||
| 165 | * @idx: Index | ||
| 166 | * | ||
| 167 | * Return: next index after idx | ||
| 168 | */ | ||
| 169 | static inline u16 ionic_queue_next(struct ionic_queue *q, u16 idx) | ||
| 170 | { | ||
| 171 | return (idx + 1) & q->mask; | ||
| 172 | } | ||
| 173 | |||
| 174 | /** | ||
| 175 | * ionic_queue_produce() - Increase the producer index | ||
| 176 | * @q: Queue structure | ||
| 177 | * | ||
| 178 | * Caller must ensure that the queue is not full. It is not checked here. | ||
| 179 | */ | ||
| 180 | static inline void ionic_queue_produce(struct ionic_queue *q) | ||
| 181 | { | ||
| 182 | q->prod = ionic_queue_next(q, q->prod); | ||
| 183 | } | ||
| 184 | |||
| 185 | /** | ||
| 186 | * ionic_queue_consume() - Increase the consumer index | ||
| 187 | * @q: Queue structure | ||
| 188 | * | ||
| 189 | * Caller must ensure that the queue is not empty. It is not checked here. | ||
| 190 | * | ||
| 191 | * This is only valid for to-device queues. | ||
| 192 | */ | ||
| 193 | static inline void ionic_queue_consume(struct ionic_queue *q) | ||
| 194 | { | ||
| 195 | q->cons = ionic_queue_next(q, q->cons); | ||
| 196 | } | ||
| 197 | |||
| 198 | /** | ||
| 199 | * ionic_queue_consume_entries() - Increase the consumer index by entries | ||
| 200 | * @q: Queue structure | ||
| 201 | * @entries: Number of entries to increment | ||
| 202 | * | ||
| 203 | * Caller must ensure that the queue is not empty. It is not checked here. | ||
| 204 | * | ||
| 205 | * This is only valid for to-device queues. | ||
| 206 | */ | ||
| 207 | static inline void ionic_queue_consume_entries(struct ionic_queue *q, | ||
| 208 | u16 entries) | ||
| 209 | { | ||
| 210 | q->cons = (q->cons + entries) & q->mask; | ||
| 211 | } | ||
| 212 | |||
| 213 | /** | ||
| 214 | * ionic_queue_dbell_init() - Initialize doorbell bits for queue id | ||
| 215 | * @q: Queue structure | ||
| 216 | * @qid: Queue identifying number | ||
| 217 | */ | ||
| 218 | static inline void ionic_queue_dbell_init(struct ionic_queue *q, u32 qid) | ||
| 219 | { | ||
| 220 | q->dbell = IONIC_DBELL_QID(qid); | ||
| 221 | } | ||
| 222 | |||
| 223 | /** | ||
| 224 | * ionic_queue_dbell_val() - Get current doorbell update value | ||
| 225 | * @q: Queue structure | ||
| 226 | * | ||
| 227 | * Return: current doorbell update value | ||
| 228 | */ | ||
| 229 | static inline u64 ionic_queue_dbell_val(struct ionic_queue *q) | ||
| 230 | { | ||
| 231 | return q->dbell | q->prod; | ||
| 232 | } | ||
| 233 | |||
| 234 | #endif /* _IONIC_QUEUE_H_ */ | ||
| @@ -0,0 +1,154 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 */ | ||
| 2 | /* Copyright (C) 2018-2025, Advanced Micro Devices, Inc. */ | ||
| 3 | |||
| 4 | #ifndef _IONIC_RES_H_ | ||
| 5 | #define _IONIC_RES_H_ | ||
| 6 | |||
| 7 | #include <linux/kernel.h> | ||
| 8 | #include <linux/idr.h> | ||
| 9 | |||
| 10 | /** | ||
| 11 | * struct ionic_resid_bits - Number allocator based on IDA | ||
| 12 | * | ||
| 13 | * @inuse: IDA handle | ||
| 14 | * @inuse_size: Highest ID limit for IDA | ||
| 15 | */ | ||
| 16 | struct ionic_resid_bits { | ||
| 17 | struct ida inuse; | ||
| 18 | unsigned int inuse_size; | ||
| 19 | }; | ||
| 20 | |||
| 21 | /** | ||
| 22 | * ionic_resid_init() - Initialize a resid allocator | ||
| 23 | * @resid: Uninitialized resid allocator | ||
| 24 | * @size: Capacity of the allocator | ||
| 25 | * | ||
| 26 | * Return: Zero on success, or negative error number | ||
| 27 | */ | ||
| 28 | static inline void ionic_resid_init(struct ionic_resid_bits *resid, | ||
| 29 | unsigned int size) | ||
| 30 | { | ||
| 31 | resid->inuse_size = size; | ||
| 32 | ida_init(&resid->inuse); | ||
| 33 | } | ||
| 34 | |||
| 35 | /** | ||
| 36 | * ionic_resid_destroy() - Destroy a resid allocator | ||
| 37 | * @resid: Resid allocator | ||
| 38 | */ | ||
| 39 | static inline void ionic_resid_destroy(struct ionic_resid_bits *resid) | ||
| 40 | { | ||
| 41 | ida_destroy(&resid->inuse); | ||
| 42 | } | ||
| 43 | |||
| 44 | /** | ||
| 45 | * ionic_resid_get_shared() - Allocate an available shared resource id | ||
| 46 | * @resid: Resid allocator | ||
| 47 | * @min: Smallest valid resource id | ||
| 48 | * @size: One after largest valid resource id | ||
| 49 | * | ||
| 50 | * Return: Resource id, or negative error number | ||
| 51 | */ | ||
| 52 | static inline int ionic_resid_get_shared(struct ionic_resid_bits *resid, | ||
| 53 | unsigned int min, | ||
| 54 | unsigned int size) | ||
| 55 | { | ||
| 56 | return ida_alloc_range(&resid->inuse, min, size - 1, GFP_KERNEL); | ||
| 57 | } | ||
| 58 | |||
| 59 | /** | ||
| 60 | * ionic_resid_get() - Allocate an available resource id | ||
| 61 | * @resid: Resid allocator | ||
| 62 | * | ||
| 63 | * Return: Resource id, or negative error number | ||
| 64 | */ | ||
| 65 | static inline int ionic_resid_get(struct ionic_resid_bits *resid) | ||
| 66 | { | ||
| 67 | return ionic_resid_get_shared(resid, 0, resid->inuse_size); | ||
| 68 | } | ||
| 69 | |||
| 70 | /** | ||
| 71 | * ionic_resid_put() - Free a resource id | ||
| 72 | * @resid: Resid allocator | ||
| 73 | * @id: Resource id | ||
| 74 | */ | ||
| 75 | static inline void ionic_resid_put(struct ionic_resid_bits *resid, int id) | ||
| 76 | { | ||
| 77 | ida_free(&resid->inuse, id); | ||
| 78 | } | ||
| 79 | |||
| 80 | /** | ||
| 81 | * ionic_bitid_to_qid() - Transform a resource bit index into a queue id | ||
| 82 | * @bitid: Bit index | ||
| 83 | * @qgrp_shift: Log2 number of queues per queue group | ||
| 84 | * @half_qid_shift: Log2 of half the total number of queues | ||
| 85 | * | ||
| 86 | * Return: Queue id | ||
| 87 | * | ||
| 88 | * Udma-constrained queues (QPs and CQs) are associated with their udma by | ||
| 89 | * queue group. Even queue groups are associated with udma0, and odd queue | ||
| 90 | * groups with udma1. | ||
| 91 | * | ||
| 92 | * For allocating queue ids, we want to arrange the bits into two halves, | ||
| 93 | * with the even queue groups of udma0 in the lower half of the bitset, | ||
| 94 | * and the odd queue groups of udma1 in the upper half of the bitset. | ||
| 95 | * Then, one or two calls of find_next_zero_bit can examine all the bits | ||
| 96 | * for queues of an entire udma. | ||
| 97 | * | ||
| 98 | * For example, assuming eight queue groups with qgrp qids per group: | ||
| 99 | * | ||
| 100 | * bitid 0*qgrp..1*qgrp-1 : qid 0*qgrp..1*qgrp-1 | ||
| 101 | * bitid 1*qgrp..2*qgrp-1 : qid 2*qgrp..3*qgrp-1 | ||
| 102 | * bitid 2*qgrp..3*qgrp-1 : qid 4*qgrp..5*qgrp-1 | ||
| 103 | * bitid 3*qgrp..4*qgrp-1 : qid 6*qgrp..7*qgrp-1 | ||
| 104 | * bitid 4*qgrp..5*qgrp-1 : qid 1*qgrp..2*qgrp-1 | ||
| 105 | * bitid 5*qgrp..6*qgrp-1 : qid 3*qgrp..4*qgrp-1 | ||
| 106 | * bitid 6*qgrp..7*qgrp-1 : qid 5*qgrp..6*qgrp-1 | ||
| 107 | * bitid 7*qgrp..8*qgrp-1 : qid 7*qgrp..8*qgrp-1 | ||
| 108 | * | ||
| 109 | * There are three important ranges of bits in the qid. There is the udma | ||
| 110 | * bit "U" at qgrp_shift, which is the least significant bit of the group | ||
| 111 | * index, and determines which udma a queue is associated with. | ||
| 112 | * The bits of lesser significance we can call the idx bits "I", which are | ||
| 113 | * the index of the queue within the group. The bits of greater significance | ||
| 114 | * we can call the grp bits "G", which are other bits of the group index that | ||
| 115 | * do not determine the udma. Those bits are just rearranged in the bit index | ||
| 116 | * in the bitset. A bitid has the udma bit in the most significant place, | ||
| 117 | * then the grp bits, then the idx bits. | ||
| 118 | * | ||
| 119 | * bitid: 00000000000000 U GGG IIIIII | ||
| 120 | * qid: 00000000000000 GGG U IIIIII | ||
| 121 | * | ||
| 122 | * Transforming from bit index to qid, or from qid to bit index, can be | ||
| 123 | * accomplished by rearranging the bits by masking and shifting. | ||
| 124 | */ | ||
| 125 | static inline u32 ionic_bitid_to_qid(u32 bitid, u8 qgrp_shift, | ||
| 126 | u8 half_qid_shift) | ||
| 127 | { | ||
| 128 | u32 udma_bit = | ||
| 129 | (bitid & BIT(half_qid_shift)) >> (half_qid_shift - qgrp_shift); | ||
| 130 | u32 grp_bits = (bitid & GENMASK(half_qid_shift - 1, qgrp_shift)) << 1; | ||
| 131 | u32 idx_bits = bitid & (BIT(qgrp_shift) - 1); | ||
| 132 | |||
| 133 | return grp_bits | udma_bit | idx_bits; | ||
| 134 | } | ||
| 135 | |||
| 136 | /** | ||
| 137 | * ionic_qid_to_bitid() - Transform a queue id into a resource bit index | ||
| 138 | * @qid: queue index | ||
| 139 | * @qgrp_shift: Log2 number of queues per queue group | ||
| 140 | * @half_qid_shift: Log2 of half the total number of queues | ||
| 141 | * | ||
| 142 | * Return: Resource bit index | ||
| 143 | * | ||
| 144 | * This is the inverse of ionic_bitid_to_qid(). | ||
| 145 | */ | ||
| 146 | static inline u32 ionic_qid_to_bitid(u32 qid, u8 qgrp_shift, u8 half_qid_shift) | ||
| 147 | { | ||
| 148 | u32 udma_bit = (qid & BIT(qgrp_shift)) << (half_qid_shift - qgrp_shift); | ||
| 149 | u32 grp_bits = (qid & GENMASK(half_qid_shift, qgrp_shift + 1)) >> 1; | ||
| 150 | u32 idx_bits = qid & (BIT(qgrp_shift) - 1); | ||
| 151 | |||
| 152 | return udma_bit | grp_bits | idx_bits; | ||
| 153 | } | ||
| 154 | #endif /* _IONIC_RES_H_ */ | ||
| @@ -4,10 +4,11 @@ config INFINIBAND_IRDMA · 3 unmodified lines | |||
| 4 | depends on INET | 4 | depends on INET |
| 5 | depends on IPV6 || !IPV6 | 5 | depends on IPV6 || !IPV6 |
| 6 | depends on PCI | 6 | depends on PCI |
| 7 | depends on ICE && I40E | 7 | depends on IDPF && ICE && I40E |
| 8 | select GENERIC_ALLOCATOR | 8 | select GENERIC_ALLOCATOR |
| 9 | select AUXILIARY_BUS | 9 | select AUXILIARY_BUS |
| 10 | select CRC32 | 10 | select CRC32 |
| 11 | help | 11 | help |
| 12 | This is an Intel(R) Ethernet Protocol Driver for RDMA driver | 12 | This is an Intel(R) Ethernet Protocol Driver for RDMA that |
| 13 | that support E810 (iWARP/RoCE) and X722 (iWARP) network devices. | 13 | supports IPU E2000 (RoCEv2), E810 (iWARP/RoCEv2) and X722 (iWARP) |
| 14 | network devices. | ||
| @@ -13,7 +13,10 @@ irdma-objs := cm.o \ · 12 unmodified lines | |||
| 13 | hw.o \ | 13 | hw.o \ |
| 14 | i40iw_hw.o \ | 14 | i40iw_hw.o \ |
| 15 | i40iw_if.o \ | 15 | i40iw_if.o \ |
| 16 | ig3rdma_if.o\ | ||
| 17 | icrdma_if.o \ | ||
| 16 | icrdma_hw.o \ | 18 | icrdma_hw.o \ |
| 19 | ig3rdma_hw.o\ | ||
| 17 | main.o \ | 20 | main.o \ |
| 18 | pble.o \ | 21 | pble.o \ |
| 19 | puda.o \ | 22 | puda.o \ |
| @@ -22,6 +25,7 @@ irdma-objs := cm.o \ · 2 unmodified lines | |||
| 22 | uk.o \ | 25 | uk.o \ |
| 23 | utils.o \ | 26 | utils.o \ |
| 24 | verbs.o \ | 27 | verbs.o \ |
| 28 | virtchnl.o \ | ||
| 25 | ws.o \ | 29 | ws.o \ |
| 26 | 30 | ||
| 27 | CFLAGS_trace.o = -I$(src) | 31 | CFLAGS_trace.o = -I$(src) |
| @@ -74,6 +74,14 @@ static void irdma_set_qos_info(struct irdma_sc_vsi *vsi, · 73 unmodified lines | |||
| 74 | { | 74 | { |
| 75 | u8 i; | 75 | u8 i; |
| 76 | 76 | ||
| 77 | if (vsi->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) { | ||
| 78 | for (i = 0; i < IRDMA_MAX_USER_PRIORITY; i++) { | ||
| 79 | vsi->qos[i].qs_handle = vsi->dev->qos[i].qs_handle; | ||
| 80 | vsi->qos[i].valid = true; | ||
| 81 | } | ||
| 82 | |||
| 83 | return; | ||
| 84 | } | ||
| 77 | vsi->qos_rel_bw = l2p->vsi_rel_bw; | 85 | vsi->qos_rel_bw = l2p->vsi_rel_bw; |
| 78 | vsi->qos_prio_type = l2p->vsi_prio_type; | 86 | vsi->qos_prio_type = l2p->vsi_prio_type; |
| 79 | vsi->dscp_mode = l2p->dscp_mode; | 87 | vsi->dscp_mode = l2p->dscp_mode; |
| @@ -404,7 +412,8 @@ int irdma_sc_qp_init(struct irdma_sc_qp *qp, struct irdma_qp_init_info *info) · 324 unmodified lines | |||
| 404 | pble_obj_cnt = info->pd->dev->hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].cnt; | 412 | pble_obj_cnt = info->pd->dev->hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].cnt; |
| 405 | 413 | ||
| 406 | if ((info->virtual_map && info->sq_pa >= pble_obj_cnt) || | 414 | if ((info->virtual_map && info->sq_pa >= pble_obj_cnt) || |
| 407 | (info->virtual_map && info->rq_pa >= pble_obj_cnt)) | 415 | (!info->qp_uk_init_info.srq_uk && |
| 416 | info->virtual_map && info->rq_pa >= pble_obj_cnt)) | ||
| 408 | return -EINVAL; | 417 | return -EINVAL; |
| 409 | 418 | ||
| 410 | qp->llp_stream_handle = (void *)(-1); | 419 | qp->llp_stream_handle = (void *)(-1); |
| @@ -438,6 +447,208 @@ int irdma_sc_qp_init(struct irdma_sc_qp *qp, struct irdma_qp_init_info *info) · 27 unmodified lines | |||
| 438 | return 0; | 447 | return 0; |
| 439 | } | 448 | } |
| 440 | 449 | ||
| 450 | /** | ||
| 451 | * irdma_sc_srq_init - init sc_srq structure | ||
| 452 | * @srq: srq sc struct | ||
| 453 | * @info: parameters for srq init | ||
| 454 | */ | ||
| 455 | int irdma_sc_srq_init(struct irdma_sc_srq *srq, | ||
| 456 | struct irdma_srq_init_info *info) | ||
| 457 | { | ||
| 458 | u32 srq_size_quanta; | ||
| 459 | int ret_code; | ||
| 460 | |||
| 461 | ret_code = irdma_uk_srq_init(&srq->srq_uk, &info->srq_uk_init_info); | ||
| 462 | if (ret_code) | ||
| 463 | return ret_code; | ||
| 464 | |||
| 465 | srq->dev = info->pd->dev; | ||
| 466 | srq->pd = info->pd; | ||
| 467 | srq->vsi = info->vsi; | ||
| 468 | srq->srq_pa = info->srq_pa; | ||
| 469 | srq->first_pm_pbl_idx = info->first_pm_pbl_idx; | ||
| 470 | srq->pasid = info->pasid; | ||
| 471 | srq->pasid_valid = info->pasid_valid; | ||
| 472 | srq->srq_limit = info->srq_limit; | ||
| 473 | srq->leaf_pbl_size = info->leaf_pbl_size; | ||
| 474 | srq->virtual_map = info->virtual_map; | ||
| 475 | srq->tph_en = info->tph_en; | ||
| 476 | srq->arm_limit_event = info->arm_limit_event; | ||
| 477 | srq->tph_val = info->tph_value; | ||
| 478 | srq->shadow_area_pa = info->shadow_area_pa; | ||
| 479 | |||
| 480 | /* Smallest SRQ size is 256B i.e. 8 quanta */ | ||
| 481 | srq_size_quanta = max((u32)IRDMA_SRQ_MIN_QUANTA, | ||
| 482 | srq->srq_uk.srq_size * | ||
| 483 | srq->srq_uk.wqe_size_multiplier); | ||
| 484 | srq->hw_srq_size = irdma_get_encoded_wqe_size(srq_size_quanta, | ||
| 485 | IRDMA_QUEUE_TYPE_SRQ); | ||
| 486 | |||
| 487 | return 0; | ||
| 488 | } | ||
| 489 | |||
| 490 | /** | ||
| 491 | * irdma_sc_srq_create - send srq create CQP WQE | ||
| 492 | * @srq: srq sc struct | ||
| 493 | * @scratch: u64 saved to be used during cqp completion | ||
| 494 | * @post_sq: flag for cqp db to ring | ||
| 495 | */ | ||
| 496 | static int irdma_sc_srq_create(struct irdma_sc_srq *srq, u64 scratch, | ||
| 497 | bool post_sq) | ||
| 498 | { | ||
| 499 | struct irdma_sc_cqp *cqp; | ||
| 500 | __le64 *wqe; | ||
| 501 | u64 hdr; | ||
| 502 | |||
| 503 | cqp = srq->pd->dev->cqp; | ||
| 504 | if (srq->srq_uk.srq_id < cqp->dev->hw_attrs.min_hw_srq_id || | ||
| 505 | srq->srq_uk.srq_id > | ||
| 506 | (cqp->dev->hmc_info->hmc_obj[IRDMA_HMC_IW_SRQ].max_cnt - 1)) | ||
| 507 | return -EINVAL; | ||
| 508 | |||
| 509 | wqe = irdma_sc_cqp_get_next_send_wqe(cqp, scratch); | ||
| 510 | if (!wqe) | ||
| 511 | return -ENOMEM; | ||
| 512 | |||
| 513 | set_64bit_val(wqe, 0, | ||
| 514 | FIELD_PREP(IRDMA_CQPSQ_SRQ_SRQ_LIMIT, srq->srq_limit) | | ||
| 515 | FIELD_PREP(IRDMA_CQPSQ_SRQ_RQSIZE, srq->hw_srq_size) | | ||
| 516 | FIELD_PREP(IRDMA_CQPSQ_SRQ_RQ_WQE_SIZE, srq->srq_uk.wqe_size)); | ||
| 517 | set_64bit_val(wqe, 8, (uintptr_t)srq); | ||
| 518 | set_64bit_val(wqe, 16, | ||
| 519 | FIELD_PREP(IRDMA_CQPSQ_SRQ_PD_ID, srq->pd->pd_id)); | ||
| 520 | set_64bit_val(wqe, 32, | ||
| 521 | FIELD_PREP(IRDMA_CQPSQ_SRQ_PHYSICAL_BUFFER_ADDR, | ||
| 522 | srq->srq_pa >> | ||
| 523 | IRDMA_CQPSQ_SRQ_PHYSICAL_BUFFER_ADDR_S)); | ||
| 524 | set_64bit_val(wqe, 40, | ||
| 525 | FIELD_PREP(IRDMA_CQPSQ_SRQ_DB_SHADOW_ADDR, | ||
| 526 | srq->shadow_area_pa >> | ||
| 527 | IRDMA_CQPSQ_SRQ_DB_SHADOW_ADDR_S)); | ||
| 528 | set_64bit_val(wqe, 48, | ||
| 529 | FIELD_PREP(IRDMA_CQPSQ_SRQ_FIRST_PM_PBL_IDX, | ||
| 530 | srq->first_pm_pbl_idx)); | ||
| 531 | |||
| 532 | hdr = srq->srq_uk.srq_id | | ||
| 533 | FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_CREATE_SRQ) | | ||
| 534 | FIELD_PREP(IRDMA_CQPSQ_SRQ_LEAF_PBL_SIZE, srq->leaf_pbl_size) | | ||
| 535 | FIELD_PREP(IRDMA_CQPSQ_SRQ_VIRTMAP, srq->virtual_map) | | ||
| 536 | FIELD_PREP(IRDMA_CQPSQ_SRQ_ARM_LIMIT_EVENT, | ||
| 537 | srq->arm_limit_event) | | ||
| 538 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); | ||
| 539 | |||
| 540 | dma_wmb(); /* make sure WQE is written before valid bit is set */ | ||
| 541 | |||
| 542 | set_64bit_val(wqe, 24, hdr); | ||
| 543 | |||
| 544 | print_hex_dump_debug("WQE: SRQ_CREATE WQE", DUMP_PREFIX_OFFSET, 16, 8, | ||
| 545 | wqe, IRDMA_CQP_WQE_SIZE * 8, false); | ||
| 546 | if (post_sq) | ||
| 547 | irdma_sc_cqp_post_sq(cqp); | ||
| 548 | |||
| 549 | return 0; | ||
| 550 | } | ||
| 551 | |||
| 552 | /** | ||
| 553 | * irdma_sc_srq_modify - send modify_srq CQP WQE | ||
| 554 | * @srq: srq sc struct | ||
| 555 | * @info: parameters for srq modification | ||
| 556 | * @scratch: u64 saved to be used during cqp completion | ||
| 557 | * @post_sq: flag for cqp db to ring | ||
| 558 | */ | ||
| 559 | static int irdma_sc_srq_modify(struct irdma_sc_srq *srq, | ||
| 560 | struct irdma_modify_srq_info *info, u64 scratch, | ||
| 561 | bool post_sq) | ||
| 562 | { | ||
| 563 | struct irdma_sc_cqp *cqp; | ||
| 564 | __le64 *wqe; | ||
| 565 | u64 hdr; | ||
| 566 | |||
| 567 | cqp = srq->dev->cqp; | ||
| 568 | if (srq->srq_uk.srq_id < cqp->dev->hw_attrs.min_hw_srq_id || | ||
| 569 | srq->srq_uk.srq_id > | ||
| 570 | (cqp->dev->hmc_info->hmc_obj[IRDMA_HMC_IW_SRQ].max_cnt - 1)) | ||
| 571 | return -EINVAL; | ||
| 572 | |||
| 573 | wqe = irdma_sc_cqp_get_next_send_wqe(cqp, scratch); | ||
| 574 | if (!wqe) | ||
| 575 | return -ENOMEM; | ||
| 576 | |||
| 577 | set_64bit_val(wqe, 0, | ||
| 578 | FIELD_PREP(IRDMA_CQPSQ_SRQ_SRQ_LIMIT, info->srq_limit) | | ||
| 579 | FIELD_PREP(IRDMA_CQPSQ_SRQ_RQSIZE, srq->hw_srq_size) | | ||
| 580 | FIELD_PREP(IRDMA_CQPSQ_SRQ_RQ_WQE_SIZE, srq->srq_uk.wqe_size)); | ||
| 581 | set_64bit_val(wqe, 8, | ||
| 582 | FIELD_PREP(IRDMA_CQPSQ_SRQ_SRQCTX, srq->srq_uk.srq_id)); | ||
| 583 | set_64bit_val(wqe, 16, | ||
| 584 | FIELD_PREP(IRDMA_CQPSQ_SRQ_PD_ID, srq->pd->pd_id)); | ||
| 585 | set_64bit_val(wqe, 32, | ||
| 586 | FIELD_PREP(IRDMA_CQPSQ_SRQ_PHYSICAL_BUFFER_ADDR, | ||
| 587 | srq->srq_pa >> | ||
| 588 | IRDMA_CQPSQ_SRQ_PHYSICAL_BUFFER_ADDR_S)); | ||
| 589 | set_64bit_val(wqe, 40, | ||
| 590 | FIELD_PREP(IRDMA_CQPSQ_SRQ_DB_SHADOW_ADDR, | ||
| 591 | srq->shadow_area_pa >> | ||
| 592 | IRDMA_CQPSQ_SRQ_DB_SHADOW_ADDR_S)); | ||
| 593 | set_64bit_val(wqe, 48, | ||
| 594 | FIELD_PREP(IRDMA_CQPSQ_SRQ_FIRST_PM_PBL_IDX, | ||
| 595 | srq->first_pm_pbl_idx)); | ||
| 596 | |||
| 597 | hdr = srq->srq_uk.srq_id | | ||
| 598 | FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_MODIFY_SRQ) | | ||
| 599 | FIELD_PREP(IRDMA_CQPSQ_SRQ_LEAF_PBL_SIZE, srq->leaf_pbl_size) | | ||
| 600 | FIELD_PREP(IRDMA_CQPSQ_SRQ_VIRTMAP, srq->virtual_map) | | ||
| 601 | FIELD_PREP(IRDMA_CQPSQ_SRQ_ARM_LIMIT_EVENT, | ||
| 602 | info->arm_limit_event) | | ||
| 603 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); | ||
| 604 | dma_wmb(); /* make sure WQE is written before valid bit is set */ | ||
| 605 | |||
| 606 | set_64bit_val(wqe, 24, hdr); | ||
| 607 | |||
| 608 | print_hex_dump_debug("WQE: SRQ_MODIFY WQE", DUMP_PREFIX_OFFSET, 16, 8, | ||
| 609 | wqe, IRDMA_CQP_WQE_SIZE * 8, false); | ||
| 610 | if (post_sq) | ||
| 611 | irdma_sc_cqp_post_sq(cqp); | ||
| 612 | |||
| 613 | return 0; | ||
| 614 | } | ||
| 615 | |||
| 616 | /** | ||
| 617 | * irdma_sc_srq_destroy - send srq_destroy CQP WQE | ||
| 618 | * @srq: srq sc struct | ||
| 619 | * @scratch: u64 saved to be used during cqp completion | ||
| 620 | * @post_sq: flag for cqp db to ring | ||
| 621 | */ | ||
| 622 | static int irdma_sc_srq_destroy(struct irdma_sc_srq *srq, u64 scratch, | ||
| 623 | bool post_sq) | ||
| 624 | { | ||
| 625 | struct irdma_sc_cqp *cqp; | ||
| 626 | __le64 *wqe; | ||
| 627 | u64 hdr; | ||
| 628 | |||
| 629 | cqp = srq->dev->cqp; | ||
| 630 | |||
| 631 | wqe = irdma_sc_cqp_get_next_send_wqe(cqp, scratch); | ||
| 632 | if (!wqe) | ||
| 633 | return -ENOMEM; | ||
| 634 | |||
| 635 | set_64bit_val(wqe, 8, (uintptr_t)srq); | ||
| 636 | |||
| 637 | hdr = srq->srq_uk.srq_id | | ||
| 638 | FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_DESTROY_SRQ) | | ||
| 639 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); | ||
| 640 | dma_wmb(); /* make sure WQE is written before valid bit is set */ | ||
| 641 | |||
| 642 | set_64bit_val(wqe, 24, hdr); | ||
| 643 | |||
| 644 | print_hex_dump_debug("WQE: SRQ_DESTROY WQE", DUMP_PREFIX_OFFSET, 16, | ||
| 645 | 8, wqe, IRDMA_CQP_WQE_SIZE * 8, false); | ||
| 646 | if (post_sq) | ||
| 647 | irdma_sc_cqp_post_sq(cqp); | ||
| 648 | |||
| 649 | return 0; | ||
| 650 | } | ||
| 651 | |||
| 441 | /** | 652 | /** |
| 442 | * irdma_sc_qp_create - create qp | 653 | * irdma_sc_qp_create - create qp |
| 443 | * @qp: sc qp | 654 | * @qp: sc qp |
| @@ -629,13 +840,14 @@ static u8 irdma_sc_get_encoded_ird_size(u16 ird_size) · 185 unmodified lines | |||
| 629 | } | 840 | } |
| 630 | 841 | ||
| 631 | /** | 842 | /** |
| 632 | * irdma_sc_qp_setctx_roce - set qp's context | 843 | * irdma_sc_qp_setctx_roce_gen_2 - set qp's context |
| 633 | * @qp: sc qp | 844 | * @qp: sc qp |
| 634 | * @qp_ctx: context ptr | 845 | * @qp_ctx: context ptr |
| 635 | * @info: ctx info | 846 | * @info: ctx info |
| 636 | */ | 847 | */ |
| 637 | void irdma_sc_qp_setctx_roce(struct irdma_sc_qp *qp, __le64 *qp_ctx, | 848 | static void irdma_sc_qp_setctx_roce_gen_2(struct irdma_sc_qp *qp, |
| 638 | struct irdma_qp_host_ctx_info *info) | 849 | __le64 *qp_ctx, |
| 850 | struct irdma_qp_host_ctx_info *info) | ||
| 639 | { | 851 | { |
| 640 | struct irdma_roce_offload_info *roce_info; | 852 | struct irdma_roce_offload_info *roce_info; |
| 641 | struct irdma_udp_offload_info *udp; | 853 | struct irdma_udp_offload_info *udp; |
| @@ -753,6 +965,189 @@ void irdma_sc_qp_setctx_roce(struct irdma_sc_qp *qp, __le64 *qp_ctx, · 111 unmodified lines | |||
| 753 | 8, qp_ctx, IRDMA_QP_CTX_SIZE, false); | 965 | 8, qp_ctx, IRDMA_QP_CTX_SIZE, false); |
| 754 | } | 966 | } |
| 755 | 967 | ||
| 968 | /** | ||
| 969 | * irdma_sc_get_encoded_ird_size_gen_3 - get encoded IRD size for GEN 3 | ||
| 970 | * @ird_size: IRD size | ||
| 971 | * The ird from the connection is rounded to a supported HW setting and then encoded | ||
| 972 | * for ird_size field of qp_ctx. Consumers are expected to provide valid ird size based | ||
| 973 | * on hardware attributes. IRD size defaults to a value of 4 in case of invalid input. | ||
| 974 | */ | ||
| 975 | static u8 irdma_sc_get_encoded_ird_size_gen_3(u16 ird_size) | ||
| 976 | { | ||
| 977 | switch (ird_size ? | ||
| 978 | roundup_pow_of_two(2 * ird_size) : 4) { | ||
| 979 | case 4096: | ||
| 980 | return IRDMA_IRD_HW_SIZE_4096_GEN3; | ||
| 981 | case 2048: | ||
| 982 | return IRDMA_IRD_HW_SIZE_2048_GEN3; | ||
| 983 | case 1024: | ||
| 984 | return IRDMA_IRD_HW_SIZE_1024_GEN3; | ||
| 985 | case 512: | ||
| 986 | return IRDMA_IRD_HW_SIZE_512_GEN3; | ||
| 987 | case 256: | ||
| 988 | return IRDMA_IRD_HW_SIZE_256_GEN3; | ||
| 989 | case 128: | ||
| 990 | return IRDMA_IRD_HW_SIZE_128_GEN3; | ||
| 991 | case 64: | ||
| 992 | return IRDMA_IRD_HW_SIZE_64_GEN3; | ||
| 993 | case 32: | ||
| 994 | return IRDMA_IRD_HW_SIZE_32_GEN3; | ||
| 995 | case 16: | ||
| 996 | return IRDMA_IRD_HW_SIZE_16_GEN3; | ||
| 997 | case 8: | ||
| 998 | return IRDMA_IRD_HW_SIZE_8_GEN3; | ||
| 999 | case 4: | ||
| 1000 | default: | ||
| 1001 | break; | ||
| 1002 | } | ||
| 1003 | |||
| 1004 | return IRDMA_IRD_HW_SIZE_4_GEN3; | ||
| 1005 | } | ||
| 1006 | |||
| 1007 | /** | ||
| 1008 | * irdma_sc_qp_setctx_roce_gen_3 - set qp's context | ||
| 1009 | * @qp: sc qp | ||
| 1010 | * @qp_ctx: context ptr | ||
| 1011 | * @info: ctx info | ||
| 1012 | */ | ||
| 1013 | static void irdma_sc_qp_setctx_roce_gen_3(struct irdma_sc_qp *qp, | ||
| 1014 | __le64 *qp_ctx, | ||
| 1015 | struct irdma_qp_host_ctx_info *info) | ||
| 1016 | { | ||
| 1017 | struct irdma_roce_offload_info *roce_info = info->roce_info; | ||
| 1018 | struct irdma_udp_offload_info *udp = info->udp_info; | ||
| 1019 | u64 qw0, qw3, qw7 = 0, qw8 = 0; | ||
| 1020 | u8 push_mode_en; | ||
| 1021 | u32 push_idx; | ||
| 1022 | |||
| 1023 | qp->user_pri = info->user_pri; | ||
| 1024 | if (qp->push_idx == IRDMA_INVALID_PUSH_PAGE_INDEX) { | ||
| 1025 | push_mode_en = 0; | ||
| 1026 | push_idx = 0; | ||
| 1027 | } else { | ||
| 1028 | push_mode_en = 1; | ||
| 1029 | push_idx = qp->push_idx; | ||
| 1030 | } | ||
| 1031 | |||
| 1032 | qw0 = FIELD_PREP(IRDMAQPC_RQWQESIZE, qp->qp_uk.rq_wqe_size) | | ||
| 1033 | FIELD_PREP(IRDMAQPC_RCVTPHEN, qp->rcv_tph_en) | | ||
| 1034 | FIELD_PREP(IRDMAQPC_XMITTPHEN, qp->xmit_tph_en) | | ||
| 1035 | FIELD_PREP(IRDMAQPC_RQTPHEN, qp->rq_tph_en) | | ||
| 1036 | FIELD_PREP(IRDMAQPC_SQTPHEN, qp->sq_tph_en) | | ||
| 1037 | FIELD_PREP(IRDMAQPC_PPIDX, push_idx) | | ||
| 1038 | FIELD_PREP(IRDMAQPC_PMENA, push_mode_en) | | ||
| 1039 | FIELD_PREP(IRDMAQPC_DC_TCP_EN, roce_info->dctcp_en) | | ||
| 1040 | FIELD_PREP(IRDMAQPC_ISQP1, roce_info->is_qp1) | | ||
| 1041 | FIELD_PREP(IRDMAQPC_ROCE_TVER, roce_info->roce_tver) | | ||
| 1042 | FIELD_PREP(IRDMAQPC_IPV4, udp->ipv4) | | ||
| 1043 | FIELD_PREP(IRDMAQPC_USE_SRQ, !qp->qp_uk.srq_uk ? 0 : 1) | | ||
| 1044 | FIELD_PREP(IRDMAQPC_INSERTVLANTAG, udp->insert_vlan_tag); | ||
| 1045 | set_64bit_val(qp_ctx, 0, qw0); | ||
| 1046 | set_64bit_val(qp_ctx, 8, qp->sq_pa); | ||
| 1047 | set_64bit_val(qp_ctx, 16, qp->rq_pa); | ||
| 1048 | qw3 = FIELD_PREP(IRDMAQPC_RQSIZE, qp->hw_rq_size) | | ||
| 1049 | FIELD_PREP(IRDMAQPC_SQSIZE, qp->hw_sq_size) | | ||
| 1050 | FIELD_PREP(IRDMAQPC_TTL, udp->ttl) | | ||
| 1051 | FIELD_PREP(IRDMAQPC_TOS, udp->tos) | | ||
| 1052 | FIELD_PREP(IRDMAQPC_SRCPORTNUM, udp->src_port) | | ||
| 1053 | FIELD_PREP(IRDMAQPC_DESTPORTNUM, udp->dst_port); | ||
| 1054 | set_64bit_val(qp_ctx, 24, qw3); | ||
| 1055 | set_64bit_val(qp_ctx, 32, | ||
| 1056 | FIELD_PREP(IRDMAQPC_DESTIPADDR2, udp->dest_ip_addr[2]) | | ||
| 1057 | FIELD_PREP(IRDMAQPC_DESTIPADDR3, udp->dest_ip_addr[3])); | ||
| 1058 | set_64bit_val(qp_ctx, 40, | ||
| 1059 | FIELD_PREP(IRDMAQPC_DESTIPADDR0, udp->dest_ip_addr[0]) | | ||
| 1060 | FIELD_PREP(IRDMAQPC_DESTIPADDR1, udp->dest_ip_addr[1])); | ||
| 1061 | set_64bit_val(qp_ctx, 48, | ||
| 1062 | FIELD_PREP(IRDMAQPC_SNDMSS, udp->snd_mss) | | ||
| 1063 | FIELD_PREP(IRDMAQPC_VLANTAG, udp->vlan_tag) | | ||
| 1064 | FIELD_PREP(IRDMAQPC_ARPIDX, udp->arp_idx)); | ||
| 1065 | qw7 = FIELD_PREP(IRDMAQPC_PKEY, roce_info->p_key) | | ||
| 1066 | FIELD_PREP(IRDMAQPC_ACKCREDITS, roce_info->ack_credits) | | ||
| 1067 | FIELD_PREP(IRDMAQPC_FLOWLABEL, udp->flow_label); | ||
| 1068 | set_64bit_val(qp_ctx, 56, qw7); | ||
| 1069 | qw8 = FIELD_PREP(IRDMAQPC_QKEY, roce_info->qkey) | | ||
| 1070 | FIELD_PREP(IRDMAQPC_DESTQP, roce_info->dest_qp); | ||
| 1071 | set_64bit_val(qp_ctx, 64, qw8); | ||
| 1072 | set_64bit_val(qp_ctx, 80, | ||
| 1073 | FIELD_PREP(IRDMAQPC_PSNNXT, udp->psn_nxt) | | ||
| 1074 | FIELD_PREP(IRDMAQPC_LSN, udp->lsn)); | ||
| 1075 | set_64bit_val(qp_ctx, 88, | ||
| 1076 | FIELD_PREP(IRDMAQPC_EPSN, udp->epsn)); | ||
| 1077 | set_64bit_val(qp_ctx, 96, | ||
| 1078 | FIELD_PREP(IRDMAQPC_PSNMAX, udp->psn_max) | | ||
| 1079 | FIELD_PREP(IRDMAQPC_PSNUNA, udp->psn_una)); | ||
| 1080 | set_64bit_val(qp_ctx, 112, | ||
| 1081 | FIELD_PREP(IRDMAQPC_CWNDROCE, udp->cwnd)); | ||
| 1082 | set_64bit_val(qp_ctx, 128, | ||
| 1083 | FIELD_PREP(IRDMAQPC_MINRNR_TIMER, udp->min_rnr_timer) | | ||
| 1084 | FIELD_PREP(IRDMAQPC_RNRNAK_THRESH, udp->rnr_nak_thresh) | | ||
| 1085 | FIELD_PREP(IRDMAQPC_REXMIT_THRESH, udp->rexmit_thresh) | | ||
| 1086 | FIELD_PREP(IRDMAQPC_RNRNAK_TMR, udp->rnr_nak_tmr) | | ||
| 1087 | FIELD_PREP(IRDMAQPC_RTOMIN, roce_info->rtomin)); | ||
| 1088 | set_64bit_val(qp_ctx, 136, | ||
| 1089 | FIELD_PREP(IRDMAQPC_TXCQNUM, info->send_cq_num) | | ||
| 1090 | FIELD_PREP(IRDMAQPC_RXCQNUM, info->rcv_cq_num)); | ||
| 1091 | set_64bit_val(qp_ctx, 152, | ||
| 1092 | FIELD_PREP(IRDMAQPC_MACADDRESS, | ||
| 1093 | ether_addr_to_u64(roce_info->mac_addr)) | | ||
| 1094 | FIELD_PREP(IRDMAQPC_LOCALACKTIMEOUT, | ||
| 1095 | roce_info->local_ack_timeout)); | ||
| 1096 | set_64bit_val(qp_ctx, 160, | ||
| 1097 | FIELD_PREP(IRDMAQPC_ORDSIZE_GEN3, roce_info->ord_size) | | ||
| 1098 | FIELD_PREP(IRDMAQPC_IRDSIZE_GEN3, | ||
| 1099 | irdma_sc_get_encoded_ird_size_gen_3(roce_info->ird_size)) | | ||
| 1100 | FIELD_PREP(IRDMAQPC_WRRDRSPOK, roce_info->wr_rdresp_en) | | ||
| 1101 | FIELD_PREP(IRDMAQPC_RDOK, roce_info->rd_en) | | ||
| 1102 | FIELD_PREP(IRDMAQPC_USESTATSINSTANCE, | ||
| 1103 | info->stats_idx_valid) | | ||
| 1104 | FIELD_PREP(IRDMAQPC_BINDEN, roce_info->bind_en) | | ||
| 1105 | FIELD_PREP(IRDMAQPC_FASTREGEN, roce_info->fast_reg_en) | | ||
| 1106 | FIELD_PREP(IRDMAQPC_DCQCNENABLE, roce_info->dcqcn_en) | | ||
| 1107 | FIELD_PREP(IRDMAQPC_RCVNOICRC, roce_info->rcv_no_icrc) | | ||
| 1108 | FIELD_PREP(IRDMAQPC_FW_CC_ENABLE, | ||
| 1109 | roce_info->fw_cc_enable) | | ||
| 1110 | FIELD_PREP(IRDMAQPC_UDPRIVCQENABLE, | ||
| 1111 | roce_info->udprivcq_en) | | ||
| 1112 | FIELD_PREP(IRDMAQPC_PRIVEN, roce_info->priv_mode_en) | | ||
| 1113 | FIELD_PREP(IRDMAQPC_REMOTE_ATOMIC_EN, | ||
| 1114 | info->remote_atomics_en) | | ||
| 1115 | FIELD_PREP(IRDMAQPC_TIMELYENABLE, roce_info->timely_en)); | ||
| 1116 | set_64bit_val(qp_ctx, 168, | ||
| 1117 | FIELD_PREP(IRDMAQPC_QPCOMPCTX, info->qp_compl_ctx)); | ||
| 1118 | set_64bit_val(qp_ctx, 176, | ||
| 1119 | FIELD_PREP(IRDMAQPC_SQTPHVAL, qp->sq_tph_val) | | ||
| 1120 | FIELD_PREP(IRDMAQPC_RQTPHVAL, qp->rq_tph_val) | | ||
| 1121 | FIELD_PREP(IRDMAQPC_QSHANDLE, qp->qs_handle)); | ||
| 1122 | set_64bit_val(qp_ctx, 184, | ||
| 1123 | FIELD_PREP(IRDMAQPC_LOCAL_IPADDR3, udp->local_ipaddr[3]) | | ||
| 1124 | FIELD_PREP(IRDMAQPC_LOCAL_IPADDR2, udp->local_ipaddr[2])); | ||
| 1125 | set_64bit_val(qp_ctx, 192, | ||
| 1126 | FIELD_PREP(IRDMAQPC_LOCAL_IPADDR1, udp->local_ipaddr[1]) | | ||
| 1127 | FIELD_PREP(IRDMAQPC_LOCAL_IPADDR0, udp->local_ipaddr[0])); | ||
| 1128 | set_64bit_val(qp_ctx, 200, | ||
| 1129 | FIELD_PREP(IRDMAQPC_THIGH, roce_info->t_high) | | ||
| 1130 | FIELD_PREP(IRDMAQPC_SRQ_ID, | ||
| 1131 | !qp->qp_uk.srq_uk ? | ||
| 1132 | 0 : qp->qp_uk.srq_uk->srq_id) | | ||
| 1133 | FIELD_PREP(IRDMAQPC_TLOW, roce_info->t_low)); | ||
| 1134 | set_64bit_val(qp_ctx, 208, roce_info->pd_id | | ||
| 1135 | FIELD_PREP(IRDMAQPC_STAT_INDEX_GEN3, info->stats_idx) | | ||
| 1136 | FIELD_PREP(IRDMAQPC_PKT_LIMIT, qp->pkt_limit)); | ||
| 1137 | |||
| 1138 | print_hex_dump_debug("WQE: QP_HOST ROCE CTX WQE", DUMP_PREFIX_OFFSET, | ||
| 1139 | 16, 8, qp_ctx, IRDMA_QP_CTX_SIZE, false); | ||
| 1140 | } | ||
| 1141 | |||
| 1142 | void irdma_sc_qp_setctx_roce(struct irdma_sc_qp *qp, __le64 *qp_ctx, | ||
| 1143 | struct irdma_qp_host_ctx_info *info) | ||
| 1144 | { | ||
| 1145 | if (qp->dev->hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_2) | ||
| 1146 | irdma_sc_qp_setctx_roce_gen_2(qp, qp_ctx, info); | ||
| 1147 | else | ||
| 1148 | irdma_sc_qp_setctx_roce_gen_3(qp, qp_ctx, info); | ||
| 1149 | } | ||
| 1150 | |||
| 756 | /* irdma_sc_alloc_local_mac_entry - allocate a mac entry | 1151 | /* irdma_sc_alloc_local_mac_entry - allocate a mac entry |
| 757 | * @cqp: struct for cqp hw | 1152 | * @cqp: struct for cqp hw |
| 758 | * @scratch: u64 saved to be used during cqp completion | 1153 | * @scratch: u64 saved to be used during cqp completion |
| @@ -1080,7 +1475,8 @@ static int irdma_sc_alloc_stag(struct irdma_sc_dev *dev, · 321 unmodified lines | |||
| 1080 | FLD_LS_64(dev, info->pd_id, IRDMA_CQPSQ_STAG_PDID) | | 1475 | FLD_LS_64(dev, info->pd_id, IRDMA_CQPSQ_STAG_PDID) | |
| 1081 | FIELD_PREP(IRDMA_CQPSQ_STAG_STAGLEN, info->total_len)); | 1476 | FIELD_PREP(IRDMA_CQPSQ_STAG_STAGLEN, info->total_len)); |
| 1082 | set_64bit_val(wqe, 16, | 1477 | set_64bit_val(wqe, 16, |
| 1083 | FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->stag_idx)); | 1478 | FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->stag_idx) | |
| 1479 | FIELD_PREP(IRDMA_CQPSQ_STAG_PDID_HI, info->pd_id >> 18)); | ||
| 1084 | set_64bit_val(wqe, 40, | 1480 | set_64bit_val(wqe, 40, |
| 1085 | FIELD_PREP(IRDMA_CQPSQ_STAG_HMCFNIDX, info->hmc_fcn_index)); | 1481 | FIELD_PREP(IRDMA_CQPSQ_STAG_HMCFNIDX, info->hmc_fcn_index)); |
| 1086 | 1482 | ||
| @@ -1096,6 +1492,8 @@ static int irdma_sc_alloc_stag(struct irdma_sc_dev *dev, · 9 unmodified lines | |||
| 1096 | FIELD_PREP(IRDMA_CQPSQ_STAG_REMACCENABLED, info->remote_access) | | 1492 | FIELD_PREP(IRDMA_CQPSQ_STAG_REMACCENABLED, info->remote_access) | |
| 1097 | FIELD_PREP(IRDMA_CQPSQ_STAG_USEHMCFNIDX, info->use_hmc_fcn_index) | | 1493 | FIELD_PREP(IRDMA_CQPSQ_STAG_USEHMCFNIDX, info->use_hmc_fcn_index) | |
| 1098 | FIELD_PREP(IRDMA_CQPSQ_STAG_USEPFRID, info->use_pf_rid) | | 1494 | FIELD_PREP(IRDMA_CQPSQ_STAG_USEPFRID, info->use_pf_rid) | |
| 1495 | FIELD_PREP(IRDMA_CQPSQ_STAG_REMOTE_ATOMIC_EN, | ||
| 1496 | info->remote_atomics_en) | | ||
| 1099 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); | 1497 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); |
| 1100 | dma_wmb(); /* make sure WQE is written before valid bit is set */ | 1498 | dma_wmb(); /* make sure WQE is written before valid bit is set */ |
| 1101 | 1499 | ||
| @@ -1165,6 +1563,7 @@ static int irdma_sc_mr_reg_non_shared(struct irdma_sc_dev *dev, · 63 unmodified lines | |||
| 1165 | FLD_LS_64(dev, info->pd_id, IRDMA_CQPSQ_STAG_PDID)); | 1563 | FLD_LS_64(dev, info->pd_id, IRDMA_CQPSQ_STAG_PDID)); |
| 1166 | set_64bit_val(wqe, 16, | 1564 | set_64bit_val(wqe, 16, |
| 1167 | FIELD_PREP(IRDMA_CQPSQ_STAG_KEY, info->stag_key) | | 1565 | FIELD_PREP(IRDMA_CQPSQ_STAG_KEY, info->stag_key) | |
| 1566 | FIELD_PREP(IRDMA_CQPSQ_STAG_PDID_HI, info->pd_id >> 18) | | ||
| 1168 | FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->stag_idx)); | 1567 | FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->stag_idx)); |
| 1169 | if (!info->chunk_size) { | 1568 | if (!info->chunk_size) { |
| 1170 | set_64bit_val(wqe, 32, info->reg_addr_pa); | 1569 | set_64bit_val(wqe, 32, info->reg_addr_pa); |
| @@ -1187,6 +1586,8 @@ static int irdma_sc_mr_reg_non_shared(struct irdma_sc_dev *dev, · 16 unmodified lines | |||
| 1187 | FIELD_PREP(IRDMA_CQPSQ_STAG_VABASEDTO, addr_type) | | 1586 | FIELD_PREP(IRDMA_CQPSQ_STAG_VABASEDTO, addr_type) | |
| 1188 | FIELD_PREP(IRDMA_CQPSQ_STAG_USEHMCFNIDX, info->use_hmc_fcn_index) | | 1587 | FIELD_PREP(IRDMA_CQPSQ_STAG_USEHMCFNIDX, info->use_hmc_fcn_index) | |
| 1189 | FIELD_PREP(IRDMA_CQPSQ_STAG_USEPFRID, info->use_pf_rid) | | 1588 | FIELD_PREP(IRDMA_CQPSQ_STAG_USEPFRID, info->use_pf_rid) | |
| 1589 | FIELD_PREP(IRDMA_CQPSQ_STAG_REMOTE_ATOMIC_EN, | ||
| 1590 | info->remote_atomics_en) | | ||
| 1190 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); | 1591 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); |
| 1191 | dma_wmb(); /* make sure WQE is written before valid bit is set */ | 1592 | dma_wmb(); /* make sure WQE is written before valid bit is set */ |
| 1192 | 1593 | ||
| @@ -1223,7 +1624,8 @@ static int irdma_sc_dealloc_stag(struct irdma_sc_dev *dev, · 30 unmodified lines | |||
| 1223 | set_64bit_val(wqe, 8, | 1624 | set_64bit_val(wqe, 8, |
| 1224 | FLD_LS_64(dev, info->pd_id, IRDMA_CQPSQ_STAG_PDID)); | 1625 | FLD_LS_64(dev, info->pd_id, IRDMA_CQPSQ_STAG_PDID)); |
| 1225 | set_64bit_val(wqe, 16, | 1626 | set_64bit_val(wqe, 16, |
| 1226 | FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->stag_idx)); | 1627 | FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->stag_idx) | |
| 1628 | FIELD_PREP(IRDMA_CQPSQ_STAG_PDID_HI, info->pd_id >> 18)); | ||
| 1227 | 1629 | ||
| 1228 | hdr = FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_DEALLOC_STAG) | | 1630 | hdr = FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_DEALLOC_STAG) | |
| 1229 | FIELD_PREP(IRDMA_CQPSQ_STAG_MR, info->mr) | | 1631 | FIELD_PREP(IRDMA_CQPSQ_STAG_MR, info->mr) | |
| @@ -1263,7 +1665,8 @@ static int irdma_sc_mw_alloc(struct irdma_sc_dev *dev, · 33 unmodified lines | |||
| 1263 | set_64bit_val(wqe, 8, | 1665 | set_64bit_val(wqe, 8, |
| 1264 | FLD_LS_64(dev, info->pd_id, IRDMA_CQPSQ_STAG_PDID)); | 1666 | FLD_LS_64(dev, info->pd_id, IRDMA_CQPSQ_STAG_PDID)); |
| 1265 | set_64bit_val(wqe, 16, | 1667 | set_64bit_val(wqe, 16, |
| 1266 | FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->mw_stag_index)); | 1668 | FIELD_PREP(IRDMA_CQPSQ_STAG_IDX, info->mw_stag_index) | |
| 1669 | FIELD_PREP(IRDMA_CQPSQ_STAG_PDID_HI, info->pd_id >> 18)); | ||
| 1267 | 1670 | ||
| 1268 | hdr = FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_ALLOC_STAG) | | 1671 | hdr = FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_ALLOC_STAG) | |
| 1269 | FIELD_PREP(IRDMA_CQPSQ_STAG_MWTYPE, info->mw_wide) | | 1672 | FIELD_PREP(IRDMA_CQPSQ_STAG_MWTYPE, info->mw_wide) | |
| @@ -1343,6 +1746,7 @@ int irdma_sc_mr_fast_register(struct irdma_sc_qp *qp, · 73 unmodified lines | |||
| 1343 | FIELD_PREP(IRDMAQPSQ_READFENCE, info->read_fence) | | 1746 | FIELD_PREP(IRDMAQPSQ_READFENCE, info->read_fence) | |
| 1344 | FIELD_PREP(IRDMAQPSQ_LOCALFENCE, info->local_fence) | | 1747 | FIELD_PREP(IRDMAQPSQ_LOCALFENCE, info->local_fence) | |
| 1345 | FIELD_PREP(IRDMAQPSQ_SIGCOMPL, info->signaled) | | 1748 | FIELD_PREP(IRDMAQPSQ_SIGCOMPL, info->signaled) | |
| 1749 | FIELD_PREP(IRDMAQPSQ_REMOTE_ATOMICS_EN, info->remote_atomics_en) | | ||
| 1346 | FIELD_PREP(IRDMAQPSQ_VALID, qp->qp_uk.swqe_polarity); | 1750 | FIELD_PREP(IRDMAQPSQ_VALID, qp->qp_uk.swqe_polarity); |
| 1347 | dma_wmb(); /* make sure WQE is written before valid bit is set */ | 1751 | dma_wmb(); /* make sure WQE is written before valid bit is set */ |
| 1348 | 1752 | ||
| @@ -1873,7 +2277,7 @@ void irdma_sc_vsi_init(struct irdma_sc_vsi *vsi, · 524 unmodified lines | |||
| 1873 | mutex_init(&vsi->qos[i].qos_mutex); | 2277 | mutex_init(&vsi->qos[i].qos_mutex); |
| 1874 | INIT_LIST_HEAD(&vsi->qos[i].qplist); | 2278 | INIT_LIST_HEAD(&vsi->qos[i].qplist); |
| 1875 | } | 2279 | } |
| 1876 | if (vsi->register_qset) { | 2280 | if (vsi->dev->hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_2) { |
| 1877 | vsi->dev->ws_add = irdma_ws_add; | 2281 | vsi->dev->ws_add = irdma_ws_add; |
| 1878 | vsi->dev->ws_remove = irdma_ws_remove; | 2282 | vsi->dev->ws_remove = irdma_ws_remove; |
| 1879 | vsi->dev->ws_reset = irdma_ws_reset; | 2283 | vsi->dev->ws_reset = irdma_ws_reset; |
| @@ -1888,7 +2292,7 @@ void irdma_sc_vsi_init(struct irdma_sc_vsi *vsi, · 8 unmodified lines | |||
| 1888 | * irdma_get_stats_idx - Return stats index | 2292 | * irdma_get_stats_idx - Return stats index |
| 1889 | * @vsi: pointer to the vsi | 2293 | * @vsi: pointer to the vsi |
| 1890 | */ | 2294 | */ |
| 1891 | static u8 irdma_get_stats_idx(struct irdma_sc_vsi *vsi) | 2295 | static u16 irdma_get_stats_idx(struct irdma_sc_vsi *vsi) |
| 1892 | { | 2296 | { |
| 1893 | struct irdma_stats_inst_info stats_info = {}; | 2297 | struct irdma_stats_inst_info stats_info = {}; |
| 1894 | struct irdma_sc_dev *dev = vsi->dev; | 2298 | struct irdma_sc_dev *dev = vsi->dev; |
| @@ -1964,12 +2368,13 @@ int irdma_vsi_stats_init(struct irdma_sc_vsi *vsi, · 69 unmodified lines | |||
| 1964 | (void *)((uintptr_t)stats_buff_mem->va + | 2368 | (void *)((uintptr_t)stats_buff_mem->va + |
| 1965 | IRDMA_GATHER_STATS_BUF_SIZE); | 2369 | IRDMA_GATHER_STATS_BUF_SIZE); |
| 1966 | 2370 | ||
| 1967 | irdma_hw_stats_start_timer(vsi); | 2371 | if (vsi->dev->hw_attrs.uk_attrs.hw_rev < IRDMA_GEN_3) |
| 2372 | irdma_hw_stats_start_timer(vsi); | ||
| 1968 | 2373 | ||
| 1969 | /* when stat allocation is not required default to fcn_id. */ | 2374 | /* when stat allocation is not required default to fcn_id. */ |
| 1970 | vsi->stats_idx = info->fcn_id; | 2375 | vsi->stats_idx = info->fcn_id; |
| 1971 | if (info->alloc_stats_inst) { | 2376 | if (info->alloc_stats_inst) { |
| 1972 | u8 stats_idx = irdma_get_stats_idx(vsi); | 2377 | u16 stats_idx = irdma_get_stats_idx(vsi); |
| 1973 | 2378 | ||
| 1974 | if (stats_idx != IRDMA_INVALID_STATS_IDX) { | 2379 | if (stats_idx != IRDMA_INVALID_STATS_IDX) { |
| 1975 | vsi->stats_inst_alloc = true; | 2380 | vsi->stats_inst_alloc = true; |
| @@ -1993,7 +2398,7 @@ void irdma_vsi_stats_free(struct irdma_sc_vsi *vsi) · 17 unmodified lines | |||
| 1993 | { | 2398 | { |
| 1994 | struct irdma_stats_inst_info stats_info = {}; | 2399 | struct irdma_stats_inst_info stats_info = {}; |
| 1995 | struct irdma_sc_dev *dev = vsi->dev; | 2400 | struct irdma_sc_dev *dev = vsi->dev; |
| 1996 | u8 stats_idx = vsi->stats_idx; | 2401 | u16 stats_idx = vsi->stats_idx; |
| 1997 | 2402 | ||
| 1998 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_2) { | 2403 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_2) { |
| 1999 | if (vsi->stats_inst_alloc) { | 2404 | if (vsi->stats_inst_alloc) { |
| @@ -2009,7 +2414,9 @@ void irdma_vsi_stats_free(struct irdma_sc_vsi *vsi) · 9 unmodified lines | |||
| 2009 | 2414 | ||
| 2010 | if (!vsi->pestat) | 2415 | if (!vsi->pestat) |
| 2011 | return; | 2416 | return; |
| 2012 | irdma_hw_stats_stop_timer(vsi); | 2417 | |
| 2418 | if (dev->hw_attrs.uk_attrs.hw_rev < IRDMA_GEN_3) | ||
| 2419 | irdma_hw_stats_stop_timer(vsi); | ||
| 2013 | dma_free_coherent(vsi->pestat->hw->device, | 2420 | dma_free_coherent(vsi->pestat->hw->device, |
| 2014 | vsi->pestat->gather_info.stats_buff_mem.size, | 2421 | vsi->pestat->gather_info.stats_buff_mem.size, |
| 2015 | vsi->pestat->gather_info.stats_buff_mem.va, | 2422 | vsi->pestat->gather_info.stats_buff_mem.va, |
| @@ -2026,6 +2433,14 @@ u8 irdma_get_encoded_wqe_size(u32 wqsize, enum irdma_queue_type queue_type) · 10 unmodified lines | |||
| 2026 | { | 2433 | { |
| 2027 | u8 encoded_size = 0; | 2434 | u8 encoded_size = 0; |
| 2028 | 2435 | ||
| 2436 | if (queue_type == IRDMA_QUEUE_TYPE_SRQ) { | ||
| 2437 | /* Smallest SRQ size is 256B (8 quanta) that gets | ||
| 2438 | * encoded to 0. | ||
| 2439 | */ | ||
| 2440 | encoded_size = ilog2(wqsize) - 3; | ||
| 2441 | |||
| 2442 | return encoded_size; | ||
| 2443 | } | ||
| 2029 | /* cqp sq's hw coded value starts from 1 for size of 4 | 2444 | /* cqp sq's hw coded value starts from 1 for size of 4 |
| 2030 | * while it starts from 0 for qp' wq's. | 2445 | * while it starts from 0 for qp' wq's. |
| 2031 | */ | 2446 | */ |
| @@ -2259,6 +2674,12 @@ int irdma_sc_qp_flush_wqes(struct irdma_sc_qp *qp, · 227 unmodified lines | |||
| 2259 | info->ae_code | FIELD_PREP(IRDMA_CQPSQ_FWQE_AESOURCE, | 2674 | info->ae_code | FIELD_PREP(IRDMA_CQPSQ_FWQE_AESOURCE, |
| 2260 | info->ae_src) : 0; | 2675 | info->ae_src) : 0; |
| 2261 | set_64bit_val(wqe, 8, temp); | 2676 | set_64bit_val(wqe, 8, temp); |
| 2677 | if (cqp->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) { | ||
| 2678 | set_64bit_val(wqe, 40, | ||
| 2679 | FIELD_PREP(IRDMA_CQPSQ_FWQE_ERR_SQ_IDX, info->err_sq_idx)); | ||
| 2680 | set_64bit_val(wqe, 48, | ||
| 2681 | FIELD_PREP(IRDMA_CQPSQ_FWQE_ERR_RQ_IDX, info->err_rq_idx)); | ||
| 2682 | } | ||
| 2262 | 2683 | ||
| 2263 | hdr = qp->qp_uk.qp_id | | 2684 | hdr = qp->qp_uk.qp_id | |
| 2264 | FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_FLUSH_WQES) | | 2685 | FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_FLUSH_WQES) | |
| @@ -2267,6 +2688,9 @@ int irdma_sc_qp_flush_wqes(struct irdma_sc_qp *qp, · 2 unmodified lines | |||
| 2267 | FIELD_PREP(IRDMA_CQPSQ_FWQE_FLUSHSQ, flush_sq) | | 2688 | FIELD_PREP(IRDMA_CQPSQ_FWQE_FLUSHSQ, flush_sq) | |
| 2268 | FIELD_PREP(IRDMA_CQPSQ_FWQE_FLUSHRQ, flush_rq) | | 2689 | FIELD_PREP(IRDMA_CQPSQ_FWQE_FLUSHRQ, flush_rq) | |
| 2269 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); | 2690 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); |
| 2691 | if (cqp->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) | ||
| 2692 | hdr |= FIELD_PREP(IRDMA_CQPSQ_FWQE_ERR_SQ_IDX_VALID, info->err_sq_idx_valid) | | ||
| 2693 | FIELD_PREP(IRDMA_CQPSQ_FWQE_ERR_RQ_IDX_VALID, info->err_rq_idx_valid); | ||
| 2270 | dma_wmb(); /* make sure WQE is written before valid bit is set */ | 2694 | dma_wmb(); /* make sure WQE is written before valid bit is set */ |
| 2271 | 2695 | ||
| 2272 | set_64bit_val(wqe, 24, hdr); | 2696 | set_64bit_val(wqe, 24, hdr); |
| @@ -2562,6 +2986,9 @@ static int irdma_sc_cq_create(struct irdma_sc_cq *cq, u64 scratch, · 289 unmodified lines | |||
| 2562 | FIELD_PREP(IRDMA_CQPSQ_CQ_LPBLSIZE, cq->pbl_chunk_size) | | 2986 | FIELD_PREP(IRDMA_CQPSQ_CQ_LPBLSIZE, cq->pbl_chunk_size) | |
| 2563 | FIELD_PREP(IRDMA_CQPSQ_CQ_CHKOVERFLOW, check_overflow) | | 2987 | FIELD_PREP(IRDMA_CQPSQ_CQ_CHKOVERFLOW, check_overflow) | |
| 2564 | FIELD_PREP(IRDMA_CQPSQ_CQ_VIRTMAP, cq->virtual_map) | | 2988 | FIELD_PREP(IRDMA_CQPSQ_CQ_VIRTMAP, cq->virtual_map) | |
| 2989 | FIELD_PREP(IRDMA_CQPSQ_CQ_CQID_HIGH, cq->cq_uk.cq_id >> 22) | | ||
| 2990 | FIELD_PREP(IRDMA_CQPSQ_CQ_CEQID_HIGH, | ||
| 2991 | (cq->ceq_id_valid ? cq->ceq_id : 0) >> 10) | | ||
| 2565 | FIELD_PREP(IRDMA_CQPSQ_CQ_ENCEQEMASK, cq->ceqe_mask) | | 2992 | FIELD_PREP(IRDMA_CQPSQ_CQ_ENCEQEMASK, cq->ceqe_mask) | |
| 2566 | FIELD_PREP(IRDMA_CQPSQ_CQ_CEQIDVALID, cq->ceq_id_valid) | | 2993 | FIELD_PREP(IRDMA_CQPSQ_CQ_CEQIDVALID, cq->ceq_id_valid) | |
| 2567 | FIELD_PREP(IRDMA_CQPSQ_TPHEN, cq->tph_en) | | 2994 | FIELD_PREP(IRDMA_CQPSQ_TPHEN, cq->tph_en) | |
| @@ -2705,6 +3132,41 @@ static int irdma_sc_cq_modify(struct irdma_sc_cq *cq, · 137 unmodified lines | |||
| 2705 | return 0; | 3132 | return 0; |
| 2706 | } | 3133 | } |
| 2707 | 3134 | ||
| 3135 | /** | ||
| 3136 | * irdma_sc_get_decoded_ird_size_gen_3 - get decoded IRD size for GEN 3 | ||
| 3137 | * @ird_enc: IRD encoding | ||
| 3138 | * IRD size defaults to a value of 4 in case of invalid input. | ||
| 3139 | */ | ||
| 3140 | static u16 irdma_sc_get_decoded_ird_size_gen_3(u8 ird_enc) | ||
| 3141 | { | ||
| 3142 | switch (ird_enc) { | ||
| 3143 | case IRDMA_IRD_HW_SIZE_4096_GEN3: | ||
| 3144 | return 4096; | ||
| 3145 | case IRDMA_IRD_HW_SIZE_2048_GEN3: | ||
| 3146 | return 2048; | ||
| 3147 | case IRDMA_IRD_HW_SIZE_1024_GEN3: | ||
| 3148 | return 1024; | ||
| 3149 | case IRDMA_IRD_HW_SIZE_512_GEN3: | ||
| 3150 | return 512; | ||
| 3151 | case IRDMA_IRD_HW_SIZE_256_GEN3: | ||
| 3152 | return 256; | ||
| 3153 | case IRDMA_IRD_HW_SIZE_128_GEN3: | ||
| 3154 | return 128; | ||
| 3155 | case IRDMA_IRD_HW_SIZE_64_GEN3: | ||
| 3156 | return 64; | ||
| 3157 | case IRDMA_IRD_HW_SIZE_32_GEN3: | ||
| 3158 | return 32; | ||
| 3159 | case IRDMA_IRD_HW_SIZE_16_GEN3: | ||
| 3160 | return 16; | ||
| 3161 | case IRDMA_IRD_HW_SIZE_8_GEN3: | ||
| 3162 | return 8; | ||
| 3163 | case IRDMA_IRD_HW_SIZE_4_GEN3: | ||
| 3164 | return 4; | ||
| 3165 | default: | ||
| 3166 | return 4; | ||
| 3167 | } | ||
| 3168 | } | ||
| 3169 | |||
| 2708 | /** | 3170 | /** |
| 2709 | * irdma_check_cqp_progress - check cqp processing progress | 3171 | * irdma_check_cqp_progress - check cqp processing progress |
| 2710 | * @timeout: timeout info struct | 3172 | * @timeout: timeout info struct |
| @@ -2738,23 +3200,106 @@ static inline void irdma_get_cqp_reg_info(struct irdma_sc_cqp *cqp, u32 *val, · 27 unmodified lines | |||
| 2738 | } | 3200 | } |
| 2739 | 3201 | ||
| 2740 | /** | 3202 | /** |
| 2741 | * irdma_cqp_poll_registers - poll cqp registers | 3203 | * irdma_sc_cqp_def_cmpl_ae_handler - remove completed requests from pending list |
| 2742 | * @cqp: struct for cqp hw | 3204 | * @dev: sc device struct |
| 2743 | * @tail: wqtail register value | 3205 | * @info: AE entry info |
| 2744 | * @count: how many times to try for completion | 3206 | * @first: true if this is the first call to this handler for given AEQE |
| 3207 | * @scratch: (out) scratch entry pointer | ||
| 3208 | * @sw_def_info: (in/out) SW ticket value for this AE | ||
| 3209 | * | ||
| 3210 | * In case of AE_DEF_CMPL event, this function should be called in a loop | ||
| 3211 | * until it returns NULL-ptr via scratch. | ||
| 3212 | * For each call, it looks for a matching CQP request on pending list, | ||
| 3213 | * removes it from the list and returns the pointer to the associated scratch | ||
| 3214 | * entry. | ||
| 3215 | * If this is the first call to this function for given AEQE, sw_def_info | ||
| 3216 | * value is not used to find matching requests. Instead, it is populated | ||
| 3217 | * with the value from the first matching cqp_request on the list. | ||
| 3218 | * For subsequent calls, ooo_op->sw_def_info need to match the value passed | ||
| 3219 | * by a caller. | ||
| 3220 | * | ||
| 3221 | * Return: scratch entry pointer for cqp_request to be released or NULL | ||
| 3222 | * if no matching request is found. | ||
| 2745 | */ | 3223 | */ |
| 2746 | static int irdma_cqp_poll_registers(struct irdma_sc_cqp *cqp, u32 tail, | 3224 | void irdma_sc_cqp_def_cmpl_ae_handler(struct irdma_sc_dev *dev, |
| 2747 | u32 count) | 3225 | struct irdma_aeqe_info *info, |
| 3226 | bool first, u64 *scratch, | ||
| 3227 | u32 *sw_def_info) | ||
| 2748 | { | 3228 | { |
| 2749 | u32 i = 0; | 3229 | struct irdma_ooo_cqp_op *ooo_op; |
| 2750 | u32 newtail, error, val; | 3230 | unsigned long flags; |
| 2751 | 3231 | ||
| 2752 | while (i++ < count) { | 3232 | *scratch = 0; |
| 2753 | irdma_get_cqp_reg_info(cqp, &val, &newtail, &error); | 3233 | |
| 2754 | if (error) { | 3234 | spin_lock_irqsave(&dev->cqp->ooo_list_lock, flags); |
| 2755 | error = readl(cqp->dev->hw_regs[IRDMA_CQPERRCODES]); | 3235 | list_for_each_entry(ooo_op, &dev->cqp->ooo_pnd, list_entry) { |
| 2756 | ibdev_dbg(to_ibdev(cqp->dev), | 3236 | if (ooo_op->deferred && |
| 2757 | "CQP: CQPERRCODES error_code[x%08X]\n", | 3237 | ((first && ooo_op->def_info == info->def_info) || |
| 3238 | (!first && ooo_op->sw_def_info == *sw_def_info))) { | ||
| 3239 | *sw_def_info = ooo_op->sw_def_info; | ||
| 3240 | *scratch = ooo_op->scratch; | ||
| 3241 | |||
| 3242 | list_move(&ooo_op->list_entry, &dev->cqp->ooo_avail); | ||
| 3243 | atomic64_inc(&dev->cqp->completed_ops); | ||
| 3244 | |||
| 3245 | break; | ||
| 3246 | } | ||
| 3247 | } | ||
| 3248 | spin_unlock_irqrestore(&dev->cqp->ooo_list_lock, flags); | ||
| 3249 | |||
| 3250 | if (first && !*scratch) | ||
| 3251 | ibdev_dbg(to_ibdev(dev), | ||
| 3252 | "AEQ: deferred completion with unknown ticket: def_info 0x%x\n", | ||
| 3253 | info->def_info); | ||
| 3254 | } | ||
| 3255 | |||
| 3256 | /** | ||
| 3257 | * irdma_sc_cqp_cleanup_handler - remove requests from pending list | ||
| 3258 | * @dev: sc device struct | ||
| 3259 | * | ||
| 3260 | * This function should be called in a loop from irdma_cleanup_pending_cqp_op. | ||
| 3261 | * For each call, it returns first CQP request on pending list, removes it | ||
| 3262 | * from the list and returns the pointer to the associated scratch entry. | ||
| 3263 | * | ||
| 3264 | * Return: scratch entry pointer for cqp_request to be released or NULL | ||
| 3265 | * if pending list is empty. | ||
| 3266 | */ | ||
| 3267 | u64 irdma_sc_cqp_cleanup_handler(struct irdma_sc_dev *dev) | ||
| 3268 | { | ||
| 3269 | struct irdma_ooo_cqp_op *ooo_op; | ||
| 3270 | u64 scratch = 0; | ||
| 3271 | |||
| 3272 | list_for_each_entry(ooo_op, &dev->cqp->ooo_pnd, list_entry) { | ||
| 3273 | scratch = ooo_op->scratch; | ||
| 3274 | |||
| 3275 | list_del(&ooo_op->list_entry); | ||
| 3276 | list_add(&ooo_op->list_entry, &dev->cqp->ooo_avail); | ||
| 3277 | atomic64_inc(&dev->cqp->completed_ops); | ||
| 3278 | |||
| 3279 | break; | ||
| 3280 | } | ||
| 3281 | |||
| 3282 | return scratch; | ||
| 3283 | } | ||
| 3284 | |||
| 3285 | /** | ||
| 3286 | * irdma_cqp_poll_registers - poll cqp registers | ||
| 3287 | * @cqp: struct for cqp hw | ||
| 3288 | * @tail: wqtail register value | ||
| 3289 | * @count: how many times to try for completion | ||
| 3290 | */ | ||
| 3291 | static int irdma_cqp_poll_registers(struct irdma_sc_cqp *cqp, u32 tail, | ||
| 3292 | u32 count) | ||
| 3293 | { | ||
| 3294 | u32 i = 0; | ||
| 3295 | u32 newtail, error, val; | ||
| 3296 | |||
| 3297 | while (i++ < count) { | ||
| 3298 | irdma_get_cqp_reg_info(cqp, &val, &newtail, &error); | ||
| 3299 | if (error) { | ||
| 3300 | error = readl(cqp->dev->hw_regs[IRDMA_CQPERRCODES]); | ||
| 3301 | ibdev_dbg(to_ibdev(cqp->dev), | ||
| 3302 | "CQP: CQPERRCODES error_code[x%08X]\n", | ||
| 2758 | error); | 3303 | error); |
| 2759 | return -EIO; | 3304 | return -EIO; |
| 2760 | } | 3305 | } |
| @@ -2794,7 +3339,10 @@ static u64 irdma_sc_decode_fpm_commit(struct irdma_sc_dev *dev, __le64 *buf, · 33 unmodified lines | |||
| 2794 | obj_info[rsrc_idx].cnt = (u32)FLD_RS_64(dev, temp, IRDMA_COMMIT_FPM_CQCNT); | 3339 | obj_info[rsrc_idx].cnt = (u32)FLD_RS_64(dev, temp, IRDMA_COMMIT_FPM_CQCNT); |
| 2795 | break; | 3340 | break; |
| 2796 | case IRDMA_HMC_IW_APBVT_ENTRY: | 3341 | case IRDMA_HMC_IW_APBVT_ENTRY: |
| 2797 | obj_info[rsrc_idx].cnt = 1; | 3342 | if (dev->hw_attrs.uk_attrs.hw_rev <= IRDMA_GEN_2) |
| 3343 | obj_info[rsrc_idx].cnt = 1; | ||
| 3344 | else | ||
| 3345 | obj_info[rsrc_idx].cnt = 0; | ||
| 2798 | break; | 3346 | break; |
| 2799 | default: | 3347 | default: |
| 2800 | obj_info[rsrc_idx].cnt = (u32)temp; | 3348 | obj_info[rsrc_idx].cnt = (u32)temp; |
| @@ -2829,7 +3377,8 @@ irdma_sc_parse_fpm_commit_buf(struct irdma_sc_dev *dev, __le64 *buf, · 28 unmodified lines | |||
| 2829 | IRDMA_HMC_IW_QP); | 3377 | IRDMA_HMC_IW_QP); |
| 2830 | irdma_sc_decode_fpm_commit(dev, buf, 8, info, | 3378 | irdma_sc_decode_fpm_commit(dev, buf, 8, info, |
| 2831 | IRDMA_HMC_IW_CQ); | 3379 | IRDMA_HMC_IW_CQ); |
| 2832 | /* skiping RSRVD */ | 3380 | irdma_sc_decode_fpm_commit(dev, buf, 16, info, |
| 3381 | IRDMA_HMC_IW_SRQ); | ||
| 2833 | irdma_sc_decode_fpm_commit(dev, buf, 24, info, | 3382 | irdma_sc_decode_fpm_commit(dev, buf, 24, info, |
| 2834 | IRDMA_HMC_IW_HTE); | 3383 | IRDMA_HMC_IW_HTE); |
| 2835 | irdma_sc_decode_fpm_commit(dev, buf, 32, info, | 3384 | irdma_sc_decode_fpm_commit(dev, buf, 32, info, |
| @@ -2864,15 +3413,17 @@ irdma_sc_parse_fpm_commit_buf(struct irdma_sc_dev *dev, __le64 *buf, · 28 unmodified lines | |||
| 2864 | IRDMA_HMC_IW_HDR); | 3413 | IRDMA_HMC_IW_HDR); |
| 2865 | irdma_sc_decode_fpm_commit(dev, buf, 152, info, | 3414 | irdma_sc_decode_fpm_commit(dev, buf, 152, info, |
| 2866 | IRDMA_HMC_IW_MD); | 3415 | IRDMA_HMC_IW_MD); |
| 2867 | irdma_sc_decode_fpm_commit(dev, buf, 160, info, | 3416 | if (dev->cqp->protocol_used == IRDMA_IWARP_PROTOCOL_ONLY) { |
| 2868 | IRDMA_HMC_IW_OOISC); | 3417 | irdma_sc_decode_fpm_commit(dev, buf, 160, info, |
| 2869 | irdma_sc_decode_fpm_commit(dev, buf, 168, info, | 3418 | IRDMA_HMC_IW_OOISC); |
| 2870 | IRDMA_HMC_IW_OOISCFFL); | 3419 | irdma_sc_decode_fpm_commit(dev, buf, 168, info, |
| 3420 | IRDMA_HMC_IW_OOISCFFL); | ||
| 3421 | } | ||
| 2871 | } | 3422 | } |
| 2872 | 3423 | ||
| 2873 | /* searching for the last object in HMC to find the size of the HMC area. */ | 3424 | /* searching for the last object in HMC to find the size of the HMC area. */ |
| 2874 | for (i = IRDMA_HMC_IW_QP; i < IRDMA_HMC_IW_MAX; i++) { | 3425 | for (i = IRDMA_HMC_IW_QP; i < IRDMA_HMC_IW_MAX; i++) { |
| 2875 | if (info[i].base > max_base) { | 3426 | if (info[i].base > max_base && info[i].cnt) { |
| 2876 | max_base = info[i].base; | 3427 | max_base = info[i].base; |
| 2877 | last_hmc_obj = i; | 3428 | last_hmc_obj = i; |
| 2878 | } | 3429 | } |
| @@ -2927,6 +3478,7 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, · 48 unmodified lines | |||
| 2927 | struct irdma_hmc_fpm_misc *hmc_fpm_misc) | 3478 | struct irdma_hmc_fpm_misc *hmc_fpm_misc) |
| 2928 | { | 3479 | { |
| 2929 | struct irdma_hmc_obj_info *obj_info; | 3480 | struct irdma_hmc_obj_info *obj_info; |
| 3481 | u8 ird_encoding; | ||
| 2930 | u64 temp; | 3482 | u64 temp; |
| 2931 | u32 size; | 3483 | u32 size; |
| 2932 | u16 max_pe_sds; | 3484 | u16 max_pe_sds; |
| @@ -2935,7 +3487,19 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, · 2 unmodified lines | |||
| 2935 | 3487 | ||
| 2936 | get_64bit_val(buf, 0, &temp); | 3488 | get_64bit_val(buf, 0, &temp); |
| 2937 | hmc_info->first_sd_index = (u16)FIELD_GET(IRDMA_QUERY_FPM_FIRST_PE_SD_INDEX, temp); | 3489 | hmc_info->first_sd_index = (u16)FIELD_GET(IRDMA_QUERY_FPM_FIRST_PE_SD_INDEX, temp); |
| 2938 | max_pe_sds = (u16)FIELD_GET(IRDMA_QUERY_FPM_MAX_PE_SDS, temp); | 3490 | |
| 3491 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) | ||
| 3492 | max_pe_sds = (u16)FIELD_GET(IRDMA_QUERY_FPM_MAX_PE_SDS_GEN3, temp); | ||
| 3493 | else | ||
| 3494 | max_pe_sds = (u16)FIELD_GET(IRDMA_QUERY_FPM_MAX_PE_SDS, temp); | ||
| 3495 | |||
| 3496 | /* Reduce SD count for unprivleged functions by 1 to account for PBLE | ||
| 3497 | * backing page rounding | ||
| 3498 | */ | ||
| 3499 | if (dev->hw_attrs.uk_attrs.hw_rev <= IRDMA_GEN_2 && | ||
| 3500 | (hmc_info->hmc_fn_id >= dev->hw_attrs.first_hw_vf_fpm_id || | ||
| 3501 | !dev->privileged)) | ||
| 3502 | max_pe_sds--; | ||
| 2939 | 3503 | ||
| 2940 | hmc_fpm_misc->max_sds = max_pe_sds; | 3504 | hmc_fpm_misc->max_sds = max_pe_sds; |
| 2941 | hmc_info->sd_table.sd_cnt = max_pe_sds + hmc_info->first_sd_index; | 3505 | hmc_info->sd_table.sd_cnt = max_pe_sds + hmc_info->first_sd_index; |
| @@ -2949,11 +3513,17 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, · 7 unmodified lines | |||
| 2949 | size = (u32)(temp >> 32); | 3513 | size = (u32)(temp >> 32); |
| 2950 | obj_info[IRDMA_HMC_IW_CQ].size = BIT_ULL(size); | 3514 | obj_info[IRDMA_HMC_IW_CQ].size = BIT_ULL(size); |
| 2951 | 3515 | ||
| 3516 | irdma_sc_decode_fpm_query(buf, 24, obj_info, IRDMA_HMC_IW_SRQ); | ||
| 2952 | irdma_sc_decode_fpm_query(buf, 32, obj_info, IRDMA_HMC_IW_HTE); | 3517 | irdma_sc_decode_fpm_query(buf, 32, obj_info, IRDMA_HMC_IW_HTE); |
| 2953 | irdma_sc_decode_fpm_query(buf, 40, obj_info, IRDMA_HMC_IW_ARP); | 3518 | irdma_sc_decode_fpm_query(buf, 40, obj_info, IRDMA_HMC_IW_ARP); |
| 2954 | 3519 | ||
| 2955 | obj_info[IRDMA_HMC_IW_APBVT_ENTRY].size = 8192; | 3520 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) { |
| 2956 | obj_info[IRDMA_HMC_IW_APBVT_ENTRY].max_cnt = 1; | 3521 | obj_info[IRDMA_HMC_IW_APBVT_ENTRY].size = 0; |
| 3522 | obj_info[IRDMA_HMC_IW_APBVT_ENTRY].max_cnt = 0; | ||
| 3523 | } else { | ||
| 3524 | obj_info[IRDMA_HMC_IW_APBVT_ENTRY].size = 8192; | ||
| 3525 | obj_info[IRDMA_HMC_IW_APBVT_ENTRY].max_cnt = 1; | ||
| 3526 | } | ||
| 2957 | 3527 | ||
| 2958 | irdma_sc_decode_fpm_query(buf, 48, obj_info, IRDMA_HMC_IW_MR); | 3528 | irdma_sc_decode_fpm_query(buf, 48, obj_info, IRDMA_HMC_IW_MR); |
| 2959 | irdma_sc_decode_fpm_query(buf, 56, obj_info, IRDMA_HMC_IW_XF); | 3529 | irdma_sc_decode_fpm_query(buf, 56, obj_info, IRDMA_HMC_IW_XF); |
| @@ -2962,7 +3532,7 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, · 2 unmodified lines | |||
| 2962 | obj_info[IRDMA_HMC_IW_XFFL].max_cnt = (u32)temp; | 3532 | obj_info[IRDMA_HMC_IW_XFFL].max_cnt = (u32)temp; |
| 2963 | obj_info[IRDMA_HMC_IW_XFFL].size = 4; | 3533 | obj_info[IRDMA_HMC_IW_XFFL].size = 4; |
| 2964 | hmc_fpm_misc->xf_block_size = FIELD_GET(IRDMA_QUERY_FPM_XFBLOCKSIZE, temp); | 3534 | hmc_fpm_misc->xf_block_size = FIELD_GET(IRDMA_QUERY_FPM_XFBLOCKSIZE, temp); |
| 2965 | if (!hmc_fpm_misc->xf_block_size) | 3535 | if (obj_info[IRDMA_HMC_IW_XF].max_cnt && !hmc_fpm_misc->xf_block_size) |
| 2966 | return -EINVAL; | 3536 | return -EINVAL; |
| 2967 | 3537 | ||
| 2968 | irdma_sc_decode_fpm_query(buf, 72, obj_info, IRDMA_HMC_IW_Q1); | 3538 | irdma_sc_decode_fpm_query(buf, 72, obj_info, IRDMA_HMC_IW_Q1); |
| @@ -2984,6 +3554,14 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, · 15 unmodified lines | |||
| 2984 | hmc_fpm_misc->max_ceqs = FIELD_GET(IRDMA_QUERY_FPM_MAX_CEQS, temp); | 3554 | hmc_fpm_misc->max_ceqs = FIELD_GET(IRDMA_QUERY_FPM_MAX_CEQS, temp); |
| 2985 | hmc_fpm_misc->ht_multiplier = FIELD_GET(IRDMA_QUERY_FPM_HTMULTIPLIER, temp); | 3555 | hmc_fpm_misc->ht_multiplier = FIELD_GET(IRDMA_QUERY_FPM_HTMULTIPLIER, temp); |
| 2986 | hmc_fpm_misc->timer_bucket = FIELD_GET(IRDMA_QUERY_FPM_TIMERBUCKET, temp); | 3556 | hmc_fpm_misc->timer_bucket = FIELD_GET(IRDMA_QUERY_FPM_TIMERBUCKET, temp); |
| 3557 | if (FIELD_GET(IRDMA_MANAGE_RSRC_VER2, | ||
| 3558 | dev->feature_info[IRDMA_FTN_FLAGS])) { | ||
| 3559 | ird_encoding = (u8)FIELD_GET(IRDMA_QUERY_FPM_MAX_IRD, temp); | ||
| 3560 | hmc_fpm_misc->ird = | ||
| 3561 | irdma_sc_get_decoded_ird_size_gen_3(ird_encoding) / 2; | ||
| 3562 | dev->hw_attrs.max_hw_ird = hmc_fpm_misc->ird; | ||
| 3563 | dev->hw_attrs.max_hw_ord = hmc_fpm_misc->ird; | ||
| 3564 | } | ||
| 2987 | if (dev->hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_1) | 3565 | if (dev->hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_1) |
| 2988 | return 0; | 3566 | return 0; |
| 2989 | irdma_sc_decode_fpm_query(buf, 96, obj_info, IRDMA_HMC_IW_FSIMC); | 3567 | irdma_sc_decode_fpm_query(buf, 96, obj_info, IRDMA_HMC_IW_FSIMC); |
| @@ -3000,15 +3578,25 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, · 10 unmodified lines | |||
| 3000 | 3578 | ||
| 3001 | irdma_sc_decode_fpm_query(buf, 144, obj_info, IRDMA_HMC_IW_HDR); | 3579 | irdma_sc_decode_fpm_query(buf, 144, obj_info, IRDMA_HMC_IW_HDR); |
| 3002 | irdma_sc_decode_fpm_query(buf, 152, obj_info, IRDMA_HMC_IW_MD); | 3580 | irdma_sc_decode_fpm_query(buf, 152, obj_info, IRDMA_HMC_IW_MD); |
| 3003 | irdma_sc_decode_fpm_query(buf, 160, obj_info, IRDMA_HMC_IW_OOISC); | 3581 | |
| 3004 | 3582 | if (dev->cqp->protocol_used == IRDMA_IWARP_PROTOCOL_ONLY) { | |
| 3005 | get_64bit_val(buf, 168, &temp); | 3583 | irdma_sc_decode_fpm_query(buf, 160, obj_info, IRDMA_HMC_IW_OOISC); |
| 3006 | obj_info[IRDMA_HMC_IW_OOISCFFL].max_cnt = (u32)temp; | 3584 | |
| 3007 | obj_info[IRDMA_HMC_IW_OOISCFFL].size = 4; | 3585 | get_64bit_val(buf, 168, &temp); |
| 3008 | hmc_fpm_misc->ooiscf_block_size = FIELD_GET(IRDMA_QUERY_FPM_OOISCFBLOCKSIZE, temp); | 3586 | obj_info[IRDMA_HMC_IW_OOISCFFL].max_cnt = (u32)temp; |
| 3009 | if (!hmc_fpm_misc->ooiscf_block_size && | 3587 | obj_info[IRDMA_HMC_IW_OOISCFFL].size = 4; |
| 3010 | obj_info[IRDMA_HMC_IW_OOISCFFL].max_cnt) | 3588 | hmc_fpm_misc->ooiscf_block_size = FIELD_GET(IRDMA_QUERY_FPM_OOISCFBLOCKSIZE, temp); |
| 3011 | return -EINVAL; | 3589 | if (!hmc_fpm_misc->ooiscf_block_size && |
| 3590 | obj_info[IRDMA_HMC_IW_OOISCFFL].max_cnt) | ||
| 3591 | return -EINVAL; | ||
| 3592 | } | ||
| 3593 | |||
| 3594 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) { | ||
| 3595 | get_64bit_val(buf, 176, &temp); | ||
| 3596 | hmc_fpm_misc->loc_mem_pages = (u32)FIELD_GET(IRDMA_QUERY_FPM_LOC_MEM_PAGES, temp); | ||
| 3597 | if (!hmc_fpm_misc->loc_mem_pages) | ||
| 3598 | return -EINVAL; | ||
| 3599 | } | ||
| 3012 | 3600 | ||
| 3013 | return 0; | 3601 | return 0; |
| 3014 | } | 3602 | } |
| @@ -3088,6 +3676,8 @@ exit: · 73 unmodified lines | |||
| 3088 | int irdma_sc_cqp_init(struct irdma_sc_cqp *cqp, | 3676 | int irdma_sc_cqp_init(struct irdma_sc_cqp *cqp, |
| 3089 | struct irdma_cqp_init_info *info) | 3677 | struct irdma_cqp_init_info *info) |
| 3090 | { | 3678 | { |
| 3679 | struct irdma_ooo_cqp_op *ooo_op; | ||
| 3680 | u32 num_ooo_ops; | ||
| 3091 | u8 hw_sq_size; | 3681 | u8 hw_sq_size; |
| 3092 | 3682 | ||
| 3093 | if (info->sq_size > IRDMA_CQP_SW_SQSIZE_2048 || | 3683 | if (info->sq_size > IRDMA_CQP_SW_SQSIZE_2048 || |
| @@ -3118,17 +3708,43 @@ int irdma_sc_cqp_init(struct irdma_sc_cqp *cqp, · 24 unmodified lines | |||
| 3118 | cqp->rocev2_rto_policy = info->rocev2_rto_policy; | 3708 | cqp->rocev2_rto_policy = info->rocev2_rto_policy; |
| 3119 | cqp->protocol_used = info->protocol_used; | 3709 | cqp->protocol_used = info->protocol_used; |
| 3120 | memcpy(&cqp->dcqcn_params, &info->dcqcn_params, sizeof(cqp->dcqcn_params)); | 3710 | memcpy(&cqp->dcqcn_params, &info->dcqcn_params, sizeof(cqp->dcqcn_params)); |
| 3711 | if (cqp->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) { | ||
| 3712 | cqp->ooisc_blksize = info->ooisc_blksize; | ||
| 3713 | cqp->rrsp_blksize = info->rrsp_blksize; | ||
| 3714 | cqp->q1_blksize = info->q1_blksize; | ||
| 3715 | cqp->xmit_blksize = info->xmit_blksize; | ||
| 3716 | cqp->blksizes_valid = info->blksizes_valid; | ||
| 3717 | cqp->ts_shift = info->ts_shift; | ||
| 3718 | cqp->ts_override = info->ts_override; | ||
| 3719 | cqp->en_fine_grained_timers = info->en_fine_grained_timers; | ||
| 3720 | cqp->pe_en_vf_cnt = info->pe_en_vf_cnt; | ||
| 3721 | cqp->ooo_op_array = info->ooo_op_array; | ||
| 3722 | /* initialize the OOO lists */ | ||
| 3723 | INIT_LIST_HEAD(&cqp->ooo_avail); | ||
| 3724 | INIT_LIST_HEAD(&cqp->ooo_pnd); | ||
| 3725 | if (cqp->ooo_op_array) { | ||
| 3726 | /* Populate avail list entries */ | ||
| 3727 | for (num_ooo_ops = 0, ooo_op = info->ooo_op_array; | ||
| 3728 | num_ooo_ops < cqp->sq_size; | ||
| 3729 | num_ooo_ops++, ooo_op++) | ||
| 3730 | list_add(&ooo_op->list_entry, &cqp->ooo_avail); | ||
| 3731 | } | ||
| 3732 | } | ||
| 3121 | info->dev->cqp = cqp; | 3733 | info->dev->cqp = cqp; |
| 3122 | 3734 | ||
| 3123 | IRDMA_RING_INIT(cqp->sq_ring, cqp->sq_size); | 3735 | IRDMA_RING_INIT(cqp->sq_ring, cqp->sq_size); |
| 3736 | cqp->last_def_cmpl_ticket = 0; | ||
| 3737 | cqp->sw_def_cmpl_ticket = 0; | ||
| 3124 | cqp->requested_ops = 0; | 3738 | cqp->requested_ops = 0; |
| 3125 | atomic64_set(&cqp->completed_ops, 0); | 3739 | atomic64_set(&cqp->completed_ops, 0); |
| 3126 | /* for the cqp commands backlog. */ | 3740 | /* for the cqp commands backlog. */ |
| 3127 | INIT_LIST_HEAD(&cqp->dev->cqp_cmd_head); | 3741 | INIT_LIST_HEAD(&cqp->dev->cqp_cmd_head); |
| 3128 | 3742 | ||
| 3129 | writel(0, cqp->dev->hw_regs[IRDMA_CQPTAIL]); | 3743 | writel(0, cqp->dev->hw_regs[IRDMA_CQPTAIL]); |
| 3130 | writel(0, cqp->dev->hw_regs[IRDMA_CQPDB]); | 3744 | if (cqp->dev->hw_attrs.uk_attrs.hw_rev <= IRDMA_GEN_2) { |
| 3131 | writel(0, cqp->dev->hw_regs[IRDMA_CCQPSTATUS]); | 3745 | writel(0, cqp->dev->hw_regs[IRDMA_CQPDB]); |
| 3746 | writel(0, cqp->dev->hw_regs[IRDMA_CCQPSTATUS]); | ||
| 3747 | } | ||
| 3132 | 3748 | ||
| 3133 | ibdev_dbg(to_ibdev(cqp->dev), | 3749 | ibdev_dbg(to_ibdev(cqp->dev), |
| 3134 | "WQE: sq_size[%04d] hw_sq_size[%04d] sq_base[%p] sq_pa[%p] cqp[%p] polarity[x%04x]\n", | 3750 | "WQE: sq_size[%04d] hw_sq_size[%04d] sq_base[%p] sq_pa[%p] cqp[%p] polarity[x%04x]\n", |
| @@ -3160,6 +3776,7 @@ int irdma_sc_cqp_create(struct irdma_sc_cqp *cqp, u16 *maj_err, u16 *min_err) · 25 unmodified lines | |||
| 3160 | return -ENOMEM; | 3776 | return -ENOMEM; |
| 3161 | 3777 | ||
| 3162 | spin_lock_init(&cqp->dev->cqp_lock); | 3778 | spin_lock_init(&cqp->dev->cqp_lock); |
| 3779 | spin_lock_init(&cqp->ooo_list_lock); | ||
| 3163 | 3780 | ||
| 3164 | temp = FIELD_PREP(IRDMA_CQPHC_SQSIZE, cqp->hw_sq_size) | | 3781 | temp = FIELD_PREP(IRDMA_CQPHC_SQSIZE, cqp->hw_sq_size) | |
| 3165 | FIELD_PREP(IRDMA_CQPHC_SVER, cqp->struct_ver) | | 3782 | FIELD_PREP(IRDMA_CQPHC_SVER, cqp->struct_ver) | |
| @@ -3171,12 +3788,29 @@ int irdma_sc_cqp_create(struct irdma_sc_cqp *cqp, u16 *maj_err, u16 *min_err) · 5 unmodified lines | |||
| 3171 | FIELD_PREP(IRDMA_CQPHC_PROTOCOL_USED, | 3788 | FIELD_PREP(IRDMA_CQPHC_PROTOCOL_USED, |
| 3172 | cqp->protocol_used); | 3789 | cqp->protocol_used); |
| 3173 | } | 3790 | } |
| 3791 | if (hw_rev >= IRDMA_GEN_3) | ||
| 3792 | temp |= FIELD_PREP(IRDMA_CQPHC_EN_FINE_GRAINED_TIMERS, | ||
| 3793 | cqp->en_fine_grained_timers); | ||
| 3174 | 3794 | ||
| 3175 | set_64bit_val(cqp->host_ctx, 0, temp); | 3795 | set_64bit_val(cqp->host_ctx, 0, temp); |
| 3176 | set_64bit_val(cqp->host_ctx, 8, cqp->sq_pa); | 3796 | set_64bit_val(cqp->host_ctx, 8, cqp->sq_pa); |
| 3177 | 3797 | ||
| 3178 | temp = FIELD_PREP(IRDMA_CQPHC_ENABLED_VFS, cqp->ena_vf_count) | | 3798 | temp = FIELD_PREP(IRDMA_CQPHC_ENABLED_VFS, cqp->ena_vf_count) | |
| 3179 | FIELD_PREP(IRDMA_CQPHC_HMC_PROFILE, cqp->hmc_profile); | 3799 | FIELD_PREP(IRDMA_CQPHC_HMC_PROFILE, cqp->hmc_profile); |
| 3800 | |||
| 3801 | if (hw_rev >= IRDMA_GEN_3) | ||
| 3802 | temp |= FIELD_PREP(IRDMA_CQPHC_OOISC_BLKSIZE, | ||
| 3803 | cqp->ooisc_blksize) | | ||
| 3804 | FIELD_PREP(IRDMA_CQPHC_RRSP_BLKSIZE, | ||
| 3805 | cqp->rrsp_blksize) | | ||
| 3806 | FIELD_PREP(IRDMA_CQPHC_Q1_BLKSIZE, cqp->q1_blksize) | | ||
| 3807 | FIELD_PREP(IRDMA_CQPHC_XMIT_BLKSIZE, | ||
| 3808 | cqp->xmit_blksize) | | ||
| 3809 | FIELD_PREP(IRDMA_CQPHC_BLKSIZES_VALID, | ||
| 3810 | cqp->blksizes_valid) | | ||
| 3811 | FIELD_PREP(IRDMA_CQPHC_TIMESTAMP_OVERRIDE, | ||
| 3812 | cqp->ts_override) | | ||
| 3813 | FIELD_PREP(IRDMA_CQPHC_TS_SHIFT, cqp->ts_shift); | ||
| 3180 | set_64bit_val(cqp->host_ctx, 16, temp); | 3814 | set_64bit_val(cqp->host_ctx, 16, temp); |
| 3181 | set_64bit_val(cqp->host_ctx, 24, (uintptr_t)cqp); | 3815 | set_64bit_val(cqp->host_ctx, 24, (uintptr_t)cqp); |
| 3182 | temp = FIELD_PREP(IRDMA_CQPHC_HW_MAJVER, cqp->hw_maj_ver) | | 3816 | temp = FIELD_PREP(IRDMA_CQPHC_HW_MAJVER, cqp->hw_maj_ver) | |
| @@ -3337,6 +3971,87 @@ void irdma_sc_ccq_arm(struct irdma_sc_cq *ccq) · 154 unmodified lines | |||
| 3337 | writel(ccq->cq_uk.cq_id, ccq->dev->cq_arm_db); | 3971 | writel(ccq->cq_uk.cq_id, ccq->dev->cq_arm_db); |
| 3338 | } | 3972 | } |
| 3339 | 3973 | ||
| 3974 | /** | ||
| 3975 | * irdma_sc_process_def_cmpl - process deferred or pending completion | ||
| 3976 | * @cqp: CQP sc struct | ||
| 3977 | * @info: CQP CQE info | ||
| 3978 | * @wqe_idx: CQP WQE descriptor index | ||
| 3979 | * @def_info: deferred op ticket value or out-of-order completion id | ||
| 3980 | * @def_cmpl: true for deferred completion, false for pending (RCA) | ||
| 3981 | */ | ||
| 3982 | static void irdma_sc_process_def_cmpl(struct irdma_sc_cqp *cqp, | ||
| 3983 | struct irdma_ccq_cqe_info *info, | ||
| 3984 | u32 wqe_idx, u32 def_info, bool def_cmpl) | ||
| 3985 | { | ||
| 3986 | struct irdma_ooo_cqp_op *ooo_op; | ||
| 3987 | unsigned long flags; | ||
| 3988 | |||
| 3989 | /* Deferred and out-of-order completions share the same list of pending | ||
| 3990 | * completions. Since the list can be also accessed from AE handler, | ||
| 3991 | * it must be protected by a lock. | ||
| 3992 | */ | ||
| 3993 | spin_lock_irqsave(&cqp->ooo_list_lock, flags); | ||
| 3994 | |||
| 3995 | /* For deferred completions bump up SW completion ticket value. */ | ||
| 3996 | if (def_cmpl) { | ||
| 3997 | cqp->last_def_cmpl_ticket = def_info; | ||
| 3998 | cqp->sw_def_cmpl_ticket++; | ||
| 3999 | } | ||
| 4000 | if (!list_empty(&cqp->ooo_avail)) { | ||
| 4001 | ooo_op = (struct irdma_ooo_cqp_op *) | ||
| 4002 | list_entry(cqp->ooo_avail.next, | ||
| 4003 | struct irdma_ooo_cqp_op, list_entry); | ||
| 4004 | |||
| 4005 | list_del(&ooo_op->list_entry); | ||
| 4006 | ooo_op->scratch = info->scratch; | ||
| 4007 | ooo_op->def_info = def_info; | ||
| 4008 | ooo_op->sw_def_info = cqp->sw_def_cmpl_ticket; | ||
| 4009 | ooo_op->deferred = def_cmpl; | ||
| 4010 | ooo_op->wqe_idx = wqe_idx; | ||
| 4011 | /* Pending completions must be chronologically ordered, | ||
| 4012 | * so adding at the end of list. | ||
| 4013 | */ | ||
| 4014 | list_add_tail(&ooo_op->list_entry, &cqp->ooo_pnd); | ||
| 4015 | } | ||
| 4016 | spin_unlock_irqrestore(&cqp->ooo_list_lock, flags); | ||
| 4017 | |||
| 4018 | info->pending = true; | ||
| 4019 | } | ||
| 4020 | |||
| 4021 | /** | ||
| 4022 | * irdma_sc_process_ooo_cmpl - process out-of-order (final) completion | ||
| 4023 | * @cqp: CQP sc struct | ||
| 4024 | * @info: CQP CQE info | ||
| 4025 | * @def_info: out-of-order completion id | ||
| 4026 | */ | ||
| 4027 | static void irdma_sc_process_ooo_cmpl(struct irdma_sc_cqp *cqp, | ||
| 4028 | struct irdma_ccq_cqe_info *info, | ||
| 4029 | u32 def_info) | ||
| 4030 | { | ||
| 4031 | struct irdma_ooo_cqp_op *ooo_op_tmp; | ||
| 4032 | struct irdma_ooo_cqp_op *ooo_op; | ||
| 4033 | unsigned long flags; | ||
| 4034 | |||
| 4035 | info->scratch = 0; | ||
| 4036 | |||
| 4037 | spin_lock_irqsave(&cqp->ooo_list_lock, flags); | ||
| 4038 | list_for_each_entry_safe(ooo_op, ooo_op_tmp, &cqp->ooo_pnd, | ||
| 4039 | list_entry) { | ||
| 4040 | if (!ooo_op->deferred && ooo_op->def_info == def_info) { | ||
| 4041 | list_del(&ooo_op->list_entry); | ||
| 4042 | info->scratch = ooo_op->scratch; | ||
| 4043 | list_add(&ooo_op->list_entry, &cqp->ooo_avail); | ||
| 4044 | break; | ||
| 4045 | } | ||
| 4046 | } | ||
| 4047 | spin_unlock_irqrestore(&cqp->ooo_list_lock, flags); | ||
| 4048 | |||
| 4049 | if (!info->scratch) | ||
| 4050 | ibdev_dbg(to_ibdev(cqp->dev), | ||
| 4051 | "CQP: DEBUG_FW_OOO out-of-order completion with unknown def_info = 0x%x\n", | ||
| 4052 | def_info); | ||
| 4053 | } | ||
| 4054 | |||
| 3340 | /** | 4055 | /** |
| 3341 | * irdma_sc_ccq_get_cqe_info - get ccq's cq entry | 4056 | * irdma_sc_ccq_get_cqe_info - get ccq's cq entry |
| 3342 | * @ccq: ccq sc struct | 4057 | * @ccq: ccq sc struct |
| @@ -3345,6 +4060,10 @@ void irdma_sc_ccq_arm(struct irdma_sc_cq *ccq) · 2 unmodified lines | |||
| 3345 | int irdma_sc_ccq_get_cqe_info(struct irdma_sc_cq *ccq, | 4060 | int irdma_sc_ccq_get_cqe_info(struct irdma_sc_cq *ccq, |
| 3346 | struct irdma_ccq_cqe_info *info) | 4061 | struct irdma_ccq_cqe_info *info) |
| 3347 | { | 4062 | { |
| 4063 | u32 def_info; | ||
| 4064 | bool def_cmpl = false; | ||
| 4065 | bool pend_cmpl = false; | ||
| 4066 | bool ooo_final_cmpl = false; | ||
| 3348 | u64 qp_ctx, temp, temp1; | 4067 | u64 qp_ctx, temp, temp1; |
| 3349 | __le64 *cqe; | 4068 | __le64 *cqe; |
| 3350 | struct irdma_sc_cqp *cqp; | 4069 | struct irdma_sc_cqp *cqp; |
| @@ -3352,6 +4071,7 @@ int irdma_sc_ccq_get_cqe_info(struct irdma_sc_cq *ccq, · 1 unmodified line | |||
| 3352 | u32 error; | 4071 | u32 error; |
| 3353 | u8 polarity; | 4072 | u8 polarity; |
| 3354 | int ret_code = 0; | 4073 | int ret_code = 0; |
| 4074 | unsigned long flags; | ||
| 3355 | 4075 | ||
| 3356 | if (ccq->cq_uk.avoid_mem_cflct) | 4076 | if (ccq->cq_uk.avoid_mem_cflct) |
| 3357 | cqe = IRDMA_GET_CURRENT_EXTENDED_CQ_ELEM(&ccq->cq_uk); | 4077 | cqe = IRDMA_GET_CURRENT_EXTENDED_CQ_ELEM(&ccq->cq_uk); |
| @@ -3383,6 +4103,25 @@ int irdma_sc_ccq_get_cqe_info(struct irdma_sc_cq *ccq, · 25 unmodified lines | |||
| 3383 | 4103 | ||
| 3384 | get_64bit_val(cqe, 16, &temp1); | 4104 | get_64bit_val(cqe, 16, &temp1); |
| 3385 | info->op_ret_val = (u32)FIELD_GET(IRDMA_CCQ_OPRETVAL, temp1); | 4105 | info->op_ret_val = (u32)FIELD_GET(IRDMA_CCQ_OPRETVAL, temp1); |
| 4106 | if (cqp->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) { | ||
| 4107 | def_cmpl = info->maj_err_code == IRDMA_CQPSQ_MAJ_NO_ERROR && | ||
| 4108 | info->min_err_code == IRDMA_CQPSQ_MIN_DEF_CMPL; | ||
| 4109 | def_info = (u32)FIELD_GET(IRDMA_CCQ_DEFINFO, temp1); | ||
| 4110 | |||
| 4111 | pend_cmpl = info->maj_err_code == IRDMA_CQPSQ_MAJ_NO_ERROR && | ||
| 4112 | info->min_err_code == IRDMA_CQPSQ_MIN_OOO_CMPL; | ||
| 4113 | |||
| 4114 | ooo_final_cmpl = (bool)FIELD_GET(IRDMA_OOO_CMPL, temp); | ||
| 4115 | |||
| 4116 | if (def_cmpl || pend_cmpl || ooo_final_cmpl) { | ||
| 4117 | if (ooo_final_cmpl) | ||
| 4118 | irdma_sc_process_ooo_cmpl(cqp, info, def_info); | ||
| 4119 | else | ||
| 4120 | irdma_sc_process_def_cmpl(cqp, info, wqe_idx, | ||
| 4121 | def_info, def_cmpl); | ||
| 4122 | } | ||
| 4123 | } | ||
| 4124 | |||
| 3386 | get_64bit_val(cqp->sq_base[wqe_idx].elem, 24, &temp1); | 4125 | get_64bit_val(cqp->sq_base[wqe_idx].elem, 24, &temp1); |
| 3387 | info->op_code = (u8)FIELD_GET(IRDMA_CQPSQ_OPCODE, temp1); | 4126 | info->op_code = (u8)FIELD_GET(IRDMA_CQPSQ_OPCODE, temp1); |
| 3388 | info->cqp = cqp; | 4127 | info->cqp = cqp; |
| @@ -3399,7 +4138,16 @@ int irdma_sc_ccq_get_cqe_info(struct irdma_sc_cq *ccq, · 10 unmodified lines | |||
| 3399 | 4138 | ||
| 3400 | dma_wmb(); /* make sure shadow area is updated before moving tail */ | 4139 | dma_wmb(); /* make sure shadow area is updated before moving tail */ |
| 3401 | 4140 | ||
| 3402 | IRDMA_RING_MOVE_TAIL(cqp->sq_ring); | 4141 | spin_lock_irqsave(&cqp->dev->cqp_lock, flags); |
| 4142 | if (!ooo_final_cmpl) | ||
| 4143 | IRDMA_RING_MOVE_TAIL(cqp->sq_ring); | ||
| 4144 | spin_unlock_irqrestore(&cqp->dev->cqp_lock, flags); | ||
| 4145 | |||
| 4146 | /* Do not increment completed_ops counter on pending or deferred | ||
| 4147 | * completions. | ||
| 4148 | */ | ||
| 4149 | if (pend_cmpl || def_cmpl) | ||
| 4150 | return ret_code; | ||
| 3403 | atomic64_inc(&cqp->completed_ops); | 4151 | atomic64_inc(&cqp->completed_ops); |
| 3404 | 4152 | ||
| 3405 | return ret_code; | 4153 | return ret_code; |
| @@ -3647,7 +4395,7 @@ int irdma_sc_ceq_init(struct irdma_sc_ceq *ceq, · 241 unmodified lines | |||
| 3647 | ceq->pbl_list = (ceq->virtual_map ? info->pbl_list : NULL); | 4395 | ceq->pbl_list = (ceq->virtual_map ? info->pbl_list : NULL); |
| 3648 | ceq->tph_en = info->tph_en; | 4396 | ceq->tph_en = info->tph_en; |
| 3649 | ceq->tph_val = info->tph_val; | 4397 | ceq->tph_val = info->tph_val; |
| 3650 | ceq->vsi = info->vsi; | 4398 | ceq->vsi_idx = info->vsi_idx; |
| 3651 | ceq->polarity = 1; | 4399 | ceq->polarity = 1; |
| 3652 | IRDMA_RING_INIT(ceq->ceq_ring, ceq->elem_cnt); | 4400 | IRDMA_RING_INIT(ceq->ceq_ring, ceq->elem_cnt); |
| 3653 | ceq->dev->ceq[info->ceq_id] = ceq; | 4401 | ceq->dev->ceq[info->ceq_id] = ceq; |
| @@ -3680,13 +4428,16 @@ static int irdma_sc_ceq_create(struct irdma_sc_ceq *ceq, u64 scratch, · 26 unmodified lines | |||
| 3680 | (ceq->virtual_map ? ceq->first_pm_pbl_idx : 0)); | 4428 | (ceq->virtual_map ? ceq->first_pm_pbl_idx : 0)); |
| 3681 | set_64bit_val(wqe, 56, | 4429 | set_64bit_val(wqe, 56, |
| 3682 | FIELD_PREP(IRDMA_CQPSQ_TPHVAL, ceq->tph_val) | | 4430 | FIELD_PREP(IRDMA_CQPSQ_TPHVAL, ceq->tph_val) | |
| 3683 | FIELD_PREP(IRDMA_CQPSQ_VSIIDX, ceq->vsi->vsi_idx)); | 4431 | FIELD_PREP(IRDMA_CQPSQ_PASID, ceq->pasid) | |
| 4432 | FIELD_PREP(IRDMA_CQPSQ_VSIIDX, ceq->vsi_idx)); | ||
| 3684 | hdr = FIELD_PREP(IRDMA_CQPSQ_CEQ_CEQID, ceq->ceq_id) | | 4433 | hdr = FIELD_PREP(IRDMA_CQPSQ_CEQ_CEQID, ceq->ceq_id) | |
| 4434 | FIELD_PREP(IRDMA_CQPSQ_CEQ_CEQID_HIGH, ceq->ceq_id >> 10) | | ||
| 3685 | FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_CREATE_CEQ) | | 4435 | FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_CREATE_CEQ) | |
| 3686 | FIELD_PREP(IRDMA_CQPSQ_CEQ_LPBLSIZE, ceq->pbl_chunk_size) | | 4436 | FIELD_PREP(IRDMA_CQPSQ_CEQ_LPBLSIZE, ceq->pbl_chunk_size) | |
| 3687 | FIELD_PREP(IRDMA_CQPSQ_CEQ_VMAP, ceq->virtual_map) | | 4437 | FIELD_PREP(IRDMA_CQPSQ_CEQ_VMAP, ceq->virtual_map) | |
| 3688 | FIELD_PREP(IRDMA_CQPSQ_CEQ_ITRNOEXPIRE, ceq->itr_no_expire) | | 4438 | FIELD_PREP(IRDMA_CQPSQ_CEQ_ITRNOEXPIRE, ceq->itr_no_expire) | |
| 3689 | FIELD_PREP(IRDMA_CQPSQ_TPHEN, ceq->tph_en) | | 4439 | FIELD_PREP(IRDMA_CQPSQ_TPHEN, ceq->tph_en) | |
| 4440 | FIELD_PREP(IRDMA_CQPSQ_PASID_VALID, ceq->pasid_valid) | | ||
| 3690 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); | 4441 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); |
| 3691 | dma_wmb(); /* make sure WQE is written before valid bit is set */ | 4442 | dma_wmb(); /* make sure WQE is written before valid bit is set */ |
| 3692 | 4443 | ||
| @@ -3741,7 +4492,7 @@ int irdma_sc_cceq_create(struct irdma_sc_ceq *ceq, u64 scratch) · 48 unmodified lines | |||
| 3741 | int ret_code; | 4492 | int ret_code; |
| 3742 | struct irdma_sc_dev *dev = ceq->dev; | 4493 | struct irdma_sc_dev *dev = ceq->dev; |
| 3743 | 4494 | ||
| 3744 | dev->ccq->vsi = ceq->vsi; | 4495 | dev->ccq->vsi_idx = ceq->vsi_idx; |
| 3745 | if (ceq->reg_cq) { | 4496 | if (ceq->reg_cq) { |
| 3746 | ret_code = irdma_sc_add_cq_ctx(ceq, ceq->dev->ccq); | 4497 | ret_code = irdma_sc_add_cq_ctx(ceq, ceq->dev->ccq); |
| 3747 | if (ret_code) | 4498 | if (ret_code) |
| @@ -3774,11 +4525,14 @@ int irdma_sc_ceq_destroy(struct irdma_sc_ceq *ceq, u64 scratch, bool post_sq) · 26 unmodified lines | |||
| 3774 | 4525 | ||
| 3775 | set_64bit_val(wqe, 16, ceq->elem_cnt); | 4526 | set_64bit_val(wqe, 16, ceq->elem_cnt); |
| 3776 | set_64bit_val(wqe, 48, ceq->first_pm_pbl_idx); | 4527 | set_64bit_val(wqe, 48, ceq->first_pm_pbl_idx); |
| 4528 | set_64bit_val(wqe, 56, | ||
| 4529 | FIELD_PREP(IRDMA_CQPSQ_PASID, ceq->pasid)); | ||
| 3777 | hdr = ceq->ceq_id | | 4530 | hdr = ceq->ceq_id | |
| 3778 | FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_DESTROY_CEQ) | | 4531 | FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_DESTROY_CEQ) | |
| 3779 | FIELD_PREP(IRDMA_CQPSQ_CEQ_LPBLSIZE, ceq->pbl_chunk_size) | | 4532 | FIELD_PREP(IRDMA_CQPSQ_CEQ_LPBLSIZE, ceq->pbl_chunk_size) | |
| 3780 | FIELD_PREP(IRDMA_CQPSQ_CEQ_VMAP, ceq->virtual_map) | | 4533 | FIELD_PREP(IRDMA_CQPSQ_CEQ_VMAP, ceq->virtual_map) | |
| 3781 | FIELD_PREP(IRDMA_CQPSQ_TPHEN, ceq->tph_en) | | 4534 | FIELD_PREP(IRDMA_CQPSQ_TPHEN, ceq->tph_en) | |
| 4535 | FIELD_PREP(IRDMA_CQPSQ_PASID_VALID, ceq->pasid_valid) | | ||
| 3782 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); | 4536 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); |
| 3783 | dma_wmb(); /* make sure WQE is written before valid bit is set */ | 4537 | dma_wmb(); /* make sure WQE is written before valid bit is set */ |
| 3784 | 4538 | ||
| @@ -3942,10 +4696,13 @@ static int irdma_sc_aeq_create(struct irdma_sc_aeq *aeq, u64 scratch, · 157 unmodified lines | |||
| 3942 | (aeq->virtual_map ? 0 : aeq->aeq_elem_pa)); | 4696 | (aeq->virtual_map ? 0 : aeq->aeq_elem_pa)); |
| 3943 | set_64bit_val(wqe, 48, | 4697 | set_64bit_val(wqe, 48, |
| 3944 | (aeq->virtual_map ? aeq->first_pm_pbl_idx : 0)); | 4698 | (aeq->virtual_map ? aeq->first_pm_pbl_idx : 0)); |
| 4699 | set_64bit_val(wqe, 56, | ||
| 4700 | FIELD_PREP(IRDMA_CQPSQ_PASID, aeq->pasid)); | ||
| 3945 | 4701 | ||
| 3946 | hdr = FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_CREATE_AEQ) | | 4702 | hdr = FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_CREATE_AEQ) | |
| 3947 | FIELD_PREP(IRDMA_CQPSQ_AEQ_LPBLSIZE, aeq->pbl_chunk_size) | | 4703 | FIELD_PREP(IRDMA_CQPSQ_AEQ_LPBLSIZE, aeq->pbl_chunk_size) | |
| 3948 | FIELD_PREP(IRDMA_CQPSQ_AEQ_VMAP, aeq->virtual_map) | | 4704 | FIELD_PREP(IRDMA_CQPSQ_AEQ_VMAP, aeq->virtual_map) | |
| 4705 | FIELD_PREP(IRDMA_CQPSQ_PASID_VALID, aeq->pasid_valid) | | ||
| 3949 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); | 4706 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); |
| 3950 | dma_wmb(); /* make sure WQE is written before valid bit is set */ | 4707 | dma_wmb(); /* make sure WQE is written before valid bit is set */ |
| 3951 | 4708 | ||
| @@ -3974,7 +4731,8 @@ static int irdma_sc_aeq_destroy(struct irdma_sc_aeq *aeq, u64 scratch, · 22 unmodified lines | |||
| 3974 | u64 hdr; | 4731 | u64 hdr; |
| 3975 | 4732 | ||
| 3976 | dev = aeq->dev; | 4733 | dev = aeq->dev; |
| 3977 | writel(0, dev->hw_regs[IRDMA_PFINT_AEQCTL]); | 4734 | if (dev->privileged) |
| 4735 | writel(0, dev->hw_regs[IRDMA_PFINT_AEQCTL]); | ||
| 3978 | 4736 | ||
| 3979 | cqp = dev->cqp; | 4737 | cqp = dev->cqp; |
| 3980 | wqe = irdma_sc_cqp_get_next_send_wqe(cqp, scratch); | 4738 | wqe = irdma_sc_cqp_get_next_send_wqe(cqp, scratch); |
| @@ -3982,9 +4740,12 @@ static int irdma_sc_aeq_destroy(struct irdma_sc_aeq *aeq, u64 scratch, · 1 unmodified line | |||
| 3982 | return -ENOMEM; | 4740 | return -ENOMEM; |
| 3983 | set_64bit_val(wqe, 16, aeq->elem_cnt); | 4741 | set_64bit_val(wqe, 16, aeq->elem_cnt); |
| 3984 | set_64bit_val(wqe, 48, aeq->first_pm_pbl_idx); | 4742 | set_64bit_val(wqe, 48, aeq->first_pm_pbl_idx); |
| 4743 | set_64bit_val(wqe, 56, | ||
| 4744 | FIELD_PREP(IRDMA_CQPSQ_PASID, aeq->pasid)); | ||
| 3985 | hdr = FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_DESTROY_AEQ) | | 4745 | hdr = FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_DESTROY_AEQ) | |
| 3986 | FIELD_PREP(IRDMA_CQPSQ_AEQ_LPBLSIZE, aeq->pbl_chunk_size) | | 4746 | FIELD_PREP(IRDMA_CQPSQ_AEQ_LPBLSIZE, aeq->pbl_chunk_size) | |
| 3987 | FIELD_PREP(IRDMA_CQPSQ_AEQ_VMAP, aeq->virtual_map) | | 4747 | FIELD_PREP(IRDMA_CQPSQ_AEQ_VMAP, aeq->virtual_map) | |
| 4748 | FIELD_PREP(IRDMA_CQPSQ_PASID_VALID, aeq->pasid_valid) | | ||
| 3988 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); | 4749 | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); |
| 3989 | dma_wmb(); /* make sure WQE is written before valid bit is set */ | 4750 | dma_wmb(); /* make sure WQE is written before valid bit is set */ |
| 3990 | 4751 | ||
| @@ -4025,18 +4786,39 @@ int irdma_sc_get_next_aeqe(struct irdma_sc_aeq *aeq, · 34 unmodified lines | |||
| 4025 | print_hex_dump_debug("WQE: AEQ_ENTRY WQE", DUMP_PREFIX_OFFSET, 16, 8, | 4786 | print_hex_dump_debug("WQE: AEQ_ENTRY WQE", DUMP_PREFIX_OFFSET, 16, 8, |
| 4026 | aeqe, 16, false); | 4787 | aeqe, 16, false); |
| 4027 | 4788 | ||
| 4028 | ae_src = (u8)FIELD_GET(IRDMA_AEQE_AESRC, temp); | 4789 | if (aeq->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) { |
| 4029 | info->wqe_idx = (u16)FIELD_GET(IRDMA_AEQE_WQDESCIDX, temp); | 4790 | ae_src = (u8)FIELD_GET(IRDMA_AEQE_AESRC_GEN_3, temp); |
| 4030 | info->qp_cq_id = (u32)FIELD_GET(IRDMA_AEQE_QPCQID_LOW, temp) | | 4791 | info->wqe_idx = (u16)FIELD_GET(IRDMA_AEQE_WQDESCIDX_GEN_3, |
| 4792 | temp); | ||
| 4793 | info->qp_cq_id = (u32)FIELD_GET(IRDMA_AEQE_QPCQID_GEN_3, temp); | ||
| 4794 | info->ae_id = (u16)FIELD_GET(IRDMA_AEQE_AECODE_GEN_3, temp); | ||
| 4795 | info->tcp_state = (u8)FIELD_GET(IRDMA_AEQE_TCPSTATE_GEN_3, compl_ctx); | ||
| 4796 | info->iwarp_state = (u8)FIELD_GET(IRDMA_AEQE_IWSTATE_GEN_3, temp); | ||
| 4797 | info->q2_data_written = (u8)FIELD_GET(IRDMA_AEQE_Q2DATA_GEN_3, compl_ctx); | ||
| 4798 | info->aeqe_overflow = (bool)FIELD_GET(IRDMA_AEQE_OVERFLOW_GEN_3, temp); | ||
| 4799 | info->compl_ctx = FIELD_GET(IRDMA_AEQE_CMPL_CTXT, compl_ctx); | ||
| 4800 | compl_ctx = FIELD_GET(IRDMA_AEQE_CMPL_CTXT, compl_ctx) << IRDMA_AEQE_CMPL_CTXT_S; | ||
| 4801 | } else { | ||
| 4802 | ae_src = (u8)FIELD_GET(IRDMA_AEQE_AESRC, temp); | ||
| 4803 | info->wqe_idx = (u16)FIELD_GET(IRDMA_AEQE_WQDESCIDX, temp); | ||
| 4804 | info->qp_cq_id = (u32)FIELD_GET(IRDMA_AEQE_QPCQID_LOW, temp) | | ||
| 4031 | ((u32)FIELD_GET(IRDMA_AEQE_QPCQID_HI, temp) << 18); | 4805 | ((u32)FIELD_GET(IRDMA_AEQE_QPCQID_HI, temp) << 18); |
| 4032 | info->ae_id = (u16)FIELD_GET(IRDMA_AEQE_AECODE, temp); | 4806 | info->ae_id = (u16)FIELD_GET(IRDMA_AEQE_AECODE, temp); |
| 4033 | info->tcp_state = (u8)FIELD_GET(IRDMA_AEQE_TCPSTATE, temp); | 4807 | info->tcp_state = (u8)FIELD_GET(IRDMA_AEQE_TCPSTATE, temp); |
| 4034 | info->iwarp_state = (u8)FIELD_GET(IRDMA_AEQE_IWSTATE, temp); | 4808 | info->iwarp_state = (u8)FIELD_GET(IRDMA_AEQE_IWSTATE, temp); |
| 4035 | info->q2_data_written = (u8)FIELD_GET(IRDMA_AEQE_Q2DATA, temp); | 4809 | info->q2_data_written = (u8)FIELD_GET(IRDMA_AEQE_Q2DATA, temp); |
| 4036 | info->aeqe_overflow = (bool)FIELD_GET(IRDMA_AEQE_OVERFLOW, temp); | 4810 | info->aeqe_overflow = (bool)FIELD_GET(IRDMA_AEQE_OVERFLOW, |
| 4811 | temp); | ||
| 4812 | } | ||
| 4037 | 4813 | ||
| 4038 | info->ae_src = ae_src; | 4814 | info->ae_src = ae_src; |
| 4039 | switch (info->ae_id) { | 4815 | switch (info->ae_id) { |
| 4816 | case IRDMA_AE_SRQ_LIMIT: | ||
| 4817 | info->srq = true; | ||
| 4818 | /* [63:6] from CMPL_CTXT, [5:0] from WQDESCIDX. */ | ||
| 4819 | info->compl_ctx = compl_ctx; | ||
| 4820 | ae_src = IRDMA_AE_SOURCE_RSVD; | ||
| 4821 | break; | ||
| 4040 | case IRDMA_AE_PRIV_OPERATION_DENIED: | 4822 | case IRDMA_AE_PRIV_OPERATION_DENIED: |
| 4041 | case IRDMA_AE_AMP_INVALIDATE_TYPE1_MW: | 4823 | case IRDMA_AE_AMP_INVALIDATE_TYPE1_MW: |
| 4042 | case IRDMA_AE_AMP_MWBIND_ZERO_BASED_TYPE1_MW: | 4824 | case IRDMA_AE_AMP_MWBIND_ZERO_BASED_TYPE1_MW: |
| @@ -4069,6 +4851,10 @@ int irdma_sc_get_next_aeqe(struct irdma_sc_aeq *aeq, · 26 unmodified lines | |||
| 4069 | case IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR: | 4851 | case IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR: |
| 4070 | case IRDMA_AE_LLP_SEGMENT_TOO_SMALL: | 4852 | case IRDMA_AE_LLP_SEGMENT_TOO_SMALL: |
| 4071 | case IRDMA_AE_LLP_TOO_MANY_RETRIES: | 4853 | case IRDMA_AE_LLP_TOO_MANY_RETRIES: |
| 4854 | case IRDMA_AE_LLP_TOO_MANY_RNRS: | ||
| 4855 | case IRDMA_AE_REMOTE_QP_CATASTROPHIC: | ||
| 4856 | case IRDMA_AE_LOCAL_QP_CATASTROPHIC: | ||
| 4857 | case IRDMA_AE_RCE_QP_CATASTROPHIC: | ||
| 4072 | case IRDMA_AE_LLP_DOUBT_REACHABILITY: | 4858 | case IRDMA_AE_LLP_DOUBT_REACHABILITY: |
| 4073 | case IRDMA_AE_LLP_CONNECTION_ESTABLISHED: | 4859 | case IRDMA_AE_LLP_CONNECTION_ESTABLISHED: |
| 4074 | case IRDMA_AE_RESET_SENT: | 4860 | case IRDMA_AE_RESET_SENT: |
| @@ -4085,6 +4871,10 @@ int irdma_sc_get_next_aeqe(struct irdma_sc_aeq *aeq, · 10 unmodified lines | |||
| 4085 | info->compl_ctx = compl_ctx << 1; | 4871 | info->compl_ctx = compl_ctx << 1; |
| 4086 | ae_src = IRDMA_AE_SOURCE_RSVD; | 4872 | ae_src = IRDMA_AE_SOURCE_RSVD; |
| 4087 | break; | 4873 | break; |
| 4874 | case IRDMA_AE_CQP_DEFERRED_COMPLETE: | ||
| 4875 | info->def_info = info->wqe_idx; | ||
| 4876 | ae_src = IRDMA_AE_SOURCE_RSVD; | ||
| 4877 | break; | ||
| 4088 | case IRDMA_AE_ROCE_EMPTY_MCG: | 4878 | case IRDMA_AE_ROCE_EMPTY_MCG: |
| 4089 | case IRDMA_AE_ROCE_BAD_MC_IP_ADDR: | 4879 | case IRDMA_AE_ROCE_BAD_MC_IP_ADDR: |
| 4090 | case IRDMA_AE_ROCE_BAD_MC_QPID: | 4880 | case IRDMA_AE_ROCE_BAD_MC_QPID: |
| @@ -4110,6 +4900,7 @@ int irdma_sc_get_next_aeqe(struct irdma_sc_aeq *aeq, · 19 unmodified lines | |||
| 4110 | info->qp = true; | 4900 | info->qp = true; |
| 4111 | info->rq = true; | 4901 | info->rq = true; |
| 4112 | info->compl_ctx = compl_ctx; | 4902 | info->compl_ctx = compl_ctx; |
| 4903 | info->err_rq_idx_valid = true; | ||
| 4113 | break; | 4904 | break; |
| 4114 | case IRDMA_AE_SOURCE_CQ: | 4905 | case IRDMA_AE_SOURCE_CQ: |
| 4115 | case IRDMA_AE_SOURCE_CQ_0110: | 4906 | case IRDMA_AE_SOURCE_CQ_0110: |
| @@ -4125,8 +4916,18 @@ int irdma_sc_get_next_aeqe(struct irdma_sc_aeq *aeq, · 9 unmodified lines | |||
| 4125 | info->compl_ctx = compl_ctx; | 4916 | info->compl_ctx = compl_ctx; |
| 4126 | break; | 4917 | break; |
| 4127 | case IRDMA_AE_SOURCE_IN_RR_WR: | 4918 | case IRDMA_AE_SOURCE_IN_RR_WR: |
| 4919 | info->qp = true; | ||
| 4920 | if (aeq->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) | ||
| 4921 | info->err_rq_idx_valid = true; | ||
| 4922 | info->compl_ctx = compl_ctx; | ||
| 4923 | info->in_rdrsp_wr = true; | ||
| 4924 | break; | ||
| 4128 | case IRDMA_AE_SOURCE_IN_RR_WR_1011: | 4925 | case IRDMA_AE_SOURCE_IN_RR_WR_1011: |
| 4129 | info->qp = true; | 4926 | info->qp = true; |
| 4927 | if (aeq->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) { | ||
| 4928 | info->sq = true; | ||
| 4929 | info->err_rq_idx_valid = true; | ||
| 4930 | } | ||
| 4130 | info->compl_ctx = compl_ctx; | 4931 | info->compl_ctx = compl_ctx; |
| 4131 | info->in_rdrsp_wr = true; | 4932 | info->in_rdrsp_wr = true; |
| 4132 | break; | 4933 | break; |
| @@ -4335,6 +5136,26 @@ int irdma_sc_init_iw_hmc(struct irdma_sc_dev *dev, u8 hmc_fn_id) · 202 unmodified lines | |||
| 4335 | return ret_code; | 5136 | return ret_code; |
| 4336 | } | 5137 | } |
| 4337 | 5138 | ||
| 5139 | /** | ||
| 5140 | * irdma_set_loc_mem() - set a local memory bit field | ||
| 5141 | * @buf: ptr to a buffer where local memory gets enabled | ||
| 5142 | */ | ||
| 5143 | static void irdma_set_loc_mem(__le64 *buf) | ||
| 5144 | { | ||
| 5145 | u64 loc_mem_en = BIT_ULL(ENABLE_LOC_MEM); | ||
| 5146 | u32 offset; | ||
| 5147 | u64 temp; | ||
| 5148 | |||
| 5149 | for (offset = 0; offset < IRDMA_COMMIT_FPM_BUF_SIZE; | ||
| 5150 | offset += sizeof(__le64)) { | ||
| 5151 | if (offset == IRDMA_PBLE_COMMIT_OFFSET) | ||
| 5152 | continue; | ||
| 5153 | get_64bit_val(buf, offset, &temp); | ||
| 5154 | if (temp) | ||
| 5155 | set_64bit_val(buf, offset, temp | loc_mem_en); | ||
| 5156 | } | ||
| 5157 | } | ||
| 5158 | |||
| 4338 | /** | 5159 | /** |
| 4339 | * irdma_sc_cfg_iw_fpm() - commits hmc obj cnt values using cqp | 5160 | * irdma_sc_cfg_iw_fpm() - commits hmc obj cnt values using cqp |
| 4340 | * command and populates fpm base address in hmc_info | 5161 | * command and populates fpm base address in hmc_info |
| @@ -4356,7 +5177,7 @@ static int irdma_sc_cfg_iw_fpm(struct irdma_sc_dev *dev, u8 hmc_fn_id) · 15 unmodified lines | |||
| 4356 | 5177 | ||
| 4357 | set_64bit_val(buf, 0, (u64)obj_info[IRDMA_HMC_IW_QP].cnt); | 5178 | set_64bit_val(buf, 0, (u64)obj_info[IRDMA_HMC_IW_QP].cnt); |
| 4358 | set_64bit_val(buf, 8, (u64)obj_info[IRDMA_HMC_IW_CQ].cnt); | 5179 | set_64bit_val(buf, 8, (u64)obj_info[IRDMA_HMC_IW_CQ].cnt); |
| 4359 | set_64bit_val(buf, 16, (u64)0); /* RSRVD */ | 5180 | set_64bit_val(buf, 16, (u64)obj_info[IRDMA_HMC_IW_SRQ].cnt); |
| 4360 | set_64bit_val(buf, 24, (u64)obj_info[IRDMA_HMC_IW_HTE].cnt); | 5181 | set_64bit_val(buf, 24, (u64)obj_info[IRDMA_HMC_IW_HTE].cnt); |
| 4361 | set_64bit_val(buf, 32, (u64)obj_info[IRDMA_HMC_IW_ARP].cnt); | 5182 | set_64bit_val(buf, 32, (u64)obj_info[IRDMA_HMC_IW_ARP].cnt); |
| 4362 | set_64bit_val(buf, 40, (u64)0); /* RSVD */ | 5183 | set_64bit_val(buf, 40, (u64)0); /* RSVD */ |
| @@ -4383,7 +5204,9 @@ static int irdma_sc_cfg_iw_fpm(struct irdma_sc_dev *dev, u8 hmc_fn_id) · 20 unmodified lines | |||
| 4383 | (u64)obj_info[IRDMA_HMC_IW_OOISC].cnt); | 5204 | (u64)obj_info[IRDMA_HMC_IW_OOISC].cnt); |
| 4384 | set_64bit_val(buf, 168, | 5205 | set_64bit_val(buf, 168, |
| 4385 | (u64)obj_info[IRDMA_HMC_IW_OOISCFFL].cnt); | 5206 | (u64)obj_info[IRDMA_HMC_IW_OOISCFFL].cnt); |
| 4386 | 5207 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3 && | |
| 5208 | dev->hmc_fpm_misc.loc_mem_pages) | ||
| 5209 | irdma_set_loc_mem(buf); | ||
| 4387 | commit_fpm_mem.pa = dev->fpm_commit_buf_pa; | 5210 | commit_fpm_mem.pa = dev->fpm_commit_buf_pa; |
| 4388 | commit_fpm_mem.va = dev->fpm_commit_buf; | 5211 | commit_fpm_mem.va = dev->fpm_commit_buf; |
| 4389 | 5212 | ||
| @@ -4592,6 +5415,7 @@ static bool irdma_cqp_ring_full(struct irdma_sc_cqp *cqp) · 202 unmodified lines | |||
| 4592 | static u32 irdma_est_sd(struct irdma_sc_dev *dev, | 5415 | static u32 irdma_est_sd(struct irdma_sc_dev *dev, |
| 4593 | struct irdma_hmc_info *hmc_info) | 5416 | struct irdma_hmc_info *hmc_info) |
| 4594 | { | 5417 | { |
| 5418 | struct irdma_hmc_obj_info *pble_info; | ||
| 4595 | int i; | 5419 | int i; |
| 4596 | u64 size = 0; | 5420 | u64 size = 0; |
| 4597 | u64 sd; | 5421 | u64 sd; |
| @@ -4600,12 +5424,22 @@ static u32 irdma_est_sd(struct irdma_sc_dev *dev, · 2 unmodified lines | |||
| 4600 | if (i != IRDMA_HMC_IW_PBLE) | 5424 | if (i != IRDMA_HMC_IW_PBLE) |
| 4601 | size += round_up(hmc_info->hmc_obj[i].cnt * | 5425 | size += round_up(hmc_info->hmc_obj[i].cnt * |
| 4602 | hmc_info->hmc_obj[i].size, 512); | 5426 | hmc_info->hmc_obj[i].size, 512); |
| 4603 | size += round_up(hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].cnt * | 5427 | |
| 4604 | hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].size, 512); | 5428 | pble_info = &hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE]; |
| 5429 | if (dev->privileged) | ||
| 5430 | size += round_up(pble_info->cnt * pble_info->size, 512); | ||
| 4605 | if (size & 0x1FFFFF) | 5431 | if (size & 0x1FFFFF) |
| 4606 | sd = (size >> 21) + 1; /* add 1 for remainder */ | 5432 | sd = (size >> 21) + 1; /* add 1 for remainder */ |
| 4607 | else | 5433 | else |
| 4608 | sd = size >> 21; | 5434 | sd = size >> 21; |
| 5435 | if (!dev->privileged && !dev->hmc_fpm_misc.loc_mem_pages) { | ||
| 5436 | /* 2MB alignment for VF PBLE HMC */ | ||
| 5437 | size = pble_info->cnt * pble_info->size; | ||
| 5438 | if (size & 0x1FFFFF) | ||
| 5439 | sd += (size >> 21) + 1; /* add 1 for remainder */ | ||
| 5440 | else | ||
| 5441 | sd += size >> 21; | ||
| 5442 | } | ||
| 4609 | if (sd > 0xFFFFFFFF) { | 5443 | if (sd > 0xFFFFFFFF) { |
| 4610 | ibdev_dbg(to_ibdev(dev), "HMC: sd overflow[%lld]\n", sd); | 5444 | ibdev_dbg(to_ibdev(dev), "HMC: sd overflow[%lld]\n", sd); |
| 4611 | sd = 0xFFFFFFFF - 1; | 5445 | sd = 0xFFFFFFFF - 1; |
| @@ -4614,17 +5448,6 @@ static u32 irdma_est_sd(struct irdma_sc_dev *dev, · 2 unmodified lines | |||
| 4614 | return (u32)sd; | 5448 | return (u32)sd; |
| 4615 | } | 5449 | } |
| 4616 | 5450 | ||
| 4617 | /** | ||
| 4618 | * irdma_sc_query_rdma_features_done - poll cqp for query features done | ||
| 4619 | * @cqp: struct for cqp hw | ||
| 4620 | */ | ||
| 4621 | static int irdma_sc_query_rdma_features_done(struct irdma_sc_cqp *cqp) | ||
| 4622 | { | ||
| 4623 | return irdma_sc_poll_for_cqp_op_done(cqp, | ||
| 4624 | IRDMA_CQP_OP_QUERY_RDMA_FEATURES, | ||
| 4625 | NULL); | ||
| 4626 | } | ||
| 4627 | |||
| 4628 | /** | 5451 | /** |
| 4629 | * irdma_sc_query_rdma_features - query RDMA features and FW ver | 5452 | * irdma_sc_query_rdma_features - query RDMA features and FW ver |
| 4630 | * @cqp: struct for cqp hw | 5453 | * @cqp: struct for cqp hw |
| @@ -4634,7 +5457,9 @@ static int irdma_sc_query_rdma_features_done(struct irdma_sc_cqp *cqp) · 3 unmodified lines | |||
| 4634 | static int irdma_sc_query_rdma_features(struct irdma_sc_cqp *cqp, | 5457 | static int irdma_sc_query_rdma_features(struct irdma_sc_cqp *cqp, |
| 4635 | struct irdma_dma_mem *buf, u64 scratch) | 5458 | struct irdma_dma_mem *buf, u64 scratch) |
| 4636 | { | 5459 | { |
| 5460 | u32 tail, val, error; | ||
| 4637 | __le64 *wqe; | 5461 | __le64 *wqe; |
| 5462 | int status; | ||
| 4638 | u64 temp; | 5463 | u64 temp; |
| 4639 | 5464 | ||
| 4640 | wqe = irdma_sc_cqp_get_next_send_wqe(cqp, scratch); | 5465 | wqe = irdma_sc_cqp_get_next_send_wqe(cqp, scratch); |
| @@ -4654,9 +5479,15 @@ static int irdma_sc_query_rdma_features(struct irdma_sc_cqp *cqp, · 13 unmodified lines | |||
| 4654 | 5479 | ||
| 4655 | print_hex_dump_debug("WQE: QUERY RDMA FEATURES", DUMP_PREFIX_OFFSET, | 5480 | print_hex_dump_debug("WQE: QUERY RDMA FEATURES", DUMP_PREFIX_OFFSET, |
| 4656 | 16, 8, wqe, IRDMA_CQP_WQE_SIZE * 8, false); | 5481 | 16, 8, wqe, IRDMA_CQP_WQE_SIZE * 8, false); |
| 5482 | irdma_get_cqp_reg_info(cqp, &val, &tail, &error); | ||
| 5483 | |||
| 4657 | irdma_sc_cqp_post_sq(cqp); | 5484 | irdma_sc_cqp_post_sq(cqp); |
| 5485 | status = irdma_cqp_poll_registers(cqp, tail, | ||
| 5486 | cqp->dev->hw_attrs.max_done_count); | ||
| 5487 | if (error || status) | ||
| 5488 | status = -EINVAL; | ||
| 4658 | 5489 | ||
| 4659 | return 0; | 5490 | return status; |
| 4660 | } | 5491 | } |
| 4661 | 5492 | ||
| 4662 | /** | 5493 | /** |
| @@ -4678,8 +5509,6 @@ int irdma_get_rdma_features(struct irdma_sc_dev *dev) · 15 unmodified lines | |||
| 4678 | return -ENOMEM; | 5509 | return -ENOMEM; |
| 4679 | 5510 | ||
| 4680 | ret_code = irdma_sc_query_rdma_features(dev->cqp, &feat_buf, 0); | 5511 | ret_code = irdma_sc_query_rdma_features(dev->cqp, &feat_buf, 0); |
| 4681 | if (!ret_code) | ||
| 4682 | ret_code = irdma_sc_query_rdma_features_done(dev->cqp); | ||
| 4683 | if (ret_code) | 5512 | if (ret_code) |
| 4684 | goto exit; | 5513 | goto exit; |
| 4685 | 5514 | ||
| @@ -4703,8 +5532,6 @@ int irdma_get_rdma_features(struct irdma_sc_dev *dev) · 17 unmodified lines | |||
| 4703 | return -ENOMEM; | 5532 | return -ENOMEM; |
| 4704 | 5533 | ||
| 4705 | ret_code = irdma_sc_query_rdma_features(dev->cqp, &feat_buf, 0); | 5534 | ret_code = irdma_sc_query_rdma_features(dev->cqp, &feat_buf, 0); |
| 4706 | if (!ret_code) | ||
| 4707 | ret_code = irdma_sc_query_rdma_features_done(dev->cqp); | ||
| 4708 | if (ret_code) | 5535 | if (ret_code) |
| 4709 | goto exit; | 5536 | goto exit; |
| 4710 | 5537 | ||
| @@ -4731,6 +5558,10 @@ int irdma_get_rdma_features(struct irdma_sc_dev *dev) · 20 unmodified lines | |||
| 4731 | } | 5558 | } |
| 4732 | dev->feature_info[feat_type] = temp; | 5559 | dev->feature_info[feat_type] = temp; |
| 4733 | } | 5560 | } |
| 5561 | |||
| 5562 | if (dev->feature_info[IRDMA_FTN_FLAGS] & IRDMA_ATOMICS_ALLOWED_BIT) | ||
| 5563 | dev->hw_attrs.uk_attrs.feature_flags |= IRDMA_FEATURE_ATOMIC_OPS; | ||
| 5564 | |||
| 4734 | exit: | 5565 | exit: |
| 4735 | dma_free_coherent(dev->hw->device, feat_buf.size, feat_buf.va, | 5566 | dma_free_coherent(dev->hw->device, feat_buf.size, feat_buf.va, |
| 4736 | feat_buf.pa); | 5567 | feat_buf.pa); |
| @@ -4785,6 +5616,339 @@ static void cfg_fpm_value_gen_2(struct irdma_sc_dev *dev, · 48 unmodified lines | |||
| 4785 | hmc_fpm_misc->ooiscf_block_size; | 5616 | hmc_fpm_misc->ooiscf_block_size; |
| 4786 | } | 5617 | } |
| 4787 | 5618 | ||
| 5619 | /** | ||
| 5620 | * irdma_get_rsrc_mem_config - configure resources if local memory or host | ||
| 5621 | * @dev: sc device struct | ||
| 5622 | * @is_mrte_loc_mem: if true, MR's to be in local memory because sd=loc pages | ||
| 5623 | * | ||
| 5624 | * Only mr can be configured host or local memory if qp's are in local memory. | ||
| 5625 | * If qp is in local memory, then all resource object will be in local memory | ||
| 5626 | * except mr which can be either host or local memory. The only exception | ||
| 5627 | * is pble's which are always in host memory. | ||
| 5628 | */ | ||
| 5629 | static void irdma_get_rsrc_mem_config(struct irdma_sc_dev *dev, bool is_mrte_loc_mem) | ||
| 5630 | { | ||
| 5631 | struct irdma_hmc_info *hmc_info = dev->hmc_info; | ||
| 5632 | int i; | ||
| 5633 | |||
| 5634 | for (i = IRDMA_HMC_IW_QP; i < IRDMA_HMC_IW_MAX; i++) | ||
| 5635 | hmc_info->hmc_obj[i].mem_loc = IRDMA_LOC_MEM; | ||
| 5636 | |||
| 5637 | if (dev->feature_info[IRDMA_OBJ_1] && !is_mrte_loc_mem) { | ||
| 5638 | u8 mem_type; | ||
| 5639 | |||
| 5640 | mem_type = (u8)FIELD_GET(IRDMA_MR_MEM_LOC, dev->feature_info[IRDMA_OBJ_1]); | ||
| 5641 | |||
| 5642 | hmc_info->hmc_obj[IRDMA_HMC_IW_MR].mem_loc = | ||
| 5643 | (mem_type & IRDMA_OBJ_LOC_MEM_BIT) ? | ||
| 5644 | IRDMA_LOC_MEM : IRDMA_HOST_MEM; | ||
| 5645 | } else { | ||
| 5646 | hmc_info->hmc_obj[IRDMA_HMC_IW_MR].mem_loc = IRDMA_LOC_MEM; | ||
| 5647 | } | ||
| 5648 | |||
| 5649 | hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].mem_loc = IRDMA_HOST_MEM; | ||
| 5650 | |||
| 5651 | ibdev_dbg(to_ibdev(dev), "HMC: INFO: mrte_mem_loc = %d pble = %d\n", | ||
| 5652 | hmc_info->hmc_obj[IRDMA_HMC_IW_MR].mem_loc, | ||
| 5653 | hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].mem_loc); | ||
| 5654 | } | ||
| 5655 | |||
| 5656 | /** | ||
| 5657 | * irdma_cfg_sd_mem - allocate sd memory | ||
| 5658 | * @dev: sc device struct | ||
| 5659 | * @hmc_info: ptr to irdma_hmc_obj_info struct | ||
| 5660 | */ | ||
| 5661 | static int irdma_cfg_sd_mem(struct irdma_sc_dev *dev, | ||
| 5662 | struct irdma_hmc_info *hmc_info) | ||
| 5663 | { | ||
| 5664 | struct irdma_virt_mem virt_mem; | ||
| 5665 | u32 mem_size; | ||
| 5666 | |||
| 5667 | mem_size = sizeof(struct irdma_hmc_sd_entry) * hmc_info->sd_table.sd_cnt; | ||
| 5668 | virt_mem.size = mem_size; | ||
| 5669 | virt_mem.va = kzalloc(virt_mem.size, GFP_KERNEL); | ||
| 5670 | if (!virt_mem.va) | ||
| 5671 | return -ENOMEM; | ||
| 5672 | hmc_info->sd_table.sd_entry = virt_mem.va; | ||
| 5673 | |||
| 5674 | return 0; | ||
| 5675 | } | ||
| 5676 | |||
| 5677 | /** | ||
| 5678 | * irdma_get_objs_pages - get number of 2M pages needed | ||
| 5679 | * @dev: sc device struct | ||
| 5680 | * @hmc_info: pointer to the HMC configuration information struct | ||
| 5681 | * @mem_loc: pages for local or host memory | ||
| 5682 | */ | ||
| 5683 | static u32 irdma_get_objs_pages(struct irdma_sc_dev *dev, | ||
| 5684 | struct irdma_hmc_info *hmc_info, | ||
| 5685 | enum irdma_hmc_obj_mem mem_loc) | ||
| 5686 | { | ||
| 5687 | u64 size = 0; | ||
| 5688 | int i; | ||
| 5689 | |||
| 5690 | for (i = IRDMA_HMC_IW_QP; i < IRDMA_HMC_IW_MAX; i++) { | ||
| 5691 | if (hmc_info->hmc_obj[i].mem_loc == mem_loc) { | ||
| 5692 | size += round_up(hmc_info->hmc_obj[i].cnt * | ||
| 5693 | hmc_info->hmc_obj[i].size, 512); | ||
| 5694 | } | ||
| 5695 | } | ||
| 5696 | |||
| 5697 | return DIV_ROUND_UP(size, IRDMA_HMC_PAGE_SIZE); | ||
| 5698 | } | ||
| 5699 | |||
| 5700 | /** | ||
| 5701 | * irdma_set_host_hmc_rsrc_gen_3 - calculate host hmc resources for gen 3 | ||
| 5702 | * @dev: sc device struct | ||
| 5703 | */ | ||
| 5704 | static void irdma_set_host_hmc_rsrc_gen_3(struct irdma_sc_dev *dev) | ||
| 5705 | { | ||
| 5706 | struct irdma_hmc_fpm_misc *hmc_fpm_misc; | ||
| 5707 | struct irdma_hmc_info *hmc_info; | ||
| 5708 | enum irdma_hmc_obj_mem mrte_loc; | ||
| 5709 | u32 mrwanted, pblewanted; | ||
| 5710 | u32 avail_sds, mr_sds; | ||
| 5711 | |||
| 5712 | hmc_info = dev->hmc_info; | ||
| 5713 | hmc_fpm_misc = &dev->hmc_fpm_misc; | ||
| 5714 | avail_sds = hmc_fpm_misc->max_sds; | ||
| 5715 | mrte_loc = hmc_info->hmc_obj[IRDMA_HMC_IW_MR].mem_loc; | ||
| 5716 | mrwanted = hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt; | ||
| 5717 | pblewanted = hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].max_cnt; | ||
| 5718 | |||
| 5719 | if (mrte_loc == IRDMA_HOST_MEM && avail_sds > IRDMA_MIN_PBLE_PAGES) { | ||
| 5720 | mr_sds = avail_sds - IRDMA_MIN_PBLE_PAGES; | ||
| 5721 | mrwanted = min(mrwanted, mr_sds * MAX_MR_PER_SD); | ||
| 5722 | hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt = mrwanted; | ||
| 5723 | avail_sds -= DIV_ROUND_UP(mrwanted, MAX_MR_PER_SD); | ||
| 5724 | } | ||
| 5725 | |||
| 5726 | if (FIELD_GET(IRDMA_MANAGE_RSRC_VER2, dev->feature_info[IRDMA_FTN_FLAGS]) && | ||
| 5727 | pblewanted > avail_sds * MAX_PBLE_PER_SD) | ||
| 5728 | ibdev_dbg(to_ibdev(dev), | ||
| 5729 | "HMC: Warn: Resource version 2: pble wanted = 0x%x available = 0x%x\n", | ||
| 5730 | pblewanted, avail_sds * MAX_PBLE_PER_SD); | ||
| 5731 | |||
| 5732 | pblewanted = min(pblewanted, avail_sds * MAX_PBLE_PER_SD); | ||
| 5733 | hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].cnt = pblewanted; | ||
| 5734 | } | ||
| 5735 | |||
| 5736 | /** | ||
| 5737 | * irdma_verify_commit_fpm_gen_3 - verify query fpm values | ||
| 5738 | * @dev: sc device struct | ||
| 5739 | * @max_pages: max local memory available | ||
| 5740 | * @qpwanted: number of qp's wanted | ||
| 5741 | */ | ||
| 5742 | static int irdma_verify_commit_fpm_gen_3(struct irdma_sc_dev *dev, | ||
| 5743 | u32 max_pages, | ||
| 5744 | u32 qpwanted) | ||
| 5745 | { | ||
| 5746 | struct irdma_hmc_fpm_misc *hmc_fpm_misc; | ||
| 5747 | u32 rrf_cnt, xf_cnt, timer_cnt, pages_needed; | ||
| 5748 | struct irdma_hmc_info *hmc_info; | ||
| 5749 | u32 rrffl_cnt = 0; | ||
| 5750 | u32 xffl_cnt = 0; | ||
| 5751 | u32 q1fl_cnt; | ||
| 5752 | |||
| 5753 | hmc_info = dev->hmc_info; | ||
| 5754 | hmc_fpm_misc = &dev->hmc_fpm_misc; | ||
| 5755 | |||
| 5756 | rrf_cnt = roundup_pow_of_two(IRDMA_RRF_MULTIPLIER * qpwanted); | ||
| 5757 | |||
| 5758 | if (hmc_info->hmc_obj[IRDMA_HMC_IW_RRFFL].max_cnt) | ||
| 5759 | rrffl_cnt = | ||
| 5760 | hmc_info->hmc_obj[IRDMA_HMC_IW_RRF].cnt / | ||
| 5761 | hmc_fpm_misc->rrf_block_size; | ||
| 5762 | |||
| 5763 | xf_cnt = roundup_pow_of_two(IRDMA_XF_MULTIPLIER * qpwanted); | ||
| 5764 | |||
| 5765 | if (xf_cnt) | ||
| 5766 | xffl_cnt = xf_cnt / hmc_fpm_misc->xf_block_size; | ||
| 5767 | |||
| 5768 | timer_cnt = (round_up(qpwanted, 512) / 512 + 1) * | ||
| 5769 | hmc_fpm_misc->timer_bucket; | ||
| 5770 | |||
| 5771 | q1fl_cnt = hmc_info->hmc_obj[IRDMA_HMC_IW_Q1].cnt / hmc_fpm_misc->q1_block_size; | ||
| 5772 | |||
| 5773 | pages_needed = irdma_get_objs_pages(dev, hmc_info, IRDMA_LOC_MEM); | ||
| 5774 | if (pages_needed > max_pages) { | ||
| 5775 | ibdev_dbg(to_ibdev(dev), | ||
| 5776 | "HMC: FAIL: SW counts rrf_cnt = %u rrffl_cnt = %u timer_cnt = %u", | ||
| 5777 | rrf_cnt, rrffl_cnt, timer_cnt); | ||
| 5778 | ibdev_dbg(to_ibdev(dev), | ||
| 5779 | "HMC: FAIL: SW counts xf_cnt = %u xffl_cnt = %u q1fl_cnt = %u", | ||
| 5780 | xf_cnt, xffl_cnt, q1fl_cnt); | ||
| 5781 | |||
| 5782 | return -EINVAL; | ||
| 5783 | } | ||
| 5784 | |||
| 5785 | hmc_fpm_misc->max_sds -= pages_needed; | ||
| 5786 | hmc_fpm_misc->loc_mem_pages -= pages_needed; | ||
| 5787 | |||
| 5788 | return 0; | ||
| 5789 | } | ||
| 5790 | |||
| 5791 | /** | ||
| 5792 | * irdma_set_loc_hmc_rsrc_gen_3 - calculate hmc resources for gen 3 | ||
| 5793 | * @dev: sc device struct | ||
| 5794 | * @max_pages: max local memory available | ||
| 5795 | * @qpwanted: number of qp's wanted | ||
| 5796 | */ | ||
| 5797 | static int irdma_set_loc_hmc_rsrc_gen_3(struct irdma_sc_dev *dev, | ||
| 5798 | u32 max_pages, | ||
| 5799 | u32 qpwanted) | ||
| 5800 | { | ||
| 5801 | struct irdma_hmc_fpm_misc *hmc_fpm_misc; | ||
| 5802 | u32 rrf_cnt, xf_cnt, timer_cnt, pages_needed; | ||
| 5803 | struct irdma_hmc_info *hmc_info; | ||
| 5804 | u32 ird, ord; | ||
| 5805 | |||
| 5806 | if (FIELD_GET(IRDMA_MANAGE_RSRC_VER2, dev->feature_info[IRDMA_FTN_FLAGS])) | ||
| 5807 | return irdma_verify_commit_fpm_gen_3(dev, max_pages, qpwanted); | ||
| 5808 | |||
| 5809 | hmc_info = dev->hmc_info; | ||
| 5810 | hmc_fpm_misc = &dev->hmc_fpm_misc; | ||
| 5811 | ird = dev->hw_attrs.max_hw_ird; | ||
| 5812 | ord = dev->hw_attrs.max_hw_ord; | ||
| 5813 | |||
| 5814 | hmc_info->hmc_obj[IRDMA_HMC_IW_HDR].cnt = qpwanted; | ||
| 5815 | hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt = qpwanted; | ||
| 5816 | |||
| 5817 | hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt = | ||
| 5818 | min(hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt, qpwanted * 2); | ||
| 5819 | |||
| 5820 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].cnt = | ||
| 5821 | min(qpwanted * 8, hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].max_cnt); | ||
| 5822 | |||
| 5823 | rrf_cnt = roundup_pow_of_two(IRDMA_RRF_MULTIPLIER * qpwanted); | ||
| 5824 | hmc_info->hmc_obj[IRDMA_HMC_IW_RRF].cnt = | ||
| 5825 | min(hmc_info->hmc_obj[IRDMA_HMC_IW_RRF].max_cnt, rrf_cnt); | ||
| 5826 | |||
| 5827 | if (hmc_info->hmc_obj[IRDMA_HMC_IW_RRFFL].max_cnt) | ||
| 5828 | hmc_info->hmc_obj[IRDMA_HMC_IW_RRFFL].cnt = | ||
| 5829 | hmc_info->hmc_obj[IRDMA_HMC_IW_RRF].cnt / | ||
| 5830 | hmc_fpm_misc->rrf_block_size; | ||
| 5831 | |||
| 5832 | xf_cnt = roundup_pow_of_two(IRDMA_XF_MULTIPLIER * qpwanted); | ||
| 5833 | hmc_info->hmc_obj[IRDMA_HMC_IW_XF].cnt = | ||
| 5834 | min(hmc_info->hmc_obj[IRDMA_HMC_IW_XF].max_cnt, xf_cnt); | ||
| 5835 | hmc_info->hmc_obj[IRDMA_HMC_IW_XFFL].cnt = | ||
| 5836 | xf_cnt / hmc_fpm_misc->xf_block_size; | ||
| 5837 | |||
| 5838 | timer_cnt = (round_up(qpwanted, 512) / 512 + 1) * | ||
| 5839 | hmc_fpm_misc->timer_bucket; | ||
| 5840 | hmc_info->hmc_obj[IRDMA_HMC_IW_TIMER].cnt = | ||
| 5841 | min(timer_cnt, hmc_info->hmc_obj[IRDMA_HMC_IW_TIMER].cnt); | ||
| 5842 | |||
| 5843 | do { | ||
| 5844 | hmc_info->hmc_obj[IRDMA_HMC_IW_Q1].cnt = roundup_pow_of_two(ird * 2 * qpwanted); | ||
| 5845 | hmc_info->hmc_obj[IRDMA_HMC_IW_Q1FL].cnt = | ||
| 5846 | hmc_info->hmc_obj[IRDMA_HMC_IW_Q1].cnt / hmc_fpm_misc->q1_block_size; | ||
| 5847 | |||
| 5848 | pages_needed = irdma_get_objs_pages(dev, hmc_info, IRDMA_LOC_MEM); | ||
| 5849 | if (pages_needed <= max_pages) | ||
| 5850 | break; | ||
| 5851 | |||
| 5852 | ird /= 2; | ||
| 5853 | ord /= 2; | ||
| 5854 | } while (ird >= IRDMA_MIN_IRD); | ||
| 5855 | |||
| 5856 | if (ird < IRDMA_MIN_IRD) { | ||
| 5857 | ibdev_dbg(to_ibdev(dev), "HMC: FAIL: IRD=%u Q1 CNT = %u\n", | ||
| 5858 | ird, hmc_info->hmc_obj[IRDMA_HMC_IW_Q1].cnt); | ||
| 5859 | return -EINVAL; | ||
| 5860 | } | ||
| 5861 | |||
| 5862 | dev->hw_attrs.max_hw_ird = ird; | ||
| 5863 | dev->hw_attrs.max_hw_ord = ord; | ||
| 5864 | hmc_fpm_misc->max_sds -= pages_needed; | ||
| 5865 | |||
| 5866 | return 0; | ||
| 5867 | } | ||
| 5868 | |||
| 5869 | /** | ||
| 5870 | * cfg_fpm_value_gen_3 - configure fpm for gen 3 | ||
| 5871 | * @dev: sc device struct | ||
| 5872 | * @hmc_info: ptr to irdma_hmc_obj_info struct | ||
| 5873 | * @hmc_fpm_misc: ptr to fpm data | ||
| 5874 | */ | ||
| 5875 | static int cfg_fpm_value_gen_3(struct irdma_sc_dev *dev, | ||
| 5876 | struct irdma_hmc_info *hmc_info, | ||
| 5877 | struct irdma_hmc_fpm_misc *hmc_fpm_misc) | ||
| 5878 | { | ||
| 5879 | enum irdma_hmc_obj_mem mrte_loc; | ||
| 5880 | u32 mrwanted, qpwanted; | ||
| 5881 | int i, ret_code = 0; | ||
| 5882 | u32 loc_mem_pages; | ||
| 5883 | bool is_mrte_loc_mem; | ||
| 5884 | |||
| 5885 | loc_mem_pages = hmc_fpm_misc->loc_mem_pages; | ||
| 5886 | is_mrte_loc_mem = hmc_fpm_misc->loc_mem_pages == hmc_fpm_misc->max_sds ? | ||
| 5887 | true : false; | ||
| 5888 | |||
| 5889 | irdma_get_rsrc_mem_config(dev, is_mrte_loc_mem); | ||
| 5890 | mrte_loc = hmc_info->hmc_obj[IRDMA_HMC_IW_MR].mem_loc; | ||
| 5891 | |||
| 5892 | if (is_mrte_loc_mem) | ||
| 5893 | loc_mem_pages -= IRDMA_MIN_PBLE_PAGES; | ||
| 5894 | |||
| 5895 | ibdev_dbg(to_ibdev(dev), | ||
| 5896 | "HMC: mrte_loc %d loc_mem %u fpm max sds %u host_obj %d\n", | ||
| 5897 | hmc_info->hmc_obj[IRDMA_HMC_IW_MR].mem_loc, | ||
| 5898 | hmc_fpm_misc->loc_mem_pages, hmc_fpm_misc->max_sds, | ||
| 5899 | is_mrte_loc_mem); | ||
| 5900 | |||
| 5901 | mrwanted = hmc_info->hmc_obj[IRDMA_HMC_IW_MR].max_cnt; | ||
| 5902 | qpwanted = hmc_info->hmc_obj[IRDMA_HMC_IW_QP].max_cnt; | ||
| 5903 | hmc_info->hmc_obj[IRDMA_HMC_IW_HDR].cnt = qpwanted; | ||
| 5904 | |||
| 5905 | hmc_info->hmc_obj[IRDMA_HMC_IW_OOISC].max_cnt = 0; | ||
| 5906 | hmc_info->hmc_obj[IRDMA_HMC_IW_OOISCFFL].max_cnt = 0; | ||
| 5907 | hmc_info->hmc_obj[IRDMA_HMC_IW_HTE].max_cnt = 0; | ||
| 5908 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].max_cnt = 0; | ||
| 5909 | |||
| 5910 | if (!FIELD_GET(IRDMA_MANAGE_RSRC_VER2, dev->feature_info[IRDMA_FTN_FLAGS])) | ||
| 5911 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].max_cnt = | ||
| 5912 | min(hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].max_cnt, | ||
| 5913 | (u32)IRDMA_FSIAV_CNT_MAX); | ||
| 5914 | |||
| 5915 | for (i = IRDMA_HMC_IW_QP; i < IRDMA_HMC_IW_MAX; i++) | ||
| 5916 | hmc_info->hmc_obj[i].cnt = hmc_info->hmc_obj[i].max_cnt; | ||
| 5917 | |||
| 5918 | while (qpwanted >= IRDMA_MIN_QP_CNT) { | ||
| 5919 | if (!irdma_set_loc_hmc_rsrc_gen_3(dev, loc_mem_pages, qpwanted)) | ||
| 5920 | break; | ||
| 5921 | |||
| 5922 | if (FIELD_GET(IRDMA_MANAGE_RSRC_VER2, dev->feature_info[IRDMA_FTN_FLAGS])) | ||
| 5923 | return -EINVAL; | ||
| 5924 | |||
| 5925 | qpwanted /= 2; | ||
| 5926 | if (mrte_loc == IRDMA_LOC_MEM) { | ||
| 5927 | mrwanted = qpwanted * IRDMA_MIN_MR_PER_QP; | ||
| 5928 | hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt = | ||
| 5929 | min(hmc_info->hmc_obj[IRDMA_HMC_IW_MR].max_cnt, mrwanted); | ||
| 5930 | } | ||
| 5931 | } | ||
| 5932 | |||
| 5933 | if (qpwanted < IRDMA_MIN_QP_CNT) { | ||
| 5934 | ibdev_dbg(to_ibdev(dev), | ||
| 5935 | "HMC: ERROR: could not allocate fpm resources\n"); | ||
| 5936 | return -EINVAL; | ||
| 5937 | } | ||
| 5938 | |||
| 5939 | irdma_set_host_hmc_rsrc_gen_3(dev); | ||
| 5940 | ret_code = irdma_sc_cfg_iw_fpm(dev, dev->hmc_fn_id); | ||
| 5941 | if (ret_code) { | ||
| 5942 | ibdev_dbg(to_ibdev(dev), | ||
| 5943 | "HMC: cfg_iw_fpm returned error_code[x%08X]\n", | ||
| 5944 | readl(dev->hw_regs[IRDMA_CQPERRCODES])); | ||
| 5945 | |||
| 5946 | return ret_code; | ||
| 5947 | } | ||
| 5948 | |||
| 5949 | return irdma_cfg_sd_mem(dev, hmc_info); | ||
| 5950 | } | ||
| 5951 | |||
| 4788 | /** | 5952 | /** |
| 4789 | * irdma_cfg_fpm_val - configure HMC objects | 5953 | * irdma_cfg_fpm_val - configure HMC objects |
| 4790 | * @dev: sc device struct | 5954 | * @dev: sc device struct |
| @@ -4792,16 +5956,15 @@ static void cfg_fpm_value_gen_2(struct irdma_sc_dev *dev, · 1 unmodified line | |||
| 4792 | */ | 5956 | */ |
| 4793 | int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) | 5957 | int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) |
| 4794 | { | 5958 | { |
| 4795 | struct irdma_virt_mem virt_mem; | ||
| 4796 | u32 i, mem_size; | ||
| 4797 | u32 qpwanted, mrwanted, pblewanted; | 5959 | u32 qpwanted, mrwanted, pblewanted; |
| 4798 | u32 powerof2, hte; | 5960 | u32 powerof2, hte, i; |
| 4799 | u32 sd_needed; | 5961 | u32 sd_needed; |
| 4800 | u32 sd_diff; | 5962 | u32 sd_diff; |
| 4801 | u32 loop_count = 0; | 5963 | u32 loop_count = 0; |
| 4802 | struct irdma_hmc_info *hmc_info; | 5964 | struct irdma_hmc_info *hmc_info; |
| 4803 | struct irdma_hmc_fpm_misc *hmc_fpm_misc; | 5965 | struct irdma_hmc_fpm_misc *hmc_fpm_misc; |
| 4804 | int ret_code = 0; | 5966 | int ret_code = 0; |
| 5967 | u32 max_sds; | ||
| 4805 | 5968 | ||
| 4806 | hmc_info = dev->hmc_info; | 5969 | hmc_info = dev->hmc_info; |
| 4807 | hmc_fpm_misc = &dev->hmc_fpm_misc; | 5970 | hmc_fpm_misc = &dev->hmc_fpm_misc; |
| @@ -4814,14 +5977,16 @@ int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) · 6 unmodified lines | |||
| 4814 | return ret_code; | 5977 | return ret_code; |
| 4815 | } | 5978 | } |
| 4816 | 5979 | ||
| 5980 | max_sds = hmc_fpm_misc->max_sds; | ||
| 5981 | |||
| 5982 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) | ||
| 5983 | return cfg_fpm_value_gen_3(dev, hmc_info, hmc_fpm_misc); | ||
| 5984 | |||
| 4817 | for (i = IRDMA_HMC_IW_QP; i < IRDMA_HMC_IW_MAX; i++) | 5985 | for (i = IRDMA_HMC_IW_QP; i < IRDMA_HMC_IW_MAX; i++) |
| 4818 | hmc_info->hmc_obj[i].cnt = hmc_info->hmc_obj[i].max_cnt; | 5986 | hmc_info->hmc_obj[i].cnt = hmc_info->hmc_obj[i].max_cnt; |
| 4819 | sd_needed = irdma_est_sd(dev, hmc_info); | 5987 | sd_needed = irdma_est_sd(dev, hmc_info); |
| 4820 | ibdev_dbg(to_ibdev(dev), | 5988 | ibdev_dbg(to_ibdev(dev), "HMC: sd count %u where max sd is %u\n", |
| 4821 | "HMC: FW max resources sd_needed[%08d] first_sd_index[%04d]\n", | 5989 | hmc_info->sd_table.sd_cnt, max_sds); |
| 4822 | sd_needed, hmc_info->first_sd_index); | ||
| 4823 | ibdev_dbg(to_ibdev(dev), "HMC: sd count %d where max sd is %d\n", | ||
| 4824 | hmc_info->sd_table.sd_cnt, hmc_fpm_misc->max_sds); | ||
| 4825 | 5990 | ||
| 4826 | qpwanted = min(qp_count, hmc_info->hmc_obj[IRDMA_HMC_IW_QP].max_cnt); | 5991 | qpwanted = min(qp_count, hmc_info->hmc_obj[IRDMA_HMC_IW_QP].max_cnt); |
| 4827 | 5992 | ||
| @@ -4835,21 +6000,21 @@ int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) · 7 unmodified lines | |||
| 4835 | pblewanted = hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].max_cnt; | 6000 | pblewanted = hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].max_cnt; |
| 4836 | 6001 | ||
| 4837 | ibdev_dbg(to_ibdev(dev), | 6002 | ibdev_dbg(to_ibdev(dev), |
| 4838 | "HMC: req_qp=%d max_sd=%d, max_qp = %d, max_cq=%d, max_mr=%d, max_pble=%d, mc=%d, av=%d\n", | 6003 | "HMC: req_qp=%d max_sd=%u, max_qp = %u, max_cq=%u, max_mr=%u, max_pble=%u, mc=%d, av=%u\n", |
| 4839 | qp_count, hmc_fpm_misc->max_sds, | 6004 | qp_count, max_sds, |
| 4840 | hmc_info->hmc_obj[IRDMA_HMC_IW_QP].max_cnt, | 6005 | hmc_info->hmc_obj[IRDMA_HMC_IW_QP].max_cnt, |
| 4841 | hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].max_cnt, | 6006 | hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].max_cnt, |
| 4842 | hmc_info->hmc_obj[IRDMA_HMC_IW_MR].max_cnt, | 6007 | hmc_info->hmc_obj[IRDMA_HMC_IW_MR].max_cnt, |
| 4843 | hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].max_cnt, | 6008 | hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].max_cnt, |
| 4844 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].max_cnt, | 6009 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].max_cnt, |
| 4845 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].max_cnt); | 6010 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].max_cnt); |
| 6011 | |||
| 4846 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].cnt = | 6012 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].cnt = |
| 4847 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].max_cnt; | 6013 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].max_cnt; |
| 4848 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].cnt = | 6014 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].cnt = |
| 4849 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].max_cnt; | 6015 | hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].max_cnt; |
| 4850 | hmc_info->hmc_obj[IRDMA_HMC_IW_ARP].cnt = | 6016 | hmc_info->hmc_obj[IRDMA_HMC_IW_ARP].cnt = |
| 4851 | hmc_info->hmc_obj[IRDMA_HMC_IW_ARP].max_cnt; | 6017 | hmc_info->hmc_obj[IRDMA_HMC_IW_ARP].max_cnt; |
| 4852 | |||
| 4853 | hmc_info->hmc_obj[IRDMA_HMC_IW_APBVT_ENTRY].cnt = 1; | 6018 | hmc_info->hmc_obj[IRDMA_HMC_IW_APBVT_ENTRY].cnt = 1; |
| 4854 | 6019 | ||
| 4855 | while (irdma_q1_cnt(dev, hmc_info, qpwanted) > hmc_info->hmc_obj[IRDMA_HMC_IW_Q1].max_cnt) | 6020 | while (irdma_q1_cnt(dev, hmc_info, qpwanted) > hmc_info->hmc_obj[IRDMA_HMC_IW_Q1].max_cnt) |
| @@ -4860,7 +6025,7 @@ int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) · 4 unmodified lines | |||
| 4860 | hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt = qpwanted; | 6025 | hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt = qpwanted; |
| 4861 | hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt = | 6026 | hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt = |
| 4862 | min(2 * qpwanted, hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt); | 6027 | min(2 * qpwanted, hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt); |
| 4863 | hmc_info->hmc_obj[IRDMA_HMC_IW_RESERVED].cnt = 0; /* Reserved */ | 6028 | hmc_info->hmc_obj[IRDMA_HMC_IW_SRQ].cnt = 0; /* Reserved */ |
| 4864 | hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt = mrwanted; | 6029 | hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt = mrwanted; |
| 4865 | 6030 | ||
| 4866 | hte = round_up(qpwanted + hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].cnt, 512); | 6031 | hte = round_up(qpwanted + hmc_info->hmc_obj[IRDMA_HMC_IW_FSIMC].cnt, 512); |
| @@ -4898,11 +6063,12 @@ int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) · 31 unmodified lines | |||
| 4898 | if (!(loop_count % 2) && qpwanted > 128) { | 6063 | if (!(loop_count % 2) && qpwanted > 128) { |
| 4899 | qpwanted /= 2; | 6064 | qpwanted /= 2; |
| 4900 | } else { | 6065 | } else { |
| 4901 | mrwanted /= 2; | ||
| 4902 | pblewanted /= 2; | 6066 | pblewanted /= 2; |
| 6067 | mrwanted /= 2; | ||
| 4903 | } | 6068 | } |
| 4904 | continue; | 6069 | continue; |
| 4905 | } | 6070 | } |
| 6071 | |||
| 4906 | if (dev->cqp->hmc_profile != IRDMA_HMC_PROFILE_FAVOR_VF && | 6072 | if (dev->cqp->hmc_profile != IRDMA_HMC_PROFILE_FAVOR_VF && |
| 4907 | pblewanted > (512 * FPM_MULTIPLIER * sd_diff)) { | 6073 | pblewanted > (512 * FPM_MULTIPLIER * sd_diff)) { |
| 4908 | pblewanted -= 256 * FPM_MULTIPLIER * sd_diff; | 6074 | pblewanted -= 256 * FPM_MULTIPLIER * sd_diff; |
| @@ -4928,14 +6094,13 @@ int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) · 19 unmodified lines | |||
| 4928 | 6094 | ||
| 4929 | if (sd_needed > hmc_fpm_misc->max_sds) { | 6095 | if (sd_needed > hmc_fpm_misc->max_sds) { |
| 4930 | ibdev_dbg(to_ibdev(dev), | 6096 | ibdev_dbg(to_ibdev(dev), |
| 4931 | "HMC: cfg_fpm failed loop_cnt=%d, sd_needed=%d, max sd count %d\n", | 6097 | "HMC: cfg_fpm failed loop_cnt=%u, sd_needed=%u, max sd count %u\n", |
| 4932 | loop_count, sd_needed, hmc_info->sd_table.sd_cnt); | 6098 | loop_count, sd_needed, hmc_info->sd_table.sd_cnt); |
| 4933 | return -EINVAL; | 6099 | return -EINVAL; |
| 4934 | } | 6100 | } |
| 4935 | 6101 | ||
| 4936 | if (loop_count > 1 && sd_needed < hmc_fpm_misc->max_sds) { | 6102 | if (loop_count > 1 && sd_needed < max_sds) { |
| 4937 | pblewanted += (hmc_fpm_misc->max_sds - sd_needed) * 256 * | 6103 | pblewanted += (max_sds - sd_needed) * 256 * FPM_MULTIPLIER; |
| 4938 | FPM_MULTIPLIER; | ||
| 4939 | hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].cnt = pblewanted; | 6104 | hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].cnt = pblewanted; |
| 4940 | sd_needed = irdma_est_sd(dev, hmc_info); | 6105 | sd_needed = irdma_est_sd(dev, hmc_info); |
| 4941 | } | 6106 | } |
| @@ -4959,18 +6124,7 @@ int irdma_cfg_fpm_val(struct irdma_sc_dev *dev, u32 qp_count) · 17 unmodified lines | |||
| 4959 | return ret_code; | 6124 | return ret_code; |
| 4960 | } | 6125 | } |
| 4961 | 6126 | ||
| 4962 | mem_size = sizeof(struct irdma_hmc_sd_entry) * | 6127 | return irdma_cfg_sd_mem(dev, hmc_info); |
| 4963 | (hmc_info->sd_table.sd_cnt + hmc_info->first_sd_index + 1); | ||
| 4964 | virt_mem.size = mem_size; | ||
| 4965 | virt_mem.va = kzalloc(virt_mem.size, GFP_KERNEL); | ||
| 4966 | if (!virt_mem.va) { | ||
| 4967 | ibdev_dbg(to_ibdev(dev), | ||
| 4968 | "HMC: failed to allocate memory for sd_entry buffer\n"); | ||
| 4969 | return -ENOMEM; | ||
| 4970 | } | ||
| 4971 | hmc_info->sd_table.sd_entry = virt_mem.va; | ||
| 4972 | |||
| 4973 | return ret_code; | ||
| 4974 | } | 6128 | } |
| 4975 | 6129 | ||
| 4976 | /** | 6130 | /** |
| @@ -5242,6 +6396,22 @@ static int irdma_exec_cqp_cmd(struct irdma_sc_dev *dev, · 265 unmodified lines | |||
| 5242 | &pcmdinfo->in.u.mc_modify.info, | 6396 | &pcmdinfo->in.u.mc_modify.info, |
| 5243 | pcmdinfo->in.u.mc_modify.scratch); | 6397 | pcmdinfo->in.u.mc_modify.scratch); |
| 5244 | break; | 6398 | break; |
| 6399 | case IRDMA_OP_SRQ_CREATE: | ||
| 6400 | status = irdma_sc_srq_create(pcmdinfo->in.u.srq_create.srq, | ||
| 6401 | pcmdinfo->in.u.srq_create.scratch, | ||
| 6402 | pcmdinfo->post_sq); | ||
| 6403 | break; | ||
| 6404 | case IRDMA_OP_SRQ_MODIFY: | ||
| 6405 | status = irdma_sc_srq_modify(pcmdinfo->in.u.srq_modify.srq, | ||
| 6406 | &pcmdinfo->in.u.srq_modify.info, | ||
| 6407 | pcmdinfo->in.u.srq_modify.scratch, | ||
| 6408 | pcmdinfo->post_sq); | ||
| 6409 | break; | ||
| 6410 | case IRDMA_OP_SRQ_DESTROY: | ||
| 6411 | status = irdma_sc_srq_destroy(pcmdinfo->in.u.srq_destroy.srq, | ||
| 6412 | pcmdinfo->in.u.srq_destroy.scratch, | ||
| 6413 | pcmdinfo->post_sq); | ||
| 6414 | break; | ||
| 5245 | default: | 6415 | default: |
| 5246 | status = -EOPNOTSUPP; | 6416 | status = -EOPNOTSUPP; |
| 5247 | break; | 6417 | break; |
| @@ -5314,14 +6484,26 @@ void irdma_cfg_aeq(struct irdma_sc_dev *dev, u32 idx, bool enable) · 66 unmodified lines | |||
| 5314 | */ | 6484 | */ |
| 5315 | void sc_vsi_update_stats(struct irdma_sc_vsi *vsi) | 6485 | void sc_vsi_update_stats(struct irdma_sc_vsi *vsi) |
| 5316 | { | 6486 | { |
| 5317 | struct irdma_gather_stats *gather_stats; | 6487 | struct irdma_dev_hw_stats *hw_stats = &vsi->pestat->hw_stats; |
| 5318 | struct irdma_gather_stats *last_gather_stats; | 6488 | struct irdma_gather_stats *gather_stats = |
| 6489 | vsi->pestat->gather_info.gather_stats_va; | ||
| 6490 | struct irdma_gather_stats *last_gather_stats = | ||
| 6491 | vsi->pestat->gather_info.last_gather_stats_va; | ||
| 6492 | const struct irdma_hw_stat_map *map = vsi->dev->hw_stats_map; | ||
| 6493 | u16 max_stat_idx = vsi->dev->hw_attrs.max_stat_idx; | ||
| 6494 | u16 i; | ||
| 6495 | |||
| 6496 | if (vsi->dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) { | ||
| 6497 | for (i = 0; i < max_stat_idx; i++) { | ||
| 6498 | u16 idx = map[i].byteoff / sizeof(u64); | ||
| 5319 | 6499 | ||
| 5320 | gather_stats = vsi->pestat->gather_info.gather_stats_va; | 6500 | hw_stats->stats_val[i] = gather_stats->val[idx]; |
| 5321 | last_gather_stats = vsi->pestat->gather_info.last_gather_stats_va; | 6501 | } |
| 5322 | irdma_update_stats(&vsi->pestat->hw_stats, gather_stats, | 6502 | return; |
| 5323 | last_gather_stats, vsi->dev->hw_stats_map, | 6503 | } |
| 5324 | vsi->dev->hw_attrs.max_stat_idx); | 6504 | |
| 6505 | irdma_update_stats(hw_stats, gather_stats, last_gather_stats, | ||
| 6506 | map, max_stat_idx); | ||
| 5325 | } | 6507 | } |
| 5326 | 6508 | ||
| 5327 | /** | 6509 | /** |
| @@ -5356,6 +6538,9 @@ static inline void irdma_sc_init_hw(struct irdma_sc_dev *dev) · 28 unmodified lines | |||
| 5356 | case IRDMA_GEN_2: | 6538 | case IRDMA_GEN_2: |
| 5357 | icrdma_init_hw(dev); | 6539 | icrdma_init_hw(dev); |
| 5358 | break; | 6540 | break; |
| 6541 | case IRDMA_GEN_3: | ||
| 6542 | ig3rdma_init_hw(dev); | ||
| 6543 | break; | ||
| 5359 | } | 6544 | } |
| 5360 | } | 6545 | } |
| 5361 | 6546 | ||
| @@ -5381,10 +6566,15 @@ int irdma_sc_dev_init(enum irdma_vers ver, struct irdma_sc_dev *dev, · 19 unmodified lines | |||
| 5381 | dev->fpm_commit_buf = info->fpm_commit_buf; | 6566 | dev->fpm_commit_buf = info->fpm_commit_buf; |
| 5382 | dev->hw = info->hw; | 6567 | dev->hw = info->hw; |
| 5383 | dev->hw->hw_addr = info->bar0; | 6568 | dev->hw->hw_addr = info->bar0; |
| 6569 | dev->protocol_used = info->protocol_used; | ||
| 5384 | /* Setup the hardware limits, hmc may limit further */ | 6570 | /* Setup the hardware limits, hmc may limit further */ |
| 5385 | dev->hw_attrs.min_hw_qp_id = IRDMA_MIN_IW_QP_ID; | 6571 | dev->hw_attrs.min_hw_qp_id = IRDMA_MIN_IW_QP_ID; |
| 6572 | dev->hw_attrs.min_hw_srq_id = IRDMA_MIN_IW_SRQ_ID; | ||
| 5386 | dev->hw_attrs.min_hw_aeq_size = IRDMA_MIN_AEQ_ENTRIES; | 6573 | dev->hw_attrs.min_hw_aeq_size = IRDMA_MIN_AEQ_ENTRIES; |
| 5387 | dev->hw_attrs.max_hw_aeq_size = IRDMA_MAX_AEQ_ENTRIES; | 6574 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) |
| 6575 | dev->hw_attrs.max_hw_aeq_size = IRDMA_MAX_AEQ_ENTRIES_GEN_3; | ||
| 6576 | else | ||
| 6577 | dev->hw_attrs.max_hw_aeq_size = IRDMA_MAX_AEQ_ENTRIES; | ||
| 5388 | dev->hw_attrs.min_hw_ceq_size = IRDMA_MIN_CEQ_ENTRIES; | 6578 | dev->hw_attrs.min_hw_ceq_size = IRDMA_MIN_CEQ_ENTRIES; |
| 5389 | dev->hw_attrs.max_hw_ceq_size = IRDMA_MAX_CEQ_ENTRIES; | 6579 | dev->hw_attrs.max_hw_ceq_size = IRDMA_MAX_CEQ_ENTRIES; |
| 5390 | dev->hw_attrs.uk_attrs.min_hw_cq_size = IRDMA_MIN_CQ_SIZE; | 6580 | dev->hw_attrs.uk_attrs.min_hw_cq_size = IRDMA_MIN_CQ_SIZE; |
| @@ -5409,21 +6599,39 @@ int irdma_sc_dev_init(enum irdma_vers ver, struct irdma_sc_dev *dev, · 18 unmodified lines | |||
| 5409 | dev->hw_attrs.max_sleep_count = IRDMA_SLEEP_COUNT; | 6599 | dev->hw_attrs.max_sleep_count = IRDMA_SLEEP_COUNT; |
| 5410 | dev->hw_attrs.max_cqp_compl_wait_time_ms = CQP_COMPL_WAIT_TIME_MS; | 6600 | dev->hw_attrs.max_cqp_compl_wait_time_ms = CQP_COMPL_WAIT_TIME_MS; |
| 5411 | 6601 | ||
| 5412 | dev->hw_attrs.uk_attrs.hw_rev = ver; | 6602 | if (!dev->privileged) { |
| 6603 | ret_code = irdma_vchnl_req_get_hmc_fcn(dev); | ||
| 6604 | if (ret_code) { | ||
| 6605 | ibdev_dbg(to_ibdev(dev), | ||
| 6606 | "DEV: Get HMC function ret = %d\n", | ||
| 6607 | ret_code); | ||
| 6608 | |||
| 6609 | return ret_code; | ||
| 6610 | } | ||
| 6611 | } | ||
| 6612 | |||
| 5413 | irdma_sc_init_hw(dev); | 6613 | irdma_sc_init_hw(dev); |
| 5414 | 6614 | ||
| 5415 | if (irdma_wait_pe_ready(dev)) | 6615 | if (dev->privileged) { |
| 5416 | return -ETIMEDOUT; | 6616 | if (irdma_wait_pe_ready(dev)) |
| 6617 | return -ETIMEDOUT; | ||
| 5417 | 6618 | ||
| 5418 | val = readl(dev->hw_regs[IRDMA_GLPCI_LBARCTRL]); | 6619 | val = readl(dev->hw_regs[IRDMA_GLPCI_LBARCTRL]); |
| 5419 | db_size = (u8)FIELD_GET(IRDMA_GLPCI_LBARCTRL_PE_DB_SIZE, val); | 6620 | db_size = (u8)FIELD_GET(IRDMA_GLPCI_LBARCTRL_PE_DB_SIZE, val); |
| 5420 | if (db_size != IRDMA_PE_DB_SIZE_4M && db_size != IRDMA_PE_DB_SIZE_8M) { | 6621 | if (db_size != IRDMA_PE_DB_SIZE_4M && |
| 5421 | ibdev_dbg(to_ibdev(dev), | 6622 | db_size != IRDMA_PE_DB_SIZE_8M) { |
| 5422 | "DEV: RDMA PE doorbell is not enabled in CSR val 0x%x db_size=%d\n", | 6623 | ibdev_dbg(to_ibdev(dev), |
| 5423 | val, db_size); | 6624 | "DEV: RDMA PE doorbell is not enabled in CSR val 0x%x db_size=%d\n", |
| 5424 | return -ENODEV; | 6625 | val, db_size); |
| 6626 | return -ENODEV; | ||
| 6627 | } | ||
| 6628 | } else { | ||
| 6629 | ret_code = irdma_vchnl_req_get_reg_layout(dev); | ||
| 6630 | if (ret_code) | ||
| 6631 | ibdev_dbg(to_ibdev(dev), | ||
| 6632 | "DEV: Get Register layout failed ret = %d\n", | ||
| 6633 | ret_code); | ||
| 5425 | } | 6634 | } |
| 5426 | dev->db_addr = dev->hw->hw_addr + (uintptr_t)dev->hw_regs[IRDMA_DB_ADDR_OFFSET]; | ||
| 5427 | 6635 | ||
| 5428 | return ret_code; | 6636 | return ret_code; |
| 5429 | } | 6637 | } |
| @@ -14,6 +14,18 @@ · 13 unmodified lines | |||
| 14 | #define IRDMA_PE_DB_SIZE_4M 1 | 14 | #define IRDMA_PE_DB_SIZE_4M 1 |
| 15 | #define IRDMA_PE_DB_SIZE_8M 2 | 15 | #define IRDMA_PE_DB_SIZE_8M 2 |
| 16 | 16 | ||
| 17 | #define IRDMA_IRD_HW_SIZE_4_GEN3 0 | ||
| 18 | #define IRDMA_IRD_HW_SIZE_8_GEN3 1 | ||
| 19 | #define IRDMA_IRD_HW_SIZE_16_GEN3 2 | ||
| 20 | #define IRDMA_IRD_HW_SIZE_32_GEN3 3 | ||
| 21 | #define IRDMA_IRD_HW_SIZE_64_GEN3 4 | ||
| 22 | #define IRDMA_IRD_HW_SIZE_128_GEN3 5 | ||
| 23 | #define IRDMA_IRD_HW_SIZE_256_GEN3 6 | ||
| 24 | #define IRDMA_IRD_HW_SIZE_512_GEN3 7 | ||
| 25 | #define IRDMA_IRD_HW_SIZE_1024_GEN3 8 | ||
| 26 | #define IRDMA_IRD_HW_SIZE_2048_GEN3 9 | ||
| 27 | #define IRDMA_IRD_HW_SIZE_4096_GEN3 10 | ||
| 28 | |||
| 17 | #define IRDMA_IRD_HW_SIZE_4 0 | 29 | #define IRDMA_IRD_HW_SIZE_4 0 |
| 18 | #define IRDMA_IRD_HW_SIZE_16 1 | 30 | #define IRDMA_IRD_HW_SIZE_16 1 |
| 19 | #define IRDMA_IRD_HW_SIZE_64 2 | 31 | #define IRDMA_IRD_HW_SIZE_64 2 |
| @@ -114,6 +126,13 @@ enum irdma_protocol_used { · 94 unmodified lines | |||
| 114 | #define IRDMA_UPDATE_SD_BUFF_SIZE 128 | 126 | #define IRDMA_UPDATE_SD_BUFF_SIZE 128 |
| 115 | #define IRDMA_FEATURE_BUF_SIZE (8 * IRDMA_MAX_FEATURES) | 127 | #define IRDMA_FEATURE_BUF_SIZE (8 * IRDMA_MAX_FEATURES) |
| 116 | 128 | ||
| 129 | #define ENABLE_LOC_MEM 63 | ||
| 130 | #define IRDMA_ATOMICS_ALLOWED_BIT 1 | ||
| 131 | #define MAX_PBLE_PER_SD 0x40000 | ||
| 132 | #define MAX_PBLE_SD_PER_FCN 0x400 | ||
| 133 | #define MAX_MR_PER_SD 0x8000 | ||
| 134 | #define MAX_MR_SD_PER_FCN 0x80 | ||
| 135 | #define IRDMA_PBLE_COMMIT_OFFSET 112 | ||
| 117 | #define IRDMA_MAX_QUANTA_PER_WR 8 | 136 | #define IRDMA_MAX_QUANTA_PER_WR 8 |
| 118 | 137 | ||
| 119 | #define IRDMA_QP_SW_MAX_WQ_QUANTA 32768 | 138 | #define IRDMA_QP_SW_MAX_WQ_QUANTA 32768 |
| @@ -121,6 +140,10 @@ enum irdma_protocol_used { · 1 unmodified line | |||
| 121 | #define IRDMA_QP_SW_MAX_RQ_QUANTA 32768 | 140 | #define IRDMA_QP_SW_MAX_RQ_QUANTA 32768 |
| 122 | #define IRDMA_MAX_QP_WRS(max_quanta_per_wr) \ | 141 | #define IRDMA_MAX_QP_WRS(max_quanta_per_wr) \ |
| 123 | ((IRDMA_QP_SW_MAX_WQ_QUANTA - IRDMA_SQ_RSVD) / (max_quanta_per_wr)) | 142 | ((IRDMA_QP_SW_MAX_WQ_QUANTA - IRDMA_SQ_RSVD) / (max_quanta_per_wr)) |
| 143 | #define IRDMA_SRQ_MIN_QUANTA 8 | ||
| 144 | #define IRDMA_SRQ_MAX_QUANTA 262144 | ||
| 145 | #define IRDMA_MAX_SRQ_WRS \ | ||
| 146 | ((IRDMA_SRQ_MAX_QUANTA - IRDMA_RQ_RSVD) / IRDMA_MAX_QUANTA_PER_WR) | ||
| 124 | 147 | ||
| 125 | #define IRDMAQP_TERM_SEND_TERM_AND_FIN 0 | 148 | #define IRDMAQP_TERM_SEND_TERM_AND_FIN 0 |
| 126 | #define IRDMAQP_TERM_SEND_TERM_ONLY 1 | 149 | #define IRDMAQP_TERM_SEND_TERM_ONLY 1 |
| @@ -147,8 +170,13 @@ enum irdma_protocol_used { · 20 unmodified lines | |||
| 147 | #define IRDMA_SQ_RSVD 258 | 170 | #define IRDMA_SQ_RSVD 258 |
| 148 | #define IRDMA_RQ_RSVD 1 | 171 | #define IRDMA_RQ_RSVD 1 |
| 149 | 172 | ||
| 150 | #define IRDMA_FEATURE_RTS_AE 1ULL | 173 | #define IRDMA_FEATURE_RTS_AE BIT_ULL(0) |
| 151 | #define IRDMA_FEATURE_CQ_RESIZE 2ULL | 174 | #define IRDMA_FEATURE_CQ_RESIZE BIT_ULL(1) |
| 175 | #define IRDMA_FEATURE_64_BYTE_CQE BIT_ULL(5) | ||
| 176 | #define IRDMA_FEATURE_ATOMIC_OPS BIT_ULL(6) | ||
| 177 | #define IRDMA_FEATURE_SRQ BIT_ULL(7) | ||
| 178 | #define IRDMA_FEATURE_CQE_TIMESTAMPING BIT_ULL(8) | ||
| 179 | |||
| 152 | #define IRDMAQP_OP_RDMA_WRITE 0x00 | 180 | #define IRDMAQP_OP_RDMA_WRITE 0x00 |
| 153 | #define IRDMAQP_OP_RDMA_READ 0x01 | 181 | #define IRDMAQP_OP_RDMA_READ 0x01 |
| 154 | #define IRDMAQP_OP_RDMA_SEND 0x03 | 182 | #define IRDMAQP_OP_RDMA_SEND 0x03 |
| @@ -161,6 +189,8 @@ enum irdma_protocol_used { · 6 unmodified lines | |||
| 161 | #define IRDMAQP_OP_RDMA_READ_LOC_INV 0x0b | 189 | #define IRDMAQP_OP_RDMA_READ_LOC_INV 0x0b |
| 162 | #define IRDMAQP_OP_NOP 0x0c | 190 | #define IRDMAQP_OP_NOP 0x0c |
| 163 | #define IRDMAQP_OP_RDMA_WRITE_SOL 0x0d | 191 | #define IRDMAQP_OP_RDMA_WRITE_SOL 0x0d |
| 192 | #define IRDMAQP_OP_ATOMIC_FETCH_ADD 0x0f | ||
| 193 | #define IRDMAQP_OP_ATOMIC_COMPARE_SWAP_ADD 0x11 | ||
| 164 | #define IRDMAQP_OP_GEN_RTS_AE 0x30 | 194 | #define IRDMAQP_OP_GEN_RTS_AE 0x30 |
| 165 | 195 | ||
| 166 | enum irdma_cqp_op_type { | 196 | enum irdma_cqp_op_type { |
| @@ -212,9 +242,12 @@ enum irdma_cqp_op_type { · 45 unmodified lines | |||
| 212 | IRDMA_OP_ADD_LOCAL_MAC_ENTRY = 46, | 242 | IRDMA_OP_ADD_LOCAL_MAC_ENTRY = 46, |
| 213 | IRDMA_OP_DELETE_LOCAL_MAC_ENTRY = 47, | 243 | IRDMA_OP_DELETE_LOCAL_MAC_ENTRY = 47, |
| 214 | IRDMA_OP_CQ_MODIFY = 48, | 244 | IRDMA_OP_CQ_MODIFY = 48, |
| 245 | IRDMA_OP_SRQ_CREATE = 49, | ||
| 246 | IRDMA_OP_SRQ_MODIFY = 50, | ||
| 247 | IRDMA_OP_SRQ_DESTROY = 51, | ||
| 215 | 248 | ||
| 216 | /* Must be last entry*/ | 249 | /* Must be last entry*/ |
| 217 | IRDMA_MAX_CQP_OPS = 49, | 250 | IRDMA_MAX_CQP_OPS = 52, |
| 218 | }; | 251 | }; |
| 219 | 252 | ||
| 220 | /* CQP SQ WQES */ | 253 | /* CQP SQ WQES */ |
| @@ -224,6 +257,9 @@ enum irdma_cqp_op_type { · 3 unmodified lines | |||
| 224 | #define IRDMA_CQP_OP_CREATE_CQ 0x03 | 257 | #define IRDMA_CQP_OP_CREATE_CQ 0x03 |
| 225 | #define IRDMA_CQP_OP_MODIFY_CQ 0x04 | 258 | #define IRDMA_CQP_OP_MODIFY_CQ 0x04 |
| 226 | #define IRDMA_CQP_OP_DESTROY_CQ 0x05 | 259 | #define IRDMA_CQP_OP_DESTROY_CQ 0x05 |
| 260 | #define IRDMA_CQP_OP_CREATE_SRQ 0x06 | ||
| 261 | #define IRDMA_CQP_OP_MODIFY_SRQ 0x07 | ||
| 262 | #define IRDMA_CQP_OP_DESTROY_SRQ 0x08 | ||
| 227 | #define IRDMA_CQP_OP_ALLOC_STAG 0x09 | 263 | #define IRDMA_CQP_OP_ALLOC_STAG 0x09 |
| 228 | #define IRDMA_CQP_OP_REG_MR 0x0a | 264 | #define IRDMA_CQP_OP_REG_MR 0x0a |
| 229 | #define IRDMA_CQP_OP_QUERY_STAG 0x0b | 265 | #define IRDMA_CQP_OP_QUERY_STAG 0x0b |
| @@ -265,97 +301,6 @@ enum irdma_cqp_op_type { · 35 unmodified lines | |||
| 265 | #define IRDMA_CQP_OP_GATHER_STATS 0x2e | 301 | #define IRDMA_CQP_OP_GATHER_STATS 0x2e |
| 266 | #define IRDMA_CQP_OP_UP_MAP 0x2f | 302 | #define IRDMA_CQP_OP_UP_MAP 0x2f |
| 267 | 303 | ||
| 268 | /* Async Events codes */ | ||
| 269 | #define IRDMA_AE_AMP_UNALLOCATED_STAG 0x0102 | ||
| 270 | #define IRDMA_AE_AMP_INVALID_STAG 0x0103 | ||
| 271 | #define IRDMA_AE_AMP_BAD_QP 0x0104 | ||
| 272 | #define IRDMA_AE_AMP_BAD_PD 0x0105 | ||
| 273 | #define IRDMA_AE_AMP_BAD_STAG_KEY 0x0106 | ||
| 274 | #define IRDMA_AE_AMP_BAD_STAG_INDEX 0x0107 | ||
| 275 | #define IRDMA_AE_AMP_BOUNDS_VIOLATION 0x0108 | ||
| 276 | #define IRDMA_AE_AMP_RIGHTS_VIOLATION 0x0109 | ||
| 277 | #define IRDMA_AE_AMP_TO_WRAP 0x010a | ||
| 278 | #define IRDMA_AE_AMP_FASTREG_VALID_STAG 0x010c | ||
| 279 | #define IRDMA_AE_AMP_FASTREG_MW_STAG 0x010d | ||
| 280 | #define IRDMA_AE_AMP_FASTREG_INVALID_RIGHTS 0x010e | ||
| 281 | #define IRDMA_AE_AMP_FASTREG_INVALID_LENGTH 0x0110 | ||
| 282 | #define IRDMA_AE_AMP_INVALIDATE_SHARED 0x0111 | ||
| 283 | #define IRDMA_AE_AMP_INVALIDATE_NO_REMOTE_ACCESS_RIGHTS 0x0112 | ||
| 284 | #define IRDMA_AE_AMP_INVALIDATE_MR_WITH_BOUND_WINDOWS 0x0113 | ||
| 285 | #define IRDMA_AE_AMP_MWBIND_VALID_STAG 0x0114 | ||
| 286 | #define IRDMA_AE_AMP_MWBIND_OF_MR_STAG 0x0115 | ||
| 287 | #define IRDMA_AE_AMP_MWBIND_TO_ZERO_BASED_STAG 0x0116 | ||
| 288 | #define IRDMA_AE_AMP_MWBIND_TO_MW_STAG 0x0117 | ||
| 289 | #define IRDMA_AE_AMP_MWBIND_INVALID_RIGHTS 0x0118 | ||
| 290 | #define IRDMA_AE_AMP_MWBIND_INVALID_BOUNDS 0x0119 | ||
| 291 | #define IRDMA_AE_AMP_MWBIND_TO_INVALID_PARENT 0x011a | ||
| 292 | #define IRDMA_AE_AMP_MWBIND_BIND_DISABLED 0x011b | ||
| 293 | #define IRDMA_AE_PRIV_OPERATION_DENIED 0x011c | ||
| 294 | #define IRDMA_AE_AMP_INVALIDATE_TYPE1_MW 0x011d | ||
| 295 | #define IRDMA_AE_AMP_MWBIND_ZERO_BASED_TYPE1_MW 0x011e | ||
| 296 | #define IRDMA_AE_AMP_FASTREG_INVALID_PBL_HPS_CFG 0x011f | ||
| 297 | #define IRDMA_AE_AMP_MWBIND_WRONG_TYPE 0x0120 | ||
| 298 | #define IRDMA_AE_AMP_FASTREG_PBLE_MISMATCH 0x0121 | ||
| 299 | #define IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG 0x0132 | ||
| 300 | #define IRDMA_AE_UDA_XMIT_BAD_PD 0x0133 | ||
| 301 | #define IRDMA_AE_UDA_XMIT_DGRAM_TOO_SHORT 0x0134 | ||
| 302 | #define IRDMA_AE_UDA_L4LEN_INVALID 0x0135 | ||
| 303 | #define IRDMA_AE_BAD_CLOSE 0x0201 | ||
| 304 | #define IRDMA_AE_RDMAP_ROE_BAD_LLP_CLOSE 0x0202 | ||
| 305 | #define IRDMA_AE_CQ_OPERATION_ERROR 0x0203 | ||
| 306 | #define IRDMA_AE_RDMA_READ_WHILE_ORD_ZERO 0x0205 | ||
| 307 | #define IRDMA_AE_STAG_ZERO_INVALID 0x0206 | ||
| 308 | #define IRDMA_AE_IB_RREQ_AND_Q1_FULL 0x0207 | ||
| 309 | #define IRDMA_AE_IB_INVALID_REQUEST 0x0208 | ||
| 310 | #define IRDMA_AE_WQE_UNEXPECTED_OPCODE 0x020a | ||
| 311 | #define IRDMA_AE_WQE_INVALID_PARAMETER 0x020b | ||
| 312 | #define IRDMA_AE_WQE_INVALID_FRAG_DATA 0x020c | ||
| 313 | #define IRDMA_AE_IB_REMOTE_ACCESS_ERROR 0x020d | ||
| 314 | #define IRDMA_AE_IB_REMOTE_OP_ERROR 0x020e | ||
| 315 | #define IRDMA_AE_WQE_LSMM_TOO_LONG 0x0220 | ||
| 316 | #define IRDMA_AE_INVALID_REQUEST 0x0223 | ||
| 317 | #define IRDMA_AE_DDP_INVALID_MSN_GAP_IN_MSN 0x0301 | ||
| 318 | #define IRDMA_AE_DDP_UBE_DDP_MESSAGE_TOO_LONG_FOR_AVAILABLE_BUFFER 0x0303 | ||
| 319 | #define IRDMA_AE_DDP_UBE_INVALID_DDP_VERSION 0x0304 | ||
| 320 | #define IRDMA_AE_DDP_UBE_INVALID_MO 0x0305 | ||
| 321 | #define IRDMA_AE_DDP_UBE_INVALID_MSN_NO_BUFFER_AVAILABLE 0x0306 | ||
| 322 | #define IRDMA_AE_DDP_UBE_INVALID_QN 0x0307 | ||
| 323 | #define IRDMA_AE_DDP_NO_L_BIT 0x0308 | ||
| 324 | #define IRDMA_AE_RDMAP_ROE_INVALID_RDMAP_VERSION 0x0311 | ||
| 325 | #define IRDMA_AE_RDMAP_ROE_UNEXPECTED_OPCODE 0x0312 | ||
| 326 | #define IRDMA_AE_ROE_INVALID_RDMA_READ_REQUEST 0x0313 | ||
| 327 | #define IRDMA_AE_ROE_INVALID_RDMA_WRITE_OR_READ_RESP 0x0314 | ||
| 328 | #define IRDMA_AE_ROCE_RSP_LENGTH_ERROR 0x0316 | ||
| 329 | #define IRDMA_AE_ROCE_EMPTY_MCG 0x0380 | ||
| 330 | #define IRDMA_AE_ROCE_BAD_MC_IP_ADDR 0x0381 | ||
| 331 | #define IRDMA_AE_ROCE_BAD_MC_QPID 0x0382 | ||
| 332 | #define IRDMA_AE_MCG_QP_PROTOCOL_MISMATCH 0x0383 | ||
| 333 | #define IRDMA_AE_INVALID_ARP_ENTRY 0x0401 | ||
| 334 | #define IRDMA_AE_INVALID_TCP_OPTION_RCVD 0x0402 | ||
| 335 | #define IRDMA_AE_STALE_ARP_ENTRY 0x0403 | ||
| 336 | #define IRDMA_AE_INVALID_AH_ENTRY 0x0406 | ||
| 337 | #define IRDMA_AE_LLP_CLOSE_COMPLETE 0x0501 | ||
| 338 | #define IRDMA_AE_LLP_CONNECTION_RESET 0x0502 | ||
| 339 | #define IRDMA_AE_LLP_FIN_RECEIVED 0x0503 | ||
| 340 | #define IRDMA_AE_LLP_RECEIVED_MARKER_AND_LENGTH_FIELDS_DONT_MATCH 0x0504 | ||
| 341 | #define IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR 0x0505 | ||
| 342 | #define IRDMA_AE_LLP_SEGMENT_TOO_SMALL 0x0507 | ||
| 343 | #define IRDMA_AE_LLP_SYN_RECEIVED 0x0508 | ||
| 344 | #define IRDMA_AE_LLP_TERMINATE_RECEIVED 0x0509 | ||
| 345 | #define IRDMA_AE_LLP_TOO_MANY_RETRIES 0x050a | ||
| 346 | #define IRDMA_AE_LLP_TOO_MANY_KEEPALIVE_RETRIES 0x050b | ||
| 347 | #define IRDMA_AE_LLP_DOUBT_REACHABILITY 0x050c | ||
| 348 | #define IRDMA_AE_LLP_CONNECTION_ESTABLISHED 0x050e | ||
| 349 | #define IRDMA_AE_LLP_TOO_MANY_RNRS 0x050f | ||
| 350 | #define IRDMA_AE_RESOURCE_EXHAUSTION 0x0520 | ||
| 351 | #define IRDMA_AE_RESET_SENT 0x0601 | ||
| 352 | #define IRDMA_AE_TERMINATE_SENT 0x0602 | ||
| 353 | #define IRDMA_AE_RESET_NOT_SENT 0x0603 | ||
| 354 | #define IRDMA_AE_LCE_QP_CATASTROPHIC 0x0700 | ||
| 355 | #define IRDMA_AE_LCE_FUNCTION_CATASTROPHIC 0x0701 | ||
| 356 | #define IRDMA_AE_LCE_CQ_CATASTROPHIC 0x0702 | ||
| 357 | #define IRDMA_AE_QP_SUSPEND_COMPLETE 0x0900 | ||
| 358 | |||
| 359 | #define FLD_LS_64(dev, val, field) \ | 304 | #define FLD_LS_64(dev, val, field) \ |
| 360 | (((u64)(val) << (dev)->hw_shifts[field ## _S]) & (dev)->hw_masks[field ## _M]) | 305 | (((u64)(val) << (dev)->hw_shifts[field ## _S]) & (dev)->hw_masks[field ## _M]) |
| 361 | #define FLD_RS_64(dev, val, field) \ | 306 | #define FLD_RS_64(dev, val, field) \ |
| @@ -393,9 +338,13 @@ enum irdma_cqp_op_type { · 31 unmodified lines | |||
| 393 | #define IRDMA_CQPSQ_STATS_USE_INST BIT_ULL(61) | 338 | #define IRDMA_CQPSQ_STATS_USE_INST BIT_ULL(61) |
| 394 | #define IRDMA_CQPSQ_STATS_OP GENMASK_ULL(37, 32) | 339 | #define IRDMA_CQPSQ_STATS_OP GENMASK_ULL(37, 32) |
| 395 | #define IRDMA_CQPSQ_STATS_INST_INDEX GENMASK_ULL(6, 0) | 340 | #define IRDMA_CQPSQ_STATS_INST_INDEX GENMASK_ULL(6, 0) |
| 396 | #define IRDMA_CQPSQ_STATS_HMC_FCN_INDEX GENMASK_ULL(5, 0) | 341 | #define IRDMA_CQPSQ_STATS_HMC_FCN_INDEX GENMASK_ULL(15, 0) |
| 397 | #define IRDMA_CQPSQ_WS_WQEVALID BIT_ULL(63) | 342 | #define IRDMA_CQPSQ_WS_WQEVALID BIT_ULL(63) |
| 398 | #define IRDMA_CQPSQ_WS_NODEOP GENMASK_ULL(53, 52) | 343 | #define IRDMA_CQPSQ_WS_NODEOP GENMASK_ULL(55, 52) |
| 344 | #define IRDMA_SD_MAX GENMASK_ULL(15, 0) | ||
| 345 | #define IRDMA_MEM_MAX GENMASK_ULL(15, 0) | ||
| 346 | #define IRDMA_QP_MEM_LOC GENMASK_ULL(47, 44) | ||
| 347 | #define IRDMA_MR_MEM_LOC GENMASK_ULL(27, 24) | ||
| 399 | 348 | ||
| 400 | #define IRDMA_CQPSQ_WS_ENABLENODE BIT_ULL(62) | 349 | #define IRDMA_CQPSQ_WS_ENABLENODE BIT_ULL(62) |
| 401 | #define IRDMA_CQPSQ_WS_NODETYPE BIT_ULL(61) | 350 | #define IRDMA_CQPSQ_WS_NODETYPE BIT_ULL(61) |
| @@ -404,16 +353,16 @@ enum irdma_cqp_op_type { · 2 unmodified lines | |||
| 404 | #define IRDMA_CQPSQ_WS_VMVFTYPE GENMASK_ULL(55, 54) | 353 | #define IRDMA_CQPSQ_WS_VMVFTYPE GENMASK_ULL(55, 54) |
| 405 | #define IRDMA_CQPSQ_WS_VMVFNUM GENMASK_ULL(51, 42) | 354 | #define IRDMA_CQPSQ_WS_VMVFNUM GENMASK_ULL(51, 42) |
| 406 | #define IRDMA_CQPSQ_WS_OP GENMASK_ULL(37, 32) | 355 | #define IRDMA_CQPSQ_WS_OP GENMASK_ULL(37, 32) |
| 407 | #define IRDMA_CQPSQ_WS_PARENTID GENMASK_ULL(25, 16) | 356 | #define IRDMA_CQPSQ_WS_PARENTID GENMASK_ULL(29, 16) |
| 408 | #define IRDMA_CQPSQ_WS_NODEID GENMASK_ULL(9, 0) | 357 | #define IRDMA_CQPSQ_WS_NODEID GENMASK_ULL(13, 0) |
| 409 | #define IRDMA_CQPSQ_WS_VSI GENMASK_ULL(57, 48) | 358 | #define IRDMA_CQPSQ_WS_VSI GENMASK_ULL(63, 48) |
| 410 | #define IRDMA_CQPSQ_WS_WEIGHT GENMASK_ULL(38, 32) | 359 | #define IRDMA_CQPSQ_WS_WEIGHT GENMASK_ULL(38, 32) |
| 411 | 360 | ||
| 412 | #define IRDMA_CQPSQ_UP_WQEVALID BIT_ULL(63) | 361 | #define IRDMA_CQPSQ_UP_WQEVALID BIT_ULL(63) |
| 413 | #define IRDMA_CQPSQ_UP_USEVLAN BIT_ULL(62) | 362 | #define IRDMA_CQPSQ_UP_USEVLAN BIT_ULL(62) |
| 414 | #define IRDMA_CQPSQ_UP_USEOVERRIDE BIT_ULL(61) | 363 | #define IRDMA_CQPSQ_UP_USEOVERRIDE BIT_ULL(61) |
| 415 | #define IRDMA_CQPSQ_UP_OP GENMASK_ULL(37, 32) | 364 | #define IRDMA_CQPSQ_UP_OP GENMASK_ULL(37, 32) |
| 416 | #define IRDMA_CQPSQ_UP_HMCFCNIDX GENMASK_ULL(5, 0) | 365 | #define IRDMA_CQPSQ_UP_HMCFCNIDX GENMASK_ULL(15, 0) |
| 417 | #define IRDMA_CQPSQ_UP_CNPOVERRIDE GENMASK_ULL(37, 32) | 366 | #define IRDMA_CQPSQ_UP_CNPOVERRIDE GENMASK_ULL(37, 32) |
| 418 | #define IRDMA_CQPSQ_QUERY_RDMA_FEATURES_WQEVALID BIT_ULL(63) | 367 | #define IRDMA_CQPSQ_QUERY_RDMA_FEATURES_WQEVALID BIT_ULL(63) |
| 419 | #define IRDMA_CQPSQ_QUERY_RDMA_FEATURES_BUF_LEN GENMASK_ULL(31, 0) | 368 | #define IRDMA_CQPSQ_QUERY_RDMA_FEATURES_BUF_LEN GENMASK_ULL(31, 0) |
| @@ -448,6 +397,16 @@ enum irdma_cqp_op_type { · 28 unmodified lines | |||
| 448 | #define IRDMA_CQPHC_SVER GENMASK_ULL(31, 24) | 397 | #define IRDMA_CQPHC_SVER GENMASK_ULL(31, 24) |
| 449 | #define IRDMA_CQPHC_SQBASE GENMASK_ULL(63, 9) | 398 | #define IRDMA_CQPHC_SQBASE GENMASK_ULL(63, 9) |
| 450 | 399 | ||
| 400 | #define IRDMA_CQPHC_TIMESTAMP_OVERRIDE BIT_ULL(5) | ||
| 401 | #define IRDMA_CQPHC_TS_SHIFT GENMASK_ULL(12, 8) | ||
| 402 | #define IRDMA_CQPHC_EN_FINE_GRAINED_TIMERS BIT_ULL(0) | ||
| 403 | |||
| 404 | #define IRDMA_CQPHC_OOISC_BLKSIZE GENMASK_ULL(63, 60) | ||
| 405 | #define IRDMA_CQPHC_RRSP_BLKSIZE GENMASK_ULL(59, 56) | ||
| 406 | #define IRDMA_CQPHC_Q1_BLKSIZE GENMASK_ULL(55, 52) | ||
| 407 | #define IRDMA_CQPHC_XMIT_BLKSIZE GENMASK_ULL(51, 48) | ||
| 408 | #define IRDMA_CQPHC_BLKSIZES_VALID BIT_ULL(4) | ||
| 409 | |||
| 451 | #define IRDMA_CQPHC_QPCTX GENMASK_ULL(63, 0) | 410 | #define IRDMA_CQPHC_QPCTX GENMASK_ULL(63, 0) |
| 452 | #define IRDMA_QP_DBSA_HW_SQ_TAIL GENMASK_ULL(14, 0) | 411 | #define IRDMA_QP_DBSA_HW_SQ_TAIL GENMASK_ULL(14, 0) |
| 453 | #define IRDMA_CQ_DBSA_CQEIDX GENMASK_ULL(19, 0) | 412 | #define IRDMA_CQ_DBSA_CQEIDX GENMASK_ULL(19, 0) |
| @@ -461,6 +420,8 @@ enum irdma_cqp_op_type { · 7 unmodified lines | |||
| 461 | 420 | ||
| 462 | #define IRDMA_CCQ_OPRETVAL GENMASK_ULL(31, 0) | 421 | #define IRDMA_CCQ_OPRETVAL GENMASK_ULL(31, 0) |
| 463 | 422 | ||
| 423 | #define IRDMA_CCQ_DEFINFO GENMASK_ULL(63, 32) | ||
| 424 | |||
| 464 | #define IRDMA_CQ_MINERR GENMASK_ULL(15, 0) | 425 | #define IRDMA_CQ_MINERR GENMASK_ULL(15, 0) |
| 465 | #define IRDMA_CQ_MAJERR GENMASK_ULL(31, 16) | 426 | #define IRDMA_CQ_MAJERR GENMASK_ULL(31, 16) |
| 466 | #define IRDMA_CQ_WQEIDX GENMASK_ULL(46, 32) | 427 | #define IRDMA_CQ_WQEIDX GENMASK_ULL(46, 32) |
| @@ -469,6 +430,7 @@ enum irdma_cqp_op_type { · 2 unmodified lines | |||
| 469 | #define IRDMA_CQ_ERROR BIT_ULL(55) | 430 | #define IRDMA_CQ_ERROR BIT_ULL(55) |
| 470 | #define IRDMA_CQ_SQ BIT_ULL(62) | 431 | #define IRDMA_CQ_SQ BIT_ULL(62) |
| 471 | 432 | ||
| 433 | #define IRDMA_CQ_SRQ BIT_ULL(52) | ||
| 472 | #define IRDMA_CQ_VALID BIT_ULL(63) | 434 | #define IRDMA_CQ_VALID BIT_ULL(63) |
| 473 | #define IRDMA_CQ_IMMVALID BIT_ULL(62) | 435 | #define IRDMA_CQ_IMMVALID BIT_ULL(62) |
| 474 | #define IRDMA_CQ_UDSMACVALID BIT_ULL(61) | 436 | #define IRDMA_CQ_UDSMACVALID BIT_ULL(61) |
| @@ -476,8 +438,6 @@ enum irdma_cqp_op_type { · 1 unmodified line | |||
| 476 | #define IRDMA_CQ_UDSMAC GENMASK_ULL(47, 0) | 438 | #define IRDMA_CQ_UDSMAC GENMASK_ULL(47, 0) |
| 477 | #define IRDMA_CQ_UDVLAN GENMASK_ULL(63, 48) | 439 | #define IRDMA_CQ_UDVLAN GENMASK_ULL(63, 48) |
| 478 | 440 | ||
| 479 | #define IRDMA_CQ_IMMDATA_S 0 | ||
| 480 | #define IRDMA_CQ_IMMDATA_M (0xffffffffffffffffULL << IRDMA_CQ_IMMVALID_S) | ||
| 481 | #define IRDMA_CQ_IMMDATALOW32 GENMASK_ULL(31, 0) | 441 | #define IRDMA_CQ_IMMDATALOW32 GENMASK_ULL(31, 0) |
| 482 | #define IRDMA_CQ_IMMDATAUP32 GENMASK_ULL(63, 32) | 442 | #define IRDMA_CQ_IMMDATAUP32 GENMASK_ULL(63, 32) |
| 483 | #define IRDMACQ_PAYLDLEN GENMASK_ULL(31, 0) | 443 | #define IRDMACQ_PAYLDLEN GENMASK_ULL(31, 0) |
| @@ -508,6 +468,17 @@ enum irdma_cqp_op_type { · 24 unmodified lines | |||
| 508 | #define IRDMA_AEQE_Q2DATA GENMASK_ULL(62, 61) | 468 | #define IRDMA_AEQE_Q2DATA GENMASK_ULL(62, 61) |
| 509 | #define IRDMA_AEQE_VALID BIT_ULL(63) | 469 | #define IRDMA_AEQE_VALID BIT_ULL(63) |
| 510 | 470 | ||
| 471 | #define IRDMA_AEQE_Q2DATA_GEN_3 GENMASK_ULL(5, 4) | ||
| 472 | #define IRDMA_AEQE_TCPSTATE_GEN_3 GENMASK_ULL(3, 0) | ||
| 473 | #define IRDMA_AEQE_QPCQID_GEN_3 GENMASK_ULL(24, 0) | ||
| 474 | #define IRDMA_AEQE_AECODE_GEN_3 GENMASK_ULL(61, 50) | ||
| 475 | #define IRDMA_AEQE_OVERFLOW_GEN_3 BIT_ULL(62) | ||
| 476 | #define IRDMA_AEQE_WQDESCIDX_GEN_3 GENMASK_ULL(49, 32) | ||
| 477 | #define IRDMA_AEQE_IWSTATE_GEN_3 GENMASK_ULL(31, 29) | ||
| 478 | #define IRDMA_AEQE_AESRC_GEN_3 GENMASK_ULL(28, 25) | ||
| 479 | #define IRDMA_AEQE_CMPL_CTXT_S 6 | ||
| 480 | #define IRDMA_AEQE_CMPL_CTXT GENMASK_ULL(63, 6) | ||
| 481 | |||
| 511 | #define IRDMA_UDA_QPSQ_NEXT_HDR GENMASK_ULL(23, 16) | 482 | #define IRDMA_UDA_QPSQ_NEXT_HDR GENMASK_ULL(23, 16) |
| 512 | #define IRDMA_UDA_QPSQ_OPCODE GENMASK_ULL(37, 32) | 483 | #define IRDMA_UDA_QPSQ_OPCODE GENMASK_ULL(37, 32) |
| 513 | #define IRDMA_UDA_QPSQ_L4LEN GENMASK_ULL(45, 42) | 484 | #define IRDMA_UDA_QPSQ_L4LEN GENMASK_ULL(45, 42) |
| @@ -530,11 +501,14 @@ enum irdma_cqp_op_type { · 16 unmodified lines | |||
| 530 | #define IRDMA_CQPSQ_WQEVALID BIT_ULL(63) | 501 | #define IRDMA_CQPSQ_WQEVALID BIT_ULL(63) |
| 531 | #define IRDMA_CQPSQ_TPHVAL GENMASK_ULL(7, 0) | 502 | #define IRDMA_CQPSQ_TPHVAL GENMASK_ULL(7, 0) |
| 532 | 503 | ||
| 533 | #define IRDMA_CQPSQ_VSIIDX GENMASK_ULL(17, 8) | 504 | #define IRDMA_CQPSQ_VSIIDX GENMASK_ULL(23, 8) |
| 534 | #define IRDMA_CQPSQ_TPHEN BIT_ULL(60) | 505 | #define IRDMA_CQPSQ_TPHEN BIT_ULL(60) |
| 535 | 506 | ||
| 536 | #define IRDMA_CQPSQ_PBUFADDR IRDMA_CQPHC_QPCTX | 507 | #define IRDMA_CQPSQ_PBUFADDR IRDMA_CQPHC_QPCTX |
| 537 | 508 | ||
| 509 | #define IRDMA_CQPSQ_PASID GENMASK_ULL(51, 32) | ||
| 510 | #define IRDMA_CQPSQ_PASID_VALID BIT_ULL(62) | ||
| 511 | |||
| 538 | /* Create/Modify/Destroy QP */ | 512 | /* Create/Modify/Destroy QP */ |
| 539 | 513 | ||
| 540 | #define IRDMA_CQPSQ_QP_NEWMSS GENMASK_ULL(45, 32) | 514 | #define IRDMA_CQPSQ_QP_NEWMSS GENMASK_ULL(45, 32) |
| @@ -566,10 +540,30 @@ enum irdma_cqp_op_type { · 25 unmodified lines | |||
| 566 | 540 | ||
| 567 | #define IRDMA_CQPSQ_QP_DBSHADOWADDR IRDMA_CQPHC_QPCTX | 541 | #define IRDMA_CQPSQ_QP_DBSHADOWADDR IRDMA_CQPHC_QPCTX |
| 568 | 542 | ||
| 543 | #define IRDMA_CQPSQ_SRQ_RQSIZE GENMASK_ULL(3, 0) | ||
| 544 | #define IRDMA_CQPSQ_SRQ_RQ_WQE_SIZE GENMASK_ULL(5, 4) | ||
| 545 | #define IRDMA_CQPSQ_SRQ_SRQ_LIMIT GENMASK_ULL(43, 32) | ||
| 546 | #define IRDMA_CQPSQ_SRQ_SRQCTX GENMASK_ULL(63, 6) | ||
| 547 | #define IRDMA_CQPSQ_SRQ_PD_ID GENMASK_ULL(39, 16) | ||
| 548 | #define IRDMA_CQPSQ_SRQ_SRQ_ID GENMASK_ULL(15, 0) | ||
| 549 | #define IRDMA_CQPSQ_SRQ_OP GENMASK_ULL(37, 32) | ||
| 550 | #define IRDMA_CQPSQ_SRQ_LEAF_PBL_SIZE GENMASK_ULL(45, 44) | ||
| 551 | #define IRDMA_CQPSQ_SRQ_VIRTMAP BIT_ULL(47) | ||
| 552 | #define IRDMA_CQPSQ_SRQ_TPH_EN BIT_ULL(60) | ||
| 553 | #define IRDMA_CQPSQ_SRQ_ARM_LIMIT_EVENT BIT_ULL(61) | ||
| 554 | #define IRDMA_CQPSQ_SRQ_FIRST_PM_PBL_IDX GENMASK_ULL(27, 0) | ||
| 555 | #define IRDMA_CQPSQ_SRQ_TPH_VALUE GENMASK_ULL(7, 0) | ||
| 556 | #define IRDMA_CQPSQ_SRQ_PHYSICAL_BUFFER_ADDR_S 8 | ||
| 557 | #define IRDMA_CQPSQ_SRQ_PHYSICAL_BUFFER_ADDR GENMASK_ULL(63, 8) | ||
| 558 | #define IRDMA_CQPSQ_SRQ_DB_SHADOW_ADDR_S 6 | ||
| 559 | #define IRDMA_CQPSQ_SRQ_DB_SHADOW_ADDR GENMASK_ULL(63, 6) | ||
| 560 | |||
| 569 | #define IRDMA_CQPSQ_CQ_CQSIZE GENMASK_ULL(20, 0) | 561 | #define IRDMA_CQPSQ_CQ_CQSIZE GENMASK_ULL(20, 0) |
| 570 | #define IRDMA_CQPSQ_CQ_CQCTX GENMASK_ULL(62, 0) | 562 | #define IRDMA_CQPSQ_CQ_CQCTX GENMASK_ULL(62, 0) |
| 571 | #define IRDMA_CQPSQ_CQ_SHADOW_READ_THRESHOLD GENMASK(17, 0) | 563 | #define IRDMA_CQPSQ_CQ_SHADOW_READ_THRESHOLD GENMASK(17, 0) |
| 572 | 564 | ||
| 565 | #define IRDMA_CQPSQ_CQ_CQID_HIGH GENMASK_ULL(52, 50) | ||
| 566 | #define IRDMA_CQPSQ_CQ_CEQID_HIGH GENMASK_ULL(59, 54) | ||
| 573 | #define IRDMA_CQPSQ_CQ_OP GENMASK_ULL(37, 32) | 567 | #define IRDMA_CQPSQ_CQ_OP GENMASK_ULL(37, 32) |
| 574 | #define IRDMA_CQPSQ_CQ_CQRESIZE BIT_ULL(43) | 568 | #define IRDMA_CQPSQ_CQ_CQRESIZE BIT_ULL(43) |
| 575 | #define IRDMA_CQPSQ_CQ_LPBLSIZE GENMASK_ULL(45, 44) | 569 | #define IRDMA_CQPSQ_CQ_LPBLSIZE GENMASK_ULL(45, 44) |
| @@ -590,6 +584,7 @@ enum irdma_cqp_op_type { · 14 unmodified lines | |||
| 590 | #define IRDMA_CQPSQ_STAG_MR BIT_ULL(43) | 584 | #define IRDMA_CQPSQ_STAG_MR BIT_ULL(43) |
| 591 | #define IRDMA_CQPSQ_STAG_MWTYPE BIT_ULL(42) | 585 | #define IRDMA_CQPSQ_STAG_MWTYPE BIT_ULL(42) |
| 592 | #define IRDMA_CQPSQ_STAG_MW1_BIND_DONT_VLDT_KEY BIT_ULL(58) | 586 | #define IRDMA_CQPSQ_STAG_MW1_BIND_DONT_VLDT_KEY BIT_ULL(58) |
| 587 | #define IRDMA_CQPSQ_STAG_PDID_HI GENMASK_ULL(59, 54) | ||
| 593 | 588 | ||
| 594 | #define IRDMA_CQPSQ_STAG_LPBLSIZE IRDMA_CQPSQ_CQ_LPBLSIZE | 589 | #define IRDMA_CQPSQ_STAG_LPBLSIZE IRDMA_CQPSQ_CQ_LPBLSIZE |
| 595 | #define IRDMA_CQPSQ_STAG_HPAGESIZE GENMASK_ULL(47, 46) | 590 | #define IRDMA_CQPSQ_STAG_HPAGESIZE GENMASK_ULL(47, 46) |
| @@ -600,7 +595,8 @@ enum irdma_cqp_op_type { · 4 unmodified lines | |||
| 600 | #define IRDMA_CQPSQ_STAG_USEPFRID BIT_ULL(61) | 595 | #define IRDMA_CQPSQ_STAG_USEPFRID BIT_ULL(61) |
| 601 | 596 | ||
| 602 | #define IRDMA_CQPSQ_STAG_PBA IRDMA_CQPHC_QPCTX | 597 | #define IRDMA_CQPSQ_STAG_PBA IRDMA_CQPHC_QPCTX |
| 603 | #define IRDMA_CQPSQ_STAG_HMCFNIDX GENMASK_ULL(5, 0) | 598 | #define IRDMA_CQPSQ_STAG_HMCFNIDX GENMASK_ULL(15, 0) |
| 599 | #define IRDMA_CQPSQ_STAG_REMOTE_ATOMIC_EN BIT_ULL(61) | ||
| 604 | 600 | ||
| 605 | #define IRDMA_CQPSQ_STAG_FIRSTPMPBLIDX GENMASK_ULL(27, 0) | 601 | #define IRDMA_CQPSQ_STAG_FIRSTPMPBLIDX GENMASK_ULL(27, 0) |
| 606 | #define IRDMA_CQPSQ_QUERYSTAG_IDX IRDMA_CQPSQ_STAG_IDX | 602 | #define IRDMA_CQPSQ_QUERYSTAG_IDX IRDMA_CQPSQ_STAG_IDX |
| @@ -628,11 +624,8 @@ enum irdma_cqp_op_type { · 21 unmodified lines | |||
| 628 | /* Manage Push Page - MPP */ | 624 | /* Manage Push Page - MPP */ |
| 629 | #define IRDMA_INVALID_PUSH_PAGE_INDEX_GEN_1 0xffff | 625 | #define IRDMA_INVALID_PUSH_PAGE_INDEX_GEN_1 0xffff |
| 630 | #define IRDMA_INVALID_PUSH_PAGE_INDEX 0xffffffff | 626 | #define IRDMA_INVALID_PUSH_PAGE_INDEX 0xffffffff |
| 631 | 627 | #define IRDMA_CQPSQ_MPP_PPIDX GENMASK_ULL(31, 0) | |
| 632 | #define IRDMA_CQPSQ_MPP_QS_HANDLE GENMASK_ULL(9, 0) | ||
| 633 | #define IRDMA_CQPSQ_MPP_PPIDX GENMASK_ULL(9, 0) | ||
| 634 | #define IRDMA_CQPSQ_MPP_PPTYPE GENMASK_ULL(61, 60) | 628 | #define IRDMA_CQPSQ_MPP_PPTYPE GENMASK_ULL(61, 60) |
| 635 | |||
| 636 | #define IRDMA_CQPSQ_MPP_FREE_PAGE BIT_ULL(62) | 629 | #define IRDMA_CQPSQ_MPP_FREE_PAGE BIT_ULL(62) |
| 637 | 630 | ||
| 638 | /* Upload Context - UCTX */ | 631 | /* Upload Context - UCTX */ |
| @@ -651,6 +644,8 @@ enum irdma_cqp_op_type { · 12 unmodified lines | |||
| 651 | #define IRDMA_CQPSQ_CEQ_CEQSIZE GENMASK_ULL(21, 0) | 644 | #define IRDMA_CQPSQ_CEQ_CEQSIZE GENMASK_ULL(21, 0) |
| 652 | #define IRDMA_CQPSQ_CEQ_CEQID GENMASK_ULL(9, 0) | 645 | #define IRDMA_CQPSQ_CEQ_CEQID GENMASK_ULL(9, 0) |
| 653 | 646 | ||
| 647 | #define IRDMA_CQPSQ_CEQ_CEQID_HIGH GENMASK_ULL(15, 10) | ||
| 648 | |||
| 654 | #define IRDMA_CQPSQ_CEQ_LPBLSIZE IRDMA_CQPSQ_CQ_LPBLSIZE | 649 | #define IRDMA_CQPSQ_CEQ_LPBLSIZE IRDMA_CQPSQ_CQ_LPBLSIZE |
| 655 | #define IRDMA_CQPSQ_CEQ_VMAP BIT_ULL(47) | 650 | #define IRDMA_CQPSQ_CEQ_VMAP BIT_ULL(47) |
| 656 | #define IRDMA_CQPSQ_CEQ_ITRNOEXPIRE BIT_ULL(46) | 651 | #define IRDMA_CQPSQ_CEQ_ITRNOEXPIRE BIT_ULL(46) |
| @@ -660,10 +655,10 @@ enum irdma_cqp_op_type { · 3 unmodified lines | |||
| 660 | #define IRDMA_CQPSQ_AEQ_VMAP BIT_ULL(47) | 655 | #define IRDMA_CQPSQ_AEQ_VMAP BIT_ULL(47) |
| 661 | #define IRDMA_CQPSQ_AEQ_FIRSTPMPBLIDX GENMASK_ULL(27, 0) | 656 | #define IRDMA_CQPSQ_AEQ_FIRSTPMPBLIDX GENMASK_ULL(27, 0) |
| 662 | 657 | ||
| 663 | #define IRDMA_COMMIT_FPM_QPCNT GENMASK_ULL(18, 0) | 658 | #define IRDMA_COMMIT_FPM_QPCNT GENMASK_ULL(20, 0) |
| 664 | |||
| 665 | #define IRDMA_COMMIT_FPM_BASE_S 32 | 659 | #define IRDMA_COMMIT_FPM_BASE_S 32 |
| 666 | #define IRDMA_CQPSQ_CFPM_HMCFNID GENMASK_ULL(5, 0) | 660 | #define IRDMA_CQPSQ_CFPM_HMCFNID GENMASK_ULL(15, 0) |
| 661 | |||
| 667 | #define IRDMA_CQPSQ_FWQE_AECODE GENMASK_ULL(15, 0) | 662 | #define IRDMA_CQPSQ_FWQE_AECODE GENMASK_ULL(15, 0) |
| 668 | #define IRDMA_CQPSQ_FWQE_AESOURCE GENMASK_ULL(19, 16) | 663 | #define IRDMA_CQPSQ_FWQE_AESOURCE GENMASK_ULL(19, 16) |
| 669 | #define IRDMA_CQPSQ_FWQE_RQMNERR GENMASK_ULL(15, 0) | 664 | #define IRDMA_CQPSQ_FWQE_RQMNERR GENMASK_ULL(15, 0) |
| @@ -675,6 +670,10 @@ enum irdma_cqp_op_type { · 5 unmodified lines | |||
| 675 | #define IRDMA_CQPSQ_FWQE_USERFLCODE BIT_ULL(60) | 670 | #define IRDMA_CQPSQ_FWQE_USERFLCODE BIT_ULL(60) |
| 676 | #define IRDMA_CQPSQ_FWQE_FLUSHSQ BIT_ULL(61) | 671 | #define IRDMA_CQPSQ_FWQE_FLUSHSQ BIT_ULL(61) |
| 677 | #define IRDMA_CQPSQ_FWQE_FLUSHRQ BIT_ULL(62) | 672 | #define IRDMA_CQPSQ_FWQE_FLUSHRQ BIT_ULL(62) |
| 673 | #define IRDMA_CQPSQ_FWQE_ERR_SQ_IDX_VALID BIT_ULL(42) | ||
| 674 | #define IRDMA_CQPSQ_FWQE_ERR_SQ_IDX GENMASK_ULL(49, 32) | ||
| 675 | #define IRDMA_CQPSQ_FWQE_ERR_RQ_IDX_VALID BIT_ULL(43) | ||
| 676 | #define IRDMA_CQPSQ_FWQE_ERR_RQ_IDX GENMASK_ULL(46, 32) | ||
| 678 | #define IRDMA_CQPSQ_MAPT_PORT GENMASK_ULL(15, 0) | 677 | #define IRDMA_CQPSQ_MAPT_PORT GENMASK_ULL(15, 0) |
| 679 | #define IRDMA_CQPSQ_MAPT_ADDPORT BIT_ULL(62) | 678 | #define IRDMA_CQPSQ_MAPT_ADDPORT BIT_ULL(62) |
| 680 | #define IRDMA_CQPSQ_UPESD_SDCMD GENMASK_ULL(31, 0) | 679 | #define IRDMA_CQPSQ_UPESD_SDCMD GENMASK_ULL(31, 0) |
| @@ -693,9 +692,12 @@ enum irdma_cqp_op_type { · 12 unmodified lines | |||
| 693 | #define IRDMA_CQPSQ_SUSPENDQP_QPID GENMASK_ULL(23, 0) | 692 | #define IRDMA_CQPSQ_SUSPENDQP_QPID GENMASK_ULL(23, 0) |
| 694 | #define IRDMA_CQPSQ_RESUMEQP_QSHANDLE GENMASK_ULL(31, 0) | 693 | #define IRDMA_CQPSQ_RESUMEQP_QSHANDLE GENMASK_ULL(31, 0) |
| 695 | #define IRDMA_CQPSQ_RESUMEQP_QPID GENMASK(23, 0) | 694 | #define IRDMA_CQPSQ_RESUMEQP_QPID GENMASK(23, 0) |
| 695 | #define IRDMA_MANAGE_RSRC_VER2 BIT_ULL(2) | ||
| 696 | 696 | ||
| 697 | #define IRDMA_CQPSQ_MIN_STAG_INVALID 0x0001 | 697 | #define IRDMA_CQPSQ_MIN_STAG_INVALID 0x0001 |
| 698 | #define IRDMA_CQPSQ_MIN_SUSPEND_PND 0x0005 | 698 | #define IRDMA_CQPSQ_MIN_SUSPEND_PND 0x0005 |
| 699 | #define IRDMA_CQPSQ_MIN_DEF_CMPL 0x0006 | ||
| 700 | #define IRDMA_CQPSQ_MIN_OOO_CMPL 0x0007 | ||
| 699 | 701 | ||
| 700 | #define IRDMA_CQPSQ_MAJ_NO_ERROR 0x0000 | 702 | #define IRDMA_CQPSQ_MAJ_NO_ERROR 0x0000 |
| 701 | #define IRDMA_CQPSQ_MAJ_OBJCACHE_ERROR 0xF000 | 703 | #define IRDMA_CQPSQ_MAJ_OBJCACHE_ERROR 0xF000 |
| @@ -712,6 +714,11 @@ enum irdma_cqp_op_type { · 10 unmodified lines | |||
| 712 | #define IRDMAQPC_INSERTL2TAG2 BIT_ULL(11) | 714 | #define IRDMAQPC_INSERTL2TAG2 BIT_ULL(11) |
| 713 | #define IRDMAQPC_LIMIT GENMASK_ULL(13, 12) | 715 | #define IRDMAQPC_LIMIT GENMASK_ULL(13, 12) |
| 714 | 716 | ||
| 717 | #define IRDMAQPC_USE_SRQ BIT_ULL(10) | ||
| 718 | #define IRDMAQPC_SRQ_ID GENMASK_ULL(15, 0) | ||
| 719 | #define IRDMAQPC_PASID GENMASK_ULL(19, 0) | ||
| 720 | #define IRDMAQPC_PASID_VALID BIT_ULL(11) | ||
| 721 | |||
| 715 | #define IRDMAQPC_ECN_EN BIT_ULL(14) | 722 | #define IRDMAQPC_ECN_EN BIT_ULL(14) |
| 716 | #define IRDMAQPC_DROPOOOSEG BIT_ULL(15) | 723 | #define IRDMAQPC_DROPOOOSEG BIT_ULL(15) |
| 717 | #define IRDMAQPC_DUPACK_THRESH GENMASK_ULL(18, 16) | 724 | #define IRDMAQPC_DUPACK_THRESH GENMASK_ULL(18, 16) |
| @@ -782,21 +789,31 @@ enum irdma_cqp_op_type { · 64 unmodified lines | |||
| 782 | #define IRDMAQPC_CWNDROCE GENMASK_ULL(55, 32) | 789 | #define IRDMAQPC_CWNDROCE GENMASK_ULL(55, 32) |
| 783 | #define IRDMAQPC_SNDWL1 GENMASK_ULL(31, 0) | 790 | #define IRDMAQPC_SNDWL1 GENMASK_ULL(31, 0) |
| 784 | #define IRDMAQPC_SNDWL2 GENMASK_ULL(63, 32) | 791 | #define IRDMAQPC_SNDWL2 GENMASK_ULL(63, 32) |
| 785 | #define IRDMAQPC_ERR_RQ_IDX GENMASK_ULL(45, 32) | 792 | #define IRDMAQPC_MINRNR_TIMER GENMASK_ULL(4, 0) |
| 793 | #define IRDMAQPC_ERR_RQ_IDX GENMASK_ULL(46, 32) | ||
| 786 | #define IRDMAQPC_RTOMIN GENMASK_ULL(63, 57) | 794 | #define IRDMAQPC_RTOMIN GENMASK_ULL(63, 57) |
| 787 | #define IRDMAQPC_MAXSNDWND GENMASK_ULL(31, 0) | 795 | #define IRDMAQPC_MAXSNDWND GENMASK_ULL(31, 0) |
| 788 | #define IRDMAQPC_REXMIT_THRESH GENMASK_ULL(53, 48) | 796 | #define IRDMAQPC_REXMIT_THRESH GENMASK_ULL(53, 48) |
| 789 | #define IRDMAQPC_RNRNAK_THRESH GENMASK_ULL(56, 54) | 797 | #define IRDMAQPC_RNRNAK_THRESH GENMASK_ULL(56, 54) |
| 790 | #define IRDMAQPC_TXCQNUM GENMASK_ULL(18, 0) | 798 | #define IRDMAQPC_TXCQNUM GENMASK_ULL(24, 0) |
| 791 | #define IRDMAQPC_RXCQNUM GENMASK_ULL(50, 32) | 799 | #define IRDMAQPC_RXCQNUM GENMASK_ULL(56, 32) |
| 792 | #define IRDMAQPC_STAT_INDEX GENMASK_ULL(6, 0) | 800 | #define IRDMAQPC_STAT_INDEX GENMASK_ULL(6, 0) |
| 793 | #define IRDMAQPC_Q2ADDR GENMASK_ULL(63, 8) | 801 | #define IRDMAQPC_Q2ADDR GENMASK_ULL(63, 8) |
| 794 | #define IRDMAQPC_LASTBYTESENT GENMASK_ULL(7, 0) | 802 | #define IRDMAQPC_LASTBYTESENT GENMASK_ULL(7, 0) |
| 795 | #define IRDMAQPC_MACADDRESS GENMASK_ULL(63, 16) | 803 | #define IRDMAQPC_MACADDRESS GENMASK_ULL(63, 16) |
| 796 | #define IRDMAQPC_ORDSIZE GENMASK_ULL(7, 0) | 804 | #define IRDMAQPC_ORDSIZE GENMASK_ULL(7, 0) |
| 797 | 805 | ||
| 806 | #define IRDMAQPC_LOCALACKTIMEOUT GENMASK_ULL(12, 8) | ||
| 807 | #define IRDMAQPC_RNRNAK_TMR GENMASK_ULL(4, 0) | ||
| 808 | #define IRDMAQPC_ORDSIZE_GEN3 GENMASK_ULL(10, 0) | ||
| 809 | #define IRDMAQPC_REMOTE_ATOMIC_EN BIT_ULL(18) | ||
| 810 | #define IRDMAQPC_STAT_INDEX_GEN3 GENMASK_ULL(47, 32) | ||
| 811 | #define IRDMAQPC_PKT_LIMIT GENMASK_ULL(55, 48) | ||
| 812 | |||
| 798 | #define IRDMAQPC_IRDSIZE GENMASK_ULL(18, 16) | 813 | #define IRDMAQPC_IRDSIZE GENMASK_ULL(18, 16) |
| 799 | 814 | ||
| 815 | #define IRDMAQPC_IRDSIZE_GEN3 GENMASK_ULL(17, 14) | ||
| 816 | |||
| 800 | #define IRDMAQPC_UDPRIVCQENABLE BIT_ULL(19) | 817 | #define IRDMAQPC_UDPRIVCQENABLE BIT_ULL(19) |
| 801 | #define IRDMAQPC_WRRDRSPOK BIT_ULL(20) | 818 | #define IRDMAQPC_WRRDRSPOK BIT_ULL(20) |
| 802 | #define IRDMAQPC_RDOK BIT_ULL(21) | 819 | #define IRDMAQPC_RDOK BIT_ULL(21) |
| @@ -833,6 +850,7 @@ enum irdma_cqp_op_type { · 30 unmodified lines | |||
| 833 | #define IRDMA_FEATURE_INFO GENMASK_ULL(47, 0) | 850 | #define IRDMA_FEATURE_INFO GENMASK_ULL(47, 0) |
| 834 | #define IRDMA_FEATURE_CNT GENMASK_ULL(47, 32) | 851 | #define IRDMA_FEATURE_CNT GENMASK_ULL(47, 32) |
| 835 | #define IRDMA_FEATURE_TYPE GENMASK_ULL(63, 48) | 852 | #define IRDMA_FEATURE_TYPE GENMASK_ULL(63, 48) |
| 853 | #define IRDMA_FEATURE_RSRC_MAX GENMASK_ULL(31, 0) | ||
| 836 | 854 | ||
| 837 | #define IRDMAQPSQ_OPCODE GENMASK_ULL(37, 32) | 855 | #define IRDMAQPSQ_OPCODE GENMASK_ULL(37, 32) |
| 838 | #define IRDMAQPSQ_COPY_HOST_PBL BIT_ULL(43) | 856 | #define IRDMAQPSQ_COPY_HOST_PBL BIT_ULL(43) |
| @@ -856,7 +874,7 @@ enum irdma_cqp_op_type { · 17 unmodified lines | |||
| 856 | #define IRDMAQPSQ_REMSTAGINV GENMASK_ULL(31, 0) | 874 | #define IRDMAQPSQ_REMSTAGINV GENMASK_ULL(31, 0) |
| 857 | #define IRDMAQPSQ_DESTQKEY GENMASK_ULL(31, 0) | 875 | #define IRDMAQPSQ_DESTQKEY GENMASK_ULL(31, 0) |
| 858 | #define IRDMAQPSQ_DESTQPN GENMASK_ULL(55, 32) | 876 | #define IRDMAQPSQ_DESTQPN GENMASK_ULL(55, 32) |
| 859 | #define IRDMAQPSQ_AHID GENMASK_ULL(16, 0) | 877 | #define IRDMAQPSQ_AHID GENMASK_ULL(24, 0) |
| 860 | #define IRDMAQPSQ_INLINEDATAFLAG BIT_ULL(57) | 878 | #define IRDMAQPSQ_INLINEDATAFLAG BIT_ULL(57) |
| 861 | 879 | ||
| 862 | #define IRDMA_INLINE_VALID_S 7 | 880 | #define IRDMA_INLINE_VALID_S 7 |
| @@ -869,6 +887,9 @@ enum irdma_cqp_op_type { · 6 unmodified lines | |||
| 869 | 887 | ||
| 870 | #define IRDMAQPSQ_REMTO IRDMA_CQPHC_QPCTX | 888 | #define IRDMAQPSQ_REMTO IRDMA_CQPHC_QPCTX |
| 871 | 889 | ||
| 890 | #define IRDMAQPSQ_STAG GENMASK_ULL(31, 0) | ||
| 891 | #define IRDMAQPSQ_REMOTE_STAG GENMASK_ULL(31, 0) | ||
| 892 | |||
| 872 | #define IRDMAQPSQ_STAGRIGHTS GENMASK_ULL(52, 48) | 893 | #define IRDMAQPSQ_STAGRIGHTS GENMASK_ULL(52, 48) |
| 873 | #define IRDMAQPSQ_VABASEDTO BIT_ULL(53) | 894 | #define IRDMAQPSQ_VABASEDTO BIT_ULL(53) |
| 874 | #define IRDMAQPSQ_MEMWINDOWTYPE BIT_ULL(54) | 895 | #define IRDMAQPSQ_MEMWINDOWTYPE BIT_ULL(54) |
| @@ -879,6 +900,8 @@ enum irdma_cqp_op_type { · 4 unmodified lines | |||
| 879 | 900 | ||
| 880 | #define IRDMAQPSQ_BASEVA_TO_FBO IRDMA_CQPHC_QPCTX | 901 | #define IRDMAQPSQ_BASEVA_TO_FBO IRDMA_CQPHC_QPCTX |
| 881 | 902 | ||
| 903 | #define IRDMAQPSQ_REMOTE_ATOMICS_EN BIT_ULL(55) | ||
| 904 | |||
| 882 | #define IRDMAQPSQ_LOCSTAG GENMASK_ULL(31, 0) | 905 | #define IRDMAQPSQ_LOCSTAG GENMASK_ULL(31, 0) |
| 883 | 906 | ||
| 884 | #define IRDMAQPSQ_STAGKEY GENMASK_ULL(7, 0) | 907 | #define IRDMAQPSQ_STAGKEY GENMASK_ULL(7, 0) |
| @@ -903,11 +926,14 @@ enum irdma_cqp_op_type { · 18 unmodified lines | |||
| 903 | #define IRDMAPFINT_OICR_PE_PUSH_M BIT(27) | 926 | #define IRDMAPFINT_OICR_PE_PUSH_M BIT(27) |
| 904 | #define IRDMAPFINT_OICR_PE_CRITERR_M BIT(28) | 927 | #define IRDMAPFINT_OICR_PE_CRITERR_M BIT(28) |
| 905 | 928 | ||
| 906 | #define IRDMA_QUERY_FPM_MAX_QPS GENMASK_ULL(18, 0) | 929 | #define IRDMA_QUERY_FPM_LOC_MEM_PAGES GENMASK_ULL(63, 32) |
| 907 | #define IRDMA_QUERY_FPM_MAX_CQS GENMASK_ULL(19, 0) | 930 | #define IRDMA_QUERY_FPM_MAX_QPS GENMASK_ULL(31, 0) |
| 931 | #define IRDMA_QUERY_FPM_MAX_CQS GENMASK_ULL(31, 0) | ||
| 908 | #define IRDMA_QUERY_FPM_FIRST_PE_SD_INDEX GENMASK_ULL(13, 0) | 932 | #define IRDMA_QUERY_FPM_FIRST_PE_SD_INDEX GENMASK_ULL(13, 0) |
| 909 | #define IRDMA_QUERY_FPM_MAX_PE_SDS GENMASK_ULL(45, 32) | 933 | #define IRDMA_QUERY_FPM_MAX_PE_SDS GENMASK_ULL(44, 32) |
| 934 | #define IRDMA_QUERY_FPM_MAX_PE_SDS_GEN3 GENMASK_ULL(47, 32) | ||
| 910 | #define IRDMA_QUERY_FPM_MAX_CEQS GENMASK_ULL(9, 0) | 935 | #define IRDMA_QUERY_FPM_MAX_CEQS GENMASK_ULL(9, 0) |
| 936 | #define IRDMA_QUERY_FPM_MAX_IRD GENMASK_ULL(53, 50) | ||
| 911 | #define IRDMA_QUERY_FPM_XFBLOCKSIZE GENMASK_ULL(63, 32) | 937 | #define IRDMA_QUERY_FPM_XFBLOCKSIZE GENMASK_ULL(63, 32) |
| 912 | #define IRDMA_QUERY_FPM_Q1BLOCKSIZE GENMASK_ULL(63, 32) | 938 | #define IRDMA_QUERY_FPM_Q1BLOCKSIZE GENMASK_ULL(63, 32) |
| 913 | #define IRDMA_QUERY_FPM_HTMULTIPLIER GENMASK_ULL(19, 16) | 939 | #define IRDMA_QUERY_FPM_HTMULTIPLIER GENMASK_ULL(19, 16) |
| @@ -1103,7 +1129,7 @@ enum irdma_alignment { · 189 unmodified lines | |||
| 1103 | IRDMA_CEQ_ALIGNMENT = 0x100, | 1129 | IRDMA_CEQ_ALIGNMENT = 0x100, |
| 1104 | IRDMA_CQ0_ALIGNMENT = 0x100, | 1130 | IRDMA_CQ0_ALIGNMENT = 0x100, |
| 1105 | IRDMA_SD_BUF_ALIGNMENT = 0x80, | 1131 | IRDMA_SD_BUF_ALIGNMENT = 0x80, |
| 1106 | IRDMA_FEATURE_BUF_ALIGNMENT = 0x8, | 1132 | IRDMA_FEATURE_BUF_ALIGNMENT = 0x10, |
| 1107 | }; | 1133 | }; |
| 1108 | 1134 | ||
| 1109 | enum icrdma_protocol_used { | 1135 | enum icrdma_protocol_used { |
| @@ -5,6 +5,7 @@ · 4 unmodified lines | |||
| 5 | #include "defs.h" | 5 | #include "defs.h" |
| 6 | #include "type.h" | 6 | #include "type.h" |
| 7 | #include "protos.h" | 7 | #include "protos.h" |
| 8 | #include "virtchnl.h" | ||
| 8 | 9 | ||
| 9 | /** | 10 | /** |
| 10 | * irdma_find_sd_index_limit - finds segment descriptor index limit | 11 | * irdma_find_sd_index_limit - finds segment descriptor index limit |
| @@ -228,6 +229,10 @@ int irdma_sc_create_hmc_obj(struct irdma_sc_dev *dev, · 217 unmodified lines | |||
| 228 | bool pd_error = false; | 229 | bool pd_error = false; |
| 229 | int ret_code = 0; | 230 | int ret_code = 0; |
| 230 | 231 | ||
| 232 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3 && | ||
| 233 | dev->hmc_info->hmc_obj[info->rsrc_type].mem_loc == IRDMA_LOC_MEM) | ||
| 234 | return 0; | ||
| 235 | |||
| 231 | if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt) | 236 | if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt) |
| 232 | return -EINVAL; | 237 | return -EINVAL; |
| 233 | 238 | ||
| @@ -330,7 +335,7 @@ static int irdma_finish_del_sd_reg(struct irdma_sc_dev *dev, · 96 unmodified lines | |||
| 330 | u32 i, sd_idx; | 335 | u32 i, sd_idx; |
| 331 | struct irdma_dma_mem *mem; | 336 | struct irdma_dma_mem *mem; |
| 332 | 337 | ||
| 333 | if (!reset) | 338 | if (dev->privileged && !reset) |
| 334 | ret_code = irdma_hmc_sd_grp(dev, info->hmc_info, | 339 | ret_code = irdma_hmc_sd_grp(dev, info->hmc_info, |
| 335 | info->hmc_info->sd_indexes[0], | 340 | info->hmc_info->sd_indexes[0], |
| 336 | info->del_sd_cnt, false); | 341 | info->del_sd_cnt, false); |
| @@ -376,6 +381,9 @@ int irdma_sc_del_hmc_obj(struct irdma_sc_dev *dev, · 39 unmodified lines | |||
| 376 | u32 i, j; | 381 | u32 i, j; |
| 377 | int ret_code = 0; | 382 | int ret_code = 0; |
| 378 | 383 | ||
| 384 | if (dev->hmc_info->hmc_obj[info->rsrc_type].mem_loc == IRDMA_LOC_MEM) | ||
| 385 | return 0; | ||
| 386 | |||
| 379 | if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt) { | 387 | if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt) { |
| 380 | ibdev_dbg(to_ibdev(dev), | 388 | ibdev_dbg(to_ibdev(dev), |
| 381 | "HMC: error start_idx[%04d] >= [type %04d].cnt[%04d]\n", | 389 | "HMC: error start_idx[%04d] >= [type %04d].cnt[%04d]\n", |
| @@ -589,7 +597,10 @@ int irdma_add_pd_table_entry(struct irdma_sc_dev *dev, · 207 unmodified lines | |||
| 589 | pd_entry->sd_index = sd_idx; | 597 | pd_entry->sd_index = sd_idx; |
| 590 | pd_entry->valid = true; | 598 | pd_entry->valid = true; |
| 591 | pd_table->use_cnt++; | 599 | pd_table->use_cnt++; |
| 592 | irdma_invalidate_pf_hmc_pd(dev, sd_idx, rel_pd_idx); | 600 | |
| 601 | if (hmc_info->hmc_fn_id < dev->hw_attrs.first_hw_vf_fpm_id && | ||
| 602 | dev->privileged) | ||
| 603 | irdma_invalidate_pf_hmc_pd(dev, sd_idx, rel_pd_idx); | ||
| 593 | } | 604 | } |
| 594 | pd_entry->bp.use_cnt++; | 605 | pd_entry->bp.use_cnt++; |
| 595 | 606 | ||
| @@ -640,7 +651,8 @@ int irdma_remove_pd_bp(struct irdma_sc_dev *dev, · 44 unmodified lines | |||
| 640 | pd_addr = pd_table->pd_page_addr.va; | 651 | pd_addr = pd_table->pd_page_addr.va; |
| 641 | pd_addr += rel_pd_idx; | 652 | pd_addr += rel_pd_idx; |
| 642 | memset(pd_addr, 0, sizeof(u64)); | 653 | memset(pd_addr, 0, sizeof(u64)); |
| 643 | irdma_invalidate_pf_hmc_pd(dev, sd_idx, idx); | 654 | if (dev->privileged && dev->hmc_fn_id == hmc_info->hmc_fn_id) |
| 655 | irdma_invalidate_pf_hmc_pd(dev, sd_idx, idx); | ||
| 644 | 656 | ||
| 645 | if (!pd_entry->rsrc_pg) { | 657 | if (!pd_entry->rsrc_pg) { |
| 646 | mem = &pd_entry->bp.addr; | 658 | mem = &pd_entry->bp.addr; |
| @@ -16,11 +16,21 @@ · 15 unmodified lines | |||
| 16 | #define IRDMA_HMC_PD_BP_BUF_ALIGNMENT 4096 | 16 | #define IRDMA_HMC_PD_BP_BUF_ALIGNMENT 4096 |
| 17 | #define IRDMA_FIRST_VF_FPM_ID 8 | 17 | #define IRDMA_FIRST_VF_FPM_ID 8 |
| 18 | #define FPM_MULTIPLIER 1024 | 18 | #define FPM_MULTIPLIER 1024 |
| 19 | #define IRDMA_OBJ_LOC_MEM_BIT 0x4 | ||
| 20 | #define IRDMA_XF_MULTIPLIER 16 | ||
| 21 | #define IRDMA_RRF_MULTIPLIER 8 | ||
| 22 | #define IRDMA_MIN_PBLE_PAGES 3 | ||
| 23 | #define IRDMA_HMC_PAGE_SIZE 2097152 | ||
| 24 | #define IRDMA_MIN_MR_PER_QP 4 | ||
| 25 | #define IRDMA_MIN_QP_CNT 64 | ||
| 26 | #define IRDMA_FSIAV_CNT_MAX 1048576 | ||
| 27 | #define IRDMA_MIN_IRD 8 | ||
| 28 | #define IRDMA_HMC_MIN_RRF 16 | ||
| 19 | 29 | ||
| 20 | enum irdma_hmc_rsrc_type { | 30 | enum irdma_hmc_rsrc_type { |
| 21 | IRDMA_HMC_IW_QP = 0, | 31 | IRDMA_HMC_IW_QP = 0, |
| 22 | IRDMA_HMC_IW_CQ = 1, | 32 | IRDMA_HMC_IW_CQ = 1, |
| 23 | IRDMA_HMC_IW_RESERVED = 2, | 33 | IRDMA_HMC_IW_SRQ = 2, |
| 24 | IRDMA_HMC_IW_HTE = 3, | 34 | IRDMA_HMC_IW_HTE = 3, |
| 25 | IRDMA_HMC_IW_ARP = 4, | 35 | IRDMA_HMC_IW_ARP = 4, |
| 26 | IRDMA_HMC_IW_APBVT_ENTRY = 5, | 36 | IRDMA_HMC_IW_APBVT_ENTRY = 5, |
| @@ -48,11 +58,17 @@ enum irdma_sd_entry_type { · 21 unmodified lines | |||
| 48 | IRDMA_SD_TYPE_DIRECT = 2, | 58 | IRDMA_SD_TYPE_DIRECT = 2, |
| 49 | }; | 59 | }; |
| 50 | 60 | ||
| 61 | enum irdma_hmc_obj_mem { | ||
| 62 | IRDMA_HOST_MEM = 0, | ||
| 63 | IRDMA_LOC_MEM = 1, | ||
| 64 | }; | ||
| 65 | |||
| 51 | struct irdma_hmc_obj_info { | 66 | struct irdma_hmc_obj_info { |
| 52 | u64 base; | 67 | u64 base; |
| 53 | u32 max_cnt; | 68 | u32 max_cnt; |
| 54 | u32 cnt; | 69 | u32 cnt; |
| 55 | u64 size; | 70 | u64 size; |
| 71 | enum irdma_hmc_obj_mem mem_loc; | ||
| 56 | }; | 72 | }; |
| 57 | 73 | ||
| 58 | struct irdma_hmc_bp { | 74 | struct irdma_hmc_bp { |
| @@ -117,6 +133,7 @@ struct irdma_update_sds_info { · 58 unmodified lines | |||
| 117 | struct irdma_ccq_cqe_info; | 133 | struct irdma_ccq_cqe_info; |
| 118 | struct irdma_hmc_fcn_info { | 134 | struct irdma_hmc_fcn_info { |
| 119 | u32 vf_id; | 135 | u32 vf_id; |
| 136 | u8 protocol_used; | ||
| 120 | u8 free_fcn; | 137 | u8 free_fcn; |
| 121 | }; | 138 | }; |
| 122 | 139 | ||
| @@ -33,6 +33,7 @@ static struct irdma_rsrc_limits rsrc_limits_table[] = { · 32 unmodified lines | |||
| 33 | static enum irdma_hmc_rsrc_type iw_hmc_obj_types[] = { | 33 | static enum irdma_hmc_rsrc_type iw_hmc_obj_types[] = { |
| 34 | IRDMA_HMC_IW_QP, | 34 | IRDMA_HMC_IW_QP, |
| 35 | IRDMA_HMC_IW_CQ, | 35 | IRDMA_HMC_IW_CQ, |
| 36 | IRDMA_HMC_IW_SRQ, | ||
| 36 | IRDMA_HMC_IW_HTE, | 37 | IRDMA_HMC_IW_HTE, |
| 37 | IRDMA_HMC_IW_ARP, | 38 | IRDMA_HMC_IW_ARP, |
| 38 | IRDMA_HMC_IW_APBVT_ENTRY, | 39 | IRDMA_HMC_IW_APBVT_ENTRY, |
| @@ -134,75 +135,68 @@ static void irdma_process_ceq(struct irdma_pci_f *rf, struct irdma_ceq *ceq) · 95 unmodified lines | |||
| 134 | static void irdma_set_flush_fields(struct irdma_sc_qp *qp, | 135 | static void irdma_set_flush_fields(struct irdma_sc_qp *qp, |
| 135 | struct irdma_aeqe_info *info) | 136 | struct irdma_aeqe_info *info) |
| 136 | { | 137 | { |
| 138 | struct qp_err_code qp_err; | ||
| 139 | |||
| 137 | qp->sq_flush_code = info->sq; | 140 | qp->sq_flush_code = info->sq; |
| 138 | qp->rq_flush_code = info->rq; | 141 | qp->rq_flush_code = info->rq; |
| 139 | qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC; | 142 | if (qp->qp_uk.uk_attrs->hw_rev >= IRDMA_GEN_3) { |
| 140 | 143 | if (info->sq) { | |
| 141 | switch (info->ae_id) { | 144 | qp->err_sq_idx_valid = true; |
| 142 | case IRDMA_AE_AMP_BOUNDS_VIOLATION: | 145 | qp->err_sq_idx = info->wqe_idx; |
| 143 | case IRDMA_AE_AMP_INVALID_STAG: | 146 | } |
| 144 | case IRDMA_AE_AMP_RIGHTS_VIOLATION: | 147 | if (info->rq) { |
| 145 | case IRDMA_AE_AMP_UNALLOCATED_STAG: | 148 | qp->err_rq_idx_valid = true; |
| 146 | case IRDMA_AE_AMP_BAD_PD: | 149 | qp->err_rq_idx = info->wqe_idx; |
| 147 | case IRDMA_AE_AMP_BAD_QP: | 150 | } |
| 148 | case IRDMA_AE_AMP_BAD_STAG_KEY: | 151 | } |
| 149 | case IRDMA_AE_AMP_BAD_STAG_INDEX: | 152 | |
| 150 | case IRDMA_AE_AMP_TO_WRAP: | 153 | qp_err = irdma_ae_to_qp_err_code(info->ae_id); |
| 151 | case IRDMA_AE_PRIV_OPERATION_DENIED: | 154 | qp->flush_code = qp_err.flush_code; |
| 152 | qp->flush_code = FLUSH_PROT_ERR; | 155 | qp->event_type = qp_err.event_type; |
| 153 | qp->event_type = IRDMA_QP_EVENT_ACCESS_ERR; | 156 | } |
| 154 | break; | 157 | |
| 155 | case IRDMA_AE_UDA_XMIT_BAD_PD: | 158 | /** |
| 156 | case IRDMA_AE_WQE_UNEXPECTED_OPCODE: | 159 | * irdma_complete_cqp_request - perform post-completion cleanup |
| 157 | qp->flush_code = FLUSH_LOC_QP_OP_ERR; | 160 | * @cqp: device CQP |
| 158 | qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC; | 161 | * @cqp_request: CQP request |
| 159 | break; | 162 | * |
| 160 | case IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG: | 163 | * Mark CQP request as done, wake up waiting thread or invoke |
| 161 | case IRDMA_AE_UDA_XMIT_DGRAM_TOO_SHORT: | 164 | * callback function and release/free CQP request. |
| 162 | case IRDMA_AE_UDA_L4LEN_INVALID: | 165 | */ |
| 163 | case IRDMA_AE_DDP_UBE_INVALID_MO: | 166 | static void irdma_complete_cqp_request(struct irdma_cqp *cqp, |
| 164 | case IRDMA_AE_DDP_UBE_DDP_MESSAGE_TOO_LONG_FOR_AVAILABLE_BUFFER: | 167 | struct irdma_cqp_request *cqp_request) |
| 165 | qp->flush_code = FLUSH_LOC_LEN_ERR; | 168 | { |
| 166 | qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC; | 169 | if (cqp_request->waiting) { |
| 167 | break; | 170 | WRITE_ONCE(cqp_request->request_done, true); |
| 168 | case IRDMA_AE_AMP_INVALIDATE_NO_REMOTE_ACCESS_RIGHTS: | 171 | wake_up(&cqp_request->waitq); |
| 169 | case IRDMA_AE_IB_REMOTE_ACCESS_ERROR: | 172 | } else if (cqp_request->callback_fcn) { |
| 170 | qp->flush_code = FLUSH_REM_ACCESS_ERR; | 173 | cqp_request->callback_fcn(cqp_request); |
| 171 | qp->event_type = IRDMA_QP_EVENT_ACCESS_ERR; | 174 | } |
| 172 | break; | 175 | irdma_put_cqp_request(cqp, cqp_request); |
| 173 | case IRDMA_AE_LLP_SEGMENT_TOO_SMALL: | 176 | } |
| 174 | case IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR: | 177 | |
| 175 | case IRDMA_AE_ROCE_RSP_LENGTH_ERROR: | 178 | /** |
| 176 | case IRDMA_AE_IB_REMOTE_OP_ERROR: | 179 | * irdma_process_ae_def_cmpl - handle IRDMA_AE_CQP_DEFERRED_COMPLETE event |
| 177 | qp->flush_code = FLUSH_REM_OP_ERR; | 180 | * @rf: RDMA PCI function |
| 178 | qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC; | 181 | * @info: AEQ entry info |
| 179 | break; | 182 | */ |
| 180 | case IRDMA_AE_LCE_QP_CATASTROPHIC: | 183 | static void irdma_process_ae_def_cmpl(struct irdma_pci_f *rf, |
| 181 | qp->flush_code = FLUSH_FATAL_ERR; | 184 | struct irdma_aeqe_info *info) |
| 182 | qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC; | 185 | { |
| 183 | break; | 186 | u32 sw_def_info; |
| 184 | case IRDMA_AE_IB_RREQ_AND_Q1_FULL: | 187 | u64 scratch; |
| 185 | qp->flush_code = FLUSH_GENERAL_ERR; | 188 | |
| 186 | break; | 189 | irdma_cqp_ce_handler(rf, &rf->ccq.sc_cq); |
| 187 | case IRDMA_AE_LLP_TOO_MANY_RETRIES: | 190 | |
| 188 | qp->flush_code = FLUSH_RETRY_EXC_ERR; | 191 | irdma_sc_cqp_def_cmpl_ae_handler(&rf->sc_dev, info, true, |
| 189 | qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC; | 192 | &scratch, &sw_def_info); |
| 190 | break; | 193 | while (scratch) { |
| 191 | case IRDMA_AE_AMP_MWBIND_INVALID_RIGHTS: | 194 | struct irdma_cqp_request *cqp_request = |
| 192 | case IRDMA_AE_AMP_MWBIND_BIND_DISABLED: | 195 | (struct irdma_cqp_request *)(uintptr_t)scratch; |
| 193 | case IRDMA_AE_AMP_MWBIND_INVALID_BOUNDS: | 196 | |
| 194 | case IRDMA_AE_AMP_MWBIND_VALID_STAG: | 197 | irdma_complete_cqp_request(&rf->cqp, cqp_request); |
| 195 | qp->flush_code = FLUSH_MW_BIND_ERR; | 198 | irdma_sc_cqp_def_cmpl_ae_handler(&rf->sc_dev, info, false, |
| 196 | qp->event_type = IRDMA_QP_EVENT_ACCESS_ERR; | 199 | &scratch, &sw_def_info); |
| 197 | break; | ||
| 198 | case IRDMA_AE_IB_INVALID_REQUEST: | ||
| 199 | qp->flush_code = FLUSH_REM_INV_REQ_ERR; | ||
| 200 | qp->event_type = IRDMA_QP_EVENT_REQ_ERR; | ||
| 201 | break; | ||
| 202 | default: | ||
| 203 | qp->flush_code = FLUSH_GENERAL_ERR; | ||
| 204 | qp->event_type = IRDMA_QP_EVENT_CATASTROPHIC; | ||
| 205 | break; | ||
| 206 | } | 200 | } |
| 207 | } | 201 | } |
| 208 | 202 | ||
| @@ -223,6 +217,7 @@ static void irdma_process_aeq(struct irdma_pci_f *rf) · 14 unmodified lines | |||
| 223 | struct irdma_sc_qp *qp = NULL; | 217 | struct irdma_sc_qp *qp = NULL; |
| 224 | struct irdma_qp_host_ctx_info *ctx_info = NULL; | 218 | struct irdma_qp_host_ctx_info *ctx_info = NULL; |
| 225 | struct irdma_device *iwdev = rf->iwdev; | 219 | struct irdma_device *iwdev = rf->iwdev; |
| 220 | struct irdma_sc_srq *srq; | ||
| 226 | unsigned long flags; | 221 | unsigned long flags; |
| 227 | 222 | ||
| 228 | u32 aeqcnt = 0; | 223 | u32 aeqcnt = 0; |
| @@ -236,6 +231,13 @@ static void irdma_process_aeq(struct irdma_pci_f *rf) · 7 unmodified lines | |||
| 236 | if (ret) | 231 | if (ret) |
| 237 | break; | 232 | break; |
| 238 | 233 | ||
| 234 | if (info->aeqe_overflow) { | ||
| 235 | ibdev_err(&iwdev->ibdev, "AEQ has overflowed\n"); | ||
| 236 | rf->reset = true; | ||
| 237 | rf->gen_ops.request_reset(rf); | ||
| 238 | return; | ||
| 239 | } | ||
| 240 | |||
| 239 | aeqcnt++; | 241 | aeqcnt++; |
| 240 | ibdev_dbg(&iwdev->ibdev, | 242 | ibdev_dbg(&iwdev->ibdev, |
| 241 | "AEQ: ae_id = 0x%x bool qp=%d qp_id = %d tcp_state=%d iwarp_state=%d ae_src=%d\n", | 243 | "AEQ: ae_id = 0x%x bool qp=%d qp_id = %d tcp_state=%d iwarp_state=%d ae_src=%d\n", |
| @@ -266,9 +268,12 @@ static void irdma_process_aeq(struct irdma_pci_f *rf) · 24 unmodified lines | |||
| 266 | if (info->ae_id != IRDMA_AE_QP_SUSPEND_COMPLETE) | 268 | if (info->ae_id != IRDMA_AE_QP_SUSPEND_COMPLETE) |
| 267 | iwqp->last_aeq = info->ae_id; | 269 | iwqp->last_aeq = info->ae_id; |
| 268 | spin_unlock_irqrestore(&iwqp->lock, flags); | 270 | spin_unlock_irqrestore(&iwqp->lock, flags); |
| 269 | ctx_info = &iwqp->ctx_info; | 271 | } else if (info->srq) { |
| 272 | if (info->ae_id != IRDMA_AE_SRQ_LIMIT) | ||
| 273 | continue; | ||
| 270 | } else { | 274 | } else { |
| 271 | if (info->ae_id != IRDMA_AE_CQ_OPERATION_ERROR) | 275 | if (info->ae_id != IRDMA_AE_CQ_OPERATION_ERROR && |
| 276 | info->ae_id != IRDMA_AE_CQP_DEFERRED_COMPLETE) | ||
| 272 | continue; | 277 | continue; |
| 273 | } | 278 | } |
| 274 | 279 | ||
| @@ -363,6 +368,18 @@ static void irdma_process_aeq(struct irdma_pci_f *rf) · 88 unmodified lines | |||
| 363 | } | 368 | } |
| 364 | irdma_cq_rem_ref(&iwcq->ibcq); | 369 | irdma_cq_rem_ref(&iwcq->ibcq); |
| 365 | break; | 370 | break; |
| 371 | case IRDMA_AE_SRQ_LIMIT: | ||
| 372 | srq = (struct irdma_sc_srq *)(uintptr_t)info->compl_ctx; | ||
| 373 | irdma_srq_event(srq); | ||
| 374 | break; | ||
| 375 | case IRDMA_AE_SRQ_CATASTROPHIC_ERROR: | ||
| 376 | break; | ||
| 377 | case IRDMA_AE_CQP_DEFERRED_COMPLETE: | ||
| 378 | /* Remove completed CQP requests from pending list | ||
| 379 | * and notify about those CQP ops completion. | ||
| 380 | */ | ||
| 381 | irdma_process_ae_def_cmpl(rf, info); | ||
| 382 | break; | ||
| 366 | case IRDMA_AE_RESET_NOT_SENT: | 383 | case IRDMA_AE_RESET_NOT_SENT: |
| 367 | case IRDMA_AE_LLP_DOUBT_REACHABILITY: | 384 | case IRDMA_AE_LLP_DOUBT_REACHABILITY: |
| 368 | case IRDMA_AE_RESOURCE_EXHAUSTION: | 385 | case IRDMA_AE_RESOURCE_EXHAUSTION: |
| @@ -389,13 +406,18 @@ static void irdma_process_aeq(struct irdma_pci_f *rf) · 20 unmodified lines | |||
| 389 | case IRDMA_AE_LCE_FUNCTION_CATASTROPHIC: | 406 | case IRDMA_AE_LCE_FUNCTION_CATASTROPHIC: |
| 390 | case IRDMA_AE_LLP_TOO_MANY_RNRS: | 407 | case IRDMA_AE_LLP_TOO_MANY_RNRS: |
| 391 | case IRDMA_AE_LCE_CQ_CATASTROPHIC: | 408 | case IRDMA_AE_LCE_CQ_CATASTROPHIC: |
| 409 | case IRDMA_AE_REMOTE_QP_CATASTROPHIC: | ||
| 410 | case IRDMA_AE_LOCAL_QP_CATASTROPHIC: | ||
| 411 | case IRDMA_AE_RCE_QP_CATASTROPHIC: | ||
| 392 | case IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG: | 412 | case IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG: |
| 393 | default: | 413 | default: |
| 394 | ibdev_err(&iwdev->ibdev, "abnormal ae_id = 0x%x bool qp=%d qp_id = %d, ae_src=%d\n", | 414 | ibdev_err(&iwdev->ibdev, "abnormal ae_id = 0x%x bool qp=%d qp_id = %d, ae_src=%d\n", |
| 395 | info->ae_id, info->qp, info->qp_cq_id, info->ae_src); | 415 | info->ae_id, info->qp, info->qp_cq_id, info->ae_src); |
| 396 | if (rdma_protocol_roce(&iwdev->ibdev, 1)) { | 416 | ctx_info = &iwqp->ctx_info; |
| 397 | ctx_info->roce_info->err_rq_idx_valid = info->rq; | 417 | if (rdma_protocol_roce(&iwqp->iwdev->ibdev, 1)) { |
| 398 | if (info->rq) { | 418 | ctx_info->roce_info->err_rq_idx_valid = |
| 419 | ctx_info->srq_valid ? false : info->err_rq_idx_valid; | ||
| 420 | if (ctx_info->roce_info->err_rq_idx_valid) { | ||
| 399 | ctx_info->roce_info->err_rq_idx = info->wqe_idx; | 421 | ctx_info->roce_info->err_rq_idx = info->wqe_idx; |
| 400 | irdma_sc_qp_setctx_roce(&iwqp->sc_qp, iwqp->host_ctx.va, | 422 | irdma_sc_qp_setctx_roce(&iwqp->sc_qp, iwqp->host_ctx.va, |
| 401 | ctx_info); | 423 | ctx_info); |
| @@ -599,6 +621,8 @@ static void irdma_destroy_cqp(struct irdma_pci_f *rf) · 197 unmodified lines | |||
| 599 | dma_free_coherent(dev->hw->device, cqp->sq.size, cqp->sq.va, | 621 | dma_free_coherent(dev->hw->device, cqp->sq.size, cqp->sq.va, |
| 600 | cqp->sq.pa); | 622 | cqp->sq.pa); |
| 601 | cqp->sq.va = NULL; | 623 | cqp->sq.va = NULL; |
| 624 | kfree(cqp->oop_op_array); | ||
| 625 | cqp->oop_op_array = NULL; | ||
| 602 | kfree(cqp->scratch_array); | 626 | kfree(cqp->scratch_array); |
| 603 | cqp->scratch_array = NULL; | 627 | cqp->scratch_array = NULL; |
| 604 | kfree(cqp->cqp_requests); | 628 | kfree(cqp->cqp_requests); |
| @@ -631,7 +655,9 @@ static void irdma_destroy_aeq(struct irdma_pci_f *rf) · 26 unmodified lines | |||
| 631 | int status = -EBUSY; | 655 | int status = -EBUSY; |
| 632 | 656 | ||
| 633 | if (!rf->msix_shared) { | 657 | if (!rf->msix_shared) { |
| 634 | rf->sc_dev.irq_ops->irdma_cfg_aeq(&rf->sc_dev, rf->iw_msixtbl->idx, false); | 658 | if (rf->sc_dev.privileged) |
| 659 | rf->sc_dev.irq_ops->irdma_cfg_aeq(&rf->sc_dev, | ||
| 660 | rf->iw_msixtbl->idx, false); | ||
| 635 | irdma_destroy_irq(rf, rf->iw_msixtbl, rf); | 661 | irdma_destroy_irq(rf, rf->iw_msixtbl, rf); |
| 636 | } | 662 | } |
| 637 | if (rf->reset) | 663 | if (rf->reset) |
| @@ -697,9 +723,10 @@ static void irdma_del_ceq_0(struct irdma_pci_f *rf) · 59 unmodified lines | |||
| 697 | 723 | ||
| 698 | if (rf->msix_shared) { | 724 | if (rf->msix_shared) { |
| 699 | msix_vec = &rf->iw_msixtbl[0]; | 725 | msix_vec = &rf->iw_msixtbl[0]; |
| 700 | rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev, | 726 | if (rf->sc_dev.privileged) |
| 701 | msix_vec->ceq_id, | 727 | rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev, |
| 702 | msix_vec->idx, false); | 728 | msix_vec->ceq_id, |
| 729 | msix_vec->idx, false); | ||
| 703 | irdma_destroy_irq(rf, msix_vec, rf); | 730 | irdma_destroy_irq(rf, msix_vec, rf); |
| 704 | } else { | 731 | } else { |
| 705 | msix_vec = &rf->iw_msixtbl[1]; | 732 | msix_vec = &rf->iw_msixtbl[1]; |
| @@ -730,8 +757,10 @@ static void irdma_del_ceqs(struct irdma_pci_f *rf) · 24 unmodified lines | |||
| 730 | msix_vec = &rf->iw_msixtbl[2]; | 757 | msix_vec = &rf->iw_msixtbl[2]; |
| 731 | 758 | ||
| 732 | for (i = 1; i < rf->ceqs_count; i++, msix_vec++, iwceq++) { | 759 | for (i = 1; i < rf->ceqs_count; i++, msix_vec++, iwceq++) { |
| 733 | rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev, msix_vec->ceq_id, | 760 | if (rf->sc_dev.privileged) |
| 734 | msix_vec->idx, false); | 761 | rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev, |
| 762 | msix_vec->ceq_id, | ||
| 763 | msix_vec->idx, false); | ||
| 735 | irdma_destroy_irq(rf, msix_vec, iwceq); | 764 | irdma_destroy_irq(rf, msix_vec, iwceq); |
| 736 | irdma_cqp_ceq_cmd(&rf->sc_dev, &iwceq->sc_ceq, | 765 | irdma_cqp_ceq_cmd(&rf->sc_dev, &iwceq->sc_ceq, |
| 737 | IRDMA_OP_CEQ_DESTROY); | 766 | IRDMA_OP_CEQ_DESTROY); |
| @@ -942,6 +971,13 @@ static int irdma_create_cqp(struct irdma_pci_f *rf) · 204 unmodified lines | |||
| 942 | goto err_scratch; | 971 | goto err_scratch; |
| 943 | } | 972 | } |
| 944 | 973 | ||
| 974 | cqp->oop_op_array = kcalloc(sqsize, sizeof(*cqp->oop_op_array), | ||
| 975 | GFP_KERNEL); | ||
| 976 | if (!cqp->oop_op_array) { | ||
| 977 | status = -ENOMEM; | ||
| 978 | goto err_oop; | ||
| 979 | } | ||
| 980 | cqp_init_info.ooo_op_array = cqp->oop_op_array; | ||
| 945 | dev->cqp = &cqp->sc_cqp; | 981 | dev->cqp = &cqp->sc_cqp; |
| 946 | dev->cqp->dev = dev; | 982 | dev->cqp->dev = dev; |
| 947 | cqp->sq.size = ALIGN(sizeof(struct irdma_cqp_sq_wqe) * sqsize, | 983 | cqp->sq.size = ALIGN(sizeof(struct irdma_cqp_sq_wqe) * sqsize, |
| @@ -978,6 +1014,10 @@ static int irdma_create_cqp(struct irdma_pci_f *rf) · 30 unmodified lines | |||
| 978 | case IRDMA_GEN_2: | 1014 | case IRDMA_GEN_2: |
| 979 | cqp_init_info.hw_maj_ver = IRDMA_CQPHC_HW_MAJVER_GEN_2; | 1015 | cqp_init_info.hw_maj_ver = IRDMA_CQPHC_HW_MAJVER_GEN_2; |
| 980 | break; | 1016 | break; |
| 1017 | case IRDMA_GEN_3: | ||
| 1018 | cqp_init_info.hw_maj_ver = IRDMA_CQPHC_HW_MAJVER_GEN_3; | ||
| 1019 | cqp_init_info.ts_override = 1; | ||
| 1020 | break; | ||
| 981 | } | 1021 | } |
| 982 | status = irdma_sc_cqp_init(dev->cqp, &cqp_init_info); | 1022 | status = irdma_sc_cqp_init(dev->cqp, &cqp_init_info); |
| 983 | if (status) { | 1023 | if (status) { |
| @@ -1012,6 +1052,9 @@ err_ctx: · 28 unmodified lines | |||
| 1012 | cqp->sq.va, cqp->sq.pa); | 1052 | cqp->sq.va, cqp->sq.pa); |
| 1013 | cqp->sq.va = NULL; | 1053 | cqp->sq.va = NULL; |
| 1014 | err_sq: | 1054 | err_sq: |
| 1055 | kfree(cqp->oop_op_array); | ||
| 1056 | cqp->oop_op_array = NULL; | ||
| 1057 | err_oop: | ||
| 1015 | kfree(cqp->scratch_array); | 1058 | kfree(cqp->scratch_array); |
| 1016 | cqp->scratch_array = NULL; | 1059 | cqp->scratch_array = NULL; |
| 1017 | err_scratch: | 1060 | err_scratch: |
| @@ -1033,13 +1076,15 @@ static int irdma_create_ccq(struct irdma_pci_f *rf) · 15 unmodified lines | |||
| 1033 | struct irdma_sc_dev *dev = &rf->sc_dev; | 1076 | struct irdma_sc_dev *dev = &rf->sc_dev; |
| 1034 | struct irdma_ccq_init_info info = {}; | 1077 | struct irdma_ccq_init_info info = {}; |
| 1035 | struct irdma_ccq *ccq = &rf->ccq; | 1078 | struct irdma_ccq *ccq = &rf->ccq; |
| 1079 | int ccq_size; | ||
| 1036 | int status; | 1080 | int status; |
| 1037 | 1081 | ||
| 1038 | dev->ccq = &ccq->sc_cq; | 1082 | dev->ccq = &ccq->sc_cq; |
| 1039 | dev->ccq->dev = dev; | 1083 | dev->ccq->dev = dev; |
| 1040 | info.dev = dev; | 1084 | info.dev = dev; |
| 1085 | ccq_size = (rf->rdma_ver >= IRDMA_GEN_3) ? IW_GEN_3_CCQ_SIZE : IW_CCQ_SIZE; | ||
| 1041 | ccq->shadow_area.size = sizeof(struct irdma_cq_shadow_area); | 1086 | ccq->shadow_area.size = sizeof(struct irdma_cq_shadow_area); |
| 1042 | ccq->mem_cq.size = ALIGN(sizeof(struct irdma_cqe) * IW_CCQ_SIZE, | 1087 | ccq->mem_cq.size = ALIGN(sizeof(struct irdma_cqe) * ccq_size, |
| 1043 | IRDMA_CQ0_ALIGNMENT); | 1088 | IRDMA_CQ0_ALIGNMENT); |
| 1044 | ccq->mem_cq.va = dma_alloc_coherent(dev->hw->device, ccq->mem_cq.size, | 1089 | ccq->mem_cq.va = dma_alloc_coherent(dev->hw->device, ccq->mem_cq.size, |
| 1045 | &ccq->mem_cq.pa, GFP_KERNEL); | 1090 | &ccq->mem_cq.pa, GFP_KERNEL); |
| @@ -1056,7 +1101,7 @@ static int irdma_create_ccq(struct irdma_pci_f *rf) · 10 unmodified lines | |||
| 1056 | /* populate the ccq init info */ | 1101 | /* populate the ccq init info */ |
| 1057 | info.cq_base = ccq->mem_cq.va; | 1102 | info.cq_base = ccq->mem_cq.va; |
| 1058 | info.cq_pa = ccq->mem_cq.pa; | 1103 | info.cq_pa = ccq->mem_cq.pa; |
| 1059 | info.num_elem = IW_CCQ_SIZE; | 1104 | info.num_elem = ccq_size; |
| 1060 | info.shadow_area = ccq->shadow_area.va; | 1105 | info.shadow_area = ccq->shadow_area.va; |
| 1061 | info.shadow_area_pa = ccq->shadow_area.pa; | 1106 | info.shadow_area_pa = ccq->shadow_area.pa; |
| 1062 | info.ceqe_mask = false; | 1107 | info.ceqe_mask = false; |
| @@ -1140,9 +1185,13 @@ static int irdma_cfg_ceq_vector(struct irdma_pci_f *rf, struct irdma_ceq *iwceq, · 77 unmodified lines | |||
| 1140 | } | 1185 | } |
| 1141 | 1186 | ||
| 1142 | msix_vec->ceq_id = ceq_id; | 1187 | msix_vec->ceq_id = ceq_id; |
| 1143 | rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev, ceq_id, msix_vec->idx, true); | 1188 | if (rf->sc_dev.privileged) |
| 1144 | 1189 | rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev, ceq_id, | |
| 1145 | return 0; | 1190 | msix_vec->idx, true); |
| 1191 | else | ||
| 1192 | status = irdma_vchnl_req_ceq_vec_map(&rf->sc_dev, ceq_id, | ||
| 1193 | msix_vec->idx); | ||
| 1194 | return status; | ||
| 1146 | } | 1195 | } |
| 1147 | 1196 | ||
| 1148 | /** | 1197 | /** |
| @@ -1155,7 +1204,7 @@ static int irdma_cfg_ceq_vector(struct irdma_pci_f *rf, struct irdma_ceq *iwceq, · 6 unmodified lines | |||
| 1155 | static int irdma_cfg_aeq_vector(struct irdma_pci_f *rf) | 1204 | static int irdma_cfg_aeq_vector(struct irdma_pci_f *rf) |
| 1156 | { | 1205 | { |
| 1157 | struct irdma_msix_vector *msix_vec = rf->iw_msixtbl; | 1206 | struct irdma_msix_vector *msix_vec = rf->iw_msixtbl; |
| 1158 | u32 ret = 0; | 1207 | int ret = 0; |
| 1159 | 1208 | ||
| 1160 | if (!rf->msix_shared) { | 1209 | if (!rf->msix_shared) { |
| 1161 | snprintf(msix_vec->name, sizeof(msix_vec->name) - 1, | 1210 | snprintf(msix_vec->name, sizeof(msix_vec->name) - 1, |
| @@ -1166,12 +1215,16 @@ static int irdma_cfg_aeq_vector(struct irdma_pci_f *rf) · 4 unmodified lines | |||
| 1166 | } | 1215 | } |
| 1167 | if (ret) { | 1216 | if (ret) { |
| 1168 | ibdev_dbg(&rf->iwdev->ibdev, "ERR: aeq irq config fail\n"); | 1217 | ibdev_dbg(&rf->iwdev->ibdev, "ERR: aeq irq config fail\n"); |
| 1169 | return -EINVAL; | 1218 | return ret; |
| 1170 | } | 1219 | } |
| 1171 | 1220 | ||
| 1172 | rf->sc_dev.irq_ops->irdma_cfg_aeq(&rf->sc_dev, msix_vec->idx, true); | 1221 | if (rf->sc_dev.privileged) |
| 1222 | rf->sc_dev.irq_ops->irdma_cfg_aeq(&rf->sc_dev, msix_vec->idx, | ||
| 1223 | true); | ||
| 1224 | else | ||
| 1225 | ret = irdma_vchnl_req_aeq_vec_map(&rf->sc_dev, msix_vec->idx); | ||
| 1173 | 1226 | ||
| 1174 | return 0; | 1227 | return ret; |
| 1175 | } | 1228 | } |
| 1176 | 1229 | ||
| 1177 | /** | 1230 | /** |
| @@ -1179,13 +1232,13 @@ static int irdma_cfg_aeq_vector(struct irdma_pci_f *rf) · 1 unmodified line | |||
| 1179 | * @rf: RDMA PCI function | 1232 | * @rf: RDMA PCI function |
| 1180 | * @iwceq: pointer to the ceq resources to be created | 1233 | * @iwceq: pointer to the ceq resources to be created |
| 1181 | * @ceq_id: the id number of the iwceq | 1234 | * @ceq_id: the id number of the iwceq |
| 1182 | * @vsi: SC vsi struct | 1235 | * @vsi_idx: vsi idx |
| 1183 | * | 1236 | * |
| 1184 | * Return 0, if the ceq and the resources associated with it | 1237 | * Return 0, if the ceq and the resources associated with it |
| 1185 | * are successfully created, otherwise return error | 1238 | * are successfully created, otherwise return error |
| 1186 | */ | 1239 | */ |
| 1187 | static int irdma_create_ceq(struct irdma_pci_f *rf, struct irdma_ceq *iwceq, | 1240 | static int irdma_create_ceq(struct irdma_pci_f *rf, struct irdma_ceq *iwceq, |
| 1188 | u32 ceq_id, struct irdma_sc_vsi *vsi) | 1241 | u32 ceq_id, u16 vsi_idx) |
| 1189 | { | 1242 | { |
| 1190 | int status; | 1243 | int status; |
| 1191 | struct irdma_ceq_init_info info = {}; | 1244 | struct irdma_ceq_init_info info = {}; |
| @@ -1209,7 +1262,7 @@ static int irdma_create_ceq(struct irdma_pci_f *rf, struct irdma_ceq *iwceq, · 17 unmodified lines | |||
| 1209 | info.elem_cnt = ceq_size; | 1262 | info.elem_cnt = ceq_size; |
| 1210 | iwceq->sc_ceq.ceq_id = ceq_id; | 1263 | iwceq->sc_ceq.ceq_id = ceq_id; |
| 1211 | info.dev = dev; | 1264 | info.dev = dev; |
| 1212 | info.vsi = vsi; | 1265 | info.vsi_idx = vsi_idx; |
| 1213 | status = irdma_sc_ceq_init(&iwceq->sc_ceq, &info); | 1266 | status = irdma_sc_ceq_init(&iwceq->sc_ceq, &info); |
| 1214 | if (!status) { | 1267 | if (!status) { |
| 1215 | if (dev->ceq_valid) | 1268 | if (dev->ceq_valid) |
| @@ -1252,7 +1305,7 @@ static int irdma_setup_ceq_0(struct irdma_pci_f *rf) · 36 unmodified lines | |||
| 1252 | } | 1305 | } |
| 1253 | 1306 | ||
| 1254 | iwceq = &rf->ceqlist[0]; | 1307 | iwceq = &rf->ceqlist[0]; |
| 1255 | status = irdma_create_ceq(rf, iwceq, 0, &rf->default_vsi); | 1308 | status = irdma_create_ceq(rf, iwceq, 0, rf->default_vsi.vsi_idx); |
| 1256 | if (status) { | 1309 | if (status) { |
| 1257 | ibdev_dbg(&rf->iwdev->ibdev, "ERR: create ceq status = %d\n", | 1310 | ibdev_dbg(&rf->iwdev->ibdev, "ERR: create ceq status = %d\n", |
| 1258 | status); | 1311 | status); |
| @@ -1287,13 +1340,13 @@ exit: · 28 unmodified lines | |||
| 1287 | /** | 1340 | /** |
| 1288 | * irdma_setup_ceqs - manage the device ceq's and their interrupt resources | 1341 | * irdma_setup_ceqs - manage the device ceq's and their interrupt resources |
| 1289 | * @rf: RDMA PCI function | 1342 | * @rf: RDMA PCI function |
| 1290 | * @vsi: VSI structure for this CEQ | 1343 | * @vsi_idx: vsi_idx for this CEQ |
| 1291 | * | 1344 | * |
| 1292 | * Allocate a list for all device completion event queues | 1345 | * Allocate a list for all device completion event queues |
| 1293 | * Create the ceq's and configure their msix interrupt vectors | 1346 | * Create the ceq's and configure their msix interrupt vectors |
| 1294 | * Return 0, if ceqs are successfully set up, otherwise return error | 1347 | * Return 0, if ceqs are successfully set up, otherwise return error |
| 1295 | */ | 1348 | */ |
| 1296 | static int irdma_setup_ceqs(struct irdma_pci_f *rf, struct irdma_sc_vsi *vsi) | 1349 | static int irdma_setup_ceqs(struct irdma_pci_f *rf, u16 vsi_idx) |
| 1297 | { | 1350 | { |
| 1298 | u32 i; | 1351 | u32 i; |
| 1299 | u32 ceq_id; | 1352 | u32 ceq_id; |
| @@ -1306,7 +1359,7 @@ static int irdma_setup_ceqs(struct irdma_pci_f *rf, struct irdma_sc_vsi *vsi) · 6 unmodified lines | |||
| 1306 | i = (rf->msix_shared) ? 1 : 2; | 1359 | i = (rf->msix_shared) ? 1 : 2; |
| 1307 | for (ceq_id = 1; i < num_ceqs; i++, ceq_id++) { | 1360 | for (ceq_id = 1; i < num_ceqs; i++, ceq_id++) { |
| 1308 | iwceq = &rf->ceqlist[ceq_id]; | 1361 | iwceq = &rf->ceqlist[ceq_id]; |
| 1309 | status = irdma_create_ceq(rf, iwceq, ceq_id, vsi); | 1362 | status = irdma_create_ceq(rf, iwceq, ceq_id, vsi_idx); |
| 1310 | if (status) { | 1363 | if (status) { |
| 1311 | ibdev_dbg(&rf->iwdev->ibdev, | 1364 | ibdev_dbg(&rf->iwdev->ibdev, |
| 1312 | "ERR: create ceq status = %d\n", status); | 1365 | "ERR: create ceq status = %d\n", status); |
| @@ -1387,7 +1440,10 @@ static int irdma_create_aeq(struct irdma_pci_f *rf) · 74 unmodified lines | |||
| 1387 | aeq_size = multiplier * hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt + | 1440 | aeq_size = multiplier * hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt + |
| 1388 | hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt; | 1441 | hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt; |
| 1389 | aeq_size = min(aeq_size, dev->hw_attrs.max_hw_aeq_size); | 1442 | aeq_size = min(aeq_size, dev->hw_attrs.max_hw_aeq_size); |
| 1390 | 1443 | /* GEN_3 does not support virtual AEQ. Cap at max Kernel alloc size */ | |
| 1444 | if (rf->rdma_ver == IRDMA_GEN_3) | ||
| 1445 | aeq_size = min(aeq_size, (u32)((PAGE_SIZE << MAX_PAGE_ORDER) / | ||
| 1446 | sizeof(struct irdma_sc_aeqe))); | ||
| 1391 | aeq->mem.size = ALIGN(sizeof(struct irdma_sc_aeqe) * aeq_size, | 1447 | aeq->mem.size = ALIGN(sizeof(struct irdma_sc_aeqe) * aeq_size, |
| 1392 | IRDMA_AEQ_ALIGNMENT); | 1448 | IRDMA_AEQ_ALIGNMENT); |
| 1393 | aeq->mem.va = dma_alloc_coherent(dev->hw->device, aeq->mem.size, | 1449 | aeq->mem.va = dma_alloc_coherent(dev->hw->device, aeq->mem.size, |
| @@ -1395,6 +1451,8 @@ static int irdma_create_aeq(struct irdma_pci_f *rf) · 1 unmodified line | |||
| 1395 | GFP_KERNEL | __GFP_NOWARN); | 1451 | GFP_KERNEL | __GFP_NOWARN); |
| 1396 | if (aeq->mem.va) | 1452 | if (aeq->mem.va) |
| 1397 | goto skip_virt_aeq; | 1453 | goto skip_virt_aeq; |
| 1454 | else if (rf->rdma_ver == IRDMA_GEN_3) | ||
| 1455 | return -ENOMEM; | ||
| 1398 | 1456 | ||
| 1399 | /* physically mapped aeq failed. setup virtual aeq */ | 1457 | /* physically mapped aeq failed. setup virtual aeq */ |
| 1400 | status = irdma_create_virt_aeq(rf, aeq_size); | 1458 | status = irdma_create_virt_aeq(rf, aeq_size); |
| @@ -1569,6 +1627,8 @@ static void irdma_del_init_mem(struct irdma_pci_f *rf) · 168 unmodified lines | |||
| 1569 | { | 1627 | { |
| 1570 | struct irdma_sc_dev *dev = &rf->sc_dev; | 1628 | struct irdma_sc_dev *dev = &rf->sc_dev; |
| 1571 | 1629 | ||
| 1630 | if (!rf->sc_dev.privileged) | ||
| 1631 | irdma_vchnl_req_put_hmc_fcn(&rf->sc_dev); | ||
| 1572 | kfree(dev->hmc_info->sd_table.sd_entry); | 1632 | kfree(dev->hmc_info->sd_table.sd_entry); |
| 1573 | dev->hmc_info->sd_table.sd_entry = NULL; | 1633 | dev->hmc_info->sd_table.sd_entry = NULL; |
| 1574 | vfree(rf->mem_rsrc); | 1634 | vfree(rf->mem_rsrc); |
| @@ -1635,6 +1695,7 @@ static int irdma_initialize_dev(struct irdma_pci_f *rf) · 60 unmodified lines | |||
| 1635 | 1695 | ||
| 1636 | info.bar0 = rf->hw.hw_addr; | 1696 | info.bar0 = rf->hw.hw_addr; |
| 1637 | info.hmc_fn_id = rf->pf_id; | 1697 | info.hmc_fn_id = rf->pf_id; |
| 1698 | info.protocol_used = rf->protocol_used; | ||
| 1638 | info.hw = &rf->hw; | 1699 | info.hw = &rf->hw; |
| 1639 | status = irdma_sc_dev_init(rf->rdma_ver, &rf->sc_dev, &info); | 1700 | status = irdma_sc_dev_init(rf->rdma_ver, &rf->sc_dev, &info); |
| 1640 | if (status) | 1701 | if (status) |
| @@ -1665,9 +1726,6 @@ void irdma_rt_deinit_hw(struct irdma_device *iwdev) · 24 unmodified lines | |||
| 1665 | irdma_del_local_mac_entry(iwdev->rf, | 1726 | irdma_del_local_mac_entry(iwdev->rf, |
| 1666 | (u8)iwdev->mac_ip_table_idx); | 1727 | (u8)iwdev->mac_ip_table_idx); |
| 1667 | fallthrough; | 1728 | fallthrough; |
| 1668 | case AEQ_CREATED: | ||
| 1669 | case PBLE_CHUNK_MEM: | ||
| 1670 | case CEQS_CREATED: | ||
| 1671 | case IEQ_CREATED: | 1729 | case IEQ_CREATED: |
| 1672 | if (!iwdev->roce_mode) | 1730 | if (!iwdev->roce_mode) |
| 1673 | irdma_puda_dele_rsrc(&iwdev->vsi, IRDMA_PUDA_RSRC_TYPE_IEQ, | 1731 | irdma_puda_dele_rsrc(&iwdev->vsi, IRDMA_PUDA_RSRC_TYPE_IEQ, |
| @@ -1740,7 +1798,9 @@ static void irdma_get_used_rsrc(struct irdma_device *iwdev) · 66 unmodified lines | |||
| 1740 | iwdev->rf->used_qps = find_first_zero_bit(iwdev->rf->allocated_qps, | 1798 | iwdev->rf->used_qps = find_first_zero_bit(iwdev->rf->allocated_qps, |
| 1741 | iwdev->rf->max_qp); | 1799 | iwdev->rf->max_qp); |
| 1742 | iwdev->rf->used_cqs = find_first_zero_bit(iwdev->rf->allocated_cqs, | 1800 | iwdev->rf->used_cqs = find_first_zero_bit(iwdev->rf->allocated_cqs, |
| 1743 | iwdev->rf->max_cq); | 1801 | iwdev->rf->max_cq); |
| 1802 | iwdev->rf->used_srqs = find_first_zero_bit(iwdev->rf->allocated_srqs, | ||
| 1803 | iwdev->rf->max_srq); | ||
| 1744 | iwdev->rf->used_mrs = find_first_zero_bit(iwdev->rf->allocated_mrs, | 1804 | iwdev->rf->used_mrs = find_first_zero_bit(iwdev->rf->allocated_mrs, |
| 1745 | iwdev->rf->max_mr); | 1805 | iwdev->rf->max_mr); |
| 1746 | } | 1806 | } |
| @@ -1750,13 +1810,17 @@ void irdma_ctrl_deinit_hw(struct irdma_pci_f *rf) · 3 unmodified lines | |||
| 1750 | enum init_completion_state state = rf->init_state; | 1810 | enum init_completion_state state = rf->init_state; |
| 1751 | 1811 | ||
| 1752 | rf->init_state = INVALID_STATE; | 1812 | rf->init_state = INVALID_STATE; |
| 1753 | if (rf->rsrc_created) { | 1813 | |
| 1814 | switch (state) { | ||
| 1815 | case AEQ_CREATED: | ||
| 1754 | irdma_destroy_aeq(rf); | 1816 | irdma_destroy_aeq(rf); |
| 1817 | fallthrough; | ||
| 1818 | case PBLE_CHUNK_MEM: | ||
| 1755 | irdma_destroy_pble_prm(rf->pble_rsrc); | 1819 | irdma_destroy_pble_prm(rf->pble_rsrc); |
| 1820 | fallthrough; | ||
| 1821 | case CEQS_CREATED: | ||
| 1756 | irdma_del_ceqs(rf); | 1822 | irdma_del_ceqs(rf); |
| 1757 | rf->rsrc_created = false; | 1823 | fallthrough; |
| 1758 | } | ||
| 1759 | switch (state) { | ||
| 1760 | case CEQ0_CREATED: | 1824 | case CEQ0_CREATED: |
| 1761 | irdma_del_ceq_0(rf); | 1825 | irdma_del_ceq_0(rf); |
| 1762 | fallthrough; | 1826 | fallthrough; |
| @@ -1835,32 +1899,6 @@ int irdma_rt_init_hw(struct irdma_device *iwdev, · 72 unmodified lines | |||
| 1835 | break; | 1899 | break; |
| 1836 | iwdev->init_state = IEQ_CREATED; | 1900 | iwdev->init_state = IEQ_CREATED; |
| 1837 | } | 1901 | } |
| 1838 | if (!rf->rsrc_created) { | ||
| 1839 | status = irdma_setup_ceqs(rf, &iwdev->vsi); | ||
| 1840 | if (status) | ||
| 1841 | break; | ||
| 1842 | |||
| 1843 | iwdev->init_state = CEQS_CREATED; | ||
| 1844 | |||
| 1845 | status = irdma_hmc_init_pble(&rf->sc_dev, | ||
| 1846 | rf->pble_rsrc); | ||
| 1847 | if (status) { | ||
| 1848 | irdma_del_ceqs(rf); | ||
| 1849 | break; | ||
| 1850 | } | ||
| 1851 | |||
| 1852 | iwdev->init_state = PBLE_CHUNK_MEM; | ||
| 1853 | |||
| 1854 | status = irdma_setup_aeq(rf); | ||
| 1855 | if (status) { | ||
| 1856 | irdma_destroy_pble_prm(rf->pble_rsrc); | ||
| 1857 | irdma_del_ceqs(rf); | ||
| 1858 | break; | ||
| 1859 | } | ||
| 1860 | iwdev->init_state = AEQ_CREATED; | ||
| 1861 | rf->rsrc_created = true; | ||
| 1862 | } | ||
| 1863 | |||
| 1864 | if (iwdev->rf->sc_dev.hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_1) | 1902 | if (iwdev->rf->sc_dev.hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_1) |
| 1865 | irdma_alloc_set_mac(iwdev); | 1903 | irdma_alloc_set_mac(iwdev); |
| 1866 | irdma_add_ip(iwdev); | 1904 | irdma_add_ip(iwdev); |
| @@ -1907,6 +1945,13 @@ int irdma_ctrl_init_hw(struct irdma_pci_f *rf) · 40 unmodified lines | |||
| 1907 | break; | 1945 | break; |
| 1908 | rf->init_state = CQP_CREATED; | 1946 | rf->init_state = CQP_CREATED; |
| 1909 | 1947 | ||
| 1948 | dev->feature_info[IRDMA_FEATURE_FW_INFO] = IRDMA_FW_VER_DEFAULT; | ||
| 1949 | if (rf->rdma_ver != IRDMA_GEN_1) { | ||
| 1950 | status = irdma_get_rdma_features(dev); | ||
| 1951 | if (status) | ||
| 1952 | break; | ||
| 1953 | } | ||
| 1954 | |||
| 1910 | status = irdma_hmc_setup(rf); | 1955 | status = irdma_hmc_setup(rf); |
| 1911 | if (status) | 1956 | if (status) |
| 1912 | break; | 1957 | break; |
| @@ -1922,13 +1967,6 @@ int irdma_ctrl_init_hw(struct irdma_pci_f *rf) · 9 unmodified lines | |||
| 1922 | break; | 1967 | break; |
| 1923 | rf->init_state = CCQ_CREATED; | 1968 | rf->init_state = CCQ_CREATED; |
| 1924 | 1969 | ||
| 1925 | dev->feature_info[IRDMA_FEATURE_FW_INFO] = IRDMA_FW_VER_DEFAULT; | ||
| 1926 | if (rf->rdma_ver != IRDMA_GEN_1) { | ||
| 1927 | status = irdma_get_rdma_features(dev); | ||
| 1928 | if (status) | ||
| 1929 | break; | ||
| 1930 | } | ||
| 1931 | |||
| 1932 | status = irdma_setup_ceq_0(rf); | 1970 | status = irdma_setup_ceq_0(rf); |
| 1933 | if (status) | 1971 | if (status) |
| 1934 | break; | 1972 | break; |
| @@ -1942,6 +1980,25 @@ int irdma_ctrl_init_hw(struct irdma_pci_f *rf) · 7 unmodified lines | |||
| 1942 | } | 1980 | } |
| 1943 | INIT_WORK(&rf->cqp_cmpl_work, cqp_compl_worker); | 1981 | INIT_WORK(&rf->cqp_cmpl_work, cqp_compl_worker); |
| 1944 | irdma_sc_ccq_arm(dev->ccq); | 1982 | irdma_sc_ccq_arm(dev->ccq); |
| 1983 | |||
| 1984 | status = irdma_setup_ceqs(rf, rf->iwdev ? rf->iwdev->vsi_num : 0); | ||
| 1985 | if (status) | ||
| 1986 | break; | ||
| 1987 | |||
| 1988 | rf->init_state = CEQS_CREATED; | ||
| 1989 | |||
| 1990 | status = irdma_hmc_init_pble(&rf->sc_dev, | ||
| 1991 | rf->pble_rsrc); | ||
| 1992 | if (status) | ||
| 1993 | break; | ||
| 1994 | |||
| 1995 | rf->init_state = PBLE_CHUNK_MEM; | ||
| 1996 | |||
| 1997 | status = irdma_setup_aeq(rf); | ||
| 1998 | if (status) | ||
| 1999 | break; | ||
| 2000 | rf->init_state = AEQ_CREATED; | ||
| 2001 | |||
| 1945 | return 0; | 2002 | return 0; |
| 1946 | } while (0); | 2003 | } while (0); |
| 1947 | 2004 | ||
| @@ -1960,7 +2017,8 @@ static void irdma_set_hw_rsrc(struct irdma_pci_f *rf) · 12 unmodified lines | |||
| 1960 | rf->allocated_qps = (void *)(rf->mem_rsrc + | 2017 | rf->allocated_qps = (void *)(rf->mem_rsrc + |
| 1961 | (sizeof(struct irdma_arp_entry) * rf->arp_table_size)); | 2018 | (sizeof(struct irdma_arp_entry) * rf->arp_table_size)); |
| 1962 | rf->allocated_cqs = &rf->allocated_qps[BITS_TO_LONGS(rf->max_qp)]; | 2019 | rf->allocated_cqs = &rf->allocated_qps[BITS_TO_LONGS(rf->max_qp)]; |
| 1963 | rf->allocated_mrs = &rf->allocated_cqs[BITS_TO_LONGS(rf->max_cq)]; | 2020 | rf->allocated_srqs = &rf->allocated_cqs[BITS_TO_LONGS(rf->max_cq)]; |
| 2021 | rf->allocated_mrs = &rf->allocated_srqs[BITS_TO_LONGS(rf->max_srq)]; | ||
| 1964 | rf->allocated_pds = &rf->allocated_mrs[BITS_TO_LONGS(rf->max_mr)]; | 2022 | rf->allocated_pds = &rf->allocated_mrs[BITS_TO_LONGS(rf->max_mr)]; |
| 1965 | rf->allocated_ahs = &rf->allocated_pds[BITS_TO_LONGS(rf->max_pd)]; | 2023 | rf->allocated_ahs = &rf->allocated_pds[BITS_TO_LONGS(rf->max_pd)]; |
| 1966 | rf->allocated_mcgs = &rf->allocated_ahs[BITS_TO_LONGS(rf->max_ah)]; | 2024 | rf->allocated_mcgs = &rf->allocated_ahs[BITS_TO_LONGS(rf->max_ah)]; |
| @@ -1988,12 +2046,14 @@ static u32 irdma_calc_mem_rsrc_size(struct irdma_pci_f *rf) · 21 unmodified lines | |||
| 1988 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_qp); | 2046 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_qp); |
| 1989 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_mr); | 2047 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_mr); |
| 1990 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_cq); | 2048 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_cq); |
| 2049 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_srq); | ||
| 1991 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_pd); | 2050 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_pd); |
| 1992 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->arp_table_size); | 2051 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->arp_table_size); |
| 1993 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_ah); | 2052 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_ah); |
| 1994 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_mcg); | 2053 | rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(rf->max_mcg); |
| 1995 | rsrc_size += sizeof(struct irdma_qp **) * rf->max_qp; | 2054 | rsrc_size += sizeof(struct irdma_qp **) * rf->max_qp; |
| 1996 | rsrc_size += sizeof(struct irdma_cq **) * rf->max_cq; | 2055 | rsrc_size += sizeof(struct irdma_cq **) * rf->max_cq; |
| 2056 | rsrc_size += sizeof(struct irdma_srq **) * rf->max_srq; | ||
| 1997 | 2057 | ||
| 1998 | return rsrc_size; | 2058 | return rsrc_size; |
| 1999 | } | 2059 | } |
| @@ -2021,6 +2081,7 @@ u32 irdma_initialize_hw_rsrc(struct irdma_pci_f *rf) · 21 unmodified lines | |||
| 2021 | rf->max_qp = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt; | 2081 | rf->max_qp = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt; |
| 2022 | rf->max_mr = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt; | 2082 | rf->max_mr = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt; |
| 2023 | rf->max_cq = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt; | 2083 | rf->max_cq = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt; |
| 2084 | rf->max_srq = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_SRQ].cnt; | ||
| 2024 | rf->max_pd = rf->sc_dev.hw_attrs.max_hw_pds; | 2085 | rf->max_pd = rf->sc_dev.hw_attrs.max_hw_pds; |
| 2025 | rf->arp_table_size = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_ARP].cnt; | 2086 | rf->arp_table_size = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_ARP].cnt; |
| 2026 | rf->max_ah = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].cnt; | 2087 | rf->max_ah = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_FSIAV].cnt; |
| @@ -2040,6 +2101,7 @@ u32 irdma_initialize_hw_rsrc(struct irdma_pci_f *rf) · 13 unmodified lines | |||
| 2040 | set_bit(0, rf->allocated_mrs); | 2101 | set_bit(0, rf->allocated_mrs); |
| 2041 | set_bit(0, rf->allocated_qps); | 2102 | set_bit(0, rf->allocated_qps); |
| 2042 | set_bit(0, rf->allocated_cqs); | 2103 | set_bit(0, rf->allocated_cqs); |
| 2104 | set_bit(0, rf->allocated_srqs); | ||
| 2043 | set_bit(0, rf->allocated_pds); | 2105 | set_bit(0, rf->allocated_pds); |
| 2044 | set_bit(0, rf->allocated_arps); | 2106 | set_bit(0, rf->allocated_arps); |
| 2045 | set_bit(0, rf->allocated_ahs); | 2107 | set_bit(0, rf->allocated_ahs); |
| @@ -2100,15 +2162,16 @@ void irdma_cqp_ce_handler(struct irdma_pci_f *rf, struct irdma_sc_cq *cq) · 54 unmodified lines | |||
| 2100 | cqp_request->compl_info.op_ret_val = info.op_ret_val; | 2162 | cqp_request->compl_info.op_ret_val = info.op_ret_val; |
| 2101 | cqp_request->compl_info.error = info.error; | 2163 | cqp_request->compl_info.error = info.error; |
| 2102 | 2164 | ||
| 2103 | if (cqp_request->waiting) { | 2165 | /* |
| 2104 | WRITE_ONCE(cqp_request->request_done, true); | 2166 | * If this is deferred or pending completion, then mark |
| 2105 | wake_up(&cqp_request->waitq); | 2167 | * CQP request as pending to not block the CQ, but don't |
| 2106 | irdma_put_cqp_request(&rf->cqp, cqp_request); | 2168 | * release CQP request, as it is still on the OOO list. |
| 2107 | } else { | 2169 | */ |
| 2108 | if (cqp_request->callback_fcn) | 2170 | if (info.pending) |
| 2109 | cqp_request->callback_fcn(cqp_request); | 2171 | cqp_request->pending = true; |
| 2110 | irdma_put_cqp_request(&rf->cqp, cqp_request); | 2172 | else |
| 2111 | } | 2173 | irdma_complete_cqp_request(&rf->cqp, |
| 2174 | cqp_request); | ||
| 2112 | } | 2175 | } |
| 2113 | 2176 | ||
| 2114 | cqe_count++; | 2177 | cqe_count++; |
| @@ -2718,7 +2781,9 @@ void irdma_flush_wqes(struct irdma_qp *iwqp, u32 flush_mask) · 603 unmodified lines | |||
| 2718 | struct irdma_pci_f *rf = iwqp->iwdev->rf; | 2781 | struct irdma_pci_f *rf = iwqp->iwdev->rf; |
| 2719 | u8 flush_code = iwqp->sc_qp.flush_code; | 2782 | u8 flush_code = iwqp->sc_qp.flush_code; |
| 2720 | 2783 | ||
| 2721 | if (!(flush_mask & IRDMA_FLUSH_SQ) && !(flush_mask & IRDMA_FLUSH_RQ)) | 2784 | if ((!(flush_mask & IRDMA_FLUSH_SQ) && |
| 2785 | !(flush_mask & IRDMA_FLUSH_RQ)) || | ||
| 2786 | ((flush_mask & IRDMA_REFLUSH) && rf->rdma_ver >= IRDMA_GEN_3)) | ||
| 2722 | return; | 2787 | return; |
| 2723 | 2788 | ||
| 2724 | /* Set flush info fields*/ | 2789 | /* Set flush info fields*/ |
| @@ -2731,6 +2796,10 @@ void irdma_flush_wqes(struct irdma_qp *iwqp, u32 flush_mask) · 6 unmodified lines | |||
| 2731 | info.rq_major_code = IRDMA_FLUSH_MAJOR_ERR; | 2796 | info.rq_major_code = IRDMA_FLUSH_MAJOR_ERR; |
| 2732 | info.rq_minor_code = FLUSH_GENERAL_ERR; | 2797 | info.rq_minor_code = FLUSH_GENERAL_ERR; |
| 2733 | info.userflushcode = true; | 2798 | info.userflushcode = true; |
| 2799 | info.err_sq_idx_valid = iwqp->sc_qp.err_sq_idx_valid; | ||
| 2800 | info.err_sq_idx = iwqp->sc_qp.err_sq_idx; | ||
| 2801 | info.err_rq_idx_valid = iwqp->sc_qp.err_rq_idx_valid; | ||
| 2802 | info.err_rq_idx = iwqp->sc_qp.err_rq_idx; | ||
| 2734 | 2803 | ||
| 2735 | if (flush_mask & IRDMA_REFLUSH) { | 2804 | if (flush_mask & IRDMA_REFLUSH) { |
| 2736 | if (info.sq) | 2805 | if (info.sq) |
| @@ -85,6 +85,7 @@ static u64 i40iw_masks[IRDMA_MAX_MASKS] = { · 84 unmodified lines | |||
| 85 | I40E_CQPSQ_CQ_CEQID, | 85 | I40E_CQPSQ_CQ_CEQID, |
| 86 | I40E_CQPSQ_CQ_CQID, | 86 | I40E_CQPSQ_CQ_CQID, |
| 87 | I40E_COMMIT_FPM_CQCNT, | 87 | I40E_COMMIT_FPM_CQCNT, |
| 88 | I40E_CQPSQ_UPESD_HMCFNID, | ||
| 88 | }; | 89 | }; |
| 89 | 90 | ||
| 90 | static u64 i40iw_shifts[IRDMA_MAX_SHIFTS] = { | 91 | static u64 i40iw_shifts[IRDMA_MAX_SHIFTS] = { |
| @@ -94,6 +95,7 @@ static u64 i40iw_shifts[IRDMA_MAX_SHIFTS] = { · 3 unmodified lines | |||
| 94 | I40E_CQPSQ_CQ_CEQID_S, | 95 | I40E_CQPSQ_CQ_CEQID_S, |
| 95 | I40E_CQPSQ_CQ_CQID_S, | 96 | I40E_CQPSQ_CQ_CQID_S, |
| 96 | I40E_COMMIT_FPM_CQCNT_S, | 97 | I40E_COMMIT_FPM_CQCNT_S, |
| 98 | I40E_CQPSQ_UPESD_HMCFNID_S, | ||
| 97 | }; | 99 | }; |
| 98 | 100 | ||
| 99 | /** | 101 | /** |
| @@ -123,6 +123,8 @@ · 122 unmodified lines | |||
| 123 | #define I40E_CQPSQ_CQ_CQID GENMASK_ULL(15, 0) | 123 | #define I40E_CQPSQ_CQ_CQID GENMASK_ULL(15, 0) |
| 124 | #define I40E_COMMIT_FPM_CQCNT_S 0 | 124 | #define I40E_COMMIT_FPM_CQCNT_S 0 |
| 125 | #define I40E_COMMIT_FPM_CQCNT GENMASK_ULL(17, 0) | 125 | #define I40E_COMMIT_FPM_CQCNT GENMASK_ULL(17, 0) |
| 126 | #define I40E_CQPSQ_UPESD_HMCFNID_S 0 | ||
| 127 | #define I40E_CQPSQ_UPESD_HMCFNID GENMASK_ULL(5, 0) | ||
| 126 | 128 | ||
| 127 | #define I40E_VSIQF_CTL(_VSI) (0x0020D800 + ((_VSI) * 4)) | 129 | #define I40E_VSIQF_CTL(_VSI) (0x0020D800 + ((_VSI) * 4)) |
| 128 | 130 | ||
| @@ -75,6 +75,9 @@ static void i40iw_fill_device_info(struct irdma_device *iwdev, struct i40e_info · 74 unmodified lines | |||
| 75 | struct irdma_pci_f *rf = iwdev->rf; | 75 | struct irdma_pci_f *rf = iwdev->rf; |
| 76 | 76 | ||
| 77 | rf->rdma_ver = IRDMA_GEN_1; | 77 | rf->rdma_ver = IRDMA_GEN_1; |
| 78 | rf->sc_dev.hw = &rf->hw; | ||
| 79 | rf->sc_dev.hw_attrs.uk_attrs.hw_rev = IRDMA_GEN_1; | ||
| 80 | rf->sc_dev.privileged = true; | ||
| 78 | rf->gen_ops.request_reset = i40iw_request_reset; | 81 | rf->gen_ops.request_reset = i40iw_request_reset; |
| 79 | rf->pcidev = cdev_info->pcidev; | 82 | rf->pcidev = cdev_info->pcidev; |
| 80 | rf->pf_id = cdev_info->fid; | 83 | rf->pf_id = cdev_info->fid; |
| @@ -38,6 +38,7 @@ static u64 icrdma_masks[IRDMA_MAX_MASKS] = { · 37 unmodified lines | |||
| 38 | ICRDMA_CQPSQ_CQ_CEQID, | 38 | ICRDMA_CQPSQ_CQ_CEQID, |
| 39 | ICRDMA_CQPSQ_CQ_CQID, | 39 | ICRDMA_CQPSQ_CQ_CQID, |
| 40 | ICRDMA_COMMIT_FPM_CQCNT, | 40 | ICRDMA_COMMIT_FPM_CQCNT, |
| 41 | ICRDMA_CQPSQ_UPESD_HMCFNID, | ||
| 41 | }; | 42 | }; |
| 42 | 43 | ||
| 43 | static u64 icrdma_shifts[IRDMA_MAX_SHIFTS] = { | 44 | static u64 icrdma_shifts[IRDMA_MAX_SHIFTS] = { |
| @@ -47,6 +48,7 @@ static u64 icrdma_shifts[IRDMA_MAX_SHIFTS] = { · 3 unmodified lines | |||
| 47 | ICRDMA_CQPSQ_CQ_CEQID_S, | 48 | ICRDMA_CQPSQ_CQ_CEQID_S, |
| 48 | ICRDMA_CQPSQ_CQ_CQID_S, | 49 | ICRDMA_CQPSQ_CQ_CQID_S, |
| 49 | ICRDMA_COMMIT_FPM_CQCNT_S, | 50 | ICRDMA_COMMIT_FPM_CQCNT_S, |
| 51 | ICRDMA_CQPSQ_UPESD_HMCFNID_S, | ||
| 50 | }; | 52 | }; |
| 51 | 53 | ||
| 52 | /** | 54 | /** |
| @@ -194,6 +196,7 @@ void icrdma_init_hw(struct irdma_sc_dev *dev) · 141 unmodified lines | |||
| 194 | dev->hw_attrs.max_hw_ord = ICRDMA_MAX_ORD_SIZE; | 196 | dev->hw_attrs.max_hw_ord = ICRDMA_MAX_ORD_SIZE; |
| 195 | dev->hw_attrs.max_stat_inst = ICRDMA_MAX_STATS_COUNT; | 197 | dev->hw_attrs.max_stat_inst = ICRDMA_MAX_STATS_COUNT; |
| 196 | dev->hw_attrs.max_stat_idx = IRDMA_HW_STAT_INDEX_MAX_GEN_2; | 198 | dev->hw_attrs.max_stat_idx = IRDMA_HW_STAT_INDEX_MAX_GEN_2; |
| 199 | dev->hw_attrs.max_hw_device_pages = ICRDMA_MAX_PUSH_PAGE_COUNT; | ||
| 197 | 200 | ||
| 198 | dev->hw_attrs.uk_attrs.min_hw_wq_size = ICRDMA_MIN_WQ_SIZE; | 201 | dev->hw_attrs.uk_attrs.min_hw_wq_size = ICRDMA_MIN_WQ_SIZE; |
| 199 | dev->hw_attrs.uk_attrs.max_hw_sq_chunk = IRDMA_MAX_QUANTA_PER_WR; | 202 | dev->hw_attrs.uk_attrs.max_hw_sq_chunk = IRDMA_MAX_QUANTA_PER_WR; |
| @@ -58,14 +58,15 @@ · 57 unmodified lines | |||
| 58 | #define ICRDMA_CQPSQ_CQ_CQID GENMASK_ULL(18, 0) | 58 | #define ICRDMA_CQPSQ_CQ_CQID GENMASK_ULL(18, 0) |
| 59 | #define ICRDMA_COMMIT_FPM_CQCNT_S 0 | 59 | #define ICRDMA_COMMIT_FPM_CQCNT_S 0 |
| 60 | #define ICRDMA_COMMIT_FPM_CQCNT GENMASK_ULL(19, 0) | 60 | #define ICRDMA_COMMIT_FPM_CQCNT GENMASK_ULL(19, 0) |
| 61 | 61 | #define ICRDMA_CQPSQ_UPESD_HMCFNID_S 0 | |
| 62 | #define ICRDMA_CQPSQ_UPESD_HMCFNID GENMASK_ULL(5, 0) | ||
| 62 | enum icrdma_device_caps_const { | 63 | enum icrdma_device_caps_const { |
| 63 | ICRDMA_MAX_STATS_COUNT = 128, | 64 | ICRDMA_MAX_STATS_COUNT = 128, |
| 64 | 65 | ||
| 65 | ICRDMA_MAX_IRD_SIZE = 127, | 66 | ICRDMA_MAX_IRD_SIZE = 127, |
| 66 | ICRDMA_MAX_ORD_SIZE = 255, | 67 | ICRDMA_MAX_ORD_SIZE = 255, |
| 67 | ICRDMA_MIN_WQ_SIZE = 8 /* WQEs */, | 68 | ICRDMA_MIN_WQ_SIZE = 8 /* WQEs */, |
| 68 | 69 | ICRDMA_MAX_PUSH_PAGE_COUNT = 256, | |
| 69 | }; | 70 | }; |
| 70 | 71 | ||
| 71 | void icrdma_init_hw(struct irdma_sc_dev *dev); | 72 | void icrdma_init_hw(struct irdma_sc_dev *dev); |
| @@ -0,0 +1,343 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB | ||
| 2 | /* Copyright (c) 2015 - 2024 Intel Corporation */ | ||
| 3 | |||
| 4 | #include "main.h" | ||
| 5 | #include <linux/net/intel/iidc_rdma_ice.h> | ||
| 6 | |||
| 7 | static void icrdma_prep_tc_change(struct irdma_device *iwdev) | ||
| 8 | { | ||
| 9 | iwdev->vsi.tc_change_pending = true; | ||
| 10 | irdma_sc_suspend_resume_qps(&iwdev->vsi, IRDMA_OP_SUSPEND); | ||
| 11 | |||
| 12 | /* Wait for all qp's to suspend */ | ||
| 13 | wait_event_timeout(iwdev->suspend_wq, | ||
| 14 | !atomic_read(&iwdev->vsi.qp_suspend_reqs), | ||
| 15 | msecs_to_jiffies(IRDMA_EVENT_TIMEOUT_MS)); | ||
| 16 | irdma_ws_reset(&iwdev->vsi); | ||
| 17 | } | ||
| 18 | |||
| 19 | static void icrdma_fill_qos_info(struct irdma_l2params *l2params, | ||
| 20 | struct iidc_rdma_qos_params *qos_info) | ||
| 21 | { | ||
| 22 | int i; | ||
| 23 | |||
| 24 | l2params->num_tc = qos_info->num_tc; | ||
| 25 | l2params->vsi_prio_type = qos_info->vport_priority_type; | ||
| 26 | l2params->vsi_rel_bw = qos_info->vport_relative_bw; | ||
| 27 | for (i = 0; i < l2params->num_tc; i++) { | ||
| 28 | l2params->tc_info[i].egress_virt_up = | ||
| 29 | qos_info->tc_info[i].egress_virt_up; | ||
| 30 | l2params->tc_info[i].ingress_virt_up = | ||
| 31 | qos_info->tc_info[i].ingress_virt_up; | ||
| 32 | l2params->tc_info[i].prio_type = qos_info->tc_info[i].prio_type; | ||
| 33 | l2params->tc_info[i].rel_bw = qos_info->tc_info[i].rel_bw; | ||
| 34 | l2params->tc_info[i].tc_ctx = qos_info->tc_info[i].tc_ctx; | ||
| 35 | } | ||
| 36 | for (i = 0; i < IIDC_MAX_USER_PRIORITY; i++) | ||
| 37 | l2params->up2tc[i] = qos_info->up2tc[i]; | ||
| 38 | if (qos_info->pfc_mode == IIDC_DSCP_PFC_MODE) { | ||
| 39 | l2params->dscp_mode = true; | ||
| 40 | memcpy(l2params->dscp_map, qos_info->dscp_map, sizeof(l2params->dscp_map)); | ||
| 41 | } | ||
| 42 | } | ||
| 43 | |||
| 44 | static void icrdma_iidc_event_handler(struct iidc_rdma_core_dev_info *cdev_info, | ||
| 45 | struct iidc_rdma_event *event) | ||
| 46 | { | ||
| 47 | struct irdma_device *iwdev = dev_get_drvdata(&cdev_info->adev->dev); | ||
| 48 | struct irdma_l2params l2params = {}; | ||
| 49 | |||
| 50 | if (*event->type & BIT(IIDC_RDMA_EVENT_AFTER_MTU_CHANGE)) { | ||
| 51 | ibdev_dbg(&iwdev->ibdev, "CLNT: new MTU = %d\n", iwdev->netdev->mtu); | ||
| 52 | if (iwdev->vsi.mtu != iwdev->netdev->mtu) { | ||
| 53 | l2params.mtu = iwdev->netdev->mtu; | ||
| 54 | l2params.mtu_changed = true; | ||
| 55 | irdma_log_invalid_mtu(l2params.mtu, &iwdev->rf->sc_dev); | ||
| 56 | irdma_change_l2params(&iwdev->vsi, &l2params); | ||
| 57 | } | ||
| 58 | } else if (*event->type & BIT(IIDC_RDMA_EVENT_BEFORE_TC_CHANGE)) { | ||
| 59 | if (iwdev->vsi.tc_change_pending) | ||
| 60 | return; | ||
| 61 | |||
| 62 | icrdma_prep_tc_change(iwdev); | ||
| 63 | } else if (*event->type & BIT(IIDC_RDMA_EVENT_AFTER_TC_CHANGE)) { | ||
| 64 | struct iidc_rdma_priv_dev_info *idc_priv = cdev_info->iidc_priv; | ||
| 65 | |||
| 66 | if (!iwdev->vsi.tc_change_pending) | ||
| 67 | return; | ||
| 68 | |||
| 69 | l2params.tc_changed = true; | ||
| 70 | ibdev_dbg(&iwdev->ibdev, "CLNT: TC Change\n"); | ||
| 71 | |||
| 72 | icrdma_fill_qos_info(&l2params, &idc_priv->qos_info); | ||
| 73 | if (iwdev->rf->protocol_used != IRDMA_IWARP_PROTOCOL_ONLY) | ||
| 74 | iwdev->dcb_vlan_mode = | ||
| 75 | l2params.num_tc > 1 && !l2params.dscp_mode; | ||
| 76 | irdma_change_l2params(&iwdev->vsi, &l2params); | ||
| 77 | } else if (*event->type & BIT(IIDC_RDMA_EVENT_CRIT_ERR)) { | ||
| 78 | ibdev_warn(&iwdev->ibdev, "ICE OICR event notification: oicr = 0x%08x\n", | ||
| 79 | event->reg); | ||
| 80 | if (event->reg & IRDMAPFINT_OICR_PE_CRITERR_M) { | ||
| 81 | u32 pe_criterr; | ||
| 82 | |||
| 83 | pe_criterr = readl(iwdev->rf->sc_dev.hw_regs[IRDMA_GLPE_CRITERR]); | ||
| 84 | #define IRDMA_Q1_RESOURCE_ERR 0x0001024d | ||
| 85 | if (pe_criterr != IRDMA_Q1_RESOURCE_ERR) { | ||
| 86 | ibdev_err(&iwdev->ibdev, "critical PE Error, GLPE_CRITERR=0x%08x\n", | ||
| 87 | pe_criterr); | ||
| 88 | iwdev->rf->reset = true; | ||
| 89 | } else { | ||
| 90 | ibdev_warn(&iwdev->ibdev, "Q1 Resource Check\n"); | ||
| 91 | } | ||
| 92 | } | ||
| 93 | if (event->reg & IRDMAPFINT_OICR_HMC_ERR_M) { | ||
| 94 | ibdev_err(&iwdev->ibdev, "HMC Error\n"); | ||
| 95 | iwdev->rf->reset = true; | ||
| 96 | } | ||
| 97 | if (event->reg & IRDMAPFINT_OICR_PE_PUSH_M) { | ||
| 98 | ibdev_err(&iwdev->ibdev, "PE Push Error\n"); | ||
| 99 | iwdev->rf->reset = true; | ||
| 100 | } | ||
| 101 | if (iwdev->rf->reset) | ||
| 102 | iwdev->rf->gen_ops.request_reset(iwdev->rf); | ||
| 103 | } | ||
| 104 | } | ||
| 105 | |||
| 106 | /** | ||
| 107 | * icrdma_lan_register_qset - Register qset with LAN driver | ||
| 108 | * @vsi: vsi structure | ||
| 109 | * @tc_node: Traffic class node | ||
| 110 | */ | ||
| 111 | static int icrdma_lan_register_qset(struct irdma_sc_vsi *vsi, | ||
| 112 | struct irdma_ws_node *tc_node) | ||
| 113 | { | ||
| 114 | struct irdma_device *iwdev = vsi->back_vsi; | ||
| 115 | struct iidc_rdma_core_dev_info *cdev_info = iwdev->rf->cdev; | ||
| 116 | struct iidc_rdma_qset_params qset = {}; | ||
| 117 | int ret; | ||
| 118 | |||
| 119 | qset.qs_handle = tc_node->qs_handle; | ||
| 120 | qset.tc = tc_node->traffic_class; | ||
| 121 | qset.vport_id = vsi->vsi_idx; | ||
| 122 | ret = ice_add_rdma_qset(cdev_info, &qset); | ||
| 123 | if (ret) { | ||
| 124 | ibdev_dbg(&iwdev->ibdev, "WS: LAN alloc_res for rdma qset failed.\n"); | ||
| 125 | return ret; | ||
| 126 | } | ||
| 127 | |||
| 128 | tc_node->l2_sched_node_id = qset.teid; | ||
| 129 | vsi->qos[tc_node->user_pri].l2_sched_node_id = qset.teid; | ||
| 130 | |||
| 131 | return 0; | ||
| 132 | } | ||
| 133 | |||
| 134 | /** | ||
| 135 | * icrdma_lan_unregister_qset - Unregister qset with LAN driver | ||
| 136 | * @vsi: vsi structure | ||
| 137 | * @tc_node: Traffic class node | ||
| 138 | */ | ||
| 139 | static void icrdma_lan_unregister_qset(struct irdma_sc_vsi *vsi, | ||
| 140 | struct irdma_ws_node *tc_node) | ||
| 141 | { | ||
| 142 | struct irdma_device *iwdev = vsi->back_vsi; | ||
| 143 | struct iidc_rdma_core_dev_info *cdev_info = iwdev->rf->cdev; | ||
| 144 | struct iidc_rdma_qset_params qset = {}; | ||
| 145 | |||
| 146 | qset.qs_handle = tc_node->qs_handle; | ||
| 147 | qset.tc = tc_node->traffic_class; | ||
| 148 | qset.vport_id = vsi->vsi_idx; | ||
| 149 | qset.teid = tc_node->l2_sched_node_id; | ||
| 150 | |||
| 151 | if (ice_del_rdma_qset(cdev_info, &qset)) | ||
| 152 | ibdev_dbg(&iwdev->ibdev, "WS: LAN free_res for rdma qset failed.\n"); | ||
| 153 | } | ||
| 154 | |||
| 155 | /** | ||
| 156 | * icrdma_request_reset - Request a reset | ||
| 157 | * @rf: RDMA PCI function | ||
| 158 | */ | ||
| 159 | static void icrdma_request_reset(struct irdma_pci_f *rf) | ||
| 160 | { | ||
| 161 | ibdev_warn(&rf->iwdev->ibdev, "Requesting a reset\n"); | ||
| 162 | ice_rdma_request_reset(rf->cdev, IIDC_FUNC_RESET); | ||
| 163 | } | ||
| 164 | |||
| 165 | static int icrdma_init_interrupts(struct irdma_pci_f *rf, struct iidc_rdma_core_dev_info *cdev) | ||
| 166 | { | ||
| 167 | int i; | ||
| 168 | |||
| 169 | rf->msix_count = num_online_cpus() + IRDMA_NUM_AEQ_MSIX; | ||
| 170 | rf->msix_entries = kcalloc(rf->msix_count, sizeof(*rf->msix_entries), | ||
| 171 | GFP_KERNEL); | ||
| 172 | if (!rf->msix_entries) | ||
| 173 | return -ENOMEM; | ||
| 174 | |||
| 175 | for (i = 0; i < rf->msix_count; i++) | ||
| 176 | if (ice_alloc_rdma_qvector(cdev, &rf->msix_entries[i])) | ||
| 177 | break; | ||
| 178 | |||
| 179 | if (i < IRDMA_MIN_MSIX) { | ||
| 180 | while (--i >= 0) | ||
| 181 | ice_free_rdma_qvector(cdev, &rf->msix_entries[i]); | ||
| 182 | |||
| 183 | kfree(rf->msix_entries); | ||
| 184 | return -ENOMEM; | ||
| 185 | } | ||
| 186 | |||
| 187 | rf->msix_count = i; | ||
| 188 | |||
| 189 | return 0; | ||
| 190 | } | ||
| 191 | |||
| 192 | static void icrdma_deinit_interrupts(struct irdma_pci_f *rf, struct iidc_rdma_core_dev_info *cdev) | ||
| 193 | { | ||
| 194 | int i; | ||
| 195 | |||
| 196 | for (i = 0; i < rf->msix_count; i++) | ||
| 197 | ice_free_rdma_qvector(cdev, &rf->msix_entries[i]); | ||
| 198 | |||
| 199 | kfree(rf->msix_entries); | ||
| 200 | } | ||
| 201 | |||
| 202 | static void icrdma_fill_device_info(struct irdma_device *iwdev, | ||
| 203 | struct iidc_rdma_core_dev_info *cdev_info) | ||
| 204 | { | ||
| 205 | struct iidc_rdma_priv_dev_info *idc_priv = cdev_info->iidc_priv; | ||
| 206 | struct irdma_pci_f *rf = iwdev->rf; | ||
| 207 | |||
| 208 | rf->sc_dev.hw = &rf->hw; | ||
| 209 | rf->iwdev = iwdev; | ||
| 210 | rf->cdev = cdev_info; | ||
| 211 | rf->hw.hw_addr = idc_priv->hw_addr; | ||
| 212 | rf->pcidev = cdev_info->pdev; | ||
| 213 | rf->hw.device = &rf->pcidev->dev; | ||
| 214 | rf->pf_id = idc_priv->pf_id; | ||
| 215 | rf->rdma_ver = IRDMA_GEN_2; | ||
| 216 | rf->sc_dev.hw_attrs.uk_attrs.hw_rev = IRDMA_GEN_2; | ||
| 217 | rf->sc_dev.is_pf = true; | ||
| 218 | rf->sc_dev.privileged = true; | ||
| 219 | |||
| 220 | rf->gen_ops.register_qset = icrdma_lan_register_qset; | ||
| 221 | rf->gen_ops.unregister_qset = icrdma_lan_unregister_qset; | ||
| 222 | |||
| 223 | rf->default_vsi.vsi_idx = idc_priv->vport_id; | ||
| 224 | rf->protocol_used = | ||
| 225 | cdev_info->rdma_protocol == IIDC_RDMA_PROTOCOL_ROCEV2 ? | ||
| 226 | IRDMA_ROCE_PROTOCOL_ONLY : IRDMA_IWARP_PROTOCOL_ONLY; | ||
| 227 | rf->rsrc_profile = IRDMA_HMC_PROFILE_DEFAULT; | ||
| 228 | rf->rst_to = IRDMA_RST_TIMEOUT_HZ; | ||
| 229 | rf->gen_ops.request_reset = icrdma_request_reset; | ||
| 230 | rf->limits_sel = 7; | ||
| 231 | mutex_init(&rf->ah_tbl_lock); | ||
| 232 | |||
| 233 | iwdev->netdev = idc_priv->netdev; | ||
| 234 | iwdev->vsi_num = idc_priv->vport_id; | ||
| 235 | iwdev->init_state = INITIAL_STATE; | ||
| 236 | iwdev->roce_cwnd = IRDMA_ROCE_CWND_DEFAULT; | ||
| 237 | iwdev->roce_ackcreds = IRDMA_ROCE_ACKCREDS_DEFAULT; | ||
| 238 | iwdev->rcv_wnd = IRDMA_CM_DEFAULT_RCV_WND_SCALED; | ||
| 239 | iwdev->rcv_wscale = IRDMA_CM_DEFAULT_RCV_WND_SCALE; | ||
| 240 | if (iwdev->rf->protocol_used != IRDMA_IWARP_PROTOCOL_ONLY) | ||
| 241 | iwdev->roce_mode = true; | ||
| 242 | } | ||
| 243 | |||
| 244 | static int icrdma_probe(struct auxiliary_device *aux_dev, const struct auxiliary_device_id *id) | ||
| 245 | { | ||
| 246 | struct iidc_rdma_core_auxiliary_dev *iidc_adev; | ||
| 247 | struct iidc_rdma_core_dev_info *cdev_info; | ||
| 248 | struct iidc_rdma_priv_dev_info *idc_priv; | ||
| 249 | struct irdma_l2params l2params = {}; | ||
| 250 | struct irdma_device *iwdev; | ||
| 251 | struct irdma_pci_f *rf; | ||
| 252 | int err; | ||
| 253 | |||
| 254 | iidc_adev = container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev); | ||
| 255 | cdev_info = iidc_adev->cdev_info; | ||
| 256 | idc_priv = cdev_info->iidc_priv; | ||
| 257 | |||
| 258 | iwdev = ib_alloc_device(irdma_device, ibdev); | ||
| 259 | if (!iwdev) | ||
| 260 | return -ENOMEM; | ||
| 261 | iwdev->rf = kzalloc(sizeof(*rf), GFP_KERNEL); | ||
| 262 | if (!iwdev->rf) { | ||
| 263 | ib_dealloc_device(&iwdev->ibdev); | ||
| 264 | return -ENOMEM; | ||
| 265 | } | ||
| 266 | |||
| 267 | icrdma_fill_device_info(iwdev, cdev_info); | ||
| 268 | rf = iwdev->rf; | ||
| 269 | |||
| 270 | err = icrdma_init_interrupts(rf, cdev_info); | ||
| 271 | if (err) | ||
| 272 | goto err_init_interrupts; | ||
| 273 | |||
| 274 | err = irdma_ctrl_init_hw(rf); | ||
| 275 | if (err) | ||
| 276 | goto err_ctrl_init; | ||
| 277 | |||
| 278 | l2params.mtu = iwdev->netdev->mtu; | ||
| 279 | icrdma_fill_qos_info(&l2params, &idc_priv->qos_info); | ||
| 280 | if (iwdev->rf->protocol_used != IRDMA_IWARP_PROTOCOL_ONLY) | ||
| 281 | iwdev->dcb_vlan_mode = l2params.num_tc > 1 && !l2params.dscp_mode; | ||
| 282 | |||
| 283 | err = irdma_rt_init_hw(iwdev, &l2params); | ||
| 284 | if (err) | ||
| 285 | goto err_rt_init; | ||
| 286 | |||
| 287 | err = irdma_ib_register_device(iwdev); | ||
| 288 | if (err) | ||
| 289 | goto err_ibreg; | ||
| 290 | |||
| 291 | ice_rdma_update_vsi_filter(cdev_info, iwdev->vsi_num, true); | ||
| 292 | |||
| 293 | ibdev_dbg(&iwdev->ibdev, "INIT: Gen2 PF[%d] device probe success\n", PCI_FUNC(rf->pcidev->devfn)); | ||
| 294 | auxiliary_set_drvdata(aux_dev, iwdev); | ||
| 295 | |||
| 296 | return 0; | ||
| 297 | |||
| 298 | err_ibreg: | ||
| 299 | irdma_rt_deinit_hw(iwdev); | ||
| 300 | err_rt_init: | ||
| 301 | irdma_ctrl_deinit_hw(rf); | ||
| 302 | err_ctrl_init: | ||
| 303 | icrdma_deinit_interrupts(rf, cdev_info); | ||
| 304 | err_init_interrupts: | ||
| 305 | kfree(iwdev->rf); | ||
| 306 | ib_dealloc_device(&iwdev->ibdev); | ||
| 307 | |||
| 308 | return err; | ||
| 309 | } | ||
| 310 | |||
| 311 | static void icrdma_remove(struct auxiliary_device *aux_dev) | ||
| 312 | { | ||
| 313 | struct iidc_rdma_core_auxiliary_dev *idc_adev = | ||
| 314 | container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev); | ||
| 315 | struct iidc_rdma_core_dev_info *cdev_info = idc_adev->cdev_info; | ||
| 316 | struct irdma_device *iwdev = auxiliary_get_drvdata(aux_dev); | ||
| 317 | u8 rdma_ver = iwdev->rf->rdma_ver; | ||
| 318 | |||
| 319 | ice_rdma_update_vsi_filter(cdev_info, iwdev->vsi_num, false); | ||
| 320 | irdma_ib_unregister_device(iwdev); | ||
| 321 | icrdma_deinit_interrupts(iwdev->rf, cdev_info); | ||
| 322 | |||
| 323 | pr_debug("INIT: Gen[%d] func[%d] device remove success\n", | ||
| 324 | rdma_ver, PCI_FUNC(cdev_info->pdev->devfn)); | ||
| 325 | } | ||
| 326 | |||
| 327 | static const struct auxiliary_device_id icrdma_auxiliary_id_table[] = { | ||
| 328 | {.name = "ice.iwarp", }, | ||
| 329 | {.name = "ice.roce", }, | ||
| 330 | {}, | ||
| 331 | }; | ||
| 332 | |||
| 333 | MODULE_DEVICE_TABLE(auxiliary, icrdma_auxiliary_id_table); | ||
| 334 | |||
| 335 | struct iidc_rdma_core_auxiliary_drv icrdma_core_auxiliary_drv = { | ||
| 336 | .adrv = { | ||
| 337 | .name = "gen_2", | ||
| 338 | .id_table = icrdma_auxiliary_id_table, | ||
| 339 | .probe = icrdma_probe, | ||
| 340 | .remove = icrdma_remove, | ||
| 341 | }, | ||
| 342 | .event_handler = icrdma_iidc_event_handler, | ||
| 343 | }; | ||
| @@ -0,0 +1,170 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB | ||
| 2 | /* Copyright (c) 2018 - 2024 Intel Corporation */ | ||
| 3 | #include "osdep.h" | ||
| 4 | #include "type.h" | ||
| 5 | #include "protos.h" | ||
| 6 | #include "ig3rdma_hw.h" | ||
| 7 | |||
| 8 | /** | ||
| 9 | * ig3rdma_ena_irq - Enable interrupt | ||
| 10 | * @dev: pointer to the device structure | ||
| 11 | * @idx: vector index | ||
| 12 | */ | ||
| 13 | static void ig3rdma_ena_irq(struct irdma_sc_dev *dev, u32 idx) | ||
| 14 | { | ||
| 15 | u32 val; | ||
| 16 | u32 int_stride = 1; /* one u32 per register */ | ||
| 17 | |||
| 18 | if (dev->is_pf) | ||
| 19 | int_stride = 0x400; | ||
| 20 | else | ||
| 21 | idx--; /* VFs use DYN_CTL_N */ | ||
| 22 | |||
| 23 | val = FIELD_PREP(IRDMA_GLINT_DYN_CTL_INTENA, 1) | | ||
| 24 | FIELD_PREP(IRDMA_GLINT_DYN_CTL_CLEARPBA, 1); | ||
| 25 | |||
| 26 | writel(val, dev->hw_regs[IRDMA_GLINT_DYN_CTL] + (idx * int_stride)); | ||
| 27 | } | ||
| 28 | |||
| 29 | /** | ||
| 30 | * ig3rdma_disable_irq - Disable interrupt | ||
| 31 | * @dev: pointer to the device structure | ||
| 32 | * @idx: vector index | ||
| 33 | */ | ||
| 34 | static void ig3rdma_disable_irq(struct irdma_sc_dev *dev, u32 idx) | ||
| 35 | { | ||
| 36 | u32 int_stride = 1; /* one u32 per register */ | ||
| 37 | |||
| 38 | if (dev->is_pf) | ||
| 39 | int_stride = 0x400; | ||
| 40 | else | ||
| 41 | idx--; /* VFs use DYN_CTL_N */ | ||
| 42 | |||
| 43 | writel(0, dev->hw_regs[IRDMA_GLINT_DYN_CTL] + (idx * int_stride)); | ||
| 44 | } | ||
| 45 | |||
| 46 | static const struct irdma_irq_ops ig3rdma_irq_ops = { | ||
| 47 | .irdma_dis_irq = ig3rdma_disable_irq, | ||
| 48 | .irdma_en_irq = ig3rdma_ena_irq, | ||
| 49 | }; | ||
| 50 | |||
| 51 | static const struct irdma_hw_stat_map ig3rdma_hw_stat_map[] = { | ||
| 52 | [IRDMA_HW_STAT_INDEX_RXVLANERR] = { 0, 0, 0 }, | ||
| 53 | [IRDMA_HW_STAT_INDEX_IP4RXOCTS] = { 8, 0, 0 }, | ||
| 54 | [IRDMA_HW_STAT_INDEX_IP4RXPKTS] = { 16, 0, 0 }, | ||
| 55 | [IRDMA_HW_STAT_INDEX_IP4RXDISCARD] = { 24, 0, 0 }, | ||
| 56 | [IRDMA_HW_STAT_INDEX_IP4RXTRUNC] = { 32, 0, 0 }, | ||
| 57 | [IRDMA_HW_STAT_INDEX_IP4RXFRAGS] = { 40, 0, 0 }, | ||
| 58 | [IRDMA_HW_STAT_INDEX_IP4RXMCOCTS] = { 48, 0, 0 }, | ||
| 59 | [IRDMA_HW_STAT_INDEX_IP4RXMCPKTS] = { 56, 0, 0 }, | ||
| 60 | [IRDMA_HW_STAT_INDEX_IP6RXOCTS] = { 64, 0, 0 }, | ||
| 61 | [IRDMA_HW_STAT_INDEX_IP6RXPKTS] = { 72, 0, 0 }, | ||
| 62 | [IRDMA_HW_STAT_INDEX_IP6RXDISCARD] = { 80, 0, 0 }, | ||
| 63 | [IRDMA_HW_STAT_INDEX_IP6RXTRUNC] = { 88, 0, 0 }, | ||
| 64 | [IRDMA_HW_STAT_INDEX_IP6RXFRAGS] = { 96, 0, 0 }, | ||
| 65 | [IRDMA_HW_STAT_INDEX_IP6RXMCOCTS] = { 104, 0, 0 }, | ||
| 66 | [IRDMA_HW_STAT_INDEX_IP6RXMCPKTS] = { 112, 0, 0 }, | ||
| 67 | [IRDMA_HW_STAT_INDEX_IP4TXOCTS] = { 120, 0, 0 }, | ||
| 68 | [IRDMA_HW_STAT_INDEX_IP4TXPKTS] = { 128, 0, 0 }, | ||
| 69 | [IRDMA_HW_STAT_INDEX_IP4TXFRAGS] = { 136, 0, 0 }, | ||
| 70 | [IRDMA_HW_STAT_INDEX_IP4TXMCOCTS] = { 144, 0, 0 }, | ||
| 71 | [IRDMA_HW_STAT_INDEX_IP4TXMCPKTS] = { 152, 0, 0 }, | ||
| 72 | [IRDMA_HW_STAT_INDEX_IP6TXOCTS] = { 160, 0, 0 }, | ||
| 73 | [IRDMA_HW_STAT_INDEX_IP6TXPKTS] = { 168, 0, 0 }, | ||
| 74 | [IRDMA_HW_STAT_INDEX_IP6TXFRAGS] = { 176, 0, 0 }, | ||
| 75 | [IRDMA_HW_STAT_INDEX_IP6TXMCOCTS] = { 184, 0, 0 }, | ||
| 76 | [IRDMA_HW_STAT_INDEX_IP6TXMCPKTS] = { 192, 0, 0 }, | ||
| 77 | [IRDMA_HW_STAT_INDEX_IP4TXNOROUTE] = { 200, 0, 0 }, | ||
| 78 | [IRDMA_HW_STAT_INDEX_IP6TXNOROUTE] = { 208, 0, 0 }, | ||
| 79 | [IRDMA_HW_STAT_INDEX_TCPRTXSEG] = { 216, 0, 0 }, | ||
| 80 | [IRDMA_HW_STAT_INDEX_TCPRXOPTERR] = { 224, 0, 0 }, | ||
| 81 | [IRDMA_HW_STAT_INDEX_TCPRXPROTOERR] = { 232, 0, 0 }, | ||
| 82 | [IRDMA_HW_STAT_INDEX_TCPTXSEG] = { 240, 0, 0 }, | ||
| 83 | [IRDMA_HW_STAT_INDEX_TCPRXSEGS] = { 248, 0, 0 }, | ||
| 84 | [IRDMA_HW_STAT_INDEX_UDPRXPKTS] = { 256, 0, 0 }, | ||
| 85 | [IRDMA_HW_STAT_INDEX_UDPTXPKTS] = { 264, 0, 0 }, | ||
| 86 | [IRDMA_HW_STAT_INDEX_RDMARXWRS] = { 272, 0, 0 }, | ||
| 87 | [IRDMA_HW_STAT_INDEX_RDMARXRDS] = { 280, 0, 0 }, | ||
| 88 | [IRDMA_HW_STAT_INDEX_RDMARXSNDS] = { 288, 0, 0 }, | ||
| 89 | [IRDMA_HW_STAT_INDEX_RDMATXWRS] = { 296, 0, 0 }, | ||
| 90 | [IRDMA_HW_STAT_INDEX_RDMATXRDS] = { 304, 0, 0 }, | ||
| 91 | [IRDMA_HW_STAT_INDEX_RDMATXSNDS] = { 312, 0, 0 }, | ||
| 92 | [IRDMA_HW_STAT_INDEX_RDMAVBND] = { 320, 0, 0 }, | ||
| 93 | [IRDMA_HW_STAT_INDEX_RDMAVINV] = { 328, 0, 0 }, | ||
| 94 | [IRDMA_HW_STAT_INDEX_RXNPECNMARKEDPKTS] = { 336, 0, 0 }, | ||
| 95 | [IRDMA_HW_STAT_INDEX_RXRPCNPHANDLED] = { 344, 0, 0 }, | ||
| 96 | [IRDMA_HW_STAT_INDEX_RXRPCNPIGNORED] = { 352, 0, 0 }, | ||
| 97 | [IRDMA_HW_STAT_INDEX_TXNPCNPSENT] = { 360, 0, 0 }, | ||
| 98 | [IRDMA_HW_STAT_INDEX_RNR_SENT] = { 368, 0, 0 }, | ||
| 99 | [IRDMA_HW_STAT_INDEX_RNR_RCVD] = { 376, 0, 0 }, | ||
| 100 | [IRDMA_HW_STAT_INDEX_RDMAORDLMTCNT] = { 384, 0, 0 }, | ||
| 101 | [IRDMA_HW_STAT_INDEX_RDMAIRDLMTCNT] = { 392, 0, 0 }, | ||
| 102 | [IRDMA_HW_STAT_INDEX_RDMARXATS] = { 408, 0, 0 }, | ||
| 103 | [IRDMA_HW_STAT_INDEX_RDMATXATS] = { 416, 0, 0 }, | ||
| 104 | [IRDMA_HW_STAT_INDEX_NAKSEQERR] = { 424, 0, 0 }, | ||
| 105 | [IRDMA_HW_STAT_INDEX_NAKSEQERR_IMPLIED] = { 432, 0, 0 }, | ||
| 106 | [IRDMA_HW_STAT_INDEX_RTO] = { 440, 0, 0 }, | ||
| 107 | [IRDMA_HW_STAT_INDEX_RXOOOPKTS] = { 448, 0, 0 }, | ||
| 108 | [IRDMA_HW_STAT_INDEX_ICRCERR] = { 456, 0, 0 }, | ||
| 109 | }; | ||
| 110 | |||
| 111 | void ig3rdma_init_hw(struct irdma_sc_dev *dev) | ||
| 112 | { | ||
| 113 | dev->irq_ops = &ig3rdma_irq_ops; | ||
| 114 | dev->hw_stats_map = ig3rdma_hw_stat_map; | ||
| 115 | |||
| 116 | dev->hw_attrs.uk_attrs.hw_rev = IRDMA_GEN_3; | ||
| 117 | dev->hw_attrs.uk_attrs.max_hw_wq_frags = IG3RDMA_MAX_WQ_FRAGMENT_COUNT; | ||
| 118 | dev->hw_attrs.uk_attrs.max_hw_read_sges = IG3RDMA_MAX_SGE_RD; | ||
| 119 | dev->hw_attrs.uk_attrs.max_hw_sq_chunk = IRDMA_MAX_QUANTA_PER_WR; | ||
| 120 | dev->hw_attrs.first_hw_vf_fpm_id = 0; | ||
| 121 | dev->hw_attrs.max_hw_vf_fpm_id = IG3_MAX_APFS + IG3_MAX_AVFS; | ||
| 122 | dev->hw_attrs.uk_attrs.feature_flags |= IRDMA_FEATURE_64_BYTE_CQE; | ||
| 123 | dev->hw_attrs.uk_attrs.feature_flags |= IRDMA_FEATURE_CQE_TIMESTAMPING; | ||
| 124 | |||
| 125 | dev->hw_attrs.uk_attrs.feature_flags |= IRDMA_FEATURE_SRQ; | ||
| 126 | dev->hw_attrs.uk_attrs.feature_flags |= IRDMA_FEATURE_RTS_AE | | ||
| 127 | IRDMA_FEATURE_CQ_RESIZE; | ||
| 128 | dev->hw_attrs.page_size_cap = SZ_4K | SZ_2M | SZ_1G; | ||
| 129 | dev->hw_attrs.max_hw_ird = IG3RDMA_MAX_IRD_SIZE; | ||
| 130 | dev->hw_attrs.max_hw_ord = IG3RDMA_MAX_ORD_SIZE; | ||
| 131 | dev->hw_attrs.max_stat_inst = IG3RDMA_MAX_STATS_COUNT; | ||
| 132 | dev->hw_attrs.max_stat_idx = IRDMA_HW_STAT_INDEX_MAX_GEN_3; | ||
| 133 | dev->hw_attrs.uk_attrs.min_hw_wq_size = IG3RDMA_MIN_WQ_SIZE; | ||
| 134 | dev->hw_attrs.uk_attrs.max_hw_srq_quanta = IRDMA_SRQ_MAX_QUANTA; | ||
| 135 | dev->hw_attrs.uk_attrs.max_hw_inline = IG3RDMA_MAX_INLINE_DATA_SIZE; | ||
| 136 | dev->hw_attrs.max_hw_device_pages = | ||
| 137 | dev->is_pf ? IG3RDMA_MAX_PF_PUSH_PAGE_COUNT : IG3RDMA_MAX_VF_PUSH_PAGE_COUNT; | ||
| 138 | } | ||
| 139 | |||
| 140 | static void __iomem *__ig3rdma_get_reg_addr(struct irdma_mmio_region *region, u64 reg_offset) | ||
| 141 | { | ||
| 142 | if (reg_offset >= region->offset && | ||
| 143 | reg_offset < (region->offset + region->len)) { | ||
| 144 | reg_offset -= region->offset; | ||
| 145 | |||
| 146 | return region->addr + reg_offset; | ||
| 147 | } | ||
| 148 | |||
| 149 | return NULL; | ||
| 150 | } | ||
| 151 | |||
| 152 | void __iomem *ig3rdma_get_reg_addr(struct irdma_hw *hw, u64 reg_offset) | ||
| 153 | { | ||
| 154 | u8 __iomem *reg_addr; | ||
| 155 | int i; | ||
| 156 | |||
| 157 | reg_addr = __ig3rdma_get_reg_addr(&hw->rdma_reg, reg_offset); | ||
| 158 | if (reg_addr) | ||
| 159 | return reg_addr; | ||
| 160 | |||
| 161 | for (i = 0; i < hw->num_io_regions; i++) { | ||
| 162 | reg_addr = __ig3rdma_get_reg_addr(&hw->io_regs[i], reg_offset); | ||
| 163 | if (reg_addr) | ||
| 164 | return reg_addr; | ||
| 165 | } | ||
| 166 | |||
| 167 | WARN_ON_ONCE(1); | ||
| 168 | |||
| 169 | return NULL; | ||
| 170 | } | ||
| @@ -0,0 +1,32 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB */ | ||
| 2 | /* Copyright (c) 2021 - 2024 Intel Corporation */ | ||
| 3 | #ifndef IG3RDMA_HW_H | ||
| 4 | #define IG3RDMA_HW_H | ||
| 5 | |||
| 6 | #define IG3_MAX_APFS 1 | ||
| 7 | #define IG3_MAX_AVFS 0 | ||
| 8 | |||
| 9 | #define IG3_PF_RDMA_REGION_OFFSET 0xBC00000 | ||
| 10 | #define IG3_PF_RDMA_REGION_LEN 0x401000 | ||
| 11 | #define IG3_VF_RDMA_REGION_OFFSET 0x8C00 | ||
| 12 | #define IG3_VF_RDMA_REGION_LEN 0x8400 | ||
| 13 | |||
| 14 | enum ig3rdma_device_caps_const { | ||
| 15 | IG3RDMA_MAX_WQ_FRAGMENT_COUNT = 14, | ||
| 16 | IG3RDMA_MAX_SGE_RD = 14, | ||
| 17 | |||
| 18 | IG3RDMA_MAX_STATS_COUNT = 128, | ||
| 19 | |||
| 20 | IG3RDMA_MAX_IRD_SIZE = 64, | ||
| 21 | IG3RDMA_MAX_ORD_SIZE = 64, | ||
| 22 | IG3RDMA_MIN_WQ_SIZE = 16 /* WQEs */, | ||
| 23 | IG3RDMA_MAX_INLINE_DATA_SIZE = 216, | ||
| 24 | IG3RDMA_MAX_PF_PUSH_PAGE_COUNT = 8192, | ||
| 25 | IG3RDMA_MAX_VF_PUSH_PAGE_COUNT = 16, | ||
| 26 | }; | ||
| 27 | |||
| 28 | void __iomem *ig3rdma_get_reg_addr(struct irdma_hw *hw, u64 reg_offset); | ||
| 29 | int ig3rdma_vchnl_send_sync(struct irdma_sc_dev *dev, u8 *msg, u16 len, | ||
| 30 | u8 *recv_msg, u16 *recv_len); | ||
| 31 | |||
| 32 | #endif /* IG3RDMA_HW_H*/ | ||
| @@ -0,0 +1,232 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB | ||
| 2 | /* Copyright (c) 2023 - 2024 Intel Corporation */ | ||
| 3 | |||
| 4 | #include "main.h" | ||
| 5 | #include <linux/net/intel/iidc_rdma_idpf.h> | ||
| 6 | #include "ig3rdma_hw.h" | ||
| 7 | |||
| 8 | static void ig3rdma_idc_core_event_handler(struct iidc_rdma_core_dev_info *cdev_info, | ||
| 9 | struct iidc_rdma_event *event) | ||
| 10 | { | ||
| 11 | struct irdma_pci_f *rf = auxiliary_get_drvdata(cdev_info->adev); | ||
| 12 | |||
| 13 | if (*event->type & BIT(IIDC_RDMA_EVENT_WARN_RESET)) { | ||
| 14 | rf->reset = true; | ||
| 15 | rf->sc_dev.vchnl_up = false; | ||
| 16 | } | ||
| 17 | } | ||
| 18 | |||
| 19 | int ig3rdma_vchnl_send_sync(struct irdma_sc_dev *dev, u8 *msg, u16 len, | ||
| 20 | u8 *recv_msg, u16 *recv_len) | ||
| 21 | { | ||
| 22 | struct iidc_rdma_core_dev_info *cdev_info = dev_to_rf(dev)->cdev; | ||
| 23 | int ret; | ||
| 24 | |||
| 25 | ret = idpf_idc_rdma_vc_send_sync(cdev_info, msg, len, recv_msg, | ||
| 26 | recv_len); | ||
| 27 | if (ret == -ETIMEDOUT) { | ||
| 28 | ibdev_err(&(dev_to_rf(dev)->iwdev->ibdev), | ||
| 29 | "Virtual channel Req <-> Resp completion timeout\n"); | ||
| 30 | dev->vchnl_up = false; | ||
| 31 | } | ||
| 32 | |||
| 33 | return ret; | ||
| 34 | } | ||
| 35 | |||
| 36 | static int ig3rdma_vchnl_init(struct irdma_pci_f *rf, | ||
| 37 | struct iidc_rdma_core_dev_info *cdev_info, | ||
| 38 | u8 *rdma_ver) | ||
| 39 | { | ||
| 40 | struct iidc_rdma_priv_dev_info *idc_priv = cdev_info->iidc_priv; | ||
| 41 | struct irdma_vchnl_init_info virt_info; | ||
| 42 | u8 gen = rf->rdma_ver; | ||
| 43 | int ret; | ||
| 44 | |||
| 45 | rf->vchnl_wq = alloc_ordered_workqueue("irdma-virtchnl-wq", 0); | ||
| 46 | if (!rf->vchnl_wq) | ||
| 47 | return -ENOMEM; | ||
| 48 | |||
| 49 | mutex_init(&rf->sc_dev.vchnl_mutex); | ||
| 50 | |||
| 51 | virt_info.is_pf = !idc_priv->ftype; | ||
| 52 | virt_info.hw_rev = gen; | ||
| 53 | virt_info.privileged = gen == IRDMA_GEN_2; | ||
| 54 | virt_info.vchnl_wq = rf->vchnl_wq; | ||
| 55 | ret = irdma_sc_vchnl_init(&rf->sc_dev, &virt_info); | ||
| 56 | if (ret) { | ||
| 57 | destroy_workqueue(rf->vchnl_wq); | ||
| 58 | return ret; | ||
| 59 | } | ||
| 60 | |||
| 61 | *rdma_ver = rf->sc_dev.hw_attrs.uk_attrs.hw_rev; | ||
| 62 | |||
| 63 | return 0; | ||
| 64 | } | ||
| 65 | |||
| 66 | /** | ||
| 67 | * ig3rdma_request_reset - Request a reset | ||
| 68 | * @rf: RDMA PCI function | ||
| 69 | */ | ||
| 70 | static void ig3rdma_request_reset(struct irdma_pci_f *rf) | ||
| 71 | { | ||
| 72 | ibdev_warn(&rf->iwdev->ibdev, "Requesting a reset\n"); | ||
| 73 | idpf_idc_request_reset(rf->cdev, IIDC_FUNC_RESET); | ||
| 74 | } | ||
| 75 | |||
| 76 | static int ig3rdma_cfg_regions(struct irdma_hw *hw, | ||
| 77 | struct iidc_rdma_core_dev_info *cdev_info) | ||
| 78 | { | ||
| 79 | struct iidc_rdma_priv_dev_info *idc_priv = cdev_info->iidc_priv; | ||
| 80 | struct pci_dev *pdev = cdev_info->pdev; | ||
| 81 | int i; | ||
| 82 | |||
| 83 | switch (idc_priv->ftype) { | ||
| 84 | case IIDC_FUNCTION_TYPE_PF: | ||
| 85 | hw->rdma_reg.len = IG3_PF_RDMA_REGION_LEN; | ||
| 86 | hw->rdma_reg.offset = IG3_PF_RDMA_REGION_OFFSET; | ||
| 87 | break; | ||
| 88 | case IIDC_FUNCTION_TYPE_VF: | ||
| 89 | hw->rdma_reg.len = IG3_VF_RDMA_REGION_LEN; | ||
| 90 | hw->rdma_reg.offset = IG3_VF_RDMA_REGION_OFFSET; | ||
| 91 | break; | ||
| 92 | default: | ||
| 93 | return -ENODEV; | ||
| 94 | } | ||
| 95 | |||
| 96 | hw->rdma_reg.addr = ioremap(pci_resource_start(pdev, 0) + hw->rdma_reg.offset, | ||
| 97 | hw->rdma_reg.len); | ||
| 98 | |||
| 99 | if (!hw->rdma_reg.addr) | ||
| 100 | return -ENOMEM; | ||
| 101 | |||
| 102 | hw->num_io_regions = le16_to_cpu(idc_priv->num_memory_regions); | ||
| 103 | hw->io_regs = kcalloc(hw->num_io_regions, | ||
| 104 | sizeof(struct irdma_mmio_region), GFP_KERNEL); | ||
| 105 | |||
| 106 | if (!hw->io_regs) { | ||
| 107 | iounmap(hw->rdma_reg.addr); | ||
| 108 | return -ENOMEM; | ||
| 109 | } | ||
| 110 | |||
| 111 | for (i = 0; i < hw->num_io_regions; i++) { | ||
| 112 | hw->io_regs[i].addr = | ||
| 113 | idc_priv->mapped_mem_regions[i].region_addr; | ||
| 114 | hw->io_regs[i].len = | ||
| 115 | le64_to_cpu(idc_priv->mapped_mem_regions[i].size); | ||
| 116 | hw->io_regs[i].offset = | ||
| 117 | le64_to_cpu(idc_priv->mapped_mem_regions[i].start_offset); | ||
| 118 | } | ||
| 119 | |||
| 120 | return 0; | ||
| 121 | } | ||
| 122 | |||
| 123 | static void ig3rdma_decfg_rf(struct irdma_pci_f *rf) | ||
| 124 | { | ||
| 125 | struct irdma_hw *hw = &rf->hw; | ||
| 126 | |||
| 127 | destroy_workqueue(rf->vchnl_wq); | ||
| 128 | kfree(hw->io_regs); | ||
| 129 | iounmap(hw->rdma_reg.addr); | ||
| 130 | } | ||
| 131 | |||
| 132 | static int ig3rdma_cfg_rf(struct irdma_pci_f *rf, | ||
| 133 | struct iidc_rdma_core_dev_info *cdev_info) | ||
| 134 | { | ||
| 135 | struct iidc_rdma_priv_dev_info *idc_priv = cdev_info->iidc_priv; | ||
| 136 | int err; | ||
| 137 | |||
| 138 | rf->sc_dev.hw = &rf->hw; | ||
| 139 | rf->cdev = cdev_info; | ||
| 140 | rf->pcidev = cdev_info->pdev; | ||
| 141 | rf->hw.device = &rf->pcidev->dev; | ||
| 142 | rf->msix_count = idc_priv->msix_count; | ||
| 143 | rf->msix_entries = idc_priv->msix_entries; | ||
| 144 | |||
| 145 | err = ig3rdma_vchnl_init(rf, cdev_info, &rf->rdma_ver); | ||
| 146 | if (err) | ||
| 147 | return err; | ||
| 148 | |||
| 149 | err = ig3rdma_cfg_regions(&rf->hw, cdev_info); | ||
| 150 | if (err) { | ||
| 151 | destroy_workqueue(rf->vchnl_wq); | ||
| 152 | return err; | ||
| 153 | } | ||
| 154 | |||
| 155 | rf->protocol_used = IRDMA_ROCE_PROTOCOL_ONLY; | ||
| 156 | rf->rsrc_profile = IRDMA_HMC_PROFILE_DEFAULT; | ||
| 157 | rf->rst_to = IRDMA_RST_TIMEOUT_HZ; | ||
| 158 | rf->gen_ops.request_reset = ig3rdma_request_reset; | ||
| 159 | rf->limits_sel = 7; | ||
| 160 | mutex_init(&rf->ah_tbl_lock); | ||
| 161 | |||
| 162 | return 0; | ||
| 163 | } | ||
| 164 | |||
| 165 | static int ig3rdma_core_probe(struct auxiliary_device *aux_dev, | ||
| 166 | const struct auxiliary_device_id *id) | ||
| 167 | { | ||
| 168 | struct iidc_rdma_core_auxiliary_dev *idc_adev = | ||
| 169 | container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev); | ||
| 170 | struct iidc_rdma_core_dev_info *cdev_info = idc_adev->cdev_info; | ||
| 171 | struct irdma_pci_f *rf; | ||
| 172 | int err; | ||
| 173 | |||
| 174 | rf = kzalloc(sizeof(*rf), GFP_KERNEL); | ||
| 175 | if (!rf) | ||
| 176 | return -ENOMEM; | ||
| 177 | |||
| 178 | err = ig3rdma_cfg_rf(rf, cdev_info); | ||
| 179 | if (err) | ||
| 180 | goto err_cfg_rf; | ||
| 181 | |||
| 182 | err = irdma_ctrl_init_hw(rf); | ||
| 183 | if (err) | ||
| 184 | goto err_ctrl_init; | ||
| 185 | |||
| 186 | auxiliary_set_drvdata(aux_dev, rf); | ||
| 187 | |||
| 188 | err = idpf_idc_vport_dev_ctrl(cdev_info, true); | ||
| 189 | if (err) | ||
| 190 | goto err_vport_ctrl; | ||
| 191 | |||
| 192 | return 0; | ||
| 193 | |||
| 194 | err_vport_ctrl: | ||
| 195 | irdma_ctrl_deinit_hw(rf); | ||
| 196 | err_ctrl_init: | ||
| 197 | ig3rdma_decfg_rf(rf); | ||
| 198 | err_cfg_rf: | ||
| 199 | kfree(rf); | ||
| 200 | |||
| 201 | return err; | ||
| 202 | } | ||
| 203 | |||
| 204 | static void ig3rdma_core_remove(struct auxiliary_device *aux_dev) | ||
| 205 | { | ||
| 206 | struct iidc_rdma_core_auxiliary_dev *idc_adev = | ||
| 207 | container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev); | ||
| 208 | struct iidc_rdma_core_dev_info *cdev_info = idc_adev->cdev_info; | ||
| 209 | struct irdma_pci_f *rf = auxiliary_get_drvdata(aux_dev); | ||
| 210 | |||
| 211 | idpf_idc_vport_dev_ctrl(cdev_info, false); | ||
| 212 | irdma_ctrl_deinit_hw(rf); | ||
| 213 | ig3rdma_decfg_rf(rf); | ||
| 214 | kfree(rf); | ||
| 215 | } | ||
| 216 | |||
| 217 | static const struct auxiliary_device_id ig3rdma_core_auxiliary_id_table[] = { | ||
| 218 | {.name = "idpf.8086.rdma.core", }, | ||
| 219 | {}, | ||
| 220 | }; | ||
| 221 | |||
| 222 | MODULE_DEVICE_TABLE(auxiliary, ig3rdma_core_auxiliary_id_table); | ||
| 223 | |||
| 224 | struct iidc_rdma_core_auxiliary_drv ig3rdma_core_auxiliary_drv = { | ||
| 225 | .adrv = { | ||
| 226 | .name = "core", | ||
| 227 | .id_table = ig3rdma_core_auxiliary_id_table, | ||
| 228 | .probe = ig3rdma_core_probe, | ||
| 229 | .remove = ig3rdma_core_remove, | ||
| 230 | }, | ||
| 231 | .event_handler = ig3rdma_idc_core_event_handler, | ||
| 232 | }; | ||
| @@ -32,7 +32,16 @@ · 31 unmodified lines | |||
| 32 | #define IRDMA_PFHMC_SDDATALOW_PMSDDATALOW GENMASK(31, 12) | 32 | #define IRDMA_PFHMC_SDDATALOW_PMSDDATALOW GENMASK(31, 12) |
| 33 | #define IRDMA_PFHMC_SDCMD_PMSDWR BIT(31) | 33 | #define IRDMA_PFHMC_SDCMD_PMSDWR BIT(31) |
| 34 | 34 | ||
| 35 | #define IRDMA_INVALID_CQ_IDX 0xffffffff | 35 | #define IRDMA_INVALID_CQ_IDX 0xffffffff |
| 36 | #define IRDMA_Q_INVALID_IDX 0xffff | ||
| 37 | |||
| 38 | enum irdma_dyn_idx_t { | ||
| 39 | IRDMA_IDX_ITR0 = 0, | ||
| 40 | IRDMA_IDX_ITR1 = 1, | ||
| 41 | IRDMA_IDX_ITR2 = 2, | ||
| 42 | IRDMA_IDX_NOITR = 3, | ||
| 43 | }; | ||
| 44 | |||
| 36 | enum irdma_registers { | 45 | enum irdma_registers { |
| 37 | IRDMA_CQPTAIL, | 46 | IRDMA_CQPTAIL, |
| 38 | IRDMA_CQPDB, | 47 | IRDMA_CQPDB, |
| @@ -67,6 +76,7 @@ enum irdma_shifts { · 28 unmodified lines | |||
| 67 | IRDMA_CQPSQ_CQ_CEQID_S, | 76 | IRDMA_CQPSQ_CQ_CEQID_S, |
| 68 | IRDMA_CQPSQ_CQ_CQID_S, | 77 | IRDMA_CQPSQ_CQ_CQID_S, |
| 69 | IRDMA_COMMIT_FPM_CQCNT_S, | 78 | IRDMA_COMMIT_FPM_CQCNT_S, |
| 79 | IRDMA_CQPSQ_UPESD_HMCFNID_S, | ||
| 70 | IRDMA_MAX_SHIFTS, | 80 | IRDMA_MAX_SHIFTS, |
| 71 | }; | 81 | }; |
| 72 | 82 | ||
| @@ -77,6 +87,7 @@ enum irdma_masks { · 4 unmodified lines | |||
| 77 | IRDMA_CQPSQ_CQ_CEQID_M, | 87 | IRDMA_CQPSQ_CQ_CEQID_M, |
| 78 | IRDMA_CQPSQ_CQ_CQID_M, | 88 | IRDMA_CQPSQ_CQ_CQID_M, |
| 79 | IRDMA_COMMIT_FPM_CQCNT_M, | 89 | IRDMA_COMMIT_FPM_CQCNT_M, |
| 90 | IRDMA_CQPSQ_UPESD_HMCFNID_M, | ||
| 80 | IRDMA_MAX_MASKS, /* Must be last entry */ | 91 | IRDMA_MAX_MASKS, /* Must be last entry */ |
| 81 | }; | 92 | }; |
| 82 | 93 | ||
| @@ -92,7 +103,7 @@ struct irdma_mcast_grp_ctx_entry_info { · 9 unmodified lines | |||
| 92 | struct irdma_mcast_grp_info { | 103 | struct irdma_mcast_grp_info { |
| 93 | u8 dest_mac_addr[ETH_ALEN]; | 104 | u8 dest_mac_addr[ETH_ALEN]; |
| 94 | u16 vlan_id; | 105 | u16 vlan_id; |
| 95 | u8 hmc_fcn_id; | 106 | u16 hmc_fcn_id; |
| 96 | bool ipv4_valid:1; | 107 | bool ipv4_valid:1; |
| 97 | bool vlan_valid:1; | 108 | bool vlan_valid:1; |
| 98 | u16 mg_id; | 109 | u16 mg_id; |
| @@ -107,6 +118,9 @@ enum irdma_vers { · 8 unmodified lines | |||
| 107 | IRDMA_GEN_RSVD, | 118 | IRDMA_GEN_RSVD, |
| 108 | IRDMA_GEN_1, | 119 | IRDMA_GEN_1, |
| 109 | IRDMA_GEN_2, | 120 | IRDMA_GEN_2, |
| 121 | IRDMA_GEN_3, | ||
| 122 | IRDMA_GEN_NEXT, | ||
| 123 | IRDMA_GEN_MAX = IRDMA_GEN_NEXT-1 | ||
| 110 | }; | 124 | }; |
| 111 | 125 | ||
| 112 | struct irdma_uk_attrs { | 126 | struct irdma_uk_attrs { |
| @@ -118,6 +132,7 @@ struct irdma_uk_attrs { · 5 unmodified lines | |||
| 118 | u32 max_hw_wq_quanta; | 132 | u32 max_hw_wq_quanta; |
| 119 | u32 min_hw_cq_size; | 133 | u32 min_hw_cq_size; |
| 120 | u32 max_hw_cq_size; | 134 | u32 max_hw_cq_size; |
| 135 | u32 max_hw_srq_quanta; | ||
| 121 | u16 max_hw_sq_chunk; | 136 | u16 max_hw_sq_chunk; |
| 122 | u16 min_hw_wq_size; | 137 | u16 min_hw_wq_size; |
| 123 | u8 hw_rev; | 138 | u8 hw_rev; |
| @@ -147,10 +162,13 @@ struct irdma_hw_attrs { · 23 unmodified lines | |||
| 147 | u32 max_done_count; | 162 | u32 max_done_count; |
| 148 | u32 max_sleep_count; | 163 | u32 max_sleep_count; |
| 149 | u32 max_cqp_compl_wait_time_ms; | 164 | u32 max_cqp_compl_wait_time_ms; |
| 165 | u32 min_hw_srq_id; | ||
| 150 | u16 max_stat_inst; | 166 | u16 max_stat_inst; |
| 151 | u16 max_stat_idx; | 167 | u16 max_stat_idx; |
| 152 | }; | 168 | }; |
| 153 | 169 | ||
| 154 | void i40iw_init_hw(struct irdma_sc_dev *dev); | 170 | void i40iw_init_hw(struct irdma_sc_dev *dev); |
| 155 | void icrdma_init_hw(struct irdma_sc_dev *dev); | 171 | void icrdma_init_hw(struct irdma_sc_dev *dev); |
| 172 | void ig3rdma_init_hw(struct irdma_sc_dev *dev); | ||
| 173 | void __iomem *ig3rdma_get_reg_addr(struct irdma_hw *hw, u64 reg_offset); | ||
| 156 | #endif /* IRDMA_H*/ | 174 | #endif /* IRDMA_H*/ |
| @@ -1,6 +1,7 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB | 1 | // SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB |
| 2 | /* Copyright (c) 2015 - 2021 Intel Corporation */ | 2 | /* Copyright (c) 2015 - 2021 Intel Corporation */ |
| 3 | #include "main.h" | 3 | #include "main.h" |
| 4 | #include <linux/net/intel/iidc_rdma_idpf.h> | ||
| 4 | 5 | ||
| 5 | MODULE_ALIAS("i40iw"); | 6 | MODULE_ALIAS("i40iw"); |
| 6 | MODULE_DESCRIPTION("Intel(R) Ethernet Protocol Driver for RDMA"); | 7 | MODULE_DESCRIPTION("Intel(R) Ethernet Protocol Driver for RDMA"); |
| @@ -38,19 +39,7 @@ static void irdma_unregister_notifiers(void) · 31 unmodified lines | |||
| 38 | unregister_netdevice_notifier(&irdma_netdevice_notifier); | 39 | unregister_netdevice_notifier(&irdma_netdevice_notifier); |
| 39 | } | 40 | } |
| 40 | 41 | ||
| 41 | static void irdma_prep_tc_change(struct irdma_device *iwdev) | 42 | void irdma_log_invalid_mtu(u16 mtu, struct irdma_sc_dev *dev) |
| 42 | { | ||
| 43 | iwdev->vsi.tc_change_pending = true; | ||
| 44 | irdma_sc_suspend_resume_qps(&iwdev->vsi, IRDMA_OP_SUSPEND); | ||
| 45 | |||
| 46 | /* Wait for all qp's to suspend */ | ||
| 47 | wait_event_timeout(iwdev->suspend_wq, | ||
| 48 | !atomic_read(&iwdev->vsi.qp_suspend_reqs), | ||
| 49 | msecs_to_jiffies(IRDMA_EVENT_TIMEOUT_MS)); | ||
| 50 | irdma_ws_reset(&iwdev->vsi); | ||
| 51 | } | ||
| 52 | |||
| 53 | static void irdma_log_invalid_mtu(u16 mtu, struct irdma_sc_dev *dev) | ||
| 54 | { | 43 | { |
| 55 | if (mtu < IRDMA_MIN_MTU_IPV4) | 44 | if (mtu < IRDMA_MIN_MTU_IPV4) |
| 56 | ibdev_warn(to_ibdev(dev), "MTU setting [%d] too low for RDMA traffic. Minimum MTU is 576 for IPv4\n", mtu); | 45 | ibdev_warn(to_ibdev(dev), "MTU setting [%d] too low for RDMA traffic. Minimum MTU is 576 for IPv4\n", mtu); |
| @@ -58,35 +47,10 @@ static void irdma_log_invalid_mtu(u16 mtu, struct irdma_sc_dev *dev) · 1 unmodified line | |||
| 58 | ibdev_warn(to_ibdev(dev), "MTU setting [%d] too low for RDMA traffic. Minimum MTU is 1280 for IPv6\\n", mtu); | 47 | ibdev_warn(to_ibdev(dev), "MTU setting [%d] too low for RDMA traffic. Minimum MTU is 1280 for IPv6\\n", mtu); |
| 59 | } | 48 | } |
| 60 | 49 | ||
| 61 | static void irdma_fill_qos_info(struct irdma_l2params *l2params, | 50 | static void ig3rdma_idc_vport_event_handler(struct iidc_rdma_vport_dev_info *cdev_info, |
| 62 | struct iidc_rdma_qos_params *qos_info) | 51 | struct iidc_rdma_event *event) |
| 63 | { | 52 | { |
| 64 | int i; | 53 | struct irdma_device *iwdev = auxiliary_get_drvdata(cdev_info->adev); |
| 65 | |||
| 66 | l2params->num_tc = qos_info->num_tc; | ||
| 67 | l2params->vsi_prio_type = qos_info->vport_priority_type; | ||
| 68 | l2params->vsi_rel_bw = qos_info->vport_relative_bw; | ||
| 69 | for (i = 0; i < l2params->num_tc; i++) { | ||
| 70 | l2params->tc_info[i].egress_virt_up = | ||
| 71 | qos_info->tc_info[i].egress_virt_up; | ||
| 72 | l2params->tc_info[i].ingress_virt_up = | ||
| 73 | qos_info->tc_info[i].ingress_virt_up; | ||
| 74 | l2params->tc_info[i].prio_type = qos_info->tc_info[i].prio_type; | ||
| 75 | l2params->tc_info[i].rel_bw = qos_info->tc_info[i].rel_bw; | ||
| 76 | l2params->tc_info[i].tc_ctx = qos_info->tc_info[i].tc_ctx; | ||
| 77 | } | ||
| 78 | for (i = 0; i < IIDC_MAX_USER_PRIORITY; i++) | ||
| 79 | l2params->up2tc[i] = qos_info->up2tc[i]; | ||
| 80 | if (qos_info->pfc_mode == IIDC_DSCP_PFC_MODE) { | ||
| 81 | l2params->dscp_mode = true; | ||
| 82 | memcpy(l2params->dscp_map, qos_info->dscp_map, sizeof(l2params->dscp_map)); | ||
| 83 | } | ||
| 84 | } | ||
| 85 | |||
| 86 | static void irdma_iidc_event_handler(struct iidc_rdma_core_dev_info *cdev_info, | ||
| 87 | struct iidc_rdma_event *event) | ||
| 88 | { | ||
| 89 | struct irdma_device *iwdev = dev_get_drvdata(&cdev_info->adev->dev); | ||
| 90 | struct irdma_l2params l2params = {}; | 54 | struct irdma_l2params l2params = {}; |
| 91 | 55 | ||
| 92 | if (*event->type & BIT(IIDC_RDMA_EVENT_AFTER_MTU_CHANGE)) { | 56 | if (*event->type & BIT(IIDC_RDMA_EVENT_AFTER_MTU_CHANGE)) { |
| @@ -97,248 +61,39 @@ static void irdma_iidc_event_handler(struct iidc_rdma_core_dev_info *cdev_info, · 4 unmodified lines | |||
| 97 | irdma_log_invalid_mtu(l2params.mtu, &iwdev->rf->sc_dev); | 61 | irdma_log_invalid_mtu(l2params.mtu, &iwdev->rf->sc_dev); |
| 98 | irdma_change_l2params(&iwdev->vsi, &l2params); | 62 | irdma_change_l2params(&iwdev->vsi, &l2params); |
| 99 | } | 63 | } |
| 100 | } else if (*event->type & BIT(IIDC_RDMA_EVENT_BEFORE_TC_CHANGE)) { | ||
| 101 | if (iwdev->vsi.tc_change_pending) | ||
| 102 | return; | ||
| 103 | |||
| 104 | irdma_prep_tc_change(iwdev); | ||
| 105 | } else if (*event->type & BIT(IIDC_RDMA_EVENT_AFTER_TC_CHANGE)) { | ||
| 106 | struct iidc_rdma_priv_dev_info *iidc_priv = cdev_info->iidc_priv; | ||
| 107 | |||
| 108 | if (!iwdev->vsi.tc_change_pending) | ||
| 109 | return; | ||
| 110 | |||
| 111 | l2params.tc_changed = true; | ||
| 112 | ibdev_dbg(&iwdev->ibdev, "CLNT: TC Change\n"); | ||
| 113 | |||
| 114 | irdma_fill_qos_info(&l2params, &iidc_priv->qos_info); | ||
| 115 | if (iwdev->rf->protocol_used != IRDMA_IWARP_PROTOCOL_ONLY) | ||
| 116 | iwdev->dcb_vlan_mode = | ||
| 117 | l2params.num_tc > 1 && !l2params.dscp_mode; | ||
| 118 | irdma_change_l2params(&iwdev->vsi, &l2params); | ||
| 119 | } else if (*event->type & BIT(IIDC_RDMA_EVENT_CRIT_ERR)) { | ||
| 120 | ibdev_warn(&iwdev->ibdev, "ICE OICR event notification: oicr = 0x%08x\n", | ||
| 121 | event->reg); | ||
| 122 | if (event->reg & IRDMAPFINT_OICR_PE_CRITERR_M) { | ||
| 123 | u32 pe_criterr; | ||
| 124 | |||
| 125 | pe_criterr = readl(iwdev->rf->sc_dev.hw_regs[IRDMA_GLPE_CRITERR]); | ||
| 126 | #define IRDMA_Q1_RESOURCE_ERR 0x0001024d | ||
| 127 | if (pe_criterr != IRDMA_Q1_RESOURCE_ERR) { | ||
| 128 | ibdev_err(&iwdev->ibdev, "critical PE Error, GLPE_CRITERR=0x%08x\n", | ||
| 129 | pe_criterr); | ||
| 130 | iwdev->rf->reset = true; | ||
| 131 | } else { | ||
| 132 | ibdev_warn(&iwdev->ibdev, "Q1 Resource Check\n"); | ||
| 133 | } | ||
| 134 | } | ||
| 135 | if (event->reg & IRDMAPFINT_OICR_HMC_ERR_M) { | ||
| 136 | ibdev_err(&iwdev->ibdev, "HMC Error\n"); | ||
| 137 | iwdev->rf->reset = true; | ||
| 138 | } | ||
| 139 | if (event->reg & IRDMAPFINT_OICR_PE_PUSH_M) { | ||
| 140 | ibdev_err(&iwdev->ibdev, "PE Push Error\n"); | ||
| 141 | iwdev->rf->reset = true; | ||
| 142 | } | ||
| 143 | if (iwdev->rf->reset) | ||
| 144 | iwdev->rf->gen_ops.request_reset(iwdev->rf); | ||
| 145 | } | 64 | } |
| 146 | } | 65 | } |
| 147 | 66 | ||
| 148 | /** | 67 | static int ig3rdma_vport_probe(struct auxiliary_device *aux_dev, |
| 149 | * irdma_request_reset - Request a reset | 68 | const struct auxiliary_device_id *id) |
| 150 | * @rf: RDMA PCI function | ||
| 151 | */ | ||
| 152 | static void irdma_request_reset(struct irdma_pci_f *rf) | ||
| 153 | { | ||
| 154 | ibdev_warn(&rf->iwdev->ibdev, "Requesting a reset\n"); | ||
| 155 | ice_rdma_request_reset(rf->cdev, IIDC_FUNC_RESET); | ||
| 156 | } | ||
| 157 | |||
| 158 | /** | ||
| 159 | * irdma_lan_register_qset - Register qset with LAN driver | ||
| 160 | * @vsi: vsi structure | ||
| 161 | * @tc_node: Traffic class node | ||
| 162 | */ | ||
| 163 | static int irdma_lan_register_qset(struct irdma_sc_vsi *vsi, | ||
| 164 | struct irdma_ws_node *tc_node) | ||
| 165 | { | ||
| 166 | struct irdma_device *iwdev = vsi->back_vsi; | ||
| 167 | struct iidc_rdma_core_dev_info *cdev_info; | ||
| 168 | struct iidc_rdma_qset_params qset = {}; | ||
| 169 | int ret; | ||
| 170 | |||
| 171 | cdev_info = iwdev->rf->cdev; | ||
| 172 | qset.qs_handle = tc_node->qs_handle; | ||
| 173 | qset.tc = tc_node->traffic_class; | ||
| 174 | qset.vport_id = vsi->vsi_idx; | ||
| 175 | ret = ice_add_rdma_qset(cdev_info, &qset); | ||
| 176 | if (ret) { | ||
| 177 | ibdev_dbg(&iwdev->ibdev, "WS: LAN alloc_res for rdma qset failed.\n"); | ||
| 178 | return ret; | ||
| 179 | } | ||
| 180 | |||
| 181 | tc_node->l2_sched_node_id = qset.teid; | ||
| 182 | vsi->qos[tc_node->user_pri].l2_sched_node_id = qset.teid; | ||
| 183 | |||
| 184 | return 0; | ||
| 185 | } | ||
| 186 | |||
| 187 | /** | ||
| 188 | * irdma_lan_unregister_qset - Unregister qset with LAN driver | ||
| 189 | * @vsi: vsi structure | ||
| 190 | * @tc_node: Traffic class node | ||
| 191 | */ | ||
| 192 | static void irdma_lan_unregister_qset(struct irdma_sc_vsi *vsi, | ||
| 193 | struct irdma_ws_node *tc_node) | ||
| 194 | { | 69 | { |
| 195 | struct irdma_device *iwdev = vsi->back_vsi; | 70 | struct iidc_rdma_vport_auxiliary_dev *idc_adev = |
| 196 | struct iidc_rdma_core_dev_info *cdev_info; | 71 | container_of(aux_dev, struct iidc_rdma_vport_auxiliary_dev, adev); |
| 197 | struct iidc_rdma_qset_params qset = {}; | 72 | struct auxiliary_device *aux_core_dev = idc_adev->vdev_info->core_adev; |
| 198 | 73 | struct irdma_pci_f *rf = auxiliary_get_drvdata(aux_core_dev); | |
| 199 | cdev_info = iwdev->rf->cdev; | 74 | struct irdma_l2params l2params = {}; |
| 200 | qset.qs_handle = tc_node->qs_handle; | 75 | struct irdma_device *iwdev; |
| 201 | qset.tc = tc_node->traffic_class; | 76 | int err; |
| 202 | qset.vport_id = vsi->vsi_idx; | ||
| 203 | qset.teid = tc_node->l2_sched_node_id; | ||
| 204 | |||
| 205 | if (ice_del_rdma_qset(cdev_info, &qset)) | ||
| 206 | ibdev_dbg(&iwdev->ibdev, "WS: LAN free_res for rdma qset failed.\n"); | ||
| 207 | } | ||
| 208 | |||
| 209 | static int irdma_init_interrupts(struct irdma_pci_f *rf, struct iidc_rdma_core_dev_info *cdev) | ||
| 210 | { | ||
| 211 | int i; | ||
| 212 | |||
| 213 | rf->msix_count = num_online_cpus() + IRDMA_NUM_AEQ_MSIX; | ||
| 214 | rf->msix_entries = kcalloc(rf->msix_count, sizeof(*rf->msix_entries), | ||
| 215 | GFP_KERNEL); | ||
| 216 | if (!rf->msix_entries) | ||
| 217 | return -ENOMEM; | ||
| 218 | |||
| 219 | for (i = 0; i < rf->msix_count; i++) | ||
| 220 | if (ice_alloc_rdma_qvector(cdev, &rf->msix_entries[i])) | ||
| 221 | break; | ||
| 222 | |||
| 223 | if (i < IRDMA_MIN_MSIX) { | ||
| 224 | while (--i >= 0) | ||
| 225 | ice_free_rdma_qvector(cdev, &rf->msix_entries[i]); | ||
| 226 | 77 | ||
| 227 | kfree(rf->msix_entries); | 78 | if (!rf) { |
| 79 | WARN_ON_ONCE(1); | ||
| 228 | return -ENOMEM; | 80 | return -ENOMEM; |
| 229 | } | 81 | } |
| 230 | 82 | iwdev = ib_alloc_device(irdma_device, ibdev); | |
| 231 | rf->msix_count = i; | 83 | /* Fill iwdev info */ |
| 232 | 84 | iwdev->is_vport = true; | |
| 233 | return 0; | 85 | iwdev->rf = rf; |
| 234 | } | 86 | iwdev->vport_id = idc_adev->vdev_info->vport_id; |
| 235 | 87 | iwdev->netdev = idc_adev->vdev_info->netdev; | |
| 236 | static void irdma_deinit_interrupts(struct irdma_pci_f *rf, struct iidc_rdma_core_dev_info *cdev) | ||
| 237 | { | ||
| 238 | int i; | ||
| 239 | |||
| 240 | for (i = 0; i < rf->msix_count; i++) | ||
| 241 | ice_free_rdma_qvector(cdev, &rf->msix_entries[i]); | ||
| 242 | |||
| 243 | kfree(rf->msix_entries); | ||
| 244 | } | ||
| 245 | |||
| 246 | static void irdma_remove(struct auxiliary_device *aux_dev) | ||
| 247 | { | ||
| 248 | struct irdma_device *iwdev = auxiliary_get_drvdata(aux_dev); | ||
| 249 | struct iidc_rdma_core_auxiliary_dev *iidc_adev; | ||
| 250 | struct iidc_rdma_core_dev_info *cdev_info; | ||
| 251 | |||
| 252 | iidc_adev = container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev); | ||
| 253 | cdev_info = iidc_adev->cdev_info; | ||
| 254 | |||
| 255 | ice_rdma_update_vsi_filter(cdev_info, iwdev->vsi_num, false); | ||
| 256 | irdma_ib_unregister_device(iwdev); | ||
| 257 | irdma_deinit_interrupts(iwdev->rf, cdev_info); | ||
| 258 | |||
| 259 | kfree(iwdev->rf); | ||
| 260 | |||
| 261 | pr_debug("INIT: Gen2 PF[%d] device remove success\n", PCI_FUNC(cdev_info->pdev->devfn)); | ||
| 262 | } | ||
| 263 | |||
| 264 | static void irdma_fill_device_info(struct irdma_device *iwdev, | ||
| 265 | struct iidc_rdma_core_dev_info *cdev_info) | ||
| 266 | { | ||
| 267 | struct iidc_rdma_priv_dev_info *iidc_priv = cdev_info->iidc_priv; | ||
| 268 | struct irdma_pci_f *rf = iwdev->rf; | ||
| 269 | |||
| 270 | rf->sc_dev.hw = &rf->hw; | ||
| 271 | rf->iwdev = iwdev; | ||
| 272 | rf->cdev = cdev_info; | ||
| 273 | rf->hw.hw_addr = iidc_priv->hw_addr; | ||
| 274 | rf->pcidev = cdev_info->pdev; | ||
| 275 | rf->hw.device = &rf->pcidev->dev; | ||
| 276 | rf->pf_id = iidc_priv->pf_id; | ||
| 277 | rf->gen_ops.register_qset = irdma_lan_register_qset; | ||
| 278 | rf->gen_ops.unregister_qset = irdma_lan_unregister_qset; | ||
| 279 | |||
| 280 | rf->default_vsi.vsi_idx = iidc_priv->vport_id; | ||
| 281 | rf->protocol_used = | ||
| 282 | cdev_info->rdma_protocol == IIDC_RDMA_PROTOCOL_ROCEV2 ? | ||
| 283 | IRDMA_ROCE_PROTOCOL_ONLY : IRDMA_IWARP_PROTOCOL_ONLY; | ||
| 284 | rf->rdma_ver = IRDMA_GEN_2; | ||
| 285 | rf->rsrc_profile = IRDMA_HMC_PROFILE_DEFAULT; | ||
| 286 | rf->rst_to = IRDMA_RST_TIMEOUT_HZ; | ||
| 287 | rf->gen_ops.request_reset = irdma_request_reset; | ||
| 288 | rf->limits_sel = 7; | ||
| 289 | rf->iwdev = iwdev; | ||
| 290 | |||
| 291 | mutex_init(&iwdev->ah_tbl_lock); | ||
| 292 | |||
| 293 | iwdev->netdev = iidc_priv->netdev; | ||
| 294 | iwdev->vsi_num = iidc_priv->vport_id; | ||
| 295 | iwdev->init_state = INITIAL_STATE; | 88 | iwdev->init_state = INITIAL_STATE; |
| 296 | iwdev->roce_cwnd = IRDMA_ROCE_CWND_DEFAULT; | 89 | iwdev->roce_cwnd = IRDMA_ROCE_CWND_DEFAULT; |
| 297 | iwdev->roce_ackcreds = IRDMA_ROCE_ACKCREDS_DEFAULT; | 90 | iwdev->roce_ackcreds = IRDMA_ROCE_ACKCREDS_DEFAULT; |
| 298 | iwdev->rcv_wnd = IRDMA_CM_DEFAULT_RCV_WND_SCALED; | 91 | iwdev->rcv_wnd = IRDMA_CM_DEFAULT_RCV_WND_SCALED; |
| 299 | iwdev->rcv_wscale = IRDMA_CM_DEFAULT_RCV_WND_SCALE; | 92 | iwdev->rcv_wscale = IRDMA_CM_DEFAULT_RCV_WND_SCALE; |
| 300 | if (rf->protocol_used == IRDMA_ROCE_PROTOCOL_ONLY) | 93 | iwdev->roce_mode = true; |
| 301 | iwdev->roce_mode = true; | 94 | iwdev->push_mode = false; |
| 302 | } | ||
| 303 | |||
| 304 | static int irdma_probe(struct auxiliary_device *aux_dev, const struct auxiliary_device_id *id) | ||
| 305 | { | ||
| 306 | struct iidc_rdma_core_auxiliary_dev *iidc_adev; | ||
| 307 | struct iidc_rdma_core_dev_info *cdev_info; | ||
| 308 | struct iidc_rdma_priv_dev_info *iidc_priv; | ||
| 309 | struct irdma_l2params l2params = {}; | ||
| 310 | struct irdma_device *iwdev; | ||
| 311 | struct irdma_pci_f *rf; | ||
| 312 | int err; | ||
| 313 | |||
| 314 | iidc_adev = container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev); | ||
| 315 | cdev_info = iidc_adev->cdev_info; | ||
| 316 | iidc_priv = cdev_info->iidc_priv; | ||
| 317 | |||
| 318 | iwdev = ib_alloc_device(irdma_device, ibdev); | ||
| 319 | if (!iwdev) | ||
| 320 | return -ENOMEM; | ||
| 321 | iwdev->rf = kzalloc(sizeof(*rf), GFP_KERNEL); | ||
| 322 | if (!iwdev->rf) { | ||
| 323 | ib_dealloc_device(&iwdev->ibdev); | ||
| 324 | return -ENOMEM; | ||
| 325 | } | ||
| 326 | |||
| 327 | irdma_fill_device_info(iwdev, cdev_info); | ||
| 328 | rf = iwdev->rf; | ||
| 329 | |||
| 330 | err = irdma_init_interrupts(rf, cdev_info); | ||
| 331 | if (err) | ||
| 332 | goto err_init_interrupts; | ||
| 333 | |||
| 334 | err = irdma_ctrl_init_hw(rf); | ||
| 335 | if (err) | ||
| 336 | goto err_ctrl_init; | ||
| 337 | 95 | ||
| 338 | l2params.mtu = iwdev->netdev->mtu; | 96 | l2params.mtu = iwdev->netdev->mtu; |
| 339 | irdma_fill_qos_info(&l2params, &iidc_priv->qos_info); | ||
| 340 | if (iwdev->rf->protocol_used != IRDMA_IWARP_PROTOCOL_ONLY) | ||
| 341 | iwdev->dcb_vlan_mode = l2params.num_tc > 1 && !l2params.dscp_mode; | ||
| 342 | 97 | ||
| 343 | err = irdma_rt_init_hw(iwdev, &l2params); | 98 | err = irdma_rt_init_hw(iwdev, &l2params); |
| 344 | if (err) | 99 | if (err) |
| @@ -348,43 +103,57 @@ static int irdma_probe(struct auxiliary_device *aux_dev, const struct auxiliary_ · 3 unmodified lines | |||
| 348 | if (err) | 103 | if (err) |
| 349 | goto err_ibreg; | 104 | goto err_ibreg; |
| 350 | 105 | ||
| 351 | ice_rdma_update_vsi_filter(cdev_info, iwdev->vsi_num, true); | ||
| 352 | |||
| 353 | ibdev_dbg(&iwdev->ibdev, "INIT: Gen2 PF[%d] device probe success\n", PCI_FUNC(rf->pcidev->devfn)); | ||
| 354 | auxiliary_set_drvdata(aux_dev, iwdev); | 106 | auxiliary_set_drvdata(aux_dev, iwdev); |
| 355 | 107 | ||
| 356 | return 0; | 108 | ibdev_dbg(&iwdev->ibdev, |
| 109 | "INIT: Gen[%d] vport[%d] probe success. dev_name = %s, core_dev_name = %s, netdev=%s\n", | ||
| 110 | rf->rdma_ver, idc_adev->vdev_info->vport_id, | ||
| 111 | dev_name(&aux_dev->dev), | ||
| 112 | dev_name(&idc_adev->vdev_info->core_adev->dev), | ||
| 113 | netdev_name(idc_adev->vdev_info->netdev)); | ||
| 357 | 114 | ||
| 115 | return 0; | ||
| 358 | err_ibreg: | 116 | err_ibreg: |
| 359 | irdma_rt_deinit_hw(iwdev); | 117 | irdma_rt_deinit_hw(iwdev); |
| 360 | err_rt_init: | 118 | err_rt_init: |
| 361 | irdma_ctrl_deinit_hw(rf); | ||
| 362 | err_ctrl_init: | ||
| 363 | irdma_deinit_interrupts(rf, cdev_info); | ||
| 364 | err_init_interrupts: | ||
| 365 | kfree(iwdev->rf); | ||
| 366 | ib_dealloc_device(&iwdev->ibdev); | 119 | ib_dealloc_device(&iwdev->ibdev); |
| 367 | 120 | ||
| 368 | return err; | 121 | return err; |
| 369 | } | 122 | } |
| 370 | 123 | ||
| 371 | static const struct auxiliary_device_id irdma_auxiliary_id_table[] = { | 124 | static void ig3rdma_vport_remove(struct auxiliary_device *aux_dev) |
| 372 | {.name = "ice.iwarp", }, | 125 | { |
| 373 | {.name = "ice.roce", }, | 126 | struct iidc_rdma_vport_auxiliary_dev *idc_adev = |
| 127 | container_of(aux_dev, struct iidc_rdma_vport_auxiliary_dev, adev); | ||
| 128 | struct irdma_device *iwdev = auxiliary_get_drvdata(aux_dev); | ||
| 129 | |||
| 130 | ibdev_dbg(&iwdev->ibdev, | ||
| 131 | "INIT: Gen[%d] dev_name = %s, core_dev_name = %s, netdev=%s\n", | ||
| 132 | iwdev->rf->rdma_ver, dev_name(&aux_dev->dev), | ||
| 133 | dev_name(&idc_adev->vdev_info->core_adev->dev), | ||
| 134 | netdev_name(idc_adev->vdev_info->netdev)); | ||
| 135 | |||
| 136 | irdma_ib_unregister_device(iwdev); | ||
| 137 | } | ||
| 138 | |||
| 139 | static const struct auxiliary_device_id ig3rdma_vport_auxiliary_id_table[] = { | ||
| 140 | {.name = "idpf.8086.rdma.vdev", }, | ||
| 374 | {}, | 141 | {}, |
| 375 | }; | 142 | }; |
| 376 | 143 | ||
| 377 | MODULE_DEVICE_TABLE(auxiliary, irdma_auxiliary_id_table); | 144 | MODULE_DEVICE_TABLE(auxiliary, ig3rdma_vport_auxiliary_id_table); |
| 378 | 145 | ||
| 379 | static struct iidc_rdma_core_auxiliary_drv irdma_auxiliary_drv = { | 146 | static struct iidc_rdma_vport_auxiliary_drv ig3rdma_vport_auxiliary_drv = { |
| 380 | .adrv = { | 147 | .adrv = { |
| 381 | .id_table = irdma_auxiliary_id_table, | 148 | .name = "vdev", |
| 382 | .probe = irdma_probe, | 149 | .id_table = ig3rdma_vport_auxiliary_id_table, |
| 383 | .remove = irdma_remove, | 150 | .probe = ig3rdma_vport_probe, |
| 151 | .remove = ig3rdma_vport_remove, | ||
| 384 | }, | 152 | }, |
| 385 | .event_handler = irdma_iidc_event_handler, | 153 | .event_handler = ig3rdma_idc_vport_event_handler, |
| 386 | }; | 154 | }; |
| 387 | 155 | ||
| 156 | |||
| 388 | static int __init irdma_init_module(void) | 157 | static int __init irdma_init_module(void) |
| 389 | { | 158 | { |
| 390 | int ret; | 159 | int ret; |
| @@ -396,14 +165,34 @@ static int __init irdma_init_module(void) · 5 unmodified lines | |||
| 396 | return ret; | 165 | return ret; |
| 397 | } | 166 | } |
| 398 | 167 | ||
| 399 | ret = auxiliary_driver_register(&irdma_auxiliary_drv.adrv); | 168 | ret = auxiliary_driver_register(&icrdma_core_auxiliary_drv.adrv); |
| 169 | if (ret) { | ||
| 170 | auxiliary_driver_unregister(&i40iw_auxiliary_drv); | ||
| 171 | pr_err("Failed icrdma(gen_2) auxiliary_driver_register() ret=%d\n", | ||
| 172 | ret); | ||
| 173 | return ret; | ||
| 174 | } | ||
| 175 | |||
| 176 | ret = auxiliary_driver_register(&ig3rdma_core_auxiliary_drv.adrv); | ||
| 400 | if (ret) { | 177 | if (ret) { |
| 178 | auxiliary_driver_unregister(&icrdma_core_auxiliary_drv.adrv); | ||
| 401 | auxiliary_driver_unregister(&i40iw_auxiliary_drv); | 179 | auxiliary_driver_unregister(&i40iw_auxiliary_drv); |
| 402 | pr_err("Failed irdma auxiliary_driver_register() ret=%d\n", | 180 | pr_err("Failed ig3rdma(gen_3) core auxiliary_driver_register() ret=%d\n", |
| 403 | ret); | 181 | ret); |
| 182 | |||
| 404 | return ret; | 183 | return ret; |
| 405 | } | 184 | } |
| 406 | 185 | ||
| 186 | ret = auxiliary_driver_register(&ig3rdma_vport_auxiliary_drv.adrv); | ||
| 187 | if (ret) { | ||
| 188 | auxiliary_driver_unregister(&ig3rdma_core_auxiliary_drv.adrv); | ||
| 189 | auxiliary_driver_unregister(&icrdma_core_auxiliary_drv.adrv); | ||
| 190 | auxiliary_driver_unregister(&i40iw_auxiliary_drv); | ||
| 191 | pr_err("Failed ig3rdma vport auxiliary_driver_register() ret=%d\n", | ||
| 192 | ret); | ||
| 193 | |||
| 194 | return ret; | ||
| 195 | } | ||
| 407 | irdma_register_notifiers(); | 196 | irdma_register_notifiers(); |
| 408 | 197 | ||
| 409 | return 0; | 198 | return 0; |
| @@ -412,8 +201,10 @@ static int __init irdma_init_module(void) · 2 unmodified lines | |||
| 412 | static void __exit irdma_exit_module(void) | 201 | static void __exit irdma_exit_module(void) |
| 413 | { | 202 | { |
| 414 | irdma_unregister_notifiers(); | 203 | irdma_unregister_notifiers(); |
| 415 | auxiliary_driver_unregister(&irdma_auxiliary_drv.adrv); | 204 | auxiliary_driver_unregister(&icrdma_core_auxiliary_drv.adrv); |
| 416 | auxiliary_driver_unregister(&i40iw_auxiliary_drv); | 205 | auxiliary_driver_unregister(&i40iw_auxiliary_drv); |
| 206 | auxiliary_driver_unregister(&ig3rdma_core_auxiliary_drv.adrv); | ||
| 207 | auxiliary_driver_unregister(&ig3rdma_vport_auxiliary_drv.adrv); | ||
| 417 | } | 208 | } |
| 418 | 209 | ||
| 419 | module_init(irdma_init_module); | 210 | module_init(irdma_init_module); |
| @@ -30,7 +30,6 @@ · 29 unmodified lines | |||
| 30 | #endif | 30 | #endif |
| 31 | #include <linux/auxiliary_bus.h> | 31 | #include <linux/auxiliary_bus.h> |
| 32 | #include <linux/net/intel/iidc_rdma.h> | 32 | #include <linux/net/intel/iidc_rdma.h> |
| 33 | #include <linux/net/intel/iidc_rdma_ice.h> | ||
| 34 | #include <rdma/ib_smi.h> | 33 | #include <rdma/ib_smi.h> |
| 35 | #include <rdma/ib_verbs.h> | 34 | #include <rdma/ib_verbs.h> |
| 36 | #include <rdma/ib_pack.h> | 35 | #include <rdma/ib_pack.h> |
| @@ -54,6 +53,8 @@ · 17 unmodified lines | |||
| 54 | #include "puda.h" | 53 | #include "puda.h" |
| 55 | 54 | ||
| 56 | extern struct auxiliary_driver i40iw_auxiliary_drv; | 55 | extern struct auxiliary_driver i40iw_auxiliary_drv; |
| 56 | extern struct iidc_rdma_core_auxiliary_drv icrdma_core_auxiliary_drv; | ||
| 57 | extern struct iidc_rdma_core_auxiliary_drv ig3rdma_core_auxiliary_drv; | ||
| 57 | 58 | ||
| 58 | #define IRDMA_FW_VER_DEFAULT 2 | 59 | #define IRDMA_FW_VER_DEFAULT 2 |
| 59 | #define IRDMA_HW_VER 2 | 60 | #define IRDMA_HW_VER 2 |
| @@ -65,7 +66,8 @@ extern struct auxiliary_driver i40iw_auxiliary_drv; · 5 unmodified lines | |||
| 65 | #define IRDMA_MACIP_ADD 1 | 66 | #define IRDMA_MACIP_ADD 1 |
| 66 | #define IRDMA_MACIP_DELETE 2 | 67 | #define IRDMA_MACIP_DELETE 2 |
| 67 | 68 | ||
| 68 | #define IW_CCQ_SIZE (IRDMA_CQP_SW_SQSIZE_2048 + 1) | 69 | #define IW_GEN_3_CCQ_SIZE (2 * IRDMA_CQP_SW_SQSIZE_2048 + 2) |
| 70 | #define IW_CCQ_SIZE (IRDMA_CQP_SW_SQSIZE_2048 + 2) | ||
| 69 | #define IW_CEQ_SIZE 2048 | 71 | #define IW_CEQ_SIZE 2048 |
| 70 | #define IW_AEQ_SIZE 2048 | 72 | #define IW_AEQ_SIZE 2048 |
| 71 | 73 | ||
| @@ -127,12 +129,12 @@ enum init_completion_state { · 55 unmodified lines | |||
| 127 | HMC_OBJS_CREATED, | 129 | HMC_OBJS_CREATED, |
| 128 | HW_RSRC_INITIALIZED, | 130 | HW_RSRC_INITIALIZED, |
| 129 | CCQ_CREATED, | 131 | CCQ_CREATED, |
| 130 | CEQ0_CREATED, /* Last state of probe */ | 132 | CEQ0_CREATED, |
| 131 | ILQ_CREATED, | ||
| 132 | IEQ_CREATED, | ||
| 133 | CEQS_CREATED, | 133 | CEQS_CREATED, |
| 134 | PBLE_CHUNK_MEM, | 134 | PBLE_CHUNK_MEM, |
| 135 | AEQ_CREATED, | 135 | AEQ_CREATED, |
| 136 | ILQ_CREATED, | ||
| 137 | IEQ_CREATED, /* Last state of probe */ | ||
| 136 | IP_ADDR_REGISTERED, /* Last state of open */ | 138 | IP_ADDR_REGISTERED, /* Last state of open */ |
| 137 | }; | 139 | }; |
| 138 | 140 | ||
| @@ -167,6 +169,7 @@ struct irdma_cqp_request { · 28 unmodified lines | |||
| 167 | bool request_done; /* READ/WRITE_ONCE macros operate on it */ | 169 | bool request_done; /* READ/WRITE_ONCE macros operate on it */ |
| 168 | bool waiting:1; | 170 | bool waiting:1; |
| 169 | bool dynamic:1; | 171 | bool dynamic:1; |
| 172 | bool pending:1; | ||
| 170 | }; | 173 | }; |
| 171 | 174 | ||
| 172 | struct irdma_cqp { | 175 | struct irdma_cqp { |
| @@ -179,6 +182,7 @@ struct irdma_cqp { · 6 unmodified lines | |||
| 179 | struct irdma_dma_mem host_ctx; | 182 | struct irdma_dma_mem host_ctx; |
| 180 | u64 *scratch_array; | 183 | u64 *scratch_array; |
| 181 | struct irdma_cqp_request *cqp_requests; | 184 | struct irdma_cqp_request *cqp_requests; |
| 185 | struct irdma_ooo_cqp_op *oop_op_array; | ||
| 182 | struct list_head cqp_avail_reqs; | 186 | struct list_head cqp_avail_reqs; |
| 183 | struct list_head cqp_pending_reqs; | 187 | struct list_head cqp_pending_reqs; |
| 184 | }; | 188 | }; |
| @@ -257,6 +261,7 @@ struct irdma_pci_f { · 72 unmodified lines | |||
| 257 | bool reset:1; | 261 | bool reset:1; |
| 258 | bool rsrc_created:1; | 262 | bool rsrc_created:1; |
| 259 | bool msix_shared:1; | 263 | bool msix_shared:1; |
| 264 | bool hwqp1_rsvd:1; | ||
| 260 | u8 rsrc_profile; | 265 | u8 rsrc_profile; |
| 261 | u8 *hmc_info_mem; | 266 | u8 *hmc_info_mem; |
| 262 | u8 *mem_rsrc; | 267 | u8 *mem_rsrc; |
| @@ -269,6 +274,8 @@ struct irdma_pci_f { · 6 unmodified lines | |||
| 269 | u32 max_mr; | 274 | u32 max_mr; |
| 270 | u32 max_qp; | 275 | u32 max_qp; |
| 271 | u32 max_cq; | 276 | u32 max_cq; |
| 277 | u32 max_srq; | ||
| 278 | u32 next_srq; | ||
| 272 | u32 max_ah; | 279 | u32 max_ah; |
| 273 | u32 next_ah; | 280 | u32 next_ah; |
| 274 | u32 max_mcg; | 281 | u32 max_mcg; |
| @@ -282,6 +289,7 @@ struct irdma_pci_f { · 7 unmodified lines | |||
| 282 | u32 mr_stagmask; | 289 | u32 mr_stagmask; |
| 283 | u32 used_pds; | 290 | u32 used_pds; |
| 284 | u32 used_cqs; | 291 | u32 used_cqs; |
| 292 | u32 used_srqs; | ||
| 285 | u32 used_mrs; | 293 | u32 used_mrs; |
| 286 | u32 used_qps; | 294 | u32 used_qps; |
| 287 | u32 arp_table_size; | 295 | u32 arp_table_size; |
| @@ -293,6 +301,7 @@ struct irdma_pci_f { · 5 unmodified lines | |||
| 293 | unsigned long *allocated_ws_nodes; | 301 | unsigned long *allocated_ws_nodes; |
| 294 | unsigned long *allocated_qps; | 302 | unsigned long *allocated_qps; |
| 295 | unsigned long *allocated_cqs; | 303 | unsigned long *allocated_cqs; |
| 304 | unsigned long *allocated_srqs; | ||
| 296 | unsigned long *allocated_mrs; | 305 | unsigned long *allocated_mrs; |
| 297 | unsigned long *allocated_pds; | 306 | unsigned long *allocated_pds; |
| 298 | unsigned long *allocated_mcgs; | 307 | unsigned long *allocated_mcgs; |
| @@ -327,10 +336,13 @@ struct irdma_pci_f { · 28 unmodified lines | |||
| 327 | wait_queue_head_t vchnl_waitq; | 336 | wait_queue_head_t vchnl_waitq; |
| 328 | struct workqueue_struct *cqp_cmpl_wq; | 337 | struct workqueue_struct *cqp_cmpl_wq; |
| 329 | struct work_struct cqp_cmpl_work; | 338 | struct work_struct cqp_cmpl_work; |
| 339 | struct workqueue_struct *vchnl_wq; | ||
| 330 | struct irdma_sc_vsi default_vsi; | 340 | struct irdma_sc_vsi default_vsi; |
| 331 | void *back_fcn; | 341 | void *back_fcn; |
| 332 | struct irdma_gen_ops gen_ops; | 342 | struct irdma_gen_ops gen_ops; |
| 333 | struct irdma_device *iwdev; | 343 | struct irdma_device *iwdev; |
| 344 | DECLARE_HASHTABLE(ah_hash_tbl, 8); | ||
| 345 | struct mutex ah_tbl_lock; /* protect AH hash table access */ | ||
| 334 | }; | 346 | }; |
| 335 | 347 | ||
| 336 | struct irdma_device { | 348 | struct irdma_device { |
| @@ -340,8 +352,6 @@ struct irdma_device { · 3 unmodified lines | |||
| 340 | struct workqueue_struct *cleanup_wq; | 352 | struct workqueue_struct *cleanup_wq; |
| 341 | struct irdma_sc_vsi vsi; | 353 | struct irdma_sc_vsi vsi; |
| 342 | struct irdma_cm_core cm_core; | 354 | struct irdma_cm_core cm_core; |
| 343 | DECLARE_HASHTABLE(ah_hash_tbl, 8); | ||
| 344 | struct mutex ah_tbl_lock; /* protect AH hash table access */ | ||
| 345 | u32 roce_cwnd; | 355 | u32 roce_cwnd; |
| 346 | u32 roce_ackcreds; | 356 | u32 roce_ackcreds; |
| 347 | u32 vendor_id; | 357 | u32 vendor_id; |
| @@ -350,12 +360,14 @@ struct irdma_device { · 2 unmodified lines | |||
| 350 | u32 rcv_wnd; | 360 | u32 rcv_wnd; |
| 351 | u16 mac_ip_table_idx; | 361 | u16 mac_ip_table_idx; |
| 352 | u16 vsi_num; | 362 | u16 vsi_num; |
| 363 | u16 vport_id; | ||
| 353 | u8 rcv_wscale; | 364 | u8 rcv_wscale; |
| 354 | u8 iw_status; | 365 | u8 iw_status; |
| 355 | bool roce_mode:1; | 366 | bool roce_mode:1; |
| 356 | bool roce_dcqcn_en:1; | 367 | bool roce_dcqcn_en:1; |
| 357 | bool dcb_vlan_mode:1; | 368 | bool dcb_vlan_mode:1; |
| 358 | bool iw_ooo:1; | 369 | bool iw_ooo:1; |
| 370 | bool is_vport:1; | ||
| 359 | enum init_completion_state init_state; | 371 | enum init_completion_state init_state; |
| 360 | 372 | ||
| 361 | wait_queue_head_t suspend_wq; | 373 | wait_queue_head_t suspend_wq; |
| @@ -413,6 +425,11 @@ static inline struct irdma_pci_f *dev_to_rf(struct irdma_sc_dev *dev) · 51 unmodified lines | |||
| 413 | return container_of(dev, struct irdma_pci_f, sc_dev); | 425 | return container_of(dev, struct irdma_pci_f, sc_dev); |
| 414 | } | 426 | } |
| 415 | 427 | ||
| 428 | static inline struct irdma_srq *to_iwsrq(struct ib_srq *ibsrq) | ||
| 429 | { | ||
| 430 | return container_of(ibsrq, struct irdma_srq, ibsrq); | ||
| 431 | } | ||
| 432 | |||
| 416 | /** | 433 | /** |
| 417 | * irdma_alloc_resource - allocate a resource | 434 | * irdma_alloc_resource - allocate a resource |
| 418 | * @iwdev: device pointer | 435 | * @iwdev: device pointer |
| @@ -508,7 +525,8 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 89 unmodified lines | |||
| 508 | void irdma_cq_add_ref(struct ib_cq *ibcq); | 525 | void irdma_cq_add_ref(struct ib_cq *ibcq); |
| 509 | void irdma_cq_rem_ref(struct ib_cq *ibcq); | 526 | void irdma_cq_rem_ref(struct ib_cq *ibcq); |
| 510 | void irdma_cq_wq_destroy(struct irdma_pci_f *rf, struct irdma_sc_cq *cq); | 527 | void irdma_cq_wq_destroy(struct irdma_pci_f *rf, struct irdma_sc_cq *cq); |
| 511 | 528 | void irdma_srq_event(struct irdma_sc_srq *srq); | |
| 529 | void irdma_srq_wq_destroy(struct irdma_pci_f *rf, struct irdma_sc_srq *srq); | ||
| 512 | void irdma_cleanup_pending_cqp_op(struct irdma_pci_f *rf); | 530 | void irdma_cleanup_pending_cqp_op(struct irdma_pci_f *rf); |
| 513 | int irdma_hw_modify_qp(struct irdma_device *iwdev, struct irdma_qp *iwqp, | 531 | int irdma_hw_modify_qp(struct irdma_device *iwdev, struct irdma_qp *iwqp, |
| 514 | struct irdma_modify_qp_info *info, bool wait); | 532 | struct irdma_modify_qp_info *info, bool wait); |
| @@ -557,4 +575,5 @@ int irdma_netdevice_event(struct notifier_block *notifier, unsigned long event, · 42 unmodified lines | |||
| 557 | void *ptr); | 575 | void *ptr); |
| 558 | void irdma_add_ip(struct irdma_device *iwdev); | 576 | void irdma_add_ip(struct irdma_device *iwdev); |
| 559 | void cqp_compl_worker(struct work_struct *work); | 577 | void cqp_compl_worker(struct work_struct *work); |
| 578 | void irdma_log_invalid_mtu(u16 mtu, struct irdma_sc_dev *dev); | ||
| 560 | #endif /* IRDMA_MAIN_H */ | 579 | #endif /* IRDMA_MAIN_H */ |
| @@ -193,8 +193,15 @@ static enum irdma_sd_entry_type irdma_get_type(struct irdma_sc_dev *dev, · 192 unmodified lines | |||
| 193 | { | 193 | { |
| 194 | enum irdma_sd_entry_type sd_entry_type; | 194 | enum irdma_sd_entry_type sd_entry_type; |
| 195 | 195 | ||
| 196 | sd_entry_type = !idx->rel_pd_idx && pages == IRDMA_HMC_PD_CNT_IN_SD ? | 196 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) |
| 197 | IRDMA_SD_TYPE_DIRECT : IRDMA_SD_TYPE_PAGED; | 197 | sd_entry_type = (!idx->rel_pd_idx && |
| 198 | pages == IRDMA_HMC_PD_CNT_IN_SD) ? | ||
| 199 | IRDMA_SD_TYPE_DIRECT : IRDMA_SD_TYPE_PAGED; | ||
| 200 | else | ||
| 201 | sd_entry_type = (!idx->rel_pd_idx && | ||
| 202 | pages == IRDMA_HMC_PD_CNT_IN_SD && | ||
| 203 | dev->privileged) ? | ||
| 204 | IRDMA_SD_TYPE_DIRECT : IRDMA_SD_TYPE_PAGED; | ||
| 198 | return sd_entry_type; | 205 | return sd_entry_type; |
| 199 | } | 206 | } |
| 200 | 207 | ||
| @@ -279,10 +286,11 @@ static int add_pble_prm(struct irdma_hmc_pble_rsrc *pble_rsrc) · 78 unmodified lines | |||
| 279 | sd_reg_val = (sd_entry_type == IRDMA_SD_TYPE_PAGED) ? | 286 | sd_reg_val = (sd_entry_type == IRDMA_SD_TYPE_PAGED) ? |
| 280 | sd_entry->u.pd_table.pd_page_addr.pa : | 287 | sd_entry->u.pd_table.pd_page_addr.pa : |
| 281 | sd_entry->u.bp.addr.pa; | 288 | sd_entry->u.bp.addr.pa; |
| 282 | 289 | if ((dev->privileged && !sd_entry->valid) || | |
| 283 | if (!sd_entry->valid) { | 290 | dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) { |
| 284 | ret_code = irdma_hmc_sd_one(dev, hmc_info->hmc_fn_id, sd_reg_val, | 291 | ret_code = irdma_hmc_sd_one(dev, hmc_info->hmc_fn_id, |
| 285 | idx->sd_idx, sd_entry->entry_type, true); | 292 | sd_reg_val, idx->sd_idx, |
| 293 | sd_entry->entry_type, true); | ||
| 286 | if (ret_code) | 294 | if (ret_code) |
| 287 | goto error; | 295 | goto error; |
| 288 | } | 296 | } |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | |||
| 10 | #define ALL_TC2PFC 0xff | 10 | #define ALL_TC2PFC 0xff |
| 11 | #define CQP_COMPL_WAIT_TIME_MS 10 | 11 | #define CQP_COMPL_WAIT_TIME_MS 10 |
| 12 | #define CQP_TIMEOUT_THRESHOLD 500 | 12 | #define CQP_TIMEOUT_THRESHOLD 500 |
| 13 | #define CQP_DEF_CMPL_TIMEOUT_THRESHOLD 2500 | ||
| 13 | 14 | ||
| 14 | /* init operations */ | 15 | /* init operations */ |
| 15 | int irdma_sc_dev_init(enum irdma_vers ver, struct irdma_sc_dev *dev, | 16 | int irdma_sc_dev_init(enum irdma_vers ver, struct irdma_sc_dev *dev, |
| @@ -91,7 +91,7 @@ struct irdma_puda_rsrc_info { · 90 unmodified lines | |||
| 91 | u32 rq_size; | 91 | u32 rq_size; |
| 92 | u32 tx_buf_cnt; /* total bufs allocated will be rq_size + tx_buf_cnt */ | 92 | u32 tx_buf_cnt; /* total bufs allocated will be rq_size + tx_buf_cnt */ |
| 93 | u16 buf_size; | 93 | u16 buf_size; |
| 94 | u8 stats_idx; | 94 | u16 stats_idx; |
| 95 | bool stats_idx_valid:1; | 95 | bool stats_idx_valid:1; |
| 96 | int abi_ver; | 96 | int abi_ver; |
| 97 | }; | 97 | }; |
| @@ -140,7 +140,7 @@ struct irdma_puda_rsrc { · 42 unmodified lines | |||
| 140 | u64 crc_err; | 140 | u64 crc_err; |
| 141 | u64 pmode_count; | 141 | u64 pmode_count; |
| 142 | u64 partials_handled; | 142 | u64 partials_handled; |
| 143 | u8 stats_idx; | 143 | u16 stats_idx; |
| 144 | bool check_crc:1; | 144 | bool check_crc:1; |
| 145 | bool stats_idx_valid:1; | 145 | bool stats_idx_valid:1; |
| 146 | }; | 146 | }; |
| @@ -8,6 +8,8 @@ · 7 unmodified lines | |||
| 8 | #include "hmc.h" | 8 | #include "hmc.h" |
| 9 | #include "uda.h" | 9 | #include "uda.h" |
| 10 | #include "ws.h" | 10 | #include "ws.h" |
| 11 | #include "virtchnl.h" | ||
| 12 | |||
| 11 | #define IRDMA_DEBUG_ERR "ERR" | 13 | #define IRDMA_DEBUG_ERR "ERR" |
| 12 | #define IRDMA_DEBUG_INIT "INIT" | 14 | #define IRDMA_DEBUG_INIT "INIT" |
| 13 | #define IRDMA_DEBUG_DEV "DEV" | 15 | #define IRDMA_DEBUG_DEV "DEV" |
| @@ -95,12 +97,6 @@ enum irdma_term_mpa_errors { · 81 unmodified lines | |||
| 95 | MPA_REQ_RSP = 0x04, | 97 | MPA_REQ_RSP = 0x04, |
| 96 | }; | 98 | }; |
| 97 | 99 | ||
| 98 | enum irdma_qp_event_type { | ||
| 99 | IRDMA_QP_EVENT_CATASTROPHIC, | ||
| 100 | IRDMA_QP_EVENT_ACCESS_ERR, | ||
| 101 | IRDMA_QP_EVENT_REQ_ERR, | ||
| 102 | }; | ||
| 103 | |||
| 104 | enum irdma_hw_stats_index { | 100 | enum irdma_hw_stats_index { |
| 105 | /* gen1 - 32-bit */ | 101 | /* gen1 - 32-bit */ |
| 106 | IRDMA_HW_STAT_INDEX_IP4RXDISCARD = 0, | 102 | IRDMA_HW_STAT_INDEX_IP4RXDISCARD = 0, |
| @@ -154,12 +150,46 @@ enum irdma_hw_stats_index { · 47 unmodified lines | |||
| 154 | IRDMA_HW_STAT_INDEX_RXRPCNPIGNORED = 44, | 150 | IRDMA_HW_STAT_INDEX_RXRPCNPIGNORED = 44, |
| 155 | IRDMA_HW_STAT_INDEX_TXNPCNPSENT = 45, | 151 | IRDMA_HW_STAT_INDEX_TXNPCNPSENT = 45, |
| 156 | IRDMA_HW_STAT_INDEX_MAX_GEN_2 = 46, | 152 | IRDMA_HW_STAT_INDEX_MAX_GEN_2 = 46, |
| 153 | |||
| 154 | /* gen3 */ | ||
| 155 | IRDMA_HW_STAT_INDEX_RNR_SENT = 46, | ||
| 156 | IRDMA_HW_STAT_INDEX_RNR_RCVD = 47, | ||
| 157 | IRDMA_HW_STAT_INDEX_RDMAORDLMTCNT = 48, | ||
| 158 | IRDMA_HW_STAT_INDEX_RDMAIRDLMTCNT = 49, | ||
| 159 | IRDMA_HW_STAT_INDEX_RDMARXATS = 50, | ||
| 160 | IRDMA_HW_STAT_INDEX_RDMATXATS = 51, | ||
| 161 | IRDMA_HW_STAT_INDEX_NAKSEQERR = 52, | ||
| 162 | IRDMA_HW_STAT_INDEX_NAKSEQERR_IMPLIED = 53, | ||
| 163 | IRDMA_HW_STAT_INDEX_RTO = 54, | ||
| 164 | IRDMA_HW_STAT_INDEX_RXOOOPKTS = 55, | ||
| 165 | IRDMA_HW_STAT_INDEX_ICRCERR = 56, | ||
| 166 | |||
| 167 | IRDMA_HW_STAT_INDEX_MAX_GEN_3 = 57, | ||
| 157 | }; | 168 | }; |
| 158 | 169 | ||
| 159 | enum irdma_feature_type { | 170 | enum irdma_feature_type { |
| 160 | IRDMA_FEATURE_FW_INFO = 0, | 171 | IRDMA_FEATURE_FW_INFO = 0, |
| 161 | IRDMA_HW_VERSION_INFO = 1, | 172 | IRDMA_HW_VERSION_INFO = 1, |
| 173 | IRDMA_QP_MAX_INCR = 2, | ||
| 174 | IRDMA_CQ_MAX_INCR = 3, | ||
| 175 | IRDMA_CEQ_MAX_INCR = 4, | ||
| 176 | IRDMA_SD_MAX_INCR = 5, | ||
| 177 | IRDMA_MR_MAX_INCR = 6, | ||
| 178 | IRDMA_Q1_MAX_INCR = 7, | ||
| 179 | IRDMA_AH_MAX_INCR = 8, | ||
| 180 | IRDMA_SRQ_MAX_INCR = 9, | ||
| 181 | IRDMA_TIMER_MAX_INCR = 10, | ||
| 182 | IRDMA_XF_MAX_INCR = 11, | ||
| 183 | IRDMA_RRF_MAX_INCR = 12, | ||
| 184 | IRDMA_PBLE_MAX_INCR = 13, | ||
| 185 | IRDMA_OBJ_1 = 22, | ||
| 186 | IRDMA_OBJ_2 = 23, | ||
| 187 | IRDMA_ENDPT_TRK = 24, | ||
| 188 | IRDMA_FTN_INLINE_MAX = 25, | ||
| 162 | IRDMA_QSETS_MAX = 26, | 189 | IRDMA_QSETS_MAX = 26, |
| 190 | IRDMA_ASO = 27, | ||
| 191 | IRDMA_FTN_FLAGS = 32, | ||
| 192 | IRDMA_FTN_NOP = 33, | ||
| 163 | IRDMA_MAX_FEATURES, /* Must be last entry */ | 193 | IRDMA_MAX_FEATURES, /* Must be last entry */ |
| 164 | }; | 194 | }; |
| 165 | 195 | ||
| @@ -206,6 +236,7 @@ enum irdma_syn_rst_handling { · 40 unmodified lines | |||
| 206 | enum irdma_queue_type { | 236 | enum irdma_queue_type { |
| 207 | IRDMA_QUEUE_TYPE_SQ_RQ = 0, | 237 | IRDMA_QUEUE_TYPE_SQ_RQ = 0, |
| 208 | IRDMA_QUEUE_TYPE_CQP, | 238 | IRDMA_QUEUE_TYPE_CQP, |
| 239 | IRDMA_QUEUE_TYPE_SRQ, | ||
| 209 | }; | 240 | }; |
| 210 | 241 | ||
| 211 | struct irdma_sc_dev; | 242 | struct irdma_sc_dev; |
| @@ -233,12 +264,22 @@ struct irdma_cqp_init_info { · 21 unmodified lines | |||
| 233 | __le64 *host_ctx; | 264 | __le64 *host_ctx; |
| 234 | u64 *scratch_array; | 265 | u64 *scratch_array; |
| 235 | u32 sq_size; | 266 | u32 sq_size; |
| 267 | struct irdma_ooo_cqp_op *ooo_op_array; | ||
| 268 | u32 pe_en_vf_cnt; | ||
| 236 | u16 hw_maj_ver; | 269 | u16 hw_maj_ver; |
| 237 | u16 hw_min_ver; | 270 | u16 hw_min_ver; |
| 238 | u8 struct_ver; | 271 | u8 struct_ver; |
| 239 | u8 hmc_profile; | 272 | u8 hmc_profile; |
| 240 | u8 ena_vf_count; | 273 | u8 ena_vf_count; |
| 241 | u8 ceqs_per_vf; | 274 | u8 ceqs_per_vf; |
| 275 | u8 ooisc_blksize; | ||
| 276 | u8 rrsp_blksize; | ||
| 277 | u8 q1_blksize; | ||
| 278 | u8 xmit_blksize; | ||
| 279 | u8 ts_override; | ||
| 280 | u8 ts_shift; | ||
| 281 | u8 en_fine_grained_timers; | ||
| 282 | u8 blksizes_valid; | ||
| 242 | bool en_datacenter_tcp:1; | 283 | bool en_datacenter_tcp:1; |
| 243 | bool disable_packed:1; | 284 | bool disable_packed:1; |
| 244 | bool rocev2_rto_policy:1; | 285 | bool rocev2_rto_policy:1; |
| @@ -310,9 +351,21 @@ struct irdma_vsi_pestat { · 65 unmodified lines | |||
| 310 | spinlock_t lock; /* rdma stats lock */ | 351 | spinlock_t lock; /* rdma stats lock */ |
| 311 | }; | 352 | }; |
| 312 | 353 | ||
| 354 | struct irdma_mmio_region { | ||
| 355 | u8 __iomem *addr; | ||
| 356 | resource_size_t len; | ||
| 357 | resource_size_t offset; | ||
| 358 | }; | ||
| 359 | |||
| 313 | struct irdma_hw { | 360 | struct irdma_hw { |
| 314 | u8 __iomem *hw_addr; | 361 | union { |
| 315 | u8 __iomem *priv_hw_addr; | 362 | u8 __iomem *hw_addr; |
| 363 | struct { | ||
| 364 | struct irdma_mmio_region rdma_reg; /* RDMA region */ | ||
| 365 | struct irdma_mmio_region *io_regs; /* Non-RDMA MMIO regions */ | ||
| 366 | u16 num_io_regions; /* Number of Non-RDMA MMIO regions */ | ||
| 367 | }; | ||
| 368 | }; | ||
| 316 | struct device *device; | 369 | struct device *device; |
| 317 | struct irdma_hmc_info hmc; | 370 | struct irdma_hmc_info hmc; |
| 318 | }; | 371 | }; |
| @@ -351,7 +404,21 @@ struct irdma_cqp_quanta { · 32 unmodified lines | |||
| 351 | __le64 elem[IRDMA_CQP_WQE_SIZE]; | 404 | __le64 elem[IRDMA_CQP_WQE_SIZE]; |
| 352 | }; | 405 | }; |
| 353 | 406 | ||
| 407 | struct irdma_ooo_cqp_op { | ||
| 408 | struct list_head list_entry; | ||
| 409 | u64 scratch; | ||
| 410 | u32 def_info; | ||
| 411 | u32 sw_def_info; | ||
| 412 | u32 wqe_idx; | ||
| 413 | bool deferred:1; | ||
| 414 | }; | ||
| 415 | |||
| 354 | struct irdma_sc_cqp { | 416 | struct irdma_sc_cqp { |
| 417 | spinlock_t ooo_list_lock; /* protects list of pending completions */ | ||
| 418 | struct list_head ooo_avail; | ||
| 419 | struct list_head ooo_pnd; | ||
| 420 | u32 last_def_cmpl_ticket; | ||
| 421 | u32 sw_def_cmpl_ticket; | ||
| 355 | u32 size; | 422 | u32 size; |
| 356 | u64 sq_pa; | 423 | u64 sq_pa; |
| 357 | u64 host_ctx_pa; | 424 | u64 host_ctx_pa; |
| @@ -367,8 +434,10 @@ struct irdma_sc_cqp { · 9 unmodified lines | |||
| 367 | u64 *scratch_array; | 434 | u64 *scratch_array; |
| 368 | u64 requested_ops; | 435 | u64 requested_ops; |
| 369 | atomic64_t completed_ops; | 436 | atomic64_t completed_ops; |
| 437 | struct irdma_ooo_cqp_op *ooo_op_array; | ||
| 370 | u32 cqp_id; | 438 | u32 cqp_id; |
| 371 | u32 sq_size; | 439 | u32 sq_size; |
| 440 | u32 pe_en_vf_cnt; | ||
| 372 | u32 hw_sq_size; | 441 | u32 hw_sq_size; |
| 373 | u16 hw_maj_ver; | 442 | u16 hw_maj_ver; |
| 374 | u16 hw_min_ver; | 443 | u16 hw_min_ver; |
| @@ -378,6 +447,14 @@ struct irdma_sc_cqp { · 3 unmodified lines | |||
| 378 | u8 ena_vf_count; | 447 | u8 ena_vf_count; |
| 379 | u8 timeout_count; | 448 | u8 timeout_count; |
| 380 | u8 ceqs_per_vf; | 449 | u8 ceqs_per_vf; |
| 450 | u8 ooisc_blksize; | ||
| 451 | u8 rrsp_blksize; | ||
| 452 | u8 q1_blksize; | ||
| 453 | u8 xmit_blksize; | ||
| 454 | u8 ts_override; | ||
| 455 | u8 ts_shift; | ||
| 456 | u8 en_fine_grained_timers; | ||
| 457 | u8 blksizes_valid; | ||
| 381 | bool en_datacenter_tcp:1; | 458 | bool en_datacenter_tcp:1; |
| 382 | bool disable_packed:1; | 459 | bool disable_packed:1; |
| 383 | bool rocev2_rto_policy:1; | 460 | bool rocev2_rto_policy:1; |
| @@ -397,6 +474,8 @@ struct irdma_sc_aeq { · 13 unmodified lines | |||
| 397 | u32 msix_idx; | 474 | u32 msix_idx; |
| 398 | u8 polarity; | 475 | u8 polarity; |
| 399 | bool virtual_map:1; | 476 | bool virtual_map:1; |
| 477 | bool pasid_valid:1; | ||
| 478 | u32 pasid; | ||
| 400 | }; | 479 | }; |
| 401 | 480 | ||
| 402 | struct irdma_sc_ceq { | 481 | struct irdma_sc_ceq { |
| @@ -412,13 +491,15 @@ struct irdma_sc_ceq { · 9 unmodified lines | |||
| 412 | u8 tph_val; | 491 | u8 tph_val; |
| 413 | u32 first_pm_pbl_idx; | 492 | u32 first_pm_pbl_idx; |
| 414 | u8 polarity; | 493 | u8 polarity; |
| 415 | struct irdma_sc_vsi *vsi; | 494 | u16 vsi_idx; |
| 416 | struct irdma_sc_cq **reg_cq; | 495 | struct irdma_sc_cq **reg_cq; |
| 417 | u32 reg_cq_size; | 496 | u32 reg_cq_size; |
| 418 | spinlock_t req_cq_lock; /* protect access to reg_cq array */ | 497 | spinlock_t req_cq_lock; /* protect access to reg_cq array */ |
| 419 | bool virtual_map:1; | 498 | bool virtual_map:1; |
| 420 | bool tph_en:1; | 499 | bool tph_en:1; |
| 421 | bool itr_no_expire:1; | 500 | bool itr_no_expire:1; |
| 501 | bool pasid_valid:1; | ||
| 502 | u32 pasid; | ||
| 422 | }; | 503 | }; |
| 423 | 504 | ||
| 424 | struct irdma_sc_cq { | 505 | struct irdma_sc_cq { |
| @@ -426,6 +507,7 @@ struct irdma_sc_cq { · 1 unmodified line | |||
| 426 | u64 cq_pa; | 507 | u64 cq_pa; |
| 427 | u64 shadow_area_pa; | 508 | u64 shadow_area_pa; |
| 428 | struct irdma_sc_dev *dev; | 509 | struct irdma_sc_dev *dev; |
| 510 | u16 vsi_idx; | ||
| 429 | struct irdma_sc_vsi *vsi; | 511 | struct irdma_sc_vsi *vsi; |
| 430 | void *pbl_list; | 512 | void *pbl_list; |
| 431 | void *back_cq; | 513 | void *back_cq; |
| @@ -477,8 +559,13 @@ struct irdma_sc_qp { · 45 unmodified lines | |||
| 477 | bool virtual_map:1; | 559 | bool virtual_map:1; |
| 478 | bool flush_sq:1; | 560 | bool flush_sq:1; |
| 479 | bool flush_rq:1; | 561 | bool flush_rq:1; |
| 562 | bool err_sq_idx_valid:1; | ||
| 563 | bool err_rq_idx_valid:1; | ||
| 564 | u32 err_sq_idx; | ||
| 565 | u32 err_rq_idx; | ||
| 480 | bool sq_flush_code:1; | 566 | bool sq_flush_code:1; |
| 481 | bool rq_flush_code:1; | 567 | bool rq_flush_code:1; |
| 568 | u32 pkt_limit; | ||
| 482 | enum irdma_flush_opcode flush_code; | 569 | enum irdma_flush_opcode flush_code; |
| 483 | enum irdma_qp_event_type event_type; | 570 | enum irdma_qp_event_type event_type; |
| 484 | u8 term_flags; | 571 | u8 term_flags; |
| @@ -489,13 +576,13 @@ struct irdma_sc_qp { · 4 unmodified lines | |||
| 489 | struct irdma_stats_inst_info { | 576 | struct irdma_stats_inst_info { |
| 490 | bool use_hmc_fcn_index; | 577 | bool use_hmc_fcn_index; |
| 491 | u8 hmc_fn_id; | 578 | u8 hmc_fn_id; |
| 492 | u8 stats_idx; | 579 | u16 stats_idx; |
| 493 | }; | 580 | }; |
| 494 | 581 | ||
| 495 | struct irdma_up_info { | 582 | struct irdma_up_info { |
| 496 | u8 map[8]; | 583 | u8 map[8]; |
| 497 | u8 cnp_up_override; | 584 | u8 cnp_up_override; |
| 498 | u8 hmc_fcn_idx; | 585 | u16 hmc_fcn_idx; |
| 499 | bool use_vlan:1; | 586 | bool use_vlan:1; |
| 500 | bool use_cnp_up_override:1; | 587 | bool use_cnp_up_override:1; |
| 501 | }; | 588 | }; |
| @@ -518,6 +605,8 @@ struct irdma_ws_node_info { · 16 unmodified lines | |||
| 518 | struct irdma_hmc_fpm_misc { | 605 | struct irdma_hmc_fpm_misc { |
| 519 | u32 max_ceqs; | 606 | u32 max_ceqs; |
| 520 | u32 max_sds; | 607 | u32 max_sds; |
| 608 | u32 loc_mem_pages; | ||
| 609 | u8 ird; | ||
| 521 | u32 xf_block_size; | 610 | u32 xf_block_size; |
| 522 | u32 q1_block_size; | 611 | u32 q1_block_size; |
| 523 | u32 ht_multiplier; | 612 | u32 ht_multiplier; |
| @@ -526,6 +615,7 @@ struct irdma_hmc_fpm_misc { · 2 unmodified lines | |||
| 526 | u32 ooiscf_block_size; | 615 | u32 ooiscf_block_size; |
| 527 | }; | 616 | }; |
| 528 | 617 | ||
| 618 | #define IRDMA_VCHNL_MAX_MSG_SIZE 512 | ||
| 529 | #define IRDMA_LEAF_DEFAULT_REL_BW 64 | 619 | #define IRDMA_LEAF_DEFAULT_REL_BW 64 |
| 530 | #define IRDMA_PARENT_DEFAULT_REL_BW 1 | 620 | #define IRDMA_PARENT_DEFAULT_REL_BW 1 |
| 531 | 621 | ||
| @@ -601,19 +691,28 @@ struct irdma_sc_dev { · 69 unmodified lines | |||
| 601 | u64 cqp_cmd_stats[IRDMA_MAX_CQP_OPS]; | 691 | u64 cqp_cmd_stats[IRDMA_MAX_CQP_OPS]; |
| 602 | struct irdma_hw_attrs hw_attrs; | 692 | struct irdma_hw_attrs hw_attrs; |
| 603 | struct irdma_hmc_info *hmc_info; | 693 | struct irdma_hmc_info *hmc_info; |
| 694 | struct irdma_vchnl_rdma_caps vc_caps; | ||
| 695 | u8 vc_recv_buf[IRDMA_VCHNL_MAX_MSG_SIZE]; | ||
| 696 | u16 vc_recv_len; | ||
| 604 | struct irdma_sc_cqp *cqp; | 697 | struct irdma_sc_cqp *cqp; |
| 605 | struct irdma_sc_aeq *aeq; | 698 | struct irdma_sc_aeq *aeq; |
| 606 | struct irdma_sc_ceq *ceq[IRDMA_CEQ_MAX_COUNT]; | 699 | struct irdma_sc_ceq *ceq[IRDMA_CEQ_MAX_COUNT]; |
| 607 | struct irdma_sc_cq *ccq; | 700 | struct irdma_sc_cq *ccq; |
| 608 | const struct irdma_irq_ops *irq_ops; | 701 | const struct irdma_irq_ops *irq_ops; |
| 702 | struct irdma_qos qos[IRDMA_MAX_USER_PRIORITY]; | ||
| 609 | struct irdma_hmc_fpm_misc hmc_fpm_misc; | 703 | struct irdma_hmc_fpm_misc hmc_fpm_misc; |
| 610 | struct irdma_ws_node *ws_tree_root; | 704 | struct irdma_ws_node *ws_tree_root; |
| 611 | struct mutex ws_mutex; /* ws tree mutex */ | 705 | struct mutex ws_mutex; /* ws tree mutex */ |
| 706 | u32 vchnl_ver; | ||
| 612 | u16 num_vfs; | 707 | u16 num_vfs; |
| 613 | u8 hmc_fn_id; | 708 | u16 hmc_fn_id; |
| 614 | u8 vf_id; | 709 | u8 vf_id; |
| 710 | bool privileged:1; | ||
| 615 | bool vchnl_up:1; | 711 | bool vchnl_up:1; |
| 616 | bool ceq_valid:1; | 712 | bool ceq_valid:1; |
| 713 | bool is_pf:1; | ||
| 714 | u8 protocol_used; | ||
| 715 | struct mutex vchnl_mutex; /* mutex to synchronize RDMA virtual channel messages */ | ||
| 617 | u8 pci_rev; | 716 | u8 pci_rev; |
| 618 | int (*ws_add)(struct irdma_sc_vsi *vsi, u8 user_pri); | 717 | int (*ws_add)(struct irdma_sc_vsi *vsi, u8 user_pri); |
| 619 | void (*ws_remove)(struct irdma_sc_vsi *vsi, u8 user_pri); | 718 | void (*ws_remove)(struct irdma_sc_vsi *vsi, u8 user_pri); |
| @@ -632,6 +731,51 @@ struct irdma_modify_cq_info { · 12 unmodified lines | |||
| 632 | bool cq_resize:1; | 731 | bool cq_resize:1; |
| 633 | }; | 732 | }; |
| 634 | 733 | ||
| 734 | struct irdma_srq_init_info { | ||
| 735 | struct irdma_sc_pd *pd; | ||
| 736 | struct irdma_sc_vsi *vsi; | ||
| 737 | u64 srq_pa; | ||
| 738 | u64 shadow_area_pa; | ||
| 739 | u32 first_pm_pbl_idx; | ||
| 740 | u32 pasid; | ||
| 741 | u32 srq_size; | ||
| 742 | u16 srq_limit; | ||
| 743 | u8 pasid_valid; | ||
| 744 | u8 wqe_size; | ||
| 745 | u8 leaf_pbl_size; | ||
| 746 | u8 virtual_map; | ||
| 747 | u8 tph_en; | ||
| 748 | u8 arm_limit_event; | ||
| 749 | u8 tph_value; | ||
| 750 | u8 pbl_chunk_size; | ||
| 751 | struct irdma_srq_uk_init_info srq_uk_init_info; | ||
| 752 | }; | ||
| 753 | |||
| 754 | struct irdma_sc_srq { | ||
| 755 | struct irdma_sc_dev *dev; | ||
| 756 | struct irdma_sc_vsi *vsi; | ||
| 757 | struct irdma_sc_pd *pd; | ||
| 758 | struct irdma_srq_uk srq_uk; | ||
| 759 | void *back_srq; | ||
| 760 | u64 srq_pa; | ||
| 761 | u64 shadow_area_pa; | ||
| 762 | u32 first_pm_pbl_idx; | ||
| 763 | u32 pasid; | ||
| 764 | u32 hw_srq_size; | ||
| 765 | u16 srq_limit; | ||
| 766 | u8 pasid_valid; | ||
| 767 | u8 leaf_pbl_size; | ||
| 768 | u8 virtual_map; | ||
| 769 | u8 tph_en; | ||
| 770 | u8 arm_limit_event; | ||
| 771 | u8 tph_val; | ||
| 772 | }; | ||
| 773 | |||
| 774 | struct irdma_modify_srq_info { | ||
| 775 | u16 srq_limit; | ||
| 776 | u8 arm_limit_event; | ||
| 777 | }; | ||
| 778 | |||
| 635 | struct irdma_create_qp_info { | 779 | struct irdma_create_qp_info { |
| 636 | bool ord_valid:1; | 780 | bool ord_valid:1; |
| 637 | bool tcp_ctx_valid:1; | 781 | bool tcp_ctx_valid:1; |
| @@ -671,7 +815,8 @@ struct irdma_ccq_cqe_info { · 33 unmodified lines | |||
| 671 | u16 maj_err_code; | 815 | u16 maj_err_code; |
| 672 | u16 min_err_code; | 816 | u16 min_err_code; |
| 673 | u8 op_code; | 817 | u8 op_code; |
| 674 | bool error; | 818 | bool error:1; |
| 819 | bool pending:1; | ||
| 675 | }; | 820 | }; |
| 676 | 821 | ||
| 677 | struct irdma_dcb_app_info { | 822 | struct irdma_dcb_app_info { |
| @@ -720,7 +865,7 @@ struct irdma_vsi_init_info { · 42 unmodified lines | |||
| 720 | 865 | ||
| 721 | struct irdma_vsi_stats_info { | 866 | struct irdma_vsi_stats_info { |
| 722 | struct irdma_vsi_pestat *pestat; | 867 | struct irdma_vsi_pestat *pestat; |
| 723 | u8 fcn_id; | 868 | u16 fcn_id; |
| 724 | bool alloc_stats_inst; | 869 | bool alloc_stats_inst; |
| 725 | }; | 870 | }; |
| 726 | 871 | ||
| @@ -731,7 +876,8 @@ struct irdma_device_init_info { · 4 unmodified lines | |||
| 731 | __le64 *fpm_commit_buf; | 876 | __le64 *fpm_commit_buf; |
| 732 | struct irdma_hw *hw; | 877 | struct irdma_hw *hw; |
| 733 | void __iomem *bar0; | 878 | void __iomem *bar0; |
| 734 | u8 hmc_fn_id; | 879 | enum irdma_protocol_used protocol_used; |
| 880 | u16 hmc_fn_id; | ||
| 735 | }; | 881 | }; |
| 736 | 882 | ||
| 737 | struct irdma_ceq_init_info { | 883 | struct irdma_ceq_init_info { |
| @@ -746,8 +892,8 @@ struct irdma_ceq_init_info { · 8 unmodified lines | |||
| 746 | bool itr_no_expire:1; | 892 | bool itr_no_expire:1; |
| 747 | u8 pbl_chunk_size; | 893 | u8 pbl_chunk_size; |
| 748 | u8 tph_val; | 894 | u8 tph_val; |
| 895 | u16 vsi_idx; | ||
| 749 | u32 first_pm_pbl_idx; | 896 | u32 first_pm_pbl_idx; |
| 750 | struct irdma_sc_vsi *vsi; | ||
| 751 | struct irdma_sc_cq **reg_cq; | 897 | struct irdma_sc_cq **reg_cq; |
| 752 | u32 reg_cq_idx; | 898 | u32 reg_cq_idx; |
| 753 | }; | 899 | }; |
| @@ -807,6 +953,8 @@ struct irdma_udp_offload_info { · 53 unmodified lines | |||
| 807 | u32 cwnd; | 953 | u32 cwnd; |
| 808 | u8 rexmit_thresh; | 954 | u8 rexmit_thresh; |
| 809 | u8 rnr_nak_thresh; | 955 | u8 rnr_nak_thresh; |
| 956 | u8 rnr_nak_tmr; | ||
| 957 | u8 min_rnr_timer; | ||
| 810 | }; | 958 | }; |
| 811 | 959 | ||
| 812 | struct irdma_roce_offload_info { | 960 | struct irdma_roce_offload_info { |
| @@ -833,6 +981,7 @@ struct irdma_roce_offload_info { · 20 unmodified lines | |||
| 833 | bool dctcp_en:1; | 981 | bool dctcp_en:1; |
| 834 | bool fw_cc_enable:1; | 982 | bool fw_cc_enable:1; |
| 835 | bool use_stats_inst:1; | 983 | bool use_stats_inst:1; |
| 984 | u8 local_ack_timeout; | ||
| 836 | u16 t_high; | 985 | u16 t_high; |
| 837 | u16 t_low; | 986 | u16 t_low; |
| 838 | u8 last_byte_sent; | 987 | u8 last_byte_sent; |
| @@ -933,8 +1082,10 @@ struct irdma_qp_host_ctx_info { · 94 unmodified lines | |||
| 933 | }; | 1082 | }; |
| 934 | u32 send_cq_num; | 1083 | u32 send_cq_num; |
| 935 | u32 rcv_cq_num; | 1084 | u32 rcv_cq_num; |
| 1085 | u32 srq_id; | ||
| 936 | u32 rem_endpoint_idx; | 1086 | u32 rem_endpoint_idx; |
| 937 | u8 stats_idx; | 1087 | u16 stats_idx; |
| 1088 | bool remote_atomics_en:1; | ||
| 938 | bool srq_valid:1; | 1089 | bool srq_valid:1; |
| 939 | bool tcp_info_valid:1; | 1090 | bool tcp_info_valid:1; |
| 940 | bool iwarp_info_valid:1; | 1091 | bool iwarp_info_valid:1; |
| @@ -945,6 +1096,7 @@ struct irdma_qp_host_ctx_info { · 4 unmodified lines | |||
| 945 | struct irdma_aeqe_info { | 1096 | struct irdma_aeqe_info { |
| 946 | u64 compl_ctx; | 1097 | u64 compl_ctx; |
| 947 | u32 qp_cq_id; | 1098 | u32 qp_cq_id; |
| 1099 | u32 def_info; /* only valid for DEF_CMPL */ | ||
| 948 | u16 ae_id; | 1100 | u16 ae_id; |
| 949 | u16 wqe_idx; | 1101 | u16 wqe_idx; |
| 950 | u8 tcp_state; | 1102 | u8 tcp_state; |
| @@ -953,9 +1105,11 @@ struct irdma_aeqe_info { · 2 unmodified lines | |||
| 953 | bool cq:1; | 1105 | bool cq:1; |
| 954 | bool sq:1; | 1106 | bool sq:1; |
| 955 | bool rq:1; | 1107 | bool rq:1; |
| 1108 | bool srq:1; | ||
| 956 | bool in_rdrsp_wr:1; | 1109 | bool in_rdrsp_wr:1; |
| 957 | bool out_rdrsp:1; | 1110 | bool out_rdrsp:1; |
| 958 | bool aeqe_overflow:1; | 1111 | bool aeqe_overflow:1; |
| 1112 | bool err_rq_idx_valid:1; | ||
| 959 | u8 q2_data_written; | 1113 | u8 q2_data_written; |
| 960 | u8 ae_src; | 1114 | u8 ae_src; |
| 961 | }; | 1115 | }; |
| @@ -972,7 +1126,8 @@ struct irdma_allocate_stag_info { · 10 unmodified lines | |||
| 972 | bool use_hmc_fcn_index:1; | 1126 | bool use_hmc_fcn_index:1; |
| 973 | bool use_pf_rid:1; | 1127 | bool use_pf_rid:1; |
| 974 | bool all_memory:1; | 1128 | bool all_memory:1; |
| 975 | u8 hmc_fcn_index; | 1129 | bool remote_atomics_en:1; |
| 1130 | u16 hmc_fcn_index; | ||
| 976 | }; | 1131 | }; |
| 977 | 1132 | ||
| 978 | struct irdma_mw_alloc_info { | 1133 | struct irdma_mw_alloc_info { |
| @@ -1000,6 +1155,7 @@ struct irdma_reg_ns_stag_info { · 21 unmodified lines | |||
| 1000 | u8 hmc_fcn_index; | 1155 | u8 hmc_fcn_index; |
| 1001 | bool use_pf_rid:1; | 1156 | bool use_pf_rid:1; |
| 1002 | bool all_memory:1; | 1157 | bool all_memory:1; |
| 1158 | bool remote_atomics_en:1; | ||
| 1003 | }; | 1159 | }; |
| 1004 | 1160 | ||
| 1005 | struct irdma_fast_reg_stag_info { | 1161 | struct irdma_fast_reg_stag_info { |
| @@ -1023,6 +1179,7 @@ struct irdma_fast_reg_stag_info { · 17 unmodified lines | |||
| 1023 | u8 hmc_fcn_index; | 1179 | u8 hmc_fcn_index; |
| 1024 | bool use_pf_rid:1; | 1180 | bool use_pf_rid:1; |
| 1025 | bool defer_flag:1; | 1181 | bool defer_flag:1; |
| 1182 | bool remote_atomics_en:1; | ||
| 1026 | }; | 1183 | }; |
| 1027 | 1184 | ||
| 1028 | struct irdma_dealloc_stag_info { | 1185 | struct irdma_dealloc_stag_info { |
| @@ -1130,6 +1287,8 @@ struct irdma_cqp_manage_push_page_info { · 101 unmodified lines | |||
| 1130 | }; | 1287 | }; |
| 1131 | 1288 | ||
| 1132 | struct irdma_qp_flush_info { | 1289 | struct irdma_qp_flush_info { |
| 1290 | u32 err_sq_idx; | ||
| 1291 | u32 err_rq_idx; | ||
| 1133 | u16 sq_minor_code; | 1292 | u16 sq_minor_code; |
| 1134 | u16 sq_major_code; | 1293 | u16 sq_major_code; |
| 1135 | u16 rq_minor_code; | 1294 | u16 rq_minor_code; |
| @@ -1140,6 +1299,8 @@ struct irdma_qp_flush_info { · 4 unmodified lines | |||
| 1140 | bool rq:1; | 1299 | bool rq:1; |
| 1141 | bool userflushcode:1; | 1300 | bool userflushcode:1; |
| 1142 | bool generate_ae:1; | 1301 | bool generate_ae:1; |
| 1302 | bool err_sq_idx_valid:1; | ||
| 1303 | bool err_rq_idx_valid:1; | ||
| 1143 | }; | 1304 | }; |
| 1144 | 1305 | ||
| 1145 | struct irdma_gen_ae_info { | 1306 | struct irdma_gen_ae_info { |
| @@ -1189,6 +1350,11 @@ void irdma_sc_pd_init(struct irdma_sc_dev *dev, struct irdma_sc_pd *pd, u32 pd_i · 43 unmodified lines | |||
| 1189 | void irdma_cfg_aeq(struct irdma_sc_dev *dev, u32 idx, bool enable); | 1350 | void irdma_cfg_aeq(struct irdma_sc_dev *dev, u32 idx, bool enable); |
| 1190 | void irdma_check_cqp_progress(struct irdma_cqp_timeout *cqp_timeout, | 1351 | void irdma_check_cqp_progress(struct irdma_cqp_timeout *cqp_timeout, |
| 1191 | struct irdma_sc_dev *dev); | 1352 | struct irdma_sc_dev *dev); |
| 1353 | void irdma_sc_cqp_def_cmpl_ae_handler(struct irdma_sc_dev *dev, | ||
| 1354 | struct irdma_aeqe_info *info, | ||
| 1355 | bool first, u64 *scratch, | ||
| 1356 | u32 *sw_def_info); | ||
| 1357 | u64 irdma_sc_cqp_cleanup_handler(struct irdma_sc_dev *dev); | ||
| 1192 | int irdma_sc_cqp_create(struct irdma_sc_cqp *cqp, u16 *maj_err, u16 *min_err); | 1358 | int irdma_sc_cqp_create(struct irdma_sc_cqp *cqp, u16 *maj_err, u16 *min_err); |
| 1193 | int irdma_sc_cqp_destroy(struct irdma_sc_cqp *cqp); | 1359 | int irdma_sc_cqp_destroy(struct irdma_sc_cqp *cqp); |
| 1194 | int irdma_sc_cqp_init(struct irdma_sc_cqp *cqp, | 1360 | int irdma_sc_cqp_init(struct irdma_sc_cqp *cqp, |
| @@ -1224,6 +1390,8 @@ void irdma_sc_cq_resize(struct irdma_sc_cq *cq, struct irdma_modify_cq_info *inf · 29 unmodified lines | |||
| 1224 | int irdma_sc_static_hmc_pages_allocated(struct irdma_sc_cqp *cqp, u64 scratch, | 1390 | int irdma_sc_static_hmc_pages_allocated(struct irdma_sc_cqp *cqp, u64 scratch, |
| 1225 | u8 hmc_fn_id, bool post_sq, | 1391 | u8 hmc_fn_id, bool post_sq, |
| 1226 | bool poll_registers); | 1392 | bool poll_registers); |
| 1393 | int irdma_sc_srq_init(struct irdma_sc_srq *srq, | ||
| 1394 | struct irdma_srq_init_info *info); | ||
| 1227 | 1395 | ||
| 1228 | void sc_vsi_update_stats(struct irdma_sc_vsi *vsi); | 1396 | void sc_vsi_update_stats(struct irdma_sc_vsi *vsi); |
| 1229 | struct cqp_info { | 1397 | struct cqp_info { |
| @@ -1467,6 +1635,23 @@ struct cqp_info { · 237 unmodified lines | |||
| 1467 | struct irdma_dma_mem query_buff_mem; | 1635 | struct irdma_dma_mem query_buff_mem; |
| 1468 | u64 scratch; | 1636 | u64 scratch; |
| 1469 | } query_rdma; | 1637 | } query_rdma; |
| 1638 | |||
| 1639 | struct { | ||
| 1640 | struct irdma_sc_srq *srq; | ||
| 1641 | u64 scratch; | ||
| 1642 | } srq_create; | ||
| 1643 | |||
| 1644 | struct { | ||
| 1645 | struct irdma_sc_srq *srq; | ||
| 1646 | struct irdma_modify_srq_info info; | ||
| 1647 | u64 scratch; | ||
| 1648 | } srq_modify; | ||
| 1649 | |||
| 1650 | struct { | ||
| 1651 | struct irdma_sc_srq *srq; | ||
| 1652 | u64 scratch; | ||
| 1653 | } srq_destroy; | ||
| 1654 | |||
| 1470 | } u; | 1655 | } u; |
| 1471 | }; | 1656 | }; |
| 1472 | 1657 | ||
| @@ -78,8 +78,7 @@ · 77 unmodified lines | |||
| 78 | #define IRDMA_UDAQPC_IPID GENMASK_ULL(47, 32) | 78 | #define IRDMA_UDAQPC_IPID GENMASK_ULL(47, 32) |
| 79 | #define IRDMA_UDAQPC_SNDMSS GENMASK_ULL(29, 16) | 79 | #define IRDMA_UDAQPC_SNDMSS GENMASK_ULL(29, 16) |
| 80 | #define IRDMA_UDAQPC_VLANTAG GENMASK_ULL(15, 0) | 80 | #define IRDMA_UDAQPC_VLANTAG GENMASK_ULL(15, 0) |
| 81 | 81 | #define IRDMA_UDA_CQPSQ_MAV_PDINDEXHI GENMASK_ULL(27, 20) | |
| 82 | #define IRDMA_UDA_CQPSQ_MAV_PDINDEXHI GENMASK_ULL(21, 20) | ||
| 83 | #define IRDMA_UDA_CQPSQ_MAV_PDINDEXLO GENMASK_ULL(63, 48) | 82 | #define IRDMA_UDA_CQPSQ_MAV_PDINDEXLO GENMASK_ULL(63, 48) |
| 84 | #define IRDMA_UDA_CQPSQ_MAV_SRCMACADDRINDEX GENMASK_ULL(29, 24) | 83 | #define IRDMA_UDA_CQPSQ_MAV_SRCMACADDRINDEX GENMASK_ULL(29, 24) |
| 85 | #define IRDMA_UDA_CQPSQ_MAV_ARPINDEX GENMASK_ULL(63, 48) | 84 | #define IRDMA_UDA_CQPSQ_MAV_ARPINDEX GENMASK_ULL(63, 48) |
| @@ -94,7 +93,7 @@ · 8 unmodified lines | |||
| 94 | #define IRDMA_UDA_CQPSQ_MAV_OPCODE GENMASK_ULL(37, 32) | 93 | #define IRDMA_UDA_CQPSQ_MAV_OPCODE GENMASK_ULL(37, 32) |
| 95 | #define IRDMA_UDA_CQPSQ_MAV_DOLOOPBACKK BIT_ULL(62) | 94 | #define IRDMA_UDA_CQPSQ_MAV_DOLOOPBACKK BIT_ULL(62) |
| 96 | #define IRDMA_UDA_CQPSQ_MAV_IPV4VALID BIT_ULL(59) | 95 | #define IRDMA_UDA_CQPSQ_MAV_IPV4VALID BIT_ULL(59) |
| 97 | #define IRDMA_UDA_CQPSQ_MAV_AVIDX GENMASK_ULL(16, 0) | 96 | #define IRDMA_UDA_CQPSQ_MAV_AVIDX GENMASK_ULL(23, 0) |
| 98 | #define IRDMA_UDA_CQPSQ_MAV_INSERTVLANTAG BIT_ULL(60) | 97 | #define IRDMA_UDA_CQPSQ_MAV_INSERTVLANTAG BIT_ULL(60) |
| 99 | #define IRDMA_UDA_MGCTX_VFFLAG BIT_ULL(29) | 98 | #define IRDMA_UDA_MGCTX_VFFLAG BIT_ULL(29) |
| 100 | #define IRDMA_UDA_MGCTX_DESTPORT GENMASK_ULL(47, 32) | 99 | #define IRDMA_UDA_MGCTX_DESTPORT GENMASK_ULL(47, 32) |
| @@ -198,6 +198,26 @@ __le64 *irdma_qp_get_next_send_wqe(struct irdma_qp_uk *qp, u32 *wqe_idx, · 197 unmodified lines | |||
| 198 | return wqe; | 198 | return wqe; |
| 199 | } | 199 | } |
| 200 | 200 | ||
| 201 | __le64 *irdma_srq_get_next_recv_wqe(struct irdma_srq_uk *srq, u32 *wqe_idx) | ||
| 202 | { | ||
| 203 | int ret_code; | ||
| 204 | __le64 *wqe; | ||
| 205 | |||
| 206 | if (IRDMA_RING_FULL_ERR(srq->srq_ring)) | ||
| 207 | return NULL; | ||
| 208 | |||
| 209 | IRDMA_ATOMIC_RING_MOVE_HEAD(srq->srq_ring, *wqe_idx, ret_code); | ||
| 210 | if (ret_code) | ||
| 211 | return NULL; | ||
| 212 | |||
| 213 | if (!*wqe_idx) | ||
| 214 | srq->srwqe_polarity = !srq->srwqe_polarity; | ||
| 215 | /* rq_wqe_size_multiplier is no of 32 byte quanta in one rq wqe */ | ||
| 216 | wqe = srq->srq_base[*wqe_idx * (srq->wqe_size_multiplier)].elem; | ||
| 217 | |||
| 218 | return wqe; | ||
| 219 | } | ||
| 220 | |||
| 201 | /** | 221 | /** |
| 202 | * irdma_qp_get_next_recv_wqe - get next qp's rcv wqe | 222 | * irdma_qp_get_next_recv_wqe - get next qp's rcv wqe |
| 203 | * @qp: hw qp ptr | 223 | * @qp: hw qp ptr |
| @@ -317,6 +337,160 @@ int irdma_uk_rdma_write(struct irdma_qp_uk *qp, struct irdma_post_sq_info *info, · 113 unmodified lines | |||
| 317 | return 0; | 337 | return 0; |
| 318 | } | 338 | } |
| 319 | 339 | ||
| 340 | /** | ||
| 341 | * irdma_uk_atomic_fetch_add - atomic fetch and add operation | ||
| 342 | * @qp: hw qp ptr | ||
| 343 | * @info: post sq information | ||
| 344 | * @post_sq: flag to post sq | ||
| 345 | */ | ||
| 346 | int irdma_uk_atomic_fetch_add(struct irdma_qp_uk *qp, | ||
| 347 | struct irdma_post_sq_info *info, bool post_sq) | ||
| 348 | { | ||
| 349 | struct irdma_atomic_fetch_add *op_info; | ||
| 350 | u32 total_size = 0; | ||
| 351 | u16 quanta = 2; | ||
| 352 | u32 wqe_idx; | ||
| 353 | __le64 *wqe; | ||
| 354 | u64 hdr; | ||
| 355 | |||
| 356 | op_info = &info->op.atomic_fetch_add; | ||
| 357 | wqe = irdma_qp_get_next_send_wqe(qp, &wqe_idx, quanta, total_size, | ||
| 358 | info); | ||
| 359 | if (!wqe) | ||
| 360 | return -ENOMEM; | ||
| 361 | |||
| 362 | set_64bit_val(wqe, 0, op_info->tagged_offset); | ||
| 363 | set_64bit_val(wqe, 8, | ||
| 364 | FIELD_PREP(IRDMAQPSQ_STAG, op_info->stag)); | ||
| 365 | set_64bit_val(wqe, 16, op_info->remote_tagged_offset); | ||
| 366 | |||
| 367 | hdr = FIELD_PREP(IRDMAQPSQ_ADDFRAGCNT, 1) | | ||
| 368 | FIELD_PREP(IRDMAQPSQ_REMOTE_STAG, op_info->remote_stag) | | ||
| 369 | FIELD_PREP(IRDMAQPSQ_OPCODE, IRDMAQP_OP_ATOMIC_FETCH_ADD) | | ||
| 370 | FIELD_PREP(IRDMAQPSQ_READFENCE, info->read_fence) | | ||
| 371 | FIELD_PREP(IRDMAQPSQ_LOCALFENCE, info->local_fence) | | ||
| 372 | FIELD_PREP(IRDMAQPSQ_SIGCOMPL, info->signaled) | | ||
| 373 | FIELD_PREP(IRDMAQPSQ_VALID, qp->swqe_polarity); | ||
| 374 | |||
| 375 | set_64bit_val(wqe, 32, op_info->fetch_add_data_bytes); | ||
| 376 | set_64bit_val(wqe, 40, 0); | ||
| 377 | set_64bit_val(wqe, 48, 0); | ||
| 378 | set_64bit_val(wqe, 56, | ||
| 379 | FIELD_PREP(IRDMAQPSQ_VALID, qp->swqe_polarity)); | ||
| 380 | |||
| 381 | dma_wmb(); /* make sure WQE is populated before valid bit is set */ | ||
| 382 | |||
| 383 | set_64bit_val(wqe, 24, hdr); | ||
| 384 | |||
| 385 | if (post_sq) | ||
| 386 | irdma_uk_qp_post_wr(qp); | ||
| 387 | |||
| 388 | return 0; | ||
| 389 | } | ||
| 390 | |||
| 391 | /** | ||
| 392 | * irdma_uk_atomic_compare_swap - atomic compare and swap operation | ||
| 393 | * @qp: hw qp ptr | ||
| 394 | * @info: post sq information | ||
| 395 | * @post_sq: flag to post sq | ||
| 396 | */ | ||
| 397 | int irdma_uk_atomic_compare_swap(struct irdma_qp_uk *qp, | ||
| 398 | struct irdma_post_sq_info *info, bool post_sq) | ||
| 399 | { | ||
| 400 | struct irdma_atomic_compare_swap *op_info; | ||
| 401 | u32 total_size = 0; | ||
| 402 | u16 quanta = 2; | ||
| 403 | u32 wqe_idx; | ||
| 404 | __le64 *wqe; | ||
| 405 | u64 hdr; | ||
| 406 | |||
| 407 | op_info = &info->op.atomic_compare_swap; | ||
| 408 | wqe = irdma_qp_get_next_send_wqe(qp, &wqe_idx, quanta, total_size, | ||
| 409 | info); | ||
| 410 | if (!wqe) | ||
| 411 | return -ENOMEM; | ||
| 412 | |||
| 413 | set_64bit_val(wqe, 0, op_info->tagged_offset); | ||
| 414 | set_64bit_val(wqe, 8, | ||
| 415 | FIELD_PREP(IRDMAQPSQ_STAG, op_info->stag)); | ||
| 416 | set_64bit_val(wqe, 16, op_info->remote_tagged_offset); | ||
| 417 | |||
| 418 | hdr = FIELD_PREP(IRDMAQPSQ_ADDFRAGCNT, 1) | | ||
| 419 | FIELD_PREP(IRDMAQPSQ_REMOTE_STAG, op_info->remote_stag) | | ||
| 420 | FIELD_PREP(IRDMAQPSQ_OPCODE, IRDMAQP_OP_ATOMIC_COMPARE_SWAP_ADD) | | ||
| 421 | FIELD_PREP(IRDMAQPSQ_READFENCE, info->read_fence) | | ||
| 422 | FIELD_PREP(IRDMAQPSQ_LOCALFENCE, info->local_fence) | | ||
| 423 | FIELD_PREP(IRDMAQPSQ_SIGCOMPL, info->signaled) | | ||
| 424 | FIELD_PREP(IRDMAQPSQ_VALID, qp->swqe_polarity); | ||
| 425 | |||
| 426 | set_64bit_val(wqe, 32, op_info->swap_data_bytes); | ||
| 427 | set_64bit_val(wqe, 40, op_info->compare_data_bytes); | ||
| 428 | set_64bit_val(wqe, 48, 0); | ||
| 429 | set_64bit_val(wqe, 56, | ||
| 430 | FIELD_PREP(IRDMAQPSQ_VALID, qp->swqe_polarity)); | ||
| 431 | |||
| 432 | dma_wmb(); /* make sure WQE is populated before valid bit is set */ | ||
| 433 | |||
| 434 | set_64bit_val(wqe, 24, hdr); | ||
| 435 | |||
| 436 | if (post_sq) | ||
| 437 | irdma_uk_qp_post_wr(qp); | ||
| 438 | |||
| 439 | return 0; | ||
| 440 | } | ||
| 441 | |||
| 442 | /** | ||
| 443 | * irdma_uk_srq_post_receive - post a receive wqe to a shared rq | ||
| 444 | * @srq: shared rq ptr | ||
| 445 | * @info: post rq information | ||
| 446 | */ | ||
| 447 | int irdma_uk_srq_post_receive(struct irdma_srq_uk *srq, | ||
| 448 | struct irdma_post_rq_info *info) | ||
| 449 | { | ||
| 450 | u32 wqe_idx, i, byte_off; | ||
| 451 | u32 addl_frag_cnt; | ||
| 452 | __le64 *wqe; | ||
| 453 | u64 hdr; | ||
| 454 | |||
| 455 | if (srq->max_srq_frag_cnt < info->num_sges) | ||
| 456 | return -EINVAL; | ||
| 457 | |||
| 458 | wqe = irdma_srq_get_next_recv_wqe(srq, &wqe_idx); | ||
| 459 | if (!wqe) | ||
| 460 | return -ENOMEM; | ||
| 461 | |||
| 462 | addl_frag_cnt = info->num_sges > 1 ? info->num_sges - 1 : 0; | ||
| 463 | srq->wqe_ops.iw_set_fragment(wqe, 0, info->sg_list, | ||
| 464 | srq->srwqe_polarity); | ||
| 465 | |||
| 466 | for (i = 1, byte_off = 32; i < info->num_sges; i++) { | ||
| 467 | srq->wqe_ops.iw_set_fragment(wqe, byte_off, &info->sg_list[i], | ||
| 468 | srq->srwqe_polarity); | ||
| 469 | byte_off += 16; | ||
| 470 | } | ||
| 471 | |||
| 472 | /* if not an odd number set valid bit in next fragment */ | ||
| 473 | if (srq->uk_attrs->hw_rev >= IRDMA_GEN_2 && !(info->num_sges & 0x01) && | ||
| 474 | info->num_sges) { | ||
| 475 | srq->wqe_ops.iw_set_fragment(wqe, byte_off, NULL, | ||
| 476 | srq->srwqe_polarity); | ||
| 477 | if (srq->uk_attrs->hw_rev == IRDMA_GEN_2) | ||
| 478 | ++addl_frag_cnt; | ||
| 479 | } | ||
| 480 | |||
| 481 | set_64bit_val(wqe, 16, (u64)info->wr_id); | ||
| 482 | hdr = FIELD_PREP(IRDMAQPSQ_ADDFRAGCNT, addl_frag_cnt) | | ||
| 483 | FIELD_PREP(IRDMAQPSQ_VALID, srq->srwqe_polarity); | ||
| 484 | |||
| 485 | dma_wmb(); /* make sure WQE is populated before valid bit is set */ | ||
| 486 | |||
| 487 | set_64bit_val(wqe, 24, hdr); | ||
| 488 | |||
| 489 | set_64bit_val(srq->shadow_area, 0, (wqe_idx + 1) % srq->srq_ring.size); | ||
| 490 | |||
| 491 | return 0; | ||
| 492 | } | ||
| 493 | |||
| 320 | /** | 494 | /** |
| 321 | * irdma_uk_rdma_read - rdma read command | 495 | * irdma_uk_rdma_read - rdma read command |
| 322 | * @qp: hw qp ptr | 496 | * @qp: hw qp ptr |
| @@ -973,6 +1147,9 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq, · 650 unmodified lines | |||
| 973 | u64 comp_ctx, qword0, qword2, qword3; | 1147 | u64 comp_ctx, qword0, qword2, qword3; |
| 974 | __le64 *cqe; | 1148 | __le64 *cqe; |
| 975 | struct irdma_qp_uk *qp; | 1149 | struct irdma_qp_uk *qp; |
| 1150 | struct irdma_srq_uk *srq; | ||
| 1151 | struct qp_err_code qp_err; | ||
| 1152 | u8 is_srq; | ||
| 976 | struct irdma_ring *pring = NULL; | 1153 | struct irdma_ring *pring = NULL; |
| 977 | u32 wqe_idx; | 1154 | u32 wqe_idx; |
| 978 | int ret_code; | 1155 | int ret_code; |
| @@ -1046,21 +1223,46 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq, · 67 unmodified lines | |||
| 1046 | } | 1223 | } |
| 1047 | 1224 | ||
| 1048 | info->q_type = (u8)FIELD_GET(IRDMA_CQ_SQ, qword3); | 1225 | info->q_type = (u8)FIELD_GET(IRDMA_CQ_SQ, qword3); |
| 1226 | is_srq = (u8)FIELD_GET(IRDMA_CQ_SRQ, qword3); | ||
| 1049 | info->error = (bool)FIELD_GET(IRDMA_CQ_ERROR, qword3); | 1227 | info->error = (bool)FIELD_GET(IRDMA_CQ_ERROR, qword3); |
| 1050 | info->ipv4 = (bool)FIELD_GET(IRDMACQ_IPV4, qword3); | 1228 | info->ipv4 = (bool)FIELD_GET(IRDMACQ_IPV4, qword3); |
| 1229 | get_64bit_val(cqe, 8, &comp_ctx); | ||
| 1230 | if (is_srq) | ||
| 1231 | get_64bit_val(cqe, 40, (u64 *)&qp); | ||
| 1232 | else | ||
| 1233 | qp = (struct irdma_qp_uk *)(unsigned long)comp_ctx; | ||
| 1051 | if (info->error) { | 1234 | if (info->error) { |
| 1052 | info->major_err = FIELD_GET(IRDMA_CQ_MAJERR, qword3); | 1235 | info->major_err = FIELD_GET(IRDMA_CQ_MAJERR, qword3); |
| 1053 | info->minor_err = FIELD_GET(IRDMA_CQ_MINERR, qword3); | 1236 | info->minor_err = FIELD_GET(IRDMA_CQ_MINERR, qword3); |
| 1054 | if (info->major_err == IRDMA_FLUSH_MAJOR_ERR) { | 1237 | switch (info->major_err) { |
| 1055 | info->comp_status = IRDMA_COMPL_STATUS_FLUSHED; | 1238 | case IRDMA_SRQFLUSH_RSVD_MAJOR_ERR: |
| 1239 | qp_err = irdma_ae_to_qp_err_code(info->minor_err); | ||
| 1240 | info->minor_err = qp_err.flush_code; | ||
| 1241 | fallthrough; | ||
| 1242 | case IRDMA_FLUSH_MAJOR_ERR: | ||
| 1056 | /* Set the min error to standard flush error code for remaining cqes */ | 1243 | /* Set the min error to standard flush error code for remaining cqes */ |
| 1057 | if (info->minor_err != FLUSH_GENERAL_ERR) { | 1244 | if (info->minor_err != FLUSH_GENERAL_ERR) { |
| 1058 | qword3 &= ~IRDMA_CQ_MINERR; | 1245 | qword3 &= ~IRDMA_CQ_MINERR; |
| 1059 | qword3 |= FIELD_PREP(IRDMA_CQ_MINERR, FLUSH_GENERAL_ERR); | 1246 | qword3 |= FIELD_PREP(IRDMA_CQ_MINERR, FLUSH_GENERAL_ERR); |
| 1060 | set_64bit_val(cqe, 24, qword3); | 1247 | set_64bit_val(cqe, 24, qword3); |
| 1061 | } | 1248 | } |
| 1062 | } else { | 1249 | info->comp_status = IRDMA_COMPL_STATUS_FLUSHED; |
| 1063 | info->comp_status = IRDMA_COMPL_STATUS_UNKNOWN; | 1250 | break; |
| 1251 | default: | ||
| 1252 | #define IRDMA_CIE_SIGNATURE 0xE | ||
| 1253 | #define IRDMA_CQMAJERR_HIGH_NIBBLE GENMASK(15, 12) | ||
| 1254 | if (info->q_type == IRDMA_CQE_QTYPE_SQ && | ||
| 1255 | qp->qp_type == IRDMA_QP_TYPE_ROCE_UD && | ||
| 1256 | FIELD_GET(IRDMA_CQMAJERR_HIGH_NIBBLE, info->major_err) | ||
| 1257 | == IRDMA_CIE_SIGNATURE) { | ||
| 1258 | info->error = 0; | ||
| 1259 | info->major_err = 0; | ||
| 1260 | info->minor_err = 0; | ||
| 1261 | info->comp_status = IRDMA_COMPL_STATUS_SUCCESS; | ||
| 1262 | } else { | ||
| 1263 | info->comp_status = IRDMA_COMPL_STATUS_UNKNOWN; | ||
| 1264 | } | ||
| 1265 | break; | ||
| 1064 | } | 1266 | } |
| 1065 | } else { | 1267 | } else { |
| 1066 | info->comp_status = IRDMA_COMPL_STATUS_SUCCESS; | 1268 | info->comp_status = IRDMA_COMPL_STATUS_SUCCESS; |
| @@ -1069,7 +1271,6 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq, · 2 unmodified lines | |||
| 1069 | get_64bit_val(cqe, 0, &qword0); | 1271 | get_64bit_val(cqe, 0, &qword0); |
| 1070 | get_64bit_val(cqe, 16, &qword2); | 1272 | get_64bit_val(cqe, 16, &qword2); |
| 1071 | 1273 | ||
| 1072 | info->tcp_seq_num_rtt = (u32)FIELD_GET(IRDMACQ_TCPSEQNUMRTT, qword0); | ||
| 1073 | info->qp_id = (u32)FIELD_GET(IRDMACQ_QPID, qword2); | 1274 | info->qp_id = (u32)FIELD_GET(IRDMACQ_QPID, qword2); |
| 1074 | info->ud_src_qpn = (u32)FIELD_GET(IRDMACQ_UDSRCQPN, qword2); | 1275 | info->ud_src_qpn = (u32)FIELD_GET(IRDMACQ_UDSRCQPN, qword2); |
| 1075 | 1276 | ||
| @@ -1085,7 +1286,22 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq, · 9 unmodified lines | |||
| 1085 | info->qp_handle = (irdma_qp_handle)(unsigned long)qp; | 1286 | info->qp_handle = (irdma_qp_handle)(unsigned long)qp; |
| 1086 | info->op_type = (u8)FIELD_GET(IRDMACQ_OP, qword3); | 1287 | info->op_type = (u8)FIELD_GET(IRDMACQ_OP, qword3); |
| 1087 | 1288 | ||
| 1088 | if (info->q_type == IRDMA_CQE_QTYPE_RQ) { | 1289 | if (info->q_type == IRDMA_CQE_QTYPE_RQ && is_srq) { |
| 1290 | srq = qp->srq_uk; | ||
| 1291 | |||
| 1292 | get_64bit_val(cqe, 8, &info->wr_id); | ||
| 1293 | info->bytes_xfered = (u32)FIELD_GET(IRDMACQ_PAYLDLEN, qword0); | ||
| 1294 | |||
| 1295 | if (qword3 & IRDMACQ_STAG) { | ||
| 1296 | info->stag_invalid_set = true; | ||
| 1297 | info->inv_stag = (u32)FIELD_GET(IRDMACQ_INVSTAG, | ||
| 1298 | qword2); | ||
| 1299 | } else { | ||
| 1300 | info->stag_invalid_set = false; | ||
| 1301 | } | ||
| 1302 | IRDMA_RING_MOVE_TAIL(srq->srq_ring); | ||
| 1303 | pring = &srq->srq_ring; | ||
| 1304 | } else if (info->q_type == IRDMA_CQE_QTYPE_RQ && !is_srq) { | ||
| 1089 | u32 array_idx; | 1305 | u32 array_idx; |
| 1090 | 1306 | ||
| 1091 | array_idx = wqe_idx / qp->rq_wqe_size_multiplier; | 1307 | array_idx = wqe_idx / qp->rq_wqe_size_multiplier; |
| @@ -1180,9 +1396,15 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq, · 88 unmodified lines | |||
| 1180 | ret_code = 0; | 1396 | ret_code = 0; |
| 1181 | 1397 | ||
| 1182 | exit: | 1398 | exit: |
| 1183 | if (!ret_code && info->comp_status == IRDMA_COMPL_STATUS_FLUSHED) | 1399 | if (!ret_code && info->comp_status == IRDMA_COMPL_STATUS_FLUSHED) { |
| 1184 | if (pring && IRDMA_RING_MORE_WORK(*pring)) | 1400 | if (pring && IRDMA_RING_MORE_WORK(*pring)) |
| 1185 | move_cq_head = false; | 1401 | /* Park CQ head during a flush to generate additional CQEs |
| 1402 | * from SW for all unprocessed WQEs. For GEN3 and beyond | ||
| 1403 | * FW will generate/flush these CQEs so move to the next CQE | ||
| 1404 | */ | ||
| 1405 | move_cq_head = qp->uk_attrs->hw_rev <= IRDMA_GEN_2 ? | ||
| 1406 | false : true; | ||
| 1407 | } | ||
| 1186 | 1408 | ||
| 1187 | if (move_cq_head) { | 1409 | if (move_cq_head) { |
| 1188 | IRDMA_RING_MOVE_HEAD_NOCHECK(cq->cq_ring); | 1410 | IRDMA_RING_MOVE_HEAD_NOCHECK(cq->cq_ring); |
| @@ -1210,10 +1432,10 @@ exit: · 21 unmodified lines | |||
| 1210 | } | 1432 | } |
| 1211 | 1433 | ||
| 1212 | /** | 1434 | /** |
| 1213 | * irdma_qp_round_up - return round up qp wq depth | 1435 | * irdma_round_up_wq - return round up qp wq depth |
| 1214 | * @wqdepth: wq depth in quanta to round up | 1436 | * @wqdepth: wq depth in quanta to round up |
| 1215 | */ | 1437 | */ |
| 1216 | static int irdma_qp_round_up(u32 wqdepth) | 1438 | static int irdma_round_up_wq(u32 wqdepth) |
| 1217 | { | 1439 | { |
| 1218 | int scount = 1; | 1440 | int scount = 1; |
| 1219 | 1441 | ||
| @@ -1268,7 +1490,7 @@ int irdma_get_sqdepth(struct irdma_uk_attrs *uk_attrs, u32 sq_size, u8 shift, · 48 unmodified lines | |||
| 1268 | { | 1490 | { |
| 1269 | u32 min_size = (u32)uk_attrs->min_hw_wq_size << shift; | 1491 | u32 min_size = (u32)uk_attrs->min_hw_wq_size << shift; |
| 1270 | 1492 | ||
| 1271 | *sqdepth = irdma_qp_round_up((sq_size << shift) + IRDMA_SQ_RSVD); | 1493 | *sqdepth = irdma_round_up_wq((sq_size << shift) + IRDMA_SQ_RSVD); |
| 1272 | 1494 | ||
| 1273 | if (*sqdepth < min_size) | 1495 | if (*sqdepth < min_size) |
| 1274 | *sqdepth = min_size; | 1496 | *sqdepth = min_size; |
| @@ -1290,7 +1512,7 @@ int irdma_get_rqdepth(struct irdma_uk_attrs *uk_attrs, u32 rq_size, u8 shift, · 15 unmodified lines | |||
| 1290 | { | 1512 | { |
| 1291 | u32 min_size = (u32)uk_attrs->min_hw_wq_size << shift; | 1513 | u32 min_size = (u32)uk_attrs->min_hw_wq_size << shift; |
| 1292 | 1514 | ||
| 1293 | *rqdepth = irdma_qp_round_up((rq_size << shift) + IRDMA_RQ_RSVD); | 1515 | *rqdepth = irdma_round_up_wq((rq_size << shift) + IRDMA_RQ_RSVD); |
| 1294 | 1516 | ||
| 1295 | if (*rqdepth < min_size) | 1517 | if (*rqdepth < min_size) |
| 1296 | *rqdepth = min_size; | 1518 | *rqdepth = min_size; |
| @@ -1300,6 +1522,26 @@ int irdma_get_rqdepth(struct irdma_uk_attrs *uk_attrs, u32 rq_size, u8 shift, · 3 unmodified lines | |||
| 1300 | return 0; | 1522 | return 0; |
| 1301 | } | 1523 | } |
| 1302 | 1524 | ||
| 1525 | /* | ||
| 1526 | * irdma_get_srqdepth - get SRQ depth (quanta) | ||
| 1527 | * @uk_attrs: qp HW attributes | ||
| 1528 | * @srq_size: SRQ size | ||
| 1529 | * @shift: shift which determines size of WQE | ||
| 1530 | * @srqdepth: depth of SRQ | ||
| 1531 | */ | ||
| 1532 | int irdma_get_srqdepth(struct irdma_uk_attrs *uk_attrs, u32 srq_size, u8 shift, | ||
| 1533 | u32 *srqdepth) | ||
| 1534 | { | ||
| 1535 | *srqdepth = irdma_round_up_wq((srq_size << shift) + IRDMA_RQ_RSVD); | ||
| 1536 | |||
| 1537 | if (*srqdepth < ((u32)uk_attrs->min_hw_wq_size << shift)) | ||
| 1538 | *srqdepth = uk_attrs->min_hw_wq_size << shift; | ||
| 1539 | else if (*srqdepth > uk_attrs->max_hw_srq_quanta) | ||
| 1540 | return -EINVAL; | ||
| 1541 | |||
| 1542 | return 0; | ||
| 1543 | } | ||
| 1544 | |||
| 1303 | static const struct irdma_wqe_uk_ops iw_wqe_uk_ops = { | 1545 | static const struct irdma_wqe_uk_ops iw_wqe_uk_ops = { |
| 1304 | .iw_copy_inline_data = irdma_copy_inline_data, | 1546 | .iw_copy_inline_data = irdma_copy_inline_data, |
| 1305 | .iw_inline_data_size_to_quanta = irdma_inline_data_size_to_quanta, | 1547 | .iw_inline_data_size_to_quanta = irdma_inline_data_size_to_quanta, |
| @@ -1335,6 +1577,42 @@ static void irdma_setup_connection_wqes(struct irdma_qp_uk *qp, · 29 unmodified lines | |||
| 1335 | IRDMA_RING_MOVE_HEAD_BY_COUNT_NOCHECK(qp->initial_ring, move_cnt); | 1577 | IRDMA_RING_MOVE_HEAD_BY_COUNT_NOCHECK(qp->initial_ring, move_cnt); |
| 1336 | } | 1578 | } |
| 1337 | 1579 | ||
| 1580 | /** | ||
| 1581 | * irdma_uk_srq_init - initialize shared qp | ||
| 1582 | * @srq: hw srq (user and kernel) | ||
| 1583 | * @info: srq initialization info | ||
| 1584 | * | ||
| 1585 | * Initializes the vars used in both user and kernel mode. | ||
| 1586 | * The size of the wqe depends on number of max fragments | ||
| 1587 | * allowed. Then size of wqe * the number of wqes should be the | ||
| 1588 | * amount of memory allocated for srq. | ||
| 1589 | */ | ||
| 1590 | int irdma_uk_srq_init(struct irdma_srq_uk *srq, | ||
| 1591 | struct irdma_srq_uk_init_info *info) | ||
| 1592 | { | ||
| 1593 | u8 rqshift; | ||
| 1594 | |||
| 1595 | srq->uk_attrs = info->uk_attrs; | ||
| 1596 | if (info->max_srq_frag_cnt > srq->uk_attrs->max_hw_wq_frags) | ||
| 1597 | return -EINVAL; | ||
| 1598 | |||
| 1599 | irdma_get_wqe_shift(srq->uk_attrs, info->max_srq_frag_cnt, 0, &rqshift); | ||
| 1600 | srq->srq_caps = info->srq_caps; | ||
| 1601 | srq->srq_base = info->srq; | ||
| 1602 | srq->shadow_area = info->shadow_area; | ||
| 1603 | srq->srq_id = info->srq_id; | ||
| 1604 | srq->srwqe_polarity = 0; | ||
| 1605 | srq->srq_size = info->srq_size; | ||
| 1606 | srq->wqe_size = rqshift; | ||
| 1607 | srq->max_srq_frag_cnt = min(srq->uk_attrs->max_hw_wq_frags, | ||
| 1608 | ((u32)2 << rqshift) - 1); | ||
| 1609 | IRDMA_RING_INIT(srq->srq_ring, srq->srq_size); | ||
| 1610 | srq->wqe_size_multiplier = 1 << rqshift; | ||
| 1611 | srq->wqe_ops = iw_wqe_uk_ops; | ||
| 1612 | |||
| 1613 | return 0; | ||
| 1614 | } | ||
| 1615 | |||
| 1338 | /** | 1616 | /** |
| 1339 | * irdma_uk_calc_shift_wq - calculate WQE shift for both SQ and RQ | 1617 | * irdma_uk_calc_shift_wq - calculate WQE shift for both SQ and RQ |
| 1340 | * @ukinfo: qp initialization info | 1618 | * @ukinfo: qp initialization info |
| @@ -1461,6 +1739,7 @@ int irdma_uk_qp_init(struct irdma_qp_uk *qp, struct irdma_qp_uk_init_info *info) · 120 unmodified lines | |||
| 1461 | qp->wqe_ops = iw_wqe_uk_ops_gen_1; | 1739 | qp->wqe_ops = iw_wqe_uk_ops_gen_1; |
| 1462 | else | 1740 | else |
| 1463 | qp->wqe_ops = iw_wqe_uk_ops; | 1741 | qp->wqe_ops = iw_wqe_uk_ops; |
| 1742 | qp->srq_uk = info->srq_uk; | ||
| 1464 | return ret_code; | 1743 | return ret_code; |
| 1465 | } | 1744 | } |
| 1466 | 1745 | ||
| @@ -41,10 +41,114 @@ · 40 unmodified lines | |||
| 41 | #define IRDMA_OP_TYPE_INV_STAG 0x0a | 41 | #define IRDMA_OP_TYPE_INV_STAG 0x0a |
| 42 | #define IRDMA_OP_TYPE_RDMA_READ_INV_STAG 0x0b | 42 | #define IRDMA_OP_TYPE_RDMA_READ_INV_STAG 0x0b |
| 43 | #define IRDMA_OP_TYPE_NOP 0x0c | 43 | #define IRDMA_OP_TYPE_NOP 0x0c |
| 44 | #define IRDMA_OP_TYPE_ATOMIC_FETCH_AND_ADD 0x0f | ||
| 45 | #define IRDMA_OP_TYPE_ATOMIC_COMPARE_AND_SWAP 0x11 | ||
| 44 | #define IRDMA_OP_TYPE_REC 0x3e | 46 | #define IRDMA_OP_TYPE_REC 0x3e |
| 45 | #define IRDMA_OP_TYPE_REC_IMM 0x3f | 47 | #define IRDMA_OP_TYPE_REC_IMM 0x3f |
| 46 | 48 | ||
| 47 | #define IRDMA_FLUSH_MAJOR_ERR 1 | 49 | #define IRDMA_FLUSH_MAJOR_ERR 1 |
| 50 | #define IRDMA_SRQFLUSH_RSVD_MAJOR_ERR 0xfffe | ||
| 51 | |||
| 52 | /* Async Events codes */ | ||
| 53 | #define IRDMA_AE_AMP_UNALLOCATED_STAG 0x0102 | ||
| 54 | #define IRDMA_AE_AMP_INVALID_STAG 0x0103 | ||
| 55 | #define IRDMA_AE_AMP_BAD_QP 0x0104 | ||
| 56 | #define IRDMA_AE_AMP_BAD_PD 0x0105 | ||
| 57 | #define IRDMA_AE_AMP_BAD_STAG_KEY 0x0106 | ||
| 58 | #define IRDMA_AE_AMP_BAD_STAG_INDEX 0x0107 | ||
| 59 | #define IRDMA_AE_AMP_BOUNDS_VIOLATION 0x0108 | ||
| 60 | #define IRDMA_AE_AMP_RIGHTS_VIOLATION 0x0109 | ||
| 61 | #define IRDMA_AE_AMP_TO_WRAP 0x010a | ||
| 62 | #define IRDMA_AE_AMP_FASTREG_VALID_STAG 0x010c | ||
| 63 | #define IRDMA_AE_AMP_FASTREG_MW_STAG 0x010d | ||
| 64 | #define IRDMA_AE_AMP_FASTREG_INVALID_RIGHTS 0x010e | ||
| 65 | #define IRDMA_AE_AMP_FASTREG_INVALID_LENGTH 0x0110 | ||
| 66 | #define IRDMA_AE_AMP_INVALIDATE_SHARED 0x0111 | ||
| 67 | #define IRDMA_AE_AMP_INVALIDATE_NO_REMOTE_ACCESS_RIGHTS 0x0112 | ||
| 68 | #define IRDMA_AE_AMP_INVALIDATE_MR_WITH_BOUND_WINDOWS 0x0113 | ||
| 69 | #define IRDMA_AE_AMP_MWBIND_VALID_STAG 0x0114 | ||
| 70 | #define IRDMA_AE_AMP_MWBIND_OF_MR_STAG 0x0115 | ||
| 71 | #define IRDMA_AE_AMP_MWBIND_TO_ZERO_BASED_STAG 0x0116 | ||
| 72 | #define IRDMA_AE_AMP_MWBIND_TO_MW_STAG 0x0117 | ||
| 73 | #define IRDMA_AE_AMP_MWBIND_INVALID_RIGHTS 0x0118 | ||
| 74 | #define IRDMA_AE_AMP_MWBIND_INVALID_BOUNDS 0x0119 | ||
| 75 | #define IRDMA_AE_AMP_MWBIND_TO_INVALID_PARENT 0x011a | ||
| 76 | #define IRDMA_AE_AMP_MWBIND_BIND_DISABLED 0x011b | ||
| 77 | #define IRDMA_AE_PRIV_OPERATION_DENIED 0x011c | ||
| 78 | #define IRDMA_AE_AMP_INVALIDATE_TYPE1_MW 0x011d | ||
| 79 | #define IRDMA_AE_AMP_MWBIND_ZERO_BASED_TYPE1_MW 0x011e | ||
| 80 | #define IRDMA_AE_AMP_FASTREG_INVALID_PBL_HPS_CFG 0x011f | ||
| 81 | #define IRDMA_AE_AMP_MWBIND_WRONG_TYPE 0x0120 | ||
| 82 | #define IRDMA_AE_AMP_FASTREG_PBLE_MISMATCH 0x0121 | ||
| 83 | #define IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG 0x0132 | ||
| 84 | #define IRDMA_AE_UDA_XMIT_BAD_PD 0x0133 | ||
| 85 | #define IRDMA_AE_UDA_XMIT_DGRAM_TOO_SHORT 0x0134 | ||
| 86 | #define IRDMA_AE_UDA_L4LEN_INVALID 0x0135 | ||
| 87 | #define IRDMA_AE_BAD_CLOSE 0x0201 | ||
| 88 | #define IRDMA_AE_RDMAP_ROE_BAD_LLP_CLOSE 0x0202 | ||
| 89 | #define IRDMA_AE_CQ_OPERATION_ERROR 0x0203 | ||
| 90 | #define IRDMA_AE_RDMA_READ_WHILE_ORD_ZERO 0x0205 | ||
| 91 | #define IRDMA_AE_STAG_ZERO_INVALID 0x0206 | ||
| 92 | #define IRDMA_AE_IB_RREQ_AND_Q1_FULL 0x0207 | ||
| 93 | #define IRDMA_AE_IB_INVALID_REQUEST 0x0208 | ||
| 94 | #define IRDMA_AE_SRQ_LIMIT 0x0209 | ||
| 95 | #define IRDMA_AE_WQE_UNEXPECTED_OPCODE 0x020a | ||
| 96 | #define IRDMA_AE_WQE_INVALID_PARAMETER 0x020b | ||
| 97 | #define IRDMA_AE_WQE_INVALID_FRAG_DATA 0x020c | ||
| 98 | #define IRDMA_AE_IB_REMOTE_ACCESS_ERROR 0x020d | ||
| 99 | #define IRDMA_AE_IB_REMOTE_OP_ERROR 0x020e | ||
| 100 | #define IRDMA_AE_SRQ_CATASTROPHIC_ERROR 0x020f | ||
| 101 | #define IRDMA_AE_WQE_LSMM_TOO_LONG 0x0220 | ||
| 102 | #define IRDMA_AE_ATOMIC_ALIGNMENT 0x0221 | ||
| 103 | #define IRDMA_AE_ATOMIC_MASK 0x0222 | ||
| 104 | #define IRDMA_AE_INVALID_REQUEST 0x0223 | ||
| 105 | #define IRDMA_AE_PCIE_ATOMIC_DISABLE 0x0224 | ||
| 106 | #define IRDMA_AE_DDP_INVALID_MSN_GAP_IN_MSN 0x0301 | ||
| 107 | #define IRDMA_AE_DDP_UBE_DDP_MESSAGE_TOO_LONG_FOR_AVAILABLE_BUFFER 0x0303 | ||
| 108 | #define IRDMA_AE_DDP_UBE_INVALID_DDP_VERSION 0x0304 | ||
| 109 | #define IRDMA_AE_DDP_UBE_INVALID_MO 0x0305 | ||
| 110 | #define IRDMA_AE_DDP_UBE_INVALID_MSN_NO_BUFFER_AVAILABLE 0x0306 | ||
| 111 | #define IRDMA_AE_DDP_UBE_INVALID_QN 0x0307 | ||
| 112 | #define IRDMA_AE_DDP_NO_L_BIT 0x0308 | ||
| 113 | #define IRDMA_AE_RDMAP_ROE_INVALID_RDMAP_VERSION 0x0311 | ||
| 114 | #define IRDMA_AE_RDMAP_ROE_UNEXPECTED_OPCODE 0x0312 | ||
| 115 | #define IRDMA_AE_ROE_INVALID_RDMA_READ_REQUEST 0x0313 | ||
| 116 | #define IRDMA_AE_ROE_INVALID_RDMA_WRITE_OR_READ_RESP 0x0314 | ||
| 117 | #define IRDMA_AE_ROCE_RSP_LENGTH_ERROR 0x0316 | ||
| 118 | #define IRDMA_AE_ROCE_EMPTY_MCG 0x0380 | ||
| 119 | #define IRDMA_AE_ROCE_BAD_MC_IP_ADDR 0x0381 | ||
| 120 | #define IRDMA_AE_ROCE_BAD_MC_QPID 0x0382 | ||
| 121 | #define IRDMA_AE_MCG_QP_PROTOCOL_MISMATCH 0x0383 | ||
| 122 | #define IRDMA_AE_INVALID_ARP_ENTRY 0x0401 | ||
| 123 | #define IRDMA_AE_INVALID_TCP_OPTION_RCVD 0x0402 | ||
| 124 | #define IRDMA_AE_STALE_ARP_ENTRY 0x0403 | ||
| 125 | #define IRDMA_AE_INVALID_AH_ENTRY 0x0406 | ||
| 126 | #define IRDMA_AE_LLP_CLOSE_COMPLETE 0x0501 | ||
| 127 | #define IRDMA_AE_LLP_CONNECTION_RESET 0x0502 | ||
| 128 | #define IRDMA_AE_LLP_FIN_RECEIVED 0x0503 | ||
| 129 | #define IRDMA_AE_LLP_RECEIVED_MARKER_AND_LENGTH_FIELDS_DONT_MATCH 0x0504 | ||
| 130 | #define IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR 0x0505 | ||
| 131 | #define IRDMA_AE_LLP_SEGMENT_TOO_SMALL 0x0507 | ||
| 132 | #define IRDMA_AE_LLP_SYN_RECEIVED 0x0508 | ||
| 133 | #define IRDMA_AE_LLP_TERMINATE_RECEIVED 0x0509 | ||
| 134 | #define IRDMA_AE_LLP_TOO_MANY_RETRIES 0x050a | ||
| 135 | #define IRDMA_AE_LLP_TOO_MANY_KEEPALIVE_RETRIES 0x050b | ||
| 136 | #define IRDMA_AE_LLP_DOUBT_REACHABILITY 0x050c | ||
| 137 | #define IRDMA_AE_LLP_CONNECTION_ESTABLISHED 0x050e | ||
| 138 | #define IRDMA_AE_LLP_TOO_MANY_RNRS 0x050f | ||
| 139 | #define IRDMA_AE_RESOURCE_EXHAUSTION 0x0520 | ||
| 140 | #define IRDMA_AE_RESET_SENT 0x0601 | ||
| 141 | #define IRDMA_AE_TERMINATE_SENT 0x0602 | ||
| 142 | #define IRDMA_AE_RESET_NOT_SENT 0x0603 | ||
| 143 | #define IRDMA_AE_LCE_QP_CATASTROPHIC 0x0700 | ||
| 144 | #define IRDMA_AE_LCE_FUNCTION_CATASTROPHIC 0x0701 | ||
| 145 | #define IRDMA_AE_LCE_CQ_CATASTROPHIC 0x0702 | ||
| 146 | #define IRDMA_AE_REMOTE_QP_CATASTROPHIC 0x0703 | ||
| 147 | #define IRDMA_AE_LOCAL_QP_CATASTROPHIC 0x0704 | ||
| 148 | #define IRDMA_AE_RCE_QP_CATASTROPHIC 0x0705 | ||
| 149 | #define IRDMA_AE_QP_SUSPEND_COMPLETE 0x0900 | ||
| 150 | #define IRDMA_AE_CQP_DEFERRED_COMPLETE 0x0901 | ||
| 151 | #define IRDMA_AE_ADAPTER_CATASTROPHIC 0x0B0B | ||
| 48 | 152 | ||
| 49 | enum irdma_device_caps_const { | 153 | enum irdma_device_caps_const { |
| 50 | IRDMA_WQE_SIZE = 4, | 154 | IRDMA_WQE_SIZE = 4, |
| @@ -55,11 +159,12 @@ enum irdma_device_caps_const { · 4 unmodified lines | |||
| 55 | IRDMA_CEQE_SIZE = 1, | 159 | IRDMA_CEQE_SIZE = 1, |
| 56 | IRDMA_CQP_CTX_SIZE = 8, | 160 | IRDMA_CQP_CTX_SIZE = 8, |
| 57 | IRDMA_SHADOW_AREA_SIZE = 8, | 161 | IRDMA_SHADOW_AREA_SIZE = 8, |
| 58 | IRDMA_QUERY_FPM_BUF_SIZE = 176, | 162 | IRDMA_QUERY_FPM_BUF_SIZE = 192, |
| 59 | IRDMA_COMMIT_FPM_BUF_SIZE = 176, | 163 | IRDMA_COMMIT_FPM_BUF_SIZE = 192, |
| 60 | IRDMA_GATHER_STATS_BUF_SIZE = 1024, | 164 | IRDMA_GATHER_STATS_BUF_SIZE = 1024, |
| 61 | IRDMA_MIN_IW_QP_ID = 0, | 165 | IRDMA_MIN_IW_QP_ID = 0, |
| 62 | IRDMA_MAX_IW_QP_ID = 262143, | 166 | IRDMA_MAX_IW_QP_ID = 262143, |
| 167 | IRDMA_MIN_IW_SRQ_ID = 0, | ||
| 63 | IRDMA_MIN_CEQID = 0, | 168 | IRDMA_MIN_CEQID = 0, |
| 64 | IRDMA_MAX_CEQID = 1023, | 169 | IRDMA_MAX_CEQID = 1023, |
| 65 | IRDMA_CEQ_MAX_COUNT = IRDMA_MAX_CEQID + 1, | 170 | IRDMA_CEQ_MAX_COUNT = IRDMA_MAX_CEQID + 1, |
| @@ -67,6 +172,7 @@ enum irdma_device_caps_const { · 1 unmodified line | |||
| 67 | IRDMA_MAX_CQID = 524287, | 172 | IRDMA_MAX_CQID = 524287, |
| 68 | IRDMA_MIN_AEQ_ENTRIES = 1, | 173 | IRDMA_MIN_AEQ_ENTRIES = 1, |
| 69 | IRDMA_MAX_AEQ_ENTRIES = 524287, | 174 | IRDMA_MAX_AEQ_ENTRIES = 524287, |
| 175 | IRDMA_MAX_AEQ_ENTRIES_GEN_3 = 262144, | ||
| 70 | IRDMA_MIN_CEQ_ENTRIES = 1, | 176 | IRDMA_MIN_CEQ_ENTRIES = 1, |
| 71 | IRDMA_MAX_CEQ_ENTRIES = 262143, | 177 | IRDMA_MAX_CEQ_ENTRIES = 262143, |
| 72 | IRDMA_MIN_CQ_SIZE = 1, | 178 | IRDMA_MIN_CQ_SIZE = 1, |
| @@ -105,6 +211,13 @@ enum irdma_flush_opcode { · 32 unmodified lines | |||
| 105 | FLUSH_RETRY_EXC_ERR, | 211 | FLUSH_RETRY_EXC_ERR, |
| 106 | FLUSH_MW_BIND_ERR, | 212 | FLUSH_MW_BIND_ERR, |
| 107 | FLUSH_REM_INV_REQ_ERR, | 213 | FLUSH_REM_INV_REQ_ERR, |
| 214 | FLUSH_RNR_RETRY_EXC_ERR, | ||
| 215 | }; | ||
| 216 | |||
| 217 | enum irdma_qp_event_type { | ||
| 218 | IRDMA_QP_EVENT_CATASTROPHIC, | ||
| 219 | IRDMA_QP_EVENT_ACCESS_ERR, | ||
| 220 | IRDMA_QP_EVENT_REQ_ERR, | ||
| 108 | }; | 221 | }; |
| 109 | 222 | ||
| 110 | enum irdma_cmpl_status { | 223 | enum irdma_cmpl_status { |
| @@ -147,6 +260,8 @@ enum irdma_qp_caps { · 36 unmodified lines | |||
| 147 | IRDMA_PUSH_MODE = 8, | 260 | IRDMA_PUSH_MODE = 8, |
| 148 | }; | 261 | }; |
| 149 | 262 | ||
| 263 | struct irdma_srq_uk; | ||
| 264 | struct irdma_srq_uk_init_info; | ||
| 150 | struct irdma_qp_uk; | 265 | struct irdma_qp_uk; |
| 151 | struct irdma_cq_uk; | 266 | struct irdma_cq_uk; |
| 152 | struct irdma_qp_uk_init_info; | 267 | struct irdma_qp_uk_init_info; |
| @@ -201,6 +316,24 @@ struct irdma_bind_window { · 48 unmodified lines | |||
| 201 | bool ena_writes:1; | 316 | bool ena_writes:1; |
| 202 | irdma_stag mw_stag; | 317 | irdma_stag mw_stag; |
| 203 | bool mem_window_type_1:1; | 318 | bool mem_window_type_1:1; |
| 319 | bool remote_atomics_en:1; | ||
| 320 | }; | ||
| 321 | |||
| 322 | struct irdma_atomic_fetch_add { | ||
| 323 | u64 tagged_offset; | ||
| 324 | u64 remote_tagged_offset; | ||
| 325 | u64 fetch_add_data_bytes; | ||
| 326 | u32 stag; | ||
| 327 | u32 remote_stag; | ||
| 328 | }; | ||
| 329 | |||
| 330 | struct irdma_atomic_compare_swap { | ||
| 331 | u64 tagged_offset; | ||
| 332 | u64 remote_tagged_offset; | ||
| 333 | u64 swap_data_bytes; | ||
| 334 | u64 compare_data_bytes; | ||
| 335 | u32 stag; | ||
| 336 | u32 remote_stag; | ||
| 204 | }; | 337 | }; |
| 205 | 338 | ||
| 206 | struct irdma_inv_local_stag { | 339 | struct irdma_inv_local_stag { |
| @@ -219,6 +352,7 @@ struct irdma_post_sq_info { · 12 unmodified lines | |||
| 219 | bool report_rtt:1; | 352 | bool report_rtt:1; |
| 220 | bool udp_hdr:1; | 353 | bool udp_hdr:1; |
| 221 | bool defer_flag:1; | 354 | bool defer_flag:1; |
| 355 | bool remote_atomic_en:1; | ||
| 222 | u32 imm_data; | 356 | u32 imm_data; |
| 223 | u32 stag_to_inv; | 357 | u32 stag_to_inv; |
| 224 | union { | 358 | union { |
| @@ -227,6 +361,8 @@ struct irdma_post_sq_info { · 2 unmodified lines | |||
| 227 | struct irdma_rdma_read rdma_read; | 361 | struct irdma_rdma_read rdma_read; |
| 228 | struct irdma_bind_window bind_window; | 362 | struct irdma_bind_window bind_window; |
| 229 | struct irdma_inv_local_stag inv_local_stag; | 363 | struct irdma_inv_local_stag inv_local_stag; |
| 364 | struct irdma_atomic_fetch_add atomic_fetch_add; | ||
| 365 | struct irdma_atomic_compare_swap atomic_compare_swap; | ||
| 230 | } op; | 366 | } op; |
| 231 | }; | 367 | }; |
| 232 | 368 | ||
| @@ -255,6 +391,15 @@ struct irdma_cq_poll_info { · 22 unmodified lines | |||
| 255 | bool imm_valid:1; | 391 | bool imm_valid:1; |
| 256 | }; | 392 | }; |
| 257 | 393 | ||
| 394 | struct qp_err_code { | ||
| 395 | enum irdma_flush_opcode flush_code; | ||
| 396 | enum irdma_qp_event_type event_type; | ||
| 397 | }; | ||
| 398 | |||
| 399 | int irdma_uk_atomic_compare_swap(struct irdma_qp_uk *qp, | ||
| 400 | struct irdma_post_sq_info *info, bool post_sq); | ||
| 401 | int irdma_uk_atomic_fetch_add(struct irdma_qp_uk *qp, | ||
| 402 | struct irdma_post_sq_info *info, bool post_sq); | ||
| 258 | int irdma_uk_inline_rdma_write(struct irdma_qp_uk *qp, | 403 | int irdma_uk_inline_rdma_write(struct irdma_qp_uk *qp, |
| 259 | struct irdma_post_sq_info *info, bool post_sq); | 404 | struct irdma_post_sq_info *info, bool post_sq); |
| 260 | int irdma_uk_inline_send(struct irdma_qp_uk *qp, | 405 | int irdma_uk_inline_send(struct irdma_qp_uk *qp, |
| @@ -300,6 +445,39 @@ int irdma_uk_calc_depth_shift_sq(struct irdma_qp_uk_init_info *ukinfo, · 39 unmodified lines | |||
| 300 | u32 *sq_depth, u8 *sq_shift); | 445 | u32 *sq_depth, u8 *sq_shift); |
| 301 | int irdma_uk_calc_depth_shift_rq(struct irdma_qp_uk_init_info *ukinfo, | 446 | int irdma_uk_calc_depth_shift_rq(struct irdma_qp_uk_init_info *ukinfo, |
| 302 | u32 *rq_depth, u8 *rq_shift); | 447 | u32 *rq_depth, u8 *rq_shift); |
| 448 | int irdma_uk_srq_init(struct irdma_srq_uk *srq, | ||
| 449 | struct irdma_srq_uk_init_info *info); | ||
| 450 | int irdma_uk_srq_post_receive(struct irdma_srq_uk *srq, | ||
| 451 | struct irdma_post_rq_info *info); | ||
| 452 | |||
| 453 | struct irdma_srq_uk { | ||
| 454 | u32 srq_caps; | ||
| 455 | struct irdma_qp_quanta *srq_base; | ||
| 456 | struct irdma_uk_attrs *uk_attrs; | ||
| 457 | __le64 *shadow_area; | ||
| 458 | struct irdma_ring srq_ring; | ||
| 459 | struct irdma_ring initial_ring; | ||
| 460 | u32 srq_id; | ||
| 461 | u32 srq_size; | ||
| 462 | u32 max_srq_frag_cnt; | ||
| 463 | struct irdma_wqe_uk_ops wqe_ops; | ||
| 464 | u8 srwqe_polarity; | ||
| 465 | u8 wqe_size; | ||
| 466 | u8 wqe_size_multiplier; | ||
| 467 | u8 deferred_flag; | ||
| 468 | }; | ||
| 469 | |||
| 470 | struct irdma_srq_uk_init_info { | ||
| 471 | struct irdma_qp_quanta *srq; | ||
| 472 | struct irdma_uk_attrs *uk_attrs; | ||
| 473 | __le64 *shadow_area; | ||
| 474 | u64 *srq_wrid_array; | ||
| 475 | u32 srq_id; | ||
| 476 | u32 srq_caps; | ||
| 477 | u32 srq_size; | ||
| 478 | u32 max_srq_frag_cnt; | ||
| 479 | }; | ||
| 480 | |||
| 303 | struct irdma_sq_uk_wr_trk_info { | 481 | struct irdma_sq_uk_wr_trk_info { |
| 304 | u64 wrid; | 482 | u64 wrid; |
| 305 | u32 wr_len; | 483 | u32 wr_len; |
| @@ -344,6 +522,7 @@ struct irdma_qp_uk { · 38 unmodified lines | |||
| 344 | bool destroy_pending:1; /* Indicates the QP is being destroyed */ | 522 | bool destroy_pending:1; /* Indicates the QP is being destroyed */ |
| 345 | void *back_qp; | 523 | void *back_qp; |
| 346 | u8 dbg_rq_flushed; | 524 | u8 dbg_rq_flushed; |
| 525 | struct irdma_srq_uk *srq_uk; | ||
| 347 | u8 sq_flush_seen; | 526 | u8 sq_flush_seen; |
| 348 | u8 rq_flush_seen; | 527 | u8 rq_flush_seen; |
| 349 | }; | 528 | }; |
| @@ -383,6 +562,7 @@ struct irdma_qp_uk_init_info { · 33 unmodified lines | |||
| 383 | u8 rq_shift; | 562 | u8 rq_shift; |
| 384 | int abi_ver; | 563 | int abi_ver; |
| 385 | bool legacy_mode; | 564 | bool legacy_mode; |
| 565 | struct irdma_srq_uk *srq_uk; | ||
| 386 | }; | 566 | }; |
| 387 | 567 | ||
| 388 | struct irdma_cq_uk_init_info { | 568 | struct irdma_cq_uk_init_info { |
| @@ -398,6 +578,7 @@ struct irdma_cq_uk_init_info { · 9 unmodified lines | |||
| 398 | __le64 *irdma_qp_get_next_send_wqe(struct irdma_qp_uk *qp, u32 *wqe_idx, | 578 | __le64 *irdma_qp_get_next_send_wqe(struct irdma_qp_uk *qp, u32 *wqe_idx, |
| 399 | u16 quanta, u32 total_size, | 579 | u16 quanta, u32 total_size, |
| 400 | struct irdma_post_sq_info *info); | 580 | struct irdma_post_sq_info *info); |
| 581 | __le64 *irdma_srq_get_next_recv_wqe(struct irdma_srq_uk *srq, u32 *wqe_idx); | ||
| 401 | __le64 *irdma_qp_get_next_recv_wqe(struct irdma_qp_uk *qp, u32 *wqe_idx); | 582 | __le64 *irdma_qp_get_next_recv_wqe(struct irdma_qp_uk *qp, u32 *wqe_idx); |
| 402 | void irdma_uk_clean_cq(void *q, struct irdma_cq_uk *cq); | 583 | void irdma_uk_clean_cq(void *q, struct irdma_cq_uk *cq); |
| 403 | int irdma_nop(struct irdma_qp_uk *qp, u64 wr_id, bool signaled, bool post_sq); | 584 | int irdma_nop(struct irdma_qp_uk *qp, u64 wr_id, bool signaled, bool post_sq); |
| @@ -409,5 +590,85 @@ int irdma_get_sqdepth(struct irdma_uk_attrs *uk_attrs, u32 sq_size, u8 shift, · 5 unmodified lines | |||
| 409 | u32 *wqdepth); | 590 | u32 *wqdepth); |
| 410 | int irdma_get_rqdepth(struct irdma_uk_attrs *uk_attrs, u32 rq_size, u8 shift, | 591 | int irdma_get_rqdepth(struct irdma_uk_attrs *uk_attrs, u32 rq_size, u8 shift, |
| 411 | u32 *wqdepth); | 592 | u32 *wqdepth); |
| 593 | int irdma_get_srqdepth(struct irdma_uk_attrs *uk_attrs, u32 srq_size, u8 shift, | ||
| 594 | u32 *srqdepth); | ||
| 412 | void irdma_clr_wqes(struct irdma_qp_uk *qp, u32 qp_wqe_idx); | 595 | void irdma_clr_wqes(struct irdma_qp_uk *qp, u32 qp_wqe_idx); |
| 596 | |||
| 597 | static inline struct qp_err_code irdma_ae_to_qp_err_code(u16 ae_id) | ||
| 598 | { | ||
| 599 | struct qp_err_code qp_err = {}; | ||
| 600 | |||
| 601 | switch (ae_id) { | ||
| 602 | case IRDMA_AE_AMP_BOUNDS_VIOLATION: | ||
| 603 | case IRDMA_AE_AMP_INVALID_STAG: | ||
| 604 | case IRDMA_AE_AMP_RIGHTS_VIOLATION: | ||
| 605 | case IRDMA_AE_AMP_UNALLOCATED_STAG: | ||
| 606 | case IRDMA_AE_AMP_BAD_PD: | ||
| 607 | case IRDMA_AE_AMP_BAD_QP: | ||
| 608 | case IRDMA_AE_AMP_BAD_STAG_KEY: | ||
| 609 | case IRDMA_AE_AMP_BAD_STAG_INDEX: | ||
| 610 | case IRDMA_AE_AMP_TO_WRAP: | ||
| 611 | case IRDMA_AE_PRIV_OPERATION_DENIED: | ||
| 612 | qp_err.flush_code = FLUSH_PROT_ERR; | ||
| 613 | qp_err.event_type = IRDMA_QP_EVENT_ACCESS_ERR; | ||
| 614 | break; | ||
| 615 | case IRDMA_AE_UDA_XMIT_BAD_PD: | ||
| 616 | case IRDMA_AE_WQE_UNEXPECTED_OPCODE: | ||
| 617 | qp_err.flush_code = FLUSH_LOC_QP_OP_ERR; | ||
| 618 | qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; | ||
| 619 | break; | ||
| 620 | case IRDMA_AE_UDA_XMIT_DGRAM_TOO_SHORT: | ||
| 621 | case IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG: | ||
| 622 | case IRDMA_AE_UDA_L4LEN_INVALID: | ||
| 623 | case IRDMA_AE_DDP_UBE_INVALID_MO: | ||
| 624 | case IRDMA_AE_DDP_UBE_DDP_MESSAGE_TOO_LONG_FOR_AVAILABLE_BUFFER: | ||
| 625 | qp_err.flush_code = FLUSH_LOC_LEN_ERR; | ||
| 626 | qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; | ||
| 627 | break; | ||
| 628 | case IRDMA_AE_AMP_INVALIDATE_NO_REMOTE_ACCESS_RIGHTS: | ||
| 629 | case IRDMA_AE_IB_REMOTE_ACCESS_ERROR: | ||
| 630 | qp_err.flush_code = FLUSH_REM_ACCESS_ERR; | ||
| 631 | qp_err.event_type = IRDMA_QP_EVENT_ACCESS_ERR; | ||
| 632 | break; | ||
| 633 | case IRDMA_AE_AMP_MWBIND_INVALID_RIGHTS: | ||
| 634 | case IRDMA_AE_AMP_MWBIND_BIND_DISABLED: | ||
| 635 | case IRDMA_AE_AMP_MWBIND_INVALID_BOUNDS: | ||
| 636 | case IRDMA_AE_AMP_MWBIND_VALID_STAG: | ||
| 637 | qp_err.flush_code = FLUSH_MW_BIND_ERR; | ||
| 638 | qp_err.event_type = IRDMA_QP_EVENT_ACCESS_ERR; | ||
| 639 | break; | ||
| 640 | case IRDMA_AE_LLP_TOO_MANY_RETRIES: | ||
| 641 | qp_err.flush_code = FLUSH_RETRY_EXC_ERR; | ||
| 642 | qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; | ||
| 643 | break; | ||
| 644 | case IRDMA_AE_IB_INVALID_REQUEST: | ||
| 645 | qp_err.flush_code = FLUSH_REM_INV_REQ_ERR; | ||
| 646 | qp_err.event_type = IRDMA_QP_EVENT_REQ_ERR; | ||
| 647 | break; | ||
| 648 | case IRDMA_AE_LLP_SEGMENT_TOO_SMALL: | ||
| 649 | case IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR: | ||
| 650 | case IRDMA_AE_ROCE_RSP_LENGTH_ERROR: | ||
| 651 | case IRDMA_AE_IB_REMOTE_OP_ERROR: | ||
| 652 | qp_err.flush_code = FLUSH_REM_OP_ERR; | ||
| 653 | qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; | ||
| 654 | break; | ||
| 655 | case IRDMA_AE_LLP_TOO_MANY_RNRS: | ||
| 656 | qp_err.flush_code = FLUSH_RNR_RETRY_EXC_ERR; | ||
| 657 | qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; | ||
| 658 | break; | ||
| 659 | case IRDMA_AE_LCE_QP_CATASTROPHIC: | ||
| 660 | case IRDMA_AE_REMOTE_QP_CATASTROPHIC: | ||
| 661 | case IRDMA_AE_LOCAL_QP_CATASTROPHIC: | ||
| 662 | case IRDMA_AE_RCE_QP_CATASTROPHIC: | ||
| 663 | qp_err.flush_code = FLUSH_FATAL_ERR; | ||
| 664 | qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; | ||
| 665 | break; | ||
| 666 | default: | ||
| 667 | qp_err.flush_code = FLUSH_GENERAL_ERR; | ||
| 668 | qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; | ||
| 669 | break; | ||
| 670 | } | ||
| 671 | |||
| 672 | return qp_err; | ||
| 673 | } | ||
| 413 | #endif /* IRDMA_USER_H */ | 674 | #endif /* IRDMA_USER_H */ |
| @@ -481,6 +481,7 @@ void irdma_free_cqp_request(struct irdma_cqp *cqp, · 480 unmodified lines | |||
| 481 | WRITE_ONCE(cqp_request->request_done, false); | 481 | WRITE_ONCE(cqp_request->request_done, false); |
| 482 | cqp_request->callback_fcn = NULL; | 482 | cqp_request->callback_fcn = NULL; |
| 483 | cqp_request->waiting = false; | 483 | cqp_request->waiting = false; |
| 484 | cqp_request->pending = false; | ||
| 484 | 485 | ||
| 485 | spin_lock_irqsave(&cqp->req_lock, flags); | 486 | spin_lock_irqsave(&cqp->req_lock, flags); |
| 486 | list_add_tail(&cqp_request->list, &cqp->cqp_avail_reqs); | 487 | list_add_tail(&cqp_request->list, &cqp->cqp_avail_reqs); |
| @@ -520,6 +521,22 @@ irdma_free_pending_cqp_request(struct irdma_cqp *cqp, · 33 unmodified lines | |||
| 520 | irdma_put_cqp_request(cqp, cqp_request); | 521 | irdma_put_cqp_request(cqp, cqp_request); |
| 521 | } | 522 | } |
| 522 | 523 | ||
| 524 | /** | ||
| 525 | * irdma_cleanup_deferred_cqp_ops - clean-up cqp with no completions | ||
| 526 | * @dev: sc_dev | ||
| 527 | * @cqp: cqp | ||
| 528 | */ | ||
| 529 | static void irdma_cleanup_deferred_cqp_ops(struct irdma_sc_dev *dev, | ||
| 530 | struct irdma_cqp *cqp) | ||
| 531 | { | ||
| 532 | u64 scratch; | ||
| 533 | |||
| 534 | /* process all CQP requests with deferred/pending completions */ | ||
| 535 | while ((scratch = irdma_sc_cqp_cleanup_handler(dev))) | ||
| 536 | irdma_free_pending_cqp_request(cqp, (struct irdma_cqp_request *) | ||
| 537 | (uintptr_t)scratch); | ||
| 538 | } | ||
| 539 | |||
| 523 | /** | 540 | /** |
| 524 | * irdma_cleanup_pending_cqp_op - clean-up cqp with no | 541 | * irdma_cleanup_pending_cqp_op - clean-up cqp with no |
| 525 | * completions | 542 | * completions |
| @@ -533,6 +550,8 @@ void irdma_cleanup_pending_cqp_op(struct irdma_pci_f *rf) · 7 unmodified lines | |||
| 533 | struct cqp_cmds_info *pcmdinfo = NULL; | 550 | struct cqp_cmds_info *pcmdinfo = NULL; |
| 534 | u32 i, pending_work, wqe_idx; | 551 | u32 i, pending_work, wqe_idx; |
| 535 | 552 | ||
| 553 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) | ||
| 554 | irdma_cleanup_deferred_cqp_ops(dev, cqp); | ||
| 536 | pending_work = IRDMA_RING_USED_QUANTA(cqp->sc_cqp.sq_ring); | 555 | pending_work = IRDMA_RING_USED_QUANTA(cqp->sc_cqp.sq_ring); |
| 537 | wqe_idx = IRDMA_RING_CURRENT_TAIL(cqp->sc_cqp.sq_ring); | 556 | wqe_idx = IRDMA_RING_CURRENT_TAIL(cqp->sc_cqp.sq_ring); |
| 538 | for (i = 0; i < pending_work; i++) { | 557 | for (i = 0; i < pending_work; i++) { |
| @@ -552,6 +571,26 @@ void irdma_cleanup_pending_cqp_op(struct irdma_pci_f *rf) · 13 unmodified lines | |||
| 552 | } | 571 | } |
| 553 | } | 572 | } |
| 554 | 573 | ||
| 574 | static int irdma_get_timeout_threshold(struct irdma_sc_dev *dev) | ||
| 575 | { | ||
| 576 | u16 time_s = dev->vc_caps.cqp_timeout_s; | ||
| 577 | |||
| 578 | if (!time_s) | ||
| 579 | return CQP_TIMEOUT_THRESHOLD; | ||
| 580 | |||
| 581 | return time_s * 1000 / dev->hw_attrs.max_cqp_compl_wait_time_ms; | ||
| 582 | } | ||
| 583 | |||
| 584 | static int irdma_get_def_timeout_threshold(struct irdma_sc_dev *dev) | ||
| 585 | { | ||
| 586 | u16 time_s = dev->vc_caps.cqp_def_timeout_s; | ||
| 587 | |||
| 588 | if (!time_s) | ||
| 589 | return CQP_DEF_CMPL_TIMEOUT_THRESHOLD; | ||
| 590 | |||
| 591 | return time_s * 1000 / dev->hw_attrs.max_cqp_compl_wait_time_ms; | ||
| 592 | } | ||
| 593 | |||
| 555 | /** | 594 | /** |
| 556 | * irdma_wait_event - wait for completion | 595 | * irdma_wait_event - wait for completion |
| 557 | * @rf: RDMA PCI function | 596 | * @rf: RDMA PCI function |
| @@ -561,6 +600,7 @@ static int irdma_wait_event(struct irdma_pci_f *rf, · 3 unmodified lines | |||
| 561 | struct irdma_cqp_request *cqp_request) | 600 | struct irdma_cqp_request *cqp_request) |
| 562 | { | 601 | { |
| 563 | struct irdma_cqp_timeout cqp_timeout = {}; | 602 | struct irdma_cqp_timeout cqp_timeout = {}; |
| 603 | int timeout_threshold = irdma_get_timeout_threshold(&rf->sc_dev); | ||
| 564 | bool cqp_error = false; | 604 | bool cqp_error = false; |
| 565 | int err_code = 0; | 605 | int err_code = 0; |
| 566 | 606 | ||
| @@ -572,9 +612,17 @@ static int irdma_wait_event(struct irdma_pci_f *rf, · 5 unmodified lines | |||
| 572 | msecs_to_jiffies(CQP_COMPL_WAIT_TIME_MS))) | 612 | msecs_to_jiffies(CQP_COMPL_WAIT_TIME_MS))) |
| 573 | break; | 613 | break; |
| 574 | 614 | ||
| 615 | if (cqp_request->pending) | ||
| 616 | /* There was a deferred or pending completion | ||
| 617 | * received for this CQP request, so we need | ||
| 618 | * to wait longer than usual. | ||
| 619 | */ | ||
| 620 | timeout_threshold = | ||
| 621 | irdma_get_def_timeout_threshold(&rf->sc_dev); | ||
| 622 | |||
| 575 | irdma_check_cqp_progress(&cqp_timeout, &rf->sc_dev); | 623 | irdma_check_cqp_progress(&cqp_timeout, &rf->sc_dev); |
| 576 | 624 | ||
| 577 | if (cqp_timeout.count < CQP_TIMEOUT_THRESHOLD) | 625 | if (cqp_timeout.count < timeout_threshold) |
| 578 | continue; | 626 | continue; |
| 579 | 627 | ||
| 580 | if (!rf->reset) { | 628 | if (!rf->reset) { |
| @@ -649,6 +697,9 @@ static const char *const irdma_cqp_cmd_names[IRDMA_MAX_CQP_OPS] = { · 68 unmodified lines | |||
| 649 | [IRDMA_OP_ADD_LOCAL_MAC_ENTRY] = "Add Local MAC Entry Cmd", | 697 | [IRDMA_OP_ADD_LOCAL_MAC_ENTRY] = "Add Local MAC Entry Cmd", |
| 650 | [IRDMA_OP_DELETE_LOCAL_MAC_ENTRY] = "Delete Local MAC Entry Cmd", | 698 | [IRDMA_OP_DELETE_LOCAL_MAC_ENTRY] = "Delete Local MAC Entry Cmd", |
| 651 | [IRDMA_OP_CQ_MODIFY] = "CQ Modify Cmd", | 699 | [IRDMA_OP_CQ_MODIFY] = "CQ Modify Cmd", |
| 700 | [IRDMA_OP_SRQ_CREATE] = "Create SRQ Cmd", | ||
| 701 | [IRDMA_OP_SRQ_MODIFY] = "Modify SRQ Cmd", | ||
| 702 | [IRDMA_OP_SRQ_DESTROY] = "Destroy SRQ Cmd", | ||
| 652 | }; | 703 | }; |
| 653 | 704 | ||
| 654 | static const struct irdma_cqp_err_info irdma_noncrit_err_list[] = { | 705 | static const struct irdma_cqp_err_info irdma_noncrit_err_list[] = { |
| @@ -1065,6 +1116,26 @@ static void irdma_dealloc_push_page(struct irdma_pci_f *rf, · 410 unmodified lines | |||
| 1065 | irdma_put_cqp_request(&rf->cqp, cqp_request); | 1116 | irdma_put_cqp_request(&rf->cqp, cqp_request); |
| 1066 | } | 1117 | } |
| 1067 | 1118 | ||
| 1119 | static void irdma_free_gsi_qp_rsrc(struct irdma_qp *iwqp, u32 qp_num) | ||
| 1120 | { | ||
| 1121 | struct irdma_device *iwdev = iwqp->iwdev; | ||
| 1122 | struct irdma_pci_f *rf = iwdev->rf; | ||
| 1123 | unsigned long flags; | ||
| 1124 | |||
| 1125 | if (rf->sc_dev.hw_attrs.uk_attrs.hw_rev < IRDMA_GEN_3) | ||
| 1126 | return; | ||
| 1127 | |||
| 1128 | irdma_vchnl_req_del_vport(&rf->sc_dev, iwdev->vport_id, qp_num); | ||
| 1129 | |||
| 1130 | if (qp_num == 1) { | ||
| 1131 | spin_lock_irqsave(&rf->rsrc_lock, flags); | ||
| 1132 | rf->hwqp1_rsvd = false; | ||
| 1133 | spin_unlock_irqrestore(&rf->rsrc_lock, flags); | ||
| 1134 | } else if (qp_num > 2) { | ||
| 1135 | irdma_free_rsrc(rf, rf->allocated_qps, qp_num); | ||
| 1136 | } | ||
| 1137 | } | ||
| 1138 | |||
| 1068 | /** | 1139 | /** |
| 1069 | * irdma_free_qp_rsrc - free up memory resources for qp | 1140 | * irdma_free_qp_rsrc - free up memory resources for qp |
| 1070 | * @iwqp: qp ptr (user or kernel) | 1141 | * @iwqp: qp ptr (user or kernel) |
| @@ -1073,7 +1144,7 @@ void irdma_free_qp_rsrc(struct irdma_qp *iwqp) · 2 unmodified lines | |||
| 1073 | { | 1144 | { |
| 1074 | struct irdma_device *iwdev = iwqp->iwdev; | 1145 | struct irdma_device *iwdev = iwqp->iwdev; |
| 1075 | struct irdma_pci_f *rf = iwdev->rf; | 1146 | struct irdma_pci_f *rf = iwdev->rf; |
| 1076 | u32 qp_num = iwqp->ibqp.qp_num; | 1147 | u32 qp_num = iwqp->sc_qp.qp_uk.qp_id; |
| 1077 | 1148 | ||
| 1078 | irdma_ieq_cleanup_qp(iwdev->vsi.ieq, &iwqp->sc_qp); | 1149 | irdma_ieq_cleanup_qp(iwdev->vsi.ieq, &iwqp->sc_qp); |
| 1079 | irdma_dealloc_push_page(rf, &iwqp->sc_qp); | 1150 | irdma_dealloc_push_page(rf, &iwqp->sc_qp); |
| @@ -1083,8 +1154,12 @@ void irdma_free_qp_rsrc(struct irdma_qp *iwqp) · 3 unmodified lines | |||
| 1083 | iwqp->sc_qp.user_pri); | 1154 | iwqp->sc_qp.user_pri); |
| 1084 | } | 1155 | } |
| 1085 | 1156 | ||
| 1086 | if (qp_num > 2) | 1157 | if (iwqp->ibqp.qp_type == IB_QPT_GSI) { |
| 1087 | irdma_free_rsrc(rf, rf->allocated_qps, qp_num); | 1158 | irdma_free_gsi_qp_rsrc(iwqp, qp_num); |
| 1159 | } else { | ||
| 1160 | if (qp_num > 2) | ||
| 1161 | irdma_free_rsrc(rf, rf->allocated_qps, qp_num); | ||
| 1162 | } | ||
| 1088 | dma_free_coherent(rf->sc_dev.hw->device, iwqp->q2_ctx_mem.size, | 1163 | dma_free_coherent(rf->sc_dev.hw->device, iwqp->q2_ctx_mem.size, |
| 1089 | iwqp->q2_ctx_mem.va, iwqp->q2_ctx_mem.pa); | 1164 | iwqp->q2_ctx_mem.va, iwqp->q2_ctx_mem.pa); |
| 1090 | iwqp->q2_ctx_mem.va = NULL; | 1165 | iwqp->q2_ctx_mem.va = NULL; |
| @@ -1095,6 +1170,30 @@ void irdma_free_qp_rsrc(struct irdma_qp *iwqp) · 4 unmodified lines | |||
| 1095 | kfree(iwqp->kqp.rq_wrid_mem); | 1170 | kfree(iwqp->kqp.rq_wrid_mem); |
| 1096 | } | 1171 | } |
| 1097 | 1172 | ||
| 1173 | /** | ||
| 1174 | * irdma_srq_wq_destroy - send srq destroy cqp | ||
| 1175 | * @rf: RDMA PCI function | ||
| 1176 | * @srq: hardware control srq | ||
| 1177 | */ | ||
| 1178 | void irdma_srq_wq_destroy(struct irdma_pci_f *rf, struct irdma_sc_srq *srq) | ||
| 1179 | { | ||
| 1180 | struct irdma_cqp_request *cqp_request; | ||
| 1181 | struct cqp_cmds_info *cqp_info; | ||
| 1182 | |||
| 1183 | cqp_request = irdma_alloc_and_get_cqp_request(&rf->cqp, true); | ||
| 1184 | if (!cqp_request) | ||
| 1185 | return; | ||
| 1186 | |||
| 1187 | cqp_info = &cqp_request->info; | ||
| 1188 | cqp_info->cqp_cmd = IRDMA_OP_SRQ_DESTROY; | ||
| 1189 | cqp_info->post_sq = 1; | ||
| 1190 | cqp_info->in.u.srq_destroy.srq = srq; | ||
| 1191 | cqp_info->in.u.srq_destroy.scratch = (uintptr_t)cqp_request; | ||
| 1192 | |||
| 1193 | irdma_handle_cqp_op(rf, cqp_request); | ||
| 1194 | irdma_put_cqp_request(&rf->cqp, cqp_request); | ||
| 1195 | } | ||
| 1196 | |||
| 1098 | /** | 1197 | /** |
| 1099 | * irdma_cq_wq_destroy - send cq destroy cqp | 1198 | * irdma_cq_wq_destroy - send cq destroy cqp |
| 1100 | * @rf: RDMA PCI function | 1199 | * @rf: RDMA PCI function |
| @@ -2266,7 +2365,10 @@ bool irdma_cq_empty(struct irdma_cq *iwcq) · 1165 unmodified lines | |||
| 2266 | u8 polarity; | 2365 | u8 polarity; |
| 2267 | 2366 | ||
| 2268 | ukcq = &iwcq->sc_cq.cq_uk; | 2367 | ukcq = &iwcq->sc_cq.cq_uk; |
| 2269 | cqe = IRDMA_GET_CURRENT_CQ_ELEM(ukcq); | 2368 | if (ukcq->avoid_mem_cflct) |
| 2369 | cqe = IRDMA_GET_CURRENT_EXTENDED_CQ_ELEM(ukcq); | ||
| 2370 | else | ||
| 2371 | cqe = IRDMA_GET_CURRENT_CQ_ELEM(ukcq); | ||
| 2270 | get_64bit_val(cqe, 24, &qword3); | 2372 | get_64bit_val(cqe, 24, &qword3); |
| 2271 | polarity = (u8)FIELD_GET(IRDMA_CQ_VALID, qword3); | 2373 | polarity = (u8)FIELD_GET(IRDMA_CQ_VALID, qword3); |
| 2272 | 2374 | ||
| @@ -41,7 +41,8 @@ static int irdma_query_device(struct ib_device *ibdev, · 40 unmodified lines | |||
| 41 | props->max_cq = rf->max_cq - rf->used_cqs; | 41 | props->max_cq = rf->max_cq - rf->used_cqs; |
| 42 | props->max_cqe = rf->max_cqe - 1; | 42 | props->max_cqe = rf->max_cqe - 1; |
| 43 | props->max_mr = rf->max_mr - rf->used_mrs; | 43 | props->max_mr = rf->max_mr - rf->used_mrs; |
| 44 | props->max_mw = props->max_mr; | 44 | if (hw_attrs->uk_attrs.hw_rev >= IRDMA_GEN_3) |
| 45 | props->max_mw = props->max_mr; | ||
| 45 | props->max_pd = rf->max_pd - rf->used_pds; | 46 | props->max_pd = rf->max_pd - rf->used_pds; |
| 46 | props->max_sge_rd = hw_attrs->uk_attrs.max_hw_read_sges; | 47 | props->max_sge_rd = hw_attrs->uk_attrs.max_hw_read_sges; |
| 47 | props->max_qp_rd_atom = hw_attrs->max_hw_ird; | 48 | props->max_qp_rd_atom = hw_attrs->max_hw_ird; |
| @@ -56,9 +57,21 @@ static int irdma_query_device(struct ib_device *ibdev, · 8 unmodified lines | |||
| 56 | props->max_mcast_qp_attach = IRDMA_MAX_MGS_PER_CTX; | 57 | props->max_mcast_qp_attach = IRDMA_MAX_MGS_PER_CTX; |
| 57 | props->max_total_mcast_qp_attach = rf->max_qp * IRDMA_MAX_MGS_PER_CTX; | 58 | props->max_total_mcast_qp_attach = rf->max_qp * IRDMA_MAX_MGS_PER_CTX; |
| 58 | props->max_fast_reg_page_list_len = IRDMA_MAX_PAGES_PER_FMR; | 59 | props->max_fast_reg_page_list_len = IRDMA_MAX_PAGES_PER_FMR; |
| 59 | #define HCA_CLOCK_TIMESTAMP_MASK 0x1ffff | 60 | props->max_srq = rf->max_srq - rf->used_srqs; |
| 60 | if (hw_attrs->uk_attrs.hw_rev >= IRDMA_GEN_2) | 61 | props->max_srq_wr = IRDMA_MAX_SRQ_WRS; |
| 61 | props->timestamp_mask = HCA_CLOCK_TIMESTAMP_MASK; | 62 | props->max_srq_sge = hw_attrs->uk_attrs.max_hw_wq_frags; |
| 63 | if (hw_attrs->uk_attrs.feature_flags & IRDMA_FEATURE_ATOMIC_OPS) | ||
| 64 | props->atomic_cap = IB_ATOMIC_HCA; | ||
| 65 | else | ||
| 66 | props->atomic_cap = IB_ATOMIC_NONE; | ||
| 67 | props->masked_atomic_cap = props->atomic_cap; | ||
| 68 | if (hw_attrs->uk_attrs.hw_rev >= IRDMA_GEN_3) { | ||
| 69 | #define HCA_CORE_CLOCK_KHZ 1000000UL | ||
| 70 | props->timestamp_mask = GENMASK(31, 0); | ||
| 71 | props->hca_core_clock = HCA_CORE_CLOCK_KHZ; | ||
| 72 | } | ||
| 73 | if (hw_attrs->uk_attrs.hw_rev >= IRDMA_GEN_3) | ||
| 74 | props->device_cap_flags |= IB_DEVICE_MEM_WINDOW_TYPE_2B; | ||
| 62 | 75 | ||
| 63 | return 0; | 76 | return 0; |
| 64 | } | 77 | } |
| @@ -292,6 +305,10 @@ static int irdma_alloc_ucontext(struct ib_ucontext *uctx, · 227 unmodified lines | |||
| 292 | ucontext->iwdev = iwdev; | 305 | ucontext->iwdev = iwdev; |
| 293 | ucontext->abi_ver = req.userspace_ver; | 306 | ucontext->abi_ver = req.userspace_ver; |
| 294 | 307 | ||
| 308 | if (!(req.comp_mask & IRDMA_SUPPORT_WQE_FORMAT_V2) && | ||
| 309 | uk_attrs->hw_rev >= IRDMA_GEN_3) | ||
| 310 | return -EOPNOTSUPP; | ||
| 311 | |||
| 295 | if (req.comp_mask & IRDMA_ALLOC_UCTX_USE_RAW_ATTR) | 312 | if (req.comp_mask & IRDMA_ALLOC_UCTX_USE_RAW_ATTR) |
| 296 | ucontext->use_raw_attrs = true; | 313 | ucontext->use_raw_attrs = true; |
| 297 | 314 | ||
| @@ -332,6 +349,8 @@ static int irdma_alloc_ucontext(struct ib_ucontext *uctx, · 34 unmodified lines | |||
| 332 | uresp.comp_mask |= IRDMA_ALLOC_UCTX_USE_RAW_ATTR; | 349 | uresp.comp_mask |= IRDMA_ALLOC_UCTX_USE_RAW_ATTR; |
| 333 | uresp.min_hw_wq_size = uk_attrs->min_hw_wq_size; | 350 | uresp.min_hw_wq_size = uk_attrs->min_hw_wq_size; |
| 334 | uresp.comp_mask |= IRDMA_ALLOC_UCTX_MIN_HW_WQ_SIZE; | 351 | uresp.comp_mask |= IRDMA_ALLOC_UCTX_MIN_HW_WQ_SIZE; |
| 352 | uresp.max_hw_srq_quanta = uk_attrs->max_hw_srq_quanta; | ||
| 353 | uresp.comp_mask |= IRDMA_ALLOC_UCTX_MAX_HW_SRQ_QUANTA; | ||
| 335 | if (ib_copy_to_udata(udata, &uresp, | 354 | if (ib_copy_to_udata(udata, &uresp, |
| 336 | min(sizeof(uresp), udata->outlen))) { | 355 | min(sizeof(uresp), udata->outlen))) { |
| 337 | rdma_user_mmap_entry_remove(ucontext->db_mmap_entry); | 356 | rdma_user_mmap_entry_remove(ucontext->db_mmap_entry); |
| @@ -343,6 +362,8 @@ static int irdma_alloc_ucontext(struct ib_ucontext *uctx, · 5 unmodified lines | |||
| 343 | spin_lock_init(&ucontext->cq_reg_mem_list_lock); | 362 | spin_lock_init(&ucontext->cq_reg_mem_list_lock); |
| 344 | INIT_LIST_HEAD(&ucontext->qp_reg_mem_list); | 363 | INIT_LIST_HEAD(&ucontext->qp_reg_mem_list); |
| 345 | spin_lock_init(&ucontext->qp_reg_mem_list_lock); | 364 | spin_lock_init(&ucontext->qp_reg_mem_list_lock); |
| 365 | INIT_LIST_HEAD(&ucontext->srq_reg_mem_list); | ||
| 366 | spin_lock_init(&ucontext->srq_reg_mem_list_lock); | ||
| 346 | 367 | ||
| 347 | return 0; | 368 | return 0; |
| 348 | 369 | ||
| @@ -521,7 +542,7 @@ static int irdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) · 172 unmodified lines | |||
| 521 | 542 | ||
| 522 | iwqp->sc_qp.qp_uk.destroy_pending = true; | 543 | iwqp->sc_qp.qp_uk.destroy_pending = true; |
| 523 | 544 | ||
| 524 | if (iwqp->iwarp_state == IRDMA_QP_STATE_RTS) | 545 | if (iwqp->iwarp_state >= IRDMA_QP_STATE_IDLE) |
| 525 | irdma_modify_qp_to_err(&iwqp->sc_qp); | 546 | irdma_modify_qp_to_err(&iwqp->sc_qp); |
| 526 | 547 | ||
| 527 | if (!iwqp->user_mode) | 548 | if (!iwqp->user_mode) |
| @@ -541,6 +562,9 @@ static int irdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) · 13 unmodified lines | |||
| 541 | irdma_cqp_qp_destroy_cmd(&iwdev->rf->sc_dev, &iwqp->sc_qp); | 562 | irdma_cqp_qp_destroy_cmd(&iwdev->rf->sc_dev, &iwqp->sc_qp); |
| 542 | 563 | ||
| 543 | irdma_remove_push_mmap_entries(iwqp); | 564 | irdma_remove_push_mmap_entries(iwqp); |
| 565 | |||
| 566 | if (iwqp->sc_qp.qp_uk.qp_id == 1) | ||
| 567 | iwdev->rf->hwqp1_rsvd = false; | ||
| 544 | irdma_free_qp_rsrc(iwqp); | 568 | irdma_free_qp_rsrc(iwqp); |
| 545 | 569 | ||
| 546 | return 0; | 570 | return 0; |
| @@ -564,7 +588,11 @@ static void irdma_setup_virt_qp(struct irdma_device *iwdev, · 17 unmodified lines | |||
| 564 | if (iwpbl->pbl_allocated) { | 588 | if (iwpbl->pbl_allocated) { |
| 565 | init_info->virtual_map = true; | 589 | init_info->virtual_map = true; |
| 566 | init_info->sq_pa = qpmr->sq_pbl.idx; | 590 | init_info->sq_pa = qpmr->sq_pbl.idx; |
| 567 | init_info->rq_pa = qpmr->rq_pbl.idx; | 591 | /* Need to use contiguous buffer for RQ of QP |
| 592 | * in case it is associated with SRQ. | ||
| 593 | */ | ||
| 594 | init_info->rq_pa = init_info->qp_uk_init_info.srq_uk ? | ||
| 595 | qpmr->rq_pa : qpmr->rq_pbl.idx; | ||
| 568 | } else { | 596 | } else { |
| 569 | init_info->sq_pa = qpmr->sq_pbl.addr; | 597 | init_info->sq_pa = qpmr->sq_pbl.addr; |
| 570 | init_info->rq_pa = qpmr->rq_pbl.addr; | 598 | init_info->rq_pa = qpmr->rq_pbl.addr; |
| @@ -719,6 +747,7 @@ static int irdma_setup_kmode_qp(struct irdma_device *iwdev, · 148 unmodified lines | |||
| 719 | info->rq_pa + (ukinfo->rq_depth * IRDMA_QP_WQE_MIN_SIZE); | 747 | info->rq_pa + (ukinfo->rq_depth * IRDMA_QP_WQE_MIN_SIZE); |
| 720 | ukinfo->sq_size = ukinfo->sq_depth >> ukinfo->sq_shift; | 748 | ukinfo->sq_size = ukinfo->sq_depth >> ukinfo->sq_shift; |
| 721 | ukinfo->rq_size = ukinfo->rq_depth >> ukinfo->rq_shift; | 749 | ukinfo->rq_size = ukinfo->rq_depth >> ukinfo->rq_shift; |
| 750 | ukinfo->qp_id = info->qp_uk_init_info.qp_id; | ||
| 722 | 751 | ||
| 723 | iwqp->max_send_wr = (ukinfo->sq_depth - IRDMA_SQ_RSVD) >> ukinfo->sq_shift; | 752 | iwqp->max_send_wr = (ukinfo->sq_depth - IRDMA_SQ_RSVD) >> ukinfo->sq_shift; |
| 724 | iwqp->max_recv_wr = (ukinfo->rq_depth - IRDMA_RQ_RSVD) >> ukinfo->rq_shift; | 753 | iwqp->max_recv_wr = (ukinfo->rq_depth - IRDMA_RQ_RSVD) >> ukinfo->rq_shift; |
| @@ -775,9 +804,12 @@ static void irdma_roce_fill_and_set_qpctx_info(struct irdma_qp *iwqp, · 50 unmodified lines | |||
| 775 | roce_info = &iwqp->roce_info; | 804 | roce_info = &iwqp->roce_info; |
| 776 | ether_addr_copy(roce_info->mac_addr, iwdev->netdev->dev_addr); | 805 | ether_addr_copy(roce_info->mac_addr, iwdev->netdev->dev_addr); |
| 777 | 806 | ||
| 807 | if (iwqp->ibqp.qp_type == IB_QPT_GSI && iwqp->ibqp.qp_num != 1) | ||
| 808 | roce_info->is_qp1 = true; | ||
| 778 | roce_info->rd_en = true; | 809 | roce_info->rd_en = true; |
| 779 | roce_info->wr_rdresp_en = true; | 810 | roce_info->wr_rdresp_en = true; |
| 780 | roce_info->bind_en = true; | 811 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) |
| 812 | roce_info->bind_en = true; | ||
| 781 | roce_info->dcqcn_en = false; | 813 | roce_info->dcqcn_en = false; |
| 782 | roce_info->rtomin = 5; | 814 | roce_info->rtomin = 5; |
| 783 | 815 | ||
| @@ -808,7 +840,6 @@ static void irdma_iw_fill_and_set_qpctx_info(struct irdma_qp *iwqp, · 24 unmodified lines | |||
| 808 | ether_addr_copy(iwarp_info->mac_addr, iwdev->netdev->dev_addr); | 840 | ether_addr_copy(iwarp_info->mac_addr, iwdev->netdev->dev_addr); |
| 809 | iwarp_info->rd_en = true; | 841 | iwarp_info->rd_en = true; |
| 810 | iwarp_info->wr_rdresp_en = true; | 842 | iwarp_info->wr_rdresp_en = true; |
| 811 | iwarp_info->bind_en = true; | ||
| 812 | iwarp_info->ecn_en = true; | 843 | iwarp_info->ecn_en = true; |
| 813 | iwarp_info->rtomin = 5; | 844 | iwarp_info->rtomin = 5; |
| 814 | 845 | ||
| @@ -864,6 +895,47 @@ static void irdma_flush_worker(struct work_struct *work) · 49 unmodified lines | |||
| 864 | irdma_generate_flush_completions(iwqp); | 895 | irdma_generate_flush_completions(iwqp); |
| 865 | } | 896 | } |
| 866 | 897 | ||
| 898 | static int irdma_setup_gsi_qp_rsrc(struct irdma_qp *iwqp, u32 *qp_num) | ||
| 899 | { | ||
| 900 | struct irdma_device *iwdev = iwqp->iwdev; | ||
| 901 | struct irdma_pci_f *rf = iwdev->rf; | ||
| 902 | unsigned long flags; | ||
| 903 | int ret; | ||
| 904 | |||
| 905 | if (rf->rdma_ver <= IRDMA_GEN_2) { | ||
| 906 | *qp_num = 1; | ||
| 907 | return 0; | ||
| 908 | } | ||
| 909 | |||
| 910 | spin_lock_irqsave(&rf->rsrc_lock, flags); | ||
| 911 | if (!rf->hwqp1_rsvd) { | ||
| 912 | *qp_num = 1; | ||
| 913 | rf->hwqp1_rsvd = true; | ||
| 914 | spin_unlock_irqrestore(&rf->rsrc_lock, flags); | ||
| 915 | } else { | ||
| 916 | spin_unlock_irqrestore(&rf->rsrc_lock, flags); | ||
| 917 | ret = irdma_alloc_rsrc(rf, rf->allocated_qps, rf->max_qp, | ||
| 918 | qp_num, &rf->next_qp); | ||
| 919 | if (ret) | ||
| 920 | return ret; | ||
| 921 | } | ||
| 922 | |||
| 923 | ret = irdma_vchnl_req_add_vport(&rf->sc_dev, iwdev->vport_id, *qp_num, | ||
| 924 | (&iwdev->vsi)->qos); | ||
| 925 | if (ret) { | ||
| 926 | if (*qp_num != 1) { | ||
| 927 | irdma_free_rsrc(rf, rf->allocated_qps, *qp_num); | ||
| 928 | } else { | ||
| 929 | spin_lock_irqsave(&rf->rsrc_lock, flags); | ||
| 930 | rf->hwqp1_rsvd = false; | ||
| 931 | spin_unlock_irqrestore(&rf->rsrc_lock, flags); | ||
| 932 | } | ||
| 933 | return ret; | ||
| 934 | } | ||
| 935 | |||
| 936 | return 0; | ||
| 937 | } | ||
| 938 | |||
| 867 | /** | 939 | /** |
| 868 | * irdma_create_qp - create qp | 940 | * irdma_create_qp - create qp |
| 869 | * @ibqp: ptr of qp | 941 | * @ibqp: ptr of qp |
| @@ -889,6 +961,18 @@ static int irdma_create_qp(struct ib_qp *ibqp, · 19 unmodified lines | |||
| 889 | struct irdma_uk_attrs *uk_attrs = &dev->hw_attrs.uk_attrs; | 961 | struct irdma_uk_attrs *uk_attrs = &dev->hw_attrs.uk_attrs; |
| 890 | struct irdma_qp_init_info init_info = {}; | 962 | struct irdma_qp_init_info init_info = {}; |
| 891 | struct irdma_qp_host_ctx_info *ctx_info; | 963 | struct irdma_qp_host_ctx_info *ctx_info; |
| 964 | struct irdma_srq *iwsrq; | ||
| 965 | bool srq_valid = false; | ||
| 966 | u32 srq_id = 0; | ||
| 967 | |||
| 968 | if (init_attr->srq) { | ||
| 969 | iwsrq = to_iwsrq(init_attr->srq); | ||
| 970 | srq_valid = true; | ||
| 971 | srq_id = iwsrq->srq_num; | ||
| 972 | init_attr->cap.max_recv_sge = uk_attrs->max_hw_wq_frags; | ||
| 973 | init_attr->cap.max_recv_wr = 4; | ||
| 974 | init_info.qp_uk_init_info.srq_uk = &iwsrq->sc_srq.srq_uk; | ||
| 975 | } | ||
| 892 | 976 | ||
| 893 | err_code = irdma_validate_qp_attrs(init_attr, iwdev); | 977 | err_code = irdma_validate_qp_attrs(init_attr, iwdev); |
| 894 | if (err_code) | 978 | if (err_code) |
| @@ -925,16 +1009,20 @@ static int irdma_create_qp(struct ib_qp *ibqp, · 30 unmodified lines | |||
| 925 | init_info.host_ctx = (__le64 *)(init_info.q2 + IRDMA_Q2_BUF_SIZE); | 1009 | init_info.host_ctx = (__le64 *)(init_info.q2 + IRDMA_Q2_BUF_SIZE); |
| 926 | init_info.host_ctx_pa = init_info.q2_pa + IRDMA_Q2_BUF_SIZE; | 1010 | init_info.host_ctx_pa = init_info.q2_pa + IRDMA_Q2_BUF_SIZE; |
| 927 | 1011 | ||
| 928 | if (init_attr->qp_type == IB_QPT_GSI) | 1012 | if (init_attr->qp_type == IB_QPT_GSI) { |
| 929 | qp_num = 1; | 1013 | err_code = irdma_setup_gsi_qp_rsrc(iwqp, &qp_num); |
| 930 | else | 1014 | if (err_code) |
| 1015 | goto error; | ||
| 1016 | iwqp->ibqp.qp_num = 1; | ||
| 1017 | } else { | ||
| 931 | err_code = irdma_alloc_rsrc(rf, rf->allocated_qps, rf->max_qp, | 1018 | err_code = irdma_alloc_rsrc(rf, rf->allocated_qps, rf->max_qp, |
| 932 | &qp_num, &rf->next_qp); | 1019 | &qp_num, &rf->next_qp); |
| 933 | if (err_code) | 1020 | if (err_code) |
| 934 | goto error; | 1021 | goto error; |
| 1022 | iwqp->ibqp.qp_num = qp_num; | ||
| 1023 | } | ||
| 935 | 1024 | ||
| 936 | iwqp->iwpd = iwpd; | 1025 | iwqp->iwpd = iwpd; |
| 937 | iwqp->ibqp.qp_num = qp_num; | ||
| 938 | qp = &iwqp->sc_qp; | 1026 | qp = &iwqp->sc_qp; |
| 939 | iwqp->iwscq = to_iwcq(init_attr->send_cq); | 1027 | iwqp->iwscq = to_iwcq(init_attr->send_cq); |
| 940 | iwqp->iwrcq = to_iwcq(init_attr->recv_cq); | 1028 | iwqp->iwrcq = to_iwcq(init_attr->recv_cq); |
| @@ -991,13 +1079,22 @@ static int irdma_create_qp(struct ib_qp *ibqp, · 50 unmodified lines | |||
| 991 | } | 1079 | } |
| 992 | 1080 | ||
| 993 | ctx_info = &iwqp->ctx_info; | 1081 | ctx_info = &iwqp->ctx_info; |
| 1082 | ctx_info->srq_valid = srq_valid; | ||
| 1083 | ctx_info->srq_id = srq_id; | ||
| 994 | ctx_info->send_cq_num = iwqp->iwscq->sc_cq.cq_uk.cq_id; | 1084 | ctx_info->send_cq_num = iwqp->iwscq->sc_cq.cq_uk.cq_id; |
| 995 | ctx_info->rcv_cq_num = iwqp->iwrcq->sc_cq.cq_uk.cq_id; | 1085 | ctx_info->rcv_cq_num = iwqp->iwrcq->sc_cq.cq_uk.cq_id; |
| 996 | 1086 | ||
| 997 | if (rdma_protocol_roce(&iwdev->ibdev, 1)) | 1087 | if (rdma_protocol_roce(&iwdev->ibdev, 1)) { |
| 1088 | if (dev->ws_add(&iwdev->vsi, 0)) { | ||
| 1089 | irdma_cqp_qp_destroy_cmd(&rf->sc_dev, &iwqp->sc_qp); | ||
| 1090 | err_code = -EINVAL; | ||
| 1091 | goto error; | ||
| 1092 | } | ||
| 1093 | irdma_qp_add_qos(&iwqp->sc_qp); | ||
| 998 | irdma_roce_fill_and_set_qpctx_info(iwqp, ctx_info); | 1094 | irdma_roce_fill_and_set_qpctx_info(iwqp, ctx_info); |
| 999 | else | 1095 | } else { |
| 1000 | irdma_iw_fill_and_set_qpctx_info(iwqp, ctx_info); | 1096 | irdma_iw_fill_and_set_qpctx_info(iwqp, ctx_info); |
| 1097 | } | ||
| 1001 | 1098 | ||
| 1002 | err_code = irdma_cqp_create_qp_cmd(iwqp); | 1099 | err_code = irdma_cqp_create_qp_cmd(iwqp); |
| 1003 | if (err_code) | 1100 | if (err_code) |
| @@ -1009,16 +1106,6 @@ static int irdma_create_qp(struct ib_qp *ibqp, · 5 unmodified lines | |||
| 1009 | iwqp->sig_all = init_attr->sq_sig_type == IB_SIGNAL_ALL_WR; | 1106 | iwqp->sig_all = init_attr->sq_sig_type == IB_SIGNAL_ALL_WR; |
| 1010 | rf->qp_table[qp_num] = iwqp; | 1107 | rf->qp_table[qp_num] = iwqp; |
| 1011 | 1108 | ||
| 1012 | if (rdma_protocol_roce(&iwdev->ibdev, 1)) { | ||
| 1013 | if (dev->ws_add(&iwdev->vsi, 0)) { | ||
| 1014 | irdma_cqp_qp_destroy_cmd(&rf->sc_dev, &iwqp->sc_qp); | ||
| 1015 | err_code = -EINVAL; | ||
| 1016 | goto error; | ||
| 1017 | } | ||
| 1018 | |||
| 1019 | irdma_qp_add_qos(&iwqp->sc_qp); | ||
| 1020 | } | ||
| 1021 | |||
| 1022 | if (udata) { | 1109 | if (udata) { |
| 1023 | /* GEN_1 legacy support with libi40iw does not have expanded uresp struct */ | 1110 | /* GEN_1 legacy support with libi40iw does not have expanded uresp struct */ |
| 1024 | if (udata->outlen < sizeof(uresp)) { | 1111 | if (udata->outlen < sizeof(uresp)) { |
| @@ -1063,6 +1150,8 @@ static int irdma_get_ib_acc_flags(struct irdma_qp *iwqp) · 38 unmodified lines | |||
| 1063 | acc_flags |= IB_ACCESS_REMOTE_READ; | 1150 | acc_flags |= IB_ACCESS_REMOTE_READ; |
| 1064 | if (iwqp->roce_info.bind_en) | 1151 | if (iwqp->roce_info.bind_en) |
| 1065 | acc_flags |= IB_ACCESS_MW_BIND; | 1152 | acc_flags |= IB_ACCESS_MW_BIND; |
| 1153 | if (iwqp->ctx_info.remote_atomics_en) | ||
| 1154 | acc_flags |= IB_ACCESS_REMOTE_ATOMIC; | ||
| 1066 | } else { | 1155 | } else { |
| 1067 | if (iwqp->iwarp_info.wr_rdresp_en) { | 1156 | if (iwqp->iwarp_info.wr_rdresp_en) { |
| 1068 | acc_flags |= IB_ACCESS_LOCAL_WRITE; | 1157 | acc_flags |= IB_ACCESS_LOCAL_WRITE; |
| @@ -1070,8 +1159,8 @@ static int irdma_get_ib_acc_flags(struct irdma_qp *iwqp) · 1 unmodified line | |||
| 1070 | } | 1159 | } |
| 1071 | if (iwqp->iwarp_info.rd_en) | 1160 | if (iwqp->iwarp_info.rd_en) |
| 1072 | acc_flags |= IB_ACCESS_REMOTE_READ; | 1161 | acc_flags |= IB_ACCESS_REMOTE_READ; |
| 1073 | if (iwqp->iwarp_info.bind_en) | 1162 | if (iwqp->ctx_info.remote_atomics_en) |
| 1074 | acc_flags |= IB_ACCESS_MW_BIND; | 1163 | acc_flags |= IB_ACCESS_REMOTE_ATOMIC; |
| 1075 | } | 1164 | } |
| 1076 | return acc_flags; | 1165 | return acc_flags; |
| 1077 | } | 1166 | } |
| @@ -1110,6 +1199,7 @@ static int irdma_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 32 unmodified lines | |||
| 1110 | attr->pkey_index = iwqp->roce_info.p_key; | 1199 | attr->pkey_index = iwqp->roce_info.p_key; |
| 1111 | attr->retry_cnt = iwqp->udp_info.rexmit_thresh; | 1200 | attr->retry_cnt = iwqp->udp_info.rexmit_thresh; |
| 1112 | attr->rnr_retry = iwqp->udp_info.rnr_nak_thresh; | 1201 | attr->rnr_retry = iwqp->udp_info.rnr_nak_thresh; |
| 1202 | attr->min_rnr_timer = iwqp->udp_info.min_rnr_timer; | ||
| 1113 | attr->max_rd_atomic = iwqp->roce_info.ord_size; | 1203 | attr->max_rd_atomic = iwqp->roce_info.ord_size; |
| 1114 | attr->max_dest_rd_atomic = iwqp->roce_info.ird_size; | 1204 | attr->max_dest_rd_atomic = iwqp->roce_info.ird_size; |
| 1115 | } | 1205 | } |
| @@ -1118,6 +1208,7 @@ static int irdma_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 2 unmodified lines | |||
| 1118 | init_attr->qp_context = iwqp->ibqp.qp_context; | 1208 | init_attr->qp_context = iwqp->ibqp.qp_context; |
| 1119 | init_attr->send_cq = iwqp->ibqp.send_cq; | 1209 | init_attr->send_cq = iwqp->ibqp.send_cq; |
| 1120 | init_attr->recv_cq = iwqp->ibqp.recv_cq; | 1210 | init_attr->recv_cq = iwqp->ibqp.recv_cq; |
| 1211 | init_attr->srq = iwqp->ibqp.srq; | ||
| 1121 | init_attr->cap = attr->cap; | 1212 | init_attr->cap = attr->cap; |
| 1122 | 1213 | ||
| 1123 | return 0; | 1214 | return 0; |
| @@ -1242,6 +1333,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 118 unmodified lines | |||
| 1242 | if (attr_mask & IB_QP_RNR_RETRY) | 1333 | if (attr_mask & IB_QP_RNR_RETRY) |
| 1243 | udp_info->rnr_nak_thresh = attr->rnr_retry; | 1334 | udp_info->rnr_nak_thresh = attr->rnr_retry; |
| 1244 | 1335 | ||
| 1336 | if (attr_mask & IB_QP_MIN_RNR_TIMER && | ||
| 1337 | dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3) | ||
| 1338 | udp_info->min_rnr_timer = attr->min_rnr_timer; | ||
| 1339 | |||
| 1245 | if (attr_mask & IB_QP_RETRY_CNT) | 1340 | if (attr_mask & IB_QP_RETRY_CNT) |
| 1246 | udp_info->rexmit_thresh = attr->retry_cnt; | 1341 | udp_info->rexmit_thresh = attr->retry_cnt; |
| 1247 | 1342 | ||
| @@ -1362,6 +1457,9 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · 114 unmodified lines | |||
| 1362 | roce_info->wr_rdresp_en = true; | 1457 | roce_info->wr_rdresp_en = true; |
| 1363 | if (attr->qp_access_flags & IB_ACCESS_REMOTE_READ) | 1458 | if (attr->qp_access_flags & IB_ACCESS_REMOTE_READ) |
| 1364 | roce_info->rd_en = true; | 1459 | roce_info->rd_en = true; |
| 1460 | if (dev->hw_attrs.uk_attrs.feature_flags & IRDMA_FEATURE_ATOMIC_OPS) | ||
| 1461 | if (attr->qp_access_flags & IB_ACCESS_REMOTE_ATOMIC) | ||
| 1462 | ctx_info->remote_atomics_en = true; | ||
| 1365 | } | 1463 | } |
| 1366 | 1464 | ||
| 1367 | wait_event(iwqp->mod_qp_waitq, !atomic_read(&iwqp->hw_mod_qp_pend)); | 1465 | wait_event(iwqp->mod_qp_waitq, !atomic_read(&iwqp->hw_mod_qp_pend)); |
| @@ -1776,6 +1874,24 @@ exit: · 408 unmodified lines | |||
| 1776 | return err; | 1874 | return err; |
| 1777 | } | 1875 | } |
| 1778 | 1876 | ||
| 1877 | /** | ||
| 1878 | * irdma_srq_free_rsrc - free up resources for srq | ||
| 1879 | * @rf: RDMA PCI function | ||
| 1880 | * @iwsrq: srq ptr | ||
| 1881 | */ | ||
| 1882 | static void irdma_srq_free_rsrc(struct irdma_pci_f *rf, struct irdma_srq *iwsrq) | ||
| 1883 | { | ||
| 1884 | struct irdma_sc_srq *srq = &iwsrq->sc_srq; | ||
| 1885 | |||
| 1886 | if (!iwsrq->user_mode) { | ||
| 1887 | dma_free_coherent(rf->sc_dev.hw->device, iwsrq->kmem.size, | ||
| 1888 | iwsrq->kmem.va, iwsrq->kmem.pa); | ||
| 1889 | iwsrq->kmem.va = NULL; | ||
| 1890 | } | ||
| 1891 | |||
| 1892 | irdma_free_rsrc(rf, rf->allocated_srqs, srq->srq_uk.srq_id); | ||
| 1893 | } | ||
| 1894 | |||
| 1779 | /** | 1895 | /** |
| 1780 | * irdma_cq_free_rsrc - free up resources for cq | 1896 | * irdma_cq_free_rsrc - free up resources for cq |
| 1781 | * @rf: RDMA PCI function | 1897 | * @rf: RDMA PCI function |
| @@ -1839,6 +1955,22 @@ static int irdma_process_resize_list(struct irdma_cq *iwcq, · 57 unmodified lines | |||
| 1839 | return cnt; | 1955 | return cnt; |
| 1840 | } | 1956 | } |
| 1841 | 1957 | ||
| 1958 | /** | ||
| 1959 | * irdma_destroy_srq - destroy srq | ||
| 1960 | * @ibsrq: srq pointer | ||
| 1961 | * @udata: user data | ||
| 1962 | */ | ||
| 1963 | static int irdma_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata) | ||
| 1964 | { | ||
| 1965 | struct irdma_device *iwdev = to_iwdev(ibsrq->device); | ||
| 1966 | struct irdma_srq *iwsrq = to_iwsrq(ibsrq); | ||
| 1967 | struct irdma_sc_srq *srq = &iwsrq->sc_srq; | ||
| 1968 | |||
| 1969 | irdma_srq_wq_destroy(iwdev->rf, srq); | ||
| 1970 | irdma_srq_free_rsrc(iwdev->rf, iwsrq); | ||
| 1971 | return 0; | ||
| 1972 | } | ||
| 1973 | |||
| 1842 | /** | 1974 | /** |
| 1843 | * irdma_destroy_cq - destroy cq | 1975 | * irdma_destroy_cq - destroy cq |
| 1844 | * @ib_cq: cq pointer | 1976 | * @ib_cq: cq pointer |
| @@ -1914,8 +2046,13 @@ static int irdma_resize_cq(struct ib_cq *ibcq, int entries, · 69 unmodified lines | |||
| 1914 | 2046 | ||
| 1915 | if (!iwcq->user_mode) { | 2047 | if (!iwcq->user_mode) { |
| 1916 | entries++; | 2048 | entries++; |
| 1917 | if (rf->sc_dev.hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_2) | 2049 | |
| 2050 | if (!iwcq->sc_cq.cq_uk.avoid_mem_cflct && | ||
| 2051 | dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_2) | ||
| 1918 | entries *= 2; | 2052 | entries *= 2; |
| 2053 | |||
| 2054 | if (entries & 1) | ||
| 2055 | entries += 1; /* cq size must be an even number */ | ||
| 1919 | } | 2056 | } |
| 1920 | 2057 | ||
| 1921 | info.cq_size = max(entries, 4); | 2058 | info.cq_size = max(entries, 4); |
| @@ -2022,10 +2159,297 @@ error: · 100 unmodified lines | |||
| 2022 | return ret; | 2159 | return ret; |
| 2023 | } | 2160 | } |
| 2024 | 2161 | ||
| 2162 | /** | ||
| 2163 | * irdma_srq_event - event notification for srq limit | ||
| 2164 | * @srq: shared srq struct | ||
| 2165 | */ | ||
| 2166 | void irdma_srq_event(struct irdma_sc_srq *srq) | ||
| 2167 | { | ||
| 2168 | struct irdma_srq *iwsrq = container_of(srq, struct irdma_srq, sc_srq); | ||
| 2169 | struct ib_srq *ibsrq = &iwsrq->ibsrq; | ||
| 2170 | struct ib_event event; | ||
| 2171 | |||
| 2172 | srq->srq_limit = 0; | ||
| 2173 | |||
| 2174 | if (!ibsrq->event_handler) | ||
| 2175 | return; | ||
| 2176 | |||
| 2177 | event.device = ibsrq->device; | ||
| 2178 | event.element.port_num = 1; | ||
| 2179 | event.element.srq = ibsrq; | ||
| 2180 | event.event = IB_EVENT_SRQ_LIMIT_REACHED; | ||
| 2181 | ibsrq->event_handler(&event, ibsrq->srq_context); | ||
| 2182 | } | ||
| 2183 | |||
| 2184 | /** | ||
| 2185 | * irdma_modify_srq - modify srq request | ||
| 2186 | * @ibsrq: srq's pointer for modify | ||
| 2187 | * @attr: access attributes | ||
| 2188 | * @attr_mask: state mask | ||
| 2189 | * @udata: user data | ||
| 2190 | */ | ||
| 2191 | static int irdma_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr, | ||
| 2192 | enum ib_srq_attr_mask attr_mask, | ||
| 2193 | struct ib_udata *udata) | ||
| 2194 | { | ||
| 2195 | struct irdma_device *iwdev = to_iwdev(ibsrq->device); | ||
| 2196 | struct irdma_srq *iwsrq = to_iwsrq(ibsrq); | ||
| 2197 | struct irdma_cqp_request *cqp_request; | ||
| 2198 | struct irdma_pci_f *rf = iwdev->rf; | ||
| 2199 | struct irdma_modify_srq_info *info; | ||
| 2200 | struct cqp_cmds_info *cqp_info; | ||
| 2201 | int status; | ||
| 2202 | |||
| 2203 | if (attr_mask & IB_SRQ_MAX_WR) | ||
| 2204 | return -EINVAL; | ||
| 2205 | |||
| 2206 | if (!(attr_mask & IB_SRQ_LIMIT)) | ||
| 2207 | return 0; | ||
| 2208 | |||
| 2209 | if (attr->srq_limit > iwsrq->sc_srq.srq_uk.srq_size) | ||
| 2210 | return -EINVAL; | ||
| 2211 | |||
| 2212 | /* Execute this cqp op synchronously, so we can update srq_limit | ||
| 2213 | * upon successful completion. | ||
| 2214 | */ | ||
| 2215 | cqp_request = irdma_alloc_and_get_cqp_request(&rf->cqp, true); | ||
| 2216 | if (!cqp_request) | ||
| 2217 | return -ENOMEM; | ||
| 2218 | |||
| 2219 | cqp_info = &cqp_request->info; | ||
| 2220 | info = &cqp_info->in.u.srq_modify.info; | ||
| 2221 | info->srq_limit = attr->srq_limit; | ||
| 2222 | if (info->srq_limit > 0xFFF) | ||
| 2223 | info->srq_limit = 0xFFF; | ||
| 2224 | info->arm_limit_event = 1; | ||
| 2225 | |||
| 2226 | cqp_info->cqp_cmd = IRDMA_OP_SRQ_MODIFY; | ||
| 2227 | cqp_info->post_sq = 1; | ||
| 2228 | cqp_info->in.u.srq_modify.srq = &iwsrq->sc_srq; | ||
| 2229 | cqp_info->in.u.srq_modify.scratch = (uintptr_t)cqp_request; | ||
| 2230 | status = irdma_handle_cqp_op(rf, cqp_request); | ||
| 2231 | irdma_put_cqp_request(&rf->cqp, cqp_request); | ||
| 2232 | if (status) | ||
| 2233 | return status; | ||
| 2234 | |||
| 2235 | iwsrq->sc_srq.srq_limit = info->srq_limit; | ||
| 2236 | |||
| 2237 | return 0; | ||
| 2238 | } | ||
| 2239 | |||
| 2240 | static int irdma_setup_umode_srq(struct irdma_device *iwdev, | ||
| 2241 | struct irdma_srq *iwsrq, | ||
| 2242 | struct irdma_srq_init_info *info, | ||
| 2243 | struct ib_udata *udata) | ||
| 2244 | { | ||
| 2245 | #define IRDMA_CREATE_SRQ_MIN_REQ_LEN \ | ||
| 2246 | offsetofend(struct irdma_create_srq_req, user_shadow_area) | ||
| 2247 | struct irdma_create_srq_req req = {}; | ||
| 2248 | struct irdma_ucontext *ucontext; | ||
| 2249 | struct irdma_srq_mr *srqmr; | ||
| 2250 | struct irdma_pbl *iwpbl; | ||
| 2251 | unsigned long flags; | ||
| 2252 | |||
| 2253 | iwsrq->user_mode = true; | ||
| 2254 | ucontext = rdma_udata_to_drv_context(udata, struct irdma_ucontext, | ||
| 2255 | ibucontext); | ||
| 2256 | |||
| 2257 | if (udata->inlen < IRDMA_CREATE_SRQ_MIN_REQ_LEN) | ||
| 2258 | return -EINVAL; | ||
| 2259 | |||
| 2260 | if (ib_copy_from_udata(&req, udata, | ||
| 2261 | min(sizeof(req), udata->inlen))) | ||
| 2262 | return -EFAULT; | ||
| 2263 | |||
| 2264 | spin_lock_irqsave(&ucontext->srq_reg_mem_list_lock, flags); | ||
| 2265 | iwpbl = irdma_get_pbl((unsigned long)req.user_srq_buf, | ||
| 2266 | &ucontext->srq_reg_mem_list); | ||
| 2267 | spin_unlock_irqrestore(&ucontext->srq_reg_mem_list_lock, flags); | ||
| 2268 | if (!iwpbl) | ||
| 2269 | return -EPROTO; | ||
| 2270 | |||
| 2271 | iwsrq->iwpbl = iwpbl; | ||
| 2272 | srqmr = &iwpbl->srq_mr; | ||
| 2273 | |||
| 2274 | if (iwpbl->pbl_allocated) { | ||
| 2275 | info->virtual_map = true; | ||
| 2276 | info->pbl_chunk_size = 1; | ||
| 2277 | info->first_pm_pbl_idx = srqmr->srq_pbl.idx; | ||
| 2278 | info->leaf_pbl_size = 1; | ||
| 2279 | } else { | ||
| 2280 | info->srq_pa = srqmr->srq_pbl.addr; | ||
| 2281 | } | ||
| 2282 | info->shadow_area_pa = srqmr->shadow; | ||
| 2283 | |||
| 2284 | return 0; | ||
| 2285 | } | ||
| 2286 | |||
| 2287 | static int irdma_setup_kmode_srq(struct irdma_device *iwdev, | ||
| 2288 | struct irdma_srq *iwsrq, | ||
| 2289 | struct irdma_srq_init_info *info, u32 depth, | ||
| 2290 | u8 shift) | ||
| 2291 | { | ||
| 2292 | struct irdma_srq_uk_init_info *ukinfo = &info->srq_uk_init_info; | ||
| 2293 | struct irdma_dma_mem *mem = &iwsrq->kmem; | ||
| 2294 | u32 size, ring_size; | ||
| 2295 | |||
| 2296 | ring_size = depth * IRDMA_QP_WQE_MIN_SIZE; | ||
| 2297 | size = ring_size + (IRDMA_SHADOW_AREA_SIZE << 3); | ||
| 2298 | |||
| 2299 | mem->size = ALIGN(size, 256); | ||
| 2300 | mem->va = dma_alloc_coherent(iwdev->rf->hw.device, mem->size, | ||
| 2301 | &mem->pa, GFP_KERNEL); | ||
| 2302 | if (!mem->va) | ||
| 2303 | return -ENOMEM; | ||
| 2304 | |||
| 2305 | ukinfo->srq = mem->va; | ||
| 2306 | ukinfo->srq_size = depth >> shift; | ||
| 2307 | ukinfo->shadow_area = mem->va + ring_size; | ||
| 2308 | |||
| 2309 | info->shadow_area_pa = info->srq_pa + ring_size; | ||
| 2310 | info->srq_pa = mem->pa; | ||
| 2311 | |||
| 2312 | return 0; | ||
| 2313 | } | ||
| 2314 | |||
| 2315 | /** | ||
| 2316 | * irdma_create_srq - create srq | ||
| 2317 | * @ibsrq: ib's srq pointer | ||
| 2318 | * @initattrs: attributes for srq | ||
| 2319 | * @udata: user data for create srq | ||
| 2320 | */ | ||
| 2321 | static int irdma_create_srq(struct ib_srq *ibsrq, | ||
| 2322 | struct ib_srq_init_attr *initattrs, | ||
| 2323 | struct ib_udata *udata) | ||
| 2324 | { | ||
| 2325 | struct irdma_device *iwdev = to_iwdev(ibsrq->device); | ||
| 2326 | struct ib_srq_attr *attr = &initattrs->attr; | ||
| 2327 | struct irdma_pd *iwpd = to_iwpd(ibsrq->pd); | ||
| 2328 | struct irdma_srq *iwsrq = to_iwsrq(ibsrq); | ||
| 2329 | struct irdma_srq_uk_init_info *ukinfo; | ||
| 2330 | struct irdma_cqp_request *cqp_request; | ||
| 2331 | struct irdma_srq_init_info info = {}; | ||
| 2332 | struct irdma_pci_f *rf = iwdev->rf; | ||
| 2333 | struct irdma_uk_attrs *uk_attrs; | ||
| 2334 | struct cqp_cmds_info *cqp_info; | ||
| 2335 | int err_code = 0; | ||
| 2336 | u32 depth; | ||
| 2337 | u8 shift; | ||
| 2338 | |||
| 2339 | uk_attrs = &rf->sc_dev.hw_attrs.uk_attrs; | ||
| 2340 | ukinfo = &info.srq_uk_init_info; | ||
| 2341 | |||
| 2342 | if (initattrs->srq_type != IB_SRQT_BASIC) | ||
| 2343 | return -EOPNOTSUPP; | ||
| 2344 | |||
| 2345 | if (!(uk_attrs->feature_flags & IRDMA_FEATURE_SRQ) || | ||
| 2346 | attr->max_sge > uk_attrs->max_hw_wq_frags) | ||
| 2347 | return -EINVAL; | ||
| 2348 | |||
| 2349 | refcount_set(&iwsrq->refcnt, 1); | ||
| 2350 | spin_lock_init(&iwsrq->lock); | ||
| 2351 | err_code = irdma_alloc_rsrc(rf, rf->allocated_srqs, rf->max_srq, | ||
| 2352 | &iwsrq->srq_num, &rf->next_srq); | ||
| 2353 | if (err_code) | ||
| 2354 | return err_code; | ||
| 2355 | |||
| 2356 | ukinfo->max_srq_frag_cnt = attr->max_sge; | ||
| 2357 | ukinfo->uk_attrs = uk_attrs; | ||
| 2358 | ukinfo->srq_id = iwsrq->srq_num; | ||
| 2359 | |||
| 2360 | irdma_get_wqe_shift(ukinfo->uk_attrs, ukinfo->max_srq_frag_cnt, 0, | ||
| 2361 | &shift); | ||
| 2362 | |||
| 2363 | err_code = irdma_get_srqdepth(ukinfo->uk_attrs, attr->max_wr, | ||
| 2364 | shift, &depth); | ||
| 2365 | if (err_code) | ||
| 2366 | return err_code; | ||
| 2367 | |||
| 2368 | /* Actual SRQ size in WRs for ring and HW */ | ||
| 2369 | ukinfo->srq_size = depth >> shift; | ||
| 2370 | |||
| 2371 | /* Max postable WRs to SRQ */ | ||
| 2372 | iwsrq->max_wr = (depth - IRDMA_RQ_RSVD) >> shift; | ||
| 2373 | attr->max_wr = iwsrq->max_wr; | ||
| 2374 | |||
| 2375 | if (udata) | ||
| 2376 | err_code = irdma_setup_umode_srq(iwdev, iwsrq, &info, udata); | ||
| 2377 | else | ||
| 2378 | err_code = irdma_setup_kmode_srq(iwdev, iwsrq, &info, depth, | ||
| 2379 | shift); | ||
| 2380 | |||
| 2381 | if (err_code) | ||
| 2382 | goto free_rsrc; | ||
| 2383 | |||
| 2384 | info.vsi = &iwdev->vsi; | ||
| 2385 | info.pd = &iwpd->sc_pd; | ||
| 2386 | |||
| 2387 | err_code = irdma_sc_srq_init(&iwsrq->sc_srq, &info); | ||
| 2388 | if (err_code) | ||
| 2389 | goto free_dmem; | ||
| 2390 | |||
| 2391 | cqp_request = irdma_alloc_and_get_cqp_request(&rf->cqp, true); | ||
| 2392 | if (!cqp_request) { | ||
| 2393 | err_code = -ENOMEM; | ||
| 2394 | goto free_dmem; | ||
| 2395 | } | ||
| 2396 | |||
| 2397 | cqp_info = &cqp_request->info; | ||
| 2398 | cqp_info->cqp_cmd = IRDMA_OP_SRQ_CREATE; | ||
| 2399 | cqp_info->post_sq = 1; | ||
| 2400 | cqp_info->in.u.srq_create.srq = &iwsrq->sc_srq; | ||
| 2401 | cqp_info->in.u.srq_create.scratch = (uintptr_t)cqp_request; | ||
| 2402 | err_code = irdma_handle_cqp_op(rf, cqp_request); | ||
| 2403 | irdma_put_cqp_request(&rf->cqp, cqp_request); | ||
| 2404 | if (err_code) | ||
| 2405 | goto free_dmem; | ||
| 2406 | |||
| 2407 | if (udata) { | ||
| 2408 | struct irdma_create_srq_resp resp = {}; | ||
| 2409 | |||
| 2410 | resp.srq_id = iwsrq->srq_num; | ||
| 2411 | resp.srq_size = ukinfo->srq_size; | ||
| 2412 | if (ib_copy_to_udata(udata, &resp, | ||
| 2413 | min(sizeof(resp), udata->outlen))) { | ||
| 2414 | err_code = -EPROTO; | ||
| 2415 | goto srq_destroy; | ||
| 2416 | } | ||
| 2417 | } | ||
| 2418 | |||
| 2419 | return 0; | ||
| 2420 | |||
| 2421 | srq_destroy: | ||
| 2422 | irdma_srq_wq_destroy(rf, &iwsrq->sc_srq); | ||
| 2423 | |||
| 2424 | free_dmem: | ||
| 2425 | if (!iwsrq->user_mode) | ||
| 2426 | dma_free_coherent(rf->hw.device, iwsrq->kmem.size, | ||
| 2427 | iwsrq->kmem.va, iwsrq->kmem.pa); | ||
| 2428 | free_rsrc: | ||
| 2429 | irdma_free_rsrc(rf, rf->allocated_srqs, iwsrq->srq_num); | ||
| 2430 | return err_code; | ||
| 2431 | } | ||
| 2432 | |||
| 2433 | /** | ||
| 2434 | * irdma_query_srq - get SRQ attributes | ||
| 2435 | * @ibsrq: the SRQ to query | ||
| 2436 | * @attr: the attributes of the SRQ | ||
| 2437 | */ | ||
| 2438 | static int irdma_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr) | ||
| 2439 | { | ||
| 2440 | struct irdma_srq *iwsrq = to_iwsrq(ibsrq); | ||
| 2441 | |||
| 2442 | attr->max_wr = iwsrq->max_wr; | ||
| 2443 | attr->max_sge = iwsrq->sc_srq.srq_uk.max_srq_frag_cnt; | ||
| 2444 | attr->srq_limit = iwsrq->sc_srq.srq_limit; | ||
| 2445 | |||
| 2446 | return 0; | ||
| 2447 | } | ||
| 2448 | |||
| 2025 | static inline int cq_validate_flags(u32 flags, u8 hw_rev) | 2449 | static inline int cq_validate_flags(u32 flags, u8 hw_rev) |
| 2026 | { | 2450 | { |
| 2027 | /* GEN1 does not support CQ create flags */ | 2451 | /* GEN1/2 does not support CQ create flags */ |
| 2028 | if (hw_rev == IRDMA_GEN_1) | 2452 | if (hw_rev <= IRDMA_GEN_2) |
| 2029 | return flags ? -EOPNOTSUPP : 0; | 2453 | return flags ? -EOPNOTSUPP : 0; |
| 2030 | 2454 | ||
| 2031 | return flags & ~IB_UVERBS_CQ_FLAGS_TIMESTAMP_COMPLETION ? -EOPNOTSUPP : 0; | 2455 | return flags & ~IB_UVERBS_CQ_FLAGS_TIMESTAMP_COMPLETION ? -EOPNOTSUPP : 0; |
| @@ -2058,6 +2482,7 @@ static int irdma_create_cq(struct ib_cq *ibcq, · 26 unmodified lines | |||
| 2058 | unsigned long flags; | 2482 | unsigned long flags; |
| 2059 | int err_code; | 2483 | int err_code; |
| 2060 | int entries = attr->cqe; | 2484 | int entries = attr->cqe; |
| 2485 | bool cqe_64byte_ena; | ||
| 2061 | 2486 | ||
| 2062 | err_code = cq_validate_flags(attr->flags, dev->hw_attrs.uk_attrs.hw_rev); | 2487 | err_code = cq_validate_flags(attr->flags, dev->hw_attrs.uk_attrs.hw_rev); |
| 2063 | if (err_code) | 2488 | if (err_code) |
| @@ -2081,6 +2506,9 @@ static int irdma_create_cq(struct ib_cq *ibcq, · 17 unmodified lines | |||
| 2081 | info.dev = dev; | 2506 | info.dev = dev; |
| 2082 | ukinfo->cq_size = max(entries, 4); | 2507 | ukinfo->cq_size = max(entries, 4); |
| 2083 | ukinfo->cq_id = cq_num; | 2508 | ukinfo->cq_id = cq_num; |
| 2509 | cqe_64byte_ena = dev->hw_attrs.uk_attrs.feature_flags & IRDMA_FEATURE_64_BYTE_CQE ? | ||
| 2510 | true : false; | ||
| 2511 | ukinfo->avoid_mem_cflct = cqe_64byte_ena; | ||
| 2084 | iwcq->ibcq.cqe = info.cq_uk_init_info.cq_size; | 2512 | iwcq->ibcq.cqe = info.cq_uk_init_info.cq_size; |
| 2085 | if (attr->comp_vector < rf->ceqs_count) | 2513 | if (attr->comp_vector < rf->ceqs_count) |
| 2086 | info.ceq_id = attr->comp_vector; | 2514 | info.ceq_id = attr->comp_vector; |
| @@ -2116,8 +2544,6 @@ static int irdma_create_cq(struct ib_cq *ibcq, · 29 unmodified lines | |||
| 2116 | goto cq_free_rsrc; | 2544 | goto cq_free_rsrc; |
| 2117 | } | 2545 | } |
| 2118 | 2546 | ||
| 2119 | iwcq->iwpbl = iwpbl; | ||
| 2120 | iwcq->cq_mem_size = 0; | ||
| 2121 | cqmr = &iwpbl->cq_mr; | 2547 | cqmr = &iwpbl->cq_mr; |
| 2122 | 2548 | ||
| 2123 | if (rf->sc_dev.hw_attrs.uk_attrs.feature_flags & | 2549 | if (rf->sc_dev.hw_attrs.uk_attrs.feature_flags & |
| @@ -2132,7 +2558,6 @@ static int irdma_create_cq(struct ib_cq *ibcq, · 8 unmodified lines | |||
| 2132 | err_code = -EPROTO; | 2558 | err_code = -EPROTO; |
| 2133 | goto cq_free_rsrc; | 2559 | goto cq_free_rsrc; |
| 2134 | } | 2560 | } |
| 2135 | iwcq->iwpbl_shadow = iwpbl_shadow; | ||
| 2136 | cqmr_shadow = &iwpbl_shadow->cq_mr; | 2561 | cqmr_shadow = &iwpbl_shadow->cq_mr; |
| 2137 | info.shadow_area_pa = cqmr_shadow->cq_pbl.addr; | 2562 | info.shadow_area_pa = cqmr_shadow->cq_pbl.addr; |
| 2138 | cqmr->split = true; | 2563 | cqmr->split = true; |
| @@ -2156,11 +2581,18 @@ static int irdma_create_cq(struct ib_cq *ibcq, · 17 unmodified lines | |||
| 2156 | } | 2581 | } |
| 2157 | 2582 | ||
| 2158 | entries++; | 2583 | entries++; |
| 2159 | if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_2) | 2584 | if (!cqe_64byte_ena && dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_2) |
| 2160 | entries *= 2; | 2585 | entries *= 2; |
| 2586 | |||
| 2587 | if (entries & 1) | ||
| 2588 | entries += 1; /* cq size must be an even number */ | ||
| 2589 | |||
| 2161 | ukinfo->cq_size = entries; | 2590 | ukinfo->cq_size = entries; |
| 2162 | 2591 | ||
| 2163 | rsize = info.cq_uk_init_info.cq_size * sizeof(struct irdma_cqe); | 2592 | if (cqe_64byte_ena) |
| 2593 | rsize = info.cq_uk_init_info.cq_size * sizeof(struct irdma_extended_cqe); | ||
| 2594 | else | ||
| 2595 | rsize = info.cq_uk_init_info.cq_size * sizeof(struct irdma_cqe); | ||
| 2164 | iwcq->kmem.size = ALIGN(round_up(rsize, 256), 256); | 2596 | iwcq->kmem.size = ALIGN(round_up(rsize, 256), 256); |
| 2165 | iwcq->kmem.va = dma_alloc_coherent(dev->hw->device, | 2597 | iwcq->kmem.va = dma_alloc_coherent(dev->hw->device, |
| 2166 | iwcq->kmem.size, | 2598 | iwcq->kmem.size, |
| @@ -2240,8 +2672,9 @@ cq_free_rsrc: · 73 unmodified lines | |||
| 2240 | /** | 2672 | /** |
| 2241 | * irdma_get_mr_access - get hw MR access permissions from IB access flags | 2673 | * irdma_get_mr_access - get hw MR access permissions from IB access flags |
| 2242 | * @access: IB access flags | 2674 | * @access: IB access flags |
| 2675 | * @hw_rev: Hardware version | ||
| 2243 | */ | 2676 | */ |
| 2244 | static inline u16 irdma_get_mr_access(int access) | 2677 | static inline u16 irdma_get_mr_access(int access, u8 hw_rev) |
| 2245 | { | 2678 | { |
| 2246 | u16 hw_access = 0; | 2679 | u16 hw_access = 0; |
| 2247 | 2680 | ||
| @@ -2251,8 +2684,10 @@ static inline u16 irdma_get_mr_access(int access) · 3 unmodified lines | |||
| 2251 | IRDMA_ACCESS_FLAGS_REMOTEWRITE : 0; | 2684 | IRDMA_ACCESS_FLAGS_REMOTEWRITE : 0; |
| 2252 | hw_access |= (access & IB_ACCESS_REMOTE_READ) ? | 2685 | hw_access |= (access & IB_ACCESS_REMOTE_READ) ? |
| 2253 | IRDMA_ACCESS_FLAGS_REMOTEREAD : 0; | 2686 | IRDMA_ACCESS_FLAGS_REMOTEREAD : 0; |
| 2254 | hw_access |= (access & IB_ACCESS_MW_BIND) ? | 2687 | if (hw_rev >= IRDMA_GEN_3) { |
| 2255 | IRDMA_ACCESS_FLAGS_BIND_WINDOW : 0; | 2688 | hw_access |= (access & IB_ACCESS_MW_BIND) ? |
| 2689 | IRDMA_ACCESS_FLAGS_BIND_WINDOW : 0; | ||
| 2690 | } | ||
| 2256 | hw_access |= (access & IB_ZERO_BASED) ? | 2691 | hw_access |= (access & IB_ZERO_BASED) ? |
| 2257 | IRDMA_ACCESS_FLAGS_ZERO_BASED : 0; | 2692 | IRDMA_ACCESS_FLAGS_ZERO_BASED : 0; |
| 2258 | hw_access |= IRDMA_ACCESS_FLAGS_LOCALREAD; | 2693 | hw_access |= IRDMA_ACCESS_FLAGS_LOCALREAD; |
| @@ -2463,6 +2898,7 @@ static int irdma_handle_q_mem(struct irdma_device *iwdev, · 204 unmodified lines | |||
| 2463 | struct irdma_mr *iwmr = iwpbl->iwmr; | 2898 | struct irdma_mr *iwmr = iwpbl->iwmr; |
| 2464 | struct irdma_qp_mr *qpmr = &iwpbl->qp_mr; | 2899 | struct irdma_qp_mr *qpmr = &iwpbl->qp_mr; |
| 2465 | struct irdma_cq_mr *cqmr = &iwpbl->cq_mr; | 2900 | struct irdma_cq_mr *cqmr = &iwpbl->cq_mr; |
| 2901 | struct irdma_srq_mr *srqmr = &iwpbl->srq_mr; | ||
| 2466 | struct irdma_hmc_pble *hmc_p; | 2902 | struct irdma_hmc_pble *hmc_p; |
| 2467 | u64 *arr = iwmr->pgaddrmem; | 2903 | u64 *arr = iwmr->pgaddrmem; |
| 2468 | u32 pg_size, total; | 2904 | u32 pg_size, total; |
| @@ -2482,7 +2918,10 @@ static int irdma_handle_q_mem(struct irdma_device *iwdev, · 13 unmodified lines | |||
| 2482 | total = req->sq_pages + req->rq_pages; | 2918 | total = req->sq_pages + req->rq_pages; |
| 2483 | hmc_p = &qpmr->sq_pbl; | 2919 | hmc_p = &qpmr->sq_pbl; |
| 2484 | qpmr->shadow = (dma_addr_t)arr[total]; | 2920 | qpmr->shadow = (dma_addr_t)arr[total]; |
| 2485 | 2921 | /* Need to use physical address for RQ of QP | |
| 2922 | * in case it is associated with SRQ. | ||
| 2923 | */ | ||
| 2924 | qpmr->rq_pa = (dma_addr_t)arr[req->sq_pages]; | ||
| 2486 | if (lvl) { | 2925 | if (lvl) { |
| 2487 | ret = irdma_check_mem_contiguous(arr, req->sq_pages, | 2926 | ret = irdma_check_mem_contiguous(arr, req->sq_pages, |
| 2488 | pg_size); | 2927 | pg_size); |
| @@ -2502,6 +2941,18 @@ static int irdma_handle_q_mem(struct irdma_device *iwdev, · 13 unmodified lines | |||
| 2502 | hmc_p->addr = arr[req->sq_pages]; | 2941 | hmc_p->addr = arr[req->sq_pages]; |
| 2503 | } | 2942 | } |
| 2504 | break; | 2943 | break; |
| 2944 | case IRDMA_MEMREG_TYPE_SRQ: | ||
| 2945 | hmc_p = &srqmr->srq_pbl; | ||
| 2946 | srqmr->shadow = (dma_addr_t)arr[req->rq_pages]; | ||
| 2947 | if (lvl) | ||
| 2948 | ret = irdma_check_mem_contiguous(arr, req->rq_pages, | ||
| 2949 | pg_size); | ||
| 2950 | |||
| 2951 | if (!ret) | ||
| 2952 | hmc_p->idx = palloc->level1.idx; | ||
| 2953 | else | ||
| 2954 | hmc_p->addr = arr[0]; | ||
| 2955 | break; | ||
| 2505 | case IRDMA_MEMREG_TYPE_CQ: | 2956 | case IRDMA_MEMREG_TYPE_CQ: |
| 2506 | hmc_p = &cqmr->cq_pbl; | 2957 | hmc_p = &cqmr->cq_pbl; |
| 2507 | 2958 | ||
| @@ -2806,7 +3257,10 @@ static int irdma_hwreg_mr(struct irdma_device *iwdev, struct irdma_mr *iwmr, · 298 unmodified lines | |||
| 2806 | stag_info->stag_idx = iwmr->stag >> IRDMA_CQPSQ_STAG_IDX_S; | 3257 | stag_info->stag_idx = iwmr->stag >> IRDMA_CQPSQ_STAG_IDX_S; |
| 2807 | stag_info->stag_key = (u8)iwmr->stag; | 3258 | stag_info->stag_key = (u8)iwmr->stag; |
| 2808 | stag_info->total_len = iwmr->len; | 3259 | stag_info->total_len = iwmr->len; |
| 2809 | stag_info->access_rights = irdma_get_mr_access(access); | 3260 | stag_info->access_rights = irdma_get_mr_access(access, |
| 3261 | iwdev->rf->sc_dev.hw_attrs.uk_attrs.hw_rev); | ||
| 3262 | if (iwdev->rf->sc_dev.hw_attrs.uk_attrs.feature_flags & IRDMA_FEATURE_ATOMIC_OPS) | ||
| 3263 | stag_info->remote_atomics_en = (access & IB_ACCESS_REMOTE_ATOMIC) ? 1 : 0; | ||
| 2810 | stag_info->pd_id = iwpd->sc_pd.pd_id; | 3264 | stag_info->pd_id = iwpd->sc_pd.pd_id; |
| 2811 | stag_info->all_memory = pd->flags & IB_PD_UNSAFE_GLOBAL_RKEY; | 3265 | stag_info->all_memory = pd->flags & IB_PD_UNSAFE_GLOBAL_RKEY; |
| 2812 | if (stag_info->access_rights & IRDMA_ACCESS_FLAGS_ZERO_BASED) | 3266 | if (stag_info->access_rights & IRDMA_ACCESS_FLAGS_ZERO_BASED) |
| @@ -2972,6 +3426,37 @@ static int irdma_reg_user_mr_type_qp(struct irdma_mem_reg_req req, · 159 unmodified lines | |||
| 2972 | return 0; | 3426 | return 0; |
| 2973 | } | 3427 | } |
| 2974 | 3428 | ||
| 3429 | static int irdma_reg_user_mr_type_srq(struct irdma_mem_reg_req req, | ||
| 3430 | struct ib_udata *udata, | ||
| 3431 | struct irdma_mr *iwmr) | ||
| 3432 | { | ||
| 3433 | struct irdma_device *iwdev = to_iwdev(iwmr->ibmr.device); | ||
| 3434 | struct irdma_pbl *iwpbl = &iwmr->iwpbl; | ||
| 3435 | struct irdma_ucontext *ucontext; | ||
| 3436 | unsigned long flags; | ||
| 3437 | u32 total; | ||
| 3438 | int err; | ||
| 3439 | u8 lvl; | ||
| 3440 | |||
| 3441 | total = req.rq_pages + IRDMA_SHADOW_PGCNT; | ||
| 3442 | if (total > iwmr->page_cnt) | ||
| 3443 | return -EINVAL; | ||
| 3444 | |||
| 3445 | lvl = req.rq_pages > 1 ? PBLE_LEVEL_1 : PBLE_LEVEL_0; | ||
| 3446 | err = irdma_handle_q_mem(iwdev, &req, iwpbl, lvl); | ||
| 3447 | if (err) | ||
| 3448 | return err; | ||
| 3449 | |||
| 3450 | ucontext = rdma_udata_to_drv_context(udata, struct irdma_ucontext, | ||
| 3451 | ibucontext); | ||
| 3452 | spin_lock_irqsave(&ucontext->srq_reg_mem_list_lock, flags); | ||
| 3453 | list_add_tail(&iwpbl->list, &ucontext->srq_reg_mem_list); | ||
| 3454 | iwpbl->on_list = true; | ||
| 3455 | spin_unlock_irqrestore(&ucontext->srq_reg_mem_list_lock, flags); | ||
| 3456 | |||
| 3457 | return 0; | ||
| 3458 | } | ||
| 3459 | |||
| 2975 | static int irdma_reg_user_mr_type_cq(struct irdma_mem_reg_req req, | 3460 | static int irdma_reg_user_mr_type_cq(struct irdma_mem_reg_req req, |
| 2976 | struct ib_udata *udata, | 3461 | struct ib_udata *udata, |
| 2977 | struct irdma_mr *iwmr) | 3462 | struct irdma_mr *iwmr) |
| @@ -3062,6 +3547,12 @@ static struct ib_mr *irdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 len, · 84 unmodified lines | |||
| 3062 | if (err) | 3547 | if (err) |
| 3063 | goto error; | 3548 | goto error; |
| 3064 | 3549 | ||
| 3550 | break; | ||
| 3551 | case IRDMA_MEMREG_TYPE_SRQ: | ||
| 3552 | err = irdma_reg_user_mr_type_srq(req, udata, iwmr); | ||
| 3553 | if (err) | ||
| 3554 | goto error; | ||
| 3555 | |||
| 3065 | break; | 3556 | break; |
| 3066 | case IRDMA_MEMREG_TYPE_CQ: | 3557 | case IRDMA_MEMREG_TYPE_CQ: |
| 3067 | err = irdma_reg_user_mr_type_cq(req, udata, iwmr); | 3558 | err = irdma_reg_user_mr_type_cq(req, udata, iwmr); |
| @@ -3106,9 +3597,9 @@ static struct ib_mr *irdma_reg_user_mr_dmabuf(struct ib_pd *pd, u64 start, · 38 unmodified lines | |||
| 3106 | 3597 | ||
| 3107 | umem_dmabuf = ib_umem_dmabuf_get_pinned(pd->device, start, len, fd, access); | 3598 | umem_dmabuf = ib_umem_dmabuf_get_pinned(pd->device, start, len, fd, access); |
| 3108 | if (IS_ERR(umem_dmabuf)) { | 3599 | if (IS_ERR(umem_dmabuf)) { |
| 3109 | err = PTR_ERR(umem_dmabuf); | 3600 | ibdev_dbg(&iwdev->ibdev, "Failed to get dmabuf umem[%pe]\n", |
| 3110 | ibdev_dbg(&iwdev->ibdev, "Failed to get dmabuf umem[%d]\n", err); | 3601 | umem_dmabuf); |
| 3111 | return ERR_PTR(err); | 3602 | return ERR_CAST(umem_dmabuf); |
| 3112 | } | 3603 | } |
| 3113 | 3604 | ||
| 3114 | iwmr = irdma_alloc_iwmr(&umem_dmabuf->umem, pd, virt, IRDMA_MEMREG_TYPE_MEM); | 3605 | iwmr = irdma_alloc_iwmr(&umem_dmabuf->umem, pd, virt, IRDMA_MEMREG_TYPE_MEM); |
| @@ -3382,6 +3873,14 @@ static void irdma_del_memlist(struct irdma_mr *iwmr, · 267 unmodified lines | |||
| 3382 | } | 3873 | } |
| 3383 | spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags); | 3874 | spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags); |
| 3384 | break; | 3875 | break; |
| 3876 | case IRDMA_MEMREG_TYPE_SRQ: | ||
| 3877 | spin_lock_irqsave(&ucontext->srq_reg_mem_list_lock, flags); | ||
| 3878 | if (iwpbl->on_list) { | ||
| 3879 | iwpbl->on_list = false; | ||
| 3880 | list_del(&iwpbl->list); | ||
| 3881 | } | ||
| 3882 | spin_unlock_irqrestore(&ucontext->srq_reg_mem_list_lock, flags); | ||
| 3883 | break; | ||
| 3385 | default: | 3884 | default: |
| 3386 | break; | 3885 | break; |
| 3387 | } | 3886 | } |
| @@ -3461,6 +3960,40 @@ static int irdma_post_send(struct ib_qp *ibqp, · 73 unmodified lines | |||
| 3461 | if (ib_wr->send_flags & IB_SEND_FENCE) | 3960 | if (ib_wr->send_flags & IB_SEND_FENCE) |
| 3462 | info.read_fence = true; | 3961 | info.read_fence = true; |
| 3463 | switch (ib_wr->opcode) { | 3962 | switch (ib_wr->opcode) { |
| 3963 | case IB_WR_ATOMIC_CMP_AND_SWP: | ||
| 3964 | if (unlikely(!(dev->hw_attrs.uk_attrs.feature_flags & | ||
| 3965 | IRDMA_FEATURE_ATOMIC_OPS))) { | ||
| 3966 | err = -EINVAL; | ||
| 3967 | break; | ||
| 3968 | } | ||
| 3969 | info.op_type = IRDMA_OP_TYPE_ATOMIC_COMPARE_AND_SWAP; | ||
| 3970 | info.op.atomic_compare_swap.tagged_offset = ib_wr->sg_list[0].addr; | ||
| 3971 | info.op.atomic_compare_swap.remote_tagged_offset = | ||
| 3972 | atomic_wr(ib_wr)->remote_addr; | ||
| 3973 | info.op.atomic_compare_swap.swap_data_bytes = atomic_wr(ib_wr)->swap; | ||
| 3974 | info.op.atomic_compare_swap.compare_data_bytes = | ||
| 3975 | atomic_wr(ib_wr)->compare_add; | ||
| 3976 | info.op.atomic_compare_swap.stag = ib_wr->sg_list[0].lkey; | ||
| 3977 | info.op.atomic_compare_swap.remote_stag = atomic_wr(ib_wr)->rkey; | ||
| 3978 | err = irdma_uk_atomic_compare_swap(ukqp, &info, false); | ||
| 3979 | break; | ||
| 3980 | case IB_WR_ATOMIC_FETCH_AND_ADD: | ||
| 3981 | if (unlikely(!(dev->hw_attrs.uk_attrs.feature_flags & | ||
| 3982 | IRDMA_FEATURE_ATOMIC_OPS))) { | ||
| 3983 | err = -EINVAL; | ||
| 3984 | break; | ||
| 3985 | } | ||
| 3986 | info.op_type = IRDMA_OP_TYPE_ATOMIC_FETCH_AND_ADD; | ||
| 3987 | info.op.atomic_fetch_add.tagged_offset = ib_wr->sg_list[0].addr; | ||
| 3988 | info.op.atomic_fetch_add.remote_tagged_offset = | ||
| 3989 | atomic_wr(ib_wr)->remote_addr; | ||
| 3990 | info.op.atomic_fetch_add.fetch_add_data_bytes = | ||
| 3991 | atomic_wr(ib_wr)->compare_add; | ||
| 3992 | info.op.atomic_fetch_add.stag = ib_wr->sg_list[0].lkey; | ||
| 3993 | info.op.atomic_fetch_add.remote_stag = | ||
| 3994 | atomic_wr(ib_wr)->rkey; | ||
| 3995 | err = irdma_uk_atomic_fetch_add(ukqp, &info, false); | ||
| 3996 | break; | ||
| 3464 | case IB_WR_SEND_WITH_IMM: | 3997 | case IB_WR_SEND_WITH_IMM: |
| 3465 | if (ukqp->qp_caps & IRDMA_SEND_WITH_IMM) { | 3998 | if (ukqp->qp_caps & IRDMA_SEND_WITH_IMM) { |
| 3466 | info.imm_data_valid = true; | 3999 | info.imm_data_valid = true; |
| @@ -3555,7 +4088,9 @@ static int irdma_post_send(struct ib_qp *ibqp, · 88 unmodified lines | |||
| 3555 | 4088 | ||
| 3556 | stag_info.signaled = info.signaled; | 4089 | stag_info.signaled = info.signaled; |
| 3557 | stag_info.read_fence = info.read_fence; | 4090 | stag_info.read_fence = info.read_fence; |
| 3558 | stag_info.access_rights = irdma_get_mr_access(reg_wr(ib_wr)->access); | 4091 | stag_info.access_rights = |
| 4092 | irdma_get_mr_access(reg_wr(ib_wr)->access, | ||
| 4093 | dev->hw_attrs.uk_attrs.hw_rev); | ||
| 3559 | stag_info.stag_key = reg_wr(ib_wr)->key & 0xff; | 4094 | stag_info.stag_key = reg_wr(ib_wr)->key & 0xff; |
| 3560 | stag_info.stag_idx = reg_wr(ib_wr)->key >> 8; | 4095 | stag_info.stag_idx = reg_wr(ib_wr)->key >> 8; |
| 3561 | stag_info.page_size = reg_wr(ib_wr)->mr->page_size; | 4096 | stag_info.page_size = reg_wr(ib_wr)->mr->page_size; |
| @@ -3594,6 +4129,48 @@ static int irdma_post_send(struct ib_qp *ibqp, · 32 unmodified lines | |||
| 3594 | mod_delayed_work(iwqp->iwdev->cleanup_wq, &iwqp->dwork_flush, | 4129 | mod_delayed_work(iwqp->iwdev->cleanup_wq, &iwqp->dwork_flush, |
| 3595 | msecs_to_jiffies(IRDMA_FLUSH_DELAY_MS)); | 4130 | msecs_to_jiffies(IRDMA_FLUSH_DELAY_MS)); |
| 3596 | } | 4131 | } |
| 4132 | |||
| 4133 | if (err) | ||
| 4134 | *bad_wr = ib_wr; | ||
| 4135 | |||
| 4136 | return err; | ||
| 4137 | } | ||
| 4138 | |||
| 4139 | /** | ||
| 4140 | * irdma_post_srq_recv - post receive wr for kernel application | ||
| 4141 | * @ibsrq: ib srq pointer | ||
| 4142 | * @ib_wr: work request for receive | ||
| 4143 | * @bad_wr: bad wr caused an error | ||
| 4144 | */ | ||
| 4145 | static int irdma_post_srq_recv(struct ib_srq *ibsrq, | ||
| 4146 | const struct ib_recv_wr *ib_wr, | ||
| 4147 | const struct ib_recv_wr **bad_wr) | ||
| 4148 | { | ||
| 4149 | struct irdma_srq *iwsrq = to_iwsrq(ibsrq); | ||
| 4150 | struct irdma_srq_uk *uksrq = &iwsrq->sc_srq.srq_uk; | ||
| 4151 | struct irdma_post_rq_info post_recv = {}; | ||
| 4152 | unsigned long flags; | ||
| 4153 | int err = 0; | ||
| 4154 | |||
| 4155 | spin_lock_irqsave(&iwsrq->lock, flags); | ||
| 4156 | while (ib_wr) { | ||
| 4157 | if (ib_wr->num_sge > uksrq->max_srq_frag_cnt) { | ||
| 4158 | err = -EINVAL; | ||
| 4159 | goto out; | ||
| 4160 | } | ||
| 4161 | post_recv.num_sges = ib_wr->num_sge; | ||
| 4162 | post_recv.wr_id = ib_wr->wr_id; | ||
| 4163 | post_recv.sg_list = ib_wr->sg_list; | ||
| 4164 | err = irdma_uk_srq_post_receive(uksrq, &post_recv); | ||
| 4165 | if (err) | ||
| 4166 | goto out; | ||
| 4167 | |||
| 4168 | ib_wr = ib_wr->next; | ||
| 4169 | } | ||
| 4170 | |||
| 4171 | out: | ||
| 4172 | spin_unlock_irqrestore(&iwsrq->lock, flags); | ||
| 4173 | |||
| 3597 | if (err) | 4174 | if (err) |
| 3598 | *bad_wr = ib_wr; | 4175 | *bad_wr = ib_wr; |
| 3599 | 4176 | ||
| @@ -3619,6 +4196,11 @@ static int irdma_post_recv(struct ib_qp *ibqp, · 19 unmodified lines | |||
| 3619 | iwqp = to_iwqp(ibqp); | 4196 | iwqp = to_iwqp(ibqp); |
| 3620 | ukqp = &iwqp->sc_qp.qp_uk; | 4197 | ukqp = &iwqp->sc_qp.qp_uk; |
| 3621 | 4198 | ||
| 4199 | if (ukqp->srq_uk) { | ||
| 4200 | *bad_wr = ib_wr; | ||
| 4201 | return -EINVAL; | ||
| 4202 | } | ||
| 4203 | |||
| 3622 | spin_lock_irqsave(&iwqp->lock, flags); | 4204 | spin_lock_irqsave(&iwqp->lock, flags); |
| 3623 | while (ib_wr) { | 4205 | while (ib_wr) { |
| 3624 | post_recv.num_sges = ib_wr->num_sge; | 4206 | post_recv.num_sges = ib_wr->num_sge; |
| @@ -3671,6 +4253,8 @@ static enum ib_wc_status irdma_flush_err_to_ib_wc_status(enum irdma_flush_opcode · 46 unmodified lines | |||
| 3671 | return IB_WC_MW_BIND_ERR; | 4253 | return IB_WC_MW_BIND_ERR; |
| 3672 | case FLUSH_REM_INV_REQ_ERR: | 4254 | case FLUSH_REM_INV_REQ_ERR: |
| 3673 | return IB_WC_REM_INV_REQ_ERR; | 4255 | return IB_WC_REM_INV_REQ_ERR; |
| 4256 | case FLUSH_RNR_RETRY_EXC_ERR: | ||
| 4257 | return IB_WC_RNR_RETRY_EXC_ERR; | ||
| 3674 | case FLUSH_FATAL_ERR: | 4258 | case FLUSH_FATAL_ERR: |
| 3675 | default: | 4259 | default: |
| 3676 | return IB_WC_FATAL_ERR; | 4260 | return IB_WC_FATAL_ERR; |
| @@ -3727,8 +4311,12 @@ static void irdma_process_cqe(struct ib_wc *entry, · 50 unmodified lines | |||
| 3727 | if (cq_poll_info->q_type == IRDMA_CQE_QTYPE_SQ) { | 4311 | if (cq_poll_info->q_type == IRDMA_CQE_QTYPE_SQ) { |
| 3728 | set_ib_wc_op_sq(cq_poll_info, entry); | 4312 | set_ib_wc_op_sq(cq_poll_info, entry); |
| 3729 | } else { | 4313 | } else { |
| 3730 | set_ib_wc_op_rq(cq_poll_info, entry, | 4314 | if (qp->dev->hw_attrs.uk_attrs.hw_rev <= IRDMA_GEN_2) |
| 3731 | qp->qp_uk.qp_caps & IRDMA_SEND_WITH_IMM); | 4315 | set_ib_wc_op_rq(cq_poll_info, entry, |
| 4316 | qp->qp_uk.qp_caps & IRDMA_SEND_WITH_IMM ? | ||
| 4317 | true : false); | ||
| 4318 | else | ||
| 4319 | set_ib_wc_op_rq_gen_3(cq_poll_info, entry); | ||
| 3732 | if (qp->qp_uk.qp_type != IRDMA_QP_TYPE_ROCE_UD && | 4320 | if (qp->qp_uk.qp_type != IRDMA_QP_TYPE_ROCE_UD && |
| 3733 | cq_poll_info->stag_invalid_set) { | 4321 | cq_poll_info->stag_invalid_set) { |
| 3734 | entry->ex.invalidate_rkey = cq_poll_info->inv_stag; | 4322 | entry->ex.invalidate_rkey = cq_poll_info->inv_stag; |
| @@ -3923,40 +4511,7 @@ static int irdma_req_notify_cq(struct ib_cq *ibcq, · 188 unmodified lines | |||
| 3923 | return ret; | 4511 | return ret; |
| 3924 | } | 4512 | } |
| 3925 | 4513 | ||
| 3926 | static int irdma_roce_port_immutable(struct ib_device *ibdev, u32 port_num, | 4514 | static const struct rdma_stat_desc irdma_hw_stat_descs[] = { |
| 3927 | struct ib_port_immutable *immutable) | ||
| 3928 | { | ||
| 3929 | struct ib_port_attr attr; | ||
| 3930 | int err; | ||
| 3931 | |||
| 3932 | immutable->core_cap_flags = RDMA_CORE_PORT_IBA_ROCE_UDP_ENCAP; | ||
| 3933 | err = ib_query_port(ibdev, port_num, &attr); | ||
| 3934 | if (err) | ||
| 3935 | return err; | ||
| 3936 | |||
| 3937 | immutable->max_mad_size = IB_MGMT_MAD_SIZE; | ||
| 3938 | immutable->pkey_tbl_len = attr.pkey_tbl_len; | ||
| 3939 | immutable->gid_tbl_len = attr.gid_tbl_len; | ||
| 3940 | |||
| 3941 | return 0; | ||
| 3942 | } | ||
| 3943 | |||
| 3944 | static int irdma_iw_port_immutable(struct ib_device *ibdev, u32 port_num, | ||
| 3945 | struct ib_port_immutable *immutable) | ||
| 3946 | { | ||
| 3947 | struct ib_port_attr attr; | ||
| 3948 | int err; | ||
| 3949 | |||
| 3950 | immutable->core_cap_flags = RDMA_CORE_PORT_IWARP; | ||
| 3951 | err = ib_query_port(ibdev, port_num, &attr); | ||
| 3952 | if (err) | ||
| 3953 | return err; | ||
| 3954 | immutable->gid_tbl_len = attr.gid_tbl_len; | ||
| 3955 | |||
| 3956 | return 0; | ||
| 3957 | } | ||
| 3958 | |||
| 3959 | static const struct rdma_stat_desc irdma_hw_stat_names[] = { | ||
| 3960 | /* gen1 - 32-bit */ | 4515 | /* gen1 - 32-bit */ |
| 3961 | [IRDMA_HW_STAT_INDEX_IP4RXDISCARD].name = "ip4InDiscards", | 4516 | [IRDMA_HW_STAT_INDEX_IP4RXDISCARD].name = "ip4InDiscards", |
| 3962 | [IRDMA_HW_STAT_INDEX_IP4RXTRUNC].name = "ip4InTruncatedPkts", | 4517 | [IRDMA_HW_STAT_INDEX_IP4RXTRUNC].name = "ip4InTruncatedPkts", |
| @@ -3964,9 +4519,6 @@ static const struct rdma_stat_desc irdma_hw_stat_names[] = { · 1 unmodified line | |||
| 3964 | [IRDMA_HW_STAT_INDEX_IP6RXDISCARD].name = "ip6InDiscards", | 4519 | [IRDMA_HW_STAT_INDEX_IP6RXDISCARD].name = "ip6InDiscards", |
| 3965 | [IRDMA_HW_STAT_INDEX_IP6RXTRUNC].name = "ip6InTruncatedPkts", | 4520 | [IRDMA_HW_STAT_INDEX_IP6RXTRUNC].name = "ip6InTruncatedPkts", |
| 3966 | [IRDMA_HW_STAT_INDEX_IP6TXNOROUTE].name = "ip6OutNoRoutes", | 4521 | [IRDMA_HW_STAT_INDEX_IP6TXNOROUTE].name = "ip6OutNoRoutes", |
| 3967 | [IRDMA_HW_STAT_INDEX_TCPRTXSEG].name = "tcpRetransSegs", | ||
| 3968 | [IRDMA_HW_STAT_INDEX_TCPRXOPTERR].name = "tcpInOptErrors", | ||
| 3969 | [IRDMA_HW_STAT_INDEX_TCPRXPROTOERR].name = "tcpInProtoErrors", | ||
| 3970 | [IRDMA_HW_STAT_INDEX_RXVLANERR].name = "rxVlanErrors", | 4522 | [IRDMA_HW_STAT_INDEX_RXVLANERR].name = "rxVlanErrors", |
| 3971 | /* gen1 - 64-bit */ | 4523 | /* gen1 - 64-bit */ |
| 3972 | [IRDMA_HW_STAT_INDEX_IP4RXOCTS].name = "ip4InOctets", | 4524 | [IRDMA_HW_STAT_INDEX_IP4RXOCTS].name = "ip4InOctets", |
| @@ -3985,16 +4537,14 @@ static const struct rdma_stat_desc irdma_hw_stat_names[] = { · 12 unmodified lines | |||
| 3985 | [IRDMA_HW_STAT_INDEX_IP6TXPKTS].name = "ip6OutPkts", | 4537 | [IRDMA_HW_STAT_INDEX_IP6TXPKTS].name = "ip6OutPkts", |
| 3986 | [IRDMA_HW_STAT_INDEX_IP6TXFRAGS].name = "ip6OutSegRqd", | 4538 | [IRDMA_HW_STAT_INDEX_IP6TXFRAGS].name = "ip6OutSegRqd", |
| 3987 | [IRDMA_HW_STAT_INDEX_IP6TXMCPKTS].name = "ip6OutMcastPkts", | 4539 | [IRDMA_HW_STAT_INDEX_IP6TXMCPKTS].name = "ip6OutMcastPkts", |
| 3988 | [IRDMA_HW_STAT_INDEX_TCPRXSEGS].name = "tcpInSegs", | 4540 | [IRDMA_HW_STAT_INDEX_RDMARXRDS].name = "InRdmaReads", |
| 3989 | [IRDMA_HW_STAT_INDEX_TCPTXSEG].name = "tcpOutSegs", | 4541 | [IRDMA_HW_STAT_INDEX_RDMARXSNDS].name = "InRdmaSends", |
| 3990 | [IRDMA_HW_STAT_INDEX_RDMARXRDS].name = "iwInRdmaReads", | 4542 | [IRDMA_HW_STAT_INDEX_RDMARXWRS].name = "InRdmaWrites", |
| 3991 | [IRDMA_HW_STAT_INDEX_RDMARXSNDS].name = "iwInRdmaSends", | 4543 | [IRDMA_HW_STAT_INDEX_RDMATXRDS].name = "OutRdmaReads", |
| 3992 | [IRDMA_HW_STAT_INDEX_RDMARXWRS].name = "iwInRdmaWrites", | 4544 | [IRDMA_HW_STAT_INDEX_RDMATXSNDS].name = "OutRdmaSends", |
| 3993 | [IRDMA_HW_STAT_INDEX_RDMATXRDS].name = "iwOutRdmaReads", | 4545 | [IRDMA_HW_STAT_INDEX_RDMATXWRS].name = "OutRdmaWrites", |
| 3994 | [IRDMA_HW_STAT_INDEX_RDMATXSNDS].name = "iwOutRdmaSends", | 4546 | [IRDMA_HW_STAT_INDEX_RDMAVBND].name = "RdmaBnd", |
| 3995 | [IRDMA_HW_STAT_INDEX_RDMATXWRS].name = "iwOutRdmaWrites", | 4547 | [IRDMA_HW_STAT_INDEX_RDMAVINV].name = "RdmaInv", |
| 3996 | [IRDMA_HW_STAT_INDEX_RDMAVBND].name = "iwRdmaBnd", | ||
| 3997 | [IRDMA_HW_STAT_INDEX_RDMAVINV].name = "iwRdmaInv", | ||
| 3998 | 4548 | ||
| 3999 | /* gen2 - 32-bit */ | 4549 | /* gen2 - 32-bit */ |
| 4000 | [IRDMA_HW_STAT_INDEX_RXRPCNPHANDLED].name = "cnpHandled", | 4550 | [IRDMA_HW_STAT_INDEX_RXRPCNPHANDLED].name = "cnpHandled", |
| @@ -4008,9 +4558,59 @@ static const struct rdma_stat_desc irdma_hw_stat_names[] = { · 7 unmodified lines | |||
| 4008 | [IRDMA_HW_STAT_INDEX_UDPRXPKTS].name = "RxUDP", | 4558 | [IRDMA_HW_STAT_INDEX_UDPRXPKTS].name = "RxUDP", |
| 4009 | [IRDMA_HW_STAT_INDEX_UDPTXPKTS].name = "TxUDP", | 4559 | [IRDMA_HW_STAT_INDEX_UDPTXPKTS].name = "TxUDP", |
| 4010 | [IRDMA_HW_STAT_INDEX_RXNPECNMARKEDPKTS].name = "RxECNMrkd", | 4560 | [IRDMA_HW_STAT_INDEX_RXNPECNMARKEDPKTS].name = "RxECNMrkd", |
| 4011 | 4561 | [IRDMA_HW_STAT_INDEX_TCPRTXSEG].name = "RetransSegs", | |
| 4562 | [IRDMA_HW_STAT_INDEX_TCPRXOPTERR].name = "InOptErrors", | ||
| 4563 | [IRDMA_HW_STAT_INDEX_TCPRXPROTOERR].name = "InProtoErrors", | ||
| 4564 | [IRDMA_HW_STAT_INDEX_TCPRXSEGS].name = "InSegs", | ||
| 4565 | [IRDMA_HW_STAT_INDEX_TCPTXSEG].name = "OutSegs", | ||
| 4566 | |||
| 4567 | /* gen3 */ | ||
| 4568 | [IRDMA_HW_STAT_INDEX_RNR_SENT].name = "RNR sent", | ||
| 4569 | [IRDMA_HW_STAT_INDEX_RNR_RCVD].name = "RNR received", | ||
| 4570 | [IRDMA_HW_STAT_INDEX_RDMAORDLMTCNT].name = "ord limit count", | ||
| 4571 | [IRDMA_HW_STAT_INDEX_RDMAIRDLMTCNT].name = "ird limit count", | ||
| 4572 | [IRDMA_HW_STAT_INDEX_RDMARXATS].name = "Rx atomics", | ||
| 4573 | [IRDMA_HW_STAT_INDEX_RDMATXATS].name = "Tx atomics", | ||
| 4574 | [IRDMA_HW_STAT_INDEX_NAKSEQERR].name = "Nak Sequence Error", | ||
| 4575 | [IRDMA_HW_STAT_INDEX_NAKSEQERR_IMPLIED].name = "Nak Sequence Error Implied", | ||
| 4576 | [IRDMA_HW_STAT_INDEX_RTO].name = "RTO", | ||
| 4577 | [IRDMA_HW_STAT_INDEX_RXOOOPKTS].name = "Rcvd Out of order packets", | ||
| 4578 | [IRDMA_HW_STAT_INDEX_ICRCERR].name = "CRC errors", | ||
| 4012 | }; | 4579 | }; |
| 4013 | 4580 | ||
| 4581 | static int irdma_roce_port_immutable(struct ib_device *ibdev, u32 port_num, | ||
| 4582 | struct ib_port_immutable *immutable) | ||
| 4583 | { | ||
| 4584 | struct ib_port_attr attr; | ||
| 4585 | int err; | ||
| 4586 | |||
| 4587 | immutable->core_cap_flags = RDMA_CORE_PORT_IBA_ROCE_UDP_ENCAP; | ||
| 4588 | err = ib_query_port(ibdev, port_num, &attr); | ||
| 4589 | if (err) | ||
| 4590 | return err; | ||
| 4591 | |||
| 4592 | immutable->max_mad_size = IB_MGMT_MAD_SIZE; | ||
| 4593 | immutable->pkey_tbl_len = attr.pkey_tbl_len; | ||
| 4594 | immutable->gid_tbl_len = attr.gid_tbl_len; | ||
| 4595 | |||
| 4596 | return 0; | ||
| 4597 | } | ||
| 4598 | |||
| 4599 | static int irdma_iw_port_immutable(struct ib_device *ibdev, u32 port_num, | ||
| 4600 | struct ib_port_immutable *immutable) | ||
| 4601 | { | ||
| 4602 | struct ib_port_attr attr; | ||
| 4603 | int err; | ||
| 4604 | |||
| 4605 | immutable->core_cap_flags = RDMA_CORE_PORT_IWARP; | ||
| 4606 | err = ib_query_port(ibdev, port_num, &attr); | ||
| 4607 | if (err) | ||
| 4608 | return err; | ||
| 4609 | immutable->gid_tbl_len = attr.gid_tbl_len; | ||
| 4610 | |||
| 4611 | return 0; | ||
| 4612 | } | ||
| 4613 | |||
| 4014 | static void irdma_get_dev_fw_str(struct ib_device *dev, char *str) | 4614 | static void irdma_get_dev_fw_str(struct ib_device *dev, char *str) |
| 4015 | { | 4615 | { |
| 4016 | struct irdma_device *iwdev = to_iwdev(dev); | 4616 | struct irdma_device *iwdev = to_iwdev(dev); |
| @@ -4034,7 +4634,7 @@ static struct rdma_hw_stats *irdma_alloc_hw_port_stats(struct ib_device *ibdev, · 17 unmodified lines | |||
| 4034 | int num_counters = dev->hw_attrs.max_stat_idx; | 4634 | int num_counters = dev->hw_attrs.max_stat_idx; |
| 4035 | unsigned long lifespan = RDMA_HW_STATS_DEFAULT_LIFESPAN; | 4635 | unsigned long lifespan = RDMA_HW_STATS_DEFAULT_LIFESPAN; |
| 4036 | 4636 | ||
| 4037 | return rdma_alloc_hw_stats_struct(irdma_hw_stat_names, num_counters, | 4637 | return rdma_alloc_hw_stats_struct(irdma_hw_stat_descs, num_counters, |
| 4038 | lifespan); | 4638 | lifespan); |
| 4039 | } | 4639 | } |
| 4040 | 4640 | ||
| @@ -4539,7 +5139,7 @@ static bool irdma_ah_exists(struct irdma_device *iwdev, · 498 unmodified lines | |||
| 4539 | new_ah->sc_ah.ah_info.dest_ip_addr[2] ^ | 5139 | new_ah->sc_ah.ah_info.dest_ip_addr[2] ^ |
| 4540 | new_ah->sc_ah.ah_info.dest_ip_addr[3]; | 5140 | new_ah->sc_ah.ah_info.dest_ip_addr[3]; |
| 4541 | 5141 | ||
| 4542 | hash_for_each_possible(iwdev->ah_hash_tbl, ah, list, key) { | 5142 | hash_for_each_possible(iwdev->rf->ah_hash_tbl, ah, list, key) { |
| 4543 | /* Set ah_valid and ah_id the same so memcmp can work */ | 5143 | /* Set ah_valid and ah_id the same so memcmp can work */ |
| 4544 | new_ah->sc_ah.ah_info.ah_idx = ah->sc_ah.ah_info.ah_idx; | 5144 | new_ah->sc_ah.ah_info.ah_idx = ah->sc_ah.ah_info.ah_idx; |
| 4545 | new_ah->sc_ah.ah_info.ah_valid = ah->sc_ah.ah_info.ah_valid; | 5145 | new_ah->sc_ah.ah_info.ah_valid = ah->sc_ah.ah_info.ah_valid; |
| @@ -4565,14 +5165,14 @@ static int irdma_destroy_ah(struct ib_ah *ibah, u32 ah_flags) · 19 unmodified lines | |||
| 4565 | struct irdma_ah *ah = to_iwah(ibah); | 5165 | struct irdma_ah *ah = to_iwah(ibah); |
| 4566 | 5166 | ||
| 4567 | if ((ah_flags & RDMA_DESTROY_AH_SLEEPABLE) && ah->parent_ah) { | 5167 | if ((ah_flags & RDMA_DESTROY_AH_SLEEPABLE) && ah->parent_ah) { |
| 4568 | mutex_lock(&iwdev->ah_tbl_lock); | 5168 | mutex_lock(&iwdev->rf->ah_tbl_lock); |
| 4569 | if (!refcount_dec_and_test(&ah->parent_ah->refcnt)) { | 5169 | if (!refcount_dec_and_test(&ah->parent_ah->refcnt)) { |
| 4570 | mutex_unlock(&iwdev->ah_tbl_lock); | 5170 | mutex_unlock(&iwdev->rf->ah_tbl_lock); |
| 4571 | return 0; | 5171 | return 0; |
| 4572 | } | 5172 | } |
| 4573 | hash_del(&ah->parent_ah->list); | 5173 | hash_del(&ah->parent_ah->list); |
| 4574 | kfree(ah->parent_ah); | 5174 | kfree(ah->parent_ah); |
| 4575 | mutex_unlock(&iwdev->ah_tbl_lock); | 5175 | mutex_unlock(&iwdev->rf->ah_tbl_lock); |
| 4576 | } | 5176 | } |
| 4577 | 5177 | ||
| 4578 | irdma_ah_cqp_op(iwdev->rf, &ah->sc_ah, IRDMA_OP_AH_DESTROY, | 5178 | irdma_ah_cqp_op(iwdev->rf, &ah->sc_ah, IRDMA_OP_AH_DESTROY, |
| @@ -4609,11 +5209,11 @@ static int irdma_create_user_ah(struct ib_ah *ibah, · 30 unmodified lines | |||
| 4609 | err = irdma_setup_ah(ibah, attr); | 5209 | err = irdma_setup_ah(ibah, attr); |
| 4610 | if (err) | 5210 | if (err) |
| 4611 | return err; | 5211 | return err; |
| 4612 | mutex_lock(&iwdev->ah_tbl_lock); | 5212 | mutex_lock(&iwdev->rf->ah_tbl_lock); |
| 4613 | if (!irdma_ah_exists(iwdev, ah)) { | 5213 | if (!irdma_ah_exists(iwdev, ah)) { |
| 4614 | err = irdma_create_hw_ah(iwdev, ah, true); | 5214 | err = irdma_create_hw_ah(iwdev, ah, true); |
| 4615 | if (err) { | 5215 | if (err) { |
| 4616 | mutex_unlock(&iwdev->ah_tbl_lock); | 5216 | mutex_unlock(&iwdev->rf->ah_tbl_lock); |
| 4617 | return err; | 5217 | return err; |
| 4618 | } | 5218 | } |
| 4619 | /* Add new AH to list */ | 5219 | /* Add new AH to list */ |
| @@ -4625,11 +5225,11 @@ static int irdma_create_user_ah(struct ib_ah *ibah, · 5 unmodified lines | |||
| 4625 | parent_ah->sc_ah.ah_info.dest_ip_addr[3]; | 5225 | parent_ah->sc_ah.ah_info.dest_ip_addr[3]; |
| 4626 | 5226 | ||
| 4627 | ah->parent_ah = parent_ah; | 5227 | ah->parent_ah = parent_ah; |
| 4628 | hash_add(iwdev->ah_hash_tbl, &parent_ah->list, key); | 5228 | hash_add(iwdev->rf->ah_hash_tbl, &parent_ah->list, key); |
| 4629 | refcount_set(&parent_ah->refcnt, 1); | 5229 | refcount_set(&parent_ah->refcnt, 1); |
| 4630 | } | 5230 | } |
| 4631 | } | 5231 | } |
| 4632 | mutex_unlock(&iwdev->ah_tbl_lock); | 5232 | mutex_unlock(&iwdev->rf->ah_tbl_lock); |
| 4633 | 5233 | ||
| 4634 | uresp.ah_id = ah->sc_ah.ah_info.ah_idx; | 5234 | uresp.ah_id = ah->sc_ah.ah_info.ah_idx; |
| 4635 | err = ib_copy_to_udata(udata, &uresp, min(sizeof(uresp), udata->outlen)); | 5235 | err = ib_copy_to_udata(udata, &uresp, min(sizeof(uresp), udata->outlen)); |
| @@ -4691,6 +5291,20 @@ static enum rdma_link_layer irdma_get_link_layer(struct ib_device *ibdev, · 55 unmodified lines | |||
| 4691 | return IB_LINK_LAYER_ETHERNET; | 5291 | return IB_LINK_LAYER_ETHERNET; |
| 4692 | } | 5292 | } |
| 4693 | 5293 | ||
| 5294 | static const struct ib_device_ops irdma_gen1_dev_ops = { | ||
| 5295 | .dealloc_driver = irdma_ib_dealloc_device, | ||
| 5296 | }; | ||
| 5297 | |||
| 5298 | static const struct ib_device_ops irdma_gen3_dev_ops = { | ||
| 5299 | .alloc_mw = irdma_alloc_mw, | ||
| 5300 | .create_srq = irdma_create_srq, | ||
| 5301 | .dealloc_mw = irdma_dealloc_mw, | ||
| 5302 | .destroy_srq = irdma_destroy_srq, | ||
| 5303 | .modify_srq = irdma_modify_srq, | ||
| 5304 | .post_srq_recv = irdma_post_srq_recv, | ||
| 5305 | .query_srq = irdma_query_srq, | ||
| 5306 | }; | ||
| 5307 | |||
| 4694 | static const struct ib_device_ops irdma_roce_dev_ops = { | 5308 | static const struct ib_device_ops irdma_roce_dev_ops = { |
| 4695 | .attach_mcast = irdma_attach_mcast, | 5309 | .attach_mcast = irdma_attach_mcast, |
| 4696 | .create_ah = irdma_create_ah, | 5310 | .create_ah = irdma_create_ah, |
| @@ -4725,7 +5339,6 @@ static const struct ib_device_ops irdma_dev_ops = { · 28 unmodified lines | |||
| 4725 | 5339 | ||
| 4726 | .alloc_hw_port_stats = irdma_alloc_hw_port_stats, | 5340 | .alloc_hw_port_stats = irdma_alloc_hw_port_stats, |
| 4727 | .alloc_mr = irdma_alloc_mr, | 5341 | .alloc_mr = irdma_alloc_mr, |
| 4728 | .alloc_mw = irdma_alloc_mw, | ||
| 4729 | .alloc_pd = irdma_alloc_pd, | 5342 | .alloc_pd = irdma_alloc_pd, |
| 4730 | .alloc_ucontext = irdma_alloc_ucontext, | 5343 | .alloc_ucontext = irdma_alloc_ucontext, |
| 4731 | .create_cq = irdma_create_cq, | 5344 | .create_cq = irdma_create_cq, |
| @@ -4761,6 +5374,7 @@ static const struct ib_device_ops irdma_dev_ops = { · 29 unmodified lines | |||
| 4761 | INIT_RDMA_OBJ_SIZE(ib_cq, irdma_cq, ibcq), | 5374 | INIT_RDMA_OBJ_SIZE(ib_cq, irdma_cq, ibcq), |
| 4762 | INIT_RDMA_OBJ_SIZE(ib_mw, irdma_mr, ibmw), | 5375 | INIT_RDMA_OBJ_SIZE(ib_mw, irdma_mr, ibmw), |
| 4763 | INIT_RDMA_OBJ_SIZE(ib_qp, irdma_qp, ibqp), | 5376 | INIT_RDMA_OBJ_SIZE(ib_qp, irdma_qp, ibqp), |
| 5377 | INIT_RDMA_OBJ_SIZE(ib_srq, irdma_srq, ibsrq), | ||
| 4764 | }; | 5378 | }; |
| 4765 | 5379 | ||
| 4766 | /** | 5380 | /** |
| @@ -4808,6 +5422,10 @@ static void irdma_init_rdma_device(struct irdma_device *iwdev) · 41 unmodified lines | |||
| 4808 | iwdev->ibdev.num_comp_vectors = iwdev->rf->ceqs_count; | 5422 | iwdev->ibdev.num_comp_vectors = iwdev->rf->ceqs_count; |
| 4809 | iwdev->ibdev.dev.parent = &pcidev->dev; | 5423 | iwdev->ibdev.dev.parent = &pcidev->dev; |
| 4810 | ib_set_device_ops(&iwdev->ibdev, &irdma_dev_ops); | 5424 | ib_set_device_ops(&iwdev->ibdev, &irdma_dev_ops); |
| 5425 | if (iwdev->rf->rdma_ver == IRDMA_GEN_1) | ||
| 5426 | ib_set_device_ops(&iwdev->ibdev, &irdma_gen1_dev_ops); | ||
| 5427 | if (iwdev->rf->rdma_ver >= IRDMA_GEN_3) | ||
| 5428 | ib_set_device_ops(&iwdev->ibdev, &irdma_gen3_dev_ops); | ||
| 4811 | } | 5429 | } |
| 4812 | 5430 | ||
| 4813 | /** | 5431 | /** |
| @@ -4879,5 +5497,9 @@ void irdma_ib_dealloc_device(struct ib_device *ibdev) · 65 unmodified lines | |||
| 4879 | struct irdma_device *iwdev = to_iwdev(ibdev); | 5497 | struct irdma_device *iwdev = to_iwdev(ibdev); |
| 4880 | 5498 | ||
| 4881 | irdma_rt_deinit_hw(iwdev); | 5499 | irdma_rt_deinit_hw(iwdev); |
| 4882 | irdma_ctrl_deinit_hw(iwdev->rf); | 5500 | if (!iwdev->is_vport) { |
| 5501 | irdma_ctrl_deinit_hw(iwdev->rf); | ||
| 5502 | if (iwdev->rf->vchnl_wq) | ||
| 5503 | destroy_workqueue(iwdev->rf->vchnl_wq); | ||
| 5504 | } | ||
| 4883 | } | 5505 | } |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | |||
| 8 | 8 | ||
| 9 | #define IRDMA_PKEY_TBL_SZ 1 | 9 | #define IRDMA_PKEY_TBL_SZ 1 |
| 10 | #define IRDMA_DEFAULT_PKEY 0xFFFF | 10 | #define IRDMA_DEFAULT_PKEY 0xFFFF |
| 11 | #define IRDMA_SHADOW_PGCNT 1 | ||
| 11 | 12 | ||
| 12 | struct irdma_ucontext { | 13 | struct irdma_ucontext { |
| 13 | struct ib_ucontext ibucontext; | 14 | struct ib_ucontext ibucontext; |
| @@ -17,6 +18,8 @@ struct irdma_ucontext { · 3 unmodified lines | |||
| 17 | spinlock_t cq_reg_mem_list_lock; /* protect CQ memory list */ | 18 | spinlock_t cq_reg_mem_list_lock; /* protect CQ memory list */ |
| 18 | struct list_head qp_reg_mem_list; | 19 | struct list_head qp_reg_mem_list; |
| 19 | spinlock_t qp_reg_mem_list_lock; /* protect QP memory list */ | 20 | spinlock_t qp_reg_mem_list_lock; /* protect QP memory list */ |
| 21 | struct list_head srq_reg_mem_list; | ||
| 22 | spinlock_t srq_reg_mem_list_lock; /* protect SRQ memory list */ | ||
| 20 | int abi_ver; | 23 | int abi_ver; |
| 21 | u8 legacy_mode : 1; | 24 | u8 legacy_mode : 1; |
| 22 | u8 use_raw_attrs : 1; | 25 | u8 use_raw_attrs : 1; |
| @@ -65,10 +68,16 @@ struct irdma_cq_mr { · 42 unmodified lines | |||
| 65 | bool split; | 68 | bool split; |
| 66 | }; | 69 | }; |
| 67 | 70 | ||
| 71 | struct irdma_srq_mr { | ||
| 72 | struct irdma_hmc_pble srq_pbl; | ||
| 73 | dma_addr_t shadow; | ||
| 74 | }; | ||
| 75 | |||
| 68 | struct irdma_qp_mr { | 76 | struct irdma_qp_mr { |
| 69 | struct irdma_hmc_pble sq_pbl; | 77 | struct irdma_hmc_pble sq_pbl; |
| 70 | struct irdma_hmc_pble rq_pbl; | 78 | struct irdma_hmc_pble rq_pbl; |
| 71 | dma_addr_t shadow; | 79 | dma_addr_t shadow; |
| 80 | dma_addr_t rq_pa; | ||
| 72 | struct page *sq_page; | 81 | struct page *sq_page; |
| 73 | }; | 82 | }; |
| 74 | 83 | ||
| @@ -85,6 +94,7 @@ struct irdma_pbl { · 10 unmodified lines | |||
| 85 | union { | 94 | union { |
| 86 | struct irdma_qp_mr qp_mr; | 95 | struct irdma_qp_mr qp_mr; |
| 87 | struct irdma_cq_mr cq_mr; | 96 | struct irdma_cq_mr cq_mr; |
| 97 | struct irdma_srq_mr srq_mr; | ||
| 88 | }; | 98 | }; |
| 89 | 99 | ||
| 90 | bool pbl_allocated:1; | 100 | bool pbl_allocated:1; |
| @@ -112,24 +122,33 @@ struct irdma_mr { · 21 unmodified lines | |||
| 112 | struct irdma_pbl iwpbl; | 122 | struct irdma_pbl iwpbl; |
| 113 | }; | 123 | }; |
| 114 | 124 | ||
| 125 | struct irdma_srq { | ||
| 126 | struct ib_srq ibsrq; | ||
| 127 | struct irdma_sc_srq sc_srq __aligned(64); | ||
| 128 | struct irdma_dma_mem kmem; | ||
| 129 | u64 *srq_wrid_mem; | ||
| 130 | refcount_t refcnt; | ||
| 131 | spinlock_t lock; /* for poll srq */ | ||
| 132 | struct irdma_pbl *iwpbl; | ||
| 133 | struct irdma_sge *sg_list; | ||
| 134 | u16 srq_head; | ||
| 135 | u32 srq_num; | ||
| 136 | u32 max_wr; | ||
| 137 | bool user_mode:1; | ||
| 138 | }; | ||
| 139 | |||
| 115 | struct irdma_cq { | 140 | struct irdma_cq { |
| 116 | struct ib_cq ibcq; | 141 | struct ib_cq ibcq; |
| 117 | struct irdma_sc_cq sc_cq; | 142 | struct irdma_sc_cq sc_cq; |
| 118 | u16 cq_head; | ||
| 119 | u16 cq_size; | ||
| 120 | u16 cq_num; | 143 | u16 cq_num; |
| 121 | bool user_mode; | 144 | bool user_mode; |
| 122 | atomic_t armed; | 145 | atomic_t armed; |
| 123 | enum irdma_cmpl_notify last_notify; | 146 | enum irdma_cmpl_notify last_notify; |
| 124 | u32 polled_cmpls; | ||
| 125 | u32 cq_mem_size; | ||
| 126 | struct irdma_dma_mem kmem; | 147 | struct irdma_dma_mem kmem; |
| 127 | struct irdma_dma_mem kmem_shadow; | 148 | struct irdma_dma_mem kmem_shadow; |
| 128 | struct completion free_cq; | 149 | struct completion free_cq; |
| 129 | refcount_t refcnt; | 150 | refcount_t refcnt; |
| 130 | spinlock_t lock; /* for poll cq */ | 151 | spinlock_t lock; /* for poll cq */ |
| 131 | struct irdma_pbl *iwpbl; | ||
| 132 | struct irdma_pbl *iwpbl_shadow; | ||
| 133 | struct list_head resize_list; | 152 | struct list_head resize_list; |
| 134 | struct irdma_cq_poll_info cur_cqe; | 153 | struct irdma_cq_poll_info cur_cqe; |
| 135 | struct list_head cmpl_generated; | 154 | struct list_head cmpl_generated; |
| @@ -259,6 +278,12 @@ static inline void set_ib_wc_op_sq(struct irdma_cq_poll_info *cq_poll_info, · 123 unmodified lines | |||
| 259 | case IRDMA_OP_TYPE_FAST_REG_NSMR: | 278 | case IRDMA_OP_TYPE_FAST_REG_NSMR: |
| 260 | entry->opcode = IB_WC_REG_MR; | 279 | entry->opcode = IB_WC_REG_MR; |
| 261 | break; | 280 | break; |
| 281 | case IRDMA_OP_TYPE_ATOMIC_COMPARE_AND_SWAP: | ||
| 282 | entry->opcode = IB_WC_COMP_SWAP; | ||
| 283 | break; | ||
| 284 | case IRDMA_OP_TYPE_ATOMIC_FETCH_AND_ADD: | ||
| 285 | entry->opcode = IB_WC_FETCH_ADD; | ||
| 286 | break; | ||
| 262 | case IRDMA_OP_TYPE_INV_STAG: | 287 | case IRDMA_OP_TYPE_INV_STAG: |
| 263 | entry->opcode = IB_WC_LOCAL_INV; | 288 | entry->opcode = IB_WC_LOCAL_INV; |
| 264 | break; | 289 | break; |
| @@ -267,6 +292,19 @@ static inline void set_ib_wc_op_sq(struct irdma_cq_poll_info *cq_poll_info, · 2 unmodified lines | |||
| 267 | } | 292 | } |
| 268 | } | 293 | } |
| 269 | 294 | ||
| 295 | static inline void set_ib_wc_op_rq_gen_3(struct irdma_cq_poll_info *info, | ||
| 296 | struct ib_wc *entry) | ||
| 297 | { | ||
| 298 | switch (info->op_type) { | ||
| 299 | case IRDMA_OP_TYPE_RDMA_WRITE: | ||
| 300 | case IRDMA_OP_TYPE_RDMA_WRITE_SOL: | ||
| 301 | entry->opcode = IB_WC_RECV_RDMA_WITH_IMM; | ||
| 302 | break; | ||
| 303 | default: | ||
| 304 | entry->opcode = IB_WC_RECV; | ||
| 305 | } | ||
| 306 | } | ||
| 307 | |||
| 270 | static inline void set_ib_wc_op_rq(struct irdma_cq_poll_info *cq_poll_info, | 308 | static inline void set_ib_wc_op_rq(struct irdma_cq_poll_info *cq_poll_info, |
| 271 | struct ib_wc *entry, bool send_imm_support) | 309 | struct ib_wc *entry, bool send_imm_support) |
| 272 | { | 310 | { |
| @@ -0,0 +1,618 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB | ||
| 2 | /* Copyright (c) 2015 - 2024 Intel Corporation */ | ||
| 3 | |||
| 4 | #include "osdep.h" | ||
| 5 | #include "hmc.h" | ||
| 6 | #include "defs.h" | ||
| 7 | #include "type.h" | ||
| 8 | #include "protos.h" | ||
| 9 | #include "virtchnl.h" | ||
| 10 | #include "ws.h" | ||
| 11 | #include "i40iw_hw.h" | ||
| 12 | #include "ig3rdma_hw.h" | ||
| 13 | |||
| 14 | struct vchnl_reg_map_elem { | ||
| 15 | u16 reg_id; | ||
| 16 | u16 reg_idx; | ||
| 17 | bool pg_rel; | ||
| 18 | }; | ||
| 19 | |||
| 20 | struct vchnl_regfld_map_elem { | ||
| 21 | u16 regfld_id; | ||
| 22 | u16 regfld_idx; | ||
| 23 | }; | ||
| 24 | |||
| 25 | static struct vchnl_reg_map_elem vchnl_reg_map[] = { | ||
| 26 | {IRDMA_VCHNL_REG_ID_CQPTAIL, IRDMA_CQPTAIL, false}, | ||
| 27 | {IRDMA_VCHNL_REG_ID_CQPDB, IRDMA_CQPDB, false}, | ||
| 28 | {IRDMA_VCHNL_REG_ID_CCQPSTATUS, IRDMA_CCQPSTATUS, false}, | ||
| 29 | {IRDMA_VCHNL_REG_ID_CCQPHIGH, IRDMA_CCQPHIGH, false}, | ||
| 30 | {IRDMA_VCHNL_REG_ID_CCQPLOW, IRDMA_CCQPLOW, false}, | ||
| 31 | {IRDMA_VCHNL_REG_ID_CQARM, IRDMA_CQARM, false}, | ||
| 32 | {IRDMA_VCHNL_REG_ID_CQACK, IRDMA_CQACK, false}, | ||
| 33 | {IRDMA_VCHNL_REG_ID_AEQALLOC, IRDMA_AEQALLOC, false}, | ||
| 34 | {IRDMA_VCHNL_REG_ID_CQPERRCODES, IRDMA_CQPERRCODES, false}, | ||
| 35 | {IRDMA_VCHNL_REG_ID_WQEALLOC, IRDMA_WQEALLOC, false}, | ||
| 36 | {IRDMA_VCHNL_REG_ID_DB_ADDR_OFFSET, IRDMA_DB_ADDR_OFFSET, false }, | ||
| 37 | {IRDMA_VCHNL_REG_ID_DYN_CTL, IRDMA_GLINT_DYN_CTL, false }, | ||
| 38 | {IRDMA_VCHNL_REG_INV_ID, IRDMA_VCHNL_REG_INV_ID, false } | ||
| 39 | }; | ||
| 40 | |||
| 41 | static struct vchnl_regfld_map_elem vchnl_regfld_map[] = { | ||
| 42 | {IRDMA_VCHNL_REGFLD_ID_CCQPSTATUS_CQP_OP_ERR, IRDMA_CCQPSTATUS_CCQP_ERR_M}, | ||
| 43 | {IRDMA_VCHNL_REGFLD_ID_CCQPSTATUS_CCQP_DONE, IRDMA_CCQPSTATUS_CCQP_DONE_M}, | ||
| 44 | {IRDMA_VCHNL_REGFLD_ID_CQPSQ_STAG_PDID, IRDMA_CQPSQ_STAG_PDID_M}, | ||
| 45 | {IRDMA_VCHNL_REGFLD_ID_CQPSQ_CQ_CEQID, IRDMA_CQPSQ_CQ_CEQID_M}, | ||
| 46 | {IRDMA_VCHNL_REGFLD_ID_CQPSQ_CQ_CQID, IRDMA_CQPSQ_CQ_CQID_M}, | ||
| 47 | {IRDMA_VCHNL_REGFLD_ID_COMMIT_FPM_CQCNT, IRDMA_COMMIT_FPM_CQCNT_M}, | ||
| 48 | {IRDMA_VCHNL_REGFLD_ID_UPESD_HMCN_ID, IRDMA_CQPSQ_UPESD_HMCFNID_M}, | ||
| 49 | {IRDMA_VCHNL_REGFLD_INV_ID, IRDMA_VCHNL_REGFLD_INV_ID} | ||
| 50 | }; | ||
| 51 | |||
| 52 | #define IRDMA_VCHNL_REG_COUNT ARRAY_SIZE(vchnl_reg_map) | ||
| 53 | #define IRDMA_VCHNL_REGFLD_COUNT ARRAY_SIZE(vchnl_regfld_map) | ||
| 54 | #define IRDMA_VCHNL_REGFLD_BUF_SIZE \ | ||
| 55 | (IRDMA_VCHNL_REG_COUNT * sizeof(struct irdma_vchnl_reg_info) + \ | ||
| 56 | IRDMA_VCHNL_REGFLD_COUNT * sizeof(struct irdma_vchnl_reg_field_info)) | ||
| 57 | #define IRDMA_REGMAP_RESP_BUF_SIZE (IRDMA_VCHNL_RESP_MIN_SIZE + IRDMA_VCHNL_REGFLD_BUF_SIZE) | ||
| 58 | |||
| 59 | /** | ||
| 60 | * irdma_sc_vchnl_init - Initialize dev virtchannel and get hw_rev | ||
| 61 | * @dev: dev structure to update | ||
| 62 | * @info: virtchannel info parameters to fill into the dev structure | ||
| 63 | */ | ||
| 64 | int irdma_sc_vchnl_init(struct irdma_sc_dev *dev, | ||
| 65 | struct irdma_vchnl_init_info *info) | ||
| 66 | { | ||
| 67 | dev->vchnl_up = true; | ||
| 68 | dev->privileged = info->privileged; | ||
| 69 | dev->is_pf = info->is_pf; | ||
| 70 | dev->hw_attrs.uk_attrs.hw_rev = info->hw_rev; | ||
| 71 | |||
| 72 | if (!dev->privileged) { | ||
| 73 | int ret = irdma_vchnl_req_get_ver(dev, IRDMA_VCHNL_CHNL_VER_MAX, | ||
| 74 | &dev->vchnl_ver); | ||
| 75 | |||
| 76 | ibdev_dbg(to_ibdev(dev), | ||
| 77 | "DEV: Get Channel version ret = %d, version is %u\n", | ||
| 78 | ret, dev->vchnl_ver); | ||
| 79 | |||
| 80 | if (ret) | ||
| 81 | return ret; | ||
| 82 | |||
| 83 | ret = irdma_vchnl_req_get_caps(dev); | ||
| 84 | if (ret) | ||
| 85 | return ret; | ||
| 86 | |||
| 87 | dev->hw_attrs.uk_attrs.hw_rev = dev->vc_caps.hw_rev; | ||
| 88 | } | ||
| 89 | |||
| 90 | return 0; | ||
| 91 | } | ||
| 92 | |||
| 93 | /** | ||
| 94 | * irdma_vchnl_req_verify_resp - Verify requested response size | ||
| 95 | * @vchnl_req: vchnl message requested | ||
| 96 | * @resp_len: response length sent from vchnl peer | ||
| 97 | */ | ||
| 98 | static int irdma_vchnl_req_verify_resp(struct irdma_vchnl_req *vchnl_req, | ||
| 99 | u16 resp_len) | ||
| 100 | { | ||
| 101 | switch (vchnl_req->vchnl_msg->op_code) { | ||
| 102 | case IRDMA_VCHNL_OP_GET_VER: | ||
| 103 | case IRDMA_VCHNL_OP_GET_HMC_FCN: | ||
| 104 | case IRDMA_VCHNL_OP_PUT_HMC_FCN: | ||
| 105 | if (resp_len != vchnl_req->parm_len) | ||
| 106 | return -EBADMSG; | ||
| 107 | break; | ||
| 108 | case IRDMA_VCHNL_OP_GET_RDMA_CAPS: | ||
| 109 | if (resp_len < IRDMA_VCHNL_OP_GET_RDMA_CAPS_MIN_SIZE) | ||
| 110 | return -EBADMSG; | ||
| 111 | break; | ||
| 112 | case IRDMA_VCHNL_OP_GET_REG_LAYOUT: | ||
| 113 | case IRDMA_VCHNL_OP_QUEUE_VECTOR_MAP: | ||
| 114 | case IRDMA_VCHNL_OP_QUEUE_VECTOR_UNMAP: | ||
| 115 | case IRDMA_VCHNL_OP_ADD_VPORT: | ||
| 116 | case IRDMA_VCHNL_OP_DEL_VPORT: | ||
| 117 | break; | ||
| 118 | default: | ||
| 119 | return -EOPNOTSUPP; | ||
| 120 | } | ||
| 121 | |||
| 122 | return 0; | ||
| 123 | } | ||
| 124 | |||
| 125 | static void irdma_free_vchnl_req_msg(struct irdma_vchnl_req *vchnl_req) | ||
| 126 | { | ||
| 127 | kfree(vchnl_req->vchnl_msg); | ||
| 128 | } | ||
| 129 | |||
| 130 | static int irdma_alloc_vchnl_req_msg(struct irdma_vchnl_req *vchnl_req, | ||
| 131 | struct irdma_vchnl_req_init_info *info) | ||
| 132 | { | ||
| 133 | struct irdma_vchnl_op_buf *vchnl_msg; | ||
| 134 | |||
| 135 | vchnl_msg = kzalloc(IRDMA_VCHNL_MAX_MSG_SIZE, GFP_KERNEL); | ||
| 136 | |||
| 137 | if (!vchnl_msg) | ||
| 138 | return -ENOMEM; | ||
| 139 | |||
| 140 | vchnl_msg->op_ctx = (uintptr_t)vchnl_req; | ||
| 141 | vchnl_msg->buf_len = sizeof(*vchnl_msg) + info->req_parm_len; | ||
| 142 | if (info->req_parm_len) | ||
| 143 | memcpy(vchnl_msg->buf, info->req_parm, info->req_parm_len); | ||
| 144 | vchnl_msg->op_code = info->op_code; | ||
| 145 | vchnl_msg->op_ver = info->op_ver; | ||
| 146 | |||
| 147 | vchnl_req->vchnl_msg = vchnl_msg; | ||
| 148 | vchnl_req->parm = info->resp_parm; | ||
| 149 | vchnl_req->parm_len = info->resp_parm_len; | ||
| 150 | |||
| 151 | return 0; | ||
| 152 | } | ||
| 153 | |||
| 154 | static int irdma_vchnl_req_send_sync(struct irdma_sc_dev *dev, | ||
| 155 | struct irdma_vchnl_req_init_info *info) | ||
| 156 | { | ||
| 157 | u16 resp_len = sizeof(dev->vc_recv_buf); | ||
| 158 | struct irdma_vchnl_req vchnl_req = {}; | ||
| 159 | u16 msg_len; | ||
| 160 | u8 *msg; | ||
| 161 | int ret; | ||
| 162 | |||
| 163 | ret = irdma_alloc_vchnl_req_msg(&vchnl_req, info); | ||
| 164 | if (ret) | ||
| 165 | return ret; | ||
| 166 | |||
| 167 | msg_len = vchnl_req.vchnl_msg->buf_len; | ||
| 168 | msg = (u8 *)vchnl_req.vchnl_msg; | ||
| 169 | |||
| 170 | mutex_lock(&dev->vchnl_mutex); | ||
| 171 | ret = ig3rdma_vchnl_send_sync(dev, msg, msg_len, dev->vc_recv_buf, | ||
| 172 | &resp_len); | ||
| 173 | dev->vc_recv_len = resp_len; | ||
| 174 | if (ret) | ||
| 175 | goto exit; | ||
| 176 | |||
| 177 | ret = irdma_vchnl_req_get_resp(dev, &vchnl_req); | ||
| 178 | exit: | ||
| 179 | mutex_unlock(&dev->vchnl_mutex); | ||
| 180 | ibdev_dbg(to_ibdev(dev), | ||
| 181 | "VIRT: virtual channel send %s caller: %pS ret=%d op=%u op_ver=%u req_len=%u parm_len=%u resp_len=%u\n", | ||
| 182 | !ret ? "SUCCEEDS" : "FAILS", __builtin_return_address(0), | ||
| 183 | ret, vchnl_req.vchnl_msg->op_code, | ||
| 184 | vchnl_req.vchnl_msg->op_ver, vchnl_req.vchnl_msg->buf_len, | ||
| 185 | vchnl_req.parm_len, vchnl_req.resp_len); | ||
| 186 | irdma_free_vchnl_req_msg(&vchnl_req); | ||
| 187 | |||
| 188 | return ret; | ||
| 189 | } | ||
| 190 | |||
| 191 | /** | ||
| 192 | * irdma_vchnl_req_get_reg_layout - Get Register Layout | ||
| 193 | * @dev: RDMA device pointer | ||
| 194 | */ | ||
| 195 | int irdma_vchnl_req_get_reg_layout(struct irdma_sc_dev *dev) | ||
| 196 | { | ||
| 197 | u16 reg_idx, reg_id, tmp_reg_id, regfld_idx, regfld_id, tmp_regfld_id; | ||
| 198 | struct irdma_vchnl_reg_field_info *regfld_array = NULL; | ||
| 199 | u8 resp_buffer[IRDMA_REGMAP_RESP_BUF_SIZE] = {}; | ||
| 200 | struct vchnl_regfld_map_elem *regfld_map_array; | ||
| 201 | struct irdma_vchnl_req_init_info info = {}; | ||
| 202 | struct vchnl_reg_map_elem *reg_map_array; | ||
| 203 | struct irdma_vchnl_reg_info *reg_array; | ||
| 204 | u8 num_bits, shift_cnt; | ||
| 205 | u16 buf_len = 0; | ||
| 206 | u64 bitmask; | ||
| 207 | u32 rindex; | ||
| 208 | int ret; | ||
| 209 | |||
| 210 | if (!dev->vchnl_up) | ||
| 211 | return -EBUSY; | ||
| 212 | |||
| 213 | info.op_code = IRDMA_VCHNL_OP_GET_REG_LAYOUT; | ||
| 214 | info.op_ver = IRDMA_VCHNL_OP_GET_REG_LAYOUT_V0; | ||
| 215 | info.resp_parm = resp_buffer; | ||
| 216 | info.resp_parm_len = sizeof(resp_buffer); | ||
| 217 | |||
| 218 | ret = irdma_vchnl_req_send_sync(dev, &info); | ||
| 219 | |||
| 220 | if (ret) | ||
| 221 | return ret; | ||
| 222 | |||
| 223 | /* parse the response buffer and update reg info*/ | ||
| 224 | /* Parse registers till invalid */ | ||
| 225 | /* Parse register fields till invalid */ | ||
| 226 | reg_array = (struct irdma_vchnl_reg_info *)resp_buffer; | ||
| 227 | for (rindex = 0; rindex < IRDMA_VCHNL_REG_COUNT; rindex++) { | ||
| 228 | buf_len += sizeof(struct irdma_vchnl_reg_info); | ||
| 229 | if (buf_len >= sizeof(resp_buffer)) | ||
| 230 | return -ENOMEM; | ||
| 231 | |||
| 232 | regfld_array = | ||
| 233 | (struct irdma_vchnl_reg_field_info *)®_array[rindex + 1]; | ||
| 234 | reg_id = reg_array[rindex].reg_id; | ||
| 235 | if (reg_id == IRDMA_VCHNL_REG_INV_ID) | ||
| 236 | break; | ||
| 237 | |||
| 238 | reg_id &= ~IRDMA_VCHNL_REG_PAGE_REL; | ||
| 239 | if (reg_id >= IRDMA_VCHNL_REG_COUNT) | ||
| 240 | return -EINVAL; | ||
| 241 | |||
| 242 | /* search regmap for register index in hw_regs.*/ | ||
| 243 | reg_map_array = vchnl_reg_map; | ||
| 244 | do { | ||
| 245 | tmp_reg_id = reg_map_array->reg_id; | ||
| 246 | if (tmp_reg_id == reg_id) | ||
| 247 | break; | ||
| 248 | |||
| 249 | reg_map_array++; | ||
| 250 | } while (tmp_reg_id != IRDMA_VCHNL_REG_INV_ID); | ||
| 251 | if (tmp_reg_id != reg_id) | ||
| 252 | continue; | ||
| 253 | |||
| 254 | reg_idx = reg_map_array->reg_idx; | ||
| 255 | |||
| 256 | /* Page relative, DB Offset do not need bar offset */ | ||
| 257 | if (reg_idx == IRDMA_DB_ADDR_OFFSET || | ||
| 258 | (reg_array[rindex].reg_id & IRDMA_VCHNL_REG_PAGE_REL)) { | ||
| 259 | dev->hw_regs[reg_idx] = | ||
| 260 | (u32 __iomem *)(uintptr_t)reg_array[rindex].reg_offset; | ||
| 261 | continue; | ||
| 262 | } | ||
| 263 | |||
| 264 | /* Update the local HW struct */ | ||
| 265 | dev->hw_regs[reg_idx] = ig3rdma_get_reg_addr(dev->hw, | ||
| 266 | reg_array[rindex].reg_offset); | ||
| 267 | if (!dev->hw_regs[reg_idx]) | ||
| 268 | return -EINVAL; | ||
| 269 | } | ||
| 270 | |||
| 271 | if (!regfld_array) | ||
| 272 | return -ENOMEM; | ||
| 273 | |||
| 274 | /* set up doorbell variables using mapped DB page */ | ||
| 275 | dev->wqe_alloc_db = dev->hw_regs[IRDMA_WQEALLOC]; | ||
| 276 | dev->cq_arm_db = dev->hw_regs[IRDMA_CQARM]; | ||
| 277 | dev->aeq_alloc_db = dev->hw_regs[IRDMA_AEQALLOC]; | ||
| 278 | dev->cqp_db = dev->hw_regs[IRDMA_CQPDB]; | ||
| 279 | dev->cq_ack_db = dev->hw_regs[IRDMA_CQACK]; | ||
| 280 | |||
| 281 | for (rindex = 0; rindex < IRDMA_VCHNL_REGFLD_COUNT; rindex++) { | ||
| 282 | buf_len += sizeof(struct irdma_vchnl_reg_field_info); | ||
| 283 | if ((buf_len - 1) > sizeof(resp_buffer)) | ||
| 284 | break; | ||
| 285 | |||
| 286 | if (regfld_array[rindex].fld_id == IRDMA_VCHNL_REGFLD_INV_ID) | ||
| 287 | break; | ||
| 288 | |||
| 289 | regfld_id = regfld_array[rindex].fld_id; | ||
| 290 | regfld_map_array = vchnl_regfld_map; | ||
| 291 | do { | ||
| 292 | tmp_regfld_id = regfld_map_array->regfld_id; | ||
| 293 | if (tmp_regfld_id == regfld_id) | ||
| 294 | break; | ||
| 295 | |||
| 296 | regfld_map_array++; | ||
| 297 | } while (tmp_regfld_id != IRDMA_VCHNL_REGFLD_INV_ID); | ||
| 298 | |||
| 299 | if (tmp_regfld_id != regfld_id) | ||
| 300 | continue; | ||
| 301 | |||
| 302 | regfld_idx = regfld_map_array->regfld_idx; | ||
| 303 | |||
| 304 | num_bits = regfld_array[rindex].fld_bits; | ||
| 305 | shift_cnt = regfld_array[rindex].fld_shift; | ||
| 306 | if ((num_bits + shift_cnt > 64) || !num_bits) { | ||
| 307 | ibdev_dbg(to_ibdev(dev), | ||
| 308 | "ERR: Invalid field mask id %d bits %d shift %d", | ||
| 309 | regfld_id, num_bits, shift_cnt); | ||
| 310 | |||
| 311 | continue; | ||
| 312 | } | ||
| 313 | |||
| 314 | bitmask = (1ULL << num_bits) - 1; | ||
| 315 | dev->hw_masks[regfld_idx] = bitmask << shift_cnt; | ||
| 316 | dev->hw_shifts[regfld_idx] = shift_cnt; | ||
| 317 | } | ||
| 318 | |||
| 319 | return 0; | ||
| 320 | } | ||
| 321 | |||
| 322 | int irdma_vchnl_req_add_vport(struct irdma_sc_dev *dev, u16 vport_id, | ||
| 323 | u32 qp1_id, struct irdma_qos *qos) | ||
| 324 | { | ||
| 325 | struct irdma_vchnl_resp_vport_info resp_vport = { 0 }; | ||
| 326 | struct irdma_vchnl_req_vport_info req_vport = { 0 }; | ||
| 327 | struct irdma_vchnl_req_init_info info = { 0 }; | ||
| 328 | int ret, i; | ||
| 329 | |||
| 330 | if (!dev->vchnl_up) | ||
| 331 | return -EBUSY; | ||
| 332 | |||
| 333 | info.op_code = IRDMA_VCHNL_OP_ADD_VPORT; | ||
| 334 | info.op_ver = IRDMA_VCHNL_OP_ADD_VPORT_V0; | ||
| 335 | req_vport.vport_id = vport_id; | ||
| 336 | req_vport.qp1_id = qp1_id; | ||
| 337 | info.req_parm_len = sizeof(req_vport); | ||
| 338 | info.req_parm = &req_vport; | ||
| 339 | info.resp_parm = &resp_vport; | ||
| 340 | info.resp_parm_len = sizeof(resp_vport); | ||
278 more lines aren’t shown here.
This diff is large: its 177 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 239 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 132 lines aren’t shown here.
This diff is large: its 103 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 329 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 231 lines aren’t shown here.
This diff is large: its 145 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 116 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.