History

Merge branch 'io_uring-zero-copy-rx'

David Wei says: ==================== io_uring zero copy rx This patchset contains net/ patches needed by a new io_uring request implementing zero copy rx into userspace pages, eliminating a kernel to user copy. We configure a page pool that a driver uses to fill a hw rx queue to hand out user pages instead of kernel pages. Any data that ends up hitting this hw rx queue will thus be dma'd into userspace memory directly, without needing to be bounced through kernel memory. 'Reading' data out of a socket instead becomes a _notification_ mechanism, where the kernel tells userspace where the data is. The overall approach is similar to the devmem TCP proposal. This relies on hw header/data split, flow steering and RSS to ensure packet headers remain in kernel memory and only desired flows hit a hw rx queue configured for zero copy. Configuring this is outside of the scope of this patchset. We share netdev core infra with devmem TCP. The main difference is that io_uring is used for the uAPI and the lifetime of all objects are bound to an io_uring instance. Data is 'read' using a new io_uring request type. When done, data is returned via a new shared refill queue. A zero copy page pool refills a hw rx queue from this refill queue directly. Of course, the lifetime of these data buffers are managed by io_uring rather than the networking stack, with different refcounting rules. This patchset is the first step adding basic zero copy support. We will extend this iteratively with new features e.g. dynamically allocated zero copy areas, THP support, dmabuf support, improved copy fallback, general optimisations and more. In terms of netdev support, we're first targeting Broadcom bnxt. Patches aren't included since Taehee Yoo has already sent a more comprehensive patchset adding support in [1]. Google gve should already support this, and Mellanox mlx5 support is WIP pending driver changes. =========== Performance =========== Note: Comparison with epoll + TCP_ZEROCOPY_RECEIVE isn't done yet. Test setup: * AMD EPYC 9454 * Broadcom BCM957508 200G * Kernel v6.11 base [2] * liburing fork [3] * kperf fork [4] * 4K MTU * Single TCP flow With application thread + net rx softirq pinned to _different_ cores: +-------------------------------+ | epoll | io_uring | |-----------|-------------------| | 82.2 Gbps | 116.2 Gbps (+41%) | +-------------------------------+ Pinned to _same_ core: +-------------------------------+ | epoll | io_uring | |-----------|-------------------| | 62.6 Gbps | 80.9 Gbps (+29%) | +-------------------------------+ ===== Links ===== Broadcom bnxt support: [1]: https://lore.kernel.org/20241003160620.1521626-8-ap420073@gmail.com Linux kernel branch including io_uring bits: [2]: https://github.com/isilence/linux.git zcrx/v13 liburing for testing: [3]: https://github.com/isilence/liburing.git zcrx/next kperf for testing: [4]: https://git.kernel.dk/kperf.git ====================

Jakub Kicinski committed 1 year ago

Commit 71f0dd52 parents ba6ec09 6e18ed914 files · +321 −81Browse files

Unified · Split

14 files
  1. Documentation/netlink/specs/netdev.yaml +15 −0
  2. include/net/netmem.h +20 −1
  3. include/net/page_pool/memory_provider.h +45 −0
  4. include/net/page_pool/types.h +4 −0
  5. include/uapi/linux/netdev.h +7 −0
  6. net/core/dev.c +15 −1
  7. net/core/devmem.c +59 −34
  8. net/core/devmem.h +22 −27
  9. net/core/netdev-genl.c +6 −5
  10. net/core/netdev_rx_queue.c +69 −0
  11. net/core/page_pool.c +43 −8
  12. net/core/page_pool_user.c +3 −4
  13. net/ipv4/tcp.c +6 −1
  14. tools/include/uapi/linux/netdev.h +7 −0
Documentation/netlink/specs/netdev.yaml+15−0
@@ -114,6 +114,9 @@ attribute-sets: · 113 unmodified lines
114114 doc: Bitmask of enabled AF_XDP features.
115115 type: u64
116116 enum: xsk-flags
117+ -
118+ name: io-uring-provider-info
119+ attributes: []
117120 -
118121 name: page-pool
119122 attributes:
@@ -171,6 +174,11 @@ attribute-sets: · 51 unmodified lines
171174 name: dmabuf
172175 doc: ID of the dmabuf this page-pool is attached to.
173176 type: u32
177+ -
178+ name: io-uring
179+ doc: io-uring memory provider information.
180+ type: nest
181+ nested-attributes: io-uring-provider-info
174182 -
175183 name: page-pool-info
176184 subset-of: page-pool
@@ -296,6 +304,11 @@ attribute-sets: · 119 unmodified lines
296304 name: dmabuf
297305 doc: ID of the dmabuf attached to this queue, if any.
298306 type: u32
307+ -
308+ name: io-uring
309+ doc: io_uring memory provider information.
310+ type: nest
311+ nested-attributes: io-uring-provider-info
299312
300313 -
301314 name: qstats
@@ -572,6 +585,7 @@ operations: · 270 unmodified lines
572585 - inflight-mem
573586 - detach-time
574587 - dmabuf
588+ - io-uring
575589 dump:
576590 reply: *pp-reply
577591 config-cond: page-pool
@@ -637,6 +651,7 @@ operations: · 59 unmodified lines
637651 - napi-id
638652 - ifindex
639653 - dmabuf
654+ - io-uring
640655 dump:
641656 request:
642657 attributes:
include/net/netmem.h+20−1
@@ -24,11 +24,20 @@ struct net_iov { · 23 unmodified lines
2424 unsigned long __unused_padding;
2525 unsigned long pp_magic;
2626 struct page_pool *pp;
27− struct dmabuf_genpool_chunk_owner *owner;
27+ struct net_iov_area *owner;
2828 unsigned long dma_addr;
2929 atomic_long_t pp_ref_count;
3030 };
3131
32+struct net_iov_area {
33+ /* Array of net_iovs for this area. */
34+ struct net_iov *niovs;
35+ size_t num_niovs;
36+
37+ /* Offset into the dma-buf where this chunk starts. */
38+ unsigned long base_virtual;
39+};
40+
3241 /* These fields in struct page are used by the page_pool and net stack:
3342 *
3443 * struct {
@@ -54,6 +63,16 @@ NET_IOV_ASSERT_OFFSET(dma_addr, dma_addr); · 19 unmodified lines
5463 NET_IOV_ASSERT_OFFSET(pp_ref_count, pp_ref_count);
5564 #undef NET_IOV_ASSERT_OFFSET
5665
66+static inline struct net_iov_area *net_iov_owner(const struct net_iov *niov)
67+{
68+ return niov->owner;
69+}
70+
71+static inline unsigned int net_iov_idx(const struct net_iov *niov)
72+{
73+ return niov - net_iov_owner(niov)->niovs;
74+}
75+
5776 /* netmem */
5877
5978 /**
include/net/page_pool/memory_provider.h+45−0
@@ -0,0 +1,45 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+#ifndef _NET_PAGE_POOL_MEMORY_PROVIDER_H
3+#define _NET_PAGE_POOL_MEMORY_PROVIDER_H
4+
5+#include <net/netmem.h>
6+#include <net/page_pool/types.h>
7+
8+struct netdev_rx_queue;
9+struct sk_buff;
10+
11+struct memory_provider_ops {
12+ netmem_ref (*alloc_netmems)(struct page_pool *pool, gfp_t gfp);
13+ bool (*release_netmem)(struct page_pool *pool, netmem_ref netmem);
14+ int (*init)(struct page_pool *pool);
15+ void (*destroy)(struct page_pool *pool);
16+ int (*nl_fill)(void *mp_priv, struct sk_buff *rsp,
17+ struct netdev_rx_queue *rxq);
18+ void (*uninstall)(void *mp_priv, struct netdev_rx_queue *rxq);
19+};
20+
21+bool net_mp_niov_set_dma_addr(struct net_iov *niov, dma_addr_t addr);
22+void net_mp_niov_set_page_pool(struct page_pool *pool, struct net_iov *niov);
23+void net_mp_niov_clear_page_pool(struct net_iov *niov);
24+
25+int net_mp_open_rxq(struct net_device *dev, unsigned ifq_idx,
26+ struct pp_memory_provider_params *p);
27+void net_mp_close_rxq(struct net_device *dev, unsigned ifq_idx,
28+ struct pp_memory_provider_params *old_p);
29+
30+/**
31+ * net_mp_netmem_place_in_cache() - give a netmem to a page pool
32+ * @pool: the page pool to place the netmem into
33+ * @netmem: netmem to give
34+ *
35+ * Push an accounted netmem into the page pool's allocation cache. The caller
36+ * must ensure that there is space in the cache. It should only be called off
37+ * the mp_ops->alloc_netmems() path.
38+ */
39+static inline void net_mp_netmem_place_in_cache(struct page_pool *pool,
40+ netmem_ref netmem)
41+{
42+ pool->alloc.cache[pool->alloc.count++] = netmem;
43+}
44+
45+#endif
include/net/page_pool/types.h+4−0
@@ -152,8 +152,11 @@ struct page_pool_stats { · 151 unmodified lines
152152 */
153153 #define PAGE_POOL_FRAG_GROUP_ALIGN (4 * sizeof(long))
154154
155+struct memory_provider_ops;
156+
155157 struct pp_memory_provider_params {
156158 void *mp_priv;
159+ const struct memory_provider_ops *mp_ops;
157160 };
158161
159162 struct page_pool {
@@ -216,6 +219,7 @@ struct page_pool { · 56 unmodified lines
216219 struct ptr_ring ring;
217220
218221 void *mp_priv;
222+ const struct memory_provider_ops *mp_ops;
219223
220224 #ifdef CONFIG_PAGE_POOL_STATS
221225 /* recycle stats are per-cpu to avoid locking */
include/uapi/linux/netdev.h+7−0
@@ -86,6 +86,11 @@ enum { · 85 unmodified lines
8686 NETDEV_A_DEV_MAX = (__NETDEV_A_DEV_MAX - 1)
8787 };
8888
89+enum {
90+ __NETDEV_A_IO_URING_PROVIDER_INFO_MAX,
91+ NETDEV_A_IO_URING_PROVIDER_INFO_MAX = (__NETDEV_A_IO_URING_PROVIDER_INFO_MAX - 1)
92+};
93+
8994 enum {
9095 NETDEV_A_PAGE_POOL_ID = 1,
9196 NETDEV_A_PAGE_POOL_IFINDEX,
@@ -94,6 +99,7 @@ enum { · 2 unmodified lines
9499 NETDEV_A_PAGE_POOL_INFLIGHT_MEM,
95100 NETDEV_A_PAGE_POOL_DETACH_TIME,
96101 NETDEV_A_PAGE_POOL_DMABUF,
102+ NETDEV_A_PAGE_POOL_IO_URING,
97103
98104 __NETDEV_A_PAGE_POOL_MAX,
99105 NETDEV_A_PAGE_POOL_MAX = (__NETDEV_A_PAGE_POOL_MAX - 1)
@@ -136,6 +142,7 @@ enum { · 36 unmodified lines
136142 NETDEV_A_QUEUE_TYPE,
137143 NETDEV_A_QUEUE_NAPI_ID,
138144 NETDEV_A_QUEUE_DMABUF,
145+ NETDEV_A_QUEUE_IO_URING,
139146
140147 __NETDEV_A_QUEUE_MAX,
141148 NETDEV_A_QUEUE_MAX = (__NETDEV_A_QUEUE_MAX - 1)
net/core/dev.c+15−1
@@ -159,6 +159,7 @@ · 158 unmodified lines
159159 #include <net/netdev_rx_queue.h>
160160 #include <net/page_pool/types.h>
161161 #include <net/page_pool/helpers.h>
162+#include <net/page_pool/memory_provider.h>
162163 #include <net/rps.h>
163164 #include <linux/phy_link_topology.h>
164165
@@ -11745,6 +11746,19 @@ void unregister_netdevice_queue(struct net_device *dev, struct list_head *head) · 11580 unmodified lines
1174511746 }
1174611747 EXPORT_SYMBOL(unregister_netdevice_queue);
1174711748
11749+static void dev_memory_provider_uninstall(struct net_device *dev)
11750+{
11751+ unsigned int i;
11752+
11753+ for (i = 0; i < dev->real_num_rx_queues; i++) {
11754+ struct netdev_rx_queue *rxq = &dev->_rx[i];
11755+ struct pp_memory_provider_params *p = &rxq->mp_params;
11756+
11757+ if (p->mp_ops && p->mp_ops->uninstall)
11758+ p->mp_ops->uninstall(rxq->mp_params.mp_priv, rxq);
11759+ }
11760+}
11761+
1174811762 void unregister_netdevice_many_notify(struct list_head *head,
1174911763 u32 portid, const struct nlmsghdr *nlh)
1175011764 {
@@ -11799,7 +11813,7 @@ void unregister_netdevice_many_notify(struct list_head *head, · 48 unmodified lines
1179911813 dev_tcx_uninstall(dev);
1180011814 dev_xdp_uninstall(dev);
1180111815 bpf_dev_bound_netdev_unregister(dev);
11802− dev_dmabuf_uninstall(dev);
11816+ dev_memory_provider_uninstall(dev);
1180311817
1180411818 netdev_offload_xstats_disable_all(dev);
1180511819
net/core/devmem.c+59−34
@@ -16,6 +16,7 @@ · 15 unmodified lines
1616 #include <net/netdev_queues.h>
1717 #include <net/netdev_rx_queue.h>
1818 #include <net/page_pool/helpers.h>
19+#include <net/page_pool/memory_provider.h>
1920 #include <trace/events/page_pool.h>
2021
2122 #include "devmem.h"
@@ -27,20 +28,28 @@ · 5 unmodified lines
2728 /* Protected by rtnl_lock() */
2829 static DEFINE_XARRAY_FLAGS(net_devmem_dmabuf_bindings, XA_FLAGS_ALLOC1);
2930
31+static const struct memory_provider_ops dmabuf_devmem_ops;
32+
33+bool net_is_devmem_iov(struct net_iov *niov)
34+{
35+ return niov->pp->mp_ops == &dmabuf_devmem_ops;
36+}
37+
3038 static void net_devmem_dmabuf_free_chunk_owner(struct gen_pool *genpool,
3139 struct gen_pool_chunk *chunk,
3240 void *not_used)
3341 {
3442 struct dmabuf_genpool_chunk_owner *owner = chunk->owner;
3543
36− kvfree(owner->niovs);
44+ kvfree(owner->area.niovs);
3745 kfree(owner);
3846 }
3947
4048 static dma_addr_t net_devmem_get_dma_addr(const struct net_iov *niov)
4149 {
42− struct dmabuf_genpool_chunk_owner *owner = net_iov_owner(niov);
50+ struct dmabuf_genpool_chunk_owner *owner;
4351
52+ owner = net_devmem_iov_to_chunk_owner(niov);
4453 return owner->base_dma_addr +
4554 ((dma_addr_t)net_iov_idx(niov) << PAGE_SHIFT);
4655 }
@@ -83,7 +92,7 @@ net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding) · 36 unmodified lines
8392
8493 offset = dma_addr - owner->base_dma_addr;
8594 index = offset / PAGE_SIZE;
86− niov = &owner->niovs[index];
95+ niov = &owner->area.niovs[index];
8796
8897 niov->pp_magic = 0;
8998 niov->pp = NULL;
@@ -94,7 +103,7 @@ net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding) · 4 unmodified lines
94103
95104 void net_devmem_free_dmabuf(struct net_iov *niov)
96105 {
97− struct net_devmem_dmabuf_binding *binding = net_iov_binding(niov);
106+ struct net_devmem_dmabuf_binding *binding = net_devmem_iov_binding(niov);
98107 unsigned long dma_addr = net_devmem_get_dma_addr(niov);
99108
100109 if (WARN_ON(!gen_pool_has_addr(binding->chunk_pool, dma_addr,
@@ -117,6 +126,7 @@ void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding) · 16 unmodified lines
117126 WARN_ON(rxq->mp_params.mp_priv != binding);
118127
119128 rxq->mp_params.mp_priv = NULL;
129+ rxq->mp_params.mp_ops = NULL;
120130
121131 rxq_idx = get_netdev_rx_queue_index(rxq);
122132
@@ -152,7 +162,7 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, · 29 unmodified lines
152162 }
153163
154164 rxq = __netif_get_rx_queue(dev, rxq_idx);
155− if (rxq->mp_params.mp_priv) {
165+ if (rxq->mp_params.mp_ops) {
156166 NL_SET_ERR_MSG(extack, "designated queue already memory provider bound");
157167 return -EEXIST;
158168 }
@@ -170,6 +180,7 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, · 11 unmodified lines
170180 return err;
171181
172182 rxq->mp_params.mp_priv = binding;
183+ rxq->mp_params.mp_ops = &dmabuf_devmem_ops;
173184
174185 err = netdev_rx_queue_restart(dev, rxq_idx);
175186 if (err)
@@ -179,6 +190,7 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, · 3 unmodified lines
179190
180191 err_xa_erase:
181192 rxq->mp_params.mp_priv = NULL;
193+ rxq->mp_params.mp_ops = NULL;
182194 xa_erase(&binding->bound_rxqs, xa_idx);
183195
184196 return err;
@@ -261,9 +273,9 @@ net_devmem_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, · 76 unmodified lines
261273 goto err_free_chunks;
262274 }
263275
264− owner->base_virtual = virtual;
276+ owner->area.base_virtual = virtual;
265277 owner->base_dma_addr = dma_addr;
266− owner->num_niovs = len / PAGE_SIZE;
278+ owner->area.num_niovs = len / PAGE_SIZE;
267279 owner->binding = binding;
268280
269281 err = gen_pool_add_owner(binding->chunk_pool, dma_addr,
@@ -275,17 +287,17 @@ net_devmem_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, · 5 unmodified lines
275287 goto err_free_chunks;
276288 }
277289
278− owner->niovs = kvmalloc_array(owner->num_niovs,
279− sizeof(*owner->niovs),
280− GFP_KERNEL);
281− if (!owner->niovs) {
290+ owner->area.niovs = kvmalloc_array(owner->area.num_niovs,
291+ sizeof(*owner->area.niovs),
292+ GFP_KERNEL);
293+ if (!owner->area.niovs) {
282294 err = -ENOMEM;
283295 goto err_free_chunks;
284296 }
285297
286− for (i = 0; i < owner->num_niovs; i++) {
287− niov = &owner->niovs[i];
288− niov->owner = owner;
298+ for (i = 0; i < owner->area.num_niovs; i++) {
299+ niov = &owner->area.niovs[i];
300+ niov->owner = &owner->area;
289301 page_pool_set_dma_addr_netmem(net_iov_to_netmem(niov),
290302 net_devmem_get_dma_addr(niov));
291303 }
@@ -313,26 +325,6 @@ err_put_dmabuf: · 21 unmodified lines
313325 return ERR_PTR(err);
314326 }
315327
316−void dev_dmabuf_uninstall(struct net_device *dev)
317−{
318− struct net_devmem_dmabuf_binding *binding;
319− struct netdev_rx_queue *rxq;
320− unsigned long xa_idx;
321− unsigned int i;
322−
323− for (i = 0; i < dev->real_num_rx_queues; i++) {
324− binding = dev->_rx[i].mp_params.mp_priv;
325− if (!binding)
326− continue;
327−
328− xa_for_each(&binding->bound_rxqs, xa_idx, rxq)
329− if (rxq == &dev->_rx[i]) {
330− xa_erase(&binding->bound_rxqs, xa_idx);
331− break;
332− }
333− }
334−}
335−
336328 /*** "Dmabuf devmem memory provider" ***/
337329
338330 int mp_dmabuf_devmem_init(struct page_pool *pool)
@@ -398,3 +390,36 @@ bool mp_dmabuf_devmem_release_page(struct page_pool *pool, netmem_ref netmem) · 59 unmodified lines
398390 /* We don't want the page pool put_page()ing our net_iovs. */
399391 return false;
400392 }
393+
394+static int mp_dmabuf_devmem_nl_fill(void *mp_priv, struct sk_buff *rsp,
395+ struct netdev_rx_queue *rxq)
396+{
397+ const struct net_devmem_dmabuf_binding *binding = mp_priv;
398+ int type = rxq ? NETDEV_A_QUEUE_DMABUF : NETDEV_A_PAGE_POOL_DMABUF;
399+
400+ return nla_put_u32(rsp, type, binding->id);
401+}
402+
403+static void mp_dmabuf_devmem_uninstall(void *mp_priv,
404+ struct netdev_rx_queue *rxq)
405+{
406+ struct net_devmem_dmabuf_binding *binding = mp_priv;
407+ struct netdev_rx_queue *bound_rxq;
408+ unsigned long xa_idx;
409+
410+ xa_for_each(&binding->bound_rxqs, xa_idx, bound_rxq) {
411+ if (bound_rxq == rxq) {
412+ xa_erase(&binding->bound_rxqs, xa_idx);
413+ break;
414+ }
415+ }
416+}
417+
418+static const struct memory_provider_ops dmabuf_devmem_ops = {
419+ .init = mp_dmabuf_devmem_init,
420+ .destroy = mp_dmabuf_devmem_destroy,
421+ .alloc_netmems = mp_dmabuf_devmem_alloc_netmems,
422+ .release_netmem = mp_dmabuf_devmem_release_page,
423+ .nl_fill = mp_dmabuf_devmem_nl_fill,
424+ .uninstall = mp_dmabuf_devmem_uninstall,
425+};
net/core/devmem.h+22−27
@@ -10,6 +10,8 @@ · 9 unmodified lines
1010 #ifndef _NET_DEVMEM_H
1111 #define _NET_DEVMEM_H
1212
13+#include <net/netmem.h>
14+
1315 struct netlink_ext_ack;
1416
1517 struct net_devmem_dmabuf_binding {
@@ -51,17 +53,11 @@ struct net_devmem_dmabuf_binding { · 35 unmodified lines
5153 * allocations from this chunk.
5254 */
5355 struct dmabuf_genpool_chunk_owner {
54− /* Offset into the dma-buf where this chunk starts. */
55− unsigned long base_virtual;
56+ struct net_iov_area area;
57+ struct net_devmem_dmabuf_binding *binding;
5658
5759 /* dma_addr of the start of the chunk. */
5860 dma_addr_t base_dma_addr;
59−
60− /* Array of net_iovs for this chunk. */
61− struct net_iov *niovs;
62− size_t num_niovs;
63−
64− struct net_devmem_dmabuf_binding *binding;
6561 };
6662
6763 void __net_devmem_dmabuf_binding_free(struct net_devmem_dmabuf_binding *binding);
@@ -72,38 +68,34 @@ void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding); · 4 unmodified lines
7268 int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx,
7369 struct net_devmem_dmabuf_binding *binding,
7470 struct netlink_ext_ack *extack);
75−void dev_dmabuf_uninstall(struct net_device *dev);
7671
7772 static inline struct dmabuf_genpool_chunk_owner *
78−net_iov_owner(const struct net_iov *niov)
73+net_devmem_iov_to_chunk_owner(const struct net_iov *niov)
7974 {
80− return niov->owner;
75+ struct net_iov_area *owner = net_iov_owner(niov);
76+
77+ return container_of(owner, struct dmabuf_genpool_chunk_owner, area);
8178 }
8279
83−static inline unsigned int net_iov_idx(const struct net_iov *niov)
80+static inline struct net_devmem_dmabuf_binding *
81+net_devmem_iov_binding(const struct net_iov *niov)
8482 {
85− return niov - net_iov_owner(niov)->niovs;
83+ return net_devmem_iov_to_chunk_owner(niov)->binding;
8684 }
8785
88−static inline struct net_devmem_dmabuf_binding *
89−net_iov_binding(const struct net_iov *niov)
86+static inline u32 net_devmem_iov_binding_id(const struct net_iov *niov)
9087 {
91− return net_iov_owner(niov)->binding;
88+ return net_devmem_iov_binding(niov)->id;
9289 }
9390
9491 static inline unsigned long net_iov_virtual_addr(const struct net_iov *niov)
9592 {
96− struct dmabuf_genpool_chunk_owner *owner = net_iov_owner(niov);
93+ struct net_iov_area *owner = net_iov_owner(niov);
9794
9895 return owner->base_virtual +
9996 ((unsigned long)net_iov_idx(niov) << PAGE_SHIFT);
10097 }
10198
102−static inline u32 net_iov_binding_id(const struct net_iov *niov)
103−{
104− return net_iov_owner(niov)->binding->id;
105−}
106−
10799 static inline void
108100 net_devmem_dmabuf_binding_get(struct net_devmem_dmabuf_binding *binding)
109101 {
@@ -123,6 +115,8 @@ struct net_iov * · 13 unmodified lines
123115 net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding);
124116 void net_devmem_free_dmabuf(struct net_iov *ppiov);
125117
118+bool net_is_devmem_iov(struct net_iov *niov);
119+
126120 #else
127121 struct net_devmem_dmabuf_binding;
128122
@@ -152,10 +146,6 @@ net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, · 23 unmodified lines
152146 return -EOPNOTSUPP;
153147 }
154148
155−static inline void dev_dmabuf_uninstall(struct net_device *dev)
156−{
157−}
158−
159149 static inline struct net_iov *
160150 net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding)
161151 {
@@ -171,10 +161,15 @@ static inline unsigned long net_iov_virtual_addr(const struct net_iov *niov) · 9 unmodified lines
171161 return 0;
172162 }
173163
174−static inline u32 net_iov_binding_id(const struct net_iov *niov)
164+static inline u32 net_devmem_iov_binding_id(const struct net_iov *niov)
175165 {
176166 return 0;
177167 }
168+
169+static inline bool net_is_devmem_iov(struct net_iov *niov)
170+{
171+ return false;
172+}
178173 #endif
179174
180175 #endif /* _NET_DEVMEM_H */
net/core/netdev-genl.c+6−5
@@ -10,6 +10,7 @@ · 9 unmodified lines
1010 #include <net/sock.h>
1111 #include <net/xdp.h>
1212 #include <net/xdp_sock.h>
13+#include <net/page_pool/memory_provider.h>
1314
1415 #include "dev.h"
1516 #include "devmem.h"
@@ -368,7 +369,7 @@ static int · 352 unmodified lines
368369 netdev_nl_queue_fill_one(struct sk_buff *rsp, struct net_device *netdev,
369370 u32 q_idx, u32 q_type, const struct genl_info *info)
370371 {
371− struct net_devmem_dmabuf_binding *binding;
372+ struct pp_memory_provider_params *params;
372373 struct netdev_rx_queue *rxq;
373374 struct netdev_queue *txq;
374375 void *hdr;
@@ -385,15 +386,15 @@ netdev_nl_queue_fill_one(struct sk_buff *rsp, struct net_device *netdev, · 10 unmodified lines
385386 switch (q_type) {
386387 case NETDEV_QUEUE_TYPE_RX:
387388 rxq = __netif_get_rx_queue(netdev, q_idx);
389+
388390 if (rxq->napi && nla_put_u32(rsp, NETDEV_A_QUEUE_NAPI_ID,
389391 rxq->napi->napi_id))
390392 goto nla_put_failure;
391393
392− binding = rxq->mp_params.mp_priv;
393− if (binding &&
394− nla_put_u32(rsp, NETDEV_A_QUEUE_DMABUF, binding->id))
394+ params = &rxq->mp_params;
395+ if (params->mp_ops &&
396+ params->mp_ops->nl_fill(params->mp_priv, rsp, rxq))
395397 goto nla_put_failure;
396−
397398 break;
398399 case NETDEV_QUEUE_TYPE_TX:
399400 txq = netdev_get_tx_queue(netdev, q_idx);
net/core/netdev_rx_queue.c+69−0
@@ -3,6 +3,7 @@ · 2 unmodified lines
33 #include <linux/netdevice.h>
44 #include <net/netdev_queues.h>
55 #include <net/netdev_rx_queue.h>
6+#include <net/page_pool/memory_provider.h>
67
78 #include "page_pool_priv.h"
89
@@ -80,3 +81,71 @@ err_free_new_mem: · 71 unmodified lines
8081 return err;
8182 }
8283 EXPORT_SYMBOL_NS_GPL(netdev_rx_queue_restart, "NETDEV_INTERNAL");
84+
85+static int __net_mp_open_rxq(struct net_device *dev, unsigned ifq_idx,
86+ struct pp_memory_provider_params *p)
87+{
88+ struct netdev_rx_queue *rxq;
89+ int ret;
90+
91+ if (ifq_idx >= dev->real_num_rx_queues)
92+ return -EINVAL;
93+ ifq_idx = array_index_nospec(ifq_idx, dev->real_num_rx_queues);
94+
95+ rxq = __netif_get_rx_queue(dev, ifq_idx);
96+ if (rxq->mp_params.mp_ops)
97+ return -EEXIST;
98+
99+ rxq->mp_params = *p;
100+ ret = netdev_rx_queue_restart(dev, ifq_idx);
101+ if (ret) {
102+ rxq->mp_params.mp_ops = NULL;
103+ rxq->mp_params.mp_priv = NULL;
104+ }
105+ return ret;
106+}
107+
108+int net_mp_open_rxq(struct net_device *dev, unsigned ifq_idx,
109+ struct pp_memory_provider_params *p)
110+{
111+ int ret;
112+
113+ rtnl_lock();
114+ ret = __net_mp_open_rxq(dev, ifq_idx, p);
115+ rtnl_unlock();
116+ return ret;
117+}
118+
119+static void __net_mp_close_rxq(struct net_device *dev, unsigned ifq_idx,
120+ struct pp_memory_provider_params *old_p)
121+{
122+ struct netdev_rx_queue *rxq;
123+
124+ if (WARN_ON_ONCE(ifq_idx >= dev->real_num_rx_queues))
125+ return;
126+
127+ rxq = __netif_get_rx_queue(dev, ifq_idx);
128+
129+ /* Callers holding a netdev ref may get here after we already
130+ * went thru shutdown via dev_memory_provider_uninstall().
131+ */
132+ if (dev->reg_state > NETREG_REGISTERED &&
133+ !rxq->mp_params.mp_ops)
134+ return;
135+
136+ if (WARN_ON_ONCE(rxq->mp_params.mp_ops != old_p->mp_ops ||
137+ rxq->mp_params.mp_priv != old_p->mp_priv))
138+ return;
139+
140+ rxq->mp_params.mp_ops = NULL;
141+ rxq->mp_params.mp_priv = NULL;
142+ WARN_ON(netdev_rx_queue_restart(dev, ifq_idx));
143+}
144+
145+void net_mp_close_rxq(struct net_device *dev, unsigned ifq_idx,
146+ struct pp_memory_provider_params *old_p)
147+{
148+ rtnl_lock();
149+ __net_mp_close_rxq(dev, ifq_idx, old_p);
150+ rtnl_unlock();
151+}
net/core/page_pool.c+43−8
@@ -13,6 +13,7 @@ · 12 unmodified lines
1313
1414 #include <net/netdev_rx_queue.h>
1515 #include <net/page_pool/helpers.h>
16+#include <net/page_pool/memory_provider.h>
1617 #include <net/xdp.h>
1718
1819 #include <linux/dma-direction.h>
@@ -285,13 +286,19 @@ static int page_pool_init(struct page_pool *pool, · 266 unmodified lines
285286 rxq = __netif_get_rx_queue(pool->slow.netdev,
286287 pool->slow.queue_idx);
287288 pool->mp_priv = rxq->mp_params.mp_priv;
289+ pool->mp_ops = rxq->mp_params.mp_ops;
288290 }
289291
290− if (pool->mp_priv) {
292+ if (pool->mp_ops) {
291293 if (!pool->dma_map || !pool->dma_sync)
292294 return -EOPNOTSUPP;
293295
294− err = mp_dmabuf_devmem_init(pool);
296+ if (WARN_ON(!is_kernel_rodata((unsigned long)pool->mp_ops))) {
297+ err = -EFAULT;
298+ goto free_ptr_ring;
299+ }
300+
301+ err = pool->mp_ops->init(pool);
295302 if (err) {
296303 pr_warn("%s() mem-provider init failed %d\n", __func__,
297304 err);
@@ -587,8 +594,8 @@ netmem_ref page_pool_alloc_netmems(struct page_pool *pool, gfp_t gfp) · 289 unmodified lines
587594 return netmem;
588595
589596 /* Slow-path: cache empty, do real allocation */
590− if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_priv)
591− netmem = mp_dmabuf_devmem_alloc_netmems(pool, gfp);
597+ if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_ops)
598+ netmem = pool->mp_ops->alloc_netmems(pool, gfp);
592599 else
593600 netmem = __page_pool_alloc_pages_slow(pool, gfp);
594601 return netmem;
@@ -679,8 +686,8 @@ void page_pool_return_page(struct page_pool *pool, netmem_ref netmem) · 84 unmodified lines
679686 bool put;
680687
681688 put = true;
682− if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_priv)
683− put = mp_dmabuf_devmem_release_page(pool, netmem);
689+ if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_ops)
690+ put = pool->mp_ops->release_netmem(pool, netmem);
684691 else
685692 __page_pool_release_page_dma(pool, netmem);
686693
@@ -1048,8 +1055,8 @@ static void __page_pool_destroy(struct page_pool *pool) · 361 unmodified lines
10481055 page_pool_unlist(pool);
10491056 page_pool_uninit(pool);
10501057
1051− if (pool->mp_priv) {
1052− mp_dmabuf_devmem_destroy(pool);
1058+ if (pool->mp_ops) {
1059+ pool->mp_ops->destroy(pool);
10531060 static_branch_dec(&page_pool_mem_providers);
10541061 }
10551062
@@ -1190,3 +1197,31 @@ void page_pool_update_nid(struct page_pool *pool, int new_nid) · 134 unmodified lines
11901197 }
11911198 }
11921199 EXPORT_SYMBOL(page_pool_update_nid);
1200+
1201+bool net_mp_niov_set_dma_addr(struct net_iov *niov, dma_addr_t addr)
1202+{
1203+ return page_pool_set_dma_addr_netmem(net_iov_to_netmem(niov), addr);
1204+}
1205+
1206+/* Associate a niov with a page pool. Should follow with a matching
1207+ * net_mp_niov_clear_page_pool()
1208+ */
1209+void net_mp_niov_set_page_pool(struct page_pool *pool, struct net_iov *niov)
1210+{
1211+ netmem_ref netmem = net_iov_to_netmem(niov);
1212+
1213+ page_pool_set_pp_info(pool, netmem);
1214+
1215+ pool->pages_state_hold_cnt++;
1216+ trace_page_pool_state_hold(pool, netmem, pool->pages_state_hold_cnt);
1217+}
1218+
1219+/* Disassociate a niov from a page pool. Should only be used in the
1220+ * ->release_netmem() path.
1221+ */
1222+void net_mp_niov_clear_page_pool(struct net_iov *niov)
1223+{
1224+ netmem_ref netmem = net_iov_to_netmem(niov);
1225+
1226+ page_pool_clear_pp_info(netmem);
1227+}
net/core/page_pool_user.c+3−4
@@ -8,9 +8,9 @@ · 7 unmodified lines
88 #include <net/netdev_rx_queue.h>
99 #include <net/page_pool/helpers.h>
1010 #include <net/page_pool/types.h>
11+#include <net/page_pool/memory_provider.h>
1112 #include <net/sock.h>
1213
13−#include "devmem.h"
1414 #include "page_pool_priv.h"
1515 #include "netdev-genl-gen.h"
1616
@@ -216,7 +216,6 @@ static int · 199 unmodified lines
216216 page_pool_nl_fill(struct sk_buff *rsp, const struct page_pool *pool,
217217 const struct genl_info *info)
218218 {
219− struct net_devmem_dmabuf_binding *binding = pool->mp_priv;
220219 size_t inflight, refsz;
221220 unsigned int napi_id;
222221 void *hdr;
@@ -249,7 +248,7 @@ page_pool_nl_fill(struct sk_buff *rsp, const struct page_pool *pool, · 26 unmodified lines
249248 pool->user.detach_time))
250249 goto err_cancel;
251250
252− if (binding && nla_put_u32(rsp, NETDEV_A_PAGE_POOL_DMABUF, binding->id))
251+ if (pool->mp_ops && pool->mp_ops->nl_fill(pool->mp_priv, rsp, NULL))
253252 goto err_cancel;
254253
255254 genlmsg_end(rsp, hdr);
@@ -356,7 +355,7 @@ void page_pool_unlist(struct page_pool *pool) · 100 unmodified lines
356355 int page_pool_check_memory_provider(struct net_device *dev,
357356 struct netdev_rx_queue *rxq)
358357 {
359− struct net_devmem_dmabuf_binding *binding = rxq->mp_params.mp_priv;
358+ void *binding = rxq->mp_params.mp_priv;
360359 struct page_pool *pool;
361360 struct hlist_node *n;
362361
net/ipv4/tcp.c+6−1
@@ -2476,6 +2476,11 @@ static int tcp_recvmsg_dmabuf(struct sock *sk, const struct sk_buff *skb, · 2475 unmodified lines
24762476 }
24772477
24782478 niov = skb_frag_net_iov(frag);
2479+ if (!net_is_devmem_iov(niov)) {
2480+ err = -ENODEV;
2481+ goto out;
2482+ }
2483+
24792484 end = start + skb_frag_size(frag);
24802485 copy = end - offset;
24812486
@@ -2494,7 +2499,7 @@ static int tcp_recvmsg_dmabuf(struct sock *sk, const struct sk_buff *skb, · 12 unmodified lines
24942499
24952500 /* Will perform the exchange later */
24962501 dmabuf_cmsg.frag_token = tcp_xa_pool.tokens[tcp_xa_pool.idx];
2497− dmabuf_cmsg.dmabuf_id = net_iov_binding_id(niov);
2502+ dmabuf_cmsg.dmabuf_id = net_devmem_iov_binding_id(niov);
24982503
24992504 offset += copy;
25002505 remaining_len -= copy;
tools/include/uapi/linux/netdev.h+7−0
@@ -86,6 +86,11 @@ enum { · 85 unmodified lines
8686 NETDEV_A_DEV_MAX = (__NETDEV_A_DEV_MAX - 1)
8787 };
8888
89+enum {
90+ __NETDEV_A_IO_URING_PROVIDER_INFO_MAX,
91+ NETDEV_A_IO_URING_PROVIDER_INFO_MAX = (__NETDEV_A_IO_URING_PROVIDER_INFO_MAX - 1)
92+};
93+
8994 enum {
9095 NETDEV_A_PAGE_POOL_ID = 1,
9196 NETDEV_A_PAGE_POOL_IFINDEX,
@@ -94,6 +99,7 @@ enum { · 2 unmodified lines
9499 NETDEV_A_PAGE_POOL_INFLIGHT_MEM,
95100 NETDEV_A_PAGE_POOL_DETACH_TIME,
96101 NETDEV_A_PAGE_POOL_DMABUF,
102+ NETDEV_A_PAGE_POOL_IO_URING,
97103
98104 __NETDEV_A_PAGE_POOL_MAX,
99105 NETDEV_A_PAGE_POOL_MAX = (__NETDEV_A_PAGE_POOL_MAX - 1)
@@ -136,6 +142,7 @@ enum { · 36 unmodified lines
136142 NETDEV_A_QUEUE_TYPE,
137143 NETDEV_A_QUEUE_NAPI_ID,
138144 NETDEV_A_QUEUE_DMABUF,
145+ NETDEV_A_QUEUE_IO_URING,
139146
140147 __NETDEV_A_QUEUE_MAX,
141148 NETDEV_A_QUEUE_MAX = (__NETDEV_A_QUEUE_MAX - 1)

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t