lael-labs/linux

Documentation/bpf/bpf_devel_QA.rst+12−7

 again in a second or later revision, it is also required to add a
 version number (``v2``, ``v3``, ...) into the subject prefix::
-  git format-patch --subject-prefix='PATCH net-next v2' start..finish
+  git format-patch --subject-prefix='PATCH bpf-next v2' start..finish
 When changes have been requested to the patch series, always send the
 whole patch series again with the feedback incorporated (never send
      $ llc --version
      LLVM (http://llvm.org/):
-       LLVM version 6.0.0svn
+       LLVM version 10.0.0
        Optimized build.
        Default target: x86_64-unknown-linux-gnu
        Host CPU: skylake
        Registered Targets:
-         bpf    - BPF (host endian)
+         aarch64    - AArch64 (little endian)
-         bpfeb  - BPF (big endian)
+         bpf        - BPF (host endian)
-         bpfel  - BPF (little endian)
+         bpfeb      - BPF (big endian)
-         x86    - 32-bit X86: Pentium-Pro and above
+         bpfel      - BPF (little endian)
-         x86-64 - 64-bit X86: EM64T and AMD64
+         x86        - 32-bit X86: Pentium-Pro and above
+         x86-64     - 64-bit X86: EM64T and AMD64
 For developers in order to utilize the latest features added to LLVM's
 BPF back end, it is advisable to run the latest LLVM releases. Support
 The built binaries can then be found in the build/bin/ directory, where
 you can point the PATH variable to.
+Set ``-DLLVM_TARGETS_TO_BUILD`` equal to the target you wish to build, you
+will find a full list of targets within the llvm-project/llvm/lib/Target
+directory.
 Q: Reporting LLVM BPF issues
 ----------------------------
 Q: Should I notify BPF kernel maintainers about issues in LLVM's BPF code

Documentation/bpf/btf.rst+25−0

@@ -724,6 +724,31 @@ want to define unused entry in BTF_ID_LIST, like:: · 723 unmodified lines
724	BTF_ID_UNUSED	724	BTF_ID_UNUSED
725	BTF_ID(struct, task_struct)	725	BTF_ID(struct, task_struct)
726		726
		727	The ``BTF_SET_START/END`` macros pair defines sorted list of BTF ID values
		728	and their count, with following syntax::
		729
		730	BTF_SET_START(set)
		731	BTF_ID(type1, name1)
		732	BTF_ID(type2, name2)
		733	BTF_SET_END(set)
		734
		735	resulting in following layout in .BTF_ids section::
		736
		737	__BTF_ID__set__set:
		738	.zero 4
		739	__BTF_ID__type1__name1__3:
		740	.zero 4
		741	__BTF_ID__type2__name2__4:
		742	.zero 4
		743
		744	The ``struct btf_id_set set;`` variable is defined to access the list.
		745
		746	The ``typeX`` name can be one of following::
		747
		748	struct, union, typedef, func
		749
		750	and is used as a filter when resolving the BTF ID value.
		751
727	All the BTF ID lists and sets are compiled in the .BTF_ids section and	752	All the BTF ID lists and sets are compiled in the .BTF_ids section and
728	resolved during the linking phase of kernel build by ``resolve_btfids`` tool.	753	resolved during the linking phase of kernel build by ``resolve_btfids`` tool.
729		754

Documentation/bpf/index.rst+1−0

@@ -52,6 +52,7 @@ Program types · 51 unmodified lines
52	prog_cgroup_sysctl	52	prog_cgroup_sysctl
53	prog_flow_dissector	53	prog_flow_dissector
54	bpf_lsm	54	bpf_lsm
		55	prog_sk_lookup
55		56
56		57
57	Map types	58	Map types

Documentation/bpf/prog_sk_lookup.rst+98−0

@@ -0,0 +1,98 @@
	1	.. SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
	2
	3	=====================
	4	BPF sk_lookup program
	5	=====================
	6
	7	BPF sk_lookup program type (``BPF_PROG_TYPE_SK_LOOKUP``) introduces programmability
	8	into the socket lookup performed by the transport layer when a packet is to be
	9	delivered locally.
	10
	11	When invoked BPF sk_lookup program can select a socket that will receive the
	12	incoming packet by calling the ``bpf_sk_assign()`` BPF helper function.
	13
	14	Hooks for a common attach point (``BPF_SK_LOOKUP``) exist for both TCP and UDP.
	15
	16	Motivation
	17	==========
	18
	19	BPF sk_lookup program type was introduced to address setup scenarios where
	20	binding sockets to an address with ``bind()`` socket call is impractical, such
	21	as:
	22
	23	1. receiving connections on a range of IP addresses, e.g. 192.0.2.0/24, when
	24	binding to a wildcard address ``INADRR_ANY`` is not possible due to a port
	25	conflict,
	26	2. receiving connections on all or a wide range of ports, i.e. an L7 proxy use
	27	case.
	28
	29	Such setups would require creating and ``bind()``'ing one socket to each of the
	30	IP address/port in the range, leading to resource consumption and potential
	31	latency spikes during socket lookup.
	32
	33	Attachment
	34	==========
	35
	36	BPF sk_lookup program can be attached to a network namespace with
	37	``bpf(BPF_LINK_CREATE, ...)`` syscall using the ``BPF_SK_LOOKUP`` attach type and a
	38	netns FD as attachment ``target_fd``.
	39
	40	Multiple programs can be attached to one network namespace. Programs will be
	41	invoked in the same order as they were attached.
	42
	43	Hooks
	44	=====
	45
	46	The attached BPF sk_lookup programs run whenever the transport layer needs to
	47	find a listening (TCP) or an unconnected (UDP) socket for an incoming packet.
	48
	49	Incoming traffic to established (TCP) and connected (UDP) sockets is delivered
	50	as usual without triggering the BPF sk_lookup hook.
	51
	52	The attached BPF programs must return with either ``SK_PASS`` or ``SK_DROP``
	53	verdict code. As for other BPF program types that are network filters,
	54	``SK_PASS`` signifies that the socket lookup should continue on to regular
	55	hashtable-based lookup, while ``SK_DROP`` causes the transport layer to drop the
	56	packet.
	57
	58	A BPF sk_lookup program can also select a socket to receive the packet by
	59	calling ``bpf_sk_assign()`` BPF helper. Typically, the program looks up a socket
	60	in a map holding sockets, such as ``SOCKMAP`` or ``SOCKHASH``, and passes a
	61	``struct bpf_sock *`` to ``bpf_sk_assign()`` helper to record the
	62	selection. Selecting a socket only takes effect if the program has terminated
	63	with ``SK_PASS`` code.
	64
	65	When multiple programs are attached, the end result is determined from return
	66	codes of all the programs according to the following rules:
	67
	68	1. If any program returned ``SK_PASS`` and selected a valid socket, the socket
	69	is used as the result of the socket lookup.
	70	2. If more than one program returned ``SK_PASS`` and selected a socket, the last
	71	selection takes effect.
	72	3. If any program returned ``SK_DROP``, and no program returned ``SK_PASS`` and
	73	selected a socket, socket lookup fails.
	74	4. If all programs returned ``SK_PASS`` and none of them selected a socket,
	75	socket lookup continues on.
	76
	77	API
	78	===
	79
	80	In its context, an instance of ``struct bpf_sk_lookup``, BPF sk_lookup program
	81	receives information about the packet that triggered the socket lookup. Namely:
	82
	83	* IP version (``AF_INET`` or ``AF_INET6``),
	84	* L4 protocol identifier (``IPPROTO_TCP`` or ``IPPROTO_UDP``),
	85	* source and destination IP address,
	86	* source and destination L4 port,
	87	* the socket that has been selected with ``bpf_sk_assign()``.
	88
	89	Refer to ``struct bpf_sk_lookup`` declaration in ``linux/bpf.h`` user API
	90	header, and `bpf-helpers(7)
	91	<https://man7.org/linux/man-pages/man7/bpf-helpers.7.html>`_ man-page section
	92	for ``bpf_sk_assign()`` for details.
	93
	94	Example
	95	=======
	96
	97	See ``tools/testing/selftests/bpf/prog_tests/sk_lookup.c`` for the reference
	98	implementation.

Documentation/networking/af_xdp.rst+58−10

@@ -258,14 +258,21 @@ socket into zero-copy mode or fail. · 257 unmodified lines
258	XDP_SHARED_UMEM bind flag	258	XDP_SHARED_UMEM bind flag
259	-------------------------	259	-------------------------
260		260
261	This flag enables you to bind multiple sockets to the same UMEM, but	261	This flag enables you to bind multiple sockets to the same UMEM. It
262	only if they share the same queue id. In this mode, each socket has	262	works on the same queue id, between queue ids and between
263	their own RX and TX rings, but the UMEM (tied to the fist socket	263	netdevs/devices. In this mode, each socket has their own RX and TX
264	created) only has a single FILL ring and a single COMPLETION	264	rings as usual, but you are going to have one or more FILL and
265	ring. To use this mode, create the first socket and bind it in the normal	265	COMPLETION ring pairs. You have to create one of these pairs per
266	way. Create a second socket and create an RX and a TX ring, or at	266	unique netdev and queue id tuple that you bind to.
267	least one of them, but no FILL or COMPLETION rings as the ones from	267
268	the first socket will be used. In the bind call, set he	268	Starting with the case were we would like to share a UMEM between
		269	sockets bound to the same netdev and queue id. The UMEM (tied to the
		270	fist socket created) will only have a single FILL ring and a single
		271	COMPLETION ring as there is only on unique netdev,queue_id tuple that
		272	we have bound to. To use this mode, create the first socket and bind
		273	it in the normal way. Create a second socket and create an RX and a TX
		274	ring, or at least one of them, but no FILL or COMPLETION rings as the
		275	ones from the first socket will be used. In the bind call, set he
269	XDP_SHARED_UMEM option and provide the initial socket's fd in the	276	XDP_SHARED_UMEM option and provide the initial socket's fd in the
270	sxdp_shared_umem_fd field. You can attach an arbitrary number of extra	277	sxdp_shared_umem_fd field. You can attach an arbitrary number of extra
271	sockets this way.	278	sockets this way.
@@ -305,11 +312,41 @@ concurrently. There are no synchronization primitives in the · 33 unmodified lines
305	libbpf code that protects multiple users at this point in time.	312	libbpf code that protects multiple users at this point in time.
306		313
307	Libbpf uses this mode if you create more than one socket tied to the	314	Libbpf uses this mode if you create more than one socket tied to the
308	same umem. However, note that you need to supply the	315	same UMEM. However, note that you need to supply the
309	XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD libbpf_flag with the	316	XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD libbpf_flag with the
310	xsk_socket__create calls and load your own XDP program as there is no	317	xsk_socket__create calls and load your own XDP program as there is no
311	built in one in libbpf that will route the traffic for you.	318	built in one in libbpf that will route the traffic for you.
312		319
		320	The second case is when you share a UMEM between sockets that are
		321	bound to different queue ids and/or netdevs. In this case you have to
		322	create one FILL ring and one COMPLETION ring for each unique
		323	netdev,queue_id pair. Let us say you want to create two sockets bound
		324	to two different queue ids on the same netdev. Create the first socket
		325	and bind it in the normal way. Create a second socket and create an RX
		326	and a TX ring, or at least one of them, and then one FILL and
		327	COMPLETION ring for this socket. Then in the bind call, set he
		328	XDP_SHARED_UMEM option and provide the initial socket's fd in the
		329	sxdp_shared_umem_fd field as you registered the UMEM on that
		330	socket. These two sockets will now share one and the same UMEM.
		331
		332	There is no need to supply an XDP program like the one in the previous
		333	case where sockets were bound to the same queue id and
		334	device. Instead, use the NIC's packet steering capabilities to steer
		335	the packets to the right queue. In the previous example, there is only
		336	one queue shared among sockets, so the NIC cannot do this steering. It
		337	can only steer between queues.
		338
		339	In libbpf, you need to use the xsk_socket__create_shared() API as it
		340	takes a reference to a FILL ring and a COMPLETION ring that will be
		341	created for you and bound to the shared UMEM. You can use this
		342	function for all the sockets you create, or you can use it for the
		343	second and following ones and use xsk_socket__create() for the first
		344	one. Both methods yield the same result.
		345
		346	Note that a UMEM can be shared between sockets on the same queue id
		347	and device, as well as between queues on the same device and between
		348	devices at the same time.
		349
313	XDP_USE_NEED_WAKEUP bind flag	350	XDP_USE_NEED_WAKEUP bind flag
314	-----------------------------	351	-----------------------------
315		352
@@ -364,7 +401,7 @@ resources by only setting up one of them. Both the FILL ring and the · 48 unmodified lines
364	COMPLETION ring are mandatory as you need to have a UMEM tied to your	401	COMPLETION ring are mandatory as you need to have a UMEM tied to your
365	socket. But if the XDP_SHARED_UMEM flag is used, any socket after the	402	socket. But if the XDP_SHARED_UMEM flag is used, any socket after the
366	first one does not have a UMEM and should in that case not have any	403	first one does not have a UMEM and should in that case not have any
367	FILL or COMPLETION rings created as the ones from the shared umem will	404	FILL or COMPLETION rings created as the ones from the shared UMEM will
368	be used. Note, that the rings are single-producer single-consumer, so	405	be used. Note, that the rings are single-producer single-consumer, so
369	do not try to access them from multiple processes at the same	406	do not try to access them from multiple processes at the same
370	time. See the XDP_SHARED_UMEM section.	407	time. See the XDP_SHARED_UMEM section.
@@ -567,6 +604,17 @@ A: The short answer is no, that is not supported at the moment. The · 196 unmodified lines
567	switch, or other distribution mechanism, in your NIC to direct	604	switch, or other distribution mechanism, in your NIC to direct
568	traffic to the correct queue id and socket.	605	traffic to the correct queue id and socket.
569		606
		607	Q: My packets are sometimes corrupted. What is wrong?
		608
		609	A: Care has to be taken not to feed the same buffer in the UMEM into
		610	more than one ring at the same time. If you for example feed the
		611	same buffer into the FILL ring and the TX ring at the same time, the
		612	NIC might receive data into the buffer at the same time it is
		613	sending it. This will cause some packets to become corrupted. Same
		614	thing goes for feeding the same buffer into the FILL rings
		615	belonging to different queue ids or netdevs bound with the
		616	XDP_SHARED_UMEM flag.
		617
570	Credits	618	Credits
571	=======	619	=======
572		620

arch/x86/net/bpf_jit_comp.c+21−11

 	u8 *prog = *pprog;
 	int cnt = 0;
-	if (emit_call(&prog, __bpf_prog_enter, prog))
+	if (p->aux->sleepable) {
-		return -EINVAL;
+		if (emit_call(&prog, __bpf_prog_enter_sleepable, prog))
-	/* remember prog start time returned by __bpf_prog_enter */
+			return -EINVAL;
-	emit_mov_reg(&prog, true, BPF_REG_6, BPF_REG_0);
+	} else {
+		if (emit_call(&prog, __bpf_prog_enter, prog))
+			return -EINVAL;
+		/* remember prog start time returned by __bpf_prog_enter */
+		emit_mov_reg(&prog, true, BPF_REG_6, BPF_REG_0);
+	}
 	/* arg1: lea rdi, [rbp - stack_size] */
 	EMIT4(0x48, 0x8D, 0x7D, -stack_size);
 	if (mod_ret)
 		emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
-	/* arg1: mov rdi, progs[i] */
+	if (p->aux->sleepable) {
-	emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32,
+		if (emit_call(&prog, __bpf_prog_exit_sleepable, prog))
-		       (u32) (long) p);
+			return -EINVAL;
-	/* arg2: mov rsi, rbx <- start time in nsec */
+	} else {
-	emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6);
+		/* arg1: mov rdi, progs[i] */
-	if (emit_call(&prog, __bpf_prog_exit, prog))
+		emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32,
-		return -EINVAL;
+			       (u32) (long) p);
+		/* arg2: mov rsi, rbx <- start time in nsec */
+		emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6);
+		if (emit_call(&prog, __bpf_prog_exit, prog))
+			return -EINVAL;
+	}
 	*pprog = prog;
 	return 0;

drivers/net/ethernet/intel/i40e/i40e_ethtool.c+1−1

@@ -1967,7 +1967,7 @@ static int i40e_set_ringparam(struct net_device *netdev, · 1966 unmodified lines
1967	(new_rx_count == vsi->rx_rings[0]->count))	1967	(new_rx_count == vsi->rx_rings[0]->count))
1968	return 0;	1968	return 0;
1969		1969
1970	/* If there is a AF_XDP UMEM attached to any of Rx rings,	1970	/* If there is a AF_XDP page pool attached to any of Rx rings,
1971	* disallow changing the number of descriptors -- regardless	1971	* disallow changing the number of descriptors -- regardless
1972	* if the netdev is running or not.	1972	* if the netdev is running or not.
1973	*/	1973	*/

drivers/net/ethernet/intel/i40e/i40e_main.c+15−14

 }
 /**
- * i40e_xsk_umem - Retrieve the AF_XDP ZC if XDP and ZC is enabled
+ * i40e_xsk_pool - Retrieve the AF_XDP buffer pool if XDP and ZC is enabled
  * @ring: The Tx or Rx ring
  *
- * Returns the UMEM or NULL.
+ * Returns the AF_XDP buffer pool or NULL.
  **/
-static struct xdp_umem *i40e_xsk_umem(struct i40e_ring *ring)
+static struct xsk_buff_pool *i40e_xsk_pool(struct i40e_ring *ring)
 {
 	bool xdp_on = i40e_enabled_xdp_vsi(ring->vsi);
 	int qid = ring->queue_index;
 	if (!xdp_on || !test_bit(qid, ring->vsi->af_xdp_zc_qps))
 		return NULL;
-	return xdp_get_umem_from_qid(ring->vsi->netdev, qid);
+	return xsk_get_pool_from_qid(ring->vsi->netdev, qid);
 }
 /**
 	u32 qtx_ctl = 0;
 	if (ring_is_xdp(ring))
-		ring->xsk_umem = i40e_xsk_umem(ring);
+		ring->xsk_pool = i40e_xsk_pool(ring);
 	/* some ATR related tx ring init */
 	if (vsi->back->flags & I40E_FLAG_FD_ATR_ENABLED) {
 		xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
 	kfree(ring->rx_bi);
-	ring->xsk_umem = i40e_xsk_umem(ring);
+	ring->xsk_pool = i40e_xsk_pool(ring);
-	if (ring->xsk_umem) {
+	if (ring->xsk_pool) {
 		ret = i40e_alloc_rx_bi_zc(ring);
 		if (ret)
 			return ret;
-		ring->rx_buf_len = xsk_umem_get_rx_frame_size(ring->xsk_umem);
+		ring->rx_buf_len =
+		  xsk_pool_get_rx_frame_size(ring->xsk_pool);
 		/* For AF_XDP ZC, we disallow packets to span on
 		 * multiple buffers, thus letting us skip that
 		 * handling in the fast-path.
 	ring->tail = hw->hw_addr + I40E_QRX_TAIL(pf_q);
 	writel(0, ring->tail);
-	if (ring->xsk_umem) {
+	if (ring->xsk_pool) {
-		xsk_buff_set_rxq_info(ring->xsk_umem, &ring->xdp_rxq);
+		xsk_pool_set_rxq_info(ring->xsk_pool, &ring->xdp_rxq);
 		ok = i40e_alloc_rx_buffers_zc(ring, I40E_DESC_UNUSED(ring));
 	} else {
 		ok = !i40e_alloc_rx_buffers(ring, I40E_DESC_UNUSED(ring));
 		 */
 		dev_info(&vsi->back->pdev->dev,
 			 "Failed to allocate some buffers on %sRx ring %d (pf_q %d)\n",
-			 ring->xsk_umem ? "UMEM enabled " : "",
+			 ring->xsk_pool ? "AF_XDP ZC enabled " : "",
 			 ring->queue_index, pf_q);
 	}
 	 */
 	if (need_reset && prog)
 		for (i = 0; i < vsi->num_queue_pairs; i++)
-			if (vsi->xdp_rings[i]->xsk_umem)
+			if (vsi->xdp_rings[i]->xsk_pool)
 				(void)i40e_xsk_wakeup(vsi->netdev, i,
 						      XDP_WAKEUP_RX);
 	switch (xdp->command) {
 	case XDP_SETUP_PROG:
 		return i40e_xdp_setup(vsi, xdp->prog);
-	case XDP_SETUP_XSK_UMEM:
+	case XDP_SETUP_XSK_POOL:
-		return i40e_xsk_umem_setup(vsi, xdp->xsk.umem,
+		return i40e_xsk_pool_setup(vsi, xdp->xsk.pool,
 					   xdp->xsk.queue_id);
 	default:
 		return -EINVAL;

drivers/net/ethernet/intel/i40e/i40e_txrx.c+5−5

@@ -636,7 +636,7 @@ void i40e_clean_tx_ring(struct i40e_ring *tx_ring) · 635 unmodified lines
636	unsigned long bi_size;	636	unsigned long bi_size;
637	u16 i;	637	u16 i;
638		638
639	if (ring_is_xdp(tx_ring) && tx_ring->xsk_umem) {	639	if (ring_is_xdp(tx_ring) && tx_ring->xsk_pool) {
640	i40e_xsk_clean_tx_ring(tx_ring);	640	i40e_xsk_clean_tx_ring(tx_ring);
641	} else {	641	} else {
642	/* ring already cleared, nothing to do */	642	/* ring already cleared, nothing to do */
@@ -1335,7 +1335,7 @@ void i40e_clean_rx_ring(struct i40e_ring *rx_ring) · 692 unmodified lines
1335	rx_ring->skb = NULL;	1335	rx_ring->skb = NULL;
1336	}	1336	}
1337		1337
1338	if (rx_ring->xsk_umem) {	1338	if (rx_ring->xsk_pool) {
1339	i40e_xsk_clean_rx_ring(rx_ring);	1339	i40e_xsk_clean_rx_ring(rx_ring);
1340	goto skip_free;	1340	goto skip_free;
1341	}	1341	}
@@ -1369,7 +1369,7 @@ void i40e_clean_rx_ring(struct i40e_ring *rx_ring) · 27 unmodified lines
1369	}	1369	}
1370		1370
1371	skip_free:	1371	skip_free:
1372	if (rx_ring->xsk_umem)	1372	if (rx_ring->xsk_pool)
1373	i40e_clear_rx_bi_zc(rx_ring);	1373	i40e_clear_rx_bi_zc(rx_ring);
1374	else	1374	else
1375	i40e_clear_rx_bi(rx_ring);	1375	i40e_clear_rx_bi(rx_ring);
@@ -2575,7 +2575,7 @@ int i40e_napi_poll(struct napi_struct *napi, int budget) · 1199 unmodified lines
2575	* budget and be more aggressive about cleaning up the Tx descriptors.	2575	* budget and be more aggressive about cleaning up the Tx descriptors.
2576	*/	2576	*/
2577	i40e_for_each_ring(ring, q_vector->tx) {	2577	i40e_for_each_ring(ring, q_vector->tx) {
2578	bool wd = ring->xsk_umem ?	2578	bool wd = ring->xsk_pool ?
2579	i40e_clean_xdp_tx_irq(vsi, ring) :	2579	i40e_clean_xdp_tx_irq(vsi, ring) :
2580	i40e_clean_tx_irq(vsi, ring, budget);	2580	i40e_clean_tx_irq(vsi, ring, budget);
2581		2581
@@ -2603,7 +2603,7 @@ int i40e_napi_poll(struct napi_struct *napi, int budget) · 21 unmodified lines
2603	budget_per_ring = budget;	2603	budget_per_ring = budget;
2604		2604
2605	i40e_for_each_ring(ring, q_vector->rx) {	2605	i40e_for_each_ring(ring, q_vector->rx) {
2606	int cleaned = ring->xsk_umem ?	2606	int cleaned = ring->xsk_pool ?
2607	i40e_clean_rx_irq_zc(ring, budget_per_ring) :	2607	i40e_clean_rx_irq_zc(ring, budget_per_ring) :
2608	i40e_clean_rx_irq(ring, budget_per_ring);	2608	i40e_clean_rx_irq(ring, budget_per_ring);
2609		2609

drivers/net/ethernet/intel/i40e/i40e_txrx.h+1−1

@@ -388,7 +388,7 @@ struct i40e_ring { · 387 unmodified lines
388		388
389	struct i40e_channel *ch;	389	struct i40e_channel *ch;
390	struct xdp_rxq_info xdp_rxq;	390	struct xdp_rxq_info xdp_rxq;
391	struct xdp_umem *xsk_umem;	391	struct xsk_buff_pool *xsk_pool;
392	} ____cacheline_internodealigned_in_smp;	392	} ____cacheline_internodealigned_in_smp;
393		393
394	static inline bool ring_uses_build_skb(struct i40e_ring *ring)	394	static inline bool ring_uses_build_skb(struct i40e_ring *ring)

drivers/net/ethernet/intel/i40e/i40e_xsk.c+43−38

 }
 /**
- * i40e_xsk_umem_enable - Enable/associate a UMEM to a certain ring/qid
+ * i40e_xsk_pool_enable - Enable/associate an AF_XDP buffer pool to a
+ * certain ring/qid
  * @vsi: Current VSI
- * @umem: UMEM
+ * @pool: buffer pool
- * @qid: Rx ring to associate UMEM to
+ * @qid: Rx ring to associate buffer pool with
  *
  * Returns 0 on success, <0 on failure
  **/
-static int i40e_xsk_umem_enable(struct i40e_vsi *vsi, struct xdp_umem *umem,
+static int i40e_xsk_pool_enable(struct i40e_vsi *vsi,
+				struct xsk_buff_pool *pool,
 				u16 qid)
 {
 	struct net_device *netdev = vsi->netdev;
 	    qid >= netdev->real_num_tx_queues)
 		return -EINVAL;
-	err = xsk_buff_dma_map(umem, &vsi->back->pdev->dev, I40E_RX_DMA_ATTR);
+	err = xsk_pool_dma_map(pool, &vsi->back->pdev->dev, I40E_RX_DMA_ATTR);
 	if (err)
 		return err;
 }
 /**
- * i40e_xsk_umem_disable - Disassociate a UMEM from a certain ring/qid
+ * i40e_xsk_pool_disable - Disassociate an AF_XDP buffer pool from a
+ * certain ring/qid
  * @vsi: Current VSI
- * @qid: Rx ring to associate UMEM to
+ * @qid: Rx ring to associate buffer pool with
  *
  * Returns 0 on success, <0 on failure
  **/
-static int i40e_xsk_umem_disable(struct i40e_vsi *vsi, u16 qid)
+static int i40e_xsk_pool_disable(struct i40e_vsi *vsi, u16 qid)
 {
 	struct net_device *netdev = vsi->netdev;
-	struct xdp_umem *umem;
+	struct xsk_buff_pool *pool;
 	bool if_running;
 	int err;
-	umem = xdp_get_umem_from_qid(netdev, qid);
+	pool = xsk_get_pool_from_qid(netdev, qid);
-	if (!umem)
+	if (!pool)
 		return -EINVAL;
 	if_running = netif_running(vsi->netdev) && i40e_enabled_xdp_vsi(vsi);
 	}
 	clear_bit(qid, vsi->af_xdp_zc_qps);
-	xsk_buff_dma_unmap(umem, I40E_RX_DMA_ATTR);
+	xsk_pool_dma_unmap(pool, I40E_RX_DMA_ATTR);
 	if (if_running) {
 		err = i40e_queue_pair_enable(vsi, qid);
 }
 /**
- * i40e_xsk_umem_setup - Enable/disassociate a UMEM to/from a ring/qid
+ * i40e_xsk_pool_setup - Enable/disassociate an AF_XDP buffer pool to/from
+ * a ring/qid
  * @vsi: Current VSI
- * @umem: UMEM to enable/associate to a ring, or NULL to disable
+ * @pool: Buffer pool to enable/associate to a ring, or NULL to disable
- * @qid: Rx ring to (dis)associate UMEM (from)to
+ * @qid: Rx ring to (dis)associate buffer pool (from)to
  *
- * This function enables or disables a UMEM to a certain ring.
+ * This function enables or disables a buffer pool to a certain ring.
  *
  * Returns 0 on success, <0 on failure
  **/
-int i40e_xsk_umem_setup(struct i40e_vsi *vsi, struct xdp_umem *umem,
+int i40e_xsk_pool_setup(struct i40e_vsi *vsi, struct xsk_buff_pool *pool,
 			u16 qid)
 {
-	return umem ? i40e_xsk_umem_enable(vsi, umem, qid) :
+	return pool ? i40e_xsk_pool_enable(vsi, pool, qid) :
-		i40e_xsk_umem_disable(vsi, qid);
+		i40e_xsk_pool_disable(vsi, qid);
 }
 /**
 	rx_desc = I40E_RX_DESC(rx_ring, ntu);
 	bi = i40e_rx_bi(rx_ring, ntu);
 	do {
-		xdp = xsk_buff_alloc(rx_ring->xsk_umem);
+		xdp = xsk_buff_alloc(rx_ring->xsk_pool);
 		if (!xdp) {
 			ok = false;
 			goto no_buffers;
 		bi = i40e_rx_bi(rx_ring, rx_ring->next_to_clean);
 		(*bi)->data_end = (*bi)->data + size;
-		xsk_buff_dma_sync_for_cpu(*bi);
+		xsk_buff_dma_sync_for_cpu(*bi, rx_ring->xsk_pool);
 		xdp_res = i40e_run_xdp_zc(rx_ring, *bi);
 		if (xdp_res) {
 	i40e_finalize_xdp_rx(rx_ring, xdp_xmit);
 	i40e_update_rx_stats(rx_ring, total_rx_bytes, total_rx_packets);
-	if (xsk_umem_uses_need_wakeup(rx_ring->xsk_umem)) {
+	if (xsk_uses_need_wakeup(rx_ring->xsk_pool)) {
 		if (failure || rx_ring->next_to_clean == rx_ring->next_to_use)
-			xsk_set_rx_need_wakeup(rx_ring->xsk_umem);
+			xsk_set_rx_need_wakeup(rx_ring->xsk_pool);
 		else
-			xsk_clear_rx_need_wakeup(rx_ring->xsk_umem);
+			xsk_clear_rx_need_wakeup(rx_ring->xsk_pool);
 		return (int)total_rx_packets;
 	}
 	dma_addr_t dma;
 	while (budget-- > 0) {
-		if (!xsk_umem_consume_tx(xdp_ring->xsk_umem, &desc))
+		if (!xsk_tx_peek_desc(xdp_ring->xsk_pool, &desc))
 			break;
-		dma = xsk_buff_raw_get_dma(xdp_ring->xsk_umem, desc.addr);
+		dma = xsk_buff_raw_get_dma(xdp_ring->xsk_pool, desc.addr);
-		xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_umem, dma,
+		xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_pool, dma,
 						 desc.len);
 		tx_bi = &xdp_ring->tx_bi[xdp_ring->next_to_use];
 						 I40E_TXD_QW1_CMD_SHIFT);
 		i40e_xdp_ring_update_tail(xdp_ring);
-		xsk_umem_consume_tx_done(xdp_ring->xsk_umem);
+		xsk_tx_release(xdp_ring->xsk_pool);
 		i40e_update_tx_stats(xdp_ring, sent_frames, total_bytes);
 	}
  **/
 bool i40e_clean_xdp_tx_irq(struct i40e_vsi *vsi, struct i40e_ring *tx_ring)
 {
-	struct xdp_umem *umem = tx_ring->xsk_umem;
+	struct xsk_buff_pool *bp = tx_ring->xsk_pool;
 	u32 i, completed_frames, xsk_frames = 0;
 	u32 head_idx = i40e_get_head(tx_ring);
 	struct i40e_tx_buffer *tx_bi;
 		tx_ring->next_to_clean -= tx_ring->count;
 	if (xsk_frames)
-		xsk_umem_complete_tx(umem, xsk_frames);
+		xsk_tx_completed(bp, xsk_frames);
 	i40e_arm_wb(tx_ring, vsi, completed_frames);
 out_xmit:
-	if (xsk_umem_uses_need_wakeup(tx_ring->xsk_umem))
+	if (xsk_uses_need_wakeup(tx_ring->xsk_pool))
-		xsk_set_tx_need_wakeup(tx_ring->xsk_umem);
+		xsk_set_tx_need_wakeup(tx_ring->xsk_pool);
 	return i40e_xmit_zc(tx_ring, I40E_DESC_UNUSED(tx_ring));
 }
 	if (queue_id >= vsi->num_queue_pairs)
 		return -ENXIO;
-	if (!vsi->xdp_rings[queue_id]->xsk_umem)
+	if (!vsi->xdp_rings[queue_id]->xsk_pool)
 		return -ENXIO;
 	ring = vsi->xdp_rings[queue_id];
 void i40e_xsk_clean_tx_ring(struct i40e_ring *tx_ring)
 {
 	u16 ntc = tx_ring->next_to_clean, ntu = tx_ring->next_to_use;
-	struct xdp_umem *umem = tx_ring->xsk_umem;
+	struct xsk_buff_pool *bp = tx_ring->xsk_pool;
 	struct i40e_tx_buffer *tx_bi;
 	u32 xsk_frames = 0;
 	}
 	if (xsk_frames)
-		xsk_umem_complete_tx(umem, xsk_frames);
+		xsk_tx_completed(bp, xsk_frames);
 }
 /**
- * i40e_xsk_any_rx_ring_enabled - Checks if Rx rings have AF_XDP UMEM attached
+ * i40e_xsk_any_rx_ring_enabled - Checks if Rx rings have an AF_XDP
+ * buffer pool attached
  * @vsi: vsi
  *
- * Returns true if any of the Rx rings has an AF_XDP UMEM attached
+ * Returns true if any of the Rx rings has an AF_XDP buffer pool attached
  **/
 bool i40e_xsk_any_rx_ring_enabled(struct i40e_vsi *vsi)
 {
 	int i;
 	for (i = 0; i < vsi->num_queue_pairs; i++) {
-		if (xdp_get_umem_from_qid(netdev, i))
+		if (xsk_get_pool_from_qid(netdev, i))
 			return true;
 	}

drivers/net/ethernet/intel/i40e/i40e_xsk.h+2−2

@@ -5,12 +5,12 @@ · 4 unmodified lines
5	#define _I40E_XSK_H_	5	#define _I40E_XSK_H_
6		6
7	struct i40e_vsi;	7	struct i40e_vsi;
8	struct xdp_umem;	8	struct xsk_buff_pool;
9	struct zero_copy_allocator;	9	struct zero_copy_allocator;
10		10
11	int i40e_queue_pair_disable(struct i40e_vsi *vsi, int queue_pair);	11	int i40e_queue_pair_disable(struct i40e_vsi *vsi, int queue_pair);
12	int i40e_queue_pair_enable(struct i40e_vsi *vsi, int queue_pair);	12	int i40e_queue_pair_enable(struct i40e_vsi *vsi, int queue_pair);
13	int i40e_xsk_umem_setup(struct i40e_vsi vsi, struct xdp_umem umem,	13	int i40e_xsk_pool_setup(struct i40e_vsi vsi, struct xsk_buff_pool pool,
14	u16 qid);	14	u16 qid);
15	bool i40e_alloc_rx_buffers_zc(struct i40e_ring *rx_ring, u16 cleaned_count);	15	bool i40e_alloc_rx_buffers_zc(struct i40e_ring *rx_ring, u16 cleaned_count);
16	int i40e_clean_rx_irq_zc(struct i40e_ring *rx_ring, int budget);	16	int i40e_clean_rx_irq_zc(struct i40e_ring *rx_ring, int budget);

drivers/net/ethernet/intel/ice/ice.h+9−9

@@ -321,9 +321,9 @@ struct ice_vsi { · 320 unmodified lines
321	struct ice_ring *xdp_rings; / XDP ring array */	321	struct ice_ring *xdp_rings; / XDP ring array */
322	u16 num_xdp_txq; /* Used XDP queues */	322	u16 num_xdp_txq; /* Used XDP queues */
323	u8 xdp_mapping_mode; /* ICE_MAP_MODE_[CONTIG\|SCATTER] */	323	u8 xdp_mapping_mode; /* ICE_MAP_MODE_[CONTIG\|SCATTER] */
324	struct xdp_umem **xsk_umems;	324	struct xsk_buff_pool **xsk_pools;
325	u16 num_xsk_umems_used;	325	u16 num_xsk_pools_used;
326	u16 num_xsk_umems;	326	u16 num_xsk_pools;
327	} ____cacheline_internodealigned_in_smp;	327	} ____cacheline_internodealigned_in_smp;
328		328
329	/* struct that defines an interrupt vector */	329	/* struct that defines an interrupt vector */
@@ -507,25 +507,25 @@ static inline void ice_set_ring_xdp(struct ice_ring *ring) · 177 unmodified lines
507	}	507	}
508		508
509	/**	509	/**
510	* ice_xsk_umem - get XDP UMEM bound to a ring	510	* ice_xsk_pool - get XSK buffer pool bound to a ring
511	* @ring - ring to use	511	* @ring - ring to use
512	*	512	*
513	* Returns a pointer to xdp_umem structure if there is an UMEM present,	513	* Returns a pointer to xdp_umem structure if there is a buffer pool present,
514	* NULL otherwise.	514	* NULL otherwise.
515	*/	515	*/
516	static inline struct xdp_umem ice_xsk_umem(struct ice_ring ring)	516	static inline struct xsk_buff_pool ice_xsk_pool(struct ice_ring ring)
517	{	517	{
518	struct xdp_umem **umems = ring->vsi->xsk_umems;	518	struct xsk_buff_pool **pools = ring->vsi->xsk_pools;
519	u16 qid = ring->q_index;	519	u16 qid = ring->q_index;
520		520
521	if (ice_ring_is_xdp(ring))	521	if (ice_ring_is_xdp(ring))
522	qid -= ring->vsi->num_xdp_txq;	522	qid -= ring->vsi->num_xdp_txq;
523		523
524	if (qid >= ring->vsi->num_xsk_umems \|\| !umems \|\| !umems[qid] \|\|	524	if (qid >= ring->vsi->num_xsk_pools \|\| !pools \|\| !pools[qid] \|\|
525	!ice_is_xdp_ena_vsi(ring->vsi))	525	!ice_is_xdp_ena_vsi(ring->vsi))
526	return NULL;	526	return NULL;
527		527
528	return umems[qid];	528	return pools[qid];
529	}	529	}
530		530
531	/**	531	/**

drivers/net/ethernet/intel/ice/ice_base.c+8−8

@@ -308,12 +308,12 @@ int ice_setup_rx_ctx(struct ice_ring *ring) · 307 unmodified lines
308	xdp_rxq_info_reg(&ring->xdp_rxq, ring->netdev,	308	xdp_rxq_info_reg(&ring->xdp_rxq, ring->netdev,
309	ring->q_index);	309	ring->q_index);
310		310
311	ring->xsk_umem = ice_xsk_umem(ring);	311	ring->xsk_pool = ice_xsk_pool(ring);
312	if (ring->xsk_umem) {	312	if (ring->xsk_pool) {
313	xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);	313	xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
314		314
315	ring->rx_buf_len =	315	ring->rx_buf_len =
316	xsk_umem_get_rx_frame_size(ring->xsk_umem);	316	xsk_pool_get_rx_frame_size(ring->xsk_pool);
317	/* For AF_XDP ZC, we disallow packets to span on	317	/* For AF_XDP ZC, we disallow packets to span on
318	* multiple buffers, thus letting us skip that	318	* multiple buffers, thus letting us skip that
319	* handling in the fast-path.	319	* handling in the fast-path.
@@ -324,7 +324,7 @@ int ice_setup_rx_ctx(struct ice_ring *ring) · 4 unmodified lines
324	NULL);	324	NULL);
325	if (err)	325	if (err)
326	return err;	326	return err;
327	xsk_buff_set_rxq_info(ring->xsk_umem, &ring->xdp_rxq);	327	xsk_pool_set_rxq_info(ring->xsk_pool, &ring->xdp_rxq);
328		328
329	dev_info(dev, "Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring %d\n",	329	dev_info(dev, "Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring %d\n",
330	ring->q_index);	330	ring->q_index);
@@ -417,9 +417,9 @@ int ice_setup_rx_ctx(struct ice_ring *ring) · 86 unmodified lines
417	ring->tail = hw->hw_addr + QRX_TAIL(pf_q);	417	ring->tail = hw->hw_addr + QRX_TAIL(pf_q);
418	writel(0, ring->tail);	418	writel(0, ring->tail);
419		419
420	if (ring->xsk_umem) {	420	if (ring->xsk_pool) {
421	if (!xsk_buff_can_alloc(ring->xsk_umem, num_bufs)) {	421	if (!xsk_buff_can_alloc(ring->xsk_pool, num_bufs)) {
422	dev_warn(dev, "UMEM does not provide enough addresses to fill %d buffers on Rx ring %d\n",	422	dev_warn(dev, "XSK buffer pool does not provide enough addresses to fill %d buffers on Rx ring %d\n",
423	num_bufs, ring->q_index);	423	num_bufs, ring->q_index);
424	dev_warn(dev, "Change Rx ring/fill queue size to avoid performance issues\n");	424	dev_warn(dev, "Change Rx ring/fill queue size to avoid performance issues\n");
425		425
@@ -428,7 +428,7 @@ int ice_setup_rx_ctx(struct ice_ring *ring) · 2 unmodified lines
428		428
429	err = ice_alloc_rx_bufs_zc(ring, num_bufs);	429	err = ice_alloc_rx_bufs_zc(ring, num_bufs);
430	if (err)	430	if (err)
431	dev_info(dev, "Failed to allocate some buffers on UMEM enabled Rx ring %d (pf_q %d)\n",	431	dev_info(dev, "Failed to allocate some buffers on XSK buffer pool enabled Rx ring %d (pf_q %d)\n",
432	ring->q_index, pf_q);	432	ring->q_index, pf_q);
433	return 0;	433	return 0;
434	}	434	}

drivers/net/ethernet/intel/ice/ice_lib.c+1−1

@@ -1743,7 +1743,7 @@ int ice_vsi_cfg_xdp_txqs(struct ice_vsi *vsi) · 1742 unmodified lines
1743	return ret;	1743	return ret;
1744		1744
1745	for (i = 0; i < vsi->num_xdp_txq; i++)	1745	for (i = 0; i < vsi->num_xdp_txq; i++)
1746	vsi->xdp_rings[i]->xsk_umem = ice_xsk_umem(vsi->xdp_rings[i]);	1746	vsi->xdp_rings[i]->xsk_pool = ice_xsk_pool(vsi->xdp_rings[i]);
1747		1747
1748	return ret;	1748	return ret;
1749	}	1749	}

drivers/net/ethernet/intel/ice/ice_main.c+5−5

@@ -2273,7 +2273,7 @@ static int ice_xdp_alloc_setup_rings(struct ice_vsi *vsi) · 2272 unmodified lines
2273	if (ice_setup_tx_ring(xdp_ring))	2273	if (ice_setup_tx_ring(xdp_ring))
2274	goto free_xdp_rings;	2274	goto free_xdp_rings;
2275	ice_set_ring_xdp(xdp_ring);	2275	ice_set_ring_xdp(xdp_ring);
2276	xdp_ring->xsk_umem = ice_xsk_umem(xdp_ring);	2276	xdp_ring->xsk_pool = ice_xsk_pool(xdp_ring);
2277	}	2277	}
2278		2278
2279	return 0;	2279	return 0;
@@ -2517,13 +2517,13 @@ ice_xdp_setup_prog(struct ice_vsi vsi, struct bpf_prog prog, · 237 unmodified lines
2517	if (if_running)	2517	if (if_running)
2518	ret = ice_up(vsi);	2518	ret = ice_up(vsi);
2519		2519
2520	if (!ret && prog && vsi->xsk_umems) {	2520	if (!ret && prog && vsi->xsk_pools) {
2521	int i;	2521	int i;
2522		2522
2523	ice_for_each_rxq(vsi, i) {	2523	ice_for_each_rxq(vsi, i) {
2524	struct ice_ring *rx_ring = vsi->rx_rings[i];	2524	struct ice_ring *rx_ring = vsi->rx_rings[i];
2525		2525
2526	if (rx_ring->xsk_umem)	2526	if (rx_ring->xsk_pool)
2527	napi_schedule(&rx_ring->q_vector->napi);	2527	napi_schedule(&rx_ring->q_vector->napi);
2528	}	2528	}
2529	}	2529	}
@@ -2549,8 +2549,8 @@ static int ice_xdp(struct net_device dev, struct netdev_bpf xdp) · 19 unmodified lines
2549	switch (xdp->command) {	2549	switch (xdp->command) {
2550	case XDP_SETUP_PROG:	2550	case XDP_SETUP_PROG:
2551	return ice_xdp_setup_prog(vsi, xdp->prog, xdp->extack);	2551	return ice_xdp_setup_prog(vsi, xdp->prog, xdp->extack);
2552	case XDP_SETUP_XSK_UMEM:	2552	case XDP_SETUP_XSK_POOL:
2553	return ice_xsk_umem_setup(vsi, xdp->xsk.umem,	2553	return ice_xsk_pool_setup(vsi, xdp->xsk.pool,
2554	xdp->xsk.queue_id);	2554	xdp->xsk.queue_id);
2555	default:	2555	default:
2556	return -EINVAL;	2556	return -EINVAL;

drivers/net/ethernet/intel/ice/ice_txrx.c+4−4

@@ -145,7 +145,7 @@ void ice_clean_tx_ring(struct ice_ring *tx_ring) · 144 unmodified lines
145	{	145	{
146	u16 i;	146	u16 i;
147		147
148	if (ice_ring_is_xdp(tx_ring) && tx_ring->xsk_umem) {	148	if (ice_ring_is_xdp(tx_ring) && tx_ring->xsk_pool) {
149	ice_xsk_clean_xdp_ring(tx_ring);	149	ice_xsk_clean_xdp_ring(tx_ring);
150	goto tx_skip_free;	150	goto tx_skip_free;
151	}	151	}
@@ -375,7 +375,7 @@ void ice_clean_rx_ring(struct ice_ring *rx_ring) · 223 unmodified lines
375	if (!rx_ring->rx_buf)	375	if (!rx_ring->rx_buf)
376	return;	376	return;
377		377
378	if (rx_ring->xsk_umem) {	378	if (rx_ring->xsk_pool) {
379	ice_xsk_clean_rx_ring(rx_ring);	379	ice_xsk_clean_rx_ring(rx_ring);
380	goto rx_skip_free;	380	goto rx_skip_free;
381	}	381	}
@@ -1610,7 +1610,7 @@ int ice_napi_poll(struct napi_struct *napi, int budget) · 1228 unmodified lines
1610	* budget and be more aggressive about cleaning up the Tx descriptors.	1610	* budget and be more aggressive about cleaning up the Tx descriptors.
1611	*/	1611	*/
1612	ice_for_each_ring(ring, q_vector->tx) {	1612	ice_for_each_ring(ring, q_vector->tx) {
1613	bool wd = ring->xsk_umem ?	1613	bool wd = ring->xsk_pool ?
1614	ice_clean_tx_irq_zc(ring, budget) :	1614	ice_clean_tx_irq_zc(ring, budget) :
1615	ice_clean_tx_irq(ring, budget);	1615	ice_clean_tx_irq(ring, budget);
1616		1616
@@ -1640,7 +1640,7 @@ int ice_napi_poll(struct napi_struct *napi, int budget) · 23 unmodified lines
1640	* comparison in the irq context instead of many inside the	1640	* comparison in the irq context instead of many inside the
1641	* ice_clean_rx_irq function and makes the codebase cleaner.	1641	* ice_clean_rx_irq function and makes the codebase cleaner.
1642	*/	1642	*/
1643	cleaned = ring->xsk_umem ?	1643	cleaned = ring->xsk_pool ?
1644	ice_clean_rx_irq_zc(ring, budget_per_ring) :	1644	ice_clean_rx_irq_zc(ring, budget_per_ring) :
1645	ice_clean_rx_irq(ring, budget_per_ring);	1645	ice_clean_rx_irq(ring, budget_per_ring);
1646	work_done += cleaned;	1646	work_done += cleaned;

drivers/net/ethernet/intel/ice/ice_txrx.h+1−1

@@ -295,7 +295,7 @@ struct ice_ring { · 294 unmodified lines
295		295
296	struct rcu_head rcu; /* to avoid race on free */	296	struct rcu_head rcu; /* to avoid race on free */
297	struct bpf_prog *xdp_prog;	297	struct bpf_prog *xdp_prog;
298	struct xdp_umem *xsk_umem;	298	struct xsk_buff_pool *xsk_pool;
299	/* CL3 - 3rd cacheline starts here */	299	/* CL3 - 3rd cacheline starts here */
300	struct xdp_rxq_info xdp_rxq;	300	struct xdp_rxq_info xdp_rxq;
301	/* CLX - the below items are only accessed infrequently and should be	301	/* CLX - the below items are only accessed infrequently and should be

drivers/net/ethernet/intel/ice/ice_xsk.c+69−69

@@ -236,7 +236,7 @@ static int ice_qp_ena(struct ice_vsi *vsi, u16 q_idx) · 235 unmodified lines
236	if (err)	236	if (err)
237	goto free_buf;	237	goto free_buf;
238	ice_set_ring_xdp(xdp_ring);	238	ice_set_ring_xdp(xdp_ring);
239	xdp_ring->xsk_umem = ice_xsk_umem(xdp_ring);	239	xdp_ring->xsk_pool = ice_xsk_pool(xdp_ring);
240	}	240	}
241		241
242	err = ice_setup_rx_ctx(rx_ring);	242	err = ice_setup_rx_ctx(rx_ring);
@@ -260,21 +260,21 @@ free_buf: · 17 unmodified lines
260	}	260	}
261		261
262	/**	262	/**
263	* ice_xsk_alloc_umems - allocate a UMEM region for an XDP socket	263	* ice_xsk_alloc_pools - allocate a buffer pool for an XDP socket
264	* @vsi: VSI to allocate the UMEM on	264	* @vsi: VSI to allocate the buffer pool on
265	*	265	*
266	* Returns 0 on success, negative on error	266	* Returns 0 on success, negative on error
267	*/	267	*/
268	static int ice_xsk_alloc_umems(struct ice_vsi *vsi)	268	static int ice_xsk_alloc_pools(struct ice_vsi *vsi)
269	{	269	{
270	if (vsi->xsk_umems)	270	if (vsi->xsk_pools)
271	return 0;	271	return 0;
272		272
273	vsi->xsk_umems = kcalloc(vsi->num_xsk_umems, sizeof(*vsi->xsk_umems),	273	vsi->xsk_pools = kcalloc(vsi->num_xsk_pools, sizeof(*vsi->xsk_pools),
274	GFP_KERNEL);	274	GFP_KERNEL);
275		275
276	if (!vsi->xsk_umems) {	276	if (!vsi->xsk_pools) {
277	vsi->num_xsk_umems = 0;	277	vsi->num_xsk_pools = 0;
278	return -ENOMEM;	278	return -ENOMEM;
279	}	279	}
280		280
@@ -282,73 +282,73 @@ static int ice_xsk_alloc_umems(struct ice_vsi *vsi) · 1 unmodified line
282	}	282	}
283		283
284	/**	284	/**
285	* ice_xsk_remove_umem - Remove an UMEM for a certain ring/qid	285	* ice_xsk_remove_pool - Remove an buffer pool for a certain ring/qid
286	* @vsi: VSI from which the VSI will be removed	286	* @vsi: VSI from which the VSI will be removed
287	* @qid: Ring/qid associated with the UMEM	287	* @qid: Ring/qid associated with the buffer pool
288	*/	288	*/
289	static void ice_xsk_remove_umem(struct ice_vsi *vsi, u16 qid)	289	static void ice_xsk_remove_pool(struct ice_vsi *vsi, u16 qid)
290	{	290	{
291	vsi->xsk_umems[qid] = NULL;	291	vsi->xsk_pools[qid] = NULL;
292	vsi->num_xsk_umems_used--;	292	vsi->num_xsk_pools_used--;
293		293
294	if (vsi->num_xsk_umems_used == 0) {	294	if (vsi->num_xsk_pools_used == 0) {
295	kfree(vsi->xsk_umems);	295	kfree(vsi->xsk_pools);
296	vsi->xsk_umems = NULL;	296	vsi->xsk_pools = NULL;
297	vsi->num_xsk_umems = 0;	297	vsi->num_xsk_pools = 0;
298	}	298	}
299	}	299	}
300		300
301	/**	301	/**
302	* ice_xsk_umem_disable - disable a UMEM region	302	* ice_xsk_pool_disable - disable a buffer pool region
303	* @vsi: Current VSI	303	* @vsi: Current VSI
304	* @qid: queue ID	304	* @qid: queue ID
305	*	305	*
306	* Returns 0 on success, negative on failure	306	* Returns 0 on success, negative on failure
307	*/	307	*/
308	static int ice_xsk_umem_disable(struct ice_vsi *vsi, u16 qid)	308	static int ice_xsk_pool_disable(struct ice_vsi *vsi, u16 qid)
309	{	309	{
310	if (!vsi->xsk_umems \|\| qid >= vsi->num_xsk_umems \|\|	310	if (!vsi->xsk_pools \|\| qid >= vsi->num_xsk_pools \|\|
311	!vsi->xsk_umems[qid])	311	!vsi->xsk_pools[qid])
312	return -EINVAL;	312	return -EINVAL;
313		313
314	xsk_buff_dma_unmap(vsi->xsk_umems[qid], ICE_RX_DMA_ATTR);	314	xsk_pool_dma_unmap(vsi->xsk_pools[qid], ICE_RX_DMA_ATTR);
315	ice_xsk_remove_umem(vsi, qid);	315	ice_xsk_remove_pool(vsi, qid);
316		316
317	return 0;	317	return 0;
318	}	318	}
319		319
320	/**	320	/**
321	* ice_xsk_umem_enable - enable a UMEM region	321	* ice_xsk_pool_enable - enable a buffer pool region
322	* @vsi: Current VSI	322	* @vsi: Current VSI
323	* @umem: pointer to a requested UMEM region	323	* @pool: pointer to a requested buffer pool region
324	* @qid: queue ID	324	* @qid: queue ID
325	*	325	*
326	* Returns 0 on success, negative on failure	326	* Returns 0 on success, negative on failure
327	*/	327	*/
328	static int	328	static int
329	ice_xsk_umem_enable(struct ice_vsi vsi, struct xdp_umem umem, u16 qid)	329	ice_xsk_pool_enable(struct ice_vsi vsi, struct xsk_buff_pool pool, u16 qid)
330	{	330	{
331	int err;	331	int err;
332		332
333	if (vsi->type != ICE_VSI_PF)	333	if (vsi->type != ICE_VSI_PF)
334	return -EINVAL;	334	return -EINVAL;
335		335
336	if (!vsi->num_xsk_umems)	336	if (!vsi->num_xsk_pools)
337	vsi->num_xsk_umems = min_t(u16, vsi->num_rxq, vsi->num_txq);	337	vsi->num_xsk_pools = min_t(u16, vsi->num_rxq, vsi->num_txq);
338	if (qid >= vsi->num_xsk_umems)	338	if (qid >= vsi->num_xsk_pools)
339	return -EINVAL;	339	return -EINVAL;
340		340
341	err = ice_xsk_alloc_umems(vsi);	341	err = ice_xsk_alloc_pools(vsi);
342	if (err)	342	if (err)
343	return err;	343	return err;
344		344
345	if (vsi->xsk_umems && vsi->xsk_umems[qid])	345	if (vsi->xsk_pools && vsi->xsk_pools[qid])
346	return -EBUSY;	346	return -EBUSY;
347		347
348	vsi->xsk_umems[qid] = umem;	348	vsi->xsk_pools[qid] = pool;
349	vsi->num_xsk_umems_used++;	349	vsi->num_xsk_pools_used++;
350		350
351	err = xsk_buff_dma_map(vsi->xsk_umems[qid], ice_pf_to_dev(vsi->back),	351	err = xsk_pool_dma_map(vsi->xsk_pools[qid], ice_pf_to_dev(vsi->back),
352	ICE_RX_DMA_ATTR);	352	ICE_RX_DMA_ATTR);
353	if (err)	353	if (err)
354	return err;	354	return err;
@@ -357,17 +357,17 @@ ice_xsk_umem_enable(struct ice_vsi vsi, struct xdp_umem umem, u16 qid) · 2 unmodified lines
357	}	357	}
358		358
359	/**	359	/**
360	* ice_xsk_umem_setup - enable/disable a UMEM region depending on its state	360	* ice_xsk_pool_setup - enable/disable a buffer pool region depending on its state
361	* @vsi: Current VSI	361	* @vsi: Current VSI
362	* @umem: UMEM to enable/associate to a ring, NULL to disable	362	* @pool: buffer pool to enable/associate to a ring, NULL to disable
363	* @qid: queue ID	363	* @qid: queue ID
364	*	364	*
365	* Returns 0 on success, negative on failure	365	* Returns 0 on success, negative on failure
366	*/	366	*/
367	int ice_xsk_umem_setup(struct ice_vsi vsi, struct xdp_umem umem, u16 qid)	367	int ice_xsk_pool_setup(struct ice_vsi vsi, struct xsk_buff_pool pool, u16 qid)
368	{	368	{
369	bool if_running, umem_present = !!umem;	369	bool if_running, pool_present = !!pool;
370	int ret = 0, umem_failure = 0;	370	int ret = 0, pool_failure = 0;
371		371
372	if_running = netif_running(vsi->netdev) && ice_is_xdp_ena_vsi(vsi);	372	if_running = netif_running(vsi->netdev) && ice_is_xdp_ena_vsi(vsi);
373		373
@@ -375,26 +375,26 @@ int ice_xsk_umem_setup(struct ice_vsi vsi, struct xdp_umem umem, u16 qid) · 1 unmodified line
375	ret = ice_qp_dis(vsi, qid);	375	ret = ice_qp_dis(vsi, qid);
376	if (ret) {	376	if (ret) {
377	netdev_err(vsi->netdev, "ice_qp_dis error = %d\n", ret);	377	netdev_err(vsi->netdev, "ice_qp_dis error = %d\n", ret);
378	goto xsk_umem_if_up;	378	goto xsk_pool_if_up;
379	}	379	}
380	}	380	}
381		381
382	umem_failure = umem_present ? ice_xsk_umem_enable(vsi, umem, qid) :	382	pool_failure = pool_present ? ice_xsk_pool_enable(vsi, pool, qid) :
383	ice_xsk_umem_disable(vsi, qid);	383	ice_xsk_pool_disable(vsi, qid);
384		384
385	xsk_umem_if_up:	385	xsk_pool_if_up:
386	if (if_running) {	386	if (if_running) {
387	ret = ice_qp_ena(vsi, qid);	387	ret = ice_qp_ena(vsi, qid);
388	if (!ret && umem_present)	388	if (!ret && pool_present)
389	napi_schedule(&vsi->xdp_rings[qid]->q_vector->napi);	389	napi_schedule(&vsi->xdp_rings[qid]->q_vector->napi);
390	else if (ret)	390	else if (ret)
391	netdev_err(vsi->netdev, "ice_qp_ena error = %d\n", ret);	391	netdev_err(vsi->netdev, "ice_qp_ena error = %d\n", ret);
392	}	392	}
393		393
394	if (umem_failure) {	394	if (pool_failure) {
395	netdev_err(vsi->netdev, "Could not %sable UMEM, error = %d\n",	395	netdev_err(vsi->netdev, "Could not %sable buffer pool, error = %d\n",
396	umem_present ? "en" : "dis", umem_failure);	396	pool_present ? "en" : "dis", pool_failure);
397	return umem_failure;	397	return pool_failure;
398	}	398	}
399		399
400	return ret;	400	return ret;
@@ -425,7 +425,7 @@ bool ice_alloc_rx_bufs_zc(struct ice_ring *rx_ring, u16 count) · 24 unmodified lines
425	rx_buf = &rx_ring->rx_buf[ntu];	425	rx_buf = &rx_ring->rx_buf[ntu];
426		426
427	do {	427	do {
428	rx_buf->xdp = xsk_buff_alloc(rx_ring->xsk_umem);	428	rx_buf->xdp = xsk_buff_alloc(rx_ring->xsk_pool);
429	if (!rx_buf->xdp) {	429	if (!rx_buf->xdp) {
430	ret = true;	430	ret = true;
431	break;	431	break;
@@ -595,7 +595,7 @@ int ice_clean_rx_irq_zc(struct ice_ring *rx_ring, int budget) · 163 unmodified lines
595		595
596	rx_buf = &rx_ring->rx_buf[rx_ring->next_to_clean];	596	rx_buf = &rx_ring->rx_buf[rx_ring->next_to_clean];
597	rx_buf->xdp->data_end = rx_buf->xdp->data + size;	597	rx_buf->xdp->data_end = rx_buf->xdp->data + size;
598	xsk_buff_dma_sync_for_cpu(rx_buf->xdp);	598	xsk_buff_dma_sync_for_cpu(rx_buf->xdp, rx_ring->xsk_pool);
599		599
600	xdp_res = ice_run_xdp_zc(rx_ring, rx_buf->xdp);	600	xdp_res = ice_run_xdp_zc(rx_ring, rx_buf->xdp);
601	if (xdp_res) {	601	if (xdp_res) {
@@ -645,11 +645,11 @@ int ice_clean_rx_irq_zc(struct ice_ring *rx_ring, int budget) · 43 unmodified lines
645	ice_finalize_xdp_rx(rx_ring, xdp_xmit);	645	ice_finalize_xdp_rx(rx_ring, xdp_xmit);
646	ice_update_rx_ring_stats(rx_ring, total_rx_packets, total_rx_bytes);	646	ice_update_rx_ring_stats(rx_ring, total_rx_packets, total_rx_bytes);
647		647
648	if (xsk_umem_uses_need_wakeup(rx_ring->xsk_umem)) {	648	if (xsk_uses_need_wakeup(rx_ring->xsk_pool)) {
649	if (failure \|\| rx_ring->next_to_clean == rx_ring->next_to_use)	649	if (failure \|\| rx_ring->next_to_clean == rx_ring->next_to_use)
650	xsk_set_rx_need_wakeup(rx_ring->xsk_umem);	650	xsk_set_rx_need_wakeup(rx_ring->xsk_pool);
651	else	651	else
652	xsk_clear_rx_need_wakeup(rx_ring->xsk_umem);	652	xsk_clear_rx_need_wakeup(rx_ring->xsk_pool);
653		653
654	return (int)total_rx_packets;	654	return (int)total_rx_packets;
655	}	655	}
@@ -682,11 +682,11 @@ static bool ice_xmit_zc(struct ice_ring *xdp_ring, int budget) · 26 unmodified lines
682		682
683	tx_buf = &xdp_ring->tx_buf[xdp_ring->next_to_use];	683	tx_buf = &xdp_ring->tx_buf[xdp_ring->next_to_use];
684		684
685	if (!xsk_umem_consume_tx(xdp_ring->xsk_umem, &desc))	685	if (!xsk_tx_peek_desc(xdp_ring->xsk_pool, &desc))
686	break;	686	break;
687		687
688	dma = xsk_buff_raw_get_dma(xdp_ring->xsk_umem, desc.addr);	688	dma = xsk_buff_raw_get_dma(xdp_ring->xsk_pool, desc.addr);
689	xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_umem, dma,	689	xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_pool, dma,
690	desc.len);	690	desc.len);
691		691
692	tx_buf->bytecount = desc.len;	692	tx_buf->bytecount = desc.len;
@@ -703,7 +703,7 @@ static bool ice_xmit_zc(struct ice_ring *xdp_ring, int budget) · 10 unmodified lines
703		703
704	if (tx_desc) {	704	if (tx_desc) {
705	ice_xdp_ring_update_tail(xdp_ring);	705	ice_xdp_ring_update_tail(xdp_ring);
706	xsk_umem_consume_tx_done(xdp_ring->xsk_umem);	706	xsk_tx_release(xdp_ring->xsk_pool);
707	}	707	}
708		708
709	return budget > 0 && work_done;	709	return budget > 0 && work_done;
@@ -777,10 +777,10 @@ bool ice_clean_tx_irq_zc(struct ice_ring *xdp_ring, int budget) · 67 unmodified lines
777	xdp_ring->next_to_clean = ntc;	777	xdp_ring->next_to_clean = ntc;
778		778
779	if (xsk_frames)	779	if (xsk_frames)
780	xsk_umem_complete_tx(xdp_ring->xsk_umem, xsk_frames);	780	xsk_tx_completed(xdp_ring->xsk_pool, xsk_frames);
781		781
782	if (xsk_umem_uses_need_wakeup(xdp_ring->xsk_umem))	782	if (xsk_uses_need_wakeup(xdp_ring->xsk_pool))
783	xsk_set_tx_need_wakeup(xdp_ring->xsk_umem);	783	xsk_set_tx_need_wakeup(xdp_ring->xsk_pool);
784		784
785	ice_update_tx_ring_stats(xdp_ring, total_packets, total_bytes);	785	ice_update_tx_ring_stats(xdp_ring, total_packets, total_bytes);
786	xmit_done = ice_xmit_zc(xdp_ring, ICE_DFLT_IRQ_WORK);	786	xmit_done = ice_xmit_zc(xdp_ring, ICE_DFLT_IRQ_WORK);
@@ -814,7 +814,7 @@ ice_xsk_wakeup(struct net_device *netdev, u32 queue_id, · 27 unmodified lines
814	if (queue_id >= vsi->num_txq)	814	if (queue_id >= vsi->num_txq)
815	return -ENXIO;	815	return -ENXIO;
816		816
817	if (!vsi->xdp_rings[queue_id]->xsk_umem)	817	if (!vsi->xdp_rings[queue_id]->xsk_pool)
818	return -ENXIO;	818	return -ENXIO;
819		819
820	ring = vsi->xdp_rings[queue_id];	820	ring = vsi->xdp_rings[queue_id];
@@ -833,20 +833,20 @@ ice_xsk_wakeup(struct net_device *netdev, u32 queue_id, · 12 unmodified lines
833	}	833	}
834		834
835	/**	835	/**
836	* ice_xsk_any_rx_ring_ena - Checks if Rx rings have AF_XDP UMEM attached	836	* ice_xsk_any_rx_ring_ena - Checks if Rx rings have AF_XDP buff pool attached
837	* @vsi: VSI to be checked	837	* @vsi: VSI to be checked
838	*	838	*
839	* Returns true if any of the Rx rings has an AF_XDP UMEM attached	839	* Returns true if any of the Rx rings has an AF_XDP buff pool attached
840	*/	840	*/
841	bool ice_xsk_any_rx_ring_ena(struct ice_vsi *vsi)	841	bool ice_xsk_any_rx_ring_ena(struct ice_vsi *vsi)
842	{	842	{
843	int i;	843	int i;
844		844
845	if (!vsi->xsk_umems)	845	if (!vsi->xsk_pools)
846	return false;	846	return false;
847		847
848	for (i = 0; i < vsi->num_xsk_umems; i++) {	848	for (i = 0; i < vsi->num_xsk_pools; i++) {
849	if (vsi->xsk_umems[i])	849	if (vsi->xsk_pools[i])
850	return true;	850	return true;
851	}	851	}
852		852
@@ -854,7 +854,7 @@ bool ice_xsk_any_rx_ring_ena(struct ice_vsi *vsi) · 1 unmodified line
854	}	854	}
855		855
856	/**	856	/**
857	* ice_xsk_clean_rx_ring - clean UMEM queues connected to a given Rx ring	857	* ice_xsk_clean_rx_ring - clean buffer pool queues connected to a given Rx ring
858	* @rx_ring: ring to be cleaned	858	* @rx_ring: ring to be cleaned
859	*/	859	*/
860	void ice_xsk_clean_rx_ring(struct ice_ring *rx_ring)	860	void ice_xsk_clean_rx_ring(struct ice_ring *rx_ring)
@@ -872,7 +872,7 @@ void ice_xsk_clean_rx_ring(struct ice_ring *rx_ring) · 11 unmodified lines
872	}	872	}
873		873
874	/**	874	/**
875	* ice_xsk_clean_xdp_ring - Clean the XDP Tx ring and its UMEM queues	875	* ice_xsk_clean_xdp_ring - Clean the XDP Tx ring and its buffer pool queues
876	* @xdp_ring: XDP_Tx ring	876	* @xdp_ring: XDP_Tx ring
877	*/	877	*/
878	void ice_xsk_clean_xdp_ring(struct ice_ring *xdp_ring)	878	void ice_xsk_clean_xdp_ring(struct ice_ring *xdp_ring)
@@ -896,5 +896,5 @@ void ice_xsk_clean_xdp_ring(struct ice_ring *xdp_ring) · 17 unmodified lines
896	}	896	}
897		897
898	if (xsk_frames)	898	if (xsk_frames)
899	xsk_umem_complete_tx(xdp_ring->xsk_umem, xsk_frames);	899	xsk_tx_completed(xdp_ring->xsk_pool, xsk_frames);
900	}	900	}

drivers/net/ethernet/intel/ice/ice_xsk.h+4−3

 struct ice_vsi;
 #ifdef CONFIG_XDP_SOCKETS
-int ice_xsk_umem_setup(struct ice_vsi *vsi, struct xdp_umem *umem, u16 qid);
+int ice_xsk_pool_setup(struct ice_vsi *vsi, struct xsk_buff_pool *pool,
+		       u16 qid);
 int ice_clean_rx_irq_zc(struct ice_ring *rx_ring, int budget);
 bool ice_clean_tx_irq_zc(struct ice_ring *xdp_ring, int budget);
 int ice_xsk_wakeup(struct net_device *netdev, u32 queue_id, u32 flags);
 void ice_xsk_clean_xdp_ring(struct ice_ring *xdp_ring);
 #else
 static inline int
-ice_xsk_umem_setup(struct ice_vsi __always_unused *vsi,
+ice_xsk_pool_setup(struct ice_vsi __always_unused *vsi,
-		   struct xdp_umem __always_unused *umem,
+		   struct xsk_buff_pool __always_unused *pool,
 		   u16 __always_unused qid)
 {
 	return -EOPNOTSUPP;

drivers/net/ethernet/intel/ixgbe/ixgbe.h+1−1

@@ -350,7 +350,7 @@ struct ixgbe_ring { · 349 unmodified lines
350	struct ixgbe_rx_queue_stats rx_stats;	350	struct ixgbe_rx_queue_stats rx_stats;
351	};	351	};
352	struct xdp_rxq_info xdp_rxq;	352	struct xdp_rxq_info xdp_rxq;
353	struct xdp_umem *xsk_umem;	353	struct xsk_buff_pool *xsk_pool;
354	u16 ring_idx; /* {rx,tx,xdp}_ring back reference idx */	354	u16 ring_idx; /* {rx,tx,xdp}_ring back reference idx */
355	u16 rx_buf_len;	355	u16 rx_buf_len;
356	} ____cacheline_internodealigned_in_smp;	356	} ____cacheline_internodealigned_in_smp;

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c+17−17

@@ -3151,7 +3151,7 @@ int ixgbe_poll(struct napi_struct *napi, int budget) · 3150 unmodified lines
3151	#endif	3151	#endif
3152		3152
3153	ixgbe_for_each_ring(ring, q_vector->tx) {	3153	ixgbe_for_each_ring(ring, q_vector->tx) {
3154	bool wd = ring->xsk_umem ?	3154	bool wd = ring->xsk_pool ?
3155	ixgbe_clean_xdp_tx_irq(q_vector, ring, budget) :	3155	ixgbe_clean_xdp_tx_irq(q_vector, ring, budget) :
3156	ixgbe_clean_tx_irq(q_vector, ring, budget);	3156	ixgbe_clean_tx_irq(q_vector, ring, budget);
3157		3157
@@ -3171,7 +3171,7 @@ int ixgbe_poll(struct napi_struct *napi, int budget) · 13 unmodified lines
3171	per_ring_budget = budget;	3171	per_ring_budget = budget;
3172		3172
3173	ixgbe_for_each_ring(ring, q_vector->rx) {	3173	ixgbe_for_each_ring(ring, q_vector->rx) {
3174	int cleaned = ring->xsk_umem ?	3174	int cleaned = ring->xsk_pool ?
3175	ixgbe_clean_rx_irq_zc(q_vector, ring,	3175	ixgbe_clean_rx_irq_zc(q_vector, ring,
3176	per_ring_budget) :	3176	per_ring_budget) :
3177	ixgbe_clean_rx_irq(q_vector, ring,	3177	ixgbe_clean_rx_irq(q_vector, ring,
@@ -3466,9 +3466,9 @@ void ixgbe_configure_tx_ring(struct ixgbe_adapter *adapter, · 288 unmodified lines
3466	u32 txdctl = IXGBE_TXDCTL_ENABLE;	3466	u32 txdctl = IXGBE_TXDCTL_ENABLE;
3467	u8 reg_idx = ring->reg_idx;	3467	u8 reg_idx = ring->reg_idx;
3468		3468
3469	ring->xsk_umem = NULL;	3469	ring->xsk_pool = NULL;
3470	if (ring_is_xdp(ring))	3470	if (ring_is_xdp(ring))
3471	ring->xsk_umem = ixgbe_xsk_umem(adapter, ring);	3471	ring->xsk_pool = ixgbe_xsk_pool(adapter, ring);
3472		3472
3473	/* disable queue to avoid issues while updating state */	3473	/* disable queue to avoid issues while updating state */
3474	IXGBE_WRITE_REG(hw, IXGBE_TXDCTL(reg_idx), 0);	3474	IXGBE_WRITE_REG(hw, IXGBE_TXDCTL(reg_idx), 0);
@@ -3708,8 +3708,8 @@ static void ixgbe_configure_srrctl(struct ixgbe_adapter *adapter, · 233 unmodified lines
3708	srrctl = IXGBE_RX_HDR_SIZE << IXGBE_SRRCTL_BSIZEHDRSIZE_SHIFT;	3708	srrctl = IXGBE_RX_HDR_SIZE << IXGBE_SRRCTL_BSIZEHDRSIZE_SHIFT;
3709		3709
3710	/* configure the packet buffer length */	3710	/* configure the packet buffer length */
3711	if (rx_ring->xsk_umem) {	3711	if (rx_ring->xsk_pool) {
3712	u32 xsk_buf_len = xsk_umem_get_rx_frame_size(rx_ring->xsk_umem);	3712	u32 xsk_buf_len = xsk_pool_get_rx_frame_size(rx_ring->xsk_pool);
3713		3713
3714	/* If the MAC support setting RXDCTL.RLPML, the	3714	/* If the MAC support setting RXDCTL.RLPML, the
3715	* SRRCTL[n].BSIZEPKT is set to PAGE_SIZE and	3715	* SRRCTL[n].BSIZEPKT is set to PAGE_SIZE and
@@ -4054,12 +4054,12 @@ void ixgbe_configure_rx_ring(struct ixgbe_adapter *adapter, · 338 unmodified lines
4054	u8 reg_idx = ring->reg_idx;	4054	u8 reg_idx = ring->reg_idx;
4055		4055
4056	xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);	4056	xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
4057	ring->xsk_umem = ixgbe_xsk_umem(adapter, ring);	4057	ring->xsk_pool = ixgbe_xsk_pool(adapter, ring);
4058	if (ring->xsk_umem) {	4058	if (ring->xsk_pool) {
4059	WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,	4059	WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
4060	MEM_TYPE_XSK_BUFF_POOL,	4060	MEM_TYPE_XSK_BUFF_POOL,
4061	NULL));	4061	NULL));
4062	xsk_buff_set_rxq_info(ring->xsk_umem, &ring->xdp_rxq);	4062	xsk_pool_set_rxq_info(ring->xsk_pool, &ring->xdp_rxq);
4063	} else {	4063	} else {
4064	WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,	4064	WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
4065	MEM_TYPE_PAGE_SHARED, NULL));	4065	MEM_TYPE_PAGE_SHARED, NULL));
@@ -4114,8 +4114,8 @@ void ixgbe_configure_rx_ring(struct ixgbe_adapter *adapter, · 48 unmodified lines
4114	#endif	4114	#endif
4115	}	4115	}
4116		4116
4117	if (ring->xsk_umem && hw->mac.type != ixgbe_mac_82599EB) {	4117	if (ring->xsk_pool && hw->mac.type != ixgbe_mac_82599EB) {
4118	u32 xsk_buf_len = xsk_umem_get_rx_frame_size(ring->xsk_umem);	4118	u32 xsk_buf_len = xsk_pool_get_rx_frame_size(ring->xsk_pool);
4119		4119
4120	rxdctl &= ~(IXGBE_RXDCTL_RLPMLMASK \|	4120	rxdctl &= ~(IXGBE_RXDCTL_RLPMLMASK \|
4121	IXGBE_RXDCTL_RLPML_EN);	4121	IXGBE_RXDCTL_RLPML_EN);
@@ -4137,7 +4137,7 @@ void ixgbe_configure_rx_ring(struct ixgbe_adapter *adapter, · 15 unmodified lines
4137	IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(reg_idx), rxdctl);	4137	IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(reg_idx), rxdctl);
4138		4138
4139	ixgbe_rx_desc_queue_enable(adapter, ring);	4139	ixgbe_rx_desc_queue_enable(adapter, ring);
4140	if (ring->xsk_umem)	4140	if (ring->xsk_pool)
4141	ixgbe_alloc_rx_buffers_zc(ring, ixgbe_desc_unused(ring));	4141	ixgbe_alloc_rx_buffers_zc(ring, ixgbe_desc_unused(ring));
4142	else	4142	else
4143	ixgbe_alloc_rx_buffers(ring, ixgbe_desc_unused(ring));	4143	ixgbe_alloc_rx_buffers(ring, ixgbe_desc_unused(ring));
@@ -5287,7 +5287,7 @@ static void ixgbe_clean_rx_ring(struct ixgbe_ring *rx_ring) · 1143 unmodified lines
5287	u16 i = rx_ring->next_to_clean;	5287	u16 i = rx_ring->next_to_clean;
5288	struct ixgbe_rx_buffer *rx_buffer = &rx_ring->rx_buffer_info[i];	5288	struct ixgbe_rx_buffer *rx_buffer = &rx_ring->rx_buffer_info[i];
5289		5289
5290	if (rx_ring->xsk_umem) {	5290	if (rx_ring->xsk_pool) {
5291	ixgbe_xsk_clean_rx_ring(rx_ring);	5291	ixgbe_xsk_clean_rx_ring(rx_ring);
5292	goto skip_free;	5292	goto skip_free;
5293	}	5293	}
@@ -5979,7 +5979,7 @@ static void ixgbe_clean_tx_ring(struct ixgbe_ring *tx_ring) · 685 unmodified lines
5979	u16 i = tx_ring->next_to_clean;	5979	u16 i = tx_ring->next_to_clean;
5980	struct ixgbe_tx_buffer *tx_buffer = &tx_ring->tx_buffer_info[i];	5980	struct ixgbe_tx_buffer *tx_buffer = &tx_ring->tx_buffer_info[i];
5981		5981
5982	if (tx_ring->xsk_umem) {	5982	if (tx_ring->xsk_pool) {
5983	ixgbe_xsk_clean_tx_ring(tx_ring);	5983	ixgbe_xsk_clean_tx_ring(tx_ring);
5984	goto out;	5984	goto out;
5985	}	5985	}
@@ -10141,7 +10141,7 @@ static int ixgbe_xdp_setup(struct net_device dev, struct bpf_prog prog) · 4155 unmodified lines
10141	*/	10141	*/
10142	if (need_reset && prog)	10142	if (need_reset && prog)
10143	for (i = 0; i < adapter->num_rx_queues; i++)	10143	for (i = 0; i < adapter->num_rx_queues; i++)
10144	if (adapter->xdp_ring[i]->xsk_umem)	10144	if (adapter->xdp_ring[i]->xsk_pool)
10145	(void)ixgbe_xsk_wakeup(adapter->netdev, i,	10145	(void)ixgbe_xsk_wakeup(adapter->netdev, i,
10146	XDP_WAKEUP_RX);	10146	XDP_WAKEUP_RX);
10147		10147
@@ -10155,8 +10155,8 @@ static int ixgbe_xdp(struct net_device dev, struct netdev_bpf xdp) · 7 unmodified lines
10155	switch (xdp->command) {	10155	switch (xdp->command) {
10156	case XDP_SETUP_PROG:	10156	case XDP_SETUP_PROG:
10157	return ixgbe_xdp_setup(dev, xdp->prog);	10157	return ixgbe_xdp_setup(dev, xdp->prog);
10158	case XDP_SETUP_XSK_UMEM:	10158	case XDP_SETUP_XSK_POOL:
10159	return ixgbe_xsk_umem_setup(adapter, xdp->xsk.umem,	10159	return ixgbe_xsk_pool_setup(adapter, xdp->xsk.pool,
10160	xdp->xsk.queue_id);	10160	xdp->xsk.queue_id);
10161		10161
10162	default:	10162	default:

drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h+4−3

 void ixgbe_txrx_ring_disable(struct ixgbe_adapter *adapter, int ring);
 void ixgbe_txrx_ring_enable(struct ixgbe_adapter *adapter, int ring);
-struct xdp_umem *ixgbe_xsk_umem(struct ixgbe_adapter *adapter,
+struct xsk_buff_pool *ixgbe_xsk_pool(struct ixgbe_adapter *adapter,
-				struct ixgbe_ring *ring);
+				     struct ixgbe_ring *ring);
-int ixgbe_xsk_umem_setup(struct ixgbe_adapter *adapter, struct xdp_umem *umem,
+int ixgbe_xsk_pool_setup(struct ixgbe_adapter *adapter,
+			 struct xsk_buff_pool *pool,
 			 u16 qid);
 void ixgbe_zca_free(struct zero_copy_allocator *alloc, unsigned long handle);

drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c+32−31

 #include "ixgbe.h"
 #include "ixgbe_txrx_common.h"
-struct xdp_umem *ixgbe_xsk_umem(struct ixgbe_adapter *adapter,
+struct xsk_buff_pool *ixgbe_xsk_pool(struct ixgbe_adapter *adapter,
-				struct ixgbe_ring *ring)
+				     struct ixgbe_ring *ring)
 {
 	bool xdp_on = READ_ONCE(adapter->xdp_prog);
 	int qid = ring->ring_idx;
 	if (!xdp_on || !test_bit(qid, adapter->af_xdp_zc_qps))
 		return NULL;
-	return xdp_get_umem_from_qid(adapter->netdev, qid);
+	return xsk_get_pool_from_qid(adapter->netdev, qid);
 }
-static int ixgbe_xsk_umem_enable(struct ixgbe_adapter *adapter,
+static int ixgbe_xsk_pool_enable(struct ixgbe_adapter *adapter,
-				 struct xdp_umem *umem,
+				 struct xsk_buff_pool *pool,
 				 u16 qid)
 {
 	struct net_device *netdev = adapter->netdev;
 	    qid >= netdev->real_num_tx_queues)
 		return -EINVAL;
-	err = xsk_buff_dma_map(umem, &adapter->pdev->dev, IXGBE_RX_DMA_ATTR);
+	err = xsk_pool_dma_map(pool, &adapter->pdev->dev, IXGBE_RX_DMA_ATTR);
 	if (err)
 		return err;
 	return 0;
 }
-static int ixgbe_xsk_umem_disable(struct ixgbe_adapter *adapter, u16 qid)
+static int ixgbe_xsk_pool_disable(struct ixgbe_adapter *adapter, u16 qid)
 {
-	struct xdp_umem *umem;
+	struct xsk_buff_pool *pool;
 	bool if_running;
-	umem = xdp_get_umem_from_qid(adapter->netdev, qid);
+	pool = xsk_get_pool_from_qid(adapter->netdev, qid);
-	if (!umem)
+	if (!pool)
 		return -EINVAL;
 	if_running = netif_running(adapter->netdev) &&
 		ixgbe_txrx_ring_disable(adapter, qid);
 	clear_bit(qid, adapter->af_xdp_zc_qps);
-	xsk_buff_dma_unmap(umem, IXGBE_RX_DMA_ATTR);
+	xsk_pool_dma_unmap(pool, IXGBE_RX_DMA_ATTR);
 	if (if_running)
 		ixgbe_txrx_ring_enable(adapter, qid);
 	return 0;
 }
-int ixgbe_xsk_umem_setup(struct ixgbe_adapter *adapter, struct xdp_umem *umem,
+int ixgbe_xsk_pool_setup(struct ixgbe_adapter *adapter,
+			 struct xsk_buff_pool *pool,
 			 u16 qid)
 {
-	return umem ? ixgbe_xsk_umem_enable(adapter, umem, qid) :
+	return pool ? ixgbe_xsk_pool_enable(adapter, pool, qid) :
-		ixgbe_xsk_umem_disable(adapter, qid);
+		ixgbe_xsk_pool_disable(adapter, qid);
 }
 static int ixgbe_run_xdp_zc(struct ixgbe_adapter *adapter,
 	i -= rx_ring->count;
 	do {
-		bi->xdp = xsk_buff_alloc(rx_ring->xsk_umem);
+		bi->xdp = xsk_buff_alloc(rx_ring->xsk_pool);
 		if (!bi->xdp) {
 			ok = false;
 			break;
 		}
 		bi->xdp->data_end = bi->xdp->data + size;
-		xsk_buff_dma_sync_for_cpu(bi->xdp);
+		xsk_buff_dma_sync_for_cpu(bi->xdp, rx_ring->xsk_pool);
 		xdp_res = ixgbe_run_xdp_zc(adapter, rx_ring, bi->xdp);
 		if (xdp_res) {
 	q_vector->rx.total_packets += total_rx_packets;
 	q_vector->rx.total_bytes += total_rx_bytes;
-	if (xsk_umem_uses_need_wakeup(rx_ring->xsk_umem)) {
+	if (xsk_uses_need_wakeup(rx_ring->xsk_pool)) {
 		if (failure || rx_ring->next_to_clean == rx_ring->next_to_use)
-			xsk_set_rx_need_wakeup(rx_ring->xsk_umem);
+			xsk_set_rx_need_wakeup(rx_ring->xsk_pool);
 		else
-			xsk_clear_rx_need_wakeup(rx_ring->xsk_umem);
+			xsk_clear_rx_need_wakeup(rx_ring->xsk_pool);
 		return (int)total_rx_packets;
 	}
 static bool ixgbe_xmit_zc(struct ixgbe_ring *xdp_ring, unsigned int budget)
 {
+	struct xsk_buff_pool *pool = xdp_ring->xsk_pool;
 	union ixgbe_adv_tx_desc *tx_desc = NULL;
 	struct ixgbe_tx_buffer *tx_bi;
 	bool work_done = true;
 			break;
 		}
-		if (!xsk_umem_consume_tx(xdp_ring->xsk_umem, &desc))
+		if (!xsk_tx_peek_desc(pool, &desc))
 			break;
-		dma = xsk_buff_raw_get_dma(xdp_ring->xsk_umem, desc.addr);
+		dma = xsk_buff_raw_get_dma(pool, desc.addr);
-		xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_umem, dma,
+		xsk_buff_raw_dma_sync_for_device(pool, dma, desc.len);
-						 desc.len);
 		tx_bi = &xdp_ring->tx_buffer_info[xdp_ring->next_to_use];
 		tx_bi->bytecount = desc.len;
 	if (tx_desc) {
 		ixgbe_xdp_ring_update_tail(xdp_ring);
-		xsk_umem_consume_tx_done(xdp_ring->xsk_umem);
+		xsk_tx_release(pool);
 	}
 	return !!budget && work_done;
 {
 	u16 ntc = tx_ring->next_to_clean, ntu = tx_ring->next_to_use;
 	unsigned int total_packets = 0, total_bytes = 0;
-	struct xdp_umem *umem = tx_ring->xsk_umem;
+	struct xsk_buff_pool *pool = tx_ring->xsk_pool;
 	union ixgbe_adv_tx_desc *tx_desc;
 	struct ixgbe_tx_buffer *tx_bi;
 	u32 xsk_frames = 0;
 	q_vector->tx.total_packets += total_packets;
 	if (xsk_frames)
-		xsk_umem_complete_tx(umem, xsk_frames);
+		xsk_tx_completed(pool, xsk_frames);
-	if (xsk_umem_uses_need_wakeup(tx_ring->xsk_umem))
+	if (xsk_uses_need_wakeup(pool))
-		xsk_set_tx_need_wakeup(tx_ring->xsk_umem);
+		xsk_set_tx_need_wakeup(pool);
 	return ixgbe_xmit_zc(tx_ring, q_vector->tx.work_limit);
 }
 	if (test_bit(__IXGBE_TX_DISABLED, &ring->state))
 		return -ENETDOWN;
-	if (!ring->xsk_umem)
+	if (!ring->xsk_pool)
 		return -ENXIO;
 	if (!napi_if_scheduled_mark_missed(&ring->q_vector->napi)) {
 void ixgbe_xsk_clean_tx_ring(struct ixgbe_ring *tx_ring)
 {
 	u16 ntc = tx_ring->next_to_clean, ntu = tx_ring->next_to_use;
-	struct xdp_umem *umem = tx_ring->xsk_umem;
+	struct xsk_buff_pool *pool = tx_ring->xsk_pool;
 	struct ixgbe_tx_buffer *tx_bi;
 	u32 xsk_frames = 0;
 	}
 	if (xsk_frames)
-		xsk_umem_complete_tx(umem, xsk_frames);
+		xsk_tx_completed(pool, xsk_frames);
 }

drivers/net/ethernet/mellanox/mlx5/core/Makefile+1−1

@@ -24,7 +24,7 @@ mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \ · 23 unmodified lines
24	mlx5_core-$(CONFIG_MLX5_CORE_EN) += en_main.o en_common.o en_fs.o en_ethtool.o \	24	mlx5_core-$(CONFIG_MLX5_CORE_EN) += en_main.o en_common.o en_fs.o en_ethtool.o \
25	en_tx.o en_rx.o en_dim.o en_txrx.o en/xdp.o en_stats.o \	25	en_tx.o en_rx.o en_dim.o en_txrx.o en/xdp.o en_stats.o \
26	en_selftest.o en/port.o en/monitor_stats.o en/health.o \	26	en_selftest.o en/port.o en/monitor_stats.o en/health.o \
27	en/reporter_tx.o en/reporter_rx.o en/params.o en/xsk/umem.o \	27	en/reporter_tx.o en/reporter_rx.o en/params.o en/xsk/pool.o \
28	en/xsk/setup.o en/xsk/rx.o en/xsk/tx.o en/devlink.o	28	en/xsk/setup.o en/xsk/rx.o en/xsk/tx.o en/devlink.o
29		29
30	#	30	#

drivers/net/ethernet/mellanox/mlx5/core/en.h+10−9

 	struct mlx5e_cq            cq;
 	/* read only */
-	struct xdp_umem           *umem;
+	struct xsk_buff_pool      *xsk_pool;
 	struct mlx5_wq_cyc         wq;
 	struct mlx5e_xdpsq_stats  *stats;
 	mlx5e_fp_xmit_xdp_frame_check xmit_xdp_frame_check;
 	struct page_pool      *page_pool;
 	/* AF_XDP zero-copy */
-	struct xdp_umem       *umem;
+	struct xsk_buff_pool  *xsk_pool;
 	struct work_struct     recover_work;
 #endif
 struct mlx5e_xsk {
-	/* UMEMs are stored separately from channels, because we don't want to
+	/* XSK buffer pools are stored separately from channels,
-	 * lose them when channels are recreated. The kernel also stores UMEMs,
+	 * because we don't want to lose them when channels are
-	 * but it doesn't distinguish between zero-copy and non-zero-copy UMEMs,
+	 * recreated. The kernel also stores buffer pool, but it doesn't
-	 * so rely on our mechanism.
+	 * distinguish between zero-copy and non-zero-copy UMEMs, so
+	 * rely on our mechanism.
 	 */
-	struct xdp_umem **umems;
+	struct xsk_buff_pool **pools;
 	u16 refcnt;
 	bool ever_used;
 };
 struct mlx5e_rq_param;
 int mlx5e_open_rq(struct mlx5e_channel *c, struct mlx5e_params *params,
 		  struct mlx5e_rq_param *param, struct mlx5e_xsk_param *xsk,
-		  struct xdp_umem *umem, struct mlx5e_rq *rq);
+		  struct xsk_buff_pool *xsk_pool, struct mlx5e_rq *rq);
 int mlx5e_wait_for_min_rx_wqes(struct mlx5e_rq *rq, int wait_time);
 void mlx5e_deactivate_rq(struct mlx5e_rq *rq);
 void mlx5e_close_rq(struct mlx5e_rq *rq);
 		     struct mlx5e_sq_param *param, struct mlx5e_icosq *sq);
 void mlx5e_close_icosq(struct mlx5e_icosq *sq);
 int mlx5e_open_xdpsq(struct mlx5e_channel *c, struct mlx5e_params *params,
-		     struct mlx5e_sq_param *param, struct xdp_umem *umem,
+		     struct mlx5e_sq_param *param, struct xsk_buff_pool *xsk_pool,
 		     struct mlx5e_xdpsq *sq, bool is_redirect);
 void mlx5e_close_xdpsq(struct mlx5e_xdpsq *sq);

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c+2−3

@@ -445,7 +445,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq) · 444 unmodified lines
445	} while ((++i < MLX5E_TX_CQ_POLL_BUDGET) && (cqe = mlx5_cqwq_get_cqe(&cq->wq)));	445	} while ((++i < MLX5E_TX_CQ_POLL_BUDGET) && (cqe = mlx5_cqwq_get_cqe(&cq->wq)));
446		446
447	if (xsk_frames)	447	if (xsk_frames)
448	xsk_umem_complete_tx(sq->umem, xsk_frames);	448	xsk_tx_completed(sq->xsk_pool, xsk_frames);
449		449
450	sq->stats->cqes += i;	450	sq->stats->cqes += i;
451		451
@@ -475,7 +475,7 @@ void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq) · 23 unmodified lines
475	}	475	}
476		476
477	if (xsk_frames)	477	if (xsk_frames)
478	xsk_umem_complete_tx(sq->umem, xsk_frames);	478	xsk_tx_completed(sq->xsk_pool, xsk_frames);
479	}	479	}
480		480
481	int mlx5e_xdp_xmit(struct net_device dev, int n, struct xdp_frame *frames,	481	int mlx5e_xdp_xmit(struct net_device dev, int n, struct xdp_frame *frames,
@@ -563,4 +563,3 @@ void mlx5e_set_xmit_fp(struct mlx5e_xdpsq *sq, bool is_mpw) · 81 unmodified lines
563	sq->xmit_xdp_frame = is_mpw ?	563	sq->xmit_xdp_frame = is_mpw ?
564	mlx5e_xmit_xdp_frame_mpwqe : mlx5e_xmit_xdp_frame;	564	mlx5e_xmit_xdp_frame_mpwqe : mlx5e_xmit_xdp_frame;
565	}	565	}
566

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/pool.crenamed from drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.c+55−55

@@ -1,31 +1,31 @@
1	// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB	1	// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2	/* Copyright (c) 2019 Mellanox Technologies. */	2	/* Copyright (c) 2019-2020, Mellanox Technologies inc. All rights reserved. */
3		3
4	#include <net/xdp_sock_drv.h>	4	#include <net/xdp_sock_drv.h>
5	#include "umem.h"	5	#include "pool.h"
6	#include "setup.h"	6	#include "setup.h"
7	#include "en/params.h"	7	#include "en/params.h"
8		8
9	static int mlx5e_xsk_map_umem(struct mlx5e_priv *priv,	9	static int mlx5e_xsk_map_pool(struct mlx5e_priv *priv,
10	struct xdp_umem *umem)	10	struct xsk_buff_pool *pool)
11	{	11	{
12	struct device *dev = priv->mdev->device;	12	struct device *dev = priv->mdev->device;
13		13
14	return xsk_buff_dma_map(umem, dev, 0);	14	return xsk_pool_dma_map(pool, dev, 0);
15	}	15	}
16		16
17	static void mlx5e_xsk_unmap_umem(struct mlx5e_priv *priv,	17	static void mlx5e_xsk_unmap_pool(struct mlx5e_priv *priv,
18	struct xdp_umem *umem)	18	struct xsk_buff_pool *pool)
19	{	19	{
20	return xsk_buff_dma_unmap(umem, 0);	20	return xsk_pool_dma_unmap(pool, 0);
21	}	21	}
22		22
23	static int mlx5e_xsk_get_umems(struct mlx5e_xsk *xsk)	23	static int mlx5e_xsk_get_pools(struct mlx5e_xsk *xsk)
24	{	24	{
25	if (!xsk->umems) {	25	if (!xsk->pools) {
26	xsk->umems = kcalloc(MLX5E_MAX_NUM_CHANNELS,	26	xsk->pools = kcalloc(MLX5E_MAX_NUM_CHANNELS,
27	sizeof(*xsk->umems), GFP_KERNEL);	27	sizeof(*xsk->pools), GFP_KERNEL);
28	if (unlikely(!xsk->umems))	28	if (unlikely(!xsk->pools))
29	return -ENOMEM;	29	return -ENOMEM;
30	}	30	}
31		31
@@ -35,68 +35,68 @@ static int mlx5e_xsk_get_umems(struct mlx5e_xsk *xsk) · 3 unmodified lines
35	return 0;	35	return 0;
36	}	36	}
37		37
38	static void mlx5e_xsk_put_umems(struct mlx5e_xsk *xsk)	38	static void mlx5e_xsk_put_pools(struct mlx5e_xsk *xsk)
39	{	39	{
40	if (!--xsk->refcnt) {	40	if (!--xsk->refcnt) {
41	kfree(xsk->umems);	41	kfree(xsk->pools);
42	xsk->umems = NULL;	42	xsk->pools = NULL;
43	}	43	}
44	}	44	}
45		45
46	static int mlx5e_xsk_add_umem(struct mlx5e_xsk xsk, struct xdp_umem umem, u16 ix)	46	static int mlx5e_xsk_add_pool(struct mlx5e_xsk xsk, struct xsk_buff_pool pool, u16 ix)
47	{	47	{
48	int err;	48	int err;
49		49
50	err = mlx5e_xsk_get_umems(xsk);	50	err = mlx5e_xsk_get_pools(xsk);
51	if (unlikely(err))	51	if (unlikely(err))
52	return err;	52	return err;
53		53
54	xsk->umems[ix] = umem;	54	xsk->pools[ix] = pool;
55	return 0;	55	return 0;
56	}	56	}
57		57
58	static void mlx5e_xsk_remove_umem(struct mlx5e_xsk *xsk, u16 ix)	58	static void mlx5e_xsk_remove_pool(struct mlx5e_xsk *xsk, u16 ix)
59	{	59	{
60	xsk->umems[ix] = NULL;	60	xsk->pools[ix] = NULL;
61		61
62	mlx5e_xsk_put_umems(xsk);	62	mlx5e_xsk_put_pools(xsk);
63	}	63	}
64		64
65	static bool mlx5e_xsk_is_umem_sane(struct xdp_umem *umem)	65	static bool mlx5e_xsk_is_pool_sane(struct xsk_buff_pool *pool)
66	{	66	{
67	return xsk_umem_get_headroom(umem) <= 0xffff &&	67	return xsk_pool_get_headroom(pool) <= 0xffff &&
68	xsk_umem_get_chunk_size(umem) <= 0xffff;	68	xsk_pool_get_chunk_size(pool) <= 0xffff;
69	}	69	}
70		70
71	void mlx5e_build_xsk_param(struct xdp_umem umem, struct mlx5e_xsk_param xsk)	71	void mlx5e_build_xsk_param(struct xsk_buff_pool pool, struct mlx5e_xsk_param xsk)
72	{	72	{
73	xsk->headroom = xsk_umem_get_headroom(umem);	73	xsk->headroom = xsk_pool_get_headroom(pool);
74	xsk->chunk_size = xsk_umem_get_chunk_size(umem);	74	xsk->chunk_size = xsk_pool_get_chunk_size(pool);
75	}	75	}
76		76
77	static int mlx5e_xsk_enable_locked(struct mlx5e_priv *priv,	77	static int mlx5e_xsk_enable_locked(struct mlx5e_priv *priv,
78	struct xdp_umem *umem, u16 ix)	78	struct xsk_buff_pool *pool, u16 ix)
79	{	79	{
80	struct mlx5e_params *params = &priv->channels.params;	80	struct mlx5e_params *params = &priv->channels.params;
81	struct mlx5e_xsk_param xsk;	81	struct mlx5e_xsk_param xsk;
82	struct mlx5e_channel *c;	82	struct mlx5e_channel *c;
83	int err;	83	int err;
84		84
85	if (unlikely(mlx5e_xsk_get_umem(&priv->channels.params, &priv->xsk, ix)))	85	if (unlikely(mlx5e_xsk_get_pool(&priv->channels.params, &priv->xsk, ix)))
86	return -EBUSY;	86	return -EBUSY;
87		87
88	if (unlikely(!mlx5e_xsk_is_umem_sane(umem)))	88	if (unlikely(!mlx5e_xsk_is_pool_sane(pool)))
89	return -EINVAL;	89	return -EINVAL;
90		90
91	err = mlx5e_xsk_map_umem(priv, umem);	91	err = mlx5e_xsk_map_pool(priv, pool);
92	if (unlikely(err))	92	if (unlikely(err))
93	return err;	93	return err;
94		94
95	err = mlx5e_xsk_add_umem(&priv->xsk, umem, ix);	95	err = mlx5e_xsk_add_pool(&priv->xsk, pool, ix);
96	if (unlikely(err))	96	if (unlikely(err))
97	goto err_unmap_umem;	97	goto err_unmap_pool;
98		98
99	mlx5e_build_xsk_param(umem, &xsk);	99	mlx5e_build_xsk_param(pool, &xsk);
100		100
101	if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {	101	if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
102	/* XSK objects will be created on open. */	102	/* XSK objects will be created on open. */
@@ -112,9 +112,9 @@ static int mlx5e_xsk_enable_locked(struct mlx5e_priv *priv, · 9 unmodified lines
112		112
113	c = priv->channels.c[ix];	113	c = priv->channels.c[ix];
114		114
115	err = mlx5e_open_xsk(priv, params, &xsk, umem, c);	115	err = mlx5e_open_xsk(priv, params, &xsk, pool, c);
116	if (unlikely(err))	116	if (unlikely(err))
117	goto err_remove_umem;	117	goto err_remove_pool;
118		118
119	mlx5e_activate_xsk(c);	119	mlx5e_activate_xsk(c);
120		120
@@ -132,11 +132,11 @@ err_deactivate: · 11 unmodified lines
132	mlx5e_deactivate_xsk(c);	132	mlx5e_deactivate_xsk(c);
133	mlx5e_close_xsk(c);	133	mlx5e_close_xsk(c);
134		134
135	err_remove_umem:	135	err_remove_pool:
136	mlx5e_xsk_remove_umem(&priv->xsk, ix);	136	mlx5e_xsk_remove_pool(&priv->xsk, ix);
137		137
138	err_unmap_umem:	138	err_unmap_pool:
139	mlx5e_xsk_unmap_umem(priv, umem);	139	mlx5e_xsk_unmap_pool(priv, pool);
140		140
141	return err;	141	return err;
142		142
@@ -146,7 +146,7 @@ validate_closed: · 3 unmodified lines
146	*/	146	*/
147	if (!mlx5e_validate_xsk_param(params, &xsk, priv->mdev)) {	147	if (!mlx5e_validate_xsk_param(params, &xsk, priv->mdev)) {
148	err = -EINVAL;	148	err = -EINVAL;
149	goto err_remove_umem;	149	goto err_remove_pool;
150	}	150	}
151		151
152	return 0;	152	return 0;
@@ -154,45 +154,45 @@ validate_closed: · 1 unmodified line
154		154
155	static int mlx5e_xsk_disable_locked(struct mlx5e_priv *priv, u16 ix)	155	static int mlx5e_xsk_disable_locked(struct mlx5e_priv *priv, u16 ix)
156	{	156	{
157	struct xdp_umem *umem = mlx5e_xsk_get_umem(&priv->channels.params,	157	struct xsk_buff_pool *pool = mlx5e_xsk_get_pool(&priv->channels.params,
158	&priv->xsk, ix);	158	&priv->xsk, ix);
159	struct mlx5e_channel *c;	159	struct mlx5e_channel *c;
160		160
161	if (unlikely(!umem))	161	if (unlikely(!pool))
162	return -EINVAL;	162	return -EINVAL;
163		163
164	if (!test_bit(MLX5E_STATE_OPENED, &priv->state))	164	if (!test_bit(MLX5E_STATE_OPENED, &priv->state))
165	goto remove_umem;	165	goto remove_pool;
166		166
167	/* XSK RQ and SQ are only created if XDP program is set. */	167	/* XSK RQ and SQ are only created if XDP program is set. */
168	if (!priv->channels.params.xdp_prog)	168	if (!priv->channels.params.xdp_prog)
169	goto remove_umem;	169	goto remove_pool;
170		170
171	c = priv->channels.c[ix];	171	c = priv->channels.c[ix];
172	mlx5e_xsk_redirect_rqt_to_drop(priv, ix);	172	mlx5e_xsk_redirect_rqt_to_drop(priv, ix);
173	mlx5e_deactivate_xsk(c);	173	mlx5e_deactivate_xsk(c);
174	mlx5e_close_xsk(c);	174	mlx5e_close_xsk(c);
175		175
176	remove_umem:	176	remove_pool:
177	mlx5e_xsk_remove_umem(&priv->xsk, ix);	177	mlx5e_xsk_remove_pool(&priv->xsk, ix);
178	mlx5e_xsk_unmap_umem(priv, umem);	178	mlx5e_xsk_unmap_pool(priv, pool);
179		179
180	return 0;	180	return 0;
181	}	181	}
182		182
183	static int mlx5e_xsk_enable_umem(struct mlx5e_priv priv, struct xdp_umem umem,	183	static int mlx5e_xsk_enable_pool(struct mlx5e_priv priv, struct xsk_buff_pool pool,
184	u16 ix)	184	u16 ix)
185	{	185	{
186	int err;	186	int err;
187		187
188	mutex_lock(&priv->state_lock);	188	mutex_lock(&priv->state_lock);
189	err = mlx5e_xsk_enable_locked(priv, umem, ix);	189	err = mlx5e_xsk_enable_locked(priv, pool, ix);
190	mutex_unlock(&priv->state_lock);	190	mutex_unlock(&priv->state_lock);
191		191
192	return err;	192	return err;
193	}	193	}
194		194
195	static int mlx5e_xsk_disable_umem(struct mlx5e_priv *priv, u16 ix)	195	static int mlx5e_xsk_disable_pool(struct mlx5e_priv *priv, u16 ix)
196	{	196	{
197	int err;	197	int err;
198		198
@@ -203,7 +203,7 @@ static int mlx5e_xsk_disable_umem(struct mlx5e_priv *priv, u16 ix) · 4 unmodified lines
203	return err;	203	return err;
204	}	204	}
205		205
206	int mlx5e_xsk_setup_umem(struct net_device dev, struct xdp_umem umem, u16 qid)	206	int mlx5e_xsk_setup_pool(struct net_device dev, struct xsk_buff_pool pool, u16 qid)
207	{	207	{
208	struct mlx5e_priv *priv = netdev_priv(dev);	208	struct mlx5e_priv *priv = netdev_priv(dev);
209	struct mlx5e_params *params = &priv->channels.params;	209	struct mlx5e_params *params = &priv->channels.params;
@@ -212,6 +212,6 @@ int mlx5e_xsk_setup_umem(struct net_device dev, struct xdp_umem umem, u16 qid) · 2 unmodified lines
212	if (unlikely(!mlx5e_qid_get_ch_if_in_group(params, qid, MLX5E_RQ_GROUP_XSK, &ix)))	212	if (unlikely(!mlx5e_qid_get_ch_if_in_group(params, qid, MLX5E_RQ_GROUP_XSK, &ix)))
213	return -EINVAL;	213	return -EINVAL;
214		214
215	return umem ? mlx5e_xsk_enable_umem(priv, umem, ix) :	215	return pool ? mlx5e_xsk_enable_pool(priv, pool, ix) :
216	mlx5e_xsk_disable_umem(priv, ix);	216	mlx5e_xsk_disable_pool(priv, ix);
217	}	217	}

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/pool.h+27−0

@@ -0,0 +1,27 @@
	1	/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
	2	/* Copyright (c) 2019-2020, Mellanox Technologies inc. All rights reserved. */
	3
	4	#ifndef __MLX5_EN_XSK_POOL_H__
	5	#define __MLX5_EN_XSK_POOL_H__
	6
	7	#include "en.h"
	8
	9	static inline struct xsk_buff_pool mlx5e_xsk_get_pool(struct mlx5e_params params,
	10	struct mlx5e_xsk *xsk, u16 ix)
	11	{
	12	if (!xsk \|\| !xsk->pools)
	13	return NULL;
	14
	15	if (unlikely(ix >= params->num_channels))
	16	return NULL;
	17
	18	return xsk->pools[ix];
	19	}
	20
	21	struct mlx5e_xsk_param;
	22	void mlx5e_build_xsk_param(struct xsk_buff_pool pool, struct mlx5e_xsk_param xsk);
	23
	24	/* .ndo_bpf callback. */
	25	int mlx5e_xsk_setup_pool(struct net_device dev, struct xsk_buff_pool pool, u16 qid);
	26
	27	#endif /* __MLX5_EN_XSK_POOL_H__ */

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c+2−2

@@ -48,7 +48,7 @@ struct sk_buff mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq rq, · 47 unmodified lines
48		48
49	xdp->data_end = xdp->data + cqe_bcnt32;	49	xdp->data_end = xdp->data + cqe_bcnt32;
50	xdp_set_data_meta_invalid(xdp);	50	xdp_set_data_meta_invalid(xdp);
51	xsk_buff_dma_sync_for_cpu(xdp);	51	xsk_buff_dma_sync_for_cpu(xdp, rq->xsk_pool);
52	net_prefetch(xdp->data);	52	net_prefetch(xdp->data);
53		53
54	rcu_read_lock();	54	rcu_read_lock();
@@ -99,7 +99,7 @@ struct sk_buff mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq rq, · 44 unmodified lines
99		99
100	xdp->data_end = xdp->data + cqe_bcnt;	100	xdp->data_end = xdp->data + cqe_bcnt;
101	xdp_set_data_meta_invalid(xdp);	101	xdp_set_data_meta_invalid(xdp);
102	xsk_buff_dma_sync_for_cpu(xdp);	102	xsk_buff_dma_sync_for_cpu(xdp, rq->xsk_pool);
103	net_prefetch(xdp->data);	103	net_prefetch(xdp->data);
104		104
105	if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_RESP_SEND)) {	105	if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_RESP_SEND)) {

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h+5−5

@@ -19,10 +19,10 @@ struct sk_buff mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq rq, · 18 unmodified lines
19	struct mlx5e_wqe_frag_info *wi,	19	struct mlx5e_wqe_frag_info *wi,
20	u32 cqe_bcnt);	20	u32 cqe_bcnt);
21		21
22	static inline int mlx5e_xsk_page_alloc_umem(struct mlx5e_rq *rq,	22	static inline int mlx5e_xsk_page_alloc_pool(struct mlx5e_rq *rq,
23	struct mlx5e_dma_info *dma_info)	23	struct mlx5e_dma_info *dma_info)
24	{	24	{
25	dma_info->xsk = xsk_buff_alloc(rq->umem);	25	dma_info->xsk = xsk_buff_alloc(rq->xsk_pool);
26	if (!dma_info->xsk)	26	if (!dma_info->xsk)
27	return -ENOMEM;	27	return -ENOMEM;
28		28
@@ -38,13 +38,13 @@ static inline int mlx5e_xsk_page_alloc_umem(struct mlx5e_rq *rq, · 9 unmodified lines
38		38
39	static inline bool mlx5e_xsk_update_rx_wakeup(struct mlx5e_rq *rq, bool alloc_err)	39	static inline bool mlx5e_xsk_update_rx_wakeup(struct mlx5e_rq *rq, bool alloc_err)
40	{	40	{
41	if (!xsk_umem_uses_need_wakeup(rq->umem))	41	if (!xsk_uses_need_wakeup(rq->xsk_pool))
42	return alloc_err;	42	return alloc_err;
43		43
44	if (unlikely(alloc_err))	44	if (unlikely(alloc_err))
45	xsk_set_rx_need_wakeup(rq->umem);	45	xsk_set_rx_need_wakeup(rq->xsk_pool);
46	else	46	else
47	xsk_clear_rx_need_wakeup(rq->umem);	47	xsk_clear_rx_need_wakeup(rq->xsk_pool);
48		48
49	return false;	49	return false;
50	}	50	}

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c+6−6

@@ -45,7 +45,7 @@ static void mlx5e_build_xsk_cparam(struct mlx5e_priv *priv, · 44 unmodified lines
45	}	45	}
46		46
47	int mlx5e_open_xsk(struct mlx5e_priv priv, struct mlx5e_params params,	47	int mlx5e_open_xsk(struct mlx5e_priv priv, struct mlx5e_params params,
48	struct mlx5e_xsk_param xsk, struct xdp_umem umem,	48	struct mlx5e_xsk_param xsk, struct xsk_buff_pool pool,
49	struct mlx5e_channel *c)	49	struct mlx5e_channel *c)
50	{	50	{
51	struct mlx5e_channel_param *cparam;	51	struct mlx5e_channel_param *cparam;
@@ -64,7 +64,7 @@ int mlx5e_open_xsk(struct mlx5e_priv priv, struct mlx5e_params params, · 12 unmodified lines
64	if (unlikely(err))	64	if (unlikely(err))
65	goto err_free_cparam;	65	goto err_free_cparam;
66		66
67	err = mlx5e_open_rq(c, params, &cparam->rq, xsk, umem, &c->xskrq);	67	err = mlx5e_open_rq(c, params, &cparam->rq, xsk, pool, &c->xskrq);
68	if (unlikely(err))	68	if (unlikely(err))
69	goto err_close_rx_cq;	69	goto err_close_rx_cq;
70		70
@@ -72,13 +72,13 @@ int mlx5e_open_xsk(struct mlx5e_priv priv, struct mlx5e_params params, · 1 unmodified line
72	if (unlikely(err))	72	if (unlikely(err))
73	goto err_close_rq;	73	goto err_close_rq;
74		74
75	/* Create a separate SQ, so that when the UMEM is disabled, we could	75	/* Create a separate SQ, so that when the buff pool is disabled, we could
76	* close this SQ safely and stop receiving CQEs. In other case, e.g., if	76	* close this SQ safely and stop receiving CQEs. In other case, e.g., if
77	* the XDPSQ was used instead, we might run into trouble when the UMEM	77	* the XDPSQ was used instead, we might run into trouble when the buff pool
78	* is disabled and then reenabled, but the SQ continues receiving CQEs	78	* is disabled and then reenabled, but the SQ continues receiving CQEs
79	* from the old UMEM.	79	* from the old buff pool.
80	*/	80	*/
81	err = mlx5e_open_xdpsq(c, params, &cparam->xdp_sq, umem, &c->xsksq, true);	81	err = mlx5e_open_xdpsq(c, params, &cparam->xdp_sq, pool, &c->xsksq, true);
82	if (unlikely(err))	82	if (unlikely(err))
83	goto err_close_tx_cq;	83	goto err_close_tx_cq;
84		84

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.h+1−1

@@ -12,7 +12,7 @@ bool mlx5e_validate_xsk_param(struct mlx5e_params *params, · 11 unmodified lines
12	struct mlx5e_xsk_param *xsk,	12	struct mlx5e_xsk_param *xsk,
13	struct mlx5_core_dev *mdev);	13	struct mlx5_core_dev *mdev);
14	int mlx5e_open_xsk(struct mlx5e_priv priv, struct mlx5e_params params,	14	int mlx5e_open_xsk(struct mlx5e_priv priv, struct mlx5e_params params,
15	struct mlx5e_xsk_param xsk, struct xdp_umem umem,	15	struct mlx5e_xsk_param xsk, struct xsk_buff_pool pool,
16	struct mlx5e_channel *c);	16	struct mlx5e_channel *c);
17	void mlx5e_close_xsk(struct mlx5e_channel *c);	17	void mlx5e_close_xsk(struct mlx5e_channel *c);
18	void mlx5e_activate_xsk(struct mlx5e_channel *c);	18	void mlx5e_activate_xsk(struct mlx5e_channel *c);

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c+7−7

@@ -2,7 +2,7 @@ · 1 unmodified line
2	/* Copyright (c) 2019 Mellanox Technologies. */	2	/* Copyright (c) 2019 Mellanox Technologies. */
3		3
4	#include "tx.h"	4	#include "tx.h"
5	#include "umem.h"	5	#include "pool.h"
6	#include "en/xdp.h"	6	#include "en/xdp.h"
7	#include "en/params.h"	7	#include "en/params.h"
8	#include <net/xdp_sock_drv.h>	8	#include <net/xdp_sock_drv.h>
@@ -66,7 +66,7 @@ static void mlx5e_xsk_tx_post_err(struct mlx5e_xdpsq *sq, · 57 unmodified lines
66		66
67	bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget)	67	bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget)
68	{	68	{
69	struct xdp_umem *umem = sq->umem;	69	struct xsk_buff_pool *pool = sq->xsk_pool;
70	struct mlx5e_xdp_info xdpi;	70	struct mlx5e_xdp_info xdpi;
71	struct mlx5e_xdp_xmit_data xdptxd;	71	struct mlx5e_xdp_xmit_data xdptxd;
72	bool work_done = true;	72	bool work_done = true;
@@ -87,7 +87,7 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget) · 14 unmodified lines
87	break;	87	break;
88	}	88	}
89		89
90	if (!xsk_umem_consume_tx(umem, &desc)) {	90	if (!xsk_tx_peek_desc(pool, &desc)) {
91	/* TX will get stuck until something wakes it up by	91	/* TX will get stuck until something wakes it up by
92	* triggering NAPI. Currently it's expected that the	92	* triggering NAPI. Currently it's expected that the
93	* application calls sendto() if there are consumed, but	93	* application calls sendto() if there are consumed, but
@@ -96,11 +96,11 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget) · 2 unmodified lines
96	break;	96	break;
97	}	97	}
98		98
99	xdptxd.dma_addr = xsk_buff_raw_get_dma(umem, desc.addr);	99	xdptxd.dma_addr = xsk_buff_raw_get_dma(pool, desc.addr);
100	xdptxd.data = xsk_buff_raw_get_data(umem, desc.addr);	100	xdptxd.data = xsk_buff_raw_get_data(pool, desc.addr);
101	xdptxd.len = desc.len;	101	xdptxd.len = desc.len;
102		102
103	xsk_buff_raw_dma_sync_for_device(umem, xdptxd.dma_addr, xdptxd.len);	103	xsk_buff_raw_dma_sync_for_device(pool, xdptxd.dma_addr, xdptxd.len);
104		104
105	ret = INDIRECT_CALL_2(sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe,	105	ret = INDIRECT_CALL_2(sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe,
106	mlx5e_xmit_xdp_frame, sq, &xdptxd, &xdpi, check_result);	106	mlx5e_xmit_xdp_frame, sq, &xdptxd, &xdpi, check_result);
@@ -119,7 +119,7 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget) · 12 unmodified lines
119	mlx5e_xdp_mpwqe_complete(sq);	119	mlx5e_xdp_mpwqe_complete(sq);
120	mlx5e_xmit_xdp_doorbell(sq);	120	mlx5e_xmit_xdp_doorbell(sq);
121		121
122	xsk_umem_consume_tx_done(umem);	122	xsk_tx_release(pool);
123	}	123	}
124		124
125	return !(budget && work_done);	125	return !(budget && work_done);

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.h+3−3

@@ -15,13 +15,13 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget); · 14 unmodified lines
15		15
16	static inline void mlx5e_xsk_update_tx_wakeup(struct mlx5e_xdpsq *sq)	16	static inline void mlx5e_xsk_update_tx_wakeup(struct mlx5e_xdpsq *sq)
17	{	17	{
18	if (!xsk_umem_uses_need_wakeup(sq->umem))	18	if (!xsk_uses_need_wakeup(sq->xsk_pool))
19	return;	19	return;
20		20
21	if (sq->pc != sq->cc)	21	if (sq->pc != sq->cc)
22	xsk_clear_tx_need_wakeup(sq->umem);	22	xsk_clear_tx_need_wakeup(sq->xsk_pool);
23	else	23	else
24	xsk_set_tx_need_wakeup(sq->umem);	24	xsk_set_tx_need_wakeup(sq->xsk_pool);
25	}	25	}
26		26
27	#endif /* __MLX5_EN_XSK_TX_H__ */	27	#endif /* __MLX5_EN_XSK_TX_H__ */

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.h+0−29

@@ -1,29 +0,0 @@
1	/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
2	/* Copyright (c) 2019 Mellanox Technologies. */
3
4	#ifndef __MLX5_EN_XSK_UMEM_H__
5	#define __MLX5_EN_XSK_UMEM_H__
6
7	#include "en.h"
8
9	static inline struct xdp_umem mlx5e_xsk_get_umem(struct mlx5e_params params,
10	struct mlx5e_xsk *xsk, u16 ix)
11	{
12	if (!xsk \|\| !xsk->umems)
13	return NULL;
14
15	if (unlikely(ix >= params->num_channels))
16	return NULL;
17
18	return xsk->umems[ix];
19	}
20
21	struct mlx5e_xsk_param;
22	void mlx5e_build_xsk_param(struct xdp_umem umem, struct mlx5e_xsk_param xsk);
23
24	/* .ndo_bpf callback. */
25	int mlx5e_xsk_setup_umem(struct net_device dev, struct xdp_umem umem, u16 qid);
26
27	int mlx5e_xsk_resize_reuseq(struct xdp_umem *umem, u32 nentries);
28
29	#endif /* __MLX5_EN_XSK_UMEM_H__ */

drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c+1−1

@@ -32,7 +32,7 @@ · 31 unmodified lines
32		32
33	#include "en.h"	33	#include "en.h"
34	#include "en/port.h"	34	#include "en/port.h"
35	#include "en/xsk/umem.h"	35	#include "en/xsk/pool.h"
36	#include "lib/clock.h"	36	#include "lib/clock.h"
37		37
38	void mlx5e_ethtool_get_drvinfo(struct mlx5e_priv *priv,	38	void mlx5e_ethtool_get_drvinfo(struct mlx5e_priv *priv,

drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.c+1−1

@@ -33,7 +33,7 @@ · 32 unmodified lines
33	#include <linux/mlx5/fs.h>	33	#include <linux/mlx5/fs.h>
34	#include "en.h"	34	#include "en.h"
35	#include "en/params.h"	35	#include "en/params.h"
36	#include "en/xsk/umem.h"	36	#include "en/xsk/pool.h"
37		37
38	struct mlx5e_ethtool_rule {	38	struct mlx5e_ethtool_rule {
39	struct list_head list;	39	struct list_head list;

drivers/net/ethernet/mellanox/mlx5/core/en_main.c+25−24

 #include "en/monitor_stats.h"
 #include "en/health.h"
 #include "en/params.h"
-#include "en/xsk/umem.h"
+#include "en/xsk/pool.h"
 #include "en/xsk/setup.h"
 #include "en/xsk/rx.h"
 #include "en/xsk/tx.h"
 static int mlx5e_alloc_rq(struct mlx5e_channel *c,
 			  struct mlx5e_params *params,
 			  struct mlx5e_xsk_param *xsk,
-			  struct xdp_umem *umem,
+			  struct xsk_buff_pool *xsk_pool,
 			  struct mlx5e_rq_param *rqp,
 			  struct mlx5e_rq *rq)
 {
 	rq->mdev    = mdev;
 	rq->hw_mtu  = MLX5E_SW2HW_MTU(params, params->sw_mtu);
 	rq->xdpsq   = &c->rq_xdpsq;
-	rq->umem    = umem;
+	rq->xsk_pool = xsk_pool;
-	if (rq->umem)
+	if (rq->xsk_pool)
 		rq->stats = &c->priv->channel_stats[c->ix].xskrq;
 	else
 		rq->stats = &c->priv->channel_stats[c->ix].rq;
 	if (xsk) {
 		err = xdp_rxq_info_reg_mem_model(&rq->xdp_rxq,
 						 MEM_TYPE_XSK_BUFF_POOL, NULL);
-		xsk_buff_set_rxq_info(rq->umem, &rq->xdp_rxq);
+		xsk_pool_set_rxq_info(rq->xsk_pool, &rq->xdp_rxq);
 	} else {
 		/* Create a page_pool and register it with rxq */
 		pp_params.order     = 0;
 int mlx5e_open_rq(struct mlx5e_channel *c, struct mlx5e_params *params,
 		  struct mlx5e_rq_param *param, struct mlx5e_xsk_param *xsk,
-		  struct xdp_umem *umem, struct mlx5e_rq *rq)
+		  struct xsk_buff_pool *xsk_pool, struct mlx5e_rq *rq)
 {
 	int err;
-	err = mlx5e_alloc_rq(c, params, xsk, umem, param, rq);
+	err = mlx5e_alloc_rq(c, params, xsk, xsk_pool, param, rq);
 	if (err)
 		return err;
 static int mlx5e_alloc_xdpsq(struct mlx5e_channel *c,
 			     struct mlx5e_params *params,
-			     struct xdp_umem *umem,
+			     struct xsk_buff_pool *xsk_pool,
 			     struct mlx5e_sq_param *param,
 			     struct mlx5e_xdpsq *sq,
 			     bool is_redirect)
 	sq->uar_map   = mdev->mlx5e_res.bfreg.map;
 	sq->min_inline_mode = params->tx_min_inline_mode;
 	sq->hw_mtu    = MLX5E_SW2HW_MTU(params, params->sw_mtu);
-	sq->umem      = umem;
+	sq->xsk_pool  = xsk_pool;
-	sq->stats = sq->umem ?
+	sq->stats = sq->xsk_pool ?
 		&c->priv->channel_stats[c->ix].xsksq :
 		is_redirect ?
 			&c->priv->channel_stats[c->ix].xdpsq :
 }
 int mlx5e_open_xdpsq(struct mlx5e_channel *c, struct mlx5e_params *params,
-		     struct mlx5e_sq_param *param, struct xdp_umem *umem,
+		     struct mlx5e_sq_param *param, struct xsk_buff_pool *xsk_pool,
 		     struct mlx5e_xdpsq *sq, bool is_redirect)
 {
 	struct mlx5e_create_sq_param csp = {};
 	int err;
-	err = mlx5e_alloc_xdpsq(c, params, umem, param, sq, is_redirect);
+	err = mlx5e_alloc_xdpsq(c, params, xsk_pool, param, sq, is_redirect);
 	if (err)
 		return err;
 static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
 			      struct mlx5e_params *params,
 			      struct mlx5e_channel_param *cparam,
-			      struct xdp_umem *umem,
+			      struct xsk_buff_pool *xsk_pool,
 			      struct mlx5e_channel **cp)
 {
 	int cpu = cpumask_first(mlx5_comp_irq_get_affinity_mask(priv->mdev, ix));
 	if (unlikely(err))
 		goto err_napi_del;
-	if (umem) {
+	if (xsk_pool) {
-		mlx5e_build_xsk_param(umem, &xsk);
+		mlx5e_build_xsk_param(xsk_pool, &xsk);
-		err = mlx5e_open_xsk(priv, params, &xsk, umem, c);
+		err = mlx5e_open_xsk(priv, params, &xsk, xsk_pool, c);
 		if (unlikely(err))
 			goto err_close_queues;
 	}
 	mlx5e_build_channel_param(priv, &chs->params, cparam);
 	for (i = 0; i < chs->num; i++) {
-		struct xdp_umem *umem = NULL;
+		struct xsk_buff_pool *xsk_pool = NULL;
 		if (chs->params.xdp_prog)
-			umem = mlx5e_xsk_get_umem(&chs->params, chs->params.xsk, i);
+			xsk_pool = mlx5e_xsk_get_pool(&chs->params, chs->params.xsk, i);
-		err = mlx5e_open_channel(priv, i, &chs->params, cparam, umem, &chs->c[i]);
+		err = mlx5e_open_channel(priv, i, &chs->params, cparam, xsk_pool, &chs->c[i]);
 		if (err)
 			goto err_close_channels;
 	}
 	u16 ix;
 	for (ix = 0; ix < chs->params.num_channels; ix++) {
-		struct xdp_umem *umem = mlx5e_xsk_get_umem(&chs->params, chs->params.xsk, ix);
+		struct xsk_buff_pool *xsk_pool =
+			mlx5e_xsk_get_pool(&chs->params, chs->params.xsk, ix);
 		struct mlx5e_xsk_param xsk;
-		if (!umem)
+		if (!xsk_pool)
 			continue;
-		mlx5e_build_xsk_param(umem, &xsk);
+		mlx5e_build_xsk_param(xsk_pool, &xsk);
 		if (!mlx5e_validate_xsk_param(new_params, &xsk, mdev)) {
 			u32 hr = mlx5e_get_linear_rq_headroom(new_params, &xsk);
 	switch (xdp->command) {
 	case XDP_SETUP_PROG:
 		return mlx5e_xdp_set(dev, xdp->prog);
-	case XDP_SETUP_XSK_UMEM:
+	case XDP_SETUP_XSK_POOL:
-		return mlx5e_xsk_setup_umem(dev, xdp->xsk.umem,
+		return mlx5e_xsk_setup_pool(dev, xdp->xsk.pool,
 					    xdp->xsk.queue_id);
 	default:
 		return -EINVAL;

drivers/net/ethernet/mellanox/mlx5/core/en_rx.c+8−8

@@ -280,8 +280,8 @@ static inline int mlx5e_page_alloc_pool(struct mlx5e_rq *rq, · 279 unmodified lines
280	static inline int mlx5e_page_alloc(struct mlx5e_rq *rq,	280	static inline int mlx5e_page_alloc(struct mlx5e_rq *rq,
281	struct mlx5e_dma_info *dma_info)	281	struct mlx5e_dma_info *dma_info)
282	{	282	{
283	if (rq->umem)	283	if (rq->xsk_pool)
284	return mlx5e_xsk_page_alloc_umem(rq, dma_info);	284	return mlx5e_xsk_page_alloc_pool(rq, dma_info);
285	else	285	else
286	return mlx5e_page_alloc_pool(rq, dma_info);	286	return mlx5e_page_alloc_pool(rq, dma_info);
287	}	287	}
@@ -312,7 +312,7 @@ static inline void mlx5e_page_release(struct mlx5e_rq *rq, · 24 unmodified lines
312	struct mlx5e_dma_info *dma_info,	312	struct mlx5e_dma_info *dma_info,
313	bool recycle)	313	bool recycle)
314	{	314	{
315	if (rq->umem)	315	if (rq->xsk_pool)
316	/* The `recycle` parameter is ignored, and the page is always	316	/* The `recycle` parameter is ignored, and the page is always
317	* put into the Reuse Ring, because there is no way to return	317	* put into the Reuse Ring, because there is no way to return
318	* the page to the userspace when the interface goes down.	318	* the page to the userspace when the interface goes down.
@@ -399,14 +399,14 @@ static int mlx5e_alloc_rx_wqes(struct mlx5e_rq *rq, u16 ix, u8 wqe_bulk) · 80 unmodified lines
399	int err;	399	int err;
400	int i;	400	int i;
401		401
402	if (rq->umem) {	402	if (rq->xsk_pool) {
403	int pages_desired = wqe_bulk << rq->wqe.info.log_num_frags;	403	int pages_desired = wqe_bulk << rq->wqe.info.log_num_frags;
404		404
405	/* Check in advance that we have enough frames, instead of	405	/* Check in advance that we have enough frames, instead of
406	* allocating one-by-one, failing and moving frames to the	406	* allocating one-by-one, failing and moving frames to the
407	* Reuse Ring.	407	* Reuse Ring.
408	*/	408	*/
409	if (unlikely(!xsk_buff_can_alloc(rq->umem, pages_desired)))	409	if (unlikely(!xsk_buff_can_alloc(rq->xsk_pool, pages_desired)))
410	return -ENOMEM;	410	return -ENOMEM;
411	}	411	}
412		412
@@ -504,8 +504,8 @@ static int mlx5e_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix) · 91 unmodified lines
504	/* Check in advance that we have enough frames, instead of allocating	504	/* Check in advance that we have enough frames, instead of allocating
505	* one-by-one, failing and moving frames to the Reuse Ring.	505	* one-by-one, failing and moving frames to the Reuse Ring.
506	*/	506	*/
507	if (rq->umem &&	507	if (rq->xsk_pool &&
508	unlikely(!xsk_buff_can_alloc(rq->umem, MLX5_MPWRQ_PAGES_PER_WQE))) {	508	unlikely(!xsk_buff_can_alloc(rq->xsk_pool, MLX5_MPWRQ_PAGES_PER_WQE))) {
509	err = -ENOMEM;	509	err = -ENOMEM;
510	goto err;	510	goto err;
511	}	511	}
@@ -753,7 +753,7 @@ INDIRECT_CALLABLE_SCOPE bool mlx5e_post_rx_mpwqes(struct mlx5e_rq *rq) · 241 unmodified lines
753	* the driver when it refills the Fill Ring.	753	* the driver when it refills the Fill Ring.
754	* 2. Otherwise, busy poll by rescheduling the NAPI poll.	754	* 2. Otherwise, busy poll by rescheduling the NAPI poll.
755	*/	755	*/
756	if (unlikely(alloc_err == -ENOMEM && rq->umem))	756	if (unlikely(alloc_err == -ENOMEM && rq->xsk_pool))
757	return true;	757	return true;
758		758
759	return false;	759	return false;

drivers/net/tun.c+0−18

@@ -219,24 +219,6 @@ struct veth { · 218 unmodified lines
219	__be16 h_vlan_TCI;	219	__be16 h_vlan_TCI;
220	};	220	};
221		221
222	bool tun_is_xdp_frame(void *ptr)
223	{
224	return (unsigned long)ptr & TUN_XDP_FLAG;
225	}
226	EXPORT_SYMBOL(tun_is_xdp_frame);
227
228	void tun_xdp_to_ptr(void ptr)
229	{
230	return (void *)((unsigned long)ptr \| TUN_XDP_FLAG);
231	}
232	EXPORT_SYMBOL(tun_xdp_to_ptr);
233
234	void tun_ptr_to_xdp(void ptr)
235	{
236	return (void *)((unsigned long)ptr & ~TUN_XDP_FLAG);
237	}
238	EXPORT_SYMBOL(tun_ptr_to_xdp);
239
240	static int tun_napi_receive(struct napi_struct *napi, int budget)	222	static int tun_napi_receive(struct napi_struct *napi, int budget)
241	{	223	{
242	struct tun_file *tfile = container_of(napi, struct tun_file, napi);	224	struct tun_file *tfile = container_of(napi, struct tun_file, napi);

drivers/net/veth.c+3−3

@@ -234,14 +234,14 @@ static bool veth_is_xdp_frame(void *ptr) · 233 unmodified lines
234	return (unsigned long)ptr & VETH_XDP_FLAG;	234	return (unsigned long)ptr & VETH_XDP_FLAG;
235	}	235	}
236		236
237	static void veth_ptr_to_xdp(void ptr)	237	static struct xdp_frame veth_ptr_to_xdp(void ptr)
238	{	238	{
239	return (void *)((unsigned long)ptr & ~VETH_XDP_FLAG);	239	return (void *)((unsigned long)ptr & ~VETH_XDP_FLAG);
240	}	240	}
241		241
242	static void veth_xdp_to_ptr(void ptr)	242	static void veth_xdp_to_ptr(struct xdp_frame xdp)
243	{	243	{
244	return (void *)((unsigned long)ptr \| VETH_XDP_FLAG);	244	return (void *)((unsigned long)xdp \| VETH_XDP_FLAG);
245	}	245	}
246		246
247	static void veth_ptr_free(void *ptr)	247	static void veth_ptr_free(void *ptr)

include/linux/bpf-cgroup.h+25−0

@@ -279,6 +279,31 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map map, void key, · 278 unmodified lines
279	#define BPF_CGROUP_RUN_PROG_UDP6_RECVMSG_LOCK(sk, uaddr) \	279	#define BPF_CGROUP_RUN_PROG_UDP6_RECVMSG_LOCK(sk, uaddr) \
280	BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, BPF_CGROUP_UDP6_RECVMSG, NULL)	280	BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, BPF_CGROUP_UDP6_RECVMSG, NULL)
281		281
		282	/* The SOCK_OPS"_SK" macro should be used when sock_ops->sk is not a
		283	* fullsock and its parent fullsock cannot be traced by
		284	* sk_to_full_sk().
		285	*
		286	* e.g. sock_ops->sk is a request_sock and it is under syncookie mode.
		287	* Its listener-sk is not attached to the rsk_listener.
		288	* In this case, the caller holds the listener-sk (unlocked),
		289	* set its sock_ops->sk to req_sk, and call this SOCK_OPS"_SK" with
		290	* the listener-sk such that the cgroup-bpf-progs of the
		291	* listener-sk will be run.
		292	*
		293	* Regardless of syncookie mode or not,
		294	* calling bpf_setsockopt on listener-sk will not make sense anyway,
		295	* so passing 'sock_ops->sk == req_sk' to the bpf prog is appropriate here.
		296	*/
		297	#define BPF_CGROUP_RUN_PROG_SOCK_OPS_SK(sock_ops, sk) \
		298	({ \
		299	int __ret = 0; \
		300	if (cgroup_bpf_enabled) \
		301	__ret = __cgroup_bpf_run_filter_sock_ops(sk, \
		302	sock_ops, \
		303	BPF_CGROUP_SOCK_OPS); \
		304	__ret; \
		305	})
		306
282	#define BPF_CGROUP_RUN_PROG_SOCK_OPS(sock_ops) \	307	#define BPF_CGROUP_RUN_PROG_SOCK_OPS(sock_ops) \
283	({ \	308	({ \
284	int __ret = 0; \	309	int __ret = 0; \

include/linux/bpf.h+52−0

 struct exception_table_entry;
 struct seq_operations;
 struct bpf_iter_aux_info;
+struct bpf_local_storage;
+struct bpf_local_storage_map;
 extern struct idr btf_idr;
 extern spinlock_t btf_idr_lock;
 	__poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
 			     struct poll_table_struct *pts);
+	/* Functions called by bpf_local_storage maps */
+	int (*map_local_storage_charge)(struct bpf_local_storage_map *smap,
+					void *owner, u32 size);
+	void (*map_local_storage_uncharge)(struct bpf_local_storage_map *smap,
+					   void *owner, u32 size);
+	struct bpf_local_storage __rcu ** (*map_owner_storage_ptr)(void *owner);
+	/* map_meta_equal must be implemented for maps that can be
+	 * used as an inner map.  It is a runtime check to ensure
+	 * an inner map can be inserted to an outer map.
+	 *
+	 * Some properties of the inner map has been used during the
+	 * verification time.  When inserting an inner map at the runtime,
+	 * map_meta_equal has to ensure the inserting map has the same
+	 * properties that the verifier has used earlier.
+	 */
+	bool (*map_meta_equal)(const struct bpf_map *meta0,
+			       const struct bpf_map *meta1);
 	/* BTF name and id of struct allocated by map_alloc */
 	const char * const map_btf_name;
 	int *map_btf_id;
 		     const struct btf_type *key_type,
 		     const struct btf_type *value_type);
+bool bpf_map_meta_equal(const struct bpf_map *meta0,
+			const struct bpf_map *meta1);
 extern const struct bpf_map_ops bpf_map_offload_ops;
 /* function argument constraints */
 						    * for this argument.
 						    */
 	int *ret_btf_id; /* return value btf_id */
+	bool (*allowed)(const struct bpf_prog *prog);
 };
 /* bpf_context is intentionally undefined structure. Pointer to bpf_context is
 /* these two functions are called from generated trampoline */
 u64 notrace __bpf_prog_enter(void);
 void notrace __bpf_prog_exit(struct bpf_prog *prog, u64 start);
+void notrace __bpf_prog_enter_sleepable(void);
+void notrace __bpf_prog_exit_sleepable(void);
 struct bpf_ksym {
 	unsigned long		 start;
 	bool offload_requested;
 	bool attach_btf_trace; /* true if attaching to BTF-enabled raw tp */
 	bool func_proto_unreliable;
+	bool sleepable;
 	enum bpf_tramp_prog_type trampoline_prog_type;
 	struct bpf_trampoline *trampoline;
 	struct hlist_node tramp_hlist;
 					union bpf_iter_link_info *linfo,
 					struct bpf_iter_aux_info *aux);
 typedef void (*bpf_iter_detach_target_t)(struct bpf_iter_aux_info *aux);
+typedef void (*bpf_iter_show_fdinfo_t) (const struct bpf_iter_aux_info *aux,
+					struct seq_file *seq);
+typedef int (*bpf_iter_fill_link_info_t)(const struct bpf_iter_aux_info *aux,
+					 struct bpf_link_info *info);
 #define BPF_ITER_CTX_ARG_MAX 2
 struct bpf_iter_reg {
 	const char *target;
 	bpf_iter_attach_target_t attach_target;
 	bpf_iter_detach_target_t detach_target;
+	bpf_iter_show_fdinfo_t show_fdinfo;
+	bpf_iter_fill_link_info_t fill_link_info;
 	u32 ctx_arg_info_size;
 	struct bpf_ctx_arg_aux ctx_arg_info[BPF_ITER_CTX_ARG_MAX];
 	const struct bpf_iter_seq_info *seq_info;
 bool bpf_link_is_iter(struct bpf_link *link);
 struct bpf_prog *bpf_iter_get_info(struct bpf_iter_meta *meta, bool in_stop);
 int bpf_iter_run_prog(struct bpf_prog *prog, void *ctx);
+void bpf_iter_map_show_fdinfo(const struct bpf_iter_aux_info *aux,
+			      struct seq_file *seq);
+int bpf_iter_map_fill_link_info(const struct bpf_iter_aux_info *aux,
+				struct bpf_link_info *info);
 int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value);
 int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value);
 		      const struct btf_type *t, int off, int size,
 		      enum bpf_access_type atype,
 		      u32 *next_btf_id);
+bool btf_struct_ids_match(struct bpf_verifier_log *log,
+			  int off, u32 id, u32 need_type_id);
 int btf_resolve_helper_id(struct bpf_verifier_log *log,
 			  const struct bpf_func_proto *fn, int);
 			 struct btf *btf, const struct btf_type *t);
 struct bpf_prog *bpf_prog_by_id(u32 id);
+struct bpf_link *bpf_link_by_id(u32 id);
 const struct bpf_func_proto *bpf_base_func_proto(enum bpf_func_id func_id);
 #else /* !CONFIG_BPF_SYSCALL */
 			 struct bpf_prog *old, u32 which);
 int sock_map_get_from_fd(const union bpf_attr *attr, struct bpf_prog *prog);
 int sock_map_prog_detach(const union bpf_attr *attr, enum bpf_prog_type ptype);
+int sock_map_update_elem_sys(struct bpf_map *map, void *key, void *value, u64 flags);
 void sock_map_unhash(struct sock *sk);
 void sock_map_close(struct sock *sk, long timeout);
 #else
 {
 	return -EOPNOTSUPP;
 }
+static inline int sock_map_update_elem_sys(struct bpf_map *map, void *key, void *value,
+					   u64 flags)
+{
+	return -EOPNOTSUPP;
+}
 #endif /* CONFIG_BPF_STREAM_PARSER */
 #if defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL)
 extern const struct bpf_func_proto bpf_skc_to_tcp_timewait_sock_proto;
 extern const struct bpf_func_proto bpf_skc_to_tcp_request_sock_proto;
 extern const struct bpf_func_proto bpf_skc_to_udp6_sock_proto;
+extern const struct bpf_func_proto bpf_copy_from_user_proto;
 const struct bpf_func_proto *bpf_tracing_func_proto(
 	enum bpf_func_id func_id, const struct bpf_prog *prog);
 int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,
 		       void *addr1, void *addr2);
+struct btf_id_set;
+bool btf_id_set_contains(struct btf_id_set *set, u32 id);
 #endif /* _LINUX_BPF_H */

include/linux/bpf_local_storage.h+163−0

@@ -0,0 +1,163 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	/*
	3	* Copyright (c) 2019 Facebook
	4	* Copyright 2020 Google LLC.
	5	*/
	6
	7	#ifndef _BPF_LOCAL_STORAGE_H
	8	#define _BPF_LOCAL_STORAGE_H
	9
	10	#include <linux/bpf.h>
	11	#include <linux/rculist.h>
	12	#include <linux/list.h>
	13	#include <linux/hash.h>
	14	#include <linux/types.h>
	15	#include <uapi/linux/btf.h>
	16
	17	#define BPF_LOCAL_STORAGE_CACHE_SIZE 16
	18
	19	struct bpf_local_storage_map_bucket {
	20	struct hlist_head list;
	21	raw_spinlock_t lock;
	22	};
	23
	24	/* Thp map is not the primary owner of a bpf_local_storage_elem.
	25	* Instead, the container object (eg. sk->sk_bpf_storage) is.
	26	*
	27	* The map (bpf_local_storage_map) is for two purposes
	28	* 1. Define the size of the "local storage". It is
	29	* the map's value_size.
	30	*
	31	* 2. Maintain a list to keep track of all elems such
	32	* that they can be cleaned up during the map destruction.
	33	*
	34	* When a bpf local storage is being looked up for a
	35	* particular object, the "bpf_map" pointer is actually used
	36	* as the "key" to search in the list of elem in
	37	* the respective bpf_local_storage owned by the object.
	38	*
	39	* e.g. sk->sk_bpf_storage is the mini-map with the "bpf_map" pointer
	40	* as the searching key.
	41	*/
	42	struct bpf_local_storage_map {
	43	struct bpf_map map;
	44	/* Lookup elem does not require accessing the map.
	45	*
	46	* Updating/Deleting requires a bucket lock to
	47	* link/unlink the elem from the map. Having
	48	* multiple buckets to improve contention.
	49	*/
	50	struct bpf_local_storage_map_bucket *buckets;
	51	u32 bucket_log;
	52	u16 elem_size;
	53	u16 cache_idx;
	54	};
	55
	56	struct bpf_local_storage_data {
	57	/* smap is used as the searching key when looking up
	58	* from the object's bpf_local_storage.
	59	*
	60	* Put it in the same cacheline as the data to minimize
	61	* the number of cachelines access during the cache hit case.
	62	*/
	63	struct bpf_local_storage_map __rcu *smap;
	64	u8 data[] __aligned(8);
	65	};
	66
	67	/* Linked to bpf_local_storage and bpf_local_storage_map */
	68	struct bpf_local_storage_elem {
	69	struct hlist_node map_node; /* Linked to bpf_local_storage_map */
	70	struct hlist_node snode; /* Linked to bpf_local_storage */
	71	struct bpf_local_storage __rcu *local_storage;
	72	struct rcu_head rcu;
	73	/* 8 bytes hole */
	74	/* The data is stored in aother cacheline to minimize
	75	* the number of cachelines access during a cache hit.
	76	*/
	77	struct bpf_local_storage_data sdata ____cacheline_aligned;
	78	};
	79
	80	struct bpf_local_storage {
	81	struct bpf_local_storage_data __rcu *cache[BPF_LOCAL_STORAGE_CACHE_SIZE];
	82	struct hlist_head list; /* List of bpf_local_storage_elem */
	83	void owner; / The object that owns the above "list" of
	84	* bpf_local_storage_elem.
	85	*/
	86	struct rcu_head rcu;
	87	raw_spinlock_t lock; /* Protect adding/removing from the "list" */
	88	};
	89
	90	/* U16_MAX is much more than enough for sk local storage
	91	* considering a tcp_sock is ~2k.
	92	*/
	93	#define BPF_LOCAL_STORAGE_MAX_VALUE_SIZE \
	94	min_t(u32, \
	95	(KMALLOC_MAX_SIZE - MAX_BPF_STACK - \
	96	sizeof(struct bpf_local_storage_elem)), \
	97	(U16_MAX - sizeof(struct bpf_local_storage_elem)))
	98
	99	#define SELEM(_SDATA) \
	100	container_of((_SDATA), struct bpf_local_storage_elem, sdata)
	101	#define SDATA(_SELEM) (&(_SELEM)->sdata)
	102
	103	#define BPF_LOCAL_STORAGE_CACHE_SIZE 16
	104
	105	struct bpf_local_storage_cache {
	106	spinlock_t idx_lock;
	107	u64 idx_usage_counts[BPF_LOCAL_STORAGE_CACHE_SIZE];
	108	};
	109
	110	#define DEFINE_BPF_STORAGE_CACHE(name) \
	111	static struct bpf_local_storage_cache name = { \
	112	.idx_lock = __SPIN_LOCK_UNLOCKED(name.idx_lock), \
	113	}
	114
	115	u16 bpf_local_storage_cache_idx_get(struct bpf_local_storage_cache *cache);
	116	void bpf_local_storage_cache_idx_free(struct bpf_local_storage_cache *cache,
	117	u16 idx);
	118
	119	/* Helper functions for bpf_local_storage */
	120	int bpf_local_storage_map_alloc_check(union bpf_attr *attr);
	121
	122	struct bpf_local_storage_map bpf_local_storage_map_alloc(union bpf_attr attr);
	123
	124	struct bpf_local_storage_data *
	125	bpf_local_storage_lookup(struct bpf_local_storage *local_storage,
	126	struct bpf_local_storage_map *smap,
	127	bool cacheit_lockit);
	128
	129	void bpf_local_storage_map_free(struct bpf_local_storage_map *smap);
	130
	131	int bpf_local_storage_map_check_btf(const struct bpf_map *map,
	132	const struct btf *btf,
	133	const struct btf_type *key_type,
	134	const struct btf_type *value_type);
	135
	136	void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,
	137	struct bpf_local_storage_elem *selem);
	138
	139	bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage,
	140	struct bpf_local_storage_elem *selem,
	141	bool uncharge_omem);
	142
	143	void bpf_selem_unlink(struct bpf_local_storage_elem *selem);
	144
	145	void bpf_selem_link_map(struct bpf_local_storage_map *smap,
	146	struct bpf_local_storage_elem *selem);
	147
	148	void bpf_selem_unlink_map(struct bpf_local_storage_elem *selem);
	149
	150	struct bpf_local_storage_elem *
	151	bpf_selem_alloc(struct bpf_local_storage_map smap, void owner, void *value,
	152	bool charge_mem);
	153
	154	int
	155	bpf_local_storage_alloc(void *owner,
	156	struct bpf_local_storage_map *smap,
	157	struct bpf_local_storage_elem *first_selem);
	158
	159	struct bpf_local_storage_data *
	160	bpf_local_storage_update(void owner, struct bpf_local_storage_map smap,
	161	void *value, u64 map_flags);
	162
	163	#endif /* _BPF_LOCAL_STORAGE_H */

include/linux/bpf_lsm.h+29−0

@@ -17,9 +17,28 @@ · 16 unmodified lines
17	#include <linux/lsm_hook_defs.h>	17	#include <linux/lsm_hook_defs.h>
18	#undef LSM_HOOK	18	#undef LSM_HOOK
19		19
		20	struct bpf_storage_blob {
		21	struct bpf_local_storage __rcu *storage;
		22	};
		23
		24	extern struct lsm_blob_sizes bpf_lsm_blob_sizes;
		25
20	int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,	26	int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,
21	const struct bpf_prog *prog);	27	const struct bpf_prog *prog);
22		28
		29	static inline struct bpf_storage_blob *bpf_inode(
		30	const struct inode *inode)
		31	{
		32	if (unlikely(!inode->i_security))
		33	return NULL;
		34
		35	return inode->i_security + bpf_lsm_blob_sizes.lbs_inode;
		36	}
		37
		38	extern const struct bpf_func_proto bpf_inode_storage_get_proto;
		39	extern const struct bpf_func_proto bpf_inode_storage_delete_proto;
		40	void bpf_inode_storage_free(struct inode *inode);
		41
23	#else /* !CONFIG_BPF_LSM */	42	#else /* !CONFIG_BPF_LSM */
24		43
25	static inline int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,	44	static inline int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,
@@ -28,6 +47,16 @@ static inline int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog, · 2 unmodified lines
28	return -EOPNOTSUPP;	47	return -EOPNOTSUPP;
29	}	48	}
30		49
		50	static inline struct bpf_storage_blob *bpf_inode(
		51	const struct inode *inode)
		52	{
		53	return NULL;
		54	}
		55
		56	static inline void bpf_inode_storage_free(struct inode *inode)
		57	{
		58	}
		59
31	#endif /* CONFIG_BPF_LSM */	60	#endif /* CONFIG_BPF_LSM */
32		61
33	#endif /* _LINUX_BPF_LSM_H */	62	#endif /* _LINUX_BPF_LSM_H */

include/linux/bpf_types.h+3−0

 BPF_MAP_TYPE(BPF_MAP_TYPE_SOCKMAP, sock_map_ops)
 BPF_MAP_TYPE(BPF_MAP_TYPE_SOCKHASH, sock_hash_ops)
 #endif
+#ifdef CONFIG_BPF_LSM
+BPF_MAP_TYPE(BPF_MAP_TYPE_INODE_STORAGE, inode_storage_map_ops)
+#endif
 BPF_MAP_TYPE(BPF_MAP_TYPE_CPUMAP, cpu_map_ops)
 #if defined(CONFIG_XDP_SOCKETS)
 BPF_MAP_TYPE(BPF_MAP_TYPE_XSKMAP, xsk_map_ops)

include/linux/btf.h+1−2

@@ -64,8 +64,7 @@ const struct btf_type btf_type_resolve_func_ptr(const struct btf btf, · 63 unmodified lines
64	u32 id, u32 *res_id);	64	u32 id, u32 *res_id);
65	const struct btf_type *	65	const struct btf_type *
66	btf_resolve_size(const struct btf btf, const struct btf_type type,	66	btf_resolve_size(const struct btf btf, const struct btf_type type,
67	u32 type_size, const struct btf_type *elem_type,	67	u32 *type_size);
68	u32 *total_nelems);
69		68
70	#define for_each_member(i, struct_type, member) \	69	#define for_each_member(i, struct_type, member) \
71	for (i = 0, member = btf_type_member(struct_type); \	70	for (i = 0, member = btf_type_member(struct_type); \

include/linux/btf_ids.h+50−1

@@ -3,6 +3,11 @@ · 2 unmodified lines
3	#ifndef _LINUX_BTF_IDS_H	3	#ifndef _LINUX_BTF_IDS_H
4	#define _LINUX_BTF_IDS_H	4	#define _LINUX_BTF_IDS_H
5		5
		6	struct btf_id_set {
		7	u32 cnt;
		8	u32 ids[];
		9	};
		10
6	#ifdef CONFIG_DEBUG_INFO_BTF	11	#ifdef CONFIG_DEBUG_INFO_BTF
7		12
8	#include <linux/compiler.h> /* for __PASTE */	13	#include <linux/compiler.h> /* for __PASTE */
@@ -62,7 +67,7 @@ asm( \ · 53 unmodified lines
62	".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \	67	".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
63	"." #scope " " #name "; \n" \	68	"." #scope " " #name "; \n" \
64	#name ":; \n" \	69	#name ":; \n" \
65	".popsection; \n"); \	70	".popsection; \n");
66		71
67	#define BTF_ID_LIST(name) \	72	#define BTF_ID_LIST(name) \
68	__BTF_ID_LIST(name, local) \	73	__BTF_ID_LIST(name, local) \
@@ -88,12 +93,56 @@ asm( \ · 19 unmodified lines
88	".zero 4 \n" \	93	".zero 4 \n" \
89	".popsection; \n");	94	".popsection; \n");
90		95
		96	/*
		97	* The BTF_SET_START/END macros pair defines sorted list of
		98	* BTF IDs plus its members count, with following layout:
		99	*
		100	* BTF_SET_START(list)
		101	* BTF_ID(type1, name1)
		102	* BTF_ID(type2, name2)
		103	* BTF_SET_END(list)
		104	*
		105	* __BTF_ID__set__list:
		106	* .zero 4
		107	* list:
		108	* __BTF_ID__type1__name1__3:
		109	* .zero 4
		110	* __BTF_ID__type2__name2__4:
		111	* .zero 4
		112	*
		113	*/
		114	#define __BTF_SET_START(name, scope) \
		115	asm( \
		116	".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
		117	"." #scope " __BTF_ID__set__" #name "; \n" \
		118	"__BTF_ID__set__" #name ":; \n" \
		119	".zero 4 \n" \
		120	".popsection; \n");
		121
		122	#define BTF_SET_START(name) \
		123	__BTF_ID_LIST(name, local) \
		124	__BTF_SET_START(name, local)
		125
		126	#define BTF_SET_START_GLOBAL(name) \
		127	__BTF_ID_LIST(name, globl) \
		128	__BTF_SET_START(name, globl)
		129
		130	#define BTF_SET_END(name) \
		131	asm( \
		132	".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
		133	".size __BTF_ID__set__" #name ", .-" #name " \n" \
		134	".popsection; \n"); \
		135	extern struct btf_id_set name;
		136
91	#else	137	#else
92		138
93	#define BTF_ID_LIST(name) static u32 name[5];	139	#define BTF_ID_LIST(name) static u32 name[5];
94	#define BTF_ID(prefix, name)	140	#define BTF_ID(prefix, name)
95	#define BTF_ID_UNUSED	141	#define BTF_ID_UNUSED
96	#define BTF_ID_LIST_GLOBAL(name) u32 name[1];	142	#define BTF_ID_LIST_GLOBAL(name) u32 name[1];
		143	#define BTF_SET_START(name) static struct btf_id_set name = { 0 };
		144	#define BTF_SET_START_GLOBAL(name) static struct btf_id_set name = { 0 };
		145	#define BTF_SET_END(name)
97		146
98	#endif /* CONFIG_DEBUG_INFO_BTF */	147	#endif /* CONFIG_DEBUG_INFO_BTF */
99		148

include/linux/filter.h+6−2

 struct bpf_sock_ops_kern {
 	struct	sock *sk;
-	u32	op;
 	union {
 		u32 args[4];
 		u32 reply;
 		u32 replylong[4];
 	};
-	u32	is_fullsock;
+	struct sk_buff	*syn_skb;
+	struct sk_buff	*skb;
+	void	*skb_data_end;
+	u8	op;
+	u8	is_fullsock;
+	u8	remaining_opt_len;
 	u64	temp;			/* temp and everything after is not
 					 * initialized to 0 before calling
 					 * the BPF program. New fields that

include/linux/if_tun.h+14−5

 #if defined(CONFIG_TUN) || defined(CONFIG_TUN_MODULE)
 struct socket *tun_get_socket(struct file *);
 struct ptr_ring *tun_get_tx_ring(struct file *file);
-bool tun_is_xdp_frame(void *ptr);
+static inline bool tun_is_xdp_frame(void *ptr)
-void *tun_xdp_to_ptr(void *ptr);
+{
-void *tun_ptr_to_xdp(void *ptr);
+       return (unsigned long)ptr & TUN_XDP_FLAG;
+}
+static inline void *tun_xdp_to_ptr(struct xdp_frame *xdp)
+{
+       return (void *)((unsigned long)xdp | TUN_XDP_FLAG);
+}
+static inline struct xdp_frame *tun_ptr_to_xdp(void *ptr)
+{
+       return (void *)((unsigned long)ptr & ~TUN_XDP_FLAG);
+}
 void tun_ptr_free(void *ptr);
 #else
 #include <linux/err.h>
 {
 	return false;
 }
-static inline void *tun_xdp_to_ptr(void *ptr)
+static inline void *tun_xdp_to_ptr(struct xdp_frame *xdp)
 {
 	return NULL;
 }
-static inline void *tun_ptr_to_xdp(void *ptr)
+static inline struct xdp_frame *tun_ptr_to_xdp(void *ptr)
 {
 	return NULL;
 }

include/linux/netdevice.h+5−5

@@ -618,7 +618,7 @@ struct netdev_queue { · 617 unmodified lines
618	/* Subordinate device that the queue has been assigned to */	618	/* Subordinate device that the queue has been assigned to */
619	struct net_device *sb_dev;	619	struct net_device *sb_dev;
620	#ifdef CONFIG_XDP_SOCKETS	620	#ifdef CONFIG_XDP_SOCKETS
621	struct xdp_umem *umem;	621	struct xsk_buff_pool *pool;
622	#endif	622	#endif
623	/*	623	/*
624	* write-mostly part	624	* write-mostly part
@@ -755,7 +755,7 @@ struct netdev_rx_queue { · 130 unmodified lines
755	struct net_device *dev;	755	struct net_device *dev;
756	struct xdp_rxq_info xdp_rxq;	756	struct xdp_rxq_info xdp_rxq;
757	#ifdef CONFIG_XDP_SOCKETS	757	#ifdef CONFIG_XDP_SOCKETS
758	struct xdp_umem *umem;	758	struct xsk_buff_pool *pool;
759	#endif	759	#endif
760	} ____cacheline_aligned_in_smp;	760	} ____cacheline_aligned_in_smp;
761		761
@@ -883,7 +883,7 @@ enum bpf_netdev_command { · 121 unmodified lines
883	/* BPF program for offload callbacks, invoked at program load time. */	883	/* BPF program for offload callbacks, invoked at program load time. */
884	BPF_OFFLOAD_MAP_ALLOC,	884	BPF_OFFLOAD_MAP_ALLOC,
885	BPF_OFFLOAD_MAP_FREE,	885	BPF_OFFLOAD_MAP_FREE,
886	XDP_SETUP_XSK_UMEM,	886	XDP_SETUP_XSK_POOL,
887	};	887	};
888		888
889	struct bpf_prog_offload_ops;	889	struct bpf_prog_offload_ops;
@@ -917,9 +917,9 @@ struct netdev_bpf { · 27 unmodified lines
917	struct {	917	struct {
918	struct bpf_offloaded_map *offmap;	918	struct bpf_offloaded_map *offmap;
919	};	919	};
920	/* XDP_SETUP_XSK_UMEM */	920	/* XDP_SETUP_XSK_POOL */
921	struct {	921	struct {
922	struct xdp_umem *umem;	922	struct xsk_buff_pool *pool;
923	u16 queue_id;	923	u16 queue_id;
924	} xsk;	924	} xsk;
925	};	925	};

include/linux/rcupdate_trace.h+8−1

@@ -82,7 +82,14 @@ static inline void rcu_read_unlock_trace(void) · 81 unmodified lines
82	void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func);	82	void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func);
83	void synchronize_rcu_tasks_trace(void);	83	void synchronize_rcu_tasks_trace(void);
84	void rcu_barrier_tasks_trace(void);	84	void rcu_barrier_tasks_trace(void);
85		85	#else
		86	/*
		87	* The BPF JIT forms these addresses even when it doesn't call these
		88	* functions, so provide definitions that result in runtime errors.
		89	*/
		90	static inline void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func) { BUG(); }
		91	static inline void rcu_read_lock_trace(void) { BUG(); }
		92	static inline void rcu_read_unlock_trace(void) { BUG(); }
86	#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */	93	#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */
87		94
88	#endif /* __LINUX_RCUPDATE_TRACE_H */	95	#endif /* __LINUX_RCUPDATE_TRACE_H */

include/linux/skmsg.h+0−17

@@ -340,23 +340,6 @@ static inline void sk_psock_update_proto(struct sock *sk, · 339 unmodified lines
340	struct sk_psock *psock,	340	struct sk_psock *psock,
341	struct proto *ops)	341	struct proto *ops)
342	{	342	{
343	/* Initialize saved callbacks and original proto only once, since this
344	* function may be called multiple times for a psock, e.g. when
345	* psock->progs.msg_parser is updated.
346	*
347	* Since we've not installed the new proto, psock is not yet in use and
348	* we can initialize it without synchronization.
349	*/
350	if (!psock->sk_proto) {
351	struct proto *orig = READ_ONCE(sk->sk_prot);
352
353	psock->saved_unhash = orig->unhash;
354	psock->saved_close = orig->close;
355	psock->saved_write_space = sk->sk_write_space;
356
357	psock->sk_proto = orig;
358	}
359
360	/* Pairs with lockless read in sk_clone_lock() */	343	/* Pairs with lockless read in sk_clone_lock() */
361	WRITE_ONCE(sk->sk_prot, ops);	344	WRITE_ONCE(sk->sk_prot, ops);
362	}	345	}

include/linux/tcp.h+15−5

 		smc_ok : 1,	/* SMC seen on SYN packet		*/
 		snd_wscale : 4,	/* Window scaling received from sender	*/
 		rcv_wscale : 4;	/* Window scaling to send to receiver	*/
+	u8	saw_unknown:1,	/* Received unknown option		*/
+		unused:7;
 	u8	num_sacks;	/* Number of SACK blocks		*/
 	u16	user_mss;	/* mss requested by user in ioctl	*/
 	u16	mss_clamp;	/* Maximal mss, negotiated at connection setup */
 		repair      : 1,
 		frto        : 1;/* F-RTO (RFC5682) activated in CA_Loss */
 	u8	repair_queue;
-	u8	syn_data:1,	/* SYN includes data */
+	u8	save_syn:2,	/* Save headers of SYN packet */
+		syn_data:1,	/* SYN includes data */
 		syn_fastopen:1,	/* SYN includes Fast Open option */
 		syn_fastopen_exp:1,/* SYN includes Fast Open exp. option */
 		syn_fastopen_ch:1, /* Active TFO re-enabling probe */
 		syn_data_acked:1,/* data in SYN is acked by SYN-ACK */
-		save_syn:1,	/* Save headers of SYN packet */
+		is_cwnd_limited:1;/* forward progress limited by snd_cwnd? */
-		is_cwnd_limited:1,/* forward progress limited by snd_cwnd? */
-		syn_smc:1;	/* SYN includes SMC */
 	u32	tlp_high_seq;	/* snd_nxt at the time of TLP */
 	u32	tcp_tx_delay;	/* delay (in usec) added to TX packets */
 #if IS_ENABLED(CONFIG_MPTCP)
 	bool	is_mptcp;
 #endif
+#if IS_ENABLED(CONFIG_SMC)
+	bool	syn_smc;	/* SYN includes SMC */
+#endif
 #ifdef CONFIG_TCP_MD5SIG
 /* TCP AF-Specific parts; only used by MD5 Signature support so far */
 	 * socket. Used to retransmit SYNACKs etc.
 	 */
 	struct request_sock __rcu *fastopen_rsk;
-	u32	*saved_syn;
+	struct saved_syn *saved_syn;
 };
 enum tsq_enum {
 	tp->saved_syn = NULL;
 }
+static inline u32 tcp_saved_syn_len(const struct saved_syn *saved_syn)
+{
+	return saved_syn->mac_hdrlen + saved_syn->network_hdrlen +
+		saved_syn->tcp_hdrlen;
+}
 struct sk_buff *tcp_get_timestamping_opt_stats(const struct sock *sk,
 					       const struct sk_buff *orig_skb);

include/net/bpf_sk_storage.h+14−0

@@ -3,13 +3,27 @@ · 2 unmodified lines
3	#ifndef _BPF_SK_STORAGE_H	3	#ifndef _BPF_SK_STORAGE_H
4	#define _BPF_SK_STORAGE_H	4	#define _BPF_SK_STORAGE_H
5		5
		6	#include <linux/rculist.h>
		7	#include <linux/list.h>
		8	#include <linux/hash.h>
		9	#include <linux/types.h>
		10	#include <linux/spinlock.h>
		11	#include <linux/bpf.h>
		12	#include <net/sock.h>
		13	#include <uapi/linux/sock_diag.h>
		14	#include <uapi/linux/btf.h>
		15	#include <linux/bpf_local_storage.h>
		16
6	struct sock;	17	struct sock;
7		18
8	void bpf_sk_storage_free(struct sock *sk);	19	void bpf_sk_storage_free(struct sock *sk);
9		20
10	extern const struct bpf_func_proto bpf_sk_storage_get_proto;	21	extern const struct bpf_func_proto bpf_sk_storage_get_proto;
11	extern const struct bpf_func_proto bpf_sk_storage_delete_proto;	22	extern const struct bpf_func_proto bpf_sk_storage_delete_proto;
		23	extern const struct bpf_func_proto sk_storage_get_btf_proto;
		24	extern const struct bpf_func_proto sk_storage_delete_btf_proto;
12		25
		26	struct bpf_local_storage_elem;
13	struct bpf_sk_storage_diag;	27	struct bpf_sk_storage_diag;
14	struct sk_buff;	28	struct sk_buff;
15	struct nlattr;	29	struct nlattr;

include/net/inet_connection_sock.h+2−0

  	struct timer_list	  icsk_retransmit_timer;
  	struct timer_list	  icsk_delack_timer;
 	__u32			  icsk_rto;
+	__u32                     icsk_rto_min;
+	__u32                     icsk_delack_max;
 	__u32			  icsk_pmtu_cookie;
 	const struct tcp_congestion_ops *icsk_ca_ops;
 	const struct inet_connection_sock_af_ops *icsk_af_ops;

include/net/request_sock.h+8−1

 int inet_rtx_syn_ack(const struct sock *parent, struct request_sock *req);
+struct saved_syn {
+	u32 mac_hdrlen;
+	u32 network_hdrlen;
+	u32 tcp_hdrlen;
+	u8 data[];
+};
 /* struct request_sock - mini sock to represent a connection request
  */
 struct request_sock {
 	struct timer_list		rsk_timer;
 	const struct request_sock_ops	*rsk_ops;
 	struct sock			*sk;
-	u32				*saved_syn;
+	struct saved_syn		*saved_syn;
 	u32				secid;
 	u32				peer_secid;
 };

include/net/sock.h+2−2

@@ -246,7 +246,7 @@ struct sock_common { · 245 unmodified lines
246	/* public: */	246	/* public: */
247	};	247	};
248		248
249	struct bpf_sk_storage;	249	struct bpf_local_storage;
250		250
251	/**	251	/**
252	* struct sock - network layer representation of sockets	252	* struct sock - network layer representation of sockets
@@ -517,7 +517,7 @@ struct sock { · 264 unmodified lines
517	void (sk_destruct)(struct sock sk);	517	void (sk_destruct)(struct sock sk);
518	struct sock_reuseport __rcu *sk_reuseport_cb;	518	struct sock_reuseport __rcu *sk_reuseport_cb;
519	#ifdef CONFIG_BPF_SYSCALL	519	#ifdef CONFIG_BPF_SYSCALL
520	struct bpf_sk_storage __rcu *sk_bpf_storage;	520	struct bpf_local_storage __rcu *sk_bpf_storage;
521	#endif	521	#endif
522	struct rcu_head sk_rcu;	522	struct rcu_head sk_rcu;
523	};	523	};

include/net/tcp.h+55−4

@@ -394,7 +394,7 @@ void tcp_metrics_init(void); · 393 unmodified lines
394	bool tcp_peer_is_proven(struct request_sock req, struct dst_entry dst);	394	bool tcp_peer_is_proven(struct request_sock req, struct dst_entry dst);
395	void tcp_close(struct sock *sk, long timeout);	395	void tcp_close(struct sock *sk, long timeout);
396	void tcp_init_sock(struct sock *sk);	396	void tcp_init_sock(struct sock *sk);
397	void tcp_init_transfer(struct sock *sk, int bpf_op);	397	void tcp_init_transfer(struct sock sk, int bpf_op, struct sk_buff skb);
398	__poll_t tcp_poll(struct file file, struct socket sock,	398	__poll_t tcp_poll(struct file file, struct socket sock,
399	struct poll_table_struct *wait);	399	struct poll_table_struct *wait);
400	int tcp_getsockopt(struct sock *sk, int level, int optname,	400	int tcp_getsockopt(struct sock *sk, int level, int optname,
@@ -455,7 +455,8 @@ enum tcp_synack_type { · 54 unmodified lines
455	struct sk_buff tcp_make_synack(const struct sock sk, struct dst_entry *dst,	455	struct sk_buff tcp_make_synack(const struct sock sk, struct dst_entry *dst,
456	struct request_sock *req,	456	struct request_sock *req,
457	struct tcp_fastopen_cookie *foc,	457	struct tcp_fastopen_cookie *foc,
458	enum tcp_synack_type synack_type);	458	enum tcp_synack_type synack_type,
		459	struct sk_buff *syn_skb);
459	int tcp_disconnect(struct sock *sk, int flags);	460	int tcp_disconnect(struct sock *sk, int flags);
460		461
461	void tcp_finish_connect(struct sock sk, struct sk_buff skb);	462	void tcp_finish_connect(struct sock sk, struct sk_buff skb);
@@ -699,7 +700,7 @@ static inline void tcp_fast_path_check(struct sock *sk) · 237 unmodified lines
699	static inline u32 tcp_rto_min(struct sock *sk)	700	static inline u32 tcp_rto_min(struct sock *sk)
700	{	701	{
701	const struct dst_entry *dst = __sk_dst_get(sk);	702	const struct dst_entry *dst = __sk_dst_get(sk);
702	u32 rto_min = TCP_RTO_MIN;	703	u32 rto_min = inet_csk(sk)->icsk_rto_min;
703		704
704	if (dst && dst_metric_locked(dst, RTAX_RTO_MIN))	705	if (dst && dst_metric_locked(dst, RTAX_RTO_MIN))
705	rto_min = dst_metric_rtt(dst, RTAX_RTO_MIN);	706	rto_min = dst_metric_rtt(dst, RTAX_RTO_MIN);
@@ -2025,7 +2026,8 @@ struct tcp_request_sock_ops { · 1319 unmodified lines
2025	int (send_synack)(const struct sock sk, struct dst_entry *dst,	2026	int (send_synack)(const struct sock sk, struct dst_entry *dst,
2026	struct flowi fl, struct request_sock req,	2027	struct flowi fl, struct request_sock req,
2027	struct tcp_fastopen_cookie *foc,	2028	struct tcp_fastopen_cookie *foc,
2028	enum tcp_synack_type synack_type);	2029	enum tcp_synack_type synack_type,
		2030	struct sk_buff *syn_skb);
2029	};	2031	};
2030		2032
2031	extern const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops;	2033	extern const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops;
@@ -2223,6 +2225,55 @@ int __tcp_bpf_recvmsg(struct sock sk, struct sk_psock psock, · 191 unmodified lines
2223	struct msghdr *msg, int len, int flags);	2225	struct msghdr *msg, int len, int flags);
2224	#endif /* CONFIG_NET_SOCK_MSG */	2226	#endif /* CONFIG_NET_SOCK_MSG */
2225		2227
		2228	#ifdef CONFIG_CGROUP_BPF
		2229	/* Copy the listen sk's HDR_OPT_CB flags to its child.
		2230	*
		2231	* During 3-Way-HandShake, the synack is usually sent from
		2232	* the listen sk with the HDR_OPT_CB flags set so that
		2233	* bpf-prog will be called to write the BPF hdr option.
		2234	*
		2235	* In fastopen, the child sk is used to send synack instead
		2236	* of the listen sk. Thus, inheriting the HDR_OPT_CB flags
		2237	* from the listen sk gives the bpf-prog a chance to write
		2238	* BPF hdr option in the synack pkt during fastopen.
		2239	*
		2240	* Both fastopen and non-fastopen child will inherit the
		2241	* HDR_OPT_CB flags to keep the bpf-prog having a consistent
		2242	* behavior when deciding to clear this cb flags (or not)
		2243	* during the PASSIVE_ESTABLISHED_CB.
		2244	*
		2245	* In the future, other cb flags could be inherited here also.
		2246	*/
		2247	static inline void bpf_skops_init_child(const struct sock *sk,
		2248	struct sock *child)
		2249	{
		2250	tcp_sk(child)->bpf_sock_ops_cb_flags =
		2251	tcp_sk(sk)->bpf_sock_ops_cb_flags &
		2252	(BPF_SOCK_OPS_PARSE_ALL_HDR_OPT_CB_FLAG \|
		2253	BPF_SOCK_OPS_PARSE_UNKNOWN_HDR_OPT_CB_FLAG \|
		2254	BPF_SOCK_OPS_WRITE_HDR_OPT_CB_FLAG);
		2255	}
		2256
		2257	static inline void bpf_skops_init_skb(struct bpf_sock_ops_kern *skops,
		2258	struct sk_buff *skb,
		2259	unsigned int end_offset)
		2260	{
		2261	skops->skb = skb;
		2262	skops->skb_data_end = skb->data + end_offset;
		2263	}
		2264	#else
		2265	static inline void bpf_skops_init_child(const struct sock *sk,
		2266	struct sock *child)
		2267	{
		2268	}
		2269
		2270	static inline void bpf_skops_init_skb(struct bpf_sock_ops_kern *skops,
		2271	struct sk_buff *skb,
		2272	unsigned int end_offset)
		2273	{
		2274	}
		2275	#endif
		2276
2226	/* Call BPF_SOCK_OPS program that returns an int. If the return value	2277	/* Call BPF_SOCK_OPS program that returns an int. If the return value
2227	* is < 0, then the BPF op failed (for example if the loaded BPF	2278	* is < 0, then the BPF op failed (for example if the loaded BPF
2228	* program does not support the chosen operation or there is no BPF	2279	* program does not support the chosen operation or there is no BPF

include/net/xdp_sock.h+14−16

@@ -18,25 +18,19 @@ struct xsk_queue; · 17 unmodified lines
18	struct xdp_buff;	18	struct xdp_buff;
19		19
20	struct xdp_umem {	20	struct xdp_umem {
21	struct xsk_queue *fq;	21	void *addrs;
22	struct xsk_queue *cq;
23	struct xsk_buff_pool *pool;
24	u64 size;	22	u64 size;
25	u32 headroom;	23	u32 headroom;
26	u32 chunk_size;	24	u32 chunk_size;
		25	u32 chunks;
		26	u32 npgs;
27	struct user_struct *user;	27	struct user_struct *user;
28	refcount_t users;	28	refcount_t users;
29	struct work_struct work;
30	struct page **pgs;
31	u32 npgs;
32	u16 queue_id;
33	u8 need_wakeup;
34	u8 flags;	29	u8 flags;
35	int id;
36	struct net_device *dev;
37	bool zc;	30	bool zc;
38	spinlock_t xsk_tx_list_lock;	31	struct page **pgs;
39	struct list_head xsk_tx_list;	32	int id;
		33	struct list_head xsk_dma_list;
40	};	34	};
41		35
42	struct xsk_map {	36	struct xsk_map {
@@ -48,10 +42,11 @@ struct xsk_map { · 5 unmodified lines
48	struct xdp_sock {	42	struct xdp_sock {
49	/* struct sock must be the first member of struct xdp_sock */	43	/* struct sock must be the first member of struct xdp_sock */
50	struct sock sk;	44	struct sock sk;
51	struct xsk_queue *rx;	45	struct xsk_queue *rx ____cacheline_aligned_in_smp;
52	struct net_device *dev;	46	struct net_device *dev;
53	struct xdp_umem *umem;	47	struct xdp_umem *umem;
54	struct list_head flush_node;	48	struct list_head flush_node;
		49	struct xsk_buff_pool *pool;
55	u16 queue_id;	50	u16 queue_id;
56	bool zc;	51	bool zc;
57	enum {	52	enum {
@@ -59,10 +54,9 @@ struct xdp_sock { · 1 unmodified line
59	XSK_BOUND,	54	XSK_BOUND,
60	XSK_UNBOUND,	55	XSK_UNBOUND,
61	} state;	56	} state;
62	/* Protects multiple processes in the control path */	57
63	struct mutex mutex;
64	struct xsk_queue *tx ____cacheline_aligned_in_smp;	58	struct xsk_queue *tx ____cacheline_aligned_in_smp;
65	struct list_head list;	59	struct list_head tx_list;
66	/* Mutual exclusion of NAPI TX thread and sendmsg error paths	60	/* Mutual exclusion of NAPI TX thread and sendmsg error paths
67	* in the SKB destructor callback.	61	* in the SKB destructor callback.
68	*/	62	*/
@@ -77,6 +71,10 @@ struct xdp_sock { · 8 unmodified lines
77	struct list_head map_list;	71	struct list_head map_list;
78	/* Protects map_list */	72	/* Protects map_list */
79	spinlock_t map_list_lock;	73	spinlock_t map_list_lock;
		74	/* Protects multiple processes in the control path */
		75	struct mutex mutex;
		76	struct xsk_queue fq_tmp; / Only as tmp storage before bind */
		77	struct xsk_queue cq_tmp; / Only as tmp storage before bind */
80	};	78	};
81		79
82	#ifdef CONFIG_XDP_SOCKETS	80	#ifdef CONFIG_XDP_SOCKETS

include/net/xdp_sock_drv.h+65−57

 #ifdef CONFIG_XDP_SOCKETS
-void xsk_umem_complete_tx(struct xdp_umem *umem, u32 nb_entries);
+void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries);
-bool xsk_umem_consume_tx(struct xdp_umem *umem, struct xdp_desc *desc);
+bool xsk_tx_peek_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc);
-void xsk_umem_consume_tx_done(struct xdp_umem *umem);
+void xsk_tx_release(struct xsk_buff_pool *pool);
-struct xdp_umem *xdp_get_umem_from_qid(struct net_device *dev, u16 queue_id);
+struct xsk_buff_pool *xsk_get_pool_from_qid(struct net_device *dev,
-void xsk_set_rx_need_wakeup(struct xdp_umem *umem);
+					    u16 queue_id);
-void xsk_set_tx_need_wakeup(struct xdp_umem *umem);
+void xsk_set_rx_need_wakeup(struct xsk_buff_pool *pool);
-void xsk_clear_rx_need_wakeup(struct xdp_umem *umem);
+void xsk_set_tx_need_wakeup(struct xsk_buff_pool *pool);
-void xsk_clear_tx_need_wakeup(struct xdp_umem *umem);
+void xsk_clear_rx_need_wakeup(struct xsk_buff_pool *pool);
-bool xsk_umem_uses_need_wakeup(struct xdp_umem *umem);
+void xsk_clear_tx_need_wakeup(struct xsk_buff_pool *pool);
+bool xsk_uses_need_wakeup(struct xsk_buff_pool *pool);
-static inline u32 xsk_umem_get_headroom(struct xdp_umem *umem)
+static inline u32 xsk_pool_get_headroom(struct xsk_buff_pool *pool)
 {
-	return XDP_PACKET_HEADROOM + umem->headroom;
+	return XDP_PACKET_HEADROOM + pool->headroom;
 }
-static inline u32 xsk_umem_get_chunk_size(struct xdp_umem *umem)
+static inline u32 xsk_pool_get_chunk_size(struct xsk_buff_pool *pool)
 {
-	return umem->chunk_size;
+	return pool->chunk_size;
 }
-static inline u32 xsk_umem_get_rx_frame_size(struct xdp_umem *umem)
+static inline u32 xsk_pool_get_rx_frame_size(struct xsk_buff_pool *pool)
 {
-	return xsk_umem_get_chunk_size(umem) - xsk_umem_get_headroom(umem);
+	return xsk_pool_get_chunk_size(pool) - xsk_pool_get_headroom(pool);
 }
-static inline void xsk_buff_set_rxq_info(struct xdp_umem *umem,
+static inline void xsk_pool_set_rxq_info(struct xsk_buff_pool *pool,
 					 struct xdp_rxq_info *rxq)
 {
-	xp_set_rxq_info(umem->pool, rxq);
+	xp_set_rxq_info(pool, rxq);
 }
-static inline void xsk_buff_dma_unmap(struct xdp_umem *umem,
+static inline void xsk_pool_dma_unmap(struct xsk_buff_pool *pool,
 				      unsigned long attrs)
 {
-	xp_dma_unmap(umem->pool, attrs);
+	xp_dma_unmap(pool, attrs);
 }
-static inline int xsk_buff_dma_map(struct xdp_umem *umem, struct device *dev,
+static inline int xsk_pool_dma_map(struct xsk_buff_pool *pool,
-				   unsigned long attrs)
+				   struct device *dev, unsigned long attrs)
 {
-	return xp_dma_map(umem->pool, dev, attrs, umem->pgs, umem->npgs);
+	struct xdp_umem *umem = pool->umem;
+	return xp_dma_map(pool, dev, attrs, umem->pgs, umem->npgs);
 }
 static inline dma_addr_t xsk_buff_xdp_get_dma(struct xdp_buff *xdp)
 	return xp_get_frame_dma(xskb);
 }
-static inline struct xdp_buff *xsk_buff_alloc(struct xdp_umem *umem)
+static inline struct xdp_buff *xsk_buff_alloc(struct xsk_buff_pool *pool)
 {
-	return xp_alloc(umem->pool);
+	return xp_alloc(pool);
 }
-static inline bool xsk_buff_can_alloc(struct xdp_umem *umem, u32 count)
+static inline bool xsk_buff_can_alloc(struct xsk_buff_pool *pool, u32 count)
 {
-	return xp_can_alloc(umem->pool, count);
+	return xp_can_alloc(pool, count);
 }
 static inline void xsk_buff_free(struct xdp_buff *xdp)
 	xp_free(xskb);
 }
-static inline dma_addr_t xsk_buff_raw_get_dma(struct xdp_umem *umem, u64 addr)
+static inline dma_addr_t xsk_buff_raw_get_dma(struct xsk_buff_pool *pool,
+					      u64 addr)
 {
-	return xp_raw_get_dma(umem->pool, addr);
+	return xp_raw_get_dma(pool, addr);
 }
-static inline void *xsk_buff_raw_get_data(struct xdp_umem *umem, u64 addr)
+static inline void *xsk_buff_raw_get_data(struct xsk_buff_pool *pool, u64 addr)
 {
-	return xp_raw_get_data(umem->pool, addr);
+	return xp_raw_get_data(pool, addr);
 }
-static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp)
+static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp, struct xsk_buff_pool *pool)
 {
 	struct xdp_buff_xsk *xskb = container_of(xdp, struct xdp_buff_xsk, xdp);
+	if (!pool->dma_need_sync)
+		return;
 	xp_dma_sync_for_cpu(xskb);
 }
-static inline void xsk_buff_raw_dma_sync_for_device(struct xdp_umem *umem,
+static inline void xsk_buff_raw_dma_sync_for_device(struct xsk_buff_pool *pool,
 						    dma_addr_t dma,
 						    size_t size)
 {
-	xp_dma_sync_for_device(umem->pool, dma, size);
+	xp_dma_sync_for_device(pool, dma, size);
 }
 #else
-static inline void xsk_umem_complete_tx(struct xdp_umem *umem, u32 nb_entries)
+static inline void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries)
 {
 }
-static inline bool xsk_umem_consume_tx(struct xdp_umem *umem,
+static inline bool xsk_tx_peek_desc(struct xsk_buff_pool *pool,
-				       struct xdp_desc *desc)
+				    struct xdp_desc *desc)
 {
 	return false;
 }
-static inline void xsk_umem_consume_tx_done(struct xdp_umem *umem)
+static inline void xsk_tx_release(struct xsk_buff_pool *pool)
 {
 }
-static inline struct xdp_umem *xdp_get_umem_from_qid(struct net_device *dev,
+static inline struct xsk_buff_pool *
-						     u16 queue_id)
+xsk_get_pool_from_qid(struct net_device *dev, u16 queue_id)
 {
 	return NULL;
 }
-static inline void xsk_set_rx_need_wakeup(struct xdp_umem *umem)
+static inline void xsk_set_rx_need_wakeup(struct xsk_buff_pool *pool)
 {
 }
-static inline void xsk_set_tx_need_wakeup(struct xdp_umem *umem)
+static inline void xsk_set_tx_need_wakeup(struct xsk_buff_pool *pool)
 {
 }
-static inline void xsk_clear_rx_need_wakeup(struct xdp_umem *umem)
+static inline void xsk_clear_rx_need_wakeup(struct xsk_buff_pool *pool)
 {
 }
-static inline void xsk_clear_tx_need_wakeup(struct xdp_umem *umem)
+static inline void xsk_clear_tx_need_wakeup(struct xsk_buff_pool *pool)
 {
 }
-static inline bool xsk_umem_uses_need_wakeup(struct xdp_umem *umem)
+static inline bool xsk_uses_need_wakeup(struct xsk_buff_pool *pool)
 {
 	return false;
 }
-static inline u32 xsk_umem_get_headroom(struct xdp_umem *umem)
+static inline u32 xsk_pool_get_headroom(struct xsk_buff_pool *pool)
 {
 	return 0;
 }
-static inline u32 xsk_umem_get_chunk_size(struct xdp_umem *umem)
+static inline u32 xsk_pool_get_chunk_size(struct xsk_buff_pool *pool)
 {
 	return 0;
 }
-static inline u32 xsk_umem_get_rx_frame_size(struct xdp_umem *umem)
+static inline u32 xsk_pool_get_rx_frame_size(struct xsk_buff_pool *pool)
 {
 	return 0;
 }
-static inline void xsk_buff_set_rxq_info(struct xdp_umem *umem,
+static inline void xsk_pool_set_rxq_info(struct xsk_buff_pool *pool,
 					 struct xdp_rxq_info *rxq)
 {
 }
-static inline void xsk_buff_dma_unmap(struct xdp_umem *umem,
+static inline void xsk_pool_dma_unmap(struct xsk_buff_pool *pool,
 				      unsigned long attrs)
 {
 }
-static inline int xsk_buff_dma_map(struct xdp_umem *umem, struct device *dev,
+static inline int xsk_pool_dma_map(struct xsk_buff_pool *pool,
-				   unsigned long attrs)
+				   struct device *dev, unsigned long attrs)
 {
 	return 0;
 }
 	return 0;
 }
-static inline struct xdp_buff *xsk_buff_alloc(struct xdp_umem *umem)
+static inline struct xdp_buff *xsk_buff_alloc(struct xsk_buff_pool *pool)
 {
 	return NULL;
 }
-static inline bool xsk_buff_can_alloc(struct xdp_umem *umem, u32 count)
+static inline bool xsk_buff_can_alloc(struct xsk_buff_pool *pool, u32 count)
 {
 	return false;
 }
 {
 }
-static inline dma_addr_t xsk_buff_raw_get_dma(struct xdp_umem *umem, u64 addr)
+static inline dma_addr_t xsk_buff_raw_get_dma(struct xsk_buff_pool *pool,
+					      u64 addr)
 {
 	return 0;
 }
-static inline void *xsk_buff_raw_get_data(struct xdp_umem *umem, u64 addr)
+static inline void *xsk_buff_raw_get_data(struct xsk_buff_pool *pool, u64 addr)
 {
 	return NULL;
 }
-static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp)
+static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp, struct xsk_buff_pool *pool)
 {
 }
-static inline void xsk_buff_raw_dma_sync_for_device(struct xdp_umem *umem,
+static inline void xsk_buff_raw_dma_sync_for_device(struct xsk_buff_pool *pool,
 						    dma_addr_t dma,
 						    size_t size)
 {

include/net/xsk_buff_pool.h+43−10

 struct xdp_rxq_info;
 struct xsk_queue;
 struct xdp_desc;
+struct xdp_umem;
+struct xdp_sock;
 struct device;
 struct page;
 	struct list_head free_list_node;
 };
+struct xsk_dma_map {
+	dma_addr_t *dma_pages;
+	struct device *dev;
+	struct net_device *netdev;
+	refcount_t users;
+	struct list_head list; /* Protected by the RTNL_LOCK */
+	u32 dma_pages_cnt;
+	bool dma_need_sync;
+};
 struct xsk_buff_pool {
-	struct xsk_queue *fq;
+	/* Members only used in the control path first. */
+	struct device *dev;
+	struct net_device *netdev;
+	struct list_head xsk_tx_list;
+	/* Protects modifications to the xsk_tx_list */
+	spinlock_t xsk_tx_list_lock;
+	refcount_t users;
+	struct xdp_umem *umem;
+	struct work_struct work;
 	struct list_head free_list;
+	u32 heads_cnt;
+	u16 queue_id;
+	/* Data path members as close to free_heads at the end as possible. */
+	struct xsk_queue *fq ____cacheline_aligned_in_smp;
+	struct xsk_queue *cq;
+	/* For performance reasons, each buff pool has its own array of dma_pages
+	 * even when they are identical.
+	 */
 	dma_addr_t *dma_pages;
 	struct xdp_buff_xsk *heads;
 	u64 chunk_mask;
 	u64 addrs_cnt;
 	u32 free_list_cnt;
 	u32 dma_pages_cnt;
-	u32 heads_cnt;
 	u32 free_heads_cnt;
 	u32 headroom;
 	u32 chunk_size;
 	u32 frame_len;
+	u8 cached_need_wakeup;
+	bool uses_need_wakeup;
 	bool dma_need_sync;
 	bool unaligned;
 	void *addrs;
-	struct device *dev;
 	struct xdp_buff_xsk *free_heads[];
 };
 /* AF_XDP core. */
-struct xsk_buff_pool *xp_create(struct page **pages, u32 nr_pages, u32 chunks,
+struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,
-				u32 chunk_size, u32 headroom, u64 size,
+						struct xdp_umem *umem);
-				bool unaligned);
+int xp_assign_dev(struct xsk_buff_pool *pool, struct net_device *dev,
-void xp_set_fq(struct xsk_buff_pool *pool, struct xsk_queue *fq);
+		  u16 queue_id, u16 flags);
+int xp_assign_dev_shared(struct xsk_buff_pool *pool, struct xdp_umem *umem,
+			 struct net_device *dev, u16 queue_id);
 void xp_destroy(struct xsk_buff_pool *pool);
 void xp_release(struct xdp_buff_xsk *xskb);
+void xp_get_pool(struct xsk_buff_pool *pool);
+void xp_put_pool(struct xsk_buff_pool *pool);
+void xp_clear_dev(struct xsk_buff_pool *pool);
+void xp_add_xsk(struct xsk_buff_pool *pool, struct xdp_sock *xs);
+void xp_del_xsk(struct xsk_buff_pool *pool, struct xdp_sock *xs);
 /* AF_XDP, and XDP core. */
 void xp_free(struct xdp_buff_xsk *xskb);
 void xp_dma_sync_for_cpu_slow(struct xdp_buff_xsk *xskb);
 static inline void xp_dma_sync_for_cpu(struct xdp_buff_xsk *xskb)
 {
-	if (!xskb->pool->dma_need_sync)
-		return;
 	xp_dma_sync_for_cpu_slow(xskb);
 }

include/uapi/linux/bpf.h+393−5

@@ -155,6 +155,7 @@ enum bpf_map_type { · 154 unmodified lines
155	BPF_MAP_TYPE_DEVMAP_HASH,	155	BPF_MAP_TYPE_DEVMAP_HASH,
156	BPF_MAP_TYPE_STRUCT_OPS,	156	BPF_MAP_TYPE_STRUCT_OPS,
157	BPF_MAP_TYPE_RINGBUF,	157	BPF_MAP_TYPE_RINGBUF,
		158	BPF_MAP_TYPE_INODE_STORAGE,
158	};	159	};
159		160
160	/* Note that tracing related programs such as	161	/* Note that tracing related programs such as
@@ -345,6 +346,14 @@ enum bpf_link_type { · 184 unmodified lines
345	/* The verifier internal test flag. Behavior is undefined */	346	/* The verifier internal test flag. Behavior is undefined */
346	#define BPF_F_TEST_STATE_FREQ (1U << 3)	347	#define BPF_F_TEST_STATE_FREQ (1U << 3)
347		348
		349	/* If BPF_F_SLEEPABLE is used in BPF_PROG_LOAD command, the verifier will
		350	* restrict map and helper usage for such programs. Sleepable BPF programs can
		351	* only be attached to hooks where kernel execution context allows sleeping.
		352	* Such programs are allowed to use helpers that may sleep like
		353	* bpf_copy_from_user().
		354	*/
		355	#define BPF_F_SLEEPABLE (1U << 4)
		356
348	/* When BPF ldimm64's insn[0].src_reg != 0 then this can have	357	/* When BPF ldimm64's insn[0].src_reg != 0 then this can have
349	* two extensions:	358	* two extensions:
350	*	359	*
@@ -2807,7 +2816,7 @@ union bpf_attr { · 2456 unmodified lines
2807	*	2816	*
2808	* -ERANGE if resulting value was out of range.	2817	* -ERANGE if resulting value was out of range.
2809	*	2818	*
2810	* void bpf_sk_storage_get(struct bpf_map map, struct bpf_sock sk, void value, u64 flags)	2819	* void bpf_sk_storage_get(struct bpf_map map, void sk, void value, u64 flags)
2811	* Description	2820	* Description
2812	* Get a bpf-local-storage from a sk.	2821	* Get a bpf-local-storage from a sk.
2813	*	2822	*
@@ -2823,6 +2832,9 @@ union bpf_attr { · 9 unmodified lines
2823	* "type". The bpf-local-storage "type" (i.e. the map) is	2832	* "type". The bpf-local-storage "type" (i.e. the map) is
2824	* searched against all bpf-local-storages residing at sk.	2833	* searched against all bpf-local-storages residing at sk.
2825	*	2834	*
		2835	* sk is a kernel struct sock pointer for LSM program.
		2836	* sk is a struct bpf_sock pointer for other program types.
		2837	*
2826	* An optional flags (BPF_SK_STORAGE_GET_F_CREATE) can be	2838	* An optional flags (BPF_SK_STORAGE_GET_F_CREATE) can be
2827	* used such that a new bpf-local-storage will be	2839	* used such that a new bpf-local-storage will be
2828	* created if one does not exist. value can be used	2840	* created if one does not exist. value can be used
@@ -2835,7 +2847,7 @@ union bpf_attr { · 6 unmodified lines
2835	* NULL if not found or there was an error in adding	2847	* NULL if not found or there was an error in adding
2836	* a new bpf-local-storage.	2848	* a new bpf-local-storage.
2837	*	2849	*
2838	* long bpf_sk_storage_delete(struct bpf_map map, struct bpf_sock sk)	2850	* long bpf_sk_storage_delete(struct bpf_map map, void sk)
2839	* Description	2851	* Description
2840	* Delete a bpf-local-storage from a sk.	2852	* Delete a bpf-local-storage from a sk.
2841	* Return	2853	* Return
@@ -3395,6 +3407,175 @@ union bpf_attr { · 553 unmodified lines
3395	* A non-negative value equal to or less than size on success,	3407	* A non-negative value equal to or less than size on success,
3396	* or a negative error in case of failure.	3408	* or a negative error in case of failure.
3397	*	3409	*
		3410	* long bpf_load_hdr_opt(struct bpf_sock_ops skops, void searchby_res, u32 len, u64 flags)
		3411	* Description
		3412	* Load header option. Support reading a particular TCP header
		3413	* option for bpf program (BPF_PROG_TYPE_SOCK_OPS).
		3414	*
		3415	* If flags is 0, it will search the option from the
		3416	* sock_ops->skb_data. The comment in "struct bpf_sock_ops"
		3417	* has details on what skb_data contains under different
		3418	* sock_ops->op.
		3419	*
		3420	* The first byte of the searchby_res specifies the
		3421	* kind that it wants to search.
		3422	*
		3423	* If the searching kind is an experimental kind
		3424	* (i.e. 253 or 254 according to RFC6994). It also
		3425	* needs to specify the "magic" which is either
		3426	* 2 bytes or 4 bytes. It then also needs to
		3427	* specify the size of the magic by using
		3428	* the 2nd byte which is "kind-length" of a TCP
		3429	* header option and the "kind-length" also
		3430	* includes the first 2 bytes "kind" and "kind-length"
		3431	* itself as a normal TCP header option also does.
		3432	*
		3433	* For example, to search experimental kind 254 with
		3434	* 2 byte magic 0xeB9F, the searchby_res should be
		3435	* [ 254, 4, 0xeB, 0x9F, 0, 0, .... 0 ].
		3436	*
		3437	* To search for the standard window scale option (3),
		3438	* the searchby_res should be [ 3, 0, 0, .... 0 ].
		3439	* Note, kind-length must be 0 for regular option.
		3440	*
		3441	* Searching for No-Op (0) and End-of-Option-List (1) are
		3442	* not supported.
		3443	*
		3444	* len must be at least 2 bytes which is the minimal size
		3445	* of a header option.
		3446	*
		3447	* Supported flags:
		3448	* * BPF_LOAD_HDR_OPT_TCP_SYN to search from the
		3449	* saved_syn packet or the just-received syn packet.
		3450	*
		3451	* Return
		3452	* >0 when found, the header option is copied to searchby_res.
		3453	* The return value is the total length copied.
		3454	*
		3455	* -EINVAL If param is invalid
		3456	*
		3457	* -ENOMSG The option is not found
		3458	*
		3459	* -ENOENT No syn packet available when
		3460	* BPF_LOAD_HDR_OPT_TCP_SYN is used
		3461	*
		3462	* -ENOSPC Not enough space. Only len number of
		3463	* bytes are copied.
		3464	*
		3465	* -EFAULT Cannot parse the header options in the packet
		3466	*
		3467	* -EPERM This helper cannot be used under the
		3468	* current sock_ops->op.
		3469	*
		3470	* long bpf_store_hdr_opt(struct bpf_sock_ops skops, const void from, u32 len, u64 flags)
		3471	* Description
		3472	* Store header option. The data will be copied
		3473	* from buffer from with length len to the TCP header.
		3474	*
		3475	* The buffer from should have the whole option that
		3476	* includes the kind, kind-length, and the actual
		3477	* option data. The len must be at least kind-length
		3478	* long. The kind-length does not have to be 4 byte
		3479	* aligned. The kernel will take care of the padding
		3480	* and setting the 4 bytes aligned value to th->doff.
		3481	*
		3482	* This helper will check for duplicated option
		3483	* by searching the same option in the outgoing skb.
		3484	*
		3485	* This helper can only be called during
		3486	* BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
		3487	*
		3488	* Return
		3489	* 0 on success, or negative error in case of failure:
		3490	*
		3491	* -EINVAL If param is invalid
		3492	*
		3493	* -ENOSPC Not enough space in the header.
		3494	* Nothing has been written
		3495	*
		3496	* -EEXIST The option has already existed
		3497	*
		3498	* -EFAULT Cannot parse the existing header options
		3499	*
		3500	* -EPERM This helper cannot be used under the
		3501	* current sock_ops->op.
		3502	*
		3503	* long bpf_reserve_hdr_opt(struct bpf_sock_ops *skops, u32 len, u64 flags)
		3504	* Description
		3505	* Reserve len bytes for the bpf header option. The
		3506	* space will be used by bpf_store_hdr_opt() later in
		3507	* BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
		3508	*
		3509	* If bpf_reserve_hdr_opt() is called multiple times,
		3510	* the total number of bytes will be reserved.
		3511	*
		3512	* This helper can only be called during
		3513	* BPF_SOCK_OPS_HDR_OPT_LEN_CB.
		3514	*
		3515	* Return
		3516	* 0 on success, or negative error in case of failure:
		3517	*
		3518	* -EINVAL if param is invalid
		3519	*
		3520	* -ENOSPC Not enough space in the header.
		3521	*
		3522	* -EPERM This helper cannot be used under the
		3523	* current sock_ops->op.
		3524	*
		3525	* void bpf_inode_storage_get(struct bpf_map map, void inode, void value, u64 flags)
		3526	* Description
		3527	* Get a bpf_local_storage from an inode.
		3528	*
		3529	* Logically, it could be thought of as getting the value from
		3530	* a map with inode as the key. From this
		3531	* perspective, the usage is not much different from
		3532	* bpf_map_lookup_elem\ (map, &\ inode) except this
		3533	* helper enforces the key must be an inode and the map must also
		3534	* be a BPF_MAP_TYPE_INODE_STORAGE.
		3535	*
		3536	* Underneath, the value is stored locally at inode instead of
		3537	* the map. The map is used as the bpf-local-storage
		3538	* "type". The bpf-local-storage "type" (i.e. the map) is
		3539	* searched against all bpf_local_storage residing at inode.
		3540	*
		3541	* An optional flags (BPF_LOCAL_STORAGE_GET_F_CREATE) can be
		3542	* used such that a new bpf_local_storage will be
		3543	* created if one does not exist. value can be used
		3544	* together with BPF_LOCAL_STORAGE_GET_F_CREATE to specify
		3545	* the initial value of a bpf_local_storage. If value is
		3546	* NULL, the new bpf_local_storage will be zero initialized.
		3547	* Return
		3548	* A bpf_local_storage pointer is returned on success.
		3549	*
		3550	* NULL if not found or there was an error in adding
		3551	* a new bpf_local_storage.
		3552	*
		3553	* int bpf_inode_storage_delete(struct bpf_map map, void inode)
		3554	* Description
		3555	* Delete a bpf_local_storage from an inode.
		3556	* Return
		3557	* 0 on success.
		3558	*
		3559	* -ENOENT if the bpf_local_storage cannot be found.
		3560	*
		3561	* long bpf_d_path(struct path path, char buf, u32 sz)
		3562	* Description
		3563	* Return full path for given 'struct path' object, which
		3564	* needs to be the kernel BTF 'path' object. The path is
		3565	* returned in the provided buffer 'buf' of size 'sz' and
		3566	* is zero terminated.
		3567	*
		3568	* Return
		3569	* On success, the strictly positive length of the string,
		3570	* including the trailing NUL character. On error, a negative
		3571	* value.
		3572	*
		3573	* long bpf_copy_from_user(void dst, u32 size, const void user_ptr)
		3574	* Description
		3575	* Read size bytes from user space address user_ptr and store
		3576	* the data in dst. This is a wrapper of copy_from_user().
		3577	* Return
		3578	* 0 on success, or a negative error in case of failure.
3398	*/	3579	*/
3399	#define __BPF_FUNC_MAPPER(FN) \	3580	#define __BPF_FUNC_MAPPER(FN) \
3400	FN(unspec), \	3581	FN(unspec), \
@@ -3539,6 +3720,13 @@ union bpf_attr { · 138 unmodified lines
3539	FN(skc_to_tcp_request_sock), \	3720	FN(skc_to_tcp_request_sock), \
3540	FN(skc_to_udp6_sock), \	3721	FN(skc_to_udp6_sock), \
3541	FN(get_task_stack), \	3722	FN(get_task_stack), \
		3723	FN(load_hdr_opt), \
		3724	FN(store_hdr_opt), \
		3725	FN(reserve_hdr_opt), \
		3726	FN(inode_storage_get), \
		3727	FN(inode_storage_delete), \
		3728	FN(d_path), \
		3729	FN(copy_from_user), \
3542	/* */	3730	/* */
3543		3731
3544	/* integer value in 'imm' field of BPF_CALL instruction selects which helper	3732	/* integer value in 'imm' field of BPF_CALL instruction selects which helper
@@ -3648,9 +3836,13 @@ enum { · 103 unmodified lines
3648	BPF_F_SYSCTL_BASE_NAME = (1ULL << 0),	3836	BPF_F_SYSCTL_BASE_NAME = (1ULL << 0),
3649	};	3837	};
3650		3838
3651	/* BPF_FUNC_sk_storage_get flags */	3839	/* BPF_FUNC_<kernel_obj>_storage_get flags */
3652	enum {	3840	enum {
3653	BPF_SK_STORAGE_GET_F_CREATE = (1ULL << 0),	3841	BPF_LOCAL_STORAGE_GET_F_CREATE = (1ULL << 0),
		3842	/* BPF_SK_STORAGE_GET_F_CREATE is only kept for backward compatibility
		3843	* and BPF_LOCAL_STORAGE_GET_F_CREATE must be used instead.
		3844	*/
		3845	BPF_SK_STORAGE_GET_F_CREATE = BPF_LOCAL_STORAGE_GET_F_CREATE,
3654	};	3846	};
3655		3847
3656	/* BPF_FUNC_read_branch_records flags. */	3848	/* BPF_FUNC_read_branch_records flags. */
@@ -4071,6 +4263,15 @@ struct bpf_link_info { · 414 unmodified lines
4071	__u64 cgroup_id;	4263	__u64 cgroup_id;
4072	__u32 attach_type;	4264	__u32 attach_type;
4073	} cgroup;	4265	} cgroup;
		4266	struct {
		4267	__aligned_u64 target_name; /* in/out: target_name buffer ptr */
		4268	__u32 target_name_len; /* in/out: target_name buffer len */
		4269	union {
		4270	struct {
		4271	__u32 map_id;
		4272	} map;
		4273	};
		4274	} iter;
4074	struct {	4275	struct {
4075	__u32 netns_ino;	4276	__u32 netns_ino;
4076	__u32 attach_type;	4277	__u32 attach_type;
@@ -4158,6 +4359,36 @@ struct bpf_sock_ops { · 81 unmodified lines
4158	__u64 bytes_received;	4359	__u64 bytes_received;
4159	__u64 bytes_acked;	4360	__u64 bytes_acked;
4160	__bpf_md_ptr(struct bpf_sock *, sk);	4361	__bpf_md_ptr(struct bpf_sock *, sk);
		4362	/* [skb_data, skb_data_end) covers the whole TCP header.
		4363	*
		4364	* BPF_SOCK_OPS_PARSE_HDR_OPT_CB: The packet received
		4365	* BPF_SOCK_OPS_HDR_OPT_LEN_CB: Not useful because the
		4366	* header has not been written.
		4367	* BPF_SOCK_OPS_WRITE_HDR_OPT_CB: The header and options have
		4368	* been written so far.
		4369	* BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB: The SYNACK that concludes
		4370	* the 3WHS.
		4371	* BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB: The ACK that concludes
		4372	* the 3WHS.
		4373	*
		4374	* bpf_load_hdr_opt() can also be used to read a particular option.
		4375	*/
		4376	__bpf_md_ptr(void *, skb_data);
		4377	__bpf_md_ptr(void *, skb_data_end);
		4378	__u32 skb_len; /* The total length of a packet.
		4379	* It includes the header, options,
		4380	* and payload.
		4381	*/
		4382	__u32 skb_tcp_flags; /* tcp_flags of the header. It provides
		4383	* an easy way to check for tcp_flags
		4384	* without parsing skb_data.
		4385	*
		4386	* In particular, the skb_tcp_flags
		4387	* will still be available in
		4388	* BPF_SOCK_OPS_HDR_OPT_LEN even though
		4389	* the outgoing header has not
		4390	* been written yet.
		4391	*/
4161	};	4392	};
4162		4393
4163	/* Definitions for bpf_sock_ops_cb_flags */	4394	/* Definitions for bpf_sock_ops_cb_flags */
@@ -4166,8 +4397,51 @@ enum { · 2 unmodified lines
4166	BPF_SOCK_OPS_RETRANS_CB_FLAG = (1<<1),	4397	BPF_SOCK_OPS_RETRANS_CB_FLAG = (1<<1),
4167	BPF_SOCK_OPS_STATE_CB_FLAG = (1<<2),	4398	BPF_SOCK_OPS_STATE_CB_FLAG = (1<<2),
4168	BPF_SOCK_OPS_RTT_CB_FLAG = (1<<3),	4399	BPF_SOCK_OPS_RTT_CB_FLAG = (1<<3),
		4400	/* Call bpf for all received TCP headers. The bpf prog will be
		4401	* called under sock_ops->op == BPF_SOCK_OPS_PARSE_HDR_OPT_CB
		4402	*
		4403	* Please refer to the comment in BPF_SOCK_OPS_PARSE_HDR_OPT_CB
		4404	* for the header option related helpers that will be useful
		4405	* to the bpf programs.
		4406	*
		4407	* It could be used at the client/active side (i.e. connect() side)
		4408	* when the server told it that the server was in syncookie
		4409	* mode and required the active side to resend the bpf-written
		4410	* options. The active side can keep writing the bpf-options until
		4411	* it received a valid packet from the server side to confirm
		4412	* the earlier packet (and options) has been received. The later
		4413	* example patch is using it like this at the active side when the
		4414	* server is in syncookie mode.
		4415	*
		4416	* The bpf prog will usually turn this off in the common cases.
		4417	*/
		4418	BPF_SOCK_OPS_PARSE_ALL_HDR_OPT_CB_FLAG = (1<<4),
		4419	/* Call bpf when kernel has received a header option that
		4420	* the kernel cannot handle. The bpf prog will be called under
		4421	* sock_ops->op == BPF_SOCK_OPS_PARSE_HDR_OPT_CB.
		4422	*
		4423	* Please refer to the comment in BPF_SOCK_OPS_PARSE_HDR_OPT_CB
		4424	* for the header option related helpers that will be useful
		4425	* to the bpf programs.
		4426	*/
		4427	BPF_SOCK_OPS_PARSE_UNKNOWN_HDR_OPT_CB_FLAG = (1<<5),
		4428	/* Call bpf when the kernel is writing header options for the
		4429	* outgoing packet. The bpf prog will first be called
		4430	* to reserve space in a skb under
		4431	* sock_ops->op == BPF_SOCK_OPS_HDR_OPT_LEN_CB. Then
		4432	* the bpf prog will be called to write the header option(s)
		4433	* under sock_ops->op == BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
		4434	*
		4435	* Please refer to the comment in BPF_SOCK_OPS_HDR_OPT_LEN_CB
		4436	* and BPF_SOCK_OPS_WRITE_HDR_OPT_CB for the header option
		4437	* related helpers that will be useful to the bpf programs.
		4438	*
		4439	* The kernel gets its chance to reserve space and write
		4440	* options first before the BPF program does.
		4441	*/
		4442	BPF_SOCK_OPS_WRITE_HDR_OPT_CB_FLAG = (1<<6),
4169	/* Mask of all currently supported cb flags */	4443	/* Mask of all currently supported cb flags */
4170	BPF_SOCK_OPS_ALL_CB_FLAGS = 0xF,	4444	BPF_SOCK_OPS_ALL_CB_FLAGS = 0x7F,
4171	};	4445	};
4172		4446
4173	/* List of known BPF sock_ops operators.	4447	/* List of known BPF sock_ops operators.
@@ -4223,6 +4497,63 @@ enum { · 49 unmodified lines
4223	*/	4497	*/
4224	BPF_SOCK_OPS_RTT_CB, /* Called on every RTT.	4498	BPF_SOCK_OPS_RTT_CB, /* Called on every RTT.
4225	*/	4499	*/
		4500	BPF_SOCK_OPS_PARSE_HDR_OPT_CB, /* Parse the header option.
		4501	* It will be called to handle
		4502	* the packets received at
		4503	* an already established
		4504	* connection.
		4505	*
		4506	* sock_ops->skb_data:
		4507	* Referring to the received skb.
		4508	* It covers the TCP header only.
		4509	*
		4510	* bpf_load_hdr_opt() can also
		4511	* be used to search for a
		4512	* particular option.
		4513	*/
		4514	BPF_SOCK_OPS_HDR_OPT_LEN_CB, /* Reserve space for writing the
		4515	* header option later in
		4516	* BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
		4517	* Arg1: bool want_cookie. (in
		4518	* writing SYNACK only)
		4519	*
		4520	* sock_ops->skb_data:
		4521	* Not available because no header has
		4522	* been written yet.
		4523	*
		4524	* sock_ops->skb_tcp_flags:
		4525	* The tcp_flags of the
		4526	* outgoing skb. (e.g. SYN, ACK, FIN).
		4527	*
		4528	* bpf_reserve_hdr_opt() should
		4529	* be used to reserve space.
		4530	*/
		4531	BPF_SOCK_OPS_WRITE_HDR_OPT_CB, /* Write the header options
		4532	* Arg1: bool want_cookie. (in
		4533	* writing SYNACK only)
		4534	*
		4535	* sock_ops->skb_data:
		4536	* Referring to the outgoing skb.
		4537	* It covers the TCP header
		4538	* that has already been written
		4539	* by the kernel and the
		4540	* earlier bpf-progs.
		4541	*
		4542	* sock_ops->skb_tcp_flags:
		4543	* The tcp_flags of the outgoing
		4544	* skb. (e.g. SYN, ACK, FIN).
		4545	*
		4546	* bpf_store_hdr_opt() should
		4547	* be used to write the
		4548	* option.
		4549	*
		4550	* bpf_load_hdr_opt() can also
		4551	* be used to search for a
		4552	* particular option that
		4553	* has already been written
		4554	* by the kernel or the
		4555	* earlier bpf-progs.
		4556	*/
4226	};	4557	};
4227		4558
4228	/* List of TCP states. There is a build check in net/ipv4/tcp.c to detect	4559	/* List of TCP states. There is a build check in net/ipv4/tcp.c to detect
@@ -4250,6 +4581,63 @@ enum { · 21 unmodified lines
4250	enum {	4581	enum {
4251	TCP_BPF_IW = 1001, /* Set TCP initial congestion window */	4582	TCP_BPF_IW = 1001, /* Set TCP initial congestion window */
4252	TCP_BPF_SNDCWND_CLAMP = 1002, /* Set sndcwnd_clamp */	4583	TCP_BPF_SNDCWND_CLAMP = 1002, /* Set sndcwnd_clamp */
		4584	TCP_BPF_DELACK_MAX = 1003, /* Max delay ack in usecs */
		4585	TCP_BPF_RTO_MIN = 1004, /* Min delay ack in usecs */
		4586	/* Copy the SYN pkt to optval
		4587	*
		4588	* BPF_PROG_TYPE_SOCK_OPS only. It is similar to the
		4589	* bpf_getsockopt(TCP_SAVED_SYN) but it does not limit
		4590	* to only getting from the saved_syn. It can either get the
		4591	* syn packet from:
		4592	*
		4593	* 1. the just-received SYN packet (only available when writing the
		4594	* SYNACK). It will be useful when it is not necessary to
		4595	* save the SYN packet for latter use. It is also the only way
		4596	* to get the SYN during syncookie mode because the syn
		4597	* packet cannot be saved during syncookie.
		4598	*
		4599	* OR
		4600	*
		4601	* 2. the earlier saved syn which was done by
		4602	* bpf_setsockopt(TCP_SAVE_SYN).
		4603	*
		4604	* The bpf_getsockopt(TCP_BPF_SYN*) option will hide where the
		4605	* SYN packet is obtained.
		4606	*
		4607	* If the bpf-prog does not need the IP[46] header, the
		4608	* bpf-prog can avoid parsing the IP header by using
		4609	* TCP_BPF_SYN. Otherwise, the bpf-prog can get both
		4610	* IP[46] and TCP header by using TCP_BPF_SYN_IP.
		4611	*
		4612	* >0: Total number of bytes copied
		4613	* -ENOSPC: Not enough space in optval. Only optlen number of
		4614	* bytes is copied.
		4615	* -ENOENT: The SYN skb is not available now and the earlier SYN pkt
		4616	* is not saved by setsockopt(TCP_SAVE_SYN).
		4617	*/
		4618	TCP_BPF_SYN = 1005, /* Copy the TCP header */
		4619	TCP_BPF_SYN_IP = 1006, /* Copy the IP[46] and TCP header */
		4620	TCP_BPF_SYN_MAC = 1007, /* Copy the MAC, IP[46], and TCP header */
		4621	};
		4622
		4623	enum {
		4624	BPF_LOAD_HDR_OPT_TCP_SYN = (1ULL << 0),
		4625	};
		4626
		4627	/* args[0] value during BPF_SOCK_OPS_HDR_OPT_LEN_CB and
		4628	* BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
		4629	*/
		4630	enum {
		4631	BPF_WRITE_HDR_TCP_CURRENT_MSS = 1, /* Kernel is finding the
		4632	* total option spaces
		4633	* required for an established
		4634	* sk in order to calculate the
		4635	* MSS. No skb is actually
		4636	* sent.
		4637	*/
		4638	BPF_WRITE_HDR_TCP_SYNACK_COOKIE = 2, /* Kernel is in syncookie mode
		4639	* when sending a SYN.
		4640	*/
4253	};	4641	};
4254		4642
4255	struct bpf_perf_event_value {	4643	struct bpf_perf_event_value {

init/Kconfig+3−0

 	bool "Enable bpf() system call"
 	select BPF
 	select IRQ_WORK
+	select TASKS_TRACE_RCU
 	default n
 	help
 	  Enable the bpf() system call that allows to manipulate eBPF
 	def_bool ARCH_WANT_DEFAULT_BPF_JIT || BPF_JIT_ALWAYS_ON
 	depends on HAVE_EBPF_JIT && BPF_JIT
+source "kernel/bpf/preload/Kconfig"
 config USERFAULTFD
 	bool "Enable userfaultfd() system call"
 	depends on MMU

kernel/Makefile+1−1

@@ -12,7 +12,7 @@ obj-y = fork.o exec_domain.o panic.o \ · 11 unmodified lines
12	notifier.o ksysfs.o cred.o reboot.o \	12	notifier.o ksysfs.o cred.o reboot.o \
13	async.o range.o smpboot.o ucount.o regset.o	13	async.o range.o smpboot.o ucount.o regset.o
14		14
15	obj-$(CONFIG_BPFILTER) += usermode_driver.o	15	obj-$(CONFIG_USERMODE_DRIVER) += usermode_driver.o
16	obj-$(CONFIG_MODULES) += kmod.o	16	obj-$(CONFIG_MODULES) += kmod.o
17	obj-$(CONFIG_MULTIUSER) += groups.o	17	obj-$(CONFIG_MULTIUSER) += groups.o
18		18

kernel/bpf/Makefile+3−0

 obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o task_iter.o prog_iter.o
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
+obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
 obj-$(CONFIG_BPF_SYSCALL) += btf.o
 ifeq ($(CONFIG_NET),y)
 obj-$(CONFIG_BPF_SYSCALL) += devmap.o
 obj-$(CONFIG_BPF_SYSCALL) += cpumap.o
+obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += offload.o
 obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o
 endif
 obj-$(CONFIG_BPF_SYSCALL) += bpf_struct_ops.o
 obj-${CONFIG_BPF_LSM} += bpf_lsm.o
 endif
+obj-$(CONFIG_BPF_PRELOAD) += preload/

kernel/bpf/arraymap.c+17−0

 #include <linux/filter.h>
 #include <linux/perf_event.h>
 #include <uapi/linux/btf.h>
+#include <linux/rcupdate_trace.h>
 #include "map_in_map.h"
 				   vma->vm_pgoff + pgoff);
 }
+static bool array_map_meta_equal(const struct bpf_map *meta0,
+				 const struct bpf_map *meta1)
+{
+	return meta0->max_entries == meta1->max_entries &&
+		bpf_map_meta_equal(meta0, meta1);
+}
 struct bpf_iter_seq_array_map_info {
 	struct bpf_map *map;
 	void *percpu_value_buf;
 static int array_map_btf_id;
 const struct bpf_map_ops array_map_ops = {
+	.map_meta_equal = array_map_meta_equal,
 	.map_alloc_check = array_map_alloc_check,
 	.map_alloc = array_map_alloc,
 	.map_free = array_map_free,
 static int percpu_array_map_btf_id;
 const struct bpf_map_ops percpu_array_map_ops = {
+	.map_meta_equal = bpf_map_meta_equal,
 	.map_alloc_check = array_map_alloc_check,
 	.map_alloc = array_map_alloc,
 	.map_free = array_map_free,
 	fd_array_map_free(map);
 }
+/* prog_array->aux->{type,jited} is a runtime binding.
+ * Doing static check alone in the verifier is not enough.
+ * Thus, prog_array_map cannot be used as an inner_map
+ * and map_meta_equal is not implemented.
+ */
 static int prog_array_map_btf_id;
 const struct bpf_map_ops prog_array_map_ops = {
 	.map_alloc_check = fd_array_map_alloc_check,
 static int perf_event_array_map_btf_id;
 const struct bpf_map_ops perf_event_array_map_ops = {
+	.map_meta_equal = bpf_map_meta_equal,
 	.map_alloc_check = fd_array_map_alloc_check,
 	.map_alloc = array_map_alloc,
 	.map_free = fd_array_map_free,
 static int cgroup_array_map_btf_id;
 const struct bpf_map_ops cgroup_array_map_ops = {
+	.map_meta_equal = bpf_map_meta_equal,
 	.map_alloc_check = fd_array_map_alloc_check,
 	.map_alloc = array_map_alloc,
 	.map_free = cgroup_fd_array_free,

kernel/bpf/bpf_inode_storage.c+274−0

@@ -0,0 +1,274 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	/*
	3	* Copyright (c) 2019 Facebook
	4	* Copyright 2020 Google LLC.
	5	*/
	6
	7	#include <linux/rculist.h>
	8	#include <linux/list.h>
	9	#include <linux/hash.h>
	10	#include <linux/types.h>
	11	#include <linux/spinlock.h>
	12	#include <linux/bpf.h>
	13	#include <linux/bpf_local_storage.h>
	14	#include <net/sock.h>
	15	#include <uapi/linux/sock_diag.h>
	16	#include <uapi/linux/btf.h>
	17	#include <linux/bpf_lsm.h>
	18	#include <linux/btf_ids.h>
	19	#include <linux/fdtable.h>
	20
	21	DEFINE_BPF_STORAGE_CACHE(inode_cache);
	22
	23	static struct bpf_local_storage __rcu **
	24	inode_storage_ptr(void *owner)
	25	{
	26	struct inode *inode = owner;
	27	struct bpf_storage_blob *bsb;
	28
	29	bsb = bpf_inode(inode);
	30	if (!bsb)
	31	return NULL;
	32	return &bsb->storage;
	33	}
	34
	35	static struct bpf_local_storage_data inode_storage_lookup(struct inode inode,
	36	struct bpf_map *map,
	37	bool cacheit_lockit)
	38	{
	39	struct bpf_local_storage *inode_storage;
	40	struct bpf_local_storage_map *smap;
	41	struct bpf_storage_blob *bsb;
	42
	43	bsb = bpf_inode(inode);
	44	if (!bsb)
	45	return NULL;
	46
	47	inode_storage = rcu_dereference(bsb->storage);
	48	if (!inode_storage)
	49	return NULL;
	50
	51	smap = (struct bpf_local_storage_map *)map;
	52	return bpf_local_storage_lookup(inode_storage, smap, cacheit_lockit);
	53	}
	54
	55	void bpf_inode_storage_free(struct inode *inode)
	56	{
	57	struct bpf_local_storage_elem *selem;
	58	struct bpf_local_storage *local_storage;
	59	bool free_inode_storage = false;
	60	struct bpf_storage_blob *bsb;
	61	struct hlist_node *n;
	62
	63	bsb = bpf_inode(inode);
	64	if (!bsb)
	65	return;
	66
	67	rcu_read_lock();
	68
	69	local_storage = rcu_dereference(bsb->storage);
	70	if (!local_storage) {
	71	rcu_read_unlock();
	72	return;
	73	}
	74
	75	/* Netiher the bpf_prog nor the bpf-map's syscall
	76	* could be modifying the local_storage->list now.
	77	* Thus, no elem can be added-to or deleted-from the
	78	* local_storage->list by the bpf_prog or by the bpf-map's syscall.
	79	*
	80	* It is racing with bpf_local_storage_map_free() alone
	81	* when unlinking elem from the local_storage->list and
	82	* the map's bucket->list.
	83	*/
	84	raw_spin_lock_bh(&local_storage->lock);
	85	hlist_for_each_entry_safe(selem, n, &local_storage->list, snode) {
	86	/* Always unlink from map before unlinking from
	87	* local_storage.
	88	*/
	89	bpf_selem_unlink_map(selem);
	90	free_inode_storage = bpf_selem_unlink_storage_nolock(
	91	local_storage, selem, false);
	92	}
	93	raw_spin_unlock_bh(&local_storage->lock);
	94	rcu_read_unlock();
	95
	96	/* free_inoode_storage should always be true as long as
	97	* local_storage->list was non-empty.
	98	*/
	99	if (free_inode_storage)
	100	kfree_rcu(local_storage, rcu);
	101	}
	102
	103	static void bpf_fd_inode_storage_lookup_elem(struct bpf_map map, void *key)
	104	{
	105	struct bpf_local_storage_data *sdata;
	106	struct file *f;
	107	int fd;
	108
	109	fd = (int )key;
	110	f = fget_raw(fd);
	111	if (!f)
	112	return NULL;
	113
	114	sdata = inode_storage_lookup(f->f_inode, map, true);
	115	fput(f);
	116	return sdata ? sdata->data : NULL;
	117	}
	118
	119	static int bpf_fd_inode_storage_update_elem(struct bpf_map map, void key,
	120	void *value, u64 map_flags)
	121	{
	122	struct bpf_local_storage_data *sdata;
	123	struct file *f;
	124	int fd;
	125
	126	fd = (int )key;
	127	f = fget_raw(fd);
	128	if (!f \|\| !inode_storage_ptr(f->f_inode))
	129	return -EBADF;
	130
	131	sdata = bpf_local_storage_update(f->f_inode,
	132	(struct bpf_local_storage_map *)map,
	133	value, map_flags);
	134	fput(f);
	135	return PTR_ERR_OR_ZERO(sdata);
	136	}
	137
	138	static int inode_storage_delete(struct inode inode, struct bpf_map map)
	139	{
	140	struct bpf_local_storage_data *sdata;
	141
	142	sdata = inode_storage_lookup(inode, map, false);
	143	if (!sdata)
	144	return -ENOENT;
	145
	146	bpf_selem_unlink(SELEM(sdata));
	147
	148	return 0;
	149	}
	150
	151	static int bpf_fd_inode_storage_delete_elem(struct bpf_map map, void key)
	152	{
	153	struct file *f;
	154	int fd, err;
	155
	156	fd = (int )key;
	157	f = fget_raw(fd);
	158	if (!f)
	159	return -EBADF;
	160
	161	err = inode_storage_delete(f->f_inode, map);
	162	fput(f);
	163	return err;
	164	}
	165
	166	BPF_CALL_4(bpf_inode_storage_get, struct bpf_map , map, struct inode , inode,
	167	void *, value, u64, flags)
	168	{
	169	struct bpf_local_storage_data *sdata;
	170
	171	if (flags & ~(BPF_LOCAL_STORAGE_GET_F_CREATE))
	172	return (unsigned long)NULL;
	173
	174	/* explicitly check that the inode_storage_ptr is not
	175	* NULL as inode_storage_lookup returns NULL in this case and
	176	* bpf_local_storage_update expects the owner to have a
	177	* valid storage pointer.
	178	*/
	179	if (!inode_storage_ptr(inode))
	180	return (unsigned long)NULL;
	181
	182	sdata = inode_storage_lookup(inode, map, true);
	183	if (sdata)
	184	return (unsigned long)sdata->data;
	185
	186	/* This helper must only called from where the inode is gurranteed
	187	* to have a refcount and cannot be freed.
	188	*/
	189	if (flags & BPF_LOCAL_STORAGE_GET_F_CREATE) {
	190	sdata = bpf_local_storage_update(
	191	inode, (struct bpf_local_storage_map *)map, value,
	192	BPF_NOEXIST);
	193	return IS_ERR(sdata) ? (unsigned long)NULL :
	194	(unsigned long)sdata->data;
	195	}
	196
	197	return (unsigned long)NULL;
	198	}
	199
	200	BPF_CALL_2(bpf_inode_storage_delete,
	201	struct bpf_map , map, struct inode , inode)
	202	{
	203	/* This helper must only called from where the inode is gurranteed
	204	* to have a refcount and cannot be freed.
	205	*/
	206	return inode_storage_delete(inode, map);
	207	}
	208
	209	static int notsupp_get_next_key(struct bpf_map map, void key,
	210	void *next_key)
	211	{
	212	return -ENOTSUPP;
	213	}
	214
	215	static struct bpf_map inode_storage_map_alloc(union bpf_attr attr)
	216	{
	217	struct bpf_local_storage_map *smap;
	218
	219	smap = bpf_local_storage_map_alloc(attr);
	220	if (IS_ERR(smap))
	221	return ERR_CAST(smap);
	222
	223	smap->cache_idx = bpf_local_storage_cache_idx_get(&inode_cache);
	224	return &smap->map;
	225	}
	226
	227	static void inode_storage_map_free(struct bpf_map *map)
	228	{
	229	struct bpf_local_storage_map *smap;
	230
	231	smap = (struct bpf_local_storage_map *)map;
	232	bpf_local_storage_cache_idx_free(&inode_cache, smap->cache_idx);
	233	bpf_local_storage_map_free(smap);
	234	}
	235
	236	static int inode_storage_map_btf_id;
	237	const struct bpf_map_ops inode_storage_map_ops = {
	238	.map_meta_equal = bpf_map_meta_equal,
	239	.map_alloc_check = bpf_local_storage_map_alloc_check,
	240	.map_alloc = inode_storage_map_alloc,
	241	.map_free = inode_storage_map_free,
	242	.map_get_next_key = notsupp_get_next_key,
	243	.map_lookup_elem = bpf_fd_inode_storage_lookup_elem,
	244	.map_update_elem = bpf_fd_inode_storage_update_elem,
	245	.map_delete_elem = bpf_fd_inode_storage_delete_elem,
	246	.map_check_btf = bpf_local_storage_map_check_btf,
	247	.map_btf_name = "bpf_local_storage_map",
	248	.map_btf_id = &inode_storage_map_btf_id,
	249	.map_owner_storage_ptr = inode_storage_ptr,
	250	};
	251
	252	BTF_ID_LIST(bpf_inode_storage_btf_ids)
	253	BTF_ID_UNUSED
	254	BTF_ID(struct, inode)
	255
	256	const struct bpf_func_proto bpf_inode_storage_get_proto = {
	257	.func = bpf_inode_storage_get,
	258	.gpl_only = false,
	259	.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
	260	.arg1_type = ARG_CONST_MAP_PTR,
	261	.arg2_type = ARG_PTR_TO_BTF_ID,
	262	.arg3_type = ARG_PTR_TO_MAP_VALUE_OR_NULL,
	263	.arg4_type = ARG_ANYTHING,
	264	.btf_id = bpf_inode_storage_btf_ids,
	265	};
	266
	267	const struct bpf_func_proto bpf_inode_storage_delete_proto = {
	268	.func = bpf_inode_storage_delete,
	269	.gpl_only = false,
	270	.ret_type = RET_INTEGER,
	271	.arg1_type = ARG_CONST_MAP_PTR,
	272	.arg2_type = ARG_PTR_TO_BTF_ID,
	273	.btf_id = bpf_inode_storage_btf_ids,
	274	};

kernel/bpf/bpf_iter.c+58−0

@@ -390,10 +390,68 @@ out_unlock: · 389 unmodified lines
390	return ret;	390	return ret;
391	}	391	}
392		392
		393	static void bpf_iter_link_show_fdinfo(const struct bpf_link *link,
		394	struct seq_file *seq)
		395	{
		396	struct bpf_iter_link *iter_link =
		397	container_of(link, struct bpf_iter_link, link);
		398	bpf_iter_show_fdinfo_t show_fdinfo;
		399
		400	seq_printf(seq,
		401	"target_name:\t%s\n",
		402	iter_link->tinfo->reg_info->target);
		403
		404	show_fdinfo = iter_link->tinfo->reg_info->show_fdinfo;
		405	if (show_fdinfo)
		406	show_fdinfo(&iter_link->aux, seq);
		407	}
		408
		409	static int bpf_iter_link_fill_link_info(const struct bpf_link *link,
		410	struct bpf_link_info *info)
		411	{
		412	struct bpf_iter_link *iter_link =
		413	container_of(link, struct bpf_iter_link, link);
		414	char __user *ubuf = u64_to_user_ptr(info->iter.target_name);
		415	bpf_iter_fill_link_info_t fill_link_info;
		416	u32 ulen = info->iter.target_name_len;
		417	const char *target_name;
		418	u32 target_len;
		419
		420	if (!ulen ^ !ubuf)
		421	return -EINVAL;
		422
		423	target_name = iter_link->tinfo->reg_info->target;
		424	target_len = strlen(target_name);
		425	info->iter.target_name_len = target_len + 1;
		426
		427	if (ubuf) {
		428	if (ulen >= target_len + 1) {
		429	if (copy_to_user(ubuf, target_name, target_len + 1))
		430	return -EFAULT;
		431	} else {
		432	char zero = '\0';
		433
		434	if (copy_to_user(ubuf, target_name, ulen - 1))
		435	return -EFAULT;
		436	if (put_user(zero, ubuf + ulen - 1))
		437	return -EFAULT;
		438	return -ENOSPC;
		439	}
		440	}
		441
		442	fill_link_info = iter_link->tinfo->reg_info->fill_link_info;
		443	if (fill_link_info)
		444	return fill_link_info(&iter_link->aux, info);
		445
		446	return 0;
		447	}
		448
393	static const struct bpf_link_ops bpf_iter_link_lops = {	449	static const struct bpf_link_ops bpf_iter_link_lops = {
394	.release = bpf_iter_link_release,	450	.release = bpf_iter_link_release,
395	.dealloc = bpf_iter_link_dealloc,	451	.dealloc = bpf_iter_link_dealloc,
396	.update_prog = bpf_iter_link_replace,	452	.update_prog = bpf_iter_link_replace,
		453	.show_fdinfo = bpf_iter_link_show_fdinfo,
		454	.fill_link_info = bpf_iter_link_fill_link_info,
397	};	455	};
398		456
399	bool bpf_link_is_iter(struct bpf_link *link)	457	bool bpf_link_is_iter(struct bpf_link *link)

kernel/bpf/bpf_local_storage.c+600−0

@@ -0,0 +1,600 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	/* Copyright (c) 2019 Facebook */
	3	#include <linux/rculist.h>
	4	#include <linux/list.h>
	5	#include <linux/hash.h>
	6	#include <linux/types.h>
	7	#include <linux/spinlock.h>
	8	#include <linux/bpf.h>
	9	#include <linux/btf_ids.h>
	10	#include <linux/bpf_local_storage.h>
	11	#include <net/sock.h>
	12	#include <uapi/linux/sock_diag.h>
	13	#include <uapi/linux/btf.h>
	14
	15	#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK (BPF_F_NO_PREALLOC \| BPF_F_CLONE)
	16
	17	static struct bpf_local_storage_map_bucket *
	18	select_bucket(struct bpf_local_storage_map *smap,
	19	struct bpf_local_storage_elem *selem)
	20	{
	21	return &smap->buckets[hash_ptr(selem, smap->bucket_log)];
	22	}
	23
	24	static int mem_charge(struct bpf_local_storage_map smap, void owner, u32 size)
	25	{
	26	struct bpf_map *map = &smap->map;
	27
	28	if (!map->ops->map_local_storage_charge)
	29	return 0;
	30
	31	return map->ops->map_local_storage_charge(smap, owner, size);
	32	}
	33
	34	static void mem_uncharge(struct bpf_local_storage_map smap, void owner,
	35	u32 size)
	36	{
	37	struct bpf_map *map = &smap->map;
	38
	39	if (map->ops->map_local_storage_uncharge)
	40	map->ops->map_local_storage_uncharge(smap, owner, size);
	41	}
	42
	43	static struct bpf_local_storage __rcu **
	44	owner_storage(struct bpf_local_storage_map smap, void owner)
	45	{
	46	struct bpf_map *map = &smap->map;
	47
	48	return map->ops->map_owner_storage_ptr(owner);
	49	}
	50
	51	static bool selem_linked_to_storage(const struct bpf_local_storage_elem *selem)
	52	{
	53	return !hlist_unhashed(&selem->snode);
	54	}
	55
	56	static bool selem_linked_to_map(const struct bpf_local_storage_elem *selem)
	57	{
	58	return !hlist_unhashed(&selem->map_node);
	59	}
	60
	61	struct bpf_local_storage_elem *
	62	bpf_selem_alloc(struct bpf_local_storage_map smap, void owner,
	63	void *value, bool charge_mem)
	64	{
	65	struct bpf_local_storage_elem *selem;
	66
	67	if (charge_mem && mem_charge(smap, owner, smap->elem_size))
	68	return NULL;
	69
	70	selem = kzalloc(smap->elem_size, GFP_ATOMIC \| __GFP_NOWARN);
	71	if (selem) {
	72	if (value)
	73	memcpy(SDATA(selem)->data, value, smap->map.value_size);
	74	return selem;
	75	}
	76
	77	if (charge_mem)
	78	mem_uncharge(smap, owner, smap->elem_size);
	79
	80	return NULL;
	81	}
	82
	83	/* local_storage->lock must be held and selem->local_storage == local_storage.
	84	* The caller must ensure selem->smap is still valid to be
	85	* dereferenced for its smap->elem_size and smap->cache_idx.
	86	*/
	87	bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage,
	88	struct bpf_local_storage_elem *selem,
	89	bool uncharge_mem)
	90	{
	91	struct bpf_local_storage_map *smap;
	92	bool free_local_storage;
	93	void *owner;
	94
	95	smap = rcu_dereference(SDATA(selem)->smap);
	96	owner = local_storage->owner;
	97
	98	/* All uncharging on the owner must be done first.
	99	* The owner may be freed once the last selem is unlinked
	100	* from local_storage.
	101	*/
	102	if (uncharge_mem)
	103	mem_uncharge(smap, owner, smap->elem_size);
	104
	105	free_local_storage = hlist_is_singular_node(&selem->snode,
	106	&local_storage->list);
	107	if (free_local_storage) {
	108	mem_uncharge(smap, owner, sizeof(struct bpf_local_storage));
	109	local_storage->owner = NULL;
	110
	111	/* After this RCU_INIT, owner may be freed and cannot be used */
	112	RCU_INIT_POINTER(*owner_storage(smap, owner), NULL);
	113
	114	/* local_storage is not freed now. local_storage->lock is
	115	* still held and raw_spin_unlock_bh(&local_storage->lock)
	116	* will be done by the caller.
	117	*
	118	* Although the unlock will be done under
	119	* rcu_read_lock(), it is more intutivie to
	120	* read if kfree_rcu(local_storage, rcu) is done
	121	* after the raw_spin_unlock_bh(&local_storage->lock).
	122	*
	123	* Hence, a "bool free_local_storage" is returned
	124	* to the caller which then calls the kfree_rcu()
	125	* after unlock.
	126	*/
	127	}
	128	hlist_del_init_rcu(&selem->snode);
	129	if (rcu_access_pointer(local_storage->cache[smap->cache_idx]) ==
	130	SDATA(selem))
	131	RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL);
	132
	133	kfree_rcu(selem, rcu);
	134
	135	return free_local_storage;
	136	}
	137
	138	static void __bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem)
	139	{
	140	struct bpf_local_storage *local_storage;
	141	bool free_local_storage = false;
	142
	143	if (unlikely(!selem_linked_to_storage(selem)))
	144	/* selem has already been unlinked from sk */
	145	return;
	146
	147	local_storage = rcu_dereference(selem->local_storage);
	148	raw_spin_lock_bh(&local_storage->lock);
	149	if (likely(selem_linked_to_storage(selem)))
	150	free_local_storage = bpf_selem_unlink_storage_nolock(
	151	local_storage, selem, true);
	152	raw_spin_unlock_bh(&local_storage->lock);
	153
	154	if (free_local_storage)
	155	kfree_rcu(local_storage, rcu);
	156	}
	157
	158	void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,
	159	struct bpf_local_storage_elem *selem)
	160	{
	161	RCU_INIT_POINTER(selem->local_storage, local_storage);
	162	hlist_add_head(&selem->snode, &local_storage->list);
	163	}
	164
	165	void bpf_selem_unlink_map(struct bpf_local_storage_elem *selem)
	166	{
	167	struct bpf_local_storage_map *smap;
	168	struct bpf_local_storage_map_bucket *b;
	169
	170	if (unlikely(!selem_linked_to_map(selem)))
	171	/* selem has already be unlinked from smap */
	172	return;
	173
	174	smap = rcu_dereference(SDATA(selem)->smap);
	175	b = select_bucket(smap, selem);
	176	raw_spin_lock_bh(&b->lock);
	177	if (likely(selem_linked_to_map(selem)))
	178	hlist_del_init_rcu(&selem->map_node);
	179	raw_spin_unlock_bh(&b->lock);
	180	}
	181
	182	void bpf_selem_link_map(struct bpf_local_storage_map *smap,
	183	struct bpf_local_storage_elem *selem)
	184	{
	185	struct bpf_local_storage_map_bucket *b = select_bucket(smap, selem);
	186
	187	raw_spin_lock_bh(&b->lock);
	188	RCU_INIT_POINTER(SDATA(selem)->smap, smap);
	189	hlist_add_head_rcu(&selem->map_node, &b->list);
	190	raw_spin_unlock_bh(&b->lock);
	191	}
	192
	193	void bpf_selem_unlink(struct bpf_local_storage_elem *selem)
	194	{
	195	/* Always unlink from map before unlinking from local_storage
	196	* because selem will be freed after successfully unlinked from
	197	* the local_storage.
	198	*/
	199	bpf_selem_unlink_map(selem);
	200	__bpf_selem_unlink_storage(selem);
	201	}
	202
	203	struct bpf_local_storage_data *
	204	bpf_local_storage_lookup(struct bpf_local_storage *local_storage,
	205	struct bpf_local_storage_map *smap,
	206	bool cacheit_lockit)
	207	{
	208	struct bpf_local_storage_data *sdata;
	209	struct bpf_local_storage_elem *selem;
	210
	211	/* Fast path (cache hit) */
	212	sdata = rcu_dereference(local_storage->cache[smap->cache_idx]);
	213	if (sdata && rcu_access_pointer(sdata->smap) == smap)
	214	return sdata;
	215
	216	/* Slow path (cache miss) */
	217	hlist_for_each_entry_rcu(selem, &local_storage->list, snode)
	218	if (rcu_access_pointer(SDATA(selem)->smap) == smap)
	219	break;
	220
	221	if (!selem)
	222	return NULL;
	223
	224	sdata = SDATA(selem);
	225	if (cacheit_lockit) {
	226	/* spinlock is needed to avoid racing with the
	227	* parallel delete. Otherwise, publishing an already
	228	* deleted sdata to the cache will become a use-after-free
	229	* problem in the next bpf_local_storage_lookup().
	230	*/
	231	raw_spin_lock_bh(&local_storage->lock);
	232	if (selem_linked_to_storage(selem))
	233	rcu_assign_pointer(local_storage->cache[smap->cache_idx],
	234	sdata);
	235	raw_spin_unlock_bh(&local_storage->lock);
	236	}
	237
	238	return sdata;
	239	}
	240
	241	static int check_flags(const struct bpf_local_storage_data *old_sdata,
	242	u64 map_flags)
	243	{
	244	if (old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_NOEXIST)
	245	/* elem already exists */
	246	return -EEXIST;
	247
	248	if (!old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_EXIST)
	249	/* elem doesn't exist, cannot update it */
	250	return -ENOENT;
	251
	252	return 0;
	253	}
	254
	255	int bpf_local_storage_alloc(void *owner,
	256	struct bpf_local_storage_map *smap,
	257	struct bpf_local_storage_elem *first_selem)
	258	{
	259	struct bpf_local_storage prev_storage, storage;
	260	struct bpf_local_storage **owner_storage_ptr;
	261	int err;
	262
	263	err = mem_charge(smap, owner, sizeof(*storage));
	264	if (err)
	265	return err;
	266
	267	storage = kzalloc(sizeof(*storage), GFP_ATOMIC \| __GFP_NOWARN);
	268	if (!storage) {
	269	err = -ENOMEM;
	270	goto uncharge;
	271	}
	272
	273	INIT_HLIST_HEAD(&storage->list);
	274	raw_spin_lock_init(&storage->lock);
	275	storage->owner = owner;
	276
	277	bpf_selem_link_storage_nolock(storage, first_selem);
	278	bpf_selem_link_map(smap, first_selem);
	279
	280	owner_storage_ptr =
	281	(struct bpf_local_storage **)owner_storage(smap, owner);
	282	/* Publish storage to the owner.
	283	* Instead of using any lock of the kernel object (i.e. owner),
	284	* cmpxchg will work with any kernel object regardless what
	285	* the running context is, bh, irq...etc.
	286	*
	287	* From now on, the owner->storage pointer (e.g. sk->sk_bpf_storage)
	288	* is protected by the storage->lock. Hence, when freeing
	289	* the owner->storage, the storage->lock must be held before
	290	* setting owner->storage ptr to NULL.
	291	*/
	292	prev_storage = cmpxchg(owner_storage_ptr, NULL, storage);
	293	if (unlikely(prev_storage)) {
	294	bpf_selem_unlink_map(first_selem);
	295	err = -EAGAIN;
	296	goto uncharge;
	297
	298	/* Note that even first_selem was linked to smap's
	299	* bucket->list, first_selem can be freed immediately
	300	* (instead of kfree_rcu) because
	301	* bpf_local_storage_map_free() does a
	302	* synchronize_rcu() before walking the bucket->list.
	303	* Hence, no one is accessing selem from the
	304	* bucket->list under rcu_read_lock().
	305	*/
	306	}
	307
	308	return 0;
	309
	310	uncharge:
	311	kfree(storage);
	312	mem_uncharge(smap, owner, sizeof(*storage));
	313	return err;
	314	}
	315
	316	/* sk cannot be going away because it is linking new elem
	317	* to sk->sk_bpf_storage. (i.e. sk->sk_refcnt cannot be 0).
	318	* Otherwise, it will become a leak (and other memory issues
	319	* during map destruction).
	320	*/
	321	struct bpf_local_storage_data *
	322	bpf_local_storage_update(void owner, struct bpf_local_storage_map smap,
	323	void *value, u64 map_flags)
	324	{
	325	struct bpf_local_storage_data *old_sdata = NULL;
	326	struct bpf_local_storage_elem *selem;
	327	struct bpf_local_storage *local_storage;
	328	int err;
	329
	330	/* BPF_EXIST and BPF_NOEXIST cannot be both set */
	331	if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST) \|\|
	332	/* BPF_F_LOCK can only be used in a value with spin_lock */
	333	unlikely((map_flags & BPF_F_LOCK) &&
	334	!map_value_has_spin_lock(&smap->map)))
	335	return ERR_PTR(-EINVAL);
	336
	337	local_storage = rcu_dereference(*owner_storage(smap, owner));
	338	if (!local_storage \|\| hlist_empty(&local_storage->list)) {
	339	/* Very first elem for the owner */
	340	err = check_flags(NULL, map_flags);
	341	if (err)
	342	return ERR_PTR(err);
	343
	344	selem = bpf_selem_alloc(smap, owner, value, true);
	345	if (!selem)
	346	return ERR_PTR(-ENOMEM);
	347
	348	err = bpf_local_storage_alloc(owner, smap, selem);
	349	if (err) {
	350	kfree(selem);
	351	mem_uncharge(smap, owner, smap->elem_size);
	352	return ERR_PTR(err);
	353	}
	354
	355	return SDATA(selem);
	356	}
	357
	358	if ((map_flags & BPF_F_LOCK) && !(map_flags & BPF_NOEXIST)) {
	359	/* Hoping to find an old_sdata to do inline update
	360	* such that it can avoid taking the local_storage->lock
	361	* and changing the lists.
	362	*/
	363	old_sdata =
	364	bpf_local_storage_lookup(local_storage, smap, false);
	365	err = check_flags(old_sdata, map_flags);
	366	if (err)
	367	return ERR_PTR(err);
	368	if (old_sdata && selem_linked_to_storage(SELEM(old_sdata))) {
	369	copy_map_value_locked(&smap->map, old_sdata->data,
	370	value, false);
	371	return old_sdata;
	372	}
	373	}
	374
	375	raw_spin_lock_bh(&local_storage->lock);
	376
	377	/* Recheck local_storage->list under local_storage->lock */
	378	if (unlikely(hlist_empty(&local_storage->list))) {
	379	/* A parallel del is happening and local_storage is going
	380	* away. It has just been checked before, so very
	381	* unlikely. Return instead of retry to keep things
	382	* simple.
	383	*/
	384	err = -EAGAIN;
	385	goto unlock_err;
	386	}
	387
	388	old_sdata = bpf_local_storage_lookup(local_storage, smap, false);
	389	err = check_flags(old_sdata, map_flags);
	390	if (err)
	391	goto unlock_err;
	392
	393	if (old_sdata && (map_flags & BPF_F_LOCK)) {
	394	copy_map_value_locked(&smap->map, old_sdata->data, value,
	395	false);
	396	selem = SELEM(old_sdata);
	397	goto unlock;
	398	}
	399
	400	/* local_storage->lock is held. Hence, we are sure
	401	* we can unlink and uncharge the old_sdata successfully
	402	* later. Hence, instead of charging the new selem now
	403	* and then uncharge the old selem later (which may cause
	404	* a potential but unnecessary charge failure), avoid taking
	405	* a charge at all here (the "!old_sdata" check) and the
	406	* old_sdata will not be uncharged later during
	407	* bpf_selem_unlink_storage_nolock().
	408	*/
	409	selem = bpf_selem_alloc(smap, owner, value, !old_sdata);
	410	if (!selem) {
	411	err = -ENOMEM;
	412	goto unlock_err;
	413	}
	414
	415	/* First, link the new selem to the map */
	416	bpf_selem_link_map(smap, selem);
	417
	418	/* Second, link (and publish) the new selem to local_storage */
	419	bpf_selem_link_storage_nolock(local_storage, selem);
	420
	421	/* Third, remove old selem, SELEM(old_sdata) */
	422	if (old_sdata) {
	423	bpf_selem_unlink_map(SELEM(old_sdata));
	424	bpf_selem_unlink_storage_nolock(local_storage, SELEM(old_sdata),
	425	false);
	426	}
	427
	428	unlock:
	429	raw_spin_unlock_bh(&local_storage->lock);
	430	return SDATA(selem);
	431
	432	unlock_err:
	433	raw_spin_unlock_bh(&local_storage->lock);
	434	return ERR_PTR(err);
	435	}
	436
	437	u16 bpf_local_storage_cache_idx_get(struct bpf_local_storage_cache *cache)
	438	{
	439	u64 min_usage = U64_MAX;
	440	u16 i, res = 0;
	441
	442	spin_lock(&cache->idx_lock);
	443
	444	for (i = 0; i < BPF_LOCAL_STORAGE_CACHE_SIZE; i++) {
	445	if (cache->idx_usage_counts[i] < min_usage) {
	446	min_usage = cache->idx_usage_counts[i];
	447	res = i;
	448
	449	/* Found a free cache_idx */
	450	if (!min_usage)
	451	break;
	452	}
	453	}
	454	cache->idx_usage_counts[res]++;
	455
	456	spin_unlock(&cache->idx_lock);
	457
	458	return res;
	459	}
	460
	461	void bpf_local_storage_cache_idx_free(struct bpf_local_storage_cache *cache,
	462	u16 idx)
	463	{
	464	spin_lock(&cache->idx_lock);
	465	cache->idx_usage_counts[idx]--;
	466	spin_unlock(&cache->idx_lock);
	467	}
	468
	469	void bpf_local_storage_map_free(struct bpf_local_storage_map *smap)
	470	{
	471	struct bpf_local_storage_elem *selem;
	472	struct bpf_local_storage_map_bucket *b;
	473	unsigned int i;
	474
	475	/* Note that this map might be concurrently cloned from
	476	* bpf_sk_storage_clone. Wait for any existing bpf_sk_storage_clone
	477	* RCU read section to finish before proceeding. New RCU
	478	* read sections should be prevented via bpf_map_inc_not_zero.
	479	*/
	480	synchronize_rcu();
	481
	482	/* bpf prog and the userspace can no longer access this map
	483	* now. No new selem (of this map) can be added
	484	* to the owner->storage or to the map bucket's list.
	485	*
	486	* The elem of this map can be cleaned up here
	487	* or when the storage is freed e.g.
	488	* by bpf_sk_storage_free() during __sk_destruct().
	489	*/
	490	for (i = 0; i < (1U << smap->bucket_log); i++) {
	491	b = &smap->buckets[i];
	492
	493	rcu_read_lock();
	494	/* No one is adding to b->list now */
	495	while ((selem = hlist_entry_safe(
	496	rcu_dereference_raw(hlist_first_rcu(&b->list)),
	497	struct bpf_local_storage_elem, map_node))) {
	498	bpf_selem_unlink(selem);
	499	cond_resched_rcu();
	500	}
	501	rcu_read_unlock();
	502	}
	503
	504	/* While freeing the storage we may still need to access the map.
	505	*
	506	* e.g. when bpf_sk_storage_free() has unlinked selem from the map
	507	* which then made the above while((selem = ...)) loop
	508	* exit immediately.
	509	*
	510	* However, while freeing the storage one still needs to access the
	511	* smap->elem_size to do the uncharging in
	512	* bpf_selem_unlink_storage_nolock().
	513	*
	514	* Hence, wait another rcu grace period for the storage to be freed.
	515	*/
	516	synchronize_rcu();
	517
	518	kvfree(smap->buckets);
	519	kfree(smap);
	520	}
	521
	522	int bpf_local_storage_map_alloc_check(union bpf_attr *attr)
	523	{
	524	if (attr->map_flags & ~BPF_LOCAL_STORAGE_CREATE_FLAG_MASK \|\|
	525	!(attr->map_flags & BPF_F_NO_PREALLOC) \|\|
	526	attr->max_entries \|\|
	527	attr->key_size != sizeof(int) \|\| !attr->value_size \|\|
	528	/* Enforce BTF for userspace sk dumping */
	529	!attr->btf_key_type_id \|\| !attr->btf_value_type_id)
	530	return -EINVAL;
	531
	532	if (!bpf_capable())
	533	return -EPERM;
	534
	535	if (attr->value_size > BPF_LOCAL_STORAGE_MAX_VALUE_SIZE)
	536	return -E2BIG;
	537
	538	return 0;
	539	}
	540
	541	struct bpf_local_storage_map bpf_local_storage_map_alloc(union bpf_attr attr)
	542	{
	543	struct bpf_local_storage_map *smap;
	544	unsigned int i;
	545	u32 nbuckets;
	546	u64 cost;
	547	int ret;
	548
	549	smap = kzalloc(sizeof(*smap), GFP_USER \| __GFP_NOWARN);
	550	if (!smap)
	551	return ERR_PTR(-ENOMEM);
	552	bpf_map_init_from_attr(&smap->map, attr);
	553
	554	nbuckets = roundup_pow_of_two(num_possible_cpus());
	555	/* Use at least 2 buckets, select_bucket() is undefined behavior with 1 bucket */
	556	nbuckets = max_t(u32, 2, nbuckets);
	557	smap->bucket_log = ilog2(nbuckets);
	558	cost = sizeof(smap->buckets) nbuckets + sizeof(*smap);
	559
	560	ret = bpf_map_charge_init(&smap->map.memory, cost);
	561	if (ret < 0) {
	562	kfree(smap);
	563	return ERR_PTR(ret);
	564	}
	565
	566	smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets,
	567	GFP_USER \| __GFP_NOWARN);
	568	if (!smap->buckets) {
	569	bpf_map_charge_finish(&smap->map.memory);
	570	kfree(smap);
	571	return ERR_PTR(-ENOMEM);
	572	}
	573
	574	for (i = 0; i < nbuckets; i++) {
	575	INIT_HLIST_HEAD(&smap->buckets[i].list);
	576	raw_spin_lock_init(&smap->buckets[i].lock);
	577	}
	578
	579	smap->elem_size =
	580	sizeof(struct bpf_local_storage_elem) + attr->value_size;
	581
	582	return smap;
	583	}
	584
	585	int bpf_local_storage_map_check_btf(const struct bpf_map *map,
	586	const struct btf *btf,
	587	const struct btf_type *key_type,
	588	const struct btf_type *value_type)
	589	{
	590	u32 int_data;
	591
	592	if (BTF_INFO_KIND(key_type->info) != BTF_KIND_INT)
	593	return -EINVAL;
	594
	595	int_data = (u32 )(key_type + 1);
	596	if (BTF_INT_BITS(int_data) != 32 \|\| BTF_INT_OFFSET(int_data))
	597	return -EINVAL;
	598
	599	return 0;
	600	}

kernel/bpf/bpf_lsm.c+20−1

@@ -11,6 +11,8 @@ · 10 unmodified lines
11	#include <linux/bpf_lsm.h>	11	#include <linux/bpf_lsm.h>
12	#include <linux/kallsyms.h>	12	#include <linux/kallsyms.h>
13	#include <linux/bpf_verifier.h>	13	#include <linux/bpf_verifier.h>
		14	#include <net/bpf_sk_storage.h>
		15	#include <linux/bpf_local_storage.h>
14		16
15	/* For every LSM hook that allows attachment of BPF programs, declare a nop	17	/* For every LSM hook that allows attachment of BPF programs, declare a nop
16	* function where a BPF program can be attached.	18	* function where a BPF program can be attached.
@@ -45,10 +47,27 @@ int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog, · 28 unmodified lines
45	return 0;	47	return 0;
46	}	48	}
47		49
		50	static const struct bpf_func_proto *
		51	bpf_lsm_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
		52	{
		53	switch (func_id) {
		54	case BPF_FUNC_inode_storage_get:
		55	return &bpf_inode_storage_get_proto;
		56	case BPF_FUNC_inode_storage_delete:
		57	return &bpf_inode_storage_delete_proto;
		58	case BPF_FUNC_sk_storage_get:
		59	return &sk_storage_get_btf_proto;
		60	case BPF_FUNC_sk_storage_delete:
		61	return &sk_storage_delete_btf_proto;
		62	default:
		63	return tracing_prog_func_proto(func_id, prog);
		64	}
		65	}
		66
48	const struct bpf_prog_ops lsm_prog_ops = {	67	const struct bpf_prog_ops lsm_prog_ops = {
49	};	68	};
50		69
51	const struct bpf_verifier_ops lsm_verifier_ops = {	70	const struct bpf_verifier_ops lsm_verifier_ops = {
52	.get_func_proto = tracing_prog_func_proto,	71	.get_func_proto = bpf_lsm_func_proto,
53	.is_valid_access = btf_ctx_access,	72	.is_valid_access = btf_ctx_access,
54	};	73	};

kernel/bpf/bpf_struct_ops.c+2−4

 			return -EINVAL;
 		mtype = btf_type_by_id(btf_vmlinux, member->type);
-		mtype = btf_resolve_size(btf_vmlinux, mtype, &msize,
+		mtype = btf_resolve_size(btf_vmlinux, mtype, &msize);
-					 NULL, NULL);
 		if (IS_ERR(mtype))
 			return PTR_ERR(mtype);
 		prev_mend = moff + msize;
 			u32 msize;
 			mtype = btf_type_by_id(btf_vmlinux, member->type);
-			mtype = btf_resolve_size(btf_vmlinux, mtype, &msize,
+			mtype = btf_resolve_size(btf_vmlinux, mtype, &msize);
-						 NULL, NULL);
 			if (IS_ERR(mtype)) {
 				err = PTR_ERR(mtype);
 				goto reset_unlock;

kernel/bpf/btf.c+139−24

 #include <linux/btf_ids.h>
 #include <linux/skmsg.h>
 #include <linux/perf_event.h>
+#include <linux/bsearch.h>
+#include <linux/btf_ids.h>
 #include <net/sock.h>
 /* BTF (BPF Type Format) is the meta data format which describes
  * *type_size: (x * y * sizeof(u32)).  Hence, *type_size always
  *             corresponds to the return type.
  * *elem_type: u32
+ * *elem_id: id of u32
  * *total_nelems: (x * y).  Hence, individual elem size is
  *                (*type_size / *total_nelems)
+ * *type_id: id of type if it's changed within the function, 0 if not
  *
  * type: is not an array (e.g. const struct X)
  * return type: type "struct X"
  * *type_size: sizeof(struct X)
  * *elem_type: same as return type ("struct X")
+ * *elem_id: 0
  * *total_nelems: 1
+ * *type_id: id of type if it's changed within the function, 0 if not
  */
-const struct btf_type *
+static const struct btf_type *
-btf_resolve_size(const struct btf *btf, const struct btf_type *type,
+__btf_resolve_size(const struct btf *btf, const struct btf_type *type,
-		 u32 *type_size, const struct btf_type **elem_type,
+		   u32 *type_size, const struct btf_type **elem_type,
-		 u32 *total_nelems)
+		   u32 *elem_id, u32 *total_nelems, u32 *type_id)
 {
 	const struct btf_type *array_type = NULL;
-	const struct btf_array *array;
+	const struct btf_array *array = NULL;
-	u32 i, size, nelems = 1;
+	u32 i, size, nelems = 1, id = 0;
 	for (i = 0; i < MAX_RESOLVE_DEPTH; i++) {
 		switch (BTF_INFO_KIND(type->info)) {
 		case BTF_KIND_VOLATILE:
 		case BTF_KIND_CONST:
 		case BTF_KIND_RESTRICT:
+			id = type->type;
 			type = btf_type_by_id(btf, type->type);
 			break;
 		*total_nelems = nelems;
 	if (elem_type)
 		*elem_type = type;
+	if (elem_id)
+		*elem_id = array ? array->type : 0;
+	if (type_id && id)
+		*type_id = id;
 	return array_type ? : type;
 }
+const struct btf_type *
+btf_resolve_size(const struct btf *btf, const struct btf_type *type,
+		 u32 *type_size)
+{
+	return __btf_resolve_size(btf, type, type_size, NULL, NULL, NULL, NULL);
+}
 /* The input param "type_id" must point to a needs_resolve type */
 static const struct btf_type *btf_type_id_resolve(const struct btf *btf,
 						  u32 *type_id)
 	return true;
 }
-int btf_struct_access(struct bpf_verifier_log *log,
+enum bpf_struct_walk_result {
-		      const struct btf_type *t, int off, int size,
+	/* < 0 error */
-		      enum bpf_access_type atype,
+	WALK_SCALAR = 0,
-		      u32 *next_btf_id)
+	WALK_PTR,
+	WALK_STRUCT,
+};
+static int btf_struct_walk(struct bpf_verifier_log *log,
+			   const struct btf_type *t, int off, int size,
+			   u32 *next_btf_id)
 {
 	u32 i, moff, mtrue_end, msize = 0, total_nelems = 0;
 	const struct btf_type *mtype, *elem_type = NULL;
 	const struct btf_member *member;
 	const char *tname, *mname;
-	u32 vlen;
+	u32 vlen, elem_id, mid;
 again:
 	tname = __btf_name_by_offset(btf_vmlinux, t->name_off);
 		/* Only allow structure for now, can be relaxed for
 		 * other types later.
 		 */
-		elem_type = btf_type_skip_modifiers(btf_vmlinux,
+		t = btf_type_skip_modifiers(btf_vmlinux, array_elem->type,
-						    array_elem->type, NULL);
+					    NULL);
-		if (!btf_type_is_struct(elem_type))
+		if (!btf_type_is_struct(t))
 			goto error;
-		off = (off - moff) % elem_type->size;
+		off = (off - moff) % t->size;
-		return btf_struct_access(log, elem_type, off, size, atype,
+		goto again;
-					 next_btf_id);
 error:
 		bpf_log(log, "access beyond struct %s at off %u size %u\n",
 			 */
 			if (off <= moff &&
 			    BITS_ROUNDUP_BYTES(end_bit) <= off + size)
-				return SCALAR_VALUE;
+				return WALK_SCALAR;
 			/* off may be accessing a following member
 			 *
 			break;
 		/* type of the field */
+		mid = member->type;
 		mtype = btf_type_by_id(btf_vmlinux, member->type);
 		mname = __btf_name_by_offset(btf_vmlinux, member->name_off);
-		mtype = btf_resolve_size(btf_vmlinux, mtype, &msize,
+		mtype = __btf_resolve_size(btf_vmlinux, mtype, &msize,
-					 &elem_type, &total_nelems);
+					   &elem_type, &elem_id, &total_nelems,
+					   &mid);
 		if (IS_ERR(mtype)) {
 			bpf_log(log, "field %s doesn't have size\n", mname);
 			return -EFAULT;
 		if (btf_type_is_array(mtype)) {
 			u32 elem_idx;
-			/* btf_resolve_size() above helps to
+			/* __btf_resolve_size() above helps to
 			 * linearize a multi-dimensional array.
 			 *
 			 * The logic here is treating an array
 			elem_idx = (off - moff) / msize;
 			moff += elem_idx * msize;
 			mtype = elem_type;
+			mid = elem_id;
 		}
 		/* the 'off' we're looking for is either equal to start
 			/* our field must be inside that union or struct */
 			t = mtype;
+			/* return if the offset matches the member offset */
+			if (off == moff) {
+				*next_btf_id = mid;
+				return WALK_STRUCT;
+			}
 			/* adjust offset we're looking for */
 			off -= moff;
 			goto again;
 					mname, moff, tname, off, size);
 				return -EACCES;
 			}
 			stype = btf_type_skip_modifiers(btf_vmlinux, mtype->type, &id);
 			if (btf_type_is_struct(stype)) {
 				*next_btf_id = id;
-				return PTR_TO_BTF_ID;
+				return WALK_PTR;
 			}
 		}
 			return -EACCES;
 		}
-		return SCALAR_VALUE;
+		return WALK_SCALAR;
 	}
 	bpf_log(log, "struct %s doesn't have field at offset %d\n", tname, off);
 	return -EINVAL;
 }
+int btf_struct_access(struct bpf_verifier_log *log,
+		      const struct btf_type *t, int off, int size,
+		      enum bpf_access_type atype __maybe_unused,
+		      u32 *next_btf_id)
+{
+	int err;
+	u32 id;
+	do {
+		err = btf_struct_walk(log, t, off, size, &id);
+		switch (err) {
+		case WALK_PTR:
+			/* If we found the pointer or scalar on t+off,
+			 * we're done.
+			 */
+			*next_btf_id = id;
+			return PTR_TO_BTF_ID;
+		case WALK_SCALAR:
+			return SCALAR_VALUE;
+		case WALK_STRUCT:
+			/* We found nested struct, so continue the search
+			 * by diving in it. At this point the offset is
+			 * aligned with the new type, so set it to 0.
+			 */
+			t = btf_type_by_id(btf_vmlinux, id);
+			off = 0;
+			break;
+		default:
+			/* It's either error or unknown return value..
+			 * scream and leave.
+			 */
+			if (WARN_ONCE(err > 0, "unknown btf_struct_walk return value"))
+				return -EINVAL;
+			return err;
+		}
+	} while (t);
+	return -EINVAL;
+}
+bool btf_struct_ids_match(struct bpf_verifier_log *log,
+			  int off, u32 id, u32 need_type_id)
+{
+	const struct btf_type *type;
+	int err;
+	/* Are we already done? */
+	if (need_type_id == id && off == 0)
+		return true;
+again:
+	type = btf_type_by_id(btf_vmlinux, id);
+	if (!type)
+		return false;
+	err = btf_struct_walk(log, type, off, 1, &id);
+	if (err != WALK_STRUCT)
+		return false;
+	/* We found nested struct object. If it matches
+	 * the requested ID, we're done. Otherwise let's
+	 * continue the search with offset 0 in the new
+	 * type.
+	 */
+	if (need_type_id != id) {
+		off = 0;
+		goto again;
+	}
+	return true;
+}
 int btf_resolve_helper_id(struct bpf_verifier_log *log,
 			  const struct bpf_func_proto *fn, int arg)
 {
 {
 	return btf->id;
 }
+static int btf_id_cmp_func(const void *a, const void *b)
+{
+	const int *pa = a, *pb = b;
+	return *pa - *pb;
+}
+bool btf_id_set_contains(struct btf_id_set *set, u32 id)
+{
+	return bsearch(&id, set->ids, set->cnt, sizeof(u32), btf_id_cmp_func) != NULL;
+}

kernel/bpf/cpumap.c+4−8

 static DEFINE_PER_CPU(struct list_head, cpu_map_flush_list);
-static int bq_flush_to_queue(struct xdp_bulk_queue *bq);
 static struct bpf_map *cpu_map_alloc(union bpf_attr *attr)
 {
 	u32 value_size = attr->value_size;
 static int cpu_map_btf_id;
 const struct bpf_map_ops cpu_map_ops = {
+	.map_meta_equal		= bpf_map_meta_equal,
 	.map_alloc		= cpu_map_alloc,
 	.map_free		= cpu_map_free,
 	.map_delete_elem	= cpu_map_delete_elem,
 	.map_btf_id		= &cpu_map_btf_id,
 };
-static int bq_flush_to_queue(struct xdp_bulk_queue *bq)
+static void bq_flush_to_queue(struct xdp_bulk_queue *bq)
 {
 	struct bpf_cpu_map_entry *rcpu = bq->obj;
 	unsigned int processed = 0, drops = 0;
 	int i;
 	if (unlikely(!bq->count))
-		return 0;
+		return;
 	q = rcpu->queue;
 	spin_lock(&q->producer_lock);
 	/* Feedback loop via tracepoints */
 	trace_xdp_cpumap_enqueue(rcpu->map_id, processed, drops, to_cpu);
-	return 0;
 }
 /* Runs under RCU-read-side, plus in softirq under NAPI protection.
  * Thus, safe percpu variable access.
  */
-static int bq_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf)
+static void bq_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf)
 {
 	struct list_head *flush_list = this_cpu_ptr(&cpu_map_flush_list);
 	struct xdp_bulk_queue *bq = this_cpu_ptr(rcpu->bulkq);
 	if (!bq->flush_node.prev)
 		list_add(&bq->flush_node, flush_list);
-	return 0;
 }
 int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp,

kernel/bpf/devmap.c+9−8

 	return false;
 }
-static int bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
+static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
 {
 	struct net_device *dev = bq->dev;
 	int sent = 0, drops = 0, err = 0;
 	int i;
 	if (unlikely(!bq->count))
-		return 0;
+		return;
 	for (i = 0; i < bq->count; i++) {
 		struct xdp_frame *xdpf = bq->q[i];
 	trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err);
 	bq->dev_rx = NULL;
 	__list_del_clearprev(&bq->flush_node);
-	return 0;
+	return;
 error:
 	/* If ndo_xdp_xmit fails with an errno, no frames have been
 	 * xmit'ed and it's our responsibility to them free all.
 /* Runs under RCU-read-side, plus in softirq under NAPI protection.
  * Thus, safe percpu variable access.
  */
-static int bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf,
+static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf,
-		      struct net_device *dev_rx)
+		       struct net_device *dev_rx)
 {
 	struct list_head *flush_list = this_cpu_ptr(&dev_flush_list);
 	struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq);
 	if (!bq->flush_node.prev)
 		list_add(&bq->flush_node, flush_list);
-	return 0;
 }
 static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp,
 	if (unlikely(!xdpf))
 		return -EOVERFLOW;
-	return bq_enqueue(dev, xdpf, dev_rx);
+	bq_enqueue(dev, xdpf, dev_rx);
+	return 0;
 }
 static struct xdp_buff *dev_map_run_prog(struct net_device *dev,
 static int dev_map_btf_id;
 const struct bpf_map_ops dev_map_ops = {
+	.map_meta_equal = bpf_map_meta_equal,
 	.map_alloc = dev_map_alloc,
 	.map_free = dev_map_free,
 	.map_get_next_key = dev_map_get_next_key,
 static int dev_map_hash_map_btf_id;
 const struct bpf_map_ops dev_map_hash_ops = {
+	.map_meta_equal = bpf_map_meta_equal,
 	.map_alloc = dev_map_alloc,
 	.map_free = dev_map_free,
 	.map_get_next_key = dev_map_hash_get_next_key,

kernel/bpf/hashtab.c+10−6

 #include <linux/rculist_nulls.h>
 #include <linux/random.h>
 #include <uapi/linux/btf.h>
+#include <linux/rcupdate_trace.h>
 #include "percpu_freelist.h"
 #include "bpf_lru_list.h"
 #include "map_in_map.h"
 	struct htab_elem *l;
 	u32 hash, key_size;
-	/* Must be called with rcu_read_lock. */
+	WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
-	WARN_ON_ONCE(!rcu_read_lock_held());
 	key_size = map->key_size;
 		/* unknown flags */
 		return -EINVAL;
-	WARN_ON_ONCE(!rcu_read_lock_held());
+	WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
 	key_size = map->key_size;
 		/* unknown flags */
 		return -EINVAL;
-	WARN_ON_ONCE(!rcu_read_lock_held());
+	WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
 	key_size = map->key_size;
 	u32 hash, key_size;
 	int ret = -ENOENT;
-	WARN_ON_ONCE(!rcu_read_lock_held());
+	WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
 	key_size = map->key_size;
 	u32 hash, key_size;
 	int ret = -ENOENT;
-	WARN_ON_ONCE(!rcu_read_lock_held());
+	WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
 	key_size = map->key_size;
 static int htab_map_btf_id;
 const struct bpf_map_ops htab_map_ops = {
+	.map_meta_equal = bpf_map_meta_equal,
 	.map_alloc_check = htab_map_alloc_check,
 	.map_alloc = htab_map_alloc,
 	.map_free = htab_map_free,
 static int htab_lru_map_btf_id;
 const struct bpf_map_ops htab_lru_map_ops = {
+	.map_meta_equal = bpf_map_meta_equal,
 	.map_alloc_check = htab_map_alloc_check,
 	.map_alloc = htab_map_alloc,
 	.map_free = htab_map_free,
 static int htab_percpu_map_btf_id;
 const struct bpf_map_ops htab_percpu_map_ops = {
+	.map_meta_equal = bpf_map_meta_equal,
 	.map_alloc_check = htab_map_alloc_check,
 	.map_alloc = htab_map_alloc,
 	.map_free = htab_map_free,
 static int htab_lru_percpu_map_btf_id;
 const struct bpf_map_ops htab_lru_percpu_map_ops = {
+	.map_meta_equal = bpf_map_meta_equal,
 	.map_alloc_check = htab_map_alloc_check,
 	.map_alloc = htab_map_alloc,
 	.map_free = htab_map_free,

kernel/bpf/helpers.c+22−0

@@ -601,6 +601,28 @@ const struct bpf_func_proto bpf_event_output_data_proto = { · 600 unmodified lines
601	.arg5_type = ARG_CONST_SIZE_OR_ZERO,	601	.arg5_type = ARG_CONST_SIZE_OR_ZERO,
602	};	602	};
603		603
		604	BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
		605	const void __user *, user_ptr)
		606	{
		607	int ret = copy_from_user(dst, user_ptr, size);
		608
		609	if (unlikely(ret)) {
		610	memset(dst, 0, size);
		611	ret = -EFAULT;
		612	}
		613
		614	return ret;
		615	}
		616
		617	const struct bpf_func_proto bpf_copy_from_user_proto = {
		618	.func = bpf_copy_from_user,
		619	.gpl_only = false,
		620	.ret_type = RET_INTEGER,
		621	.arg1_type = ARG_PTR_TO_UNINIT_MEM,
		622	.arg2_type = ARG_CONST_SIZE_OR_ZERO,
		623	.arg3_type = ARG_ANYTHING,
		624	};
		625
604	const struct bpf_func_proto bpf_get_current_task_proto __weak;	626	const struct bpf_func_proto bpf_get_current_task_proto __weak;
605	const struct bpf_func_proto bpf_probe_read_user_proto __weak;	627	const struct bpf_func_proto bpf_probe_read_user_proto __weak;
606	const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;	628	const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;

kernel/bpf/inode.c+113−3

@@ -20,6 +20,7 @@ · 19 unmodified lines
20	#include <linux/filter.h>	20	#include <linux/filter.h>
21	#include <linux/bpf.h>	21	#include <linux/bpf.h>
22	#include <linux/bpf_trace.h>	22	#include <linux/bpf_trace.h>
		23	#include "preload/bpf_preload.h"
23		24
24	enum bpf_type {	25	enum bpf_type {
25	BPF_TYPE_UNSPEC = 0,	26	BPF_TYPE_UNSPEC = 0,
@@ -369,9 +370,10 @@ static struct dentry * · 343 unmodified lines
369	bpf_lookup(struct inode dir, struct dentry dentry, unsigned flags)	370	bpf_lookup(struct inode dir, struct dentry dentry, unsigned flags)
370	{	371	{
371	/* Dots in names (e.g. "/sys/fs/bpf/foo.bar") are reserved for future	372	/* Dots in names (e.g. "/sys/fs/bpf/foo.bar") are reserved for future
372	* extensions.	373	* extensions. That allows popoulate_bpffs() create special files.
373	*/	374	*/
374	if (strchr(dentry->d_name.name, '.'))	375	if ((dir->i_mode & S_IALLUGO) &&
		376	strchr(dentry->d_name.name, '.'))
375	return ERR_PTR(-EPERM);	377	return ERR_PTR(-EPERM);
376		378
377	return simple_lookup(dir, dentry, flags);	379	return simple_lookup(dir, dentry, flags);
@@ -409,6 +411,27 @@ static const struct inode_operations bpf_dir_iops = { · 31 unmodified lines
409	.unlink = simple_unlink,	411	.unlink = simple_unlink,
410	};	412	};
411		413
		414	/* pin iterator link into bpffs */
		415	static int bpf_iter_link_pin_kernel(struct dentry *parent,
		416	const char name, struct bpf_link link)
		417	{
		418	umode_t mode = S_IFREG \| S_IRUSR;
		419	struct dentry *dentry;
		420	int ret;
		421
		422	inode_lock(parent->d_inode);
		423	dentry = lookup_one_len(name, parent, strlen(name));
		424	if (IS_ERR(dentry)) {
		425	inode_unlock(parent->d_inode);
		426	return PTR_ERR(dentry);
		427	}
		428	ret = bpf_mkobj_ops(dentry, mode, link, &bpf_link_iops,
		429	&bpf_iter_fops);
		430	dput(dentry);
		431	inode_unlock(parent->d_inode);
		432	return ret;
		433	}
		434
412	static int bpf_obj_do_pin(const char __user pathname, void raw,	435	static int bpf_obj_do_pin(const char __user pathname, void raw,
413	enum bpf_type type)	436	enum bpf_type type)
414	{	437	{
@@ -638,6 +661,91 @@ static int bpf_parse_param(struct fs_context fc, struct fs_parameter param) · 223 unmodified lines
638	return 0;	661	return 0;
639	}	662	}
640		663
		664	struct bpf_preload_ops *bpf_preload_ops;
		665	EXPORT_SYMBOL_GPL(bpf_preload_ops);
		666
		667	static bool bpf_preload_mod_get(void)
		668	{
		669	/* If bpf_preload.ko wasn't loaded earlier then load it now.
		670	* When bpf_preload is built into vmlinux the module's __init
		671	* function will populate it.
		672	*/
		673	if (!bpf_preload_ops) {
		674	request_module("bpf_preload");
		675	if (!bpf_preload_ops)
		676	return false;
		677	}
		678	/* And grab the reference, so the module doesn't disappear while the
		679	* kernel is interacting with the kernel module and its UMD.
		680	*/
		681	if (!try_module_get(bpf_preload_ops->owner)) {
		682	pr_err("bpf_preload module get failed.\n");
		683	return false;
		684	}
		685	return true;
		686	}
		687
		688	static void bpf_preload_mod_put(void)
		689	{
		690	if (bpf_preload_ops)
		691	/* now user can "rmmod bpf_preload" if necessary */
		692	module_put(bpf_preload_ops->owner);
		693	}
		694
		695	static DEFINE_MUTEX(bpf_preload_lock);
		696
		697	static int populate_bpffs(struct dentry *parent)
		698	{
		699	struct bpf_preload_info objs[BPF_PRELOAD_LINKS] = {};
		700	struct bpf_link *links[BPF_PRELOAD_LINKS] = {};
		701	int err = 0, i;
		702
		703	/* grab the mutex to make sure the kernel interactions with bpf_preload
		704	* UMD are serialized
		705	*/
		706	mutex_lock(&bpf_preload_lock);
		707
		708	/* if bpf_preload.ko wasn't built into vmlinux then load it */
		709	if (!bpf_preload_mod_get())
		710	goto out;
		711
		712	if (!bpf_preload_ops->info.tgid) {
		713	/* preload() will start UMD that will load BPF iterator programs */
		714	err = bpf_preload_ops->preload(objs);
		715	if (err)
		716	goto out_put;
		717	for (i = 0; i < BPF_PRELOAD_LINKS; i++) {
		718	links[i] = bpf_link_by_id(objs[i].link_id);
		719	if (IS_ERR(links[i])) {
		720	err = PTR_ERR(links[i]);
		721	goto out_put;
		722	}
		723	}
		724	for (i = 0; i < BPF_PRELOAD_LINKS; i++) {
		725	err = bpf_iter_link_pin_kernel(parent,
		726	objs[i].link_name, links[i]);
		727	if (err)
		728	goto out_put;
		729	/* do not unlink successfully pinned links even
		730	* if later link fails to pin
		731	*/
		732	links[i] = NULL;
		733	}
		734	/* finish() will tell UMD process to exit */
		735	err = bpf_preload_ops->finish();
		736	if (err)
		737	goto out_put;
		738	}
		739	out_put:
		740	bpf_preload_mod_put();
		741	out:
		742	mutex_unlock(&bpf_preload_lock);
		743	for (i = 0; i < BPF_PRELOAD_LINKS && err; i++)
		744	if (!IS_ERR_OR_NULL(links[i]))
		745	bpf_link_put(links[i]);
		746	return err;
		747	}
		748
641	static int bpf_fill_super(struct super_block sb, struct fs_context fc)	749	static int bpf_fill_super(struct super_block sb, struct fs_context fc)
642	{	750	{
643	static const struct tree_descr bpf_rfiles[] = { { "" } };	751	static const struct tree_descr bpf_rfiles[] = { { "" } };
@@ -654,8 +762,8 @@ static int bpf_fill_super(struct super_block sb, struct fs_context fc) · 10 unmodified lines
654	inode = sb->s_root->d_inode;	762	inode = sb->s_root->d_inode;
655	inode->i_op = &bpf_dir_iops;	763	inode->i_op = &bpf_dir_iops;
656	inode->i_mode &= ~S_IALLUGO;	764	inode->i_mode &= ~S_IALLUGO;
		765	populate_bpffs(sb->s_root);
657	inode->i_mode \|= S_ISVTX \| opts->mode;	766	inode->i_mode \|= S_ISVTX \| opts->mode;
658
659	return 0;	767	return 0;
660	}	768	}
661		769
@@ -705,6 +813,8 @@ static int __init bpf_init(void) · 43 unmodified lines
705	{	813	{
706	int ret;	814	int ret;
707		815
		816	mutex_init(&bpf_preload_lock);
		817
708	ret = sysfs_create_mount_point(fs_kobj, "bpf");	818	ret = sysfs_create_mount_point(fs_kobj, "bpf");
709	if (ret)	819	if (ret)
710	return ret;	820	return ret;

kernel/bpf/lpm_trie.c+1−0

@@ -732,6 +732,7 @@ static int trie_check_btf(const struct bpf_map *map, · 731 unmodified lines
732		732
733	static int trie_map_btf_id;	733	static int trie_map_btf_id;
734	const struct bpf_map_ops trie_map_ops = {	734	const struct bpf_map_ops trie_map_ops = {
		735	.map_meta_equal = bpf_map_meta_equal,
735	.map_alloc = trie_alloc,	736	.map_alloc = trie_alloc,
736	.map_free = trie_free,	737	.map_free = trie_free,
737	.map_get_next_key = trie_get_next_key,	738	.map_get_next_key = trie_get_next_key,

kernel/bpf/map_in_map.c+9−15

 	if (IS_ERR(inner_map))
 		return inner_map;
-	/* prog_array->aux->{type,jited} is a runtime binding.
-	 * Doing static check alone in the verifier is not enough.
-	 */
-	if (inner_map->map_type == BPF_MAP_TYPE_PROG_ARRAY ||
-	    inner_map->map_type == BPF_MAP_TYPE_CGROUP_STORAGE ||
-	    inner_map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE ||
-	    inner_map->map_type == BPF_MAP_TYPE_STRUCT_OPS) {
-		fdput(f);
-		return ERR_PTR(-ENOTSUPP);
-	}
 	/* Does not support >1 level map-in-map */
 	if (inner_map->inner_map_meta) {
 		fdput(f);
 		return ERR_PTR(-EINVAL);
 	}
+	if (!inner_map->ops->map_meta_equal) {
+		fdput(f);
+		return ERR_PTR(-ENOTSUPP);
+	}
 	if (map_value_has_spin_lock(inner_map)) {
 		fdput(f);
 		return ERR_PTR(-ENOTSUPP);
 	return meta0->map_type == meta1->map_type &&
 		meta0->key_size == meta1->key_size &&
 		meta0->value_size == meta1->value_size &&
-		meta0->map_flags == meta1->map_flags &&
+		meta0->map_flags == meta1->map_flags;
-		meta0->max_entries == meta1->max_entries;
 }
 void *bpf_map_fd_get_ptr(struct bpf_map *map,
 			 struct file *map_file /* not used */,
 			 int ufd)
 {
-	struct bpf_map *inner_map;
+	struct bpf_map *inner_map, *inner_map_meta;
 	struct fd f;
 	f = fdget(ufd);
 	if (IS_ERR(inner_map))
 		return inner_map;
-	if (bpf_map_meta_equal(map->inner_map_meta, inner_map))
+	inner_map_meta = map->inner_map_meta;
+	if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map))
 		bpf_map_inc(inner_map);
 	else
 		inner_map = ERR_PTR(-EINVAL);

kernel/bpf/map_in_map.h+0−2

@@ -11,8 +11,6 @@ struct bpf_map; · 10 unmodified lines
11		11
12	struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd);	12	struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd);
13	void bpf_map_meta_free(struct bpf_map *map_meta);	13	void bpf_map_meta_free(struct bpf_map *map_meta);
14	bool bpf_map_meta_equal(const struct bpf_map *meta0,
15	const struct bpf_map *meta1);
16	void bpf_map_fd_get_ptr(struct bpf_map map, struct file *map_file,	14	void bpf_map_fd_get_ptr(struct bpf_map map, struct file *map_file,
17	int ufd);	15	int ufd);
18	void bpf_map_fd_put_ptr(void *ptr);	16	void bpf_map_fd_put_ptr(void *ptr);

kernel/bpf/map_iter.c+15−0

@@ -149,6 +149,19 @@ static void bpf_iter_detach_map(struct bpf_iter_aux_info *aux) · 148 unmodified lines
149	bpf_map_put_with_uref(aux->map);	149	bpf_map_put_with_uref(aux->map);
150	}	150	}
151		151
		152	void bpf_iter_map_show_fdinfo(const struct bpf_iter_aux_info *aux,
		153	struct seq_file *seq)
		154	{
		155	seq_printf(seq, "map_id:\t%u\n", aux->map->id);
		156	}
		157
		158	int bpf_iter_map_fill_link_info(const struct bpf_iter_aux_info *aux,
		159	struct bpf_link_info *info)
		160	{
		161	info->iter.map.map_id = aux->map->id;
		162	return 0;
		163	}
		164
152	DEFINE_BPF_ITER_FUNC(bpf_map_elem, struct bpf_iter_meta *meta,	165	DEFINE_BPF_ITER_FUNC(bpf_map_elem, struct bpf_iter_meta *meta,
153	struct bpf_map map, void key, void *value)	166	struct bpf_map map, void key, void *value)
154		167
@@ -156,6 +169,8 @@ static const struct bpf_iter_reg bpf_map_elem_reg_info = { · 1 unmodified line
156	.target = "bpf_map_elem",	169	.target = "bpf_map_elem",
157	.attach_target = bpf_iter_attach_map,	170	.attach_target = bpf_iter_attach_map,
158	.detach_target = bpf_iter_detach_map,	171	.detach_target = bpf_iter_detach_map,
		172	.show_fdinfo = bpf_iter_map_show_fdinfo,
		173	.fill_link_info = bpf_iter_map_fill_link_info,
159	.ctx_arg_info_size = 2,	174	.ctx_arg_info_size = 2,
160	.ctx_arg_info = {	175	.ctx_arg_info = {
161	{ offsetof(struct bpf_iter__bpf_map_elem, key),	176	{ offsetof(struct bpf_iter__bpf_map_elem, key),

kernel/bpf/preload/Kconfig+26−0

@@ -0,0 +1,26 @@
	1	# SPDX-License-Identifier: GPL-2.0-only
	2	config USERMODE_DRIVER
	3	bool
	4	default n
	5
	6	menuconfig BPF_PRELOAD
	7	bool "Preload BPF file system with kernel specific program and map iterators"
	8	depends on BPF
	9	# The dependency on !COMPILE_TEST prevents it from being enabled
	10	# in allmodconfig or allyesconfig configurations
	11	depends on !COMPILE_TEST
	12	select USERMODE_DRIVER
	13	help
	14	This builds kernel module with several embedded BPF programs that are
	15	pinned into BPF FS mount point as human readable files that are
	16	useful in debugging and introspection of BPF programs and maps.
	17
	18	if BPF_PRELOAD
	19	config BPF_PRELOAD_UMD
	20	tristate "bpf_preload kernel module with user mode driver"
	21	depends on CC_CAN_LINK
	22	depends on m \|\| CC_CAN_LINK_STATIC
	23	default m
	24	help
	25	This builds bpf_preload kernel module with embedded user mode driver.
	26	endif

kernel/bpf/preload/Makefile+23−0

@@ -0,0 +1,23 @@
	1	# SPDX-License-Identifier: GPL-2.0
	2
	3	LIBBPF_SRCS = $(srctree)/tools/lib/bpf/
	4	LIBBPF_A = $(obj)/libbpf.a
	5	LIBBPF_OUT = $(abspath $(obj))
	6
	7	$(LIBBPF_A):
	8	$(Q)$(MAKE) -C $(LIBBPF_SRCS) OUTPUT=$(LIBBPF_OUT)/ $(LIBBPF_OUT)/libbpf.a
	9
	10	userccflags += -I $(srctree)/tools/include/ -I $(srctree)/tools/include/uapi \
	11	-I $(srctree)/tools/lib/ -Wno-unused-result
	12
	13	userprogs := bpf_preload_umd
	14
	15	bpf_preload_umd-objs := iterators/iterators.o
	16	bpf_preload_umd-userldlibs := $(LIBBPF_A) -lelf -lz
	17
	18	$(obj)/bpf_preload_umd: $(LIBBPF_A)
	19
	20	$(obj)/bpf_preload_umd_blob.o: $(obj)/bpf_preload_umd
	21
	22	obj-$(CONFIG_BPF_PRELOAD_UMD) += bpf_preload.o
	23	bpf_preload-objs += bpf_preload_kern.o bpf_preload_umd_blob.o

kernel/bpf/preload/bpf_preload.h+16−0

@@ -0,0 +1,16 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef _BPF_PRELOAD_H
	3	#define _BPF_PRELOAD_H
	4
	5	#include <linux/usermode_driver.h>
	6	#include "iterators/bpf_preload_common.h"
	7
	8	struct bpf_preload_ops {
	9	struct umd_info info;
	10	int (preload)(struct bpf_preload_info );
	11	int (*finish)(void);
	12	struct module *owner;
	13	};
	14	extern struct bpf_preload_ops *bpf_preload_ops;
	15	#define BPF_PRELOAD_LINKS 2
	16	#endif

kernel/bpf/preload/bpf_preload_kern.c+91−0

@@ -0,0 +1,91 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
	3	#include <linux/init.h>
	4	#include <linux/module.h>
	5	#include <linux/pid.h>
	6	#include <linux/fs.h>
	7	#include <linux/sched/signal.h>
	8	#include "bpf_preload.h"
	9
	10	extern char bpf_preload_umd_start;
	11	extern char bpf_preload_umd_end;
	12
	13	static int preload(struct bpf_preload_info *obj);
	14	static int finish(void);
	15
	16	static struct bpf_preload_ops umd_ops = {
	17	.info.driver_name = "bpf_preload",
	18	.preload = preload,
	19	.finish = finish,
	20	.owner = THIS_MODULE,
	21	};
	22
	23	static int preload(struct bpf_preload_info *obj)
	24	{
	25	int magic = BPF_PRELOAD_START;
	26	loff_t pos = 0;
	27	int i, err;
	28	ssize_t n;
	29
	30	err = fork_usermode_driver(&umd_ops.info);
	31	if (err)
	32	return err;
	33
	34	/* send the start magic to let UMD proceed with loading BPF progs */
	35	n = kernel_write(umd_ops.info.pipe_to_umh,
	36	&magic, sizeof(magic), &pos);
	37	if (n != sizeof(magic))
	38	return -EPIPE;
	39
	40	/* receive bpf_link IDs and names from UMD */
	41	pos = 0;
	42	for (i = 0; i < BPF_PRELOAD_LINKS; i++) {
	43	n = kernel_read(umd_ops.info.pipe_from_umh,
	44	&obj[i], sizeof(*obj), &pos);
	45	if (n != sizeof(*obj))
	46	return -EPIPE;
	47	}
	48	return 0;
	49	}
	50
	51	static int finish(void)
	52	{
	53	int magic = BPF_PRELOAD_END;
	54	struct pid *tgid;
	55	loff_t pos = 0;
	56	ssize_t n;
	57
	58	/* send the last magic to UMD. It will do a normal exit. */
	59	n = kernel_write(umd_ops.info.pipe_to_umh,
	60	&magic, sizeof(magic), &pos);
	61	if (n != sizeof(magic))
	62	return -EPIPE;
	63	tgid = umd_ops.info.tgid;
	64	wait_event(tgid->wait_pidfd, thread_group_exited(tgid));
	65	umd_ops.info.tgid = NULL;
	66	return 0;
	67	}
	68
	69	static int __init load_umd(void)
	70	{
	71	int err;
	72
	73	err = umd_load_blob(&umd_ops.info, &bpf_preload_umd_start,
	74	&bpf_preload_umd_end - &bpf_preload_umd_start);
	75	if (err)
	76	return err;
	77	bpf_preload_ops = &umd_ops;
	78	return err;
	79	}
	80
	81	static void __exit fini_umd(void)
	82	{
	83	bpf_preload_ops = NULL;
	84	/* kill UMD in case it's still there due to earlier error */
	85	kill_pid(umd_ops.info.tgid, SIGKILL, 1);
	86	umd_ops.info.tgid = NULL;
	87	umd_unload_blob(&umd_ops.info);
	88	}
	89	late_initcall(load_umd);
	90	module_exit(fini_umd);
	91	MODULE_LICENSE("GPL");

kernel/bpf/preload/bpf_preload_umd_blob.S+7−0

@@ -0,0 +1,7 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	.section .init.rodata, "a"
	3	.global bpf_preload_umd_start
	4	bpf_preload_umd_start:
	5	.incbin "kernel/bpf/preload/bpf_preload_umd"
	6	.global bpf_preload_umd_end
	7	bpf_preload_umd_end:

kernel/bpf/preload/iterators/.gitignore+2−0

@@ -0,0 +1,2 @@
	1	# SPDX-License-Identifier: GPL-2.0-only
	2	/.output

kernel/bpf/preload/iterators/Makefile+57−0

@@ -0,0 +1,57 @@
	1	# SPDX-License-Identifier: GPL-2.0
	2	OUTPUT := .output
	3	CLANG ?= clang
	4	LLC ?= llc
	5	LLVM_STRIP ?= llvm-strip
	6	DEFAULT_BPFTOOL := $(OUTPUT)/sbin/bpftool
	7	BPFTOOL ?= $(DEFAULT_BPFTOOL)
	8	LIBBPF_SRC := $(abspath ../../../../tools/lib/bpf)
	9	BPFOBJ := $(OUTPUT)/libbpf.a
	10	BPF_INCLUDE := $(OUTPUT)
	11	INCLUDES := -I$(OUTPUT) -I$(BPF_INCLUDE) -I$(abspath ../../../../tools/lib) \
	12	-I$(abspath ../../../../tools/include/uapi)
	13	CFLAGS := -g -Wall
	14
	15	abs_out := $(abspath $(OUTPUT))
	16	ifeq ($(V),1)
	17	Q =
	18	msg =
	19	else
	20	Q = @
	21	msg = @printf ' %-8s %s%s\n' "$(1)" "$(notdir $(2))" "$(if $(3), $(3))";
	22	MAKEFLAGS += --no-print-directory
	23	submake_extras := feature_display=0
	24	endif
	25
	26	.DELETE_ON_ERROR:
	27
	28	.PHONY: all clean
	29
	30	all: iterators.skel.h
	31
	32	clean:
	33	$(call msg,CLEAN)
	34	$(Q)rm -rf $(OUTPUT) iterators
	35
	36	iterators.skel.h: $(OUTPUT)/iterators.bpf.o \| $(BPFTOOL)
	37	$(call msg,GEN-SKEL,$@)
	38	$(Q)$(BPFTOOL) gen skeleton $< > $@
	39
	40
	41	$(OUTPUT)/iterators.bpf.o: iterators.bpf.c $(BPFOBJ) \| $(OUTPUT)
	42	$(call msg,BPF,$@)
	43	$(Q)$(CLANG) -g -O2 -target bpf $(INCLUDES) \
	44	-c $(filter %.c,$^) -o $@ && \
	45	$(LLVM_STRIP) -g $@
	46
	47	$(OUTPUT):
	48	$(call msg,MKDIR,$@)
	49	$(Q)mkdir -p $(OUTPUT)
	50
	51	$(BPFOBJ): $(wildcard $(LIBBPF_SRC)/*.[ch] $(LIBBPF_SRC)/Makefile) \| $(OUTPUT)
	52	$(Q)$(MAKE) $(submake_extras) -C $(LIBBPF_SRC) \
	53	OUTPUT=$(abspath $(dir $@))/ $(abspath $@)
	54
	55	$(DEFAULT_BPFTOOL):
	56	$(Q)$(MAKE) $(submake_extras) -C ../../../../tools/bpf/bpftool \
	57	prefix= OUTPUT=$(abs_out)/ DESTDIR=$(abs_out) install

kernel/bpf/preload/iterators/README+4−0

@@ -0,0 +1,4 @@
	1	WARNING:
	2	If you change "iterators.bpf.c" do "make -j" in this directory to rebuild "iterators.skel.h".
	3	Make sure to have clang 10 installed.
	4	See Documentation/bpf/bpf_devel_QA.rst

kernel/bpf/preload/iterators/bpf_preload_common.h+13−0

@@ -0,0 +1,13 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef _BPF_PRELOAD_COMMON_H
	3	#define _BPF_PRELOAD_COMMON_H
	4
	5	#define BPF_PRELOAD_START 0x5555
	6	#define BPF_PRELOAD_END 0xAAAA
	7
	8	struct bpf_preload_info {
	9	char link_name[16];
	10	int link_id;
	11	};
	12
	13	#endif

kernel/bpf/preload/iterators/iterators.bpf.c+114−0

@@ -0,0 +1,114 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	/* Copyright (c) 2020 Facebook */
	3	#include <linux/bpf.h>
	4	#include <bpf/bpf_helpers.h>
	5	#include <bpf/bpf_tracing.h>
	6	#include <bpf/bpf_core_read.h>
	7
	8	#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
	9	struct seq_file;
	10	struct bpf_iter_meta {
	11	struct seq_file *seq;
	12	__u64 session_id;
	13	__u64 seq_num;
	14	};
	15
	16	struct bpf_map {
	17	__u32 id;
	18	char name[16];
	19	__u32 max_entries;
	20	};
	21
	22	struct bpf_iter__bpf_map {
	23	struct bpf_iter_meta *meta;
	24	struct bpf_map *map;
	25	};
	26
	27	struct btf_type {
	28	__u32 name_off;
	29	};
	30
	31	struct btf_header {
	32	__u32 str_len;
	33	};
	34
	35	struct btf {
	36	const char *strings;
	37	struct btf_type **types;
	38	struct btf_header hdr;
	39	};
	40
	41	struct bpf_prog_aux {
	42	__u32 id;
	43	char name[16];
	44	const char *attach_func_name;
	45	struct bpf_prog *linked_prog;
	46	struct bpf_func_info *func_info;
	47	struct btf *btf;
	48	};
	49
	50	struct bpf_prog {
	51	struct bpf_prog_aux *aux;
	52	};
	53
	54	struct bpf_iter__bpf_prog {
	55	struct bpf_iter_meta *meta;
	56	struct bpf_prog *prog;
	57	};
	58	#pragma clang attribute pop
	59
	60	static const char get_name(struct btf btf, long btf_id, const char *fallback)
	61	{
	62	struct btf_type *types, t;
	63	unsigned int name_off;
	64	const char *str;
	65
	66	if (!btf)
	67	return fallback;
	68	str = btf->strings;
	69	types = btf->types;
	70	bpf_probe_read_kernel(&t, sizeof(t), types + btf_id);
	71	name_off = BPF_CORE_READ(t, name_off);
	72	if (name_off >= btf->hdr.str_len)
	73	return fallback;
	74	return str + name_off;
	75	}
	76
	77	SEC("iter/bpf_map")
	78	int dump_bpf_map(struct bpf_iter__bpf_map *ctx)
	79	{
	80	struct seq_file *seq = ctx->meta->seq;
	81	__u64 seq_num = ctx->meta->seq_num;
	82	struct bpf_map *map = ctx->map;
	83
	84	if (!map)
	85	return 0;
	86
	87	if (seq_num == 0)
	88	BPF_SEQ_PRINTF(seq, " id name max_entries\n");
	89
	90	BPF_SEQ_PRINTF(seq, "%4u %-16s%6d\n", map->id, map->name, map->max_entries);
	91	return 0;
	92	}
	93
	94	SEC("iter/bpf_prog")
	95	int dump_bpf_prog(struct bpf_iter__bpf_prog *ctx)
	96	{
	97	struct seq_file *seq = ctx->meta->seq;
	98	__u64 seq_num = ctx->meta->seq_num;
	99	struct bpf_prog *prog = ctx->prog;
	100	struct bpf_prog_aux *aux;
	101
	102	if (!prog)
	103	return 0;
	104
	105	aux = prog->aux;
	106	if (seq_num == 0)
	107	BPF_SEQ_PRINTF(seq, " id name attached\n");
	108
	109	BPF_SEQ_PRINTF(seq, "%4u %-16s %s %s\n", aux->id,
	110	get_name(aux->btf, aux->func_info[0].type_id, aux->name),
	111	aux->attach_func_name, aux->linked_prog->aux->name);
	112	return 0;
	113	}
	114	char LICENSE[] SEC("license") = "GPL";

kernel/bpf/preload/iterators/iterators.c+94−0

@@ -0,0 +1,94 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	/* Copyright (c) 2020 Facebook */
	3	#include <argp.h>
	4	#include <stdio.h>
	5	#include <stdlib.h>
	6	#include <string.h>
	7	#include <unistd.h>
	8	#include <fcntl.h>
	9	#include <sys/resource.h>
	10	#include <bpf/libbpf.h>
	11	#include <bpf/bpf.h>
	12	#include <sys/mount.h>
	13	#include "iterators.skel.h"
	14	#include "bpf_preload_common.h"
	15
	16	int to_kernel = -1;
	17	int from_kernel = 0;
	18
	19	static int send_link_to_kernel(struct bpf_link link, const char link_name)
	20	{
	21	struct bpf_preload_info obj = {};
	22	struct bpf_link_info info = {};
	23	__u32 info_len = sizeof(info);
	24	int err;
	25
	26	err = bpf_obj_get_info_by_fd(bpf_link__fd(link), &info, &info_len);
	27	if (err)
	28	return err;
	29	obj.link_id = info.id;
	30	if (strlen(link_name) >= sizeof(obj.link_name))
	31	return -E2BIG;
	32	strcpy(obj.link_name, link_name);
	33	if (write(to_kernel, &obj, sizeof(obj)) != sizeof(obj))
	34	return -EPIPE;
	35	return 0;
	36	}
	37
	38	int main(int argc, char **argv)
	39	{
	40	struct rlimit rlim = { RLIM_INFINITY, RLIM_INFINITY };
	41	struct iterators_bpf *skel;
	42	int err, magic;
	43	int debug_fd;
	44
	45	debug_fd = open("/dev/console", O_WRONLY \| O_NOCTTY \| O_CLOEXEC);
	46	if (debug_fd < 0)
	47	return 1;
	48	to_kernel = dup(1);
	49	close(1);
	50	dup(debug_fd);
	51	/* now stdin and stderr point to /dev/console */
	52
	53	read(from_kernel, &magic, sizeof(magic));
	54	if (magic != BPF_PRELOAD_START) {
	55	printf("bad start magic %d\n", magic);
	56	return 1;
	57	}
	58	setrlimit(RLIMIT_MEMLOCK, &rlim);
	59	/* libbpf opens BPF object and loads it into the kernel */
	60	skel = iterators_bpf__open_and_load();
	61	if (!skel) {
	62	/* iterators.skel.h is little endian.
	63	* libbpf doesn't support automatic little->big conversion
	64	* of BPF bytecode yet.
	65	* The program load will fail in such case.
	66	*/
	67	printf("Failed load could be due to wrong endianness\n");
	68	return 1;
	69	}
	70	err = iterators_bpf__attach(skel);
	71	if (err)
	72	goto cleanup;
	73
	74	/* send two bpf_link IDs with names to the kernel */
	75	err = send_link_to_kernel(skel->links.dump_bpf_map, "maps.debug");
	76	if (err)
	77	goto cleanup;
	78	err = send_link_to_kernel(skel->links.dump_bpf_prog, "progs.debug");
	79	if (err)
	80	goto cleanup;
	81
	82	/* The kernel will proceed with pinnging the links in bpffs.
	83	* UMD will wait on read from pipe.
	84	*/
	85	read(from_kernel, &magic, sizeof(magic));
	86	if (magic != BPF_PRELOAD_END) {
	87	printf("bad final magic %d\n", magic);
	88	err = -EINVAL;
	89	}
	90	cleanup:
	91	iterators_bpf__destroy(skel);
	92
	93	return err != 0;
	94	}

kernel/bpf/preload/iterators/iterators.skel.h+410−0

@@ -0,0 +1,410 @@
	1	/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
	2
	3	/* THIS FILE IS AUTOGENERATED! */
	4	#ifndef __ITERATORS_BPF_SKEL_H__
	5	#define __ITERATORS_BPF_SKEL_H__
	6
	7	#include <stdlib.h>
	8	#include <bpf/libbpf.h>
	9
	10	struct iterators_bpf {
	11	struct bpf_object_skeleton *skeleton;
	12	struct bpf_object *obj;
	13	struct {
	14	struct bpf_map *rodata;
	15	} maps;
	16	struct {
	17	struct bpf_program *dump_bpf_map;
	18	struct bpf_program *dump_bpf_prog;
	19	} progs;
	20	struct {
	21	struct bpf_link *dump_bpf_map;
	22	struct bpf_link *dump_bpf_prog;
	23	} links;
	24	struct iterators_bpf__rodata {
	25	char dump_bpf_map____fmt[35];
	26	char dump_bpf_map____fmt_1[14];
	27	char dump_bpf_prog____fmt[32];
	28	char dump_bpf_prog____fmt_2[17];
	29	} *rodata;
	30	};
	31
	32	static void
	33	iterators_bpf__destroy(struct iterators_bpf *obj)
	34	{
	35	if (!obj)
	36	return;
	37	if (obj->skeleton)
	38	bpf_object__destroy_skeleton(obj->skeleton);
	39	free(obj);
	40	}
	41
	42	static inline int
	43	iterators_bpf__create_skeleton(struct iterators_bpf *obj);
	44
	45	static inline struct iterators_bpf *
	46	iterators_bpf__open_opts(const struct bpf_object_open_opts *opts)
	47	{
	48	struct iterators_bpf *obj;
	49
	50	obj = (typeof(obj))calloc(1, sizeof(*obj));
	51	if (!obj)
	52	return NULL;
	53	if (iterators_bpf__create_skeleton(obj))
	54	goto err;
	55	if (bpf_object__open_skeleton(obj->skeleton, opts))
	56	goto err;
	57
	58	return obj;
	59	err:
	60	iterators_bpf__destroy(obj);
	61	return NULL;
	62	}
	63
	64	static inline struct iterators_bpf *
	65	iterators_bpf__open(void)
	66	{
	67	return iterators_bpf__open_opts(NULL);
	68	}
	69
	70	static inline int
	71	iterators_bpf__load(struct iterators_bpf *obj)
	72	{
	73	return bpf_object__load_skeleton(obj->skeleton);
	74	}
	75
	76	static inline struct iterators_bpf *
	77	iterators_bpf__open_and_load(void)
	78	{
	79	struct iterators_bpf *obj;
	80
	81	obj = iterators_bpf__open();
	82	if (!obj)
	83	return NULL;
	84	if (iterators_bpf__load(obj)) {
	85	iterators_bpf__destroy(obj);
	86	return NULL;
	87	}
	88	return obj;
	89	}
	90
	91	static inline int
	92	iterators_bpf__attach(struct iterators_bpf *obj)
	93	{
	94	return bpf_object__attach_skeleton(obj->skeleton);
	95	}
	96
	97	static inline void
	98	iterators_bpf__detach(struct iterators_bpf *obj)
	99	{
	100	return bpf_object__detach_skeleton(obj->skeleton);
	101	}
	102
	103	static inline int
	104	iterators_bpf__create_skeleton(struct iterators_bpf *obj)
	105	{
	106	struct bpf_object_skeleton *s;
	107
	108	s = (typeof(s))calloc(1, sizeof(*s));
	109	if (!s)
	110	return -1;
	111	obj->skeleton = s;
	112
	113	s->sz = sizeof(*s);
	114	s->name = "iterators_bpf";
	115	s->obj = &obj->obj;
	116
	117	/* maps */
	118	s->map_cnt = 1;
	119	s->map_skel_sz = sizeof(*s->maps);
	120	s->maps = (typeof(s->maps))calloc(s->map_cnt, s->map_skel_sz);
	121	if (!s->maps)
	122	goto err;
	123
	124	s->maps[0].name = "iterator.rodata";
	125	s->maps[0].map = &obj->maps.rodata;
	126	s->maps[0].mmaped = (void **)&obj->rodata;
	127
	128	/* programs */
	129	s->prog_cnt = 2;
	130	s->prog_skel_sz = sizeof(*s->progs);
	131	s->progs = (typeof(s->progs))calloc(s->prog_cnt, s->prog_skel_sz);
	132	if (!s->progs)
	133	goto err;
	134
	135	s->progs[0].name = "dump_bpf_map";
	136	s->progs[0].prog = &obj->progs.dump_bpf_map;
	137	s->progs[0].link = &obj->links.dump_bpf_map;
	138
	139	s->progs[1].name = "dump_bpf_prog";
	140	s->progs[1].prog = &obj->progs.dump_bpf_prog;
	141	s->progs[1].link = &obj->links.dump_bpf_prog;
	142
	143	s->data_sz = 7128;
	144	s->data = (void *)"\
	145	\x7f\x45\x4c\x46\x02\x01\x01\0\0\0\0\0\0\0\0\0\x01\0\xf7\0\x01\0\0\0\0\0\0\0\0\
	146	\0\0\0\0\0\0\0\0\0\0\0\x18\x18\0\0\0\0\0\0\0\0\0\0\x40\0\0\0\0\0\x40\0\x0f\0\
	147	\x0e\0\x79\x12\0\0\0\0\0\0\x79\x26\0\0\0\0\0\0\x79\x17\x08\0\0\0\0\0\x15\x07\
	148	\x1a\0\0\0\0\0\x79\x21\x10\0\0\0\0\0\x55\x01\x08\0\0\0\0\0\xbf\xa4\0\0\0\0\0\0\
	149	\x07\x04\0\0\xe8\xff\xff\xff\xbf\x61\0\0\0\0\0\0\x18\x02\0\0\0\0\0\0\0\0\0\0\0\
	150	\0\0\0\xb7\x03\0\0\x23\0\0\0\xb7\x05\0\0\0\0\0\0\x85\0\0\0\x7e\0\0\0\x61\x71\0\
	151	\0\0\0\0\0\x7b\x1a\xe8\xff\0\0\0\0\xb7\x01\0\0\x04\0\0\0\xbf\x72\0\0\0\0\0\0\
	152	\x0f\x12\0\0\0\0\0\0\x7b\x2a\xf0\xff\0\0\0\0\x61\x71\x14\0\0\0\0\0\x7b\x1a\xf8\
	153	\xff\0\0\0\0\xbf\xa4\0\0\0\0\0\0\x07\x04\0\0\xe8\xff\xff\xff\xbf\x61\0\0\0\0\0\
	154	\0\x18\x02\0\0\x23\0\0\0\0\0\0\0\0\0\0\0\xb7\x03\0\0\x0e\0\0\0\xb7\x05\0\0\x18\
	155	\0\0\0\x85\0\0\0\x7e\0\0\0\xb7\0\0\0\0\0\0\0\x95\0\0\0\0\0\0\0\x79\x12\0\0\0\0\
	156	\0\0\x79\x26\0\0\0\0\0\0\x79\x11\x08\0\0\0\0\0\x15\x01\x3b\0\0\0\0\0\x79\x17\0\
	157	\0\0\0\0\0\x79\x21\x10\0\0\0\0\0\x55\x01\x08\0\0\0\0\0\xbf\xa4\0\0\0\0\0\0\x07\
	158	\x04\0\0\xd0\xff\xff\xff\xbf\x61\0\0\0\0\0\0\x18\x02\0\0\x31\0\0\0\0\0\0\0\0\0\
	159	\0\0\xb7\x03\0\0\x20\0\0\0\xb7\x05\0\0\0\0\0\0\x85\0\0\0\x7e\0\0\0\x7b\x6a\xc8\
	160	\xff\0\0\0\0\x61\x71\0\0\0\0\0\0\x7b\x1a\xd0\xff\0\0\0\0\xb7\x03\0\0\x04\0\0\0\
	161	\xbf\x79\0\0\0\0\0\0\x0f\x39\0\0\0\0\0\0\x79\x71\x28\0\0\0\0\0\x79\x78\x30\0\0\
	162	\0\0\0\x15\x08\x18\0\0\0\0\0\xb7\x02\0\0\0\0\0\0\x0f\x21\0\0\0\0\0\0\x61\x11\
	163	\x04\0\0\0\0\0\x79\x83\x08\0\0\0\0\0\x67\x01\0\0\x03\0\0\0\x0f\x13\0\0\0\0\0\0\
	164	\x79\x86\0\0\0\0\0\0\xbf\xa1\0\0\0\0\0\0\x07\x01\0\0\xf8\xff\xff\xff\xb7\x02\0\
	165	\0\x08\0\0\0\x85\0\0\0\x71\0\0\0\xb7\x01\0\0\0\0\0\0\x79\xa3\xf8\xff\0\0\0\0\
	166	\x0f\x13\0\0\0\0\0\0\xbf\xa1\0\0\0\0\0\0\x07\x01\0\0\xf4\xff\xff\xff\xb7\x02\0\
	167	\0\x04\0\0\0\x85\0\0\0\x04\0\0\0\xb7\x03\0\0\x04\0\0\0\x61\xa1\xf4\xff\0\0\0\0\
	168	\x61\x82\x10\0\0\0\0\0\x3d\x21\x02\0\0\0\0\0\x0f\x16\0\0\0\0\0\0\xbf\x69\0\0\0\
	169	\0\0\0\x7b\x9a\xd8\xff\0\0\0\0\x79\x71\x18\0\0\0\0\0\x7b\x1a\xe0\xff\0\0\0\0\
	170	\x79\x71\x20\0\0\0\0\0\x79\x11\0\0\0\0\0\0\x0f\x31\0\0\0\0\0\0\x7b\x1a\xe8\xff\
	171	\0\0\0\0\xbf\xa4\0\0\0\0\0\0\x07\x04\0\0\xd0\xff\xff\xff\x79\xa1\xc8\xff\0\0\0\
	172	\0\x18\x02\0\0\x51\0\0\0\0\0\0\0\0\0\0\0\xb7\x03\0\0\x11\0\0\0\xb7\x05\0\0\x20\
	173	\0\0\0\x85\0\0\0\x7e\0\0\0\xb7\0\0\0\0\0\0\0\x95\0\0\0\0\0\0\0\x20\x20\x69\x64\
	174	\x20\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x6d\
	175	\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\x0a\0\x25\x34\x75\x20\x25\x2d\x31\x36\
	176	\x73\x25\x36\x64\x0a\0\x20\x20\x69\x64\x20\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\
	177	\x20\x20\x20\x20\x20\x20\x20\x20\x61\x74\x74\x61\x63\x68\x65\x64\x0a\0\x25\x34\
	178	\x75\x20\x25\x2d\x31\x36\x73\x20\x25\x73\x20\x25\x73\x0a\0\x47\x50\x4c\0\x9f\
	179	\xeb\x01\0\x18\0\0\0\0\0\0\0\x1c\x04\0\0\x1c\x04\0\0\0\x05\0\0\0\0\0\0\0\0\0\
	180	\x02\x02\0\0\0\x01\0\0\0\x02\0\0\x04\x10\0\0\0\x13\0\0\0\x03\0\0\0\0\0\0\0\x18\
	181	\0\0\0\x04\0\0\0\x40\0\0\0\0\0\0\0\0\0\0\x02\x08\0\0\0\0\0\0\0\0\0\0\x02\x0d\0\
	182	\0\0\0\0\0\0\x01\0\0\x0d\x06\0\0\0\x1c\0\0\0\x01\0\0\0\x20\0\0\0\0\0\0\x01\x04\
	183	\0\0\0\x20\0\0\x01\x24\0\0\0\x01\0\0\x0c\x05\0\0\0\xa3\0\0\0\x03\0\0\x04\x18\0\
	184	\0\0\xb1\0\0\0\x09\0\0\0\0\0\0\0\xb5\0\0\0\x0b\0\0\0\x40\0\0\0\xc0\0\0\0\x0b\0\
	185	\0\0\x80\0\0\0\0\0\0\0\0\0\0\x02\x0a\0\0\0\xc8\0\0\0\0\0\0\x07\0\0\0\0\xd1\0\0\
	186	\0\0\0\0\x08\x0c\0\0\0\xd7\0\0\0\0\0\0\x01\x08\0\0\0\x40\0\0\0\x98\x01\0\0\x03\
	187	\0\0\x04\x18\0\0\0\xa0\x01\0\0\x0e\0\0\0\0\0\0\0\xa3\x01\0\0\x11\0\0\0\x20\0\0\
	188	\0\xa8\x01\0\0\x0e\0\0\0\xa0\0\0\0\xb4\x01\0\0\0\0\0\x08\x0f\0\0\0\xba\x01\0\0\
	189	\0\0\0\x01\x04\0\0\0\x20\0\0\0\xc7\x01\0\0\0\0\0\x01\x01\0\0\0\x08\0\0\x01\0\0\
	190	\0\0\0\0\0\x03\0\0\0\0\x10\0\0\0\x12\0\0\0\x10\0\0\0\xcc\x01\0\0\0\0\0\x01\x04\
	191	\0\0\0\x20\0\0\0\0\0\0\0\0\0\0\x02\x14\0\0\0\x30\x02\0\0\x02\0\0\x04\x10\0\0\0\
	192	\x13\0\0\0\x03\0\0\0\0\0\0\0\x43\x02\0\0\x15\0\0\0\x40\0\0\0\0\0\0\0\0\0\0\x02\
	193	\x18\0\0\0\0\0\0\0\x01\0\0\x0d\x06\0\0\0\x1c\0\0\0\x13\0\0\0\x48\x02\0\0\x01\0\
	194	\0\x0c\x16\0\0\0\x94\x02\0\0\x01\0\0\x04\x08\0\0\0\x9d\x02\0\0\x19\0\0\0\0\0\0\
	195	\0\0\0\0\0\0\0\0\x02\x1a\0\0\0\xee\x02\0\0\x06\0\0\x04\x38\0\0\0\xa0\x01\0\0\
	196	\x0e\0\0\0\0\0\0\0\xa3\x01\0\0\x11\0\0\0\x20\0\0\0\xfb\x02\0\0\x1b\0\0\0\xc0\0\
	197	\0\0\x0c\x03\0\0\x15\0\0\0\0\x01\0\0\x18\x03\0\0\x1d\0\0\0\x40\x01\0\0\x22\x03\
	198	\0\0\x1e\0\0\0\x80\x01\0\0\0\0\0\0\0\0\0\x02\x1c\0\0\0\0\0\0\0\0\0\0\x0a\x10\0\
	199	\0\0\0\0\0\0\0\0\0\x02\x1f\0\0\0\0\0\0\0\0\0\0\x02\x20\0\0\0\x6c\x03\0\0\x02\0\
	200	\0\x04\x08\0\0\0\x7a\x03\0\0\x0e\0\0\0\0\0\0\0\x83\x03\0\0\x0e\0\0\0\x20\0\0\0\
	201	\x22\x03\0\0\x03\0\0\x04\x18\0\0\0\x8d\x03\0\0\x1b\0\0\0\0\0\0\0\x95\x03\0\0\
	202	\x21\0\0\0\x40\0\0\0\x9b\x03\0\0\x23\0\0\0\x80\0\0\0\0\0\0\0\0\0\0\x02\x22\0\0\
	203	\0\0\0\0\0\0\0\0\x02\x24\0\0\0\x9f\x03\0\0\x01\0\0\x04\x04\0\0\0\xaa\x03\0\0\
	204	\x0e\0\0\0\0\0\0\0\x13\x04\0\0\x01\0\0\x04\x04\0\0\0\x1c\x04\0\0\x0e\0\0\0\0\0\
	205	\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\x1c\0\0\0\x12\0\0\0\x23\0\0\0\x92\x04\0\0\0\0\0\
	206	\x0e\x25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\x1c\0\0\0\x12\0\0\0\x0e\0\0\0\
	207	\xa6\x04\0\0\0\0\0\x0e\x27\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\x1c\0\0\0\
	208	\x12\0\0\0\x20\0\0\0\xbc\x04\0\0\0\0\0\x0e\x29\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\
	209	\0\0\0\0\x1c\0\0\0\x12\0\0\0\x11\0\0\0\xd1\x04\0\0\0\0\0\x0e\x2b\0\0\0\0\0\0\0\
	210	\0\0\0\0\0\0\0\x03\0\0\0\0\x10\0\0\0\x12\0\0\0\x04\0\0\0\xe8\x04\0\0\0\0\0\x0e\
	211	\x2d\0\0\0\x01\0\0\0\xf0\x04\0\0\x04\0\0\x0f\0\0\0\0\x26\0\0\0\0\0\0\0\x23\0\0\
	212	\0\x28\0\0\0\x23\0\0\0\x0e\0\0\0\x2a\0\0\0\x31\0\0\0\x20\0\0\0\x2c\0\0\0\x51\0\
	213	\0\0\x11\0\0\0\xf8\x04\0\0\x01\0\0\x0f\0\0\0\0\x2e\0\0\0\0\0\0\0\x04\0\0\0\0\
	214	\x62\x70\x66\x5f\x69\x74\x65\x72\x5f\x5f\x62\x70\x66\x5f\x6d\x61\x70\0\x6d\x65\
	215	\x74\x61\0\x6d\x61\x70\0\x63\x74\x78\0\x69\x6e\x74\0\x64\x75\x6d\x70\x5f\x62\
	216	\x70\x66\x5f\x6d\x61\x70\0\x69\x74\x65\x72\x2f\x62\x70\x66\x5f\x6d\x61\x70\0\
	217	\x30\x3a\x30\0\x2f\x77\x2f\x6e\x65\x74\x2d\x6e\x65\x78\x74\x2f\x6b\x65\x72\x6e\
	218	\x65\x6c\x2f\x62\x70\x66\x2f\x70\x72\x65\x6c\x6f\x61\x64\x2f\x69\x74\x65\x72\
	219	\x61\x74\x6f\x72\x73\x2f\x69\x74\x65\x72\x61\x74\x6f\x72\x73\x2e\x62\x70\x66\
	220	\x2e\x63\0\x09\x73\x74\x72\x75\x63\x74\x20\x73\x65\x71\x5f\x66\x69\x6c\x65\x20\
	221	\x2a\x73\x65\x71\x20\x3d\x20\x63\x74\x78\x2d\x3e\x6d\x65\x74\x61\x2d\x3e\x73\
	222	\x65\x71\x3b\0\x62\x70\x66\x5f\x69\x74\x65\x72\x5f\x6d\x65\x74\x61\0\x73\x65\
	223	\x71\0\x73\x65\x73\x73\x69\x6f\x6e\x5f\x69\x64\0\x73\x65\x71\x5f\x6e\x75\x6d\0\
	224	\x73\x65\x71\x5f\x66\x69\x6c\x65\0\x5f\x5f\x75\x36\x34\0\x6c\x6f\x6e\x67\x20\
	225	\x6c\x6f\x6e\x67\x20\x75\x6e\x73\x69\x67\x6e\x65\x64\x20\x69\x6e\x74\0\x30\x3a\
	226	\x31\0\x09\x73\x74\x72\x75\x63\x74\x20\x62\x70\x66\x5f\x6d\x61\x70\x20\x2a\x6d\
	227	\x61\x70\x20\x3d\x20\x63\x74\x78\x2d\x3e\x6d\x61\x70\x3b\0\x09\x69\x66\x20\x28\
	228	\x21\x6d\x61\x70\x29\0\x30\x3a\x32\0\x09\x5f\x5f\x75\x36\x34\x20\x73\x65\x71\
	229	\x5f\x6e\x75\x6d\x20\x3d\x20\x63\x74\x78\x2d\x3e\x6d\x65\x74\x61\x2d\x3e\x73\
	230	\x65\x71\x5f\x6e\x75\x6d\x3b\0\x09\x69\x66\x20\x28\x73\x65\x71\x5f\x6e\x75\x6d\
	231	\x20\x3d\x3d\x20\x30\x29\0\x09\x09\x42\x50\x46\x5f\x53\x45\x51\x5f\x50\x52\x49\
	232	\x4e\x54\x46\x28\x73\x65\x71\x2c\x20\x22\x20\x20\x69\x64\x20\x6e\x61\x6d\x65\
	233	\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x6d\x61\x78\x5f\x65\x6e\
	234	\x74\x72\x69\x65\x73\x5c\x6e\x22\x29\x3b\0\x62\x70\x66\x5f\x6d\x61\x70\0\x69\
	235	\x64\0\x6e\x61\x6d\x65\0\x6d\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\0\x5f\x5f\
	236	\x75\x33\x32\0\x75\x6e\x73\x69\x67\x6e\x65\x64\x20\x69\x6e\x74\0\x63\x68\x61\
	237	\x72\0\x5f\x5f\x41\x52\x52\x41\x59\x5f\x53\x49\x5a\x45\x5f\x54\x59\x50\x45\x5f\
	238	\x5f\0\x09\x42\x50\x46\x5f\x53\x45\x51\x5f\x50\x52\x49\x4e\x54\x46\x28\x73\x65\
	239	\x71\x2c\x20\x22\x25\x34\x75\x20\x25\x2d\x31\x36\x73\x25\x36\x64\x5c\x6e\x22\
	240	\x2c\x20\x6d\x61\x70\x2d\x3e\x69\x64\x2c\x20\x6d\x61\x70\x2d\x3e\x6e\x61\x6d\
	241	\x65\x2c\x20\x6d\x61\x70\x2d\x3e\x6d\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\
	242	\x29\x3b\0\x7d\0\x62\x70\x66\x5f\x69\x74\x65\x72\x5f\x5f\x62\x70\x66\x5f\x70\
	243	\x72\x6f\x67\0\x70\x72\x6f\x67\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\
	244	\x6f\x67\0\x69\x74\x65\x72\x2f\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x09\x73\x74\
	245	\x72\x75\x63\x74\x20\x62\x70\x66\x5f\x70\x72\x6f\x67\x20\x2a\x70\x72\x6f\x67\
	246	\x20\x3d\x20\x63\x74\x78\x2d\x3e\x70\x72\x6f\x67\x3b\0\x09\x69\x66\x20\x28\x21\
	247	\x70\x72\x6f\x67\x29\0\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x61\x75\x78\0\x09\x61\
	248	\x75\x78\x20\x3d\x20\x70\x72\x6f\x67\x2d\x3e\x61\x75\x78\x3b\0\x09\x09\x42\x50\
	249	\x46\x5f\x53\x45\x51\x5f\x50\x52\x49\x4e\x54\x46\x28\x73\x65\x71\x2c\x20\x22\
	250	\x20\x20\x69\x64\x20\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\
	251	\x20\x20\x20\x61\x74\x74\x61\x63\x68\x65\x64\x5c\x6e\x22\x29\x3b\0\x62\x70\x66\
	252	\x5f\x70\x72\x6f\x67\x5f\x61\x75\x78\0\x61\x74\x74\x61\x63\x68\x5f\x66\x75\x6e\
	253	\x63\x5f\x6e\x61\x6d\x65\0\x6c\x69\x6e\x6b\x65\x64\x5f\x70\x72\x6f\x67\0\x66\
	254	\x75\x6e\x63\x5f\x69\x6e\x66\x6f\0\x62\x74\x66\0\x09\x42\x50\x46\x5f\x53\x45\
	255	\x51\x5f\x50\x52\x49\x4e\x54\x46\x28\x73\x65\x71\x2c\x20\x22\x25\x34\x75\x20\
	256	\x25\x2d\x31\x36\x73\x20\x25\x73\x20\x25\x73\x5c\x6e\x22\x2c\x20\x61\x75\x78\
	257	\x2d\x3e\x69\x64\x2c\0\x30\x3a\x34\0\x30\x3a\x35\0\x09\x69\x66\x20\x28\x21\x62\
	258	\x74\x66\x29\0\x62\x70\x66\x5f\x66\x75\x6e\x63\x5f\x69\x6e\x66\x6f\0\x69\x6e\
	259	\x73\x6e\x5f\x6f\x66\x66\0\x74\x79\x70\x65\x5f\x69\x64\0\x30\0\x73\x74\x72\x69\
	260	\x6e\x67\x73\0\x74\x79\x70\x65\x73\0\x68\x64\x72\0\x62\x74\x66\x5f\x68\x65\x61\
	261	\x64\x65\x72\0\x73\x74\x72\x5f\x6c\x65\x6e\0\x09\x74\x79\x70\x65\x73\x20\x3d\
	262	\x20\x62\x74\x66\x2d\x3e\x74\x79\x70\x65\x73\x3b\0\x09\x62\x70\x66\x5f\x70\x72\
	263	\x6f\x62\x65\x5f\x72\x65\x61\x64\x5f\x6b\x65\x72\x6e\x65\x6c\x28\x26\x74\x2c\
	264	\x20\x73\x69\x7a\x65\x6f\x66\x28\x74\x29\x2c\x20\x74\x79\x70\x65\x73\x20\x2b\
	265	\x20\x62\x74\x66\x5f\x69\x64\x29\x3b\0\x09\x73\x74\x72\x20\x3d\x20\x62\x74\x66\
	266	\x2d\x3e\x73\x74\x72\x69\x6e\x67\x73\x3b\0\x62\x74\x66\x5f\x74\x79\x70\x65\0\
	267	\x6e\x61\x6d\x65\x5f\x6f\x66\x66\0\x09\x6e\x61\x6d\x65\x5f\x6f\x66\x66\x20\x3d\
	268	\x20\x42\x50\x46\x5f\x43\x4f\x52\x45\x5f\x52\x45\x41\x44\x28\x74\x2c\x20\x6e\
	269	\x61\x6d\x65\x5f\x6f\x66\x66\x29\x3b\0\x30\x3a\x32\x3a\x30\0\x09\x69\x66\x20\
	270	\x28\x6e\x61\x6d\x65\x5f\x6f\x66\x66\x20\x3e\x3d\x20\x62\x74\x66\x2d\x3e\x68\
	271	\x64\x72\x2e\x73\x74\x72\x5f\x6c\x65\x6e\x29\0\x09\x72\x65\x74\x75\x72\x6e\x20\
	272	\x73\x74\x72\x20\x2b\x20\x6e\x61\x6d\x65\x5f\x6f\x66\x66\x3b\0\x30\x3a\x33\0\
	273	\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\0\
	274	\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\
	275	\x2e\x31\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\x2e\x5f\x5f\x5f\
	276	\x66\x6d\x74\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\x2e\x5f\x5f\
	277	\x5f\x66\x6d\x74\x2e\x32\0\x4c\x49\x43\x45\x4e\x53\x45\0\x2e\x72\x6f\x64\x61\
	278	\x74\x61\0\x6c\x69\x63\x65\x6e\x73\x65\0\x9f\xeb\x01\0\x20\0\0\0\0\0\0\0\x24\0\
	279	\0\0\x24\0\0\0\x44\x02\0\0\x68\x02\0\0\xa4\x01\0\0\x08\0\0\0\x31\0\0\0\x01\0\0\
	280	\0\0\0\0\0\x07\0\0\0\x56\x02\0\0\x01\0\0\0\0\0\0\0\x17\0\0\0\x10\0\0\0\x31\0\0\
	281	\0\x09\0\0\0\0\0\0\0\x42\0\0\0\x7b\0\0\0\x1e\x40\x01\0\x08\0\0\0\x42\0\0\0\x7b\
	282	\0\0\0\x24\x40\x01\0\x10\0\0\0\x42\0\0\0\xf2\0\0\0\x1d\x48\x01\0\x18\0\0\0\x42\
	283	\0\0\0\x13\x01\0\0\x06\x50\x01\0\x20\0\0\0\x42\0\0\0\x22\x01\0\0\x1d\x44\x01\0\
	284	\x28\0\0\0\x42\0\0\0\x47\x01\0\0\x06\x5c\x01\0\x38\0\0\0\x42\0\0\0\x5a\x01\0\0\
	285	\x03\x60\x01\0\x70\0\0\0\x42\0\0\0\xe0\x01\0\0\x02\x68\x01\0\xf0\0\0\0\x42\0\0\
	286	\0\x2e\x02\0\0\x01\x70\x01\0\x56\x02\0\0\x1a\0\0\0\0\0\0\0\x42\0\0\0\x7b\0\0\0\
	287	\x1e\x84\x01\0\x08\0\0\0\x42\0\0\0\x7b\0\0\0\x24\x84\x01\0\x10\0\0\0\x42\0\0\0\
	288	\x64\x02\0\0\x1f\x8c\x01\0\x18\0\0\0\x42\0\0\0\x88\x02\0\0\x06\x98\x01\0\x20\0\
	289	\0\0\x42\0\0\0\xa1\x02\0\0\x0e\xa4\x01\0\x28\0\0\0\x42\0\0\0\x22\x01\0\0\x1d\
	290	\x88\x01\0\x30\0\0\0\x42\0\0\0\x47\x01\0\0\x06\xa8\x01\0\x40\0\0\0\x42\0\0\0\
	291	\xb3\x02\0\0\x03\xac\x01\0\x80\0\0\0\x42\0\0\0\x26\x03\0\0\x02\xb4\x01\0\xb8\0\
	292	\0\0\x42\0\0\0\x61\x03\0\0\x06\x08\x01\0\xd0\0\0\0\x42\0\0\0\0\0\0\0\0\0\0\0\
	293	\xd8\0\0\0\x42\0\0\0\xb2\x03\0\0\x0f\x14\x01\0\xe0\0\0\0\x42\0\0\0\xc7\x03\0\0\
	294	\x2d\x18\x01\0\xf0\0\0\0\x42\0\0\0\xfe\x03\0\0\x0d\x10\x01\0\0\x01\0\0\x42\0\0\
	295	\0\0\0\0\0\0\0\0\0\x08\x01\0\0\x42\0\0\0\xc7\x03\0\0\x02\x18\x01\0\x20\x01\0\0\
	296	\x42\0\0\0\x25\x04\0\0\x0d\x1c\x01\0\x38\x01\0\0\x42\0\0\0\0\0\0\0\0\0\0\0\x40\
	297	\x01\0\0\x42\0\0\0\x25\x04\0\0\x0d\x1c\x01\0\x58\x01\0\0\x42\0\0\0\x25\x04\0\0\
	298	\x0d\x1c\x01\0\x60\x01\0\0\x42\0\0\0\x53\x04\0\0\x1b\x20\x01\0\x68\x01\0\0\x42\
	299	\0\0\0\x53\x04\0\0\x06\x20\x01\0\x70\x01\0\0\x42\0\0\0\x76\x04\0\0\x0d\x28\x01\
	300	\0\x78\x01\0\0\x42\0\0\0\0\0\0\0\0\0\0\0\x80\x01\0\0\x42\0\0\0\x26\x03\0\0\x02\
	301	\xb4\x01\0\xf8\x01\0\0\x42\0\0\0\x2e\x02\0\0\x01\xc4\x01\0\x10\0\0\0\x31\0\0\0\
	302	\x07\0\0\0\0\0\0\0\x02\0\0\0\x3e\0\0\0\0\0\0\0\x08\0\0\0\x08\0\0\0\x3e\0\0\0\0\
	303	\0\0\0\x10\0\0\0\x02\0\0\0\xee\0\0\0\0\0\0\0\x20\0\0\0\x08\0\0\0\x1e\x01\0\0\0\
	304	\0\0\0\x70\0\0\0\x0d\0\0\0\x3e\0\0\0\0\0\0\0\x80\0\0\0\x0d\0\0\0\xee\0\0\0\0\0\
	305	\0\0\xa0\0\0\0\x0d\0\0\0\x1e\x01\0\0\0\0\0\0\x56\x02\0\0\x12\0\0\0\0\0\0\0\x14\
	306	\0\0\0\x3e\0\0\0\0\0\0\0\x08\0\0\0\x08\0\0\0\x3e\0\0\0\0\0\0\0\x10\0\0\0\x14\0\
	307	\0\0\xee\0\0\0\0\0\0\0\x20\0\0\0\x18\0\0\0\x3e\0\0\0\0\0\0\0\x28\0\0\0\x08\0\0\
	308	\0\x1e\x01\0\0\0\0\0\0\x80\0\0\0\x1a\0\0\0\x3e\0\0\0\0\0\0\0\x90\0\0\0\x1a\0\0\
	309	\0\xee\0\0\0\0\0\0\0\xa8\0\0\0\x1a\0\0\0\x59\x03\0\0\0\0\0\0\xb0\0\0\0\x1a\0\0\
	310	\0\x5d\x03\0\0\0\0\0\0\xc0\0\0\0\x1f\0\0\0\x8b\x03\0\0\0\0\0\0\xd8\0\0\0\x20\0\
	311	\0\0\xee\0\0\0\0\0\0\0\xf0\0\0\0\x20\0\0\0\x3e\0\0\0\0\0\0\0\x18\x01\0\0\x24\0\
	312	\0\0\x3e\0\0\0\0\0\0\0\x50\x01\0\0\x1a\0\0\0\xee\0\0\0\0\0\0\0\x60\x01\0\0\x20\
	313	\0\0\0\x4d\x04\0\0\0\0\0\0\x88\x01\0\0\x1a\0\0\0\x1e\x01\0\0\0\0\0\0\x98\x01\0\
	314	\0\x1a\0\0\0\x8e\x04\0\0\0\0\0\0\xa0\x01\0\0\x18\0\0\0\x3e\0\0\0\0\0\0\0\0\0\0\
	315	\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xd6\0\0\0\0\0\x02\0\x70\0\0\0\0\
	316	\0\0\0\0\0\0\0\0\0\0\0\xc8\0\0\0\0\0\x02\0\xf0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
	317	\xcf\0\0\0\0\0\x03\0\x78\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xc1\0\0\0\0\0\x03\0\x80\
	318	\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xba\0\0\0\0\0\x03\0\xf8\x01\0\0\0\0\0\0\0\0\0\
	319	\0\0\0\0\0\x14\0\0\0\x01\0\x04\0\0\0\0\0\0\0\0\0\x23\0\0\0\0\0\0\0\xf4\0\0\0\
	320	\x01\0\x04\0\x23\0\0\0\0\0\0\0\x0e\0\0\0\0\0\0\0\x28\0\0\0\x01\0\x04\0\x31\0\0\
	321	\0\0\0\0\0\x20\0\0\0\0\0\0\0\xdd\0\0\0\x01\0\x04\0\x51\0\0\0\0\0\0\0\x11\0\0\0\
	322	\0\0\0\0\0\0\0\0\x03\0\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\x03\
	323	\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\x04\0\0\0\0\0\0\0\0\0\0\0\0\0\
	324	\0\0\0\0\xb2\0\0\0\x11\0\x05\0\0\0\0\0\0\0\0\0\x04\0\0\0\0\0\0\0\x3d\0\0\0\x12\
	325	\0\x02\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\x5b\0\0\0\x12\0\x03\0\0\0\0\0\0\0\0\
	326	\0\x08\x02\0\0\0\0\0\0\x48\0\0\0\0\0\0\0\x01\0\0\0\x0c\0\0\0\xc8\0\0\0\0\0\0\0\
	327	\x01\0\0\0\x0c\0\0\0\x50\0\0\0\0\0\0\0\x01\0\0\0\x0c\0\0\0\xd0\x01\0\0\0\0\0\0\
	328	\x01\0\0\0\x0c\0\0\0\xf0\x03\0\0\0\0\0\0\x0a\0\0\0\x0c\0\0\0\xfc\x03\0\0\0\0\0\
	329	\0\x0a\0\0\0\x0c\0\0\0\x08\x04\0\0\0\0\0\0\x0a\0\0\0\x0c\0\0\0\x14\x04\0\0\0\0\
	330	\0\0\x0a\0\0\0\x0c\0\0\0\x2c\x04\0\0\0\0\0\0\0\0\0\0\x0d\0\0\0\x2c\0\0\0\0\0\0\
	331	\0\0\0\0\0\x0a\0\0\0\x3c\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x50\0\0\0\0\0\0\0\0\0\
	332	\0\0\x0a\0\0\0\x60\0\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\x70\0\0\0\0\0\0\0\0\0\0\0\
	333	\x0a\0\0\0\x80\0\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\x90\0\0\0\0\0\0\0\0\0\0\0\x0a\0\
	334	\0\0\xa0\0\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\xb0\0\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\
	335	\xc0\0\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\xd0\0\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\xe8\0\
	336	\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xf8\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x08\x01\0\0\
	337	\0\0\0\0\0\0\0\0\x0b\0\0\0\x18\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x28\x01\0\0\0\
	338	\0\0\0\0\0\0\0\x0b\0\0\0\x38\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x48\x01\0\0\0\0\
	339	\0\0\0\0\0\0\x0b\0\0\0\x58\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x68\x01\0\0\0\0\0\
	340	\0\0\0\0\0\x0b\0\0\0\x78\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x88\x01\0\0\0\0\0\0\
	341	\0\0\0\0\x0b\0\0\0\x98\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xa8\x01\0\0\0\0\0\0\0\
	342	\0\0\0\x0b\0\0\0\xb8\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xc8\x01\0\0\0\0\0\0\0\0\
	343	\0\0\x0b\0\0\0\xd8\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xe8\x01\0\0\0\0\0\0\0\0\0\
	344	\0\x0b\0\0\0\xf8\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x08\x02\0\0\0\0\0\0\0\0\0\0\
	345	\x0b\0\0\0\x18\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x28\x02\0\0\0\0\0\0\0\0\0\0\
	346	\x0b\0\0\0\x38\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x48\x02\0\0\0\0\0\0\0\0\0\0\
	347	\x0b\0\0\0\x58\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x68\x02\0\0\0\0\0\0\0\0\0\0\
	348	\x0b\0\0\0\x78\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x94\x02\0\0\0\0\0\0\0\0\0\0\
	349	\x0a\0\0\0\xa4\x02\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\xb4\x02\0\0\0\0\0\0\0\0\0\0\
	350	\x0a\0\0\0\xc4\x02\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\xd4\x02\0\0\0\0\0\0\0\0\0\0\
	351	\x0a\0\0\0\xe4\x02\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\xf4\x02\0\0\0\0\0\0\0\0\0\0\
	352	\x0a\0\0\0\x0c\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x1c\x03\0\0\0\0\0\0\0\0\0\0\
	353	\x0b\0\0\0\x2c\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x3c\x03\0\0\0\0\0\0\0\0\0\0\
	354	\x0b\0\0\0\x4c\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x5c\x03\0\0\0\0\0\0\0\0\0\0\
	355	\x0b\0\0\0\x6c\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x7c\x03\0\0\0\0\0\0\0\0\0\0\
	356	\x0b\0\0\0\x8c\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x9c\x03\0\0\0\0\0\0\0\0\0\0\
	357	\x0b\0\0\0\xac\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xbc\x03\0\0\0\0\0\0\0\0\0\0\
	358	\x0b\0\0\0\xcc\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xdc\x03\0\0\0\0\0\0\0\0\0\0\
	359	\x0b\0\0\0\xec\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xfc\x03\0\0\0\0\0\0\0\0\0\0\
	360	\x0b\0\0\0\x0c\x04\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x1c\x04\0\0\0\0\0\0\0\0\0\0\
	361	\x0b\0\0\0\x4e\x4f\x41\x42\x43\x44\x4d\0\x2e\x74\x65\x78\x74\0\x2e\x72\x65\x6c\
	362	\x2e\x42\x54\x46\x2e\x65\x78\x74\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\x61\
	363	\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\
	364	\x6f\x67\x2e\x5f\x5f\x5f\x66\x6d\x74\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\
	365	\x61\x70\0\x2e\x72\x65\x6c\x69\x74\x65\x72\x2f\x62\x70\x66\x5f\x6d\x61\x70\0\
	366	\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x2e\x72\x65\x6c\x69\x74\
	367	\x65\x72\x2f\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x2e\x6c\x6c\x76\x6d\x5f\x61\x64\
	368	\x64\x72\x73\x69\x67\0\x6c\x69\x63\x65\x6e\x73\x65\0\x2e\x73\x74\x72\x74\x61\
	369	\x62\0\x2e\x73\x79\x6d\x74\x61\x62\0\x2e\x72\x6f\x64\x61\x74\x61\0\x2e\x72\x65\
	370	\x6c\x2e\x42\x54\x46\0\x4c\x49\x43\x45\x4e\x53\x45\0\x4c\x42\x42\x31\x5f\x37\0\
	371	\x4c\x42\x42\x31\x5f\x36\0\x4c\x42\x42\x30\x5f\x34\0\x4c\x42\x42\x31\x5f\x33\0\
	372	\x4c\x42\x42\x30\x5f\x33\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\
	373	\x2e\x5f\x5f\x5f\x66\x6d\x74\x2e\x32\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\
	374	\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\x2e\x31\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
	375	\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
	376	\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\x01\0\0\0\x06\0\0\0\0\0\0\0\0\0\0\0\
	377	\0\0\0\0\x40\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x04\0\0\0\0\0\0\0\0\
	378	\0\0\0\0\0\0\0\x4e\0\0\0\x01\0\0\0\x06\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x40\0\0\0\
	379	\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x08\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
	380	\x6d\0\0\0\x01\0\0\0\x06\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x40\x01\0\0\0\0\0\0\x08\
	381	\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x08\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xa1\0\0\0\
	382	\x01\0\0\0\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x48\x03\0\0\0\0\0\0\x62\0\0\0\0\0\
	383	\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x89\0\0\0\x01\0\0\0\x03\
	384	\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xaa\x03\0\0\0\0\0\0\x04\0\0\0\0\0\0\0\0\0\0\0\0\
	385	\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xad\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\
	386	\0\0\0\0\0\0\0\xae\x03\0\0\0\0\0\0\x34\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\
	387	\0\0\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
	388	\xe2\x0c\0\0\0\0\0\0\x2c\x04\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\
	389	\0\0\0\0\0\0\x99\0\0\0\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x10\x11\0\0\0\
	390	\0\0\0\x80\x01\0\0\0\0\0\0\x0e\0\0\0\x0d\0\0\0\x08\0\0\0\0\0\0\0\x18\0\0\0\0\0\
	391	\0\0\x4a\0\0\0\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x90\x12\0\0\0\0\0\0\
	392	\x20\0\0\0\0\0\0\0\x08\0\0\0\x02\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\0\0\0\0\x69\
	393	\0\0\0\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xb0\x12\0\0\0\0\0\0\x20\0\0\0\
	394	\0\0\0\0\x08\0\0\0\x03\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\0\0\0\0\xa9\0\0\0\x09\
	395	\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xd0\x12\0\0\0\0\0\0\x50\0\0\0\0\0\0\0\
	396	\x08\0\0\0\x06\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\0\0\0\0\x07\0\0\0\x09\0\0\0\0\
	397	\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x20\x13\0\0\0\0\0\0\xe0\x03\0\0\0\0\0\0\x08\0\0\
	398	\0\x07\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\0\0\0\0\x7b\0\0\0\x03\x4c\xff\x6f\0\0\
	399	\0\x80\0\0\0\0\0\0\0\0\0\0\0\0\0\x17\0\0\0\0\0\0\x07\0\0\0\0\0\0\0\0\0\0\0\0\0\
	400	\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x91\0\0\0\x03\0\0\0\0\0\0\0\0\0\0\0\0\0\
	401	\0\0\0\0\0\0\x07\x17\0\0\0\0\0\0\x0a\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\
	402	\0\0\0\0\0\0\0\0\0\0\0\0";
	403
	404	return 0;
	405	err:
	406	bpf_object__destroy_skeleton(s);
	407	return -1;
	408	}
	409
	410	#endif /* __ITERATORS_BPF_SKEL_H__ */

kernel/bpf/queue_stack_maps.c+2−0

 static int queue_map_btf_id;
 const struct bpf_map_ops queue_map_ops = {
+	.map_meta_equal = bpf_map_meta_equal,
 	.map_alloc_check = queue_stack_map_alloc_check,
 	.map_alloc = queue_stack_map_alloc,
 	.map_free = queue_stack_map_free,
 static int stack_map_btf_id;
 const struct bpf_map_ops stack_map_ops = {
+	.map_meta_equal = bpf_map_meta_equal,
 	.map_alloc_check = queue_stack_map_alloc_check,
 	.map_alloc = queue_stack_map_alloc,
 	.map_free = queue_stack_map_free,

kernel/bpf/reuseport_array.c+1−0

@@ -351,6 +351,7 @@ static int reuseport_array_get_next_key(struct bpf_map map, void key, · 350 unmodified lines
351		351
352	static int reuseport_array_map_btf_id;	352	static int reuseport_array_map_btf_id;
353	const struct bpf_map_ops reuseport_array_ops = {	353	const struct bpf_map_ops reuseport_array_ops = {
		354	.map_meta_equal = bpf_map_meta_equal,
354	.map_alloc_check = reuseport_array_alloc_check,	355	.map_alloc_check = reuseport_array_alloc_check,
355	.map_alloc = reuseport_array_alloc,	356	.map_alloc = reuseport_array_alloc,
356	.map_free = reuseport_array_free,	357	.map_free = reuseport_array_free,

kernel/bpf/ringbuf.c+1−0

@@ -287,6 +287,7 @@ static __poll_t ringbuf_map_poll(struct bpf_map map, struct file filp, · 286 unmodified lines
287		287
288	static int ringbuf_map_btf_id;	288	static int ringbuf_map_btf_id;
289	const struct bpf_map_ops ringbuf_map_ops = {	289	const struct bpf_map_ops ringbuf_map_ops = {
		290	.map_meta_equal = bpf_map_meta_equal,
290	.map_alloc = ringbuf_map_alloc,	291	.map_alloc = ringbuf_map_alloc,
291	.map_free = ringbuf_map_free,	292	.map_free = ringbuf_map_free,
292	.map_mmap = ringbuf_map_mmap,	293	.map_mmap = ringbuf_map_mmap,

kernel/bpf/stackmap.c+1−0

@@ -839,6 +839,7 @@ static void stack_map_free(struct bpf_map *map) · 838 unmodified lines
839		839
840	static int stack_trace_map_btf_id;	840	static int stack_trace_map_btf_id;
841	const struct bpf_map_ops stack_trace_map_ops = {	841	const struct bpf_map_ops stack_trace_map_ops = {
		842	.map_meta_equal = bpf_map_meta_equal,
842	.map_alloc = stack_map_alloc,	843	.map_alloc = stack_map_alloc,
843	.map_free = stack_map_free,	844	.map_free = stack_map_free,
844	.map_get_next_key = stack_map_get_next_key,	845	.map_get_next_key = stack_map_get_next_key,

kernel/bpf/syscall.c+44−24

@@ -29,6 +29,7 @@ · 28 unmodified lines
29	#include <linux/bpf_lsm.h>	29	#include <linux/bpf_lsm.h>
30	#include <linux/poll.h>	30	#include <linux/poll.h>
31	#include <linux/bpf-netns.h>	31	#include <linux/bpf-netns.h>
		32	#include <linux/rcupdate_trace.h>
32		33
33	#define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY \|\| \	34	#define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY \|\| \
34	(map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY \|\| \	35	(map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY \|\| \
@@ -90,6 +91,7 @@ int bpf_check_uarg_tail_zero(void __user *uaddr, · 55 unmodified lines
90	}	91	}
91		92
92	const struct bpf_map_ops bpf_map_offload_ops = {	93	const struct bpf_map_ops bpf_map_offload_ops = {
		94	.map_meta_equal = bpf_map_meta_equal,
93	.map_alloc = bpf_map_offload_map_alloc,	95	.map_alloc = bpf_map_offload_map_alloc,
94	.map_free = bpf_map_offload_map_free,	96	.map_free = bpf_map_offload_map_free,
95	.map_check_btf = map_check_no_btf,	97	.map_check_btf = map_check_no_btf,
@@ -157,10 +159,11 @@ static int bpf_map_update_value(struct bpf_map map, struct fd f, void key, · 61 unmodified lines
157	if (bpf_map_is_dev_bound(map)) {	159	if (bpf_map_is_dev_bound(map)) {
158	return bpf_map_offload_update_elem(map, key, value, flags);	160	return bpf_map_offload_update_elem(map, key, value, flags);
159	} else if (map->map_type == BPF_MAP_TYPE_CPUMAP \|\|	161	} else if (map->map_type == BPF_MAP_TYPE_CPUMAP \|\|
160	map->map_type == BPF_MAP_TYPE_SOCKHASH \|\|
161	map->map_type == BPF_MAP_TYPE_SOCKMAP \|\|
162	map->map_type == BPF_MAP_TYPE_STRUCT_OPS) {	162	map->map_type == BPF_MAP_TYPE_STRUCT_OPS) {
163	return map->ops->map_update_elem(map, key, value, flags);	163	return map->ops->map_update_elem(map, key, value, flags);
		164	} else if (map->map_type == BPF_MAP_TYPE_SOCKHASH \|\|
		165	map->map_type == BPF_MAP_TYPE_SOCKMAP) {
		166	return sock_map_update_elem_sys(map, key, value, flags);
164	} else if (IS_FD_PROG_ARRAY(map)) {	167	} else if (IS_FD_PROG_ARRAY(map)) {
165	return bpf_fd_array_map_update_elem(map, f.file, key, value,	168	return bpf_fd_array_map_update_elem(map, f.file, key, value,
166	flags);	169	flags);
@@ -768,7 +771,8 @@ static int map_check_btf(struct bpf_map map, const struct btf btf, · 601 unmodified lines
768	if (map->map_type != BPF_MAP_TYPE_HASH &&	771	if (map->map_type != BPF_MAP_TYPE_HASH &&
769	map->map_type != BPF_MAP_TYPE_ARRAY &&	772	map->map_type != BPF_MAP_TYPE_ARRAY &&
770	map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE &&	773	map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE &&
771	map->map_type != BPF_MAP_TYPE_SK_STORAGE)	774	map->map_type != BPF_MAP_TYPE_SK_STORAGE &&
		775	map->map_type != BPF_MAP_TYPE_INODE_STORAGE)
772	return -ENOTSUPP;	776	return -ENOTSUPP;
773	if (map->spin_lock_off + sizeof(struct bpf_spin_lock) >	777	if (map->spin_lock_off + sizeof(struct bpf_spin_lock) >
774	map->value_size) {	778	map->value_size) {
@@ -1728,10 +1732,14 @@ static void __bpf_prog_put_noref(struct bpf_prog *prog, bool deferred) · 953 unmodified lines
1728	btf_put(prog->aux->btf);	1732	btf_put(prog->aux->btf);
1729	bpf_prog_free_linfo(prog);	1733	bpf_prog_free_linfo(prog);
1730		1734
1731	if (deferred)	1735	if (deferred) {
1732	call_rcu(&prog->aux->rcu, __bpf_prog_put_rcu);	1736	if (prog->aux->sleepable)
1733	else	1737	call_rcu_tasks_trace(&prog->aux->rcu, __bpf_prog_put_rcu);
		1738	else
		1739	call_rcu(&prog->aux->rcu, __bpf_prog_put_rcu);
		1740	} else {
1734	__bpf_prog_put_rcu(&prog->aux->rcu);	1741	__bpf_prog_put_rcu(&prog->aux->rcu);
		1742	}
1735	}	1743	}
1736		1744
1737	static void __bpf_prog_put(struct bpf_prog *prog, bool do_idr_lock)	1745	static void __bpf_prog_put(struct bpf_prog *prog, bool do_idr_lock)
@@ -2101,6 +2109,7 @@ static int bpf_prog_load(union bpf_attr attr, union bpf_attr __user uattr) · 363 unmodified lines
2101	if (attr->prog_flags & ~(BPF_F_STRICT_ALIGNMENT \|	2109	if (attr->prog_flags & ~(BPF_F_STRICT_ALIGNMENT \|
2102	BPF_F_ANY_ALIGNMENT \|	2110	BPF_F_ANY_ALIGNMENT \|
2103	BPF_F_TEST_STATE_FREQ \|	2111	BPF_F_TEST_STATE_FREQ \|
		2112	BPF_F_SLEEPABLE \|
2104	BPF_F_TEST_RND_HI32))	2113	BPF_F_TEST_RND_HI32))
2105	return -EINVAL;	2114	return -EINVAL;
2106		2115
@@ -2156,6 +2165,7 @@ static int bpf_prog_load(union bpf_attr attr, union bpf_attr __user uattr) · 49 unmodified lines
2156	}	2165	}
2157		2166
2158	prog->aux->offload_requested = !!attr->prog_ifindex;	2167	prog->aux->offload_requested = !!attr->prog_ifindex;
		2168	prog->aux->sleepable = attr->prog_flags & BPF_F_SLEEPABLE;
2159		2169
2160	err = security_bpf_prog_alloc(prog->aux);	2170	err = security_bpf_prog_alloc(prog->aux);
2161	if (err)	2171	if (err)
@@ -4014,40 +4024,50 @@ static int link_detach(union bpf_attr *attr) · 1852 unmodified lines
4014	return ret;	4024	return ret;
4015	}	4025	}
4016		4026
4017	static int bpf_link_inc_not_zero(struct bpf_link *link)	4027	static struct bpf_link bpf_link_inc_not_zero(struct bpf_link link)
4018	{	4028	{
4019	return atomic64_fetch_add_unless(&link->refcnt, 1, 0) ? 0 : -ENOENT;	4029	return atomic64_fetch_add_unless(&link->refcnt, 1, 0) ? link : ERR_PTR(-ENOENT);
4020	}	4030	}
4021		4031
4022	#define BPF_LINK_GET_FD_BY_ID_LAST_FIELD link_id	4032	struct bpf_link *bpf_link_by_id(u32 id)
4023
4024	static int bpf_link_get_fd_by_id(const union bpf_attr *attr)
4025	{	4033	{
4026	struct bpf_link *link;	4034	struct bpf_link *link;
4027	u32 id = attr->link_id;
4028	int fd, err;
4029		4035
4030	if (CHECK_ATTR(BPF_LINK_GET_FD_BY_ID))	4036	if (!id)
4031	return -EINVAL;	4037	return ERR_PTR(-ENOENT);
4032
4033	if (!capable(CAP_SYS_ADMIN))
4034	return -EPERM;
4035		4038
4036	spin_lock_bh(&link_idr_lock);	4039	spin_lock_bh(&link_idr_lock);
4037	link = idr_find(&link_idr, id);
4038	/* before link is "settled", ID is 0, pretend it doesn't exist yet */	4040	/* before link is "settled", ID is 0, pretend it doesn't exist yet */
		4041	link = idr_find(&link_idr, id);
4039	if (link) {	4042	if (link) {
4040	if (link->id)	4043	if (link->id)
4041	err = bpf_link_inc_not_zero(link);	4044	link = bpf_link_inc_not_zero(link);
4042	else	4045	else
4043	err = -EAGAIN;	4046	link = ERR_PTR(-EAGAIN);
4044	} else {	4047	} else {
4045	err = -ENOENT;	4048	link = ERR_PTR(-ENOENT);
4046	}	4049	}
4047	spin_unlock_bh(&link_idr_lock);	4050	spin_unlock_bh(&link_idr_lock);
		4051	return link;
		4052	}
4048		4053
4049	if (err)	4054	#define BPF_LINK_GET_FD_BY_ID_LAST_FIELD link_id
4050	return err;	4055
		4056	static int bpf_link_get_fd_by_id(const union bpf_attr *attr)
		4057	{
		4058	struct bpf_link *link;
		4059	u32 id = attr->link_id;
		4060	int fd;
		4061
		4062	if (CHECK_ATTR(BPF_LINK_GET_FD_BY_ID))
		4063	return -EINVAL;
		4064
		4065	if (!capable(CAP_SYS_ADMIN))
		4066	return -EPERM;
		4067
		4068	link = bpf_link_by_id(id);
		4069	if (IS_ERR(link))
		4070	return PTR_ERR(link);
4051		4071
4052	fd = bpf_link_new_fd(link);	4072	fd = bpf_link_new_fd(link);
4053	if (fd < 0)	4073	if (fd < 0)

kernel/bpf/trampoline.c+26−3

@@ -7,6 +7,8 @@ · 6 unmodified lines
7	#include <linux/rbtree_latch.h>	7	#include <linux/rbtree_latch.h>
8	#include <linux/perf_event.h>	8	#include <linux/perf_event.h>
9	#include <linux/btf.h>	9	#include <linux/btf.h>
		10	#include <linux/rcupdate_trace.h>
		11	#include <linux/rcupdate_wait.h>
10		12
11	/* dummy _ops. The verifier will operate on target program's ops. */	13	/* dummy _ops. The verifier will operate on target program's ops. */
12	const struct bpf_verifier_ops bpf_extension_verifier_ops = {	14	const struct bpf_verifier_ops bpf_extension_verifier_ops = {
@@ -210,9 +212,12 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr) · 197 unmodified lines
210	* updates to trampoline would change the code from underneath the	212	* updates to trampoline would change the code from underneath the
211	* preempted task. Hence wait for tasks to voluntarily schedule or go	213	* preempted task. Hence wait for tasks to voluntarily schedule or go
212	* to userspace.	214	* to userspace.
		215	* The same trampoline can hold both sleepable and non-sleepable progs.
		216	* synchronize_rcu_tasks_trace() is needed to make sure all sleepable
		217	* programs finish executing.
		218	* Wait for these two grace periods together.
213	*/	219	*/
214		220	synchronize_rcu_mult(call_rcu_tasks, call_rcu_tasks_trace);
215	synchronize_rcu_tasks();
216		221
217	err = arch_prepare_bpf_trampoline(new_image, new_image + PAGE_SIZE / 2,	222	err = arch_prepare_bpf_trampoline(new_image, new_image + PAGE_SIZE / 2,
218	&tr->func.model, flags, tprogs,	223	&tr->func.model, flags, tprogs,
@@ -344,7 +349,14 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) · 125 unmodified lines
344	if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[BPF_TRAMP_FEXIT])))	349	if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[BPF_TRAMP_FEXIT])))
345	goto out;	350	goto out;
346	bpf_image_ksym_del(&tr->ksym);	351	bpf_image_ksym_del(&tr->ksym);
347	/* wait for tasks to get out of trampoline before freeing it */	352	/* This code will be executed when all bpf progs (both sleepable and
		353	* non-sleepable) went through
		354	* bpf_prog_put()->call_rcu[_tasks_trace]()->bpf_prog_free_deferred().
		355	* Hence no need for another synchronize_rcu_tasks_trace() here,
		356	* but synchronize_rcu_tasks() is still needed, since trampoline
		357	* may not have had any sleepable programs and we need to wait
		358	* for tasks to get out of trampoline code before freeing it.
		359	*/
348	synchronize_rcu_tasks();	360	synchronize_rcu_tasks();
349	bpf_jit_free_exec(tr->image);	361	bpf_jit_free_exec(tr->image);
350	hlist_del(&tr->hlist);	362	hlist_del(&tr->hlist);
@@ -394,6 +406,17 @@ void notrace __bpf_prog_exit(struct bpf_prog *prog, u64 start) · 43 unmodified lines
394	rcu_read_unlock();	406	rcu_read_unlock();
395	}	407	}
396		408
		409	void notrace __bpf_prog_enter_sleepable(void)
		410	{
		411	rcu_read_lock_trace();
		412	might_fault();
		413	}
		414
		415	void notrace __bpf_prog_exit_sleepable(void)
		416	{
		417	rcu_read_unlock_trace();
		418	}
		419
397	int __weak	420	int __weak
398	arch_prepare_bpf_trampoline(void image, void image_end,	421	arch_prepare_bpf_trampoline(void image, void image_end,
399	const struct btf_func_model *m, u32 flags,	422	const struct btf_func_model *m, u32 flags,

kernel/bpf/verifier.c+263−20

@@ -21,6 +21,7 @@ · 20 unmodified lines
21	#include <linux/ctype.h>	21	#include <linux/ctype.h>
22	#include <linux/error-injection.h>	22	#include <linux/error-injection.h>
23	#include <linux/bpf_lsm.h>	23	#include <linux/bpf_lsm.h>
		24	#include <linux/btf_ids.h>
24		25
25	#include "disasm.h"	26	#include "disasm.h"
26		27
@@ -2625,11 +2626,19 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 2598 unmodified lines
2625		2626
2626	#define MAX_PACKET_OFF 0xffff	2627	#define MAX_PACKET_OFF 0xffff
2627		2628
		2629	static enum bpf_prog_type resolve_prog_type(struct bpf_prog *prog)
		2630	{
		2631	return prog->aux->linked_prog ? prog->aux->linked_prog->type
		2632	: prog->type;
		2633	}
		2634
2628	static bool may_access_direct_pkt_data(struct bpf_verifier_env *env,	2635	static bool may_access_direct_pkt_data(struct bpf_verifier_env *env,
2629	const struct bpf_call_arg_meta *meta,	2636	const struct bpf_call_arg_meta *meta,
2630	enum bpf_access_type t)	2637	enum bpf_access_type t)
2631	{	2638	{
2632	switch (env->prog->type) {	2639	enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
		2640
		2641	switch (prog_type) {
2633	/* Program types only with direct read access go here! */	2642	/* Program types only with direct read access go here! */
2634	case BPF_PROG_TYPE_LWT_IN:	2643	case BPF_PROG_TYPE_LWT_IN:
2635	case BPF_PROG_TYPE_LWT_OUT:	2644	case BPF_PROG_TYPE_LWT_OUT:
@@ -3872,6 +3881,33 @@ static int int_ptr_type_to_size(enum bpf_arg_type type) · 1236 unmodified lines
3872	return -EINVAL;	3881	return -EINVAL;
3873	}	3882	}
3874		3883
		3884	static int resolve_map_arg_type(struct bpf_verifier_env *env,
		3885	const struct bpf_call_arg_meta *meta,
		3886	enum bpf_arg_type *arg_type)
		3887	{
		3888	if (!meta->map_ptr) {
		3889	/* kernel subsystem misconfigured verifier */
		3890	verbose(env, "invalid map_ptr to access map->type\n");
		3891	return -EACCES;
		3892	}
		3893
		3894	switch (meta->map_ptr->map_type) {
		3895	case BPF_MAP_TYPE_SOCKMAP:
		3896	case BPF_MAP_TYPE_SOCKHASH:
		3897	if (*arg_type == ARG_PTR_TO_MAP_VALUE) {
		3898	*arg_type = ARG_PTR_TO_SOCKET;
		3899	} else {
		3900	verbose(env, "invalid arg_type for sockmap/sockhash\n");
		3901	return -EINVAL;
		3902	}
		3903	break;
		3904
		3905	default:
		3906	break;
		3907	}
		3908	return 0;
		3909	}
		3910
3875	static int check_func_arg(struct bpf_verifier_env *env, u32 arg,	3911	static int check_func_arg(struct bpf_verifier_env *env, u32 arg,
3876	struct bpf_call_arg_meta *meta,	3912	struct bpf_call_arg_meta *meta,
3877	const struct bpf_func_proto *fn)	3913	const struct bpf_func_proto *fn)
@@ -3904,6 +3940,14 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 26 unmodified lines
3904	return -EACCES;	3940	return -EACCES;
3905	}	3941	}
3906		3942
		3943	if (arg_type == ARG_PTR_TO_MAP_VALUE \|\|
		3944	arg_type == ARG_PTR_TO_UNINIT_MAP_VALUE \|\|
		3945	arg_type == ARG_PTR_TO_MAP_VALUE_OR_NULL) {
		3946	err = resolve_map_arg_type(env, meta, &arg_type);
		3947	if (err)
		3948	return err;
		3949	}
		3950
3907	if (arg_type == ARG_PTR_TO_MAP_KEY \|\|	3951	if (arg_type == ARG_PTR_TO_MAP_KEY \|\|
3908	arg_type == ARG_PTR_TO_MAP_VALUE \|\|	3952	arg_type == ARG_PTR_TO_MAP_VALUE \|\|
3909	arg_type == ARG_PTR_TO_UNINIT_MAP_VALUE \|\|	3953	arg_type == ARG_PTR_TO_UNINIT_MAP_VALUE \|\|
@@ -3960,16 +4004,21 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 50 unmodified lines
3960	goto err_type;	4004	goto err_type;
3961	}	4005	}
3962	} else if (arg_type == ARG_PTR_TO_BTF_ID) {	4006	} else if (arg_type == ARG_PTR_TO_BTF_ID) {
		4007	bool ids_match = false;
		4008
3963	expected_type = PTR_TO_BTF_ID;	4009	expected_type = PTR_TO_BTF_ID;
3964	if (type != expected_type)	4010	if (type != expected_type)
3965	goto err_type;	4011	goto err_type;
3966	if (!fn->check_btf_id) {	4012	if (!fn->check_btf_id) {
3967	if (reg->btf_id != meta->btf_id) {	4013	if (reg->btf_id != meta->btf_id) {
3968	verbose(env, "Helper has type %s got %s in R%d\n",	4014	ids_match = btf_struct_ids_match(&env->log, reg->off, reg->btf_id,
3969	kernel_type_name(meta->btf_id),	4015	meta->btf_id);
3970	kernel_type_name(reg->btf_id), regno);	4016	if (!ids_match) {
3971		4017	verbose(env, "Helper has type %s got %s in R%d\n",
3972	return -EACCES;	4018	kernel_type_name(meta->btf_id),
		4019	kernel_type_name(reg->btf_id), regno);
		4020	return -EACCES;
		4021	}
3973	}	4022	}
3974	} else if (!fn->check_btf_id(reg->btf_id, arg)) {	4023	} else if (!fn->check_btf_id(reg->btf_id, arg)) {
3975	verbose(env, "Helper does not support %s in R%d\n",	4024	verbose(env, "Helper does not support %s in R%d\n",
@@ -3977,7 +4026,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 1 unmodified line
3977		4026
3978	return -EACCES;	4027	return -EACCES;
3979	}	4028	}
3980	if (!tnum_is_const(reg->var_off) \|\| reg->var_off.value \|\| reg->off) {	4029	if ((reg->off && !ids_match) \|\| !tnum_is_const(reg->var_off) \|\| reg->var_off.value) {
3981	verbose(env, "R%d is a pointer to in-kernel struct with non-zero offset\n",	4030	verbose(env, "R%d is a pointer to in-kernel struct with non-zero offset\n",
3982	regno);	4031	regno);
3983	return -EACCES;	4032	return -EACCES;
@@ -4143,6 +4192,38 @@ err_type: · 159 unmodified lines
4143	return -EACCES;	4192	return -EACCES;
4144	}	4193	}
4145		4194
		4195	static bool may_update_sockmap(struct bpf_verifier_env *env, int func_id)
		4196	{
		4197	enum bpf_attach_type eatype = env->prog->expected_attach_type;
		4198	enum bpf_prog_type type = resolve_prog_type(env->prog);
		4199
		4200	if (func_id != BPF_FUNC_map_update_elem)
		4201	return false;
		4202
		4203	/* It's not possible to get access to a locked struct sock in these
		4204	* contexts, so updating is safe.
		4205	*/
		4206	switch (type) {
		4207	case BPF_PROG_TYPE_TRACING:
		4208	if (eatype == BPF_TRACE_ITER)
		4209	return true;
		4210	break;
		4211	case BPF_PROG_TYPE_SOCKET_FILTER:
		4212	case BPF_PROG_TYPE_SCHED_CLS:
		4213	case BPF_PROG_TYPE_SCHED_ACT:
		4214	case BPF_PROG_TYPE_XDP:
		4215	case BPF_PROG_TYPE_SK_REUSEPORT:
		4216	case BPF_PROG_TYPE_FLOW_DISSECTOR:
		4217	case BPF_PROG_TYPE_SK_LOOKUP:
		4218	return true;
		4219	default:
		4220	break;
		4221	}
		4222
		4223	verbose(env, "cannot update sockmap in this context\n");
		4224	return false;
		4225	}
		4226
4146	static int check_map_func_compatibility(struct bpf_verifier_env *env,	4227	static int check_map_func_compatibility(struct bpf_verifier_env *env,
4147	struct bpf_map *map, int func_id)	4228	struct bpf_map *map, int func_id)
4148	{	4229	{
@@ -4214,7 +4295,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 65 unmodified lines
4214	func_id != BPF_FUNC_map_delete_elem &&	4295	func_id != BPF_FUNC_map_delete_elem &&
4215	func_id != BPF_FUNC_msg_redirect_map &&	4296	func_id != BPF_FUNC_msg_redirect_map &&
4216	func_id != BPF_FUNC_sk_select_reuseport &&	4297	func_id != BPF_FUNC_sk_select_reuseport &&
4217	func_id != BPF_FUNC_map_lookup_elem)	4298	func_id != BPF_FUNC_map_lookup_elem &&
		4299	!may_update_sockmap(env, func_id))
4218	goto error;	4300	goto error;
4219	break;	4301	break;
4220	case BPF_MAP_TYPE_SOCKHASH:	4302	case BPF_MAP_TYPE_SOCKHASH:
@@ -4223,7 +4305,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 2 unmodified lines
4223	func_id != BPF_FUNC_map_delete_elem &&	4305	func_id != BPF_FUNC_map_delete_elem &&
4224	func_id != BPF_FUNC_msg_redirect_hash &&	4306	func_id != BPF_FUNC_msg_redirect_hash &&
4225	func_id != BPF_FUNC_sk_select_reuseport &&	4307	func_id != BPF_FUNC_sk_select_reuseport &&
4226	func_id != BPF_FUNC_map_lookup_elem)	4308	func_id != BPF_FUNC_map_lookup_elem &&
		4309	!may_update_sockmap(env, func_id))
4227	goto error;	4310	goto error;
4228	break;	4311	break;
4229	case BPF_MAP_TYPE_REUSEPORT_SOCKARRAY:	4312	case BPF_MAP_TYPE_REUSEPORT_SOCKARRAY:
@@ -4242,6 +4325,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 12 unmodified lines
4242	func_id != BPF_FUNC_sk_storage_delete)	4325	func_id != BPF_FUNC_sk_storage_delete)
4243	goto error;	4326	goto error;
4244	break;	4327	break;
		4328	case BPF_MAP_TYPE_INODE_STORAGE:
		4329	if (func_id != BPF_FUNC_inode_storage_get &&
		4330	func_id != BPF_FUNC_inode_storage_delete)
		4331	goto error;
		4332	break;
4245	default:	4333	default:
4246	break;	4334	break;
4247	}	4335	}
@@ -4315,6 +4403,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 67 unmodified lines
4315	if (map->map_type != BPF_MAP_TYPE_SK_STORAGE)	4403	if (map->map_type != BPF_MAP_TYPE_SK_STORAGE)
4316	goto error;	4404	goto error;
4317	break;	4405	break;
		4406	case BPF_FUNC_inode_storage_get:
		4407	case BPF_FUNC_inode_storage_delete:
		4408	if (map->map_type != BPF_MAP_TYPE_INODE_STORAGE)
		4409	goto error;
		4410	break;
4318	default:	4411	default:
4319	break;	4412	break;
4320	}	4413	}
@@ -4775,6 +4868,11 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 454 unmodified lines
4775	return -EINVAL;	4868	return -EINVAL;
4776	}	4869	}
4777		4870
		4871	if (fn->allowed && !fn->allowed(env->prog)) {
		4872	verbose(env, "helper call is not allowed in probe\n");
		4873	return -EINVAL;
		4874	}
		4875
4778	/* With LD_ABS/IND some JITs save/restore skb from r1. */	4876	/* With LD_ABS/IND some JITs save/restore skb from r1. */
4779	changes_data = bpf_helper_changes_pkt_data(fn->func);	4877	changes_data = bpf_helper_changes_pkt_data(fn->func);
4780	if (changes_data && fn->arg1_type != ARG_PTR_TO_CTX) {	4878	if (changes_data && fn->arg1_type != ARG_PTR_TO_CTX) {
@@ -5732,6 +5830,67 @@ static void scalar_min_max_or(struct bpf_reg_state *dst_reg, · 951 unmodified lines
5732	__update_reg_bounds(dst_reg);	5830	__update_reg_bounds(dst_reg);
5733	}	5831	}
5734		5832
		5833	static void scalar32_min_max_xor(struct bpf_reg_state *dst_reg,
		5834	struct bpf_reg_state *src_reg)
		5835	{
		5836	bool src_known = tnum_subreg_is_const(src_reg->var_off);
		5837	bool dst_known = tnum_subreg_is_const(dst_reg->var_off);
		5838	struct tnum var32_off = tnum_subreg(dst_reg->var_off);
		5839	s32 smin_val = src_reg->s32_min_value;
		5840
		5841	/* Assuming scalar64_min_max_xor will be called so it is safe
		5842	* to skip updating register for known case.
		5843	*/
		5844	if (src_known && dst_known)
		5845	return;
		5846
		5847	/* We get both minimum and maximum from the var32_off. */
		5848	dst_reg->u32_min_value = var32_off.value;
		5849	dst_reg->u32_max_value = var32_off.value \| var32_off.mask;
		5850
		5851	if (dst_reg->s32_min_value >= 0 && smin_val >= 0) {
		5852	/* XORing two positive sign numbers gives a positive,
		5853	* so safe to cast u32 result into s32.
		5854	*/
		5855	dst_reg->s32_min_value = dst_reg->u32_min_value;
		5856	dst_reg->s32_max_value = dst_reg->u32_max_value;
		5857	} else {
		5858	dst_reg->s32_min_value = S32_MIN;
		5859	dst_reg->s32_max_value = S32_MAX;
		5860	}
		5861	}
		5862
		5863	static void scalar_min_max_xor(struct bpf_reg_state *dst_reg,
		5864	struct bpf_reg_state *src_reg)
		5865	{
		5866	bool src_known = tnum_is_const(src_reg->var_off);
		5867	bool dst_known = tnum_is_const(dst_reg->var_off);
		5868	s64 smin_val = src_reg->smin_value;
		5869
		5870	if (src_known && dst_known) {
		5871	/* dst_reg->var_off.value has been updated earlier */
		5872	__mark_reg_known(dst_reg, dst_reg->var_off.value);
		5873	return;
		5874	}
		5875
		5876	/* We get both minimum and maximum from the var_off. */
		5877	dst_reg->umin_value = dst_reg->var_off.value;
		5878	dst_reg->umax_value = dst_reg->var_off.value \| dst_reg->var_off.mask;
		5879
		5880	if (dst_reg->smin_value >= 0 && smin_val >= 0) {
		5881	/* XORing two positive sign numbers gives a positive,
		5882	* so safe to cast u64 result into s64.
		5883	*/
		5884	dst_reg->smin_value = dst_reg->umin_value;
		5885	dst_reg->smax_value = dst_reg->umax_value;
		5886	} else {
		5887	dst_reg->smin_value = S64_MIN;
		5888	dst_reg->smax_value = S64_MAX;
		5889	}
		5890
		5891	__update_reg_bounds(dst_reg);
		5892	}
		5893
5735	static void __scalar32_min_max_lsh(struct bpf_reg_state *dst_reg,	5894	static void __scalar32_min_max_lsh(struct bpf_reg_state *dst_reg,
5736	u64 umin_val, u64 umax_val)	5895	u64 umin_val, u64 umax_val)
5737	{	5896	{
@@ -6040,6 +6199,11 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env, · 302 unmodified lines
6040	scalar32_min_max_or(dst_reg, &src_reg);	6199	scalar32_min_max_or(dst_reg, &src_reg);
6041	scalar_min_max_or(dst_reg, &src_reg);	6200	scalar_min_max_or(dst_reg, &src_reg);
6042	break;	6201	break;
		6202	case BPF_XOR:
		6203	dst_reg->var_off = tnum_xor(dst_reg->var_off, src_reg.var_off);
		6204	scalar32_min_max_xor(dst_reg, &src_reg);
		6205	scalar_min_max_xor(dst_reg, &src_reg);
		6206	break;
6043	case BPF_LSH:	6207	case BPF_LSH:
6044	if (umax_val >= insn_bitness) {	6208	if (umax_val >= insn_bitness) {
6045	/* Shifts greater than 31 or 63 are undefined.	6209	/* Shifts greater than 31 or 63 are undefined.
@@ -7287,7 +7451,7 @@ static int check_ld_abs(struct bpf_verifier_env env, struct bpf_insn insn) · 1241 unmodified lines
7287	u8 mode = BPF_MODE(insn->code);	7451	u8 mode = BPF_MODE(insn->code);
7288	int i, err;	7452	int i, err;
7289		7453
7290	if (!may_access_skb(env->prog->type)) {	7454	if (!may_access_skb(resolve_prog_type(env->prog))) {
7291	verbose(env, "BPF_LD_[ABS\|IND] instructions not allowed for this program type\n");	7455	verbose(env, "BPF_LD_[ABS\|IND] instructions not allowed for this program type\n");
7292	return -EINVAL;	7456	return -EINVAL;
7293	}	7457	}
@@ -7375,11 +7539,12 @@ static int check_return_code(struct bpf_verifier_env *env) · 81 unmodified lines
7375	const struct bpf_prog *prog = env->prog;	7539	const struct bpf_prog *prog = env->prog;
7376	struct bpf_reg_state *reg;	7540	struct bpf_reg_state *reg;
7377	struct tnum range = tnum_range(0, 1);	7541	struct tnum range = tnum_range(0, 1);
		7542	enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
7378	int err;	7543	int err;
7379		7544
7380	/* LSM and struct_ops func-ptr's return type could be "void" */	7545	/* LSM and struct_ops func-ptr's return type could be "void" */
7381	if ((env->prog->type == BPF_PROG_TYPE_STRUCT_OPS \|\|	7546	if ((prog_type == BPF_PROG_TYPE_STRUCT_OPS \|\|
7382	env->prog->type == BPF_PROG_TYPE_LSM) &&	7547	prog_type == BPF_PROG_TYPE_LSM) &&
7383	!prog->aux->attach_func_proto->type)	7548	!prog->aux->attach_func_proto->type)
7384	return 0;	7549	return 0;
7385		7550
@@ -7398,7 +7563,7 @@ static int check_return_code(struct bpf_verifier_env *env) · 12 unmodified lines
7398	return -EACCES;	7563	return -EACCES;
7399	}	7564	}
7400		7565
7401	switch (env->prog->type) {	7566	switch (prog_type) {
7402	case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:	7567	case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
7403	if (env->prog->expected_attach_type == BPF_CGROUP_UDP4_RECVMSG \|\|	7568	if (env->prog->expected_attach_type == BPF_CGROUP_UDP4_RECVMSG \|\|
7404	env->prog->expected_attach_type == BPF_CGROUP_UDP6_RECVMSG \|\|	7569	env->prog->expected_attach_type == BPF_CGROUP_UDP6_RECVMSG \|\|
@@ -9154,6 +9319,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 1749 unmodified lines
9154	struct bpf_prog *prog)	9319	struct bpf_prog *prog)
9155		9320
9156	{	9321	{
		9322	enum bpf_prog_type prog_type = resolve_prog_type(prog);
9157	/*	9323	/*
9158	* Validate that trace type programs use preallocated hash maps.	9324	* Validate that trace type programs use preallocated hash maps.
9159	*	9325	*
@@ -9171,8 +9337,8 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 11 unmodified lines
9171	* now, but warnings are emitted so developers are made aware of	9337	* now, but warnings are emitted so developers are made aware of
9172	* the unsafety and can fix their programs before this is enforced.	9338	* the unsafety and can fix their programs before this is enforced.
9173	*/	9339	*/
9174	if (is_tracing_prog_type(prog->type) && !is_preallocated_map(map)) {	9340	if (is_tracing_prog_type(prog_type) && !is_preallocated_map(map)) {
9175	if (prog->type == BPF_PROG_TYPE_PERF_EVENT) {	9341	if (prog_type == BPF_PROG_TYPE_PERF_EVENT) {
9176	verbose(env, "perf_event programs can only use preallocated hash map\n");	9342	verbose(env, "perf_event programs can only use preallocated hash map\n");
9177	return -EINVAL;	9343	return -EINVAL;
9178	}	9344	}
@@ -9184,8 +9350,8 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 5 unmodified lines
9184	verbose(env, "trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.\n");	9350	verbose(env, "trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.\n");
9185	}	9351	}
9186		9352
9187	if ((is_tracing_prog_type(prog->type) \|\|	9353	if ((is_tracing_prog_type(prog_type) \|\|
9188	prog->type == BPF_PROG_TYPE_SOCKET_FILTER) &&	9354	prog_type == BPF_PROG_TYPE_SOCKET_FILTER) &&
9189	map_value_has_spin_lock(map)) {	9355	map_value_has_spin_lock(map)) {
9190	verbose(env, "tracing progs cannot use bpf_spin_lock yet\n");	9356	verbose(env, "tracing progs cannot use bpf_spin_lock yet\n");
9191	return -EINVAL;	9357	return -EINVAL;
@@ -9202,6 +9368,23 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 10 unmodified lines
9202	return -EINVAL;	9368	return -EINVAL;
9203	}	9369	}
9204		9370
		9371	if (prog->aux->sleepable)
		9372	switch (map->map_type) {
		9373	case BPF_MAP_TYPE_HASH:
		9374	case BPF_MAP_TYPE_LRU_HASH:
		9375	case BPF_MAP_TYPE_ARRAY:
		9376	if (!is_preallocated_map(map)) {
		9377	verbose(env,
		9378	"Sleepable programs can only use preallocated hash maps\n");
		9379	return -EINVAL;
		9380	}
		9381	break;
		9382	default:
		9383	verbose(env,
		9384	"Sleepable programs can only use array and hash maps\n");
		9385	return -EINVAL;
		9386	}
		9387
9205	return 0;	9388	return 0;
9206	}	9389	}
9207		9390
@@ -9897,7 +10080,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) · 689 unmodified lines
9897	insn->code = BPF_LDX \| BPF_PROBE_MEM \|	10080	insn->code = BPF_LDX \| BPF_PROBE_MEM \|
9898	BPF_SIZE((insn)->code);	10081	BPF_SIZE((insn)->code);
9899	env->prog->aux->num_exentries++;	10082	env->prog->aux->num_exentries++;
9900	} else if (env->prog->type != BPF_PROG_TYPE_STRUCT_OPS) {	10083	} else if (resolve_prog_type(env->prog) != BPF_PROG_TYPE_STRUCT_OPS) {
9901	verbose(env, "Writes through BTF pointers are not allowed\n");	10084	verbose(env, "Writes through BTF pointers are not allowed\n");
9902	return -EINVAL;	10085	return -EINVAL;
9903	}	10086	}
@@ -10820,6 +11003,37 @@ static int check_attach_modify_return(struct bpf_prog *prog, unsigned long addr) · 916 unmodified lines
10820	return -EINVAL;	11003	return -EINVAL;
10821	}	11004	}
10822		11005
		11006	/* non exhaustive list of sleepable bpf_lsm_() functions /
		11007	BTF_SET_START(btf_sleepable_lsm_hooks)
		11008	#ifdef CONFIG_BPF_LSM
		11009	BTF_ID(func, bpf_lsm_bprm_committed_creds)
		11010	#else
		11011	BTF_ID_UNUSED
		11012	#endif
		11013	BTF_SET_END(btf_sleepable_lsm_hooks)
		11014
		11015	static int check_sleepable_lsm_hook(u32 btf_id)
		11016	{
		11017	return btf_id_set_contains(&btf_sleepable_lsm_hooks, btf_id);
		11018	}
		11019
		11020	/* list of non-sleepable functions that are otherwise on
		11021	* ALLOW_ERROR_INJECTION list
		11022	*/
		11023	BTF_SET_START(btf_non_sleepable_error_inject)
		11024	/* Three functions below can be called from sleepable and non-sleepable context.
		11025	* Assume non-sleepable from bpf safety point of view.
		11026	*/
		11027	BTF_ID(func, __add_to_page_cache_locked)
		11028	BTF_ID(func, should_fail_alloc_page)
		11029	BTF_ID(func, should_failslab)
		11030	BTF_SET_END(btf_non_sleepable_error_inject)
		11031
		11032	static int check_non_sleepable_error_inject(u32 btf_id)
		11033	{
		11034	return btf_id_set_contains(&btf_non_sleepable_error_inject, btf_id);
		11035	}
		11036
10823	static int check_attach_btf_id(struct bpf_verifier_env *env)	11037	static int check_attach_btf_id(struct bpf_verifier_env *env)
10824	{	11038	{
10825	struct bpf_prog *prog = env->prog;	11039	struct bpf_prog *prog = env->prog;
@@ -10837,6 +11051,12 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 11 unmodified lines
10837	long addr;	11051	long addr;
10838	u64 key;	11052	u64 key;
10839		11053
		11054	if (prog->aux->sleepable && prog->type != BPF_PROG_TYPE_TRACING &&
		11055	prog->type != BPF_PROG_TYPE_LSM) {
		11056	verbose(env, "Only fentry/fexit/fmod_ret and lsm programs can be sleepable\n");
		11057	return -EINVAL;
		11058	}
		11059
10840	if (prog->type == BPF_PROG_TYPE_STRUCT_OPS)	11060	if (prog->type == BPF_PROG_TYPE_STRUCT_OPS)
10841	return check_struct_ops_btf_id(env);	11061	return check_struct_ops_btf_id(env);
10842		11062
@@ -11045,13 +11265,36 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 202 unmodified lines
11045	}	11265	}
11046	}	11266	}
11047		11267
11048	if (prog->expected_attach_type == BPF_MODIFY_RETURN) {	11268	if (prog->aux->sleepable) {
		11269	ret = -EINVAL;
		11270	switch (prog->type) {
		11271	case BPF_PROG_TYPE_TRACING:
		11272	/* fentry/fexit/fmod_ret progs can be sleepable only if they are
		11273	* attached to ALLOW_ERROR_INJECTION and are not in denylist.
		11274	*/
		11275	if (!check_non_sleepable_error_inject(btf_id) &&
		11276	within_error_injection_list(addr))
		11277	ret = 0;
		11278	break;
		11279	case BPF_PROG_TYPE_LSM:
		11280	/* LSM progs check that they are attached to bpf_lsm_*() funcs.
		11281	* Only some of them are sleepable.
		11282	*/
		11283	if (check_sleepable_lsm_hook(btf_id))
		11284	ret = 0;
		11285	break;
		11286	default:
		11287	break;
		11288	}
		11289	if (ret)
		11290	verbose(env, "%s is not sleepable\n",
		11291	prog->aux->attach_func_name);
		11292	} else if (prog->expected_attach_type == BPF_MODIFY_RETURN) {
11049	ret = check_attach_modify_return(prog, addr);	11293	ret = check_attach_modify_return(prog, addr);
11050	if (ret)	11294	if (ret)
11051	verbose(env, "%s() is not modifiable\n",	11295	verbose(env, "%s() is not modifiable\n",
11052	prog->aux->attach_func_name);	11296	prog->aux->attach_func_name);
11053	}	11297	}
11054
11055	if (ret)	11298	if (ret)
11056	goto out;	11299	goto out;
11057	tr->func.addr = (void *)addr;	11300	tr->func.addr = (void *)addr;

kernel/trace/bpf_trace.c+50−0

@@ -1098,6 +1098,52 @@ static const struct bpf_func_proto bpf_send_signal_thread_proto = { · 1097 unmodified lines
1098	.arg1_type = ARG_ANYTHING,	1098	.arg1_type = ARG_ANYTHING,
1099	};	1099	};
1100		1100
		1101	BPF_CALL_3(bpf_d_path, struct path , path, char , buf, u32, sz)
		1102	{
		1103	long len;
		1104	char *p;
		1105
		1106	if (!sz)
		1107	return 0;
		1108
		1109	p = d_path(path, buf, sz);
		1110	if (IS_ERR(p)) {
		1111	len = PTR_ERR(p);
		1112	} else {
		1113	len = buf + sz - p;
		1114	memmove(buf, p, len);
		1115	}
		1116
		1117	return len;
		1118	}
		1119
		1120	BTF_SET_START(btf_allowlist_d_path)
		1121	BTF_ID(func, vfs_truncate)
		1122	BTF_ID(func, vfs_fallocate)
		1123	BTF_ID(func, dentry_open)
		1124	BTF_ID(func, vfs_getattr)
		1125	BTF_ID(func, filp_close)
		1126	BTF_SET_END(btf_allowlist_d_path)
		1127
		1128	static bool bpf_d_path_allowed(const struct bpf_prog *prog)
		1129	{
		1130	return btf_id_set_contains(&btf_allowlist_d_path, prog->aux->attach_btf_id);
		1131	}
		1132
		1133	BTF_ID_LIST(bpf_d_path_btf_ids)
		1134	BTF_ID(struct, path)
		1135
		1136	static const struct bpf_func_proto bpf_d_path_proto = {
		1137	.func = bpf_d_path,
		1138	.gpl_only = false,
		1139	.ret_type = RET_INTEGER,
		1140	.arg1_type = ARG_PTR_TO_BTF_ID,
		1141	.arg2_type = ARG_PTR_TO_MEM,
		1142	.arg3_type = ARG_CONST_SIZE_OR_ZERO,
		1143	.btf_id = bpf_d_path_btf_ids,
		1144	.allowed = bpf_d_path_allowed,
		1145	};
		1146
1101	const struct bpf_func_proto *	1147	const struct bpf_func_proto *
1102	bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)	1148	bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
1103	{	1149	{
@@ -1182,6 +1228,8 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 78 unmodified lines
1182	return &bpf_jiffies64_proto;	1228	return &bpf_jiffies64_proto;
1183	case BPF_FUNC_get_task_stack:	1229	case BPF_FUNC_get_task_stack:
1184	return &bpf_get_task_stack_proto;	1230	return &bpf_get_task_stack_proto;
		1231	case BPF_FUNC_copy_from_user:
		1232	return prog->aux->sleepable ? &bpf_copy_from_user_proto : NULL;
1185	default:	1233	default:
1186	return NULL;	1234	return NULL;
1187	}	1235	}
@@ -1579,6 +1627,8 @@ tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 391 unmodified lines
1579	return prog->expected_attach_type == BPF_TRACE_ITER ?	1627	return prog->expected_attach_type == BPF_TRACE_ITER ?
1580	&bpf_seq_write_proto :	1628	&bpf_seq_write_proto :
1581	NULL;	1629	NULL;
		1630	case BPF_FUNC_d_path:
		1631	return &bpf_d_path_proto;
1582	default:	1632	default:
1583	return raw_tp_prog_func_proto(func_id, prog);	1633	return raw_tp_prog_func_proto(func_id, prog);
1584	}	1634	}

mm/filemap.c+4−4

@@ -827,10 +827,10 @@ int replace_page_cache_page(struct page old, struct page new, gfp_t gfp_mask) · 826 unmodified lines
827	}	827	}
828	EXPORT_SYMBOL_GPL(replace_page_cache_page);	828	EXPORT_SYMBOL_GPL(replace_page_cache_page);
829		829
830	static int __add_to_page_cache_locked(struct page *page,	830	noinline int __add_to_page_cache_locked(struct page *page,
831	struct address_space *mapping,	831	struct address_space *mapping,
832	pgoff_t offset, gfp_t gfp_mask,	832	pgoff_t offset, gfp_t gfp_mask,
833	void **shadowp)	833	void **shadowp)
834	{	834	{
835	XA_STATE(xas, &mapping->i_pages, offset);	835	XA_STATE(xas, &mapping->i_pages, offset);
836	int huge = PageHuge(page);	836	int huge = PageHuge(page);

mm/page_alloc.c+1−1

@@ -3482,7 +3482,7 @@ static inline bool __should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) · 3481 unmodified lines
3482		3482
3483	#endif /* CONFIG_FAIL_PAGE_ALLOC */	3483	#endif /* CONFIG_FAIL_PAGE_ALLOC */
3484		3484
3485	static noinline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)	3485	noinline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
3486	{	3486	{
3487	return __should_fail_alloc_page(gfp_mask, order);	3487	return __should_fail_alloc_page(gfp_mask, order);
3488	}	3488	}

net/bpfilter/Kconfig+1−0

@@ -2,6 +2,7 @@ · 1 unmodified line
2	menuconfig BPFILTER	2	menuconfig BPFILTER
3	bool "BPF based packet filtering framework (BPFILTER)"	3	bool "BPF based packet filtering framework (BPFILTER)"
4	depends on NET && BPF && INET	4	depends on NET && BPF && INET
		5	select USERMODE_DRIVER
5	help	6	help
6	This builds experimental bpfilter framework that is aiming to	7	This builds experimental bpfilter framework that is aiming to
7	provide netfilter compatible functionality via BPF	8	provide netfilter compatible functionality via BPF

net/core/bpf_sk_storage.c+138−695

@@ -7,97 +7,14 @@ · 6 unmodified lines
7	#include <linux/spinlock.h>	7	#include <linux/spinlock.h>
8	#include <linux/bpf.h>	8	#include <linux/bpf.h>
9	#include <linux/btf_ids.h>	9	#include <linux/btf_ids.h>
		10	#include <linux/bpf_local_storage.h>
10	#include <net/bpf_sk_storage.h>	11	#include <net/bpf_sk_storage.h>
11	#include <net/sock.h>	12	#include <net/sock.h>
12	#include <uapi/linux/sock_diag.h>	13	#include <uapi/linux/sock_diag.h>
13	#include <uapi/linux/btf.h>	14	#include <uapi/linux/btf.h>
		15	#include <linux/btf_ids.h>
14		16
15	#define SK_STORAGE_CREATE_FLAG_MASK \	17	DEFINE_BPF_STORAGE_CACHE(sk_cache);
16	(BPF_F_NO_PREALLOC \| BPF_F_CLONE)
17
18	struct bucket {
19	struct hlist_head list;
20	raw_spinlock_t lock;
21	};
22
23	/* Thp map is not the primary owner of a bpf_sk_storage_elem.
24	* Instead, the sk->sk_bpf_storage is.
25	*
26	* The map (bpf_sk_storage_map) is for two purposes
27	* 1. Define the size of the "sk local storage". It is
28	* the map's value_size.
29	*
30	* 2. Maintain a list to keep track of all elems such
31	* that they can be cleaned up during the map destruction.
32	*
33	* When a bpf local storage is being looked up for a
34	* particular sk, the "bpf_map" pointer is actually used
35	* as the "key" to search in the list of elem in
36	* sk->sk_bpf_storage.
37	*
38	* Hence, consider sk->sk_bpf_storage is the mini-map
39	* with the "bpf_map" pointer as the searching key.
40	*/
41	struct bpf_sk_storage_map {
42	struct bpf_map map;
43	/* Lookup elem does not require accessing the map.
44	*
45	* Updating/Deleting requires a bucket lock to
46	* link/unlink the elem from the map. Having
47	* multiple buckets to improve contention.
48	*/
49	struct bucket *buckets;
50	u32 bucket_log;
51	u16 elem_size;
52	u16 cache_idx;
53	};
54
55	struct bpf_sk_storage_data {
56	/* smap is used as the searching key when looking up
57	* from sk->sk_bpf_storage.
58	*
59	* Put it in the same cacheline as the data to minimize
60	* the number of cachelines access during the cache hit case.
61	*/
62	struct bpf_sk_storage_map __rcu *smap;
63	u8 data[] __aligned(8);
64	};
65
66	/* Linked to bpf_sk_storage and bpf_sk_storage_map */
67	struct bpf_sk_storage_elem {
68	struct hlist_node map_node; /* Linked to bpf_sk_storage_map */
69	struct hlist_node snode; /* Linked to bpf_sk_storage */
70	struct bpf_sk_storage __rcu *sk_storage;
71	struct rcu_head rcu;
72	/* 8 bytes hole */
73	/* The data is stored in aother cacheline to minimize
74	* the number of cachelines access during a cache hit.
75	*/
76	struct bpf_sk_storage_data sdata ____cacheline_aligned;
77	};
78
79	#define SELEM(_SDATA) container_of((_SDATA), struct bpf_sk_storage_elem, sdata)
80	#define SDATA(_SELEM) (&(_SELEM)->sdata)
81	#define BPF_SK_STORAGE_CACHE_SIZE 16
82
83	static DEFINE_SPINLOCK(cache_idx_lock);
84	static u64 cache_idx_usage_counts[BPF_SK_STORAGE_CACHE_SIZE];
85
86	struct bpf_sk_storage {
87	struct bpf_sk_storage_data __rcu *cache[BPF_SK_STORAGE_CACHE_SIZE];
88	struct hlist_head list; /* List of bpf_sk_storage_elem */
89	struct sock sk; / The sk that owns the the above "list" of
90	* bpf_sk_storage_elem.
91	*/
92	struct rcu_head rcu;
93	raw_spinlock_t lock; /* Protect adding/removing from the "list" */
94	};
95
96	static struct bucket select_bucket(struct bpf_sk_storage_map smap,
97	struct bpf_sk_storage_elem *selem)
98	{
99	return &smap->buckets[hash_ptr(selem, smap->bucket_log)];
100	}
101		18
102	static int omem_charge(struct sock *sk, unsigned int size)	19	static int omem_charge(struct sock *sk, unsigned int size)
103	{	20	{
@@ -111,445 +28,38 @@ static int omem_charge(struct sock *sk, unsigned int size) · 7 unmodified lines
111	return -ENOMEM;	28	return -ENOMEM;
112	}	29	}
113		30
114	static bool selem_linked_to_sk(const struct bpf_sk_storage_elem *selem)	31	static struct bpf_local_storage_data *
115	{
116	return !hlist_unhashed(&selem->snode);
117	}
118
119	static bool selem_linked_to_map(const struct bpf_sk_storage_elem *selem)
120	{
121	return !hlist_unhashed(&selem->map_node);
122	}
123
124	static struct bpf_sk_storage_elem selem_alloc(struct bpf_sk_storage_map smap,
125	struct sock sk, void value,
126	bool charge_omem)
127	{
128	struct bpf_sk_storage_elem *selem;
129
130	if (charge_omem && omem_charge(sk, smap->elem_size))
131	return NULL;
132
133	selem = kzalloc(smap->elem_size, GFP_ATOMIC \| __GFP_NOWARN);
134	if (selem) {
135	if (value)
136	memcpy(SDATA(selem)->data, value, smap->map.value_size);
137	return selem;
138	}
139
140	if (charge_omem)
141	atomic_sub(smap->elem_size, &sk->sk_omem_alloc);
142
143	return NULL;
144	}
145
146	/* sk_storage->lock must be held and selem->sk_storage == sk_storage.
147	* The caller must ensure selem->smap is still valid to be
148	* dereferenced for its smap->elem_size and smap->cache_idx.
149	*/
150	static bool __selem_unlink_sk(struct bpf_sk_storage *sk_storage,
151	struct bpf_sk_storage_elem *selem,
152	bool uncharge_omem)
153	{
154	struct bpf_sk_storage_map *smap;
155	bool free_sk_storage;
156	struct sock *sk;
157
158	smap = rcu_dereference(SDATA(selem)->smap);
159	sk = sk_storage->sk;
160
161	/* All uncharging on sk->sk_omem_alloc must be done first.
162	* sk may be freed once the last selem is unlinked from sk_storage.
163	*/
164	if (uncharge_omem)
165	atomic_sub(smap->elem_size, &sk->sk_omem_alloc);
166
167	free_sk_storage = hlist_is_singular_node(&selem->snode,
168	&sk_storage->list);
169	if (free_sk_storage) {
170	atomic_sub(sizeof(struct bpf_sk_storage), &sk->sk_omem_alloc);
171	sk_storage->sk = NULL;
172	/* After this RCU_INIT, sk may be freed and cannot be used */
173	RCU_INIT_POINTER(sk->sk_bpf_storage, NULL);
174
175	/* sk_storage is not freed now. sk_storage->lock is
176	* still held and raw_spin_unlock_bh(&sk_storage->lock)
177	* will be done by the caller.
178	*
179	* Although the unlock will be done under
180	* rcu_read_lock(), it is more intutivie to
181	* read if kfree_rcu(sk_storage, rcu) is done
182	* after the raw_spin_unlock_bh(&sk_storage->lock).
183	*
184	* Hence, a "bool free_sk_storage" is returned
185	* to the caller which then calls the kfree_rcu()
186	* after unlock.
187	*/
188	}
189	hlist_del_init_rcu(&selem->snode);
190	if (rcu_access_pointer(sk_storage->cache[smap->cache_idx]) ==
191	SDATA(selem))
192	RCU_INIT_POINTER(sk_storage->cache[smap->cache_idx], NULL);
193
194	kfree_rcu(selem, rcu);
195
196	return free_sk_storage;
197	}
198
199	static void selem_unlink_sk(struct bpf_sk_storage_elem *selem)
200	{
201	struct bpf_sk_storage *sk_storage;
202	bool free_sk_storage = false;
203
204	if (unlikely(!selem_linked_to_sk(selem)))
205	/* selem has already been unlinked from sk */
206	return;
207
208	sk_storage = rcu_dereference(selem->sk_storage);
209	raw_spin_lock_bh(&sk_storage->lock);
210	if (likely(selem_linked_to_sk(selem)))
211	free_sk_storage = __selem_unlink_sk(sk_storage, selem, true);
212	raw_spin_unlock_bh(&sk_storage->lock);
213
214	if (free_sk_storage)
215	kfree_rcu(sk_storage, rcu);
216	}
217
218	static void __selem_link_sk(struct bpf_sk_storage *sk_storage,
219	struct bpf_sk_storage_elem *selem)
220	{
221	RCU_INIT_POINTER(selem->sk_storage, sk_storage);
222	hlist_add_head(&selem->snode, &sk_storage->list);
223	}
224
225	static void selem_unlink_map(struct bpf_sk_storage_elem *selem)
226	{
227	struct bpf_sk_storage_map *smap;
228	struct bucket *b;
229
230	if (unlikely(!selem_linked_to_map(selem)))
231	/* selem has already be unlinked from smap */
232	return;
233
234	smap = rcu_dereference(SDATA(selem)->smap);
235	b = select_bucket(smap, selem);
236	raw_spin_lock_bh(&b->lock);
237	if (likely(selem_linked_to_map(selem)))
238	hlist_del_init_rcu(&selem->map_node);
239	raw_spin_unlock_bh(&b->lock);
240	}
241
242	static void selem_link_map(struct bpf_sk_storage_map *smap,
243	struct bpf_sk_storage_elem *selem)
244	{
245	struct bucket *b = select_bucket(smap, selem);
246
247	raw_spin_lock_bh(&b->lock);
248	RCU_INIT_POINTER(SDATA(selem)->smap, smap);
249	hlist_add_head_rcu(&selem->map_node, &b->list);
250	raw_spin_unlock_bh(&b->lock);
251	}
252
253	static void selem_unlink(struct bpf_sk_storage_elem *selem)
254	{
255	/* Always unlink from map before unlinking from sk_storage
256	* because selem will be freed after successfully unlinked from
257	* the sk_storage.
258	*/
259	selem_unlink_map(selem);
260	selem_unlink_sk(selem);
261	}
262
263	static struct bpf_sk_storage_data *
264	__sk_storage_lookup(struct bpf_sk_storage *sk_storage,
265	struct bpf_sk_storage_map *smap,
266	bool cacheit_lockit)
267	{
268	struct bpf_sk_storage_data *sdata;
269	struct bpf_sk_storage_elem *selem;
270
271	/* Fast path (cache hit) */
272	sdata = rcu_dereference(sk_storage->cache[smap->cache_idx]);
273	if (sdata && rcu_access_pointer(sdata->smap) == smap)
274	return sdata;
275
276	/* Slow path (cache miss) */
277	hlist_for_each_entry_rcu(selem, &sk_storage->list, snode)
278	if (rcu_access_pointer(SDATA(selem)->smap) == smap)
279	break;
280
281	if (!selem)
282	return NULL;
283
284	sdata = SDATA(selem);
285	if (cacheit_lockit) {
286	/* spinlock is needed to avoid racing with the
287	* parallel delete. Otherwise, publishing an already
288	* deleted sdata to the cache will become a use-after-free
289	* problem in the next __sk_storage_lookup().
290	*/
291	raw_spin_lock_bh(&sk_storage->lock);
292	if (selem_linked_to_sk(selem))
293	rcu_assign_pointer(sk_storage->cache[smap->cache_idx],
294	sdata);
295	raw_spin_unlock_bh(&sk_storage->lock);
296	}
297
298	return sdata;
299	}
300
301	static struct bpf_sk_storage_data *
302	sk_storage_lookup(struct sock sk, struct bpf_map map, bool cacheit_lockit)	32	sk_storage_lookup(struct sock sk, struct bpf_map map, bool cacheit_lockit)
303	{	33	{
304	struct bpf_sk_storage *sk_storage;	34	struct bpf_local_storage *sk_storage;
305	struct bpf_sk_storage_map *smap;	35	struct bpf_local_storage_map *smap;
306		36
307	sk_storage = rcu_dereference(sk->sk_bpf_storage);	37	sk_storage = rcu_dereference(sk->sk_bpf_storage);
308	if (!sk_storage)	38	if (!sk_storage)
309	return NULL;	39	return NULL;
310		40
311	smap = (struct bpf_sk_storage_map *)map;	41	smap = (struct bpf_local_storage_map *)map;
312	return __sk_storage_lookup(sk_storage, smap, cacheit_lockit);	42	return bpf_local_storage_lookup(sk_storage, smap, cacheit_lockit);
313	}
314
315	static int check_flags(const struct bpf_sk_storage_data *old_sdata,
316	u64 map_flags)
317	{
318	if (old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_NOEXIST)
319	/* elem already exists */
320	return -EEXIST;
321
322	if (!old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_EXIST)
323	/* elem doesn't exist, cannot update it */
324	return -ENOENT;
325
326	return 0;
327	}
328
329	static int sk_storage_alloc(struct sock *sk,
330	struct bpf_sk_storage_map *smap,
331	struct bpf_sk_storage_elem *first_selem)
332	{
333	struct bpf_sk_storage prev_sk_storage, sk_storage;
334	int err;
335
336	err = omem_charge(sk, sizeof(*sk_storage));
337	if (err)
338	return err;
339
340	sk_storage = kzalloc(sizeof(*sk_storage), GFP_ATOMIC \| __GFP_NOWARN);
341	if (!sk_storage) {
342	err = -ENOMEM;
343	goto uncharge;
344	}
345	INIT_HLIST_HEAD(&sk_storage->list);
346	raw_spin_lock_init(&sk_storage->lock);
347	sk_storage->sk = sk;
348
349	__selem_link_sk(sk_storage, first_selem);
350	selem_link_map(smap, first_selem);
351	/* Publish sk_storage to sk. sk->sk_lock cannot be acquired.
352	* Hence, atomic ops is used to set sk->sk_bpf_storage
353	* from NULL to the newly allocated sk_storage ptr.
354	*
355	* From now on, the sk->sk_bpf_storage pointer is protected
356	* by the sk_storage->lock. Hence, when freeing
357	* the sk->sk_bpf_storage, the sk_storage->lock must
358	* be held before setting sk->sk_bpf_storage to NULL.
359	*/
360	prev_sk_storage = cmpxchg((struct bpf_sk_storage **)&sk->sk_bpf_storage,
361	NULL, sk_storage);
362	if (unlikely(prev_sk_storage)) {
363	selem_unlink_map(first_selem);
364	err = -EAGAIN;
365	goto uncharge;
366
367	/* Note that even first_selem was linked to smap's
368	* bucket->list, first_selem can be freed immediately
369	* (instead of kfree_rcu) because
370	* bpf_sk_storage_map_free() does a
371	* synchronize_rcu() before walking the bucket->list.
372	* Hence, no one is accessing selem from the
373	* bucket->list under rcu_read_lock().
374	*/
375	}
376
377	return 0;
378
379	uncharge:
380	kfree(sk_storage);
381	atomic_sub(sizeof(*sk_storage), &sk->sk_omem_alloc);
382	return err;
383	}
384
385	/* sk cannot be going away because it is linking new elem
386	* to sk->sk_bpf_storage. (i.e. sk->sk_refcnt cannot be 0).
387	* Otherwise, it will become a leak (and other memory issues
388	* during map destruction).
389	*/
390	static struct bpf_sk_storage_data sk_storage_update(struct sock sk,
391	struct bpf_map *map,
392	void *value,
393	u64 map_flags)
394	{
395	struct bpf_sk_storage_data *old_sdata = NULL;
396	struct bpf_sk_storage_elem *selem;
397	struct bpf_sk_storage *sk_storage;
398	struct bpf_sk_storage_map *smap;
399	int err;
400
401	/* BPF_EXIST and BPF_NOEXIST cannot be both set */
402	if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST) \|\|
403	/* BPF_F_LOCK can only be used in a value with spin_lock */
404	unlikely((map_flags & BPF_F_LOCK) && !map_value_has_spin_lock(map)))
405	return ERR_PTR(-EINVAL);
406
407	smap = (struct bpf_sk_storage_map *)map;
408	sk_storage = rcu_dereference(sk->sk_bpf_storage);
409	if (!sk_storage \|\| hlist_empty(&sk_storage->list)) {
410	/* Very first elem for this sk */
411	err = check_flags(NULL, map_flags);
412	if (err)
413	return ERR_PTR(err);
414
415	selem = selem_alloc(smap, sk, value, true);
416	if (!selem)
417	return ERR_PTR(-ENOMEM);
418
419	err = sk_storage_alloc(sk, smap, selem);
420	if (err) {
421	kfree(selem);
422	atomic_sub(smap->elem_size, &sk->sk_omem_alloc);
423	return ERR_PTR(err);
424	}
425
426	return SDATA(selem);
427	}
428
429	if ((map_flags & BPF_F_LOCK) && !(map_flags & BPF_NOEXIST)) {
430	/* Hoping to find an old_sdata to do inline update
431	* such that it can avoid taking the sk_storage->lock
432	* and changing the lists.
433	*/
434	old_sdata = __sk_storage_lookup(sk_storage, smap, false);
435	err = check_flags(old_sdata, map_flags);
436	if (err)
437	return ERR_PTR(err);
438	if (old_sdata && selem_linked_to_sk(SELEM(old_sdata))) {
439	copy_map_value_locked(map, old_sdata->data,
440	value, false);
441	return old_sdata;
442	}
443	}
444
445	raw_spin_lock_bh(&sk_storage->lock);
446
447	/* Recheck sk_storage->list under sk_storage->lock */
448	if (unlikely(hlist_empty(&sk_storage->list))) {
449	/* A parallel del is happening and sk_storage is going
450	* away. It has just been checked before, so very
451	* unlikely. Return instead of retry to keep things
452	* simple.
453	*/
454	err = -EAGAIN;
455	goto unlock_err;
456	}
457
458	old_sdata = __sk_storage_lookup(sk_storage, smap, false);
459	err = check_flags(old_sdata, map_flags);
460	if (err)
461	goto unlock_err;
462
463	if (old_sdata && (map_flags & BPF_F_LOCK)) {
464	copy_map_value_locked(map, old_sdata->data, value, false);
465	selem = SELEM(old_sdata);
466	goto unlock;
467	}
468
469	/* sk_storage->lock is held. Hence, we are sure
470	* we can unlink and uncharge the old_sdata successfully
471	* later. Hence, instead of charging the new selem now
472	* and then uncharge the old selem later (which may cause
473	* a potential but unnecessary charge failure), avoid taking
474	* a charge at all here (the "!old_sdata" check) and the
475	* old_sdata will not be uncharged later during __selem_unlink_sk().
476	*/
477	selem = selem_alloc(smap, sk, value, !old_sdata);
478	if (!selem) {
479	err = -ENOMEM;
480	goto unlock_err;
481	}
482
483	/* First, link the new selem to the map */
484	selem_link_map(smap, selem);
485
486	/* Second, link (and publish) the new selem to sk_storage */
487	__selem_link_sk(sk_storage, selem);
488
489	/* Third, remove old selem, SELEM(old_sdata) */
490	if (old_sdata) {
491	selem_unlink_map(SELEM(old_sdata));
492	__selem_unlink_sk(sk_storage, SELEM(old_sdata), false);
493	}
494
495	unlock:
496	raw_spin_unlock_bh(&sk_storage->lock);
497	return SDATA(selem);
498
499	unlock_err:
500	raw_spin_unlock_bh(&sk_storage->lock);
501	return ERR_PTR(err);
502	}	43	}
503		44
504	static int sk_storage_delete(struct sock sk, struct bpf_map map)	45	static int sk_storage_delete(struct sock sk, struct bpf_map map)
505	{	46	{
506	struct bpf_sk_storage_data *sdata;	47	struct bpf_local_storage_data *sdata;
507		48
508	sdata = sk_storage_lookup(sk, map, false);	49	sdata = sk_storage_lookup(sk, map, false);
509	if (!sdata)	50	if (!sdata)
510	return -ENOENT;	51	return -ENOENT;
511		52
512	selem_unlink(SELEM(sdata));	53	bpf_selem_unlink(SELEM(sdata));
513		54
514	return 0;	55	return 0;
515	}	56	}
516		57
517	static u16 cache_idx_get(void)
518	{
519	u64 min_usage = U64_MAX;
520	u16 i, res = 0;
521
522	spin_lock(&cache_idx_lock);
523
524	for (i = 0; i < BPF_SK_STORAGE_CACHE_SIZE; i++) {
525	if (cache_idx_usage_counts[i] < min_usage) {
526	min_usage = cache_idx_usage_counts[i];
527	res = i;
528
529	/* Found a free cache_idx */
530	if (!min_usage)
531	break;
532	}
533	}
534	cache_idx_usage_counts[res]++;
535
536	spin_unlock(&cache_idx_lock);
537
538	return res;
539	}
540
541	static void cache_idx_free(u16 idx)
542	{
543	spin_lock(&cache_idx_lock);
544	cache_idx_usage_counts[idx]--;
545	spin_unlock(&cache_idx_lock);
546	}
547
548	/* Called by __sk_destruct() & bpf_sk_storage_clone() */	58	/* Called by __sk_destruct() & bpf_sk_storage_clone() */
549	void bpf_sk_storage_free(struct sock *sk)	59	void bpf_sk_storage_free(struct sock *sk)
550	{	60	{
551	struct bpf_sk_storage_elem *selem;	61	struct bpf_local_storage_elem *selem;
552	struct bpf_sk_storage *sk_storage;	62	struct bpf_local_storage *sk_storage;
553	bool free_sk_storage = false;	63	bool free_sk_storage = false;
554	struct hlist_node *n;	64	struct hlist_node *n;
555		65
@@ -565,7 +75,7 @@ void bpf_sk_storage_free(struct sock *sk) · 9 unmodified lines
565	* Thus, no elem can be added-to or deleted-from the	75	* Thus, no elem can be added-to or deleted-from the
566	* sk_storage->list by the bpf_prog or by the bpf-map's syscall.	76	* sk_storage->list by the bpf_prog or by the bpf-map's syscall.
567	*	77	*
568	* It is racing with bpf_sk_storage_map_free() alone	78	* It is racing with bpf_local_storage_map_free() alone
569	* when unlinking elem from the sk_storage->list and	79	* when unlinking elem from the sk_storage->list and
570	* the map's bucket->list.	80	* the map's bucket->list.
571	*/	81	*/
@@ -574,8 +84,9 @@ void bpf_sk_storage_free(struct sock *sk) · 2 unmodified lines
574	/* Always unlink from map before unlinking from	84	/* Always unlink from map before unlinking from
575	* sk_storage.	85	* sk_storage.
576	*/	86	*/
577	selem_unlink_map(selem);	87	bpf_selem_unlink_map(selem);
578	free_sk_storage = __selem_unlink_sk(sk_storage, selem, true);	88	free_sk_storage = bpf_selem_unlink_storage_nolock(sk_storage,
		89	selem, true);
579	}	90	}
580	raw_spin_unlock_bh(&sk_storage->lock);	91	raw_spin_unlock_bh(&sk_storage->lock);
581	rcu_read_unlock();	92	rcu_read_unlock();
@@ -584,132 +95,24 @@ void bpf_sk_storage_free(struct sock *sk) · 2 unmodified lines
584	kfree_rcu(sk_storage, rcu);	95	kfree_rcu(sk_storage, rcu);
585	}	96	}
586		97
587	static void bpf_sk_storage_map_free(struct bpf_map *map)	98	static void sk_storage_map_free(struct bpf_map *map)
588	{
589	struct bpf_sk_storage_elem *selem;
590	struct bpf_sk_storage_map *smap;
591	struct bucket *b;
592	unsigned int i;
593
594	smap = (struct bpf_sk_storage_map *)map;
595
596	cache_idx_free(smap->cache_idx);
597
598	/* Note that this map might be concurrently cloned from
599	* bpf_sk_storage_clone. Wait for any existing bpf_sk_storage_clone
600	* RCU read section to finish before proceeding. New RCU
601	* read sections should be prevented via bpf_map_inc_not_zero.
602	*/
603	synchronize_rcu();
604
605	/* bpf prog and the userspace can no longer access this map
606	* now. No new selem (of this map) can be added
607	* to the sk->sk_bpf_storage or to the map bucket's list.
608	*
609	* The elem of this map can be cleaned up here
610	* or
611	* by bpf_sk_storage_free() during __sk_destruct().
612	*/
613	for (i = 0; i < (1U << smap->bucket_log); i++) {
614	b = &smap->buckets[i];
615
616	rcu_read_lock();
617	/* No one is adding to b->list now */
618	while ((selem = hlist_entry_safe(rcu_dereference_raw(hlist_first_rcu(&b->list)),
619	struct bpf_sk_storage_elem,
620	map_node))) {
621	selem_unlink(selem);
622	cond_resched_rcu();
623	}
624	rcu_read_unlock();
625	}
626
627	/* bpf_sk_storage_free() may still need to access the map.
628	* e.g. bpf_sk_storage_free() has unlinked selem from the map
629	* which then made the above while((selem = ...)) loop
630	* exited immediately.
631	*
632	* However, the bpf_sk_storage_free() still needs to access
633	* the smap->elem_size to do the uncharging in
634	* __selem_unlink_sk().
635	*
636	* Hence, wait another rcu grace period for the
637	* bpf_sk_storage_free() to finish.
638	*/
639	synchronize_rcu();
640
641	kvfree(smap->buckets);
642	kfree(map);
643	}
644
645	/* U16_MAX is much more than enough for sk local storage
646	* considering a tcp_sock is ~2k.
647	*/
648	#define MAX_VALUE_SIZE \
649	min_t(u32, \
650	(KMALLOC_MAX_SIZE - MAX_BPF_STACK - sizeof(struct bpf_sk_storage_elem)), \
651	(U16_MAX - sizeof(struct bpf_sk_storage_elem)))
652
653	static int bpf_sk_storage_map_alloc_check(union bpf_attr *attr)
654	{	99	{
655	if (attr->map_flags & ~SK_STORAGE_CREATE_FLAG_MASK \|\|	100	struct bpf_local_storage_map *smap;
656	!(attr->map_flags & BPF_F_NO_PREALLOC) \|\|
657	attr->max_entries \|\|
658	attr->key_size != sizeof(int) \|\| !attr->value_size \|\|
659	/* Enforce BTF for userspace sk dumping */
660	!attr->btf_key_type_id \|\| !attr->btf_value_type_id)
661	return -EINVAL;
662
663	if (!bpf_capable())
664	return -EPERM;
665		101
666	if (attr->value_size > MAX_VALUE_SIZE)	102	smap = (struct bpf_local_storage_map *)map;
667	return -E2BIG;	103	bpf_local_storage_cache_idx_free(&sk_cache, smap->cache_idx);
668		104	bpf_local_storage_map_free(smap);
669	return 0;
670	}	105	}
671		106
672	static struct bpf_map bpf_sk_storage_map_alloc(union bpf_attr attr)	107	static struct bpf_map sk_storage_map_alloc(union bpf_attr attr)
673	{	108	{
674	struct bpf_sk_storage_map *smap;	109	struct bpf_local_storage_map *smap;
675	unsigned int i;
676	u32 nbuckets;
677	u64 cost;
678	int ret;
679
680	smap = kzalloc(sizeof(*smap), GFP_USER \| __GFP_NOWARN);
681	if (!smap)
682	return ERR_PTR(-ENOMEM);
683	bpf_map_init_from_attr(&smap->map, attr);
684
685	nbuckets = roundup_pow_of_two(num_possible_cpus());
686	/* Use at least 2 buckets, select_bucket() is undefined behavior with 1 bucket */
687	nbuckets = max_t(u32, 2, nbuckets);
688	smap->bucket_log = ilog2(nbuckets);
689	cost = sizeof(smap->buckets) nbuckets + sizeof(*smap);
690
691	ret = bpf_map_charge_init(&smap->map.memory, cost);
692	if (ret < 0) {
693	kfree(smap);
694	return ERR_PTR(ret);
695	}
696
697	smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets,
698	GFP_USER \| __GFP_NOWARN);
699	if (!smap->buckets) {
700	bpf_map_charge_finish(&smap->map.memory);
701	kfree(smap);
702	return ERR_PTR(-ENOMEM);
703	}
704		110
705	for (i = 0; i < nbuckets; i++) {	111	smap = bpf_local_storage_map_alloc(attr);
706	INIT_HLIST_HEAD(&smap->buckets[i].list);	112	if (IS_ERR(smap))
707	raw_spin_lock_init(&smap->buckets[i].lock);	113	return ERR_CAST(smap);
708	}
709
710	smap->elem_size = sizeof(struct bpf_sk_storage_elem) + attr->value_size;
711	smap->cache_idx = cache_idx_get();
712		114
		115	smap->cache_idx = bpf_local_storage_cache_idx_get(&sk_cache);
713	return &smap->map;	116	return &smap->map;
714	}	117	}
715		118
@@ -719,26 +122,9 @@ static int notsupp_get_next_key(struct bpf_map map, void key, · 3 unmodified lines
719	return -ENOTSUPP;	122	return -ENOTSUPP;
720	}	123	}
721		124
722	static int bpf_sk_storage_map_check_btf(const struct bpf_map *map,
723	const struct btf *btf,
724	const struct btf_type *key_type,
725	const struct btf_type *value_type)
726	{
727	u32 int_data;
728
729	if (BTF_INFO_KIND(key_type->info) != BTF_KIND_INT)
730	return -EINVAL;
731
732	int_data = (u32 )(key_type + 1);
733	if (BTF_INT_BITS(int_data) != 32 \|\| BTF_INT_OFFSET(int_data))
734	return -EINVAL;
735
736	return 0;
737	}
738
739	static void bpf_fd_sk_storage_lookup_elem(struct bpf_map map, void *key)	125	static void bpf_fd_sk_storage_lookup_elem(struct bpf_map map, void *key)
740	{	126	{
741	struct bpf_sk_storage_data *sdata;	127	struct bpf_local_storage_data *sdata;
742	struct socket *sock;	128	struct socket *sock;
743	int fd, err;	129	int fd, err;
744		130
@@ -756,14 +142,16 @@ static void bpf_fd_sk_storage_lookup_elem(struct bpf_map map, void *key) · 11 unmodified lines
756	static int bpf_fd_sk_storage_update_elem(struct bpf_map map, void key,	142	static int bpf_fd_sk_storage_update_elem(struct bpf_map map, void key,
757	void *value, u64 map_flags)	143	void *value, u64 map_flags)
758	{	144	{
759	struct bpf_sk_storage_data *sdata;	145	struct bpf_local_storage_data *sdata;
760	struct socket *sock;	146	struct socket *sock;
761	int fd, err;	147	int fd, err;
762		148
763	fd = (int )key;	149	fd = (int )key;
764	sock = sockfd_lookup(fd, &err);	150	sock = sockfd_lookup(fd, &err);
765	if (sock) {	151	if (sock) {
766	sdata = sk_storage_update(sock->sk, map, value, map_flags);	152	sdata = bpf_local_storage_update(
		153	sock->sk, (struct bpf_local_storage_map *)map, value,
		154	map_flags);
767	sockfd_put(sock);	155	sockfd_put(sock);
768	return PTR_ERR_OR_ZERO(sdata);	156	return PTR_ERR_OR_ZERO(sdata);
769	}	157	}
@@ -787,14 +175,14 @@ static int bpf_fd_sk_storage_delete_elem(struct bpf_map map, void key) · 17 unmodified lines
787	return err;	175	return err;
788	}	176	}
789		177
790	static struct bpf_sk_storage_elem *	178	static struct bpf_local_storage_elem *
791	bpf_sk_storage_clone_elem(struct sock *newsk,	179	bpf_sk_storage_clone_elem(struct sock *newsk,
792	struct bpf_sk_storage_map *smap,	180	struct bpf_local_storage_map *smap,
793	struct bpf_sk_storage_elem *selem)	181	struct bpf_local_storage_elem *selem)
794	{	182	{
795	struct bpf_sk_storage_elem *copy_selem;	183	struct bpf_local_storage_elem *copy_selem;
796		184
797	copy_selem = selem_alloc(smap, newsk, NULL, true);	185	copy_selem = bpf_selem_alloc(smap, newsk, NULL, true);
798	if (!copy_selem)	186	if (!copy_selem)
799	return NULL;	187	return NULL;
800		188
@@ -810,9 +198,9 @@ bpf_sk_storage_clone_elem(struct sock *newsk, · 9 unmodified lines
810		198
811	int bpf_sk_storage_clone(const struct sock sk, struct sock newsk)	199	int bpf_sk_storage_clone(const struct sock sk, struct sock newsk)
812	{	200	{
813	struct bpf_sk_storage *new_sk_storage = NULL;	201	struct bpf_local_storage *new_sk_storage = NULL;
814	struct bpf_sk_storage *sk_storage;	202	struct bpf_local_storage *sk_storage;
815	struct bpf_sk_storage_elem *selem;	203	struct bpf_local_storage_elem *selem;
816	int ret = 0;	204	int ret = 0;
817		205
818	RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL);	206	RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL);
@@ -824,8 +212,8 @@ int bpf_sk_storage_clone(const struct sock sk, struct sock newsk) · 5 unmodified lines
824	goto out;	212	goto out;
825		213
826	hlist_for_each_entry_rcu(selem, &sk_storage->list, snode) {	214	hlist_for_each_entry_rcu(selem, &sk_storage->list, snode) {
827	struct bpf_sk_storage_elem *copy_selem;	215	struct bpf_local_storage_elem *copy_selem;
828	struct bpf_sk_storage_map *smap;	216	struct bpf_local_storage_map *smap;
829	struct bpf_map *map;	217	struct bpf_map *map;
830		218
831	smap = rcu_dereference(SDATA(selem)->smap);	219	smap = rcu_dereference(SDATA(selem)->smap);
@@ -833,7 +221,7 @@ int bpf_sk_storage_clone(const struct sock sk, struct sock newsk) · 1 unmodified line
833	continue;	221	continue;
834		222
835	/* Note that for lockless listeners adding new element	223	/* Note that for lockless listeners adding new element
836	* here can race with cleanup in bpf_sk_storage_map_free.	224	* here can race with cleanup in bpf_local_storage_map_free.
837	* Try to grab map refcnt to make sure that it's still	225	* Try to grab map refcnt to make sure that it's still
838	* alive and prevent concurrent removal.	226	* alive and prevent concurrent removal.
839	*/	227	*/
@@ -849,10 +237,10 @@ int bpf_sk_storage_clone(const struct sock sk, struct sock newsk) · 9 unmodified lines
849	}	237	}
850		238
851	if (new_sk_storage) {	239	if (new_sk_storage) {
852	selem_link_map(smap, copy_selem);	240	bpf_selem_link_map(smap, copy_selem);
853	__selem_link_sk(new_sk_storage, copy_selem);	241	bpf_selem_link_storage_nolock(new_sk_storage, copy_selem);
854	} else {	242	} else {
855	ret = sk_storage_alloc(newsk, smap, copy_selem);	243	ret = bpf_local_storage_alloc(newsk, smap, copy_selem);
856	if (ret) {	244	if (ret) {
857	kfree(copy_selem);	245	kfree(copy_selem);
858	atomic_sub(smap->elem_size,	246	atomic_sub(smap->elem_size,
@@ -861,7 +249,8 @@ int bpf_sk_storage_clone(const struct sock sk, struct sock newsk) · 2 unmodified lines
861	goto out;	249	goto out;
862	}	250	}
863		251
864	new_sk_storage = rcu_dereference(copy_selem->sk_storage);	252	new_sk_storage =
		253	rcu_dereference(copy_selem->local_storage);
865	}	254	}
866	bpf_map_put(map);	255	bpf_map_put(map);
867	}	256	}
@@ -879,7 +268,7 @@ out: · 11 unmodified lines
879	BPF_CALL_4(bpf_sk_storage_get, struct bpf_map , map, struct sock , sk,	268	BPF_CALL_4(bpf_sk_storage_get, struct bpf_map , map, struct sock , sk,
880	void *, value, u64, flags)	269	void *, value, u64, flags)
881	{	270	{
882	struct bpf_sk_storage_data *sdata;	271	struct bpf_local_storage_data *sdata;
883		272
884	if (flags > BPF_SK_STORAGE_GET_F_CREATE)	273	if (flags > BPF_SK_STORAGE_GET_F_CREATE)
885	return (unsigned long)NULL;	274	return (unsigned long)NULL;
@@ -895,7 +284,9 @@ BPF_CALL_4(bpf_sk_storage_get, struct bpf_map , map, struct sock , sk, · 9 unmodified lines
895	* destruction).	284	* destruction).
896	*/	285	*/
897	refcount_inc_not_zero(&sk->sk_refcnt)) {	286	refcount_inc_not_zero(&sk->sk_refcnt)) {
898	sdata = sk_storage_update(sk, map, value, BPF_NOEXIST);	287	sdata = bpf_local_storage_update(
		288	sk, (struct bpf_local_storage_map *)map, value,
		289	BPF_NOEXIST);
899	/* sk must be a fullsock (guaranteed by verifier),	290	/* sk must be a fullsock (guaranteed by verifier),
900	* so sock_gen_put() is unnecessary.	291	* so sock_gen_put() is unnecessary.
901	*/	292	*/
@@ -920,18 +311,44 @@ BPF_CALL_2(bpf_sk_storage_delete, struct bpf_map , map, struct sock , sk) · 18 unmodified lines
920	return -ENOENT;	311	return -ENOENT;
921	}	312	}
922		313
		314	static int sk_storage_charge(struct bpf_local_storage_map *smap,
		315	void *owner, u32 size)
		316	{
		317	return omem_charge(owner, size);
		318	}
		319
		320	static void sk_storage_uncharge(struct bpf_local_storage_map *smap,
		321	void *owner, u32 size)
		322	{
		323	struct sock *sk = owner;
		324
		325	atomic_sub(size, &sk->sk_omem_alloc);
		326	}
		327
		328	static struct bpf_local_storage __rcu **
		329	sk_storage_ptr(void *owner)
		330	{
		331	struct sock *sk = owner;
		332
		333	return &sk->sk_bpf_storage;
		334	}
		335
923	static int sk_storage_map_btf_id;	336	static int sk_storage_map_btf_id;
924	const struct bpf_map_ops sk_storage_map_ops = {	337	const struct bpf_map_ops sk_storage_map_ops = {
925	.map_alloc_check = bpf_sk_storage_map_alloc_check,	338	.map_meta_equal = bpf_map_meta_equal,
926	.map_alloc = bpf_sk_storage_map_alloc,	339	.map_alloc_check = bpf_local_storage_map_alloc_check,
927	.map_free = bpf_sk_storage_map_free,	340	.map_alloc = sk_storage_map_alloc,
		341	.map_free = sk_storage_map_free,
928	.map_get_next_key = notsupp_get_next_key,	342	.map_get_next_key = notsupp_get_next_key,
929	.map_lookup_elem = bpf_fd_sk_storage_lookup_elem,	343	.map_lookup_elem = bpf_fd_sk_storage_lookup_elem,
930	.map_update_elem = bpf_fd_sk_storage_update_elem,	344	.map_update_elem = bpf_fd_sk_storage_update_elem,
931	.map_delete_elem = bpf_fd_sk_storage_delete_elem,	345	.map_delete_elem = bpf_fd_sk_storage_delete_elem,
932	.map_check_btf = bpf_sk_storage_map_check_btf,	346	.map_check_btf = bpf_local_storage_map_check_btf,
933	.map_btf_name = "bpf_sk_storage_map",	347	.map_btf_name = "bpf_local_storage_map",
934	.map_btf_id = &sk_storage_map_btf_id,	348	.map_btf_id = &sk_storage_map_btf_id,
		349	.map_local_storage_charge = sk_storage_charge,
		350	.map_local_storage_uncharge = sk_storage_uncharge,
		351	.map_owner_storage_ptr = sk_storage_ptr,
935	};	352	};
936		353
937	const struct bpf_func_proto bpf_sk_storage_get_proto = {	354	const struct bpf_func_proto bpf_sk_storage_get_proto = {
@@ -962,6 +379,30 @@ const struct bpf_func_proto bpf_sk_storage_delete_proto = { · 24 unmodified lines
962	.arg2_type = ARG_PTR_TO_SOCKET,	379	.arg2_type = ARG_PTR_TO_SOCKET,
963	};	380	};
964		381
		382	BTF_ID_LIST(sk_storage_btf_ids)
		383	BTF_ID_UNUSED
		384	BTF_ID(struct, sock)
		385
		386	const struct bpf_func_proto sk_storage_get_btf_proto = {
		387	.func = bpf_sk_storage_get,
		388	.gpl_only = false,
		389	.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
		390	.arg1_type = ARG_CONST_MAP_PTR,
		391	.arg2_type = ARG_PTR_TO_BTF_ID,
		392	.arg3_type = ARG_PTR_TO_MAP_VALUE_OR_NULL,
		393	.arg4_type = ARG_ANYTHING,
		394	.btf_id = sk_storage_btf_ids,
		395	};
		396
		397	const struct bpf_func_proto sk_storage_delete_btf_proto = {
		398	.func = bpf_sk_storage_delete,
		399	.gpl_only = false,
		400	.ret_type = RET_INTEGER,
		401	.arg1_type = ARG_CONST_MAP_PTR,
		402	.arg2_type = ARG_PTR_TO_BTF_ID,
		403	.btf_id = sk_storage_btf_ids,
		404	};
		405
965	struct bpf_sk_storage_diag {	406	struct bpf_sk_storage_diag {
966	u32 nr_maps;	407	u32 nr_maps;
967	struct bpf_map *maps[];	408	struct bpf_map *maps[];
@@ -1022,7 +463,7 @@ bpf_sk_storage_diag_alloc(const struct nlattr *nla_stgs) · 54 unmodified lines
1022	u32 nr_maps = 0;	463	u32 nr_maps = 0;
1023	int rem, err;	464	int rem, err;
1024		465
1025	/* bpf_sk_storage_map is currently limited to CAP_SYS_ADMIN as	466	/* bpf_local_storage_map is currently limited to CAP_SYS_ADMIN as
1026	* the map_alloc_check() side also does.	467	* the map_alloc_check() side also does.
1027	*/	468	*/
1028	if (!bpf_capable())	469	if (!bpf_capable())
@@ -1072,13 +513,13 @@ err_free: · 43 unmodified lines
1072	}	513	}
1073	EXPORT_SYMBOL_GPL(bpf_sk_storage_diag_alloc);	514	EXPORT_SYMBOL_GPL(bpf_sk_storage_diag_alloc);
1074		515
1075	static int diag_get(struct bpf_sk_storage_data sdata, struct sk_buff skb)	516	static int diag_get(struct bpf_local_storage_data sdata, struct sk_buff skb)
1076	{	517	{
1077	struct nlattr nla_stg, nla_value;	518	struct nlattr nla_stg, nla_value;
1078	struct bpf_sk_storage_map *smap;	519	struct bpf_local_storage_map *smap;
1079		520
1080	/* It cannot exceed max nlattr's payload */	521	/* It cannot exceed max nlattr's payload */
1081	BUILD_BUG_ON(U16_MAX - NLA_HDRLEN < MAX_VALUE_SIZE);	522	BUILD_BUG_ON(U16_MAX - NLA_HDRLEN < BPF_LOCAL_STORAGE_MAX_VALUE_SIZE);
1082		523
1083	nla_stg = nla_nest_start(skb, SK_DIAG_BPF_STORAGE);	524	nla_stg = nla_nest_start(skb, SK_DIAG_BPF_STORAGE);
1084	if (!nla_stg)	525	if (!nla_stg)
@@ -1114,9 +555,9 @@ static int bpf_sk_storage_diag_put_all(struct sock sk, struct sk_buff skb, · 29 unmodified lines
1114	{	555	{
1115	/* stg_array_type (e.g. INET_DIAG_BPF_SK_STORAGES) */	556	/* stg_array_type (e.g. INET_DIAG_BPF_SK_STORAGES) */
1116	unsigned int diag_size = nla_total_size(0);	557	unsigned int diag_size = nla_total_size(0);
1117	struct bpf_sk_storage *sk_storage;	558	struct bpf_local_storage *sk_storage;
1118	struct bpf_sk_storage_elem *selem;	559	struct bpf_local_storage_elem *selem;
1119	struct bpf_sk_storage_map *smap;	560	struct bpf_local_storage_map *smap;
1120	struct nlattr *nla_stgs;	561	struct nlattr *nla_stgs;
1121	unsigned int saved_len;	562	unsigned int saved_len;
1122	int err = 0;	563	int err = 0;
@@ -1169,8 +610,8 @@ int bpf_sk_storage_diag_put(struct bpf_sk_storage_diag *diag, · 46 unmodified lines
1169	{	610	{
1170	/* stg_array_type (e.g. INET_DIAG_BPF_SK_STORAGES) */	611	/* stg_array_type (e.g. INET_DIAG_BPF_SK_STORAGES) */
1171	unsigned int diag_size = nla_total_size(0);	612	unsigned int diag_size = nla_total_size(0);
1172	struct bpf_sk_storage *sk_storage;	613	struct bpf_local_storage *sk_storage;
1173	struct bpf_sk_storage_data *sdata;	614	struct bpf_local_storage_data *sdata;
1174	struct nlattr *nla_stgs;	615	struct nlattr *nla_stgs;
1175	unsigned int saved_len;	616	unsigned int saved_len;
1176	int err = 0;	617	int err = 0;
@@ -1197,8 +638,8 @@ int bpf_sk_storage_diag_put(struct bpf_sk_storage_diag *diag, · 20 unmodified lines
1197		638
1198	saved_len = skb->len;	639	saved_len = skb->len;
1199	for (i = 0; i < diag->nr_maps; i++) {	640	for (i = 0; i < diag->nr_maps; i++) {
1200	sdata = __sk_storage_lookup(sk_storage,	641	sdata = bpf_local_storage_lookup(sk_storage,
1201	(struct bpf_sk_storage_map *)diag->maps[i],	642	(struct bpf_local_storage_map *)diag->maps[i],
1202	false);	643	false);
1203		644
1204	if (!sdata)	645	if (!sdata)
@@ -1235,19 +676,19 @@ struct bpf_iter_seq_sk_storage_map_info { · 30 unmodified lines
1235	unsigned skip_elems;	676	unsigned skip_elems;
1236	};	677	};
1237		678
1238	static struct bpf_sk_storage_elem *	679	static struct bpf_local_storage_elem *
1239	bpf_sk_storage_map_seq_find_next(struct bpf_iter_seq_sk_storage_map_info *info,	680	bpf_sk_storage_map_seq_find_next(struct bpf_iter_seq_sk_storage_map_info *info,
1240	struct bpf_sk_storage_elem *prev_selem)	681	struct bpf_local_storage_elem *prev_selem)
1241	{	682	{
1242	struct bpf_sk_storage *sk_storage;	683	struct bpf_local_storage *sk_storage;
1243	struct bpf_sk_storage_elem *selem;	684	struct bpf_local_storage_elem *selem;
1244	u32 skip_elems = info->skip_elems;	685	u32 skip_elems = info->skip_elems;
1245	struct bpf_sk_storage_map *smap;	686	struct bpf_local_storage_map *smap;
1246	u32 bucket_id = info->bucket_id;	687	u32 bucket_id = info->bucket_id;
1247	u32 i, count, n_buckets;	688	u32 i, count, n_buckets;
1248	struct bucket *b;	689	struct bpf_local_storage_map_bucket *b;
1249		690
1250	smap = (struct bpf_sk_storage_map *)info->map;	691	smap = (struct bpf_local_storage_map *)info->map;
1251	n_buckets = 1U << smap->bucket_log;	692	n_buckets = 1U << smap->bucket_log;
1252	if (bucket_id >= n_buckets)	693	if (bucket_id >= n_buckets)
1253	return NULL;	694	return NULL;
@@ -1257,7 +698,7 @@ bpf_sk_storage_map_seq_find_next(struct bpf_iter_seq_sk_storage_map_info *info, · 3 unmodified lines
1257	count = 0;	698	count = 0;
1258	while (selem) {	699	while (selem) {
1259	selem = hlist_entry_safe(selem->map_node.next,	700	selem = hlist_entry_safe(selem->map_node.next,
1260	struct bpf_sk_storage_elem, map_node);	701	struct bpf_local_storage_elem, map_node);
1261	if (!selem) {	702	if (!selem) {
1262	/* not found, unlock and go to the next bucket */	703	/* not found, unlock and go to the next bucket */
1263	b = &smap->buckets[bucket_id++];	704	b = &smap->buckets[bucket_id++];
@@ -1265,7 +706,7 @@ bpf_sk_storage_map_seq_find_next(struct bpf_iter_seq_sk_storage_map_info *info, · 1 unmodified line
1265	skip_elems = 0;	706	skip_elems = 0;
1266	break;	707	break;
1267	}	708	}
1268	sk_storage = rcu_dereference_raw(selem->sk_storage);	709	sk_storage = rcu_dereference_raw(selem->local_storage);
1269	if (sk_storage) {	710	if (sk_storage) {
1270	info->skip_elems = skip_elems + count;	711	info->skip_elems = skip_elems + count;
1271	return selem;	712	return selem;
@@ -1278,7 +719,7 @@ bpf_sk_storage_map_seq_find_next(struct bpf_iter_seq_sk_storage_map_info *info, · 6 unmodified lines
1278	raw_spin_lock_bh(&b->lock);	719	raw_spin_lock_bh(&b->lock);
1279	count = 0;	720	count = 0;
1280	hlist_for_each_entry(selem, &b->list, map_node) {	721	hlist_for_each_entry(selem, &b->list, map_node) {
1281	sk_storage = rcu_dereference_raw(selem->sk_storage);	722	sk_storage = rcu_dereference_raw(selem->local_storage);
1282	if (sk_storage && count >= skip_elems) {	723	if (sk_storage && count >= skip_elems) {
1283	info->bucket_id = i;	724	info->bucket_id = i;
1284	info->skip_elems = count;	725	info->skip_elems = count;
@@ -1297,7 +738,7 @@ bpf_sk_storage_map_seq_find_next(struct bpf_iter_seq_sk_storage_map_info *info, · 12 unmodified lines
1297		738
1298	static void bpf_sk_storage_map_seq_start(struct seq_file seq, loff_t *pos)	739	static void bpf_sk_storage_map_seq_start(struct seq_file seq, loff_t *pos)
1299	{	740	{
1300	struct bpf_sk_storage_elem *selem;	741	struct bpf_local_storage_elem *selem;
1301		742
1302	selem = bpf_sk_storage_map_seq_find_next(seq->private, NULL);	743	selem = bpf_sk_storage_map_seq_find_next(seq->private, NULL);
1303	if (!selem)	744	if (!selem)
@@ -1330,11 +771,11 @@ DEFINE_BPF_ITER_FUNC(bpf_sk_storage_map, struct bpf_iter_meta *meta, · 26 unmodified lines
1330	void *value)	771	void *value)
1331		772
1332	static int __bpf_sk_storage_map_seq_show(struct seq_file *seq,	773	static int __bpf_sk_storage_map_seq_show(struct seq_file *seq,
1333	struct bpf_sk_storage_elem *selem)	774	struct bpf_local_storage_elem *selem)
1334	{	775	{
1335	struct bpf_iter_seq_sk_storage_map_info *info = seq->private;	776	struct bpf_iter_seq_sk_storage_map_info *info = seq->private;
1336	struct bpf_iter__bpf_sk_storage_map ctx = {};	777	struct bpf_iter__bpf_sk_storage_map ctx = {};
1337	struct bpf_sk_storage *sk_storage;	778	struct bpf_local_storage *sk_storage;
1338	struct bpf_iter_meta meta;	779	struct bpf_iter_meta meta;
1339	struct bpf_prog *prog;	780	struct bpf_prog *prog;
1340	int ret = 0;	781	int ret = 0;
@@ -1345,8 +786,8 @@ static int __bpf_sk_storage_map_seq_show(struct seq_file *seq, · 4 unmodified lines
1345	ctx.meta = &meta;	786	ctx.meta = &meta;
1346	ctx.map = info->map;	787	ctx.map = info->map;
1347	if (selem) {	788	if (selem) {
1348	sk_storage = rcu_dereference_raw(selem->sk_storage);	789	sk_storage = rcu_dereference_raw(selem->local_storage);
1349	ctx.sk = sk_storage->sk;	790	ctx.sk = sk_storage->owner;
1350	ctx.value = SDATA(selem)->data;	791	ctx.value = SDATA(selem)->data;
1351	}	792	}
1352	ret = bpf_iter_run_prog(prog, &ctx);	793	ret = bpf_iter_run_prog(prog, &ctx);
@@ -1363,13 +804,13 @@ static int bpf_sk_storage_map_seq_show(struct seq_file seq, void v) · 10 unmodified lines
1363	static void bpf_sk_storage_map_seq_stop(struct seq_file seq, void v)	804	static void bpf_sk_storage_map_seq_stop(struct seq_file seq, void v)
1364	{	805	{
1365	struct bpf_iter_seq_sk_storage_map_info *info = seq->private;	806	struct bpf_iter_seq_sk_storage_map_info *info = seq->private;
1366	struct bpf_sk_storage_map *smap;	807	struct bpf_local_storage_map *smap;
1367	struct bucket *b;	808	struct bpf_local_storage_map_bucket *b;
1368		809
1369	if (!v) {	810	if (!v) {
1370	(void)__bpf_sk_storage_map_seq_show(seq, v);	811	(void)__bpf_sk_storage_map_seq_show(seq, v);
1371	} else {	812	} else {
1372	smap = (struct bpf_sk_storage_map *)info->map;	813	smap = (struct bpf_local_storage_map *)info->map;
1373	b = &smap->buckets[info->bucket_id];	814	b = &smap->buckets[info->bucket_id];
1374	raw_spin_unlock_bh(&b->lock);	815	raw_spin_unlock_bh(&b->lock);
1375	}	816	}
@@ -1437,6 +878,8 @@ static struct bpf_iter_reg bpf_sk_storage_map_reg_info = { · 61 unmodified lines
1437	.target = "bpf_sk_storage_map",	878	.target = "bpf_sk_storage_map",
1438	.attach_target = bpf_iter_attach_map,	879	.attach_target = bpf_iter_attach_map,
1439	.detach_target = bpf_iter_detach_map,	880	.detach_target = bpf_iter_detach_map,
		881	.show_fdinfo = bpf_iter_map_show_fdinfo,
		882	.fill_link_info = bpf_iter_map_fill_link_info,
1440	.ctx_arg_info_size = 2,	883	.ctx_arg_info_size = 2,
1441	.ctx_arg_info = {	884	.ctx_arg_info = {
1442	{ offsetof(struct bpf_iter__bpf_sk_storage_map, sk),	885	{ offsetof(struct bpf_iter__bpf_sk_storage_map, sk),

net/core/filter.c+409−7

@@ -4459,6 +4459,7 @@ static int _bpf_setsockopt(struct sock *sk, int level, int optname, · 4458 unmodified lines
4459	} else {	4459	} else {
4460	struct inet_connection_sock *icsk = inet_csk(sk);	4460	struct inet_connection_sock *icsk = inet_csk(sk);
4461	struct tcp_sock *tp = tcp_sk(sk);	4461	struct tcp_sock *tp = tcp_sk(sk);
		4462	unsigned long timeout;
4462		4463
4463	if (optlen != sizeof(int))	4464	if (optlen != sizeof(int))
4464	return -EINVAL;	4465	return -EINVAL;
@@ -4480,6 +4481,20 @@ static int _bpf_setsockopt(struct sock *sk, int level, int optname, · 15 unmodified lines
4480	tp->snd_ssthresh = val;	4481	tp->snd_ssthresh = val;
4481	}	4482	}
4482	break;	4483	break;
		4484	case TCP_BPF_DELACK_MAX:
		4485	timeout = usecs_to_jiffies(val);
		4486	if (timeout > TCP_DELACK_MAX \|\|
		4487	timeout < TCP_TIMEOUT_MIN)
		4488	return -EINVAL;
		4489	inet_csk(sk)->icsk_delack_max = timeout;
		4490	break;
		4491	case TCP_BPF_RTO_MIN:
		4492	timeout = usecs_to_jiffies(val);
		4493	if (timeout > TCP_RTO_MIN \|\|
		4494	timeout < TCP_TIMEOUT_MIN)
		4495	return -EINVAL;
		4496	inet_csk(sk)->icsk_rto_min = timeout;
		4497	break;
4483	case TCP_SAVE_SYN:	4498	case TCP_SAVE_SYN:
4484	if (val < 0 \|\| val > 1)	4499	if (val < 0 \|\| val > 1)
4485	ret = -EINVAL;	4500	ret = -EINVAL;
@@ -4550,9 +4565,9 @@ static int _bpf_getsockopt(struct sock *sk, int level, int optname, · 64 unmodified lines
4550	tp = tcp_sk(sk);	4565	tp = tcp_sk(sk);
4551		4566
4552	if (optlen <= 0 \|\| !tp->saved_syn \|\|	4567	if (optlen <= 0 \|\| !tp->saved_syn \|\|
4553	optlen > tp->saved_syn[0])	4568	optlen > tcp_saved_syn_len(tp->saved_syn))
4554	goto err_clear;	4569	goto err_clear;
4555	memcpy(optval, tp->saved_syn + 1, optlen);	4570	memcpy(optval, tp->saved_syn->data, optlen);
4556	break;	4571	break;
4557	default:	4572	default:
4558	goto err_clear;	4573	goto err_clear;
@@ -4654,9 +4669,99 @@ static const struct bpf_func_proto bpf_sock_ops_setsockopt_proto = { · 95 unmodified lines
4654	.arg5_type = ARG_CONST_SIZE,	4669	.arg5_type = ARG_CONST_SIZE,
4655	};	4670	};
4656		4671
		4672	static int bpf_sock_ops_get_syn(struct bpf_sock_ops_kern *bpf_sock,
		4673	int optname, const u8 **start)
		4674	{
		4675	struct sk_buff *syn_skb = bpf_sock->syn_skb;
		4676	const u8 *hdr_start;
		4677	int ret;
		4678
		4679	if (syn_skb) {
		4680	/* sk is a request_sock here */
		4681
		4682	if (optname == TCP_BPF_SYN) {
		4683	hdr_start = syn_skb->data;
		4684	ret = tcp_hdrlen(syn_skb);
		4685	} else if (optname == TCP_BPF_SYN_IP) {
		4686	hdr_start = skb_network_header(syn_skb);
		4687	ret = skb_network_header_len(syn_skb) +
		4688	tcp_hdrlen(syn_skb);
		4689	} else {
		4690	/* optname == TCP_BPF_SYN_MAC */
		4691	hdr_start = skb_mac_header(syn_skb);
		4692	ret = skb_mac_header_len(syn_skb) +
		4693	skb_network_header_len(syn_skb) +
		4694	tcp_hdrlen(syn_skb);
		4695	}
		4696	} else {
		4697	struct sock *sk = bpf_sock->sk;
		4698	struct saved_syn *saved_syn;
		4699
		4700	if (sk->sk_state == TCP_NEW_SYN_RECV)
		4701	/* synack retransmit. bpf_sock->syn_skb will
		4702	* not be available. It has to resort to
		4703	* saved_syn (if it is saved).
		4704	*/
		4705	saved_syn = inet_reqsk(sk)->saved_syn;
		4706	else
		4707	saved_syn = tcp_sk(sk)->saved_syn;
		4708
		4709	if (!saved_syn)
		4710	return -ENOENT;
		4711
		4712	if (optname == TCP_BPF_SYN) {
		4713	hdr_start = saved_syn->data +
		4714	saved_syn->mac_hdrlen +
		4715	saved_syn->network_hdrlen;
		4716	ret = saved_syn->tcp_hdrlen;
		4717	} else if (optname == TCP_BPF_SYN_IP) {
		4718	hdr_start = saved_syn->data +
		4719	saved_syn->mac_hdrlen;
		4720	ret = saved_syn->network_hdrlen +
		4721	saved_syn->tcp_hdrlen;
		4722	} else {
		4723	/* optname == TCP_BPF_SYN_MAC */
		4724
		4725	/* TCP_SAVE_SYN may not have saved the mac hdr */
		4726	if (!saved_syn->mac_hdrlen)
		4727	return -ENOENT;
		4728
		4729	hdr_start = saved_syn->data;
		4730	ret = saved_syn->mac_hdrlen +
		4731	saved_syn->network_hdrlen +
		4732	saved_syn->tcp_hdrlen;
		4733	}
		4734	}
		4735
		4736	*start = hdr_start;
		4737	return ret;
		4738	}
		4739
4657	BPF_CALL_5(bpf_sock_ops_getsockopt, struct bpf_sock_ops_kern *, bpf_sock,	4740	BPF_CALL_5(bpf_sock_ops_getsockopt, struct bpf_sock_ops_kern *, bpf_sock,
4658	int, level, int, optname, char *, optval, int, optlen)	4741	int, level, int, optname, char *, optval, int, optlen)
4659	{	4742	{
		4743	if (IS_ENABLED(CONFIG_INET) && level == SOL_TCP &&
		4744	optname >= TCP_BPF_SYN && optname <= TCP_BPF_SYN_MAC) {
		4745	int ret, copy_len = 0;
		4746	const u8 *start;
		4747
		4748	ret = bpf_sock_ops_get_syn(bpf_sock, optname, &start);
		4749	if (ret > 0) {
		4750	copy_len = ret;
		4751	if (optlen < copy_len) {
		4752	copy_len = optlen;
		4753	ret = -ENOSPC;
		4754	}
		4755
		4756	memcpy(optval, start, copy_len);
		4757	}
		4758
		4759	/* Zero out unused buffer at the end */
		4760	memset(optval + copy_len, 0, optlen - copy_len);
		4761
		4762	return ret;
		4763	}
		4764
4660	return _bpf_getsockopt(bpf_sock->sk, level, optname, optval, optlen);	4765	return _bpf_getsockopt(bpf_sock->sk, level, optname, optval, optlen);
4661	}	4766	}
4662		4767
@@ -6150,6 +6255,232 @@ static const struct bpf_func_proto bpf_sk_assign_proto = { · 1487 unmodified lines
6150	.arg3_type = ARG_ANYTHING,	6255	.arg3_type = ARG_ANYTHING,
6151	};	6256	};
6152		6257
		6258	static const u8 bpf_search_tcp_opt(const u8 op, const u8 *opend,
		6259	u8 search_kind, const u8 *magic,
		6260	u8 magic_len, bool *eol)
		6261	{
		6262	u8 kind, kind_len;
		6263
		6264	*eol = false;
		6265
		6266	while (op < opend) {
		6267	kind = op[0];
		6268
		6269	if (kind == TCPOPT_EOL) {
		6270	*eol = true;
		6271	return ERR_PTR(-ENOMSG);
		6272	} else if (kind == TCPOPT_NOP) {
		6273	op++;
		6274	continue;
		6275	}
		6276
		6277	if (opend - op < 2 \|\| opend - op < op[1] \|\| op[1] < 2)
		6278	/* Something is wrong in the received header.
		6279	* Follow the TCP stack's tcp_parse_options()
		6280	* and just bail here.
		6281	*/
		6282	return ERR_PTR(-EFAULT);
		6283
		6284	kind_len = op[1];
		6285	if (search_kind == kind) {
		6286	if (!magic_len)
		6287	return op;
		6288
		6289	if (magic_len > kind_len - 2)
		6290	return ERR_PTR(-ENOMSG);
		6291
		6292	if (!memcmp(&op[2], magic, magic_len))
		6293	return op;
		6294	}
		6295
		6296	op += kind_len;
		6297	}
		6298
		6299	return ERR_PTR(-ENOMSG);
		6300	}
		6301
		6302	BPF_CALL_4(bpf_sock_ops_load_hdr_opt, struct bpf_sock_ops_kern *, bpf_sock,
		6303	void *, search_res, u32, len, u64, flags)
		6304	{
		6305	bool eol, load_syn = flags & BPF_LOAD_HDR_OPT_TCP_SYN;
		6306	const u8 op, opend, magic, search = search_res;
		6307	u8 search_kind, search_len, copy_len, magic_len;
		6308	int ret;
		6309
		6310	/* 2 byte is the minimal option len except TCPOPT_NOP and
		6311	* TCPOPT_EOL which are useless for the bpf prog to learn
		6312	* and this helper disallow loading them also.
		6313	*/
		6314	if (len < 2 \|\| flags & ~BPF_LOAD_HDR_OPT_TCP_SYN)
		6315	return -EINVAL;
		6316
		6317	search_kind = search[0];
		6318	search_len = search[1];
		6319
		6320	if (search_len > len \|\| search_kind == TCPOPT_NOP \|\|
		6321	search_kind == TCPOPT_EOL)
		6322	return -EINVAL;
		6323
		6324	if (search_kind == TCPOPT_EXP \|\| search_kind == 253) {
		6325	/* 16 or 32 bit magic. +2 for kind and kind length */
		6326	if (search_len != 4 && search_len != 6)
		6327	return -EINVAL;
		6328	magic = &search[2];
		6329	magic_len = search_len - 2;
		6330	} else {
		6331	if (search_len)
		6332	return -EINVAL;
		6333	magic = NULL;
		6334	magic_len = 0;
		6335	}
		6336
		6337	if (load_syn) {
		6338	ret = bpf_sock_ops_get_syn(bpf_sock, TCP_BPF_SYN, &op);
		6339	if (ret < 0)
		6340	return ret;
		6341
		6342	opend = op + ret;
		6343	op += sizeof(struct tcphdr);
		6344	} else {
		6345	if (!bpf_sock->skb \|\|
		6346	bpf_sock->op == BPF_SOCK_OPS_HDR_OPT_LEN_CB)
		6347	/* This bpf_sock->op cannot call this helper */
		6348	return -EPERM;
		6349
		6350	opend = bpf_sock->skb_data_end;
		6351	op = bpf_sock->skb->data + sizeof(struct tcphdr);
		6352	}
		6353
		6354	op = bpf_search_tcp_opt(op, opend, search_kind, magic, magic_len,
		6355	&eol);
		6356	if (IS_ERR(op))
		6357	return PTR_ERR(op);
		6358
		6359	copy_len = op[1];
		6360	ret = copy_len;
		6361	if (copy_len > len) {
		6362	ret = -ENOSPC;
		6363	copy_len = len;
		6364	}
		6365
		6366	memcpy(search_res, op, copy_len);
		6367	return ret;
		6368	}
		6369
		6370	static const struct bpf_func_proto bpf_sock_ops_load_hdr_opt_proto = {
		6371	.func = bpf_sock_ops_load_hdr_opt,
		6372	.gpl_only = false,
		6373	.ret_type = RET_INTEGER,
		6374	.arg1_type = ARG_PTR_TO_CTX,
		6375	.arg2_type = ARG_PTR_TO_MEM,
		6376	.arg3_type = ARG_CONST_SIZE,
		6377	.arg4_type = ARG_ANYTHING,
		6378	};
		6379
		6380	BPF_CALL_4(bpf_sock_ops_store_hdr_opt, struct bpf_sock_ops_kern *, bpf_sock,
		6381	const void *, from, u32, len, u64, flags)
		6382	{
		6383	u8 new_kind, new_kind_len, magic_len = 0, *opend;
		6384	const u8 op, new_op, *magic = NULL;
		6385	struct sk_buff *skb;
		6386	bool eol;
		6387
		6388	if (bpf_sock->op != BPF_SOCK_OPS_WRITE_HDR_OPT_CB)
		6389	return -EPERM;
		6390
		6391	if (len < 2 \|\| flags)
		6392	return -EINVAL;
		6393
		6394	new_op = from;
		6395	new_kind = new_op[0];
		6396	new_kind_len = new_op[1];
		6397
		6398	if (new_kind_len > len \|\| new_kind == TCPOPT_NOP \|\|
		6399	new_kind == TCPOPT_EOL)
		6400	return -EINVAL;
		6401
		6402	if (new_kind_len > bpf_sock->remaining_opt_len)
		6403	return -ENOSPC;
		6404
		6405	/* 253 is another experimental kind */
		6406	if (new_kind == TCPOPT_EXP \|\| new_kind == 253) {
		6407	if (new_kind_len < 4)
		6408	return -EINVAL;
		6409	/* Match for the 2 byte magic also.
		6410	* RFC 6994: the magic could be 2 or 4 bytes.
		6411	* Hence, matching by 2 byte only is on the
		6412	* conservative side but it is the right
		6413	* thing to do for the 'search-for-duplication'
		6414	* purpose.
		6415	*/
		6416	magic = &new_op[2];
		6417	magic_len = 2;
		6418	}
		6419
		6420	/* Check for duplication */
		6421	skb = bpf_sock->skb;
		6422	op = skb->data + sizeof(struct tcphdr);
		6423	opend = bpf_sock->skb_data_end;
		6424
		6425	op = bpf_search_tcp_opt(op, opend, new_kind, magic, magic_len,
		6426	&eol);
		6427	if (!IS_ERR(op))
		6428	return -EEXIST;
		6429
		6430	if (PTR_ERR(op) != -ENOMSG)
		6431	return PTR_ERR(op);
		6432
		6433	if (eol)
		6434	/* The option has been ended. Treat it as no more
		6435	* header option can be written.
		6436	*/
		6437	return -ENOSPC;
		6438
		6439	/* No duplication found. Store the header option. */
		6440	memcpy(opend, from, new_kind_len);
		6441
		6442	bpf_sock->remaining_opt_len -= new_kind_len;
		6443	bpf_sock->skb_data_end += new_kind_len;
		6444
		6445	return 0;
		6446	}
		6447
		6448	static const struct bpf_func_proto bpf_sock_ops_store_hdr_opt_proto = {
		6449	.func = bpf_sock_ops_store_hdr_opt,
		6450	.gpl_only = false,
		6451	.ret_type = RET_INTEGER,
		6452	.arg1_type = ARG_PTR_TO_CTX,
		6453	.arg2_type = ARG_PTR_TO_MEM,
		6454	.arg3_type = ARG_CONST_SIZE,
		6455	.arg4_type = ARG_ANYTHING,
		6456	};
		6457
		6458	BPF_CALL_3(bpf_sock_ops_reserve_hdr_opt, struct bpf_sock_ops_kern *, bpf_sock,
		6459	u32, len, u64, flags)
		6460	{
		6461	if (bpf_sock->op != BPF_SOCK_OPS_HDR_OPT_LEN_CB)
		6462	return -EPERM;
		6463
		6464	if (flags \|\| len < 2)
		6465	return -EINVAL;
		6466
		6467	if (len > bpf_sock->remaining_opt_len)
		6468	return -ENOSPC;
		6469
		6470	bpf_sock->remaining_opt_len -= len;
		6471
		6472	return 0;
		6473	}
		6474
		6475	static const struct bpf_func_proto bpf_sock_ops_reserve_hdr_opt_proto = {
		6476	.func = bpf_sock_ops_reserve_hdr_opt,
		6477	.gpl_only = false,
		6478	.ret_type = RET_INTEGER,
		6479	.arg1_type = ARG_PTR_TO_CTX,
		6480	.arg2_type = ARG_ANYTHING,
		6481	.arg3_type = ARG_ANYTHING,
		6482	};
		6483
6153	#endif /* CONFIG_INET */	6484	#endif /* CONFIG_INET */
6154		6485
6155	bool bpf_helper_changes_pkt_data(void *func)	6486	bool bpf_helper_changes_pkt_data(void *func)
@@ -6178,6 +6509,9 @@ bool bpf_helper_changes_pkt_data(void *func) · 22 unmodified lines
6178	func == bpf_lwt_seg6_store_bytes \|\|	6509	func == bpf_lwt_seg6_store_bytes \|\|
6179	func == bpf_lwt_seg6_adjust_srh \|\|	6510	func == bpf_lwt_seg6_adjust_srh \|\|
6180	func == bpf_lwt_seg6_action \|\|	6511	func == bpf_lwt_seg6_action \|\|
		6512	#endif
		6513	#ifdef CONFIG_INET
		6514	func == bpf_sock_ops_store_hdr_opt \|\|
6181	#endif	6515	#endif
6182	func == bpf_lwt_in_push_encap \|\|	6516	func == bpf_lwt_in_push_encap \|\|
6183	func == bpf_lwt_xmit_push_encap)	6517	func == bpf_lwt_xmit_push_encap)
@@ -6550,6 +6884,12 @@ sock_ops_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 366 unmodified lines
6550	case BPF_FUNC_sk_storage_delete:	6884	case BPF_FUNC_sk_storage_delete:
6551	return &bpf_sk_storage_delete_proto;	6885	return &bpf_sk_storage_delete_proto;
6552	#ifdef CONFIG_INET	6886	#ifdef CONFIG_INET
		6887	case BPF_FUNC_load_hdr_opt:
		6888	return &bpf_sock_ops_load_hdr_opt_proto;
		6889	case BPF_FUNC_store_hdr_opt:
		6890	return &bpf_sock_ops_store_hdr_opt_proto;
		6891	case BPF_FUNC_reserve_hdr_opt:
		6892	return &bpf_sock_ops_reserve_hdr_opt_proto;
6553	case BPF_FUNC_tcp_sock:	6893	case BPF_FUNC_tcp_sock:
6554	return &bpf_tcp_sock_proto;	6894	return &bpf_tcp_sock_proto;
6555	#endif /* CONFIG_INET */	6895	#endif /* CONFIG_INET */
@@ -7349,6 +7689,20 @@ static bool sock_ops_is_valid_access(int off, int size, · 793 unmodified lines
7349	return false;	7689	return false;
7350	info->reg_type = PTR_TO_SOCKET_OR_NULL;	7690	info->reg_type = PTR_TO_SOCKET_OR_NULL;
7351	break;	7691	break;
		7692	case offsetof(struct bpf_sock_ops, skb_data):
		7693	if (size != sizeof(__u64))
		7694	return false;
		7695	info->reg_type = PTR_TO_PACKET;
		7696	break;
		7697	case offsetof(struct bpf_sock_ops, skb_data_end):
		7698	if (size != sizeof(__u64))
		7699	return false;
		7700	info->reg_type = PTR_TO_PACKET_END;
		7701	break;
		7702	case offsetof(struct bpf_sock_ops, skb_tcp_flags):
		7703	bpf_ctx_record_field_size(info, size_default);
		7704	return bpf_ctx_narrow_access_ok(off, size,
		7705	size_default);
7352	default:	7706	default:
7353	if (size != size_default)	7707	if (size != size_default)
7354	return false;	7708	return false;
@@ -8450,17 +8804,22 @@ static u32 sock_ops_convert_ctx_access(enum bpf_access_type type, · 1095 unmodified lines
8450	return insn - insn_buf;	8804	return insn - insn_buf;
8451		8805
8452	switch (si->off) {	8806	switch (si->off) {
8453	case offsetof(struct bpf_sock_ops, op) ...	8807	case offsetof(struct bpf_sock_ops, op):
		8808	*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sock_ops_kern,
		8809	op),
		8810	si->dst_reg, si->src_reg,
		8811	offsetof(struct bpf_sock_ops_kern, op));
		8812	break;
		8813
		8814	case offsetof(struct bpf_sock_ops, replylong[0]) ...
8454	offsetof(struct bpf_sock_ops, replylong[3]):	8815	offsetof(struct bpf_sock_ops, replylong[3]):
8455	BUILD_BUG_ON(sizeof_field(struct bpf_sock_ops, op) !=
8456	sizeof_field(struct bpf_sock_ops_kern, op));
8457	BUILD_BUG_ON(sizeof_field(struct bpf_sock_ops, reply) !=	8816	BUILD_BUG_ON(sizeof_field(struct bpf_sock_ops, reply) !=
8458	sizeof_field(struct bpf_sock_ops_kern, reply));	8817	sizeof_field(struct bpf_sock_ops_kern, reply));
8459	BUILD_BUG_ON(sizeof_field(struct bpf_sock_ops, replylong) !=	8818	BUILD_BUG_ON(sizeof_field(struct bpf_sock_ops, replylong) !=
8460	sizeof_field(struct bpf_sock_ops_kern, replylong));	8819	sizeof_field(struct bpf_sock_ops_kern, replylong));
8461	off = si->off;	8820	off = si->off;
8462	off -= offsetof(struct bpf_sock_ops, op);	8821	off -= offsetof(struct bpf_sock_ops, replylong[0]);
8463	off += offsetof(struct bpf_sock_ops_kern, op);	8822	off += offsetof(struct bpf_sock_ops_kern, replylong[0]);
8464	if (type == BPF_WRITE)	8823	if (type == BPF_WRITE)
8465	*insn++ = BPF_STX_MEM(BPF_W, si->dst_reg, si->src_reg,	8824	*insn++ = BPF_STX_MEM(BPF_W, si->dst_reg, si->src_reg,
8466	off);	8825	off);
@@ -8681,6 +9040,49 @@ static u32 sock_ops_convert_ctx_access(enum bpf_access_type type, · 214 unmodified lines
8681	case offsetof(struct bpf_sock_ops, sk):	9040	case offsetof(struct bpf_sock_ops, sk):
8682	SOCK_OPS_GET_SK();	9041	SOCK_OPS_GET_SK();
8683	break;	9042	break;
		9043	case offsetof(struct bpf_sock_ops, skb_data_end):
		9044	*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sock_ops_kern,
		9045	skb_data_end),
		9046	si->dst_reg, si->src_reg,
		9047	offsetof(struct bpf_sock_ops_kern,
		9048	skb_data_end));
		9049	break;
		9050	case offsetof(struct bpf_sock_ops, skb_data):
		9051	*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sock_ops_kern,
		9052	skb),
		9053	si->dst_reg, si->src_reg,
		9054	offsetof(struct bpf_sock_ops_kern,
		9055	skb));
		9056	*insn++ = BPF_JMP_IMM(BPF_JEQ, si->dst_reg, 0, 1);
		9057	*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct sk_buff, data),
		9058	si->dst_reg, si->dst_reg,
		9059	offsetof(struct sk_buff, data));
		9060	break;
		9061	case offsetof(struct bpf_sock_ops, skb_len):
		9062	*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sock_ops_kern,
		9063	skb),
		9064	si->dst_reg, si->src_reg,
		9065	offsetof(struct bpf_sock_ops_kern,
		9066	skb));
		9067	*insn++ = BPF_JMP_IMM(BPF_JEQ, si->dst_reg, 0, 1);
		9068	*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct sk_buff, len),
		9069	si->dst_reg, si->dst_reg,
		9070	offsetof(struct sk_buff, len));
		9071	break;
		9072	case offsetof(struct bpf_sock_ops, skb_tcp_flags):
		9073	off = offsetof(struct sk_buff, cb);
		9074	off += offsetof(struct tcp_skb_cb, tcp_flags);
		9075	*target_size = sizeof_field(struct tcp_skb_cb, tcp_flags);
		9076	*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sock_ops_kern,
		9077	skb),
		9078	si->dst_reg, si->src_reg,
		9079	offsetof(struct bpf_sock_ops_kern,
		9080	skb));
		9081	*insn++ = BPF_JMP_IMM(BPF_JEQ, si->dst_reg, 0, 1);
		9082	*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct tcp_skb_cb,
		9083	tcp_flags),
		9084	si->dst_reg, si->dst_reg, off);
		9085	break;
8684	}	9086	}
8685	return insn - insn_buf;	9087	return insn - insn_buf;
8686	}	9088	}

net/core/skmsg.c+28−6

@@ -494,14 +494,34 @@ end: · 493 unmodified lines
494		494
495	struct sk_psock sk_psock_init(struct sock sk, int node)	495	struct sk_psock sk_psock_init(struct sock sk, int node)
496	{	496	{
497	struct sk_psock psock = kzalloc_node(sizeof(psock),	497	struct sk_psock *psock;
498	GFP_ATOMIC \| __GFP_NOWARN,	498	struct proto *prot;
499	node);	499
500	if (!psock)	500	write_lock_bh(&sk->sk_callback_lock);
501	return NULL;	501
		502	if (inet_csk_has_ulp(sk)) {
		503	psock = ERR_PTR(-EINVAL);
		504	goto out;
		505	}
		506
		507	if (sk->sk_user_data) {
		508	psock = ERR_PTR(-EBUSY);
		509	goto out;
		510	}
502		511
		512	psock = kzalloc_node(sizeof(*psock), GFP_ATOMIC \| __GFP_NOWARN, node);
		513	if (!psock) {
		514	psock = ERR_PTR(-ENOMEM);
		515	goto out;
		516	}
		517
		518	prot = READ_ONCE(sk->sk_prot);
503	psock->sk = sk;	519	psock->sk = sk;
504	psock->eval = __SK_NONE;	520	psock->eval = __SK_NONE;
		521	psock->sk_proto = prot;
		522	psock->saved_unhash = prot->unhash;
		523	psock->saved_close = prot->close;
		524	psock->saved_write_space = sk->sk_write_space;
505		525
506	INIT_LIST_HEAD(&psock->link);	526	INIT_LIST_HEAD(&psock->link);
507	spin_lock_init(&psock->link_lock);	527	spin_lock_init(&psock->link_lock);
@@ -516,6 +536,8 @@ struct sk_psock sk_psock_init(struct sock sk, int node) · 8 unmodified lines
516	rcu_assign_sk_user_data_nocopy(sk, psock);	536	rcu_assign_sk_user_data_nocopy(sk, psock);
517	sock_hold(sk);	537	sock_hold(sk);
518		538
		539	out:
		540	write_unlock_bh(&sk->sk_callback_lock);
519	return psock;	541	return psock;
520	}	542	}
521	EXPORT_SYMBOL_GPL(sk_psock_init);	543	EXPORT_SYMBOL_GPL(sk_psock_init);

net/core/sock_map.c+37−54

 {
 	struct proto *prot;
-	sock_owned_by_me(sk);
 	switch (sk->sk_type) {
 	case SOCK_STREAM:
 		prot = tcp_bpf_get_proto(sk, psock);
 		}
 	} else {
 		psock = sk_psock_init(sk, map->numa_node);
-		if (!psock) {
+		if (IS_ERR(psock)) {
-			ret = -ENOMEM;
+			ret = PTR_ERR(psock);
 			goto out_progs;
 		}
 	}
 	if (!psock) {
 		psock = sk_psock_init(sk, map->numa_node);
-		if (!psock)
+		if (IS_ERR(psock))
-			return -ENOMEM;
+			return PTR_ERR(psock);
 	}
 	ret = sock_map_init_proto(sk, psock);
 		return -EINVAL;
 	if (unlikely(idx >= map->max_entries))
 		return -E2BIG;
-	if (inet_csk_has_ulp(sk))
-		return -EINVAL;
 	link = sk_psock_init_link();
 	if (!link)
 	return false;
 }
-static int sock_map_update_elem(struct bpf_map *map, void *key,
+static int sock_hash_update_common(struct bpf_map *map, void *key,
-				void *value, u64 flags)
+				   struct sock *sk, u64 flags);
+int sock_map_update_elem_sys(struct bpf_map *map, void *key, void *value,
+			     u64 flags)
 {
-	u32 idx = *(u32 *)key;
 	struct socket *sock;
 	struct sock *sk;
 	int ret;
 	sock_map_sk_acquire(sk);
 	if (!sock_map_sk_state_allowed(sk))
 		ret = -EOPNOTSUPP;
+	else if (map->map_type == BPF_MAP_TYPE_SOCKMAP)
+		ret = sock_map_update_common(map, *(u32 *)key, sk, flags);
 	else
-		ret = sock_map_update_common(map, idx, sk, flags);
+		ret = sock_hash_update_common(map, key, sk, flags);
 	sock_map_sk_release(sk);
 out:
 	fput(sock->file);
 	return ret;
 }
+static int sock_map_update_elem(struct bpf_map *map, void *key,
+				void *value, u64 flags)
+{
+	struct sock *sk = (struct sock *)value;
+	int ret;
+	if (!sock_map_sk_is_suitable(sk))
+		return -EOPNOTSUPP;
+	local_bh_disable();
+	bh_lock_sock(sk);
+	if (!sock_map_sk_state_allowed(sk))
+		ret = -EOPNOTSUPP;
+	else if (map->map_type == BPF_MAP_TYPE_SOCKMAP)
+		ret = sock_map_update_common(map, *(u32 *)key, sk, flags);
+	else
+		ret = sock_hash_update_common(map, key, sk, flags);
+	bh_unlock_sock(sk);
+	local_bh_enable();
+	return ret;
+}
 BPF_CALL_4(bpf_sock_map_update, struct bpf_sock_ops_kern *, sops,
 	   struct bpf_map *, map, void *, key, u64, flags)
 {
 static int sock_map_btf_id;
 const struct bpf_map_ops sock_map_ops = {
+	.map_meta_equal		= bpf_map_meta_equal,
 	.map_alloc		= sock_map_alloc,
 	.map_free		= sock_map_free,
 	.map_get_next_key	= sock_map_get_next_key,
 	WARN_ON_ONCE(!rcu_read_lock_held());
 	if (unlikely(flags > BPF_EXIST))
 		return -EINVAL;
-	if (inet_csk_has_ulp(sk))
-		return -EINVAL;
 	link = sk_psock_init_link();
 	if (!link)
 	return ret;
 }
-static int sock_hash_update_elem(struct bpf_map *map, void *key,
-				 void *value, u64 flags)
-{
-	struct socket *sock;
-	struct sock *sk;
-	int ret;
-	u64 ufd;
-	if (map->value_size == sizeof(u64))
-		ufd = *(u64 *)value;
-	else
-		ufd = *(u32 *)value;
-	if (ufd > S32_MAX)
-		return -EINVAL;
-	sock = sockfd_lookup(ufd, &ret);
-	if (!sock)
-		return ret;
-	sk = sock->sk;
-	if (!sk) {
-		ret = -EINVAL;
-		goto out;
-	}
-	if (!sock_map_sk_is_suitable(sk)) {
-		ret = -EOPNOTSUPP;
-		goto out;
-	}
-	sock_map_sk_acquire(sk);
-	if (!sock_map_sk_state_allowed(sk))
-		ret = -EOPNOTSUPP;
-	else
-		ret = sock_hash_update_common(map, key, sk, flags);
-	sock_map_sk_release(sk);
-out:
-	fput(sock->file);
-	return ret;
-}
 static int sock_hash_get_next_key(struct bpf_map *map, void *key,
 				  void *key_next)
 {
 static int sock_hash_map_btf_id;
 const struct bpf_map_ops sock_hash_ops = {
+	.map_meta_equal		= bpf_map_meta_equal,
 	.map_alloc		= sock_hash_alloc,
 	.map_free		= sock_hash_free,
 	.map_get_next_key	= sock_hash_get_next_key,
-	.map_update_elem	= sock_hash_update_elem,
+	.map_update_elem	= sock_map_update_elem,
 	.map_delete_elem	= sock_hash_delete_elem,
 	.map_lookup_elem	= sock_hash_lookup,
 	.map_lookup_elem_sys_only = sock_hash_lookup_sys,

net/ethtool/channels.c+1−1

@@ -223,7 +223,7 @@ int ethnl_set_channels(struct sk_buff skb, struct genl_info info) · 222 unmodified lines
223	from_channel = channels.combined_count +	223	from_channel = channels.combined_count +
224	min(channels.rx_count, channels.tx_count);	224	min(channels.rx_count, channels.tx_count);
225	for (i = from_channel; i < old_total; i++)	225	for (i = from_channel; i < old_total; i++)
226	if (xdp_get_umem_from_qid(dev, i)) {	226	if (xsk_get_pool_from_qid(dev, i)) {
227	GENL_SET_ERR_MSG(info, "requested channel counts are too low for existing zerocopy AF_XDP sockets");	227	GENL_SET_ERR_MSG(info, "requested channel counts are too low for existing zerocopy AF_XDP sockets");
228	return -EINVAL;	228	return -EINVAL;
229	}	229	}

net/ethtool/ioctl.c+1−1

@@ -1706,7 +1706,7 @@ static noinline_for_stack int ethtool_set_channels(struct net_device *dev, · 1705 unmodified lines
1706	min(channels.rx_count, channels.tx_count);	1706	min(channels.rx_count, channels.tx_count);
1707	to_channel = curr.combined_count + max(curr.rx_count, curr.tx_count);	1707	to_channel = curr.combined_count + max(curr.rx_count, curr.tx_count);
1708	for (i = from_channel; i < to_channel; i++)	1708	for (i = from_channel; i < to_channel; i++)
1709	if (xdp_get_umem_from_qid(dev, i))	1709	if (xsk_get_pool_from_qid(dev, i))
1710	return -EINVAL;	1710	return -EINVAL;
1711		1711
1712	ret = dev->ethtool_ops->set_channels(dev, &channels);	1712	ret = dev->ethtool_ops->set_channels(dev, &channels);

net/ipv4/tcp.c+11−5

 	INIT_LIST_HEAD(&tp->tsorted_sent_queue);
 	icsk->icsk_rto = TCP_TIMEOUT_INIT;
+	icsk->icsk_rto_min = TCP_RTO_MIN;
+	icsk->icsk_delack_max = TCP_DELACK_MAX;
 	tp->mdev_us = jiffies_to_usecs(TCP_TIMEOUT_INIT);
 	minmax_reset(&tp->rtt_min, tcp_jiffies32, ~0U);
 	icsk->icsk_backoff = 0;
 	icsk->icsk_probes_out = 0;
 	icsk->icsk_rto = TCP_TIMEOUT_INIT;
+	icsk->icsk_rto_min = TCP_RTO_MIN;
+	icsk->icsk_delack_max = TCP_DELACK_MAX;
 	tp->snd_ssthresh = TCP_INFINITE_SSTHRESH;
 	tp->snd_cwnd = TCP_INIT_CWND;
 	tp->snd_cwnd_cnt = 0;
 		break;
 	case TCP_SAVE_SYN:
-		if (val < 0 || val > 1)
+		/* 0: disable, 1: enable, 2: start from ether_header */
+		if (val < 0 || val > 2)
 			err = -EINVAL;
 		else
 			tp->save_syn = val;
 		lock_sock(sk);
 		if (tp->saved_syn) {
-			if (len < tp->saved_syn[0]) {
+			if (len < tcp_saved_syn_len(tp->saved_syn)) {
-				if (put_user(tp->saved_syn[0], optlen)) {
+				if (put_user(tcp_saved_syn_len(tp->saved_syn),
+					     optlen)) {
 					release_sock(sk);
 					return -EFAULT;
 				}
 				release_sock(sk);
 				return -EINVAL;
 			}
-			len = tp->saved_syn[0];
+			len = tcp_saved_syn_len(tp->saved_syn);
 			if (put_user(len, optlen)) {
 				release_sock(sk);
 				return -EFAULT;
 			}
-			if (copy_to_user(optval, tp->saved_syn + 1, len)) {
+			if (copy_to_user(optval, tp->saved_syn->data, len)) {
 				release_sock(sk);
 				return -EFAULT;
 			}

net/ipv4/tcp_bpf.c+5−8

 	prot[TCP_BPF_TX].sendpage		= tcp_bpf_sendpage;
 }
-static void tcp_bpf_check_v6_needs_rebuild(struct sock *sk, struct proto *ops)
+static void tcp_bpf_check_v6_needs_rebuild(struct proto *ops)
 {
-	if (sk->sk_family == AF_INET6 &&
+	if (unlikely(ops != smp_load_acquire(&tcpv6_prot_saved))) {
-	    unlikely(ops != smp_load_acquire(&tcpv6_prot_saved))) {
 		spin_lock_bh(&tcpv6_prot_lock);
 		if (likely(ops != tcpv6_prot_saved)) {
 			tcp_bpf_rebuild_protos(tcp_bpf_prots[TCP_BPF_IPV6], ops);
 	int family = sk->sk_family == AF_INET6 ? TCP_BPF_IPV6 : TCP_BPF_IPV4;
 	int config = psock->progs.msg_parser   ? TCP_BPF_TX   : TCP_BPF_BASE;
-	if (!psock->sk_proto) {
+	if (sk->sk_family == AF_INET6) {
-		struct proto *ops = READ_ONCE(sk->sk_prot);
+		if (tcp_bpf_assert_proto_ops(psock->sk_proto))
-		if (tcp_bpf_assert_proto_ops(ops))
 			return ERR_PTR(-EINVAL);
-		tcp_bpf_check_v6_needs_rebuild(sk, ops);
+		tcp_bpf_check_v6_needs_rebuild(psock->sk_proto);
 	}
 	return &tcp_bpf_prots[family][config];

net/ipv4/tcp_fastopen.c+1−1

@@ -295,7 +295,7 @@ static struct sock tcp_fastopen_create_child(struct sock sk, · 294 unmodified lines
295	refcount_set(&req->rsk_refcnt, 2);	295	refcount_set(&req->rsk_refcnt, 2);
296		296
297	/* Now finish processing the fastopen child socket. */	297	/* Now finish processing the fastopen child socket. */
298	tcp_init_transfer(child, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB);	298	tcp_init_transfer(child, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB, skb);
299		299
300	tp->rcv_nxt = TCP_SKB_CB(skb)->seq + 1;	300	tp->rcv_nxt = TCP_SKB_CB(skb)->seq + 1;
301		301

net/ipv4/tcp_input.c+109−18

@@ -138,6 +138,69 @@ void clean_acked_data_flush(void) · 137 unmodified lines
138	EXPORT_SYMBOL_GPL(clean_acked_data_flush);	138	EXPORT_SYMBOL_GPL(clean_acked_data_flush);
139	#endif	139	#endif
140		140
		141	#ifdef CONFIG_CGROUP_BPF
		142	static void bpf_skops_parse_hdr(struct sock sk, struct sk_buff skb)
		143	{
		144	bool unknown_opt = tcp_sk(sk)->rx_opt.saw_unknown &&
		145	BPF_SOCK_OPS_TEST_FLAG(tcp_sk(sk),
		146	BPF_SOCK_OPS_PARSE_UNKNOWN_HDR_OPT_CB_FLAG);
		147	bool parse_all_opt = BPF_SOCK_OPS_TEST_FLAG(tcp_sk(sk),
		148	BPF_SOCK_OPS_PARSE_ALL_HDR_OPT_CB_FLAG);
		149	struct bpf_sock_ops_kern sock_ops;
		150
		151	if (likely(!unknown_opt && !parse_all_opt))
		152	return;
		153
		154	/* The skb will be handled in the
		155	* bpf_skops_established() or
		156	* bpf_skops_write_hdr_opt().
		157	*/
		158	switch (sk->sk_state) {
		159	case TCP_SYN_RECV:
		160	case TCP_SYN_SENT:
		161	case TCP_LISTEN:
		162	return;
		163	}
		164
		165	sock_owned_by_me(sk);
		166
		167	memset(&sock_ops, 0, offsetof(struct bpf_sock_ops_kern, temp));
		168	sock_ops.op = BPF_SOCK_OPS_PARSE_HDR_OPT_CB;
		169	sock_ops.is_fullsock = 1;
		170	sock_ops.sk = sk;
		171	bpf_skops_init_skb(&sock_ops, skb, tcp_hdrlen(skb));
		172
		173	BPF_CGROUP_RUN_PROG_SOCK_OPS(&sock_ops);
		174	}
		175
		176	static void bpf_skops_established(struct sock *sk, int bpf_op,
		177	struct sk_buff *skb)
		178	{
		179	struct bpf_sock_ops_kern sock_ops;
		180
		181	sock_owned_by_me(sk);
		182
		183	memset(&sock_ops, 0, offsetof(struct bpf_sock_ops_kern, temp));
		184	sock_ops.op = bpf_op;
		185	sock_ops.is_fullsock = 1;
		186	sock_ops.sk = sk;
		187	/* sk with TCP_REPAIR_ON does not have skb in tcp_finish_connect */
		188	if (skb)
		189	bpf_skops_init_skb(&sock_ops, skb, tcp_hdrlen(skb));
		190
		191	BPF_CGROUP_RUN_PROG_SOCK_OPS(&sock_ops);
		192	}
		193	#else
		194	static void bpf_skops_parse_hdr(struct sock sk, struct sk_buff skb)
		195	{
		196	}
		197
		198	static void bpf_skops_established(struct sock *sk, int bpf_op,
		199	struct sk_buff *skb)
		200	{
		201	}
		202	#endif
		203
141	static void tcp_gro_dev_warn(struct sock sk, const struct sk_buff skb,	204	static void tcp_gro_dev_warn(struct sock sk, const struct sk_buff skb,
142	unsigned int len)	205	unsigned int len)
143	{	206	{
@@ -3801,7 +3864,7 @@ static void tcp_parse_fastopen_option(int len, const unsigned char *cookie, · 3657 unmodified lines
3801	foc->exp = exp_opt;	3864	foc->exp = exp_opt;
3802	}	3865	}
3803		3866
3804	static void smc_parse_options(const struct tcphdr *th,	3867	static bool smc_parse_options(const struct tcphdr *th,
3805	struct tcp_options_received *opt_rx,	3868	struct tcp_options_received *opt_rx,
3806	const unsigned char *ptr,	3869	const unsigned char *ptr,
3807	int opsize)	3870	int opsize)
@@ -3810,10 +3873,13 @@ static void smc_parse_options(const struct tcphdr *th, · 2 unmodified lines
3810	if (static_branch_unlikely(&tcp_have_smc)) {	3873	if (static_branch_unlikely(&tcp_have_smc)) {
3811	if (th->syn && !(opsize & 1) &&	3874	if (th->syn && !(opsize & 1) &&
3812	opsize >= TCPOLEN_EXP_SMC_BASE &&	3875	opsize >= TCPOLEN_EXP_SMC_BASE &&
3813	get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC)	3876	get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC) {
3814	opt_rx->smc_ok = 1;	3877	opt_rx->smc_ok = 1;
		3878	return true;
		3879	}
3815	}	3880	}
3816	#endif	3881	#endif
		3882	return false;
3817	}	3883	}
3818		3884
3819	/* Try to parse the MSS option from the TCP header. Return 0 on failure, clamped	3885	/* Try to parse the MSS option from the TCP header. Return 0 on failure, clamped
@@ -3874,6 +3940,7 @@ void tcp_parse_options(const struct net *net, · 54 unmodified lines
3874		3940
3875	ptr = (const unsigned char *)(th + 1);	3941	ptr = (const unsigned char *)(th + 1);
3876	opt_rx->saw_tstamp = 0;	3942	opt_rx->saw_tstamp = 0;
		3943	opt_rx->saw_unknown = 0;
3877		3944
3878	while (length > 0) {	3945	while (length > 0) {
3879	int opcode = *ptr++;	3946	int opcode = *ptr++;
@@ -3964,15 +4031,21 @@ void tcp_parse_options(const struct net *net, · 84 unmodified lines
3964	*/	4031	*/
3965	if (opsize >= TCPOLEN_EXP_FASTOPEN_BASE &&	4032	if (opsize >= TCPOLEN_EXP_FASTOPEN_BASE &&
3966	get_unaligned_be16(ptr) ==	4033	get_unaligned_be16(ptr) ==
3967	TCPOPT_FASTOPEN_MAGIC)	4034	TCPOPT_FASTOPEN_MAGIC) {
3968	tcp_parse_fastopen_option(opsize -	4035	tcp_parse_fastopen_option(opsize -
3969	TCPOLEN_EXP_FASTOPEN_BASE,	4036	TCPOLEN_EXP_FASTOPEN_BASE,
3970	ptr + 2, th->syn, foc, true);	4037	ptr + 2, th->syn, foc, true);
3971	else	4038	break;
3972	smc_parse_options(th, opt_rx, ptr,	4039	}
3973	opsize);	4040
		4041	if (smc_parse_options(th, opt_rx, ptr, opsize))
		4042	break;
		4043
		4044	opt_rx->saw_unknown = 1;
3974	break;	4045	break;
3975		4046
		4047	default:
		4048	opt_rx->saw_unknown = 1;
3976	}	4049	}
3977	ptr += opsize-2;	4050	ptr += opsize-2;
3978	length -= opsize;	4051	length -= opsize;
@@ -5590,6 +5663,8 @@ syn_challenge: · 1611 unmodified lines
5590	goto discard;	5663	goto discard;
5591	}	5664	}
5592		5665
		5666	bpf_skops_parse_hdr(sk, skb);
		5667
5593	return true;	5668	return true;
5594		5669
5595	discard:	5670	discard:
@@ -5798,7 +5873,7 @@ discard: · 202 unmodified lines
5798	}	5873	}
5799	EXPORT_SYMBOL(tcp_rcv_established);	5874	EXPORT_SYMBOL(tcp_rcv_established);
5800		5875
5801	void tcp_init_transfer(struct sock *sk, int bpf_op)	5876	void tcp_init_transfer(struct sock sk, int bpf_op, struct sk_buff skb)
5802	{	5877	{
5803	struct inet_connection_sock *icsk = inet_csk(sk);	5878	struct inet_connection_sock *icsk = inet_csk(sk);
5804	struct tcp_sock *tp = tcp_sk(sk);	5879	struct tcp_sock *tp = tcp_sk(sk);
@@ -5819,7 +5894,7 @@ void tcp_init_transfer(struct sock *sk, int bpf_op) · 14 unmodified lines
5819	tp->snd_cwnd = tcp_init_cwnd(tp, __sk_dst_get(sk));	5894	tp->snd_cwnd = tcp_init_cwnd(tp, __sk_dst_get(sk));
5820	tp->snd_cwnd_stamp = tcp_jiffies32;	5895	tp->snd_cwnd_stamp = tcp_jiffies32;
5821		5896
5822	tcp_call_bpf(sk, bpf_op, 0, NULL);	5897	bpf_skops_established(sk, bpf_op, skb);
5823	tcp_init_congestion_control(sk);	5898	tcp_init_congestion_control(sk);
5824	tcp_init_buffer_space(sk);	5899	tcp_init_buffer_space(sk);
5825	}	5900	}
@@ -5838,7 +5913,7 @@ void tcp_finish_connect(struct sock sk, struct sk_buff skb) · 12 unmodified lines
5838	sk_mark_napi_id(sk, skb);	5913	sk_mark_napi_id(sk, skb);
5839	}	5914	}
5840		5915
5841	tcp_init_transfer(sk, BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB);	5916	tcp_init_transfer(sk, BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB, skb);
5842		5917
5843	/* Prevent spurious tcp_cwnd_restart() on first data	5918	/* Prevent spurious tcp_cwnd_restart() on first data
5844	* packet.	5919	* packet.
@@ -6310,7 +6385,8 @@ int tcp_rcv_state_process(struct sock sk, struct sk_buff skb) · 465 unmodified lines
6310	} else {	6385	} else {
6311	tcp_try_undo_spurious_syn(sk);	6386	tcp_try_undo_spurious_syn(sk);
6312	tp->retrans_stamp = 0;	6387	tp->retrans_stamp = 0;
6313	tcp_init_transfer(sk, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB);	6388	tcp_init_transfer(sk, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB,
		6389	skb);
6314	WRITE_ONCE(tp->copied_seq, tp->rcv_nxt);	6390	WRITE_ONCE(tp->copied_seq, tp->rcv_nxt);
6315	}	6391	}
6316	smp_mb();	6392	smp_mb();
@@ -6599,13 +6675,27 @@ static void tcp_reqsk_record_syn(const struct sock *sk, · 282 unmodified lines
6599	{	6675	{
6600	if (tcp_sk(sk)->save_syn) {	6676	if (tcp_sk(sk)->save_syn) {
6601	u32 len = skb_network_header_len(skb) + tcp_hdrlen(skb);	6677	u32 len = skb_network_header_len(skb) + tcp_hdrlen(skb);
6602	u32 *copy;	6678	struct saved_syn *saved_syn;
		6679	u32 mac_hdrlen;
		6680	void *base;
		6681
		6682	if (tcp_sk(sk)->save_syn == 2) { /* Save full header. */
		6683	base = skb_mac_header(skb);
		6684	mac_hdrlen = skb_mac_header_len(skb);
		6685	len += mac_hdrlen;
		6686	} else {
		6687	base = skb_network_header(skb);
		6688	mac_hdrlen = 0;
		6689	}
6603		6690
6604	copy = kmalloc(len + sizeof(u32), GFP_ATOMIC);	6691	saved_syn = kmalloc(struct_size(saved_syn, data, len),
6605	if (copy) {	6692	GFP_ATOMIC);
6606	copy[0] = len;	6693	if (saved_syn) {
6607	memcpy(&copy[1], skb_network_header(skb), len);	6694	saved_syn->mac_hdrlen = mac_hdrlen;
6608	req->saved_syn = copy;	6695	saved_syn->network_hdrlen = skb_network_header_len(skb);
		6696	saved_syn->tcp_hdrlen = tcp_hdrlen(skb);
		6697	memcpy(saved_syn->data, base, len);
		6698	req->saved_syn = saved_syn;
6609	}	6699	}
6610	}	6700	}
6611	}	6701	}
@@ -6752,7 +6842,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops, · 140 unmodified lines
6752	}	6842	}
6753	if (fastopen_sk) {	6843	if (fastopen_sk) {
6754	af_ops->send_synack(fastopen_sk, dst, &fl, req,	6844	af_ops->send_synack(fastopen_sk, dst, &fl, req,
6755	&foc, TCP_SYNACK_FASTOPEN);	6845	&foc, TCP_SYNACK_FASTOPEN, skb);
6756	/* Add the child socket directly into the accept queue */	6846	/* Add the child socket directly into the accept queue */
6757	if (!inet_csk_reqsk_queue_add(sk, req, fastopen_sk)) {	6847	if (!inet_csk_reqsk_queue_add(sk, req, fastopen_sk)) {
6758	reqsk_fastopen_remove(fastopen_sk, req, false);	6848	reqsk_fastopen_remove(fastopen_sk, req, false);
@@ -6770,7 +6860,8 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops, · 11 unmodified lines
6770	tcp_timeout_init((struct sock *)req));	6860	tcp_timeout_init((struct sock *)req));
6771	af_ops->send_synack(sk, dst, &fl, req, &foc,	6861	af_ops->send_synack(sk, dst, &fl, req, &foc,
6772	!want_cookie ? TCP_SYNACK_NORMAL :	6862	!want_cookie ? TCP_SYNACK_NORMAL :
6773	TCP_SYNACK_COOKIE);	6863	TCP_SYNACK_COOKIE,
		6864	skb);
6774	if (want_cookie) {	6865	if (want_cookie) {
6775	reqsk_free(req);	6866	reqsk_free(req);
6776	return 0;	6867	return 0;

net/ipv4/tcp_ipv4.c+3−2

 			      struct flowi *fl,
 			      struct request_sock *req,
 			      struct tcp_fastopen_cookie *foc,
-			      enum tcp_synack_type synack_type)
+			      enum tcp_synack_type synack_type,
+			      struct sk_buff *syn_skb)
 {
 	const struct inet_request_sock *ireq = inet_rsk(req);
 	struct flowi4 fl4;
 	if (!dst && (dst = inet_csk_route_req(sk, &fl4, req)) == NULL)
 		return -1;
-	skb = tcp_make_synack(sk, dst, req, foc, synack_type);
+	skb = tcp_make_synack(sk, dst, req, foc, synack_type, syn_skb);
 	if (skb) {
 		__tcp_v4_send_check(skb, ireq->ir_loc_addr, ireq->ir_rmt_addr);

net/ipv4/tcp_minisocks.c+1−0

@@ -548,6 +548,7 @@ struct sock tcp_create_openreq_child(const struct sock sk, · 547 unmodified lines
548	newtp->fastopen_req = NULL;	548	newtp->fastopen_req = NULL;
549	RCU_INIT_POINTER(newtp->fastopen_rsk, NULL);	549	RCU_INIT_POINTER(newtp->fastopen_rsk, NULL);
550		550
		551	bpf_skops_init_child(sk, newsk);
551	tcp_bpf_clone(sk, newsk);	552	tcp_bpf_clone(sk, newsk);
552		553
553	__TCP_INC_STATS(sock_net(sk), TCP_MIB_PASSIVEOPENS);	554	__TCP_INC_STATS(sock_net(sk), TCP_MIB_PASSIVEOPENS);

net/ipv4/tcp_output.c+180−13

@@ -438,6 +438,7 @@ struct tcp_out_options { · 437 unmodified lines
438	u8 ws; /* window scale, 0 to disable */	438	u8 ws; /* window scale, 0 to disable */
439	u8 num_sack_blocks; /* number of SACK blocks to include */	439	u8 num_sack_blocks; /* number of SACK blocks to include */
440	u8 hash_size; /* bytes in hash_location */	440	u8 hash_size; /* bytes in hash_location */
		441	u8 bpf_opt_len; /* length of BPF hdr option */
441	__u8 hash_location; / temporary pointer, overloaded */	442	__u8 hash_location; / temporary pointer, overloaded */
442	__u32 tsval, tsecr; /* need to include OPTION_TS */	443	__u32 tsval, tsecr; /* need to include OPTION_TS */
443	struct tcp_fastopen_cookie fastopen_cookie; / Fast open cookie */	444	struct tcp_fastopen_cookie fastopen_cookie; / Fast open cookie */
@@ -452,6 +453,145 @@ static void mptcp_options_write(__be32 ptr, struct tcp_out_options opts) · 8 unmodified lines
452	#endif	453	#endif
453	}	454	}
454		455
		456	#ifdef CONFIG_CGROUP_BPF
		457	static int bpf_skops_write_hdr_opt_arg0(struct sk_buff *skb,
		458	enum tcp_synack_type synack_type)
		459	{
		460	if (unlikely(!skb))
		461	return BPF_WRITE_HDR_TCP_CURRENT_MSS;
		462
		463	if (unlikely(synack_type == TCP_SYNACK_COOKIE))
		464	return BPF_WRITE_HDR_TCP_SYNACK_COOKIE;
		465
		466	return 0;
		467	}
		468
		469	/* req, syn_skb and synack_type are used when writing synack */
		470	static void bpf_skops_hdr_opt_len(struct sock sk, struct sk_buff skb,
		471	struct request_sock *req,
		472	struct sk_buff *syn_skb,
		473	enum tcp_synack_type synack_type,
		474	struct tcp_out_options *opts,
		475	unsigned int *remaining)
		476	{
		477	struct bpf_sock_ops_kern sock_ops;
		478	int err;
		479
		480	if (likely(!BPF_SOCK_OPS_TEST_FLAG(tcp_sk(sk),
		481	BPF_SOCK_OPS_WRITE_HDR_OPT_CB_FLAG)) \|\|
		482	!*remaining)
		483	return;
		484
		485	/* remaining has already been aligned to 4 bytes, so remaining >= 4 */
		486
		487	/* init sock_ops */
		488	memset(&sock_ops, 0, offsetof(struct bpf_sock_ops_kern, temp));
		489
		490	sock_ops.op = BPF_SOCK_OPS_HDR_OPT_LEN_CB;
		491
		492	if (req) {
		493	/* The listen "sk" cannot be passed here because
		494	* it is not locked. It would not make too much
		495	* sense to do bpf_setsockopt(listen_sk) based
		496	* on individual connection request also.
		497	*
		498	* Thus, "req" is passed here and the cgroup-bpf-progs
		499	* of the listen "sk" will be run.
		500	*
		501	* "req" is also used here for fastopen even the "sk" here is
		502	* a fullsock "child" sk. It is to keep the behavior
		503	* consistent between fastopen and non-fastopen on
		504	* the bpf programming side.
		505	*/
		506	sock_ops.sk = (struct sock *)req;
		507	sock_ops.syn_skb = syn_skb;
		508	} else {
		509	sock_owned_by_me(sk);
		510
		511	sock_ops.is_fullsock = 1;
		512	sock_ops.sk = sk;
		513	}
		514
		515	sock_ops.args[0] = bpf_skops_write_hdr_opt_arg0(skb, synack_type);
		516	sock_ops.remaining_opt_len = *remaining;
		517	/* tcp_current_mss() does not pass a skb */
		518	if (skb)
		519	bpf_skops_init_skb(&sock_ops, skb, 0);
		520
		521	err = BPF_CGROUP_RUN_PROG_SOCK_OPS_SK(&sock_ops, sk);
		522
		523	if (err \|\| sock_ops.remaining_opt_len == *remaining)
		524	return;
		525
		526	opts->bpf_opt_len = *remaining - sock_ops.remaining_opt_len;
		527	/* round up to 4 bytes */
		528	opts->bpf_opt_len = (opts->bpf_opt_len + 3) & ~3;
		529
		530	*remaining -= opts->bpf_opt_len;
		531	}
		532
		533	static void bpf_skops_write_hdr_opt(struct sock sk, struct sk_buff skb,
		534	struct request_sock *req,
		535	struct sk_buff *syn_skb,
		536	enum tcp_synack_type synack_type,
		537	struct tcp_out_options *opts)
		538	{
		539	u8 first_opt_off, nr_written, max_opt_len = opts->bpf_opt_len;
		540	struct bpf_sock_ops_kern sock_ops;
		541	int err;
		542
		543	if (likely(!max_opt_len))
		544	return;
		545
		546	memset(&sock_ops, 0, offsetof(struct bpf_sock_ops_kern, temp));
		547
		548	sock_ops.op = BPF_SOCK_OPS_WRITE_HDR_OPT_CB;
		549
		550	if (req) {
		551	sock_ops.sk = (struct sock *)req;
		552	sock_ops.syn_skb = syn_skb;
		553	} else {
		554	sock_owned_by_me(sk);
		555
		556	sock_ops.is_fullsock = 1;
		557	sock_ops.sk = sk;
		558	}
		559
		560	sock_ops.args[0] = bpf_skops_write_hdr_opt_arg0(skb, synack_type);
		561	sock_ops.remaining_opt_len = max_opt_len;
		562	first_opt_off = tcp_hdrlen(skb) - max_opt_len;
		563	bpf_skops_init_skb(&sock_ops, skb, first_opt_off);
		564
		565	err = BPF_CGROUP_RUN_PROG_SOCK_OPS_SK(&sock_ops, sk);
		566
		567	if (err)
		568	nr_written = 0;
		569	else
		570	nr_written = max_opt_len - sock_ops.remaining_opt_len;
		571
		572	if (nr_written < max_opt_len)
		573	memset(skb->data + first_opt_off + nr_written, TCPOPT_NOP,
		574	max_opt_len - nr_written);
		575	}
		576	#else
		577	static void bpf_skops_hdr_opt_len(struct sock sk, struct sk_buff skb,
		578	struct request_sock *req,
		579	struct sk_buff *syn_skb,
		580	enum tcp_synack_type synack_type,
		581	struct tcp_out_options *opts,
		582	unsigned int *remaining)
		583	{
		584	}
		585
		586	static void bpf_skops_write_hdr_opt(struct sock sk, struct sk_buff skb,
		587	struct request_sock *req,
		588	struct sk_buff *syn_skb,
		589	enum tcp_synack_type synack_type,
		590	struct tcp_out_options *opts)
		591	{
		592	}
		593	#endif
		594
455	/* Write previously computed TCP options to the packet.	595	/* Write previously computed TCP options to the packet.
456	*	596	*
457	* Beware: Something in the Internet is very sensitive to the ordering of	597	* Beware: Something in the Internet is very sensitive to the ordering of
@@ -691,6 +831,8 @@ static unsigned int tcp_syn_options(struct sock sk, struct sk_buff skb, · 233 unmodified lines
691	}	831	}
692	}	832	}
693		833
		834	bpf_skops_hdr_opt_len(sk, skb, NULL, NULL, 0, opts, &remaining);
		835
694	return MAX_TCP_OPTION_SPACE - remaining;	836	return MAX_TCP_OPTION_SPACE - remaining;
695	}	837	}
696		838
@@ -701,7 +843,8 @@ static unsigned int tcp_synack_options(const struct sock *sk, · 4 unmodified lines
701	struct tcp_out_options *opts,	843	struct tcp_out_options *opts,
702	const struct tcp_md5sig_key *md5,	844	const struct tcp_md5sig_key *md5,
703	struct tcp_fastopen_cookie *foc,	845	struct tcp_fastopen_cookie *foc,
704	enum tcp_synack_type synack_type)	846	enum tcp_synack_type synack_type,
		847	struct sk_buff *syn_skb)
705	{	848	{
706	struct inet_request_sock *ireq = inet_rsk(req);	849	struct inet_request_sock *ireq = inet_rsk(req);
707	unsigned int remaining = MAX_TCP_OPTION_SPACE;	850	unsigned int remaining = MAX_TCP_OPTION_SPACE;
@@ -758,6 +901,9 @@ static unsigned int tcp_synack_options(const struct sock *sk, · 50 unmodified lines
758		901
759	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);	902	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
760		903
		904	bpf_skops_hdr_opt_len((struct sock *)sk, skb, req, syn_skb,
		905	synack_type, opts, &remaining);
		906
761	return MAX_TCP_OPTION_SPACE - remaining;	907	return MAX_TCP_OPTION_SPACE - remaining;
762	}	908	}
763		909
@@ -826,6 +972,15 @@ static unsigned int tcp_established_options(struct sock sk, struct sk_buff skb · 62 unmodified lines
826	opts->num_sack_blocks * TCPOLEN_SACK_PERBLOCK;	972	opts->num_sack_blocks * TCPOLEN_SACK_PERBLOCK;
827	}	973	}
828		974
		975	if (unlikely(BPF_SOCK_OPS_TEST_FLAG(tp,
		976	BPF_SOCK_OPS_WRITE_HDR_OPT_CB_FLAG))) {
		977	unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
		978
		979	bpf_skops_hdr_opt_len(sk, skb, NULL, NULL, 0, opts, &remaining);
		980
		981	size = MAX_TCP_OPTION_SPACE - remaining;
		982	}
		983
829	return size;	984	return size;
830	}	985	}
831		986
@@ -1213,6 +1368,9 @@ static int __tcp_transmit_skb(struct sock sk, struct sk_buff skb, · 381 unmodified lines
1213	}	1368	}
1214	#endif	1369	#endif
1215		1370
		1371	/* BPF prog is the last one writing header option */
		1372	bpf_skops_write_hdr_opt(sk, skb, NULL, NULL, 0, &opts);
		1373
1216	INDIRECT_CALL_INET(icsk->icsk_af_ops->send_check,	1374	INDIRECT_CALL_INET(icsk->icsk_af_ops->send_check,
1217	tcp_v6_send_check, tcp_v4_send_check,	1375	tcp_v6_send_check, tcp_v4_send_check,
1218	sk, skb);	1376	sk, skb);
@@ -3336,20 +3494,20 @@ int tcp_send_synack(struct sock *sk) · 2117 unmodified lines
3336	}	3494	}
3337		3495
3338	/**	3496	/**
3339	* tcp_make_synack - Prepare a SYN-ACK.	3497	* tcp_make_synack - Allocate one skb and build a SYNACK packet.
3340	* sk: listener socket	3498	* @sk: listener socket
3341	* dst: dst entry attached to the SYNACK	3499	* @dst: dst entry attached to the SYNACK. It is consumed and caller
3342	* req: request_sock pointer	3500	* should not use it again.
3343	* foc: cookie for tcp fast open	3501	* @req: request_sock pointer
3344	* synack_type: Type of synback to prepare	3502	* @foc: cookie for tcp fast open
3345	*	3503	* @synack_type: Type of synack to prepare
3346	* Allocate one skb and build a SYNACK packet.	3504	* @syn_skb: SYN packet just received. It could be NULL for rtx case.
3347	* @dst is consumed : Caller should not use it again.
3348	*/	3505	*/
3349	struct sk_buff tcp_make_synack(const struct sock sk, struct dst_entry *dst,	3506	struct sk_buff tcp_make_synack(const struct sock sk, struct dst_entry *dst,
3350	struct request_sock *req,	3507	struct request_sock *req,
3351	struct tcp_fastopen_cookie *foc,	3508	struct tcp_fastopen_cookie *foc,
3352	enum tcp_synack_type synack_type)	3509	enum tcp_synack_type synack_type,
		3510	struct sk_buff *syn_skb)
3353	{	3511	{
3354	struct inet_request_sock *ireq = inet_rsk(req);	3512	struct inet_request_sock *ireq = inet_rsk(req);
3355	const struct tcp_sock *tp = tcp_sk(sk);	3513	const struct tcp_sock *tp = tcp_sk(sk);
@@ -3408,8 +3566,11 @@ struct sk_buff tcp_make_synack(const struct sock sk, struct dst_entry *dst, · 52 unmodified lines
3408	md5 = tcp_rsk(req)->af_specific->req_md5_lookup(sk, req_to_sk(req));	3566	md5 = tcp_rsk(req)->af_specific->req_md5_lookup(sk, req_to_sk(req));
3409	#endif	3567	#endif
3410	skb_set_hash(skb, tcp_rsk(req)->txhash, PKT_HASH_TYPE_L4);	3568	skb_set_hash(skb, tcp_rsk(req)->txhash, PKT_HASH_TYPE_L4);
		3569	/* bpf program will be interested in the tcp_flags */
		3570	TCP_SKB_CB(skb)->tcp_flags = TCPHDR_SYN \| TCPHDR_ACK;
3411	tcp_header_size = tcp_synack_options(sk, req, mss, skb, &opts, md5,	3571	tcp_header_size = tcp_synack_options(sk, req, mss, skb, &opts, md5,
3412	foc, synack_type) + sizeof(*th);	3572	foc, synack_type,
		3573	syn_skb) + sizeof(*th);
3413		3574
3414	skb_push(skb, tcp_header_size);	3575	skb_push(skb, tcp_header_size);
3415	skb_reset_transport_header(skb);	3576	skb_reset_transport_header(skb);
@@ -3441,6 +3602,9 @@ struct sk_buff tcp_make_synack(const struct sock sk, struct dst_entry *dst, · 25 unmodified lines
3441	rcu_read_unlock();	3602	rcu_read_unlock();
3442	#endif	3603	#endif
3443		3604
		3605	bpf_skops_write_hdr_opt((struct sock *)sk, skb, req, syn_skb,
		3606	synack_type, &opts);
		3607
3444	skb->skb_mstamp_ns = now;	3608	skb->skb_mstamp_ns = now;
3445	tcp_add_tx_delay(skb, tp);	3609	tcp_add_tx_delay(skb, tp);
3446		3610
@@ -3741,6 +3905,8 @@ void tcp_send_delayed_ack(struct sock *sk) · 294 unmodified lines
3741	ato = min(ato, max_ato);	3905	ato = min(ato, max_ato);
3742	}	3906	}
3743		3907
		3908	ato = min_t(u32, ato, inet_csk(sk)->icsk_delack_max);
		3909
3744	/* Stay within the limit we were given */	3910	/* Stay within the limit we were given */
3745	timeout = jiffies + ato;	3911	timeout = jiffies + ato;
3746		3912
@@ -3934,7 +4100,8 @@ int tcp_rtx_synack(const struct sock sk, struct request_sock req) · 187 unmodified lines
3934	int res;	4100	int res;
3935		4101
3936	tcp_rsk(req)->txhash = net_tx_rndhash();	4102	tcp_rsk(req)->txhash = net_tx_rndhash();
3937	res = af_ops->send_synack(sk, NULL, &fl, req, NULL, TCP_SYNACK_NORMAL);	4103	res = af_ops->send_synack(sk, NULL, &fl, req, NULL, TCP_SYNACK_NORMAL,
		4104	NULL);
3938	if (!res) {	4105	if (!res) {
3939	__TCP_INC_STATS(sock_net(sk), TCP_MIB_RETRANSSEGS);	4106	__TCP_INC_STATS(sock_net(sk), TCP_MIB_RETRANSSEGS);
3940	__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPSYNRETRANS);	4107	__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPSYNRETRANS);

net/ipv4/udp_bpf.c+4−5

 	prot->close  = sock_map_close;
 }
-static void udp_bpf_check_v6_needs_rebuild(struct sock *sk, struct proto *ops)
+static void udp_bpf_check_v6_needs_rebuild(struct proto *ops)
 {
-	if (sk->sk_family == AF_INET6 &&
+	if (unlikely(ops != smp_load_acquire(&udpv6_prot_saved))) {
-	    unlikely(ops != smp_load_acquire(&udpv6_prot_saved))) {
 		spin_lock_bh(&udpv6_prot_lock);
 		if (likely(ops != udpv6_prot_saved)) {
 			udp_bpf_rebuild_protos(&udp_bpf_prots[UDP_BPF_IPV6], ops);
 {
 	int family = sk->sk_family == AF_INET ? UDP_BPF_IPV4 : UDP_BPF_IPV6;
-	if (!psock->sk_proto)
+	if (sk->sk_family == AF_INET6)
-		udp_bpf_check_v6_needs_rebuild(sk, READ_ONCE(sk->sk_prot));
+		udp_bpf_check_v6_needs_rebuild(psock->sk_proto);
 	return &udp_bpf_prots[family];
 }

net/ipv6/tcp_ipv6.c+3−2

 			      struct flowi *fl,
 			      struct request_sock *req,
 			      struct tcp_fastopen_cookie *foc,
-			      enum tcp_synack_type synack_type)
+			      enum tcp_synack_type synack_type,
+			      struct sk_buff *syn_skb)
 {
 	struct inet_request_sock *ireq = inet_rsk(req);
 	struct ipv6_pinfo *np = tcp_inet6_sk(sk);
 					       IPPROTO_TCP)) == NULL)
 		goto done;
-	skb = tcp_make_synack(sk, dst, req, foc, synack_type);
+	skb = tcp_make_synack(sk, dst, req, foc, synack_type, syn_skb);
 	if (skb) {
 		__tcp_v6_send_check(skb, &ireq->ir_v6_loc_addr,

net/xdp/xdp_umem.c+27−198

@@ -23,162 +23,6 @@ · 22 unmodified lines
23		23
24	static DEFINE_IDA(umem_ida);	24	static DEFINE_IDA(umem_ida);
25		25
26	void xdp_add_sk_umem(struct xdp_umem umem, struct xdp_sock xs)
27	{
28	unsigned long flags;
29
30	if (!xs->tx)
31	return;
32
33	spin_lock_irqsave(&umem->xsk_tx_list_lock, flags);
34	list_add_rcu(&xs->list, &umem->xsk_tx_list);
35	spin_unlock_irqrestore(&umem->xsk_tx_list_lock, flags);
36	}
37
38	void xdp_del_sk_umem(struct xdp_umem umem, struct xdp_sock xs)
39	{
40	unsigned long flags;
41
42	if (!xs->tx)
43	return;
44
45	spin_lock_irqsave(&umem->xsk_tx_list_lock, flags);
46	list_del_rcu(&xs->list);
47	spin_unlock_irqrestore(&umem->xsk_tx_list_lock, flags);
48	}
49
50	/* The umem is stored both in the _rx struct and the _tx struct as we do
51	* not know if the device has more tx queues than rx, or the opposite.
52	* This might also change during run time.
53	*/
54	static int xdp_reg_umem_at_qid(struct net_device dev, struct xdp_umem umem,
55	u16 queue_id)
56	{
57	if (queue_id >= max_t(unsigned int,
58	dev->real_num_rx_queues,
59	dev->real_num_tx_queues))
60	return -EINVAL;
61
62	if (queue_id < dev->real_num_rx_queues)
63	dev->_rx[queue_id].umem = umem;
64	if (queue_id < dev->real_num_tx_queues)
65	dev->_tx[queue_id].umem = umem;
66
67	return 0;
68	}
69
70	struct xdp_umem xdp_get_umem_from_qid(struct net_device dev,
71	u16 queue_id)
72	{
73	if (queue_id < dev->real_num_rx_queues)
74	return dev->_rx[queue_id].umem;
75	if (queue_id < dev->real_num_tx_queues)
76	return dev->_tx[queue_id].umem;
77
78	return NULL;
79	}
80	EXPORT_SYMBOL(xdp_get_umem_from_qid);
81
82	static void xdp_clear_umem_at_qid(struct net_device *dev, u16 queue_id)
83	{
84	if (queue_id < dev->real_num_rx_queues)
85	dev->_rx[queue_id].umem = NULL;
86	if (queue_id < dev->real_num_tx_queues)
87	dev->_tx[queue_id].umem = NULL;
88	}
89
90	int xdp_umem_assign_dev(struct xdp_umem umem, struct net_device dev,
91	u16 queue_id, u16 flags)
92	{
93	bool force_zc, force_copy;
94	struct netdev_bpf bpf;
95	int err = 0;
96
97	ASSERT_RTNL();
98
99	force_zc = flags & XDP_ZEROCOPY;
100	force_copy = flags & XDP_COPY;
101
102	if (force_zc && force_copy)
103	return -EINVAL;
104
105	if (xdp_get_umem_from_qid(dev, queue_id))
106	return -EBUSY;
107
108	err = xdp_reg_umem_at_qid(dev, umem, queue_id);
109	if (err)
110	return err;
111
112	umem->dev = dev;
113	umem->queue_id = queue_id;
114
115	if (flags & XDP_USE_NEED_WAKEUP) {
116	umem->flags \|= XDP_UMEM_USES_NEED_WAKEUP;
117	/* Tx needs to be explicitly woken up the first time.
118	* Also for supporting drivers that do not implement this
119	* feature. They will always have to call sendto().
120	*/
121	xsk_set_tx_need_wakeup(umem);
122	}
123
124	dev_hold(dev);
125
126	if (force_copy)
127	/* For copy-mode, we are done. */
128	return 0;
129
130	if (!dev->netdev_ops->ndo_bpf \|\| !dev->netdev_ops->ndo_xsk_wakeup) {
131	err = -EOPNOTSUPP;
132	goto err_unreg_umem;
133	}
134
135	bpf.command = XDP_SETUP_XSK_UMEM;
136	bpf.xsk.umem = umem;
137	bpf.xsk.queue_id = queue_id;
138
139	err = dev->netdev_ops->ndo_bpf(dev, &bpf);
140	if (err)
141	goto err_unreg_umem;
142
143	umem->zc = true;
144	return 0;
145
146	err_unreg_umem:
147	if (!force_zc)
148	err = 0; /* fallback to copy mode */
149	if (err)
150	xdp_clear_umem_at_qid(dev, queue_id);
151	return err;
152	}
153
154	void xdp_umem_clear_dev(struct xdp_umem *umem)
155	{
156	struct netdev_bpf bpf;
157	int err;
158
159	ASSERT_RTNL();
160
161	if (!umem->dev)
162	return;
163
164	if (umem->zc) {
165	bpf.command = XDP_SETUP_XSK_UMEM;
166	bpf.xsk.umem = NULL;
167	bpf.xsk.queue_id = umem->queue_id;
168
169	err = umem->dev->netdev_ops->ndo_bpf(umem->dev, &bpf);
170
171	if (err)
172	WARN(1, "failed to disable umem!\n");
173	}
174
175	xdp_clear_umem_at_qid(umem->dev, umem->queue_id);
176
177	dev_put(umem->dev);
178	umem->dev = NULL;
179	umem->zc = false;
180	}
181
182	static void xdp_umem_unpin_pages(struct xdp_umem *umem)	26	static void xdp_umem_unpin_pages(struct xdp_umem *umem)
183	{	27	{
184	unpin_user_pages_dirty_lock(umem->pgs, umem->npgs, true);	28	unpin_user_pages_dirty_lock(umem->pgs, umem->npgs, true);
@@ -195,38 +39,33 @@ static void xdp_umem_unaccount_pages(struct xdp_umem *umem) · 10 unmodified lines
195	}	39	}
196	}	40	}
197		41
198	static void xdp_umem_release(struct xdp_umem *umem)	42	static void xdp_umem_addr_unmap(struct xdp_umem *umem)
199	{	43	{
200	rtnl_lock();	44	vunmap(umem->addrs);
201	xdp_umem_clear_dev(umem);	45	umem->addrs = NULL;
202	rtnl_unlock();	46	}
203
204	ida_simple_remove(&umem_ida, umem->id);
205		47
206	if (umem->fq) {	48	static int xdp_umem_addr_map(struct xdp_umem umem, struct page *pages,
207	xskq_destroy(umem->fq);	49	u32 nr_pages)
208	umem->fq = NULL;	50	{
209	}	51	umem->addrs = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL);
		52	if (!umem->addrs)
		53	return -ENOMEM;
		54	return 0;
		55	}
210		56
211	if (umem->cq) {	57	static void xdp_umem_release(struct xdp_umem *umem)
212	xskq_destroy(umem->cq);	58	{
213	umem->cq = NULL;	59	umem->zc = false;
214	}	60	ida_simple_remove(&umem_ida, umem->id);
215		61
216	xp_destroy(umem->pool);	62	xdp_umem_addr_unmap(umem);
217	xdp_umem_unpin_pages(umem);	63	xdp_umem_unpin_pages(umem);
218		64
219	xdp_umem_unaccount_pages(umem);	65	xdp_umem_unaccount_pages(umem);
220	kfree(umem);	66	kfree(umem);
221	}	67	}
222		68
223	static void xdp_umem_release_deferred(struct work_struct *work)
224	{
225	struct xdp_umem *umem = container_of(work, struct xdp_umem, work);
226
227	xdp_umem_release(umem);
228	}
229
230	void xdp_get_umem(struct xdp_umem *umem)	69	void xdp_get_umem(struct xdp_umem *umem)
231	{	70	{
232	refcount_inc(&umem->users);	71	refcount_inc(&umem->users);
@@ -237,10 +76,8 @@ void xdp_put_umem(struct xdp_umem *umem) · 4 unmodified lines
237	if (!umem)	76	if (!umem)
238	return;	77	return;
239		78
240	if (refcount_dec_and_test(&umem->users)) {	79	if (refcount_dec_and_test(&umem->users))
241	INIT_WORK(&umem->work, xdp_umem_release_deferred);	80	xdp_umem_release(umem);
242	schedule_work(&umem->work);
243	}
244	}	81	}
245		82
246	static int xdp_umem_pin_pages(struct xdp_umem *umem, unsigned long address)	83	static int xdp_umem_pin_pages(struct xdp_umem *umem, unsigned long address)
@@ -319,8 +156,7 @@ static int xdp_umem_reg(struct xdp_umem umem, struct xdp_umem_reg mr) · 72 unmodified lines
319	return -EINVAL;	156	return -EINVAL;
320	}	157	}
321		158
322	if (mr->flags & ~(XDP_UMEM_UNALIGNED_CHUNK_FLAG \|	159	if (mr->flags & ~XDP_UMEM_UNALIGNED_CHUNK_FLAG)
323	XDP_UMEM_USES_NEED_WAKEUP))
324	return -EINVAL;	160	return -EINVAL;
325		161
326	if (!unaligned_chunks && !is_power_of_2(chunk_size))	162	if (!unaligned_chunks && !is_power_of_2(chunk_size))
@@ -356,13 +192,13 @@ static int xdp_umem_reg(struct xdp_umem umem, struct xdp_umem_reg mr) · 29 unmodified lines
356	umem->size = size;	192	umem->size = size;
357	umem->headroom = headroom;	193	umem->headroom = headroom;
358	umem->chunk_size = chunk_size;	194	umem->chunk_size = chunk_size;
		195	umem->chunks = chunks;
359	umem->npgs = (u32)npgs;	196	umem->npgs = (u32)npgs;
360	umem->pgs = NULL;	197	umem->pgs = NULL;
361	umem->user = NULL;	198	umem->user = NULL;
362	umem->flags = mr->flags;	199	umem->flags = mr->flags;
363	INIT_LIST_HEAD(&umem->xsk_tx_list);
364	spin_lock_init(&umem->xsk_tx_list_lock);
365		200
		201	INIT_LIST_HEAD(&umem->xsk_dma_list);
366	refcount_set(&umem->users, 1);	202	refcount_set(&umem->users, 1);
367		203
368	err = xdp_umem_account_pages(umem);	204	err = xdp_umem_account_pages(umem);
@@ -373,15 +209,13 @@ static int xdp_umem_reg(struct xdp_umem umem, struct xdp_umem_reg mr) · 4 unmodified lines
373	if (err)	209	if (err)
374	goto out_account;	210	goto out_account;
375		211
376	umem->pool = xp_create(umem->pgs, umem->npgs, chunks, chunk_size,	212	err = xdp_umem_addr_map(umem, umem->pgs, umem->npgs);
377	headroom, size, unaligned_chunks);	213	if (err)
378	if (!umem->pool) {	214	goto out_unpin;
379	err = -ENOMEM;	215
380	goto out_pin;
381	}
382	return 0;	216	return 0;
383		217
384	out_pin:	218	out_unpin:
385	xdp_umem_unpin_pages(umem);	219	xdp_umem_unpin_pages(umem);
386	out_account:	220	out_account:
387	xdp_umem_unaccount_pages(umem);	221	xdp_umem_unaccount_pages(umem);
@@ -413,8 +247,3 @@ struct xdp_umem xdp_umem_create(struct xdp_umem_reg mr) · 25 unmodified lines
413		247
414	return umem;	248	return umem;
415	}	249	}
416
417	bool xdp_umem_validate_queues(struct xdp_umem *umem)
418	{
419	return umem->fq && umem->cq;
420	}

net/xdp/xdp_umem.h+0−6

 #include <net/xdp_sock_drv.h>
-int xdp_umem_assign_dev(struct xdp_umem *umem, struct net_device *dev,
-			u16 queue_id, u16 flags);
-void xdp_umem_clear_dev(struct xdp_umem *umem);
-bool xdp_umem_validate_queues(struct xdp_umem *umem);
 void xdp_get_umem(struct xdp_umem *umem);
 void xdp_put_umem(struct xdp_umem *umem);
-void xdp_add_sk_umem(struct xdp_umem *umem, struct xdp_sock *xs);
-void xdp_del_sk_umem(struct xdp_umem *umem, struct xdp_sock *xs);
 struct xdp_umem *xdp_umem_create(struct xdp_umem_reg *mr);
 #endif /* XDP_UMEM_H_ */

net/xdp/xsk.c+142−71

@@ -36,68 +36,108 @@ static DEFINE_PER_CPU(struct list_head, xskmap_flush_list); · 35 unmodified lines
36	bool xsk_is_setup_for_bpf_map(struct xdp_sock *xs)	36	bool xsk_is_setup_for_bpf_map(struct xdp_sock *xs)
37	{	37	{
38	return READ_ONCE(xs->rx) && READ_ONCE(xs->umem) &&	38	return READ_ONCE(xs->rx) && READ_ONCE(xs->umem) &&
39	READ_ONCE(xs->umem->fq);	39	(xs->pool->fq \|\| READ_ONCE(xs->fq_tmp));
40	}	40	}
41		41
42	void xsk_set_rx_need_wakeup(struct xdp_umem *umem)	42	void xsk_set_rx_need_wakeup(struct xsk_buff_pool *pool)
43	{	43	{
44	if (umem->need_wakeup & XDP_WAKEUP_RX)	44	if (pool->cached_need_wakeup & XDP_WAKEUP_RX)
45	return;	45	return;
46		46
47	umem->fq->ring->flags \|= XDP_RING_NEED_WAKEUP;	47	pool->fq->ring->flags \|= XDP_RING_NEED_WAKEUP;
48	umem->need_wakeup \|= XDP_WAKEUP_RX;	48	pool->cached_need_wakeup \|= XDP_WAKEUP_RX;
49	}	49	}
50	EXPORT_SYMBOL(xsk_set_rx_need_wakeup);	50	EXPORT_SYMBOL(xsk_set_rx_need_wakeup);
51		51
52	void xsk_set_tx_need_wakeup(struct xdp_umem *umem)	52	void xsk_set_tx_need_wakeup(struct xsk_buff_pool *pool)
53	{	53	{
54	struct xdp_sock *xs;	54	struct xdp_sock *xs;
55		55
56	if (umem->need_wakeup & XDP_WAKEUP_TX)	56	if (pool->cached_need_wakeup & XDP_WAKEUP_TX)
57	return;	57	return;
58		58
59	rcu_read_lock();	59	rcu_read_lock();
60	list_for_each_entry_rcu(xs, &umem->xsk_tx_list, list) {	60	list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) {
61	xs->tx->ring->flags \|= XDP_RING_NEED_WAKEUP;	61	xs->tx->ring->flags \|= XDP_RING_NEED_WAKEUP;
62	}	62	}
63	rcu_read_unlock();	63	rcu_read_unlock();
64		64
65	umem->need_wakeup \|= XDP_WAKEUP_TX;	65	pool->cached_need_wakeup \|= XDP_WAKEUP_TX;
66	}	66	}
67	EXPORT_SYMBOL(xsk_set_tx_need_wakeup);	67	EXPORT_SYMBOL(xsk_set_tx_need_wakeup);
68		68
69	void xsk_clear_rx_need_wakeup(struct xdp_umem *umem)	69	void xsk_clear_rx_need_wakeup(struct xsk_buff_pool *pool)
70	{	70	{
71	if (!(umem->need_wakeup & XDP_WAKEUP_RX))	71	if (!(pool->cached_need_wakeup & XDP_WAKEUP_RX))
72	return;	72	return;
73		73
74	umem->fq->ring->flags &= ~XDP_RING_NEED_WAKEUP;	74	pool->fq->ring->flags &= ~XDP_RING_NEED_WAKEUP;
75	umem->need_wakeup &= ~XDP_WAKEUP_RX;	75	pool->cached_need_wakeup &= ~XDP_WAKEUP_RX;
76	}	76	}
77	EXPORT_SYMBOL(xsk_clear_rx_need_wakeup);	77	EXPORT_SYMBOL(xsk_clear_rx_need_wakeup);
78		78
79	void xsk_clear_tx_need_wakeup(struct xdp_umem *umem)	79	void xsk_clear_tx_need_wakeup(struct xsk_buff_pool *pool)
80	{	80	{
81	struct xdp_sock *xs;	81	struct xdp_sock *xs;
82		82
83	if (!(umem->need_wakeup & XDP_WAKEUP_TX))	83	if (!(pool->cached_need_wakeup & XDP_WAKEUP_TX))
84	return;	84	return;
85		85
86	rcu_read_lock();	86	rcu_read_lock();
87	list_for_each_entry_rcu(xs, &umem->xsk_tx_list, list) {	87	list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) {
88	xs->tx->ring->flags &= ~XDP_RING_NEED_WAKEUP;	88	xs->tx->ring->flags &= ~XDP_RING_NEED_WAKEUP;
89	}	89	}
90	rcu_read_unlock();	90	rcu_read_unlock();
91		91
92	umem->need_wakeup &= ~XDP_WAKEUP_TX;	92	pool->cached_need_wakeup &= ~XDP_WAKEUP_TX;
93	}	93	}
94	EXPORT_SYMBOL(xsk_clear_tx_need_wakeup);	94	EXPORT_SYMBOL(xsk_clear_tx_need_wakeup);
95		95
96	bool xsk_umem_uses_need_wakeup(struct xdp_umem *umem)	96	bool xsk_uses_need_wakeup(struct xsk_buff_pool *pool)
97	{	97	{
98	return umem->flags & XDP_UMEM_USES_NEED_WAKEUP;	98	return pool->uses_need_wakeup;
		99	}
		100	EXPORT_SYMBOL(xsk_uses_need_wakeup);
		101
		102	struct xsk_buff_pool xsk_get_pool_from_qid(struct net_device dev,
		103	u16 queue_id)
		104	{
		105	if (queue_id < dev->real_num_rx_queues)
		106	return dev->_rx[queue_id].pool;
		107	if (queue_id < dev->real_num_tx_queues)
		108	return dev->_tx[queue_id].pool;
		109
		110	return NULL;
		111	}
		112	EXPORT_SYMBOL(xsk_get_pool_from_qid);
		113
		114	void xsk_clear_pool_at_qid(struct net_device *dev, u16 queue_id)
		115	{
		116	if (queue_id < dev->real_num_rx_queues)
		117	dev->_rx[queue_id].pool = NULL;
		118	if (queue_id < dev->real_num_tx_queues)
		119	dev->_tx[queue_id].pool = NULL;
		120	}
		121
		122	/* The buffer pool is stored both in the _rx struct and the _tx struct as we do
		123	* not know if the device has more tx queues than rx, or the opposite.
		124	* This might also change during run time.
		125	*/
		126	int xsk_reg_pool_at_qid(struct net_device dev, struct xsk_buff_pool pool,
		127	u16 queue_id)
		128	{
		129	if (queue_id >= max_t(unsigned int,
		130	dev->real_num_rx_queues,
		131	dev->real_num_tx_queues))
		132	return -EINVAL;
		133
		134	if (queue_id < dev->real_num_rx_queues)
		135	dev->_rx[queue_id].pool = pool;
		136	if (queue_id < dev->real_num_tx_queues)
		137	dev->_tx[queue_id].pool = pool;
		138
		139	return 0;
99	}	140	}
100	EXPORT_SYMBOL(xsk_umem_uses_need_wakeup);
101		141
102	void xp_release(struct xdp_buff_xsk *xskb)	142	void xp_release(struct xdp_buff_xsk *xskb)
103	{	143	{
@@ -155,12 +195,12 @@ static int __xsk_rcv(struct xdp_sock xs, struct xdp_buff xdp, u32 len, · 51 unmodified lines
155	struct xdp_buff *xsk_xdp;	195	struct xdp_buff *xsk_xdp;
156	int err;	196	int err;
157		197
158	if (len > xsk_umem_get_rx_frame_size(xs->umem)) {	198	if (len > xsk_pool_get_rx_frame_size(xs->pool)) {
159	xs->rx_dropped++;	199	xs->rx_dropped++;
160	return -ENOSPC;	200	return -ENOSPC;
161	}	201	}
162		202
163	xsk_xdp = xsk_buff_alloc(xs->umem);	203	xsk_xdp = xsk_buff_alloc(xs->pool);
164	if (!xsk_xdp) {	204	if (!xsk_xdp) {
165	xs->rx_dropped++;	205	xs->rx_dropped++;
166	return -ENOSPC;	206	return -ENOSPC;
@@ -208,7 +248,7 @@ static int xsk_rcv(struct xdp_sock xs, struct xdp_buff xdp, · 41 unmodified lines
208	static void xsk_flush(struct xdp_sock *xs)	248	static void xsk_flush(struct xdp_sock *xs)
209	{	249	{
210	xskq_prod_submit(xs->rx);	250	xskq_prod_submit(xs->rx);
211	__xskq_cons_release(xs->umem->fq);	251	__xskq_cons_release(xs->pool->fq);
212	sock_def_readable(&xs->sk);	252	sock_def_readable(&xs->sk);
213	}	253	}
214		254
@@ -249,32 +289,32 @@ void __xsk_map_flush(void) · 34 unmodified lines
249	}	289	}
250	}	290	}
251		291
252	void xsk_umem_complete_tx(struct xdp_umem *umem, u32 nb_entries)	292	void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries)
253	{	293	{
254	xskq_prod_submit_n(umem->cq, nb_entries);	294	xskq_prod_submit_n(pool->cq, nb_entries);
255	}	295	}
256	EXPORT_SYMBOL(xsk_umem_complete_tx);	296	EXPORT_SYMBOL(xsk_tx_completed);
257		297
258	void xsk_umem_consume_tx_done(struct xdp_umem *umem)	298	void xsk_tx_release(struct xsk_buff_pool *pool)
259	{	299	{
260	struct xdp_sock *xs;	300	struct xdp_sock *xs;
261		301
262	rcu_read_lock();	302	rcu_read_lock();
263	list_for_each_entry_rcu(xs, &umem->xsk_tx_list, list) {	303	list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) {
264	__xskq_cons_release(xs->tx);	304	__xskq_cons_release(xs->tx);
265	xs->sk.sk_write_space(&xs->sk);	305	xs->sk.sk_write_space(&xs->sk);
266	}	306	}
267	rcu_read_unlock();	307	rcu_read_unlock();
268	}	308	}
269	EXPORT_SYMBOL(xsk_umem_consume_tx_done);	309	EXPORT_SYMBOL(xsk_tx_release);
270		310
271	bool xsk_umem_consume_tx(struct xdp_umem umem, struct xdp_desc desc)	311	bool xsk_tx_peek_desc(struct xsk_buff_pool pool, struct xdp_desc desc)
272	{	312	{
273	struct xdp_sock *xs;	313	struct xdp_sock *xs;
274		314
275	rcu_read_lock();	315	rcu_read_lock();
276	list_for_each_entry_rcu(xs, &umem->xsk_tx_list, list) {	316	list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) {
277	if (!xskq_cons_peek_desc(xs->tx, desc, umem)) {	317	if (!xskq_cons_peek_desc(xs->tx, desc, pool)) {
278	xs->tx->queue_empty_descs++;	318	xs->tx->queue_empty_descs++;
279	continue;	319	continue;
280	}	320	}
@@ -284,7 +324,7 @@ bool xsk_umem_consume_tx(struct xdp_umem umem, struct xdp_desc desc) · 3 unmodified lines
284	* if there is space in it. This avoids having to implement	324	* if there is space in it. This avoids having to implement
285	* any buffering in the Tx path.	325	* any buffering in the Tx path.
286	*/	326	*/
287	if (xskq_prod_reserve_addr(umem->cq, desc->addr))	327	if (xskq_prod_reserve_addr(pool->cq, desc->addr))
288	goto out;	328	goto out;
289		329
290	xskq_cons_release(xs->tx);	330	xskq_cons_release(xs->tx);
@@ -296,7 +336,7 @@ out: · 5 unmodified lines
296	rcu_read_unlock();	336	rcu_read_unlock();
297	return false;	337	return false;
298	}	338	}
299	EXPORT_SYMBOL(xsk_umem_consume_tx);	339	EXPORT_SYMBOL(xsk_tx_peek_desc);
300		340
301	static int xsk_wakeup(struct xdp_sock *xs, u8 flags)	341	static int xsk_wakeup(struct xdp_sock *xs, u8 flags)
302	{	342	{
@@ -322,7 +362,7 @@ static void xsk_destruct_skb(struct sk_buff *skb) · 19 unmodified lines
322	unsigned long flags;	362	unsigned long flags;
323		363
324	spin_lock_irqsave(&xs->tx_completion_lock, flags);	364	spin_lock_irqsave(&xs->tx_completion_lock, flags);
325	xskq_prod_submit_addr(xs->umem->cq, addr);	365	xskq_prod_submit_addr(xs->pool->cq, addr);
326	spin_unlock_irqrestore(&xs->tx_completion_lock, flags);	366	spin_unlock_irqrestore(&xs->tx_completion_lock, flags);
327		367
328	sock_wfree(skb);	368	sock_wfree(skb);
@@ -342,7 +382,7 @@ static int xsk_generic_xmit(struct sock *sk) · 13 unmodified lines
342	if (xs->queue_id >= xs->dev->real_num_tx_queues)	382	if (xs->queue_id >= xs->dev->real_num_tx_queues)
343	goto out;	383	goto out;
344		384
345	while (xskq_cons_peek_desc(xs->tx, &desc, xs->umem)) {	385	while (xskq_cons_peek_desc(xs->tx, &desc, xs->pool)) {
346	char *buffer;	386	char *buffer;
347	u64 addr;	387	u64 addr;
348	u32 len;	388	u32 len;
@@ -359,14 +399,14 @@ static int xsk_generic_xmit(struct sock *sk) · 10 unmodified lines
359		399
360	skb_put(skb, len);	400	skb_put(skb, len);
361	addr = desc.addr;	401	addr = desc.addr;
362	buffer = xsk_buff_raw_get_data(xs->umem, addr);	402	buffer = xsk_buff_raw_get_data(xs->pool, addr);
363	err = skb_store_bits(skb, 0, buffer, len);	403	err = skb_store_bits(skb, 0, buffer, len);
364	/* This is the backpressure mechanism for the Tx path.	404	/* This is the backpressure mechanism for the Tx path.
365	* Reserve space in the completion queue and only proceed	405	* Reserve space in the completion queue and only proceed
366	* if there is space in it. This avoids having to implement	406	* if there is space in it. This avoids having to implement
367	* any buffering in the Tx path.	407	* any buffering in the Tx path.
368	*/	408	*/
369	if (unlikely(err) \|\| xskq_prod_reserve(xs->umem->cq)) {	409	if (unlikely(err) \|\| xskq_prod_reserve(xs->pool->cq)) {
370	kfree_skb(skb);	410	kfree_skb(skb);
371	goto out;	411	goto out;
372	}	412	}
@@ -431,16 +471,16 @@ static __poll_t xsk_poll(struct file file, struct socket sock, · 58 unmodified lines
431	__poll_t mask = datagram_poll(file, sock, wait);	471	__poll_t mask = datagram_poll(file, sock, wait);
432	struct sock *sk = sock->sk;	472	struct sock *sk = sock->sk;
433	struct xdp_sock *xs = xdp_sk(sk);	473	struct xdp_sock *xs = xdp_sk(sk);
434	struct xdp_umem *umem;	474	struct xsk_buff_pool *pool;
435		475
436	if (unlikely(!xsk_is_bound(xs)))	476	if (unlikely(!xsk_is_bound(xs)))
437	return mask;	477	return mask;
438		478
439	umem = xs->umem;	479	pool = xs->pool;
440		480
441	if (umem->need_wakeup) {	481	if (pool->cached_need_wakeup) {
442	if (xs->zc)	482	if (xs->zc)
443	xsk_wakeup(xs, umem->need_wakeup);	483	xsk_wakeup(xs, pool->cached_need_wakeup);
444	else	484	else
445	/* Poll needs to drive Tx also in copy mode */	485	/* Poll needs to drive Tx also in copy mode */
446	__xsk_sendmsg(sk);	486	__xsk_sendmsg(sk);
@@ -481,7 +521,7 @@ static void xsk_unbind_dev(struct xdp_sock *xs) · 34 unmodified lines
481	WRITE_ONCE(xs->state, XSK_UNBOUND);	521	WRITE_ONCE(xs->state, XSK_UNBOUND);
482		522
483	/* Wait for driver to stop using the xdp socket. */	523	/* Wait for driver to stop using the xdp socket. */
484	xdp_del_sk_umem(xs->umem, xs);	524	xp_del_xsk(xs->pool, xs);
485	xs->dev = NULL;	525	xs->dev = NULL;
486	synchronize_net();	526	synchronize_net();
487	dev_put(dev);	527	dev_put(dev);
@@ -559,6 +599,8 @@ static int xsk_release(struct socket *sock) · 71 unmodified lines
559		599
560	xskq_destroy(xs->rx);	600	xskq_destroy(xs->rx);
561	xskq_destroy(xs->tx);	601	xskq_destroy(xs->tx);
		602	xskq_destroy(xs->fq_tmp);
		603	xskq_destroy(xs->cq_tmp);
562		604
563	sock_orphan(sk);	605	sock_orphan(sk);
564	sock->sk = NULL;	606	sock->sk = NULL;
@@ -586,6 +628,11 @@ static struct socket *xsk_lookup_xsk_from_fd(int fd) · 21 unmodified lines
586	return sock;	628	return sock;
587	}	629	}
588		630
		631	static bool xsk_validate_queues(struct xdp_sock *xs)
		632	{
		633	return xs->fq_tmp && xs->cq_tmp;
		634	}
		635
589	static int xsk_bind(struct socket sock, struct sockaddr addr, int addr_len)	636	static int xsk_bind(struct socket sock, struct sockaddr addr, int addr_len)
590	{	637	{
591	struct sockaddr_xdp sxdp = (struct sockaddr_xdp )addr;	638	struct sockaddr_xdp sxdp = (struct sockaddr_xdp )addr;
@@ -654,29 +701,64 @@ static int xsk_bind(struct socket sock, struct sockaddr addr, int addr_len) · 62 unmodified lines
654	sockfd_put(sock);	701	sockfd_put(sock);
655	goto out_unlock;	702	goto out_unlock;
656	}	703	}
657	if (umem_xs->dev != dev \|\| umem_xs->queue_id != qid) {	704
658	err = -EINVAL;	705	if (umem_xs->queue_id != qid \|\| umem_xs->dev != dev) {
659	sockfd_put(sock);	706	/* Share the umem with another socket on another qid
660	goto out_unlock;	707	* and/or device.
		708	*/
		709	xs->pool = xp_create_and_assign_umem(xs,
		710	umem_xs->umem);
		711	if (!xs->pool) {
		712	sockfd_put(sock);
		713	goto out_unlock;
		714	}
		715
		716	err = xp_assign_dev_shared(xs->pool, umem_xs->umem,
		717	dev, qid);
		718	if (err) {
		719	xp_destroy(xs->pool);
		720	sockfd_put(sock);
		721	goto out_unlock;
		722	}
		723	} else {
		724	/* Share the buffer pool with the other socket. */
		725	if (xs->fq_tmp \|\| xs->cq_tmp) {
		726	/* Do not allow setting your own fq or cq. */
		727	err = -EINVAL;
		728	sockfd_put(sock);
		729	goto out_unlock;
		730	}
		731
		732	xp_get_pool(umem_xs->pool);
		733	xs->pool = umem_xs->pool;
661	}	734	}
662		735
663	xdp_get_umem(umem_xs->umem);	736	xdp_get_umem(umem_xs->umem);
664	WRITE_ONCE(xs->umem, umem_xs->umem);	737	WRITE_ONCE(xs->umem, umem_xs->umem);
665	sockfd_put(sock);	738	sockfd_put(sock);
666	} else if (!xs->umem \|\| !xdp_umem_validate_queues(xs->umem)) {	739	} else if (!xs->umem \|\| !xsk_validate_queues(xs)) {
667	err = -EINVAL;	740	err = -EINVAL;
668	goto out_unlock;	741	goto out_unlock;
669	} else {	742	} else {
670	/* This xsk has its own umem. */	743	/* This xsk has its own umem. */
671	err = xdp_umem_assign_dev(xs->umem, dev, qid, flags);	744	xs->pool = xp_create_and_assign_umem(xs, xs->umem);
672	if (err)	745	if (!xs->pool) {
		746	err = -ENOMEM;
		747	goto out_unlock;
		748	}
		749
		750	err = xp_assign_dev(xs->pool, dev, qid, flags);
		751	if (err) {
		752	xp_destroy(xs->pool);
		753	xs->pool = NULL;
673	goto out_unlock;	754	goto out_unlock;
		755	}
674	}	756	}
675		757
676	xs->dev = dev;	758	xs->dev = dev;
677	xs->zc = xs->umem->zc;	759	xs->zc = xs->umem->zc;
678	xs->queue_id = qid;	760	xs->queue_id = qid;
679	xdp_add_sk_umem(xs->umem, xs);	761	xp_add_xsk(xs->pool, xs);
680		762
681	out_unlock:	763	out_unlock:
682	if (err) {	764	if (err) {
@@ -782,16 +864,10 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname, · 99 unmodified lines
782	mutex_unlock(&xs->mutex);	864	mutex_unlock(&xs->mutex);
783	return -EBUSY;	865	return -EBUSY;
784	}	866	}
785	if (!xs->umem) {
786	mutex_unlock(&xs->mutex);
787	return -EINVAL;
788	}
789		867
790	q = (optname == XDP_UMEM_FILL_RING) ? &xs->umem->fq :	868	q = (optname == XDP_UMEM_FILL_RING) ? &xs->fq_tmp :
791	&xs->umem->cq;	869	&xs->cq_tmp;
792	err = xsk_init_queue(entries, q, true);	870	err = xsk_init_queue(entries, q, true);
793	if (optname == XDP_UMEM_FILL_RING)
794	xp_set_fq(xs->umem->pool, *q);
795	mutex_unlock(&xs->mutex);	871	mutex_unlock(&xs->mutex);
796	return err;	872	return err;
797	}	873	}
@@ -858,7 +934,7 @@ static int xsk_getsockopt(struct socket *sock, int level, int optname, · 60 unmodified lines
858	if (extra_stats) {	934	if (extra_stats) {
859	stats.rx_ring_full = xs->rx_queue_full;	935	stats.rx_ring_full = xs->rx_queue_full;
860	stats.rx_fill_ring_empty_descs =	936	stats.rx_fill_ring_empty_descs =
861	xs->umem ? xskq_nb_queue_empty_descs(xs->umem->fq) : 0;	937	xs->pool ? xskq_nb_queue_empty_descs(xs->pool->fq) : 0;
862	stats.tx_ring_empty_descs = xskq_nb_queue_empty_descs(xs->tx);	938	stats.tx_ring_empty_descs = xskq_nb_queue_empty_descs(xs->tx);
863	} else {	939	} else {
864	stats.rx_dropped += xs->rx_queue_full;	940	stats.rx_dropped += xs->rx_queue_full;
@@ -960,7 +1036,6 @@ static int xsk_mmap(struct file file, struct socket sock, · 95 unmodified lines
960	unsigned long size = vma->vm_end - vma->vm_start;	1036	unsigned long size = vma->vm_end - vma->vm_start;
961	struct xdp_sock *xs = xdp_sk(sock->sk);	1037	struct xdp_sock *xs = xdp_sk(sock->sk);
962	struct xsk_queue *q = NULL;	1038	struct xsk_queue *q = NULL;
963	struct xdp_umem *umem;
964	unsigned long pfn;	1039	unsigned long pfn;
965	struct page *qpg;	1040	struct page *qpg;
966		1041
@@ -972,16 +1047,12 @@ static int xsk_mmap(struct file file, struct socket sock, · 5 unmodified lines
972	} else if (offset == XDP_PGOFF_TX_RING) {	1047	} else if (offset == XDP_PGOFF_TX_RING) {
973	q = READ_ONCE(xs->tx);	1048	q = READ_ONCE(xs->tx);
974	} else {	1049	} else {
975	umem = READ_ONCE(xs->umem);
976	if (!umem)
977	return -EINVAL;
978
979	/* Matches the smp_wmb() in XDP_UMEM_REG */	1050	/* Matches the smp_wmb() in XDP_UMEM_REG */
980	smp_rmb();	1051	smp_rmb();
981	if (offset == XDP_UMEM_PGOFF_FILL_RING)	1052	if (offset == XDP_UMEM_PGOFF_FILL_RING)
982	q = READ_ONCE(umem->fq);	1053	q = READ_ONCE(xs->fq_tmp);
983	else if (offset == XDP_UMEM_PGOFF_COMPLETION_RING)	1054	else if (offset == XDP_UMEM_PGOFF_COMPLETION_RING)
984	q = READ_ONCE(umem->cq);	1055	q = READ_ONCE(xs->cq_tmp);
985	}	1056	}
986		1057
987	if (!q)	1058	if (!q)
@@ -1019,8 +1090,8 @@ static int xsk_notifier(struct notifier_block *this, · 31 unmodified lines
1019		1090
1020	xsk_unbind_dev(xs);	1091	xsk_unbind_dev(xs);
1021		1092
1022	/* Clear device references in umem. */	1093	/* Clear device references. */
1023	xdp_umem_clear_dev(xs->umem);	1094	xp_clear_dev(xs->pool);
1024	}	1095	}
1025	mutex_unlock(&xs->mutex);	1096	mutex_unlock(&xs->mutex);
1026	}	1097	}
@@ -1064,7 +1135,7 @@ static void xsk_destruct(struct sock *sk) · 37 unmodified lines
1064	if (!sock_flag(sk, SOCK_DEAD))	1135	if (!sock_flag(sk, SOCK_DEAD))
1065	return;	1136	return;
1066		1137
1067	xdp_put_umem(xs->umem);	1138	xp_put_pool(xs->pool);
1068		1139
1069	sk_refcnt_debug_dec(sk);	1140	sk_refcnt_debug_dec(sk);
1070	}	1141	}
@@ -1072,8 +1143,8 @@ static void xsk_destruct(struct sock *sk) · 1 unmodified line
1072	static int xsk_create(struct net net, struct socket sock, int protocol,	1143	static int xsk_create(struct net net, struct socket sock, int protocol,
1073	int kern)	1144	int kern)
1074	{	1145	{
1075	struct sock *sk;
1076	struct xdp_sock *xs;	1146	struct xdp_sock *xs;
		1147	struct sock *sk;
1077		1148
1078	if (!ns_capable(net->user_ns, CAP_NET_RAW))	1149	if (!ns_capable(net->user_ns, CAP_NET_RAW))
1079	return -EPERM;	1150	return -EPERM;

net/xdp/xsk.h+3−7

 #define XSK_NEXT_PG_CONTIG_SHIFT 0
 #define XSK_NEXT_PG_CONTIG_MASK BIT_ULL(XSK_NEXT_PG_CONTIG_SHIFT)
-/* Flags for the umem flags field.
- *
- * The NEED_WAKEUP flag is 1 due to the reuse of the flags field for public
- * flags. See inlude/uapi/include/linux/if_xdp.h.
- */
-#define XDP_UMEM_USES_NEED_WAKEUP BIT(1)
 struct xdp_ring_offset_v1 {
 	__u64 producer;
 	__u64 consumer;
 			     struct xdp_sock **map_entry);
 int xsk_map_inc(struct xsk_map *map);
 void xsk_map_put(struct xsk_map *map);
+void xsk_clear_pool_at_qid(struct net_device *dev, u16 queue_id);
+int xsk_reg_pool_at_qid(struct net_device *dev, struct xsk_buff_pool *pool,
+			u16 queue_id);
 #endif /* XSK_H_ */

net/xdp/xsk_buff_pool.c+324−56

@@ -2,21 +2,37 @@ · 1 unmodified line
2		2
3	#include <net/xsk_buff_pool.h>	3	#include <net/xsk_buff_pool.h>
4	#include <net/xdp_sock.h>	4	#include <net/xdp_sock.h>
		5	#include <net/xdp_sock_drv.h>
		6	#include <linux/dma-direct.h>
		7	#include <linux/dma-noncoherent.h>
		8	#include <linux/swiotlb.h>
5		9
6	#include "xsk_queue.h"	10	#include "xsk_queue.h"
		11	#include "xdp_umem.h"
		12	#include "xsk.h"
7		13
8	static void xp_addr_unmap(struct xsk_buff_pool *pool)	14	void xp_add_xsk(struct xsk_buff_pool pool, struct xdp_sock xs)
9	{	15	{
10	vunmap(pool->addrs);	16	unsigned long flags;
		17
		18	if (!xs->tx)
		19	return;
		20
		21	spin_lock_irqsave(&pool->xsk_tx_list_lock, flags);
		22	list_add_rcu(&xs->tx_list, &pool->xsk_tx_list);
		23	spin_unlock_irqrestore(&pool->xsk_tx_list_lock, flags);
11	}	24	}
12		25
13	static int xp_addr_map(struct xsk_buff_pool *pool,	26	void xp_del_xsk(struct xsk_buff_pool pool, struct xdp_sock xs)
14	struct page **pages, u32 nr_pages)
15	{	27	{
16	pool->addrs = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL);	28	unsigned long flags;
17	if (!pool->addrs)	29
18	return -ENOMEM;	30	if (!xs->tx)
19	return 0;	31	return;
		32
		33	spin_lock_irqsave(&pool->xsk_tx_list_lock, flags);
		34	list_del_rcu(&xs->tx_list);
		35	spin_unlock_irqrestore(&pool->xsk_tx_list_lock, flags);
20	}	36	}
21		37
22	void xp_destroy(struct xsk_buff_pool *pool)	38	void xp_destroy(struct xsk_buff_pool *pool)
@@ -24,59 +40,61 @@ void xp_destroy(struct xsk_buff_pool *pool) · 1 unmodified line
24	if (!pool)	40	if (!pool)
25	return;	41	return;
26		42
27	xp_addr_unmap(pool);
28	kvfree(pool->heads);	43	kvfree(pool->heads);
29	kvfree(pool);	44	kvfree(pool);
30	}	45	}
31		46
32	struct xsk_buff_pool xp_create(struct page *pages, u32 nr_pages, u32 chunks,	47	struct xsk_buff_pool xp_create_and_assign_umem(struct xdp_sock xs,
33	u32 chunk_size, u32 headroom, u64 size,	48	struct xdp_umem *umem)
34	bool unaligned)
35	{	49	{
36	struct xsk_buff_pool *pool;	50	struct xsk_buff_pool *pool;
37	struct xdp_buff_xsk *xskb;	51	struct xdp_buff_xsk *xskb;
38	int err;
39	u32 i;	52	u32 i;
40		53
41	pool = kvzalloc(struct_size(pool, free_heads, chunks), GFP_KERNEL);	54	pool = kvzalloc(struct_size(pool, free_heads, umem->chunks),
		55	GFP_KERNEL);
42	if (!pool)	56	if (!pool)
43	goto out;	57	goto out;
44		58
45	pool->heads = kvcalloc(chunks, sizeof(*pool->heads), GFP_KERNEL);	59	pool->heads = kvcalloc(umem->chunks, sizeof(*pool->heads), GFP_KERNEL);
46	if (!pool->heads)	60	if (!pool->heads)
47	goto out;	61	goto out;
48		62
49	pool->chunk_mask = ~((u64)chunk_size - 1);	63	pool->chunk_mask = ~((u64)umem->chunk_size - 1);
50	pool->addrs_cnt = size;	64	pool->addrs_cnt = umem->size;
51	pool->heads_cnt = chunks;	65	pool->heads_cnt = umem->chunks;
52	pool->free_heads_cnt = chunks;	66	pool->free_heads_cnt = umem->chunks;
53	pool->headroom = headroom;	67	pool->headroom = umem->headroom;
54	pool->chunk_size = chunk_size;	68	pool->chunk_size = umem->chunk_size;
55	pool->unaligned = unaligned;	69	pool->unaligned = umem->flags & XDP_UMEM_UNALIGNED_CHUNK_FLAG;
56	pool->frame_len = chunk_size - headroom - XDP_PACKET_HEADROOM;	70	pool->frame_len = umem->chunk_size - umem->headroom -
		71	XDP_PACKET_HEADROOM;
		72	pool->umem = umem;
		73	pool->addrs = umem->addrs;
57	INIT_LIST_HEAD(&pool->free_list);	74	INIT_LIST_HEAD(&pool->free_list);
		75	INIT_LIST_HEAD(&pool->xsk_tx_list);
		76	spin_lock_init(&pool->xsk_tx_list_lock);
		77	refcount_set(&pool->users, 1);
		78
		79	pool->fq = xs->fq_tmp;
		80	pool->cq = xs->cq_tmp;
		81	xs->fq_tmp = NULL;
		82	xs->cq_tmp = NULL;
58		83
59	for (i = 0; i < pool->free_heads_cnt; i++) {	84	for (i = 0; i < pool->free_heads_cnt; i++) {
60	xskb = &pool->heads[i];	85	xskb = &pool->heads[i];
61	xskb->pool = pool;	86	xskb->pool = pool;
62	xskb->xdp.frame_sz = chunk_size - headroom;	87	xskb->xdp.frame_sz = umem->chunk_size - umem->headroom;
63	pool->free_heads[i] = xskb;	88	pool->free_heads[i] = xskb;
64	}	89	}
65		90
66	err = xp_addr_map(pool, pages, nr_pages);	91	return pool;
67	if (!err)
68	return pool;
69		92
70	out:	93	out:
71	xp_destroy(pool);	94	xp_destroy(pool);
72	return NULL;	95	return NULL;
73	}	96	}
74		97
75	void xp_set_fq(struct xsk_buff_pool pool, struct xsk_queue fq)
76	{
77	pool->fq = fq;
78	}
79
80	void xp_set_rxq_info(struct xsk_buff_pool pool, struct xdp_rxq_info rxq)	98	void xp_set_rxq_info(struct xsk_buff_pool pool, struct xdp_rxq_info rxq)
81	{	99	{
82	u32 i;	100	u32 i;
@@ -86,70 +104,320 @@ void xp_set_rxq_info(struct xsk_buff_pool pool, struct xdp_rxq_info rxq) · 3 unmodified lines
86	}	104	}
87	EXPORT_SYMBOL(xp_set_rxq_info);	105	EXPORT_SYMBOL(xp_set_rxq_info);
88		106
89	void xp_dma_unmap(struct xsk_buff_pool *pool, unsigned long attrs)	107	static void xp_disable_drv_zc(struct xsk_buff_pool *pool)
90	{	108	{
91	dma_addr_t *dma;	109	struct netdev_bpf bpf;
92	u32 i;	110	int err;
93		111
94	if (pool->dma_pages_cnt == 0)	112	ASSERT_RTNL();
		113
		114	if (pool->umem->zc) {
		115	bpf.command = XDP_SETUP_XSK_POOL;
		116	bpf.xsk.pool = NULL;
		117	bpf.xsk.queue_id = pool->queue_id;
		118
		119	err = pool->netdev->netdev_ops->ndo_bpf(pool->netdev, &bpf);
		120
		121	if (err)
		122	WARN(1, "Failed to disable zero-copy!\n");
		123	}
		124	}
		125
		126	static int __xp_assign_dev(struct xsk_buff_pool *pool,
		127	struct net_device *netdev, u16 queue_id, u16 flags)
		128	{
		129	bool force_zc, force_copy;
		130	struct netdev_bpf bpf;
		131	int err = 0;
		132
		133	ASSERT_RTNL();
		134
		135	force_zc = flags & XDP_ZEROCOPY;
		136	force_copy = flags & XDP_COPY;
		137
		138	if (force_zc && force_copy)
		139	return -EINVAL;
		140
		141	if (xsk_get_pool_from_qid(netdev, queue_id))
		142	return -EBUSY;
		143
		144	pool->netdev = netdev;
		145	pool->queue_id = queue_id;
		146	err = xsk_reg_pool_at_qid(netdev, pool, queue_id);
		147	if (err)
		148	return err;
		149
		150	if (flags & XDP_USE_NEED_WAKEUP) {
		151	pool->uses_need_wakeup = true;
		152	/* Tx needs to be explicitly woken up the first time.
		153	* Also for supporting drivers that do not implement this
		154	* feature. They will always have to call sendto().
		155	*/
		156	pool->cached_need_wakeup = XDP_WAKEUP_TX;
		157	}
		158
		159	dev_hold(netdev);
		160
		161	if (force_copy)
		162	/* For copy-mode, we are done. */
		163	return 0;
		164
		165	if (!netdev->netdev_ops->ndo_bpf \|\|
		166	!netdev->netdev_ops->ndo_xsk_wakeup) {
		167	err = -EOPNOTSUPP;
		168	goto err_unreg_pool;
		169	}
		170
		171	bpf.command = XDP_SETUP_XSK_POOL;
		172	bpf.xsk.pool = pool;
		173	bpf.xsk.queue_id = queue_id;
		174
		175	err = netdev->netdev_ops->ndo_bpf(netdev, &bpf);
		176	if (err)
		177	goto err_unreg_pool;
		178
		179	if (!pool->dma_pages) {
		180	WARN(1, "Driver did not DMA map zero-copy buffers");
		181	goto err_unreg_xsk;
		182	}
		183	pool->umem->zc = true;
		184	return 0;
		185
		186	err_unreg_xsk:
		187	xp_disable_drv_zc(pool);
		188	err_unreg_pool:
		189	if (!force_zc)
		190	err = 0; /* fallback to copy mode */
		191	if (err)
		192	xsk_clear_pool_at_qid(netdev, queue_id);
		193	return err;
		194	}
		195
		196	int xp_assign_dev(struct xsk_buff_pool pool, struct net_device dev,
		197	u16 queue_id, u16 flags)
		198	{
		199	return __xp_assign_dev(pool, dev, queue_id, flags);
		200	}
		201
		202	int xp_assign_dev_shared(struct xsk_buff_pool pool, struct xdp_umem umem,
		203	struct net_device *dev, u16 queue_id)
		204	{
		205	u16 flags;
		206
		207	/* One fill and completion ring required for each queue id. */
		208	if (!pool->fq \|\| !pool->cq)
		209	return -EINVAL;
		210
		211	flags = umem->zc ? XDP_ZEROCOPY : XDP_COPY;
		212	if (pool->uses_need_wakeup)
		213	flags \|= XDP_USE_NEED_WAKEUP;
		214
		215	return __xp_assign_dev(pool, dev, queue_id, flags);
		216	}
		217
		218	void xp_clear_dev(struct xsk_buff_pool *pool)
		219	{
		220	if (!pool->netdev)
95	return;	221	return;
96		222
97	for (i = 0; i < pool->dma_pages_cnt; i++) {	223	xp_disable_drv_zc(pool);
98	dma = &pool->dma_pages[i];	224	xsk_clear_pool_at_qid(pool->netdev, pool->queue_id);
		225	dev_put(pool->netdev);
		226	pool->netdev = NULL;
		227	}
		228
		229	static void xp_release_deferred(struct work_struct *work)
		230	{
		231	struct xsk_buff_pool *pool = container_of(work, struct xsk_buff_pool,
		232	work);
		233
		234	rtnl_lock();
		235	xp_clear_dev(pool);
		236	rtnl_unlock();
		237
		238	if (pool->fq) {
		239	xskq_destroy(pool->fq);
		240	pool->fq = NULL;
		241	}
		242
		243	if (pool->cq) {
		244	xskq_destroy(pool->cq);
		245	pool->cq = NULL;
		246	}
		247
		248	xdp_put_umem(pool->umem);
		249	xp_destroy(pool);
		250	}
		251
		252	void xp_get_pool(struct xsk_buff_pool *pool)
		253	{
		254	refcount_inc(&pool->users);
		255	}
		256
		257	void xp_put_pool(struct xsk_buff_pool *pool)
		258	{
		259	if (!pool)
		260	return;
		261
		262	if (refcount_dec_and_test(&pool->users)) {
		263	INIT_WORK(&pool->work, xp_release_deferred);
		264	schedule_work(&pool->work);
		265	}
		266	}
		267
		268	static struct xsk_dma_map xp_find_dma_map(struct xsk_buff_pool pool)
		269	{
		270	struct xsk_dma_map *dma_map;
		271
		272	list_for_each_entry(dma_map, &pool->umem->xsk_dma_list, list) {
		273	if (dma_map->netdev == pool->netdev)
		274	return dma_map;
		275	}
		276
		277	return NULL;
		278	}
		279
		280	static struct xsk_dma_map xp_create_dma_map(struct device dev, struct net_device *netdev,
		281	u32 nr_pages, struct xdp_umem *umem)
		282	{
		283	struct xsk_dma_map *dma_map;
		284
		285	dma_map = kzalloc(sizeof(*dma_map), GFP_KERNEL);
		286	if (!dma_map)
		287	return NULL;
		288
		289	dma_map->dma_pages = kvcalloc(nr_pages, sizeof(*dma_map->dma_pages), GFP_KERNEL);
		290	if (!dma_map) {
		291	kfree(dma_map);
		292	return NULL;
		293	}
		294
		295	dma_map->netdev = netdev;
		296	dma_map->dev = dev;
		297	dma_map->dma_need_sync = false;
		298	dma_map->dma_pages_cnt = nr_pages;
		299	refcount_set(&dma_map->users, 0);
		300	list_add(&dma_map->list, &umem->xsk_dma_list);
		301	return dma_map;
		302	}
		303
		304	static void xp_destroy_dma_map(struct xsk_dma_map *dma_map)
		305	{
		306	list_del(&dma_map->list);
		307	kvfree(dma_map->dma_pages);
		308	kfree(dma_map);
		309	}
		310
		311	static void __xp_dma_unmap(struct xsk_dma_map *dma_map, unsigned long attrs)
		312	{
		313	dma_addr_t *dma;
		314	u32 i;
		315
		316	for (i = 0; i < dma_map->dma_pages_cnt; i++) {
		317	dma = &dma_map->dma_pages[i];
99	if (*dma) {	318	if (*dma) {
100	dma_unmap_page_attrs(pool->dev, *dma, PAGE_SIZE,	319	dma_unmap_page_attrs(dma_map->dev, *dma, PAGE_SIZE,
101	DMA_BIDIRECTIONAL, attrs);	320	DMA_BIDIRECTIONAL, attrs);
102	*dma = 0;	321	*dma = 0;
103	}	322	}
104	}	323	}
105		324
		325	xp_destroy_dma_map(dma_map);
		326	}
		327
		328	void xp_dma_unmap(struct xsk_buff_pool *pool, unsigned long attrs)
		329	{
		330	struct xsk_dma_map *dma_map;
		331
		332	if (pool->dma_pages_cnt == 0)
		333	return;
		334
		335	dma_map = xp_find_dma_map(pool);
		336	if (!dma_map) {
		337	WARN(1, "Could not find dma_map for device");
		338	return;
		339	}
		340
		341	if (!refcount_dec_and_test(&dma_map->users))
		342	return;
		343
		344	__xp_dma_unmap(dma_map, attrs);
106	kvfree(pool->dma_pages);	345	kvfree(pool->dma_pages);
107	pool->dma_pages_cnt = 0;	346	pool->dma_pages_cnt = 0;
108	pool->dev = NULL;	347	pool->dev = NULL;
109	}	348	}
110	EXPORT_SYMBOL(xp_dma_unmap);	349	EXPORT_SYMBOL(xp_dma_unmap);
111		350
112	static void xp_check_dma_contiguity(struct xsk_buff_pool *pool)	351	static void xp_check_dma_contiguity(struct xsk_dma_map *dma_map)
113	{	352	{
114	u32 i;	353	u32 i;
115		354
116	for (i = 0; i < pool->dma_pages_cnt - 1; i++) {	355	for (i = 0; i < dma_map->dma_pages_cnt - 1; i++) {
117	if (pool->dma_pages[i] + PAGE_SIZE == pool->dma_pages[i + 1])	356	if (dma_map->dma_pages[i] + PAGE_SIZE == dma_map->dma_pages[i + 1])
118	pool->dma_pages[i] \|= XSK_NEXT_PG_CONTIG_MASK;	357	dma_map->dma_pages[i] \|= XSK_NEXT_PG_CONTIG_MASK;
119	else	358	else
120	pool->dma_pages[i] &= ~XSK_NEXT_PG_CONTIG_MASK;	359	dma_map->dma_pages[i] &= ~XSK_NEXT_PG_CONTIG_MASK;
121	}	360	}
122	}	361	}
123		362
		363	static int xp_init_dma_info(struct xsk_buff_pool pool, struct xsk_dma_map dma_map)
		364	{
		365	pool->dma_pages = kvcalloc(dma_map->dma_pages_cnt, sizeof(*pool->dma_pages), GFP_KERNEL);
		366	if (!pool->dma_pages)
		367	return -ENOMEM;
		368
		369	pool->dev = dma_map->dev;
		370	pool->dma_pages_cnt = dma_map->dma_pages_cnt;
		371	pool->dma_need_sync = dma_map->dma_need_sync;
		372	refcount_inc(&dma_map->users);
		373	memcpy(pool->dma_pages, dma_map->dma_pages,
		374	pool->dma_pages_cnt * sizeof(*pool->dma_pages));
		375
		376	return 0;
		377	}
		378
124	int xp_dma_map(struct xsk_buff_pool pool, struct device dev,	379	int xp_dma_map(struct xsk_buff_pool pool, struct device dev,
125	unsigned long attrs, struct page **pages, u32 nr_pages)	380	unsigned long attrs, struct page **pages, u32 nr_pages)
126	{	381	{
		382	struct xsk_dma_map *dma_map;
127	dma_addr_t dma;	383	dma_addr_t dma;
		384	int err;
128	u32 i;	385	u32 i;
129		386
130	pool->dma_pages = kvcalloc(nr_pages, sizeof(*pool->dma_pages),	387	dma_map = xp_find_dma_map(pool);
131	GFP_KERNEL);	388	if (dma_map) {
132	if (!pool->dma_pages)	389	err = xp_init_dma_info(pool, dma_map);
133	return -ENOMEM;	390	if (err)
		391	return err;
		392
		393	return 0;
		394	}
134		395
135	pool->dev = dev;	396	dma_map = xp_create_dma_map(dev, pool->netdev, nr_pages, pool->umem);
136	pool->dma_pages_cnt = nr_pages;	397	if (!dma_map)
137	pool->dma_need_sync = false;	398	return -ENOMEM;
138		399
139	for (i = 0; i < pool->dma_pages_cnt; i++) {	400	for (i = 0; i < dma_map->dma_pages_cnt; i++) {
140	dma = dma_map_page_attrs(dev, pages[i], 0, PAGE_SIZE,	401	dma = dma_map_page_attrs(dev, pages[i], 0, PAGE_SIZE,
141	DMA_BIDIRECTIONAL, attrs);	402	DMA_BIDIRECTIONAL, attrs);
142	if (dma_mapping_error(dev, dma)) {	403	if (dma_mapping_error(dev, dma)) {
143	xp_dma_unmap(pool, attrs);	404	__xp_dma_unmap(dma_map, attrs);
144	return -ENOMEM;	405	return -ENOMEM;
145	}	406	}
146	if (dma_need_sync(dev, dma))	407	if (dma_need_sync(dev, dma))
147	pool->dma_need_sync = true;	408	dma_map->dma_need_sync = true;
148	pool->dma_pages[i] = dma;	409	dma_map->dma_pages[i] = dma;
149	}	410	}
150		411
151	if (pool->unaligned)	412	if (pool->unaligned)
152	xp_check_dma_contiguity(pool);	413	xp_check_dma_contiguity(dma_map);
		414
		415	err = xp_init_dma_info(pool, dma_map);
		416	if (err) {
		417	__xp_dma_unmap(dma_map, attrs);
		418	return err;
		419	}
		420
153	return 0;	421	return 0;
154	}	422	}
155	EXPORT_SYMBOL(xp_dma_map);	423	EXPORT_SYMBOL(xp_dma_map);

net/xdp/xsk_diag.c+9−7

 static int xsk_diag_put_umem(const struct xdp_sock *xs, struct sk_buff *nlskb)
 {
+	struct xsk_buff_pool *pool = xs->pool;
 	struct xdp_umem *umem = xs->umem;
 	struct xdp_diag_umem du = {};
 	int err;
 	du.num_pages = umem->npgs;
 	du.chunk_size = umem->chunk_size;
 	du.headroom = umem->headroom;
-	du.ifindex = umem->dev ? umem->dev->ifindex : 0;
+	du.ifindex = pool->netdev ? pool->netdev->ifindex : 0;
-	du.queue_id = umem->queue_id;
+	du.queue_id = pool->queue_id;
 	du.flags = 0;
 	if (umem->zc)
 		du.flags |= XDP_DU_F_ZEROCOPY;
 	err = nla_put(nlskb, XDP_DIAG_UMEM, sizeof(du), &du);
-	if (!err && umem->fq)
+	if (!err && pool->fq)
-		err = xsk_diag_put_ring(umem->fq, XDP_DIAG_UMEM_FILL_RING, nlskb);
+		err = xsk_diag_put_ring(pool->fq,
-	if (!err && umem->cq) {
+					XDP_DIAG_UMEM_FILL_RING, nlskb);
-		err = xsk_diag_put_ring(umem->cq, XDP_DIAG_UMEM_COMPLETION_RING,
+	if (!err && pool->cq) {
+		err = xsk_diag_put_ring(pool->cq, XDP_DIAG_UMEM_COMPLETION_RING,
 					nlskb);
 	}
 	return err;
 	du.n_rx_dropped = xs->rx_dropped;
 	du.n_rx_invalid = xskq_nb_invalid_descs(xs->rx);
 	du.n_rx_full = xs->rx_queue_full;
-	du.n_fill_ring_empty = xs->umem ? xskq_nb_queue_empty_descs(xs->umem->fq) : 0;
+	du.n_fill_ring_empty = xs->pool ? xskq_nb_queue_empty_descs(xs->pool->fq) : 0;
 	du.n_tx_invalid = xskq_nb_invalid_descs(xs->tx);
 	du.n_tx_ring_empty = xskq_nb_queue_empty_descs(xs->tx);
 	return nla_put(nlskb, XDP_DIAG_STATS, sizeof(du), &du);

net/xdp/xsk_queue.h+6−6

@@ -166,9 +166,9 @@ static inline bool xp_validate_desc(struct xsk_buff_pool *pool, · 165 unmodified lines
166		166
167	static inline bool xskq_cons_is_valid_desc(struct xsk_queue *q,	167	static inline bool xskq_cons_is_valid_desc(struct xsk_queue *q,
168	struct xdp_desc *d,	168	struct xdp_desc *d,
169	struct xdp_umem *umem)	169	struct xsk_buff_pool *pool)
170	{	170	{
171	if (!xp_validate_desc(umem->pool, d)) {	171	if (!xp_validate_desc(pool, d)) {
172	q->invalid_descs++;	172	q->invalid_descs++;
173	return false;	173	return false;
174	}	174	}
@@ -177,14 +177,14 @@ static inline bool xskq_cons_is_valid_desc(struct xsk_queue *q, · 2 unmodified lines
177		177
178	static inline bool xskq_cons_read_desc(struct xsk_queue *q,	178	static inline bool xskq_cons_read_desc(struct xsk_queue *q,
179	struct xdp_desc *desc,	179	struct xdp_desc *desc,
180	struct xdp_umem *umem)	180	struct xsk_buff_pool *pool)
181	{	181	{
182	while (q->cached_cons != q->cached_prod) {	182	while (q->cached_cons != q->cached_prod) {
183	struct xdp_rxtx_ring ring = (struct xdp_rxtx_ring )q->ring;	183	struct xdp_rxtx_ring ring = (struct xdp_rxtx_ring )q->ring;
184	u32 idx = q->cached_cons & q->ring_mask;	184	u32 idx = q->cached_cons & q->ring_mask;
185		185
186	*desc = ring->desc[idx];	186	*desc = ring->desc[idx];
187	if (xskq_cons_is_valid_desc(q, desc, umem))	187	if (xskq_cons_is_valid_desc(q, desc, pool))
188	return true;	188	return true;
189		189
190	q->cached_cons++;	190	q->cached_cons++;
@@ -236,11 +236,11 @@ static inline bool xskq_cons_peek_addr_unchecked(struct xsk_queue q, u64 addr) · 45 unmodified lines
236		236
237	static inline bool xskq_cons_peek_desc(struct xsk_queue *q,	237	static inline bool xskq_cons_peek_desc(struct xsk_queue *q,
238	struct xdp_desc *desc,	238	struct xdp_desc *desc,
239	struct xdp_umem *umem)	239	struct xsk_buff_pool *pool)
240	{	240	{
241	if (q->cached_prod == q->cached_cons)	241	if (q->cached_prod == q->cached_cons)
242	xskq_cons_get_entries(q);	242	xskq_cons_get_entries(q);
243	return xskq_cons_read_desc(q, desc, umem);	243	return xskq_cons_read_desc(q, desc, pool);
244	}	244	}
245		245
246	static inline void xskq_cons_release(struct xsk_queue *q)	246	static inline void xskq_cons_release(struct xsk_queue *q)

net/xdp/xskmap.c+8−0

 	spin_unlock_bh(&map->lock);
 }
+static bool xsk_map_meta_equal(const struct bpf_map *meta0,
+			       const struct bpf_map *meta1)
+{
+	return meta0->max_entries == meta1->max_entries &&
+		bpf_map_meta_equal(meta0, meta1);
+}
 static int xsk_map_btf_id;
 const struct bpf_map_ops xsk_map_ops = {
+	.map_meta_equal = xsk_map_meta_equal,
 	.map_alloc = xsk_map_alloc,
 	.map_free = xsk_map_free,
 	.map_get_next_key = xsk_map_get_next_key,

samples/bpf/Makefile+12−9

 tprogs-y += cpustat
 tprogs-y += xdp_adjust_tail
 tprogs-y += xdpsock
+tprogs-y += xsk_fwd
 tprogs-y += xdp_fwd
 tprogs-y += task_fd_query
 tprogs-y += xdp_sample_pkts
 tracex5-objs := tracex5_user.o $(TRACE_HELPERS)
 tracex6-objs := tracex6_user.o
 tracex7-objs := tracex7_user.o
-test_probe_write_user-objs := bpf_load.o test_probe_write_user_user.o
+test_probe_write_user-objs := test_probe_write_user_user.o
-trace_output-objs := bpf_load.o trace_output_user.o $(TRACE_HELPERS)
+trace_output-objs := trace_output_user.o $(TRACE_HELPERS)
-lathist-objs := bpf_load.o lathist_user.o
+lathist-objs := lathist_user.o
-offwaketime-objs := bpf_load.o offwaketime_user.o $(TRACE_HELPERS)
+offwaketime-objs := offwaketime_user.o $(TRACE_HELPERS)
-spintest-objs := bpf_load.o spintest_user.o $(TRACE_HELPERS)
+spintest-objs := spintest_user.o $(TRACE_HELPERS)
-map_perf_test-objs := bpf_load.o map_perf_test_user.o
+map_perf_test-objs := map_perf_test_user.o
 test_overhead-objs := bpf_load.o test_overhead_user.o
 test_cgrp2_array_pin-objs := test_cgrp2_array_pin.o
 test_cgrp2_attach-objs := test_cgrp2_attach.o
 # reuse xdp1 source intentionally
 xdp2-objs := xdp1_user.o
 xdp_router_ipv4-objs := xdp_router_ipv4_user.o
-test_current_task_under_cgroup-objs := bpf_load.o $(CGROUP_HELPERS) \
+test_current_task_under_cgroup-objs := $(CGROUP_HELPERS) \
 				       test_current_task_under_cgroup_user.o
 trace_event-objs := trace_event_user.o $(TRACE_HELPERS)
 sampleip-objs := sampleip_user.o $(TRACE_HELPERS)
 xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o
 xdp_monitor-objs := bpf_load.o xdp_monitor_user.o
 xdp_rxq_info-objs := xdp_rxq_info_user.o
-syscall_tp-objs := bpf_load.o syscall_tp_user.o
+syscall_tp-objs := syscall_tp_user.o
-cpustat-objs := bpf_load.o cpustat_user.o
+cpustat-objs := cpustat_user.o
 xdp_adjust_tail-objs := xdp_adjust_tail_user.o
 xdpsock-objs := xdpsock_user.o
+xsk_fwd-objs := xsk_fwd.o
 xdp_fwd-objs := xdp_fwd_user.o
 task_fd_query-objs := bpf_load.o task_fd_query_user.o $(TRACE_HELPERS)
 xdp_sample_pkts-objs := xdp_sample_pkts_user.o $(TRACE_HELPERS)
 TPROGLDLIBS_map_perf_test	+= -lrt
 TPROGLDLIBS_test_overhead	+= -lrt
 TPROGLDLIBS_xdpsock		+= -pthread
+TPROGLDLIBS_xsk_fwd		+= -pthread
 # Allows pointing LLC/CLANG to a LLVM backend with bpf support, redefine on cmdline:
 #  make M=samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang

samples/bpf/cpustat_kern.c+18−18

@@ -51,28 +51,28 @@ static int cpu_opps[] = { 208000, 432000, 729000, 960000, 1200000 }; · 50 unmodified lines
51	#define MAP_OFF_PSTATE_IDX 3	51	#define MAP_OFF_PSTATE_IDX 3
52	#define MAP_OFF_NUM 4	52	#define MAP_OFF_NUM 4
53		53
54	struct bpf_map_def SEC("maps") my_map = {	54	struct {
55	.type = BPF_MAP_TYPE_ARRAY,	55	__uint(type, BPF_MAP_TYPE_ARRAY);
56	.key_size = sizeof(u32),	56	__type(key, u32);
57	.value_size = sizeof(u64),	57	__type(value, u64);
58	.max_entries = MAX_CPU * MAP_OFF_NUM,	58	__uint(max_entries, MAX_CPU * MAP_OFF_NUM);
59	};	59	} my_map SEC(".maps");
60		60
61	/* cstate_duration records duration time for every idle state per CPU */	61	/* cstate_duration records duration time for every idle state per CPU */
62	struct bpf_map_def SEC("maps") cstate_duration = {	62	struct {
63	.type = BPF_MAP_TYPE_ARRAY,	63	__uint(type, BPF_MAP_TYPE_ARRAY);
64	.key_size = sizeof(u32),	64	__type(key, u32);
65	.value_size = sizeof(u64),	65	__type(value, u64);
66	.max_entries = MAX_CPU * MAX_CSTATE_ENTRIES,	66	__uint(max_entries, MAX_CPU * MAX_CSTATE_ENTRIES);
67	};	67	} cstate_duration SEC(".maps");
68		68
69	/* pstate_duration records duration time for every operating point per CPU */	69	/* pstate_duration records duration time for every operating point per CPU */
70	struct bpf_map_def SEC("maps") pstate_duration = {	70	struct {
71	.type = BPF_MAP_TYPE_ARRAY,	71	__uint(type, BPF_MAP_TYPE_ARRAY);
72	.key_size = sizeof(u32),	72	__type(key, u32);
73	.value_size = sizeof(u64),	73	__type(value, u64);
74	.max_entries = MAX_CPU * MAX_PSTATE_ENTRIES,	74	__uint(max_entries, MAX_CPU * MAX_PSTATE_ENTRIES);
75	};	75	} pstate_duration SEC(".maps");
76		76
77	/*	77	/*
78	* The trace events for cpu_idle and cpu_frequency are taken from:	78	* The trace events for cpu_idle and cpu_frequency are taken from:

samples/bpf/cpustat_user.c+40−7

 #include <string.h>
 #include <unistd.h>
 #include <fcntl.h>
-#include <linux/bpf.h>
 #include <locale.h>
 #include <sys/types.h>
 #include <sys/stat.h>
 #include <sys/wait.h>
 #include <bpf/bpf.h>
-#include "bpf_load.h"
+#include <bpf/libbpf.h>
+static int cstate_map_fd, pstate_map_fd;
 #define MAX_CPU			8
 #define MAX_PSTATE_ENTRIES	5
 {
 	cpu_stat_inject_cpu_idle_event();
 	cpu_stat_inject_cpu_frequency_event();
-	cpu_stat_update(map_fd[1], map_fd[2]);
+	cpu_stat_update(cstate_map_fd, pstate_map_fd);
 	cpu_stat_print();
 	exit(0);
 }
 int main(int argc, char **argv)
 {
+	struct bpf_link *link = NULL;
+	struct bpf_program *prog;
+	struct bpf_object *obj;
 	char filename[256];
 	int ret;
 	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+	obj = bpf_object__open_file(filename, NULL);
+	if (libbpf_get_error(obj)) {
+		fprintf(stderr, "ERROR: opening BPF object file failed\n");
+		return 0;
+	}
-	if (load_bpf_file(filename)) {
+	prog = bpf_object__find_program_by_name(obj, "bpf_prog1");
-		printf("%s", bpf_log_buf);
+	if (!prog) {
-		return 1;
+		printf("finding a prog in obj file failed\n");
+		goto cleanup;
+	}
+	/* load BPF program */
+	if (bpf_object__load(obj)) {
+		fprintf(stderr, "ERROR: loading BPF object file failed\n");
+		goto cleanup;
+	}
+	cstate_map_fd = bpf_object__find_map_fd_by_name(obj, "cstate_duration");
+	pstate_map_fd = bpf_object__find_map_fd_by_name(obj, "pstate_duration");
+	if (cstate_map_fd < 0 || pstate_map_fd < 0) {
+		fprintf(stderr, "ERROR: finding a map in obj file failed\n");
+		goto cleanup;
+	}
+	link = bpf_program__attach(prog);
+	if (libbpf_get_error(link)) {
+		fprintf(stderr, "ERROR: bpf_program__attach failed\n");
+		link = NULL;
+		goto cleanup;
 	}
 	ret = cpu_stat_inject_cpu_idle_event();
 	signal(SIGTERM, int_exit);
 	while (1) {
-		cpu_stat_update(map_fd[1], map_fd[2]);
+		cpu_stat_update(cstate_map_fd, pstate_map_fd);
 		cpu_stat_print();
 		sleep(5);
 	}
+cleanup:
+	bpf_link__destroy(link);
+	bpf_object__close(obj);
 	return 0;
 }

samples/bpf/lathist_kern.c+12−12

@@ -18,12 +18,12 @@ · 17 unmodified lines
18	* trace_preempt_[on\|off] tracepoints hooks is not supported.	18	* trace_preempt_[on\|off] tracepoints hooks is not supported.
19	*/	19	*/
20		20
21	struct bpf_map_def SEC("maps") my_map = {	21	struct {
22	.type = BPF_MAP_TYPE_ARRAY,	22	__uint(type, BPF_MAP_TYPE_ARRAY);
23	.key_size = sizeof(int),	23	__type(key, int);
24	.value_size = sizeof(u64),	24	__type(value, u64);
25	.max_entries = MAX_CPU,	25	__uint(max_entries, MAX_CPU);
26	};	26	} my_map SEC(".maps");
27		27
28	SEC("kprobe/trace_preempt_off")	28	SEC("kprobe/trace_preempt_off")
29	int bpf_prog1(struct pt_regs *ctx)	29	int bpf_prog1(struct pt_regs *ctx)
@@ -61,12 +61,12 @@ static unsigned int log2l(unsigned long v) · 31 unmodified lines
61	return log2(v);	61	return log2(v);
62	}	62	}
63		63
64	struct bpf_map_def SEC("maps") my_lat = {	64	struct {
65	.type = BPF_MAP_TYPE_ARRAY,	65	__uint(type, BPF_MAP_TYPE_ARRAY);
66	.key_size = sizeof(int),	66	__type(key, int);
67	.value_size = sizeof(long),	67	__type(value, long);
68	.max_entries = MAX_CPU * MAX_ENTRIES,	68	__uint(max_entries, MAX_CPU * MAX_ENTRIES);
69	};	69	} my_lat SEC(".maps");
70		70
71	SEC("kprobe/trace_preempt_on")	71	SEC("kprobe/trace_preempt_on")
72	int bpf_prog2(struct pt_regs *ctx)	72	int bpf_prog2(struct pt_regs *ctx)

samples/bpf/lathist_user.c+36−6

@@ -6,9 +6,8 @@ · 5 unmodified lines
6	#include <unistd.h>	6	#include <unistd.h>
7	#include <stdlib.h>	7	#include <stdlib.h>
8	#include <signal.h>	8	#include <signal.h>
9	#include <linux/bpf.h>	9	#include <bpf/libbpf.h>
10	#include <bpf/bpf.h>	10	#include <bpf/bpf.h>
11	#include "bpf_load.h"
12		11
13	#define MAX_ENTRIES 20	12	#define MAX_ENTRIES 20
14	#define MAX_CPU 4	13	#define MAX_CPU 4
@@ -81,20 +80,51 @@ static void get_data(int fd) · 66 unmodified lines
81		80
82	int main(int argc, char **argv)	81	int main(int argc, char **argv)
83	{	82	{
		83	struct bpf_link *links[2];
		84	struct bpf_program *prog;
		85	struct bpf_object *obj;
84	char filename[256];	86	char filename[256];
		87	int map_fd, i = 0;
85		88
86	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);	89	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
		90	obj = bpf_object__open_file(filename, NULL);
		91	if (libbpf_get_error(obj)) {
		92	fprintf(stderr, "ERROR: opening BPF object file failed\n");
		93	return 0;
		94	}
		95
		96	/* load BPF program */
		97	if (bpf_object__load(obj)) {
		98	fprintf(stderr, "ERROR: loading BPF object file failed\n");
		99	goto cleanup;
		100	}
87		101
88	if (load_bpf_file(filename)) {	102	map_fd = bpf_object__find_map_fd_by_name(obj, "my_lat");
89	printf("%s", bpf_log_buf);	103	if (map_fd < 0) {
90	return 1;	104	fprintf(stderr, "ERROR: finding a map in obj file failed\n");
		105	goto cleanup;
		106	}
		107
		108	bpf_object__for_each_program(prog, obj) {
		109	links[i] = bpf_program__attach(prog);
		110	if (libbpf_get_error(links[i])) {
		111	fprintf(stderr, "ERROR: bpf_program__attach failed\n");
		112	links[i] = NULL;
		113	goto cleanup;
		114	}
		115	i++;
91	}	116	}
92		117
93	while (1) {	118	while (1) {
94	get_data(map_fd[1]);	119	get_data(map_fd);
95	print_hist();	120	print_hist();
96	sleep(5);	121	sleep(5);
97	}	122	}
98		123
		124	cleanup:
		125	for (i--; i >= 0; i--)
		126	bpf_link__destroy(links[i]);
		127
		128	bpf_object__close(obj);
99	return 0;	129	return 0;
100	}	130	}

samples/bpf/offwaketime_kern.c+26−26

@@ -28,38 +28,38 @@ struct key_t { · 27 unmodified lines
28	u32 tret;	28	u32 tret;
29	};	29	};
30		30
31	struct bpf_map_def SEC("maps") counts = {	31	struct {
32	.type = BPF_MAP_TYPE_HASH,	32	__uint(type, BPF_MAP_TYPE_HASH);
33	.key_size = sizeof(struct key_t),	33	__type(key, struct key_t);
34	.value_size = sizeof(u64),	34	__type(value, u64);
35	.max_entries = 10000,	35	__uint(max_entries, 10000);
36	};	36	} counts SEC(".maps");
37		37
38	struct bpf_map_def SEC("maps") start = {	38	struct {
39	.type = BPF_MAP_TYPE_HASH,	39	__uint(type, BPF_MAP_TYPE_HASH);
40	.key_size = sizeof(u32),	40	__type(key, u32);
41	.value_size = sizeof(u64),	41	__type(value, u64);
42	.max_entries = 10000,	42	__uint(max_entries, 10000);
43	};	43	} start SEC(".maps");
44		44
45	struct wokeby_t {	45	struct wokeby_t {
46	char name[TASK_COMM_LEN];	46	char name[TASK_COMM_LEN];
47	u32 ret;	47	u32 ret;
48	};	48	};
49		49
50	struct bpf_map_def SEC("maps") wokeby = {	50	struct {
51	.type = BPF_MAP_TYPE_HASH,	51	__uint(type, BPF_MAP_TYPE_HASH);
52	.key_size = sizeof(u32),	52	__type(key, u32);
53	.value_size = sizeof(struct wokeby_t),	53	__type(value, struct wokeby_t);
54	.max_entries = 10000,	54	__uint(max_entries, 10000);
55	};	55	} wokeby SEC(".maps");
56		56
57	struct bpf_map_def SEC("maps") stackmap = {	57	struct {
58	.type = BPF_MAP_TYPE_STACK_TRACE,	58	__uint(type, BPF_MAP_TYPE_STACK_TRACE);
59	.key_size = sizeof(u32),	59	__uint(key_size, sizeof(u32));
60	.value_size = PERF_MAX_STACK_DEPTH * sizeof(u64),	60	__uint(value_size, PERF_MAX_STACK_DEPTH * sizeof(u64));
61	.max_entries = 10000,	61	__uint(max_entries, 10000);
62	};	62	} stackmap SEC(".maps");
63		63
64	#define STACKID_FLAGS (0 \| BPF_F_FAST_STACK_CMP)	64	#define STACKID_FLAGS (0 \| BPF_F_FAST_STACK_CMP)
65		65

samples/bpf/offwaketime_user.c+51−15

 #include <unistd.h>
 #include <stdlib.h>
 #include <signal.h>
-#include <linux/bpf.h>
-#include <string.h>
 #include <linux/perf_event.h>
 #include <errno.h>
-#include <assert.h>
 #include <stdbool.h>
 #include <sys/resource.h>
 #include <bpf/libbpf.h>
-#include "bpf_load.h"
+#include <bpf/bpf.h>
 #include "trace_helpers.h"
 #define PRINT_RAW_ADDR 0
+/* counts, stackmap */
+static int map_fd[2];
 static void print_ksym(__u64 addr)
 {
 	struct ksym *sym;
 	int i;
 	printf("%s;", key->target);
-	if (bpf_map_lookup_elem(map_fd[3], &key->tret, ip) != 0) {
+	if (bpf_map_lookup_elem(map_fd[1], &key->tret, ip) != 0) {
 		printf("---;");
 	} else {
 		for (i = PERF_MAX_STACK_DEPTH - 1; i >= 0; i--)
 			print_ksym(ip[i]);
 	}
 	printf("-;");
-	if (bpf_map_lookup_elem(map_fd[3], &key->wret, ip) != 0) {
+	if (bpf_map_lookup_elem(map_fd[1], &key->wret, ip) != 0) {
 		printf("---;");
 	} else {
 		for (i = 0; i < PERF_MAX_STACK_DEPTH; i++)
 int main(int argc, char **argv)
 {
 	struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
+	struct bpf_object *obj = NULL;
+	struct bpf_link *links[2];
+	struct bpf_program *prog;
+	int delay = 1, i = 0;
 	char filename[256];
-	int delay = 1;
-	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
-	setrlimit(RLIMIT_MEMLOCK, &r);
-	signal(SIGINT, int_exit);
+	if (setrlimit(RLIMIT_MEMLOCK, &r)) {
-	signal(SIGTERM, int_exit);
+		perror("setrlimit(RLIMIT_MEMLOCK)");
+		return 1;
+	}
 	if (load_kallsyms()) {
 		printf("failed to process /proc/kallsyms\n");
 		return 2;
 	}
-	if (load_bpf_file(filename)) {
+	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
-		printf("%s", bpf_log_buf);
+	obj = bpf_object__open_file(filename, NULL);
-		return 1;
+	if (libbpf_get_error(obj)) {
+		fprintf(stderr, "ERROR: opening BPF object file failed\n");
+		obj = NULL;
+		goto cleanup;
+	}
+	/* load BPF program */
+	if (bpf_object__load(obj)) {
+		fprintf(stderr, "ERROR: loading BPF object file failed\n");
+		goto cleanup;
+	}
+	map_fd[0] = bpf_object__find_map_fd_by_name(obj, "counts");
+	map_fd[1] = bpf_object__find_map_fd_by_name(obj, "stackmap");
+	if (map_fd[0] < 0 || map_fd[1] < 0) {
+		fprintf(stderr, "ERROR: finding a map in obj file failed\n");
+		goto cleanup;
+	}
+	signal(SIGINT, int_exit);
+	signal(SIGTERM, int_exit);
+	bpf_object__for_each_program(prog, obj) {
+		links[i] = bpf_program__attach(prog);
+		if (libbpf_get_error(links[i])) {
+			fprintf(stderr, "ERROR: bpf_program__attach failed\n");
+			links[i] = NULL;
+			goto cleanup;
+		}
+		i++;
 	}
 	if (argc > 1)
 	sleep(delay);
 	print_stacks(map_fd[0]);
+cleanup:
+	for (i--; i >= 0; i--)
+		bpf_link__destroy(links[i]);
+	bpf_object__close(obj);
 	return 0;
 }

samples/bpf/spintest_kern.c+18−18

@@ -12,25 +12,25 @@ · 11 unmodified lines
12	#include <bpf/bpf_helpers.h>	12	#include <bpf/bpf_helpers.h>
13	#include <bpf/bpf_tracing.h>	13	#include <bpf/bpf_tracing.h>
14		14
15	struct bpf_map_def SEC("maps") my_map = {	15	struct {
16	.type = BPF_MAP_TYPE_HASH,	16	__uint(type, BPF_MAP_TYPE_HASH);
17	.key_size = sizeof(long),	17	__type(key, long);
18	.value_size = sizeof(long),	18	__type(value, long);
19	.max_entries = 1024,	19	__uint(max_entries, 1024);
20	};	20	} my_map SEC(".maps");
21	struct bpf_map_def SEC("maps") my_map2 = {	21	struct {
22	.type = BPF_MAP_TYPE_PERCPU_HASH,	22	__uint(type, BPF_MAP_TYPE_PERCPU_HASH);
23	.key_size = sizeof(long),	23	__uint(key_size, sizeof(long));
24	.value_size = sizeof(long),	24	__uint(value_size, sizeof(long));
25	.max_entries = 1024,	25	__uint(max_entries, 1024);
26	};	26	} my_map2 SEC(".maps");
27		27
28	struct bpf_map_def SEC("maps") stackmap = {	28	struct {
29	.type = BPF_MAP_TYPE_STACK_TRACE,	29	__uint(type, BPF_MAP_TYPE_STACK_TRACE);
30	.key_size = sizeof(u32),	30	__uint(key_size, sizeof(u32));
31	.value_size = PERF_MAX_STACK_DEPTH * sizeof(u64),	31	__uint(value_size, PERF_MAX_STACK_DEPTH * sizeof(u64));
32	.max_entries = 10000,	32	__uint(max_entries, 10000);
33	};	33	} stackmap SEC(".maps");
34		34
35	#define PROG(foo) \	35	#define PROG(foo) \
36	int foo(struct pt_regs *ctx) \	36	int foo(struct pt_regs *ctx) \

samples/bpf/spintest_user.c+55−13

 // SPDX-License-Identifier: GPL-2.0
 #include <stdio.h>
 #include <unistd.h>
-#include <linux/bpf.h>
 #include <string.h>
 #include <assert.h>
 #include <sys/resource.h>
 #include <bpf/libbpf.h>
-#include "bpf_load.h"
+#include <bpf/bpf.h>
 #include "trace_helpers.h"
 int main(int ac, char **argv)
 {
 	struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
+	char filename[256], symbol[256];
+	struct bpf_object *obj = NULL;
+	struct bpf_link *links[20];
 	long key, next_key, value;
-	char filename[256];
+	struct bpf_program *prog;
+	int map_fd, i, j = 0;
+	const char *title;
 	struct ksym *sym;
-	int i;
-	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+	if (setrlimit(RLIMIT_MEMLOCK, &r)) {
-	setrlimit(RLIMIT_MEMLOCK, &r);
+		perror("setrlimit(RLIMIT_MEMLOCK)");
+		return 1;
+	}
 	if (load_kallsyms()) {
 		printf("failed to process /proc/kallsyms\n");
 		return 2;
 	}
-	if (load_bpf_file(filename)) {
+	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
-		printf("%s", bpf_log_buf);
+	obj = bpf_object__open_file(filename, NULL);
-		return 1;
+	if (libbpf_get_error(obj)) {
+		fprintf(stderr, "ERROR: opening BPF object file failed\n");
+		obj = NULL;
+		goto cleanup;
+	}
+	/* load BPF program */
+	if (bpf_object__load(obj)) {
+		fprintf(stderr, "ERROR: loading BPF object file failed\n");
+		goto cleanup;
+	}
+	map_fd = bpf_object__find_map_fd_by_name(obj, "my_map");
+	if (map_fd < 0) {
+		fprintf(stderr, "ERROR: finding a map in obj file failed\n");
+		goto cleanup;
+	}
+	bpf_object__for_each_program(prog, obj) {
+		title = bpf_program__title(prog, false);
+		if (sscanf(title, "kprobe/%s", symbol) != 1)
+			continue;
+		/* Attach prog only when symbol exists */
+		if (ksym_get_addr(symbol)) {
+			links[j] = bpf_program__attach(prog);
+			if (libbpf_get_error(links[j])) {
+				fprintf(stderr, "bpf_program__attach failed\n");
+				links[j] = NULL;
+				goto cleanup;
+			}
+			j++;
+		}
 	}
 	for (i = 0; i < 5; i++) {
 		key = 0;
 		printf("kprobing funcs:");
-		while (bpf_map_get_next_key(map_fd[0], &key, &next_key) == 0) {
+		while (bpf_map_get_next_key(map_fd, &key, &next_key) == 0) {
-			bpf_map_lookup_elem(map_fd[0], &next_key, &value);
+			bpf_map_lookup_elem(map_fd, &next_key, &value);
 			assert(next_key == value);
 			sym = ksym_search(value);
 			key = next_key;
 		if (key)
 			printf("\n");
 		key = 0;
-		while (bpf_map_get_next_key(map_fd[0], &key, &next_key) == 0)
+		while (bpf_map_get_next_key(map_fd, &key, &next_key) == 0)
-			bpf_map_delete_elem(map_fd[0], &next_key);
+			bpf_map_delete_elem(map_fd, &next_key);
 		sleep(1);
 	}
+cleanup:
+	for (j--; j >= 0; j--)
+		bpf_link__destroy(links[j]);
+	bpf_object__close(obj);
 	return 0;
 }

samples/bpf/syscall_tp_kern.c+12−12

@@ -18,19 +18,19 @@ struct syscalls_exit_open_args { · 17 unmodified lines
18	long ret;	18	long ret;
19	};	19	};
20		20
21	struct bpf_map_def SEC("maps") enter_open_map = {	21	struct {
22	.type = BPF_MAP_TYPE_ARRAY,	22	__uint(type, BPF_MAP_TYPE_ARRAY);
23	.key_size = sizeof(u32),	23	__type(key, u32);
24	.value_size = sizeof(u32),	24	__type(value, u32);
25	.max_entries = 1,	25	__uint(max_entries, 1);
26	};	26	} enter_open_map SEC(".maps");
27		27
28	struct bpf_map_def SEC("maps") exit_open_map = {	28	struct {
29	.type = BPF_MAP_TYPE_ARRAY,	29	__uint(type, BPF_MAP_TYPE_ARRAY);
30	.key_size = sizeof(u32),	30	__type(key, u32);
31	.value_size = sizeof(u32),	31	__type(value, u32);
32	.max_entries = 1,	32	__uint(max_entries, 1);
33	};	33	} exit_open_map SEC(".maps");
34		34
35	static __always_inline void count(void *map)	35	static __always_inline void count(void *map)
36	{	36	{

samples/bpf/syscall_tp_user.c+42−12

@@ -5,16 +5,12 @@ · 4 unmodified lines
5	#include <unistd.h>	5	#include <unistd.h>
6	#include <fcntl.h>	6	#include <fcntl.h>
7	#include <stdlib.h>	7	#include <stdlib.h>
8	#include <signal.h>
9	#include <linux/bpf.h>
10	#include <string.h>	8	#include <string.h>
11	#include <linux/perf_event.h>	9	#include <linux/perf_event.h>
12	#include <errno.h>	10	#include <errno.h>
13	#include <assert.h>
14	#include <stdbool.h>
15	#include <sys/resource.h>	11	#include <sys/resource.h>
		12	#include <bpf/libbpf.h>
16	#include <bpf/bpf.h>	13	#include <bpf/bpf.h>
17	#include "bpf_load.h"
18		14
19	/* This program verifies bpf attachment to tracepoint sys_enter_* and sys_exit_*.	15	/* This program verifies bpf attachment to tracepoint sys_enter_* and sys_exit_*.
20	* This requires kernel CONFIG_FTRACE_SYSCALLS to be set.	16	* This requires kernel CONFIG_FTRACE_SYSCALLS to be set.
@@ -49,16 +45,44 @@ static void verify_map(int map_id) · 28 unmodified lines
49		45
50	static int test(char *filename, int num_progs)	46	static int test(char *filename, int num_progs)
51	{	47	{
52	int i, fd, map0_fds[num_progs], map1_fds[num_progs];	48	int map0_fds[num_progs], map1_fds[num_progs], fd, i, j = 0;
		49	struct bpf_link links[num_progs 4];
		50	struct bpf_object *objs[num_progs];
		51	struct bpf_program *prog;
53		52
54	for (i = 0; i < num_progs; i++) {	53	for (i = 0; i < num_progs; i++) {
55	if (load_bpf_file(filename)) {	54	objs[i] = bpf_object__open_file(filename, NULL);
56	fprintf(stderr, "%s", bpf_log_buf);	55	if (libbpf_get_error(objs[i])) {
57	return 1;	56	fprintf(stderr, "opening BPF object file failed\n");
		57	objs[i] = NULL;
		58	goto cleanup;
58	}	59	}
59	printf("prog #%d: map ids %d %d\n", i, map_fd[0], map_fd[1]);	60
60	map0_fds[i] = map_fd[0];	61	/* load BPF program */
61	map1_fds[i] = map_fd[1];	62	if (bpf_object__load(objs[i])) {
		63	fprintf(stderr, "loading BPF object file failed\n");
		64	goto cleanup;
		65	}
		66
		67	map0_fds[i] = bpf_object__find_map_fd_by_name(objs[i],
		68	"enter_open_map");
		69	map1_fds[i] = bpf_object__find_map_fd_by_name(objs[i],
		70	"exit_open_map");
		71	if (map0_fds[i] < 0 \|\| map1_fds[i] < 0) {
		72	fprintf(stderr, "finding a map in obj file failed\n");
		73	goto cleanup;
		74	}
		75
		76	bpf_object__for_each_program(prog, objs[i]) {
		77	links[j] = bpf_program__attach(prog);
		78	if (libbpf_get_error(links[j])) {
		79	fprintf(stderr, "bpf_program__attach failed\n");
		80	links[j] = NULL;
		81	goto cleanup;
		82	}
		83	j++;
		84	}
		85	printf("prog #%d: map ids %d %d\n", i, map0_fds[i], map1_fds[i]);
62	}	86	}
63		87
64	/* current load_bpf_file has perf_event_open default pid = -1	88	/* current load_bpf_file has perf_event_open default pid = -1
@@ -80,6 +104,12 @@ static int test(char *filename, int num_progs) · 15 unmodified lines
80	verify_map(map1_fds[i]);	104	verify_map(map1_fds[i]);
81	}	105	}
82		106
		107	cleanup:
		108	for (j--; j >= 0; j--)
		109	bpf_link__destroy(links[j]);
		110
		111	for (i--; i >= 0; i--)
		112	bpf_object__close(objs[i]);
83	return 0;	113	return 0;
84	}	114	}
85		115

samples/bpf/task_fd_query_kern.c+1−1

@@ -10,7 +10,7 @@ int bpf_prog1(struct pt_regs *ctx) · 9 unmodified lines
10	return 0;	10	return 0;
11	}	11	}
12		12
13	SEC("kretprobe/blk_account_io_completion")	13	SEC("kretprobe/blk_account_io_done")
14	int bpf_prog2(struct pt_regs *ctx)	14	int bpf_prog2(struct pt_regs *ctx)
15	{	15	{
16	return 0;	16	return 0;

samples/bpf/task_fd_query_user.c+1−1

@@ -314,7 +314,7 @@ int main(int argc, char **argv) · 313 unmodified lines
314	/* test two functions in the corresponding _kern.c file /	314	/* test two functions in the corresponding _kern.c file /
315	CHECK_AND_RET(test_debug_fs_kprobe(0, "blk_mq_start_request",	315	CHECK_AND_RET(test_debug_fs_kprobe(0, "blk_mq_start_request",
316	BPF_FD_TYPE_KPROBE));	316	BPF_FD_TYPE_KPROBE));
317	CHECK_AND_RET(test_debug_fs_kprobe(1, "blk_account_io_completion",	317	CHECK_AND_RET(test_debug_fs_kprobe(1, "blk_account_io_done",
318	BPF_FD_TYPE_KRETPROBE));	318	BPF_FD_TYPE_KRETPROBE));
319		319
320	/* test nondebug fs kprobe */	320	/* test nondebug fs kprobe */

samples/bpf/test_current_task_under_cgroup_kern.c+14−13

@@ -10,23 +10,24 @@ · 9 unmodified lines
10	#include <linux/version.h>	10	#include <linux/version.h>
11	#include <bpf/bpf_helpers.h>	11	#include <bpf/bpf_helpers.h>
12	#include <uapi/linux/utsname.h>	12	#include <uapi/linux/utsname.h>
		13	#include "trace_common.h"
13		14
14	struct bpf_map_def SEC("maps") cgroup_map = {	15	struct {
15	.type = BPF_MAP_TYPE_CGROUP_ARRAY,	16	__uint(type, BPF_MAP_TYPE_CGROUP_ARRAY);
16	.key_size = sizeof(u32),	17	__uint(key_size, sizeof(u32));
17	.value_size = sizeof(u32),	18	__uint(value_size, sizeof(u32));
18	.max_entries = 1,	19	__uint(max_entries, 1);
19	};	20	} cgroup_map SEC(".maps");
20		21
21	struct bpf_map_def SEC("maps") perf_map = {	22	struct {
22	.type = BPF_MAP_TYPE_ARRAY,	23	__uint(type, BPF_MAP_TYPE_ARRAY);
23	.key_size = sizeof(u32),	24	__type(key, u32);
24	.value_size = sizeof(u64),	25	__type(value, u64);
25	.max_entries = 1,	26	__uint(max_entries, 1);
26	};	27	} perf_map SEC(".maps");
27		28
28	/* Writes the last PID that called sync to a map at index 0 */	29	/* Writes the last PID that called sync to a map at index 0 */
29	SEC("kprobe/sys_sync")	30	SEC("kprobe/" SYSCALL(sys_sync))
30	int bpf_prog1(struct pt_regs *ctx)	31	int bpf_prog1(struct pt_regs *ctx)
31	{	32	{
32	u64 pid = bpf_get_current_pid_tgid();	33	u64 pid = bpf_get_current_pid_tgid();

samples/bpf/test_current_task_under_cgroup_user.c+42−10

 #define _GNU_SOURCE
 #include <stdio.h>
-#include <linux/bpf.h>
 #include <unistd.h>
 #include <bpf/bpf.h>
-#include "bpf_load.h"
+#include <bpf/libbpf.h>
 #include "cgroup_helpers.h"
 #define CGROUP_PATH		"/my-cgroup"
 int main(int argc, char **argv)
 {
 	pid_t remote_pid, local_pid = getpid();
-	int cg2, idx = 0, rc = 0;
+	struct bpf_link *link = NULL;
+	struct bpf_program *prog;
+	int cg2, idx = 0, rc = 1;
+	struct bpf_object *obj;
 	char filename[256];
+	int map_fd[2];
 	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
-	if (load_bpf_file(filename)) {
+	obj = bpf_object__open_file(filename, NULL);
-		printf("%s", bpf_log_buf);
+	if (libbpf_get_error(obj)) {
-		return 1;
+		fprintf(stderr, "ERROR: opening BPF object file failed\n");
+		return 0;
+	}
+	prog = bpf_object__find_program_by_name(obj, "bpf_prog1");
+	if (!prog) {
+		printf("finding a prog in obj file failed\n");
+		goto cleanup;
+	}
+	/* load BPF program */
+	if (bpf_object__load(obj)) {
+		fprintf(stderr, "ERROR: loading BPF object file failed\n");
+		goto cleanup;
+	}
+	map_fd[0] = bpf_object__find_map_fd_by_name(obj, "cgroup_map");
+	map_fd[1] = bpf_object__find_map_fd_by_name(obj, "perf_map");
+	if (map_fd[0] < 0 || map_fd[1] < 0) {
+		fprintf(stderr, "ERROR: finding a map in obj file failed\n");
+		goto cleanup;
+	}
+	link = bpf_program__attach(prog);
+	if (libbpf_get_error(link)) {
+		fprintf(stderr, "ERROR: bpf_program__attach failed\n");
+		link = NULL;
+		goto cleanup;
 	}
 	if (setup_cgroup_environment())
 		goto err;
 	}
-	goto out;
+	rc = 0;
-err:
-	rc = 1;
-out:
+err:
 	close(cg2);
 	cleanup_cgroup_environment();
+cleanup:
+	bpf_link__destroy(link);
+	bpf_object__close(obj);
 	return rc;
 }

samples/bpf/test_probe_write_user_kern.c+6−6

@@ -13,12 +13,12 @@ · 12 unmodified lines
13	#include <bpf/bpf_core_read.h>	13	#include <bpf/bpf_core_read.h>
14	#include "trace_common.h"	14	#include "trace_common.h"
15		15
16	struct bpf_map_def SEC("maps") dnat_map = {	16	struct {
17	.type = BPF_MAP_TYPE_HASH,	17	__uint(type, BPF_MAP_TYPE_HASH);
18	.key_size = sizeof(struct sockaddr_in),	18	__type(key, struct sockaddr_in);
19	.value_size = sizeof(struct sockaddr_in),	19	__type(value, struct sockaddr_in);
20	.max_entries = 256,	20	__uint(max_entries, 256);
21	};	21	} dnat_map SEC(".maps");
22		22
23	/* kprobe is NOT a stable ABI	23	/* kprobe is NOT a stable ABI
24	* kernel functions can be removed, renamed or completely change semantics.	24	* kernel functions can be removed, renamed or completely change semantics.

samples/bpf/test_probe_write_user_user.c+39−10

 // SPDX-License-Identifier: GPL-2.0
 #include <stdio.h>
 #include <assert.h>
-#include <linux/bpf.h>
 #include <unistd.h>
 #include <bpf/bpf.h>
-#include "bpf_load.h"
+#include <bpf/libbpf.h>
 #include <sys/socket.h>
-#include <string.h>
 #include <netinet/in.h>
 #include <arpa/inet.h>
 int main(int ac, char **argv)
 {
-	int serverfd, serverconnfd, clientfd;
-	socklen_t sockaddr_len;
-	struct sockaddr serv_addr, mapped_addr, tmp_addr;
 	struct sockaddr_in *serv_addr_in, *mapped_addr_in, *tmp_addr_in;
+	struct sockaddr serv_addr, mapped_addr, tmp_addr;
+	int serverfd, serverconnfd, clientfd, map_fd;
+	struct bpf_link *link = NULL;
+	struct bpf_program *prog;
+	struct bpf_object *obj;
+	socklen_t sockaddr_len;
 	char filename[256];
 	char *ip;
 	tmp_addr_in = (struct sockaddr_in *)&tmp_addr;
 	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+	obj = bpf_object__open_file(filename, NULL);
+	if (libbpf_get_error(obj)) {
+		fprintf(stderr, "ERROR: opening BPF object file failed\n");
+		return 0;
+	}
+	prog = bpf_object__find_program_by_name(obj, "bpf_prog1");
+	if (libbpf_get_error(prog)) {
+		fprintf(stderr, "ERROR: finding a prog in obj file failed\n");
+		goto cleanup;
+	}
+	/* load BPF program */
+	if (bpf_object__load(obj)) {
+		fprintf(stderr, "ERROR: loading BPF object file failed\n");
+		goto cleanup;
+	}
+	map_fd = bpf_object__find_map_fd_by_name(obj, "dnat_map");
+	if (map_fd < 0) {
+		fprintf(stderr, "ERROR: finding a map in obj file failed\n");
+		goto cleanup;
+	}
-	if (load_bpf_file(filename)) {
+	link = bpf_program__attach(prog);
-		printf("%s", bpf_log_buf);
+	if (libbpf_get_error(link)) {
-		return 1;
+		fprintf(stderr, "ERROR: bpf_program__attach failed\n");
+		link = NULL;
+		goto cleanup;
 	}
 	assert((serverfd = socket(AF_INET, SOCK_STREAM, 0)) > 0);
 	mapped_addr_in->sin_port = htons(5555);
 	mapped_addr_in->sin_addr.s_addr = inet_addr("255.255.255.255");
-	assert(!bpf_map_update_elem(map_fd[0], &mapped_addr, &serv_addr, BPF_ANY));
+	assert(!bpf_map_update_elem(map_fd, &mapped_addr, &serv_addr, BPF_ANY));
 	assert(listen(serverfd, 5) == 0);
 	/* Is the server's getsockname = the socket getpeername */
 	assert(memcmp(&serv_addr, &tmp_addr, sizeof(struct sockaddr_in)) == 0);
+cleanup:
+	bpf_link__destroy(link);
+	bpf_object__close(obj);
 	return 0;
 }

samples/bpf/trace_output_kern.c+8−7

@@ -2,15 +2,16 @@ · 1 unmodified line
2	#include <linux/version.h>	2	#include <linux/version.h>
3	#include <uapi/linux/bpf.h>	3	#include <uapi/linux/bpf.h>
4	#include <bpf/bpf_helpers.h>	4	#include <bpf/bpf_helpers.h>
		5	#include "trace_common.h"
5		6
6	struct bpf_map_def SEC("maps") my_map = {	7	struct {
7	.type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,	8	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
8	.key_size = sizeof(int),	9	__uint(key_size, sizeof(int));
9	.value_size = sizeof(u32),	10	__uint(value_size, sizeof(u32));
10	.max_entries = 2,	11	__uint(max_entries, 2);
11	};	12	} my_map SEC(".maps");
12		13
13	SEC("kprobe/sys_write")	14	SEC("kprobe/" SYSCALL(sys_write))
14	int bpf_prog1(struct pt_regs *ctx)	15	int bpf_prog1(struct pt_regs *ctx)
15	{	16	{
16	struct S {	17	struct S {

samples/bpf/trace_output_user.c+37−18

 // SPDX-License-Identifier: GPL-2.0-only
 #include <stdio.h>
-#include <unistd.h>
-#include <stdlib.h>
-#include <stdbool.h>
-#include <string.h>
 #include <fcntl.h>
 #include <poll.h>
-#include <linux/perf_event.h>
-#include <linux/bpf.h>
-#include <errno.h>
-#include <assert.h>
-#include <sys/syscall.h>
-#include <sys/ioctl.h>
-#include <sys/mman.h>
 #include <time.h>
 #include <signal.h>
 #include <bpf/libbpf.h>
-#include "bpf_load.h"
-#include "perf-sys.h"
 static __u64 time_get_ns(void)
 {
 int main(int argc, char **argv)
 {
 	struct perf_buffer_opts pb_opts = {};
+	struct bpf_link *link = NULL;
+	struct bpf_program *prog;
 	struct perf_buffer *pb;
+	struct bpf_object *obj;
+	int map_fd, ret = 0;
 	char filename[256];
 	FILE *f;
-	int ret;
 	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+	obj = bpf_object__open_file(filename, NULL);
+	if (libbpf_get_error(obj)) {
+		fprintf(stderr, "ERROR: opening BPF object file failed\n");
+		return 0;
+	}
-	if (load_bpf_file(filename)) {
+	/* load BPF program */
-		printf("%s", bpf_log_buf);
+	if (bpf_object__load(obj)) {
-		return 1;
+		fprintf(stderr, "ERROR: loading BPF object file failed\n");
+		goto cleanup;
+	}
+	map_fd = bpf_object__find_map_fd_by_name(obj, "my_map");
+	if (map_fd < 0) {
+		fprintf(stderr, "ERROR: finding a map in obj file failed\n");
+		goto cleanup;
+	}
+	prog = bpf_object__find_program_by_name(obj, "bpf_prog1");
+	if (libbpf_get_error(prog)) {
+		fprintf(stderr, "ERROR: finding a prog in obj file failed\n");
+		goto cleanup;
+	}
+	link = bpf_program__attach(prog);
+	if (libbpf_get_error(link)) {
+		fprintf(stderr, "ERROR: bpf_program__attach failed\n");
+		link = NULL;
+		goto cleanup;
 	}
 	pb_opts.sample_cb = print_bpf_output;
-	pb = perf_buffer__new(map_fd[0], 8, &pb_opts);
+	pb = perf_buffer__new(map_fd, 8, &pb_opts);
 	ret = libbpf_get_error(pb);
 	if (ret) {
 		printf("failed to setup perf_buffer: %d\n", ret);
 	while ((ret = perf_buffer__poll(pb, 1000)) >= 0 && cnt < MAX_CNT) {
 	}
 	kill(0, SIGINT);
+cleanup:
+	bpf_link__destroy(link);
+	bpf_object__close(obj);
 	return ret;
 }

samples/bpf/tracex3_kern.c+1−1

@@ -49,7 +49,7 @@ struct { · 48 unmodified lines
49	__uint(max_entries, SLOTS);	49	__uint(max_entries, SLOTS);
50	} lat_map SEC(".maps");	50	} lat_map SEC(".maps");
51		51
52	SEC("kprobe/blk_account_io_completion")	52	SEC("kprobe/blk_account_io_done")
53	int bpf_prog2(struct pt_regs *ctx)	53	int bpf_prog2(struct pt_regs *ctx)
54	{	54	{
55	long rq = PT_REGS_PARM1(ctx);	55	long rq = PT_REGS_PARM1(ctx);

samples/bpf/xdpsock_user.c+19−13

 {
 	struct xsk_umem_info *umem;
 	struct xsk_umem_config cfg = {
-		.fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS,
+		/* We recommend that you set the fill ring size >= HW RX ring size +
+		 * AF_XDP RX ring size. Make sure you fill up the fill ring
+		 * with buffers at regular intervals, and you will with this setting
+		 * avoid allocation failures in the driver. These are usually quite
+		 * expensive since drivers have not been written to assume that
+		 * allocation failures are common. For regular sockets, kernel
+		 * allocated memory is used that only runs out in OOM situations
+		 * that should be rare.
+		 */
+		.fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS * 2,
 		.comp_size = XSK_RING_CONS__DEFAULT_NUM_DESCS,
 		.frame_size = opt_xsk_frame_size,
 		.frame_headroom = XSK_UMEM__DEFAULT_FRAME_HEADROOM,
 	u32 idx;
 	ret = xsk_ring_prod__reserve(&umem->fq,
-				     XSK_RING_PROD__DEFAULT_NUM_DESCS, &idx);
+				     XSK_RING_PROD__DEFAULT_NUM_DESCS * 2, &idx);
-	if (ret != XSK_RING_PROD__DEFAULT_NUM_DESCS)
+	if (ret != XSK_RING_PROD__DEFAULT_NUM_DESCS * 2)
 		exit_with_error(-ret);
-	for (i = 0; i < XSK_RING_PROD__DEFAULT_NUM_DESCS; i++)
+	for (i = 0; i < XSK_RING_PROD__DEFAULT_NUM_DESCS * 2; i++)
 		*xsk_ring_prod__fill_addr(&umem->fq, idx++) =
 			i * opt_xsk_frame_size;
-	xsk_ring_prod__submit(&umem->fq, XSK_RING_PROD__DEFAULT_NUM_DESCS);
+	xsk_ring_prod__submit(&umem->fq, XSK_RING_PROD__DEFAULT_NUM_DESCS * 2);
 }
 static struct xsk_socket_info *xsk_configure_socket(struct xsk_umem_info *umem,
 	if (!xsk->outstanding_tx)
 		return;
-	if (!opt_need_wakeup || xsk_ring_prod__needs_wakeup(&xsk->tx))
-		kick_tx(xsk);
 	ndescs = (xsk->outstanding_tx > opt_batch_size) ? opt_batch_size :
 		xsk->outstanding_tx;
 	}
 }
-static void tx_only(struct xsk_socket_info *xsk, u32 frame_nb, int batch_size)
+static void tx_only(struct xsk_socket_info *xsk, u32 *frame_nb, int batch_size)
 {
 	u32 idx;
 	unsigned int i;
 	for (i = 0; i < batch_size; i++) {
 		struct xdp_desc *tx_desc = xsk_ring_prod__tx_desc(&xsk->tx,
 								  idx + i);
-		tx_desc->addr = (frame_nb + i) << XSK_UMEM__DEFAULT_FRAME_SHIFT;
+		tx_desc->addr = (*frame_nb + i) << XSK_UMEM__DEFAULT_FRAME_SHIFT;
 		tx_desc->len = PKT_SIZE;
 	}
 	xsk_ring_prod__submit(&xsk->tx, batch_size);
 	xsk->outstanding_tx += batch_size;
-	frame_nb += batch_size;
+	*frame_nb += batch_size;
-	frame_nb %= NUM_FRAMES;
+	*frame_nb %= NUM_FRAMES;
 	complete_tx_only(xsk, batch_size);
 }
 		}
 		for (i = 0; i < num_socks; i++)
-			tx_only(xsks[i], frame_nb[i], batch_size);
+			tx_only(xsks[i], &frame_nb[i], batch_size);
 		pkt_cnt += batch_size;

samples/bpf/xsk_fwd.c+1085−0

@@ -0,0 +1,1085 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	/* Copyright(c) 2020 Intel Corporation. */
	3
	4	#define _GNU_SOURCE
	5	#include <poll.h>
	6	#include <pthread.h>
	7	#include <signal.h>
	8	#include <sched.h>
	9	#include <stdio.h>
	10	#include <stdlib.h>
	11	#include <string.h>
	12	#include <sys/mman.h>
	13	#include <sys/resource.h>
	14	#include <sys/socket.h>
	15	#include <sys/types.h>
	16	#include <time.h>
	17	#include <unistd.h>
	18	#include <getopt.h>
	19	#include <netinet/ether.h>
	20	#include <net/if.h>
	21
	22	#include <linux/bpf.h>
	23	#include <linux/if_link.h>
	24	#include <linux/if_xdp.h>
	25
	26	#include <bpf/libbpf.h>
	27	#include <bpf/xsk.h>
	28	#include <bpf/bpf.h>
	29
	30	#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
	31
	32	typedef __u64 u64;
	33	typedef __u32 u32;
	34	typedef __u16 u16;
	35	typedef __u8 u8;
	36
	37	/* This program illustrates the packet forwarding between multiple AF_XDP
	38	* sockets in multi-threaded environment. All threads are sharing a common
	39	* buffer pool, with each socket having its own private buffer cache.
	40	*
	41	* Example 1: Single thread handling two sockets. The packets received by socket
	42	* A (interface IFA, queue QA) are forwarded to socket B (interface IFB, queue
	43	* QB), while the packets received by socket B are forwarded to socket A. The
	44	* thread is running on CPU core X:
	45	*
	46	* ./xsk_fwd -i IFA -q QA -i IFB -q QB -c X
	47	*
	48	* Example 2: Two threads, each handling two sockets. The thread running on CPU
	49	* core X forwards all the packets received by socket A to socket B, and all the
	50	* packets received by socket B to socket A. The thread running on CPU core Y is
	51	* performing the same packet forwarding between sockets C and D:
	52	*
	53	* ./xsk_fwd -i IFA -q QA -i IFB -q QB -i IFC -q QC -i IFD -q QD
	54	* -c CX -c CY
	55	*/
	56
	57	/*
	58	* Buffer pool and buffer cache
	59	*
	60	* For packet forwarding, the packet buffers are typically allocated from the
	61	* pool for packet reception and freed back to the pool for further reuse once
	62	* the packet transmission is completed.
	63	*
	64	* The buffer pool is shared between multiple threads. In order to minimize the
	65	* access latency to the shared buffer pool, each thread creates one (or
	66	* several) buffer caches, which, unlike the buffer pool, are private to the
	67	* thread that creates them and therefore cannot be shared with other threads.
	68	* The access to the shared pool is only needed either (A) when the cache gets
	69	* empty due to repeated buffer allocations and it needs to be replenished from
	70	* the pool, or (B) when the cache gets full due to repeated buffer free and it
	71	* needs to be flushed back to the pull.
	72	*
	73	* In a packet forwarding system, a packet received on any input port can
	74	* potentially be transmitted on any output port, depending on the forwarding
	75	* configuration. For AF_XDP sockets, for this to work with zero-copy of the
	76	* packet buffers when, it is required that the buffer pool memory fits into the
	77	* UMEM area shared by all the sockets.
	78	*/
	79
	80	struct bpool_params {
	81	u32 n_buffers;
	82	u32 buffer_size;
	83	int mmap_flags;
	84
	85	u32 n_users_max;
	86	u32 n_buffers_per_slab;
	87	};
	88
	89	/* This buffer pool implementation organizes the buffers into equally sized
	90	* slabs of n_buffers_per_slab. Initially, there are n_slabs slabs in the
	91	* pool that are completely filled with buffer pointers (full slabs).
	92	*
	93	* Each buffer cache has a slab for buffer allocation and a slab for buffer
	94	* free, with both of these slabs initially empty. When the cache's allocation
	95	* slab goes empty, it is swapped with one of the available full slabs from the
	96	* pool, if any is available. When the cache's free slab goes full, it is
	97	* swapped for one of the empty slabs from the pool, which is guaranteed to
	98	* succeed.
	99	*
	100	* Partially filled slabs never get traded between the cache and the pool
	101	* (except when the cache itself is destroyed), which enables fast operation
	102	* through pointer swapping.
	103	*/
	104	struct bpool {
	105	struct bpool_params params;
	106	pthread_mutex_t lock;
	107	void *addr;
	108
	109	u64 **slabs;
	110	u64 **slabs_reserved;
	111	u64 *buffers;
	112	u64 *buffers_reserved;
	113
	114	u64 n_slabs;
	115	u64 n_slabs_reserved;
	116	u64 n_buffers;
	117
	118	u64 n_slabs_available;
	119	u64 n_slabs_reserved_available;
	120
	121	struct xsk_umem_config umem_cfg;
	122	struct xsk_ring_prod umem_fq;
	123	struct xsk_ring_cons umem_cq;
	124	struct xsk_umem *umem;
	125	};
	126
	127	static struct bpool *
	128	bpool_init(struct bpool_params *params,
	129	struct xsk_umem_config *umem_cfg)
	130	{
	131	struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
	132	u64 n_slabs, n_slabs_reserved, n_buffers, n_buffers_reserved;
	133	u64 slabs_size, slabs_reserved_size;
	134	u64 buffers_size, buffers_reserved_size;
	135	u64 total_size, i;
	136	struct bpool *bp;
	137	u8 *p;
	138	int status;
	139
	140	/* mmap prep. */
	141	if (setrlimit(RLIMIT_MEMLOCK, &r))
	142	return NULL;
	143
	144	/* bpool internals dimensioning. */
	145	n_slabs = (params->n_buffers + params->n_buffers_per_slab - 1) /
	146	params->n_buffers_per_slab;
	147	n_slabs_reserved = params->n_users_max * 2;
	148	n_buffers = n_slabs * params->n_buffers_per_slab;
	149	n_buffers_reserved = n_slabs_reserved * params->n_buffers_per_slab;
	150
	151	slabs_size = n_slabs * sizeof(u64 *);
	152	slabs_reserved_size = n_slabs_reserved * sizeof(u64 *);
	153	buffers_size = n_buffers * sizeof(u64);
	154	buffers_reserved_size = n_buffers_reserved * sizeof(u64);
	155
	156	total_size = sizeof(struct bpool) +
	157	slabs_size + slabs_reserved_size +
	158	buffers_size + buffers_reserved_size;
	159
	160	/* bpool memory allocation. */
	161	p = calloc(total_size, sizeof(u8));
	162	if (!p)
	163	return NULL;
	164
	165	/* bpool memory initialization. */
	166	bp = (struct bpool *)p;
	167	memcpy(&bp->params, params, sizeof(*params));
	168	bp->params.n_buffers = n_buffers;
	169
	170	bp->slabs = (u64 **)&p[sizeof(struct bpool)];
	171	bp->slabs_reserved = (u64 **)&p[sizeof(struct bpool) +
	172	slabs_size];
	173	bp->buffers = (u64 *)&p[sizeof(struct bpool) +
	174	slabs_size + slabs_reserved_size];
	175	bp->buffers_reserved = (u64 *)&p[sizeof(struct bpool) +
	176	slabs_size + slabs_reserved_size + buffers_size];
	177
	178	bp->n_slabs = n_slabs;
	179	bp->n_slabs_reserved = n_slabs_reserved;
	180	bp->n_buffers = n_buffers;
	181
	182	for (i = 0; i < n_slabs; i++)
	183	bp->slabs[i] = &bp->buffers[i * params->n_buffers_per_slab];
	184	bp->n_slabs_available = n_slabs;
	185
	186	for (i = 0; i < n_slabs_reserved; i++)
	187	bp->slabs_reserved[i] = &bp->buffers_reserved[i *
	188	params->n_buffers_per_slab];
	189	bp->n_slabs_reserved_available = n_slabs_reserved;
	190
	191	for (i = 0; i < n_buffers; i++)
	192	bp->buffers[i] = i * params->buffer_size;
	193
	194	/* lock. */
	195	status = pthread_mutex_init(&bp->lock, NULL);
	196	if (status) {
	197	free(p);
	198	return NULL;
	199	}
	200
	201	/* mmap. */
	202	bp->addr = mmap(NULL,
	203	n_buffers * params->buffer_size,
	204	PROT_READ \| PROT_WRITE,
	205	MAP_PRIVATE \| MAP_ANONYMOUS \| params->mmap_flags,
	206	-1,
	207	0);
	208	if (bp->addr == MAP_FAILED) {
	209	pthread_mutex_destroy(&bp->lock);
	210	free(p);
	211	return NULL;
	212	}
	213
	214	/* umem. */
	215	status = xsk_umem__create(&bp->umem,
	216	bp->addr,
	217	bp->params.n_buffers * bp->params.buffer_size,
	218	&bp->umem_fq,
	219	&bp->umem_cq,
	220	umem_cfg);
	221	if (status) {
	222	munmap(bp->addr, bp->params.n_buffers * bp->params.buffer_size);
	223	pthread_mutex_destroy(&bp->lock);
	224	free(p);
	225	return NULL;
	226	}
	227	memcpy(&bp->umem_cfg, umem_cfg, sizeof(*umem_cfg));
	228
	229	return bp;
	230	}
	231
	232	static void
	233	bpool_free(struct bpool *bp)
	234	{
	235	if (!bp)
	236	return;
	237
	238	xsk_umem__delete(bp->umem);
	239	munmap(bp->addr, bp->params.n_buffers * bp->params.buffer_size);
	240	pthread_mutex_destroy(&bp->lock);
	241	free(bp);
	242	}
	243
	244	struct bcache {
	245	struct bpool *bp;
	246
	247	u64 *slab_cons;
	248	u64 *slab_prod;
	249
	250	u64 n_buffers_cons;
	251	u64 n_buffers_prod;
	252	};
	253
	254	static u32
	255	bcache_slab_size(struct bcache *bc)
	256	{
	257	struct bpool *bp = bc->bp;
	258
	259	return bp->params.n_buffers_per_slab;
	260	}
	261
	262	static struct bcache *
	263	bcache_init(struct bpool *bp)
	264	{
	265	struct bcache *bc;
	266
	267	bc = calloc(1, sizeof(struct bcache));
	268	if (!bc)
	269	return NULL;
	270
	271	bc->bp = bp;
	272	bc->n_buffers_cons = 0;
	273	bc->n_buffers_prod = 0;
	274
	275	pthread_mutex_lock(&bp->lock);
	276	if (bp->n_slabs_reserved_available == 0) {
	277	pthread_mutex_unlock(&bp->lock);
	278	free(bc);
	279	return NULL;
	280	}
	281
	282	bc->slab_cons = bp->slabs_reserved[bp->n_slabs_reserved_available - 1];
	283	bc->slab_prod = bp->slabs_reserved[bp->n_slabs_reserved_available - 2];
	284	bp->n_slabs_reserved_available -= 2;
	285	pthread_mutex_unlock(&bp->lock);
	286
	287	return bc;
	288	}
	289
	290	static void
	291	bcache_free(struct bcache *bc)
	292	{
	293	struct bpool *bp;
	294
	295	if (!bc)
	296	return;
	297
	298	/* In order to keep this example simple, the case of freeing any
	299	* existing buffers from the cache back to the pool is ignored.
	300	*/
	301
	302	bp = bc->bp;
	303	pthread_mutex_lock(&bp->lock);
	304	bp->slabs_reserved[bp->n_slabs_reserved_available] = bc->slab_prod;
	305	bp->slabs_reserved[bp->n_slabs_reserved_available + 1] = bc->slab_cons;
	306	bp->n_slabs_reserved_available += 2;
	307	pthread_mutex_unlock(&bp->lock);
	308
	309	free(bc);
	310	}
	311
	312	/* To work correctly, the implementation requires that the n_buffers input
	313	* argument is never greater than the buffer pool's n_buffers_per_slab. This
	314	* is typically the case, with one exception taking place when large number of
	315	* buffers are allocated at init time (e.g. for the UMEM fill queue setup).
	316	*/
	317	static inline u32
	318	bcache_cons_check(struct bcache *bc, u32 n_buffers)
	319	{
	320	struct bpool *bp = bc->bp;
	321	u64 n_buffers_per_slab = bp->params.n_buffers_per_slab;
	322	u64 n_buffers_cons = bc->n_buffers_cons;
	323	u64 n_slabs_available;
	324	u64 *slab_full;
	325
	326	/*
	327	* Consumer slab is not empty: Use what's available locally. Do not
	328	* look for more buffers from the pool when the ask can only be
	329	* partially satisfied.
	330	*/
	331	if (n_buffers_cons)
	332	return (n_buffers_cons < n_buffers) ?
	333	n_buffers_cons :
	334	n_buffers;
	335
	336	/*
	337	* Consumer slab is empty: look to trade the current consumer slab
	338	* (full) for a full slab from the pool, if any is available.
	339	*/
	340	pthread_mutex_lock(&bp->lock);
	341	n_slabs_available = bp->n_slabs_available;
	342	if (!n_slabs_available) {
	343	pthread_mutex_unlock(&bp->lock);
	344	return 0;
	345	}
	346
	347	n_slabs_available--;
	348	slab_full = bp->slabs[n_slabs_available];
	349	bp->slabs[n_slabs_available] = bc->slab_cons;
	350	bp->n_slabs_available = n_slabs_available;
	351	pthread_mutex_unlock(&bp->lock);
	352
	353	bc->slab_cons = slab_full;
	354	bc->n_buffers_cons = n_buffers_per_slab;
	355	return n_buffers;
	356	}
	357
	358	static inline u64
	359	bcache_cons(struct bcache *bc)
	360	{
	361	u64 n_buffers_cons = bc->n_buffers_cons - 1;
	362	u64 buffer;
	363
	364	buffer = bc->slab_cons[n_buffers_cons];
	365	bc->n_buffers_cons = n_buffers_cons;
	366	return buffer;
	367	}
	368
	369	static inline void
	370	bcache_prod(struct bcache *bc, u64 buffer)
	371	{
	372	struct bpool *bp = bc->bp;
	373	u64 n_buffers_per_slab = bp->params.n_buffers_per_slab;
	374	u64 n_buffers_prod = bc->n_buffers_prod;
	375	u64 n_slabs_available;
	376	u64 *slab_empty;
	377
	378	/*
	379	* Producer slab is not yet full: store the current buffer to it.
	380	*/
	381	if (n_buffers_prod < n_buffers_per_slab) {
	382	bc->slab_prod[n_buffers_prod] = buffer;
	383	bc->n_buffers_prod = n_buffers_prod + 1;
	384	return;
	385	}
	386
	387	/*
	388	* Producer slab is full: trade the cache's current producer slab
	389	* (full) for an empty slab from the pool, then store the current
	390	* buffer to the new producer slab. As one full slab exists in the
	391	* cache, it is guaranteed that there is at least one empty slab
	392	* available in the pool.
	393	*/
	394	pthread_mutex_lock(&bp->lock);
	395	n_slabs_available = bp->n_slabs_available;
	396	slab_empty = bp->slabs[n_slabs_available];
	397	bp->slabs[n_slabs_available] = bc->slab_prod;
	398	bp->n_slabs_available = n_slabs_available + 1;
	399	pthread_mutex_unlock(&bp->lock);
	400
	401	slab_empty[0] = buffer;
	402	bc->slab_prod = slab_empty;
	403	bc->n_buffers_prod = 1;
	404	}
	405
	406	/*
	407	* Port
	408	*
	409	* Each of the forwarding ports sits on top of an AF_XDP socket. In order for
	410	* packet forwarding to happen with no packet buffer copy, all the sockets need
	411	* to share the same UMEM area, which is used as the buffer pool memory.
	412	*/
	413	#ifndef MAX_BURST_RX
	414	#define MAX_BURST_RX 64
	415	#endif
	416
	417	#ifndef MAX_BURST_TX
	418	#define MAX_BURST_TX 64
	419	#endif
	420
	421	struct burst_rx {
	422	u64 addr[MAX_BURST_RX];
	423	u32 len[MAX_BURST_RX];
	424	};
	425
	426	struct burst_tx {
	427	u64 addr[MAX_BURST_TX];
	428	u32 len[MAX_BURST_TX];
	429	u32 n_pkts;
	430	};
	431
	432	struct port_params {
	433	struct xsk_socket_config xsk_cfg;
	434	struct bpool *bp;
	435	const char *iface;
	436	u32 iface_queue;
	437	};
	438
	439	struct port {
	440	struct port_params params;
	441
	442	struct bcache *bc;
	443
	444	struct xsk_ring_cons rxq;
	445	struct xsk_ring_prod txq;
	446	struct xsk_ring_prod umem_fq;
	447	struct xsk_ring_cons umem_cq;
	448	struct xsk_socket *xsk;
	449	int umem_fq_initialized;
	450
	451	u64 n_pkts_rx;
	452	u64 n_pkts_tx;
	453	};
	454
	455	static void
	456	port_free(struct port *p)
	457	{
	458	if (!p)
	459	return;
	460
	461	/* To keep this example simple, the code to free the buffers from the
	462	* socket's receive and transmit queues, as well as from the UMEM fill
	463	* and completion queues, is not included.
	464	*/
	465
	466	if (p->xsk)
	467	xsk_socket__delete(p->xsk);
	468
	469	bcache_free(p->bc);
	470
	471	free(p);
	472	}
	473
	474	static struct port *
	475	port_init(struct port_params *params)
	476	{
	477	struct port *p;
	478	u32 umem_fq_size, pos = 0;
	479	int status, i;
	480
	481	/* Memory allocation and initialization. */
	482	p = calloc(sizeof(struct port), 1);
	483	if (!p)
	484	return NULL;
	485
	486	memcpy(&p->params, params, sizeof(p->params));
	487	umem_fq_size = params->bp->umem_cfg.fill_size;
	488
	489	/* bcache. */
	490	p->bc = bcache_init(params->bp);
	491	if (!p->bc \|\|
	492	(bcache_slab_size(p->bc) < umem_fq_size) \|\|
	493	(bcache_cons_check(p->bc, umem_fq_size) < umem_fq_size)) {
	494	port_free(p);
	495	return NULL;
	496	}
	497
	498	/* xsk socket. */
	499	status = xsk_socket__create_shared(&p->xsk,
	500	params->iface,
	501	params->iface_queue,
	502	params->bp->umem,
	503	&p->rxq,
	504	&p->txq,
	505	&p->umem_fq,
	506	&p->umem_cq,
	507	&params->xsk_cfg);
	508	if (status) {
	509	port_free(p);
	510	return NULL;
	511	}
	512
	513	/* umem fq. */
	514	xsk_ring_prod__reserve(&p->umem_fq, umem_fq_size, &pos);
	515
	516	for (i = 0; i < umem_fq_size; i++)
	517	*xsk_ring_prod__fill_addr(&p->umem_fq, pos + i) =
	518	bcache_cons(p->bc);
	519
	520	xsk_ring_prod__submit(&p->umem_fq, umem_fq_size);
	521	p->umem_fq_initialized = 1;
	522
	523	return p;
	524	}
	525
	526	static inline u32
	527	port_rx_burst(struct port p, struct burst_rx b)
	528	{
	529	u32 n_pkts, pos, i;
	530
	531	/* Free buffers for FQ replenish. */
	532	n_pkts = ARRAY_SIZE(b->addr);
	533
	534	n_pkts = bcache_cons_check(p->bc, n_pkts);
	535	if (!n_pkts)
	536	return 0;
	537
	538	/* RXQ. */
	539	n_pkts = xsk_ring_cons__peek(&p->rxq, n_pkts, &pos);
	540	if (!n_pkts) {
	541	if (xsk_ring_prod__needs_wakeup(&p->umem_fq)) {
	542	struct pollfd pollfd = {
	543	.fd = xsk_socket__fd(p->xsk),
	544	.events = POLLIN,
	545	};
	546
	547	poll(&pollfd, 1, 0);
	548	}
	549	return 0;
	550	}
	551
	552	for (i = 0; i < n_pkts; i++) {
	553	b->addr[i] = xsk_ring_cons__rx_desc(&p->rxq, pos + i)->addr;
	554	b->len[i] = xsk_ring_cons__rx_desc(&p->rxq, pos + i)->len;
	555	}
	556
	557	xsk_ring_cons__release(&p->rxq, n_pkts);
	558	p->n_pkts_rx += n_pkts;
	559
	560	/* UMEM FQ. */
	561	for ( ; ; ) {
	562	int status;
	563
	564	status = xsk_ring_prod__reserve(&p->umem_fq, n_pkts, &pos);
	565	if (status == n_pkts)
	566	break;
	567
	568	if (xsk_ring_prod__needs_wakeup(&p->umem_fq)) {
	569	struct pollfd pollfd = {
	570	.fd = xsk_socket__fd(p->xsk),
	571	.events = POLLIN,
	572	};
	573
	574	poll(&pollfd, 1, 0);
	575	}
	576	}
	577
	578	for (i = 0; i < n_pkts; i++)
	579	*xsk_ring_prod__fill_addr(&p->umem_fq, pos + i) =
	580	bcache_cons(p->bc);
	581
	582	xsk_ring_prod__submit(&p->umem_fq, n_pkts);
	583
	584	return n_pkts;
	585	}
	586
	587	static inline void
	588	port_tx_burst(struct port p, struct burst_tx b)
	589	{
	590	u32 n_pkts, pos, i;
	591	int status;
	592
	593	/* UMEM CQ. */
	594	n_pkts = p->params.bp->umem_cfg.comp_size;
	595
	596	n_pkts = xsk_ring_cons__peek(&p->umem_cq, n_pkts, &pos);
	597
	598	for (i = 0; i < n_pkts; i++) {
	599	u64 addr = *xsk_ring_cons__comp_addr(&p->umem_cq, pos + i);
	600
	601	bcache_prod(p->bc, addr);
	602	}
	603
	604	xsk_ring_cons__release(&p->umem_cq, n_pkts);
	605
	606	/* TXQ. */
	607	n_pkts = b->n_pkts;
	608
	609	for ( ; ; ) {
	610	status = xsk_ring_prod__reserve(&p->txq, n_pkts, &pos);
	611	if (status == n_pkts)
	612	break;
	613
	614	if (xsk_ring_prod__needs_wakeup(&p->txq))
	615	sendto(xsk_socket__fd(p->xsk), NULL, 0, MSG_DONTWAIT,
	616	NULL, 0);
	617	}
	618
	619	for (i = 0; i < n_pkts; i++) {
	620	xsk_ring_prod__tx_desc(&p->txq, pos + i)->addr = b->addr[i];
	621	xsk_ring_prod__tx_desc(&p->txq, pos + i)->len = b->len[i];
	622	}
	623
	624	xsk_ring_prod__submit(&p->txq, n_pkts);
	625	if (xsk_ring_prod__needs_wakeup(&p->txq))
	626	sendto(xsk_socket__fd(p->xsk), NULL, 0, MSG_DONTWAIT, NULL, 0);
	627	p->n_pkts_tx += n_pkts;
	628	}
	629
	630	/*
	631	* Thread
	632	*
	633	* Packet forwarding threads.
	634	*/
	635	#ifndef MAX_PORTS_PER_THREAD
	636	#define MAX_PORTS_PER_THREAD 16
	637	#endif
	638
	639	struct thread_data {
	640	struct port *ports_rx[MAX_PORTS_PER_THREAD];
	641	struct port *ports_tx[MAX_PORTS_PER_THREAD];
	642	u32 n_ports_rx;
	643	struct burst_rx burst_rx;
	644	struct burst_tx burst_tx[MAX_PORTS_PER_THREAD];
	645	u32 cpu_core_id;
	646	int quit;
	647	};
	648
	649	static void swap_mac_addresses(void *data)
	650	{
	651	struct ether_header eth = (struct ether_header )data;
	652	struct ether_addr src_addr = (struct ether_addr )&eth->ether_shost;
	653	struct ether_addr dst_addr = (struct ether_addr )&eth->ether_dhost;
	654	struct ether_addr tmp;
	655
	656	tmp = *src_addr;
	657	src_addr = dst_addr;
	658	*dst_addr = tmp;
	659	}
	660
	661	static void *
	662	thread_func(void *arg)
	663	{
	664	struct thread_data *t = arg;
	665	cpu_set_t cpu_cores;
	666	u32 i;
	667
	668	CPU_ZERO(&cpu_cores);
	669	CPU_SET(t->cpu_core_id, &cpu_cores);
	670	pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpu_cores);
	671
	672	for (i = 0; !t->quit; i = (i + 1) & (t->n_ports_rx - 1)) {
	673	struct port *port_rx = t->ports_rx[i];
	674	struct port *port_tx = t->ports_tx[i];
	675	struct burst_rx *brx = &t->burst_rx;
	676	struct burst_tx *btx = &t->burst_tx[i];
	677	u32 n_pkts, j;
	678
	679	/* RX. */
	680	n_pkts = port_rx_burst(port_rx, brx);
	681	if (!n_pkts)
	682	continue;
	683
	684	/* Process & TX. */
	685	for (j = 0; j < n_pkts; j++) {
	686	u64 addr = xsk_umem__add_offset_to_addr(brx->addr[j]);
	687	u8 *pkt = xsk_umem__get_data(port_rx->params.bp->addr,
	688	addr);
	689
	690	swap_mac_addresses(pkt);
	691
	692	btx->addr[btx->n_pkts] = brx->addr[j];
	693	btx->len[btx->n_pkts] = brx->len[j];
	694	btx->n_pkts++;
	695
	696	if (btx->n_pkts == MAX_BURST_TX) {
	697	port_tx_burst(port_tx, btx);
	698	btx->n_pkts = 0;
	699	}
	700	}
	701	}
	702
	703	return NULL;
	704	}
	705
	706	/*
	707	* Process
	708	*/
	709	static const struct bpool_params bpool_params_default = {
	710	.n_buffers = 64 * 1024,
	711	.buffer_size = XSK_UMEM__DEFAULT_FRAME_SIZE,
	712	.mmap_flags = 0,
	713
	714	.n_users_max = 16,
	715	.n_buffers_per_slab = XSK_RING_PROD__DEFAULT_NUM_DESCS * 2,
	716	};
	717
	718	static const struct xsk_umem_config umem_cfg_default = {
	719	.fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS * 2,
	720	.comp_size = XSK_RING_CONS__DEFAULT_NUM_DESCS,
	721	.frame_size = XSK_UMEM__DEFAULT_FRAME_SIZE,
	722	.frame_headroom = XSK_UMEM__DEFAULT_FRAME_HEADROOM,
	723	.flags = 0,
	724	};
	725
	726	static const struct port_params port_params_default = {
	727	.xsk_cfg = {
	728	.rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS,
	729	.tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS,
	730	.libbpf_flags = 0,
	731	.xdp_flags = XDP_FLAGS_DRV_MODE,
	732	.bind_flags = XDP_USE_NEED_WAKEUP \| XDP_ZEROCOPY,
	733	},
	734
	735	.bp = NULL,
	736	.iface = NULL,
	737	.iface_queue = 0,
	738	};
	739
	740	#ifndef MAX_PORTS
	741	#define MAX_PORTS 64
	742	#endif
	743
	744	#ifndef MAX_THREADS
	745	#define MAX_THREADS 64
	746	#endif
	747
	748	static struct bpool_params bpool_params;
	749	static struct xsk_umem_config umem_cfg;
	750	static struct bpool *bp;
	751
	752	static struct port_params port_params[MAX_PORTS];
	753	static struct port *ports[MAX_PORTS];
	754	static u64 n_pkts_rx[MAX_PORTS];
	755	static u64 n_pkts_tx[MAX_PORTS];
	756	static int n_ports;
	757
	758	static pthread_t threads[MAX_THREADS];
	759	static struct thread_data thread_data[MAX_THREADS];
	760	static int n_threads;
	761
	762	static void
	763	print_usage(char *prog_name)
	764	{
	765	const char *usage =
	766	"Usage:\n"
	767	"\t%s [ -b SIZE ] -c CORE -i INTERFACE [ -q QUEUE ]\n"
	768	"\n"
	769	"-c CORE CPU core to run a packet forwarding thread\n"
	770	" on. May be invoked multiple times.\n"
	771	"\n"
	772	"-b SIZE Number of buffers in the buffer pool shared\n"
	773	" by all the forwarding threads. Default: %u.\n"
	774	"\n"
	775	"-i INTERFACE Network interface. Each (INTERFACE, QUEUE)\n"
	776	" pair specifies one forwarding port. May be\n"
	777	" invoked multiple times.\n"
	778	"\n"
	779	"-q QUEUE Network interface queue for RX and TX. Each\n"
	780	" (INTERFACE, QUEUE) pair specified one\n"
	781	" forwarding port. Default: %u. May be invoked\n"
	782	" multiple times.\n"
	783	"\n";
	784	printf(usage,
	785	prog_name,
	786	bpool_params_default.n_buffers,
	787	port_params_default.iface_queue);
	788	}
	789
	790	static int
	791	parse_args(int argc, char **argv)
	792	{
	793	struct option lgopts[] = {
	794	{ NULL, 0, 0, 0 }
	795	};
	796	int opt, option_index;
	797
	798	/* Parse the input arguments. */
	799	for ( ; ;) {
	800	opt = getopt_long(argc, argv, "c:i:q:", lgopts, &option_index);
	801	if (opt == EOF)
	802	break;
	803
	804	switch (opt) {
	805	case 'b':
	806	bpool_params.n_buffers = atoi(optarg);
	807	break;
	808
	809	case 'c':
	810	if (n_threads == MAX_THREADS) {
	811	printf("Max number of threads (%d) reached.\n",
	812	MAX_THREADS);
	813	return -1;
	814	}
	815
	816	thread_data[n_threads].cpu_core_id = atoi(optarg);
	817	n_threads++;
	818	break;
	819
	820	case 'i':
	821	if (n_ports == MAX_PORTS) {
	822	printf("Max number of ports (%d) reached.\n",
	823	MAX_PORTS);
	824	return -1;
	825	}
	826
	827	port_params[n_ports].iface = optarg;
	828	port_params[n_ports].iface_queue = 0;
	829	n_ports++;
	830	break;
	831
	832	case 'q':
	833	if (n_ports == 0) {
	834	printf("No port specified for queue.\n");
	835	return -1;
	836	}
	837	port_params[n_ports - 1].iface_queue = atoi(optarg);
	838	break;
	839
	840	default:
	841	printf("Illegal argument.\n");
	842	return -1;
	843	}
	844	}
	845
	846	optind = 1; /* reset getopt lib */
	847
	848	/* Check the input arguments. */
	849	if (!n_ports) {
	850	printf("No ports specified.\n");
	851	return -1;
	852	}
	853
	854	if (!n_threads) {
	855	printf("No threads specified.\n");
	856	return -1;
	857	}
	858
	859	if (n_ports % n_threads) {
	860	printf("Ports cannot be evenly distributed to threads.\n");
	861	return -1;
	862	}
	863
	864	return 0;
	865	}
	866
	867	static void
	868	print_port(u32 port_id)
	869	{
	870	struct port *port = ports[port_id];
	871
	872	printf("Port %u: interface = %s, queue = %u\n",
	873	port_id, port->params.iface, port->params.iface_queue);
	874	}
	875
	876	static void
	877	print_thread(u32 thread_id)
	878	{
	879	struct thread_data *t = &thread_data[thread_id];
	880	u32 i;
	881
	882	printf("Thread %u (CPU core %u): ",
	883	thread_id, t->cpu_core_id);
	884
	885	for (i = 0; i < t->n_ports_rx; i++) {
	886	struct port *port_rx = t->ports_rx[i];
	887	struct port *port_tx = t->ports_tx[i];
	888
	889	printf("(%s, %u) -> (%s, %u), ",
	890	port_rx->params.iface,
	891	port_rx->params.iface_queue,
	892	port_tx->params.iface,
	893	port_tx->params.iface_queue);
	894	}
	895
	896	printf("\n");
	897	}
	898
	899	static void
	900	print_port_stats_separator(void)
	901	{
	902	printf("+-%4s-+-%12s-+-%13s-+-%12s-+-%13s-+\n",
	903	"----",
	904	"------------",
	905	"-------------",
	906	"------------",
	907	"-------------");
	908	}
	909
	910	static void
	911	print_port_stats_header(void)
	912	{
	913	print_port_stats_separator();
	914	printf("\| %4s \| %12s \| %13s \| %12s \| %13s \|\n",
	915	"Port",
	916	"RX packets",
	917	"RX rate (pps)",
	918	"TX packets",
	919	"TX_rate (pps)");
	920	print_port_stats_separator();
	921	}
	922
	923	static void
	924	print_port_stats_trailer(void)
	925	{
	926	print_port_stats_separator();
	927	printf("\n");
	928	}
	929
	930	static void
	931	print_port_stats(int port_id, u64 ns_diff)
	932	{
	933	struct port *p = ports[port_id];
	934	double rx_pps, tx_pps;
	935
	936	rx_pps = (p->n_pkts_rx - n_pkts_rx[port_id]) * 1000000000. / ns_diff;
	937	tx_pps = (p->n_pkts_tx - n_pkts_tx[port_id]) * 1000000000. / ns_diff;
	938
	939	printf("\| %4d \| %12llu \| %13.0f \| %12llu \| %13.0f \|\n",
	940	port_id,
	941	p->n_pkts_rx,
	942	rx_pps,
	943	p->n_pkts_tx,
	944	tx_pps);
	945
	946	n_pkts_rx[port_id] = p->n_pkts_rx;
	947	n_pkts_tx[port_id] = p->n_pkts_tx;
	948	}
	949
	950	static void
	951	print_port_stats_all(u64 ns_diff)
	952	{
	953	int i;
	954
	955	print_port_stats_header();
	956	for (i = 0; i < n_ports; i++)
	957	print_port_stats(i, ns_diff);
	958	print_port_stats_trailer();
	959	}
	960
	961	static int quit;
	962
	963	static void
	964	signal_handler(int sig)
	965	{
	966	quit = 1;
	967	}
	968
	969	static void remove_xdp_program(void)
	970	{
	971	int i;
	972
	973	for (i = 0 ; i < n_ports; i++)
	974	bpf_set_link_xdp_fd(if_nametoindex(port_params[i].iface), -1,
	975	port_params[i].xsk_cfg.xdp_flags);
	976	}
	977
	978	int main(int argc, char **argv)
	979	{
	980	struct timespec time;
	981	u64 ns0;
	982	int i;
	983
	984	/* Parse args. */
	985	memcpy(&bpool_params, &bpool_params_default,
	986	sizeof(struct bpool_params));
	987	memcpy(&umem_cfg, &umem_cfg_default,
	988	sizeof(struct xsk_umem_config));
	989	for (i = 0; i < MAX_PORTS; i++)
	990	memcpy(&port_params[i], &port_params_default,
	991	sizeof(struct port_params));
	992
	993	if (parse_args(argc, argv)) {
	994	print_usage(argv[0]);
	995	return -1;
	996	}
	997
	998	/* Buffer pool initialization. */
	999	bp = bpool_init(&bpool_params, &umem_cfg);
	1000	if (!bp) {
	1001	printf("Buffer pool initialization failed.\n");
	1002	return -1;
	1003	}
	1004	printf("Buffer pool created successfully.\n");
	1005
	1006	/* Ports initialization. */
	1007	for (i = 0; i < MAX_PORTS; i++)
	1008	port_params[i].bp = bp;
	1009
	1010	for (i = 0; i < n_ports; i++) {
	1011	ports[i] = port_init(&port_params[i]);
	1012	if (!ports[i]) {
	1013	printf("Port %d initialization failed.\n", i);
	1014	return -1;
	1015	}
	1016	print_port(i);
	1017	}
	1018	printf("All ports created successfully.\n");
	1019
	1020	/* Threads. */
	1021	for (i = 0; i < n_threads; i++) {
	1022	struct thread_data *t = &thread_data[i];
	1023	u32 n_ports_per_thread = n_ports / n_threads, j;
	1024
	1025	for (j = 0; j < n_ports_per_thread; j++) {
	1026	t->ports_rx[j] = ports[i * n_ports_per_thread + j];
	1027	t->ports_tx[j] = ports[i * n_ports_per_thread +
	1028	(j + 1) % n_ports_per_thread];
	1029	}
	1030
	1031	t->n_ports_rx = n_ports_per_thread;
	1032
	1033	print_thread(i);
	1034	}
	1035
	1036	for (i = 0; i < n_threads; i++) {
	1037	int status;
	1038
	1039	status = pthread_create(&threads[i],
	1040	NULL,
	1041	thread_func,
	1042	&thread_data[i]);
	1043	if (status) {
	1044	printf("Thread %d creation failed.\n", i);
	1045	return -1;
	1046	}
	1047	}
	1048	printf("All threads created successfully.\n");
	1049
	1050	/* Print statistics. */
	1051	signal(SIGINT, signal_handler);
	1052	signal(SIGTERM, signal_handler);
	1053	signal(SIGABRT, signal_handler);
	1054
	1055	clock_gettime(CLOCK_MONOTONIC, &time);
	1056	ns0 = time.tv_sec * 1000000000UL + time.tv_nsec;
	1057	for ( ; !quit; ) {
	1058	u64 ns1, ns_diff;
	1059
	1060	sleep(1);
	1061	clock_gettime(CLOCK_MONOTONIC, &time);
	1062	ns1 = time.tv_sec * 1000000000UL + time.tv_nsec;
	1063	ns_diff = ns1 - ns0;
	1064	ns0 = ns1;
	1065
	1066	print_port_stats_all(ns_diff);
	1067	}
	1068
	1069	/* Threads completion. */
	1070	printf("Quit.\n");
	1071	for (i = 0; i < n_threads; i++)
	1072	thread_data[i].quit = 1;
	1073
	1074	for (i = 0; i < n_threads; i++)
	1075	pthread_join(threads[i], NULL);
	1076
	1077	for (i = 0; i < n_ports; i++)
	1078	port_free(ports[i]);
	1079
	1080	bpool_free(bp);
	1081
	1082	remove_xdp_program();
	1083
	1084	return 0;
	1085	}

scripts/bpf_helpers_doc.py+2−0

             'struct __sk_buff',
             'struct sk_msg_md',
             'struct xdp_md',
+            'struct path',
     ]
     known_types = {
             '...',
             'struct tcp_request_sock',
             'struct udp6_sock',
             'struct task_struct',
+            'struct path',
     }
     mapped_types = {
             'u8': '__u8',

security/bpf/hooks.c+6−0

 	LSM_HOOK_INIT(NAME, bpf_lsm_##NAME),
 	#include <linux/lsm_hook_defs.h>
 	#undef LSM_HOOK
+	LSM_HOOK_INIT(inode_free_security, bpf_inode_storage_free),
 };
 static int __init bpf_lsm_init(void)
 	return 0;
 }
+struct lsm_blob_sizes bpf_lsm_blob_sizes __lsm_ro_after_init = {
+	.lbs_inode = sizeof(struct bpf_storage_blob),
+};
 DEFINE_LSM(bpf) = {
 	.name = "bpf",
 	.init = bpf_lsm_init,
+	.blobs = &bpf_lsm_blob_sizes
 };

tools/bpf/bpftool/Documentation/bpftool-map.rst+1−1

@@ -49,7 +49,7 @@ MAP COMMANDS · 48 unmodified lines
49	\| \| lru_percpu_hash \| lpm_trie \| array_of_maps \| hash_of_maps	49	\| \| lru_percpu_hash \| lpm_trie \| array_of_maps \| hash_of_maps
50	\| \| devmap \| devmap_hash \| sockmap \| cpumap \| xskmap \| sockhash	50	\| \| devmap \| devmap_hash \| sockmap \| cpumap \| xskmap \| sockhash
51	\| \| cgroup_storage \| reuseport_sockarray \| percpu_cgroup_storage	51	\| \| cgroup_storage \| reuseport_sockarray \| percpu_cgroup_storage
52	\| \| queue \| stack \| sk_storage \| struct_ops \| ringbuf }	52	\| \| queue \| stack \| sk_storage \| struct_ops \| ringbuf \| inode_storage }
53		53
54	DESCRIPTION	54	DESCRIPTION
55	===========	55	===========

tools/bpf/bpftool/Makefile+5−1

 $(OUTPUT)%.o: %.c
 	$(QUIET_CC)$(CC) $(CFLAGS) -c -MMD -o $@ $<
-clean: $(LIBBPF)-clean
+feature-detect-clean:
+	$(call QUIET_CLEAN, feature-detect)
+	$(Q)$(MAKE) -C $(srctree)/tools/build/feature/ clean >/dev/null
+clean: $(LIBBPF)-clean feature-detect-clean
 	$(call QUIET_CLEAN, bpftool)
 	$(Q)$(RM) -- $(OUTPUT)bpftool $(OUTPUT)*.o $(OUTPUT)*.d
 	$(Q)$(RM) -- $(BPFTOOL_BOOTSTRAP) $(OUTPUT)*.skel.h $(OUTPUT)vmlinux.h

tools/bpf/bpftool/bash-completion/bpftool+2−1

                                 lru_percpu_hash lpm_trie array_of_maps \
                                 hash_of_maps devmap devmap_hash sockmap cpumap \
                                 xskmap sockhash cgroup_storage reuseport_sockarray \
-                                percpu_cgroup_storage queue stack' -- \
+                                percpu_cgroup_storage queue stack sk_storage \
+                                struct_ops inode_storage' -- \
                                                    "$cur" ) )
                             return 0
                             ;;

tools/bpf/bpftool/gen.c+0−2

@@ -19,11 +19,9 @@ · 18 unmodified lines
19	#include <sys/mman.h>	19	#include <sys/mman.h>
20	#include <bpf/btf.h>	20	#include <bpf/btf.h>
21		21
22	#include "bpf/libbpf_internal.h"
23	#include "json_writer.h"	22	#include "json_writer.h"
24	#include "main.h"	23	#include "main.h"
25		24
26
27	#define MAX_OBJ_NAME_LEN 64	25	#define MAX_OBJ_NAME_LEN 64
28		26
29	static void sanitize_identifier(char *name)	27	static void sanitize_identifier(char *name)

tools/bpf/bpftool/link.c+41−3

 		jsonw_uint_field(wtr, "attach_type", attach_type);
 }
+static bool is_iter_map_target(const char *target_name)
+{
+	return strcmp(target_name, "bpf_map_elem") == 0 ||
+	       strcmp(target_name, "bpf_sk_storage_map") == 0;
+}
+static void show_iter_json(struct bpf_link_info *info, json_writer_t *wtr)
+{
+	const char *target_name = u64_to_ptr(info->iter.target_name);
+	jsonw_string_field(wtr, "target_name", target_name);
+	if (is_iter_map_target(target_name))
+		jsonw_uint_field(wtr, "map_id", info->iter.map.map_id);
+}
 static int get_prog_info(int prog_id, struct bpf_prog_info *info)
 {
 	__u32 len = sizeof(*info);
 				   info->cgroup.cgroup_id);
 		show_link_attach_type_json(info->cgroup.attach_type, json_wtr);
 		break;
+	case BPF_LINK_TYPE_ITER:
+		show_iter_json(info, json_wtr);
+		break;
 	case BPF_LINK_TYPE_NETNS:
 		jsonw_uint_field(json_wtr, "netns_ino",
 				 info->netns.netns_ino);
 		printf("attach_type %u  ", attach_type);
 }
+static void show_iter_plain(struct bpf_link_info *info)
+{
+	const char *target_name = u64_to_ptr(info->iter.target_name);
+	printf("target_name %s  ", target_name);
+	if (is_iter_map_target(target_name))
+		printf("map_id %u  ", info->iter.map.map_id);
+}
 static int show_link_close_plain(int fd, struct bpf_link_info *info)
 {
 	struct bpf_prog_info prog_info;
 		printf("\n\tcgroup_id %zu  ", (size_t)info->cgroup.cgroup_id);
 		show_link_attach_type_plain(info->cgroup.attach_type);
 		break;
+	case BPF_LINK_TYPE_ITER:
+		show_iter_plain(info);
+		break;
 	case BPF_LINK_TYPE_NETNS:
 		printf("\n\tnetns_ino %u  ", info->netns.netns_ino);
 		show_link_attach_type_plain(info->netns.attach_type);
 {
 	struct bpf_link_info info;
 	__u32 len = sizeof(info);
-	char raw_tp_name[256];
+	char buf[256];
 	int err;
 	memset(&info, 0, sizeof(info));
 	}
 	if (info.type == BPF_LINK_TYPE_RAW_TRACEPOINT &&
 	    !info.raw_tracepoint.tp_name) {
-		info.raw_tracepoint.tp_name = (unsigned long)&raw_tp_name;
+		info.raw_tracepoint.tp_name = (unsigned long)&buf;
-		info.raw_tracepoint.tp_name_len = sizeof(raw_tp_name);
+		info.raw_tracepoint.tp_name_len = sizeof(buf);
+		goto again;
+	}
+	if (info.type == BPF_LINK_TYPE_ITER &&
+	    !info.iter.target_name) {
+		info.iter.target_name = (unsigned long)&buf;
+		info.iter.target_name_len = sizeof(buf);
 		goto again;
 	}

tools/bpf/bpftool/map.c+2−1

 	[BPF_MAP_TYPE_SK_STORAGE]		= "sk_storage",
 	[BPF_MAP_TYPE_STRUCT_OPS]		= "struct_ops",
 	[BPF_MAP_TYPE_RINGBUF]			= "ringbuf",
+	[BPF_MAP_TYPE_INODE_STORAGE]		= "inode_storage",
 };
 const size_t map_type_name_size = ARRAY_SIZE(map_type_name);
 		"                 lru_percpu_hash | lpm_trie | array_of_maps | hash_of_maps |\n"
 		"                 devmap | devmap_hash | sockmap | cpumap | xskmap | sockhash |\n"
 		"                 cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n"
-		"                 queue | stack | sk_storage | struct_ops | ringbuf }\n"
+		"                 queue | stack | sk_storage | struct_ops | ringbuf | inode_storage }\n"
 		"       " HELP_SPEC_OPTIONS "\n"
 		"",
 		bin_name, argv[-2]);

tools/bpf/bpftool/net.c+282−17

@@ -6,22 +6,27 @@ · 5 unmodified lines
6	#include <fcntl.h>	6	#include <fcntl.h>
7	#include <stdlib.h>	7	#include <stdlib.h>
8	#include <string.h>	8	#include <string.h>
		9	#include <time.h>
9	#include <unistd.h>	10	#include <unistd.h>
10	#include <bpf/bpf.h>	11	#include <bpf/bpf.h>
11	#include <bpf/libbpf.h>	12	#include <bpf/libbpf.h>
12	#include <net/if.h>	13	#include <net/if.h>
13	#include <linux/if.h>	14	#include <linux/if.h>
14	#include <linux/rtnetlink.h>	15	#include <linux/rtnetlink.h>
		16	#include <linux/socket.h>
15	#include <linux/tc_act/tc_bpf.h>	17	#include <linux/tc_act/tc_bpf.h>
16	#include <sys/socket.h>	18	#include <sys/socket.h>
17	#include <sys/stat.h>	19	#include <sys/stat.h>
18	#include <sys/types.h>	20	#include <sys/types.h>
19		21
20	#include "bpf/nlattr.h"	22	#include "bpf/nlattr.h"
21	#include "bpf/libbpf_internal.h"
22	#include "main.h"	23	#include "main.h"
23	#include "netlink_dumper.h"	24	#include "netlink_dumper.h"
24		25
		26	#ifndef SOL_NETLINK
		27	#define SOL_NETLINK 270
		28	#endif
		29
25	struct ip_devname_ifindex {	30	struct ip_devname_ifindex {
26	char devname[64];	31	char devname[64];
27	int ifindex;	32	int ifindex;
@@ -85,6 +90,266 @@ static enum net_attach_type parse_attach_type(const char *str) · 57 unmodified lines
85	return net_attach_type_size;	90	return net_attach_type_size;
86	}	91	}
87		92
		93	typedef int (dump_nlmsg_t)(void cookie, void msg, struct nlattr *tb);
		94
		95	typedef int (__dump_nlmsg_t)(struct nlmsghdr nlmsg, dump_nlmsg_t, void *cookie);
		96
		97	static int netlink_open(__u32 *nl_pid)
		98	{
		99	struct sockaddr_nl sa;
		100	socklen_t addrlen;
		101	int one = 1, ret;
		102	int sock;
		103
		104	memset(&sa, 0, sizeof(sa));
		105	sa.nl_family = AF_NETLINK;
		106
		107	sock = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);
		108	if (sock < 0)
		109	return -errno;
		110
		111	if (setsockopt(sock, SOL_NETLINK, NETLINK_EXT_ACK,
		112	&one, sizeof(one)) < 0) {
		113	p_err("Netlink error reporting not supported");
		114	}
		115
		116	if (bind(sock, (struct sockaddr *)&sa, sizeof(sa)) < 0) {
		117	ret = -errno;
		118	goto cleanup;
		119	}
		120
		121	addrlen = sizeof(sa);
		122	if (getsockname(sock, (struct sockaddr *)&sa, &addrlen) < 0) {
		123	ret = -errno;
		124	goto cleanup;
		125	}
		126
		127	if (addrlen != sizeof(sa)) {
		128	ret = -LIBBPF_ERRNO__INTERNAL;
		129	goto cleanup;
		130	}
		131
		132	*nl_pid = sa.nl_pid;
		133	return sock;
		134
		135	cleanup:
		136	close(sock);
		137	return ret;
		138	}
		139
		140	static int netlink_recv(int sock, __u32 nl_pid, __u32 seq,
		141	__dump_nlmsg_t _fn, dump_nlmsg_t fn,
		142	void *cookie)
		143	{
		144	bool multipart = true;
		145	struct nlmsgerr *err;
		146	struct nlmsghdr *nh;
		147	char buf[4096];
		148	int len, ret;
		149
		150	while (multipart) {
		151	multipart = false;
		152	len = recv(sock, buf, sizeof(buf), 0);
		153	if (len < 0) {
		154	ret = -errno;
		155	goto done;
		156	}
		157
		158	if (len == 0)
		159	break;
		160
		161	for (nh = (struct nlmsghdr *)buf; NLMSG_OK(nh, len);
		162	nh = NLMSG_NEXT(nh, len)) {
		163	if (nh->nlmsg_pid != nl_pid) {
		164	ret = -LIBBPF_ERRNO__WRNGPID;
		165	goto done;
		166	}
		167	if (nh->nlmsg_seq != seq) {
		168	ret = -LIBBPF_ERRNO__INVSEQ;
		169	goto done;
		170	}
		171	if (nh->nlmsg_flags & NLM_F_MULTI)
		172	multipart = true;
		173	switch (nh->nlmsg_type) {
		174	case NLMSG_ERROR:
		175	err = (struct nlmsgerr *)NLMSG_DATA(nh);
		176	if (!err->error)
		177	continue;
		178	ret = err->error;
		179	libbpf_nla_dump_errormsg(nh);
		180	goto done;
		181	case NLMSG_DONE:
		182	return 0;
		183	default:
		184	break;
		185	}
		186	if (_fn) {
		187	ret = _fn(nh, fn, cookie);
		188	if (ret)
		189	return ret;
		190	}
		191	}
		192	}
		193	ret = 0;
		194	done:
		195	return ret;
		196	}
		197
		198	static int __dump_class_nlmsg(struct nlmsghdr *nlh,
		199	dump_nlmsg_t dump_class_nlmsg,
		200	void *cookie)
		201	{
		202	struct nlattr tb[TCA_MAX + 1], attr;
		203	struct tcmsg *t = NLMSG_DATA(nlh);
		204	int len;
		205
		206	len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*t));
		207	attr = (struct nlattr ) ((void ) t + NLMSG_ALIGN(sizeof(*t)));
		208	if (libbpf_nla_parse(tb, TCA_MAX, attr, len, NULL) != 0)
		209	return -LIBBPF_ERRNO__NLPARSE;
		210
		211	return dump_class_nlmsg(cookie, t, tb);
		212	}
		213
		214	static int netlink_get_class(int sock, unsigned int nl_pid, int ifindex,
		215	dump_nlmsg_t dump_class_nlmsg, void *cookie)
		216	{
		217	struct {
		218	struct nlmsghdr nlh;
		219	struct tcmsg t;
		220	} req = {
		221	.nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg)),
		222	.nlh.nlmsg_type = RTM_GETTCLASS,
		223	.nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
		224	.t.tcm_family = AF_UNSPEC,
		225	.t.tcm_ifindex = ifindex,
		226	};
		227	int seq = time(NULL);
		228
		229	req.nlh.nlmsg_seq = seq;
		230	if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
		231	return -errno;
		232
		233	return netlink_recv(sock, nl_pid, seq, __dump_class_nlmsg,
		234	dump_class_nlmsg, cookie);
		235	}
		236
		237	static int __dump_qdisc_nlmsg(struct nlmsghdr *nlh,
		238	dump_nlmsg_t dump_qdisc_nlmsg,
		239	void *cookie)
		240	{
		241	struct nlattr tb[TCA_MAX + 1], attr;
		242	struct tcmsg *t = NLMSG_DATA(nlh);
		243	int len;
		244
		245	len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*t));
		246	attr = (struct nlattr ) ((void ) t + NLMSG_ALIGN(sizeof(*t)));
		247	if (libbpf_nla_parse(tb, TCA_MAX, attr, len, NULL) != 0)
		248	return -LIBBPF_ERRNO__NLPARSE;
		249
		250	return dump_qdisc_nlmsg(cookie, t, tb);
		251	}
		252
		253	static int netlink_get_qdisc(int sock, unsigned int nl_pid, int ifindex,
		254	dump_nlmsg_t dump_qdisc_nlmsg, void *cookie)
		255	{
		256	struct {
		257	struct nlmsghdr nlh;
		258	struct tcmsg t;
		259	} req = {
		260	.nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg)),
		261	.nlh.nlmsg_type = RTM_GETQDISC,
		262	.nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
		263	.t.tcm_family = AF_UNSPEC,
		264	.t.tcm_ifindex = ifindex,
		265	};
		266	int seq = time(NULL);
		267
		268	req.nlh.nlmsg_seq = seq;
		269	if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
		270	return -errno;
		271
		272	return netlink_recv(sock, nl_pid, seq, __dump_qdisc_nlmsg,
		273	dump_qdisc_nlmsg, cookie);
		274	}
		275
		276	static int __dump_filter_nlmsg(struct nlmsghdr *nlh,
		277	dump_nlmsg_t dump_filter_nlmsg,
		278	void *cookie)
		279	{
		280	struct nlattr tb[TCA_MAX + 1], attr;
		281	struct tcmsg *t = NLMSG_DATA(nlh);
		282	int len;
		283
		284	len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*t));
		285	attr = (struct nlattr ) ((void ) t + NLMSG_ALIGN(sizeof(*t)));
		286	if (libbpf_nla_parse(tb, TCA_MAX, attr, len, NULL) != 0)
		287	return -LIBBPF_ERRNO__NLPARSE;
		288
		289	return dump_filter_nlmsg(cookie, t, tb);
		290	}
		291
		292	static int netlink_get_filter(int sock, unsigned int nl_pid, int ifindex, int handle,
		293	dump_nlmsg_t dump_filter_nlmsg, void *cookie)
		294	{
		295	struct {
		296	struct nlmsghdr nlh;
		297	struct tcmsg t;
		298	} req = {
		299	.nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg)),
		300	.nlh.nlmsg_type = RTM_GETTFILTER,
		301	.nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
		302	.t.tcm_family = AF_UNSPEC,
		303	.t.tcm_ifindex = ifindex,
		304	.t.tcm_parent = handle,
		305	};
		306	int seq = time(NULL);
		307
		308	req.nlh.nlmsg_seq = seq;
		309	if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
		310	return -errno;
		311
		312	return netlink_recv(sock, nl_pid, seq, __dump_filter_nlmsg,
		313	dump_filter_nlmsg, cookie);
		314	}
		315
		316	static int __dump_link_nlmsg(struct nlmsghdr *nlh,
		317	dump_nlmsg_t dump_link_nlmsg, void *cookie)
		318	{
		319	struct nlattr tb[IFLA_MAX + 1], attr;
		320	struct ifinfomsg *ifi = NLMSG_DATA(nlh);
		321	int len;
		322
		323	len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*ifi));
		324	attr = (struct nlattr ) ((void ) ifi + NLMSG_ALIGN(sizeof(*ifi)));
		325	if (libbpf_nla_parse(tb, IFLA_MAX, attr, len, NULL) != 0)
		326	return -LIBBPF_ERRNO__NLPARSE;
		327
		328	return dump_link_nlmsg(cookie, ifi, tb);
		329	}
		330
		331	static int netlink_get_link(int sock, unsigned int nl_pid,
		332	dump_nlmsg_t dump_link_nlmsg, void *cookie)
		333	{
		334	struct {
		335	struct nlmsghdr nlh;
		336	struct ifinfomsg ifm;
		337	} req = {
		338	.nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct ifinfomsg)),
		339	.nlh.nlmsg_type = RTM_GETLINK,
		340	.nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
		341	.ifm.ifi_family = AF_PACKET,
		342	};
		343	int seq = time(NULL);
		344
		345	req.nlh.nlmsg_seq = seq;
		346	if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
		347	return -errno;
		348
		349	return netlink_recv(sock, nl_pid, seq, __dump_link_nlmsg,
		350	dump_link_nlmsg, cookie);
		351	}
		352
88	static int dump_link_nlmsg(void cookie, void msg, struct nlattr **tb)	353	static int dump_link_nlmsg(void cookie, void msg, struct nlattr **tb)
89	{	354	{
90	struct bpf_netdev_t *netinfo = cookie;	355	struct bpf_netdev_t *netinfo = cookie;
@@ -168,14 +433,14 @@ static int show_dev_tc_bpf(int sock, unsigned int nl_pid, · 77 unmodified lines
168	tcinfo.array_len = 0;	433	tcinfo.array_len = 0;
169		434
170	tcinfo.is_qdisc = false;	435	tcinfo.is_qdisc = false;
171	ret = libbpf_nl_get_class(sock, nl_pid, dev->ifindex,	436	ret = netlink_get_class(sock, nl_pid, dev->ifindex,
172	dump_class_qdisc_nlmsg, &tcinfo);	437	dump_class_qdisc_nlmsg, &tcinfo);
173	if (ret)	438	if (ret)
174	goto out;	439	goto out;
175		440
176	tcinfo.is_qdisc = true;	441	tcinfo.is_qdisc = true;
177	ret = libbpf_nl_get_qdisc(sock, nl_pid, dev->ifindex,	442	ret = netlink_get_qdisc(sock, nl_pid, dev->ifindex,
178	dump_class_qdisc_nlmsg, &tcinfo);	443	dump_class_qdisc_nlmsg, &tcinfo);
179	if (ret)	444	if (ret)
180	goto out;	445	goto out;
181		446
@@ -183,9 +448,9 @@ static int show_dev_tc_bpf(int sock, unsigned int nl_pid, · 1 unmodified line
183	filter_info.ifindex = dev->ifindex;	448	filter_info.ifindex = dev->ifindex;
184	for (i = 0; i < tcinfo.used_len; i++) {	449	for (i = 0; i < tcinfo.used_len; i++) {
185	filter_info.kind = tcinfo.handle_array[i].kind;	450	filter_info.kind = tcinfo.handle_array[i].kind;
186	ret = libbpf_nl_get_filter(sock, nl_pid, dev->ifindex,	451	ret = netlink_get_filter(sock, nl_pid, dev->ifindex,
187	tcinfo.handle_array[i].handle,	452	tcinfo.handle_array[i].handle,
188	dump_filter_nlmsg, &filter_info);	453	dump_filter_nlmsg, &filter_info);
189	if (ret)	454	if (ret)
190	goto out;	455	goto out;
191	}	456	}
@@ -193,22 +458,22 @@ static int show_dev_tc_bpf(int sock, unsigned int nl_pid, · 1 unmodified line
193	/* root, ingress and egress handle */	458	/* root, ingress and egress handle */
194	handle = TC_H_ROOT;	459	handle = TC_H_ROOT;
195	filter_info.kind = "root";	460	filter_info.kind = "root";
196	ret = libbpf_nl_get_filter(sock, nl_pid, dev->ifindex, handle,	461	ret = netlink_get_filter(sock, nl_pid, dev->ifindex, handle,
197	dump_filter_nlmsg, &filter_info);	462	dump_filter_nlmsg, &filter_info);
198	if (ret)	463	if (ret)
199	goto out;	464	goto out;
200		465
201	handle = TC_H_MAKE(TC_H_CLSACT, TC_H_MIN_INGRESS);	466	handle = TC_H_MAKE(TC_H_CLSACT, TC_H_MIN_INGRESS);
202	filter_info.kind = "clsact/ingress";	467	filter_info.kind = "clsact/ingress";
203	ret = libbpf_nl_get_filter(sock, nl_pid, dev->ifindex, handle,	468	ret = netlink_get_filter(sock, nl_pid, dev->ifindex, handle,
204	dump_filter_nlmsg, &filter_info);	469	dump_filter_nlmsg, &filter_info);
205	if (ret)	470	if (ret)
206	goto out;	471	goto out;
207		472
208	handle = TC_H_MAKE(TC_H_CLSACT, TC_H_MIN_EGRESS);	473	handle = TC_H_MAKE(TC_H_CLSACT, TC_H_MIN_EGRESS);
209	filter_info.kind = "clsact/egress";	474	filter_info.kind = "clsact/egress";
210	ret = libbpf_nl_get_filter(sock, nl_pid, dev->ifindex, handle,	475	ret = netlink_get_filter(sock, nl_pid, dev->ifindex, handle,
211	dump_filter_nlmsg, &filter_info);	476	dump_filter_nlmsg, &filter_info);
212	if (ret)	477	if (ret)
213	goto out;	478	goto out;
214		479
@@ -386,7 +651,7 @@ static int do_show(int argc, char **argv) · 171 unmodified lines
386	struct bpf_attach_info attach_info = {};	651	struct bpf_attach_info attach_info = {};
387	int i, sock, ret, filter_idx = -1;	652	int i, sock, ret, filter_idx = -1;
388	struct bpf_netdev_t dev_array;	653	struct bpf_netdev_t dev_array;
389	unsigned int nl_pid;	654	unsigned int nl_pid = 0;
390	char err_buf[256];	655	char err_buf[256];
391		656
392	if (argc == 2) {	657	if (argc == 2) {
@@ -401,7 +666,7 @@ static int do_show(int argc, char **argv) · 8 unmodified lines
401	if (ret)	666	if (ret)
402	return -1;	667	return -1;
403		668
404	sock = libbpf_netlink_open(&nl_pid);	669	sock = netlink_open(&nl_pid);
405	if (sock < 0) {	670	if (sock < 0) {
406	fprintf(stderr, "failed to open netlink sock\n");	671	fprintf(stderr, "failed to open netlink sock\n");
407	return -1;	672	return -1;
@@ -416,7 +681,7 @@ static int do_show(int argc, char **argv) · 8 unmodified lines
416	jsonw_start_array(json_wtr);	681	jsonw_start_array(json_wtr);
417	NET_START_OBJECT;	682	NET_START_OBJECT;
418	NET_START_ARRAY("xdp", "%s:\n");	683	NET_START_ARRAY("xdp", "%s:\n");
419	ret = libbpf_nl_get_link(sock, nl_pid, dump_link_nlmsg, &dev_array);	684	ret = netlink_get_link(sock, nl_pid, dump_link_nlmsg, &dev_array);
420	NET_END_ARRAY("\n");	685	NET_END_ARRAY("\n");
421		686
422	if (!ret) {	687	if (!ret) {

tools/bpf/resolve_btfids/main.c+27−2

@@ -199,9 +199,16 @@ static char get_id(const char prefix_end) · 198 unmodified lines
199	/*	199	/*
200	* __BTF_ID__func__vfs_truncate__0	200	* __BTF_ID__func__vfs_truncate__0
201	* prefix_end = ^	201	* prefix_end = ^
		202	* pos = ^
202	*/	203	*/
203	char p, id = strdup(prefix_end + sizeof("__") - 1);	204	int len = strlen(prefix_end);
		205	int pos = sizeof("__") - 1;
		206	char p, id;
204		207
		208	if (pos >= len)
		209	return NULL;
		210
		211	id = strdup(prefix_end + pos);
205	if (id) {	212	if (id) {
206	/*	213	/*
207	* __BTF_ID__func__vfs_truncate__0	214	* __BTF_ID__func__vfs_truncate__0
@@ -220,6 +227,24 @@ static char get_id(const char prefix_end) · 12 unmodified lines
220	return id;	227	return id;
221	}	228	}
222		229
		230	static struct btf_id add_set(struct object obj, char *name)
		231	{
		232	/*
		233	* __BTF_ID__set__name
		234	* name = ^
		235	* id = ^
		236	*/
		237	char *id = name + sizeof(BTF_SET "__") - 1;
		238	int len = strlen(name);
		239
		240	if (id >= name + len) {
		241	pr_err("FAILED to parse set name: %s\n", name);
		242	return NULL;
		243	}
		244
		245	return btf_id__add(&obj->sets, id, true);
		246	}
		247
223	static struct btf_id add_symbol(struct rb_root root, char *name, size_t size)	248	static struct btf_id add_symbol(struct rb_root root, char *name, size_t size)
224	{	249	{
225	char *id;	250	char *id;
@@ -412,7 +437,7 @@ static int symbols_collect(struct object *obj) · 186 unmodified lines
412	id = add_symbol(&obj->funcs, prefix, sizeof(BTF_FUNC) - 1);	437	id = add_symbol(&obj->funcs, prefix, sizeof(BTF_FUNC) - 1);
413	/* set */	438	/* set */
414	} else if (!strncmp(prefix, BTF_SET, sizeof(BTF_SET) - 1)) {	439	} else if (!strncmp(prefix, BTF_SET, sizeof(BTF_SET) - 1)) {
415	id = add_symbol(&obj->sets, prefix, sizeof(BTF_SET) - 1);	440	id = add_set(obj, prefix);
416	/*	441	/*
417	* SET objects store list's count, which is encoded	442	* SET objects store list's count, which is encoded
418	* in symbol's size, together with 'cnt' field hence	443	* in symbol's size, together with 'cnt' field hence

tools/build/Makefile+2−0

@@ -38,6 +38,8 @@ clean: · 37 unmodified lines
38	$(call QUIET_CLEAN, fixdep)	38	$(call QUIET_CLEAN, fixdep)
39	$(Q)find $(if $(OUTPUT),$(OUTPUT),.) -name '.o' -delete -o -name '\..cmd' -delete -o -name '\.*.d' -delete	39	$(Q)find $(if $(OUTPUT),$(OUTPUT),.) -name '.o' -delete -o -name '\..cmd' -delete -o -name '\.*.d' -delete
40	$(Q)rm -f $(OUTPUT)fixdep	40	$(Q)rm -f $(OUTPUT)fixdep
		41	$(call QUIET_CLEAN, feature-detect)
		42	$(Q)$(MAKE) -C feature/ clean >/dev/null
41		43
42	$(OUTPUT)fixdep-in.o: FORCE	44	$(OUTPUT)fixdep-in.o: FORCE
43	$(Q)$(MAKE) $(build)=fixdep	45	$(Q)$(MAKE) $(build)=fixdep

tools/build/Makefile.feature+0−1

@@ -46,7 +46,6 @@ FEATURE_TESTS_BASIC := \ · 45 unmodified lines
46	libelf-getphdrnum \	46	libelf-getphdrnum \
47	libelf-gelf_getnote \	47	libelf-gelf_getnote \
48	libelf-getshdrstrndx \	48	libelf-getshdrstrndx \
49	libelf-mmap \
50	libnuma \	49	libnuma \
51	numa_num_possible_cpus \	50	numa_num_possible_cpus \
52	libperl \	51	libperl \

tools/build/feature/Makefile+0−4

          test-libelf-getphdrnum.bin             \
          test-libelf-gelf_getnote.bin           \
          test-libelf-getshdrstrndx.bin          \
-         test-libelf-mmap.bin                   \
          test-libdebuginfod.bin                 \
          test-libnuma.bin                       \
          test-numa_num_possible_cpus.bin        \
 $(OUTPUT)test-dwarf_getlocations.bin:
 	$(BUILD) $(DWARFLIBS)
-$(OUTPUT)test-libelf-mmap.bin:
-	$(BUILD) -lelf
 $(OUTPUT)test-libelf-getphdrnum.bin:
 	$(BUILD) -lelf

tools/build/feature/test-all.c+0−4

 # include "test-libelf.c"
 #undef main
-#define main main_test_libelf_mmap
-# include "test-libelf-mmap.c"
-#undef main
 #define main main_test_get_current_dir_name
 # include "test-get_current_dir_name.c"
 #undef main

tools/build/feature/test-libelf-mmap.c+0−9

@@ -1,9 +0,0 @@
1	// SPDX-License-Identifier: GPL-2.0
2	#include <libelf.h>
3
4	int main(void)
5	{
6	Elf *elf = elf_begin(0, ELF_C_READ_MMAP, 0);
7
8	return (long)elf;
9	}

tools/include/linux/btf_ids.h+50−1

@@ -3,6 +3,11 @@ · 2 unmodified lines
3	#ifndef _LINUX_BTF_IDS_H	3	#ifndef _LINUX_BTF_IDS_H
4	#define _LINUX_BTF_IDS_H	4	#define _LINUX_BTF_IDS_H
5		5
		6	struct btf_id_set {
		7	u32 cnt;
		8	u32 ids[];
		9	};
		10
6	#ifdef CONFIG_DEBUG_INFO_BTF	11	#ifdef CONFIG_DEBUG_INFO_BTF
7		12
8	#include <linux/compiler.h> /* for __PASTE */	13	#include <linux/compiler.h> /* for __PASTE */
@@ -62,7 +67,7 @@ asm( \ · 53 unmodified lines
62	".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \	67	".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
63	"." #scope " " #name "; \n" \	68	"." #scope " " #name "; \n" \
64	#name ":; \n" \	69	#name ":; \n" \
65	".popsection; \n"); \	70	".popsection; \n");
66		71
67	#define BTF_ID_LIST(name) \	72	#define BTF_ID_LIST(name) \
68	__BTF_ID_LIST(name, local) \	73	__BTF_ID_LIST(name, local) \
@@ -88,12 +93,56 @@ asm( \ · 19 unmodified lines
88	".zero 4 \n" \	93	".zero 4 \n" \
89	".popsection; \n");	94	".popsection; \n");
90		95
		96	/*
		97	* The BTF_SET_START/END macros pair defines sorted list of
		98	* BTF IDs plus its members count, with following layout:
		99	*
		100	* BTF_SET_START(list)
		101	* BTF_ID(type1, name1)
		102	* BTF_ID(type2, name2)
		103	* BTF_SET_END(list)
		104	*
		105	* __BTF_ID__set__list:
		106	* .zero 4
		107	* list:
		108	* __BTF_ID__type1__name1__3:
		109	* .zero 4
		110	* __BTF_ID__type2__name2__4:
		111	* .zero 4
		112	*
		113	*/
		114	#define __BTF_SET_START(name, scope) \
		115	asm( \
		116	".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
		117	"." #scope " __BTF_ID__set__" #name "; \n" \
		118	"__BTF_ID__set__" #name ":; \n" \
		119	".zero 4 \n" \
		120	".popsection; \n");
		121
		122	#define BTF_SET_START(name) \
		123	__BTF_ID_LIST(name, local) \
		124	__BTF_SET_START(name, local)
		125
		126	#define BTF_SET_START_GLOBAL(name) \
		127	__BTF_ID_LIST(name, globl) \
		128	__BTF_SET_START(name, globl)
		129
		130	#define BTF_SET_END(name) \
		131	asm( \
		132	".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
		133	".size __BTF_ID__set__" #name ", .-" #name " \n" \
		134	".popsection; \n"); \
		135	extern struct btf_id_set name;
		136
91	#else	137	#else
92		138
93	#define BTF_ID_LIST(name) static u32 name[5];	139	#define BTF_ID_LIST(name) static u32 name[5];
94	#define BTF_ID(prefix, name)	140	#define BTF_ID(prefix, name)
95	#define BTF_ID_UNUSED	141	#define BTF_ID_UNUSED
96	#define BTF_ID_LIST_GLOBAL(name) u32 name[1];	142	#define BTF_ID_LIST_GLOBAL(name) u32 name[1];
		143	#define BTF_SET_START(name) static struct btf_id_set name = { 0 };
		144	#define BTF_SET_START_GLOBAL(name) static struct btf_id_set name = { 0 };
		145	#define BTF_SET_END(name)
97		146
98	#endif /* CONFIG_DEBUG_INFO_BTF */	147	#endif /* CONFIG_DEBUG_INFO_BTF */
99		148

tools/include/uapi/linux/bpf.h+393−5

@@ -155,6 +155,7 @@ enum bpf_map_type { · 154 unmodified lines
155	BPF_MAP_TYPE_DEVMAP_HASH,	155	BPF_MAP_TYPE_DEVMAP_HASH,
156	BPF_MAP_TYPE_STRUCT_OPS,	156	BPF_MAP_TYPE_STRUCT_OPS,
157	BPF_MAP_TYPE_RINGBUF,	157	BPF_MAP_TYPE_RINGBUF,
		158	BPF_MAP_TYPE_INODE_STORAGE,
158	};	159	};
159		160
160	/* Note that tracing related programs such as	161	/* Note that tracing related programs such as
@@ -345,6 +346,14 @@ enum bpf_link_type { · 184 unmodified lines
345	/* The verifier internal test flag. Behavior is undefined */	346	/* The verifier internal test flag. Behavior is undefined */
346	#define BPF_F_TEST_STATE_FREQ (1U << 3)	347	#define BPF_F_TEST_STATE_FREQ (1U << 3)
347		348
		349	/* If BPF_F_SLEEPABLE is used in BPF_PROG_LOAD command, the verifier will
		350	* restrict map and helper usage for such programs. Sleepable BPF programs can
		351	* only be attached to hooks where kernel execution context allows sleeping.
		352	* Such programs are allowed to use helpers that may sleep like
		353	* bpf_copy_from_user().
		354	*/
		355	#define BPF_F_SLEEPABLE (1U << 4)
		356
348	/* When BPF ldimm64's insn[0].src_reg != 0 then this can have	357	/* When BPF ldimm64's insn[0].src_reg != 0 then this can have
349	* two extensions:	358	* two extensions:
350	*	359	*
@@ -2807,7 +2816,7 @@ union bpf_attr { · 2456 unmodified lines
2807	*	2816	*
2808	* -ERANGE if resulting value was out of range.	2817	* -ERANGE if resulting value was out of range.
2809	*	2818	*
2810	* void bpf_sk_storage_get(struct bpf_map map, struct bpf_sock sk, void value, u64 flags)	2819	* void bpf_sk_storage_get(struct bpf_map map, void sk, void value, u64 flags)
2811	* Description	2820	* Description
2812	* Get a bpf-local-storage from a sk.	2821	* Get a bpf-local-storage from a sk.
2813	*	2822	*
@@ -2823,6 +2832,9 @@ union bpf_attr { · 9 unmodified lines
2823	* "type". The bpf-local-storage "type" (i.e. the map) is	2832	* "type". The bpf-local-storage "type" (i.e. the map) is
2824	* searched against all bpf-local-storages residing at sk.	2833	* searched against all bpf-local-storages residing at sk.
2825	*	2834	*
		2835	* sk is a kernel struct sock pointer for LSM program.
		2836	* sk is a struct bpf_sock pointer for other program types.
		2837	*
2826	* An optional flags (BPF_SK_STORAGE_GET_F_CREATE) can be	2838	* An optional flags (BPF_SK_STORAGE_GET_F_CREATE) can be
2827	* used such that a new bpf-local-storage will be	2839	* used such that a new bpf-local-storage will be
2828	* created if one does not exist. value can be used	2840	* created if one does not exist. value can be used
@@ -2835,7 +2847,7 @@ union bpf_attr { · 6 unmodified lines
2835	* NULL if not found or there was an error in adding	2847	* NULL if not found or there was an error in adding
2836	* a new bpf-local-storage.	2848	* a new bpf-local-storage.
2837	*	2849	*
2838	* long bpf_sk_storage_delete(struct bpf_map map, struct bpf_sock sk)	2850	* long bpf_sk_storage_delete(struct bpf_map map, void sk)
2839	* Description	2851	* Description
2840	* Delete a bpf-local-storage from a sk.	2852	* Delete a bpf-local-storage from a sk.
2841	* Return	2853	* Return
@@ -3395,6 +3407,175 @@ union bpf_attr { · 553 unmodified lines
3395	* A non-negative value equal to or less than size on success,	3407	* A non-negative value equal to or less than size on success,
3396	* or a negative error in case of failure.	3408	* or a negative error in case of failure.
3397	*	3409	*
		3410	* long bpf_load_hdr_opt(struct bpf_sock_ops skops, void searchby_res, u32 len, u64 flags)
		3411	* Description
		3412	* Load header option. Support reading a particular TCP header
		3413	* option for bpf program (BPF_PROG_TYPE_SOCK_OPS).
		3414	*
		3415	* If flags is 0, it will search the option from the
		3416	* sock_ops->skb_data. The comment in "struct bpf_sock_ops"
		3417	* has details on what skb_data contains under different
		3418	* sock_ops->op.
		3419	*
		3420	* The first byte of the searchby_res specifies the
		3421	* kind that it wants to search.
		3422	*
		3423	* If the searching kind is an experimental kind
		3424	* (i.e. 253 or 254 according to RFC6994). It also
		3425	* needs to specify the "magic" which is either
		3426	* 2 bytes or 4 bytes. It then also needs to
		3427	* specify the size of the magic by using
		3428	* the 2nd byte which is "kind-length" of a TCP
		3429	* header option and the "kind-length" also
		3430	* includes the first 2 bytes "kind" and "kind-length"
		3431	* itself as a normal TCP header option also does.
		3432	*
		3433	* For example, to search experimental kind 254 with
		3434	* 2 byte magic 0xeB9F, the searchby_res should be
		3435	* [ 254, 4, 0xeB, 0x9F, 0, 0, .... 0 ].
		3436	*
		3437	* To search for the standard window scale option (3),
		3438	* the searchby_res should be [ 3, 0, 0, .... 0 ].
		3439	* Note, kind-length must be 0 for regular option.
		3440	*
		3441	* Searching for No-Op (0) and End-of-Option-List (1) are
		3442	* not supported.
		3443	*
		3444	* len must be at least 2 bytes which is the minimal size
		3445	* of a header option.
		3446	*
		3447	* Supported flags:
		3448	* * BPF_LOAD_HDR_OPT_TCP_SYN to search from the
		3449	* saved_syn packet or the just-received syn packet.
		3450	*
		3451	* Return
		3452	* >0 when found, the header option is copied to searchby_res.
		3453	* The return value is the total length copied.
		3454	*
		3455	* -EINVAL If param is invalid
		3456	*
		3457	* -ENOMSG The option is not found
		3458	*
		3459	* -ENOENT No syn packet available when
		3460	* BPF_LOAD_HDR_OPT_TCP_SYN is used
		3461	*
		3462	* -ENOSPC Not enough space. Only len number of
		3463	* bytes are copied.
		3464	*
		3465	* -EFAULT Cannot parse the header options in the packet
		3466	*
		3467	* -EPERM This helper cannot be used under the
		3468	* current sock_ops->op.
		3469	*
		3470	* long bpf_store_hdr_opt(struct bpf_sock_ops skops, const void from, u32 len, u64 flags)
		3471	* Description
		3472	* Store header option. The data will be copied
		3473	* from buffer from with length len to the TCP header.
		3474	*
		3475	* The buffer from should have the whole option that
		3476	* includes the kind, kind-length, and the actual
		3477	* option data. The len must be at least kind-length
		3478	* long. The kind-length does not have to be 4 byte
		3479	* aligned. The kernel will take care of the padding
		3480	* and setting the 4 bytes aligned value to th->doff.
		3481	*
		3482	* This helper will check for duplicated option
		3483	* by searching the same option in the outgoing skb.
		3484	*
		3485	* This helper can only be called during
		3486	* BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
		3487	*
		3488	* Return
		3489	* 0 on success, or negative error in case of failure:
		3490	*
		3491	* -EINVAL If param is invalid
		3492	*
		3493	* -ENOSPC Not enough space in the header.
		3494	* Nothing has been written
		3495	*
		3496	* -EEXIST The option has already existed
		3497	*
		3498	* -EFAULT Cannot parse the existing header options
		3499	*
		3500	* -EPERM This helper cannot be used under the
		3501	* current sock_ops->op.
		3502	*
		3503	* long bpf_reserve_hdr_opt(struct bpf_sock_ops *skops, u32 len, u64 flags)
		3504	* Description
		3505	* Reserve len bytes for the bpf header option. The
		3506	* space will be used by bpf_store_hdr_opt() later in
		3507	* BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
		3508	*
		3509	* If bpf_reserve_hdr_opt() is called multiple times,
		3510	* the total number of bytes will be reserved.
		3511	*
		3512	* This helper can only be called during
		3513	* BPF_SOCK_OPS_HDR_OPT_LEN_CB.
		3514	*
		3515	* Return
		3516	* 0 on success, or negative error in case of failure:
		3517	*
		3518	* -EINVAL if param is invalid
		3519	*
		3520	* -ENOSPC Not enough space in the header.
		3521	*
		3522	* -EPERM This helper cannot be used under the
		3523	* current sock_ops->op.
		3524	*
		3525	* void bpf_inode_storage_get(struct bpf_map map, void inode, void value, u64 flags)
		3526	* Description
		3527	* Get a bpf_local_storage from an inode.
		3528	*
		3529	* Logically, it could be thought of as getting the value from
		3530	* a map with inode as the key. From this
		3531	* perspective, the usage is not much different from
		3532	* bpf_map_lookup_elem\ (map, &\ inode) except this
		3533	* helper enforces the key must be an inode and the map must also
		3534	* be a BPF_MAP_TYPE_INODE_STORAGE.
		3535	*
		3536	* Underneath, the value is stored locally at inode instead of
		3537	* the map. The map is used as the bpf-local-storage
		3538	* "type". The bpf-local-storage "type" (i.e. the map) is
		3539	* searched against all bpf_local_storage residing at inode.
		3540	*
		3541	* An optional flags (BPF_LOCAL_STORAGE_GET_F_CREATE) can be
		3542	* used such that a new bpf_local_storage will be
		3543	* created if one does not exist. value can be used
		3544	* together with BPF_LOCAL_STORAGE_GET_F_CREATE to specify
		3545	* the initial value of a bpf_local_storage. If value is
		3546	* NULL, the new bpf_local_storage will be zero initialized.
		3547	* Return
		3548	* A bpf_local_storage pointer is returned on success.
		3549	*
		3550	* NULL if not found or there was an error in adding
		3551	* a new bpf_local_storage.
		3552	*
		3553	* int bpf_inode_storage_delete(struct bpf_map map, void inode)
		3554	* Description
		3555	* Delete a bpf_local_storage from an inode.
		3556	* Return
		3557	* 0 on success.
		3558	*
		3559	* -ENOENT if the bpf_local_storage cannot be found.
		3560	*
		3561	* long bpf_d_path(struct path path, char buf, u32 sz)
		3562	* Description
		3563	* Return full path for given 'struct path' object, which
		3564	* needs to be the kernel BTF 'path' object. The path is
		3565	* returned in the provided buffer 'buf' of size 'sz' and
		3566	* is zero terminated.
		3567	*
		3568	* Return
		3569	* On success, the strictly positive length of the string,
		3570	* including the trailing NUL character. On error, a negative
		3571	* value.
		3572	*
		3573	* long bpf_copy_from_user(void dst, u32 size, const void user_ptr)
		3574	* Description
		3575	* Read size bytes from user space address user_ptr and store
		3576	* the data in dst. This is a wrapper of copy_from_user().
		3577	* Return
		3578	* 0 on success, or a negative error in case of failure.
3398	*/	3579	*/
3399	#define __BPF_FUNC_MAPPER(FN) \	3580	#define __BPF_FUNC_MAPPER(FN) \
3400	FN(unspec), \	3581	FN(unspec), \
@@ -3539,6 +3720,13 @@ union bpf_attr { · 138 unmodified lines
3539	FN(skc_to_tcp_request_sock), \	3720	FN(skc_to_tcp_request_sock), \
3540	FN(skc_to_udp6_sock), \	3721	FN(skc_to_udp6_sock), \
3541	FN(get_task_stack), \	3722	FN(get_task_stack), \
		3723	FN(load_hdr_opt), \
		3724	FN(store_hdr_opt), \
		3725	FN(reserve_hdr_opt), \
		3726	FN(inode_storage_get), \
		3727	FN(inode_storage_delete), \
		3728	FN(d_path), \
		3729	FN(copy_from_user), \
3542	/* */	3730	/* */
3543		3731
3544	/* integer value in 'imm' field of BPF_CALL instruction selects which helper	3732	/* integer value in 'imm' field of BPF_CALL instruction selects which helper
@@ -3648,9 +3836,13 @@ enum { · 103 unmodified lines
3648	BPF_F_SYSCTL_BASE_NAME = (1ULL << 0),	3836	BPF_F_SYSCTL_BASE_NAME = (1ULL << 0),
3649	};	3837	};
3650		3838
3651	/* BPF_FUNC_sk_storage_get flags */	3839	/* BPF_FUNC_<kernel_obj>_storage_get flags */
3652	enum {	3840	enum {
3653	BPF_SK_STORAGE_GET_F_CREATE = (1ULL << 0),	3841	BPF_LOCAL_STORAGE_GET_F_CREATE = (1ULL << 0),
		3842	/* BPF_SK_STORAGE_GET_F_CREATE is only kept for backward compatibility
		3843	* and BPF_LOCAL_STORAGE_GET_F_CREATE must be used instead.
		3844	*/
		3845	BPF_SK_STORAGE_GET_F_CREATE = BPF_LOCAL_STORAGE_GET_F_CREATE,
3654	};	3846	};
3655		3847
3656	/* BPF_FUNC_read_branch_records flags. */	3848	/* BPF_FUNC_read_branch_records flags. */
@@ -4071,6 +4263,15 @@ struct bpf_link_info { · 414 unmodified lines
4071	__u64 cgroup_id;	4263	__u64 cgroup_id;
4072	__u32 attach_type;	4264	__u32 attach_type;
4073	} cgroup;	4265	} cgroup;
		4266	struct {
		4267	__aligned_u64 target_name; /* in/out: target_name buffer ptr */
		4268	__u32 target_name_len; /* in/out: target_name buffer len */
		4269	union {
		4270	struct {
		4271	__u32 map_id;
		4272	} map;
		4273	};
		4274	} iter;
4074	struct {	4275	struct {
4075	__u32 netns_ino;	4276	__u32 netns_ino;
4076	__u32 attach_type;	4277	__u32 attach_type;
@@ -4158,6 +4359,36 @@ struct bpf_sock_ops { · 81 unmodified lines
4158	__u64 bytes_received;	4359	__u64 bytes_received;
4159	__u64 bytes_acked;	4360	__u64 bytes_acked;
4160	__bpf_md_ptr(struct bpf_sock *, sk);	4361	__bpf_md_ptr(struct bpf_sock *, sk);
		4362	/* [skb_data, skb_data_end) covers the whole TCP header.
		4363	*
		4364	* BPF_SOCK_OPS_PARSE_HDR_OPT_CB: The packet received
		4365	* BPF_SOCK_OPS_HDR_OPT_LEN_CB: Not useful because the
		4366	* header has not been written.
		4367	* BPF_SOCK_OPS_WRITE_HDR_OPT_CB: The header and options have
		4368	* been written so far.
		4369	* BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB: The SYNACK that concludes
		4370	* the 3WHS.
		4371	* BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB: The ACK that concludes
		4372	* the 3WHS.
		4373	*
		4374	* bpf_load_hdr_opt() can also be used to read a particular option.
		4375	*/
		4376	__bpf_md_ptr(void *, skb_data);
		4377	__bpf_md_ptr(void *, skb_data_end);
		4378	__u32 skb_len; /* The total length of a packet.
		4379	* It includes the header, options,
		4380	* and payload.
		4381	*/
		4382	__u32 skb_tcp_flags; /* tcp_flags of the header. It provides
		4383	* an easy way to check for tcp_flags
		4384	* without parsing skb_data.
		4385	*
		4386	* In particular, the skb_tcp_flags
		4387	* will still be available in
		4388	* BPF_SOCK_OPS_HDR_OPT_LEN even though
		4389	* the outgoing header has not
		4390	* been written yet.
		4391	*/
4161	};	4392	};
4162		4393
4163	/* Definitions for bpf_sock_ops_cb_flags */	4394	/* Definitions for bpf_sock_ops_cb_flags */
@@ -4166,8 +4397,51 @@ enum { · 2 unmodified lines
4166	BPF_SOCK_OPS_RETRANS_CB_FLAG = (1<<1),	4397	BPF_SOCK_OPS_RETRANS_CB_FLAG = (1<<1),
4167	BPF_SOCK_OPS_STATE_CB_FLAG = (1<<2),	4398	BPF_SOCK_OPS_STATE_CB_FLAG = (1<<2),
4168	BPF_SOCK_OPS_RTT_CB_FLAG = (1<<3),	4399	BPF_SOCK_OPS_RTT_CB_FLAG = (1<<3),
		4400	/* Call bpf for all received TCP headers. The bpf prog will be
		4401	* called under sock_ops->op == BPF_SOCK_OPS_PARSE_HDR_OPT_CB
		4402	*
		4403	* Please refer to the comment in BPF_SOCK_OPS_PARSE_HDR_OPT_CB
		4404	* for the header option related helpers that will be useful
		4405	* to the bpf programs.
		4406	*
		4407	* It could be used at the client/active side (i.e. connect() side)
		4408	* when the server told it that the server was in syncookie
		4409	* mode and required the active side to resend the bpf-written
		4410	* options. The active side can keep writing the bpf-options until
		4411	* it received a valid packet from the server side to confirm
		4412	* the earlier packet (and options) has been received. The later
		4413	* example patch is using it like this at the active side when the
		4414	* server is in syncookie mode.
		4415	*
		4416	* The bpf prog will usually turn this off in the common cases.
		4417	*/
		4418	BPF_SOCK_OPS_PARSE_ALL_HDR_OPT_CB_FLAG = (1<<4),
		4419	/* Call bpf when kernel has received a header option that
		4420	* the kernel cannot handle. The bpf prog will be called under
		4421	* sock_ops->op == BPF_SOCK_OPS_PARSE_HDR_OPT_CB.
		4422	*
		4423	* Please refer to the comment in BPF_SOCK_OPS_PARSE_HDR_OPT_CB
		4424	* for the header option related helpers that will be useful
		4425	* to the bpf programs.
		4426	*/
		4427	BPF_SOCK_OPS_PARSE_UNKNOWN_HDR_OPT_CB_FLAG = (1<<5),
		4428	/* Call bpf when the kernel is writing header options for the
		4429	* outgoing packet. The bpf prog will first be called
		4430	* to reserve space in a skb under
		4431	* sock_ops->op == BPF_SOCK_OPS_HDR_OPT_LEN_CB. Then
		4432	* the bpf prog will be called to write the header option(s)
		4433	* under sock_ops->op == BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
		4434	*
		4435	* Please refer to the comment in BPF_SOCK_OPS_HDR_OPT_LEN_CB
		4436	* and BPF_SOCK_OPS_WRITE_HDR_OPT_CB for the header option
		4437	* related helpers that will be useful to the bpf programs.
		4438	*
		4439	* The kernel gets its chance to reserve space and write
		4440	* options first before the BPF program does.
		4441	*/
		4442	BPF_SOCK_OPS_WRITE_HDR_OPT_CB_FLAG = (1<<6),
4169	/* Mask of all currently supported cb flags */	4443	/* Mask of all currently supported cb flags */
4170	BPF_SOCK_OPS_ALL_CB_FLAGS = 0xF,	4444	BPF_SOCK_OPS_ALL_CB_FLAGS = 0x7F,
4171	};	4445	};
4172		4446
4173	/* List of known BPF sock_ops operators.	4447	/* List of known BPF sock_ops operators.
@@ -4223,6 +4497,63 @@ enum { · 49 unmodified lines
4223	*/	4497	*/
4224	BPF_SOCK_OPS_RTT_CB, /* Called on every RTT.	4498	BPF_SOCK_OPS_RTT_CB, /* Called on every RTT.
4225	*/	4499	*/
		4500	BPF_SOCK_OPS_PARSE_HDR_OPT_CB, /* Parse the header option.
		4501	* It will be called to handle
		4502	* the packets received at
		4503	* an already established
		4504	* connection.
		4505	*
		4506	* sock_ops->skb_data:
		4507	* Referring to the received skb.
		4508	* It covers the TCP header only.
		4509	*
		4510	* bpf_load_hdr_opt() can also
		4511	* be used to search for a
		4512	* particular option.
		4513	*/
		4514	BPF_SOCK_OPS_HDR_OPT_LEN_CB, /* Reserve space for writing the
		4515	* header option later in
		4516	* BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
		4517	* Arg1: bool want_cookie. (in
		4518	* writing SYNACK only)
		4519	*
		4520	* sock_ops->skb_data:
		4521	* Not available because no header has
		4522	* been written yet.
		4523	*
		4524	* sock_ops->skb_tcp_flags:
		4525	* The tcp_flags of the
		4526	* outgoing skb. (e.g. SYN, ACK, FIN).
		4527	*
		4528	* bpf_reserve_hdr_opt() should
		4529	* be used to reserve space.
		4530	*/
		4531	BPF_SOCK_OPS_WRITE_HDR_OPT_CB, /* Write the header options
		4532	* Arg1: bool want_cookie. (in
		4533	* writing SYNACK only)
		4534	*
		4535	* sock_ops->skb_data:
		4536	* Referring to the outgoing skb.
		4537	* It covers the TCP header
		4538	* that has already been written
		4539	* by the kernel and the
		4540	* earlier bpf-progs.
		4541	*
		4542	* sock_ops->skb_tcp_flags:
		4543	* The tcp_flags of the outgoing
		4544	* skb. (e.g. SYN, ACK, FIN).
		4545	*
		4546	* bpf_store_hdr_opt() should
		4547	* be used to write the
		4548	* option.
		4549	*
		4550	* bpf_load_hdr_opt() can also
		4551	* be used to search for a
		4552	* particular option that
		4553	* has already been written
		4554	* by the kernel or the
		4555	* earlier bpf-progs.
		4556	*/
4226	};	4557	};
4227		4558
4228	/* List of TCP states. There is a build check in net/ipv4/tcp.c to detect	4559	/* List of TCP states. There is a build check in net/ipv4/tcp.c to detect
@@ -4250,6 +4581,63 @@ enum { · 21 unmodified lines
4250	enum {	4581	enum {
4251	TCP_BPF_IW = 1001, /* Set TCP initial congestion window */	4582	TCP_BPF_IW = 1001, /* Set TCP initial congestion window */
4252	TCP_BPF_SNDCWND_CLAMP = 1002, /* Set sndcwnd_clamp */	4583	TCP_BPF_SNDCWND_CLAMP = 1002, /* Set sndcwnd_clamp */
		4584	TCP_BPF_DELACK_MAX = 1003, /* Max delay ack in usecs */
		4585	TCP_BPF_RTO_MIN = 1004, /* Min delay ack in usecs */
		4586	/* Copy the SYN pkt to optval
		4587	*
		4588	* BPF_PROG_TYPE_SOCK_OPS only. It is similar to the
		4589	* bpf_getsockopt(TCP_SAVED_SYN) but it does not limit
		4590	* to only getting from the saved_syn. It can either get the
		4591	* syn packet from:
		4592	*
		4593	* 1. the just-received SYN packet (only available when writing the
		4594	* SYNACK). It will be useful when it is not necessary to
		4595	* save the SYN packet for latter use. It is also the only way
		4596	* to get the SYN during syncookie mode because the syn
		4597	* packet cannot be saved during syncookie.
		4598	*
		4599	* OR
		4600	*
		4601	* 2. the earlier saved syn which was done by
		4602	* bpf_setsockopt(TCP_SAVE_SYN).
		4603	*
		4604	* The bpf_getsockopt(TCP_BPF_SYN*) option will hide where the
		4605	* SYN packet is obtained.
		4606	*
		4607	* If the bpf-prog does not need the IP[46] header, the
		4608	* bpf-prog can avoid parsing the IP header by using
		4609	* TCP_BPF_SYN. Otherwise, the bpf-prog can get both
		4610	* IP[46] and TCP header by using TCP_BPF_SYN_IP.
		4611	*
		4612	* >0: Total number of bytes copied
		4613	* -ENOSPC: Not enough space in optval. Only optlen number of
		4614	* bytes is copied.
		4615	* -ENOENT: The SYN skb is not available now and the earlier SYN pkt
		4616	* is not saved by setsockopt(TCP_SAVE_SYN).
		4617	*/
		4618	TCP_BPF_SYN = 1005, /* Copy the TCP header */
		4619	TCP_BPF_SYN_IP = 1006, /* Copy the IP[46] and TCP header */
		4620	TCP_BPF_SYN_MAC = 1007, /* Copy the MAC, IP[46], and TCP header */
		4621	};
		4622
		4623	enum {
		4624	BPF_LOAD_HDR_OPT_TCP_SYN = (1ULL << 0),
		4625	};
		4626
		4627	/* args[0] value during BPF_SOCK_OPS_HDR_OPT_LEN_CB and
		4628	* BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
		4629	*/
		4630	enum {
		4631	BPF_WRITE_HDR_TCP_CURRENT_MSS = 1, /* Kernel is finding the
		4632	* total option spaces
		4633	* required for an established
		4634	* sk in order to calculate the
		4635	* MSS. No skb is actually
		4636	* sent.
		4637	*/
		4638	BPF_WRITE_HDR_TCP_SYNACK_COOKIE = 2, /* Kernel is in syncookie mode
		4639	* when sending a SYN.
		4640	*/
4253	};	4641	};
4254		4642
4255	struct bpf_perf_event_value {	4643	struct bpf_perf_event_value {

tools/lib/bpf/Makefile+9−14

 # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
 # Most of this file is copied from tools/lib/traceevent/Makefile
+RM ?= rm
+srctree = $(abs_srctree)
 LIBBPF_VERSION := $(shell \
 	grep -oE '^LIBBPF_([0-9.]+)' libbpf.map | \
 	sort -rV | head -n1 | cut -d'_' -f2)
 endif
 FEATURE_USER = .libbpf
-FEATURE_TESTS = libelf libelf-mmap zlib bpf reallocarray
+FEATURE_TESTS = libelf zlib bpf
 FEATURE_DISPLAY = libelf zlib bpf
 INCLUDES = -I. -I$(srctree)/tools/include -I$(srctree)/tools/arch/$(ARCH)/include/uapi -I$(srctree)/tools/include/uapi
   CFLAGS := -g -Wall
 endif
-ifeq ($(feature-libelf-mmap), 1)
-  override CFLAGS += -DHAVE_LIBELF_MMAP_SUPPORT
-endif
-ifeq ($(feature-reallocarray), 0)
-  override CFLAGS += -DCOMPAT_NEED_REALLOCARRAY
-endif
 # Append required CFLAGS
-override CFLAGS += $(EXTRA_WARNINGS)
+override CFLAGS += $(EXTRA_WARNINGS) -Wno-switch-enum
 override CFLAGS += -Werror -Wall
 override CFLAGS += -fPIC
 override CFLAGS += $(INCLUDES)
 	@ln -sf $(@F) $(OUTPUT)libbpf.so.$(LIBBPF_MAJOR_VERSION)
 $(OUTPUT)libbpf.a: $(BPF_IN_STATIC)
-	$(QUIET_LINK)$(RM) $@; $(AR) rcs $@ $^
+	$(QUIET_LINK)$(RM) -f $@; $(AR) rcs $@ $^
 $(OUTPUT)libbpf.pc:
 	$(QUIET_GEN)sed -e "s|@PREFIX@|$(prefix)|" \
 ### Cleaning rules
 config-clean:
-	$(call QUIET_CLEAN, config)
+	$(call QUIET_CLEAN, feature-detect)
 	$(Q)$(MAKE) -C $(srctree)/tools/build/feature/ clean >/dev/null
-clean:
+clean: config-clean
 	$(call QUIET_CLEAN, libbpf) $(RM) -rf $(CMD_TARGETS)		     \
 		*~ .*.d .*.cmd LIBBPF-CFLAGS $(BPF_HELPER_DEFS)		     \
 		$(SHARED_OBJDIR) $(STATIC_OBJDIR)			     \
 	cscope -b -q -I $(srctree)/include -f cscope.out
 tags:
-	rm -f TAGS tags
+	$(RM) -f TAGS tags
 	ls *.c *.h | xargs $(TAGS_PROG) -a
 # Declare the contents of the .PHONY variable as phony.  We keep that

tools/lib/bpf/bpf.c+0−3

@@ -32,9 +32,6 @@ · 31 unmodified lines
32	#include "libbpf.h"	32	#include "libbpf.h"
33	#include "libbpf_internal.h"	33	#include "libbpf_internal.h"
34		34
35	/* make sure libbpf doesn't use kernel-only integer typedefs */
36	#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
37
38	/*	35	/*
39	* When building perf, unistd.h is overridden. __NR_bpf is	36	* When building perf, unistd.h is overridden. __NR_bpf is
40	* required to be defined explicitly.	37	* required to be defined explicitly.

tools/lib/bpf/bpf_core_read.h+101−19

@@ -19,32 +19,52 @@ enum bpf_field_info_kind { · 18 unmodified lines
19	BPF_FIELD_RSHIFT_U64 = 5,	19	BPF_FIELD_RSHIFT_U64 = 5,
20	};	20	};
21		21
		22	/* second argument to __builtin_btf_type_id() built-in */
		23	enum bpf_type_id_kind {
		24	BPF_TYPE_ID_LOCAL = 0, /* BTF type ID in local program */
		25	BPF_TYPE_ID_TARGET = 1, /* BTF type ID in target kernel */
		26	};
		27
		28	/* second argument to __builtin_preserve_type_info() built-in */
		29	enum bpf_type_info_kind {
		30	BPF_TYPE_EXISTS = 0, /* type existence in target kernel */
		31	BPF_TYPE_SIZE = 1, /* type size in target kernel */
		32	};
		33
		34	/* second argument to __builtin_preserve_enum_value() built-in */
		35	enum bpf_enum_value_kind {
		36	BPF_ENUMVAL_EXISTS = 0, /* enum value existence in kernel */
		37	BPF_ENUMVAL_VALUE = 1, /* enum value value relocation */
		38	};
		39
22	#define __CORE_RELO(src, field, info) \	40	#define __CORE_RELO(src, field, info) \
23	__builtin_preserve_field_info((src)->field, BPF_FIELD_##info)	41	__builtin_preserve_field_info((src)->field, BPF_FIELD_##info)
24		42
25	#if __BYTE_ORDER == __LITTLE_ENDIAN	43	#if __BYTE_ORDER == __LITTLE_ENDIAN
26	#define __CORE_BITFIELD_PROBE_READ(dst, src, fld) \	44	#define __CORE_BITFIELD_PROBE_READ(dst, src, fld) \
27	bpf_probe_read((void *)dst, \	45	bpf_probe_read_kernel( \
28	__CORE_RELO(src, fld, BYTE_SIZE), \	46	(void *)dst, \
29	(const void *)src + __CORE_RELO(src, fld, BYTE_OFFSET))	47	__CORE_RELO(src, fld, BYTE_SIZE), \
		48	(const void *)src + __CORE_RELO(src, fld, BYTE_OFFSET))
30	#else	49	#else
31	/* semantics of LSHIFT_64 assumes loading values into low-ordered bytes, so	50	/* semantics of LSHIFT_64 assumes loading values into low-ordered bytes, so
32	* for big-endian we need to adjust destination pointer accordingly, based on	51	* for big-endian we need to adjust destination pointer accordingly, based on
33	* field byte size	52	* field byte size
34	*/	53	*/
35	#define __CORE_BITFIELD_PROBE_READ(dst, src, fld) \	54	#define __CORE_BITFIELD_PROBE_READ(dst, src, fld) \
36	bpf_probe_read((void *)dst + (8 - __CORE_RELO(src, fld, BYTE_SIZE)), \	55	bpf_probe_read_kernel( \
37	__CORE_RELO(src, fld, BYTE_SIZE), \	56	(void *)dst + (8 - __CORE_RELO(src, fld, BYTE_SIZE)), \
38	(const void *)src + __CORE_RELO(src, fld, BYTE_OFFSET))	57	__CORE_RELO(src, fld, BYTE_SIZE), \
		58	(const void *)src + __CORE_RELO(src, fld, BYTE_OFFSET))
39	#endif	59	#endif
40		60
41	/*	61	/*
42	* Extract bitfield, identified by s->field, and return its value as u64.	62	* Extract bitfield, identified by s->field, and return its value as u64.
43	* All this is done in relocatable manner, so bitfield changes such as	63	* All this is done in relocatable manner, so bitfield changes such as
44	* signedness, bit size, offset changes, this will be handled automatically.	64	* signedness, bit size, offset changes, this will be handled automatically.
45	* This version of macro is using bpf_probe_read() to read underlying integer	65	* This version of macro is using bpf_probe_read_kernel() to read underlying
46	* storage. Macro functions as an expression and its return type is	66	* integer storage. Macro functions as an expression and its return type is
47	* bpf_probe_read()'s return value: 0, on success, <0 on error.	67	* bpf_probe_read_kernel()'s return value: 0, on success, <0 on error.
48	*/	68	*/
49	#define BPF_CORE_READ_BITFIELD_PROBED(s, field) ({ \	69	#define BPF_CORE_READ_BITFIELD_PROBED(s, field) ({ \
50	unsigned long long val = 0; \	70	unsigned long long val = 0; \
@@ -92,15 +112,75 @@ enum bpf_field_info_kind { · 41 unmodified lines
92	__builtin_preserve_field_info(field, BPF_FIELD_EXISTS)	112	__builtin_preserve_field_info(field, BPF_FIELD_EXISTS)
93		113
94	/*	114	/*
95	* Convenience macro to get byte size of a field. Works for integers,	115	* Convenience macro to get the byte size of a field. Works for integers,
96	* struct/unions, pointers, arrays, and enums.	116	* struct/unions, pointers, arrays, and enums.
97	*/	117	*/
98	#define bpf_core_field_size(field) \	118	#define bpf_core_field_size(field) \
99	__builtin_preserve_field_info(field, BPF_FIELD_BYTE_SIZE)	119	__builtin_preserve_field_info(field, BPF_FIELD_BYTE_SIZE)
100		120
101	/*	121	/*
102	* bpf_core_read() abstracts away bpf_probe_read() call and captures offset	122	* Convenience macro to get BTF type ID of a specified type, using a local BTF
103	* relocation for source address using __builtin_preserve_access_index()	123	* information. Return 32-bit unsigned integer with type ID from program's own
		124	* BTF. Always succeeds.
		125	*/
		126	#define bpf_core_type_id_local(type) \
		127	__builtin_btf_type_id((typeof(type) )0, BPF_TYPE_ID_LOCAL)
		128
		129	/*
		130	* Convenience macro to get BTF type ID of a target kernel's type that matches
		131	* specified local type.
		132	* Returns:
		133	* - valid 32-bit unsigned type ID in kernel BTF;
		134	* - 0, if no matching type was found in a target kernel BTF.
		135	*/
		136	#define bpf_core_type_id_kernel(type) \
		137	__builtin_btf_type_id((typeof(type) )0, BPF_TYPE_ID_TARGET)
		138
		139	/*
		140	* Convenience macro to check that provided named type
		141	* (struct/union/enum/typedef) exists in a target kernel.
		142	* Returns:
		143	* 1, if such type is present in target kernel's BTF;
		144	* 0, if no matching type is found.
		145	*/
		146	#define bpf_core_type_exists(type) \
		147	__builtin_preserve_type_info((typeof(type) )0, BPF_TYPE_EXISTS)
		148
		149	/*
		150	* Convenience macro to get the byte size of a provided named type
		151	* (struct/union/enum/typedef) in a target kernel.
		152	* Returns:
		153	* >= 0 size (in bytes), if type is present in target kernel's BTF;
		154	* 0, if no matching type is found.
		155	*/
		156	#define bpf_core_type_size(type) \
		157	__builtin_preserve_type_info((typeof(type) )0, BPF_TYPE_SIZE)
		158
		159	/*
		160	* Convenience macro to check that provided enumerator value is defined in
		161	* a target kernel.
		162	* Returns:
		163	* 1, if specified enum type and its enumerator value are present in target
		164	* kernel's BTF;
		165	* 0, if no matching enum and/or enum value within that enum is found.
		166	*/
		167	#define bpf_core_enum_value_exists(enum_type, enum_value) \
		168	__builtin_preserve_enum_value((typeof(enum_type) )enum_value, BPF_ENUMVAL_EXISTS)
		169
		170	/*
		171	* Convenience macro to get the integer value of an enumerator value in
		172	* a target kernel.
		173	* Returns:
		174	* 64-bit value, if specified enum type and its enumerator value are
		175	* present in target kernel's BTF;
		176	* 0, if no matching enum and/or enum value within that enum is found.
		177	*/
		178	#define bpf_core_enum_value(enum_type, enum_value) \
		179	__builtin_preserve_enum_value((typeof(enum_type) )enum_value, BPF_ENUMVAL_VALUE)
		180
		181	/*
		182	* bpf_core_read() abstracts away bpf_probe_read_kernel() call and captures
		183	* offset relocation for source address using __builtin_preserve_access_index()
104	* built-in, provided by Clang.	184	* built-in, provided by Clang.
105	*	185	*
106	* __builtin_preserve_access_index() takes as an argument an expression of	186	* __builtin_preserve_access_index() takes as an argument an expression of
@@ -115,8 +195,8 @@ enum bpf_field_info_kind { · 8 unmodified lines
115	* (local) BTF, used to record relocation.	195	* (local) BTF, used to record relocation.
116	*/	196	*/
117	#define bpf_core_read(dst, sz, src) \	197	#define bpf_core_read(dst, sz, src) \
118	bpf_probe_read(dst, sz, \	198	bpf_probe_read_kernel(dst, sz, \
119	(const void *)__builtin_preserve_access_index(src))	199	(const void *)__builtin_preserve_access_index(src))
120		200
121	/*	201	/*
122	* bpf_core_read_str() is a thin wrapper around bpf_probe_read_str()	202	* bpf_core_read_str() is a thin wrapper around bpf_probe_read_str()
@@ -124,8 +204,8 @@ enum bpf_field_info_kind { · 1 unmodified line
124	* argument.	204	* argument.
125	*/	205	*/
126	#define bpf_core_read_str(dst, sz, src) \	206	#define bpf_core_read_str(dst, sz, src) \
127	bpf_probe_read_str(dst, sz, \	207	bpf_probe_read_kernel_str(dst, sz, \
128	(const void *)__builtin_preserve_access_index(src))	208	(const void *)__builtin_preserve_access_index(src))
129		209
130	#define ___concat(a, b) a ## b	210	#define ___concat(a, b) a ## b
131	#define ___apply(fn, n) ___concat(fn, n)	211	#define ___apply(fn, n) ___concat(fn, n)
@@ -239,15 +319,17 @@ enum bpf_field_info_kind { · 107 unmodified lines
239	* int x = BPF_CORE_READ(s, a.b.c, d.e, f, g);	319	* int x = BPF_CORE_READ(s, a.b.c, d.e, f, g);
240	*	320	*
241	* BPF_CORE_READ will decompose above statement into 4 bpf_core_read (BPF	321	* BPF_CORE_READ will decompose above statement into 4 bpf_core_read (BPF
242	* CO-RE relocatable bpf_probe_read() wrapper) calls, logically equivalent to:	322	* CO-RE relocatable bpf_probe_read_kernel() wrapper) calls, logically
		323	* equivalent to:
243	* 1. const void *__t = s->a.b.c;	324	* 1. const void *__t = s->a.b.c;
244	* 2. __t = __t->d.e;	325	* 2. __t = __t->d.e;
245	* 3. __t = __t->f;	326	* 3. __t = __t->f;
246	* 4. return __t->g;	327	* 4. return __t->g;
247	*	328	*
248	* Equivalence is logical, because there is a heavy type casting/preservation	329	* Equivalence is logical, because there is a heavy type casting/preservation
249	* involved, as well as all the reads are happening through bpf_probe_read()	330	* involved, as well as all the reads are happening through
250	* calls using __builtin_preserve_access_index() to emit CO-RE relocations.	331	* bpf_probe_read_kernel() calls using __builtin_preserve_access_index() to
		332	* emit CO-RE relocations.
251	*	333	*
252	* N.B. Only up to 9 "field accessors" are supported, which should be more	334	* N.B. Only up to 9 "field accessors" are supported, which should be more
253	* than enough for any practical purpose.	335	* than enough for any practical purpose.

tools/lib/bpf/bpf_helpers.h+3−0

 #ifndef __always_inline
 #define __always_inline __attribute__((always_inline))
 #endif
+#ifndef __noinline
+#define __noinline __attribute__((noinline))
+#endif
 #ifndef __weak
 #define __weak __attribute__((weak))
 #endif

tools/lib/bpf/bpf_prog_linfo.c+0−3

@@ -8,9 +8,6 @@ · 7 unmodified lines
8	#include "libbpf.h"	8	#include "libbpf.h"
9	#include "libbpf_internal.h"	9	#include "libbpf_internal.h"
10		10
11	/* make sure libbpf doesn't use kernel-only integer typedefs */
12	#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
13
14	struct bpf_prog_linfo {	11	struct bpf_prog_linfo {
15	void *raw_linfo;	12	void *raw_linfo;
16	void *raw_jited_linfo;	13	void *raw_jited_linfo;

tools/lib/bpf/bpf_tracing.h+2−2

@@ -289,9 +289,9 @@ struct pt_regs; · 288 unmodified lines
289	#define BPF_KRETPROBE_READ_RET_IP BPF_KPROBE_READ_RET_IP	289	#define BPF_KRETPROBE_READ_RET_IP BPF_KPROBE_READ_RET_IP
290	#else	290	#else
291	#define BPF_KPROBE_READ_RET_IP(ip, ctx) \	291	#define BPF_KPROBE_READ_RET_IP(ip, ctx) \
292	({ bpf_probe_read(&(ip), sizeof(ip), (void *)PT_REGS_RET(ctx)); })	292	({ bpf_probe_read_kernel(&(ip), sizeof(ip), (void *)PT_REGS_RET(ctx)); })
293	#define BPF_KRETPROBE_READ_RET_IP(ip, ctx) \	293	#define BPF_KRETPROBE_READ_RET_IP(ip, ctx) \
294	({ bpf_probe_read(&(ip), sizeof(ip), \	294	({ bpf_probe_read_kernel(&(ip), sizeof(ip), \
295	(void *)(PT_REGS_FP(ctx) + sizeof(ip))); })	295	(void *)(PT_REGS_FP(ctx) + sizeof(ip))); })
296	#endif	296	#endif
297		297

tools/lib/bpf/btf.c+13−18

 #include "libbpf_internal.h"
 #include "hashmap.h"
-/* make sure libbpf doesn't use kernel-only integer typedefs */
-#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
 #define BTF_MAX_NR_TYPES 0x7fffffffU
 #define BTF_MAX_STR_OFFSET 0x7fffffffU
 		expand_by = max(btf->types_size >> 2, 16U);
 		new_size = min(BTF_MAX_NR_TYPES, btf->types_size + expand_by);
-		new_types = realloc(btf->types, sizeof(*new_types) * new_size);
+		new_types = libbpf_reallocarray(btf->types, new_size, sizeof(*new_types));
 		if (!new_types)
 			return -ENOMEM;
 	return btf_ext_setup_info(btf_ext, &param);
 }
-static int btf_ext_setup_field_reloc(struct btf_ext *btf_ext)
+static int btf_ext_setup_core_relos(struct btf_ext *btf_ext)
 {
 	struct btf_ext_sec_setup_param param = {
-		.off = btf_ext->hdr->field_reloc_off,
+		.off = btf_ext->hdr->core_relo_off,
-		.len = btf_ext->hdr->field_reloc_len,
+		.len = btf_ext->hdr->core_relo_len,
-		.min_rec_size = sizeof(struct bpf_field_reloc),
+		.min_rec_size = sizeof(struct bpf_core_relo),
-		.ext_info = &btf_ext->field_reloc_info,
+		.ext_info = &btf_ext->core_relo_info,
-		.desc = "field_reloc",
+		.desc = "core_relo",
 	};
 	return btf_ext_setup_info(btf_ext, &param);
 	if (err)
 		goto done;
-	if (btf_ext->hdr->hdr_len <
+	if (btf_ext->hdr->hdr_len < offsetofend(struct btf_ext_header, core_relo_len))
-	    offsetofend(struct btf_ext_header, field_reloc_len))
 		goto done;
-	err = btf_ext_setup_field_reloc(btf_ext);
+	err = btf_ext_setup_core_relos(btf_ext);
 	if (err)
 		goto done;
 		__u32 *new_list;
 		d->hypot_cap += max((size_t)16, d->hypot_cap / 2);
-		new_list = realloc(d->hypot_list, sizeof(__u32) * d->hypot_cap);
+		new_list = libbpf_reallocarray(d->hypot_list, d->hypot_cap, sizeof(__u32));
 		if (!new_list)
 			return -ENOMEM;
 		d->hypot_list = new_list;
 			struct btf_str_ptr *new_ptrs;
 			strs.cap += max(strs.cnt / 2, 16U);
-			new_ptrs = realloc(strs.ptrs,
+			new_ptrs = libbpf_reallocarray(strs.ptrs, strs.cap, sizeof(strs.ptrs[0]));
-					   sizeof(strs.ptrs[0]) * strs.cap);
 			if (!new_ptrs) {
 				err = -ENOMEM;
 				goto done;
 	d->btf->nr_types = next_type_id - 1;
 	d->btf->types_size = d->btf->nr_types;
 	d->btf->hdr->type_len = p - types_start;
-	new_types = realloc(d->btf->types,
+	new_types = libbpf_reallocarray(d->btf->types, (1 + d->btf->nr_types),
-			    (1 + d->btf->nr_types) * sizeof(struct btf_type *));
+					sizeof(struct btf_type *));
 	if (!new_types)
 		return -ENOMEM;
 	d->btf->types = new_types;

tools/lib/bpf/btf.h+0−38

@@ -24,44 +24,6 @@ struct btf_type; · 23 unmodified lines
24		24
25	struct bpf_object;	25	struct bpf_object;
26		26
27	/*
28	* The .BTF.ext ELF section layout defined as
29	* struct btf_ext_header
30	* func_info subsection
31	*
32	* The func_info subsection layout:
33	* record size for struct bpf_func_info in the func_info subsection
34	* struct btf_sec_func_info for section #1
35	* a list of bpf_func_info records for section #1
36	* where struct bpf_func_info mimics one in include/uapi/linux/bpf.h
37	* but may not be identical
38	* struct btf_sec_func_info for section #2
39	* a list of bpf_func_info records for section #2
40	* ......
41	*
42	* Note that the bpf_func_info record size in .BTF.ext may not
43	* be the same as the one defined in include/uapi/linux/bpf.h.
44	* The loader should ensure that record_size meets minimum
45	* requirement and pass the record as is to the kernel. The
46	* kernel will handle the func_info properly based on its contents.
47	*/
48	struct btf_ext_header {
49	__u16 magic;
50	__u8 version;
51	__u8 flags;
52	__u32 hdr_len;
53
54	/* All offsets are in bytes relative to the end of this header */
55	__u32 func_info_off;
56	__u32 func_info_len;
57	__u32 line_info_off;
58	__u32 line_info_len;
59
60	/* optional part of .BTF.ext header */
61	__u32 field_reloc_off;
62	__u32 field_reloc_len;
63	};
64
65	LIBBPF_API void btf__free(struct btf *btf);	27	LIBBPF_API void btf__free(struct btf *btf);
66	LIBBPF_API struct btf btf__new(const void data, __u32 size);	28	LIBBPF_API struct btf btf__new(const void data, __u32 size);
67	LIBBPF_API struct btf btf__parse(const char path, struct btf_ext **btf_ext);	29	LIBBPF_API struct btf btf__parse(const char path, struct btf_ext **btf_ext);

tools/lib/bpf/btf_dump.c+2−7

 #include "libbpf.h"
 #include "libbpf_internal.h"
-/* make sure libbpf doesn't use kernel-only integer typedefs */
-#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
 static const char PREFIXES[] = "\t\t\t\t\t\t\t\t\t\t\t\t\t";
 static const size_t PREFIX_CNT = sizeof(PREFIXES) - 1;
 	if (d->emit_queue_cnt >= d->emit_queue_cap) {
 		new_cap = max(16, d->emit_queue_cap * 3 / 2);
-		new_queue = realloc(d->emit_queue,
+		new_queue = libbpf_reallocarray(d->emit_queue, new_cap, sizeof(new_queue[0]));
-				    new_cap * sizeof(new_queue[0]));
 		if (!new_queue)
 			return -ENOMEM;
 		d->emit_queue = new_queue;
 	if (d->decl_stack_cnt >= d->decl_stack_cap) {
 		new_cap = max(16, d->decl_stack_cap * 3 / 2);
-		new_stack = realloc(d->decl_stack,
+		new_stack = libbpf_reallocarray(d->decl_stack, new_cap, sizeof(new_stack[0]));
-				    new_cap * sizeof(new_stack[0]));
 		if (!new_stack)
 			return -ENOMEM;
 		d->decl_stack = new_stack;

tools/lib/bpf/hashmap.c+3−0

@@ -15,6 +15,9 @@ · 14 unmodified lines
15	/* make sure libbpf doesn't use kernel-only integer typedefs */	15	/* make sure libbpf doesn't use kernel-only integer typedefs */
16	#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64	16	#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
17		17
		18	/* prevent accidental re-addition of reallocarray() */
		19	#pragma GCC poison reallocarray
		20
18	/* start with 4 buckets */	21	/* start with 4 buckets */
19	#define HASHMAP_MIN_CAP_BITS 2	22	#define HASHMAP_MIN_CAP_BITS 2
20		23

tools/lib/bpf/libbpf.c+1242−517

@@ -44,7 +44,6 @@ · 43 unmodified lines
44	#include <sys/vfs.h>	44	#include <sys/vfs.h>
45	#include <sys/utsname.h>	45	#include <sys/utsname.h>
46	#include <sys/resource.h>	46	#include <sys/resource.h>
47	#include <tools/libc_compat.h>
48	#include <libelf.h>	47	#include <libelf.h>
49	#include <gelf.h>	48	#include <gelf.h>
50	#include <zlib.h>	49	#include <zlib.h>
@@ -56,9 +55,6 @@ · 5 unmodified lines
56	#include "libbpf_internal.h"	55	#include "libbpf_internal.h"
57	#include "hashmap.h"	56	#include "hashmap.h"
58		57
59	/* make sure libbpf doesn't use kernel-only integer typedefs */
60	#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
61
62	#ifndef EM_BPF	58	#ifndef EM_BPF
63	#define EM_BPF 247	59	#define EM_BPF 247
64	#endif	60	#endif
@@ -67,6 +63,8 @@ · 2 unmodified lines
67	#define BPF_FS_MAGIC 0xcafe4a11	63	#define BPF_FS_MAGIC 0xcafe4a11
68	#endif	64	#endif
69		65
		66	#define BPF_INSN_SZ (sizeof(struct bpf_insn))
		67
70	/* vsprintf() in __base_pr() uses nonliteral format string. It may break	68	/* vsprintf() in __base_pr() uses nonliteral format string. It may break
71	* compilation if user enables corresponding warning. Disable it explicitly.	69	* compilation if user enables corresponding warning. Disable it explicitly.
72	*/	70	*/
@@ -154,34 +152,35 @@ static void pr_perm_msg(int err) · 81 unmodified lines
154	___err; })	152	___err; })
155	#endif	153	#endif
156		154
157	#ifdef HAVE_LIBELF_MMAP_SUPPORT
158	# define LIBBPF_ELF_C_READ_MMAP ELF_C_READ_MMAP
159	#else
160	# define LIBBPF_ELF_C_READ_MMAP ELF_C_READ
161	#endif
162
163	static inline __u64 ptr_to_u64(const void *ptr)	155	static inline __u64 ptr_to_u64(const void *ptr)
164	{	156	{
165	return (__u64) (unsigned long) ptr;	157	return (__u64) (unsigned long) ptr;
166	}	158	}
167		159
168	struct bpf_capabilities {	160	enum kern_feature_id {
169	/* v4.14: kernel support for program & map names. */	161	/* v4.14: kernel support for program & map names. */
170	__u32 name:1;	162	FEAT_PROG_NAME,
171	/* v5.2: kernel support for global data sections. */	163	/* v5.2: kernel support for global data sections. */
172	__u32 global_data:1;	164	FEAT_GLOBAL_DATA,
		165	/* BTF support */
		166	FEAT_BTF,
173	/* BTF_KIND_FUNC and BTF_KIND_FUNC_PROTO support */	167	/* BTF_KIND_FUNC and BTF_KIND_FUNC_PROTO support */
174	__u32 btf_func:1;	168	FEAT_BTF_FUNC,
175	/* BTF_KIND_VAR and BTF_KIND_DATASEC support */	169	/* BTF_KIND_VAR and BTF_KIND_DATASEC support */
176	__u32 btf_datasec:1;	170	FEAT_BTF_DATASEC,
177	/* BPF_F_MMAPABLE is supported for arrays */
178	__u32 array_mmap:1;
179	/* BTF_FUNC_GLOBAL is supported */	171	/* BTF_FUNC_GLOBAL is supported */
180	__u32 btf_func_global:1;	172	FEAT_BTF_GLOBAL_FUNC,
		173	/* BPF_F_MMAPABLE is supported for arrays */
		174	FEAT_ARRAY_MMAP,
181	/* kernel support for expected_attach_type in BPF_PROG_LOAD */	175	/* kernel support for expected_attach_type in BPF_PROG_LOAD */
182	__u32 exp_attach_type:1;	176	FEAT_EXP_ATTACH_TYPE,
		177	/* bpf_probe_read_{kernel,user}[_str] helpers */
		178	FEAT_PROBE_READ_KERN,
		179	__FEAT_CNT,
183	};	180	};
184		181
		182	static bool kernel_supports(enum kern_feature_id feat_id);
		183
185	enum reloc_type {	184	enum reloc_type {
186	RELO_LD64,	185	RELO_LD64,
187	RELO_CALL,	186	RELO_CALL,
@@ -209,6 +208,7 @@ struct bpf_sec_def { · 21 unmodified lines
209	bool is_exp_attach_type_optional;	208	bool is_exp_attach_type_optional;
210	bool is_attachable;	209	bool is_attachable;
211	bool is_attach_btf;	210	bool is_attach_btf;
		211	bool is_sleepable;
212	attach_fn_t attach_fn;	212	attach_fn_t attach_fn;
213	};	213	};
214		214
@@ -253,8 +253,6 @@ struct bpf_program { · 38 unmodified lines
253	__u32 func_info_rec_size;	253	__u32 func_info_rec_size;
254	__u32 func_info_cnt;	254	__u32 func_info_cnt;
255		255
256	struct bpf_capabilities *caps;
257
258	void *line_info;	256	void *line_info;
259	__u32 line_info_rec_size;	257	__u32 line_info_rec_size;
260	__u32 line_info_cnt;	258	__u32 line_info_cnt;
@@ -403,6 +401,7 @@ struct bpf_object { · 142 unmodified lines
403	Elf_Data *rodata;	401	Elf_Data *rodata;
404	Elf_Data *bss;	402	Elf_Data *bss;
405	Elf_Data *st_ops_data;	403	Elf_Data *st_ops_data;
		404	size_t shstrndx; /* section index for section name strings */
406	size_t strtabidx;	405	size_t strtabidx;
407	struct {	406	struct {
408	GElf_Shdr shdr;	407	GElf_Shdr shdr;
@@ -436,12 +435,18 @@ struct bpf_object { · 27 unmodified lines
436	void *priv;	435	void *priv;
437	bpf_object_clear_priv_t clear_priv;	436	bpf_object_clear_priv_t clear_priv;
438		437
439	struct bpf_capabilities caps;
440
441	char path[];	438	char path[];
442	};	439	};
443	#define obj_elf_valid(o) ((o)->efile.elf)	440	#define obj_elf_valid(o) ((o)->efile.elf)
444		441
		442	static const char elf_sym_str(const struct bpf_object obj, size_t off);
		443	static const char elf_sec_str(const struct bpf_object obj, size_t off);
		444	static Elf_Scn elf_sec_by_idx(const struct bpf_object obj, size_t idx);
		445	static Elf_Scn elf_sec_by_name(const struct bpf_object obj, const char *name);
		446	static int elf_sec_hdr(const struct bpf_object obj, Elf_Scn scn, GElf_Shdr *hdr);
		447	static const char elf_sec_name(const struct bpf_object obj, Elf_Scn *scn);
		448	static Elf_Data elf_sec_data(const struct bpf_object obj, Elf_Scn *scn);
		449
445	void bpf_program__unload(struct bpf_program *prog)	450	void bpf_program__unload(struct bpf_program *prog)
446	{	451	{
447	int i;	452	int i;
@@ -503,7 +508,7 @@ static char __bpf_program__pin_name(struct bpf_program prog) · 55 unmodified lines
503	}	508	}
504		509
505	static int	510	static int
506	bpf_program__init(void data, size_t size, char section_name, int idx,	511	bpf_program__init(void data, size_t size, const char section_name, int idx,
507	struct bpf_program *prog)	512	struct bpf_program *prog)
508	{	513	{
509	const size_t bpf_insn_sz = sizeof(struct bpf_insn);	514	const size_t bpf_insn_sz = sizeof(struct bpf_insn);
@@ -552,7 +557,7 @@ errout: · 42 unmodified lines
552		557
553	static int	558	static int
554	bpf_object__add_program(struct bpf_object obj, void data, size_t size,	559	bpf_object__add_program(struct bpf_object obj, void data, size_t size,
555	char *section_name, int idx)	560	const char *section_name, int idx)
556	{	561	{
557	struct bpf_program prog, *progs;	562	struct bpf_program prog, *progs;
558	int nr_progs, err;	563	int nr_progs, err;
@@ -561,11 +566,10 @@ bpf_object__add_program(struct bpf_object obj, void data, size_t size, · 2 unmodified lines
561	if (err)	566	if (err)
562	return err;	567	return err;
563		568
564	prog.caps = &obj->caps;
565	progs = obj->programs;	569	progs = obj->programs;
566	nr_progs = obj->nr_programs;	570	nr_progs = obj->nr_programs;
567		571
568	progs = reallocarray(progs, nr_progs + 1, sizeof(progs[0]));	572	progs = libbpf_reallocarray(progs, nr_progs + 1, sizeof(progs[0]));
569	if (!progs) {	573	if (!progs) {
570	/*	574	/*
571	* In this case the original obj->programs	575	* In this case the original obj->programs
@@ -578,7 +582,7 @@ bpf_object__add_program(struct bpf_object obj, void data, size_t size, · 6 unmodified lines
578	return -ENOMEM;	582	return -ENOMEM;
579	}	583	}
580		584
581	pr_debug("found program %s\n", prog.section_name);	585	pr_debug("elf: found program '%s'\n", prog.section_name);
582	obj->programs = progs;	586	obj->programs = progs;
583	obj->nr_programs = nr_progs + 1;	587	obj->nr_programs = nr_progs + 1;
584	prog.obj = obj;	588	prog.obj = obj;
@@ -598,8 +602,7 @@ bpf_object__init_prog_names(struct bpf_object *obj) · 13 unmodified lines
598		602
599	prog = &obj->programs[pi];	603	prog = &obj->programs[pi];
600		604
601	for (si = 0; si < symbols->d_size / sizeof(GElf_Sym) && !name;	605	for (si = 0; si < symbols->d_size / sizeof(GElf_Sym) && !name; si++) {
602	si++) {
603	GElf_Sym sym;	606	GElf_Sym sym;
604		607
605	if (!gelf_getsym(symbols, si, &sym))	608	if (!gelf_getsym(symbols, si, &sym))
@@ -609,11 +612,9 @@ bpf_object__init_prog_names(struct bpf_object *obj) · 3 unmodified lines
609	if (GELF_ST_BIND(sym.st_info) != STB_GLOBAL)	612	if (GELF_ST_BIND(sym.st_info) != STB_GLOBAL)
610	continue;	613	continue;
611		614
612	name = elf_strptr(obj->efile.elf,	615	name = elf_sym_str(obj, sym.st_name);
613	obj->efile.strtabidx,
614	sym.st_name);
615	if (!name) {	616	if (!name) {
616	pr_warn("failed to get sym name string for prog %s\n",	617	pr_warn("prog '%s': failed to get symbol name\n",
617	prog->section_name);	618	prog->section_name);
618	return -LIBBPF_ERRNO__LIBELF;	619	return -LIBBPF_ERRNO__LIBELF;
619	}	620	}
@@ -623,17 +624,14 @@ bpf_object__init_prog_names(struct bpf_object *obj) · 3 unmodified lines
623	name = ".text";	624	name = ".text";
624		625
625	if (!name) {	626	if (!name) {
626	pr_warn("failed to find sym for prog %s\n",	627	pr_warn("prog '%s': failed to find program symbol\n",
627	prog->section_name);	628	prog->section_name);
628	return -EINVAL;	629	return -EINVAL;
629	}	630	}
630		631
631	prog->name = strdup(name);	632	prog->name = strdup(name);
632	if (!prog->name) {	633	if (!prog->name)
633	pr_warn("failed to allocate memory for prog sym %s\n",
634	name);
635	return -ENOMEM;	634	return -ENOMEM;
636	}
637	}	635	}
638		636
639	return 0;	637	return 0;
@@ -1066,13 +1064,18 @@ static void bpf_object__elf_finish(struct bpf_object *obj) · 426 unmodified lines
1066	obj->efile.obj_buf_sz = 0;	1064	obj->efile.obj_buf_sz = 0;
1067	}	1065	}
1068		1066
		1067	/* if libelf is old and doesn't support mmap(), fall back to read() */
		1068	#ifndef ELF_C_READ_MMAP
		1069	#define ELF_C_READ_MMAP ELF_C_READ
		1070	#endif
		1071
1069	static int bpf_object__elf_init(struct bpf_object *obj)	1072	static int bpf_object__elf_init(struct bpf_object *obj)
1070	{	1073	{
1071	int err = 0;	1074	int err = 0;
1072	GElf_Ehdr *ep;	1075	GElf_Ehdr *ep;
1073		1076
1074	if (obj_elf_valid(obj)) {	1077	if (obj_elf_valid(obj)) {
1075	pr_warn("elf init: internal error\n");	1078	pr_warn("elf: init internal error\n");
1076	return -LIBBPF_ERRNO__LIBELF;	1079	return -LIBBPF_ERRNO__LIBELF;
1077	}	1080	}
1078		1081
@@ -1090,31 +1093,44 @@ static int bpf_object__elf_init(struct bpf_object *obj) · 11 unmodified lines
1090		1093
1091	err = -errno;	1094	err = -errno;
1092	cp = libbpf_strerror_r(err, errmsg, sizeof(errmsg));	1095	cp = libbpf_strerror_r(err, errmsg, sizeof(errmsg));
1093	pr_warn("failed to open %s: %s\n", obj->path, cp);	1096	pr_warn("elf: failed to open %s: %s\n", obj->path, cp);
1094	return err;	1097	return err;
1095	}	1098	}
1096		1099
1097	obj->efile.elf = elf_begin(obj->efile.fd,	1100	obj->efile.elf = elf_begin(obj->efile.fd, ELF_C_READ_MMAP, NULL);
1098	LIBBPF_ELF_C_READ_MMAP, NULL);
1099	}	1101	}
1100		1102
1101	if (!obj->efile.elf) {	1103	if (!obj->efile.elf) {
1102	pr_warn("failed to open %s as ELF file\n", obj->path);	1104	pr_warn("elf: failed to open %s as ELF file: %s\n", obj->path, elf_errmsg(-1));
1103	err = -LIBBPF_ERRNO__LIBELF;	1105	err = -LIBBPF_ERRNO__LIBELF;
1104	goto errout;	1106	goto errout;
1105	}	1107	}
1106		1108
1107	if (!gelf_getehdr(obj->efile.elf, &obj->efile.ehdr)) {	1109	if (!gelf_getehdr(obj->efile.elf, &obj->efile.ehdr)) {
1108	pr_warn("failed to get EHDR from %s\n", obj->path);	1110	pr_warn("elf: failed to get ELF header from %s: %s\n", obj->path, elf_errmsg(-1));
1109	err = -LIBBPF_ERRNO__FORMAT;	1111	err = -LIBBPF_ERRNO__FORMAT;
1110	goto errout;	1112	goto errout;
1111	}	1113	}
1112	ep = &obj->efile.ehdr;	1114	ep = &obj->efile.ehdr;
1113		1115
		1116	if (elf_getshdrstrndx(obj->efile.elf, &obj->efile.shstrndx)) {
		1117	pr_warn("elf: failed to get section names section index for %s: %s\n",
		1118	obj->path, elf_errmsg(-1));
		1119	err = -LIBBPF_ERRNO__FORMAT;
		1120	goto errout;
		1121	}
		1122
		1123	/* Elf is corrupted/truncated, avoid calling elf_strptr. */
		1124	if (!elf_rawdata(elf_getscn(obj->efile.elf, obj->efile.shstrndx), NULL)) {
		1125	pr_warn("elf: failed to get section names strings from %s: %s\n",
		1126	obj->path, elf_errmsg(-1));
		1127	return -LIBBPF_ERRNO__FORMAT;
		1128	}
		1129
1114	/* Old LLVM set e_machine to EM_NONE */	1130	/* Old LLVM set e_machine to EM_NONE */
1115	if (ep->e_type != ET_REL \|\|	1131	if (ep->e_type != ET_REL \|\|
1116	(ep->e_machine && ep->e_machine != EM_BPF)) {	1132	(ep->e_machine && ep->e_machine != EM_BPF)) {
1117	pr_warn("%s is not an eBPF object file\n", obj->path);	1133	pr_warn("elf: %s is not a valid eBPF object file\n", obj->path);
1118	err = -LIBBPF_ERRNO__FORMAT;	1134	err = -LIBBPF_ERRNO__FORMAT;
1119	goto errout;	1135	goto errout;
1120	}	1136	}
@@ -1136,7 +1152,7 @@ static int bpf_object__check_endianness(struct bpf_object *obj) · 15 unmodified lines
1136	#else	1152	#else
1137	# error "Unrecognized __BYTE_ORDER__"	1153	# error "Unrecognized __BYTE_ORDER__"
1138	#endif	1154	#endif
1139	pr_warn("endianness mismatch.\n");	1155	pr_warn("elf: endianness mismatch in %s.\n", obj->path);
1140	return -LIBBPF_ERRNO__ENDIAN;	1156	return -LIBBPF_ERRNO__ENDIAN;
1141	}	1157	}
1142		1158
@@ -1171,55 +1187,10 @@ static bool bpf_map_type__is_map_in_map(enum bpf_map_type type) · 28 unmodified lines
1171	return false;	1187	return false;
1172	}	1188	}
1173		1189
1174	static int bpf_object_search_section_size(const struct bpf_object *obj,
1175	const char name, size_t d_size)
1176	{
1177	const GElf_Ehdr *ep = &obj->efile.ehdr;
1178	Elf *elf = obj->efile.elf;
1179	Elf_Scn *scn = NULL;
1180	int idx = 0;
1181
1182	while ((scn = elf_nextscn(elf, scn)) != NULL) {
1183	const char *sec_name;
1184	Elf_Data *data;
1185	GElf_Shdr sh;
1186
1187	idx++;
1188	if (gelf_getshdr(scn, &sh) != &sh) {
1189	pr_warn("failed to get section(%d) header from %s\n",
1190	idx, obj->path);
1191	return -EIO;
1192	}
1193
1194	sec_name = elf_strptr(elf, ep->e_shstrndx, sh.sh_name);
1195	if (!sec_name) {
1196	pr_warn("failed to get section(%d) name from %s\n",
1197	idx, obj->path);
1198	return -EIO;
1199	}
1200
1201	if (strcmp(name, sec_name))
1202	continue;
1203
1204	data = elf_getdata(scn, 0);
1205	if (!data) {
1206	pr_warn("failed to get section(%d) data from %s(%s)\n",
1207	idx, name, obj->path);
1208	return -EIO;
1209	}
1210
1211	*d_size = data->d_size;
1212	return 0;
1213	}
1214
1215	return -ENOENT;
1216	}
1217
1218	int bpf_object__section_size(const struct bpf_object obj, const char name,	1190	int bpf_object__section_size(const struct bpf_object obj, const char name,
1219	__u32 *size)	1191	__u32 *size)
1220	{	1192	{
1221	int ret = -ENOENT;	1193	int ret = -ENOENT;
1222	size_t d_size;
1223		1194
1224	*size = 0;	1195	*size = 0;
1225	if (!name) {	1196	if (!name) {
@@ -1237,9 +1208,13 @@ int bpf_object__section_size(const struct bpf_object obj, const char name, · 11 unmodified lines
1237	if (obj->efile.st_ops_data)	1208	if (obj->efile.st_ops_data)
1238	*size = obj->efile.st_ops_data->d_size;	1209	*size = obj->efile.st_ops_data->d_size;
1239	} else {	1210	} else {
1240	ret = bpf_object_search_section_size(obj, name, &d_size);	1211	Elf_Scn *scn = elf_sec_by_name(obj, name);
1241	if (!ret)	1212	Elf_Data *data = elf_sec_data(obj, scn);
1242	*size = d_size;	1213
		1214	if (data) {
		1215	ret = 0; /* found it */
		1216	*size = data->d_size;
		1217	}
1243	}	1218	}
1244		1219
1245	return *size ? 0 : ret;	1220	return *size ? 0 : ret;
@@ -1264,8 +1239,7 @@ int bpf_object__variable_offset(const struct bpf_object obj, const char name, · 18 unmodified lines
1264	GELF_ST_TYPE(sym.st_info) != STT_OBJECT)	1239	GELF_ST_TYPE(sym.st_info) != STT_OBJECT)
1265	continue;	1240	continue;
1266		1241
1267	sname = elf_strptr(obj->efile.elf, obj->efile.strtabidx,	1242	sname = elf_sym_str(obj, sym.st_name);
1268	sym.st_name);
1269	if (!sname) {	1243	if (!sname) {
1270	pr_warn("failed to get sym name string for var %s\n",	1244	pr_warn("failed to get sym name string for var %s\n",
1271	name);	1245	name);
@@ -1290,7 +1264,7 @@ static struct bpf_map bpf_object__add_map(struct bpf_object obj) · 18 unmodified lines
1290	return &obj->maps[obj->nr_maps++];	1264	return &obj->maps[obj->nr_maps++];
1291		1265
1292	new_cap = max((size_t)4, obj->maps_cap * 3 / 2);	1266	new_cap = max((size_t)4, obj->maps_cap * 3 / 2);
1293	new_maps = realloc(obj->maps, new_cap * sizeof(*obj->maps));	1267	new_maps = libbpf_reallocarray(obj->maps, new_cap, sizeof(*obj->maps));
1294	if (!new_maps) {	1268	if (!new_maps) {
1295	pr_warn("alloc maps for object failed\n");	1269	pr_warn("alloc maps for object failed\n");
1296	return ERR_PTR(-ENOMEM);	1270	return ERR_PTR(-ENOMEM);
@@ -1742,12 +1716,12 @@ static int bpf_object__init_user_maps(struct bpf_object *obj, bool strict) · 445 unmodified lines
1742	if (!symbols)	1716	if (!symbols)
1743	return -EINVAL;	1717	return -EINVAL;
1744		1718
1745	scn = elf_getscn(obj->efile.elf, obj->efile.maps_shndx);	1719
1746	if (scn)	1720	scn = elf_sec_by_idx(obj, obj->efile.maps_shndx);
1747	data = elf_getdata(scn, NULL);	1721	data = elf_sec_data(obj, scn);
1748	if (!scn \|\| !data) {	1722	if (!scn \|\| !data) {
1749	pr_warn("failed to get Elf_Data from map section %d\n",	1723	pr_warn("elf: failed to get legacy map definitions for %s\n",
1750	obj->efile.maps_shndx);	1724	obj->path);
1751	return -EINVAL;	1725	return -EINVAL;
1752	}	1726	}
1753		1727
@@ -1769,12 +1743,12 @@ static int bpf_object__init_user_maps(struct bpf_object *obj, bool strict) · 15 unmodified lines
1769	nr_maps++;	1743	nr_maps++;
1770	}	1744	}
1771	/* Assume equally sized map definitions */	1745	/* Assume equally sized map definitions */
1772	pr_debug("maps in %s: %d maps in %zd bytes\n",	1746	pr_debug("elf: found %d legacy map definitions (%zd bytes) in %s\n",
1773	obj->path, nr_maps, data->d_size);	1747	nr_maps, data->d_size, obj->path);
1774		1748
1775	if (!data->d_size \|\| nr_maps == 0 \|\| (data->d_size % nr_maps) != 0) {	1749	if (!data->d_size \|\| nr_maps == 0 \|\| (data->d_size % nr_maps) != 0) {
1776	pr_warn("unable to determine map definition size section %s, %d maps in %zd bytes\n",	1750	pr_warn("elf: unable to determine legacy map definition size in %s\n",
1777	obj->path, nr_maps, data->d_size);	1751	obj->path);
1778	return -EINVAL;	1752	return -EINVAL;
1779	}	1753	}
1780	map_def_sz = data->d_size / nr_maps;	1754	map_def_sz = data->d_size / nr_maps;
@@ -1795,8 +1769,7 @@ static int bpf_object__init_user_maps(struct bpf_object *obj, bool strict) · 14 unmodified lines
1795	if (IS_ERR(map))	1769	if (IS_ERR(map))
1796	return PTR_ERR(map);	1770	return PTR_ERR(map);
1797		1771
1798	map_name = elf_strptr(obj->efile.elf, obj->efile.strtabidx,	1772	map_name = elf_sym_str(obj, sym.st_name);
1799	sym.st_name);
1800	if (!map_name) {	1773	if (!map_name) {
1801	pr_warn("failed to get map #%d name sym string for obj %s\n",	1774	pr_warn("failed to get map #%d name sym string for obj %s\n",
1802	i, obj->path);	1775	i, obj->path);
@@ -1884,6 +1857,29 @@ resolve_func_ptr(const struct btf btf, __u32 id, __u32 res_id) · 81 unmodified lines
1884	return btf_is_func_proto(t) ? t : NULL;	1857	return btf_is_func_proto(t) ? t : NULL;
1885	}	1858	}
1886		1859
		1860	static const char btf_kind_str(const struct btf_type t)
		1861	{
		1862	switch (btf_kind(t)) {
		1863	case BTF_KIND_UNKN: return "void";
		1864	case BTF_KIND_INT: return "int";
		1865	case BTF_KIND_PTR: return "ptr";
		1866	case BTF_KIND_ARRAY: return "array";
		1867	case BTF_KIND_STRUCT: return "struct";
		1868	case BTF_KIND_UNION: return "union";
		1869	case BTF_KIND_ENUM: return "enum";
		1870	case BTF_KIND_FWD: return "fwd";
		1871	case BTF_KIND_TYPEDEF: return "typedef";
		1872	case BTF_KIND_VOLATILE: return "volatile";
		1873	case BTF_KIND_CONST: return "const";
		1874	case BTF_KIND_RESTRICT: return "restrict";
		1875	case BTF_KIND_FUNC: return "func";
		1876	case BTF_KIND_FUNC_PROTO: return "func_proto";
		1877	case BTF_KIND_VAR: return "var";
		1878	case BTF_KIND_DATASEC: return "datasec";
		1879	default: return "unknown";
		1880	}
		1881	}
		1882
1887	/*	1883	/*
1888	* Fetch integer attribute of BTF map definition. Such attributes are	1884	* Fetch integer attribute of BTF map definition. Such attributes are
1889	* represented using a pointer to an array, in which dimensionality of array	1885	* represented using a pointer to an array, in which dimensionality of array
@@ -1900,8 +1896,8 @@ static bool get_map_field_int(const char map_name, const struct btf btf, · 10 unmodified lines
1900	const struct btf_type *arr_t;	1896	const struct btf_type *arr_t;
1901		1897
1902	if (!btf_is_ptr(t)) {	1898	if (!btf_is_ptr(t)) {
1903	pr_warn("map '%s': attr '%s': expected PTR, got %u.\n",	1899	pr_warn("map '%s': attr '%s': expected PTR, got %s.\n",
1904	map_name, name, btf_kind(t));	1900	map_name, name, btf_kind_str(t));
1905	return false;	1901	return false;
1906	}	1902	}
1907		1903
@@ -1912,8 +1908,8 @@ static bool get_map_field_int(const char map_name, const struct btf btf, · 4 unmodified lines
1912	return false;	1908	return false;
1913	}	1909	}
1914	if (!btf_is_array(arr_t)) {	1910	if (!btf_is_array(arr_t)) {
1915	pr_warn("map '%s': attr '%s': expected ARRAY, got %u.\n",	1911	pr_warn("map '%s': attr '%s': expected ARRAY, got %s.\n",
1916	map_name, name, btf_kind(arr_t));	1912	map_name, name, btf_kind_str(arr_t));
1917	return false;	1913	return false;
1918	}	1914	}
1919	arr_info = btf_array(arr_t);	1915	arr_info = btf_array(arr_t);
@@ -1924,7 +1920,7 @@ static bool get_map_field_int(const char map_name, const struct btf btf, · 4 unmodified lines
1924	static int build_map_pin_path(struct bpf_map map, const char path)	1920	static int build_map_pin_path(struct bpf_map map, const char path)
1925	{	1921	{
1926	char buf[PATH_MAX];	1922	char buf[PATH_MAX];
1927	int err, len;	1923	int len;
1928		1924
1929	if (!path)	1925	if (!path)
1930	path = "/sys/fs/bpf";	1926	path = "/sys/fs/bpf";
@@ -1935,11 +1931,7 @@ static int build_map_pin_path(struct bpf_map map, const char path) · 4 unmodified lines
1935	else if (len >= PATH_MAX)	1931	else if (len >= PATH_MAX)
1936	return -ENAMETOOLONG;	1932	return -ENAMETOOLONG;
1937		1933
1938	err = bpf_map__set_pin_path(map, buf);	1934	return bpf_map__set_pin_path(map, buf);
1939	if (err)
1940	return err;
1941
1942	return 0;
1943	}	1935	}
1944		1936
1945		1937
@@ -2007,8 +1999,8 @@ static int parse_btf_map_def(struct bpf_object *obj, · 61 unmodified lines
2007	return -EINVAL;	1999	return -EINVAL;
2008	}	2000	}
2009	if (!btf_is_ptr(t)) {	2001	if (!btf_is_ptr(t)) {
2010	pr_warn("map '%s': key spec is not PTR: %u.\n",	2002	pr_warn("map '%s': key spec is not PTR: %s.\n",
2011	map->name, btf_kind(t));	2003	map->name, btf_kind_str(t));
2012	return -EINVAL;	2004	return -EINVAL;
2013	}	2005	}
2014	sz = btf__resolve_size(obj->btf, t->type);	2006	sz = btf__resolve_size(obj->btf, t->type);
@@ -2049,8 +2041,8 @@ static int parse_btf_map_def(struct bpf_object *obj, · 34 unmodified lines
2049	return -EINVAL;	2041	return -EINVAL;
2050	}	2042	}
2051	if (!btf_is_ptr(t)) {	2043	if (!btf_is_ptr(t)) {
2052	pr_warn("map '%s': value spec is not PTR: %u.\n",	2044	pr_warn("map '%s': value spec is not PTR: %s.\n",
2053	map->name, btf_kind(t));	2045	map->name, btf_kind_str(t));
2054	return -EINVAL;	2046	return -EINVAL;
2055	}	2047	}
2056	sz = btf__resolve_size(obj->btf, t->type);	2048	sz = btf__resolve_size(obj->btf, t->type);
@@ -2107,14 +2099,14 @@ static int parse_btf_map_def(struct bpf_object *obj, · 50 unmodified lines
2107	t = skip_mods_and_typedefs(obj->btf, btf_array(t)->type,	2099	t = skip_mods_and_typedefs(obj->btf, btf_array(t)->type,
2108	NULL);	2100	NULL);
2109	if (!btf_is_ptr(t)) {	2101	if (!btf_is_ptr(t)) {
2110	pr_warn("map '%s': map-in-map inner def is of unexpected kind %u.\n",	2102	pr_warn("map '%s': map-in-map inner def is of unexpected kind %s.\n",
2111	map->name, btf_kind(t));	2103	map->name, btf_kind_str(t));
2112	return -EINVAL;	2104	return -EINVAL;
2113	}	2105	}
2114	t = skip_mods_and_typedefs(obj->btf, t->type, NULL);	2106	t = skip_mods_and_typedefs(obj->btf, t->type, NULL);
2115	if (!btf_is_struct(t)) {	2107	if (!btf_is_struct(t)) {
2116	pr_warn("map '%s': map-in-map inner def is of unexpected kind %u.\n",	2108	pr_warn("map '%s': map-in-map inner def is of unexpected kind %s.\n",
2117	map->name, btf_kind(t));	2109	map->name, btf_kind_str(t));
2118	return -EINVAL;	2110	return -EINVAL;
2119	}	2111	}
2120		2112
@@ -2205,8 +2197,8 @@ static int bpf_object__init_user_btf_map(struct bpf_object *obj, · 84 unmodified lines
2205	return -EINVAL;	2197	return -EINVAL;
2206	}	2198	}
2207	if (!btf_is_var(var)) {	2199	if (!btf_is_var(var)) {
2208	pr_warn("map '%s': unexpected var kind %u.\n",	2200	pr_warn("map '%s': unexpected var kind %s.\n",
2209	map_name, btf_kind(var));	2201	map_name, btf_kind_str(var));
2210	return -EINVAL;	2202	return -EINVAL;
2211	}	2203	}
2212	if (var_extra->linkage != BTF_VAR_GLOBAL_ALLOCATED &&	2204	if (var_extra->linkage != BTF_VAR_GLOBAL_ALLOCATED &&
@@ -2218,8 +2210,8 @@ static int bpf_object__init_user_btf_map(struct bpf_object *obj, · 5 unmodified lines
2218		2210
2219	def = skip_mods_and_typedefs(obj->btf, var->type, NULL);	2211	def = skip_mods_and_typedefs(obj->btf, var->type, NULL);
2220	if (!btf_is_struct(def)) {	2212	if (!btf_is_struct(def)) {
2221	pr_warn("map '%s': unexpected def kind %u.\n",	2213	pr_warn("map '%s': unexpected def kind %s.\n",
2222	map_name, btf_kind(var));	2214	map_name, btf_kind_str(var));
2223	return -EINVAL;	2215	return -EINVAL;
2224	}	2216	}
2225	if (def->size > vi->size) {	2217	if (def->size > vi->size) {
@@ -2259,12 +2251,11 @@ static int bpf_object__init_user_btf_maps(struct bpf_object *obj, bool strict, · 33 unmodified lines
2259	if (obj->efile.btf_maps_shndx < 0)	2251	if (obj->efile.btf_maps_shndx < 0)
2260	return 0;	2252	return 0;
2261		2253
2262	scn = elf_getscn(obj->efile.elf, obj->efile.btf_maps_shndx);	2254	scn = elf_sec_by_idx(obj, obj->efile.btf_maps_shndx);
2263	if (scn)	2255	data = elf_sec_data(obj, scn);
2264	data = elf_getdata(scn, NULL);
2265	if (!scn \|\| !data) {	2256	if (!scn \|\| !data) {
2266	pr_warn("failed to get Elf_Data from map section %d (%s)\n",	2257	pr_warn("elf: failed to get %s map definitions for %s\n",
2267	obj->efile.btf_maps_shndx, MAPS_ELF_SEC);	2258	MAPS_ELF_SEC, obj->path);
2268	return -EINVAL;	2259	return -EINVAL;
2269	}	2260	}
2270		2261
@@ -2322,36 +2313,28 @@ static int bpf_object__init_maps(struct bpf_object *obj, · 51 unmodified lines
2322		2313
2323	static bool section_have_execinstr(struct bpf_object *obj, int idx)	2314	static bool section_have_execinstr(struct bpf_object *obj, int idx)
2324	{	2315	{
2325	Elf_Scn *scn;
2326	GElf_Shdr sh;	2316	GElf_Shdr sh;
2327		2317
2328	scn = elf_getscn(obj->efile.elf, idx);	2318	if (elf_sec_hdr(obj, elf_sec_by_idx(obj, idx), &sh))
2329	if (!scn)
2330	return false;
2331
2332	if (gelf_getshdr(scn, &sh) != &sh)
2333	return false;	2319	return false;
2334		2320
2335	if (sh.sh_flags & SHF_EXECINSTR)	2321	return sh.sh_flags & SHF_EXECINSTR;
2336	return true;
2337
2338	return false;
2339	}	2322	}
2340		2323
2341	static bool btf_needs_sanitization(struct bpf_object *obj)	2324	static bool btf_needs_sanitization(struct bpf_object *obj)
2342	{	2325	{
2343	bool has_func_global = obj->caps.btf_func_global;	2326	bool has_func_global = kernel_supports(FEAT_BTF_GLOBAL_FUNC);
2344	bool has_datasec = obj->caps.btf_datasec;	2327	bool has_datasec = kernel_supports(FEAT_BTF_DATASEC);
2345	bool has_func = obj->caps.btf_func;	2328	bool has_func = kernel_supports(FEAT_BTF_FUNC);
2346		2329
2347	return !has_func \|\| !has_datasec \|\| !has_func_global;	2330	return !has_func \|\| !has_datasec \|\| !has_func_global;
2348	}	2331	}
2349		2332
2350	static void bpf_object__sanitize_btf(struct bpf_object obj, struct btf btf)	2333	static void bpf_object__sanitize_btf(struct bpf_object obj, struct btf btf)
2351	{	2334	{
2352	bool has_func_global = obj->caps.btf_func_global;	2335	bool has_func_global = kernel_supports(FEAT_BTF_GLOBAL_FUNC);
2353	bool has_datasec = obj->caps.btf_datasec;	2336	bool has_datasec = kernel_supports(FEAT_BTF_DATASEC);
2354	bool has_func = obj->caps.btf_func;	2337	bool has_func = kernel_supports(FEAT_BTF_FUNC);
2355	struct btf_type *t;	2338	struct btf_type *t;
2356	int i, j, vlen;	2339	int i, j, vlen;
2357		2340
@@ -2499,7 +2482,7 @@ static int bpf_object__load_vmlinux_btf(struct bpf_object *obj) · 141 unmodified lines
2499	int err;	2482	int err;
2500		2483
2501	/* CO-RE relocations need kernel BTF */	2484	/* CO-RE relocations need kernel BTF */
2502	if (obj->btf_ext && obj->btf_ext->field_reloc_info.len)	2485	if (obj->btf_ext && obj->btf_ext->core_relo_info.len)
2503	need_vmlinux_btf = true;	2486	need_vmlinux_btf = true;
2504		2487
2505	bpf_object__for_each_program(prog, obj) {	2488	bpf_object__for_each_program(prog, obj) {
@@ -2533,6 +2516,15 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) · 27 unmodified lines
2533	if (!obj->btf)	2516	if (!obj->btf)
2534	return 0;	2517	return 0;
2535		2518
		2519	if (!kernel_supports(FEAT_BTF)) {
		2520	if (kernel_needs_btf(obj)) {
		2521	err = -EOPNOTSUPP;
		2522	goto report;
		2523	}
		2524	pr_debug("Kernel doesn't support BTF, skipping uploading it.\n");
		2525	return 0;
		2526	}
		2527
2536	sanitize = btf_needs_sanitization(obj);	2528	sanitize = btf_needs_sanitization(obj);
2537	if (sanitize) {	2529	if (sanitize) {
2538	const void *raw_data;	2530	const void *raw_data;
@@ -2558,6 +2550,7 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) · 19 unmodified lines
2558	}	2550	}
2559	btf__free(kern_btf);	2551	btf__free(kern_btf);
2560	}	2552	}
		2553	report:
2561	if (err) {	2554	if (err) {
2562	btf_mandatory = kernel_needs_btf(obj);	2555	btf_mandatory = kernel_needs_btf(obj);
2563	pr_warn("Error loading .BTF into kernel: %d. %s\n", err,	2556	pr_warn("Error loading .BTF into kernel: %d. %s\n", err,
@@ -2569,61 +2562,199 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) · 5 unmodified lines
2569	return err;	2562	return err;
2570	}	2563	}
2571		2564
		2565	static const char elf_sym_str(const struct bpf_object obj, size_t off)
		2566	{
		2567	const char *name;
		2568
		2569	name = elf_strptr(obj->efile.elf, obj->efile.strtabidx, off);
		2570	if (!name) {
		2571	pr_warn("elf: failed to get section name string at offset %zu from %s: %s\n",
		2572	off, obj->path, elf_errmsg(-1));
		2573	return NULL;
		2574	}
		2575
		2576	return name;
		2577	}
		2578
		2579	static const char elf_sec_str(const struct bpf_object obj, size_t off)
		2580	{
		2581	const char *name;
		2582
		2583	name = elf_strptr(obj->efile.elf, obj->efile.shstrndx, off);
		2584	if (!name) {
		2585	pr_warn("elf: failed to get section name string at offset %zu from %s: %s\n",
		2586	off, obj->path, elf_errmsg(-1));
		2587	return NULL;
		2588	}
		2589
		2590	return name;
		2591	}
		2592
		2593	static Elf_Scn elf_sec_by_idx(const struct bpf_object obj, size_t idx)
		2594	{
		2595	Elf_Scn *scn;
		2596
		2597	scn = elf_getscn(obj->efile.elf, idx);
		2598	if (!scn) {
		2599	pr_warn("elf: failed to get section(%zu) from %s: %s\n",
		2600	idx, obj->path, elf_errmsg(-1));
		2601	return NULL;
		2602	}
		2603	return scn;
		2604	}
		2605
		2606	static Elf_Scn elf_sec_by_name(const struct bpf_object obj, const char *name)
		2607	{
		2608	Elf_Scn *scn = NULL;
		2609	Elf *elf = obj->efile.elf;
		2610	const char *sec_name;
		2611
		2612	while ((scn = elf_nextscn(elf, scn)) != NULL) {
		2613	sec_name = elf_sec_name(obj, scn);
		2614	if (!sec_name)
		2615	return NULL;
		2616
		2617	if (strcmp(sec_name, name) != 0)
		2618	continue;
		2619
		2620	return scn;
		2621	}
		2622	return NULL;
		2623	}
		2624
		2625	static int elf_sec_hdr(const struct bpf_object obj, Elf_Scn scn, GElf_Shdr *hdr)
		2626	{
		2627	if (!scn)
		2628	return -EINVAL;
		2629
		2630	if (gelf_getshdr(scn, hdr) != hdr) {
		2631	pr_warn("elf: failed to get section(%zu) header from %s: %s\n",
		2632	elf_ndxscn(scn), obj->path, elf_errmsg(-1));
		2633	return -EINVAL;
		2634	}
		2635
		2636	return 0;
		2637	}
		2638
		2639	static const char elf_sec_name(const struct bpf_object obj, Elf_Scn *scn)
		2640	{
		2641	const char *name;
		2642	GElf_Shdr sh;
		2643
		2644	if (!scn)
		2645	return NULL;
		2646
		2647	if (elf_sec_hdr(obj, scn, &sh))
		2648	return NULL;
		2649
		2650	name = elf_sec_str(obj, sh.sh_name);
		2651	if (!name) {
		2652	pr_warn("elf: failed to get section(%zu) name from %s: %s\n",
		2653	elf_ndxscn(scn), obj->path, elf_errmsg(-1));
		2654	return NULL;
		2655	}
		2656
		2657	return name;
		2658	}
		2659
		2660	static Elf_Data elf_sec_data(const struct bpf_object obj, Elf_Scn *scn)
		2661	{
		2662	Elf_Data *data;
		2663
		2664	if (!scn)
		2665	return NULL;
		2666
		2667	data = elf_getdata(scn, 0);
		2668	if (!data) {
		2669	pr_warn("elf: failed to get section(%zu) %s data from %s: %s\n",
		2670	elf_ndxscn(scn), elf_sec_name(obj, scn) ?: "<?>",
		2671	obj->path, elf_errmsg(-1));
		2672	return NULL;
		2673	}
		2674
		2675	return data;
		2676	}
		2677
		2678	static bool is_sec_name_dwarf(const char *name)
		2679	{
		2680	/* approximation, but the actual list is too long */
		2681	return strncmp(name, ".debug_", sizeof(".debug_") - 1) == 0;
		2682	}
		2683
		2684	static bool ignore_elf_section(GElf_Shdr hdr, const char name)
		2685	{
		2686	/* no special handling of .strtab */
		2687	if (hdr->sh_type == SHT_STRTAB)
		2688	return true;
		2689
		2690	/* ignore .llvm_addrsig section as well */
		2691	if (hdr->sh_type == 0x6FFF4C03 /* SHT_LLVM_ADDRSIG */)
		2692	return true;
		2693
		2694	/* no subprograms will lead to an empty .text section, ignore it */
		2695	if (hdr->sh_type == SHT_PROGBITS && hdr->sh_size == 0 &&
		2696	strcmp(name, ".text") == 0)
		2697	return true;
		2698
		2699	/* DWARF sections */
		2700	if (is_sec_name_dwarf(name))
		2701	return true;
		2702
		2703	if (strncmp(name, ".rel", sizeof(".rel") - 1) == 0) {
		2704	name += sizeof(".rel") - 1;
		2705	/* DWARF section relocations */
		2706	if (is_sec_name_dwarf(name))
		2707	return true;
		2708
		2709	/* .BTF and .BTF.ext don't need relocations */
		2710	if (strcmp(name, BTF_ELF_SEC) == 0 \|\|
		2711	strcmp(name, BTF_EXT_ELF_SEC) == 0)
		2712	return true;
		2713	}
		2714
		2715	return false;
		2716	}
		2717
2572	static int bpf_object__elf_collect(struct bpf_object *obj)	2718	static int bpf_object__elf_collect(struct bpf_object *obj)
2573	{	2719	{
2574	Elf *elf = obj->efile.elf;	2720	Elf *elf = obj->efile.elf;
2575	GElf_Ehdr *ep = &obj->efile.ehdr;
2576	Elf_Data *btf_ext_data = NULL;	2721	Elf_Data *btf_ext_data = NULL;
2577	Elf_Data *btf_data = NULL;	2722	Elf_Data *btf_data = NULL;
2578	Elf_Scn *scn = NULL;	2723	Elf_Scn *scn = NULL;
2579	int idx = 0, err = 0;	2724	int idx = 0, err = 0;
2580		2725
2581	/* Elf is corrupted/truncated, avoid calling elf_strptr. */
2582	if (!elf_rawdata(elf_getscn(elf, ep->e_shstrndx), NULL)) {
2583	pr_warn("failed to get e_shstrndx from %s\n", obj->path);
2584	return -LIBBPF_ERRNO__FORMAT;
2585	}
2586
2587	while ((scn = elf_nextscn(elf, scn)) != NULL) {	2726	while ((scn = elf_nextscn(elf, scn)) != NULL) {
2588	char *name;	2727	const char *name;
2589	GElf_Shdr sh;	2728	GElf_Shdr sh;
2590	Elf_Data *data;	2729	Elf_Data *data;
2591		2730
2592	idx++;	2731	idx++;
2593	if (gelf_getshdr(scn, &sh) != &sh) {	2732
2594	pr_warn("failed to get section(%d) header from %s\n",	2733	if (elf_sec_hdr(obj, scn, &sh))
2595	idx, obj->path);
2596	return -LIBBPF_ERRNO__FORMAT;	2734	return -LIBBPF_ERRNO__FORMAT;
2597	}
2598		2735
2599	name = elf_strptr(elf, ep->e_shstrndx, sh.sh_name);	2736	name = elf_sec_str(obj, sh.sh_name);
2600	if (!name) {	2737	if (!name)
2601	pr_warn("failed to get section(%d) name from %s\n",
2602	idx, obj->path);
2603	return -LIBBPF_ERRNO__FORMAT;	2738	return -LIBBPF_ERRNO__FORMAT;
2604	}
2605		2739
2606	data = elf_getdata(scn, 0);	2740	if (ignore_elf_section(&sh, name))
2607	if (!data) {	2741	continue;
2608	pr_warn("failed to get section(%d) data from %s(%s)\n",	2742
2609	idx, name, obj->path);	2743	data = elf_sec_data(obj, scn);
		2744	if (!data)
2610	return -LIBBPF_ERRNO__FORMAT;	2745	return -LIBBPF_ERRNO__FORMAT;
2611	}	2746
2612	pr_debug("section(%d) %s, size %ld, link %d, flags %lx, type=%d\n",	2747	pr_debug("elf: section(%d) %s, size %ld, link %d, flags %lx, type=%d\n",
2613	idx, name, (unsigned long)data->d_size,	2748	idx, name, (unsigned long)data->d_size,
2614	(int)sh.sh_link, (unsigned long)sh.sh_flags,	2749	(int)sh.sh_link, (unsigned long)sh.sh_flags,
2615	(int)sh.sh_type);	2750	(int)sh.sh_type);
2616		2751
2617	if (strcmp(name, "license") == 0) {	2752	if (strcmp(name, "license") == 0) {
2618	err = bpf_object__init_license(obj,	2753	err = bpf_object__init_license(obj, data->d_buf, data->d_size);
2619	data->d_buf,
2620	data->d_size);
2621	if (err)	2754	if (err)
2622	return err;	2755	return err;
2623	} else if (strcmp(name, "version") == 0) {	2756	} else if (strcmp(name, "version") == 0) {
2624	err = bpf_object__init_kversion(obj,	2757	err = bpf_object__init_kversion(obj, data->d_buf, data->d_size);
2625	data->d_buf,
2626	data->d_size);
2627	if (err)	2758	if (err)
2628	return err;	2759	return err;
2629	} else if (strcmp(name, "maps") == 0) {	2760	} else if (strcmp(name, "maps") == 0) {
@@ -2636,8 +2767,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj) · 6 unmodified lines
2636	btf_ext_data = data;	2767	btf_ext_data = data;
2637	} else if (sh.sh_type == SHT_SYMTAB) {	2768	} else if (sh.sh_type == SHT_SYMTAB) {
2638	if (obj->efile.symbols) {	2769	if (obj->efile.symbols) {
2639	pr_warn("bpf: multiple SYMTAB in %s\n",	2770	pr_warn("elf: multiple symbol tables in %s\n", obj->path);
2640	obj->path);
2641	return -LIBBPF_ERRNO__FORMAT;	2771	return -LIBBPF_ERRNO__FORMAT;
2642	}	2772	}
2643	obj->efile.symbols = data;	2773	obj->efile.symbols = data;
@@ -2650,16 +2780,8 @@ static int bpf_object__elf_collect(struct bpf_object *obj) · 6 unmodified lines
2650	err = bpf_object__add_program(obj, data->d_buf,	2780	err = bpf_object__add_program(obj, data->d_buf,
2651	data->d_size,	2781	data->d_size,
2652	name, idx);	2782	name, idx);
2653	if (err) {	2783	if (err)
2654	char errmsg[STRERR_BUFSIZE];
2655	char *cp;
2656
2657	cp = libbpf_strerror_r(-err, errmsg,
2658	sizeof(errmsg));
2659	pr_warn("failed to alloc program %s (%s): %s",
2660	name, obj->path, cp);
2661	return err;	2784	return err;
2662	}
2663	} else if (strcmp(name, DATA_SEC) == 0) {	2785	} else if (strcmp(name, DATA_SEC) == 0) {
2664	obj->efile.data = data;	2786	obj->efile.data = data;
2665	obj->efile.data_shndx = idx;	2787	obj->efile.data_shndx = idx;
@@ -2670,7 +2792,8 @@ static int bpf_object__elf_collect(struct bpf_object *obj) · 4 unmodified lines
2670	obj->efile.st_ops_data = data;	2792	obj->efile.st_ops_data = data;
2671	obj->efile.st_ops_shndx = idx;	2793	obj->efile.st_ops_shndx = idx;
2672	} else {	2794	} else {
2673	pr_debug("skip section(%d) %s\n", idx, name);	2795	pr_info("elf: skipping unrecognized data section(%d) %s\n",
		2796	idx, name);
2674	}	2797	}
2675	} else if (sh.sh_type == SHT_REL) {	2798	} else if (sh.sh_type == SHT_REL) {
2676	int nr_sects = obj->efile.nr_reloc_sects;	2799	int nr_sects = obj->efile.nr_reloc_sects;
@@ -2681,34 +2804,33 @@ static int bpf_object__elf_collect(struct bpf_object *obj) · 4 unmodified lines
2681	if (!section_have_execinstr(obj, sec) &&	2804	if (!section_have_execinstr(obj, sec) &&
2682	strcmp(name, ".rel" STRUCT_OPS_SEC) &&	2805	strcmp(name, ".rel" STRUCT_OPS_SEC) &&
2683	strcmp(name, ".rel" MAPS_ELF_SEC)) {	2806	strcmp(name, ".rel" MAPS_ELF_SEC)) {
2684	pr_debug("skip relo %s(%d) for section(%d)\n",	2807	pr_info("elf: skipping relo section(%d) %s for section(%d) %s\n",
2685	name, idx, sec);	2808	idx, name, sec,
		2809	elf_sec_name(obj, elf_sec_by_idx(obj, sec)) ?: "<?>");
2686	continue;	2810	continue;
2687	}	2811	}
2688		2812
2689	sects = reallocarray(sects, nr_sects + 1,	2813	sects = libbpf_reallocarray(sects, nr_sects + 1,
2690	sizeof(*obj->efile.reloc_sects));	2814	sizeof(*obj->efile.reloc_sects));
2691	if (!sects) {	2815	if (!sects)
2692	pr_warn("reloc_sects realloc failed\n");
2693	return -ENOMEM;	2816	return -ENOMEM;
2694	}
2695		2817
2696	obj->efile.reloc_sects = sects;	2818	obj->efile.reloc_sects = sects;
2697	obj->efile.nr_reloc_sects++;	2819	obj->efile.nr_reloc_sects++;
2698		2820
2699	obj->efile.reloc_sects[nr_sects].shdr = sh;	2821	obj->efile.reloc_sects[nr_sects].shdr = sh;
2700	obj->efile.reloc_sects[nr_sects].data = data;	2822	obj->efile.reloc_sects[nr_sects].data = data;
2701	} else if (sh.sh_type == SHT_NOBITS &&	2823	} else if (sh.sh_type == SHT_NOBITS && strcmp(name, BSS_SEC) == 0) {
2702	strcmp(name, BSS_SEC) == 0) {
2703	obj->efile.bss = data;	2824	obj->efile.bss = data;
2704	obj->efile.bss_shndx = idx;	2825	obj->efile.bss_shndx = idx;
2705	} else {	2826	} else {
2706	pr_debug("skip section(%d) %s\n", idx, name);	2827	pr_info("elf: skipping section(%d) %s (size %zu)\n", idx, name,
		2828	(size_t)sh.sh_size);
2707	}	2829	}
2708	}	2830	}
2709		2831
2710	if (!obj->efile.strtabidx \|\| obj->efile.strtabidx > idx) {	2832	if (!obj->efile.strtabidx \|\| obj->efile.strtabidx > idx) {
2711	pr_warn("Corrupted ELF file: index of strtab invalid\n");	2833	pr_warn("elf: symbol strings section missing or invalid in %s\n", obj->path);
2712	return -LIBBPF_ERRNO__FORMAT;	2834	return -LIBBPF_ERRNO__FORMAT;
2713	}	2835	}
2714	return bpf_object__init_btf(obj, btf_data, btf_ext_data);	2836	return bpf_object__init_btf(obj, btf_data, btf_ext_data);
@@ -2869,14 +2991,13 @@ static int bpf_object__collect_externs(struct bpf_object *obj) · 154 unmodified lines
2869	if (!obj->efile.symbols)	2991	if (!obj->efile.symbols)
2870	return 0;	2992	return 0;
2871		2993
2872	scn = elf_getscn(obj->efile.elf, obj->efile.symbols_shndx);	2994	scn = elf_sec_by_idx(obj, obj->efile.symbols_shndx);
2873	if (!scn)	2995	if (elf_sec_hdr(obj, scn, &sh))
2874	return -LIBBPF_ERRNO__FORMAT;
2875	if (gelf_getshdr(scn, &sh) != &sh)
2876	return -LIBBPF_ERRNO__FORMAT;	2996	return -LIBBPF_ERRNO__FORMAT;
2877	n = sh.sh_size / sh.sh_entsize;
2878		2997
		2998	n = sh.sh_size / sh.sh_entsize;
2879	pr_debug("looking for externs among %d symbols...\n", n);	2999	pr_debug("looking for externs among %d symbols...\n", n);
		3000
2880	for (i = 0; i < n; i++) {	3001	for (i = 0; i < n; i++) {
2881	GElf_Sym sym;	3002	GElf_Sym sym;
2882		3003
@@ -2884,13 +3005,12 @@ static int bpf_object__collect_externs(struct bpf_object *obj) · 1 unmodified line
2884	return -LIBBPF_ERRNO__FORMAT;	3005	return -LIBBPF_ERRNO__FORMAT;
2885	if (!sym_is_extern(&sym))	3006	if (!sym_is_extern(&sym))
2886	continue;	3007	continue;
2887	ext_name = elf_strptr(obj->efile.elf, obj->efile.strtabidx,	3008	ext_name = elf_sym_str(obj, sym.st_name);
2888	sym.st_name);
2889	if (!ext_name \|\| !ext_name[0])	3009	if (!ext_name \|\| !ext_name[0])
2890	continue;	3010	continue;
2891		3011
2892	ext = obj->externs;	3012	ext = obj->externs;
2893	ext = reallocarray(ext, obj->nr_extern + 1, sizeof(*ext));	3013	ext = libbpf_reallocarray(ext, obj->nr_extern + 1, sizeof(*ext));
2894	if (!ext)	3014	if (!ext)
2895	return -ENOMEM;	3015	return -ENOMEM;
2896	obj->externs = ext;	3016	obj->externs = ext;
@@ -3109,7 +3229,7 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object *obj, int shndx) · 212 unmodified lines
3109		3229
3110	static int bpf_program__record_reloc(struct bpf_program *prog,	3230	static int bpf_program__record_reloc(struct bpf_program *prog,
3111	struct reloc_desc *reloc_desc,	3231	struct reloc_desc *reloc_desc,
3112	__u32 insn_idx, const char *name,	3232	__u32 insn_idx, const char *sym_name,
3113	const GElf_Sym sym, const GElf_Rel rel)	3233	const GElf_Sym sym, const GElf_Rel rel)
3114	{	3234	{
3115	struct bpf_insn *insn = &prog->insns[insn_idx];	3235	struct bpf_insn *insn = &prog->insns[insn_idx];
@@ -3117,22 +3237,25 @@ static int bpf_program__record_reloc(struct bpf_program *prog, · 1 unmodified line
3117	struct bpf_object *obj = prog->obj;	3237	struct bpf_object *obj = prog->obj;
3118	__u32 shdr_idx = sym->st_shndx;	3238	__u32 shdr_idx = sym->st_shndx;
3119	enum libbpf_map_type type;	3239	enum libbpf_map_type type;
		3240	const char *sym_sec_name;
3120	struct bpf_map *map;	3241	struct bpf_map *map;
3121		3242
3122	/* sub-program call relocation */	3243	/* sub-program call relocation */
3123	if (insn->code == (BPF_JMP \| BPF_CALL)) {	3244	if (insn->code == (BPF_JMP \| BPF_CALL)) {
3124	if (insn->src_reg != BPF_PSEUDO_CALL) {	3245	if (insn->src_reg != BPF_PSEUDO_CALL) {
3125	pr_warn("incorrect bpf_call opcode\n");	3246	pr_warn("prog '%s': incorrect bpf_call opcode\n", prog->name);
3126	return -LIBBPF_ERRNO__RELOC;	3247	return -LIBBPF_ERRNO__RELOC;
3127	}	3248	}
3128	/* text_shndx can be 0, if no default "main" program exists */	3249	/* text_shndx can be 0, if no default "main" program exists */
3129	if (!shdr_idx \|\| shdr_idx != obj->efile.text_shndx) {	3250	if (!shdr_idx \|\| shdr_idx != obj->efile.text_shndx) {
3130	pr_warn("bad call relo against section %u\n", shdr_idx);	3251	sym_sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, shdr_idx));
		3252	pr_warn("prog '%s': bad call relo against '%s' in section '%s'\n",
		3253	prog->name, sym_name, sym_sec_name);
3131	return -LIBBPF_ERRNO__RELOC;	3254	return -LIBBPF_ERRNO__RELOC;
3132	}	3255	}
3133	if (sym->st_value % 8) {	3256	if (sym->st_value % BPF_INSN_SZ) {
3134	pr_warn("bad call relo offset: %zu\n",	3257	pr_warn("prog '%s': bad call relo against '%s' at offset %zu\n",
3135	(size_t)sym->st_value);	3258	prog->name, sym_name, (size_t)sym->st_value);
3136	return -LIBBPF_ERRNO__RELOC;	3259	return -LIBBPF_ERRNO__RELOC;
3137	}	3260	}
3138	reloc_desc->type = RELO_CALL;	3261	reloc_desc->type = RELO_CALL;
@@ -3143,8 +3266,8 @@ static int bpf_program__record_reloc(struct bpf_program *prog, · 4 unmodified lines
3143	}	3266	}
3144		3267
3145	if (insn->code != (BPF_LD \| BPF_IMM \| BPF_DW)) {	3268	if (insn->code != (BPF_LD \| BPF_IMM \| BPF_DW)) {
3146	pr_warn("invalid relo for insns[%d].code 0x%x\n",	3269	pr_warn("prog '%s': invalid relo against '%s' for insns[%d].code 0x%x\n",
3147	insn_idx, insn->code);	3270	prog->name, sym_name, insn_idx, insn->code);
3148	return -LIBBPF_ERRNO__RELOC;	3271	return -LIBBPF_ERRNO__RELOC;
3149	}	3272	}
3150		3273
@@ -3159,12 +3282,12 @@ static int bpf_program__record_reloc(struct bpf_program *prog, · 8 unmodified lines
3159	break;	3282	break;
3160	}	3283	}
3161	if (i >= n) {	3284	if (i >= n) {
3162	pr_warn("extern relo failed to find extern for sym %d\n",	3285	pr_warn("prog '%s': extern relo failed to find extern for '%s' (%d)\n",
3163	sym_idx);	3286	prog->name, sym_name, sym_idx);
3164	return -LIBBPF_ERRNO__RELOC;	3287	return -LIBBPF_ERRNO__RELOC;
3165	}	3288	}
3166	pr_debug("found extern #%d '%s' (sym %d) for insn %u\n",	3289	pr_debug("prog '%s': found extern #%d '%s' (sym %d) for insn #%u\n",
3167	i, ext->name, ext->sym_idx, insn_idx);	3290	prog->name, i, ext->name, ext->sym_idx, insn_idx);
3168	reloc_desc->type = RELO_EXTERN;	3291	reloc_desc->type = RELO_EXTERN;
3169	reloc_desc->insn_idx = insn_idx;	3292	reloc_desc->insn_idx = insn_idx;
3170	reloc_desc->sym_off = i; /* sym_off stores extern index */	3293	reloc_desc->sym_off = i; /* sym_off stores extern index */
@@ -3172,18 +3295,19 @@ static int bpf_program__record_reloc(struct bpf_program *prog, · 1 unmodified line
3172	}	3295	}
3173		3296
3174	if (!shdr_idx \|\| shdr_idx >= SHN_LORESERVE) {	3297	if (!shdr_idx \|\| shdr_idx >= SHN_LORESERVE) {
3175	pr_warn("invalid relo for \'%s\' in special section 0x%x; forgot to initialize global var?..\n",	3298	pr_warn("prog '%s': invalid relo against '%s' in special section 0x%x; forgot to initialize global var?..\n",
3176	name, shdr_idx);	3299	prog->name, sym_name, shdr_idx);
3177	return -LIBBPF_ERRNO__RELOC;	3300	return -LIBBPF_ERRNO__RELOC;
3178	}	3301	}
3179		3302
3180	type = bpf_object__section_to_libbpf_map_type(obj, shdr_idx);	3303	type = bpf_object__section_to_libbpf_map_type(obj, shdr_idx);
		3304	sym_sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, shdr_idx));
3181		3305
3182	/* generic map reference relocation */	3306	/* generic map reference relocation */
3183	if (type == LIBBPF_MAP_UNSPEC) {	3307	if (type == LIBBPF_MAP_UNSPEC) {
3184	if (!bpf_object__shndx_is_maps(obj, shdr_idx)) {	3308	if (!bpf_object__shndx_is_maps(obj, shdr_idx)) {
3185	pr_warn("bad map relo against section %u\n",	3309	pr_warn("prog '%s': bad map relo against '%s' in section '%s'\n",
3186	shdr_idx);	3310	prog->name, sym_name, sym_sec_name);
3187	return -LIBBPF_ERRNO__RELOC;	3311	return -LIBBPF_ERRNO__RELOC;
3188	}	3312	}
3189	for (map_idx = 0; map_idx < nr_maps; map_idx++) {	3313	for (map_idx = 0; map_idx < nr_maps; map_idx++) {
@@ -3192,14 +3316,14 @@ static int bpf_program__record_reloc(struct bpf_program *prog, · 2 unmodified lines
3192	map->sec_idx != sym->st_shndx \|\|	3316	map->sec_idx != sym->st_shndx \|\|
3193	map->sec_offset != sym->st_value)	3317	map->sec_offset != sym->st_value)
3194	continue;	3318	continue;
3195	pr_debug("found map %zd (%s, sec %d, off %zu) for insn %u\n",	3319	pr_debug("prog '%s': found map %zd (%s, sec %d, off %zu) for insn #%u\n",
3196	map_idx, map->name, map->sec_idx,	3320	prog->name, map_idx, map->name, map->sec_idx,
3197	map->sec_offset, insn_idx);	3321	map->sec_offset, insn_idx);
3198	break;	3322	break;
3199	}	3323	}
3200	if (map_idx >= nr_maps) {	3324	if (map_idx >= nr_maps) {
3201	pr_warn("map relo failed to find map for sec %u, off %zu\n",	3325	pr_warn("prog '%s': map relo failed to find map for section '%s', off %zu\n",
3202	shdr_idx, (size_t)sym->st_value);	3326	prog->name, sym_sec_name, (size_t)sym->st_value);
3203	return -LIBBPF_ERRNO__RELOC;	3327	return -LIBBPF_ERRNO__RELOC;
3204	}	3328	}
3205	reloc_desc->type = RELO_LD64;	3329	reloc_desc->type = RELO_LD64;
@@ -3211,21 +3335,22 @@ static int bpf_program__record_reloc(struct bpf_program *prog, · 5 unmodified lines
3211		3335
3212	/* global data map relocation */	3336	/* global data map relocation */
3213	if (!bpf_object__shndx_is_data(obj, shdr_idx)) {	3337	if (!bpf_object__shndx_is_data(obj, shdr_idx)) {
3214	pr_warn("bad data relo against section %u\n", shdr_idx);	3338	pr_warn("prog '%s': bad data relo against section '%s'\n",
		3339	prog->name, sym_sec_name);
3215	return -LIBBPF_ERRNO__RELOC;	3340	return -LIBBPF_ERRNO__RELOC;
3216	}	3341	}
3217	for (map_idx = 0; map_idx < nr_maps; map_idx++) {	3342	for (map_idx = 0; map_idx < nr_maps; map_idx++) {
3218	map = &obj->maps[map_idx];	3343	map = &obj->maps[map_idx];
3219	if (map->libbpf_type != type)	3344	if (map->libbpf_type != type)
3220	continue;	3345	continue;
3221	pr_debug("found data map %zd (%s, sec %d, off %zu) for insn %u\n",	3346	pr_debug("prog '%s': found data map %zd (%s, sec %d, off %zu) for insn %u\n",
3222	map_idx, map->name, map->sec_idx, map->sec_offset,	3347	prog->name, map_idx, map->name, map->sec_idx,
3223	insn_idx);	3348	map->sec_offset, insn_idx);
3224	break;	3349	break;
3225	}	3350	}
3226	if (map_idx >= nr_maps) {	3351	if (map_idx >= nr_maps) {
3227	pr_warn("data relo failed to find map for sec %u\n",	3352	pr_warn("prog '%s': data relo failed to find map for section '%s'\n",
3228	shdr_idx);	3353	prog->name, sym_sec_name);
3229	return -LIBBPF_ERRNO__RELOC;	3354	return -LIBBPF_ERRNO__RELOC;
3230	}	3355	}
3231		3356
@@ -3241,9 +3366,17 @@ bpf_program__collect_reloc(struct bpf_program prog, GElf_Shdr shdr, · 9 unmodified lines
3241	Elf_Data data, struct bpf_object obj)	3366	Elf_Data data, struct bpf_object obj)
3242	{	3367	{
3243	Elf_Data *symbols = obj->efile.symbols;	3368	Elf_Data *symbols = obj->efile.symbols;
		3369	const char relo_sec_name, sec_name;
		3370	size_t sec_idx = shdr->sh_info;
3244	int err, i, nrels;	3371	int err, i, nrels;
3245		3372
3246	pr_debug("collecting relocating info for: '%s'\n", prog->section_name);	3373	relo_sec_name = elf_sec_str(obj, shdr->sh_name);
		3374	sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx));
		3375	if (!relo_sec_name \|\| !sec_name)
		3376	return -EINVAL;
		3377
		3378	pr_debug("sec '%s': collecting relocation for section(%zu) '%s'\n",
		3379	relo_sec_name, sec_idx, sec_name);
3247	nrels = shdr->sh_size / shdr->sh_entsize;	3380	nrels = shdr->sh_size / shdr->sh_entsize;
3248		3381
3249	prog->reloc_desc = malloc(sizeof(prog->reloc_desc) nrels);	3382	prog->reloc_desc = malloc(sizeof(prog->reloc_desc) nrels);
@@ -3254,35 +3387,34 @@ bpf_program__collect_reloc(struct bpf_program prog, GElf_Shdr shdr, · 4 unmodified lines
3254	prog->nr_reloc = nrels;	3387	prog->nr_reloc = nrels;
3255		3388
3256	for (i = 0; i < nrels; i++) {	3389	for (i = 0; i < nrels; i++) {
3257	const char *name;	3390	const char *sym_name;
3258	__u32 insn_idx;	3391	__u32 insn_idx;
3259	GElf_Sym sym;	3392	GElf_Sym sym;
3260	GElf_Rel rel;	3393	GElf_Rel rel;
3261		3394
3262	if (!gelf_getrel(data, i, &rel)) {	3395	if (!gelf_getrel(data, i, &rel)) {
3263	pr_warn("relocation: failed to get %d reloc\n", i);	3396	pr_warn("sec '%s': failed to get relo #%d\n", relo_sec_name, i);
3264	return -LIBBPF_ERRNO__FORMAT;	3397	return -LIBBPF_ERRNO__FORMAT;
3265	}	3398	}
3266	if (!gelf_getsym(symbols, GELF_R_SYM(rel.r_info), &sym)) {	3399	if (!gelf_getsym(symbols, GELF_R_SYM(rel.r_info), &sym)) {
3267	pr_warn("relocation: symbol %"PRIx64" not found\n",	3400	pr_warn("sec '%s': symbol 0x%zx not found for relo #%d\n",
3268	GELF_R_SYM(rel.r_info));	3401	relo_sec_name, (size_t)GELF_R_SYM(rel.r_info), i);
3269	return -LIBBPF_ERRNO__FORMAT;	3402	return -LIBBPF_ERRNO__FORMAT;
3270	}	3403	}
3271	if (rel.r_offset % sizeof(struct bpf_insn))	3404	if (rel.r_offset % BPF_INSN_SZ) {
		3405	pr_warn("sec '%s': invalid offset 0x%zx for relo #%d\n",
		3406	relo_sec_name, (size_t)GELF_R_SYM(rel.r_info), i);
3272	return -LIBBPF_ERRNO__FORMAT;	3407	return -LIBBPF_ERRNO__FORMAT;
		3408	}
3273		3409
3274	insn_idx = rel.r_offset / sizeof(struct bpf_insn);	3410	insn_idx = rel.r_offset / BPF_INSN_SZ;
3275	name = elf_strptr(obj->efile.elf, obj->efile.strtabidx,	3411	sym_name = elf_sym_str(obj, sym.st_name) ?: "<?>";
3276	sym.st_name) ? : "<?>";
3277		3412
3278	pr_debug("relo for shdr %u, symb %zu, value %zu, type %d, bind %d, name %d (\'%s\'), insn %u\n",	3413	pr_debug("sec '%s': relo #%d: insn #%u against '%s'\n",
3279	(__u32)sym.st_shndx, (size_t)GELF_R_SYM(rel.r_info),	3414	relo_sec_name, i, insn_idx, sym_name);
3280	(size_t)sym.st_value, GELF_ST_TYPE(sym.st_info),
3281	GELF_ST_BIND(sym.st_info), sym.st_name, name,
3282	insn_idx);
3283		3415
3284	err = bpf_program__record_reloc(prog, &prog->reloc_desc[i],	3416	err = bpf_program__record_reloc(prog, &prog->reloc_desc[i],
3285	insn_idx, name, &sym, &rel);	3417	insn_idx, sym_name, &sym, &rel);
3286	if (err)	3418	if (err)
3287	return err;	3419	return err;
3288	}	3420	}
@@ -3433,8 +3565,14 @@ bpf_object__probe_loading(struct bpf_object *obj) · 144 unmodified lines
3433	return 0;	3565	return 0;
3434	}	3566	}
3435		3567
3436	static int	3568	static int probe_fd(int fd)
3437	bpf_object__probe_name(struct bpf_object *obj)	3569	{
		3570	if (fd >= 0)
		3571	close(fd);
		3572	return fd >= 0;
		3573	}
		3574
		3575	static int probe_kern_prog_name(void)
3438	{	3576	{
3439	struct bpf_load_program_attr attr;	3577	struct bpf_load_program_attr attr;
3440	struct bpf_insn insns[] = {	3578	struct bpf_insn insns[] = {
@@ -3452,16 +3590,10 @@ bpf_object__probe_name(struct bpf_object *obj) · 11 unmodified lines
3452	attr.license = "GPL";	3590	attr.license = "GPL";
3453	attr.name = "test";	3591	attr.name = "test";
3454	ret = bpf_load_program_xattr(&attr, NULL, 0);	3592	ret = bpf_load_program_xattr(&attr, NULL, 0);
3455	if (ret >= 0) {	3593	return probe_fd(ret);
3456	obj->caps.name = 1;
3457	close(ret);
3458	}
3459
3460	return 0;
3461	}	3594	}
3462		3595
3463	static int	3596	static int probe_kern_global_data(void)
3464	bpf_object__probe_global_data(struct bpf_object *obj)
3465	{	3597	{
3466	struct bpf_load_program_attr prg_attr;	3598	struct bpf_load_program_attr prg_attr;
3467	struct bpf_create_map_attr map_attr;	3599	struct bpf_create_map_attr map_attr;
@@ -3498,16 +3630,23 @@ bpf_object__probe_global_data(struct bpf_object *obj) · 30 unmodified lines
3498	prg_attr.license = "GPL";	3630	prg_attr.license = "GPL";
3499		3631
3500	ret = bpf_load_program_xattr(&prg_attr, NULL, 0);	3632	ret = bpf_load_program_xattr(&prg_attr, NULL, 0);
3501	if (ret >= 0) {
3502	obj->caps.global_data = 1;
3503	close(ret);
3504	}
3505
3506	close(map);	3633	close(map);
3507	return 0;	3634	return probe_fd(ret);
		3635	}
		3636
		3637	static int probe_kern_btf(void)
		3638	{
		3639	static const char strs[] = "\0int";
		3640	__u32 types[] = {
		3641	/* int */
		3642	BTF_TYPE_INT_ENC(1, BTF_INT_SIGNED, 0, 32, 4),
		3643	};
		3644
		3645	return probe_fd(libbpf__load_raw_btf((char *)types, sizeof(types),
		3646	strs, sizeof(strs)));
3508	}	3647	}
3509		3648
3510	static int bpf_object__probe_btf_func(struct bpf_object *obj)	3649	static int probe_kern_btf_func(void)
3511	{	3650	{
3512	static const char strs[] = "\0int\0x\0a";	3651	static const char strs[] = "\0int\0x\0a";
3513	/* void x(int a) {} */	3652	/* void x(int a) {} */
@@ -3520,20 +3659,12 @@ static int bpf_object__probe_btf_func(struct bpf_object *obj) · 6 unmodified lines
3520	/* FUNC x / / [3] */	3659	/* FUNC x / / [3] */
3521	BTF_TYPE_ENC(5, BTF_INFO_ENC(BTF_KIND_FUNC, 0, 0), 2),	3660	BTF_TYPE_ENC(5, BTF_INFO_ENC(BTF_KIND_FUNC, 0, 0), 2),
3522	};	3661	};
3523	int btf_fd;
3524		3662
3525	btf_fd = libbpf__load_raw_btf((char *)types, sizeof(types),	3663	return probe_fd(libbpf__load_raw_btf((char *)types, sizeof(types),
3526	strs, sizeof(strs));	3664	strs, sizeof(strs)));
3527	if (btf_fd >= 0) {
3528	obj->caps.btf_func = 1;
3529	close(btf_fd);
3530	return 1;
3531	}
3532
3533	return 0;
3534	}	3665	}
3535		3666
3536	static int bpf_object__probe_btf_func_global(struct bpf_object *obj)	3667	static int probe_kern_btf_func_global(void)
3537	{	3668	{
3538	static const char strs[] = "\0int\0x\0a";	3669	static const char strs[] = "\0int\0x\0a";
3539	/* static void x(int a) {} */	3670	/* static void x(int a) {} */
@@ -3546,20 +3677,12 @@ static int bpf_object__probe_btf_func_global(struct bpf_object *obj) · 6 unmodified lines
3546	/* FUNC x BTF_FUNC_GLOBAL / / [3] */	3677	/* FUNC x BTF_FUNC_GLOBAL / / [3] */
3547	BTF_TYPE_ENC(5, BTF_INFO_ENC(BTF_KIND_FUNC, 0, BTF_FUNC_GLOBAL), 2),	3678	BTF_TYPE_ENC(5, BTF_INFO_ENC(BTF_KIND_FUNC, 0, BTF_FUNC_GLOBAL), 2),
3548	};	3679	};
3549	int btf_fd;
3550		3680
3551	btf_fd = libbpf__load_raw_btf((char *)types, sizeof(types),	3681	return probe_fd(libbpf__load_raw_btf((char *)types, sizeof(types),
3552	strs, sizeof(strs));	3682	strs, sizeof(strs)));
3553	if (btf_fd >= 0) {
3554	obj->caps.btf_func_global = 1;
3555	close(btf_fd);
3556	return 1;
3557	}
3558
3559	return 0;
3560	}	3683	}
3561		3684
3562	static int bpf_object__probe_btf_datasec(struct bpf_object *obj)	3685	static int probe_kern_btf_datasec(void)
3563	{	3686	{
3564	static const char strs[] = "\0x\0.data";	3687	static const char strs[] = "\0x\0.data";
3565	/* static int a; */	3688	/* static int a; */
@@ -3573,20 +3696,12 @@ static int bpf_object__probe_btf_datasec(struct bpf_object *obj) · 7 unmodified lines
3573	BTF_TYPE_ENC(3, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4),	3696	BTF_TYPE_ENC(3, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4),
3574	BTF_VAR_SECINFO_ENC(2, 0, 4),	3697	BTF_VAR_SECINFO_ENC(2, 0, 4),
3575	};	3698	};
3576	int btf_fd;
3577
3578	btf_fd = libbpf__load_raw_btf((char *)types, sizeof(types),
3579	strs, sizeof(strs));
3580	if (btf_fd >= 0) {
3581	obj->caps.btf_datasec = 1;
3582	close(btf_fd);
3583	return 1;
3584	}
3585		3699
3586	return 0;	3700	return probe_fd(libbpf__load_raw_btf((char *)types, sizeof(types),
		3701	strs, sizeof(strs)));
3587	}	3702	}
3588		3703
3589	static int bpf_object__probe_array_mmap(struct bpf_object *obj)	3704	static int probe_kern_array_mmap(void)
3590	{	3705	{
3591	struct bpf_create_map_attr attr = {	3706	struct bpf_create_map_attr attr = {
3592	.map_type = BPF_MAP_TYPE_ARRAY,	3707	.map_type = BPF_MAP_TYPE_ARRAY,
@@ -3595,27 +3710,17 @@ static int bpf_object__probe_array_mmap(struct bpf_object *obj) · 2 unmodified lines
3595	.value_size = sizeof(int),	3710	.value_size = sizeof(int),
3596	.max_entries = 1,	3711	.max_entries = 1,
3597	};	3712	};
3598	int fd;
3599
3600	fd = bpf_create_map_xattr(&attr);
3601	if (fd >= 0) {
3602	obj->caps.array_mmap = 1;
3603	close(fd);
3604	return 1;
3605	}
3606		3713
3607	return 0;	3714	return probe_fd(bpf_create_map_xattr(&attr));
3608	}	3715	}
3609		3716
3610	static int	3717	static int probe_kern_exp_attach_type(void)
3611	bpf_object__probe_exp_attach_type(struct bpf_object *obj)
3612	{	3718	{
3613	struct bpf_load_program_attr attr;	3719	struct bpf_load_program_attr attr;
3614	struct bpf_insn insns[] = {	3720	struct bpf_insn insns[] = {
3615	BPF_MOV64_IMM(BPF_REG_0, 0),	3721	BPF_MOV64_IMM(BPF_REG_0, 0),
3616	BPF_EXIT_INSN(),	3722	BPF_EXIT_INSN(),
3617	};	3723	};
3618	int fd;
3619		3724
3620	memset(&attr, 0, sizeof(attr));	3725	memset(&attr, 0, sizeof(attr));
3621	/* use any valid combination of program type and (optional)	3726	/* use any valid combination of program type and (optional)
@@ -3629,36 +3734,91 @@ bpf_object__probe_exp_attach_type(struct bpf_object *obj) · 7 unmodified lines
3629	attr.insns_cnt = ARRAY_SIZE(insns);	3734	attr.insns_cnt = ARRAY_SIZE(insns);
3630	attr.license = "GPL";	3735	attr.license = "GPL";
3631		3736
3632	fd = bpf_load_program_xattr(&attr, NULL, 0);	3737	return probe_fd(bpf_load_program_xattr(&attr, NULL, 0));
3633	if (fd >= 0) {
3634	obj->caps.exp_attach_type = 1;
3635	close(fd);
3636	return 1;
3637	}
3638	return 0;
3639	}	3738	}
3640		3739
3641	static int	3740	static int probe_kern_probe_read_kernel(void)
3642	bpf_object__probe_caps(struct bpf_object *obj)	3741	{
3643	{	3742	struct bpf_load_program_attr attr;
3644	int (probe_fn[])(struct bpf_object obj) = {	3743	struct bpf_insn insns[] = {
3645	bpf_object__probe_name,	3744	BPF_MOV64_REG(BPF_REG_1, BPF_REG_10), /* r1 = r10 (fp) */
3646	bpf_object__probe_global_data,	3745	BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -8), /* r1 += -8 */
3647	bpf_object__probe_btf_func,	3746	BPF_MOV64_IMM(BPF_REG_2, 8), /* r2 = 8 */
3648	bpf_object__probe_btf_func_global,	3747	BPF_MOV64_IMM(BPF_REG_3, 0), /* r3 = 0 */
3649	bpf_object__probe_btf_datasec,	3748	BPF_RAW_INSN(BPF_JMP \| BPF_CALL, 0, 0, 0, BPF_FUNC_probe_read_kernel),
3650	bpf_object__probe_array_mmap,	3749	BPF_EXIT_INSN(),
3651	bpf_object__probe_exp_attach_type,
3652	};	3750	};
3653	int i, ret;
3654		3751
3655	for (i = 0; i < ARRAY_SIZE(probe_fn); i++) {	3752	memset(&attr, 0, sizeof(attr));
3656	ret = probe_fn[i](obj);	3753	attr.prog_type = BPF_PROG_TYPE_KPROBE;
3657	if (ret < 0)	3754	attr.insns = insns;
3658	pr_debug("Probe #%d failed with %d.\n", i, ret);	3755	attr.insns_cnt = ARRAY_SIZE(insns);
		3756	attr.license = "GPL";
		3757
		3758	return probe_fd(bpf_load_program_xattr(&attr, NULL, 0));
		3759	}
		3760
		3761	enum kern_feature_result {
		3762	FEAT_UNKNOWN = 0,
		3763	FEAT_SUPPORTED = 1,
		3764	FEAT_MISSING = 2,
		3765	};
		3766
		3767	typedef int (*feature_probe_fn)(void);
		3768
		3769	static struct kern_feature_desc {
		3770	const char *desc;
		3771	feature_probe_fn probe;
		3772	enum kern_feature_result res;
		3773	} feature_probes[__FEAT_CNT] = {
		3774	[FEAT_PROG_NAME] = {
		3775	"BPF program name", probe_kern_prog_name,
		3776	},
		3777	[FEAT_GLOBAL_DATA] = {
		3778	"global variables", probe_kern_global_data,
		3779	},
		3780	[FEAT_BTF] = {
		3781	"minimal BTF", probe_kern_btf,
		3782	},
		3783	[FEAT_BTF_FUNC] = {
		3784	"BTF functions", probe_kern_btf_func,
		3785	},
		3786	[FEAT_BTF_GLOBAL_FUNC] = {
		3787	"BTF global function", probe_kern_btf_func_global,
		3788	},
		3789	[FEAT_BTF_DATASEC] = {
		3790	"BTF data section and variable", probe_kern_btf_datasec,
		3791	},
		3792	[FEAT_ARRAY_MMAP] = {
		3793	"ARRAY map mmap()", probe_kern_array_mmap,
		3794	},
		3795	[FEAT_EXP_ATTACH_TYPE] = {
		3796	"BPF_PROG_LOAD expected_attach_type attribute",
		3797	probe_kern_exp_attach_type,
		3798	},
		3799	[FEAT_PROBE_READ_KERN] = {
		3800	"bpf_probe_read_kernel() helper", probe_kern_probe_read_kernel,
3659	}	3801	}
		3802	};
3660		3803
3661	return 0;	3804	static bool kernel_supports(enum kern_feature_id feat_id)
		3805	{
		3806	struct kern_feature_desc *feat = &feature_probes[feat_id];
		3807	int ret;
		3808
		3809	if (READ_ONCE(feat->res) == FEAT_UNKNOWN) {
		3810	ret = feat->probe();
		3811	if (ret > 0) {
		3812	WRITE_ONCE(feat->res, FEAT_SUPPORTED);
		3813	} else if (ret == 0) {
		3814	WRITE_ONCE(feat->res, FEAT_MISSING);
		3815	} else {
		3816	pr_warn("Detection of kernel %s support failed: %d\n", feat->desc, ret);
		3817	WRITE_ONCE(feat->res, FEAT_MISSING);
		3818	}
		3819	}
		3820
		3821	return READ_ONCE(feat->res) == FEAT_SUPPORTED;
3662	}	3822	}
3663		3823
3664	static bool map_is_reuse_compat(const struct bpf_map *map, int map_fd)	3824	static bool map_is_reuse_compat(const struct bpf_map *map, int map_fd)
@@ -3760,7 +3920,7 @@ static int bpf_object__create_map(struct bpf_object obj, struct bpf_map map) · 95 unmodified lines
3760		3920
3761	memset(&create_attr, 0, sizeof(create_attr));	3921	memset(&create_attr, 0, sizeof(create_attr));
3762		3922
3763	if (obj->caps.name)	3923	if (kernel_supports(FEAT_PROG_NAME))
3764	create_attr.name = map->name;	3924	create_attr.name = map->name;
3765	create_attr.map_ifindex = map->map_ifindex;	3925	create_attr.map_ifindex = map->map_ifindex;
3766	create_attr.map_type = def->type;	3926	create_attr.map_type = def->type;
@@ -4011,6 +4171,10 @@ struct bpf_core_spec { · 244 unmodified lines
4011	const struct btf *btf;	4171	const struct btf *btf;
4012	/* high-level spec: named fields and array indices only */	4172	/* high-level spec: named fields and array indices only */
4013	struct bpf_core_accessor spec[BPF_CORE_SPEC_MAX_LEN];	4173	struct bpf_core_accessor spec[BPF_CORE_SPEC_MAX_LEN];
		4174	/* original unresolved (no skip_mods_or_typedefs) root type ID */
		4175	__u32 root_type_id;
		4176	/* CO-RE relocation kind */
		4177	enum bpf_core_relo_kind relo_kind;
4014	/* high-level spec length */	4178	/* high-level spec length */
4015	int len;	4179	int len;
4016	/* raw, low-level spec: 1-to-1 with accessor spec string */	4180	/* raw, low-level spec: 1-to-1 with accessor spec string */
@@ -4041,8 +4205,66 @@ static bool is_flex_arr(const struct btf *btf, · 24 unmodified lines
4041	return acc->idx == btf_vlen(t) - 1;	4205	return acc->idx == btf_vlen(t) - 1;
4042	}	4206	}
4043		4207
		4208	static const char *core_relo_kind_str(enum bpf_core_relo_kind kind)
		4209	{
		4210	switch (kind) {
		4211	case BPF_FIELD_BYTE_OFFSET: return "byte_off";
		4212	case BPF_FIELD_BYTE_SIZE: return "byte_sz";
		4213	case BPF_FIELD_EXISTS: return "field_exists";
		4214	case BPF_FIELD_SIGNED: return "signed";
		4215	case BPF_FIELD_LSHIFT_U64: return "lshift_u64";
		4216	case BPF_FIELD_RSHIFT_U64: return "rshift_u64";
		4217	case BPF_TYPE_ID_LOCAL: return "local_type_id";
		4218	case BPF_TYPE_ID_TARGET: return "target_type_id";
		4219	case BPF_TYPE_EXISTS: return "type_exists";
		4220	case BPF_TYPE_SIZE: return "type_size";
		4221	case BPF_ENUMVAL_EXISTS: return "enumval_exists";
		4222	case BPF_ENUMVAL_VALUE: return "enumval_value";
		4223	default: return "unknown";
		4224	}
		4225	}
		4226
		4227	static bool core_relo_is_field_based(enum bpf_core_relo_kind kind)
		4228	{
		4229	switch (kind) {
		4230	case BPF_FIELD_BYTE_OFFSET:
		4231	case BPF_FIELD_BYTE_SIZE:
		4232	case BPF_FIELD_EXISTS:
		4233	case BPF_FIELD_SIGNED:
		4234	case BPF_FIELD_LSHIFT_U64:
		4235	case BPF_FIELD_RSHIFT_U64:
		4236	return true;
		4237	default:
		4238	return false;
		4239	}
		4240	}
		4241
		4242	static bool core_relo_is_type_based(enum bpf_core_relo_kind kind)
		4243	{
		4244	switch (kind) {
		4245	case BPF_TYPE_ID_LOCAL:
		4246	case BPF_TYPE_ID_TARGET:
		4247	case BPF_TYPE_EXISTS:
		4248	case BPF_TYPE_SIZE:
		4249	return true;
		4250	default:
		4251	return false;
		4252	}
		4253	}
		4254
		4255	static bool core_relo_is_enumval_based(enum bpf_core_relo_kind kind)
		4256	{
		4257	switch (kind) {
		4258	case BPF_ENUMVAL_EXISTS:
		4259	case BPF_ENUMVAL_VALUE:
		4260	return true;
		4261	default:
		4262	return false;
		4263	}
		4264	}
		4265
4044	/*	4266	/*
4045	* Turn bpf_field_reloc into a low- and high-level spec representation,	4267	* Turn bpf_core_relo into a low- and high-level spec representation,
4046	* validating correctness along the way, as well as calculating resulting	4268	* validating correctness along the way, as well as calculating resulting
4047	* field bit offset, specified by accessor string. Low-level spec captures	4269	* field bit offset, specified by accessor string. Low-level spec captures
4048	* every single level of nestedness, including traversing anonymous	4270	* every single level of nestedness, including traversing anonymous
@@ -4071,10 +4293,17 @@ static bool is_flex_arr(const struct btf *btf, · 22 unmodified lines
4071	* - field 'a' access (corresponds to '2' in low-level spec);	4293	* - field 'a' access (corresponds to '2' in low-level spec);
4072	* - array element #3 access (corresponds to '3' in low-level spec).	4294	* - array element #3 access (corresponds to '3' in low-level spec).
4073	*	4295	*
		4296	* Type-based relocations (TYPE_EXISTS/TYPE_SIZE,
		4297	* TYPE_ID_LOCAL/TYPE_ID_TARGET) don't capture any field information. Their
		4298	* spec and raw_spec are kept empty.
		4299	*
		4300	* Enum value-based relocations (ENUMVAL_EXISTS/ENUMVAL_VALUE) use access
		4301	* string to specify enumerator's value index that need to be relocated.
4074	*/	4302	*/
4075	static int bpf_core_spec_parse(const struct btf *btf,	4303	static int bpf_core_parse_spec(const struct btf *btf,
4076	__u32 type_id,	4304	__u32 type_id,
4077	const char *spec_str,	4305	const char *spec_str,
		4306	enum bpf_core_relo_kind relo_kind,
4078	struct bpf_core_spec *spec)	4307	struct bpf_core_spec *spec)
4079	{	4308	{
4080	int access_idx, parsed_len, i;	4309	int access_idx, parsed_len, i;
@@ -4089,6 +4318,15 @@ static int bpf_core_spec_parse(const struct btf *btf, · 8 unmodified lines
4089		4318
4090	memset(spec, 0, sizeof(*spec));	4319	memset(spec, 0, sizeof(*spec));
4091	spec->btf = btf;	4320	spec->btf = btf;
		4321	spec->root_type_id = type_id;
		4322	spec->relo_kind = relo_kind;
		4323
		4324	/* type-based relocations don't have a field access string */
		4325	if (core_relo_is_type_based(relo_kind)) {
		4326	if (strcmp(spec_str, "0"))
		4327	return -EINVAL;
		4328	return 0;
		4329	}
4092		4330
4093	/* parse spec_str="0:1:2:3:4" into array raw_spec=[0, 1, 2, 3, 4] */	4331	/* parse spec_str="0:1:2:3:4" into array raw_spec=[0, 1, 2, 3, 4] */
4094	while (*spec_str) {	4332	while (*spec_str) {
@@ -4105,16 +4343,28 @@ static int bpf_core_spec_parse(const struct btf *btf, · 10 unmodified lines
4105	if (spec->raw_len == 0)	4343	if (spec->raw_len == 0)
4106	return -EINVAL;	4344	return -EINVAL;
4107		4345
4108	/* first spec value is always reloc type array index */
4109	t = skip_mods_and_typedefs(btf, type_id, &id);	4346	t = skip_mods_and_typedefs(btf, type_id, &id);
4110	if (!t)	4347	if (!t)
4111	return -EINVAL;	4348	return -EINVAL;
4112		4349
4113	access_idx = spec->raw_spec[0];	4350	access_idx = spec->raw_spec[0];
4114	spec->spec[0].type_id = id;	4351	acc = &spec->spec[0];
4115	spec->spec[0].idx = access_idx;	4352	acc->type_id = id;
		4353	acc->idx = access_idx;
4116	spec->len++;	4354	spec->len++;
4117		4355
		4356	if (core_relo_is_enumval_based(relo_kind)) {
		4357	if (!btf_is_enum(t) \|\| spec->raw_len > 1 \|\| access_idx >= btf_vlen(t))
		4358	return -EINVAL;
		4359
		4360	/* record enumerator name in a first accessor */
		4361	acc->name = btf__name_by_offset(btf, btf_enum(t)[access_idx].name_off);
		4362	return 0;
		4363	}
		4364
		4365	if (!core_relo_is_field_based(relo_kind))
		4366	return -EINVAL;
		4367
4118	sz = btf__resolve_size(btf, id);	4368	sz = btf__resolve_size(btf, id);
4119	if (sz < 0)	4369	if (sz < 0)
4120	return sz;	4370	return sz;
@@ -4172,8 +4422,8 @@ static int bpf_core_spec_parse(const struct btf *btf, · 51 unmodified lines
4172	return sz;	4422	return sz;
4173	spec->bit_offset += access_idx * sz * 8;	4423	spec->bit_offset += access_idx * sz * 8;
4174	} else {	4424	} else {
4175	pr_warn("relo for [%u] %s (at idx %d) captures type [%d] of unexpected kind %d\n",	4425	pr_warn("relo for [%u] %s (at idx %d) captures type [%d] of unexpected kind %s\n",
4176	type_id, spec_str, i, id, btf_kind(t));	4426	type_id, spec_str, i, id, btf_kind_str(t));
4177	return -EINVAL;	4427	return -EINVAL;
4178	}	4428	}
4179	}	4429	}
@@ -4223,16 +4473,16 @@ static struct ids_vec bpf_core_find_cands(const struct btf local_btf, · 43 unmodified lines
4223	{	4473	{
4224	size_t local_essent_len, targ_essent_len;	4474	size_t local_essent_len, targ_essent_len;
4225	const char local_name, targ_name;	4475	const char local_name, targ_name;
4226	const struct btf_type *t;	4476	const struct btf_type t, local_t;
4227	struct ids_vec *cand_ids;	4477	struct ids_vec *cand_ids;
4228	__u32 *new_ids;	4478	__u32 *new_ids;
4229	int i, err, n;	4479	int i, err, n;
4230		4480
4231	t = btf__type_by_id(local_btf, local_type_id);	4481	local_t = btf__type_by_id(local_btf, local_type_id);
4232	if (!t)	4482	if (!local_t)
4233	return ERR_PTR(-EINVAL);	4483	return ERR_PTR(-EINVAL);
4234		4484
4235	local_name = btf__name_by_offset(local_btf, t->name_off);	4485	local_name = btf__name_by_offset(local_btf, local_t->name_off);
4236	if (str_is_empty(local_name))	4486	if (str_is_empty(local_name))
4237	return ERR_PTR(-EINVAL);	4487	return ERR_PTR(-EINVAL);
4238	local_essent_len = bpf_core_essential_name_len(local_name);	4488	local_essent_len = bpf_core_essential_name_len(local_name);
@@ -4244,12 +4494,11 @@ static struct ids_vec bpf_core_find_cands(const struct btf local_btf, · 5 unmodified lines
4244	n = btf__get_nr_types(targ_btf);	4494	n = btf__get_nr_types(targ_btf);
4245	for (i = 1; i <= n; i++) {	4495	for (i = 1; i <= n; i++) {
4246	t = btf__type_by_id(targ_btf, i);	4496	t = btf__type_by_id(targ_btf, i);
4247	targ_name = btf__name_by_offset(targ_btf, t->name_off);	4497	if (btf_kind(t) != btf_kind(local_t))
4248	if (str_is_empty(targ_name))
4249	continue;	4498	continue;
4250		4499
4251	t = skip_mods_and_typedefs(targ_btf, i, NULL);	4500	targ_name = btf__name_by_offset(targ_btf, t->name_off);
4252	if (!btf_is_composite(t) && !btf_is_array(t))	4501	if (str_is_empty(targ_name))
4253	continue;	4502	continue;
4254		4503
4255	targ_essent_len = bpf_core_essential_name_len(targ_name);	4504	targ_essent_len = bpf_core_essential_name_len(targ_name);
@@ -4257,11 +4506,12 @@ static struct ids_vec bpf_core_find_cands(const struct btf local_btf, · 1 unmodified line
4257	continue;	4506	continue;
4258		4507
4259	if (strncmp(local_name, targ_name, local_essent_len) == 0) {	4508	if (strncmp(local_name, targ_name, local_essent_len) == 0) {
4260	pr_debug("[%d] %s: found candidate [%d] %s\n",	4509	pr_debug("CO-RE relocating [%d] %s %s: found target candidate [%d] %s %s\n",
4261	local_type_id, local_name, i, targ_name);	4510	local_type_id, btf_kind_str(local_t),
4262	new_ids = reallocarray(cand_ids->data,	4511	local_name, i, btf_kind_str(t), targ_name);
4263	cand_ids->len + 1,	4512	new_ids = libbpf_reallocarray(cand_ids->data,
4264	sizeof(*cand_ids->data));	4513	cand_ids->len + 1,
		4514	sizeof(*cand_ids->data));
4265	if (!new_ids) {	4515	if (!new_ids) {
4266	err = -ENOMEM;	4516	err = -ENOMEM;
4267	goto err_out;	4517	goto err_out;
@@ -4276,8 +4526,9 @@ err_out: · 8 unmodified lines
4276	return ERR_PTR(err);	4526	return ERR_PTR(err);
4277	}	4527	}
4278		4528
4279	/* Check two types for compatibility, skipping const/volatile/restrict and	4529	/* Check two types for compatibility for the purpose of field access
4280	* typedefs, to ensure we are relocating compatible entities:	4530	* relocation. const/volatile/restrict and typedefs are skipped to ensure we
		4531	* are relocating semantically compatible entities:
4281	* - any two STRUCTs/UNIONs are compatible and can be mixed;	4532	* - any two STRUCTs/UNIONs are compatible and can be mixed;
4282	* - any two FWDs are compatible, if their names match (modulo flavor suffix);	4533	* - any two FWDs are compatible, if their names match (modulo flavor suffix);
4283	* - any two PTRs are always compatible;	4534	* - any two PTRs are always compatible;
@@ -4411,25 +4662,119 @@ static int bpf_core_match_member(const struct btf *local_btf, · 127 unmodified lines
4411	/* matching named field */	4662	/* matching named field */
4412	struct bpf_core_accessor *targ_acc;	4663	struct bpf_core_accessor *targ_acc;
4413		4664
4414	targ_acc = &spec->spec[spec->len++];	4665	targ_acc = &spec->spec[spec->len++];
4415	targ_acc->type_id = targ_id;	4666	targ_acc->type_id = targ_id;
4416	targ_acc->idx = i;	4667	targ_acc->idx = i;
4417	targ_acc->name = targ_name;	4668	targ_acc->name = targ_name;
		4669
		4670	*next_targ_id = m->type;
		4671	found = bpf_core_fields_are_compat(local_btf,
		4672	local_member->type,
		4673	targ_btf, m->type);
		4674	if (!found)
		4675	spec->len--; /* pop accessor */
		4676	return found;
		4677	}
		4678	/* member turned out not to be what we looked for */
		4679	spec->bit_offset -= bit_offset;
		4680	spec->raw_len--;
		4681	}
		4682
		4683	return 0;
		4684	}
		4685
		4686	/* Check local and target types for compatibility. This check is used for
		4687	* type-based CO-RE relocations and follow slightly different rules than
		4688	* field-based relocations. This function assumes that root types were already
		4689	* checked for name match. Beyond that initial root-level name check, names
		4690	* are completely ignored. Compatibility rules are as follows:
		4691	* - any two STRUCTs/UNIONs/FWDs/ENUMs/INTs are considered compatible, but
		4692	* kind should match for local and target types (i.e., STRUCT is not
		4693	* compatible with UNION);
		4694	* - for ENUMs, the size is ignored;
		4695	* - for INT, size and signedness are ignored;
		4696	* - for ARRAY, dimensionality is ignored, element types are checked for
		4697	* compatibility recursively;
		4698	* - CONST/VOLATILE/RESTRICT modifiers are ignored;
		4699	* - TYPEDEFs/PTRs are compatible if types they pointing to are compatible;
		4700	* - FUNC_PROTOs are compatible if they have compatible signature: same
		4701	* number of input args and compatible return and argument types.
		4702	* These rules are not set in stone and probably will be adjusted as we get
		4703	* more experience with using BPF CO-RE relocations.
		4704	*/
		4705	static int bpf_core_types_are_compat(const struct btf *local_btf, __u32 local_id,
		4706	const struct btf *targ_btf, __u32 targ_id)
		4707	{
		4708	const struct btf_type local_type, targ_type;
		4709	int depth = 32; /* max recursion depth */
		4710
		4711	/* caller made sure that names match (ignoring flavor suffix) */
		4712	local_type = btf__type_by_id(local_btf, local_id);
		4713	targ_type = btf__type_by_id(targ_btf, targ_id);
		4714	if (btf_kind(local_type) != btf_kind(targ_type))
		4715	return 0;
		4716
		4717	recur:
		4718	depth--;
		4719	if (depth < 0)
		4720	return -EINVAL;
4418		4721
4419	*next_targ_id = m->type;	4722	local_type = skip_mods_and_typedefs(local_btf, local_id, &local_id);
4420	found = bpf_core_fields_are_compat(local_btf,	4723	targ_type = skip_mods_and_typedefs(targ_btf, targ_id, &targ_id);
4421	local_member->type,	4724	if (!local_type \|\| !targ_type)
4422	targ_btf, m->type);	4725	return -EINVAL;
4423	if (!found)	4726
4424	spec->len--; /* pop accessor */	4727	if (btf_kind(local_type) != btf_kind(targ_type))
4425	return found;	4728	return 0;
		4729
		4730	switch (btf_kind(local_type)) {
		4731	case BTF_KIND_UNKN:
		4732	case BTF_KIND_STRUCT:
		4733	case BTF_KIND_UNION:
		4734	case BTF_KIND_ENUM:
		4735	case BTF_KIND_FWD:
		4736	return 1;
		4737	case BTF_KIND_INT:
		4738	/* just reject deprecated bitfield-like integers; all other
		4739	* integers are by default compatible between each other
		4740	*/
		4741	return btf_int_offset(local_type) == 0 && btf_int_offset(targ_type) == 0;
		4742	case BTF_KIND_PTR:
		4743	local_id = local_type->type;
		4744	targ_id = targ_type->type;
		4745	goto recur;
		4746	case BTF_KIND_ARRAY:
		4747	local_id = btf_array(local_type)->type;
		4748	targ_id = btf_array(targ_type)->type;
		4749	goto recur;
		4750	case BTF_KIND_FUNC_PROTO: {
		4751	struct btf_param *local_p = btf_params(local_type);
		4752	struct btf_param *targ_p = btf_params(targ_type);
		4753	__u16 local_vlen = btf_vlen(local_type);
		4754	__u16 targ_vlen = btf_vlen(targ_type);
		4755	int i, err;
		4756
		4757	if (local_vlen != targ_vlen)
		4758	return 0;
		4759
		4760	for (i = 0; i < local_vlen; i++, local_p++, targ_p++) {
		4761	skip_mods_and_typedefs(local_btf, local_p->type, &local_id);
		4762	skip_mods_and_typedefs(targ_btf, targ_p->type, &targ_id);
		4763	err = bpf_core_types_are_compat(local_btf, local_id, targ_btf, targ_id);
		4764	if (err <= 0)
		4765	return err;
4426	}	4766	}
4427	/* member turned out not to be what we looked for */
4428	spec->bit_offset -= bit_offset;
4429	spec->raw_len--;
4430	}
4431		4767
4432	return 0;	4768	/* tail recurse for return type check */
		4769	skip_mods_and_typedefs(local_btf, local_type->type, &local_id);
		4770	skip_mods_and_typedefs(targ_btf, targ_type->type, &targ_id);
		4771	goto recur;
		4772	}
		4773	default:
		4774	pr_warn("unexpected kind %s relocated, local [%d], target [%d]\n",
		4775	btf_kind_str(local_type), local_id, targ_id);
		4776	return 0;
		4777	}
4433	}	4778	}
4434		4779
4435	/*	4780	/*
@@ -4447,10 +4792,51 @@ static int bpf_core_spec_match(struct bpf_core_spec *local_spec, · 11 unmodified lines
4447		4792
4448	memset(targ_spec, 0, sizeof(*targ_spec));	4793	memset(targ_spec, 0, sizeof(*targ_spec));
4449	targ_spec->btf = targ_btf;	4794	targ_spec->btf = targ_btf;
		4795	targ_spec->root_type_id = targ_id;
		4796	targ_spec->relo_kind = local_spec->relo_kind;
		4797
		4798	if (core_relo_is_type_based(local_spec->relo_kind)) {
		4799	return bpf_core_types_are_compat(local_spec->btf,
		4800	local_spec->root_type_id,
		4801	targ_btf, targ_id);
		4802	}
4450		4803
4451	local_acc = &local_spec->spec[0];	4804	local_acc = &local_spec->spec[0];
4452	targ_acc = &targ_spec->spec[0];	4805	targ_acc = &targ_spec->spec[0];
4453		4806
		4807	if (core_relo_is_enumval_based(local_spec->relo_kind)) {
		4808	size_t local_essent_len, targ_essent_len;
		4809	const struct btf_enum *e;
		4810	const char *targ_name;
		4811
		4812	/* has to resolve to an enum */
		4813	targ_type = skip_mods_and_typedefs(targ_spec->btf, targ_id, &targ_id);
		4814	if (!btf_is_enum(targ_type))
		4815	return 0;
		4816
		4817	local_essent_len = bpf_core_essential_name_len(local_acc->name);
		4818
		4819	for (i = 0, e = btf_enum(targ_type); i < btf_vlen(targ_type); i++, e++) {
		4820	targ_name = btf__name_by_offset(targ_spec->btf, e->name_off);
		4821	targ_essent_len = bpf_core_essential_name_len(targ_name);
		4822	if (targ_essent_len != local_essent_len)
		4823	continue;
		4824	if (strncmp(local_acc->name, targ_name, local_essent_len) == 0) {
		4825	targ_acc->type_id = targ_id;
		4826	targ_acc->idx = i;
		4827	targ_acc->name = targ_name;
		4828	targ_spec->len++;
		4829	targ_spec->raw_spec[targ_spec->raw_len] = targ_acc->idx;
		4830	targ_spec->raw_len++;
		4831	return 1;
		4832	}
		4833	}
		4834	return 0;
		4835	}
		4836
		4837	if (!core_relo_is_field_based(local_spec->relo_kind))
		4838	return -EINVAL;
		4839
4454	for (i = 0; i < local_spec->len; i++, local_acc++, targ_acc++) {	4840	for (i = 0; i < local_spec->len; i++, local_acc++, targ_acc++) {
4455	targ_type = skip_mods_and_typedefs(targ_spec->btf, targ_id,	4841	targ_type = skip_mods_and_typedefs(targ_spec->btf, targ_id,
4456	&targ_id);	4842	&targ_id);
@@ -4507,18 +4893,29 @@ static int bpf_core_spec_match(struct bpf_core_spec *local_spec, · 50 unmodified lines
4507	}	4893	}
4508		4894
4509	static int bpf_core_calc_field_relo(const struct bpf_program *prog,	4895	static int bpf_core_calc_field_relo(const struct bpf_program *prog,
4510	const struct bpf_field_reloc *relo,	4896	const struct bpf_core_relo *relo,
4511	const struct bpf_core_spec *spec,	4897	const struct bpf_core_spec *spec,
4512	__u32 val, bool validate)	4898	__u32 val, bool validate)
4513	{	4899	{
4514	const struct bpf_core_accessor *acc = &spec->spec[spec->len - 1];	4900	const struct bpf_core_accessor *acc;
4515	const struct btf_type *t = btf__type_by_id(spec->btf, acc->type_id);	4901	const struct btf_type *t;
4516	__u32 byte_off, byte_sz, bit_off, bit_sz;	4902	__u32 byte_off, byte_sz, bit_off, bit_sz;
4517	const struct btf_member *m;	4903	const struct btf_member *m;
4518	const struct btf_type *mt;	4904	const struct btf_type *mt;
4519	bool bitfield;	4905	bool bitfield;
4520	__s64 sz;	4906	__s64 sz;
4521		4907
		4908	if (relo->kind == BPF_FIELD_EXISTS) {
		4909	*val = spec ? 1 : 0;
		4910	return 0;
		4911	}
		4912
		4913	if (!spec)
		4914	return -EUCLEAN; /* request instruction poisoning */
		4915
		4916	acc = &spec->spec[spec->len - 1];
		4917	t = btf__type_by_id(spec->btf, acc->type_id);
		4918
4522	/* a[n] accessor needs special handling */	4919	/* a[n] accessor needs special handling */
4523	if (!acc->name) {	4920	if (!acc->name) {
4524	if (relo->kind == BPF_FIELD_BYTE_OFFSET) {	4921	if (relo->kind == BPF_FIELD_BYTE_OFFSET) {
@@ -4604,21 +5001,158 @@ static int bpf_core_calc_field_relo(const struct bpf_program *prog, · 79 unmodified lines
4604	break;	5001	break;
4605	case BPF_FIELD_EXISTS:	5002	case BPF_FIELD_EXISTS:
4606	default:	5003	default:
4607	pr_warn("prog '%s': unknown relo %d at insn #%d\n",	5004	return -EOPNOTSUPP;
4608	bpf_program__title(prog, false),	5005	}
4609	relo->kind, relo->insn_off / 8);	5006
4610	return -EINVAL;	5007	return 0;
		5008	}
		5009
		5010	static int bpf_core_calc_type_relo(const struct bpf_core_relo *relo,
		5011	const struct bpf_core_spec *spec,
		5012	__u32 *val)
		5013	{
		5014	__s64 sz;
		5015
		5016	/* type-based relos return zero when target type is not found */
		5017	if (!spec) {
		5018	*val = 0;
		5019	return 0;
		5020	}
		5021
		5022	switch (relo->kind) {
		5023	case BPF_TYPE_ID_TARGET:
		5024	*val = spec->root_type_id;
		5025	break;
		5026	case BPF_TYPE_EXISTS:
		5027	*val = 1;
		5028	break;
		5029	case BPF_TYPE_SIZE:
		5030	sz = btf__resolve_size(spec->btf, spec->root_type_id);
		5031	if (sz < 0)
		5032	return -EINVAL;
		5033	*val = sz;
		5034	break;
		5035	case BPF_TYPE_ID_LOCAL:
		5036	/* BPF_TYPE_ID_LOCAL is handled specially and shouldn't get here */
		5037	default:
		5038	return -EOPNOTSUPP;
		5039	}
		5040
		5041	return 0;
		5042	}
		5043
		5044	static int bpf_core_calc_enumval_relo(const struct bpf_core_relo *relo,
		5045	const struct bpf_core_spec *spec,
		5046	__u32 *val)
		5047	{
		5048	const struct btf_type *t;
		5049	const struct btf_enum *e;
		5050
		5051	switch (relo->kind) {
		5052	case BPF_ENUMVAL_EXISTS:
		5053	*val = spec ? 1 : 0;
		5054	break;
		5055	case BPF_ENUMVAL_VALUE:
		5056	if (!spec)
		5057	return -EUCLEAN; /* request instruction poisoning */
		5058	t = btf__type_by_id(spec->btf, spec->spec[0].type_id);
		5059	e = btf_enum(t) + spec->spec[0].idx;
		5060	*val = e->val;
		5061	break;
		5062	default:
		5063	return -EOPNOTSUPP;
4611	}	5064	}
4612		5065
4613	return 0;	5066	return 0;
4614	}	5067	}
4615		5068
		5069	struct bpf_core_relo_res
		5070	{
		5071	/* expected value in the instruction, unless validate == false */
		5072	__u32 orig_val;
		5073	/* new value that needs to be patched up to */
		5074	__u32 new_val;
		5075	/* relocation unsuccessful, poison instruction, but don't fail load */
		5076	bool poison;
		5077	/* some relocations can't be validated against orig_val */
		5078	bool validate;
		5079	};
		5080
		5081	/* Calculate original and target relocation values, given local and target
		5082	* specs and relocation kind. These values are calculated for each candidate.
		5083	* If there are multiple candidates, resulting values should all be consistent
		5084	* with each other. Otherwise, libbpf will refuse to proceed due to ambiguity.
		5085	* If instruction has to be poisoned, *poison will be set to true.
		5086	*/
		5087	static int bpf_core_calc_relo(const struct bpf_program *prog,
		5088	const struct bpf_core_relo *relo,
		5089	int relo_idx,
		5090	const struct bpf_core_spec *local_spec,
		5091	const struct bpf_core_spec *targ_spec,
		5092	struct bpf_core_relo_res *res)
		5093	{
		5094	int err = -EOPNOTSUPP;
		5095
		5096	res->orig_val = 0;
		5097	res->new_val = 0;
		5098	res->poison = false;
		5099	res->validate = true;
		5100
		5101	if (core_relo_is_field_based(relo->kind)) {
		5102	err = bpf_core_calc_field_relo(prog, relo, local_spec, &res->orig_val, &res->validate);
		5103	err = err ?: bpf_core_calc_field_relo(prog, relo, targ_spec, &res->new_val, NULL);
		5104	} else if (core_relo_is_type_based(relo->kind)) {
		5105	err = bpf_core_calc_type_relo(relo, local_spec, &res->orig_val);
		5106	err = err ?: bpf_core_calc_type_relo(relo, targ_spec, &res->new_val);
		5107	} else if (core_relo_is_enumval_based(relo->kind)) {
		5108	err = bpf_core_calc_enumval_relo(relo, local_spec, &res->orig_val);
		5109	err = err ?: bpf_core_calc_enumval_relo(relo, targ_spec, &res->new_val);
		5110	}
		5111
		5112	if (err == -EUCLEAN) {
		5113	/* EUCLEAN is used to signal instruction poisoning request */
		5114	res->poison = true;
		5115	err = 0;
		5116	} else if (err == -EOPNOTSUPP) {
		5117	/* EOPNOTSUPP means unknown/unsupported relocation */
		5118	pr_warn("prog '%s': relo #%d: unrecognized CO-RE relocation %s (%d) at insn #%d\n",
		5119	bpf_program__title(prog, false), relo_idx,
		5120	core_relo_kind_str(relo->kind), relo->kind, relo->insn_off / 8);
		5121	}
		5122
		5123	return err;
		5124	}
		5125
		5126	/*
		5127	* Turn instruction for which CO_RE relocation failed into invalid one with
		5128	* distinct signature.
		5129	*/
		5130	static void bpf_core_poison_insn(struct bpf_program *prog, int relo_idx,
		5131	int insn_idx, struct bpf_insn *insn)
		5132	{
		5133	pr_debug("prog '%s': relo #%d: substituting insn #%d w/ invalid insn\n",
		5134	bpf_program__title(prog, false), relo_idx, insn_idx);
		5135	insn->code = BPF_JMP \| BPF_CALL;
		5136	insn->dst_reg = 0;
		5137	insn->src_reg = 0;
		5138	insn->off = 0;
		5139	/* if this instruction is reachable (not a dead code),
		5140	* verifier will complain with the following message:
		5141	* invalid func unknown#195896080
		5142	*/
		5143	insn->imm = 195896080; /* => 0xbad2310 => "bad relo" */
		5144	}
		5145
		5146	static bool is_ldimm64(struct bpf_insn *insn)
		5147	{
		5148	return insn->code == (BPF_LD \| BPF_IMM \| BPF_DW);
		5149	}
		5150
4616	/*	5151	/*
4617	* Patch relocatable BPF instruction.	5152	* Patch relocatable BPF instruction.
4618	*	5153	*
4619	* Patched value is determined by relocation kind and target specification.	5154	* Patched value is determined by relocation kind and target specification.
4620	* For field existence relocation target spec will be NULL if field is not	5155	* For existence relocations target spec will be NULL if field/type is not found.
4621	* found.
4622	* Expected insn->imm value is determined using relocation kind and local	5156	* Expected insn->imm value is determined using relocation kind and local
4623	* spec, and is checked before patching instruction. If actual insn->imm value	5157	* spec, and is checked before patching instruction. If actual insn->imm value
4624	* is wrong, bail out with error.	5158	* is wrong, bail out with error.
@@ -4626,58 +5160,43 @@ static int bpf_core_calc_field_relo(const struct bpf_program *prog, · 1 unmodified line
4626	* Currently three kinds of BPF instructions are supported:	5160	* Currently three kinds of BPF instructions are supported:
4627	* 1. rX = <imm> (assignment with immediate operand);	5161	* 1. rX = <imm> (assignment with immediate operand);
4628	* 2. rX += <imm> (arithmetic operations with immediate operand);	5162	* 2. rX += <imm> (arithmetic operations with immediate operand);
		5163	* 3. rX = <imm64> (load with 64-bit immediate value).
4629	*/	5164	*/
4630	static int bpf_core_reloc_insn(struct bpf_program *prog,	5165	static int bpf_core_patch_insn(struct bpf_program *prog,
4631	const struct bpf_field_reloc *relo,	5166	const struct bpf_core_relo *relo,
4632	int relo_idx,	5167	int relo_idx,
4633	const struct bpf_core_spec *local_spec,	5168	const struct bpf_core_relo_res *res)
4634	const struct bpf_core_spec *targ_spec)
4635	{	5169	{
4636	__u32 orig_val, new_val;	5170	__u32 orig_val, new_val;
4637	struct bpf_insn *insn;	5171	struct bpf_insn *insn;
4638	bool validate = true;	5172	int insn_idx;
4639	int insn_idx, err;
4640	__u8 class;	5173	__u8 class;
4641		5174
4642	if (relo->insn_off % sizeof(struct bpf_insn))	5175	if (relo->insn_off % BPF_INSN_SZ)
4643	return -EINVAL;	5176	return -EINVAL;
4644	insn_idx = relo->insn_off / sizeof(struct bpf_insn);	5177	insn_idx = relo->insn_off / BPF_INSN_SZ;
4645	insn = &prog->insns[insn_idx];	5178	insn = &prog->insns[insn_idx];
4646	class = BPF_CLASS(insn->code);	5179	class = BPF_CLASS(insn->code);
4647		5180
4648	if (relo->kind == BPF_FIELD_EXISTS) {	5181	if (res->poison) {
4649	orig_val = 1; /* can't generate EXISTS relo w/o local field */	5182	/* poison second part of ldimm64 to avoid confusing error from
4650	new_val = targ_spec ? 1 : 0;	5183	* verifier about "unknown opcode 00"
4651	} else if (!targ_spec) {
4652	pr_debug("prog '%s': relo #%d: substituting insn #%d w/ invalid insn\n",
4653	bpf_program__title(prog, false), relo_idx, insn_idx);
4654	insn->code = BPF_JMP \| BPF_CALL;
4655	insn->dst_reg = 0;
4656	insn->src_reg = 0;
4657	insn->off = 0;
4658	/* if this instruction is reachable (not a dead code),
4659	* verifier will complain with the following message:
4660	* invalid func unknown#195896080
4661	*/	5184	*/
4662	insn->imm = 195896080; /* => 0xbad2310 => "bad relo" */	5185	if (is_ldimm64(insn))
		5186	bpf_core_poison_insn(prog, relo_idx, insn_idx + 1, insn + 1);
		5187	bpf_core_poison_insn(prog, relo_idx, insn_idx, insn);
4663	return 0;	5188	return 0;
4664	} else {
4665	err = bpf_core_calc_field_relo(prog, relo, local_spec,
4666	&orig_val, &validate);
4667	if (err)
4668	return err;
4669	err = bpf_core_calc_field_relo(prog, relo, targ_spec,
4670	&new_val, NULL);
4671	if (err)
4672	return err;
4673	}	5189	}
4674		5190
		5191	orig_val = res->orig_val;
		5192	new_val = res->new_val;
		5193
4675	switch (class) {	5194	switch (class) {
4676	case BPF_ALU:	5195	case BPF_ALU:
4677	case BPF_ALU64:	5196	case BPF_ALU64:
4678	if (BPF_SRC(insn->code) != BPF_K)	5197	if (BPF_SRC(insn->code) != BPF_K)
4679	return -EINVAL;	5198	return -EINVAL;
4680	if (validate && insn->imm != orig_val) {	5199	if (res->validate && insn->imm != orig_val) {
4681	pr_warn("prog '%s': relo #%d: unexpected insn #%d (ALU/ALU64) value: got %u, exp %u -> %u\n",	5200	pr_warn("prog '%s': relo #%d: unexpected insn #%d (ALU/ALU64) value: got %u, exp %u -> %u\n",
4682	bpf_program__title(prog, false), relo_idx,	5201	bpf_program__title(prog, false), relo_idx,
4683	insn_idx, insn->imm, orig_val, new_val);	5202	insn_idx, insn->imm, orig_val, new_val);
@@ -4692,8 +5211,8 @@ static int bpf_core_reloc_insn(struct bpf_program *prog, · 8 unmodified lines
4692	case BPF_LDX:	5211	case BPF_LDX:
4693	case BPF_ST:	5212	case BPF_ST:
4694	case BPF_STX:	5213	case BPF_STX:
4695	if (validate && insn->off != orig_val) {	5214	if (res->validate && insn->off != orig_val) {
4696	pr_warn("prog '%s': relo #%d: unexpected insn #%d (LD/LDX/ST/STX) value: got %u, exp %u -> %u\n",	5215	pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDX/ST/STX) value: got %u, exp %u -> %u\n",
4697	bpf_program__title(prog, false), relo_idx,	5216	bpf_program__title(prog, false), relo_idx,
4698	insn_idx, insn->off, orig_val, new_val);	5217	insn_idx, insn->off, orig_val, new_val);
4699	return -EINVAL;	5218	return -EINVAL;
@@ -4710,8 +5229,37 @@ static int bpf_core_reloc_insn(struct bpf_program *prog, · 10 unmodified lines
4710	bpf_program__title(prog, false), relo_idx, insn_idx,	5229	bpf_program__title(prog, false), relo_idx, insn_idx,
4711	orig_val, new_val);	5230	orig_val, new_val);
4712	break;	5231	break;
		5232	case BPF_LD: {
		5233	__u64 imm;
		5234
		5235	if (!is_ldimm64(insn) \|\|
		5236	insn[0].src_reg != 0 \|\| insn[0].off != 0 \|\|
		5237	insn_idx + 1 >= prog->insns_cnt \|\|
		5238	insn[1].code != 0 \|\| insn[1].dst_reg != 0 \|\|
		5239	insn[1].src_reg != 0 \|\| insn[1].off != 0) {
		5240	pr_warn("prog '%s': relo #%d: insn #%d (LDIMM64) has unexpected form\n",
		5241	bpf_program__title(prog, false), relo_idx, insn_idx);
		5242	return -EINVAL;
		5243	}
		5244
		5245	imm = insn[0].imm + ((__u64)insn[1].imm << 32);
		5246	if (res->validate && imm != orig_val) {
		5247	pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDIMM64) value: got %llu, exp %u -> %u\n",
		5248	bpf_program__title(prog, false), relo_idx,
		5249	insn_idx, (unsigned long long)imm,
		5250	orig_val, new_val);
		5251	return -EINVAL;
		5252	}
		5253
		5254	insn[0].imm = new_val;
		5255	insn[1].imm = 0; /* currently only 32-bit values are supported */
		5256	pr_debug("prog '%s': relo #%d: patched insn #%d (LDIMM64) imm64 %llu -> %u\n",
		5257	bpf_program__title(prog, false), relo_idx, insn_idx,
		5258	(unsigned long long)imm, new_val);
		5259	break;
		5260	}
4713	default:	5261	default:
4714	pr_warn("prog '%s': relo #%d: trying to relocate unrecognized insn #%d, code:%x, src:%x, dst:%x, off:%x, imm:%x\n",	5262	pr_warn("prog '%s': relo #%d: trying to relocate unrecognized insn #%d, code:0x%x, src:0x%x, dst:0x%x, off:0x%x, imm:0x%x\n",
4715	bpf_program__title(prog, false), relo_idx,	5263	bpf_program__title(prog, false), relo_idx,
4716	insn_idx, insn->code, insn->src_reg, insn->dst_reg,	5264	insn_idx, insn->code, insn->src_reg, insn->dst_reg,
4717	insn->off, insn->imm);	5265	insn->off, insn->imm);
@@ -4728,29 +5276,48 @@ static int bpf_core_reloc_insn(struct bpf_program *prog, · 10 unmodified lines
4728	static void bpf_core_dump_spec(int level, const struct bpf_core_spec *spec)	5276	static void bpf_core_dump_spec(int level, const struct bpf_core_spec *spec)
4729	{	5277	{
4730	const struct btf_type *t;	5278	const struct btf_type *t;
		5279	const struct btf_enum *e;
4731	const char *s;	5280	const char *s;
4732	__u32 type_id;	5281	__u32 type_id;
4733	int i;	5282	int i;
4734		5283
4735	type_id = spec->spec[0].type_id;	5284	type_id = spec->root_type_id;
4736	t = btf__type_by_id(spec->btf, type_id);	5285	t = btf__type_by_id(spec->btf, type_id);
4737	s = btf__name_by_offset(spec->btf, t->name_off);	5286	s = btf__name_by_offset(spec->btf, t->name_off);
4738	libbpf_print(level, "[%u] %s + ", type_id, s);
4739		5287
4740	for (i = 0; i < spec->raw_len; i++)	5288	libbpf_print(level, "[%u] %s %s", type_id, btf_kind_str(t), str_is_empty(s) ? "<anon>" : s);
4741	libbpf_print(level, "%d%s", spec->raw_spec[i],
4742	i == spec->raw_len - 1 ? " => " : ":");
4743		5289
4744	libbpf_print(level, "%u.%u @ &x",	5290	if (core_relo_is_type_based(spec->relo_kind))
4745	spec->bit_offset / 8, spec->bit_offset % 8);	5291	return;
4746		5292
4747	for (i = 0; i < spec->len; i++) {	5293	if (core_relo_is_enumval_based(spec->relo_kind)) {
4748	if (spec->spec[i].name)	5294	t = skip_mods_and_typedefs(spec->btf, type_id, NULL);
4749	libbpf_print(level, ".%s", spec->spec[i].name);	5295	e = btf_enum(t) + spec->raw_spec[0];
4750	else	5296	s = btf__name_by_offset(spec->btf, e->name_off);
4751	libbpf_print(level, "[%u]", spec->spec[i].idx);	5297
		5298	libbpf_print(level, "::%s = %u", s, e->val);
		5299	return;
4752	}	5300	}
4753		5301
		5302	if (core_relo_is_field_based(spec->relo_kind)) {
		5303	for (i = 0; i < spec->len; i++) {
		5304	if (spec->spec[i].name)
		5305	libbpf_print(level, ".%s", spec->spec[i].name);
		5306	else if (i > 0 \|\| spec->spec[i].idx > 0)
		5307	libbpf_print(level, "[%u]", spec->spec[i].idx);
		5308	}
		5309
		5310	libbpf_print(level, " (");
		5311	for (i = 0; i < spec->raw_len; i++)
		5312	libbpf_print(level, "%s%d", i == 0 ? "" : ":", spec->raw_spec[i]);
		5313
		5314	if (spec->bit_offset % 8)
		5315	libbpf_print(level, " @ offset %u.%u)",
		5316	spec->bit_offset / 8, spec->bit_offset % 8);
		5317	else
		5318	libbpf_print(level, " @ offset %u)", spec->bit_offset / 8);
		5319	return;
		5320	}
4754	}	5321	}
4755		5322
4756	static size_t bpf_core_hash_fn(const void key, void ctx)	5323	static size_t bpf_core_hash_fn(const void key, void ctx)
@@ -4814,22 +5381,23 @@ static void *u32_as_hash_key(__u32 x) · 57 unmodified lines
4814	* CPU-wise compared to prebuilding a map from all local type names to	5381	* CPU-wise compared to prebuilding a map from all local type names to
4815	* a list of candidate type names. It's also sped up by caching resolved	5382	* a list of candidate type names. It's also sped up by caching resolved
4816	* list of matching candidates per each local "root" type ID, that has at	5383	* list of matching candidates per each local "root" type ID, that has at
4817	* least one bpf_field_reloc associated with it. This list is shared	5384	* least one bpf_core_relo associated with it. This list is shared
4818	* between multiple relocations for the same type ID and is updated as some	5385	* between multiple relocations for the same type ID and is updated as some
4819	* of the candidates are pruned due to structural incompatibility.	5386	* of the candidates are pruned due to structural incompatibility.
4820	*/	5387	*/
4821	static int bpf_core_reloc_field(struct bpf_program *prog,	5388	static int bpf_core_apply_relo(struct bpf_program *prog,
4822	const struct bpf_field_reloc *relo,	5389	const struct bpf_core_relo *relo,
4823	int relo_idx,	5390	int relo_idx,
4824	const struct btf *local_btf,	5391	const struct btf *local_btf,
4825	const struct btf *targ_btf,	5392	const struct btf *targ_btf,
4826	struct hashmap *cand_cache)	5393	struct hashmap *cand_cache)
4827	{	5394	{
4828	const char *prog_name = bpf_program__title(prog, false);	5395	const char *prog_name = bpf_program__title(prog, false);
4829	struct bpf_core_spec local_spec, cand_spec, targ_spec;	5396	struct bpf_core_spec local_spec, cand_spec, targ_spec = {};
4830	const void *type_key = u32_as_hash_key(relo->type_id);	5397	const void *type_key = u32_as_hash_key(relo->type_id);
4831	const struct btf_type local_type, cand_type;	5398	struct bpf_core_relo_res cand_res, targ_res;
4832	const char local_name, cand_name;	5399	const struct btf_type *local_type;
		5400	const char *local_name;
4833	struct ids_vec *cand_ids;	5401	struct ids_vec *cand_ids;
4834	__u32 local_id, cand_id;	5402	__u32 local_id, cand_id;
4835	const char *spec_str;	5403	const char *spec_str;
@@ -4841,32 +5409,49 @@ static int bpf_core_reloc_field(struct bpf_program *prog, · 5 unmodified lines
4841	return -EINVAL;	5409	return -EINVAL;
4842		5410
4843	local_name = btf__name_by_offset(local_btf, local_type->name_off);	5411	local_name = btf__name_by_offset(local_btf, local_type->name_off);
4844	if (str_is_empty(local_name))	5412	if (!local_name)
4845	return -EINVAL;	5413	return -EINVAL;
4846		5414
4847	spec_str = btf__name_by_offset(local_btf, relo->access_str_off);	5415	spec_str = btf__name_by_offset(local_btf, relo->access_str_off);
4848	if (str_is_empty(spec_str))	5416	if (str_is_empty(spec_str))
4849	return -EINVAL;	5417	return -EINVAL;
4850		5418
4851	err = bpf_core_spec_parse(local_btf, local_id, spec_str, &local_spec);	5419	err = bpf_core_parse_spec(local_btf, local_id, spec_str, relo->kind, &local_spec);
4852	if (err) {	5420	if (err) {
4853	pr_warn("prog '%s': relo #%d: parsing [%d] %s + %s failed: %d\n",	5421	pr_warn("prog '%s': relo #%d: parsing [%d] %s %s + %s failed: %d\n",
4854	prog_name, relo_idx, local_id, local_name, spec_str,	5422	prog_name, relo_idx, local_id, btf_kind_str(local_type),
4855	err);	5423	str_is_empty(local_name) ? "<anon>" : local_name,
		5424	spec_str, err);
4856	return -EINVAL;	5425	return -EINVAL;
4857	}	5426	}
4858		5427
4859	pr_debug("prog '%s': relo #%d: kind %d, spec is ", prog_name, relo_idx,	5428	pr_debug("prog '%s': relo #%d: kind <%s> (%d), spec is ", prog_name,
4860	relo->kind);	5429	relo_idx, core_relo_kind_str(relo->kind), relo->kind);
4861	bpf_core_dump_spec(LIBBPF_DEBUG, &local_spec);	5430	bpf_core_dump_spec(LIBBPF_DEBUG, &local_spec);
4862	libbpf_print(LIBBPF_DEBUG, "\n");	5431	libbpf_print(LIBBPF_DEBUG, "\n");
4863		5432
		5433	/* TYPE_ID_LOCAL relo is special and doesn't need candidate search */
		5434	if (relo->kind == BPF_TYPE_ID_LOCAL) {
		5435	targ_res.validate = true;
		5436	targ_res.poison = false;
		5437	targ_res.orig_val = local_spec.root_type_id;
		5438	targ_res.new_val = local_spec.root_type_id;
		5439	goto patch_insn;
		5440	}
		5441
		5442	/* libbpf doesn't support candidate search for anonymous types */
		5443	if (str_is_empty(spec_str)) {
		5444	pr_warn("prog '%s': relo #%d: <%s> (%d) relocation doesn't support anonymous types\n",
		5445	prog_name, relo_idx, core_relo_kind_str(relo->kind), relo->kind);
		5446	return -EOPNOTSUPP;
		5447	}
		5448
4864	if (!hashmap__find(cand_cache, type_key, (void **)&cand_ids)) {	5449	if (!hashmap__find(cand_cache, type_key, (void **)&cand_ids)) {
4865	cand_ids = bpf_core_find_cands(local_btf, local_id, targ_btf);	5450	cand_ids = bpf_core_find_cands(local_btf, local_id, targ_btf);
4866	if (IS_ERR(cand_ids)) {	5451	if (IS_ERR(cand_ids)) {
4867	pr_warn("prog '%s': relo #%d: target candidate search failed for [%d] %s: %ld",	5452	pr_warn("prog '%s': relo #%d: target candidate search failed for [%d] %s %s: %ld",
4868	prog_name, relo_idx, local_id, local_name,	5453	prog_name, relo_idx, local_id, btf_kind_str(local_type),
4869	PTR_ERR(cand_ids));	5454	local_name, PTR_ERR(cand_ids));
4870	return PTR_ERR(cand_ids);	5455	return PTR_ERR(cand_ids);
4871	}	5456	}
4872	err = hashmap__set(cand_cache, type_key, cand_ids, NULL, NULL);	5457	err = hashmap__set(cand_cache, type_key, cand_ids, NULL, NULL);
@@ -4878,36 +5463,51 @@ static int bpf_core_reloc_field(struct bpf_program *prog, · 5 unmodified lines
4878		5463
4879	for (i = 0, j = 0; i < cand_ids->len; i++) {	5464	for (i = 0, j = 0; i < cand_ids->len; i++) {
4880	cand_id = cand_ids->data[i];	5465	cand_id = cand_ids->data[i];
4881	cand_type = btf__type_by_id(targ_btf, cand_id);	5466	err = bpf_core_spec_match(&local_spec, targ_btf, cand_id, &cand_spec);
4882	cand_name = btf__name_by_offset(targ_btf, cand_type->name_off);
4883
4884	err = bpf_core_spec_match(&local_spec, targ_btf,
4885	cand_id, &cand_spec);
4886	pr_debug("prog '%s': relo #%d: matching candidate #%d %s against spec ",
4887	prog_name, relo_idx, i, cand_name);
4888	bpf_core_dump_spec(LIBBPF_DEBUG, &cand_spec);
4889	libbpf_print(LIBBPF_DEBUG, ": %d\n", err);
4890	if (err < 0) {	5467	if (err < 0) {
4891	pr_warn("prog '%s': relo #%d: matching error: %d\n",	5468	pr_warn("prog '%s': relo #%d: error matching candidate #%d ",
4892	prog_name, relo_idx, err);	5469	prog_name, relo_idx, i);
		5470	bpf_core_dump_spec(LIBBPF_WARN, &cand_spec);
		5471	libbpf_print(LIBBPF_WARN, ": %d\n", err);
4893	return err;	5472	return err;
4894	}	5473	}
		5474
		5475	pr_debug("prog '%s': relo #%d: %s candidate #%d ", prog_name,
		5476	relo_idx, err == 0 ? "non-matching" : "matching", i);
		5477	bpf_core_dump_spec(LIBBPF_DEBUG, &cand_spec);
		5478	libbpf_print(LIBBPF_DEBUG, "\n");
		5479
4895	if (err == 0)	5480	if (err == 0)
4896	continue;	5481	continue;
4897		5482
		5483	err = bpf_core_calc_relo(prog, relo, relo_idx, &local_spec, &cand_spec, &cand_res);
		5484	if (err)
		5485	return err;
		5486
4898	if (j == 0) {	5487	if (j == 0) {
		5488	targ_res = cand_res;
4899	targ_spec = cand_spec;	5489	targ_spec = cand_spec;
4900	} else if (cand_spec.bit_offset != targ_spec.bit_offset) {	5490	} else if (cand_spec.bit_offset != targ_spec.bit_offset) {
4901	/* if there are many candidates, they should all	5491	/* if there are many field relo candidates, they
4902	* resolve to the same bit offset	5492	* should all resolve to the same bit offset
4903	*/	5493	*/
4904	pr_warn("prog '%s': relo #%d: offset ambiguity: %u != %u\n",	5494	pr_warn("prog '%s': relo #%d: field offset ambiguity: %u != %u\n",
4905	prog_name, relo_idx, cand_spec.bit_offset,	5495	prog_name, relo_idx, cand_spec.bit_offset,
4906	targ_spec.bit_offset);	5496	targ_spec.bit_offset);
4907	return -EINVAL;	5497	return -EINVAL;
		5498	} else if (cand_res.poison != targ_res.poison \|\| cand_res.new_val != targ_res.new_val) {
		5499	/* all candidates should result in the same relocation
		5500	* decision and value, otherwise it's dangerous to
		5501	* proceed due to ambiguity
		5502	*/
		5503	pr_warn("prog '%s': relo #%d: relocation decision ambiguity: %s %u != %s %u\n",
		5504	prog_name, relo_idx,
		5505	cand_res.poison ? "failure" : "success", cand_res.new_val,
		5506	targ_res.poison ? "failure" : "success", targ_res.new_val);
		5507	return -EINVAL;
4908	}	5508	}
4909		5509
4910	cand_ids->data[j++] = cand_spec.spec[0].type_id;	5510	cand_ids->data[j++] = cand_spec.root_type_id;
4911	}	5511	}
4912		5512
4913	/*	5513	/*
@@ -4926,19 +5526,25 @@ static int bpf_core_reloc_field(struct bpf_program *prog, · 12 unmodified lines
4926	* as well as expected case, depending whether instruction w/	5526	* as well as expected case, depending whether instruction w/
4927	* relocation is guarded in some way that makes it unreachable (dead	5527	* relocation is guarded in some way that makes it unreachable (dead
4928	* code) if relocation can't be resolved. This is handled in	5528	* code) if relocation can't be resolved. This is handled in
4929	* bpf_core_reloc_insn() uniformly by replacing that instruction with	5529	* bpf_core_patch_insn() uniformly by replacing that instruction with
4930	* BPF helper call insn (using invalid helper ID). If that instruction	5530	* BPF helper call insn (using invalid helper ID). If that instruction
4931	* is indeed unreachable, then it will be ignored and eliminated by	5531	* is indeed unreachable, then it will be ignored and eliminated by
4932	* verifier. If it was an error, then verifier will complain and point	5532	* verifier. If it was an error, then verifier will complain and point
4933	* to a specific instruction number in its log.	5533	* to a specific instruction number in its log.
4934	*/	5534	*/
4935	if (j == 0)	5535	if (j == 0) {
4936	pr_debug("prog '%s': relo #%d: no matching targets found for [%d] %s + %s\n",	5536	pr_debug("prog '%s': relo #%d: no matching targets found\n",
4937	prog_name, relo_idx, local_id, local_name, spec_str);	5537	prog_name, relo_idx);
		5538
		5539	/* calculate single target relo result explicitly */
		5540	err = bpf_core_calc_relo(prog, relo, relo_idx, &local_spec, NULL, &targ_res);
		5541	if (err)
		5542	return err;
		5543	}
4938		5544
4939	/* bpf_core_reloc_insn should know how to handle missing targ_spec */	5545	patch_insn:
4940	err = bpf_core_reloc_insn(prog, relo, relo_idx, &local_spec,	5546	/* bpf_core_patch_insn() should know how to handle missing targ_spec */
4941	j ? &targ_spec : NULL);	5547	err = bpf_core_patch_insn(prog, relo, relo_idx, &targ_res);
4942	if (err) {	5548	if (err) {
4943	pr_warn("prog '%s': relo #%d: failed to patch insn at offset %d: %d\n",	5549	pr_warn("prog '%s': relo #%d: failed to patch insn at offset %d: %d\n",
4944	prog_name, relo_idx, relo->insn_off, err);	5550	prog_name, relo_idx, relo->insn_off, err);
@@ -4949,10 +5555,10 @@ static int bpf_core_reloc_field(struct bpf_program *prog, · 4 unmodified lines
4949	}	5555	}
4950		5556
4951	static int	5557	static int
4952	bpf_core_reloc_fields(struct bpf_object obj, const char targ_btf_path)	5558	bpf_object__relocate_core(struct bpf_object obj, const char targ_btf_path)
4953	{	5559	{
4954	const struct btf_ext_info_sec *sec;	5560	const struct btf_ext_info_sec *sec;
4955	const struct bpf_field_reloc *rec;	5561	const struct bpf_core_relo *rec;
4956	const struct btf_ext_info *seg;	5562	const struct btf_ext_info *seg;
4957	struct hashmap_entry *entry;	5563	struct hashmap_entry *entry;
4958	struct hashmap *cand_cache = NULL;	5564	struct hashmap *cand_cache = NULL;
@@ -4961,6 +5567,9 @@ bpf_core_reloc_fields(struct bpf_object obj, const char targ_btf_path) · 2 unmodified lines
4961	const char *sec_name;	5567	const char *sec_name;
4962	int i, err = 0;	5568	int i, err = 0;
4963		5569
		5570	if (obj->btf_ext->core_relo_info.len == 0)
		5571	return 0;
		5572
4964	if (targ_btf_path)	5573	if (targ_btf_path)
4965	targ_btf = btf__parse_elf(targ_btf_path, NULL);	5574	targ_btf = btf__parse_elf(targ_btf_path, NULL);
4966	else	5575	else
@@ -4976,7 +5585,7 @@ bpf_core_reloc_fields(struct bpf_object obj, const char targ_btf_path) · 9 unmodified lines
4976	goto out;	5585	goto out;
4977	}	5586	}
4978		5587
4979	seg = &obj->btf_ext->field_reloc_info;	5588	seg = &obj->btf_ext->core_relo_info;
4980	for_each_btf_ext_sec(seg, sec) {	5589	for_each_btf_ext_sec(seg, sec) {
4981	sec_name = btf__name_by_offset(obj->btf, sec->sec_name_off);	5590	sec_name = btf__name_by_offset(obj->btf, sec->sec_name_off);
4982	if (str_is_empty(sec_name)) {	5591	if (str_is_empty(sec_name)) {
@@ -4997,15 +5606,15 @@ bpf_core_reloc_fields(struct bpf_object obj, const char targ_btf_path) · 14 unmodified lines
4997	goto out;	5606	goto out;
4998	}	5607	}
4999		5608
5000	pr_debug("prog '%s': performing %d CO-RE offset relocs\n",	5609	pr_debug("sec '%s': found %d CO-RE relocations\n",
5001	sec_name, sec->num_info);	5610	sec_name, sec->num_info);
5002		5611
5003	for_each_btf_ext_rec(seg, sec, i, rec) {	5612	for_each_btf_ext_rec(seg, sec, i, rec) {
5004	err = bpf_core_reloc_field(prog, rec, i, obj->btf,	5613	err = bpf_core_apply_relo(prog, rec, i, obj->btf,
5005	targ_btf, cand_cache);	5614	targ_btf, cand_cache);
5006	if (err) {	5615	if (err) {
5007	pr_warn("prog '%s': relo #%d: failed to relocate: %d\n",	5616	pr_warn("prog '%s': relo #%d: failed to relocate: %d\n",
5008	sec_name, i, err);	5617	prog->name, i, err);
5009	goto out;	5618	goto out;
5010	}	5619	}
5011	}	5620	}
@@ -5024,17 +5633,6 @@ out: · 12 unmodified lines
5024	return err;	5633	return err;
5025	}	5634	}
5026		5635
5027	static int
5028	bpf_object__relocate_core(struct bpf_object obj, const char targ_btf_path)
5029	{
5030	int err = 0;
5031
5032	if (obj->btf_ext->field_reloc_info.len)
5033	err = bpf_core_reloc_fields(obj, targ_btf_path);
5034
5035	return err;
5036	}
5037
5038	static int	5636	static int
5039	bpf_program__reloc_text(struct bpf_program prog, struct bpf_object obj,	5637	bpf_program__reloc_text(struct bpf_program prog, struct bpf_object obj,
5040	struct reloc_desc *relo)	5638	struct reloc_desc *relo)
@@ -5051,7 +5649,7 @@ bpf_program__reloc_text(struct bpf_program prog, struct bpf_object obj, · 10 unmodified lines
5051	return -LIBBPF_ERRNO__RELOC;	5649	return -LIBBPF_ERRNO__RELOC;
5052	}	5650	}
5053	new_cnt = prog->insns_cnt + text->insns_cnt;	5651	new_cnt = prog->insns_cnt + text->insns_cnt;
5054	new_insn = reallocarray(prog->insns, new_cnt, sizeof(*insn));	5652	new_insn = libbpf_reallocarray(prog->insns, new_cnt, sizeof(*insn));
5055	if (!new_insn) {	5653	if (!new_insn) {
5056	pr_warn("oom in prog realloc\n");	5654	pr_warn("oom in prog realloc\n");
5057	return -ENOMEM;	5655	return -ENOMEM;
@@ -5136,7 +5734,8 @@ bpf_program__relocate(struct bpf_program prog, struct bpf_object obj) · 78 unmodified lines
5136	return err;	5734	return err;
5137	break;	5735	break;
5138	default:	5736	default:
5139	pr_warn("relo #%d: bad relo type %d\n", i, relo->type);	5737	pr_warn("prog '%s': relo #%d: bad relo type %d\n",
		5738	prog->name, i, relo->type);
5140	return -EINVAL;	5739	return -EINVAL;
5141	}	5740	}
5142	}	5741	}
@@ -5171,7 +5770,8 @@ bpf_object__relocate(struct bpf_object obj, const char targ_btf_path) · 28 unmodified lines
5171		5770
5172	err = bpf_program__relocate(prog, obj);	5771	err = bpf_program__relocate(prog, obj);
5173	if (err) {	5772	if (err) {
5174	pr_warn("failed to relocate '%s'\n", prog->section_name);	5773	pr_warn("prog '%s': failed to relocate data references: %d\n",
		5774	prog->name, err);
5175	return err;	5775	return err;
5176	}	5776	}
5177	break;	5777	break;
@@ -5186,7 +5786,8 @@ bpf_object__relocate(struct bpf_object obj, const char targ_btf_path) · 8 unmodified lines
5186		5786
5187	err = bpf_program__relocate(prog, obj);	5787	err = bpf_program__relocate(prog, obj);
5188	if (err) {	5788	if (err) {
5189	pr_warn("failed to relocate '%s'\n", prog->section_name);	5789	pr_warn("prog '%s': failed to relocate calls: %d\n",
		5790	prog->name, err);
5190	return err;	5791	return err;
5191	}	5792	}
5192	}	5793	}
@@ -5230,8 +5831,7 @@ static int bpf_object__collect_map_relos(struct bpf_object *obj, · 37 unmodified lines
5230	i, (size_t)GELF_R_SYM(rel.r_info));	5831	i, (size_t)GELF_R_SYM(rel.r_info));
5231	return -LIBBPF_ERRNO__FORMAT;	5832	return -LIBBPF_ERRNO__FORMAT;
5232	}	5833	}
5233	name = elf_strptr(obj->efile.elf, obj->efile.strtabidx,	5834	name = elf_sym_str(obj, sym.st_name) ?: "<?>";
5234	sym.st_name) ? : "<?>";
5235	if (sym.st_shndx != obj->efile.btf_maps_shndx) {	5835	if (sym.st_shndx != obj->efile.btf_maps_shndx) {
5236	pr_warn(".maps relo #%d: '%s' isn't a BTF-defined map\n",	5836	pr_warn(".maps relo #%d: '%s' isn't a BTF-defined map\n",
5237	i, name);	5837	i, name);
@@ -5293,7 +5893,7 @@ static int bpf_object__collect_map_relos(struct bpf_object *obj, · 55 unmodified lines
5293	moff /= bpf_ptr_sz;	5893	moff /= bpf_ptr_sz;
5294	if (moff >= map->init_slots_sz) {	5894	if (moff >= map->init_slots_sz) {
5295	new_sz = moff + 1;	5895	new_sz = moff + 1;
5296	tmp = realloc(map->init_slots, new_sz * host_ptr_sz);	5896	tmp = libbpf_reallocarray(map->init_slots, new_sz, host_ptr_sz);
5297	if (!tmp)	5897	if (!tmp)
5298	return -ENOMEM;	5898	return -ENOMEM;
5299	map->init_slots = tmp;	5899	map->init_slots = tmp;
@@ -5348,6 +5948,51 @@ static int bpf_object__collect_reloc(struct bpf_object *obj) · 48 unmodified lines
5348	return 0;	5948	return 0;
5349	}	5949	}
5350		5950
		5951	static bool insn_is_helper_call(struct bpf_insn insn, enum bpf_func_id func_id)
		5952	{
		5953	if (BPF_CLASS(insn->code) == BPF_JMP &&
		5954	BPF_OP(insn->code) == BPF_CALL &&
		5955	BPF_SRC(insn->code) == BPF_K &&
		5956	insn->src_reg == 0 &&
		5957	insn->dst_reg == 0) {
		5958	*func_id = insn->imm;
		5959	return true;
		5960	}
		5961	return false;
		5962	}
		5963
		5964	static int bpf_object__sanitize_prog(struct bpf_object* obj, struct bpf_program *prog)
		5965	{
		5966	struct bpf_insn *insn = prog->insns;
		5967	enum bpf_func_id func_id;
		5968	int i;
		5969
		5970	for (i = 0; i < prog->insns_cnt; i++, insn++) {
		5971	if (!insn_is_helper_call(insn, &func_id))
		5972	continue;
		5973
		5974	/* on kernels that don't yet support
		5975	* bpf_probe_read_{kernel,user}[_str] helpers, fall back
		5976	* to bpf_probe_read() which works well for old kernels
		5977	*/
		5978	switch (func_id) {
		5979	case BPF_FUNC_probe_read_kernel:
		5980	case BPF_FUNC_probe_read_user:
		5981	if (!kernel_supports(FEAT_PROBE_READ_KERN))
		5982	insn->imm = BPF_FUNC_probe_read;
		5983	break;
		5984	case BPF_FUNC_probe_read_kernel_str:
		5985	case BPF_FUNC_probe_read_user_str:
		5986	if (!kernel_supports(FEAT_PROBE_READ_KERN))
		5987	insn->imm = BPF_FUNC_probe_read_str;
		5988	break;
		5989	default:
		5990	break;
		5991	}
		5992	}
		5993	return 0;
		5994	}
		5995
5351	static int	5996	static int
5352	load_program(struct bpf_program prog, struct bpf_insn insns, int insns_cnt,	5997	load_program(struct bpf_program prog, struct bpf_insn insns, int insns_cnt,
5353	char license, __u32 kern_version, int pfd)	5998	char license, __u32 kern_version, int pfd)
@@ -5364,12 +6009,12 @@ load_program(struct bpf_program prog, struct bpf_insn insns, int insns_cnt, · 10 unmodified lines
5364	memset(&load_attr, 0, sizeof(struct bpf_load_program_attr));	6009	memset(&load_attr, 0, sizeof(struct bpf_load_program_attr));
5365	load_attr.prog_type = prog->type;	6010	load_attr.prog_type = prog->type;
5366	/* old kernels might not support specifying expected_attach_type */	6011	/* old kernels might not support specifying expected_attach_type */
5367	if (!prog->caps->exp_attach_type && prog->sec_def &&	6012	if (!kernel_supports(FEAT_EXP_ATTACH_TYPE) && prog->sec_def &&
5368	prog->sec_def->is_exp_attach_type_optional)	6013	prog->sec_def->is_exp_attach_type_optional)
5369	load_attr.expected_attach_type = 0;	6014	load_attr.expected_attach_type = 0;
5370	else	6015	else
5371	load_attr.expected_attach_type = prog->expected_attach_type;	6016	load_attr.expected_attach_type = prog->expected_attach_type;
5372	if (prog->caps->name)	6017	if (kernel_supports(FEAT_PROG_NAME))
5373	load_attr.name = prog->name;	6018	load_attr.name = prog->name;
5374	load_attr.insns = insns;	6019	load_attr.insns = insns;
5375	load_attr.insns_cnt = insns_cnt;	6020	load_attr.insns_cnt = insns_cnt;
@@ -5387,7 +6032,7 @@ load_program(struct bpf_program prog, struct bpf_insn insns, int insns_cnt, · 11 unmodified lines
5387	}	6032	}
5388	/* specify func_info/line_info only if kernel supports them */	6033	/* specify func_info/line_info only if kernel supports them */
5389	btf_fd = bpf_object__btf_fd(prog->obj);	6034	btf_fd = bpf_object__btf_fd(prog->obj);
5390	if (btf_fd >= 0 && prog->obj->caps.btf_func) {	6035	if (btf_fd >= 0 && kernel_supports(FEAT_BTF_FUNC)) {
5391	load_attr.prog_btf_fd = btf_fd;	6036	load_attr.prog_btf_fd = btf_fd;
5392	load_attr.func_info = prog->func_info;	6037	load_attr.func_info = prog->func_info;
5393	load_attr.func_info_rec_size = prog->func_info_rec_size;	6038	load_attr.func_info_rec_size = prog->func_info_rec_size;
@@ -5425,7 +6070,7 @@ retry_load: · 31 unmodified lines
5425	free(log_buf);	6070	free(log_buf);
5426	goto retry_load;	6071	goto retry_load;
5427	}	6072	}
5428	ret = -errno;	6073	ret = errno ? -errno : -LIBBPF_ERRNO__LOAD;
5429	cp = libbpf_strerror_r(errno, errmsg, sizeof(errmsg));	6074	cp = libbpf_strerror_r(errno, errmsg, sizeof(errmsg));
5430	pr_warn("load bpf program failed: %s\n", cp);	6075	pr_warn("load bpf program failed: %s\n", cp);
5431	pr_perm_msg(ret);	6076	pr_perm_msg(ret);
@@ -5562,13 +6207,19 @@ bpf_object__load_progs(struct bpf_object *obj, int log_level) · 130 unmodified lines
5562	size_t i;	6207	size_t i;
5563	int err;	6208	int err;
5564		6209
		6210	for (i = 0; i < obj->nr_programs; i++) {
		6211	prog = &obj->programs[i];
		6212	err = bpf_object__sanitize_prog(obj, prog);
		6213	if (err)
		6214	return err;
		6215	}
		6216
5565	for (i = 0; i < obj->nr_programs; i++) {	6217	for (i = 0; i < obj->nr_programs; i++) {
5566	prog = &obj->programs[i];	6218	prog = &obj->programs[i];
5567	if (bpf_program__is_function_storage(prog, obj))	6219	if (bpf_program__is_function_storage(prog, obj))
5568	continue;	6220	continue;
5569	if (!prog->load) {	6221	if (!prog->load) {
5570	pr_debug("prog '%s'('%s'): skipped loading\n",	6222	pr_debug("prog '%s': skipped loading\n", prog->name);
5571	prog->name, prog->section_name);
5572	continue;	6223	continue;
5573	}	6224	}
5574	prog->log_level \|= log_level;	6225	prog->log_level \|= log_level;
@@ -5641,6 +6292,8 @@ __bpf_object__open(const char path, const void obj_buf, size_t obj_buf_sz, · 66 unmodified lines
5641	/* couldn't guess, but user might manually specify */	6292	/* couldn't guess, but user might manually specify */
5642	continue;	6293	continue;
5643		6294
		6295	if (prog->sec_def->is_sleepable)
		6296	prog->prog_flags \|= BPF_F_SLEEPABLE;
5644	bpf_program__set_type(prog, prog->sec_def->prog_type);	6297	bpf_program__set_type(prog, prog->sec_def->prog_type);
5645	bpf_program__set_expected_attach_type(prog,	6298	bpf_program__set_expected_attach_type(prog,
5646	prog->sec_def->expected_attach_type);	6299	prog->sec_def->expected_attach_type);
@@ -5750,11 +6403,11 @@ static int bpf_object__sanitize_maps(struct bpf_object *obj) · 103 unmodified lines
5750	bpf_object__for_each_map(m, obj) {	6403	bpf_object__for_each_map(m, obj) {
5751	if (!bpf_map__is_internal(m))	6404	if (!bpf_map__is_internal(m))
5752	continue;	6405	continue;
5753	if (!obj->caps.global_data) {	6406	if (!kernel_supports(FEAT_GLOBAL_DATA)) {
5754	pr_warn("kernel doesn't support global data\n");	6407	pr_warn("kernel doesn't support global data\n");
5755	return -ENOTSUP;	6408	return -ENOTSUP;
5756	}	6409	}
5757	if (!obj->caps.array_mmap)	6410	if (!kernel_supports(FEAT_ARRAY_MMAP))
5758	m->def.map_flags ^= BPF_F_MMAPABLE;	6411	m->def.map_flags ^= BPF_F_MMAPABLE;
5759	}	6412	}
5760		6413
@@ -5904,7 +6557,6 @@ int bpf_object__load_xattr(struct bpf_object_load_attr *attr) · 143 unmodified lines
5904	}	6557	}
5905		6558
5906	err = bpf_object__probe_loading(obj);	6559	err = bpf_object__probe_loading(obj);
5907	err = err ? : bpf_object__probe_caps(obj);
5908	err = err ? : bpf_object__resolve_externs(obj, obj->kconfig);	6560	err = err ? : bpf_object__resolve_externs(obj, obj->kconfig);
5909	err = err ? : bpf_object__sanitize_and_load_btf(obj);	6561	err = err ? : bpf_object__sanitize_and_load_btf(obj);
5910	err = err ? : bpf_object__sanitize_maps(obj);	6562	err = err ? : bpf_object__sanitize_maps(obj);
@@ -6713,7 +7365,7 @@ int bpf_program__fd(const struct bpf_program *prog) · 802 unmodified lines
6713		7365
6714	size_t bpf_program__size(const struct bpf_program *prog)	7366	size_t bpf_program__size(const struct bpf_program *prog)
6715	{	7367	{
6716	return prog->insns_cnt * sizeof(struct bpf_insn);	7368	return prog->insns_cnt * BPF_INSN_SZ;
6717	}	7369	}
6718		7370
6719	int bpf_program__set_prep(struct bpf_program *prog, int nr_instances,	7371	int bpf_program__set_prep(struct bpf_program *prog, int nr_instances,
@@ -6910,6 +7562,21 @@ static const struct bpf_sec_def section_defs[] = { · 190 unmodified lines
6910	.expected_attach_type = BPF_TRACE_FEXIT,	7562	.expected_attach_type = BPF_TRACE_FEXIT,
6911	.is_attach_btf = true,	7563	.is_attach_btf = true,
6912	.attach_fn = attach_trace),	7564	.attach_fn = attach_trace),
		7565	SEC_DEF("fentry.s/", TRACING,
		7566	.expected_attach_type = BPF_TRACE_FENTRY,
		7567	.is_attach_btf = true,
		7568	.is_sleepable = true,
		7569	.attach_fn = attach_trace),
		7570	SEC_DEF("fmod_ret.s/", TRACING,
		7571	.expected_attach_type = BPF_MODIFY_RETURN,
		7572	.is_attach_btf = true,
		7573	.is_sleepable = true,
		7574	.attach_fn = attach_trace),
		7575	SEC_DEF("fexit.s/", TRACING,
		7576	.expected_attach_type = BPF_TRACE_FEXIT,
		7577	.is_attach_btf = true,
		7578	.is_sleepable = true,
		7579	.attach_fn = attach_trace),
6913	SEC_DEF("freplace/", EXT,	7580	SEC_DEF("freplace/", EXT,
6914	.is_attach_btf = true,	7581	.is_attach_btf = true,
6915	.attach_fn = attach_trace),	7582	.attach_fn = attach_trace),
@@ -6917,6 +7584,11 @@ static const struct bpf_sec_def section_defs[] = { · 1 unmodified line
6917	.is_attach_btf = true,	7584	.is_attach_btf = true,
6918	.expected_attach_type = BPF_LSM_MAC,	7585	.expected_attach_type = BPF_LSM_MAC,
6919	.attach_fn = attach_lsm),	7586	.attach_fn = attach_lsm),
		7587	SEC_DEF("lsm.s/", LSM,
		7588	.is_attach_btf = true,
		7589	.is_sleepable = true,
		7590	.expected_attach_type = BPF_LSM_MAC,
		7591	.attach_fn = attach_lsm),
6920	SEC_DEF("iter/", TRACING,	7592	SEC_DEF("iter/", TRACING,
6921	.expected_attach_type = BPF_TRACE_ITER,	7593	.expected_attach_type = BPF_TRACE_ITER,
6922	.is_attach_btf = true,	7594	.is_attach_btf = true,
@@ -7122,8 +7794,7 @@ static int bpf_object__collect_st_ops_relos(struct bpf_object *obj, · 199 unmodified lines
7122	return -LIBBPF_ERRNO__FORMAT;	7794	return -LIBBPF_ERRNO__FORMAT;
7123	}	7795	}
7124		7796
7125	name = elf_strptr(obj->efile.elf, obj->efile.strtabidx,	7797	name = elf_sym_str(obj, sym.st_name) ?: "<?>";
7126	sym.st_name) ? : "<?>";
7127	map = find_struct_ops_map_by_offset(obj, rel.r_offset);	7798	map = find_struct_ops_map_by_offset(obj, rel.r_offset);
7128	if (!map) {	7799	if (!map) {
7129	pr_warn("struct_ops reloc: cannot find map at rel.r_offset %zu\n",	7800	pr_warn("struct_ops reloc: cannot find map at rel.r_offset %zu\n",
@@ -7640,7 +8311,7 @@ int bpf_prog_load_xattr(const struct bpf_prog_load_attr *attr, · 510 unmodified lines
7640		8311
7641	prog->prog_ifindex = attr->ifindex;	8312	prog->prog_ifindex = attr->ifindex;
7642	prog->log_level = attr->log_level;	8313	prog->log_level = attr->log_level;
7643	prog->prog_flags = attr->prog_flags;	8314	prog->prog_flags \|= attr->prog_flags;
7644	if (!first_prog)	8315	if (!first_prog)
7645	first_prog = prog;	8316	first_prog = prog;
7646	}	8317	}
@@ -8594,7 +9265,7 @@ struct perf_buffer *perf_buffer__new(int map_fd, size_t page_cnt, · 947 unmodified lines
8594	struct perf_buffer_params p = {};	9265	struct perf_buffer_params p = {};
8595	struct perf_event_attr attr = { 0, };	9266	struct perf_event_attr attr = { 0, };
8596		9267
8597	attr.config = PERF_COUNT_SW_BPF_OUTPUT,	9268	attr.config = PERF_COUNT_SW_BPF_OUTPUT;
8598	attr.type = PERF_TYPE_SOFTWARE;	9269	attr.type = PERF_TYPE_SOFTWARE;
8599	attr.sample_type = PERF_SAMPLE_RAW;	9270	attr.sample_type = PERF_SAMPLE_RAW;
8600	attr.sample_period = 1;	9271	attr.sample_period = 1;
@@ -8832,6 +9503,11 @@ static int perf_buffer__process_records(struct perf_buffer *pb, · 231 unmodified lines
8832	return 0;	9503	return 0;
8833	}	9504	}
8834		9505
		9506	int perf_buffer__epoll_fd(const struct perf_buffer *pb)
		9507	{
		9508	return pb->epoll_fd;
		9509	}
		9510
8835	int perf_buffer__poll(struct perf_buffer *pb, int timeout_ms)	9511	int perf_buffer__poll(struct perf_buffer *pb, int timeout_ms)
8836	{	9512	{
8837	int i, cnt, err;	9513	int i, cnt, err;
@@ -8849,6 +9525,55 @@ int perf_buffer__poll(struct perf_buffer *pb, int timeout_ms) · 11 unmodified lines
8849	return cnt < 0 ? -errno : cnt;	9525	return cnt < 0 ? -errno : cnt;
8850	}	9526	}
8851		9527
		9528	/* Return number of PERF_EVENT_ARRAY map slots set up by this perf_buffer
		9529	* manager.
		9530	*/
		9531	size_t perf_buffer__buffer_cnt(const struct perf_buffer *pb)
		9532	{
		9533	return pb->cpu_cnt;
		9534	}
		9535
		9536	/*
		9537	* Return perf_event FD of a ring buffer in buf_idx slot of
		9538	* PERF_EVENT_ARRAY BPF map. This FD can be polled for new data using
		9539	* select()/poll()/epoll() Linux syscalls.
		9540	*/
		9541	int perf_buffer__buffer_fd(const struct perf_buffer *pb, size_t buf_idx)
		9542	{
		9543	struct perf_cpu_buf *cpu_buf;
		9544
		9545	if (buf_idx >= pb->cpu_cnt)
		9546	return -EINVAL;
		9547
		9548	cpu_buf = pb->cpu_bufs[buf_idx];
		9549	if (!cpu_buf)
		9550	return -ENOENT;
		9551
		9552	return cpu_buf->fd;
		9553	}
		9554
		9555	/*
		9556	* Consume data from perf ring buffer corresponding to slot buf_idx in
		9557	* PERF_EVENT_ARRAY BPF map without waiting/polling. If there is no data to
		9558	* consume, do nothing and return success.
		9559	* Returns:
		9560	* - 0 on success;
		9561	* - <0 on failure.
		9562	*/
		9563	int perf_buffer__consume_buffer(struct perf_buffer *pb, size_t buf_idx)
		9564	{
		9565	struct perf_cpu_buf *cpu_buf;
		9566
		9567	if (buf_idx >= pb->cpu_cnt)
		9568	return -EINVAL;
		9569
		9570	cpu_buf = pb->cpu_bufs[buf_idx];
		9571	if (!cpu_buf)
		9572	return -ENOENT;
		9573
		9574	return perf_buffer__process_records(pb, cpu_buf);
		9575	}
		9576
8852	int perf_buffer__consume(struct perf_buffer *pb)	9577	int perf_buffer__consume(struct perf_buffer *pb)
8853	{	9578	{
8854	int i, err;	9579	int i, err;
@@ -8861,7 +9586,7 @@ int perf_buffer__consume(struct perf_buffer *pb) · 6 unmodified lines
8861		9586
8862	err = perf_buffer__process_records(pb, cpu_buf);	9587	err = perf_buffer__process_records(pb, cpu_buf);
8863	if (err) {	9588	if (err) {
8864	pr_warn("error while processing records: %d\n", err);	9589	pr_warn("perf_buffer: failed to process records in buffer #%d: %d\n", i, err);
8865	return err;	9590	return err;
8866	}	9591	}
8867	}	9592	}

tools/lib/bpf/libbpf.h+4−0

 		     const struct perf_buffer_raw_opts *opts);
 LIBBPF_API void perf_buffer__free(struct perf_buffer *pb);
+LIBBPF_API int perf_buffer__epoll_fd(const struct perf_buffer *pb);
 LIBBPF_API int perf_buffer__poll(struct perf_buffer *pb, int timeout_ms);
 LIBBPF_API int perf_buffer__consume(struct perf_buffer *pb);
+LIBBPF_API int perf_buffer__consume_buffer(struct perf_buffer *pb, size_t buf_idx);
+LIBBPF_API size_t perf_buffer__buffer_cnt(const struct perf_buffer *pb);
+LIBBPF_API int perf_buffer__buffer_fd(const struct perf_buffer *pb, size_t buf_idx);
 typedef enum bpf_perf_event_ret
 	(*bpf_perf_event_print_t)(struct perf_event_header *hdr,

tools/lib/bpf/libbpf.map+9−0

@@ -299,3 +299,12 @@ LIBBPF_0.1.0 { · 298 unmodified lines
299	btf__set_fd;	299	btf__set_fd;
300	btf__set_pointer_size;	300	btf__set_pointer_size;
301	} LIBBPF_0.0.9;	301	} LIBBPF_0.0.9;
		302
		303	LIBBPF_0.2.0 {
		304	global:
		305	perf_buffer__buffer_cnt;
		306	perf_buffer__buffer_fd;
		307	perf_buffer__epoll_fd;
		308	perf_buffer__consume_buffer;
		309	xsk_socket__create_shared;
		310	} LIBBPF_0.1.0;

tools/lib/bpf/libbpf_internal.h+107−31

 #ifndef __LIBBPF_LIBBPF_INTERNAL_H
 #define __LIBBPF_LIBBPF_INTERNAL_H
+#include <stdlib.h>
+#include <limits.h>
+/* make sure libbpf doesn't use kernel-only integer typedefs */
+#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
+/* prevent accidental re-addition of reallocarray() */
+#pragma GCC poison reallocarray
 #include "libbpf.h"
 #define BTF_INFO_ENC(kind, kind_flag, vlen) \
 #define BTF_PARAM_ENC(name, type) (name), (type)
 #define BTF_VAR_SECINFO_ENC(type, offset, size) (type), (offset), (size)
+#ifndef likely
+#define likely(x) __builtin_expect(!!(x), 1)
+#endif
+#ifndef unlikely
+#define unlikely(x) __builtin_expect(!!(x), 0)
+#endif
 #ifndef min
 # define min(x, y) ((x) < (y) ? (x) : (y))
 #endif
 #define pr_info(fmt, ...)	__pr(LIBBPF_INFO, fmt, ##__VA_ARGS__)
 #define pr_debug(fmt, ...)	__pr(LIBBPF_DEBUG, fmt, ##__VA_ARGS__)
+#ifndef __has_builtin
+#define __has_builtin(x) 0
+#endif
+/*
+ * Re-implement glibc's reallocarray() for libbpf internal-only use.
+ * reallocarray(), unfortunately, is not available in all versions of glibc,
+ * so requires extra feature detection and using reallocarray() stub from
+ * <tools/libc_compat.h> and COMPAT_NEED_REALLOCARRAY. All this complicates
+ * build of libbpf unnecessarily and is just a maintenance burden. Instead,
+ * it's trivial to implement libbpf-specific internal version and use it
+ * throughout libbpf.
+ */
+static inline void *libbpf_reallocarray(void *ptr, size_t nmemb, size_t size)
+{
+	size_t total;
+#if __has_builtin(__builtin_mul_overflow)
+	if (unlikely(__builtin_mul_overflow(nmemb, size, &total)))
+		return NULL;
+#else
+	if (size == 0 || nmemb > ULONG_MAX / size)
+		return NULL;
+	total = nmemb * size;
+#endif
+	return realloc(ptr, total);
+}
 static inline bool libbpf_validate_opts(const char *opts,
 					size_t opts_sz, size_t user_sz,
 					const char *type_name)
 int bpf_object__variable_offset(const struct bpf_object *obj, const char *name,
 				__u32 *off);
-struct nlattr;
-typedef int (*libbpf_dump_nlmsg_t)(void *cookie, void *msg, struct nlattr **tb);
-int libbpf_netlink_open(unsigned int *nl_pid);
-int libbpf_nl_get_link(int sock, unsigned int nl_pid,
-		       libbpf_dump_nlmsg_t dump_link_nlmsg, void *cookie);
-int libbpf_nl_get_class(int sock, unsigned int nl_pid, int ifindex,
-			libbpf_dump_nlmsg_t dump_class_nlmsg, void *cookie);
-int libbpf_nl_get_qdisc(int sock, unsigned int nl_pid, int ifindex,
-			libbpf_dump_nlmsg_t dump_qdisc_nlmsg, void *cookie);
-int libbpf_nl_get_filter(int sock, unsigned int nl_pid, int ifindex, int handle,
-			 libbpf_dump_nlmsg_t dump_filter_nlmsg, void *cookie);
 struct btf_ext_info {
 	/*
 	 * info points to the individual info section (e.g. func_info and
 	     i < (sec)->num_info;					\
 	     i++, rec = (void *)rec + (seg)->rec_size)
+/*
+ * The .BTF.ext ELF section layout defined as
+ *   struct btf_ext_header
+ *   func_info subsection
+ *
+ * The func_info subsection layout:
+ *   record size for struct bpf_func_info in the func_info subsection
+ *   struct btf_sec_func_info for section #1
+ *   a list of bpf_func_info records for section #1
+ *     where struct bpf_func_info mimics one in include/uapi/linux/bpf.h
+ *     but may not be identical
+ *   struct btf_sec_func_info for section #2
+ *   a list of bpf_func_info records for section #2
+ *   ......
+ *
+ * Note that the bpf_func_info record size in .BTF.ext may not
+ * be the same as the one defined in include/uapi/linux/bpf.h.
+ * The loader should ensure that record_size meets minimum
+ * requirement and pass the record as is to the kernel. The
+ * kernel will handle the func_info properly based on its contents.
+ */
+struct btf_ext_header {
+	__u16	magic;
+	__u8	version;
+	__u8	flags;
+	__u32	hdr_len;
+	/* All offsets are in bytes relative to the end of this header */
+	__u32	func_info_off;
+	__u32	func_info_len;
+	__u32	line_info_off;
+	__u32	line_info_len;
+	/* optional part of .BTF.ext header */
+	__u32	core_relo_off;
+	__u32	core_relo_len;
+};
 struct btf_ext {
 	union {
 		struct btf_ext_header *hdr;
 	};
 	struct btf_ext_info func_info;
 	struct btf_ext_info line_info;
-	struct btf_ext_info field_reloc_info;
+	struct btf_ext_info core_relo_info;
 	__u32 data_size;
 };
 	__u32	line_col;
 };
-/* bpf_field_info_kind encodes which aspect of captured field has to be
+/* bpf_core_relo_kind encodes which aspect of captured field/type/enum value
- * adjusted by relocations. Currently supported values are:
+ * has to be adjusted by relocations.
- *   - BPF_FIELD_BYTE_OFFSET: field offset (in bytes);
- *   - BPF_FIELD_EXISTS: field existence (1, if field exists; 0, otherwise);
  */
-enum bpf_field_info_kind {
+enum bpf_core_relo_kind {
 	BPF_FIELD_BYTE_OFFSET = 0,	/* field byte offset */
-	BPF_FIELD_BYTE_SIZE = 1,
+	BPF_FIELD_BYTE_SIZE = 1,	/* field size in bytes */
 	BPF_FIELD_EXISTS = 2,		/* field existence in target kernel */
-	BPF_FIELD_SIGNED = 3,
+	BPF_FIELD_SIGNED = 3,		/* field signedness (0 - unsigned, 1 - signed) */
-	BPF_FIELD_LSHIFT_U64 = 4,
+	BPF_FIELD_LSHIFT_U64 = 4,	/* bitfield-specific left bitshift */
-	BPF_FIELD_RSHIFT_U64 = 5,
+	BPF_FIELD_RSHIFT_U64 = 5,	/* bitfield-specific right bitshift */
+	BPF_TYPE_ID_LOCAL = 6,		/* type ID in local BPF object */
+	BPF_TYPE_ID_TARGET = 7,		/* type ID in target kernel */
+	BPF_TYPE_EXISTS = 8,		/* type existence in target kernel */
+	BPF_TYPE_SIZE = 9,		/* type size in bytes */
+	BPF_ENUMVAL_EXISTS = 10,	/* enum value existence in target kernel */
+	BPF_ENUMVAL_VALUE = 11,		/* enum value integer value */
 };
-/* The minimum bpf_field_reloc checked by the loader
+/* The minimum bpf_core_relo checked by the loader
  *
- * Field relocation captures the following data:
+ * CO-RE relocation captures the following data:
  * - insn_off - instruction offset (in bytes) within a BPF program that needs
  *   its insn->imm field to be relocated with actual field info;
  * - type_id - BTF type ID of the "root" (containing) entity of a relocatable
- *   field;
+ *   type or field;
  * - access_str_off - offset into corresponding .BTF string section. String
- *   itself encodes an accessed field using a sequence of field and array
+ *   interpretation depends on specific relocation kind:
- *   indicies, separated by colon (:). It's conceptually very close to LLVM's
+ *     - for field-based relocations, string encodes an accessed field using
- *   getelementptr ([0]) instruction's arguments for identifying offset to
+ *     a sequence of field and array indices, separated by colon (:). It's
- *   a field.
+ *     conceptually very close to LLVM's getelementptr ([0]) instruction's
+ *     arguments for identifying offset to a field.
+ *     - for type-based relocations, strings is expected to be just "0";
+ *     - for enum value-based relocations, string contains an index of enum
+ *     value within its enum type;
  *
  * Example to provide a better feel.
  *
  *
  *   [0] https://llvm.org/docs/LangRef.html#getelementptr-instruction
  */
-struct bpf_field_reloc {
+struct bpf_core_relo {
 	__u32   insn_off;
 	__u32   type_id;
 	__u32   access_str_off;
-	enum bpf_field_info_kind kind;
+	enum bpf_core_relo_kind kind;
 };
 #endif /* __LIBBPF_LIBBPF_INTERNAL_H */

tools/lib/bpf/libbpf_probes.c+3−5

 #include "libbpf.h"
 #include "libbpf_internal.h"
-/* make sure libbpf doesn't use kernel-only integer typedefs */
-#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
 static bool grep(const char *buffer, const char *pattern)
 {
 	return !!strstr(buffer, pattern);
 	return btf_fd;
 }
-static int load_sk_storage_btf(void)
+static int load_local_storage_btf(void)
 {
 	const char strs[] = "\0bpf_spin_lock\0val\0cnt\0l";
 	/* struct bpf_spin_lock {
 		key_size	= 0;
 		break;
 	case BPF_MAP_TYPE_SK_STORAGE:
+	case BPF_MAP_TYPE_INODE_STORAGE:
 		btf_key_type_id = 1;
 		btf_value_type_id = 3;
 		value_size = 8;
 		max_entries = 0;
 		map_flags = BPF_F_NO_PREALLOC;
-		btf_fd = load_sk_storage_btf();
+		btf_fd = load_local_storage_btf();
 		if (btf_fd < 0)
 			return false;
 		break;

tools/lib/bpf/netlink.c+6−122

@@ -15,13 +15,12 @@ · 14 unmodified lines
15	#include "libbpf_internal.h"	15	#include "libbpf_internal.h"
16	#include "nlattr.h"	16	#include "nlattr.h"
17		17
18	/* make sure libbpf doesn't use kernel-only integer typedefs */
19	#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
20
21	#ifndef SOL_NETLINK	18	#ifndef SOL_NETLINK
22	#define SOL_NETLINK 270	19	#define SOL_NETLINK 270
23	#endif	20	#endif
24		21
		22	typedef int (libbpf_dump_nlmsg_t)(void cookie, void msg, struct nlattr *tb);
		23
25	typedef int (__dump_nlmsg_t)(struct nlmsghdr nlmsg, libbpf_dump_nlmsg_t,	24	typedef int (__dump_nlmsg_t)(struct nlmsghdr nlmsg, libbpf_dump_nlmsg_t,
26	void *cookie);	25	void *cookie);
27		26
@@ -31,7 +30,7 @@ struct xdp_id_md { · 3 unmodified lines
31	struct xdp_link_info info;	30	struct xdp_link_info info;
32	};	31	};
33		32
34	int libbpf_netlink_open(__u32 *nl_pid)	33	static int libbpf_netlink_open(__u32 *nl_pid)
35	{	34	{
36	struct sockaddr_nl sa;	35	struct sockaddr_nl sa;
37	socklen_t addrlen;	36	socklen_t addrlen;
@@ -283,6 +282,9 @@ static int get_xdp_info(void cookie, void msg, struct nlattr **tb) · 245 unmodified lines
283	return 0;	282	return 0;
284	}	283	}
285		284
		285	static int libbpf_nl_get_link(int sock, unsigned int nl_pid,
		286	libbpf_dump_nlmsg_t dump_link_nlmsg, void *cookie);
		287
286	int bpf_get_link_xdp_info(int ifindex, struct xdp_link_info *info,	288	int bpf_get_link_xdp_info(int ifindex, struct xdp_link_info *info,
287	size_t info_size, __u32 flags)	289	size_t info_size, __u32 flags)
288	{	290	{
@@ -368,121 +370,3 @@ int libbpf_nl_get_link(int sock, unsigned int nl_pid, · 79 unmodified lines
368	return bpf_netlink_recv(sock, nl_pid, seq, __dump_link_nlmsg,	370	return bpf_netlink_recv(sock, nl_pid, seq, __dump_link_nlmsg,
369	dump_link_nlmsg, cookie);	371	dump_link_nlmsg, cookie);
370	}	372	}
371
372	static int __dump_class_nlmsg(struct nlmsghdr *nlh,
373	libbpf_dump_nlmsg_t dump_class_nlmsg,
374	void *cookie)
375	{
376	struct nlattr tb[TCA_MAX + 1], attr;
377	struct tcmsg *t = NLMSG_DATA(nlh);
378	int len;
379
380	len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*t));
381	attr = (struct nlattr ) ((void ) t + NLMSG_ALIGN(sizeof(*t)));
382	if (libbpf_nla_parse(tb, TCA_MAX, attr, len, NULL) != 0)
383	return -LIBBPF_ERRNO__NLPARSE;
384
385	return dump_class_nlmsg(cookie, t, tb);
386	}
387
388	int libbpf_nl_get_class(int sock, unsigned int nl_pid, int ifindex,
389	libbpf_dump_nlmsg_t dump_class_nlmsg, void *cookie)
390	{
391	struct {
392	struct nlmsghdr nlh;
393	struct tcmsg t;
394	} req = {
395	.nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg)),
396	.nlh.nlmsg_type = RTM_GETTCLASS,
397	.nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
398	.t.tcm_family = AF_UNSPEC,
399	.t.tcm_ifindex = ifindex,
400	};
401	int seq = time(NULL);
402
403	req.nlh.nlmsg_seq = seq;
404	if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
405	return -errno;
406
407	return bpf_netlink_recv(sock, nl_pid, seq, __dump_class_nlmsg,
408	dump_class_nlmsg, cookie);
409	}
410
411	static int __dump_qdisc_nlmsg(struct nlmsghdr *nlh,
412	libbpf_dump_nlmsg_t dump_qdisc_nlmsg,
413	void *cookie)
414	{
415	struct nlattr tb[TCA_MAX + 1], attr;
416	struct tcmsg *t = NLMSG_DATA(nlh);
417	int len;
418
419	len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*t));
420	attr = (struct nlattr ) ((void ) t + NLMSG_ALIGN(sizeof(*t)));
421	if (libbpf_nla_parse(tb, TCA_MAX, attr, len, NULL) != 0)
422	return -LIBBPF_ERRNO__NLPARSE;
423
424	return dump_qdisc_nlmsg(cookie, t, tb);
425	}
426
427	int libbpf_nl_get_qdisc(int sock, unsigned int nl_pid, int ifindex,
428	libbpf_dump_nlmsg_t dump_qdisc_nlmsg, void *cookie)
429	{
430	struct {
431	struct nlmsghdr nlh;
432	struct tcmsg t;
433	} req = {
434	.nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg)),
435	.nlh.nlmsg_type = RTM_GETQDISC,
436	.nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
437	.t.tcm_family = AF_UNSPEC,
438	.t.tcm_ifindex = ifindex,
439	};
440	int seq = time(NULL);
441
442	req.nlh.nlmsg_seq = seq;
443	if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
444	return -errno;
445
446	return bpf_netlink_recv(sock, nl_pid, seq, __dump_qdisc_nlmsg,
447	dump_qdisc_nlmsg, cookie);
448	}
449
450	static int __dump_filter_nlmsg(struct nlmsghdr *nlh,
451	libbpf_dump_nlmsg_t dump_filter_nlmsg,
452	void *cookie)
453	{
454	struct nlattr tb[TCA_MAX + 1], attr;
455	struct tcmsg *t = NLMSG_DATA(nlh);
456	int len;
457
458	len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*t));
459	attr = (struct nlattr ) ((void ) t + NLMSG_ALIGN(sizeof(*t)));
460	if (libbpf_nla_parse(tb, TCA_MAX, attr, len, NULL) != 0)
461	return -LIBBPF_ERRNO__NLPARSE;
462
463	return dump_filter_nlmsg(cookie, t, tb);
464	}
465
466	int libbpf_nl_get_filter(int sock, unsigned int nl_pid, int ifindex, int handle,
467	libbpf_dump_nlmsg_t dump_filter_nlmsg, void *cookie)
468	{
469	struct {
470	struct nlmsghdr nlh;
471	struct tcmsg t;
472	} req = {
473	.nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg)),
474	.nlh.nlmsg_type = RTM_GETTFILTER,
475	.nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
476	.t.tcm_family = AF_UNSPEC,
477	.t.tcm_ifindex = ifindex,
478	.t.tcm_parent = handle,
479	};
480	int seq = time(NULL);
481
482	req.nlh.nlmsg_seq = seq;
483	if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
484	return -errno;
485
486	return bpf_netlink_recv(sock, nl_pid, seq, __dump_filter_nlmsg,
487	dump_filter_nlmsg, cookie);
488	}

tools/lib/bpf/nlattr.c+3−6

@@ -7,14 +7,11 @@ · 6 unmodified lines
7	*/	7	*/
8		8
9	#include <errno.h>	9	#include <errno.h>
10	#include "nlattr.h"
11	#include "libbpf_internal.h"
12	#include <linux/rtnetlink.h>
13	#include <string.h>	10	#include <string.h>
14	#include <stdio.h>	11	#include <stdio.h>
15		12	#include <linux/rtnetlink.h>
16	/* make sure libbpf doesn't use kernel-only integer typedefs */	13	#include "nlattr.h"
17	#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64	14	#include "libbpf_internal.h"
18		15
19	static uint16_t nla_attr_minlen[LIBBPF_NLA_TYPE_MAX+1] = {	16	static uint16_t nla_attr_minlen[LIBBPF_NLA_TYPE_MAX+1] = {
20	[LIBBPF_NLA_U8] = sizeof(uint8_t),	17	[LIBBPF_NLA_U8] = sizeof(uint8_t),

tools/lib/bpf/ringbuf.c+2−6

 #include <asm/barrier.h>
 #include <sys/mman.h>
 #include <sys/epoll.h>
-#include <tools/libc_compat.h>
 #include "libbpf.h"
 #include "libbpf_internal.h"
 #include "bpf.h"
-/* make sure libbpf doesn't use kernel-only integer typedefs */
-#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
 struct ring {
 	ring_buffer_sample_fn sample_cb;
 	void *ctx;
 		return -EINVAL;
 	}
-	tmp = reallocarray(rb->rings, rb->ring_cnt + 1, sizeof(*rb->rings));
+	tmp = libbpf_reallocarray(rb->rings, rb->ring_cnt + 1, sizeof(*rb->rings));
 	if (!tmp)
 		return -ENOMEM;
 	rb->rings = tmp;
-	tmp = reallocarray(rb->events, rb->ring_cnt + 1, sizeof(*rb->events));
+	tmp = libbpf_reallocarray(rb->events, rb->ring_cnt + 1, sizeof(*rb->events));
 	if (!tmp)
 		return -ENOMEM;
 	rb->events = tmp;

tools/lib/bpf/xsk.c+244−135

@@ -20,6 +20,7 @@ · 19 unmodified lines
20	#include <linux/if_ether.h>	20	#include <linux/if_ether.h>
21	#include <linux/if_packet.h>	21	#include <linux/if_packet.h>
22	#include <linux/if_xdp.h>	22	#include <linux/if_xdp.h>
		23	#include <linux/list.h>
23	#include <linux/sockios.h>	24	#include <linux/sockios.h>
24	#include <net/if.h>	25	#include <net/if.h>
25	#include <sys/ioctl.h>	26	#include <sys/ioctl.h>
@@ -32,9 +33,6 @@ · 6 unmodified lines
32	#include "libbpf_internal.h"	33	#include "libbpf_internal.h"
33	#include "xsk.h"	34	#include "xsk.h"
34		35
35	/* make sure libbpf doesn't use kernel-only integer typedefs */
36	#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
37
38	#ifndef SOL_XDP	36	#ifndef SOL_XDP
39	#define SOL_XDP 283	37	#define SOL_XDP 283
40	#endif	38	#endif
@@ -48,26 +46,35 @@ · 7 unmodified lines
48	#endif	46	#endif
49		47
50	struct xsk_umem {	48	struct xsk_umem {
51	struct xsk_ring_prod *fill;	49	struct xsk_ring_prod *fill_save;
52	struct xsk_ring_cons *comp;	50	struct xsk_ring_cons *comp_save;
53	char *umem_area;	51	char *umem_area;
54	struct xsk_umem_config config;	52	struct xsk_umem_config config;
55	int fd;	53	int fd;
56	int refcount;	54	int refcount;
		55	struct list_head ctx_list;
		56	};
		57
		58	struct xsk_ctx {
		59	struct xsk_ring_prod *fill;
		60	struct xsk_ring_cons *comp;
		61	__u32 queue_id;
		62	struct xsk_umem *umem;
		63	int refcount;
		64	int ifindex;
		65	struct list_head list;
		66	int prog_fd;
		67	int xsks_map_fd;
		68	char ifname[IFNAMSIZ];
57	};	69	};
58		70
59	struct xsk_socket {	71	struct xsk_socket {
60	struct xsk_ring_cons *rx;	72	struct xsk_ring_cons *rx;
61	struct xsk_ring_prod *tx;	73	struct xsk_ring_prod *tx;
62	__u64 outstanding_tx;	74	__u64 outstanding_tx;
63	struct xsk_umem *umem;	75	struct xsk_ctx *ctx;
64	struct xsk_socket_config config;	76	struct xsk_socket_config config;
65	int fd;	77	int fd;
66	int ifindex;
67	int prog_fd;
68	int xsks_map_fd;
69	__u32 queue_id;
70	char ifname[IFNAMSIZ];
71	};	78	};
72		79
73	struct xsk_nl_info {	80	struct xsk_nl_info {
@@ -203,15 +210,73 @@ static int xsk_get_mmap_offsets(int fd, struct xdp_mmap_offsets *off) · 129 unmodified lines
203	return -EINVAL;	210	return -EINVAL;
204	}	211	}
205		212
		213	static int xsk_create_umem_rings(struct xsk_umem *umem, int fd,
		214	struct xsk_ring_prod *fill,
		215	struct xsk_ring_cons *comp)
		216	{
		217	struct xdp_mmap_offsets off;
		218	void *map;
		219	int err;
		220
		221	err = setsockopt(fd, SOL_XDP, XDP_UMEM_FILL_RING,
		222	&umem->config.fill_size,
		223	sizeof(umem->config.fill_size));
		224	if (err)
		225	return -errno;
		226
		227	err = setsockopt(fd, SOL_XDP, XDP_UMEM_COMPLETION_RING,
		228	&umem->config.comp_size,
		229	sizeof(umem->config.comp_size));
		230	if (err)
		231	return -errno;
		232
		233	err = xsk_get_mmap_offsets(fd, &off);
		234	if (err)
		235	return -errno;
		236
		237	map = mmap(NULL, off.fr.desc + umem->config.fill_size * sizeof(__u64),
		238	PROT_READ \| PROT_WRITE, MAP_SHARED \| MAP_POPULATE, fd,
		239	XDP_UMEM_PGOFF_FILL_RING);
		240	if (map == MAP_FAILED)
		241	return -errno;
		242
		243	fill->mask = umem->config.fill_size - 1;
		244	fill->size = umem->config.fill_size;
		245	fill->producer = map + off.fr.producer;
		246	fill->consumer = map + off.fr.consumer;
		247	fill->flags = map + off.fr.flags;
		248	fill->ring = map + off.fr.desc;
		249	fill->cached_cons = umem->config.fill_size;
		250
		251	map = mmap(NULL, off.cr.desc + umem->config.comp_size * sizeof(__u64),
		252	PROT_READ \| PROT_WRITE, MAP_SHARED \| MAP_POPULATE, fd,
		253	XDP_UMEM_PGOFF_COMPLETION_RING);
		254	if (map == MAP_FAILED) {
		255	err = -errno;
		256	goto out_mmap;
		257	}
		258
		259	comp->mask = umem->config.comp_size - 1;
		260	comp->size = umem->config.comp_size;
		261	comp->producer = map + off.cr.producer;
		262	comp->consumer = map + off.cr.consumer;
		263	comp->flags = map + off.cr.flags;
		264	comp->ring = map + off.cr.desc;
		265
		266	return 0;
		267
		268	out_mmap:
		269	munmap(map, off.fr.desc + umem->config.fill_size * sizeof(__u64));
		270	return err;
		271	}
		272
206	int xsk_umem__create_v0_0_4(struct xsk_umem *umem_ptr, void umem_area,	273	int xsk_umem__create_v0_0_4(struct xsk_umem *umem_ptr, void umem_area,
207	__u64 size, struct xsk_ring_prod *fill,	274	__u64 size, struct xsk_ring_prod *fill,
208	struct xsk_ring_cons *comp,	275	struct xsk_ring_cons *comp,
209	const struct xsk_umem_config *usr_config)	276	const struct xsk_umem_config *usr_config)
210	{	277	{
211	struct xdp_mmap_offsets off;
212	struct xdp_umem_reg mr;	278	struct xdp_umem_reg mr;
213	struct xsk_umem *umem;	279	struct xsk_umem *umem;
214	void *map;
215	int err;	280	int err;
216		281
217	if (!umem_area \|\| !umem_ptr \|\| !fill \|\| !comp)	282	if (!umem_area \|\| !umem_ptr \|\| !fill \|\| !comp)
@@ -230,6 +295,7 @@ int xsk_umem__create_v0_0_4(struct xsk_umem *umem_ptr, void umem_area, · 12 unmodified lines
230	}	295	}
231		296
232	umem->umem_area = umem_area;	297	umem->umem_area = umem_area;
		298	INIT_LIST_HEAD(&umem->ctx_list);
233	xsk_set_umem_config(&umem->config, usr_config);	299	xsk_set_umem_config(&umem->config, usr_config);
234		300
235	memset(&mr, 0, sizeof(mr));	301	memset(&mr, 0, sizeof(mr));
@@ -244,71 +310,16 @@ int xsk_umem__create_v0_0_4(struct xsk_umem *umem_ptr, void umem_area, · 8 unmodified lines
244	err = -errno;	310	err = -errno;
245	goto out_socket;	311	goto out_socket;
246	}	312	}
247	err = setsockopt(umem->fd, SOL_XDP, XDP_UMEM_FILL_RING,
248	&umem->config.fill_size,
249	sizeof(umem->config.fill_size));
250	if (err) {
251	err = -errno;
252	goto out_socket;
253	}
254	err = setsockopt(umem->fd, SOL_XDP, XDP_UMEM_COMPLETION_RING,
255	&umem->config.comp_size,
256	sizeof(umem->config.comp_size));
257	if (err) {
258	err = -errno;
259	goto out_socket;
260	}
261		313
262	err = xsk_get_mmap_offsets(umem->fd, &off);	314	err = xsk_create_umem_rings(umem, umem->fd, fill, comp);
263	if (err) {	315	if (err)
264	err = -errno;
265	goto out_socket;
266	}
267
268	map = mmap(NULL, off.fr.desc + umem->config.fill_size * sizeof(__u64),
269	PROT_READ \| PROT_WRITE, MAP_SHARED \| MAP_POPULATE, umem->fd,
270	XDP_UMEM_PGOFF_FILL_RING);
271	if (map == MAP_FAILED) {
272	err = -errno;
273	goto out_socket;	316	goto out_socket;
274	}
275
276	umem->fill = fill;
277	fill->mask = umem->config.fill_size - 1;
278	fill->size = umem->config.fill_size;
279	fill->producer = map + off.fr.producer;
280	fill->consumer = map + off.fr.consumer;
281	fill->flags = map + off.fr.flags;
282	fill->ring = map + off.fr.desc;
283	fill->cached_prod = *fill->producer;
284	/* cached_cons is "size" bigger than the real consumer pointer
285	* See xsk_prod_nb_free
286	*/
287	fill->cached_cons = *fill->consumer + umem->config.fill_size;
288
289	map = mmap(NULL, off.cr.desc + umem->config.comp_size * sizeof(__u64),
290	PROT_READ \| PROT_WRITE, MAP_SHARED \| MAP_POPULATE, umem->fd,
291	XDP_UMEM_PGOFF_COMPLETION_RING);
292	if (map == MAP_FAILED) {
293	err = -errno;
294	goto out_mmap;
295	}
296
297	umem->comp = comp;
298	comp->mask = umem->config.comp_size - 1;
299	comp->size = umem->config.comp_size;
300	comp->producer = map + off.cr.producer;
301	comp->consumer = map + off.cr.consumer;
302	comp->flags = map + off.cr.flags;
303	comp->ring = map + off.cr.desc;
304	comp->cached_prod = *comp->producer;
305	comp->cached_cons = *comp->consumer;
306		317
		318	umem->fill_save = fill;
		319	umem->comp_save = comp;
307	*umem_ptr = umem;	320	*umem_ptr = umem;
308	return 0;	321	return 0;
309		322
310	out_mmap:
311	munmap(map, off.fr.desc + umem->config.fill_size * sizeof(__u64));
312	out_socket:	323	out_socket:
313	close(umem->fd);	324	close(umem->fd);
314	out_umem_alloc:	325	out_umem_alloc:
@@ -342,6 +353,7 @@ DEFAULT_VERSION(xsk_umem__create_v0_0_4, xsk_umem__create, LIBBPF_0.0.4) · 27 unmodified lines
342	static int xsk_load_xdp_prog(struct xsk_socket *xsk)	353	static int xsk_load_xdp_prog(struct xsk_socket *xsk)
343	{	354	{
344	static const int log_buf_size = 16 * 1024;	355	static const int log_buf_size = 16 * 1024;
		356	struct xsk_ctx *ctx = xsk->ctx;
345	char log_buf[log_buf_size];	357	char log_buf[log_buf_size];
346	int err, prog_fd;	358	int err, prog_fd;
347		359
@@ -369,7 +381,7 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk) · 21 unmodified lines
369	/* (u32 )(r10 - 4) = r2 */	381	/* (u32 )(r10 - 4) = r2 */
370	BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_2, -4),	382	BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_2, -4),
371	/* r1 = xskmap[] */	383	/* r1 = xskmap[] */
372	BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),	384	BPF_LD_MAP_FD(BPF_REG_1, ctx->xsks_map_fd),
373	/* r3 = XDP_PASS */	385	/* r3 = XDP_PASS */
374	BPF_MOV64_IMM(BPF_REG_3, 2),	386	BPF_MOV64_IMM(BPF_REG_3, 2),
375	/* call bpf_redirect_map */	387	/* call bpf_redirect_map */
@@ -381,7 +393,7 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk) · 5 unmodified lines
381	/* r2 += -4 */	393	/* r2 += -4 */
382	BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),	394	BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
383	/* r1 = xskmap[] */	395	/* r1 = xskmap[] */
384	BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),	396	BPF_LD_MAP_FD(BPF_REG_1, ctx->xsks_map_fd),
385	/* call bpf_map_lookup_elem */	397	/* call bpf_map_lookup_elem */
386	BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),	398	BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),
387	/* r1 = r0 */	399	/* r1 = r0 */
@@ -393,7 +405,7 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk) · 5 unmodified lines
393	/* r2 = (u32 )(r10 - 4) */	405	/* r2 = (u32 )(r10 - 4) */
394	BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_10, -4),	406	BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_10, -4),
395	/* r1 = xskmap[] */	407	/* r1 = xskmap[] */
396	BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),	408	BPF_LD_MAP_FD(BPF_REG_1, ctx->xsks_map_fd),
397	/* r3 = 0 */	409	/* r3 = 0 */
398	BPF_MOV64_IMM(BPF_REG_3, 0),	410	BPF_MOV64_IMM(BPF_REG_3, 0),
399	/* call bpf_redirect_map */	411	/* call bpf_redirect_map */
@@ -411,19 +423,21 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk) · 11 unmodified lines
411	return prog_fd;	423	return prog_fd;
412	}	424	}
413		425
414	err = bpf_set_link_xdp_fd(xsk->ifindex, prog_fd, xsk->config.xdp_flags);	426	err = bpf_set_link_xdp_fd(xsk->ctx->ifindex, prog_fd,
		427	xsk->config.xdp_flags);
415	if (err) {	428	if (err) {
416	close(prog_fd);	429	close(prog_fd);
417	return err;	430	return err;
418	}	431	}
419		432
420	xsk->prog_fd = prog_fd;	433	ctx->prog_fd = prog_fd;
421	return 0;	434	return 0;
422	}	435	}
423		436
424	static int xsk_get_max_queues(struct xsk_socket *xsk)	437	static int xsk_get_max_queues(struct xsk_socket *xsk)
425	{	438	{
426	struct ethtool_channels channels = { .cmd = ETHTOOL_GCHANNELS };	439	struct ethtool_channels channels = { .cmd = ETHTOOL_GCHANNELS };
		440	struct xsk_ctx *ctx = xsk->ctx;
427	struct ifreq ifr = {};	441	struct ifreq ifr = {};
428	int fd, err, ret;	442	int fd, err, ret;
429		443
@@ -432,7 +446,7 @@ static int xsk_get_max_queues(struct xsk_socket *xsk) · 2 unmodified lines
432	return -errno;	446	return -errno;
433		447
434	ifr.ifr_data = (void *)&channels;	448	ifr.ifr_data = (void *)&channels;
435	memcpy(ifr.ifr_name, xsk->ifname, IFNAMSIZ - 1);	449	memcpy(ifr.ifr_name, ctx->ifname, IFNAMSIZ - 1);
436	ifr.ifr_name[IFNAMSIZ - 1] = '\0';	450	ifr.ifr_name[IFNAMSIZ - 1] = '\0';
437	err = ioctl(fd, SIOCETHTOOL, &ifr);	451	err = ioctl(fd, SIOCETHTOOL, &ifr);
438	if (err && errno != EOPNOTSUPP) {	452	if (err && errno != EOPNOTSUPP) {
@@ -460,6 +474,7 @@ out: · 21 unmodified lines
460		474
461	static int xsk_create_bpf_maps(struct xsk_socket *xsk)	475	static int xsk_create_bpf_maps(struct xsk_socket *xsk)
462	{	476	{
		477	struct xsk_ctx *ctx = xsk->ctx;
463	int max_queues;	478	int max_queues;
464	int fd;	479	int fd;
465		480
@@ -472,15 +487,17 @@ static int xsk_create_bpf_maps(struct xsk_socket *xsk) · 6 unmodified lines
472	if (fd < 0)	487	if (fd < 0)
473	return fd;	488	return fd;
474		489
475	xsk->xsks_map_fd = fd;	490	ctx->xsks_map_fd = fd;
476		491
477	return 0;	492	return 0;
478	}	493	}
479		494
480	static void xsk_delete_bpf_maps(struct xsk_socket *xsk)	495	static void xsk_delete_bpf_maps(struct xsk_socket *xsk)
481	{	496	{
482	bpf_map_delete_elem(xsk->xsks_map_fd, &xsk->queue_id);	497	struct xsk_ctx *ctx = xsk->ctx;
483	close(xsk->xsks_map_fd);	498
		499	bpf_map_delete_elem(ctx->xsks_map_fd, &ctx->queue_id);
		500	close(ctx->xsks_map_fd);
484	}	501	}
485		502
486	static int xsk_lookup_bpf_maps(struct xsk_socket *xsk)	503	static int xsk_lookup_bpf_maps(struct xsk_socket *xsk)
@@ -488,10 +505,11 @@ static int xsk_lookup_bpf_maps(struct xsk_socket *xsk) · 1 unmodified line
488	__u32 i, *map_ids, num_maps, prog_len = sizeof(struct bpf_prog_info);	505	__u32 i, *map_ids, num_maps, prog_len = sizeof(struct bpf_prog_info);
489	__u32 map_len = sizeof(struct bpf_map_info);	506	__u32 map_len = sizeof(struct bpf_map_info);
490	struct bpf_prog_info prog_info = {};	507	struct bpf_prog_info prog_info = {};
		508	struct xsk_ctx *ctx = xsk->ctx;
491	struct bpf_map_info map_info;	509	struct bpf_map_info map_info;
492	int fd, err;	510	int fd, err;
493		511
494	err = bpf_obj_get_info_by_fd(xsk->prog_fd, &prog_info, &prog_len);	512	err = bpf_obj_get_info_by_fd(ctx->prog_fd, &prog_info, &prog_len);
495	if (err)	513	if (err)
496	return err;	514	return err;
497		515
@@ -505,11 +523,11 @@ static int xsk_lookup_bpf_maps(struct xsk_socket *xsk) · 7 unmodified lines
505	prog_info.nr_map_ids = num_maps;	523	prog_info.nr_map_ids = num_maps;
506	prog_info.map_ids = (__u64)(unsigned long)map_ids;	524	prog_info.map_ids = (__u64)(unsigned long)map_ids;
507		525
508	err = bpf_obj_get_info_by_fd(xsk->prog_fd, &prog_info, &prog_len);	526	err = bpf_obj_get_info_by_fd(ctx->prog_fd, &prog_info, &prog_len);
509	if (err)	527	if (err)
510	goto out_map_ids;	528	goto out_map_ids;
511		529
512	xsk->xsks_map_fd = -1;	530	ctx->xsks_map_fd = -1;
513		531
514	for (i = 0; i < prog_info.nr_map_ids; i++) {	532	for (i = 0; i < prog_info.nr_map_ids; i++) {
515	fd = bpf_map_get_fd_by_id(map_ids[i]);	533	fd = bpf_map_get_fd_by_id(map_ids[i]);
@@ -523,7 +541,7 @@ static int xsk_lookup_bpf_maps(struct xsk_socket *xsk) · 7 unmodified lines
523	}	541	}
524		542
525	if (!strcmp(map_info.name, "xsks_map")) {	543	if (!strcmp(map_info.name, "xsks_map")) {
526	xsk->xsks_map_fd = fd;	544	ctx->xsks_map_fd = fd;
527	continue;	545	continue;
528	}	546	}
529		547
@@ -531,7 +549,7 @@ static int xsk_lookup_bpf_maps(struct xsk_socket *xsk) · 1 unmodified line
531	}	549	}
532		550
533	err = 0;	551	err = 0;
534	if (xsk->xsks_map_fd == -1)	552	if (ctx->xsks_map_fd == -1)
535	err = -ENOENT;	553	err = -ENOENT;
536		554
537	out_map_ids:	555	out_map_ids:
@@ -541,16 +559,19 @@ out_map_ids: · 3 unmodified lines
541		559
542	static int xsk_set_bpf_maps(struct xsk_socket *xsk)	560	static int xsk_set_bpf_maps(struct xsk_socket *xsk)
543	{	561	{
544	return bpf_map_update_elem(xsk->xsks_map_fd, &xsk->queue_id,	562	struct xsk_ctx *ctx = xsk->ctx;
		563
		564	return bpf_map_update_elem(ctx->xsks_map_fd, &ctx->queue_id,
545	&xsk->fd, 0);	565	&xsk->fd, 0);
546	}	566	}
547		567
548	static int xsk_setup_xdp_prog(struct xsk_socket *xsk)	568	static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
549	{	569	{
		570	struct xsk_ctx *ctx = xsk->ctx;
550	__u32 prog_id = 0;	571	__u32 prog_id = 0;
551	int err;	572	int err;
552		573
553	err = bpf_get_link_xdp_id(xsk->ifindex, &prog_id,	574	err = bpf_get_link_xdp_id(ctx->ifindex, &prog_id,
554	xsk->config.xdp_flags);	575	xsk->config.xdp_flags);
555	if (err)	576	if (err)
556	return err;	577	return err;
@@ -566,12 +587,12 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk) · 9 unmodified lines
566	return err;	587	return err;
567	}	588	}
568	} else {	589	} else {
569	xsk->prog_fd = bpf_prog_get_fd_by_id(prog_id);	590	ctx->prog_fd = bpf_prog_get_fd_by_id(prog_id);
570	if (xsk->prog_fd < 0)	591	if (ctx->prog_fd < 0)
571	return -errno;	592	return -errno;
572	err = xsk_lookup_bpf_maps(xsk);	593	err = xsk_lookup_bpf_maps(xsk);
573	if (err) {	594	if (err) {
574	close(xsk->prog_fd);	595	close(ctx->prog_fd);
575	return err;	596	return err;
576	}	597	}
577	}	598	}
@@ -580,25 +601,110 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk) · 2 unmodified lines
580	err = xsk_set_bpf_maps(xsk);	601	err = xsk_set_bpf_maps(xsk);
581	if (err) {	602	if (err) {
582	xsk_delete_bpf_maps(xsk);	603	xsk_delete_bpf_maps(xsk);
583	close(xsk->prog_fd);	604	close(ctx->prog_fd);
584	return err;	605	return err;
585	}	606	}
586		607
587	return 0;	608	return 0;
588	}	609	}
589		610
590	int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname,	611	static struct xsk_ctx xsk_get_ctx(struct xsk_umem umem, int ifindex,
591	__u32 queue_id, struct xsk_umem *umem,	612	__u32 queue_id)
592	struct xsk_ring_cons rx, struct xsk_ring_prod tx,	613	{
593	const struct xsk_socket_config *usr_config)	614	struct xsk_ctx *ctx;
		615
		616	if (list_empty(&umem->ctx_list))
		617	return NULL;
		618
		619	list_for_each_entry(ctx, &umem->ctx_list, list) {
		620	if (ctx->ifindex == ifindex && ctx->queue_id == queue_id) {
		621	ctx->refcount++;
		622	return ctx;
		623	}
		624	}
		625
		626	return NULL;
		627	}
		628
		629	static void xsk_put_ctx(struct xsk_ctx *ctx)
		630	{
		631	struct xsk_umem *umem = ctx->umem;
		632	struct xdp_mmap_offsets off;
		633	int err;
		634
		635	if (--ctx->refcount == 0) {
		636	err = xsk_get_mmap_offsets(umem->fd, &off);
		637	if (!err) {
		638	munmap(ctx->fill->ring - off.fr.desc,
		639	off.fr.desc + umem->config.fill_size *
		640	sizeof(__u64));
		641	munmap(ctx->comp->ring - off.cr.desc,
		642	off.cr.desc + umem->config.comp_size *
		643	sizeof(__u64));
		644	}
		645
		646	list_del(&ctx->list);
		647	free(ctx);
		648	}
		649	}
		650
		651	static struct xsk_ctx xsk_create_ctx(struct xsk_socket xsk,
		652	struct xsk_umem *umem, int ifindex,
		653	const char *ifname, __u32 queue_id,
		654	struct xsk_ring_prod *fill,
		655	struct xsk_ring_cons *comp)
		656	{
		657	struct xsk_ctx *ctx;
		658	int err;
		659
		660	ctx = calloc(1, sizeof(*ctx));
		661	if (!ctx)
		662	return NULL;
		663
		664	if (!umem->fill_save) {
		665	err = xsk_create_umem_rings(umem, xsk->fd, fill, comp);
		666	if (err) {
		667	free(ctx);
		668	return NULL;
		669	}
		670	} else if (umem->fill_save != fill \|\| umem->comp_save != comp) {
		671	/* Copy over rings to new structs. */
		672	memcpy(fill, umem->fill_save, sizeof(*fill));
		673	memcpy(comp, umem->comp_save, sizeof(*comp));
		674	}
		675
		676	ctx->ifindex = ifindex;
		677	ctx->refcount = 1;
		678	ctx->umem = umem;
		679	ctx->queue_id = queue_id;
		680	memcpy(ctx->ifname, ifname, IFNAMSIZ - 1);
		681	ctx->ifname[IFNAMSIZ - 1] = '\0';
		682
		683	umem->fill_save = NULL;
		684	umem->comp_save = NULL;
		685	ctx->fill = fill;
		686	ctx->comp = comp;
		687	list_add(&ctx->list, &umem->ctx_list);
		688	return ctx;
		689	}
		690
		691	int xsk_socket__create_shared(struct xsk_socket **xsk_ptr,
		692	const char *ifname,
		693	__u32 queue_id, struct xsk_umem *umem,
		694	struct xsk_ring_cons *rx,
		695	struct xsk_ring_prod *tx,
		696	struct xsk_ring_prod *fill,
		697	struct xsk_ring_cons *comp,
		698	const struct xsk_socket_config *usr_config)
594	{	699	{
595	void rx_map = NULL, tx_map = NULL;	700	void rx_map = NULL, tx_map = NULL;
596	struct sockaddr_xdp sxdp = {};	701	struct sockaddr_xdp sxdp = {};
597	struct xdp_mmap_offsets off;	702	struct xdp_mmap_offsets off;
598	struct xsk_socket *xsk;	703	struct xsk_socket *xsk;
599	int err;	704	struct xsk_ctx *ctx;
		705	int err, ifindex;
600		706
601	if (!umem \|\| !xsk_ptr \|\| !(rx \|\| tx))	707	if (!umem \|\| !xsk_ptr \|\| !(rx \|\| tx) \|\| !fill \|\| !comp)
602	return -EFAULT;	708	return -EFAULT;
603		709
604	xsk = calloc(1, sizeof(*xsk));	710	xsk = calloc(1, sizeof(*xsk));
@@ -609,10 +715,10 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 4 unmodified lines
609	if (err)	715	if (err)
610	goto out_xsk_alloc;	716	goto out_xsk_alloc;
611		717
612	if (umem->refcount &&	718	xsk->outstanding_tx = 0;
613	!(xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)) {	719	ifindex = if_nametoindex(ifname);
614	pr_warn("Error: shared umems not supported by libbpf supplied XDP program.\n");	720	if (!ifindex) {
615	err = -EBUSY;	721	err = -errno;
616	goto out_xsk_alloc;	722	goto out_xsk_alloc;
617	}	723	}
618		724
@@ -626,16 +732,16 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 7 unmodified lines
626	xsk->fd = umem->fd;	732	xsk->fd = umem->fd;
627	}	733	}
628		734
629	xsk->outstanding_tx = 0;	735	ctx = xsk_get_ctx(umem, ifindex, queue_id);
630	xsk->queue_id = queue_id;	736	if (!ctx) {
631	xsk->umem = umem;	737	ctx = xsk_create_ctx(xsk, umem, ifindex, ifname, queue_id,
632	xsk->ifindex = if_nametoindex(ifname);	738	fill, comp);
633	if (!xsk->ifindex) {	739	if (!ctx) {
634	err = -errno;	740	err = -ENOMEM;
635	goto out_socket;	741	goto out_socket;
		742	}
636	}	743	}
637	memcpy(xsk->ifname, ifname, IFNAMSIZ - 1);	744	xsk->ctx = ctx;
638	xsk->ifname[IFNAMSIZ - 1] = '\0';
639		745
640	if (rx) {	746	if (rx) {
641	err = setsockopt(xsk->fd, SOL_XDP, XDP_RX_RING,	747	err = setsockopt(xsk->fd, SOL_XDP, XDP_RX_RING,
@@ -643,7 +749,7 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 1 unmodified line
643	sizeof(xsk->config.rx_size));	749	sizeof(xsk->config.rx_size));
644	if (err) {	750	if (err) {
645	err = -errno;	751	err = -errno;
646	goto out_socket;	752	goto out_put_ctx;
647	}	753	}
648	}	754	}
649	if (tx) {	755	if (tx) {
@@ -652,14 +758,14 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 2 unmodified lines
652	sizeof(xsk->config.tx_size));	758	sizeof(xsk->config.tx_size));
653	if (err) {	759	if (err) {
654	err = -errno;	760	err = -errno;
655	goto out_socket;	761	goto out_put_ctx;
656	}	762	}
657	}	763	}
658		764
659	err = xsk_get_mmap_offsets(xsk->fd, &off);	765	err = xsk_get_mmap_offsets(xsk->fd, &off);
660	if (err) {	766	if (err) {
661	err = -errno;	767	err = -errno;
662	goto out_socket;	768	goto out_put_ctx;
663	}	769	}
664		770
665	if (rx) {	771	if (rx) {
@@ -669,7 +775,7 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 3 unmodified lines
669	xsk->fd, XDP_PGOFF_RX_RING);	775	xsk->fd, XDP_PGOFF_RX_RING);
670	if (rx_map == MAP_FAILED) {	776	if (rx_map == MAP_FAILED) {
671	err = -errno;	777	err = -errno;
672	goto out_socket;	778	goto out_put_ctx;
673	}	779	}
674		780
675	rx->mask = xsk->config.rx_size - 1;	781	rx->mask = xsk->config.rx_size - 1;
@@ -708,10 +814,10 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 32 unmodified lines
708	xsk->tx = tx;	814	xsk->tx = tx;
709		815
710	sxdp.sxdp_family = PF_XDP;	816	sxdp.sxdp_family = PF_XDP;
711	sxdp.sxdp_ifindex = xsk->ifindex;	817	sxdp.sxdp_ifindex = ctx->ifindex;
712	sxdp.sxdp_queue_id = xsk->queue_id;	818	sxdp.sxdp_queue_id = ctx->queue_id;
713	if (umem->refcount > 1) {	819	if (umem->refcount > 1) {
714	sxdp.sxdp_flags = XDP_SHARED_UMEM;	820	sxdp.sxdp_flags \|= XDP_SHARED_UMEM;
715	sxdp.sxdp_shared_umem_fd = umem->fd;	821	sxdp.sxdp_shared_umem_fd = umem->fd;
716	} else {	822	} else {
717	sxdp.sxdp_flags = xsk->config.bind_flags;	823	sxdp.sxdp_flags = xsk->config.bind_flags;
@@ -723,7 +829,7 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 5 unmodified lines
723	goto out_mmap_tx;	829	goto out_mmap_tx;
724	}	830	}
725		831
726	xsk->prog_fd = -1;	832	ctx->prog_fd = -1;
727		833
728	if (!(xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)) {	834	if (!(xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)) {
729	err = xsk_setup_xdp_prog(xsk);	835	err = xsk_setup_xdp_prog(xsk);
@@ -742,6 +848,8 @@ out_mmap_rx: · 12 unmodified lines
742	if (rx)	848	if (rx)
743	munmap(rx_map, off.rx.desc +	849	munmap(rx_map, off.rx.desc +
744	xsk->config.rx_size * sizeof(struct xdp_desc));	850	xsk->config.rx_size * sizeof(struct xdp_desc));
		851	out_put_ctx:
		852	xsk_put_ctx(ctx);
745	out_socket:	853	out_socket:
746	if (--umem->refcount)	854	if (--umem->refcount)
747	close(xsk->fd);	855	close(xsk->fd);
@@ -750,25 +858,24 @@ out_xsk_alloc: · 2 unmodified lines
750	return err;	858	return err;
751	}	859	}
752		860
753	int xsk_umem__delete(struct xsk_umem *umem)	861	int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname,
		862	__u32 queue_id, struct xsk_umem *umem,
		863	struct xsk_ring_cons rx, struct xsk_ring_prod tx,
		864	const struct xsk_socket_config *usr_config)
754	{	865	{
755	struct xdp_mmap_offsets off;	866	return xsk_socket__create_shared(xsk_ptr, ifname, queue_id, umem,
756	int err;	867	rx, tx, umem->fill_save,
		868	umem->comp_save, usr_config);
		869	}
757		870
		871	int xsk_umem__delete(struct xsk_umem *umem)
		872	{
758	if (!umem)	873	if (!umem)
759	return 0;	874	return 0;
760		875
761	if (umem->refcount)	876	if (umem->refcount)
762	return -EBUSY;	877	return -EBUSY;
763		878
764	err = xsk_get_mmap_offsets(umem->fd, &off);
765	if (!err) {
766	munmap(umem->fill->ring - off.fr.desc,
767	off.fr.desc + umem->config.fill_size * sizeof(__u64));
768	munmap(umem->comp->ring - off.cr.desc,
769	off.cr.desc + umem->config.comp_size * sizeof(__u64));
770	}
771
772	close(umem->fd);	879	close(umem->fd);
773	free(umem);	880	free(umem);
774		881
@@ -778,15 +885,16 @@ int xsk_umem__delete(struct xsk_umem *umem) · 3 unmodified lines
778	void xsk_socket__delete(struct xsk_socket *xsk)	885	void xsk_socket__delete(struct xsk_socket *xsk)
779	{	886	{
780	size_t desc_sz = sizeof(struct xdp_desc);	887	size_t desc_sz = sizeof(struct xdp_desc);
		888	struct xsk_ctx *ctx = xsk->ctx;
781	struct xdp_mmap_offsets off;	889	struct xdp_mmap_offsets off;
782	int err;	890	int err;
783		891
784	if (!xsk)	892	if (!xsk)
785	return;	893	return;
786		894
787	if (xsk->prog_fd != -1) {	895	if (ctx->prog_fd != -1) {
788	xsk_delete_bpf_maps(xsk);	896	xsk_delete_bpf_maps(xsk);
789	close(xsk->prog_fd);	897	close(ctx->prog_fd);
790	}	898	}
791		899
792	err = xsk_get_mmap_offsets(xsk->fd, &off);	900	err = xsk_get_mmap_offsets(xsk->fd, &off);
@@ -799,14 +907,15 @@ void xsk_socket__delete(struct xsk_socket *xsk) · 6 unmodified lines
799	munmap(xsk->tx->ring - off.tx.desc,	907	munmap(xsk->tx->ring - off.tx.desc,
800	off.tx.desc + xsk->config.tx_size * desc_sz);	908	off.tx.desc + xsk->config.tx_size * desc_sz);
801	}	909	}
802
803	}	910	}
804		911
805	xsk->umem->refcount--;	912	xsk_put_ctx(ctx);
		913
		914	ctx->umem->refcount--;
806	/* Do not close an fd that also has an associated umem connected	915	/* Do not close an fd that also has an associated umem connected
807	* to it.	916	* to it.
808	*/	917	*/
809	if (xsk->fd != xsk->umem->fd)	918	if (xsk->fd != ctx->umem->fd)
810	close(xsk->fd);	919	close(xsk->fd);
811	free(xsk);	920	free(xsk);
812	}	921	}

tools/lib/bpf/xsk.h+9−0

@@ -234,6 +234,15 @@ LIBBPF_API int xsk_socket__create(struct xsk_socket **xsk, · 233 unmodified lines
234	struct xsk_ring_cons *rx,	234	struct xsk_ring_cons *rx,
235	struct xsk_ring_prod *tx,	235	struct xsk_ring_prod *tx,
236	const struct xsk_socket_config *config);	236	const struct xsk_socket_config *config);
		237	LIBBPF_API int
		238	xsk_socket__create_shared(struct xsk_socket **xsk_ptr,
		239	const char *ifname,
		240	__u32 queue_id, struct xsk_umem *umem,
		241	struct xsk_ring_cons *rx,
		242	struct xsk_ring_prod *tx,
		243	struct xsk_ring_prod *fill,
		244	struct xsk_ring_cons *comp,
		245	const struct xsk_socket_config *config);
237		246
238	/* Returns 0 for success and -EBUSY if the umem is still in use. */	247	/* Returns 0 for success and -EBUSY if the umem is still in use. */
239	LIBBPF_API int xsk_umem__delete(struct xsk_umem *umem);	248	LIBBPF_API int xsk_umem__delete(struct xsk_umem *umem);

tools/perf/Makefile.config+0−4

   EXTLIBS += -lelf
   $(call detected,CONFIG_LIBELF)
-  ifeq ($(feature-libelf-mmap), 1)
-    CFLAGS += -DHAVE_LIBELF_MMAP_SUPPORT
-  endif
   ifeq ($(feature-libelf-getphdrnum), 1)
     CFLAGS += -DHAVE_ELF_GETPHDRNUM_SUPPORT
   endif

tools/perf/util/symbol.h+1−1

@@ -28,7 +28,7 @@ struct option; · 27 unmodified lines
28	* libelf 0.8.x and earlier do not support ELF_C_READ_MMAP;	28	* libelf 0.8.x and earlier do not support ELF_C_READ_MMAP;
29	* for newer versions we can use mmap to reduce memory usage:	29	* for newer versions we can use mmap to reduce memory usage:
30	*/	30	*/
31	#ifdef HAVE_LIBELF_MMAP_SUPPORT	31	#ifdef ELF_C_READ_MMAP
32	# define PERF_ELF_C_READ_MMAP ELF_C_READ_MMAP	32	# define PERF_ELF_C_READ_MMAP ELF_C_READ_MMAP
33	#else	33	#else
34	# define PERF_ELF_C_READ_MMAP ELF_C_READ	34	# define PERF_ELF_C_READ_MMAP ELF_C_READ

tools/testing/selftests/bpf/Makefile+1−1

@@ -316,7 +316,7 @@ $(TRUNNER_BPF_OBJS): $(TRUNNER_OUTPUT)/%.o: \ · 315 unmodified lines
316	$(TRUNNER_BPF_PROGS_DIR)/%.c \	316	$(TRUNNER_BPF_PROGS_DIR)/%.c \
317	$(TRUNNER_BPF_PROGS_DIR)/*.h \	317	$(TRUNNER_BPF_PROGS_DIR)/*.h \
318	$$(INCLUDE_DIR)/vmlinux.h \	318	$$(INCLUDE_DIR)/vmlinux.h \
319	$$(BPFOBJ) \| $(TRUNNER_OUTPUT)	319	$(wildcard $(BPFDIR)/bpf_*.h) \| $(TRUNNER_OUTPUT)
320	$$(call $(TRUNNER_BPF_BUILD_RULE),$$<,$$@, \	320	$$(call $(TRUNNER_BPF_BUILD_RULE),$$<,$$@, \
321	$(TRUNNER_BPF_CFLAGS), \	321	$(TRUNNER_BPF_CFLAGS), \
322	$(TRUNNER_BPF_LDFLAGS))	322	$(TRUNNER_BPF_LDFLAGS))

tools/testing/selftests/bpf/README.rst+21−0

@@ -43,3 +43,24 @@ This is due to a llvm BPF backend bug. The fix · 42 unmodified lines
43	https://reviews.llvm.org/D78466	43	https://reviews.llvm.org/D78466
44	has been pushed to llvm 10.x release branch and will be	44	has been pushed to llvm 10.x release branch and will be
45	available in 10.0.1. The fix is available in llvm 11.0.0 trunk.	45	available in 10.0.1. The fix is available in llvm 11.0.0 trunk.
		46
		47	BPF CO-RE-based tests and Clang version
		48	=======================================
		49
		50	A set of selftests use BPF target-specific built-ins, which might require
		51	bleeding-edge Clang versions (Clang 12 nightly at this time).
		52
		53	Few sub-tests of core_reloc test suit (part of test_progs test runner) require
		54	the following built-ins, listed with corresponding Clang diffs introducing
		55	them to Clang/LLVM. These sub-tests are going to be skipped if Clang is too
		56	old to support them, they shouldn't cause build failures or runtime test
		57	failures:
		58
		59	- __builtin_btf_type_id() ([0], [1], [2]);
		60	- __builtin_preserve_type_info(), __builtin_preserve_enum_value() ([3], [4]).
		61
		62	[0] https://reviews.llvm.org/D74572
		63	[1] https://reviews.llvm.org/D74668
		64	[2] https://reviews.llvm.org/D85174
		65	[3] https://reviews.llvm.org/D83878
		66	[4] https://reviews.llvm.org/D83242

tools/testing/selftests/bpf/bench.c+2−0

 extern const struct bench bench_trig_rawtp;
 extern const struct bench bench_trig_kprobe;
 extern const struct bench bench_trig_fentry;
+extern const struct bench bench_trig_fentry_sleep;
 extern const struct bench bench_trig_fmodret;
 extern const struct bench bench_rb_libbpf;
 extern const struct bench bench_rb_custom;
 	&bench_trig_rawtp,
 	&bench_trig_kprobe,
 	&bench_trig_fentry,
+	&bench_trig_fentry_sleep,
 	&bench_trig_fmodret,
 	&bench_rb_libbpf,
 	&bench_rb_custom,

tools/testing/selftests/bpf/benchs/bench_trigger.c+17−0

@@ -90,6 +90,12 @@ static void trigger_fentry_setup() · 89 unmodified lines
90	attach_bpf(ctx.skel->progs.bench_trigger_fentry);	90	attach_bpf(ctx.skel->progs.bench_trigger_fentry);
91	}	91	}
92		92
		93	static void trigger_fentry_sleep_setup()
		94	{
		95	setup_ctx();
		96	attach_bpf(ctx.skel->progs.bench_trigger_fentry_sleep);
		97	}
		98
93	static void trigger_fmodret_setup()	99	static void trigger_fmodret_setup()
94	{	100	{
95	setup_ctx();	101	setup_ctx();
@@ -155,6 +161,17 @@ const struct bench bench_trig_fentry = { · 59 unmodified lines
155	.report_final = hits_drops_report_final,	161	.report_final = hits_drops_report_final,
156	};	162	};
157		163
		164	const struct bench bench_trig_fentry_sleep = {
		165	.name = "trig-fentry-sleep",
		166	.validate = trigger_validate,
		167	.setup = trigger_fentry_sleep_setup,
		168	.producer_thread = trigger_producer,
		169	.consumer_thread = trigger_consumer,
		170	.measure = trigger_measure,
		171	.report_progress = hits_drops_report_progress,
		172	.report_final = hits_drops_report_final,
		173	};
		174
158	const struct bench bench_trig_fmodret = {	175	const struct bench bench_trig_fmodret = {
159	.name = "trig-fmodret",	176	.name = "trig-fmodret",
160	.validate = trigger_validate,	177	.validate = trigger_validate,

tools/testing/selftests/bpf/network_helpers.c+37−0

@@ -104,6 +104,43 @@ error_close: · 103 unmodified lines
104	return -1;	104	return -1;
105	}	105	}
106		106
		107	int fastopen_connect(int server_fd, const char *data, unsigned int data_len,
		108	int timeout_ms)
		109	{
		110	struct sockaddr_storage addr;
		111	socklen_t addrlen = sizeof(addr);
		112	struct sockaddr_in *addr_in;
		113	int fd, ret;
		114
		115	if (getsockname(server_fd, (struct sockaddr *)&addr, &addrlen)) {
		116	log_err("Failed to get server addr");
		117	return -1;
		118	}
		119
		120	addr_in = (struct sockaddr_in *)&addr;
		121	fd = socket(addr_in->sin_family, SOCK_STREAM, 0);
		122	if (fd < 0) {
		123	log_err("Failed to create client socket");
		124	return -1;
		125	}
		126
		127	if (settimeo(fd, timeout_ms))
		128	goto error_close;
		129
		130	ret = sendto(fd, data, data_len, MSG_FASTOPEN, (struct sockaddr *)&addr,
		131	addrlen);
		132	if (ret != data_len) {
		133	log_err("sendto(data, %u) != %d\n", data_len, ret);
		134	goto error_close;
		135	}
		136
		137	return fd;
		138
		139	error_close:
		140	save_errno_close(fd);
		141	return -1;
		142	}
		143
107	static int connect_fd_to_addr(int fd,	144	static int connect_fd_to_addr(int fd,
108	const struct sockaddr_storage *addr,	145	const struct sockaddr_storage *addr,
109	socklen_t addrlen)	146	socklen_t addrlen)

tools/testing/selftests/bpf/network_helpers.h+2−0

@@ -37,6 +37,8 @@ int start_server(int family, int type, const char *addr, __u16 port, · 36 unmodified lines
37	int timeout_ms);	37	int timeout_ms);
38	int connect_to_fd(int server_fd, int timeout_ms);	38	int connect_to_fd(int server_fd, int timeout_ms);
39	int connect_fd_to_fd(int client_fd, int server_fd, int timeout_ms);	39	int connect_fd_to_fd(int client_fd, int server_fd, int timeout_ms);
		40	int fastopen_connect(int server_fd, const char *data, unsigned int data_len,
		41	int timeout_ms);
40	int make_sockaddr(int family, const char *addr_str, __u16 port,	42	int make_sockaddr(int family, const char *addr_str, __u16 port,
41	struct sockaddr_storage addr, socklen_t len);	43	struct sockaddr_storage addr, socklen_t len);
42		44

tools/testing/selftests/bpf/prog_tests/btf_map_in_map.c+34−1

@@ -53,7 +53,7 @@ static int kern_sync_rcu(void) · 52 unmodified lines
53	return err;	53	return err;
54	}	54	}
55		55
56	void test_btf_map_in_map(void)	56	static void test_lookup_update(void)
57	{	57	{
58	int err, key = 0, val, i;	58	int err, key = 0, val, i;
59	struct test_btf_map_in_map *skel;	59	struct test_btf_map_in_map *skel;
@@ -143,3 +143,36 @@ void test_btf_map_in_map(void) · 83 unmodified lines
143	cleanup:	143	cleanup:
144	test_btf_map_in_map__destroy(skel);	144	test_btf_map_in_map__destroy(skel);
145	}	145	}
		146
		147	static void test_diff_size(void)
		148	{
		149	struct test_btf_map_in_map *skel;
		150	int err, inner_map_fd, zero = 0;
		151
		152	skel = test_btf_map_in_map__open_and_load();
		153	if (CHECK(!skel, "skel_open", "failed to open&load skeleton\n"))
		154	return;
		155
		156	inner_map_fd = bpf_map__fd(skel->maps.sockarr_sz2);
		157	err = bpf_map_update_elem(bpf_map__fd(skel->maps.outer_sockarr), &zero,
		158	&inner_map_fd, 0);
		159	CHECK(err, "outer_sockarr inner map size check",
		160	"cannot use a different size inner_map\n");
		161
		162	inner_map_fd = bpf_map__fd(skel->maps.inner_map_sz2);
		163	err = bpf_map_update_elem(bpf_map__fd(skel->maps.outer_arr), &zero,
		164	&inner_map_fd, 0);
		165	CHECK(!err, "outer_arr inner map size check",
		166	"incorrectly updated with a different size inner_map\n");
		167
		168	test_btf_map_in_map__destroy(skel);
		169	}
		170
		171	void test_btf_map_in_map(void)
		172	{
		173	if (test__start_subtest("lookup_update"))
		174	test_lookup_update();
		175
		176	if (test__start_subtest("diff_size"))
		177	test_diff_size();
		178	}

tools/testing/selftests/bpf/prog_tests/core_reloc.c+324−26

 #include "progs/core_reloc_types.h"
 #include <sys/mman.h>
 #include <sys/syscall.h>
+#include <bpf/btf.h>
+static int duration = 0;
 #define STRUCT_TO_CHAR_PTR(struct_name) (const char *)&(struct struct_name)
 	.fails = true,							\
 }
-#define EXISTENCE_CASE_COMMON(name)					\
+#define FIELD_EXISTS_CASE_COMMON(name)					\
 	.case_name = #name,						\
 	.bpf_obj_file = "test_core_reloc_existence.o",			\
-	.btf_src_file = "btf__core_reloc_" #name ".o",			\
+	.btf_src_file = "btf__core_reloc_" #name ".o"			\
-	.relaxed_core_relocs = true
-#define EXISTENCE_ERR_CASE(name) {					\
+#define FIELD_EXISTS_ERR_CASE(name) {					\
-	EXISTENCE_CASE_COMMON(name),					\

421 more lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/d_path.c+147−0

This diff is large: its 148 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c+68−0

This diff is large: its 86 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/perf_buffer.c+54−11

This diff is large: its 106 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/resolve_btfids.c+38−1

This diff is large: its 58 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/sk_assign.c+3−2

This diff is large: its 26 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/sockmap_basic.c+76−0

This diff is large: its 95 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/tcp_hdr_options.c+622−0

This diff is large: its 623 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/test_bpffs.c+94−0

This diff is large: its 95 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/test_local_storage.c+60−0

This diff is large: its 61 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/test_lsm.c+9−0

This diff is large: its 30 lines aren’t shown here.

tools/testing/selftests/bpf/progs/btf__core_reloc_enumval.c+3−0