Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Daniel Borkmann says: ==================== pull-request: bpf-next 2020-09-01 The following pull-request contains BPF updates for your *net-next* tree. There are two small conflicts when pulling, resolve as follows: 1) Merge conflict in tools/lib/bpf/libbpf.c between 88a82120282b ("libbpf: Factor out common ELF operations and improve logging") in bpf-next and 1e891e513e16 ("libbpf: Fix map index used in error message") in net-next. Resolve by taking the hunk in bpf-next: [...] scn = elf_sec_by_idx(obj, obj->efile.btf_maps_shndx); data = elf_sec_data(obj, scn); if (!scn || !data) { pr_warn("elf: failed to get %s map definitions for %s\n", MAPS_ELF_SEC, obj->path); return -EINVAL; } [...] 2) Merge conflict in drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c between 9647c57b11e5 ("xsk: i40e: ice: ixgbe: mlx5: Test for dma_need_sync earlier for better performance") in bpf-next and e20f0dbf204f ("net/mlx5e: RX, Add a prefetch command for small L1_CACHE_BYTES") in net-next. Resolve the two locations by retaining net_prefetch() and taking xsk_buff_dma_sync_for_cpu() from bpf-next. Should look like: [...] xdp_set_data_meta_invalid(xdp); xsk_buff_dma_sync_for_cpu(xdp, rq->xsk_pool); net_prefetch(xdp->data); [...] We've added 133 non-merge commits during the last 14 day(s) which contain a total of 246 files changed, 13832 insertions(+), 3105 deletions(-). The main changes are: 1) Initial support for sleepable BPF programs along with bpf_copy_from_user() helper for tracing to reliably access user memory, from Alexei Starovoitov. 2) Add BPF infra for writing and parsing TCP header options, from Martin KaFai Lau. 3) bpf_d_path() helper for returning full path for given 'struct path', from Jiri Olsa. 4) AF_XDP support for shared umems between devices and queues, from Magnus Karlsson. 5) Initial prep work for full BPF-to-BPF call support in libbpf, from Andrii Nakryiko. 6) Generalize bpf_sk_storage map & add local storage for inodes, from KP Singh. 7) Implement sockmap/hash updates from BPF context, from Lorenz Bauer. 8) BPF xor verification for scalar types & add BPF link iterator, from Yonghong Song. 9) Use target's prog type for BPF_PROG_TYPE_EXT prog verification, from Udip Pant. 10) Rework BPF tracing samples to use libbpf loader, from Daniel T. Lee. 11) Fix xdpsock sample to really cycle through all buffers, from Weqaar Janjua. 12) Improve type safety for tun/veth XDP frame handling, from Maciej Żenczykowski. 13) Various smaller cleanups and improvements all over the place. ====================

David S. Miller committed 6 years ago

Commit 150f29f2 parents 8aa639e ebc4ecd246 files · +13832 −3105Browse files

Unified · Split

246 files

Documentation/bpf/bpf_devel_QA.rst +12 −7
Documentation/bpf/btf.rst +25 −0
Documentation/bpf/index.rst +1 −0
Documentation/bpf/prog_sk_lookup.rst +98 −0
Documentation/networking/af_xdp.rst +58 −10
arch/x86/net/bpf_jit_comp.c +21 −11
drivers/net/ethernet/intel/i40e/i40e_ethtool.c +1 −1
drivers/net/ethernet/intel/i40e/i40e_main.c +15 −14
drivers/net/ethernet/intel/i40e/i40e_txrx.c +5 −5
drivers/net/ethernet/intel/i40e/i40e_txrx.h +1 −1
drivers/net/ethernet/intel/i40e/i40e_xsk.c +43 −38
drivers/net/ethernet/intel/i40e/i40e_xsk.h +2 −2
drivers/net/ethernet/intel/ice/ice.h +9 −9
drivers/net/ethernet/intel/ice/ice_base.c +8 −8
drivers/net/ethernet/intel/ice/ice_lib.c +1 −1
drivers/net/ethernet/intel/ice/ice_main.c +5 −5
drivers/net/ethernet/intel/ice/ice_txrx.c +4 −4
drivers/net/ethernet/intel/ice/ice_txrx.h +1 −1
drivers/net/ethernet/intel/ice/ice_xsk.c +69 −69
drivers/net/ethernet/intel/ice/ice_xsk.h +4 −3
drivers/net/ethernet/intel/ixgbe/ixgbe.h +1 −1
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +17 −17
drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h +4 −3
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +32 −31
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 −1
drivers/net/ethernet/mellanox/mlx5/core/en.h +10 −9
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +2 −3
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/pool.c +55 −55
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/pool.h +27 −0
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +2 −2
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h +5 −5
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c +6 −6
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.h +1 −1
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c +7 −7
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.h +3 −3
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.h +0 −29
drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c +1 −1
drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.c +1 −1
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +25 −24
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +8 −8
drivers/net/tun.c +0 −18
drivers/net/veth.c +3 −3
include/linux/bpf-cgroup.h +25 −0
include/linux/bpf.h +52 −0
include/linux/bpf_local_storage.h +163 −0
include/linux/bpf_lsm.h +29 −0
include/linux/bpf_types.h +3 −0
include/linux/btf.h +1 −2
include/linux/btf_ids.h +50 −1
include/linux/filter.h +6 −2
include/linux/if_tun.h +14 −5
include/linux/netdevice.h +5 −5
include/linux/rcupdate_trace.h +8 −1
include/linux/skmsg.h +0 −17
include/linux/tcp.h +15 −5
include/net/bpf_sk_storage.h +14 −0
include/net/inet_connection_sock.h +2 −0
include/net/request_sock.h +8 −1
include/net/sock.h +2 −2
include/net/tcp.h +55 −4
include/net/xdp_sock.h +14 −16
include/net/xdp_sock_drv.h +65 −57
include/net/xsk_buff_pool.h +43 −10
include/uapi/linux/bpf.h +393 −5
init/Kconfig +3 −0
kernel/Makefile +1 −1
kernel/bpf/Makefile +3 −0
kernel/bpf/arraymap.c +17 −0
kernel/bpf/bpf_inode_storage.c +274 −0
kernel/bpf/bpf_iter.c +58 −0
kernel/bpf/bpf_local_storage.c +600 −0
kernel/bpf/bpf_lsm.c +20 −1
kernel/bpf/bpf_struct_ops.c +2 −4
kernel/bpf/btf.c +139 −24
kernel/bpf/cpumap.c +4 −8
kernel/bpf/devmap.c +9 −8
kernel/bpf/hashtab.c +10 −6
kernel/bpf/helpers.c +22 −0
kernel/bpf/inode.c +113 −3
kernel/bpf/lpm_trie.c +1 −0
kernel/bpf/map_in_map.c +9 −15
kernel/bpf/map_in_map.h +0 −2
kernel/bpf/map_iter.c +15 −0
kernel/bpf/preload/Kconfig +26 −0
kernel/bpf/preload/Makefile +23 −0
kernel/bpf/preload/bpf_preload.h +16 −0
kernel/bpf/preload/bpf_preload_kern.c +91 −0
kernel/bpf/preload/bpf_preload_umd_blob.S +7 −0
kernel/bpf/preload/iterators/.gitignore +2 −0
kernel/bpf/preload/iterators/Makefile +57 −0
kernel/bpf/preload/iterators/README +4 −0
kernel/bpf/preload/iterators/bpf_preload_common.h +13 −0
kernel/bpf/preload/iterators/iterators.bpf.c +114 −0
kernel/bpf/preload/iterators/iterators.c +94 −0
kernel/bpf/preload/iterators/iterators.skel.h +410 −0
kernel/bpf/queue_stack_maps.c +2 −0
kernel/bpf/reuseport_array.c +1 −0
kernel/bpf/ringbuf.c +1 −0
kernel/bpf/stackmap.c +1 −0
kernel/bpf/syscall.c +44 −24
kernel/bpf/trampoline.c +26 −3
kernel/bpf/verifier.c +263 −20
kernel/trace/bpf_trace.c +50 −0
mm/filemap.c +4 −4
mm/page_alloc.c +1 −1
net/bpfilter/Kconfig +1 −0
net/core/bpf_sk_storage.c +138 −695
net/core/filter.c +409 −7
net/core/skmsg.c +28 −6
net/core/sock_map.c +37 −54
net/ethtool/channels.c +1 −1
net/ethtool/ioctl.c +1 −1
net/ipv4/tcp.c +11 −5
net/ipv4/tcp_bpf.c +5 −8
net/ipv4/tcp_fastopen.c +1 −1
net/ipv4/tcp_input.c +109 −18
net/ipv4/tcp_ipv4.c +3 −2
net/ipv4/tcp_minisocks.c +1 −0
net/ipv4/tcp_output.c +180 −13
net/ipv4/udp_bpf.c +4 −5
net/ipv6/tcp_ipv6.c +3 −2
net/xdp/xdp_umem.c +27 −198
net/xdp/xdp_umem.h +0 −6
net/xdp/xsk.c +142 −71
net/xdp/xsk.h +3 −7
net/xdp/xsk_buff_pool.c +324 −56
net/xdp/xsk_diag.c +9 −7
net/xdp/xsk_queue.h +6 −6
net/xdp/xskmap.c +8 −0
samples/bpf/Makefile +12 −9
samples/bpf/cpustat_kern.c +18 −18
samples/bpf/cpustat_user.c +40 −7
samples/bpf/lathist_kern.c +12 −12
samples/bpf/lathist_user.c +36 −6
samples/bpf/offwaketime_kern.c +26 −26
samples/bpf/offwaketime_user.c +51 −15
samples/bpf/spintest_kern.c +18 −18
samples/bpf/spintest_user.c +55 −13
samples/bpf/syscall_tp_kern.c +12 −12
samples/bpf/syscall_tp_user.c +42 −12
samples/bpf/task_fd_query_kern.c +1 −1
samples/bpf/task_fd_query_user.c +1 −1
samples/bpf/test_current_task_under_cgroup_kern.c +14 −13
samples/bpf/test_current_task_under_cgroup_user.c +42 −10
samples/bpf/test_probe_write_user_kern.c +6 −6
samples/bpf/test_probe_write_user_user.c +39 −10
samples/bpf/trace_output_kern.c +8 −7
samples/bpf/trace_output_user.c +37 −18
samples/bpf/tracex3_kern.c +1 −1
samples/bpf/xdpsock_user.c +19 −13
samples/bpf/xsk_fwd.c +1085 −0
scripts/bpf_helpers_doc.py +2 −0
security/bpf/hooks.c +6 −0
tools/bpf/bpftool/Documentation/bpftool-map.rst +1 −1
tools/bpf/bpftool/Makefile +5 −1
tools/bpf/bpftool/bash-completion/bpftool +2 −1
tools/bpf/bpftool/gen.c +0 −2
tools/bpf/bpftool/link.c +41 −3
tools/bpf/bpftool/map.c +2 −1
tools/bpf/bpftool/net.c +282 −17
tools/bpf/resolve_btfids/main.c +27 −2
tools/build/Makefile +2 −0
tools/build/Makefile.feature +0 −1
tools/build/feature/Makefile +0 −4
tools/build/feature/test-all.c +0 −4
tools/build/feature/test-libelf-mmap.c +0 −9
tools/include/linux/btf_ids.h +50 −1
tools/include/uapi/linux/bpf.h +393 −5
tools/lib/bpf/Makefile +9 −14
tools/lib/bpf/bpf.c +0 −3
tools/lib/bpf/bpf_core_read.h +101 −19
tools/lib/bpf/bpf_helpers.h +3 −0
tools/lib/bpf/bpf_prog_linfo.c +0 −3
tools/lib/bpf/bpf_tracing.h +2 −2
tools/lib/bpf/btf.c +13 −18
tools/lib/bpf/btf.h +0 −38
tools/lib/bpf/btf_dump.c +2 −7
tools/lib/bpf/hashmap.c +3 −0
tools/lib/bpf/libbpf.c +1242 −517
tools/lib/bpf/libbpf.h +4 −0
tools/lib/bpf/libbpf.map +9 −0
tools/lib/bpf/libbpf_internal.h +107 −31
tools/lib/bpf/libbpf_probes.c +3 −5
tools/lib/bpf/netlink.c +6 −122
tools/lib/bpf/nlattr.c +3 −6
tools/lib/bpf/ringbuf.c +2 −6
tools/lib/bpf/xsk.c +244 −135
tools/lib/bpf/xsk.h +9 −0
tools/perf/Makefile.config +0 −4
tools/perf/util/symbol.h +1 −1
tools/testing/selftests/bpf/Makefile +1 −1
tools/testing/selftests/bpf/README.rst +21 −0
tools/testing/selftests/bpf/bench.c +2 −0
tools/testing/selftests/bpf/benchs/bench_trigger.c +17 −0
tools/testing/selftests/bpf/network_helpers.c +37 −0
tools/testing/selftests/bpf/network_helpers.h +2 −0
tools/testing/selftests/bpf/prog_tests/btf_map_in_map.c +34 −1
tools/testing/selftests/bpf/prog_tests/core_reloc.c +324 −26
tools/testing/selftests/bpf/prog_tests/d_path.c +147 −0
tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c +68 −0
tools/testing/selftests/bpf/prog_tests/perf_buffer.c +54 −11
tools/testing/selftests/bpf/prog_tests/resolve_btfids.c +38 −1
tools/testing/selftests/bpf/prog_tests/sk_assign.c +3 −2
tools/testing/selftests/bpf/prog_tests/sockmap_basic.c +76 −0
tools/testing/selftests/bpf/prog_tests/tcp_hdr_options.c +622 −0
tools/testing/selftests/bpf/prog_tests/test_bpffs.c +94 −0
tools/testing/selftests/bpf/prog_tests/test_local_storage.c +60 −0
tools/testing/selftests/bpf/prog_tests/test_lsm.c +9 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_enumval.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_enumval___diff.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_enumval___err_missing.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_enumval___val3_missing.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_size___err_ambiguous.c +4 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_type_based.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_type_based___all_missing.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_type_based___diff_sz.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_type_based___fn_wrong_args.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_type_based___incompat.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_type_id.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_type_id___missing_targets.c +3 −0
tools/testing/selftests/bpf/progs/core_reloc_types.h +351 −1
tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c +27 −0
tools/testing/selftests/bpf/progs/freplace_attach_probe.c +40 −0
tools/testing/selftests/bpf/progs/freplace_cls_redirect.c +34 −0
tools/testing/selftests/bpf/progs/freplace_connect_v4_prog.c +19 −0
tools/testing/selftests/bpf/progs/local_storage.c +140 −0
tools/testing/selftests/bpf/progs/lsm.c +63 −1
tools/testing/selftests/bpf/progs/map_ptr_kern.c +3 −3
tools/testing/selftests/bpf/progs/test_btf_map_in_map.c +31 −0
tools/testing/selftests/bpf/progs/test_core_reloc_enumval.c +72 −0
tools/testing/selftests/bpf/progs/test_core_reloc_kernel.c +2 −0
tools/testing/selftests/bpf/progs/test_core_reloc_type_based.c +110 −0
tools/testing/selftests/bpf/progs/test_core_reloc_type_id.c +115 −0
tools/testing/selftests/bpf/progs/test_d_path.c +58 −0
tools/testing/selftests/bpf/progs/test_misc_tcp_hdr_options.c +325 −0
tools/testing/selftests/bpf/progs/test_pkt_access.c +20 −0
tools/testing/selftests/bpf/progs/test_sockmap_invalid_update.c +23 −0
tools/testing/selftests/bpf/progs/test_sockmap_update.c +48 −0
tools/testing/selftests/bpf/progs/test_tcp_hdr_options.c +623 −0
tools/testing/selftests/bpf/progs/test_vmlinux.c +9 −3
tools/testing/selftests/bpf/progs/trigger_bench.c +7 −0
tools/testing/selftests/bpf/test_current_pid_tgid_new_ns.c +1 −0
tools/testing/selftests/bpf/test_tcp_hdr_options.h +151 −0
tools/testing/selftests/bpf/test_verifier.c +18 −1
tools/testing/selftests/bpf/verifier/bounds.c +146 −0
tools/testing/selftests/bpf/verifier/d_path.c +37 −0

Documentation/bpf/bpf_devel_QA.rst+12−7

@@ -149,7 +149,7 @@ In case the patch or patch series has to be reworked and sent out · 148 unmodified lines
149	149	again in a second or later revision, it is also required to add a
150	150	version number (``v2``, ``v3``, ...) into the subject prefix::
151	151
152		− git format-patch --subject-prefix='PATCH net-next v2' start..finish
	152	+ git format-patch --subject-prefix='PATCH bpf-next v2' start..finish
153	153
154	154	When changes have been requested to the patch series, always send the
155	155	whole patch series again with the feedback incorporated (never send
@@ -479,17 +479,18 @@ LLVM's static compiler lists the supported targets through · 323 unmodified lines
479	479
480	480	$ llc --version
481	481	LLVM (http://llvm.org/):
482		− LLVM version 6.0.0svn
	482	+ LLVM version 10.0.0
483	483	Optimized build.
484	484	Default target: x86_64-unknown-linux-gnu
485	485	Host CPU: skylake
486	486
487	487	Registered Targets:
488		− bpf - BPF (host endian)
489		− bpfeb - BPF (big endian)
490		− bpfel - BPF (little endian)
491		− x86 - 32-bit X86: Pentium-Pro and above
492		− x86-64 - 64-bit X86: EM64T and AMD64
	488	+ aarch64 - AArch64 (little endian)
	489	+ bpf - BPF (host endian)
	490	+ bpfeb - BPF (big endian)
	491	+ bpfel - BPF (little endian)
	492	+ x86 - 32-bit X86: Pentium-Pro and above
	493	+ x86-64 - 64-bit X86: EM64T and AMD64
493	494
494	495	For developers in order to utilize the latest features added to LLVM's
495	496	BPF back end, it is advisable to run the latest LLVM releases. Support
@@ -517,6 +518,10 @@ from the git repositories:: · 21 unmodified lines
517	518	The built binaries can then be found in the build/bin/ directory, where
518	519	you can point the PATH variable to.
519	520
	521	+Set ``-DLLVM_TARGETS_TO_BUILD`` equal to the target you wish to build, you
	522	+will find a full list of targets within the llvm-project/llvm/lib/Target
	523	+directory.
	524	+
520	525	Q: Reporting LLVM BPF issues
521	526	----------------------------
522	527	Q: Should I notify BPF kernel maintainers about issues in LLVM's BPF code

Documentation/bpf/btf.rst+25−0

@@ -724,6 +724,31 @@ want to define unused entry in BTF_ID_LIST, like:: · 723 unmodified lines
724	724	BTF_ID_UNUSED
725	725	BTF_ID(struct, task_struct)
726	726
	727	+The ``BTF_SET_START/END`` macros pair defines sorted list of BTF ID values
	728	+and their count, with following syntax::
	729	+
	730	+ BTF_SET_START(set)
	731	+ BTF_ID(type1, name1)
	732	+ BTF_ID(type2, name2)
	733	+ BTF_SET_END(set)
	734	+
	735	+resulting in following layout in .BTF_ids section::
	736	+
	737	+ __BTF_ID__set__set:
	738	+ .zero 4
	739	+ __BTF_ID__type1__name1__3:
	740	+ .zero 4
	741	+ __BTF_ID__type2__name2__4:
	742	+ .zero 4
	743	+
	744	+The ``struct btf_id_set set;`` variable is defined to access the list.
	745	+
	746	+The ``typeX`` name can be one of following::
	747	+
	748	+ struct, union, typedef, func
	749	+
	750	+and is used as a filter when resolving the BTF ID value.
	751	+
727	752	All the BTF ID lists and sets are compiled in the .BTF_ids section and
728	753	resolved during the linking phase of kernel build by ``resolve_btfids`` tool.
729	754

Documentation/bpf/index.rst+1−0

@@ -52,6 +52,7 @@ Program types · 51 unmodified lines
52	52	prog_cgroup_sysctl
53	53	prog_flow_dissector
54	54	bpf_lsm
	55	+ prog_sk_lookup
55	56
56	57
57	58	Map types

Documentation/bpf/prog_sk_lookup.rst+98−0

@@ -0,0 +1,98 @@
	1	+.. SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
	2	+
	3	+=====================
	4	+BPF sk_lookup program
	5	+=====================
	6	+
	7	+BPF sk_lookup program type (``BPF_PROG_TYPE_SK_LOOKUP``) introduces programmability
	8	+into the socket lookup performed by the transport layer when a packet is to be
	9	+delivered locally.
	10	+
	11	+When invoked BPF sk_lookup program can select a socket that will receive the
	12	+incoming packet by calling the ``bpf_sk_assign()`` BPF helper function.
	13	+
	14	+Hooks for a common attach point (``BPF_SK_LOOKUP``) exist for both TCP and UDP.
	15	+
	16	+Motivation
	17	+==========
	18	+
	19	+BPF sk_lookup program type was introduced to address setup scenarios where
	20	+binding sockets to an address with ``bind()`` socket call is impractical, such
	21	+as:
	22	+
	23	+1. receiving connections on a range of IP addresses, e.g. 192.0.2.0/24, when
	24	+ binding to a wildcard address ``INADRR_ANY`` is not possible due to a port
	25	+ conflict,
	26	+2. receiving connections on all or a wide range of ports, i.e. an L7 proxy use
	27	+ case.
	28	+
	29	+Such setups would require creating and ``bind()``'ing one socket to each of the
	30	+IP address/port in the range, leading to resource consumption and potential
	31	+latency spikes during socket lookup.
	32	+
	33	+Attachment
	34	+==========
	35	+
	36	+BPF sk_lookup program can be attached to a network namespace with
	37	+``bpf(BPF_LINK_CREATE, ...)`` syscall using the ``BPF_SK_LOOKUP`` attach type and a
	38	+netns FD as attachment ``target_fd``.
	39	+
	40	+Multiple programs can be attached to one network namespace. Programs will be
	41	+invoked in the same order as they were attached.
	42	+
	43	+Hooks
	44	+=====
	45	+
	46	+The attached BPF sk_lookup programs run whenever the transport layer needs to
	47	+find a listening (TCP) or an unconnected (UDP) socket for an incoming packet.
	48	+
	49	+Incoming traffic to established (TCP) and connected (UDP) sockets is delivered
	50	+as usual without triggering the BPF sk_lookup hook.
	51	+
	52	+The attached BPF programs must return with either ``SK_PASS`` or ``SK_DROP``
	53	+verdict code. As for other BPF program types that are network filters,
	54	+``SK_PASS`` signifies that the socket lookup should continue on to regular
	55	+hashtable-based lookup, while ``SK_DROP`` causes the transport layer to drop the
	56	+packet.
	57	+
	58	+A BPF sk_lookup program can also select a socket to receive the packet by
	59	+calling ``bpf_sk_assign()`` BPF helper. Typically, the program looks up a socket
	60	+in a map holding sockets, such as ``SOCKMAP`` or ``SOCKHASH``, and passes a
	61	+``struct bpf_sock *`` to ``bpf_sk_assign()`` helper to record the
	62	+selection. Selecting a socket only takes effect if the program has terminated
	63	+with ``SK_PASS`` code.
	64	+
	65	+When multiple programs are attached, the end result is determined from return
	66	+codes of all the programs according to the following rules:
	67	+
	68	+1. If any program returned ``SK_PASS`` and selected a valid socket, the socket
	69	+ is used as the result of the socket lookup.
	70	+2. If more than one program returned ``SK_PASS`` and selected a socket, the last
	71	+ selection takes effect.
	72	+3. If any program returned ``SK_DROP``, and no program returned ``SK_PASS`` and
	73	+ selected a socket, socket lookup fails.
	74	+4. If all programs returned ``SK_PASS`` and none of them selected a socket,
	75	+ socket lookup continues on.
	76	+
	77	+API
	78	+===
	79	+
	80	+In its context, an instance of ``struct bpf_sk_lookup``, BPF sk_lookup program
	81	+receives information about the packet that triggered the socket lookup. Namely:
	82	+
	83	+* IP version (``AF_INET`` or ``AF_INET6``),
	84	+* L4 protocol identifier (``IPPROTO_TCP`` or ``IPPROTO_UDP``),
	85	+* source and destination IP address,
	86	+* source and destination L4 port,
	87	+* the socket that has been selected with ``bpf_sk_assign()``.
	88	+
	89	+Refer to ``struct bpf_sk_lookup`` declaration in ``linux/bpf.h`` user API
	90	+header, and `bpf-helpers(7)
	91	+<https://man7.org/linux/man-pages/man7/bpf-helpers.7.html>`_ man-page section
	92	+for ``bpf_sk_assign()`` for details.
	93	+
	94	+Example
	95	+=======
	96	+
	97	+See ``tools/testing/selftests/bpf/prog_tests/sk_lookup.c`` for the reference
	98	+implementation.

Documentation/networking/af_xdp.rst+58−10

@@ -258,14 +258,21 @@ socket into zero-copy mode or fail. · 257 unmodified lines
258	258	XDP_SHARED_UMEM bind flag
259	259	-------------------------
260	260
261		−This flag enables you to bind multiple sockets to the same UMEM, but
262		−only if they share the same queue id. In this mode, each socket has
263		−their own RX and TX rings, but the UMEM (tied to the fist socket
264		−created) only has a single FILL ring and a single COMPLETION
265		−ring. To use this mode, create the first socket and bind it in the normal
266		−way. Create a second socket and create an RX and a TX ring, or at
267		−least one of them, but no FILL or COMPLETION rings as the ones from
268		−the first socket will be used. In the bind call, set he
	261	+This flag enables you to bind multiple sockets to the same UMEM. It
	262	+works on the same queue id, between queue ids and between
	263	+netdevs/devices. In this mode, each socket has their own RX and TX
	264	+rings as usual, but you are going to have one or more FILL and
	265	+COMPLETION ring pairs. You have to create one of these pairs per
	266	+unique netdev and queue id tuple that you bind to.
	267	+
	268	+Starting with the case were we would like to share a UMEM between
	269	+sockets bound to the same netdev and queue id. The UMEM (tied to the
	270	+fist socket created) will only have a single FILL ring and a single
	271	+COMPLETION ring as there is only on unique netdev,queue_id tuple that
	272	+we have bound to. To use this mode, create the first socket and bind
	273	+it in the normal way. Create a second socket and create an RX and a TX
	274	+ring, or at least one of them, but no FILL or COMPLETION rings as the
	275	+ones from the first socket will be used. In the bind call, set he
269	276	XDP_SHARED_UMEM option and provide the initial socket's fd in the
270	277	sxdp_shared_umem_fd field. You can attach an arbitrary number of extra
271	278	sockets this way.
@@ -305,11 +312,41 @@ concurrently. There are no synchronization primitives in the · 33 unmodified lines
305	312	libbpf code that protects multiple users at this point in time.
306	313
307	314	Libbpf uses this mode if you create more than one socket tied to the
308		−same umem. However, note that you need to supply the
	315	+same UMEM. However, note that you need to supply the
309	316	XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD libbpf_flag with the
310	317	xsk_socket__create calls and load your own XDP program as there is no
311	318	built in one in libbpf that will route the traffic for you.
312	319
	320	+The second case is when you share a UMEM between sockets that are
	321	+bound to different queue ids and/or netdevs. In this case you have to
	322	+create one FILL ring and one COMPLETION ring for each unique
	323	+netdev,queue_id pair. Let us say you want to create two sockets bound
	324	+to two different queue ids on the same netdev. Create the first socket
	325	+and bind it in the normal way. Create a second socket and create an RX
	326	+and a TX ring, or at least one of them, and then one FILL and
	327	+COMPLETION ring for this socket. Then in the bind call, set he
	328	+XDP_SHARED_UMEM option and provide the initial socket's fd in the
	329	+sxdp_shared_umem_fd field as you registered the UMEM on that
	330	+socket. These two sockets will now share one and the same UMEM.
	331	+
	332	+There is no need to supply an XDP program like the one in the previous
	333	+case where sockets were bound to the same queue id and
	334	+device. Instead, use the NIC's packet steering capabilities to steer
	335	+the packets to the right queue. In the previous example, there is only
	336	+one queue shared among sockets, so the NIC cannot do this steering. It
	337	+can only steer between queues.
	338	+
	339	+In libbpf, you need to use the xsk_socket__create_shared() API as it
	340	+takes a reference to a FILL ring and a COMPLETION ring that will be
	341	+created for you and bound to the shared UMEM. You can use this
	342	+function for all the sockets you create, or you can use it for the
	343	+second and following ones and use xsk_socket__create() for the first
	344	+one. Both methods yield the same result.
	345	+
	346	+Note that a UMEM can be shared between sockets on the same queue id
	347	+and device, as well as between queues on the same device and between
	348	+devices at the same time.
	349	+
313	350	XDP_USE_NEED_WAKEUP bind flag
314	351	-----------------------------
315	352
@@ -364,7 +401,7 @@ resources by only setting up one of them. Both the FILL ring and the · 48 unmodified lines
364	401	COMPLETION ring are mandatory as you need to have a UMEM tied to your
365	402	socket. But if the XDP_SHARED_UMEM flag is used, any socket after the
366	403	first one does not have a UMEM and should in that case not have any
367		−FILL or COMPLETION rings created as the ones from the shared umem will
	404	+FILL or COMPLETION rings created as the ones from the shared UMEM will
368	405	be used. Note, that the rings are single-producer single-consumer, so
369	406	do not try to access them from multiple processes at the same
370	407	time. See the XDP_SHARED_UMEM section.
@@ -567,6 +604,17 @@ A: The short answer is no, that is not supported at the moment. The · 196 unmodified lines
567	604	switch, or other distribution mechanism, in your NIC to direct
568	605	traffic to the correct queue id and socket.
569	606
	607	+Q: My packets are sometimes corrupted. What is wrong?
	608	+
	609	+A: Care has to be taken not to feed the same buffer in the UMEM into
	610	+ more than one ring at the same time. If you for example feed the
	611	+ same buffer into the FILL ring and the TX ring at the same time, the
	612	+ NIC might receive data into the buffer at the same time it is
	613	+ sending it. This will cause some packets to become corrupted. Same
	614	+ thing goes for feeding the same buffer into the FILL rings
	615	+ belonging to different queue ids or netdevs bound with the
	616	+ XDP_SHARED_UMEM flag.
	617	+
570	618	Credits
571	619	=======
572	620

arch/x86/net/bpf_jit_comp.c+21−11

@@ -1379,10 +1379,15 @@ static int invoke_bpf_prog(const struct btf_func_model m, u8 *pprog, · 1378 unmodified lines
1379	1379	u8 prog = pprog;
1380	1380	int cnt = 0;
1381	1381
1382		− if (emit_call(&prog, __bpf_prog_enter, prog))
1383		− return -EINVAL;
1384		− /* remember prog start time returned by __bpf_prog_enter */
1385		− emit_mov_reg(&prog, true, BPF_REG_6, BPF_REG_0);
	1382	+ if (p->aux->sleepable) {
	1383	+ if (emit_call(&prog, __bpf_prog_enter_sleepable, prog))
	1384	+ return -EINVAL;
	1385	+ } else {
	1386	+ if (emit_call(&prog, __bpf_prog_enter, prog))
	1387	+ return -EINVAL;
	1388	+ /* remember prog start time returned by __bpf_prog_enter */
	1389	+ emit_mov_reg(&prog, true, BPF_REG_6, BPF_REG_0);
	1390	+ }
1386	1391
1387	1392	/* arg1: lea rdi, [rbp - stack_size] */
1388	1393	EMIT4(0x48, 0x8D, 0x7D, -stack_size);
@@ -1402,13 +1407,18 @@ static int invoke_bpf_prog(const struct btf_func_model m, u8 *pprog, · 13 unmodified lines
1402	1407	if (mod_ret)
1403	1408	emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
1404	1409
1405		− /* arg1: mov rdi, progs[i] */
1406		− emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32,
1407		− (u32) (long) p);
1408		− /* arg2: mov rsi, rbx <- start time in nsec */
1409		− emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6);
1410		− if (emit_call(&prog, __bpf_prog_exit, prog))
1411		− return -EINVAL;
	1410	+ if (p->aux->sleepable) {
	1411	+ if (emit_call(&prog, __bpf_prog_exit_sleepable, prog))
	1412	+ return -EINVAL;
	1413	+ } else {
	1414	+ /* arg1: mov rdi, progs[i] */
	1415	+ emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32,
	1416	+ (u32) (long) p);
	1417	+ /* arg2: mov rsi, rbx <- start time in nsec */
	1418	+ emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6);
	1419	+ if (emit_call(&prog, __bpf_prog_exit, prog))
	1420	+ return -EINVAL;
	1421	+ }
1412	1422
1413	1423	*pprog = prog;
1414	1424	return 0;

drivers/net/ethernet/intel/i40e/i40e_ethtool.c+1−1

@@ -1967,7 +1967,7 @@ static int i40e_set_ringparam(struct net_device *netdev, · 1966 unmodified lines
1967	1967	(new_rx_count == vsi->rx_rings[0]->count))
1968	1968	return 0;
1969	1969
1970		− /* If there is a AF_XDP UMEM attached to any of Rx rings,
	1970	+ /* If there is a AF_XDP page pool attached to any of Rx rings,
1971	1971	* disallow changing the number of descriptors -- regardless
1972	1972	* if the netdev is running or not.
1973	1973	*/

drivers/net/ethernet/intel/i40e/i40e_main.c+15−14

@@ -3122,12 +3122,12 @@ static void i40e_config_xps_tx_ring(struct i40e_ring *ring) · 3121 unmodified lines
3122	3122	}
3123	3123
3124	3124	/**
3125		− * i40e_xsk_umem - Retrieve the AF_XDP ZC if XDP and ZC is enabled
	3125	+ * i40e_xsk_pool - Retrieve the AF_XDP buffer pool if XDP and ZC is enabled
3126	3126	* @ring: The Tx or Rx ring
3127	3127	*
3128		− * Returns the UMEM or NULL.
	3128	+ * Returns the AF_XDP buffer pool or NULL.
3129	3129	**/
3130		−static struct xdp_umem i40e_xsk_umem(struct i40e_ring ring)
	3130	+static struct xsk_buff_pool i40e_xsk_pool(struct i40e_ring ring)
3131	3131	{
3132	3132	bool xdp_on = i40e_enabled_xdp_vsi(ring->vsi);
3133	3133	int qid = ring->queue_index;
@@ -3138,7 +3138,7 @@ static struct xdp_umem i40e_xsk_umem(struct i40e_ring ring) · 4 unmodified lines
3138	3138	if (!xdp_on \|\| !test_bit(qid, ring->vsi->af_xdp_zc_qps))
3139	3139	return NULL;
3140	3140
3141		− return xdp_get_umem_from_qid(ring->vsi->netdev, qid);
	3141	+ return xsk_get_pool_from_qid(ring->vsi->netdev, qid);
3142	3142	}
3143	3143
3144	3144	/**
@@ -3157,7 +3157,7 @@ static int i40e_configure_tx_ring(struct i40e_ring *ring) · 12 unmodified lines
3157	3157	u32 qtx_ctl = 0;
3158	3158
3159	3159	if (ring_is_xdp(ring))
3160		− ring->xsk_umem = i40e_xsk_umem(ring);
	3160	+ ring->xsk_pool = i40e_xsk_pool(ring);
3161	3161
3162	3162	/* some ATR related tx ring init */
3163	3163	if (vsi->back->flags & I40E_FLAG_FD_ATR_ENABLED) {
@@ -3280,12 +3280,13 @@ static int i40e_configure_rx_ring(struct i40e_ring *ring) · 116 unmodified lines
3280	3280	xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
3281	3281
3282	3282	kfree(ring->rx_bi);
3283		− ring->xsk_umem = i40e_xsk_umem(ring);
3284		− if (ring->xsk_umem) {
	3283	+ ring->xsk_pool = i40e_xsk_pool(ring);
	3284	+ if (ring->xsk_pool) {
3285	3285	ret = i40e_alloc_rx_bi_zc(ring);
3286	3286	if (ret)
3287	3287	return ret;
3288		− ring->rx_buf_len = xsk_umem_get_rx_frame_size(ring->xsk_umem);
	3288	+ ring->rx_buf_len =
	3289	+ xsk_pool_get_rx_frame_size(ring->xsk_pool);
3289	3290	/* For AF_XDP ZC, we disallow packets to span on
3290	3291	* multiple buffers, thus letting us skip that
3291	3292	* handling in the fast-path.
@@ -3368,8 +3369,8 @@ static int i40e_configure_rx_ring(struct i40e_ring *ring) · 76 unmodified lines
3368	3369	ring->tail = hw->hw_addr + I40E_QRX_TAIL(pf_q);
3369	3370	writel(0, ring->tail);
3370	3371
3371		− if (ring->xsk_umem) {
3372		− xsk_buff_set_rxq_info(ring->xsk_umem, &ring->xdp_rxq);
	3372	+ if (ring->xsk_pool) {
	3373	+ xsk_pool_set_rxq_info(ring->xsk_pool, &ring->xdp_rxq);
3373	3374	ok = i40e_alloc_rx_buffers_zc(ring, I40E_DESC_UNUSED(ring));
3374	3375	} else {
3375	3376	ok = !i40e_alloc_rx_buffers(ring, I40E_DESC_UNUSED(ring));
@@ -3380,7 +3381,7 @@ static int i40e_configure_rx_ring(struct i40e_ring *ring) · 4 unmodified lines
3380	3381	*/
3381	3382	dev_info(&vsi->back->pdev->dev,
3382	3383	"Failed to allocate some buffers on %sRx ring %d (pf_q %d)\n",
3383		− ring->xsk_umem ? "UMEM enabled " : "",
	3384	+ ring->xsk_pool ? "AF_XDP ZC enabled " : "",
3384	3385	ring->queue_index, pf_q);
3385	3386	}
3386	3387
@@ -12644,7 +12645,7 @@ static int i40e_xdp_setup(struct i40e_vsi *vsi, · 9257 unmodified lines
12644	12645	*/
12645	12646	if (need_reset && prog)
12646	12647	for (i = 0; i < vsi->num_queue_pairs; i++)
12647		− if (vsi->xdp_rings[i]->xsk_umem)
	12648	+ if (vsi->xdp_rings[i]->xsk_pool)
12648	12649	(void)i40e_xsk_wakeup(vsi->netdev, i,
12649	12650	XDP_WAKEUP_RX);
12650	12651
@@ -12923,8 +12924,8 @@ static int i40e_xdp(struct net_device *dev, · 272 unmodified lines
12923	12924	switch (xdp->command) {
12924	12925	case XDP_SETUP_PROG:
12925	12926	return i40e_xdp_setup(vsi, xdp->prog);
12926		− case XDP_SETUP_XSK_UMEM:
12927		− return i40e_xsk_umem_setup(vsi, xdp->xsk.umem,
	12927	+ case XDP_SETUP_XSK_POOL:
	12928	+ return i40e_xsk_pool_setup(vsi, xdp->xsk.pool,
12928	12929	xdp->xsk.queue_id);
12929	12930	default:
12930	12931	return -EINVAL;

drivers/net/ethernet/intel/i40e/i40e_txrx.c+5−5

@@ -636,7 +636,7 @@ void i40e_clean_tx_ring(struct i40e_ring *tx_ring) · 635 unmodified lines
636	636	unsigned long bi_size;
637	637	u16 i;
638	638
639		− if (ring_is_xdp(tx_ring) && tx_ring->xsk_umem) {
	639	+ if (ring_is_xdp(tx_ring) && tx_ring->xsk_pool) {
640	640	i40e_xsk_clean_tx_ring(tx_ring);
641	641	} else {
642	642	/* ring already cleared, nothing to do */
@@ -1335,7 +1335,7 @@ void i40e_clean_rx_ring(struct i40e_ring *rx_ring) · 692 unmodified lines
1335	1335	rx_ring->skb = NULL;
1336	1336	}
1337	1337
1338		− if (rx_ring->xsk_umem) {
	1338	+ if (rx_ring->xsk_pool) {
1339	1339	i40e_xsk_clean_rx_ring(rx_ring);
1340	1340	goto skip_free;
1341	1341	}
@@ -1369,7 +1369,7 @@ void i40e_clean_rx_ring(struct i40e_ring *rx_ring) · 27 unmodified lines
1369	1369	}
1370	1370
1371	1371	skip_free:
1372		− if (rx_ring->xsk_umem)
	1372	+ if (rx_ring->xsk_pool)
1373	1373	i40e_clear_rx_bi_zc(rx_ring);
1374	1374	else
1375	1375	i40e_clear_rx_bi(rx_ring);
@@ -2575,7 +2575,7 @@ int i40e_napi_poll(struct napi_struct *napi, int budget) · 1199 unmodified lines
2575	2575	* budget and be more aggressive about cleaning up the Tx descriptors.
2576	2576	*/
2577	2577	i40e_for_each_ring(ring, q_vector->tx) {
2578		− bool wd = ring->xsk_umem ?
	2578	+ bool wd = ring->xsk_pool ?
2579	2579	i40e_clean_xdp_tx_irq(vsi, ring) :
2580	2580	i40e_clean_tx_irq(vsi, ring, budget);
2581	2581
@@ -2603,7 +2603,7 @@ int i40e_napi_poll(struct napi_struct *napi, int budget) · 21 unmodified lines
2603	2603	budget_per_ring = budget;
2604	2604
2605	2605	i40e_for_each_ring(ring, q_vector->rx) {
2606		− int cleaned = ring->xsk_umem ?
	2606	+ int cleaned = ring->xsk_pool ?
2607	2607	i40e_clean_rx_irq_zc(ring, budget_per_ring) :
2608	2608	i40e_clean_rx_irq(ring, budget_per_ring);
2609	2609

drivers/net/ethernet/intel/i40e/i40e_txrx.h+1−1

@@ -388,7 +388,7 @@ struct i40e_ring { · 387 unmodified lines
388	388
389	389	struct i40e_channel *ch;
390	390	struct xdp_rxq_info xdp_rxq;
391		− struct xdp_umem *xsk_umem;
	391	+ struct xsk_buff_pool *xsk_pool;
392	392	} ____cacheline_internodealigned_in_smp;
393	393
394	394	static inline bool ring_uses_build_skb(struct i40e_ring *ring)

drivers/net/ethernet/intel/i40e/i40e_xsk.c+43−38

@@ -29,14 +29,16 @@ static struct xdp_buff *i40e_rx_bi(struct i40e_ring rx_ring, u32 idx) · 28 unmodified lines
29	29	}
30	30
31	31	/**
32		− * i40e_xsk_umem_enable - Enable/associate a UMEM to a certain ring/qid
	32	+ * i40e_xsk_pool_enable - Enable/associate an AF_XDP buffer pool to a
	33	+ * certain ring/qid
33	34	* @vsi: Current VSI
34		− * @umem: UMEM
35		− * @qid: Rx ring to associate UMEM to
	35	+ * @pool: buffer pool
	36	+ * @qid: Rx ring to associate buffer pool with
36	37	*
37	38	* Returns 0 on success, <0 on failure
38	39	**/
39		−static int i40e_xsk_umem_enable(struct i40e_vsi vsi, struct xdp_umem umem,
	40	+static int i40e_xsk_pool_enable(struct i40e_vsi *vsi,
	41	+ struct xsk_buff_pool *pool,
40	42	u16 qid)
41	43	{
42	44	struct net_device *netdev = vsi->netdev;
@@ -53,7 +55,7 @@ static int i40e_xsk_umem_enable(struct i40e_vsi vsi, struct xdp_umem umem, · 10 unmodified lines
53	55	qid >= netdev->real_num_tx_queues)
54	56	return -EINVAL;
55	57
56		− err = xsk_buff_dma_map(umem, &vsi->back->pdev->dev, I40E_RX_DMA_ATTR);
	58	+ err = xsk_pool_dma_map(pool, &vsi->back->pdev->dev, I40E_RX_DMA_ATTR);
57	59	if (err)
58	60	return err;
59	61
@@ -80,21 +82,22 @@ static int i40e_xsk_umem_enable(struct i40e_vsi vsi, struct xdp_umem umem, · 20 unmodified lines
80	82	}
81	83
82	84	/**
83		− * i40e_xsk_umem_disable - Disassociate a UMEM from a certain ring/qid
	85	+ * i40e_xsk_pool_disable - Disassociate an AF_XDP buffer pool from a
	86	+ * certain ring/qid
84	87	* @vsi: Current VSI
85		− * @qid: Rx ring to associate UMEM to
	88	+ * @qid: Rx ring to associate buffer pool with
86	89	*
87	90	* Returns 0 on success, <0 on failure
88	91	**/
89		−static int i40e_xsk_umem_disable(struct i40e_vsi *vsi, u16 qid)
	92	+static int i40e_xsk_pool_disable(struct i40e_vsi *vsi, u16 qid)
90	93	{
91	94	struct net_device *netdev = vsi->netdev;
92		− struct xdp_umem *umem;
	95	+ struct xsk_buff_pool *pool;
93	96	bool if_running;
94	97	int err;
95	98
96		− umem = xdp_get_umem_from_qid(netdev, qid);
97		− if (!umem)
	99	+ pool = xsk_get_pool_from_qid(netdev, qid);
	100	+ if (!pool)
98	101	return -EINVAL;
99	102
100	103	if_running = netif_running(vsi->netdev) && i40e_enabled_xdp_vsi(vsi);
@@ -106,7 +109,7 @@ static int i40e_xsk_umem_disable(struct i40e_vsi *vsi, u16 qid) · 5 unmodified lines
106	109	}
107	110
108	111	clear_bit(qid, vsi->af_xdp_zc_qps);
109		− xsk_buff_dma_unmap(umem, I40E_RX_DMA_ATTR);
	112	+ xsk_pool_dma_unmap(pool, I40E_RX_DMA_ATTR);
110	113
111	114	if (if_running) {
112	115	err = i40e_queue_pair_enable(vsi, qid);
@@ -118,20 +121,21 @@ static int i40e_xsk_umem_disable(struct i40e_vsi *vsi, u16 qid) · 5 unmodified lines
118	121	}
119	122
120	123	/**
121		− * i40e_xsk_umem_setup - Enable/disassociate a UMEM to/from a ring/qid
	124	+ * i40e_xsk_pool_setup - Enable/disassociate an AF_XDP buffer pool to/from
	125	+ * a ring/qid
122	126	* @vsi: Current VSI
123		− * @umem: UMEM to enable/associate to a ring, or NULL to disable
124		− * @qid: Rx ring to (dis)associate UMEM (from)to
	127	+ * @pool: Buffer pool to enable/associate to a ring, or NULL to disable
	128	+ * @qid: Rx ring to (dis)associate buffer pool (from)to
125	129	*
126		− * This function enables or disables a UMEM to a certain ring.
	130	+ * This function enables or disables a buffer pool to a certain ring.
127	131	*
128	132	* Returns 0 on success, <0 on failure
129	133	**/
130		−int i40e_xsk_umem_setup(struct i40e_vsi vsi, struct xdp_umem umem,
	134	+int i40e_xsk_pool_setup(struct i40e_vsi vsi, struct xsk_buff_pool pool,
131	135	u16 qid)
132	136	{
133		− return umem ? i40e_xsk_umem_enable(vsi, umem, qid) :
134		− i40e_xsk_umem_disable(vsi, qid);
	137	+ return pool ? i40e_xsk_pool_enable(vsi, pool, qid) :
	138	+ i40e_xsk_pool_disable(vsi, qid);
135	139	}
136	140
137	141	/**
@@ -191,7 +195,7 @@ bool i40e_alloc_rx_buffers_zc(struct i40e_ring *rx_ring, u16 count) · 53 unmodified lines
191	195	rx_desc = I40E_RX_DESC(rx_ring, ntu);
192	196	bi = i40e_rx_bi(rx_ring, ntu);
193	197	do {
194		− xdp = xsk_buff_alloc(rx_ring->xsk_umem);
	198	+ xdp = xsk_buff_alloc(rx_ring->xsk_pool);
195	199	if (!xdp) {
196	200	ok = false;
197	201	goto no_buffers;
@@ -310,7 +314,7 @@ int i40e_clean_rx_irq_zc(struct i40e_ring *rx_ring, int budget) · 112 unmodified lines
310	314
311	315	bi = i40e_rx_bi(rx_ring, rx_ring->next_to_clean);
312	316	(bi)->data_end = (bi)->data + size;
313		− xsk_buff_dma_sync_for_cpu(*bi);
	317	+ xsk_buff_dma_sync_for_cpu(*bi, rx_ring->xsk_pool);
314	318
315	319	xdp_res = i40e_run_xdp_zc(rx_ring, *bi);
316	320	if (xdp_res) {
@@ -358,11 +362,11 @@ int i40e_clean_rx_irq_zc(struct i40e_ring *rx_ring, int budget) · 41 unmodified lines
358	362	i40e_finalize_xdp_rx(rx_ring, xdp_xmit);
359	363	i40e_update_rx_stats(rx_ring, total_rx_bytes, total_rx_packets);
360	364
361		− if (xsk_umem_uses_need_wakeup(rx_ring->xsk_umem)) {
	365	+ if (xsk_uses_need_wakeup(rx_ring->xsk_pool)) {
362	366	if (failure \|\| rx_ring->next_to_clean == rx_ring->next_to_use)
363		− xsk_set_rx_need_wakeup(rx_ring->xsk_umem);
	367	+ xsk_set_rx_need_wakeup(rx_ring->xsk_pool);
364	368	else
365		− xsk_clear_rx_need_wakeup(rx_ring->xsk_umem);
	369	+ xsk_clear_rx_need_wakeup(rx_ring->xsk_pool);
366	370
367	371	return (int)total_rx_packets;
368	372	}
@@ -385,11 +389,11 @@ static bool i40e_xmit_zc(struct i40e_ring *xdp_ring, unsigned int budget) · 16 unmodified lines
385	389	dma_addr_t dma;
386	390
387	391	while (budget-- > 0) {
388		− if (!xsk_umem_consume_tx(xdp_ring->xsk_umem, &desc))
	392	+ if (!xsk_tx_peek_desc(xdp_ring->xsk_pool, &desc))
389	393	break;
390	394
391		− dma = xsk_buff_raw_get_dma(xdp_ring->xsk_umem, desc.addr);
392		− xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_umem, dma,
	395	+ dma = xsk_buff_raw_get_dma(xdp_ring->xsk_pool, desc.addr);
	396	+ xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_pool, dma,
393	397	desc.len);
394	398
395	399	tx_bi = &xdp_ring->tx_bi[xdp_ring->next_to_use];
@@ -416,7 +420,7 @@ static bool i40e_xmit_zc(struct i40e_ring *xdp_ring, unsigned int budget) · 20 unmodified lines
416	420	I40E_TXD_QW1_CMD_SHIFT);
417	421	i40e_xdp_ring_update_tail(xdp_ring);
418	422
419		− xsk_umem_consume_tx_done(xdp_ring->xsk_umem);
	423	+ xsk_tx_release(xdp_ring->xsk_pool);
420	424	i40e_update_tx_stats(xdp_ring, sent_frames, total_bytes);
421	425	}
422	426
@@ -448,7 +452,7 @@ static void i40e_clean_xdp_tx_buffer(struct i40e_ring *tx_ring, · 25 unmodified lines
448	452	**/
449	453	bool i40e_clean_xdp_tx_irq(struct i40e_vsi vsi, struct i40e_ring tx_ring)
450	454	{
451		− struct xdp_umem *umem = tx_ring->xsk_umem;
	455	+ struct xsk_buff_pool *bp = tx_ring->xsk_pool;
452	456	u32 i, completed_frames, xsk_frames = 0;
453	457	u32 head_idx = i40e_get_head(tx_ring);
454	458	struct i40e_tx_buffer *tx_bi;
@@ -488,13 +492,13 @@ skip: · 33 unmodified lines
488	492	tx_ring->next_to_clean -= tx_ring->count;
489	493
490	494	if (xsk_frames)
491		− xsk_umem_complete_tx(umem, xsk_frames);
	495	+ xsk_tx_completed(bp, xsk_frames);
492	496
493	497	i40e_arm_wb(tx_ring, vsi, completed_frames);
494	498
495	499	out_xmit:
496		− if (xsk_umem_uses_need_wakeup(tx_ring->xsk_umem))
497		− xsk_set_tx_need_wakeup(tx_ring->xsk_umem);
	500	+ if (xsk_uses_need_wakeup(tx_ring->xsk_pool))
	501	+ xsk_set_tx_need_wakeup(tx_ring->xsk_pool);
498	502
499	503	return i40e_xmit_zc(tx_ring, I40E_DESC_UNUSED(tx_ring));
500	504	}
@@ -526,7 +530,7 @@ int i40e_xsk_wakeup(struct net_device *dev, u32 queue_id, u32 flags) · 25 unmodified lines
526	530	if (queue_id >= vsi->num_queue_pairs)
527	531	return -ENXIO;
528	532
529		− if (!vsi->xdp_rings[queue_id]->xsk_umem)
	533	+ if (!vsi->xdp_rings[queue_id]->xsk_pool)
530	534	return -ENXIO;
531	535
532	536	ring = vsi->xdp_rings[queue_id];
@@ -565,7 +569,7 @@ void i40e_xsk_clean_rx_ring(struct i40e_ring *rx_ring) · 32 unmodified lines
565	569	void i40e_xsk_clean_tx_ring(struct i40e_ring *tx_ring)
566	570	{
567	571	u16 ntc = tx_ring->next_to_clean, ntu = tx_ring->next_to_use;
568		− struct xdp_umem *umem = tx_ring->xsk_umem;
	572	+ struct xsk_buff_pool *bp = tx_ring->xsk_pool;
569	573	struct i40e_tx_buffer *tx_bi;
570	574	u32 xsk_frames = 0;
571	575
@@ -585,14 +589,15 @@ void i40e_xsk_clean_tx_ring(struct i40e_ring *tx_ring) · 13 unmodified lines
585	589	}
586	590
587	591	if (xsk_frames)
588		− xsk_umem_complete_tx(umem, xsk_frames);
	592	+ xsk_tx_completed(bp, xsk_frames);
589	593	}
590	594
591	595	/**
592		− * i40e_xsk_any_rx_ring_enabled - Checks if Rx rings have AF_XDP UMEM attached
	596	+ * i40e_xsk_any_rx_ring_enabled - Checks if Rx rings have an AF_XDP
	597	+ * buffer pool attached
593	598	* @vsi: vsi
594	599	*
595		− * Returns true if any of the Rx rings has an AF_XDP UMEM attached
	600	+ * Returns true if any of the Rx rings has an AF_XDP buffer pool attached
596	601	**/
597	602	bool i40e_xsk_any_rx_ring_enabled(struct i40e_vsi *vsi)
598	603	{
@@ -600,7 +605,7 @@ bool i40e_xsk_any_rx_ring_enabled(struct i40e_vsi *vsi) · 1 unmodified line
600	605	int i;
601	606
602	607	for (i = 0; i < vsi->num_queue_pairs; i++) {
603		− if (xdp_get_umem_from_qid(netdev, i))
	608	+ if (xsk_get_pool_from_qid(netdev, i))
604	609	return true;
605	610	}
606	611

drivers/net/ethernet/intel/i40e/i40e_xsk.h+2−2

@@ -5,12 +5,12 @@ · 4 unmodified lines
5	5	#define _I40E_XSK_H_
6	6
7	7	struct i40e_vsi;
8		−struct xdp_umem;
	8	+struct xsk_buff_pool;
9	9	struct zero_copy_allocator;
10	10
11	11	int i40e_queue_pair_disable(struct i40e_vsi *vsi, int queue_pair);
12	12	int i40e_queue_pair_enable(struct i40e_vsi *vsi, int queue_pair);
13		−int i40e_xsk_umem_setup(struct i40e_vsi vsi, struct xdp_umem umem,
	13	+int i40e_xsk_pool_setup(struct i40e_vsi vsi, struct xsk_buff_pool pool,
14	14	u16 qid);
15	15	bool i40e_alloc_rx_buffers_zc(struct i40e_ring *rx_ring, u16 cleaned_count);
16	16	int i40e_clean_rx_irq_zc(struct i40e_ring *rx_ring, int budget);

drivers/net/ethernet/intel/ice/ice.h+9−9

@@ -321,9 +321,9 @@ struct ice_vsi { · 320 unmodified lines
321	321	struct ice_ring *xdp_rings; / XDP ring array */
322	322	u16 num_xdp_txq; /* Used XDP queues */
323	323	u8 xdp_mapping_mode; /* ICE_MAP_MODE_[CONTIG\|SCATTER] */
324		− struct xdp_umem **xsk_umems;
325		− u16 num_xsk_umems_used;
326		− u16 num_xsk_umems;
	324	+ struct xsk_buff_pool **xsk_pools;
	325	+ u16 num_xsk_pools_used;
	326	+ u16 num_xsk_pools;
327	327	} ____cacheline_internodealigned_in_smp;
328	328
329	329	/* struct that defines an interrupt vector */
@@ -507,25 +507,25 @@ static inline void ice_set_ring_xdp(struct ice_ring *ring) · 177 unmodified lines
507	507	}
508	508
509	509	/**
510		− * ice_xsk_umem - get XDP UMEM bound to a ring
	510	+ * ice_xsk_pool - get XSK buffer pool bound to a ring
511	511	* @ring - ring to use
512	512	*
513		− * Returns a pointer to xdp_umem structure if there is an UMEM present,
	513	+ * Returns a pointer to xdp_umem structure if there is a buffer pool present,
514	514	* NULL otherwise.
515	515	*/
516		−static inline struct xdp_umem ice_xsk_umem(struct ice_ring ring)
	516	+static inline struct xsk_buff_pool ice_xsk_pool(struct ice_ring ring)
517	517	{
518		− struct xdp_umem **umems = ring->vsi->xsk_umems;
	518	+ struct xsk_buff_pool **pools = ring->vsi->xsk_pools;
519	519	u16 qid = ring->q_index;
520	520
521	521	if (ice_ring_is_xdp(ring))
522	522	qid -= ring->vsi->num_xdp_txq;
523	523
524		− if (qid >= ring->vsi->num_xsk_umems \|\| !umems \|\| !umems[qid] \|\|
	524	+ if (qid >= ring->vsi->num_xsk_pools \|\| !pools \|\| !pools[qid] \|\|
525	525	!ice_is_xdp_ena_vsi(ring->vsi))
526	526	return NULL;
527	527
528		− return umems[qid];
	528	+ return pools[qid];
529	529	}
530	530
531	531	/**

drivers/net/ethernet/intel/ice/ice_base.c+8−8

@@ -308,12 +308,12 @@ int ice_setup_rx_ctx(struct ice_ring *ring) · 307 unmodified lines
308	308	xdp_rxq_info_reg(&ring->xdp_rxq, ring->netdev,
309	309	ring->q_index);
310	310
311		− ring->xsk_umem = ice_xsk_umem(ring);
312		− if (ring->xsk_umem) {
	311	+ ring->xsk_pool = ice_xsk_pool(ring);
	312	+ if (ring->xsk_pool) {
313	313	xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
314	314
315	315	ring->rx_buf_len =
316		− xsk_umem_get_rx_frame_size(ring->xsk_umem);
	316	+ xsk_pool_get_rx_frame_size(ring->xsk_pool);
317	317	/* For AF_XDP ZC, we disallow packets to span on
318	318	* multiple buffers, thus letting us skip that
319	319	* handling in the fast-path.
@@ -324,7 +324,7 @@ int ice_setup_rx_ctx(struct ice_ring *ring) · 4 unmodified lines
324	324	NULL);
325	325	if (err)
326	326	return err;
327		− xsk_buff_set_rxq_info(ring->xsk_umem, &ring->xdp_rxq);
	327	+ xsk_pool_set_rxq_info(ring->xsk_pool, &ring->xdp_rxq);
328	328
329	329	dev_info(dev, "Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring %d\n",
330	330	ring->q_index);
@@ -417,9 +417,9 @@ int ice_setup_rx_ctx(struct ice_ring *ring) · 86 unmodified lines
417	417	ring->tail = hw->hw_addr + QRX_TAIL(pf_q);
418	418	writel(0, ring->tail);
419	419
420		− if (ring->xsk_umem) {
421		− if (!xsk_buff_can_alloc(ring->xsk_umem, num_bufs)) {
422		− dev_warn(dev, "UMEM does not provide enough addresses to fill %d buffers on Rx ring %d\n",
	420	+ if (ring->xsk_pool) {
	421	+ if (!xsk_buff_can_alloc(ring->xsk_pool, num_bufs)) {
	422	+ dev_warn(dev, "XSK buffer pool does not provide enough addresses to fill %d buffers on Rx ring %d\n",
423	423	num_bufs, ring->q_index);
424	424	dev_warn(dev, "Change Rx ring/fill queue size to avoid performance issues\n");
425	425
@@ -428,7 +428,7 @@ int ice_setup_rx_ctx(struct ice_ring *ring) · 2 unmodified lines
428	428
429	429	err = ice_alloc_rx_bufs_zc(ring, num_bufs);
430	430	if (err)
431		− dev_info(dev, "Failed to allocate some buffers on UMEM enabled Rx ring %d (pf_q %d)\n",
	431	+ dev_info(dev, "Failed to allocate some buffers on XSK buffer pool enabled Rx ring %d (pf_q %d)\n",
432	432	ring->q_index, pf_q);
433	433	return 0;
434	434	}

drivers/net/ethernet/intel/ice/ice_lib.c+1−1

@@ -1743,7 +1743,7 @@ int ice_vsi_cfg_xdp_txqs(struct ice_vsi *vsi) · 1742 unmodified lines
1743	1743	return ret;
1744	1744
1745	1745	for (i = 0; i < vsi->num_xdp_txq; i++)
1746		− vsi->xdp_rings[i]->xsk_umem = ice_xsk_umem(vsi->xdp_rings[i]);
	1746	+ vsi->xdp_rings[i]->xsk_pool = ice_xsk_pool(vsi->xdp_rings[i]);
1747	1747
1748	1748	return ret;
1749	1749	}

drivers/net/ethernet/intel/ice/ice_main.c+5−5

@@ -2273,7 +2273,7 @@ static int ice_xdp_alloc_setup_rings(struct ice_vsi *vsi) · 2272 unmodified lines
2273	2273	if (ice_setup_tx_ring(xdp_ring))
2274	2274	goto free_xdp_rings;
2275	2275	ice_set_ring_xdp(xdp_ring);
2276		− xdp_ring->xsk_umem = ice_xsk_umem(xdp_ring);
	2276	+ xdp_ring->xsk_pool = ice_xsk_pool(xdp_ring);
2277	2277	}
2278	2278
2279	2279	return 0;
@@ -2517,13 +2517,13 @@ ice_xdp_setup_prog(struct ice_vsi vsi, struct bpf_prog prog, · 237 unmodified lines
2517	2517	if (if_running)
2518	2518	ret = ice_up(vsi);
2519	2519
2520		− if (!ret && prog && vsi->xsk_umems) {
	2520	+ if (!ret && prog && vsi->xsk_pools) {
2521	2521	int i;
2522	2522
2523	2523	ice_for_each_rxq(vsi, i) {
2524	2524	struct ice_ring *rx_ring = vsi->rx_rings[i];
2525	2525
2526		− if (rx_ring->xsk_umem)
	2526	+ if (rx_ring->xsk_pool)
2527	2527	napi_schedule(&rx_ring->q_vector->napi);
2528	2528	}
2529	2529	}
@@ -2549,8 +2549,8 @@ static int ice_xdp(struct net_device dev, struct netdev_bpf xdp) · 19 unmodified lines
2549	2549	switch (xdp->command) {
2550	2550	case XDP_SETUP_PROG:
2551	2551	return ice_xdp_setup_prog(vsi, xdp->prog, xdp->extack);
2552		− case XDP_SETUP_XSK_UMEM:
2553		− return ice_xsk_umem_setup(vsi, xdp->xsk.umem,
	2552	+ case XDP_SETUP_XSK_POOL:
	2553	+ return ice_xsk_pool_setup(vsi, xdp->xsk.pool,
2554	2554	xdp->xsk.queue_id);
2555	2555	default:
2556	2556	return -EINVAL;

drivers/net/ethernet/intel/ice/ice_txrx.c+4−4

@@ -145,7 +145,7 @@ void ice_clean_tx_ring(struct ice_ring *tx_ring) · 144 unmodified lines
145	145	{
146	146	u16 i;
147	147
148		− if (ice_ring_is_xdp(tx_ring) && tx_ring->xsk_umem) {
	148	+ if (ice_ring_is_xdp(tx_ring) && tx_ring->xsk_pool) {
149	149	ice_xsk_clean_xdp_ring(tx_ring);
150	150	goto tx_skip_free;
151	151	}
@@ -375,7 +375,7 @@ void ice_clean_rx_ring(struct ice_ring *rx_ring) · 223 unmodified lines
375	375	if (!rx_ring->rx_buf)
376	376	return;
377	377
378		− if (rx_ring->xsk_umem) {
	378	+ if (rx_ring->xsk_pool) {
379	379	ice_xsk_clean_rx_ring(rx_ring);
380	380	goto rx_skip_free;
381	381	}
@@ -1610,7 +1610,7 @@ int ice_napi_poll(struct napi_struct *napi, int budget) · 1228 unmodified lines
1610	1610	* budget and be more aggressive about cleaning up the Tx descriptors.
1611	1611	*/
1612	1612	ice_for_each_ring(ring, q_vector->tx) {
1613		− bool wd = ring->xsk_umem ?
	1613	+ bool wd = ring->xsk_pool ?
1614	1614	ice_clean_tx_irq_zc(ring, budget) :
1615	1615	ice_clean_tx_irq(ring, budget);
1616	1616
@@ -1640,7 +1640,7 @@ int ice_napi_poll(struct napi_struct *napi, int budget) · 23 unmodified lines
1640	1640	* comparison in the irq context instead of many inside the
1641	1641	* ice_clean_rx_irq function and makes the codebase cleaner.
1642	1642	*/
1643		− cleaned = ring->xsk_umem ?
	1643	+ cleaned = ring->xsk_pool ?
1644	1644	ice_clean_rx_irq_zc(ring, budget_per_ring) :
1645	1645	ice_clean_rx_irq(ring, budget_per_ring);
1646	1646	work_done += cleaned;

drivers/net/ethernet/intel/ice/ice_txrx.h+1−1

@@ -295,7 +295,7 @@ struct ice_ring { · 294 unmodified lines
295	295
296	296	struct rcu_head rcu; /* to avoid race on free */
297	297	struct bpf_prog *xdp_prog;
298		− struct xdp_umem *xsk_umem;
	298	+ struct xsk_buff_pool *xsk_pool;
299	299	/* CL3 - 3rd cacheline starts here */
300	300	struct xdp_rxq_info xdp_rxq;
301	301	/* CLX - the below items are only accessed infrequently and should be

drivers/net/ethernet/intel/ice/ice_xsk.c+69−69

@@ -236,7 +236,7 @@ static int ice_qp_ena(struct ice_vsi *vsi, u16 q_idx) · 235 unmodified lines
236	236	if (err)
237	237	goto free_buf;
238	238	ice_set_ring_xdp(xdp_ring);
239		− xdp_ring->xsk_umem = ice_xsk_umem(xdp_ring);
	239	+ xdp_ring->xsk_pool = ice_xsk_pool(xdp_ring);
240	240	}
241	241
242	242	err = ice_setup_rx_ctx(rx_ring);
@@ -260,21 +260,21 @@ free_buf: · 17 unmodified lines
260	260	}
261	261
262	262	/**
263		− * ice_xsk_alloc_umems - allocate a UMEM region for an XDP socket
264		− * @vsi: VSI to allocate the UMEM on
	263	+ * ice_xsk_alloc_pools - allocate a buffer pool for an XDP socket
	264	+ * @vsi: VSI to allocate the buffer pool on
265	265	*
266	266	* Returns 0 on success, negative on error
267	267	*/
268		−static int ice_xsk_alloc_umems(struct ice_vsi *vsi)
	268	+static int ice_xsk_alloc_pools(struct ice_vsi *vsi)
269	269	{
270		− if (vsi->xsk_umems)
	270	+ if (vsi->xsk_pools)
271	271	return 0;
272	272
273		− vsi->xsk_umems = kcalloc(vsi->num_xsk_umems, sizeof(*vsi->xsk_umems),
	273	+ vsi->xsk_pools = kcalloc(vsi->num_xsk_pools, sizeof(*vsi->xsk_pools),
274	274	GFP_KERNEL);
275	275
276		− if (!vsi->xsk_umems) {
277		− vsi->num_xsk_umems = 0;
	276	+ if (!vsi->xsk_pools) {
	277	+ vsi->num_xsk_pools = 0;
278	278	return -ENOMEM;
279	279	}
280	280
@@ -282,73 +282,73 @@ static int ice_xsk_alloc_umems(struct ice_vsi *vsi) · 1 unmodified line
282	282	}
283	283
284	284	/**
285		− * ice_xsk_remove_umem - Remove an UMEM for a certain ring/qid
	285	+ * ice_xsk_remove_pool - Remove an buffer pool for a certain ring/qid
286	286	* @vsi: VSI from which the VSI will be removed
287		− * @qid: Ring/qid associated with the UMEM
	287	+ * @qid: Ring/qid associated with the buffer pool
288	288	*/
289		−static void ice_xsk_remove_umem(struct ice_vsi *vsi, u16 qid)
	289	+static void ice_xsk_remove_pool(struct ice_vsi *vsi, u16 qid)
290	290	{
291		− vsi->xsk_umems[qid] = NULL;
292		− vsi->num_xsk_umems_used--;
	291	+ vsi->xsk_pools[qid] = NULL;
	292	+ vsi->num_xsk_pools_used--;
293	293
294		− if (vsi->num_xsk_umems_used == 0) {
295		− kfree(vsi->xsk_umems);
296		− vsi->xsk_umems = NULL;
297		− vsi->num_xsk_umems = 0;
	294	+ if (vsi->num_xsk_pools_used == 0) {
	295	+ kfree(vsi->xsk_pools);
	296	+ vsi->xsk_pools = NULL;
	297	+ vsi->num_xsk_pools = 0;
298	298	}
299	299	}
300	300
301	301	/**
302		− * ice_xsk_umem_disable - disable a UMEM region
	302	+ * ice_xsk_pool_disable - disable a buffer pool region
303	303	* @vsi: Current VSI
304	304	* @qid: queue ID
305	305	*
306	306	* Returns 0 on success, negative on failure
307	307	*/
308		−static int ice_xsk_umem_disable(struct ice_vsi *vsi, u16 qid)
	308	+static int ice_xsk_pool_disable(struct ice_vsi *vsi, u16 qid)
309	309	{
310		− if (!vsi->xsk_umems \|\| qid >= vsi->num_xsk_umems \|\|
311		− !vsi->xsk_umems[qid])
	310	+ if (!vsi->xsk_pools \|\| qid >= vsi->num_xsk_pools \|\|
	311	+ !vsi->xsk_pools[qid])
312	312	return -EINVAL;
313	313
314		− xsk_buff_dma_unmap(vsi->xsk_umems[qid], ICE_RX_DMA_ATTR);
315		− ice_xsk_remove_umem(vsi, qid);
	314	+ xsk_pool_dma_unmap(vsi->xsk_pools[qid], ICE_RX_DMA_ATTR);
	315	+ ice_xsk_remove_pool(vsi, qid);
316	316
317	317	return 0;
318	318	}
319	319
320	320	/**
321		− * ice_xsk_umem_enable - enable a UMEM region
	321	+ * ice_xsk_pool_enable - enable a buffer pool region
322	322	* @vsi: Current VSI
323		− * @umem: pointer to a requested UMEM region
	323	+ * @pool: pointer to a requested buffer pool region
324	324	* @qid: queue ID
325	325	*
326	326	* Returns 0 on success, negative on failure
327	327	*/
328	328	static int
329		−ice_xsk_umem_enable(struct ice_vsi vsi, struct xdp_umem umem, u16 qid)
	329	+ice_xsk_pool_enable(struct ice_vsi vsi, struct xsk_buff_pool pool, u16 qid)
330	330	{
331	331	int err;
332	332
333	333	if (vsi->type != ICE_VSI_PF)
334	334	return -EINVAL;
335	335
336		− if (!vsi->num_xsk_umems)
337		− vsi->num_xsk_umems = min_t(u16, vsi->num_rxq, vsi->num_txq);
338		− if (qid >= vsi->num_xsk_umems)
	336	+ if (!vsi->num_xsk_pools)
	337	+ vsi->num_xsk_pools = min_t(u16, vsi->num_rxq, vsi->num_txq);
	338	+ if (qid >= vsi->num_xsk_pools)
339	339	return -EINVAL;
340	340
341		− err = ice_xsk_alloc_umems(vsi);
	341	+ err = ice_xsk_alloc_pools(vsi);
342	342	if (err)
343	343	return err;
344	344
345		− if (vsi->xsk_umems && vsi->xsk_umems[qid])
	345	+ if (vsi->xsk_pools && vsi->xsk_pools[qid])
346	346	return -EBUSY;
347	347
348		− vsi->xsk_umems[qid] = umem;
349		− vsi->num_xsk_umems_used++;
	348	+ vsi->xsk_pools[qid] = pool;
	349	+ vsi->num_xsk_pools_used++;
350	350
351		− err = xsk_buff_dma_map(vsi->xsk_umems[qid], ice_pf_to_dev(vsi->back),
	351	+ err = xsk_pool_dma_map(vsi->xsk_pools[qid], ice_pf_to_dev(vsi->back),
352	352	ICE_RX_DMA_ATTR);
353	353	if (err)
354	354	return err;
@@ -357,17 +357,17 @@ ice_xsk_umem_enable(struct ice_vsi vsi, struct xdp_umem umem, u16 qid) · 2 unmodified lines
357	357	}
358	358
359	359	/**
360		− * ice_xsk_umem_setup - enable/disable a UMEM region depending on its state
	360	+ * ice_xsk_pool_setup - enable/disable a buffer pool region depending on its state
361	361	* @vsi: Current VSI
362		− * @umem: UMEM to enable/associate to a ring, NULL to disable
	362	+ * @pool: buffer pool to enable/associate to a ring, NULL to disable
363	363	* @qid: queue ID
364	364	*
365	365	* Returns 0 on success, negative on failure
366	366	*/
367		−int ice_xsk_umem_setup(struct ice_vsi vsi, struct xdp_umem umem, u16 qid)
	367	+int ice_xsk_pool_setup(struct ice_vsi vsi, struct xsk_buff_pool pool, u16 qid)
368	368	{
369		− bool if_running, umem_present = !!umem;
370		− int ret = 0, umem_failure = 0;
	369	+ bool if_running, pool_present = !!pool;
	370	+ int ret = 0, pool_failure = 0;
371	371
372	372	if_running = netif_running(vsi->netdev) && ice_is_xdp_ena_vsi(vsi);
373	373
@@ -375,26 +375,26 @@ int ice_xsk_umem_setup(struct ice_vsi vsi, struct xdp_umem umem, u16 qid) · 1 unmodified line
375	375	ret = ice_qp_dis(vsi, qid);
376	376	if (ret) {
377	377	netdev_err(vsi->netdev, "ice_qp_dis error = %d\n", ret);
378		− goto xsk_umem_if_up;
	378	+ goto xsk_pool_if_up;
379	379	}
380	380	}
381	381
382		− umem_failure = umem_present ? ice_xsk_umem_enable(vsi, umem, qid) :
383		− ice_xsk_umem_disable(vsi, qid);
	382	+ pool_failure = pool_present ? ice_xsk_pool_enable(vsi, pool, qid) :
	383	+ ice_xsk_pool_disable(vsi, qid);
384	384
385		−xsk_umem_if_up:
	385	+xsk_pool_if_up:
386	386	if (if_running) {
387	387	ret = ice_qp_ena(vsi, qid);
388		− if (!ret && umem_present)
	388	+ if (!ret && pool_present)
389	389	napi_schedule(&vsi->xdp_rings[qid]->q_vector->napi);
390	390	else if (ret)
391	391	netdev_err(vsi->netdev, "ice_qp_ena error = %d\n", ret);
392	392	}
393	393
394		− if (umem_failure) {
395		− netdev_err(vsi->netdev, "Could not %sable UMEM, error = %d\n",
396		− umem_present ? "en" : "dis", umem_failure);
397		− return umem_failure;
	394	+ if (pool_failure) {
	395	+ netdev_err(vsi->netdev, "Could not %sable buffer pool, error = %d\n",
	396	+ pool_present ? "en" : "dis", pool_failure);
	397	+ return pool_failure;
398	398	}
399	399
400	400	return ret;
@@ -425,7 +425,7 @@ bool ice_alloc_rx_bufs_zc(struct ice_ring *rx_ring, u16 count) · 24 unmodified lines
425	425	rx_buf = &rx_ring->rx_buf[ntu];
426	426
427	427	do {
428		− rx_buf->xdp = xsk_buff_alloc(rx_ring->xsk_umem);
	428	+ rx_buf->xdp = xsk_buff_alloc(rx_ring->xsk_pool);
429	429	if (!rx_buf->xdp) {
430	430	ret = true;
431	431	break;
@@ -595,7 +595,7 @@ int ice_clean_rx_irq_zc(struct ice_ring *rx_ring, int budget) · 163 unmodified lines
595	595
596	596	rx_buf = &rx_ring->rx_buf[rx_ring->next_to_clean];
597	597	rx_buf->xdp->data_end = rx_buf->xdp->data + size;
598		− xsk_buff_dma_sync_for_cpu(rx_buf->xdp);
	598	+ xsk_buff_dma_sync_for_cpu(rx_buf->xdp, rx_ring->xsk_pool);
599	599
600	600	xdp_res = ice_run_xdp_zc(rx_ring, rx_buf->xdp);
601	601	if (xdp_res) {
@@ -645,11 +645,11 @@ int ice_clean_rx_irq_zc(struct ice_ring *rx_ring, int budget) · 43 unmodified lines
645	645	ice_finalize_xdp_rx(rx_ring, xdp_xmit);
646	646	ice_update_rx_ring_stats(rx_ring, total_rx_packets, total_rx_bytes);
647	647
648		− if (xsk_umem_uses_need_wakeup(rx_ring->xsk_umem)) {
	648	+ if (xsk_uses_need_wakeup(rx_ring->xsk_pool)) {
649	649	if (failure \|\| rx_ring->next_to_clean == rx_ring->next_to_use)
650		− xsk_set_rx_need_wakeup(rx_ring->xsk_umem);
	650	+ xsk_set_rx_need_wakeup(rx_ring->xsk_pool);
651	651	else
652		− xsk_clear_rx_need_wakeup(rx_ring->xsk_umem);
	652	+ xsk_clear_rx_need_wakeup(rx_ring->xsk_pool);
653	653
654	654	return (int)total_rx_packets;
655	655	}
@@ -682,11 +682,11 @@ static bool ice_xmit_zc(struct ice_ring *xdp_ring, int budget) · 26 unmodified lines
682	682
683	683	tx_buf = &xdp_ring->tx_buf[xdp_ring->next_to_use];
684	684
685		− if (!xsk_umem_consume_tx(xdp_ring->xsk_umem, &desc))
	685	+ if (!xsk_tx_peek_desc(xdp_ring->xsk_pool, &desc))
686	686	break;
687	687
688		− dma = xsk_buff_raw_get_dma(xdp_ring->xsk_umem, desc.addr);
689		− xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_umem, dma,
	688	+ dma = xsk_buff_raw_get_dma(xdp_ring->xsk_pool, desc.addr);
	689	+ xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_pool, dma,
690	690	desc.len);
691	691
692	692	tx_buf->bytecount = desc.len;
@@ -703,7 +703,7 @@ static bool ice_xmit_zc(struct ice_ring *xdp_ring, int budget) · 10 unmodified lines
703	703
704	704	if (tx_desc) {
705	705	ice_xdp_ring_update_tail(xdp_ring);
706		− xsk_umem_consume_tx_done(xdp_ring->xsk_umem);
	706	+ xsk_tx_release(xdp_ring->xsk_pool);
707	707	}
708	708
709	709	return budget > 0 && work_done;
@@ -777,10 +777,10 @@ bool ice_clean_tx_irq_zc(struct ice_ring *xdp_ring, int budget) · 67 unmodified lines
777	777	xdp_ring->next_to_clean = ntc;
778	778
779	779	if (xsk_frames)
780		− xsk_umem_complete_tx(xdp_ring->xsk_umem, xsk_frames);
	780	+ xsk_tx_completed(xdp_ring->xsk_pool, xsk_frames);
781	781
782		− if (xsk_umem_uses_need_wakeup(xdp_ring->xsk_umem))
783		− xsk_set_tx_need_wakeup(xdp_ring->xsk_umem);
	782	+ if (xsk_uses_need_wakeup(xdp_ring->xsk_pool))
	783	+ xsk_set_tx_need_wakeup(xdp_ring->xsk_pool);
784	784
785	785	ice_update_tx_ring_stats(xdp_ring, total_packets, total_bytes);
786	786	xmit_done = ice_xmit_zc(xdp_ring, ICE_DFLT_IRQ_WORK);
@@ -814,7 +814,7 @@ ice_xsk_wakeup(struct net_device *netdev, u32 queue_id, · 27 unmodified lines
814	814	if (queue_id >= vsi->num_txq)
815	815	return -ENXIO;
816	816
817		− if (!vsi->xdp_rings[queue_id]->xsk_umem)
	817	+ if (!vsi->xdp_rings[queue_id]->xsk_pool)
818	818	return -ENXIO;
819	819
820	820	ring = vsi->xdp_rings[queue_id];
@@ -833,20 +833,20 @@ ice_xsk_wakeup(struct net_device *netdev, u32 queue_id, · 12 unmodified lines
833	833	}
834	834
835	835	/**
836		− * ice_xsk_any_rx_ring_ena - Checks if Rx rings have AF_XDP UMEM attached
	836	+ * ice_xsk_any_rx_ring_ena - Checks if Rx rings have AF_XDP buff pool attached
837	837	* @vsi: VSI to be checked
838	838	*
839		− * Returns true if any of the Rx rings has an AF_XDP UMEM attached
	839	+ * Returns true if any of the Rx rings has an AF_XDP buff pool attached
840	840	*/
841	841	bool ice_xsk_any_rx_ring_ena(struct ice_vsi *vsi)
842	842	{
843	843	int i;
844	844
845		− if (!vsi->xsk_umems)
	845	+ if (!vsi->xsk_pools)
846	846	return false;
847	847
848		− for (i = 0; i < vsi->num_xsk_umems; i++) {
849		− if (vsi->xsk_umems[i])
	848	+ for (i = 0; i < vsi->num_xsk_pools; i++) {
	849	+ if (vsi->xsk_pools[i])
850	850	return true;
851	851	}
852	852
@@ -854,7 +854,7 @@ bool ice_xsk_any_rx_ring_ena(struct ice_vsi *vsi) · 1 unmodified line
854	854	}
855	855
856	856	/**
857		− * ice_xsk_clean_rx_ring - clean UMEM queues connected to a given Rx ring
	857	+ * ice_xsk_clean_rx_ring - clean buffer pool queues connected to a given Rx ring
858	858	* @rx_ring: ring to be cleaned
859	859	*/
860	860	void ice_xsk_clean_rx_ring(struct ice_ring *rx_ring)
@@ -872,7 +872,7 @@ void ice_xsk_clean_rx_ring(struct ice_ring *rx_ring) · 11 unmodified lines
872	872	}
873	873
874	874	/**
875		− * ice_xsk_clean_xdp_ring - Clean the XDP Tx ring and its UMEM queues
	875	+ * ice_xsk_clean_xdp_ring - Clean the XDP Tx ring and its buffer pool queues
876	876	* @xdp_ring: XDP_Tx ring
877	877	*/
878	878	void ice_xsk_clean_xdp_ring(struct ice_ring *xdp_ring)
@@ -896,5 +896,5 @@ void ice_xsk_clean_xdp_ring(struct ice_ring *xdp_ring) · 17 unmodified lines
896	896	}
897	897
898	898	if (xsk_frames)
899		− xsk_umem_complete_tx(xdp_ring->xsk_umem, xsk_frames);
	899	+ xsk_tx_completed(xdp_ring->xsk_pool, xsk_frames);
900	900	}

drivers/net/ethernet/intel/ice/ice_xsk.h+4−3

@@ -9,7 +9,8 @@ · 8 unmodified lines
9	9	struct ice_vsi;
10	10
11	11	#ifdef CONFIG_XDP_SOCKETS
12		−int ice_xsk_umem_setup(struct ice_vsi vsi, struct xdp_umem umem, u16 qid);
	12	+int ice_xsk_pool_setup(struct ice_vsi vsi, struct xsk_buff_pool pool,
	13	+ u16 qid);
13	14	int ice_clean_rx_irq_zc(struct ice_ring *rx_ring, int budget);
14	15	bool ice_clean_tx_irq_zc(struct ice_ring *xdp_ring, int budget);
15	16	int ice_xsk_wakeup(struct net_device *netdev, u32 queue_id, u32 flags);
@@ -19,8 +20,8 @@ void ice_xsk_clean_rx_ring(struct ice_ring *rx_ring); · 3 unmodified lines
19	20	void ice_xsk_clean_xdp_ring(struct ice_ring *xdp_ring);
20	21	#else
21	22	static inline int
22		−ice_xsk_umem_setup(struct ice_vsi __always_unused *vsi,
23		− struct xdp_umem __always_unused *umem,
	23	+ice_xsk_pool_setup(struct ice_vsi __always_unused *vsi,
	24	+ struct xsk_buff_pool __always_unused *pool,
24	25	u16 __always_unused qid)
25	26	{
26	27	return -EOPNOTSUPP;

drivers/net/ethernet/intel/ixgbe/ixgbe.h+1−1

@@ -350,7 +350,7 @@ struct ixgbe_ring { · 349 unmodified lines
350	350	struct ixgbe_rx_queue_stats rx_stats;
351	351	};
352	352	struct xdp_rxq_info xdp_rxq;
353		− struct xdp_umem *xsk_umem;
	353	+ struct xsk_buff_pool *xsk_pool;
354	354	u16 ring_idx; /* {rx,tx,xdp}_ring back reference idx */
355	355	u16 rx_buf_len;
356	356	} ____cacheline_internodealigned_in_smp;

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c+17−17

@@ -3151,7 +3151,7 @@ int ixgbe_poll(struct napi_struct *napi, int budget) · 3150 unmodified lines
3151	3151	#endif
3152	3152
3153	3153	ixgbe_for_each_ring(ring, q_vector->tx) {
3154		− bool wd = ring->xsk_umem ?
	3154	+ bool wd = ring->xsk_pool ?
3155	3155	ixgbe_clean_xdp_tx_irq(q_vector, ring, budget) :
3156	3156	ixgbe_clean_tx_irq(q_vector, ring, budget);
3157	3157
@@ -3171,7 +3171,7 @@ int ixgbe_poll(struct napi_struct *napi, int budget) · 13 unmodified lines
3171	3171	per_ring_budget = budget;
3172	3172
3173	3173	ixgbe_for_each_ring(ring, q_vector->rx) {
3174		− int cleaned = ring->xsk_umem ?
	3174	+ int cleaned = ring->xsk_pool ?
3175	3175	ixgbe_clean_rx_irq_zc(q_vector, ring,
3176	3176	per_ring_budget) :
3177	3177	ixgbe_clean_rx_irq(q_vector, ring,
@@ -3466,9 +3466,9 @@ void ixgbe_configure_tx_ring(struct ixgbe_adapter *adapter, · 288 unmodified lines
3466	3466	u32 txdctl = IXGBE_TXDCTL_ENABLE;
3467	3467	u8 reg_idx = ring->reg_idx;
3468	3468
3469		− ring->xsk_umem = NULL;
	3469	+ ring->xsk_pool = NULL;
3470	3470	if (ring_is_xdp(ring))
3471		− ring->xsk_umem = ixgbe_xsk_umem(adapter, ring);
	3471	+ ring->xsk_pool = ixgbe_xsk_pool(adapter, ring);
3472	3472
3473	3473	/* disable queue to avoid issues while updating state */
3474	3474	IXGBE_WRITE_REG(hw, IXGBE_TXDCTL(reg_idx), 0);
@@ -3708,8 +3708,8 @@ static void ixgbe_configure_srrctl(struct ixgbe_adapter *adapter, · 233 unmodified lines
3708	3708	srrctl = IXGBE_RX_HDR_SIZE << IXGBE_SRRCTL_BSIZEHDRSIZE_SHIFT;
3709	3709
3710	3710	/* configure the packet buffer length */
3711		− if (rx_ring->xsk_umem) {
3712		− u32 xsk_buf_len = xsk_umem_get_rx_frame_size(rx_ring->xsk_umem);
	3711	+ if (rx_ring->xsk_pool) {
	3712	+ u32 xsk_buf_len = xsk_pool_get_rx_frame_size(rx_ring->xsk_pool);
3713	3713
3714	3714	/* If the MAC support setting RXDCTL.RLPML, the
3715	3715	* SRRCTL[n].BSIZEPKT is set to PAGE_SIZE and
@@ -4054,12 +4054,12 @@ void ixgbe_configure_rx_ring(struct ixgbe_adapter *adapter, · 338 unmodified lines
4054	4054	u8 reg_idx = ring->reg_idx;
4055	4055
4056	4056	xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
4057		− ring->xsk_umem = ixgbe_xsk_umem(adapter, ring);
4058		− if (ring->xsk_umem) {
	4057	+ ring->xsk_pool = ixgbe_xsk_pool(adapter, ring);
	4058	+ if (ring->xsk_pool) {
4059	4059	WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
4060	4060	MEM_TYPE_XSK_BUFF_POOL,
4061	4061	NULL));
4062		− xsk_buff_set_rxq_info(ring->xsk_umem, &ring->xdp_rxq);
	4062	+ xsk_pool_set_rxq_info(ring->xsk_pool, &ring->xdp_rxq);
4063	4063	} else {
4064	4064	WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
4065	4065	MEM_TYPE_PAGE_SHARED, NULL));
@@ -4114,8 +4114,8 @@ void ixgbe_configure_rx_ring(struct ixgbe_adapter *adapter, · 48 unmodified lines
4114	4114	#endif
4115	4115	}
4116	4116
4117		− if (ring->xsk_umem && hw->mac.type != ixgbe_mac_82599EB) {
4118		− u32 xsk_buf_len = xsk_umem_get_rx_frame_size(ring->xsk_umem);
	4117	+ if (ring->xsk_pool && hw->mac.type != ixgbe_mac_82599EB) {
	4118	+ u32 xsk_buf_len = xsk_pool_get_rx_frame_size(ring->xsk_pool);
4119	4119
4120	4120	rxdctl &= ~(IXGBE_RXDCTL_RLPMLMASK \|
4121	4121	IXGBE_RXDCTL_RLPML_EN);
@@ -4137,7 +4137,7 @@ void ixgbe_configure_rx_ring(struct ixgbe_adapter *adapter, · 15 unmodified lines
4137	4137	IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(reg_idx), rxdctl);
4138	4138
4139	4139	ixgbe_rx_desc_queue_enable(adapter, ring);
4140		− if (ring->xsk_umem)
	4140	+ if (ring->xsk_pool)
4141	4141	ixgbe_alloc_rx_buffers_zc(ring, ixgbe_desc_unused(ring));
4142	4142	else
4143	4143	ixgbe_alloc_rx_buffers(ring, ixgbe_desc_unused(ring));
@@ -5287,7 +5287,7 @@ static void ixgbe_clean_rx_ring(struct ixgbe_ring *rx_ring) · 1143 unmodified lines
5287	5287	u16 i = rx_ring->next_to_clean;
5288	5288	struct ixgbe_rx_buffer *rx_buffer = &rx_ring->rx_buffer_info[i];
5289	5289
5290		− if (rx_ring->xsk_umem) {
	5290	+ if (rx_ring->xsk_pool) {
5291	5291	ixgbe_xsk_clean_rx_ring(rx_ring);
5292	5292	goto skip_free;
5293	5293	}
@@ -5979,7 +5979,7 @@ static void ixgbe_clean_tx_ring(struct ixgbe_ring *tx_ring) · 685 unmodified lines
5979	5979	u16 i = tx_ring->next_to_clean;
5980	5980	struct ixgbe_tx_buffer *tx_buffer = &tx_ring->tx_buffer_info[i];
5981	5981
5982		− if (tx_ring->xsk_umem) {
	5982	+ if (tx_ring->xsk_pool) {
5983	5983	ixgbe_xsk_clean_tx_ring(tx_ring);
5984	5984	goto out;
5985	5985	}
@@ -10141,7 +10141,7 @@ static int ixgbe_xdp_setup(struct net_device dev, struct bpf_prog prog) · 4155 unmodified lines
10141	10141	*/
10142	10142	if (need_reset && prog)
10143	10143	for (i = 0; i < adapter->num_rx_queues; i++)
10144		− if (adapter->xdp_ring[i]->xsk_umem)
	10144	+ if (adapter->xdp_ring[i]->xsk_pool)
10145	10145	(void)ixgbe_xsk_wakeup(adapter->netdev, i,
10146	10146	XDP_WAKEUP_RX);
10147	10147
@@ -10155,8 +10155,8 @@ static int ixgbe_xdp(struct net_device dev, struct netdev_bpf xdp) · 7 unmodified lines
10155	10155	switch (xdp->command) {
10156	10156	case XDP_SETUP_PROG:
10157	10157	return ixgbe_xdp_setup(dev, xdp->prog);
10158		− case XDP_SETUP_XSK_UMEM:
10159		− return ixgbe_xsk_umem_setup(adapter, xdp->xsk.umem,
	10158	+ case XDP_SETUP_XSK_POOL:
	10159	+ return ixgbe_xsk_pool_setup(adapter, xdp->xsk.pool,
10160	10160	xdp->xsk.queue_id);
10161	10161
10162	10162	default:

drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h+4−3

@@ -28,9 +28,10 @@ void ixgbe_irq_rearm_queues(struct ixgbe_adapter *adapter, u64 qmask); · 27 unmodified lines
28	28	void ixgbe_txrx_ring_disable(struct ixgbe_adapter *adapter, int ring);
29	29	void ixgbe_txrx_ring_enable(struct ixgbe_adapter *adapter, int ring);
30	30
31		−struct xdp_umem ixgbe_xsk_umem(struct ixgbe_adapter adapter,
32		− struct ixgbe_ring *ring);
33		−int ixgbe_xsk_umem_setup(struct ixgbe_adapter adapter, struct xdp_umem umem,
	31	+struct xsk_buff_pool ixgbe_xsk_pool(struct ixgbe_adapter adapter,
	32	+ struct ixgbe_ring *ring);
	33	+int ixgbe_xsk_pool_setup(struct ixgbe_adapter *adapter,
	34	+ struct xsk_buff_pool *pool,
34	35	u16 qid);
35	36
36	37	void ixgbe_zca_free(struct zero_copy_allocator *alloc, unsigned long handle);

drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c+32−31

@@ -8,8 +8,8 @@ · 7 unmodified lines
8	8	#include "ixgbe.h"
9	9	#include "ixgbe_txrx_common.h"
10	10
11		−struct xdp_umem ixgbe_xsk_umem(struct ixgbe_adapter adapter,
12		− struct ixgbe_ring *ring)
	11	+struct xsk_buff_pool ixgbe_xsk_pool(struct ixgbe_adapter adapter,
	12	+ struct ixgbe_ring *ring)
13	13	{
14	14	bool xdp_on = READ_ONCE(adapter->xdp_prog);
15	15	int qid = ring->ring_idx;
@@ -17,11 +17,11 @@ struct xdp_umem ixgbe_xsk_umem(struct ixgbe_adapter adapter, · 1 unmodified line
17	17	if (!xdp_on \|\| !test_bit(qid, adapter->af_xdp_zc_qps))
18	18	return NULL;
19	19
20		− return xdp_get_umem_from_qid(adapter->netdev, qid);
	20	+ return xsk_get_pool_from_qid(adapter->netdev, qid);
21	21	}
22	22
23		−static int ixgbe_xsk_umem_enable(struct ixgbe_adapter *adapter,
24		− struct xdp_umem *umem,
	23	+static int ixgbe_xsk_pool_enable(struct ixgbe_adapter *adapter,
	24	+ struct xsk_buff_pool *pool,
25	25	u16 qid)
26	26	{
27	27	struct net_device *netdev = adapter->netdev;
@@ -35,7 +35,7 @@ static int ixgbe_xsk_umem_enable(struct ixgbe_adapter *adapter, · 7 unmodified lines
35	35	qid >= netdev->real_num_tx_queues)
36	36	return -EINVAL;
37	37
38		− err = xsk_buff_dma_map(umem, &adapter->pdev->dev, IXGBE_RX_DMA_ATTR);
	38	+ err = xsk_pool_dma_map(pool, &adapter->pdev->dev, IXGBE_RX_DMA_ATTR);
39	39	if (err)
40	40	return err;
41	41
@@ -59,13 +59,13 @@ static int ixgbe_xsk_umem_enable(struct ixgbe_adapter *adapter, · 17 unmodified lines
59	59	return 0;
60	60	}
61	61
62		−static int ixgbe_xsk_umem_disable(struct ixgbe_adapter *adapter, u16 qid)
	62	+static int ixgbe_xsk_pool_disable(struct ixgbe_adapter *adapter, u16 qid)
63	63	{
64		− struct xdp_umem *umem;
	64	+ struct xsk_buff_pool *pool;
65	65	bool if_running;
66	66
67		− umem = xdp_get_umem_from_qid(adapter->netdev, qid);
68		− if (!umem)
	67	+ pool = xsk_get_pool_from_qid(adapter->netdev, qid);
	68	+ if (!pool)
69	69	return -EINVAL;
70	70
71	71	if_running = netif_running(adapter->netdev) &&
@@ -75,7 +75,7 @@ static int ixgbe_xsk_umem_disable(struct ixgbe_adapter *adapter, u16 qid) · 3 unmodified lines
75	75	ixgbe_txrx_ring_disable(adapter, qid);
76	76
77	77	clear_bit(qid, adapter->af_xdp_zc_qps);
78		− xsk_buff_dma_unmap(umem, IXGBE_RX_DMA_ATTR);
	78	+ xsk_pool_dma_unmap(pool, IXGBE_RX_DMA_ATTR);
79	79
80	80	if (if_running)
81	81	ixgbe_txrx_ring_enable(adapter, qid);
@@ -83,11 +83,12 @@ static int ixgbe_xsk_umem_disable(struct ixgbe_adapter *adapter, u16 qid) · 1 unmodified line
83	83	return 0;
84	84	}
85	85
86		−int ixgbe_xsk_umem_setup(struct ixgbe_adapter adapter, struct xdp_umem umem,
	86	+int ixgbe_xsk_pool_setup(struct ixgbe_adapter *adapter,
	87	+ struct xsk_buff_pool *pool,
87	88	u16 qid)
88	89	{
89		− return umem ? ixgbe_xsk_umem_enable(adapter, umem, qid) :
90		− ixgbe_xsk_umem_disable(adapter, qid);
	90	+ return pool ? ixgbe_xsk_pool_enable(adapter, pool, qid) :
	91	+ ixgbe_xsk_pool_disable(adapter, qid);
91	92	}
92	93
93	94	static int ixgbe_run_xdp_zc(struct ixgbe_adapter *adapter,
@@ -149,7 +150,7 @@ bool ixgbe_alloc_rx_buffers_zc(struct ixgbe_ring *rx_ring, u16 count) · 55 unmodified lines
149	150	i -= rx_ring->count;
150	151
151	152	do {
152		− bi->xdp = xsk_buff_alloc(rx_ring->xsk_umem);
	153	+ bi->xdp = xsk_buff_alloc(rx_ring->xsk_pool);
153	154	if (!bi->xdp) {
154	155	ok = false;
155	156	break;
@@ -286,7 +287,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, · 130 unmodified lines
286	287	}
287	288
288	289	bi->xdp->data_end = bi->xdp->data + size;
289		− xsk_buff_dma_sync_for_cpu(bi->xdp);
	290	+ xsk_buff_dma_sync_for_cpu(bi->xdp, rx_ring->xsk_pool);
290	291	xdp_res = ixgbe_run_xdp_zc(adapter, rx_ring, bi->xdp);
291	292
292	293	if (xdp_res) {
@@ -344,11 +345,11 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, · 51 unmodified lines
344	345	q_vector->rx.total_packets += total_rx_packets;
345	346	q_vector->rx.total_bytes += total_rx_bytes;
346	347
347		− if (xsk_umem_uses_need_wakeup(rx_ring->xsk_umem)) {
	348	+ if (xsk_uses_need_wakeup(rx_ring->xsk_pool)) {
348	349	if (failure \|\| rx_ring->next_to_clean == rx_ring->next_to_use)
349		− xsk_set_rx_need_wakeup(rx_ring->xsk_umem);
	350	+ xsk_set_rx_need_wakeup(rx_ring->xsk_pool);
350	351	else
351		− xsk_clear_rx_need_wakeup(rx_ring->xsk_umem);
	352	+ xsk_clear_rx_need_wakeup(rx_ring->xsk_pool);
352	353
353	354	return (int)total_rx_packets;
354	355	}
@@ -373,6 +374,7 @@ void ixgbe_xsk_clean_rx_ring(struct ixgbe_ring *rx_ring) · 18 unmodified lines
373	374
374	375	static bool ixgbe_xmit_zc(struct ixgbe_ring *xdp_ring, unsigned int budget)
375	376	{
	377	+ struct xsk_buff_pool *pool = xdp_ring->xsk_pool;
376	378	union ixgbe_adv_tx_desc *tx_desc = NULL;
377	379	struct ixgbe_tx_buffer *tx_bi;
378	380	bool work_done = true;
@@ -387,12 +389,11 @@ static bool ixgbe_xmit_zc(struct ixgbe_ring *xdp_ring, unsigned int budget) · 8 unmodified lines
387	389	break;
388	390	}
389	391
390		− if (!xsk_umem_consume_tx(xdp_ring->xsk_umem, &desc))
	392	+ if (!xsk_tx_peek_desc(pool, &desc))
391	393	break;
392	394
393		− dma = xsk_buff_raw_get_dma(xdp_ring->xsk_umem, desc.addr);
394		− xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_umem, dma,
395		− desc.len);
	395	+ dma = xsk_buff_raw_get_dma(pool, desc.addr);
	396	+ xsk_buff_raw_dma_sync_for_device(pool, dma, desc.len);
396	397
397	398	tx_bi = &xdp_ring->tx_buffer_info[xdp_ring->next_to_use];
398	399	tx_bi->bytecount = desc.len;
@@ -418,7 +419,7 @@ static bool ixgbe_xmit_zc(struct ixgbe_ring *xdp_ring, unsigned int budget) · 19 unmodified lines
418	419
419	420	if (tx_desc) {
420	421	ixgbe_xdp_ring_update_tail(xdp_ring);
421		− xsk_umem_consume_tx_done(xdp_ring->xsk_umem);
	422	+ xsk_tx_release(pool);
422	423	}
423	424
424	425	return !!budget && work_done;
@@ -439,7 +440,7 @@ bool ixgbe_clean_xdp_tx_irq(struct ixgbe_q_vector *q_vector, · 14 unmodified lines
439	440	{
440	441	u16 ntc = tx_ring->next_to_clean, ntu = tx_ring->next_to_use;
441	442	unsigned int total_packets = 0, total_bytes = 0;
442		− struct xdp_umem *umem = tx_ring->xsk_umem;
	443	+ struct xsk_buff_pool *pool = tx_ring->xsk_pool;
443	444	union ixgbe_adv_tx_desc *tx_desc;
444	445	struct ixgbe_tx_buffer *tx_bi;
445	446	u32 xsk_frames = 0;
@@ -484,10 +485,10 @@ bool ixgbe_clean_xdp_tx_irq(struct ixgbe_q_vector *q_vector, · 38 unmodified lines
484	485	q_vector->tx.total_packets += total_packets;
485	486
486	487	if (xsk_frames)
487		− xsk_umem_complete_tx(umem, xsk_frames);
	488	+ xsk_tx_completed(pool, xsk_frames);
488	489
489		− if (xsk_umem_uses_need_wakeup(tx_ring->xsk_umem))
490		− xsk_set_tx_need_wakeup(tx_ring->xsk_umem);
	490	+ if (xsk_uses_need_wakeup(pool))
	491	+ xsk_set_tx_need_wakeup(pool);
491	492
492	493	return ixgbe_xmit_zc(tx_ring, q_vector->tx.work_limit);
493	494	}
@@ -511,7 +512,7 @@ int ixgbe_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags) · 17 unmodified lines
511	512	if (test_bit(__IXGBE_TX_DISABLED, &ring->state))
512	513	return -ENETDOWN;
513	514
514		− if (!ring->xsk_umem)
	515	+ if (!ring->xsk_pool)
515	516	return -ENXIO;
516	517
517	518	if (!napi_if_scheduled_mark_missed(&ring->q_vector->napi)) {
@@ -526,7 +527,7 @@ int ixgbe_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags) · 8 unmodified lines
526	527	void ixgbe_xsk_clean_tx_ring(struct ixgbe_ring *tx_ring)
527	528	{
528	529	u16 ntc = tx_ring->next_to_clean, ntu = tx_ring->next_to_use;
529		− struct xdp_umem *umem = tx_ring->xsk_umem;
	530	+ struct xsk_buff_pool *pool = tx_ring->xsk_pool;
530	531	struct ixgbe_tx_buffer *tx_bi;
531	532	u32 xsk_frames = 0;
532	533
@@ -546,5 +547,5 @@ void ixgbe_xsk_clean_tx_ring(struct ixgbe_ring *tx_ring) · 13 unmodified lines
546	547	}
547	548
548	549	if (xsk_frames)
549		− xsk_umem_complete_tx(umem, xsk_frames);
	550	+ xsk_tx_completed(pool, xsk_frames);
550	551	}

drivers/net/ethernet/mellanox/mlx5/core/Makefile+1−1

@@ -24,7 +24,7 @@ mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \ · 23 unmodified lines
24	24	mlx5_core-$(CONFIG_MLX5_CORE_EN) += en_main.o en_common.o en_fs.o en_ethtool.o \
25	25	en_tx.o en_rx.o en_dim.o en_txrx.o en/xdp.o en_stats.o \
26	26	en_selftest.o en/port.o en/monitor_stats.o en/health.o \
27		− en/reporter_tx.o en/reporter_rx.o en/params.o en/xsk/umem.o \
	27	+ en/reporter_tx.o en/reporter_rx.o en/params.o en/xsk/pool.o \
28	28	en/xsk/setup.o en/xsk/rx.o en/xsk/tx.o en/devlink.o
29	29
30	30	#

drivers/net/ethernet/mellanox/mlx5/core/en.h+10−9

@@ -442,7 +442,7 @@ struct mlx5e_xdpsq { · 441 unmodified lines
442	442	struct mlx5e_cq cq;
443	443
444	444	/* read only */
445		− struct xdp_umem *umem;
	445	+ struct xsk_buff_pool *xsk_pool;
446	446	struct mlx5_wq_cyc wq;
447	447	struct mlx5e_xdpsq_stats *stats;
448	448	mlx5e_fp_xmit_xdp_frame_check xmit_xdp_frame_check;
@@ -606,7 +606,7 @@ struct mlx5e_rq { · 157 unmodified lines
606	606	struct page_pool *page_pool;
607	607
608	608	/* AF_XDP zero-copy */
609		− struct xdp_umem *umem;
	609	+ struct xsk_buff_pool *xsk_pool;
610	610
611	611	struct work_struct recover_work;
612	612
@@ -729,12 +729,13 @@ struct mlx5e_hv_vhca_stats_agent { · 116 unmodified lines
729	729	#endif
730	730
731	731	struct mlx5e_xsk {
732		− /* UMEMs are stored separately from channels, because we don't want to
733		− * lose them when channels are recreated. The kernel also stores UMEMs,
734		− * but it doesn't distinguish between zero-copy and non-zero-copy UMEMs,
735		− * so rely on our mechanism.
	732	+ /* XSK buffer pools are stored separately from channels,
	733	+ * because we don't want to lose them when channels are
	734	+ * recreated. The kernel also stores buffer pool, but it doesn't
	735	+ * distinguish between zero-copy and non-zero-copy UMEMs, so
	736	+ * rely on our mechanism.
736	737	*/
737		− struct xdp_umem **umems;
	738	+ struct xsk_buff_pool **pools;
738	739	u16 refcnt;
739	740	bool ever_used;
740	741	};
@@ -893,7 +894,7 @@ struct mlx5e_xsk_param; · 152 unmodified lines
893	894	struct mlx5e_rq_param;
894	895	int mlx5e_open_rq(struct mlx5e_channel c, struct mlx5e_params params,
895	896	struct mlx5e_rq_param param, struct mlx5e_xsk_param xsk,
896		− struct xdp_umem umem, struct mlx5e_rq rq);
	897	+ struct xsk_buff_pool xsk_pool, struct mlx5e_rq rq);
897	898	int mlx5e_wait_for_min_rx_wqes(struct mlx5e_rq *rq, int wait_time);
898	899	void mlx5e_deactivate_rq(struct mlx5e_rq *rq);
899	900	void mlx5e_close_rq(struct mlx5e_rq *rq);
@@ -903,7 +904,7 @@ int mlx5e_open_icosq(struct mlx5e_channel c, struct mlx5e_params params, · 3 unmodified lines
903	904	struct mlx5e_sq_param param, struct mlx5e_icosq sq);
904	905	void mlx5e_close_icosq(struct mlx5e_icosq *sq);
905	906	int mlx5e_open_xdpsq(struct mlx5e_channel c, struct mlx5e_params params,
906		− struct mlx5e_sq_param param, struct xdp_umem umem,
	907	+ struct mlx5e_sq_param param, struct xsk_buff_pool xsk_pool,
907	908	struct mlx5e_xdpsq *sq, bool is_redirect);
908	909	void mlx5e_close_xdpsq(struct mlx5e_xdpsq *sq);
909	910

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c+2−3

@@ -445,7 +445,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq) · 444 unmodified lines
445	445	} while ((++i < MLX5E_TX_CQ_POLL_BUDGET) && (cqe = mlx5_cqwq_get_cqe(&cq->wq)));
446	446
447	447	if (xsk_frames)
448		− xsk_umem_complete_tx(sq->umem, xsk_frames);
	448	+ xsk_tx_completed(sq->xsk_pool, xsk_frames);
449	449
450	450	sq->stats->cqes += i;
451	451
@@ -475,7 +475,7 @@ void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq) · 23 unmodified lines
475	475	}
476	476
477	477	if (xsk_frames)
478		− xsk_umem_complete_tx(sq->umem, xsk_frames);
	478	+ xsk_tx_completed(sq->xsk_pool, xsk_frames);
479	479	}
480	480
481	481	int mlx5e_xdp_xmit(struct net_device dev, int n, struct xdp_frame *frames,
@@ -563,4 +563,3 @@ void mlx5e_set_xmit_fp(struct mlx5e_xdpsq *sq, bool is_mpw) · 81 unmodified lines
563	563	sq->xmit_xdp_frame = is_mpw ?
564	564	mlx5e_xmit_xdp_frame_mpwqe : mlx5e_xmit_xdp_frame;
565	565	}
566		−

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/pool.crenamed from drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.c+55−55

@@ -1,31 +1,31 @@
1	1	// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2		−/* Copyright (c) 2019 Mellanox Technologies. */
	2	+/* Copyright (c) 2019-2020, Mellanox Technologies inc. All rights reserved. */
3	3
4	4	#include <net/xdp_sock_drv.h>
5		−#include "umem.h"
	5	+#include "pool.h"
6	6	#include "setup.h"
7	7	#include "en/params.h"
8	8
9		−static int mlx5e_xsk_map_umem(struct mlx5e_priv *priv,
10		− struct xdp_umem *umem)
	9	+static int mlx5e_xsk_map_pool(struct mlx5e_priv *priv,
	10	+ struct xsk_buff_pool *pool)
11	11	{
12	12	struct device *dev = priv->mdev->device;
13	13
14		− return xsk_buff_dma_map(umem, dev, 0);
	14	+ return xsk_pool_dma_map(pool, dev, 0);
15	15	}
16	16
17		−static void mlx5e_xsk_unmap_umem(struct mlx5e_priv *priv,
18		− struct xdp_umem *umem)
	17	+static void mlx5e_xsk_unmap_pool(struct mlx5e_priv *priv,
	18	+ struct xsk_buff_pool *pool)
19	19	{
20		− return xsk_buff_dma_unmap(umem, 0);
	20	+ return xsk_pool_dma_unmap(pool, 0);
21	21	}
22	22
23		−static int mlx5e_xsk_get_umems(struct mlx5e_xsk *xsk)
	23	+static int mlx5e_xsk_get_pools(struct mlx5e_xsk *xsk)
24	24	{
25		− if (!xsk->umems) {
26		− xsk->umems = kcalloc(MLX5E_MAX_NUM_CHANNELS,
27		− sizeof(*xsk->umems), GFP_KERNEL);
28		− if (unlikely(!xsk->umems))
	25	+ if (!xsk->pools) {
	26	+ xsk->pools = kcalloc(MLX5E_MAX_NUM_CHANNELS,
	27	+ sizeof(*xsk->pools), GFP_KERNEL);
	28	+ if (unlikely(!xsk->pools))
29	29	return -ENOMEM;
30	30	}
31	31
@@ -35,68 +35,68 @@ static int mlx5e_xsk_get_umems(struct mlx5e_xsk *xsk) · 3 unmodified lines
35	35	return 0;
36	36	}
37	37
38		−static void mlx5e_xsk_put_umems(struct mlx5e_xsk *xsk)
	38	+static void mlx5e_xsk_put_pools(struct mlx5e_xsk *xsk)
39	39	{
40	40	if (!--xsk->refcnt) {
41		− kfree(xsk->umems);
42		− xsk->umems = NULL;
	41	+ kfree(xsk->pools);
	42	+ xsk->pools = NULL;
43	43	}
44	44	}
45	45
46		−static int mlx5e_xsk_add_umem(struct mlx5e_xsk xsk, struct xdp_umem umem, u16 ix)
	46	+static int mlx5e_xsk_add_pool(struct mlx5e_xsk xsk, struct xsk_buff_pool pool, u16 ix)
47	47	{
48	48	int err;
49	49
50		− err = mlx5e_xsk_get_umems(xsk);
	50	+ err = mlx5e_xsk_get_pools(xsk);
51	51	if (unlikely(err))
52	52	return err;
53	53
54		− xsk->umems[ix] = umem;
	54	+ xsk->pools[ix] = pool;
55	55	return 0;
56	56	}
57	57
58		−static void mlx5e_xsk_remove_umem(struct mlx5e_xsk *xsk, u16 ix)
	58	+static void mlx5e_xsk_remove_pool(struct mlx5e_xsk *xsk, u16 ix)
59	59	{
60		− xsk->umems[ix] = NULL;
	60	+ xsk->pools[ix] = NULL;
61	61
62		− mlx5e_xsk_put_umems(xsk);
	62	+ mlx5e_xsk_put_pools(xsk);
63	63	}
64	64
65		−static bool mlx5e_xsk_is_umem_sane(struct xdp_umem *umem)
	65	+static bool mlx5e_xsk_is_pool_sane(struct xsk_buff_pool *pool)
66	66	{
67		− return xsk_umem_get_headroom(umem) <= 0xffff &&
68		− xsk_umem_get_chunk_size(umem) <= 0xffff;
	67	+ return xsk_pool_get_headroom(pool) <= 0xffff &&
	68	+ xsk_pool_get_chunk_size(pool) <= 0xffff;
69	69	}
70	70
71		−void mlx5e_build_xsk_param(struct xdp_umem umem, struct mlx5e_xsk_param xsk)
	71	+void mlx5e_build_xsk_param(struct xsk_buff_pool pool, struct mlx5e_xsk_param xsk)
72	72	{
73		− xsk->headroom = xsk_umem_get_headroom(umem);
74		− xsk->chunk_size = xsk_umem_get_chunk_size(umem);
	73	+ xsk->headroom = xsk_pool_get_headroom(pool);
	74	+ xsk->chunk_size = xsk_pool_get_chunk_size(pool);
75	75	}
76	76
77	77	static int mlx5e_xsk_enable_locked(struct mlx5e_priv *priv,
78		− struct xdp_umem *umem, u16 ix)
	78	+ struct xsk_buff_pool *pool, u16 ix)
79	79	{
80	80	struct mlx5e_params *params = &priv->channels.params;
81	81	struct mlx5e_xsk_param xsk;
82	82	struct mlx5e_channel *c;
83	83	int err;
84	84
85		− if (unlikely(mlx5e_xsk_get_umem(&priv->channels.params, &priv->xsk, ix)))
	85	+ if (unlikely(mlx5e_xsk_get_pool(&priv->channels.params, &priv->xsk, ix)))
86	86	return -EBUSY;
87	87
88		− if (unlikely(!mlx5e_xsk_is_umem_sane(umem)))
	88	+ if (unlikely(!mlx5e_xsk_is_pool_sane(pool)))
89	89	return -EINVAL;
90	90
91		− err = mlx5e_xsk_map_umem(priv, umem);
	91	+ err = mlx5e_xsk_map_pool(priv, pool);
92	92	if (unlikely(err))
93	93	return err;
94	94
95		− err = mlx5e_xsk_add_umem(&priv->xsk, umem, ix);
	95	+ err = mlx5e_xsk_add_pool(&priv->xsk, pool, ix);
96	96	if (unlikely(err))
97		− goto err_unmap_umem;
	97	+ goto err_unmap_pool;
98	98
99		− mlx5e_build_xsk_param(umem, &xsk);
	99	+ mlx5e_build_xsk_param(pool, &xsk);
100	100
101	101	if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
102	102	/* XSK objects will be created on open. */
@@ -112,9 +112,9 @@ static int mlx5e_xsk_enable_locked(struct mlx5e_priv *priv, · 9 unmodified lines
112	112
113	113	c = priv->channels.c[ix];
114	114
115		− err = mlx5e_open_xsk(priv, params, &xsk, umem, c);
	115	+ err = mlx5e_open_xsk(priv, params, &xsk, pool, c);
116	116	if (unlikely(err))
117		− goto err_remove_umem;
	117	+ goto err_remove_pool;
118	118
119	119	mlx5e_activate_xsk(c);
120	120
@@ -132,11 +132,11 @@ err_deactivate: · 11 unmodified lines
132	132	mlx5e_deactivate_xsk(c);
133	133	mlx5e_close_xsk(c);
134	134
135		−err_remove_umem:
136		− mlx5e_xsk_remove_umem(&priv->xsk, ix);
	135	+err_remove_pool:
	136	+ mlx5e_xsk_remove_pool(&priv->xsk, ix);
137	137
138		−err_unmap_umem:
139		− mlx5e_xsk_unmap_umem(priv, umem);
	138	+err_unmap_pool:
	139	+ mlx5e_xsk_unmap_pool(priv, pool);
140	140
141	141	return err;
142	142
@@ -146,7 +146,7 @@ validate_closed: · 3 unmodified lines
146	146	*/
147	147	if (!mlx5e_validate_xsk_param(params, &xsk, priv->mdev)) {
148	148	err = -EINVAL;
149		− goto err_remove_umem;
	149	+ goto err_remove_pool;
150	150	}
151	151
152	152	return 0;
@@ -154,45 +154,45 @@ validate_closed: · 1 unmodified line
154	154
155	155	static int mlx5e_xsk_disable_locked(struct mlx5e_priv *priv, u16 ix)
156	156	{
157		− struct xdp_umem *umem = mlx5e_xsk_get_umem(&priv->channels.params,
	157	+ struct xsk_buff_pool *pool = mlx5e_xsk_get_pool(&priv->channels.params,
158	158	&priv->xsk, ix);
159	159	struct mlx5e_channel *c;
160	160
161		− if (unlikely(!umem))
	161	+ if (unlikely(!pool))
162	162	return -EINVAL;
163	163
164	164	if (!test_bit(MLX5E_STATE_OPENED, &priv->state))
165		− goto remove_umem;
	165	+ goto remove_pool;
166	166
167	167	/* XSK RQ and SQ are only created if XDP program is set. */
168	168	if (!priv->channels.params.xdp_prog)
169		− goto remove_umem;
	169	+ goto remove_pool;
170	170
171	171	c = priv->channels.c[ix];
172	172	mlx5e_xsk_redirect_rqt_to_drop(priv, ix);
173	173	mlx5e_deactivate_xsk(c);
174	174	mlx5e_close_xsk(c);
175	175
176		−remove_umem:
177		− mlx5e_xsk_remove_umem(&priv->xsk, ix);
178		− mlx5e_xsk_unmap_umem(priv, umem);
	176	+remove_pool:
	177	+ mlx5e_xsk_remove_pool(&priv->xsk, ix);
	178	+ mlx5e_xsk_unmap_pool(priv, pool);
179	179
180	180	return 0;
181	181	}
182	182
183		−static int mlx5e_xsk_enable_umem(struct mlx5e_priv priv, struct xdp_umem umem,
	183	+static int mlx5e_xsk_enable_pool(struct mlx5e_priv priv, struct xsk_buff_pool pool,
184	184	u16 ix)
185	185	{
186	186	int err;
187	187
188	188	mutex_lock(&priv->state_lock);
189		− err = mlx5e_xsk_enable_locked(priv, umem, ix);
	189	+ err = mlx5e_xsk_enable_locked(priv, pool, ix);
190	190	mutex_unlock(&priv->state_lock);
191	191
192	192	return err;
193	193	}
194	194
195		−static int mlx5e_xsk_disable_umem(struct mlx5e_priv *priv, u16 ix)
	195	+static int mlx5e_xsk_disable_pool(struct mlx5e_priv *priv, u16 ix)
196	196	{
197	197	int err;
198	198
@@ -203,7 +203,7 @@ static int mlx5e_xsk_disable_umem(struct mlx5e_priv *priv, u16 ix) · 4 unmodified lines
203	203	return err;
204	204	}
205	205
206		−int mlx5e_xsk_setup_umem(struct net_device dev, struct xdp_umem umem, u16 qid)
	206	+int mlx5e_xsk_setup_pool(struct net_device dev, struct xsk_buff_pool pool, u16 qid)
207	207	{
208	208	struct mlx5e_priv *priv = netdev_priv(dev);
209	209	struct mlx5e_params *params = &priv->channels.params;
@@ -212,6 +212,6 @@ int mlx5e_xsk_setup_umem(struct net_device dev, struct xdp_umem umem, u16 qid) · 2 unmodified lines
212	212	if (unlikely(!mlx5e_qid_get_ch_if_in_group(params, qid, MLX5E_RQ_GROUP_XSK, &ix)))
213	213	return -EINVAL;
214	214
215		− return umem ? mlx5e_xsk_enable_umem(priv, umem, ix) :
216		− mlx5e_xsk_disable_umem(priv, ix);
	215	+ return pool ? mlx5e_xsk_enable_pool(priv, pool, ix) :
	216	+ mlx5e_xsk_disable_pool(priv, ix);
217	217	}

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/pool.h+27−0

@@ -0,0 +1,27 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
	2	+/* Copyright (c) 2019-2020, Mellanox Technologies inc. All rights reserved. */
	3	+
	4	+#ifndef __MLX5_EN_XSK_POOL_H__
	5	+#define __MLX5_EN_XSK_POOL_H__
	6	+
	7	+#include "en.h"
	8	+
	9	+static inline struct xsk_buff_pool mlx5e_xsk_get_pool(struct mlx5e_params params,
	10	+ struct mlx5e_xsk *xsk, u16 ix)
	11	+{
	12	+ if (!xsk \|\| !xsk->pools)
	13	+ return NULL;
	14	+
	15	+ if (unlikely(ix >= params->num_channels))
	16	+ return NULL;
	17	+
	18	+ return xsk->pools[ix];
	19	+}
	20	+
	21	+struct mlx5e_xsk_param;
	22	+void mlx5e_build_xsk_param(struct xsk_buff_pool pool, struct mlx5e_xsk_param xsk);
	23	+
	24	+/* .ndo_bpf callback. */
	25	+int mlx5e_xsk_setup_pool(struct net_device dev, struct xsk_buff_pool pool, u16 qid);
	26	+
	27	+#endif /* __MLX5_EN_XSK_POOL_H__ */

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c+2−2

@@ -48,7 +48,7 @@ struct sk_buff mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq rq, · 47 unmodified lines
48	48
49	49	xdp->data_end = xdp->data + cqe_bcnt32;
50	50	xdp_set_data_meta_invalid(xdp);
51		− xsk_buff_dma_sync_for_cpu(xdp);
	51	+ xsk_buff_dma_sync_for_cpu(xdp, rq->xsk_pool);
52	52	net_prefetch(xdp->data);
53	53
54	54	rcu_read_lock();
@@ -99,7 +99,7 @@ struct sk_buff mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq rq, · 44 unmodified lines
99	99
100	100	xdp->data_end = xdp->data + cqe_bcnt;
101	101	xdp_set_data_meta_invalid(xdp);
102		− xsk_buff_dma_sync_for_cpu(xdp);
	102	+ xsk_buff_dma_sync_for_cpu(xdp, rq->xsk_pool);
103	103	net_prefetch(xdp->data);
104	104
105	105	if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_RESP_SEND)) {

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h+5−5

@@ -19,10 +19,10 @@ struct sk_buff mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq rq, · 18 unmodified lines
19	19	struct mlx5e_wqe_frag_info *wi,
20	20	u32 cqe_bcnt);
21	21
22		−static inline int mlx5e_xsk_page_alloc_umem(struct mlx5e_rq *rq,
	22	+static inline int mlx5e_xsk_page_alloc_pool(struct mlx5e_rq *rq,
23	23	struct mlx5e_dma_info *dma_info)
24	24	{
25		− dma_info->xsk = xsk_buff_alloc(rq->umem);
	25	+ dma_info->xsk = xsk_buff_alloc(rq->xsk_pool);
26	26	if (!dma_info->xsk)
27	27	return -ENOMEM;
28	28
@@ -38,13 +38,13 @@ static inline int mlx5e_xsk_page_alloc_umem(struct mlx5e_rq *rq, · 9 unmodified lines
38	38
39	39	static inline bool mlx5e_xsk_update_rx_wakeup(struct mlx5e_rq *rq, bool alloc_err)
40	40	{
41		− if (!xsk_umem_uses_need_wakeup(rq->umem))
	41	+ if (!xsk_uses_need_wakeup(rq->xsk_pool))
42	42	return alloc_err;
43	43
44	44	if (unlikely(alloc_err))
45		− xsk_set_rx_need_wakeup(rq->umem);
	45	+ xsk_set_rx_need_wakeup(rq->xsk_pool);
46	46	else
47		− xsk_clear_rx_need_wakeup(rq->umem);
	47	+ xsk_clear_rx_need_wakeup(rq->xsk_pool);
48	48
49	49	return false;
50	50	}

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c+6−6

@@ -45,7 +45,7 @@ static void mlx5e_build_xsk_cparam(struct mlx5e_priv *priv, · 44 unmodified lines
45	45	}
46	46
47	47	int mlx5e_open_xsk(struct mlx5e_priv priv, struct mlx5e_params params,
48		− struct mlx5e_xsk_param xsk, struct xdp_umem umem,
	48	+ struct mlx5e_xsk_param xsk, struct xsk_buff_pool pool,
49	49	struct mlx5e_channel *c)
50	50	{
51	51	struct mlx5e_channel_param *cparam;
@@ -64,7 +64,7 @@ int mlx5e_open_xsk(struct mlx5e_priv priv, struct mlx5e_params params, · 12 unmodified lines
64	64	if (unlikely(err))
65	65	goto err_free_cparam;
66	66
67		− err = mlx5e_open_rq(c, params, &cparam->rq, xsk, umem, &c->xskrq);
	67	+ err = mlx5e_open_rq(c, params, &cparam->rq, xsk, pool, &c->xskrq);
68	68	if (unlikely(err))
69	69	goto err_close_rx_cq;
70	70
@@ -72,13 +72,13 @@ int mlx5e_open_xsk(struct mlx5e_priv priv, struct mlx5e_params params, · 1 unmodified line
72	72	if (unlikely(err))
73	73	goto err_close_rq;
74	74
75		− /* Create a separate SQ, so that when the UMEM is disabled, we could
	75	+ /* Create a separate SQ, so that when the buff pool is disabled, we could
76	76	* close this SQ safely and stop receiving CQEs. In other case, e.g., if
77		− * the XDPSQ was used instead, we might run into trouble when the UMEM
	77	+ * the XDPSQ was used instead, we might run into trouble when the buff pool
78	78	* is disabled and then reenabled, but the SQ continues receiving CQEs
79		− * from the old UMEM.
	79	+ * from the old buff pool.
80	80	*/
81		− err = mlx5e_open_xdpsq(c, params, &cparam->xdp_sq, umem, &c->xsksq, true);
	81	+ err = mlx5e_open_xdpsq(c, params, &cparam->xdp_sq, pool, &c->xsksq, true);
82	82	if (unlikely(err))
83	83	goto err_close_tx_cq;
84	84

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.h+1−1

@@ -12,7 +12,7 @@ bool mlx5e_validate_xsk_param(struct mlx5e_params *params, · 11 unmodified lines
12	12	struct mlx5e_xsk_param *xsk,
13	13	struct mlx5_core_dev *mdev);
14	14	int mlx5e_open_xsk(struct mlx5e_priv priv, struct mlx5e_params params,
15		− struct mlx5e_xsk_param xsk, struct xdp_umem umem,
	15	+ struct mlx5e_xsk_param xsk, struct xsk_buff_pool pool,
16	16	struct mlx5e_channel *c);
17	17	void mlx5e_close_xsk(struct mlx5e_channel *c);
18	18	void mlx5e_activate_xsk(struct mlx5e_channel *c);

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c+7−7

@@ -2,7 +2,7 @@ · 1 unmodified line
2	2	/* Copyright (c) 2019 Mellanox Technologies. */
3	3
4	4	#include "tx.h"
5		−#include "umem.h"
	5	+#include "pool.h"
6	6	#include "en/xdp.h"
7	7	#include "en/params.h"
8	8	#include <net/xdp_sock_drv.h>
@@ -66,7 +66,7 @@ static void mlx5e_xsk_tx_post_err(struct mlx5e_xdpsq *sq, · 57 unmodified lines
66	66
67	67	bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget)
68	68	{
69		− struct xdp_umem *umem = sq->umem;
	69	+ struct xsk_buff_pool *pool = sq->xsk_pool;
70	70	struct mlx5e_xdp_info xdpi;
71	71	struct mlx5e_xdp_xmit_data xdptxd;
72	72	bool work_done = true;
@@ -87,7 +87,7 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget) · 14 unmodified lines
87	87	break;
88	88	}
89	89
90		− if (!xsk_umem_consume_tx(umem, &desc)) {
	90	+ if (!xsk_tx_peek_desc(pool, &desc)) {
91	91	/* TX will get stuck until something wakes it up by
92	92	* triggering NAPI. Currently it's expected that the
93	93	* application calls sendto() if there are consumed, but
@@ -96,11 +96,11 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget) · 2 unmodified lines
96	96	break;
97	97	}
98	98
99		− xdptxd.dma_addr = xsk_buff_raw_get_dma(umem, desc.addr);
100		− xdptxd.data = xsk_buff_raw_get_data(umem, desc.addr);
	99	+ xdptxd.dma_addr = xsk_buff_raw_get_dma(pool, desc.addr);
	100	+ xdptxd.data = xsk_buff_raw_get_data(pool, desc.addr);
101	101	xdptxd.len = desc.len;
102	102
103		− xsk_buff_raw_dma_sync_for_device(umem, xdptxd.dma_addr, xdptxd.len);
	103	+ xsk_buff_raw_dma_sync_for_device(pool, xdptxd.dma_addr, xdptxd.len);
104	104
105	105	ret = INDIRECT_CALL_2(sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe,
106	106	mlx5e_xmit_xdp_frame, sq, &xdptxd, &xdpi, check_result);
@@ -119,7 +119,7 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget) · 12 unmodified lines
119	119	mlx5e_xdp_mpwqe_complete(sq);
120	120	mlx5e_xmit_xdp_doorbell(sq);
121	121
122		− xsk_umem_consume_tx_done(umem);
	122	+ xsk_tx_release(pool);
123	123	}
124	124
125	125	return !(budget && work_done);

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.h+3−3

@@ -15,13 +15,13 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget); · 14 unmodified lines
15	15
16	16	static inline void mlx5e_xsk_update_tx_wakeup(struct mlx5e_xdpsq *sq)
17	17	{
18		− if (!xsk_umem_uses_need_wakeup(sq->umem))
	18	+ if (!xsk_uses_need_wakeup(sq->xsk_pool))
19	19	return;
20	20
21	21	if (sq->pc != sq->cc)
22		− xsk_clear_tx_need_wakeup(sq->umem);
	22	+ xsk_clear_tx_need_wakeup(sq->xsk_pool);
23	23	else
24		− xsk_set_tx_need_wakeup(sq->umem);
	24	+ xsk_set_tx_need_wakeup(sq->xsk_pool);
25	25	}
26	26
27	27	#endif /* __MLX5_EN_XSK_TX_H__ */

drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.h+0−29

@@ -1,29 +0,0 @@
1		−/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
2		−/* Copyright (c) 2019 Mellanox Technologies. */
3		−
4		−#ifndef __MLX5_EN_XSK_UMEM_H__
5		−#define __MLX5_EN_XSK_UMEM_H__
6		−
7		−#include "en.h"
8		−
9		−static inline struct xdp_umem mlx5e_xsk_get_umem(struct mlx5e_params params,
10		− struct mlx5e_xsk *xsk, u16 ix)
11		−{
12		− if (!xsk \|\| !xsk->umems)
13		− return NULL;
14		−
15		− if (unlikely(ix >= params->num_channels))
16		− return NULL;
17		−
18		− return xsk->umems[ix];
19		−}
20		−
21		−struct mlx5e_xsk_param;
22		−void mlx5e_build_xsk_param(struct xdp_umem umem, struct mlx5e_xsk_param xsk);
23		−
24		−/* .ndo_bpf callback. */
25		−int mlx5e_xsk_setup_umem(struct net_device dev, struct xdp_umem umem, u16 qid);
26		−
27		−int mlx5e_xsk_resize_reuseq(struct xdp_umem *umem, u32 nentries);
28		−
29		−#endif /* __MLX5_EN_XSK_UMEM_H__ */

drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c+1−1

@@ -32,7 +32,7 @@ · 31 unmodified lines
32	32
33	33	#include "en.h"
34	34	#include "en/port.h"
35		−#include "en/xsk/umem.h"
	35	+#include "en/xsk/pool.h"
36	36	#include "lib/clock.h"
37	37
38	38	void mlx5e_ethtool_get_drvinfo(struct mlx5e_priv *priv,

drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.c+1−1

@@ -33,7 +33,7 @@ · 32 unmodified lines
33	33	#include <linux/mlx5/fs.h>
34	34	#include "en.h"
35	35	#include "en/params.h"
36		−#include "en/xsk/umem.h"
	36	+#include "en/xsk/pool.h"
37	37
38	38	struct mlx5e_ethtool_rule {
39	39	struct list_head list;

drivers/net/ethernet/mellanox/mlx5/core/en_main.c+25−24

@@ -57,7 +57,7 @@ · 56 unmodified lines
57	57	#include "en/monitor_stats.h"
58	58	#include "en/health.h"
59	59	#include "en/params.h"
60		−#include "en/xsk/umem.h"
	60	+#include "en/xsk/pool.h"
61	61	#include "en/xsk/setup.h"
62	62	#include "en/xsk/rx.h"
63	63	#include "en/xsk/tx.h"
@@ -363,7 +363,7 @@ static void mlx5e_rq_err_cqe_work(struct work_struct *recover_work) · 299 unmodified lines
363	363	static int mlx5e_alloc_rq(struct mlx5e_channel *c,
364	364	struct mlx5e_params *params,
365	365	struct mlx5e_xsk_param *xsk,
366		− struct xdp_umem *umem,
	366	+ struct xsk_buff_pool *xsk_pool,
367	367	struct mlx5e_rq_param *rqp,
368	368	struct mlx5e_rq *rq)
369	369	{
@@ -389,9 +389,9 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, · 19 unmodified lines
389	389	rq->mdev = mdev;
390	390	rq->hw_mtu = MLX5E_SW2HW_MTU(params, params->sw_mtu);
391	391	rq->xdpsq = &c->rq_xdpsq;
392		− rq->umem = umem;
	392	+ rq->xsk_pool = xsk_pool;
393	393
394		− if (rq->umem)
	394	+ if (rq->xsk_pool)
395	395	rq->stats = &c->priv->channel_stats[c->ix].xskrq;
396	396	else
397	397	rq->stats = &c->priv->channel_stats[c->ix].rq;
@@ -477,7 +477,7 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, · 79 unmodified lines
477	477	if (xsk) {
478	478	err = xdp_rxq_info_reg_mem_model(&rq->xdp_rxq,
479	479	MEM_TYPE_XSK_BUFF_POOL, NULL);
480		− xsk_buff_set_rxq_info(rq->umem, &rq->xdp_rxq);
	480	+ xsk_pool_set_rxq_info(rq->xsk_pool, &rq->xdp_rxq);
481	481	} else {
482	482	/* Create a page_pool and register it with rxq */
483	483	pp_params.order = 0;
@@ -816,11 +816,11 @@ void mlx5e_free_rx_descs(struct mlx5e_rq *rq) · 332 unmodified lines
816	816
817	817	int mlx5e_open_rq(struct mlx5e_channel c, struct mlx5e_params params,
818	818	struct mlx5e_rq_param param, struct mlx5e_xsk_param xsk,
819		− struct xdp_umem umem, struct mlx5e_rq rq)
	819	+ struct xsk_buff_pool xsk_pool, struct mlx5e_rq rq)
820	820	{
821	821	int err;
822	822
823		− err = mlx5e_alloc_rq(c, params, xsk, umem, param, rq);
	823	+ err = mlx5e_alloc_rq(c, params, xsk, xsk_pool, param, rq);
824	824	if (err)
825	825	return err;
826	826
@@ -925,7 +925,7 @@ static int mlx5e_alloc_xdpsq_db(struct mlx5e_xdpsq *sq, int numa) · 98 unmodified lines
925	925
926	926	static int mlx5e_alloc_xdpsq(struct mlx5e_channel *c,
927	927	struct mlx5e_params *params,
928		− struct xdp_umem *umem,
	928	+ struct xsk_buff_pool *xsk_pool,
929	929	struct mlx5e_sq_param *param,
930	930	struct mlx5e_xdpsq *sq,
931	931	bool is_redirect)
@@ -941,9 +941,9 @@ static int mlx5e_alloc_xdpsq(struct mlx5e_channel *c, · 9 unmodified lines
941	941	sq->uar_map = mdev->mlx5e_res.bfreg.map;
942	942	sq->min_inline_mode = params->tx_min_inline_mode;
943	943	sq->hw_mtu = MLX5E_SW2HW_MTU(params, params->sw_mtu);
944		− sq->umem = umem;
	944	+ sq->xsk_pool = xsk_pool;
945	945
946		− sq->stats = sq->umem ?
	946	+ sq->stats = sq->xsk_pool ?
947	947	&c->priv->channel_stats[c->ix].xsksq :
948	948	is_redirect ?
949	949	&c->priv->channel_stats[c->ix].xdpsq :
@@ -1408,13 +1408,13 @@ void mlx5e_close_icosq(struct mlx5e_icosq *sq) · 458 unmodified lines
1408	1408	}
1409	1409
1410	1410	int mlx5e_open_xdpsq(struct mlx5e_channel c, struct mlx5e_params params,
1411		− struct mlx5e_sq_param param, struct xdp_umem umem,
	1411	+ struct mlx5e_sq_param param, struct xsk_buff_pool xsk_pool,
1412	1412	struct mlx5e_xdpsq *sq, bool is_redirect)
1413	1413	{
1414	1414	struct mlx5e_create_sq_param csp = {};
1415	1415	int err;
1416	1416
1417		− err = mlx5e_alloc_xdpsq(c, params, umem, param, sq, is_redirect);
	1417	+ err = mlx5e_alloc_xdpsq(c, params, xsk_pool, param, sq, is_redirect);
1418	1418	if (err)
1419	1419	return err;
1420	1420
@@ -1907,7 +1907,7 @@ static u8 mlx5e_enumerate_lag_port(struct mlx5_core_dev *mdev, int ix) · 486 unmodified lines
1907	1907	static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
1908	1908	struct mlx5e_params *params,
1909	1909	struct mlx5e_channel_param *cparam,
1910		− struct xdp_umem *umem,
	1910	+ struct xsk_buff_pool *xsk_pool,
1911	1911	struct mlx5e_channel **cp)
1912	1912	{
1913	1913	int cpu = cpumask_first(mlx5_comp_irq_get_affinity_mask(priv->mdev, ix));
@@ -1946,9 +1946,9 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix, · 32 unmodified lines
1946	1946	if (unlikely(err))
1947	1947	goto err_napi_del;
1948	1948
1949		− if (umem) {
1950		− mlx5e_build_xsk_param(umem, &xsk);
1951		− err = mlx5e_open_xsk(priv, params, &xsk, umem, c);
	1949	+ if (xsk_pool) {
	1950	+ mlx5e_build_xsk_param(xsk_pool, &xsk);
	1951	+ err = mlx5e_open_xsk(priv, params, &xsk, xsk_pool, c);
1952	1952	if (unlikely(err))
1953	1953	goto err_close_queues;
1954	1954	}
@@ -2309,12 +2309,12 @@ int mlx5e_open_channels(struct mlx5e_priv *priv, · 354 unmodified lines
2309	2309
2310	2310	mlx5e_build_channel_param(priv, &chs->params, cparam);
2311	2311	for (i = 0; i < chs->num; i++) {
2312		− struct xdp_umem *umem = NULL;
	2312	+ struct xsk_buff_pool *xsk_pool = NULL;
2313	2313
2314	2314	if (chs->params.xdp_prog)
2315		− umem = mlx5e_xsk_get_umem(&chs->params, chs->params.xsk, i);
	2315	+ xsk_pool = mlx5e_xsk_get_pool(&chs->params, chs->params.xsk, i);
2316	2316
2317		− err = mlx5e_open_channel(priv, i, &chs->params, cparam, umem, &chs->c[i]);
	2317	+ err = mlx5e_open_channel(priv, i, &chs->params, cparam, xsk_pool, &chs->c[i]);
2318	2318	if (err)
2319	2319	goto err_close_channels;
2320	2320	}
@@ -3892,13 +3892,14 @@ static bool mlx5e_xsk_validate_mtu(struct net_device *netdev, · 1571 unmodified lines
3892	3892	u16 ix;
3893	3893
3894	3894	for (ix = 0; ix < chs->params.num_channels; ix++) {
3895		− struct xdp_umem *umem = mlx5e_xsk_get_umem(&chs->params, chs->params.xsk, ix);
	3895	+ struct xsk_buff_pool *xsk_pool =
	3896	+ mlx5e_xsk_get_pool(&chs->params, chs->params.xsk, ix);
3896	3897	struct mlx5e_xsk_param xsk;
3897	3898
3898		− if (!umem)
	3899	+ if (!xsk_pool)
3899	3900	continue;
3900	3901
3901		− mlx5e_build_xsk_param(umem, &xsk);
	3902	+ mlx5e_build_xsk_param(xsk_pool, &xsk);
3902	3903
3903	3904	if (!mlx5e_validate_xsk_param(new_params, &xsk, mdev)) {
3904	3905	u32 hr = mlx5e_get_linear_rq_headroom(new_params, &xsk);
@@ -4423,8 +4424,8 @@ static int mlx5e_xdp(struct net_device dev, struct netdev_bpf xdp) · 518 unmodified lines
4423	4424	switch (xdp->command) {
4424	4425	case XDP_SETUP_PROG:
4425	4426	return mlx5e_xdp_set(dev, xdp->prog);
4426		− case XDP_SETUP_XSK_UMEM:
4427		− return mlx5e_xsk_setup_umem(dev, xdp->xsk.umem,
	4427	+ case XDP_SETUP_XSK_POOL:
	4428	+ return mlx5e_xsk_setup_pool(dev, xdp->xsk.pool,
4428	4429	xdp->xsk.queue_id);
4429	4430	default:
4430	4431	return -EINVAL;

drivers/net/ethernet/mellanox/mlx5/core/en_rx.c+8−8

@@ -280,8 +280,8 @@ static inline int mlx5e_page_alloc_pool(struct mlx5e_rq *rq, · 279 unmodified lines
280	280	static inline int mlx5e_page_alloc(struct mlx5e_rq *rq,
281	281	struct mlx5e_dma_info *dma_info)
282	282	{
283		− if (rq->umem)
284		− return mlx5e_xsk_page_alloc_umem(rq, dma_info);
	283	+ if (rq->xsk_pool)
	284	+ return mlx5e_xsk_page_alloc_pool(rq, dma_info);
285	285	else
286	286	return mlx5e_page_alloc_pool(rq, dma_info);
287	287	}
@@ -312,7 +312,7 @@ static inline void mlx5e_page_release(struct mlx5e_rq *rq, · 24 unmodified lines
312	312	struct mlx5e_dma_info *dma_info,
313	313	bool recycle)
314	314	{
315		− if (rq->umem)
	315	+ if (rq->xsk_pool)
316	316	/* The `recycle` parameter is ignored, and the page is always
317	317	* put into the Reuse Ring, because there is no way to return
318	318	* the page to the userspace when the interface goes down.
@@ -399,14 +399,14 @@ static int mlx5e_alloc_rx_wqes(struct mlx5e_rq *rq, u16 ix, u8 wqe_bulk) · 80 unmodified lines
399	399	int err;
400	400	int i;
401	401
402		− if (rq->umem) {
	402	+ if (rq->xsk_pool) {
403	403	int pages_desired = wqe_bulk << rq->wqe.info.log_num_frags;
404	404
405	405	/* Check in advance that we have enough frames, instead of
406	406	* allocating one-by-one, failing and moving frames to the
407	407	* Reuse Ring.
408	408	*/
409		− if (unlikely(!xsk_buff_can_alloc(rq->umem, pages_desired)))
	409	+ if (unlikely(!xsk_buff_can_alloc(rq->xsk_pool, pages_desired)))
410	410	return -ENOMEM;
411	411	}
412	412
@@ -504,8 +504,8 @@ static int mlx5e_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix) · 91 unmodified lines
504	504	/* Check in advance that we have enough frames, instead of allocating
505	505	* one-by-one, failing and moving frames to the Reuse Ring.
506	506	*/
507		− if (rq->umem &&
508		− unlikely(!xsk_buff_can_alloc(rq->umem, MLX5_MPWRQ_PAGES_PER_WQE))) {
	507	+ if (rq->xsk_pool &&
	508	+ unlikely(!xsk_buff_can_alloc(rq->xsk_pool, MLX5_MPWRQ_PAGES_PER_WQE))) {
509	509	err = -ENOMEM;
510	510	goto err;
511	511	}
@@ -753,7 +753,7 @@ INDIRECT_CALLABLE_SCOPE bool mlx5e_post_rx_mpwqes(struct mlx5e_rq *rq) · 241 unmodified lines
753	753	* the driver when it refills the Fill Ring.
754	754	* 2. Otherwise, busy poll by rescheduling the NAPI poll.
755	755	*/
756		− if (unlikely(alloc_err == -ENOMEM && rq->umem))
	756	+ if (unlikely(alloc_err == -ENOMEM && rq->xsk_pool))
757	757	return true;
758	758
759	759	return false;

drivers/net/tun.c+0−18

@@ -219,24 +219,6 @@ struct veth { · 218 unmodified lines
219	219	__be16 h_vlan_TCI;
220	220	};
221	221
222		−bool tun_is_xdp_frame(void *ptr)
223		−{
224		− return (unsigned long)ptr & TUN_XDP_FLAG;
225		−}
226		−EXPORT_SYMBOL(tun_is_xdp_frame);
227		−
228		−void tun_xdp_to_ptr(void ptr)
229		−{
230		− return (void *)((unsigned long)ptr \| TUN_XDP_FLAG);
231		−}
232		−EXPORT_SYMBOL(tun_xdp_to_ptr);
233		−
234		−void tun_ptr_to_xdp(void ptr)
235		−{
236		− return (void *)((unsigned long)ptr & ~TUN_XDP_FLAG);
237		−}
238		−EXPORT_SYMBOL(tun_ptr_to_xdp);
239		−
240	222	static int tun_napi_receive(struct napi_struct *napi, int budget)
241	223	{
242	224	struct tun_file *tfile = container_of(napi, struct tun_file, napi);

drivers/net/veth.c+3−3

@@ -234,14 +234,14 @@ static bool veth_is_xdp_frame(void *ptr) · 233 unmodified lines
234	234	return (unsigned long)ptr & VETH_XDP_FLAG;
235	235	}
236	236
237		−static void veth_ptr_to_xdp(void ptr)
	237	+static struct xdp_frame veth_ptr_to_xdp(void ptr)
238	238	{
239	239	return (void *)((unsigned long)ptr & ~VETH_XDP_FLAG);
240	240	}
241	241
242		−static void veth_xdp_to_ptr(void ptr)
	242	+static void veth_xdp_to_ptr(struct xdp_frame xdp)
243	243	{
244		− return (void *)((unsigned long)ptr \| VETH_XDP_FLAG);
	244	+ return (void *)((unsigned long)xdp \| VETH_XDP_FLAG);
245	245	}
246	246
247	247	static void veth_ptr_free(void *ptr)

include/linux/bpf-cgroup.h+25−0

@@ -279,6 +279,31 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map map, void key, · 278 unmodified lines
279	279	#define BPF_CGROUP_RUN_PROG_UDP6_RECVMSG_LOCK(sk, uaddr) \
280	280	BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, BPF_CGROUP_UDP6_RECVMSG, NULL)
281	281
	282	+/* The SOCK_OPS"_SK" macro should be used when sock_ops->sk is not a
	283	+ * fullsock and its parent fullsock cannot be traced by
	284	+ * sk_to_full_sk().
	285	+ *
	286	+ * e.g. sock_ops->sk is a request_sock and it is under syncookie mode.
	287	+ * Its listener-sk is not attached to the rsk_listener.
	288	+ * In this case, the caller holds the listener-sk (unlocked),
	289	+ * set its sock_ops->sk to req_sk, and call this SOCK_OPS"_SK" with
	290	+ * the listener-sk such that the cgroup-bpf-progs of the
	291	+ * listener-sk will be run.
	292	+ *
	293	+ * Regardless of syncookie mode or not,
	294	+ * calling bpf_setsockopt on listener-sk will not make sense anyway,
	295	+ * so passing 'sock_ops->sk == req_sk' to the bpf prog is appropriate here.
	296	+ */
	297	+#define BPF_CGROUP_RUN_PROG_SOCK_OPS_SK(sock_ops, sk) \
	298	+({ \
	299	+ int __ret = 0; \
	300	+ if (cgroup_bpf_enabled) \
	301	+ __ret = __cgroup_bpf_run_filter_sock_ops(sk, \
	302	+ sock_ops, \
	303	+ BPF_CGROUP_SOCK_OPS); \
	304	+ __ret; \
	305	+})
	306	+
282	307	#define BPF_CGROUP_RUN_PROG_SOCK_OPS(sock_ops) \
283	308	({ \
284	309	int __ret = 0; \

include/linux/bpf.h+52−0

@@ -34,6 +34,8 @@ struct btf_type; · 33 unmodified lines
34	34	struct exception_table_entry;
35	35	struct seq_operations;
36	36	struct bpf_iter_aux_info;
	37	+struct bpf_local_storage;
	38	+struct bpf_local_storage_map;
37	39
38	40	extern struct idr btf_idr;
39	41	extern spinlock_t btf_idr_lock;
@@ -104,6 +106,25 @@ struct bpf_map_ops { · 64 unmodified lines
104	106	__poll_t (map_poll)(struct bpf_map map, struct file *filp,
105	107	struct poll_table_struct *pts);
106	108
	109	+ /* Functions called by bpf_local_storage maps */
	110	+ int (map_local_storage_charge)(struct bpf_local_storage_map smap,
	111	+ void *owner, u32 size);
	112	+ void (map_local_storage_uncharge)(struct bpf_local_storage_map smap,
	113	+ void *owner, u32 size);
	114	+ struct bpf_local_storage __rcu ** (map_owner_storage_ptr)(void owner);
	115	+
	116	+ /* map_meta_equal must be implemented for maps that can be
	117	+ * used as an inner map. It is a runtime check to ensure
	118	+ * an inner map can be inserted to an outer map.
	119	+ *
	120	+ * Some properties of the inner map has been used during the
	121	+ * verification time. When inserting an inner map at the runtime,
	122	+ * map_meta_equal has to ensure the inserting map has the same
	123	+ * properties that the verifier has used earlier.
	124	+ */
	125	+ bool (map_meta_equal)(const struct bpf_map meta0,
	126	+ const struct bpf_map *meta1);
	127	+
107	128	/* BTF name and id of struct allocated by map_alloc */
108	129	const char * const map_btf_name;
109	130	int *map_btf_id;
@@ -227,6 +248,9 @@ int map_check_no_btf(const struct bpf_map *map, · 117 unmodified lines
227	248	const struct btf_type *key_type,
228	249	const struct btf_type *value_type);
229	250
	251	+bool bpf_map_meta_equal(const struct bpf_map *meta0,
	252	+ const struct bpf_map *meta1);
	253	+
230	254	extern const struct bpf_map_ops bpf_map_offload_ops;
231	255
232	256	/* function argument constraints */
@@ -309,6 +333,7 @@ struct bpf_func_proto { · 76 unmodified lines
309	333	* for this argument.
310	334	*/
311	335	int ret_btf_id; / return value btf_id */
	336	+ bool (allowed)(const struct bpf_prog prog);
312	337	};
313	338
314	339	/* bpf_context is intentionally undefined structure. Pointer to bpf_context is
@@ -514,6 +539,8 @@ int arch_prepare_bpf_trampoline(void image, void image_end, · 199 unmodified lines
514	539	/* these two functions are called from generated trampoline */
515	540	u64 notrace __bpf_prog_enter(void);
516	541	void notrace __bpf_prog_exit(struct bpf_prog *prog, u64 start);
	542	+void notrace __bpf_prog_enter_sleepable(void);
	543	+void notrace __bpf_prog_exit_sleepable(void);
517	544
518	545	struct bpf_ksym {
519	546	unsigned long start;
@@ -709,6 +736,7 @@ struct bpf_prog_aux { · 189 unmodified lines
709	736	bool offload_requested;
710	737	bool attach_btf_trace; /* true if attaching to BTF-enabled raw tp */
711	738	bool func_proto_unreliable;
	739	+ bool sleepable;
712	740	enum bpf_tramp_prog_type trampoline_prog_type;
713	741	struct bpf_trampoline *trampoline;
714	742	struct hlist_node tramp_hlist;
@@ -1218,12 +1246,18 @@ typedef int (bpf_iter_attach_target_t)(struct bpf_prog prog, · 503 unmodified lines
1218	1246	union bpf_iter_link_info *linfo,
1219	1247	struct bpf_iter_aux_info *aux);
1220	1248	typedef void (bpf_iter_detach_target_t)(struct bpf_iter_aux_info aux);
	1249	+typedef void (bpf_iter_show_fdinfo_t) (const struct bpf_iter_aux_info aux,
	1250	+ struct seq_file *seq);
	1251	+typedef int (bpf_iter_fill_link_info_t)(const struct bpf_iter_aux_info aux,
	1252	+ struct bpf_link_info *info);
1221	1253
1222	1254	#define BPF_ITER_CTX_ARG_MAX 2
1223	1255	struct bpf_iter_reg {
1224	1256	const char *target;
1225	1257	bpf_iter_attach_target_t attach_target;
1226	1258	bpf_iter_detach_target_t detach_target;
	1259	+ bpf_iter_show_fdinfo_t show_fdinfo;
	1260	+ bpf_iter_fill_link_info_t fill_link_info;
1227	1261	u32 ctx_arg_info_size;
1228	1262	struct bpf_ctx_arg_aux ctx_arg_info[BPF_ITER_CTX_ARG_MAX];
1229	1263	const struct bpf_iter_seq_info *seq_info;
@@ -1250,6 +1284,10 @@ int bpf_iter_new_fd(struct bpf_link *link); · 20 unmodified lines
1250	1284	bool bpf_link_is_iter(struct bpf_link *link);
1251	1285	struct bpf_prog bpf_iter_get_info(struct bpf_iter_meta meta, bool in_stop);
1252	1286	int bpf_iter_run_prog(struct bpf_prog prog, void ctx);
	1287	+void bpf_iter_map_show_fdinfo(const struct bpf_iter_aux_info *aux,
	1288	+ struct seq_file *seq);
	1289	+int bpf_iter_map_fill_link_info(const struct bpf_iter_aux_info *aux,
	1290	+ struct bpf_link_info *info);
1253	1291
1254	1292	int bpf_percpu_hash_copy(struct bpf_map map, void key, void *value);
1255	1293	int bpf_percpu_array_copy(struct bpf_map map, void key, void *value);
@@ -1340,6 +1378,8 @@ int btf_struct_access(struct bpf_verifier_log *log, · 84 unmodified lines
1340	1378	const struct btf_type *t, int off, int size,
1341	1379	enum bpf_access_type atype,
1342	1380	u32 *next_btf_id);
	1381	+bool btf_struct_ids_match(struct bpf_verifier_log *log,
	1382	+ int off, u32 id, u32 need_type_id);
1343	1383	int btf_resolve_helper_id(struct bpf_verifier_log *log,
1344	1384	const struct bpf_func_proto *fn, int);
1345	1385
@@ -1358,6 +1398,7 @@ int btf_check_type_match(struct bpf_verifier_env env, struct bpf_prog prog, · 12 unmodified lines
1358	1398	struct btf btf, const struct btf_type t);
1359	1399
1360	1400	struct bpf_prog *bpf_prog_by_id(u32 id);
	1401	+struct bpf_link *bpf_link_by_id(u32 id);
1361	1402
1362	1403	const struct bpf_func_proto *bpf_base_func_proto(enum bpf_func_id func_id);
1363	1404	#else /* !CONFIG_BPF_SYSCALL */
@@ -1637,6 +1678,7 @@ int sock_map_prog_update(struct bpf_map map, struct bpf_prog prog, · 273 unmodified lines
1637	1678	struct bpf_prog *old, u32 which);
1638	1679	int sock_map_get_from_fd(const union bpf_attr attr, struct bpf_prog prog);
1639	1680	int sock_map_prog_detach(const union bpf_attr *attr, enum bpf_prog_type ptype);
	1681	+int sock_map_update_elem_sys(struct bpf_map map, void key, void *value, u64 flags);
1640	1682	void sock_map_unhash(struct sock *sk);
1641	1683	void sock_map_close(struct sock *sk, long timeout);
1642	1684	#else
@@ -1658,6 +1700,12 @@ static inline int sock_map_prog_detach(const union bpf_attr *attr, · 15 unmodified lines
1658	1700	{
1659	1701	return -EOPNOTSUPP;
1660	1702	}
	1703	+
	1704	+static inline int sock_map_update_elem_sys(struct bpf_map map, void key, void *value,
	1705	+ u64 flags)
	1706	+{
	1707	+ return -EOPNOTSUPP;
	1708	+}
1661	1709	#endif /* CONFIG_BPF_STREAM_PARSER */
1662	1710
1663	1711	#if defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL)
@@ -1736,6 +1784,7 @@ extern const struct bpf_func_proto bpf_skc_to_tcp_sock_proto; · 72 unmodified lines
1736	1784	extern const struct bpf_func_proto bpf_skc_to_tcp_timewait_sock_proto;
1737	1785	extern const struct bpf_func_proto bpf_skc_to_tcp_request_sock_proto;
1738	1786	extern const struct bpf_func_proto bpf_skc_to_udp6_sock_proto;
	1787	+extern const struct bpf_func_proto bpf_copy_from_user_proto;
1739	1788
1740	1789	const struct bpf_func_proto *bpf_tracing_func_proto(
1741	1790	enum bpf_func_id func_id, const struct bpf_prog *prog);
@@ -1850,4 +1899,7 @@ enum bpf_text_poke_type { · 108 unmodified lines
1850	1899	int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,
1851	1900	void addr1, void addr2);
1852	1901
	1902	+struct btf_id_set;
	1903	+bool btf_id_set_contains(struct btf_id_set *set, u32 id);
	1904	+
1853	1905	#endif /* _LINUX_BPF_H */

include/linux/bpf_local_storage.h+163−0

@@ -0,0 +1,163 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 */
	2	+/*
	3	+ * Copyright (c) 2019 Facebook
	4	+ * Copyright 2020 Google LLC.
	5	+ */
	6	+
	7	+#ifndef _BPF_LOCAL_STORAGE_H
	8	+#define _BPF_LOCAL_STORAGE_H
	9	+
	10	+#include <linux/bpf.h>
	11	+#include <linux/rculist.h>
	12	+#include <linux/list.h>
	13	+#include <linux/hash.h>
	14	+#include <linux/types.h>
	15	+#include <uapi/linux/btf.h>
	16	+
	17	+#define BPF_LOCAL_STORAGE_CACHE_SIZE 16
	18	+
	19	+struct bpf_local_storage_map_bucket {
	20	+ struct hlist_head list;
	21	+ raw_spinlock_t lock;
	22	+};
	23	+
	24	+/* Thp map is not the primary owner of a bpf_local_storage_elem.
	25	+ * Instead, the container object (eg. sk->sk_bpf_storage) is.
	26	+ *
	27	+ * The map (bpf_local_storage_map) is for two purposes
	28	+ * 1. Define the size of the "local storage". It is
	29	+ * the map's value_size.
	30	+ *
	31	+ * 2. Maintain a list to keep track of all elems such
	32	+ * that they can be cleaned up during the map destruction.
	33	+ *
	34	+ * When a bpf local storage is being looked up for a
	35	+ * particular object, the "bpf_map" pointer is actually used
	36	+ * as the "key" to search in the list of elem in
	37	+ * the respective bpf_local_storage owned by the object.
	38	+ *
	39	+ * e.g. sk->sk_bpf_storage is the mini-map with the "bpf_map" pointer
	40	+ * as the searching key.
	41	+ */
	42	+struct bpf_local_storage_map {
	43	+ struct bpf_map map;
	44	+ /* Lookup elem does not require accessing the map.
	45	+ *
	46	+ * Updating/Deleting requires a bucket lock to
	47	+ * link/unlink the elem from the map. Having
	48	+ * multiple buckets to improve contention.
	49	+ */
	50	+ struct bpf_local_storage_map_bucket *buckets;
	51	+ u32 bucket_log;
	52	+ u16 elem_size;
	53	+ u16 cache_idx;
	54	+};
	55	+
	56	+struct bpf_local_storage_data {
	57	+ /* smap is used as the searching key when looking up
	58	+ * from the object's bpf_local_storage.
	59	+ *
	60	+ * Put it in the same cacheline as the data to minimize
	61	+ * the number of cachelines access during the cache hit case.
	62	+ */
	63	+ struct bpf_local_storage_map __rcu *smap;
	64	+ u8 data[] __aligned(8);
	65	+};
	66	+
	67	+/* Linked to bpf_local_storage and bpf_local_storage_map */
	68	+struct bpf_local_storage_elem {
	69	+ struct hlist_node map_node; /* Linked to bpf_local_storage_map */
	70	+ struct hlist_node snode; /* Linked to bpf_local_storage */
	71	+ struct bpf_local_storage __rcu *local_storage;
	72	+ struct rcu_head rcu;
	73	+ /* 8 bytes hole */
	74	+ /* The data is stored in aother cacheline to minimize
	75	+ * the number of cachelines access during a cache hit.
	76	+ */
	77	+ struct bpf_local_storage_data sdata ____cacheline_aligned;
	78	+};
	79	+
	80	+struct bpf_local_storage {
	81	+ struct bpf_local_storage_data __rcu *cache[BPF_LOCAL_STORAGE_CACHE_SIZE];
	82	+ struct hlist_head list; /* List of bpf_local_storage_elem */
	83	+ void owner; / The object that owns the above "list" of
	84	+ * bpf_local_storage_elem.
	85	+ */
	86	+ struct rcu_head rcu;
	87	+ raw_spinlock_t lock; /* Protect adding/removing from the "list" */
	88	+};
	89	+
	90	+/* U16_MAX is much more than enough for sk local storage
	91	+ * considering a tcp_sock is ~2k.
	92	+ */
	93	+#define BPF_LOCAL_STORAGE_MAX_VALUE_SIZE \
	94	+ min_t(u32, \
	95	+ (KMALLOC_MAX_SIZE - MAX_BPF_STACK - \
	96	+ sizeof(struct bpf_local_storage_elem)), \
	97	+ (U16_MAX - sizeof(struct bpf_local_storage_elem)))
	98	+
	99	+#define SELEM(_SDATA) \
	100	+ container_of((_SDATA), struct bpf_local_storage_elem, sdata)
	101	+#define SDATA(_SELEM) (&(_SELEM)->sdata)
	102	+
	103	+#define BPF_LOCAL_STORAGE_CACHE_SIZE 16
	104	+
	105	+struct bpf_local_storage_cache {
	106	+ spinlock_t idx_lock;
	107	+ u64 idx_usage_counts[BPF_LOCAL_STORAGE_CACHE_SIZE];
	108	+};
	109	+
	110	+#define DEFINE_BPF_STORAGE_CACHE(name) \
	111	+static struct bpf_local_storage_cache name = { \
	112	+ .idx_lock = __SPIN_LOCK_UNLOCKED(name.idx_lock), \
	113	+}
	114	+
	115	+u16 bpf_local_storage_cache_idx_get(struct bpf_local_storage_cache *cache);
	116	+void bpf_local_storage_cache_idx_free(struct bpf_local_storage_cache *cache,
	117	+ u16 idx);
	118	+
	119	+/* Helper functions for bpf_local_storage */
	120	+int bpf_local_storage_map_alloc_check(union bpf_attr *attr);
	121	+
	122	+struct bpf_local_storage_map bpf_local_storage_map_alloc(union bpf_attr attr);
	123	+
	124	+struct bpf_local_storage_data *
	125	+bpf_local_storage_lookup(struct bpf_local_storage *local_storage,
	126	+ struct bpf_local_storage_map *smap,
	127	+ bool cacheit_lockit);
	128	+
	129	+void bpf_local_storage_map_free(struct bpf_local_storage_map *smap);
	130	+
	131	+int bpf_local_storage_map_check_btf(const struct bpf_map *map,
	132	+ const struct btf *btf,
	133	+ const struct btf_type *key_type,
	134	+ const struct btf_type *value_type);
	135	+
	136	+void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,
	137	+ struct bpf_local_storage_elem *selem);
	138	+
	139	+bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage,
	140	+ struct bpf_local_storage_elem *selem,
	141	+ bool uncharge_omem);
	142	+
	143	+void bpf_selem_unlink(struct bpf_local_storage_elem *selem);
	144	+
	145	+void bpf_selem_link_map(struct bpf_local_storage_map *smap,
	146	+ struct bpf_local_storage_elem *selem);
	147	+
	148	+void bpf_selem_unlink_map(struct bpf_local_storage_elem *selem);
	149	+
	150	+struct bpf_local_storage_elem *
	151	+bpf_selem_alloc(struct bpf_local_storage_map smap, void owner, void *value,
	152	+ bool charge_mem);
	153	+
	154	+int
	155	+bpf_local_storage_alloc(void *owner,
	156	+ struct bpf_local_storage_map *smap,
	157	+ struct bpf_local_storage_elem *first_selem);
	158	+
	159	+struct bpf_local_storage_data *
	160	+bpf_local_storage_update(void owner, struct bpf_local_storage_map smap,
	161	+ void *value, u64 map_flags);
	162	+
	163	+#endif /* _BPF_LOCAL_STORAGE_H */

include/linux/bpf_lsm.h+29−0

@@ -17,9 +17,28 @@ · 16 unmodified lines
17	17	#include <linux/lsm_hook_defs.h>
18	18	#undef LSM_HOOK
19	19
	20	+struct bpf_storage_blob {
	21	+ struct bpf_local_storage __rcu *storage;
	22	+};
	23	+
	24	+extern struct lsm_blob_sizes bpf_lsm_blob_sizes;
	25	+
20	26	int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,
21	27	const struct bpf_prog *prog);
22	28
	29	+static inline struct bpf_storage_blob *bpf_inode(
	30	+ const struct inode *inode)
	31	+{
	32	+ if (unlikely(!inode->i_security))
	33	+ return NULL;
	34	+
	35	+ return inode->i_security + bpf_lsm_blob_sizes.lbs_inode;
	36	+}
	37	+
	38	+extern const struct bpf_func_proto bpf_inode_storage_get_proto;
	39	+extern const struct bpf_func_proto bpf_inode_storage_delete_proto;
	40	+void bpf_inode_storage_free(struct inode *inode);
	41	+
23	42	#else /* !CONFIG_BPF_LSM */
24	43
25	44	static inline int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,
@@ -28,6 +47,16 @@ static inline int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog, · 2 unmodified lines
28	47	return -EOPNOTSUPP;
29	48	}
30	49
	50	+static inline struct bpf_storage_blob *bpf_inode(
	51	+ const struct inode *inode)
	52	+{
	53	+ return NULL;
	54	+}
	55	+
	56	+static inline void bpf_inode_storage_free(struct inode *inode)
	57	+{
	58	+}
	59	+
31	60	#endif /* CONFIG_BPF_LSM */
32	61
33	62	#endif /* _LINUX_BPF_LSM_H */

include/linux/bpf_types.h+3−0

@@ -107,6 +107,9 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_SK_STORAGE, sk_storage_map_ops) · 106 unmodified lines
107	107	BPF_MAP_TYPE(BPF_MAP_TYPE_SOCKMAP, sock_map_ops)
108	108	BPF_MAP_TYPE(BPF_MAP_TYPE_SOCKHASH, sock_hash_ops)
109	109	#endif
	110	+#ifdef CONFIG_BPF_LSM
	111	+BPF_MAP_TYPE(BPF_MAP_TYPE_INODE_STORAGE, inode_storage_map_ops)
	112	+#endif
110	113	BPF_MAP_TYPE(BPF_MAP_TYPE_CPUMAP, cpu_map_ops)
111	114	#if defined(CONFIG_XDP_SOCKETS)
112	115	BPF_MAP_TYPE(BPF_MAP_TYPE_XSKMAP, xsk_map_ops)

include/linux/btf.h+1−2

@@ -64,8 +64,7 @@ const struct btf_type btf_type_resolve_func_ptr(const struct btf btf, · 63 unmodified lines
64	64	u32 id, u32 *res_id);
65	65	const struct btf_type *
66	66	btf_resolve_size(const struct btf btf, const struct btf_type type,
67		− u32 type_size, const struct btf_type *elem_type,
68		− u32 *total_nelems);
	67	+ u32 *type_size);
69	68
70	69	#define for_each_member(i, struct_type, member) \
71	70	for (i = 0, member = btf_type_member(struct_type); \

include/linux/btf_ids.h+50−1

@@ -3,6 +3,11 @@ · 2 unmodified lines
3	3	#ifndef _LINUX_BTF_IDS_H
4	4	#define _LINUX_BTF_IDS_H
5	5
	6	+struct btf_id_set {
	7	+ u32 cnt;
	8	+ u32 ids[];
	9	+};
	10	+
6	11	#ifdef CONFIG_DEBUG_INFO_BTF
7	12
8	13	#include <linux/compiler.h> /* for __PASTE */
@@ -62,7 +67,7 @@ asm( \ · 53 unmodified lines
62	67	".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
63	68	"." #scope " " #name "; \n" \
64	69	#name ":; \n" \
65		−".popsection; \n"); \
	70	+".popsection; \n");
66	71
67	72	#define BTF_ID_LIST(name) \
68	73	__BTF_ID_LIST(name, local) \
@@ -88,12 +93,56 @@ asm( \ · 19 unmodified lines
88	93	".zero 4 \n" \
89	94	".popsection; \n");
90	95
	96	+/*
	97	+ * The BTF_SET_START/END macros pair defines sorted list of
	98	+ * BTF IDs plus its members count, with following layout:
	99	+ *
	100	+ * BTF_SET_START(list)
	101	+ * BTF_ID(type1, name1)
	102	+ * BTF_ID(type2, name2)
	103	+ * BTF_SET_END(list)
	104	+ *
	105	+ * __BTF_ID__set__list:
	106	+ * .zero 4
	107	+ * list:
	108	+ * __BTF_ID__type1__name1__3:
	109	+ * .zero 4
	110	+ * __BTF_ID__type2__name2__4:
	111	+ * .zero 4
	112	+ *
	113	+ */
	114	+#define __BTF_SET_START(name, scope) \
	115	+asm( \
	116	+".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
	117	+"." #scope " __BTF_ID__set__" #name "; \n" \
	118	+"__BTF_ID__set__" #name ":; \n" \
	119	+".zero 4 \n" \
	120	+".popsection; \n");
	121	+
	122	+#define BTF_SET_START(name) \
	123	+__BTF_ID_LIST(name, local) \
	124	+__BTF_SET_START(name, local)
	125	+
	126	+#define BTF_SET_START_GLOBAL(name) \
	127	+__BTF_ID_LIST(name, globl) \
	128	+__BTF_SET_START(name, globl)
	129	+
	130	+#define BTF_SET_END(name) \
	131	+asm( \
	132	+".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
	133	+".size __BTF_ID__set__" #name ", .-" #name " \n" \
	134	+".popsection; \n"); \
	135	+extern struct btf_id_set name;
	136	+
91	137	#else
92	138
93	139	#define BTF_ID_LIST(name) static u32 name[5];
94	140	#define BTF_ID(prefix, name)
95	141	#define BTF_ID_UNUSED
96	142	#define BTF_ID_LIST_GLOBAL(name) u32 name[1];
	143	+#define BTF_SET_START(name) static struct btf_id_set name = { 0 };
	144	+#define BTF_SET_START_GLOBAL(name) static struct btf_id_set name = { 0 };
	145	+#define BTF_SET_END(name)
97	146
98	147	#endif /* CONFIG_DEBUG_INFO_BTF */
99	148

include/linux/filter.h+6−2

@@ -1236,13 +1236,17 @@ struct bpf_sock_addr_kern { · 1235 unmodified lines
1236	1236
1237	1237	struct bpf_sock_ops_kern {
1238	1238	struct sock *sk;
1239		− u32 op;
1240	1239	union {
1241	1240	u32 args[4];
1242	1241	u32 reply;
1243	1242	u32 replylong[4];
1244	1243	};
1245		− u32 is_fullsock;
	1244	+ struct sk_buff *syn_skb;
	1245	+ struct sk_buff *skb;
	1246	+ void *skb_data_end;
	1247	+ u8 op;
	1248	+ u8 is_fullsock;
	1249	+ u8 remaining_opt_len;
1246	1250	u64 temp; /* temp and everything after is not
1247	1251	* initialized to 0 before calling
1248	1252	* the BPF program. New fields that

include/linux/if_tun.h+14−5

@@ -27,9 +27,18 @@ struct tun_xdp_hdr { · 26 unmodified lines
27	27	#if defined(CONFIG_TUN) \|\| defined(CONFIG_TUN_MODULE)
28	28	struct socket tun_get_socket(struct file );
29	29	struct ptr_ring tun_get_tx_ring(struct file file);
30		−bool tun_is_xdp_frame(void *ptr);
31		−void tun_xdp_to_ptr(void ptr);
32		−void tun_ptr_to_xdp(void ptr);
	30	+static inline bool tun_is_xdp_frame(void *ptr)
	31	+{
	32	+ return (unsigned long)ptr & TUN_XDP_FLAG;
	33	+}
	34	+static inline void tun_xdp_to_ptr(struct xdp_frame xdp)
	35	+{
	36	+ return (void *)((unsigned long)xdp \| TUN_XDP_FLAG);
	37	+}
	38	+static inline struct xdp_frame tun_ptr_to_xdp(void ptr)
	39	+{
	40	+ return (void *)((unsigned long)ptr & ~TUN_XDP_FLAG);
	41	+}
33	42	void tun_ptr_free(void *ptr);
34	43	#else
35	44	#include <linux/err.h>
@@ -48,11 +57,11 @@ static inline bool tun_is_xdp_frame(void *ptr) · 12 unmodified lines
48	57	{
49	58	return false;
50	59	}
51		−static inline void tun_xdp_to_ptr(void ptr)
	60	+static inline void tun_xdp_to_ptr(struct xdp_frame xdp)
52	61	{
53	62	return NULL;
54	63	}
55		−static inline void tun_ptr_to_xdp(void ptr)
	64	+static inline struct xdp_frame tun_ptr_to_xdp(void ptr)
56	65	{
57	66	return NULL;
58	67	}

include/linux/netdevice.h+5−5

@@ -618,7 +618,7 @@ struct netdev_queue { · 617 unmodified lines
618	618	/* Subordinate device that the queue has been assigned to */
619	619	struct net_device *sb_dev;
620	620	#ifdef CONFIG_XDP_SOCKETS
621		− struct xdp_umem *umem;
	621	+ struct xsk_buff_pool *pool;
622	622	#endif
623	623	/*
624	624	* write-mostly part
@@ -755,7 +755,7 @@ struct netdev_rx_queue { · 130 unmodified lines
755	755	struct net_device *dev;
756	756	struct xdp_rxq_info xdp_rxq;
757	757	#ifdef CONFIG_XDP_SOCKETS
758		− struct xdp_umem *umem;
	758	+ struct xsk_buff_pool *pool;
759	759	#endif
760	760	} ____cacheline_aligned_in_smp;
761	761
@@ -883,7 +883,7 @@ enum bpf_netdev_command { · 121 unmodified lines
883	883	/* BPF program for offload callbacks, invoked at program load time. */
884	884	BPF_OFFLOAD_MAP_ALLOC,
885	885	BPF_OFFLOAD_MAP_FREE,
886		− XDP_SETUP_XSK_UMEM,
	886	+ XDP_SETUP_XSK_POOL,
887	887	};
888	888
889	889	struct bpf_prog_offload_ops;
@@ -917,9 +917,9 @@ struct netdev_bpf { · 27 unmodified lines
917	917	struct {
918	918	struct bpf_offloaded_map *offmap;
919	919	};
920		− /* XDP_SETUP_XSK_UMEM */
	920	+ /* XDP_SETUP_XSK_POOL */
921	921	struct {
922		− struct xdp_umem *umem;
	922	+ struct xsk_buff_pool *pool;
923	923	u16 queue_id;
924	924	} xsk;
925	925	};

include/linux/rcupdate_trace.h+8−1

@@ -82,7 +82,14 @@ static inline void rcu_read_unlock_trace(void) · 81 unmodified lines
82	82	void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func);
83	83	void synchronize_rcu_tasks_trace(void);
84	84	void rcu_barrier_tasks_trace(void);
85		−
	85	+#else
	86	+/*
	87	+ * The BPF JIT forms these addresses even when it doesn't call these
	88	+ * functions, so provide definitions that result in runtime errors.
	89	+ */
	90	+static inline void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func) { BUG(); }
	91	+static inline void rcu_read_lock_trace(void) { BUG(); }
	92	+static inline void rcu_read_unlock_trace(void) { BUG(); }
86	93	#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */
87	94
88	95	#endif /* __LINUX_RCUPDATE_TRACE_H */

include/linux/skmsg.h+0−17

@@ -340,23 +340,6 @@ static inline void sk_psock_update_proto(struct sock *sk, · 339 unmodified lines
340	340	struct sk_psock *psock,
341	341	struct proto *ops)
342	342	{
343		− /* Initialize saved callbacks and original proto only once, since this
344		− * function may be called multiple times for a psock, e.g. when
345		− * psock->progs.msg_parser is updated.
346		− *
347		− * Since we've not installed the new proto, psock is not yet in use and
348		− * we can initialize it without synchronization.
349		− */
350		− if (!psock->sk_proto) {
351		− struct proto *orig = READ_ONCE(sk->sk_prot);
352		−
353		− psock->saved_unhash = orig->unhash;
354		− psock->saved_close = orig->close;
355		− psock->saved_write_space = sk->sk_write_space;
356		−
357		− psock->sk_proto = orig;
358		− }
359		−
360	343	/* Pairs with lockless read in sk_clone_lock() */
361	344	WRITE_ONCE(sk->sk_prot, ops);
362	345	}

include/linux/tcp.h+15−5

@@ -92,6 +92,8 @@ struct tcp_options_received { · 91 unmodified lines
92	92	smc_ok : 1, /* SMC seen on SYN packet */
93	93	snd_wscale : 4, /* Window scaling received from sender */
94	94	rcv_wscale : 4; /* Window scaling to send to receiver */
	95	+ u8 saw_unknown:1, /* Received unknown option */
	96	+ unused:7;
95	97	u8 num_sacks; /* Number of SACK blocks */
96	98	u16 user_mss; /* mss requested by user in ioctl */
97	99	u16 mss_clamp; /* Maximal mss, negotiated at connection setup */
@@ -237,14 +239,13 @@ struct tcp_sock { · 139 unmodified lines
237	239	repair : 1,
238	240	frto : 1;/* F-RTO (RFC5682) activated in CA_Loss */
239	241	u8 repair_queue;
240		− u8 syn_data:1, /* SYN includes data */
	242	+ u8 save_syn:2, /* Save headers of SYN packet */
	243	+ syn_data:1, /* SYN includes data */
241	244	syn_fastopen:1, /* SYN includes Fast Open option */
242	245	syn_fastopen_exp:1,/* SYN includes Fast Open exp. option */
243	246	syn_fastopen_ch:1, /* Active TFO re-enabling probe */
244	247	syn_data_acked:1,/* data in SYN is acked by SYN-ACK */
245		− save_syn:1, /* Save headers of SYN packet */
246		− is_cwnd_limited:1,/* forward progress limited by snd_cwnd? */
247		− syn_smc:1; /* SYN includes SMC */
	248	+ is_cwnd_limited:1;/* forward progress limited by snd_cwnd? */
248	249	u32 tlp_high_seq; /* snd_nxt at the time of TLP */
249	250
250	251	u32 tcp_tx_delay; /* delay (in usec) added to TX packets */
@@ -391,6 +392,9 @@ struct tcp_sock { · 140 unmodified lines
391	392	#if IS_ENABLED(CONFIG_MPTCP)
392	393	bool is_mptcp;
393	394	#endif
	395	+#if IS_ENABLED(CONFIG_SMC)
	396	+ bool syn_smc; /* SYN includes SMC */
	397	+#endif
394	398
395	399	#ifdef CONFIG_TCP_MD5SIG
396	400	/* TCP AF-Specific parts; only used by MD5 Signature support so far */
@@ -406,7 +410,7 @@ struct tcp_sock { · 9 unmodified lines
406	410	* socket. Used to retransmit SYNACKs etc.
407	411	*/
408	412	struct request_sock __rcu *fastopen_rsk;
409		− u32 *saved_syn;
	413	+ struct saved_syn *saved_syn;
410	414	};
411	415
412	416	enum tsq_enum {
@@ -484,6 +488,12 @@ static inline void tcp_saved_syn_free(struct tcp_sock *tp) · 71 unmodified lines
484	488	tp->saved_syn = NULL;
485	489	}
486	490
	491	+static inline u32 tcp_saved_syn_len(const struct saved_syn *saved_syn)
	492	+{
	493	+ return saved_syn->mac_hdrlen + saved_syn->network_hdrlen +
	494	+ saved_syn->tcp_hdrlen;
	495	+}
	496	+
487	497	struct sk_buff tcp_get_timestamping_opt_stats(const struct sock sk,
488	498	const struct sk_buff *orig_skb);
489	499

include/net/bpf_sk_storage.h+14−0

@@ -3,13 +3,27 @@ · 2 unmodified lines
3	3	#ifndef _BPF_SK_STORAGE_H
4	4	#define _BPF_SK_STORAGE_H
5	5
	6	+#include <linux/rculist.h>
	7	+#include <linux/list.h>
	8	+#include <linux/hash.h>
	9	+#include <linux/types.h>
	10	+#include <linux/spinlock.h>
	11	+#include <linux/bpf.h>
	12	+#include <net/sock.h>
	13	+#include <uapi/linux/sock_diag.h>
	14	+#include <uapi/linux/btf.h>
	15	+#include <linux/bpf_local_storage.h>
	16	+
6	17	struct sock;
7	18
8	19	void bpf_sk_storage_free(struct sock *sk);
9	20
10	21	extern const struct bpf_func_proto bpf_sk_storage_get_proto;
11	22	extern const struct bpf_func_proto bpf_sk_storage_delete_proto;
	23	+extern const struct bpf_func_proto sk_storage_get_btf_proto;
	24	+extern const struct bpf_func_proto sk_storage_delete_btf_proto;
12	25
	26	+struct bpf_local_storage_elem;
13	27	struct bpf_sk_storage_diag;
14	28	struct sk_buff;
15	29	struct nlattr;

include/net/inet_connection_sock.h+2−0

@@ -86,6 +86,8 @@ struct inet_connection_sock { · 85 unmodified lines
86	86	struct timer_list icsk_retransmit_timer;
87	87	struct timer_list icsk_delack_timer;
88	88	__u32 icsk_rto;
	89	+ __u32 icsk_rto_min;
	90	+ __u32 icsk_delack_max;
89	91	__u32 icsk_pmtu_cookie;
90	92	const struct tcp_congestion_ops *icsk_ca_ops;
91	93	const struct inet_connection_sock_af_ops *icsk_af_ops;

include/net/request_sock.h+8−1

@@ -41,6 +41,13 @@ struct request_sock_ops { · 40 unmodified lines
41	41
42	42	int inet_rtx_syn_ack(const struct sock parent, struct request_sock req);
43	43
	44	+struct saved_syn {
	45	+ u32 mac_hdrlen;
	46	+ u32 network_hdrlen;
	47	+ u32 tcp_hdrlen;
	48	+ u8 data[];
	49	+};
	50	+
44	51	/* struct request_sock - mini sock to represent a connection request
45	52	*/
46	53	struct request_sock {
@@ -60,7 +67,7 @@ struct request_sock { · 13 unmodified lines
60	67	struct timer_list rsk_timer;
61	68	const struct request_sock_ops *rsk_ops;
62	69	struct sock *sk;
63		− u32 *saved_syn;
	70	+ struct saved_syn *saved_syn;
64	71	u32 secid;
65	72	u32 peer_secid;
66	73	};

include/net/sock.h+2−2

@@ -246,7 +246,7 @@ struct sock_common { · 245 unmodified lines
246	246	/* public: */
247	247	};
248	248
249		−struct bpf_sk_storage;
	249	+struct bpf_local_storage;
250	250
251	251	/**
252	252	* struct sock - network layer representation of sockets
@@ -517,7 +517,7 @@ struct sock { · 264 unmodified lines
517	517	void (sk_destruct)(struct sock sk);
518	518	struct sock_reuseport __rcu *sk_reuseport_cb;
519	519	#ifdef CONFIG_BPF_SYSCALL
520		− struct bpf_sk_storage __rcu *sk_bpf_storage;
	520	+ struct bpf_local_storage __rcu *sk_bpf_storage;
521	521	#endif
522	522	struct rcu_head sk_rcu;
523	523	};

include/net/tcp.h+55−4

@@ -394,7 +394,7 @@ void tcp_metrics_init(void); · 393 unmodified lines
394	394	bool tcp_peer_is_proven(struct request_sock req, struct dst_entry dst);
395	395	void tcp_close(struct sock *sk, long timeout);
396	396	void tcp_init_sock(struct sock *sk);
397		−void tcp_init_transfer(struct sock *sk, int bpf_op);
	397	+void tcp_init_transfer(struct sock sk, int bpf_op, struct sk_buff skb);
398	398	__poll_t tcp_poll(struct file file, struct socket sock,
399	399	struct poll_table_struct *wait);
400	400	int tcp_getsockopt(struct sock *sk, int level, int optname,
@@ -455,7 +455,8 @@ enum tcp_synack_type { · 54 unmodified lines
455	455	struct sk_buff tcp_make_synack(const struct sock sk, struct dst_entry *dst,
456	456	struct request_sock *req,
457	457	struct tcp_fastopen_cookie *foc,
458		− enum tcp_synack_type synack_type);
	458	+ enum tcp_synack_type synack_type,
	459	+ struct sk_buff *syn_skb);
459	460	int tcp_disconnect(struct sock *sk, int flags);
460	461
461	462	void tcp_finish_connect(struct sock sk, struct sk_buff skb);
@@ -699,7 +700,7 @@ static inline void tcp_fast_path_check(struct sock *sk) · 237 unmodified lines
699	700	static inline u32 tcp_rto_min(struct sock *sk)
700	701	{
701	702	const struct dst_entry *dst = __sk_dst_get(sk);
702		− u32 rto_min = TCP_RTO_MIN;
	703	+ u32 rto_min = inet_csk(sk)->icsk_rto_min;
703	704
704	705	if (dst && dst_metric_locked(dst, RTAX_RTO_MIN))
705	706	rto_min = dst_metric_rtt(dst, RTAX_RTO_MIN);
@@ -2025,7 +2026,8 @@ struct tcp_request_sock_ops { · 1319 unmodified lines
2025	2026	int (send_synack)(const struct sock sk, struct dst_entry *dst,
2026	2027	struct flowi fl, struct request_sock req,
2027	2028	struct tcp_fastopen_cookie *foc,
2028		− enum tcp_synack_type synack_type);
	2029	+ enum tcp_synack_type synack_type,
	2030	+ struct sk_buff *syn_skb);
2029	2031	};
2030	2032
2031	2033	extern const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops;
@@ -2223,6 +2225,55 @@ int __tcp_bpf_recvmsg(struct sock sk, struct sk_psock psock, · 191 unmodified lines
2223	2225	struct msghdr *msg, int len, int flags);
2224	2226	#endif /* CONFIG_NET_SOCK_MSG */
2225	2227
	2228	+#ifdef CONFIG_CGROUP_BPF
	2229	+/* Copy the listen sk's HDR_OPT_CB flags to its child.
	2230	+ *
	2231	+ * During 3-Way-HandShake, the synack is usually sent from
	2232	+ * the listen sk with the HDR_OPT_CB flags set so that
	2233	+ * bpf-prog will be called to write the BPF hdr option.
	2234	+ *
	2235	+ * In fastopen, the child sk is used to send synack instead
	2236	+ * of the listen sk. Thus, inheriting the HDR_OPT_CB flags
	2237	+ * from the listen sk gives the bpf-prog a chance to write
	2238	+ * BPF hdr option in the synack pkt during fastopen.
	2239	+ *
	2240	+ * Both fastopen and non-fastopen child will inherit the
	2241	+ * HDR_OPT_CB flags to keep the bpf-prog having a consistent
	2242	+ * behavior when deciding to clear this cb flags (or not)
	2243	+ * during the PASSIVE_ESTABLISHED_CB.
	2244	+ *
	2245	+ * In the future, other cb flags could be inherited here also.
	2246	+ */
	2247	+static inline void bpf_skops_init_child(const struct sock *sk,
	2248	+ struct sock *child)
	2249	+{
	2250	+ tcp_sk(child)->bpf_sock_ops_cb_flags =
	2251	+ tcp_sk(sk)->bpf_sock_ops_cb_flags &
	2252	+ (BPF_SOCK_OPS_PARSE_ALL_HDR_OPT_CB_FLAG \|
	2253	+ BPF_SOCK_OPS_PARSE_UNKNOWN_HDR_OPT_CB_FLAG \|
	2254	+ BPF_SOCK_OPS_WRITE_HDR_OPT_CB_FLAG);
	2255	+}
	2256	+
	2257	+static inline void bpf_skops_init_skb(struct bpf_sock_ops_kern *skops,
	2258	+ struct sk_buff *skb,
	2259	+ unsigned int end_offset)
	2260	+{
	2261	+ skops->skb = skb;
	2262	+ skops->skb_data_end = skb->data + end_offset;
	2263	+}
	2264	+#else
	2265	+static inline void bpf_skops_init_child(const struct sock *sk,
	2266	+ struct sock *child)
	2267	+{
	2268	+}
	2269	+
	2270	+static inline void bpf_skops_init_skb(struct bpf_sock_ops_kern *skops,
	2271	+ struct sk_buff *skb,
	2272	+ unsigned int end_offset)
	2273	+{
	2274	+}
	2275	+#endif
	2276	+
2226	2277	/* Call BPF_SOCK_OPS program that returns an int. If the return value
2227	2278	* is < 0, then the BPF op failed (for example if the loaded BPF
2228	2279	* program does not support the chosen operation or there is no BPF

include/net/xdp_sock.h+14−16

@@ -18,25 +18,19 @@ struct xsk_queue; · 17 unmodified lines
18	18	struct xdp_buff;
19	19
20	20	struct xdp_umem {
21		− struct xsk_queue *fq;
22		− struct xsk_queue *cq;
23		− struct xsk_buff_pool *pool;
	21	+ void *addrs;
24	22	u64 size;
25	23	u32 headroom;
26	24	u32 chunk_size;
	25	+ u32 chunks;
	26	+ u32 npgs;
27	27	struct user_struct *user;
28	28	refcount_t users;
29		− struct work_struct work;
30		− struct page **pgs;
31		− u32 npgs;
32		− u16 queue_id;
33		− u8 need_wakeup;
34	29	u8 flags;
35		− int id;
36		− struct net_device *dev;
37	30	bool zc;
38		− spinlock_t xsk_tx_list_lock;
39		− struct list_head xsk_tx_list;
	31	+ struct page **pgs;
	32	+ int id;
	33	+ struct list_head xsk_dma_list;
40	34	};
41	35
42	36	struct xsk_map {
@@ -48,10 +42,11 @@ struct xsk_map { · 5 unmodified lines
48	42	struct xdp_sock {
49	43	/* struct sock must be the first member of struct xdp_sock */
50	44	struct sock sk;
51		− struct xsk_queue *rx;
	45	+ struct xsk_queue *rx ____cacheline_aligned_in_smp;
52	46	struct net_device *dev;
53	47	struct xdp_umem *umem;
54	48	struct list_head flush_node;
	49	+ struct xsk_buff_pool *pool;
55	50	u16 queue_id;
56	51	bool zc;
57	52	enum {
@@ -59,10 +54,9 @@ struct xdp_sock { · 1 unmodified line
59	54	XSK_BOUND,
60	55	XSK_UNBOUND,
61	56	} state;
62		− /* Protects multiple processes in the control path */
63		− struct mutex mutex;
	57	+
64	58	struct xsk_queue *tx ____cacheline_aligned_in_smp;
65		− struct list_head list;
	59	+ struct list_head tx_list;
66	60	/* Mutual exclusion of NAPI TX thread and sendmsg error paths
67	61	* in the SKB destructor callback.
68	62	*/
@@ -77,6 +71,10 @@ struct xdp_sock { · 8 unmodified lines
77	71	struct list_head map_list;
78	72	/* Protects map_list */
79	73	spinlock_t map_list_lock;
	74	+ /* Protects multiple processes in the control path */
	75	+ struct mutex mutex;
	76	+ struct xsk_queue fq_tmp; / Only as tmp storage before bind */
	77	+ struct xsk_queue cq_tmp; / Only as tmp storage before bind */
80	78	};
81	79
82	80	#ifdef CONFIG_XDP_SOCKETS

include/net/xdp_sock_drv.h+65−57

@@ -11,47 +11,50 @@ · 10 unmodified lines
11	11
12	12	#ifdef CONFIG_XDP_SOCKETS
13	13
14		−void xsk_umem_complete_tx(struct xdp_umem *umem, u32 nb_entries);
15		−bool xsk_umem_consume_tx(struct xdp_umem umem, struct xdp_desc desc);
16		−void xsk_umem_consume_tx_done(struct xdp_umem *umem);
17		−struct xdp_umem xdp_get_umem_from_qid(struct net_device dev, u16 queue_id);
18		−void xsk_set_rx_need_wakeup(struct xdp_umem *umem);
19		−void xsk_set_tx_need_wakeup(struct xdp_umem *umem);
20		−void xsk_clear_rx_need_wakeup(struct xdp_umem *umem);
21		−void xsk_clear_tx_need_wakeup(struct xdp_umem *umem);
22		−bool xsk_umem_uses_need_wakeup(struct xdp_umem *umem);
	14	+void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries);
	15	+bool xsk_tx_peek_desc(struct xsk_buff_pool pool, struct xdp_desc desc);
	16	+void xsk_tx_release(struct xsk_buff_pool *pool);
	17	+struct xsk_buff_pool xsk_get_pool_from_qid(struct net_device dev,
	18	+ u16 queue_id);
	19	+void xsk_set_rx_need_wakeup(struct xsk_buff_pool *pool);
	20	+void xsk_set_tx_need_wakeup(struct xsk_buff_pool *pool);
	21	+void xsk_clear_rx_need_wakeup(struct xsk_buff_pool *pool);
	22	+void xsk_clear_tx_need_wakeup(struct xsk_buff_pool *pool);
	23	+bool xsk_uses_need_wakeup(struct xsk_buff_pool *pool);
23	24
24		−static inline u32 xsk_umem_get_headroom(struct xdp_umem *umem)
	25	+static inline u32 xsk_pool_get_headroom(struct xsk_buff_pool *pool)
25	26	{
26		− return XDP_PACKET_HEADROOM + umem->headroom;
	27	+ return XDP_PACKET_HEADROOM + pool->headroom;
27	28	}
28	29
29		−static inline u32 xsk_umem_get_chunk_size(struct xdp_umem *umem)
	30	+static inline u32 xsk_pool_get_chunk_size(struct xsk_buff_pool *pool)
30	31	{
31		− return umem->chunk_size;
	32	+ return pool->chunk_size;
32	33	}
33	34
34		−static inline u32 xsk_umem_get_rx_frame_size(struct xdp_umem *umem)
	35	+static inline u32 xsk_pool_get_rx_frame_size(struct xsk_buff_pool *pool)
35	36	{
36		− return xsk_umem_get_chunk_size(umem) - xsk_umem_get_headroom(umem);
	37	+ return xsk_pool_get_chunk_size(pool) - xsk_pool_get_headroom(pool);
37	38	}
38	39
39		−static inline void xsk_buff_set_rxq_info(struct xdp_umem *umem,
	40	+static inline void xsk_pool_set_rxq_info(struct xsk_buff_pool *pool,
40	41	struct xdp_rxq_info *rxq)
41	42	{
42		− xp_set_rxq_info(umem->pool, rxq);
	43	+ xp_set_rxq_info(pool, rxq);
43	44	}
44	45
45		−static inline void xsk_buff_dma_unmap(struct xdp_umem *umem,
	46	+static inline void xsk_pool_dma_unmap(struct xsk_buff_pool *pool,
46	47	unsigned long attrs)
47	48	{
48		− xp_dma_unmap(umem->pool, attrs);
	49	+ xp_dma_unmap(pool, attrs);
49	50	}
50	51
51		−static inline int xsk_buff_dma_map(struct xdp_umem umem, struct device dev,
52		− unsigned long attrs)
	52	+static inline int xsk_pool_dma_map(struct xsk_buff_pool *pool,
	53	+ struct device *dev, unsigned long attrs)
53	54	{
54		− return xp_dma_map(umem->pool, dev, attrs, umem->pgs, umem->npgs);
	55	+ struct xdp_umem *umem = pool->umem;
	56	+
	57	+ return xp_dma_map(pool, dev, attrs, umem->pgs, umem->npgs);
55	58	}
56	59
57	60	static inline dma_addr_t xsk_buff_xdp_get_dma(struct xdp_buff *xdp)
@@ -68,14 +71,14 @@ static inline dma_addr_t xsk_buff_xdp_get_frame_dma(struct xdp_buff *xdp) · 10 unmodified lines
68	71	return xp_get_frame_dma(xskb);
69	72	}
70	73
71		−static inline struct xdp_buff xsk_buff_alloc(struct xdp_umem umem)
	74	+static inline struct xdp_buff xsk_buff_alloc(struct xsk_buff_pool pool)
72	75	{
73		− return xp_alloc(umem->pool);
	76	+ return xp_alloc(pool);
74	77	}
75	78
76		−static inline bool xsk_buff_can_alloc(struct xdp_umem *umem, u32 count)
	79	+static inline bool xsk_buff_can_alloc(struct xsk_buff_pool *pool, u32 count)
77	80	{
78		− return xp_can_alloc(umem->pool, count);
	81	+ return xp_can_alloc(pool, count);
79	82	}
80	83
81	84	static inline void xsk_buff_free(struct xdp_buff *xdp)
@@ -85,100 +88,104 @@ static inline void xsk_buff_free(struct xdp_buff *xdp) · 3 unmodified lines
85	88	xp_free(xskb);
86	89	}
87	90
88		−static inline dma_addr_t xsk_buff_raw_get_dma(struct xdp_umem *umem, u64 addr)
	91	+static inline dma_addr_t xsk_buff_raw_get_dma(struct xsk_buff_pool *pool,
	92	+ u64 addr)
89	93	{
90		− return xp_raw_get_dma(umem->pool, addr);
	94	+ return xp_raw_get_dma(pool, addr);
91	95	}
92	96
93		−static inline void xsk_buff_raw_get_data(struct xdp_umem umem, u64 addr)
	97	+static inline void xsk_buff_raw_get_data(struct xsk_buff_pool pool, u64 addr)
94	98	{
95		− return xp_raw_get_data(umem->pool, addr);
	99	+ return xp_raw_get_data(pool, addr);
96	100	}
97	101
98		−static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp)
	102	+static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff xdp, struct xsk_buff_pool pool)
99	103	{
100	104	struct xdp_buff_xsk *xskb = container_of(xdp, struct xdp_buff_xsk, xdp);
101	105
	106	+ if (!pool->dma_need_sync)
	107	+ return;
	108	+
102	109	xp_dma_sync_for_cpu(xskb);
103	110	}
104	111
105		−static inline void xsk_buff_raw_dma_sync_for_device(struct xdp_umem *umem,
	112	+static inline void xsk_buff_raw_dma_sync_for_device(struct xsk_buff_pool *pool,
106	113	dma_addr_t dma,
107	114	size_t size)
108	115	{
109		− xp_dma_sync_for_device(umem->pool, dma, size);
	116	+ xp_dma_sync_for_device(pool, dma, size);
110	117	}
111	118
112	119	#else
113	120
114		−static inline void xsk_umem_complete_tx(struct xdp_umem *umem, u32 nb_entries)
	121	+static inline void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries)
115	122	{
116	123	}
117	124
118		−static inline bool xsk_umem_consume_tx(struct xdp_umem *umem,
119		− struct xdp_desc *desc)
	125	+static inline bool xsk_tx_peek_desc(struct xsk_buff_pool *pool,
	126	+ struct xdp_desc *desc)
120	127	{
121	128	return false;
122	129	}
123	130
124		−static inline void xsk_umem_consume_tx_done(struct xdp_umem *umem)
	131	+static inline void xsk_tx_release(struct xsk_buff_pool *pool)
125	132	{
126	133	}
127	134
128		−static inline struct xdp_umem xdp_get_umem_from_qid(struct net_device dev,
129		− u16 queue_id)
	135	+static inline struct xsk_buff_pool *
	136	+xsk_get_pool_from_qid(struct net_device *dev, u16 queue_id)
130	137	{
131	138	return NULL;
132	139	}
133	140
134		−static inline void xsk_set_rx_need_wakeup(struct xdp_umem *umem)
	141	+static inline void xsk_set_rx_need_wakeup(struct xsk_buff_pool *pool)
135	142	{
136	143	}
137	144
138		−static inline void xsk_set_tx_need_wakeup(struct xdp_umem *umem)
	145	+static inline void xsk_set_tx_need_wakeup(struct xsk_buff_pool *pool)
139	146	{
140	147	}
141	148
142		−static inline void xsk_clear_rx_need_wakeup(struct xdp_umem *umem)
	149	+static inline void xsk_clear_rx_need_wakeup(struct xsk_buff_pool *pool)
143	150	{
144	151	}
145	152
146		−static inline void xsk_clear_tx_need_wakeup(struct xdp_umem *umem)
	153	+static inline void xsk_clear_tx_need_wakeup(struct xsk_buff_pool *pool)
147	154	{
148	155	}
149	156
150		−static inline bool xsk_umem_uses_need_wakeup(struct xdp_umem *umem)
	157	+static inline bool xsk_uses_need_wakeup(struct xsk_buff_pool *pool)
151	158	{
152	159	return false;
153	160	}
154	161
155		−static inline u32 xsk_umem_get_headroom(struct xdp_umem *umem)
	162	+static inline u32 xsk_pool_get_headroom(struct xsk_buff_pool *pool)
156	163	{
157	164	return 0;
158	165	}
159	166
160		−static inline u32 xsk_umem_get_chunk_size(struct xdp_umem *umem)
	167	+static inline u32 xsk_pool_get_chunk_size(struct xsk_buff_pool *pool)
161	168	{
162	169	return 0;
163	170	}
164	171
165		−static inline u32 xsk_umem_get_rx_frame_size(struct xdp_umem *umem)
	172	+static inline u32 xsk_pool_get_rx_frame_size(struct xsk_buff_pool *pool)
166	173	{
167	174	return 0;
168	175	}
169	176
170		−static inline void xsk_buff_set_rxq_info(struct xdp_umem *umem,
	177	+static inline void xsk_pool_set_rxq_info(struct xsk_buff_pool *pool,
171	178	struct xdp_rxq_info *rxq)
172	179	{
173	180	}
174	181
175		−static inline void xsk_buff_dma_unmap(struct xdp_umem *umem,
	182	+static inline void xsk_pool_dma_unmap(struct xsk_buff_pool *pool,
176	183	unsigned long attrs)
177	184	{
178	185	}
179	186
180		−static inline int xsk_buff_dma_map(struct xdp_umem umem, struct device dev,
181		− unsigned long attrs)
	187	+static inline int xsk_pool_dma_map(struct xsk_buff_pool *pool,
	188	+ struct device *dev, unsigned long attrs)
182	189	{
183	190	return 0;
184	191	}
@@ -193,12 +200,12 @@ static inline dma_addr_t xsk_buff_xdp_get_frame_dma(struct xdp_buff *xdp) · 8 unmodified lines
193	200	return 0;
194	201	}
195	202
196		−static inline struct xdp_buff xsk_buff_alloc(struct xdp_umem umem)
	203	+static inline struct xdp_buff xsk_buff_alloc(struct xsk_buff_pool pool)
197	204	{
198	205	return NULL;
199	206	}
200	207
201		−static inline bool xsk_buff_can_alloc(struct xdp_umem *umem, u32 count)
	208	+static inline bool xsk_buff_can_alloc(struct xsk_buff_pool *pool, u32 count)
202	209	{
203	210	return false;
204	211	}
@@ -207,21 +214,22 @@ static inline void xsk_buff_free(struct xdp_buff *xdp) · 2 unmodified lines
207	214	{
208	215	}
209	216
210		−static inline dma_addr_t xsk_buff_raw_get_dma(struct xdp_umem *umem, u64 addr)
	217	+static inline dma_addr_t xsk_buff_raw_get_dma(struct xsk_buff_pool *pool,
	218	+ u64 addr)
211	219	{
212	220	return 0;
213	221	}
214	222
215		−static inline void xsk_buff_raw_get_data(struct xdp_umem umem, u64 addr)
	223	+static inline void xsk_buff_raw_get_data(struct xsk_buff_pool pool, u64 addr)
216	224	{
217	225	return NULL;
218	226	}
219	227
220		−static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp)
	228	+static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff xdp, struct xsk_buff_pool pool)
221	229	{
222	230	}
223	231
224		−static inline void xsk_buff_raw_dma_sync_for_device(struct xdp_umem *umem,
	232	+static inline void xsk_buff_raw_dma_sync_for_device(struct xsk_buff_pool *pool,
225	233	dma_addr_t dma,
226	234	size_t size)
227	235	{

include/net/xsk_buff_pool.h+43−10

@@ -13,6 +13,8 @@ struct xsk_buff_pool; · 12 unmodified lines
13	13	struct xdp_rxq_info;
14	14	struct xsk_queue;
15	15	struct xdp_desc;
	16	+struct xdp_umem;
	17	+struct xdp_sock;
16	18	struct device;
17	19	struct page;
18	20
@@ -26,34 +28,68 @@ struct xdp_buff_xsk { · 7 unmodified lines
26	28	struct list_head free_list_node;
27	29	};
28	30
	31	+struct xsk_dma_map {
	32	+ dma_addr_t *dma_pages;
	33	+ struct device *dev;
	34	+ struct net_device *netdev;
	35	+ refcount_t users;
	36	+ struct list_head list; /* Protected by the RTNL_LOCK */
	37	+ u32 dma_pages_cnt;
	38	+ bool dma_need_sync;
	39	+};
	40	+
29	41	struct xsk_buff_pool {
30		− struct xsk_queue *fq;
	42	+ /* Members only used in the control path first. */
	43	+ struct device *dev;
	44	+ struct net_device *netdev;
	45	+ struct list_head xsk_tx_list;
	46	+ /* Protects modifications to the xsk_tx_list */
	47	+ spinlock_t xsk_tx_list_lock;
	48	+ refcount_t users;
	49	+ struct xdp_umem *umem;
	50	+ struct work_struct work;
31	51	struct list_head free_list;
	52	+ u32 heads_cnt;
	53	+ u16 queue_id;
	54	+
	55	+ /* Data path members as close to free_heads at the end as possible. */
	56	+ struct xsk_queue *fq ____cacheline_aligned_in_smp;
	57	+ struct xsk_queue *cq;
	58	+ /* For performance reasons, each buff pool has its own array of dma_pages
	59	+ * even when they are identical.
	60	+ */
32	61	dma_addr_t *dma_pages;
33	62	struct xdp_buff_xsk *heads;
34	63	u64 chunk_mask;
35	64	u64 addrs_cnt;
36	65	u32 free_list_cnt;
37	66	u32 dma_pages_cnt;
38		− u32 heads_cnt;
39	67	u32 free_heads_cnt;
40	68	u32 headroom;
41	69	u32 chunk_size;
42	70	u32 frame_len;
	71	+ u8 cached_need_wakeup;
	72	+ bool uses_need_wakeup;
43	73	bool dma_need_sync;
44	74	bool unaligned;
45	75	void *addrs;
46		− struct device *dev;
47	76	struct xdp_buff_xsk *free_heads[];
48	77	};
49	78
50	79	/* AF_XDP core. */
51		−struct xsk_buff_pool xp_create(struct page *pages, u32 nr_pages, u32 chunks,
52		− u32 chunk_size, u32 headroom, u64 size,
53		− bool unaligned);
54		−void xp_set_fq(struct xsk_buff_pool pool, struct xsk_queue fq);
	80	+struct xsk_buff_pool xp_create_and_assign_umem(struct xdp_sock xs,
	81	+ struct xdp_umem *umem);
	82	+int xp_assign_dev(struct xsk_buff_pool pool, struct net_device dev,
	83	+ u16 queue_id, u16 flags);
	84	+int xp_assign_dev_shared(struct xsk_buff_pool pool, struct xdp_umem umem,
	85	+ struct net_device *dev, u16 queue_id);
55	86	void xp_destroy(struct xsk_buff_pool *pool);
56	87	void xp_release(struct xdp_buff_xsk *xskb);
	88	+void xp_get_pool(struct xsk_buff_pool *pool);
	89	+void xp_put_pool(struct xsk_buff_pool *pool);
	90	+void xp_clear_dev(struct xsk_buff_pool *pool);
	91	+void xp_add_xsk(struct xsk_buff_pool pool, struct xdp_sock xs);
	92	+void xp_del_xsk(struct xsk_buff_pool pool, struct xdp_sock xs);
57	93
58	94	/* AF_XDP, and XDP core. */
59	95	void xp_free(struct xdp_buff_xsk *xskb);
@@ -80,9 +116,6 @@ static inline dma_addr_t xp_get_frame_dma(struct xdp_buff_xsk *xskb) · 20 unmodified lines
80	116	void xp_dma_sync_for_cpu_slow(struct xdp_buff_xsk *xskb);
81	117	static inline void xp_dma_sync_for_cpu(struct xdp_buff_xsk *xskb)
82	118	{
83		− if (!xskb->pool->dma_need_sync)
84		− return;
85		−
86	119	xp_dma_sync_for_cpu_slow(xskb);
87	120	}
88	121

include/uapi/linux/bpf.h+393−5

@@ -155,6 +155,7 @@ enum bpf_map_type { · 154 unmodified lines
155	155	BPF_MAP_TYPE_DEVMAP_HASH,
156	156	BPF_MAP_TYPE_STRUCT_OPS,
157	157	BPF_MAP_TYPE_RINGBUF,
	158	+ BPF_MAP_TYPE_INODE_STORAGE,
158	159	};
159	160
160	161	/* Note that tracing related programs such as
@@ -345,6 +346,14 @@ enum bpf_link_type { · 184 unmodified lines
345	346	/* The verifier internal test flag. Behavior is undefined */
346	347	#define BPF_F_TEST_STATE_FREQ (1U << 3)
347	348
	349	+/* If BPF_F_SLEEPABLE is used in BPF_PROG_LOAD command, the verifier will
	350	+ * restrict map and helper usage for such programs. Sleepable BPF programs can
	351	+ * only be attached to hooks where kernel execution context allows sleeping.
	352	+ * Such programs are allowed to use helpers that may sleep like
	353	+ * bpf_copy_from_user().
	354	+ */
	355	+#define BPF_F_SLEEPABLE (1U << 4)
	356	+
348	357	/* When BPF ldimm64's insn[0].src_reg != 0 then this can have
349	358	* two extensions:
350	359	*
@@ -2807,7 +2816,7 @@ union bpf_attr { · 2456 unmodified lines
2807	2816	*
2808	2817	* -ERANGE if resulting value was out of range.
2809	2818	*
2810		− * void bpf_sk_storage_get(struct bpf_map map, struct bpf_sock sk, void value, u64 flags)
	2819	+ * void bpf_sk_storage_get(struct bpf_map map, void sk, void value, u64 flags)
2811	2820	* Description
2812	2821	* Get a bpf-local-storage from a sk.
2813	2822	*
@@ -2823,6 +2832,9 @@ union bpf_attr { · 9 unmodified lines
2823	2832	* "type". The bpf-local-storage "type" (i.e. the map) is
2824	2833	* searched against all bpf-local-storages residing at sk.
2825	2834	*
	2835	+ * sk is a kernel struct sock pointer for LSM program.
	2836	+ * sk is a struct bpf_sock pointer for other program types.
	2837	+ *
2826	2838	* An optional flags (BPF_SK_STORAGE_GET_F_CREATE) can be
2827	2839	* used such that a new bpf-local-storage will be
2828	2840	* created if one does not exist. value can be used
@@ -2835,7 +2847,7 @@ union bpf_attr { · 6 unmodified lines
2835	2847	* NULL if not found or there was an error in adding
2836	2848	* a new bpf-local-storage.
2837	2849	*
2838		− * long bpf_sk_storage_delete(struct bpf_map map, struct bpf_sock sk)
	2850	+ * long bpf_sk_storage_delete(struct bpf_map map, void sk)
2839	2851	* Description
2840	2852	* Delete a bpf-local-storage from a sk.
2841	2853	* Return
@@ -3395,6 +3407,175 @@ union bpf_attr { · 553 unmodified lines
3395	3407	* A non-negative value equal to or less than size on success,
3396	3408	* or a negative error in case of failure.
3397	3409	*
	3410	+ * long bpf_load_hdr_opt(struct bpf_sock_ops skops, void searchby_res, u32 len, u64 flags)
	3411	+ * Description
	3412	+ * Load header option. Support reading a particular TCP header
	3413	+ * option for bpf program (BPF_PROG_TYPE_SOCK_OPS).
	3414	+ *
	3415	+ * If flags is 0, it will search the option from the
	3416	+ * sock_ops->skb_data. The comment in "struct bpf_sock_ops"
	3417	+ * has details on what skb_data contains under different
	3418	+ * sock_ops->op.
	3419	+ *
	3420	+ * The first byte of the searchby_res specifies the
	3421	+ * kind that it wants to search.
	3422	+ *
	3423	+ * If the searching kind is an experimental kind
	3424	+ * (i.e. 253 or 254 according to RFC6994). It also
	3425	+ * needs to specify the "magic" which is either
	3426	+ * 2 bytes or 4 bytes. It then also needs to
	3427	+ * specify the size of the magic by using
	3428	+ * the 2nd byte which is "kind-length" of a TCP
	3429	+ * header option and the "kind-length" also
	3430	+ * includes the first 2 bytes "kind" and "kind-length"
	3431	+ * itself as a normal TCP header option also does.
	3432	+ *
	3433	+ * For example, to search experimental kind 254 with
	3434	+ * 2 byte magic 0xeB9F, the searchby_res should be
	3435	+ * [ 254, 4, 0xeB, 0x9F, 0, 0, .... 0 ].
	3436	+ *
	3437	+ * To search for the standard window scale option (3),
	3438	+ * the searchby_res should be [ 3, 0, 0, .... 0 ].
	3439	+ * Note, kind-length must be 0 for regular option.
	3440	+ *
	3441	+ * Searching for No-Op (0) and End-of-Option-List (1) are
	3442	+ * not supported.
	3443	+ *
	3444	+ * len must be at least 2 bytes which is the minimal size
	3445	+ * of a header option.
	3446	+ *
	3447	+ * Supported flags:
	3448	+ * * BPF_LOAD_HDR_OPT_TCP_SYN to search from the
	3449	+ * saved_syn packet or the just-received syn packet.
	3450	+ *
	3451	+ * Return
	3452	+ * >0 when found, the header option is copied to searchby_res.
	3453	+ * The return value is the total length copied.
	3454	+ *
	3455	+ * -EINVAL If param is invalid
	3456	+ *
	3457	+ * -ENOMSG The option is not found
	3458	+ *
	3459	+ * -ENOENT No syn packet available when
	3460	+ * BPF_LOAD_HDR_OPT_TCP_SYN is used
	3461	+ *
	3462	+ * -ENOSPC Not enough space. Only len number of
	3463	+ * bytes are copied.
	3464	+ *
	3465	+ * -EFAULT Cannot parse the header options in the packet
	3466	+ *
	3467	+ * -EPERM This helper cannot be used under the
	3468	+ * current sock_ops->op.
	3469	+ *
	3470	+ * long bpf_store_hdr_opt(struct bpf_sock_ops skops, const void from, u32 len, u64 flags)
	3471	+ * Description
	3472	+ * Store header option. The data will be copied
	3473	+ * from buffer from with length len to the TCP header.
	3474	+ *
	3475	+ * The buffer from should have the whole option that
	3476	+ * includes the kind, kind-length, and the actual
	3477	+ * option data. The len must be at least kind-length
	3478	+ * long. The kind-length does not have to be 4 byte
	3479	+ * aligned. The kernel will take care of the padding
	3480	+ * and setting the 4 bytes aligned value to th->doff.
	3481	+ *
	3482	+ * This helper will check for duplicated option
	3483	+ * by searching the same option in the outgoing skb.
	3484	+ *
	3485	+ * This helper can only be called during
	3486	+ * BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
	3487	+ *
	3488	+ * Return
	3489	+ * 0 on success, or negative error in case of failure:
	3490	+ *
	3491	+ * -EINVAL If param is invalid
	3492	+ *
	3493	+ * -ENOSPC Not enough space in the header.
	3494	+ * Nothing has been written
	3495	+ *
	3496	+ * -EEXIST The option has already existed
	3497	+ *
	3498	+ * -EFAULT Cannot parse the existing header options
	3499	+ *
	3500	+ * -EPERM This helper cannot be used under the
	3501	+ * current sock_ops->op.
	3502	+ *
	3503	+ * long bpf_reserve_hdr_opt(struct bpf_sock_ops *skops, u32 len, u64 flags)
	3504	+ * Description
	3505	+ * Reserve len bytes for the bpf header option. The
	3506	+ * space will be used by bpf_store_hdr_opt() later in
	3507	+ * BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
	3508	+ *
	3509	+ * If bpf_reserve_hdr_opt() is called multiple times,
	3510	+ * the total number of bytes will be reserved.
	3511	+ *
	3512	+ * This helper can only be called during
	3513	+ * BPF_SOCK_OPS_HDR_OPT_LEN_CB.
	3514	+ *
	3515	+ * Return
	3516	+ * 0 on success, or negative error in case of failure:
	3517	+ *
	3518	+ * -EINVAL if param is invalid
	3519	+ *
	3520	+ * -ENOSPC Not enough space in the header.
	3521	+ *
	3522	+ * -EPERM This helper cannot be used under the
	3523	+ * current sock_ops->op.
	3524	+ *
	3525	+ * void bpf_inode_storage_get(struct bpf_map map, void inode, void value, u64 flags)
	3526	+ * Description
	3527	+ * Get a bpf_local_storage from an inode.
	3528	+ *
	3529	+ * Logically, it could be thought of as getting the value from
	3530	+ * a map with inode as the key. From this
	3531	+ * perspective, the usage is not much different from
	3532	+ * bpf_map_lookup_elem\ (map, &\ inode) except this
	3533	+ * helper enforces the key must be an inode and the map must also
	3534	+ * be a BPF_MAP_TYPE_INODE_STORAGE.
	3535	+ *
	3536	+ * Underneath, the value is stored locally at inode instead of
	3537	+ * the map. The map is used as the bpf-local-storage
	3538	+ * "type". The bpf-local-storage "type" (i.e. the map) is
	3539	+ * searched against all bpf_local_storage residing at inode.
	3540	+ *
	3541	+ * An optional flags (BPF_LOCAL_STORAGE_GET_F_CREATE) can be
	3542	+ * used such that a new bpf_local_storage will be
	3543	+ * created if one does not exist. value can be used
	3544	+ * together with BPF_LOCAL_STORAGE_GET_F_CREATE to specify
	3545	+ * the initial value of a bpf_local_storage. If value is
	3546	+ * NULL, the new bpf_local_storage will be zero initialized.
	3547	+ * Return
	3548	+ * A bpf_local_storage pointer is returned on success.
	3549	+ *
	3550	+ * NULL if not found or there was an error in adding
	3551	+ * a new bpf_local_storage.
	3552	+ *
	3553	+ * int bpf_inode_storage_delete(struct bpf_map map, void inode)
	3554	+ * Description
	3555	+ * Delete a bpf_local_storage from an inode.
	3556	+ * Return
	3557	+ * 0 on success.
	3558	+ *
	3559	+ * -ENOENT if the bpf_local_storage cannot be found.
	3560	+ *
	3561	+ * long bpf_d_path(struct path path, char buf, u32 sz)
	3562	+ * Description
	3563	+ * Return full path for given 'struct path' object, which
	3564	+ * needs to be the kernel BTF 'path' object. The path is
	3565	+ * returned in the provided buffer 'buf' of size 'sz' and
	3566	+ * is zero terminated.
	3567	+ *
	3568	+ * Return
	3569	+ * On success, the strictly positive length of the string,
	3570	+ * including the trailing NUL character. On error, a negative
	3571	+ * value.
	3572	+ *
	3573	+ * long bpf_copy_from_user(void dst, u32 size, const void user_ptr)
	3574	+ * Description
	3575	+ * Read size bytes from user space address user_ptr and store
	3576	+ * the data in dst. This is a wrapper of copy_from_user().
	3577	+ * Return
	3578	+ * 0 on success, or a negative error in case of failure.
3398	3579	*/
3399	3580	#define __BPF_FUNC_MAPPER(FN) \
3400	3581	FN(unspec), \
@@ -3539,6 +3720,13 @@ union bpf_attr { · 138 unmodified lines
3539	3720	FN(skc_to_tcp_request_sock), \
3540	3721	FN(skc_to_udp6_sock), \
3541	3722	FN(get_task_stack), \
	3723	+ FN(load_hdr_opt), \
	3724	+ FN(store_hdr_opt), \
	3725	+ FN(reserve_hdr_opt), \
	3726	+ FN(inode_storage_get), \
	3727	+ FN(inode_storage_delete), \
	3728	+ FN(d_path), \
	3729	+ FN(copy_from_user), \
3542	3730	/* */
3543	3731
3544	3732	/* integer value in 'imm' field of BPF_CALL instruction selects which helper
@@ -3648,9 +3836,13 @@ enum { · 103 unmodified lines
3648	3836	BPF_F_SYSCTL_BASE_NAME = (1ULL << 0),
3649	3837	};
3650	3838
3651		−/* BPF_FUNC_sk_storage_get flags */
	3839	+/* BPF_FUNC_<kernel_obj>_storage_get flags */
3652	3840	enum {
3653		− BPF_SK_STORAGE_GET_F_CREATE = (1ULL << 0),
	3841	+ BPF_LOCAL_STORAGE_GET_F_CREATE = (1ULL << 0),
	3842	+ /* BPF_SK_STORAGE_GET_F_CREATE is only kept for backward compatibility
	3843	+ * and BPF_LOCAL_STORAGE_GET_F_CREATE must be used instead.
	3844	+ */
	3845	+ BPF_SK_STORAGE_GET_F_CREATE = BPF_LOCAL_STORAGE_GET_F_CREATE,
3654	3846	};
3655	3847
3656	3848	/* BPF_FUNC_read_branch_records flags. */
@@ -4071,6 +4263,15 @@ struct bpf_link_info { · 414 unmodified lines
4071	4263	__u64 cgroup_id;
4072	4264	__u32 attach_type;
4073	4265	} cgroup;
	4266	+ struct {
	4267	+ __aligned_u64 target_name; /* in/out: target_name buffer ptr */
	4268	+ __u32 target_name_len; /* in/out: target_name buffer len */
	4269	+ union {
	4270	+ struct {
	4271	+ __u32 map_id;
	4272	+ } map;
	4273	+ };
	4274	+ } iter;
4074	4275	struct {
4075	4276	__u32 netns_ino;
4076	4277	__u32 attach_type;
@@ -4158,6 +4359,36 @@ struct bpf_sock_ops { · 81 unmodified lines
4158	4359	__u64 bytes_received;
4159	4360	__u64 bytes_acked;
4160	4361	__bpf_md_ptr(struct bpf_sock *, sk);
	4362	+ /* [skb_data, skb_data_end) covers the whole TCP header.
	4363	+ *
	4364	+ * BPF_SOCK_OPS_PARSE_HDR_OPT_CB: The packet received
	4365	+ * BPF_SOCK_OPS_HDR_OPT_LEN_CB: Not useful because the
	4366	+ * header has not been written.
	4367	+ * BPF_SOCK_OPS_WRITE_HDR_OPT_CB: The header and options have
	4368	+ * been written so far.
	4369	+ * BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB: The SYNACK that concludes
	4370	+ * the 3WHS.
	4371	+ * BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB: The ACK that concludes
	4372	+ * the 3WHS.
	4373	+ *
	4374	+ * bpf_load_hdr_opt() can also be used to read a particular option.
	4375	+ */
	4376	+ __bpf_md_ptr(void *, skb_data);
	4377	+ __bpf_md_ptr(void *, skb_data_end);
	4378	+ __u32 skb_len; /* The total length of a packet.
	4379	+ * It includes the header, options,
	4380	+ * and payload.
	4381	+ */
	4382	+ __u32 skb_tcp_flags; /* tcp_flags of the header. It provides
	4383	+ * an easy way to check for tcp_flags
	4384	+ * without parsing skb_data.
	4385	+ *
	4386	+ * In particular, the skb_tcp_flags
	4387	+ * will still be available in
	4388	+ * BPF_SOCK_OPS_HDR_OPT_LEN even though
	4389	+ * the outgoing header has not
	4390	+ * been written yet.
	4391	+ */
4161	4392	};
4162	4393
4163	4394	/* Definitions for bpf_sock_ops_cb_flags */
@@ -4166,8 +4397,51 @@ enum { · 2 unmodified lines
4166	4397	BPF_SOCK_OPS_RETRANS_CB_FLAG = (1<<1),
4167	4398	BPF_SOCK_OPS_STATE_CB_FLAG = (1<<2),
4168	4399	BPF_SOCK_OPS_RTT_CB_FLAG = (1<<3),
	4400	+ /* Call bpf for all received TCP headers. The bpf prog will be
	4401	+ * called under sock_ops->op == BPF_SOCK_OPS_PARSE_HDR_OPT_CB
	4402	+ *
	4403	+ * Please refer to the comment in BPF_SOCK_OPS_PARSE_HDR_OPT_CB
	4404	+ * for the header option related helpers that will be useful
	4405	+ * to the bpf programs.
	4406	+ *
	4407	+ * It could be used at the client/active side (i.e. connect() side)
	4408	+ * when the server told it that the server was in syncookie
	4409	+ * mode and required the active side to resend the bpf-written
	4410	+ * options. The active side can keep writing the bpf-options until
	4411	+ * it received a valid packet from the server side to confirm
	4412	+ * the earlier packet (and options) has been received. The later
	4413	+ * example patch is using it like this at the active side when the
	4414	+ * server is in syncookie mode.
	4415	+ *
	4416	+ * The bpf prog will usually turn this off in the common cases.
	4417	+ */
	4418	+ BPF_SOCK_OPS_PARSE_ALL_HDR_OPT_CB_FLAG = (1<<4),
	4419	+ /* Call bpf when kernel has received a header option that
	4420	+ * the kernel cannot handle. The bpf prog will be called under
	4421	+ * sock_ops->op == BPF_SOCK_OPS_PARSE_HDR_OPT_CB.
	4422	+ *
	4423	+ * Please refer to the comment in BPF_SOCK_OPS_PARSE_HDR_OPT_CB
	4424	+ * for the header option related helpers that will be useful
	4425	+ * to the bpf programs.
	4426	+ */
	4427	+ BPF_SOCK_OPS_PARSE_UNKNOWN_HDR_OPT_CB_FLAG = (1<<5),
	4428	+ /* Call bpf when the kernel is writing header options for the
	4429	+ * outgoing packet. The bpf prog will first be called
	4430	+ * to reserve space in a skb under
	4431	+ * sock_ops->op == BPF_SOCK_OPS_HDR_OPT_LEN_CB. Then
	4432	+ * the bpf prog will be called to write the header option(s)
	4433	+ * under sock_ops->op == BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
	4434	+ *
	4435	+ * Please refer to the comment in BPF_SOCK_OPS_HDR_OPT_LEN_CB
	4436	+ * and BPF_SOCK_OPS_WRITE_HDR_OPT_CB for the header option
	4437	+ * related helpers that will be useful to the bpf programs.
	4438	+ *
	4439	+ * The kernel gets its chance to reserve space and write
	4440	+ * options first before the BPF program does.
	4441	+ */
	4442	+ BPF_SOCK_OPS_WRITE_HDR_OPT_CB_FLAG = (1<<6),
4169	4443	/* Mask of all currently supported cb flags */
4170		− BPF_SOCK_OPS_ALL_CB_FLAGS = 0xF,
	4444	+ BPF_SOCK_OPS_ALL_CB_FLAGS = 0x7F,
4171	4445	};
4172	4446
4173	4447	/* List of known BPF sock_ops operators.
@@ -4223,6 +4497,63 @@ enum { · 49 unmodified lines
4223	4497	*/
4224	4498	BPF_SOCK_OPS_RTT_CB, /* Called on every RTT.
4225	4499	*/
	4500	+ BPF_SOCK_OPS_PARSE_HDR_OPT_CB, /* Parse the header option.
	4501	+ * It will be called to handle
	4502	+ * the packets received at
	4503	+ * an already established
	4504	+ * connection.
	4505	+ *
	4506	+ * sock_ops->skb_data:
	4507	+ * Referring to the received skb.
	4508	+ * It covers the TCP header only.
	4509	+ *
	4510	+ * bpf_load_hdr_opt() can also
	4511	+ * be used to search for a
	4512	+ * particular option.
	4513	+ */
	4514	+ BPF_SOCK_OPS_HDR_OPT_LEN_CB, /* Reserve space for writing the
	4515	+ * header option later in
	4516	+ * BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
	4517	+ * Arg1: bool want_cookie. (in
	4518	+ * writing SYNACK only)
	4519	+ *
	4520	+ * sock_ops->skb_data:
	4521	+ * Not available because no header has
	4522	+ * been written yet.
	4523	+ *
	4524	+ * sock_ops->skb_tcp_flags:
	4525	+ * The tcp_flags of the
	4526	+ * outgoing skb. (e.g. SYN, ACK, FIN).
	4527	+ *
	4528	+ * bpf_reserve_hdr_opt() should
	4529	+ * be used to reserve space.
	4530	+ */
	4531	+ BPF_SOCK_OPS_WRITE_HDR_OPT_CB, /* Write the header options
	4532	+ * Arg1: bool want_cookie. (in
	4533	+ * writing SYNACK only)
	4534	+ *
	4535	+ * sock_ops->skb_data:
	4536	+ * Referring to the outgoing skb.
	4537	+ * It covers the TCP header
	4538	+ * that has already been written
	4539	+ * by the kernel and the
	4540	+ * earlier bpf-progs.
	4541	+ *
	4542	+ * sock_ops->skb_tcp_flags:
	4543	+ * The tcp_flags of the outgoing
	4544	+ * skb. (e.g. SYN, ACK, FIN).
	4545	+ *
	4546	+ * bpf_store_hdr_opt() should
	4547	+ * be used to write the
	4548	+ * option.
	4549	+ *
	4550	+ * bpf_load_hdr_opt() can also
	4551	+ * be used to search for a
	4552	+ * particular option that
	4553	+ * has already been written
	4554	+ * by the kernel or the
	4555	+ * earlier bpf-progs.
	4556	+ */
4226	4557	};
4227	4558
4228	4559	/* List of TCP states. There is a build check in net/ipv4/tcp.c to detect
@@ -4250,6 +4581,63 @@ enum { · 21 unmodified lines
4250	4581	enum {
4251	4582	TCP_BPF_IW = 1001, /* Set TCP initial congestion window */
4252	4583	TCP_BPF_SNDCWND_CLAMP = 1002, /* Set sndcwnd_clamp */
	4584	+ TCP_BPF_DELACK_MAX = 1003, /* Max delay ack in usecs */
	4585	+ TCP_BPF_RTO_MIN = 1004, /* Min delay ack in usecs */
	4586	+ /* Copy the SYN pkt to optval
	4587	+ *
	4588	+ * BPF_PROG_TYPE_SOCK_OPS only. It is similar to the
	4589	+ * bpf_getsockopt(TCP_SAVED_SYN) but it does not limit
	4590	+ * to only getting from the saved_syn. It can either get the
	4591	+ * syn packet from:
	4592	+ *
	4593	+ * 1. the just-received SYN packet (only available when writing the
	4594	+ * SYNACK). It will be useful when it is not necessary to
	4595	+ * save the SYN packet for latter use. It is also the only way
	4596	+ * to get the SYN during syncookie mode because the syn
	4597	+ * packet cannot be saved during syncookie.
	4598	+ *
	4599	+ * OR
	4600	+ *
	4601	+ * 2. the earlier saved syn which was done by
	4602	+ * bpf_setsockopt(TCP_SAVE_SYN).
	4603	+ *
	4604	+ * The bpf_getsockopt(TCP_BPF_SYN*) option will hide where the
	4605	+ * SYN packet is obtained.
	4606	+ *
	4607	+ * If the bpf-prog does not need the IP[46] header, the
	4608	+ * bpf-prog can avoid parsing the IP header by using
	4609	+ * TCP_BPF_SYN. Otherwise, the bpf-prog can get both
	4610	+ * IP[46] and TCP header by using TCP_BPF_SYN_IP.
	4611	+ *
	4612	+ * >0: Total number of bytes copied
	4613	+ * -ENOSPC: Not enough space in optval. Only optlen number of
	4614	+ * bytes is copied.
	4615	+ * -ENOENT: The SYN skb is not available now and the earlier SYN pkt
	4616	+ * is not saved by setsockopt(TCP_SAVE_SYN).
	4617	+ */
	4618	+ TCP_BPF_SYN = 1005, /* Copy the TCP header */
	4619	+ TCP_BPF_SYN_IP = 1006, /* Copy the IP[46] and TCP header */
	4620	+ TCP_BPF_SYN_MAC = 1007, /* Copy the MAC, IP[46], and TCP header */
	4621	+};
	4622	+
	4623	+enum {
	4624	+ BPF_LOAD_HDR_OPT_TCP_SYN = (1ULL << 0),
	4625	+};
	4626	+
	4627	+/* args[0] value during BPF_SOCK_OPS_HDR_OPT_LEN_CB and
	4628	+ * BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
	4629	+ */
	4630	+enum {
	4631	+ BPF_WRITE_HDR_TCP_CURRENT_MSS = 1, /* Kernel is finding the
	4632	+ * total option spaces
	4633	+ * required for an established
	4634	+ * sk in order to calculate the
	4635	+ * MSS. No skb is actually
	4636	+ * sent.
	4637	+ */
	4638	+ BPF_WRITE_HDR_TCP_SYNACK_COOKIE = 2, /* Kernel is in syncookie mode
	4639	+ * when sending a SYN.
	4640	+ */
4253	4641	};
4254	4642
4255	4643	struct bpf_perf_event_value {

init/Kconfig+3−0

@@ -1691,6 +1691,7 @@ config BPF_SYSCALL · 1690 unmodified lines
1691	1691	bool "Enable bpf() system call"
1692	1692	select BPF
1693	1693	select IRQ_WORK
	1694	+ select TASKS_TRACE_RCU
1694	1695	default n
1695	1696	help
1696	1697	Enable the bpf() system call that allows to manipulate eBPF
@@ -1710,6 +1711,8 @@ config BPF_JIT_DEFAULT_ON · 13 unmodified lines
1710	1711	def_bool ARCH_WANT_DEFAULT_BPF_JIT \|\| BPF_JIT_ALWAYS_ON
1711	1712	depends on HAVE_EBPF_JIT && BPF_JIT
1712	1713
	1714	+source "kernel/bpf/preload/Kconfig"
	1715	+
1713	1716	config USERFAULTFD
1714	1717	bool "Enable userfaultfd() system call"
1715	1718	depends on MMU

kernel/Makefile+1−1

@@ -12,7 +12,7 @@ obj-y = fork.o exec_domain.o panic.o \ · 11 unmodified lines
12	12	notifier.o ksysfs.o cred.o reboot.o \
13	13	async.o range.o smpboot.o ucount.o regset.o
14	14
15		−obj-$(CONFIG_BPFILTER) += usermode_driver.o
	15	+obj-$(CONFIG_USERMODE_DRIVER) += usermode_driver.o
16	16	obj-$(CONFIG_MODULES) += kmod.o
17	17	obj-$(CONFIG_MULTIUSER) += groups.o
18	18

kernel/bpf/Makefile+3−0

@@ -5,6 +5,7 @@ CFLAGS_core.o += $(call cc-disable-warning, override-init) · 4 unmodified lines
5	5	obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o task_iter.o prog_iter.o
6	6	obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o
7	7	obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
	8	+obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
8	9	obj-$(CONFIG_BPF_SYSCALL) += disasm.o
9	10	obj-$(CONFIG_BPF_JIT) += trampoline.o
10	11	obj-$(CONFIG_BPF_SYSCALL) += btf.o
@@ -12,6 +13,7 @@ obj-$(CONFIG_BPF_JIT) += dispatcher.o · 1 unmodified line
12	13	ifeq ($(CONFIG_NET),y)
13	14	obj-$(CONFIG_BPF_SYSCALL) += devmap.o
14	15	obj-$(CONFIG_BPF_SYSCALL) += cpumap.o
	16	+obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o
15	17	obj-$(CONFIG_BPF_SYSCALL) += offload.o
16	18	obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o
17	19	endif
@@ -29,3 +31,4 @@ ifeq ($(CONFIG_BPF_JIT),y) · 11 unmodified lines
29	31	obj-$(CONFIG_BPF_SYSCALL) += bpf_struct_ops.o
30	32	obj-${CONFIG_BPF_LSM} += bpf_lsm.o
31	33	endif
	34	+obj-$(CONFIG_BPF_PRELOAD) += preload/

kernel/bpf/arraymap.c+17−0

@@ -10,6 +10,7 @@ · 9 unmodified lines
10	10	#include <linux/filter.h>
11	11	#include <linux/perf_event.h>
12	12	#include <uapi/linux/btf.h>
	13	+#include <linux/rcupdate_trace.h>
13	14
14	15	#include "map_in_map.h"
15	16
@@ -487,6 +488,13 @@ static int array_map_mmap(struct bpf_map map, struct vm_area_struct vma) · 471 unmodified lines
487	488	vma->vm_pgoff + pgoff);
488	489	}
489	490
	491	+static bool array_map_meta_equal(const struct bpf_map *meta0,
	492	+ const struct bpf_map *meta1)
	493	+{
	494	+ return meta0->max_entries == meta1->max_entries &&
	495	+ bpf_map_meta_equal(meta0, meta1);
	496	+}
	497	+
490	498	struct bpf_iter_seq_array_map_info {
491	499	struct bpf_map *map;
492	500	void *percpu_value_buf;
@@ -625,6 +633,7 @@ static const struct bpf_iter_seq_info iter_seq_info = { · 132 unmodified lines
625	633
626	634	static int array_map_btf_id;
627	635	const struct bpf_map_ops array_map_ops = {
	636	+ .map_meta_equal = array_map_meta_equal,
628	637	.map_alloc_check = array_map_alloc_check,
629	638	.map_alloc = array_map_alloc,
630	639	.map_free = array_map_free,
@@ -647,6 +656,7 @@ const struct bpf_map_ops array_map_ops = { · 16 unmodified lines
647	656
648	657	static int percpu_array_map_btf_id;
649	658	const struct bpf_map_ops percpu_array_map_ops = {
	659	+ .map_meta_equal = bpf_map_meta_equal,
650	660	.map_alloc_check = array_map_alloc_check,
651	661	.map_alloc = array_map_alloc,
652	662	.map_free = array_map_free,
@@ -1003,6 +1013,11 @@ static void prog_array_map_free(struct bpf_map *map) · 350 unmodified lines
1003	1013	fd_array_map_free(map);
1004	1014	}
1005	1015
	1016	+/* prog_array->aux->{type,jited} is a runtime binding.
	1017	+ * Doing static check alone in the verifier is not enough.
	1018	+ * Thus, prog_array_map cannot be used as an inner_map
	1019	+ * and map_meta_equal is not implemented.
	1020	+ */
1006	1021	static int prog_array_map_btf_id;
1007	1022	const struct bpf_map_ops prog_array_map_ops = {
1008	1023	.map_alloc_check = fd_array_map_alloc_check,
@@ -1101,6 +1116,7 @@ static void perf_event_fd_array_release(struct bpf_map *map, · 92 unmodified lines
1101	1116
1102	1117	static int perf_event_array_map_btf_id;
1103	1118	const struct bpf_map_ops perf_event_array_map_ops = {
	1119	+ .map_meta_equal = bpf_map_meta_equal,
1104	1120	.map_alloc_check = fd_array_map_alloc_check,
1105	1121	.map_alloc = array_map_alloc,
1106	1122	.map_free = fd_array_map_free,
@@ -1137,6 +1153,7 @@ static void cgroup_fd_array_free(struct bpf_map *map) · 30 unmodified lines
1137	1153
1138	1154	static int cgroup_array_map_btf_id;
1139	1155	const struct bpf_map_ops cgroup_array_map_ops = {
	1156	+ .map_meta_equal = bpf_map_meta_equal,
1140	1157	.map_alloc_check = fd_array_map_alloc_check,
1141	1158	.map_alloc = array_map_alloc,
1142	1159	.map_free = cgroup_fd_array_free,

kernel/bpf/bpf_inode_storage.c+274−0

@@ -0,0 +1,274 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+/*
	3	+ * Copyright (c) 2019 Facebook
	4	+ * Copyright 2020 Google LLC.
	5	+ */
	6	+
	7	+#include <linux/rculist.h>
	8	+#include <linux/list.h>
	9	+#include <linux/hash.h>
	10	+#include <linux/types.h>
	11	+#include <linux/spinlock.h>
	12	+#include <linux/bpf.h>
	13	+#include <linux/bpf_local_storage.h>
	14	+#include <net/sock.h>
	15	+#include <uapi/linux/sock_diag.h>
	16	+#include <uapi/linux/btf.h>
	17	+#include <linux/bpf_lsm.h>
	18	+#include <linux/btf_ids.h>
	19	+#include <linux/fdtable.h>
	20	+
	21	+DEFINE_BPF_STORAGE_CACHE(inode_cache);
	22	+
	23	+static struct bpf_local_storage __rcu **
	24	+inode_storage_ptr(void *owner)
	25	+{
	26	+ struct inode *inode = owner;
	27	+ struct bpf_storage_blob *bsb;
	28	+
	29	+ bsb = bpf_inode(inode);
	30	+ if (!bsb)
	31	+ return NULL;
	32	+ return &bsb->storage;
	33	+}
	34	+
	35	+static struct bpf_local_storage_data inode_storage_lookup(struct inode inode,
	36	+ struct bpf_map *map,
	37	+ bool cacheit_lockit)
	38	+{
	39	+ struct bpf_local_storage *inode_storage;
	40	+ struct bpf_local_storage_map *smap;
	41	+ struct bpf_storage_blob *bsb;
	42	+
	43	+ bsb = bpf_inode(inode);
	44	+ if (!bsb)
	45	+ return NULL;
	46	+
	47	+ inode_storage = rcu_dereference(bsb->storage);
	48	+ if (!inode_storage)
	49	+ return NULL;
	50	+
	51	+ smap = (struct bpf_local_storage_map *)map;
	52	+ return bpf_local_storage_lookup(inode_storage, smap, cacheit_lockit);
	53	+}
	54	+
	55	+void bpf_inode_storage_free(struct inode *inode)
	56	+{
	57	+ struct bpf_local_storage_elem *selem;
	58	+ struct bpf_local_storage *local_storage;
	59	+ bool free_inode_storage = false;
	60	+ struct bpf_storage_blob *bsb;
	61	+ struct hlist_node *n;
	62	+
	63	+ bsb = bpf_inode(inode);
	64	+ if (!bsb)
	65	+ return;
	66	+
	67	+ rcu_read_lock();
	68	+
	69	+ local_storage = rcu_dereference(bsb->storage);
	70	+ if (!local_storage) {
	71	+ rcu_read_unlock();
	72	+ return;
	73	+ }
	74	+
	75	+ /* Netiher the bpf_prog nor the bpf-map's syscall
	76	+ * could be modifying the local_storage->list now.
	77	+ * Thus, no elem can be added-to or deleted-from the
	78	+ * local_storage->list by the bpf_prog or by the bpf-map's syscall.
	79	+ *
	80	+ * It is racing with bpf_local_storage_map_free() alone
	81	+ * when unlinking elem from the local_storage->list and
	82	+ * the map's bucket->list.
	83	+ */
	84	+ raw_spin_lock_bh(&local_storage->lock);
	85	+ hlist_for_each_entry_safe(selem, n, &local_storage->list, snode) {
	86	+ /* Always unlink from map before unlinking from
	87	+ * local_storage.
	88	+ */
	89	+ bpf_selem_unlink_map(selem);
	90	+ free_inode_storage = bpf_selem_unlink_storage_nolock(
	91	+ local_storage, selem, false);
	92	+ }
	93	+ raw_spin_unlock_bh(&local_storage->lock);
	94	+ rcu_read_unlock();
	95	+
	96	+ /* free_inoode_storage should always be true as long as
	97	+ * local_storage->list was non-empty.
	98	+ */
	99	+ if (free_inode_storage)
	100	+ kfree_rcu(local_storage, rcu);
	101	+}
	102	+
	103	+static void bpf_fd_inode_storage_lookup_elem(struct bpf_map map, void *key)
	104	+{
	105	+ struct bpf_local_storage_data *sdata;
	106	+ struct file *f;
	107	+ int fd;
	108	+
	109	+ fd = (int )key;
	110	+ f = fget_raw(fd);
	111	+ if (!f)
	112	+ return NULL;
	113	+
	114	+ sdata = inode_storage_lookup(f->f_inode, map, true);
	115	+ fput(f);
	116	+ return sdata ? sdata->data : NULL;
	117	+}
	118	+
	119	+static int bpf_fd_inode_storage_update_elem(struct bpf_map map, void key,
	120	+ void *value, u64 map_flags)
	121	+{
	122	+ struct bpf_local_storage_data *sdata;
	123	+ struct file *f;
	124	+ int fd;
	125	+
	126	+ fd = (int )key;
	127	+ f = fget_raw(fd);
	128	+ if (!f \|\| !inode_storage_ptr(f->f_inode))
	129	+ return -EBADF;
	130	+
	131	+ sdata = bpf_local_storage_update(f->f_inode,
	132	+ (struct bpf_local_storage_map *)map,
	133	+ value, map_flags);
	134	+ fput(f);
	135	+ return PTR_ERR_OR_ZERO(sdata);
	136	+}
	137	+
	138	+static int inode_storage_delete(struct inode inode, struct bpf_map map)
	139	+{
	140	+ struct bpf_local_storage_data *sdata;
	141	+
	142	+ sdata = inode_storage_lookup(inode, map, false);
	143	+ if (!sdata)
	144	+ return -ENOENT;
	145	+
	146	+ bpf_selem_unlink(SELEM(sdata));
	147	+
	148	+ return 0;
	149	+}
	150	+
	151	+static int bpf_fd_inode_storage_delete_elem(struct bpf_map map, void key)
	152	+{
	153	+ struct file *f;
	154	+ int fd, err;
	155	+
	156	+ fd = (int )key;
	157	+ f = fget_raw(fd);
	158	+ if (!f)
	159	+ return -EBADF;
	160	+
	161	+ err = inode_storage_delete(f->f_inode, map);
	162	+ fput(f);
	163	+ return err;
	164	+}
	165	+
	166	+BPF_CALL_4(bpf_inode_storage_get, struct bpf_map , map, struct inode , inode,
	167	+ void *, value, u64, flags)
	168	+{
	169	+ struct bpf_local_storage_data *sdata;
	170	+
	171	+ if (flags & ~(BPF_LOCAL_STORAGE_GET_F_CREATE))
	172	+ return (unsigned long)NULL;
	173	+
	174	+ /* explicitly check that the inode_storage_ptr is not
	175	+ * NULL as inode_storage_lookup returns NULL in this case and
	176	+ * bpf_local_storage_update expects the owner to have a
	177	+ * valid storage pointer.
	178	+ */
	179	+ if (!inode_storage_ptr(inode))
	180	+ return (unsigned long)NULL;
	181	+
	182	+ sdata = inode_storage_lookup(inode, map, true);
	183	+ if (sdata)
	184	+ return (unsigned long)sdata->data;
	185	+
	186	+ /* This helper must only called from where the inode is gurranteed
	187	+ * to have a refcount and cannot be freed.
	188	+ */
	189	+ if (flags & BPF_LOCAL_STORAGE_GET_F_CREATE) {
	190	+ sdata = bpf_local_storage_update(
	191	+ inode, (struct bpf_local_storage_map *)map, value,
	192	+ BPF_NOEXIST);
	193	+ return IS_ERR(sdata) ? (unsigned long)NULL :
	194	+ (unsigned long)sdata->data;
	195	+ }
	196	+
	197	+ return (unsigned long)NULL;
	198	+}
	199	+
	200	+BPF_CALL_2(bpf_inode_storage_delete,
	201	+ struct bpf_map , map, struct inode , inode)
	202	+{
	203	+ /* This helper must only called from where the inode is gurranteed
	204	+ * to have a refcount and cannot be freed.
	205	+ */
	206	+ return inode_storage_delete(inode, map);
	207	+}
	208	+
	209	+static int notsupp_get_next_key(struct bpf_map map, void key,
	210	+ void *next_key)
	211	+{
	212	+ return -ENOTSUPP;
	213	+}
	214	+
	215	+static struct bpf_map inode_storage_map_alloc(union bpf_attr attr)
	216	+{
	217	+ struct bpf_local_storage_map *smap;
	218	+
	219	+ smap = bpf_local_storage_map_alloc(attr);
	220	+ if (IS_ERR(smap))
	221	+ return ERR_CAST(smap);
	222	+
	223	+ smap->cache_idx = bpf_local_storage_cache_idx_get(&inode_cache);
	224	+ return &smap->map;
	225	+}
	226	+
	227	+static void inode_storage_map_free(struct bpf_map *map)
	228	+{
	229	+ struct bpf_local_storage_map *smap;
	230	+
	231	+ smap = (struct bpf_local_storage_map *)map;
	232	+ bpf_local_storage_cache_idx_free(&inode_cache, smap->cache_idx);
	233	+ bpf_local_storage_map_free(smap);
	234	+}
	235	+
	236	+static int inode_storage_map_btf_id;
	237	+const struct bpf_map_ops inode_storage_map_ops = {
	238	+ .map_meta_equal = bpf_map_meta_equal,
	239	+ .map_alloc_check = bpf_local_storage_map_alloc_check,
	240	+ .map_alloc = inode_storage_map_alloc,
	241	+ .map_free = inode_storage_map_free,
	242	+ .map_get_next_key = notsupp_get_next_key,
	243	+ .map_lookup_elem = bpf_fd_inode_storage_lookup_elem,
	244	+ .map_update_elem = bpf_fd_inode_storage_update_elem,
	245	+ .map_delete_elem = bpf_fd_inode_storage_delete_elem,
	246	+ .map_check_btf = bpf_local_storage_map_check_btf,
	247	+ .map_btf_name = "bpf_local_storage_map",
	248	+ .map_btf_id = &inode_storage_map_btf_id,
	249	+ .map_owner_storage_ptr = inode_storage_ptr,
	250	+};
	251	+
	252	+BTF_ID_LIST(bpf_inode_storage_btf_ids)
	253	+BTF_ID_UNUSED
	254	+BTF_ID(struct, inode)
	255	+
	256	+const struct bpf_func_proto bpf_inode_storage_get_proto = {
	257	+ .func = bpf_inode_storage_get,
	258	+ .gpl_only = false,
	259	+ .ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
	260	+ .arg1_type = ARG_CONST_MAP_PTR,
	261	+ .arg2_type = ARG_PTR_TO_BTF_ID,
	262	+ .arg3_type = ARG_PTR_TO_MAP_VALUE_OR_NULL,
	263	+ .arg4_type = ARG_ANYTHING,
	264	+ .btf_id = bpf_inode_storage_btf_ids,
	265	+};
	266	+
	267	+const struct bpf_func_proto bpf_inode_storage_delete_proto = {
	268	+ .func = bpf_inode_storage_delete,
	269	+ .gpl_only = false,
	270	+ .ret_type = RET_INTEGER,
	271	+ .arg1_type = ARG_CONST_MAP_PTR,
	272	+ .arg2_type = ARG_PTR_TO_BTF_ID,
	273	+ .btf_id = bpf_inode_storage_btf_ids,
	274	+};

kernel/bpf/bpf_iter.c+58−0

@@ -390,10 +390,68 @@ out_unlock: · 389 unmodified lines
390	390	return ret;
391	391	}
392	392
	393	+static void bpf_iter_link_show_fdinfo(const struct bpf_link *link,
	394	+ struct seq_file *seq)
	395	+{
	396	+ struct bpf_iter_link *iter_link =
	397	+ container_of(link, struct bpf_iter_link, link);
	398	+ bpf_iter_show_fdinfo_t show_fdinfo;
	399	+
	400	+ seq_printf(seq,
	401	+ "target_name:\t%s\n",
	402	+ iter_link->tinfo->reg_info->target);
	403	+
	404	+ show_fdinfo = iter_link->tinfo->reg_info->show_fdinfo;
	405	+ if (show_fdinfo)
	406	+ show_fdinfo(&iter_link->aux, seq);
	407	+}
	408	+
	409	+static int bpf_iter_link_fill_link_info(const struct bpf_link *link,
	410	+ struct bpf_link_info *info)
	411	+{
	412	+ struct bpf_iter_link *iter_link =
	413	+ container_of(link, struct bpf_iter_link, link);
	414	+ char __user *ubuf = u64_to_user_ptr(info->iter.target_name);
	415	+ bpf_iter_fill_link_info_t fill_link_info;
	416	+ u32 ulen = info->iter.target_name_len;
	417	+ const char *target_name;
	418	+ u32 target_len;
	419	+
	420	+ if (!ulen ^ !ubuf)
	421	+ return -EINVAL;
	422	+
	423	+ target_name = iter_link->tinfo->reg_info->target;
	424	+ target_len = strlen(target_name);
	425	+ info->iter.target_name_len = target_len + 1;
	426	+
	427	+ if (ubuf) {
	428	+ if (ulen >= target_len + 1) {
	429	+ if (copy_to_user(ubuf, target_name, target_len + 1))
	430	+ return -EFAULT;
	431	+ } else {
	432	+ char zero = '\0';
	433	+
	434	+ if (copy_to_user(ubuf, target_name, ulen - 1))
	435	+ return -EFAULT;
	436	+ if (put_user(zero, ubuf + ulen - 1))
	437	+ return -EFAULT;
	438	+ return -ENOSPC;
	439	+ }
	440	+ }
	441	+
	442	+ fill_link_info = iter_link->tinfo->reg_info->fill_link_info;
	443	+ if (fill_link_info)
	444	+ return fill_link_info(&iter_link->aux, info);
	445	+
	446	+ return 0;
	447	+}
	448	+
393	449	static const struct bpf_link_ops bpf_iter_link_lops = {
394	450	.release = bpf_iter_link_release,
395	451	.dealloc = bpf_iter_link_dealloc,
396	452	.update_prog = bpf_iter_link_replace,
	453	+ .show_fdinfo = bpf_iter_link_show_fdinfo,
	454	+ .fill_link_info = bpf_iter_link_fill_link_info,
397	455	};
398	456
399	457	bool bpf_link_is_iter(struct bpf_link *link)

kernel/bpf/bpf_local_storage.c+600−0

@@ -0,0 +1,600 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+/* Copyright (c) 2019 Facebook */
	3	+#include <linux/rculist.h>
	4	+#include <linux/list.h>
	5	+#include <linux/hash.h>
	6	+#include <linux/types.h>
	7	+#include <linux/spinlock.h>
	8	+#include <linux/bpf.h>
	9	+#include <linux/btf_ids.h>
	10	+#include <linux/bpf_local_storage.h>
	11	+#include <net/sock.h>
	12	+#include <uapi/linux/sock_diag.h>
	13	+#include <uapi/linux/btf.h>
	14	+
	15	+#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK (BPF_F_NO_PREALLOC \| BPF_F_CLONE)
	16	+
	17	+static struct bpf_local_storage_map_bucket *
	18	+select_bucket(struct bpf_local_storage_map *smap,
	19	+ struct bpf_local_storage_elem *selem)
	20	+{
	21	+ return &smap->buckets[hash_ptr(selem, smap->bucket_log)];
	22	+}
	23	+
	24	+static int mem_charge(struct bpf_local_storage_map smap, void owner, u32 size)
	25	+{
	26	+ struct bpf_map *map = &smap->map;
	27	+
	28	+ if (!map->ops->map_local_storage_charge)
	29	+ return 0;
	30	+
	31	+ return map->ops->map_local_storage_charge(smap, owner, size);
	32	+}
	33	+
	34	+static void mem_uncharge(struct bpf_local_storage_map smap, void owner,
	35	+ u32 size)
	36	+{
	37	+ struct bpf_map *map = &smap->map;
	38	+
	39	+ if (map->ops->map_local_storage_uncharge)
	40	+ map->ops->map_local_storage_uncharge(smap, owner, size);
	41	+}
	42	+
	43	+static struct bpf_local_storage __rcu **
	44	+owner_storage(struct bpf_local_storage_map smap, void owner)
	45	+{
	46	+ struct bpf_map *map = &smap->map;
	47	+
	48	+ return map->ops->map_owner_storage_ptr(owner);
	49	+}
	50	+
	51	+static bool selem_linked_to_storage(const struct bpf_local_storage_elem *selem)
	52	+{
	53	+ return !hlist_unhashed(&selem->snode);
	54	+}
	55	+
	56	+static bool selem_linked_to_map(const struct bpf_local_storage_elem *selem)
	57	+{
	58	+ return !hlist_unhashed(&selem->map_node);
	59	+}
	60	+
	61	+struct bpf_local_storage_elem *
	62	+bpf_selem_alloc(struct bpf_local_storage_map smap, void owner,
	63	+ void *value, bool charge_mem)
	64	+{
	65	+ struct bpf_local_storage_elem *selem;
	66	+
	67	+ if (charge_mem && mem_charge(smap, owner, smap->elem_size))
	68	+ return NULL;
	69	+
	70	+ selem = kzalloc(smap->elem_size, GFP_ATOMIC \| __GFP_NOWARN);
	71	+ if (selem) {
	72	+ if (value)
	73	+ memcpy(SDATA(selem)->data, value, smap->map.value_size);
	74	+ return selem;
	75	+ }
	76	+
	77	+ if (charge_mem)
	78	+ mem_uncharge(smap, owner, smap->elem_size);
	79	+
	80	+ return NULL;
	81	+}
	82	+
	83	+/* local_storage->lock must be held and selem->local_storage == local_storage.
	84	+ * The caller must ensure selem->smap is still valid to be
	85	+ * dereferenced for its smap->elem_size and smap->cache_idx.
	86	+ */
	87	+bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage,
	88	+ struct bpf_local_storage_elem *selem,
	89	+ bool uncharge_mem)
	90	+{
	91	+ struct bpf_local_storage_map *smap;
	92	+ bool free_local_storage;
	93	+ void *owner;
	94	+
	95	+ smap = rcu_dereference(SDATA(selem)->smap);
	96	+ owner = local_storage->owner;
	97	+
	98	+ /* All uncharging on the owner must be done first.
	99	+ * The owner may be freed once the last selem is unlinked
	100	+ * from local_storage.
	101	+ */
	102	+ if (uncharge_mem)
	103	+ mem_uncharge(smap, owner, smap->elem_size);
	104	+
	105	+ free_local_storage = hlist_is_singular_node(&selem->snode,
	106	+ &local_storage->list);
	107	+ if (free_local_storage) {
	108	+ mem_uncharge(smap, owner, sizeof(struct bpf_local_storage));
	109	+ local_storage->owner = NULL;
	110	+
	111	+ /* After this RCU_INIT, owner may be freed and cannot be used */
	112	+ RCU_INIT_POINTER(*owner_storage(smap, owner), NULL);
	113	+
	114	+ /* local_storage is not freed now. local_storage->lock is
	115	+ * still held and raw_spin_unlock_bh(&local_storage->lock)
	116	+ * will be done by the caller.
	117	+ *
	118	+ * Although the unlock will be done under
	119	+ * rcu_read_lock(), it is more intutivie to
	120	+ * read if kfree_rcu(local_storage, rcu) is done
	121	+ * after the raw_spin_unlock_bh(&local_storage->lock).
	122	+ *
	123	+ * Hence, a "bool free_local_storage" is returned
	124	+ * to the caller which then calls the kfree_rcu()
	125	+ * after unlock.
	126	+ */
	127	+ }
	128	+ hlist_del_init_rcu(&selem->snode);
	129	+ if (rcu_access_pointer(local_storage->cache[smap->cache_idx]) ==
	130	+ SDATA(selem))
	131	+ RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL);
	132	+
	133	+ kfree_rcu(selem, rcu);
	134	+
	135	+ return free_local_storage;
	136	+}
	137	+
	138	+static void __bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem)
	139	+{
	140	+ struct bpf_local_storage *local_storage;
	141	+ bool free_local_storage = false;
	142	+
	143	+ if (unlikely(!selem_linked_to_storage(selem)))
	144	+ /* selem has already been unlinked from sk */
	145	+ return;
	146	+
	147	+ local_storage = rcu_dereference(selem->local_storage);
	148	+ raw_spin_lock_bh(&local_storage->lock);
	149	+ if (likely(selem_linked_to_storage(selem)))
	150	+ free_local_storage = bpf_selem_unlink_storage_nolock(
	151	+ local_storage, selem, true);
	152	+ raw_spin_unlock_bh(&local_storage->lock);
	153	+
	154	+ if (free_local_storage)
	155	+ kfree_rcu(local_storage, rcu);
	156	+}
	157	+
	158	+void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,
	159	+ struct bpf_local_storage_elem *selem)
	160	+{
	161	+ RCU_INIT_POINTER(selem->local_storage, local_storage);
	162	+ hlist_add_head(&selem->snode, &local_storage->list);
	163	+}
	164	+
	165	+void bpf_selem_unlink_map(struct bpf_local_storage_elem *selem)
	166	+{
	167	+ struct bpf_local_storage_map *smap;
	168	+ struct bpf_local_storage_map_bucket *b;
	169	+
	170	+ if (unlikely(!selem_linked_to_map(selem)))
	171	+ /* selem has already be unlinked from smap */
	172	+ return;
	173	+
	174	+ smap = rcu_dereference(SDATA(selem)->smap);
	175	+ b = select_bucket(smap, selem);
	176	+ raw_spin_lock_bh(&b->lock);
	177	+ if (likely(selem_linked_to_map(selem)))
	178	+ hlist_del_init_rcu(&selem->map_node);
	179	+ raw_spin_unlock_bh(&b->lock);
	180	+}
	181	+
	182	+void bpf_selem_link_map(struct bpf_local_storage_map *smap,
	183	+ struct bpf_local_storage_elem *selem)
	184	+{
	185	+ struct bpf_local_storage_map_bucket *b = select_bucket(smap, selem);
	186	+
	187	+ raw_spin_lock_bh(&b->lock);
	188	+ RCU_INIT_POINTER(SDATA(selem)->smap, smap);
	189	+ hlist_add_head_rcu(&selem->map_node, &b->list);
	190	+ raw_spin_unlock_bh(&b->lock);
	191	+}
	192	+
	193	+void bpf_selem_unlink(struct bpf_local_storage_elem *selem)
	194	+{
	195	+ /* Always unlink from map before unlinking from local_storage
	196	+ * because selem will be freed after successfully unlinked from
	197	+ * the local_storage.
	198	+ */
	199	+ bpf_selem_unlink_map(selem);
	200	+ __bpf_selem_unlink_storage(selem);
	201	+}
	202	+
	203	+struct bpf_local_storage_data *
	204	+bpf_local_storage_lookup(struct bpf_local_storage *local_storage,
	205	+ struct bpf_local_storage_map *smap,
	206	+ bool cacheit_lockit)
	207	+{
	208	+ struct bpf_local_storage_data *sdata;
	209	+ struct bpf_local_storage_elem *selem;
	210	+
	211	+ /* Fast path (cache hit) */
	212	+ sdata = rcu_dereference(local_storage->cache[smap->cache_idx]);
	213	+ if (sdata && rcu_access_pointer(sdata->smap) == smap)
	214	+ return sdata;
	215	+
	216	+ /* Slow path (cache miss) */
	217	+ hlist_for_each_entry_rcu(selem, &local_storage->list, snode)
	218	+ if (rcu_access_pointer(SDATA(selem)->smap) == smap)
	219	+ break;
	220	+
	221	+ if (!selem)
	222	+ return NULL;
	223	+
	224	+ sdata = SDATA(selem);
	225	+ if (cacheit_lockit) {
	226	+ /* spinlock is needed to avoid racing with the
	227	+ * parallel delete. Otherwise, publishing an already
	228	+ * deleted sdata to the cache will become a use-after-free
	229	+ * problem in the next bpf_local_storage_lookup().
	230	+ */
	231	+ raw_spin_lock_bh(&local_storage->lock);
	232	+ if (selem_linked_to_storage(selem))
	233	+ rcu_assign_pointer(local_storage->cache[smap->cache_idx],
	234	+ sdata);
	235	+ raw_spin_unlock_bh(&local_storage->lock);
	236	+ }
	237	+
	238	+ return sdata;
	239	+}
	240	+
	241	+static int check_flags(const struct bpf_local_storage_data *old_sdata,
	242	+ u64 map_flags)
	243	+{
	244	+ if (old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_NOEXIST)
	245	+ /* elem already exists */
	246	+ return -EEXIST;
	247	+
	248	+ if (!old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_EXIST)
	249	+ /* elem doesn't exist, cannot update it */
	250	+ return -ENOENT;
	251	+
	252	+ return 0;
	253	+}
	254	+
	255	+int bpf_local_storage_alloc(void *owner,
	256	+ struct bpf_local_storage_map *smap,
	257	+ struct bpf_local_storage_elem *first_selem)
	258	+{
	259	+ struct bpf_local_storage prev_storage, storage;
	260	+ struct bpf_local_storage **owner_storage_ptr;
	261	+ int err;
	262	+
	263	+ err = mem_charge(smap, owner, sizeof(*storage));
	264	+ if (err)
	265	+ return err;
	266	+
	267	+ storage = kzalloc(sizeof(*storage), GFP_ATOMIC \| __GFP_NOWARN);
	268	+ if (!storage) {
	269	+ err = -ENOMEM;
	270	+ goto uncharge;
	271	+ }
	272	+
	273	+ INIT_HLIST_HEAD(&storage->list);
	274	+ raw_spin_lock_init(&storage->lock);
	275	+ storage->owner = owner;
	276	+
	277	+ bpf_selem_link_storage_nolock(storage, first_selem);
	278	+ bpf_selem_link_map(smap, first_selem);
	279	+
	280	+ owner_storage_ptr =
	281	+ (struct bpf_local_storage **)owner_storage(smap, owner);
	282	+ /* Publish storage to the owner.
	283	+ * Instead of using any lock of the kernel object (i.e. owner),
	284	+ * cmpxchg will work with any kernel object regardless what
	285	+ * the running context is, bh, irq...etc.
	286	+ *
	287	+ * From now on, the owner->storage pointer (e.g. sk->sk_bpf_storage)
	288	+ * is protected by the storage->lock. Hence, when freeing
	289	+ * the owner->storage, the storage->lock must be held before
	290	+ * setting owner->storage ptr to NULL.
	291	+ */
	292	+ prev_storage = cmpxchg(owner_storage_ptr, NULL, storage);
	293	+ if (unlikely(prev_storage)) {
	294	+ bpf_selem_unlink_map(first_selem);
	295	+ err = -EAGAIN;
	296	+ goto uncharge;
	297	+
	298	+ /* Note that even first_selem was linked to smap's
	299	+ * bucket->list, first_selem can be freed immediately
	300	+ * (instead of kfree_rcu) because
	301	+ * bpf_local_storage_map_free() does a
	302	+ * synchronize_rcu() before walking the bucket->list.
	303	+ * Hence, no one is accessing selem from the
	304	+ * bucket->list under rcu_read_lock().
	305	+ */
	306	+ }
	307	+
	308	+ return 0;
	309	+
	310	+uncharge:
	311	+ kfree(storage);
	312	+ mem_uncharge(smap, owner, sizeof(*storage));
	313	+ return err;
	314	+}
	315	+
	316	+/* sk cannot be going away because it is linking new elem
	317	+ * to sk->sk_bpf_storage. (i.e. sk->sk_refcnt cannot be 0).
	318	+ * Otherwise, it will become a leak (and other memory issues
	319	+ * during map destruction).
	320	+ */
	321	+struct bpf_local_storage_data *
	322	+bpf_local_storage_update(void owner, struct bpf_local_storage_map smap,
	323	+ void *value, u64 map_flags)
	324	+{
	325	+ struct bpf_local_storage_data *old_sdata = NULL;
	326	+ struct bpf_local_storage_elem *selem;
	327	+ struct bpf_local_storage *local_storage;
	328	+ int err;
	329	+
	330	+ /* BPF_EXIST and BPF_NOEXIST cannot be both set */
	331	+ if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST) \|\|
	332	+ /* BPF_F_LOCK can only be used in a value with spin_lock */
	333	+ unlikely((map_flags & BPF_F_LOCK) &&
	334	+ !map_value_has_spin_lock(&smap->map)))
	335	+ return ERR_PTR(-EINVAL);
	336	+
	337	+ local_storage = rcu_dereference(*owner_storage(smap, owner));
	338	+ if (!local_storage \|\| hlist_empty(&local_storage->list)) {
	339	+ /* Very first elem for the owner */
	340	+ err = check_flags(NULL, map_flags);
	341	+ if (err)
	342	+ return ERR_PTR(err);
	343	+
	344	+ selem = bpf_selem_alloc(smap, owner, value, true);
	345	+ if (!selem)
	346	+ return ERR_PTR(-ENOMEM);
	347	+
	348	+ err = bpf_local_storage_alloc(owner, smap, selem);
	349	+ if (err) {
	350	+ kfree(selem);
	351	+ mem_uncharge(smap, owner, smap->elem_size);
	352	+ return ERR_PTR(err);
	353	+ }
	354	+
	355	+ return SDATA(selem);
	356	+ }
	357	+
	358	+ if ((map_flags & BPF_F_LOCK) && !(map_flags & BPF_NOEXIST)) {
	359	+ /* Hoping to find an old_sdata to do inline update
	360	+ * such that it can avoid taking the local_storage->lock
	361	+ * and changing the lists.
	362	+ */
	363	+ old_sdata =
	364	+ bpf_local_storage_lookup(local_storage, smap, false);
	365	+ err = check_flags(old_sdata, map_flags);
	366	+ if (err)
	367	+ return ERR_PTR(err);
	368	+ if (old_sdata && selem_linked_to_storage(SELEM(old_sdata))) {
	369	+ copy_map_value_locked(&smap->map, old_sdata->data,
	370	+ value, false);
	371	+ return old_sdata;
	372	+ }
	373	+ }
	374	+
	375	+ raw_spin_lock_bh(&local_storage->lock);
	376	+
	377	+ /* Recheck local_storage->list under local_storage->lock */
	378	+ if (unlikely(hlist_empty(&local_storage->list))) {
	379	+ /* A parallel del is happening and local_storage is going
	380	+ * away. It has just been checked before, so very
	381	+ * unlikely. Return instead of retry to keep things
	382	+ * simple.
	383	+ */
	384	+ err = -EAGAIN;
	385	+ goto unlock_err;
	386	+ }
	387	+
	388	+ old_sdata = bpf_local_storage_lookup(local_storage, smap, false);
	389	+ err = check_flags(old_sdata, map_flags);
	390	+ if (err)
	391	+ goto unlock_err;
	392	+
	393	+ if (old_sdata && (map_flags & BPF_F_LOCK)) {
	394	+ copy_map_value_locked(&smap->map, old_sdata->data, value,
	395	+ false);
	396	+ selem = SELEM(old_sdata);
	397	+ goto unlock;
	398	+ }
	399	+
	400	+ /* local_storage->lock is held. Hence, we are sure
	401	+ * we can unlink and uncharge the old_sdata successfully
	402	+ * later. Hence, instead of charging the new selem now
	403	+ * and then uncharge the old selem later (which may cause
	404	+ * a potential but unnecessary charge failure), avoid taking
	405	+ * a charge at all here (the "!old_sdata" check) and the
	406	+ * old_sdata will not be uncharged later during
	407	+ * bpf_selem_unlink_storage_nolock().
	408	+ */
	409	+ selem = bpf_selem_alloc(smap, owner, value, !old_sdata);
	410	+ if (!selem) {
	411	+ err = -ENOMEM;
	412	+ goto unlock_err;
	413	+ }
	414	+
	415	+ /* First, link the new selem to the map */
	416	+ bpf_selem_link_map(smap, selem);
	417	+
	418	+ /* Second, link (and publish) the new selem to local_storage */
	419	+ bpf_selem_link_storage_nolock(local_storage, selem);
	420	+
	421	+ /* Third, remove old selem, SELEM(old_sdata) */
	422	+ if (old_sdata) {
	423	+ bpf_selem_unlink_map(SELEM(old_sdata));
	424	+ bpf_selem_unlink_storage_nolock(local_storage, SELEM(old_sdata),
	425	+ false);
	426	+ }
	427	+
	428	+unlock:
	429	+ raw_spin_unlock_bh(&local_storage->lock);
	430	+ return SDATA(selem);
	431	+
	432	+unlock_err:
	433	+ raw_spin_unlock_bh(&local_storage->lock);
	434	+ return ERR_PTR(err);
	435	+}
	436	+
	437	+u16 bpf_local_storage_cache_idx_get(struct bpf_local_storage_cache *cache)
	438	+{
	439	+ u64 min_usage = U64_MAX;
	440	+ u16 i, res = 0;
	441	+
	442	+ spin_lock(&cache->idx_lock);
	443	+
	444	+ for (i = 0; i < BPF_LOCAL_STORAGE_CACHE_SIZE; i++) {
	445	+ if (cache->idx_usage_counts[i] < min_usage) {
	446	+ min_usage = cache->idx_usage_counts[i];
	447	+ res = i;
	448	+
	449	+ /* Found a free cache_idx */
	450	+ if (!min_usage)
	451	+ break;
	452	+ }
	453	+ }
	454	+ cache->idx_usage_counts[res]++;
	455	+
	456	+ spin_unlock(&cache->idx_lock);
	457	+
	458	+ return res;
	459	+}
	460	+
	461	+void bpf_local_storage_cache_idx_free(struct bpf_local_storage_cache *cache,
	462	+ u16 idx)
	463	+{
	464	+ spin_lock(&cache->idx_lock);
	465	+ cache->idx_usage_counts[idx]--;
	466	+ spin_unlock(&cache->idx_lock);
	467	+}
	468	+
	469	+void bpf_local_storage_map_free(struct bpf_local_storage_map *smap)
	470	+{
	471	+ struct bpf_local_storage_elem *selem;
	472	+ struct bpf_local_storage_map_bucket *b;
	473	+ unsigned int i;
	474	+
	475	+ /* Note that this map might be concurrently cloned from
	476	+ * bpf_sk_storage_clone. Wait for any existing bpf_sk_storage_clone
	477	+ * RCU read section to finish before proceeding. New RCU
	478	+ * read sections should be prevented via bpf_map_inc_not_zero.
	479	+ */
	480	+ synchronize_rcu();
	481	+
	482	+ /* bpf prog and the userspace can no longer access this map
	483	+ * now. No new selem (of this map) can be added
	484	+ * to the owner->storage or to the map bucket's list.
	485	+ *
	486	+ * The elem of this map can be cleaned up here
	487	+ * or when the storage is freed e.g.
	488	+ * by bpf_sk_storage_free() during __sk_destruct().
	489	+ */
	490	+ for (i = 0; i < (1U << smap->bucket_log); i++) {
	491	+ b = &smap->buckets[i];
	492	+
	493	+ rcu_read_lock();
	494	+ /* No one is adding to b->list now */
	495	+ while ((selem = hlist_entry_safe(
	496	+ rcu_dereference_raw(hlist_first_rcu(&b->list)),
	497	+ struct bpf_local_storage_elem, map_node))) {
	498	+ bpf_selem_unlink(selem);
	499	+ cond_resched_rcu();
	500	+ }
	501	+ rcu_read_unlock();
	502	+ }
	503	+
	504	+ /* While freeing the storage we may still need to access the map.
	505	+ *
	506	+ * e.g. when bpf_sk_storage_free() has unlinked selem from the map
	507	+ * which then made the above while((selem = ...)) loop
	508	+ * exit immediately.
	509	+ *
	510	+ * However, while freeing the storage one still needs to access the
	511	+ * smap->elem_size to do the uncharging in
	512	+ * bpf_selem_unlink_storage_nolock().
	513	+ *
	514	+ * Hence, wait another rcu grace period for the storage to be freed.
	515	+ */
	516	+ synchronize_rcu();
	517	+
	518	+ kvfree(smap->buckets);
	519	+ kfree(smap);
	520	+}
	521	+
	522	+int bpf_local_storage_map_alloc_check(union bpf_attr *attr)
	523	+{
	524	+ if (attr->map_flags & ~BPF_LOCAL_STORAGE_CREATE_FLAG_MASK \|\|
	525	+ !(attr->map_flags & BPF_F_NO_PREALLOC) \|\|
	526	+ attr->max_entries \|\|
	527	+ attr->key_size != sizeof(int) \|\| !attr->value_size \|\|
	528	+ /* Enforce BTF for userspace sk dumping */
	529	+ !attr->btf_key_type_id \|\| !attr->btf_value_type_id)
	530	+ return -EINVAL;
	531	+
	532	+ if (!bpf_capable())
	533	+ return -EPERM;
	534	+
	535	+ if (attr->value_size > BPF_LOCAL_STORAGE_MAX_VALUE_SIZE)
	536	+ return -E2BIG;
	537	+
	538	+ return 0;
	539	+}
	540	+
	541	+struct bpf_local_storage_map bpf_local_storage_map_alloc(union bpf_attr attr)
	542	+{
	543	+ struct bpf_local_storage_map *smap;
	544	+ unsigned int i;
	545	+ u32 nbuckets;
	546	+ u64 cost;
	547	+ int ret;
	548	+
	549	+ smap = kzalloc(sizeof(*smap), GFP_USER \| __GFP_NOWARN);
	550	+ if (!smap)
	551	+ return ERR_PTR(-ENOMEM);
	552	+ bpf_map_init_from_attr(&smap->map, attr);
	553	+
	554	+ nbuckets = roundup_pow_of_two(num_possible_cpus());
	555	+ /* Use at least 2 buckets, select_bucket() is undefined behavior with 1 bucket */
	556	+ nbuckets = max_t(u32, 2, nbuckets);
	557	+ smap->bucket_log = ilog2(nbuckets);
	558	+ cost = sizeof(smap->buckets) nbuckets + sizeof(*smap);
	559	+
	560	+ ret = bpf_map_charge_init(&smap->map.memory, cost);
	561	+ if (ret < 0) {
	562	+ kfree(smap);
	563	+ return ERR_PTR(ret);
	564	+ }
	565	+
	566	+ smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets,
	567	+ GFP_USER \| __GFP_NOWARN);
	568	+ if (!smap->buckets) {
	569	+ bpf_map_charge_finish(&smap->map.memory);
	570	+ kfree(smap);
	571	+ return ERR_PTR(-ENOMEM);
	572	+ }
	573	+
	574	+ for (i = 0; i < nbuckets; i++) {
	575	+ INIT_HLIST_HEAD(&smap->buckets[i].list);
	576	+ raw_spin_lock_init(&smap->buckets[i].lock);
	577	+ }
	578	+
	579	+ smap->elem_size =
	580	+ sizeof(struct bpf_local_storage_elem) + attr->value_size;
	581	+
	582	+ return smap;
	583	+}
	584	+
	585	+int bpf_local_storage_map_check_btf(const struct bpf_map *map,
	586	+ const struct btf *btf,
	587	+ const struct btf_type *key_type,
	588	+ const struct btf_type *value_type)
	589	+{
	590	+ u32 int_data;
	591	+
	592	+ if (BTF_INFO_KIND(key_type->info) != BTF_KIND_INT)
	593	+ return -EINVAL;
	594	+
	595	+ int_data = (u32 )(key_type + 1);
	596	+ if (BTF_INT_BITS(int_data) != 32 \|\| BTF_INT_OFFSET(int_data))
	597	+ return -EINVAL;
	598	+
	599	+ return 0;
	600	+}

kernel/bpf/bpf_lsm.c+20−1

@@ -11,6 +11,8 @@ · 10 unmodified lines
11	11	#include <linux/bpf_lsm.h>
12	12	#include <linux/kallsyms.h>
13	13	#include <linux/bpf_verifier.h>
	14	+#include <net/bpf_sk_storage.h>
	15	+#include <linux/bpf_local_storage.h>
14	16
15	17	/* For every LSM hook that allows attachment of BPF programs, declare a nop
16	18	* function where a BPF program can be attached.
@@ -45,10 +47,27 @@ int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog, · 28 unmodified lines
45	47	return 0;
46	48	}
47	49
	50	+static const struct bpf_func_proto *
	51	+bpf_lsm_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
	52	+{
	53	+ switch (func_id) {
	54	+ case BPF_FUNC_inode_storage_get:
	55	+ return &bpf_inode_storage_get_proto;
	56	+ case BPF_FUNC_inode_storage_delete:
	57	+ return &bpf_inode_storage_delete_proto;
	58	+ case BPF_FUNC_sk_storage_get:
	59	+ return &sk_storage_get_btf_proto;
	60	+ case BPF_FUNC_sk_storage_delete:
	61	+ return &sk_storage_delete_btf_proto;
	62	+ default:
	63	+ return tracing_prog_func_proto(func_id, prog);
	64	+ }
	65	+}
	66	+
48	67	const struct bpf_prog_ops lsm_prog_ops = {
49	68	};
50	69
51	70	const struct bpf_verifier_ops lsm_verifier_ops = {
52		− .get_func_proto = tracing_prog_func_proto,
	71	+ .get_func_proto = bpf_lsm_func_proto,
53	72	.is_valid_access = btf_ctx_access,
54	73	};

kernel/bpf/bpf_struct_ops.c+2−4

@@ -298,8 +298,7 @@ static int check_zero_holes(const struct btf_type t, void data) · 297 unmodified lines
298	298	return -EINVAL;
299	299
300	300	mtype = btf_type_by_id(btf_vmlinux, member->type);
301		− mtype = btf_resolve_size(btf_vmlinux, mtype, &msize,
302		− NULL, NULL);
	301	+ mtype = btf_resolve_size(btf_vmlinux, mtype, &msize);
303	302	if (IS_ERR(mtype))
304	303	return PTR_ERR(mtype);
305	304	prev_mend = moff + msize;
@@ -396,8 +395,7 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map map, void key, · 90 unmodified lines
396	395	u32 msize;
397	396
398	397	mtype = btf_type_by_id(btf_vmlinux, member->type);
399		− mtype = btf_resolve_size(btf_vmlinux, mtype, &msize,
400		− NULL, NULL);
	398	+ mtype = btf_resolve_size(btf_vmlinux, mtype, &msize);
401	399	if (IS_ERR(mtype)) {
402	400	err = PTR_ERR(mtype);
403	401	goto reset_unlock;

kernel/bpf/btf.c+139−24

@@ -21,6 +21,8 @@ · 20 unmodified lines
21	21	#include <linux/btf_ids.h>
22	22	#include <linux/skmsg.h>
23	23	#include <linux/perf_event.h>
	24	+#include <linux/bsearch.h>
	25	+#include <linux/btf_ids.h>
24	26	#include <net/sock.h>
25	27
26	28	/* BTF (BPF Type Format) is the meta data format which describes
@@ -1079,23 +1081,27 @@ static const struct resolve_vertex env_stack_peak(struct btf_verifier_env env) · 1052 unmodified lines
1079	1081	* type_size: (x y * sizeof(u32)). Hence, *type_size always
1080	1082	* corresponds to the return type.
1081	1083	* *elem_type: u32
	1084	+ * *elem_id: id of u32
1082	1085	* total_nelems: (x y). Hence, individual elem size is
1083	1086	* (type_size / total_nelems)
	1087	+ * *type_id: id of type if it's changed within the function, 0 if not
1084	1088	*
1085	1089	* type: is not an array (e.g. const struct X)
1086	1090	* return type: type "struct X"
1087	1091	* *type_size: sizeof(struct X)
1088	1092	* *elem_type: same as return type ("struct X")
	1093	+ * *elem_id: 0
1089	1094	* *total_nelems: 1
	1095	+ * *type_id: id of type if it's changed within the function, 0 if not
1090	1096	*/
1091		−const struct btf_type *
1092		−btf_resolve_size(const struct btf btf, const struct btf_type type,
1093		− u32 type_size, const struct btf_type *elem_type,
1094		− u32 *total_nelems)
	1097	+static const struct btf_type *
	1098	+__btf_resolve_size(const struct btf btf, const struct btf_type type,
	1099	+ u32 type_size, const struct btf_type *elem_type,
	1100	+ u32 elem_id, u32 total_nelems, u32 *type_id)
1095	1101	{
1096	1102	const struct btf_type *array_type = NULL;
1097		− const struct btf_array *array;
1098		− u32 i, size, nelems = 1;
	1103	+ const struct btf_array *array = NULL;
	1104	+ u32 i, size, nelems = 1, id = 0;
1099	1105
1100	1106	for (i = 0; i < MAX_RESOLVE_DEPTH; i++) {
1101	1107	switch (BTF_INFO_KIND(type->info)) {
@@ -1116,6 +1122,7 @@ btf_resolve_size(const struct btf btf, const struct btf_type type, · 14 unmodified lines
1116	1122	case BTF_KIND_VOLATILE:
1117	1123	case BTF_KIND_CONST:
1118	1124	case BTF_KIND_RESTRICT:
	1125	+ id = type->type;
1119	1126	type = btf_type_by_id(btf, type->type);
1120	1127	break;
1121	1128
@@ -1146,10 +1153,21 @@ resolved: · 24 unmodified lines
1146	1153	*total_nelems = nelems;
1147	1154	if (elem_type)
1148	1155	*elem_type = type;
	1156	+ if (elem_id)
	1157	+ *elem_id = array ? array->type : 0;
	1158	+ if (type_id && id)
	1159	+ *type_id = id;
1149	1160
1150	1161	return array_type ? : type;
1151	1162	}
1152	1163
	1164	+const struct btf_type *
	1165	+btf_resolve_size(const struct btf btf, const struct btf_type type,
	1166	+ u32 *type_size)
	1167	+{
	1168	+ return __btf_resolve_size(btf, type, type_size, NULL, NULL, NULL, NULL);
	1169	+}
	1170	+
1153	1171	/* The input param "type_id" must point to a needs_resolve type */
1154	1172	static const struct btf_type btf_type_id_resolve(const struct btf btf,
1155	1173	u32 *type_id)
@@ -3870,16 +3888,22 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 2714 unmodified lines
3870	3888	return true;
3871	3889	}
3872	3890
3873		−int btf_struct_access(struct bpf_verifier_log *log,
3874		− const struct btf_type *t, int off, int size,
3875		− enum bpf_access_type atype,
3876		− u32 *next_btf_id)
	3891	+enum bpf_struct_walk_result {
	3892	+ /* < 0 error */
	3893	+ WALK_SCALAR = 0,
	3894	+ WALK_PTR,
	3895	+ WALK_STRUCT,
	3896	+};
	3897	+
	3898	+static int btf_struct_walk(struct bpf_verifier_log *log,
	3899	+ const struct btf_type *t, int off, int size,
	3900	+ u32 *next_btf_id)
3877	3901	{
3878	3902	u32 i, moff, mtrue_end, msize = 0, total_nelems = 0;
3879	3903	const struct btf_type mtype, elem_type = NULL;
3880	3904	const struct btf_member *member;
3881	3905	const char tname, mname;
3882		− u32 vlen;
	3906	+ u32 vlen, elem_id, mid;
3883	3907
3884	3908	again:
3885	3909	tname = __btf_name_by_offset(btf_vmlinux, t->name_off);
@@ -3915,14 +3939,13 @@ again: · 29 unmodified lines
3915	3939	/* Only allow structure for now, can be relaxed for
3916	3940	* other types later.
3917	3941	*/
3918		− elem_type = btf_type_skip_modifiers(btf_vmlinux,
3919		− array_elem->type, NULL);
3920		− if (!btf_type_is_struct(elem_type))
	3942	+ t = btf_type_skip_modifiers(btf_vmlinux, array_elem->type,
	3943	+ NULL);
	3944	+ if (!btf_type_is_struct(t))
3921	3945	goto error;
3922	3946
3923		− off = (off - moff) % elem_type->size;
3924		− return btf_struct_access(log, elem_type, off, size, atype,
3925		− next_btf_id);
	3947	+ off = (off - moff) % t->size;
	3948	+ goto again;
3926	3949
3927	3950	error:
3928	3951	bpf_log(log, "access beyond struct %s at off %u size %u\n",
@@ -3951,7 +3974,7 @@ error: · 22 unmodified lines
3951	3974	*/
3952	3975	if (off <= moff &&
3953	3976	BITS_ROUNDUP_BYTES(end_bit) <= off + size)
3954		− return SCALAR_VALUE;
	3977	+ return WALK_SCALAR;
3955	3978
3956	3979	/* off may be accessing a following member
3957	3980	*
@@ -3973,11 +3996,13 @@ error: · 15 unmodified lines
3973	3996	break;
3974	3997
3975	3998	/* type of the field */
	3999	+ mid = member->type;
3976	4000	mtype = btf_type_by_id(btf_vmlinux, member->type);
3977	4001	mname = __btf_name_by_offset(btf_vmlinux, member->name_off);
3978	4002
3979		− mtype = btf_resolve_size(btf_vmlinux, mtype, &msize,
3980		− &elem_type, &total_nelems);
	4003	+ mtype = __btf_resolve_size(btf_vmlinux, mtype, &msize,
	4004	+ &elem_type, &elem_id, &total_nelems,
	4005	+ &mid);
3981	4006	if (IS_ERR(mtype)) {
3982	4007	bpf_log(log, "field %s doesn't have size\n", mname);
3983	4008	return -EFAULT;
@@ -3991,7 +4016,7 @@ error: · 7 unmodified lines
3991	4016	if (btf_type_is_array(mtype)) {
3992	4017	u32 elem_idx;
3993	4018
3994		− /* btf_resolve_size() above helps to
	4019	+ /* __btf_resolve_size() above helps to
3995	4020	* linearize a multi-dimensional array.
3996	4021	*
3997	4022	* The logic here is treating an array
@@ -4039,6 +4064,7 @@ error: · 41 unmodified lines
4039	4064	elem_idx = (off - moff) / msize;
4040	4065	moff += elem_idx * msize;
4041	4066	mtype = elem_type;
	4067	+ mid = elem_id;
4042	4068	}
4043	4069
4044	4070	/* the 'off' we're looking for is either equal to start
@@ -4048,6 +4074,12 @@ error: · 3 unmodified lines
4048	4074	/* our field must be inside that union or struct */
4049	4075	t = mtype;
4050	4076
	4077	+ /* return if the offset matches the member offset */
	4078	+ if (off == moff) {
	4079	+ *next_btf_id = mid;
	4080	+ return WALK_STRUCT;
	4081	+ }
	4082	+
4051	4083	/* adjust offset we're looking for */
4052	4084	off -= moff;
4053	4085	goto again;
@@ -4063,11 +4095,10 @@ error: · 9 unmodified lines
4063	4095	mname, moff, tname, off, size);
4064	4096	return -EACCES;
4065	4097	}
4066		−
4067	4098	stype = btf_type_skip_modifiers(btf_vmlinux, mtype->type, &id);
4068	4099	if (btf_type_is_struct(stype)) {
4069	4100	*next_btf_id = id;
4070		− return PTR_TO_BTF_ID;
	4101	+ return WALK_PTR;
4071	4102	}
4072	4103	}
4073	4104
@@ -4084,12 +4115,84 @@ error: · 10 unmodified lines
4084	4115	return -EACCES;
4085	4116	}
4086	4117
4087		− return SCALAR_VALUE;
	4118	+ return WALK_SCALAR;
4088	4119	}
4089	4120	bpf_log(log, "struct %s doesn't have field at offset %d\n", tname, off);
4090	4121	return -EINVAL;
4091	4122	}
4092	4123
	4124	+int btf_struct_access(struct bpf_verifier_log *log,
	4125	+ const struct btf_type *t, int off, int size,
	4126	+ enum bpf_access_type atype __maybe_unused,
	4127	+ u32 *next_btf_id)
	4128	+{
	4129	+ int err;
	4130	+ u32 id;
	4131	+
	4132	+ do {
	4133	+ err = btf_struct_walk(log, t, off, size, &id);
	4134	+
	4135	+ switch (err) {
	4136	+ case WALK_PTR:
	4137	+ /* If we found the pointer or scalar on t+off,
	4138	+ * we're done.
	4139	+ */
	4140	+ *next_btf_id = id;
	4141	+ return PTR_TO_BTF_ID;
	4142	+ case WALK_SCALAR:
	4143	+ return SCALAR_VALUE;
	4144	+ case WALK_STRUCT:
	4145	+ /* We found nested struct, so continue the search
	4146	+ * by diving in it. At this point the offset is
	4147	+ * aligned with the new type, so set it to 0.
	4148	+ */
	4149	+ t = btf_type_by_id(btf_vmlinux, id);
	4150	+ off = 0;
	4151	+ break;
	4152	+ default:
	4153	+ /* It's either error or unknown return value..
	4154	+ * scream and leave.
	4155	+ */
	4156	+ if (WARN_ONCE(err > 0, "unknown btf_struct_walk return value"))
	4157	+ return -EINVAL;
	4158	+ return err;
	4159	+ }
	4160	+ } while (t);
	4161	+
	4162	+ return -EINVAL;
	4163	+}
	4164	+
	4165	+bool btf_struct_ids_match(struct bpf_verifier_log *log,
	4166	+ int off, u32 id, u32 need_type_id)
	4167	+{
	4168	+ const struct btf_type *type;
	4169	+ int err;
	4170	+
	4171	+ /* Are we already done? */
	4172	+ if (need_type_id == id && off == 0)
	4173	+ return true;
	4174	+
	4175	+again:
	4176	+ type = btf_type_by_id(btf_vmlinux, id);
	4177	+ if (!type)
	4178	+ return false;
	4179	+ err = btf_struct_walk(log, type, off, 1, &id);
	4180	+ if (err != WALK_STRUCT)
	4181	+ return false;
	4182	+
	4183	+ /* We found nested struct object. If it matches
	4184	+ * the requested ID, we're done. Otherwise let's
	4185	+ * continue the search with offset 0 in the new
	4186	+ * type.
	4187	+ */
	4188	+ if (need_type_id != id) {
	4189	+ off = 0;
	4190	+ goto again;
	4191	+ }
	4192	+
	4193	+ return true;
	4194	+}
	4195	+
4093	4196	int btf_resolve_helper_id(struct bpf_verifier_log *log,
4094	4197	const struct bpf_func_proto *fn, int arg)
4095	4198	{
@@ -4661,3 +4764,15 @@ u32 btf_id(const struct btf *btf) · 565 unmodified lines
4661	4764	{
4662	4765	return btf->id;
4663	4766	}
	4767	+
	4768	+static int btf_id_cmp_func(const void a, const void b)
	4769	+{
	4770	+ const int pa = a, pb = b;
	4771	+
	4772	+ return pa - pb;
	4773	+}
	4774	+
	4775	+bool btf_id_set_contains(struct btf_id_set *set, u32 id)
	4776	+{
	4777	+ return bsearch(&id, set->ids, set->cnt, sizeof(u32), btf_id_cmp_func) != NULL;
	4778	+}

kernel/bpf/cpumap.c+4−8

@@ -79,8 +79,6 @@ struct bpf_cpu_map { · 78 unmodified lines
79	79
80	80	static DEFINE_PER_CPU(struct list_head, cpu_map_flush_list);
81	81
82		−static int bq_flush_to_queue(struct xdp_bulk_queue *bq);
83		−
84	82	static struct bpf_map cpu_map_alloc(union bpf_attr attr)
85	83	{
86	84	u32 value_size = attr->value_size;
@@ -658,6 +656,7 @@ static int cpu_map_get_next_key(struct bpf_map map, void key, void *next_key) · 571 unmodified lines
658	656
659	657	static int cpu_map_btf_id;
660	658	const struct bpf_map_ops cpu_map_ops = {
	659	+ .map_meta_equal = bpf_map_meta_equal,
661	660	.map_alloc = cpu_map_alloc,
662	661	.map_free = cpu_map_free,
663	662	.map_delete_elem = cpu_map_delete_elem,
@@ -669,7 +668,7 @@ const struct bpf_map_ops cpu_map_ops = { · 5 unmodified lines
669	668	.map_btf_id = &cpu_map_btf_id,
670	669	};
671	670
672		−static int bq_flush_to_queue(struct xdp_bulk_queue *bq)
	671	+static void bq_flush_to_queue(struct xdp_bulk_queue *bq)
673	672	{
674	673	struct bpf_cpu_map_entry *rcpu = bq->obj;
675	674	unsigned int processed = 0, drops = 0;
@@ -678,7 +677,7 @@ static int bq_flush_to_queue(struct xdp_bulk_queue *bq) · 2 unmodified lines
678	677	int i;
679	678
680	679	if (unlikely(!bq->count))
681		− return 0;
	680	+ return;
682	681
683	682	q = rcpu->queue;
684	683	spin_lock(&q->producer_lock);
@@ -701,13 +700,12 @@ static int bq_flush_to_queue(struct xdp_bulk_queue *bq) · 16 unmodified lines
701	700
702	701	/* Feedback loop via tracepoints */
703	702	trace_xdp_cpumap_enqueue(rcpu->map_id, processed, drops, to_cpu);
704		− return 0;
705	703	}
706	704
707	705	/* Runs under RCU-read-side, plus in softirq under NAPI protection.
708	706	* Thus, safe percpu variable access.
709	707	*/
710		−static int bq_enqueue(struct bpf_cpu_map_entry rcpu, struct xdp_frame xdpf)
	708	+static void bq_enqueue(struct bpf_cpu_map_entry rcpu, struct xdp_frame xdpf)
711	709	{
712	710	struct list_head *flush_list = this_cpu_ptr(&cpu_map_flush_list);
713	711	struct xdp_bulk_queue *bq = this_cpu_ptr(rcpu->bulkq);
@@ -728,8 +726,6 @@ static int bq_enqueue(struct bpf_cpu_map_entry rcpu, struct xdp_frame xdpf) · 14 unmodified lines
728	726
729	727	if (!bq->flush_node.prev)
730	728	list_add(&bq->flush_node, flush_list);
731		−
732		− return 0;
733	729	}
734	730
735	731	int cpu_map_enqueue(struct bpf_cpu_map_entry rcpu, struct xdp_buff xdp,

kernel/bpf/devmap.c+9−8

@@ -341,14 +341,14 @@ bool dev_map_can_have_prog(struct bpf_map *map) · 340 unmodified lines
341	341	return false;
342	342	}
343	343
344		−static int bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
	344	+static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
345	345	{
346	346	struct net_device *dev = bq->dev;
347	347	int sent = 0, drops = 0, err = 0;
348	348	int i;
349	349
350	350	if (unlikely(!bq->count))
351		− return 0;
	351	+ return;
352	352
353	353	for (i = 0; i < bq->count; i++) {
354	354	struct xdp_frame *xdpf = bq->q[i];
@@ -369,7 +369,7 @@ out: · 14 unmodified lines
369	369	trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err);
370	370	bq->dev_rx = NULL;
371	371	__list_del_clearprev(&bq->flush_node);
372		− return 0;
	372	+ return;
373	373	error:
374	374	/* If ndo_xdp_xmit fails with an errno, no frames have been
375	375	* xmit'ed and it's our responsibility to them free all.
@@ -421,8 +421,8 @@ struct bpf_dtab_netdev __dev_map_lookup_elem(struct bpf_map map, u32 key) · 45 unmodified lines
421	421	/* Runs under RCU-read-side, plus in softirq under NAPI protection.
422	422	* Thus, safe percpu variable access.
423	423	*/
424		−static int bq_enqueue(struct net_device dev, struct xdp_frame xdpf,
425		− struct net_device *dev_rx)
	424	+static void bq_enqueue(struct net_device dev, struct xdp_frame xdpf,
	425	+ struct net_device *dev_rx)
426	426	{
427	427	struct list_head *flush_list = this_cpu_ptr(&dev_flush_list);
428	428	struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq);
@@ -441,8 +441,6 @@ static int bq_enqueue(struct net_device dev, struct xdp_frame xdpf, · 12 unmodified lines
441	441
442	442	if (!bq->flush_node.prev)
443	443	list_add(&bq->flush_node, flush_list);
444		−
445		− return 0;
446	444	}
447	445
448	446	static inline int __xdp_enqueue(struct net_device dev, struct xdp_buff xdp,
@@ -462,7 +460,8 @@ static inline int __xdp_enqueue(struct net_device dev, struct xdp_buff xdp, · 13 unmodified lines
462	460	if (unlikely(!xdpf))
463	461	return -EOVERFLOW;
464	462
465		− return bq_enqueue(dev, xdpf, dev_rx);
	463	+ bq_enqueue(dev, xdpf, dev_rx);
	464	+ return 0;
466	465	}
467	466
468	467	static struct xdp_buff dev_map_run_prog(struct net_device dev,
@@ -751,6 +750,7 @@ static int dev_map_hash_update_elem(struct bpf_map map, void key, void *value, · 282 unmodified lines
751	750
752	751	static int dev_map_btf_id;
753	752	const struct bpf_map_ops dev_map_ops = {
	753	+ .map_meta_equal = bpf_map_meta_equal,
754	754	.map_alloc = dev_map_alloc,
755	755	.map_free = dev_map_free,
756	756	.map_get_next_key = dev_map_get_next_key,
@@ -764,6 +764,7 @@ const struct bpf_map_ops dev_map_ops = { · 7 unmodified lines
764	764
765	765	static int dev_map_hash_map_btf_id;
766	766	const struct bpf_map_ops dev_map_hash_ops = {
	767	+ .map_meta_equal = bpf_map_meta_equal,
767	768	.map_alloc = dev_map_alloc,
768	769	.map_free = dev_map_free,
769	770	.map_get_next_key = dev_map_hash_get_next_key,

kernel/bpf/hashtab.c+10−6

@@ -9,6 +9,7 @@ · 8 unmodified lines
9	9	#include <linux/rculist_nulls.h>
10	10	#include <linux/random.h>
11	11	#include <uapi/linux/btf.h>
	12	+#include <linux/rcupdate_trace.h>
12	13	#include "percpu_freelist.h"
13	14	#include "bpf_lru_list.h"
14	15	#include "map_in_map.h"
@@ -577,8 +578,7 @@ static void __htab_map_lookup_elem(struct bpf_map map, void *key) · 562 unmodified lines
577	578	struct htab_elem *l;
578	579	u32 hash, key_size;
579	580
580		− /* Must be called with rcu_read_lock. */
581		− WARN_ON_ONCE(!rcu_read_lock_held());
	581	+ WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
582	582
583	583	key_size = map->key_size;
584	584
@@ -941,7 +941,7 @@ static int htab_map_update_elem(struct bpf_map map, void key, void *value, · 356 unmodified lines
941	941	/* unknown flags */
942	942	return -EINVAL;
943	943
944		− WARN_ON_ONCE(!rcu_read_lock_held());
	944	+ WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
945	945
946	946	key_size = map->key_size;
947	947
@@ -1032,7 +1032,7 @@ static int htab_lru_map_update_elem(struct bpf_map map, void key, void *value, · 84 unmodified lines
1032	1032	/* unknown flags */
1033	1033	return -EINVAL;
1034	1034
1035		− WARN_ON_ONCE(!rcu_read_lock_held());
	1035	+ WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
1036	1036
1037	1037	key_size = map->key_size;
1038	1038
@@ -1220,7 +1220,7 @@ static int htab_map_delete_elem(struct bpf_map map, void key) · 181 unmodified lines
1220	1220	u32 hash, key_size;
1221	1221	int ret = -ENOENT;
1222	1222
1223		− WARN_ON_ONCE(!rcu_read_lock_held());
	1223	+ WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
1224	1224
1225	1225	key_size = map->key_size;
1226	1226
@@ -1252,7 +1252,7 @@ static int htab_lru_map_delete_elem(struct bpf_map map, void key) · 25 unmodified lines
1252	1252	u32 hash, key_size;
1253	1253	int ret = -ENOENT;
1254	1254
1255		− WARN_ON_ONCE(!rcu_read_lock_held());
	1255	+ WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
1256	1256
1257	1257	key_size = map->key_size;
1258	1258
@@ -1810,6 +1810,7 @@ static const struct bpf_iter_seq_info iter_seq_info = { · 551 unmodified lines
1810	1810
1811	1811	static int htab_map_btf_id;
1812	1812	const struct bpf_map_ops htab_map_ops = {
	1813	+ .map_meta_equal = bpf_map_meta_equal,
1813	1814	.map_alloc_check = htab_map_alloc_check,
1814	1815	.map_alloc = htab_map_alloc,
1815	1816	.map_free = htab_map_free,
@@ -1827,6 +1828,7 @@ const struct bpf_map_ops htab_map_ops = { · 11 unmodified lines
1827	1828
1828	1829	static int htab_lru_map_btf_id;
1829	1830	const struct bpf_map_ops htab_lru_map_ops = {
	1831	+ .map_meta_equal = bpf_map_meta_equal,
1830	1832	.map_alloc_check = htab_map_alloc_check,
1831	1833	.map_alloc = htab_map_alloc,
1832	1834	.map_free = htab_map_free,
@@ -1947,6 +1949,7 @@ static void htab_percpu_map_seq_show_elem(struct bpf_map map, void key, · 114 unmodified lines
1947	1949
1948	1950	static int htab_percpu_map_btf_id;
1949	1951	const struct bpf_map_ops htab_percpu_map_ops = {
	1952	+ .map_meta_equal = bpf_map_meta_equal,
1950	1953	.map_alloc_check = htab_map_alloc_check,
1951	1954	.map_alloc = htab_map_alloc,
1952	1955	.map_free = htab_map_free,
@@ -1963,6 +1966,7 @@ const struct bpf_map_ops htab_percpu_map_ops = { · 10 unmodified lines
1963	1966
1964	1967	static int htab_lru_percpu_map_btf_id;
1965	1968	const struct bpf_map_ops htab_lru_percpu_map_ops = {
	1969	+ .map_meta_equal = bpf_map_meta_equal,
1966	1970	.map_alloc_check = htab_map_alloc_check,
1967	1971	.map_alloc = htab_map_alloc,
1968	1972	.map_free = htab_map_free,

kernel/bpf/helpers.c+22−0

@@ -601,6 +601,28 @@ const struct bpf_func_proto bpf_event_output_data_proto = { · 600 unmodified lines
601	601	.arg5_type = ARG_CONST_SIZE_OR_ZERO,
602	602	};
603	603
	604	+BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
	605	+ const void __user *, user_ptr)
	606	+{
	607	+ int ret = copy_from_user(dst, user_ptr, size);
	608	+
	609	+ if (unlikely(ret)) {
	610	+ memset(dst, 0, size);
	611	+ ret = -EFAULT;
	612	+ }
	613	+
	614	+ return ret;
	615	+}
	616	+
	617	+const struct bpf_func_proto bpf_copy_from_user_proto = {
	618	+ .func = bpf_copy_from_user,
	619	+ .gpl_only = false,
	620	+ .ret_type = RET_INTEGER,
	621	+ .arg1_type = ARG_PTR_TO_UNINIT_MEM,
	622	+ .arg2_type = ARG_CONST_SIZE_OR_ZERO,
	623	+ .arg3_type = ARG_ANYTHING,
	624	+};
	625	+
604	626	const struct bpf_func_proto bpf_get_current_task_proto __weak;
605	627	const struct bpf_func_proto bpf_probe_read_user_proto __weak;
606	628	const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;

kernel/bpf/inode.c+113−3

@@ -20,6 +20,7 @@ · 19 unmodified lines
20	20	#include <linux/filter.h>
21	21	#include <linux/bpf.h>
22	22	#include <linux/bpf_trace.h>
	23	+#include "preload/bpf_preload.h"
23	24
24	25	enum bpf_type {
25	26	BPF_TYPE_UNSPEC = 0,
@@ -369,9 +370,10 @@ static struct dentry * · 343 unmodified lines
369	370	bpf_lookup(struct inode dir, struct dentry dentry, unsigned flags)
370	371	{
371	372	/* Dots in names (e.g. "/sys/fs/bpf/foo.bar") are reserved for future
372		− * extensions.
	373	+ * extensions. That allows popoulate_bpffs() create special files.
373	374	*/
374		− if (strchr(dentry->d_name.name, '.'))
	375	+ if ((dir->i_mode & S_IALLUGO) &&
	376	+ strchr(dentry->d_name.name, '.'))
375	377	return ERR_PTR(-EPERM);
376	378
377	379	return simple_lookup(dir, dentry, flags);
@@ -409,6 +411,27 @@ static const struct inode_operations bpf_dir_iops = { · 31 unmodified lines
409	411	.unlink = simple_unlink,
410	412	};
411	413
	414	+/* pin iterator link into bpffs */
	415	+static int bpf_iter_link_pin_kernel(struct dentry *parent,
	416	+ const char name, struct bpf_link link)
	417	+{
	418	+ umode_t mode = S_IFREG \| S_IRUSR;
	419	+ struct dentry *dentry;
	420	+ int ret;
	421	+
	422	+ inode_lock(parent->d_inode);
	423	+ dentry = lookup_one_len(name, parent, strlen(name));
	424	+ if (IS_ERR(dentry)) {
	425	+ inode_unlock(parent->d_inode);
	426	+ return PTR_ERR(dentry);
	427	+ }
	428	+ ret = bpf_mkobj_ops(dentry, mode, link, &bpf_link_iops,
	429	+ &bpf_iter_fops);
	430	+ dput(dentry);
	431	+ inode_unlock(parent->d_inode);
	432	+ return ret;
	433	+}
	434	+
412	435	static int bpf_obj_do_pin(const char __user pathname, void raw,
413	436	enum bpf_type type)
414	437	{
@@ -638,6 +661,91 @@ static int bpf_parse_param(struct fs_context fc, struct fs_parameter param) · 223 unmodified lines
638	661	return 0;
639	662	}
640	663
	664	+struct bpf_preload_ops *bpf_preload_ops;
	665	+EXPORT_SYMBOL_GPL(bpf_preload_ops);
	666	+
	667	+static bool bpf_preload_mod_get(void)
	668	+{
	669	+ /* If bpf_preload.ko wasn't loaded earlier then load it now.
	670	+ * When bpf_preload is built into vmlinux the module's __init
	671	+ * function will populate it.
	672	+ */
	673	+ if (!bpf_preload_ops) {
	674	+ request_module("bpf_preload");
	675	+ if (!bpf_preload_ops)
	676	+ return false;
	677	+ }
	678	+ /* And grab the reference, so the module doesn't disappear while the
	679	+ * kernel is interacting with the kernel module and its UMD.
	680	+ */
	681	+ if (!try_module_get(bpf_preload_ops->owner)) {
	682	+ pr_err("bpf_preload module get failed.\n");
	683	+ return false;
	684	+ }
	685	+ return true;
	686	+}
	687	+
	688	+static void bpf_preload_mod_put(void)
	689	+{
	690	+ if (bpf_preload_ops)
	691	+ /* now user can "rmmod bpf_preload" if necessary */
	692	+ module_put(bpf_preload_ops->owner);
	693	+}
	694	+
	695	+static DEFINE_MUTEX(bpf_preload_lock);
	696	+
	697	+static int populate_bpffs(struct dentry *parent)
	698	+{
	699	+ struct bpf_preload_info objs[BPF_PRELOAD_LINKS] = {};
	700	+ struct bpf_link *links[BPF_PRELOAD_LINKS] = {};
	701	+ int err = 0, i;
	702	+
	703	+ /* grab the mutex to make sure the kernel interactions with bpf_preload
	704	+ * UMD are serialized
	705	+ */
	706	+ mutex_lock(&bpf_preload_lock);
	707	+
	708	+ /* if bpf_preload.ko wasn't built into vmlinux then load it */
	709	+ if (!bpf_preload_mod_get())
	710	+ goto out;
	711	+
	712	+ if (!bpf_preload_ops->info.tgid) {
	713	+ /* preload() will start UMD that will load BPF iterator programs */
	714	+ err = bpf_preload_ops->preload(objs);
	715	+ if (err)
	716	+ goto out_put;
	717	+ for (i = 0; i < BPF_PRELOAD_LINKS; i++) {
	718	+ links[i] = bpf_link_by_id(objs[i].link_id);
	719	+ if (IS_ERR(links[i])) {
	720	+ err = PTR_ERR(links[i]);
	721	+ goto out_put;
	722	+ }
	723	+ }
	724	+ for (i = 0; i < BPF_PRELOAD_LINKS; i++) {
	725	+ err = bpf_iter_link_pin_kernel(parent,
	726	+ objs[i].link_name, links[i]);
	727	+ if (err)
	728	+ goto out_put;
	729	+ /* do not unlink successfully pinned links even
	730	+ * if later link fails to pin
	731	+ */
	732	+ links[i] = NULL;
	733	+ }
	734	+ /* finish() will tell UMD process to exit */
	735	+ err = bpf_preload_ops->finish();
	736	+ if (err)
	737	+ goto out_put;
	738	+ }
	739	+out_put:
	740	+ bpf_preload_mod_put();
	741	+out:
	742	+ mutex_unlock(&bpf_preload_lock);
	743	+ for (i = 0; i < BPF_PRELOAD_LINKS && err; i++)
	744	+ if (!IS_ERR_OR_NULL(links[i]))
	745	+ bpf_link_put(links[i]);
	746	+ return err;
	747	+}
	748	+
641	749	static int bpf_fill_super(struct super_block sb, struct fs_context fc)
642	750	{
643	751	static const struct tree_descr bpf_rfiles[] = { { "" } };
@@ -654,8 +762,8 @@ static int bpf_fill_super(struct super_block sb, struct fs_context fc) · 10 unmodified lines
654	762	inode = sb->s_root->d_inode;
655	763	inode->i_op = &bpf_dir_iops;
656	764	inode->i_mode &= ~S_IALLUGO;
	765	+ populate_bpffs(sb->s_root);
657	766	inode->i_mode \|= S_ISVTX \| opts->mode;
658		−
659	767	return 0;
660	768	}
661	769
@@ -705,6 +813,8 @@ static int __init bpf_init(void) · 43 unmodified lines
705	813	{
706	814	int ret;
707	815
	816	+ mutex_init(&bpf_preload_lock);
	817	+
708	818	ret = sysfs_create_mount_point(fs_kobj, "bpf");
709	819	if (ret)
710	820	return ret;

kernel/bpf/lpm_trie.c+1−0

@@ -732,6 +732,7 @@ static int trie_check_btf(const struct bpf_map *map, · 731 unmodified lines
732	732
733	733	static int trie_map_btf_id;
734	734	const struct bpf_map_ops trie_map_ops = {
	735	+ .map_meta_equal = bpf_map_meta_equal,
735	736	.map_alloc = trie_alloc,
736	737	.map_free = trie_free,
737	738	.map_get_next_key = trie_get_next_key,

kernel/bpf/map_in_map.c+9−15

@@ -17,23 +17,17 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) · 16 unmodified lines
17	17	if (IS_ERR(inner_map))
18	18	return inner_map;
19	19
20		− /* prog_array->aux->{type,jited} is a runtime binding.
21		− * Doing static check alone in the verifier is not enough.
22		− */
23		− if (inner_map->map_type == BPF_MAP_TYPE_PROG_ARRAY \|\|
24		− inner_map->map_type == BPF_MAP_TYPE_CGROUP_STORAGE \|\|
25		− inner_map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE \|\|
26		− inner_map->map_type == BPF_MAP_TYPE_STRUCT_OPS) {
27		− fdput(f);
28		− return ERR_PTR(-ENOTSUPP);
29		− }
30		−
31	20	/* Does not support >1 level map-in-map */
32	21	if (inner_map->inner_map_meta) {
33	22	fdput(f);
34	23	return ERR_PTR(-EINVAL);
35	24	}
36	25
	26	+ if (!inner_map->ops->map_meta_equal) {
	27	+ fdput(f);
	28	+ return ERR_PTR(-ENOTSUPP);
	29	+ }
	30	+
37	31	if (map_value_has_spin_lock(inner_map)) {
38	32	fdput(f);
39	33	return ERR_PTR(-ENOTSUPP);
@@ -81,15 +75,14 @@ bool bpf_map_meta_equal(const struct bpf_map *meta0, · 41 unmodified lines
81	75	return meta0->map_type == meta1->map_type &&
82	76	meta0->key_size == meta1->key_size &&
83	77	meta0->value_size == meta1->value_size &&
84		− meta0->map_flags == meta1->map_flags &&
85		− meta0->max_entries == meta1->max_entries;
	78	+ meta0->map_flags == meta1->map_flags;
86	79	}
87	80
88	81	void bpf_map_fd_get_ptr(struct bpf_map map,
89	82	struct file map_file / not used */,
90	83	int ufd)
91	84	{
92		− struct bpf_map *inner_map;
	85	+ struct bpf_map inner_map, inner_map_meta;
93	86	struct fd f;
94	87
95	88	f = fdget(ufd);
@@ -97,7 +90,8 @@ void bpf_map_fd_get_ptr(struct bpf_map map, · 1 unmodified line
97	90	if (IS_ERR(inner_map))
98	91	return inner_map;
99	92
100		− if (bpf_map_meta_equal(map->inner_map_meta, inner_map))
	93	+ inner_map_meta = map->inner_map_meta;
	94	+ if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map))
101	95	bpf_map_inc(inner_map);
102	96	else
103	97	inner_map = ERR_PTR(-EINVAL);

kernel/bpf/map_in_map.h+0−2

@@ -11,8 +11,6 @@ struct bpf_map; · 10 unmodified lines
11	11
12	12	struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd);
13	13	void bpf_map_meta_free(struct bpf_map *map_meta);
14		−bool bpf_map_meta_equal(const struct bpf_map *meta0,
15		− const struct bpf_map *meta1);
16	14	void bpf_map_fd_get_ptr(struct bpf_map map, struct file *map_file,
17	15	int ufd);
18	16	void bpf_map_fd_put_ptr(void *ptr);

kernel/bpf/map_iter.c+15−0

@@ -149,6 +149,19 @@ static void bpf_iter_detach_map(struct bpf_iter_aux_info *aux) · 148 unmodified lines
149	149	bpf_map_put_with_uref(aux->map);
150	150	}
151	151
	152	+void bpf_iter_map_show_fdinfo(const struct bpf_iter_aux_info *aux,
	153	+ struct seq_file *seq)
	154	+{
	155	+ seq_printf(seq, "map_id:\t%u\n", aux->map->id);
	156	+}
	157	+
	158	+int bpf_iter_map_fill_link_info(const struct bpf_iter_aux_info *aux,
	159	+ struct bpf_link_info *info)
	160	+{
	161	+ info->iter.map.map_id = aux->map->id;
	162	+ return 0;
	163	+}
	164	+
152	165	DEFINE_BPF_ITER_FUNC(bpf_map_elem, struct bpf_iter_meta *meta,
153	166	struct bpf_map map, void key, void *value)
154	167
@@ -156,6 +169,8 @@ static const struct bpf_iter_reg bpf_map_elem_reg_info = { · 1 unmodified line
156	169	.target = "bpf_map_elem",
157	170	.attach_target = bpf_iter_attach_map,
158	171	.detach_target = bpf_iter_detach_map,
	172	+ .show_fdinfo = bpf_iter_map_show_fdinfo,
	173	+ .fill_link_info = bpf_iter_map_fill_link_info,
159	174	.ctx_arg_info_size = 2,
160	175	.ctx_arg_info = {
161	176	{ offsetof(struct bpf_iter__bpf_map_elem, key),

kernel/bpf/preload/Kconfig+26−0

@@ -0,0 +1,26 @@
	1	+# SPDX-License-Identifier: GPL-2.0-only
	2	+config USERMODE_DRIVER
	3	+ bool
	4	+ default n
	5	+
	6	+menuconfig BPF_PRELOAD
	7	+ bool "Preload BPF file system with kernel specific program and map iterators"
	8	+ depends on BPF
	9	+ # The dependency on !COMPILE_TEST prevents it from being enabled
	10	+ # in allmodconfig or allyesconfig configurations
	11	+ depends on !COMPILE_TEST
	12	+ select USERMODE_DRIVER
	13	+ help
	14	+ This builds kernel module with several embedded BPF programs that are
	15	+ pinned into BPF FS mount point as human readable files that are
	16	+ useful in debugging and introspection of BPF programs and maps.
	17	+
	18	+if BPF_PRELOAD
	19	+config BPF_PRELOAD_UMD
	20	+ tristate "bpf_preload kernel module with user mode driver"
	21	+ depends on CC_CAN_LINK
	22	+ depends on m \|\| CC_CAN_LINK_STATIC
	23	+ default m
	24	+ help
	25	+ This builds bpf_preload kernel module with embedded user mode driver.
	26	+endif

kernel/bpf/preload/Makefile+23−0

@@ -0,0 +1,23 @@
	1	+# SPDX-License-Identifier: GPL-2.0
	2	+
	3	+LIBBPF_SRCS = $(srctree)/tools/lib/bpf/
	4	+LIBBPF_A = $(obj)/libbpf.a
	5	+LIBBPF_OUT = $(abspath $(obj))
	6	+
	7	+$(LIBBPF_A):
	8	+ $(Q)$(MAKE) -C $(LIBBPF_SRCS) OUTPUT=$(LIBBPF_OUT)/ $(LIBBPF_OUT)/libbpf.a
	9	+
	10	+userccflags += -I $(srctree)/tools/include/ -I $(srctree)/tools/include/uapi \
	11	+ -I $(srctree)/tools/lib/ -Wno-unused-result
	12	+
	13	+userprogs := bpf_preload_umd
	14	+
	15	+bpf_preload_umd-objs := iterators/iterators.o
	16	+bpf_preload_umd-userldlibs := $(LIBBPF_A) -lelf -lz
	17	+
	18	+$(obj)/bpf_preload_umd: $(LIBBPF_A)
	19	+
	20	+$(obj)/bpf_preload_umd_blob.o: $(obj)/bpf_preload_umd
	21	+
	22	+obj-$(CONFIG_BPF_PRELOAD_UMD) += bpf_preload.o
	23	+bpf_preload-objs += bpf_preload_kern.o bpf_preload_umd_blob.o

kernel/bpf/preload/bpf_preload.h+16−0

@@ -0,0 +1,16 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 */
	2	+#ifndef _BPF_PRELOAD_H
	3	+#define _BPF_PRELOAD_H
	4	+
	5	+#include <linux/usermode_driver.h>
	6	+#include "iterators/bpf_preload_common.h"
	7	+
	8	+struct bpf_preload_ops {
	9	+ struct umd_info info;
	10	+ int (preload)(struct bpf_preload_info );
	11	+ int (*finish)(void);
	12	+ struct module *owner;
	13	+};
	14	+extern struct bpf_preload_ops *bpf_preload_ops;
	15	+#define BPF_PRELOAD_LINKS 2
	16	+#endif

kernel/bpf/preload/bpf_preload_kern.c+91−0

@@ -0,0 +1,91 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
	3	+#include <linux/init.h>
	4	+#include <linux/module.h>
	5	+#include <linux/pid.h>
	6	+#include <linux/fs.h>
	7	+#include <linux/sched/signal.h>
	8	+#include "bpf_preload.h"
	9	+
	10	+extern char bpf_preload_umd_start;
	11	+extern char bpf_preload_umd_end;
	12	+
	13	+static int preload(struct bpf_preload_info *obj);
	14	+static int finish(void);
	15	+
	16	+static struct bpf_preload_ops umd_ops = {
	17	+ .info.driver_name = "bpf_preload",
	18	+ .preload = preload,
	19	+ .finish = finish,
	20	+ .owner = THIS_MODULE,
	21	+};
	22	+
	23	+static int preload(struct bpf_preload_info *obj)
	24	+{
	25	+ int magic = BPF_PRELOAD_START;
	26	+ loff_t pos = 0;
	27	+ int i, err;
	28	+ ssize_t n;
	29	+
	30	+ err = fork_usermode_driver(&umd_ops.info);
	31	+ if (err)
	32	+ return err;
	33	+
	34	+ /* send the start magic to let UMD proceed with loading BPF progs */
	35	+ n = kernel_write(umd_ops.info.pipe_to_umh,
	36	+ &magic, sizeof(magic), &pos);
	37	+ if (n != sizeof(magic))
	38	+ return -EPIPE;
	39	+
	40	+ /* receive bpf_link IDs and names from UMD */
	41	+ pos = 0;
	42	+ for (i = 0; i < BPF_PRELOAD_LINKS; i++) {
	43	+ n = kernel_read(umd_ops.info.pipe_from_umh,
	44	+ &obj[i], sizeof(*obj), &pos);
	45	+ if (n != sizeof(*obj))
	46	+ return -EPIPE;
	47	+ }
	48	+ return 0;
	49	+}
	50	+
	51	+static int finish(void)
	52	+{
	53	+ int magic = BPF_PRELOAD_END;
	54	+ struct pid *tgid;
	55	+ loff_t pos = 0;
	56	+ ssize_t n;
	57	+
	58	+ /* send the last magic to UMD. It will do a normal exit. */
	59	+ n = kernel_write(umd_ops.info.pipe_to_umh,
	60	+ &magic, sizeof(magic), &pos);
	61	+ if (n != sizeof(magic))
	62	+ return -EPIPE;
	63	+ tgid = umd_ops.info.tgid;
	64	+ wait_event(tgid->wait_pidfd, thread_group_exited(tgid));
	65	+ umd_ops.info.tgid = NULL;
	66	+ return 0;
	67	+}
	68	+
	69	+static int __init load_umd(void)
	70	+{
	71	+ int err;
	72	+
	73	+ err = umd_load_blob(&umd_ops.info, &bpf_preload_umd_start,
	74	+ &bpf_preload_umd_end - &bpf_preload_umd_start);
	75	+ if (err)
	76	+ return err;
	77	+ bpf_preload_ops = &umd_ops;
	78	+ return err;
	79	+}
	80	+
	81	+static void __exit fini_umd(void)
	82	+{
	83	+ bpf_preload_ops = NULL;
	84	+ /* kill UMD in case it's still there due to earlier error */
	85	+ kill_pid(umd_ops.info.tgid, SIGKILL, 1);
	86	+ umd_ops.info.tgid = NULL;
	87	+ umd_unload_blob(&umd_ops.info);
	88	+}
	89	+late_initcall(load_umd);
	90	+module_exit(fini_umd);
	91	+MODULE_LICENSE("GPL");

kernel/bpf/preload/bpf_preload_umd_blob.S+7−0

@@ -0,0 +1,7 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 */
	2	+ .section .init.rodata, "a"
	3	+ .global bpf_preload_umd_start
	4	+bpf_preload_umd_start:
	5	+ .incbin "kernel/bpf/preload/bpf_preload_umd"
	6	+ .global bpf_preload_umd_end
	7	+bpf_preload_umd_end:

kernel/bpf/preload/iterators/.gitignore+2−0

@@ -0,0 +1,2 @@
	1	+# SPDX-License-Identifier: GPL-2.0-only
	2	+/.output

kernel/bpf/preload/iterators/Makefile+57−0

@@ -0,0 +1,57 @@
	1	+# SPDX-License-Identifier: GPL-2.0
	2	+OUTPUT := .output
	3	+CLANG ?= clang
	4	+LLC ?= llc
	5	+LLVM_STRIP ?= llvm-strip
	6	+DEFAULT_BPFTOOL := $(OUTPUT)/sbin/bpftool
	7	+BPFTOOL ?= $(DEFAULT_BPFTOOL)
	8	+LIBBPF_SRC := $(abspath ../../../../tools/lib/bpf)
	9	+BPFOBJ := $(OUTPUT)/libbpf.a
	10	+BPF_INCLUDE := $(OUTPUT)
	11	+INCLUDES := -I$(OUTPUT) -I$(BPF_INCLUDE) -I$(abspath ../../../../tools/lib) \
	12	+ -I$(abspath ../../../../tools/include/uapi)
	13	+CFLAGS := -g -Wall
	14	+
	15	+abs_out := $(abspath $(OUTPUT))
	16	+ifeq ($(V),1)
	17	+Q =
	18	+msg =
	19	+else
	20	+Q = @
	21	+msg = @printf ' %-8s %s%s\n' "$(1)" "$(notdir $(2))" "$(if $(3), $(3))";
	22	+MAKEFLAGS += --no-print-directory
	23	+submake_extras := feature_display=0
	24	+endif
	25	+
	26	+.DELETE_ON_ERROR:
	27	+
	28	+.PHONY: all clean
	29	+
	30	+all: iterators.skel.h
	31	+
	32	+clean:
	33	+ $(call msg,CLEAN)
	34	+ $(Q)rm -rf $(OUTPUT) iterators
	35	+
	36	+iterators.skel.h: $(OUTPUT)/iterators.bpf.o \| $(BPFTOOL)
	37	+ $(call msg,GEN-SKEL,$@)
	38	+ $(Q)$(BPFTOOL) gen skeleton $< > $@
	39	+
	40	+
	41	+$(OUTPUT)/iterators.bpf.o: iterators.bpf.c $(BPFOBJ) \| $(OUTPUT)
	42	+ $(call msg,BPF,$@)
	43	+ $(Q)$(CLANG) -g -O2 -target bpf $(INCLUDES) \
	44	+ -c $(filter %.c,$^) -o $@ && \
	45	+ $(LLVM_STRIP) -g $@
	46	+
	47	+$(OUTPUT):
	48	+ $(call msg,MKDIR,$@)
	49	+ $(Q)mkdir -p $(OUTPUT)
	50	+
	51	+$(BPFOBJ): $(wildcard $(LIBBPF_SRC)/*.[ch] $(LIBBPF_SRC)/Makefile) \| $(OUTPUT)
	52	+ $(Q)$(MAKE) $(submake_extras) -C $(LIBBPF_SRC) \
	53	+ OUTPUT=$(abspath $(dir $@))/ $(abspath $@)
	54	+
	55	+$(DEFAULT_BPFTOOL):
	56	+ $(Q)$(MAKE) $(submake_extras) -C ../../../../tools/bpf/bpftool \
	57	+ prefix= OUTPUT=$(abs_out)/ DESTDIR=$(abs_out) install

kernel/bpf/preload/iterators/README+4−0

@@ -0,0 +1,4 @@
	1	+WARNING:
	2	+If you change "iterators.bpf.c" do "make -j" in this directory to rebuild "iterators.skel.h".
	3	+Make sure to have clang 10 installed.
	4	+See Documentation/bpf/bpf_devel_QA.rst

kernel/bpf/preload/iterators/bpf_preload_common.h+13−0

@@ -0,0 +1,13 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 */
	2	+#ifndef _BPF_PRELOAD_COMMON_H
	3	+#define _BPF_PRELOAD_COMMON_H
	4	+
	5	+#define BPF_PRELOAD_START 0x5555
	6	+#define BPF_PRELOAD_END 0xAAAA
	7	+
	8	+struct bpf_preload_info {
	9	+ char link_name[16];
	10	+ int link_id;
	11	+};
	12	+
	13	+#endif

kernel/bpf/preload/iterators/iterators.bpf.c+114−0

@@ -0,0 +1,114 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+/* Copyright (c) 2020 Facebook */
	3	+#include <linux/bpf.h>
	4	+#include <bpf/bpf_helpers.h>
	5	+#include <bpf/bpf_tracing.h>
	6	+#include <bpf/bpf_core_read.h>
	7	+
	8	+#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
	9	+struct seq_file;
	10	+struct bpf_iter_meta {
	11	+ struct seq_file *seq;
	12	+ __u64 session_id;
	13	+ __u64 seq_num;
	14	+};
	15	+
	16	+struct bpf_map {
	17	+ __u32 id;
	18	+ char name[16];
	19	+ __u32 max_entries;
	20	+};
	21	+
	22	+struct bpf_iter__bpf_map {
	23	+ struct bpf_iter_meta *meta;
	24	+ struct bpf_map *map;
	25	+};
	26	+
	27	+struct btf_type {
	28	+ __u32 name_off;
	29	+};
	30	+
	31	+struct btf_header {
	32	+ __u32 str_len;
	33	+};
	34	+
	35	+struct btf {
	36	+ const char *strings;
	37	+ struct btf_type **types;
	38	+ struct btf_header hdr;
	39	+};
	40	+
	41	+struct bpf_prog_aux {
	42	+ __u32 id;
	43	+ char name[16];
	44	+ const char *attach_func_name;
	45	+ struct bpf_prog *linked_prog;
	46	+ struct bpf_func_info *func_info;
	47	+ struct btf *btf;
	48	+};
	49	+
	50	+struct bpf_prog {
	51	+ struct bpf_prog_aux *aux;
	52	+};
	53	+
	54	+struct bpf_iter__bpf_prog {
	55	+ struct bpf_iter_meta *meta;
	56	+ struct bpf_prog *prog;
	57	+};
	58	+#pragma clang attribute pop
	59	+
	60	+static const char get_name(struct btf btf, long btf_id, const char *fallback)
	61	+{
	62	+ struct btf_type *types, t;
	63	+ unsigned int name_off;
	64	+ const char *str;
	65	+
	66	+ if (!btf)
	67	+ return fallback;
	68	+ str = btf->strings;
	69	+ types = btf->types;
	70	+ bpf_probe_read_kernel(&t, sizeof(t), types + btf_id);
	71	+ name_off = BPF_CORE_READ(t, name_off);
	72	+ if (name_off >= btf->hdr.str_len)
	73	+ return fallback;
	74	+ return str + name_off;
	75	+}
	76	+
	77	+SEC("iter/bpf_map")
	78	+int dump_bpf_map(struct bpf_iter__bpf_map *ctx)
	79	+{
	80	+ struct seq_file *seq = ctx->meta->seq;
	81	+ __u64 seq_num = ctx->meta->seq_num;
	82	+ struct bpf_map *map = ctx->map;
	83	+
	84	+ if (!map)
	85	+ return 0;
	86	+
	87	+ if (seq_num == 0)
	88	+ BPF_SEQ_PRINTF(seq, " id name max_entries\n");
	89	+
	90	+ BPF_SEQ_PRINTF(seq, "%4u %-16s%6d\n", map->id, map->name, map->max_entries);
	91	+ return 0;
	92	+}
	93	+
	94	+SEC("iter/bpf_prog")
	95	+int dump_bpf_prog(struct bpf_iter__bpf_prog *ctx)
	96	+{
	97	+ struct seq_file *seq = ctx->meta->seq;
	98	+ __u64 seq_num = ctx->meta->seq_num;
	99	+ struct bpf_prog *prog = ctx->prog;
	100	+ struct bpf_prog_aux *aux;
	101	+
	102	+ if (!prog)
	103	+ return 0;
	104	+
	105	+ aux = prog->aux;
	106	+ if (seq_num == 0)
	107	+ BPF_SEQ_PRINTF(seq, " id name attached\n");
	108	+
	109	+ BPF_SEQ_PRINTF(seq, "%4u %-16s %s %s\n", aux->id,
	110	+ get_name(aux->btf, aux->func_info[0].type_id, aux->name),
	111	+ aux->attach_func_name, aux->linked_prog->aux->name);
	112	+ return 0;
	113	+}
	114	+char LICENSE[] SEC("license") = "GPL";

kernel/bpf/preload/iterators/iterators.c+94−0

@@ -0,0 +1,94 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+/* Copyright (c) 2020 Facebook */
	3	+#include <argp.h>
	4	+#include <stdio.h>
	5	+#include <stdlib.h>
	6	+#include <string.h>
	7	+#include <unistd.h>
	8	+#include <fcntl.h>
	9	+#include <sys/resource.h>
	10	+#include <bpf/libbpf.h>
	11	+#include <bpf/bpf.h>
	12	+#include <sys/mount.h>
	13	+#include "iterators.skel.h"
	14	+#include "bpf_preload_common.h"
	15	+
	16	+int to_kernel = -1;
	17	+int from_kernel = 0;
	18	+
	19	+static int send_link_to_kernel(struct bpf_link link, const char link_name)
	20	+{
	21	+ struct bpf_preload_info obj = {};
	22	+ struct bpf_link_info info = {};
	23	+ __u32 info_len = sizeof(info);
	24	+ int err;
	25	+
	26	+ err = bpf_obj_get_info_by_fd(bpf_link__fd(link), &info, &info_len);
	27	+ if (err)
	28	+ return err;
	29	+ obj.link_id = info.id;
	30	+ if (strlen(link_name) >= sizeof(obj.link_name))
	31	+ return -E2BIG;
	32	+ strcpy(obj.link_name, link_name);
	33	+ if (write(to_kernel, &obj, sizeof(obj)) != sizeof(obj))
	34	+ return -EPIPE;
	35	+ return 0;
	36	+}
	37	+
	38	+int main(int argc, char **argv)
	39	+{
	40	+ struct rlimit rlim = { RLIM_INFINITY, RLIM_INFINITY };
	41	+ struct iterators_bpf *skel;
	42	+ int err, magic;
	43	+ int debug_fd;
	44	+
	45	+ debug_fd = open("/dev/console", O_WRONLY \| O_NOCTTY \| O_CLOEXEC);
	46	+ if (debug_fd < 0)
	47	+ return 1;
	48	+ to_kernel = dup(1);
	49	+ close(1);
	50	+ dup(debug_fd);
	51	+ /* now stdin and stderr point to /dev/console */
	52	+
	53	+ read(from_kernel, &magic, sizeof(magic));
	54	+ if (magic != BPF_PRELOAD_START) {
	55	+ printf("bad start magic %d\n", magic);
	56	+ return 1;
	57	+ }
	58	+ setrlimit(RLIMIT_MEMLOCK, &rlim);
	59	+ /* libbpf opens BPF object and loads it into the kernel */
	60	+ skel = iterators_bpf__open_and_load();
	61	+ if (!skel) {
	62	+ /* iterators.skel.h is little endian.
	63	+ * libbpf doesn't support automatic little->big conversion
	64	+ * of BPF bytecode yet.
	65	+ * The program load will fail in such case.
	66	+ */
	67	+ printf("Failed load could be due to wrong endianness\n");
	68	+ return 1;
	69	+ }
	70	+ err = iterators_bpf__attach(skel);
	71	+ if (err)
	72	+ goto cleanup;
	73	+
	74	+ /* send two bpf_link IDs with names to the kernel */
	75	+ err = send_link_to_kernel(skel->links.dump_bpf_map, "maps.debug");
	76	+ if (err)
	77	+ goto cleanup;
	78	+ err = send_link_to_kernel(skel->links.dump_bpf_prog, "progs.debug");
	79	+ if (err)
	80	+ goto cleanup;
	81	+
	82	+ /* The kernel will proceed with pinnging the links in bpffs.
	83	+ * UMD will wait on read from pipe.
	84	+ */
	85	+ read(from_kernel, &magic, sizeof(magic));
	86	+ if (magic != BPF_PRELOAD_END) {
	87	+ printf("bad final magic %d\n", magic);
	88	+ err = -EINVAL;
	89	+ }
	90	+cleanup:
	91	+ iterators_bpf__destroy(skel);
	92	+
	93	+ return err != 0;
	94	+}

kernel/bpf/preload/iterators/iterators.skel.h+410−0

@@ -0,0 +1,410 @@
	1	+/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
	2	+
	3	+/* THIS FILE IS AUTOGENERATED! */
	4	+#ifndef __ITERATORS_BPF_SKEL_H__
	5	+#define __ITERATORS_BPF_SKEL_H__
	6	+
	7	+#include <stdlib.h>
	8	+#include <bpf/libbpf.h>
	9	+
	10	+struct iterators_bpf {
	11	+ struct bpf_object_skeleton *skeleton;
	12	+ struct bpf_object *obj;
	13	+ struct {
	14	+ struct bpf_map *rodata;
	15	+ } maps;
	16	+ struct {
	17	+ struct bpf_program *dump_bpf_map;
	18	+ struct bpf_program *dump_bpf_prog;
	19	+ } progs;
	20	+ struct {
	21	+ struct bpf_link *dump_bpf_map;
	22	+ struct bpf_link *dump_bpf_prog;
	23	+ } links;
	24	+ struct iterators_bpf__rodata {
	25	+ char dump_bpf_map____fmt[35];
	26	+ char dump_bpf_map____fmt_1[14];
	27	+ char dump_bpf_prog____fmt[32];
	28	+ char dump_bpf_prog____fmt_2[17];
	29	+ } *rodata;
	30	+};
	31	+
	32	+static void
	33	+iterators_bpf__destroy(struct iterators_bpf *obj)
	34	+{
	35	+ if (!obj)
	36	+ return;
	37	+ if (obj->skeleton)
	38	+ bpf_object__destroy_skeleton(obj->skeleton);
	39	+ free(obj);
	40	+}
	41	+
	42	+static inline int
	43	+iterators_bpf__create_skeleton(struct iterators_bpf *obj);
	44	+
	45	+static inline struct iterators_bpf *
	46	+iterators_bpf__open_opts(const struct bpf_object_open_opts *opts)
	47	+{
	48	+ struct iterators_bpf *obj;
	49	+
	50	+ obj = (typeof(obj))calloc(1, sizeof(*obj));
	51	+ if (!obj)
	52	+ return NULL;
	53	+ if (iterators_bpf__create_skeleton(obj))
	54	+ goto err;
	55	+ if (bpf_object__open_skeleton(obj->skeleton, opts))
	56	+ goto err;
	57	+
	58	+ return obj;
	59	+err:
	60	+ iterators_bpf__destroy(obj);
	61	+ return NULL;
	62	+}
	63	+
	64	+static inline struct iterators_bpf *
	65	+iterators_bpf__open(void)
	66	+{
	67	+ return iterators_bpf__open_opts(NULL);
	68	+}
	69	+
	70	+static inline int
	71	+iterators_bpf__load(struct iterators_bpf *obj)
	72	+{
	73	+ return bpf_object__load_skeleton(obj->skeleton);
	74	+}
	75	+
	76	+static inline struct iterators_bpf *
	77	+iterators_bpf__open_and_load(void)
	78	+{
	79	+ struct iterators_bpf *obj;
	80	+
	81	+ obj = iterators_bpf__open();
	82	+ if (!obj)
	83	+ return NULL;
	84	+ if (iterators_bpf__load(obj)) {
	85	+ iterators_bpf__destroy(obj);
	86	+ return NULL;
	87	+ }
	88	+ return obj;
	89	+}
	90	+
	91	+static inline int
	92	+iterators_bpf__attach(struct iterators_bpf *obj)
	93	+{
	94	+ return bpf_object__attach_skeleton(obj->skeleton);
	95	+}
	96	+
	97	+static inline void
	98	+iterators_bpf__detach(struct iterators_bpf *obj)
	99	+{
	100	+ return bpf_object__detach_skeleton(obj->skeleton);
	101	+}
	102	+
	103	+static inline int
	104	+iterators_bpf__create_skeleton(struct iterators_bpf *obj)
	105	+{
	106	+ struct bpf_object_skeleton *s;
	107	+
	108	+ s = (typeof(s))calloc(1, sizeof(*s));
	109	+ if (!s)
	110	+ return -1;
	111	+ obj->skeleton = s;
	112	+
	113	+ s->sz = sizeof(*s);
	114	+ s->name = "iterators_bpf";
	115	+ s->obj = &obj->obj;
	116	+
	117	+ /* maps */
	118	+ s->map_cnt = 1;
	119	+ s->map_skel_sz = sizeof(*s->maps);
	120	+ s->maps = (typeof(s->maps))calloc(s->map_cnt, s->map_skel_sz);
	121	+ if (!s->maps)
	122	+ goto err;
	123	+
	124	+ s->maps[0].name = "iterator.rodata";
	125	+ s->maps[0].map = &obj->maps.rodata;
	126	+ s->maps[0].mmaped = (void **)&obj->rodata;
	127	+
	128	+ /* programs */
	129	+ s->prog_cnt = 2;
	130	+ s->prog_skel_sz = sizeof(*s->progs);
	131	+ s->progs = (typeof(s->progs))calloc(s->prog_cnt, s->prog_skel_sz);
	132	+ if (!s->progs)
	133	+ goto err;
	134	+
	135	+ s->progs[0].name = "dump_bpf_map";
	136	+ s->progs[0].prog = &obj->progs.dump_bpf_map;
	137	+ s->progs[0].link = &obj->links.dump_bpf_map;
	138	+
	139	+ s->progs[1].name = "dump_bpf_prog";
	140	+ s->progs[1].prog = &obj->progs.dump_bpf_prog;
	141	+ s->progs[1].link = &obj->links.dump_bpf_prog;
	142	+
	143	+ s->data_sz = 7128;
	144	+ s->data = (void *)"\
	145	+\x7f\x45\x4c\x46\x02\x01\x01\0\0\0\0\0\0\0\0\0\x01\0\xf7\0\x01\0\0\0\0\0\0\0\0\
	146	+\0\0\0\0\0\0\0\0\0\0\0\x18\x18\0\0\0\0\0\0\0\0\0\0\x40\0\0\0\0\0\x40\0\x0f\0\
	147	+\x0e\0\x79\x12\0\0\0\0\0\0\x79\x26\0\0\0\0\0\0\x79\x17\x08\0\0\0\0\0\x15\x07\
	148	+\x1a\0\0\0\0\0\x79\x21\x10\0\0\0\0\0\x55\x01\x08\0\0\0\0\0\xbf\xa4\0\0\0\0\0\0\
	149	+\x07\x04\0\0\xe8\xff\xff\xff\xbf\x61\0\0\0\0\0\0\x18\x02\0\0\0\0\0\0\0\0\0\0\0\
	150	+\0\0\0\xb7\x03\0\0\x23\0\0\0\xb7\x05\0\0\0\0\0\0\x85\0\0\0\x7e\0\0\0\x61\x71\0\
	151	+\0\0\0\0\0\x7b\x1a\xe8\xff\0\0\0\0\xb7\x01\0\0\x04\0\0\0\xbf\x72\0\0\0\0\0\0\
	152	+\x0f\x12\0\0\0\0\0\0\x7b\x2a\xf0\xff\0\0\0\0\x61\x71\x14\0\0\0\0\0\x7b\x1a\xf8\
	153	+\xff\0\0\0\0\xbf\xa4\0\0\0\0\0\0\x07\x04\0\0\xe8\xff\xff\xff\xbf\x61\0\0\0\0\0\
	154	+\0\x18\x02\0\0\x23\0\0\0\0\0\0\0\0\0\0\0\xb7\x03\0\0\x0e\0\0\0\xb7\x05\0\0\x18\
	155	+\0\0\0\x85\0\0\0\x7e\0\0\0\xb7\0\0\0\0\0\0\0\x95\0\0\0\0\0\0\0\x79\x12\0\0\0\0\
	156	+\0\0\x79\x26\0\0\0\0\0\0\x79\x11\x08\0\0\0\0\0\x15\x01\x3b\0\0\0\0\0\x79\x17\0\
	157	+\0\0\0\0\0\x79\x21\x10\0\0\0\0\0\x55\x01\x08\0\0\0\0\0\xbf\xa4\0\0\0\0\0\0\x07\
	158	+\x04\0\0\xd0\xff\xff\xff\xbf\x61\0\0\0\0\0\0\x18\x02\0\0\x31\0\0\0\0\0\0\0\0\0\
	159	+\0\0\xb7\x03\0\0\x20\0\0\0\xb7\x05\0\0\0\0\0\0\x85\0\0\0\x7e\0\0\0\x7b\x6a\xc8\
	160	+\xff\0\0\0\0\x61\x71\0\0\0\0\0\0\x7b\x1a\xd0\xff\0\0\0\0\xb7\x03\0\0\x04\0\0\0\
	161	+\xbf\x79\0\0\0\0\0\0\x0f\x39\0\0\0\0\0\0\x79\x71\x28\0\0\0\0\0\x79\x78\x30\0\0\
	162	+\0\0\0\x15\x08\x18\0\0\0\0\0\xb7\x02\0\0\0\0\0\0\x0f\x21\0\0\0\0\0\0\x61\x11\
	163	+\x04\0\0\0\0\0\x79\x83\x08\0\0\0\0\0\x67\x01\0\0\x03\0\0\0\x0f\x13\0\0\0\0\0\0\
	164	+\x79\x86\0\0\0\0\0\0\xbf\xa1\0\0\0\0\0\0\x07\x01\0\0\xf8\xff\xff\xff\xb7\x02\0\
	165	+\0\x08\0\0\0\x85\0\0\0\x71\0\0\0\xb7\x01\0\0\0\0\0\0\x79\xa3\xf8\xff\0\0\0\0\
	166	+\x0f\x13\0\0\0\0\0\0\xbf\xa1\0\0\0\0\0\0\x07\x01\0\0\xf4\xff\xff\xff\xb7\x02\0\
	167	+\0\x04\0\0\0\x85\0\0\0\x04\0\0\0\xb7\x03\0\0\x04\0\0\0\x61\xa1\xf4\xff\0\0\0\0\
	168	+\x61\x82\x10\0\0\0\0\0\x3d\x21\x02\0\0\0\0\0\x0f\x16\0\0\0\0\0\0\xbf\x69\0\0\0\
	169	+\0\0\0\x7b\x9a\xd8\xff\0\0\0\0\x79\x71\x18\0\0\0\0\0\x7b\x1a\xe0\xff\0\0\0\0\
	170	+\x79\x71\x20\0\0\0\0\0\x79\x11\0\0\0\0\0\0\x0f\x31\0\0\0\0\0\0\x7b\x1a\xe8\xff\
	171	+\0\0\0\0\xbf\xa4\0\0\0\0\0\0\x07\x04\0\0\xd0\xff\xff\xff\x79\xa1\xc8\xff\0\0\0\
	172	+\0\x18\x02\0\0\x51\0\0\0\0\0\0\0\0\0\0\0\xb7\x03\0\0\x11\0\0\0\xb7\x05\0\0\x20\
	173	+\0\0\0\x85\0\0\0\x7e\0\0\0\xb7\0\0\0\0\0\0\0\x95\0\0\0\0\0\0\0\x20\x20\x69\x64\
	174	+\x20\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x6d\
	175	+\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\x0a\0\x25\x34\x75\x20\x25\x2d\x31\x36\
	176	+\x73\x25\x36\x64\x0a\0\x20\x20\x69\x64\x20\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\
	177	+\x20\x20\x20\x20\x20\x20\x20\x20\x61\x74\x74\x61\x63\x68\x65\x64\x0a\0\x25\x34\
	178	+\x75\x20\x25\x2d\x31\x36\x73\x20\x25\x73\x20\x25\x73\x0a\0\x47\x50\x4c\0\x9f\
	179	+\xeb\x01\0\x18\0\0\0\0\0\0\0\x1c\x04\0\0\x1c\x04\0\0\0\x05\0\0\0\0\0\0\0\0\0\
	180	+\x02\x02\0\0\0\x01\0\0\0\x02\0\0\x04\x10\0\0\0\x13\0\0\0\x03\0\0\0\0\0\0\0\x18\
	181	+\0\0\0\x04\0\0\0\x40\0\0\0\0\0\0\0\0\0\0\x02\x08\0\0\0\0\0\0\0\0\0\0\x02\x0d\0\
	182	+\0\0\0\0\0\0\x01\0\0\x0d\x06\0\0\0\x1c\0\0\0\x01\0\0\0\x20\0\0\0\0\0\0\x01\x04\
	183	+\0\0\0\x20\0\0\x01\x24\0\0\0\x01\0\0\x0c\x05\0\0\0\xa3\0\0\0\x03\0\0\x04\x18\0\
	184	+\0\0\xb1\0\0\0\x09\0\0\0\0\0\0\0\xb5\0\0\0\x0b\0\0\0\x40\0\0\0\xc0\0\0\0\x0b\0\
	185	+\0\0\x80\0\0\0\0\0\0\0\0\0\0\x02\x0a\0\0\0\xc8\0\0\0\0\0\0\x07\0\0\0\0\xd1\0\0\
	186	+\0\0\0\0\x08\x0c\0\0\0\xd7\0\0\0\0\0\0\x01\x08\0\0\0\x40\0\0\0\x98\x01\0\0\x03\
	187	+\0\0\x04\x18\0\0\0\xa0\x01\0\0\x0e\0\0\0\0\0\0\0\xa3\x01\0\0\x11\0\0\0\x20\0\0\
	188	+\0\xa8\x01\0\0\x0e\0\0\0\xa0\0\0\0\xb4\x01\0\0\0\0\0\x08\x0f\0\0\0\xba\x01\0\0\
	189	+\0\0\0\x01\x04\0\0\0\x20\0\0\0\xc7\x01\0\0\0\0\0\x01\x01\0\0\0\x08\0\0\x01\0\0\
	190	+\0\0\0\0\0\x03\0\0\0\0\x10\0\0\0\x12\0\0\0\x10\0\0\0\xcc\x01\0\0\0\0\0\x01\x04\
	191	+\0\0\0\x20\0\0\0\0\0\0\0\0\0\0\x02\x14\0\0\0\x30\x02\0\0\x02\0\0\x04\x10\0\0\0\
	192	+\x13\0\0\0\x03\0\0\0\0\0\0\0\x43\x02\0\0\x15\0\0\0\x40\0\0\0\0\0\0\0\0\0\0\x02\
	193	+\x18\0\0\0\0\0\0\0\x01\0\0\x0d\x06\0\0\0\x1c\0\0\0\x13\0\0\0\x48\x02\0\0\x01\0\
	194	+\0\x0c\x16\0\0\0\x94\x02\0\0\x01\0\0\x04\x08\0\0\0\x9d\x02\0\0\x19\0\0\0\0\0\0\
	195	+\0\0\0\0\0\0\0\0\x02\x1a\0\0\0\xee\x02\0\0\x06\0\0\x04\x38\0\0\0\xa0\x01\0\0\
	196	+\x0e\0\0\0\0\0\0\0\xa3\x01\0\0\x11\0\0\0\x20\0\0\0\xfb\x02\0\0\x1b\0\0\0\xc0\0\
	197	+\0\0\x0c\x03\0\0\x15\0\0\0\0\x01\0\0\x18\x03\0\0\x1d\0\0\0\x40\x01\0\0\x22\x03\
	198	+\0\0\x1e\0\0\0\x80\x01\0\0\0\0\0\0\0\0\0\x02\x1c\0\0\0\0\0\0\0\0\0\0\x0a\x10\0\
	199	+\0\0\0\0\0\0\0\0\0\x02\x1f\0\0\0\0\0\0\0\0\0\0\x02\x20\0\0\0\x6c\x03\0\0\x02\0\
	200	+\0\x04\x08\0\0\0\x7a\x03\0\0\x0e\0\0\0\0\0\0\0\x83\x03\0\0\x0e\0\0\0\x20\0\0\0\
	201	+\x22\x03\0\0\x03\0\0\x04\x18\0\0\0\x8d\x03\0\0\x1b\0\0\0\0\0\0\0\x95\x03\0\0\
	202	+\x21\0\0\0\x40\0\0\0\x9b\x03\0\0\x23\0\0\0\x80\0\0\0\0\0\0\0\0\0\0\x02\x22\0\0\
	203	+\0\0\0\0\0\0\0\0\x02\x24\0\0\0\x9f\x03\0\0\x01\0\0\x04\x04\0\0\0\xaa\x03\0\0\
	204	+\x0e\0\0\0\0\0\0\0\x13\x04\0\0\x01\0\0\x04\x04\0\0\0\x1c\x04\0\0\x0e\0\0\0\0\0\
	205	+\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\x1c\0\0\0\x12\0\0\0\x23\0\0\0\x92\x04\0\0\0\0\0\
	206	+\x0e\x25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\x1c\0\0\0\x12\0\0\0\x0e\0\0\0\
	207	+\xa6\x04\0\0\0\0\0\x0e\x27\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\x1c\0\0\0\
	208	+\x12\0\0\0\x20\0\0\0\xbc\x04\0\0\0\0\0\x0e\x29\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\
	209	+\0\0\0\0\x1c\0\0\0\x12\0\0\0\x11\0\0\0\xd1\x04\0\0\0\0\0\x0e\x2b\0\0\0\0\0\0\0\
	210	+\0\0\0\0\0\0\0\x03\0\0\0\0\x10\0\0\0\x12\0\0\0\x04\0\0\0\xe8\x04\0\0\0\0\0\x0e\
	211	+\x2d\0\0\0\x01\0\0\0\xf0\x04\0\0\x04\0\0\x0f\0\0\0\0\x26\0\0\0\0\0\0\0\x23\0\0\
	212	+\0\x28\0\0\0\x23\0\0\0\x0e\0\0\0\x2a\0\0\0\x31\0\0\0\x20\0\0\0\x2c\0\0\0\x51\0\
	213	+\0\0\x11\0\0\0\xf8\x04\0\0\x01\0\0\x0f\0\0\0\0\x2e\0\0\0\0\0\0\0\x04\0\0\0\0\
	214	+\x62\x70\x66\x5f\x69\x74\x65\x72\x5f\x5f\x62\x70\x66\x5f\x6d\x61\x70\0\x6d\x65\
	215	+\x74\x61\0\x6d\x61\x70\0\x63\x74\x78\0\x69\x6e\x74\0\x64\x75\x6d\x70\x5f\x62\
	216	+\x70\x66\x5f\x6d\x61\x70\0\x69\x74\x65\x72\x2f\x62\x70\x66\x5f\x6d\x61\x70\0\
	217	+\x30\x3a\x30\0\x2f\x77\x2f\x6e\x65\x74\x2d\x6e\x65\x78\x74\x2f\x6b\x65\x72\x6e\
	218	+\x65\x6c\x2f\x62\x70\x66\x2f\x70\x72\x65\x6c\x6f\x61\x64\x2f\x69\x74\x65\x72\
	219	+\x61\x74\x6f\x72\x73\x2f\x69\x74\x65\x72\x61\x74\x6f\x72\x73\x2e\x62\x70\x66\
	220	+\x2e\x63\0\x09\x73\x74\x72\x75\x63\x74\x20\x73\x65\x71\x5f\x66\x69\x6c\x65\x20\
	221	+\x2a\x73\x65\x71\x20\x3d\x20\x63\x74\x78\x2d\x3e\x6d\x65\x74\x61\x2d\x3e\x73\
	222	+\x65\x71\x3b\0\x62\x70\x66\x5f\x69\x74\x65\x72\x5f\x6d\x65\x74\x61\0\x73\x65\
	223	+\x71\0\x73\x65\x73\x73\x69\x6f\x6e\x5f\x69\x64\0\x73\x65\x71\x5f\x6e\x75\x6d\0\
	224	+\x73\x65\x71\x5f\x66\x69\x6c\x65\0\x5f\x5f\x75\x36\x34\0\x6c\x6f\x6e\x67\x20\
	225	+\x6c\x6f\x6e\x67\x20\x75\x6e\x73\x69\x67\x6e\x65\x64\x20\x69\x6e\x74\0\x30\x3a\
	226	+\x31\0\x09\x73\x74\x72\x75\x63\x74\x20\x62\x70\x66\x5f\x6d\x61\x70\x20\x2a\x6d\
	227	+\x61\x70\x20\x3d\x20\x63\x74\x78\x2d\x3e\x6d\x61\x70\x3b\0\x09\x69\x66\x20\x28\
	228	+\x21\x6d\x61\x70\x29\0\x30\x3a\x32\0\x09\x5f\x5f\x75\x36\x34\x20\x73\x65\x71\
	229	+\x5f\x6e\x75\x6d\x20\x3d\x20\x63\x74\x78\x2d\x3e\x6d\x65\x74\x61\x2d\x3e\x73\
	230	+\x65\x71\x5f\x6e\x75\x6d\x3b\0\x09\x69\x66\x20\x28\x73\x65\x71\x5f\x6e\x75\x6d\
	231	+\x20\x3d\x3d\x20\x30\x29\0\x09\x09\x42\x50\x46\x5f\x53\x45\x51\x5f\x50\x52\x49\
	232	+\x4e\x54\x46\x28\x73\x65\x71\x2c\x20\x22\x20\x20\x69\x64\x20\x6e\x61\x6d\x65\
	233	+\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x6d\x61\x78\x5f\x65\x6e\
	234	+\x74\x72\x69\x65\x73\x5c\x6e\x22\x29\x3b\0\x62\x70\x66\x5f\x6d\x61\x70\0\x69\
	235	+\x64\0\x6e\x61\x6d\x65\0\x6d\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\0\x5f\x5f\
	236	+\x75\x33\x32\0\x75\x6e\x73\x69\x67\x6e\x65\x64\x20\x69\x6e\x74\0\x63\x68\x61\
	237	+\x72\0\x5f\x5f\x41\x52\x52\x41\x59\x5f\x53\x49\x5a\x45\x5f\x54\x59\x50\x45\x5f\
	238	+\x5f\0\x09\x42\x50\x46\x5f\x53\x45\x51\x5f\x50\x52\x49\x4e\x54\x46\x28\x73\x65\
	239	+\x71\x2c\x20\x22\x25\x34\x75\x20\x25\x2d\x31\x36\x73\x25\x36\x64\x5c\x6e\x22\
	240	+\x2c\x20\x6d\x61\x70\x2d\x3e\x69\x64\x2c\x20\x6d\x61\x70\x2d\x3e\x6e\x61\x6d\
	241	+\x65\x2c\x20\x6d\x61\x70\x2d\x3e\x6d\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\
	242	+\x29\x3b\0\x7d\0\x62\x70\x66\x5f\x69\x74\x65\x72\x5f\x5f\x62\x70\x66\x5f\x70\
	243	+\x72\x6f\x67\0\x70\x72\x6f\x67\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\
	244	+\x6f\x67\0\x69\x74\x65\x72\x2f\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x09\x73\x74\
	245	+\x72\x75\x63\x74\x20\x62\x70\x66\x5f\x70\x72\x6f\x67\x20\x2a\x70\x72\x6f\x67\
	246	+\x20\x3d\x20\x63\x74\x78\x2d\x3e\x70\x72\x6f\x67\x3b\0\x09\x69\x66\x20\x28\x21\
	247	+\x70\x72\x6f\x67\x29\0\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x61\x75\x78\0\x09\x61\
	248	+\x75\x78\x20\x3d\x20\x70\x72\x6f\x67\x2d\x3e\x61\x75\x78\x3b\0\x09\x09\x42\x50\
	249	+\x46\x5f\x53\x45\x51\x5f\x50\x52\x49\x4e\x54\x46\x28\x73\x65\x71\x2c\x20\x22\
	250	+\x20\x20\x69\x64\x20\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\
	251	+\x20\x20\x20\x61\x74\x74\x61\x63\x68\x65\x64\x5c\x6e\x22\x29\x3b\0\x62\x70\x66\
	252	+\x5f\x70\x72\x6f\x67\x5f\x61\x75\x78\0\x61\x74\x74\x61\x63\x68\x5f\x66\x75\x6e\
	253	+\x63\x5f\x6e\x61\x6d\x65\0\x6c\x69\x6e\x6b\x65\x64\x5f\x70\x72\x6f\x67\0\x66\
	254	+\x75\x6e\x63\x5f\x69\x6e\x66\x6f\0\x62\x74\x66\0\x09\x42\x50\x46\x5f\x53\x45\
	255	+\x51\x5f\x50\x52\x49\x4e\x54\x46\x28\x73\x65\x71\x2c\x20\x22\x25\x34\x75\x20\
	256	+\x25\x2d\x31\x36\x73\x20\x25\x73\x20\x25\x73\x5c\x6e\x22\x2c\x20\x61\x75\x78\
	257	+\x2d\x3e\x69\x64\x2c\0\x30\x3a\x34\0\x30\x3a\x35\0\x09\x69\x66\x20\x28\x21\x62\
	258	+\x74\x66\x29\0\x62\x70\x66\x5f\x66\x75\x6e\x63\x5f\x69\x6e\x66\x6f\0\x69\x6e\
	259	+\x73\x6e\x5f\x6f\x66\x66\0\x74\x79\x70\x65\x5f\x69\x64\0\x30\0\x73\x74\x72\x69\
	260	+\x6e\x67\x73\0\x74\x79\x70\x65\x73\0\x68\x64\x72\0\x62\x74\x66\x5f\x68\x65\x61\
	261	+\x64\x65\x72\0\x73\x74\x72\x5f\x6c\x65\x6e\0\x09\x74\x79\x70\x65\x73\x20\x3d\
	262	+\x20\x62\x74\x66\x2d\x3e\x74\x79\x70\x65\x73\x3b\0\x09\x62\x70\x66\x5f\x70\x72\
	263	+\x6f\x62\x65\x5f\x72\x65\x61\x64\x5f\x6b\x65\x72\x6e\x65\x6c\x28\x26\x74\x2c\
	264	+\x20\x73\x69\x7a\x65\x6f\x66\x28\x74\x29\x2c\x20\x74\x79\x70\x65\x73\x20\x2b\
	265	+\x20\x62\x74\x66\x5f\x69\x64\x29\x3b\0\x09\x73\x74\x72\x20\x3d\x20\x62\x74\x66\
	266	+\x2d\x3e\x73\x74\x72\x69\x6e\x67\x73\x3b\0\x62\x74\x66\x5f\x74\x79\x70\x65\0\
	267	+\x6e\x61\x6d\x65\x5f\x6f\x66\x66\0\x09\x6e\x61\x6d\x65\x5f\x6f\x66\x66\x20\x3d\
	268	+\x20\x42\x50\x46\x5f\x43\x4f\x52\x45\x5f\x52\x45\x41\x44\x28\x74\x2c\x20\x6e\
	269	+\x61\x6d\x65\x5f\x6f\x66\x66\x29\x3b\0\x30\x3a\x32\x3a\x30\0\x09\x69\x66\x20\
	270	+\x28\x6e\x61\x6d\x65\x5f\x6f\x66\x66\x20\x3e\x3d\x20\x62\x74\x66\x2d\x3e\x68\
	271	+\x64\x72\x2e\x73\x74\x72\x5f\x6c\x65\x6e\x29\0\x09\x72\x65\x74\x75\x72\x6e\x20\
	272	+\x73\x74\x72\x20\x2b\x20\x6e\x61\x6d\x65\x5f\x6f\x66\x66\x3b\0\x30\x3a\x33\0\
	273	+\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\0\
	274	+\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\
	275	+\x2e\x31\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\x2e\x5f\x5f\x5f\
	276	+\x66\x6d\x74\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\x2e\x5f\x5f\
	277	+\x5f\x66\x6d\x74\x2e\x32\0\x4c\x49\x43\x45\x4e\x53\x45\0\x2e\x72\x6f\x64\x61\
	278	+\x74\x61\0\x6c\x69\x63\x65\x6e\x73\x65\0\x9f\xeb\x01\0\x20\0\0\0\0\0\0\0\x24\0\
	279	+\0\0\x24\0\0\0\x44\x02\0\0\x68\x02\0\0\xa4\x01\0\0\x08\0\0\0\x31\0\0\0\x01\0\0\
	280	+\0\0\0\0\0\x07\0\0\0\x56\x02\0\0\x01\0\0\0\0\0\0\0\x17\0\0\0\x10\0\0\0\x31\0\0\
	281	+\0\x09\0\0\0\0\0\0\0\x42\0\0\0\x7b\0\0\0\x1e\x40\x01\0\x08\0\0\0\x42\0\0\0\x7b\
	282	+\0\0\0\x24\x40\x01\0\x10\0\0\0\x42\0\0\0\xf2\0\0\0\x1d\x48\x01\0\x18\0\0\0\x42\
	283	+\0\0\0\x13\x01\0\0\x06\x50\x01\0\x20\0\0\0\x42\0\0\0\x22\x01\0\0\x1d\x44\x01\0\
	284	+\x28\0\0\0\x42\0\0\0\x47\x01\0\0\x06\x5c\x01\0\x38\0\0\0\x42\0\0\0\x5a\x01\0\0\
	285	+\x03\x60\x01\0\x70\0\0\0\x42\0\0\0\xe0\x01\0\0\x02\x68\x01\0\xf0\0\0\0\x42\0\0\
	286	+\0\x2e\x02\0\0\x01\x70\x01\0\x56\x02\0\0\x1a\0\0\0\0\0\0\0\x42\0\0\0\x7b\0\0\0\
	287	+\x1e\x84\x01\0\x08\0\0\0\x42\0\0\0\x7b\0\0\0\x24\x84\x01\0\x10\0\0\0\x42\0\0\0\
	288	+\x64\x02\0\0\x1f\x8c\x01\0\x18\0\0\0\x42\0\0\0\x88\x02\0\0\x06\x98\x01\0\x20\0\
	289	+\0\0\x42\0\0\0\xa1\x02\0\0\x0e\xa4\x01\0\x28\0\0\0\x42\0\0\0\x22\x01\0\0\x1d\
	290	+\x88\x01\0\x30\0\0\0\x42\0\0\0\x47\x01\0\0\x06\xa8\x01\0\x40\0\0\0\x42\0\0\0\
	291	+\xb3\x02\0\0\x03\xac\x01\0\x80\0\0\0\x42\0\0\0\x26\x03\0\0\x02\xb4\x01\0\xb8\0\
	292	+\0\0\x42\0\0\0\x61\x03\0\0\x06\x08\x01\0\xd0\0\0\0\x42\0\0\0\0\0\0\0\0\0\0\0\
	293	+\xd8\0\0\0\x42\0\0\0\xb2\x03\0\0\x0f\x14\x01\0\xe0\0\0\0\x42\0\0\0\xc7\x03\0\0\
	294	+\x2d\x18\x01\0\xf0\0\0\0\x42\0\0\0\xfe\x03\0\0\x0d\x10\x01\0\0\x01\0\0\x42\0\0\
	295	+\0\0\0\0\0\0\0\0\0\x08\x01\0\0\x42\0\0\0\xc7\x03\0\0\x02\x18\x01\0\x20\x01\0\0\
	296	+\x42\0\0\0\x25\x04\0\0\x0d\x1c\x01\0\x38\x01\0\0\x42\0\0\0\0\0\0\0\0\0\0\0\x40\
	297	+\x01\0\0\x42\0\0\0\x25\x04\0\0\x0d\x1c\x01\0\x58\x01\0\0\x42\0\0\0\x25\x04\0\0\
	298	+\x0d\x1c\x01\0\x60\x01\0\0\x42\0\0\0\x53\x04\0\0\x1b\x20\x01\0\x68\x01\0\0\x42\
	299	+\0\0\0\x53\x04\0\0\x06\x20\x01\0\x70\x01\0\0\x42\0\0\0\x76\x04\0\0\x0d\x28\x01\
	300	+\0\x78\x01\0\0\x42\0\0\0\0\0\0\0\0\0\0\0\x80\x01\0\0\x42\0\0\0\x26\x03\0\0\x02\
	301	+\xb4\x01\0\xf8\x01\0\0\x42\0\0\0\x2e\x02\0\0\x01\xc4\x01\0\x10\0\0\0\x31\0\0\0\
	302	+\x07\0\0\0\0\0\0\0\x02\0\0\0\x3e\0\0\0\0\0\0\0\x08\0\0\0\x08\0\0\0\x3e\0\0\0\0\
	303	+\0\0\0\x10\0\0\0\x02\0\0\0\xee\0\0\0\0\0\0\0\x20\0\0\0\x08\0\0\0\x1e\x01\0\0\0\
	304	+\0\0\0\x70\0\0\0\x0d\0\0\0\x3e\0\0\0\0\0\0\0\x80\0\0\0\x0d\0\0\0\xee\0\0\0\0\0\
	305	+\0\0\xa0\0\0\0\x0d\0\0\0\x1e\x01\0\0\0\0\0\0\x56\x02\0\0\x12\0\0\0\0\0\0\0\x14\
	306	+\0\0\0\x3e\0\0\0\0\0\0\0\x08\0\0\0\x08\0\0\0\x3e\0\0\0\0\0\0\0\x10\0\0\0\x14\0\
	307	+\0\0\xee\0\0\0\0\0\0\0\x20\0\0\0\x18\0\0\0\x3e\0\0\0\0\0\0\0\x28\0\0\0\x08\0\0\
	308	+\0\x1e\x01\0\0\0\0\0\0\x80\0\0\0\x1a\0\0\0\x3e\0\0\0\0\0\0\0\x90\0\0\0\x1a\0\0\
	309	+\0\xee\0\0\0\0\0\0\0\xa8\0\0\0\x1a\0\0\0\x59\x03\0\0\0\0\0\0\xb0\0\0\0\x1a\0\0\
	310	+\0\x5d\x03\0\0\0\0\0\0\xc0\0\0\0\x1f\0\0\0\x8b\x03\0\0\0\0\0\0\xd8\0\0\0\x20\0\
	311	+\0\0\xee\0\0\0\0\0\0\0\xf0\0\0\0\x20\0\0\0\x3e\0\0\0\0\0\0\0\x18\x01\0\0\x24\0\
	312	+\0\0\x3e\0\0\0\0\0\0\0\x50\x01\0\0\x1a\0\0\0\xee\0\0\0\0\0\0\0\x60\x01\0\0\x20\
	313	+\0\0\0\x4d\x04\0\0\0\0\0\0\x88\x01\0\0\x1a\0\0\0\x1e\x01\0\0\0\0\0\0\x98\x01\0\
	314	+\0\x1a\0\0\0\x8e\x04\0\0\0\0\0\0\xa0\x01\0\0\x18\0\0\0\x3e\0\0\0\0\0\0\0\0\0\0\
	315	+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xd6\0\0\0\0\0\x02\0\x70\0\0\0\0\
	316	+\0\0\0\0\0\0\0\0\0\0\0\xc8\0\0\0\0\0\x02\0\xf0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
	317	+\xcf\0\0\0\0\0\x03\0\x78\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xc1\0\0\0\0\0\x03\0\x80\
	318	+\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xba\0\0\0\0\0\x03\0\xf8\x01\0\0\0\0\0\0\0\0\0\
	319	+\0\0\0\0\0\x14\0\0\0\x01\0\x04\0\0\0\0\0\0\0\0\0\x23\0\0\0\0\0\0\0\xf4\0\0\0\
	320	+\x01\0\x04\0\x23\0\0\0\0\0\0\0\x0e\0\0\0\0\0\0\0\x28\0\0\0\x01\0\x04\0\x31\0\0\
	321	+\0\0\0\0\0\x20\0\0\0\0\0\0\0\xdd\0\0\0\x01\0\x04\0\x51\0\0\0\0\0\0\0\x11\0\0\0\
	322	+\0\0\0\0\0\0\0\0\x03\0\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\x03\
	323	+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\x04\0\0\0\0\0\0\0\0\0\0\0\0\0\
	324	+\0\0\0\0\xb2\0\0\0\x11\0\x05\0\0\0\0\0\0\0\0\0\x04\0\0\0\0\0\0\0\x3d\0\0\0\x12\
	325	+\0\x02\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\x5b\0\0\0\x12\0\x03\0\0\0\0\0\0\0\0\
	326	+\0\x08\x02\0\0\0\0\0\0\x48\0\0\0\0\0\0\0\x01\0\0\0\x0c\0\0\0\xc8\0\0\0\0\0\0\0\
	327	+\x01\0\0\0\x0c\0\0\0\x50\0\0\0\0\0\0\0\x01\0\0\0\x0c\0\0\0\xd0\x01\0\0\0\0\0\0\
	328	+\x01\0\0\0\x0c\0\0\0\xf0\x03\0\0\0\0\0\0\x0a\0\0\0\x0c\0\0\0\xfc\x03\0\0\0\0\0\
	329	+\0\x0a\0\0\0\x0c\0\0\0\x08\x04\0\0\0\0\0\0\x0a\0\0\0\x0c\0\0\0\x14\x04\0\0\0\0\
	330	+\0\0\x0a\0\0\0\x0c\0\0\0\x2c\x04\0\0\0\0\0\0\0\0\0\0\x0d\0\0\0\x2c\0\0\0\0\0\0\
	331	+\0\0\0\0\0\x0a\0\0\0\x3c\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x50\0\0\0\0\0\0\0\0\0\
	332	+\0\0\x0a\0\0\0\x60\0\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\x70\0\0\0\0\0\0\0\0\0\0\0\
	333	+\x0a\0\0\0\x80\0\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\x90\0\0\0\0\0\0\0\0\0\0\0\x0a\0\
	334	+\0\0\xa0\0\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\xb0\0\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\
	335	+\xc0\0\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\xd0\0\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\xe8\0\
	336	+\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xf8\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x08\x01\0\0\
	337	+\0\0\0\0\0\0\0\0\x0b\0\0\0\x18\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x28\x01\0\0\0\
	338	+\0\0\0\0\0\0\0\x0b\0\0\0\x38\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x48\x01\0\0\0\0\
	339	+\0\0\0\0\0\0\x0b\0\0\0\x58\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x68\x01\0\0\0\0\0\
	340	+\0\0\0\0\0\x0b\0\0\0\x78\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x88\x01\0\0\0\0\0\0\
	341	+\0\0\0\0\x0b\0\0\0\x98\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xa8\x01\0\0\0\0\0\0\0\
	342	+\0\0\0\x0b\0\0\0\xb8\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xc8\x01\0\0\0\0\0\0\0\0\
	343	+\0\0\x0b\0\0\0\xd8\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xe8\x01\0\0\0\0\0\0\0\0\0\
	344	+\0\x0b\0\0\0\xf8\x01\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x08\x02\0\0\0\0\0\0\0\0\0\0\
	345	+\x0b\0\0\0\x18\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x28\x02\0\0\0\0\0\0\0\0\0\0\
	346	+\x0b\0\0\0\x38\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x48\x02\0\0\0\0\0\0\0\0\0\0\
	347	+\x0b\0\0\0\x58\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x68\x02\0\0\0\0\0\0\0\0\0\0\
	348	+\x0b\0\0\0\x78\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x94\x02\0\0\0\0\0\0\0\0\0\0\
	349	+\x0a\0\0\0\xa4\x02\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\xb4\x02\0\0\0\0\0\0\0\0\0\0\
	350	+\x0a\0\0\0\xc4\x02\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\xd4\x02\0\0\0\0\0\0\0\0\0\0\
	351	+\x0a\0\0\0\xe4\x02\0\0\0\0\0\0\0\0\0\0\x0a\0\0\0\xf4\x02\0\0\0\0\0\0\0\0\0\0\
	352	+\x0a\0\0\0\x0c\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x1c\x03\0\0\0\0\0\0\0\0\0\0\
	353	+\x0b\0\0\0\x2c\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x3c\x03\0\0\0\0\0\0\0\0\0\0\
	354	+\x0b\0\0\0\x4c\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x5c\x03\0\0\0\0\0\0\0\0\0\0\
	355	+\x0b\0\0\0\x6c\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x7c\x03\0\0\0\0\0\0\0\0\0\0\
	356	+\x0b\0\0\0\x8c\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x9c\x03\0\0\0\0\0\0\0\0\0\0\
	357	+\x0b\0\0\0\xac\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xbc\x03\0\0\0\0\0\0\0\0\0\0\
	358	+\x0b\0\0\0\xcc\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xdc\x03\0\0\0\0\0\0\0\0\0\0\
	359	+\x0b\0\0\0\xec\x03\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xfc\x03\0\0\0\0\0\0\0\0\0\0\
	360	+\x0b\0\0\0\x0c\x04\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x1c\x04\0\0\0\0\0\0\0\0\0\0\
	361	+\x0b\0\0\0\x4e\x4f\x41\x42\x43\x44\x4d\0\x2e\x74\x65\x78\x74\0\x2e\x72\x65\x6c\
	362	+\x2e\x42\x54\x46\x2e\x65\x78\x74\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\x61\
	363	+\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\
	364	+\x6f\x67\x2e\x5f\x5f\x5f\x66\x6d\x74\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\
	365	+\x61\x70\0\x2e\x72\x65\x6c\x69\x74\x65\x72\x2f\x62\x70\x66\x5f\x6d\x61\x70\0\
	366	+\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x2e\x72\x65\x6c\x69\x74\
	367	+\x65\x72\x2f\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x2e\x6c\x6c\x76\x6d\x5f\x61\x64\
	368	+\x64\x72\x73\x69\x67\0\x6c\x69\x63\x65\x6e\x73\x65\0\x2e\x73\x74\x72\x74\x61\
	369	+\x62\0\x2e\x73\x79\x6d\x74\x61\x62\0\x2e\x72\x6f\x64\x61\x74\x61\0\x2e\x72\x65\
	370	+\x6c\x2e\x42\x54\x46\0\x4c\x49\x43\x45\x4e\x53\x45\0\x4c\x42\x42\x31\x5f\x37\0\
	371	+\x4c\x42\x42\x31\x5f\x36\0\x4c\x42\x42\x30\x5f\x34\0\x4c\x42\x42\x31\x5f\x33\0\
	372	+\x4c\x42\x42\x30\x5f\x33\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\
	373	+\x2e\x5f\x5f\x5f\x66\x6d\x74\x2e\x32\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\
	374	+\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\x2e\x31\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
	375	+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
	376	+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\x01\0\0\0\x06\0\0\0\0\0\0\0\0\0\0\0\
	377	+\0\0\0\0\x40\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x04\0\0\0\0\0\0\0\0\
	378	+\0\0\0\0\0\0\0\x4e\0\0\0\x01\0\0\0\x06\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x40\0\0\0\
	379	+\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x08\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
	380	+\x6d\0\0\0\x01\0\0\0\x06\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x40\x01\0\0\0\0\0\0\x08\
	381	+\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x08\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xa1\0\0\0\
	382	+\x01\0\0\0\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x48\x03\0\0\0\0\0\0\x62\0\0\0\0\0\
	383	+\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x89\0\0\0\x01\0\0\0\x03\
	384	+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xaa\x03\0\0\0\0\0\0\x04\0\0\0\0\0\0\0\0\0\0\0\0\
	385	+\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xad\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\
	386	+\0\0\0\0\0\0\0\xae\x03\0\0\0\0\0\0\x34\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\
	387	+\0\0\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
	388	+\xe2\x0c\0\0\0\0\0\0\x2c\x04\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\
	389	+\0\0\0\0\0\0\x99\0\0\0\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x10\x11\0\0\0\
	390	+\0\0\0\x80\x01\0\0\0\0\0\0\x0e\0\0\0\x0d\0\0\0\x08\0\0\0\0\0\0\0\x18\0\0\0\0\0\
	391	+\0\0\x4a\0\0\0\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x90\x12\0\0\0\0\0\0\
	392	+\x20\0\0\0\0\0\0\0\x08\0\0\0\x02\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\0\0\0\0\x69\
	393	+\0\0\0\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xb0\x12\0\0\0\0\0\0\x20\0\0\0\
	394	+\0\0\0\0\x08\0\0\0\x03\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\0\0\0\0\xa9\0\0\0\x09\
	395	+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xd0\x12\0\0\0\0\0\0\x50\0\0\0\0\0\0\0\
	396	+\x08\0\0\0\x06\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\0\0\0\0\x07\0\0\0\x09\0\0\0\0\
	397	+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x20\x13\0\0\0\0\0\0\xe0\x03\0\0\0\0\0\0\x08\0\0\
	398	+\0\x07\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\0\0\0\0\x7b\0\0\0\x03\x4c\xff\x6f\0\0\
	399	+\0\x80\0\0\0\0\0\0\0\0\0\0\0\0\0\x17\0\0\0\0\0\0\x07\0\0\0\0\0\0\0\0\0\0\0\0\0\
	400	+\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x91\0\0\0\x03\0\0\0\0\0\0\0\0\0\0\0\0\0\
	401	+\0\0\0\0\0\0\x07\x17\0\0\0\0\0\0\x0a\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\
	402	+\0\0\0\0\0\0\0\0\0\0\0\0";
	403	+
	404	+ return 0;
	405	+err:
	406	+ bpf_object__destroy_skeleton(s);
	407	+ return -1;
	408	+}
	409	+
	410	+#endif /* __ITERATORS_BPF_SKEL_H__ */

kernel/bpf/queue_stack_maps.c+2−0

@@ -257,6 +257,7 @@ static int queue_stack_map_get_next_key(struct bpf_map map, void key, · 256 unmodified lines
257	257
258	258	static int queue_map_btf_id;
259	259	const struct bpf_map_ops queue_map_ops = {
	260	+ .map_meta_equal = bpf_map_meta_equal,
260	261	.map_alloc_check = queue_stack_map_alloc_check,
261	262	.map_alloc = queue_stack_map_alloc,
262	263	.map_free = queue_stack_map_free,
@@ -273,6 +274,7 @@ const struct bpf_map_ops queue_map_ops = { · 10 unmodified lines
273	274
274	275	static int stack_map_btf_id;
275	276	const struct bpf_map_ops stack_map_ops = {
	277	+ .map_meta_equal = bpf_map_meta_equal,
276	278	.map_alloc_check = queue_stack_map_alloc_check,
277	279	.map_alloc = queue_stack_map_alloc,
278	280	.map_free = queue_stack_map_free,

kernel/bpf/reuseport_array.c+1−0

@@ -351,6 +351,7 @@ static int reuseport_array_get_next_key(struct bpf_map map, void key, · 350 unmodified lines
351	351
352	352	static int reuseport_array_map_btf_id;
353	353	const struct bpf_map_ops reuseport_array_ops = {
	354	+ .map_meta_equal = bpf_map_meta_equal,
354	355	.map_alloc_check = reuseport_array_alloc_check,
355	356	.map_alloc = reuseport_array_alloc,
356	357	.map_free = reuseport_array_free,

kernel/bpf/ringbuf.c+1−0

@@ -287,6 +287,7 @@ static __poll_t ringbuf_map_poll(struct bpf_map map, struct file filp, · 286 unmodified lines
287	287
288	288	static int ringbuf_map_btf_id;
289	289	const struct bpf_map_ops ringbuf_map_ops = {
	290	+ .map_meta_equal = bpf_map_meta_equal,
290	291	.map_alloc = ringbuf_map_alloc,
291	292	.map_free = ringbuf_map_free,
292	293	.map_mmap = ringbuf_map_mmap,

kernel/bpf/stackmap.c+1−0

@@ -839,6 +839,7 @@ static void stack_map_free(struct bpf_map *map) · 838 unmodified lines
839	839
840	840	static int stack_trace_map_btf_id;
841	841	const struct bpf_map_ops stack_trace_map_ops = {
	842	+ .map_meta_equal = bpf_map_meta_equal,
842	843	.map_alloc = stack_map_alloc,
843	844	.map_free = stack_map_free,
844	845	.map_get_next_key = stack_map_get_next_key,

kernel/bpf/syscall.c+44−24

@@ -29,6 +29,7 @@ · 28 unmodified lines
29	29	#include <linux/bpf_lsm.h>
30	30	#include <linux/poll.h>
31	31	#include <linux/bpf-netns.h>
	32	+#include <linux/rcupdate_trace.h>
32	33
33	34	#define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY \|\| \
34	35	(map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY \|\| \
@@ -90,6 +91,7 @@ int bpf_check_uarg_tail_zero(void __user *uaddr, · 55 unmodified lines
90	91	}
91	92
92	93	const struct bpf_map_ops bpf_map_offload_ops = {
	94	+ .map_meta_equal = bpf_map_meta_equal,
93	95	.map_alloc = bpf_map_offload_map_alloc,
94	96	.map_free = bpf_map_offload_map_free,
95	97	.map_check_btf = map_check_no_btf,
@@ -157,10 +159,11 @@ static int bpf_map_update_value(struct bpf_map map, struct fd f, void key, · 61 unmodified lines
157	159	if (bpf_map_is_dev_bound(map)) {
158	160	return bpf_map_offload_update_elem(map, key, value, flags);
159	161	} else if (map->map_type == BPF_MAP_TYPE_CPUMAP \|\|
160		− map->map_type == BPF_MAP_TYPE_SOCKHASH \|\|
161		− map->map_type == BPF_MAP_TYPE_SOCKMAP \|\|
162	162	map->map_type == BPF_MAP_TYPE_STRUCT_OPS) {
163	163	return map->ops->map_update_elem(map, key, value, flags);
	164	+ } else if (map->map_type == BPF_MAP_TYPE_SOCKHASH \|\|
	165	+ map->map_type == BPF_MAP_TYPE_SOCKMAP) {
	166	+ return sock_map_update_elem_sys(map, key, value, flags);
164	167	} else if (IS_FD_PROG_ARRAY(map)) {
165	168	return bpf_fd_array_map_update_elem(map, f.file, key, value,
166	169	flags);
@@ -768,7 +771,8 @@ static int map_check_btf(struct bpf_map map, const struct btf btf, · 601 unmodified lines
768	771	if (map->map_type != BPF_MAP_TYPE_HASH &&
769	772	map->map_type != BPF_MAP_TYPE_ARRAY &&
770	773	map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE &&
771		− map->map_type != BPF_MAP_TYPE_SK_STORAGE)
	774	+ map->map_type != BPF_MAP_TYPE_SK_STORAGE &&
	775	+ map->map_type != BPF_MAP_TYPE_INODE_STORAGE)
772	776	return -ENOTSUPP;
773	777	if (map->spin_lock_off + sizeof(struct bpf_spin_lock) >
774	778	map->value_size) {
@@ -1728,10 +1732,14 @@ static void __bpf_prog_put_noref(struct bpf_prog *prog, bool deferred) · 953 unmodified lines
1728	1732	btf_put(prog->aux->btf);
1729	1733	bpf_prog_free_linfo(prog);
1730	1734
1731		− if (deferred)
1732		− call_rcu(&prog->aux->rcu, __bpf_prog_put_rcu);
1733		− else
	1735	+ if (deferred) {
	1736	+ if (prog->aux->sleepable)
	1737	+ call_rcu_tasks_trace(&prog->aux->rcu, __bpf_prog_put_rcu);
	1738	+ else
	1739	+ call_rcu(&prog->aux->rcu, __bpf_prog_put_rcu);
	1740	+ } else {
1734	1741	__bpf_prog_put_rcu(&prog->aux->rcu);
	1742	+ }
1735	1743	}
1736	1744
1737	1745	static void __bpf_prog_put(struct bpf_prog *prog, bool do_idr_lock)
@@ -2101,6 +2109,7 @@ static int bpf_prog_load(union bpf_attr attr, union bpf_attr __user uattr) · 363 unmodified lines
2101	2109	if (attr->prog_flags & ~(BPF_F_STRICT_ALIGNMENT \|
2102	2110	BPF_F_ANY_ALIGNMENT \|
2103	2111	BPF_F_TEST_STATE_FREQ \|
	2112	+ BPF_F_SLEEPABLE \|
2104	2113	BPF_F_TEST_RND_HI32))
2105	2114	return -EINVAL;
2106	2115
@@ -2156,6 +2165,7 @@ static int bpf_prog_load(union bpf_attr attr, union bpf_attr __user uattr) · 49 unmodified lines
2156	2165	}
2157	2166
2158	2167	prog->aux->offload_requested = !!attr->prog_ifindex;
	2168	+ prog->aux->sleepable = attr->prog_flags & BPF_F_SLEEPABLE;
2159	2169
2160	2170	err = security_bpf_prog_alloc(prog->aux);
2161	2171	if (err)
@@ -4014,40 +4024,50 @@ static int link_detach(union bpf_attr *attr) · 1852 unmodified lines
4014	4024	return ret;
4015	4025	}
4016	4026
4017		−static int bpf_link_inc_not_zero(struct bpf_link *link)
	4027	+static struct bpf_link bpf_link_inc_not_zero(struct bpf_link link)
4018	4028	{
4019		− return atomic64_fetch_add_unless(&link->refcnt, 1, 0) ? 0 : -ENOENT;
	4029	+ return atomic64_fetch_add_unless(&link->refcnt, 1, 0) ? link : ERR_PTR(-ENOENT);
4020	4030	}
4021	4031
4022		−#define BPF_LINK_GET_FD_BY_ID_LAST_FIELD link_id
4023		−
4024		−static int bpf_link_get_fd_by_id(const union bpf_attr *attr)
	4032	+struct bpf_link *bpf_link_by_id(u32 id)
4025	4033	{
4026	4034	struct bpf_link *link;
4027		− u32 id = attr->link_id;
4028		− int fd, err;
4029	4035
4030		− if (CHECK_ATTR(BPF_LINK_GET_FD_BY_ID))
4031		− return -EINVAL;
4032		−
4033		− if (!capable(CAP_SYS_ADMIN))
4034		− return -EPERM;
	4036	+ if (!id)
	4037	+ return ERR_PTR(-ENOENT);
4035	4038
4036	4039	spin_lock_bh(&link_idr_lock);
4037		− link = idr_find(&link_idr, id);
4038	4040	/* before link is "settled", ID is 0, pretend it doesn't exist yet */
	4041	+ link = idr_find(&link_idr, id);
4039	4042	if (link) {
4040	4043	if (link->id)
4041		− err = bpf_link_inc_not_zero(link);
	4044	+ link = bpf_link_inc_not_zero(link);
4042	4045	else
4043		− err = -EAGAIN;
	4046	+ link = ERR_PTR(-EAGAIN);
4044	4047	} else {
4045		− err = -ENOENT;
	4048	+ link = ERR_PTR(-ENOENT);
4046	4049	}
4047	4050	spin_unlock_bh(&link_idr_lock);
	4051	+ return link;
	4052	+}
4048	4053
4049		− if (err)
4050		− return err;
	4054	+#define BPF_LINK_GET_FD_BY_ID_LAST_FIELD link_id
	4055	+
	4056	+static int bpf_link_get_fd_by_id(const union bpf_attr *attr)
	4057	+{
	4058	+ struct bpf_link *link;
	4059	+ u32 id = attr->link_id;
	4060	+ int fd;
	4061	+
	4062	+ if (CHECK_ATTR(BPF_LINK_GET_FD_BY_ID))
	4063	+ return -EINVAL;
	4064	+
	4065	+ if (!capable(CAP_SYS_ADMIN))
	4066	+ return -EPERM;
	4067	+
	4068	+ link = bpf_link_by_id(id);
	4069	+ if (IS_ERR(link))
	4070	+ return PTR_ERR(link);
4051	4071
4052	4072	fd = bpf_link_new_fd(link);
4053	4073	if (fd < 0)

kernel/bpf/trampoline.c+26−3

@@ -7,6 +7,8 @@ · 6 unmodified lines
7	7	#include <linux/rbtree_latch.h>
8	8	#include <linux/perf_event.h>
9	9	#include <linux/btf.h>
	10	+#include <linux/rcupdate_trace.h>
	11	+#include <linux/rcupdate_wait.h>
10	12
11	13	/* dummy _ops. The verifier will operate on target program's ops. */
12	14	const struct bpf_verifier_ops bpf_extension_verifier_ops = {
@@ -210,9 +212,12 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr) · 197 unmodified lines
210	212	* updates to trampoline would change the code from underneath the
211	213	* preempted task. Hence wait for tasks to voluntarily schedule or go
212	214	* to userspace.
	215	+ * The same trampoline can hold both sleepable and non-sleepable progs.
	216	+ * synchronize_rcu_tasks_trace() is needed to make sure all sleepable
	217	+ * programs finish executing.
	218	+ * Wait for these two grace periods together.
213	219	*/
214		−
215		− synchronize_rcu_tasks();
	220	+ synchronize_rcu_mult(call_rcu_tasks, call_rcu_tasks_trace);
216	221
217	222	err = arch_prepare_bpf_trampoline(new_image, new_image + PAGE_SIZE / 2,
218	223	&tr->func.model, flags, tprogs,
@@ -344,7 +349,14 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) · 125 unmodified lines
344	349	if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[BPF_TRAMP_FEXIT])))
345	350	goto out;
346	351	bpf_image_ksym_del(&tr->ksym);
347		− /* wait for tasks to get out of trampoline before freeing it */
	352	+ /* This code will be executed when all bpf progs (both sleepable and
	353	+ * non-sleepable) went through
	354	+ * bpf_prog_put()->call_rcu[_tasks_trace]()->bpf_prog_free_deferred().
	355	+ * Hence no need for another synchronize_rcu_tasks_trace() here,
	356	+ * but synchronize_rcu_tasks() is still needed, since trampoline
	357	+ * may not have had any sleepable programs and we need to wait
	358	+ * for tasks to get out of trampoline code before freeing it.
	359	+ */
348	360	synchronize_rcu_tasks();
349	361	bpf_jit_free_exec(tr->image);
350	362	hlist_del(&tr->hlist);
@@ -394,6 +406,17 @@ void notrace __bpf_prog_exit(struct bpf_prog *prog, u64 start) · 43 unmodified lines
394	406	rcu_read_unlock();
395	407	}
396	408
	409	+void notrace __bpf_prog_enter_sleepable(void)
	410	+{
	411	+ rcu_read_lock_trace();
	412	+ might_fault();
	413	+}
	414	+
	415	+void notrace __bpf_prog_exit_sleepable(void)
	416	+{
	417	+ rcu_read_unlock_trace();
	418	+}
	419	+
397	420	int __weak
398	421	arch_prepare_bpf_trampoline(void image, void image_end,
399	422	const struct btf_func_model *m, u32 flags,

kernel/bpf/verifier.c+263−20

@@ -21,6 +21,7 @@ · 20 unmodified lines
21	21	#include <linux/ctype.h>
22	22	#include <linux/error-injection.h>
23	23	#include <linux/bpf_lsm.h>
	24	+#include <linux/btf_ids.h>
24	25
25	26	#include "disasm.h"
26	27
@@ -2625,11 +2626,19 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 2598 unmodified lines
2625	2626
2626	2627	#define MAX_PACKET_OFF 0xffff
2627	2628
	2629	+static enum bpf_prog_type resolve_prog_type(struct bpf_prog *prog)
	2630	+{
	2631	+ return prog->aux->linked_prog ? prog->aux->linked_prog->type
	2632	+ : prog->type;
	2633	+}
	2634	+
2628	2635	static bool may_access_direct_pkt_data(struct bpf_verifier_env *env,
2629	2636	const struct bpf_call_arg_meta *meta,
2630	2637	enum bpf_access_type t)
2631	2638	{
2632		− switch (env->prog->type) {
	2639	+ enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
	2640	+
	2641	+ switch (prog_type) {
2633	2642	/* Program types only with direct read access go here! */
2634	2643	case BPF_PROG_TYPE_LWT_IN:
2635	2644	case BPF_PROG_TYPE_LWT_OUT:
@@ -3872,6 +3881,33 @@ static int int_ptr_type_to_size(enum bpf_arg_type type) · 1236 unmodified lines
3872	3881	return -EINVAL;
3873	3882	}
3874	3883
	3884	+static int resolve_map_arg_type(struct bpf_verifier_env *env,
	3885	+ const struct bpf_call_arg_meta *meta,
	3886	+ enum bpf_arg_type *arg_type)
	3887	+{
	3888	+ if (!meta->map_ptr) {
	3889	+ /* kernel subsystem misconfigured verifier */
	3890	+ verbose(env, "invalid map_ptr to access map->type\n");
	3891	+ return -EACCES;
	3892	+ }
	3893	+
	3894	+ switch (meta->map_ptr->map_type) {
	3895	+ case BPF_MAP_TYPE_SOCKMAP:
	3896	+ case BPF_MAP_TYPE_SOCKHASH:
	3897	+ if (*arg_type == ARG_PTR_TO_MAP_VALUE) {
	3898	+ *arg_type = ARG_PTR_TO_SOCKET;
	3899	+ } else {
	3900	+ verbose(env, "invalid arg_type for sockmap/sockhash\n");
	3901	+ return -EINVAL;
	3902	+ }
	3903	+ break;
	3904	+
	3905	+ default:
	3906	+ break;
	3907	+ }
	3908	+ return 0;
	3909	+}
	3910	+
3875	3911	static int check_func_arg(struct bpf_verifier_env *env, u32 arg,
3876	3912	struct bpf_call_arg_meta *meta,
3877	3913	const struct bpf_func_proto *fn)
@@ -3904,6 +3940,14 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 26 unmodified lines
3904	3940	return -EACCES;
3905	3941	}
3906	3942
	3943	+ if (arg_type == ARG_PTR_TO_MAP_VALUE \|\|
	3944	+ arg_type == ARG_PTR_TO_UNINIT_MAP_VALUE \|\|
	3945	+ arg_type == ARG_PTR_TO_MAP_VALUE_OR_NULL) {
	3946	+ err = resolve_map_arg_type(env, meta, &arg_type);
	3947	+ if (err)
	3948	+ return err;
	3949	+ }
	3950	+
3907	3951	if (arg_type == ARG_PTR_TO_MAP_KEY \|\|
3908	3952	arg_type == ARG_PTR_TO_MAP_VALUE \|\|
3909	3953	arg_type == ARG_PTR_TO_UNINIT_MAP_VALUE \|\|
@@ -3960,16 +4004,21 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 50 unmodified lines
3960	4004	goto err_type;
3961	4005	}
3962	4006	} else if (arg_type == ARG_PTR_TO_BTF_ID) {
	4007	+ bool ids_match = false;
	4008	+
3963	4009	expected_type = PTR_TO_BTF_ID;
3964	4010	if (type != expected_type)
3965	4011	goto err_type;
3966	4012	if (!fn->check_btf_id) {
3967	4013	if (reg->btf_id != meta->btf_id) {
3968		− verbose(env, "Helper has type %s got %s in R%d\n",
3969		− kernel_type_name(meta->btf_id),
3970		− kernel_type_name(reg->btf_id), regno);
3971		−
3972		− return -EACCES;
	4014	+ ids_match = btf_struct_ids_match(&env->log, reg->off, reg->btf_id,
	4015	+ meta->btf_id);
	4016	+ if (!ids_match) {
	4017	+ verbose(env, "Helper has type %s got %s in R%d\n",
	4018	+ kernel_type_name(meta->btf_id),
	4019	+ kernel_type_name(reg->btf_id), regno);
	4020	+ return -EACCES;
	4021	+ }
3973	4022	}
3974	4023	} else if (!fn->check_btf_id(reg->btf_id, arg)) {
3975	4024	verbose(env, "Helper does not support %s in R%d\n",
@@ -3977,7 +4026,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 1 unmodified line
3977	4026
3978	4027	return -EACCES;
3979	4028	}
3980		− if (!tnum_is_const(reg->var_off) \|\| reg->var_off.value \|\| reg->off) {
	4029	+ if ((reg->off && !ids_match) \|\| !tnum_is_const(reg->var_off) \|\| reg->var_off.value) {
3981	4030	verbose(env, "R%d is a pointer to in-kernel struct with non-zero offset\n",
3982	4031	regno);
3983	4032	return -EACCES;
@@ -4143,6 +4192,38 @@ err_type: · 159 unmodified lines
4143	4192	return -EACCES;
4144	4193	}
4145	4194
	4195	+static bool may_update_sockmap(struct bpf_verifier_env *env, int func_id)
	4196	+{
	4197	+ enum bpf_attach_type eatype = env->prog->expected_attach_type;
	4198	+ enum bpf_prog_type type = resolve_prog_type(env->prog);
	4199	+
	4200	+ if (func_id != BPF_FUNC_map_update_elem)
	4201	+ return false;
	4202	+
	4203	+ /* It's not possible to get access to a locked struct sock in these
	4204	+ * contexts, so updating is safe.
	4205	+ */
	4206	+ switch (type) {
	4207	+ case BPF_PROG_TYPE_TRACING:
	4208	+ if (eatype == BPF_TRACE_ITER)
	4209	+ return true;
	4210	+ break;
	4211	+ case BPF_PROG_TYPE_SOCKET_FILTER:
	4212	+ case BPF_PROG_TYPE_SCHED_CLS:
	4213	+ case BPF_PROG_TYPE_SCHED_ACT:
	4214	+ case BPF_PROG_TYPE_XDP:
	4215	+ case BPF_PROG_TYPE_SK_REUSEPORT:
	4216	+ case BPF_PROG_TYPE_FLOW_DISSECTOR:
	4217	+ case BPF_PROG_TYPE_SK_LOOKUP:
	4218	+ return true;
	4219	+ default:
	4220	+ break;
	4221	+ }
	4222	+
	4223	+ verbose(env, "cannot update sockmap in this context\n");
	4224	+ return false;
	4225	+}
	4226	+
4146	4227	static int check_map_func_compatibility(struct bpf_verifier_env *env,
4147	4228	struct bpf_map *map, int func_id)
4148	4229	{
@@ -4214,7 +4295,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 65 unmodified lines
4214	4295	func_id != BPF_FUNC_map_delete_elem &&
4215	4296	func_id != BPF_FUNC_msg_redirect_map &&
4216	4297	func_id != BPF_FUNC_sk_select_reuseport &&
4217		− func_id != BPF_FUNC_map_lookup_elem)
	4298	+ func_id != BPF_FUNC_map_lookup_elem &&
	4299	+ !may_update_sockmap(env, func_id))
4218	4300	goto error;
4219	4301	break;
4220	4302	case BPF_MAP_TYPE_SOCKHASH:
@@ -4223,7 +4305,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 2 unmodified lines
4223	4305	func_id != BPF_FUNC_map_delete_elem &&
4224	4306	func_id != BPF_FUNC_msg_redirect_hash &&
4225	4307	func_id != BPF_FUNC_sk_select_reuseport &&
4226		− func_id != BPF_FUNC_map_lookup_elem)
	4308	+ func_id != BPF_FUNC_map_lookup_elem &&
	4309	+ !may_update_sockmap(env, func_id))
4227	4310	goto error;
4228	4311	break;
4229	4312	case BPF_MAP_TYPE_REUSEPORT_SOCKARRAY:
@@ -4242,6 +4325,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 12 unmodified lines
4242	4325	func_id != BPF_FUNC_sk_storage_delete)
4243	4326	goto error;
4244	4327	break;
	4328	+ case BPF_MAP_TYPE_INODE_STORAGE:
	4329	+ if (func_id != BPF_FUNC_inode_storage_get &&
	4330	+ func_id != BPF_FUNC_inode_storage_delete)
	4331	+ goto error;
	4332	+ break;
4245	4333	default:
4246	4334	break;
4247	4335	}
@@ -4315,6 +4403,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 67 unmodified lines
4315	4403	if (map->map_type != BPF_MAP_TYPE_SK_STORAGE)
4316	4404	goto error;
4317	4405	break;
	4406	+ case BPF_FUNC_inode_storage_get:
	4407	+ case BPF_FUNC_inode_storage_delete:
	4408	+ if (map->map_type != BPF_MAP_TYPE_INODE_STORAGE)
	4409	+ goto error;
	4410	+ break;
4318	4411	default:
4319	4412	break;
4320	4413	}
@@ -4775,6 +4868,11 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 454 unmodified lines
4775	4868	return -EINVAL;
4776	4869	}
4777	4870
	4871	+ if (fn->allowed && !fn->allowed(env->prog)) {
	4872	+ verbose(env, "helper call is not allowed in probe\n");
	4873	+ return -EINVAL;
	4874	+ }
	4875	+
4778	4876	/* With LD_ABS/IND some JITs save/restore skb from r1. */
4779	4877	changes_data = bpf_helper_changes_pkt_data(fn->func);
4780	4878	if (changes_data && fn->arg1_type != ARG_PTR_TO_CTX) {
@@ -5732,6 +5830,67 @@ static void scalar_min_max_or(struct bpf_reg_state *dst_reg, · 951 unmodified lines
5732	5830	__update_reg_bounds(dst_reg);
5733	5831	}
5734	5832
	5833	+static void scalar32_min_max_xor(struct bpf_reg_state *dst_reg,
	5834	+ struct bpf_reg_state *src_reg)
	5835	+{
	5836	+ bool src_known = tnum_subreg_is_const(src_reg->var_off);
	5837	+ bool dst_known = tnum_subreg_is_const(dst_reg->var_off);
	5838	+ struct tnum var32_off = tnum_subreg(dst_reg->var_off);
	5839	+ s32 smin_val = src_reg->s32_min_value;
	5840	+
	5841	+ /* Assuming scalar64_min_max_xor will be called so it is safe
	5842	+ * to skip updating register for known case.
	5843	+ */
	5844	+ if (src_known && dst_known)
	5845	+ return;
	5846	+
	5847	+ /* We get both minimum and maximum from the var32_off. */
	5848	+ dst_reg->u32_min_value = var32_off.value;
	5849	+ dst_reg->u32_max_value = var32_off.value \| var32_off.mask;
	5850	+
	5851	+ if (dst_reg->s32_min_value >= 0 && smin_val >= 0) {
	5852	+ /* XORing two positive sign numbers gives a positive,
	5853	+ * so safe to cast u32 result into s32.
	5854	+ */
	5855	+ dst_reg->s32_min_value = dst_reg->u32_min_value;
	5856	+ dst_reg->s32_max_value = dst_reg->u32_max_value;
	5857	+ } else {
	5858	+ dst_reg->s32_min_value = S32_MIN;
	5859	+ dst_reg->s32_max_value = S32_MAX;
	5860	+ }
	5861	+}
	5862	+
	5863	+static void scalar_min_max_xor(struct bpf_reg_state *dst_reg,
	5864	+ struct bpf_reg_state *src_reg)
	5865	+{
	5866	+ bool src_known = tnum_is_const(src_reg->var_off);
	5867	+ bool dst_known = tnum_is_const(dst_reg->var_off);
	5868	+ s64 smin_val = src_reg->smin_value;
	5869	+
	5870	+ if (src_known && dst_known) {
	5871	+ /* dst_reg->var_off.value has been updated earlier */
	5872	+ __mark_reg_known(dst_reg, dst_reg->var_off.value);
	5873	+ return;
	5874	+ }
	5875	+
	5876	+ /* We get both minimum and maximum from the var_off. */
	5877	+ dst_reg->umin_value = dst_reg->var_off.value;
	5878	+ dst_reg->umax_value = dst_reg->var_off.value \| dst_reg->var_off.mask;
	5879	+
	5880	+ if (dst_reg->smin_value >= 0 && smin_val >= 0) {
	5881	+ /* XORing two positive sign numbers gives a positive,
	5882	+ * so safe to cast u64 result into s64.
	5883	+ */
	5884	+ dst_reg->smin_value = dst_reg->umin_value;
	5885	+ dst_reg->smax_value = dst_reg->umax_value;
	5886	+ } else {
	5887	+ dst_reg->smin_value = S64_MIN;
	5888	+ dst_reg->smax_value = S64_MAX;
	5889	+ }
	5890	+
	5891	+ __update_reg_bounds(dst_reg);
	5892	+}
	5893	+
5735	5894	static void __scalar32_min_max_lsh(struct bpf_reg_state *dst_reg,
5736	5895	u64 umin_val, u64 umax_val)
5737	5896	{
@@ -6040,6 +6199,11 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env, · 302 unmodified lines
6040	6199	scalar32_min_max_or(dst_reg, &src_reg);
6041	6200	scalar_min_max_or(dst_reg, &src_reg);
6042	6201	break;
	6202	+ case BPF_XOR:
	6203	+ dst_reg->var_off = tnum_xor(dst_reg->var_off, src_reg.var_off);
	6204	+ scalar32_min_max_xor(dst_reg, &src_reg);
	6205	+ scalar_min_max_xor(dst_reg, &src_reg);
	6206	+ break;
6043	6207	case BPF_LSH:
6044	6208	if (umax_val >= insn_bitness) {
6045	6209	/* Shifts greater than 31 or 63 are undefined.
@@ -7287,7 +7451,7 @@ static int check_ld_abs(struct bpf_verifier_env env, struct bpf_insn insn) · 1241 unmodified lines
7287	7451	u8 mode = BPF_MODE(insn->code);
7288	7452	int i, err;
7289	7453
7290		− if (!may_access_skb(env->prog->type)) {
	7454	+ if (!may_access_skb(resolve_prog_type(env->prog))) {
7291	7455	verbose(env, "BPF_LD_[ABS\|IND] instructions not allowed for this program type\n");
7292	7456	return -EINVAL;
7293	7457	}
@@ -7375,11 +7539,12 @@ static int check_return_code(struct bpf_verifier_env *env) · 81 unmodified lines
7375	7539	const struct bpf_prog *prog = env->prog;
7376	7540	struct bpf_reg_state *reg;
7377	7541	struct tnum range = tnum_range(0, 1);
	7542	+ enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
7378	7543	int err;
7379	7544
7380	7545	/* LSM and struct_ops func-ptr's return type could be "void" */
7381		− if ((env->prog->type == BPF_PROG_TYPE_STRUCT_OPS \|\|
7382		− env->prog->type == BPF_PROG_TYPE_LSM) &&
	7546	+ if ((prog_type == BPF_PROG_TYPE_STRUCT_OPS \|\|
	7547	+ prog_type == BPF_PROG_TYPE_LSM) &&
7383	7548	!prog->aux->attach_func_proto->type)
7384	7549	return 0;
7385	7550
@@ -7398,7 +7563,7 @@ static int check_return_code(struct bpf_verifier_env *env) · 12 unmodified lines
7398	7563	return -EACCES;
7399	7564	}
7400	7565
7401		− switch (env->prog->type) {
	7566	+ switch (prog_type) {
7402	7567	case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
7403	7568	if (env->prog->expected_attach_type == BPF_CGROUP_UDP4_RECVMSG \|\|
7404	7569	env->prog->expected_attach_type == BPF_CGROUP_UDP6_RECVMSG \|\|
@@ -9154,6 +9319,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 1749 unmodified lines
9154	9319	struct bpf_prog *prog)
9155	9320
9156	9321	{
	9322	+ enum bpf_prog_type prog_type = resolve_prog_type(prog);
9157	9323	/*
9158	9324	* Validate that trace type programs use preallocated hash maps.
9159	9325	*
@@ -9171,8 +9337,8 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 11 unmodified lines
9171	9337	* now, but warnings are emitted so developers are made aware of
9172	9338	* the unsafety and can fix their programs before this is enforced.
9173	9339	*/
9174		− if (is_tracing_prog_type(prog->type) && !is_preallocated_map(map)) {
9175		− if (prog->type == BPF_PROG_TYPE_PERF_EVENT) {
	9340	+ if (is_tracing_prog_type(prog_type) && !is_preallocated_map(map)) {
	9341	+ if (prog_type == BPF_PROG_TYPE_PERF_EVENT) {
9176	9342	verbose(env, "perf_event programs can only use preallocated hash map\n");
9177	9343	return -EINVAL;
9178	9344	}
@@ -9184,8 +9350,8 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 5 unmodified lines
9184	9350	verbose(env, "trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.\n");
9185	9351	}
9186	9352
9187		− if ((is_tracing_prog_type(prog->type) \|\|
9188		− prog->type == BPF_PROG_TYPE_SOCKET_FILTER) &&
	9353	+ if ((is_tracing_prog_type(prog_type) \|\|
	9354	+ prog_type == BPF_PROG_TYPE_SOCKET_FILTER) &&
9189	9355	map_value_has_spin_lock(map)) {
9190	9356	verbose(env, "tracing progs cannot use bpf_spin_lock yet\n");
9191	9357	return -EINVAL;
@@ -9202,6 +9368,23 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 10 unmodified lines
9202	9368	return -EINVAL;
9203	9369	}
9204	9370
	9371	+ if (prog->aux->sleepable)
	9372	+ switch (map->map_type) {
	9373	+ case BPF_MAP_TYPE_HASH:
	9374	+ case BPF_MAP_TYPE_LRU_HASH:
	9375	+ case BPF_MAP_TYPE_ARRAY:
	9376	+ if (!is_preallocated_map(map)) {
	9377	+ verbose(env,
	9378	+ "Sleepable programs can only use preallocated hash maps\n");
	9379	+ return -EINVAL;
	9380	+ }
	9381	+ break;
	9382	+ default:
	9383	+ verbose(env,
	9384	+ "Sleepable programs can only use array and hash maps\n");
	9385	+ return -EINVAL;
	9386	+ }
	9387	+
9205	9388	return 0;
9206	9389	}
9207	9390
@@ -9897,7 +10080,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) · 689 unmodified lines
9897	10080	insn->code = BPF_LDX \| BPF_PROBE_MEM \|
9898	10081	BPF_SIZE((insn)->code);
9899	10082	env->prog->aux->num_exentries++;
9900		− } else if (env->prog->type != BPF_PROG_TYPE_STRUCT_OPS) {
	10083	+ } else if (resolve_prog_type(env->prog) != BPF_PROG_TYPE_STRUCT_OPS) {
9901	10084	verbose(env, "Writes through BTF pointers are not allowed\n");
9902	10085	return -EINVAL;
9903	10086	}
@@ -10820,6 +11003,37 @@ static int check_attach_modify_return(struct bpf_prog *prog, unsigned long addr) · 916 unmodified lines
10820	11003	return -EINVAL;
10821	11004	}
10822	11005
	11006	+/* non exhaustive list of sleepable bpf_lsm_() functions /
	11007	+BTF_SET_START(btf_sleepable_lsm_hooks)
	11008	+#ifdef CONFIG_BPF_LSM
	11009	+BTF_ID(func, bpf_lsm_bprm_committed_creds)
	11010	+#else
	11011	+BTF_ID_UNUSED
	11012	+#endif
	11013	+BTF_SET_END(btf_sleepable_lsm_hooks)
	11014	+
	11015	+static int check_sleepable_lsm_hook(u32 btf_id)
	11016	+{
	11017	+ return btf_id_set_contains(&btf_sleepable_lsm_hooks, btf_id);
	11018	+}
	11019	+
	11020	+/* list of non-sleepable functions that are otherwise on
	11021	+ * ALLOW_ERROR_INJECTION list
	11022	+ */
	11023	+BTF_SET_START(btf_non_sleepable_error_inject)
	11024	+/* Three functions below can be called from sleepable and non-sleepable context.
	11025	+ * Assume non-sleepable from bpf safety point of view.
	11026	+ */
	11027	+BTF_ID(func, __add_to_page_cache_locked)
	11028	+BTF_ID(func, should_fail_alloc_page)
	11029	+BTF_ID(func, should_failslab)
	11030	+BTF_SET_END(btf_non_sleepable_error_inject)
	11031	+
	11032	+static int check_non_sleepable_error_inject(u32 btf_id)
	11033	+{
	11034	+ return btf_id_set_contains(&btf_non_sleepable_error_inject, btf_id);
	11035	+}
	11036	+
10823	11037	static int check_attach_btf_id(struct bpf_verifier_env *env)
10824	11038	{
10825	11039	struct bpf_prog *prog = env->prog;
@@ -10837,6 +11051,12 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 11 unmodified lines
10837	11051	long addr;
10838	11052	u64 key;
10839	11053
	11054	+ if (prog->aux->sleepable && prog->type != BPF_PROG_TYPE_TRACING &&
	11055	+ prog->type != BPF_PROG_TYPE_LSM) {
	11056	+ verbose(env, "Only fentry/fexit/fmod_ret and lsm programs can be sleepable\n");
	11057	+ return -EINVAL;
	11058	+ }
	11059	+
10840	11060	if (prog->type == BPF_PROG_TYPE_STRUCT_OPS)
10841	11061	return check_struct_ops_btf_id(env);
10842	11062
@@ -11045,13 +11265,36 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 202 unmodified lines
11045	11265	}
11046	11266	}
11047	11267
11048		− if (prog->expected_attach_type == BPF_MODIFY_RETURN) {
	11268	+ if (prog->aux->sleepable) {
	11269	+ ret = -EINVAL;
	11270	+ switch (prog->type) {
	11271	+ case BPF_PROG_TYPE_TRACING:
	11272	+ /* fentry/fexit/fmod_ret progs can be sleepable only if they are
	11273	+ * attached to ALLOW_ERROR_INJECTION and are not in denylist.
	11274	+ */
	11275	+ if (!check_non_sleepable_error_inject(btf_id) &&
	11276	+ within_error_injection_list(addr))
	11277	+ ret = 0;
	11278	+ break;
	11279	+ case BPF_PROG_TYPE_LSM:
	11280	+ /* LSM progs check that they are attached to bpf_lsm_*() funcs.
	11281	+ * Only some of them are sleepable.
	11282	+ */
	11283	+ if (check_sleepable_lsm_hook(btf_id))
	11284	+ ret = 0;
	11285	+ break;
	11286	+ default:
	11287	+ break;
	11288	+ }
	11289	+ if (ret)
	11290	+ verbose(env, "%s is not sleepable\n",
	11291	+ prog->aux->attach_func_name);
	11292	+ } else if (prog->expected_attach_type == BPF_MODIFY_RETURN) {
11049	11293	ret = check_attach_modify_return(prog, addr);
11050	11294	if (ret)
11051	11295	verbose(env, "%s() is not modifiable\n",
11052	11296	prog->aux->attach_func_name);
11053	11297	}
11054		−
11055	11298	if (ret)
11056	11299	goto out;
11057	11300	tr->func.addr = (void *)addr;

kernel/trace/bpf_trace.c+50−0

@@ -1098,6 +1098,52 @@ static const struct bpf_func_proto bpf_send_signal_thread_proto = { · 1097 unmodified lines
1098	1098	.arg1_type = ARG_ANYTHING,
1099	1099	};
1100	1100
	1101	+BPF_CALL_3(bpf_d_path, struct path , path, char , buf, u32, sz)
	1102	+{
	1103	+ long len;
	1104	+ char *p;
	1105	+
	1106	+ if (!sz)
	1107	+ return 0;
	1108	+
	1109	+ p = d_path(path, buf, sz);
	1110	+ if (IS_ERR(p)) {
	1111	+ len = PTR_ERR(p);
	1112	+ } else {
	1113	+ len = buf + sz - p;
	1114	+ memmove(buf, p, len);
	1115	+ }
	1116	+
	1117	+ return len;
	1118	+}
	1119	+
	1120	+BTF_SET_START(btf_allowlist_d_path)
	1121	+BTF_ID(func, vfs_truncate)
	1122	+BTF_ID(func, vfs_fallocate)
	1123	+BTF_ID(func, dentry_open)
	1124	+BTF_ID(func, vfs_getattr)
	1125	+BTF_ID(func, filp_close)
	1126	+BTF_SET_END(btf_allowlist_d_path)
	1127	+
	1128	+static bool bpf_d_path_allowed(const struct bpf_prog *prog)
	1129	+{
	1130	+ return btf_id_set_contains(&btf_allowlist_d_path, prog->aux->attach_btf_id);
	1131	+}
	1132	+
	1133	+BTF_ID_LIST(bpf_d_path_btf_ids)
	1134	+BTF_ID(struct, path)
	1135	+
	1136	+static const struct bpf_func_proto bpf_d_path_proto = {
	1137	+ .func = bpf_d_path,
	1138	+ .gpl_only = false,
	1139	+ .ret_type = RET_INTEGER,
	1140	+ .arg1_type = ARG_PTR_TO_BTF_ID,
	1141	+ .arg2_type = ARG_PTR_TO_MEM,
	1142	+ .arg3_type = ARG_CONST_SIZE_OR_ZERO,
	1143	+ .btf_id = bpf_d_path_btf_ids,
	1144	+ .allowed = bpf_d_path_allowed,
	1145	+};
	1146	+
1101	1147	const struct bpf_func_proto *
1102	1148	bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
1103	1149	{
@@ -1182,6 +1228,8 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 78 unmodified lines
1182	1228	return &bpf_jiffies64_proto;
1183	1229	case BPF_FUNC_get_task_stack:
1184	1230	return &bpf_get_task_stack_proto;
	1231	+ case BPF_FUNC_copy_from_user:
	1232	+ return prog->aux->sleepable ? &bpf_copy_from_user_proto : NULL;
1185	1233	default:
1186	1234	return NULL;
1187	1235	}
@@ -1579,6 +1627,8 @@ tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 391 unmodified lines
1579	1627	return prog->expected_attach_type == BPF_TRACE_ITER ?
1580	1628	&bpf_seq_write_proto :
1581	1629	NULL;
	1630	+ case BPF_FUNC_d_path:
	1631	+ return &bpf_d_path_proto;
1582	1632	default:
1583	1633	return raw_tp_prog_func_proto(func_id, prog);
1584	1634	}

mm/filemap.c+4−4

@@ -827,10 +827,10 @@ int replace_page_cache_page(struct page old, struct page new, gfp_t gfp_mask) · 826 unmodified lines
827	827	}
828	828	EXPORT_SYMBOL_GPL(replace_page_cache_page);
829	829
830		−static int __add_to_page_cache_locked(struct page *page,
831		− struct address_space *mapping,
832		− pgoff_t offset, gfp_t gfp_mask,
833		− void **shadowp)
	830	+noinline int __add_to_page_cache_locked(struct page *page,
	831	+ struct address_space *mapping,
	832	+ pgoff_t offset, gfp_t gfp_mask,
	833	+ void **shadowp)
834	834	{
835	835	XA_STATE(xas, &mapping->i_pages, offset);
836	836	int huge = PageHuge(page);

mm/page_alloc.c+1−1

@@ -3482,7 +3482,7 @@ static inline bool __should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) · 3481 unmodified lines
3482	3482
3483	3483	#endif /* CONFIG_FAIL_PAGE_ALLOC */
3484	3484
3485		−static noinline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
	3485	+noinline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
3486	3486	{
3487	3487	return __should_fail_alloc_page(gfp_mask, order);
3488	3488	}

net/bpfilter/Kconfig+1−0

@@ -2,6 +2,7 @@ · 1 unmodified line
2	2	menuconfig BPFILTER
3	3	bool "BPF based packet filtering framework (BPFILTER)"
4	4	depends on NET && BPF && INET
	5	+ select USERMODE_DRIVER
5	6	help
6	7	This builds experimental bpfilter framework that is aiming to
7	8	provide netfilter compatible functionality via BPF

net/core/bpf_sk_storage.c+138−695

@@ -7,97 +7,14 @@ · 6 unmodified lines
7	7	#include <linux/spinlock.h>
8	8	#include <linux/bpf.h>
9	9	#include <linux/btf_ids.h>
	10	+#include <linux/bpf_local_storage.h>
10	11	#include <net/bpf_sk_storage.h>
11	12	#include <net/sock.h>
12	13	#include <uapi/linux/sock_diag.h>
13	14	#include <uapi/linux/btf.h>
	15	+#include <linux/btf_ids.h>
14	16
15		−#define SK_STORAGE_CREATE_FLAG_MASK \
16		− (BPF_F_NO_PREALLOC \| BPF_F_CLONE)
17		−
18		−struct bucket {
19		− struct hlist_head list;
20		− raw_spinlock_t lock;
21		−};
22		−
23		−/* Thp map is not the primary owner of a bpf_sk_storage_elem.
24		− * Instead, the sk->sk_bpf_storage is.
25		− *
26		− * The map (bpf_sk_storage_map) is for two purposes
27		− * 1. Define the size of the "sk local storage". It is
28		− * the map's value_size.
29		− *
30		− * 2. Maintain a list to keep track of all elems such
31		− * that they can be cleaned up during the map destruction.
32		− *
33		− * When a bpf local storage is being looked up for a
34		− * particular sk, the "bpf_map" pointer is actually used
35		− * as the "key" to search in the list of elem in
36		− * sk->sk_bpf_storage.
37		− *
38		− * Hence, consider sk->sk_bpf_storage is the mini-map
39		− * with the "bpf_map" pointer as the searching key.
40		− */
41		−struct bpf_sk_storage_map {
42		− struct bpf_map map;
43		− /* Lookup elem does not require accessing the map.
44		− *
45		− * Updating/Deleting requires a bucket lock to
46		− * link/unlink the elem from the map. Having
47		− * multiple buckets to improve contention.
48		− */
49		− struct bucket *buckets;
50		− u32 bucket_log;
51		− u16 elem_size;
52		− u16 cache_idx;
53		−};
54		−
55		−struct bpf_sk_storage_data {
56		− /* smap is used as the searching key when looking up
57		− * from sk->sk_bpf_storage.
58		− *
59		− * Put it in the same cacheline as the data to minimize
60		− * the number of cachelines access during the cache hit case.
61		− */
62		− struct bpf_sk_storage_map __rcu *smap;
63		− u8 data[] __aligned(8);
64		−};
65		−
66		−/* Linked to bpf_sk_storage and bpf_sk_storage_map */
67		−struct bpf_sk_storage_elem {
68		− struct hlist_node map_node; /* Linked to bpf_sk_storage_map */
69		− struct hlist_node snode; /* Linked to bpf_sk_storage */
70		− struct bpf_sk_storage __rcu *sk_storage;
71		− struct rcu_head rcu;
72		− /* 8 bytes hole */
73		− /* The data is stored in aother cacheline to minimize
74		− * the number of cachelines access during a cache hit.
75		− */
76		− struct bpf_sk_storage_data sdata ____cacheline_aligned;
77		−};
78		−
79		−#define SELEM(_SDATA) container_of((_SDATA), struct bpf_sk_storage_elem, sdata)
80		−#define SDATA(_SELEM) (&(_SELEM)->sdata)
81		−#define BPF_SK_STORAGE_CACHE_SIZE 16
82		−
83		−static DEFINE_SPINLOCK(cache_idx_lock);
84		−static u64 cache_idx_usage_counts[BPF_SK_STORAGE_CACHE_SIZE];
85		−
86		−struct bpf_sk_storage {
87		− struct bpf_sk_storage_data __rcu *cache[BPF_SK_STORAGE_CACHE_SIZE];
88		− struct hlist_head list; /* List of bpf_sk_storage_elem */
89		− struct sock sk; / The sk that owns the the above "list" of
90		− * bpf_sk_storage_elem.
91		− */
92		− struct rcu_head rcu;
93		− raw_spinlock_t lock; /* Protect adding/removing from the "list" */
94		−};
95		−
96		−static struct bucket select_bucket(struct bpf_sk_storage_map smap,
97		− struct bpf_sk_storage_elem *selem)
98		−{
99		− return &smap->buckets[hash_ptr(selem, smap->bucket_log)];
100		−}
	17	+DEFINE_BPF_STORAGE_CACHE(sk_cache);
101	18
102	19	static int omem_charge(struct sock *sk, unsigned int size)
103	20	{
@@ -111,445 +28,38 @@ static int omem_charge(struct sock *sk, unsigned int size) · 7 unmodified lines
111	28	return -ENOMEM;
112	29	}
113	30
114		−static bool selem_linked_to_sk(const struct bpf_sk_storage_elem *selem)
115		−{
116		− return !hlist_unhashed(&selem->snode);
117		−}
118		−
119		−static bool selem_linked_to_map(const struct bpf_sk_storage_elem *selem)
120		−{
121		− return !hlist_unhashed(&selem->map_node);
122		−}
123		−
124		−static struct bpf_sk_storage_elem selem_alloc(struct bpf_sk_storage_map smap,
125		− struct sock sk, void value,
126		− bool charge_omem)
127		−{
128		− struct bpf_sk_storage_elem *selem;
129		−
130		− if (charge_omem && omem_charge(sk, smap->elem_size))
131		− return NULL;
132		−
133		− selem = kzalloc(smap->elem_size, GFP_ATOMIC \| __GFP_NOWARN);
134		− if (selem) {
135		− if (value)
136		− memcpy(SDATA(selem)->data, value, smap->map.value_size);
137		− return selem;
138		− }
139		−
140		− if (charge_omem)
141		− atomic_sub(smap->elem_size, &sk->sk_omem_alloc);
142		−
143		− return NULL;
144		−}
145		−
146		−/* sk_storage->lock must be held and selem->sk_storage == sk_storage.
147		− * The caller must ensure selem->smap is still valid to be
148		− * dereferenced for its smap->elem_size and smap->cache_idx.
149		− */
150		−static bool __selem_unlink_sk(struct bpf_sk_storage *sk_storage,
151		− struct bpf_sk_storage_elem *selem,
152		− bool uncharge_omem)
153		−{
154		− struct bpf_sk_storage_map *smap;
155		− bool free_sk_storage;
156		− struct sock *sk;
157		−
158		− smap = rcu_dereference(SDATA(selem)->smap);
159		− sk = sk_storage->sk;
160		−
161		− /* All uncharging on sk->sk_omem_alloc must be done first.
162		− * sk may be freed once the last selem is unlinked from sk_storage.
163		− */
164		− if (uncharge_omem)
165		− atomic_sub(smap->elem_size, &sk->sk_omem_alloc);
166		−
167		− free_sk_storage = hlist_is_singular_node(&selem->snode,
168		− &sk_storage->list);
169		− if (free_sk_storage) {
170		− atomic_sub(sizeof(struct bpf_sk_storage), &sk->sk_omem_alloc);
171		− sk_storage->sk = NULL;
172		− /* After this RCU_INIT, sk may be freed and cannot be used */
173		− RCU_INIT_POINTER(sk->sk_bpf_storage, NULL);
174		−
175		− /* sk_storage is not freed now. sk_storage->lock is
176		− * still held and raw_spin_unlock_bh(&sk_storage->lock)
177		− * will be done by the caller.
178		− *
179		− * Although the unlock will be done under
180		− * rcu_read_lock(), it is more intutivie to
181		− * read if kfree_rcu(sk_storage, rcu) is done
182		− * after the raw_spin_unlock_bh(&sk_storage->lock).
183		− *
184		− * Hence, a "bool free_sk_storage" is returned
185		− * to the caller which then calls the kfree_rcu()
186		− * after unlock.
187		− */
188		− }
189		− hlist_del_init_rcu(&selem->snode);
190		− if (rcu_access_pointer(sk_storage->cache[smap->cache_idx]) ==
191		− SDATA(selem))
192		− RCU_INIT_POINTER(sk_storage->cache[smap->cache_idx], NULL);
193		−
194		− kfree_rcu(selem, rcu);
195		−
196		− return free_sk_storage;
197		−}
198		−
199		−static void selem_unlink_sk(struct bpf_sk_storage_elem *selem)
200		−{
201		− struct bpf_sk_storage *sk_storage;
202		− bool free_sk_storage = false;
203		−
204		− if (unlikely(!selem_linked_to_sk(selem)))
205		− /* selem has already been unlinked from sk */
206		− return;
207		−
208		− sk_storage = rcu_dereference(selem->sk_storage);
209		− raw_spin_lock_bh(&sk_storage->lock);
210		− if (likely(selem_linked_to_sk(selem)))
211		− free_sk_storage = __selem_unlink_sk(sk_storage, selem, true);
212		− raw_spin_unlock_bh(&sk_storage->lock);
213		−
214		− if (free_sk_storage)
215		− kfree_rcu(sk_storage, rcu);
216		−}
217		−
218		−static void __selem_link_sk(struct bpf_sk_storage *sk_storage,
219		− struct bpf_sk_storage_elem *selem)
220		−{
221		− RCU_INIT_POINTER(selem->sk_storage, sk_storage);
222		− hlist_add_head(&selem->snode, &sk_storage->list);
223		−}
224		−
225		−static void selem_unlink_map(struct bpf_sk_storage_elem *selem)
226		−{
227		− struct bpf_sk_storage_map *smap;
228		− struct bucket *b;
229		−
230		− if (unlikely(!selem_linked_to_map(selem)))
231		− /* selem has already be unlinked from smap */
232		− return;
233		−
234		− smap = rcu_dereference(SDATA(selem)->smap);
235		− b = select_bucket(smap, selem);
236		− raw_spin_lock_bh(&b->lock);
237		− if (likely(selem_linked_to_map(selem)))
238		− hlist_del_init_rcu(&selem->map_node);
239		− raw_spin_unlock_bh(&b->lock);
240		−}
241		−
242		−static void selem_link_map(struct bpf_sk_storage_map *smap,
243		− struct bpf_sk_storage_elem *selem)
244		−{
245		− struct bucket *b = select_bucket(smap, selem);
246		−
247		− raw_spin_lock_bh(&b->lock);
248		− RCU_INIT_POINTER(SDATA(selem)->smap, smap);
249		− hlist_add_head_rcu(&selem->map_node, &b->list);
250		− raw_spin_unlock_bh(&b->lock);
251		−}
252		−
253		−static void selem_unlink(struct bpf_sk_storage_elem *selem)
254		−{
255		− /* Always unlink from map before unlinking from sk_storage
256		− * because selem will be freed after successfully unlinked from
257		− * the sk_storage.
258		− */
259		− selem_unlink_map(selem);
260		− selem_unlink_sk(selem);
261		−}
262		−
263		−static struct bpf_sk_storage_data *
264		−__sk_storage_lookup(struct bpf_sk_storage *sk_storage,
265		− struct bpf_sk_storage_map *smap,
266		− bool cacheit_lockit)
267		−{
268		− struct bpf_sk_storage_data *sdata;
269		− struct bpf_sk_storage_elem *selem;
270		−
271		− /* Fast path (cache hit) */
272		− sdata = rcu_dereference(sk_storage->cache[smap->cache_idx]);
273		− if (sdata && rcu_access_pointer(sdata->smap) == smap)
274		− return sdata;
275		−
276		− /* Slow path (cache miss) */
277		− hlist_for_each_entry_rcu(selem, &sk_storage->list, snode)
278		− if (rcu_access_pointer(SDATA(selem)->smap) == smap)
279		− break;
280		−
281		− if (!selem)
282		− return NULL;
283		−
284		− sdata = SDATA(selem);
285		− if (cacheit_lockit) {
286		− /* spinlock is needed to avoid racing with the
287		− * parallel delete. Otherwise, publishing an already
288		− * deleted sdata to the cache will become a use-after-free
289		− * problem in the next __sk_storage_lookup().
290		− */
291		− raw_spin_lock_bh(&sk_storage->lock);
292		− if (selem_linked_to_sk(selem))
293		− rcu_assign_pointer(sk_storage->cache[smap->cache_idx],
294		− sdata);
295		− raw_spin_unlock_bh(&sk_storage->lock);
296		− }
297		−
298		− return sdata;
299		−}
300		−
301		−static struct bpf_sk_storage_data *
	31	+static struct bpf_local_storage_data *
302	32	sk_storage_lookup(struct sock sk, struct bpf_map map, bool cacheit_lockit)
303	33	{
304		− struct bpf_sk_storage *sk_storage;
305		− struct bpf_sk_storage_map *smap;
	34	+ struct bpf_local_storage *sk_storage;
	35	+ struct bpf_local_storage_map *smap;
306	36
307	37	sk_storage = rcu_dereference(sk->sk_bpf_storage);
308	38	if (!sk_storage)
309	39	return NULL;
310	40
311		− smap = (struct bpf_sk_storage_map *)map;
312		− return __sk_storage_lookup(sk_storage, smap, cacheit_lockit);
313		−}
314		−
315		−static int check_flags(const struct bpf_sk_storage_data *old_sdata,
316		− u64 map_flags)
317		−{
318		− if (old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_NOEXIST)
319		− /* elem already exists */
320		− return -EEXIST;
321		−
322		− if (!old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_EXIST)
323		− /* elem doesn't exist, cannot update it */
324		− return -ENOENT;
325		−
326		− return 0;
327		−}
328		−
329		−static int sk_storage_alloc(struct sock *sk,
330		− struct bpf_sk_storage_map *smap,
331		− struct bpf_sk_storage_elem *first_selem)
332		−{
333		− struct bpf_sk_storage prev_sk_storage, sk_storage;
334		− int err;
335		−
336		− err = omem_charge(sk, sizeof(*sk_storage));
337		− if (err)
338		− return err;
339		−
340		− sk_storage = kzalloc(sizeof(*sk_storage), GFP_ATOMIC \| __GFP_NOWARN);
341		− if (!sk_storage) {
342		− err = -ENOMEM;
343		− goto uncharge;
344		− }
345		− INIT_HLIST_HEAD(&sk_storage->list);
346		− raw_spin_lock_init(&sk_storage->lock);
347		− sk_storage->sk = sk;
348		−
349		− __selem_link_sk(sk_storage, first_selem);
350		− selem_link_map(smap, first_selem);
351		− /* Publish sk_storage to sk. sk->sk_lock cannot be acquired.
352		− * Hence, atomic ops is used to set sk->sk_bpf_storage
353		− * from NULL to the newly allocated sk_storage ptr.
354		− *
355		− * From now on, the sk->sk_bpf_storage pointer is protected
356		− * by the sk_storage->lock. Hence, when freeing
357		− * the sk->sk_bpf_storage, the sk_storage->lock must
358		− * be held before setting sk->sk_bpf_storage to NULL.
359		− */
360		− prev_sk_storage = cmpxchg((struct bpf_sk_storage **)&sk->sk_bpf_storage,
361		− NULL, sk_storage);
362		− if (unlikely(prev_sk_storage)) {
363		− selem_unlink_map(first_selem);
364		− err = -EAGAIN;
365		− goto uncharge;
366		−
367		− /* Note that even first_selem was linked to smap's
368		− * bucket->list, first_selem can be freed immediately
369		− * (instead of kfree_rcu) because
370		− * bpf_sk_storage_map_free() does a
371		− * synchronize_rcu() before walking the bucket->list.
372		− * Hence, no one is accessing selem from the
373		− * bucket->list under rcu_read_lock().
374		− */
375		− }
376		−
377		− return 0;
378		−
379		−uncharge:
380		− kfree(sk_storage);
381		− atomic_sub(sizeof(*sk_storage), &sk->sk_omem_alloc);
382		− return err;
383		−}
384		−
385		−/* sk cannot be going away because it is linking new elem
386		− * to sk->sk_bpf_storage. (i.e. sk->sk_refcnt cannot be 0).
387		− * Otherwise, it will become a leak (and other memory issues
388		− * during map destruction).
389		− */
390		−static struct bpf_sk_storage_data sk_storage_update(struct sock sk,
391		− struct bpf_map *map,
392		− void *value,
393		− u64 map_flags)
394		−{
395		− struct bpf_sk_storage_data *old_sdata = NULL;
396		− struct bpf_sk_storage_elem *selem;
397		− struct bpf_sk_storage *sk_storage;
398		− struct bpf_sk_storage_map *smap;
399		− int err;
400		−
401		− /* BPF_EXIST and BPF_NOEXIST cannot be both set */
402		− if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST) \|\|
403		− /* BPF_F_LOCK can only be used in a value with spin_lock */
404		− unlikely((map_flags & BPF_F_LOCK) && !map_value_has_spin_lock(map)))
405		− return ERR_PTR(-EINVAL);
406		−
407		− smap = (struct bpf_sk_storage_map *)map;
408		− sk_storage = rcu_dereference(sk->sk_bpf_storage);
409		− if (!sk_storage \|\| hlist_empty(&sk_storage->list)) {
410		− /* Very first elem for this sk */
411		− err = check_flags(NULL, map_flags);
412		− if (err)
413		− return ERR_PTR(err);
414		−
415		− selem = selem_alloc(smap, sk, value, true);
416		− if (!selem)
417		− return ERR_PTR(-ENOMEM);
418		−
419		− err = sk_storage_alloc(sk, smap, selem);
420		− if (err) {
421		− kfree(selem);
422		− atomic_sub(smap->elem_size, &sk->sk_omem_alloc);
423		− return ERR_PTR(err);
424		− }
425		−
426		− return SDATA(selem);
427		− }
428		−
429		− if ((map_flags & BPF_F_LOCK) && !(map_flags & BPF_NOEXIST)) {
430		− /* Hoping to find an old_sdata to do inline update
431		− * such that it can avoid taking the sk_storage->lock
432		− * and changing the lists.
433		− */
434		− old_sdata = __sk_storage_lookup(sk_storage, smap, false);
435		− err = check_flags(old_sdata, map_flags);
436		− if (err)
437		− return ERR_PTR(err);
438		− if (old_sdata && selem_linked_to_sk(SELEM(old_sdata))) {
439		− copy_map_value_locked(map, old_sdata->data,
440		− value, false);
441		− return old_sdata;
442		− }
443		− }
444		−
445		− raw_spin_lock_bh(&sk_storage->lock);
446		−
447		− /* Recheck sk_storage->list under sk_storage->lock */
448		− if (unlikely(hlist_empty(&sk_storage->list))) {
449		− /* A parallel del is happening and sk_storage is going
450		− * away. It has just been checked before, so very
451		− * unlikely. Return instead of retry to keep things
452		− * simple.
453		− */
454		− err = -EAGAIN;
455		− goto unlock_err;
456		− }
457		−
458		− old_sdata = __sk_storage_lookup(sk_storage, smap, false);
459		− err = check_flags(old_sdata, map_flags);
460		− if (err)
461		− goto unlock_err;
462		−
463		− if (old_sdata && (map_flags & BPF_F_LOCK)) {
464		− copy_map_value_locked(map, old_sdata->data, value, false);
465		− selem = SELEM(old_sdata);
466		− goto unlock;
467		− }
468		−
469		− /* sk_storage->lock is held. Hence, we are sure
470		− * we can unlink and uncharge the old_sdata successfully
471		− * later. Hence, instead of charging the new selem now
472		− * and then uncharge the old selem later (which may cause
473		− * a potential but unnecessary charge failure), avoid taking
474		− * a charge at all here (the "!old_sdata" check) and the
475		− * old_sdata will not be uncharged later during __selem_unlink_sk().
476		− */
477		− selem = selem_alloc(smap, sk, value, !old_sdata);
478		− if (!selem) {
479		− err = -ENOMEM;
480		− goto unlock_err;
481		− }
482		−
483		− /* First, link the new selem to the map */
484		− selem_link_map(smap, selem);
485		−
486		− /* Second, link (and publish) the new selem to sk_storage */
487		− __selem_link_sk(sk_storage, selem);
488		−
489		− /* Third, remove old selem, SELEM(old_sdata) */
490		− if (old_sdata) {
491		− selem_unlink_map(SELEM(old_sdata));
492		− __selem_unlink_sk(sk_storage, SELEM(old_sdata), false);
493		− }
494		−
495		−unlock:
496		− raw_spin_unlock_bh(&sk_storage->lock);
497		− return SDATA(selem);
498		−
499		−unlock_err:
500		− raw_spin_unlock_bh(&sk_storage->lock);
501		− return ERR_PTR(err);
	41	+ smap = (struct bpf_local_storage_map *)map;
	42	+ return bpf_local_storage_lookup(sk_storage, smap, cacheit_lockit);
502	43	}
503	44
504	45	static int sk_storage_delete(struct sock sk, struct bpf_map map)
505	46	{
506		− struct bpf_sk_storage_data *sdata;
	47	+ struct bpf_local_storage_data *sdata;
507	48
508	49	sdata = sk_storage_lookup(sk, map, false);
509	50	if (!sdata)
510	51	return -ENOENT;
511	52
512		− selem_unlink(SELEM(sdata));
	53	+ bpf_selem_unlink(SELEM(sdata));
513	54
514	55	return 0;
515	56	}
516	57
517		−static u16 cache_idx_get(void)
518		−{
519		− u64 min_usage = U64_MAX;
520		− u16 i, res = 0;
521		−
522		− spin_lock(&cache_idx_lock);
523		−
524		− for (i = 0; i < BPF_SK_STORAGE_CACHE_SIZE; i++) {
525		− if (cache_idx_usage_counts[i] < min_usage) {
526		− min_usage = cache_idx_usage_counts[i];
527		− res = i;
528		−
529		− /* Found a free cache_idx */
530		− if (!min_usage)
531		− break;
532		− }
533		− }
534		− cache_idx_usage_counts[res]++;
535		−
536		− spin_unlock(&cache_idx_lock);
537		−
538		− return res;
539		−}
540		−
541		−static void cache_idx_free(u16 idx)
542		−{
543		− spin_lock(&cache_idx_lock);
544		− cache_idx_usage_counts[idx]--;
545		− spin_unlock(&cache_idx_lock);
546		−}
547		−
548	58	/* Called by __sk_destruct() & bpf_sk_storage_clone() */
549	59	void bpf_sk_storage_free(struct sock *sk)
550	60	{
551		− struct bpf_sk_storage_elem *selem;
552		− struct bpf_sk_storage *sk_storage;
	61	+ struct bpf_local_storage_elem *selem;
	62	+ struct bpf_local_storage *sk_storage;
553	63	bool free_sk_storage = false;
554	64	struct hlist_node *n;
555	65
@@ -565,7 +75,7 @@ void bpf_sk_storage_free(struct sock *sk) · 9 unmodified lines
565	75	* Thus, no elem can be added-to or deleted-from the
566	76	* sk_storage->list by the bpf_prog or by the bpf-map's syscall.
567	77	*
568		− * It is racing with bpf_sk_storage_map_free() alone
	78	+ * It is racing with bpf_local_storage_map_free() alone
569	79	* when unlinking elem from the sk_storage->list and
570	80	* the map's bucket->list.
571	81	*/
@@ -574,8 +84,9 @@ void bpf_sk_storage_free(struct sock *sk) · 2 unmodified lines
574	84	/* Always unlink from map before unlinking from
575	85	* sk_storage.
576	86	*/
577		− selem_unlink_map(selem);
578		− free_sk_storage = __selem_unlink_sk(sk_storage, selem, true);
	87	+ bpf_selem_unlink_map(selem);
	88	+ free_sk_storage = bpf_selem_unlink_storage_nolock(sk_storage,
	89	+ selem, true);
579	90	}
580	91	raw_spin_unlock_bh(&sk_storage->lock);
581	92	rcu_read_unlock();
@@ -584,132 +95,24 @@ void bpf_sk_storage_free(struct sock *sk) · 2 unmodified lines
584	95	kfree_rcu(sk_storage, rcu);
585	96	}
586	97
587		−static void bpf_sk_storage_map_free(struct bpf_map *map)
588		−{
589		− struct bpf_sk_storage_elem *selem;
590		− struct bpf_sk_storage_map *smap;
591		− struct bucket *b;
592		− unsigned int i;
593		−
594		− smap = (struct bpf_sk_storage_map *)map;
595		−
596		− cache_idx_free(smap->cache_idx);
597		−
598		− /* Note that this map might be concurrently cloned from
599		− * bpf_sk_storage_clone. Wait for any existing bpf_sk_storage_clone
600		− * RCU read section to finish before proceeding. New RCU
601		− * read sections should be prevented via bpf_map_inc_not_zero.
602		− */
603		− synchronize_rcu();
604		−
605		− /* bpf prog and the userspace can no longer access this map
606		− * now. No new selem (of this map) can be added
607		− * to the sk->sk_bpf_storage or to the map bucket's list.
608		− *
609		− * The elem of this map can be cleaned up here
610		− * or
611		− * by bpf_sk_storage_free() during __sk_destruct().
612		− */
613		− for (i = 0; i < (1U << smap->bucket_log); i++) {
614		− b = &smap->buckets[i];
615		−
616		− rcu_read_lock();
617		− /* No one is adding to b->list now */
618		− while ((selem = hlist_entry_safe(rcu_dereference_raw(hlist_first_rcu(&b->list)),
619		− struct bpf_sk_storage_elem,
620		− map_node))) {
621		− selem_unlink(selem);
622		− cond_resched_rcu();
623		− }
624		− rcu_read_unlock();
625		− }
626		−
627		− /* bpf_sk_storage_free() may still need to access the map.
628		− * e.g. bpf_sk_storage_free() has unlinked selem from the map
629		− * which then made the above while((selem = ...)) loop
630		− * exited immediately.
631		− *
632		− * However, the bpf_sk_storage_free() still needs to access
633		− * the smap->elem_size to do the uncharging in
634		− * __selem_unlink_sk().
635		− *
636		− * Hence, wait another rcu grace period for the
637		− * bpf_sk_storage_free() to finish.
638		− */
639		− synchronize_rcu();
640		−
641		− kvfree(smap->buckets);
642		− kfree(map);
643		−}
644		−
645		−/* U16_MAX is much more than enough for sk local storage
646		− * considering a tcp_sock is ~2k.
647		− */
648		−#define MAX_VALUE_SIZE \
649		− min_t(u32, \
650		− (KMALLOC_MAX_SIZE - MAX_BPF_STACK - sizeof(struct bpf_sk_storage_elem)), \
651		− (U16_MAX - sizeof(struct bpf_sk_storage_elem)))
652		−
653		−static int bpf_sk_storage_map_alloc_check(union bpf_attr *attr)
	98	+static void sk_storage_map_free(struct bpf_map *map)
654	99	{
655		− if (attr->map_flags & ~SK_STORAGE_CREATE_FLAG_MASK \|\|
656		− !(attr->map_flags & BPF_F_NO_PREALLOC) \|\|
657		− attr->max_entries \|\|
658		− attr->key_size != sizeof(int) \|\| !attr->value_size \|\|
659		− /* Enforce BTF for userspace sk dumping */
660		− !attr->btf_key_type_id \|\| !attr->btf_value_type_id)
661		− return -EINVAL;
662		−
663		− if (!bpf_capable())
664		− return -EPERM;
	100	+ struct bpf_local_storage_map *smap;
665	101
666		− if (attr->value_size > MAX_VALUE_SIZE)
667		− return -E2BIG;
668		−
669		− return 0;
	102	+ smap = (struct bpf_local_storage_map *)map;
	103	+ bpf_local_storage_cache_idx_free(&sk_cache, smap->cache_idx);
	104	+ bpf_local_storage_map_free(smap);
670	105	}
671	106
672		−static struct bpf_map bpf_sk_storage_map_alloc(union bpf_attr attr)
	107	+static struct bpf_map sk_storage_map_alloc(union bpf_attr attr)
673	108	{
674		− struct bpf_sk_storage_map *smap;
675		− unsigned int i;
676		− u32 nbuckets;
677		− u64 cost;
678		− int ret;
679		−
680		− smap = kzalloc(sizeof(*smap), GFP_USER \| __GFP_NOWARN);
681		− if (!smap)
682		− return ERR_PTR(-ENOMEM);
683		− bpf_map_init_from_attr(&smap->map, attr);
684		−
685		− nbuckets = roundup_pow_of_two(num_possible_cpus());
686		− /* Use at least 2 buckets, select_bucket() is undefined behavior with 1 bucket */
687		− nbuckets = max_t(u32, 2, nbuckets);
688		− smap->bucket_log = ilog2(nbuckets);
689		− cost = sizeof(smap->buckets) nbuckets + sizeof(*smap);
690		−
691		− ret = bpf_map_charge_init(&smap->map.memory, cost);
692		− if (ret < 0) {
693		− kfree(smap);
694		− return ERR_PTR(ret);
695		− }
696		−
697		− smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets,
698		− GFP_USER \| __GFP_NOWARN);
699		− if (!smap->buckets) {
700		− bpf_map_charge_finish(&smap->map.memory);
701		− kfree(smap);
702		− return ERR_PTR(-ENOMEM);
703		− }
	109	+ struct bpf_local_storage_map *smap;
704	110
705		− for (i = 0; i < nbuckets; i++) {
706		− INIT_HLIST_HEAD(&smap->buckets[i].list);
707		− raw_spin_lock_init(&smap->buckets[i].lock);
708		− }
709		−
710		− smap->elem_size = sizeof(struct bpf_sk_storage_elem) + attr->value_size;
711		− smap->cache_idx = cache_idx_get();
	111	+ smap = bpf_local_storage_map_alloc(attr);
	112	+ if (IS_ERR(smap))
	113	+ return ERR_CAST(smap);
712	114
	115	+ smap->cache_idx = bpf_local_storage_cache_idx_get(&sk_cache);
713	116	return &smap->map;
714	117	}
715	118
@@ -719,26 +122,9 @@ static int notsupp_get_next_key(struct bpf_map map, void key, · 3 unmodified lines
719	122	return -ENOTSUPP;
720	123	}
721	124
722		−static int bpf_sk_storage_map_check_btf(const struct bpf_map *map,
723		− const struct btf *btf,
724		− const struct btf_type *key_type,
725		− const struct btf_type *value_type)
726		−{
727		− u32 int_data;
728		−
729		− if (BTF_INFO_KIND(key_type->info) != BTF_KIND_INT)
730		− return -EINVAL;
731		−
732		− int_data = (u32 )(key_type + 1);
733		− if (BTF_INT_BITS(int_data) != 32 \|\| BTF_INT_OFFSET(int_data))
734		− return -EINVAL;
735		−
736		− return 0;
737		−}
738		−
739	125	static void bpf_fd_sk_storage_lookup_elem(struct bpf_map map, void *key)
740	126	{
741		− struct bpf_sk_storage_data *sdata;
	127	+ struct bpf_local_storage_data *sdata;
742	128	struct socket *sock;
743	129	int fd, err;
744	130
@@ -756,14 +142,16 @@ static void bpf_fd_sk_storage_lookup_elem(struct bpf_map map, void *key) · 11 unmodified lines
756	142	static int bpf_fd_sk_storage_update_elem(struct bpf_map map, void key,
757	143	void *value, u64 map_flags)
758	144	{
759		− struct bpf_sk_storage_data *sdata;
	145	+ struct bpf_local_storage_data *sdata;
760	146	struct socket *sock;
761	147	int fd, err;
762	148
763	149	fd = (int )key;
764	150	sock = sockfd_lookup(fd, &err);
765	151	if (sock) {
766		− sdata = sk_storage_update(sock->sk, map, value, map_flags);
	152	+ sdata = bpf_local_storage_update(
	153	+ sock->sk, (struct bpf_local_storage_map *)map, value,
	154	+ map_flags);
767	155	sockfd_put(sock);
768	156	return PTR_ERR_OR_ZERO(sdata);
769	157	}
@@ -787,14 +175,14 @@ static int bpf_fd_sk_storage_delete_elem(struct bpf_map map, void key) · 17 unmodified lines
787	175	return err;
788	176	}
789	177
790		−static struct bpf_sk_storage_elem *
	178	+static struct bpf_local_storage_elem *
791	179	bpf_sk_storage_clone_elem(struct sock *newsk,
792		− struct bpf_sk_storage_map *smap,
793		− struct bpf_sk_storage_elem *selem)
	180	+ struct bpf_local_storage_map *smap,
	181	+ struct bpf_local_storage_elem *selem)
794	182	{
795		− struct bpf_sk_storage_elem *copy_selem;
	183	+ struct bpf_local_storage_elem *copy_selem;
796	184
797		− copy_selem = selem_alloc(smap, newsk, NULL, true);
	185	+ copy_selem = bpf_selem_alloc(smap, newsk, NULL, true);
798	186	if (!copy_selem)
799	187	return NULL;
800	188
@@ -810,9 +198,9 @@ bpf_sk_storage_clone_elem(struct sock *newsk, · 9 unmodified lines
810	198
811	199	int bpf_sk_storage_clone(const struct sock sk, struct sock newsk)
812	200	{
813		− struct bpf_sk_storage *new_sk_storage = NULL;
814		− struct bpf_sk_storage *sk_storage;
815		− struct bpf_sk_storage_elem *selem;
	201	+ struct bpf_local_storage *new_sk_storage = NULL;
	202	+ struct bpf_local_storage *sk_storage;
	203	+ struct bpf_local_storage_elem *selem;
816	204	int ret = 0;
817	205
818	206	RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL);
@@ -824,8 +212,8 @@ int bpf_sk_storage_clone(const struct sock sk, struct sock newsk) · 5 unmodified lines
824	212	goto out;
825	213
826	214	hlist_for_each_entry_rcu(selem, &sk_storage->list, snode) {
827		− struct bpf_sk_storage_elem *copy_selem;
828		− struct bpf_sk_storage_map *smap;
	215	+ struct bpf_local_storage_elem *copy_selem;
	216	+ struct bpf_local_storage_map *smap;
829	217	struct bpf_map *map;
830	218
831	219	smap = rcu_dereference(SDATA(selem)->smap);
@@ -833,7 +221,7 @@ int bpf_sk_storage_clone(const struct sock sk, struct sock newsk) · 1 unmodified line
833	221	continue;
834	222
835	223	/* Note that for lockless listeners adding new element
836		− * here can race with cleanup in bpf_sk_storage_map_free.
	224	+ * here can race with cleanup in bpf_local_storage_map_free.
837	225	* Try to grab map refcnt to make sure that it's still
838	226	* alive and prevent concurrent removal.
839	227	*/
@@ -849,10 +237,10 @@ int bpf_sk_storage_clone(const struct sock sk, struct sock newsk) · 9 unmodified lines
849	237	}
850	238
851	239	if (new_sk_storage) {
852		− selem_link_map(smap, copy_selem);
853		− __selem_link_sk(new_sk_storage, copy_selem);
	240	+ bpf_selem_link_map(smap, copy_selem);
	241	+ bpf_selem_link_storage_nolock(new_sk_storage, copy_selem);
854	242	} else {
855		− ret = sk_storage_alloc(newsk, smap, copy_selem);
	243	+ ret = bpf_local_storage_alloc(newsk, smap, copy_selem);
856	244	if (ret) {
857	245	kfree(copy_selem);
858	246	atomic_sub(smap->elem_size,
@@ -861,7 +249,8 @@ int bpf_sk_storage_clone(const struct sock sk, struct sock newsk) · 2 unmodified lines
861	249	goto out;
862	250	}
863	251
864		− new_sk_storage = rcu_dereference(copy_selem->sk_storage);
	252	+ new_sk_storage =
	253	+ rcu_dereference(copy_selem->local_storage);
865	254	}
866	255	bpf_map_put(map);
867	256	}
@@ -879,7 +268,7 @@ out: · 11 unmodified lines
879	268	BPF_CALL_4(bpf_sk_storage_get, struct bpf_map , map, struct sock , sk,
880	269	void *, value, u64, flags)
881	270	{
882		− struct bpf_sk_storage_data *sdata;
	271	+ struct bpf_local_storage_data *sdata;
883	272
884	273	if (flags > BPF_SK_STORAGE_GET_F_CREATE)
885	274	return (unsigned long)NULL;
@@ -895,7 +284,9 @@ BPF_CALL_4(bpf_sk_storage_get, struct bpf_map , map, struct sock , sk, · 9 unmodified lines
895	284	* destruction).
896	285	*/
897	286	refcount_inc_not_zero(&sk->sk_refcnt)) {
898		− sdata = sk_storage_update(sk, map, value, BPF_NOEXIST);
	287	+ sdata = bpf_local_storage_update(
	288	+ sk, (struct bpf_local_storage_map *)map, value,
	289	+ BPF_NOEXIST);
899	290	/* sk must be a fullsock (guaranteed by verifier),
900	291	* so sock_gen_put() is unnecessary.
901	292	*/
@@ -920,18 +311,44 @@ BPF_CALL_2(bpf_sk_storage_delete, struct bpf_map , map, struct sock , sk) · 18 unmodified lines
920	311	return -ENOENT;
921	312	}
922	313
	314	+static int sk_storage_charge(struct bpf_local_storage_map *smap,
	315	+ void *owner, u32 size)
	316	+{
	317	+ return omem_charge(owner, size);
	318	+}
	319	+
	320	+static void sk_storage_uncharge(struct bpf_local_storage_map *smap,
	321	+ void *owner, u32 size)
	322	+{
	323	+ struct sock *sk = owner;
	324	+
	325	+ atomic_sub(size, &sk->sk_omem_alloc);
	326	+}
	327	+
	328	+static struct bpf_local_storage __rcu **
	329	+sk_storage_ptr(void *owner)
	330	+{
	331	+ struct sock *sk = owner;
	332	+
	333	+ return &sk->sk_bpf_storage;
	334	+}
	335	+
923	336	static int sk_storage_map_btf_id;
924	337	const struct bpf_map_ops sk_storage_map_ops = {
925		− .map_alloc_check = bpf_sk_storage_map_alloc_check,
926		− .map_alloc = bpf_sk_storage_map_alloc,
927		− .map_free = bpf_sk_storage_map_free,
	338	+ .map_meta_equal = bpf_map_meta_equal,
	339	+ .map_alloc_check = bpf_local_storage_map_alloc_check,
	340	+ .map_alloc = sk_storage_map_alloc,
	341	+ .map_free = sk_storage_map_free,
928	342	.map_get_next_key = notsupp_get_next_key,
929	343	.map_lookup_elem = bpf_fd_sk_storage_lookup_elem,
930	344	.map_update_elem = bpf_fd_sk_storage_update_elem,
931	345	.map_delete_elem = bpf_fd_sk_storage_delete_elem,
932		− .map_check_btf = bpf_sk_storage_map_check_btf,
933		− .map_btf_name = "bpf_sk_storage_map",
	346	+ .map_check_btf = bpf_local_storage_map_check_btf,
	347	+ .map_btf_name = "bpf_local_storage_map",
934	348	.map_btf_id = &sk_storage_map_btf_id,
	349	+ .map_local_storage_charge = sk_storage_charge,
	350	+ .map_local_storage_uncharge = sk_storage_uncharge,
	351	+ .map_owner_storage_ptr = sk_storage_ptr,
935	352	};
936	353
937	354	const struct bpf_func_proto bpf_sk_storage_get_proto = {
@@ -962,6 +379,30 @@ const struct bpf_func_proto bpf_sk_storage_delete_proto = { · 24 unmodified lines
962	379	.arg2_type = ARG_PTR_TO_SOCKET,
963	380	};
964	381
	382	+BTF_ID_LIST(sk_storage_btf_ids)
	383	+BTF_ID_UNUSED
	384	+BTF_ID(struct, sock)
	385	+
	386	+const struct bpf_func_proto sk_storage_get_btf_proto = {
	387	+ .func = bpf_sk_storage_get,
	388	+ .gpl_only = false,
	389	+ .ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
	390	+ .arg1_type = ARG_CONST_MAP_PTR,
	391	+ .arg2_type = ARG_PTR_TO_BTF_ID,
	392	+ .arg3_type = ARG_PTR_TO_MAP_VALUE_OR_NULL,
	393	+ .arg4_type = ARG_ANYTHING,
	394	+ .btf_id = sk_storage_btf_ids,
	395	+};
	396	+
	397	+const struct bpf_func_proto sk_storage_delete_btf_proto = {
	398	+ .func = bpf_sk_storage_delete,
	399	+ .gpl_only = false,
	400	+ .ret_type = RET_INTEGER,
	401	+ .arg1_type = ARG_CONST_MAP_PTR,
	402	+ .arg2_type = ARG_PTR_TO_BTF_ID,
	403	+ .btf_id = sk_storage_btf_ids,
	404	+};
	405	+
965	406	struct bpf_sk_storage_diag {
966	407	u32 nr_maps;
967	408	struct bpf_map *maps[];
@@ -1022,7 +463,7 @@ bpf_sk_storage_diag_alloc(const struct nlattr *nla_stgs) · 54 unmodified lines
1022	463	u32 nr_maps = 0;
1023	464	int rem, err;
1024	465
1025		− /* bpf_sk_storage_map is currently limited to CAP_SYS_ADMIN as
	466	+ /* bpf_local_storage_map is currently limited to CAP_SYS_ADMIN as
1026	467	* the map_alloc_check() side also does.
1027	468	*/
1028	469	if (!bpf_capable())
@@ -1072,13 +513,13 @@ err_free: · 43 unmodified lines
1072	513	}
1073	514	EXPORT_SYMBOL_GPL(bpf_sk_storage_diag_alloc);
1074	515
1075		−static int diag_get(struct bpf_sk_storage_data sdata, struct sk_buff skb)
	516	+static int diag_get(struct bpf_local_storage_data sdata, struct sk_buff skb)
1076	517	{
1077	518	struct nlattr nla_stg, nla_value;
1078		− struct bpf_sk_storage_map *smap;
	519	+ struct bpf_local_storage_map *smap;
1079	520
1080	521	/* It cannot exceed max nlattr's payload */
1081		− BUILD_BUG_ON(U16_MAX - NLA_HDRLEN < MAX_VALUE_SIZE);
	522	+ BUILD_BUG_ON(U16_MAX - NLA_HDRLEN < BPF_LOCAL_STORAGE_MAX_VALUE_SIZE);
1082	523
1083	524	nla_stg = nla_nest_start(skb, SK_DIAG_BPF_STORAGE);
1084	525	if (!nla_stg)
@@ -1114,9 +555,9 @@ static int bpf_sk_storage_diag_put_all(struct sock sk, struct sk_buff skb, · 29 unmodified lines
1114	555	{
1115	556	/* stg_array_type (e.g. INET_DIAG_BPF_SK_STORAGES) */
1116	557	unsigned int diag_size = nla_total_size(0);
1117		− struct bpf_sk_storage *sk_storage;
1118		− struct bpf_sk_storage_elem *selem;
1119		− struct bpf_sk_storage_map *smap;
	558	+ struct bpf_local_storage *sk_storage;
	559	+ struct bpf_local_storage_elem *selem;
	560	+ struct bpf_local_storage_map *smap;
1120	561	struct nlattr *nla_stgs;
1121	562	unsigned int saved_len;
1122	563	int err = 0;
@@ -1169,8 +610,8 @@ int bpf_sk_storage_diag_put(struct bpf_sk_storage_diag *diag, · 46 unmodified lines
1169	610	{
1170	611	/* stg_array_type (e.g. INET_DIAG_BPF_SK_STORAGES) */
1171	612	unsigned int diag_size = nla_total_size(0);
1172		− struct bpf_sk_storage *sk_storage;
1173		− struct bpf_sk_storage_data *sdata;
	613	+ struct bpf_local_storage *sk_storage;
	614	+ struct bpf_local_storage_data *sdata;
1174	615	struct nlattr *nla_stgs;
1175	616	unsigned int saved_len;
1176	617	int err = 0;
@@ -1197,8 +638,8 @@ int bpf_sk_storage_diag_put(struct bpf_sk_storage_diag *diag, · 20 unmodified lines
1197	638
1198	639	saved_len = skb->len;
1199	640	for (i = 0; i < diag->nr_maps; i++) {
1200		− sdata = __sk_storage_lookup(sk_storage,
1201		− (struct bpf_sk_storage_map *)diag->maps[i],
	641	+ sdata = bpf_local_storage_lookup(sk_storage,
	642	+ (struct bpf_local_storage_map *)diag->maps[i],
1202	643	false);
1203	644
1204	645	if (!sdata)
@@ -1235,19 +676,19 @@ struct bpf_iter_seq_sk_storage_map_info { · 30 unmodified lines
1235	676	unsigned skip_elems;
1236	677	};
1237	678
1238		−static struct bpf_sk_storage_elem *
	679	+static struct bpf_local_storage_elem *
1239	680	bpf_sk_storage_map_seq_find_next(struct bpf_iter_seq_sk_storage_map_info *info,
1240		− struct bpf_sk_storage_elem *prev_selem)
	681	+ struct bpf_local_storage_elem *prev_selem)
1241	682	{
1242		− struct bpf_sk_storage *sk_storage;
1243		− struct bpf_sk_storage_elem *selem;
	683	+ struct bpf_local_storage *sk_storage;
	684	+ struct bpf_local_storage_elem *selem;
1244	685	u32 skip_elems = info->skip_elems;
1245		− struct bpf_sk_storage_map *smap;
	686	+ struct bpf_local_storage_map *smap;
1246	687	u32 bucket_id = info->bucket_id;
1247	688	u32 i, count, n_buckets;
1248		− struct bucket *b;
	689	+ struct bpf_local_storage_map_bucket *b;
1249	690
1250		− smap = (struct bpf_sk_storage_map *)info->map;
	691	+ smap = (struct bpf_local_storage_map *)info->map;
1251	692	n_buckets = 1U << smap->bucket_log;
1252	693	if (bucket_id >= n_buckets)
1253	694	return NULL;
@@ -1257,7 +698,7 @@ bpf_sk_storage_map_seq_find_next(struct bpf_iter_seq_sk_storage_map_info *info, · 3 unmodified lines
1257	698	count = 0;
1258	699	while (selem) {
1259	700	selem = hlist_entry_safe(selem->map_node.next,
1260		− struct bpf_sk_storage_elem, map_node);
	701	+ struct bpf_local_storage_elem, map_node);
1261	702	if (!selem) {
1262	703	/* not found, unlock and go to the next bucket */
1263	704	b = &smap->buckets[bucket_id++];
@@ -1265,7 +706,7 @@ bpf_sk_storage_map_seq_find_next(struct bpf_iter_seq_sk_storage_map_info *info, · 1 unmodified line
1265	706	skip_elems = 0;
1266	707	break;
1267	708	}
1268		− sk_storage = rcu_dereference_raw(selem->sk_storage);
	709	+ sk_storage = rcu_dereference_raw(selem->local_storage);
1269	710	if (sk_storage) {
1270	711	info->skip_elems = skip_elems + count;
1271	712	return selem;
@@ -1278,7 +719,7 @@ bpf_sk_storage_map_seq_find_next(struct bpf_iter_seq_sk_storage_map_info *info, · 6 unmodified lines
1278	719	raw_spin_lock_bh(&b->lock);
1279	720	count = 0;
1280	721	hlist_for_each_entry(selem, &b->list, map_node) {
1281		− sk_storage = rcu_dereference_raw(selem->sk_storage);
	722	+ sk_storage = rcu_dereference_raw(selem->local_storage);
1282	723	if (sk_storage && count >= skip_elems) {
1283	724	info->bucket_id = i;
1284	725	info->skip_elems = count;
@@ -1297,7 +738,7 @@ bpf_sk_storage_map_seq_find_next(struct bpf_iter_seq_sk_storage_map_info *info, · 12 unmodified lines
1297	738
1298	739	static void bpf_sk_storage_map_seq_start(struct seq_file seq, loff_t *pos)
1299	740	{
1300		− struct bpf_sk_storage_elem *selem;
	741	+ struct bpf_local_storage_elem *selem;
1301	742
1302	743	selem = bpf_sk_storage_map_seq_find_next(seq->private, NULL);
1303	744	if (!selem)
@@ -1330,11 +771,11 @@ DEFINE_BPF_ITER_FUNC(bpf_sk_storage_map, struct bpf_iter_meta *meta, · 26 unmodified lines
1330	771	void *value)
1331	772
1332	773	static int __bpf_sk_storage_map_seq_show(struct seq_file *seq,
1333		− struct bpf_sk_storage_elem *selem)
	774	+ struct bpf_local_storage_elem *selem)
1334	775	{
1335	776	struct bpf_iter_seq_sk_storage_map_info *info = seq->private;
1336	777	struct bpf_iter__bpf_sk_storage_map ctx = {};
1337		− struct bpf_sk_storage *sk_storage;
	778	+ struct bpf_local_storage *sk_storage;
1338	779	struct bpf_iter_meta meta;
1339	780	struct bpf_prog *prog;
1340	781	int ret = 0;
@@ -1345,8 +786,8 @@ static int __bpf_sk_storage_map_seq_show(struct seq_file *seq, · 4 unmodified lines
1345	786	ctx.meta = &meta;
1346	787	ctx.map = info->map;
1347	788	if (selem) {
1348		− sk_storage = rcu_dereference_raw(selem->sk_storage);
1349		− ctx.sk = sk_storage->sk;
	789	+ sk_storage = rcu_dereference_raw(selem->local_storage);
	790	+ ctx.sk = sk_storage->owner;
1350	791	ctx.value = SDATA(selem)->data;
1351	792	}
1352	793	ret = bpf_iter_run_prog(prog, &ctx);
@@ -1363,13 +804,13 @@ static int bpf_sk_storage_map_seq_show(struct seq_file seq, void v) · 10 unmodified lines
1363	804	static void bpf_sk_storage_map_seq_stop(struct seq_file seq, void v)
1364	805	{
1365	806	struct bpf_iter_seq_sk_storage_map_info *info = seq->private;
1366		− struct bpf_sk_storage_map *smap;
1367		− struct bucket *b;
	807	+ struct bpf_local_storage_map *smap;
	808	+ struct bpf_local_storage_map_bucket *b;
1368	809
1369	810	if (!v) {
1370	811	(void)__bpf_sk_storage_map_seq_show(seq, v);
1371	812	} else {
1372		− smap = (struct bpf_sk_storage_map *)info->map;
	813	+ smap = (struct bpf_local_storage_map *)info->map;
1373	814	b = &smap->buckets[info->bucket_id];
1374	815	raw_spin_unlock_bh(&b->lock);
1375	816	}
@@ -1437,6 +878,8 @@ static struct bpf_iter_reg bpf_sk_storage_map_reg_info = { · 61 unmodified lines
1437	878	.target = "bpf_sk_storage_map",
1438	879	.attach_target = bpf_iter_attach_map,
1439	880	.detach_target = bpf_iter_detach_map,
	881	+ .show_fdinfo = bpf_iter_map_show_fdinfo,
	882	+ .fill_link_info = bpf_iter_map_fill_link_info,
1440	883	.ctx_arg_info_size = 2,
1441	884	.ctx_arg_info = {
1442	885	{ offsetof(struct bpf_iter__bpf_sk_storage_map, sk),

net/core/filter.c+409−7

@@ -4459,6 +4459,7 @@ static int _bpf_setsockopt(struct sock *sk, int level, int optname, · 4458 unmodified lines
4459	4459	} else {
4460	4460	struct inet_connection_sock *icsk = inet_csk(sk);
4461	4461	struct tcp_sock *tp = tcp_sk(sk);
	4462	+ unsigned long timeout;
4462	4463
4463	4464	if (optlen != sizeof(int))
4464	4465	return -EINVAL;
@@ -4480,6 +4481,20 @@ static int _bpf_setsockopt(struct sock *sk, int level, int optname, · 15 unmodified lines
4480	4481	tp->snd_ssthresh = val;
4481	4482	}
4482	4483	break;
	4484	+ case TCP_BPF_DELACK_MAX:
	4485	+ timeout = usecs_to_jiffies(val);
	4486	+ if (timeout > TCP_DELACK_MAX \|\|
	4487	+ timeout < TCP_TIMEOUT_MIN)
	4488	+ return -EINVAL;
	4489	+ inet_csk(sk)->icsk_delack_max = timeout;
	4490	+ break;
	4491	+ case TCP_BPF_RTO_MIN:
	4492	+ timeout = usecs_to_jiffies(val);
	4493	+ if (timeout > TCP_RTO_MIN \|\|
	4494	+ timeout < TCP_TIMEOUT_MIN)
	4495	+ return -EINVAL;
	4496	+ inet_csk(sk)->icsk_rto_min = timeout;
	4497	+ break;
4483	4498	case TCP_SAVE_SYN:
4484	4499	if (val < 0 \|\| val > 1)
4485	4500	ret = -EINVAL;
@@ -4550,9 +4565,9 @@ static int _bpf_getsockopt(struct sock *sk, int level, int optname, · 64 unmodified lines
4550	4565	tp = tcp_sk(sk);
4551	4566
4552	4567	if (optlen <= 0 \|\| !tp->saved_syn \|\|
4553		− optlen > tp->saved_syn[0])
	4568	+ optlen > tcp_saved_syn_len(tp->saved_syn))
4554	4569	goto err_clear;
4555		− memcpy(optval, tp->saved_syn + 1, optlen);
	4570	+ memcpy(optval, tp->saved_syn->data, optlen);
4556	4571	break;
4557	4572	default:
4558	4573	goto err_clear;
@@ -4654,9 +4669,99 @@ static const struct bpf_func_proto bpf_sock_ops_setsockopt_proto = { · 95 unmodified lines
4654	4669	.arg5_type = ARG_CONST_SIZE,
4655	4670	};
4656	4671
	4672	+static int bpf_sock_ops_get_syn(struct bpf_sock_ops_kern *bpf_sock,
	4673	+ int optname, const u8 **start)
	4674	+{
	4675	+ struct sk_buff *syn_skb = bpf_sock->syn_skb;
	4676	+ const u8 *hdr_start;
	4677	+ int ret;
	4678	+
	4679	+ if (syn_skb) {
	4680	+ /* sk is a request_sock here */
	4681	+
	4682	+ if (optname == TCP_BPF_SYN) {
	4683	+ hdr_start = syn_skb->data;
	4684	+ ret = tcp_hdrlen(syn_skb);
	4685	+ } else if (optname == TCP_BPF_SYN_IP) {
	4686	+ hdr_start = skb_network_header(syn_skb);
	4687	+ ret = skb_network_header_len(syn_skb) +
	4688	+ tcp_hdrlen(syn_skb);
	4689	+ } else {
	4690	+ /* optname == TCP_BPF_SYN_MAC */
	4691	+ hdr_start = skb_mac_header(syn_skb);
	4692	+ ret = skb_mac_header_len(syn_skb) +
	4693	+ skb_network_header_len(syn_skb) +
	4694	+ tcp_hdrlen(syn_skb);
	4695	+ }
	4696	+ } else {
	4697	+ struct sock *sk = bpf_sock->sk;
	4698	+ struct saved_syn *saved_syn;
	4699	+
	4700	+ if (sk->sk_state == TCP_NEW_SYN_RECV)
	4701	+ /* synack retransmit. bpf_sock->syn_skb will
	4702	+ * not be available. It has to resort to
	4703	+ * saved_syn (if it is saved).
	4704	+ */
	4705	+ saved_syn = inet_reqsk(sk)->saved_syn;
	4706	+ else
	4707	+ saved_syn = tcp_sk(sk)->saved_syn;
	4708	+
	4709	+ if (!saved_syn)
	4710	+ return -ENOENT;
	4711	+
	4712	+ if (optname == TCP_BPF_SYN) {
	4713	+ hdr_start = saved_syn->data +
	4714	+ saved_syn->mac_hdrlen +
	4715	+ saved_syn->network_hdrlen;
	4716	+ ret = saved_syn->tcp_hdrlen;
	4717	+ } else if (optname == TCP_BPF_SYN_IP) {
	4718	+ hdr_start = saved_syn->data +
	4719	+ saved_syn->mac_hdrlen;
	4720	+ ret = saved_syn->network_hdrlen +
	4721	+ saved_syn->tcp_hdrlen;
	4722	+ } else {
	4723	+ /* optname == TCP_BPF_SYN_MAC */
	4724	+
	4725	+ /* TCP_SAVE_SYN may not have saved the mac hdr */
	4726	+ if (!saved_syn->mac_hdrlen)
	4727	+ return -ENOENT;
	4728	+
	4729	+ hdr_start = saved_syn->data;
	4730	+ ret = saved_syn->mac_hdrlen +
	4731	+ saved_syn->network_hdrlen +
	4732	+ saved_syn->tcp_hdrlen;
	4733	+ }
	4734	+ }
	4735	+
	4736	+ *start = hdr_start;
	4737	+ return ret;
	4738	+}
	4739	+
4657	4740	BPF_CALL_5(bpf_sock_ops_getsockopt, struct bpf_sock_ops_kern *, bpf_sock,
4658	4741	int, level, int, optname, char *, optval, int, optlen)
4659	4742	{
	4743	+ if (IS_ENABLED(CONFIG_INET) && level == SOL_TCP &&
	4744	+ optname >= TCP_BPF_SYN && optname <= TCP_BPF_SYN_MAC) {
	4745	+ int ret, copy_len = 0;
	4746	+ const u8 *start;
	4747	+
	4748	+ ret = bpf_sock_ops_get_syn(bpf_sock, optname, &start);
	4749	+ if (ret > 0) {
	4750	+ copy_len = ret;
	4751	+ if (optlen < copy_len) {
	4752	+ copy_len = optlen;
	4753	+ ret = -ENOSPC;
	4754	+ }
	4755	+
	4756	+ memcpy(optval, start, copy_len);
	4757	+ }
	4758	+
	4759	+ /* Zero out unused buffer at the end */
	4760	+ memset(optval + copy_len, 0, optlen - copy_len);
	4761	+
	4762	+ return ret;
	4763	+ }
	4764	+
4660	4765	return _bpf_getsockopt(bpf_sock->sk, level, optname, optval, optlen);
4661	4766	}
4662	4767
@@ -6150,6 +6255,232 @@ static const struct bpf_func_proto bpf_sk_assign_proto = { · 1487 unmodified lines
6150	6255	.arg3_type = ARG_ANYTHING,
6151	6256	};
6152	6257
	6258	+static const u8 bpf_search_tcp_opt(const u8 op, const u8 *opend,
	6259	+ u8 search_kind, const u8 *magic,
	6260	+ u8 magic_len, bool *eol)
	6261	+{
	6262	+ u8 kind, kind_len;
	6263	+
	6264	+ *eol = false;
	6265	+
	6266	+ while (op < opend) {
	6267	+ kind = op[0];
	6268	+
	6269	+ if (kind == TCPOPT_EOL) {
	6270	+ *eol = true;
	6271	+ return ERR_PTR(-ENOMSG);
	6272	+ } else if (kind == TCPOPT_NOP) {
	6273	+ op++;
	6274	+ continue;
	6275	+ }
	6276	+
	6277	+ if (opend - op < 2 \|\| opend - op < op[1] \|\| op[1] < 2)
	6278	+ /* Something is wrong in the received header.
	6279	+ * Follow the TCP stack's tcp_parse_options()
	6280	+ * and just bail here.
	6281	+ */
	6282	+ return ERR_PTR(-EFAULT);
	6283	+
	6284	+ kind_len = op[1];
	6285	+ if (search_kind == kind) {
	6286	+ if (!magic_len)
	6287	+ return op;
	6288	+
	6289	+ if (magic_len > kind_len - 2)
	6290	+ return ERR_PTR(-ENOMSG);
	6291	+
	6292	+ if (!memcmp(&op[2], magic, magic_len))
	6293	+ return op;
	6294	+ }
	6295	+
	6296	+ op += kind_len;
	6297	+ }
	6298	+
	6299	+ return ERR_PTR(-ENOMSG);
	6300	+}
	6301	+
	6302	+BPF_CALL_4(bpf_sock_ops_load_hdr_opt, struct bpf_sock_ops_kern *, bpf_sock,
	6303	+ void *, search_res, u32, len, u64, flags)
	6304	+{
	6305	+ bool eol, load_syn = flags & BPF_LOAD_HDR_OPT_TCP_SYN;
	6306	+ const u8 op, opend, magic, search = search_res;
	6307	+ u8 search_kind, search_len, copy_len, magic_len;
	6308	+ int ret;
	6309	+
	6310	+ /* 2 byte is the minimal option len except TCPOPT_NOP and
	6311	+ * TCPOPT_EOL which are useless for the bpf prog to learn
	6312	+ * and this helper disallow loading them also.
	6313	+ */
	6314	+ if (len < 2 \|\| flags & ~BPF_LOAD_HDR_OPT_TCP_SYN)
	6315	+ return -EINVAL;
	6316	+
	6317	+ search_kind = search[0];
	6318	+ search_len = search[1];
	6319	+
	6320	+ if (search_len > len \|\| search_kind == TCPOPT_NOP \|\|
	6321	+ search_kind == TCPOPT_EOL)
	6322	+ return -EINVAL;
	6323	+
	6324	+ if (search_kind == TCPOPT_EXP \|\| search_kind == 253) {
	6325	+ /* 16 or 32 bit magic. +2 for kind and kind length */
	6326	+ if (search_len != 4 && search_len != 6)
	6327	+ return -EINVAL;
	6328	+ magic = &search[2];
	6329	+ magic_len = search_len - 2;
	6330	+ } else {
	6331	+ if (search_len)
	6332	+ return -EINVAL;
	6333	+ magic = NULL;
	6334	+ magic_len = 0;
	6335	+ }
	6336	+
	6337	+ if (load_syn) {
	6338	+ ret = bpf_sock_ops_get_syn(bpf_sock, TCP_BPF_SYN, &op);
	6339	+ if (ret < 0)
	6340	+ return ret;
	6341	+
	6342	+ opend = op + ret;
	6343	+ op += sizeof(struct tcphdr);
	6344	+ } else {
	6345	+ if (!bpf_sock->skb \|\|
	6346	+ bpf_sock->op == BPF_SOCK_OPS_HDR_OPT_LEN_CB)
	6347	+ /* This bpf_sock->op cannot call this helper */
	6348	+ return -EPERM;
	6349	+
	6350	+ opend = bpf_sock->skb_data_end;
	6351	+ op = bpf_sock->skb->data + sizeof(struct tcphdr);
	6352	+ }
	6353	+
	6354	+ op = bpf_search_tcp_opt(op, opend, search_kind, magic, magic_len,
	6355	+ &eol);
	6356	+ if (IS_ERR(op))
	6357	+ return PTR_ERR(op);
	6358	+
	6359	+ copy_len = op[1];
	6360	+ ret = copy_len;
	6361	+ if (copy_len > len) {
	6362	+ ret = -ENOSPC;
	6363	+ copy_len = len;
	6364	+ }
	6365	+
	6366	+ memcpy(search_res, op, copy_len);
	6367	+ return ret;
	6368	+}
	6369	+
	6370	+static const struct bpf_func_proto bpf_sock_ops_load_hdr_opt_proto = {
	6371	+ .func = bpf_sock_ops_load_hdr_opt,
	6372	+ .gpl_only = false,
	6373	+ .ret_type = RET_INTEGER,
	6374	+ .arg1_type = ARG_PTR_TO_CTX,
	6375	+ .arg2_type = ARG_PTR_TO_MEM,
	6376	+ .arg3_type = ARG_CONST_SIZE,
	6377	+ .arg4_type = ARG_ANYTHING,
	6378	+};
	6379	+
	6380	+BPF_CALL_4(bpf_sock_ops_store_hdr_opt, struct bpf_sock_ops_kern *, bpf_sock,
	6381	+ const void *, from, u32, len, u64, flags)
	6382	+{
	6383	+ u8 new_kind, new_kind_len, magic_len = 0, *opend;
	6384	+ const u8 op, new_op, *magic = NULL;
	6385	+ struct sk_buff *skb;
	6386	+ bool eol;
	6387	+
	6388	+ if (bpf_sock->op != BPF_SOCK_OPS_WRITE_HDR_OPT_CB)
	6389	+ return -EPERM;
	6390	+
	6391	+ if (len < 2 \|\| flags)
	6392	+ return -EINVAL;
	6393	+
	6394	+ new_op = from;
	6395	+ new_kind = new_op[0];
	6396	+ new_kind_len = new_op[1];
	6397	+
	6398	+ if (new_kind_len > len \|\| new_kind == TCPOPT_NOP \|\|
	6399	+ new_kind == TCPOPT_EOL)
	6400	+ return -EINVAL;
	6401	+
	6402	+ if (new_kind_len > bpf_sock->remaining_opt_len)
	6403	+ return -ENOSPC;
	6404	+
	6405	+ /* 253 is another experimental kind */
	6406	+ if (new_kind == TCPOPT_EXP \|\| new_kind == 253) {
	6407	+ if (new_kind_len < 4)
	6408	+ return -EINVAL;
	6409	+ /* Match for the 2 byte magic also.
	6410	+ * RFC 6994: the magic could be 2 or 4 bytes.
	6411	+ * Hence, matching by 2 byte only is on the
	6412	+ * conservative side but it is the right
	6413	+ * thing to do for the 'search-for-duplication'
	6414	+ * purpose.
	6415	+ */
	6416	+ magic = &new_op[2];
	6417	+ magic_len = 2;
	6418	+ }
	6419	+
	6420	+ /* Check for duplication */
	6421	+ skb = bpf_sock->skb;
	6422	+ op = skb->data + sizeof(struct tcphdr);
	6423	+ opend = bpf_sock->skb_data_end;
	6424	+
	6425	+ op = bpf_search_tcp_opt(op, opend, new_kind, magic, magic_len,
	6426	+ &eol);
	6427	+ if (!IS_ERR(op))
	6428	+ return -EEXIST;
	6429	+
	6430	+ if (PTR_ERR(op) != -ENOMSG)
	6431	+ return PTR_ERR(op);
	6432	+
	6433	+ if (eol)
	6434	+ /* The option has been ended. Treat it as no more
	6435	+ * header option can be written.
	6436	+ */
	6437	+ return -ENOSPC;
	6438	+
	6439	+ /* No duplication found. Store the header option. */
	6440	+ memcpy(opend, from, new_kind_len);
	6441	+
	6442	+ bpf_sock->remaining_opt_len -= new_kind_len;
	6443	+ bpf_sock->skb_data_end += new_kind_len;
	6444	+
	6445	+ return 0;
	6446	+}
	6447	+
	6448	+static const struct bpf_func_proto bpf_sock_ops_store_hdr_opt_proto = {
	6449	+ .func = bpf_sock_ops_store_hdr_opt,
	6450	+ .gpl_only = false,
	6451	+ .ret_type = RET_INTEGER,
	6452	+ .arg1_type = ARG_PTR_TO_CTX,
	6453	+ .arg2_type = ARG_PTR_TO_MEM,
	6454	+ .arg3_type = ARG_CONST_SIZE,
	6455	+ .arg4_type = ARG_ANYTHING,
	6456	+};
	6457	+
	6458	+BPF_CALL_3(bpf_sock_ops_reserve_hdr_opt, struct bpf_sock_ops_kern *, bpf_sock,
	6459	+ u32, len, u64, flags)
	6460	+{
	6461	+ if (bpf_sock->op != BPF_SOCK_OPS_HDR_OPT_LEN_CB)
	6462	+ return -EPERM;
	6463	+
	6464	+ if (flags \|\| len < 2)
	6465	+ return -EINVAL;
	6466	+
	6467	+ if (len > bpf_sock->remaining_opt_len)
	6468	+ return -ENOSPC;
	6469	+
	6470	+ bpf_sock->remaining_opt_len -= len;
	6471	+
	6472	+ return 0;
	6473	+}
	6474	+
	6475	+static const struct bpf_func_proto bpf_sock_ops_reserve_hdr_opt_proto = {
	6476	+ .func = bpf_sock_ops_reserve_hdr_opt,
	6477	+ .gpl_only = false,
	6478	+ .ret_type = RET_INTEGER,
	6479	+ .arg1_type = ARG_PTR_TO_CTX,
	6480	+ .arg2_type = ARG_ANYTHING,
	6481	+ .arg3_type = ARG_ANYTHING,
	6482	+};
	6483	+
6153	6484	#endif /* CONFIG_INET */
6154	6485
6155	6486	bool bpf_helper_changes_pkt_data(void *func)
@@ -6178,6 +6509,9 @@ bool bpf_helper_changes_pkt_data(void *func) · 22 unmodified lines
6178	6509	func == bpf_lwt_seg6_store_bytes \|\|
6179	6510	func == bpf_lwt_seg6_adjust_srh \|\|
6180	6511	func == bpf_lwt_seg6_action \|\|
	6512	+#endif
	6513	+#ifdef CONFIG_INET
	6514	+ func == bpf_sock_ops_store_hdr_opt \|\|
6181	6515	#endif
6182	6516	func == bpf_lwt_in_push_encap \|\|
6183	6517	func == bpf_lwt_xmit_push_encap)
@@ -6550,6 +6884,12 @@ sock_ops_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 366 unmodified lines
6550	6884	case BPF_FUNC_sk_storage_delete:
6551	6885	return &bpf_sk_storage_delete_proto;
6552	6886	#ifdef CONFIG_INET
	6887	+ case BPF_FUNC_load_hdr_opt:
	6888	+ return &bpf_sock_ops_load_hdr_opt_proto;
	6889	+ case BPF_FUNC_store_hdr_opt:
	6890	+ return &bpf_sock_ops_store_hdr_opt_proto;
	6891	+ case BPF_FUNC_reserve_hdr_opt:
	6892	+ return &bpf_sock_ops_reserve_hdr_opt_proto;
6553	6893	case BPF_FUNC_tcp_sock:
6554	6894	return &bpf_tcp_sock_proto;
6555	6895	#endif /* CONFIG_INET */
@@ -7349,6 +7689,20 @@ static bool sock_ops_is_valid_access(int off, int size, · 793 unmodified lines
7349	7689	return false;
7350	7690	info->reg_type = PTR_TO_SOCKET_OR_NULL;
7351	7691	break;
	7692	+ case offsetof(struct bpf_sock_ops, skb_data):
	7693	+ if (size != sizeof(__u64))
	7694	+ return false;
	7695	+ info->reg_type = PTR_TO_PACKET;
	7696	+ break;
	7697	+ case offsetof(struct bpf_sock_ops, skb_data_end):
	7698	+ if (size != sizeof(__u64))
	7699	+ return false;
	7700	+ info->reg_type = PTR_TO_PACKET_END;
	7701	+ break;
	7702	+ case offsetof(struct bpf_sock_ops, skb_tcp_flags):
	7703	+ bpf_ctx_record_field_size(info, size_default);
	7704	+ return bpf_ctx_narrow_access_ok(off, size,
	7705	+ size_default);
7352	7706	default:
7353	7707	if (size != size_default)
7354	7708	return false;
@@ -8450,17 +8804,22 @@ static u32 sock_ops_convert_ctx_access(enum bpf_access_type type, · 1095 unmodified lines
8450	8804	return insn - insn_buf;
8451	8805
8452	8806	switch (si->off) {
8453		− case offsetof(struct bpf_sock_ops, op) ...
	8807	+ case offsetof(struct bpf_sock_ops, op):
	8808	+ *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sock_ops_kern,
	8809	+ op),
	8810	+ si->dst_reg, si->src_reg,
	8811	+ offsetof(struct bpf_sock_ops_kern, op));
	8812	+ break;
	8813	+
	8814	+ case offsetof(struct bpf_sock_ops, replylong[0]) ...
8454	8815	offsetof(struct bpf_sock_ops, replylong[3]):
8455		− BUILD_BUG_ON(sizeof_field(struct bpf_sock_ops, op) !=
8456		− sizeof_field(struct bpf_sock_ops_kern, op));
8457	8816	BUILD_BUG_ON(sizeof_field(struct bpf_sock_ops, reply) !=
8458	8817	sizeof_field(struct bpf_sock_ops_kern, reply));
8459	8818	BUILD_BUG_ON(sizeof_field(struct bpf_sock_ops, replylong) !=
8460	8819	sizeof_field(struct bpf_sock_ops_kern, replylong));
8461	8820	off = si->off;
8462		− off -= offsetof(struct bpf_sock_ops, op);
8463		− off += offsetof(struct bpf_sock_ops_kern, op);
	8821	+ off -= offsetof(struct bpf_sock_ops, replylong[0]);
	8822	+ off += offsetof(struct bpf_sock_ops_kern, replylong[0]);
8464	8823	if (type == BPF_WRITE)
8465	8824	*insn++ = BPF_STX_MEM(BPF_W, si->dst_reg, si->src_reg,
8466	8825	off);
@@ -8681,6 +9040,49 @@ static u32 sock_ops_convert_ctx_access(enum bpf_access_type type, · 214 unmodified lines
8681	9040	case offsetof(struct bpf_sock_ops, sk):
8682	9041	SOCK_OPS_GET_SK();
8683	9042	break;
	9043	+ case offsetof(struct bpf_sock_ops, skb_data_end):
	9044	+ *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sock_ops_kern,
	9045	+ skb_data_end),
	9046	+ si->dst_reg, si->src_reg,
	9047	+ offsetof(struct bpf_sock_ops_kern,
	9048	+ skb_data_end));
	9049	+ break;
	9050	+ case offsetof(struct bpf_sock_ops, skb_data):
	9051	+ *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sock_ops_kern,
	9052	+ skb),
	9053	+ si->dst_reg, si->src_reg,
	9054	+ offsetof(struct bpf_sock_ops_kern,
	9055	+ skb));
	9056	+ *insn++ = BPF_JMP_IMM(BPF_JEQ, si->dst_reg, 0, 1);
	9057	+ *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct sk_buff, data),
	9058	+ si->dst_reg, si->dst_reg,
	9059	+ offsetof(struct sk_buff, data));
	9060	+ break;
	9061	+ case offsetof(struct bpf_sock_ops, skb_len):
	9062	+ *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sock_ops_kern,
	9063	+ skb),
	9064	+ si->dst_reg, si->src_reg,
	9065	+ offsetof(struct bpf_sock_ops_kern,
	9066	+ skb));
	9067	+ *insn++ = BPF_JMP_IMM(BPF_JEQ, si->dst_reg, 0, 1);
	9068	+ *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct sk_buff, len),
	9069	+ si->dst_reg, si->dst_reg,
	9070	+ offsetof(struct sk_buff, len));
	9071	+ break;
	9072	+ case offsetof(struct bpf_sock_ops, skb_tcp_flags):
	9073	+ off = offsetof(struct sk_buff, cb);
	9074	+ off += offsetof(struct tcp_skb_cb, tcp_flags);
	9075	+ *target_size = sizeof_field(struct tcp_skb_cb, tcp_flags);
	9076	+ *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sock_ops_kern,
	9077	+ skb),
	9078	+ si->dst_reg, si->src_reg,
	9079	+ offsetof(struct bpf_sock_ops_kern,
	9080	+ skb));
	9081	+ *insn++ = BPF_JMP_IMM(BPF_JEQ, si->dst_reg, 0, 1);
	9082	+ *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct tcp_skb_cb,
	9083	+ tcp_flags),
	9084	+ si->dst_reg, si->dst_reg, off);
	9085	+ break;
8684	9086	}
8685	9087	return insn - insn_buf;
8686	9088	}

net/core/skmsg.c+28−6

@@ -494,14 +494,34 @@ end: · 493 unmodified lines
494	494
495	495	struct sk_psock sk_psock_init(struct sock sk, int node)
496	496	{
497		− struct sk_psock psock = kzalloc_node(sizeof(psock),
498		− GFP_ATOMIC \| __GFP_NOWARN,
499		− node);
500		− if (!psock)
501		− return NULL;
	497	+ struct sk_psock *psock;
	498	+ struct proto *prot;
	499	+
	500	+ write_lock_bh(&sk->sk_callback_lock);
	501	+
	502	+ if (inet_csk_has_ulp(sk)) {
	503	+ psock = ERR_PTR(-EINVAL);
	504	+ goto out;
	505	+ }
	506	+
	507	+ if (sk->sk_user_data) {
	508	+ psock = ERR_PTR(-EBUSY);
	509	+ goto out;
	510	+ }
502	511
	512	+ psock = kzalloc_node(sizeof(*psock), GFP_ATOMIC \| __GFP_NOWARN, node);
	513	+ if (!psock) {
	514	+ psock = ERR_PTR(-ENOMEM);
	515	+ goto out;
	516	+ }
	517	+
	518	+ prot = READ_ONCE(sk->sk_prot);
503	519	psock->sk = sk;
504		− psock->eval = __SK_NONE;
	520	+ psock->eval = __SK_NONE;
	521	+ psock->sk_proto = prot;
	522	+ psock->saved_unhash = prot->unhash;
	523	+ psock->saved_close = prot->close;
	524	+ psock->saved_write_space = sk->sk_write_space;
505	525
506	526	INIT_LIST_HEAD(&psock->link);
507	527	spin_lock_init(&psock->link_lock);
@@ -516,6 +536,8 @@ struct sk_psock sk_psock_init(struct sock sk, int node) · 8 unmodified lines
516	536	rcu_assign_sk_user_data_nocopy(sk, psock);
517	537	sock_hold(sk);
518	538
	539	+out:
	540	+ write_unlock_bh(&sk->sk_callback_lock);
519	541	return psock;
520	542	}
521	543	EXPORT_SYMBOL_GPL(sk_psock_init);

net/core/sock_map.c+37−54

@@ -184,8 +184,6 @@ static int sock_map_init_proto(struct sock sk, struct sk_psock psock) · 183 unmodified lines
184	184	{
185	185	struct proto *prot;
186	186
187		− sock_owned_by_me(sk);
188		−
189	187	switch (sk->sk_type) {
190	188	case SOCK_STREAM:
191	189	prot = tcp_bpf_get_proto(sk, psock);
@@ -272,8 +270,8 @@ static int sock_map_link(struct bpf_map map, struct sk_psock_progs progs, · 80 unmodified lines
272	270	}
273	271	} else {
274	272	psock = sk_psock_init(sk, map->numa_node);
275		− if (!psock) {
276		− ret = -ENOMEM;
	273	+ if (IS_ERR(psock)) {
	274	+ ret = PTR_ERR(psock);
277	275	goto out_progs;
278	276	}
279	277	}
@@ -322,8 +320,8 @@ static int sock_map_link_no_progs(struct bpf_map map, struct sock sk) · 42 unmodified lines
322	320
323	321	if (!psock) {
324	322	psock = sk_psock_init(sk, map->numa_node);
325		− if (!psock)
326		− return -ENOMEM;
	323	+ if (IS_ERR(psock))
	324	+ return PTR_ERR(psock);
327	325	}
328	326
329	327	ret = sock_map_init_proto(sk, psock);
@@ -478,8 +476,6 @@ static int sock_map_update_common(struct bpf_map *map, u32 idx, · 148 unmodified lines
478	476	return -EINVAL;
479	477	if (unlikely(idx >= map->max_entries))
480	478	return -E2BIG;
481		− if (inet_csk_has_ulp(sk))
482		− return -EINVAL;
483	479
484	480	link = sk_psock_init_link();
485	481	if (!link)
@@ -563,10 +559,12 @@ static bool sock_map_sk_state_allowed(const struct sock *sk) · 77 unmodified lines
563	559	return false;
564	560	}
565	561
566		−static int sock_map_update_elem(struct bpf_map map, void key,
567		− void *value, u64 flags)
	562	+static int sock_hash_update_common(struct bpf_map map, void key,
	563	+ struct sock *sk, u64 flags);
	564	+
	565	+int sock_map_update_elem_sys(struct bpf_map map, void key, void *value,
	566	+ u64 flags)
568	567	{
569		− u32 idx = (u32 )key;
570	568	struct socket *sock;
571	569	struct sock *sk;
572	570	int ret;
@@ -595,14 +593,38 @@ static int sock_map_update_elem(struct bpf_map map, void key, · 22 unmodified lines
595	593	sock_map_sk_acquire(sk);
596	594	if (!sock_map_sk_state_allowed(sk))
597	595	ret = -EOPNOTSUPP;
	596	+ else if (map->map_type == BPF_MAP_TYPE_SOCKMAP)
	597	+ ret = sock_map_update_common(map, (u32 )key, sk, flags);
598	598	else
599		− ret = sock_map_update_common(map, idx, sk, flags);
	599	+ ret = sock_hash_update_common(map, key, sk, flags);
600	600	sock_map_sk_release(sk);
601	601	out:
602	602	fput(sock->file);
603	603	return ret;
604	604	}
605	605
	606	+static int sock_map_update_elem(struct bpf_map map, void key,
	607	+ void *value, u64 flags)
	608	+{
	609	+ struct sock sk = (struct sock )value;
	610	+ int ret;
	611	+
	612	+ if (!sock_map_sk_is_suitable(sk))
	613	+ return -EOPNOTSUPP;
	614	+
	615	+ local_bh_disable();
	616	+ bh_lock_sock(sk);
	617	+ if (!sock_map_sk_state_allowed(sk))
	618	+ ret = -EOPNOTSUPP;
	619	+ else if (map->map_type == BPF_MAP_TYPE_SOCKMAP)
	620	+ ret = sock_map_update_common(map, (u32 )key, sk, flags);
	621	+ else
	622	+ ret = sock_hash_update_common(map, key, sk, flags);
	623	+ bh_unlock_sock(sk);
	624	+ local_bh_enable();
	625	+ return ret;
	626	+}
	627	+
606	628	BPF_CALL_4(bpf_sock_map_update, struct bpf_sock_ops_kern *, sops,
607	629	struct bpf_map , map, void , key, u64, flags)
608	630	{
@@ -683,6 +705,7 @@ const struct bpf_func_proto bpf_msg_redirect_map_proto = { · 74 unmodified lines
683	705
684	706	static int sock_map_btf_id;
685	707	const struct bpf_map_ops sock_map_ops = {
	708	+ .map_meta_equal = bpf_map_meta_equal,
686	709	.map_alloc = sock_map_alloc,
687	710	.map_free = sock_map_free,
688	711	.map_get_next_key = sock_map_get_next_key,
@@ -855,8 +878,6 @@ static int sock_hash_update_common(struct bpf_map map, void key, · 166 unmodified lines
855	878	WARN_ON_ONCE(!rcu_read_lock_held());
856	879	if (unlikely(flags > BPF_EXIST))
857	880	return -EINVAL;
858		− if (inet_csk_has_ulp(sk))
859		− return -EINVAL;
860	881
861	882	link = sk_psock_init_link();
862	883	if (!link)
@@ -915,45 +936,6 @@ out_free: · 52 unmodified lines
915	936	return ret;
916	937	}
917	938
918		−static int sock_hash_update_elem(struct bpf_map map, void key,
919		− void *value, u64 flags)
920		−{
921		− struct socket *sock;
922		− struct sock *sk;
923		− int ret;
924		− u64 ufd;
925		−
926		− if (map->value_size == sizeof(u64))
927		− ufd = (u64 )value;
928		− else
929		− ufd = (u32 )value;
930		− if (ufd > S32_MAX)
931		− return -EINVAL;
932		−
933		− sock = sockfd_lookup(ufd, &ret);
934		− if (!sock)
935		− return ret;
936		− sk = sock->sk;
937		− if (!sk) {
938		− ret = -EINVAL;
939		− goto out;
940		− }
941		− if (!sock_map_sk_is_suitable(sk)) {
942		− ret = -EOPNOTSUPP;
943		− goto out;
944		− }
945		−
946		− sock_map_sk_acquire(sk);
947		− if (!sock_map_sk_state_allowed(sk))
948		− ret = -EOPNOTSUPP;
949		− else
950		− ret = sock_hash_update_common(map, key, sk, flags);
951		− sock_map_sk_release(sk);
952		−out:
953		− fput(sock->file);
954		− return ret;
955		−}
956		−
957	939	static int sock_hash_get_next_key(struct bpf_map map, void key,
958	940	void *key_next)
959	941	{
@@ -1219,10 +1201,11 @@ const struct bpf_func_proto bpf_msg_redirect_hash_proto = { · 259 unmodified lines
1219	1201
1220	1202	static int sock_hash_map_btf_id;
1221	1203	const struct bpf_map_ops sock_hash_ops = {
	1204	+ .map_meta_equal = bpf_map_meta_equal,
1222	1205	.map_alloc = sock_hash_alloc,
1223	1206	.map_free = sock_hash_free,
1224	1207	.map_get_next_key = sock_hash_get_next_key,
1225		− .map_update_elem = sock_hash_update_elem,
	1208	+ .map_update_elem = sock_map_update_elem,
1226	1209	.map_delete_elem = sock_hash_delete_elem,
1227	1210	.map_lookup_elem = sock_hash_lookup,
1228	1211	.map_lookup_elem_sys_only = sock_hash_lookup_sys,

net/ethtool/channels.c+1−1

@@ -223,7 +223,7 @@ int ethnl_set_channels(struct sk_buff skb, struct genl_info info) · 222 unmodified lines
223	223	from_channel = channels.combined_count +
224	224	min(channels.rx_count, channels.tx_count);
225	225	for (i = from_channel; i < old_total; i++)
226		− if (xdp_get_umem_from_qid(dev, i)) {
	226	+ if (xsk_get_pool_from_qid(dev, i)) {
227	227	GENL_SET_ERR_MSG(info, "requested channel counts are too low for existing zerocopy AF_XDP sockets");
228	228	return -EINVAL;
229	229	}

net/ethtool/ioctl.c+1−1

@@ -1706,7 +1706,7 @@ static noinline_for_stack int ethtool_set_channels(struct net_device *dev, · 1705 unmodified lines
1706	1706	min(channels.rx_count, channels.tx_count);
1707	1707	to_channel = curr.combined_count + max(curr.rx_count, curr.tx_count);
1708	1708	for (i = from_channel; i < to_channel; i++)
1709		− if (xdp_get_umem_from_qid(dev, i))
	1709	+ if (xsk_get_pool_from_qid(dev, i))
1710	1710	return -EINVAL;
1711	1711
1712	1712	ret = dev->ethtool_ops->set_channels(dev, &channels);

net/ipv4/tcp.c+11−5

@@ -418,6 +418,8 @@ void tcp_init_sock(struct sock *sk) · 417 unmodified lines
418	418	INIT_LIST_HEAD(&tp->tsorted_sent_queue);
419	419
420	420	icsk->icsk_rto = TCP_TIMEOUT_INIT;
	421	+ icsk->icsk_rto_min = TCP_RTO_MIN;
	422	+ icsk->icsk_delack_max = TCP_DELACK_MAX;
421	423	tp->mdev_us = jiffies_to_usecs(TCP_TIMEOUT_INIT);
422	424	minmax_reset(&tp->rtt_min, tcp_jiffies32, ~0U);
423	425
@@ -2685,6 +2687,8 @@ int tcp_disconnect(struct sock *sk, int flags) · 2261 unmodified lines
2685	2687	icsk->icsk_backoff = 0;
2686	2688	icsk->icsk_probes_out = 0;
2687	2689	icsk->icsk_rto = TCP_TIMEOUT_INIT;
	2690	+ icsk->icsk_rto_min = TCP_RTO_MIN;
	2691	+ icsk->icsk_delack_max = TCP_DELACK_MAX;
2688	2692	tp->snd_ssthresh = TCP_INFINITE_SSTHRESH;
2689	2693	tp->snd_cwnd = TCP_INIT_CWND;
2690	2694	tp->snd_cwnd_cnt = 0;
@@ -3207,7 +3211,8 @@ static int do_tcp_setsockopt(struct sock *sk, int level, int optname, · 516 unmodified lines
3207	3211	break;
3208	3212
3209	3213	case TCP_SAVE_SYN:
3210		− if (val < 0 \|\| val > 1)
	3214	+ /* 0: disable, 1: enable, 2: start from ether_header */
	3215	+ if (val < 0 \|\| val > 2)
3211	3216	err = -EINVAL;
3212	3217	else
3213	3218	tp->save_syn = val;
@@ -3788,20 +3793,21 @@ static int do_tcp_getsockopt(struct sock *sk, int level, · 574 unmodified lines
3788	3793
3789	3794	lock_sock(sk);
3790	3795	if (tp->saved_syn) {
3791		− if (len < tp->saved_syn[0]) {
3792		− if (put_user(tp->saved_syn[0], optlen)) {
	3796	+ if (len < tcp_saved_syn_len(tp->saved_syn)) {
	3797	+ if (put_user(tcp_saved_syn_len(tp->saved_syn),
	3798	+ optlen)) {
3793	3799	release_sock(sk);
3794	3800	return -EFAULT;
3795	3801	}
3796	3802	release_sock(sk);
3797	3803	return -EINVAL;
3798	3804	}
3799		− len = tp->saved_syn[0];
	3805	+ len = tcp_saved_syn_len(tp->saved_syn);
3800	3806	if (put_user(len, optlen)) {
3801	3807	release_sock(sk);
3802	3808	return -EFAULT;
3803	3809	}
3804		− if (copy_to_user(optval, tp->saved_syn + 1, len)) {
	3810	+ if (copy_to_user(optval, tp->saved_syn->data, len)) {
3805	3811	release_sock(sk);
3806	3812	return -EFAULT;
3807	3813	}

net/ipv4/tcp_bpf.c+5−8

@@ -567,10 +567,9 @@ static void tcp_bpf_rebuild_protos(struct proto prot[TCP_BPF_NUM_CFGS], · 566 unmodified lines
567	567	prot[TCP_BPF_TX].sendpage = tcp_bpf_sendpage;
568	568	}
569	569
570		−static void tcp_bpf_check_v6_needs_rebuild(struct sock sk, struct proto ops)
	570	+static void tcp_bpf_check_v6_needs_rebuild(struct proto *ops)
571	571	{
572		− if (sk->sk_family == AF_INET6 &&
573		− unlikely(ops != smp_load_acquire(&tcpv6_prot_saved))) {
	572	+ if (unlikely(ops != smp_load_acquire(&tcpv6_prot_saved))) {
574	573	spin_lock_bh(&tcpv6_prot_lock);
575	574	if (likely(ops != tcpv6_prot_saved)) {
576	575	tcp_bpf_rebuild_protos(tcp_bpf_prots[TCP_BPF_IPV6], ops);
@@ -603,13 +602,11 @@ struct proto tcp_bpf_get_proto(struct sock sk, struct sk_psock *psock) · 26 unmodified lines
603	602	int family = sk->sk_family == AF_INET6 ? TCP_BPF_IPV6 : TCP_BPF_IPV4;
604	603	int config = psock->progs.msg_parser ? TCP_BPF_TX : TCP_BPF_BASE;
605	604
606		− if (!psock->sk_proto) {
607		− struct proto *ops = READ_ONCE(sk->sk_prot);
608		−
609		− if (tcp_bpf_assert_proto_ops(ops))
	605	+ if (sk->sk_family == AF_INET6) {
	606	+ if (tcp_bpf_assert_proto_ops(psock->sk_proto))
610	607	return ERR_PTR(-EINVAL);
611	608
612		− tcp_bpf_check_v6_needs_rebuild(sk, ops);
	609	+ tcp_bpf_check_v6_needs_rebuild(psock->sk_proto);
613	610	}
614	611
615	612	return &tcp_bpf_prots[family][config];

net/ipv4/tcp_fastopen.c+1−1

@@ -295,7 +295,7 @@ static struct sock tcp_fastopen_create_child(struct sock sk, · 294 unmodified lines
295	295	refcount_set(&req->rsk_refcnt, 2);
296	296
297	297	/* Now finish processing the fastopen child socket. */
298		− tcp_init_transfer(child, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB);
	298	+ tcp_init_transfer(child, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB, skb);
299	299
300	300	tp->rcv_nxt = TCP_SKB_CB(skb)->seq + 1;
301	301

net/ipv4/tcp_input.c+109−18

@@ -138,6 +138,69 @@ void clean_acked_data_flush(void) · 137 unmodified lines
138	138	EXPORT_SYMBOL_GPL(clean_acked_data_flush);
139	139	#endif
140	140
	141	+#ifdef CONFIG_CGROUP_BPF
	142	+static void bpf_skops_parse_hdr(struct sock sk, struct sk_buff skb)
	143	+{
	144	+ bool unknown_opt = tcp_sk(sk)->rx_opt.saw_unknown &&
	145	+ BPF_SOCK_OPS_TEST_FLAG(tcp_sk(sk),
	146	+ BPF_SOCK_OPS_PARSE_UNKNOWN_HDR_OPT_CB_FLAG);
	147	+ bool parse_all_opt = BPF_SOCK_OPS_TEST_FLAG(tcp_sk(sk),
	148	+ BPF_SOCK_OPS_PARSE_ALL_HDR_OPT_CB_FLAG);
	149	+ struct bpf_sock_ops_kern sock_ops;
	150	+
	151	+ if (likely(!unknown_opt && !parse_all_opt))
	152	+ return;
	153	+
	154	+ /* The skb will be handled in the
	155	+ * bpf_skops_established() or
	156	+ * bpf_skops_write_hdr_opt().
	157	+ */
	158	+ switch (sk->sk_state) {
	159	+ case TCP_SYN_RECV:
	160	+ case TCP_SYN_SENT:
	161	+ case TCP_LISTEN:
	162	+ return;
	163	+ }
	164	+
	165	+ sock_owned_by_me(sk);
	166	+
	167	+ memset(&sock_ops, 0, offsetof(struct bpf_sock_ops_kern, temp));
	168	+ sock_ops.op = BPF_SOCK_OPS_PARSE_HDR_OPT_CB;
	169	+ sock_ops.is_fullsock = 1;
	170	+ sock_ops.sk = sk;
	171	+ bpf_skops_init_skb(&sock_ops, skb, tcp_hdrlen(skb));
	172	+
	173	+ BPF_CGROUP_RUN_PROG_SOCK_OPS(&sock_ops);
	174	+}
	175	+
	176	+static void bpf_skops_established(struct sock *sk, int bpf_op,
	177	+ struct sk_buff *skb)
	178	+{
	179	+ struct bpf_sock_ops_kern sock_ops;
	180	+
	181	+ sock_owned_by_me(sk);
	182	+
	183	+ memset(&sock_ops, 0, offsetof(struct bpf_sock_ops_kern, temp));
	184	+ sock_ops.op = bpf_op;
	185	+ sock_ops.is_fullsock = 1;
	186	+ sock_ops.sk = sk;
	187	+ /* sk with TCP_REPAIR_ON does not have skb in tcp_finish_connect */
	188	+ if (skb)
	189	+ bpf_skops_init_skb(&sock_ops, skb, tcp_hdrlen(skb));
	190	+
	191	+ BPF_CGROUP_RUN_PROG_SOCK_OPS(&sock_ops);
	192	+}
	193	+#else
	194	+static void bpf_skops_parse_hdr(struct sock sk, struct sk_buff skb)
	195	+{
	196	+}
	197	+
	198	+static void bpf_skops_established(struct sock *sk, int bpf_op,
	199	+ struct sk_buff *skb)
	200	+{
	201	+}
	202	+#endif
	203	+
141	204	static void tcp_gro_dev_warn(struct sock sk, const struct sk_buff skb,
142	205	unsigned int len)
143	206	{
@@ -3801,7 +3864,7 @@ static void tcp_parse_fastopen_option(int len, const unsigned char *cookie, · 3657 unmodified lines
3801	3864	foc->exp = exp_opt;
3802	3865	}
3803	3866
3804		−static void smc_parse_options(const struct tcphdr *th,
	3867	+static bool smc_parse_options(const struct tcphdr *th,
3805	3868	struct tcp_options_received *opt_rx,
3806	3869	const unsigned char *ptr,
3807	3870	int opsize)
@@ -3810,10 +3873,13 @@ static void smc_parse_options(const struct tcphdr *th, · 2 unmodified lines
3810	3873	if (static_branch_unlikely(&tcp_have_smc)) {
3811	3874	if (th->syn && !(opsize & 1) &&
3812	3875	opsize >= TCPOLEN_EXP_SMC_BASE &&
3813		− get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC)
	3876	+ get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC) {
3814	3877	opt_rx->smc_ok = 1;
	3878	+ return true;
	3879	+ }
3815	3880	}
3816	3881	#endif
	3882	+ return false;
3817	3883	}
3818	3884
3819	3885	/* Try to parse the MSS option from the TCP header. Return 0 on failure, clamped
@@ -3874,6 +3940,7 @@ void tcp_parse_options(const struct net *net, · 54 unmodified lines
3874	3940
3875	3941	ptr = (const unsigned char *)(th + 1);
3876	3942	opt_rx->saw_tstamp = 0;
	3943	+ opt_rx->saw_unknown = 0;
3877	3944
3878	3945	while (length > 0) {
3879	3946	int opcode = *ptr++;
@@ -3964,15 +4031,21 @@ void tcp_parse_options(const struct net *net, · 84 unmodified lines
3964	4031	*/
3965	4032	if (opsize >= TCPOLEN_EXP_FASTOPEN_BASE &&
3966	4033	get_unaligned_be16(ptr) ==
3967		− TCPOPT_FASTOPEN_MAGIC)
	4034	+ TCPOPT_FASTOPEN_MAGIC) {
3968	4035	tcp_parse_fastopen_option(opsize -
3969	4036	TCPOLEN_EXP_FASTOPEN_BASE,
3970	4037	ptr + 2, th->syn, foc, true);
3971		− else
3972		− smc_parse_options(th, opt_rx, ptr,
3973		− opsize);
	4038	+ break;
	4039	+ }
	4040	+
	4041	+ if (smc_parse_options(th, opt_rx, ptr, opsize))
	4042	+ break;
	4043	+
	4044	+ opt_rx->saw_unknown = 1;
3974	4045	break;
3975	4046
	4047	+ default:
	4048	+ opt_rx->saw_unknown = 1;
3976	4049	}
3977	4050	ptr += opsize-2;
3978	4051	length -= opsize;
@@ -5590,6 +5663,8 @@ syn_challenge: · 1611 unmodified lines
5590	5663	goto discard;
5591	5664	}
5592	5665
	5666	+ bpf_skops_parse_hdr(sk, skb);
	5667	+
5593	5668	return true;
5594	5669
5595	5670	discard:
@@ -5798,7 +5873,7 @@ discard: · 202 unmodified lines
5798	5873	}
5799	5874	EXPORT_SYMBOL(tcp_rcv_established);
5800	5875
5801		−void tcp_init_transfer(struct sock *sk, int bpf_op)
	5876	+void tcp_init_transfer(struct sock sk, int bpf_op, struct sk_buff skb)
5802	5877	{
5803	5878	struct inet_connection_sock *icsk = inet_csk(sk);
5804	5879	struct tcp_sock *tp = tcp_sk(sk);
@@ -5819,7 +5894,7 @@ void tcp_init_transfer(struct sock *sk, int bpf_op) · 14 unmodified lines
5819	5894	tp->snd_cwnd = tcp_init_cwnd(tp, __sk_dst_get(sk));
5820	5895	tp->snd_cwnd_stamp = tcp_jiffies32;
5821	5896
5822		− tcp_call_bpf(sk, bpf_op, 0, NULL);
	5897	+ bpf_skops_established(sk, bpf_op, skb);
5823	5898	tcp_init_congestion_control(sk);
5824	5899	tcp_init_buffer_space(sk);
5825	5900	}
@@ -5838,7 +5913,7 @@ void tcp_finish_connect(struct sock sk, struct sk_buff skb) · 12 unmodified lines
5838	5913	sk_mark_napi_id(sk, skb);
5839	5914	}
5840	5915
5841		− tcp_init_transfer(sk, BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB);
	5916	+ tcp_init_transfer(sk, BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB, skb);
5842	5917
5843	5918	/* Prevent spurious tcp_cwnd_restart() on first data
5844	5919	* packet.
@@ -6310,7 +6385,8 @@ int tcp_rcv_state_process(struct sock sk, struct sk_buff skb) · 465 unmodified lines
6310	6385	} else {
6311	6386	tcp_try_undo_spurious_syn(sk);
6312	6387	tp->retrans_stamp = 0;
6313		− tcp_init_transfer(sk, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB);
	6388	+ tcp_init_transfer(sk, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB,
	6389	+ skb);
6314	6390	WRITE_ONCE(tp->copied_seq, tp->rcv_nxt);
6315	6391	}
6316	6392	smp_mb();
@@ -6599,13 +6675,27 @@ static void tcp_reqsk_record_syn(const struct sock *sk, · 282 unmodified lines
6599	6675	{
6600	6676	if (tcp_sk(sk)->save_syn) {
6601	6677	u32 len = skb_network_header_len(skb) + tcp_hdrlen(skb);
6602		− u32 *copy;
	6678	+ struct saved_syn *saved_syn;
	6679	+ u32 mac_hdrlen;
	6680	+ void *base;
	6681	+
	6682	+ if (tcp_sk(sk)->save_syn == 2) { /* Save full header. */
	6683	+ base = skb_mac_header(skb);
	6684	+ mac_hdrlen = skb_mac_header_len(skb);
	6685	+ len += mac_hdrlen;
	6686	+ } else {
	6687	+ base = skb_network_header(skb);
	6688	+ mac_hdrlen = 0;
	6689	+ }
6603	6690
6604		− copy = kmalloc(len + sizeof(u32), GFP_ATOMIC);
6605		− if (copy) {
6606		− copy[0] = len;
6607		− memcpy(&copy[1], skb_network_header(skb), len);
6608		− req->saved_syn = copy;
	6691	+ saved_syn = kmalloc(struct_size(saved_syn, data, len),
	6692	+ GFP_ATOMIC);
	6693	+ if (saved_syn) {
	6694	+ saved_syn->mac_hdrlen = mac_hdrlen;
	6695	+ saved_syn->network_hdrlen = skb_network_header_len(skb);
	6696	+ saved_syn->tcp_hdrlen = tcp_hdrlen(skb);
	6697	+ memcpy(saved_syn->data, base, len);
	6698	+ req->saved_syn = saved_syn;
6609	6699	}
6610	6700	}
6611	6701	}
@@ -6752,7 +6842,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops, · 140 unmodified lines
6752	6842	}
6753	6843	if (fastopen_sk) {
6754	6844	af_ops->send_synack(fastopen_sk, dst, &fl, req,
6755		− &foc, TCP_SYNACK_FASTOPEN);
	6845	+ &foc, TCP_SYNACK_FASTOPEN, skb);
6756	6846	/* Add the child socket directly into the accept queue */
6757	6847	if (!inet_csk_reqsk_queue_add(sk, req, fastopen_sk)) {
6758	6848	reqsk_fastopen_remove(fastopen_sk, req, false);
@@ -6770,7 +6860,8 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops, · 11 unmodified lines
6770	6860	tcp_timeout_init((struct sock *)req));
6771	6861	af_ops->send_synack(sk, dst, &fl, req, &foc,
6772	6862	!want_cookie ? TCP_SYNACK_NORMAL :
6773		− TCP_SYNACK_COOKIE);
	6863	+ TCP_SYNACK_COOKIE,
	6864	+ skb);
6774	6865	if (want_cookie) {
6775	6866	reqsk_free(req);
6776	6867	return 0;

net/ipv4/tcp_ipv4.c+3−2

@@ -965,7 +965,8 @@ static int tcp_v4_send_synack(const struct sock sk, struct dst_entry dst, · 964 unmodified lines
965	965	struct flowi *fl,
966	966	struct request_sock *req,
967	967	struct tcp_fastopen_cookie *foc,
968		− enum tcp_synack_type synack_type)
	968	+ enum tcp_synack_type synack_type,
	969	+ struct sk_buff *syn_skb)
969	970	{
970	971	const struct inet_request_sock *ireq = inet_rsk(req);
971	972	struct flowi4 fl4;
@@ -976,7 +977,7 @@ static int tcp_v4_send_synack(const struct sock sk, struct dst_entry dst, · 4 unmodified lines
976	977	if (!dst && (dst = inet_csk_route_req(sk, &fl4, req)) == NULL)
977	978	return -1;
978	979
979		− skb = tcp_make_synack(sk, dst, req, foc, synack_type);
	980	+ skb = tcp_make_synack(sk, dst, req, foc, synack_type, syn_skb);
980	981
981	982	if (skb) {
982	983	__tcp_v4_send_check(skb, ireq->ir_loc_addr, ireq->ir_rmt_addr);

net/ipv4/tcp_minisocks.c+1−0

@@ -548,6 +548,7 @@ struct sock tcp_create_openreq_child(const struct sock sk, · 547 unmodified lines
548	548	newtp->fastopen_req = NULL;
549	549	RCU_INIT_POINTER(newtp->fastopen_rsk, NULL);
550	550
	551	+ bpf_skops_init_child(sk, newsk);
551	552	tcp_bpf_clone(sk, newsk);
552	553
553	554	__TCP_INC_STATS(sock_net(sk), TCP_MIB_PASSIVEOPENS);

net/ipv4/tcp_output.c+180−13

@@ -438,6 +438,7 @@ struct tcp_out_options { · 437 unmodified lines
438	438	u8 ws; /* window scale, 0 to disable */
439	439	u8 num_sack_blocks; /* number of SACK blocks to include */
440	440	u8 hash_size; /* bytes in hash_location */
	441	+ u8 bpf_opt_len; /* length of BPF hdr option */
441	442	__u8 hash_location; / temporary pointer, overloaded */
442	443	__u32 tsval, tsecr; /* need to include OPTION_TS */
443	444	struct tcp_fastopen_cookie fastopen_cookie; / Fast open cookie */
@@ -452,6 +453,145 @@ static void mptcp_options_write(__be32 ptr, struct tcp_out_options opts) · 8 unmodified lines
452	453	#endif
453	454	}
454	455
	456	+#ifdef CONFIG_CGROUP_BPF
	457	+static int bpf_skops_write_hdr_opt_arg0(struct sk_buff *skb,
	458	+ enum tcp_synack_type synack_type)
	459	+{
	460	+ if (unlikely(!skb))
	461	+ return BPF_WRITE_HDR_TCP_CURRENT_MSS;
	462	+
	463	+ if (unlikely(synack_type == TCP_SYNACK_COOKIE))
	464	+ return BPF_WRITE_HDR_TCP_SYNACK_COOKIE;
	465	+
	466	+ return 0;
	467	+}
	468	+
	469	+/* req, syn_skb and synack_type are used when writing synack */
	470	+static void bpf_skops_hdr_opt_len(struct sock sk, struct sk_buff skb,
	471	+ struct request_sock *req,
	472	+ struct sk_buff *syn_skb,
	473	+ enum tcp_synack_type synack_type,
	474	+ struct tcp_out_options *opts,
	475	+ unsigned int *remaining)
	476	+{
	477	+ struct bpf_sock_ops_kern sock_ops;
	478	+ int err;
	479	+
	480	+ if (likely(!BPF_SOCK_OPS_TEST_FLAG(tcp_sk(sk),
	481	+ BPF_SOCK_OPS_WRITE_HDR_OPT_CB_FLAG)) \|\|
	482	+ !*remaining)
	483	+ return;
	484	+
	485	+ /* remaining has already been aligned to 4 bytes, so remaining >= 4 */
	486	+
	487	+ /* init sock_ops */
	488	+ memset(&sock_ops, 0, offsetof(struct bpf_sock_ops_kern, temp));
	489	+
	490	+ sock_ops.op = BPF_SOCK_OPS_HDR_OPT_LEN_CB;
	491	+
	492	+ if (req) {
	493	+ /* The listen "sk" cannot be passed here because
	494	+ * it is not locked. It would not make too much
	495	+ * sense to do bpf_setsockopt(listen_sk) based
	496	+ * on individual connection request also.
	497	+ *
	498	+ * Thus, "req" is passed here and the cgroup-bpf-progs
	499	+ * of the listen "sk" will be run.
	500	+ *
	501	+ * "req" is also used here for fastopen even the "sk" here is
	502	+ * a fullsock "child" sk. It is to keep the behavior
	503	+ * consistent between fastopen and non-fastopen on
	504	+ * the bpf programming side.
	505	+ */
	506	+ sock_ops.sk = (struct sock *)req;
	507	+ sock_ops.syn_skb = syn_skb;
	508	+ } else {
	509	+ sock_owned_by_me(sk);
	510	+
	511	+ sock_ops.is_fullsock = 1;
	512	+ sock_ops.sk = sk;
	513	+ }
	514	+
	515	+ sock_ops.args[0] = bpf_skops_write_hdr_opt_arg0(skb, synack_type);
	516	+ sock_ops.remaining_opt_len = *remaining;
	517	+ /* tcp_current_mss() does not pass a skb */
	518	+ if (skb)
	519	+ bpf_skops_init_skb(&sock_ops, skb, 0);
	520	+
	521	+ err = BPF_CGROUP_RUN_PROG_SOCK_OPS_SK(&sock_ops, sk);
	522	+
	523	+ if (err \|\| sock_ops.remaining_opt_len == *remaining)
	524	+ return;
	525	+
	526	+ opts->bpf_opt_len = *remaining - sock_ops.remaining_opt_len;
	527	+ /* round up to 4 bytes */
	528	+ opts->bpf_opt_len = (opts->bpf_opt_len + 3) & ~3;
	529	+
	530	+ *remaining -= opts->bpf_opt_len;
	531	+}
	532	+
	533	+static void bpf_skops_write_hdr_opt(struct sock sk, struct sk_buff skb,
	534	+ struct request_sock *req,
	535	+ struct sk_buff *syn_skb,
	536	+ enum tcp_synack_type synack_type,
	537	+ struct tcp_out_options *opts)
	538	+{
	539	+ u8 first_opt_off, nr_written, max_opt_len = opts->bpf_opt_len;
	540	+ struct bpf_sock_ops_kern sock_ops;
	541	+ int err;
	542	+
	543	+ if (likely(!max_opt_len))
	544	+ return;
	545	+
	546	+ memset(&sock_ops, 0, offsetof(struct bpf_sock_ops_kern, temp));
	547	+
	548	+ sock_ops.op = BPF_SOCK_OPS_WRITE_HDR_OPT_CB;
	549	+
	550	+ if (req) {
	551	+ sock_ops.sk = (struct sock *)req;
	552	+ sock_ops.syn_skb = syn_skb;
	553	+ } else {
	554	+ sock_owned_by_me(sk);
	555	+
	556	+ sock_ops.is_fullsock = 1;
	557	+ sock_ops.sk = sk;
	558	+ }
	559	+
	560	+ sock_ops.args[0] = bpf_skops_write_hdr_opt_arg0(skb, synack_type);
	561	+ sock_ops.remaining_opt_len = max_opt_len;
	562	+ first_opt_off = tcp_hdrlen(skb) - max_opt_len;
	563	+ bpf_skops_init_skb(&sock_ops, skb, first_opt_off);
	564	+
	565	+ err = BPF_CGROUP_RUN_PROG_SOCK_OPS_SK(&sock_ops, sk);
	566	+
	567	+ if (err)
	568	+ nr_written = 0;
	569	+ else
	570	+ nr_written = max_opt_len - sock_ops.remaining_opt_len;
	571	+
	572	+ if (nr_written < max_opt_len)
	573	+ memset(skb->data + first_opt_off + nr_written, TCPOPT_NOP,
	574	+ max_opt_len - nr_written);
	575	+}
	576	+#else
	577	+static void bpf_skops_hdr_opt_len(struct sock sk, struct sk_buff skb,
	578	+ struct request_sock *req,
	579	+ struct sk_buff *syn_skb,
	580	+ enum tcp_synack_type synack_type,
	581	+ struct tcp_out_options *opts,
	582	+ unsigned int *remaining)
	583	+{
	584	+}
	585	+
	586	+static void bpf_skops_write_hdr_opt(struct sock sk, struct sk_buff skb,
	587	+ struct request_sock *req,
	588	+ struct sk_buff *syn_skb,
	589	+ enum tcp_synack_type synack_type,
	590	+ struct tcp_out_options *opts)
	591	+{
	592	+}
	593	+#endif
	594	+
455	595	/* Write previously computed TCP options to the packet.
456	596	*
457	597	* Beware: Something in the Internet is very sensitive to the ordering of
@@ -691,6 +831,8 @@ static unsigned int tcp_syn_options(struct sock sk, struct sk_buff skb, · 233 unmodified lines
691	831	}
692	832	}
693	833
	834	+ bpf_skops_hdr_opt_len(sk, skb, NULL, NULL, 0, opts, &remaining);
	835	+
694	836	return MAX_TCP_OPTION_SPACE - remaining;
695	837	}
696	838
@@ -701,7 +843,8 @@ static unsigned int tcp_synack_options(const struct sock *sk, · 4 unmodified lines
701	843	struct tcp_out_options *opts,
702	844	const struct tcp_md5sig_key *md5,
703	845	struct tcp_fastopen_cookie *foc,
704		− enum tcp_synack_type synack_type)
	846	+ enum tcp_synack_type synack_type,
	847	+ struct sk_buff *syn_skb)
705	848	{
706	849	struct inet_request_sock *ireq = inet_rsk(req);
707	850	unsigned int remaining = MAX_TCP_OPTION_SPACE;
@@ -758,6 +901,9 @@ static unsigned int tcp_synack_options(const struct sock *sk, · 50 unmodified lines
758	901
759	902	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
760	903
	904	+ bpf_skops_hdr_opt_len((struct sock *)sk, skb, req, syn_skb,
	905	+ synack_type, opts, &remaining);
	906	+
761	907	return MAX_TCP_OPTION_SPACE - remaining;
762	908	}
763	909
@@ -826,6 +972,15 @@ static unsigned int tcp_established_options(struct sock sk, struct sk_buff skb · 62 unmodified lines
826	972	opts->num_sack_blocks * TCPOLEN_SACK_PERBLOCK;
827	973	}
828	974
	975	+ if (unlikely(BPF_SOCK_OPS_TEST_FLAG(tp,
	976	+ BPF_SOCK_OPS_WRITE_HDR_OPT_CB_FLAG))) {
	977	+ unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
	978	+
	979	+ bpf_skops_hdr_opt_len(sk, skb, NULL, NULL, 0, opts, &remaining);
	980	+
	981	+ size = MAX_TCP_OPTION_SPACE - remaining;
	982	+ }
	983	+
829	984	return size;
830	985	}
831	986
@@ -1213,6 +1368,9 @@ static int __tcp_transmit_skb(struct sock sk, struct sk_buff skb, · 381 unmodified lines
1213	1368	}
1214	1369	#endif
1215	1370
	1371	+ /* BPF prog is the last one writing header option */
	1372	+ bpf_skops_write_hdr_opt(sk, skb, NULL, NULL, 0, &opts);
	1373	+
1216	1374	INDIRECT_CALL_INET(icsk->icsk_af_ops->send_check,
1217	1375	tcp_v6_send_check, tcp_v4_send_check,
1218	1376	sk, skb);
@@ -3336,20 +3494,20 @@ int tcp_send_synack(struct sock *sk) · 2117 unmodified lines
3336	3494	}
3337	3495
3338	3496	/**
3339		− * tcp_make_synack - Prepare a SYN-ACK.
3340		− * sk: listener socket
3341		− * dst: dst entry attached to the SYNACK
3342		− * req: request_sock pointer
3343		− * foc: cookie for tcp fast open
3344		− * synack_type: Type of synback to prepare
3345		− *
3346		− * Allocate one skb and build a SYNACK packet.
3347		− * @dst is consumed : Caller should not use it again.
	3497	+ * tcp_make_synack - Allocate one skb and build a SYNACK packet.
	3498	+ * @sk: listener socket
	3499	+ * @dst: dst entry attached to the SYNACK. It is consumed and caller
	3500	+ * should not use it again.
	3501	+ * @req: request_sock pointer
	3502	+ * @foc: cookie for tcp fast open
	3503	+ * @synack_type: Type of synack to prepare
	3504	+ * @syn_skb: SYN packet just received. It could be NULL for rtx case.
3348	3505	*/
3349	3506	struct sk_buff tcp_make_synack(const struct sock sk, struct dst_entry *dst,
3350	3507	struct request_sock *req,
3351	3508	struct tcp_fastopen_cookie *foc,
3352		− enum tcp_synack_type synack_type)
	3509	+ enum tcp_synack_type synack_type,
	3510	+ struct sk_buff *syn_skb)
3353	3511	{
3354	3512	struct inet_request_sock *ireq = inet_rsk(req);
3355	3513	const struct tcp_sock *tp = tcp_sk(sk);
@@ -3408,8 +3566,11 @@ struct sk_buff tcp_make_synack(const struct sock sk, struct dst_entry *dst, · 52 unmodified lines
3408	3566	md5 = tcp_rsk(req)->af_specific->req_md5_lookup(sk, req_to_sk(req));
3409	3567	#endif
3410	3568	skb_set_hash(skb, tcp_rsk(req)->txhash, PKT_HASH_TYPE_L4);
	3569	+ /* bpf program will be interested in the tcp_flags */
	3570	+ TCP_SKB_CB(skb)->tcp_flags = TCPHDR_SYN \| TCPHDR_ACK;
3411	3571	tcp_header_size = tcp_synack_options(sk, req, mss, skb, &opts, md5,
3412		− foc, synack_type) + sizeof(*th);
	3572	+ foc, synack_type,
	3573	+ syn_skb) + sizeof(*th);
3413	3574
3414	3575	skb_push(skb, tcp_header_size);
3415	3576	skb_reset_transport_header(skb);
@@ -3441,6 +3602,9 @@ struct sk_buff tcp_make_synack(const struct sock sk, struct dst_entry *dst, · 25 unmodified lines
3441	3602	rcu_read_unlock();
3442	3603	#endif
3443	3604
	3605	+ bpf_skops_write_hdr_opt((struct sock *)sk, skb, req, syn_skb,
	3606	+ synack_type, &opts);
	3607	+
3444	3608	skb->skb_mstamp_ns = now;
3445	3609	tcp_add_tx_delay(skb, tp);
3446	3610
@@ -3741,6 +3905,8 @@ void tcp_send_delayed_ack(struct sock *sk) · 294 unmodified lines
3741	3905	ato = min(ato, max_ato);
3742	3906	}
3743	3907
	3908	+ ato = min_t(u32, ato, inet_csk(sk)->icsk_delack_max);
	3909	+
3744	3910	/* Stay within the limit we were given */
3745	3911	timeout = jiffies + ato;
3746	3912
@@ -3934,7 +4100,8 @@ int tcp_rtx_synack(const struct sock sk, struct request_sock req) · 187 unmodified lines
3934	4100	int res;
3935	4101
3936	4102	tcp_rsk(req)->txhash = net_tx_rndhash();
3937		− res = af_ops->send_synack(sk, NULL, &fl, req, NULL, TCP_SYNACK_NORMAL);
	4103	+ res = af_ops->send_synack(sk, NULL, &fl, req, NULL, TCP_SYNACK_NORMAL,
	4104	+ NULL);
3938	4105	if (!res) {
3939	4106	__TCP_INC_STATS(sock_net(sk), TCP_MIB_RETRANSSEGS);
3940	4107	__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPSYNRETRANS);

net/ipv4/udp_bpf.c+4−5

@@ -22,10 +22,9 @@ static void udp_bpf_rebuild_protos(struct proto prot, const struct proto base) · 21 unmodified lines
22	22	prot->close = sock_map_close;
23	23	}
24	24
25		−static void udp_bpf_check_v6_needs_rebuild(struct sock sk, struct proto ops)
	25	+static void udp_bpf_check_v6_needs_rebuild(struct proto *ops)
26	26	{
27		− if (sk->sk_family == AF_INET6 &&
28		− unlikely(ops != smp_load_acquire(&udpv6_prot_saved))) {
	27	+ if (unlikely(ops != smp_load_acquire(&udpv6_prot_saved))) {
29	28	spin_lock_bh(&udpv6_prot_lock);
30	29	if (likely(ops != udpv6_prot_saved)) {
31	30	udp_bpf_rebuild_protos(&udp_bpf_prots[UDP_BPF_IPV6], ops);
@@ -46,8 +45,8 @@ struct proto udp_bpf_get_proto(struct sock sk, struct sk_psock *psock) · 14 unmodified lines
46	45	{
47	46	int family = sk->sk_family == AF_INET ? UDP_BPF_IPV4 : UDP_BPF_IPV6;
48	47
49		− if (!psock->sk_proto)
50		− udp_bpf_check_v6_needs_rebuild(sk, READ_ONCE(sk->sk_prot));
	48	+ if (sk->sk_family == AF_INET6)
	49	+ udp_bpf_check_v6_needs_rebuild(psock->sk_proto);
51	50
52	51	return &udp_bpf_prots[family];
53	52	}

net/ipv6/tcp_ipv6.c+3−2

@@ -501,7 +501,8 @@ static int tcp_v6_send_synack(const struct sock sk, struct dst_entry dst, · 500 unmodified lines
501	501	struct flowi *fl,
502	502	struct request_sock *req,
503	503	struct tcp_fastopen_cookie *foc,
504		− enum tcp_synack_type synack_type)
	504	+ enum tcp_synack_type synack_type,
	505	+ struct sk_buff *syn_skb)
505	506	{
506	507	struct inet_request_sock *ireq = inet_rsk(req);
507	508	struct ipv6_pinfo *np = tcp_inet6_sk(sk);
@@ -515,7 +516,7 @@ static int tcp_v6_send_synack(const struct sock sk, struct dst_entry dst, · 7 unmodified lines
515	516	IPPROTO_TCP)) == NULL)
516	517	goto done;
517	518
518		− skb = tcp_make_synack(sk, dst, req, foc, synack_type);
	519	+ skb = tcp_make_synack(sk, dst, req, foc, synack_type, syn_skb);
519	520
520	521	if (skb) {
521	522	__tcp_v6_send_check(skb, &ireq->ir_v6_loc_addr,

net/xdp/xdp_umem.c+27−198

@@ -23,162 +23,6 @@ · 22 unmodified lines
23	23
24	24	static DEFINE_IDA(umem_ida);
25	25
26		−void xdp_add_sk_umem(struct xdp_umem umem, struct xdp_sock xs)
27		−{
28		− unsigned long flags;
29		−
30		− if (!xs->tx)
31		− return;
32		−
33		− spin_lock_irqsave(&umem->xsk_tx_list_lock, flags);
34		− list_add_rcu(&xs->list, &umem->xsk_tx_list);
35		− spin_unlock_irqrestore(&umem->xsk_tx_list_lock, flags);
36		−}
37		−
38		−void xdp_del_sk_umem(struct xdp_umem umem, struct xdp_sock xs)
39		−{
40		− unsigned long flags;
41		−
42		− if (!xs->tx)
43		− return;
44		−
45		− spin_lock_irqsave(&umem->xsk_tx_list_lock, flags);
46		− list_del_rcu(&xs->list);
47		− spin_unlock_irqrestore(&umem->xsk_tx_list_lock, flags);
48		−}
49		−
50		−/* The umem is stored both in the _rx struct and the _tx struct as we do
51		− * not know if the device has more tx queues than rx, or the opposite.
52		− * This might also change during run time.
53		− */
54		−static int xdp_reg_umem_at_qid(struct net_device dev, struct xdp_umem umem,
55		− u16 queue_id)
56		−{
57		− if (queue_id >= max_t(unsigned int,
58		− dev->real_num_rx_queues,
59		− dev->real_num_tx_queues))
60		− return -EINVAL;
61		−
62		− if (queue_id < dev->real_num_rx_queues)
63		− dev->_rx[queue_id].umem = umem;
64		− if (queue_id < dev->real_num_tx_queues)
65		− dev->_tx[queue_id].umem = umem;
66		−
67		− return 0;
68		−}
69		−
70		−struct xdp_umem xdp_get_umem_from_qid(struct net_device dev,
71		− u16 queue_id)
72		−{
73		− if (queue_id < dev->real_num_rx_queues)
74		− return dev->_rx[queue_id].umem;
75		− if (queue_id < dev->real_num_tx_queues)
76		− return dev->_tx[queue_id].umem;
77		−
78		− return NULL;
79		−}
80		−EXPORT_SYMBOL(xdp_get_umem_from_qid);
81		−
82		−static void xdp_clear_umem_at_qid(struct net_device *dev, u16 queue_id)
83		−{
84		− if (queue_id < dev->real_num_rx_queues)
85		− dev->_rx[queue_id].umem = NULL;
86		− if (queue_id < dev->real_num_tx_queues)
87		− dev->_tx[queue_id].umem = NULL;
88		−}
89		−
90		−int xdp_umem_assign_dev(struct xdp_umem umem, struct net_device dev,
91		− u16 queue_id, u16 flags)
92		−{
93		− bool force_zc, force_copy;
94		− struct netdev_bpf bpf;
95		− int err = 0;
96		−
97		− ASSERT_RTNL();
98		−
99		− force_zc = flags & XDP_ZEROCOPY;
100		− force_copy = flags & XDP_COPY;
101		−
102		− if (force_zc && force_copy)
103		− return -EINVAL;
104		−
105		− if (xdp_get_umem_from_qid(dev, queue_id))
106		− return -EBUSY;
107		−
108		− err = xdp_reg_umem_at_qid(dev, umem, queue_id);
109		− if (err)
110		− return err;
111		−
112		− umem->dev = dev;
113		− umem->queue_id = queue_id;
114		−
115		− if (flags & XDP_USE_NEED_WAKEUP) {
116		− umem->flags \|= XDP_UMEM_USES_NEED_WAKEUP;
117		− /* Tx needs to be explicitly woken up the first time.
118		− * Also for supporting drivers that do not implement this
119		− * feature. They will always have to call sendto().
120		− */
121		− xsk_set_tx_need_wakeup(umem);
122		− }
123		−
124		− dev_hold(dev);
125		−
126		− if (force_copy)
127		− /* For copy-mode, we are done. */
128		− return 0;
129		−
130		− if (!dev->netdev_ops->ndo_bpf \|\| !dev->netdev_ops->ndo_xsk_wakeup) {
131		− err = -EOPNOTSUPP;
132		− goto err_unreg_umem;
133		− }
134		−
135		− bpf.command = XDP_SETUP_XSK_UMEM;
136		− bpf.xsk.umem = umem;
137		− bpf.xsk.queue_id = queue_id;
138		−
139		− err = dev->netdev_ops->ndo_bpf(dev, &bpf);
140		− if (err)
141		− goto err_unreg_umem;
142		−
143		− umem->zc = true;
144		− return 0;
145		−
146		−err_unreg_umem:
147		− if (!force_zc)
148		− err = 0; /* fallback to copy mode */
149		− if (err)
150		− xdp_clear_umem_at_qid(dev, queue_id);
151		− return err;
152		−}
153		−
154		−void xdp_umem_clear_dev(struct xdp_umem *umem)
155		−{
156		− struct netdev_bpf bpf;
157		− int err;
158		−
159		− ASSERT_RTNL();
160		−
161		− if (!umem->dev)
162		− return;
163		−
164		− if (umem->zc) {
165		− bpf.command = XDP_SETUP_XSK_UMEM;
166		− bpf.xsk.umem = NULL;
167		− bpf.xsk.queue_id = umem->queue_id;
168		−
169		− err = umem->dev->netdev_ops->ndo_bpf(umem->dev, &bpf);
170		−
171		− if (err)
172		− WARN(1, "failed to disable umem!\n");
173		− }
174		−
175		− xdp_clear_umem_at_qid(umem->dev, umem->queue_id);
176		−
177		− dev_put(umem->dev);
178		− umem->dev = NULL;
179		− umem->zc = false;
180		−}
181		−
182	26	static void xdp_umem_unpin_pages(struct xdp_umem *umem)
183	27	{
184	28	unpin_user_pages_dirty_lock(umem->pgs, umem->npgs, true);
@@ -195,38 +39,33 @@ static void xdp_umem_unaccount_pages(struct xdp_umem *umem) · 10 unmodified lines
195	39	}
196	40	}
197	41
198		−static void xdp_umem_release(struct xdp_umem *umem)
	42	+static void xdp_umem_addr_unmap(struct xdp_umem *umem)
199	43	{
200		− rtnl_lock();
201		− xdp_umem_clear_dev(umem);
202		− rtnl_unlock();
203		−
204		− ida_simple_remove(&umem_ida, umem->id);
	44	+ vunmap(umem->addrs);
	45	+ umem->addrs = NULL;
	46	+}
205	47
206		− if (umem->fq) {
207		− xskq_destroy(umem->fq);
208		− umem->fq = NULL;
209		− }
	48	+static int xdp_umem_addr_map(struct xdp_umem umem, struct page *pages,
	49	+ u32 nr_pages)
	50	+{
	51	+ umem->addrs = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL);
	52	+ if (!umem->addrs)
	53	+ return -ENOMEM;
	54	+ return 0;
	55	+}
210	56
211		− if (umem->cq) {
212		− xskq_destroy(umem->cq);
213		− umem->cq = NULL;
214		− }
	57	+static void xdp_umem_release(struct xdp_umem *umem)
	58	+{
	59	+ umem->zc = false;
	60	+ ida_simple_remove(&umem_ida, umem->id);
215	61
216		− xp_destroy(umem->pool);
	62	+ xdp_umem_addr_unmap(umem);
217	63	xdp_umem_unpin_pages(umem);
218	64
219	65	xdp_umem_unaccount_pages(umem);
220	66	kfree(umem);
221	67	}
222	68
223		−static void xdp_umem_release_deferred(struct work_struct *work)
224		−{
225		− struct xdp_umem *umem = container_of(work, struct xdp_umem, work);
226		−
227		− xdp_umem_release(umem);
228		−}
229		−
230	69	void xdp_get_umem(struct xdp_umem *umem)
231	70	{
232	71	refcount_inc(&umem->users);
@@ -237,10 +76,8 @@ void xdp_put_umem(struct xdp_umem *umem) · 4 unmodified lines
237	76	if (!umem)
238	77	return;
239	78
240		− if (refcount_dec_and_test(&umem->users)) {
241		− INIT_WORK(&umem->work, xdp_umem_release_deferred);
242		− schedule_work(&umem->work);
243		− }
	79	+ if (refcount_dec_and_test(&umem->users))
	80	+ xdp_umem_release(umem);
244	81	}
245	82
246	83	static int xdp_umem_pin_pages(struct xdp_umem *umem, unsigned long address)
@@ -319,8 +156,7 @@ static int xdp_umem_reg(struct xdp_umem umem, struct xdp_umem_reg mr) · 72 unmodified lines
319	156	return -EINVAL;
320	157	}
321	158
322		− if (mr->flags & ~(XDP_UMEM_UNALIGNED_CHUNK_FLAG \|
323		− XDP_UMEM_USES_NEED_WAKEUP))
	159	+ if (mr->flags & ~XDP_UMEM_UNALIGNED_CHUNK_FLAG)
324	160	return -EINVAL;
325	161
326	162	if (!unaligned_chunks && !is_power_of_2(chunk_size))
@@ -356,13 +192,13 @@ static int xdp_umem_reg(struct xdp_umem umem, struct xdp_umem_reg mr) · 29 unmodified lines
356	192	umem->size = size;
357	193	umem->headroom = headroom;
358	194	umem->chunk_size = chunk_size;
	195	+ umem->chunks = chunks;
359	196	umem->npgs = (u32)npgs;
360	197	umem->pgs = NULL;
361	198	umem->user = NULL;
362	199	umem->flags = mr->flags;
363		− INIT_LIST_HEAD(&umem->xsk_tx_list);
364		− spin_lock_init(&umem->xsk_tx_list_lock);
365	200
	201	+ INIT_LIST_HEAD(&umem->xsk_dma_list);
366	202	refcount_set(&umem->users, 1);
367	203
368	204	err = xdp_umem_account_pages(umem);
@@ -373,15 +209,13 @@ static int xdp_umem_reg(struct xdp_umem umem, struct xdp_umem_reg mr) · 4 unmodified lines
373	209	if (err)
374	210	goto out_account;
375	211
376		− umem->pool = xp_create(umem->pgs, umem->npgs, chunks, chunk_size,
377		− headroom, size, unaligned_chunks);
378		− if (!umem->pool) {
379		− err = -ENOMEM;
380		− goto out_pin;
381		− }
	212	+ err = xdp_umem_addr_map(umem, umem->pgs, umem->npgs);
	213	+ if (err)
	214	+ goto out_unpin;
	215	+
382	216	return 0;
383	217
384		−out_pin:
	218	+out_unpin:
385	219	xdp_umem_unpin_pages(umem);
386	220	out_account:
387	221	xdp_umem_unaccount_pages(umem);
@@ -413,8 +247,3 @@ struct xdp_umem xdp_umem_create(struct xdp_umem_reg mr) · 25 unmodified lines
413	247
414	248	return umem;
415	249	}
416		−
417		−bool xdp_umem_validate_queues(struct xdp_umem *umem)
418		−{
419		− return umem->fq && umem->cq;
420		−}

net/xdp/xdp_umem.h+0−6

@@ -8,14 +8,8 @@ · 7 unmodified lines
8	8
9	9	#include <net/xdp_sock_drv.h>
10	10
11		−int xdp_umem_assign_dev(struct xdp_umem umem, struct net_device dev,
12		− u16 queue_id, u16 flags);
13		−void xdp_umem_clear_dev(struct xdp_umem *umem);
14		−bool xdp_umem_validate_queues(struct xdp_umem *umem);
15	11	void xdp_get_umem(struct xdp_umem *umem);
16	12	void xdp_put_umem(struct xdp_umem *umem);
17		−void xdp_add_sk_umem(struct xdp_umem umem, struct xdp_sock xs);
18		−void xdp_del_sk_umem(struct xdp_umem umem, struct xdp_sock xs);
19	13	struct xdp_umem xdp_umem_create(struct xdp_umem_reg mr);
20	14
21	15	#endif /* XDP_UMEM_H_ */

net/xdp/xsk.c+142−71

@@ -36,68 +36,108 @@ static DEFINE_PER_CPU(struct list_head, xskmap_flush_list); · 35 unmodified lines
36	36	bool xsk_is_setup_for_bpf_map(struct xdp_sock *xs)
37	37	{
38	38	return READ_ONCE(xs->rx) && READ_ONCE(xs->umem) &&
39		− READ_ONCE(xs->umem->fq);
	39	+ (xs->pool->fq \|\| READ_ONCE(xs->fq_tmp));
40	40	}
41	41
42		−void xsk_set_rx_need_wakeup(struct xdp_umem *umem)
	42	+void xsk_set_rx_need_wakeup(struct xsk_buff_pool *pool)
43	43	{
44		− if (umem->need_wakeup & XDP_WAKEUP_RX)
	44	+ if (pool->cached_need_wakeup & XDP_WAKEUP_RX)
45	45	return;
46	46
47		− umem->fq->ring->flags \|= XDP_RING_NEED_WAKEUP;
48		− umem->need_wakeup \|= XDP_WAKEUP_RX;
	47	+ pool->fq->ring->flags \|= XDP_RING_NEED_WAKEUP;
	48	+ pool->cached_need_wakeup \|= XDP_WAKEUP_RX;
49	49	}
50	50	EXPORT_SYMBOL(xsk_set_rx_need_wakeup);
51	51
52		−void xsk_set_tx_need_wakeup(struct xdp_umem *umem)
	52	+void xsk_set_tx_need_wakeup(struct xsk_buff_pool *pool)
53	53	{
54	54	struct xdp_sock *xs;
55	55
56		− if (umem->need_wakeup & XDP_WAKEUP_TX)
	56	+ if (pool->cached_need_wakeup & XDP_WAKEUP_TX)
57	57	return;
58	58
59	59	rcu_read_lock();
60		− list_for_each_entry_rcu(xs, &umem->xsk_tx_list, list) {
	60	+ list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) {
61	61	xs->tx->ring->flags \|= XDP_RING_NEED_WAKEUP;
62	62	}
63	63	rcu_read_unlock();
64	64
65		− umem->need_wakeup \|= XDP_WAKEUP_TX;
	65	+ pool->cached_need_wakeup \|= XDP_WAKEUP_TX;
66	66	}
67	67	EXPORT_SYMBOL(xsk_set_tx_need_wakeup);
68	68
69		−void xsk_clear_rx_need_wakeup(struct xdp_umem *umem)
	69	+void xsk_clear_rx_need_wakeup(struct xsk_buff_pool *pool)
70	70	{
71		− if (!(umem->need_wakeup & XDP_WAKEUP_RX))
	71	+ if (!(pool->cached_need_wakeup & XDP_WAKEUP_RX))
72	72	return;
73	73
74		− umem->fq->ring->flags &= ~XDP_RING_NEED_WAKEUP;
75		− umem->need_wakeup &= ~XDP_WAKEUP_RX;
	74	+ pool->fq->ring->flags &= ~XDP_RING_NEED_WAKEUP;
	75	+ pool->cached_need_wakeup &= ~XDP_WAKEUP_RX;
76	76	}
77	77	EXPORT_SYMBOL(xsk_clear_rx_need_wakeup);
78	78
79		−void xsk_clear_tx_need_wakeup(struct xdp_umem *umem)
	79	+void xsk_clear_tx_need_wakeup(struct xsk_buff_pool *pool)
80	80	{
81	81	struct xdp_sock *xs;
82	82
83		− if (!(umem->need_wakeup & XDP_WAKEUP_TX))
	83	+ if (!(pool->cached_need_wakeup & XDP_WAKEUP_TX))
84	84	return;
85	85
86	86	rcu_read_lock();
87		− list_for_each_entry_rcu(xs, &umem->xsk_tx_list, list) {
	87	+ list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) {
88	88	xs->tx->ring->flags &= ~XDP_RING_NEED_WAKEUP;
89	89	}
90	90	rcu_read_unlock();
91	91
92		− umem->need_wakeup &= ~XDP_WAKEUP_TX;
	92	+ pool->cached_need_wakeup &= ~XDP_WAKEUP_TX;
93	93	}
94	94	EXPORT_SYMBOL(xsk_clear_tx_need_wakeup);
95	95
96		−bool xsk_umem_uses_need_wakeup(struct xdp_umem *umem)
	96	+bool xsk_uses_need_wakeup(struct xsk_buff_pool *pool)
97	97	{
98		− return umem->flags & XDP_UMEM_USES_NEED_WAKEUP;
	98	+ return pool->uses_need_wakeup;
	99	+}
	100	+EXPORT_SYMBOL(xsk_uses_need_wakeup);
	101	+
	102	+struct xsk_buff_pool xsk_get_pool_from_qid(struct net_device dev,
	103	+ u16 queue_id)
	104	+{
	105	+ if (queue_id < dev->real_num_rx_queues)
	106	+ return dev->_rx[queue_id].pool;
	107	+ if (queue_id < dev->real_num_tx_queues)
	108	+ return dev->_tx[queue_id].pool;
	109	+
	110	+ return NULL;
	111	+}
	112	+EXPORT_SYMBOL(xsk_get_pool_from_qid);
	113	+
	114	+void xsk_clear_pool_at_qid(struct net_device *dev, u16 queue_id)
	115	+{
	116	+ if (queue_id < dev->real_num_rx_queues)
	117	+ dev->_rx[queue_id].pool = NULL;
	118	+ if (queue_id < dev->real_num_tx_queues)
	119	+ dev->_tx[queue_id].pool = NULL;
	120	+}
	121	+
	122	+/* The buffer pool is stored both in the _rx struct and the _tx struct as we do
	123	+ * not know if the device has more tx queues than rx, or the opposite.
	124	+ * This might also change during run time.
	125	+ */
	126	+int xsk_reg_pool_at_qid(struct net_device dev, struct xsk_buff_pool pool,
	127	+ u16 queue_id)
	128	+{
	129	+ if (queue_id >= max_t(unsigned int,
	130	+ dev->real_num_rx_queues,
	131	+ dev->real_num_tx_queues))
	132	+ return -EINVAL;
	133	+
	134	+ if (queue_id < dev->real_num_rx_queues)
	135	+ dev->_rx[queue_id].pool = pool;
	136	+ if (queue_id < dev->real_num_tx_queues)
	137	+ dev->_tx[queue_id].pool = pool;
	138	+
	139	+ return 0;
99	140	}
100		−EXPORT_SYMBOL(xsk_umem_uses_need_wakeup);
101	141
102	142	void xp_release(struct xdp_buff_xsk *xskb)
103	143	{
@@ -155,12 +195,12 @@ static int __xsk_rcv(struct xdp_sock xs, struct xdp_buff xdp, u32 len, · 51 unmodified lines
155	195	struct xdp_buff *xsk_xdp;
156	196	int err;
157	197
158		− if (len > xsk_umem_get_rx_frame_size(xs->umem)) {
	198	+ if (len > xsk_pool_get_rx_frame_size(xs->pool)) {
159	199	xs->rx_dropped++;
160	200	return -ENOSPC;
161	201	}
162	202
163		− xsk_xdp = xsk_buff_alloc(xs->umem);
	203	+ xsk_xdp = xsk_buff_alloc(xs->pool);
164	204	if (!xsk_xdp) {
165	205	xs->rx_dropped++;
166	206	return -ENOSPC;
@@ -208,7 +248,7 @@ static int xsk_rcv(struct xdp_sock xs, struct xdp_buff xdp, · 41 unmodified lines
208	248	static void xsk_flush(struct xdp_sock *xs)
209	249	{
210	250	xskq_prod_submit(xs->rx);
211		− __xskq_cons_release(xs->umem->fq);
	251	+ __xskq_cons_release(xs->pool->fq);
212	252	sock_def_readable(&xs->sk);
213	253	}
214	254
@@ -249,32 +289,32 @@ void __xsk_map_flush(void) · 34 unmodified lines
249	289	}
250	290	}
251	291
252		−void xsk_umem_complete_tx(struct xdp_umem *umem, u32 nb_entries)
	292	+void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries)
253	293	{
254		− xskq_prod_submit_n(umem->cq, nb_entries);
	294	+ xskq_prod_submit_n(pool->cq, nb_entries);
255	295	}
256		−EXPORT_SYMBOL(xsk_umem_complete_tx);
	296	+EXPORT_SYMBOL(xsk_tx_completed);
257	297
258		−void xsk_umem_consume_tx_done(struct xdp_umem *umem)
	298	+void xsk_tx_release(struct xsk_buff_pool *pool)
259	299	{
260	300	struct xdp_sock *xs;
261	301
262	302	rcu_read_lock();
263		− list_for_each_entry_rcu(xs, &umem->xsk_tx_list, list) {
	303	+ list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) {
264	304	__xskq_cons_release(xs->tx);
265	305	xs->sk.sk_write_space(&xs->sk);
266	306	}
267	307	rcu_read_unlock();
268	308	}
269		−EXPORT_SYMBOL(xsk_umem_consume_tx_done);
	309	+EXPORT_SYMBOL(xsk_tx_release);
270	310
271		−bool xsk_umem_consume_tx(struct xdp_umem umem, struct xdp_desc desc)
	311	+bool xsk_tx_peek_desc(struct xsk_buff_pool pool, struct xdp_desc desc)
272	312	{
273	313	struct xdp_sock *xs;
274	314
275	315	rcu_read_lock();
276		− list_for_each_entry_rcu(xs, &umem->xsk_tx_list, list) {
277		− if (!xskq_cons_peek_desc(xs->tx, desc, umem)) {
	316	+ list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) {
	317	+ if (!xskq_cons_peek_desc(xs->tx, desc, pool)) {
278	318	xs->tx->queue_empty_descs++;
279	319	continue;
280	320	}
@@ -284,7 +324,7 @@ bool xsk_umem_consume_tx(struct xdp_umem umem, struct xdp_desc desc) · 3 unmodified lines
284	324	* if there is space in it. This avoids having to implement
285	325	* any buffering in the Tx path.
286	326	*/
287		− if (xskq_prod_reserve_addr(umem->cq, desc->addr))
	327	+ if (xskq_prod_reserve_addr(pool->cq, desc->addr))
288	328	goto out;
289	329
290	330	xskq_cons_release(xs->tx);
@@ -296,7 +336,7 @@ out: · 5 unmodified lines
296	336	rcu_read_unlock();
297	337	return false;
298	338	}
299		−EXPORT_SYMBOL(xsk_umem_consume_tx);
	339	+EXPORT_SYMBOL(xsk_tx_peek_desc);
300	340
301	341	static int xsk_wakeup(struct xdp_sock *xs, u8 flags)
302	342	{
@@ -322,7 +362,7 @@ static void xsk_destruct_skb(struct sk_buff *skb) · 19 unmodified lines
322	362	unsigned long flags;
323	363
324	364	spin_lock_irqsave(&xs->tx_completion_lock, flags);
325		− xskq_prod_submit_addr(xs->umem->cq, addr);
	365	+ xskq_prod_submit_addr(xs->pool->cq, addr);
326	366	spin_unlock_irqrestore(&xs->tx_completion_lock, flags);
327	367
328	368	sock_wfree(skb);
@@ -342,7 +382,7 @@ static int xsk_generic_xmit(struct sock *sk) · 13 unmodified lines
342	382	if (xs->queue_id >= xs->dev->real_num_tx_queues)
343	383	goto out;
344	384
345		− while (xskq_cons_peek_desc(xs->tx, &desc, xs->umem)) {
	385	+ while (xskq_cons_peek_desc(xs->tx, &desc, xs->pool)) {
346	386	char *buffer;
347	387	u64 addr;
348	388	u32 len;
@@ -359,14 +399,14 @@ static int xsk_generic_xmit(struct sock *sk) · 10 unmodified lines
359	399
360	400	skb_put(skb, len);
361	401	addr = desc.addr;
362		− buffer = xsk_buff_raw_get_data(xs->umem, addr);
	402	+ buffer = xsk_buff_raw_get_data(xs->pool, addr);
363	403	err = skb_store_bits(skb, 0, buffer, len);
364	404	/* This is the backpressure mechanism for the Tx path.
365	405	* Reserve space in the completion queue and only proceed
366	406	* if there is space in it. This avoids having to implement
367	407	* any buffering in the Tx path.
368	408	*/
369		− if (unlikely(err) \|\| xskq_prod_reserve(xs->umem->cq)) {
	409	+ if (unlikely(err) \|\| xskq_prod_reserve(xs->pool->cq)) {
370	410	kfree_skb(skb);
371	411	goto out;
372	412	}
@@ -431,16 +471,16 @@ static __poll_t xsk_poll(struct file file, struct socket sock, · 58 unmodified lines
431	471	__poll_t mask = datagram_poll(file, sock, wait);
432	472	struct sock *sk = sock->sk;
433	473	struct xdp_sock *xs = xdp_sk(sk);
434		− struct xdp_umem *umem;
	474	+ struct xsk_buff_pool *pool;
435	475
436	476	if (unlikely(!xsk_is_bound(xs)))
437	477	return mask;
438	478
439		− umem = xs->umem;
	479	+ pool = xs->pool;
440	480
441		− if (umem->need_wakeup) {
	481	+ if (pool->cached_need_wakeup) {
442	482	if (xs->zc)
443		− xsk_wakeup(xs, umem->need_wakeup);
	483	+ xsk_wakeup(xs, pool->cached_need_wakeup);
444	484	else
445	485	/* Poll needs to drive Tx also in copy mode */
446	486	__xsk_sendmsg(sk);
@@ -481,7 +521,7 @@ static void xsk_unbind_dev(struct xdp_sock *xs) · 34 unmodified lines
481	521	WRITE_ONCE(xs->state, XSK_UNBOUND);
482	522
483	523	/* Wait for driver to stop using the xdp socket. */
484		− xdp_del_sk_umem(xs->umem, xs);
	524	+ xp_del_xsk(xs->pool, xs);
485	525	xs->dev = NULL;
486	526	synchronize_net();
487	527	dev_put(dev);
@@ -559,6 +599,8 @@ static int xsk_release(struct socket *sock) · 71 unmodified lines
559	599
560	600	xskq_destroy(xs->rx);
561	601	xskq_destroy(xs->tx);
	602	+ xskq_destroy(xs->fq_tmp);
	603	+ xskq_destroy(xs->cq_tmp);
562	604
563	605	sock_orphan(sk);
564	606	sock->sk = NULL;
@@ -586,6 +628,11 @@ static struct socket *xsk_lookup_xsk_from_fd(int fd) · 21 unmodified lines
586	628	return sock;
587	629	}
588	630
	631	+static bool xsk_validate_queues(struct xdp_sock *xs)
	632	+{
	633	+ return xs->fq_tmp && xs->cq_tmp;
	634	+}
	635	+
589	636	static int xsk_bind(struct socket sock, struct sockaddr addr, int addr_len)
590	637	{
591	638	struct sockaddr_xdp sxdp = (struct sockaddr_xdp )addr;
@@ -654,29 +701,64 @@ static int xsk_bind(struct socket sock, struct sockaddr addr, int addr_len) · 62 unmodified lines
654	701	sockfd_put(sock);
655	702	goto out_unlock;
656	703	}
657		− if (umem_xs->dev != dev \|\| umem_xs->queue_id != qid) {
658		− err = -EINVAL;
659		− sockfd_put(sock);
660		− goto out_unlock;
	704	+
	705	+ if (umem_xs->queue_id != qid \|\| umem_xs->dev != dev) {
	706	+ /* Share the umem with another socket on another qid
	707	+ * and/or device.
	708	+ */
	709	+ xs->pool = xp_create_and_assign_umem(xs,
	710	+ umem_xs->umem);
	711	+ if (!xs->pool) {
	712	+ sockfd_put(sock);
	713	+ goto out_unlock;
	714	+ }
	715	+
	716	+ err = xp_assign_dev_shared(xs->pool, umem_xs->umem,
	717	+ dev, qid);
	718	+ if (err) {
	719	+ xp_destroy(xs->pool);
	720	+ sockfd_put(sock);
	721	+ goto out_unlock;
	722	+ }
	723	+ } else {
	724	+ /* Share the buffer pool with the other socket. */
	725	+ if (xs->fq_tmp \|\| xs->cq_tmp) {
	726	+ /* Do not allow setting your own fq or cq. */
	727	+ err = -EINVAL;
	728	+ sockfd_put(sock);
	729	+ goto out_unlock;
	730	+ }
	731	+
	732	+ xp_get_pool(umem_xs->pool);
	733	+ xs->pool = umem_xs->pool;
661	734	}
662	735
663	736	xdp_get_umem(umem_xs->umem);
664	737	WRITE_ONCE(xs->umem, umem_xs->umem);
665	738	sockfd_put(sock);
666		− } else if (!xs->umem \|\| !xdp_umem_validate_queues(xs->umem)) {
	739	+ } else if (!xs->umem \|\| !xsk_validate_queues(xs)) {
667	740	err = -EINVAL;
668	741	goto out_unlock;
669	742	} else {
670	743	/* This xsk has its own umem. */
671		− err = xdp_umem_assign_dev(xs->umem, dev, qid, flags);
672		− if (err)
	744	+ xs->pool = xp_create_and_assign_umem(xs, xs->umem);
	745	+ if (!xs->pool) {
	746	+ err = -ENOMEM;
	747	+ goto out_unlock;
	748	+ }
	749	+
	750	+ err = xp_assign_dev(xs->pool, dev, qid, flags);
	751	+ if (err) {
	752	+ xp_destroy(xs->pool);
	753	+ xs->pool = NULL;
673	754	goto out_unlock;
	755	+ }
674	756	}
675	757
676	758	xs->dev = dev;
677	759	xs->zc = xs->umem->zc;
678	760	xs->queue_id = qid;
679		− xdp_add_sk_umem(xs->umem, xs);
	761	+ xp_add_xsk(xs->pool, xs);
680	762
681	763	out_unlock:
682	764	if (err) {
@@ -782,16 +864,10 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname, · 99 unmodified lines
782	864	mutex_unlock(&xs->mutex);
783	865	return -EBUSY;
784	866	}
785		− if (!xs->umem) {
786		− mutex_unlock(&xs->mutex);
787		− return -EINVAL;
788		− }
789	867
790		− q = (optname == XDP_UMEM_FILL_RING) ? &xs->umem->fq :
791		− &xs->umem->cq;
	868	+ q = (optname == XDP_UMEM_FILL_RING) ? &xs->fq_tmp :
	869	+ &xs->cq_tmp;
792	870	err = xsk_init_queue(entries, q, true);
793		− if (optname == XDP_UMEM_FILL_RING)
794		− xp_set_fq(xs->umem->pool, *q);
795	871	mutex_unlock(&xs->mutex);
796	872	return err;
797	873	}
@@ -858,7 +934,7 @@ static int xsk_getsockopt(struct socket *sock, int level, int optname, · 60 unmodified lines
858	934	if (extra_stats) {
859	935	stats.rx_ring_full = xs->rx_queue_full;
860	936	stats.rx_fill_ring_empty_descs =
861		− xs->umem ? xskq_nb_queue_empty_descs(xs->umem->fq) : 0;
	937	+ xs->pool ? xskq_nb_queue_empty_descs(xs->pool->fq) : 0;
862	938	stats.tx_ring_empty_descs = xskq_nb_queue_empty_descs(xs->tx);
863	939	} else {
864	940	stats.rx_dropped += xs->rx_queue_full;
@@ -960,7 +1036,6 @@ static int xsk_mmap(struct file file, struct socket sock, · 95 unmodified lines
960	1036	unsigned long size = vma->vm_end - vma->vm_start;
961	1037	struct xdp_sock *xs = xdp_sk(sock->sk);
962	1038	struct xsk_queue *q = NULL;
963		− struct xdp_umem *umem;
964	1039	unsigned long pfn;
965	1040	struct page *qpg;
966	1041
@@ -972,16 +1047,12 @@ static int xsk_mmap(struct file file, struct socket sock, · 5 unmodified lines
972	1047	} else if (offset == XDP_PGOFF_TX_RING) {
973	1048	q = READ_ONCE(xs->tx);
974	1049	} else {
975		− umem = READ_ONCE(xs->umem);
976		− if (!umem)
977		− return -EINVAL;
978		−
979	1050	/* Matches the smp_wmb() in XDP_UMEM_REG */
980	1051	smp_rmb();
981	1052	if (offset == XDP_UMEM_PGOFF_FILL_RING)
982		− q = READ_ONCE(umem->fq);
	1053	+ q = READ_ONCE(xs->fq_tmp);
983	1054	else if (offset == XDP_UMEM_PGOFF_COMPLETION_RING)
984		− q = READ_ONCE(umem->cq);
	1055	+ q = READ_ONCE(xs->cq_tmp);
985	1056	}
986	1057
987	1058	if (!q)
@@ -1019,8 +1090,8 @@ static int xsk_notifier(struct notifier_block *this, · 31 unmodified lines
1019	1090
1020	1091	xsk_unbind_dev(xs);
1021	1092
1022		− /* Clear device references in umem. */
1023		− xdp_umem_clear_dev(xs->umem);
	1093	+ /* Clear device references. */
	1094	+ xp_clear_dev(xs->pool);
1024	1095	}
1025	1096	mutex_unlock(&xs->mutex);
1026	1097	}
@@ -1064,7 +1135,7 @@ static void xsk_destruct(struct sock *sk) · 37 unmodified lines
1064	1135	if (!sock_flag(sk, SOCK_DEAD))
1065	1136	return;
1066	1137
1067		− xdp_put_umem(xs->umem);
	1138	+ xp_put_pool(xs->pool);
1068	1139
1069	1140	sk_refcnt_debug_dec(sk);
1070	1141	}
@@ -1072,8 +1143,8 @@ static void xsk_destruct(struct sock *sk) · 1 unmodified line
1072	1143	static int xsk_create(struct net net, struct socket sock, int protocol,
1073	1144	int kern)
1074	1145	{
1075		− struct sock *sk;
1076	1146	struct xdp_sock *xs;
	1147	+ struct sock *sk;
1077	1148
1078	1149	if (!ns_capable(net->user_ns, CAP_NET_RAW))
1079	1150	return -EPERM;

net/xdp/xsk.h+3−7

@@ -11,13 +11,6 @@ · 10 unmodified lines
11	11	#define XSK_NEXT_PG_CONTIG_SHIFT 0
12	12	#define XSK_NEXT_PG_CONTIG_MASK BIT_ULL(XSK_NEXT_PG_CONTIG_SHIFT)
13	13
14		−/* Flags for the umem flags field.
15		− *
16		− * The NEED_WAKEUP flag is 1 due to the reuse of the flags field for public
17		− * flags. See inlude/uapi/include/linux/if_xdp.h.
18		− */
19		−#define XDP_UMEM_USES_NEED_WAKEUP BIT(1)
20		−
21	14	struct xdp_ring_offset_v1 {
22	15	__u64 producer;
23	16	__u64 consumer;
@@ -51,5 +44,8 @@ void xsk_map_try_sock_delete(struct xsk_map map, struct xdp_sock xs, · 27 unmodified lines
51	44	struct xdp_sock **map_entry);
52	45	int xsk_map_inc(struct xsk_map *map);
53	46	void xsk_map_put(struct xsk_map *map);
	47	+void xsk_clear_pool_at_qid(struct net_device *dev, u16 queue_id);
	48	+int xsk_reg_pool_at_qid(struct net_device dev, struct xsk_buff_pool pool,
	49	+ u16 queue_id);
54	50
55	51	#endif /* XSK_H_ */

net/xdp/xsk_buff_pool.c+324−56

@@ -2,21 +2,37 @@ · 1 unmodified line
2	2
3	3	#include <net/xsk_buff_pool.h>
4	4	#include <net/xdp_sock.h>
	5	+#include <net/xdp_sock_drv.h>
	6	+#include <linux/dma-direct.h>
	7	+#include <linux/dma-noncoherent.h>
	8	+#include <linux/swiotlb.h>
5	9
6	10	#include "xsk_queue.h"
	11	+#include "xdp_umem.h"
	12	+#include "xsk.h"
7	13
8		−static void xp_addr_unmap(struct xsk_buff_pool *pool)
	14	+void xp_add_xsk(struct xsk_buff_pool pool, struct xdp_sock xs)
9	15	{
10		− vunmap(pool->addrs);
	16	+ unsigned long flags;
	17	+
	18	+ if (!xs->tx)
	19	+ return;
	20	+
	21	+ spin_lock_irqsave(&pool->xsk_tx_list_lock, flags);
	22	+ list_add_rcu(&xs->tx_list, &pool->xsk_tx_list);
	23	+ spin_unlock_irqrestore(&pool->xsk_tx_list_lock, flags);
11	24	}
12	25
13		−static int xp_addr_map(struct xsk_buff_pool *pool,
14		− struct page **pages, u32 nr_pages)
	26	+void xp_del_xsk(struct xsk_buff_pool pool, struct xdp_sock xs)
15	27	{
16		− pool->addrs = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL);
17		− if (!pool->addrs)
18		− return -ENOMEM;
19		− return 0;
	28	+ unsigned long flags;
	29	+
	30	+ if (!xs->tx)
	31	+ return;
	32	+
	33	+ spin_lock_irqsave(&pool->xsk_tx_list_lock, flags);
	34	+ list_del_rcu(&xs->tx_list);
	35	+ spin_unlock_irqrestore(&pool->xsk_tx_list_lock, flags);
20	36	}
21	37
22	38	void xp_destroy(struct xsk_buff_pool *pool)
@@ -24,59 +40,61 @@ void xp_destroy(struct xsk_buff_pool *pool) · 1 unmodified line
24	40	if (!pool)
25	41	return;
26	42
27		− xp_addr_unmap(pool);
28	43	kvfree(pool->heads);
29	44	kvfree(pool);
30	45	}
31	46
32		−struct xsk_buff_pool xp_create(struct page *pages, u32 nr_pages, u32 chunks,
33		− u32 chunk_size, u32 headroom, u64 size,
34		− bool unaligned)
	47	+struct xsk_buff_pool xp_create_and_assign_umem(struct xdp_sock xs,
	48	+ struct xdp_umem *umem)
35	49	{
36	50	struct xsk_buff_pool *pool;
37	51	struct xdp_buff_xsk *xskb;
38		− int err;
39	52	u32 i;
40	53
41		− pool = kvzalloc(struct_size(pool, free_heads, chunks), GFP_KERNEL);
	54	+ pool = kvzalloc(struct_size(pool, free_heads, umem->chunks),
	55	+ GFP_KERNEL);
42	56	if (!pool)
43	57	goto out;
44	58
45		− pool->heads = kvcalloc(chunks, sizeof(*pool->heads), GFP_KERNEL);
	59	+ pool->heads = kvcalloc(umem->chunks, sizeof(*pool->heads), GFP_KERNEL);
46	60	if (!pool->heads)
47	61	goto out;
48	62
49		− pool->chunk_mask = ~((u64)chunk_size - 1);
50		− pool->addrs_cnt = size;
51		− pool->heads_cnt = chunks;
52		− pool->free_heads_cnt = chunks;
53		− pool->headroom = headroom;
54		− pool->chunk_size = chunk_size;
55		− pool->unaligned = unaligned;
56		− pool->frame_len = chunk_size - headroom - XDP_PACKET_HEADROOM;
	63	+ pool->chunk_mask = ~((u64)umem->chunk_size - 1);
	64	+ pool->addrs_cnt = umem->size;
	65	+ pool->heads_cnt = umem->chunks;
	66	+ pool->free_heads_cnt = umem->chunks;
	67	+ pool->headroom = umem->headroom;
	68	+ pool->chunk_size = umem->chunk_size;
	69	+ pool->unaligned = umem->flags & XDP_UMEM_UNALIGNED_CHUNK_FLAG;
	70	+ pool->frame_len = umem->chunk_size - umem->headroom -
	71	+ XDP_PACKET_HEADROOM;
	72	+ pool->umem = umem;
	73	+ pool->addrs = umem->addrs;
57	74	INIT_LIST_HEAD(&pool->free_list);
	75	+ INIT_LIST_HEAD(&pool->xsk_tx_list);
	76	+ spin_lock_init(&pool->xsk_tx_list_lock);
	77	+ refcount_set(&pool->users, 1);
	78	+
	79	+ pool->fq = xs->fq_tmp;
	80	+ pool->cq = xs->cq_tmp;
	81	+ xs->fq_tmp = NULL;
	82	+ xs->cq_tmp = NULL;
58	83
59	84	for (i = 0; i < pool->free_heads_cnt; i++) {
60	85	xskb = &pool->heads[i];
61	86	xskb->pool = pool;
62		− xskb->xdp.frame_sz = chunk_size - headroom;
	87	+ xskb->xdp.frame_sz = umem->chunk_size - umem->headroom;
63	88	pool->free_heads[i] = xskb;
64	89	}
65	90
66		− err = xp_addr_map(pool, pages, nr_pages);
67		− if (!err)
68		− return pool;
	91	+ return pool;
69	92
70	93	out:
71	94	xp_destroy(pool);
72	95	return NULL;
73	96	}
74	97
75		−void xp_set_fq(struct xsk_buff_pool pool, struct xsk_queue fq)
76		−{
77		− pool->fq = fq;
78		−}
79		−
80	98	void xp_set_rxq_info(struct xsk_buff_pool pool, struct xdp_rxq_info rxq)
81	99	{
82	100	u32 i;
@@ -86,70 +104,320 @@ void xp_set_rxq_info(struct xsk_buff_pool pool, struct xdp_rxq_info rxq) · 3 unmodified lines
86	104	}
87	105	EXPORT_SYMBOL(xp_set_rxq_info);
88	106
89		−void xp_dma_unmap(struct xsk_buff_pool *pool, unsigned long attrs)
	107	+static void xp_disable_drv_zc(struct xsk_buff_pool *pool)
90	108	{
91		− dma_addr_t *dma;
92		− u32 i;
	109	+ struct netdev_bpf bpf;
	110	+ int err;
93	111
94		− if (pool->dma_pages_cnt == 0)
	112	+ ASSERT_RTNL();
	113	+
	114	+ if (pool->umem->zc) {
	115	+ bpf.command = XDP_SETUP_XSK_POOL;
	116	+ bpf.xsk.pool = NULL;
	117	+ bpf.xsk.queue_id = pool->queue_id;
	118	+
	119	+ err = pool->netdev->netdev_ops->ndo_bpf(pool->netdev, &bpf);
	120	+
	121	+ if (err)
	122	+ WARN(1, "Failed to disable zero-copy!\n");
	123	+ }
	124	+}
	125	+
	126	+static int __xp_assign_dev(struct xsk_buff_pool *pool,
	127	+ struct net_device *netdev, u16 queue_id, u16 flags)
	128	+{
	129	+ bool force_zc, force_copy;
	130	+ struct netdev_bpf bpf;
	131	+ int err = 0;
	132	+
	133	+ ASSERT_RTNL();
	134	+
	135	+ force_zc = flags & XDP_ZEROCOPY;
	136	+ force_copy = flags & XDP_COPY;
	137	+
	138	+ if (force_zc && force_copy)
	139	+ return -EINVAL;
	140	+
	141	+ if (xsk_get_pool_from_qid(netdev, queue_id))
	142	+ return -EBUSY;
	143	+
	144	+ pool->netdev = netdev;
	145	+ pool->queue_id = queue_id;
	146	+ err = xsk_reg_pool_at_qid(netdev, pool, queue_id);
	147	+ if (err)
	148	+ return err;
	149	+
	150	+ if (flags & XDP_USE_NEED_WAKEUP) {
	151	+ pool->uses_need_wakeup = true;
	152	+ /* Tx needs to be explicitly woken up the first time.
	153	+ * Also for supporting drivers that do not implement this
	154	+ * feature. They will always have to call sendto().
	155	+ */
	156	+ pool->cached_need_wakeup = XDP_WAKEUP_TX;
	157	+ }
	158	+
	159	+ dev_hold(netdev);
	160	+
	161	+ if (force_copy)
	162	+ /* For copy-mode, we are done. */
	163	+ return 0;
	164	+
	165	+ if (!netdev->netdev_ops->ndo_bpf \|\|
	166	+ !netdev->netdev_ops->ndo_xsk_wakeup) {
	167	+ err = -EOPNOTSUPP;
	168	+ goto err_unreg_pool;
	169	+ }
	170	+
	171	+ bpf.command = XDP_SETUP_XSK_POOL;
	172	+ bpf.xsk.pool = pool;
	173	+ bpf.xsk.queue_id = queue_id;
	174	+
	175	+ err = netdev->netdev_ops->ndo_bpf(netdev, &bpf);
	176	+ if (err)
	177	+ goto err_unreg_pool;
	178	+
	179	+ if (!pool->dma_pages) {
	180	+ WARN(1, "Driver did not DMA map zero-copy buffers");
	181	+ goto err_unreg_xsk;
	182	+ }
	183	+ pool->umem->zc = true;
	184	+ return 0;
	185	+
	186	+err_unreg_xsk:
	187	+ xp_disable_drv_zc(pool);
	188	+err_unreg_pool:
	189	+ if (!force_zc)
	190	+ err = 0; /* fallback to copy mode */
	191	+ if (err)
	192	+ xsk_clear_pool_at_qid(netdev, queue_id);
	193	+ return err;
	194	+}
	195	+
	196	+int xp_assign_dev(struct xsk_buff_pool pool, struct net_device dev,
	197	+ u16 queue_id, u16 flags)
	198	+{
	199	+ return __xp_assign_dev(pool, dev, queue_id, flags);
	200	+}
	201	+
	202	+int xp_assign_dev_shared(struct xsk_buff_pool pool, struct xdp_umem umem,
	203	+ struct net_device *dev, u16 queue_id)
	204	+{
	205	+ u16 flags;
	206	+
	207	+ /* One fill and completion ring required for each queue id. */
	208	+ if (!pool->fq \|\| !pool->cq)
	209	+ return -EINVAL;
	210	+
	211	+ flags = umem->zc ? XDP_ZEROCOPY : XDP_COPY;
	212	+ if (pool->uses_need_wakeup)
	213	+ flags \|= XDP_USE_NEED_WAKEUP;
	214	+
	215	+ return __xp_assign_dev(pool, dev, queue_id, flags);
	216	+}
	217	+
	218	+void xp_clear_dev(struct xsk_buff_pool *pool)
	219	+{
	220	+ if (!pool->netdev)
95	221	return;
96	222
97		− for (i = 0; i < pool->dma_pages_cnt; i++) {
98		− dma = &pool->dma_pages[i];
	223	+ xp_disable_drv_zc(pool);
	224	+ xsk_clear_pool_at_qid(pool->netdev, pool->queue_id);
	225	+ dev_put(pool->netdev);
	226	+ pool->netdev = NULL;
	227	+}
	228	+
	229	+static void xp_release_deferred(struct work_struct *work)
	230	+{
	231	+ struct xsk_buff_pool *pool = container_of(work, struct xsk_buff_pool,
	232	+ work);
	233	+
	234	+ rtnl_lock();
	235	+ xp_clear_dev(pool);
	236	+ rtnl_unlock();
	237	+
	238	+ if (pool->fq) {
	239	+ xskq_destroy(pool->fq);
	240	+ pool->fq = NULL;
	241	+ }
	242	+
	243	+ if (pool->cq) {
	244	+ xskq_destroy(pool->cq);
	245	+ pool->cq = NULL;
	246	+ }
	247	+
	248	+ xdp_put_umem(pool->umem);
	249	+ xp_destroy(pool);
	250	+}
	251	+
	252	+void xp_get_pool(struct xsk_buff_pool *pool)
	253	+{
	254	+ refcount_inc(&pool->users);
	255	+}
	256	+
	257	+void xp_put_pool(struct xsk_buff_pool *pool)
	258	+{
	259	+ if (!pool)
	260	+ return;
	261	+
	262	+ if (refcount_dec_and_test(&pool->users)) {
	263	+ INIT_WORK(&pool->work, xp_release_deferred);
	264	+ schedule_work(&pool->work);
	265	+ }
	266	+}
	267	+
	268	+static struct xsk_dma_map xp_find_dma_map(struct xsk_buff_pool pool)
	269	+{
	270	+ struct xsk_dma_map *dma_map;
	271	+
	272	+ list_for_each_entry(dma_map, &pool->umem->xsk_dma_list, list) {
	273	+ if (dma_map->netdev == pool->netdev)
	274	+ return dma_map;
	275	+ }
	276	+
	277	+ return NULL;
	278	+}
	279	+
	280	+static struct xsk_dma_map xp_create_dma_map(struct device dev, struct net_device *netdev,
	281	+ u32 nr_pages, struct xdp_umem *umem)
	282	+{
	283	+ struct xsk_dma_map *dma_map;
	284	+
	285	+ dma_map = kzalloc(sizeof(*dma_map), GFP_KERNEL);
	286	+ if (!dma_map)
	287	+ return NULL;
	288	+
	289	+ dma_map->dma_pages = kvcalloc(nr_pages, sizeof(*dma_map->dma_pages), GFP_KERNEL);
	290	+ if (!dma_map) {
	291	+ kfree(dma_map);
	292	+ return NULL;
	293	+ }
	294	+
	295	+ dma_map->netdev = netdev;
	296	+ dma_map->dev = dev;
	297	+ dma_map->dma_need_sync = false;
	298	+ dma_map->dma_pages_cnt = nr_pages;
	299	+ refcount_set(&dma_map->users, 0);
	300	+ list_add(&dma_map->list, &umem->xsk_dma_list);
	301	+ return dma_map;
	302	+}
	303	+
	304	+static void xp_destroy_dma_map(struct xsk_dma_map *dma_map)
	305	+{
	306	+ list_del(&dma_map->list);
	307	+ kvfree(dma_map->dma_pages);
	308	+ kfree(dma_map);
	309	+}
	310	+
	311	+static void __xp_dma_unmap(struct xsk_dma_map *dma_map, unsigned long attrs)
	312	+{
	313	+ dma_addr_t *dma;
	314	+ u32 i;
	315	+
	316	+ for (i = 0; i < dma_map->dma_pages_cnt; i++) {
	317	+ dma = &dma_map->dma_pages[i];
99	318	if (*dma) {
100		− dma_unmap_page_attrs(pool->dev, *dma, PAGE_SIZE,
	319	+ dma_unmap_page_attrs(dma_map->dev, *dma, PAGE_SIZE,
101	320	DMA_BIDIRECTIONAL, attrs);
102	321	*dma = 0;
103	322	}
104	323	}
105	324
	325	+ xp_destroy_dma_map(dma_map);
	326	+}
	327	+
	328	+void xp_dma_unmap(struct xsk_buff_pool *pool, unsigned long attrs)
	329	+{
	330	+ struct xsk_dma_map *dma_map;
	331	+
	332	+ if (pool->dma_pages_cnt == 0)
	333	+ return;
	334	+
	335	+ dma_map = xp_find_dma_map(pool);
	336	+ if (!dma_map) {
	337	+ WARN(1, "Could not find dma_map for device");
	338	+ return;
	339	+ }
	340	+
	341	+ if (!refcount_dec_and_test(&dma_map->users))
	342	+ return;
	343	+
	344	+ __xp_dma_unmap(dma_map, attrs);
106	345	kvfree(pool->dma_pages);
107	346	pool->dma_pages_cnt = 0;
108	347	pool->dev = NULL;
109	348	}
110	349	EXPORT_SYMBOL(xp_dma_unmap);
111	350
112		−static void xp_check_dma_contiguity(struct xsk_buff_pool *pool)
	351	+static void xp_check_dma_contiguity(struct xsk_dma_map *dma_map)
113	352	{
114	353	u32 i;
115	354
116		− for (i = 0; i < pool->dma_pages_cnt - 1; i++) {
117		− if (pool->dma_pages[i] + PAGE_SIZE == pool->dma_pages[i + 1])
118		− pool->dma_pages[i] \|= XSK_NEXT_PG_CONTIG_MASK;
	355	+ for (i = 0; i < dma_map->dma_pages_cnt - 1; i++) {
	356	+ if (dma_map->dma_pages[i] + PAGE_SIZE == dma_map->dma_pages[i + 1])
	357	+ dma_map->dma_pages[i] \|= XSK_NEXT_PG_CONTIG_MASK;
119	358	else
120		− pool->dma_pages[i] &= ~XSK_NEXT_PG_CONTIG_MASK;
	359	+ dma_map->dma_pages[i] &= ~XSK_NEXT_PG_CONTIG_MASK;
121	360	}
122	361	}
123	362
	363	+static int xp_init_dma_info(struct xsk_buff_pool pool, struct xsk_dma_map dma_map)
	364	+{
	365	+ pool->dma_pages = kvcalloc(dma_map->dma_pages_cnt, sizeof(*pool->dma_pages), GFP_KERNEL);
	366	+ if (!pool->dma_pages)
	367	+ return -ENOMEM;
	368	+
	369	+ pool->dev = dma_map->dev;
	370	+ pool->dma_pages_cnt = dma_map->dma_pages_cnt;
	371	+ pool->dma_need_sync = dma_map->dma_need_sync;
	372	+ refcount_inc(&dma_map->users);
	373	+ memcpy(pool->dma_pages, dma_map->dma_pages,
	374	+ pool->dma_pages_cnt * sizeof(*pool->dma_pages));
	375	+
	376	+ return 0;
	377	+}
	378	+
124	379	int xp_dma_map(struct xsk_buff_pool pool, struct device dev,
125	380	unsigned long attrs, struct page **pages, u32 nr_pages)
126	381	{
	382	+ struct xsk_dma_map *dma_map;
127	383	dma_addr_t dma;
	384	+ int err;
128	385	u32 i;
129	386
130		− pool->dma_pages = kvcalloc(nr_pages, sizeof(*pool->dma_pages),
131		− GFP_KERNEL);
132		− if (!pool->dma_pages)
133		− return -ENOMEM;
	387	+ dma_map = xp_find_dma_map(pool);
	388	+ if (dma_map) {
	389	+ err = xp_init_dma_info(pool, dma_map);
	390	+ if (err)
	391	+ return err;
	392	+
	393	+ return 0;
	394	+ }
134	395
135		− pool->dev = dev;
136		− pool->dma_pages_cnt = nr_pages;
137		− pool->dma_need_sync = false;
	396	+ dma_map = xp_create_dma_map(dev, pool->netdev, nr_pages, pool->umem);
	397	+ if (!dma_map)
	398	+ return -ENOMEM;
138	399
139		− for (i = 0; i < pool->dma_pages_cnt; i++) {
	400	+ for (i = 0; i < dma_map->dma_pages_cnt; i++) {
140	401	dma = dma_map_page_attrs(dev, pages[i], 0, PAGE_SIZE,
141	402	DMA_BIDIRECTIONAL, attrs);
142	403	if (dma_mapping_error(dev, dma)) {
143		− xp_dma_unmap(pool, attrs);
	404	+ __xp_dma_unmap(dma_map, attrs);
144	405	return -ENOMEM;
145	406	}
146	407	if (dma_need_sync(dev, dma))
147		− pool->dma_need_sync = true;
148		− pool->dma_pages[i] = dma;
	408	+ dma_map->dma_need_sync = true;
	409	+ dma_map->dma_pages[i] = dma;
149	410	}
150	411
151	412	if (pool->unaligned)
152		− xp_check_dma_contiguity(pool);
	413	+ xp_check_dma_contiguity(dma_map);
	414	+
	415	+ err = xp_init_dma_info(pool, dma_map);
	416	+ if (err) {
	417	+ __xp_dma_unmap(dma_map, attrs);
	418	+ return err;
	419	+ }
	420	+
153	421	return 0;
154	422	}
155	423	EXPORT_SYMBOL(xp_dma_map);

net/xdp/xsk_diag.c+9−7

@@ -46,6 +46,7 @@ static int xsk_diag_put_rings_cfg(const struct xdp_sock *xs, · 45 unmodified lines
46	46
47	47	static int xsk_diag_put_umem(const struct xdp_sock xs, struct sk_buff nlskb)
48	48	{
	49	+ struct xsk_buff_pool *pool = xs->pool;
49	50	struct xdp_umem *umem = xs->umem;
50	51	struct xdp_diag_umem du = {};
51	52	int err;
@@ -58,8 +59,8 @@ static int xsk_diag_put_umem(const struct xdp_sock xs, struct sk_buff nlskb) · 6 unmodified lines
58	59	du.num_pages = umem->npgs;
59	60	du.chunk_size = umem->chunk_size;
60	61	du.headroom = umem->headroom;
61		− du.ifindex = umem->dev ? umem->dev->ifindex : 0;
62		− du.queue_id = umem->queue_id;
	62	+ du.ifindex = pool->netdev ? pool->netdev->ifindex : 0;
	63	+ du.queue_id = pool->queue_id;
63	64	du.flags = 0;
64	65	if (umem->zc)
65	66	du.flags \|= XDP_DU_F_ZEROCOPY;
@@ -67,10 +68,11 @@ static int xsk_diag_put_umem(const struct xdp_sock xs, struct sk_buff nlskb) · 1 unmodified line
67	68
68	69	err = nla_put(nlskb, XDP_DIAG_UMEM, sizeof(du), &du);
69	70
70		− if (!err && umem->fq)
71		− err = xsk_diag_put_ring(umem->fq, XDP_DIAG_UMEM_FILL_RING, nlskb);
72		− if (!err && umem->cq) {
73		− err = xsk_diag_put_ring(umem->cq, XDP_DIAG_UMEM_COMPLETION_RING,
	71	+ if (!err && pool->fq)
	72	+ err = xsk_diag_put_ring(pool->fq,
	73	+ XDP_DIAG_UMEM_FILL_RING, nlskb);
	74	+ if (!err && pool->cq) {
	75	+ err = xsk_diag_put_ring(pool->cq, XDP_DIAG_UMEM_COMPLETION_RING,
74	76	nlskb);
75	77	}
76	78	return err;
@@ -83,7 +85,7 @@ static int xsk_diag_put_stats(const struct xdp_sock xs, struct sk_buff nlskb) · 6 unmodified lines
83	85	du.n_rx_dropped = xs->rx_dropped;
84	86	du.n_rx_invalid = xskq_nb_invalid_descs(xs->rx);
85	87	du.n_rx_full = xs->rx_queue_full;
86		− du.n_fill_ring_empty = xs->umem ? xskq_nb_queue_empty_descs(xs->umem->fq) : 0;
	88	+ du.n_fill_ring_empty = xs->pool ? xskq_nb_queue_empty_descs(xs->pool->fq) : 0;
87	89	du.n_tx_invalid = xskq_nb_invalid_descs(xs->tx);
88	90	du.n_tx_ring_empty = xskq_nb_queue_empty_descs(xs->tx);
89	91	return nla_put(nlskb, XDP_DIAG_STATS, sizeof(du), &du);

net/xdp/xsk_queue.h+6−6

@@ -166,9 +166,9 @@ static inline bool xp_validate_desc(struct xsk_buff_pool *pool, · 165 unmodified lines
166	166
167	167	static inline bool xskq_cons_is_valid_desc(struct xsk_queue *q,
168	168	struct xdp_desc *d,
169		− struct xdp_umem *umem)
	169	+ struct xsk_buff_pool *pool)
170	170	{
171		− if (!xp_validate_desc(umem->pool, d)) {
	171	+ if (!xp_validate_desc(pool, d)) {
172	172	q->invalid_descs++;
173	173	return false;
174	174	}
@@ -177,14 +177,14 @@ static inline bool xskq_cons_is_valid_desc(struct xsk_queue *q, · 2 unmodified lines
177	177
178	178	static inline bool xskq_cons_read_desc(struct xsk_queue *q,
179	179	struct xdp_desc *desc,
180		− struct xdp_umem *umem)
	180	+ struct xsk_buff_pool *pool)
181	181	{
182	182	while (q->cached_cons != q->cached_prod) {
183	183	struct xdp_rxtx_ring ring = (struct xdp_rxtx_ring )q->ring;
184	184	u32 idx = q->cached_cons & q->ring_mask;
185	185
186	186	*desc = ring->desc[idx];
187		− if (xskq_cons_is_valid_desc(q, desc, umem))
	187	+ if (xskq_cons_is_valid_desc(q, desc, pool))
188	188	return true;
189	189
190	190	q->cached_cons++;
@@ -236,11 +236,11 @@ static inline bool xskq_cons_peek_addr_unchecked(struct xsk_queue q, u64 addr) · 45 unmodified lines
236	236
237	237	static inline bool xskq_cons_peek_desc(struct xsk_queue *q,
238	238	struct xdp_desc *desc,
239		− struct xdp_umem *umem)
	239	+ struct xsk_buff_pool *pool)
240	240	{
241	241	if (q->cached_prod == q->cached_cons)
242	242	xskq_cons_get_entries(q);
243		− return xskq_cons_read_desc(q, desc, umem);
	243	+ return xskq_cons_read_desc(q, desc, pool);
244	244	}
245	245
246	246	static inline void xskq_cons_release(struct xsk_queue *q)

net/xdp/xskmap.c+8−0

@@ -254,8 +254,16 @@ void xsk_map_try_sock_delete(struct xsk_map map, struct xdp_sock xs, · 253 unmodified lines
254	254	spin_unlock_bh(&map->lock);
255	255	}
256	256
	257	+static bool xsk_map_meta_equal(const struct bpf_map *meta0,
	258	+ const struct bpf_map *meta1)
	259	+{
	260	+ return meta0->max_entries == meta1->max_entries &&
	261	+ bpf_map_meta_equal(meta0, meta1);
	262	+}
	263	+
257	264	static int xsk_map_btf_id;
258	265	const struct bpf_map_ops xsk_map_ops = {
	266	+ .map_meta_equal = xsk_map_meta_equal,
259	267	.map_alloc = xsk_map_alloc,
260	268	.map_free = xsk_map_free,
261	269	.map_get_next_key = xsk_map_get_next_key,

samples/bpf/Makefile+12−9

@@ -48,6 +48,7 @@ tprogs-y += syscall_tp · 47 unmodified lines
48	48	tprogs-y += cpustat
49	49	tprogs-y += xdp_adjust_tail
50	50	tprogs-y += xdpsock
	51	+tprogs-y += xsk_fwd
51	52	tprogs-y += xdp_fwd
52	53	tprogs-y += task_fd_query
53	54	tprogs-y += xdp_sample_pkts
@@ -71,12 +72,12 @@ tracex4-objs := tracex4_user.o · 17 unmodified lines
71	72	tracex5-objs := tracex5_user.o $(TRACE_HELPERS)
72	73	tracex6-objs := tracex6_user.o
73	74	tracex7-objs := tracex7_user.o
74		−test_probe_write_user-objs := bpf_load.o test_probe_write_user_user.o
75		−trace_output-objs := bpf_load.o trace_output_user.o $(TRACE_HELPERS)
76		−lathist-objs := bpf_load.o lathist_user.o
77		−offwaketime-objs := bpf_load.o offwaketime_user.o $(TRACE_HELPERS)
78		−spintest-objs := bpf_load.o spintest_user.o $(TRACE_HELPERS)
79		−map_perf_test-objs := bpf_load.o map_perf_test_user.o
	75	+test_probe_write_user-objs := test_probe_write_user_user.o
	76	+trace_output-objs := trace_output_user.o $(TRACE_HELPERS)
	77	+lathist-objs := lathist_user.o
	78	+offwaketime-objs := offwaketime_user.o $(TRACE_HELPERS)
	79	+spintest-objs := spintest_user.o $(TRACE_HELPERS)
	80	+map_perf_test-objs := map_perf_test_user.o
80	81	test_overhead-objs := bpf_load.o test_overhead_user.o
81	82	test_cgrp2_array_pin-objs := test_cgrp2_array_pin.o
82	83	test_cgrp2_attach-objs := test_cgrp2_attach.o
@@ -86,7 +87,7 @@ xdp1-objs := xdp1_user.o · 3 unmodified lines
86	87	# reuse xdp1 source intentionally
87	88	xdp2-objs := xdp1_user.o
88	89	xdp_router_ipv4-objs := xdp_router_ipv4_user.o
89		−test_current_task_under_cgroup-objs := bpf_load.o $(CGROUP_HELPERS) \
	90	+test_current_task_under_cgroup-objs := $(CGROUP_HELPERS) \
90	91	test_current_task_under_cgroup_user.o
91	92	trace_event-objs := trace_event_user.o $(TRACE_HELPERS)
92	93	sampleip-objs := sampleip_user.o $(TRACE_HELPERS)
@@ -100,10 +101,11 @@ xdp_redirect_map-objs := xdp_redirect_map_user.o · 7 unmodified lines
100	101	xdp_redirect_cpu-objs := bpf_load.o xdp_redirect_cpu_user.o
101	102	xdp_monitor-objs := bpf_load.o xdp_monitor_user.o
102	103	xdp_rxq_info-objs := xdp_rxq_info_user.o
103		−syscall_tp-objs := bpf_load.o syscall_tp_user.o
104		−cpustat-objs := bpf_load.o cpustat_user.o
	104	+syscall_tp-objs := syscall_tp_user.o
	105	+cpustat-objs := cpustat_user.o
105	106	xdp_adjust_tail-objs := xdp_adjust_tail_user.o
106	107	xdpsock-objs := xdpsock_user.o
	108	+xsk_fwd-objs := xsk_fwd.o
107	109	xdp_fwd-objs := xdp_fwd_user.o
108	110	task_fd_query-objs := bpf_load.o task_fd_query_user.o $(TRACE_HELPERS)
109	111	xdp_sample_pkts-objs := xdp_sample_pkts_user.o $(TRACE_HELPERS)
@@ -203,6 +205,7 @@ TPROGLDLIBS_trace_output += -lrt · 93 unmodified lines
203	205	TPROGLDLIBS_map_perf_test += -lrt
204	206	TPROGLDLIBS_test_overhead += -lrt
205	207	TPROGLDLIBS_xdpsock += -pthread
	208	+TPROGLDLIBS_xsk_fwd += -pthread
206	209
207	210	# Allows pointing LLC/CLANG to a LLVM backend with bpf support, redefine on cmdline:
208	211	# make M=samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang

samples/bpf/cpustat_kern.c+18−18

@@ -51,28 +51,28 @@ static int cpu_opps[] = { 208000, 432000, 729000, 960000, 1200000 }; · 50 unmodified lines
51	51	#define MAP_OFF_PSTATE_IDX 3
52	52	#define MAP_OFF_NUM 4
53	53
54		−struct bpf_map_def SEC("maps") my_map = {
55		− .type = BPF_MAP_TYPE_ARRAY,
56		− .key_size = sizeof(u32),
57		− .value_size = sizeof(u64),
58		− .max_entries = MAX_CPU * MAP_OFF_NUM,
59		−};
	54	+struct {
	55	+ __uint(type, BPF_MAP_TYPE_ARRAY);
	56	+ __type(key, u32);
	57	+ __type(value, u64);
	58	+ __uint(max_entries, MAX_CPU * MAP_OFF_NUM);
	59	+} my_map SEC(".maps");
60	60
61	61	/* cstate_duration records duration time for every idle state per CPU */
62		−struct bpf_map_def SEC("maps") cstate_duration = {
63		− .type = BPF_MAP_TYPE_ARRAY,
64		− .key_size = sizeof(u32),
65		− .value_size = sizeof(u64),
66		− .max_entries = MAX_CPU * MAX_CSTATE_ENTRIES,
67		−};
	62	+struct {
	63	+ __uint(type, BPF_MAP_TYPE_ARRAY);
	64	+ __type(key, u32);
	65	+ __type(value, u64);
	66	+ __uint(max_entries, MAX_CPU * MAX_CSTATE_ENTRIES);
	67	+} cstate_duration SEC(".maps");
68	68
69	69	/* pstate_duration records duration time for every operating point per CPU */
70		−struct bpf_map_def SEC("maps") pstate_duration = {
71		− .type = BPF_MAP_TYPE_ARRAY,
72		− .key_size = sizeof(u32),
73		− .value_size = sizeof(u64),
74		− .max_entries = MAX_CPU * MAX_PSTATE_ENTRIES,
75		−};
	70	+struct {
	71	+ __uint(type, BPF_MAP_TYPE_ARRAY);
	72	+ __type(key, u32);
	73	+ __type(value, u64);
	74	+ __uint(max_entries, MAX_CPU * MAX_PSTATE_ENTRIES);
	75	+} pstate_duration SEC(".maps");
76	76
77	77	/*
78	78	* The trace events for cpu_idle and cpu_frequency are taken from:

samples/bpf/cpustat_user.c+40−7

@@ -9,7 +9,6 @@ · 8 unmodified lines
9	9	#include <string.h>
10	10	#include <unistd.h>
11	11	#include <fcntl.h>
12		−#include <linux/bpf.h>
13	12	#include <locale.h>
14	13	#include <sys/types.h>
15	14	#include <sys/stat.h>
@@ -18,7 +17,9 @@ · 2 unmodified lines
18	17	#include <sys/wait.h>
19	18
20	19	#include <bpf/bpf.h>
21		−#include "bpf_load.h"
	20	+#include <bpf/libbpf.h>
	21	+
	22	+static int cstate_map_fd, pstate_map_fd;
22	23
23	24	#define MAX_CPU 8
24	25	#define MAX_PSTATE_ENTRIES 5
@@ -181,21 +182,50 @@ static void int_exit(int sig) · 156 unmodified lines
181	182	{
182	183	cpu_stat_inject_cpu_idle_event();
183	184	cpu_stat_inject_cpu_frequency_event();
184		− cpu_stat_update(map_fd[1], map_fd[2]);
	185	+ cpu_stat_update(cstate_map_fd, pstate_map_fd);
185	186	cpu_stat_print();
186	187	exit(0);
187	188	}
188	189
189	190	int main(int argc, char **argv)
190	191	{
	192	+ struct bpf_link *link = NULL;
	193	+ struct bpf_program *prog;
	194	+ struct bpf_object *obj;
191	195	char filename[256];
192	196	int ret;
193	197
194	198	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
	199	+ obj = bpf_object__open_file(filename, NULL);
	200	+ if (libbpf_get_error(obj)) {
	201	+ fprintf(stderr, "ERROR: opening BPF object file failed\n");
	202	+ return 0;
	203	+ }
195	204
196		− if (load_bpf_file(filename)) {
197		− printf("%s", bpf_log_buf);
198		− return 1;
	205	+ prog = bpf_object__find_program_by_name(obj, "bpf_prog1");
	206	+ if (!prog) {
	207	+ printf("finding a prog in obj file failed\n");
	208	+ goto cleanup;
	209	+ }
	210	+
	211	+ /* load BPF program */
	212	+ if (bpf_object__load(obj)) {
	213	+ fprintf(stderr, "ERROR: loading BPF object file failed\n");
	214	+ goto cleanup;
	215	+ }
	216	+
	217	+ cstate_map_fd = bpf_object__find_map_fd_by_name(obj, "cstate_duration");
	218	+ pstate_map_fd = bpf_object__find_map_fd_by_name(obj, "pstate_duration");
	219	+ if (cstate_map_fd < 0 \|\| pstate_map_fd < 0) {
	220	+ fprintf(stderr, "ERROR: finding a map in obj file failed\n");
	221	+ goto cleanup;
	222	+ }
	223	+
	224	+ link = bpf_program__attach(prog);
	225	+ if (libbpf_get_error(link)) {
	226	+ fprintf(stderr, "ERROR: bpf_program__attach failed\n");
	227	+ link = NULL;
	228	+ goto cleanup;
199	229	}
200	230
201	231	ret = cpu_stat_inject_cpu_idle_event();
@@ -210,10 +240,13 @@ int main(int argc, char **argv) · 8 unmodified lines
210	240	signal(SIGTERM, int_exit);
211	241
212	242	while (1) {
213		− cpu_stat_update(map_fd[1], map_fd[2]);
	243	+ cpu_stat_update(cstate_map_fd, pstate_map_fd);
214	244	cpu_stat_print();
215	245	sleep(5);
216	246	}
217	247
	248	+cleanup:
	249	+ bpf_link__destroy(link);
	250	+ bpf_object__close(obj);
218	251	return 0;
219	252	}

samples/bpf/lathist_kern.c+12−12

@@ -18,12 +18,12 @@ · 17 unmodified lines
18	18	* trace_preempt_[on\|off] tracepoints hooks is not supported.
19	19	*/
20	20
21		−struct bpf_map_def SEC("maps") my_map = {
22		− .type = BPF_MAP_TYPE_ARRAY,
23		− .key_size = sizeof(int),
24		− .value_size = sizeof(u64),
25		− .max_entries = MAX_CPU,
26		−};
	21	+struct {
	22	+ __uint(type, BPF_MAP_TYPE_ARRAY);
	23	+ __type(key, int);
	24	+ __type(value, u64);
	25	+ __uint(max_entries, MAX_CPU);
	26	+} my_map SEC(".maps");
27	27
28	28	SEC("kprobe/trace_preempt_off")
29	29	int bpf_prog1(struct pt_regs *ctx)
@@ -61,12 +61,12 @@ static unsigned int log2l(unsigned long v) · 31 unmodified lines
61	61	return log2(v);
62	62	}
63	63
64		−struct bpf_map_def SEC("maps") my_lat = {
65		− .type = BPF_MAP_TYPE_ARRAY,
66		− .key_size = sizeof(int),
67		− .value_size = sizeof(long),
68		− .max_entries = MAX_CPU * MAX_ENTRIES,
69		−};
	64	+struct {
	65	+ __uint(type, BPF_MAP_TYPE_ARRAY);
	66	+ __type(key, int);
	67	+ __type(value, long);
	68	+ __uint(max_entries, MAX_CPU * MAX_ENTRIES);
	69	+} my_lat SEC(".maps");
70	70
71	71	SEC("kprobe/trace_preempt_on")
72	72	int bpf_prog2(struct pt_regs *ctx)

samples/bpf/lathist_user.c+36−6

@@ -6,9 +6,8 @@ · 5 unmodified lines
6	6	#include <unistd.h>
7	7	#include <stdlib.h>
8	8	#include <signal.h>
9		−#include <linux/bpf.h>
	9	+#include <bpf/libbpf.h>
10	10	#include <bpf/bpf.h>
11		−#include "bpf_load.h"
12	11
13	12	#define MAX_ENTRIES 20
14	13	#define MAX_CPU 4
@@ -81,20 +80,51 @@ static void get_data(int fd) · 66 unmodified lines
81	80
82	81	int main(int argc, char **argv)
83	82	{
	83	+ struct bpf_link *links[2];
	84	+ struct bpf_program *prog;
	85	+ struct bpf_object *obj;
84	86	char filename[256];
	87	+ int map_fd, i = 0;
85	88
86	89	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
	90	+ obj = bpf_object__open_file(filename, NULL);
	91	+ if (libbpf_get_error(obj)) {
	92	+ fprintf(stderr, "ERROR: opening BPF object file failed\n");
	93	+ return 0;
	94	+ }
	95	+
	96	+ /* load BPF program */
	97	+ if (bpf_object__load(obj)) {
	98	+ fprintf(stderr, "ERROR: loading BPF object file failed\n");
	99	+ goto cleanup;
	100	+ }
87	101
88		− if (load_bpf_file(filename)) {
89		− printf("%s", bpf_log_buf);
90		− return 1;
	102	+ map_fd = bpf_object__find_map_fd_by_name(obj, "my_lat");
	103	+ if (map_fd < 0) {
	104	+ fprintf(stderr, "ERROR: finding a map in obj file failed\n");
	105	+ goto cleanup;
	106	+ }
	107	+
	108	+ bpf_object__for_each_program(prog, obj) {
	109	+ links[i] = bpf_program__attach(prog);
	110	+ if (libbpf_get_error(links[i])) {
	111	+ fprintf(stderr, "ERROR: bpf_program__attach failed\n");
	112	+ links[i] = NULL;
	113	+ goto cleanup;
	114	+ }
	115	+ i++;
91	116	}
92	117
93	118	while (1) {
94		− get_data(map_fd[1]);
	119	+ get_data(map_fd);
95	120	print_hist();
96	121	sleep(5);
97	122	}
98	123
	124	+cleanup:
	125	+ for (i--; i >= 0; i--)
	126	+ bpf_link__destroy(links[i]);
	127	+
	128	+ bpf_object__close(obj);
99	129	return 0;
100	130	}

samples/bpf/offwaketime_kern.c+26−26

@@ -28,38 +28,38 @@ struct key_t { · 27 unmodified lines
28	28	u32 tret;
29	29	};
30	30
31		−struct bpf_map_def SEC("maps") counts = {
32		− .type = BPF_MAP_TYPE_HASH,
33		− .key_size = sizeof(struct key_t),
34		− .value_size = sizeof(u64),
35		− .max_entries = 10000,
36		−};
37		−
38		−struct bpf_map_def SEC("maps") start = {
39		− .type = BPF_MAP_TYPE_HASH,
40		− .key_size = sizeof(u32),
41		− .value_size = sizeof(u64),
42		− .max_entries = 10000,
43		−};
	31	+struct {
	32	+ __uint(type, BPF_MAP_TYPE_HASH);
	33	+ __type(key, struct key_t);
	34	+ __type(value, u64);
	35	+ __uint(max_entries, 10000);
	36	+} counts SEC(".maps");
	37	+
	38	+struct {
	39	+ __uint(type, BPF_MAP_TYPE_HASH);
	40	+ __type(key, u32);
	41	+ __type(value, u64);
	42	+ __uint(max_entries, 10000);
	43	+} start SEC(".maps");
44	44
45	45	struct wokeby_t {
46	46	char name[TASK_COMM_LEN];
47	47	u32 ret;
48	48	};
49	49
50		−struct bpf_map_def SEC("maps") wokeby = {
51		− .type = BPF_MAP_TYPE_HASH,
52		− .key_size = sizeof(u32),
53		− .value_size = sizeof(struct wokeby_t),
54		− .max_entries = 10000,
55		−};
56		−
57		−struct bpf_map_def SEC("maps") stackmap = {
58		− .type = BPF_MAP_TYPE_STACK_TRACE,
59		− .key_size = sizeof(u32),
60		− .value_size = PERF_MAX_STACK_DEPTH * sizeof(u64),
61		− .max_entries = 10000,
62		−};
	50	+struct {
	51	+ __uint(type, BPF_MAP_TYPE_HASH);
	52	+ __type(key, u32);
	53	+ __type(value, struct wokeby_t);
	54	+ __uint(max_entries, 10000);
	55	+} wokeby SEC(".maps");
	56	+
	57	+struct {
	58	+ __uint(type, BPF_MAP_TYPE_STACK_TRACE);
	59	+ __uint(key_size, sizeof(u32));
	60	+ __uint(value_size, PERF_MAX_STACK_DEPTH * sizeof(u64));
	61	+ __uint(max_entries, 10000);
	62	+} stackmap SEC(".maps");
63	63
64	64	#define STACKID_FLAGS (0 \| BPF_F_FAST_STACK_CMP)
65	65

samples/bpf/offwaketime_user.c+51−15

@@ -5,19 +5,19 @@ · 4 unmodified lines
5	5	#include <unistd.h>
6	6	#include <stdlib.h>
7	7	#include <signal.h>
8		−#include <linux/bpf.h>
9		−#include <string.h>
10	8	#include <linux/perf_event.h>
11	9	#include <errno.h>
12		−#include <assert.h>
13	10	#include <stdbool.h>
14	11	#include <sys/resource.h>
15	12	#include <bpf/libbpf.h>
16		−#include "bpf_load.h"
	13	+#include <bpf/bpf.h>
17	14	#include "trace_helpers.h"
18	15
19	16	#define PRINT_RAW_ADDR 0
20	17
	18	+/* counts, stackmap */
	19	+static int map_fd[2];
	20	+
21	21	static void print_ksym(__u64 addr)
22	22	{
23	23	struct ksym *sym;
@@ -52,14 +52,14 @@ static void print_stack(struct key_t *key, __u64 count) · 28 unmodified lines
52	52	int i;
53	53
54	54	printf("%s;", key->target);
55		− if (bpf_map_lookup_elem(map_fd[3], &key->tret, ip) != 0) {
	55	+ if (bpf_map_lookup_elem(map_fd[1], &key->tret, ip) != 0) {
56	56	printf("---;");
57	57	} else {
58	58	for (i = PERF_MAX_STACK_DEPTH - 1; i >= 0; i--)
59	59	print_ksym(ip[i]);
60	60	}
61	61	printf("-;");
62		− if (bpf_map_lookup_elem(map_fd[3], &key->wret, ip) != 0) {
	62	+ if (bpf_map_lookup_elem(map_fd[1], &key->wret, ip) != 0) {
63	63	printf("---;");
64	64	} else {
65	65	for (i = 0; i < PERF_MAX_STACK_DEPTH; i++)
@@ -96,23 +96,54 @@ static void int_exit(int sig) · 30 unmodified lines
96	96	int main(int argc, char **argv)
97	97	{
98	98	struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
	99	+ struct bpf_object *obj = NULL;
	100	+ struct bpf_link *links[2];
	101	+ struct bpf_program *prog;
	102	+ int delay = 1, i = 0;
99	103	char filename[256];
100		− int delay = 1;
101		−
102		− snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
103		− setrlimit(RLIMIT_MEMLOCK, &r);
104	104
105		− signal(SIGINT, int_exit);
106		− signal(SIGTERM, int_exit);
	105	+ if (setrlimit(RLIMIT_MEMLOCK, &r)) {
	106	+ perror("setrlimit(RLIMIT_MEMLOCK)");
	107	+ return 1;
	108	+ }
107	109
108	110	if (load_kallsyms()) {
109	111	printf("failed to process /proc/kallsyms\n");
110	112	return 2;
111	113	}
112	114
113		− if (load_bpf_file(filename)) {
114		− printf("%s", bpf_log_buf);
115		− return 1;
	115	+ snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
	116	+ obj = bpf_object__open_file(filename, NULL);
	117	+ if (libbpf_get_error(obj)) {
	118	+ fprintf(stderr, "ERROR: opening BPF object file failed\n");
	119	+ obj = NULL;
	120	+ goto cleanup;
	121	+ }
	122	+
	123	+ /* load BPF program */
	124	+ if (bpf_object__load(obj)) {
	125	+ fprintf(stderr, "ERROR: loading BPF object file failed\n");
	126	+ goto cleanup;
	127	+ }
	128	+
	129	+ map_fd[0] = bpf_object__find_map_fd_by_name(obj, "counts");
	130	+ map_fd[1] = bpf_object__find_map_fd_by_name(obj, "stackmap");
	131	+ if (map_fd[0] < 0 \|\| map_fd[1] < 0) {
	132	+ fprintf(stderr, "ERROR: finding a map in obj file failed\n");
	133	+ goto cleanup;
	134	+ }
	135	+
	136	+ signal(SIGINT, int_exit);
	137	+ signal(SIGTERM, int_exit);
	138	+
	139	+ bpf_object__for_each_program(prog, obj) {
	140	+ links[i] = bpf_program__attach(prog);
	141	+ if (libbpf_get_error(links[i])) {
	142	+ fprintf(stderr, "ERROR: bpf_program__attach failed\n");
	143	+ links[i] = NULL;
	144	+ goto cleanup;
	145	+ }
	146	+ i++;
116	147	}
117	148
118	149	if (argc > 1)
@@ -120,5 +151,10 @@ int main(int argc, char **argv) · 1 unmodified line
120	151	sleep(delay);
121	152	print_stacks(map_fd[0]);
122	153
	154	+cleanup:
	155	+ for (i--; i >= 0; i--)
	156	+ bpf_link__destroy(links[i]);
	157	+
	158	+ bpf_object__close(obj);
123	159	return 0;
124	160	}

samples/bpf/spintest_kern.c+18−18

@@ -12,25 +12,25 @@ · 11 unmodified lines
12	12	#include <bpf/bpf_helpers.h>
13	13	#include <bpf/bpf_tracing.h>
14	14
15		−struct bpf_map_def SEC("maps") my_map = {
16		− .type = BPF_MAP_TYPE_HASH,
17		− .key_size = sizeof(long),
18		− .value_size = sizeof(long),
19		− .max_entries = 1024,
20		−};
21		−struct bpf_map_def SEC("maps") my_map2 = {
22		− .type = BPF_MAP_TYPE_PERCPU_HASH,
23		− .key_size = sizeof(long),
24		− .value_size = sizeof(long),
25		− .max_entries = 1024,
26		−};
	15	+struct {
	16	+ __uint(type, BPF_MAP_TYPE_HASH);
	17	+ __type(key, long);
	18	+ __type(value, long);
	19	+ __uint(max_entries, 1024);
	20	+} my_map SEC(".maps");
	21	+struct {
	22	+ __uint(type, BPF_MAP_TYPE_PERCPU_HASH);
	23	+ __uint(key_size, sizeof(long));
	24	+ __uint(value_size, sizeof(long));
	25	+ __uint(max_entries, 1024);
	26	+} my_map2 SEC(".maps");
27	27
28		−struct bpf_map_def SEC("maps") stackmap = {
29		− .type = BPF_MAP_TYPE_STACK_TRACE,
30		− .key_size = sizeof(u32),
31		− .value_size = PERF_MAX_STACK_DEPTH * sizeof(u64),
32		− .max_entries = 10000,
33		−};
	28	+struct {
	29	+ __uint(type, BPF_MAP_TYPE_STACK_TRACE);
	30	+ __uint(key_size, sizeof(u32));
	31	+ __uint(value_size, PERF_MAX_STACK_DEPTH * sizeof(u64));
	32	+ __uint(max_entries, 10000);
	33	+} stackmap SEC(".maps");
34	34
35	35	#define PROG(foo) \
36	36	int foo(struct pt_regs *ctx) \

samples/bpf/spintest_user.c+55−13

@@ -1,40 +1,77 @@
1	1	// SPDX-License-Identifier: GPL-2.0
2	2	#include <stdio.h>
3	3	#include <unistd.h>
4		−#include <linux/bpf.h>
5	4	#include <string.h>
6	5	#include <assert.h>
7	6	#include <sys/resource.h>
8	7	#include <bpf/libbpf.h>
9		−#include "bpf_load.h"
	8	+#include <bpf/bpf.h>
10	9	#include "trace_helpers.h"
11	10
12	11	int main(int ac, char **argv)
13	12	{
14	13	struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
	14	+ char filename[256], symbol[256];
	15	+ struct bpf_object *obj = NULL;
	16	+ struct bpf_link *links[20];
15	17	long key, next_key, value;
16		− char filename[256];
	18	+ struct bpf_program *prog;
	19	+ int map_fd, i, j = 0;
	20	+ const char *title;
17	21	struct ksym *sym;
18		− int i;
19	22
20		− snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
21		− setrlimit(RLIMIT_MEMLOCK, &r);
	23	+ if (setrlimit(RLIMIT_MEMLOCK, &r)) {
	24	+ perror("setrlimit(RLIMIT_MEMLOCK)");
	25	+ return 1;
	26	+ }
22	27
23	28	if (load_kallsyms()) {
24	29	printf("failed to process /proc/kallsyms\n");
25	30	return 2;
26	31	}
27	32
28		− if (load_bpf_file(filename)) {
29		− printf("%s", bpf_log_buf);
30		− return 1;
	33	+ snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
	34	+ obj = bpf_object__open_file(filename, NULL);
	35	+ if (libbpf_get_error(obj)) {
	36	+ fprintf(stderr, "ERROR: opening BPF object file failed\n");
	37	+ obj = NULL;
	38	+ goto cleanup;
	39	+ }
	40	+
	41	+ /* load BPF program */
	42	+ if (bpf_object__load(obj)) {
	43	+ fprintf(stderr, "ERROR: loading BPF object file failed\n");
	44	+ goto cleanup;
	45	+ }
	46	+
	47	+ map_fd = bpf_object__find_map_fd_by_name(obj, "my_map");
	48	+ if (map_fd < 0) {
	49	+ fprintf(stderr, "ERROR: finding a map in obj file failed\n");
	50	+ goto cleanup;
	51	+ }
	52	+
	53	+ bpf_object__for_each_program(prog, obj) {
	54	+ title = bpf_program__title(prog, false);
	55	+ if (sscanf(title, "kprobe/%s", symbol) != 1)
	56	+ continue;
	57	+
	58	+ /* Attach prog only when symbol exists */
	59	+ if (ksym_get_addr(symbol)) {
	60	+ links[j] = bpf_program__attach(prog);
	61	+ if (libbpf_get_error(links[j])) {
	62	+ fprintf(stderr, "bpf_program__attach failed\n");
	63	+ links[j] = NULL;
	64	+ goto cleanup;
	65	+ }
	66	+ j++;
	67	+ }
31	68	}
32	69
33	70	for (i = 0; i < 5; i++) {
34	71	key = 0;
35	72	printf("kprobing funcs:");
36		− while (bpf_map_get_next_key(map_fd[0], &key, &next_key) == 0) {
37		− bpf_map_lookup_elem(map_fd[0], &next_key, &value);
	73	+ while (bpf_map_get_next_key(map_fd, &key, &next_key) == 0) {
	74	+ bpf_map_lookup_elem(map_fd, &next_key, &value);
38	75	assert(next_key == value);
39	76	sym = ksym_search(value);
40	77	key = next_key;
@@ -48,10 +85,15 @@ int main(int ac, char **argv) · 7 unmodified lines
48	85	if (key)
49	86	printf("\n");
50	87	key = 0;
51		− while (bpf_map_get_next_key(map_fd[0], &key, &next_key) == 0)
52		− bpf_map_delete_elem(map_fd[0], &next_key);
	88	+ while (bpf_map_get_next_key(map_fd, &key, &next_key) == 0)
	89	+ bpf_map_delete_elem(map_fd, &next_key);
53	90	sleep(1);
54	91	}
55	92
	93	+cleanup:
	94	+ for (j--; j >= 0; j--)
	95	+ bpf_link__destroy(links[j]);
	96	+
	97	+ bpf_object__close(obj);
56	98	return 0;
57	99	}

samples/bpf/syscall_tp_kern.c+12−12

@@ -18,19 +18,19 @@ struct syscalls_exit_open_args { · 17 unmodified lines
18	18	long ret;
19	19	};
20	20
21		−struct bpf_map_def SEC("maps") enter_open_map = {
22		− .type = BPF_MAP_TYPE_ARRAY,
23		− .key_size = sizeof(u32),
24		− .value_size = sizeof(u32),
25		− .max_entries = 1,
26		−};
	21	+struct {
	22	+ __uint(type, BPF_MAP_TYPE_ARRAY);
	23	+ __type(key, u32);
	24	+ __type(value, u32);
	25	+ __uint(max_entries, 1);
	26	+} enter_open_map SEC(".maps");
27	27
28		−struct bpf_map_def SEC("maps") exit_open_map = {
29		− .type = BPF_MAP_TYPE_ARRAY,
30		− .key_size = sizeof(u32),
31		− .value_size = sizeof(u32),
32		− .max_entries = 1,
33		−};
	28	+struct {
	29	+ __uint(type, BPF_MAP_TYPE_ARRAY);
	30	+ __type(key, u32);
	31	+ __type(value, u32);
	32	+ __uint(max_entries, 1);
	33	+} exit_open_map SEC(".maps");
34	34
35	35	static __always_inline void count(void *map)
36	36	{

samples/bpf/syscall_tp_user.c+42−12

@@ -5,16 +5,12 @@ · 4 unmodified lines
5	5	#include <unistd.h>
6	6	#include <fcntl.h>
7	7	#include <stdlib.h>
8		−#include <signal.h>
9		−#include <linux/bpf.h>
10	8	#include <string.h>
11	9	#include <linux/perf_event.h>
12	10	#include <errno.h>
13		−#include <assert.h>
14		−#include <stdbool.h>
15	11	#include <sys/resource.h>
	12	+#include <bpf/libbpf.h>
16	13	#include <bpf/bpf.h>
17		−#include "bpf_load.h"
18	14
19	15	/* This program verifies bpf attachment to tracepoint sys_enter_* and sys_exit_*.
20	16	* This requires kernel CONFIG_FTRACE_SYSCALLS to be set.
@@ -49,16 +45,44 @@ static void verify_map(int map_id) · 28 unmodified lines
49	45
50	46	static int test(char *filename, int num_progs)
51	47	{
52		− int i, fd, map0_fds[num_progs], map1_fds[num_progs];
	48	+ int map0_fds[num_progs], map1_fds[num_progs], fd, i, j = 0;
	49	+ struct bpf_link links[num_progs 4];
	50	+ struct bpf_object *objs[num_progs];
	51	+ struct bpf_program *prog;
53	52
54	53	for (i = 0; i < num_progs; i++) {
55		− if (load_bpf_file(filename)) {
56		− fprintf(stderr, "%s", bpf_log_buf);
57		− return 1;
	54	+ objs[i] = bpf_object__open_file(filename, NULL);
	55	+ if (libbpf_get_error(objs[i])) {
	56	+ fprintf(stderr, "opening BPF object file failed\n");
	57	+ objs[i] = NULL;
	58	+ goto cleanup;
58	59	}
59		− printf("prog #%d: map ids %d %d\n", i, map_fd[0], map_fd[1]);
60		− map0_fds[i] = map_fd[0];
61		− map1_fds[i] = map_fd[1];
	60	+
	61	+ /* load BPF program */
	62	+ if (bpf_object__load(objs[i])) {
	63	+ fprintf(stderr, "loading BPF object file failed\n");
	64	+ goto cleanup;
	65	+ }
	66	+
	67	+ map0_fds[i] = bpf_object__find_map_fd_by_name(objs[i],
	68	+ "enter_open_map");
	69	+ map1_fds[i] = bpf_object__find_map_fd_by_name(objs[i],
	70	+ "exit_open_map");
	71	+ if (map0_fds[i] < 0 \|\| map1_fds[i] < 0) {
	72	+ fprintf(stderr, "finding a map in obj file failed\n");
	73	+ goto cleanup;
	74	+ }
	75	+
	76	+ bpf_object__for_each_program(prog, objs[i]) {
	77	+ links[j] = bpf_program__attach(prog);
	78	+ if (libbpf_get_error(links[j])) {
	79	+ fprintf(stderr, "bpf_program__attach failed\n");
	80	+ links[j] = NULL;
	81	+ goto cleanup;
	82	+ }
	83	+ j++;
	84	+ }
	85	+ printf("prog #%d: map ids %d %d\n", i, map0_fds[i], map1_fds[i]);
62	86	}
63	87
64	88	/* current load_bpf_file has perf_event_open default pid = -1
@@ -80,6 +104,12 @@ static int test(char *filename, int num_progs) · 15 unmodified lines
80	104	verify_map(map1_fds[i]);
81	105	}
82	106
	107	+cleanup:
	108	+ for (j--; j >= 0; j--)
	109	+ bpf_link__destroy(links[j]);
	110	+
	111	+ for (i--; i >= 0; i--)
	112	+ bpf_object__close(objs[i]);
83	113	return 0;
84	114	}
85	115

samples/bpf/task_fd_query_kern.c+1−1

@@ -10,7 +10,7 @@ int bpf_prog1(struct pt_regs *ctx) · 9 unmodified lines
10	10	return 0;
11	11	}
12	12
13		−SEC("kretprobe/blk_account_io_completion")
	13	+SEC("kretprobe/blk_account_io_done")
14	14	int bpf_prog2(struct pt_regs *ctx)
15	15	{
16	16	return 0;

samples/bpf/task_fd_query_user.c+1−1

@@ -314,7 +314,7 @@ int main(int argc, char **argv) · 313 unmodified lines
314	314	/* test two functions in the corresponding _kern.c file /
315	315	CHECK_AND_RET(test_debug_fs_kprobe(0, "blk_mq_start_request",
316	316	BPF_FD_TYPE_KPROBE));
317		− CHECK_AND_RET(test_debug_fs_kprobe(1, "blk_account_io_completion",
	317	+ CHECK_AND_RET(test_debug_fs_kprobe(1, "blk_account_io_done",
318	318	BPF_FD_TYPE_KRETPROBE));
319	319
320	320	/* test nondebug fs kprobe */

samples/bpf/test_current_task_under_cgroup_kern.c+14−13

@@ -10,23 +10,24 @@ · 9 unmodified lines
10	10	#include <linux/version.h>
11	11	#include <bpf/bpf_helpers.h>
12	12	#include <uapi/linux/utsname.h>
	13	+#include "trace_common.h"
13	14
14		−struct bpf_map_def SEC("maps") cgroup_map = {
15		− .type = BPF_MAP_TYPE_CGROUP_ARRAY,
16		− .key_size = sizeof(u32),
17		− .value_size = sizeof(u32),
18		− .max_entries = 1,
19		−};
	15	+struct {
	16	+ __uint(type, BPF_MAP_TYPE_CGROUP_ARRAY);
	17	+ __uint(key_size, sizeof(u32));
	18	+ __uint(value_size, sizeof(u32));
	19	+ __uint(max_entries, 1);
	20	+} cgroup_map SEC(".maps");
20	21
21		−struct bpf_map_def SEC("maps") perf_map = {
22		− .type = BPF_MAP_TYPE_ARRAY,
23		− .key_size = sizeof(u32),
24		− .value_size = sizeof(u64),
25		− .max_entries = 1,
26		−};
	22	+struct {
	23	+ __uint(type, BPF_MAP_TYPE_ARRAY);
	24	+ __type(key, u32);
	25	+ __type(value, u64);
	26	+ __uint(max_entries, 1);
	27	+} perf_map SEC(".maps");
27	28
28	29	/* Writes the last PID that called sync to a map at index 0 */
29		−SEC("kprobe/sys_sync")
	30	+SEC("kprobe/" SYSCALL(sys_sync))
30	31	int bpf_prog1(struct pt_regs *ctx)
31	32	{
32	33	u64 pid = bpf_get_current_pid_tgid();

samples/bpf/test_current_task_under_cgroup_user.c+42−10

@@ -4,10 +4,9 @@ · 3 unmodified lines
4	4
5	5	#define _GNU_SOURCE
6	6	#include <stdio.h>
7		−#include <linux/bpf.h>
8	7	#include <unistd.h>
9	8	#include <bpf/bpf.h>
10		−#include "bpf_load.h"
	9	+#include <bpf/libbpf.h>
11	10	#include "cgroup_helpers.h"
12	11
13	12	#define CGROUP_PATH "/my-cgroup"
@@ -15,13 +14,44 @@ · 1 unmodified line
15	14	int main(int argc, char **argv)
16	15	{
17	16	pid_t remote_pid, local_pid = getpid();
18		− int cg2, idx = 0, rc = 0;
	17	+ struct bpf_link *link = NULL;
	18	+ struct bpf_program *prog;
	19	+ int cg2, idx = 0, rc = 1;
	20	+ struct bpf_object *obj;
19	21	char filename[256];
	22	+ int map_fd[2];
20	23
21	24	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
22		− if (load_bpf_file(filename)) {
23		− printf("%s", bpf_log_buf);
24		− return 1;
	25	+ obj = bpf_object__open_file(filename, NULL);
	26	+ if (libbpf_get_error(obj)) {
	27	+ fprintf(stderr, "ERROR: opening BPF object file failed\n");
	28	+ return 0;
	29	+ }
	30	+
	31	+ prog = bpf_object__find_program_by_name(obj, "bpf_prog1");
	32	+ if (!prog) {
	33	+ printf("finding a prog in obj file failed\n");
	34	+ goto cleanup;
	35	+ }
	36	+
	37	+ /* load BPF program */
	38	+ if (bpf_object__load(obj)) {
	39	+ fprintf(stderr, "ERROR: loading BPF object file failed\n");
	40	+ goto cleanup;
	41	+ }
	42	+
	43	+ map_fd[0] = bpf_object__find_map_fd_by_name(obj, "cgroup_map");
	44	+ map_fd[1] = bpf_object__find_map_fd_by_name(obj, "perf_map");
	45	+ if (map_fd[0] < 0 \|\| map_fd[1] < 0) {
	46	+ fprintf(stderr, "ERROR: finding a map in obj file failed\n");
	47	+ goto cleanup;
	48	+ }
	49	+
	50	+ link = bpf_program__attach(prog);
	51	+ if (libbpf_get_error(link)) {
	52	+ fprintf(stderr, "ERROR: bpf_program__attach failed\n");
	53	+ link = NULL;
	54	+ goto cleanup;
25	55	}
26	56
27	57	if (setup_cgroup_environment())
@@ -70,12 +100,14 @@ int main(int argc, char **argv) · 42 unmodified lines
70	100	goto err;
71	101	}
72	102
73		− goto out;
74		−err:
75		− rc = 1;
	103	+ rc = 0;
76	104
77		−out:
	105	+err:
78	106	close(cg2);
79	107	cleanup_cgroup_environment();
	108	+
	109	+cleanup:
	110	+ bpf_link__destroy(link);
	111	+ bpf_object__close(obj);
80	112	return rc;
81	113	}

samples/bpf/test_probe_write_user_kern.c+6−6

@@ -13,12 +13,12 @@ · 12 unmodified lines
13	13	#include <bpf/bpf_core_read.h>
14	14	#include "trace_common.h"
15	15
16		−struct bpf_map_def SEC("maps") dnat_map = {
17		− .type = BPF_MAP_TYPE_HASH,
18		− .key_size = sizeof(struct sockaddr_in),
19		− .value_size = sizeof(struct sockaddr_in),
20		− .max_entries = 256,
21		−};
	16	+struct {
	17	+ __uint(type, BPF_MAP_TYPE_HASH);
	18	+ __type(key, struct sockaddr_in);
	19	+ __type(value, struct sockaddr_in);
	20	+ __uint(max_entries, 256);
	21	+} dnat_map SEC(".maps");
22	22
23	23	/* kprobe is NOT a stable ABI
24	24	* kernel functions can be removed, renamed or completely change semantics.

samples/bpf/test_probe_write_user_user.c+39−10

@@ -1,21 +1,22 @@
1	1	// SPDX-License-Identifier: GPL-2.0
2	2	#include <stdio.h>
3	3	#include <assert.h>
4		−#include <linux/bpf.h>
5	4	#include <unistd.h>
6	5	#include <bpf/bpf.h>
7		−#include "bpf_load.h"
	6	+#include <bpf/libbpf.h>
8	7	#include <sys/socket.h>
9		−#include <string.h>
10	8	#include <netinet/in.h>
11	9	#include <arpa/inet.h>
12	10
13	11	int main(int ac, char **argv)
14	12	{
15		− int serverfd, serverconnfd, clientfd;
16		− socklen_t sockaddr_len;
17		− struct sockaddr serv_addr, mapped_addr, tmp_addr;
18	13	struct sockaddr_in serv_addr_in, mapped_addr_in, *tmp_addr_in;
	14	+ struct sockaddr serv_addr, mapped_addr, tmp_addr;
	15	+ int serverfd, serverconnfd, clientfd, map_fd;
	16	+ struct bpf_link *link = NULL;
	17	+ struct bpf_program *prog;
	18	+ struct bpf_object *obj;
	19	+ socklen_t sockaddr_len;
19	20	char filename[256];
20	21	char *ip;
21	22
@@ -24,10 +25,35 @@ int main(int ac, char **argv) · 2 unmodified lines
24	25	tmp_addr_in = (struct sockaddr_in *)&tmp_addr;
25	26
26	27	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
	28	+ obj = bpf_object__open_file(filename, NULL);
	29	+ if (libbpf_get_error(obj)) {
	30	+ fprintf(stderr, "ERROR: opening BPF object file failed\n");
	31	+ return 0;
	32	+ }
	33	+
	34	+ prog = bpf_object__find_program_by_name(obj, "bpf_prog1");
	35	+ if (libbpf_get_error(prog)) {
	36	+ fprintf(stderr, "ERROR: finding a prog in obj file failed\n");
	37	+ goto cleanup;
	38	+ }
	39	+
	40	+ /* load BPF program */
	41	+ if (bpf_object__load(obj)) {
	42	+ fprintf(stderr, "ERROR: loading BPF object file failed\n");
	43	+ goto cleanup;
	44	+ }
	45	+
	46	+ map_fd = bpf_object__find_map_fd_by_name(obj, "dnat_map");
	47	+ if (map_fd < 0) {
	48	+ fprintf(stderr, "ERROR: finding a map in obj file failed\n");
	49	+ goto cleanup;
	50	+ }
27	51
28		− if (load_bpf_file(filename)) {
29		− printf("%s", bpf_log_buf);
30		− return 1;
	52	+ link = bpf_program__attach(prog);
	53	+ if (libbpf_get_error(link)) {
	54	+ fprintf(stderr, "ERROR: bpf_program__attach failed\n");
	55	+ link = NULL;
	56	+ goto cleanup;
31	57	}
32	58
33	59	assert((serverfd = socket(AF_INET, SOCK_STREAM, 0)) > 0);
@@ -51,7 +77,7 @@ int main(int ac, char **argv) · 17 unmodified lines
51	77	mapped_addr_in->sin_port = htons(5555);
52	78	mapped_addr_in->sin_addr.s_addr = inet_addr("255.255.255.255");
53	79
54		− assert(!bpf_map_update_elem(map_fd[0], &mapped_addr, &serv_addr, BPF_ANY));
	80	+ assert(!bpf_map_update_elem(map_fd, &mapped_addr, &serv_addr, BPF_ANY));
55	81
56	82	assert(listen(serverfd, 5) == 0);
57	83
@@ -75,5 +101,8 @@ int main(int ac, char **argv) · 17 unmodified lines
75	101	/* Is the server's getsockname = the socket getpeername */
76	102	assert(memcmp(&serv_addr, &tmp_addr, sizeof(struct sockaddr_in)) == 0);
77	103
	104	+cleanup:
	105	+ bpf_link__destroy(link);
	106	+ bpf_object__close(obj);
78	107	return 0;
79	108	}

samples/bpf/trace_output_kern.c+8−7

@@ -2,15 +2,16 @@ · 1 unmodified line
2	2	#include <linux/version.h>
3	3	#include <uapi/linux/bpf.h>
4	4	#include <bpf/bpf_helpers.h>
	5	+#include "trace_common.h"
5	6
6		−struct bpf_map_def SEC("maps") my_map = {
7		− .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
8		− .key_size = sizeof(int),
9		− .value_size = sizeof(u32),
10		− .max_entries = 2,
11		−};
	7	+struct {
	8	+ __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
	9	+ __uint(key_size, sizeof(int));
	10	+ __uint(value_size, sizeof(u32));
	11	+ __uint(max_entries, 2);
	12	+} my_map SEC(".maps");
12	13
13		−SEC("kprobe/sys_write")
	14	+SEC("kprobe/" SYSCALL(sys_write))
14	15	int bpf_prog1(struct pt_regs *ctx)
15	16	{
16	17	struct S {

samples/bpf/trace_output_user.c+37−18

@@ -1,23 +1,10 @@
1	1	// SPDX-License-Identifier: GPL-2.0-only
2	2	#include <stdio.h>
3		−#include <unistd.h>
4		−#include <stdlib.h>
5		−#include <stdbool.h>
6		−#include <string.h>
7	3	#include <fcntl.h>
8	4	#include <poll.h>
9		−#include <linux/perf_event.h>
10		−#include <linux/bpf.h>
11		−#include <errno.h>
12		−#include <assert.h>
13		−#include <sys/syscall.h>
14		−#include <sys/ioctl.h>
15		−#include <sys/mman.h>
16	5	#include <time.h>
17	6	#include <signal.h>
18	7	#include <bpf/libbpf.h>
19		−#include "bpf_load.h"
20		−#include "perf-sys.h"
21	8
22	9	static __u64 time_get_ns(void)
23	10	{
@@ -57,20 +44,48 @@ static void print_bpf_output(void ctx, int cpu, void data, __u32 size) · 33 unmodified lines
57	44	int main(int argc, char **argv)
58	45	{
59	46	struct perf_buffer_opts pb_opts = {};
	47	+ struct bpf_link *link = NULL;
	48	+ struct bpf_program *prog;
60	49	struct perf_buffer *pb;
	50	+ struct bpf_object *obj;
	51	+ int map_fd, ret = 0;
61	52	char filename[256];
62	53	FILE *f;
63		− int ret;
64	54
65	55	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
	56	+ obj = bpf_object__open_file(filename, NULL);
	57	+ if (libbpf_get_error(obj)) {
	58	+ fprintf(stderr, "ERROR: opening BPF object file failed\n");
	59	+ return 0;
	60	+ }
66	61
67		− if (load_bpf_file(filename)) {
68		− printf("%s", bpf_log_buf);
69		− return 1;
	62	+ /* load BPF program */
	63	+ if (bpf_object__load(obj)) {
	64	+ fprintf(stderr, "ERROR: loading BPF object file failed\n");
	65	+ goto cleanup;
	66	+ }
	67	+
	68	+ map_fd = bpf_object__find_map_fd_by_name(obj, "my_map");
	69	+ if (map_fd < 0) {
	70	+ fprintf(stderr, "ERROR: finding a map in obj file failed\n");
	71	+ goto cleanup;
	72	+ }
	73	+
	74	+ prog = bpf_object__find_program_by_name(obj, "bpf_prog1");
	75	+ if (libbpf_get_error(prog)) {
	76	+ fprintf(stderr, "ERROR: finding a prog in obj file failed\n");
	77	+ goto cleanup;
	78	+ }
	79	+
	80	+ link = bpf_program__attach(prog);
	81	+ if (libbpf_get_error(link)) {
	82	+ fprintf(stderr, "ERROR: bpf_program__attach failed\n");
	83	+ link = NULL;
	84	+ goto cleanup;
70	85	}
71	86
72	87	pb_opts.sample_cb = print_bpf_output;
73		− pb = perf_buffer__new(map_fd[0], 8, &pb_opts);
	88	+ pb = perf_buffer__new(map_fd, 8, &pb_opts);
74	89	ret = libbpf_get_error(pb);
75	90	if (ret) {
76	91	printf("failed to setup perf_buffer: %d\n", ret);
@@ -84,5 +99,9 @@ int main(int argc, char **argv) · 7 unmodified lines
84	99	while ((ret = perf_buffer__poll(pb, 1000)) >= 0 && cnt < MAX_CNT) {
85	100	}
86	101	kill(0, SIGINT);
	102	+
	103	+cleanup:
	104	+ bpf_link__destroy(link);
	105	+ bpf_object__close(obj);
87	106	return ret;
88	107	}

samples/bpf/tracex3_kern.c+1−1

@@ -49,7 +49,7 @@ struct { · 48 unmodified lines
49	49	__uint(max_entries, SLOTS);
50	50	} lat_map SEC(".maps");
51	51
52		−SEC("kprobe/blk_account_io_completion")
	52	+SEC("kprobe/blk_account_io_done")
53	53	int bpf_prog2(struct pt_regs *ctx)
54	54	{
55	55	long rq = PT_REGS_PARM1(ctx);

samples/bpf/xdpsock_user.c+19−13

@@ -613,7 +613,16 @@ static struct xsk_umem_info xsk_configure_umem(void buffer, u64 size) · 612 unmodified lines
613	613	{
614	614	struct xsk_umem_info *umem;
615	615	struct xsk_umem_config cfg = {
616		− .fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS,
	616	+ /* We recommend that you set the fill ring size >= HW RX ring size +
	617	+ * AF_XDP RX ring size. Make sure you fill up the fill ring
	618	+ * with buffers at regular intervals, and you will with this setting
	619	+ * avoid allocation failures in the driver. These are usually quite
	620	+ * expensive since drivers have not been written to assume that
	621	+ * allocation failures are common. For regular sockets, kernel
	622	+ * allocated memory is used that only runs out in OOM situations
	623	+ * that should be rare.
	624	+ */
	625	+ .fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS * 2,
617	626	.comp_size = XSK_RING_CONS__DEFAULT_NUM_DESCS,
618	627	.frame_size = opt_xsk_frame_size,
619	628	.frame_headroom = XSK_UMEM__DEFAULT_FRAME_HEADROOM,
@@ -640,13 +649,13 @@ static void xsk_populate_fill_ring(struct xsk_umem_info *umem) · 20 unmodified lines
640	649	u32 idx;
641	650
642	651	ret = xsk_ring_prod__reserve(&umem->fq,
643		− XSK_RING_PROD__DEFAULT_NUM_DESCS, &idx);
644		− if (ret != XSK_RING_PROD__DEFAULT_NUM_DESCS)
	652	+ XSK_RING_PROD__DEFAULT_NUM_DESCS * 2, &idx);
	653	+ if (ret != XSK_RING_PROD__DEFAULT_NUM_DESCS * 2)
645	654	exit_with_error(-ret);
646		− for (i = 0; i < XSK_RING_PROD__DEFAULT_NUM_DESCS; i++)
	655	+ for (i = 0; i < XSK_RING_PROD__DEFAULT_NUM_DESCS * 2; i++)
647	656	*xsk_ring_prod__fill_addr(&umem->fq, idx++) =
648	657	i * opt_xsk_frame_size;
649		− xsk_ring_prod__submit(&umem->fq, XSK_RING_PROD__DEFAULT_NUM_DESCS);
	658	+ xsk_ring_prod__submit(&umem->fq, XSK_RING_PROD__DEFAULT_NUM_DESCS * 2);
650	659	}
651	660
652	661	static struct xsk_socket_info xsk_configure_socket(struct xsk_umem_info umem,
@@ -888,9 +897,6 @@ static inline void complete_tx_l2fwd(struct xsk_socket_info *xsk, · 235 unmodified lines
888	897	if (!xsk->outstanding_tx)
889	898	return;
890	899
891		− if (!opt_need_wakeup \|\| xsk_ring_prod__needs_wakeup(&xsk->tx))
892		− kick_tx(xsk);
893		−
894	900	ndescs = (xsk->outstanding_tx > opt_batch_size) ? opt_batch_size :
895	901	xsk->outstanding_tx;
896	902
@@ -1004,7 +1010,7 @@ static void rx_drop_all(void) · 107 unmodified lines
1004	1010	}
1005	1011	}
1006	1012
1007		−static void tx_only(struct xsk_socket_info *xsk, u32 frame_nb, int batch_size)
	1013	+static void tx_only(struct xsk_socket_info xsk, u32 frame_nb, int batch_size)
1008	1014	{
1009	1015	u32 idx;
1010	1016	unsigned int i;
@@ -1017,14 +1023,14 @@ static void tx_only(struct xsk_socket_info *xsk, u32 frame_nb, int batch_size) · 6 unmodified lines
1017	1023	for (i = 0; i < batch_size; i++) {
1018	1024	struct xdp_desc *tx_desc = xsk_ring_prod__tx_desc(&xsk->tx,
1019	1025	idx + i);
1020		− tx_desc->addr = (frame_nb + i) << XSK_UMEM__DEFAULT_FRAME_SHIFT;
	1026	+ tx_desc->addr = (*frame_nb + i) << XSK_UMEM__DEFAULT_FRAME_SHIFT;
1021	1027	tx_desc->len = PKT_SIZE;
1022	1028	}
1023	1029
1024	1030	xsk_ring_prod__submit(&xsk->tx, batch_size);
1025	1031	xsk->outstanding_tx += batch_size;
1026		− frame_nb += batch_size;
1027		− frame_nb %= NUM_FRAMES;
	1032	+ *frame_nb += batch_size;
	1033	+ *frame_nb %= NUM_FRAMES;
1028	1034	complete_tx_only(xsk, batch_size);
1029	1035	}
1030	1036
@@ -1080,7 +1086,7 @@ static void tx_only_all(void) · 49 unmodified lines
1080	1086	}
1081	1087
1082	1088	for (i = 0; i < num_socks; i++)
1083		− tx_only(xsks[i], frame_nb[i], batch_size);
	1089	+ tx_only(xsks[i], &frame_nb[i], batch_size);
1084	1090
1085	1091	pkt_cnt += batch_size;
1086	1092

samples/bpf/xsk_fwd.c+1085−0

@@ -0,0 +1,1085 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+/* Copyright(c) 2020 Intel Corporation. */
	3	+
	4	+#define _GNU_SOURCE
	5	+#include <poll.h>
	6	+#include <pthread.h>
	7	+#include <signal.h>
	8	+#include <sched.h>
	9	+#include <stdio.h>
	10	+#include <stdlib.h>
	11	+#include <string.h>
	12	+#include <sys/mman.h>
	13	+#include <sys/resource.h>
	14	+#include <sys/socket.h>
	15	+#include <sys/types.h>
	16	+#include <time.h>
	17	+#include <unistd.h>
	18	+#include <getopt.h>
	19	+#include <netinet/ether.h>
	20	+#include <net/if.h>
	21	+
	22	+#include <linux/bpf.h>
	23	+#include <linux/if_link.h>
	24	+#include <linux/if_xdp.h>
	25	+
	26	+#include <bpf/libbpf.h>
	27	+#include <bpf/xsk.h>
	28	+#include <bpf/bpf.h>
	29	+
	30	+#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
	31	+
	32	+typedef __u64 u64;
	33	+typedef __u32 u32;
	34	+typedef __u16 u16;
	35	+typedef __u8 u8;
	36	+
	37	+/* This program illustrates the packet forwarding between multiple AF_XDP
	38	+ * sockets in multi-threaded environment. All threads are sharing a common
	39	+ * buffer pool, with each socket having its own private buffer cache.
	40	+ *
	41	+ * Example 1: Single thread handling two sockets. The packets received by socket
	42	+ * A (interface IFA, queue QA) are forwarded to socket B (interface IFB, queue
	43	+ * QB), while the packets received by socket B are forwarded to socket A. The
	44	+ * thread is running on CPU core X:
	45	+ *
	46	+ * ./xsk_fwd -i IFA -q QA -i IFB -q QB -c X
	47	+ *
	48	+ * Example 2: Two threads, each handling two sockets. The thread running on CPU
	49	+ * core X forwards all the packets received by socket A to socket B, and all the
	50	+ * packets received by socket B to socket A. The thread running on CPU core Y is
	51	+ * performing the same packet forwarding between sockets C and D:
	52	+ *
	53	+ * ./xsk_fwd -i IFA -q QA -i IFB -q QB -i IFC -q QC -i IFD -q QD
	54	+ * -c CX -c CY
	55	+ */
	56	+
	57	+/*
	58	+ * Buffer pool and buffer cache
	59	+ *
	60	+ * For packet forwarding, the packet buffers are typically allocated from the
	61	+ * pool for packet reception and freed back to the pool for further reuse once
	62	+ * the packet transmission is completed.
	63	+ *
	64	+ * The buffer pool is shared between multiple threads. In order to minimize the
	65	+ * access latency to the shared buffer pool, each thread creates one (or
	66	+ * several) buffer caches, which, unlike the buffer pool, are private to the
	67	+ * thread that creates them and therefore cannot be shared with other threads.
	68	+ * The access to the shared pool is only needed either (A) when the cache gets
	69	+ * empty due to repeated buffer allocations and it needs to be replenished from
	70	+ * the pool, or (B) when the cache gets full due to repeated buffer free and it
	71	+ * needs to be flushed back to the pull.
	72	+ *
	73	+ * In a packet forwarding system, a packet received on any input port can
	74	+ * potentially be transmitted on any output port, depending on the forwarding
	75	+ * configuration. For AF_XDP sockets, for this to work with zero-copy of the
	76	+ * packet buffers when, it is required that the buffer pool memory fits into the
	77	+ * UMEM area shared by all the sockets.
	78	+ */
	79	+
	80	+struct bpool_params {
	81	+ u32 n_buffers;
	82	+ u32 buffer_size;
	83	+ int mmap_flags;
	84	+
	85	+ u32 n_users_max;
	86	+ u32 n_buffers_per_slab;
	87	+};
	88	+
	89	+/* This buffer pool implementation organizes the buffers into equally sized
	90	+ * slabs of n_buffers_per_slab. Initially, there are n_slabs slabs in the
	91	+ * pool that are completely filled with buffer pointers (full slabs).
	92	+ *
	93	+ * Each buffer cache has a slab for buffer allocation and a slab for buffer
	94	+ * free, with both of these slabs initially empty. When the cache's allocation
	95	+ * slab goes empty, it is swapped with one of the available full slabs from the
	96	+ * pool, if any is available. When the cache's free slab goes full, it is
	97	+ * swapped for one of the empty slabs from the pool, which is guaranteed to
	98	+ * succeed.
	99	+ *
	100	+ * Partially filled slabs never get traded between the cache and the pool
	101	+ * (except when the cache itself is destroyed), which enables fast operation
	102	+ * through pointer swapping.
	103	+ */
	104	+struct bpool {
	105	+ struct bpool_params params;
	106	+ pthread_mutex_t lock;
	107	+ void *addr;
	108	+
	109	+ u64 **slabs;
	110	+ u64 **slabs_reserved;
	111	+ u64 *buffers;
	112	+ u64 *buffers_reserved;
	113	+
	114	+ u64 n_slabs;
	115	+ u64 n_slabs_reserved;
	116	+ u64 n_buffers;
	117	+
	118	+ u64 n_slabs_available;
	119	+ u64 n_slabs_reserved_available;
	120	+
	121	+ struct xsk_umem_config umem_cfg;
	122	+ struct xsk_ring_prod umem_fq;
	123	+ struct xsk_ring_cons umem_cq;
	124	+ struct xsk_umem *umem;
	125	+};
	126	+
	127	+static struct bpool *
	128	+bpool_init(struct bpool_params *params,
	129	+ struct xsk_umem_config *umem_cfg)
	130	+{
	131	+ struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
	132	+ u64 n_slabs, n_slabs_reserved, n_buffers, n_buffers_reserved;
	133	+ u64 slabs_size, slabs_reserved_size;
	134	+ u64 buffers_size, buffers_reserved_size;
	135	+ u64 total_size, i;
	136	+ struct bpool *bp;
	137	+ u8 *p;
	138	+ int status;
	139	+
	140	+ /* mmap prep. */
	141	+ if (setrlimit(RLIMIT_MEMLOCK, &r))
	142	+ return NULL;
	143	+
	144	+ /* bpool internals dimensioning. */
	145	+ n_slabs = (params->n_buffers + params->n_buffers_per_slab - 1) /
	146	+ params->n_buffers_per_slab;
	147	+ n_slabs_reserved = params->n_users_max * 2;
	148	+ n_buffers = n_slabs * params->n_buffers_per_slab;
	149	+ n_buffers_reserved = n_slabs_reserved * params->n_buffers_per_slab;
	150	+
	151	+ slabs_size = n_slabs * sizeof(u64 *);
	152	+ slabs_reserved_size = n_slabs_reserved * sizeof(u64 *);
	153	+ buffers_size = n_buffers * sizeof(u64);
	154	+ buffers_reserved_size = n_buffers_reserved * sizeof(u64);
	155	+
	156	+ total_size = sizeof(struct bpool) +
	157	+ slabs_size + slabs_reserved_size +
	158	+ buffers_size + buffers_reserved_size;
	159	+
	160	+ /* bpool memory allocation. */
	161	+ p = calloc(total_size, sizeof(u8));
	162	+ if (!p)
	163	+ return NULL;
	164	+
	165	+ /* bpool memory initialization. */
	166	+ bp = (struct bpool *)p;
	167	+ memcpy(&bp->params, params, sizeof(*params));
	168	+ bp->params.n_buffers = n_buffers;
	169	+
	170	+ bp->slabs = (u64 **)&p[sizeof(struct bpool)];
	171	+ bp->slabs_reserved = (u64 **)&p[sizeof(struct bpool) +
	172	+ slabs_size];
	173	+ bp->buffers = (u64 *)&p[sizeof(struct bpool) +
	174	+ slabs_size + slabs_reserved_size];
	175	+ bp->buffers_reserved = (u64 *)&p[sizeof(struct bpool) +
	176	+ slabs_size + slabs_reserved_size + buffers_size];
	177	+
	178	+ bp->n_slabs = n_slabs;
	179	+ bp->n_slabs_reserved = n_slabs_reserved;
	180	+ bp->n_buffers = n_buffers;
	181	+
	182	+ for (i = 0; i < n_slabs; i++)
	183	+ bp->slabs[i] = &bp->buffers[i * params->n_buffers_per_slab];
	184	+ bp->n_slabs_available = n_slabs;
	185	+
	186	+ for (i = 0; i < n_slabs_reserved; i++)
	187	+ bp->slabs_reserved[i] = &bp->buffers_reserved[i *
	188	+ params->n_buffers_per_slab];
	189	+ bp->n_slabs_reserved_available = n_slabs_reserved;
	190	+
	191	+ for (i = 0; i < n_buffers; i++)
	192	+ bp->buffers[i] = i * params->buffer_size;
	193	+
	194	+ /* lock. */
	195	+ status = pthread_mutex_init(&bp->lock, NULL);
	196	+ if (status) {
	197	+ free(p);
	198	+ return NULL;
	199	+ }
	200	+
	201	+ /* mmap. */
	202	+ bp->addr = mmap(NULL,
	203	+ n_buffers * params->buffer_size,
	204	+ PROT_READ \| PROT_WRITE,
	205	+ MAP_PRIVATE \| MAP_ANONYMOUS \| params->mmap_flags,
	206	+ -1,
	207	+ 0);
	208	+ if (bp->addr == MAP_FAILED) {
	209	+ pthread_mutex_destroy(&bp->lock);
	210	+ free(p);
	211	+ return NULL;
	212	+ }
	213	+
	214	+ /* umem. */
	215	+ status = xsk_umem__create(&bp->umem,
	216	+ bp->addr,
	217	+ bp->params.n_buffers * bp->params.buffer_size,
	218	+ &bp->umem_fq,
	219	+ &bp->umem_cq,
	220	+ umem_cfg);
	221	+ if (status) {
	222	+ munmap(bp->addr, bp->params.n_buffers * bp->params.buffer_size);
	223	+ pthread_mutex_destroy(&bp->lock);
	224	+ free(p);
	225	+ return NULL;
	226	+ }
	227	+ memcpy(&bp->umem_cfg, umem_cfg, sizeof(*umem_cfg));
	228	+
	229	+ return bp;
	230	+}
	231	+
	232	+static void
	233	+bpool_free(struct bpool *bp)
	234	+{
	235	+ if (!bp)
	236	+ return;
	237	+
	238	+ xsk_umem__delete(bp->umem);
	239	+ munmap(bp->addr, bp->params.n_buffers * bp->params.buffer_size);
	240	+ pthread_mutex_destroy(&bp->lock);
	241	+ free(bp);
	242	+}
	243	+
	244	+struct bcache {
	245	+ struct bpool *bp;
	246	+
	247	+ u64 *slab_cons;
	248	+ u64 *slab_prod;
	249	+
	250	+ u64 n_buffers_cons;
	251	+ u64 n_buffers_prod;
	252	+};
	253	+
	254	+static u32
	255	+bcache_slab_size(struct bcache *bc)
	256	+{
	257	+ struct bpool *bp = bc->bp;
	258	+
	259	+ return bp->params.n_buffers_per_slab;
	260	+}
	261	+
	262	+static struct bcache *
	263	+bcache_init(struct bpool *bp)
	264	+{
	265	+ struct bcache *bc;
	266	+
	267	+ bc = calloc(1, sizeof(struct bcache));
	268	+ if (!bc)
	269	+ return NULL;
	270	+
	271	+ bc->bp = bp;
	272	+ bc->n_buffers_cons = 0;
	273	+ bc->n_buffers_prod = 0;
	274	+
	275	+ pthread_mutex_lock(&bp->lock);
	276	+ if (bp->n_slabs_reserved_available == 0) {
	277	+ pthread_mutex_unlock(&bp->lock);
	278	+ free(bc);
	279	+ return NULL;
	280	+ }
	281	+
	282	+ bc->slab_cons = bp->slabs_reserved[bp->n_slabs_reserved_available - 1];
	283	+ bc->slab_prod = bp->slabs_reserved[bp->n_slabs_reserved_available - 2];
	284	+ bp->n_slabs_reserved_available -= 2;
	285	+ pthread_mutex_unlock(&bp->lock);
	286	+
	287	+ return bc;
	288	+}
	289	+
	290	+static void
	291	+bcache_free(struct bcache *bc)
	292	+{
	293	+ struct bpool *bp;
	294	+
	295	+ if (!bc)
	296	+ return;
	297	+
	298	+ /* In order to keep this example simple, the case of freeing any
	299	+ * existing buffers from the cache back to the pool is ignored.
	300	+ */
	301	+
	302	+ bp = bc->bp;
	303	+ pthread_mutex_lock(&bp->lock);
	304	+ bp->slabs_reserved[bp->n_slabs_reserved_available] = bc->slab_prod;
	305	+ bp->slabs_reserved[bp->n_slabs_reserved_available + 1] = bc->slab_cons;
	306	+ bp->n_slabs_reserved_available += 2;
	307	+ pthread_mutex_unlock(&bp->lock);
	308	+
	309	+ free(bc);
	310	+}
	311	+
	312	+/* To work correctly, the implementation requires that the n_buffers input
	313	+ * argument is never greater than the buffer pool's n_buffers_per_slab. This
	314	+ * is typically the case, with one exception taking place when large number of
	315	+ * buffers are allocated at init time (e.g. for the UMEM fill queue setup).
	316	+ */
	317	+static inline u32
	318	+bcache_cons_check(struct bcache *bc, u32 n_buffers)
	319	+{
	320	+ struct bpool *bp = bc->bp;
	321	+ u64 n_buffers_per_slab = bp->params.n_buffers_per_slab;
	322	+ u64 n_buffers_cons = bc->n_buffers_cons;
	323	+ u64 n_slabs_available;
	324	+ u64 *slab_full;
	325	+
	326	+ /*
	327	+ * Consumer slab is not empty: Use what's available locally. Do not
	328	+ * look for more buffers from the pool when the ask can only be
	329	+ * partially satisfied.
	330	+ */
	331	+ if (n_buffers_cons)
	332	+ return (n_buffers_cons < n_buffers) ?
	333	+ n_buffers_cons :
	334	+ n_buffers;
	335	+
	336	+ /*
	337	+ * Consumer slab is empty: look to trade the current consumer slab
	338	+ * (full) for a full slab from the pool, if any is available.
	339	+ */
	340	+ pthread_mutex_lock(&bp->lock);
	341	+ n_slabs_available = bp->n_slabs_available;
	342	+ if (!n_slabs_available) {
	343	+ pthread_mutex_unlock(&bp->lock);
	344	+ return 0;
	345	+ }
	346	+
	347	+ n_slabs_available--;
	348	+ slab_full = bp->slabs[n_slabs_available];
	349	+ bp->slabs[n_slabs_available] = bc->slab_cons;
	350	+ bp->n_slabs_available = n_slabs_available;
	351	+ pthread_mutex_unlock(&bp->lock);
	352	+
	353	+ bc->slab_cons = slab_full;
	354	+ bc->n_buffers_cons = n_buffers_per_slab;
	355	+ return n_buffers;
	356	+}
	357	+
	358	+static inline u64
	359	+bcache_cons(struct bcache *bc)
	360	+{
	361	+ u64 n_buffers_cons = bc->n_buffers_cons - 1;
	362	+ u64 buffer;
	363	+
	364	+ buffer = bc->slab_cons[n_buffers_cons];
	365	+ bc->n_buffers_cons = n_buffers_cons;
	366	+ return buffer;
	367	+}
	368	+
	369	+static inline void
	370	+bcache_prod(struct bcache *bc, u64 buffer)
	371	+{
	372	+ struct bpool *bp = bc->bp;
	373	+ u64 n_buffers_per_slab = bp->params.n_buffers_per_slab;
	374	+ u64 n_buffers_prod = bc->n_buffers_prod;
	375	+ u64 n_slabs_available;
	376	+ u64 *slab_empty;
	377	+
	378	+ /*
	379	+ * Producer slab is not yet full: store the current buffer to it.
	380	+ */
	381	+ if (n_buffers_prod < n_buffers_per_slab) {
	382	+ bc->slab_prod[n_buffers_prod] = buffer;
	383	+ bc->n_buffers_prod = n_buffers_prod + 1;
	384	+ return;
	385	+ }
	386	+
	387	+ /*
	388	+ * Producer slab is full: trade the cache's current producer slab
	389	+ * (full) for an empty slab from the pool, then store the current
	390	+ * buffer to the new producer slab. As one full slab exists in the
	391	+ * cache, it is guaranteed that there is at least one empty slab
	392	+ * available in the pool.
	393	+ */
	394	+ pthread_mutex_lock(&bp->lock);
	395	+ n_slabs_available = bp->n_slabs_available;
	396	+ slab_empty = bp->slabs[n_slabs_available];
	397	+ bp->slabs[n_slabs_available] = bc->slab_prod;
	398	+ bp->n_slabs_available = n_slabs_available + 1;
	399	+ pthread_mutex_unlock(&bp->lock);
	400	+
	401	+ slab_empty[0] = buffer;
	402	+ bc->slab_prod = slab_empty;
	403	+ bc->n_buffers_prod = 1;
	404	+}
	405	+
	406	+/*
	407	+ * Port
	408	+ *
	409	+ * Each of the forwarding ports sits on top of an AF_XDP socket. In order for
	410	+ * packet forwarding to happen with no packet buffer copy, all the sockets need
	411	+ * to share the same UMEM area, which is used as the buffer pool memory.
	412	+ */
	413	+#ifndef MAX_BURST_RX
	414	+#define MAX_BURST_RX 64
	415	+#endif
	416	+
	417	+#ifndef MAX_BURST_TX
	418	+#define MAX_BURST_TX 64
	419	+#endif
	420	+
	421	+struct burst_rx {
	422	+ u64 addr[MAX_BURST_RX];
	423	+ u32 len[MAX_BURST_RX];
	424	+};
	425	+
	426	+struct burst_tx {
	427	+ u64 addr[MAX_BURST_TX];
	428	+ u32 len[MAX_BURST_TX];
	429	+ u32 n_pkts;
	430	+};
	431	+
	432	+struct port_params {
	433	+ struct xsk_socket_config xsk_cfg;
	434	+ struct bpool *bp;
	435	+ const char *iface;
	436	+ u32 iface_queue;
	437	+};
	438	+
	439	+struct port {
	440	+ struct port_params params;
	441	+
	442	+ struct bcache *bc;
	443	+
	444	+ struct xsk_ring_cons rxq;
	445	+ struct xsk_ring_prod txq;
	446	+ struct xsk_ring_prod umem_fq;
	447	+ struct xsk_ring_cons umem_cq;
	448	+ struct xsk_socket *xsk;
	449	+ int umem_fq_initialized;
	450	+
	451	+ u64 n_pkts_rx;
	452	+ u64 n_pkts_tx;
	453	+};
	454	+
	455	+static void
	456	+port_free(struct port *p)
	457	+{
	458	+ if (!p)
	459	+ return;
	460	+
	461	+ /* To keep this example simple, the code to free the buffers from the
	462	+ * socket's receive and transmit queues, as well as from the UMEM fill
	463	+ * and completion queues, is not included.
	464	+ */
	465	+
	466	+ if (p->xsk)
	467	+ xsk_socket__delete(p->xsk);
	468	+
	469	+ bcache_free(p->bc);
	470	+
	471	+ free(p);
	472	+}
	473	+
	474	+static struct port *
	475	+port_init(struct port_params *params)
	476	+{
	477	+ struct port *p;
	478	+ u32 umem_fq_size, pos = 0;
	479	+ int status, i;
	480	+
	481	+ /* Memory allocation and initialization. */
	482	+ p = calloc(sizeof(struct port), 1);
	483	+ if (!p)
	484	+ return NULL;
	485	+
	486	+ memcpy(&p->params, params, sizeof(p->params));
	487	+ umem_fq_size = params->bp->umem_cfg.fill_size;
	488	+
	489	+ /* bcache. */
	490	+ p->bc = bcache_init(params->bp);
	491	+ if (!p->bc \|\|
	492	+ (bcache_slab_size(p->bc) < umem_fq_size) \|\|
	493	+ (bcache_cons_check(p->bc, umem_fq_size) < umem_fq_size)) {
	494	+ port_free(p);
	495	+ return NULL;
	496	+ }
	497	+
	498	+ /* xsk socket. */
	499	+ status = xsk_socket__create_shared(&p->xsk,
	500	+ params->iface,
	501	+ params->iface_queue,
	502	+ params->bp->umem,
	503	+ &p->rxq,
	504	+ &p->txq,
	505	+ &p->umem_fq,
	506	+ &p->umem_cq,
	507	+ &params->xsk_cfg);
	508	+ if (status) {
	509	+ port_free(p);
	510	+ return NULL;
	511	+ }
	512	+
	513	+ /* umem fq. */
	514	+ xsk_ring_prod__reserve(&p->umem_fq, umem_fq_size, &pos);
	515	+
	516	+ for (i = 0; i < umem_fq_size; i++)
	517	+ *xsk_ring_prod__fill_addr(&p->umem_fq, pos + i) =
	518	+ bcache_cons(p->bc);
	519	+
	520	+ xsk_ring_prod__submit(&p->umem_fq, umem_fq_size);
	521	+ p->umem_fq_initialized = 1;
	522	+
	523	+ return p;
	524	+}
	525	+
	526	+static inline u32
	527	+port_rx_burst(struct port p, struct burst_rx b)
	528	+{
	529	+ u32 n_pkts, pos, i;
	530	+
	531	+ /* Free buffers for FQ replenish. */
	532	+ n_pkts = ARRAY_SIZE(b->addr);
	533	+
	534	+ n_pkts = bcache_cons_check(p->bc, n_pkts);
	535	+ if (!n_pkts)
	536	+ return 0;
	537	+
	538	+ /* RXQ. */
	539	+ n_pkts = xsk_ring_cons__peek(&p->rxq, n_pkts, &pos);
	540	+ if (!n_pkts) {
	541	+ if (xsk_ring_prod__needs_wakeup(&p->umem_fq)) {
	542	+ struct pollfd pollfd = {
	543	+ .fd = xsk_socket__fd(p->xsk),
	544	+ .events = POLLIN,
	545	+ };
	546	+
	547	+ poll(&pollfd, 1, 0);
	548	+ }
	549	+ return 0;
	550	+ }
	551	+
	552	+ for (i = 0; i < n_pkts; i++) {
	553	+ b->addr[i] = xsk_ring_cons__rx_desc(&p->rxq, pos + i)->addr;
	554	+ b->len[i] = xsk_ring_cons__rx_desc(&p->rxq, pos + i)->len;
	555	+ }
	556	+
	557	+ xsk_ring_cons__release(&p->rxq, n_pkts);
	558	+ p->n_pkts_rx += n_pkts;
	559	+
	560	+ /* UMEM FQ. */
	561	+ for ( ; ; ) {
	562	+ int status;
	563	+
	564	+ status = xsk_ring_prod__reserve(&p->umem_fq, n_pkts, &pos);
	565	+ if (status == n_pkts)
	566	+ break;
	567	+
	568	+ if (xsk_ring_prod__needs_wakeup(&p->umem_fq)) {
	569	+ struct pollfd pollfd = {
	570	+ .fd = xsk_socket__fd(p->xsk),
	571	+ .events = POLLIN,
	572	+ };
	573	+
	574	+ poll(&pollfd, 1, 0);
	575	+ }
	576	+ }
	577	+
	578	+ for (i = 0; i < n_pkts; i++)
	579	+ *xsk_ring_prod__fill_addr(&p->umem_fq, pos + i) =
	580	+ bcache_cons(p->bc);
	581	+
	582	+ xsk_ring_prod__submit(&p->umem_fq, n_pkts);
	583	+
	584	+ return n_pkts;
	585	+}
	586	+
	587	+static inline void
	588	+port_tx_burst(struct port p, struct burst_tx b)
	589	+{
	590	+ u32 n_pkts, pos, i;
	591	+ int status;
	592	+
	593	+ /* UMEM CQ. */
	594	+ n_pkts = p->params.bp->umem_cfg.comp_size;
	595	+
	596	+ n_pkts = xsk_ring_cons__peek(&p->umem_cq, n_pkts, &pos);
	597	+
	598	+ for (i = 0; i < n_pkts; i++) {
	599	+ u64 addr = *xsk_ring_cons__comp_addr(&p->umem_cq, pos + i);
	600	+
	601	+ bcache_prod(p->bc, addr);
	602	+ }
	603	+
	604	+ xsk_ring_cons__release(&p->umem_cq, n_pkts);
	605	+
	606	+ /* TXQ. */
	607	+ n_pkts = b->n_pkts;
	608	+
	609	+ for ( ; ; ) {
	610	+ status = xsk_ring_prod__reserve(&p->txq, n_pkts, &pos);
	611	+ if (status == n_pkts)
	612	+ break;
	613	+
	614	+ if (xsk_ring_prod__needs_wakeup(&p->txq))
	615	+ sendto(xsk_socket__fd(p->xsk), NULL, 0, MSG_DONTWAIT,
	616	+ NULL, 0);
	617	+ }
	618	+
	619	+ for (i = 0; i < n_pkts; i++) {
	620	+ xsk_ring_prod__tx_desc(&p->txq, pos + i)->addr = b->addr[i];
	621	+ xsk_ring_prod__tx_desc(&p->txq, pos + i)->len = b->len[i];
	622	+ }
	623	+
	624	+ xsk_ring_prod__submit(&p->txq, n_pkts);
	625	+ if (xsk_ring_prod__needs_wakeup(&p->txq))
	626	+ sendto(xsk_socket__fd(p->xsk), NULL, 0, MSG_DONTWAIT, NULL, 0);
	627	+ p->n_pkts_tx += n_pkts;
	628	+}
	629	+
	630	+/*
	631	+ * Thread
	632	+ *
	633	+ * Packet forwarding threads.
	634	+ */
	635	+#ifndef MAX_PORTS_PER_THREAD
	636	+#define MAX_PORTS_PER_THREAD 16
	637	+#endif
	638	+
	639	+struct thread_data {
	640	+ struct port *ports_rx[MAX_PORTS_PER_THREAD];
	641	+ struct port *ports_tx[MAX_PORTS_PER_THREAD];
	642	+ u32 n_ports_rx;
	643	+ struct burst_rx burst_rx;
	644	+ struct burst_tx burst_tx[MAX_PORTS_PER_THREAD];
	645	+ u32 cpu_core_id;
	646	+ int quit;
	647	+};
	648	+
	649	+static void swap_mac_addresses(void *data)
	650	+{
	651	+ struct ether_header eth = (struct ether_header )data;
	652	+ struct ether_addr src_addr = (struct ether_addr )&eth->ether_shost;
	653	+ struct ether_addr dst_addr = (struct ether_addr )&eth->ether_dhost;
	654	+ struct ether_addr tmp;
	655	+
	656	+ tmp = *src_addr;
	657	+ src_addr = dst_addr;
	658	+ *dst_addr = tmp;
	659	+}
	660	+
	661	+static void *
	662	+thread_func(void *arg)
	663	+{
	664	+ struct thread_data *t = arg;
	665	+ cpu_set_t cpu_cores;
	666	+ u32 i;
	667	+
	668	+ CPU_ZERO(&cpu_cores);
	669	+ CPU_SET(t->cpu_core_id, &cpu_cores);
	670	+ pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpu_cores);
	671	+
	672	+ for (i = 0; !t->quit; i = (i + 1) & (t->n_ports_rx - 1)) {
	673	+ struct port *port_rx = t->ports_rx[i];
	674	+ struct port *port_tx = t->ports_tx[i];
	675	+ struct burst_rx *brx = &t->burst_rx;
	676	+ struct burst_tx *btx = &t->burst_tx[i];
	677	+ u32 n_pkts, j;
	678	+
	679	+ /* RX. */
	680	+ n_pkts = port_rx_burst(port_rx, brx);
	681	+ if (!n_pkts)
	682	+ continue;
	683	+
	684	+ /* Process & TX. */
	685	+ for (j = 0; j < n_pkts; j++) {
	686	+ u64 addr = xsk_umem__add_offset_to_addr(brx->addr[j]);
	687	+ u8 *pkt = xsk_umem__get_data(port_rx->params.bp->addr,
	688	+ addr);
	689	+
	690	+ swap_mac_addresses(pkt);
	691	+
	692	+ btx->addr[btx->n_pkts] = brx->addr[j];
	693	+ btx->len[btx->n_pkts] = brx->len[j];
	694	+ btx->n_pkts++;
	695	+
	696	+ if (btx->n_pkts == MAX_BURST_TX) {
	697	+ port_tx_burst(port_tx, btx);
	698	+ btx->n_pkts = 0;
	699	+ }
	700	+ }
	701	+ }
	702	+
	703	+ return NULL;
	704	+}
	705	+
	706	+/*
	707	+ * Process
	708	+ */
	709	+static const struct bpool_params bpool_params_default = {
	710	+ .n_buffers = 64 * 1024,
	711	+ .buffer_size = XSK_UMEM__DEFAULT_FRAME_SIZE,
	712	+ .mmap_flags = 0,
	713	+
	714	+ .n_users_max = 16,
	715	+ .n_buffers_per_slab = XSK_RING_PROD__DEFAULT_NUM_DESCS * 2,
	716	+};
	717	+
	718	+static const struct xsk_umem_config umem_cfg_default = {
	719	+ .fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS * 2,
	720	+ .comp_size = XSK_RING_CONS__DEFAULT_NUM_DESCS,
	721	+ .frame_size = XSK_UMEM__DEFAULT_FRAME_SIZE,
	722	+ .frame_headroom = XSK_UMEM__DEFAULT_FRAME_HEADROOM,
	723	+ .flags = 0,
	724	+};
	725	+
	726	+static const struct port_params port_params_default = {
	727	+ .xsk_cfg = {
	728	+ .rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS,
	729	+ .tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS,
	730	+ .libbpf_flags = 0,
	731	+ .xdp_flags = XDP_FLAGS_DRV_MODE,
	732	+ .bind_flags = XDP_USE_NEED_WAKEUP \| XDP_ZEROCOPY,
	733	+ },
	734	+
	735	+ .bp = NULL,
	736	+ .iface = NULL,
	737	+ .iface_queue = 0,
	738	+};
	739	+
	740	+#ifndef MAX_PORTS
	741	+#define MAX_PORTS 64
	742	+#endif
	743	+
	744	+#ifndef MAX_THREADS
	745	+#define MAX_THREADS 64
	746	+#endif
	747	+
	748	+static struct bpool_params bpool_params;
	749	+static struct xsk_umem_config umem_cfg;
	750	+static struct bpool *bp;
	751	+
	752	+static struct port_params port_params[MAX_PORTS];
	753	+static struct port *ports[MAX_PORTS];
	754	+static u64 n_pkts_rx[MAX_PORTS];
	755	+static u64 n_pkts_tx[MAX_PORTS];
	756	+static int n_ports;
	757	+
	758	+static pthread_t threads[MAX_THREADS];
	759	+static struct thread_data thread_data[MAX_THREADS];
	760	+static int n_threads;
	761	+
	762	+static void
	763	+print_usage(char *prog_name)
	764	+{
	765	+ const char *usage =
	766	+ "Usage:\n"
	767	+ "\t%s [ -b SIZE ] -c CORE -i INTERFACE [ -q QUEUE ]\n"
	768	+ "\n"
	769	+ "-c CORE CPU core to run a packet forwarding thread\n"
	770	+ " on. May be invoked multiple times.\n"
	771	+ "\n"
	772	+ "-b SIZE Number of buffers in the buffer pool shared\n"
	773	+ " by all the forwarding threads. Default: %u.\n"
	774	+ "\n"
	775	+ "-i INTERFACE Network interface. Each (INTERFACE, QUEUE)\n"
	776	+ " pair specifies one forwarding port. May be\n"
	777	+ " invoked multiple times.\n"
	778	+ "\n"
	779	+ "-q QUEUE Network interface queue for RX and TX. Each\n"
	780	+ " (INTERFACE, QUEUE) pair specified one\n"
	781	+ " forwarding port. Default: %u. May be invoked\n"
	782	+ " multiple times.\n"
	783	+ "\n";
	784	+ printf(usage,
	785	+ prog_name,
	786	+ bpool_params_default.n_buffers,
	787	+ port_params_default.iface_queue);
	788	+}
	789	+
	790	+static int
	791	+parse_args(int argc, char **argv)
	792	+{
	793	+ struct option lgopts[] = {
	794	+ { NULL, 0, 0, 0 }
	795	+ };
	796	+ int opt, option_index;
	797	+
	798	+ /* Parse the input arguments. */
	799	+ for ( ; ;) {
	800	+ opt = getopt_long(argc, argv, "c:i:q:", lgopts, &option_index);
	801	+ if (opt == EOF)
	802	+ break;
	803	+
	804	+ switch (opt) {
	805	+ case 'b':
	806	+ bpool_params.n_buffers = atoi(optarg);
	807	+ break;
	808	+
	809	+ case 'c':
	810	+ if (n_threads == MAX_THREADS) {
	811	+ printf("Max number of threads (%d) reached.\n",
	812	+ MAX_THREADS);
	813	+ return -1;
	814	+ }
	815	+
	816	+ thread_data[n_threads].cpu_core_id = atoi(optarg);
	817	+ n_threads++;
	818	+ break;
	819	+
	820	+ case 'i':
	821	+ if (n_ports == MAX_PORTS) {
	822	+ printf("Max number of ports (%d) reached.\n",
	823	+ MAX_PORTS);
	824	+ return -1;
	825	+ }
	826	+
	827	+ port_params[n_ports].iface = optarg;
	828	+ port_params[n_ports].iface_queue = 0;
	829	+ n_ports++;
	830	+ break;
	831	+
	832	+ case 'q':
	833	+ if (n_ports == 0) {
	834	+ printf("No port specified for queue.\n");
	835	+ return -1;
	836	+ }
	837	+ port_params[n_ports - 1].iface_queue = atoi(optarg);
	838	+ break;
	839	+
	840	+ default:
	841	+ printf("Illegal argument.\n");
	842	+ return -1;
	843	+ }
	844	+ }
	845	+
	846	+ optind = 1; /* reset getopt lib */
	847	+
	848	+ /* Check the input arguments. */
	849	+ if (!n_ports) {
	850	+ printf("No ports specified.\n");
	851	+ return -1;
	852	+ }
	853	+
	854	+ if (!n_threads) {
	855	+ printf("No threads specified.\n");
	856	+ return -1;
	857	+ }
	858	+
	859	+ if (n_ports % n_threads) {
	860	+ printf("Ports cannot be evenly distributed to threads.\n");
	861	+ return -1;
	862	+ }
	863	+
	864	+ return 0;
	865	+}
	866	+
	867	+static void
	868	+print_port(u32 port_id)
	869	+{
	870	+ struct port *port = ports[port_id];
	871	+
	872	+ printf("Port %u: interface = %s, queue = %u\n",
	873	+ port_id, port->params.iface, port->params.iface_queue);
	874	+}
	875	+
	876	+static void
	877	+print_thread(u32 thread_id)
	878	+{
	879	+ struct thread_data *t = &thread_data[thread_id];
	880	+ u32 i;
	881	+
	882	+ printf("Thread %u (CPU core %u): ",
	883	+ thread_id, t->cpu_core_id);
	884	+
	885	+ for (i = 0; i < t->n_ports_rx; i++) {
	886	+ struct port *port_rx = t->ports_rx[i];
	887	+ struct port *port_tx = t->ports_tx[i];
	888	+
	889	+ printf("(%s, %u) -> (%s, %u), ",
	890	+ port_rx->params.iface,
	891	+ port_rx->params.iface_queue,
	892	+ port_tx->params.iface,
	893	+ port_tx->params.iface_queue);
	894	+ }
	895	+
	896	+ printf("\n");
	897	+}
	898	+
	899	+static void
	900	+print_port_stats_separator(void)
	901	+{
	902	+ printf("+-%4s-+-%12s-+-%13s-+-%12s-+-%13s-+\n",
	903	+ "----",
	904	+ "------------",
	905	+ "-------------",
	906	+ "------------",
	907	+ "-------------");
	908	+}
	909	+
	910	+static void
	911	+print_port_stats_header(void)
	912	+{
	913	+ print_port_stats_separator();
	914	+ printf("\| %4s \| %12s \| %13s \| %12s \| %13s \|\n",
	915	+ "Port",
	916	+ "RX packets",
	917	+ "RX rate (pps)",
	918	+ "TX packets",
	919	+ "TX_rate (pps)");
	920	+ print_port_stats_separator();
	921	+}
	922	+
	923	+static void
	924	+print_port_stats_trailer(void)
	925	+{
	926	+ print_port_stats_separator();
	927	+ printf("\n");
	928	+}
	929	+
	930	+static void
	931	+print_port_stats(int port_id, u64 ns_diff)
	932	+{
	933	+ struct port *p = ports[port_id];
	934	+ double rx_pps, tx_pps;
	935	+
	936	+ rx_pps = (p->n_pkts_rx - n_pkts_rx[port_id]) * 1000000000. / ns_diff;
	937	+ tx_pps = (p->n_pkts_tx - n_pkts_tx[port_id]) * 1000000000. / ns_diff;
	938	+
	939	+ printf("\| %4d \| %12llu \| %13.0f \| %12llu \| %13.0f \|\n",
	940	+ port_id,
	941	+ p->n_pkts_rx,
	942	+ rx_pps,
	943	+ p->n_pkts_tx,
	944	+ tx_pps);
	945	+
	946	+ n_pkts_rx[port_id] = p->n_pkts_rx;
	947	+ n_pkts_tx[port_id] = p->n_pkts_tx;
	948	+}
	949	+
	950	+static void
	951	+print_port_stats_all(u64 ns_diff)
	952	+{
	953	+ int i;
	954	+
	955	+ print_port_stats_header();
	956	+ for (i = 0; i < n_ports; i++)
	957	+ print_port_stats(i, ns_diff);
	958	+ print_port_stats_trailer();
	959	+}
	960	+
	961	+static int quit;
	962	+
	963	+static void
	964	+signal_handler(int sig)
	965	+{
	966	+ quit = 1;
	967	+}
	968	+
	969	+static void remove_xdp_program(void)
	970	+{
	971	+ int i;
	972	+
	973	+ for (i = 0 ; i < n_ports; i++)
	974	+ bpf_set_link_xdp_fd(if_nametoindex(port_params[i].iface), -1,
	975	+ port_params[i].xsk_cfg.xdp_flags);
	976	+}
	977	+
	978	+int main(int argc, char **argv)
	979	+{
	980	+ struct timespec time;
	981	+ u64 ns0;
	982	+ int i;
	983	+
	984	+ /* Parse args. */
	985	+ memcpy(&bpool_params, &bpool_params_default,
	986	+ sizeof(struct bpool_params));
	987	+ memcpy(&umem_cfg, &umem_cfg_default,
	988	+ sizeof(struct xsk_umem_config));
	989	+ for (i = 0; i < MAX_PORTS; i++)
	990	+ memcpy(&port_params[i], &port_params_default,
	991	+ sizeof(struct port_params));
	992	+
	993	+ if (parse_args(argc, argv)) {
	994	+ print_usage(argv[0]);
	995	+ return -1;
	996	+ }
	997	+
	998	+ /* Buffer pool initialization. */
	999	+ bp = bpool_init(&bpool_params, &umem_cfg);
	1000	+ if (!bp) {
	1001	+ printf("Buffer pool initialization failed.\n");
	1002	+ return -1;
	1003	+ }
	1004	+ printf("Buffer pool created successfully.\n");
	1005	+
	1006	+ /* Ports initialization. */
	1007	+ for (i = 0; i < MAX_PORTS; i++)
	1008	+ port_params[i].bp = bp;
	1009	+
	1010	+ for (i = 0; i < n_ports; i++) {
	1011	+ ports[i] = port_init(&port_params[i]);
	1012	+ if (!ports[i]) {
	1013	+ printf("Port %d initialization failed.\n", i);
	1014	+ return -1;
	1015	+ }
	1016	+ print_port(i);
	1017	+ }
	1018	+ printf("All ports created successfully.\n");
	1019	+
	1020	+ /* Threads. */
	1021	+ for (i = 0; i < n_threads; i++) {
	1022	+ struct thread_data *t = &thread_data[i];
	1023	+ u32 n_ports_per_thread = n_ports / n_threads, j;
	1024	+
	1025	+ for (j = 0; j < n_ports_per_thread; j++) {
	1026	+ t->ports_rx[j] = ports[i * n_ports_per_thread + j];
	1027	+ t->ports_tx[j] = ports[i * n_ports_per_thread +
	1028	+ (j + 1) % n_ports_per_thread];
	1029	+ }
	1030	+
	1031	+ t->n_ports_rx = n_ports_per_thread;
	1032	+
	1033	+ print_thread(i);
	1034	+ }
	1035	+
	1036	+ for (i = 0; i < n_threads; i++) {
	1037	+ int status;
	1038	+
	1039	+ status = pthread_create(&threads[i],
	1040	+ NULL,
	1041	+ thread_func,
	1042	+ &thread_data[i]);
	1043	+ if (status) {
	1044	+ printf("Thread %d creation failed.\n", i);
	1045	+ return -1;
	1046	+ }
	1047	+ }
	1048	+ printf("All threads created successfully.\n");
	1049	+
	1050	+ /* Print statistics. */
	1051	+ signal(SIGINT, signal_handler);
	1052	+ signal(SIGTERM, signal_handler);
	1053	+ signal(SIGABRT, signal_handler);
	1054	+
	1055	+ clock_gettime(CLOCK_MONOTONIC, &time);
	1056	+ ns0 = time.tv_sec * 1000000000UL + time.tv_nsec;
	1057	+ for ( ; !quit; ) {
	1058	+ u64 ns1, ns_diff;
	1059	+
	1060	+ sleep(1);
	1061	+ clock_gettime(CLOCK_MONOTONIC, &time);
	1062	+ ns1 = time.tv_sec * 1000000000UL + time.tv_nsec;
	1063	+ ns_diff = ns1 - ns0;
	1064	+ ns0 = ns1;
	1065	+
	1066	+ print_port_stats_all(ns_diff);
	1067	+ }
	1068	+
	1069	+ /* Threads completion. */
	1070	+ printf("Quit.\n");
	1071	+ for (i = 0; i < n_threads; i++)
	1072	+ thread_data[i].quit = 1;
	1073	+
	1074	+ for (i = 0; i < n_threads; i++)
	1075	+ pthread_join(threads[i], NULL);
	1076	+
	1077	+ for (i = 0; i < n_ports; i++)
	1078	+ port_free(ports[i]);
	1079	+
	1080	+ bpool_free(bp);
	1081	+
	1082	+ remove_xdp_program();
	1083	+
	1084	+ return 0;
	1085	+}

scripts/bpf_helpers_doc.py+2−0

@@ -432,6 +432,7 @@ class PrinterHelpers(Printer): · 431 unmodified lines
432	432	'struct __sk_buff',
433	433	'struct sk_msg_md',
434	434	'struct xdp_md',
	435	+ 'struct path',
435	436	]
436	437	known_types = {
437	438	'...',
@@ -472,6 +473,7 @@ class PrinterHelpers(Printer): · 34 unmodified lines
472	473	'struct tcp_request_sock',
473	474	'struct udp6_sock',
474	475	'struct task_struct',
	476	+ 'struct path',
475	477	}
476	478	mapped_types = {
477	479	'u8': '__u8',

security/bpf/hooks.c+6−0

@@ -11,6 +11,7 @@ static struct security_hook_list bpf_lsm_hooks[] __lsm_ro_after_init = { · 10 unmodified lines
11	11	LSM_HOOK_INIT(NAME, bpf_lsm_##NAME),
12	12	#include <linux/lsm_hook_defs.h>
13	13	#undef LSM_HOOK
	14	+ LSM_HOOK_INIT(inode_free_security, bpf_inode_storage_free),
14	15	};
15	16
16	17	static int __init bpf_lsm_init(void)
@@ -20,7 +21,12 @@ static int __init bpf_lsm_init(void) · 3 unmodified lines
20	21	return 0;
21	22	}
22	23
	24	+struct lsm_blob_sizes bpf_lsm_blob_sizes __lsm_ro_after_init = {
	25	+ .lbs_inode = sizeof(struct bpf_storage_blob),
	26	+};
	27	+
23	28	DEFINE_LSM(bpf) = {
24	29	.name = "bpf",
25	30	.init = bpf_lsm_init,
	31	+ .blobs = &bpf_lsm_blob_sizes
26	32	};

tools/bpf/bpftool/Documentation/bpftool-map.rst+1−1

@@ -49,7 +49,7 @@ MAP COMMANDS · 48 unmodified lines
49	49	\| \| lru_percpu_hash \| lpm_trie \| array_of_maps \| hash_of_maps
50	50	\| \| devmap \| devmap_hash \| sockmap \| cpumap \| xskmap \| sockhash
51	51	\| \| cgroup_storage \| reuseport_sockarray \| percpu_cgroup_storage
52		−\| \| queue \| stack \| sk_storage \| struct_ops \| ringbuf }
	52	+\| \| queue \| stack \| sk_storage \| struct_ops \| ringbuf \| inode_storage }
53	53
54	54	DESCRIPTION
55	55	===========

tools/bpf/bpftool/Makefile+5−1

@@ -176,7 +176,11 @@ $(OUTPUT)bpftool: $(OBJS) $(LIBBPF) · 175 unmodified lines
176	176	$(OUTPUT)%.o: %.c
177	177	$(QUIET_CC)$(CC) $(CFLAGS) -c -MMD -o $@ $<
178	178
179		−clean: $(LIBBPF)-clean
	179	+feature-detect-clean:
	180	+ $(call QUIET_CLEAN, feature-detect)
	181	+ $(Q)$(MAKE) -C $(srctree)/tools/build/feature/ clean >/dev/null
	182	+
	183	+clean: $(LIBBPF)-clean feature-detect-clean
180	184	$(call QUIET_CLEAN, bpftool)
181	185	$(Q)$(RM) -- $(OUTPUT)bpftool $(OUTPUT).o $(OUTPUT).d
182	186	$(Q)$(RM) -- $(BPFTOOL_BOOTSTRAP) $(OUTPUT)*.skel.h $(OUTPUT)vmlinux.h

tools/bpf/bpftool/bash-completion/bpftool+2−1

@@ -704,7 +704,8 @@ _bpftool() · 703 unmodified lines
704	704	lru_percpu_hash lpm_trie array_of_maps \
705	705	hash_of_maps devmap devmap_hash sockmap cpumap \
706	706	xskmap sockhash cgroup_storage reuseport_sockarray \
707		− percpu_cgroup_storage queue stack' -- \
	707	+ percpu_cgroup_storage queue stack sk_storage \
	708	+ struct_ops inode_storage' -- \
708	709	"$cur" ) )
709	710	return 0
710	711	;;

tools/bpf/bpftool/gen.c+0−2

@@ -19,11 +19,9 @@ · 18 unmodified lines
19	19	#include <sys/mman.h>
20	20	#include <bpf/btf.h>
21	21
22		−#include "bpf/libbpf_internal.h"
23	22	#include "json_writer.h"
24	23	#include "main.h"
25	24
26		−
27	25	#define MAX_OBJ_NAME_LEN 64
28	26
29	27	static void sanitize_identifier(char *name)

tools/bpf/bpftool/link.c+41−3

@@ -77,6 +77,22 @@ static void show_link_attach_type_json(__u32 attach_type, json_writer_t *wtr) · 76 unmodified lines
77	77	jsonw_uint_field(wtr, "attach_type", attach_type);
78	78	}
79	79
	80	+static bool is_iter_map_target(const char *target_name)
	81	+{
	82	+ return strcmp(target_name, "bpf_map_elem") == 0 \|\|
	83	+ strcmp(target_name, "bpf_sk_storage_map") == 0;
	84	+}
	85	+
	86	+static void show_iter_json(struct bpf_link_info info, json_writer_t wtr)
	87	+{
	88	+ const char *target_name = u64_to_ptr(info->iter.target_name);
	89	+
	90	+ jsonw_string_field(wtr, "target_name", target_name);
	91	+
	92	+ if (is_iter_map_target(target_name))
	93	+ jsonw_uint_field(wtr, "map_id", info->iter.map.map_id);
	94	+}
	95	+
80	96	static int get_prog_info(int prog_id, struct bpf_prog_info *info)
81	97	{
82	98	__u32 len = sizeof(*info);
@@ -128,6 +144,9 @@ static int show_link_close_json(int fd, struct bpf_link_info *info) · 45 unmodified lines
128	144	info->cgroup.cgroup_id);
129	145	show_link_attach_type_json(info->cgroup.attach_type, json_wtr);
130	146	break;
	147	+ case BPF_LINK_TYPE_ITER:
	148	+ show_iter_json(info, json_wtr);
	149	+ break;
131	150	case BPF_LINK_TYPE_NETNS:
132	151	jsonw_uint_field(json_wtr, "netns_ino",
133	152	info->netns.netns_ino);
@@ -175,6 +194,16 @@ static void show_link_attach_type_plain(__u32 attach_type) · 41 unmodified lines
175	194	printf("attach_type %u ", attach_type);
176	195	}
177	196
	197	+static void show_iter_plain(struct bpf_link_info *info)
	198	+{
	199	+ const char *target_name = u64_to_ptr(info->iter.target_name);
	200	+
	201	+ printf("target_name %s ", target_name);
	202	+
	203	+ if (is_iter_map_target(target_name))
	204	+ printf("map_id %u ", info->iter.map.map_id);
	205	+}
	206	+
178	207	static int show_link_close_plain(int fd, struct bpf_link_info *info)
179	208	{
180	209	struct bpf_prog_info prog_info;
@@ -204,6 +233,9 @@ static int show_link_close_plain(int fd, struct bpf_link_info *info) · 23 unmodified lines
204	233	printf("\n\tcgroup_id %zu ", (size_t)info->cgroup.cgroup_id);
205	234	show_link_attach_type_plain(info->cgroup.attach_type);
206	235	break;
	236	+ case BPF_LINK_TYPE_ITER:
	237	+ show_iter_plain(info);
	238	+ break;
207	239	case BPF_LINK_TYPE_NETNS:
208	240	printf("\n\tnetns_ino %u ", info->netns.netns_ino);
209	241	show_link_attach_type_plain(info->netns.attach_type);
@@ -231,7 +263,7 @@ static int do_show_link(int fd) · 21 unmodified lines
231	263	{
232	264	struct bpf_link_info info;
233	265	__u32 len = sizeof(info);
234		− char raw_tp_name[256];
	266	+ char buf[256];
235	267	int err;
236	268
237	269	memset(&info, 0, sizeof(info));
@@ -245,8 +277,14 @@ again: · 7 unmodified lines
245	277	}
246	278	if (info.type == BPF_LINK_TYPE_RAW_TRACEPOINT &&
247	279	!info.raw_tracepoint.tp_name) {
248		− info.raw_tracepoint.tp_name = (unsigned long)&raw_tp_name;
249		− info.raw_tracepoint.tp_name_len = sizeof(raw_tp_name);
	280	+ info.raw_tracepoint.tp_name = (unsigned long)&buf;
	281	+ info.raw_tracepoint.tp_name_len = sizeof(buf);
	282	+ goto again;
	283	+ }
	284	+ if (info.type == BPF_LINK_TYPE_ITER &&
	285	+ !info.iter.target_name) {
	286	+ info.iter.target_name = (unsigned long)&buf;
	287	+ info.iter.target_name_len = sizeof(buf);
250	288	goto again;
251	289	}
252	290

tools/bpf/bpftool/map.c+2−1

@@ -50,6 +50,7 @@ const char * const map_type_name[] = { · 49 unmodified lines
50	50	[BPF_MAP_TYPE_SK_STORAGE] = "sk_storage",
51	51	[BPF_MAP_TYPE_STRUCT_OPS] = "struct_ops",
52	52	[BPF_MAP_TYPE_RINGBUF] = "ringbuf",
	53	+ [BPF_MAP_TYPE_INODE_STORAGE] = "inode_storage",
53	54	};
54	55
55	56	const size_t map_type_name_size = ARRAY_SIZE(map_type_name);
@@ -1442,7 +1443,7 @@ static int do_help(int argc, char **argv) · 1386 unmodified lines
1442	1443	" lru_percpu_hash \| lpm_trie \| array_of_maps \| hash_of_maps \|\n"
1443	1444	" devmap \| devmap_hash \| sockmap \| cpumap \| xskmap \| sockhash \|\n"
1444	1445	" cgroup_storage \| reuseport_sockarray \| percpu_cgroup_storage \|\n"
1445		− " queue \| stack \| sk_storage \| struct_ops \| ringbuf }\n"
	1446	+ " queue \| stack \| sk_storage \| struct_ops \| ringbuf \| inode_storage }\n"
1446	1447	" " HELP_SPEC_OPTIONS "\n"
1447	1448	"",
1448	1449	bin_name, argv[-2]);

tools/bpf/bpftool/net.c+282−17

@@ -6,22 +6,27 @@ · 5 unmodified lines
6	6	#include <fcntl.h>
7	7	#include <stdlib.h>
8	8	#include <string.h>
	9	+#include <time.h>
9	10	#include <unistd.h>
10	11	#include <bpf/bpf.h>
11	12	#include <bpf/libbpf.h>
12	13	#include <net/if.h>
13	14	#include <linux/if.h>
14	15	#include <linux/rtnetlink.h>
	16	+#include <linux/socket.h>
15	17	#include <linux/tc_act/tc_bpf.h>
16	18	#include <sys/socket.h>
17	19	#include <sys/stat.h>
18	20	#include <sys/types.h>
19	21
20	22	#include "bpf/nlattr.h"
21		−#include "bpf/libbpf_internal.h"
22	23	#include "main.h"
23	24	#include "netlink_dumper.h"
24	25
	26	+#ifndef SOL_NETLINK
	27	+#define SOL_NETLINK 270
	28	+#endif
	29	+
25	30	struct ip_devname_ifindex {
26	31	char devname[64];
27	32	int ifindex;
@@ -85,6 +90,266 @@ static enum net_attach_type parse_attach_type(const char *str) · 57 unmodified lines
85	90	return net_attach_type_size;
86	91	}
87	92
	93	+typedef int (dump_nlmsg_t)(void cookie, void msg, struct nlattr *tb);
	94	+
	95	+typedef int (__dump_nlmsg_t)(struct nlmsghdr nlmsg, dump_nlmsg_t, void *cookie);
	96	+
	97	+static int netlink_open(__u32 *nl_pid)
	98	+{
	99	+ struct sockaddr_nl sa;
	100	+ socklen_t addrlen;
	101	+ int one = 1, ret;
	102	+ int sock;
	103	+
	104	+ memset(&sa, 0, sizeof(sa));
	105	+ sa.nl_family = AF_NETLINK;
	106	+
	107	+ sock = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);
	108	+ if (sock < 0)
	109	+ return -errno;
	110	+
	111	+ if (setsockopt(sock, SOL_NETLINK, NETLINK_EXT_ACK,
	112	+ &one, sizeof(one)) < 0) {
	113	+ p_err("Netlink error reporting not supported");
	114	+ }
	115	+
	116	+ if (bind(sock, (struct sockaddr *)&sa, sizeof(sa)) < 0) {
	117	+ ret = -errno;
	118	+ goto cleanup;
	119	+ }
	120	+
	121	+ addrlen = sizeof(sa);
	122	+ if (getsockname(sock, (struct sockaddr *)&sa, &addrlen) < 0) {
	123	+ ret = -errno;
	124	+ goto cleanup;
	125	+ }
	126	+
	127	+ if (addrlen != sizeof(sa)) {
	128	+ ret = -LIBBPF_ERRNO__INTERNAL;
	129	+ goto cleanup;
	130	+ }
	131	+
	132	+ *nl_pid = sa.nl_pid;
	133	+ return sock;
	134	+
	135	+cleanup:
	136	+ close(sock);
	137	+ return ret;
	138	+}
	139	+
	140	+static int netlink_recv(int sock, __u32 nl_pid, __u32 seq,
	141	+ __dump_nlmsg_t _fn, dump_nlmsg_t fn,
	142	+ void *cookie)
	143	+{
	144	+ bool multipart = true;
	145	+ struct nlmsgerr *err;
	146	+ struct nlmsghdr *nh;
	147	+ char buf[4096];
	148	+ int len, ret;
	149	+
	150	+ while (multipart) {
	151	+ multipart = false;
	152	+ len = recv(sock, buf, sizeof(buf), 0);
	153	+ if (len < 0) {
	154	+ ret = -errno;
	155	+ goto done;
	156	+ }
	157	+
	158	+ if (len == 0)
	159	+ break;
	160	+
	161	+ for (nh = (struct nlmsghdr *)buf; NLMSG_OK(nh, len);
	162	+ nh = NLMSG_NEXT(nh, len)) {
	163	+ if (nh->nlmsg_pid != nl_pid) {
	164	+ ret = -LIBBPF_ERRNO__WRNGPID;
	165	+ goto done;
	166	+ }
	167	+ if (nh->nlmsg_seq != seq) {
	168	+ ret = -LIBBPF_ERRNO__INVSEQ;
	169	+ goto done;
	170	+ }
	171	+ if (nh->nlmsg_flags & NLM_F_MULTI)
	172	+ multipart = true;
	173	+ switch (nh->nlmsg_type) {
	174	+ case NLMSG_ERROR:
	175	+ err = (struct nlmsgerr *)NLMSG_DATA(nh);
	176	+ if (!err->error)
	177	+ continue;
	178	+ ret = err->error;
	179	+ libbpf_nla_dump_errormsg(nh);
	180	+ goto done;
	181	+ case NLMSG_DONE:
	182	+ return 0;
	183	+ default:
	184	+ break;
	185	+ }
	186	+ if (_fn) {
	187	+ ret = _fn(nh, fn, cookie);
	188	+ if (ret)
	189	+ return ret;
	190	+ }
	191	+ }
	192	+ }
	193	+ ret = 0;
	194	+done:
	195	+ return ret;
	196	+}
	197	+
	198	+static int __dump_class_nlmsg(struct nlmsghdr *nlh,
	199	+ dump_nlmsg_t dump_class_nlmsg,
	200	+ void *cookie)
	201	+{
	202	+ struct nlattr tb[TCA_MAX + 1], attr;
	203	+ struct tcmsg *t = NLMSG_DATA(nlh);
	204	+ int len;
	205	+
	206	+ len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*t));
	207	+ attr = (struct nlattr ) ((void ) t + NLMSG_ALIGN(sizeof(*t)));
	208	+ if (libbpf_nla_parse(tb, TCA_MAX, attr, len, NULL) != 0)
	209	+ return -LIBBPF_ERRNO__NLPARSE;
	210	+
	211	+ return dump_class_nlmsg(cookie, t, tb);
	212	+}
	213	+
	214	+static int netlink_get_class(int sock, unsigned int nl_pid, int ifindex,
	215	+ dump_nlmsg_t dump_class_nlmsg, void *cookie)
	216	+{
	217	+ struct {
	218	+ struct nlmsghdr nlh;
	219	+ struct tcmsg t;
	220	+ } req = {
	221	+ .nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg)),
	222	+ .nlh.nlmsg_type = RTM_GETTCLASS,
	223	+ .nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
	224	+ .t.tcm_family = AF_UNSPEC,
	225	+ .t.tcm_ifindex = ifindex,
	226	+ };
	227	+ int seq = time(NULL);
	228	+
	229	+ req.nlh.nlmsg_seq = seq;
	230	+ if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
	231	+ return -errno;
	232	+
	233	+ return netlink_recv(sock, nl_pid, seq, __dump_class_nlmsg,
	234	+ dump_class_nlmsg, cookie);
	235	+}
	236	+
	237	+static int __dump_qdisc_nlmsg(struct nlmsghdr *nlh,
	238	+ dump_nlmsg_t dump_qdisc_nlmsg,
	239	+ void *cookie)
	240	+{
	241	+ struct nlattr tb[TCA_MAX + 1], attr;
	242	+ struct tcmsg *t = NLMSG_DATA(nlh);
	243	+ int len;
	244	+
	245	+ len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*t));
	246	+ attr = (struct nlattr ) ((void ) t + NLMSG_ALIGN(sizeof(*t)));
	247	+ if (libbpf_nla_parse(tb, TCA_MAX, attr, len, NULL) != 0)
	248	+ return -LIBBPF_ERRNO__NLPARSE;
	249	+
	250	+ return dump_qdisc_nlmsg(cookie, t, tb);
	251	+}
	252	+
	253	+static int netlink_get_qdisc(int sock, unsigned int nl_pid, int ifindex,
	254	+ dump_nlmsg_t dump_qdisc_nlmsg, void *cookie)
	255	+{
	256	+ struct {
	257	+ struct nlmsghdr nlh;
	258	+ struct tcmsg t;
	259	+ } req = {
	260	+ .nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg)),
	261	+ .nlh.nlmsg_type = RTM_GETQDISC,
	262	+ .nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
	263	+ .t.tcm_family = AF_UNSPEC,
	264	+ .t.tcm_ifindex = ifindex,
	265	+ };
	266	+ int seq = time(NULL);
	267	+
	268	+ req.nlh.nlmsg_seq = seq;
	269	+ if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
	270	+ return -errno;
	271	+
	272	+ return netlink_recv(sock, nl_pid, seq, __dump_qdisc_nlmsg,
	273	+ dump_qdisc_nlmsg, cookie);
	274	+}
	275	+
	276	+static int __dump_filter_nlmsg(struct nlmsghdr *nlh,
	277	+ dump_nlmsg_t dump_filter_nlmsg,
	278	+ void *cookie)
	279	+{
	280	+ struct nlattr tb[TCA_MAX + 1], attr;
	281	+ struct tcmsg *t = NLMSG_DATA(nlh);
	282	+ int len;
	283	+
	284	+ len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*t));
	285	+ attr = (struct nlattr ) ((void ) t + NLMSG_ALIGN(sizeof(*t)));
	286	+ if (libbpf_nla_parse(tb, TCA_MAX, attr, len, NULL) != 0)
	287	+ return -LIBBPF_ERRNO__NLPARSE;
	288	+
	289	+ return dump_filter_nlmsg(cookie, t, tb);
	290	+}
	291	+
	292	+static int netlink_get_filter(int sock, unsigned int nl_pid, int ifindex, int handle,
	293	+ dump_nlmsg_t dump_filter_nlmsg, void *cookie)
	294	+{
	295	+ struct {
	296	+ struct nlmsghdr nlh;
	297	+ struct tcmsg t;
	298	+ } req = {
	299	+ .nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg)),
	300	+ .nlh.nlmsg_type = RTM_GETTFILTER,
	301	+ .nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
	302	+ .t.tcm_family = AF_UNSPEC,
	303	+ .t.tcm_ifindex = ifindex,
	304	+ .t.tcm_parent = handle,
	305	+ };
	306	+ int seq = time(NULL);
	307	+
	308	+ req.nlh.nlmsg_seq = seq;
	309	+ if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
	310	+ return -errno;
	311	+
	312	+ return netlink_recv(sock, nl_pid, seq, __dump_filter_nlmsg,
	313	+ dump_filter_nlmsg, cookie);
	314	+}
	315	+
	316	+static int __dump_link_nlmsg(struct nlmsghdr *nlh,
	317	+ dump_nlmsg_t dump_link_nlmsg, void *cookie)
	318	+{
	319	+ struct nlattr tb[IFLA_MAX + 1], attr;
	320	+ struct ifinfomsg *ifi = NLMSG_DATA(nlh);
	321	+ int len;
	322	+
	323	+ len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*ifi));
	324	+ attr = (struct nlattr ) ((void ) ifi + NLMSG_ALIGN(sizeof(*ifi)));
	325	+ if (libbpf_nla_parse(tb, IFLA_MAX, attr, len, NULL) != 0)
	326	+ return -LIBBPF_ERRNO__NLPARSE;
	327	+
	328	+ return dump_link_nlmsg(cookie, ifi, tb);
	329	+}
	330	+
	331	+static int netlink_get_link(int sock, unsigned int nl_pid,
	332	+ dump_nlmsg_t dump_link_nlmsg, void *cookie)
	333	+{
	334	+ struct {
	335	+ struct nlmsghdr nlh;
	336	+ struct ifinfomsg ifm;
	337	+ } req = {
	338	+ .nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct ifinfomsg)),
	339	+ .nlh.nlmsg_type = RTM_GETLINK,
	340	+ .nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
	341	+ .ifm.ifi_family = AF_PACKET,
	342	+ };
	343	+ int seq = time(NULL);
	344	+
	345	+ req.nlh.nlmsg_seq = seq;
	346	+ if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
	347	+ return -errno;
	348	+
	349	+ return netlink_recv(sock, nl_pid, seq, __dump_link_nlmsg,
	350	+ dump_link_nlmsg, cookie);
	351	+}
	352	+
88	353	static int dump_link_nlmsg(void cookie, void msg, struct nlattr **tb)
89	354	{
90	355	struct bpf_netdev_t *netinfo = cookie;
@@ -168,14 +433,14 @@ static int show_dev_tc_bpf(int sock, unsigned int nl_pid, · 77 unmodified lines
168	433	tcinfo.array_len = 0;
169	434
170	435	tcinfo.is_qdisc = false;
171		− ret = libbpf_nl_get_class(sock, nl_pid, dev->ifindex,
172		− dump_class_qdisc_nlmsg, &tcinfo);
	436	+ ret = netlink_get_class(sock, nl_pid, dev->ifindex,
	437	+ dump_class_qdisc_nlmsg, &tcinfo);
173	438	if (ret)
174	439	goto out;
175	440
176	441	tcinfo.is_qdisc = true;
177		− ret = libbpf_nl_get_qdisc(sock, nl_pid, dev->ifindex,
178		− dump_class_qdisc_nlmsg, &tcinfo);
	442	+ ret = netlink_get_qdisc(sock, nl_pid, dev->ifindex,
	443	+ dump_class_qdisc_nlmsg, &tcinfo);
179	444	if (ret)
180	445	goto out;
181	446
@@ -183,9 +448,9 @@ static int show_dev_tc_bpf(int sock, unsigned int nl_pid, · 1 unmodified line
183	448	filter_info.ifindex = dev->ifindex;
184	449	for (i = 0; i < tcinfo.used_len; i++) {
185	450	filter_info.kind = tcinfo.handle_array[i].kind;
186		− ret = libbpf_nl_get_filter(sock, nl_pid, dev->ifindex,
187		− tcinfo.handle_array[i].handle,
188		− dump_filter_nlmsg, &filter_info);
	451	+ ret = netlink_get_filter(sock, nl_pid, dev->ifindex,
	452	+ tcinfo.handle_array[i].handle,
	453	+ dump_filter_nlmsg, &filter_info);
189	454	if (ret)
190	455	goto out;
191	456	}
@@ -193,22 +458,22 @@ static int show_dev_tc_bpf(int sock, unsigned int nl_pid, · 1 unmodified line
193	458	/* root, ingress and egress handle */
194	459	handle = TC_H_ROOT;
195	460	filter_info.kind = "root";
196		− ret = libbpf_nl_get_filter(sock, nl_pid, dev->ifindex, handle,
197		− dump_filter_nlmsg, &filter_info);
	461	+ ret = netlink_get_filter(sock, nl_pid, dev->ifindex, handle,
	462	+ dump_filter_nlmsg, &filter_info);
198	463	if (ret)
199	464	goto out;
200	465
201	466	handle = TC_H_MAKE(TC_H_CLSACT, TC_H_MIN_INGRESS);
202	467	filter_info.kind = "clsact/ingress";
203		− ret = libbpf_nl_get_filter(sock, nl_pid, dev->ifindex, handle,
204		− dump_filter_nlmsg, &filter_info);
	468	+ ret = netlink_get_filter(sock, nl_pid, dev->ifindex, handle,
	469	+ dump_filter_nlmsg, &filter_info);
205	470	if (ret)
206	471	goto out;
207	472
208	473	handle = TC_H_MAKE(TC_H_CLSACT, TC_H_MIN_EGRESS);
209	474	filter_info.kind = "clsact/egress";
210		− ret = libbpf_nl_get_filter(sock, nl_pid, dev->ifindex, handle,
211		− dump_filter_nlmsg, &filter_info);
	475	+ ret = netlink_get_filter(sock, nl_pid, dev->ifindex, handle,
	476	+ dump_filter_nlmsg, &filter_info);
212	477	if (ret)
213	478	goto out;
214	479
@@ -386,7 +651,7 @@ static int do_show(int argc, char **argv) · 171 unmodified lines
386	651	struct bpf_attach_info attach_info = {};
387	652	int i, sock, ret, filter_idx = -1;
388	653	struct bpf_netdev_t dev_array;
389		− unsigned int nl_pid;
	654	+ unsigned int nl_pid = 0;
390	655	char err_buf[256];
391	656
392	657	if (argc == 2) {
@@ -401,7 +666,7 @@ static int do_show(int argc, char **argv) · 8 unmodified lines
401	666	if (ret)
402	667	return -1;
403	668
404		− sock = libbpf_netlink_open(&nl_pid);
	669	+ sock = netlink_open(&nl_pid);
405	670	if (sock < 0) {
406	671	fprintf(stderr, "failed to open netlink sock\n");
407	672	return -1;
@@ -416,7 +681,7 @@ static int do_show(int argc, char **argv) · 8 unmodified lines
416	681	jsonw_start_array(json_wtr);
417	682	NET_START_OBJECT;
418	683	NET_START_ARRAY("xdp", "%s:\n");
419		− ret = libbpf_nl_get_link(sock, nl_pid, dump_link_nlmsg, &dev_array);
	684	+ ret = netlink_get_link(sock, nl_pid, dump_link_nlmsg, &dev_array);
420	685	NET_END_ARRAY("\n");
421	686
422	687	if (!ret) {

tools/bpf/resolve_btfids/main.c+27−2

@@ -199,9 +199,16 @@ static char get_id(const char prefix_end) · 198 unmodified lines
199	199	/*
200	200	* __BTF_ID__func__vfs_truncate__0
201	201	* prefix_end = ^
	202	+ * pos = ^
202	203	*/
203		− char p, id = strdup(prefix_end + sizeof("__") - 1);
	204	+ int len = strlen(prefix_end);
	205	+ int pos = sizeof("__") - 1;
	206	+ char p, id;
204	207
	208	+ if (pos >= len)
	209	+ return NULL;
	210	+
	211	+ id = strdup(prefix_end + pos);
205	212	if (id) {
206	213	/*
207	214	* __BTF_ID__func__vfs_truncate__0
@@ -220,6 +227,24 @@ static char get_id(const char prefix_end) · 12 unmodified lines
220	227	return id;
221	228	}
222	229
	230	+static struct btf_id add_set(struct object obj, char *name)
	231	+{
	232	+ /*
	233	+ * __BTF_ID__set__name
	234	+ * name = ^
	235	+ * id = ^
	236	+ */
	237	+ char *id = name + sizeof(BTF_SET "__") - 1;
	238	+ int len = strlen(name);
	239	+
	240	+ if (id >= name + len) {
	241	+ pr_err("FAILED to parse set name: %s\n", name);
	242	+ return NULL;
	243	+ }
	244	+
	245	+ return btf_id__add(&obj->sets, id, true);
	246	+}
	247	+
223	248	static struct btf_id add_symbol(struct rb_root root, char *name, size_t size)
224	249	{
225	250	char *id;
@@ -412,7 +437,7 @@ static int symbols_collect(struct object *obj) · 186 unmodified lines
412	437	id = add_symbol(&obj->funcs, prefix, sizeof(BTF_FUNC) - 1);
413	438	/* set */
414	439	} else if (!strncmp(prefix, BTF_SET, sizeof(BTF_SET) - 1)) {
415		− id = add_symbol(&obj->sets, prefix, sizeof(BTF_SET) - 1);
	440	+ id = add_set(obj, prefix);
416	441	/*
417	442	* SET objects store list's count, which is encoded
418	443	* in symbol's size, together with 'cnt' field hence

tools/build/Makefile+2−0

@@ -38,6 +38,8 @@ clean: · 37 unmodified lines
38	38	$(call QUIET_CLEAN, fixdep)
39	39	$(Q)find $(if $(OUTPUT),$(OUTPUT),.) -name '.o' -delete -o -name '\..cmd' -delete -o -name '\.*.d' -delete
40	40	$(Q)rm -f $(OUTPUT)fixdep
	41	+ $(call QUIET_CLEAN, feature-detect)
	42	+ $(Q)$(MAKE) -C feature/ clean >/dev/null
41	43
42	44	$(OUTPUT)fixdep-in.o: FORCE
43	45	$(Q)$(MAKE) $(build)=fixdep

tools/build/Makefile.feature+0−1

@@ -46,7 +46,6 @@ FEATURE_TESTS_BASIC := \ · 45 unmodified lines
46	46	libelf-getphdrnum \
47	47	libelf-gelf_getnote \
48	48	libelf-getshdrstrndx \
49		− libelf-mmap \
50	49	libnuma \
51	50	numa_num_possible_cpus \
52	51	libperl \

tools/build/feature/Makefile+0−4

@@ -25,7 +25,6 @@ FILES= \ · 24 unmodified lines
25	25	test-libelf-getphdrnum.bin \
26	26	test-libelf-gelf_getnote.bin \
27	27	test-libelf-getshdrstrndx.bin \
28		− test-libelf-mmap.bin \
29	28	test-libdebuginfod.bin \
30	29	test-libnuma.bin \
31	30	test-numa_num_possible_cpus.bin \
@@ -146,9 +145,6 @@ $(OUTPUT)test-dwarf.bin: · 114 unmodified lines
146	145	$(OUTPUT)test-dwarf_getlocations.bin:
147	146	$(BUILD) $(DWARFLIBS)
148	147
149		−$(OUTPUT)test-libelf-mmap.bin:
150		− $(BUILD) -lelf
151		−
152	148	$(OUTPUT)test-libelf-getphdrnum.bin:
153	149	$(BUILD) -lelf
154	150

tools/build/feature/test-all.c+0−4

@@ -30,10 +30,6 @@ · 29 unmodified lines
30	30	# include "test-libelf.c"
31	31	#undef main
32	32
33		−#define main main_test_libelf_mmap
34		−# include "test-libelf-mmap.c"
35		−#undef main
36		−
37	33	#define main main_test_get_current_dir_name
38	34	# include "test-get_current_dir_name.c"
39	35	#undef main

tools/build/feature/test-libelf-mmap.c+0−9

@@ -1,9 +0,0 @@
1		−// SPDX-License-Identifier: GPL-2.0
2		−#include <libelf.h>
3		−
4		−int main(void)
5		−{
6		− Elf *elf = elf_begin(0, ELF_C_READ_MMAP, 0);
7		−
8		− return (long)elf;
9		−}

tools/include/linux/btf_ids.h+50−1

@@ -3,6 +3,11 @@ · 2 unmodified lines
3	3	#ifndef _LINUX_BTF_IDS_H
4	4	#define _LINUX_BTF_IDS_H
5	5
	6	+struct btf_id_set {
	7	+ u32 cnt;
	8	+ u32 ids[];
	9	+};
	10	+
6	11	#ifdef CONFIG_DEBUG_INFO_BTF
7	12
8	13	#include <linux/compiler.h> /* for __PASTE */
@@ -62,7 +67,7 @@ asm( \ · 53 unmodified lines
62	67	".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
63	68	"." #scope " " #name "; \n" \
64	69	#name ":; \n" \
65		−".popsection; \n"); \
	70	+".popsection; \n");
66	71
67	72	#define BTF_ID_LIST(name) \
68	73	__BTF_ID_LIST(name, local) \
@@ -88,12 +93,56 @@ asm( \ · 19 unmodified lines
88	93	".zero 4 \n" \
89	94	".popsection; \n");
90	95
	96	+/*
	97	+ * The BTF_SET_START/END macros pair defines sorted list of
	98	+ * BTF IDs plus its members count, with following layout:
	99	+ *
	100	+ * BTF_SET_START(list)
	101	+ * BTF_ID(type1, name1)
	102	+ * BTF_ID(type2, name2)
	103	+ * BTF_SET_END(list)
	104	+ *
	105	+ * __BTF_ID__set__list:
	106	+ * .zero 4
	107	+ * list:
	108	+ * __BTF_ID__type1__name1__3:
	109	+ * .zero 4
	110	+ * __BTF_ID__type2__name2__4:
	111	+ * .zero 4
	112	+ *
	113	+ */
	114	+#define __BTF_SET_START(name, scope) \
	115	+asm( \
	116	+".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
	117	+"." #scope " __BTF_ID__set__" #name "; \n" \
	118	+"__BTF_ID__set__" #name ":; \n" \
	119	+".zero 4 \n" \
	120	+".popsection; \n");
	121	+
	122	+#define BTF_SET_START(name) \
	123	+__BTF_ID_LIST(name, local) \
	124	+__BTF_SET_START(name, local)
	125	+
	126	+#define BTF_SET_START_GLOBAL(name) \
	127	+__BTF_ID_LIST(name, globl) \
	128	+__BTF_SET_START(name, globl)
	129	+
	130	+#define BTF_SET_END(name) \
	131	+asm( \
	132	+".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
	133	+".size __BTF_ID__set__" #name ", .-" #name " \n" \
	134	+".popsection; \n"); \
	135	+extern struct btf_id_set name;
	136	+
91	137	#else
92	138
93	139	#define BTF_ID_LIST(name) static u32 name[5];
94	140	#define BTF_ID(prefix, name)
95	141	#define BTF_ID_UNUSED
96	142	#define BTF_ID_LIST_GLOBAL(name) u32 name[1];
	143	+#define BTF_SET_START(name) static struct btf_id_set name = { 0 };
	144	+#define BTF_SET_START_GLOBAL(name) static struct btf_id_set name = { 0 };
	145	+#define BTF_SET_END(name)
97	146
98	147	#endif /* CONFIG_DEBUG_INFO_BTF */
99	148

tools/include/uapi/linux/bpf.h+393−5

@@ -155,6 +155,7 @@ enum bpf_map_type { · 154 unmodified lines
155	155	BPF_MAP_TYPE_DEVMAP_HASH,
156	156	BPF_MAP_TYPE_STRUCT_OPS,
157	157	BPF_MAP_TYPE_RINGBUF,
	158	+ BPF_MAP_TYPE_INODE_STORAGE,
158	159	};
159	160
160	161	/* Note that tracing related programs such as
@@ -345,6 +346,14 @@ enum bpf_link_type { · 184 unmodified lines
345	346	/* The verifier internal test flag. Behavior is undefined */
346	347	#define BPF_F_TEST_STATE_FREQ (1U << 3)
347	348
	349	+/* If BPF_F_SLEEPABLE is used in BPF_PROG_LOAD command, the verifier will
	350	+ * restrict map and helper usage for such programs. Sleepable BPF programs can
	351	+ * only be attached to hooks where kernel execution context allows sleeping.
	352	+ * Such programs are allowed to use helpers that may sleep like
	353	+ * bpf_copy_from_user().
	354	+ */
	355	+#define BPF_F_SLEEPABLE (1U << 4)
	356	+
348	357	/* When BPF ldimm64's insn[0].src_reg != 0 then this can have
349	358	* two extensions:
350	359	*
@@ -2807,7 +2816,7 @@ union bpf_attr { · 2456 unmodified lines
2807	2816	*
2808	2817	* -ERANGE if resulting value was out of range.
2809	2818	*
2810		− * void bpf_sk_storage_get(struct bpf_map map, struct bpf_sock sk, void value, u64 flags)
	2819	+ * void bpf_sk_storage_get(struct bpf_map map, void sk, void value, u64 flags)
2811	2820	* Description
2812	2821	* Get a bpf-local-storage from a sk.
2813	2822	*
@@ -2823,6 +2832,9 @@ union bpf_attr { · 9 unmodified lines
2823	2832	* "type". The bpf-local-storage "type" (i.e. the map) is
2824	2833	* searched against all bpf-local-storages residing at sk.
2825	2834	*
	2835	+ * sk is a kernel struct sock pointer for LSM program.
	2836	+ * sk is a struct bpf_sock pointer for other program types.
	2837	+ *
2826	2838	* An optional flags (BPF_SK_STORAGE_GET_F_CREATE) can be
2827	2839	* used such that a new bpf-local-storage will be
2828	2840	* created if one does not exist. value can be used
@@ -2835,7 +2847,7 @@ union bpf_attr { · 6 unmodified lines
2835	2847	* NULL if not found or there was an error in adding
2836	2848	* a new bpf-local-storage.
2837	2849	*
2838		− * long bpf_sk_storage_delete(struct bpf_map map, struct bpf_sock sk)
	2850	+ * long bpf_sk_storage_delete(struct bpf_map map, void sk)
2839	2851	* Description
2840	2852	* Delete a bpf-local-storage from a sk.
2841	2853	* Return
@@ -3395,6 +3407,175 @@ union bpf_attr { · 553 unmodified lines
3395	3407	* A non-negative value equal to or less than size on success,
3396	3408	* or a negative error in case of failure.
3397	3409	*
	3410	+ * long bpf_load_hdr_opt(struct bpf_sock_ops skops, void searchby_res, u32 len, u64 flags)
	3411	+ * Description
	3412	+ * Load header option. Support reading a particular TCP header
	3413	+ * option for bpf program (BPF_PROG_TYPE_SOCK_OPS).
	3414	+ *
	3415	+ * If flags is 0, it will search the option from the
	3416	+ * sock_ops->skb_data. The comment in "struct bpf_sock_ops"
	3417	+ * has details on what skb_data contains under different
	3418	+ * sock_ops->op.
	3419	+ *
	3420	+ * The first byte of the searchby_res specifies the
	3421	+ * kind that it wants to search.
	3422	+ *
	3423	+ * If the searching kind is an experimental kind
	3424	+ * (i.e. 253 or 254 according to RFC6994). It also
	3425	+ * needs to specify the "magic" which is either
	3426	+ * 2 bytes or 4 bytes. It then also needs to
	3427	+ * specify the size of the magic by using
	3428	+ * the 2nd byte which is "kind-length" of a TCP
	3429	+ * header option and the "kind-length" also
	3430	+ * includes the first 2 bytes "kind" and "kind-length"
	3431	+ * itself as a normal TCP header option also does.
	3432	+ *
	3433	+ * For example, to search experimental kind 254 with
	3434	+ * 2 byte magic 0xeB9F, the searchby_res should be
	3435	+ * [ 254, 4, 0xeB, 0x9F, 0, 0, .... 0 ].
	3436	+ *
	3437	+ * To search for the standard window scale option (3),
	3438	+ * the searchby_res should be [ 3, 0, 0, .... 0 ].
	3439	+ * Note, kind-length must be 0 for regular option.
	3440	+ *
	3441	+ * Searching for No-Op (0) and End-of-Option-List (1) are
	3442	+ * not supported.
	3443	+ *
	3444	+ * len must be at least 2 bytes which is the minimal size
	3445	+ * of a header option.
	3446	+ *
	3447	+ * Supported flags:
	3448	+ * * BPF_LOAD_HDR_OPT_TCP_SYN to search from the
	3449	+ * saved_syn packet or the just-received syn packet.
	3450	+ *
	3451	+ * Return
	3452	+ * >0 when found, the header option is copied to searchby_res.
	3453	+ * The return value is the total length copied.
	3454	+ *
	3455	+ * -EINVAL If param is invalid
	3456	+ *
	3457	+ * -ENOMSG The option is not found
	3458	+ *
	3459	+ * -ENOENT No syn packet available when
	3460	+ * BPF_LOAD_HDR_OPT_TCP_SYN is used
	3461	+ *
	3462	+ * -ENOSPC Not enough space. Only len number of
	3463	+ * bytes are copied.
	3464	+ *
	3465	+ * -EFAULT Cannot parse the header options in the packet
	3466	+ *
	3467	+ * -EPERM This helper cannot be used under the
	3468	+ * current sock_ops->op.
	3469	+ *
	3470	+ * long bpf_store_hdr_opt(struct bpf_sock_ops skops, const void from, u32 len, u64 flags)
	3471	+ * Description
	3472	+ * Store header option. The data will be copied
	3473	+ * from buffer from with length len to the TCP header.
	3474	+ *
	3475	+ * The buffer from should have the whole option that
	3476	+ * includes the kind, kind-length, and the actual
	3477	+ * option data. The len must be at least kind-length
	3478	+ * long. The kind-length does not have to be 4 byte
	3479	+ * aligned. The kernel will take care of the padding
	3480	+ * and setting the 4 bytes aligned value to th->doff.
	3481	+ *
	3482	+ * This helper will check for duplicated option
	3483	+ * by searching the same option in the outgoing skb.
	3484	+ *
	3485	+ * This helper can only be called during
	3486	+ * BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
	3487	+ *
	3488	+ * Return
	3489	+ * 0 on success, or negative error in case of failure:
	3490	+ *
	3491	+ * -EINVAL If param is invalid
	3492	+ *
	3493	+ * -ENOSPC Not enough space in the header.
	3494	+ * Nothing has been written
	3495	+ *
	3496	+ * -EEXIST The option has already existed
	3497	+ *
	3498	+ * -EFAULT Cannot parse the existing header options
	3499	+ *
	3500	+ * -EPERM This helper cannot be used under the
	3501	+ * current sock_ops->op.
	3502	+ *
	3503	+ * long bpf_reserve_hdr_opt(struct bpf_sock_ops *skops, u32 len, u64 flags)
	3504	+ * Description
	3505	+ * Reserve len bytes for the bpf header option. The
	3506	+ * space will be used by bpf_store_hdr_opt() later in
	3507	+ * BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
	3508	+ *
	3509	+ * If bpf_reserve_hdr_opt() is called multiple times,
	3510	+ * the total number of bytes will be reserved.
	3511	+ *
	3512	+ * This helper can only be called during
	3513	+ * BPF_SOCK_OPS_HDR_OPT_LEN_CB.
	3514	+ *
	3515	+ * Return
	3516	+ * 0 on success, or negative error in case of failure:
	3517	+ *
	3518	+ * -EINVAL if param is invalid
	3519	+ *
	3520	+ * -ENOSPC Not enough space in the header.
	3521	+ *
	3522	+ * -EPERM This helper cannot be used under the
	3523	+ * current sock_ops->op.
	3524	+ *
	3525	+ * void bpf_inode_storage_get(struct bpf_map map, void inode, void value, u64 flags)
	3526	+ * Description
	3527	+ * Get a bpf_local_storage from an inode.
	3528	+ *
	3529	+ * Logically, it could be thought of as getting the value from
	3530	+ * a map with inode as the key. From this
	3531	+ * perspective, the usage is not much different from
	3532	+ * bpf_map_lookup_elem\ (map, &\ inode) except this
	3533	+ * helper enforces the key must be an inode and the map must also
	3534	+ * be a BPF_MAP_TYPE_INODE_STORAGE.
	3535	+ *
	3536	+ * Underneath, the value is stored locally at inode instead of
	3537	+ * the map. The map is used as the bpf-local-storage
	3538	+ * "type". The bpf-local-storage "type" (i.e. the map) is
	3539	+ * searched against all bpf_local_storage residing at inode.
	3540	+ *
	3541	+ * An optional flags (BPF_LOCAL_STORAGE_GET_F_CREATE) can be
	3542	+ * used such that a new bpf_local_storage will be
	3543	+ * created if one does not exist. value can be used
	3544	+ * together with BPF_LOCAL_STORAGE_GET_F_CREATE to specify
	3545	+ * the initial value of a bpf_local_storage. If value is
	3546	+ * NULL, the new bpf_local_storage will be zero initialized.
	3547	+ * Return
	3548	+ * A bpf_local_storage pointer is returned on success.
	3549	+ *
	3550	+ * NULL if not found or there was an error in adding
	3551	+ * a new bpf_local_storage.
	3552	+ *
	3553	+ * int bpf_inode_storage_delete(struct bpf_map map, void inode)
	3554	+ * Description
	3555	+ * Delete a bpf_local_storage from an inode.
	3556	+ * Return
	3557	+ * 0 on success.
	3558	+ *
	3559	+ * -ENOENT if the bpf_local_storage cannot be found.
	3560	+ *
	3561	+ * long bpf_d_path(struct path path, char buf, u32 sz)
	3562	+ * Description
	3563	+ * Return full path for given 'struct path' object, which
	3564	+ * needs to be the kernel BTF 'path' object. The path is
	3565	+ * returned in the provided buffer 'buf' of size 'sz' and
	3566	+ * is zero terminated.
	3567	+ *
	3568	+ * Return
	3569	+ * On success, the strictly positive length of the string,
	3570	+ * including the trailing NUL character. On error, a negative
	3571	+ * value.
	3572	+ *
	3573	+ * long bpf_copy_from_user(void dst, u32 size, const void user_ptr)
	3574	+ * Description
	3575	+ * Read size bytes from user space address user_ptr and store
	3576	+ * the data in dst. This is a wrapper of copy_from_user().
	3577	+ * Return
	3578	+ * 0 on success, or a negative error in case of failure.
3398	3579	*/
3399	3580	#define __BPF_FUNC_MAPPER(FN) \
3400	3581	FN(unspec), \
@@ -3539,6 +3720,13 @@ union bpf_attr { · 138 unmodified lines
3539	3720	FN(skc_to_tcp_request_sock), \
3540	3721	FN(skc_to_udp6_sock), \
3541	3722	FN(get_task_stack), \
	3723	+ FN(load_hdr_opt), \
	3724	+ FN(store_hdr_opt), \
	3725	+ FN(reserve_hdr_opt), \
	3726	+ FN(inode_storage_get), \
	3727	+ FN(inode_storage_delete), \
	3728	+ FN(d_path), \
	3729	+ FN(copy_from_user), \
3542	3730	/* */
3543	3731
3544	3732	/* integer value in 'imm' field of BPF_CALL instruction selects which helper
@@ -3648,9 +3836,13 @@ enum { · 103 unmodified lines
3648	3836	BPF_F_SYSCTL_BASE_NAME = (1ULL << 0),
3649	3837	};
3650	3838
3651		−/* BPF_FUNC_sk_storage_get flags */
	3839	+/* BPF_FUNC_<kernel_obj>_storage_get flags */
3652	3840	enum {
3653		− BPF_SK_STORAGE_GET_F_CREATE = (1ULL << 0),
	3841	+ BPF_LOCAL_STORAGE_GET_F_CREATE = (1ULL << 0),
	3842	+ /* BPF_SK_STORAGE_GET_F_CREATE is only kept for backward compatibility
	3843	+ * and BPF_LOCAL_STORAGE_GET_F_CREATE must be used instead.
	3844	+ */
	3845	+ BPF_SK_STORAGE_GET_F_CREATE = BPF_LOCAL_STORAGE_GET_F_CREATE,
3654	3846	};
3655	3847
3656	3848	/* BPF_FUNC_read_branch_records flags. */
@@ -4071,6 +4263,15 @@ struct bpf_link_info { · 414 unmodified lines
4071	4263	__u64 cgroup_id;
4072	4264	__u32 attach_type;
4073	4265	} cgroup;
	4266	+ struct {
	4267	+ __aligned_u64 target_name; /* in/out: target_name buffer ptr */
	4268	+ __u32 target_name_len; /* in/out: target_name buffer len */
	4269	+ union {
	4270	+ struct {
	4271	+ __u32 map_id;
	4272	+ } map;
	4273	+ };
	4274	+ } iter;
4074	4275	struct {
4075	4276	__u32 netns_ino;
4076	4277	__u32 attach_type;
@@ -4158,6 +4359,36 @@ struct bpf_sock_ops { · 81 unmodified lines
4158	4359	__u64 bytes_received;
4159	4360	__u64 bytes_acked;
4160	4361	__bpf_md_ptr(struct bpf_sock *, sk);
	4362	+ /* [skb_data, skb_data_end) covers the whole TCP header.
	4363	+ *
	4364	+ * BPF_SOCK_OPS_PARSE_HDR_OPT_CB: The packet received
	4365	+ * BPF_SOCK_OPS_HDR_OPT_LEN_CB: Not useful because the
	4366	+ * header has not been written.
	4367	+ * BPF_SOCK_OPS_WRITE_HDR_OPT_CB: The header and options have
	4368	+ * been written so far.
	4369	+ * BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB: The SYNACK that concludes
	4370	+ * the 3WHS.
	4371	+ * BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB: The ACK that concludes
	4372	+ * the 3WHS.
	4373	+ *
	4374	+ * bpf_load_hdr_opt() can also be used to read a particular option.
	4375	+ */
	4376	+ __bpf_md_ptr(void *, skb_data);
	4377	+ __bpf_md_ptr(void *, skb_data_end);
	4378	+ __u32 skb_len; /* The total length of a packet.
	4379	+ * It includes the header, options,
	4380	+ * and payload.
	4381	+ */
	4382	+ __u32 skb_tcp_flags; /* tcp_flags of the header. It provides
	4383	+ * an easy way to check for tcp_flags
	4384	+ * without parsing skb_data.
	4385	+ *
	4386	+ * In particular, the skb_tcp_flags
	4387	+ * will still be available in
	4388	+ * BPF_SOCK_OPS_HDR_OPT_LEN even though
	4389	+ * the outgoing header has not
	4390	+ * been written yet.
	4391	+ */
4161	4392	};
4162	4393
4163	4394	/* Definitions for bpf_sock_ops_cb_flags */
@@ -4166,8 +4397,51 @@ enum { · 2 unmodified lines
4166	4397	BPF_SOCK_OPS_RETRANS_CB_FLAG = (1<<1),
4167	4398	BPF_SOCK_OPS_STATE_CB_FLAG = (1<<2),
4168	4399	BPF_SOCK_OPS_RTT_CB_FLAG = (1<<3),
	4400	+ /* Call bpf for all received TCP headers. The bpf prog will be
	4401	+ * called under sock_ops->op == BPF_SOCK_OPS_PARSE_HDR_OPT_CB
	4402	+ *
	4403	+ * Please refer to the comment in BPF_SOCK_OPS_PARSE_HDR_OPT_CB
	4404	+ * for the header option related helpers that will be useful
	4405	+ * to the bpf programs.
	4406	+ *
	4407	+ * It could be used at the client/active side (i.e. connect() side)
	4408	+ * when the server told it that the server was in syncookie
	4409	+ * mode and required the active side to resend the bpf-written
	4410	+ * options. The active side can keep writing the bpf-options until
	4411	+ * it received a valid packet from the server side to confirm
	4412	+ * the earlier packet (and options) has been received. The later
	4413	+ * example patch is using it like this at the active side when the
	4414	+ * server is in syncookie mode.
	4415	+ *
	4416	+ * The bpf prog will usually turn this off in the common cases.
	4417	+ */
	4418	+ BPF_SOCK_OPS_PARSE_ALL_HDR_OPT_CB_FLAG = (1<<4),
	4419	+ /* Call bpf when kernel has received a header option that
	4420	+ * the kernel cannot handle. The bpf prog will be called under
	4421	+ * sock_ops->op == BPF_SOCK_OPS_PARSE_HDR_OPT_CB.
	4422	+ *
	4423	+ * Please refer to the comment in BPF_SOCK_OPS_PARSE_HDR_OPT_CB
	4424	+ * for the header option related helpers that will be useful
	4425	+ * to the bpf programs.
	4426	+ */
	4427	+ BPF_SOCK_OPS_PARSE_UNKNOWN_HDR_OPT_CB_FLAG = (1<<5),
	4428	+ /* Call bpf when the kernel is writing header options for the
	4429	+ * outgoing packet. The bpf prog will first be called
	4430	+ * to reserve space in a skb under
	4431	+ * sock_ops->op == BPF_SOCK_OPS_HDR_OPT_LEN_CB. Then
	4432	+ * the bpf prog will be called to write the header option(s)
	4433	+ * under sock_ops->op == BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
	4434	+ *
	4435	+ * Please refer to the comment in BPF_SOCK_OPS_HDR_OPT_LEN_CB
	4436	+ * and BPF_SOCK_OPS_WRITE_HDR_OPT_CB for the header option
	4437	+ * related helpers that will be useful to the bpf programs.
	4438	+ *
	4439	+ * The kernel gets its chance to reserve space and write
	4440	+ * options first before the BPF program does.
	4441	+ */
	4442	+ BPF_SOCK_OPS_WRITE_HDR_OPT_CB_FLAG = (1<<6),
4169	4443	/* Mask of all currently supported cb flags */
4170		− BPF_SOCK_OPS_ALL_CB_FLAGS = 0xF,
	4444	+ BPF_SOCK_OPS_ALL_CB_FLAGS = 0x7F,
4171	4445	};
4172	4446
4173	4447	/* List of known BPF sock_ops operators.
@@ -4223,6 +4497,63 @@ enum { · 49 unmodified lines
4223	4497	*/
4224	4498	BPF_SOCK_OPS_RTT_CB, /* Called on every RTT.
4225	4499	*/
	4500	+ BPF_SOCK_OPS_PARSE_HDR_OPT_CB, /* Parse the header option.
	4501	+ * It will be called to handle
	4502	+ * the packets received at
	4503	+ * an already established
	4504	+ * connection.
	4505	+ *
	4506	+ * sock_ops->skb_data:
	4507	+ * Referring to the received skb.
	4508	+ * It covers the TCP header only.
	4509	+ *
	4510	+ * bpf_load_hdr_opt() can also
	4511	+ * be used to search for a
	4512	+ * particular option.
	4513	+ */
	4514	+ BPF_SOCK_OPS_HDR_OPT_LEN_CB, /* Reserve space for writing the
	4515	+ * header option later in
	4516	+ * BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
	4517	+ * Arg1: bool want_cookie. (in
	4518	+ * writing SYNACK only)
	4519	+ *
	4520	+ * sock_ops->skb_data:
	4521	+ * Not available because no header has
	4522	+ * been written yet.
	4523	+ *
	4524	+ * sock_ops->skb_tcp_flags:
	4525	+ * The tcp_flags of the
	4526	+ * outgoing skb. (e.g. SYN, ACK, FIN).
	4527	+ *
	4528	+ * bpf_reserve_hdr_opt() should
	4529	+ * be used to reserve space.
	4530	+ */
	4531	+ BPF_SOCK_OPS_WRITE_HDR_OPT_CB, /* Write the header options
	4532	+ * Arg1: bool want_cookie. (in
	4533	+ * writing SYNACK only)
	4534	+ *
	4535	+ * sock_ops->skb_data:
	4536	+ * Referring to the outgoing skb.
	4537	+ * It covers the TCP header
	4538	+ * that has already been written
	4539	+ * by the kernel and the
	4540	+ * earlier bpf-progs.
	4541	+ *
	4542	+ * sock_ops->skb_tcp_flags:
	4543	+ * The tcp_flags of the outgoing
	4544	+ * skb. (e.g. SYN, ACK, FIN).
	4545	+ *
	4546	+ * bpf_store_hdr_opt() should
	4547	+ * be used to write the
	4548	+ * option.
	4549	+ *
	4550	+ * bpf_load_hdr_opt() can also
	4551	+ * be used to search for a
	4552	+ * particular option that
	4553	+ * has already been written
	4554	+ * by the kernel or the
	4555	+ * earlier bpf-progs.
	4556	+ */
4226	4557	};
4227	4558
4228	4559	/* List of TCP states. There is a build check in net/ipv4/tcp.c to detect
@@ -4250,6 +4581,63 @@ enum { · 21 unmodified lines
4250	4581	enum {
4251	4582	TCP_BPF_IW = 1001, /* Set TCP initial congestion window */
4252	4583	TCP_BPF_SNDCWND_CLAMP = 1002, /* Set sndcwnd_clamp */
	4584	+ TCP_BPF_DELACK_MAX = 1003, /* Max delay ack in usecs */
	4585	+ TCP_BPF_RTO_MIN = 1004, /* Min delay ack in usecs */
	4586	+ /* Copy the SYN pkt to optval
	4587	+ *
	4588	+ * BPF_PROG_TYPE_SOCK_OPS only. It is similar to the
	4589	+ * bpf_getsockopt(TCP_SAVED_SYN) but it does not limit
	4590	+ * to only getting from the saved_syn. It can either get the
	4591	+ * syn packet from:
	4592	+ *
	4593	+ * 1. the just-received SYN packet (only available when writing the
	4594	+ * SYNACK). It will be useful when it is not necessary to
	4595	+ * save the SYN packet for latter use. It is also the only way
	4596	+ * to get the SYN during syncookie mode because the syn
	4597	+ * packet cannot be saved during syncookie.
	4598	+ *
	4599	+ * OR
	4600	+ *
	4601	+ * 2. the earlier saved syn which was done by
	4602	+ * bpf_setsockopt(TCP_SAVE_SYN).
	4603	+ *
	4604	+ * The bpf_getsockopt(TCP_BPF_SYN*) option will hide where the
	4605	+ * SYN packet is obtained.
	4606	+ *
	4607	+ * If the bpf-prog does not need the IP[46] header, the
	4608	+ * bpf-prog can avoid parsing the IP header by using
	4609	+ * TCP_BPF_SYN. Otherwise, the bpf-prog can get both
	4610	+ * IP[46] and TCP header by using TCP_BPF_SYN_IP.
	4611	+ *
	4612	+ * >0: Total number of bytes copied
	4613	+ * -ENOSPC: Not enough space in optval. Only optlen number of
	4614	+ * bytes is copied.
	4615	+ * -ENOENT: The SYN skb is not available now and the earlier SYN pkt
	4616	+ * is not saved by setsockopt(TCP_SAVE_SYN).
	4617	+ */
	4618	+ TCP_BPF_SYN = 1005, /* Copy the TCP header */
	4619	+ TCP_BPF_SYN_IP = 1006, /* Copy the IP[46] and TCP header */
	4620	+ TCP_BPF_SYN_MAC = 1007, /* Copy the MAC, IP[46], and TCP header */
	4621	+};
	4622	+
	4623	+enum {
	4624	+ BPF_LOAD_HDR_OPT_TCP_SYN = (1ULL << 0),
	4625	+};
	4626	+
	4627	+/* args[0] value during BPF_SOCK_OPS_HDR_OPT_LEN_CB and
	4628	+ * BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
	4629	+ */
	4630	+enum {
	4631	+ BPF_WRITE_HDR_TCP_CURRENT_MSS = 1, /* Kernel is finding the
	4632	+ * total option spaces
	4633	+ * required for an established
	4634	+ * sk in order to calculate the
	4635	+ * MSS. No skb is actually
	4636	+ * sent.
	4637	+ */
	4638	+ BPF_WRITE_HDR_TCP_SYNACK_COOKIE = 2, /* Kernel is in syncookie mode
	4639	+ * when sending a SYN.
	4640	+ */
4253	4641	};
4254	4642
4255	4643	struct bpf_perf_event_value {

tools/lib/bpf/Makefile+9−14

@@ -1,6 +1,9 @@
1	1	# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
2	2	# Most of this file is copied from tools/lib/traceevent/Makefile
3	3
	4	+RM ?= rm
	5	+srctree = $(abs_srctree)
	6	+
4	7	LIBBPF_VERSION := $(shell \
5	8	grep -oE '^LIBBPF_([0-9.]+)' libbpf.map \| \
6	9	sort -rV \| head -n1 \| cut -d'_' -f2)
@@ -56,7 +59,7 @@ ifndef VERBOSE · 49 unmodified lines
56	59	endif
57	60
58	61	FEATURE_USER = .libbpf
59		−FEATURE_TESTS = libelf libelf-mmap zlib bpf reallocarray
	62	+FEATURE_TESTS = libelf zlib bpf
60	63	FEATURE_DISPLAY = libelf zlib bpf
61	64
62	65	INCLUDES = -I. -I$(srctree)/tools/include -I$(srctree)/tools/arch/$(ARCH)/include/uapi -I$(srctree)/tools/include/uapi
@@ -98,16 +101,8 @@ else · 35 unmodified lines
98	101	CFLAGS := -g -Wall
99	102	endif
100	103
101		−ifeq ($(feature-libelf-mmap), 1)
102		− override CFLAGS += -DHAVE_LIBELF_MMAP_SUPPORT
103		−endif
104		−
105		−ifeq ($(feature-reallocarray), 0)
106		− override CFLAGS += -DCOMPAT_NEED_REALLOCARRAY
107		−endif
108		−
109	104	# Append required CFLAGS
110		−override CFLAGS += $(EXTRA_WARNINGS)
	105	+override CFLAGS += $(EXTRA_WARNINGS) -Wno-switch-enum
111	106	override CFLAGS += -Werror -Wall
112	107	override CFLAGS += -fPIC
113	108	override CFLAGS += $(INCLUDES)
@@ -196,7 +191,7 @@ $(OUTPUT)libbpf.so.$(LIBBPF_VERSION): $(BPF_IN_SHARED) · 82 unmodified lines
196	191	@ln -sf $(@F) $(OUTPUT)libbpf.so.$(LIBBPF_MAJOR_VERSION)
197	192
198	193	$(OUTPUT)libbpf.a: $(BPF_IN_STATIC)
199		− $(QUIET_LINK)$(RM) $@; $(AR) rcs $@ $^
	194	+ $(QUIET_LINK)$(RM) -f $@; $(AR) rcs $@ $^
200	195
201	196	$(OUTPUT)libbpf.pc:
202	197	$(QUIET_GEN)sed -e "s\|@PREFIX@\|$(prefix)\|" \
@@ -269,10 +264,10 @@ install: install_lib install_pkgconfig install_headers · 66 unmodified lines
269	264	### Cleaning rules
270	265
271	266	config-clean:
272		− $(call QUIET_CLEAN, config)
	267	+ $(call QUIET_CLEAN, feature-detect)
273	268	$(Q)$(MAKE) -C $(srctree)/tools/build/feature/ clean >/dev/null
274	269
275		−clean:
	270	+clean: config-clean
276	271	$(call QUIET_CLEAN, libbpf) $(RM) -rf $(CMD_TARGETS) \
277	272	~ ..d .*.cmd LIBBPF-CFLAGS $(BPF_HELPER_DEFS) \
278	273	$(SHARED_OBJDIR) $(STATIC_OBJDIR) \
@@ -299,7 +294,7 @@ cscope: · 20 unmodified lines
299	294	cscope -b -q -I $(srctree)/include -f cscope.out
300	295
301	296	tags:
302		− rm -f TAGS tags
	297	+ $(RM) -f TAGS tags
303	298	ls .c .h \| xargs $(TAGS_PROG) -a
304	299
305	300	# Declare the contents of the .PHONY variable as phony. We keep that

tools/lib/bpf/bpf.c+0−3

@@ -32,9 +32,6 @@ · 31 unmodified lines
32	32	#include "libbpf.h"
33	33	#include "libbpf_internal.h"
34	34
35		−/* make sure libbpf doesn't use kernel-only integer typedefs */
36		−#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
37		−
38	35	/*
39	36	* When building perf, unistd.h is overridden. __NR_bpf is
40	37	* required to be defined explicitly.

tools/lib/bpf/bpf_core_read.h+101−19

@@ -19,32 +19,52 @@ enum bpf_field_info_kind { · 18 unmodified lines
19	19	BPF_FIELD_RSHIFT_U64 = 5,
20	20	};
21	21
	22	+/* second argument to __builtin_btf_type_id() built-in */
	23	+enum bpf_type_id_kind {
	24	+ BPF_TYPE_ID_LOCAL = 0, /* BTF type ID in local program */
	25	+ BPF_TYPE_ID_TARGET = 1, /* BTF type ID in target kernel */
	26	+};
	27	+
	28	+/* second argument to __builtin_preserve_type_info() built-in */
	29	+enum bpf_type_info_kind {
	30	+ BPF_TYPE_EXISTS = 0, /* type existence in target kernel */
	31	+ BPF_TYPE_SIZE = 1, /* type size in target kernel */
	32	+};
	33	+
	34	+/* second argument to __builtin_preserve_enum_value() built-in */
	35	+enum bpf_enum_value_kind {
	36	+ BPF_ENUMVAL_EXISTS = 0, /* enum value existence in kernel */
	37	+ BPF_ENUMVAL_VALUE = 1, /* enum value value relocation */
	38	+};
	39	+
22	40	#define __CORE_RELO(src, field, info) \
23	41	__builtin_preserve_field_info((src)->field, BPF_FIELD_##info)
24	42
25	43	#if __BYTE_ORDER == __LITTLE_ENDIAN
26	44	#define __CORE_BITFIELD_PROBE_READ(dst, src, fld) \
27		− bpf_probe_read((void *)dst, \
28		− __CORE_RELO(src, fld, BYTE_SIZE), \
29		− (const void *)src + __CORE_RELO(src, fld, BYTE_OFFSET))
	45	+ bpf_probe_read_kernel( \
	46	+ (void *)dst, \
	47	+ __CORE_RELO(src, fld, BYTE_SIZE), \
	48	+ (const void *)src + __CORE_RELO(src, fld, BYTE_OFFSET))
30	49	#else
31	50	/* semantics of LSHIFT_64 assumes loading values into low-ordered bytes, so
32	51	* for big-endian we need to adjust destination pointer accordingly, based on
33	52	* field byte size
34	53	*/
35	54	#define __CORE_BITFIELD_PROBE_READ(dst, src, fld) \
36		− bpf_probe_read((void *)dst + (8 - __CORE_RELO(src, fld, BYTE_SIZE)), \
37		− __CORE_RELO(src, fld, BYTE_SIZE), \
38		− (const void *)src + __CORE_RELO(src, fld, BYTE_OFFSET))
	55	+ bpf_probe_read_kernel( \
	56	+ (void *)dst + (8 - __CORE_RELO(src, fld, BYTE_SIZE)), \
	57	+ __CORE_RELO(src, fld, BYTE_SIZE), \
	58	+ (const void *)src + __CORE_RELO(src, fld, BYTE_OFFSET))
39	59	#endif
40	60
41	61	/*
42	62	* Extract bitfield, identified by s->field, and return its value as u64.
43	63	* All this is done in relocatable manner, so bitfield changes such as
44	64	* signedness, bit size, offset changes, this will be handled automatically.
45		− * This version of macro is using bpf_probe_read() to read underlying integer
46		− * storage. Macro functions as an expression and its return type is
47		− * bpf_probe_read()'s return value: 0, on success, <0 on error.
	65	+ * This version of macro is using bpf_probe_read_kernel() to read underlying
	66	+ * integer storage. Macro functions as an expression and its return type is
	67	+ * bpf_probe_read_kernel()'s return value: 0, on success, <0 on error.
48	68	*/
49	69	#define BPF_CORE_READ_BITFIELD_PROBED(s, field) ({ \
50	70	unsigned long long val = 0; \
@@ -92,15 +112,75 @@ enum bpf_field_info_kind { · 41 unmodified lines
92	112	__builtin_preserve_field_info(field, BPF_FIELD_EXISTS)
93	113
94	114	/*
95		− * Convenience macro to get byte size of a field. Works for integers,
	115	+ * Convenience macro to get the byte size of a field. Works for integers,
96	116	* struct/unions, pointers, arrays, and enums.
97	117	*/
98	118	#define bpf_core_field_size(field) \
99	119	__builtin_preserve_field_info(field, BPF_FIELD_BYTE_SIZE)
100	120
101	121	/*
102		− * bpf_core_read() abstracts away bpf_probe_read() call and captures offset
103		− * relocation for source address using __builtin_preserve_access_index()
	122	+ * Convenience macro to get BTF type ID of a specified type, using a local BTF
	123	+ * information. Return 32-bit unsigned integer with type ID from program's own
	124	+ * BTF. Always succeeds.
	125	+ */
	126	+#define bpf_core_type_id_local(type) \
	127	+ __builtin_btf_type_id((typeof(type) )0, BPF_TYPE_ID_LOCAL)
	128	+
	129	+/*
	130	+ * Convenience macro to get BTF type ID of a target kernel's type that matches
	131	+ * specified local type.
	132	+ * Returns:
	133	+ * - valid 32-bit unsigned type ID in kernel BTF;
	134	+ * - 0, if no matching type was found in a target kernel BTF.
	135	+ */
	136	+#define bpf_core_type_id_kernel(type) \
	137	+ __builtin_btf_type_id((typeof(type) )0, BPF_TYPE_ID_TARGET)
	138	+
	139	+/*
	140	+ * Convenience macro to check that provided named type
	141	+ * (struct/union/enum/typedef) exists in a target kernel.
	142	+ * Returns:
	143	+ * 1, if such type is present in target kernel's BTF;
	144	+ * 0, if no matching type is found.
	145	+ */
	146	+#define bpf_core_type_exists(type) \
	147	+ __builtin_preserve_type_info((typeof(type) )0, BPF_TYPE_EXISTS)
	148	+
	149	+/*
	150	+ * Convenience macro to get the byte size of a provided named type
	151	+ * (struct/union/enum/typedef) in a target kernel.
	152	+ * Returns:
	153	+ * >= 0 size (in bytes), if type is present in target kernel's BTF;
	154	+ * 0, if no matching type is found.
	155	+ */
	156	+#define bpf_core_type_size(type) \
	157	+ __builtin_preserve_type_info((typeof(type) )0, BPF_TYPE_SIZE)
	158	+
	159	+/*
	160	+ * Convenience macro to check that provided enumerator value is defined in
	161	+ * a target kernel.
	162	+ * Returns:
	163	+ * 1, if specified enum type and its enumerator value are present in target
	164	+ * kernel's BTF;
	165	+ * 0, if no matching enum and/or enum value within that enum is found.
	166	+ */
	167	+#define bpf_core_enum_value_exists(enum_type, enum_value) \
	168	+ __builtin_preserve_enum_value((typeof(enum_type) )enum_value, BPF_ENUMVAL_EXISTS)
	169	+
	170	+/*
	171	+ * Convenience macro to get the integer value of an enumerator value in
	172	+ * a target kernel.
	173	+ * Returns:
	174	+ * 64-bit value, if specified enum type and its enumerator value are
	175	+ * present in target kernel's BTF;
	176	+ * 0, if no matching enum and/or enum value within that enum is found.
	177	+ */
	178	+#define bpf_core_enum_value(enum_type, enum_value) \
	179	+ __builtin_preserve_enum_value((typeof(enum_type) )enum_value, BPF_ENUMVAL_VALUE)
	180	+
	181	+/*
	182	+ * bpf_core_read() abstracts away bpf_probe_read_kernel() call and captures
	183	+ * offset relocation for source address using __builtin_preserve_access_index()
104	184	* built-in, provided by Clang.
105	185	*
106	186	* __builtin_preserve_access_index() takes as an argument an expression of
@@ -115,8 +195,8 @@ enum bpf_field_info_kind { · 8 unmodified lines
115	195	* (local) BTF, used to record relocation.
116	196	*/
117	197	#define bpf_core_read(dst, sz, src) \
118		− bpf_probe_read(dst, sz, \
119		− (const void *)__builtin_preserve_access_index(src))
	198	+ bpf_probe_read_kernel(dst, sz, \
	199	+ (const void *)__builtin_preserve_access_index(src))
120	200
121	201	/*
122	202	* bpf_core_read_str() is a thin wrapper around bpf_probe_read_str()
@@ -124,8 +204,8 @@ enum bpf_field_info_kind { · 1 unmodified line
124	204	* argument.
125	205	*/
126	206	#define bpf_core_read_str(dst, sz, src) \
127		− bpf_probe_read_str(dst, sz, \
128		− (const void *)__builtin_preserve_access_index(src))
	207	+ bpf_probe_read_kernel_str(dst, sz, \
	208	+ (const void *)__builtin_preserve_access_index(src))
129	209
130	210	#define ___concat(a, b) a ## b
131	211	#define ___apply(fn, n) ___concat(fn, n)
@@ -239,15 +319,17 @@ enum bpf_field_info_kind { · 107 unmodified lines
239	319	* int x = BPF_CORE_READ(s, a.b.c, d.e, f, g);
240	320	*
241	321	* BPF_CORE_READ will decompose above statement into 4 bpf_core_read (BPF
242		− * CO-RE relocatable bpf_probe_read() wrapper) calls, logically equivalent to:
	322	+ * CO-RE relocatable bpf_probe_read_kernel() wrapper) calls, logically
	323	+ * equivalent to:
243	324	* 1. const void *__t = s->a.b.c;
244	325	* 2. __t = __t->d.e;
245	326	* 3. __t = __t->f;
246	327	* 4. return __t->g;
247	328	*
248	329	* Equivalence is logical, because there is a heavy type casting/preservation
249		− * involved, as well as all the reads are happening through bpf_probe_read()
250		− * calls using __builtin_preserve_access_index() to emit CO-RE relocations.
	330	+ * involved, as well as all the reads are happening through
	331	+ * bpf_probe_read_kernel() calls using __builtin_preserve_access_index() to
	332	+ * emit CO-RE relocations.
251	333	*
252	334	* N.B. Only up to 9 "field accessors" are supported, which should be more
253	335	* than enough for any practical purpose.

tools/lib/bpf/bpf_helpers.h+3−0

@@ -32,6 +32,9 @@ · 31 unmodified lines
32	32	#ifndef __always_inline
33	33	#define __always_inline __attribute__((always_inline))
34	34	#endif
	35	+#ifndef __noinline
	36	+#define __noinline __attribute__((noinline))
	37	+#endif
35	38	#ifndef __weak
36	39	#define __weak __attribute__((weak))
37	40	#endif

tools/lib/bpf/bpf_prog_linfo.c+0−3

@@ -8,9 +8,6 @@ · 7 unmodified lines
8	8	#include "libbpf.h"
9	9	#include "libbpf_internal.h"
10	10
11		−/* make sure libbpf doesn't use kernel-only integer typedefs */
12		−#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
13		−
14	11	struct bpf_prog_linfo {
15	12	void *raw_linfo;
16	13	void *raw_jited_linfo;

tools/lib/bpf/bpf_tracing.h+2−2

@@ -289,9 +289,9 @@ struct pt_regs; · 288 unmodified lines
289	289	#define BPF_KRETPROBE_READ_RET_IP BPF_KPROBE_READ_RET_IP
290	290	#else
291	291	#define BPF_KPROBE_READ_RET_IP(ip, ctx) \
292		− ({ bpf_probe_read(&(ip), sizeof(ip), (void *)PT_REGS_RET(ctx)); })
	292	+ ({ bpf_probe_read_kernel(&(ip), sizeof(ip), (void *)PT_REGS_RET(ctx)); })
293	293	#define BPF_KRETPROBE_READ_RET_IP(ip, ctx) \
294		− ({ bpf_probe_read(&(ip), sizeof(ip), \
	294	+ ({ bpf_probe_read_kernel(&(ip), sizeof(ip), \
295	295	(void *)(PT_REGS_FP(ctx) + sizeof(ip))); })
296	296	#endif
297	297

tools/lib/bpf/btf.c+13−18

@@ -21,9 +21,6 @@ · 20 unmodified lines
21	21	#include "libbpf_internal.h"
22	22	#include "hashmap.h"
23	23
24		−/* make sure libbpf doesn't use kernel-only integer typedefs */
25		−#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
26		−
27	24	#define BTF_MAX_NR_TYPES 0x7fffffffU
28	25	#define BTF_MAX_STR_OFFSET 0x7fffffffU
29	26
@@ -61,7 +58,7 @@ static int btf_add_type(struct btf btf, struct btf_type t) · 31 unmodified lines
61	58	expand_by = max(btf->types_size >> 2, 16U);
62	59	new_size = min(BTF_MAX_NR_TYPES, btf->types_size + expand_by);
63	60
64		− new_types = realloc(btf->types, sizeof(new_types) new_size);
	61	+ new_types = libbpf_reallocarray(btf->types, new_size, sizeof(*new_types));
65	62	if (!new_types)
66	63	return -ENOMEM;
67	64
@@ -1131,14 +1128,14 @@ static int btf_ext_setup_line_info(struct btf_ext *btf_ext) · 1063 unmodified lines
1131	1128	return btf_ext_setup_info(btf_ext, &param);
1132	1129	}
1133	1130
1134		−static int btf_ext_setup_field_reloc(struct btf_ext *btf_ext)
	1131	+static int btf_ext_setup_core_relos(struct btf_ext *btf_ext)
1135	1132	{
1136	1133	struct btf_ext_sec_setup_param param = {
1137		− .off = btf_ext->hdr->field_reloc_off,
1138		− .len = btf_ext->hdr->field_reloc_len,
1139		− .min_rec_size = sizeof(struct bpf_field_reloc),
1140		− .ext_info = &btf_ext->field_reloc_info,
1141		− .desc = "field_reloc",
	1134	+ .off = btf_ext->hdr->core_relo_off,
	1135	+ .len = btf_ext->hdr->core_relo_len,
	1136	+ .min_rec_size = sizeof(struct bpf_core_relo),
	1137	+ .ext_info = &btf_ext->core_relo_info,
	1138	+ .desc = "core_relo",
1142	1139	};
1143	1140
1144	1141	return btf_ext_setup_info(btf_ext, &param);
@@ -1217,10 +1214,9 @@ struct btf_ext btf_ext__new(__u8 data, __u32 size) · 72 unmodified lines
1217	1214	if (err)
1218	1215	goto done;
1219	1216
1220		− if (btf_ext->hdr->hdr_len <
1221		− offsetofend(struct btf_ext_header, field_reloc_len))
	1217	+ if (btf_ext->hdr->hdr_len < offsetofend(struct btf_ext_header, core_relo_len))
1222	1218	goto done;
1223		− err = btf_ext_setup_field_reloc(btf_ext);
	1219	+ err = btf_ext_setup_core_relos(btf_ext);
1224	1220	if (err)
1225	1221	goto done;
1226	1222
@@ -1575,7 +1571,7 @@ static int btf_dedup_hypot_map_add(struct btf_dedup *d, · 348 unmodified lines
1575	1571	__u32 *new_list;
1576	1572
1577	1573	d->hypot_cap += max((size_t)16, d->hypot_cap / 2);
1578		− new_list = realloc(d->hypot_list, sizeof(__u32) * d->hypot_cap);
	1574	+ new_list = libbpf_reallocarray(d->hypot_list, d->hypot_cap, sizeof(__u32));
1579	1575	if (!new_list)
1580	1576	return -ENOMEM;
1581	1577	d->hypot_list = new_list;
@@ -1871,8 +1867,7 @@ static int btf_dedup_strings(struct btf_dedup *d) · 289 unmodified lines
1871	1867	struct btf_str_ptr *new_ptrs;
1872	1868
1873	1869	strs.cap += max(strs.cnt / 2, 16U);
1874		− new_ptrs = realloc(strs.ptrs,
1875		− sizeof(strs.ptrs[0]) * strs.cap);
	1870	+ new_ptrs = libbpf_reallocarray(strs.ptrs, strs.cap, sizeof(strs.ptrs[0]));
1876	1871	if (!new_ptrs) {
1877	1872	err = -ENOMEM;
1878	1873	goto done;
@@ -2957,8 +2952,8 @@ static int btf_dedup_compact_types(struct btf_dedup *d) · 1078 unmodified lines
2957	2952	d->btf->nr_types = next_type_id - 1;
2958	2953	d->btf->types_size = d->btf->nr_types;
2959	2954	d->btf->hdr->type_len = p - types_start;
2960		− new_types = realloc(d->btf->types,
2961		− (1 + d->btf->nr_types) * sizeof(struct btf_type *));
	2955	+ new_types = libbpf_reallocarray(d->btf->types, (1 + d->btf->nr_types),
	2956	+ sizeof(struct btf_type *));
2962	2957	if (!new_types)
2963	2958	return -ENOMEM;
2964	2959	d->btf->types = new_types;

tools/lib/bpf/btf.h+0−38

@@ -24,44 +24,6 @@ struct btf_type; · 23 unmodified lines
24	24
25	25	struct bpf_object;
26	26
27		−/*
28		− * The .BTF.ext ELF section layout defined as
29		− * struct btf_ext_header
30		− * func_info subsection
31		− *
32		− * The func_info subsection layout:
33		− * record size for struct bpf_func_info in the func_info subsection
34		− * struct btf_sec_func_info for section #1
35		− * a list of bpf_func_info records for section #1
36		− * where struct bpf_func_info mimics one in include/uapi/linux/bpf.h
37		− * but may not be identical
38		− * struct btf_sec_func_info for section #2
39		− * a list of bpf_func_info records for section #2
40		− * ......
41		− *
42		− * Note that the bpf_func_info record size in .BTF.ext may not
43		− * be the same as the one defined in include/uapi/linux/bpf.h.
44		− * The loader should ensure that record_size meets minimum
45		− * requirement and pass the record as is to the kernel. The
46		− * kernel will handle the func_info properly based on its contents.
47		− */
48		−struct btf_ext_header {
49		− __u16 magic;
50		− __u8 version;
51		− __u8 flags;
52		− __u32 hdr_len;
53		−
54		− /* All offsets are in bytes relative to the end of this header */
55		− __u32 func_info_off;
56		− __u32 func_info_len;
57		− __u32 line_info_off;
58		− __u32 line_info_len;
59		−
60		− /* optional part of .BTF.ext header */
61		− __u32 field_reloc_off;
62		− __u32 field_reloc_len;
63		−};
64		−
65	27	LIBBPF_API void btf__free(struct btf *btf);
66	28	LIBBPF_API struct btf btf__new(const void data, __u32 size);
67	29	LIBBPF_API struct btf btf__parse(const char path, struct btf_ext **btf_ext);

tools/lib/bpf/btf_dump.c+2−7

@@ -19,9 +19,6 @@ · 18 unmodified lines
19	19	#include "libbpf.h"
20	20	#include "libbpf_internal.h"
21	21
22		−/* make sure libbpf doesn't use kernel-only integer typedefs */
23		−#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
24		−
25	22	static const char PREFIXES[] = "\t\t\t\t\t\t\t\t\t\t\t\t\t";
26	23	static const size_t PREFIX_CNT = sizeof(PREFIXES) - 1;
27	24
@@ -323,8 +320,7 @@ static int btf_dump_add_emit_queue_id(struct btf_dump *d, __u32 id) · 295 unmodified lines
323	320
324	321	if (d->emit_queue_cnt >= d->emit_queue_cap) {
325	322	new_cap = max(16, d->emit_queue_cap * 3 / 2);
326		− new_queue = realloc(d->emit_queue,
327		− new_cap * sizeof(new_queue[0]));
	323	+ new_queue = libbpf_reallocarray(d->emit_queue, new_cap, sizeof(new_queue[0]));
328	324	if (!new_queue)
329	325	return -ENOMEM;
330	326	d->emit_queue = new_queue;
@@ -1003,8 +999,7 @@ static int btf_dump_push_decl_stack_id(struct btf_dump *d, __u32 id) · 672 unmodified lines
1003	999
1004	1000	if (d->decl_stack_cnt >= d->decl_stack_cap) {
1005	1001	new_cap = max(16, d->decl_stack_cap * 3 / 2);
1006		− new_stack = realloc(d->decl_stack,
1007		− new_cap * sizeof(new_stack[0]));
	1002	+ new_stack = libbpf_reallocarray(d->decl_stack, new_cap, sizeof(new_stack[0]));
1008	1003	if (!new_stack)
1009	1004	return -ENOMEM;
1010	1005	d->decl_stack = new_stack;

tools/lib/bpf/hashmap.c+3−0

@@ -15,6 +15,9 @@ · 14 unmodified lines
15	15	/* make sure libbpf doesn't use kernel-only integer typedefs */
16	16	#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
17	17
	18	+/* prevent accidental re-addition of reallocarray() */
	19	+#pragma GCC poison reallocarray
	20	+
18	21	/* start with 4 buckets */
19	22	#define HASHMAP_MIN_CAP_BITS 2
20	23

tools/lib/bpf/libbpf.c+1242−517

@@ -44,7 +44,6 @@ · 43 unmodified lines
44	44	#include <sys/vfs.h>
45	45	#include <sys/utsname.h>
46	46	#include <sys/resource.h>
47		−#include <tools/libc_compat.h>
48	47	#include <libelf.h>
49	48	#include <gelf.h>
50	49	#include <zlib.h>
@@ -56,9 +55,6 @@ · 5 unmodified lines
56	55	#include "libbpf_internal.h"
57	56	#include "hashmap.h"
58	57
59		−/* make sure libbpf doesn't use kernel-only integer typedefs */
60		−#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
61		−
62	58	#ifndef EM_BPF
63	59	#define EM_BPF 247
64	60	#endif
@@ -67,6 +63,8 @@ · 2 unmodified lines
67	63	#define BPF_FS_MAGIC 0xcafe4a11
68	64	#endif
69	65
	66	+#define BPF_INSN_SZ (sizeof(struct bpf_insn))
	67	+
70	68	/* vsprintf() in __base_pr() uses nonliteral format string. It may break
71	69	* compilation if user enables corresponding warning. Disable it explicitly.
72	70	*/
@@ -154,34 +152,35 @@ static void pr_perm_msg(int err) · 81 unmodified lines
154	152	___err; })
155	153	#endif
156	154
157		−#ifdef HAVE_LIBELF_MMAP_SUPPORT
158		−# define LIBBPF_ELF_C_READ_MMAP ELF_C_READ_MMAP
159		−#else
160		−# define LIBBPF_ELF_C_READ_MMAP ELF_C_READ
161		−#endif
162		−
163	155	static inline __u64 ptr_to_u64(const void *ptr)
164	156	{
165	157	return (__u64) (unsigned long) ptr;
166	158	}
167	159
168		−struct bpf_capabilities {
	160	+enum kern_feature_id {
169	161	/* v4.14: kernel support for program & map names. */
170		− __u32 name:1;
	162	+ FEAT_PROG_NAME,
171	163	/* v5.2: kernel support for global data sections. */
172		− __u32 global_data:1;
	164	+ FEAT_GLOBAL_DATA,
	165	+ /* BTF support */
	166	+ FEAT_BTF,
173	167	/* BTF_KIND_FUNC and BTF_KIND_FUNC_PROTO support */
174		− __u32 btf_func:1;
	168	+ FEAT_BTF_FUNC,
175	169	/* BTF_KIND_VAR and BTF_KIND_DATASEC support */
176		− __u32 btf_datasec:1;
177		− /* BPF_F_MMAPABLE is supported for arrays */
178		− __u32 array_mmap:1;
	170	+ FEAT_BTF_DATASEC,
179	171	/* BTF_FUNC_GLOBAL is supported */
180		− __u32 btf_func_global:1;
	172	+ FEAT_BTF_GLOBAL_FUNC,
	173	+ /* BPF_F_MMAPABLE is supported for arrays */
	174	+ FEAT_ARRAY_MMAP,
181	175	/* kernel support for expected_attach_type in BPF_PROG_LOAD */
182		− __u32 exp_attach_type:1;
	176	+ FEAT_EXP_ATTACH_TYPE,
	177	+ /* bpf_probe_read_{kernel,user}[_str] helpers */
	178	+ FEAT_PROBE_READ_KERN,
	179	+ __FEAT_CNT,
183	180	};
184	181
	182	+static bool kernel_supports(enum kern_feature_id feat_id);
	183	+
185	184	enum reloc_type {
186	185	RELO_LD64,
187	186	RELO_CALL,
@@ -209,6 +208,7 @@ struct bpf_sec_def { · 21 unmodified lines
209	208	bool is_exp_attach_type_optional;
210	209	bool is_attachable;
211	210	bool is_attach_btf;
	211	+ bool is_sleepable;
212	212	attach_fn_t attach_fn;
213	213	};
214	214
@@ -253,8 +253,6 @@ struct bpf_program { · 38 unmodified lines
253	253	__u32 func_info_rec_size;
254	254	__u32 func_info_cnt;
255	255
256		− struct bpf_capabilities *caps;
257		−
258	256	void *line_info;
259	257	__u32 line_info_rec_size;
260	258	__u32 line_info_cnt;
@@ -403,6 +401,7 @@ struct bpf_object { · 142 unmodified lines
403	401	Elf_Data *rodata;
404	402	Elf_Data *bss;
405	403	Elf_Data *st_ops_data;
	404	+ size_t shstrndx; /* section index for section name strings */
406	405	size_t strtabidx;
407	406	struct {
408	407	GElf_Shdr shdr;
@@ -436,12 +435,18 @@ struct bpf_object { · 27 unmodified lines
436	435	void *priv;
437	436	bpf_object_clear_priv_t clear_priv;
438	437
439		− struct bpf_capabilities caps;
440		−
441	438	char path[];
442	439	};
443	440	#define obj_elf_valid(o) ((o)->efile.elf)
444	441
	442	+static const char elf_sym_str(const struct bpf_object obj, size_t off);
	443	+static const char elf_sec_str(const struct bpf_object obj, size_t off);
	444	+static Elf_Scn elf_sec_by_idx(const struct bpf_object obj, size_t idx);
	445	+static Elf_Scn elf_sec_by_name(const struct bpf_object obj, const char *name);
	446	+static int elf_sec_hdr(const struct bpf_object obj, Elf_Scn scn, GElf_Shdr *hdr);
	447	+static const char elf_sec_name(const struct bpf_object obj, Elf_Scn *scn);
	448	+static Elf_Data elf_sec_data(const struct bpf_object obj, Elf_Scn *scn);
	449	+
445	450	void bpf_program__unload(struct bpf_program *prog)
446	451	{
447	452	int i;
@@ -503,7 +508,7 @@ static char __bpf_program__pin_name(struct bpf_program prog) · 55 unmodified lines
503	508	}
504	509
505	510	static int
506		−bpf_program__init(void data, size_t size, char section_name, int idx,
	511	+bpf_program__init(void data, size_t size, const char section_name, int idx,
507	512	struct bpf_program *prog)
508	513	{
509	514	const size_t bpf_insn_sz = sizeof(struct bpf_insn);
@@ -552,7 +557,7 @@ errout: · 42 unmodified lines
552	557
553	558	static int
554	559	bpf_object__add_program(struct bpf_object obj, void data, size_t size,
555		− char *section_name, int idx)
	560	+ const char *section_name, int idx)
556	561	{
557	562	struct bpf_program prog, *progs;
558	563	int nr_progs, err;
@@ -561,11 +566,10 @@ bpf_object__add_program(struct bpf_object obj, void data, size_t size, · 2 unmodified lines
561	566	if (err)
562	567	return err;
563	568
564		− prog.caps = &obj->caps;
565	569	progs = obj->programs;
566	570	nr_progs = obj->nr_programs;
567	571
568		− progs = reallocarray(progs, nr_progs + 1, sizeof(progs[0]));
	572	+ progs = libbpf_reallocarray(progs, nr_progs + 1, sizeof(progs[0]));
569	573	if (!progs) {
570	574	/*
571	575	* In this case the original obj->programs
@@ -578,7 +582,7 @@ bpf_object__add_program(struct bpf_object obj, void data, size_t size, · 6 unmodified lines
578	582	return -ENOMEM;
579	583	}
580	584
581		− pr_debug("found program %s\n", prog.section_name);
	585	+ pr_debug("elf: found program '%s'\n", prog.section_name);
582	586	obj->programs = progs;
583	587	obj->nr_programs = nr_progs + 1;
584	588	prog.obj = obj;
@@ -598,8 +602,7 @@ bpf_object__init_prog_names(struct bpf_object *obj) · 13 unmodified lines
598	602
599	603	prog = &obj->programs[pi];
600	604
601		− for (si = 0; si < symbols->d_size / sizeof(GElf_Sym) && !name;
602		− si++) {
	605	+ for (si = 0; si < symbols->d_size / sizeof(GElf_Sym) && !name; si++) {
603	606	GElf_Sym sym;
604	607
605	608	if (!gelf_getsym(symbols, si, &sym))
@@ -609,11 +612,9 @@ bpf_object__init_prog_names(struct bpf_object *obj) · 3 unmodified lines
609	612	if (GELF_ST_BIND(sym.st_info) != STB_GLOBAL)
610	613	continue;
611	614
612		− name = elf_strptr(obj->efile.elf,
613		− obj->efile.strtabidx,
614		− sym.st_name);
	615	+ name = elf_sym_str(obj, sym.st_name);
615	616	if (!name) {
616		− pr_warn("failed to get sym name string for prog %s\n",
	617	+ pr_warn("prog '%s': failed to get symbol name\n",
617	618	prog->section_name);
618	619	return -LIBBPF_ERRNO__LIBELF;
619	620	}
@@ -623,17 +624,14 @@ bpf_object__init_prog_names(struct bpf_object *obj) · 3 unmodified lines
623	624	name = ".text";
624	625
625	626	if (!name) {
626		− pr_warn("failed to find sym for prog %s\n",
	627	+ pr_warn("prog '%s': failed to find program symbol\n",
627	628	prog->section_name);
628	629	return -EINVAL;
629	630	}
630	631
631	632	prog->name = strdup(name);
632		− if (!prog->name) {
633		− pr_warn("failed to allocate memory for prog sym %s\n",
634		− name);
	633	+ if (!prog->name)
635	634	return -ENOMEM;
636		− }
637	635	}
638	636
639	637	return 0;
@@ -1066,13 +1064,18 @@ static void bpf_object__elf_finish(struct bpf_object *obj) · 426 unmodified lines
1066	1064	obj->efile.obj_buf_sz = 0;
1067	1065	}
1068	1066
	1067	+/* if libelf is old and doesn't support mmap(), fall back to read() */
	1068	+#ifndef ELF_C_READ_MMAP
	1069	+#define ELF_C_READ_MMAP ELF_C_READ
	1070	+#endif
	1071	+
1069	1072	static int bpf_object__elf_init(struct bpf_object *obj)
1070	1073	{
1071	1074	int err = 0;
1072	1075	GElf_Ehdr *ep;
1073	1076
1074	1077	if (obj_elf_valid(obj)) {
1075		− pr_warn("elf init: internal error\n");
	1078	+ pr_warn("elf: init internal error\n");
1076	1079	return -LIBBPF_ERRNO__LIBELF;
1077	1080	}
1078	1081
@@ -1090,31 +1093,44 @@ static int bpf_object__elf_init(struct bpf_object *obj) · 11 unmodified lines
1090	1093
1091	1094	err = -errno;
1092	1095	cp = libbpf_strerror_r(err, errmsg, sizeof(errmsg));
1093		− pr_warn("failed to open %s: %s\n", obj->path, cp);
	1096	+ pr_warn("elf: failed to open %s: %s\n", obj->path, cp);
1094	1097	return err;
1095	1098	}
1096	1099
1097		− obj->efile.elf = elf_begin(obj->efile.fd,
1098		− LIBBPF_ELF_C_READ_MMAP, NULL);
	1100	+ obj->efile.elf = elf_begin(obj->efile.fd, ELF_C_READ_MMAP, NULL);
1099	1101	}
1100	1102
1101	1103	if (!obj->efile.elf) {
1102		− pr_warn("failed to open %s as ELF file\n", obj->path);
	1104	+ pr_warn("elf: failed to open %s as ELF file: %s\n", obj->path, elf_errmsg(-1));
1103	1105	err = -LIBBPF_ERRNO__LIBELF;
1104	1106	goto errout;
1105	1107	}
1106	1108
1107	1109	if (!gelf_getehdr(obj->efile.elf, &obj->efile.ehdr)) {
1108		− pr_warn("failed to get EHDR from %s\n", obj->path);
	1110	+ pr_warn("elf: failed to get ELF header from %s: %s\n", obj->path, elf_errmsg(-1));
1109	1111	err = -LIBBPF_ERRNO__FORMAT;
1110	1112	goto errout;
1111	1113	}
1112	1114	ep = &obj->efile.ehdr;
1113	1115
	1116	+ if (elf_getshdrstrndx(obj->efile.elf, &obj->efile.shstrndx)) {
	1117	+ pr_warn("elf: failed to get section names section index for %s: %s\n",
	1118	+ obj->path, elf_errmsg(-1));
	1119	+ err = -LIBBPF_ERRNO__FORMAT;
	1120	+ goto errout;
	1121	+ }
	1122	+
	1123	+ /* Elf is corrupted/truncated, avoid calling elf_strptr. */
	1124	+ if (!elf_rawdata(elf_getscn(obj->efile.elf, obj->efile.shstrndx), NULL)) {
	1125	+ pr_warn("elf: failed to get section names strings from %s: %s\n",
	1126	+ obj->path, elf_errmsg(-1));
	1127	+ return -LIBBPF_ERRNO__FORMAT;
	1128	+ }
	1129	+
1114	1130	/* Old LLVM set e_machine to EM_NONE */
1115	1131	if (ep->e_type != ET_REL \|\|
1116	1132	(ep->e_machine && ep->e_machine != EM_BPF)) {
1117		− pr_warn("%s is not an eBPF object file\n", obj->path);
	1133	+ pr_warn("elf: %s is not a valid eBPF object file\n", obj->path);
1118	1134	err = -LIBBPF_ERRNO__FORMAT;
1119	1135	goto errout;
1120	1136	}
@@ -1136,7 +1152,7 @@ static int bpf_object__check_endianness(struct bpf_object *obj) · 15 unmodified lines
1136	1152	#else
1137	1153	# error "Unrecognized __BYTE_ORDER__"
1138	1154	#endif
1139		− pr_warn("endianness mismatch.\n");
	1155	+ pr_warn("elf: endianness mismatch in %s.\n", obj->path);
1140	1156	return -LIBBPF_ERRNO__ENDIAN;
1141	1157	}
1142	1158
@@ -1171,55 +1187,10 @@ static bool bpf_map_type__is_map_in_map(enum bpf_map_type type) · 28 unmodified lines
1171	1187	return false;
1172	1188	}
1173	1189
1174		−static int bpf_object_search_section_size(const struct bpf_object *obj,
1175		− const char name, size_t d_size)
1176		−{
1177		− const GElf_Ehdr *ep = &obj->efile.ehdr;
1178		− Elf *elf = obj->efile.elf;
1179		− Elf_Scn *scn = NULL;
1180		− int idx = 0;
1181		−
1182		− while ((scn = elf_nextscn(elf, scn)) != NULL) {
1183		− const char *sec_name;
1184		− Elf_Data *data;
1185		− GElf_Shdr sh;
1186		−
1187		− idx++;
1188		− if (gelf_getshdr(scn, &sh) != &sh) {
1189		− pr_warn("failed to get section(%d) header from %s\n",
1190		− idx, obj->path);
1191		− return -EIO;
1192		− }
1193		−
1194		− sec_name = elf_strptr(elf, ep->e_shstrndx, sh.sh_name);
1195		− if (!sec_name) {
1196		− pr_warn("failed to get section(%d) name from %s\n",
1197		− idx, obj->path);
1198		− return -EIO;
1199		− }
1200		−
1201		− if (strcmp(name, sec_name))
1202		− continue;
1203		−
1204		− data = elf_getdata(scn, 0);
1205		− if (!data) {
1206		− pr_warn("failed to get section(%d) data from %s(%s)\n",
1207		− idx, name, obj->path);
1208		− return -EIO;
1209		− }
1210		−
1211		− *d_size = data->d_size;
1212		− return 0;
1213		− }
1214		−
1215		− return -ENOENT;
1216		−}
1217		−
1218	1190	int bpf_object__section_size(const struct bpf_object obj, const char name,
1219	1191	__u32 *size)
1220	1192	{
1221	1193	int ret = -ENOENT;
1222		− size_t d_size;
1223	1194
1224	1195	*size = 0;
1225	1196	if (!name) {
@@ -1237,9 +1208,13 @@ int bpf_object__section_size(const struct bpf_object obj, const char name, · 11 unmodified lines
1237	1208	if (obj->efile.st_ops_data)
1238	1209	*size = obj->efile.st_ops_data->d_size;
1239	1210	} else {
1240		− ret = bpf_object_search_section_size(obj, name, &d_size);
1241		− if (!ret)
1242		− *size = d_size;
	1211	+ Elf_Scn *scn = elf_sec_by_name(obj, name);
	1212	+ Elf_Data *data = elf_sec_data(obj, scn);
	1213	+
	1214	+ if (data) {
	1215	+ ret = 0; /* found it */
	1216	+ *size = data->d_size;
	1217	+ }
1243	1218	}
1244	1219
1245	1220	return *size ? 0 : ret;
@@ -1264,8 +1239,7 @@ int bpf_object__variable_offset(const struct bpf_object obj, const char name, · 18 unmodified lines
1264	1239	GELF_ST_TYPE(sym.st_info) != STT_OBJECT)
1265	1240	continue;
1266	1241
1267		− sname = elf_strptr(obj->efile.elf, obj->efile.strtabidx,
1268		− sym.st_name);
	1242	+ sname = elf_sym_str(obj, sym.st_name);
1269	1243	if (!sname) {
1270	1244	pr_warn("failed to get sym name string for var %s\n",
1271	1245	name);
@@ -1290,7 +1264,7 @@ static struct bpf_map bpf_object__add_map(struct bpf_object obj) · 18 unmodified lines
1290	1264	return &obj->maps[obj->nr_maps++];
1291	1265
1292	1266	new_cap = max((size_t)4, obj->maps_cap * 3 / 2);
1293		− new_maps = realloc(obj->maps, new_cap * sizeof(*obj->maps));
	1267	+ new_maps = libbpf_reallocarray(obj->maps, new_cap, sizeof(*obj->maps));
1294	1268	if (!new_maps) {
1295	1269	pr_warn("alloc maps for object failed\n");
1296	1270	return ERR_PTR(-ENOMEM);
@@ -1742,12 +1716,12 @@ static int bpf_object__init_user_maps(struct bpf_object *obj, bool strict) · 445 unmodified lines
1742	1716	if (!symbols)
1743	1717	return -EINVAL;
1744	1718
1745		− scn = elf_getscn(obj->efile.elf, obj->efile.maps_shndx);
1746		− if (scn)
1747		− data = elf_getdata(scn, NULL);
	1719	+
	1720	+ scn = elf_sec_by_idx(obj, obj->efile.maps_shndx);
	1721	+ data = elf_sec_data(obj, scn);
1748	1722	if (!scn \|\| !data) {
1749		− pr_warn("failed to get Elf_Data from map section %d\n",
1750		− obj->efile.maps_shndx);
	1723	+ pr_warn("elf: failed to get legacy map definitions for %s\n",
	1724	+ obj->path);
1751	1725	return -EINVAL;
1752	1726	}
1753	1727
@@ -1769,12 +1743,12 @@ static int bpf_object__init_user_maps(struct bpf_object *obj, bool strict) · 15 unmodified lines
1769	1743	nr_maps++;
1770	1744	}
1771	1745	/* Assume equally sized map definitions */
1772		− pr_debug("maps in %s: %d maps in %zd bytes\n",
1773		− obj->path, nr_maps, data->d_size);
	1746	+ pr_debug("elf: found %d legacy map definitions (%zd bytes) in %s\n",
	1747	+ nr_maps, data->d_size, obj->path);
1774	1748
1775	1749	if (!data->d_size \|\| nr_maps == 0 \|\| (data->d_size % nr_maps) != 0) {
1776		− pr_warn("unable to determine map definition size section %s, %d maps in %zd bytes\n",
1777		− obj->path, nr_maps, data->d_size);
	1750	+ pr_warn("elf: unable to determine legacy map definition size in %s\n",
	1751	+ obj->path);
1778	1752	return -EINVAL;
1779	1753	}
1780	1754	map_def_sz = data->d_size / nr_maps;
@@ -1795,8 +1769,7 @@ static int bpf_object__init_user_maps(struct bpf_object *obj, bool strict) · 14 unmodified lines
1795	1769	if (IS_ERR(map))
1796	1770	return PTR_ERR(map);
1797	1771
1798		− map_name = elf_strptr(obj->efile.elf, obj->efile.strtabidx,
1799		− sym.st_name);
	1772	+ map_name = elf_sym_str(obj, sym.st_name);
1800	1773	if (!map_name) {
1801	1774	pr_warn("failed to get map #%d name sym string for obj %s\n",
1802	1775	i, obj->path);
@@ -1884,6 +1857,29 @@ resolve_func_ptr(const struct btf btf, __u32 id, __u32 res_id) · 81 unmodified lines
1884	1857	return btf_is_func_proto(t) ? t : NULL;
1885	1858	}
1886	1859
	1860	+static const char btf_kind_str(const struct btf_type t)
	1861	+{
	1862	+ switch (btf_kind(t)) {
	1863	+ case BTF_KIND_UNKN: return "void";
	1864	+ case BTF_KIND_INT: return "int";
	1865	+ case BTF_KIND_PTR: return "ptr";
	1866	+ case BTF_KIND_ARRAY: return "array";
	1867	+ case BTF_KIND_STRUCT: return "struct";
	1868	+ case BTF_KIND_UNION: return "union";
	1869	+ case BTF_KIND_ENUM: return "enum";
	1870	+ case BTF_KIND_FWD: return "fwd";
	1871	+ case BTF_KIND_TYPEDEF: return "typedef";
	1872	+ case BTF_KIND_VOLATILE: return "volatile";
	1873	+ case BTF_KIND_CONST: return "const";
	1874	+ case BTF_KIND_RESTRICT: return "restrict";
	1875	+ case BTF_KIND_FUNC: return "func";
	1876	+ case BTF_KIND_FUNC_PROTO: return "func_proto";
	1877	+ case BTF_KIND_VAR: return "var";
	1878	+ case BTF_KIND_DATASEC: return "datasec";
	1879	+ default: return "unknown";
	1880	+ }
	1881	+}
	1882	+
1887	1883	/*
1888	1884	* Fetch integer attribute of BTF map definition. Such attributes are
1889	1885	* represented using a pointer to an array, in which dimensionality of array
@@ -1900,8 +1896,8 @@ static bool get_map_field_int(const char map_name, const struct btf btf, · 10 unmodified lines
1900	1896	const struct btf_type *arr_t;
1901	1897
1902	1898	if (!btf_is_ptr(t)) {
1903		− pr_warn("map '%s': attr '%s': expected PTR, got %u.\n",
1904		− map_name, name, btf_kind(t));
	1899	+ pr_warn("map '%s': attr '%s': expected PTR, got %s.\n",
	1900	+ map_name, name, btf_kind_str(t));
1905	1901	return false;
1906	1902	}
1907	1903
@@ -1912,8 +1908,8 @@ static bool get_map_field_int(const char map_name, const struct btf btf, · 4 unmodified lines
1912	1908	return false;
1913	1909	}
1914	1910	if (!btf_is_array(arr_t)) {
1915		− pr_warn("map '%s': attr '%s': expected ARRAY, got %u.\n",
1916		− map_name, name, btf_kind(arr_t));
	1911	+ pr_warn("map '%s': attr '%s': expected ARRAY, got %s.\n",
	1912	+ map_name, name, btf_kind_str(arr_t));
1917	1913	return false;
1918	1914	}
1919	1915	arr_info = btf_array(arr_t);
@@ -1924,7 +1920,7 @@ static bool get_map_field_int(const char map_name, const struct btf btf, · 4 unmodified lines
1924	1920	static int build_map_pin_path(struct bpf_map map, const char path)
1925	1921	{
1926	1922	char buf[PATH_MAX];
1927		− int err, len;
	1923	+ int len;
1928	1924
1929	1925	if (!path)
1930	1926	path = "/sys/fs/bpf";
@@ -1935,11 +1931,7 @@ static int build_map_pin_path(struct bpf_map map, const char path) · 4 unmodified lines
1935	1931	else if (len >= PATH_MAX)
1936	1932	return -ENAMETOOLONG;
1937	1933
1938		− err = bpf_map__set_pin_path(map, buf);
1939		− if (err)
1940		− return err;
1941		−
1942		− return 0;
	1934	+ return bpf_map__set_pin_path(map, buf);
1943	1935	}
1944	1936
1945	1937
@@ -2007,8 +1999,8 @@ static int parse_btf_map_def(struct bpf_object *obj, · 61 unmodified lines
2007	1999	return -EINVAL;
2008	2000	}
2009	2001	if (!btf_is_ptr(t)) {
2010		− pr_warn("map '%s': key spec is not PTR: %u.\n",
2011		− map->name, btf_kind(t));
	2002	+ pr_warn("map '%s': key spec is not PTR: %s.\n",
	2003	+ map->name, btf_kind_str(t));
2012	2004	return -EINVAL;
2013	2005	}
2014	2006	sz = btf__resolve_size(obj->btf, t->type);
@@ -2049,8 +2041,8 @@ static int parse_btf_map_def(struct bpf_object *obj, · 34 unmodified lines
2049	2041	return -EINVAL;
2050	2042	}
2051	2043	if (!btf_is_ptr(t)) {
2052		− pr_warn("map '%s': value spec is not PTR: %u.\n",
2053		− map->name, btf_kind(t));
	2044	+ pr_warn("map '%s': value spec is not PTR: %s.\n",
	2045	+ map->name, btf_kind_str(t));
2054	2046	return -EINVAL;
2055	2047	}
2056	2048	sz = btf__resolve_size(obj->btf, t->type);
@@ -2107,14 +2099,14 @@ static int parse_btf_map_def(struct bpf_object *obj, · 50 unmodified lines
2107	2099	t = skip_mods_and_typedefs(obj->btf, btf_array(t)->type,
2108	2100	NULL);
2109	2101	if (!btf_is_ptr(t)) {
2110		− pr_warn("map '%s': map-in-map inner def is of unexpected kind %u.\n",
2111		− map->name, btf_kind(t));
	2102	+ pr_warn("map '%s': map-in-map inner def is of unexpected kind %s.\n",
	2103	+ map->name, btf_kind_str(t));
2112	2104	return -EINVAL;
2113	2105	}
2114	2106	t = skip_mods_and_typedefs(obj->btf, t->type, NULL);
2115	2107	if (!btf_is_struct(t)) {
2116		− pr_warn("map '%s': map-in-map inner def is of unexpected kind %u.\n",
2117		− map->name, btf_kind(t));
	2108	+ pr_warn("map '%s': map-in-map inner def is of unexpected kind %s.\n",
	2109	+ map->name, btf_kind_str(t));
2118	2110	return -EINVAL;
2119	2111	}
2120	2112
@@ -2205,8 +2197,8 @@ static int bpf_object__init_user_btf_map(struct bpf_object *obj, · 84 unmodified lines
2205	2197	return -EINVAL;
2206	2198	}
2207	2199	if (!btf_is_var(var)) {
2208		− pr_warn("map '%s': unexpected var kind %u.\n",
2209		− map_name, btf_kind(var));
	2200	+ pr_warn("map '%s': unexpected var kind %s.\n",
	2201	+ map_name, btf_kind_str(var));
2210	2202	return -EINVAL;
2211	2203	}
2212	2204	if (var_extra->linkage != BTF_VAR_GLOBAL_ALLOCATED &&
@@ -2218,8 +2210,8 @@ static int bpf_object__init_user_btf_map(struct bpf_object *obj, · 5 unmodified lines
2218	2210
2219	2211	def = skip_mods_and_typedefs(obj->btf, var->type, NULL);
2220	2212	if (!btf_is_struct(def)) {
2221		− pr_warn("map '%s': unexpected def kind %u.\n",
2222		− map_name, btf_kind(var));
	2213	+ pr_warn("map '%s': unexpected def kind %s.\n",
	2214	+ map_name, btf_kind_str(var));
2223	2215	return -EINVAL;
2224	2216	}
2225	2217	if (def->size > vi->size) {
@@ -2259,12 +2251,11 @@ static int bpf_object__init_user_btf_maps(struct bpf_object *obj, bool strict, · 33 unmodified lines
2259	2251	if (obj->efile.btf_maps_shndx < 0)
2260	2252	return 0;
2261	2253
2262		− scn = elf_getscn(obj->efile.elf, obj->efile.btf_maps_shndx);
2263		− if (scn)
2264		− data = elf_getdata(scn, NULL);
	2254	+ scn = elf_sec_by_idx(obj, obj->efile.btf_maps_shndx);
	2255	+ data = elf_sec_data(obj, scn);
2265	2256	if (!scn \|\| !data) {
2266		− pr_warn("failed to get Elf_Data from map section %d (%s)\n",
2267		− obj->efile.btf_maps_shndx, MAPS_ELF_SEC);
	2257	+ pr_warn("elf: failed to get %s map definitions for %s\n",
	2258	+ MAPS_ELF_SEC, obj->path);
2268	2259	return -EINVAL;
2269	2260	}
2270	2261
@@ -2322,36 +2313,28 @@ static int bpf_object__init_maps(struct bpf_object *obj, · 51 unmodified lines
2322	2313
2323	2314	static bool section_have_execinstr(struct bpf_object *obj, int idx)
2324	2315	{
2325		− Elf_Scn *scn;
2326	2316	GElf_Shdr sh;
2327	2317
2328		− scn = elf_getscn(obj->efile.elf, idx);
2329		− if (!scn)
2330		− return false;
2331		−
2332		− if (gelf_getshdr(scn, &sh) != &sh)
	2318	+ if (elf_sec_hdr(obj, elf_sec_by_idx(obj, idx), &sh))
2333	2319	return false;
2334	2320
2335		− if (sh.sh_flags & SHF_EXECINSTR)
2336		− return true;
2337		−
2338		− return false;
	2321	+ return sh.sh_flags & SHF_EXECINSTR;
2339	2322	}
2340	2323
2341	2324	static bool btf_needs_sanitization(struct bpf_object *obj)
2342	2325	{
2343		− bool has_func_global = obj->caps.btf_func_global;
2344		− bool has_datasec = obj->caps.btf_datasec;
2345		− bool has_func = obj->caps.btf_func;
	2326	+ bool has_func_global = kernel_supports(FEAT_BTF_GLOBAL_FUNC);
	2327	+ bool has_datasec = kernel_supports(FEAT_BTF_DATASEC);
	2328	+ bool has_func = kernel_supports(FEAT_BTF_FUNC);
2346	2329
2347	2330	return !has_func \|\| !has_datasec \|\| !has_func_global;
2348	2331	}
2349	2332
2350	2333	static void bpf_object__sanitize_btf(struct bpf_object obj, struct btf btf)
2351	2334	{
2352		− bool has_func_global = obj->caps.btf_func_global;
2353		− bool has_datasec = obj->caps.btf_datasec;
2354		− bool has_func = obj->caps.btf_func;
	2335	+ bool has_func_global = kernel_supports(FEAT_BTF_GLOBAL_FUNC);
	2336	+ bool has_datasec = kernel_supports(FEAT_BTF_DATASEC);
	2337	+ bool has_func = kernel_supports(FEAT_BTF_FUNC);
2355	2338	struct btf_type *t;
2356	2339	int i, j, vlen;
2357	2340
@@ -2499,7 +2482,7 @@ static int bpf_object__load_vmlinux_btf(struct bpf_object *obj) · 141 unmodified lines
2499	2482	int err;
2500	2483
2501	2484	/* CO-RE relocations need kernel BTF */
2502		− if (obj->btf_ext && obj->btf_ext->field_reloc_info.len)
	2485	+ if (obj->btf_ext && obj->btf_ext->core_relo_info.len)
2503	2486	need_vmlinux_btf = true;
2504	2487
2505	2488	bpf_object__for_each_program(prog, obj) {
@@ -2533,6 +2516,15 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) · 27 unmodified lines
2533	2516	if (!obj->btf)
2534	2517	return 0;
2535	2518
	2519	+ if (!kernel_supports(FEAT_BTF)) {
	2520	+ if (kernel_needs_btf(obj)) {
	2521	+ err = -EOPNOTSUPP;
	2522	+ goto report;
	2523	+ }
	2524	+ pr_debug("Kernel doesn't support BTF, skipping uploading it.\n");
	2525	+ return 0;
	2526	+ }
	2527	+
2536	2528	sanitize = btf_needs_sanitization(obj);
2537	2529	if (sanitize) {
2538	2530	const void *raw_data;
@@ -2558,6 +2550,7 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) · 19 unmodified lines
2558	2550	}
2559	2551	btf__free(kern_btf);
2560	2552	}
	2553	+report:
2561	2554	if (err) {
2562	2555	btf_mandatory = kernel_needs_btf(obj);
2563	2556	pr_warn("Error loading .BTF into kernel: %d. %s\n", err,
@@ -2569,61 +2562,199 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) · 5 unmodified lines
2569	2562	return err;
2570	2563	}
2571	2564
	2565	+static const char elf_sym_str(const struct bpf_object obj, size_t off)
	2566	+{
	2567	+ const char *name;
	2568	+
	2569	+ name = elf_strptr(obj->efile.elf, obj->efile.strtabidx, off);
	2570	+ if (!name) {
	2571	+ pr_warn("elf: failed to get section name string at offset %zu from %s: %s\n",
	2572	+ off, obj->path, elf_errmsg(-1));
	2573	+ return NULL;
	2574	+ }
	2575	+
	2576	+ return name;
	2577	+}
	2578	+
	2579	+static const char elf_sec_str(const struct bpf_object obj, size_t off)
	2580	+{
	2581	+ const char *name;
	2582	+
	2583	+ name = elf_strptr(obj->efile.elf, obj->efile.shstrndx, off);
	2584	+ if (!name) {
	2585	+ pr_warn("elf: failed to get section name string at offset %zu from %s: %s\n",
	2586	+ off, obj->path, elf_errmsg(-1));
	2587	+ return NULL;
	2588	+ }
	2589	+
	2590	+ return name;
	2591	+}
	2592	+
	2593	+static Elf_Scn elf_sec_by_idx(const struct bpf_object obj, size_t idx)
	2594	+{
	2595	+ Elf_Scn *scn;
	2596	+
	2597	+ scn = elf_getscn(obj->efile.elf, idx);
	2598	+ if (!scn) {
	2599	+ pr_warn("elf: failed to get section(%zu) from %s: %s\n",
	2600	+ idx, obj->path, elf_errmsg(-1));
	2601	+ return NULL;
	2602	+ }
	2603	+ return scn;
	2604	+}
	2605	+
	2606	+static Elf_Scn elf_sec_by_name(const struct bpf_object obj, const char *name)
	2607	+{
	2608	+ Elf_Scn *scn = NULL;
	2609	+ Elf *elf = obj->efile.elf;
	2610	+ const char *sec_name;
	2611	+
	2612	+ while ((scn = elf_nextscn(elf, scn)) != NULL) {
	2613	+ sec_name = elf_sec_name(obj, scn);
	2614	+ if (!sec_name)
	2615	+ return NULL;
	2616	+
	2617	+ if (strcmp(sec_name, name) != 0)
	2618	+ continue;
	2619	+
	2620	+ return scn;
	2621	+ }
	2622	+ return NULL;
	2623	+}
	2624	+
	2625	+static int elf_sec_hdr(const struct bpf_object obj, Elf_Scn scn, GElf_Shdr *hdr)
	2626	+{
	2627	+ if (!scn)
	2628	+ return -EINVAL;
	2629	+
	2630	+ if (gelf_getshdr(scn, hdr) != hdr) {
	2631	+ pr_warn("elf: failed to get section(%zu) header from %s: %s\n",
	2632	+ elf_ndxscn(scn), obj->path, elf_errmsg(-1));
	2633	+ return -EINVAL;
	2634	+ }
	2635	+
	2636	+ return 0;
	2637	+}
	2638	+
	2639	+static const char elf_sec_name(const struct bpf_object obj, Elf_Scn *scn)
	2640	+{
	2641	+ const char *name;
	2642	+ GElf_Shdr sh;
	2643	+
	2644	+ if (!scn)
	2645	+ return NULL;
	2646	+
	2647	+ if (elf_sec_hdr(obj, scn, &sh))
	2648	+ return NULL;
	2649	+
	2650	+ name = elf_sec_str(obj, sh.sh_name);
	2651	+ if (!name) {
	2652	+ pr_warn("elf: failed to get section(%zu) name from %s: %s\n",
	2653	+ elf_ndxscn(scn), obj->path, elf_errmsg(-1));
	2654	+ return NULL;
	2655	+ }
	2656	+
	2657	+ return name;
	2658	+}
	2659	+
	2660	+static Elf_Data elf_sec_data(const struct bpf_object obj, Elf_Scn *scn)
	2661	+{
	2662	+ Elf_Data *data;
	2663	+
	2664	+ if (!scn)
	2665	+ return NULL;
	2666	+
	2667	+ data = elf_getdata(scn, 0);
	2668	+ if (!data) {
	2669	+ pr_warn("elf: failed to get section(%zu) %s data from %s: %s\n",
	2670	+ elf_ndxscn(scn), elf_sec_name(obj, scn) ?: "<?>",
	2671	+ obj->path, elf_errmsg(-1));
	2672	+ return NULL;
	2673	+ }
	2674	+
	2675	+ return data;
	2676	+}
	2677	+
	2678	+static bool is_sec_name_dwarf(const char *name)
	2679	+{
	2680	+ /* approximation, but the actual list is too long */
	2681	+ return strncmp(name, ".debug_", sizeof(".debug_") - 1) == 0;
	2682	+}
	2683	+
	2684	+static bool ignore_elf_section(GElf_Shdr hdr, const char name)
	2685	+{
	2686	+ /* no special handling of .strtab */
	2687	+ if (hdr->sh_type == SHT_STRTAB)
	2688	+ return true;
	2689	+
	2690	+ /* ignore .llvm_addrsig section as well */
	2691	+ if (hdr->sh_type == 0x6FFF4C03 /* SHT_LLVM_ADDRSIG */)
	2692	+ return true;
	2693	+
	2694	+ /* no subprograms will lead to an empty .text section, ignore it */
	2695	+ if (hdr->sh_type == SHT_PROGBITS && hdr->sh_size == 0 &&
	2696	+ strcmp(name, ".text") == 0)
	2697	+ return true;
	2698	+
	2699	+ /* DWARF sections */
	2700	+ if (is_sec_name_dwarf(name))
	2701	+ return true;
	2702	+
	2703	+ if (strncmp(name, ".rel", sizeof(".rel") - 1) == 0) {
	2704	+ name += sizeof(".rel") - 1;
	2705	+ /* DWARF section relocations */
	2706	+ if (is_sec_name_dwarf(name))
	2707	+ return true;
	2708	+
	2709	+ /* .BTF and .BTF.ext don't need relocations */
	2710	+ if (strcmp(name, BTF_ELF_SEC) == 0 \|\|
	2711	+ strcmp(name, BTF_EXT_ELF_SEC) == 0)
	2712	+ return true;
	2713	+ }
	2714	+
	2715	+ return false;
	2716	+}
	2717	+
2572	2718	static int bpf_object__elf_collect(struct bpf_object *obj)
2573	2719	{
2574	2720	Elf *elf = obj->efile.elf;
2575		− GElf_Ehdr *ep = &obj->efile.ehdr;
2576	2721	Elf_Data *btf_ext_data = NULL;
2577	2722	Elf_Data *btf_data = NULL;
2578	2723	Elf_Scn *scn = NULL;
2579	2724	int idx = 0, err = 0;
2580	2725
2581		− /* Elf is corrupted/truncated, avoid calling elf_strptr. */
2582		− if (!elf_rawdata(elf_getscn(elf, ep->e_shstrndx), NULL)) {
2583		− pr_warn("failed to get e_shstrndx from %s\n", obj->path);
2584		− return -LIBBPF_ERRNO__FORMAT;
2585		− }
2586		−
2587	2726	while ((scn = elf_nextscn(elf, scn)) != NULL) {
2588		− char *name;
	2727	+ const char *name;
2589	2728	GElf_Shdr sh;
2590	2729	Elf_Data *data;
2591	2730
2592	2731	idx++;
2593		− if (gelf_getshdr(scn, &sh) != &sh) {
2594		− pr_warn("failed to get section(%d) header from %s\n",
2595		− idx, obj->path);
	2732	+
	2733	+ if (elf_sec_hdr(obj, scn, &sh))
2596	2734	return -LIBBPF_ERRNO__FORMAT;
2597		− }
2598	2735
2599		− name = elf_strptr(elf, ep->e_shstrndx, sh.sh_name);
2600		− if (!name) {
2601		− pr_warn("failed to get section(%d) name from %s\n",
2602		− idx, obj->path);
	2736	+ name = elf_sec_str(obj, sh.sh_name);
	2737	+ if (!name)
2603	2738	return -LIBBPF_ERRNO__FORMAT;
2604		− }
2605	2739
2606		− data = elf_getdata(scn, 0);
2607		− if (!data) {
2608		− pr_warn("failed to get section(%d) data from %s(%s)\n",
2609		− idx, name, obj->path);
	2740	+ if (ignore_elf_section(&sh, name))
	2741	+ continue;
	2742	+
	2743	+ data = elf_sec_data(obj, scn);
	2744	+ if (!data)
2610	2745	return -LIBBPF_ERRNO__FORMAT;
2611		− }
2612		− pr_debug("section(%d) %s, size %ld, link %d, flags %lx, type=%d\n",
	2746	+
	2747	+ pr_debug("elf: section(%d) %s, size %ld, link %d, flags %lx, type=%d\n",
2613	2748	idx, name, (unsigned long)data->d_size,
2614	2749	(int)sh.sh_link, (unsigned long)sh.sh_flags,
2615	2750	(int)sh.sh_type);
2616	2751
2617	2752	if (strcmp(name, "license") == 0) {
2618		− err = bpf_object__init_license(obj,
2619		− data->d_buf,
2620		− data->d_size);
	2753	+ err = bpf_object__init_license(obj, data->d_buf, data->d_size);
2621	2754	if (err)
2622	2755	return err;
2623	2756	} else if (strcmp(name, "version") == 0) {
2624		− err = bpf_object__init_kversion(obj,
2625		− data->d_buf,
2626		− data->d_size);
	2757	+ err = bpf_object__init_kversion(obj, data->d_buf, data->d_size);
2627	2758	if (err)
2628	2759	return err;
2629	2760	} else if (strcmp(name, "maps") == 0) {
@@ -2636,8 +2767,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj) · 6 unmodified lines
2636	2767	btf_ext_data = data;
2637	2768	} else if (sh.sh_type == SHT_SYMTAB) {
2638	2769	if (obj->efile.symbols) {
2639		− pr_warn("bpf: multiple SYMTAB in %s\n",
2640		− obj->path);
	2770	+ pr_warn("elf: multiple symbol tables in %s\n", obj->path);
2641	2771	return -LIBBPF_ERRNO__FORMAT;
2642	2772	}
2643	2773	obj->efile.symbols = data;
@@ -2650,16 +2780,8 @@ static int bpf_object__elf_collect(struct bpf_object *obj) · 6 unmodified lines
2650	2780	err = bpf_object__add_program(obj, data->d_buf,
2651	2781	data->d_size,
2652	2782	name, idx);
2653		− if (err) {
2654		− char errmsg[STRERR_BUFSIZE];
2655		− char *cp;
2656		−
2657		− cp = libbpf_strerror_r(-err, errmsg,
2658		− sizeof(errmsg));
2659		− pr_warn("failed to alloc program %s (%s): %s",
2660		− name, obj->path, cp);
	2783	+ if (err)
2661	2784	return err;
2662		− }
2663	2785	} else if (strcmp(name, DATA_SEC) == 0) {
2664	2786	obj->efile.data = data;
2665	2787	obj->efile.data_shndx = idx;
@@ -2670,7 +2792,8 @@ static int bpf_object__elf_collect(struct bpf_object *obj) · 4 unmodified lines
2670	2792	obj->efile.st_ops_data = data;
2671	2793	obj->efile.st_ops_shndx = idx;
2672	2794	} else {
2673		− pr_debug("skip section(%d) %s\n", idx, name);
	2795	+ pr_info("elf: skipping unrecognized data section(%d) %s\n",
	2796	+ idx, name);
2674	2797	}
2675	2798	} else if (sh.sh_type == SHT_REL) {
2676	2799	int nr_sects = obj->efile.nr_reloc_sects;
@@ -2681,34 +2804,33 @@ static int bpf_object__elf_collect(struct bpf_object *obj) · 4 unmodified lines
2681	2804	if (!section_have_execinstr(obj, sec) &&
2682	2805	strcmp(name, ".rel" STRUCT_OPS_SEC) &&
2683	2806	strcmp(name, ".rel" MAPS_ELF_SEC)) {
2684		− pr_debug("skip relo %s(%d) for section(%d)\n",
2685		− name, idx, sec);
	2807	+ pr_info("elf: skipping relo section(%d) %s for section(%d) %s\n",
	2808	+ idx, name, sec,
	2809	+ elf_sec_name(obj, elf_sec_by_idx(obj, sec)) ?: "<?>");
2686	2810	continue;
2687	2811	}
2688	2812
2689		− sects = reallocarray(sects, nr_sects + 1,
2690		− sizeof(*obj->efile.reloc_sects));
2691		− if (!sects) {
2692		− pr_warn("reloc_sects realloc failed\n");
	2813	+ sects = libbpf_reallocarray(sects, nr_sects + 1,
	2814	+ sizeof(*obj->efile.reloc_sects));
	2815	+ if (!sects)
2693	2816	return -ENOMEM;
2694		− }
2695	2817
2696	2818	obj->efile.reloc_sects = sects;
2697	2819	obj->efile.nr_reloc_sects++;
2698	2820
2699	2821	obj->efile.reloc_sects[nr_sects].shdr = sh;
2700	2822	obj->efile.reloc_sects[nr_sects].data = data;
2701		− } else if (sh.sh_type == SHT_NOBITS &&
2702		− strcmp(name, BSS_SEC) == 0) {
	2823	+ } else if (sh.sh_type == SHT_NOBITS && strcmp(name, BSS_SEC) == 0) {
2703	2824	obj->efile.bss = data;
2704	2825	obj->efile.bss_shndx = idx;
2705	2826	} else {
2706		− pr_debug("skip section(%d) %s\n", idx, name);
	2827	+ pr_info("elf: skipping section(%d) %s (size %zu)\n", idx, name,
	2828	+ (size_t)sh.sh_size);
2707	2829	}
2708	2830	}
2709	2831
2710	2832	if (!obj->efile.strtabidx \|\| obj->efile.strtabidx > idx) {
2711		− pr_warn("Corrupted ELF file: index of strtab invalid\n");
	2833	+ pr_warn("elf: symbol strings section missing or invalid in %s\n", obj->path);
2712	2834	return -LIBBPF_ERRNO__FORMAT;
2713	2835	}
2714	2836	return bpf_object__init_btf(obj, btf_data, btf_ext_data);
@@ -2869,14 +2991,13 @@ static int bpf_object__collect_externs(struct bpf_object *obj) · 154 unmodified lines
2869	2991	if (!obj->efile.symbols)
2870	2992	return 0;
2871	2993
2872		− scn = elf_getscn(obj->efile.elf, obj->efile.symbols_shndx);
2873		− if (!scn)
2874		− return -LIBBPF_ERRNO__FORMAT;
2875		− if (gelf_getshdr(scn, &sh) != &sh)
	2994	+ scn = elf_sec_by_idx(obj, obj->efile.symbols_shndx);
	2995	+ if (elf_sec_hdr(obj, scn, &sh))
2876	2996	return -LIBBPF_ERRNO__FORMAT;
2877		− n = sh.sh_size / sh.sh_entsize;
2878	2997
	2998	+ n = sh.sh_size / sh.sh_entsize;
2879	2999	pr_debug("looking for externs among %d symbols...\n", n);
	3000	+
2880	3001	for (i = 0; i < n; i++) {
2881	3002	GElf_Sym sym;
2882	3003
@@ -2884,13 +3005,12 @@ static int bpf_object__collect_externs(struct bpf_object *obj) · 1 unmodified line
2884	3005	return -LIBBPF_ERRNO__FORMAT;
2885	3006	if (!sym_is_extern(&sym))
2886	3007	continue;
2887		− ext_name = elf_strptr(obj->efile.elf, obj->efile.strtabidx,
2888		− sym.st_name);
	3008	+ ext_name = elf_sym_str(obj, sym.st_name);
2889	3009	if (!ext_name \|\| !ext_name[0])
2890	3010	continue;
2891	3011
2892	3012	ext = obj->externs;
2893		− ext = reallocarray(ext, obj->nr_extern + 1, sizeof(*ext));
	3013	+ ext = libbpf_reallocarray(ext, obj->nr_extern + 1, sizeof(*ext));
2894	3014	if (!ext)
2895	3015	return -ENOMEM;
2896	3016	obj->externs = ext;
@@ -3109,7 +3229,7 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object *obj, int shndx) · 212 unmodified lines
3109	3229
3110	3230	static int bpf_program__record_reloc(struct bpf_program *prog,
3111	3231	struct reloc_desc *reloc_desc,
3112		− __u32 insn_idx, const char *name,
	3232	+ __u32 insn_idx, const char *sym_name,
3113	3233	const GElf_Sym sym, const GElf_Rel rel)
3114	3234	{
3115	3235	struct bpf_insn *insn = &prog->insns[insn_idx];
@@ -3117,22 +3237,25 @@ static int bpf_program__record_reloc(struct bpf_program *prog, · 1 unmodified line
3117	3237	struct bpf_object *obj = prog->obj;
3118	3238	__u32 shdr_idx = sym->st_shndx;
3119	3239	enum libbpf_map_type type;
	3240	+ const char *sym_sec_name;
3120	3241	struct bpf_map *map;
3121	3242
3122	3243	/* sub-program call relocation */
3123	3244	if (insn->code == (BPF_JMP \| BPF_CALL)) {
3124	3245	if (insn->src_reg != BPF_PSEUDO_CALL) {
3125		− pr_warn("incorrect bpf_call opcode\n");
	3246	+ pr_warn("prog '%s': incorrect bpf_call opcode\n", prog->name);
3126	3247	return -LIBBPF_ERRNO__RELOC;
3127	3248	}
3128	3249	/* text_shndx can be 0, if no default "main" program exists */
3129	3250	if (!shdr_idx \|\| shdr_idx != obj->efile.text_shndx) {
3130		− pr_warn("bad call relo against section %u\n", shdr_idx);
	3251	+ sym_sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, shdr_idx));
	3252	+ pr_warn("prog '%s': bad call relo against '%s' in section '%s'\n",
	3253	+ prog->name, sym_name, sym_sec_name);
3131	3254	return -LIBBPF_ERRNO__RELOC;
3132	3255	}
3133		− if (sym->st_value % 8) {
3134		− pr_warn("bad call relo offset: %zu\n",
3135		− (size_t)sym->st_value);
	3256	+ if (sym->st_value % BPF_INSN_SZ) {
	3257	+ pr_warn("prog '%s': bad call relo against '%s' at offset %zu\n",
	3258	+ prog->name, sym_name, (size_t)sym->st_value);
3136	3259	return -LIBBPF_ERRNO__RELOC;
3137	3260	}
3138	3261	reloc_desc->type = RELO_CALL;
@@ -3143,8 +3266,8 @@ static int bpf_program__record_reloc(struct bpf_program *prog, · 4 unmodified lines
3143	3266	}
3144	3267
3145	3268	if (insn->code != (BPF_LD \| BPF_IMM \| BPF_DW)) {
3146		− pr_warn("invalid relo for insns[%d].code 0x%x\n",
3147		− insn_idx, insn->code);
	3269	+ pr_warn("prog '%s': invalid relo against '%s' for insns[%d].code 0x%x\n",
	3270	+ prog->name, sym_name, insn_idx, insn->code);
3148	3271	return -LIBBPF_ERRNO__RELOC;
3149	3272	}
3150	3273
@@ -3159,12 +3282,12 @@ static int bpf_program__record_reloc(struct bpf_program *prog, · 8 unmodified lines
3159	3282	break;
3160	3283	}
3161	3284	if (i >= n) {
3162		− pr_warn("extern relo failed to find extern for sym %d\n",
3163		− sym_idx);
	3285	+ pr_warn("prog '%s': extern relo failed to find extern for '%s' (%d)\n",
	3286	+ prog->name, sym_name, sym_idx);
3164	3287	return -LIBBPF_ERRNO__RELOC;
3165	3288	}
3166		− pr_debug("found extern #%d '%s' (sym %d) for insn %u\n",
3167		− i, ext->name, ext->sym_idx, insn_idx);
	3289	+ pr_debug("prog '%s': found extern #%d '%s' (sym %d) for insn #%u\n",
	3290	+ prog->name, i, ext->name, ext->sym_idx, insn_idx);
3168	3291	reloc_desc->type = RELO_EXTERN;
3169	3292	reloc_desc->insn_idx = insn_idx;
3170	3293	reloc_desc->sym_off = i; /* sym_off stores extern index */
@@ -3172,18 +3295,19 @@ static int bpf_program__record_reloc(struct bpf_program *prog, · 1 unmodified line
3172	3295	}
3173	3296
3174	3297	if (!shdr_idx \|\| shdr_idx >= SHN_LORESERVE) {
3175		− pr_warn("invalid relo for \'%s\' in special section 0x%x; forgot to initialize global var?..\n",
3176		− name, shdr_idx);
	3298	+ pr_warn("prog '%s': invalid relo against '%s' in special section 0x%x; forgot to initialize global var?..\n",
	3299	+ prog->name, sym_name, shdr_idx);
3177	3300	return -LIBBPF_ERRNO__RELOC;
3178	3301	}
3179	3302
3180	3303	type = bpf_object__section_to_libbpf_map_type(obj, shdr_idx);
	3304	+ sym_sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, shdr_idx));
3181	3305
3182	3306	/* generic map reference relocation */
3183	3307	if (type == LIBBPF_MAP_UNSPEC) {
3184	3308	if (!bpf_object__shndx_is_maps(obj, shdr_idx)) {
3185		− pr_warn("bad map relo against section %u\n",
3186		− shdr_idx);
	3309	+ pr_warn("prog '%s': bad map relo against '%s' in section '%s'\n",
	3310	+ prog->name, sym_name, sym_sec_name);
3187	3311	return -LIBBPF_ERRNO__RELOC;
3188	3312	}
3189	3313	for (map_idx = 0; map_idx < nr_maps; map_idx++) {
@@ -3192,14 +3316,14 @@ static int bpf_program__record_reloc(struct bpf_program *prog, · 2 unmodified lines
3192	3316	map->sec_idx != sym->st_shndx \|\|
3193	3317	map->sec_offset != sym->st_value)
3194	3318	continue;
3195		− pr_debug("found map %zd (%s, sec %d, off %zu) for insn %u\n",
3196		− map_idx, map->name, map->sec_idx,
	3319	+ pr_debug("prog '%s': found map %zd (%s, sec %d, off %zu) for insn #%u\n",
	3320	+ prog->name, map_idx, map->name, map->sec_idx,
3197	3321	map->sec_offset, insn_idx);
3198	3322	break;
3199	3323	}
3200	3324	if (map_idx >= nr_maps) {
3201		− pr_warn("map relo failed to find map for sec %u, off %zu\n",
3202		− shdr_idx, (size_t)sym->st_value);
	3325	+ pr_warn("prog '%s': map relo failed to find map for section '%s', off %zu\n",
	3326	+ prog->name, sym_sec_name, (size_t)sym->st_value);
3203	3327	return -LIBBPF_ERRNO__RELOC;
3204	3328	}
3205	3329	reloc_desc->type = RELO_LD64;
@@ -3211,21 +3335,22 @@ static int bpf_program__record_reloc(struct bpf_program *prog, · 5 unmodified lines
3211	3335
3212	3336	/* global data map relocation */
3213	3337	if (!bpf_object__shndx_is_data(obj, shdr_idx)) {
3214		− pr_warn("bad data relo against section %u\n", shdr_idx);
	3338	+ pr_warn("prog '%s': bad data relo against section '%s'\n",
	3339	+ prog->name, sym_sec_name);
3215	3340	return -LIBBPF_ERRNO__RELOC;
3216	3341	}
3217	3342	for (map_idx = 0; map_idx < nr_maps; map_idx++) {
3218	3343	map = &obj->maps[map_idx];
3219	3344	if (map->libbpf_type != type)
3220	3345	continue;
3221		− pr_debug("found data map %zd (%s, sec %d, off %zu) for insn %u\n",
3222		− map_idx, map->name, map->sec_idx, map->sec_offset,
3223		− insn_idx);
	3346	+ pr_debug("prog '%s': found data map %zd (%s, sec %d, off %zu) for insn %u\n",
	3347	+ prog->name, map_idx, map->name, map->sec_idx,
	3348	+ map->sec_offset, insn_idx);
3224	3349	break;
3225	3350	}
3226	3351	if (map_idx >= nr_maps) {
3227		− pr_warn("data relo failed to find map for sec %u\n",
3228		− shdr_idx);
	3352	+ pr_warn("prog '%s': data relo failed to find map for section '%s'\n",
	3353	+ prog->name, sym_sec_name);
3229	3354	return -LIBBPF_ERRNO__RELOC;
3230	3355	}
3231	3356
@@ -3241,9 +3366,17 @@ bpf_program__collect_reloc(struct bpf_program prog, GElf_Shdr shdr, · 9 unmodified lines
3241	3366	Elf_Data data, struct bpf_object obj)
3242	3367	{
3243	3368	Elf_Data *symbols = obj->efile.symbols;
	3369	+ const char relo_sec_name, sec_name;
	3370	+ size_t sec_idx = shdr->sh_info;
3244	3371	int err, i, nrels;
3245	3372
3246		− pr_debug("collecting relocating info for: '%s'\n", prog->section_name);
	3373	+ relo_sec_name = elf_sec_str(obj, shdr->sh_name);
	3374	+ sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx));
	3375	+ if (!relo_sec_name \|\| !sec_name)
	3376	+ return -EINVAL;
	3377	+
	3378	+ pr_debug("sec '%s': collecting relocation for section(%zu) '%s'\n",
	3379	+ relo_sec_name, sec_idx, sec_name);
3247	3380	nrels = shdr->sh_size / shdr->sh_entsize;
3248	3381
3249	3382	prog->reloc_desc = malloc(sizeof(prog->reloc_desc) nrels);
@@ -3254,35 +3387,34 @@ bpf_program__collect_reloc(struct bpf_program prog, GElf_Shdr shdr, · 4 unmodified lines
3254	3387	prog->nr_reloc = nrels;
3255	3388
3256	3389	for (i = 0; i < nrels; i++) {
3257		− const char *name;
	3390	+ const char *sym_name;
3258	3391	__u32 insn_idx;
3259	3392	GElf_Sym sym;
3260	3393	GElf_Rel rel;
3261	3394
3262	3395	if (!gelf_getrel(data, i, &rel)) {
3263		− pr_warn("relocation: failed to get %d reloc\n", i);
	3396	+ pr_warn("sec '%s': failed to get relo #%d\n", relo_sec_name, i);
3264	3397	return -LIBBPF_ERRNO__FORMAT;
3265	3398	}
3266	3399	if (!gelf_getsym(symbols, GELF_R_SYM(rel.r_info), &sym)) {
3267		− pr_warn("relocation: symbol %"PRIx64" not found\n",
3268		− GELF_R_SYM(rel.r_info));
	3400	+ pr_warn("sec '%s': symbol 0x%zx not found for relo #%d\n",
	3401	+ relo_sec_name, (size_t)GELF_R_SYM(rel.r_info), i);
3269	3402	return -LIBBPF_ERRNO__FORMAT;
3270	3403	}
3271		− if (rel.r_offset % sizeof(struct bpf_insn))
	3404	+ if (rel.r_offset % BPF_INSN_SZ) {
	3405	+ pr_warn("sec '%s': invalid offset 0x%zx for relo #%d\n",
	3406	+ relo_sec_name, (size_t)GELF_R_SYM(rel.r_info), i);
3272	3407	return -LIBBPF_ERRNO__FORMAT;
	3408	+ }
3273	3409
3274		− insn_idx = rel.r_offset / sizeof(struct bpf_insn);
3275		− name = elf_strptr(obj->efile.elf, obj->efile.strtabidx,
3276		− sym.st_name) ? : "<?>";
	3410	+ insn_idx = rel.r_offset / BPF_INSN_SZ;
	3411	+ sym_name = elf_sym_str(obj, sym.st_name) ?: "<?>";
3277	3412
3278		− pr_debug("relo for shdr %u, symb %zu, value %zu, type %d, bind %d, name %d (\'%s\'), insn %u\n",
3279		− (__u32)sym.st_shndx, (size_t)GELF_R_SYM(rel.r_info),
3280		− (size_t)sym.st_value, GELF_ST_TYPE(sym.st_info),
3281		− GELF_ST_BIND(sym.st_info), sym.st_name, name,
3282		− insn_idx);
	3413	+ pr_debug("sec '%s': relo #%d: insn #%u against '%s'\n",
	3414	+ relo_sec_name, i, insn_idx, sym_name);
3283	3415
3284	3416	err = bpf_program__record_reloc(prog, &prog->reloc_desc[i],
3285		− insn_idx, name, &sym, &rel);
	3417	+ insn_idx, sym_name, &sym, &rel);
3286	3418	if (err)
3287	3419	return err;
3288	3420	}
@@ -3433,8 +3565,14 @@ bpf_object__probe_loading(struct bpf_object *obj) · 144 unmodified lines
3433	3565	return 0;
3434	3566	}
3435	3567
3436		−static int
3437		−bpf_object__probe_name(struct bpf_object *obj)
	3568	+static int probe_fd(int fd)
	3569	+{
	3570	+ if (fd >= 0)
	3571	+ close(fd);
	3572	+ return fd >= 0;
	3573	+}
	3574	+
	3575	+static int probe_kern_prog_name(void)
3438	3576	{
3439	3577	struct bpf_load_program_attr attr;
3440	3578	struct bpf_insn insns[] = {
@@ -3452,16 +3590,10 @@ bpf_object__probe_name(struct bpf_object *obj) · 11 unmodified lines
3452	3590	attr.license = "GPL";
3453	3591	attr.name = "test";
3454	3592	ret = bpf_load_program_xattr(&attr, NULL, 0);
3455		− if (ret >= 0) {
3456		− obj->caps.name = 1;
3457		− close(ret);
3458		− }
3459		−
3460		− return 0;
	3593	+ return probe_fd(ret);
3461	3594	}
3462	3595
3463		−static int
3464		−bpf_object__probe_global_data(struct bpf_object *obj)
	3596	+static int probe_kern_global_data(void)
3465	3597	{
3466	3598	struct bpf_load_program_attr prg_attr;
3467	3599	struct bpf_create_map_attr map_attr;
@@ -3498,16 +3630,23 @@ bpf_object__probe_global_data(struct bpf_object *obj) · 30 unmodified lines
3498	3630	prg_attr.license = "GPL";
3499	3631
3500	3632	ret = bpf_load_program_xattr(&prg_attr, NULL, 0);
3501		− if (ret >= 0) {
3502		− obj->caps.global_data = 1;
3503		− close(ret);
3504		− }
3505		−
3506	3633	close(map);
3507		− return 0;
	3634	+ return probe_fd(ret);
	3635	+}
	3636	+
	3637	+static int probe_kern_btf(void)
	3638	+{
	3639	+ static const char strs[] = "\0int";
	3640	+ __u32 types[] = {
	3641	+ /* int */
	3642	+ BTF_TYPE_INT_ENC(1, BTF_INT_SIGNED, 0, 32, 4),
	3643	+ };
	3644	+
	3645	+ return probe_fd(libbpf__load_raw_btf((char *)types, sizeof(types),
	3646	+ strs, sizeof(strs)));
3508	3647	}
3509	3648
3510		−static int bpf_object__probe_btf_func(struct bpf_object *obj)
	3649	+static int probe_kern_btf_func(void)
3511	3650	{
3512	3651	static const char strs[] = "\0int\0x\0a";
3513	3652	/* void x(int a) {} */
@@ -3520,20 +3659,12 @@ static int bpf_object__probe_btf_func(struct bpf_object *obj) · 6 unmodified lines
3520	3659	/* FUNC x / / [3] */
3521	3660	BTF_TYPE_ENC(5, BTF_INFO_ENC(BTF_KIND_FUNC, 0, 0), 2),
3522	3661	};
3523		− int btf_fd;
3524	3662
3525		− btf_fd = libbpf__load_raw_btf((char *)types, sizeof(types),
3526		− strs, sizeof(strs));
3527		− if (btf_fd >= 0) {
3528		− obj->caps.btf_func = 1;
3529		− close(btf_fd);
3530		− return 1;
3531		− }
3532		−
3533		− return 0;
	3663	+ return probe_fd(libbpf__load_raw_btf((char *)types, sizeof(types),
	3664	+ strs, sizeof(strs)));
3534	3665	}
3535	3666
3536		−static int bpf_object__probe_btf_func_global(struct bpf_object *obj)
	3667	+static int probe_kern_btf_func_global(void)
3537	3668	{
3538	3669	static const char strs[] = "\0int\0x\0a";
3539	3670	/* static void x(int a) {} */
@@ -3546,20 +3677,12 @@ static int bpf_object__probe_btf_func_global(struct bpf_object *obj) · 6 unmodified lines
3546	3677	/* FUNC x BTF_FUNC_GLOBAL / / [3] */
3547	3678	BTF_TYPE_ENC(5, BTF_INFO_ENC(BTF_KIND_FUNC, 0, BTF_FUNC_GLOBAL), 2),
3548	3679	};
3549		− int btf_fd;
3550	3680
3551		− btf_fd = libbpf__load_raw_btf((char *)types, sizeof(types),
3552		− strs, sizeof(strs));
3553		− if (btf_fd >= 0) {
3554		− obj->caps.btf_func_global = 1;
3555		− close(btf_fd);
3556		− return 1;
3557		− }
3558		−
3559		− return 0;
	3681	+ return probe_fd(libbpf__load_raw_btf((char *)types, sizeof(types),
	3682	+ strs, sizeof(strs)));
3560	3683	}
3561	3684
3562		−static int bpf_object__probe_btf_datasec(struct bpf_object *obj)
	3685	+static int probe_kern_btf_datasec(void)
3563	3686	{
3564	3687	static const char strs[] = "\0x\0.data";
3565	3688	/* static int a; */
@@ -3573,20 +3696,12 @@ static int bpf_object__probe_btf_datasec(struct bpf_object *obj) · 7 unmodified lines
3573	3696	BTF_TYPE_ENC(3, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4),
3574	3697	BTF_VAR_SECINFO_ENC(2, 0, 4),
3575	3698	};
3576		− int btf_fd;
3577		−
3578		− btf_fd = libbpf__load_raw_btf((char *)types, sizeof(types),
3579		− strs, sizeof(strs));
3580		− if (btf_fd >= 0) {
3581		− obj->caps.btf_datasec = 1;
3582		− close(btf_fd);
3583		− return 1;
3584		− }
3585	3699
3586		− return 0;
	3700	+ return probe_fd(libbpf__load_raw_btf((char *)types, sizeof(types),
	3701	+ strs, sizeof(strs)));
3587	3702	}
3588	3703
3589		−static int bpf_object__probe_array_mmap(struct bpf_object *obj)
	3704	+static int probe_kern_array_mmap(void)
3590	3705	{
3591	3706	struct bpf_create_map_attr attr = {
3592	3707	.map_type = BPF_MAP_TYPE_ARRAY,
@@ -3595,27 +3710,17 @@ static int bpf_object__probe_array_mmap(struct bpf_object *obj) · 2 unmodified lines
3595	3710	.value_size = sizeof(int),
3596	3711	.max_entries = 1,
3597	3712	};
3598		− int fd;
3599		−
3600		− fd = bpf_create_map_xattr(&attr);
3601		− if (fd >= 0) {
3602		− obj->caps.array_mmap = 1;
3603		− close(fd);
3604		− return 1;
3605		− }
3606	3713
3607		− return 0;
	3714	+ return probe_fd(bpf_create_map_xattr(&attr));
3608	3715	}
3609	3716
3610		−static int
3611		−bpf_object__probe_exp_attach_type(struct bpf_object *obj)
	3717	+static int probe_kern_exp_attach_type(void)
3612	3718	{
3613	3719	struct bpf_load_program_attr attr;
3614	3720	struct bpf_insn insns[] = {
3615	3721	BPF_MOV64_IMM(BPF_REG_0, 0),
3616	3722	BPF_EXIT_INSN(),
3617	3723	};
3618		− int fd;
3619	3724
3620	3725	memset(&attr, 0, sizeof(attr));
3621	3726	/* use any valid combination of program type and (optional)
@@ -3629,36 +3734,91 @@ bpf_object__probe_exp_attach_type(struct bpf_object *obj) · 7 unmodified lines
3629	3734	attr.insns_cnt = ARRAY_SIZE(insns);
3630	3735	attr.license = "GPL";
3631	3736
3632		− fd = bpf_load_program_xattr(&attr, NULL, 0);
3633		− if (fd >= 0) {
3634		− obj->caps.exp_attach_type = 1;
3635		− close(fd);
3636		− return 1;
3637		− }
3638		− return 0;
	3737	+ return probe_fd(bpf_load_program_xattr(&attr, NULL, 0));
3639	3738	}
3640	3739
3641		−static int
3642		−bpf_object__probe_caps(struct bpf_object *obj)
3643		−{
3644		− int (probe_fn[])(struct bpf_object obj) = {
3645		− bpf_object__probe_name,
3646		− bpf_object__probe_global_data,
3647		− bpf_object__probe_btf_func,
3648		− bpf_object__probe_btf_func_global,
3649		− bpf_object__probe_btf_datasec,
3650		− bpf_object__probe_array_mmap,
3651		− bpf_object__probe_exp_attach_type,
	3740	+static int probe_kern_probe_read_kernel(void)
	3741	+{
	3742	+ struct bpf_load_program_attr attr;
	3743	+ struct bpf_insn insns[] = {
	3744	+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_10), /* r1 = r10 (fp) */
	3745	+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -8), /* r1 += -8 */
	3746	+ BPF_MOV64_IMM(BPF_REG_2, 8), /* r2 = 8 */
	3747	+ BPF_MOV64_IMM(BPF_REG_3, 0), /* r3 = 0 */
	3748	+ BPF_RAW_INSN(BPF_JMP \| BPF_CALL, 0, 0, 0, BPF_FUNC_probe_read_kernel),
	3749	+ BPF_EXIT_INSN(),
3652	3750	};
3653		− int i, ret;
3654	3751
3655		− for (i = 0; i < ARRAY_SIZE(probe_fn); i++) {
3656		− ret = probe_fn[i](obj);
3657		− if (ret < 0)
3658		− pr_debug("Probe #%d failed with %d.\n", i, ret);
	3752	+ memset(&attr, 0, sizeof(attr));
	3753	+ attr.prog_type = BPF_PROG_TYPE_KPROBE;
	3754	+ attr.insns = insns;
	3755	+ attr.insns_cnt = ARRAY_SIZE(insns);
	3756	+ attr.license = "GPL";
	3757	+
	3758	+ return probe_fd(bpf_load_program_xattr(&attr, NULL, 0));
	3759	+}
	3760	+
	3761	+enum kern_feature_result {
	3762	+ FEAT_UNKNOWN = 0,
	3763	+ FEAT_SUPPORTED = 1,
	3764	+ FEAT_MISSING = 2,
	3765	+};
	3766	+
	3767	+typedef int (*feature_probe_fn)(void);
	3768	+
	3769	+static struct kern_feature_desc {
	3770	+ const char *desc;
	3771	+ feature_probe_fn probe;
	3772	+ enum kern_feature_result res;
	3773	+} feature_probes[__FEAT_CNT] = {
	3774	+ [FEAT_PROG_NAME] = {
	3775	+ "BPF program name", probe_kern_prog_name,
	3776	+ },
	3777	+ [FEAT_GLOBAL_DATA] = {
	3778	+ "global variables", probe_kern_global_data,
	3779	+ },
	3780	+ [FEAT_BTF] = {
	3781	+ "minimal BTF", probe_kern_btf,
	3782	+ },
	3783	+ [FEAT_BTF_FUNC] = {
	3784	+ "BTF functions", probe_kern_btf_func,
	3785	+ },
	3786	+ [FEAT_BTF_GLOBAL_FUNC] = {
	3787	+ "BTF global function", probe_kern_btf_func_global,
	3788	+ },
	3789	+ [FEAT_BTF_DATASEC] = {
	3790	+ "BTF data section and variable", probe_kern_btf_datasec,
	3791	+ },
	3792	+ [FEAT_ARRAY_MMAP] = {
	3793	+ "ARRAY map mmap()", probe_kern_array_mmap,
	3794	+ },
	3795	+ [FEAT_EXP_ATTACH_TYPE] = {
	3796	+ "BPF_PROG_LOAD expected_attach_type attribute",
	3797	+ probe_kern_exp_attach_type,
	3798	+ },
	3799	+ [FEAT_PROBE_READ_KERN] = {
	3800	+ "bpf_probe_read_kernel() helper", probe_kern_probe_read_kernel,
3659	3801	}
	3802	+};
3660	3803
3661		− return 0;
	3804	+static bool kernel_supports(enum kern_feature_id feat_id)
	3805	+{
	3806	+ struct kern_feature_desc *feat = &feature_probes[feat_id];
	3807	+ int ret;
	3808	+
	3809	+ if (READ_ONCE(feat->res) == FEAT_UNKNOWN) {
	3810	+ ret = feat->probe();
	3811	+ if (ret > 0) {
	3812	+ WRITE_ONCE(feat->res, FEAT_SUPPORTED);
	3813	+ } else if (ret == 0) {
	3814	+ WRITE_ONCE(feat->res, FEAT_MISSING);
	3815	+ } else {
	3816	+ pr_warn("Detection of kernel %s support failed: %d\n", feat->desc, ret);
	3817	+ WRITE_ONCE(feat->res, FEAT_MISSING);
	3818	+ }
	3819	+ }
	3820	+
	3821	+ return READ_ONCE(feat->res) == FEAT_SUPPORTED;
3662	3822	}
3663	3823
3664	3824	static bool map_is_reuse_compat(const struct bpf_map *map, int map_fd)
@@ -3760,7 +3920,7 @@ static int bpf_object__create_map(struct bpf_object obj, struct bpf_map map) · 95 unmodified lines
3760	3920
3761	3921	memset(&create_attr, 0, sizeof(create_attr));
3762	3922
3763		− if (obj->caps.name)
	3923	+ if (kernel_supports(FEAT_PROG_NAME))
3764	3924	create_attr.name = map->name;
3765	3925	create_attr.map_ifindex = map->map_ifindex;
3766	3926	create_attr.map_type = def->type;
@@ -4011,6 +4171,10 @@ struct bpf_core_spec { · 244 unmodified lines
4011	4171	const struct btf *btf;
4012	4172	/* high-level spec: named fields and array indices only */
4013	4173	struct bpf_core_accessor spec[BPF_CORE_SPEC_MAX_LEN];
	4174	+ /* original unresolved (no skip_mods_or_typedefs) root type ID */
	4175	+ __u32 root_type_id;
	4176	+ /* CO-RE relocation kind */
	4177	+ enum bpf_core_relo_kind relo_kind;
4014	4178	/* high-level spec length */
4015	4179	int len;
4016	4180	/* raw, low-level spec: 1-to-1 with accessor spec string */
@@ -4041,8 +4205,66 @@ static bool is_flex_arr(const struct btf *btf, · 24 unmodified lines
4041	4205	return acc->idx == btf_vlen(t) - 1;
4042	4206	}
4043	4207
	4208	+static const char *core_relo_kind_str(enum bpf_core_relo_kind kind)
	4209	+{
	4210	+ switch (kind) {
	4211	+ case BPF_FIELD_BYTE_OFFSET: return "byte_off";
	4212	+ case BPF_FIELD_BYTE_SIZE: return "byte_sz";
	4213	+ case BPF_FIELD_EXISTS: return "field_exists";
	4214	+ case BPF_FIELD_SIGNED: return "signed";
	4215	+ case BPF_FIELD_LSHIFT_U64: return "lshift_u64";
	4216	+ case BPF_FIELD_RSHIFT_U64: return "rshift_u64";
	4217	+ case BPF_TYPE_ID_LOCAL: return "local_type_id";
	4218	+ case BPF_TYPE_ID_TARGET: return "target_type_id";
	4219	+ case BPF_TYPE_EXISTS: return "type_exists";
	4220	+ case BPF_TYPE_SIZE: return "type_size";
	4221	+ case BPF_ENUMVAL_EXISTS: return "enumval_exists";
	4222	+ case BPF_ENUMVAL_VALUE: return "enumval_value";
	4223	+ default: return "unknown";
	4224	+ }
	4225	+}
	4226	+
	4227	+static bool core_relo_is_field_based(enum bpf_core_relo_kind kind)
	4228	+{
	4229	+ switch (kind) {
	4230	+ case BPF_FIELD_BYTE_OFFSET:
	4231	+ case BPF_FIELD_BYTE_SIZE:
	4232	+ case BPF_FIELD_EXISTS:
	4233	+ case BPF_FIELD_SIGNED:
	4234	+ case BPF_FIELD_LSHIFT_U64:
	4235	+ case BPF_FIELD_RSHIFT_U64:
	4236	+ return true;
	4237	+ default:
	4238	+ return false;
	4239	+ }
	4240	+}
	4241	+
	4242	+static bool core_relo_is_type_based(enum bpf_core_relo_kind kind)
	4243	+{
	4244	+ switch (kind) {
	4245	+ case BPF_TYPE_ID_LOCAL:
	4246	+ case BPF_TYPE_ID_TARGET:
	4247	+ case BPF_TYPE_EXISTS:
	4248	+ case BPF_TYPE_SIZE:
	4249	+ return true;
	4250	+ default:
	4251	+ return false;
	4252	+ }
	4253	+}
	4254	+
	4255	+static bool core_relo_is_enumval_based(enum bpf_core_relo_kind kind)
	4256	+{
	4257	+ switch (kind) {
	4258	+ case BPF_ENUMVAL_EXISTS:
	4259	+ case BPF_ENUMVAL_VALUE:
	4260	+ return true;
	4261	+ default:
	4262	+ return false;
	4263	+ }
	4264	+}
	4265	+
4044	4266	/*
4045		− * Turn bpf_field_reloc into a low- and high-level spec representation,
	4267	+ * Turn bpf_core_relo into a low- and high-level spec representation,
4046	4268	* validating correctness along the way, as well as calculating resulting
4047	4269	* field bit offset, specified by accessor string. Low-level spec captures
4048	4270	* every single level of nestedness, including traversing anonymous
@@ -4071,10 +4293,17 @@ static bool is_flex_arr(const struct btf *btf, · 22 unmodified lines
4071	4293	* - field 'a' access (corresponds to '2' in low-level spec);
4072	4294	* - array element #3 access (corresponds to '3' in low-level spec).
4073	4295	*
	4296	+ * Type-based relocations (TYPE_EXISTS/TYPE_SIZE,
	4297	+ * TYPE_ID_LOCAL/TYPE_ID_TARGET) don't capture any field information. Their
	4298	+ * spec and raw_spec are kept empty.
	4299	+ *
	4300	+ * Enum value-based relocations (ENUMVAL_EXISTS/ENUMVAL_VALUE) use access
	4301	+ * string to specify enumerator's value index that need to be relocated.
4074	4302	*/
4075		−static int bpf_core_spec_parse(const struct btf *btf,
	4303	+static int bpf_core_parse_spec(const struct btf *btf,
4076	4304	__u32 type_id,
4077	4305	const char *spec_str,
	4306	+ enum bpf_core_relo_kind relo_kind,
4078	4307	struct bpf_core_spec *spec)
4079	4308	{
4080	4309	int access_idx, parsed_len, i;
@@ -4089,6 +4318,15 @@ static int bpf_core_spec_parse(const struct btf *btf, · 8 unmodified lines
4089	4318
4090	4319	memset(spec, 0, sizeof(*spec));
4091	4320	spec->btf = btf;
	4321	+ spec->root_type_id = type_id;
	4322	+ spec->relo_kind = relo_kind;
	4323	+
	4324	+ /* type-based relocations don't have a field access string */
	4325	+ if (core_relo_is_type_based(relo_kind)) {
	4326	+ if (strcmp(spec_str, "0"))
	4327	+ return -EINVAL;
	4328	+ return 0;
	4329	+ }
4092	4330
4093	4331	/* parse spec_str="0:1:2:3:4" into array raw_spec=[0, 1, 2, 3, 4] */
4094	4332	while (*spec_str) {
@@ -4105,16 +4343,28 @@ static int bpf_core_spec_parse(const struct btf *btf, · 10 unmodified lines
4105	4343	if (spec->raw_len == 0)
4106	4344	return -EINVAL;
4107	4345
4108		− /* first spec value is always reloc type array index */
4109	4346	t = skip_mods_and_typedefs(btf, type_id, &id);
4110	4347	if (!t)
4111	4348	return -EINVAL;
4112	4349
4113	4350	access_idx = spec->raw_spec[0];
4114		− spec->spec[0].type_id = id;
4115		− spec->spec[0].idx = access_idx;
	4351	+ acc = &spec->spec[0];
	4352	+ acc->type_id = id;
	4353	+ acc->idx = access_idx;
4116	4354	spec->len++;
4117	4355
	4356	+ if (core_relo_is_enumval_based(relo_kind)) {
	4357	+ if (!btf_is_enum(t) \|\| spec->raw_len > 1 \|\| access_idx >= btf_vlen(t))
	4358	+ return -EINVAL;
	4359	+
	4360	+ /* record enumerator name in a first accessor */
	4361	+ acc->name = btf__name_by_offset(btf, btf_enum(t)[access_idx].name_off);
	4362	+ return 0;
	4363	+ }
	4364	+
	4365	+ if (!core_relo_is_field_based(relo_kind))
	4366	+ return -EINVAL;
	4367	+
4118	4368	sz = btf__resolve_size(btf, id);
4119	4369	if (sz < 0)
4120	4370	return sz;
@@ -4172,8 +4422,8 @@ static int bpf_core_spec_parse(const struct btf *btf, · 51 unmodified lines
4172	4422	return sz;
4173	4423	spec->bit_offset += access_idx * sz * 8;
4174	4424	} else {
4175		− pr_warn("relo for [%u] %s (at idx %d) captures type [%d] of unexpected kind %d\n",
4176		− type_id, spec_str, i, id, btf_kind(t));
	4425	+ pr_warn("relo for [%u] %s (at idx %d) captures type [%d] of unexpected kind %s\n",
	4426	+ type_id, spec_str, i, id, btf_kind_str(t));
4177	4427	return -EINVAL;
4178	4428	}
4179	4429	}
@@ -4223,16 +4473,16 @@ static struct ids_vec bpf_core_find_cands(const struct btf local_btf, · 43 unmodified lines
4223	4473	{
4224	4474	size_t local_essent_len, targ_essent_len;
4225	4475	const char local_name, targ_name;
4226		− const struct btf_type *t;
	4476	+ const struct btf_type t, local_t;
4227	4477	struct ids_vec *cand_ids;
4228	4478	__u32 *new_ids;
4229	4479	int i, err, n;
4230	4480
4231		− t = btf__type_by_id(local_btf, local_type_id);
4232		− if (!t)
	4481	+ local_t = btf__type_by_id(local_btf, local_type_id);
	4482	+ if (!local_t)
4233	4483	return ERR_PTR(-EINVAL);
4234	4484
4235		− local_name = btf__name_by_offset(local_btf, t->name_off);
	4485	+ local_name = btf__name_by_offset(local_btf, local_t->name_off);
4236	4486	if (str_is_empty(local_name))
4237	4487	return ERR_PTR(-EINVAL);
4238	4488	local_essent_len = bpf_core_essential_name_len(local_name);
@@ -4244,12 +4494,11 @@ static struct ids_vec bpf_core_find_cands(const struct btf local_btf, · 5 unmodified lines
4244	4494	n = btf__get_nr_types(targ_btf);
4245	4495	for (i = 1; i <= n; i++) {
4246	4496	t = btf__type_by_id(targ_btf, i);
4247		− targ_name = btf__name_by_offset(targ_btf, t->name_off);
4248		− if (str_is_empty(targ_name))
	4497	+ if (btf_kind(t) != btf_kind(local_t))
4249	4498	continue;
4250	4499
4251		− t = skip_mods_and_typedefs(targ_btf, i, NULL);
4252		− if (!btf_is_composite(t) && !btf_is_array(t))
	4500	+ targ_name = btf__name_by_offset(targ_btf, t->name_off);
	4501	+ if (str_is_empty(targ_name))
4253	4502	continue;
4254	4503
4255	4504	targ_essent_len = bpf_core_essential_name_len(targ_name);
@@ -4257,11 +4506,12 @@ static struct ids_vec bpf_core_find_cands(const struct btf local_btf, · 1 unmodified line
4257	4506	continue;
4258	4507
4259	4508	if (strncmp(local_name, targ_name, local_essent_len) == 0) {
4260		− pr_debug("[%d] %s: found candidate [%d] %s\n",
4261		− local_type_id, local_name, i, targ_name);
4262		− new_ids = reallocarray(cand_ids->data,
4263		− cand_ids->len + 1,
4264		− sizeof(*cand_ids->data));
	4509	+ pr_debug("CO-RE relocating [%d] %s %s: found target candidate [%d] %s %s\n",
	4510	+ local_type_id, btf_kind_str(local_t),
	4511	+ local_name, i, btf_kind_str(t), targ_name);
	4512	+ new_ids = libbpf_reallocarray(cand_ids->data,
	4513	+ cand_ids->len + 1,
	4514	+ sizeof(*cand_ids->data));
4265	4515	if (!new_ids) {
4266	4516	err = -ENOMEM;
4267	4517	goto err_out;
@@ -4276,8 +4526,9 @@ err_out: · 8 unmodified lines
4276	4526	return ERR_PTR(err);
4277	4527	}
4278	4528
4279		−/* Check two types for compatibility, skipping const/volatile/restrict and
4280		− * typedefs, to ensure we are relocating compatible entities:
	4529	+/* Check two types for compatibility for the purpose of field access
	4530	+ * relocation. const/volatile/restrict and typedefs are skipped to ensure we
	4531	+ * are relocating semantically compatible entities:
4281	4532	* - any two STRUCTs/UNIONs are compatible and can be mixed;
4282	4533	* - any two FWDs are compatible, if their names match (modulo flavor suffix);
4283	4534	* - any two PTRs are always compatible;
@@ -4411,25 +4662,119 @@ static int bpf_core_match_member(const struct btf *local_btf, · 127 unmodified lines
4411	4662	/* matching named field */
4412	4663	struct bpf_core_accessor *targ_acc;
4413	4664
4414		− targ_acc = &spec->spec[spec->len++];
4415		− targ_acc->type_id = targ_id;
4416		− targ_acc->idx = i;
4417		− targ_acc->name = targ_name;
	4665	+ targ_acc = &spec->spec[spec->len++];
	4666	+ targ_acc->type_id = targ_id;
	4667	+ targ_acc->idx = i;
	4668	+ targ_acc->name = targ_name;
	4669	+
	4670	+ *next_targ_id = m->type;
	4671	+ found = bpf_core_fields_are_compat(local_btf,
	4672	+ local_member->type,
	4673	+ targ_btf, m->type);
	4674	+ if (!found)
	4675	+ spec->len--; /* pop accessor */
	4676	+ return found;
	4677	+ }
	4678	+ /* member turned out not to be what we looked for */
	4679	+ spec->bit_offset -= bit_offset;
	4680	+ spec->raw_len--;
	4681	+ }
	4682	+
	4683	+ return 0;
	4684	+}
	4685	+
	4686	+/* Check local and target types for compatibility. This check is used for
	4687	+ * type-based CO-RE relocations and follow slightly different rules than
	4688	+ * field-based relocations. This function assumes that root types were already
	4689	+ * checked for name match. Beyond that initial root-level name check, names
	4690	+ * are completely ignored. Compatibility rules are as follows:
	4691	+ * - any two STRUCTs/UNIONs/FWDs/ENUMs/INTs are considered compatible, but
	4692	+ * kind should match for local and target types (i.e., STRUCT is not
	4693	+ * compatible with UNION);
	4694	+ * - for ENUMs, the size is ignored;
	4695	+ * - for INT, size and signedness are ignored;
	4696	+ * - for ARRAY, dimensionality is ignored, element types are checked for
	4697	+ * compatibility recursively;
	4698	+ * - CONST/VOLATILE/RESTRICT modifiers are ignored;
	4699	+ * - TYPEDEFs/PTRs are compatible if types they pointing to are compatible;
	4700	+ * - FUNC_PROTOs are compatible if they have compatible signature: same
	4701	+ * number of input args and compatible return and argument types.
	4702	+ * These rules are not set in stone and probably will be adjusted as we get
	4703	+ * more experience with using BPF CO-RE relocations.
	4704	+ */
	4705	+static int bpf_core_types_are_compat(const struct btf *local_btf, __u32 local_id,
	4706	+ const struct btf *targ_btf, __u32 targ_id)
	4707	+{
	4708	+ const struct btf_type local_type, targ_type;
	4709	+ int depth = 32; /* max recursion depth */
	4710	+
	4711	+ /* caller made sure that names match (ignoring flavor suffix) */
	4712	+ local_type = btf__type_by_id(local_btf, local_id);
	4713	+ targ_type = btf__type_by_id(targ_btf, targ_id);
	4714	+ if (btf_kind(local_type) != btf_kind(targ_type))
	4715	+ return 0;
	4716	+
	4717	+recur:
	4718	+ depth--;
	4719	+ if (depth < 0)
	4720	+ return -EINVAL;
4418	4721
4419		− *next_targ_id = m->type;
4420		− found = bpf_core_fields_are_compat(local_btf,
4421		− local_member->type,
4422		− targ_btf, m->type);
4423		− if (!found)
4424		− spec->len--; /* pop accessor */
4425		− return found;
	4722	+ local_type = skip_mods_and_typedefs(local_btf, local_id, &local_id);
	4723	+ targ_type = skip_mods_and_typedefs(targ_btf, targ_id, &targ_id);
	4724	+ if (!local_type \|\| !targ_type)
	4725	+ return -EINVAL;
	4726	+
	4727	+ if (btf_kind(local_type) != btf_kind(targ_type))
	4728	+ return 0;
	4729	+
	4730	+ switch (btf_kind(local_type)) {
	4731	+ case BTF_KIND_UNKN:
	4732	+ case BTF_KIND_STRUCT:
	4733	+ case BTF_KIND_UNION:
	4734	+ case BTF_KIND_ENUM:
	4735	+ case BTF_KIND_FWD:
	4736	+ return 1;
	4737	+ case BTF_KIND_INT:
	4738	+ /* just reject deprecated bitfield-like integers; all other
	4739	+ * integers are by default compatible between each other
	4740	+ */
	4741	+ return btf_int_offset(local_type) == 0 && btf_int_offset(targ_type) == 0;
	4742	+ case BTF_KIND_PTR:
	4743	+ local_id = local_type->type;
	4744	+ targ_id = targ_type->type;
	4745	+ goto recur;
	4746	+ case BTF_KIND_ARRAY:
	4747	+ local_id = btf_array(local_type)->type;
	4748	+ targ_id = btf_array(targ_type)->type;
	4749	+ goto recur;
	4750	+ case BTF_KIND_FUNC_PROTO: {
	4751	+ struct btf_param *local_p = btf_params(local_type);
	4752	+ struct btf_param *targ_p = btf_params(targ_type);
	4753	+ __u16 local_vlen = btf_vlen(local_type);
	4754	+ __u16 targ_vlen = btf_vlen(targ_type);
	4755	+ int i, err;
	4756	+
	4757	+ if (local_vlen != targ_vlen)
	4758	+ return 0;
	4759	+
	4760	+ for (i = 0; i < local_vlen; i++, local_p++, targ_p++) {
	4761	+ skip_mods_and_typedefs(local_btf, local_p->type, &local_id);
	4762	+ skip_mods_and_typedefs(targ_btf, targ_p->type, &targ_id);
	4763	+ err = bpf_core_types_are_compat(local_btf, local_id, targ_btf, targ_id);
	4764	+ if (err <= 0)
	4765	+ return err;
4426	4766	}
4427		− /* member turned out not to be what we looked for */
4428		− spec->bit_offset -= bit_offset;
4429		− spec->raw_len--;
4430		− }
4431	4767
4432		− return 0;
	4768	+ /* tail recurse for return type check */
	4769	+ skip_mods_and_typedefs(local_btf, local_type->type, &local_id);
	4770	+ skip_mods_and_typedefs(targ_btf, targ_type->type, &targ_id);
	4771	+ goto recur;
	4772	+ }
	4773	+ default:
	4774	+ pr_warn("unexpected kind %s relocated, local [%d], target [%d]\n",
	4775	+ btf_kind_str(local_type), local_id, targ_id);
	4776	+ return 0;
	4777	+ }
4433	4778	}
4434	4779
4435	4780	/*
@@ -4447,10 +4792,51 @@ static int bpf_core_spec_match(struct bpf_core_spec *local_spec, · 11 unmodified lines
4447	4792
4448	4793	memset(targ_spec, 0, sizeof(*targ_spec));
4449	4794	targ_spec->btf = targ_btf;
	4795	+ targ_spec->root_type_id = targ_id;
	4796	+ targ_spec->relo_kind = local_spec->relo_kind;
	4797	+
	4798	+ if (core_relo_is_type_based(local_spec->relo_kind)) {
	4799	+ return bpf_core_types_are_compat(local_spec->btf,
	4800	+ local_spec->root_type_id,
	4801	+ targ_btf, targ_id);
	4802	+ }
4450	4803
4451	4804	local_acc = &local_spec->spec[0];
4452	4805	targ_acc = &targ_spec->spec[0];
4453	4806
	4807	+ if (core_relo_is_enumval_based(local_spec->relo_kind)) {
	4808	+ size_t local_essent_len, targ_essent_len;
	4809	+ const struct btf_enum *e;
	4810	+ const char *targ_name;
	4811	+
	4812	+ /* has to resolve to an enum */
	4813	+ targ_type = skip_mods_and_typedefs(targ_spec->btf, targ_id, &targ_id);
	4814	+ if (!btf_is_enum(targ_type))
	4815	+ return 0;
	4816	+
	4817	+ local_essent_len = bpf_core_essential_name_len(local_acc->name);
	4818	+
	4819	+ for (i = 0, e = btf_enum(targ_type); i < btf_vlen(targ_type); i++, e++) {
	4820	+ targ_name = btf__name_by_offset(targ_spec->btf, e->name_off);
	4821	+ targ_essent_len = bpf_core_essential_name_len(targ_name);
	4822	+ if (targ_essent_len != local_essent_len)
	4823	+ continue;
	4824	+ if (strncmp(local_acc->name, targ_name, local_essent_len) == 0) {
	4825	+ targ_acc->type_id = targ_id;
	4826	+ targ_acc->idx = i;
	4827	+ targ_acc->name = targ_name;
	4828	+ targ_spec->len++;
	4829	+ targ_spec->raw_spec[targ_spec->raw_len] = targ_acc->idx;
	4830	+ targ_spec->raw_len++;
	4831	+ return 1;
	4832	+ }
	4833	+ }
	4834	+ return 0;
	4835	+ }
	4836	+
	4837	+ if (!core_relo_is_field_based(local_spec->relo_kind))
	4838	+ return -EINVAL;
	4839	+
4454	4840	for (i = 0; i < local_spec->len; i++, local_acc++, targ_acc++) {
4455	4841	targ_type = skip_mods_and_typedefs(targ_spec->btf, targ_id,
4456	4842	&targ_id);
@@ -4507,18 +4893,29 @@ static int bpf_core_spec_match(struct bpf_core_spec *local_spec, · 50 unmodified lines
4507	4893	}
4508	4894
4509	4895	static int bpf_core_calc_field_relo(const struct bpf_program *prog,
4510		− const struct bpf_field_reloc *relo,
	4896	+ const struct bpf_core_relo *relo,
4511	4897	const struct bpf_core_spec *spec,
4512	4898	__u32 val, bool validate)
4513	4899	{
4514		− const struct bpf_core_accessor *acc = &spec->spec[spec->len - 1];
4515		− const struct btf_type *t = btf__type_by_id(spec->btf, acc->type_id);
	4900	+ const struct bpf_core_accessor *acc;
	4901	+ const struct btf_type *t;
4516	4902	__u32 byte_off, byte_sz, bit_off, bit_sz;
4517	4903	const struct btf_member *m;
4518	4904	const struct btf_type *mt;
4519	4905	bool bitfield;
4520	4906	__s64 sz;
4521	4907
	4908	+ if (relo->kind == BPF_FIELD_EXISTS) {
	4909	+ *val = spec ? 1 : 0;
	4910	+ return 0;
	4911	+ }
	4912	+
	4913	+ if (!spec)
	4914	+ return -EUCLEAN; /* request instruction poisoning */
	4915	+
	4916	+ acc = &spec->spec[spec->len - 1];
	4917	+ t = btf__type_by_id(spec->btf, acc->type_id);
	4918	+
4522	4919	/* a[n] accessor needs special handling */
4523	4920	if (!acc->name) {
4524	4921	if (relo->kind == BPF_FIELD_BYTE_OFFSET) {
@@ -4604,21 +5001,158 @@ static int bpf_core_calc_field_relo(const struct bpf_program *prog, · 79 unmodified lines
4604	5001	break;
4605	5002	case BPF_FIELD_EXISTS:
4606	5003	default:
4607		− pr_warn("prog '%s': unknown relo %d at insn #%d\n",
4608		− bpf_program__title(prog, false),
4609		− relo->kind, relo->insn_off / 8);
4610		− return -EINVAL;
	5004	+ return -EOPNOTSUPP;
	5005	+ }
	5006	+
	5007	+ return 0;
	5008	+}
	5009	+
	5010	+static int bpf_core_calc_type_relo(const struct bpf_core_relo *relo,
	5011	+ const struct bpf_core_spec *spec,
	5012	+ __u32 *val)
	5013	+{
	5014	+ __s64 sz;
	5015	+
	5016	+ /* type-based relos return zero when target type is not found */
	5017	+ if (!spec) {
	5018	+ *val = 0;
	5019	+ return 0;
	5020	+ }
	5021	+
	5022	+ switch (relo->kind) {
	5023	+ case BPF_TYPE_ID_TARGET:
	5024	+ *val = spec->root_type_id;
	5025	+ break;
	5026	+ case BPF_TYPE_EXISTS:
	5027	+ *val = 1;
	5028	+ break;
	5029	+ case BPF_TYPE_SIZE:
	5030	+ sz = btf__resolve_size(spec->btf, spec->root_type_id);
	5031	+ if (sz < 0)
	5032	+ return -EINVAL;
	5033	+ *val = sz;
	5034	+ break;
	5035	+ case BPF_TYPE_ID_LOCAL:
	5036	+ /* BPF_TYPE_ID_LOCAL is handled specially and shouldn't get here */
	5037	+ default:
	5038	+ return -EOPNOTSUPP;
	5039	+ }
	5040	+
	5041	+ return 0;
	5042	+}
	5043	+
	5044	+static int bpf_core_calc_enumval_relo(const struct bpf_core_relo *relo,
	5045	+ const struct bpf_core_spec *spec,
	5046	+ __u32 *val)
	5047	+{
	5048	+ const struct btf_type *t;
	5049	+ const struct btf_enum *e;
	5050	+
	5051	+ switch (relo->kind) {
	5052	+ case BPF_ENUMVAL_EXISTS:
	5053	+ *val = spec ? 1 : 0;
	5054	+ break;
	5055	+ case BPF_ENUMVAL_VALUE:
	5056	+ if (!spec)
	5057	+ return -EUCLEAN; /* request instruction poisoning */
	5058	+ t = btf__type_by_id(spec->btf, spec->spec[0].type_id);
	5059	+ e = btf_enum(t) + spec->spec[0].idx;
	5060	+ *val = e->val;
	5061	+ break;
	5062	+ default:
	5063	+ return -EOPNOTSUPP;
4611	5064	}
4612	5065
4613	5066	return 0;
4614	5067	}
4615	5068
	5069	+struct bpf_core_relo_res
	5070	+{
	5071	+ /* expected value in the instruction, unless validate == false */
	5072	+ __u32 orig_val;
	5073	+ /* new value that needs to be patched up to */
	5074	+ __u32 new_val;
	5075	+ /* relocation unsuccessful, poison instruction, but don't fail load */
	5076	+ bool poison;
	5077	+ /* some relocations can't be validated against orig_val */
	5078	+ bool validate;
	5079	+};
	5080	+
	5081	+/* Calculate original and target relocation values, given local and target
	5082	+ * specs and relocation kind. These values are calculated for each candidate.
	5083	+ * If there are multiple candidates, resulting values should all be consistent
	5084	+ * with each other. Otherwise, libbpf will refuse to proceed due to ambiguity.
	5085	+ * If instruction has to be poisoned, *poison will be set to true.
	5086	+ */
	5087	+static int bpf_core_calc_relo(const struct bpf_program *prog,
	5088	+ const struct bpf_core_relo *relo,
	5089	+ int relo_idx,
	5090	+ const struct bpf_core_spec *local_spec,
	5091	+ const struct bpf_core_spec *targ_spec,
	5092	+ struct bpf_core_relo_res *res)
	5093	+{
	5094	+ int err = -EOPNOTSUPP;
	5095	+
	5096	+ res->orig_val = 0;
	5097	+ res->new_val = 0;
	5098	+ res->poison = false;
	5099	+ res->validate = true;
	5100	+
	5101	+ if (core_relo_is_field_based(relo->kind)) {
	5102	+ err = bpf_core_calc_field_relo(prog, relo, local_spec, &res->orig_val, &res->validate);
	5103	+ err = err ?: bpf_core_calc_field_relo(prog, relo, targ_spec, &res->new_val, NULL);
	5104	+ } else if (core_relo_is_type_based(relo->kind)) {
	5105	+ err = bpf_core_calc_type_relo(relo, local_spec, &res->orig_val);
	5106	+ err = err ?: bpf_core_calc_type_relo(relo, targ_spec, &res->new_val);
	5107	+ } else if (core_relo_is_enumval_based(relo->kind)) {
	5108	+ err = bpf_core_calc_enumval_relo(relo, local_spec, &res->orig_val);
	5109	+ err = err ?: bpf_core_calc_enumval_relo(relo, targ_spec, &res->new_val);
	5110	+ }
	5111	+
	5112	+ if (err == -EUCLEAN) {
	5113	+ /* EUCLEAN is used to signal instruction poisoning request */
	5114	+ res->poison = true;
	5115	+ err = 0;
	5116	+ } else if (err == -EOPNOTSUPP) {
	5117	+ /* EOPNOTSUPP means unknown/unsupported relocation */
	5118	+ pr_warn("prog '%s': relo #%d: unrecognized CO-RE relocation %s (%d) at insn #%d\n",
	5119	+ bpf_program__title(prog, false), relo_idx,
	5120	+ core_relo_kind_str(relo->kind), relo->kind, relo->insn_off / 8);
	5121	+ }
	5122	+
	5123	+ return err;
	5124	+}
	5125	+
	5126	+/*
	5127	+ * Turn instruction for which CO_RE relocation failed into invalid one with
	5128	+ * distinct signature.
	5129	+ */
	5130	+static void bpf_core_poison_insn(struct bpf_program *prog, int relo_idx,
	5131	+ int insn_idx, struct bpf_insn *insn)
	5132	+{
	5133	+ pr_debug("prog '%s': relo #%d: substituting insn #%d w/ invalid insn\n",
	5134	+ bpf_program__title(prog, false), relo_idx, insn_idx);
	5135	+ insn->code = BPF_JMP \| BPF_CALL;
	5136	+ insn->dst_reg = 0;
	5137	+ insn->src_reg = 0;
	5138	+ insn->off = 0;
	5139	+ /* if this instruction is reachable (not a dead code),
	5140	+ * verifier will complain with the following message:
	5141	+ * invalid func unknown#195896080
	5142	+ */
	5143	+ insn->imm = 195896080; /* => 0xbad2310 => "bad relo" */
	5144	+}
	5145	+
	5146	+static bool is_ldimm64(struct bpf_insn *insn)
	5147	+{
	5148	+ return insn->code == (BPF_LD \| BPF_IMM \| BPF_DW);
	5149	+}
	5150	+
4616	5151	/*
4617	5152	* Patch relocatable BPF instruction.
4618	5153	*
4619	5154	* Patched value is determined by relocation kind and target specification.
4620		− * For field existence relocation target spec will be NULL if field is not
4621		− * found.
	5155	+ * For existence relocations target spec will be NULL if field/type is not found.
4622	5156	* Expected insn->imm value is determined using relocation kind and local
4623	5157	* spec, and is checked before patching instruction. If actual insn->imm value
4624	5158	* is wrong, bail out with error.
@@ -4626,58 +5160,43 @@ static int bpf_core_calc_field_relo(const struct bpf_program *prog, · 1 unmodified line
4626	5160	* Currently three kinds of BPF instructions are supported:
4627	5161	* 1. rX = <imm> (assignment with immediate operand);
4628	5162	* 2. rX += <imm> (arithmetic operations with immediate operand);
	5163	+ * 3. rX = <imm64> (load with 64-bit immediate value).
4629	5164	*/
4630		−static int bpf_core_reloc_insn(struct bpf_program *prog,
4631		− const struct bpf_field_reloc *relo,
	5165	+static int bpf_core_patch_insn(struct bpf_program *prog,
	5166	+ const struct bpf_core_relo *relo,
4632	5167	int relo_idx,
4633		− const struct bpf_core_spec *local_spec,
4634		− const struct bpf_core_spec *targ_spec)
	5168	+ const struct bpf_core_relo_res *res)
4635	5169	{
4636	5170	__u32 orig_val, new_val;
4637	5171	struct bpf_insn *insn;
4638		− bool validate = true;
4639		− int insn_idx, err;
	5172	+ int insn_idx;
4640	5173	__u8 class;
4641	5174
4642		− if (relo->insn_off % sizeof(struct bpf_insn))
	5175	+ if (relo->insn_off % BPF_INSN_SZ)
4643	5176	return -EINVAL;
4644		− insn_idx = relo->insn_off / sizeof(struct bpf_insn);
	5177	+ insn_idx = relo->insn_off / BPF_INSN_SZ;
4645	5178	insn = &prog->insns[insn_idx];
4646	5179	class = BPF_CLASS(insn->code);
4647	5180
4648		− if (relo->kind == BPF_FIELD_EXISTS) {
4649		− orig_val = 1; /* can't generate EXISTS relo w/o local field */
4650		− new_val = targ_spec ? 1 : 0;
4651		− } else if (!targ_spec) {
4652		− pr_debug("prog '%s': relo #%d: substituting insn #%d w/ invalid insn\n",
4653		− bpf_program__title(prog, false), relo_idx, insn_idx);
4654		− insn->code = BPF_JMP \| BPF_CALL;
4655		− insn->dst_reg = 0;
4656		− insn->src_reg = 0;
4657		− insn->off = 0;
4658		− /* if this instruction is reachable (not a dead code),
4659		− * verifier will complain with the following message:
4660		− * invalid func unknown#195896080
	5181	+ if (res->poison) {
	5182	+ /* poison second part of ldimm64 to avoid confusing error from
	5183	+ * verifier about "unknown opcode 00"
4661	5184	*/
4662		− insn->imm = 195896080; /* => 0xbad2310 => "bad relo" */
	5185	+ if (is_ldimm64(insn))
	5186	+ bpf_core_poison_insn(prog, relo_idx, insn_idx + 1, insn + 1);
	5187	+ bpf_core_poison_insn(prog, relo_idx, insn_idx, insn);
4663	5188	return 0;
4664		− } else {
4665		− err = bpf_core_calc_field_relo(prog, relo, local_spec,
4666		− &orig_val, &validate);
4667		− if (err)
4668		− return err;
4669		− err = bpf_core_calc_field_relo(prog, relo, targ_spec,
4670		− &new_val, NULL);
4671		− if (err)
4672		− return err;
4673	5189	}
4674	5190
	5191	+ orig_val = res->orig_val;
	5192	+ new_val = res->new_val;
	5193	+
4675	5194	switch (class) {
4676	5195	case BPF_ALU:
4677	5196	case BPF_ALU64:
4678	5197	if (BPF_SRC(insn->code) != BPF_K)
4679	5198	return -EINVAL;
4680		− if (validate && insn->imm != orig_val) {
	5199	+ if (res->validate && insn->imm != orig_val) {
4681	5200	pr_warn("prog '%s': relo #%d: unexpected insn #%d (ALU/ALU64) value: got %u, exp %u -> %u\n",
4682	5201	bpf_program__title(prog, false), relo_idx,
4683	5202	insn_idx, insn->imm, orig_val, new_val);
@@ -4692,8 +5211,8 @@ static int bpf_core_reloc_insn(struct bpf_program *prog, · 8 unmodified lines
4692	5211	case BPF_LDX:
4693	5212	case BPF_ST:
4694	5213	case BPF_STX:
4695		− if (validate && insn->off != orig_val) {
4696		− pr_warn("prog '%s': relo #%d: unexpected insn #%d (LD/LDX/ST/STX) value: got %u, exp %u -> %u\n",
	5214	+ if (res->validate && insn->off != orig_val) {
	5215	+ pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDX/ST/STX) value: got %u, exp %u -> %u\n",
4697	5216	bpf_program__title(prog, false), relo_idx,
4698	5217	insn_idx, insn->off, orig_val, new_val);
4699	5218	return -EINVAL;
@@ -4710,8 +5229,37 @@ static int bpf_core_reloc_insn(struct bpf_program *prog, · 10 unmodified lines
4710	5229	bpf_program__title(prog, false), relo_idx, insn_idx,
4711	5230	orig_val, new_val);
4712	5231	break;
	5232	+ case BPF_LD: {
	5233	+ __u64 imm;
	5234	+
	5235	+ if (!is_ldimm64(insn) \|\|
	5236	+ insn[0].src_reg != 0 \|\| insn[0].off != 0 \|\|
	5237	+ insn_idx + 1 >= prog->insns_cnt \|\|
	5238	+ insn[1].code != 0 \|\| insn[1].dst_reg != 0 \|\|
	5239	+ insn[1].src_reg != 0 \|\| insn[1].off != 0) {
	5240	+ pr_warn("prog '%s': relo #%d: insn #%d (LDIMM64) has unexpected form\n",
	5241	+ bpf_program__title(prog, false), relo_idx, insn_idx);
	5242	+ return -EINVAL;
	5243	+ }
	5244	+
	5245	+ imm = insn[0].imm + ((__u64)insn[1].imm << 32);
	5246	+ if (res->validate && imm != orig_val) {
	5247	+ pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDIMM64) value: got %llu, exp %u -> %u\n",
	5248	+ bpf_program__title(prog, false), relo_idx,
	5249	+ insn_idx, (unsigned long long)imm,
	5250	+ orig_val, new_val);
	5251	+ return -EINVAL;
	5252	+ }
	5253	+
	5254	+ insn[0].imm = new_val;
	5255	+ insn[1].imm = 0; /* currently only 32-bit values are supported */
	5256	+ pr_debug("prog '%s': relo #%d: patched insn #%d (LDIMM64) imm64 %llu -> %u\n",
	5257	+ bpf_program__title(prog, false), relo_idx, insn_idx,
	5258	+ (unsigned long long)imm, new_val);
	5259	+ break;
	5260	+ }
4713	5261	default:
4714		− pr_warn("prog '%s': relo #%d: trying to relocate unrecognized insn #%d, code:%x, src:%x, dst:%x, off:%x, imm:%x\n",
	5262	+ pr_warn("prog '%s': relo #%d: trying to relocate unrecognized insn #%d, code:0x%x, src:0x%x, dst:0x%x, off:0x%x, imm:0x%x\n",
4715	5263	bpf_program__title(prog, false), relo_idx,
4716	5264	insn_idx, insn->code, insn->src_reg, insn->dst_reg,
4717	5265	insn->off, insn->imm);
@@ -4728,29 +5276,48 @@ static int bpf_core_reloc_insn(struct bpf_program *prog, · 10 unmodified lines
4728	5276	static void bpf_core_dump_spec(int level, const struct bpf_core_spec *spec)
4729	5277	{
4730	5278	const struct btf_type *t;
	5279	+ const struct btf_enum *e;
4731	5280	const char *s;
4732	5281	__u32 type_id;
4733	5282	int i;
4734	5283
4735		− type_id = spec->spec[0].type_id;
	5284	+ type_id = spec->root_type_id;
4736	5285	t = btf__type_by_id(spec->btf, type_id);
4737	5286	s = btf__name_by_offset(spec->btf, t->name_off);
4738		− libbpf_print(level, "[%u] %s + ", type_id, s);
4739	5287
4740		− for (i = 0; i < spec->raw_len; i++)
4741		− libbpf_print(level, "%d%s", spec->raw_spec[i],
4742		− i == spec->raw_len - 1 ? " => " : ":");
	5288	+ libbpf_print(level, "[%u] %s %s", type_id, btf_kind_str(t), str_is_empty(s) ? "<anon>" : s);
4743	5289
4744		− libbpf_print(level, "%u.%u @ &x",
4745		− spec->bit_offset / 8, spec->bit_offset % 8);
	5290	+ if (core_relo_is_type_based(spec->relo_kind))
	5291	+ return;
4746	5292
4747		− for (i = 0; i < spec->len; i++) {
4748		− if (spec->spec[i].name)
4749		− libbpf_print(level, ".%s", spec->spec[i].name);
4750		− else
4751		− libbpf_print(level, "[%u]", spec->spec[i].idx);
	5293	+ if (core_relo_is_enumval_based(spec->relo_kind)) {
	5294	+ t = skip_mods_and_typedefs(spec->btf, type_id, NULL);
	5295	+ e = btf_enum(t) + spec->raw_spec[0];
	5296	+ s = btf__name_by_offset(spec->btf, e->name_off);
	5297	+
	5298	+ libbpf_print(level, "::%s = %u", s, e->val);
	5299	+ return;
4752	5300	}
4753	5301
	5302	+ if (core_relo_is_field_based(spec->relo_kind)) {
	5303	+ for (i = 0; i < spec->len; i++) {
	5304	+ if (spec->spec[i].name)
	5305	+ libbpf_print(level, ".%s", spec->spec[i].name);
	5306	+ else if (i > 0 \|\| spec->spec[i].idx > 0)
	5307	+ libbpf_print(level, "[%u]", spec->spec[i].idx);
	5308	+ }
	5309	+
	5310	+ libbpf_print(level, " (");
	5311	+ for (i = 0; i < spec->raw_len; i++)
	5312	+ libbpf_print(level, "%s%d", i == 0 ? "" : ":", spec->raw_spec[i]);
	5313	+
	5314	+ if (spec->bit_offset % 8)
	5315	+ libbpf_print(level, " @ offset %u.%u)",
	5316	+ spec->bit_offset / 8, spec->bit_offset % 8);
	5317	+ else
	5318	+ libbpf_print(level, " @ offset %u)", spec->bit_offset / 8);
	5319	+ return;
	5320	+ }
4754	5321	}
4755	5322
4756	5323	static size_t bpf_core_hash_fn(const void key, void ctx)
@@ -4814,22 +5381,23 @@ static void *u32_as_hash_key(__u32 x) · 57 unmodified lines
4814	5381	* CPU-wise compared to prebuilding a map from all local type names to
4815	5382	* a list of candidate type names. It's also sped up by caching resolved
4816	5383	* list of matching candidates per each local "root" type ID, that has at
4817		− * least one bpf_field_reloc associated with it. This list is shared
	5384	+ * least one bpf_core_relo associated with it. This list is shared
4818	5385	* between multiple relocations for the same type ID and is updated as some
4819	5386	* of the candidates are pruned due to structural incompatibility.
4820	5387	*/
4821		−static int bpf_core_reloc_field(struct bpf_program *prog,
4822		− const struct bpf_field_reloc *relo,
4823		− int relo_idx,
4824		− const struct btf *local_btf,
4825		− const struct btf *targ_btf,
4826		− struct hashmap *cand_cache)
	5388	+static int bpf_core_apply_relo(struct bpf_program *prog,
	5389	+ const struct bpf_core_relo *relo,
	5390	+ int relo_idx,
	5391	+ const struct btf *local_btf,
	5392	+ const struct btf *targ_btf,
	5393	+ struct hashmap *cand_cache)
4827	5394	{
4828	5395	const char *prog_name = bpf_program__title(prog, false);
4829		− struct bpf_core_spec local_spec, cand_spec, targ_spec;
	5396	+ struct bpf_core_spec local_spec, cand_spec, targ_spec = {};
4830	5397	const void *type_key = u32_as_hash_key(relo->type_id);
4831		− const struct btf_type local_type, cand_type;
4832		− const char local_name, cand_name;
	5398	+ struct bpf_core_relo_res cand_res, targ_res;
	5399	+ const struct btf_type *local_type;
	5400	+ const char *local_name;
4833	5401	struct ids_vec *cand_ids;
4834	5402	__u32 local_id, cand_id;
4835	5403	const char *spec_str;
@@ -4841,32 +5409,49 @@ static int bpf_core_reloc_field(struct bpf_program *prog, · 5 unmodified lines
4841	5409	return -EINVAL;
4842	5410
4843	5411	local_name = btf__name_by_offset(local_btf, local_type->name_off);
4844		− if (str_is_empty(local_name))
	5412	+ if (!local_name)
4845	5413	return -EINVAL;
4846	5414
4847	5415	spec_str = btf__name_by_offset(local_btf, relo->access_str_off);
4848	5416	if (str_is_empty(spec_str))
4849	5417	return -EINVAL;
4850	5418
4851		− err = bpf_core_spec_parse(local_btf, local_id, spec_str, &local_spec);
	5419	+ err = bpf_core_parse_spec(local_btf, local_id, spec_str, relo->kind, &local_spec);
4852	5420	if (err) {
4853		− pr_warn("prog '%s': relo #%d: parsing [%d] %s + %s failed: %d\n",
4854		− prog_name, relo_idx, local_id, local_name, spec_str,
4855		− err);
	5421	+ pr_warn("prog '%s': relo #%d: parsing [%d] %s %s + %s failed: %d\n",
	5422	+ prog_name, relo_idx, local_id, btf_kind_str(local_type),
	5423	+ str_is_empty(local_name) ? "<anon>" : local_name,
	5424	+ spec_str, err);
4856	5425	return -EINVAL;
4857	5426	}
4858	5427
4859		− pr_debug("prog '%s': relo #%d: kind %d, spec is ", prog_name, relo_idx,
4860		− relo->kind);
	5428	+ pr_debug("prog '%s': relo #%d: kind <%s> (%d), spec is ", prog_name,
	5429	+ relo_idx, core_relo_kind_str(relo->kind), relo->kind);
4861	5430	bpf_core_dump_spec(LIBBPF_DEBUG, &local_spec);
4862	5431	libbpf_print(LIBBPF_DEBUG, "\n");
4863	5432
	5433	+ /* TYPE_ID_LOCAL relo is special and doesn't need candidate search */
	5434	+ if (relo->kind == BPF_TYPE_ID_LOCAL) {
	5435	+ targ_res.validate = true;
	5436	+ targ_res.poison = false;
	5437	+ targ_res.orig_val = local_spec.root_type_id;
	5438	+ targ_res.new_val = local_spec.root_type_id;
	5439	+ goto patch_insn;
	5440	+ }
	5441	+
	5442	+ /* libbpf doesn't support candidate search for anonymous types */
	5443	+ if (str_is_empty(spec_str)) {
	5444	+ pr_warn("prog '%s': relo #%d: <%s> (%d) relocation doesn't support anonymous types\n",
	5445	+ prog_name, relo_idx, core_relo_kind_str(relo->kind), relo->kind);
	5446	+ return -EOPNOTSUPP;
	5447	+ }
	5448	+
4864	5449	if (!hashmap__find(cand_cache, type_key, (void **)&cand_ids)) {
4865	5450	cand_ids = bpf_core_find_cands(local_btf, local_id, targ_btf);
4866	5451	if (IS_ERR(cand_ids)) {
4867		− pr_warn("prog '%s': relo #%d: target candidate search failed for [%d] %s: %ld",
4868		− prog_name, relo_idx, local_id, local_name,
4869		− PTR_ERR(cand_ids));
	5452	+ pr_warn("prog '%s': relo #%d: target candidate search failed for [%d] %s %s: %ld",
	5453	+ prog_name, relo_idx, local_id, btf_kind_str(local_type),
	5454	+ local_name, PTR_ERR(cand_ids));
4870	5455	return PTR_ERR(cand_ids);
4871	5456	}
4872	5457	err = hashmap__set(cand_cache, type_key, cand_ids, NULL, NULL);
@@ -4878,36 +5463,51 @@ static int bpf_core_reloc_field(struct bpf_program *prog, · 5 unmodified lines
4878	5463
4879	5464	for (i = 0, j = 0; i < cand_ids->len; i++) {
4880	5465	cand_id = cand_ids->data[i];
4881		− cand_type = btf__type_by_id(targ_btf, cand_id);
4882		− cand_name = btf__name_by_offset(targ_btf, cand_type->name_off);
4883		−
4884		− err = bpf_core_spec_match(&local_spec, targ_btf,
4885		− cand_id, &cand_spec);
4886		− pr_debug("prog '%s': relo #%d: matching candidate #%d %s against spec ",
4887		− prog_name, relo_idx, i, cand_name);
4888		− bpf_core_dump_spec(LIBBPF_DEBUG, &cand_spec);
4889		− libbpf_print(LIBBPF_DEBUG, ": %d\n", err);
	5466	+ err = bpf_core_spec_match(&local_spec, targ_btf, cand_id, &cand_spec);
4890	5467	if (err < 0) {
4891		− pr_warn("prog '%s': relo #%d: matching error: %d\n",
4892		− prog_name, relo_idx, err);
	5468	+ pr_warn("prog '%s': relo #%d: error matching candidate #%d ",
	5469	+ prog_name, relo_idx, i);
	5470	+ bpf_core_dump_spec(LIBBPF_WARN, &cand_spec);
	5471	+ libbpf_print(LIBBPF_WARN, ": %d\n", err);
4893	5472	return err;
4894	5473	}
	5474	+
	5475	+ pr_debug("prog '%s': relo #%d: %s candidate #%d ", prog_name,
	5476	+ relo_idx, err == 0 ? "non-matching" : "matching", i);
	5477	+ bpf_core_dump_spec(LIBBPF_DEBUG, &cand_spec);
	5478	+ libbpf_print(LIBBPF_DEBUG, "\n");
	5479	+
4895	5480	if (err == 0)
4896	5481	continue;
4897	5482
	5483	+ err = bpf_core_calc_relo(prog, relo, relo_idx, &local_spec, &cand_spec, &cand_res);
	5484	+ if (err)
	5485	+ return err;
	5486	+
4898	5487	if (j == 0) {
	5488	+ targ_res = cand_res;
4899	5489	targ_spec = cand_spec;
4900	5490	} else if (cand_spec.bit_offset != targ_spec.bit_offset) {
4901		− /* if there are many candidates, they should all
4902		− * resolve to the same bit offset
	5491	+ /* if there are many field relo candidates, they
	5492	+ * should all resolve to the same bit offset
4903	5493	*/
4904		− pr_warn("prog '%s': relo #%d: offset ambiguity: %u != %u\n",
	5494	+ pr_warn("prog '%s': relo #%d: field offset ambiguity: %u != %u\n",
4905	5495	prog_name, relo_idx, cand_spec.bit_offset,
4906	5496	targ_spec.bit_offset);
4907	5497	return -EINVAL;
	5498	+ } else if (cand_res.poison != targ_res.poison \|\| cand_res.new_val != targ_res.new_val) {
	5499	+ /* all candidates should result in the same relocation
	5500	+ * decision and value, otherwise it's dangerous to
	5501	+ * proceed due to ambiguity
	5502	+ */
	5503	+ pr_warn("prog '%s': relo #%d: relocation decision ambiguity: %s %u != %s %u\n",
	5504	+ prog_name, relo_idx,
	5505	+ cand_res.poison ? "failure" : "success", cand_res.new_val,
	5506	+ targ_res.poison ? "failure" : "success", targ_res.new_val);
	5507	+ return -EINVAL;
4908	5508	}
4909	5509
4910		− cand_ids->data[j++] = cand_spec.spec[0].type_id;
	5510	+ cand_ids->data[j++] = cand_spec.root_type_id;
4911	5511	}
4912	5512
4913	5513	/*
@@ -4926,19 +5526,25 @@ static int bpf_core_reloc_field(struct bpf_program *prog, · 12 unmodified lines
4926	5526	* as well as expected case, depending whether instruction w/
4927	5527	* relocation is guarded in some way that makes it unreachable (dead
4928	5528	* code) if relocation can't be resolved. This is handled in
4929		− * bpf_core_reloc_insn() uniformly by replacing that instruction with
	5529	+ * bpf_core_patch_insn() uniformly by replacing that instruction with
4930	5530	* BPF helper call insn (using invalid helper ID). If that instruction
4931	5531	* is indeed unreachable, then it will be ignored and eliminated by
4932	5532	* verifier. If it was an error, then verifier will complain and point
4933	5533	* to a specific instruction number in its log.
4934	5534	*/
4935		− if (j == 0)
4936		− pr_debug("prog '%s': relo #%d: no matching targets found for [%d] %s + %s\n",
4937		− prog_name, relo_idx, local_id, local_name, spec_str);
	5535	+ if (j == 0) {
	5536	+ pr_debug("prog '%s': relo #%d: no matching targets found\n",
	5537	+ prog_name, relo_idx);
	5538	+
	5539	+ /* calculate single target relo result explicitly */
	5540	+ err = bpf_core_calc_relo(prog, relo, relo_idx, &local_spec, NULL, &targ_res);
	5541	+ if (err)
	5542	+ return err;
	5543	+ }
4938	5544
4939		− /* bpf_core_reloc_insn should know how to handle missing targ_spec */
4940		− err = bpf_core_reloc_insn(prog, relo, relo_idx, &local_spec,
4941		− j ? &targ_spec : NULL);
	5545	+patch_insn:
	5546	+ /* bpf_core_patch_insn() should know how to handle missing targ_spec */
	5547	+ err = bpf_core_patch_insn(prog, relo, relo_idx, &targ_res);
4942	5548	if (err) {
4943	5549	pr_warn("prog '%s': relo #%d: failed to patch insn at offset %d: %d\n",
4944	5550	prog_name, relo_idx, relo->insn_off, err);
@@ -4949,10 +5555,10 @@ static int bpf_core_reloc_field(struct bpf_program *prog, · 4 unmodified lines
4949	5555	}
4950	5556
4951	5557	static int
4952		−bpf_core_reloc_fields(struct bpf_object obj, const char targ_btf_path)
	5558	+bpf_object__relocate_core(struct bpf_object obj, const char targ_btf_path)
4953	5559	{
4954	5560	const struct btf_ext_info_sec *sec;
4955		− const struct bpf_field_reloc *rec;
	5561	+ const struct bpf_core_relo *rec;
4956	5562	const struct btf_ext_info *seg;
4957	5563	struct hashmap_entry *entry;
4958	5564	struct hashmap *cand_cache = NULL;
@@ -4961,6 +5567,9 @@ bpf_core_reloc_fields(struct bpf_object obj, const char targ_btf_path) · 2 unmodified lines
4961	5567	const char *sec_name;
4962	5568	int i, err = 0;
4963	5569
	5570	+ if (obj->btf_ext->core_relo_info.len == 0)
	5571	+ return 0;
	5572	+
4964	5573	if (targ_btf_path)
4965	5574	targ_btf = btf__parse_elf(targ_btf_path, NULL);
4966	5575	else
@@ -4976,7 +5585,7 @@ bpf_core_reloc_fields(struct bpf_object obj, const char targ_btf_path) · 9 unmodified lines
4976	5585	goto out;
4977	5586	}
4978	5587
4979		− seg = &obj->btf_ext->field_reloc_info;
	5588	+ seg = &obj->btf_ext->core_relo_info;
4980	5589	for_each_btf_ext_sec(seg, sec) {
4981	5590	sec_name = btf__name_by_offset(obj->btf, sec->sec_name_off);
4982	5591	if (str_is_empty(sec_name)) {
@@ -4997,15 +5606,15 @@ bpf_core_reloc_fields(struct bpf_object obj, const char targ_btf_path) · 14 unmodified lines
4997	5606	goto out;
4998	5607	}
4999	5608
5000		− pr_debug("prog '%s': performing %d CO-RE offset relocs\n",
	5609	+ pr_debug("sec '%s': found %d CO-RE relocations\n",
5001	5610	sec_name, sec->num_info);
5002	5611
5003	5612	for_each_btf_ext_rec(seg, sec, i, rec) {
5004		− err = bpf_core_reloc_field(prog, rec, i, obj->btf,
5005		− targ_btf, cand_cache);
	5613	+ err = bpf_core_apply_relo(prog, rec, i, obj->btf,
	5614	+ targ_btf, cand_cache);
5006	5615	if (err) {
5007	5616	pr_warn("prog '%s': relo #%d: failed to relocate: %d\n",
5008		− sec_name, i, err);
	5617	+ prog->name, i, err);
5009	5618	goto out;
5010	5619	}
5011	5620	}
@@ -5024,17 +5633,6 @@ out: · 12 unmodified lines
5024	5633	return err;
5025	5634	}
5026	5635
5027		−static int
5028		−bpf_object__relocate_core(struct bpf_object obj, const char targ_btf_path)
5029		−{
5030		− int err = 0;
5031		−
5032		− if (obj->btf_ext->field_reloc_info.len)
5033		− err = bpf_core_reloc_fields(obj, targ_btf_path);
5034		−
5035		− return err;
5036		−}
5037		−
5038	5636	static int
5039	5637	bpf_program__reloc_text(struct bpf_program prog, struct bpf_object obj,
5040	5638	struct reloc_desc *relo)
@@ -5051,7 +5649,7 @@ bpf_program__reloc_text(struct bpf_program prog, struct bpf_object obj, · 10 unmodified lines
5051	5649	return -LIBBPF_ERRNO__RELOC;
5052	5650	}
5053	5651	new_cnt = prog->insns_cnt + text->insns_cnt;
5054		− new_insn = reallocarray(prog->insns, new_cnt, sizeof(*insn));
	5652	+ new_insn = libbpf_reallocarray(prog->insns, new_cnt, sizeof(*insn));
5055	5653	if (!new_insn) {
5056	5654	pr_warn("oom in prog realloc\n");
5057	5655	return -ENOMEM;
@@ -5136,7 +5734,8 @@ bpf_program__relocate(struct bpf_program prog, struct bpf_object obj) · 78 unmodified lines
5136	5734	return err;
5137	5735	break;
5138	5736	default:
5139		− pr_warn("relo #%d: bad relo type %d\n", i, relo->type);
	5737	+ pr_warn("prog '%s': relo #%d: bad relo type %d\n",
	5738	+ prog->name, i, relo->type);
5140	5739	return -EINVAL;
5141	5740	}
5142	5741	}
@@ -5171,7 +5770,8 @@ bpf_object__relocate(struct bpf_object obj, const char targ_btf_path) · 28 unmodified lines
5171	5770
5172	5771	err = bpf_program__relocate(prog, obj);
5173	5772	if (err) {
5174		− pr_warn("failed to relocate '%s'\n", prog->section_name);
	5773	+ pr_warn("prog '%s': failed to relocate data references: %d\n",
	5774	+ prog->name, err);
5175	5775	return err;
5176	5776	}
5177	5777	break;
@@ -5186,7 +5786,8 @@ bpf_object__relocate(struct bpf_object obj, const char targ_btf_path) · 8 unmodified lines
5186	5786
5187	5787	err = bpf_program__relocate(prog, obj);
5188	5788	if (err) {
5189		− pr_warn("failed to relocate '%s'\n", prog->section_name);
	5789	+ pr_warn("prog '%s': failed to relocate calls: %d\n",
	5790	+ prog->name, err);
5190	5791	return err;
5191	5792	}
5192	5793	}
@@ -5230,8 +5831,7 @@ static int bpf_object__collect_map_relos(struct bpf_object *obj, · 37 unmodified lines
5230	5831	i, (size_t)GELF_R_SYM(rel.r_info));
5231	5832	return -LIBBPF_ERRNO__FORMAT;
5232	5833	}
5233		− name = elf_strptr(obj->efile.elf, obj->efile.strtabidx,
5234		− sym.st_name) ? : "<?>";
	5834	+ name = elf_sym_str(obj, sym.st_name) ?: "<?>";
5235	5835	if (sym.st_shndx != obj->efile.btf_maps_shndx) {
5236	5836	pr_warn(".maps relo #%d: '%s' isn't a BTF-defined map\n",
5237	5837	i, name);
@@ -5293,7 +5893,7 @@ static int bpf_object__collect_map_relos(struct bpf_object *obj, · 55 unmodified lines
5293	5893	moff /= bpf_ptr_sz;
5294	5894	if (moff >= map->init_slots_sz) {
5295	5895	new_sz = moff + 1;
5296		− tmp = realloc(map->init_slots, new_sz * host_ptr_sz);
	5896	+ tmp = libbpf_reallocarray(map->init_slots, new_sz, host_ptr_sz);
5297	5897	if (!tmp)
5298	5898	return -ENOMEM;
5299	5899	map->init_slots = tmp;
@@ -5348,6 +5948,51 @@ static int bpf_object__collect_reloc(struct bpf_object *obj) · 48 unmodified lines
5348	5948	return 0;
5349	5949	}
5350	5950
	5951	+static bool insn_is_helper_call(struct bpf_insn insn, enum bpf_func_id func_id)
	5952	+{
	5953	+ if (BPF_CLASS(insn->code) == BPF_JMP &&
	5954	+ BPF_OP(insn->code) == BPF_CALL &&
	5955	+ BPF_SRC(insn->code) == BPF_K &&
	5956	+ insn->src_reg == 0 &&
	5957	+ insn->dst_reg == 0) {
	5958	+ *func_id = insn->imm;
	5959	+ return true;
	5960	+ }
	5961	+ return false;
	5962	+}
	5963	+
	5964	+static int bpf_object__sanitize_prog(struct bpf_object* obj, struct bpf_program *prog)
	5965	+{
	5966	+ struct bpf_insn *insn = prog->insns;
	5967	+ enum bpf_func_id func_id;
	5968	+ int i;
	5969	+
	5970	+ for (i = 0; i < prog->insns_cnt; i++, insn++) {
	5971	+ if (!insn_is_helper_call(insn, &func_id))
	5972	+ continue;
	5973	+
	5974	+ /* on kernels that don't yet support
	5975	+ * bpf_probe_read_{kernel,user}[_str] helpers, fall back
	5976	+ * to bpf_probe_read() which works well for old kernels
	5977	+ */
	5978	+ switch (func_id) {
	5979	+ case BPF_FUNC_probe_read_kernel:
	5980	+ case BPF_FUNC_probe_read_user:
	5981	+ if (!kernel_supports(FEAT_PROBE_READ_KERN))
	5982	+ insn->imm = BPF_FUNC_probe_read;
	5983	+ break;
	5984	+ case BPF_FUNC_probe_read_kernel_str:
	5985	+ case BPF_FUNC_probe_read_user_str:
	5986	+ if (!kernel_supports(FEAT_PROBE_READ_KERN))
	5987	+ insn->imm = BPF_FUNC_probe_read_str;
	5988	+ break;
	5989	+ default:
	5990	+ break;
	5991	+ }
	5992	+ }
	5993	+ return 0;
	5994	+}
	5995	+
5351	5996	static int
5352	5997	load_program(struct bpf_program prog, struct bpf_insn insns, int insns_cnt,
5353	5998	char license, __u32 kern_version, int pfd)
@@ -5364,12 +6009,12 @@ load_program(struct bpf_program prog, struct bpf_insn insns, int insns_cnt, · 10 unmodified lines
5364	6009	memset(&load_attr, 0, sizeof(struct bpf_load_program_attr));
5365	6010	load_attr.prog_type = prog->type;
5366	6011	/* old kernels might not support specifying expected_attach_type */
5367		− if (!prog->caps->exp_attach_type && prog->sec_def &&
	6012	+ if (!kernel_supports(FEAT_EXP_ATTACH_TYPE) && prog->sec_def &&
5368	6013	prog->sec_def->is_exp_attach_type_optional)
5369	6014	load_attr.expected_attach_type = 0;
5370	6015	else
5371	6016	load_attr.expected_attach_type = prog->expected_attach_type;
5372		− if (prog->caps->name)
	6017	+ if (kernel_supports(FEAT_PROG_NAME))
5373	6018	load_attr.name = prog->name;
5374	6019	load_attr.insns = insns;
5375	6020	load_attr.insns_cnt = insns_cnt;
@@ -5387,7 +6032,7 @@ load_program(struct bpf_program prog, struct bpf_insn insns, int insns_cnt, · 11 unmodified lines
5387	6032	}
5388	6033	/* specify func_info/line_info only if kernel supports them */
5389	6034	btf_fd = bpf_object__btf_fd(prog->obj);
5390		− if (btf_fd >= 0 && prog->obj->caps.btf_func) {
	6035	+ if (btf_fd >= 0 && kernel_supports(FEAT_BTF_FUNC)) {
5391	6036	load_attr.prog_btf_fd = btf_fd;
5392	6037	load_attr.func_info = prog->func_info;
5393	6038	load_attr.func_info_rec_size = prog->func_info_rec_size;
@@ -5425,7 +6070,7 @@ retry_load: · 31 unmodified lines
5425	6070	free(log_buf);
5426	6071	goto retry_load;
5427	6072	}
5428		− ret = -errno;
	6073	+ ret = errno ? -errno : -LIBBPF_ERRNO__LOAD;
5429	6074	cp = libbpf_strerror_r(errno, errmsg, sizeof(errmsg));
5430	6075	pr_warn("load bpf program failed: %s\n", cp);
5431	6076	pr_perm_msg(ret);
@@ -5562,13 +6207,19 @@ bpf_object__load_progs(struct bpf_object *obj, int log_level) · 130 unmodified lines
5562	6207	size_t i;
5563	6208	int err;
5564	6209
	6210	+ for (i = 0; i < obj->nr_programs; i++) {
	6211	+ prog = &obj->programs[i];
	6212	+ err = bpf_object__sanitize_prog(obj, prog);
	6213	+ if (err)
	6214	+ return err;
	6215	+ }
	6216	+
5565	6217	for (i = 0; i < obj->nr_programs; i++) {
5566	6218	prog = &obj->programs[i];
5567	6219	if (bpf_program__is_function_storage(prog, obj))
5568	6220	continue;
5569	6221	if (!prog->load) {
5570		− pr_debug("prog '%s'('%s'): skipped loading\n",
5571		− prog->name, prog->section_name);
	6222	+ pr_debug("prog '%s': skipped loading\n", prog->name);
5572	6223	continue;
5573	6224	}
5574	6225	prog->log_level \|= log_level;
@@ -5641,6 +6292,8 @@ __bpf_object__open(const char path, const void obj_buf, size_t obj_buf_sz, · 66 unmodified lines
5641	6292	/* couldn't guess, but user might manually specify */
5642	6293	continue;
5643	6294
	6295	+ if (prog->sec_def->is_sleepable)
	6296	+ prog->prog_flags \|= BPF_F_SLEEPABLE;
5644	6297	bpf_program__set_type(prog, prog->sec_def->prog_type);
5645	6298	bpf_program__set_expected_attach_type(prog,
5646	6299	prog->sec_def->expected_attach_type);
@@ -5750,11 +6403,11 @@ static int bpf_object__sanitize_maps(struct bpf_object *obj) · 103 unmodified lines
5750	6403	bpf_object__for_each_map(m, obj) {
5751	6404	if (!bpf_map__is_internal(m))
5752	6405	continue;
5753		− if (!obj->caps.global_data) {
	6406	+ if (!kernel_supports(FEAT_GLOBAL_DATA)) {
5754	6407	pr_warn("kernel doesn't support global data\n");
5755	6408	return -ENOTSUP;
5756	6409	}
5757		− if (!obj->caps.array_mmap)
	6410	+ if (!kernel_supports(FEAT_ARRAY_MMAP))
5758	6411	m->def.map_flags ^= BPF_F_MMAPABLE;
5759	6412	}
5760	6413
@@ -5904,7 +6557,6 @@ int bpf_object__load_xattr(struct bpf_object_load_attr *attr) · 143 unmodified lines
5904	6557	}
5905	6558
5906	6559	err = bpf_object__probe_loading(obj);
5907		− err = err ? : bpf_object__probe_caps(obj);
5908	6560	err = err ? : bpf_object__resolve_externs(obj, obj->kconfig);
5909	6561	err = err ? : bpf_object__sanitize_and_load_btf(obj);
5910	6562	err = err ? : bpf_object__sanitize_maps(obj);
@@ -6713,7 +7365,7 @@ int bpf_program__fd(const struct bpf_program *prog) · 802 unmodified lines
6713	7365
6714	7366	size_t bpf_program__size(const struct bpf_program *prog)
6715	7367	{
6716		− return prog->insns_cnt * sizeof(struct bpf_insn);
	7368	+ return prog->insns_cnt * BPF_INSN_SZ;
6717	7369	}
6718	7370
6719	7371	int bpf_program__set_prep(struct bpf_program *prog, int nr_instances,
@@ -6910,6 +7562,21 @@ static const struct bpf_sec_def section_defs[] = { · 190 unmodified lines
6910	7562	.expected_attach_type = BPF_TRACE_FEXIT,
6911	7563	.is_attach_btf = true,
6912	7564	.attach_fn = attach_trace),
	7565	+ SEC_DEF("fentry.s/", TRACING,
	7566	+ .expected_attach_type = BPF_TRACE_FENTRY,
	7567	+ .is_attach_btf = true,
	7568	+ .is_sleepable = true,
	7569	+ .attach_fn = attach_trace),
	7570	+ SEC_DEF("fmod_ret.s/", TRACING,
	7571	+ .expected_attach_type = BPF_MODIFY_RETURN,
	7572	+ .is_attach_btf = true,
	7573	+ .is_sleepable = true,
	7574	+ .attach_fn = attach_trace),
	7575	+ SEC_DEF("fexit.s/", TRACING,
	7576	+ .expected_attach_type = BPF_TRACE_FEXIT,
	7577	+ .is_attach_btf = true,
	7578	+ .is_sleepable = true,
	7579	+ .attach_fn = attach_trace),
6913	7580	SEC_DEF("freplace/", EXT,
6914	7581	.is_attach_btf = true,
6915	7582	.attach_fn = attach_trace),
@@ -6917,6 +7584,11 @@ static const struct bpf_sec_def section_defs[] = { · 1 unmodified line
6917	7584	.is_attach_btf = true,
6918	7585	.expected_attach_type = BPF_LSM_MAC,
6919	7586	.attach_fn = attach_lsm),
	7587	+ SEC_DEF("lsm.s/", LSM,
	7588	+ .is_attach_btf = true,
	7589	+ .is_sleepable = true,
	7590	+ .expected_attach_type = BPF_LSM_MAC,
	7591	+ .attach_fn = attach_lsm),
6920	7592	SEC_DEF("iter/", TRACING,
6921	7593	.expected_attach_type = BPF_TRACE_ITER,
6922	7594	.is_attach_btf = true,
@@ -7122,8 +7794,7 @@ static int bpf_object__collect_st_ops_relos(struct bpf_object *obj, · 199 unmodified lines
7122	7794	return -LIBBPF_ERRNO__FORMAT;
7123	7795	}
7124	7796
7125		− name = elf_strptr(obj->efile.elf, obj->efile.strtabidx,
7126		− sym.st_name) ? : "<?>";
	7797	+ name = elf_sym_str(obj, sym.st_name) ?: "<?>";
7127	7798	map = find_struct_ops_map_by_offset(obj, rel.r_offset);
7128	7799	if (!map) {
7129	7800	pr_warn("struct_ops reloc: cannot find map at rel.r_offset %zu\n",
@@ -7640,7 +8311,7 @@ int bpf_prog_load_xattr(const struct bpf_prog_load_attr *attr, · 510 unmodified lines
7640	8311
7641	8312	prog->prog_ifindex = attr->ifindex;
7642	8313	prog->log_level = attr->log_level;
7643		− prog->prog_flags = attr->prog_flags;
	8314	+ prog->prog_flags \|= attr->prog_flags;
7644	8315	if (!first_prog)
7645	8316	first_prog = prog;
7646	8317	}
@@ -8594,7 +9265,7 @@ struct perf_buffer *perf_buffer__new(int map_fd, size_t page_cnt, · 947 unmodified lines
8594	9265	struct perf_buffer_params p = {};
8595	9266	struct perf_event_attr attr = { 0, };
8596	9267
8597		− attr.config = PERF_COUNT_SW_BPF_OUTPUT,
	9268	+ attr.config = PERF_COUNT_SW_BPF_OUTPUT;
8598	9269	attr.type = PERF_TYPE_SOFTWARE;
8599	9270	attr.sample_type = PERF_SAMPLE_RAW;
8600	9271	attr.sample_period = 1;
@@ -8832,6 +9503,11 @@ static int perf_buffer__process_records(struct perf_buffer *pb, · 231 unmodified lines
8832	9503	return 0;
8833	9504	}
8834	9505
	9506	+int perf_buffer__epoll_fd(const struct perf_buffer *pb)
	9507	+{
	9508	+ return pb->epoll_fd;
	9509	+}
	9510	+
8835	9511	int perf_buffer__poll(struct perf_buffer *pb, int timeout_ms)
8836	9512	{
8837	9513	int i, cnt, err;
@@ -8849,6 +9525,55 @@ int perf_buffer__poll(struct perf_buffer *pb, int timeout_ms) · 11 unmodified lines
8849	9525	return cnt < 0 ? -errno : cnt;
8850	9526	}
8851	9527
	9528	+/* Return number of PERF_EVENT_ARRAY map slots set up by this perf_buffer
	9529	+ * manager.
	9530	+ */
	9531	+size_t perf_buffer__buffer_cnt(const struct perf_buffer *pb)
	9532	+{
	9533	+ return pb->cpu_cnt;
	9534	+}
	9535	+
	9536	+/*
	9537	+ * Return perf_event FD of a ring buffer in buf_idx slot of
	9538	+ * PERF_EVENT_ARRAY BPF map. This FD can be polled for new data using
	9539	+ * select()/poll()/epoll() Linux syscalls.
	9540	+ */
	9541	+int perf_buffer__buffer_fd(const struct perf_buffer *pb, size_t buf_idx)
	9542	+{
	9543	+ struct perf_cpu_buf *cpu_buf;
	9544	+
	9545	+ if (buf_idx >= pb->cpu_cnt)
	9546	+ return -EINVAL;
	9547	+
	9548	+ cpu_buf = pb->cpu_bufs[buf_idx];
	9549	+ if (!cpu_buf)
	9550	+ return -ENOENT;
	9551	+
	9552	+ return cpu_buf->fd;
	9553	+}
	9554	+
	9555	+/*
	9556	+ * Consume data from perf ring buffer corresponding to slot buf_idx in
	9557	+ * PERF_EVENT_ARRAY BPF map without waiting/polling. If there is no data to
	9558	+ * consume, do nothing and return success.
	9559	+ * Returns:
	9560	+ * - 0 on success;
	9561	+ * - <0 on failure.
	9562	+ */
	9563	+int perf_buffer__consume_buffer(struct perf_buffer *pb, size_t buf_idx)
	9564	+{
	9565	+ struct perf_cpu_buf *cpu_buf;
	9566	+
	9567	+ if (buf_idx >= pb->cpu_cnt)
	9568	+ return -EINVAL;
	9569	+
	9570	+ cpu_buf = pb->cpu_bufs[buf_idx];
	9571	+ if (!cpu_buf)
	9572	+ return -ENOENT;
	9573	+
	9574	+ return perf_buffer__process_records(pb, cpu_buf);
	9575	+}
	9576	+
8852	9577	int perf_buffer__consume(struct perf_buffer *pb)
8853	9578	{
8854	9579	int i, err;
@@ -8861,7 +9586,7 @@ int perf_buffer__consume(struct perf_buffer *pb) · 6 unmodified lines
8861	9586
8862	9587	err = perf_buffer__process_records(pb, cpu_buf);
8863	9588	if (err) {
8864		− pr_warn("error while processing records: %d\n", err);
	9589	+ pr_warn("perf_buffer: failed to process records in buffer #%d: %d\n", i, err);
8865	9590	return err;
8866	9591	}
8867	9592	}

tools/lib/bpf/libbpf.h+4−0

@@ -588,8 +588,12 @@ perf_buffer__new_raw(int map_fd, size_t page_cnt, · 587 unmodified lines
588	588	const struct perf_buffer_raw_opts *opts);
589	589
590	590	LIBBPF_API void perf_buffer__free(struct perf_buffer *pb);
	591	+LIBBPF_API int perf_buffer__epoll_fd(const struct perf_buffer *pb);
591	592	LIBBPF_API int perf_buffer__poll(struct perf_buffer *pb, int timeout_ms);
592	593	LIBBPF_API int perf_buffer__consume(struct perf_buffer *pb);
	594	+LIBBPF_API int perf_buffer__consume_buffer(struct perf_buffer *pb, size_t buf_idx);
	595	+LIBBPF_API size_t perf_buffer__buffer_cnt(const struct perf_buffer *pb);
	596	+LIBBPF_API int perf_buffer__buffer_fd(const struct perf_buffer *pb, size_t buf_idx);
593	597
594	598	typedef enum bpf_perf_event_ret
595	599	(bpf_perf_event_print_t)(struct perf_event_header hdr,

tools/lib/bpf/libbpf.map+9−0

@@ -299,3 +299,12 @@ LIBBPF_0.1.0 { · 298 unmodified lines
299	299	btf__set_fd;
300	300	btf__set_pointer_size;
301	301	} LIBBPF_0.0.9;
	302	+
	303	+LIBBPF_0.2.0 {
	304	+ global:
	305	+ perf_buffer__buffer_cnt;
	306	+ perf_buffer__buffer_fd;
	307	+ perf_buffer__epoll_fd;
	308	+ perf_buffer__consume_buffer;
	309	+ xsk_socket__create_shared;
	310	+} LIBBPF_0.1.0;

tools/lib/bpf/libbpf_internal.h+107−31

@@ -9,6 +9,15 @@ · 8 unmodified lines
9	9	#ifndef __LIBBPF_LIBBPF_INTERNAL_H
10	10	#define __LIBBPF_LIBBPF_INTERNAL_H
11	11
	12	+#include <stdlib.h>
	13	+#include <limits.h>
	14	+
	15	+/* make sure libbpf doesn't use kernel-only integer typedefs */
	16	+#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
	17	+
	18	+/* prevent accidental re-addition of reallocarray() */
	19	+#pragma GCC poison reallocarray
	20	+
12	21	#include "libbpf.h"
13	22
14	23	#define BTF_INFO_ENC(kind, kind_flag, vlen) \
@@ -23,6 +32,12 @@ · 8 unmodified lines
23	32	#define BTF_PARAM_ENC(name, type) (name), (type)
24	33	#define BTF_VAR_SECINFO_ENC(type, offset, size) (type), (offset), (size)
25	34
	35	+#ifndef likely
	36	+#define likely(x) __builtin_expect(!!(x), 1)
	37	+#endif
	38	+#ifndef unlikely
	39	+#define unlikely(x) __builtin_expect(!!(x), 0)
	40	+#endif
26	41	#ifndef min
27	42	# define min(x, y) ((x) < (y) ? (x) : (y))
28	43	#endif
@@ -63,6 +78,33 @@ do { \ · 34 unmodified lines
63	78	#define pr_info(fmt, ...) __pr(LIBBPF_INFO, fmt, ##__VA_ARGS__)
64	79	#define pr_debug(fmt, ...) __pr(LIBBPF_DEBUG, fmt, ##__VA_ARGS__)
65	80
	81	+#ifndef __has_builtin
	82	+#define __has_builtin(x) 0
	83	+#endif
	84	+/*
	85	+ * Re-implement glibc's reallocarray() for libbpf internal-only use.
	86	+ * reallocarray(), unfortunately, is not available in all versions of glibc,
	87	+ * so requires extra feature detection and using reallocarray() stub from
	88	+ * <tools/libc_compat.h> and COMPAT_NEED_REALLOCARRAY. All this complicates
	89	+ * build of libbpf unnecessarily and is just a maintenance burden. Instead,
	90	+ * it's trivial to implement libbpf-specific internal version and use it
	91	+ * throughout libbpf.
	92	+ */
	93	+static inline void libbpf_reallocarray(void ptr, size_t nmemb, size_t size)
	94	+{
	95	+ size_t total;
	96	+
	97	+#if __has_builtin(__builtin_mul_overflow)
	98	+ if (unlikely(__builtin_mul_overflow(nmemb, size, &total)))
	99	+ return NULL;
	100	+#else
	101	+ if (size == 0 \|\| nmemb > ULONG_MAX / size)
	102	+ return NULL;
	103	+ total = nmemb * size;
	104	+#endif
	105	+ return realloc(ptr, total);
	106	+}
	107	+
66	108	static inline bool libbpf_validate_opts(const char *opts,
67	109	size_t opts_sz, size_t user_sz,
68	110	const char *type_name)
@@ -105,18 +147,6 @@ int bpf_object__section_size(const struct bpf_object obj, const char name, · 36 unmodified lines
105	147	int bpf_object__variable_offset(const struct bpf_object obj, const char name,
106	148	__u32 *off);
107	149
108		−struct nlattr;
109		−typedef int (libbpf_dump_nlmsg_t)(void cookie, void msg, struct nlattr *tb);
110		−int libbpf_netlink_open(unsigned int *nl_pid);
111		−int libbpf_nl_get_link(int sock, unsigned int nl_pid,
112		− libbpf_dump_nlmsg_t dump_link_nlmsg, void *cookie);
113		−int libbpf_nl_get_class(int sock, unsigned int nl_pid, int ifindex,
114		− libbpf_dump_nlmsg_t dump_class_nlmsg, void *cookie);
115		−int libbpf_nl_get_qdisc(int sock, unsigned int nl_pid, int ifindex,
116		− libbpf_dump_nlmsg_t dump_qdisc_nlmsg, void *cookie);
117		−int libbpf_nl_get_filter(int sock, unsigned int nl_pid, int ifindex, int handle,
118		− libbpf_dump_nlmsg_t dump_filter_nlmsg, void *cookie);
119		−
120	150	struct btf_ext_info {
121	151	/*
122	152	* info points to the individual info section (e.g. func_info and
@@ -138,6 +168,44 @@ struct btf_ext_info { · 15 unmodified lines
138	168	i < (sec)->num_info; \
139	169	i++, rec = (void *)rec + (seg)->rec_size)
140	170
	171	+/*
	172	+ * The .BTF.ext ELF section layout defined as
	173	+ * struct btf_ext_header
	174	+ * func_info subsection
	175	+ *
	176	+ * The func_info subsection layout:
	177	+ * record size for struct bpf_func_info in the func_info subsection
	178	+ * struct btf_sec_func_info for section #1
	179	+ * a list of bpf_func_info records for section #1
	180	+ * where struct bpf_func_info mimics one in include/uapi/linux/bpf.h
	181	+ * but may not be identical
	182	+ * struct btf_sec_func_info for section #2
	183	+ * a list of bpf_func_info records for section #2
	184	+ * ......
	185	+ *
	186	+ * Note that the bpf_func_info record size in .BTF.ext may not
	187	+ * be the same as the one defined in include/uapi/linux/bpf.h.
	188	+ * The loader should ensure that record_size meets minimum
	189	+ * requirement and pass the record as is to the kernel. The
	190	+ * kernel will handle the func_info properly based on its contents.
	191	+ */
	192	+struct btf_ext_header {
	193	+ __u16 magic;
	194	+ __u8 version;
	195	+ __u8 flags;
	196	+ __u32 hdr_len;
	197	+
	198	+ /* All offsets are in bytes relative to the end of this header */
	199	+ __u32 func_info_off;
	200	+ __u32 func_info_len;
	201	+ __u32 line_info_off;
	202	+ __u32 line_info_len;
	203	+
	204	+ /* optional part of .BTF.ext header */
	205	+ __u32 core_relo_off;
	206	+ __u32 core_relo_len;
	207	+};
	208	+
141	209	struct btf_ext {
142	210	union {
143	211	struct btf_ext_header *hdr;
@@ -145,7 +213,7 @@ struct btf_ext { · 1 unmodified line
145	213	};
146	214	struct btf_ext_info func_info;
147	215	struct btf_ext_info line_info;
148		− struct btf_ext_info field_reloc_info;
	216	+ struct btf_ext_info core_relo_info;
149	217	__u32 data_size;
150	218	};
151	219
@@ -170,32 +238,40 @@ struct bpf_line_info_min { · 18 unmodified lines
170	238	__u32 line_col;
171	239	};
172	240
173		−/* bpf_field_info_kind encodes which aspect of captured field has to be
174		− * adjusted by relocations. Currently supported values are:
175		− * - BPF_FIELD_BYTE_OFFSET: field offset (in bytes);
176		− * - BPF_FIELD_EXISTS: field existence (1, if field exists; 0, otherwise);
	241	+/* bpf_core_relo_kind encodes which aspect of captured field/type/enum value
	242	+ * has to be adjusted by relocations.
177	243	*/
178		−enum bpf_field_info_kind {
	244	+enum bpf_core_relo_kind {
179	245	BPF_FIELD_BYTE_OFFSET = 0, /* field byte offset */
180		− BPF_FIELD_BYTE_SIZE = 1,
	246	+ BPF_FIELD_BYTE_SIZE = 1, /* field size in bytes */
181	247	BPF_FIELD_EXISTS = 2, /* field existence in target kernel */
182		− BPF_FIELD_SIGNED = 3,
183		− BPF_FIELD_LSHIFT_U64 = 4,
184		− BPF_FIELD_RSHIFT_U64 = 5,
	248	+ BPF_FIELD_SIGNED = 3, /* field signedness (0 - unsigned, 1 - signed) */
	249	+ BPF_FIELD_LSHIFT_U64 = 4, /* bitfield-specific left bitshift */
	250	+ BPF_FIELD_RSHIFT_U64 = 5, /* bitfield-specific right bitshift */
	251	+ BPF_TYPE_ID_LOCAL = 6, /* type ID in local BPF object */
	252	+ BPF_TYPE_ID_TARGET = 7, /* type ID in target kernel */
	253	+ BPF_TYPE_EXISTS = 8, /* type existence in target kernel */
	254	+ BPF_TYPE_SIZE = 9, /* type size in bytes */
	255	+ BPF_ENUMVAL_EXISTS = 10, /* enum value existence in target kernel */
	256	+ BPF_ENUMVAL_VALUE = 11, /* enum value integer value */
185	257	};
186	258
187		−/* The minimum bpf_field_reloc checked by the loader
	259	+/* The minimum bpf_core_relo checked by the loader
188	260	*
189		− * Field relocation captures the following data:
	261	+ * CO-RE relocation captures the following data:
190	262	* - insn_off - instruction offset (in bytes) within a BPF program that needs
191	263	* its insn->imm field to be relocated with actual field info;
192	264	* - type_id - BTF type ID of the "root" (containing) entity of a relocatable
193		− * field;
	265	+ * type or field;
194	266	* - access_str_off - offset into corresponding .BTF string section. String
195		− * itself encodes an accessed field using a sequence of field and array
196		− * indicies, separated by colon (:). It's conceptually very close to LLVM's
197		− * getelementptr ([0]) instruction's arguments for identifying offset to
198		− * a field.
	267	+ * interpretation depends on specific relocation kind:
	268	+ * - for field-based relocations, string encodes an accessed field using
	269	+ * a sequence of field and array indices, separated by colon (:). It's
	270	+ * conceptually very close to LLVM's getelementptr ([0]) instruction's
	271	+ * arguments for identifying offset to a field.
	272	+ * - for type-based relocations, strings is expected to be just "0";
	273	+ * - for enum value-based relocations, string contains an index of enum
	274	+ * value within its enum type;
199	275	*
200	276	* Example to provide a better feel.
201	277	*
@@ -226,11 +302,11 @@ enum bpf_field_info_kind { · 24 unmodified lines
226	302	*
227	303	* [0] https://llvm.org/docs/LangRef.html#getelementptr-instruction
228	304	*/
229		−struct bpf_field_reloc {
	305	+struct bpf_core_relo {
230	306	__u32 insn_off;
231	307	__u32 type_id;
232	308	__u32 access_str_off;
233		− enum bpf_field_info_kind kind;
	309	+ enum bpf_core_relo_kind kind;
234	310	};
235	311
236	312	#endif /* __LIBBPF_LIBBPF_INTERNAL_H */

tools/lib/bpf/libbpf_probes.c+3−5

@@ -17,9 +17,6 @@ · 16 unmodified lines
17	17	#include "libbpf.h"
18	18	#include "libbpf_internal.h"
19	19
20		−/* make sure libbpf doesn't use kernel-only integer typedefs */
21		−#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
22		−
23	20	static bool grep(const char buffer, const char pattern)
24	21	{
25	22	return !!strstr(buffer, pattern);
@@ -173,7 +170,7 @@ int libbpf__load_raw_btf(const char *raw_types, size_t types_len, · 147 unmodified lines
173	170	return btf_fd;
174	171	}
175	172
176		−static int load_sk_storage_btf(void)
	173	+static int load_local_storage_btf(void)
177	174	{
178	175	const char strs[] = "\0bpf_spin_lock\0val\0cnt\0l";
179	176	/* struct bpf_spin_lock {
@@ -232,12 +229,13 @@ bool bpf_probe_map_type(enum bpf_map_type map_type, __u32 ifindex) · 52 unmodified lines
232	229	key_size = 0;
233	230	break;
234	231	case BPF_MAP_TYPE_SK_STORAGE:
	232	+ case BPF_MAP_TYPE_INODE_STORAGE:
235	233	btf_key_type_id = 1;
236	234	btf_value_type_id = 3;
237	235	value_size = 8;
238	236	max_entries = 0;
239	237	map_flags = BPF_F_NO_PREALLOC;
240		− btf_fd = load_sk_storage_btf();
	238	+ btf_fd = load_local_storage_btf();
241	239	if (btf_fd < 0)
242	240	return false;
243	241	break;

tools/lib/bpf/netlink.c+6−122

@@ -15,13 +15,12 @@ · 14 unmodified lines
15	15	#include "libbpf_internal.h"
16	16	#include "nlattr.h"
17	17
18		−/* make sure libbpf doesn't use kernel-only integer typedefs */
19		−#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
20		−
21	18	#ifndef SOL_NETLINK
22	19	#define SOL_NETLINK 270
23	20	#endif
24	21
	22	+typedef int (libbpf_dump_nlmsg_t)(void cookie, void msg, struct nlattr *tb);
	23	+
25	24	typedef int (__dump_nlmsg_t)(struct nlmsghdr nlmsg, libbpf_dump_nlmsg_t,
26	25	void *cookie);
27	26
@@ -31,7 +30,7 @@ struct xdp_id_md { · 3 unmodified lines
31	30	struct xdp_link_info info;
32	31	};
33	32
34		−int libbpf_netlink_open(__u32 *nl_pid)
	33	+static int libbpf_netlink_open(__u32 *nl_pid)
35	34	{
36	35	struct sockaddr_nl sa;
37	36	socklen_t addrlen;
@@ -283,6 +282,9 @@ static int get_xdp_info(void cookie, void msg, struct nlattr **tb) · 245 unmodified lines
283	282	return 0;
284	283	}
285	284
	285	+static int libbpf_nl_get_link(int sock, unsigned int nl_pid,
	286	+ libbpf_dump_nlmsg_t dump_link_nlmsg, void *cookie);
	287	+
286	288	int bpf_get_link_xdp_info(int ifindex, struct xdp_link_info *info,
287	289	size_t info_size, __u32 flags)
288	290	{
@@ -368,121 +370,3 @@ int libbpf_nl_get_link(int sock, unsigned int nl_pid, · 79 unmodified lines
368	370	return bpf_netlink_recv(sock, nl_pid, seq, __dump_link_nlmsg,
369	371	dump_link_nlmsg, cookie);
370	372	}
371		−
372		−static int __dump_class_nlmsg(struct nlmsghdr *nlh,
373		− libbpf_dump_nlmsg_t dump_class_nlmsg,
374		− void *cookie)
375		−{
376		− struct nlattr tb[TCA_MAX + 1], attr;
377		− struct tcmsg *t = NLMSG_DATA(nlh);
378		− int len;
379		−
380		− len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*t));
381		− attr = (struct nlattr ) ((void ) t + NLMSG_ALIGN(sizeof(*t)));
382		− if (libbpf_nla_parse(tb, TCA_MAX, attr, len, NULL) != 0)
383		− return -LIBBPF_ERRNO__NLPARSE;
384		−
385		− return dump_class_nlmsg(cookie, t, tb);
386		−}
387		−
388		−int libbpf_nl_get_class(int sock, unsigned int nl_pid, int ifindex,
389		− libbpf_dump_nlmsg_t dump_class_nlmsg, void *cookie)
390		−{
391		− struct {
392		− struct nlmsghdr nlh;
393		− struct tcmsg t;
394		− } req = {
395		− .nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg)),
396		− .nlh.nlmsg_type = RTM_GETTCLASS,
397		− .nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
398		− .t.tcm_family = AF_UNSPEC,
399		− .t.tcm_ifindex = ifindex,
400		− };
401		− int seq = time(NULL);
402		−
403		− req.nlh.nlmsg_seq = seq;
404		− if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
405		− return -errno;
406		−
407		− return bpf_netlink_recv(sock, nl_pid, seq, __dump_class_nlmsg,
408		− dump_class_nlmsg, cookie);
409		−}
410		−
411		−static int __dump_qdisc_nlmsg(struct nlmsghdr *nlh,
412		− libbpf_dump_nlmsg_t dump_qdisc_nlmsg,
413		− void *cookie)
414		−{
415		− struct nlattr tb[TCA_MAX + 1], attr;
416		− struct tcmsg *t = NLMSG_DATA(nlh);
417		− int len;
418		−
419		− len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*t));
420		− attr = (struct nlattr ) ((void ) t + NLMSG_ALIGN(sizeof(*t)));
421		− if (libbpf_nla_parse(tb, TCA_MAX, attr, len, NULL) != 0)
422		− return -LIBBPF_ERRNO__NLPARSE;
423		−
424		− return dump_qdisc_nlmsg(cookie, t, tb);
425		−}
426		−
427		−int libbpf_nl_get_qdisc(int sock, unsigned int nl_pid, int ifindex,
428		− libbpf_dump_nlmsg_t dump_qdisc_nlmsg, void *cookie)
429		−{
430		− struct {
431		− struct nlmsghdr nlh;
432		− struct tcmsg t;
433		− } req = {
434		− .nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg)),
435		− .nlh.nlmsg_type = RTM_GETQDISC,
436		− .nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
437		− .t.tcm_family = AF_UNSPEC,
438		− .t.tcm_ifindex = ifindex,
439		− };
440		− int seq = time(NULL);
441		−
442		− req.nlh.nlmsg_seq = seq;
443		− if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
444		− return -errno;
445		−
446		− return bpf_netlink_recv(sock, nl_pid, seq, __dump_qdisc_nlmsg,
447		− dump_qdisc_nlmsg, cookie);
448		−}
449		−
450		−static int __dump_filter_nlmsg(struct nlmsghdr *nlh,
451		− libbpf_dump_nlmsg_t dump_filter_nlmsg,
452		− void *cookie)
453		−{
454		− struct nlattr tb[TCA_MAX + 1], attr;
455		− struct tcmsg *t = NLMSG_DATA(nlh);
456		− int len;
457		−
458		− len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*t));
459		− attr = (struct nlattr ) ((void ) t + NLMSG_ALIGN(sizeof(*t)));
460		− if (libbpf_nla_parse(tb, TCA_MAX, attr, len, NULL) != 0)
461		− return -LIBBPF_ERRNO__NLPARSE;
462		−
463		− return dump_filter_nlmsg(cookie, t, tb);
464		−}
465		−
466		−int libbpf_nl_get_filter(int sock, unsigned int nl_pid, int ifindex, int handle,
467		− libbpf_dump_nlmsg_t dump_filter_nlmsg, void *cookie)
468		−{
469		− struct {
470		− struct nlmsghdr nlh;
471		− struct tcmsg t;
472		− } req = {
473		− .nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg)),
474		− .nlh.nlmsg_type = RTM_GETTFILTER,
475		− .nlh.nlmsg_flags = NLM_F_DUMP \| NLM_F_REQUEST,
476		− .t.tcm_family = AF_UNSPEC,
477		− .t.tcm_ifindex = ifindex,
478		− .t.tcm_parent = handle,
479		− };
480		− int seq = time(NULL);
481		−
482		− req.nlh.nlmsg_seq = seq;
483		− if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
484		− return -errno;
485		−
486		− return bpf_netlink_recv(sock, nl_pid, seq, __dump_filter_nlmsg,
487		− dump_filter_nlmsg, cookie);
488		−}

tools/lib/bpf/nlattr.c+3−6

@@ -7,14 +7,11 @@ · 6 unmodified lines
7	7	*/
8	8
9	9	#include <errno.h>
10		−#include "nlattr.h"
11		−#include "libbpf_internal.h"
12		−#include <linux/rtnetlink.h>
13	10	#include <string.h>
14	11	#include <stdio.h>
15		−
16		−/* make sure libbpf doesn't use kernel-only integer typedefs */
17		−#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
	12	+#include <linux/rtnetlink.h>
	13	+#include "nlattr.h"
	14	+#include "libbpf_internal.h"
18	15
19	16	static uint16_t nla_attr_minlen[LIBBPF_NLA_TYPE_MAX+1] = {
20	17	[LIBBPF_NLA_U8] = sizeof(uint8_t),

tools/lib/bpf/ringbuf.c+2−6

@@ -16,15 +16,11 @@ · 15 unmodified lines
16	16	#include <asm/barrier.h>
17	17	#include <sys/mman.h>
18	18	#include <sys/epoll.h>
19		−#include <tools/libc_compat.h>
20	19
21	20	#include "libbpf.h"
22	21	#include "libbpf_internal.h"
23	22	#include "bpf.h"
24	23
25		−/* make sure libbpf doesn't use kernel-only integer typedefs */
26		−#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
27		−
28	24	struct ring {
29	25	ring_buffer_sample_fn sample_cb;
30	26	void *ctx;
@@ -82,12 +78,12 @@ int ring_buffer__add(struct ring_buffer *rb, int map_fd, · 51 unmodified lines
82	78	return -EINVAL;
83	79	}
84	80
85		− tmp = reallocarray(rb->rings, rb->ring_cnt + 1, sizeof(*rb->rings));
	81	+ tmp = libbpf_reallocarray(rb->rings, rb->ring_cnt + 1, sizeof(*rb->rings));
86	82	if (!tmp)
87	83	return -ENOMEM;
88	84	rb->rings = tmp;
89	85
90		− tmp = reallocarray(rb->events, rb->ring_cnt + 1, sizeof(*rb->events));
	86	+ tmp = libbpf_reallocarray(rb->events, rb->ring_cnt + 1, sizeof(*rb->events));
91	87	if (!tmp)
92	88	return -ENOMEM;
93	89	rb->events = tmp;

tools/lib/bpf/xsk.c+244−135

@@ -20,6 +20,7 @@ · 19 unmodified lines
20	20	#include <linux/if_ether.h>
21	21	#include <linux/if_packet.h>
22	22	#include <linux/if_xdp.h>
	23	+#include <linux/list.h>
23	24	#include <linux/sockios.h>
24	25	#include <net/if.h>
25	26	#include <sys/ioctl.h>
@@ -32,9 +33,6 @@ · 6 unmodified lines
32	33	#include "libbpf_internal.h"
33	34	#include "xsk.h"
34	35
35		−/* make sure libbpf doesn't use kernel-only integer typedefs */
36		−#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
37		−
38	36	#ifndef SOL_XDP
39	37	#define SOL_XDP 283
40	38	#endif
@@ -48,26 +46,35 @@ · 7 unmodified lines
48	46	#endif
49	47
50	48	struct xsk_umem {
51		− struct xsk_ring_prod *fill;
52		− struct xsk_ring_cons *comp;
	49	+ struct xsk_ring_prod *fill_save;
	50	+ struct xsk_ring_cons *comp_save;
53	51	char *umem_area;
54	52	struct xsk_umem_config config;
55	53	int fd;
56	54	int refcount;
	55	+ struct list_head ctx_list;
	56	+};
	57	+
	58	+struct xsk_ctx {
	59	+ struct xsk_ring_prod *fill;
	60	+ struct xsk_ring_cons *comp;
	61	+ __u32 queue_id;
	62	+ struct xsk_umem *umem;
	63	+ int refcount;
	64	+ int ifindex;
	65	+ struct list_head list;
	66	+ int prog_fd;
	67	+ int xsks_map_fd;
	68	+ char ifname[IFNAMSIZ];
57	69	};
58	70
59	71	struct xsk_socket {
60	72	struct xsk_ring_cons *rx;
61	73	struct xsk_ring_prod *tx;
62	74	__u64 outstanding_tx;
63		− struct xsk_umem *umem;
	75	+ struct xsk_ctx *ctx;
64	76	struct xsk_socket_config config;
65	77	int fd;
66		− int ifindex;
67		− int prog_fd;
68		− int xsks_map_fd;
69		− __u32 queue_id;
70		− char ifname[IFNAMSIZ];
71	78	};
72	79
73	80	struct xsk_nl_info {
@@ -203,15 +210,73 @@ static int xsk_get_mmap_offsets(int fd, struct xdp_mmap_offsets *off) · 129 unmodified lines
203	210	return -EINVAL;
204	211	}
205	212
	213	+static int xsk_create_umem_rings(struct xsk_umem *umem, int fd,
	214	+ struct xsk_ring_prod *fill,
	215	+ struct xsk_ring_cons *comp)
	216	+{
	217	+ struct xdp_mmap_offsets off;
	218	+ void *map;
	219	+ int err;
	220	+
	221	+ err = setsockopt(fd, SOL_XDP, XDP_UMEM_FILL_RING,
	222	+ &umem->config.fill_size,
	223	+ sizeof(umem->config.fill_size));
	224	+ if (err)
	225	+ return -errno;
	226	+
	227	+ err = setsockopt(fd, SOL_XDP, XDP_UMEM_COMPLETION_RING,
	228	+ &umem->config.comp_size,
	229	+ sizeof(umem->config.comp_size));
	230	+ if (err)
	231	+ return -errno;
	232	+
	233	+ err = xsk_get_mmap_offsets(fd, &off);
	234	+ if (err)
	235	+ return -errno;
	236	+
	237	+ map = mmap(NULL, off.fr.desc + umem->config.fill_size * sizeof(__u64),
	238	+ PROT_READ \| PROT_WRITE, MAP_SHARED \| MAP_POPULATE, fd,
	239	+ XDP_UMEM_PGOFF_FILL_RING);
	240	+ if (map == MAP_FAILED)
	241	+ return -errno;
	242	+
	243	+ fill->mask = umem->config.fill_size - 1;
	244	+ fill->size = umem->config.fill_size;
	245	+ fill->producer = map + off.fr.producer;
	246	+ fill->consumer = map + off.fr.consumer;
	247	+ fill->flags = map + off.fr.flags;
	248	+ fill->ring = map + off.fr.desc;
	249	+ fill->cached_cons = umem->config.fill_size;
	250	+
	251	+ map = mmap(NULL, off.cr.desc + umem->config.comp_size * sizeof(__u64),
	252	+ PROT_READ \| PROT_WRITE, MAP_SHARED \| MAP_POPULATE, fd,
	253	+ XDP_UMEM_PGOFF_COMPLETION_RING);
	254	+ if (map == MAP_FAILED) {
	255	+ err = -errno;
	256	+ goto out_mmap;
	257	+ }
	258	+
	259	+ comp->mask = umem->config.comp_size - 1;
	260	+ comp->size = umem->config.comp_size;
	261	+ comp->producer = map + off.cr.producer;
	262	+ comp->consumer = map + off.cr.consumer;
	263	+ comp->flags = map + off.cr.flags;
	264	+ comp->ring = map + off.cr.desc;
	265	+
	266	+ return 0;
	267	+
	268	+out_mmap:
	269	+ munmap(map, off.fr.desc + umem->config.fill_size * sizeof(__u64));
	270	+ return err;
	271	+}
	272	+
206	273	int xsk_umem__create_v0_0_4(struct xsk_umem *umem_ptr, void umem_area,
207	274	__u64 size, struct xsk_ring_prod *fill,
208	275	struct xsk_ring_cons *comp,
209	276	const struct xsk_umem_config *usr_config)
210	277	{
211		− struct xdp_mmap_offsets off;
212	278	struct xdp_umem_reg mr;
213	279	struct xsk_umem *umem;
214		− void *map;
215	280	int err;
216	281
217	282	if (!umem_area \|\| !umem_ptr \|\| !fill \|\| !comp)
@@ -230,6 +295,7 @@ int xsk_umem__create_v0_0_4(struct xsk_umem *umem_ptr, void umem_area, · 12 unmodified lines
230	295	}
231	296
232	297	umem->umem_area = umem_area;
	298	+ INIT_LIST_HEAD(&umem->ctx_list);
233	299	xsk_set_umem_config(&umem->config, usr_config);
234	300
235	301	memset(&mr, 0, sizeof(mr));
@@ -244,71 +310,16 @@ int xsk_umem__create_v0_0_4(struct xsk_umem *umem_ptr, void umem_area, · 8 unmodified lines
244	310	err = -errno;
245	311	goto out_socket;
246	312	}
247		− err = setsockopt(umem->fd, SOL_XDP, XDP_UMEM_FILL_RING,
248		− &umem->config.fill_size,
249		− sizeof(umem->config.fill_size));
250		− if (err) {
251		− err = -errno;
252		− goto out_socket;
253		− }
254		− err = setsockopt(umem->fd, SOL_XDP, XDP_UMEM_COMPLETION_RING,
255		− &umem->config.comp_size,
256		− sizeof(umem->config.comp_size));
257		− if (err) {
258		− err = -errno;
259		− goto out_socket;
260		− }
261	313
262		− err = xsk_get_mmap_offsets(umem->fd, &off);
263		− if (err) {
264		− err = -errno;
265		− goto out_socket;
266		− }
267		−
268		− map = mmap(NULL, off.fr.desc + umem->config.fill_size * sizeof(__u64),
269		− PROT_READ \| PROT_WRITE, MAP_SHARED \| MAP_POPULATE, umem->fd,
270		− XDP_UMEM_PGOFF_FILL_RING);
271		− if (map == MAP_FAILED) {
272		− err = -errno;
	314	+ err = xsk_create_umem_rings(umem, umem->fd, fill, comp);
	315	+ if (err)
273	316	goto out_socket;
274		− }
275		−
276		− umem->fill = fill;
277		− fill->mask = umem->config.fill_size - 1;
278		− fill->size = umem->config.fill_size;
279		− fill->producer = map + off.fr.producer;
280		− fill->consumer = map + off.fr.consumer;
281		− fill->flags = map + off.fr.flags;
282		− fill->ring = map + off.fr.desc;
283		− fill->cached_prod = *fill->producer;
284		− /* cached_cons is "size" bigger than the real consumer pointer
285		− * See xsk_prod_nb_free
286		− */
287		− fill->cached_cons = *fill->consumer + umem->config.fill_size;
288		−
289		− map = mmap(NULL, off.cr.desc + umem->config.comp_size * sizeof(__u64),
290		− PROT_READ \| PROT_WRITE, MAP_SHARED \| MAP_POPULATE, umem->fd,
291		− XDP_UMEM_PGOFF_COMPLETION_RING);
292		− if (map == MAP_FAILED) {
293		− err = -errno;
294		− goto out_mmap;
295		− }
296		−
297		− umem->comp = comp;
298		− comp->mask = umem->config.comp_size - 1;
299		− comp->size = umem->config.comp_size;
300		− comp->producer = map + off.cr.producer;
301		− comp->consumer = map + off.cr.consumer;
302		− comp->flags = map + off.cr.flags;
303		− comp->ring = map + off.cr.desc;
304		− comp->cached_prod = *comp->producer;
305		− comp->cached_cons = *comp->consumer;
306	317
	318	+ umem->fill_save = fill;
	319	+ umem->comp_save = comp;
307	320	*umem_ptr = umem;
308	321	return 0;
309	322
310		−out_mmap:
311		− munmap(map, off.fr.desc + umem->config.fill_size * sizeof(__u64));
312	323	out_socket:
313	324	close(umem->fd);
314	325	out_umem_alloc:
@@ -342,6 +353,7 @@ DEFAULT_VERSION(xsk_umem__create_v0_0_4, xsk_umem__create, LIBBPF_0.0.4) · 27 unmodified lines
342	353	static int xsk_load_xdp_prog(struct xsk_socket *xsk)
343	354	{
344	355	static const int log_buf_size = 16 * 1024;
	356	+ struct xsk_ctx *ctx = xsk->ctx;
345	357	char log_buf[log_buf_size];
346	358	int err, prog_fd;
347	359
@@ -369,7 +381,7 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk) · 21 unmodified lines
369	381	/* (u32 )(r10 - 4) = r2 */
370	382	BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_2, -4),
371	383	/* r1 = xskmap[] */
372		− BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),
	384	+ BPF_LD_MAP_FD(BPF_REG_1, ctx->xsks_map_fd),
373	385	/* r3 = XDP_PASS */
374	386	BPF_MOV64_IMM(BPF_REG_3, 2),
375	387	/* call bpf_redirect_map */
@@ -381,7 +393,7 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk) · 5 unmodified lines
381	393	/* r2 += -4 */
382	394	BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
383	395	/* r1 = xskmap[] */
384		− BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),
	396	+ BPF_LD_MAP_FD(BPF_REG_1, ctx->xsks_map_fd),
385	397	/* call bpf_map_lookup_elem */
386	398	BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),
387	399	/* r1 = r0 */
@@ -393,7 +405,7 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk) · 5 unmodified lines
393	405	/* r2 = (u32 )(r10 - 4) */
394	406	BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_10, -4),
395	407	/* r1 = xskmap[] */
396		− BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),
	408	+ BPF_LD_MAP_FD(BPF_REG_1, ctx->xsks_map_fd),
397	409	/* r3 = 0 */
398	410	BPF_MOV64_IMM(BPF_REG_3, 0),
399	411	/* call bpf_redirect_map */
@@ -411,19 +423,21 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk) · 11 unmodified lines
411	423	return prog_fd;
412	424	}
413	425
414		− err = bpf_set_link_xdp_fd(xsk->ifindex, prog_fd, xsk->config.xdp_flags);
	426	+ err = bpf_set_link_xdp_fd(xsk->ctx->ifindex, prog_fd,
	427	+ xsk->config.xdp_flags);
415	428	if (err) {
416	429	close(prog_fd);
417	430	return err;
418	431	}
419	432
420		− xsk->prog_fd = prog_fd;
	433	+ ctx->prog_fd = prog_fd;
421	434	return 0;
422	435	}
423	436
424	437	static int xsk_get_max_queues(struct xsk_socket *xsk)
425	438	{
426	439	struct ethtool_channels channels = { .cmd = ETHTOOL_GCHANNELS };
	440	+ struct xsk_ctx *ctx = xsk->ctx;
427	441	struct ifreq ifr = {};
428	442	int fd, err, ret;
429	443
@@ -432,7 +446,7 @@ static int xsk_get_max_queues(struct xsk_socket *xsk) · 2 unmodified lines
432	446	return -errno;
433	447
434	448	ifr.ifr_data = (void *)&channels;
435		− memcpy(ifr.ifr_name, xsk->ifname, IFNAMSIZ - 1);
	449	+ memcpy(ifr.ifr_name, ctx->ifname, IFNAMSIZ - 1);
436	450	ifr.ifr_name[IFNAMSIZ - 1] = '\0';
437	451	err = ioctl(fd, SIOCETHTOOL, &ifr);
438	452	if (err && errno != EOPNOTSUPP) {
@@ -460,6 +474,7 @@ out: · 21 unmodified lines
460	474
461	475	static int xsk_create_bpf_maps(struct xsk_socket *xsk)
462	476	{
	477	+ struct xsk_ctx *ctx = xsk->ctx;
463	478	int max_queues;
464	479	int fd;
465	480
@@ -472,15 +487,17 @@ static int xsk_create_bpf_maps(struct xsk_socket *xsk) · 6 unmodified lines
472	487	if (fd < 0)
473	488	return fd;
474	489
475		− xsk->xsks_map_fd = fd;
	490	+ ctx->xsks_map_fd = fd;
476	491
477	492	return 0;
478	493	}
479	494
480	495	static void xsk_delete_bpf_maps(struct xsk_socket *xsk)
481	496	{
482		− bpf_map_delete_elem(xsk->xsks_map_fd, &xsk->queue_id);
483		− close(xsk->xsks_map_fd);
	497	+ struct xsk_ctx *ctx = xsk->ctx;
	498	+
	499	+ bpf_map_delete_elem(ctx->xsks_map_fd, &ctx->queue_id);
	500	+ close(ctx->xsks_map_fd);
484	501	}
485	502
486	503	static int xsk_lookup_bpf_maps(struct xsk_socket *xsk)
@@ -488,10 +505,11 @@ static int xsk_lookup_bpf_maps(struct xsk_socket *xsk) · 1 unmodified line
488	505	__u32 i, *map_ids, num_maps, prog_len = sizeof(struct bpf_prog_info);
489	506	__u32 map_len = sizeof(struct bpf_map_info);
490	507	struct bpf_prog_info prog_info = {};
	508	+ struct xsk_ctx *ctx = xsk->ctx;
491	509	struct bpf_map_info map_info;
492	510	int fd, err;
493	511
494		− err = bpf_obj_get_info_by_fd(xsk->prog_fd, &prog_info, &prog_len);
	512	+ err = bpf_obj_get_info_by_fd(ctx->prog_fd, &prog_info, &prog_len);
495	513	if (err)
496	514	return err;
497	515
@@ -505,11 +523,11 @@ static int xsk_lookup_bpf_maps(struct xsk_socket *xsk) · 7 unmodified lines
505	523	prog_info.nr_map_ids = num_maps;
506	524	prog_info.map_ids = (__u64)(unsigned long)map_ids;
507	525
508		− err = bpf_obj_get_info_by_fd(xsk->prog_fd, &prog_info, &prog_len);
	526	+ err = bpf_obj_get_info_by_fd(ctx->prog_fd, &prog_info, &prog_len);
509	527	if (err)
510	528	goto out_map_ids;
511	529
512		− xsk->xsks_map_fd = -1;
	530	+ ctx->xsks_map_fd = -1;
513	531
514	532	for (i = 0; i < prog_info.nr_map_ids; i++) {
515	533	fd = bpf_map_get_fd_by_id(map_ids[i]);
@@ -523,7 +541,7 @@ static int xsk_lookup_bpf_maps(struct xsk_socket *xsk) · 7 unmodified lines
523	541	}
524	542
525	543	if (!strcmp(map_info.name, "xsks_map")) {
526		− xsk->xsks_map_fd = fd;
	544	+ ctx->xsks_map_fd = fd;
527	545	continue;
528	546	}
529	547
@@ -531,7 +549,7 @@ static int xsk_lookup_bpf_maps(struct xsk_socket *xsk) · 1 unmodified line
531	549	}
532	550
533	551	err = 0;
534		− if (xsk->xsks_map_fd == -1)
	552	+ if (ctx->xsks_map_fd == -1)
535	553	err = -ENOENT;
536	554
537	555	out_map_ids:
@@ -541,16 +559,19 @@ out_map_ids: · 3 unmodified lines
541	559
542	560	static int xsk_set_bpf_maps(struct xsk_socket *xsk)
543	561	{
544		− return bpf_map_update_elem(xsk->xsks_map_fd, &xsk->queue_id,
	562	+ struct xsk_ctx *ctx = xsk->ctx;
	563	+
	564	+ return bpf_map_update_elem(ctx->xsks_map_fd, &ctx->queue_id,
545	565	&xsk->fd, 0);
546	566	}
547	567
548	568	static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
549	569	{
	570	+ struct xsk_ctx *ctx = xsk->ctx;
550	571	__u32 prog_id = 0;
551	572	int err;
552	573
553		− err = bpf_get_link_xdp_id(xsk->ifindex, &prog_id,
	574	+ err = bpf_get_link_xdp_id(ctx->ifindex, &prog_id,
554	575	xsk->config.xdp_flags);
555	576	if (err)
556	577	return err;
@@ -566,12 +587,12 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk) · 9 unmodified lines
566	587	return err;
567	588	}
568	589	} else {
569		− xsk->prog_fd = bpf_prog_get_fd_by_id(prog_id);
570		− if (xsk->prog_fd < 0)
	590	+ ctx->prog_fd = bpf_prog_get_fd_by_id(prog_id);
	591	+ if (ctx->prog_fd < 0)
571	592	return -errno;
572	593	err = xsk_lookup_bpf_maps(xsk);
573	594	if (err) {
574		− close(xsk->prog_fd);
	595	+ close(ctx->prog_fd);
575	596	return err;
576	597	}
577	598	}
@@ -580,25 +601,110 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk) · 2 unmodified lines
580	601	err = xsk_set_bpf_maps(xsk);
581	602	if (err) {
582	603	xsk_delete_bpf_maps(xsk);
583		− close(xsk->prog_fd);
	604	+ close(ctx->prog_fd);
584	605	return err;
585	606	}
586	607
587	608	return 0;
588	609	}
589	610
590		−int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname,
591		− __u32 queue_id, struct xsk_umem *umem,
592		− struct xsk_ring_cons rx, struct xsk_ring_prod tx,
593		− const struct xsk_socket_config *usr_config)
	611	+static struct xsk_ctx xsk_get_ctx(struct xsk_umem umem, int ifindex,
	612	+ __u32 queue_id)
	613	+{
	614	+ struct xsk_ctx *ctx;
	615	+
	616	+ if (list_empty(&umem->ctx_list))
	617	+ return NULL;
	618	+
	619	+ list_for_each_entry(ctx, &umem->ctx_list, list) {
	620	+ if (ctx->ifindex == ifindex && ctx->queue_id == queue_id) {
	621	+ ctx->refcount++;
	622	+ return ctx;
	623	+ }
	624	+ }
	625	+
	626	+ return NULL;
	627	+}
	628	+
	629	+static void xsk_put_ctx(struct xsk_ctx *ctx)
	630	+{
	631	+ struct xsk_umem *umem = ctx->umem;
	632	+ struct xdp_mmap_offsets off;
	633	+ int err;
	634	+
	635	+ if (--ctx->refcount == 0) {
	636	+ err = xsk_get_mmap_offsets(umem->fd, &off);
	637	+ if (!err) {
	638	+ munmap(ctx->fill->ring - off.fr.desc,
	639	+ off.fr.desc + umem->config.fill_size *
	640	+ sizeof(__u64));
	641	+ munmap(ctx->comp->ring - off.cr.desc,
	642	+ off.cr.desc + umem->config.comp_size *
	643	+ sizeof(__u64));
	644	+ }
	645	+
	646	+ list_del(&ctx->list);
	647	+ free(ctx);
	648	+ }
	649	+}
	650	+
	651	+static struct xsk_ctx xsk_create_ctx(struct xsk_socket xsk,
	652	+ struct xsk_umem *umem, int ifindex,
	653	+ const char *ifname, __u32 queue_id,
	654	+ struct xsk_ring_prod *fill,
	655	+ struct xsk_ring_cons *comp)
	656	+{
	657	+ struct xsk_ctx *ctx;
	658	+ int err;
	659	+
	660	+ ctx = calloc(1, sizeof(*ctx));
	661	+ if (!ctx)
	662	+ return NULL;
	663	+
	664	+ if (!umem->fill_save) {
	665	+ err = xsk_create_umem_rings(umem, xsk->fd, fill, comp);
	666	+ if (err) {
	667	+ free(ctx);
	668	+ return NULL;
	669	+ }
	670	+ } else if (umem->fill_save != fill \|\| umem->comp_save != comp) {
	671	+ /* Copy over rings to new structs. */
	672	+ memcpy(fill, umem->fill_save, sizeof(*fill));
	673	+ memcpy(comp, umem->comp_save, sizeof(*comp));
	674	+ }
	675	+
	676	+ ctx->ifindex = ifindex;
	677	+ ctx->refcount = 1;
	678	+ ctx->umem = umem;
	679	+ ctx->queue_id = queue_id;
	680	+ memcpy(ctx->ifname, ifname, IFNAMSIZ - 1);
	681	+ ctx->ifname[IFNAMSIZ - 1] = '\0';
	682	+
	683	+ umem->fill_save = NULL;
	684	+ umem->comp_save = NULL;
	685	+ ctx->fill = fill;
	686	+ ctx->comp = comp;
	687	+ list_add(&ctx->list, &umem->ctx_list);
	688	+ return ctx;
	689	+}
	690	+
	691	+int xsk_socket__create_shared(struct xsk_socket **xsk_ptr,
	692	+ const char *ifname,
	693	+ __u32 queue_id, struct xsk_umem *umem,
	694	+ struct xsk_ring_cons *rx,
	695	+ struct xsk_ring_prod *tx,
	696	+ struct xsk_ring_prod *fill,
	697	+ struct xsk_ring_cons *comp,
	698	+ const struct xsk_socket_config *usr_config)
594	699	{
595	700	void rx_map = NULL, tx_map = NULL;
596	701	struct sockaddr_xdp sxdp = {};
597	702	struct xdp_mmap_offsets off;
598	703	struct xsk_socket *xsk;
599		− int err;
	704	+ struct xsk_ctx *ctx;
	705	+ int err, ifindex;
600	706
601		− if (!umem \|\| !xsk_ptr \|\| !(rx \|\| tx))
	707	+ if (!umem \|\| !xsk_ptr \|\| !(rx \|\| tx) \|\| !fill \|\| !comp)
602	708	return -EFAULT;
603	709
604	710	xsk = calloc(1, sizeof(*xsk));
@@ -609,10 +715,10 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 4 unmodified lines
609	715	if (err)
610	716	goto out_xsk_alloc;
611	717
612		− if (umem->refcount &&
613		− !(xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)) {
614		− pr_warn("Error: shared umems not supported by libbpf supplied XDP program.\n");
615		− err = -EBUSY;
	718	+ xsk->outstanding_tx = 0;
	719	+ ifindex = if_nametoindex(ifname);
	720	+ if (!ifindex) {
	721	+ err = -errno;
616	722	goto out_xsk_alloc;
617	723	}
618	724
@@ -626,16 +732,16 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 7 unmodified lines
626	732	xsk->fd = umem->fd;
627	733	}
628	734
629		− xsk->outstanding_tx = 0;
630		− xsk->queue_id = queue_id;
631		− xsk->umem = umem;
632		− xsk->ifindex = if_nametoindex(ifname);
633		− if (!xsk->ifindex) {
634		− err = -errno;
635		− goto out_socket;
	735	+ ctx = xsk_get_ctx(umem, ifindex, queue_id);
	736	+ if (!ctx) {
	737	+ ctx = xsk_create_ctx(xsk, umem, ifindex, ifname, queue_id,
	738	+ fill, comp);
	739	+ if (!ctx) {
	740	+ err = -ENOMEM;
	741	+ goto out_socket;
	742	+ }
636	743	}
637		− memcpy(xsk->ifname, ifname, IFNAMSIZ - 1);
638		− xsk->ifname[IFNAMSIZ - 1] = '\0';
	744	+ xsk->ctx = ctx;
639	745
640	746	if (rx) {
641	747	err = setsockopt(xsk->fd, SOL_XDP, XDP_RX_RING,
@@ -643,7 +749,7 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 1 unmodified line
643	749	sizeof(xsk->config.rx_size));
644	750	if (err) {
645	751	err = -errno;
646		− goto out_socket;
	752	+ goto out_put_ctx;
647	753	}
648	754	}
649	755	if (tx) {
@@ -652,14 +758,14 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 2 unmodified lines
652	758	sizeof(xsk->config.tx_size));
653	759	if (err) {
654	760	err = -errno;
655		− goto out_socket;
	761	+ goto out_put_ctx;
656	762	}
657	763	}
658	764
659	765	err = xsk_get_mmap_offsets(xsk->fd, &off);
660	766	if (err) {
661	767	err = -errno;
662		− goto out_socket;
	768	+ goto out_put_ctx;
663	769	}
664	770
665	771	if (rx) {
@@ -669,7 +775,7 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 3 unmodified lines
669	775	xsk->fd, XDP_PGOFF_RX_RING);
670	776	if (rx_map == MAP_FAILED) {
671	777	err = -errno;
672		− goto out_socket;
	778	+ goto out_put_ctx;
673	779	}
674	780
675	781	rx->mask = xsk->config.rx_size - 1;
@@ -708,10 +814,10 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 32 unmodified lines
708	814	xsk->tx = tx;
709	815
710	816	sxdp.sxdp_family = PF_XDP;
711		− sxdp.sxdp_ifindex = xsk->ifindex;
712		− sxdp.sxdp_queue_id = xsk->queue_id;
	817	+ sxdp.sxdp_ifindex = ctx->ifindex;
	818	+ sxdp.sxdp_queue_id = ctx->queue_id;
713	819	if (umem->refcount > 1) {
714		− sxdp.sxdp_flags = XDP_SHARED_UMEM;
	820	+ sxdp.sxdp_flags \|= XDP_SHARED_UMEM;
715	821	sxdp.sxdp_shared_umem_fd = umem->fd;
716	822	} else {
717	823	sxdp.sxdp_flags = xsk->config.bind_flags;
@@ -723,7 +829,7 @@ int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname, · 5 unmodified lines
723	829	goto out_mmap_tx;
724	830	}
725	831
726		− xsk->prog_fd = -1;
	832	+ ctx->prog_fd = -1;
727	833
728	834	if (!(xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)) {
729	835	err = xsk_setup_xdp_prog(xsk);
@@ -742,6 +848,8 @@ out_mmap_rx: · 12 unmodified lines
742	848	if (rx)
743	849	munmap(rx_map, off.rx.desc +
744	850	xsk->config.rx_size * sizeof(struct xdp_desc));
	851	+out_put_ctx:
	852	+ xsk_put_ctx(ctx);
745	853	out_socket:
746	854	if (--umem->refcount)
747	855	close(xsk->fd);
@@ -750,25 +858,24 @@ out_xsk_alloc: · 2 unmodified lines
750	858	return err;
751	859	}
752	860
753		−int xsk_umem__delete(struct xsk_umem *umem)
	861	+int xsk_socket__create(struct xsk_socket *xsk_ptr, const char ifname,
	862	+ __u32 queue_id, struct xsk_umem *umem,
	863	+ struct xsk_ring_cons rx, struct xsk_ring_prod tx,
	864	+ const struct xsk_socket_config *usr_config)
754	865	{
755		− struct xdp_mmap_offsets off;
756		− int err;
	866	+ return xsk_socket__create_shared(xsk_ptr, ifname, queue_id, umem,
	867	+ rx, tx, umem->fill_save,
	868	+ umem->comp_save, usr_config);
	869	+}
757	870
	871	+int xsk_umem__delete(struct xsk_umem *umem)
	872	+{
758	873	if (!umem)
759	874	return 0;
760	875
761	876	if (umem->refcount)
762	877	return -EBUSY;
763	878
764		− err = xsk_get_mmap_offsets(umem->fd, &off);
765		− if (!err) {
766		− munmap(umem->fill->ring - off.fr.desc,
767		− off.fr.desc + umem->config.fill_size * sizeof(__u64));
768		− munmap(umem->comp->ring - off.cr.desc,
769		− off.cr.desc + umem->config.comp_size * sizeof(__u64));
770		− }
771		−
772	879	close(umem->fd);
773	880	free(umem);
774	881
@@ -778,15 +885,16 @@ int xsk_umem__delete(struct xsk_umem *umem) · 3 unmodified lines
778	885	void xsk_socket__delete(struct xsk_socket *xsk)
779	886	{
780	887	size_t desc_sz = sizeof(struct xdp_desc);
	888	+ struct xsk_ctx *ctx = xsk->ctx;
781	889	struct xdp_mmap_offsets off;
782	890	int err;
783	891
784	892	if (!xsk)
785	893	return;
786	894
787		− if (xsk->prog_fd != -1) {
	895	+ if (ctx->prog_fd != -1) {
788	896	xsk_delete_bpf_maps(xsk);
789		− close(xsk->prog_fd);
	897	+ close(ctx->prog_fd);
790	898	}
791	899
792	900	err = xsk_get_mmap_offsets(xsk->fd, &off);
@@ -799,14 +907,15 @@ void xsk_socket__delete(struct xsk_socket *xsk) · 6 unmodified lines
799	907	munmap(xsk->tx->ring - off.tx.desc,
800	908	off.tx.desc + xsk->config.tx_size * desc_sz);
801	909	}
802		−
803	910	}
804	911
805		− xsk->umem->refcount--;
	912	+ xsk_put_ctx(ctx);
	913	+
	914	+ ctx->umem->refcount--;
806	915	/* Do not close an fd that also has an associated umem connected
807	916	* to it.
808	917	*/
809		− if (xsk->fd != xsk->umem->fd)
	918	+ if (xsk->fd != ctx->umem->fd)
810	919	close(xsk->fd);
811	920	free(xsk);
812	921	}

tools/lib/bpf/xsk.h+9−0

@@ -234,6 +234,15 @@ LIBBPF_API int xsk_socket__create(struct xsk_socket **xsk, · 233 unmodified lines
234	234	struct xsk_ring_cons *rx,
235	235	struct xsk_ring_prod *tx,
236	236	const struct xsk_socket_config *config);
	237	+LIBBPF_API int
	238	+xsk_socket__create_shared(struct xsk_socket **xsk_ptr,
	239	+ const char *ifname,
	240	+ __u32 queue_id, struct xsk_umem *umem,
	241	+ struct xsk_ring_cons *rx,
	242	+ struct xsk_ring_prod *tx,
	243	+ struct xsk_ring_prod *fill,
	244	+ struct xsk_ring_cons *comp,
	245	+ const struct xsk_socket_config *config);
237	246
238	247	/* Returns 0 for success and -EBUSY if the umem is still in use. */
239	248	LIBBPF_API int xsk_umem__delete(struct xsk_umem *umem);

tools/perf/Makefile.config+0−4

@@ -483,10 +483,6 @@ ifndef NO_LIBELF · 482 unmodified lines
483	483	EXTLIBS += -lelf
484	484	$(call detected,CONFIG_LIBELF)
485	485
486		− ifeq ($(feature-libelf-mmap), 1)
487		− CFLAGS += -DHAVE_LIBELF_MMAP_SUPPORT
488		− endif
489		−
490	486	ifeq ($(feature-libelf-getphdrnum), 1)
491	487	CFLAGS += -DHAVE_ELF_GETPHDRNUM_SUPPORT
492	488	endif

tools/perf/util/symbol.h+1−1

@@ -28,7 +28,7 @@ struct option; · 27 unmodified lines
28	28	* libelf 0.8.x and earlier do not support ELF_C_READ_MMAP;
29	29	* for newer versions we can use mmap to reduce memory usage:
30	30	*/
31		−#ifdef HAVE_LIBELF_MMAP_SUPPORT
	31	+#ifdef ELF_C_READ_MMAP
32	32	# define PERF_ELF_C_READ_MMAP ELF_C_READ_MMAP
33	33	#else
34	34	# define PERF_ELF_C_READ_MMAP ELF_C_READ

tools/testing/selftests/bpf/Makefile+1−1

@@ -316,7 +316,7 @@ $(TRUNNER_BPF_OBJS): $(TRUNNER_OUTPUT)/%.o: \ · 315 unmodified lines
316	316	$(TRUNNER_BPF_PROGS_DIR)/%.c \
317	317	$(TRUNNER_BPF_PROGS_DIR)/*.h \
318	318	$$(INCLUDE_DIR)/vmlinux.h \
319		− $$(BPFOBJ) \| $(TRUNNER_OUTPUT)
	319	+ $(wildcard $(BPFDIR)/bpf_*.h) \| $(TRUNNER_OUTPUT)
320	320	$$(call $(TRUNNER_BPF_BUILD_RULE),$$<,$$@, \
321	321	$(TRUNNER_BPF_CFLAGS), \
322	322	$(TRUNNER_BPF_LDFLAGS))

tools/testing/selftests/bpf/README.rst+21−0

@@ -43,3 +43,24 @@ This is due to a llvm BPF backend bug. The fix · 42 unmodified lines
43	43	https://reviews.llvm.org/D78466
44	44	has been pushed to llvm 10.x release branch and will be
45	45	available in 10.0.1. The fix is available in llvm 11.0.0 trunk.
	46	+
	47	+BPF CO-RE-based tests and Clang version
	48	+=======================================
	49	+
	50	+A set of selftests use BPF target-specific built-ins, which might require
	51	+bleeding-edge Clang versions (Clang 12 nightly at this time).
	52	+
	53	+Few sub-tests of core_reloc test suit (part of test_progs test runner) require
	54	+the following built-ins, listed with corresponding Clang diffs introducing
	55	+them to Clang/LLVM. These sub-tests are going to be skipped if Clang is too
	56	+old to support them, they shouldn't cause build failures or runtime test
	57	+failures:
	58	+
	59	+ - __builtin_btf_type_id() ([0], [1], [2]);
	60	+ - __builtin_preserve_type_info(), __builtin_preserve_enum_value() ([3], [4]).
	61	+
	62	+ [0] https://reviews.llvm.org/D74572
	63	+ [1] https://reviews.llvm.org/D74668
	64	+ [2] https://reviews.llvm.org/D85174
	65	+ [3] https://reviews.llvm.org/D83878
	66	+ [4] https://reviews.llvm.org/D83242

tools/testing/selftests/bpf/bench.c+2−0

@@ -317,6 +317,7 @@ extern const struct bench bench_trig_tp; · 316 unmodified lines
317	317	extern const struct bench bench_trig_rawtp;
318	318	extern const struct bench bench_trig_kprobe;
319	319	extern const struct bench bench_trig_fentry;
	320	+extern const struct bench bench_trig_fentry_sleep;
320	321	extern const struct bench bench_trig_fmodret;
321	322	extern const struct bench bench_rb_libbpf;
322	323	extern const struct bench bench_rb_custom;
@@ -338,6 +339,7 @@ static const struct bench *benchs[] = { · 15 unmodified lines
338	339	&bench_trig_rawtp,
339	340	&bench_trig_kprobe,
340	341	&bench_trig_fentry,
	342	+ &bench_trig_fentry_sleep,
341	343	&bench_trig_fmodret,
342	344	&bench_rb_libbpf,
343	345	&bench_rb_custom,

tools/testing/selftests/bpf/benchs/bench_trigger.c+17−0

@@ -90,6 +90,12 @@ static void trigger_fentry_setup() · 89 unmodified lines
90	90	attach_bpf(ctx.skel->progs.bench_trigger_fentry);
91	91	}
92	92
	93	+static void trigger_fentry_sleep_setup()
	94	+{
	95	+ setup_ctx();
	96	+ attach_bpf(ctx.skel->progs.bench_trigger_fentry_sleep);
	97	+}
	98	+
93	99	static void trigger_fmodret_setup()
94	100	{
95	101	setup_ctx();
@@ -155,6 +161,17 @@ const struct bench bench_trig_fentry = { · 59 unmodified lines
155	161	.report_final = hits_drops_report_final,
156	162	};
157	163
	164	+const struct bench bench_trig_fentry_sleep = {
	165	+ .name = "trig-fentry-sleep",
	166	+ .validate = trigger_validate,
	167	+ .setup = trigger_fentry_sleep_setup,
	168	+ .producer_thread = trigger_producer,
	169	+ .consumer_thread = trigger_consumer,
	170	+ .measure = trigger_measure,
	171	+ .report_progress = hits_drops_report_progress,
	172	+ .report_final = hits_drops_report_final,
	173	+};
	174	+
158	175	const struct bench bench_trig_fmodret = {
159	176	.name = "trig-fmodret",
160	177	.validate = trigger_validate,

tools/testing/selftests/bpf/network_helpers.c+37−0

@@ -104,6 +104,43 @@ error_close: · 103 unmodified lines
104	104	return -1;
105	105	}
106	106
	107	+int fastopen_connect(int server_fd, const char *data, unsigned int data_len,
	108	+ int timeout_ms)
	109	+{
	110	+ struct sockaddr_storage addr;
	111	+ socklen_t addrlen = sizeof(addr);
	112	+ struct sockaddr_in *addr_in;
	113	+ int fd, ret;
	114	+
	115	+ if (getsockname(server_fd, (struct sockaddr *)&addr, &addrlen)) {
	116	+ log_err("Failed to get server addr");
	117	+ return -1;
	118	+ }
	119	+
	120	+ addr_in = (struct sockaddr_in *)&addr;
	121	+ fd = socket(addr_in->sin_family, SOCK_STREAM, 0);
	122	+ if (fd < 0) {
	123	+ log_err("Failed to create client socket");
	124	+ return -1;
	125	+ }
	126	+
	127	+ if (settimeo(fd, timeout_ms))
	128	+ goto error_close;
	129	+
	130	+ ret = sendto(fd, data, data_len, MSG_FASTOPEN, (struct sockaddr *)&addr,
	131	+ addrlen);
	132	+ if (ret != data_len) {
	133	+ log_err("sendto(data, %u) != %d\n", data_len, ret);
	134	+ goto error_close;
	135	+ }
	136	+
	137	+ return fd;
	138	+
	139	+error_close:
	140	+ save_errno_close(fd);
	141	+ return -1;
	142	+}
	143	+
107	144	static int connect_fd_to_addr(int fd,
108	145	const struct sockaddr_storage *addr,
109	146	socklen_t addrlen)

tools/testing/selftests/bpf/network_helpers.h+2−0

@@ -37,6 +37,8 @@ int start_server(int family, int type, const char *addr, __u16 port, · 36 unmodified lines
37	37	int timeout_ms);
38	38	int connect_to_fd(int server_fd, int timeout_ms);
39	39	int connect_fd_to_fd(int client_fd, int server_fd, int timeout_ms);
	40	+int fastopen_connect(int server_fd, const char *data, unsigned int data_len,
	41	+ int timeout_ms);
40	42	int make_sockaddr(int family, const char *addr_str, __u16 port,
41	43	struct sockaddr_storage addr, socklen_t len);
42	44

tools/testing/selftests/bpf/prog_tests/btf_map_in_map.c+34−1

@@ -53,7 +53,7 @@ static int kern_sync_rcu(void) · 52 unmodified lines
53	53	return err;
54	54	}
55	55
56		−void test_btf_map_in_map(void)
	56	+static void test_lookup_update(void)
57	57	{
58	58	int err, key = 0, val, i;
59	59	struct test_btf_map_in_map *skel;
@@ -143,3 +143,36 @@ void test_btf_map_in_map(void) · 83 unmodified lines
143	143	cleanup:
144	144	test_btf_map_in_map__destroy(skel);
145	145	}
	146	+
	147	+static void test_diff_size(void)
	148	+{
	149	+ struct test_btf_map_in_map *skel;
	150	+ int err, inner_map_fd, zero = 0;
	151	+
	152	+ skel = test_btf_map_in_map__open_and_load();
	153	+ if (CHECK(!skel, "skel_open", "failed to open&load skeleton\n"))
	154	+ return;
	155	+
	156	+ inner_map_fd = bpf_map__fd(skel->maps.sockarr_sz2);
	157	+ err = bpf_map_update_elem(bpf_map__fd(skel->maps.outer_sockarr), &zero,
	158	+ &inner_map_fd, 0);
	159	+ CHECK(err, "outer_sockarr inner map size check",
	160	+ "cannot use a different size inner_map\n");
	161	+
	162	+ inner_map_fd = bpf_map__fd(skel->maps.inner_map_sz2);
	163	+ err = bpf_map_update_elem(bpf_map__fd(skel->maps.outer_arr), &zero,
	164	+ &inner_map_fd, 0);
	165	+ CHECK(!err, "outer_arr inner map size check",
	166	+ "incorrectly updated with a different size inner_map\n");
	167	+
	168	+ test_btf_map_in_map__destroy(skel);
	169	+}
	170	+
	171	+void test_btf_map_in_map(void)
	172	+{
	173	+ if (test__start_subtest("lookup_update"))
	174	+ test_lookup_update();
	175	+
	176	+ if (test__start_subtest("diff_size"))
	177	+ test_diff_size();
	178	+}

tools/testing/selftests/bpf/prog_tests/core_reloc.c+324−26

@@ -3,6 +3,9 @@ · 2 unmodified lines
3	3	#include "progs/core_reloc_types.h"
4	4	#include <sys/mman.h>
5	5	#include <sys/syscall.h>
	6	+#include <bpf/btf.h>
	7	+
	8	+static int duration = 0;
6	9
7	10	#define STRUCT_TO_CHAR_PTR(struct_name) (const char *)&(struct struct_name)
8	11
@@ -177,14 +180,13 @@ · 168 unmodified lines
177	180	.fails = true, \
178	181	}
179	182
180		−#define EXISTENCE_CASE_COMMON(name) \
	183	+#define FIELD_EXISTS_CASE_COMMON(name) \
181	184	.case_name = #name, \
182	185	.bpf_obj_file = "test_core_reloc_existence.o", \
183		− .btf_src_file = "btf__core_reloc_" #name ".o", \
184		− .relaxed_core_relocs = true
	186	+ .btf_src_file = "btf__core_reloc_" #name ".o" \
185	187
186		−#define EXISTENCE_ERR_CASE(name) { \
187		− EXISTENCE_CASE_COMMON(name), \
	188	+#define FIELD_EXISTS_ERR_CASE(name) { \

421 more lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/d_path.c+147−0

This diff is large: its 148 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c+68−0

This diff is large: its 86 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/perf_buffer.c+54−11

This diff is large: its 106 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/resolve_btfids.c+38−1

This diff is large: its 58 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/sk_assign.c+3−2

This diff is large: its 26 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/sockmap_basic.c+76−0

This diff is large: its 95 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/tcp_hdr_options.c+622−0

This diff is large: its 623 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/test_bpffs.c+94−0

This diff is large: its 95 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/test_local_storage.c+60−0

This diff is large: its 61 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/test_lsm.c+9−0

This diff is large: its 30 lines aren’t shown here.

tools/testing/selftests/bpf/progs/btf__core_reloc_enumval.c+3−0