Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Daniel Borkmann says: ==================== pull-request: bpf-next 2022-06-17 We've added 72 non-merge commits during the last 15 day(s) which contain a total of 92 files changed, 4582 insertions(+), 834 deletions(-). The main changes are: 1) Add 64 bit enum value support to BTF, from Yonghong Song. 2) Implement support for sleepable BPF uprobe programs, from Delyan Kratunov. 3) Add new BPF helpers to issue and check TCP SYN cookies without binding to a socket especially useful in synproxy scenarios, from Maxim Mikityanskiy. 4) Fix libbpf's internal USDT address translation logic for shared libraries as well as uprobe's symbol file offset calculation, from Andrii Nakryiko. 5) Extend libbpf to provide an API for textual representation of the various map/prog/attach/link types and use it in bpftool, from Daniel Müller. 6) Provide BTF line info for RV64 and RV32 JITs, and fix a put_user bug in the core seen in 32 bit when storing BPF function addresses, from Pu Lehui. 7) Fix libbpf's BTF pointer size guessing by adding a list of various aliases for 'long' types, from Douglas Raillard. 8) Fix bpftool to readd setting rlimit since probing for memcg-based accounting has been unreliable and caused a regression on COS, from Quentin Monnet. 9) Fix UAF in BPF cgroup's effective program computation triggered upon BPF link detachment, from Tadeusz Struk. 10) Fix bpftool build bootstrapping during cross compilation which was pointing to the wrong AR process, from Shahab Vahedi. 11) Fix logic bug in libbpf's is_pow_of_2 implementation, from Yuze Chi. 12) BPF hash map optimization to avoid grabbing spinlocks of all CPUs when there is no free element. Also add a benchmark as reproducer, from Feng Zhou. 13) Fix bpftool's codegen to bail out when there's no BTF, from Michael Mullin. 14) Various minor cleanup and improvements all over the place. * https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (72 commits) bpf: Fix bpf_skc_lookup comment wrt. return type bpf: Fix non-static bpf_func_proto struct definitions selftests/bpf: Don't force lld on non-x86 architectures selftests/bpf: Add selftests for raw syncookie helpers in TC mode bpf: Allow the new syncookie helpers to work with SKBs selftests/bpf: Add selftests for raw syncookie helpers bpf: Add helpers to issue and check SYN cookies in XDP bpf: Allow helpers to accept pointers with a fixed size bpf: Fix documentation of th_len in bpf_tcp_{gen,check}_syncookie selftests/bpf: add tests for sleepable (uk)probes libbpf: add support for sleepable uprobe programs bpf: allow sleepable uprobe programs to attach bpf: implement sleepable uprobes by chaining gps bpf: move bpf_prog to bpf.h libbpf: Fix internal USDT address translation logic for shared libraries samples/bpf: Check detach prog exist or not in xdp_fwd selftests/bpf: Avoid skipping certain subtests selftests/bpf: Fix test_varlen verification failure with latest llvm bpftool: Do not check return value from libbpf_set_strict_mode() Revert "bpftool: Use libbpf 1.0 API mode instead of RLIMIT_MEMLOCK" ... ====================

Jakub Kicinski committed 4 years ago

Commit 9fb424c2 parents 4875d94 f5be22c92 files · +4582 −834Browse files

Unified · Split

92 files

Documentation/bpf/btf.rst +37 −6
Documentation/bpf/instruction-set.rst +1 −1
arch/arm/net/bpf_jit_32.c +0 −16
arch/riscv/net/bpf_jit.h +1 −0
arch/riscv/net/bpf_jit_core.c +7 −1
include/linux/bpf.h +102 −3
include/linux/bpf_verifier.h +1 −1
include/linux/btf.h +28 −0
include/linux/filter.h +0 −34
include/linux/skmsg.h +1 −0
include/net/tcp.h +1 −0
include/uapi/linux/bpf.h +84 −4
include/uapi/linux/btf.h +14 −3
kernel/bpf/btf.c +128 −14
kernel/bpf/cgroup.c +60 −10
kernel/bpf/core.c +16 −1
kernel/bpf/helpers.c +6 −6
kernel/bpf/percpu_freelist.c +14 −6
kernel/bpf/syscall.c +4 −3
kernel/bpf/verifier.c +35 −14
kernel/events/core.c +10 −6
kernel/trace/bpf_trace.c +2 −2
kernel/trace/trace_uprobe.c +2 −3
net/bpf/test_run.c +0 −6
net/core/filter.c +128 −2
net/core/skmsg.c +1 −0
net/core/sock_map.c +23 −0
net/ipv4/tcp_bpf.c +1 −0
net/ipv4/tcp_input.c +2 −1
net/xdp/xdp_umem.c +3 −3
samples/bpf/xdp_fwd_user.c +49 −6
samples/bpf/xdp_router_ipv4.bpf.c +9 −0
scripts/bpf_doc.py +4 −0
tools/bpf/bpftool/Documentation/bpftool-cgroup.rst +11 −5
tools/bpf/bpftool/Documentation/bpftool-prog.rst +3 −2
tools/bpf/bpftool/Makefile +1 −1
tools/bpf/bpftool/bash-completion/bpftool +10 −8
tools/bpf/bpftool/btf.c +54 −3
tools/bpf/bpftool/btf_dumper.c +29 −0
tools/bpf/bpftool/cgroup.c +36 −17
tools/bpf/bpftool/common.c +44 −46
tools/bpf/bpftool/feature.c +55 −34
tools/bpf/bpftool/gen.c +4 −0
tools/bpf/bpftool/link.c +31 −30
tools/bpf/bpftool/main.c +0 −2
tools/bpf/bpftool/main.h +16 −6
tools/bpf/bpftool/map.c +35 −49
tools/bpf/bpftool/pids.c +1 −0
tools/bpf/bpftool/prog.c +34 −45
tools/bpf/bpftool/struct_ops.c +2 −0
tools/include/uapi/linux/bpf.h +84 −4
tools/include/uapi/linux/btf.h +14 −3
tools/lib/bpf/btf.c +205 −24
tools/lib/bpf/btf.h +31 −1
tools/lib/bpf/btf_dump.c +103 −34
tools/lib/bpf/libbpf.c +242 −54
tools/lib/bpf/libbpf.h +37 −1
tools/lib/bpf/libbpf.map +8 −0
tools/lib/bpf/libbpf_internal.h +7 −0
tools/lib/bpf/linker.c +2 −5
tools/lib/bpf/relo_core.c +69 −44
tools/lib/bpf/relo_core.h +2 −2
tools/lib/bpf/usdt.c +63 −60
tools/testing/selftests/bpf/.gitignore +1 −0
tools/testing/selftests/bpf/Makefile +21 −7
tools/testing/selftests/bpf/bench.c +2 −0
tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_full_update.c +96 −0
tools/testing/selftests/bpf/benchs/run_bench_bpf_hashmap_full_update.sh +11 −0
tools/testing/selftests/bpf/btf_helpers.c +22 −3
tools/testing/selftests/bpf/prog_tests/attach_probe.c +48 −1
tools/testing/selftests/bpf/prog_tests/btf.c +131 −22
tools/testing/selftests/bpf/prog_tests/btf_write.c +92 −34
tools/testing/selftests/bpf/prog_tests/core_reloc.c +64 −1
tools/testing/selftests/bpf/prog_tests/fexit_stress.c +4 −28
tools/testing/selftests/bpf/prog_tests/libbpf_str.c +207 −0
tools/testing/selftests/bpf/prog_tests/tc_redirect.c +4 −4
tools/testing/selftests/bpf/prog_tests/xdp_synproxy.c +183 −0
tools/testing/selftests/bpf/progs/bpf_hashmap_full_update_bench.c +40 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_enum64val.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_enum64val___diff.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_enum64val___err_missing.c +3 −0
tools/testing/selftests/bpf/progs/btf__core_reloc_enum64val___val3_missing.c +3 −0
tools/testing/selftests/bpf/progs/core_reloc_types.h +78 −0
tools/testing/selftests/bpf/progs/test_attach_probe.c +60 −0
tools/testing/selftests/bpf/progs/test_core_reloc_enum64val.c +70 −0
tools/testing/selftests/bpf/progs/test_tc_dtime.c +51 −2
tools/testing/selftests/bpf/progs/test_varlen.c +4 −4
tools/testing/selftests/bpf/progs/xdp_synproxy_kern.c +833 −0
tools/testing/selftests/bpf/test_bpftool_synctypes.py +70 −96
tools/testing/selftests/bpf/test_btf.h +1 −0
tools/testing/selftests/bpf/test_xdping.sh +4 −0
tools/testing/selftests/bpf/xdp_synproxy.c +466 −0

Documentation/bpf/btf.rst+37−6

@@ -74,7 +74,7 @@ sequentially and type id is assigned to each recognized type starting from id · 73 unmodified lines
74	74	#define BTF_KIND_ARRAY 3 /* Array */
75	75	#define BTF_KIND_STRUCT 4 /* Struct */
76	76	#define BTF_KIND_UNION 5 /* Union */
77		− #define BTF_KIND_ENUM 6 /* Enumeration */
	77	+ #define BTF_KIND_ENUM 6 /* Enumeration up to 32-bit values */
78	78	#define BTF_KIND_FWD 7 /* Forward */
79	79	#define BTF_KIND_TYPEDEF 8 /* Typedef */
80	80	#define BTF_KIND_VOLATILE 9 /* Volatile */
@@ -87,6 +87,7 @@ sequentially and type id is assigned to each recognized type starting from id · 6 unmodified lines
87	87	#define BTF_KIND_FLOAT 16 /* Floating point */
88	88	#define BTF_KIND_DECL_TAG 17 /* Decl Tag */
89	89	#define BTF_KIND_TYPE_TAG 18 /* Type Tag */
	90	+ #define BTF_KIND_ENUM64 19 /* Enumeration up to 64-bit values */
90	91
91	92	Note that the type section encodes debug info, not just pure types.
92	93	``BTF_KIND_FUNC`` is not a type, and it represents a defined subprogram.
@@ -101,10 +102,10 @@ Each type contains the following common data:: · 8 unmodified lines
101	102	* bits 24-28: kind (e.g. int, ptr, array...etc)
102	103	* bits 29-30: unused
103	104	* bit 31: kind_flag, currently used by
104		− * struct, union and fwd
	105	+ * struct, union, fwd, enum and enum64.
105	106	*/
106	107	__u32 info;
107		− /* "size" is used by INT, ENUM, STRUCT and UNION.
	108	+ /* "size" is used by INT, ENUM, STRUCT, UNION and ENUM64.
108	109	* "size" tells the size of the type it is describing.
109	110	*
110	111	* "type" is used by PTR, TYPEDEF, VOLATILE, CONST, RESTRICT,
@@ -281,10 +282,10 @@ modes exist: · 170 unmodified lines
281	282
282	283	``struct btf_type`` encoding requirement:
283	284	* ``name_off``: 0 or offset to a valid C identifier
284		− * ``info.kind_flag``: 0
	285	+ * ``info.kind_flag``: 0 for unsigned, 1 for signed
285	286	* ``info.kind``: BTF_KIND_ENUM
286	287	* ``info.vlen``: number of enum values
287		− * ``size``: 4
	288	+ * ``size``: 1/2/4/8
288	289
289	290	``btf_type`` is followed by ``info.vlen`` number of ``struct btf_enum``.::
290	291
@@ -297,6 +298,10 @@ The ``btf_enum`` encoding: · 6 unmodified lines
297	298	* ``name_off``: offset to a valid C identifier
298	299	* ``val``: any value
299	300
	301	+If the original enum value is signed and the size is less than 4,
	302	+that value will be sign extended into 4 bytes. If the size is 8,
	303	+the value will be truncated into 4 bytes.
	304	+
300	305	2.2.7 BTF_KIND_FWD
301	306	~~~~~~~~~~~~~~~~~~
302	307
@@ -493,7 +498,7 @@ the attribute is applied to a ``struct``/``union`` member or · 190 unmodified lines
493	498	a ``func`` argument, and ``btf_decl_tag.component_idx`` should be a
494	499	valid index (starting from 0) pointing to a member or an argument.
495	500
496		−2.2.17 BTF_KIND_TYPE_TAG
	501	+2.2.18 BTF_KIND_TYPE_TAG
497	502	~~~~~~~~~~~~~~~~~~~~~~~~
498	503
499	504	``struct btf_type`` encoding requirement:
@@ -516,6 +521,32 @@ type_tag, then zero or more const/volatile/restrict/typedef · 16 unmodified lines
516	521	and finally the base type. The base type is one of
517	522	int, ptr, array, struct, union, enum, func_proto and float types.
518	523
	524	+2.2.19 BTF_KIND_ENUM64
	525	+~~~~~~~~~~~~~~~~~~~~~~
	526	+
	527	+``struct btf_type`` encoding requirement:
	528	+ * ``name_off``: 0 or offset to a valid C identifier
	529	+ * ``info.kind_flag``: 0 for unsigned, 1 for signed
	530	+ * ``info.kind``: BTF_KIND_ENUM64
	531	+ * ``info.vlen``: number of enum values
	532	+ * ``size``: 1/2/4/8
	533	+
	534	+``btf_type`` is followed by ``info.vlen`` number of ``struct btf_enum64``.::
	535	+
	536	+ struct btf_enum64 {
	537	+ __u32 name_off;
	538	+ __u32 val_lo32;
	539	+ __u32 val_hi32;
	540	+ };
	541	+
	542	+The ``btf_enum64`` encoding:
	543	+ * ``name_off``: offset to a valid C identifier
	544	+ * ``val_lo32``: lower 32-bit value for a 64-bit value
	545	+ * ``val_hi32``: high 32-bit value for a 64-bit value
	546	+
	547	+If the original enum value is signed and the size is less than 8,
	548	+that value will be sign extended into 8 bytes.
	549	+
519	550	3. BTF Kernel API
520	551	=================
521	552

Documentation/bpf/instruction-set.rst+1−1

@@ -127,7 +127,7 @@ BPF_XOR \| BPF_K \| BPF_ALU64 means:: · 126 unmodified lines
127	127	Byte swap instructions
128	128	----------------------
129	129
130		−The byte swap instructions use an instruction class of ``BFP_ALU`` and a 4-bit
	130	+The byte swap instructions use an instruction class of ``BPF_ALU`` and a 4-bit
131	131	code field of ``BPF_END``.
132	132
133	133	The byte swap instructions operate on the destination register

arch/arm/net/bpf_jit_32.c+0−16

@@ -712,22 +712,6 @@ static inline void emit_alu_r(const u8 dst, const u8 src, const bool is64, · 711 unmodified lines
712	712	}
713	713	}
714	714
715		−/* ALU operation (32 bit)
716		− * dst = dst (op) src
717		− */
718		−static inline void emit_a32_alu_r(const s8 dst, const s8 src,
719		− struct jit_ctx *ctx, const bool is64,
720		− const bool hi, const u8 op) {
721		− const s8 *tmp = bpf2a32[TMP_REG_1];
722		− s8 rn, rd;
723		−
724		− rn = arm_bpf_get_reg32(src, tmp[1], ctx);
725		− rd = arm_bpf_get_reg32(dst, tmp[0], ctx);
726		− /* ALU operation */
727		− emit_alu_r(rd, rn, is64, hi, op, ctx);
728		− arm_bpf_put_reg32(dst, rd, ctx);
729		−}
730		−
731	715	/* ALU operation (64 bit) */
732	716	static inline void emit_a32_alu_r64(const bool is64, const s8 dst[],
733	717	const s8 src[], struct jit_ctx *ctx,

arch/riscv/net/bpf_jit.h+1−0

@@ -69,6 +69,7 @@ struct rv_jit_context { · 68 unmodified lines
69	69	struct bpf_prog *prog;
70	70	u16 insns; / RV insns */
71	71	int ninsns;
	72	+ int body_len;
72	73	int epilogue_offset;
73	74	int offset; / BPF to RV */
74	75	int nexentries;

arch/riscv/net/bpf_jit_core.c+7−1

@@ -44,7 +44,7 @@ struct bpf_prog bpf_int_jit_compile(struct bpf_prog prog) · 43 unmodified lines
44	44	unsigned int prog_size = 0, extable_size = 0;
45	45	bool tmp_blinded = false, extra_pass = false;
46	46	struct bpf_prog tmp, orig_prog = prog;
47		− int pass = 0, prev_ninsns = 0, i;
	47	+ int pass = 0, prev_ninsns = 0, prologue_len, i;
48	48	struct rv_jit_data *jit_data;
49	49	struct rv_jit_context *ctx;
50	50
@@ -95,6 +95,7 @@ struct bpf_prog bpf_int_jit_compile(struct bpf_prog prog) · 44 unmodified lines
95	95	prog = orig_prog;
96	96	goto out_offset;
97	97	}
	98	+ ctx->body_len = ctx->ninsns;
98	99	bpf_jit_build_prologue(ctx);
99	100	ctx->epilogue_offset = ctx->ninsns;
100	101	bpf_jit_build_epilogue(ctx);
@@ -161,6 +162,11 @@ skip_init_ctx: · 60 unmodified lines
161	162
162	163	if (!prog->is_func \|\| extra_pass) {
163	164	bpf_jit_binary_lock_ro(jit_data->header);
	165	+ prologue_len = ctx->epilogue_offset - ctx->body_len;
	166	+ for (i = 0; i < prog->len; i++)
	167	+ ctx->offset[i] = ninsns_rvoff(prologue_len +
	168	+ ctx->offset[i]);
	169	+ bpf_prog_fill_jited_linfo(prog, ctx->offset);
164	170	out_offset:
165	171	kfree(ctx->offset);
166	172	kfree(jit_data);

include/linux/bpf.h+102−3

@@ -5,6 +5,7 @@ · 4 unmodified lines
5	5	#define _LINUX_BPF_H 1
6	6
7	7	#include <uapi/linux/bpf.h>
	8	+#include <uapi/linux/filter.h>
8	9
9	10	#include <linux/workqueue.h>
10	11	#include <linux/file.h>
@@ -22,8 +23,10 @@ · 11 unmodified lines
22	23	#include <linux/sched/mm.h>
23	24	#include <linux/slab.h>
24	25	#include <linux/percpu-refcount.h>
	26	+#include <linux/stddef.h>
25	27	#include <linux/bpfptr.h>
26	28	#include <linux/btf.h>
	29	+#include <linux/rcupdate_trace.h>
27	30
28	31	struct bpf_verifier_env;
29	32	struct bpf_verifier_log;
@@ -398,6 +401,9 @@ enum bpf_type_flag { · 368 unmodified lines
398	401	/* DYNPTR points to a ringbuf record. */
399	402	DYNPTR_TYPE_RINGBUF = BIT(9 + BPF_BASE_TYPE_BITS),
400	403
	404	+ /* Size is known at compile time. */
	405	+ MEM_FIXED_SIZE = BIT(10 + BPF_BASE_TYPE_BITS),
	406	+
401	407	__BPF_TYPE_FLAG_MAX,
402	408	__BPF_TYPE_LAST_FLAG = __BPF_TYPE_FLAG_MAX - 1,
403	409	};
@@ -461,6 +467,8 @@ enum bpf_arg_type { · 57 unmodified lines
461	467	* all bytes or clear them in error case.
462	468	*/
463	469	ARG_PTR_TO_UNINIT_MEM = MEM_UNINIT \| ARG_PTR_TO_MEM,
	470	+ /* Pointer to valid memory of size known at compile time. */
	471	+ ARG_PTR_TO_FIXED_SIZE_MEM = MEM_FIXED_SIZE \| ARG_PTR_TO_MEM,
464	472
465	473	/* This must be the last entry. Its purpose is to ensure the enum is
466	474	* wide enough to hold the higher bits reserved for bpf_type_flag.
@@ -526,6 +534,14 @@ struct bpf_func_proto { · 59 unmodified lines
526	534	u32 *arg5_btf_id;
527	535	};
528	536	u32 *arg_btf_id[5];
	537	+ struct {
	538	+ size_t arg1_size;
	539	+ size_t arg2_size;
	540	+ size_t arg3_size;
	541	+ size_t arg4_size;
	542	+ size_t arg5_size;
	543	+ };
	544	+ size_t arg_size[5];
529	545	};
530	546	int ret_btf_id; / return value btf_id */
531	547	bool (allowed)(const struct bpf_prog prog);
@@ -1084,6 +1100,40 @@ struct bpf_prog_aux { · 552 unmodified lines
1084	1100	};
1085	1101	};
1086	1102
	1103	+struct bpf_prog {
	1104	+ u16 pages; /* Number of allocated pages */
	1105	+ u16 jited:1, /* Is our filter JIT'ed? */
	1106	+ jit_requested:1,/* archs need to JIT the prog */
	1107	+ gpl_compatible:1, /* Is filter GPL compatible? */
	1108	+ cb_access:1, /* Is control block accessed? */
	1109	+ dst_needed:1, /* Do we need dst entry? */
	1110	+ blinding_requested:1, /* needs constant blinding */
	1111	+ blinded:1, /* Was blinded */
	1112	+ is_func:1, /* program is a bpf function */
	1113	+ kprobe_override:1, /* Do we override a kprobe? */
	1114	+ has_callchain_buf:1, /* callchain buffer allocated? */
	1115	+ enforce_expected_attach_type:1, /* Enforce expected_attach_type checking at attach time */
	1116	+ call_get_stack:1, /* Do we call bpf_get_stack() or bpf_get_stackid() */
	1117	+ call_get_func_ip:1, /* Do we call get_func_ip() */
	1118	+ tstamp_type_access:1; /* Accessed __sk_buff->tstamp_type */
	1119	+ enum bpf_prog_type type; /* Type of BPF program */
	1120	+ enum bpf_attach_type expected_attach_type; /* For some prog types */
	1121	+ u32 len; /* Number of filter blocks */
	1122	+ u32 jited_len; /* Size of jited insns in bytes */
	1123	+ u8 tag[BPF_TAG_SIZE];
	1124	+ struct bpf_prog_stats __percpu *stats;
	1125	+ int __percpu *active;
	1126	+ unsigned int (bpf_func)(const void ctx,
	1127	+ const struct bpf_insn *insn);
	1128	+ struct bpf_prog_aux aux; / Auxiliary fields */
	1129	+ struct sock_fprog_kern orig_prog; / Original BPF program */
	1130	+ /* Instructions for interpreter */
	1131	+ union {
	1132	+ DECLARE_FLEX_ARRAY(struct sock_filter, insns);
	1133	+ DECLARE_FLEX_ARRAY(struct bpf_insn, insnsi);
	1134	+ };
	1135	+};
	1136	+
1087	1137	struct bpf_array_aux {
1088	1138	/* Programs with direct jumps into programs part of this array. */
1089	1139	struct list_head poke_progs;
@@ -1336,6 +1386,8 @@ extern struct bpf_empty_prog_array bpf_empty_prog_array; · 246 unmodified lines
1336	1386
1337	1387	struct bpf_prog_array *bpf_prog_array_alloc(u32 prog_cnt, gfp_t flags);
1338	1388	void bpf_prog_array_free(struct bpf_prog_array *progs);
	1389	+/* Use when traversal over the bpf_prog_array uses tasks_trace rcu */
	1390	+void bpf_prog_array_free_sleepable(struct bpf_prog_array *progs);
1339	1391	int bpf_prog_array_length(struct bpf_prog_array *progs);
1340	1392	bool bpf_prog_array_is_empty(struct bpf_prog_array *array);
1341	1393	int bpf_prog_array_copy_to_user(struct bpf_prog_array *progs,
@@ -1427,6 +1479,55 @@ bpf_prog_run_array(const struct bpf_prog_array *array, · 85 unmodified lines
1427	1479	return ret;
1428	1480	}
1429	1481
	1482	+/* Notes on RCU design for bpf_prog_arrays containing sleepable programs:
	1483	+ *
	1484	+ * We use the tasks_trace rcu flavor read section to protect the bpf_prog_array
	1485	+ * overall. As a result, we must use the bpf_prog_array_free_sleepable
	1486	+ * in order to use the tasks_trace rcu grace period.
	1487	+ *
	1488	+ * When a non-sleepable program is inside the array, we take the rcu read
	1489	+ * section and disable preemption for that program alone, so it can access
	1490	+ * rcu-protected dynamically sized maps.
	1491	+ */
	1492	+static __always_inline u32
	1493	+bpf_prog_run_array_sleepable(const struct bpf_prog_array __rcu *array_rcu,
	1494	+ const void *ctx, bpf_prog_run_fn run_prog)
	1495	+{
	1496	+ const struct bpf_prog_array_item *item;
	1497	+ const struct bpf_prog *prog;
	1498	+ const struct bpf_prog_array *array;
	1499	+ struct bpf_run_ctx *old_run_ctx;
	1500	+ struct bpf_trace_run_ctx run_ctx;
	1501	+ u32 ret = 1;
	1502	+
	1503	+ might_fault();
	1504	+
	1505	+ rcu_read_lock_trace();
	1506	+ migrate_disable();
	1507	+
	1508	+ array = rcu_dereference_check(array_rcu, rcu_read_lock_trace_held());
	1509	+ if (unlikely(!array))
	1510	+ goto out;
	1511	+ old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
	1512	+ item = &array->items[0];
	1513	+ while ((prog = READ_ONCE(item->prog))) {
	1514	+ if (!prog->aux->sleepable)
	1515	+ rcu_read_lock();
	1516	+
	1517	+ run_ctx.bpf_cookie = item->bpf_cookie;
	1518	+ ret &= run_prog(prog, ctx);
	1519	+ item++;
	1520	+
	1521	+ if (!prog->aux->sleepable)
	1522	+ rcu_read_unlock();
	1523	+ }
	1524	+ bpf_reset_run_ctx(old_run_ctx);
	1525	+out:
	1526	+ migrate_enable();
	1527	+ rcu_read_unlock_trace();
	1528	+ return ret;
	1529	+}
	1530	+
1430	1531	#ifdef CONFIG_BPF_SYSCALL
1431	1532	DECLARE_PER_CPU(int, bpf_prog_active);
1432	1533	extern struct mutex bpf_stats_enabled_mutex;
@@ -2104,6 +2205,7 @@ int sock_map_bpf_prog_query(const union bpf_attr *attr, · 671 unmodified lines
2104	2205	union bpf_attr __user *uattr);
2105	2206
2106	2207	void sock_map_unhash(struct sock *sk);
	2208	+void sock_map_destroy(struct sock *sk);
2107	2209	void sock_map_close(struct sock *sk, long timeout);
2108	2210	#else
2109	2211	static inline int bpf_prog_offload_init(struct bpf_prog *prog,
@@ -2261,12 +2363,9 @@ extern const struct bpf_func_proto bpf_for_each_map_elem_proto; · 151 unmodified lines
2261	2363	extern const struct bpf_func_proto bpf_btf_find_by_name_kind_proto;
2262	2364	extern const struct bpf_func_proto bpf_sk_setsockopt_proto;
2263	2365	extern const struct bpf_func_proto bpf_sk_getsockopt_proto;
2264		−extern const struct bpf_func_proto bpf_kallsyms_lookup_name_proto;
2265	2366	extern const struct bpf_func_proto bpf_find_vma_proto;
2266	2367	extern const struct bpf_func_proto bpf_loop_proto;
2267		−extern const struct bpf_func_proto bpf_strncmp_proto;
2268	2368	extern const struct bpf_func_proto bpf_copy_from_user_task_proto;
2269		−extern const struct bpf_func_proto bpf_kptr_xchg_proto;
2270	2369
2271	2370	const struct bpf_func_proto *tracing_prog_func_proto(
2272	2371	enum bpf_func_id func_id, const struct bpf_prog *prog);

include/linux/bpf_verifier.h+1−1

@@ -299,7 +299,7 @@ struct bpf_verifier_state { · 298 unmodified lines
299	299	* If is_state_visited() sees a state with branches > 0 it means
300	300	* there is a loop. If such state is exactly equal to the current state
301	301	* it's an infinite loop. Note states_equal() checks for states
302		− * equvalency, so two states being 'states_equal' does not mean
	302	+ * equivalency, so two states being 'states_equal' does not mean
303	303	* infinite loop. The exact comparison is provided by
304	304	* states_maybe_looping() function. It's a stronger pre-check and
305	305	* much faster than states_equal().

include/linux/btf.h+28−0

@@ -177,6 +177,19 @@ static inline bool btf_type_is_enum(const struct btf_type *t) · 176 unmodified lines
177	177	return BTF_INFO_KIND(t->info) == BTF_KIND_ENUM;
178	178	}
179	179
	180	+static inline bool btf_is_any_enum(const struct btf_type *t)
	181	+{
	182	+ return BTF_INFO_KIND(t->info) == BTF_KIND_ENUM \|\|
	183	+ BTF_INFO_KIND(t->info) == BTF_KIND_ENUM64;
	184	+}
	185	+
	186	+static inline bool btf_kind_core_compat(const struct btf_type *t1,
	187	+ const struct btf_type *t2)
	188	+{
	189	+ return BTF_INFO_KIND(t1->info) == BTF_INFO_KIND(t2->info) \|\|
	190	+ (btf_is_any_enum(t1) && btf_is_any_enum(t2));
	191	+}
	192	+
180	193	static inline bool str_is_empty(const char *s)
181	194	{
182	195	return !s \|\| !s[0];
@@ -192,6 +205,16 @@ static inline bool btf_is_enum(const struct btf_type *t) · 9 unmodified lines
192	205	return btf_kind(t) == BTF_KIND_ENUM;
193	206	}
194	207
	208	+static inline bool btf_is_enum64(const struct btf_type *t)
	209	+{
	210	+ return btf_kind(t) == BTF_KIND_ENUM64;
	211	+}
	212	+
	213	+static inline u64 btf_enum64_value(const struct btf_enum64 *e)
	214	+{
	215	+ return ((u64)e->val_hi32 << 32) \| e->val_lo32;
	216	+}
	217	+
195	218	static inline bool btf_is_composite(const struct btf_type *t)
196	219	{
197	220	u16 kind = btf_kind(t);
@@ -332,6 +355,11 @@ static inline struct btf_enum btf_enum(const struct btf_type t) · 134 unmodified lines
332	355	return (struct btf_enum *)(t + 1);
333	356	}
334	357
	358	+static inline struct btf_enum64 btf_enum64(const struct btf_type t)
	359	+{
	360	+ return (struct btf_enum64 *)(t + 1);
	361	+}
	362	+
335	363	static inline const struct btf_var_secinfo *btf_type_var_secinfo(
336	364	const struct btf_type *t)
337	365	{

include/linux/filter.h+0−34

@@ -559,40 +559,6 @@ struct bpf_prog_stats { · 558 unmodified lines
559	559	struct u64_stats_sync syncp;
560	560	} __aligned(2 * sizeof(u64));
561	561
562		−struct bpf_prog {
563		− u16 pages; /* Number of allocated pages */
564		− u16 jited:1, /* Is our filter JIT'ed? */
565		− jit_requested:1,/* archs need to JIT the prog */
566		− gpl_compatible:1, /* Is filter GPL compatible? */
567		− cb_access:1, /* Is control block accessed? */
568		− dst_needed:1, /* Do we need dst entry? */
569		− blinding_requested:1, /* needs constant blinding */
570		− blinded:1, /* Was blinded */
571		− is_func:1, /* program is a bpf function */
572		− kprobe_override:1, /* Do we override a kprobe? */
573		− has_callchain_buf:1, /* callchain buffer allocated? */
574		− enforce_expected_attach_type:1, /* Enforce expected_attach_type checking at attach time */
575		− call_get_stack:1, /* Do we call bpf_get_stack() or bpf_get_stackid() */
576		− call_get_func_ip:1, /* Do we call get_func_ip() */
577		− tstamp_type_access:1; /* Accessed __sk_buff->tstamp_type */
578		− enum bpf_prog_type type; /* Type of BPF program */
579		− enum bpf_attach_type expected_attach_type; /* For some prog types */
580		− u32 len; /* Number of filter blocks */
581		− u32 jited_len; /* Size of jited insns in bytes */
582		− u8 tag[BPF_TAG_SIZE];
583		− struct bpf_prog_stats __percpu *stats;
584		− int __percpu *active;
585		− unsigned int (bpf_func)(const void ctx,
586		− const struct bpf_insn *insn);
587		− struct bpf_prog_aux aux; / Auxiliary fields */
588		− struct sock_fprog_kern orig_prog; / Original BPF program */
589		− /* Instructions for interpreter */
590		− union {
591		− DECLARE_FLEX_ARRAY(struct sock_filter, insns);
592		− DECLARE_FLEX_ARRAY(struct bpf_insn, insnsi);
593		− };
594		−};
595		−
596	562	struct sk_filter {
597	563	refcount_t refcnt;
598	564	struct rcu_head rcu;

include/linux/skmsg.h+1−0

@@ -95,6 +95,7 @@ struct sk_psock { · 94 unmodified lines
95	95	spinlock_t link_lock;
96	96	refcount_t refcnt;
97	97	void (saved_unhash)(struct sock sk);
	98	+ void (saved_destroy)(struct sock sk);
98	99	void (saved_close)(struct sock sk, long timeout);
99	100	void (saved_write_space)(struct sock sk);
100	101	void (saved_data_ready)(struct sock sk);

include/net/tcp.h+1−0

@@ -434,6 +434,7 @@ u16 tcp_v4_get_syncookie(struct sock sk, struct iphdr iph, · 433 unmodified lines
434	434	struct tcphdr th, u32 cookie);
435	435	u16 tcp_v6_get_syncookie(struct sock sk, struct ipv6hdr iph,
436	436	struct tcphdr th, u32 cookie);
	437	+u16 tcp_parse_mss_option(const struct tcphdr *th, u16 user_mss);
437	438	u16 tcp_get_syncookie_mss(struct request_sock_ops *rsk_ops,
438	439	const struct tcp_request_sock_ops *af_ops,
439	440	struct sock sk, struct tcphdr th);

include/uapi/linux/bpf.h+84−4

@@ -3597,10 +3597,11 @@ union bpf_attr { · 3596 unmodified lines
3597	3597	*
3598	3598	* iph points to the start of the IPv4 or IPv6 header, while
3599	3599	* iph_len contains sizeof\ (struct iphdr) or
3600		− * sizeof\ (struct ip6hdr).
	3600	+ * sizeof\ (struct ipv6hdr).
3601	3601	*
3602	3602	* th points to the start of the TCP header, while th_len
3603		− * contains sizeof\ (struct tcphdr).
	3603	+ * contains the length of the TCP header (at least
	3604	+ * sizeof\ (struct tcphdr)).
3604	3605	* Return
3605	3606	* 0 if iph and th are a valid SYN cookie ACK, or a negative
3606	3607	* error otherwise.
@@ -3783,10 +3784,11 @@ union bpf_attr { · 176 unmodified lines
3783	3784	*
3784	3785	* iph points to the start of the IPv4 or IPv6 header, while
3785	3786	* iph_len contains sizeof\ (struct iphdr) or
3786		− * sizeof\ (struct ip6hdr).
	3787	+ * sizeof\ (struct ipv6hdr).
3787	3788	*
3788	3789	* th points to the start of the TCP header, while th_len
3789		− * contains the length of the TCP header.
	3790	+ * contains the length of the TCP header with options (at least
	3791	+ * sizeof\ (struct tcphdr)).
3790	3792	* Return
3791	3793	* On success, lower 32 bits hold the generated SYN cookie in
3792	3794	* followed by 16 bits which hold the MSS value for that cookie,
@@ -5249,6 +5251,80 @@ union bpf_attr { · 1456 unmodified lines
5249	5251	* Pointer to the underlying dynptr data, NULL if the dynptr is
5250	5252	* read-only, if the dynptr is invalid, or if the offset and length
5251	5253	* is out of bounds.
	5254	+ *
	5255	+ * s64 bpf_tcp_raw_gen_syncookie_ipv4(struct iphdr iph, struct tcphdr th, u32 th_len)
	5256	+ * Description
	5257	+ * Try to issue a SYN cookie for the packet with corresponding
	5258	+ * IPv4/TCP headers, iph and th, without depending on a
	5259	+ * listening socket.
	5260	+ *
	5261	+ * iph points to the IPv4 header.
	5262	+ *
	5263	+ * th points to the start of the TCP header, while th_len
	5264	+ * contains the length of the TCP header (at least
	5265	+ * sizeof\ (struct tcphdr)).
	5266	+ * Return
	5267	+ * On success, lower 32 bits hold the generated SYN cookie in
	5268	+ * followed by 16 bits which hold the MSS value for that cookie,
	5269	+ * and the top 16 bits are unused.
	5270	+ *
	5271	+ * On failure, the returned value is one of the following:
	5272	+ *
	5273	+ * -EINVAL if th_len is invalid.
	5274	+ *
	5275	+ * s64 bpf_tcp_raw_gen_syncookie_ipv6(struct ipv6hdr iph, struct tcphdr th, u32 th_len)
	5276	+ * Description
	5277	+ * Try to issue a SYN cookie for the packet with corresponding
	5278	+ * IPv6/TCP headers, iph and th, without depending on a
	5279	+ * listening socket.
	5280	+ *
	5281	+ * iph points to the IPv6 header.
	5282	+ *
	5283	+ * th points to the start of the TCP header, while th_len
	5284	+ * contains the length of the TCP header (at least
	5285	+ * sizeof\ (struct tcphdr)).
	5286	+ * Return
	5287	+ * On success, lower 32 bits hold the generated SYN cookie in
	5288	+ * followed by 16 bits which hold the MSS value for that cookie,
	5289	+ * and the top 16 bits are unused.
	5290	+ *
	5291	+ * On failure, the returned value is one of the following:
	5292	+ *
	5293	+ * -EINVAL if th_len is invalid.
	5294	+ *
	5295	+ * -EPROTONOSUPPORT if CONFIG_IPV6 is not builtin.
	5296	+ *
	5297	+ * long bpf_tcp_raw_check_syncookie_ipv4(struct iphdr iph, struct tcphdr th)
	5298	+ * Description
	5299	+ * Check whether iph and th contain a valid SYN cookie ACK
	5300	+ * without depending on a listening socket.
	5301	+ *
	5302	+ * iph points to the IPv4 header.
	5303	+ *
	5304	+ * th points to the TCP header.
	5305	+ * Return
	5306	+ * 0 if iph and th are a valid SYN cookie ACK.
	5307	+ *
	5308	+ * On failure, the returned value is one of the following:
	5309	+ *
	5310	+ * -EACCES if the SYN cookie is not valid.
	5311	+ *
	5312	+ * long bpf_tcp_raw_check_syncookie_ipv6(struct ipv6hdr iph, struct tcphdr th)
	5313	+ * Description
	5314	+ * Check whether iph and th contain a valid SYN cookie ACK
	5315	+ * without depending on a listening socket.
	5316	+ *
	5317	+ * iph points to the IPv6 header.
	5318	+ *
	5319	+ * th points to the TCP header.
	5320	+ * Return
	5321	+ * 0 if iph and th are a valid SYN cookie ACK.
	5322	+ *
	5323	+ * On failure, the returned value is one of the following:
	5324	+ *
	5325	+ * -EACCES if the SYN cookie is not valid.
	5326	+ *
	5327	+ * -EPROTONOSUPPORT if CONFIG_IPV6 is not builtin.
5252	5328	*/
5253	5329	#define __BPF_FUNC_MAPPER(FN) \
5254	5330	FN(unspec), \
@@ -5455,6 +5531,10 @@ union bpf_attr { · 200 unmodified lines
5455	5531	FN(dynptr_read), \
5456	5532	FN(dynptr_write), \
5457	5533	FN(dynptr_data), \
	5534	+ FN(tcp_raw_gen_syncookie_ipv4), \
	5535	+ FN(tcp_raw_gen_syncookie_ipv6), \
	5536	+ FN(tcp_raw_check_syncookie_ipv4), \
	5537	+ FN(tcp_raw_check_syncookie_ipv6), \
5458	5538	/* */
5459	5539
5460	5540	/* integer value in 'imm' field of BPF_CALL instruction selects which helper

include/uapi/linux/btf.h+14−3

@@ -36,10 +36,10 @@ struct btf_type { · 35 unmodified lines
36	36	* bits 24-28: kind (e.g. int, ptr, array...etc)
37	37	* bits 29-30: unused
38	38	* bit 31: kind_flag, currently used by
39		− * struct, union and fwd
	39	+ * struct, union, enum, fwd and enum64
40	40	*/
41	41	__u32 info;
42		− /* "size" is used by INT, ENUM, STRUCT, UNION and DATASEC.
	42	+ /* "size" is used by INT, ENUM, STRUCT, UNION, DATASEC and ENUM64.
43	43	* "size" tells the size of the type it is describing.
44	44	*
45	45	* "type" is used by PTR, TYPEDEF, VOLATILE, CONST, RESTRICT,
@@ -63,7 +63,7 @@ enum { · 17 unmodified lines
63	63	BTF_KIND_ARRAY = 3, /* Array */
64	64	BTF_KIND_STRUCT = 4, /* Struct */
65	65	BTF_KIND_UNION = 5, /* Union */
66		− BTF_KIND_ENUM = 6, /* Enumeration */
	66	+ BTF_KIND_ENUM = 6, /* Enumeration up to 32-bit values */
67	67	BTF_KIND_FWD = 7, /* Forward */
68	68	BTF_KIND_TYPEDEF = 8, /* Typedef */
69	69	BTF_KIND_VOLATILE = 9, /* Volatile */
@@ -76,6 +76,7 @@ enum { · 6 unmodified lines
76	76	BTF_KIND_FLOAT = 16, /* Floating point */
77	77	BTF_KIND_DECL_TAG = 17, /* Decl Tag */
78	78	BTF_KIND_TYPE_TAG = 18, /* Type Tag */
	79	+ BTF_KIND_ENUM64 = 19, /* Enumeration up to 64-bit values */
79	80
80	81	NR_BTF_KINDS,
81	82	BTF_KIND_MAX = NR_BTF_KINDS - 1,
@@ -186,4 +187,14 @@ struct btf_decl_tag { · 104 unmodified lines
186	187	__s32 component_idx;
187	188	};
188	189
	190	+/* BTF_KIND_ENUM64 is followed by multiple "struct btf_enum64".
	191	+ * The exact number of btf_enum64 is stored in the vlen (of the
	192	+ * info in "struct btf_type").
	193	+ */
	194	+struct btf_enum64 {
	195	+ __u32 name_off;
	196	+ __u32 val_lo32;
	197	+ __u32 val_hi32;
	198	+};
	199	+
189	200	#endif /* _UAPI__LINUX_BTF_H__ */

kernel/bpf/btf.c+128−14

@@ -309,6 +309,7 @@ static const char * const btf_kind_str[NR_BTF_KINDS] = { · 308 unmodified lines
309	309	[BTF_KIND_FLOAT] = "FLOAT",
310	310	[BTF_KIND_DECL_TAG] = "DECL_TAG",
311	311	[BTF_KIND_TYPE_TAG] = "TYPE_TAG",
	312	+ [BTF_KIND_ENUM64] = "ENUM64",
312	313	};
313	314
314	315	const char btf_type_str(const struct btf_type t)
@@ -666,6 +667,7 @@ static bool btf_type_has_size(const struct btf_type *t) · 351 unmodified lines
666	667	case BTF_KIND_ENUM:
667	668	case BTF_KIND_DATASEC:
668	669	case BTF_KIND_FLOAT:
	670	+ case BTF_KIND_ENUM64:
669	671	return true;
670	672	}
671	673
@@ -711,6 +713,11 @@ static const struct btf_decl_tag btf_type_decl_tag(const struct btf_type t) · 39 unmodified lines
711	713	return (const struct btf_decl_tag *)(t + 1);
712	714	}
713	715
	716	+static const struct btf_enum64 btf_type_enum64(const struct btf_type t)
	717	+{
	718	+ return (const struct btf_enum64 *)(t + 1);
	719	+}
	720	+
714	721	static const struct btf_kind_operations btf_type_ops(const struct btf_type t)
715	722	{
716	723	return kind_ops[BTF_INFO_KIND(t->info)];
@@ -1019,6 +1026,7 @@ static const char btf_show_name(struct btf_show show) · 302 unmodified lines
1019	1026	parens = "{";
1020	1027	break;
1021	1028	case BTF_KIND_ENUM:
	1029	+ case BTF_KIND_ENUM64:
1022	1030	prefix = "enum";
1023	1031	break;
1024	1032	default:
@@ -1834,6 +1842,7 @@ __btf_resolve_size(const struct btf btf, const struct btf_type type, · 809 unmodified lines
1834	1842	case BTF_KIND_UNION:
1835	1843	case BTF_KIND_ENUM:
1836	1844	case BTF_KIND_FLOAT:
	1845	+ case BTF_KIND_ENUM64:
1837	1846	size = type->size;
1838	1847	goto resolved;
1839	1848
@@ -3670,6 +3679,7 @@ static s32 btf_enum_check_meta(struct btf_verifier_env *env, · 1830 unmodified lines
3670	3679	{
3671	3680	const struct btf_enum *enums = btf_type_enum(t);
3672	3681	struct btf *btf = env->btf;
	3682	+ const char *fmt_str;
3673	3683	u16 i, nr_enums;
3674	3684	u32 meta_needed;
3675	3685
@@ -3683,11 +3693,6 @@ static s32 btf_enum_check_meta(struct btf_verifier_env *env, · 7 unmodified lines
3683	3693	return -EINVAL;
3684	3694	}
3685	3695
3686		− if (btf_type_kflag(t)) {
3687		− btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
3688		− return -EINVAL;
3689		− }
3690		−
3691	3696	if (t->size > 8 \|\| !is_power_of_2(t->size)) {
3692	3697	btf_verifier_log_type(env, t, "Unexpected size");
3693	3698	return -EINVAL;
@@ -3718,7 +3723,8 @@ static s32 btf_enum_check_meta(struct btf_verifier_env *env, · 24 unmodified lines
3718	3723
3719	3724	if (env->log.level == BPF_LOG_KERNEL)
3720	3725	continue;
3721		− btf_verifier_log(env, "\t%s val=%d\n",
	3726	+ fmt_str = btf_type_kflag(t) ? "\t%s val=%d\n" : "\t%s val=%u\n";
	3727	+ btf_verifier_log(env, fmt_str,
3722	3728	__btf_name_by_offset(btf, enums[i].name_off),
3723	3729	enums[i].val);
3724	3730	}
@@ -3759,7 +3765,10 @@ static void btf_enum_show(const struct btf btf, const struct btf_type t, · 34 unmodified lines
3759	3765	return;
3760	3766	}
3761	3767
3762		− btf_show_type_value(show, "%d", v);
	3768	+ if (btf_type_kflag(t))
	3769	+ btf_show_type_value(show, "%d", v);
	3770	+ else
	3771	+ btf_show_type_value(show, "%u", v);
3763	3772	btf_show_end_type(show);
3764	3773	}
3765	3774
@@ -3772,6 +3781,109 @@ static struct btf_kind_operations enum_ops = { · 6 unmodified lines
3772	3781	.show = btf_enum_show,
3773	3782	};
3774	3783
	3784	+static s32 btf_enum64_check_meta(struct btf_verifier_env *env,
	3785	+ const struct btf_type *t,
	3786	+ u32 meta_left)
	3787	+{
	3788	+ const struct btf_enum64 *enums = btf_type_enum64(t);
	3789	+ struct btf *btf = env->btf;
	3790	+ const char *fmt_str;
	3791	+ u16 i, nr_enums;
	3792	+ u32 meta_needed;
	3793	+
	3794	+ nr_enums = btf_type_vlen(t);
	3795	+ meta_needed = nr_enums * sizeof(*enums);
	3796	+
	3797	+ if (meta_left < meta_needed) {
	3798	+ btf_verifier_log_basic(env, t,
	3799	+ "meta_left:%u meta_needed:%u",
	3800	+ meta_left, meta_needed);
	3801	+ return -EINVAL;
	3802	+ }
	3803	+
	3804	+ if (t->size > 8 \|\| !is_power_of_2(t->size)) {
	3805	+ btf_verifier_log_type(env, t, "Unexpected size");
	3806	+ return -EINVAL;
	3807	+ }
	3808	+
	3809	+ /* enum type either no name or a valid one */
	3810	+ if (t->name_off &&
	3811	+ !btf_name_valid_identifier(env->btf, t->name_off)) {
	3812	+ btf_verifier_log_type(env, t, "Invalid name");
	3813	+ return -EINVAL;
	3814	+ }
	3815	+
	3816	+ btf_verifier_log_type(env, t, NULL);
	3817	+
	3818	+ for (i = 0; i < nr_enums; i++) {
	3819	+ if (!btf_name_offset_valid(btf, enums[i].name_off)) {
	3820	+ btf_verifier_log(env, "\tInvalid name_offset:%u",
	3821	+ enums[i].name_off);
	3822	+ return -EINVAL;
	3823	+ }
	3824	+
	3825	+ /* enum member must have a valid name */
	3826	+ if (!enums[i].name_off \|\|
	3827	+ !btf_name_valid_identifier(btf, enums[i].name_off)) {
	3828	+ btf_verifier_log_type(env, t, "Invalid name");
	3829	+ return -EINVAL;
	3830	+ }
	3831	+
	3832	+ if (env->log.level == BPF_LOG_KERNEL)
	3833	+ continue;
	3834	+
	3835	+ fmt_str = btf_type_kflag(t) ? "\t%s val=%lld\n" : "\t%s val=%llu\n";
	3836	+ btf_verifier_log(env, fmt_str,
	3837	+ __btf_name_by_offset(btf, enums[i].name_off),
	3838	+ btf_enum64_value(enums + i));
	3839	+ }
	3840	+
	3841	+ return meta_needed;
	3842	+}
	3843	+
	3844	+static void btf_enum64_show(const struct btf btf, const struct btf_type t,
	3845	+ u32 type_id, void *data, u8 bits_offset,
	3846	+ struct btf_show *show)
	3847	+{
	3848	+ const struct btf_enum64 *enums = btf_type_enum64(t);
	3849	+ u32 i, nr_enums = btf_type_vlen(t);
	3850	+ void *safe_data;
	3851	+ s64 v;
	3852	+
	3853	+ safe_data = btf_show_start_type(show, t, type_id, data);
	3854	+ if (!safe_data)
	3855	+ return;
	3856	+
	3857	+ v = (u64 )safe_data;
	3858	+
	3859	+ for (i = 0; i < nr_enums; i++) {
	3860	+ if (v != btf_enum64_value(enums + i))
	3861	+ continue;
	3862	+
	3863	+ btf_show_type_value(show, "%s",
	3864	+ __btf_name_by_offset(btf,
	3865	+ enums[i].name_off));
	3866	+
	3867	+ btf_show_end_type(show);
	3868	+ return;
	3869	+ }
	3870	+
	3871	+ if (btf_type_kflag(t))
	3872	+ btf_show_type_value(show, "%lld", v);
	3873	+ else
	3874	+ btf_show_type_value(show, "%llu", v);
	3875	+ btf_show_end_type(show);
	3876	+}
	3877	+
	3878	+static struct btf_kind_operations enum64_ops = {
	3879	+ .check_meta = btf_enum64_check_meta,
	3880	+ .resolve = btf_df_resolve,
	3881	+ .check_member = btf_enum_check_member,
	3882	+ .check_kflag_member = btf_enum_check_kflag_member,
	3883	+ .log_details = btf_enum_log,
	3884	+ .show = btf_enum64_show,
	3885	+};
	3886	+
3775	3887	static s32 btf_func_proto_check_meta(struct btf_verifier_env *env,
3776	3888	const struct btf_type *t,
3777	3889	u32 meta_left)
@@ -4438,6 +4550,7 @@ static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS] = { · 660 unmodified lines
4438	4550	[BTF_KIND_FLOAT] = &float_ops,
4439	4551	[BTF_KIND_DECL_TAG] = &decl_tag_ops,
4440	4552	[BTF_KIND_TYPE_TAG] = &modifier_ops,
	4553	+ [BTF_KIND_ENUM64] = &enum64_ops,
4441	4554	};
4442	4555
4443	4556	static s32 btf_check_meta(struct btf_verifier_env *env,
@@ -5299,7 +5412,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 855 unmodified lines
5299	5412	/* skip modifiers */
5300	5413	while (btf_type_is_modifier(t))
5301	5414	t = btf_type_by_id(btf, t->type);
5302		− if (btf_type_is_small_int(t) \|\| btf_type_is_enum(t))
	5415	+ if (btf_type_is_small_int(t) \|\| btf_is_any_enum(t))
5303	5416	/* accessing a scalar */
5304	5417	return true;
5305	5418	if (!btf_type_is_ptr(t)) {
@@ -5763,7 +5876,7 @@ static int __get_type_size(struct btf *btf, u32 btf_id, · 457 unmodified lines
5763	5876	if (btf_type_is_ptr(t))
5764	5877	/* kernel size of pointer. Not BPF's size of pointer*/
5765	5878	return sizeof(void *);
5766		− if (btf_type_is_int(t) \|\| btf_type_is_enum(t))
	5879	+ if (btf_type_is_int(t) \|\| btf_is_any_enum(t))
5767	5880	return t->size;
5768	5881	*bad_type = t;
5769	5882	return -EINVAL;
@@ -5911,7 +6024,7 @@ static int btf_check_func_type_match(struct bpf_verifier_log *log, · 141 unmodified lines
5911	6024	* to context only. And only global functions can be replaced.
5912	6025	* Hence type check only those types.
5913	6026	*/
5914		− if (btf_type_is_int(t1) \|\| btf_type_is_enum(t1))
	6027	+ if (btf_type_is_int(t1) \|\| btf_is_any_enum(t1))
5915	6028	continue;
5916	6029	if (!btf_type_is_ptr(t1)) {
5917	6030	bpf_log(log,
@@ -6409,7 +6522,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, · 491 unmodified lines
6409	6522	t = btf_type_by_id(btf, t->type);
6410	6523	while (btf_type_is_modifier(t))
6411	6524	t = btf_type_by_id(btf, t->type);
6412		− if (!btf_type_is_int(t) && !btf_type_is_enum(t)) {
	6525	+ if (!btf_type_is_int(t) && !btf_is_any_enum(t)) {
6413	6526	bpf_log(log,
6414	6527	"Global function %s() doesn't return scalar. Only those are supported.\n",
6415	6528	tname);
@@ -6424,7 +6537,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, · 8 unmodified lines
6424	6537	t = btf_type_by_id(btf, args[i].type);
6425	6538	while (btf_type_is_modifier(t))
6426	6539	t = btf_type_by_id(btf, t->type);
6427		− if (btf_type_is_int(t) \|\| btf_type_is_enum(t)) {
	6540	+ if (btf_type_is_int(t) \|\| btf_is_any_enum(t)) {
6428	6541	reg->type = SCALAR_VALUE;
6429	6542	continue;
6430	6543	}
@@ -7336,6 +7449,7 @@ recur: · 905 unmodified lines
7336	7449	case BTF_KIND_UNION:
7337	7450	case BTF_KIND_ENUM:
7338	7451	case BTF_KIND_FWD:
	7452	+ case BTF_KIND_ENUM64:
7339	7453	return 1;
7340	7454	case BTF_KIND_INT:
7341	7455	/* just reject deprecated bitfield-like integers; all other
@@ -7388,10 +7502,10 @@ recur: · 46 unmodified lines
7388	7502	* field-based relocations. This function assumes that root types were already
7389	7503	* checked for name match. Beyond that initial root-level name check, names
7390	7504	* are completely ignored. Compatibility rules are as follows:
7391		− * - any two STRUCTs/UNIONs/FWDs/ENUMs/INTs are considered compatible, but
	7505	+ * - any two STRUCTs/UNIONs/FWDs/ENUMs/INTs/ENUM64s are considered compatible, but
7392	7506	* kind should match for local and target types (i.e., STRUCT is not
7393	7507	* compatible with UNION);
7394		− * - for ENUMs, the size is ignored;
	7508	+ * - for ENUMs/ENUM64s, the size is ignored;
7395	7509	* - for INT, size and signedness are ignored;
7396	7510	* - for ARRAY, dimensionality is ignored, element types are checked for
7397	7511	* compatibility recursively;

kernel/bpf/cgroup.c+60−10

@@ -720,6 +720,60 @@ static struct bpf_prog_list find_detach_entry(struct list_head progs, · 719 unmodified lines
720	720	return ERR_PTR(-ENOENT);
721	721	}
722	722
	723	+/**
	724	+ * purge_effective_progs() - After compute_effective_progs fails to alloc new
	725	+ * cgrp->bpf.inactive table we can recover by
	726	+ * recomputing the array in place.
	727	+ *
	728	+ * @cgrp: The cgroup which descendants to travers
	729	+ * @prog: A program to detach or NULL
	730	+ * @link: A link to detach or NULL
	731	+ * @atype: Type of detach operation
	732	+ */
	733	+static void purge_effective_progs(struct cgroup cgrp, struct bpf_prog prog,
	734	+ struct bpf_cgroup_link *link,
	735	+ enum cgroup_bpf_attach_type atype)
	736	+{
	737	+ struct cgroup_subsys_state *css;
	738	+ struct bpf_prog_array *progs;
	739	+ struct bpf_prog_list *pl;
	740	+ struct list_head *head;
	741	+ struct cgroup *cg;
	742	+ int pos;
	743	+
	744	+ /* recompute effective prog array in place */
	745	+ css_for_each_descendant_pre(css, &cgrp->self) {
	746	+ struct cgroup *desc = container_of(css, struct cgroup, self);
	747	+
	748	+ if (percpu_ref_is_zero(&desc->bpf.refcnt))
	749	+ continue;
	750	+
	751	+ /* find position of link or prog in effective progs array */
	752	+ for (pos = 0, cg = desc; cg; cg = cgroup_parent(cg)) {
	753	+ if (pos && !(cg->bpf.flags[atype] & BPF_F_ALLOW_MULTI))
	754	+ continue;
	755	+
	756	+ head = &cg->bpf.progs[atype];
	757	+ list_for_each_entry(pl, head, node) {
	758	+ if (!prog_list_prog(pl))
	759	+ continue;
	760	+ if (pl->prog == prog && pl->link == link)
	761	+ goto found;
	762	+ pos++;
	763	+ }
	764	+ }
	765	+found:
	766	+ BUG_ON(!cg);
	767	+ progs = rcu_dereference_protected(
	768	+ desc->bpf.effective[atype],
	769	+ lockdep_is_held(&cgroup_mutex));
	770	+
	771	+ /* Remove the program from the array */
	772	+ WARN_ONCE(bpf_prog_array_delete_safe_at(progs, pos),
	773	+ "Failed to purge a prog from array at index %d", pos);
	774	+ }
	775	+}
	776	+
723	777	/**
724	778	* __cgroup_bpf_detach() - Detach the program or link from a cgroup, and
725	779	* propagate the change to descendants
@@ -739,7 +793,6 @@ static int __cgroup_bpf_detach(struct cgroup cgrp, struct bpf_prog prog, · 13 unmodified lines
739	793	struct bpf_prog_list *pl;
740	794	struct list_head *progs;
741	795	u32 flags;
742		− int err;
743	796
744	797	atype = to_cgroup_bpf_attach_type(type);
745	798	if (atype < 0)
@@ -761,9 +814,12 @@ static int __cgroup_bpf_detach(struct cgroup cgrp, struct bpf_prog prog, · 15 unmodified lines
761	814	pl->prog = NULL;
762	815	pl->link = NULL;
763	816
764		− err = update_effective_progs(cgrp, atype);
765		− if (err)
766		− goto cleanup;
	817	+ if (update_effective_progs(cgrp, atype)) {
	818	+ /* if update effective array failed replace the prog with a dummy prog*/
	819	+ pl->prog = old_prog;
	820	+ pl->link = link;
	821	+ purge_effective_progs(cgrp, old_prog, link, atype);
	822	+ }
767	823
768	824	/* now can actually delete it from this cgroup list */
769	825	list_del(&pl->node);
@@ -775,12 +831,6 @@ static int __cgroup_bpf_detach(struct cgroup cgrp, struct bpf_prog prog, · 5 unmodified lines
775	831	bpf_prog_put(old_prog);
776	832	static_branch_dec(&cgroup_bpf_enabled_key[atype]);
777	833	return 0;
778		−
779		−cleanup:
780		− /* restore back prog or link */
781		− pl->prog = old_prog;
782		− pl->link = link;
783		− return err;
784	834	}
785	835
786	836	static int cgroup_bpf_detach(struct cgroup cgrp, struct bpf_prog prog,

kernel/bpf/core.c+16−1

@@ -176,7 +176,7 @@ void bpf_prog_jit_attempt_done(struct bpf_prog *prog) · 175 unmodified lines
176	176	* here is relative to the prog itself instead of the main prog.
177	177	* This array has one entry for each xlated bpf insn.
178	178	*
179		− * jited_off is the byte off to the last byte of the jited insn.
	179	+ * jited_off is the byte off to the end of the jited insn.
180	180	*
181	181	* Hence, with
182	182	* insn_start:
@@ -2279,6 +2279,21 @@ void bpf_prog_array_free(struct bpf_prog_array *progs) · 2096 unmodified lines
2279	2279	kfree_rcu(progs, rcu);
2280	2280	}
2281	2281
	2282	+static void __bpf_prog_array_free_sleepable_cb(struct rcu_head *rcu)
	2283	+{
	2284	+ struct bpf_prog_array *progs;
	2285	+
	2286	+ progs = container_of(rcu, struct bpf_prog_array, rcu);
	2287	+ kfree_rcu(progs, rcu);
	2288	+}
	2289	+
	2290	+void bpf_prog_array_free_sleepable(struct bpf_prog_array *progs)
	2291	+{
	2292	+ if (!progs \|\| progs == &bpf_empty_prog_array.hdr)
	2293	+ return;
	2294	+ call_rcu_tasks_trace(&progs->rcu, __bpf_prog_array_free_sleepable_cb);
	2295	+}
	2296	+
2282	2297	int bpf_prog_array_length(struct bpf_prog_array *array)
2283	2298	{
2284	2299	struct bpf_prog_array_item *item;

kernel/bpf/helpers.c+6−6

@@ -584,7 +584,7 @@ BPF_CALL_3(bpf_strncmp, const char , s1, u32, s1_sz, const char , s2) · 583 unmodified lines
584	584	return strncmp(s1, s2, s1_sz);
585	585	}
586	586
587		−const struct bpf_func_proto bpf_strncmp_proto = {
	587	+static const struct bpf_func_proto bpf_strncmp_proto = {
588	588	.func = bpf_strncmp,
589	589	.gpl_only = false,
590	590	.ret_type = RET_INTEGER,
@@ -1402,7 +1402,7 @@ BPF_CALL_2(bpf_kptr_xchg, void , map_value, void , ptr) · 811 unmodified lines
1402	1402	*/
1403	1403	#define BPF_PTR_POISON ((void *)((0xeB9FUL << 2) + POISON_POINTER_DELTA))
1404	1404
1405		−const struct bpf_func_proto bpf_kptr_xchg_proto = {
	1405	+static const struct bpf_func_proto bpf_kptr_xchg_proto = {
1406	1406	.func = bpf_kptr_xchg,
1407	1407	.gpl_only = false,
1408	1408	.ret_type = RET_PTR_TO_BTF_ID_OR_NULL,
@@ -1487,7 +1487,7 @@ error: · 78 unmodified lines
1487	1487	return err;
1488	1488	}
1489	1489
1490		−const struct bpf_func_proto bpf_dynptr_from_mem_proto = {
	1490	+static const struct bpf_func_proto bpf_dynptr_from_mem_proto = {
1491	1491	.func = bpf_dynptr_from_mem,
1492	1492	.gpl_only = false,
1493	1493	.ret_type = RET_INTEGER,
@@ -1513,7 +1513,7 @@ BPF_CALL_4(bpf_dynptr_read, void , dst, u32, len, struct bpf_dynptr_kern , src · 19 unmodified lines
1513	1513	return 0;
1514	1514	}
1515	1515
1516		−const struct bpf_func_proto bpf_dynptr_read_proto = {
	1516	+static const struct bpf_func_proto bpf_dynptr_read_proto = {
1517	1517	.func = bpf_dynptr_read,
1518	1518	.gpl_only = false,
1519	1519	.ret_type = RET_INTEGER,
@@ -1539,7 +1539,7 @@ BPF_CALL_4(bpf_dynptr_write, struct bpf_dynptr_kern , dst, u32, offset, void , · 19 unmodified lines
1539	1539	return 0;
1540	1540	}
1541	1541
1542		−const struct bpf_func_proto bpf_dynptr_write_proto = {
	1542	+static const struct bpf_func_proto bpf_dynptr_write_proto = {
1543	1543	.func = bpf_dynptr_write,
1544	1544	.gpl_only = false,
1545	1545	.ret_type = RET_INTEGER,
@@ -1566,7 +1566,7 @@ BPF_CALL_3(bpf_dynptr_data, struct bpf_dynptr_kern *, ptr, u32, offset, u32, len · 20 unmodified lines
1566	1566	return (unsigned long)(ptr->data + ptr->offset + offset);
1567	1567	}
1568	1568
1569		−const struct bpf_func_proto bpf_dynptr_data_proto = {
	1569	+static const struct bpf_func_proto bpf_dynptr_data_proto = {
1570	1570	.func = bpf_dynptr_data,
1571	1571	.gpl_only = false,
1572	1572	.ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,

kernel/bpf/percpu_freelist.c+14−6

@@ -31,7 +31,7 @@ static inline void pcpu_freelist_push_node(struct pcpu_freelist_head *head, · 30 unmodified lines
31	31	struct pcpu_freelist_node *node)
32	32	{
33	33	node->next = head->first;
34		− head->first = node;
	34	+ WRITE_ONCE(head->first, node);
35	35	}
36	36
37	37	static inline void ___pcpu_freelist_push(struct pcpu_freelist_head *head,
@@ -130,14 +130,17 @@ static struct pcpu_freelist_node ___pcpu_freelist_pop(struct pcpu_freelist s) · 92 unmodified lines
130	130	orig_cpu = cpu = raw_smp_processor_id();
131	131	while (1) {
132	132	head = per_cpu_ptr(s->freelist, cpu);
	133	+ if (!READ_ONCE(head->first))
	134	+ goto next_cpu;
133	135	raw_spin_lock(&head->lock);
134	136	node = head->first;
135	137	if (node) {
136		− head->first = node->next;
	138	+ WRITE_ONCE(head->first, node->next);
137	139	raw_spin_unlock(&head->lock);
138	140	return node;
139	141	}
140	142	raw_spin_unlock(&head->lock);
	143	+next_cpu:
141	144	cpu = cpumask_next(cpu, cpu_possible_mask);
142	145	if (cpu >= nr_cpu_ids)
143	146	cpu = 0;
@@ -146,10 +149,12 @@ static struct pcpu_freelist_node ___pcpu_freelist_pop(struct pcpu_freelist s) · 2 unmodified lines
146	149	}
147	150
148	151	/* per cpu lists are all empty, try extralist */
	152	+ if (!READ_ONCE(s->extralist.first))
	153	+ return NULL;
149	154	raw_spin_lock(&s->extralist.lock);
150	155	node = s->extralist.first;
151	156	if (node)
152		− s->extralist.first = node->next;
	157	+ WRITE_ONCE(s->extralist.first, node->next);
153	158	raw_spin_unlock(&s->extralist.lock);
154	159	return node;
155	160	}
@@ -164,15 +169,18 @@ ___pcpu_freelist_pop_nmi(struct pcpu_freelist *s) · 8 unmodified lines
164	169	orig_cpu = cpu = raw_smp_processor_id();
165	170	while (1) {
166	171	head = per_cpu_ptr(s->freelist, cpu);
	172	+ if (!READ_ONCE(head->first))
	173	+ goto next_cpu;
167	174	if (raw_spin_trylock(&head->lock)) {
168	175	node = head->first;
169	176	if (node) {
170		− head->first = node->next;
	177	+ WRITE_ONCE(head->first, node->next);
171	178	raw_spin_unlock(&head->lock);
172	179	return node;
173	180	}
174	181	raw_spin_unlock(&head->lock);
175	182	}
	183	+next_cpu:
176	184	cpu = cpumask_next(cpu, cpu_possible_mask);
177	185	if (cpu >= nr_cpu_ids)
178	186	cpu = 0;
@@ -181,11 +189,11 @@ ___pcpu_freelist_pop_nmi(struct pcpu_freelist *s) · 2 unmodified lines
181	189	}
182	190
183	191	/* cannot pop from per cpu lists, try extralist */
184		− if (!raw_spin_trylock(&s->extralist.lock))
	192	+ if (!READ_ONCE(s->extralist.first) \|\| !raw_spin_trylock(&s->extralist.lock))
185	193	return NULL;
186	194	node = s->extralist.first;
187	195	if (node)
188		− s->extralist.first = node->next;
	196	+ WRITE_ONCE(s->extralist.first, node->next);
189	197	raw_spin_unlock(&s->extralist.lock);
190	198	return node;
191	199	}

kernel/bpf/syscall.c+4−3

@@ -4090,14 +4090,15 @@ static int bpf_prog_get_info_by_fd(struct file *file, · 4089 unmodified lines
4090	4090	info.nr_jited_line_info = 0;
4091	4091	if (info.nr_jited_line_info && ulen) {
4092	4092	if (bpf_dump_raw_ok(file->f_cred)) {
	4093	+ unsigned long line_addr;
4093	4094	__u64 __user *user_linfo;
4094	4095	u32 i;
4095	4096
4096	4097	user_linfo = u64_to_user_ptr(info.jited_line_info);
4097	4098	ulen = min_t(u32, info.nr_jited_line_info, ulen);
4098	4099	for (i = 0; i < ulen; i++) {
4099		− if (put_user((__u64)(long)prog->aux->jited_linfo[i],
4100		− &user_linfo[i]))
	4100	+ line_addr = (unsigned long)prog->aux->jited_linfo[i];
	4101	+ if (put_user((__u64)line_addr, &user_linfo[i]))
4101	4102	return -EFAULT;
4102	4103	}
4103	4104	} else {
@@ -5130,7 +5131,7 @@ BPF_CALL_4(bpf_kallsyms_lookup_name, const char *, name, int, name_sz, int, flag · 1026 unmodified lines
5130	5131	return *res ? 0 : -ENOENT;
5131	5132	}
5132	5133
5133		−const struct bpf_func_proto bpf_kallsyms_lookup_name_proto = {
	5134	+static const struct bpf_func_proto bpf_kallsyms_lookup_name_proto = {
5134	5135	.func = bpf_kallsyms_lookup_name,
5135	5136	.gpl_only = false,
5136	5137	.ret_type = RET_INTEGER,

kernel/bpf/verifier.c+35−14

@@ -5848,6 +5848,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 5847 unmodified lines
5848	5848	struct bpf_reg_state regs = cur_regs(env), reg = &regs[regno];
5849	5849	enum bpf_arg_type arg_type = fn->arg_type[arg];
5850	5850	enum bpf_reg_type type = reg->type;
	5851	+ u32 *arg_btf_id = NULL;
5851	5852	int err = 0;
5852	5853
5853	5854	if (arg_type == ARG_DONTCARE)
@@ -5884,7 +5885,11 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 30 unmodified lines
5884	5885	*/
5885	5886	goto skip_type_check;
5886	5887
5887		− err = check_reg_type(env, regno, arg_type, fn->arg_btf_id[arg], meta);
	5888	+ /* arg_btf_id and arg_size are in a union. */
	5889	+ if (base_type(arg_type) == ARG_PTR_TO_BTF_ID)
	5890	+ arg_btf_id = fn->arg_btf_id[arg];
	5891	+
	5892	+ err = check_reg_type(env, regno, arg_type, arg_btf_id, meta);
5888	5893	if (err)
5889	5894	return err;
5890	5895
@@ -6011,6 +6016,11 @@ skip_type_check: · 120 unmodified lines
6011	6016	* next is_mem_size argument below.
6012	6017	*/
6013	6018	meta->raw_mode = arg_type & MEM_UNINIT;
	6019	+ if (arg_type & MEM_FIXED_SIZE) {
	6020	+ err = check_helper_mem_access(env, regno,
	6021	+ fn->arg_size[arg], false,
	6022	+ meta);
	6023	+ }
6014	6024	} else if (arg_type_is_mem_size(arg_type)) {
6015	6025	bool zero_size_allowed = (arg_type == ARG_CONST_SIZE_OR_ZERO);
6016	6026
@@ -6400,11 +6410,19 @@ static bool check_raw_mode_ok(const struct bpf_func_proto *fn) · 383 unmodified lines
6400	6410	return count <= 1;
6401	6411	}
6402	6412
6403		−static bool check_args_pair_invalid(enum bpf_arg_type arg_curr,
6404		− enum bpf_arg_type arg_next)
	6413	+static bool check_args_pair_invalid(const struct bpf_func_proto *fn, int arg)
6405	6414	{
6406		− return (base_type(arg_curr) == ARG_PTR_TO_MEM) !=
6407		− arg_type_is_mem_size(arg_next);
	6415	+ bool is_fixed = fn->arg_type[arg] & MEM_FIXED_SIZE;
	6416	+ bool has_size = fn->arg_size[arg] != 0;
	6417	+ bool is_next_size = false;
	6418	+
	6419	+ if (arg + 1 < ARRAY_SIZE(fn->arg_type))
	6420	+ is_next_size = arg_type_is_mem_size(fn->arg_type[arg + 1]);
	6421	+
	6422	+ if (base_type(fn->arg_type[arg]) != ARG_PTR_TO_MEM)
	6423	+ return is_next_size;
	6424	+
	6425	+ return has_size == is_next_size \|\| is_next_size == is_fixed;
6408	6426	}
6409	6427
6410	6428	static bool check_arg_pair_ok(const struct bpf_func_proto *fn)
@@ -6415,11 +6433,11 @@ static bool check_arg_pair_ok(const struct bpf_func_proto *fn) · 4 unmodified lines
6415	6433	* helper function specification.
6416	6434	*/
6417	6435	if (arg_type_is_mem_size(fn->arg1_type) \|\|
6418		− base_type(fn->arg5_type) == ARG_PTR_TO_MEM \|\|
6419		− check_args_pair_invalid(fn->arg1_type, fn->arg2_type) \|\|
6420		− check_args_pair_invalid(fn->arg2_type, fn->arg3_type) \|\|
6421		− check_args_pair_invalid(fn->arg3_type, fn->arg4_type) \|\|
6422		− check_args_pair_invalid(fn->arg4_type, fn->arg5_type))
	6436	+ check_args_pair_invalid(fn, 0) \|\|
	6437	+ check_args_pair_invalid(fn, 1) \|\|
	6438	+ check_args_pair_invalid(fn, 2) \|\|
	6439	+ check_args_pair_invalid(fn, 3) \|\|
	6440	+ check_args_pair_invalid(fn, 4))
6423	6441	return false;
6424	6442
6425	6443	return true;
@@ -6460,7 +6478,10 @@ static bool check_btf_id_ok(const struct bpf_func_proto *fn) · 34 unmodified lines
6460	6478	if (base_type(fn->arg_type[i]) == ARG_PTR_TO_BTF_ID && !fn->arg_btf_id[i])
6461	6479	return false;
6462	6480
6463		− if (base_type(fn->arg_type[i]) != ARG_PTR_TO_BTF_ID && fn->arg_btf_id[i])
	6481	+ if (base_type(fn->arg_type[i]) != ARG_PTR_TO_BTF_ID && fn->arg_btf_id[i] &&
	6482	+ /* arg_btf_id and arg_size are in a union. */
	6483	+ (base_type(fn->arg_type[i]) != ARG_PTR_TO_MEM \|\|
	6484	+ !(fn->arg_type[i] & MEM_FIXED_SIZE)))
6464	6485	return false;
6465	6486	}
6466	6487
@@ -10901,7 +10922,7 @@ static int check_btf_func(struct bpf_verifier_env *env, · 4434 unmodified lines
10901	10922	goto err_free;
10902	10923	ret_type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
10903	10924	scalar_return =
10904		− btf_type_is_small_int(ret_type) \|\| btf_type_is_enum(ret_type);
	10925	+ btf_type_is_small_int(ret_type) \|\| btf_is_any_enum(ret_type);
10905	10926	if (i && !scalar_return && env->subprog_info[i].has_ld_abs) {
10906	10927	verbose(env, "LD_ABS is only allowed in functions that return 'int'.\n");
10907	10928	goto err_free;
@@ -14829,8 +14850,8 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 3921 unmodified lines
14829	14850	}
14830	14851
14831	14852	if (prog->aux->sleepable && prog->type != BPF_PROG_TYPE_TRACING &&
14832		− prog->type != BPF_PROG_TYPE_LSM) {
14833		− verbose(env, "Only fentry/fexit/fmod_ret and lsm programs can be sleepable\n");
	14853	+ prog->type != BPF_PROG_TYPE_LSM && prog->type != BPF_PROG_TYPE_KPROBE) {
	14854	+ verbose(env, "Only fentry/fexit/fmod_ret, lsm, and kprobe/uprobe programs can be sleepable\n");
14834	14855	return -EINVAL;
14835	14856	}
14836	14857

kernel/events/core.c+10−6

@@ -10068,26 +10068,30 @@ static inline bool perf_event_is_tracing(struct perf_event *event) · 10067 unmodified lines
10068	10068	int perf_event_set_bpf_prog(struct perf_event event, struct bpf_prog prog,
10069	10069	u64 bpf_cookie)
10070	10070	{
10071		− bool is_kprobe, is_tracepoint, is_syscall_tp;
	10071	+ bool is_kprobe, is_uprobe, is_tracepoint, is_syscall_tp;
10072	10072
10073	10073	if (!perf_event_is_tracing(event))
10074	10074	return perf_event_set_bpf_handler(event, prog, bpf_cookie);
10075	10075
10076		− is_kprobe = event->tp_event->flags & TRACE_EVENT_FL_UKPROBE;
	10076	+ is_kprobe = event->tp_event->flags & TRACE_EVENT_FL_KPROBE;
	10077	+ is_uprobe = event->tp_event->flags & TRACE_EVENT_FL_UPROBE;
10077	10078	is_tracepoint = event->tp_event->flags & TRACE_EVENT_FL_TRACEPOINT;
10078	10079	is_syscall_tp = is_syscall_trace_event(event->tp_event);
10079		− if (!is_kprobe && !is_tracepoint && !is_syscall_tp)
	10080	+ if (!is_kprobe && !is_uprobe && !is_tracepoint && !is_syscall_tp)
10080	10081	/* bpf programs can only be attached to u/kprobe or tracepoint */
10081	10082	return -EINVAL;
10082	10083
10083		− if ((is_kprobe && prog->type != BPF_PROG_TYPE_KPROBE) \|\|
	10084	+ if (((is_kprobe \|\| is_uprobe) && prog->type != BPF_PROG_TYPE_KPROBE) \|\|
10084	10085	(is_tracepoint && prog->type != BPF_PROG_TYPE_TRACEPOINT) \|\|
10085	10086	(is_syscall_tp && prog->type != BPF_PROG_TYPE_TRACEPOINT))
10086	10087	return -EINVAL;
10087	10088
	10089	+ if (prog->type == BPF_PROG_TYPE_KPROBE && prog->aux->sleepable && !is_uprobe)
	10090	+ /* only uprobe programs are allowed to be sleepable */
	10091	+ return -EINVAL;
	10092	+
10088	10093	/* Kprobe override only works for kprobes, not uprobes. */
10089		− if (prog->kprobe_override &&
10090		− !(event->tp_event->flags & TRACE_EVENT_FL_KPROBE))
	10094	+ if (prog->kprobe_override && !is_kprobe)
10091	10095	return -EINVAL;
10092	10096
10093	10097	if (is_tracepoint \|\| is_syscall_tp) {

kernel/trace/bpf_trace.c+2−2

@@ -1936,7 +1936,7 @@ int perf_event_attach_bpf_prog(struct perf_event *event, · 1935 unmodified lines
1936	1936	event->prog = prog;
1937	1937	event->bpf_cookie = bpf_cookie;
1938	1938	rcu_assign_pointer(event->tp_event->prog_array, new_array);
1939		− bpf_prog_array_free(old_array);
	1939	+ bpf_prog_array_free_sleepable(old_array);
1940	1940
1941	1941	unlock:
1942	1942	mutex_unlock(&bpf_event_mutex);
@@ -1962,7 +1962,7 @@ void perf_event_detach_bpf_prog(struct perf_event *event) · 19 unmodified lines
1962	1962	bpf_prog_array_delete_safe(old_array, event->prog);
1963	1963	} else {
1964	1964	rcu_assign_pointer(event->tp_event->prog_array, new_array);
1965		− bpf_prog_array_free(old_array);
	1965	+ bpf_prog_array_free_sleepable(old_array);
1966	1966	}
1967	1967
1968	1968	bpf_prog_put(event->prog);

kernel/trace/trace_uprobe.c+2−3

@@ -16,6 +16,7 @@ · 15 unmodified lines
16	16	#include <linux/namei.h>
17	17	#include <linux/string.h>
18	18	#include <linux/rculist.h>
	19	+#include <linux/filter.h>
19	20
20	21	#include "trace_dynevent.h"
21	22	#include "trace_probe.h"
@@ -1346,9 +1347,7 @@ static void __uprobe_perf_func(struct trace_uprobe *tu, · 1324 unmodified lines
1346	1347	if (bpf_prog_array_valid(call)) {
1347	1348	u32 ret;
1348	1349
1349		− preempt_disable();
1350		− ret = trace_call_bpf(call, regs);
1351		− preempt_enable();
	1350	+ ret = bpf_prog_run_array_sleepable(call->prog_array, regs, bpf_prog_run);
1352	1351	if (!ret)
1353	1352	return;
1354	1353	}

net/bpf/test_run.c+0−6

@@ -1420,9 +1420,6 @@ int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog, · 1419 unmodified lines
1420	1420	void *data;
1421	1421	int ret;
1422	1422
1423		− if (prog->type != BPF_PROG_TYPE_FLOW_DISSECTOR)
1424		− return -EINVAL;
1425		−
1426	1423	if (kattr->test.flags \|\| kattr->test.cpu \|\| kattr->test.batch_size)
1427	1424	return -EINVAL;
1428	1425
@@ -1487,9 +1484,6 @@ int bpf_prog_test_run_sk_lookup(struct bpf_prog prog, const union bpf_attr kat · 58 unmodified lines
1487	1484	u32 retval, duration;
1488	1485	int ret = -EINVAL;
1489	1486
1490		− if (prog->type != BPF_PROG_TYPE_SK_LOOKUP)
1491		− return -EINVAL;
1492		−
1493	1487	if (kattr->test.flags \|\| kattr->test.cpu \|\| kattr->test.batch_size)
1494	1488	return -EINVAL;
1495	1489

net/core/filter.c+128−2

@@ -6463,8 +6463,6 @@ static struct sock sk_lookup(struct net net, struct bpf_sock_tuple *tuple, · 6462 unmodified lines
6463	6463
6464	6464	/* bpf_skc_lookup performs the core lookup for different types of sockets,
6465	6465	* taking a reference on the socket if it doesn't have the flag SOCK_RCU_FREE.
6466		− * Returns the socket as an 'unsigned long' to simplify the casting in the
6467		− * callers to satisfy BPF_CALL declarations.
6468	6466	*/
6469	6467	static struct sock *
6470	6468	__bpf_skc_lookup(struct sk_buff skb, struct bpf_sock_tuple tuple, u32 len,
@@ -7444,6 +7442,114 @@ static const struct bpf_func_proto bpf_skb_set_tstamp_proto = { · 973 unmodified lines
7444	7442	.arg3_type = ARG_ANYTHING,
7445	7443	};
7446	7444
	7445	+#ifdef CONFIG_SYN_COOKIES
	7446	+BPF_CALL_3(bpf_tcp_raw_gen_syncookie_ipv4, struct iphdr *, iph,
	7447	+ struct tcphdr *, th, u32, th_len)
	7448	+{
	7449	+ u32 cookie;
	7450	+ u16 mss;
	7451	+
	7452	+ if (unlikely(th_len < sizeof(th) \|\| th_len != th->doff 4))
	7453	+ return -EINVAL;
	7454	+
	7455	+ mss = tcp_parse_mss_option(th, 0) ?: TCP_MSS_DEFAULT;
	7456	+ cookie = __cookie_v4_init_sequence(iph, th, &mss);
	7457	+
	7458	+ return cookie \| ((u64)mss << 32);
	7459	+}
	7460	+
	7461	+static const struct bpf_func_proto bpf_tcp_raw_gen_syncookie_ipv4_proto = {
	7462	+ .func = bpf_tcp_raw_gen_syncookie_ipv4,
	7463	+ .gpl_only = true, /* __cookie_v4_init_sequence() is GPL */
	7464	+ .pkt_access = true,
	7465	+ .ret_type = RET_INTEGER,
	7466	+ .arg1_type = ARG_PTR_TO_FIXED_SIZE_MEM,
	7467	+ .arg1_size = sizeof(struct iphdr),
	7468	+ .arg2_type = ARG_PTR_TO_MEM,
	7469	+ .arg3_type = ARG_CONST_SIZE,
	7470	+};
	7471	+
	7472	+BPF_CALL_3(bpf_tcp_raw_gen_syncookie_ipv6, struct ipv6hdr *, iph,
	7473	+ struct tcphdr *, th, u32, th_len)
	7474	+{
	7475	+#if IS_BUILTIN(CONFIG_IPV6)
	7476	+ const u16 mss_clamp = IPV6_MIN_MTU - sizeof(struct tcphdr) -
	7477	+ sizeof(struct ipv6hdr);
	7478	+ u32 cookie;
	7479	+ u16 mss;
	7480	+
	7481	+ if (unlikely(th_len < sizeof(th) \|\| th_len != th->doff 4))
	7482	+ return -EINVAL;
	7483	+
	7484	+ mss = tcp_parse_mss_option(th, 0) ?: mss_clamp;
	7485	+ cookie = __cookie_v6_init_sequence(iph, th, &mss);
	7486	+
	7487	+ return cookie \| ((u64)mss << 32);
	7488	+#else
	7489	+ return -EPROTONOSUPPORT;
	7490	+#endif
	7491	+}
	7492	+
	7493	+static const struct bpf_func_proto bpf_tcp_raw_gen_syncookie_ipv6_proto = {
	7494	+ .func = bpf_tcp_raw_gen_syncookie_ipv6,
	7495	+ .gpl_only = true, /* __cookie_v6_init_sequence() is GPL */
	7496	+ .pkt_access = true,
	7497	+ .ret_type = RET_INTEGER,
	7498	+ .arg1_type = ARG_PTR_TO_FIXED_SIZE_MEM,
	7499	+ .arg1_size = sizeof(struct ipv6hdr),
	7500	+ .arg2_type = ARG_PTR_TO_MEM,
	7501	+ .arg3_type = ARG_CONST_SIZE,
	7502	+};
	7503	+
	7504	+BPF_CALL_2(bpf_tcp_raw_check_syncookie_ipv4, struct iphdr *, iph,
	7505	+ struct tcphdr *, th)
	7506	+{
	7507	+ u32 cookie = ntohl(th->ack_seq) - 1;
	7508	+
	7509	+ if (__cookie_v4_check(iph, th, cookie) > 0)
	7510	+ return 0;
	7511	+
	7512	+ return -EACCES;
	7513	+}
	7514	+
	7515	+static const struct bpf_func_proto bpf_tcp_raw_check_syncookie_ipv4_proto = {
	7516	+ .func = bpf_tcp_raw_check_syncookie_ipv4,
	7517	+ .gpl_only = true, /* __cookie_v4_check is GPL */
	7518	+ .pkt_access = true,
	7519	+ .ret_type = RET_INTEGER,
	7520	+ .arg1_type = ARG_PTR_TO_FIXED_SIZE_MEM,
	7521	+ .arg1_size = sizeof(struct iphdr),
	7522	+ .arg2_type = ARG_PTR_TO_FIXED_SIZE_MEM,
	7523	+ .arg2_size = sizeof(struct tcphdr),
	7524	+};
	7525	+
	7526	+BPF_CALL_2(bpf_tcp_raw_check_syncookie_ipv6, struct ipv6hdr *, iph,
	7527	+ struct tcphdr *, th)
	7528	+{
	7529	+#if IS_BUILTIN(CONFIG_IPV6)
	7530	+ u32 cookie = ntohl(th->ack_seq) - 1;
	7531	+
	7532	+ if (__cookie_v6_check(iph, th, cookie) > 0)
	7533	+ return 0;
	7534	+
	7535	+ return -EACCES;
	7536	+#else
	7537	+ return -EPROTONOSUPPORT;
	7538	+#endif
	7539	+}
	7540	+
	7541	+static const struct bpf_func_proto bpf_tcp_raw_check_syncookie_ipv6_proto = {
	7542	+ .func = bpf_tcp_raw_check_syncookie_ipv6,
	7543	+ .gpl_only = true, /* __cookie_v6_check is GPL */
	7544	+ .pkt_access = true,
	7545	+ .ret_type = RET_INTEGER,
	7546	+ .arg1_type = ARG_PTR_TO_FIXED_SIZE_MEM,
	7547	+ .arg1_size = sizeof(struct ipv6hdr),
	7548	+ .arg2_type = ARG_PTR_TO_FIXED_SIZE_MEM,
	7549	+ .arg2_size = sizeof(struct tcphdr),
	7550	+};
	7551	+#endif /* CONFIG_SYN_COOKIES */
	7552	+
7447	7553	#endif /* CONFIG_INET */
7448	7554
7449	7555	bool bpf_helper_changes_pkt_data(void *func)
@@ -7807,6 +7913,16 @@ tc_cls_act_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 357 unmodified lines
7807	7913	return &bpf_sk_assign_proto;
7808	7914	case BPF_FUNC_skb_set_tstamp:
7809	7915	return &bpf_skb_set_tstamp_proto;
	7916	+#ifdef CONFIG_SYN_COOKIES
	7917	+ case BPF_FUNC_tcp_raw_gen_syncookie_ipv4:
	7918	+ return &bpf_tcp_raw_gen_syncookie_ipv4_proto;
	7919	+ case BPF_FUNC_tcp_raw_gen_syncookie_ipv6:
	7920	+ return &bpf_tcp_raw_gen_syncookie_ipv6_proto;
	7921	+ case BPF_FUNC_tcp_raw_check_syncookie_ipv4:
	7922	+ return &bpf_tcp_raw_check_syncookie_ipv4_proto;
	7923	+ case BPF_FUNC_tcp_raw_check_syncookie_ipv6:
	7924	+ return &bpf_tcp_raw_check_syncookie_ipv6_proto;
	7925	+#endif
7810	7926	#endif
7811	7927	default:
7812	7928	return bpf_sk_base_func_proto(func_id);
@@ -7856,6 +7972,16 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 43 unmodified lines
7856	7972	return &bpf_tcp_check_syncookie_proto;
7857	7973	case BPF_FUNC_tcp_gen_syncookie:
7858	7974	return &bpf_tcp_gen_syncookie_proto;
	7975	+#ifdef CONFIG_SYN_COOKIES
	7976	+ case BPF_FUNC_tcp_raw_gen_syncookie_ipv4:
	7977	+ return &bpf_tcp_raw_gen_syncookie_ipv4_proto;
	7978	+ case BPF_FUNC_tcp_raw_gen_syncookie_ipv6:
	7979	+ return &bpf_tcp_raw_gen_syncookie_ipv6_proto;
	7980	+ case BPF_FUNC_tcp_raw_check_syncookie_ipv4:
	7981	+ return &bpf_tcp_raw_check_syncookie_ipv4_proto;
	7982	+ case BPF_FUNC_tcp_raw_check_syncookie_ipv6:
	7983	+ return &bpf_tcp_raw_check_syncookie_ipv6_proto;
	7984	+#endif
7859	7985	#endif
7860	7986	default:
7861	7987	return bpf_sk_base_func_proto(func_id);

net/core/skmsg.c+1−0

@@ -715,6 +715,7 @@ struct sk_psock sk_psock_init(struct sock sk, int node) · 714 unmodified lines
715	715	psock->eval = __SK_NONE;
716	716	psock->sk_proto = prot;
717	717	psock->saved_unhash = prot->unhash;
	718	+ psock->saved_destroy = prot->destroy;
718	719	psock->saved_close = prot->close;
719	720	psock->saved_write_space = sk->sk_write_space;
720	721

net/core/sock_map.c+23−0

@@ -1561,6 +1561,29 @@ void sock_map_unhash(struct sock *sk) · 1560 unmodified lines
1561	1561	}
1562	1562	EXPORT_SYMBOL_GPL(sock_map_unhash);
1563	1563
	1564	+void sock_map_destroy(struct sock *sk)
	1565	+{
	1566	+ void (saved_destroy)(struct sock sk);
	1567	+ struct sk_psock *psock;
	1568	+
	1569	+ rcu_read_lock();
	1570	+ psock = sk_psock_get(sk);
	1571	+ if (unlikely(!psock)) {
	1572	+ rcu_read_unlock();
	1573	+ if (sk->sk_prot->destroy)
	1574	+ sk->sk_prot->destroy(sk);
	1575	+ return;
	1576	+ }
	1577	+
	1578	+ saved_destroy = psock->saved_destroy;
	1579	+ sock_map_remove_links(sk, psock);
	1580	+ rcu_read_unlock();
	1581	+ sk_psock_stop(psock, true);
	1582	+ sk_psock_put(sk, psock);
	1583	+ saved_destroy(sk);
	1584	+}
	1585	+EXPORT_SYMBOL_GPL(sock_map_destroy);
	1586	+
1564	1587	void sock_map_close(struct sock *sk, long timeout)
1565	1588	{
1566	1589	void (saved_close)(struct sock sk, long timeout);

net/ipv4/tcp_bpf.c+1−0

@@ -540,6 +540,7 @@ static void tcp_bpf_rebuild_protos(struct proto prot[TCP_BPF_NUM_CFGS], · 539 unmodified lines
540	540	struct proto *base)
541	541	{
542	542	prot[TCP_BPF_BASE] = *base;
	543	+ prot[TCP_BPF_BASE].destroy = sock_map_destroy;
543	544	prot[TCP_BPF_BASE].close = sock_map_close;
544	545	prot[TCP_BPF_BASE].recvmsg = tcp_bpf_recvmsg;
545	546	prot[TCP_BPF_BASE].sock_is_readable = sk_msg_is_readable;

net/ipv4/tcp_input.c+2−1

@@ -3966,7 +3966,7 @@ static bool smc_parse_options(const struct tcphdr *th, · 3965 unmodified lines
3966	3966	/* Try to parse the MSS option from the TCP header. Return 0 on failure, clamped
3967	3967	* value on success.
3968	3968	*/
3969		−static u16 tcp_parse_mss_option(const struct tcphdr *th, u16 user_mss)
	3969	+u16 tcp_parse_mss_option(const struct tcphdr *th, u16 user_mss)
3970	3970	{
3971	3971	const unsigned char ptr = (const unsigned char )(th + 1);
3972	3972	int length = (th->doff * 4) - sizeof(struct tcphdr);
@@ -4005,6 +4005,7 @@ static u16 tcp_parse_mss_option(const struct tcphdr *th, u16 user_mss) · 32 unmodified lines
4005	4005	}
4006	4006	return mss;
4007	4007	}
	4008	+EXPORT_SYMBOL_GPL(tcp_parse_mss_option);
4008	4009
4009	4010	/* Look for tcp options. Normally only called on SYN and SYNACK packets.
4010	4011	* But, this can also be called on packets in the established flow when

net/xdp/xdp_umem.c+3−3

@@ -57,7 +57,7 @@ static int xdp_umem_addr_map(struct xdp_umem umem, struct page *pages, · 56 unmodified lines
57	57	static void xdp_umem_release(struct xdp_umem *umem)
58	58	{
59	59	umem->zc = false;
60		− ida_simple_remove(&umem_ida, umem->id);
	60	+ ida_free(&umem_ida, umem->id);
61	61
62	62	xdp_umem_addr_unmap(umem);
63	63	xdp_umem_unpin_pages(umem);
@@ -242,7 +242,7 @@ struct xdp_umem xdp_umem_create(struct xdp_umem_reg mr) · 178 unmodified lines
242	242	if (!umem)
243	243	return ERR_PTR(-ENOMEM);
244	244
245		− err = ida_simple_get(&umem_ida, 0, 0, GFP_KERNEL);
	245	+ err = ida_alloc(&umem_ida, GFP_KERNEL);
246	246	if (err < 0) {
247	247	kfree(umem);
248	248	return ERR_PTR(err);
@@ -251,7 +251,7 @@ struct xdp_umem xdp_umem_create(struct xdp_umem_reg mr) · 2 unmodified lines
251	251
252	252	err = xdp_umem_reg(umem, mr);
253	253	if (err) {
254		− ida_simple_remove(&umem_ida, umem->id);
	254	+ ida_free(&umem_ida, umem->id);
255	255	kfree(umem);
256	256	return ERR_PTR(err);
257	257	}

samples/bpf/xdp_fwd_user.c+49−6

@@ -47,17 +47,60 @@ static int do_attach(int idx, int prog_fd, int map_fd, const char *name) · 46 unmodified lines
47	47	return err;
48	48	}
49	49
50		−static int do_detach(int idx, const char *name)
	50	+static int do_detach(int ifindex, const char ifname, const char app_name)
51	51	{
52		− int err;
	52	+ LIBBPF_OPTS(bpf_xdp_attach_opts, opts);
	53	+ struct bpf_prog_info prog_info = {};
	54	+ char prog_name[BPF_OBJ_NAME_LEN];
	55	+ __u32 info_len, curr_prog_id;
	56	+ int prog_fd;
	57	+ int err = 1;
	58	+
	59	+ if (bpf_xdp_query_id(ifindex, xdp_flags, &curr_prog_id)) {
	60	+ printf("ERROR: bpf_xdp_query_id failed (%s)\n",
	61	+ strerror(errno));
	62	+ return err;
	63	+ }
53	64
54		− err = bpf_xdp_detach(idx, xdp_flags, NULL);
55		− if (err < 0)
56		− printf("ERROR: failed to detach program from %s\n", name);
	65	+ if (!curr_prog_id) {
	66	+ printf("ERROR: flags(0x%x) xdp prog is not attached to %s\n",
	67	+ xdp_flags, ifname);
	68	+ return err;
	69	+ }
57	70
	71	+ info_len = sizeof(prog_info);
	72	+ prog_fd = bpf_prog_get_fd_by_id(curr_prog_id);
	73	+ if (prog_fd < 0) {
	74	+ printf("ERROR: bpf_prog_get_fd_by_id failed (%s)\n",
	75	+ strerror(errno));
	76	+ return prog_fd;
	77	+ }
	78	+
	79	+ err = bpf_obj_get_info_by_fd(prog_fd, &prog_info, &info_len);
	80	+ if (err) {
	81	+ printf("ERROR: bpf_obj_get_info_by_fd failed (%s)\n",
	82	+ strerror(errno));
	83	+ goto close_out;
	84	+ }
	85	+ snprintf(prog_name, sizeof(prog_name), "%s_prog", app_name);
	86	+ prog_name[BPF_OBJ_NAME_LEN - 1] = '\0';
	87	+
	88	+ if (strcmp(prog_info.name, prog_name)) {
	89	+ printf("ERROR: %s isn't attached to %s\n", app_name, ifname);
	90	+ err = 1;
	91	+ goto close_out;
	92	+ }
	93	+
	94	+ opts.old_prog_fd = prog_fd;
	95	+ err = bpf_xdp_detach(ifindex, xdp_flags, &opts);
	96	+ if (err < 0)
	97	+ printf("ERROR: failed to detach program from %s (%s)\n",
	98	+ ifname, strerror(errno));
58	99	/* TODO: Remember to cleanup map, when adding use of shared map
59	100	* bpf_map_delete_elem((map_fd, &idx);
60	101	*/
	102	+close_out:
	103	+ close(prog_fd);
61	104	return err;
62	105	}
63	106
@@ -169,7 +212,7 @@ int main(int argc, char **argv) · 105 unmodified lines
169	212	return 1;
170	213	}
171	214	if (!attach) {
172		− err = do_detach(idx, argv[i]);
	215	+ err = do_detach(idx, argv[i], prog_name);
173	216	if (err)
174	217	ret = err;
175	218	} else {

samples/bpf/xdp_router_ipv4.bpf.c+9−0

@@ -150,6 +150,15 @@ int xdp_router_ipv4_prog(struct xdp_md *ctx) · 149 unmodified lines
150	150
151	151	dest_mac = bpf_map_lookup_elem(&arp_table,
152	152	&prefix_value->gw);
	153	+ if (!dest_mac) {
	154	+ /* Forward the packet to the kernel in
	155	+ * order to trigger ARP discovery for
	156	+ * the default gw.
	157	+ */
	158	+ if (rec)
	159	+ NO_TEAR_INC(rec->xdp_pass);
	160	+ return XDP_PASS;
	161	+ }
153	162	}
154	163	}
155	164

scripts/bpf_doc.py+4−0

@@ -635,6 +635,8 @@ class PrinterHelpers(Printer): · 634 unmodified lines
635	635	'struct bpf_timer',
636	636	'struct mptcp_sock',
637	637	'struct bpf_dynptr',
	638	+ 'struct iphdr',
	639	+ 'struct ipv6hdr',
638	640	]
639	641	known_types = {
640	642	'...',
@@ -686,6 +688,8 @@ class PrinterHelpers(Printer): · 45 unmodified lines
686	688	'struct bpf_timer',
687	689	'struct mptcp_sock',
688	690	'struct bpf_dynptr',
	691	+ 'struct iphdr',
	692	+ 'struct ipv6hdr',
689	693	}
690	694	mapped_types = {
691	695	'u8': '__u8',

tools/bpf/bpftool/Documentation/bpftool-cgroup.rst+11−5

@@ -31,11 +31,17 @@ CGROUP COMMANDS · 30 unmodified lines
31	31	\| bpftool cgroup help
32	32	\|
33	33	\| PROG := { id PROG_ID \| pinned FILE \| tag PROG_TAG }
34		−\| ATTACH_TYPE := { ingress \| egress \| sock_create \| sock_ops \| device \|
35		−\| bind4 \| bind6 \| post_bind4 \| post_bind6 \| connect4 \| connect6 \|
36		−\| getpeername4 \| getpeername6 \| getsockname4 \| getsockname6 \| sendmsg4 \|
37		−\| sendmsg6 \| recvmsg4 \| recvmsg6 \| sysctl \| getsockopt \| setsockopt \|
38		−\| sock_release }
	34	+\| ATTACH_TYPE := { cgroup_inet_ingress \| cgroup_inet_egress \|
	35	+\| cgroup_inet_sock_create \| cgroup_sock_ops \|
	36	+\| cgroup_device \| cgroup_inet4_bind \| cgroup_inet6_bind \|
	37	+\| cgroup_inet4_post_bind \| cgroup_inet6_post_bind \|
	38	+\| cgroup_inet4_connect \| cgroup_inet6_connect \|
	39	+\| cgroup_inet4_getpeername \| cgroup_inet6_getpeername \|
	40	+\| cgroup_inet4_getsockname \| cgroup_inet6_getsockname \|
	41	+\| cgroup_udp4_sendmsg \| cgroup_udp6_sendmsg \|
	42	+\| cgroup_udp4_recvmsg \| cgroup_udp6_recvmsg \|
	43	+\| cgroup_sysctl \| cgroup_getsockopt \| cgroup_setsockopt \|
	44	+\| cgroup_inet_sock_release }
39	45	\| ATTACH_FLAGS := { multi \| override }
40	46
41	47	DESCRIPTION

tools/bpf/bpftool/Documentation/bpftool-prog.rst+3−2

@@ -53,8 +53,9 @@ PROG COMMANDS · 52 unmodified lines
53	53	\| cgroup/getsockopt \| cgroup/setsockopt \| cgroup/sock_release \|
54	54	\| struct_ops \| fentry \| fexit \| freplace \| sk_lookup
55	55	\| }
56		−\| ATTACH_TYPE := {
57		−\| msg_verdict \| skb_verdict \| stream_verdict \| stream_parser \| flow_dissector
	56	+\| ATTACH_TYPE := {
	57	+\| sk_msg_verdict \| sk_skb_verdict \| sk_skb_stream_verdict \|
	58	+\| sk_skb_stream_parser \| flow_dissector
58	59	\| }
59	60	\| METRICs := {
60	61	\| cycles \| instructions \| l1d_loads \| llc_misses \|

tools/bpf/bpftool/Makefile+1−1

@@ -53,7 +53,7 @@ $(LIBBPF_INTERNAL_HDRS): $(LIBBPF_HDRS_DIR)/%.h: $(BPF_DIR)/%.h \| $(LIBBPF_HDRS_ · 52 unmodified lines
53	53	$(LIBBPF_BOOTSTRAP): $(wildcard $(BPF_DIR)/*.[ch] $(BPF_DIR)/Makefile) \| $(LIBBPF_BOOTSTRAP_OUTPUT)
54	54	$(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_BOOTSTRAP_OUTPUT) \
55	55	DESTDIR=$(LIBBPF_BOOTSTRAP_DESTDIR:/=) prefix= \
56		− ARCH= CROSS_COMPILE= CC=$(HOSTCC) LD=$(HOSTLD) $@ install_headers
	56	+ ARCH= CROSS_COMPILE= CC=$(HOSTCC) LD=$(HOSTLD) AR=$(HOSTAR) $@ install_headers
57	57
58	58	$(LIBBPF_BOOTSTRAP_INTERNAL_HDRS): $(LIBBPF_BOOTSTRAP_HDRS_DIR)/%.h: $(BPF_DIR)/%.h \| $(LIBBPF_BOOTSTRAP_HDRS_DIR)
59	59	$(call QUIET_INSTALL, $@)

tools/bpf/bpftool/bash-completion/bpftool+10−8

@@ -407,8 +407,8 @@ _bpftool() · 406 unmodified lines
407	407	return 0
408	408	;;
409	409	5)
410		− local BPFTOOL_PROG_ATTACH_TYPES='msg_verdict \
411		− skb_verdict stream_verdict stream_parser \
	410	+ local BPFTOOL_PROG_ATTACH_TYPES='sk_msg_verdict \
	411	+ sk_skb_verdict sk_skb_stream_verdict sk_skb_stream_parser \
412	412	flow_dissector'
413	413	COMPREPLY=( $( compgen -W "$BPFTOOL_PROG_ATTACH_TYPES" -- "$cur" ) )
414	414	return 0
@@ -1039,12 +1039,14 @@ _bpftool() · 624 unmodified lines
1039	1039	return 0
1040	1040	;;
1041	1041	attach\|detach)
1042		− local BPFTOOL_CGROUP_ATTACH_TYPES='ingress egress \
1043		− sock_create sock_ops device \
1044		− bind4 bind6 post_bind4 post_bind6 connect4 connect6 \
1045		− getpeername4 getpeername6 getsockname4 getsockname6 \
1046		− sendmsg4 sendmsg6 recvmsg4 recvmsg6 sysctl getsockopt \
1047		− setsockopt sock_release'
	1042	+ local BPFTOOL_CGROUP_ATTACH_TYPES='cgroup_inet_ingress cgroup_inet_egress \
	1043	+ cgroup_inet_sock_create cgroup_sock_ops cgroup_device cgroup_inet4_bind \
	1044	+ cgroup_inet6_bind cgroup_inet4_post_bind cgroup_inet6_post_bind \
	1045	+ cgroup_inet4_connect cgroup_inet6_connect cgroup_inet4_getpeername \
	1046	+ cgroup_inet6_getpeername cgroup_inet4_getsockname cgroup_inet6_getsockname \
	1047	+ cgroup_udp4_sendmsg cgroup_udp6_sendmsg cgroup_udp4_recvmsg \
	1048	+ cgroup_udp6_recvmsg cgroup_sysctl cgroup_getsockopt cgroup_setsockopt \
	1049	+ cgroup_inet_sock_release'
1048	1050	local ATTACH_FLAGS='multi override'
1049	1051	local PROG_TYPE='id pinned tag name'
1050	1052	# Check for $prev = $command first

tools/bpf/bpftool/btf.c+54−3

@@ -40,6 +40,7 @@ static const char * const btf_kind_str[NR_BTF_KINDS] = { · 39 unmodified lines
40	40	[BTF_KIND_FLOAT] = "FLOAT",
41	41	[BTF_KIND_DECL_TAG] = "DECL_TAG",
42	42	[BTF_KIND_TYPE_TAG] = "TYPE_TAG",
	43	+ [BTF_KIND_ENUM64] = "ENUM64",
43	44	};
44	45
45	46	struct btf_attach_point {
@@ -212,26 +213,76 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 166 unmodified lines
212	213	case BTF_KIND_ENUM: {
213	214	const struct btf_enum v = (const void )(t + 1);
214	215	__u16 vlen = BTF_INFO_VLEN(t->info);
	216	+ const char *encoding;
215	217	int i;
216	218
	219	+ encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED";
217	220	if (json_output) {
	221	+ jsonw_string_field(w, "encoding", encoding);
218	222	jsonw_uint_field(w, "size", t->size);
219	223	jsonw_uint_field(w, "vlen", vlen);
220	224	jsonw_name(w, "values");
221	225	jsonw_start_array(w);
222	226	} else {
223		− printf(" size=%u vlen=%u", t->size, vlen);
	227	+ printf(" encoding=%s size=%u vlen=%u", encoding, t->size, vlen);
	228	+ }
	229	+ for (i = 0; i < vlen; i++, v++) {
	230	+ const char *name = btf_str(btf, v->name_off);
	231	+
	232	+ if (json_output) {
	233	+ jsonw_start_object(w);
	234	+ jsonw_string_field(w, "name", name);
	235	+ if (btf_kflag(t))
	236	+ jsonw_int_field(w, "val", v->val);
	237	+ else
	238	+ jsonw_uint_field(w, "val", v->val);
	239	+ jsonw_end_object(w);
	240	+ } else {
	241	+ if (btf_kflag(t))
	242	+ printf("\n\t'%s' val=%d", name, v->val);
	243	+ else
	244	+ printf("\n\t'%s' val=%u", name, v->val);
	245	+ }
	246	+ }
	247	+ if (json_output)
	248	+ jsonw_end_array(w);
	249	+ break;
	250	+ }
	251	+ case BTF_KIND_ENUM64: {
	252	+ const struct btf_enum64 *v = btf_enum64(t);
	253	+ __u16 vlen = btf_vlen(t);
	254	+ const char *encoding;
	255	+ int i;
	256	+
	257	+ encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED";
	258	+ if (json_output) {
	259	+ jsonw_string_field(w, "encoding", encoding);
	260	+ jsonw_uint_field(w, "size", t->size);
	261	+ jsonw_uint_field(w, "vlen", vlen);
	262	+ jsonw_name(w, "values");
	263	+ jsonw_start_array(w);
	264	+ } else {
	265	+ printf(" encoding=%s size=%u vlen=%u", encoding, t->size, vlen);
224	266	}
225	267	for (i = 0; i < vlen; i++, v++) {
226	268	const char *name = btf_str(btf, v->name_off);
	269	+ __u64 val = ((__u64)v->val_hi32 << 32) \| v->val_lo32;
227	270
228	271	if (json_output) {
229	272	jsonw_start_object(w);
230	273	jsonw_string_field(w, "name", name);
231		− jsonw_uint_field(w, "val", v->val);
	274	+ if (btf_kflag(t))
	275	+ jsonw_int_field(w, "val", val);
	276	+ else
	277	+ jsonw_uint_field(w, "val", val);
232	278	jsonw_end_object(w);
233	279	} else {
234		− printf("\n\t'%s' val=%u", name, v->val);
	280	+ if (btf_kflag(t))
	281	+ printf("\n\t'%s' val=%lldLL", name,
	282	+ (unsigned long long)val);
	283	+ else
	284	+ printf("\n\t'%s' val=%lluULL", name,
	285	+ (unsigned long long)val);
235	286	}
236	287	}
237	288	if (json_output)

tools/bpf/bpftool/btf_dumper.c+29−0

@@ -182,6 +182,32 @@ static int btf_dumper_enum(const struct btf_dumper *d, · 181 unmodified lines
182	182	return 0;
183	183	}
184	184
	185	+static int btf_dumper_enum64(const struct btf_dumper *d,
	186	+ const struct btf_type *t,
	187	+ const void *data)
	188	+{
	189	+ const struct btf_enum64 *enums = btf_enum64(t);
	190	+ __u32 val_lo32, val_hi32;
	191	+ __u64 value;
	192	+ __u16 i;
	193	+
	194	+ value = (__u64 )data;
	195	+ val_lo32 = (__u32)value;
	196	+ val_hi32 = value >> 32;
	197	+
	198	+ for (i = 0; i < btf_vlen(t); i++) {
	199	+ if (val_lo32 == enums[i].val_lo32 && val_hi32 == enums[i].val_hi32) {
	200	+ jsonw_string(d->jw,
	201	+ btf__name_by_offset(d->btf,
	202	+ enums[i].name_off));
	203	+ return 0;
	204	+ }
	205	+ }
	206	+
	207	+ jsonw_int(d->jw, value);
	208	+ return 0;
	209	+}
	210	+
185	211	static bool is_str_array(const struct btf btf, const struct btf_array arr,
186	212	const char *s)
187	213	{
@@ -542,6 +568,8 @@ static int btf_dumper_do_type(const struct btf_dumper *d, __u32 type_id, · 354 unmodified lines
542	568	return btf_dumper_array(d, type_id, data);
543	569	case BTF_KIND_ENUM:
544	570	return btf_dumper_enum(d, t, data);
	571	+ case BTF_KIND_ENUM64:
	572	+ return btf_dumper_enum64(d, t, data);
545	573	case BTF_KIND_PTR:
546	574	btf_dumper_ptr(d, t, data);
547	575	return 0;
@@ -618,6 +646,7 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id, · 70 unmodified lines
618	646	btf__name_by_offset(btf, t->name_off));
619	647	break;
620	648	case BTF_KIND_ENUM:
	649	+ case BTF_KIND_ENUM64:
621	650	BTF_PRINT_ARG("enum %s ",
622	651	btf__name_by_offset(btf, t->name_off));
623	652	break;

tools/bpf/bpftool/cgroup.c+36−17

@@ -21,25 +21,43 @@ · 20 unmodified lines
21	21	#define HELP_SPEC_ATTACH_FLAGS \
22	22	"ATTACH_FLAGS := { multi \| override }"
23	23
24		−#define HELP_SPEC_ATTACH_TYPES \
25		− " ATTACH_TYPE := { ingress \| egress \| sock_create \|\n" \
26		− " sock_ops \| device \| bind4 \| bind6 \|\n" \
27		− " post_bind4 \| post_bind6 \| connect4 \|\n" \
28		− " connect6 \| getpeername4 \| getpeername6 \|\n" \
29		− " getsockname4 \| getsockname6 \| sendmsg4 \|\n" \
30		− " sendmsg6 \| recvmsg4 \| recvmsg6 \|\n" \
31		− " sysctl \| getsockopt \| setsockopt \|\n" \
32		− " sock_release }"
	24	+#define HELP_SPEC_ATTACH_TYPES \
	25	+ " ATTACH_TYPE := { cgroup_inet_ingress \| cgroup_inet_egress \|\n" \
	26	+ " cgroup_inet_sock_create \| cgroup_sock_ops \|\n" \
	27	+ " cgroup_device \| cgroup_inet4_bind \|\n" \
	28	+ " cgroup_inet6_bind \| cgroup_inet4_post_bind \|\n" \
	29	+ " cgroup_inet6_post_bind \| cgroup_inet4_connect \|\n" \
	30	+ " cgroup_inet6_connect \| cgroup_inet4_getpeername \|\n" \
	31	+ " cgroup_inet6_getpeername \| cgroup_inet4_getsockname \|\n" \
	32	+ " cgroup_inet6_getsockname \| cgroup_udp4_sendmsg \|\n" \
	33	+ " cgroup_udp6_sendmsg \| cgroup_udp4_recvmsg \|\n" \
	34	+ " cgroup_udp6_recvmsg \| cgroup_sysctl \|\n" \
	35	+ " cgroup_getsockopt \| cgroup_setsockopt \|\n" \
	36	+ " cgroup_inet_sock_release }"
33	37
34	38	static unsigned int query_flags;
35	39
36	40	static enum bpf_attach_type parse_attach_type(const char *str)
37	41	{
	42	+ const char *attach_type_str;
38	43	enum bpf_attach_type type;
39	44
40		− for (type = 0; type < __MAX_BPF_ATTACH_TYPE; type++) {
41		− if (attach_type_name[type] &&
42		− is_prefix(str, attach_type_name[type]))
	45	+ for (type = 0; ; type++) {
	46	+ attach_type_str = libbpf_bpf_attach_type_str(type);
	47	+ if (!attach_type_str)
	48	+ break;
	49	+ if (!strcmp(str, attach_type_str))
	50	+ return type;
	51	+ }
	52	+
	53	+ /* Also check traditionally used attach type strings. For these we keep
	54	+ * allowing prefixed usage.
	55	+ */
	56	+ for (type = 0; ; type++) {
	57	+ attach_type_str = bpf_attach_type_input_str(type);
	58	+ if (!attach_type_str)
	59	+ break;
	60	+ if (is_prefix(str, attach_type_str))
43	61	return type;
44	62	}
45	63
@@ -52,6 +70,7 @@ static int show_bpf_prog(int id, enum bpf_attach_type attach_type, · 6 unmodified lines
52	70	{
53	71	char prog_name[MAX_PROG_FULL_NAME];
54	72	struct bpf_prog_info info = {};
	73	+ const char *attach_type_str;
55	74	__u32 info_len = sizeof(info);
56	75	int prog_fd;
57	76
@@ -64,13 +83,13 @@ static int show_bpf_prog(int id, enum bpf_attach_type attach_type, · 6 unmodified lines
64	83	return -1;
65	84	}
66	85
	86	+ attach_type_str = libbpf_bpf_attach_type_str(attach_type);
67	87	get_prog_full_name(&info, prog_fd, prog_name, sizeof(prog_name));
68	88	if (json_output) {
69	89	jsonw_start_object(json_wtr);
70	90	jsonw_uint_field(json_wtr, "id", info.id);
71		− if (attach_type < ARRAY_SIZE(attach_type_name))
72		− jsonw_string_field(json_wtr, "attach_type",
73		− attach_type_name[attach_type]);
	91	+ if (attach_type_str)
	92	+ jsonw_string_field(json_wtr, "attach_type", attach_type_str);
74	93	else
75	94	jsonw_uint_field(json_wtr, "attach_type", attach_type);
76	95	jsonw_string_field(json_wtr, "attach_flags",
@@ -79,8 +98,8 @@ static int show_bpf_prog(int id, enum bpf_attach_type attach_type, · 2 unmodified lines
79	98	jsonw_end_object(json_wtr);
80	99	} else {
81	100	printf("%s%-8u ", level ? " " : "", info.id);
82		− if (attach_type < ARRAY_SIZE(attach_type_name))
83		− printf("%-15s", attach_type_name[attach_type]);
	101	+ if (attach_type_str)
	102	+ printf("%-15s", attach_type_str);
84	103	else
85	104	printf("type %-10u", attach_type);
86	105	printf(" %-15s %-15s\n", attach_flags_str, prog_name);

tools/bpf/bpftool/common.c+44−46

@@ -17,6 +17,7 @@ · 16 unmodified lines
17	17	#include <linux/magic.h>
18	18	#include <net/if.h>
19	19	#include <sys/mount.h>
	20	+#include <sys/resource.h>
20	21	#include <sys/stat.h>
21	22	#include <sys/vfs.h>
22	23
@@ -31,52 +32,6 @@ · 8 unmodified lines
31	32	#define BPF_FS_MAGIC 0xcafe4a11
32	33	#endif
33	34
34		−const char * const attach_type_name[__MAX_BPF_ATTACH_TYPE] = {
35		− [BPF_CGROUP_INET_INGRESS] = "ingress",
36		− [BPF_CGROUP_INET_EGRESS] = "egress",
37		− [BPF_CGROUP_INET_SOCK_CREATE] = "sock_create",
38		− [BPF_CGROUP_INET_SOCK_RELEASE] = "sock_release",
39		− [BPF_CGROUP_SOCK_OPS] = "sock_ops",
40		− [BPF_CGROUP_DEVICE] = "device",
41		− [BPF_CGROUP_INET4_BIND] = "bind4",
42		− [BPF_CGROUP_INET6_BIND] = "bind6",
43		− [BPF_CGROUP_INET4_CONNECT] = "connect4",
44		− [BPF_CGROUP_INET6_CONNECT] = "connect6",
45		− [BPF_CGROUP_INET4_POST_BIND] = "post_bind4",
46		− [BPF_CGROUP_INET6_POST_BIND] = "post_bind6",
47		− [BPF_CGROUP_INET4_GETPEERNAME] = "getpeername4",
48		− [BPF_CGROUP_INET6_GETPEERNAME] = "getpeername6",
49		− [BPF_CGROUP_INET4_GETSOCKNAME] = "getsockname4",
50		− [BPF_CGROUP_INET6_GETSOCKNAME] = "getsockname6",
51		− [BPF_CGROUP_UDP4_SENDMSG] = "sendmsg4",
52		− [BPF_CGROUP_UDP6_SENDMSG] = "sendmsg6",
53		− [BPF_CGROUP_SYSCTL] = "sysctl",
54		− [BPF_CGROUP_UDP4_RECVMSG] = "recvmsg4",
55		− [BPF_CGROUP_UDP6_RECVMSG] = "recvmsg6",
56		− [BPF_CGROUP_GETSOCKOPT] = "getsockopt",
57		− [BPF_CGROUP_SETSOCKOPT] = "setsockopt",
58		− [BPF_SK_SKB_STREAM_PARSER] = "sk_skb_stream_parser",
59		− [BPF_SK_SKB_STREAM_VERDICT] = "sk_skb_stream_verdict",
60		− [BPF_SK_SKB_VERDICT] = "sk_skb_verdict",
61		− [BPF_SK_MSG_VERDICT] = "sk_msg_verdict",
62		− [BPF_LIRC_MODE2] = "lirc_mode2",
63		− [BPF_FLOW_DISSECTOR] = "flow_dissector",
64		− [BPF_TRACE_RAW_TP] = "raw_tp",
65		− [BPF_TRACE_FENTRY] = "fentry",
66		− [BPF_TRACE_FEXIT] = "fexit",
67		− [BPF_MODIFY_RETURN] = "mod_ret",
68		− [BPF_LSM_MAC] = "lsm_mac",
69		− [BPF_SK_LOOKUP] = "sk_lookup",
70		− [BPF_TRACE_ITER] = "trace_iter",
71		− [BPF_XDP_DEVMAP] = "xdp_devmap",
72		− [BPF_XDP_CPUMAP] = "xdp_cpumap",
73		− [BPF_XDP] = "xdp",
74		− [BPF_SK_REUSEPORT_SELECT] = "sk_skb_reuseport_select",
75		− [BPF_SK_REUSEPORT_SELECT_OR_MIGRATE] = "sk_skb_reuseport_select_or_migrate",
76		− [BPF_PERF_EVENT] = "perf_event",
77		− [BPF_TRACE_KPROBE_MULTI] = "trace_kprobe_multi",
78		−};
79		−
80	35	void p_err(const char *fmt, ...)
81	36	{
82	37	va_list ap;
@@ -118,6 +73,13 @@ static bool is_bpffs(char *path) · 35 unmodified lines
118	73	return (unsigned long)st_fs.f_type == BPF_FS_MAGIC;
119	74	}
120	75
	76	+void set_max_rlimit(void)
	77	+{
	78	+ struct rlimit rinf = { RLIM_INFINITY, RLIM_INFINITY };
	79	+
	80	+ setrlimit(RLIMIT_MEMLOCK, &rinf);
	81	+}
	82	+
121	83	static int
122	84	mnt_fs(const char target, const char type, char *buff, size_t bufflen)
123	85	{
@@ -1009,3 +971,39 @@ bool equal_fn_for_key_as_id(const void k1, const void k2, void *ctx) · 885 unmodified lines
1009	971	{
1010	972	return k1 == k2;
1011	973	}
	974	+
	975	+const char *bpf_attach_type_input_str(enum bpf_attach_type t)
	976	+{
	977	+ switch (t) {
	978	+ case BPF_CGROUP_INET_INGRESS: return "ingress";
	979	+ case BPF_CGROUP_INET_EGRESS: return "egress";
	980	+ case BPF_CGROUP_INET_SOCK_CREATE: return "sock_create";
	981	+ case BPF_CGROUP_INET_SOCK_RELEASE: return "sock_release";
	982	+ case BPF_CGROUP_SOCK_OPS: return "sock_ops";
	983	+ case BPF_CGROUP_DEVICE: return "device";
	984	+ case BPF_CGROUP_INET4_BIND: return "bind4";
	985	+ case BPF_CGROUP_INET6_BIND: return "bind6";
	986	+ case BPF_CGROUP_INET4_CONNECT: return "connect4";
	987	+ case BPF_CGROUP_INET6_CONNECT: return "connect6";
	988	+ case BPF_CGROUP_INET4_POST_BIND: return "post_bind4";
	989	+ case BPF_CGROUP_INET6_POST_BIND: return "post_bind6";
	990	+ case BPF_CGROUP_INET4_GETPEERNAME: return "getpeername4";
	991	+ case BPF_CGROUP_INET6_GETPEERNAME: return "getpeername6";
	992	+ case BPF_CGROUP_INET4_GETSOCKNAME: return "getsockname4";
	993	+ case BPF_CGROUP_INET6_GETSOCKNAME: return "getsockname6";
	994	+ case BPF_CGROUP_UDP4_SENDMSG: return "sendmsg4";
	995	+ case BPF_CGROUP_UDP6_SENDMSG: return "sendmsg6";
	996	+ case BPF_CGROUP_SYSCTL: return "sysctl";
	997	+ case BPF_CGROUP_UDP4_RECVMSG: return "recvmsg4";
	998	+ case BPF_CGROUP_UDP6_RECVMSG: return "recvmsg6";
	999	+ case BPF_CGROUP_GETSOCKOPT: return "getsockopt";
	1000	+ case BPF_CGROUP_SETSOCKOPT: return "setsockopt";
	1001	+ case BPF_TRACE_RAW_TP: return "raw_tp";
	1002	+ case BPF_TRACE_FENTRY: return "fentry";
	1003	+ case BPF_TRACE_FEXIT: return "fexit";
	1004	+ case BPF_MODIFY_RETURN: return "mod_ret";
	1005	+ case BPF_SK_REUSEPORT_SELECT: return "sk_skb_reuseport_select";
	1006	+ case BPF_SK_REUSEPORT_SELECT_OR_MIGRATE: return "sk_skb_reuseport_select_or_migrate";
	1007	+ default: return libbpf_bpf_attach_type_str(t);
	1008	+ }
	1009	+}

tools/bpf/bpftool/feature.c+55−34

@@ -548,8 +548,8 @@ static bool probe_prog_type_ifindex(enum bpf_prog_type prog_type, __u32 ifindex) · 547 unmodified lines
548	548	}
549	549
550	550	static void
551		−probe_prog_type(enum bpf_prog_type prog_type, bool *supported_types,
552		− const char *define_prefix, __u32 ifindex)
	551	+probe_prog_type(enum bpf_prog_type prog_type, const char *prog_type_str,
	552	+ bool supported_types, const char define_prefix, __u32 ifindex)
553	553	{
554	554	char feat_name[128], plain_desc[128], define_name[128];
555	555	const char *plain_comment = "eBPF program_type ";
@@ -580,20 +580,16 @@ probe_prog_type(enum bpf_prog_type prog_type, bool *supported_types, · 24 unmodified lines
580	580
581	581	supported_types[prog_type] \|= res;
582	582
583		− if (!prog_type_name[prog_type]) {
584		− p_info("program type name not found (type %d)", prog_type);
585		− return;
586		− }
587	583	maxlen = sizeof(plain_desc) - strlen(plain_comment) - 1;
588		− if (strlen(prog_type_name[prog_type]) > maxlen) {
	584	+ if (strlen(prog_type_str) > maxlen) {
589	585	p_info("program type name too long");
590	586	return;
591	587	}
592	588
593		− sprintf(feat_name, "have_%s_prog_type", prog_type_name[prog_type]);
594		− sprintf(define_name, "%s_prog_type", prog_type_name[prog_type]);
	589	+ sprintf(feat_name, "have_%s_prog_type", prog_type_str);
	590	+ sprintf(define_name, "%s_prog_type", prog_type_str);
595	591	uppercase(define_name, sizeof(define_name));
596		− sprintf(plain_desc, "%s%s", plain_comment, prog_type_name[prog_type]);
	592	+ sprintf(plain_desc, "%s%s", plain_comment, prog_type_str);
597	593	print_bool_feature(feat_name, plain_desc, define_name, res,
598	594	define_prefix);
599	595	}
@@ -619,8 +615,8 @@ static bool probe_map_type_ifindex(enum bpf_map_type map_type, __u32 ifindex) · 19 unmodified lines
619	615	}
620	616
621	617	static void
622		−probe_map_type(enum bpf_map_type map_type, const char *define_prefix,
623		− __u32 ifindex)
	618	+probe_map_type(enum bpf_map_type map_type, char const *map_type_str,
	619	+ const char *define_prefix, __u32 ifindex)
624	620	{
625	621	char feat_name[128], plain_desc[128], define_name[128];
626	622	const char *plain_comment = "eBPF map_type ";
@@ -645,20 +641,16 @@ probe_map_type(enum bpf_map_type map_type, const char *define_prefix, · 18 unmodified lines
645	641	* check required for unprivileged users
646	642	*/
647	643
648		− if (!map_type_name[map_type]) {
649		− p_info("map type name not found (type %d)", map_type);
650		− return;
651		− }
652	644	maxlen = sizeof(plain_desc) - strlen(plain_comment) - 1;
653		− if (strlen(map_type_name[map_type]) > maxlen) {
	645	+ if (strlen(map_type_str) > maxlen) {
654	646	p_info("map type name too long");
655	647	return;
656	648	}
657	649
658		− sprintf(feat_name, "have_%s_map_type", map_type_name[map_type]);
659		− sprintf(define_name, "%s_map_type", map_type_name[map_type]);
	650	+ sprintf(feat_name, "have_%s_map_type", map_type_str);
	651	+ sprintf(define_name, "%s_map_type", map_type_str);
660	652	uppercase(define_name, sizeof(define_name));
661		− sprintf(plain_desc, "%s%s", plain_comment, map_type_name[map_type]);
	653	+ sprintf(plain_desc, "%s%s", plain_comment, map_type_str);
662	654	print_bool_feature(feat_name, plain_desc, define_name, res,
663	655	define_prefix);
664	656	}
@@ -728,10 +720,10 @@ probe_helper_for_progtype(enum bpf_prog_type prog_type, bool supported_type, · 63 unmodified lines
728	720	}
729	721
730	722	static void
731		−probe_helpers_for_progtype(enum bpf_prog_type prog_type, bool supported_type,
	723	+probe_helpers_for_progtype(enum bpf_prog_type prog_type,
	724	+ const char *prog_type_str, bool supported_type,
732	725	const char *define_prefix, __u32 ifindex)
733	726	{
734		− const char *ptype_name = prog_type_name[prog_type];
735	727	char feat_name[128];
736	728	unsigned int id;
737	729	bool probe_res = false;
@@ -747,12 +739,12 @@ probe_helpers_for_progtype(enum bpf_prog_type prog_type, bool supported_type, · 9 unmodified lines
747	739	}
748	740
749	741	if (json_output) {
750		− sprintf(feat_name, "%s_available_helpers", ptype_name);
	742	+ sprintf(feat_name, "%s_available_helpers", prog_type_str);
751	743	jsonw_name(json_wtr, feat_name);
752	744	jsonw_start_array(json_wtr);
753	745	} else if (!define_prefix) {
754	746	printf("eBPF helpers supported for program type %s:",
755		− ptype_name);
	747	+ prog_type_str);
756	748	}
757	749
758	750	for (id = 1; id < ARRAY_SIZE(helper_name); id++) {
@@ -768,7 +760,7 @@ probe_helpers_for_progtype(enum bpf_prog_type prog_type, bool supported_type, · 9 unmodified lines
768	760	/* fallthrough */
769	761	default:
770	762	probe_res \|= probe_helper_for_progtype(prog_type, supported_type,
771		− define_prefix, id, ptype_name,
	763	+ define_prefix, id, prog_type_str,
772	764	ifindex);
773	765	}
774	766	}
@@ -943,30 +935,47 @@ static void · 168 unmodified lines
943	935	section_program_types(bool supported_types, const char define_prefix,
944	936	__u32 ifindex)
945	937	{
946		− unsigned int i;
	938	+ unsigned int prog_type = BPF_PROG_TYPE_UNSPEC;
	939	+ const char *prog_type_str;
947	940
948	941	print_start_section("program_types",
949	942	"Scanning eBPF program types...",
950	943	"/* eBPF program types */",
951	944	define_prefix);
952	945
953		− for (i = BPF_PROG_TYPE_UNSPEC + 1; i < prog_type_name_size; i++)
954		− probe_prog_type(i, supported_types, define_prefix, ifindex);
	946	+ while (true) {
	947	+ prog_type++;
	948	+ prog_type_str = libbpf_bpf_prog_type_str(prog_type);
	949	+ /* libbpf will return NULL for variants unknown to it. */
	950	+ if (!prog_type_str)
	951	+ break;
	952	+
	953	+ probe_prog_type(prog_type, prog_type_str, supported_types, define_prefix,
	954	+ ifindex);
	955	+ }
955	956
956	957	print_end_section();
957	958	}
958	959
959	960	static void section_map_types(const char *define_prefix, __u32 ifindex)
960	961	{
961		− unsigned int i;
	962	+ unsigned int map_type = BPF_MAP_TYPE_UNSPEC;
	963	+ const char *map_type_str;
962	964
963	965	print_start_section("map_types",
964	966	"Scanning eBPF map types...",
965	967	"/* eBPF map types */",
966	968	define_prefix);
967	969
968		− for (i = BPF_MAP_TYPE_UNSPEC + 1; i < map_type_name_size; i++)
969		− probe_map_type(i, define_prefix, ifindex);
	970	+ while (true) {
	971	+ map_type++;
	972	+ map_type_str = libbpf_bpf_map_type_str(map_type);
	973	+ /* libbpf will return NULL for variants unknown to it. */
	974	+ if (!map_type_str)
	975	+ break;
	976	+
	977	+ probe_map_type(map_type, map_type_str, define_prefix, ifindex);
	978	+ }
970	979
971	980	print_end_section();
972	981	}
@@ -974,7 +983,8 @@ static void section_map_types(const char *define_prefix, __u32 ifindex) · 1 unmodified line
974	983	static void
975	984	section_helpers(bool supported_types, const char define_prefix, __u32 ifindex)
976	985	{
977		− unsigned int i;
	986	+ unsigned int prog_type = BPF_PROG_TYPE_UNSPEC;
	987	+ const char *prog_type_str;
978	988
979	989	print_start_section("helpers",
980	990	"Scanning eBPF helper functions...",
@@ -996,9 +1006,18 @@ section_helpers(bool supported_types, const char define_prefix, __u32 ifindex) · 15 unmodified lines
996	1006	" %sBPF__PROG_TYPE_ ## prog_type ## __HELPER_ ## helper\n",
997	1007	define_prefix, define_prefix, define_prefix,
998	1008	define_prefix);
999		− for (i = BPF_PROG_TYPE_UNSPEC + 1; i < prog_type_name_size; i++)
1000		− probe_helpers_for_progtype(i, supported_types[i], define_prefix,
	1009	+ while (true) {
	1010	+ prog_type++;
	1011	+ prog_type_str = libbpf_bpf_prog_type_str(prog_type);
	1012	+ /* libbpf will return NULL for variants unknown to it. */
	1013	+ if (!prog_type_str)
	1014	+ break;
	1015	+
	1016	+ probe_helpers_for_progtype(prog_type, prog_type_str,
	1017	+ supported_types[prog_type],
	1018	+ define_prefix,
1001	1019	ifindex);
	1020	+ }
1002	1021
1003	1022	print_end_section();
1004	1023	}
@@ -1148,6 +1167,8 @@ static int do_probe(int argc, char **argv) · 143 unmodified lines
1148	1167	__u32 ifindex = 0;
1149	1168	char *ifname;
1150	1169
	1170	+ set_max_rlimit();
	1171	+
1151	1172	while (argc) {
1152	1173	if (is_prefix(*argv, "kernel")) {
1153	1174	if (target != COMPONENT_UNSPEC) {

tools/bpf/bpftool/gen.c+4−0

@@ -474,6 +474,9 @@ static void codegen_asserts(struct bpf_object obj, const char obj_name) · 473 unmodified lines
474	474	const struct btf_type *sec;
475	475	char map_ident[256], var_ident[256];
476	476
	477	+ if (!btf)
	478	+ return;
	479	+
477	480	codegen("\
478	481	\n\
479	482	__attribute__((unused)) static void \n\
@@ -1747,6 +1750,7 @@ btfgen_mark_type(struct btfgen_info *info, unsigned int type_id, bool follow_poi · 1267 unmodified lines
1747	1750	case BTF_KIND_INT:
1748	1751	case BTF_KIND_FLOAT:
1749	1752	case BTF_KIND_ENUM:
	1753	+ case BTF_KIND_ENUM64:
1750	1754	case BTF_KIND_STRUCT:
1751	1755	case BTF_KIND_UNION:
1752	1756	break;

tools/bpf/bpftool/link.c+31−30

@@ -13,19 +13,6 @@ · 12 unmodified lines
13	13	#include "json_writer.h"
14	14	#include "main.h"
15	15
16		−static const char * const link_type_name[] = {
17		− [BPF_LINK_TYPE_UNSPEC] = "unspec",
18		− [BPF_LINK_TYPE_RAW_TRACEPOINT] = "raw_tracepoint",
19		− [BPF_LINK_TYPE_TRACING] = "tracing",
20		− [BPF_LINK_TYPE_CGROUP] = "cgroup",
21		− [BPF_LINK_TYPE_ITER] = "iter",
22		− [BPF_LINK_TYPE_NETNS] = "netns",
23		− [BPF_LINK_TYPE_XDP] = "xdp",
24		− [BPF_LINK_TYPE_PERF_EVENT] = "perf_event",
25		− [BPF_LINK_TYPE_KPROBE_MULTI] = "kprobe_multi",
26		− [BPF_LINK_TYPE_STRUCT_OPS] = "struct_ops",
27		−};
28		−
29	16	static struct hashmap *link_table;
30	17
31	18	static int link_parse_fd(int argc, char **argv)
@@ -67,9 +54,12 @@ static int link_parse_fd(int argc, char **argv) · 35 unmodified lines
67	54	static void
68	55	show_link_header_json(struct bpf_link_info info, json_writer_t wtr)
69	56	{
	57	+ const char *link_type_str;
	58	+
70	59	jsonw_uint_field(wtr, "id", info->id);
71		− if (info->type < ARRAY_SIZE(link_type_name))
72		− jsonw_string_field(wtr, "type", link_type_name[info->type]);
	60	+ link_type_str = libbpf_bpf_link_type_str(info->type);
	61	+ if (link_type_str)
	62	+ jsonw_string_field(wtr, "type", link_type_str);
73	63	else
74	64	jsonw_uint_field(wtr, "type", info->type);
75	65
@@ -78,9 +68,11 @@ show_link_header_json(struct bpf_link_info info, json_writer_t wtr) · 2 unmodified lines
78	68
79	69	static void show_link_attach_type_json(__u32 attach_type, json_writer_t *wtr)
80	70	{
81		− if (attach_type < ARRAY_SIZE(attach_type_name))
82		− jsonw_string_field(wtr, "attach_type",
83		− attach_type_name[attach_type]);
	71	+ const char *attach_type_str;
	72	+
	73	+ attach_type_str = libbpf_bpf_attach_type_str(attach_type);
	74	+ if (attach_type_str)
	75	+ jsonw_string_field(wtr, "attach_type", attach_type_str);
84	76	else
85	77	jsonw_uint_field(wtr, "attach_type", attach_type);
86	78	}
@@ -121,6 +113,7 @@ static int get_prog_info(int prog_id, struct bpf_prog_info *info) · 34 unmodified lines
121	113	static int show_link_close_json(int fd, struct bpf_link_info *info)
122	114	{
123	115	struct bpf_prog_info prog_info;
	116	+ const char *prog_type_str;
124	117	int err;
125	118
126	119	jsonw_start_object(json_wtr);
@@ -137,12 +130,12 @@ static int show_link_close_json(int fd, struct bpf_link_info *info) · 10 unmodified lines
137	130	if (err)
138	131	return err;
139	132
140		− if (prog_info.type < prog_type_name_size)
141		− jsonw_string_field(json_wtr, "prog_type",
142		− prog_type_name[prog_info.type]);
	133	+ prog_type_str = libbpf_bpf_prog_type_str(prog_info.type);
	134	+ /* libbpf will return NULL for variants unknown to it. */
	135	+ if (prog_type_str)
	136	+ jsonw_string_field(json_wtr, "prog_type", prog_type_str);
143	137	else
144		− jsonw_uint_field(json_wtr, "prog_type",
145		− prog_info.type);
	138	+ jsonw_uint_field(json_wtr, "prog_type", prog_info.type);
146	139
147	140	show_link_attach_type_json(info->tracing.attach_type,
148	141	json_wtr);
@@ -184,9 +177,12 @@ static int show_link_close_json(int fd, struct bpf_link_info *info) · 35 unmodified lines
184	177
185	178	static void show_link_header_plain(struct bpf_link_info *info)
186	179	{
	180	+ const char *link_type_str;
	181	+
187	182	printf("%u: ", info->id);
188		− if (info->type < ARRAY_SIZE(link_type_name))
189		− printf("%s ", link_type_name[info->type]);
	183	+ link_type_str = libbpf_bpf_link_type_str(info->type);
	184	+ if (link_type_str)
	185	+ printf("%s ", link_type_str);
190	186	else
191	187	printf("type %u ", info->type);
192	188
@@ -195,8 +191,11 @@ static void show_link_header_plain(struct bpf_link_info *info) · 2 unmodified lines
195	191
196	192	static void show_link_attach_type_plain(__u32 attach_type)
197	193	{
198		− if (attach_type < ARRAY_SIZE(attach_type_name))
199		− printf("attach_type %s ", attach_type_name[attach_type]);
	194	+ const char *attach_type_str;
	195	+
	196	+ attach_type_str = libbpf_bpf_attach_type_str(attach_type);
	197	+ if (attach_type_str)
	198	+ printf("attach_type %s ", attach_type_str);
200	199	else
201	200	printf("attach_type %u ", attach_type);
202	201	}
@@ -214,6 +213,7 @@ static void show_iter_plain(struct bpf_link_info *info) · 11 unmodified lines
214	213	static int show_link_close_plain(int fd, struct bpf_link_info *info)
215	214	{
216	215	struct bpf_prog_info prog_info;
	216	+ const char *prog_type_str;
217	217	int err;
218	218
219	219	show_link_header_plain(info);
@@ -228,9 +228,10 @@ static int show_link_close_plain(int fd, struct bpf_link_info *info) · 8 unmodified lines
228	228	if (err)
229	229	return err;
230	230
231		− if (prog_info.type < prog_type_name_size)
232		− printf("\n\tprog_type %s ",
233		− prog_type_name[prog_info.type]);
	231	+ prog_type_str = libbpf_bpf_prog_type_str(prog_info.type);
	232	+ /* libbpf will return NULL for variants unknown to it. */
	233	+ if (prog_type_str)
	234	+ printf("\n\tprog_type %s ", prog_type_str);
234	235	else
235	236	printf("\n\tprog_type %u ", prog_info.type);
236	237

tools/bpf/bpftool/main.c+0−2

@@ -508,8 +508,6 @@ int main(int argc, char **argv) · 507 unmodified lines
508	508	* mode for loading generated skeleton.
509	509	*/
510	510	libbpf_set_strict_mode(LIBBPF_STRICT_ALL & ~LIBBPF_STRICT_MAP_DEFINITIONS);
511		− } else {
512		− libbpf_set_strict_mode(LIBBPF_STRICT_AUTO_RLIMIT_MEMLOCK);
513	511	}
514	512
515	513	argc -= optind;

tools/bpf/bpftool/main.h+16−6

@@ -63,14 +63,8 @@ static inline void *u64_to_ptr(__u64 ptr) · 62 unmodified lines
63	63	#define HELP_SPEC_LINK \
64	64	"LINK := { id LINK_ID \| pinned FILE }"
65	65
66		−extern const char * const prog_type_name[];
67		−extern const size_t prog_type_name_size;
68		−
69	66	extern const char * const attach_type_name[__MAX_BPF_ATTACH_TYPE];
70	67
71		−extern const char * const map_type_name[];
72		−extern const size_t map_type_name_size;
73		−
74	68	/* keep in sync with the definition in skeleton/pid_iter.bpf.c */
75	69	enum bpf_obj_type {
76	70	BPF_OBJ_UNKNOWN,
@@ -102,6 +96,8 @@ int detect_common_prefix(const char *arg, ...); · 25 unmodified lines
102	96	void fprint_hex(FILE f, void arg, unsigned int n, const char *sep);
103	97	void usage(void) __noreturn;
104	98
	99	+void set_max_rlimit(void);
	100	+
105	101	int mount_tracefs(const char *target);
106	102
107	103	struct obj_ref {
@@ -249,6 +245,20 @@ int print_all_levels(__maybe_unused enum libbpf_print_level level, · 141 unmodified lines
249	245	size_t hash_fn_for_key_as_id(const void key, void ctx);
250	246	bool equal_fn_for_key_as_id(const void k1, const void k2, void *ctx);
251	247
	248	+/* bpf_attach_type_input_str - convert the provided attach type value into a
	249	+ * textual representation that we accept for input purposes.
	250	+ *
	251	+ * This function is similar in nature to libbpf_bpf_attach_type_str, but
	252	+ * recognizes some attach type names that have been used by the program in the
	253	+ * past and which do not follow the string inference scheme that libbpf uses.
	254	+ * These textual representations should only be used for user input.
	255	+ *
	256	+ * @t: The attach type
	257	+ * Returns a pointer to a static string identifying the attach type. NULL is
	258	+ * returned for unknown bpf_attach_type values.
	259	+ */
	260	+const char *bpf_attach_type_input_str(enum bpf_attach_type t);
	261	+
252	262	static inline void *u32_as_hash_field(__u32 x)
253	263	{
254	264	return (void *)(uintptr_t)x;

tools/bpf/bpftool/map.c+35−49

@@ -22,42 +22,6 @@ · 21 unmodified lines
22	22	#include "json_writer.h"
23	23	#include "main.h"
24	24
25		−const char * const map_type_name[] = {
26		− [BPF_MAP_TYPE_UNSPEC] = "unspec",
27		− [BPF_MAP_TYPE_HASH] = "hash",
28		− [BPF_MAP_TYPE_ARRAY] = "array",
29		− [BPF_MAP_TYPE_PROG_ARRAY] = "prog_array",
30		− [BPF_MAP_TYPE_PERF_EVENT_ARRAY] = "perf_event_array",
31		− [BPF_MAP_TYPE_PERCPU_HASH] = "percpu_hash",
32		− [BPF_MAP_TYPE_PERCPU_ARRAY] = "percpu_array",
33		− [BPF_MAP_TYPE_STACK_TRACE] = "stack_trace",
34		− [BPF_MAP_TYPE_CGROUP_ARRAY] = "cgroup_array",
35		− [BPF_MAP_TYPE_LRU_HASH] = "lru_hash",
36		− [BPF_MAP_TYPE_LRU_PERCPU_HASH] = "lru_percpu_hash",
37		− [BPF_MAP_TYPE_LPM_TRIE] = "lpm_trie",
38		− [BPF_MAP_TYPE_ARRAY_OF_MAPS] = "array_of_maps",
39		− [BPF_MAP_TYPE_HASH_OF_MAPS] = "hash_of_maps",
40		− [BPF_MAP_TYPE_DEVMAP] = "devmap",
41		− [BPF_MAP_TYPE_DEVMAP_HASH] = "devmap_hash",
42		− [BPF_MAP_TYPE_SOCKMAP] = "sockmap",
43		− [BPF_MAP_TYPE_CPUMAP] = "cpumap",
44		− [BPF_MAP_TYPE_XSKMAP] = "xskmap",
45		− [BPF_MAP_TYPE_SOCKHASH] = "sockhash",
46		− [BPF_MAP_TYPE_CGROUP_STORAGE] = "cgroup_storage",
47		− [BPF_MAP_TYPE_REUSEPORT_SOCKARRAY] = "reuseport_sockarray",
48		− [BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE] = "percpu_cgroup_storage",
49		− [BPF_MAP_TYPE_QUEUE] = "queue",
50		− [BPF_MAP_TYPE_STACK] = "stack",
51		− [BPF_MAP_TYPE_SK_STORAGE] = "sk_storage",
52		− [BPF_MAP_TYPE_STRUCT_OPS] = "struct_ops",
53		− [BPF_MAP_TYPE_RINGBUF] = "ringbuf",
54		− [BPF_MAP_TYPE_INODE_STORAGE] = "inode_storage",
55		− [BPF_MAP_TYPE_TASK_STORAGE] = "task_storage",
56		− [BPF_MAP_TYPE_BLOOM_FILTER] = "bloom_filter",
57		−};
58		−
59		−const size_t map_type_name_size = ARRAY_SIZE(map_type_name);
60		−
61	25	static struct hashmap *map_table;
62	26
63	27	static bool map_is_per_cpu(__u32 type)
@@ -81,12 +45,18 @@ static bool map_is_map_of_progs(__u32 type) · 17 unmodified lines
81	45
82	46	static int map_type_from_str(const char *type)
83	47	{
	48	+ const char *map_type_str;
84	49	unsigned int i;
85	50
86		− for (i = 0; i < ARRAY_SIZE(map_type_name); i++)
	51	+ for (i = 0; ; i++) {
	52	+ map_type_str = libbpf_bpf_map_type_str(i);
	53	+ if (!map_type_str)
	54	+ break;
	55	+
87	56	/* Don't allow prefixing in case of possible future shadowing */
88		− if (map_type_name[i] && !strcmp(map_type_name[i], type))
	57	+ if (!strcmp(map_type_str, type))
89	58	return i;
	59	+ }
90	60	return -1;
91	61	}
92	62
@@ -472,9 +442,12 @@ static int parse_elem(char *argv, struct bpf_map_info info, · 379 unmodified lines
472	442
473	443	static void show_map_header_json(struct bpf_map_info info, json_writer_t wtr)
474	444	{
	445	+ const char *map_type_str;
	446	+
475	447	jsonw_uint_field(wtr, "id", info->id);
476		− if (info->type < ARRAY_SIZE(map_type_name))
477		− jsonw_string_field(wtr, "type", map_type_name[info->type]);
	448	+ map_type_str = libbpf_bpf_map_type_str(info->type);
	449	+ if (map_type_str)
	450	+ jsonw_string_field(wtr, "type", map_type_str);
478	451	else
479	452	jsonw_uint_field(wtr, "type", info->type);
480	453
@@ -513,10 +486,12 @@ static int show_map_close_json(int fd, struct bpf_map_info *info) · 32 unmodified lines
513	486
514	487	if (owner_prog_type) {
515	488	unsigned int prog_type = atoi(owner_prog_type);
	489	+ const char *prog_type_str;
516	490
517		− if (prog_type < prog_type_name_size)
	491	+ prog_type_str = libbpf_bpf_prog_type_str(prog_type);
	492	+ if (prog_type_str)
518	493	jsonw_string_field(json_wtr, "owner_prog_type",
519		− prog_type_name[prog_type]);
	494	+ prog_type_str);
520	495	else
521	496	jsonw_uint_field(json_wtr, "owner_prog_type",
522	497	prog_type);
@@ -559,9 +534,13 @@ static int show_map_close_json(int fd, struct bpf_map_info *info) · 36 unmodified lines
559	534
560	535	static void show_map_header_plain(struct bpf_map_info *info)
561	536	{
	537	+ const char *map_type_str;
	538	+
562	539	printf("%u: ", info->id);
563		− if (info->type < ARRAY_SIZE(map_type_name))
564		− printf("%s ", map_type_name[info->type]);
	540	+
	541	+ map_type_str = libbpf_bpf_map_type_str(info->type);
	542	+ if (map_type_str)
	543	+ printf("%s ", map_type_str);
565	544	else
566	545	printf("type %u ", info->type);
567	546
@@ -597,10 +576,11 @@ static int show_map_close_plain(int fd, struct bpf_map_info *info) · 29 unmodified lines
597	576	printf("\n\t");
598	577	if (owner_prog_type) {
599	578	unsigned int prog_type = atoi(owner_prog_type);
	579	+ const char *prog_type_str;
600	580
601		− if (prog_type < prog_type_name_size)
602		− printf("owner_prog_type %s ",
603		− prog_type_name[prog_type]);
	581	+ prog_type_str = libbpf_bpf_prog_type_str(prog_type);
	582	+ if (prog_type_str)
	583	+ printf("owner_prog_type %s ", prog_type_str);
604	584	else
605	585	printf("owner_prog_type %d ", prog_type);
606	586	}
@@ -876,9 +856,13 @@ map_dump(int fd, struct bpf_map_info info, json_writer_t wtr, · 269 unmodified lines
876	856	}
877	857
878	858	if (info->type == BPF_MAP_TYPE_REUSEPORT_SOCKARRAY &&
879		− info->value_size != 8)
	859	+ info->value_size != 8) {
	860	+ const char *map_type_str;
	861	+
	862	+ map_type_str = libbpf_bpf_map_type_str(info->type);
880	863	p_info("Warning: cannot read values from %s map with value_size != 8",
881		− map_type_name[info->type]);
	864	+ map_type_str);
	865	+ }
882	866	while (true) {
883	867	err = bpf_map_get_next_key(fd, prev_key, key);
884	868	if (err) {
@@ -1342,6 +1326,8 @@ static int do_create(int argc, char **argv) · 457 unmodified lines
1342	1326	goto exit;
1343	1327	}
1344	1328
	1329	+ set_max_rlimit();
	1330	+
1345	1331	fd = bpf_map_create(map_type, map_name, key_size, value_size, max_entries, &attr);
1346	1332	if (fd < 0) {
1347	1333	p_err("map create failed: %s", strerror(errno));

tools/bpf/bpftool/pids.c+1−0

@@ -108,6 +108,7 @@ int build_obj_refs_table(struct hashmap **map, enum bpf_obj_type type) · 107 unmodified lines
108	108	p_err("failed to create hashmap for PID references");
109	109	return -1;
110	110	}
	111	+ set_max_rlimit();
111	112
112	113	skel = pid_iter_bpf__open();
113	114	if (!skel) {

tools/bpf/bpftool/prog.c+34−45

@@ -36,54 +36,28 @@ · 35 unmodified lines
36	36	#define BPF_METADATA_PREFIX "bpf_metadata_"
37	37	#define BPF_METADATA_PREFIX_LEN (sizeof(BPF_METADATA_PREFIX) - 1)
38	38
39		−const char * const prog_type_name[] = {
40		− [BPF_PROG_TYPE_UNSPEC] = "unspec",
41		− [BPF_PROG_TYPE_SOCKET_FILTER] = "socket_filter",
42		− [BPF_PROG_TYPE_KPROBE] = "kprobe",
43		− [BPF_PROG_TYPE_SCHED_CLS] = "sched_cls",
44		− [BPF_PROG_TYPE_SCHED_ACT] = "sched_act",
45		− [BPF_PROG_TYPE_TRACEPOINT] = "tracepoint",
46		− [BPF_PROG_TYPE_XDP] = "xdp",
47		− [BPF_PROG_TYPE_PERF_EVENT] = "perf_event",
48		− [BPF_PROG_TYPE_CGROUP_SKB] = "cgroup_skb",
49		− [BPF_PROG_TYPE_CGROUP_SOCK] = "cgroup_sock",
50		− [BPF_PROG_TYPE_LWT_IN] = "lwt_in",
51		− [BPF_PROG_TYPE_LWT_OUT] = "lwt_out",
52		− [BPF_PROG_TYPE_LWT_XMIT] = "lwt_xmit",
53		− [BPF_PROG_TYPE_SOCK_OPS] = "sock_ops",
54		− [BPF_PROG_TYPE_SK_SKB] = "sk_skb",
55		− [BPF_PROG_TYPE_CGROUP_DEVICE] = "cgroup_device",
56		− [BPF_PROG_TYPE_SK_MSG] = "sk_msg",
57		− [BPF_PROG_TYPE_RAW_TRACEPOINT] = "raw_tracepoint",
58		− [BPF_PROG_TYPE_CGROUP_SOCK_ADDR] = "cgroup_sock_addr",
59		− [BPF_PROG_TYPE_LWT_SEG6LOCAL] = "lwt_seg6local",
60		− [BPF_PROG_TYPE_LIRC_MODE2] = "lirc_mode2",
61		− [BPF_PROG_TYPE_SK_REUSEPORT] = "sk_reuseport",
62		− [BPF_PROG_TYPE_FLOW_DISSECTOR] = "flow_dissector",
63		− [BPF_PROG_TYPE_CGROUP_SYSCTL] = "cgroup_sysctl",
64		− [BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE] = "raw_tracepoint_writable",
65		− [BPF_PROG_TYPE_CGROUP_SOCKOPT] = "cgroup_sockopt",
66		− [BPF_PROG_TYPE_TRACING] = "tracing",
67		− [BPF_PROG_TYPE_STRUCT_OPS] = "struct_ops",
68		− [BPF_PROG_TYPE_EXT] = "ext",
69		− [BPF_PROG_TYPE_LSM] = "lsm",
70		− [BPF_PROG_TYPE_SK_LOOKUP] = "sk_lookup",
71		− [BPF_PROG_TYPE_SYSCALL] = "syscall",
72		−};
73		−
74		−const size_t prog_type_name_size = ARRAY_SIZE(prog_type_name);
75		−
76	39	enum dump_mode {
77	40	DUMP_JITED,
78	41	DUMP_XLATED,
79	42	};
80	43
	44	+static const bool attach_types[] = {
	45	+ [BPF_SK_SKB_STREAM_PARSER] = true,
	46	+ [BPF_SK_SKB_STREAM_VERDICT] = true,
	47	+ [BPF_SK_SKB_VERDICT] = true,
	48	+ [BPF_SK_MSG_VERDICT] = true,
	49	+ [BPF_FLOW_DISSECTOR] = true,
	50	+ [__MAX_BPF_ATTACH_TYPE] = false,
	51	+};
	52	+
	53	+/* Textual representations traditionally used by the program and kept around
	54	+ * for the sake of backwards compatibility.
	55	+ */
81	56	static const char * const attach_type_strings[] = {
82	57	[BPF_SK_SKB_STREAM_PARSER] = "stream_parser",
83	58	[BPF_SK_SKB_STREAM_VERDICT] = "stream_verdict",
84	59	[BPF_SK_SKB_VERDICT] = "skb_verdict",
85	60	[BPF_SK_MSG_VERDICT] = "msg_verdict",
86		− [BPF_FLOW_DISSECTOR] = "flow_dissector",
87	61	[__MAX_BPF_ATTACH_TYPE] = NULL,
88	62	};
89	63
@@ -94,6 +68,14 @@ static enum bpf_attach_type parse_attach_type(const char *str) · 4 unmodified lines
94	68	enum bpf_attach_type type;
95	69
96	70	for (type = 0; type < __MAX_BPF_ATTACH_TYPE; type++) {
	71	+ if (attach_types[type]) {
	72	+ const char *attach_type_str;
	73	+
	74	+ attach_type_str = libbpf_bpf_attach_type_str(type);
	75	+ if (!strcmp(str, attach_type_str))
	76	+ return type;
	77	+ }
	78	+
97	79	if (attach_type_strings[type] &&
98	80	is_prefix(str, attach_type_strings[type]))
99	81	return type;
@@ -428,12 +410,14 @@ out_free: · 328 unmodified lines
428	410
429	411	static void print_prog_header_json(struct bpf_prog_info *info, int fd)
430	412	{
	413	+ const char *prog_type_str;
431	414	char prog_name[MAX_PROG_FULL_NAME];
432	415
433	416	jsonw_uint_field(json_wtr, "id", info->id);
434		− if (info->type < ARRAY_SIZE(prog_type_name))
435		− jsonw_string_field(json_wtr, "type",
436		− prog_type_name[info->type]);
	417	+ prog_type_str = libbpf_bpf_prog_type_str(info->type);
	418	+
	419	+ if (prog_type_str)
	420	+ jsonw_string_field(json_wtr, "type", prog_type_str);
437	421	else
438	422	jsonw_uint_field(json_wtr, "type", info->type);
439	423
@@ -515,11 +499,13 @@ static void print_prog_json(struct bpf_prog_info *info, int fd) · 75 unmodified lines
515	499
516	500	static void print_prog_header_plain(struct bpf_prog_info *info, int fd)
517	501	{
	502	+ const char *prog_type_str;
518	503	char prog_name[MAX_PROG_FULL_NAME];
519	504
520	505	printf("%u: ", info->id);
521		− if (info->type < ARRAY_SIZE(prog_type_name))
522		− printf("%s ", prog_type_name[info->type]);
	506	+ prog_type_str = libbpf_bpf_prog_type_str(info->type);
	507	+ if (prog_type_str)
	508	+ printf("%s ", prog_type_str);
523	509	else
524	510	printf("type %u ", info->type);
525	511
@@ -1604,6 +1590,8 @@ static int load_with_options(int argc, char **argv, bool first_prog_only) · 1078 unmodified lines
1604	1590	}
1605	1591	}
1606	1592
	1593	+ set_max_rlimit();
	1594	+
1607	1595	if (verifier_logs)
1608	1596	/* log_level1 + log_level2 + stats, but not stable UAPI */
1609	1597	open_opts.kernel_log_level = 1 + 2 + 4;
@@ -2301,6 +2289,7 @@ static int do_profile(int argc, char **argv) · 691 unmodified lines
2301	2289	}
2302	2290	}
2303	2291
	2292	+ set_max_rlimit();
2304	2293	err = profiler_bpf__load(profile_obj);
2305	2294	if (err) {
2306	2295	p_err("failed to load profile_obj");
@@ -2374,8 +2363,8 @@ static int do_help(int argc, char **argv) · 67 unmodified lines
2374	2363	" cgroup/sendmsg6 \| cgroup/recvmsg4 \| cgroup/recvmsg6 \|\n"
2375	2364	" cgroup/getsockopt \| cgroup/setsockopt \| cgroup/sock_release \|\n"
2376	2365	" struct_ops \| fentry \| fexit \| freplace \| sk_lookup }\n"
2377		− " ATTACH_TYPE := { msg_verdict \| skb_verdict \| stream_verdict \|\n"
2378		− " stream_parser \| flow_dissector }\n"
	2366	+ " ATTACH_TYPE := { sk_msg_verdict \| sk_skb_verdict \| sk_skb_stream_verdict \|\n"
	2367	+ " sk_skb_stream_parser \| flow_dissector }\n"
2379	2368	" METRIC := { cycles \| instructions \| l1d_loads \| llc_misses \| itlb_misses \| dtlb_misses }\n"
2380	2369	" " HELP_SPEC_OPTIONS " \|\n"
2381	2370	" {-f\|--bpffs} \| {-m\|--mapcompat} \| {-n\|--nomount} \|\n"

tools/bpf/bpftool/struct_ops.c+2−0

@@ -501,6 +501,8 @@ static int do_register(int argc, char **argv) · 500 unmodified lines
501	501	if (libbpf_get_error(obj))
502	502	return -1;
503	503
	504	+ set_max_rlimit();
	505	+
504	506	if (bpf_object__load(obj)) {
505	507	bpf_object__close(obj);
506	508	return -1;

tools/include/uapi/linux/bpf.h+84−4

@@ -3597,10 +3597,11 @@ union bpf_attr { · 3596 unmodified lines
3597	3597	*
3598	3598	* iph points to the start of the IPv4 or IPv6 header, while
3599	3599	* iph_len contains sizeof\ (struct iphdr) or
3600		− * sizeof\ (struct ip6hdr).
	3600	+ * sizeof\ (struct ipv6hdr).
3601	3601	*
3602	3602	* th points to the start of the TCP header, while th_len
3603		− * contains sizeof\ (struct tcphdr).
	3603	+ * contains the length of the TCP header (at least
	3604	+ * sizeof\ (struct tcphdr)).
3604	3605	* Return
3605	3606	* 0 if iph and th are a valid SYN cookie ACK, or a negative
3606	3607	* error otherwise.
@@ -3783,10 +3784,11 @@ union bpf_attr { · 176 unmodified lines
3783	3784	*
3784	3785	* iph points to the start of the IPv4 or IPv6 header, while
3785	3786	* iph_len contains sizeof\ (struct iphdr) or
3786		− * sizeof\ (struct ip6hdr).
	3787	+ * sizeof\ (struct ipv6hdr).
3787	3788	*
3788	3789	* th points to the start of the TCP header, while th_len
3789		− * contains the length of the TCP header.
	3790	+ * contains the length of the TCP header with options (at least
	3791	+ * sizeof\ (struct tcphdr)).
3790	3792	* Return
3791	3793	* On success, lower 32 bits hold the generated SYN cookie in
3792	3794	* followed by 16 bits which hold the MSS value for that cookie,
@@ -5249,6 +5251,80 @@ union bpf_attr { · 1456 unmodified lines
5249	5251	* Pointer to the underlying dynptr data, NULL if the dynptr is
5250	5252	* read-only, if the dynptr is invalid, or if the offset and length
5251	5253	* is out of bounds.
	5254	+ *
	5255	+ * s64 bpf_tcp_raw_gen_syncookie_ipv4(struct iphdr iph, struct tcphdr th, u32 th_len)
	5256	+ * Description
	5257	+ * Try to issue a SYN cookie for the packet with corresponding
	5258	+ * IPv4/TCP headers, iph and th, without depending on a
	5259	+ * listening socket.
	5260	+ *
	5261	+ * iph points to the IPv4 header.
	5262	+ *
	5263	+ * th points to the start of the TCP header, while th_len
	5264	+ * contains the length of the TCP header (at least
	5265	+ * sizeof\ (struct tcphdr)).
	5266	+ * Return
	5267	+ * On success, lower 32 bits hold the generated SYN cookie in
	5268	+ * followed by 16 bits which hold the MSS value for that cookie,
	5269	+ * and the top 16 bits are unused.
	5270	+ *
	5271	+ * On failure, the returned value is one of the following:
	5272	+ *
	5273	+ * -EINVAL if th_len is invalid.
	5274	+ *
	5275	+ * s64 bpf_tcp_raw_gen_syncookie_ipv6(struct ipv6hdr iph, struct tcphdr th, u32 th_len)
	5276	+ * Description
	5277	+ * Try to issue a SYN cookie for the packet with corresponding
	5278	+ * IPv6/TCP headers, iph and th, without depending on a
	5279	+ * listening socket.
	5280	+ *
	5281	+ * iph points to the IPv6 header.
	5282	+ *
	5283	+ * th points to the start of the TCP header, while th_len
	5284	+ * contains the length of the TCP header (at least
	5285	+ * sizeof\ (struct tcphdr)).
	5286	+ * Return
	5287	+ * On success, lower 32 bits hold the generated SYN cookie in
	5288	+ * followed by 16 bits which hold the MSS value for that cookie,
	5289	+ * and the top 16 bits are unused.
	5290	+ *
	5291	+ * On failure, the returned value is one of the following:
	5292	+ *
	5293	+ * -EINVAL if th_len is invalid.
	5294	+ *
	5295	+ * -EPROTONOSUPPORT if CONFIG_IPV6 is not builtin.
	5296	+ *
	5297	+ * long bpf_tcp_raw_check_syncookie_ipv4(struct iphdr iph, struct tcphdr th)
	5298	+ * Description
	5299	+ * Check whether iph and th contain a valid SYN cookie ACK
	5300	+ * without depending on a listening socket.
	5301	+ *
	5302	+ * iph points to the IPv4 header.
	5303	+ *
	5304	+ * th points to the TCP header.
	5305	+ * Return
	5306	+ * 0 if iph and th are a valid SYN cookie ACK.
	5307	+ *
	5308	+ * On failure, the returned value is one of the following:
	5309	+ *
	5310	+ * -EACCES if the SYN cookie is not valid.
	5311	+ *
	5312	+ * long bpf_tcp_raw_check_syncookie_ipv6(struct ipv6hdr iph, struct tcphdr th)
	5313	+ * Description
	5314	+ * Check whether iph and th contain a valid SYN cookie ACK
	5315	+ * without depending on a listening socket.
	5316	+ *
	5317	+ * iph points to the IPv6 header.
	5318	+ *
	5319	+ * th points to the TCP header.
	5320	+ * Return
	5321	+ * 0 if iph and th are a valid SYN cookie ACK.
	5322	+ *
	5323	+ * On failure, the returned value is one of the following:
	5324	+ *
	5325	+ * -EACCES if the SYN cookie is not valid.
	5326	+ *
	5327	+ * -EPROTONOSUPPORT if CONFIG_IPV6 is not builtin.
5252	5328	*/
5253	5329	#define __BPF_FUNC_MAPPER(FN) \
5254	5330	FN(unspec), \
@@ -5455,6 +5531,10 @@ union bpf_attr { · 200 unmodified lines
5455	5531	FN(dynptr_read), \
5456	5532	FN(dynptr_write), \
5457	5533	FN(dynptr_data), \
	5534	+ FN(tcp_raw_gen_syncookie_ipv4), \
	5535	+ FN(tcp_raw_gen_syncookie_ipv6), \
	5536	+ FN(tcp_raw_check_syncookie_ipv4), \
	5537	+ FN(tcp_raw_check_syncookie_ipv6), \
5458	5538	/* */
5459	5539
5460	5540	/* integer value in 'imm' field of BPF_CALL instruction selects which helper

tools/include/uapi/linux/btf.h+14−3

@@ -36,10 +36,10 @@ struct btf_type { · 35 unmodified lines
36	36	* bits 24-28: kind (e.g. int, ptr, array...etc)
37	37	* bits 29-30: unused
38	38	* bit 31: kind_flag, currently used by
39		− * struct, union and fwd
	39	+ * struct, union, enum, fwd and enum64
40	40	*/
41	41	__u32 info;
42		− /* "size" is used by INT, ENUM, STRUCT, UNION and DATASEC.
	42	+ /* "size" is used by INT, ENUM, STRUCT, UNION, DATASEC and ENUM64.
43	43	* "size" tells the size of the type it is describing.
44	44	*
45	45	* "type" is used by PTR, TYPEDEF, VOLATILE, CONST, RESTRICT,
@@ -63,7 +63,7 @@ enum { · 17 unmodified lines
63	63	BTF_KIND_ARRAY = 3, /* Array */
64	64	BTF_KIND_STRUCT = 4, /* Struct */
65	65	BTF_KIND_UNION = 5, /* Union */
66		− BTF_KIND_ENUM = 6, /* Enumeration */
	66	+ BTF_KIND_ENUM = 6, /* Enumeration up to 32-bit values */
67	67	BTF_KIND_FWD = 7, /* Forward */
68	68	BTF_KIND_TYPEDEF = 8, /* Typedef */
69	69	BTF_KIND_VOLATILE = 9, /* Volatile */
@@ -76,6 +76,7 @@ enum { · 6 unmodified lines
76	76	BTF_KIND_FLOAT = 16, /* Floating point */
77	77	BTF_KIND_DECL_TAG = 17, /* Decl Tag */
78	78	BTF_KIND_TYPE_TAG = 18, /* Type Tag */
	79	+ BTF_KIND_ENUM64 = 19, /* Enumeration up to 64-bit values */
79	80
80	81	NR_BTF_KINDS,
81	82	BTF_KIND_MAX = NR_BTF_KINDS - 1,
@@ -186,4 +187,14 @@ struct btf_decl_tag { · 104 unmodified lines
186	187	__s32 component_idx;
187	188	};
188	189
	190	+/* BTF_KIND_ENUM64 is followed by multiple "struct btf_enum64".
	191	+ * The exact number of btf_enum64 is stored in the vlen (of the
	192	+ * info in "struct btf_type").
	193	+ */
	194	+struct btf_enum64 {
	195	+ __u32 name_off;
	196	+ __u32 val_lo32;
	197	+ __u32 val_hi32;
	198	+};
	199	+
189	200	#endif /* _UAPI__LINUX_BTF_H__ */

tools/lib/bpf/btf.c+205−24

@@ -130,7 +130,7 @@ static inline __u64 ptr_to_u64(const void *ptr) · 129 unmodified lines
130	130
131	131	/* Ensure given dynamically allocated memory region pointed to by data with
132	132	* capacity of cap_cnt elements each taking elem_sz bytes has enough
133		− * memory to accomodate add_cnt new elements, assuming cur_cnt elements
	133	+ * memory to accommodate add_cnt new elements, assuming cur_cnt elements
134	134	* are already used. At most max_cnt elements can be ever allocated.
135	135	* If necessary, memory is reallocated and all existing data is copied over,
136	136	* new pointer to the memory region is stored at *data, new memory region
@@ -305,6 +305,8 @@ static int btf_type_size(const struct btf_type *t) · 168 unmodified lines
305	305	return base_size + sizeof(__u32);
306	306	case BTF_KIND_ENUM:
307	307	return base_size + vlen * sizeof(struct btf_enum);
	308	+ case BTF_KIND_ENUM64:
	309	+ return base_size + vlen * sizeof(struct btf_enum64);
308	310	case BTF_KIND_ARRAY:
309	311	return base_size + sizeof(struct btf_array);
310	312	case BTF_KIND_STRUCT:
@@ -334,6 +336,7 @@ static void btf_bswap_type_base(struct btf_type *t) · 23 unmodified lines
334	336	static int btf_bswap_type_rest(struct btf_type *t)
335	337	{
336	338	struct btf_var_secinfo *v;
	339	+ struct btf_enum64 *e64;
337	340	struct btf_member *m;
338	341	struct btf_array *a;
339	342	struct btf_param *p;
@@ -361,6 +364,13 @@ static int btf_bswap_type_rest(struct btf_type *t) · 21 unmodified lines
361	364	e->val = bswap_32(e->val);
362	365	}
363	366	return 0;
	367	+ case BTF_KIND_ENUM64:
	368	+ for (i = 0, e64 = btf_enum64(t); i < vlen; i++, e64++) {
	369	+ e64->name_off = bswap_32(e64->name_off);
	370	+ e64->val_lo32 = bswap_32(e64->val_lo32);
	371	+ e64->val_hi32 = bswap_32(e64->val_hi32);
	372	+ }
	373	+ return 0;
364	374	case BTF_KIND_ARRAY:
365	375	a = btf_array(t);
366	376	a->type = bswap_32(a->type);
@@ -472,9 +482,22 @@ const struct btf_type btf__type_by_id(const struct btf btf, __u32 type_id) · 105 unmodified lines
472	482
473	483	static int determine_ptr_size(const struct btf *btf)
474	484	{
	485	+ static const char * const long_aliases[] = {
	486	+ "long",
	487	+ "long int",
	488	+ "int long",
	489	+ "unsigned long",
	490	+ "long unsigned",
	491	+ "unsigned long int",
	492	+ "unsigned int long",
	493	+ "long unsigned int",
	494	+ "long int unsigned",
	495	+ "int unsigned long",
	496	+ "int long unsigned",
	497	+ };
475	498	const struct btf_type *t;
476	499	const char *name;
477		− int i, n;
	500	+ int i, j, n;
478	501
479	502	if (btf->base_btf && btf->base_btf->ptr_sz > 0)
480	503	return btf->base_btf->ptr_sz;
@@ -485,15 +508,16 @@ static int determine_ptr_size(const struct btf *btf) · 4 unmodified lines
485	508	if (!btf_is_int(t))
486	509	continue;
487	510
	511	+ if (t->size != 4 && t->size != 8)
	512	+ continue;
	513	+
488	514	name = btf__name_by_offset(btf, t->name_off);
489	515	if (!name)
490	516	continue;
491	517
492		− if (strcmp(name, "long int") == 0 \|\|
493		− strcmp(name, "long unsigned int") == 0) {
494		− if (t->size != 4 && t->size != 8)
495		− continue;
496		− return t->size;
	518	+ for (j = 0; j < ARRAY_SIZE(long_aliases); j++) {
	519	+ if (strcmp(name, long_aliases[j]) == 0)
	520	+ return t->size;
497	521	}
498	522	}
499	523
@@ -597,6 +621,7 @@ __s64 btf__resolve_size(const struct btf *btf, __u32 type_id) · 97 unmodified lines
597	621	case BTF_KIND_STRUCT:
598	622	case BTF_KIND_UNION:
599	623	case BTF_KIND_ENUM:
	624	+ case BTF_KIND_ENUM64:
600	625	case BTF_KIND_DATASEC:
601	626	case BTF_KIND_FLOAT:
602	627	size = t->size;
@@ -644,6 +669,7 @@ int btf__align_of(const struct btf *btf, __u32 id) · 41 unmodified lines
644	669	switch (kind) {
645	670	case BTF_KIND_INT:
646	671	case BTF_KIND_ENUM:
	672	+ case BTF_KIND_ENUM64:
647	673	case BTF_KIND_FLOAT:
648	674	return min(btf_ptr_sz(btf), (size_t)t->size);
649	675	case BTF_KIND_PTR:
@@ -2115,20 +2141,8 @@ int btf__add_field(struct btf btf, const char name, int type_id, · 1465 unmodified lines
2115	2141	return 0;
2116	2142	}
2117	2143
2118		−/*
2119		− * Append new BTF_KIND_ENUM type with:
2120		− * - name - name of the enum, can be NULL or empty for anonymous enums;
2121		− * - byte_sz - size of the enum, in bytes.
2122		− *
2123		− * Enum initially has no enum values in it (and corresponds to enum forward
2124		− * declaration). Enumerator values can be added by btf__add_enum_value()
2125		− * immediately after btf__add_enum() succeeds.
2126		− *
2127		− * Returns:
2128		− * - >0, type ID of newly added BTF type;
2129		− * - <0, on error.
2130		− */
2131		−int btf__add_enum(struct btf btf, const char name, __u32 byte_sz)
	2144	+static int btf_add_enum_common(struct btf btf, const char name, __u32 byte_sz,
	2145	+ bool is_signed, __u8 kind)
2132	2146	{
2133	2147	struct btf_type *t;
2134	2148	int sz, name_off = 0;
@@ -2153,12 +2167,34 @@ int btf__add_enum(struct btf btf, const char name, __u32 byte_sz) · 18 unmodified lines
2153	2167
2154	2168	/* start out with vlen=0; it will be adjusted when adding enum values */
2155	2169	t->name_off = name_off;
2156		− t->info = btf_type_info(BTF_KIND_ENUM, 0, 0);
	2170	+ t->info = btf_type_info(kind, 0, is_signed);
2157	2171	t->size = byte_sz;
2158	2172
2159	2173	return btf_commit_type(btf, sz);
2160	2174	}
2161	2175
	2176	+/*
	2177	+ * Append new BTF_KIND_ENUM type with:
	2178	+ * - name - name of the enum, can be NULL or empty for anonymous enums;
	2179	+ * - byte_sz - size of the enum, in bytes.
	2180	+ *
	2181	+ * Enum initially has no enum values in it (and corresponds to enum forward
	2182	+ * declaration). Enumerator values can be added by btf__add_enum_value()
	2183	+ * immediately after btf__add_enum() succeeds.
	2184	+ *
	2185	+ * Returns:
	2186	+ * - >0, type ID of newly added BTF type;
	2187	+ * - <0, on error.
	2188	+ */
	2189	+int btf__add_enum(struct btf btf, const char name, __u32 byte_sz)
	2190	+{
	2191	+ /*
	2192	+ * set the signedness to be unsigned, it will change to signed
	2193	+ * if any later enumerator is negative.
	2194	+ */
	2195	+ return btf_add_enum_common(btf, name, byte_sz, false, BTF_KIND_ENUM);
	2196	+}
	2197	+
2162	2198	/*
2163	2199	* Append new enum value for the current ENUM type with:
2164	2200	* - name - name of the enumerator value, can't be NULL or empty;
@@ -2206,6 +2242,82 @@ int btf__add_enum_value(struct btf btf, const char name, __s64 value) · 41 unmodified lines
2206	2242	t = btf_last_type(btf);
2207	2243	btf_type_inc_vlen(t);
2208	2244
	2245	+ /* if negative value, set signedness to signed */
	2246	+ if (value < 0)
	2247	+ t->info = btf_type_info(btf_kind(t), btf_vlen(t), true);
	2248	+
	2249	+ btf->hdr->type_len += sz;
	2250	+ btf->hdr->str_off += sz;
	2251	+ return 0;
	2252	+}
	2253	+
	2254	+/*
	2255	+ * Append new BTF_KIND_ENUM64 type with:
	2256	+ * - name - name of the enum, can be NULL or empty for anonymous enums;
	2257	+ * - byte_sz - size of the enum, in bytes.
	2258	+ * - is_signed - whether the enum values are signed or not;
	2259	+ *
	2260	+ * Enum initially has no enum values in it (and corresponds to enum forward
	2261	+ * declaration). Enumerator values can be added by btf__add_enum64_value()
	2262	+ * immediately after btf__add_enum64() succeeds.
	2263	+ *
	2264	+ * Returns:
	2265	+ * - >0, type ID of newly added BTF type;
	2266	+ * - <0, on error.
	2267	+ */
	2268	+int btf__add_enum64(struct btf btf, const char name, __u32 byte_sz,
	2269	+ bool is_signed)
	2270	+{
	2271	+ return btf_add_enum_common(btf, name, byte_sz, is_signed,
	2272	+ BTF_KIND_ENUM64);
	2273	+}
	2274	+
	2275	+/*
	2276	+ * Append new enum value for the current ENUM64 type with:
	2277	+ * - name - name of the enumerator value, can't be NULL or empty;
	2278	+ * - value - integer value corresponding to enum value name;
	2279	+ * Returns:
	2280	+ * - 0, on success;
	2281	+ * - <0, on error.
	2282	+ */
	2283	+int btf__add_enum64_value(struct btf btf, const char name, __u64 value)
	2284	+{
	2285	+ struct btf_enum64 *v;
	2286	+ struct btf_type *t;
	2287	+ int sz, name_off;
	2288	+
	2289	+ /* last type should be BTF_KIND_ENUM64 */
	2290	+ if (btf->nr_types == 0)
	2291	+ return libbpf_err(-EINVAL);
	2292	+ t = btf_last_type(btf);
	2293	+ if (!btf_is_enum64(t))
	2294	+ return libbpf_err(-EINVAL);
	2295	+
	2296	+ /* non-empty name */
	2297	+ if (!name \|\| !name[0])
	2298	+ return libbpf_err(-EINVAL);
	2299	+
	2300	+ /* decompose and invalidate raw data */
	2301	+ if (btf_ensure_modifiable(btf))
	2302	+ return libbpf_err(-ENOMEM);
	2303	+
	2304	+ sz = sizeof(struct btf_enum64);
	2305	+ v = btf_add_type_mem(btf, sz);
	2306	+ if (!v)
	2307	+ return libbpf_err(-ENOMEM);
	2308	+
	2309	+ name_off = btf__add_str(btf, name);
	2310	+ if (name_off < 0)
	2311	+ return name_off;
	2312	+
	2313	+ v->name_off = name_off;
	2314	+ v->val_lo32 = (__u32)value;
	2315	+ v->val_hi32 = value >> 32;
	2316	+
	2317	+ /* update parent type's vlen */
	2318	+ t = btf_last_type(btf);
	2319	+ btf_type_inc_vlen(t);
	2320	+
2209	2321	btf->hdr->type_len += sz;
2210	2322	btf->hdr->str_off += sz;
2211	2323	return 0;
@@ -3470,7 +3582,7 @@ static bool btf_equal_int_tag(struct btf_type t1, struct btf_type t2) · 1258 unmodified lines
3470	3582	return info1 == info2;
3471	3583	}
3472	3584
3473		−/* Calculate type signature hash of ENUM. */
	3585	+/* Calculate type signature hash of ENUM/ENUM64. */
3474	3586	static long btf_hash_enum(struct btf_type *t)
3475	3587	{
3476	3588	long h;
@@ -3504,9 +3616,31 @@ static bool btf_equal_enum(struct btf_type t1, struct btf_type t2) · 27 unmodified lines
3504	3616	return true;
3505	3617	}
3506	3618
	3619	+static bool btf_equal_enum64(struct btf_type t1, struct btf_type t2)
	3620	+{
	3621	+ const struct btf_enum64 m1, m2;
	3622	+ __u16 vlen;
	3623	+ int i;
	3624	+
	3625	+ if (!btf_equal_common(t1, t2))
	3626	+ return false;
	3627	+
	3628	+ vlen = btf_vlen(t1);
	3629	+ m1 = btf_enum64(t1);
	3630	+ m2 = btf_enum64(t2);
	3631	+ for (i = 0; i < vlen; i++) {
	3632	+ if (m1->name_off != m2->name_off \|\| m1->val_lo32 != m2->val_lo32 \|\|
	3633	+ m1->val_hi32 != m2->val_hi32)
	3634	+ return false;
	3635	+ m1++;
	3636	+ m2++;
	3637	+ }
	3638	+ return true;
	3639	+}
	3640	+
3507	3641	static inline bool btf_is_enum_fwd(struct btf_type *t)
3508	3642	{
3509		− return btf_is_enum(t) && btf_vlen(t) == 0;
	3643	+ return btf_is_any_enum(t) && btf_vlen(t) == 0;
3510	3644	}
3511	3645
3512	3646	static bool btf_compat_enum(struct btf_type t1, struct btf_type t2)
@@ -3519,6 +3653,17 @@ static bool btf_compat_enum(struct btf_type t1, struct btf_type t2) · 6 unmodified lines
3519	3653	t1->size == t2->size;
3520	3654	}
3521	3655
	3656	+static bool btf_compat_enum64(struct btf_type t1, struct btf_type t2)
	3657	+{
	3658	+ if (!btf_is_enum_fwd(t1) && !btf_is_enum_fwd(t2))
	3659	+ return btf_equal_enum64(t1, t2);
	3660	+
	3661	+ /* ignore vlen when comparing */
	3662	+ return t1->name_off == t2->name_off &&
	3663	+ (t1->info & ~0xffff) == (t2->info & ~0xffff) &&
	3664	+ t1->size == t2->size;
	3665	+}
	3666	+
3522	3667	/*
3523	3668	* Calculate type signature hash of STRUCT/UNION, ignoring referenced type IDs,
3524	3669	* as referenced type IDs equivalence is established separately during type
@@ -3731,6 +3876,7 @@ static int btf_dedup_prep(struct btf_dedup *d) · 206 unmodified lines
3731	3876	h = btf_hash_int_decl_tag(t);
3732	3877	break;
3733	3878	case BTF_KIND_ENUM:
	3879	+ case BTF_KIND_ENUM64:
3734	3880	h = btf_hash_enum(t);
3735	3881	break;
3736	3882	case BTF_KIND_STRUCT:
@@ -3820,6 +3966,27 @@ static int btf_dedup_prim_type(struct btf_dedup *d, __u32 type_id) · 83 unmodified lines
3820	3966	}
3821	3967	break;
3822	3968
	3969	+ case BTF_KIND_ENUM64:
	3970	+ h = btf_hash_enum(t);
	3971	+ for_each_dedup_cand(d, hash_entry, h) {
	3972	+ cand_id = (__u32)(long)hash_entry->value;
	3973	+ cand = btf_type_by_id(d->btf, cand_id);
	3974	+ if (btf_equal_enum64(t, cand)) {
	3975	+ new_id = cand_id;
	3976	+ break;
	3977	+ }
	3978	+ if (btf_compat_enum64(t, cand)) {
	3979	+ if (btf_is_enum_fwd(t)) {
	3980	+ /* resolve fwd to full enum */
	3981	+ new_id = cand_id;
	3982	+ break;
	3983	+ }
	3984	+ /* resolve canonical enum fwd to full enum */
	3985	+ d->map[cand_id] = type_id;
	3986	+ }
	3987	+ }
	3988	+ break;
	3989	+
3823	3990	case BTF_KIND_FWD:
3824	3991	case BTF_KIND_FLOAT:
3825	3992	h = btf_hash_common(t);
@@ -4115,6 +4282,9 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id, · 289 unmodified lines
4115	4282	case BTF_KIND_ENUM:
4116	4283	return btf_compat_enum(cand_type, canon_type);
4117	4284
	4285	+ case BTF_KIND_ENUM64:
	4286	+ return btf_compat_enum64(cand_type, canon_type);
	4287	+
4118	4288	case BTF_KIND_FWD:
4119	4289	case BTF_KIND_FLOAT:
4120	4290	return btf_equal_common(cand_type, canon_type);
@@ -4717,6 +4887,7 @@ int btf_type_visit_type_ids(struct btf_type t, type_id_visit_fn visit, void ct · 596 unmodified lines
4717	4887	case BTF_KIND_INT:
4718	4888	case BTF_KIND_FLOAT:
4719	4889	case BTF_KIND_ENUM:
	4890	+ case BTF_KIND_ENUM64:
4720	4891	return 0;
4721	4892
4722	4893	case BTF_KIND_FWD:
@@ -4811,6 +4982,16 @@ int btf_type_visit_str_offs(struct btf_type t, str_off_visit_fn visit, void ct · 88 unmodified lines
4811	4982	}
4812	4983	break;
4813	4984	}
	4985	+ case BTF_KIND_ENUM64: {
	4986	+ struct btf_enum64 *m = btf_enum64(t);
	4987	+
	4988	+ for (i = 0, n = btf_vlen(t); i < n; i++, m++) {
	4989	+ err = visit(&m->name_off, ctx);
	4990	+ if (err)
	4991	+ return err;
	4992	+ }
	4993	+ break;
	4994	+ }
4814	4995	case BTF_KIND_FUNC_PROTO: {
4815	4996	struct btf_param *m = btf_params(t);
4816	4997

tools/lib/bpf/btf.h+31−1

@@ -215,6 +215,8 @@ LIBBPF_API int btf__add_field(struct btf btf, const char name, int field_type_ · 214 unmodified lines
215	215	/* enum construction APIs */
216	216	LIBBPF_API int btf__add_enum(struct btf btf, const char name, __u32 bytes_sz);
217	217	LIBBPF_API int btf__add_enum_value(struct btf btf, const char name, __s64 value);
	218	+LIBBPF_API int btf__add_enum64(struct btf btf, const char name, __u32 bytes_sz, bool is_signed);
	219	+LIBBPF_API int btf__add_enum64_value(struct btf btf, const char name, __u64 value);
218	220
219	221	enum btf_fwd_kind {
220	222	BTF_FWD_STRUCT = 0,
@@ -393,9 +395,10 @@ btf_dump__dump_type_data(struct btf_dump *d, __u32 id, · 172 unmodified lines
393	395	#ifndef BTF_KIND_FLOAT
394	396	#define BTF_KIND_FLOAT 16 /* Floating point */
395	397	#endif
396		−/* The kernel header switched to enums, so these two were never #defined */
	398	+/* The kernel header switched to enums, so the following were never #defined */
397	399	#define BTF_KIND_DECL_TAG 17 /* Decl Tag */
398	400	#define BTF_KIND_TYPE_TAG 18 /* Type Tag */
	401	+#define BTF_KIND_ENUM64 19 /* Enum for up-to 64bit values */
399	402
400	403	static inline __u16 btf_kind(const struct btf_type *t)
401	404	{
@@ -454,6 +457,11 @@ static inline bool btf_is_enum(const struct btf_type *t) · 52 unmodified lines
454	457	return btf_kind(t) == BTF_KIND_ENUM;
455	458	}
456	459
	460	+static inline bool btf_is_enum64(const struct btf_type *t)
	461	+{
	462	+ return btf_kind(t) == BTF_KIND_ENUM64;
	463	+}
	464	+
457	465	static inline bool btf_is_fwd(const struct btf_type *t)
458	466	{
459	467	return btf_kind(t) == BTF_KIND_FWD;
@@ -524,6 +532,18 @@ static inline bool btf_is_type_tag(const struct btf_type *t) · 64 unmodified lines
524	532	return btf_kind(t) == BTF_KIND_TYPE_TAG;
525	533	}
526	534
	535	+static inline bool btf_is_any_enum(const struct btf_type *t)
	536	+{
	537	+ return btf_is_enum(t) \|\| btf_is_enum64(t);
	538	+}
	539	+
	540	+static inline bool btf_kind_core_compat(const struct btf_type *t1,
	541	+ const struct btf_type *t2)
	542	+{
	543	+ return btf_kind(t1) == btf_kind(t2) \|\|
	544	+ (btf_is_any_enum(t1) && btf_is_any_enum(t2));
	545	+}
	546	+
527	547	static inline __u8 btf_int_encoding(const struct btf_type *t)
528	548	{
529	549	return BTF_INT_ENCODING((__u32 )(t + 1));
@@ -549,6 +569,16 @@ static inline struct btf_enum btf_enum(const struct btf_type t) · 19 unmodified lines
549	569	return (struct btf_enum *)(t + 1);
550	570	}
551	571
	572	+static inline struct btf_enum64 btf_enum64(const struct btf_type t)
	573	+{
	574	+ return (struct btf_enum64 *)(t + 1);
	575	+}
	576	+
	577	+static inline __u64 btf_enum64_value(const struct btf_enum64 *e)
	578	+{
	579	+ return ((__u64)e->val_hi32 << 32) \| e->val_lo32;
	580	+}
	581	+
552	582	static inline struct btf_member btf_members(const struct btf_type t)
553	583	{
554	584	return (struct btf_member *)(t + 1);

tools/lib/bpf/btf_dump.c+103−34

@@ -318,6 +318,7 @@ static int btf_dump_mark_referenced(struct btf_dump *d) · 317 unmodified lines
318	318	switch (btf_kind(t)) {
319	319	case BTF_KIND_INT:
320	320	case BTF_KIND_ENUM:
	321	+ case BTF_KIND_ENUM64:
321	322	case BTF_KIND_FWD:
322	323	case BTF_KIND_FLOAT:
323	324	break;
@@ -538,6 +539,7 @@ static int btf_dump_order_type(struct btf_dump *d, __u32 id, bool through_ptr) · 214 unmodified lines
538	539	return 1;
539	540	}
540	541	case BTF_KIND_ENUM:
	542	+ case BTF_KIND_ENUM64:
541	543	case BTF_KIND_FWD:
542	544	/*
543	545	* non-anonymous or non-referenced enums are top-level
@@ -739,6 +741,7 @@ static void btf_dump_emit_type(struct btf_dump *d, __u32 id, __u32 cont_id) · 195 unmodified lines
739	741	tstate->emit_state = EMITTED;
740	742	break;
741	743	case BTF_KIND_ENUM:
	744	+ case BTF_KIND_ENUM64:
742	745	if (top_level_def) {
743	746	btf_dump_emit_enum_def(d, id, t, 0);
744	747	btf_dump_printf(d, ";\n\n");
@@ -989,38 +992,81 @@ static void btf_dump_emit_enum_fwd(struct btf_dump *d, __u32 id, · 244 unmodified lines
989	992	btf_dump_printf(d, "enum %s", btf_dump_type_name(d, id));
990	993	}
991	994
992		−static void btf_dump_emit_enum_def(struct btf_dump *d, __u32 id,
993		− const struct btf_type *t,
994		− int lvl)
	995	+static void btf_dump_emit_enum32_val(struct btf_dump *d,
	996	+ const struct btf_type *t,
	997	+ int lvl, __u16 vlen)
995	998	{
996	999	const struct btf_enum *v = btf_enum(t);
997		− __u16 vlen = btf_vlen(t);
	1000	+ bool is_signed = btf_kflag(t);
	1001	+ const char *fmt_str;
998	1002	const char *name;
999	1003	size_t dup_cnt;
1000	1004	int i;
1001	1005
	1006	+ for (i = 0; i < vlen; i++, v++) {
	1007	+ name = btf_name_of(d, v->name_off);
	1008	+ /* enumerators share namespace with typedef idents */
	1009	+ dup_cnt = btf_dump_name_dups(d, d->ident_names, name);
	1010	+ if (dup_cnt > 1) {
	1011	+ fmt_str = is_signed ? "\n%s%s___%zd = %d," : "\n%s%s___%zd = %u,";
	1012	+ btf_dump_printf(d, fmt_str, pfx(lvl + 1), name, dup_cnt, v->val);
	1013	+ } else {
	1014	+ fmt_str = is_signed ? "\n%s%s = %d," : "\n%s%s = %u,";
	1015	+ btf_dump_printf(d, fmt_str, pfx(lvl + 1), name, v->val);
	1016	+ }
	1017	+ }
	1018	+}
	1019	+
	1020	+static void btf_dump_emit_enum64_val(struct btf_dump *d,
	1021	+ const struct btf_type *t,
	1022	+ int lvl, __u16 vlen)
	1023	+{
	1024	+ const struct btf_enum64 *v = btf_enum64(t);
	1025	+ bool is_signed = btf_kflag(t);
	1026	+ const char *fmt_str;
	1027	+ const char *name;
	1028	+ size_t dup_cnt;
	1029	+ __u64 val;
	1030	+ int i;
	1031	+
	1032	+ for (i = 0; i < vlen; i++, v++) {
	1033	+ name = btf_name_of(d, v->name_off);
	1034	+ dup_cnt = btf_dump_name_dups(d, d->ident_names, name);
	1035	+ val = btf_enum64_value(v);
	1036	+ if (dup_cnt > 1) {
	1037	+ fmt_str = is_signed ? "\n%s%s___%zd = %lldLL,"
	1038	+ : "\n%s%s___%zd = %lluULL,";
	1039	+ btf_dump_printf(d, fmt_str,
	1040	+ pfx(lvl + 1), name, dup_cnt,
	1041	+ (unsigned long long)val);
	1042	+ } else {
	1043	+ fmt_str = is_signed ? "\n%s%s = %lldLL,"
	1044	+ : "\n%s%s = %lluULL,";
	1045	+ btf_dump_printf(d, fmt_str,
	1046	+ pfx(lvl + 1), name,
	1047	+ (unsigned long long)val);
	1048	+ }
	1049	+ }
	1050	+}
	1051	+static void btf_dump_emit_enum_def(struct btf_dump *d, __u32 id,
	1052	+ const struct btf_type *t,
	1053	+ int lvl)
	1054	+{
	1055	+ __u16 vlen = btf_vlen(t);
	1056	+
1002	1057	btf_dump_printf(d, "enum%s%s",
1003	1058	t->name_off ? " " : "",
1004	1059	btf_dump_type_name(d, id));
1005	1060
1006		− if (vlen) {
1007		− btf_dump_printf(d, " {");
1008		− for (i = 0; i < vlen; i++, v++) {
1009		− name = btf_name_of(d, v->name_off);
1010		− /* enumerators share namespace with typedef idents */
1011		− dup_cnt = btf_dump_name_dups(d, d->ident_names, name);
1012		− if (dup_cnt > 1) {
1013		− btf_dump_printf(d, "\n%s%s___%zu = %u,",
1014		− pfx(lvl + 1), name, dup_cnt,
1015		− (__u32)v->val);
1016		− } else {
1017		− btf_dump_printf(d, "\n%s%s = %u,",
1018		− pfx(lvl + 1), name,
1019		− (__u32)v->val);
1020		− }
1021		− }
1022		− btf_dump_printf(d, "\n%s}", pfx(lvl));
1023		− }
	1061	+ if (!vlen)
	1062	+ return;
	1063	+
	1064	+ btf_dump_printf(d, " {");
	1065	+ if (btf_is_enum(t))
	1066	+ btf_dump_emit_enum32_val(d, t, lvl, vlen);
	1067	+ else
	1068	+ btf_dump_emit_enum64_val(d, t, lvl, vlen);
	1069	+ btf_dump_printf(d, "\n%s}", pfx(lvl));
1024	1070	}
1025	1071
1026	1072	static void btf_dump_emit_fwd_def(struct btf_dump *d, __u32 id,
@@ -1178,6 +1224,7 @@ skip_mod: · 151 unmodified lines
1178	1224	break;
1179	1225	case BTF_KIND_INT:
1180	1226	case BTF_KIND_ENUM:
	1227	+ case BTF_KIND_ENUM64:
1181	1228	case BTF_KIND_FWD:
1182	1229	case BTF_KIND_STRUCT:
1183	1230	case BTF_KIND_UNION:
@@ -1312,6 +1359,7 @@ static void btf_dump_emit_type_chain(struct btf_dump *d, · 128 unmodified lines
1312	1359	btf_dump_emit_struct_fwd(d, id, t);
1313	1360	break;
1314	1361	case BTF_KIND_ENUM:
	1362	+ case BTF_KIND_ENUM64:
1315	1363	btf_dump_emit_mods(d, decls);
1316	1364	/* inline anonymous enum */
1317	1365	if (t->name_off == 0 && !d->skip_anon_defs)
@@ -1988,7 +2036,8 @@ static int btf_dump_get_enum_value(struct btf_dump *d, · 670 unmodified lines
1988	2036	__u32 id,
1989	2037	__s64 *value)
1990	2038	{
1991		− /* handle unaligned enum value */
	2039	+ bool is_signed = btf_kflag(t);
	2040	+
1992	2041	if (!ptr_is_aligned(d->btf, id, data)) {
1993	2042	__u64 val;
1994	2043	int err;
@@ -2005,13 +2054,13 @@ static int btf_dump_get_enum_value(struct btf_dump *d, · 10 unmodified lines
2005	2054	value = (__s64 *)data;
2006	2055	return 0;
2007	2056	case 4:
2008		− value = (__s32 *)data;
	2057	+ value = is_signed ? (__s32 )data : (__u32 *)data;
2009	2058	return 0;
2010	2059	case 2:
2011		− value = (__s16 *)data;
	2060	+ value = is_signed ? (__s16 )data : (__u16 *)data;
2012	2061	return 0;
2013	2062	case 1:
2014		− value = (__s8 *)data;
	2063	+ value = is_signed ? (__s8 )data : (__u8 *)data;
2015	2064	return 0;
2016	2065	default:
2017	2066	pr_warn("unexpected size %d for enum, id:[%u]\n", t->size, id);
@@ -2024,7 +2073,7 @@ static int btf_dump_enum_data(struct btf_dump *d, · 6 unmodified lines
2024	2073	__u32 id,
2025	2074	const void *data)
2026	2075	{
2027		− const struct btf_enum *e;
	2076	+ bool is_signed;
2028	2077	__s64 value;
2029	2078	int i, err;
2030	2079
@@ -2032,14 +2081,31 @@ static int btf_dump_enum_data(struct btf_dump *d, · 1 unmodified line
2032	2081	if (err)
2033	2082	return err;
2034	2083
2035		− for (i = 0, e = btf_enum(t); i < btf_vlen(t); i++, e++) {
2036		− if (value != e->val)
2037		− continue;
2038		− btf_dump_type_values(d, "%s", btf_name_of(d, e->name_off));
2039		− return 0;
2040		− }
	2084	+ is_signed = btf_kflag(t);
	2085	+ if (btf_is_enum(t)) {
	2086	+ const struct btf_enum *e;
	2087	+
	2088	+ for (i = 0, e = btf_enum(t); i < btf_vlen(t); i++, e++) {
	2089	+ if (value != e->val)
	2090	+ continue;
	2091	+ btf_dump_type_values(d, "%s", btf_name_of(d, e->name_off));
	2092	+ return 0;
	2093	+ }
2041	2094
2042		− btf_dump_type_values(d, "%d", value);
	2095	+ btf_dump_type_values(d, is_signed ? "%d" : "%u", value);
	2096	+ } else {
	2097	+ const struct btf_enum64 *e;
	2098	+
	2099	+ for (i = 0, e = btf_enum64(t); i < btf_vlen(t); i++, e++) {
	2100	+ if (value != btf_enum64_value(e))
	2101	+ continue;
	2102	+ btf_dump_type_values(d, "%s", btf_name_of(d, e->name_off));
	2103	+ return 0;
	2104	+ }
	2105	+
	2106	+ btf_dump_type_values(d, is_signed ? "%lldLL" : "%lluULL",
	2107	+ (unsigned long long)value);
	2108	+ }
2043	2109	return 0;
2044	2110	}
2045	2111
@@ -2099,6 +2165,7 @@ static int btf_dump_type_data_check_overflow(struct btf_dump *d, · 53 unmodified lines
2099	2165	case BTF_KIND_FLOAT:
2100	2166	case BTF_KIND_PTR:
2101	2167	case BTF_KIND_ENUM:
	2168	+ case BTF_KIND_ENUM64:
2102	2169	if (data + bits_offset / 8 + size > d->typed_dump->data_end)
2103	2170	return -E2BIG;
2104	2171	break;
@@ -2203,6 +2270,7 @@ static int btf_dump_type_data_check_zero(struct btf_dump *d, · 98 unmodified lines
2203	2270	return -ENODATA;
2204	2271	}
2205	2272	case BTF_KIND_ENUM:
	2273	+ case BTF_KIND_ENUM64:
2206	2274	err = btf_dump_get_enum_value(d, t, data, id, &value);
2207	2275	if (err)
2208	2276	return err;
@@ -2275,6 +2343,7 @@ static int btf_dump_dump_type_data(struct btf_dump *d, · 66 unmodified lines
2275	2343	err = btf_dump_struct_data(d, t, id, data);
2276	2344	break;
2277	2345	case BTF_KIND_ENUM:
	2346	+ case BTF_KIND_ENUM64:
2278	2347	/* handle bitfield and int enum values */
2279	2348	if (bit_sz) {
2280	2349	__u64 print_num;

tools/lib/bpf/libbpf.c+242−54

@@ -72,6 +72,134 @@ · 71 unmodified lines
72	72	static struct bpf_map bpf_object__add_map(struct bpf_object obj);
73	73	static bool prog_is_subprog(const struct bpf_object obj, const struct bpf_program prog);
74	74
	75	+static const char * const attach_type_name[] = {
	76	+ [BPF_CGROUP_INET_INGRESS] = "cgroup_inet_ingress",
	77	+ [BPF_CGROUP_INET_EGRESS] = "cgroup_inet_egress",
	78	+ [BPF_CGROUP_INET_SOCK_CREATE] = "cgroup_inet_sock_create",
	79	+ [BPF_CGROUP_INET_SOCK_RELEASE] = "cgroup_inet_sock_release",
	80	+ [BPF_CGROUP_SOCK_OPS] = "cgroup_sock_ops",
	81	+ [BPF_CGROUP_DEVICE] = "cgroup_device",
	82	+ [BPF_CGROUP_INET4_BIND] = "cgroup_inet4_bind",
	83	+ [BPF_CGROUP_INET6_BIND] = "cgroup_inet6_bind",
	84	+ [BPF_CGROUP_INET4_CONNECT] = "cgroup_inet4_connect",
	85	+ [BPF_CGROUP_INET6_CONNECT] = "cgroup_inet6_connect",
	86	+ [BPF_CGROUP_INET4_POST_BIND] = "cgroup_inet4_post_bind",
	87	+ [BPF_CGROUP_INET6_POST_BIND] = "cgroup_inet6_post_bind",
	88	+ [BPF_CGROUP_INET4_GETPEERNAME] = "cgroup_inet4_getpeername",
	89	+ [BPF_CGROUP_INET6_GETPEERNAME] = "cgroup_inet6_getpeername",
	90	+ [BPF_CGROUP_INET4_GETSOCKNAME] = "cgroup_inet4_getsockname",
	91	+ [BPF_CGROUP_INET6_GETSOCKNAME] = "cgroup_inet6_getsockname",
	92	+ [BPF_CGROUP_UDP4_SENDMSG] = "cgroup_udp4_sendmsg",
	93	+ [BPF_CGROUP_UDP6_SENDMSG] = "cgroup_udp6_sendmsg",
	94	+ [BPF_CGROUP_SYSCTL] = "cgroup_sysctl",
	95	+ [BPF_CGROUP_UDP4_RECVMSG] = "cgroup_udp4_recvmsg",
	96	+ [BPF_CGROUP_UDP6_RECVMSG] = "cgroup_udp6_recvmsg",
	97	+ [BPF_CGROUP_GETSOCKOPT] = "cgroup_getsockopt",
	98	+ [BPF_CGROUP_SETSOCKOPT] = "cgroup_setsockopt",
	99	+ [BPF_SK_SKB_STREAM_PARSER] = "sk_skb_stream_parser",
	100	+ [BPF_SK_SKB_STREAM_VERDICT] = "sk_skb_stream_verdict",
	101	+ [BPF_SK_SKB_VERDICT] = "sk_skb_verdict",
	102	+ [BPF_SK_MSG_VERDICT] = "sk_msg_verdict",
	103	+ [BPF_LIRC_MODE2] = "lirc_mode2",
	104	+ [BPF_FLOW_DISSECTOR] = "flow_dissector",
	105	+ [BPF_TRACE_RAW_TP] = "trace_raw_tp",
	106	+ [BPF_TRACE_FENTRY] = "trace_fentry",
	107	+ [BPF_TRACE_FEXIT] = "trace_fexit",
	108	+ [BPF_MODIFY_RETURN] = "modify_return",
	109	+ [BPF_LSM_MAC] = "lsm_mac",
	110	+ [BPF_SK_LOOKUP] = "sk_lookup",
	111	+ [BPF_TRACE_ITER] = "trace_iter",
	112	+ [BPF_XDP_DEVMAP] = "xdp_devmap",
	113	+ [BPF_XDP_CPUMAP] = "xdp_cpumap",
	114	+ [BPF_XDP] = "xdp",
	115	+ [BPF_SK_REUSEPORT_SELECT] = "sk_reuseport_select",
	116	+ [BPF_SK_REUSEPORT_SELECT_OR_MIGRATE] = "sk_reuseport_select_or_migrate",
	117	+ [BPF_PERF_EVENT] = "perf_event",
	118	+ [BPF_TRACE_KPROBE_MULTI] = "trace_kprobe_multi",
	119	+};
	120	+
	121	+static const char * const link_type_name[] = {
	122	+ [BPF_LINK_TYPE_UNSPEC] = "unspec",
	123	+ [BPF_LINK_TYPE_RAW_TRACEPOINT] = "raw_tracepoint",
	124	+ [BPF_LINK_TYPE_TRACING] = "tracing",
	125	+ [BPF_LINK_TYPE_CGROUP] = "cgroup",
	126	+ [BPF_LINK_TYPE_ITER] = "iter",
	127	+ [BPF_LINK_TYPE_NETNS] = "netns",
	128	+ [BPF_LINK_TYPE_XDP] = "xdp",
	129	+ [BPF_LINK_TYPE_PERF_EVENT] = "perf_event",
	130	+ [BPF_LINK_TYPE_KPROBE_MULTI] = "kprobe_multi",
	131	+ [BPF_LINK_TYPE_STRUCT_OPS] = "struct_ops",
	132	+};
	133	+
	134	+static const char * const map_type_name[] = {
	135	+ [BPF_MAP_TYPE_UNSPEC] = "unspec",
	136	+ [BPF_MAP_TYPE_HASH] = "hash",
	137	+ [BPF_MAP_TYPE_ARRAY] = "array",
	138	+ [BPF_MAP_TYPE_PROG_ARRAY] = "prog_array",
	139	+ [BPF_MAP_TYPE_PERF_EVENT_ARRAY] = "perf_event_array",
	140	+ [BPF_MAP_TYPE_PERCPU_HASH] = "percpu_hash",
	141	+ [BPF_MAP_TYPE_PERCPU_ARRAY] = "percpu_array",
	142	+ [BPF_MAP_TYPE_STACK_TRACE] = "stack_trace",
	143	+ [BPF_MAP_TYPE_CGROUP_ARRAY] = "cgroup_array",
	144	+ [BPF_MAP_TYPE_LRU_HASH] = "lru_hash",
	145	+ [BPF_MAP_TYPE_LRU_PERCPU_HASH] = "lru_percpu_hash",
	146	+ [BPF_MAP_TYPE_LPM_TRIE] = "lpm_trie",
	147	+ [BPF_MAP_TYPE_ARRAY_OF_MAPS] = "array_of_maps",
	148	+ [BPF_MAP_TYPE_HASH_OF_MAPS] = "hash_of_maps",
	149	+ [BPF_MAP_TYPE_DEVMAP] = "devmap",
	150	+ [BPF_MAP_TYPE_DEVMAP_HASH] = "devmap_hash",
	151	+ [BPF_MAP_TYPE_SOCKMAP] = "sockmap",
	152	+ [BPF_MAP_TYPE_CPUMAP] = "cpumap",
	153	+ [BPF_MAP_TYPE_XSKMAP] = "xskmap",
	154	+ [BPF_MAP_TYPE_SOCKHASH] = "sockhash",
	155	+ [BPF_MAP_TYPE_CGROUP_STORAGE] = "cgroup_storage",
	156	+ [BPF_MAP_TYPE_REUSEPORT_SOCKARRAY] = "reuseport_sockarray",
	157	+ [BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE] = "percpu_cgroup_storage",
	158	+ [BPF_MAP_TYPE_QUEUE] = "queue",
	159	+ [BPF_MAP_TYPE_STACK] = "stack",
	160	+ [BPF_MAP_TYPE_SK_STORAGE] = "sk_storage",
	161	+ [BPF_MAP_TYPE_STRUCT_OPS] = "struct_ops",
	162	+ [BPF_MAP_TYPE_RINGBUF] = "ringbuf",
	163	+ [BPF_MAP_TYPE_INODE_STORAGE] = "inode_storage",
	164	+ [BPF_MAP_TYPE_TASK_STORAGE] = "task_storage",
	165	+ [BPF_MAP_TYPE_BLOOM_FILTER] = "bloom_filter",
	166	+};
	167	+
	168	+static const char * const prog_type_name[] = {
	169	+ [BPF_PROG_TYPE_UNSPEC] = "unspec",
	170	+ [BPF_PROG_TYPE_SOCKET_FILTER] = "socket_filter",
	171	+ [BPF_PROG_TYPE_KPROBE] = "kprobe",
	172	+ [BPF_PROG_TYPE_SCHED_CLS] = "sched_cls",
	173	+ [BPF_PROG_TYPE_SCHED_ACT] = "sched_act",
	174	+ [BPF_PROG_TYPE_TRACEPOINT] = "tracepoint",
	175	+ [BPF_PROG_TYPE_XDP] = "xdp",
	176	+ [BPF_PROG_TYPE_PERF_EVENT] = "perf_event",
	177	+ [BPF_PROG_TYPE_CGROUP_SKB] = "cgroup_skb",
	178	+ [BPF_PROG_TYPE_CGROUP_SOCK] = "cgroup_sock",
	179	+ [BPF_PROG_TYPE_LWT_IN] = "lwt_in",
	180	+ [BPF_PROG_TYPE_LWT_OUT] = "lwt_out",
	181	+ [BPF_PROG_TYPE_LWT_XMIT] = "lwt_xmit",
	182	+ [BPF_PROG_TYPE_SOCK_OPS] = "sock_ops",
	183	+ [BPF_PROG_TYPE_SK_SKB] = "sk_skb",
	184	+ [BPF_PROG_TYPE_CGROUP_DEVICE] = "cgroup_device",
	185	+ [BPF_PROG_TYPE_SK_MSG] = "sk_msg",
	186	+ [BPF_PROG_TYPE_RAW_TRACEPOINT] = "raw_tracepoint",
	187	+ [BPF_PROG_TYPE_CGROUP_SOCK_ADDR] = "cgroup_sock_addr",
	188	+ [BPF_PROG_TYPE_LWT_SEG6LOCAL] = "lwt_seg6local",
	189	+ [BPF_PROG_TYPE_LIRC_MODE2] = "lirc_mode2",
	190	+ [BPF_PROG_TYPE_SK_REUSEPORT] = "sk_reuseport",
	191	+ [BPF_PROG_TYPE_FLOW_DISSECTOR] = "flow_dissector",
	192	+ [BPF_PROG_TYPE_CGROUP_SYSCTL] = "cgroup_sysctl",
	193	+ [BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE] = "raw_tracepoint_writable",
	194	+ [BPF_PROG_TYPE_CGROUP_SOCKOPT] = "cgroup_sockopt",
	195	+ [BPF_PROG_TYPE_TRACING] = "tracing",
	196	+ [BPF_PROG_TYPE_STRUCT_OPS] = "struct_ops",
	197	+ [BPF_PROG_TYPE_EXT] = "ext",
	198	+ [BPF_PROG_TYPE_LSM] = "lsm",
	199	+ [BPF_PROG_TYPE_SK_LOOKUP] = "sk_lookup",
	200	+ [BPF_PROG_TYPE_SYSCALL] = "syscall",
	201	+};
	202	+
75	203	static int __base_pr(enum libbpf_print_level level, const char *format,
76	204	va_list args)
77	205	{
@@ -2114,6 +2242,7 @@ static const char *__btf_kind_str(__u16 kind) · 2036 unmodified lines
2114	2242	case BTF_KIND_FLOAT: return "float";
2115	2243	case BTF_KIND_DECL_TAG: return "decl_tag";
2116	2244	case BTF_KIND_TYPE_TAG: return "type_tag";
	2245	+ case BTF_KIND_ENUM64: return "enum64";
2117	2246	default: return "unknown";
2118	2247	}
2119	2248	}
@@ -2642,12 +2771,13 @@ static bool btf_needs_sanitization(struct bpf_object *obj) · 522 unmodified lines
2642	2771	bool has_func = kernel_supports(obj, FEAT_BTF_FUNC);
2643	2772	bool has_decl_tag = kernel_supports(obj, FEAT_BTF_DECL_TAG);
2644	2773	bool has_type_tag = kernel_supports(obj, FEAT_BTF_TYPE_TAG);
	2774	+ bool has_enum64 = kernel_supports(obj, FEAT_BTF_ENUM64);
2645	2775
2646	2776	return !has_func \|\| !has_datasec \|\| !has_func_global \|\| !has_float \|\|
2647		− !has_decl_tag \|\| !has_type_tag;
	2777	+ !has_decl_tag \|\| !has_type_tag \|\| !has_enum64;
2648	2778	}
2649	2779
2650		−static void bpf_object__sanitize_btf(struct bpf_object obj, struct btf btf)
	2780	+static int bpf_object__sanitize_btf(struct bpf_object obj, struct btf btf)
2651	2781	{
2652	2782	bool has_func_global = kernel_supports(obj, FEAT_BTF_GLOBAL_FUNC);
2653	2783	bool has_datasec = kernel_supports(obj, FEAT_BTF_DATASEC);
@@ -2655,6 +2785,8 @@ static void bpf_object__sanitize_btf(struct bpf_object obj, struct btf btf) · 1 unmodified line
2655	2785	bool has_func = kernel_supports(obj, FEAT_BTF_FUNC);
2656	2786	bool has_decl_tag = kernel_supports(obj, FEAT_BTF_DECL_TAG);
2657	2787	bool has_type_tag = kernel_supports(obj, FEAT_BTF_TYPE_TAG);
	2788	+ bool has_enum64 = kernel_supports(obj, FEAT_BTF_ENUM64);
	2789	+ int enum64_placeholder_id = 0;
2658	2790	struct btf_type *t;
2659	2791	int i, j, vlen;
2660	2792
@@ -2717,8 +2849,32 @@ static void bpf_object__sanitize_btf(struct bpf_object obj, struct btf btf) · 56 unmodified lines
2717	2849	/* replace TYPE_TAG with a CONST */
2718	2850	t->name_off = 0;
2719	2851	t->info = BTF_INFO_ENC(BTF_KIND_CONST, 0, 0);
2720		− }
	2852	+ } else if (!has_enum64 && btf_is_enum(t)) {
	2853	+ /* clear the kflag */
	2854	+ t->info = btf_type_info(btf_kind(t), btf_vlen(t), false);
	2855	+ } else if (!has_enum64 && btf_is_enum64(t)) {
	2856	+ /* replace ENUM64 with a union */
	2857	+ struct btf_member *m;
	2858	+
	2859	+ if (enum64_placeholder_id == 0) {
	2860	+ enum64_placeholder_id = btf__add_int(btf, "enum64_placeholder", 1, 0);
	2861	+ if (enum64_placeholder_id < 0)
	2862	+ return enum64_placeholder_id;
	2863	+
	2864	+ t = (struct btf_type *)btf__type_by_id(btf, i);
	2865	+ }
	2866	+
	2867	+ m = btf_members(t);
	2868	+ vlen = btf_vlen(t);
	2869	+ t->info = BTF_INFO_ENC(BTF_KIND_UNION, 0, vlen);
	2870	+ for (j = 0; j < vlen; j++, m++) {
	2871	+ m->type = enum64_placeholder_id;
	2872	+ m->offset = 0;
	2873	+ }
	2874	+ }
2721	2875	}
	2876	+
	2877	+ return 0;
2722	2878	}
2723	2879
2724	2880	static bool libbpf_needs_btf(const struct bpf_object *obj)
@@ -3056,7 +3212,9 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) · 331 unmodified lines
3056	3212
3057	3213	/* enforce 8-byte pointers for BPF-targeted BTFs */
3058	3214	btf__set_pointer_size(obj->btf, 8);
3059		− bpf_object__sanitize_btf(obj, kern_btf);
	3215	+ err = bpf_object__sanitize_btf(obj, kern_btf);
	3216	+ if (err)
	3217	+ return err;
3060	3218	}
3061	3219
3062	3220	if (obj->gen_loader) {
@@ -3563,6 +3721,10 @@ static enum kcfg_type find_kcfg_type(const struct btf *btf, int id, · 500 unmodified lines
3563	3721	if (strcmp(name, "libbpf_tristate"))
3564	3722	return KCFG_UNKNOWN;
3565	3723	return KCFG_TRISTATE;
	3724	+ case BTF_KIND_ENUM64:
	3725	+ if (strcmp(name, "libbpf_tristate"))
	3726	+ return KCFG_UNKNOWN;
	3727	+ return KCFG_TRISTATE;
3566	3728	case BTF_KIND_ARRAY:
3567	3729	if (btf_array(t)->nelems == 0)
3568	3730	return KCFG_UNKNOWN;
@@ -4746,6 +4908,17 @@ static int probe_kern_bpf_cookie(void) · 1177 unmodified lines
4746	4908	return probe_fd(ret);
4747	4909	}
4748	4910
	4911	+static int probe_kern_btf_enum64(void)
	4912	+{
	4913	+ static const char strs[] = "\0enum64";
	4914	+ __u32 types[] = {
	4915	+ BTF_TYPE_ENC(1, BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 0), 8),
	4916	+ };
	4917	+
	4918	+ return probe_fd(libbpf__load_raw_btf((char *)types, sizeof(types),
	4919	+ strs, sizeof(strs)));
	4920	+}
	4921	+
4749	4922	enum kern_feature_result {
4750	4923	FEAT_UNKNOWN = 0,
4751	4924	FEAT_SUPPORTED = 1,
@@ -4811,6 +4984,9 @@ static struct kern_feature_desc { · 59 unmodified lines
4811	4984	[FEAT_BPF_COOKIE] = {
4812	4985	"BPF cookie support", probe_kern_bpf_cookie,
4813	4986	},
	4987	+ [FEAT_BTF_ENUM64] = {
	4988	+ "BTF_KIND_ENUM64 support", probe_kern_btf_enum64,
	4989	+ },
4814	4990	};
4815	4991
4816	4992	bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
@@ -4943,11 +5119,6 @@ bpf_object__populate_internal_map(struct bpf_object obj, struct bpf_map map) · 126 unmodified lines
4943	5119
4944	5120	static void bpf_map__destroy(struct bpf_map *map);
4945	5121
4946		−static bool is_pow_of_2(size_t x)
4947		−{
4948		− return x && (x & (x - 1));
4949		−}
4950		−
4951	5122	static size_t adjust_ringbuf_sz(size_t sz)
4952	5123	{
4953	5124	__u32 page_sz = sysconf(_SC_PAGE_SIZE);
@@ -5353,7 +5524,7 @@ int bpf_core_add_cands(struct bpf_core_cand *local_cand, · 399 unmodified lines
5353	5524	n = btf__type_cnt(targ_btf);
5354	5525	for (i = targ_start_id; i < n; i++) {
5355	5526	t = btf__type_by_id(targ_btf, i);
5356		− if (btf_kind(t) != btf_kind(local_t))
	5527	+ if (!btf_kind_core_compat(t, local_t))
5357	5528	continue;
5358	5529
5359	5530	targ_name = btf__name_by_offset(targ_btf, t->name_off);
@@ -5567,7 +5738,7 @@ int bpf_core_types_are_compat(const struct btf *local_btf, __u32 local_id, · 207 unmodified lines
5567	5738	/* caller made sure that names match (ignoring flavor suffix) */
5568	5739	local_type = btf__type_by_id(local_btf, local_id);
5569	5740	targ_type = btf__type_by_id(targ_btf, targ_id);
5570		− if (btf_kind(local_type) != btf_kind(targ_type))
	5741	+ if (!btf_kind_core_compat(local_type, targ_type))
5571	5742	return 0;
5572	5743
5573	5744	recur:
@@ -5580,7 +5751,7 @@ recur: · 6 unmodified lines
5580	5751	if (!local_type \|\| !targ_type)
5581	5752	return -EINVAL;
5582	5753
5583		− if (btf_kind(local_type) != btf_kind(targ_type))
	5754	+ if (!btf_kind_core_compat(local_type, targ_type))
5584	5755	return 0;
5585	5756
5586	5757	switch (btf_kind(local_type)) {
@@ -5588,6 +5759,7 @@ recur: · 1 unmodified line
5588	5759	case BTF_KIND_STRUCT:
5589	5760	case BTF_KIND_UNION:
5590	5761	case BTF_KIND_ENUM:
	5762	+ case BTF_KIND_ENUM64:
5591	5763	case BTF_KIND_FWD:
5592	5764	return 1;
5593	5765	case BTF_KIND_INT:
@@ -9005,8 +9177,10 @@ static const struct bpf_sec_def section_defs[] = { · 3411 unmodified lines
9005	9177	SEC_DEF("sk_reuseport", SK_REUSEPORT, BPF_SK_REUSEPORT_SELECT, SEC_ATTACHABLE \| SEC_SLOPPY_PFX),
9006	9178	SEC_DEF("kprobe+", KPROBE, 0, SEC_NONE, attach_kprobe),
9007	9179	SEC_DEF("uprobe+", KPROBE, 0, SEC_NONE, attach_uprobe),
	9180	+ SEC_DEF("uprobe.s+", KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
9008	9181	SEC_DEF("kretprobe+", KPROBE, 0, SEC_NONE, attach_kprobe),
9009	9182	SEC_DEF("uretprobe+", KPROBE, 0, SEC_NONE, attach_uprobe),
	9183	+ SEC_DEF("uretprobe.s+", KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
9010	9184	SEC_DEF("kprobe.multi+", KPROBE, BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
9011	9185	SEC_DEF("kretprobe.multi+", KPROBE, BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
9012	9186	SEC_DEF("usdt+", KPROBE, 0, SEC_NONE, attach_usdt),
@@ -9300,6 +9474,38 @@ int libbpf_prog_type_by_name(const char name, enum bpf_prog_type prog_type, · 287 unmodified lines
9300	9474	return libbpf_err(-ESRCH);
9301	9475	}
9302	9476
	9477	+const char *libbpf_bpf_attach_type_str(enum bpf_attach_type t)
	9478	+{
	9479	+ if (t < 0 \|\| t >= ARRAY_SIZE(attach_type_name))
	9480	+ return NULL;
	9481	+
	9482	+ return attach_type_name[t];
	9483	+}
	9484	+
	9485	+const char *libbpf_bpf_link_type_str(enum bpf_link_type t)
	9486	+{
	9487	+ if (t < 0 \|\| t >= ARRAY_SIZE(link_type_name))
	9488	+ return NULL;
	9489	+
	9490	+ return link_type_name[t];
	9491	+}
	9492	+
	9493	+const char *libbpf_bpf_map_type_str(enum bpf_map_type t)
	9494	+{
	9495	+ if (t < 0 \|\| t >= ARRAY_SIZE(map_type_name))
	9496	+ return NULL;
	9497	+
	9498	+ return map_type_name[t];
	9499	+}
	9500	+
	9501	+const char *libbpf_bpf_prog_type_str(enum bpf_prog_type t)
	9502	+{
	9503	+ if (t < 0 \|\| t >= ARRAY_SIZE(prog_type_name))
	9504	+ return NULL;
	9505	+
	9506	+ return prog_type_name[t];
	9507	+}
	9508	+
9303	9509	static struct bpf_map find_struct_ops_map_by_offset(struct bpf_object obj,
9304	9510	size_t offset)
9305	9511	{
@@ -10988,43 +11194,6 @@ static int perf_event_uprobe_open_legacy(const char *probe_name, bool retprobe, · 1682 unmodified lines
10988	11194	return pfd;
10989	11195	}
10990	11196
10991		−/* uprobes deal in relative offsets; subtract the base address associated with
10992		− * the mapped binary. See Documentation/trace/uprobetracer.rst for more
10993		− * details.
10994		− */
10995		−static long elf_find_relative_offset(const char filename, Elf elf, long addr)
10996		−{
10997		− size_t n;
10998		− int i;
10999		−
11000		− if (elf_getphdrnum(elf, &n)) {
11001		− pr_warn("elf: failed to find program headers for '%s': %s\n", filename,
11002		− elf_errmsg(-1));
11003		− return -ENOENT;
11004		− }
11005		−
11006		− for (i = 0; i < n; i++) {
11007		− int seg_start, seg_end, seg_offset;
11008		− GElf_Phdr phdr;
11009		−
11010		− if (!gelf_getphdr(elf, i, &phdr)) {
11011		− pr_warn("elf: failed to get program header %d from '%s': %s\n", i, filename,
11012		− elf_errmsg(-1));
11013		− return -ENOENT;
11014		− }
11015		− if (phdr.p_type != PT_LOAD \|\| !(phdr.p_flags & PF_X))
11016		− continue;
11017		−
11018		− seg_start = phdr.p_vaddr;
11019		− seg_end = seg_start + phdr.p_memsz;
11020		− seg_offset = phdr.p_offset;
11021		− if (addr >= seg_start && addr < seg_end)
11022		− return addr - seg_start + seg_offset;
11023		− }
11024		− pr_warn("elf: failed to find prog header containing 0x%lx in '%s'\n", addr, filename);
11025		− return -ENOENT;
11026		−}
11027		−
11028	11197	/* Return next ELF section of sh_type after scn, or first of that type if scn is NULL. */
11029	11198	static Elf_Scn elf_find_next_scn_by_type(Elf elf, int sh_type, Elf_Scn *scn)
11030	11199	{
@@ -11111,6 +11280,8 @@ static long elf_find_func_offset(const char binary_path, const char name) · 80 unmodified lines
11111	11280	for (idx = 0; idx < nr_syms; idx++) {
11112	11281	int curr_bind;
11113	11282	GElf_Sym sym;
	11283	+ Elf_Scn *sym_scn;
	11284	+ GElf_Shdr sym_sh;
11114	11285
11115	11286	if (!gelf_getsym(symbols, idx, &sym))
11116	11287	continue;
@@ -11148,12 +11319,28 @@ static long elf_find_func_offset(const char binary_path, const char name) · 31 unmodified lines
11148	11319	continue;
11149	11320	}
11150	11321	}
11151		− ret = sym.st_value;
	11322	+
	11323	+ /* Transform symbol's virtual address (absolute for
	11324	+ * binaries and relative for shared libs) into file
	11325	+ * offset, which is what kernel is expecting for
	11326	+ * uprobe/uretprobe attachment.
	11327	+ * See Documentation/trace/uprobetracer.rst for more
	11328	+ * details.
	11329	+ * This is done by looking up symbol's containing
	11330	+ * section's header and using it's virtual address
	11331	+ * (sh_addr) and corresponding file offset (sh_offset)
	11332	+ * to transform sym.st_value (virtual address) into
	11333	+ * desired final file offset.
	11334	+ */
	11335	+ sym_scn = elf_getscn(elf, sym.st_shndx);
	11336	+ if (!sym_scn)
	11337	+ continue;
	11338	+ if (!gelf_getshdr(sym_scn, &sym_sh))
	11339	+ continue;
	11340	+
	11341	+ ret = sym.st_value - sym_sh.sh_addr + sym_sh.sh_offset;
11152	11342	last_bind = curr_bind;
11153	11343	}
11154		− /* For binaries that are not shared libraries, we need relative offset */
11155		− if (ret > 0 && !is_shared_lib)
11156		− ret = elf_find_relative_offset(binary_path, elf, ret);
11157	11344	if (ret > 0)
11158	11345	break;
11159	11346	}
@@ -11386,7 +11573,8 @@ static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf · 226 unmodified lines
11386	11573	break;
11387	11574	case 3:
11388	11575	case 4:
11389		− opts.retprobe = strcmp(probe_type, "uretprobe") == 0;
	11576	+ opts.retprobe = strcmp(probe_type, "uretprobe") == 0 \|\|
	11577	+ strcmp(probe_type, "uretprobe.s") == 0;
11390	11578	if (opts.retprobe && offset != 0) {
11391	11579	pr_warn("prog '%s': uretprobes do not support offset specification\n",
11392	11580	prog->name);

tools/lib/bpf/libbpf.h+37−1

@@ -51,6 +51,42 @@ enum libbpf_errno { · 50 unmodified lines
51	51
52	52	LIBBPF_API int libbpf_strerror(int err, char *buf, size_t size);
53	53
	54	+/**
	55	+ * @brief libbpf_bpf_attach_type_str() converts the provided attach type
	56	+ * value into a textual representation.
	57	+ * @param t The attach type.
	58	+ * @return Pointer to a static string identifying the attach type. NULL is
	59	+ * returned for unknown bpf_attach_type values.
	60	+ */
	61	+LIBBPF_API const char *libbpf_bpf_attach_type_str(enum bpf_attach_type t);
	62	+
	63	+/**
	64	+ * @brief libbpf_bpf_link_type_str() converts the provided link type value
	65	+ * into a textual representation.
	66	+ * @param t The link type.
	67	+ * @return Pointer to a static string identifying the link type. NULL is
	68	+ * returned for unknown bpf_link_type values.
	69	+ */
	70	+LIBBPF_API const char *libbpf_bpf_link_type_str(enum bpf_link_type t);
	71	+
	72	+/**
	73	+ * @brief libbpf_bpf_map_type_str() converts the provided map type value
	74	+ * into a textual representation.
	75	+ * @param t The map type.
	76	+ * @return Pointer to a static string identifying the map type. NULL is
	77	+ * returned for unknown bpf_map_type values.
	78	+ */
	79	+LIBBPF_API const char *libbpf_bpf_map_type_str(enum bpf_map_type t);
	80	+
	81	+/**
	82	+ * @brief libbpf_bpf_prog_type_str() converts the provided program type
	83	+ * value into a textual representation.
	84	+ * @param t The program type.
	85	+ * @return Pointer to a static string identifying the program type. NULL is
	86	+ * returned for unknown bpf_prog_type values.
	87	+ */
	88	+LIBBPF_API const char *libbpf_bpf_prog_type_str(enum bpf_prog_type t);
	89	+
54	90	enum libbpf_print_level {
55	91	LIBBPF_WARN,
56	92	LIBBPF_INFO,
@@ -71,7 +107,7 @@ struct bpf_object_open_attr { · 14 unmodified lines
71	107	};
72	108
73	109	struct bpf_object_open_opts {
74		− /* size of this struct, for forward/backward compatiblity */
	110	+ /* size of this struct, for forward/backward compatibility */
75	111	size_t sz;
76	112	/* object name override, if provided:
77	113	* - for object open from file, this will override setting object

tools/lib/bpf/libbpf.map+8−0

@@ -461,5 +461,13 @@ LIBBPF_0.8.0 { · 460 unmodified lines
461	461	} LIBBPF_0.7.0;
462	462
463	463	LIBBPF_1.0.0 {
	464	+ global:
	465	+ btf__add_enum64;
	466	+ btf__add_enum64_value;
	467	+ libbpf_bpf_attach_type_str;
	468	+ libbpf_bpf_link_type_str;
	469	+ libbpf_bpf_map_type_str;
	470	+ libbpf_bpf_prog_type_str;
	471	+
464	472	local: *;
465	473	};

tools/lib/bpf/libbpf_internal.h+7−0

@@ -351,6 +351,8 @@ enum kern_feature_id { · 350 unmodified lines
351	351	FEAT_MEMCG_ACCOUNT,
352	352	/* BPF cookie (bpf_get_attach_cookie() BPF helper) support */
353	353	FEAT_BPF_COOKIE,
	354	+ /* BTF_KIND_ENUM64 support and BTF_KIND_ENUM kflag support */
	355	+ FEAT_BTF_ENUM64,
354	356	__FEAT_CNT,
355	357	};
356	358
@@ -580,4 +582,9 @@ struct bpf_link * usdt_manager_attach_usdt(struct usdt_manager *man, · 223 unmodified lines
580	582	const char usdt_provider, const char usdt_name,
581	583	__u64 usdt_cookie);
582	584
	585	+static inline bool is_pow_of_2(size_t x)
	586	+{
	587	+ return x && (x & (x - 1)) == 0;
	588	+}
	589	+
583	590	#endif /* __LIBBPF_LIBBPF_INTERNAL_H */

tools/lib/bpf/linker.c+2−5

@@ -697,11 +697,6 @@ static int linker_load_obj_file(struct bpf_linker linker, const char filename, · 696 unmodified lines
697	697	return err;
698	698	}
699	699
700		−static bool is_pow_of_2(size_t x)
701		−{
702		− return x && (x & (x - 1)) == 0;
703		−}
704		−
705	700	static int linker_sanity_check_elf(struct src_obj *obj)
706	701	{
707	702	struct src_sec *sec;
@@ -1340,6 +1335,7 @@ recur: · 632 unmodified lines
1340	1335	case BTF_KIND_STRUCT:
1341	1336	case BTF_KIND_UNION:
1342	1337	case BTF_KIND_ENUM:
	1338	+ case BTF_KIND_ENUM64:
1343	1339	case BTF_KIND_FWD:
1344	1340	case BTF_KIND_FUNC:
1345	1341	case BTF_KIND_VAR:
@@ -1362,6 +1358,7 @@ recur: · 16 unmodified lines
1362	1358	case BTF_KIND_INT:
1363	1359	case BTF_KIND_FLOAT:
1364	1360	case BTF_KIND_ENUM:
	1361	+ case BTF_KIND_ENUM64:
1365	1362	/* ignore encoding for int and enum values for enum */
1366	1363	if (t1->size != t2->size) {
1367	1364	pr_warn("global '%s': incompatible %s '%s' size %u and %u\n",

tools/lib/bpf/relo_core.c+69−44

@@ -167,7 +167,7 @@ static bool core_relo_is_enumval_based(enum bpf_core_relo_kind kind) · 166 unmodified lines
167	167	* just a parsed access string representation): [0, 1, 2, 3].
168	168	*
169	169	* High-level spec will capture only 3 points:
170		− * - intial zero-index access by pointer (&s->... is the same as &s[0]...);
	170	+ * - initial zero-index access by pointer (&s->... is the same as &s[0]...);
171	171	* - field 'a' access (corresponds to '2' in low-level spec);
172	172	* - array element #3 access (corresponds to '3' in low-level spec).
173	173	*
@@ -186,7 +186,7 @@ int bpf_core_parse_spec(const char prog_name, const struct btf btf, · 12 unmodified lines
186	186	struct bpf_core_accessor *acc;
187	187	const struct btf_type *t;
188	188	const char name, spec_str;
189		− __u32 id;
	189	+ __u32 id, name_off;
190	190	__s64 sz;
191	191
192	192	spec_str = btf__name_by_offset(btf, relo->access_str_off);
@@ -231,11 +231,13 @@ int bpf_core_parse_spec(const char prog_name, const struct btf btf, · 38 unmodified lines
231	231	spec->len++;
232	232
233	233	if (core_relo_is_enumval_based(relo->kind)) {
234		− if (!btf_is_enum(t) \|\| spec->raw_len > 1 \|\| access_idx >= btf_vlen(t))
	234	+ if (!btf_is_any_enum(t) \|\| spec->raw_len > 1 \|\| access_idx >= btf_vlen(t))
235	235	return -EINVAL;
236	236
237	237	/* record enumerator name in a first accessor */
238		− acc->name = btf__name_by_offset(btf, btf_enum(t)[access_idx].name_off);
	238	+ name_off = btf_is_enum(t) ? btf_enum(t)[access_idx].name_off
	239	+ : btf_enum64(t)[access_idx].name_off;
	240	+ acc->name = btf__name_by_offset(btf, name_off);
239	241	return 0;
240	242	}
241	243
@@ -340,7 +342,7 @@ recur: · 98 unmodified lines
340	342
341	343	if (btf_is_composite(local_type) && btf_is_composite(targ_type))
342	344	return 1;
343		− if (btf_kind(local_type) != btf_kind(targ_type))
	345	+ if (!btf_kind_core_compat(local_type, targ_type))
344	346	return 0;
345	347
346	348	switch (btf_kind(local_type)) {
@@ -348,6 +350,7 @@ recur: · 1 unmodified line
348	350	case BTF_KIND_FLOAT:
349	351	return 1;
350	352	case BTF_KIND_FWD:
	353	+ case BTF_KIND_ENUM64:
351	354	case BTF_KIND_ENUM: {
352	355	const char local_name, targ_name;
353	356	size_t local_len, targ_len;
@@ -477,6 +480,7 @@ static int bpf_core_spec_match(struct bpf_core_spec *local_spec, · 123 unmodified lines
477	480	const struct bpf_core_accessor *local_acc;
478	481	struct bpf_core_accessor *targ_acc;
479	482	int i, sz, matched;
	483	+ __u32 name_off;
480	484
481	485	memset(targ_spec, 0, sizeof(*targ_spec));
482	486	targ_spec->btf = targ_btf;
@@ -494,18 +498,22 @@ static int bpf_core_spec_match(struct bpf_core_spec *local_spec, · 11 unmodified lines
494	498
495	499	if (core_relo_is_enumval_based(local_spec->relo_kind)) {
496	500	size_t local_essent_len, targ_essent_len;
497		− const struct btf_enum *e;
498	501	const char *targ_name;
499	502
500	503	/* has to resolve to an enum */
501	504	targ_type = skip_mods_and_typedefs(targ_spec->btf, targ_id, &targ_id);
502		− if (!btf_is_enum(targ_type))
	505	+ if (!btf_is_any_enum(targ_type))
503	506	return 0;
504	507
505	508	local_essent_len = bpf_core_essential_name_len(local_acc->name);
506	509
507		− for (i = 0, e = btf_enum(targ_type); i < btf_vlen(targ_type); i++, e++) {
508		− targ_name = btf__name_by_offset(targ_spec->btf, e->name_off);
	510	+ for (i = 0; i < btf_vlen(targ_type); i++) {
	511	+ if (btf_is_enum(targ_type))
	512	+ name_off = btf_enum(targ_type)[i].name_off;
	513	+ else
	514	+ name_off = btf_enum64(targ_type)[i].name_off;
	515	+
	516	+ targ_name = btf__name_by_offset(targ_spec->btf, name_off);
509	517	targ_essent_len = bpf_core_essential_name_len(targ_name);
510	518	if (targ_essent_len != local_essent_len)
511	519	continue;
@@ -583,7 +591,7 @@ static int bpf_core_spec_match(struct bpf_core_spec *local_spec, · 71 unmodified lines
583	591	static int bpf_core_calc_field_relo(const char *prog_name,
584	592	const struct bpf_core_relo *relo,
585	593	const struct bpf_core_spec *spec,
586		− __u32 val, __u32 field_sz, __u32 *type_id,
	594	+ __u64 val, __u32 field_sz, __u32 *type_id,
587	595	bool *validate)
588	596	{
589	597	const struct bpf_core_accessor *acc;
@@ -680,8 +688,7 @@ static int bpf_core_calc_field_relo(const char *prog_name, · 90 unmodified lines
680	688	*val = byte_sz;
681	689	break;
682	690	case BPF_CORE_FIELD_SIGNED:
683		− /* enums will be assumed unsigned */
684		− *val = btf_is_enum(mt) \|\|
	691	+ *val = (btf_is_any_enum(mt) && BTF_INFO_KFLAG(mt->info)) \|\|
685	692	(btf_int_encoding(mt) & BTF_INT_SIGNED);
686	693	if (validate)
687	694	validate = true; / signedness is never ambiguous */
@@ -708,7 +715,7 @@ static int bpf_core_calc_field_relo(const char *prog_name, · 20 unmodified lines
708	715
709	716	static int bpf_core_calc_type_relo(const struct bpf_core_relo *relo,
710	717	const struct bpf_core_spec *spec,
711		− __u32 val, bool validate)
	718	+ __u64 val, bool validate)
712	719	{
713	720	__s64 sz;
714	721
@@ -751,10 +758,9 @@ static int bpf_core_calc_type_relo(const struct bpf_core_relo *relo, · 36 unmodified lines
751	758
752	759	static int bpf_core_calc_enumval_relo(const struct bpf_core_relo *relo,
753	760	const struct bpf_core_spec *spec,
754		− __u32 *val)
	761	+ __u64 *val)
755	762	{
756	763	const struct btf_type *t;
757		− const struct btf_enum *e;
758	764
759	765	switch (relo->kind) {
760	766	case BPF_CORE_ENUMVAL_EXISTS:
@@ -764,8 +770,10 @@ static int bpf_core_calc_enumval_relo(const struct bpf_core_relo *relo, · 3 unmodified lines
764	770	if (!spec)
765	771	return -EUCLEAN; /* request instruction poisoning */
766	772	t = btf_type_by_id(spec->btf, spec->spec[0].type_id);
767		− e = btf_enum(t) + spec->spec[0].idx;
768		− *val = e->val;
	773	+ if (btf_is_enum(t))
	774	+ *val = btf_enum(t)[spec->spec[0].idx].val;
	775	+ else
	776	+ *val = btf_enum64_value(btf_enum64(t) + spec->spec[0].idx);
769	777	break;
770	778	default:
771	779	return -EOPNOTSUPP;
@@ -929,7 +937,7 @@ int bpf_core_patch_insn(const char prog_name, struct bpf_insn insn, · 157 unmodified lines
929	937	int insn_idx, const struct bpf_core_relo *relo,
930	938	int relo_idx, const struct bpf_core_relo_res *res)
931	939	{
932		− __u32 orig_val, new_val;
	940	+ __u64 orig_val, new_val;
933	941	__u8 class;
934	942
935	943	class = BPF_CLASS(insn->code);
@@ -954,28 +962,30 @@ poison: · 18 unmodified lines
954	962	if (BPF_SRC(insn->code) != BPF_K)
955	963	return -EINVAL;
956	964	if (res->validate && insn->imm != orig_val) {
957		− pr_warn("prog '%s': relo #%d: unexpected insn #%d (ALU/ALU64) value: got %u, exp %u -> %u\n",
	965	+ pr_warn("prog '%s': relo #%d: unexpected insn #%d (ALU/ALU64) value: got %u, exp %llu -> %llu\n",
958	966	prog_name, relo_idx,
959		− insn_idx, insn->imm, orig_val, new_val);
	967	+ insn_idx, insn->imm, (unsigned long long)orig_val,
	968	+ (unsigned long long)new_val);
960	969	return -EINVAL;
961	970	}
962	971	orig_val = insn->imm;
963	972	insn->imm = new_val;
964		− pr_debug("prog '%s': relo #%d: patched insn #%d (ALU/ALU64) imm %u -> %u\n",
	973	+ pr_debug("prog '%s': relo #%d: patched insn #%d (ALU/ALU64) imm %llu -> %llu\n",
965	974	prog_name, relo_idx, insn_idx,
966		− orig_val, new_val);
	975	+ (unsigned long long)orig_val, (unsigned long long)new_val);
967	976	break;
968	977	case BPF_LDX:
969	978	case BPF_ST:
970	979	case BPF_STX:
971	980	if (res->validate && insn->off != orig_val) {
972		− pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDX/ST/STX) value: got %u, exp %u -> %u\n",
973		− prog_name, relo_idx, insn_idx, insn->off, orig_val, new_val);
	981	+ pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDX/ST/STX) value: got %u, exp %llu -> %llu\n",
	982	+ prog_name, relo_idx, insn_idx, insn->off, (unsigned long long)orig_val,
	983	+ (unsigned long long)new_val);
974	984	return -EINVAL;
975	985	}
976	986	if (new_val > SHRT_MAX) {
977		− pr_warn("prog '%s': relo #%d: insn #%d (LDX/ST/STX) value too big: %u\n",
978		− prog_name, relo_idx, insn_idx, new_val);
	987	+ pr_warn("prog '%s': relo #%d: insn #%d (LDX/ST/STX) value too big: %llu\n",
	988	+ prog_name, relo_idx, insn_idx, (unsigned long long)new_val);
979	989	return -ERANGE;
980	990	}
981	991	if (res->fail_memsz_adjust) {
@@ -987,8 +997,9 @@ poison: · 5 unmodified lines
987	997
988	998	orig_val = insn->off;
989	999	insn->off = new_val;
990		− pr_debug("prog '%s': relo #%d: patched insn #%d (LDX/ST/STX) off %u -> %u\n",
991		− prog_name, relo_idx, insn_idx, orig_val, new_val);
	1000	+ pr_debug("prog '%s': relo #%d: patched insn #%d (LDX/ST/STX) off %llu -> %llu\n",
	1001	+ prog_name, relo_idx, insn_idx, (unsigned long long)orig_val,
	1002	+ (unsigned long long)new_val);
992	1003
993	1004	if (res->new_sz != res->orig_sz) {
994	1005	int insn_bytes_sz, insn_bpf_sz;
@@ -1024,20 +1035,20 @@ poison: · 29 unmodified lines
1024	1035	return -EINVAL;
1025	1036	}
1026	1037
1027		− imm = insn[0].imm + ((__u64)insn[1].imm << 32);
	1038	+ imm = (__u32)insn[0].imm \| ((__u64)insn[1].imm << 32);
1028	1039	if (res->validate && imm != orig_val) {
1029		− pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDIMM64) value: got %llu, exp %u -> %u\n",
	1040	+ pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDIMM64) value: got %llu, exp %llu -> %llu\n",
1030	1041	prog_name, relo_idx,
1031	1042	insn_idx, (unsigned long long)imm,
1032		− orig_val, new_val);
	1043	+ (unsigned long long)orig_val, (unsigned long long)new_val);
1033	1044	return -EINVAL;
1034	1045	}
1035	1046
1036	1047	insn[0].imm = new_val;
1037		− insn[1].imm = 0; /* currently only 32-bit values are supported */
1038		− pr_debug("prog '%s': relo #%d: patched insn #%d (LDIMM64) imm64 %llu -> %u\n",
	1048	+ insn[1].imm = new_val >> 32;
	1049	+ pr_debug("prog '%s': relo #%d: patched insn #%d (LDIMM64) imm64 %llu -> %llu\n",
1039	1050	prog_name, relo_idx, insn_idx,
1040		− (unsigned long long)imm, new_val);
	1051	+ (unsigned long long)imm, (unsigned long long)new_val);
1041	1052	break;
1042	1053	}
1043	1054	default:
@@ -1057,7 +1068,6 @@ poison: · 13 unmodified lines
1057	1068	int bpf_core_format_spec(char buf, size_t buf_sz, const struct bpf_core_spec spec)
1058	1069	{
1059	1070	const struct btf_type *t;
1060		− const struct btf_enum *e;
1061	1071	const char *s;
1062	1072	__u32 type_id;
1063	1073	int i, len = 0;
@@ -1086,10 +1096,23 @@ int bpf_core_format_spec(char buf, size_t buf_sz, const struct bpf_core_spec s · 22 unmodified lines
1086	1096
1087	1097	if (core_relo_is_enumval_based(spec->relo_kind)) {
1088	1098	t = skip_mods_and_typedefs(spec->btf, type_id, NULL);
1089		− e = btf_enum(t) + spec->raw_spec[0];
1090		− s = btf__name_by_offset(spec->btf, e->name_off);
	1099	+ if (btf_is_enum(t)) {
	1100	+ const struct btf_enum *e;
	1101	+ const char *fmt_str;
	1102	+
	1103	+ e = btf_enum(t) + spec->raw_spec[0];
	1104	+ s = btf__name_by_offset(spec->btf, e->name_off);
	1105	+ fmt_str = BTF_INFO_KFLAG(t->info) ? "::%s = %d" : "::%s = %u";
	1106	+ append_buf(fmt_str, s, e->val);
	1107	+ } else {
	1108	+ const struct btf_enum64 *e;
	1109	+ const char *fmt_str;
1091	1110
1092		− append_buf("::%s = %u", s, e->val);
	1111	+ e = btf_enum64(t) + spec->raw_spec[0];
	1112	+ s = btf__name_by_offset(spec->btf, e->name_off);
	1113	+ fmt_str = BTF_INFO_KFLAG(t->info) ? "::%s = %lld" : "::%s = %llu";
	1114	+ append_buf(fmt_str, s, (unsigned long long)btf_enum64_value(e));
	1115	+ }
1093	1116	return len;
1094	1117	}
1095	1118
@@ -1148,11 +1171,11 @@ int bpf_core_format_spec(char buf, size_t buf_sz, const struct bpf_core_spec s · 52 unmodified lines
1148	1171	* 3. It is supported and expected that there might be multiple flavors
1149	1172	* matching the spec. As long as all the specs resolve to the same set of
1150	1173	* offsets across all candidates, there is no error. If there is any
1151		− * ambiguity, CO-RE relocation will fail. This is necessary to accomodate
1152		− * imprefection of BTF deduplication, which can cause slight duplication of
	1174	+ * ambiguity, CO-RE relocation will fail. This is necessary to accommodate
	1175	+ * imperfection of BTF deduplication, which can cause slight duplication of
1153	1176	* the same BTF type, if some directly or indirectly referenced (by
1154	1177	* pointer) type gets resolved to different actual types in different
1155		− * object files. If such situation occurs, deduplicated BTF will end up
	1178	+ * object files. If such a situation occurs, deduplicated BTF will end up
1156	1179	* with two (or more) structurally identical types, which differ only in
1157	1180	* types they refer to through pointer. This should be OK in most cases and
1158	1181	* is not an error.
@@ -1261,10 +1284,12 @@ int bpf_core_calc_relo_insn(const char *prog_name, · 102 unmodified lines
1261	1284	* decision and value, otherwise it's dangerous to
1262	1285	* proceed due to ambiguity
1263	1286	*/
1264		− pr_warn("prog '%s': relo #%d: relocation decision ambiguity: %s %u != %s %u\n",
	1287	+ pr_warn("prog '%s': relo #%d: relocation decision ambiguity: %s %llu != %s %llu\n",
1265	1288	prog_name, relo_idx,
1266		− cand_res.poison ? "failure" : "success", cand_res.new_val,
1267		− targ_res->poison ? "failure" : "success", targ_res->new_val);
	1289	+ cand_res.poison ? "failure" : "success",
	1290	+ (unsigned long long)cand_res.new_val,
	1291	+ targ_res->poison ? "failure" : "success",
	1292	+ (unsigned long long)targ_res->new_val);
1268	1293	return -EINVAL;
1269	1294	}
1270	1295

tools/lib/bpf/relo_core.h+2−2

@@ -46,9 +46,9 @@ struct bpf_core_spec { · 45 unmodified lines
46	46
47	47	struct bpf_core_relo_res {
48	48	/* expected value in the instruction, unless validate == false */
49		− __u32 orig_val;
	49	+ __u64 orig_val;
50	50	/* new value that needs to be patched up to */
51		− __u32 new_val;
	51	+ __u64 new_val;
52	52	/* relocation unsuccessful, poison instruction, but don't fail load */
53	53	bool poison;
54	54	/* some relocations can't be validated against orig_val */

tools/lib/bpf/usdt.c+63−60

@@ -441,7 +441,7 @@ static int parse_elf_segs(Elf elf, const char path, struct elf_seg **segs, siz · 440 unmodified lines
441	441	return 0;
442	442	}
443	443
444		−static int parse_lib_segs(int pid, const char lib_path, struct elf_seg segs, size_t seg_cnt)
	444	+static int parse_vma_segs(int pid, const char lib_path, struct elf_seg segs, size_t seg_cnt)
445	445	{
446	446	char path[PATH_MAX], line[PATH_MAX], mode[16];
447	447	size_t seg_start, seg_end, seg_off;
@@ -531,35 +531,40 @@ err_out: · 83 unmodified lines
531	531	return err;
532	532	}
533	533
534		−static struct elf_seg find_elf_seg(struct elf_seg segs, size_t seg_cnt, long addr, bool relative)
	534	+static struct elf_seg find_elf_seg(struct elf_seg segs, size_t seg_cnt, long virtaddr)
535	535	{
536	536	struct elf_seg *seg;
537	537	int i;
538	538
539		− if (relative) {
540		− /* for shared libraries, address is relative offset and thus
541		− * should be fall within logical offset-based range of
542		− * [offset_start, offset_end)
543		− */
544		− for (i = 0, seg = segs; i < seg_cnt; i++, seg++) {
545		− if (seg->offset <= addr && addr < seg->offset + (seg->end - seg->start))
546		− return seg;
547		− }
548		− } else {
549		− /* for binaries, address is absolute and thus should be within
550		− * absolute address range of [seg_start, seg_end)
551		− */
552		− for (i = 0, seg = segs; i < seg_cnt; i++, seg++) {
553		− if (seg->start <= addr && addr < seg->end)
554		− return seg;
555		− }
	539	+ /* for ELF binaries (both executables and shared libraries), we are
	540	+ * given virtual address (absolute for executables, relative for
	541	+ * libraries) which should match address range of [seg_start, seg_end)
	542	+ */
	543	+ for (i = 0, seg = segs; i < seg_cnt; i++, seg++) {
	544	+ if (seg->start <= virtaddr && virtaddr < seg->end)
	545	+ return seg;
556	546	}
	547	+ return NULL;
	548	+}
557	549
	550	+static struct elf_seg find_vma_seg(struct elf_seg segs, size_t seg_cnt, long offset)
	551	+{
	552	+ struct elf_seg *seg;
	553	+ int i;
	554	+
	555	+ /* for VMA segments from /proc/<pid>/maps file, provided "address" is
	556	+ * actually a file offset, so should be fall within logical
	557	+ * offset-based range of [offset_start, offset_end)
	558	+ */
	559	+ for (i = 0, seg = segs; i < seg_cnt; i++, seg++) {
	560	+ if (seg->offset <= offset && offset < seg->offset + (seg->end - seg->start))
	561	+ return seg;
	562	+ }
558	563	return NULL;
559	564	}
560	565
561		−static int parse_usdt_note(Elf elf, const char path, long base_addr,
562		− GElf_Nhdr nhdr, const char data, size_t name_off, size_t desc_off,
	566	+static int parse_usdt_note(Elf elf, const char path, GElf_Nhdr *nhdr,
	567	+ const char *data, size_t name_off, size_t desc_off,
563	568	struct usdt_note *usdt_note);
564	569
565	570	static int parse_usdt_spec(struct usdt_spec spec, const struct usdt_note note, __u64 usdt_cookie);
@@ -568,8 +573,8 @@ static int collect_usdt_targets(struct usdt_manager man, Elf elf, const char * · 2 unmodified lines
568	573	const char usdt_provider, const char usdt_name, __u64 usdt_cookie,
569	574	struct usdt_target *out_targets, size_t out_target_cnt)
570	575	{
571		− size_t off, name_off, desc_off, seg_cnt = 0, lib_seg_cnt = 0, target_cnt = 0;
572		− struct elf_seg segs = NULL, lib_segs = NULL;
	576	+ size_t off, name_off, desc_off, seg_cnt = 0, vma_seg_cnt = 0, target_cnt = 0;
	577	+ struct elf_seg segs = NULL, vma_segs = NULL;
573	578	struct usdt_target targets = NULL, target;
574	579	long base_addr = 0;
575	580	Elf_Scn notes_scn, base_scn;
@@ -613,8 +618,7 @@ static int collect_usdt_targets(struct usdt_manager man, Elf elf, const char * · 37 unmodified lines
613	618	struct elf_seg *seg = NULL;
614	619	void *tmp;
615	620
616		− err = parse_usdt_note(elf, path, base_addr, &nhdr,
617		− data->d_buf, name_off, desc_off, &note);
	621	+ err = parse_usdt_note(elf, path, &nhdr, data->d_buf, name_off, desc_off, &note);
618	622	if (err)
619	623	goto err_out;
620	624
@@ -654,30 +658,29 @@ static int collect_usdt_targets(struct usdt_manager man, Elf elf, const char * · 33 unmodified lines
654	658	usdt_rel_ip += base_addr - note.base_addr;
655	659	}
656	660
657		− if (ehdr.e_type == ET_EXEC) {
658		− /* When attaching uprobes (which what USDTs basically
659		− * are) kernel expects a relative IP to be specified,
660		− * so if we are attaching to an executable ELF binary
661		− * (i.e., not a shared library), we need to calculate
662		− * proper relative IP based on ELF's load address
663		− */
664		− seg = find_elf_seg(segs, seg_cnt, usdt_abs_ip, false /* relative */);
665		− if (!seg) {
666		− err = -ESRCH;
667		− pr_warn("usdt: failed to find ELF program segment for '%s:%s' in '%s' at IP 0x%lx\n",
668		− usdt_provider, usdt_name, path, usdt_abs_ip);
669		− goto err_out;
670		− }
671		− if (!seg->is_exec) {
672		− err = -ESRCH;
673		− pr_warn("usdt: matched ELF binary '%s' segment [0x%lx, 0x%lx) for '%s:%s' at IP 0x%lx is not executable\n",
674		− path, seg->start, seg->end, usdt_provider, usdt_name,
675		− usdt_abs_ip);
676		− goto err_out;
677		− }
	661	+ /* When attaching uprobes (which is what USDTs basically are)
	662	+ * kernel expects file offset to be specified, not a relative
	663	+ * virtual address, so we need to translate virtual address to
	664	+ * file offset, for both ET_EXEC and ET_DYN binaries.
	665	+ */
	666	+ seg = find_elf_seg(segs, seg_cnt, usdt_abs_ip);
	667	+ if (!seg) {
	668	+ err = -ESRCH;
	669	+ pr_warn("usdt: failed to find ELF program segment for '%s:%s' in '%s' at IP 0x%lx\n",
	670	+ usdt_provider, usdt_name, path, usdt_abs_ip);
	671	+ goto err_out;
	672	+ }
	673	+ if (!seg->is_exec) {
	674	+ err = -ESRCH;
	675	+ pr_warn("usdt: matched ELF binary '%s' segment [0x%lx, 0x%lx) for '%s:%s' at IP 0x%lx is not executable\n",
	676	+ path, seg->start, seg->end, usdt_provider, usdt_name,
	677	+ usdt_abs_ip);
	678	+ goto err_out;
	679	+ }
	680	+ /* translate from virtual address to file offset */
	681	+ usdt_rel_ip = usdt_abs_ip - seg->start + seg->offset;
678	682
679		− usdt_rel_ip = usdt_abs_ip - (seg->start - seg->offset);
680		− } else if (!man->has_bpf_cookie) { /* ehdr.e_type == ET_DYN */
	683	+ if (ehdr.e_type == ET_DYN && !man->has_bpf_cookie) {
681	684	/* If we don't have BPF cookie support but need to
682	685	* attach to a shared library, we'll need to know and
683	686	* record absolute addresses of attach points due to
@@ -697,9 +700,9 @@ static int collect_usdt_targets(struct usdt_manager man, Elf elf, const char * · 13 unmodified lines
697	700	goto err_out;
698	701	}
699	702
700		− /* lib_segs are lazily initialized only if necessary */
701		− if (lib_seg_cnt == 0) {
702		− err = parse_lib_segs(pid, path, &lib_segs, &lib_seg_cnt);
	703	+ /* vma_segs are lazily initialized only if necessary */
	704	+ if (vma_seg_cnt == 0) {
	705	+ err = parse_vma_segs(pid, path, &vma_segs, &vma_seg_cnt);
703	706	if (err) {
704	707	pr_warn("usdt: failed to get memory segments in PID %d for shared library '%s': %d\n",
705	708	pid, path, err);
@@ -707,7 +710,7 @@ static int collect_usdt_targets(struct usdt_manager man, Elf elf, const char * · 1 unmodified line
707	710	}
708	711	}
709	712
710		− seg = find_elf_seg(lib_segs, lib_seg_cnt, usdt_rel_ip, true /* relative */);
	713	+ seg = find_vma_seg(vma_segs, vma_seg_cnt, usdt_rel_ip);
711	714	if (!seg) {
712	715	err = -ESRCH;
713	716	pr_warn("usdt: failed to find shared lib memory segment for '%s:%s' in '%s' at relative IP 0x%lx\n",
@@ -715,7 +718,7 @@ static int collect_usdt_targets(struct usdt_manager man, Elf elf, const char * · 1 unmodified line
715	718	goto err_out;
716	719	}
717	720
718		− usdt_abs_ip = seg->start + (usdt_rel_ip - seg->offset);
	721	+ usdt_abs_ip = seg->start - seg->offset + usdt_rel_ip;
719	722	}
720	723
721	724	pr_debug("usdt: probe for '%s:%s' in %s '%s': addr 0x%lx base 0x%lx (resolved abs_ip 0x%lx rel_ip 0x%lx) args '%s' in segment [0x%lx, 0x%lx) at offset 0x%lx\n",
@@ -723,7 +726,7 @@ static int collect_usdt_targets(struct usdt_manager man, Elf elf, const char * · 1 unmodified line
723	726	note.loc_addr, note.base_addr, usdt_abs_ip, usdt_rel_ip, note.args,
724	727	seg ? seg->start : 0, seg ? seg->end : 0, seg ? seg->offset : 0);
725	728
726		− /* Adjust semaphore address to be a relative offset */
	729	+ /* Adjust semaphore address to be a file offset */
727	730	if (note.sema_addr) {
728	731	if (!man->has_sema_refcnt) {
729	732	pr_warn("usdt: kernel doesn't support USDT semaphore refcounting for '%s:%s' in '%s'\n",
@@ -732,7 +735,7 @@ static int collect_usdt_targets(struct usdt_manager man, Elf elf, const char * · 2 unmodified lines
732	735	goto err_out;
733	736	}
734	737
735		− seg = find_elf_seg(segs, seg_cnt, note.sema_addr, false /* relative */);
	738	+ seg = find_elf_seg(segs, seg_cnt, note.sema_addr);
736	739	if (!seg) {
737	740	err = -ESRCH;
738	741	pr_warn("usdt: failed to find ELF loadable segment with semaphore of '%s:%s' in '%s' at 0x%lx\n",
@@ -747,7 +750,7 @@ static int collect_usdt_targets(struct usdt_manager man, Elf elf, const char * · 8 unmodified lines
747	750	goto err_out;
748	751	}
749	752
750		− usdt_sema_off = note.sema_addr - (seg->start - seg->offset);
	753	+ usdt_sema_off = note.sema_addr - seg->start + seg->offset;
751	754
752	755	pr_debug("usdt: sema for '%s:%s' in %s '%s': addr 0x%lx base 0x%lx (resolved 0x%lx) in segment [0x%lx, 0x%lx] at offset 0x%lx\n",
753	756	usdt_provider, usdt_name, ehdr.e_type == ET_EXEC ? "exec" : "lib ",
@@ -770,7 +773,7 @@ static int collect_usdt_targets(struct usdt_manager man, Elf elf, const char * · 16 unmodified lines
770	773	target->rel_ip = usdt_rel_ip;
771	774	target->sema_off = usdt_sema_off;
772	775
773		− /* notes->args references strings from Elf itself, so they can
	776	+ /* notes.args references strings from Elf itself, so they can
774	777	* be referenced safely until elf_end() call
775	778	*/
776	779	target->spec_str = note.args;
@@ -788,7 +791,7 @@ static int collect_usdt_targets(struct usdt_manager man, Elf elf, const char * · 11 unmodified lines
788	791
789	792	err_out:
790	793	free(segs);
791		− free(lib_segs);
	794	+ free(vma_segs);
792	795	if (err < 0)
793	796	free(targets);
794	797	return err;
@@ -1089,8 +1092,8 @@ err_out: · 294 unmodified lines
1089	1092	/* Parse out USDT ELF note from '.note.stapsdt' section.
1090	1093	* Logic inspired by perf's code.
1091	1094	*/
1092		−static int parse_usdt_note(Elf elf, const char path, long base_addr,
1093		− GElf_Nhdr nhdr, const char data, size_t name_off, size_t desc_off,
	1095	+static int parse_usdt_note(Elf elf, const char path, GElf_Nhdr *nhdr,
	1096	+ const char *data, size_t name_off, size_t desc_off,
1094	1097	struct usdt_note *note)
1095	1098	{
1096	1099	const char provider, name, *args;

tools/testing/selftests/bpf/.gitignore+1−0

@@ -43,3 +43,4 @@ test_cpp · 42 unmodified lines
43	43	*.tmp
44	44	xdpxceiver
45	45	xdp_redirect_multi
	46	+xdp_synproxy

tools/testing/selftests/bpf/Makefile+21−7

@@ -82,7 +82,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ · 81 unmodified lines
82	82	TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \
83	83	flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \
84	84	test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \
85		− xdpxceiver xdp_redirect_multi
	85	+ xdpxceiver xdp_redirect_multi xdp_synproxy
86	86
87	87	TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read
88	88
@@ -168,17 +168,26 @@ $(OUTPUT)/%:%.c · 79 unmodified lines
168	168	$(call msg,BINARY,,$@)
169	169	$(Q)$(LINK.c) $^ $(LDLIBS) -o $@
170	170
	171	+# LLVM's ld.lld doesn't support all the architectures, so use it only on x86
	172	+ifeq ($(SRCARCH),x86)
	173	+LLD := lld
	174	+else
	175	+LLD := ld
	176	+endif
	177	+
171	178	# Filter out -static for liburandom_read.so and its dependent targets so that static builds
172	179	# do not fail. Static builds leave urandom_read relying on system-wide shared libraries.
173	180	$(OUTPUT)/liburandom_read.so: urandom_read_lib1.c urandom_read_lib2.c
174	181	$(call msg,LIB,,$@)
175		− $(Q)$(CC) $(filter-out -static,$(CFLAGS) $(LDFLAGS)) $^ $(LDLIBS) -fPIC -shared -o $@
	182	+ $(Q)$(CLANG) $(filter-out -static,$(CFLAGS) $(LDFLAGS)) $^ $(LDLIBS) \
	183	+ -fuse-ld=$(LLD) -Wl,-znoseparate-code -fPIC -shared -o $@
176	184
177	185	$(OUTPUT)/urandom_read: urandom_read.c urandom_read_aux.c $(OUTPUT)/liburandom_read.so
178	186	$(call msg,BINARY,,$@)
179		− $(Q)$(CC) $(filter-out -static,$(CFLAGS) $(LDFLAGS)) $(filter %.c,$^) \
180		− liburandom_read.so $(LDLIBS) \
181		− -Wl,-rpath=. -Wl,--build-id=sha1 -o $@
	187	+ $(Q)$(CLANG) $(filter-out -static,$(CFLAGS) $(LDFLAGS)) $(filter %.c,$^) \
	188	+ liburandom_read.so $(LDLIBS) \
	189	+ -fuse-ld=$(LLD) -Wl,-znoseparate-code \
	190	+ -Wl,-rpath=. -Wl,--build-id=sha1 -o $@
182	191
183	192	$(OUTPUT)/bpf_testmod.ko: $(VMLINUX_BTF) $(wildcard bpf_testmod/Makefile bpf_testmod/*.[ch])
184	193	$(call msg,MOD,,$@)
@@ -502,6 +511,7 @@ TRUNNER_EXTRA_SOURCES := test_progs.c cgroup_helpers.c trace_helpers.c \ · 317 unmodified lines
502	511	cap_helpers.c
503	512	TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read $(OUTPUT)/bpf_testmod.ko \
504	513	$(OUTPUT)/liburandom_read.so \
	514	+ $(OUTPUT)/xdp_synproxy \
505	515	ima_setup.sh \
506	516	$(wildcard progs/btf_dump_test_case_*.c)
507	517	TRUNNER_BPF_BUILD_RULE := CLANG_BPF_BUILD_RULE
@@ -560,6 +570,7 @@ $(OUTPUT)/bench_ringbufs.o: $(OUTPUT)/ringbuf_bench.skel.h \ · 52 unmodified lines
560	570	$(OUTPUT)/bench_bloom_filter_map.o: $(OUTPUT)/bloom_filter_bench.skel.h
561	571	$(OUTPUT)/bench_bpf_loop.o: $(OUTPUT)/bpf_loop_bench.skel.h
562	572	$(OUTPUT)/bench_strncmp.o: $(OUTPUT)/strncmp_bench.skel.h
	573	+$(OUTPUT)/bench_bpf_hashmap_full_update.o: $(OUTPUT)/bpf_hashmap_full_update_bench.skel.h
563	574	$(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ)
564	575	$(OUTPUT)/bench: LDLIBS += -lm
565	576	$(OUTPUT)/bench: $(OUTPUT)/bench.o \
@@ -571,13 +582,16 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ · 5 unmodified lines
571	582	$(OUTPUT)/bench_ringbufs.o \
572	583	$(OUTPUT)/bench_bloom_filter_map.o \
573	584	$(OUTPUT)/bench_bpf_loop.o \
574		− $(OUTPUT)/bench_strncmp.o
	585	+ $(OUTPUT)/bench_strncmp.o \
	586	+ $(OUTPUT)/bench_bpf_hashmap_full_update.o
575	587	$(call msg,BINARY,,$@)
576	588	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@
577	589
578	590	EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR) \
579	591	prog_tests/tests.h map_tests/tests.h verifier/tests.h \
580	592	feature bpftool \
581		− $(addprefix $(OUTPUT)/,.o .skel.h .lskel.h .subskel.h no_alu32 bpf_gcc bpf_testmod.ko)
	593	+ $(addprefix $(OUTPUT)/,.o .skel.h .lskel.h .subskel.h \
	594	+ no_alu32 bpf_gcc bpf_testmod.ko \
	595	+ liburandom_read.so)
582	596
583	597	.PHONY: docs docs-clean

tools/testing/selftests/bpf/bench.c+2−0

@@ -396,6 +396,7 @@ extern const struct bench bench_hashmap_with_bloom; · 395 unmodified lines
396	396	extern const struct bench bench_bpf_loop;
397	397	extern const struct bench bench_strncmp_no_helper;
398	398	extern const struct bench bench_strncmp_helper;
	399	+extern const struct bench bench_bpf_hashmap_full_update;
399	400
400	401	static const struct bench *benchs[] = {
401	402	&bench_count_global,
@@ -430,6 +431,7 @@ static const struct bench *benchs[] = { · 28 unmodified lines
430	431	&bench_bpf_loop,
431	432	&bench_strncmp_no_helper,
432	433	&bench_strncmp_helper,
	434	+ &bench_bpf_hashmap_full_update,
433	435	};
434	436
435	437	static void setup_benchmark()

tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_full_update.c+96−0

@@ -0,0 +1,96 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+/* Copyright (c) 2022 Bytedance */
	3	+
	4	+#include <argp.h>
	5	+#include "bench.h"
	6	+#include "bpf_hashmap_full_update_bench.skel.h"
	7	+#include "bpf_util.h"
	8	+
	9	+/* BPF triggering benchmarks */
	10	+static struct ctx {
	11	+ struct bpf_hashmap_full_update_bench *skel;
	12	+} ctx;
	13	+
	14	+#define MAX_LOOP_NUM 10000
	15	+
	16	+static void validate(void)
	17	+{
	18	+ if (env.consumer_cnt != 1) {
	19	+ fprintf(stderr, "benchmark doesn't support multi-consumer!\n");
	20	+ exit(1);
	21	+ }
	22	+}
	23	+
	24	+static void producer(void input)
	25	+{
	26	+ while (true) {
	27	+ /* trigger the bpf program */
	28	+ syscall(__NR_getpgid);
	29	+ }
	30	+
	31	+ return NULL;
	32	+}
	33	+
	34	+static void consumer(void input)
	35	+{
	36	+ return NULL;
	37	+}
	38	+
	39	+static void measure(struct bench_res *res)
	40	+{
	41	+}
	42	+
	43	+static void setup(void)
	44	+{
	45	+ struct bpf_link *link;
	46	+ int map_fd, i, max_entries;
	47	+
	48	+ setup_libbpf();
	49	+
	50	+ ctx.skel = bpf_hashmap_full_update_bench__open_and_load();
	51	+ if (!ctx.skel) {
	52	+ fprintf(stderr, "failed to open skeleton\n");
	53	+ exit(1);
	54	+ }
	55	+
	56	+ ctx.skel->bss->nr_loops = MAX_LOOP_NUM;
	57	+
	58	+ link = bpf_program__attach(ctx.skel->progs.benchmark);
	59	+ if (!link) {
	60	+ fprintf(stderr, "failed to attach program!\n");
	61	+ exit(1);
	62	+ }
	63	+
	64	+ /* fill hash_map */
	65	+ map_fd = bpf_map__fd(ctx.skel->maps.hash_map_bench);
	66	+ max_entries = bpf_map__max_entries(ctx.skel->maps.hash_map_bench);
	67	+ for (i = 0; i < max_entries; i++)
	68	+ bpf_map_update_elem(map_fd, &i, &i, BPF_ANY);
	69	+}
	70	+
	71	+void hashmap_report_final(struct bench_res res[], int res_cnt)
	72	+{
	73	+ unsigned int nr_cpus = bpf_num_possible_cpus();
	74	+ int i;
	75	+
	76	+ for (i = 0; i < nr_cpus; i++) {
	77	+ u64 time = ctx.skel->bss->percpu_time[i];
	78	+
	79	+ if (!time)
	80	+ continue;
	81	+
	82	+ printf("%d:hash_map_full_perf %lld events per sec\n",
	83	+ i, ctx.skel->bss->nr_loops * 1000000000ll / time);
	84	+ }
	85	+}
	86	+
	87	+const struct bench bench_bpf_hashmap_full_update = {
	88	+ .name = "bpf-hashmap-ful-update",
	89	+ .validate = validate,
	90	+ .setup = setup,
	91	+ .producer_thread = producer,
	92	+ .consumer_thread = consumer,
	93	+ .measure = measure,
	94	+ .report_progress = NULL,
	95	+ .report_final = hashmap_report_final,
	96	+};

tools/testing/selftests/bpf/benchs/run_bench_bpf_hashmap_full_update.sh+11−0

@@ -0,0 +1,11 @@
	1	+#!/bin/bash
	2	+# SPDX-License-Identifier: GPL-2.0
	3	+
	4	+source ./benchs/run_common.sh
	5	+
	6	+set -eufo pipefail
	7	+
	8	+nr_threads=`expr $(cat /proc/cpuinfo \| grep "processor"\| wc -l) - 1`
	9	+summary=$($RUN_BENCH -p $nr_threads bpf-hashmap-ful-update)
	10	+printf "$summary"
	11	+printf "\n"

tools/testing/selftests/bpf/btf_helpers.c+22−3

@@ -26,11 +26,12 @@ static const char * const btf_kind_str_mapping[] = { · 25 unmodified lines
26	26	[BTF_KIND_FLOAT] = "FLOAT",
27	27	[BTF_KIND_DECL_TAG] = "DECL_TAG",
28	28	[BTF_KIND_TYPE_TAG] = "TYPE_TAG",
	29	+ [BTF_KIND_ENUM64] = "ENUM64",
29	30	};
30	31
31	32	static const char *btf_kind_str(__u16 kind)
32	33	{
33		− if (kind > BTF_KIND_TYPE_TAG)
	34	+ if (kind > BTF_KIND_ENUM64)
34	35	return "UNKNOWN";
35	36	return btf_kind_str_mapping[kind];
36	37	}
@@ -139,14 +140,32 @@ int fprintf_btf_type_raw(FILE out, const struct btf btf, __u32 id) · 102 unmodified lines
139	140	}
140	141	case BTF_KIND_ENUM: {
141	142	const struct btf_enum *v = btf_enum(t);
	143	+ const char *fmt_str;
142	144
143		− fprintf(out, " size=%u vlen=%u", t->size, vlen);
	145	+ fmt_str = btf_kflag(t) ? "\n\t'%s' val=%d" : "\n\t'%s' val=%u";
	146	+ fprintf(out, " encoding=%s size=%u vlen=%u",
	147	+ btf_kflag(t) ? "SIGNED" : "UNSIGNED", t->size, vlen);
144	148	for (i = 0; i < vlen; i++, v++) {
145		− fprintf(out, "\n\t'%s' val=%u",
	149	+ fprintf(out, fmt_str,
146	150	btf_str(btf, v->name_off), v->val);
147	151	}
148	152	break;
149	153	}
	154	+ case BTF_KIND_ENUM64: {
	155	+ const struct btf_enum64 *v = btf_enum64(t);
	156	+ const char *fmt_str;
	157	+
	158	+ fmt_str = btf_kflag(t) ? "\n\t'%s' val=%lld" : "\n\t'%s' val=%llu";
	159	+
	160	+ fprintf(out, " encoding=%s size=%u vlen=%u",
	161	+ btf_kflag(t) ? "SIGNED" : "UNSIGNED", t->size, vlen);
	162	+ for (i = 0; i < vlen; i++, v++) {
	163	+ fprintf(out, fmt_str,
	164	+ btf_str(btf, v->name_off),
	165	+ ((__u64)v->val_hi32 << 32) \| v->val_lo32);
	166	+ }
	167	+ break;
	168	+ }
150	169	case BTF_KIND_FWD:
151	170	fprintf(out, " fwd_kind=%s", btf_kflag(t) ? "union" : "struct");
152	171	break;

tools/testing/selftests/bpf/prog_tests/attach_probe.c+48−1

@@ -17,6 +17,14 @@ static void trigger_func2(void) · 16 unmodified lines
17	17	asm volatile ("");
18	18	}
19	19
	20	+/* attach point for byname sleepable uprobe */
	21	+static void trigger_func3(void)
	22	+{
	23	+ asm volatile ("");
	24	+}
	25	+
	26	+static char test_data[] = "test_data";
	27	+
20	28	void test_attach_probe(void)
21	29	{
22	30	DECLARE_LIBBPF_OPTS(bpf_uprobe_opts, uprobe_opts);
@@ -49,9 +57,17 @@ void test_attach_probe(void) · 26 unmodified lines
49	57	if (!ASSERT_GE(ref_ctr_offset, 0, "ref_ctr_offset"))
50	58	return;
51	59
52		− skel = test_attach_probe__open_and_load();
	60	+ skel = test_attach_probe__open();
53	61	if (!ASSERT_OK_PTR(skel, "skel_open"))
54	62	return;
	63	+
	64	+ /* sleepable kprobe test case needs flags set before loading */
	65	+ if (!ASSERT_OK(bpf_program__set_flags(skel->progs.handle_kprobe_sleepable,
	66	+ BPF_F_SLEEPABLE), "kprobe_sleepable_flags"))
	67	+ goto cleanup;
	68	+
	69	+ if (!ASSERT_OK(test_attach_probe__load(skel), "skel_load"))
	70	+ goto cleanup;
55	71	if (!ASSERT_OK_PTR(skel->bss, "check_bss"))
56	72	goto cleanup;
57	73
@@ -151,6 +167,30 @@ void test_attach_probe(void) · 93 unmodified lines
151	167	if (!ASSERT_OK_PTR(skel->links.handle_uretprobe_byname2, "attach_uretprobe_byname2"))
152	168	goto cleanup;
153	169
	170	+ /* sleepable kprobes should not attach successfully */
	171	+ skel->links.handle_kprobe_sleepable = bpf_program__attach(skel->progs.handle_kprobe_sleepable);
	172	+ if (!ASSERT_ERR_PTR(skel->links.handle_kprobe_sleepable, "attach_kprobe_sleepable"))
	173	+ goto cleanup;
	174	+
	175	+ /* test sleepable uprobe and uretprobe variants */
	176	+ skel->links.handle_uprobe_byname3_sleepable = bpf_program__attach(skel->progs.handle_uprobe_byname3_sleepable);
	177	+ if (!ASSERT_OK_PTR(skel->links.handle_uprobe_byname3_sleepable, "attach_uprobe_byname3_sleepable"))
	178	+ goto cleanup;
	179	+
	180	+ skel->links.handle_uprobe_byname3 = bpf_program__attach(skel->progs.handle_uprobe_byname3);
	181	+ if (!ASSERT_OK_PTR(skel->links.handle_uprobe_byname3, "attach_uprobe_byname3"))
	182	+ goto cleanup;
	183	+
	184	+ skel->links.handle_uretprobe_byname3_sleepable = bpf_program__attach(skel->progs.handle_uretprobe_byname3_sleepable);
	185	+ if (!ASSERT_OK_PTR(skel->links.handle_uretprobe_byname3_sleepable, "attach_uretprobe_byname3_sleepable"))
	186	+ goto cleanup;
	187	+
	188	+ skel->links.handle_uretprobe_byname3 = bpf_program__attach(skel->progs.handle_uretprobe_byname3);
	189	+ if (!ASSERT_OK_PTR(skel->links.handle_uretprobe_byname3, "attach_uretprobe_byname3"))
	190	+ goto cleanup;
	191	+
	192	+ skel->bss->user_ptr = test_data;
	193	+
154	194	/* trigger & validate kprobe && kretprobe */
155	195	usleep(1);
156	196
@@ -164,6 +204,9 @@ void test_attach_probe(void) · 7 unmodified lines
164	204	/* trigger & validate uprobe attached by name */
165	205	trigger_func2();
166	206
	207	+ /* trigger & validate sleepable uprobe attached by name */
	208	+ trigger_func3();
	209	+
167	210	ASSERT_EQ(skel->bss->kprobe_res, 1, "check_kprobe_res");
168	211	ASSERT_EQ(skel->bss->kprobe2_res, 11, "check_kprobe_auto_res");
169	212	ASSERT_EQ(skel->bss->kretprobe_res, 2, "check_kretprobe_res");
@@ -174,6 +217,10 @@ void test_attach_probe(void) · 4 unmodified lines
174	217	ASSERT_EQ(skel->bss->uretprobe_byname_res, 6, "check_uretprobe_byname_res");
175	218	ASSERT_EQ(skel->bss->uprobe_byname2_res, 7, "check_uprobe_byname2_res");
176	219	ASSERT_EQ(skel->bss->uretprobe_byname2_res, 8, "check_uretprobe_byname2_res");
	220	+ ASSERT_EQ(skel->bss->uprobe_byname3_sleepable_res, 9, "check_uprobe_byname3_sleepable_res");
	221	+ ASSERT_EQ(skel->bss->uprobe_byname3_res, 10, "check_uprobe_byname3_res");
	222	+ ASSERT_EQ(skel->bss->uretprobe_byname3_sleepable_res, 11, "check_uretprobe_byname3_sleepable_res");
	223	+ ASSERT_EQ(skel->bss->uretprobe_byname3_res, 12, "check_uretprobe_byname3_res");
177	224
178	225	cleanup:
179	226	test_attach_probe__destroy(skel);

tools/testing/selftests/bpf/prog_tests/btf.c+131−22

@@ -2896,26 +2896,6 @@ static struct btf_raw_test raw_tests[] = { · 2895 unmodified lines
2896	2896	.err_str = "Invalid btf_info kind_flag",
2897	2897	},
2898	2898
2899		−{
2900		− .descr = "invalid enum kind_flag",
2901		− .raw_types = {
2902		− BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
2903		− BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_ENUM, 1, 1), 4), /* [2] */
2904		− BTF_ENUM_ENC(NAME_TBD, 0),
2905		− BTF_END_RAW,
2906		− },
2907		− BTF_STR_SEC("\0A"),
2908		− .map_type = BPF_MAP_TYPE_ARRAY,
2909		− .map_name = "enum_type_check_btf",
2910		− .key_size = sizeof(int),
2911		− .value_size = sizeof(int),
2912		− .key_type_id = 1,
2913		− .value_type_id = 1,
2914		− .max_entries = 4,
2915		− .btf_load_err = true,
2916		− .err_str = "Invalid btf_info kind_flag",
2917		−},
2918		−
2919	2899	{
2920	2900	.descr = "valid fwd kind_flag",
2921	2901	.raw_types = {
@@ -4072,6 +4052,42 @@ static struct btf_raw_test raw_tests[] = { · 1150 unmodified lines
4072	4052	.btf_load_err = true,
4073	4053	.err_str = "Type tags don't precede modifiers",
4074	4054	},
	4055	+{
	4056	+ .descr = "enum64 test #1, unsigned, size 8",
	4057	+ .raw_types = {
	4058	+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
	4059	+ BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 2), 8), /* [2] */
	4060	+ BTF_ENUM64_ENC(NAME_TBD, 0, 0),
	4061	+ BTF_ENUM64_ENC(NAME_TBD, 1, 1),
	4062	+ BTF_END_RAW,
	4063	+ },
	4064	+ BTF_STR_SEC("\0a\0b\0c"),
	4065	+ .map_type = BPF_MAP_TYPE_ARRAY,
	4066	+ .map_name = "tag_type_check_btf",
	4067	+ .key_size = sizeof(int),
	4068	+ .value_size = 8,
	4069	+ .key_type_id = 1,
	4070	+ .value_type_id = 2,
	4071	+ .max_entries = 1,
	4072	+},
	4073	+{
	4074	+ .descr = "enum64 test #2, signed, size 4",
	4075	+ .raw_types = {
	4076	+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
	4077	+ BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_ENUM64, 1, 2), 4), /* [2] */
	4078	+ BTF_ENUM64_ENC(NAME_TBD, -1, 0),
	4079	+ BTF_ENUM64_ENC(NAME_TBD, 1, 0),
	4080	+ BTF_END_RAW,
	4081	+ },
	4082	+ BTF_STR_SEC("\0a\0b\0c"),
	4083	+ .map_type = BPF_MAP_TYPE_ARRAY,
	4084	+ .map_name = "tag_type_check_btf",
	4085	+ .key_size = sizeof(int),
	4086	+ .value_size = 4,
	4087	+ .key_type_id = 1,
	4088	+ .value_type_id = 2,
	4089	+ .max_entries = 1,
	4090	+},
4075	4091
4076	4092	}; /* struct btf_raw_test raw_tests[] */
4077	4093
@@ -7000,9 +7016,12 @@ static struct btf_dedup_test dedup_tests[] = { · 2922 unmodified lines
7000	7016	BTF_DECL_TAG_ENC(NAME_TBD, 13, 1), /* [16] decl_tag */
7001	7017	BTF_DECL_TAG_ENC(NAME_TBD, 7, -1), /* [17] decl_tag */
7002	7018	BTF_TYPE_TAG_ENC(NAME_TBD, 8), /* [18] type_tag */
	7019	+ BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 2), 8), /* [19] enum64 */
	7020	+ BTF_ENUM64_ENC(NAME_TBD, 0, 0),
	7021	+ BTF_ENUM64_ENC(NAME_TBD, 1, 1),
7003	7022	BTF_END_RAW,
7004	7023	},
7005		− BTF_STR_SEC("\0A\0B\0C\0D\0E\0F\0G\0H\0I\0J\0K\0L\0M\0N\0O\0P\0Q\0R"),
	7024	+ BTF_STR_SEC("\0A\0B\0C\0D\0E\0F\0G\0H\0I\0J\0K\0L\0M\0N\0O\0P\0Q\0R\0S\0T\0U"),
7006	7025	},
7007	7026	.expect = {
7008	7027	.raw_types = {
@@ -7030,9 +7049,12 @@ static struct btf_dedup_test dedup_tests[] = { · 21 unmodified lines
7030	7049	BTF_DECL_TAG_ENC(NAME_TBD, 13, 1), /* [16] decl_tag */
7031	7050	BTF_DECL_TAG_ENC(NAME_TBD, 7, -1), /* [17] decl_tag */
7032	7051	BTF_TYPE_TAG_ENC(NAME_TBD, 8), /* [18] type_tag */
	7052	+ BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 2), 8), /* [19] enum64 */
	7053	+ BTF_ENUM64_ENC(NAME_TBD, 0, 0),
	7054	+ BTF_ENUM64_ENC(NAME_TBD, 1, 1),
7033	7055	BTF_END_RAW,
7034	7056	},
7035		− BTF_STR_SEC("\0A\0B\0C\0D\0E\0F\0G\0H\0I\0J\0K\0L\0M\0N\0O\0P\0Q\0R"),
	7057	+ BTF_STR_SEC("\0A\0B\0C\0D\0E\0F\0G\0H\0I\0J\0K\0L\0M\0N\0O\0P\0Q\0R\0S\0T\0U"),
7036	7058	},
7037	7059	},
7038	7060	{
@@ -7493,6 +7515,91 @@ static struct btf_dedup_test dedup_tests[] = { · 454 unmodified lines
7493	7515	BTF_STR_SEC("\0tag1\0t\0m"),
7494	7516	},
7495	7517	},
	7518	+{
	7519	+ .descr = "dedup: enum64, standalone",
	7520	+ .input = {
	7521	+ .raw_types = {
	7522	+ BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 1), 8),
	7523	+ BTF_ENUM64_ENC(NAME_NTH(2), 1, 123),
	7524	+ BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 1), 8),
	7525	+ BTF_ENUM64_ENC(NAME_NTH(2), 1, 123),
	7526	+ BTF_END_RAW,
	7527	+ },
	7528	+ BTF_STR_SEC("\0e1\0e1_val"),
	7529	+ },
	7530	+ .expect = {
	7531	+ .raw_types = {
	7532	+ BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 1), 8),
	7533	+ BTF_ENUM64_ENC(NAME_NTH(2), 1, 123),
	7534	+ BTF_END_RAW,
	7535	+ },
	7536	+ BTF_STR_SEC("\0e1\0e1_val"),
	7537	+ },
	7538	+},
	7539	+{
	7540	+ .descr = "dedup: enum64, fwd resolution",
	7541	+ .input = {
	7542	+ .raw_types = {
	7543	+ /* [1] fwd enum64 'e1' before full enum */
	7544	+ BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 0), 8),
	7545	+ /* [2] full enum64 'e1' after fwd */
	7546	+ BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 1), 8),
	7547	+ BTF_ENUM64_ENC(NAME_NTH(2), 1, 123),
	7548	+ /* [3] full enum64 'e2' before fwd */
	7549	+ BTF_TYPE_ENC(NAME_NTH(3), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 1), 8),
	7550	+ BTF_ENUM64_ENC(NAME_NTH(4), 0, 456),
	7551	+ /* [4] fwd enum64 'e2' after full enum */
	7552	+ BTF_TYPE_ENC(NAME_NTH(3), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 0), 8),
	7553	+ /* [5] incompatible full enum64 with different value */
	7554	+ BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 1), 8),
	7555	+ BTF_ENUM64_ENC(NAME_NTH(2), 0, 321),
	7556	+ BTF_END_RAW,
	7557	+ },
	7558	+ BTF_STR_SEC("\0e1\0e1_val\0e2\0e2_val"),
	7559	+ },
	7560	+ .expect = {
	7561	+ .raw_types = {
	7562	+ /* [1] full enum64 'e1' */
	7563	+ BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 1), 8),
	7564	+ BTF_ENUM64_ENC(NAME_NTH(2), 1, 123),
	7565	+ /* [2] full enum64 'e2' */
	7566	+ BTF_TYPE_ENC(NAME_NTH(3), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 1), 8),
	7567	+ BTF_ENUM64_ENC(NAME_NTH(4), 0, 456),
	7568	+ /* [3] incompatible full enum64 with different value */
	7569	+ BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 1), 8),
	7570	+ BTF_ENUM64_ENC(NAME_NTH(2), 0, 321),
	7571	+ BTF_END_RAW,
	7572	+ },
	7573	+ BTF_STR_SEC("\0e1\0e1_val\0e2\0e2_val"),
	7574	+ },
	7575	+},
	7576	+{
	7577	+ .descr = "dedup: enum and enum64, no dedup",
	7578	+ .input = {
	7579	+ .raw_types = {
	7580	+ /* [1] enum 'e1' */
	7581	+ BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_ENUM, 0, 1), 4),
	7582	+ BTF_ENUM_ENC(NAME_NTH(2), 1),
	7583	+ /* [2] enum64 'e1' */
	7584	+ BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 1), 4),
	7585	+ BTF_ENUM64_ENC(NAME_NTH(2), 1, 0),
	7586	+ BTF_END_RAW,
	7587	+ },
	7588	+ BTF_STR_SEC("\0e1\0e1_val"),
	7589	+ },
	7590	+ .expect = {
	7591	+ .raw_types = {
	7592	+ /* [1] enum 'e1' */
	7593	+ BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_ENUM, 0, 1), 4),
	7594	+ BTF_ENUM_ENC(NAME_NTH(2), 1),
	7595	+ /* [2] enum64 'e1' */
	7596	+ BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_ENUM64, 0, 1), 4),
	7597	+ BTF_ENUM64_ENC(NAME_NTH(2), 1, 0),
	7598	+ BTF_END_RAW,
	7599	+ },
	7600	+ BTF_STR_SEC("\0e1\0e1_val"),
	7601	+ },
	7602	+},
7496	7603
7497	7604	};
7498	7605
@@ -7517,6 +7624,8 @@ static int btf_type_size(const struct btf_type *t) · 18 unmodified lines
7517	7624	return base_size + sizeof(__u32);
7518	7625	case BTF_KIND_ENUM:
7519	7626	return base_size + vlen * sizeof(struct btf_enum);
	7627	+ case BTF_KIND_ENUM64:
	7628	+ return base_size + vlen * sizeof(struct btf_enum64);
7520	7629	case BTF_KIND_ARRAY:
7521	7630	return base_size + sizeof(struct btf_array);
7522	7631	case BTF_KIND_STRUCT:

tools/testing/selftests/bpf/prog_tests/btf_write.c+92−34

@@ -9,6 +9,7 @@ static void gen_btf(struct btf *btf) · 8 unmodified lines
9	9	const struct btf_var_secinfo *vi;
10	10	const struct btf_type *t;
11	11	const struct btf_member *m;
	12	+ const struct btf_enum64 *v64;
12	13	const struct btf_enum *v;
13	14	const struct btf_param *p;
14	15	int id, err, str_off;
@@ -171,7 +172,7 @@ static void gen_btf(struct btf *btf) · 156 unmodified lines
171	172	ASSERT_STREQ(btf__str_by_offset(btf, v->name_off), "v2", "v2_name");
172	173	ASSERT_EQ(v->val, 2, "v2_val");
173	174	ASSERT_STREQ(btf_type_raw_dump(btf, 9),
174		− "[9] ENUM 'e1' size=4 vlen=2\n"
	175	+ "[9] ENUM 'e1' encoding=UNSIGNED size=4 vlen=2\n"
175	176	"\t'v1' val=1\n"
176	177	"\t'v2' val=2", "raw_dump");
177	178
@@ -202,7 +203,7 @@ static void gen_btf(struct btf *btf) · 24 unmodified lines
202	203	ASSERT_EQ(btf_vlen(t), 0, "enum_fwd_kind");
203	204	ASSERT_EQ(t->size, 4, "enum_fwd_sz");
204	205	ASSERT_STREQ(btf_type_raw_dump(btf, 12),
205		− "[12] ENUM 'enum_fwd' size=4 vlen=0", "raw_dump");
	206	+ "[12] ENUM 'enum_fwd' encoding=UNSIGNED size=4 vlen=0", "raw_dump");
206	207
207	208	/* TYPEDEF */
208	209	id = btf__add_typedef(btf, "typedef1", 1);
@@ -307,6 +308,48 @@ static void gen_btf(struct btf *btf) · 98 unmodified lines
307	308	ASSERT_EQ(t->type, 1, "tag_type");
308	309	ASSERT_STREQ(btf_type_raw_dump(btf, 20),
309	310	"[20] TYPE_TAG 'tag1' type_id=1", "raw_dump");
	311	+
	312	+ /* ENUM64 */
	313	+ id = btf__add_enum64(btf, "e1", 8, true);
	314	+ ASSERT_EQ(id, 21, "enum64_id");
	315	+ err = btf__add_enum64_value(btf, "v1", -1);
	316	+ ASSERT_OK(err, "v1_res");
	317	+ err = btf__add_enum64_value(btf, "v2", 0x123456789); /* 4886718345 */
	318	+ ASSERT_OK(err, "v2_res");
	319	+ t = btf__type_by_id(btf, 21);
	320	+ ASSERT_STREQ(btf__str_by_offset(btf, t->name_off), "e1", "enum64_name");
	321	+ ASSERT_EQ(btf_kind(t), BTF_KIND_ENUM64, "enum64_kind");
	322	+ ASSERT_EQ(btf_vlen(t), 2, "enum64_vlen");
	323	+ ASSERT_EQ(t->size, 8, "enum64_sz");
	324	+ v64 = btf_enum64(t) + 0;
	325	+ ASSERT_STREQ(btf__str_by_offset(btf, v64->name_off), "v1", "v1_name");
	326	+ ASSERT_EQ(v64->val_hi32, 0xffffffff, "v1_val");
	327	+ ASSERT_EQ(v64->val_lo32, 0xffffffff, "v1_val");
	328	+ v64 = btf_enum64(t) + 1;
	329	+ ASSERT_STREQ(btf__str_by_offset(btf, v64->name_off), "v2", "v2_name");
	330	+ ASSERT_EQ(v64->val_hi32, 0x1, "v2_val");
	331	+ ASSERT_EQ(v64->val_lo32, 0x23456789, "v2_val");
	332	+ ASSERT_STREQ(btf_type_raw_dump(btf, 21),
	333	+ "[21] ENUM64 'e1' encoding=SIGNED size=8 vlen=2\n"
	334	+ "\t'v1' val=-1\n"
	335	+ "\t'v2' val=4886718345", "raw_dump");
	336	+
	337	+ id = btf__add_enum64(btf, "e1", 8, false);
	338	+ ASSERT_EQ(id, 22, "enum64_id");
	339	+ err = btf__add_enum64_value(btf, "v1", 0xffffffffFFFFFFFF); /* 18446744073709551615 */
	340	+ ASSERT_OK(err, "v1_res");
	341	+ t = btf__type_by_id(btf, 22);
	342	+ ASSERT_STREQ(btf__str_by_offset(btf, t->name_off), "e1", "enum64_name");
	343	+ ASSERT_EQ(btf_kind(t), BTF_KIND_ENUM64, "enum64_kind");
	344	+ ASSERT_EQ(btf_vlen(t), 1, "enum64_vlen");
	345	+ ASSERT_EQ(t->size, 8, "enum64_sz");
	346	+ v64 = btf_enum64(t) + 0;
	347	+ ASSERT_STREQ(btf__str_by_offset(btf, v64->name_off), "v1", "v1_name");
	348	+ ASSERT_EQ(v64->val_hi32, 0xffffffff, "v1_val");
	349	+ ASSERT_EQ(v64->val_lo32, 0xffffffff, "v1_val");
	350	+ ASSERT_STREQ(btf_type_raw_dump(btf, 22),
	351	+ "[22] ENUM64 'e1' encoding=UNSIGNED size=8 vlen=1\n"
	352	+ "\t'v1' val=18446744073709551615", "raw_dump");
310	353	}
311	354
312	355	static void test_btf_add()
@@ -332,12 +375,12 @@ static void test_btf_add() · 19 unmodified lines
332	375	"\t'f2' type_id=1 bits_offset=32 bitfield_size=16",
333	376	"[8] UNION 'u1' size=8 vlen=1\n"
334	377	"\t'f1' type_id=1 bits_offset=0 bitfield_size=16",
335		− "[9] ENUM 'e1' size=4 vlen=2\n"
	378	+ "[9] ENUM 'e1' encoding=UNSIGNED size=4 vlen=2\n"
336	379	"\t'v1' val=1\n"
337	380	"\t'v2' val=2",
338	381	"[10] FWD 'struct_fwd' fwd_kind=struct",
339	382	"[11] FWD 'union_fwd' fwd_kind=union",
340		− "[12] ENUM 'enum_fwd' size=4 vlen=0",
	383	+ "[12] ENUM 'enum_fwd' encoding=UNSIGNED size=4 vlen=0",
341	384	"[13] TYPEDEF 'typedef1' type_id=1",
342	385	"[14] FUNC 'func1' type_id=15 linkage=global",
343	386	"[15] FUNC_PROTO '(anon)' ret_type_id=1 vlen=2\n"
@@ -348,7 +391,12 @@ static void test_btf_add() · 4 unmodified lines
348	391	"\ttype_id=1 offset=4 size=8",
349	392	"[18] DECL_TAG 'tag1' type_id=16 component_idx=-1",
350	393	"[19] DECL_TAG 'tag2' type_id=14 component_idx=1",
351		− "[20] TYPE_TAG 'tag1' type_id=1");
	394	+ "[20] TYPE_TAG 'tag1' type_id=1",
	395	+ "[21] ENUM64 'e1' encoding=SIGNED size=8 vlen=2\n"
	396	+ "\t'v1' val=-1\n"
	397	+ "\t'v2' val=4886718345",
	398	+ "[22] ENUM64 'e1' encoding=UNSIGNED size=8 vlen=1\n"
	399	+ "\t'v1' val=18446744073709551615");
352	400
353	401	btf__free(btf);
354	402	}
@@ -370,7 +418,7 @@ static void test_btf_add_btf() · 15 unmodified lines
370	418	gen_btf(btf2);
371	419
372	420	id = btf__add_btf(btf1, btf2);
373		− if (!ASSERT_EQ(id, 21, "id"))
	421	+ if (!ASSERT_EQ(id, 23, "id"))
374	422	goto cleanup;
375	423
376	424	VALIDATE_RAW_BTF(
@@ -386,12 +434,12 @@ static void test_btf_add_btf() · 9 unmodified lines
386	434	"\t'f2' type_id=1 bits_offset=32 bitfield_size=16",
387	435	"[8] UNION 'u1' size=8 vlen=1\n"
388	436	"\t'f1' type_id=1 bits_offset=0 bitfield_size=16",
389		− "[9] ENUM 'e1' size=4 vlen=2\n"
	437	+ "[9] ENUM 'e1' encoding=UNSIGNED size=4 vlen=2\n"
390	438	"\t'v1' val=1\n"
391	439	"\t'v2' val=2",
392	440	"[10] FWD 'struct_fwd' fwd_kind=struct",
393	441	"[11] FWD 'union_fwd' fwd_kind=union",
394		− "[12] ENUM 'enum_fwd' size=4 vlen=0",
	442	+ "[12] ENUM 'enum_fwd' encoding=UNSIGNED size=4 vlen=0",
395	443	"[13] TYPEDEF 'typedef1' type_id=1",
396	444	"[14] FUNC 'func1' type_id=15 linkage=global",
397	445	"[15] FUNC_PROTO '(anon)' ret_type_id=1 vlen=2\n"
@@ -403,36 +451,46 @@ static void test_btf_add_btf() · 5 unmodified lines
403	451	"[18] DECL_TAG 'tag1' type_id=16 component_idx=-1",
404	452	"[19] DECL_TAG 'tag2' type_id=14 component_idx=1",
405	453	"[20] TYPE_TAG 'tag1' type_id=1",
	454	+ "[21] ENUM64 'e1' encoding=SIGNED size=8 vlen=2\n"
	455	+ "\t'v1' val=-1\n"
	456	+ "\t'v2' val=4886718345",
	457	+ "[22] ENUM64 'e1' encoding=UNSIGNED size=8 vlen=1\n"
	458	+ "\t'v1' val=18446744073709551615",
406	459
407	460	/* types appended from the second BTF */
408		− "[21] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
409		− "[22] PTR '(anon)' type_id=21",
410		− "[23] CONST '(anon)' type_id=25",
411		− "[24] VOLATILE '(anon)' type_id=23",
412		− "[25] RESTRICT '(anon)' type_id=24",
413		− "[26] ARRAY '(anon)' type_id=22 index_type_id=21 nr_elems=10",
414		− "[27] STRUCT 's1' size=8 vlen=2\n"
415		− "\t'f1' type_id=21 bits_offset=0\n"
416		− "\t'f2' type_id=21 bits_offset=32 bitfield_size=16",
417		− "[28] UNION 'u1' size=8 vlen=1\n"
418		− "\t'f1' type_id=21 bits_offset=0 bitfield_size=16",
419		− "[29] ENUM 'e1' size=4 vlen=2\n"
	461	+ "[23] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
	462	+ "[24] PTR '(anon)' type_id=23",
	463	+ "[25] CONST '(anon)' type_id=27",
	464	+ "[26] VOLATILE '(anon)' type_id=25",
	465	+ "[27] RESTRICT '(anon)' type_id=26",
	466	+ "[28] ARRAY '(anon)' type_id=24 index_type_id=23 nr_elems=10",
	467	+ "[29] STRUCT 's1' size=8 vlen=2\n"
	468	+ "\t'f1' type_id=23 bits_offset=0\n"
	469	+ "\t'f2' type_id=23 bits_offset=32 bitfield_size=16",
	470	+ "[30] UNION 'u1' size=8 vlen=1\n"
	471	+ "\t'f1' type_id=23 bits_offset=0 bitfield_size=16",
	472	+ "[31] ENUM 'e1' encoding=UNSIGNED size=4 vlen=2\n"
420	473	"\t'v1' val=1\n"
421	474	"\t'v2' val=2",
422		− "[30] FWD 'struct_fwd' fwd_kind=struct",
423		− "[31] FWD 'union_fwd' fwd_kind=union",
424		− "[32] ENUM 'enum_fwd' size=4 vlen=0",
425		− "[33] TYPEDEF 'typedef1' type_id=21",
426		− "[34] FUNC 'func1' type_id=35 linkage=global",
427		− "[35] FUNC_PROTO '(anon)' ret_type_id=21 vlen=2\n"
428		− "\t'p1' type_id=21\n"
429		− "\t'p2' type_id=22",
430		− "[36] VAR 'var1' type_id=21, linkage=global-alloc",
431		− "[37] DATASEC 'datasec1' size=12 vlen=1\n"
432		− "\ttype_id=21 offset=4 size=8",
433		− "[38] DECL_TAG 'tag1' type_id=36 component_idx=-1",
434		− "[39] DECL_TAG 'tag2' type_id=34 component_idx=1",
435		− "[40] TYPE_TAG 'tag1' type_id=21");
	475	+ "[32] FWD 'struct_fwd' fwd_kind=struct",
	476	+ "[33] FWD 'union_fwd' fwd_kind=union",
	477	+ "[34] ENUM 'enum_fwd' encoding=UNSIGNED size=4 vlen=0",
	478	+ "[35] TYPEDEF 'typedef1' type_id=23",
	479	+ "[36] FUNC 'func1' type_id=37 linkage=global",
	480	+ "[37] FUNC_PROTO '(anon)' ret_type_id=23 vlen=2\n"
	481	+ "\t'p1' type_id=23\n"
	482	+ "\t'p2' type_id=24",
	483	+ "[38] VAR 'var1' type_id=23, linkage=global-alloc",
	484	+ "[39] DATASEC 'datasec1' size=12 vlen=1\n"
	485	+ "\ttype_id=23 offset=4 size=8",
	486	+ "[40] DECL_TAG 'tag1' type_id=38 component_idx=-1",
	487	+ "[41] DECL_TAG 'tag2' type_id=36 component_idx=1",
	488	+ "[42] TYPE_TAG 'tag1' type_id=23",
	489	+ "[43] ENUM64 'e1' encoding=SIGNED size=8 vlen=2\n"
	490	+ "\t'v1' val=-1\n"
	491	+ "\t'v2' val=4886718345",
	492	+ "[44] ENUM64 'e1' encoding=UNSIGNED size=8 vlen=1\n"
	493	+ "\t'v1' val=18446744073709551615");
436	494
437	495	cleanup:
438	496	btf__free(btf1);

tools/testing/selftests/bpf/prog_tests/core_reloc.c+64−1

@@ -84,6 +84,7 @@ static int duration = 0; · 83 unmodified lines
84	84	#define NESTING_ERR_CASE(name) { \
85	85	NESTING_CASE_COMMON(name), \
86	86	.fails = true, \
	87	+ .run_btfgen_fails = true, \
87	88	}
88	89
89	90	#define ARRAYS_DATA(struct_name) STRUCT_TO_CHAR_PTR(struct_name) { \
@@ -258,12 +259,14 @@ static int duration = 0; · 168 unmodified lines
258	259	BITFIELDS_CASE_COMMON("test_core_reloc_bitfields_probed.o", \
259	260	"probed:", name), \
260	261	.fails = true, \
	262	+ .run_btfgen_fails = true, \
261	263	.raw_tp_name = "sys_enter", \
262	264	.prog_name = "test_core_bitfields", \
263	265	}, { \
264	266	BITFIELDS_CASE_COMMON("test_core_reloc_bitfields_direct.o", \
265	267	"direct:", name), \
266	268	.fails = true, \
	269	+ .run_btfgen_fails = true, \
267	270	.prog_name = "test_core_bitfields_direct", \
268	271	}
269	272
@@ -304,6 +307,7 @@ static int duration = 0; · 34 unmodified lines
304	307	#define SIZE_ERR_CASE(name) { \
305	308	SIZE_CASE_COMMON(name), \
306	309	.fails = true, \
	310	+ .run_btfgen_fails = true, \
307	311	}
308	312
309	313	#define TYPE_BASED_CASE_COMMON(name) \
@@ -363,6 +367,25 @@ static int duration = 0; · 53 unmodified lines
363	367	.fails = true, \
364	368	}
365	369
	370	+#define ENUM64VAL_CASE_COMMON(name) \
	371	+ .case_name = #name, \
	372	+ .bpf_obj_file = "test_core_reloc_enum64val.o", \
	373	+ .btf_src_file = "btf__core_reloc_" #name ".o", \
	374	+ .raw_tp_name = "sys_enter", \
	375	+ .prog_name = "test_core_enum64val"
	376	+
	377	+#define ENUM64VAL_CASE(name, ...) { \
	378	+ ENUM64VAL_CASE_COMMON(name), \
	379	+ .output = STRUCT_TO_CHAR_PTR(core_reloc_enum64val_output) \
	380	+ __VA_ARGS__, \
	381	+ .output_len = sizeof(struct core_reloc_enum64val_output), \
	382	+}
	383	+
	384	+#define ENUM64VAL_ERR_CASE(name) { \
	385	+ ENUM64VAL_CASE_COMMON(name), \
	386	+ .fails = true, \
	387	+}
	388	+
366	389	struct core_reloc_test_case;
367	390
368	391	typedef int (setup_test_fn)(struct core_reloc_test_case test);
@@ -377,6 +400,7 @@ struct core_reloc_test_case { · 8 unmodified lines
377	400	const char *output;
378	401	int output_len;
379	402	bool fails;
	403	+ bool run_btfgen_fails;
380	404	bool needs_testmod;
381	405	bool relaxed_core_relocs;
382	406	const char *prog_name;
@@ -831,6 +855,45 @@ static const struct core_reloc_test_case test_cases[] = { · 448 unmodified lines
831	855	.anon_val2 = 0x222,
832	856	}),
833	857	ENUMVAL_ERR_CASE(enumval___err_missing),
	858	+
	859	+ /* 64bit enumerator value existence and value relocations */
	860	+ ENUM64VAL_CASE(enum64val, {
	861	+ .unsigned_val1_exists = true,
	862	+ .unsigned_val2_exists = true,
	863	+ .unsigned_val3_exists = true,
	864	+ .signed_val1_exists = true,
	865	+ .signed_val2_exists = true,
	866	+ .signed_val3_exists = true,
	867	+ .unsigned_val1 = 0x1ffffffffULL,
	868	+ .unsigned_val2 = 0x2,
	869	+ .signed_val1 = 0x1ffffffffLL,
	870	+ .signed_val2 = -2,
	871	+ }),
	872	+ ENUM64VAL_CASE(enum64val___diff, {
	873	+ .unsigned_val1_exists = true,
	874	+ .unsigned_val2_exists = true,
	875	+ .unsigned_val3_exists = true,
	876	+ .signed_val1_exists = true,
	877	+ .signed_val2_exists = true,
	878	+ .signed_val3_exists = true,
	879	+ .unsigned_val1 = 0x101ffffffffULL,
	880	+ .unsigned_val2 = 0x202ffffffffULL,
	881	+ .signed_val1 = -101,
	882	+ .signed_val2 = -202,
	883	+ }),
	884	+ ENUM64VAL_CASE(enum64val___val3_missing, {
	885	+ .unsigned_val1_exists = true,
	886	+ .unsigned_val2_exists = true,
	887	+ .unsigned_val3_exists = false,
	888	+ .signed_val1_exists = true,
	889	+ .signed_val2_exists = true,
	890	+ .signed_val3_exists = false,
	891	+ .unsigned_val1 = 0x111ffffffffULL,
	892	+ .unsigned_val2 = 0x222,
	893	+ .signed_val1 = 0x111ffffffffLL,
	894	+ .signed_val2 = -222,
	895	+ }),
	896	+ ENUM64VAL_ERR_CASE(enum64val___err_missing),
834	897	};
835	898
836	899	struct data {
@@ -894,7 +957,7 @@ static void run_core_reloc_tests(bool use_btfgen) · 57 unmodified lines
894	957	/* generate a "minimal" BTF file and use it as source */
895	958	if (use_btfgen) {
896	959
897		− if (!test_case->btf_src_file \|\| test_case->fails) {
	960	+ if (!test_case->btf_src_file \|\| test_case->run_btfgen_fails) {
898	961	test__skip();
899	962	continue;
900	963	}

tools/testing/selftests/bpf/prog_tests/fexit_stress.c+4−28

@@ -7,11 +7,9 @@ · 6 unmodified lines
7	7
8	8	void serial_test_fexit_stress(void)
9	9	{
10		− char test_skb[128] = {};
11	10	int fexit_fd[CNT] = {};
12	11	int link_fd[CNT] = {};
13		− char error[4096];
14		− int err, i, filter_fd;
	12	+ int err, i;
15	13
16	14	const struct bpf_insn trace_program[] = {
17	15	BPF_MOV64_IMM(BPF_REG_0, 0),
@@ -20,25 +18,9 @@ void serial_test_fexit_stress(void) · 2 unmodified lines
20	18
21	19	LIBBPF_OPTS(bpf_prog_load_opts, trace_opts,
22	20	.expected_attach_type = BPF_TRACE_FEXIT,
23		− .log_buf = error,
24		− .log_size = sizeof(error),
25	21	);
26	22
27		− const struct bpf_insn skb_program[] = {
28		− BPF_MOV64_IMM(BPF_REG_0, 0),
29		− BPF_EXIT_INSN(),
30		− };
31		−
32		− LIBBPF_OPTS(bpf_prog_load_opts, skb_opts,
33		− .log_buf = error,
34		− .log_size = sizeof(error),
35		− );
36		−
37		− LIBBPF_OPTS(bpf_test_run_opts, topts,
38		− .data_in = test_skb,
39		− .data_size_in = sizeof(test_skb),
40		− .repeat = 1,
41		− );
	23	+ LIBBPF_OPTS(bpf_test_run_opts, topts);
42	24
43	25	err = libbpf_find_vmlinux_btf_id("bpf_fentry_test1",
44	26	trace_opts.expected_attach_type);
@@ -58,15 +40,9 @@ void serial_test_fexit_stress(void) · 13 unmodified lines
58	40	goto out;
59	41	}
60	42
61		− filter_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL",
62		− skb_program, sizeof(skb_program) / sizeof(struct bpf_insn),
63		− &skb_opts);
64		− if (!ASSERT_GE(filter_fd, 0, "test_program_loaded"))
65		− goto out;
	43	+ err = bpf_prog_test_run_opts(fexit_fd[0], &topts);
	44	+ ASSERT_OK(err, "bpf_prog_test_run_opts");
66	45
67		− err = bpf_prog_test_run_opts(filter_fd, &topts);
68		− close(filter_fd);
69		− CHECK_FAIL(err);
70	46	out:
71	47	for (i = 0; i < CNT; i++) {
72	48	if (link_fd[i])

tools/testing/selftests/bpf/prog_tests/libbpf_str.c+207−0

@@ -0,0 +1,207 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */
	3	+
	4	+#include <ctype.h>
	5	+#include <test_progs.h>
	6	+#include <bpf/btf.h>
	7	+
	8	+/*
	9	+ * Utility function uppercasing an entire string.
	10	+ */
	11	+static void uppercase(char *s)
	12	+{
	13	+ for (; *s != '\0'; s++)
	14	+ s = toupper(s);
	15	+}
	16	+
	17	+/*
	18	+ * Test case to check that all bpf_attach_type variants are covered by
	19	+ * libbpf_bpf_attach_type_str.
	20	+ */
	21	+static void test_libbpf_bpf_attach_type_str(void)
	22	+{
	23	+ struct btf *btf;
	24	+ const struct btf_type *t;
	25	+ const struct btf_enum *e;
	26	+ int i, n, id;
	27	+
	28	+ btf = btf__parse("/sys/kernel/btf/vmlinux", NULL);
	29	+ if (!ASSERT_OK_PTR(btf, "btf_parse"))
	30	+ return;
	31	+
	32	+ /* find enum bpf_attach_type and enumerate each value */
	33	+ id = btf__find_by_name_kind(btf, "bpf_attach_type", BTF_KIND_ENUM);
	34	+ if (!ASSERT_GT(id, 0, "bpf_attach_type_id"))
	35	+ goto cleanup;
	36	+ t = btf__type_by_id(btf, id);
	37	+ e = btf_enum(t);
	38	+ n = btf_vlen(t);
	39	+ for (i = 0; i < n; e++, i++) {
	40	+ enum bpf_attach_type attach_type = (enum bpf_attach_type)e->val;
	41	+ const char *attach_type_name;
	42	+ const char *attach_type_str;
	43	+ char buf[256];
	44	+
	45	+ if (attach_type == __MAX_BPF_ATTACH_TYPE)
	46	+ continue;
	47	+
	48	+ attach_type_name = btf__str_by_offset(btf, e->name_off);
	49	+ attach_type_str = libbpf_bpf_attach_type_str(attach_type);
	50	+ ASSERT_OK_PTR(attach_type_str, attach_type_name);
	51	+
	52	+ snprintf(buf, sizeof(buf), "BPF_%s", attach_type_str);
	53	+ uppercase(buf);
	54	+
	55	+ ASSERT_STREQ(buf, attach_type_name, "exp_str_value");
	56	+ }
	57	+
	58	+cleanup:
	59	+ btf__free(btf);
	60	+}
	61	+
	62	+/*
	63	+ * Test case to check that all bpf_link_type variants are covered by
	64	+ * libbpf_bpf_link_type_str.
	65	+ */
	66	+static void test_libbpf_bpf_link_type_str(void)
	67	+{
	68	+ struct btf *btf;
	69	+ const struct btf_type *t;
	70	+ const struct btf_enum *e;
	71	+ int i, n, id;
	72	+
	73	+ btf = btf__parse("/sys/kernel/btf/vmlinux", NULL);
	74	+ if (!ASSERT_OK_PTR(btf, "btf_parse"))
	75	+ return;
	76	+
	77	+ /* find enum bpf_link_type and enumerate each value */
	78	+ id = btf__find_by_name_kind(btf, "bpf_link_type", BTF_KIND_ENUM);
	79	+ if (!ASSERT_GT(id, 0, "bpf_link_type_id"))
	80	+ goto cleanup;
	81	+ t = btf__type_by_id(btf, id);
	82	+ e = btf_enum(t);
	83	+ n = btf_vlen(t);
	84	+ for (i = 0; i < n; e++, i++) {
	85	+ enum bpf_link_type link_type = (enum bpf_link_type)e->val;
	86	+ const char *link_type_name;
	87	+ const char *link_type_str;
	88	+ char buf[256];
	89	+
	90	+ if (link_type == MAX_BPF_LINK_TYPE)
	91	+ continue;
	92	+
	93	+ link_type_name = btf__str_by_offset(btf, e->name_off);
	94	+ link_type_str = libbpf_bpf_link_type_str(link_type);
	95	+ ASSERT_OK_PTR(link_type_str, link_type_name);
	96	+
	97	+ snprintf(buf, sizeof(buf), "BPF_LINK_TYPE_%s", link_type_str);
	98	+ uppercase(buf);
	99	+
	100	+ ASSERT_STREQ(buf, link_type_name, "exp_str_value");
	101	+ }
	102	+
	103	+cleanup:
	104	+ btf__free(btf);
	105	+}
	106	+
	107	+/*
	108	+ * Test case to check that all bpf_map_type variants are covered by
	109	+ * libbpf_bpf_map_type_str.
	110	+ */
	111	+static void test_libbpf_bpf_map_type_str(void)
	112	+{
	113	+ struct btf *btf;
	114	+ const struct btf_type *t;
	115	+ const struct btf_enum *e;
	116	+ int i, n, id;
	117	+
	118	+ btf = btf__parse("/sys/kernel/btf/vmlinux", NULL);
	119	+ if (!ASSERT_OK_PTR(btf, "btf_parse"))
	120	+ return;
	121	+
	122	+ /* find enum bpf_map_type and enumerate each value */
	123	+ id = btf__find_by_name_kind(btf, "bpf_map_type", BTF_KIND_ENUM);
	124	+ if (!ASSERT_GT(id, 0, "bpf_map_type_id"))
	125	+ goto cleanup;
	126	+ t = btf__type_by_id(btf, id);
	127	+ e = btf_enum(t);
	128	+ n = btf_vlen(t);
	129	+ for (i = 0; i < n; e++, i++) {
	130	+ enum bpf_map_type map_type = (enum bpf_map_type)e->val;
	131	+ const char *map_type_name;
	132	+ const char *map_type_str;
	133	+ char buf[256];
	134	+
	135	+ map_type_name = btf__str_by_offset(btf, e->name_off);
	136	+ map_type_str = libbpf_bpf_map_type_str(map_type);
	137	+ ASSERT_OK_PTR(map_type_str, map_type_name);
	138	+
	139	+ snprintf(buf, sizeof(buf), "BPF_MAP_TYPE_%s", map_type_str);
	140	+ uppercase(buf);
	141	+
	142	+ ASSERT_STREQ(buf, map_type_name, "exp_str_value");
	143	+ }
	144	+
	145	+cleanup:
	146	+ btf__free(btf);
	147	+}
	148	+
	149	+/*
	150	+ * Test case to check that all bpf_prog_type variants are covered by
	151	+ * libbpf_bpf_prog_type_str.
	152	+ */
	153	+static void test_libbpf_bpf_prog_type_str(void)
	154	+{
	155	+ struct btf *btf;
	156	+ const struct btf_type *t;
	157	+ const struct btf_enum *e;
	158	+ int i, n, id;
	159	+
	160	+ btf = btf__parse("/sys/kernel/btf/vmlinux", NULL);
	161	+ if (!ASSERT_OK_PTR(btf, "btf_parse"))
	162	+ return;
	163	+
	164	+ /* find enum bpf_prog_type and enumerate each value */
	165	+ id = btf__find_by_name_kind(btf, "bpf_prog_type", BTF_KIND_ENUM);
	166	+ if (!ASSERT_GT(id, 0, "bpf_prog_type_id"))
	167	+ goto cleanup;
	168	+ t = btf__type_by_id(btf, id);
	169	+ e = btf_enum(t);
	170	+ n = btf_vlen(t);
	171	+ for (i = 0; i < n; e++, i++) {
	172	+ enum bpf_prog_type prog_type = (enum bpf_prog_type)e->val;
	173	+ const char *prog_type_name;
	174	+ const char *prog_type_str;
	175	+ char buf[256];
	176	+
	177	+ prog_type_name = btf__str_by_offset(btf, e->name_off);
	178	+ prog_type_str = libbpf_bpf_prog_type_str(prog_type);
	179	+ ASSERT_OK_PTR(prog_type_str, prog_type_name);
	180	+
	181	+ snprintf(buf, sizeof(buf), "BPF_PROG_TYPE_%s", prog_type_str);
	182	+ uppercase(buf);
	183	+
	184	+ ASSERT_STREQ(buf, prog_type_name, "exp_str_value");
	185	+ }
	186	+
	187	+cleanup:
	188	+ btf__free(btf);
	189	+}
	190	+
	191	+/*
	192	+ * Run all libbpf str conversion tests.
	193	+ */
	194	+void test_libbpf_str(void)
	195	+{
	196	+ if (test__start_subtest("bpf_attach_type_str"))
	197	+ test_libbpf_bpf_attach_type_str();
	198	+
	199	+ if (test__start_subtest("bpf_link_type_str"))
	200	+ test_libbpf_bpf_link_type_str();
	201	+
	202	+ if (test__start_subtest("bpf_map_type_str"))
	203	+ test_libbpf_bpf_map_type_str();
	204	+
	205	+ if (test__start_subtest("bpf_prog_type_str"))
	206	+ test_libbpf_bpf_prog_type_str();
	207	+}

tools/testing/selftests/bpf/prog_tests/tc_redirect.c+4−4

@@ -646,7 +646,7 @@ static void test_tcp_clear_dtime(struct test_tc_dtime *skel) · 645 unmodified lines
646	646	__u32 *errs = skel->bss->errs[t];
647	647
648	648	skel->bss->test = t;
649		− test_inet_dtime(AF_INET6, SOCK_STREAM, IP6_DST, 0);
	649	+ test_inet_dtime(AF_INET6, SOCK_STREAM, IP6_DST, 50000 + t);
650	650
651	651	ASSERT_EQ(dtimes[INGRESS_FWDNS_P100], 0,
652	652	dtime_cnt_str(t, INGRESS_FWDNS_P100));
@@ -683,7 +683,7 @@ static void test_tcp_dtime(struct test_tc_dtime *skel, int family, bool bpf_fwd) · 30 unmodified lines
683	683	errs = skel->bss->errs[t];
684	684
685	685	skel->bss->test = t;
686		− test_inet_dtime(family, SOCK_STREAM, addr, 0);
	686	+ test_inet_dtime(family, SOCK_STREAM, addr, 50000 + t);
687	687
688	688	/* fwdns_prio100 prog does not read delivery_time_type, so
689	689	* kernel puts the (rcv) timetamp in __sk_buff->tstamp
@@ -715,13 +715,13 @@ static void test_udp_dtime(struct test_tc_dtime *skel, int family, bool bpf_fwd) · 25 unmodified lines
715	715	errs = skel->bss->errs[t];
716	716
717	717	skel->bss->test = t;
718		− test_inet_dtime(family, SOCK_DGRAM, addr, 0);
	718	+ test_inet_dtime(family, SOCK_DGRAM, addr, 50000 + t);
719	719
720	720	ASSERT_EQ(dtimes[INGRESS_FWDNS_P100], 0,
721	721	dtime_cnt_str(t, INGRESS_FWDNS_P100));
722	722	/* non mono delivery time is not forwarded */
723	723	ASSERT_EQ(dtimes[INGRESS_FWDNS_P101], 0,
724		− dtime_cnt_str(t, INGRESS_FWDNS_P100));
	724	+ dtime_cnt_str(t, INGRESS_FWDNS_P101));
725	725	for (i = EGRESS_FWDNS_P100; i < SET_DTIME; i++)
726	726	ASSERT_GT(dtimes[i], 0, dtime_cnt_str(t, i));
727	727

tools/testing/selftests/bpf/prog_tests/xdp_synproxy.c+183−0

@@ -0,0 +1,183 @@
	1	+// SPDX-License-Identifier: LGPL-2.1 OR BSD-2-Clause
	2	+/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
	3	+
	4	+#define _GNU_SOURCE
	5	+#include <test_progs.h>
	6	+#include <network_helpers.h>
	7	+#include <ctype.h>
	8	+
	9	+#define CMD_OUT_BUF_SIZE 1023
	10	+
	11	+#define SYS(cmd) ({ \
	12	+ if (!ASSERT_OK(system(cmd), (cmd))) \
	13	+ goto out; \
	14	+})
	15	+
	16	+#define SYS_OUT(cmd, ...) ({ \
	17	+ char buf[1024]; \
	18	+ snprintf(buf, sizeof(buf), (cmd), ##__VA_ARGS__); \
	19	+ FILE *f = popen(buf, "r"); \
	20	+ if (!ASSERT_OK_PTR(f, buf)) \
	21	+ goto out; \
	22	+ f; \
	23	+})
	24	+
	25	+/* out must be at least `size * 4 + 1` bytes long */
	26	+static void escape_str(char out, const char in, size_t size)
	27	+{
	28	+ static const char *hex = "0123456789ABCDEF";
	29	+ size_t i;
	30	+
	31	+ for (i = 0; i < size; i++) {
	32	+ if (isprint(in[i]) && in[i] != '\\' && in[i] != '\'') {
	33	+ *out++ = in[i];
	34	+ } else {
	35	+ *out++ = '\\';
	36	+ *out++ = 'x';
	37	+ *out++ = hex[(in[i] >> 4) & 0xf];
	38	+ *out++ = hex[in[i] & 0xf];
	39	+ }
	40	+ }
	41	+ *out++ = '\0';
	42	+}
	43	+
	44	+static bool expect_str(char buf, size_t size, const char str, const char *name)
	45	+{
	46	+ static char escbuf_expected[CMD_OUT_BUF_SIZE * 4];
	47	+ static char escbuf_actual[CMD_OUT_BUF_SIZE * 4];
	48	+ static int duration = 0;
	49	+ bool ok;
	50	+
	51	+ ok = size == strlen(str) && !memcmp(buf, str, size);
	52	+
	53	+ if (!ok) {
	54	+ escape_str(escbuf_expected, str, strlen(str));
	55	+ escape_str(escbuf_actual, buf, size);
	56	+ }
	57	+ CHECK(!ok, name, "unexpected %s: actual '%s' != expected '%s'\n",
	58	+ name, escbuf_actual, escbuf_expected);
	59	+
	60	+ return ok;
	61	+}
	62	+
	63	+static void test_synproxy(bool xdp)
	64	+{
	65	+ int server_fd = -1, client_fd = -1, accept_fd = -1;
	66	+ char prog_id, prog_id_end;
	67	+ struct nstoken *ns = NULL;
	68	+ FILE *ctrl_file = NULL;
	69	+ char buf[CMD_OUT_BUF_SIZE];
	70	+ size_t size;
	71	+
	72	+ SYS("ip netns add synproxy");
	73	+
	74	+ SYS("ip link add tmp0 type veth peer name tmp1");
	75	+ SYS("ip link set tmp1 netns synproxy");
	76	+ SYS("ip link set tmp0 up");
	77	+ SYS("ip addr replace 198.18.0.1/24 dev tmp0");
	78	+
	79	+ /* When checksum offload is enabled, the XDP program sees wrong
	80	+ * checksums and drops packets.
	81	+ */
	82	+ SYS("ethtool -K tmp0 tx off");
	83	+ if (xdp)
	84	+ /* Workaround required for veth. */
	85	+ SYS("ip link set tmp0 xdp object xdp_dummy.o section xdp 2> /dev/null");
	86	+
	87	+ ns = open_netns("synproxy");
	88	+ if (!ASSERT_OK_PTR(ns, "setns"))
	89	+ goto out;
	90	+
	91	+ SYS("ip link set lo up");
	92	+ SYS("ip link set tmp1 up");
	93	+ SYS("ip addr replace 198.18.0.2/24 dev tmp1");
	94	+ SYS("sysctl -w net.ipv4.tcp_syncookies=2");
	95	+ SYS("sysctl -w net.ipv4.tcp_timestamps=1");
	96	+ SYS("sysctl -w net.netfilter.nf_conntrack_tcp_loose=0");
	97	+ SYS("iptables -t raw -I PREROUTING \
	98	+ -i tmp1 -p tcp -m tcp --syn --dport 8080 -j CT --notrack");
	99	+ SYS("iptables -t filter -A INPUT \
	100	+ -i tmp1 -p tcp -m tcp --dport 8080 -m state --state INVALID,UNTRACKED \
	101	+ -j SYNPROXY --sack-perm --timestamp --wscale 7 --mss 1460");
	102	+ SYS("iptables -t filter -A INPUT \
	103	+ -i tmp1 -m state --state INVALID -j DROP");
	104	+
	105	+ ctrl_file = SYS_OUT("./xdp_synproxy --iface tmp1 --ports 8080 \
	106	+ --single --mss4 1460 --mss6 1440 \
	107	+ --wscale 7 --ttl 64%s", xdp ? "" : " --tc");
	108	+ size = fread(buf, 1, sizeof(buf), ctrl_file);
	109	+ pclose(ctrl_file);
	110	+ if (!expect_str(buf, size, "Total SYNACKs generated: 0\n",
	111	+ "initial SYNACKs"))
	112	+ goto out;
	113	+
	114	+ if (!xdp) {
	115	+ ctrl_file = SYS_OUT("tc filter show dev tmp1 ingress");
	116	+ size = fread(buf, 1, sizeof(buf), ctrl_file);
	117	+ pclose(ctrl_file);
	118	+ prog_id = memmem(buf, size, " id ", 4);
	119	+ if (!ASSERT_OK_PTR(prog_id, "find prog id"))
	120	+ goto out;
	121	+ prog_id += 4;
	122	+ if (!ASSERT_LT(prog_id, buf + size, "find prog id begin"))
	123	+ goto out;
	124	+ prog_id_end = prog_id;
	125	+ while (prog_id_end < buf + size && *prog_id_end >= '0' &&
	126	+ *prog_id_end <= '9')
	127	+ prog_id_end++;
	128	+ if (!ASSERT_LT(prog_id_end, buf + size, "find prog id end"))
	129	+ goto out;
	130	+ *prog_id_end = '\0';
	131	+ }
	132	+
	133	+ server_fd = start_server(AF_INET, SOCK_STREAM, "198.18.0.2", 8080, 0);
	134	+ if (!ASSERT_GE(server_fd, 0, "start_server"))
	135	+ goto out;
	136	+
	137	+ close_netns(ns);
	138	+ ns = NULL;
	139	+
	140	+ client_fd = connect_to_fd(server_fd, 10000);
	141	+ if (!ASSERT_GE(client_fd, 0, "connect_to_fd"))
	142	+ goto out;
	143	+
	144	+ accept_fd = accept(server_fd, NULL, NULL);
	145	+ if (!ASSERT_GE(accept_fd, 0, "accept"))
	146	+ goto out;
	147	+
	148	+ ns = open_netns("synproxy");
	149	+ if (!ASSERT_OK_PTR(ns, "setns"))
	150	+ goto out;
	151	+
	152	+ if (xdp)
	153	+ ctrl_file = SYS_OUT("./xdp_synproxy --iface tmp1 --single");
	154	+ else
	155	+ ctrl_file = SYS_OUT("./xdp_synproxy --prog %s --single",
	156	+ prog_id);
	157	+ size = fread(buf, 1, sizeof(buf), ctrl_file);
	158	+ pclose(ctrl_file);
	159	+ if (!expect_str(buf, size, "Total SYNACKs generated: 1\n",
	160	+ "SYNACKs after connection"))
	161	+ goto out;
	162	+
	163	+out:
	164	+ if (accept_fd >= 0)
	165	+ close(accept_fd);
	166	+ if (client_fd >= 0)
	167	+ close(client_fd);
	168	+ if (server_fd >= 0)
	169	+ close(server_fd);
	170	+ if (ns)
	171	+ close_netns(ns);
	172	+
	173	+ system("ip link del tmp0");
	174	+ system("ip netns del synproxy");
	175	+}
	176	+
	177	+void test_xdp_synproxy(void)
	178	+{
	179	+ if (test__start_subtest("xdp"))
	180	+ test_synproxy(true);
	181	+ if (test__start_subtest("tc"))
	182	+ test_synproxy(false);
	183	+}

tools/testing/selftests/bpf/progs/bpf_hashmap_full_update_bench.c+40−0

@@ -0,0 +1,40 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+/* Copyright (c) 2022 Bytedance */
	3	+
	4	+#include "vmlinux.h"
	5	+#include <bpf/bpf_helpers.h>
	6	+#include "bpf_misc.h"
	7	+
	8	+char _license[] SEC("license") = "GPL";
	9	+
	10	+#define MAX_ENTRIES 1000
	11	+
	12	+struct {
	13	+ __uint(type, BPF_MAP_TYPE_HASH);
	14	+ __type(key, u32);
	15	+ __type(value, u64);
	16	+ __uint(max_entries, MAX_ENTRIES);
	17	+} hash_map_bench SEC(".maps");
	18	+
	19	+u64 __attribute__((__aligned__(256))) percpu_time[256];
	20	+u64 nr_loops;
	21	+
	22	+static int loop_update_callback(__u32 index, u32 *key)
	23	+{
	24	+ u64 init_val = 1;
	25	+
	26	+ bpf_map_update_elem(&hash_map_bench, key, &init_val, BPF_ANY);
	27	+ return 0;
	28	+}
	29	+
	30	+SEC("fentry/" SYS_PREFIX "sys_getpgid")
	31	+int benchmark(void *ctx)
	32	+{
	33	+ u32 cpu = bpf_get_smp_processor_id();
	34	+ u32 key = cpu + MAX_ENTRIES;
	35	+ u64 start_time = bpf_ktime_get_ns();
	36	+
	37	+ bpf_loop(nr_loops, loop_update_callback, &key, 0);
	38	+ percpu_time[cpu & 255] = bpf_ktime_get_ns() - start_time;
	39	+ return 0;
	40	+}

tools/testing/selftests/bpf/progs/btf__core_reloc_enum64val.c+3−0

@@ -0,0 +1,3 @@
	1	+#include "core_reloc_types.h"
	2	+
	3	+void f(struct core_reloc_enum64val x) {}

tools/testing/selftests/bpf/progs/btf__core_reloc_enum64val___diff.c+3−0

@@ -0,0 +1,3 @@
	1	+#include "core_reloc_types.h"
	2	+
	3	+void f(struct core_reloc_enum64val___diff x) {}

tools/testing/selftests/bpf/progs/btf__core_reloc_enum64val___err_missing.c+3−0

@@ -0,0 +1,3 @@
	1	+#include "core_reloc_types.h"
	2	+
	3	+void f(struct core_reloc_enum64val___err_missing x) {}

tools/testing/selftests/bpf/progs/btf__core_reloc_enum64val___val3_missing.c+3−0

@@ -0,0 +1,3 @@
	1	+#include "core_reloc_types.h"
	2	+
	3	+void f(struct core_reloc_enum64val___val3_missing x) {}

tools/testing/selftests/bpf/progs/core_reloc_types.h+78−0

@@ -1117,6 +1117,20 @@ struct core_reloc_enumval_output { · 1116 unmodified lines
1117	1117	int anon_val2;
1118	1118	};
1119	1119
	1120	+struct core_reloc_enum64val_output {
	1121	+ bool unsigned_val1_exists;
	1122	+ bool unsigned_val2_exists;
	1123	+ bool unsigned_val3_exists;
	1124	+ bool signed_val1_exists;
	1125	+ bool signed_val2_exists;
	1126	+ bool signed_val3_exists;
	1127	+
	1128	+ long unsigned_val1;
	1129	+ long unsigned_val2;
	1130	+ long signed_val1;
	1131	+ long signed_val2;
	1132	+};
	1133	+
1120	1134	enum named_enum {
1121	1135	NAMED_ENUM_VAL1 = 1,
1122	1136	NAMED_ENUM_VAL2 = 2,
@@ -1134,6 +1148,23 @@ struct core_reloc_enumval { · 11 unmodified lines
1134	1148	anon_enum f2;
1135	1149	};
1136	1150
	1151	+enum named_unsigned_enum64 {
	1152	+ UNSIGNED_ENUM64_VAL1 = 0x1ffffffffULL,
	1153	+ UNSIGNED_ENUM64_VAL2 = 0x2,
	1154	+ UNSIGNED_ENUM64_VAL3 = 0x3ffffffffULL,
	1155	+};
	1156	+
	1157	+enum named_signed_enum64 {
	1158	+ SIGNED_ENUM64_VAL1 = 0x1ffffffffLL,
	1159	+ SIGNED_ENUM64_VAL2 = -2,
	1160	+ SIGNED_ENUM64_VAL3 = 0x3ffffffffLL,
	1161	+};
	1162	+
	1163	+struct core_reloc_enum64val {
	1164	+ enum named_unsigned_enum64 f1;
	1165	+ enum named_signed_enum64 f2;
	1166	+};
	1167	+
1137	1168	/* differing enumerator values */
1138	1169	enum named_enum___diff {
1139	1170	NAMED_ENUM_VAL1___diff = 101,
@@ -1152,6 +1183,23 @@ struct core_reloc_enumval___diff { · 12 unmodified lines
1152	1183	anon_enum___diff f2;
1153	1184	};
1154	1185
	1186	+enum named_unsigned_enum64___diff {
	1187	+ UNSIGNED_ENUM64_VAL1___diff = 0x101ffffffffULL,
	1188	+ UNSIGNED_ENUM64_VAL2___diff = 0x202ffffffffULL,
	1189	+ UNSIGNED_ENUM64_VAL3___diff = 0x303ffffffffULL,
	1190	+};
	1191	+
	1192	+enum named_signed_enum64___diff {
	1193	+ SIGNED_ENUM64_VAL1___diff = -101,
	1194	+ SIGNED_ENUM64_VAL2___diff = -202,
	1195	+ SIGNED_ENUM64_VAL3___diff = -303,
	1196	+};
	1197	+
	1198	+struct core_reloc_enum64val___diff {
	1199	+ enum named_unsigned_enum64___diff f1;
	1200	+ enum named_signed_enum64___diff f2;
	1201	+};
	1202	+
1155	1203	/* missing (optional) third enum value */
1156	1204	enum named_enum___val3_missing {
1157	1205	NAMED_ENUM_VAL1___val3_missing = 111,
@@ -1168,6 +1216,21 @@ struct core_reloc_enumval___val3_missing { · 10 unmodified lines
1168	1216	anon_enum___val3_missing f2;
1169	1217	};
1170	1218
	1219	+enum named_unsigned_enum64___val3_missing {
	1220	+ UNSIGNED_ENUM64_VAL1___val3_missing = 0x111ffffffffULL,
	1221	+ UNSIGNED_ENUM64_VAL2___val3_missing = 0x222,
	1222	+};
	1223	+
	1224	+enum named_signed_enum64___val3_missing {
	1225	+ SIGNED_ENUM64_VAL1___val3_missing = 0x111ffffffffLL,
	1226	+ SIGNED_ENUM64_VAL2___val3_missing = -222,
	1227	+};
	1228	+
	1229	+struct core_reloc_enum64val___val3_missing {
	1230	+ enum named_unsigned_enum64___val3_missing f1;
	1231	+ enum named_signed_enum64___val3_missing f2;
	1232	+};
	1233	+
1171	1234	/* missing (mandatory) second enum value, should fail */
1172	1235	enum named_enum___err_missing {
1173	1236	NAMED_ENUM_VAL1___err_missing = 1,
@@ -1183,3 +1246,18 @@ struct core_reloc_enumval___err_missing { · 9 unmodified lines
1183	1246	enum named_enum___err_missing f1;
1184	1247	anon_enum___err_missing f2;
1185	1248	};
	1249	+
	1250	+enum named_unsigned_enum64___err_missing {
	1251	+ UNSIGNED_ENUM64_VAL1___err_missing = 0x1ffffffffULL,
	1252	+ UNSIGNED_ENUM64_VAL3___err_missing = 0x3ffffffffULL,
	1253	+};
	1254	+
	1255	+enum named_signed_enum64___err_missing {
	1256	+ SIGNED_ENUM64_VAL1___err_missing = 0x1ffffffffLL,
	1257	+ SIGNED_ENUM64_VAL3___err_missing = -3,
	1258	+};
	1259	+
	1260	+struct core_reloc_enum64val___err_missing {
	1261	+ enum named_unsigned_enum64___err_missing f1;
	1262	+ enum named_signed_enum64___err_missing f2;
	1263	+};

tools/testing/selftests/bpf/progs/test_attach_probe.c+60−0

@@ -5,6 +5,7 @@ · 4 unmodified lines
5	5	#include <linux/bpf.h>
6	6	#include <bpf/bpf_helpers.h>
7	7	#include <bpf/bpf_tracing.h>
	8	+#include <stdbool.h>
8	9	#include "bpf_misc.h"
9	10
10	11	int kprobe_res = 0;
@@ -17,6 +18,11 @@ int uprobe_byname_res = 0; · 6 unmodified lines
17	18	int uretprobe_byname_res = 0;
18	19	int uprobe_byname2_res = 0;
19	20	int uretprobe_byname2_res = 0;
	21	+int uprobe_byname3_sleepable_res = 0;
	22	+int uprobe_byname3_res = 0;
	23	+int uretprobe_byname3_sleepable_res = 0;
	24	+int uretprobe_byname3_res = 0;
	25	+void *user_ptr = 0;
20	26
21	27	SEC("kprobe")
22	28	int handle_kprobe(struct pt_regs *ctx)
@@ -32,6 +38,17 @@ int BPF_KPROBE(handle_kprobe_auto) · 9 unmodified lines
32	38	return 0;
33	39	}
34	40
	41	+/**
	42	+ * This program will be manually made sleepable on the userspace side
	43	+ * and should thus be unattachable.
	44	+ */
	45	+SEC("kprobe/" SYS_PREFIX "sys_nanosleep")
	46	+int handle_kprobe_sleepable(struct pt_regs *ctx)
	47	+{
	48	+ kprobe_res = 2;
	49	+ return 0;
	50	+}
	51	+
35	52	SEC("kretprobe")
36	53	int handle_kretprobe(struct pt_regs *ctx)
37	54	{
@@ -93,4 +110,47 @@ int handle_uretprobe_byname2(struct pt_regs *ctx) · 55 unmodified lines
93	110	return 0;
94	111	}
95	112
	113	+static __always_inline bool verify_sleepable_user_copy(void)
	114	+{
	115	+ char data[9];
	116	+
	117	+ bpf_copy_from_user(data, sizeof(data), user_ptr);
	118	+ return bpf_strncmp(data, sizeof(data), "test_data") == 0;
	119	+}
	120	+
	121	+SEC("uprobe.s//proc/self/exe:trigger_func3")
	122	+int handle_uprobe_byname3_sleepable(struct pt_regs *ctx)
	123	+{
	124	+ if (verify_sleepable_user_copy())
	125	+ uprobe_byname3_sleepable_res = 9;
	126	+ return 0;
	127	+}
	128	+
	129	+/**
	130	+ * same target as the uprobe.s above to force sleepable and non-sleepable
	131	+ * programs in the same bpf_prog_array
	132	+ */
	133	+SEC("uprobe//proc/self/exe:trigger_func3")
	134	+int handle_uprobe_byname3(struct pt_regs *ctx)
	135	+{
	136	+ uprobe_byname3_res = 10;
	137	+ return 0;
	138	+}
	139	+
	140	+SEC("uretprobe.s//proc/self/exe:trigger_func3")
	141	+int handle_uretprobe_byname3_sleepable(struct pt_regs *ctx)
	142	+{
	143	+ if (verify_sleepable_user_copy())
	144	+ uretprobe_byname3_sleepable_res = 11;
	145	+ return 0;
	146	+}
	147	+
	148	+SEC("uretprobe//proc/self/exe:trigger_func3")
	149	+int handle_uretprobe_byname3(struct pt_regs *ctx)
	150	+{
	151	+ uretprobe_byname3_res = 12;
	152	+ return 0;
	153	+}
	154	+
	155	+
96	156	char _license[] SEC("license") = "GPL";

tools/testing/selftests/bpf/progs/test_core_reloc_enum64val.c+70−0

@@ -0,0 +1,70 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */
	3	+
	4	+#include <linux/bpf.h>
	5	+#include <stdint.h>
	6	+#include <stdbool.h>
	7	+#include <bpf/bpf_helpers.h>
	8	+#include <bpf/bpf_core_read.h>
	9	+
	10	+char _license[] SEC("license") = "GPL";
	11	+
	12	+struct {
	13	+ char in[256];
	14	+ char out[256];
	15	+ bool skip;
	16	+} data = {};
	17	+
	18	+enum named_unsigned_enum64 {
	19	+ UNSIGNED_ENUM64_VAL1 = 0x1ffffffffULL,
	20	+ UNSIGNED_ENUM64_VAL2 = 0x2ffffffffULL,
	21	+ UNSIGNED_ENUM64_VAL3 = 0x3ffffffffULL,
	22	+};
	23	+
	24	+enum named_signed_enum64 {
	25	+ SIGNED_ENUM64_VAL1 = 0x1ffffffffLL,
	26	+ SIGNED_ENUM64_VAL2 = -2,
	27	+ SIGNED_ENUM64_VAL3 = 0x3ffffffffLL,
	28	+};
	29	+
	30	+struct core_reloc_enum64val_output {
	31	+ bool unsigned_val1_exists;
	32	+ bool unsigned_val2_exists;
	33	+ bool unsigned_val3_exists;
	34	+ bool signed_val1_exists;
	35	+ bool signed_val2_exists;
	36	+ bool signed_val3_exists;
	37	+
	38	+ long unsigned_val1;
	39	+ long unsigned_val2;
	40	+ long signed_val1;
	41	+ long signed_val2;
	42	+};
	43	+
	44	+SEC("raw_tracepoint/sys_enter")
	45	+int test_core_enum64val(void *ctx)
	46	+{
	47	+#if __clang_major__ >= 15
	48	+ struct core_reloc_enum64val_output out = (void )&data.out;
	49	+ enum named_unsigned_enum64 named_unsigned = 0;
	50	+ enum named_signed_enum64 named_signed = 0;
	51	+
	52	+ out->unsigned_val1_exists = bpf_core_enum_value_exists(named_unsigned, UNSIGNED_ENUM64_VAL1);
	53	+ out->unsigned_val2_exists = bpf_core_enum_value_exists(enum named_unsigned_enum64, UNSIGNED_ENUM64_VAL2);
	54	+ out->unsigned_val3_exists = bpf_core_enum_value_exists(enum named_unsigned_enum64, UNSIGNED_ENUM64_VAL3);
	55	+ out->signed_val1_exists = bpf_core_enum_value_exists(named_signed, SIGNED_ENUM64_VAL1);
	56	+ out->signed_val2_exists = bpf_core_enum_value_exists(enum named_signed_enum64, SIGNED_ENUM64_VAL2);
	57	+ out->signed_val3_exists = bpf_core_enum_value_exists(enum named_signed_enum64, SIGNED_ENUM64_VAL3);
	58	+
	59	+ out->unsigned_val1 = bpf_core_enum_value(named_unsigned, UNSIGNED_ENUM64_VAL1);
	60	+ out->unsigned_val2 = bpf_core_enum_value(named_unsigned, UNSIGNED_ENUM64_VAL2);
	61	+ out->signed_val1 = bpf_core_enum_value(named_signed, SIGNED_ENUM64_VAL1);
	62	+ out->signed_val2 = bpf_core_enum_value(named_signed, SIGNED_ENUM64_VAL2);
	63	+ /* NAMED_ENUM64_VAL3 value is optional */
	64	+
	65	+#else
	66	+ data.skip = true;
	67	+#endif
	68	+
	69	+ return 0;
	70	+}

tools/testing/selftests/bpf/progs/test_tc_dtime.c+51−2

@@ -11,6 +11,8 @@ · 10 unmodified lines
11	11	#include <linux/in.h>
12	12	#include <linux/ip.h>
13	13	#include <linux/ipv6.h>
	14	+#include <linux/tcp.h>
	15	+#include <linux/udp.h>
14	16	#include <bpf/bpf_helpers.h>
15	17	#include <bpf/bpf_endian.h>
16	18	#include <sys/socket.h>
@@ -115,6 +117,19 @@ static bool bpf_fwd(void) · 98 unmodified lines
115	117	return test < TCP_IP4_RT_FWD;
116	118	}
117	119
	120	+static __u8 get_proto(void)
	121	+{
	122	+ switch (test) {
	123	+ case UDP_IP4:
	124	+ case UDP_IP6:
	125	+ case UDP_IP4_RT_FWD:
	126	+ case UDP_IP6_RT_FWD:
	127	+ return IPPROTO_UDP;
	128	+ default:
	129	+ return IPPROTO_TCP;
	130	+ }
	131	+}
	132	+
118	133	/* -1: parse error: TC_ACT_SHOT
119	134	* 0: not testing traffic: TC_ACT_OK
120	135	* >0: first byte is the inet_proto, second byte has the netns
@@ -122,11 +137,16 @@ static bool bpf_fwd(void) · 1 unmodified line
122	137	*/
123	138	static int skb_get_type(struct __sk_buff *skb)
124	139	{
	140	+ __u16 dst_ns_port = __bpf_htons(50000 + test);
125	141	void *data_end = ctx_ptr(skb->data_end);
126	142	void *data = ctx_ptr(skb->data);
127	143	__u8 inet_proto = 0, ns = 0;
128	144	struct ipv6hdr *ip6h;
	145	+ __u16 sport, dport;
129	146	struct iphdr *iph;
	147	+ struct tcphdr *th;
	148	+ struct udphdr *uh;
	149	+ void *trans;
130	150
131	151	switch (skb->protocol) {
132	152	case __bpf_htons(ETH_P_IP):
@@ -138,6 +158,7 @@ static int skb_get_type(struct __sk_buff *skb) · 5 unmodified lines
138	158	else if (iph->saddr == ip4_dst)
139	159	ns = DST_NS;
140	160	inet_proto = iph->protocol;
	161	+ trans = iph + 1;
141	162	break;
142	163	case __bpf_htons(ETH_P_IPV6):
143	164	ip6h = data + sizeof(struct ethhdr);
@@ -148,15 +169,43 @@ static int skb_get_type(struct __sk_buff *skb) · 4 unmodified lines
148	169	else if (v6_equal(ip6h->saddr, (struct in6_addr)ip6_dst))
149	170	ns = DST_NS;
150	171	inet_proto = ip6h->nexthdr;
	172	+ trans = ip6h + 1;
151	173	break;
152	174	default:
153	175	return 0;
154	176	}
155	177
156		− if ((inet_proto != IPPROTO_TCP && inet_proto != IPPROTO_UDP) \|\| !ns)
	178	+ /* skb is not from src_ns or dst_ns.
	179	+ * skb is not the testing IPPROTO.
	180	+ */
	181	+ if (!ns \|\| inet_proto != get_proto())
157	182	return 0;
158	183
159		− return (ns << 8 \| inet_proto);
	184	+ switch (inet_proto) {
	185	+ case IPPROTO_TCP:
	186	+ th = trans;
	187	+ if (th + 1 > data_end)
	188	+ return -1;
	189	+ sport = th->source;
	190	+ dport = th->dest;
	191	+ break;
	192	+ case IPPROTO_UDP:
	193	+ uh = trans;
	194	+ if (uh + 1 > data_end)
	195	+ return -1;
	196	+ sport = uh->source;
	197	+ dport = uh->dest;
	198	+ break;
	199	+ default:
	200	+ return 0;
	201	+ }
	202	+
	203	+ /* The skb is the testing traffic */
	204	+ if ((ns == SRC_NS && dport == dst_ns_port) \|\|
	205	+ (ns == DST_NS && sport == dst_ns_port))
	206	+ return (ns << 8 \| inet_proto);
	207	+
	208	+ return 0;
160	209	}
161	210
162	211	/* format: direction@iface@netns

tools/testing/selftests/bpf/progs/test_varlen.c+4−4

@@ -41,20 +41,20 @@ int handler64_unsigned(void *regs) · 40 unmodified lines
41	41	{
42	42	int pid = bpf_get_current_pid_tgid() >> 32;
43	43	void *payload = payload1;
44		− u64 len;
	44	+ long len;
45	45
46	46	/* ignore irrelevant invocations */
47	47	if (test_pid != pid \|\| !capture)
48	48	return 0;
49	49
50	50	len = bpf_probe_read_kernel_str(payload, MAX_LEN, &buf_in1[0]);
51		− if (len <= MAX_LEN) {
	51	+ if (len >= 0) {
52	52	payload += len;
53	53	payload1_len1 = len;
54	54	}
55	55
56	56	len = bpf_probe_read_kernel_str(payload, MAX_LEN, &buf_in2[0]);
57		− if (len <= MAX_LEN) {
	57	+ if (len >= 0) {
58	58	payload += len;
59	59	payload1_len2 = len;
60	60	}
@@ -123,7 +123,7 @@ int handler32_signed(void *regs) · 62 unmodified lines
123	123	{
124	124	int pid = bpf_get_current_pid_tgid() >> 32;
125	125	void *payload = payload4;
126		− int len;
	126	+ long len;
127	127
128	128	/* ignore irrelevant invocations */
129	129	if (test_pid != pid \|\| !capture)

tools/testing/selftests/bpf/progs/xdp_synproxy_kern.c+833−0

@@ -0,0 +1,833 @@
	1	+// SPDX-License-Identifier: LGPL-2.1 OR BSD-2-Clause
	2	+/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
	3	+
	4	+#include "vmlinux.h"
	5	+
	6	+#include <bpf/bpf_helpers.h>
	7	+#include <bpf/bpf_endian.h>
	8	+#include <asm/errno.h>
	9	+
	10	+#define TC_ACT_OK 0
	11	+#define TC_ACT_SHOT 2
	12	+
	13	+#define NSEC_PER_SEC 1000000000L
	14	+
	15	+#define ETH_ALEN 6
	16	+#define ETH_P_IP 0x0800
	17	+#define ETH_P_IPV6 0x86DD
	18	+
	19	+#define tcp_flag_word(tp) (((union tcp_word_hdr *)(tp))->words[3])
	20	+
	21	+#define IP_DF 0x4000
	22	+#define IP_MF 0x2000
	23	+#define IP_OFFSET 0x1fff
	24	+
	25	+#define NEXTHDR_TCP 6
	26	+
	27	+#define TCPOPT_NOP 1
	28	+#define TCPOPT_EOL 0
	29	+#define TCPOPT_MSS 2
	30	+#define TCPOPT_WINDOW 3
	31	+#define TCPOPT_SACK_PERM 4
	32	+#define TCPOPT_TIMESTAMP 8
	33	+
	34	+#define TCPOLEN_MSS 4
	35	+#define TCPOLEN_WINDOW 3
	36	+#define TCPOLEN_SACK_PERM 2
	37	+#define TCPOLEN_TIMESTAMP 10
	38	+
	39	+#define TCP_TS_HZ 1000
	40	+#define TS_OPT_WSCALE_MASK 0xf
	41	+#define TS_OPT_SACK (1 << 4)
	42	+#define TS_OPT_ECN (1 << 5)
	43	+#define TSBITS 6
	44	+#define TSMASK (((__u32)1 << TSBITS) - 1)
	45	+#define TCP_MAX_WSCALE 14U
	46	+
	47	+#define IPV4_MAXLEN 60
	48	+#define TCP_MAXLEN 60
	49	+
	50	+#define DEFAULT_MSS4 1460
	51	+#define DEFAULT_MSS6 1440
	52	+#define DEFAULT_WSCALE 7
	53	+#define DEFAULT_TTL 64
	54	+#define MAX_ALLOWED_PORTS 8
	55	+
	56	+#define swap(a, b) \
	57	+ do { typeof(a) __tmp = (a); (a) = (b); (b) = __tmp; } while (0)
	58	+
	59	+#define __get_unaligned_t(type, ptr) ({ \
	60	+ const struct { type x; } __attribute__((__packed__)) *__pptr = (typeof(__pptr))(ptr); \
	61	+ __pptr->x; \
	62	+})
	63	+
	64	+#define get_unaligned(ptr) __get_unaligned_t(typeof(*(ptr)), (ptr))
	65	+
	66	+struct {
	67	+ __uint(type, BPF_MAP_TYPE_ARRAY);
	68	+ __type(key, __u32);
	69	+ __type(value, __u64);
	70	+ __uint(max_entries, 2);
	71	+} values SEC(".maps");
	72	+
	73	+struct {
	74	+ __uint(type, BPF_MAP_TYPE_ARRAY);
	75	+ __type(key, __u32);
	76	+ __type(value, __u16);
	77	+ __uint(max_entries, MAX_ALLOWED_PORTS);
	78	+} allowed_ports SEC(".maps");
	79	+
	80	+extern struct nf_conn bpf_xdp_ct_lookup(struct xdp_md xdp_ctx,
	81	+ struct bpf_sock_tuple *bpf_tuple,
	82	+ __u32 len_tuple,
	83	+ struct bpf_ct_opts *opts,
	84	+ __u32 len_opts) __ksym;
	85	+
	86	+extern struct nf_conn bpf_skb_ct_lookup(struct __sk_buff skb_ctx,
	87	+ struct bpf_sock_tuple *bpf_tuple,
	88	+ u32 len_tuple,
	89	+ struct bpf_ct_opts *opts,
	90	+ u32 len_opts) __ksym;
	91	+
	92	+extern void bpf_ct_release(struct nf_conn *ct) __ksym;
	93	+
	94	+static __always_inline void swap_eth_addr(__u8 a, __u8 b)
	95	+{
	96	+ __u8 tmp[ETH_ALEN];
	97	+
	98	+ __builtin_memcpy(tmp, a, ETH_ALEN);
	99	+ __builtin_memcpy(a, b, ETH_ALEN);
	100	+ __builtin_memcpy(b, tmp, ETH_ALEN);
	101	+}
	102	+
	103	+static __always_inline __u16 csum_fold(__u32 csum)
	104	+{
	105	+ csum = (csum & 0xffff) + (csum >> 16);
	106	+ csum = (csum & 0xffff) + (csum >> 16);
	107	+ return (__u16)~csum;
	108	+}
	109	+
	110	+static __always_inline __u16 csum_tcpudp_magic(__be32 saddr, __be32 daddr,
	111	+ __u32 len, __u8 proto,
	112	+ __u32 csum)
	113	+{
	114	+ __u64 s = csum;
	115	+
	116	+ s += (__u32)saddr;
	117	+ s += (__u32)daddr;
	118	+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
	119	+ s += proto + len;
	120	+#elif __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
	121	+ s += (proto + len) << 8;
	122	+#else
	123	+#error Unknown endian
	124	+#endif
	125	+ s = (s & 0xffffffff) + (s >> 32);
	126	+ s = (s & 0xffffffff) + (s >> 32);
	127	+
	128	+ return csum_fold((__u32)s);
	129	+}
	130	+
	131	+static __always_inline __u16 csum_ipv6_magic(const struct in6_addr *saddr,
	132	+ const struct in6_addr *daddr,
	133	+ __u32 len, __u8 proto, __u32 csum)
	134	+{
	135	+ __u64 sum = csum;
	136	+ int i;
	137	+
	138	+#pragma unroll
	139	+ for (i = 0; i < 4; i++)
	140	+ sum += (__u32)saddr->in6_u.u6_addr32[i];
	141	+
	142	+#pragma unroll
	143	+ for (i = 0; i < 4; i++)
	144	+ sum += (__u32)daddr->in6_u.u6_addr32[i];
	145	+
	146	+ /* Don't combine additions to avoid 32-bit overflow. */
	147	+ sum += bpf_htonl(len);
	148	+ sum += bpf_htonl(proto);
	149	+
	150	+ sum = (sum & 0xffffffff) + (sum >> 32);
	151	+ sum = (sum & 0xffffffff) + (sum >> 32);
	152	+
	153	+ return csum_fold((__u32)sum);
	154	+}
	155	+
	156	+static __always_inline __u64 tcp_clock_ns(void)
	157	+{
	158	+ return bpf_ktime_get_ns();
	159	+}
	160	+
	161	+static __always_inline __u32 tcp_ns_to_ts(__u64 ns)
	162	+{
	163	+ return ns / (NSEC_PER_SEC / TCP_TS_HZ);
	164	+}
	165	+
	166	+static __always_inline __u32 tcp_time_stamp_raw(void)
	167	+{
	168	+ return tcp_ns_to_ts(tcp_clock_ns());
	169	+}
	170	+
	171	+struct tcpopt_context {
	172	+ __u8 *ptr;
	173	+ __u8 *end;
	174	+ void *data_end;
	175	+ __be32 *tsecr;
	176	+ __u8 wscale;
	177	+ bool option_timestamp;
	178	+ bool option_sack;
	179	+};
	180	+
	181	+static int tscookie_tcpopt_parse(struct tcpopt_context *ctx)
	182	+{
	183	+ __u8 opcode, opsize;
	184	+
	185	+ if (ctx->ptr >= ctx->end)
	186	+ return 1;
	187	+ if (ctx->ptr >= ctx->data_end)
	188	+ return 1;
	189	+
	190	+ opcode = ctx->ptr[0];
	191	+
	192	+ if (opcode == TCPOPT_EOL)
	193	+ return 1;
	194	+ if (opcode == TCPOPT_NOP) {
	195	+ ++ctx->ptr;
	196	+ return 0;
	197	+ }
	198	+
	199	+ if (ctx->ptr + 1 >= ctx->end)
	200	+ return 1;
	201	+ if (ctx->ptr + 1 >= ctx->data_end)
	202	+ return 1;
	203	+ opsize = ctx->ptr[1];
	204	+ if (opsize < 2)
	205	+ return 1;
	206	+
	207	+ if (ctx->ptr + opsize > ctx->end)
	208	+ return 1;
	209	+
	210	+ switch (opcode) {
	211	+ case TCPOPT_WINDOW:
	212	+ if (opsize == TCPOLEN_WINDOW && ctx->ptr + TCPOLEN_WINDOW <= ctx->data_end)
	213	+ ctx->wscale = ctx->ptr[2] < TCP_MAX_WSCALE ? ctx->ptr[2] : TCP_MAX_WSCALE;
	214	+ break;
	215	+ case TCPOPT_TIMESTAMP:
	216	+ if (opsize == TCPOLEN_TIMESTAMP && ctx->ptr + TCPOLEN_TIMESTAMP <= ctx->data_end) {
	217	+ ctx->option_timestamp = true;
	218	+ /* Client's tsval becomes our tsecr. */
	219	+ ctx->tsecr = get_unaligned((__be32 )(ctx->ptr + 2));
	220	+ }
	221	+ break;
	222	+ case TCPOPT_SACK_PERM:
	223	+ if (opsize == TCPOLEN_SACK_PERM)
	224	+ ctx->option_sack = true;
	225	+ break;
	226	+ }
	227	+
	228	+ ctx->ptr += opsize;
	229	+
	230	+ return 0;
	231	+}
	232	+
	233	+static int tscookie_tcpopt_parse_batch(__u32 index, void *context)
	234	+{
	235	+ int i;
	236	+
	237	+ for (i = 0; i < 7; i++)
	238	+ if (tscookie_tcpopt_parse(context))
	239	+ return 1;
	240	+ return 0;
	241	+}
	242	+
	243	+static __always_inline bool tscookie_init(struct tcphdr *tcp_header,
	244	+ __u16 tcp_len, __be32 *tsval,
	245	+ __be32 tsecr, void data_end)
	246	+{
	247	+ struct tcpopt_context loop_ctx = {
	248	+ .ptr = (__u8 *)(tcp_header + 1),
	249	+ .end = (__u8 *)tcp_header + tcp_len,
	250	+ .data_end = data_end,
	251	+ .tsecr = tsecr,
	252	+ .wscale = TS_OPT_WSCALE_MASK,
	253	+ .option_timestamp = false,
	254	+ .option_sack = false,
	255	+ };
	256	+ u32 cookie;
	257	+
	258	+ bpf_loop(6, tscookie_tcpopt_parse_batch, &loop_ctx, 0);
	259	+
	260	+ if (!loop_ctx.option_timestamp)
	261	+ return false;
	262	+
	263	+ cookie = tcp_time_stamp_raw() & ~TSMASK;
	264	+ cookie \|= loop_ctx.wscale & TS_OPT_WSCALE_MASK;
	265	+ if (loop_ctx.option_sack)
	266	+ cookie \|= TS_OPT_SACK;
	267	+ if (tcp_header->ece && tcp_header->cwr)
	268	+ cookie \|= TS_OPT_ECN;
	269	+ *tsval = bpf_htonl(cookie);
	270	+
	271	+ return true;
	272	+}
	273	+
	274	+static __always_inline void values_get_tcpipopts(__u16 mss, __u8 wscale,
	275	+ __u8 *ttl, bool ipv6)
	276	+{
	277	+ __u32 key = 0;
	278	+ __u64 *value;
	279	+
	280	+ value = bpf_map_lookup_elem(&values, &key);
	281	+ if (value && *value != 0) {
	282	+ if (ipv6)
	283	+ mss = (value >> 32) & 0xffff;
	284	+ else
	285	+ mss = value & 0xffff;
	286	+ wscale = (value >> 16) & 0xf;
	287	+ ttl = (value >> 24) & 0xff;
	288	+ return;
	289	+ }
	290	+
	291	+ *mss = ipv6 ? DEFAULT_MSS6 : DEFAULT_MSS4;
	292	+ *wscale = DEFAULT_WSCALE;
	293	+ *ttl = DEFAULT_TTL;
	294	+}
	295	+
	296	+static __always_inline void values_inc_synacks(void)
	297	+{
	298	+ __u32 key = 1;
	299	+ __u32 *value;
	300	+
	301	+ value = bpf_map_lookup_elem(&values, &key);
	302	+ if (value)
	303	+ __sync_fetch_and_add(value, 1);
	304	+}
	305	+
	306	+static __always_inline bool check_port_allowed(__u16 port)
	307	+{
	308	+ __u32 i;
	309	+
	310	+ for (i = 0; i < MAX_ALLOWED_PORTS; i++) {
	311	+ __u32 key = i;
	312	+ __u16 *value;
	313	+
	314	+ value = bpf_map_lookup_elem(&allowed_ports, &key);
	315	+
	316	+ if (!value)
	317	+ break;
	318	+ /* 0 is a terminator value. Check it first to avoid matching on
	319	+ * a forbidden port == 0 and returning true.
	320	+ */
	321	+ if (*value == 0)
	322	+ break;
	323	+
	324	+ if (*value == port)
	325	+ return true;
	326	+ }
	327	+
	328	+ return false;
	329	+}
	330	+
	331	+struct header_pointers {
	332	+ struct ethhdr *eth;
	333	+ struct iphdr *ipv4;
	334	+ struct ipv6hdr *ipv6;
	335	+ struct tcphdr *tcp;
	336	+ __u16 tcp_len;
	337	+};
	338	+
	339	+static __always_inline int tcp_dissect(void data, void data_end,
	340	+ struct header_pointers *hdr)
	341	+{
	342	+ hdr->eth = data;
	343	+ if (hdr->eth + 1 > data_end)
	344	+ return XDP_DROP;
	345	+
	346	+ switch (bpf_ntohs(hdr->eth->h_proto)) {
	347	+ case ETH_P_IP:
	348	+ hdr->ipv6 = NULL;
	349	+
	350	+ hdr->ipv4 = (void )hdr->eth + sizeof(hdr->eth);
	351	+ if (hdr->ipv4 + 1 > data_end)
	352	+ return XDP_DROP;
	353	+ if (hdr->ipv4->ihl * 4 < sizeof(*hdr->ipv4))
	354	+ return XDP_DROP;
	355	+ if (hdr->ipv4->version != 4)
	356	+ return XDP_DROP;
	357	+
	358	+ if (hdr->ipv4->protocol != IPPROTO_TCP)
	359	+ return XDP_PASS;
	360	+
	361	+ hdr->tcp = (void )hdr->ipv4 + hdr->ipv4->ihl 4;
	362	+ break;
	363	+ case ETH_P_IPV6:
	364	+ hdr->ipv4 = NULL;
	365	+
	366	+ hdr->ipv6 = (void )hdr->eth + sizeof(hdr->eth);
	367	+ if (hdr->ipv6 + 1 > data_end)
	368	+ return XDP_DROP;
	369	+ if (hdr->ipv6->version != 6)
	370	+ return XDP_DROP;
	371	+
	372	+ /* XXX: Extension headers are not supported and could circumvent
	373	+ * XDP SYN flood protection.
	374	+ */
	375	+ if (hdr->ipv6->nexthdr != NEXTHDR_TCP)
	376	+ return XDP_PASS;
	377	+
	378	+ hdr->tcp = (void )hdr->ipv6 + sizeof(hdr->ipv6);
	379	+ break;
	380	+ default:
	381	+ /* XXX: VLANs will circumvent XDP SYN flood protection. */
	382	+ return XDP_PASS;
	383	+ }
	384	+
	385	+ if (hdr->tcp + 1 > data_end)
	386	+ return XDP_DROP;
	387	+ hdr->tcp_len = hdr->tcp->doff * 4;
	388	+ if (hdr->tcp_len < sizeof(*hdr->tcp))
	389	+ return XDP_DROP;
	390	+
	391	+ return XDP_TX;
	392	+}
	393	+
	394	+static __always_inline int tcp_lookup(void ctx, struct header_pointers hdr, bool xdp)
	395	+{
	396	+ struct bpf_ct_opts ct_lookup_opts = {
	397	+ .netns_id = BPF_F_CURRENT_NETNS,
	398	+ .l4proto = IPPROTO_TCP,
	399	+ };
	400	+ struct bpf_sock_tuple tup = {};
	401	+ struct nf_conn *ct;
	402	+ __u32 tup_size;
	403	+
	404	+ if (hdr->ipv4) {
	405	+ /* TCP doesn't normally use fragments, and XDP can't reassemble
	406	+ * them.
	407	+ */
	408	+ if ((hdr->ipv4->frag_off & bpf_htons(IP_DF \| IP_MF \| IP_OFFSET)) != bpf_htons(IP_DF))
	409	+ return XDP_DROP;
	410	+
	411	+ tup.ipv4.saddr = hdr->ipv4->saddr;
	412	+ tup.ipv4.daddr = hdr->ipv4->daddr;
	413	+ tup.ipv4.sport = hdr->tcp->source;
	414	+ tup.ipv4.dport = hdr->tcp->dest;
	415	+ tup_size = sizeof(tup.ipv4);
	416	+ } else if (hdr->ipv6) {
	417	+ __builtin_memcpy(tup.ipv6.saddr, &hdr->ipv6->saddr, sizeof(tup.ipv6.saddr));
	418	+ __builtin_memcpy(tup.ipv6.daddr, &hdr->ipv6->daddr, sizeof(tup.ipv6.daddr));
	419	+ tup.ipv6.sport = hdr->tcp->source;
	420	+ tup.ipv6.dport = hdr->tcp->dest;
	421	+ tup_size = sizeof(tup.ipv6);
	422	+ } else {
	423	+ /* The verifier can't track that either ipv4 or ipv6 is not
	424	+ * NULL.
	425	+ */
	426	+ return XDP_ABORTED;
	427	+ }
	428	+ if (xdp)
	429	+ ct = bpf_xdp_ct_lookup(ctx, &tup, tup_size, &ct_lookup_opts, sizeof(ct_lookup_opts));
	430	+ else
	431	+ ct = bpf_skb_ct_lookup(ctx, &tup, tup_size, &ct_lookup_opts, sizeof(ct_lookup_opts));
	432	+ if (ct) {
	433	+ unsigned long status = ct->status;
	434	+
	435	+ bpf_ct_release(ct);
	436	+ if (status & IPS_CONFIRMED_BIT)
	437	+ return XDP_PASS;
	438	+ } else if (ct_lookup_opts.error != -ENOENT) {
	439	+ return XDP_ABORTED;
	440	+ }
	441	+
	442	+ /* error == -ENOENT \|\| !(status & IPS_CONFIRMED_BIT) */
	443	+ return XDP_TX;
	444	+}
	445	+
	446	+static __always_inline __u8 tcp_mkoptions(__be32 buf, __be32 tsopt, __u16 mss,
	447	+ __u8 wscale)
	448	+{
	449	+ __be32 *start = buf;
	450	+
	451	+ *buf++ = bpf_htonl((TCPOPT_MSS << 24) \| (TCPOLEN_MSS << 16) \| mss);
	452	+
	453	+ if (!tsopt)
	454	+ return buf - start;
	455	+
	456	+ if (tsopt[0] & bpf_htonl(1 << 4))
	457	+ *buf++ = bpf_htonl((TCPOPT_SACK_PERM << 24) \|
	458	+ (TCPOLEN_SACK_PERM << 16) \|
	459	+ (TCPOPT_TIMESTAMP << 8) \|
	460	+ TCPOLEN_TIMESTAMP);
	461	+ else
	462	+ *buf++ = bpf_htonl((TCPOPT_NOP << 24) \|
	463	+ (TCPOPT_NOP << 16) \|
	464	+ (TCPOPT_TIMESTAMP << 8) \|
	465	+ TCPOLEN_TIMESTAMP);
	466	+ *buf++ = tsopt[0];
	467	+ *buf++ = tsopt[1];
	468	+
	469	+ if ((tsopt[0] & bpf_htonl(0xf)) != bpf_htonl(0xf))
	470	+ *buf++ = bpf_htonl((TCPOPT_NOP << 24) \|
	471	+ (TCPOPT_WINDOW << 16) \|
	472	+ (TCPOLEN_WINDOW << 8) \|
	473	+ wscale);
	474	+
	475	+ return buf - start;
	476	+}
	477	+
	478	+static __always_inline void tcp_gen_synack(struct tcphdr *tcp_header,
	479	+ __u32 cookie, __be32 *tsopt,
	480	+ __u16 mss, __u8 wscale)
	481	+{
	482	+ void *tcp_options;
	483	+
	484	+ tcp_flag_word(tcp_header) = TCP_FLAG_SYN \| TCP_FLAG_ACK;
	485	+ if (tsopt && (tsopt[0] & bpf_htonl(1 << 5)))
	486	+ tcp_flag_word(tcp_header) \|= TCP_FLAG_ECE;
	487	+ tcp_header->doff = 5; /* doff is part of tcp_flag_word. */
	488	+ swap(tcp_header->source, tcp_header->dest);
	489	+ tcp_header->ack_seq = bpf_htonl(bpf_ntohl(tcp_header->seq) + 1);
	490	+ tcp_header->seq = bpf_htonl(cookie);
	491	+ tcp_header->window = 0;
	492	+ tcp_header->urg_ptr = 0;
	493	+ tcp_header->check = 0; /* Calculate checksum later. */
	494	+
	495	+ tcp_options = (void *)(tcp_header + 1);
	496	+ tcp_header->doff += tcp_mkoptions(tcp_options, tsopt, mss, wscale);
	497	+}
	498	+
	499	+static __always_inline void tcpv4_gen_synack(struct header_pointers *hdr,
	500	+ __u32 cookie, __be32 *tsopt)
	501	+{
	502	+ __u8 wscale;
	503	+ __u16 mss;
	504	+ __u8 ttl;
	505	+
	506	+ values_get_tcpipopts(&mss, &wscale, &ttl, false);
	507	+
	508	+ swap_eth_addr(hdr->eth->h_source, hdr->eth->h_dest);
	509	+
	510	+ swap(hdr->ipv4->saddr, hdr->ipv4->daddr);
	511	+ hdr->ipv4->check = 0; /* Calculate checksum later. */
	512	+ hdr->ipv4->tos = 0;
	513	+ hdr->ipv4->id = 0;
	514	+ hdr->ipv4->ttl = ttl;
	515	+
	516	+ tcp_gen_synack(hdr->tcp, cookie, tsopt, mss, wscale);
	517	+
	518	+ hdr->tcp_len = hdr->tcp->doff * 4;
	519	+ hdr->ipv4->tot_len = bpf_htons(sizeof(*hdr->ipv4) + hdr->tcp_len);
	520	+}
	521	+
	522	+static __always_inline void tcpv6_gen_synack(struct header_pointers *hdr,
	523	+ __u32 cookie, __be32 *tsopt)
	524	+{
	525	+ __u8 wscale;
	526	+ __u16 mss;
	527	+ __u8 ttl;
	528	+
	529	+ values_get_tcpipopts(&mss, &wscale, &ttl, true);
	530	+
	531	+ swap_eth_addr(hdr->eth->h_source, hdr->eth->h_dest);
	532	+
	533	+ swap(hdr->ipv6->saddr, hdr->ipv6->daddr);
	534	+ (__be32 )hdr->ipv6 = bpf_htonl(0x60000000);
	535	+ hdr->ipv6->hop_limit = ttl;
	536	+
	537	+ tcp_gen_synack(hdr->tcp, cookie, tsopt, mss, wscale);
	538	+
	539	+ hdr->tcp_len = hdr->tcp->doff * 4;
	540	+ hdr->ipv6->payload_len = bpf_htons(hdr->tcp_len);
	541	+}
	542	+
	543	+static __always_inline int syncookie_handle_syn(struct header_pointers *hdr,
	544	+ void *ctx,
	545	+ void data, void data_end,
	546	+ bool xdp)
	547	+{
	548	+ __u32 old_pkt_size, new_pkt_size;
	549	+ /* Unlike clang 10, clang 11 and 12 generate code that doesn't pass the
	550	+ * BPF verifier if tsopt is not volatile. Volatile forces it to store
	551	+ * the pointer value and use it directly, otherwise tcp_mkoptions is
	552	+ * (mis)compiled like this:
	553	+ * if (!tsopt)
	554	+ * return buf - start;
	555	+ * reg = stored_return_value_of_tscookie_init;
	556	+ * if (reg)
	557	+ * tsopt = tsopt_buf;
	558	+ * else
	559	+ * tsopt = NULL;
	560	+ * ...
	561	+ * *buf++ = tsopt[1];
	562	+ * It creates a dead branch where tsopt is assigned NULL, but the
	563	+ * verifier can't prove it's dead and blocks the program.
	564	+ */
	565	+ __be32 * volatile tsopt = NULL;
	566	+ __be32 tsopt_buf[2] = {};
	567	+ __u16 ip_len;
	568	+ __u32 cookie;
	569	+ __s64 value;
	570	+
	571	+ /* Checksum is not yet verified, but both checksum failure and TCP
	572	+ * header checks return XDP_DROP, so the order doesn't matter.
	573	+ */
	574	+ if (hdr->tcp->fin \|\| hdr->tcp->rst)
	575	+ return XDP_DROP;
	576	+
	577	+ /* Issue SYN cookies on allowed ports, drop SYN packets on blocked
	578	+ * ports.
	579	+ */
	580	+ if (!check_port_allowed(bpf_ntohs(hdr->tcp->dest)))
	581	+ return XDP_DROP;
	582	+
	583	+ if (hdr->ipv4) {
	584	+ /* Check the IPv4 and TCP checksums before creating a SYNACK. */
	585	+ value = bpf_csum_diff(0, 0, (void )hdr->ipv4, hdr->ipv4->ihl 4, 0);
	586	+ if (value < 0)
	587	+ return XDP_ABORTED;
	588	+ if (csum_fold(value) != 0)
	589	+ return XDP_DROP; /* Bad IPv4 checksum. */
	590	+
	591	+ value = bpf_csum_diff(0, 0, (void *)hdr->tcp, hdr->tcp_len, 0);
	592	+ if (value < 0)
	593	+ return XDP_ABORTED;
	594	+ if (csum_tcpudp_magic(hdr->ipv4->saddr, hdr->ipv4->daddr,
	595	+ hdr->tcp_len, IPPROTO_TCP, value) != 0)
	596	+ return XDP_DROP; /* Bad TCP checksum. */
	597	+
	598	+ ip_len = sizeof(*hdr->ipv4);
	599	+
	600	+ value = bpf_tcp_raw_gen_syncookie_ipv4(hdr->ipv4, hdr->tcp,
	601	+ hdr->tcp_len);
	602	+ } else if (hdr->ipv6) {
	603	+ /* Check the TCP checksum before creating a SYNACK. */
	604	+ value = bpf_csum_diff(0, 0, (void *)hdr->tcp, hdr->tcp_len, 0);
	605	+ if (value < 0)
	606	+ return XDP_ABORTED;
	607	+ if (csum_ipv6_magic(&hdr->ipv6->saddr, &hdr->ipv6->daddr,
	608	+ hdr->tcp_len, IPPROTO_TCP, value) != 0)
	609	+ return XDP_DROP; /* Bad TCP checksum. */
	610	+
	611	+ ip_len = sizeof(*hdr->ipv6);
	612	+
	613	+ value = bpf_tcp_raw_gen_syncookie_ipv6(hdr->ipv6, hdr->tcp,
	614	+ hdr->tcp_len);
	615	+ } else {
	616	+ return XDP_ABORTED;
	617	+ }
	618	+
	619	+ if (value < 0)
	620	+ return XDP_ABORTED;
	621	+ cookie = (__u32)value;
	622	+
	623	+ if (tscookie_init((void *)hdr->tcp, hdr->tcp_len,
	624	+ &tsopt_buf[0], &tsopt_buf[1], data_end))
	625	+ tsopt = tsopt_buf;
	626	+
	627	+ /* Check that there is enough space for a SYNACK. It also covers
	628	+ * the check that the destination of the __builtin_memmove below
	629	+ * doesn't overflow.
	630	+ */
	631	+ if (data + sizeof(*hdr->eth) + ip_len + TCP_MAXLEN > data_end)
	632	+ return XDP_ABORTED;
	633	+
	634	+ if (hdr->ipv4) {
	635	+ if (hdr->ipv4->ihl * 4 > sizeof(*hdr->ipv4)) {
	636	+ struct tcphdr *new_tcp_header;
	637	+
	638	+ new_tcp_header = data + sizeof(hdr->eth) + sizeof(hdr->ipv4);
	639	+ __builtin_memmove(new_tcp_header, hdr->tcp, sizeof(*hdr->tcp));
	640	+ hdr->tcp = new_tcp_header;
	641	+
	642	+ hdr->ipv4->ihl = sizeof(*hdr->ipv4) / 4;
	643	+ }
	644	+
	645	+ tcpv4_gen_synack(hdr, cookie, tsopt);
	646	+ } else if (hdr->ipv6) {
	647	+ tcpv6_gen_synack(hdr, cookie, tsopt);
	648	+ } else {
	649	+ return XDP_ABORTED;
	650	+ }
	651	+
	652	+ /* Recalculate checksums. */
	653	+ hdr->tcp->check = 0;
	654	+ value = bpf_csum_diff(0, 0, (void *)hdr->tcp, hdr->tcp_len, 0);
	655	+ if (value < 0)
	656	+ return XDP_ABORTED;
	657	+ if (hdr->ipv4) {
	658	+ hdr->tcp->check = csum_tcpudp_magic(hdr->ipv4->saddr,
	659	+ hdr->ipv4->daddr,
	660	+ hdr->tcp_len,
	661	+ IPPROTO_TCP,
	662	+ value);
	663	+
	664	+ hdr->ipv4->check = 0;
	665	+ value = bpf_csum_diff(0, 0, (void )hdr->ipv4, sizeof(hdr->ipv4), 0);
	666	+ if (value < 0)
	667	+ return XDP_ABORTED;
	668	+ hdr->ipv4->check = csum_fold(value);
	669	+ } else if (hdr->ipv6) {
	670	+ hdr->tcp->check = csum_ipv6_magic(&hdr->ipv6->saddr,
	671	+ &hdr->ipv6->daddr,
	672	+ hdr->tcp_len,
	673	+ IPPROTO_TCP,
	674	+ value);
	675	+ } else {
	676	+ return XDP_ABORTED;
	677	+ }
	678	+
	679	+ /* Set the new packet size. */
	680	+ old_pkt_size = data_end - data;
	681	+ new_pkt_size = sizeof(hdr->eth) + ip_len + hdr->tcp->doff 4;
	682	+ if (xdp) {
	683	+ if (bpf_xdp_adjust_tail(ctx, new_pkt_size - old_pkt_size))
	684	+ return XDP_ABORTED;
	685	+ } else {
	686	+ if (bpf_skb_change_tail(ctx, new_pkt_size, 0))
	687	+ return XDP_ABORTED;
	688	+ }
	689	+
	690	+ values_inc_synacks();
	691	+
	692	+ return XDP_TX;
	693	+}
	694	+
	695	+static __always_inline int syncookie_handle_ack(struct header_pointers *hdr)
	696	+{
	697	+ int err;
	698	+
	699	+ if (hdr->tcp->rst)
	700	+ return XDP_DROP;
	701	+
	702	+ if (hdr->ipv4)
	703	+ err = bpf_tcp_raw_check_syncookie_ipv4(hdr->ipv4, hdr->tcp);
	704	+ else if (hdr->ipv6)
	705	+ err = bpf_tcp_raw_check_syncookie_ipv6(hdr->ipv6, hdr->tcp);
	706	+ else
	707	+ return XDP_ABORTED;
	708	+ if (err)
	709	+ return XDP_DROP;
	710	+
	711	+ return XDP_PASS;
	712	+}
	713	+
	714	+static __always_inline int syncookie_part1(void ctx, void data, void *data_end,
	715	+ struct header_pointers *hdr, bool xdp)
	716	+{
	717	+ struct bpf_ct_opts ct_lookup_opts = {
	718	+ .netns_id = BPF_F_CURRENT_NETNS,
	719	+ .l4proto = IPPROTO_TCP,
	720	+ };
	721	+ int ret;
	722	+
	723	+ ret = tcp_dissect(data, data_end, hdr);
	724	+ if (ret != XDP_TX)
	725	+ return ret;
	726	+
	727	+ ret = tcp_lookup(ctx, hdr, xdp);
	728	+ if (ret != XDP_TX)
	729	+ return ret;
	730	+
	731	+ /* Packet is TCP and doesn't belong to an established connection. */
	732	+
	733	+ if ((hdr->tcp->syn ^ hdr->tcp->ack) != 1)
	734	+ return XDP_DROP;
	735	+
	736	+ /* Grow the TCP header to TCP_MAXLEN to be able to pass any hdr->tcp_len
	737	+ * to bpf_tcp_raw_gen_syncookie_ipv{4,6} and pass the verifier.
	738	+ */
	739	+ if (xdp) {
	740	+ if (bpf_xdp_adjust_tail(ctx, TCP_MAXLEN - hdr->tcp_len))
	741	+ return XDP_ABORTED;
	742	+ } else {
	743	+ /* Without volatile the verifier throws this error:
	744	+ * R9 32-bit pointer arithmetic prohibited
	745	+ */
	746	+ volatile u64 old_len = data_end - data;
	747	+
	748	+ if (bpf_skb_change_tail(ctx, old_len + TCP_MAXLEN - hdr->tcp_len, 0))
	749	+ return XDP_ABORTED;
	750	+ }
	751	+
	752	+ return XDP_TX;
	753	+}
	754	+
	755	+static __always_inline int syncookie_part2(void ctx, void data, void *data_end,
	756	+ struct header_pointers *hdr, bool xdp)
	757	+{
	758	+ if (hdr->ipv4) {
	759	+ hdr->eth = data;
	760	+ hdr->ipv4 = (void )hdr->eth + sizeof(hdr->eth);
	761	+ /* IPV4_MAXLEN is needed when calculating checksum.
	762	+ * At least sizeof(struct iphdr) is needed here to access ihl.
	763	+ */
	764	+ if ((void *)hdr->ipv4 + IPV4_MAXLEN > data_end)
	765	+ return XDP_ABORTED;
	766	+ hdr->tcp = (void )hdr->ipv4 + hdr->ipv4->ihl 4;
	767	+ } else if (hdr->ipv6) {
	768	+ hdr->eth = data;
	769	+ hdr->ipv6 = (void )hdr->eth + sizeof(hdr->eth);
	770	+ hdr->tcp = (void )hdr->ipv6 + sizeof(hdr->ipv6);
	771	+ } else {
	772	+ return XDP_ABORTED;
	773	+ }
	774	+
	775	+ if ((void *)hdr->tcp + TCP_MAXLEN > data_end)
	776	+ return XDP_ABORTED;
	777	+
	778	+ /* We run out of registers, tcp_len gets spilled to the stack, and the
	779	+ * verifier forgets its min and max values checked above in tcp_dissect.
	780	+ */
	781	+ hdr->tcp_len = hdr->tcp->doff * 4;
	782	+ if (hdr->tcp_len < sizeof(*hdr->tcp))
	783	+ return XDP_ABORTED;
	784	+
	785	+ return hdr->tcp->syn ? syncookie_handle_syn(hdr, ctx, data, data_end, xdp) :
	786	+ syncookie_handle_ack(hdr);
	787	+}
	788	+
	789	+SEC("xdp")
	790	+int syncookie_xdp(struct xdp_md *ctx)
	791	+{
	792	+ void data_end = (void )(long)ctx->data_end;
	793	+ void data = (void )(long)ctx->data;
	794	+ struct header_pointers hdr;
	795	+ int ret;
	796	+
	797	+ ret = syncookie_part1(ctx, data, data_end, &hdr, true);
	798	+ if (ret != XDP_TX)
	799	+ return ret;
	800	+
	801	+ data_end = (void *)(long)ctx->data_end;
	802	+ data = (void *)(long)ctx->data;
	803	+
	804	+ return syncookie_part2(ctx, data, data_end, &hdr, true);
	805	+}
	806	+
	807	+SEC("tc")
	808	+int syncookie_tc(struct __sk_buff *skb)
	809	+{
	810	+ void data_end = (void )(long)skb->data_end;
	811	+ void data = (void )(long)skb->data;
	812	+ struct header_pointers hdr;
	813	+ int ret;
	814	+
	815	+ ret = syncookie_part1(skb, data, data_end, &hdr, false);
	816	+ if (ret != XDP_TX)
	817	+ return ret == XDP_PASS ? TC_ACT_OK : TC_ACT_SHOT;
	818	+
	819	+ data_end = (void *)(long)skb->data_end;
	820	+ data = (void *)(long)skb->data;
	821	+
	822	+ ret = syncookie_part2(skb, data, data_end, &hdr, false);
	823	+ switch (ret) {
	824	+ case XDP_PASS:
	825	+ return TC_ACT_OK;
	826	+ case XDP_TX:
	827	+ return bpf_redirect(skb->ifindex, 0);
	828	+ default:
	829	+ return TC_ACT_SHOT;
	830	+ }
	831	+}
	832	+
	833	+char _license[] SEC("license") = "GPL";

tools/testing/selftests/bpf/test_bpftool_synctypes.py+70−96

@@ -58,7 +58,7 @@ class BlockParser(object): · 57 unmodified lines
58	58
59	59	class ArrayParser(BlockParser):
60	60	"""
61		− A parser for extracting dicionaries of values from some BPF-related arrays.
	61	+ A parser for extracting a set of values from some BPF-related arrays.
62	62	@reader: a pointer to the open file to parse
63	63	@array_name: name of the array to parse
64	64	"""
@@ -66,7 +66,7 @@ class ArrayParser(BlockParser): · 1 unmodified line
66	66
67	67	def __init__(self, reader, array_name):
68	68	self.array_name = array_name
69		− self.start_marker = re.compile(f'(static )?const char \* const {self.array_name}\[.*\] = {{\n')
	69	+ self.start_marker = re.compile(f'(static )?const bool {self.array_name}\[.*\] = {{\n')
70	70	super().__init__(reader)
71	71
72	72	def search_block(self):
@@ -80,15 +80,15 @@ class ArrayParser(BlockParser): · 7 unmodified lines
80	80	Parse a block and return data as a dictionary. Items to extract must be
81	81	on separate lines in the file.
82	82	"""
83		− pattern = re.compile('\[(BPF_\w)\]\s= "(.*)",?$')
84		− entries = {}
	83	+ pattern = re.compile('\[(BPF_\w)\]\s= (true\|false),?$')
	84	+ entries = set()
85	85	while True:
86	86	line = self.reader.readline()
87	87	if line == '' or re.match(self.end_marker, line):
88	88	break
89	89	capture = pattern.search(line)
90	90	if capture:
91		− entries[capture.group(1)] = capture.group(2)
	91	+ entries \|= {capture.group(1)}
92	92	return entries
93	93
94	94	class InlineListParser(BlockParser):
@@ -115,7 +115,7 @@ class InlineListParser(BlockParser): · 20 unmodified lines
115	115	class FileExtractor(object):
116	116	"""
117	117	A generic reader for extracting data from a given file. This class contains
118		− several helper methods that wrap arround parser objects to extract values
	118	+ several helper methods that wrap around parser objects to extract values
119	119	from different structures.
120	120	This class does not offer a way to set a filename, which is expected to be
121	121	defined in children classes.
@@ -139,21 +139,19 @@ class FileExtractor(object): · 17 unmodified lines
139	139
140	140	def get_types_from_array(self, array_name):
141	141	"""
142		− Search for and parse an array associating names to BPF_* enum members,
143		− for example:
	142	+ Search for and parse a list of allowed BPF_* enum members, for example:
144	143
145		− const char * const prog_type_name[] = {
146		− [BPF_PROG_TYPE_UNSPEC] = "unspec",
147		− [BPF_PROG_TYPE_SOCKET_FILTER] = "socket_filter",
148		− [BPF_PROG_TYPE_KPROBE] = "kprobe",
	144	+ const bool prog_type_name[] = {
	145	+ [BPF_PROG_TYPE_UNSPEC] = true,
	146	+ [BPF_PROG_TYPE_SOCKET_FILTER] = true,
	147	+ [BPF_PROG_TYPE_KPROBE] = true,
149	148	};
150	149
151		− Return a dictionary with the enum member names as keys and the
152		− associated names as values, for example:
	150	+ Return a set of the enum members, for example:
153	151
154		− {'BPF_PROG_TYPE_UNSPEC': 'unspec',
155		− 'BPF_PROG_TYPE_SOCKET_FILTER': 'socket_filter',
156		− 'BPF_PROG_TYPE_KPROBE': 'kprobe'}
	152	+ {'BPF_PROG_TYPE_UNSPEC',
	153	+ 'BPF_PROG_TYPE_SOCKET_FILTER',
	154	+ 'BPF_PROG_TYPE_KPROBE'}
157	155
158	156	@array_name: name of the array to parse
159	157	"""
@@ -186,6 +184,27 @@ class FileExtractor(object): · 26 unmodified lines
186	184	parser.search_block(start_marker)
187	185	return parser.parse(pattern, end_marker)
188	186
	187	+ def make_enum_map(self, names, enum_prefix):
	188	+ """
	189	+ Search for and parse an enum containing BPF_* members, just as get_enum
	190	+ does. However, instead of just returning a set of the variant names,
	191	+ also generate a textual representation from them by (assuming and)
	192	+ removing a provided prefix and lowercasing the remainder. Then return a
	193	+ dict mapping from name to textual representation.
	194	+
	195	+ @enum_values: a set of enum values; e.g., as retrieved by get_enum
	196	+ @enum_prefix: the prefix to remove from each of the variants to infer
	197	+ textual representation
	198	+ """
	199	+ mapping = {}
	200	+ for name in names:
	201	+ if not name.startswith(enum_prefix):
	202	+ raise Exception(f"enum variant {name} does not start with {enum_prefix}")
	203	+ text = name[len(enum_prefix):].lower()
	204	+ mapping[name] = text
	205	+
	206	+ return mapping
	207	+
189	208	def __get_description_list(self, start_marker, pattern, end_marker):
190	209	parser = InlineListParser(self.reader)
191	210	parser.search_block(start_marker)
@@ -333,11 +352,9 @@ class ProgFileExtractor(SourceFileExtractor): · 141 unmodified lines
333	352	"""
334	353	filename = os.path.join(BPFTOOL_DIR, 'prog.c')
335	354
336		− def get_prog_types(self):
337		− return self.get_types_from_array('prog_type_name')
338		−
339	355	def get_attach_types(self):
340		− return self.get_types_from_array('attach_type_strings')
	356	+ types = self.get_types_from_array('attach_types')
	357	+ return self.make_enum_map(types, 'BPF_')
341	358
342	359	def get_prog_attach_help(self):
343	360	return self.get_help_list('ATTACH_TYPE')
@@ -348,9 +365,6 @@ class MapFileExtractor(SourceFileExtractor): · 4 unmodified lines
348	365	"""
349	366	filename = os.path.join(BPFTOOL_DIR, 'map.c')
350	367
351		− def get_map_types(self):
352		− return self.get_types_from_array('map_type_name')
353		−
354	368	def get_map_help(self):
355	369	return self.get_help_list('TYPE')
356	370
@@ -363,30 +377,6 @@ class CgroupFileExtractor(SourceFileExtractor): · 6 unmodified lines
363	377	def get_prog_attach_help(self):
364	378	return self.get_help_list('ATTACH_TYPE')
365	379
366		−class CommonFileExtractor(SourceFileExtractor):
367		− """
368		− An extractor for bpftool's common.c.
369		− """
370		− filename = os.path.join(BPFTOOL_DIR, 'common.c')
371		−
372		− def __init__(self):
373		− super().__init__()
374		− self.attach_types = {}
375		−
376		− def get_attach_types(self):
377		− if not self.attach_types:
378		− self.attach_types = self.get_types_from_array('attach_type_name')
379		− return self.attach_types
380		−
381		− def get_cgroup_attach_types(self):
382		− if not self.attach_types:
383		− self.get_attach_types()
384		− cgroup_types = {}
385		− for (key, value) in self.attach_types.items():
386		− if key.find('BPF_CGROUP') != -1:
387		− cgroup_types[key] = value
388		− return cgroup_types
389		−
390	380	class GenericSourceExtractor(SourceFileExtractor):
391	381	"""
392	382	An extractor for generic source code files.
@@ -403,14 +393,28 @@ class BpfHeaderExtractor(FileExtractor): · 10 unmodified lines
403	393	"""
404	394	filename = os.path.join(INCLUDE_DIR, 'uapi/linux/bpf.h')
405	395
	396	+ def __init__(self):
	397	+ super().__init__()
	398	+ self.attach_types = {}
	399	+
406	400	def get_prog_types(self):
407	401	return self.get_enum('bpf_prog_type')
408	402
409		− def get_map_types(self):
410		− return self.get_enum('bpf_map_type')
	403	+ def get_map_type_map(self):
	404	+ names = self.get_enum('bpf_map_type')
	405	+ return self.make_enum_map(names, 'BPF_MAP_TYPE_')
411	406
412		− def get_attach_types(self):
413		− return self.get_enum('bpf_attach_type')
	407	+ def get_attach_type_map(self):
	408	+ if not self.attach_types:
	409	+ names = self.get_enum('bpf_attach_type')
	410	+ self.attach_types = self.make_enum_map(names, 'BPF_')
	411	+ return self.attach_types
	412	+
	413	+ def get_cgroup_attach_type_map(self):
	414	+ if not self.attach_types:
	415	+ self.get_attach_type_map()
	416	+ return {name: text for name, text in self.attach_types.items()
	417	+ if name.startswith('BPF_CGROUP')}
414	418
415	419	class ManPageExtractor(FileExtractor):
416	420	"""
@@ -495,21 +499,12 @@ def main(): · 78 unmodified lines
495	499	""")
496	500	args = argParser.parse_args()
497	501
498		− # Map types (enum)
499		−
500	502	bpf_info = BpfHeaderExtractor()
501		− ref = bpf_info.get_map_types()
502		−
503		− map_info = MapFileExtractor()
504		− source_map_items = map_info.get_map_types()
505		− map_types_enum = set(source_map_items.keys())
506		−
507		− verify(ref, map_types_enum,
508		− f'Comparing BPF header (enum bpf_map_type) and {MapFileExtractor.filename} (map_type_name):')
509	503
510	504	# Map types (names)
511	505
512		− source_map_types = set(source_map_items.values())
	506	+ map_info = MapFileExtractor()
	507	+ source_map_types = set(bpf_info.get_map_type_map().values())
513	508	source_map_types.discard('unspec')
514	509
515	510	help_map_types = map_info.get_map_help()
@@ -525,37 +520,17 @@ def main(): · 9 unmodified lines
525	520	bashcomp_map_types = bashcomp_info.get_map_types()
526	521
527	522	verify(source_map_types, help_map_types,
528		− f'Comparing {MapFileExtractor.filename} (map_type_name) and {MapFileExtractor.filename} (do_help() TYPE):')
	523	+ f'Comparing {BpfHeaderExtractor.filename} (bpf_map_type) and {MapFileExtractor.filename} (do_help() TYPE):')
529	524	verify(source_map_types, man_map_types,
530		− f'Comparing {MapFileExtractor.filename} (map_type_name) and {ManMapExtractor.filename} (TYPE):')
	525	+ f'Comparing {BpfHeaderExtractor.filename} (bpf_map_type) and {ManMapExtractor.filename} (TYPE):')
531	526	verify(help_map_options, man_map_options,
532	527	f'Comparing {MapFileExtractor.filename} (do_help() OPTIONS) and {ManMapExtractor.filename} (OPTIONS):')
533	528	verify(source_map_types, bashcomp_map_types,
534		− f'Comparing {MapFileExtractor.filename} (map_type_name) and {BashcompExtractor.filename} (BPFTOOL_MAP_CREATE_TYPES):')
535		−
536		− # Program types (enum)
537		−
538		− ref = bpf_info.get_prog_types()
539		−
540		− prog_info = ProgFileExtractor()
541		− prog_types = set(prog_info.get_prog_types().keys())
542		−
543		− verify(ref, prog_types,
544		− f'Comparing BPF header (enum bpf_prog_type) and {ProgFileExtractor.filename} (prog_type_name):')
545		−
546		− # Attach types (enum)
547		−
548		− ref = bpf_info.get_attach_types()
549		− bpf_info.close()
550		−
551		− common_info = CommonFileExtractor()
552		− attach_types = common_info.get_attach_types()
553		−
554		− verify(ref, attach_types,
555		− f'Comparing BPF header (enum bpf_attach_type) and {CommonFileExtractor.filename} (attach_type_name):')
	529	+ f'Comparing {BpfHeaderExtractor.filename} (bpf_map_type) and {BashcompExtractor.filename} (BPFTOOL_MAP_CREATE_TYPES):')
556	530
557	531	# Attach types (names)
558	532
	533	+ prog_info = ProgFileExtractor()
559	534	source_prog_attach_types = set(prog_info.get_attach_types().values())
560	535
561	536	help_prog_attach_types = prog_info.get_prog_attach_help()
@@ -571,18 +546,17 @@ def main(): · 9 unmodified lines
571	546	bashcomp_prog_attach_types = bashcomp_info.get_prog_attach_types()
572	547
573	548	verify(source_prog_attach_types, help_prog_attach_types,
574		− f'Comparing {ProgFileExtractor.filename} (attach_type_strings) and {ProgFileExtractor.filename} (do_help() ATTACH_TYPE):')
	549	+ f'Comparing {ProgFileExtractor.filename} (bpf_attach_type) and {ProgFileExtractor.filename} (do_help() ATTACH_TYPE):')
575	550	verify(source_prog_attach_types, man_prog_attach_types,
576		− f'Comparing {ProgFileExtractor.filename} (attach_type_strings) and {ManProgExtractor.filename} (ATTACH_TYPE):')
	551	+ f'Comparing {ProgFileExtractor.filename} (bpf_attach_type) and {ManProgExtractor.filename} (ATTACH_TYPE):')
577	552	verify(help_prog_options, man_prog_options,
578	553	f'Comparing {ProgFileExtractor.filename} (do_help() OPTIONS) and {ManProgExtractor.filename} (OPTIONS):')
579	554	verify(source_prog_attach_types, bashcomp_prog_attach_types,
580		− f'Comparing {ProgFileExtractor.filename} (attach_type_strings) and {BashcompExtractor.filename} (BPFTOOL_PROG_ATTACH_TYPES):')
	555	+ f'Comparing {ProgFileExtractor.filename} (bpf_attach_type) and {BashcompExtractor.filename} (BPFTOOL_PROG_ATTACH_TYPES):')
581	556
582	557	# Cgroup attach types
583		−
584		− source_cgroup_attach_types = set(common_info.get_cgroup_attach_types().values())
585		− common_info.close()
	558	+ source_cgroup_attach_types = set(bpf_info.get_cgroup_attach_type_map().values())
	559	+ bpf_info.close()
586	560
587	561	cgroup_info = CgroupFileExtractor()
588	562	help_cgroup_attach_types = cgroup_info.get_prog_attach_help()
@@ -598,13 +572,13 @@ def main(): · 9 unmodified lines
598	572	bashcomp_info.close()
599	573
600	574	verify(source_cgroup_attach_types, help_cgroup_attach_types,
601		− f'Comparing {CommonFileExtractor.filename} (attach_type_strings) and {CgroupFileExtractor.filename} (do_help() ATTACH_TYPE):')
	575	+ f'Comparing {BpfHeaderExtractor.filename} (bpf_attach_type) and {CgroupFileExtractor.filename} (do_help() ATTACH_TYPE):')
602	576	verify(source_cgroup_attach_types, man_cgroup_attach_types,
603		− f'Comparing {CommonFileExtractor.filename} (attach_type_strings) and {ManCgroupExtractor.filename} (ATTACH_TYPE):')
	577	+ f'Comparing {BpfHeaderExtractor.filename} (bpf_attach_type) and {ManCgroupExtractor.filename} (ATTACH_TYPE):')
604	578	verify(help_cgroup_options, man_cgroup_options,
605	579	f'Comparing {CgroupFileExtractor.filename} (do_help() OPTIONS) and {ManCgroupExtractor.filename} (OPTIONS):')
606	580	verify(source_cgroup_attach_types, bashcomp_cgroup_attach_types,
607		− f'Comparing {CommonFileExtractor.filename} (attach_type_strings) and {BashcompExtractor.filename} (BPFTOOL_CGROUP_ATTACH_TYPES):')
	581	+ f'Comparing {BpfHeaderExtractor.filename} (bpf_attach_type) and {BashcompExtractor.filename} (BPFTOOL_CGROUP_ATTACH_TYPES):')
608	582
609	583	# Options for remaining commands
610	584

tools/testing/selftests/bpf/test_btf.h+1−0

@@ -39,6 +39,7 @@ · 38 unmodified lines
39	39	#define BTF_MEMBER_ENC(name, type, bits_offset) \
40	40	(name), (type), (bits_offset)
41	41	#define BTF_ENUM_ENC(name, val) (name), (val)
	42	+#define BTF_ENUM64_ENC(name, val_lo32, val_hi32) (name), (val_lo32), (val_hi32)
42	43	#define BTF_MEMBER_OFFSET(bitfield_size, bits_offset) \
43	44	((bitfield_size) << 24 \| (bits_offset))
44	45

tools/testing/selftests/bpf/test_xdping.sh+4−0

@@ -95,5 +95,9 @@ for server_args in "" "-I veth0 -s -S" ; do · 94 unmodified lines
95	95	test "$client_args" "$server_args"
96	96	done
97	97
	98	+# Test drv mode
	99	+test "-I veth1 -N" "-I veth0 -s -N"
	100	+test "-I veth1 -N -c 10" "-I veth0 -s -N"
	101	+
98	102	echo "OK. All tests passed"
99	103	exit 0

tools/testing/selftests/bpf/xdp_synproxy.c+466−0

@@ -0,0 +1,466 @@
	1	+// SPDX-License-Identifier: LGPL-2.1 OR BSD-2-Clause
	2	+/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
	3	+
	4	+#include <stdnoreturn.h>
	5	+#include <stdlib.h>
	6	+#include <stdio.h>
	7	+#include <string.h>
	8	+#include <errno.h>
	9	+#include <unistd.h>
	10	+#include <getopt.h>
	11	+#include <signal.h>
	12	+#include <sys/types.h>
	13	+#include <bpf/bpf.h>
	14	+#include <bpf/libbpf.h>
	15	+#include <net/if.h>
	16	+#include <linux/if_link.h>
	17	+#include <linux/limits.h>
	18	+
	19	+static unsigned int ifindex;
	20	+static __u32 attached_prog_id;
	21	+static bool attached_tc;
	22	+
	23	+static void noreturn cleanup(int sig)
	24	+{
	25	+ LIBBPF_OPTS(bpf_xdp_attach_opts, opts);
	26	+ int prog_fd;
	27	+ int err;
	28	+
	29	+ if (attached_prog_id == 0)
	30	+ exit(0);
	31	+
	32	+ if (attached_tc) {
	33	+ LIBBPF_OPTS(bpf_tc_hook, hook,
	34	+ .ifindex = ifindex,
	35	+ .attach_point = BPF_TC_INGRESS);
	36	+
	37	+ err = bpf_tc_hook_destroy(&hook);
	38	+ if (err < 0) {
	39	+ fprintf(stderr, "Error: bpf_tc_hook_destroy: %s\n", strerror(-err));
	40	+ fprintf(stderr, "Failed to destroy the TC hook\n");
	41	+ exit(1);
	42	+ }
	43	+ exit(0);
	44	+ }
	45	+
	46	+ prog_fd = bpf_prog_get_fd_by_id(attached_prog_id);
	47	+ if (prog_fd < 0) {
	48	+ fprintf(stderr, "Error: bpf_prog_get_fd_by_id: %s\n", strerror(-prog_fd));
	49	+ err = bpf_xdp_attach(ifindex, -1, 0, NULL);
	50	+ if (err < 0) {
	51	+ fprintf(stderr, "Error: bpf_set_link_xdp_fd: %s\n", strerror(-err));
	52	+ fprintf(stderr, "Failed to detach XDP program\n");
	53	+ exit(1);
	54	+ }
	55	+ } else {
	56	+ opts.old_prog_fd = prog_fd;
	57	+ err = bpf_xdp_attach(ifindex, -1, XDP_FLAGS_REPLACE, &opts);
	58	+ close(prog_fd);
	59	+ if (err < 0) {
	60	+ fprintf(stderr, "Error: bpf_set_link_xdp_fd_opts: %s\n", strerror(-err));
	61	+ /* Not an error if already replaced by someone else. */
	62	+ if (err != -EEXIST) {
	63	+ fprintf(stderr, "Failed to detach XDP program\n");
	64	+ exit(1);
	65	+ }
	66	+ }
	67	+ }
	68	+ exit(0);
	69	+}
	70	+
	71	+static noreturn void usage(const char *progname)
	72	+{
	73	+ fprintf(stderr, "Usage: %s [--iface <iface>\|--prog <prog_id>] [--mss4 <mss ipv4> --mss6 <mss ipv6> --wscale <wscale> --ttl <ttl>] [--ports <port1>,<port2>,...] [--single] [--tc]\n",
	74	+ progname);
	75	+ exit(1);
	76	+}
	77	+
	78	+static unsigned long parse_arg_ul(const char progname, const char arg, unsigned long limit)
	79	+{
	80	+ unsigned long res;
	81	+ char *endptr;
	82	+
	83	+ errno = 0;
	84	+ res = strtoul(arg, &endptr, 10);
	85	+ if (errno != 0 \|\| *endptr != '\0' \|\| arg[0] == '\0' \|\| res > limit)
	86	+ usage(progname);
	87	+
	88	+ return res;
	89	+}
	90	+
	91	+static void parse_options(int argc, char argv[], unsigned int ifindex, __u32 *prog_id,
	92	+ __u64 tcpipopts, char ports, bool single, bool *tc)
	93	+{
	94	+ static struct option long_options[] = {
	95	+ { "help", no_argument, NULL, 'h' },
	96	+ { "iface", required_argument, NULL, 'i' },
	97	+ { "prog", required_argument, NULL, 'x' },
	98	+ { "mss4", required_argument, NULL, 4 },
	99	+ { "mss6", required_argument, NULL, 6 },
	100	+ { "wscale", required_argument, NULL, 'w' },
	101	+ { "ttl", required_argument, NULL, 't' },
	102	+ { "ports", required_argument, NULL, 'p' },
	103	+ { "single", no_argument, NULL, 's' },
	104	+ { "tc", no_argument, NULL, 'c' },
	105	+ { NULL, 0, NULL, 0 },
	106	+ };
	107	+ unsigned long mss4, mss6, wscale, ttl;
	108	+ unsigned int tcpipopts_mask = 0;
	109	+
	110	+ if (argc < 2)
	111	+ usage(argv[0]);
	112	+
	113	+ *ifindex = 0;
	114	+ *prog_id = 0;
	115	+ *tcpipopts = 0;
	116	+ *ports = NULL;
	117	+ *single = false;
	118	+
	119	+ while (true) {
	120	+ int opt;
	121	+
	122	+ opt = getopt_long(argc, argv, "", long_options, NULL);
	123	+ if (opt == -1)
	124	+ break;
	125	+
	126	+ switch (opt) {
	127	+ case 'h':
	128	+ usage(argv[0]);
	129	+ break;
	130	+ case 'i':
	131	+ *ifindex = if_nametoindex(optarg);
	132	+ if (*ifindex == 0)
	133	+ usage(argv[0]);
	134	+ break;
	135	+ case 'x':
	136	+ *prog_id = parse_arg_ul(argv[0], optarg, UINT32_MAX);
	137	+ if (*prog_id == 0)
	138	+ usage(argv[0]);
	139	+ break;
	140	+ case 4:
	141	+ mss4 = parse_arg_ul(argv[0], optarg, UINT16_MAX);
	142	+ tcpipopts_mask \|= 1 << 0;
	143	+ break;
	144	+ case 6:
	145	+ mss6 = parse_arg_ul(argv[0], optarg, UINT16_MAX);
	146	+ tcpipopts_mask \|= 1 << 1;
	147	+ break;
	148	+ case 'w':
	149	+ wscale = parse_arg_ul(argv[0], optarg, 14);
	150	+ tcpipopts_mask \|= 1 << 2;
	151	+ break;
	152	+ case 't':
	153	+ ttl = parse_arg_ul(argv[0], optarg, UINT8_MAX);
	154	+ tcpipopts_mask \|= 1 << 3;
	155	+ break;
	156	+ case 'p':
	157	+ *ports = optarg;
	158	+ break;
	159	+ case 's':
	160	+ *single = true;
	161	+ break;
	162	+ case 'c':
	163	+ *tc = true;
	164	+ break;
	165	+ default:
	166	+ usage(argv[0]);
	167	+ }
	168	+ }
	169	+ if (optind < argc)
	170	+ usage(argv[0]);
	171	+
	172	+ if (tcpipopts_mask == 0xf) {
	173	+ if (mss4 == 0 \|\| mss6 == 0 \|\| wscale == 0 \|\| ttl == 0)
	174	+ usage(argv[0]);
	175	+ *tcpipopts = (mss6 << 32) \| (ttl << 24) \| (wscale << 16) \| mss4;
	176	+ } else if (tcpipopts_mask != 0) {
	177	+ usage(argv[0]);
	178	+ }
	179	+
	180	+ if (ifindex != 0 && prog_id != 0)
	181	+ usage(argv[0]);
	182	+ if (ifindex == 0 && prog_id == 0)
	183	+ usage(argv[0]);
	184	+}
	185	+
	186	+static int syncookie_attach(const char *argv0, unsigned int ifindex, bool tc)
	187	+{
	188	+ struct bpf_prog_info info = {};
	189	+ __u32 info_len = sizeof(info);
	190	+ char xdp_filename[PATH_MAX];
	191	+ struct bpf_program *prog;
	192	+ struct bpf_object *obj;
	193	+ int prog_fd;
	194	+ int err;
	195	+
	196	+ snprintf(xdp_filename, sizeof(xdp_filename), "%s_kern.o", argv0);
	197	+ obj = bpf_object__open_file(xdp_filename, NULL);
	198	+ err = libbpf_get_error(obj);
	199	+ if (err < 0) {
	200	+ fprintf(stderr, "Error: bpf_object__open_file: %s\n", strerror(-err));
	201	+ return err;
	202	+ }
	203	+
	204	+ err = bpf_object__load(obj);
	205	+ if (err < 0) {
	206	+ fprintf(stderr, "Error: bpf_object__open_file: %s\n", strerror(-err));
	207	+ return err;
	208	+ }
	209	+
	210	+ prog = bpf_object__find_program_by_name(obj, tc ? "syncookie_tc" : "syncookie_xdp");
	211	+ if (!prog) {
	212	+ fprintf(stderr, "Error: bpf_object__find_program_by_name: program was not found\n");
	213	+ return -ENOENT;
	214	+ }
	215	+
	216	+ prog_fd = bpf_program__fd(prog);
	217	+
	218	+ err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len);
	219	+ if (err < 0) {
	220	+ fprintf(stderr, "Error: bpf_obj_get_info_by_fd: %s\n", strerror(-err));
	221	+ goto out;
	222	+ }
	223	+ attached_tc = tc;
	224	+ attached_prog_id = info.id;
	225	+ signal(SIGINT, cleanup);
	226	+ signal(SIGTERM, cleanup);
	227	+ if (tc) {
	228	+ LIBBPF_OPTS(bpf_tc_hook, hook,
	229	+ .ifindex = ifindex,
	230	+ .attach_point = BPF_TC_INGRESS);
	231	+ LIBBPF_OPTS(bpf_tc_opts, opts,
	232	+ .handle = 1,
	233	+ .priority = 1,
	234	+ .prog_fd = prog_fd);
	235	+
	236	+ err = bpf_tc_hook_create(&hook);
	237	+ if (err < 0) {
	238	+ fprintf(stderr, "Error: bpf_tc_hook_create: %s\n",
	239	+ strerror(-err));
	240	+ goto fail;
	241	+ }
	242	+ err = bpf_tc_attach(&hook, &opts);
	243	+ if (err < 0) {
	244	+ fprintf(stderr, "Error: bpf_tc_attach: %s\n",
	245	+ strerror(-err));
	246	+ goto fail;
	247	+ }
	248	+
	249	+ } else {
	250	+ err = bpf_xdp_attach(ifindex, prog_fd,
	251	+ XDP_FLAGS_UPDATE_IF_NOEXIST, NULL);
	252	+ if (err < 0) {
	253	+ fprintf(stderr, "Error: bpf_set_link_xdp_fd: %s\n",
	254	+ strerror(-err));
	255	+ goto fail;
	256	+ }
	257	+ }
	258	+ err = 0;
	259	+out:
	260	+ bpf_object__close(obj);
	261	+ return err;
	262	+fail:
	263	+ signal(SIGINT, SIG_DFL);
	264	+ signal(SIGTERM, SIG_DFL);
	265	+ attached_prog_id = 0;
	266	+ goto out;
	267	+}
	268	+
	269	+static int syncookie_open_bpf_maps(__u32 prog_id, int values_map_fd, int ports_map_fd)
	270	+{
	271	+ struct bpf_prog_info prog_info;
	272	+ __u32 map_ids[8];
	273	+ __u32 info_len;
	274	+ int prog_fd;
	275	+ int err;
	276	+ int i;
	277	+
	278	+ *values_map_fd = -1;
	279	+ *ports_map_fd = -1;
	280	+
	281	+ prog_fd = bpf_prog_get_fd_by_id(prog_id);
	282	+ if (prog_fd < 0) {
	283	+ fprintf(stderr, "Error: bpf_prog_get_fd_by_id: %s\n", strerror(-prog_fd));
	284	+ return prog_fd;
	285	+ }
	286	+
	287	+ prog_info = (struct bpf_prog_info) {
	288	+ .nr_map_ids = 8,
	289	+ .map_ids = (__u64)map_ids,
	290	+ };
	291	+ info_len = sizeof(prog_info);
	292	+
	293	+ err = bpf_obj_get_info_by_fd(prog_fd, &prog_info, &info_len);
	294	+ if (err != 0) {
	295	+ fprintf(stderr, "Error: bpf_obj_get_info_by_fd: %s\n", strerror(-err));
	296	+ goto out;
	297	+ }
	298	+
	299	+ if (prog_info.nr_map_ids < 2) {
	300	+ fprintf(stderr, "Error: Found %u BPF maps, expected at least 2\n",
	301	+ prog_info.nr_map_ids);
	302	+ err = -ENOENT;
	303	+ goto out;
	304	+ }
	305	+
	306	+ for (i = 0; i < prog_info.nr_map_ids; i++) {
	307	+ struct bpf_map_info map_info = {};
	308	+ int map_fd;
	309	+
	310	+ err = bpf_map_get_fd_by_id(map_ids[i]);
	311	+ if (err < 0) {
	312	+ fprintf(stderr, "Error: bpf_map_get_fd_by_id: %s\n", strerror(-err));
	313	+ goto err_close_map_fds;
	314	+ }
	315	+ map_fd = err;
	316	+
	317	+ info_len = sizeof(map_info);
	318	+ err = bpf_obj_get_info_by_fd(map_fd, &map_info, &info_len);
	319	+ if (err != 0) {
	320	+ fprintf(stderr, "Error: bpf_obj_get_info_by_fd: %s\n", strerror(-err));
	321	+ close(map_fd);
	322	+ goto err_close_map_fds;
	323	+ }
	324	+ if (strcmp(map_info.name, "values") == 0) {
	325	+ *values_map_fd = map_fd;
	326	+ continue;
	327	+ }
	328	+ if (strcmp(map_info.name, "allowed_ports") == 0) {
	329	+ *ports_map_fd = map_fd;
	330	+ continue;
	331	+ }
	332	+ close(map_fd);
	333	+ }
	334	+
	335	+ if (values_map_fd != -1 && ports_map_fd != -1) {
	336	+ err = 0;
	337	+ goto out;
	338	+ }
	339	+
	340	+ err = -ENOENT;
	341	+
	342	+err_close_map_fds:
	343	+ if (*values_map_fd != -1)
	344	+ close(*values_map_fd);
	345	+ if (*ports_map_fd != -1)
	346	+ close(*ports_map_fd);
	347	+ *values_map_fd = -1;
	348	+ *ports_map_fd = -1;
	349	+
	350	+out:
	351	+ close(prog_fd);
	352	+ return err;
	353	+}
	354	+
	355	+int main(int argc, char *argv[])
	356	+{
	357	+ int values_map_fd, ports_map_fd;
	358	+ __u64 tcpipopts;
	359	+ bool firstiter;
	360	+ __u64 prevcnt;
	361	+ __u32 prog_id;
	362	+ char *ports;
	363	+ bool single;
	364	+ int err = 0;
	365	+ bool tc;
	366	+
	367	+ parse_options(argc, argv, &ifindex, &prog_id, &tcpipopts, &ports,
	368	+ &single, &tc);
	369	+
	370	+ if (prog_id == 0) {
	371	+ if (!tc) {
	372	+ err = bpf_xdp_query_id(ifindex, 0, &prog_id);
	373	+ if (err < 0) {
	374	+ fprintf(stderr, "Error: bpf_get_link_xdp_id: %s\n",
	375	+ strerror(-err));
	376	+ goto out;
	377	+ }
	378	+ }
	379	+ if (prog_id == 0) {
	380	+ err = syncookie_attach(argv[0], ifindex, tc);
	381	+ if (err < 0)
	382	+ goto out;
	383	+ prog_id = attached_prog_id;
	384	+ }
	385	+ }
	386	+
	387	+ err = syncookie_open_bpf_maps(prog_id, &values_map_fd, &ports_map_fd);
	388	+ if (err < 0)
	389	+ goto out;
	390	+
	391	+ if (ports) {
	392	+ __u16 port_last = 0;
	393	+ __u32 port_idx = 0;
	394	+ char *p = ports;
	395	+
	396	+ fprintf(stderr, "Replacing allowed ports\n");
	397	+
	398	+ while (p && *p != '\0') {
	399	+ char *token = strsep(&p, ",");
	400	+ __u16 port;
	401	+
	402	+ port = parse_arg_ul(argv[0], token, UINT16_MAX);
	403	+ err = bpf_map_update_elem(ports_map_fd, &port_idx, &port, BPF_ANY);
	404	+ if (err != 0) {
	405	+ fprintf(stderr, "Error: bpf_map_update_elem: %s\n", strerror(-err));
	406	+ fprintf(stderr, "Failed to add port %u (index %u)\n",
	407	+ port, port_idx);
	408	+ goto out_close_maps;
	409	+ }
	410	+ fprintf(stderr, "Added port %u\n", port);
	411	+ port_idx++;
	412	+ }
	413	+ err = bpf_map_update_elem(ports_map_fd, &port_idx, &port_last, BPF_ANY);
	414	+ if (err != 0) {
	415	+ fprintf(stderr, "Error: bpf_map_update_elem: %s\n", strerror(-err));
	416	+ fprintf(stderr, "Failed to add the terminator value 0 (index %u)\n",
	417	+ port_idx);
	418	+ goto out_close_maps;
	419	+ }
	420	+ }
	421	+
	422	+ if (tcpipopts) {
	423	+ __u32 key = 0;
	424	+
	425	+ fprintf(stderr, "Replacing TCP/IP options\n");
	426	+
	427	+ err = bpf_map_update_elem(values_map_fd, &key, &tcpipopts, BPF_ANY);
	428	+ if (err != 0) {
	429	+ fprintf(stderr, "Error: bpf_map_update_elem: %s\n", strerror(-err));
	430	+ goto out_close_maps;
	431	+ }
	432	+ }
	433	+
	434	+ if ((ports \|\| tcpipopts) && attached_prog_id == 0 && !single)
	435	+ goto out_close_maps;
	436	+
	437	+ prevcnt = 0;
	438	+ firstiter = true;
	439	+ while (true) {
	440	+ __u32 key = 1;
	441	+ __u64 value;
	442	+
	443	+ err = bpf_map_lookup_elem(values_map_fd, &key, &value);
	444	+ if (err != 0) {
	445	+ fprintf(stderr, "Error: bpf_map_lookup_elem: %s\n", strerror(-err));
	446	+ goto out_close_maps;
	447	+ }
	448	+ if (firstiter) {
	449	+ prevcnt = value;
	450	+ firstiter = false;
	451	+ }
	452	+ if (single) {
	453	+ printf("Total SYNACKs generated: %llu\n", value);
	454	+ break;
	455	+ }
	456	+ printf("SYNACKs generated: %llu (total %llu)\n", value - prevcnt, value);
	457	+ prevcnt = value;
	458	+ sleep(1);
	459	+ }
	460	+
	461	+out_close_maps:
	462	+ close(values_map_fd);
	463	+ close(ports_map_fd);
	464	+out:
	465	+ return err == 0 ? 0 : 1;
	466	+}

Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code