Merge branch 'bpf-ring-buffer'

Andrii Nakryiko says: ==================== Implement a new BPF ring buffer, as presented at BPF virtual conference ([0]). It presents an alternative to perf buffer, following its semantics closely, but allowing sharing same instance of ring buffer across multiple CPUs efficiently. Most patches have extensive commentary explaining various aspects, so I'll keep cover letter short. Overall structure of the patch set: - patch #1 adds BPF ring buffer implementation to kernel and necessary verifier support; - patch #2 adds libbpf consumer implementation for BPF ringbuf; - patch #3 adds selftest, both for single BPF ring buf use case, as well as using it with array/hash of maps; - patch #4 adds extensive benchmarks and provide some analysis in commit message, it builds upon selftests/bpf's bench runner. - patch #5 adds most of patch #1 commit message as a doc under Documentation/bpf/ringbuf.rst. Litmus tests, validating consumer/producer protocols and memory orderings, were moved out as discussed in [1] and are going to be posted against -rcu tree and put under Documentation/litmus-tests/bpf-rb. [0] https://docs.google.com/presentation/d/18ITdg77Bj6YDOH2LghxrnFxiPWe0fAqcmJY95t_qr0w [1] https://lkml.org/lkml/2020/5/22/1011 v3->v4: - fix ringbuf freeing (vunmap, __free_page); verified with a trivial loop creating and closing ringbuf map endlessly (Daniel); v2->v3: - dropped unnecessary smp_wmb() (Paul); - verifier reference type enhancement patch was dropped (Alexei); - better verifier message for various memory access checks (Alexei); - clarified a bit roundup_len() bit shifting (Alexei); - converted doc to .rst (Alexei); - fixed warning on 32-bit arches regarding tautological ring area size check. v1->v2: - commit()/discard()/output() accept flags (NO_WAKEUP/FORCE_WAKEUP) (Stanislav); - bpf_ringbuf_query() added, returning available data size, ringbuf size, consumer/producer positions, needed to implement smarter notification policy (Stanislav); - added ringbuf UAPI constants to include/uapi/linux/bpf.h (Jonathan); - fixed sample size check, added proper ringbuf size check (Jonathan, Alexei); - wake_up_all() is done through irq_work (Alexei); - consistent use of smp_load_acquire/smp_store_release, no READ_ONCE/WRITE_ONCE (Alexei); - added Documentation/bpf/ringbuf.txt (Stanislav); - updated litmus test with smp_load_acquire/smp_store_release changes; - added ring_buffer__consume() API to libbpf for busy-polling; - ring_buffer__poll() on success returns number of records consumed; - fixed EPOLL notifications, don't assume available data, done similarly to perfbuf's implementation; - both ringbuf and perfbuf now have --rb-sampled mode, instead of pb-raw/pb-custom mode, updated benchmark results; - extended ringbuf selftests to validate epoll logic/manual notification logic, as well as bpf_ringbuf_query(). ====================

Daniel Borkmann committed 6 years ago

Commit e255d322 parents 43dd115 97abb2b35 files · +2630 −72Browse files

Unified · Split

35 files

Documentation/bpf/ringbuf.rst +209 −0
include/linux/bpf.h +13 −0
include/linux/bpf_types.h +1 −0
include/linux/bpf_verifier.h +4 −0
include/uapi/linux/bpf.h +83 −1
kernel/bpf/Makefile +1 −1
kernel/bpf/helpers.c +10 −0
kernel/bpf/ringbuf.c +501 −0
kernel/bpf/syscall.c +12 −0
kernel/bpf/verifier.c +146 −49
kernel/trace/bpf_trace.c +10 −0
tools/include/uapi/linux/bpf.h +83 −1
tools/lib/bpf/Build +1 −1
tools/lib/bpf/libbpf.h +21 −0
tools/lib/bpf/libbpf.map +5 −0
tools/lib/bpf/libbpf_probes.c +5 −0
tools/lib/bpf/ringbuf.c +285 −0
tools/testing/selftests/bpf/Makefile +4 −1
tools/testing/selftests/bpf/bench.c +16 −0
tools/testing/selftests/bpf/benchs/bench_ringbufs.c +566 −0
tools/testing/selftests/bpf/benchs/run_bench_ringbufs.sh +75 −0
tools/testing/selftests/bpf/prog_tests/ringbuf.c +211 −0
tools/testing/selftests/bpf/prog_tests/ringbuf_multi.c +102 −0
tools/testing/selftests/bpf/progs/perfbuf_bench.c +33 −0
tools/testing/selftests/bpf/progs/ringbuf_bench.c +60 −0
tools/testing/selftests/bpf/progs/test_ringbuf.c +78 −0
tools/testing/selftests/bpf/progs/test_ringbuf_multi.c +77 −0
tools/testing/selftests/bpf/verifier/and.c +2 −2
tools/testing/selftests/bpf/verifier/array_access.c +2 −2
tools/testing/selftests/bpf/verifier/bounds.c +3 −3
tools/testing/selftests/bpf/verifier/calls.c +1 −1
tools/testing/selftests/bpf/verifier/direct_value_access.c +2 −2
tools/testing/selftests/bpf/verifier/helper_access_var_len.c +1 −1
tools/testing/selftests/bpf/verifier/helper_value_access.c +3 −3
tools/testing/selftests/bpf/verifier/value_ptr_arith.c +4 −4

Documentation/bpf/ringbuf.rst+209−0

@@ -0,0 +1,209 @@
	1	===============
	2	BPF ring buffer
	3	===============
	4
	5	This document describes BPF ring buffer design, API, and implementation details.
	6
	7	.. contents::
	8	:local:
	9	:depth: 2
	10
	11	Motivation
	12	----------
	13
	14	There are two distinctive motivators for this work, which are not satisfied by
	15	existing perf buffer, which prompted creation of a new ring buffer
	16	implementation.
	17
	18	- more efficient memory utilization by sharing ring buffer across CPUs;
	19	- preserving ordering of events that happen sequentially in time, even across
	20	multiple CPUs (e.g., fork/exec/exit events for a task).
	21
	22	These two problems are independent, but perf buffer fails to satisfy both.
	23	Both are a result of a choice to have per-CPU perf ring buffer. Both can be
	24	also solved by having an MPSC implementation of ring buffer. The ordering
	25	problem could technically be solved for perf buffer with some in-kernel
	26	counting, but given the first one requires an MPSC buffer, the same solution
	27	would solve the second problem automatically.
	28
	29	Semantics and APIs
	30	------------------
	31
	32	Single ring buffer is presented to BPF programs as an instance of BPF map of
	33	type ``BPF_MAP_TYPE_RINGBUF``. Two other alternatives considered, but
	34	ultimately rejected.
	35
	36	One way would be to, similar to ``BPF_MAP_TYPE_PERF_EVENT_ARRAY``, make
	37	``BPF_MAP_TYPE_RINGBUF`` could represent an array of ring buffers, but not
	38	enforce "same CPU only" rule. This would be more familiar interface compatible
	39	with existing perf buffer use in BPF, but would fail if application needed more
	40	advanced logic to lookup ring buffer by arbitrary key.
	41	``BPF_MAP_TYPE_HASH_OF_MAPS`` addresses this with current approach.
	42	Additionally, given the performance of BPF ringbuf, many use cases would just
	43	opt into a simple single ring buffer shared among all CPUs, for which current
	44	approach would be an overkill.
	45
	46	Another approach could introduce a new concept, alongside BPF map, to represent
	47	generic "container" object, which doesn't necessarily have key/value interface
	48	with lookup/update/delete operations. This approach would add a lot of extra
	49	infrastructure that has to be built for observability and verifier support. It
	50	would also add another concept that BPF developers would have to familiarize
	51	themselves with, new syntax in libbpf, etc. But then would really provide no
	52	additional benefits over the approach of using a map. ``BPF_MAP_TYPE_RINGBUF``
	53	doesn't support lookup/update/delete operations, but so doesn't few other map
	54	types (e.g., queue and stack; array doesn't support delete, etc).
	55
	56	The approach chosen has an advantage of re-using existing BPF map
	57	infrastructure (introspection APIs in kernel, libbpf support, etc), being
	58	familiar concept (no need to teach users a new type of object in BPF program),
	59	and utilizing existing tooling (bpftool). For common scenario of using a single
	60	ring buffer for all CPUs, it's as simple and straightforward, as would be with
	61	a dedicated "container" object. On the other hand, by being a map, it can be
	62	combined with ``ARRAY_OF_MAPS`` and ``HASH_OF_MAPS`` map-in-maps to implement
	63	a wide variety of topologies, from one ring buffer for each CPU (e.g., as
	64	a replacement for perf buffer use cases), to a complicated application
	65	hashing/sharding of ring buffers (e.g., having a small pool of ring buffers
	66	with hashed task's tgid being a look up key to preserve order, but reduce
	67	contention).
	68
	69	Key and value sizes are enforced to be zero. ``max_entries`` is used to specify
	70	the size of ring buffer and has to be a power of 2 value.
	71
	72	There are a bunch of similarities between perf buffer
	73	(``BPF_MAP_TYPE_PERF_EVENT_ARRAY``) and new BPF ring buffer semantics:
	74
	75	- variable-length records;
	76	- if there is no more space left in ring buffer, reservation fails, no
	77	blocking;
	78	- memory-mappable data area for user-space applications for ease of
	79	consumption and high performance;
	80	- epoll notifications for new incoming data;
	81	- but still the ability to do busy polling for new data to achieve the
	82	lowest latency, if necessary.
	83
	84	BPF ringbuf provides two sets of APIs to BPF programs:
	85
	86	- ``bpf_ringbuf_output()`` allows to copy data from one place to a ring
	87	buffer, similarly to ``bpf_perf_event_output()``;
	88	- ``bpf_ringbuf_reserve()``/``bpf_ringbuf_commit()``/``bpf_ringbuf_discard()``
	89	APIs split the whole process into two steps. First, a fixed amount of space
	90	is reserved. If successful, a pointer to a data inside ring buffer data
	91	area is returned, which BPF programs can use similarly to a data inside
	92	array/hash maps. Once ready, this piece of memory is either committed or
	93	discarded. Discard is similar to commit, but makes consumer ignore the
	94	record.
	95
	96	``bpf_ringbuf_output()`` has disadvantage of incurring extra memory copy,
	97	because record has to be prepared in some other place first. But it allows to
	98	submit records of the length that's not known to verifier beforehand. It also
	99	closely matches ``bpf_perf_event_output()``, so will simplify migration
	100	significantly.
	101
	102	``bpf_ringbuf_reserve()`` avoids the extra copy of memory by providing a memory
	103	pointer directly to ring buffer memory. In a lot of cases records are larger
	104	than BPF stack space allows, so many programs have use extra per-CPU array as
	105	a temporary heap for preparing sample. bpf_ringbuf_reserve() avoid this needs
	106	completely. But in exchange, it only allows a known constant size of memory to
	107	be reserved, such that verifier can verify that BPF program can't access memory
	108	outside its reserved record space. bpf_ringbuf_output(), while slightly slower
	109	due to extra memory copy, covers some use cases that are not suitable for
	110	``bpf_ringbuf_reserve()``.
	111
	112	The difference between commit and discard is very small. Discard just marks
	113	a record as discarded, and such records are supposed to be ignored by consumer
	114	code. Discard is useful for some advanced use-cases, such as ensuring
	115	all-or-nothing multi-record submission, or emulating temporary
	116	``malloc()``/``free()`` within single BPF program invocation.
	117
	118	Each reserved record is tracked by verifier through existing
	119	reference-tracking logic, similar to socket ref-tracking. It is thus
	120	impossible to reserve a record, but forget to submit (or discard) it.
	121
	122	``bpf_ringbuf_query()`` helper allows to query various properties of ring
	123	buffer. Currently 4 are supported:
	124
	125	- ``BPF_RB_AVAIL_DATA`` returns amount of unconsumed data in ring buffer;
	126	- ``BPF_RB_RING_SIZE`` returns the size of ring buffer;
	127	- ``BPF_RB_CONS_POS``/``BPF_RB_PROD_POS`` returns current logical possition
	128	of consumer/producer, respectively.
	129
	130	Returned values are momentarily snapshots of ring buffer state and could be
	131	off by the time helper returns, so this should be used only for
	132	debugging/reporting reasons or for implementing various heuristics, that take
	133	into account highly-changeable nature of some of those characteristics.
	134
	135	One such heuristic might involve more fine-grained control over poll/epoll
	136	notifications about new data availability in ring buffer. Together with
	137	``BPF_RB_NO_WAKEUP``/``BPF_RB_FORCE_WAKEUP`` flags for output/commit/discard
	138	helpers, it allows BPF program a high degree of control and, e.g., more
	139	efficient batched notifications. Default self-balancing strategy, though,
	140	should be adequate for most applications and will work reliable and efficiently
	141	already.
	142
	143	Design and Implementation
	144	-------------------------
	145
	146	This reserve/commit schema allows a natural way for multiple producers, either
	147	on different CPUs or even on the same CPU/in the same BPF program, to reserve
	148	independent records and work with them without blocking other producers. This
	149	means that if BPF program was interruped by another BPF program sharing the
	150	same ring buffer, they will both get a record reserved (provided there is
	151	enough space left) and can work with it and submit it independently. This
	152	applies to NMI context as well, except that due to using a spinlock during
	153	reservation, in NMI context, ``bpf_ringbuf_reserve()`` might fail to get
	154	a lock, in which case reservation will fail even if ring buffer is not full.
	155
	156	The ring buffer itself internally is implemented as a power-of-2 sized
	157	circular buffer, with two logical and ever-increasing counters (which might
	158	wrap around on 32-bit architectures, that's not a problem):
	159
	160	- consumer counter shows up to which logical position consumer consumed the
	161	data;
	162	- producer counter denotes amount of data reserved by all producers.
	163
	164	Each time a record is reserved, producer that "owns" the record will
	165	successfully advance producer counter. At that point, data is still not yet
	166	ready to be consumed, though. Each record has 8 byte header, which contains the
	167	length of reserved record, as well as two extra bits: busy bit to denote that
	168	record is still being worked on, and discard bit, which might be set at commit
	169	time if record is discarded. In the latter case, consumer is supposed to skip
	170	the record and move on to the next one. Record header also encodes record's
	171	relative offset from the beginning of ring buffer data area (in pages). This
	172	allows ``bpf_ringbuf_commit()``/``bpf_ringbuf_discard()`` to accept only the
	173	pointer to the record itself, without requiring also the pointer to ring buffer
	174	itself. Ring buffer memory location will be restored from record metadata
	175	header. This significantly simplifies verifier, as well as improving API
	176	usability.
	177
	178	Producer counter increments are serialized under spinlock, so there is
	179	a strict ordering between reservations. Commits, on the other hand, are
	180	completely lockless and independent. All records become available to consumer
	181	in the order of reservations, but only after all previous records where
	182	already committed. It is thus possible for slow producers to temporarily hold
	183	off submitted records, that were reserved later.
	184
	185	Reservation/commit/consumer protocol is verified by litmus tests in
	186	Documentation/litmus_tests/bpf-rb/_.
	187
	188	One interesting implementation bit, that significantly simplifies (and thus
	189	speeds up as well) implementation of both producers and consumers is how data
	190	area is mapped twice contiguously back-to-back in the virtual memory. This
	191	allows to not take any special measures for samples that have to wrap around
	192	at the end of the circular buffer data area, because the next page after the
	193	last data page would be first data page again, and thus the sample will still
	194	appear completely contiguous in virtual memory. See comment and a simple ASCII
	195	diagram showing this visually in ``bpf_ringbuf_area_alloc()``.
	196
	197	Another feature that distinguishes BPF ringbuf from perf ring buffer is
	198	a self-pacing notifications of new data being availability.
	199	``bpf_ringbuf_commit()`` implementation will send a notification of new record
	200	being available after commit only if consumer has already caught up right up to
	201	the record being committed. If not, consumer still has to catch up and thus
	202	will see new data anyways without needing an extra poll notification.
	203	Benchmarks (see tools/testing/selftests/bpf/benchs/bench_ringbuf.c_) show that
	204	this allows to achieve a very high throughput without having to resort to
	205	tricks like "notify only every Nth sample", which are necessary with perf
	206	buffer. For extreme cases, when BPF program wants more manual control of
	207	notifications, commit/discard/output helpers accept ``BPF_RB_NO_WAKEUP`` and
	208	``BPF_RB_FORCE_WAKEUP`` flags, which give full control over notifications of
	209	data availability, but require extra caution and diligence in using this API.

include/linux/bpf.h+13−0

 	int (*map_direct_value_meta)(const struct bpf_map *map,
 				     u64 imm, u32 *off);
 	int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
+	__poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
+			     struct poll_table_struct *pts);
 };
 struct bpf_map_memory {
 	ARG_PTR_TO_LONG,	/* pointer to long */
 	ARG_PTR_TO_SOCKET,	/* pointer to bpf_sock (fullsock) */
 	ARG_PTR_TO_BTF_ID,	/* pointer to in-kernel struct */
+	ARG_PTR_TO_ALLOC_MEM,	/* pointer to dynamically allocated memory */
+	ARG_PTR_TO_ALLOC_MEM_OR_NULL,	/* pointer to dynamically allocated memory or NULL */
+	ARG_CONST_ALLOC_SIZE_OR_ZERO,	/* number of allocated bytes requested */
 };
 /* type of values returned from helper functions */
 	RET_PTR_TO_SOCKET_OR_NULL,	/* returns a pointer to a socket or NULL */
 	RET_PTR_TO_TCP_SOCK_OR_NULL,	/* returns a pointer to a tcp_sock or NULL */
 	RET_PTR_TO_SOCK_COMMON_OR_NULL,	/* returns a pointer to a sock_common or NULL */
+	RET_PTR_TO_ALLOC_MEM_OR_NULL,	/* returns a pointer to dynamically allocated memory or NULL */
 };
 /* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs
 	PTR_TO_XDP_SOCK,	 /* reg points to struct xdp_sock */
 	PTR_TO_BTF_ID,		 /* reg points to kernel struct */
 	PTR_TO_BTF_ID_OR_NULL,	 /* reg points to kernel struct or NULL */
+	PTR_TO_MEM,		 /* reg points to valid memory region */
+	PTR_TO_MEM_OR_NULL,	 /* reg points to valid memory region or NULL */
 };
 /* The information passed from prog-specific *_is_valid_access
 extern const struct bpf_func_proto bpf_jiffies64_proto;
 extern const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto;
 extern const struct bpf_func_proto bpf_event_output_data_proto;
+extern const struct bpf_func_proto bpf_ringbuf_output_proto;
+extern const struct bpf_func_proto bpf_ringbuf_reserve_proto;
+extern const struct bpf_func_proto bpf_ringbuf_submit_proto;
+extern const struct bpf_func_proto bpf_ringbuf_discard_proto;
+extern const struct bpf_func_proto bpf_ringbuf_query_proto;
 const struct bpf_func_proto *bpf_tracing_func_proto(
 	enum bpf_func_id func_id, const struct bpf_prog *prog);

include/linux/bpf_types.h+1−0

@@ -118,6 +118,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_STACK, stack_map_ops) · 117 unmodified lines
118	#if defined(CONFIG_BPF_JIT)	118	#if defined(CONFIG_BPF_JIT)
119	BPF_MAP_TYPE(BPF_MAP_TYPE_STRUCT_OPS, bpf_struct_ops_map_ops)	119	BPF_MAP_TYPE(BPF_MAP_TYPE_STRUCT_OPS, bpf_struct_ops_map_ops)
120	#endif	120	#endif
		121	BPF_MAP_TYPE(BPF_MAP_TYPE_RINGBUF, ringbuf_map_ops)
121		122
122	BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)	123	BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)
123	BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing)	124	BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing)

include/linux/bpf_verifier.h+4−0

 		u32 btf_id; /* for PTR_TO_BTF_ID */
+		u32 mem_size; /* for PTR_TO_MEM | PTR_TO_MEM_OR_NULL */
 		/* Max size from any of the above. */
 		unsigned long raw;
 	};
 	 * offset, so they can share range knowledge.
 	 * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we
 	 * came from, when one is tested for != NULL.
+	 * For PTR_TO_MEM_OR_NULL this is used to identify memory allocation
+	 * for the purpose of tracking that it's freed.
 	 * For PTR_TO_SOCKET this is used to share which pointers retain the
 	 * same reference to the socket, to determine proper reference freeing.
 	 */

include/uapi/linux/bpf.h+83−1

@@ -147,6 +147,7 @@ enum bpf_map_type { · 146 unmodified lines
147	BPF_MAP_TYPE_SK_STORAGE,	147	BPF_MAP_TYPE_SK_STORAGE,
148	BPF_MAP_TYPE_DEVMAP_HASH,	148	BPF_MAP_TYPE_DEVMAP_HASH,
149	BPF_MAP_TYPE_STRUCT_OPS,	149	BPF_MAP_TYPE_STRUCT_OPS,
		150	BPF_MAP_TYPE_RINGBUF,
150	};	151	};
151		152
152	/* Note that tracing related programs such as	153	/* Note that tracing related programs such as
@@ -3157,6 +3158,59 @@ union bpf_attr { · 3004 unmodified lines
3157	* bpf_sk_cgroup_id\ ().	3158	* bpf_sk_cgroup_id\ ().
3158	* Return	3159	* Return
3159	* The id is returned or 0 in case the id could not be retrieved.	3160	* The id is returned or 0 in case the id could not be retrieved.
		3161	*
		3162	* void bpf_ringbuf_output(void ringbuf, void *data, u64 size, u64 flags)
		3163	* Description
		3164	* Copy size bytes from data into a ring buffer ringbuf.
		3165	* If BPF_RB_NO_WAKEUP is specified in flags, no notification of
		3166	* new data availability is sent.
		3167	* IF BPF_RB_FORCE_WAKEUP is specified in flags, notification of
		3168	* new data availability is sent unconditionally.
		3169	* Return
		3170	* 0, on success;
		3171	* < 0, on error.
		3172	*
		3173	* void bpf_ringbuf_reserve(void ringbuf, u64 size, u64 flags)
		3174	* Description
		3175	* Reserve size bytes of payload in a ring buffer ringbuf.
		3176	* Return
		3177	* Valid pointer with size bytes of memory available; NULL,
		3178	* otherwise.
		3179	*
		3180	* void bpf_ringbuf_submit(void *data, u64 flags)
		3181	* Description
		3182	* Submit reserved ring buffer sample, pointed to by data.
		3183	* If BPF_RB_NO_WAKEUP is specified in flags, no notification of
		3184	* new data availability is sent.
		3185	* IF BPF_RB_FORCE_WAKEUP is specified in flags, notification of
		3186	* new data availability is sent unconditionally.
		3187	* Return
		3188	* Nothing. Always succeeds.
		3189	*
		3190	* void bpf_ringbuf_discard(void *data, u64 flags)
		3191	* Description
		3192	* Discard reserved ring buffer sample, pointed to by data.
		3193	* If BPF_RB_NO_WAKEUP is specified in flags, no notification of
		3194	* new data availability is sent.
		3195	* IF BPF_RB_FORCE_WAKEUP is specified in flags, notification of
		3196	* new data availability is sent unconditionally.
		3197	* Return
		3198	* Nothing. Always succeeds.
		3199	*
		3200	* u64 bpf_ringbuf_query(void *ringbuf, u64 flags)
		3201	* Description
		3202	* Query various characteristics of provided ring buffer. What
		3203	* exactly is queries is determined by flags:
		3204	* - BPF_RB_AVAIL_DATA - amount of data not yet consumed;
		3205	* - BPF_RB_RING_SIZE - the size of ring buffer;
		3206	* - BPF_RB_CONS_POS - consumer position (can wrap around);
		3207	* - BPF_RB_PROD_POS - producer(s) position (can wrap around);
		3208	* Data returned is just a momentary snapshots of actual values
		3209	* and could be inaccurate, so this facility should be used to
		3210	* power heuristics and for reporting, not to make 100% correct
		3211	* calculation.
		3212	* Return
		3213	* Requested value, or 0, if flags are not recognized.
3160	*/	3214	*/
3161	#define __BPF_FUNC_MAPPER(FN) \	3215	#define __BPF_FUNC_MAPPER(FN) \
3162	FN(unspec), \	3216	FN(unspec), \
@@ -3288,7 +3342,12 @@ union bpf_attr { · 125 unmodified lines
3288	FN(seq_printf), \	3342	FN(seq_printf), \
3289	FN(seq_write), \	3343	FN(seq_write), \
3290	FN(sk_cgroup_id), \	3344	FN(sk_cgroup_id), \
3291	FN(sk_ancestor_cgroup_id),	3345	FN(sk_ancestor_cgroup_id), \
		3346	FN(ringbuf_output), \
		3347	FN(ringbuf_reserve), \
		3348	FN(ringbuf_submit), \
		3349	FN(ringbuf_discard), \
		3350	FN(ringbuf_query),
3292		3351
3293	/* integer value in 'imm' field of BPF_CALL instruction selects which helper	3352	/* integer value in 'imm' field of BPF_CALL instruction selects which helper
3294	* function eBPF program intends to call	3353	* function eBPF program intends to call
@@ -3398,6 +3457,29 @@ enum { · 103 unmodified lines
3398	BPF_F_GET_BRANCH_RECORDS_SIZE = (1ULL << 0),	3457	BPF_F_GET_BRANCH_RECORDS_SIZE = (1ULL << 0),
3399	};	3458	};
3400		3459
		3460	/* BPF_FUNC_bpf_ringbuf_commit, BPF_FUNC_bpf_ringbuf_discard, and
		3461	* BPF_FUNC_bpf_ringbuf_output flags.
		3462	*/
		3463	enum {
		3464	BPF_RB_NO_WAKEUP = (1ULL << 0),
		3465	BPF_RB_FORCE_WAKEUP = (1ULL << 1),
		3466	};
		3467
		3468	/* BPF_FUNC_bpf_ringbuf_query flags */
		3469	enum {
		3470	BPF_RB_AVAIL_DATA = 0,
		3471	BPF_RB_RING_SIZE = 1,
		3472	BPF_RB_CONS_POS = 2,
		3473	BPF_RB_PROD_POS = 3,
		3474	};
		3475
		3476	/* BPF ring buffer constants */
		3477	enum {
		3478	BPF_RINGBUF_BUSY_BIT = (1U << 31),
		3479	BPF_RINGBUF_DISCARD_BIT = (1U << 30),
		3480	BPF_RINGBUF_HDR_SZ = 8,
		3481	};
		3482
3401	/* Mode for BPF_FUNC_skb_adjust_room helper. */	3483	/* Mode for BPF_FUNC_skb_adjust_room helper. */
3402	enum bpf_adj_room_mode {	3484	enum bpf_adj_room_mode {
3403	BPF_ADJ_ROOM_NET,	3485	BPF_ADJ_ROOM_NET,

kernel/bpf/Makefile+1−1

@@ -4,7 +4,7 @@ CFLAGS_core.o += $(call cc-disable-warning, override-init) · 3 unmodified lines
4		4
5	obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o task_iter.o	5	obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o task_iter.o
6	obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o	6	obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o
7	obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o	7	obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
8	obj-$(CONFIG_BPF_SYSCALL) += disasm.o	8	obj-$(CONFIG_BPF_SYSCALL) += disasm.o
9	obj-$(CONFIG_BPF_JIT) += trampoline.o	9	obj-$(CONFIG_BPF_JIT) += trampoline.o
10	obj-$(CONFIG_BPF_SYSCALL) += btf.o	10	obj-$(CONFIG_BPF_SYSCALL) += btf.o

kernel/bpf/helpers.c+10−0

@@ -635,6 +635,16 @@ bpf_base_func_proto(enum bpf_func_id func_id) · 634 unmodified lines
635	return &bpf_ktime_get_ns_proto;	635	return &bpf_ktime_get_ns_proto;
636	case BPF_FUNC_ktime_get_boot_ns:	636	case BPF_FUNC_ktime_get_boot_ns:
637	return &bpf_ktime_get_boot_ns_proto;	637	return &bpf_ktime_get_boot_ns_proto;
		638	case BPF_FUNC_ringbuf_output:
		639	return &bpf_ringbuf_output_proto;
		640	case BPF_FUNC_ringbuf_reserve:
		641	return &bpf_ringbuf_reserve_proto;
		642	case BPF_FUNC_ringbuf_submit:
		643	return &bpf_ringbuf_submit_proto;
		644	case BPF_FUNC_ringbuf_discard:
		645	return &bpf_ringbuf_discard_proto;
		646	case BPF_FUNC_ringbuf_query:
		647	return &bpf_ringbuf_query_proto;
638	default:	648	default:
639	break;	649	break;
640	}	650	}

kernel/bpf/ringbuf.c+501−0

@@ -0,0 +1,501 @@
	1	#include <linux/bpf.h>
	2	#include <linux/btf.h>
	3	#include <linux/err.h>
	4	#include <linux/irq_work.h>
	5	#include <linux/slab.h>
	6	#include <linux/filter.h>
	7	#include <linux/mm.h>
	8	#include <linux/vmalloc.h>
	9	#include <linux/wait.h>
	10	#include <linux/poll.h>
	11	#include <uapi/linux/btf.h>
	12
	13	#define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)
	14
	15	/* non-mmap()'able part of bpf_ringbuf (everything up to consumer page) */
	16	#define RINGBUF_PGOFF \
	17	(offsetof(struct bpf_ringbuf, consumer_pos) >> PAGE_SHIFT)
	18	/* consumer page and producer page */
	19	#define RINGBUF_POS_PAGES 2
	20
	21	#define RINGBUF_MAX_RECORD_SZ (UINT_MAX/4)
	22
	23	/* Maximum size of ring buffer area is limited by 32-bit page offset within
	24	* record header, counted in pages. Reserve 8 bits for extensibility, and take
	25	* into account few extra pages for consumer/producer pages and
	26	* non-mmap()'able parts. This gives 64GB limit, which seems plenty for single
	27	* ring buffer.
	28	*/
	29	#define RINGBUF_MAX_DATA_SZ \
	30	(((1ULL << 24) - RINGBUF_POS_PAGES - RINGBUF_PGOFF) * PAGE_SIZE)
	31
	32	struct bpf_ringbuf {
	33	wait_queue_head_t waitq;
	34	struct irq_work work;
	35	u64 mask;
	36	struct page **pages;
	37	int nr_pages;
	38	spinlock_t spinlock ____cacheline_aligned_in_smp;
	39	/* Consumer and producer counters are put into separate pages to allow
	40	* mapping consumer page as r/w, but restrict producer page to r/o.
	41	* This protects producer position from being modified by user-space
	42	* application and ruining in-kernel position tracking.
	43	*/
	44	unsigned long consumer_pos __aligned(PAGE_SIZE);
	45	unsigned long producer_pos __aligned(PAGE_SIZE);
	46	char data[] __aligned(PAGE_SIZE);
	47	};
	48
	49	struct bpf_ringbuf_map {
	50	struct bpf_map map;
	51	struct bpf_map_memory memory;
	52	struct bpf_ringbuf *rb;
	53	};
	54
	55	/* 8-byte ring buffer record header structure */
	56	struct bpf_ringbuf_hdr {
	57	u32 len;
	58	u32 pg_off;
	59	};
	60
	61	static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node)
	62	{
	63	const gfp_t flags = GFP_KERNEL \| __GFP_RETRY_MAYFAIL \| __GFP_NOWARN \|
	64	__GFP_ZERO;
	65	int nr_meta_pages = RINGBUF_PGOFF + RINGBUF_POS_PAGES;
	66	int nr_data_pages = data_sz >> PAGE_SHIFT;
	67	int nr_pages = nr_meta_pages + nr_data_pages;
	68	struct page *pages, page;
	69	struct bpf_ringbuf *rb;
	70	size_t array_size;
	71	int i;
	72
	73	/* Each data page is mapped twice to allow "virtual"
	74	* continuous read of samples wrapping around the end of ring
	75	* buffer area:
	76	* ------------------------------------------------------
	77	* \| meta pages \| real data pages \| same data pages \|
	78	* ------------------------------------------------------
	79	* \| \| 1 2 3 4 5 6 7 8 9 \| 1 2 3 4 5 6 7 8 9 \|
	80	* ------------------------------------------------------
	81	* \| \| TA DA \| TA DA \|
	82	* ------------------------------------------------------
	83	* ^^^^^^^
	84	* \|
	85	* Here, no need to worry about special handling of wrapped-around
	86	* data due to double-mapped data pages. This works both in kernel and
	87	* when mmap()'ed in user-space, simplifying both kernel and
	88	* user-space implementations significantly.
	89	*/
	90	array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages);
	91	if (array_size > PAGE_SIZE)
	92	pages = vmalloc_node(array_size, numa_node);
	93	else
	94	pages = kmalloc_node(array_size, flags, numa_node);
	95	if (!pages)
	96	return NULL;
	97
	98	for (i = 0; i < nr_pages; i++) {
	99	page = alloc_pages_node(numa_node, flags, 0);
	100	if (!page) {
	101	nr_pages = i;
	102	goto err_free_pages;
	103	}
	104	pages[i] = page;
	105	if (i >= nr_meta_pages)
	106	pages[nr_data_pages + i] = page;
	107	}
	108
	109	rb = vmap(pages, nr_meta_pages + 2 * nr_data_pages,
	110	VM_ALLOC \| VM_USERMAP, PAGE_KERNEL);
	111	if (rb) {
	112	rb->pages = pages;
	113	rb->nr_pages = nr_pages;
	114	return rb;
	115	}
	116
	117	err_free_pages:
	118	for (i = 0; i < nr_pages; i++)
	119	__free_page(pages[i]);
	120	kvfree(pages);
	121	return NULL;
	122	}
	123
	124	static void bpf_ringbuf_notify(struct irq_work *work)
	125	{
	126	struct bpf_ringbuf *rb = container_of(work, struct bpf_ringbuf, work);
	127
	128	wake_up_all(&rb->waitq);
	129	}
	130
	131	static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, int numa_node)
	132	{
	133	struct bpf_ringbuf *rb;
	134
	135	if (!data_sz \|\| !PAGE_ALIGNED(data_sz))
	136	return ERR_PTR(-EINVAL);
	137
	138	#ifdef CONFIG_64BIT
	139	/* on 32-bit arch, it's impossible to overflow record's hdr->pgoff */
	140	if (data_sz > RINGBUF_MAX_DATA_SZ)
	141	return ERR_PTR(-E2BIG);
	142	#endif
	143
	144	rb = bpf_ringbuf_area_alloc(data_sz, numa_node);
	145	if (!rb)
	146	return ERR_PTR(-ENOMEM);
	147
	148	spin_lock_init(&rb->spinlock);
	149	init_waitqueue_head(&rb->waitq);
	150	init_irq_work(&rb->work, bpf_ringbuf_notify);
	151
	152	rb->mask = data_sz - 1;
	153	rb->consumer_pos = 0;
	154	rb->producer_pos = 0;
	155
	156	return rb;
	157	}
	158
	159	static struct bpf_map ringbuf_map_alloc(union bpf_attr attr)
	160	{
	161	struct bpf_ringbuf_map *rb_map;
	162	u64 cost;
	163	int err;
	164
	165	if (attr->map_flags & ~RINGBUF_CREATE_FLAG_MASK)
	166	return ERR_PTR(-EINVAL);
	167
	168	if (attr->key_size \|\| attr->value_size \|\|
	169	attr->max_entries == 0 \|\| !PAGE_ALIGNED(attr->max_entries))
	170	return ERR_PTR(-EINVAL);
	171
	172	rb_map = kzalloc(sizeof(*rb_map), GFP_USER);
	173	if (!rb_map)
	174	return ERR_PTR(-ENOMEM);
	175
	176	bpf_map_init_from_attr(&rb_map->map, attr);
	177
	178	cost = sizeof(struct bpf_ringbuf_map) +
	179	sizeof(struct bpf_ringbuf) +
	180	attr->max_entries;
	181	err = bpf_map_charge_init(&rb_map->map.memory, cost);
	182	if (err)
	183	goto err_free_map;
	184
	185	rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node);
	186	if (IS_ERR(rb_map->rb)) {
	187	err = PTR_ERR(rb_map->rb);
	188	goto err_uncharge;
	189	}
	190
	191	return &rb_map->map;
	192
	193	err_uncharge:
	194	bpf_map_charge_finish(&rb_map->map.memory);
	195	err_free_map:
	196	kfree(rb_map);
	197	return ERR_PTR(err);
	198	}
	199
	200	static void bpf_ringbuf_free(struct bpf_ringbuf *rb)
	201	{
	202	/* copy pages pointer and nr_pages to local variable, as we are going
	203	* to unmap rb itself with vunmap() below
	204	*/
	205	struct page **pages = rb->pages;
	206	int i, nr_pages = rb->nr_pages;
	207
	208	vunmap(rb);
	209	for (i = 0; i < nr_pages; i++)
	210	__free_page(pages[i]);
	211	kvfree(pages);
	212	}
	213
	214	static void ringbuf_map_free(struct bpf_map *map)
	215	{
	216	struct bpf_ringbuf_map *rb_map;
	217
	218	/* at this point bpf_prog->aux->refcnt == 0 and this map->refcnt == 0,
	219	* so the programs (can be more than one that used this map) were
	220	* disconnected from events. Wait for outstanding critical sections in
	221	* these programs to complete
	222	*/
	223	synchronize_rcu();
	224
	225	rb_map = container_of(map, struct bpf_ringbuf_map, map);
	226	bpf_ringbuf_free(rb_map->rb);
	227	kfree(rb_map);
	228	}
	229
	230	static void ringbuf_map_lookup_elem(struct bpf_map map, void *key)
	231	{
	232	return ERR_PTR(-ENOTSUPP);
	233	}
	234
	235	static int ringbuf_map_update_elem(struct bpf_map map, void key, void *value,
	236	u64 flags)
	237	{
	238	return -ENOTSUPP;
	239	}
	240
	241	static int ringbuf_map_delete_elem(struct bpf_map map, void key)
	242	{
	243	return -ENOTSUPP;
	244	}
	245
	246	static int ringbuf_map_get_next_key(struct bpf_map map, void key,
	247	void *next_key)
	248	{
	249	return -ENOTSUPP;
	250	}
	251
	252	static size_t bpf_ringbuf_mmap_page_cnt(const struct bpf_ringbuf *rb)
	253	{
	254	size_t data_pages = (rb->mask + 1) >> PAGE_SHIFT;
	255
	256	/* consumer page + producer page + 2 x data pages */
	257	return RINGBUF_POS_PAGES + 2 * data_pages;
	258	}
	259
	260	static int ringbuf_map_mmap(struct bpf_map map, struct vm_area_struct vma)
	261	{
	262	struct bpf_ringbuf_map *rb_map;
	263	size_t mmap_sz;
	264
	265	rb_map = container_of(map, struct bpf_ringbuf_map, map);
	266	mmap_sz = bpf_ringbuf_mmap_page_cnt(rb_map->rb) << PAGE_SHIFT;
	267
	268	if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) > mmap_sz)
	269	return -EINVAL;
	270
	271	return remap_vmalloc_range(vma, rb_map->rb,
	272	vma->vm_pgoff + RINGBUF_PGOFF);
	273	}
	274
	275	static unsigned long ringbuf_avail_data_sz(struct bpf_ringbuf *rb)
	276	{
	277	unsigned long cons_pos, prod_pos;
	278
	279	cons_pos = smp_load_acquire(&rb->consumer_pos);
	280	prod_pos = smp_load_acquire(&rb->producer_pos);
	281	return prod_pos - cons_pos;
	282	}
	283
	284	static __poll_t ringbuf_map_poll(struct bpf_map map, struct file filp,
	285	struct poll_table_struct *pts)
	286	{
	287	struct bpf_ringbuf_map *rb_map;
	288
	289	rb_map = container_of(map, struct bpf_ringbuf_map, map);
	290	poll_wait(filp, &rb_map->rb->waitq, pts);
	291
	292	if (ringbuf_avail_data_sz(rb_map->rb))
	293	return EPOLLIN \| EPOLLRDNORM;
	294	return 0;
	295	}
	296
	297	const struct bpf_map_ops ringbuf_map_ops = {
	298	.map_alloc = ringbuf_map_alloc,
	299	.map_free = ringbuf_map_free,
	300	.map_mmap = ringbuf_map_mmap,
	301	.map_poll = ringbuf_map_poll,
	302	.map_lookup_elem = ringbuf_map_lookup_elem,
	303	.map_update_elem = ringbuf_map_update_elem,
	304	.map_delete_elem = ringbuf_map_delete_elem,
	305	.map_get_next_key = ringbuf_map_get_next_key,
	306	};
	307
	308	/* Given pointer to ring buffer record metadata and struct bpf_ringbuf itself,
	309	* calculate offset from record metadata to ring buffer in pages, rounded
	310	* down. This page offset is stored as part of record metadata and allows to
	311	* restore struct bpf_ringbuf * from record pointer. This page offset is
	312	* stored at offset 4 of record metadata header.
	313	*/
	314	static size_t bpf_ringbuf_rec_pg_off(struct bpf_ringbuf *rb,
	315	struct bpf_ringbuf_hdr *hdr)
	316	{
	317	return ((void )hdr - (void )rb) >> PAGE_SHIFT;
	318	}
	319
	320	/* Given pointer to ring buffer record header, restore pointer to struct
	321	* bpf_ringbuf itself by using page offset stored at offset 4
	322	*/
	323	static struct bpf_ringbuf *
	324	bpf_ringbuf_restore_from_rec(struct bpf_ringbuf_hdr *hdr)
	325	{
	326	unsigned long addr = (unsigned long)(void *)hdr;
	327	unsigned long off = (unsigned long)hdr->pg_off << PAGE_SHIFT;
	328
	329	return (void*)((addr & PAGE_MASK) - off);
	330	}
	331
	332	static void __bpf_ringbuf_reserve(struct bpf_ringbuf rb, u64 size)
	333	{
	334	unsigned long cons_pos, prod_pos, new_prod_pos, flags;
	335	u32 len, pg_off;
	336	struct bpf_ringbuf_hdr *hdr;
	337
	338	if (unlikely(size > RINGBUF_MAX_RECORD_SZ))
	339	return NULL;
	340
	341	len = round_up(size + BPF_RINGBUF_HDR_SZ, 8);
	342	cons_pos = smp_load_acquire(&rb->consumer_pos);
	343
	344	if (in_nmi()) {
	345	if (!spin_trylock_irqsave(&rb->spinlock, flags))
	346	return NULL;
	347	} else {
	348	spin_lock_irqsave(&rb->spinlock, flags);
	349	}
	350
	351	prod_pos = rb->producer_pos;
	352	new_prod_pos = prod_pos + len;
	353
	354	/* check for out of ringbuf space by ensuring producer position
	355	* doesn't advance more than (ringbuf_size - 1) ahead
	356	*/
	357	if (new_prod_pos - cons_pos > rb->mask) {
	358	spin_unlock_irqrestore(&rb->spinlock, flags);
	359	return NULL;
	360	}
	361
	362	hdr = (void *)rb->data + (prod_pos & rb->mask);
	363	pg_off = bpf_ringbuf_rec_pg_off(rb, hdr);
	364	hdr->len = size \| BPF_RINGBUF_BUSY_BIT;
	365	hdr->pg_off = pg_off;
	366
	367	/* pairs with consumer's smp_load_acquire() */
	368	smp_store_release(&rb->producer_pos, new_prod_pos);
	369
	370	spin_unlock_irqrestore(&rb->spinlock, flags);
	371
	372	return (void *)hdr + BPF_RINGBUF_HDR_SZ;
	373	}
	374
	375	BPF_CALL_3(bpf_ringbuf_reserve, struct bpf_map *, map, u64, size, u64, flags)
	376	{
	377	struct bpf_ringbuf_map *rb_map;
	378
	379	if (unlikely(flags))
	380	return 0;
	381
	382	rb_map = container_of(map, struct bpf_ringbuf_map, map);
	383	return (unsigned long)__bpf_ringbuf_reserve(rb_map->rb, size);
	384	}
	385
	386	const struct bpf_func_proto bpf_ringbuf_reserve_proto = {
	387	.func = bpf_ringbuf_reserve,
	388	.ret_type = RET_PTR_TO_ALLOC_MEM_OR_NULL,
	389	.arg1_type = ARG_CONST_MAP_PTR,
	390	.arg2_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
	391	.arg3_type = ARG_ANYTHING,
	392	};
	393
	394	static void bpf_ringbuf_commit(void *sample, u64 flags, bool discard)
	395	{
	396	unsigned long rec_pos, cons_pos;
	397	struct bpf_ringbuf_hdr *hdr;
	398	struct bpf_ringbuf *rb;
	399	u32 new_len;
	400
	401	hdr = sample - BPF_RINGBUF_HDR_SZ;
	402	rb = bpf_ringbuf_restore_from_rec(hdr);
	403	new_len = hdr->len ^ BPF_RINGBUF_BUSY_BIT;
	404	if (discard)
	405	new_len \|= BPF_RINGBUF_DISCARD_BIT;
	406
	407	/* update record header with correct final size prefix */
	408	xchg(&hdr->len, new_len);
	409
	410	/* if consumer caught up and is waiting for our record, notify about
	411	* new data availability
	412	*/
	413	rec_pos = (void )hdr - (void )rb->data;
	414	cons_pos = smp_load_acquire(&rb->consumer_pos) & rb->mask;
	415
	416	if (flags & BPF_RB_FORCE_WAKEUP)
	417	irq_work_queue(&rb->work);
	418	else if (cons_pos == rec_pos && !(flags & BPF_RB_NO_WAKEUP))
	419	irq_work_queue(&rb->work);
	420	}
	421
	422	BPF_CALL_2(bpf_ringbuf_submit, void *, sample, u64, flags)
	423	{
	424	bpf_ringbuf_commit(sample, flags, false /* discard */);
	425	return 0;
	426	}
	427
	428	const struct bpf_func_proto bpf_ringbuf_submit_proto = {
	429	.func = bpf_ringbuf_submit,
	430	.ret_type = RET_VOID,
	431	.arg1_type = ARG_PTR_TO_ALLOC_MEM,
	432	.arg2_type = ARG_ANYTHING,
	433	};
	434
	435	BPF_CALL_2(bpf_ringbuf_discard, void *, sample, u64, flags)
	436	{
	437	bpf_ringbuf_commit(sample, flags, true /* discard */);
	438	return 0;
	439	}
	440
	441	const struct bpf_func_proto bpf_ringbuf_discard_proto = {
	442	.func = bpf_ringbuf_discard,
	443	.ret_type = RET_VOID,
	444	.arg1_type = ARG_PTR_TO_ALLOC_MEM,
	445	.arg2_type = ARG_ANYTHING,
	446	};
	447
	448	BPF_CALL_4(bpf_ringbuf_output, struct bpf_map , map, void , data, u64, size,
	449	u64, flags)
	450	{
	451	struct bpf_ringbuf_map *rb_map;
	452	void *rec;
	453
	454	if (unlikely(flags & ~(BPF_RB_NO_WAKEUP \| BPF_RB_FORCE_WAKEUP)))
	455	return -EINVAL;
	456
	457	rb_map = container_of(map, struct bpf_ringbuf_map, map);
	458	rec = __bpf_ringbuf_reserve(rb_map->rb, size);
	459	if (!rec)
	460	return -EAGAIN;
	461
	462	memcpy(rec, data, size);
	463	bpf_ringbuf_commit(rec, flags, false /* discard */);
	464	return 0;
	465	}
	466
	467	const struct bpf_func_proto bpf_ringbuf_output_proto = {
	468	.func = bpf_ringbuf_output,
	469	.ret_type = RET_INTEGER,
	470	.arg1_type = ARG_CONST_MAP_PTR,
	471	.arg2_type = ARG_PTR_TO_MEM,
	472	.arg3_type = ARG_CONST_SIZE_OR_ZERO,
	473	.arg4_type = ARG_ANYTHING,
	474	};
	475
	476	BPF_CALL_2(bpf_ringbuf_query, struct bpf_map *, map, u64, flags)
	477	{
	478	struct bpf_ringbuf *rb;
	479
	480	rb = container_of(map, struct bpf_ringbuf_map, map)->rb;
	481
	482	switch (flags) {
	483	case BPF_RB_AVAIL_DATA:
	484	return ringbuf_avail_data_sz(rb);
	485	case BPF_RB_RING_SIZE:
	486	return rb->mask + 1;
	487	case BPF_RB_CONS_POS:
	488	return smp_load_acquire(&rb->consumer_pos);
	489	case BPF_RB_PROD_POS:
	490	return smp_load_acquire(&rb->producer_pos);
	491	default:
	492	return 0;
	493	}
	494	}
	495
	496	const struct bpf_func_proto bpf_ringbuf_query_proto = {
	497	.func = bpf_ringbuf_query,
	498	.ret_type = RET_INTEGER,
	499	.arg1_type = ARG_CONST_MAP_PTR,
	500	.arg2_type = ARG_ANYTHING,
	501	};

kernel/bpf/syscall.c+12−0

 #include <linux/audit.h>
 #include <uapi/linux/btf.h>
 #include <linux/bpf_lsm.h>
+#include <linux/poll.h>
 #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \
 			  (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \
 	return err;
 }
+static __poll_t bpf_map_poll(struct file *filp, struct poll_table_struct *pts)
+{
+	struct bpf_map *map = filp->private_data;
+	if (map->ops->map_poll)
+		return map->ops->map_poll(map, filp, pts);
+	return EPOLLERR;
+}
 const struct file_operations bpf_map_fops = {
 #ifdef CONFIG_PROC_FS
 	.show_fdinfo	= bpf_map_show_fdinfo,
 	.read		= bpf_dummy_read,
 	.write		= bpf_dummy_write,
 	.mmap		= bpf_map_mmap,
+	.poll		= bpf_map_poll,
 };
 int bpf_map_new_fd(struct bpf_map *map, int flags)

kernel/bpf/verifier.c+146−49

@@ -233,6 +233,7 @@ struct bpf_call_arg_meta { · 232 unmodified lines
233	bool pkt_access;	233	bool pkt_access;
234	int regno;	234	int regno;
235	int access_size;	235	int access_size;
		236	int mem_size;
236	u64 msize_max_value;	237	u64 msize_max_value;
237	int ref_obj_id;	238	int ref_obj_id;
238	int func_id;	239	int func_id;
@@ -408,7 +409,8 @@ static bool reg_type_may_be_null(enum bpf_reg_type type) · 169 unmodified lines
408	type == PTR_TO_SOCKET_OR_NULL \|\|	409	type == PTR_TO_SOCKET_OR_NULL \|\|
409	type == PTR_TO_SOCK_COMMON_OR_NULL \|\|	410	type == PTR_TO_SOCK_COMMON_OR_NULL \|\|
410	type == PTR_TO_TCP_SOCK_OR_NULL \|\|	411	type == PTR_TO_TCP_SOCK_OR_NULL \|\|
411	type == PTR_TO_BTF_ID_OR_NULL;	412	type == PTR_TO_BTF_ID_OR_NULL \|\|
		413	type == PTR_TO_MEM_OR_NULL;
412	}	414	}
413		415
414	static bool reg_may_point_to_spin_lock(const struct bpf_reg_state *reg)	416	static bool reg_may_point_to_spin_lock(const struct bpf_reg_state *reg)
@@ -422,7 +424,9 @@ static bool reg_type_may_be_refcounted_or_null(enum bpf_reg_type type) · 7 unmodified lines
422	return type == PTR_TO_SOCKET \|\|	424	return type == PTR_TO_SOCKET \|\|
423	type == PTR_TO_SOCKET_OR_NULL \|\|	425	type == PTR_TO_SOCKET_OR_NULL \|\|
424	type == PTR_TO_TCP_SOCK \|\|	426	type == PTR_TO_TCP_SOCK \|\|
425	type == PTR_TO_TCP_SOCK_OR_NULL;	427	type == PTR_TO_TCP_SOCK_OR_NULL \|\|
		428	type == PTR_TO_MEM \|\|
		429	type == PTR_TO_MEM_OR_NULL;
426	}	430	}
427		431
428	static bool arg_type_may_be_refcounted(enum bpf_arg_type type)	432	static bool arg_type_may_be_refcounted(enum bpf_arg_type type)
@@ -436,7 +440,9 @@ static bool arg_type_may_be_refcounted(enum bpf_arg_type type) · 7 unmodified lines
436	*/	440	*/
437	static bool is_release_function(enum bpf_func_id func_id)	441	static bool is_release_function(enum bpf_func_id func_id)
438	{	442	{
439	return func_id == BPF_FUNC_sk_release;	443	return func_id == BPF_FUNC_sk_release \|\|
		444	func_id == BPF_FUNC_ringbuf_submit \|\|
		445	func_id == BPF_FUNC_ringbuf_discard;
440	}	446	}
441		447
442	static bool may_be_acquire_function(enum bpf_func_id func_id)	448	static bool may_be_acquire_function(enum bpf_func_id func_id)
@@ -444,7 +450,8 @@ static bool may_be_acquire_function(enum bpf_func_id func_id) · 1 unmodified line
444	return func_id == BPF_FUNC_sk_lookup_tcp \|\|	450	return func_id == BPF_FUNC_sk_lookup_tcp \|\|
445	func_id == BPF_FUNC_sk_lookup_udp \|\|	451	func_id == BPF_FUNC_sk_lookup_udp \|\|
446	func_id == BPF_FUNC_skc_lookup_tcp \|\|	452	func_id == BPF_FUNC_skc_lookup_tcp \|\|
447	func_id == BPF_FUNC_map_lookup_elem;	453	func_id == BPF_FUNC_map_lookup_elem \|\|
		454	func_id == BPF_FUNC_ringbuf_reserve;
448	}	455	}
449		456
450	static bool is_acquire_function(enum bpf_func_id func_id,	457	static bool is_acquire_function(enum bpf_func_id func_id,
@@ -454,7 +461,8 @@ static bool is_acquire_function(enum bpf_func_id func_id, · 3 unmodified lines
454		461
455	if (func_id == BPF_FUNC_sk_lookup_tcp \|\|	462	if (func_id == BPF_FUNC_sk_lookup_tcp \|\|
456	func_id == BPF_FUNC_sk_lookup_udp \|\|	463	func_id == BPF_FUNC_sk_lookup_udp \|\|
457	func_id == BPF_FUNC_skc_lookup_tcp)	464	func_id == BPF_FUNC_skc_lookup_tcp \|\|
		465	func_id == BPF_FUNC_ringbuf_reserve)
458	return true;	466	return true;
459		467
460	if (func_id == BPF_FUNC_map_lookup_elem &&	468	if (func_id == BPF_FUNC_map_lookup_elem &&
@@ -494,6 +502,8 @@ static const char * const reg_type_str[] = { · 33 unmodified lines
494	[PTR_TO_XDP_SOCK] = "xdp_sock",	502	[PTR_TO_XDP_SOCK] = "xdp_sock",
495	[PTR_TO_BTF_ID] = "ptr_",	503	[PTR_TO_BTF_ID] = "ptr_",
496	[PTR_TO_BTF_ID_OR_NULL] = "ptr_or_null_",	504	[PTR_TO_BTF_ID_OR_NULL] = "ptr_or_null_",
		505	[PTR_TO_MEM] = "mem",
		506	[PTR_TO_MEM_OR_NULL] = "mem_or_null",
497	};	507	};
498		508
499	static char slot_type_char[] = {	509	static char slot_type_char[] = {
@@ -2468,32 +2478,49 @@ static int check_map_access_type(struct bpf_verifier_env *env, u32 regno, · 1968 unmodified lines
2468	return 0;	2478	return 0;
2469	}	2479	}
2470		2480
2471	/* check read/write into map element returned by bpf_map_lookup_elem() */	2481	/* check read/write into memory region (e.g., map value, ringbuf sample, etc) */
2472	static int __check_map_access(struct bpf_verifier_env *env, u32 regno, int off,	2482	static int __check_mem_access(struct bpf_verifier_env *env, int regno,
2473	int size, bool zero_size_allowed)	2483	int off, int size, u32 mem_size,
		2484	bool zero_size_allowed)
2474	{	2485	{
2475	struct bpf_reg_state *regs = cur_regs(env);	2486	bool size_ok = size > 0 \|\| (size == 0 && zero_size_allowed);
2476	struct bpf_map *map = regs[regno].map_ptr;	2487	struct bpf_reg_state *reg;
		2488
		2489	if (off >= 0 && size_ok && (u64)off + size <= mem_size)
		2490	return 0;
2477		2491
2478	if (off < 0 \|\| size < 0 \|\| (size == 0 && !zero_size_allowed) \|\|	2492	reg = &cur_regs(env)[regno];
2479	off + size > map->value_size) {	2493	switch (reg->type) {
		2494	case PTR_TO_MAP_VALUE:
2480	verbose(env, "invalid access to map value, value_size=%d off=%d size=%d\n",	2495	verbose(env, "invalid access to map value, value_size=%d off=%d size=%d\n",
2481	map->value_size, off, size);	2496	mem_size, off, size);
2482	return -EACCES;	2497	break;
		2498	case PTR_TO_PACKET:
		2499	case PTR_TO_PACKET_META:
		2500	case PTR_TO_PACKET_END:
		2501	verbose(env, "invalid access to packet, off=%d size=%d, R%d(id=%d,off=%d,r=%d)\n",
		2502	off, size, regno, reg->id, off, mem_size);
		2503	break;
		2504	case PTR_TO_MEM:
		2505	default:
		2506	verbose(env, "invalid access to memory, mem_size=%u off=%d size=%d\n",
		2507	mem_size, off, size);
2483	}	2508	}
2484	return 0;	2509
		2510	return -EACCES;
2485	}	2511	}
2486		2512
2487	/* check read/write into a map element with possible variable offset */	2513	/* check read/write into a memory region with possible variable offset */
2488	static int check_map_access(struct bpf_verifier_env *env, u32 regno,	2514	static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno,
2489	int off, int size, bool zero_size_allowed)	2515	int off, int size, u32 mem_size,
		2516	bool zero_size_allowed)
2490	{	2517	{
2491	struct bpf_verifier_state *vstate = env->cur_state;	2518	struct bpf_verifier_state *vstate = env->cur_state;
2492	struct bpf_func_state *state = vstate->frame[vstate->curframe];	2519	struct bpf_func_state *state = vstate->frame[vstate->curframe];
2493	struct bpf_reg_state *reg = &state->regs[regno];	2520	struct bpf_reg_state *reg = &state->regs[regno];
2494	int err;	2521	int err;
2495		2522
2496	/* We may have adjusted the register to this map value, so we	2523	/* We may have adjusted the register pointing to memory region, so we
2497	* need to try adding each of min_value and max_value to off	2524	* need to try adding each of min_value and max_value to off
2498	* to make sure our theoretical access will be safe.	2525	* to make sure our theoretical access will be safe.
2499	*/	2526	*/
@@ -2514,10 +2541,10 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 14 unmodified lines
2514	regno);	2541	regno);
2515	return -EACCES;	2542	return -EACCES;
2516	}	2543	}
2517	err = __check_map_access(env, regno, reg->smin_value + off, size,	2544	err = __check_mem_access(env, regno, reg->smin_value + off, size,
2518	zero_size_allowed);	2545	mem_size, zero_size_allowed);
2519	if (err) {	2546	if (err) {
2520	verbose(env, "R%d min value is outside of the array range\n",	2547	verbose(env, "R%d min value is outside of the allowed memory range\n",
2521	regno);	2548	regno);
2522	return err;	2549	return err;
2523	}	2550	}
@@ -2527,18 +2554,38 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 3 unmodified lines
2527	* If reg->umax_value + off could overflow, treat that as unbounded too.	2554	* If reg->umax_value + off could overflow, treat that as unbounded too.
2528	*/	2555	*/
2529	if (reg->umax_value >= BPF_MAX_VAR_OFF) {	2556	if (reg->umax_value >= BPF_MAX_VAR_OFF) {
2530	verbose(env, "R%d unbounded memory access, make sure to bounds check any array access into a map\n",	2557	verbose(env, "R%d unbounded memory access, make sure to bounds check any such access\n",
2531	regno);	2558	regno);
2532	return -EACCES;	2559	return -EACCES;
2533	}	2560	}
2534	err = __check_map_access(env, regno, reg->umax_value + off, size,	2561	err = __check_mem_access(env, regno, reg->umax_value + off, size,
2535	zero_size_allowed);	2562	mem_size, zero_size_allowed);
2536	if (err)	2563	if (err) {
2537	verbose(env, "R%d max value is outside of the array range\n",	2564	verbose(env, "R%d max value is outside of the allowed memory range\n",
2538	regno);	2565	regno);
		2566	return err;
		2567	}
		2568
		2569	return 0;
		2570	}
2539		2571
2540	if (map_value_has_spin_lock(reg->map_ptr)) {	2572	/* check read/write into a map element with possible variable offset */
2541	u32 lock = reg->map_ptr->spin_lock_off;	2573	static int check_map_access(struct bpf_verifier_env *env, u32 regno,
		2574	int off, int size, bool zero_size_allowed)
		2575	{
		2576	struct bpf_verifier_state *vstate = env->cur_state;
		2577	struct bpf_func_state *state = vstate->frame[vstate->curframe];
		2578	struct bpf_reg_state *reg = &state->regs[regno];
		2579	struct bpf_map *map = reg->map_ptr;
		2580	int err;
		2581
		2582	err = check_mem_region_access(env, regno, off, size, map->value_size,
		2583	zero_size_allowed);
		2584	if (err)
		2585	return err;
		2586
		2587	if (map_value_has_spin_lock(map)) {
		2588	u32 lock = map->spin_lock_off;
2542		2589
2543	/* if any part of struct bpf_spin_lock can be touched by	2590	/* if any part of struct bpf_spin_lock can be touched by
2544	* load/store reject this program.	2591	* load/store reject this program.
@@ -2596,21 +2643,6 @@ static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, · 51 unmodified lines
2596	}	2643	}
2597	}	2644	}
2598		2645
2599	static int __check_packet_access(struct bpf_verifier_env *env, u32 regno,
2600	int off, int size, bool zero_size_allowed)
2601	{
2602	struct bpf_reg_state *regs = cur_regs(env);
2603	struct bpf_reg_state *reg = &regs[regno];
2604
2605	if (off < 0 \|\| size < 0 \|\| (size == 0 && !zero_size_allowed) \|\|
2606	(u64)off + size > reg->range) {
2607	verbose(env, "invalid access to packet, off=%d size=%d, R%d(id=%d,off=%d,r=%d)\n",
2608	off, size, regno, reg->id, reg->off, reg->range);
2609	return -EACCES;
2610	}
2611	return 0;
2612	}
2613
2614	static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off,	2646	static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off,
2615	int size, bool zero_size_allowed)	2647	int size, bool zero_size_allowed)
2616	{	2648	{
@@ -2631,16 +2663,17 @@ static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off, · 14 unmodified lines
2631	regno);	2663	regno);
2632	return -EACCES;	2664	return -EACCES;
2633	}	2665	}
2634	err = __check_packet_access(env, regno, off, size, zero_size_allowed);	2666	err = __check_mem_access(env, regno, off, size, reg->range,
		2667	zero_size_allowed);
2635	if (err) {	2668	if (err) {
2636	verbose(env, "R%d offset is outside of the packet\n", regno);	2669	verbose(env, "R%d offset is outside of the packet\n", regno);
2637	return err;	2670	return err;
2638	}	2671	}
2639		2672
2640	/* __check_packet_access has made sure "off + size - 1" is within u16.	2673	/* __check_mem_access has made sure "off + size - 1" is within u16.
2641	* reg->umax_value can't be bigger than MAX_PACKET_OFF which is 0xffff,	2674	* reg->umax_value can't be bigger than MAX_PACKET_OFF which is 0xffff,
2642	* otherwise find_good_pkt_pointers would have refused to set range info	2675	* otherwise find_good_pkt_pointers would have refused to set range info
2643	* that __check_packet_access would have rejected this pkt access.	2676	* that __check_mem_access would have rejected this pkt access.
2644	* Therefore, "off + reg->umax_value + size - 1" won't overflow u32.	2677	* Therefore, "off + reg->umax_value + size - 1" won't overflow u32.
2645	*/	2678	*/
2646	env->prog->aux->max_pkt_offset =	2679	env->prog->aux->max_pkt_offset =
@@ -3220,6 +3253,16 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 573 unmodified lines
3220	mark_reg_unknown(env, regs, value_regno);	3253	mark_reg_unknown(env, regs, value_regno);
3221	}	3254	}
3222	}	3255	}
		3256	} else if (reg->type == PTR_TO_MEM) {
		3257	if (t == BPF_WRITE && value_regno >= 0 &&
		3258	is_pointer_value(env, value_regno)) {
		3259	verbose(env, "R%d leaks addr into mem\n", value_regno);
		3260	return -EACCES;
		3261	}
		3262	err = check_mem_region_access(env, regno, off, size,
		3263	reg->mem_size, false);
		3264	if (!err && t == BPF_READ && value_regno >= 0)
		3265	mark_reg_unknown(env, regs, value_regno);
3223	} else if (reg->type == PTR_TO_CTX) {	3266	} else if (reg->type == PTR_TO_CTX) {
3224	enum bpf_reg_type reg_type = SCALAR_VALUE;	3267	enum bpf_reg_type reg_type = SCALAR_VALUE;
3225	u32 btf_id = 0;	3268	u32 btf_id = 0;
@@ -3557,6 +3600,10 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 331 unmodified lines
3557	return -EACCES;	3600	return -EACCES;
3558	return check_map_access(env, regno, reg->off, access_size,	3601	return check_map_access(env, regno, reg->off, access_size,
3559	zero_size_allowed);	3602	zero_size_allowed);
		3603	case PTR_TO_MEM:
		3604	return check_mem_region_access(env, regno, reg->off,
		3605	access_size, reg->mem_size,
		3606	zero_size_allowed);
3560	default: /* scalar_value\|ptr_to_stack or invalid ptr */	3607	default: /* scalar_value\|ptr_to_stack or invalid ptr */
3561	return check_stack_boundary(env, regno, access_size,	3608	return check_stack_boundary(env, regno, access_size,
3562	zero_size_allowed, meta);	3609	zero_size_allowed, meta);
@@ -3661,6 +3708,17 @@ static bool arg_type_is_mem_size(enum bpf_arg_type type) · 98 unmodified lines
3661	type == ARG_CONST_SIZE_OR_ZERO;	3708	type == ARG_CONST_SIZE_OR_ZERO;
3662	}	3709	}
3663		3710
		3711	static bool arg_type_is_alloc_mem_ptr(enum bpf_arg_type type)
		3712	{
		3713	return type == ARG_PTR_TO_ALLOC_MEM \|\|
		3714	type == ARG_PTR_TO_ALLOC_MEM_OR_NULL;
		3715	}
		3716
		3717	static bool arg_type_is_alloc_size(enum bpf_arg_type type)
		3718	{
		3719	return type == ARG_CONST_ALLOC_SIZE_OR_ZERO;
		3720	}
		3721
3664	static bool arg_type_is_int_ptr(enum bpf_arg_type type)	3722	static bool arg_type_is_int_ptr(enum bpf_arg_type type)
3665	{	3723	{
3666	return type == ARG_PTR_TO_INT \|\|	3724	return type == ARG_PTR_TO_INT \|\|
@@ -3720,7 +3778,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, · 53 unmodified lines
3720	type != expected_type)	3778	type != expected_type)
3721	goto err_type;	3779	goto err_type;
3722	} else if (arg_type == ARG_CONST_SIZE \|\|	3780	} else if (arg_type == ARG_CONST_SIZE \|\|
3723	arg_type == ARG_CONST_SIZE_OR_ZERO) {	3781	arg_type == ARG_CONST_SIZE_OR_ZERO \|\|
		3782	arg_type == ARG_CONST_ALLOC_SIZE_OR_ZERO) {
3724	expected_type = SCALAR_VALUE;	3783	expected_type = SCALAR_VALUE;
3725	if (type != expected_type)	3784	if (type != expected_type)
3726	goto err_type;	3785	goto err_type;
@@ -3791,13 +3850,29 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, · 64 unmodified lines
3791	* happens during stack boundary checking.	3850	* happens during stack boundary checking.
3792	*/	3851	*/
3793	if (register_is_null(reg) &&	3852	if (register_is_null(reg) &&
3794	arg_type == ARG_PTR_TO_MEM_OR_NULL)	3853	(arg_type == ARG_PTR_TO_MEM_OR_NULL \|\|
		3854	arg_type == ARG_PTR_TO_ALLOC_MEM_OR_NULL))
3795	/* final test in check_stack_boundary() */;	3855	/* final test in check_stack_boundary() */;
3796	else if (!type_is_pkt_pointer(type) &&	3856	else if (!type_is_pkt_pointer(type) &&
3797	type != PTR_TO_MAP_VALUE &&	3857	type != PTR_TO_MAP_VALUE &&
		3858	type != PTR_TO_MEM &&
3798	type != expected_type)	3859	type != expected_type)
3799	goto err_type;	3860	goto err_type;
3800	meta->raw_mode = arg_type == ARG_PTR_TO_UNINIT_MEM;	3861	meta->raw_mode = arg_type == ARG_PTR_TO_UNINIT_MEM;
		3862	} else if (arg_type_is_alloc_mem_ptr(arg_type)) {
		3863	expected_type = PTR_TO_MEM;
		3864	if (register_is_null(reg) &&
		3865	arg_type == ARG_PTR_TO_ALLOC_MEM_OR_NULL)
		3866	/* final test in check_stack_boundary() */;
		3867	else if (type != expected_type)
		3868	goto err_type;
		3869	if (meta->ref_obj_id) {
		3870	verbose(env, "verifier internal error: more than one arg with ref_obj_id R%d %u %u\n",
		3871	regno, reg->ref_obj_id,
		3872	meta->ref_obj_id);
		3873	return -EFAULT;
		3874	}
		3875	meta->ref_obj_id = reg->ref_obj_id;
3801	} else if (arg_type_is_int_ptr(arg_type)) {	3876	} else if (arg_type_is_int_ptr(arg_type)) {
3802	expected_type = PTR_TO_STACK;	3877	expected_type = PTR_TO_STACK;
3803	if (!type_is_pkt_pointer(type) &&	3878	if (!type_is_pkt_pointer(type) &&
@@ -3893,6 +3968,13 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, · 89 unmodified lines
3893	zero_size_allowed, meta);	3968	zero_size_allowed, meta);
3894	if (!err)	3969	if (!err)
3895	err = mark_chain_precision(env, regno);	3970	err = mark_chain_precision(env, regno);
		3971	} else if (arg_type_is_alloc_size(arg_type)) {
		3972	if (!tnum_is_const(reg->var_off)) {
		3973	verbose(env, "R%d unbounded size, use 'var &= const' or 'if (var < const)'\n",
		3974	regno);
		3975	return -EACCES;
		3976	}
		3977	meta->mem_size = reg->var_off.value;
3896	} else if (arg_type_is_int_ptr(arg_type)) {	3978	} else if (arg_type_is_int_ptr(arg_type)) {
3897	int size = int_ptr_type_to_size(arg_type);	3979	int size = int_ptr_type_to_size(arg_type);
3898		3980
@@ -3929,6 +4011,14 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 30 unmodified lines
3929	func_id != BPF_FUNC_xdp_output)	4011	func_id != BPF_FUNC_xdp_output)
3930	goto error;	4012	goto error;
3931	break;	4013	break;
		4014	case BPF_MAP_TYPE_RINGBUF:
		4015	if (func_id != BPF_FUNC_ringbuf_output &&
		4016	func_id != BPF_FUNC_ringbuf_reserve &&
		4017	func_id != BPF_FUNC_ringbuf_submit &&
		4018	func_id != BPF_FUNC_ringbuf_discard &&
		4019	func_id != BPF_FUNC_ringbuf_query)
		4020	goto error;
		4021	break;
3932	case BPF_MAP_TYPE_STACK_TRACE:	4022	case BPF_MAP_TYPE_STACK_TRACE:
3933	if (func_id != BPF_FUNC_get_stackid)	4023	if (func_id != BPF_FUNC_get_stackid)
3934	goto error;	4024	goto error;
@@ -4655,6 +4745,11 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 720 unmodified lines
4655	mark_reg_known_zero(env, regs, BPF_REG_0);	4745	mark_reg_known_zero(env, regs, BPF_REG_0);
4656	regs[BPF_REG_0].type = PTR_TO_TCP_SOCK_OR_NULL;	4746	regs[BPF_REG_0].type = PTR_TO_TCP_SOCK_OR_NULL;
4657	regs[BPF_REG_0].id = ++env->id_gen;	4747	regs[BPF_REG_0].id = ++env->id_gen;
		4748	} else if (fn->ret_type == RET_PTR_TO_ALLOC_MEM_OR_NULL) {
		4749	mark_reg_known_zero(env, regs, BPF_REG_0);
		4750	regs[BPF_REG_0].type = PTR_TO_MEM_OR_NULL;
		4751	regs[BPF_REG_0].id = ++env->id_gen;
		4752	regs[BPF_REG_0].mem_size = meta.mem_size;
4658	} else {	4753	} else {
4659	verbose(env, "unknown return type %d of func %s#%d\n",	4754	verbose(env, "unknown return type %d of func %s#%d\n",
4660	fn->ret_type, func_id_name(func_id), func_id);	4755	fn->ret_type, func_id_name(func_id), func_id);
@@ -6611,6 +6706,8 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state, · 1950 unmodified lines
6611	reg->type = PTR_TO_TCP_SOCK;	6706	reg->type = PTR_TO_TCP_SOCK;
6612	} else if (reg->type == PTR_TO_BTF_ID_OR_NULL) {	6707	} else if (reg->type == PTR_TO_BTF_ID_OR_NULL) {
6613	reg->type = PTR_TO_BTF_ID;	6708	reg->type = PTR_TO_BTF_ID;
		6709	} else if (reg->type == PTR_TO_MEM_OR_NULL) {
		6710	reg->type = PTR_TO_MEM;
6614	}	6711	}
6615	if (is_null) {	6712	if (is_null) {
6616	/* We don't need id and ref_obj_id from this point	6713	/* We don't need id and ref_obj_id from this point

kernel/trace/bpf_trace.c+10−0

@@ -1088,6 +1088,16 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 1087 unmodified lines
1088	return &bpf_perf_event_read_value_proto;	1088	return &bpf_perf_event_read_value_proto;
1089	case BPF_FUNC_get_ns_current_pid_tgid:	1089	case BPF_FUNC_get_ns_current_pid_tgid:
1090	return &bpf_get_ns_current_pid_tgid_proto;	1090	return &bpf_get_ns_current_pid_tgid_proto;
		1091	case BPF_FUNC_ringbuf_output:
		1092	return &bpf_ringbuf_output_proto;
		1093	case BPF_FUNC_ringbuf_reserve:
		1094	return &bpf_ringbuf_reserve_proto;
		1095	case BPF_FUNC_ringbuf_submit:
		1096	return &bpf_ringbuf_submit_proto;
		1097	case BPF_FUNC_ringbuf_discard:
		1098	return &bpf_ringbuf_discard_proto;
		1099	case BPF_FUNC_ringbuf_query:
		1100	return &bpf_ringbuf_query_proto;
1091	default:	1101	default:
1092	return NULL;	1102	return NULL;
1093	}	1103	}

tools/include/uapi/linux/bpf.h+83−1

@@ -147,6 +147,7 @@ enum bpf_map_type { · 146 unmodified lines
147	BPF_MAP_TYPE_SK_STORAGE,	147	BPF_MAP_TYPE_SK_STORAGE,
148	BPF_MAP_TYPE_DEVMAP_HASH,	148	BPF_MAP_TYPE_DEVMAP_HASH,
149	BPF_MAP_TYPE_STRUCT_OPS,	149	BPF_MAP_TYPE_STRUCT_OPS,
		150	BPF_MAP_TYPE_RINGBUF,
150	};	151	};
151		152
152	/* Note that tracing related programs such as	153	/* Note that tracing related programs such as
@@ -3157,6 +3158,59 @@ union bpf_attr { · 3004 unmodified lines
3157	* bpf_sk_cgroup_id\ ().	3158	* bpf_sk_cgroup_id\ ().
3158	* Return	3159	* Return
3159	* The id is returned or 0 in case the id could not be retrieved.	3160	* The id is returned or 0 in case the id could not be retrieved.
		3161	*
		3162	* void bpf_ringbuf_output(void ringbuf, void *data, u64 size, u64 flags)
		3163	* Description
		3164	* Copy size bytes from data into a ring buffer ringbuf.
		3165	* If BPF_RB_NO_WAKEUP is specified in flags, no notification of
		3166	* new data availability is sent.
		3167	* IF BPF_RB_FORCE_WAKEUP is specified in flags, notification of
		3168	* new data availability is sent unconditionally.
		3169	* Return
		3170	* 0, on success;
		3171	* < 0, on error.
		3172	*
		3173	* void bpf_ringbuf_reserve(void ringbuf, u64 size, u64 flags)
		3174	* Description
		3175	* Reserve size bytes of payload in a ring buffer ringbuf.
		3176	* Return
		3177	* Valid pointer with size bytes of memory available; NULL,
		3178	* otherwise.
		3179	*
		3180	* void bpf_ringbuf_submit(void *data, u64 flags)
		3181	* Description
		3182	* Submit reserved ring buffer sample, pointed to by data.
		3183	* If BPF_RB_NO_WAKEUP is specified in flags, no notification of
		3184	* new data availability is sent.
		3185	* IF BPF_RB_FORCE_WAKEUP is specified in flags, notification of
		3186	* new data availability is sent unconditionally.
		3187	* Return
		3188	* Nothing. Always succeeds.
		3189	*
		3190	* void bpf_ringbuf_discard(void *data, u64 flags)
		3191	* Description
		3192	* Discard reserved ring buffer sample, pointed to by data.
		3193	* If BPF_RB_NO_WAKEUP is specified in flags, no notification of
		3194	* new data availability is sent.
		3195	* IF BPF_RB_FORCE_WAKEUP is specified in flags, notification of
		3196	* new data availability is sent unconditionally.
		3197	* Return
		3198	* Nothing. Always succeeds.
		3199	*
		3200	* u64 bpf_ringbuf_query(void *ringbuf, u64 flags)
		3201	* Description
		3202	* Query various characteristics of provided ring buffer. What
		3203	* exactly is queries is determined by flags:
		3204	* - BPF_RB_AVAIL_DATA - amount of data not yet consumed;
		3205	* - BPF_RB_RING_SIZE - the size of ring buffer;
		3206	* - BPF_RB_CONS_POS - consumer position (can wrap around);
		3207	* - BPF_RB_PROD_POS - producer(s) position (can wrap around);
		3208	* Data returned is just a momentary snapshots of actual values
		3209	* and could be inaccurate, so this facility should be used to
		3210	* power heuristics and for reporting, not to make 100% correct
		3211	* calculation.
		3212	* Return
		3213	* Requested value, or 0, if flags are not recognized.
3160	*/	3214	*/
3161	#define __BPF_FUNC_MAPPER(FN) \	3215	#define __BPF_FUNC_MAPPER(FN) \
3162	FN(unspec), \	3216	FN(unspec), \
@@ -3288,7 +3342,12 @@ union bpf_attr { · 125 unmodified lines
3288	FN(seq_printf), \	3342	FN(seq_printf), \
3289	FN(seq_write), \	3343	FN(seq_write), \
3290	FN(sk_cgroup_id), \	3344	FN(sk_cgroup_id), \
3291	FN(sk_ancestor_cgroup_id),	3345	FN(sk_ancestor_cgroup_id), \
		3346	FN(ringbuf_output), \
		3347	FN(ringbuf_reserve), \
		3348	FN(ringbuf_submit), \
		3349	FN(ringbuf_discard), \
		3350	FN(ringbuf_query),
3292		3351
3293	/* integer value in 'imm' field of BPF_CALL instruction selects which helper	3352	/* integer value in 'imm' field of BPF_CALL instruction selects which helper
3294	* function eBPF program intends to call	3353	* function eBPF program intends to call
@@ -3398,6 +3457,29 @@ enum { · 103 unmodified lines
3398	BPF_F_GET_BRANCH_RECORDS_SIZE = (1ULL << 0),	3457	BPF_F_GET_BRANCH_RECORDS_SIZE = (1ULL << 0),
3399	};	3458	};
3400		3459
		3460	/* BPF_FUNC_bpf_ringbuf_commit, BPF_FUNC_bpf_ringbuf_discard, and
		3461	* BPF_FUNC_bpf_ringbuf_output flags.
		3462	*/
		3463	enum {
		3464	BPF_RB_NO_WAKEUP = (1ULL << 0),
		3465	BPF_RB_FORCE_WAKEUP = (1ULL << 1),
		3466	};
		3467
		3468	/* BPF_FUNC_bpf_ringbuf_query flags */
		3469	enum {
		3470	BPF_RB_AVAIL_DATA = 0,
		3471	BPF_RB_RING_SIZE = 1,
		3472	BPF_RB_CONS_POS = 2,
		3473	BPF_RB_PROD_POS = 3,
		3474	};
		3475
		3476	/* BPF ring buffer constants */
		3477	enum {
		3478	BPF_RINGBUF_BUSY_BIT = (1U << 31),
		3479	BPF_RINGBUF_DISCARD_BIT = (1U << 30),
		3480	BPF_RINGBUF_HDR_SZ = 8,
		3481	};
		3482
3401	/* Mode for BPF_FUNC_skb_adjust_room helper. */	3483	/* Mode for BPF_FUNC_skb_adjust_room helper. */
3402	enum bpf_adj_room_mode {	3484	enum bpf_adj_room_mode {
3403	BPF_ADJ_ROOM_NET,	3485	BPF_ADJ_ROOM_NET,

tools/lib/bpf/Build+1−1

@@ -1,3 +1,3 @@
1	libbpf-y := libbpf.o bpf.o nlattr.o btf.o libbpf_errno.o str_error.o \	1	libbpf-y := libbpf.o bpf.o nlattr.o btf.o libbpf_errno.o str_error.o \
2	netlink.o bpf_prog_linfo.o libbpf_probes.o xsk.o hashmap.o \	2	netlink.o bpf_prog_linfo.o libbpf_probes.o xsk.o hashmap.o \
3	btf_dump.o	3	btf_dump.o ringbuf.o

tools/lib/bpf/libbpf.h+21−0

@@ -478,6 +478,27 @@ LIBBPF_API int bpf_get_link_xdp_id(int ifindex, __u32 *prog_id, __u32 flags); · 477 unmodified lines
478	LIBBPF_API int bpf_get_link_xdp_info(int ifindex, struct xdp_link_info *info,	478	LIBBPF_API int bpf_get_link_xdp_info(int ifindex, struct xdp_link_info *info,
479	size_t info_size, __u32 flags);	479	size_t info_size, __u32 flags);
480		480
		481	/* Ring buffer APIs */
		482	struct ring_buffer;
		483
		484	typedef int (ring_buffer_sample_fn)(void ctx, void *data, size_t size);
		485
		486	struct ring_buffer_opts {
		487	size_t sz; /* size of this struct, for forward/backward compatiblity */
		488	};
		489
		490	#define ring_buffer_opts__last_field sz
		491
		492	LIBBPF_API struct ring_buffer *
		493	ring_buffer__new(int map_fd, ring_buffer_sample_fn sample_cb, void *ctx,
		494	const struct ring_buffer_opts *opts);
		495	LIBBPF_API void ring_buffer__free(struct ring_buffer *rb);
		496	LIBBPF_API int ring_buffer__add(struct ring_buffer *rb, int map_fd,
		497	ring_buffer_sample_fn sample_cb, void *ctx);
		498	LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms);
		499	LIBBPF_API int ring_buffer__consume(struct ring_buffer *rb);
		500
		501	/* Perf buffer APIs */
481	struct perf_buffer;	502	struct perf_buffer;
482		503
483	typedef void (perf_buffer_sample_fn)(void ctx, int cpu,	504	typedef void (perf_buffer_sample_fn)(void ctx, int cpu,

tools/lib/bpf/libbpf.map+5−0

@@ -263,4 +263,9 @@ LIBBPF_0.0.9 { · 262 unmodified lines
263	bpf_link_get_next_id;	263	bpf_link_get_next_id;
264	bpf_program__attach_iter;	264	bpf_program__attach_iter;
265	perf_buffer__consume;	265	perf_buffer__consume;
		266	ring_buffer__add;
		267	ring_buffer__consume;
		268	ring_buffer__free;
		269	ring_buffer__new;
		270	ring_buffer__poll;
266	} LIBBPF_0.0.8;	271	} LIBBPF_0.0.8;

tools/lib/bpf/libbpf_probes.c+5−0

 		if (btf_fd < 0)
 			return false;
 		break;
+	case BPF_MAP_TYPE_RINGBUF:
+		key_size = 0;
+		value_size = 0;
+		max_entries = 4096;
+		break;
 	case BPF_MAP_TYPE_UNSPEC:
 	case BPF_MAP_TYPE_HASH:
 	case BPF_MAP_TYPE_ARRAY:

tools/lib/bpf/ringbuf.c+285−0

@@ -0,0 +1,285 @@
	1	// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
	2	/*
	3	* Ring buffer operations.
	4	*
	5	* Copyright (C) 2020 Facebook, Inc.
	6	*/
	7	#include <stdlib.h>
	8	#include <stdio.h>
	9	#include <errno.h>
	10	#include <unistd.h>
	11	#include <linux/err.h>
	12	#include <linux/bpf.h>
	13	#include <asm/barrier.h>
	14	#include <sys/mman.h>
	15	#include <sys/epoll.h>
	16	#include <tools/libc_compat.h>
	17
	18	#include "libbpf.h"
	19	#include "libbpf_internal.h"
	20	#include "bpf.h"
	21
	22	/* make sure libbpf doesn't use kernel-only integer typedefs */
	23	#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
	24
	25	struct ring {
	26	ring_buffer_sample_fn sample_cb;
	27	void *ctx;
	28	void *data;
	29	unsigned long *consumer_pos;
	30	unsigned long *producer_pos;
	31	unsigned long mask;
	32	int map_fd;
	33	};
	34
	35	struct ring_buffer {
	36	struct epoll_event *events;
	37	struct ring *rings;
	38	size_t page_size;
	39	int epoll_fd;
	40	int ring_cnt;
	41	};
	42
	43	static void ringbuf_unmap_ring(struct ring_buffer rb, struct ring r)
	44	{
	45	if (r->consumer_pos) {
	46	munmap(r->consumer_pos, rb->page_size);
	47	r->consumer_pos = NULL;
	48	}
	49	if (r->producer_pos) {
	50	munmap(r->producer_pos, rb->page_size + 2 * (r->mask + 1));
	51	r->producer_pos = NULL;
	52	}
	53	}
	54
	55	/* Add extra RINGBUF maps to this ring buffer manager */
	56	int ring_buffer__add(struct ring_buffer *rb, int map_fd,
	57	ring_buffer_sample_fn sample_cb, void *ctx)
	58	{
	59	struct bpf_map_info info;
	60	__u32 len = sizeof(info);
	61	struct epoll_event *e;
	62	struct ring *r;
	63	void *tmp;
	64	int err;
	65
	66	memset(&info, 0, sizeof(info));
	67
	68	err = bpf_obj_get_info_by_fd(map_fd, &info, &len);
	69	if (err) {
	70	err = -errno;
	71	pr_warn("ringbuf: failed to get map info for fd=%d: %d\n",
	72	map_fd, err);
	73	return err;
	74	}
	75
	76	if (info.type != BPF_MAP_TYPE_RINGBUF) {
	77	pr_warn("ringbuf: map fd=%d is not BPF_MAP_TYPE_RINGBUF\n",
	78	map_fd);
	79	return -EINVAL;
	80	}
	81
	82	tmp = reallocarray(rb->rings, rb->ring_cnt + 1, sizeof(*rb->rings));
	83	if (!tmp)
	84	return -ENOMEM;
	85	rb->rings = tmp;
	86
	87	tmp = reallocarray(rb->events, rb->ring_cnt + 1, sizeof(*rb->events));
	88	if (!tmp)
	89	return -ENOMEM;
	90	rb->events = tmp;
	91
	92	r = &rb->rings[rb->ring_cnt];
	93	memset(r, 0, sizeof(*r));
	94
	95	r->map_fd = map_fd;
	96	r->sample_cb = sample_cb;
	97	r->ctx = ctx;
	98	r->mask = info.max_entries - 1;
	99
	100	/* Map writable consumer page */
	101	tmp = mmap(NULL, rb->page_size, PROT_READ \| PROT_WRITE, MAP_SHARED,
	102	map_fd, 0);
	103	if (tmp == MAP_FAILED) {
	104	err = -errno;
	105	pr_warn("ringbuf: failed to mmap consumer page for map fd=%d: %d\n",
	106	map_fd, err);
	107	return err;
	108	}
	109	r->consumer_pos = tmp;
	110
	111	/* Map read-only producer page and data pages. We map twice as big
	112	* data size to allow simple reading of samples that wrap around the
	113	* end of a ring buffer. See kernel implementation for details.
	114	* */
	115	tmp = mmap(NULL, rb->page_size + 2 * info.max_entries, PROT_READ,
	116	MAP_SHARED, map_fd, rb->page_size);
	117	if (tmp == MAP_FAILED) {
	118	err = -errno;
	119	ringbuf_unmap_ring(rb, r);
	120	pr_warn("ringbuf: failed to mmap data pages for map fd=%d: %d\n",
	121	map_fd, err);
	122	return err;
	123	}
	124	r->producer_pos = tmp;
	125	r->data = tmp + rb->page_size;
	126
	127	e = &rb->events[rb->ring_cnt];
	128	memset(e, 0, sizeof(*e));
	129
	130	e->events = EPOLLIN;
	131	e->data.fd = rb->ring_cnt;
	132	if (epoll_ctl(rb->epoll_fd, EPOLL_CTL_ADD, map_fd, e) < 0) {
	133	err = -errno;
	134	ringbuf_unmap_ring(rb, r);
	135	pr_warn("ringbuf: failed to epoll add map fd=%d: %d\n",
	136	map_fd, err);
	137	return err;
	138	}
	139
	140	rb->ring_cnt++;
	141	return 0;
	142	}
	143
	144	void ring_buffer__free(struct ring_buffer *rb)
	145	{
	146	int i;
	147
	148	if (!rb)
	149	return;
	150
	151	for (i = 0; i < rb->ring_cnt; ++i)
	152	ringbuf_unmap_ring(rb, &rb->rings[i]);
	153	if (rb->epoll_fd >= 0)
	154	close(rb->epoll_fd);
	155
	156	free(rb->events);
	157	free(rb->rings);
	158	free(rb);
	159	}
	160
	161	struct ring_buffer *
	162	ring_buffer__new(int map_fd, ring_buffer_sample_fn sample_cb, void *ctx,
	163	const struct ring_buffer_opts *opts)
	164	{
	165	struct ring_buffer *rb;
	166	int err;
	167
	168	if (!OPTS_VALID(opts, ring_buffer_opts))
	169	return NULL;
	170
	171	rb = calloc(1, sizeof(*rb));
	172	if (!rb)
	173	return NULL;
	174
	175	rb->page_size = getpagesize();
	176
	177	rb->epoll_fd = epoll_create1(EPOLL_CLOEXEC);
	178	if (rb->epoll_fd < 0) {
	179	err = -errno;
	180	pr_warn("ringbuf: failed to create epoll instance: %d\n", err);
	181	goto err_out;
	182	}
	183
	184	err = ring_buffer__add(rb, map_fd, sample_cb, ctx);
	185	if (err)
	186	goto err_out;
	187
	188	return rb;
	189
	190	err_out:
	191	ring_buffer__free(rb);
	192	return NULL;
	193	}
	194
	195	static inline int roundup_len(__u32 len)
	196	{
	197	/* clear out top 2 bits (discard and busy, if set) */
	198	len <<= 2;
	199	len >>= 2;
	200	/* add length prefix */
	201	len += BPF_RINGBUF_HDR_SZ;
	202	/* round up to 8 byte alignment */
	203	return (len + 7) / 8 * 8;
	204	}
	205
	206	static int ringbuf_process_ring(struct ring* r)
	207	{
	208	int *len_ptr, len, err, cnt = 0;
	209	unsigned long cons_pos, prod_pos;
	210	bool got_new_data;
	211	void *sample;
	212
	213	cons_pos = smp_load_acquire(r->consumer_pos);
	214	do {
	215	got_new_data = false;
	216	prod_pos = smp_load_acquire(r->producer_pos);
	217	while (cons_pos < prod_pos) {
	218	len_ptr = r->data + (cons_pos & r->mask);
	219	len = smp_load_acquire(len_ptr);
	220
	221	/* sample not committed yet, bail out for now */
	222	if (len & BPF_RINGBUF_BUSY_BIT)
	223	goto done;
	224
	225	got_new_data = true;
	226	cons_pos += roundup_len(len);
	227
	228	if ((len & BPF_RINGBUF_DISCARD_BIT) == 0) {
	229	sample = (void *)len_ptr + BPF_RINGBUF_HDR_SZ;
	230	err = r->sample_cb(r->ctx, sample, len);
	231	if (err) {
	232	/* update consumer pos and bail out */
	233	smp_store_release(r->consumer_pos,
	234	cons_pos);
	235	return err;
	236	}
	237	cnt++;
	238	}
	239
	240	smp_store_release(r->consumer_pos, cons_pos);
	241	}
	242	} while (got_new_data);
	243	done:
	244	return cnt;
	245	}
	246
	247	/* Consume available ring buffer(s) data without event polling.
	248	* Returns number of records consumed across all registered ring buffers, or
	249	* negative number if any of the callbacks return error.
	250	*/
	251	int ring_buffer__consume(struct ring_buffer *rb)
	252	{
	253	int i, err, res = 0;
	254
	255	for (i = 0; i < rb->ring_cnt; i++) {
	256	struct ring *ring = &rb->rings[i];
	257
	258	err = ringbuf_process_ring(ring);
	259	if (err < 0)
	260	return err;
	261	res += err;
	262	}
	263	return res;
	264	}
	265
	266	/* Poll for available data and consume records, if any are available.
	267	* Returns number of records consumed, or negative number, if any of the
	268	* registered callbacks returned error.
	269	*/
	270	int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms)
	271	{
	272	int i, cnt, err, res = 0;
	273
	274	cnt = epoll_wait(rb->epoll_fd, rb->events, rb->ring_cnt, timeout_ms);
	275	for (i = 0; i < cnt; i++) {
	276	__u32 ring_id = rb->events[i].data.fd;
	277	struct ring *ring = &rb->rings[ring_id];
	278
	279	err = ringbuf_process_ring(ring);
	280	if (err < 0)
	281	return err;
	282	res += cnt;
	283	}
	284	return cnt < 0 ? -errno : res;
	285	}

tools/testing/selftests/bpf/Makefile+4−1

 	$(CC) $(CFLAGS) -c $(filter %.c,$^) $(LDLIBS) -o $@
 $(OUTPUT)/bench_rename.o: $(OUTPUT)/test_overhead.skel.h
 $(OUTPUT)/bench_trigger.o: $(OUTPUT)/trigger_bench.skel.h
+$(OUTPUT)/bench_ringbufs.o: $(OUTPUT)/ringbuf_bench.skel.h \
+			    $(OUTPUT)/perfbuf_bench.skel.h
 $(OUTPUT)/bench.o: bench.h testing_helpers.h
 $(OUTPUT)/bench: LDLIBS += -lm
 $(OUTPUT)/bench: $(OUTPUT)/bench.o $(OUTPUT)/testing_helpers.o \
 		 $(OUTPUT)/bench_count.o \
 		 $(OUTPUT)/bench_rename.o \
-		 $(OUTPUT)/bench_trigger.o
+		 $(OUTPUT)/bench_trigger.o \
+		 $(OUTPUT)/bench_ringbufs.o
 	$(call msg,BINARY,,$@)
 	$(CC) $(LDFLAGS) -o $@ $(filter %.a %.o,$^) $(LDLIBS)

tools/testing/selftests/bpf/bench.c+16−0

 	{},
 };
+extern struct argp bench_ringbufs_argp;
+static const struct argp_child bench_parsers[] = {
+	{ &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 },
+	{},
+};
 static error_t parse_arg(int key, char *arg, struct argp_state *state)
 {
 	static int pos_args;
 		.options = opts,
 		.parser = parse_arg,
 		.doc = argp_program_doc,
+		.children = bench_parsers,
 	};
 	if (argp_parse(&argp, argc, argv, 0, NULL, NULL))
 		exit(1);
 extern const struct bench bench_trig_kprobe;
 extern const struct bench bench_trig_fentry;
 extern const struct bench bench_trig_fmodret;
+extern const struct bench bench_rb_libbpf;
+extern const struct bench bench_rb_custom;
+extern const struct bench bench_pb_libbpf;
+extern const struct bench bench_pb_custom;
 static const struct bench *benchs[] = {
 	&bench_count_global,
 	&bench_trig_kprobe,
 	&bench_trig_fentry,
 	&bench_trig_fmodret,
+	&bench_rb_libbpf,
+	&bench_rb_custom,
+	&bench_pb_libbpf,
+	&bench_pb_custom,
 };
 static void setup_benchmark()

tools/testing/selftests/bpf/benchs/bench_ringbufs.c+566−0

@@ -0,0 +1,566 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	/* Copyright (c) 2020 Facebook */
	3	#include <asm/barrier.h>
	4	#include <linux/perf_event.h>
	5	#include <linux/ring_buffer.h>
	6	#include <sys/epoll.h>
	7	#include <sys/mman.h>
	8	#include <argp.h>
	9	#include <stdlib.h>
	10	#include "bench.h"
	11	#include "ringbuf_bench.skel.h"
	12	#include "perfbuf_bench.skel.h"
	13
	14	static struct {
	15	bool back2back;
	16	int batch_cnt;
	17	bool sampled;
	18	int sample_rate;
	19	int ringbuf_sz; /* per-ringbuf, in bytes */
	20	bool ringbuf_use_output; /* use slower output API */
	21	int perfbuf_sz; /* per-CPU size, in pages */
	22	} args = {
	23	.back2back = false,
	24	.batch_cnt = 500,
	25	.sampled = false,
	26	.sample_rate = 500,
	27	.ringbuf_sz = 512 * 1024,
	28	.ringbuf_use_output = false,
	29	.perfbuf_sz = 128,
	30	};
	31
	32	enum {
	33	ARG_RB_BACK2BACK = 2000,
	34	ARG_RB_USE_OUTPUT = 2001,
	35	ARG_RB_BATCH_CNT = 2002,
	36	ARG_RB_SAMPLED = 2003,
	37	ARG_RB_SAMPLE_RATE = 2004,
	38	};
	39
	40	static const struct argp_option opts[] = {
	41	{ "rb-b2b", ARG_RB_BACK2BACK, NULL, 0, "Back-to-back mode"},
	42	{ "rb-use-output", ARG_RB_USE_OUTPUT, NULL, 0, "Use bpf_ringbuf_output() instead of bpf_ringbuf_reserve()"},
	43	{ "rb-batch-cnt", ARG_RB_BATCH_CNT, "CNT", 0, "Set BPF-side record batch count"},
	44	{ "rb-sampled", ARG_RB_SAMPLED, NULL, 0, "Notification sampling"},
	45	{ "rb-sample-rate", ARG_RB_SAMPLE_RATE, "RATE", 0, "Notification sample rate"},
	46	{},
	47	};
	48
	49	static error_t parse_arg(int key, char arg, struct argp_state state)
	50	{
	51	switch (key) {
	52	case ARG_RB_BACK2BACK:
	53	args.back2back = true;
	54	break;
	55	case ARG_RB_USE_OUTPUT:
	56	args.ringbuf_use_output = true;
	57	break;
	58	case ARG_RB_BATCH_CNT:
	59	args.batch_cnt = strtol(arg, NULL, 10);
	60	if (args.batch_cnt < 0) {
	61	fprintf(stderr, "Invalid batch count.");
	62	argp_usage(state);
	63	}
	64	break;
	65	case ARG_RB_SAMPLED:
	66	args.sampled = true;
	67	break;
	68	case ARG_RB_SAMPLE_RATE:
	69	args.sample_rate = strtol(arg, NULL, 10);
	70	if (args.sample_rate < 0) {
	71	fprintf(stderr, "Invalid perfbuf sample rate.");
	72	argp_usage(state);
	73	}
	74	break;
	75	default:
	76	return ARGP_ERR_UNKNOWN;
	77	}
	78	return 0;
	79	}
	80
	81	/* exported into benchmark runner */
	82	const struct argp bench_ringbufs_argp = {
	83	.options = opts,
	84	.parser = parse_arg,
	85	};
	86
	87	/* RINGBUF-LIBBPF benchmark */
	88
	89	static struct counter buf_hits;
	90
	91	static inline void bufs_trigger_batch()
	92	{
	93	(void)syscall(__NR_getpgid);
	94	}
	95
	96	static void bufs_validate()
	97	{
	98	if (env.consumer_cnt != 1) {
	99	fprintf(stderr, "rb-libbpf benchmark doesn't support multi-consumer!\n");
	100	exit(1);
	101	}
	102
	103	if (args.back2back && env.producer_cnt > 1) {
	104	fprintf(stderr, "back-to-back mode makes sense only for single-producer case!\n");
	105	exit(1);
	106	}
	107	}
	108
	109	static void bufs_sample_producer(void input)
	110	{
	111	if (args.back2back) {
	112	/* initial batch to get everything started */
	113	bufs_trigger_batch();
	114	return NULL;
	115	}
	116
	117	while (true)
	118	bufs_trigger_batch();
	119	return NULL;
	120	}
	121
	122	static struct ringbuf_libbpf_ctx {
	123	struct ringbuf_bench *skel;
	124	struct ring_buffer *ringbuf;
	125	} ringbuf_libbpf_ctx;
	126
	127	static void ringbuf_libbpf_measure(struct bench_res *res)
	128	{
	129	struct ringbuf_libbpf_ctx *ctx = &ringbuf_libbpf_ctx;
	130
	131	res->hits = atomic_swap(&buf_hits.value, 0);
	132	res->drops = atomic_swap(&ctx->skel->bss->dropped, 0);
	133	}
	134
	135	static struct ringbuf_bench *ringbuf_setup_skeleton()
	136	{
	137	struct ringbuf_bench *skel;
	138
	139	setup_libbpf();
	140
	141	skel = ringbuf_bench__open();
	142	if (!skel) {
	143	fprintf(stderr, "failed to open skeleton\n");
	144	exit(1);
	145	}
	146
	147	skel->rodata->batch_cnt = args.batch_cnt;
	148	skel->rodata->use_output = args.ringbuf_use_output ? 1 : 0;
	149
	150	if (args.sampled)
	151	/* record data + header take 16 bytes */
	152	skel->rodata->wakeup_data_size = args.sample_rate * 16;
	153
	154	bpf_map__resize(skel->maps.ringbuf, args.ringbuf_sz);
	155
	156	if (ringbuf_bench__load(skel)) {
	157	fprintf(stderr, "failed to load skeleton\n");
	158	exit(1);
	159	}
	160
	161	return skel;
	162	}
	163
	164	static int buf_process_sample(void ctx, void data, size_t len)
	165	{
	166	atomic_inc(&buf_hits.value);
	167	return 0;
	168	}
	169
	170	static void ringbuf_libbpf_setup()
	171	{
	172	struct ringbuf_libbpf_ctx *ctx = &ringbuf_libbpf_ctx;
	173	struct bpf_link *link;
	174
	175	ctx->skel = ringbuf_setup_skeleton();
	176	ctx->ringbuf = ring_buffer__new(bpf_map__fd(ctx->skel->maps.ringbuf),
	177	buf_process_sample, NULL, NULL);
	178	if (!ctx->ringbuf) {
	179	fprintf(stderr, "failed to create ringbuf\n");
	180	exit(1);
	181	}
	182
	183	link = bpf_program__attach(ctx->skel->progs.bench_ringbuf);
	184	if (IS_ERR(link)) {
	185	fprintf(stderr, "failed to attach program!\n");
	186	exit(1);
	187	}
	188	}
	189
	190	static void ringbuf_libbpf_consumer(void input)
	191	{
	192	struct ringbuf_libbpf_ctx *ctx = &ringbuf_libbpf_ctx;
	193
	194	while (ring_buffer__poll(ctx->ringbuf, -1) >= 0) {
	195	if (args.back2back)
	196	bufs_trigger_batch();
	197	}
	198	fprintf(stderr, "ringbuf polling failed!\n");
	199	return NULL;
	200	}
	201
	202	/* RINGBUF-CUSTOM benchmark */
	203	struct ringbuf_custom {
	204	__u64 *consumer_pos;
	205	__u64 *producer_pos;
	206	__u64 mask;
	207	void *data;
	208	int map_fd;
	209	};
	210
	211	static struct ringbuf_custom_ctx {
	212	struct ringbuf_bench *skel;
	213	struct ringbuf_custom ringbuf;
	214	int epoll_fd;
	215	struct epoll_event event;
	216	} ringbuf_custom_ctx;
	217
	218	static void ringbuf_custom_measure(struct bench_res *res)
	219	{
	220	struct ringbuf_custom_ctx *ctx = &ringbuf_custom_ctx;
	221
	222	res->hits = atomic_swap(&buf_hits.value, 0);
	223	res->drops = atomic_swap(&ctx->skel->bss->dropped, 0);
	224	}
	225
	226	static void ringbuf_custom_setup()
	227	{
	228	struct ringbuf_custom_ctx *ctx = &ringbuf_custom_ctx;
	229	const size_t page_size = getpagesize();
	230	struct bpf_link *link;
	231	struct ringbuf_custom *r;
	232	void *tmp;
	233	int err;
	234
	235	ctx->skel = ringbuf_setup_skeleton();
	236
	237	ctx->epoll_fd = epoll_create1(EPOLL_CLOEXEC);
	238	if (ctx->epoll_fd < 0) {
	239	fprintf(stderr, "failed to create epoll fd: %d\n", -errno);
	240	exit(1);
	241	}
	242
	243	r = &ctx->ringbuf;
	244	r->map_fd = bpf_map__fd(ctx->skel->maps.ringbuf);
	245	r->mask = args.ringbuf_sz - 1;
	246
	247	/* Map writable consumer page */
	248	tmp = mmap(NULL, page_size, PROT_READ \| PROT_WRITE, MAP_SHARED,
	249	r->map_fd, 0);
	250	if (tmp == MAP_FAILED) {
	251	fprintf(stderr, "failed to mmap consumer page: %d\n", -errno);
	252	exit(1);
	253	}
	254	r->consumer_pos = tmp;
	255
	256	/* Map read-only producer page and data pages. */
	257	tmp = mmap(NULL, page_size + 2 * args.ringbuf_sz, PROT_READ, MAP_SHARED,
	258	r->map_fd, page_size);
	259	if (tmp == MAP_FAILED) {
	260	fprintf(stderr, "failed to mmap data pages: %d\n", -errno);
	261	exit(1);
	262	}
	263	r->producer_pos = tmp;
	264	r->data = tmp + page_size;
	265
	266	ctx->event.events = EPOLLIN;
	267	err = epoll_ctl(ctx->epoll_fd, EPOLL_CTL_ADD, r->map_fd, &ctx->event);
	268	if (err < 0) {
	269	fprintf(stderr, "failed to epoll add ringbuf: %d\n", -errno);
	270	exit(1);
	271	}
	272
	273	link = bpf_program__attach(ctx->skel->progs.bench_ringbuf);
	274	if (IS_ERR(link)) {
	275	fprintf(stderr, "failed to attach program\n");
	276	exit(1);
	277	}
	278	}
	279
	280	#define RINGBUF_BUSY_BIT (1 << 31)
	281	#define RINGBUF_DISCARD_BIT (1 << 30)
	282	#define RINGBUF_META_LEN 8
	283
	284	static inline int roundup_len(__u32 len)
	285	{
	286	/* clear out top 2 bits */
	287	len <<= 2;
	288	len >>= 2;
	289	/* add length prefix */
	290	len += RINGBUF_META_LEN;
	291	/* round up to 8 byte alignment */
	292	return (len + 7) / 8 * 8;
	293	}
	294
	295	static void ringbuf_custom_process_ring(struct ringbuf_custom *r)
	296	{
	297	unsigned long cons_pos, prod_pos;
	298	int *len_ptr, len;
	299	bool got_new_data;
	300
	301	cons_pos = smp_load_acquire(r->consumer_pos);
	302	while (true) {
	303	got_new_data = false;
	304	prod_pos = smp_load_acquire(r->producer_pos);
	305	while (cons_pos < prod_pos) {
	306	len_ptr = r->data + (cons_pos & r->mask);
	307	len = smp_load_acquire(len_ptr);
	308
	309	/* sample not committed yet, bail out for now */
	310	if (len & RINGBUF_BUSY_BIT)
	311	return;
	312
	313	got_new_data = true;
	314	cons_pos += roundup_len(len);
	315
	316	atomic_inc(&buf_hits.value);
	317	}
	318	if (got_new_data)
	319	smp_store_release(r->consumer_pos, cons_pos);
	320	else
	321	break;
	322	};
	323	}
	324
	325	static void ringbuf_custom_consumer(void input)
	326	{
	327	struct ringbuf_custom_ctx *ctx = &ringbuf_custom_ctx;
	328	int cnt;
	329
	330	do {
	331	if (args.back2back)
	332	bufs_trigger_batch();
	333	cnt = epoll_wait(ctx->epoll_fd, &ctx->event, 1, -1);
	334	if (cnt > 0)
	335	ringbuf_custom_process_ring(&ctx->ringbuf);
	336	} while (cnt >= 0);
	337	fprintf(stderr, "ringbuf polling failed!\n");
	338	return 0;
	339	}
	340
	341	/* PERFBUF-LIBBPF benchmark */
	342	static struct perfbuf_libbpf_ctx {
	343	struct perfbuf_bench *skel;
	344	struct perf_buffer *perfbuf;
	345	} perfbuf_libbpf_ctx;
	346
	347	static void perfbuf_measure(struct bench_res *res)
	348	{
	349	struct perfbuf_libbpf_ctx *ctx = &perfbuf_libbpf_ctx;
	350
	351	res->hits = atomic_swap(&buf_hits.value, 0);
	352	res->drops = atomic_swap(&ctx->skel->bss->dropped, 0);
	353	}
	354
	355	static struct perfbuf_bench *perfbuf_setup_skeleton()
	356	{
	357	struct perfbuf_bench *skel;
	358
	359	setup_libbpf();
	360
	361	skel = perfbuf_bench__open();
	362	if (!skel) {
	363	fprintf(stderr, "failed to open skeleton\n");
	364	exit(1);
	365	}
	366
	367	skel->rodata->batch_cnt = args.batch_cnt;
	368
	369	if (perfbuf_bench__load(skel)) {
	370	fprintf(stderr, "failed to load skeleton\n");
	371	exit(1);
	372	}
	373
	374	return skel;
	375	}
	376
	377	static enum bpf_perf_event_ret
	378	perfbuf_process_sample_raw(void *input_ctx, int cpu,
	379	struct perf_event_header *e)
	380	{
	381	switch (e->type) {
	382	case PERF_RECORD_SAMPLE:
	383	atomic_inc(&buf_hits.value);
	384	break;
	385	case PERF_RECORD_LOST:
	386	break;
	387	default:
	388	return LIBBPF_PERF_EVENT_ERROR;
	389	}
	390	return LIBBPF_PERF_EVENT_CONT;
	391	}
	392
	393	static void perfbuf_libbpf_setup()
	394	{
	395	struct perfbuf_libbpf_ctx *ctx = &perfbuf_libbpf_ctx;
	396	struct perf_event_attr attr;
	397	struct perf_buffer_raw_opts pb_opts = {
	398	.event_cb = perfbuf_process_sample_raw,
	399	.ctx = (void *)(long)0,
	400	.attr = &attr,
	401	};
	402	struct bpf_link *link;
	403
	404	ctx->skel = perfbuf_setup_skeleton();
	405
	406	memset(&attr, 0, sizeof(attr));
	407	attr.config = PERF_COUNT_SW_BPF_OUTPUT,
	408	attr.type = PERF_TYPE_SOFTWARE;
	409	attr.sample_type = PERF_SAMPLE_RAW;
	410	/* notify only every Nth sample */
	411	if (args.sampled) {
	412	attr.sample_period = args.sample_rate;
	413	attr.wakeup_events = args.sample_rate;
	414	} else {
	415	attr.sample_period = 1;
	416	attr.wakeup_events = 1;
	417	}
	418
	419	if (args.sample_rate > args.batch_cnt) {
	420	fprintf(stderr, "sample rate %d is too high for given batch count %d\n",
	421	args.sample_rate, args.batch_cnt);
	422	exit(1);
	423	}
	424
	425	ctx->perfbuf = perf_buffer__new_raw(bpf_map__fd(ctx->skel->maps.perfbuf),
	426	args.perfbuf_sz, &pb_opts);
	427	if (!ctx->perfbuf) {
	428	fprintf(stderr, "failed to create perfbuf\n");
	429	exit(1);
	430	}
	431
	432	link = bpf_program__attach(ctx->skel->progs.bench_perfbuf);
	433	if (IS_ERR(link)) {
	434	fprintf(stderr, "failed to attach program\n");
	435	exit(1);
	436	}
	437	}
	438
	439	static void perfbuf_libbpf_consumer(void input)
	440	{
	441	struct perfbuf_libbpf_ctx *ctx = &perfbuf_libbpf_ctx;
	442
	443	while (perf_buffer__poll(ctx->perfbuf, -1) >= 0) {
	444	if (args.back2back)
	445	bufs_trigger_batch();
	446	}
	447	fprintf(stderr, "perfbuf polling failed!\n");
	448	return NULL;
	449	}
	450
	451	/* PERFBUF-CUSTOM benchmark */
	452
	453	/* copies of internal libbpf definitions */
	454	struct perf_cpu_buf {
	455	struct perf_buffer *pb;
	456	void base; / mmap()'ed memory */
	457	void buf; / for reconstructing segmented data */
	458	size_t buf_size;
	459	int fd;
	460	int cpu;
	461	int map_key;
	462	};
	463
	464	struct perf_buffer {
	465	perf_buffer_event_fn event_cb;
	466	perf_buffer_sample_fn sample_cb;
	467	perf_buffer_lost_fn lost_cb;
	468	void ctx; / passed into callbacks */
	469
	470	size_t page_size;
	471	size_t mmap_size;
	472	struct perf_cpu_buf **cpu_bufs;
	473	struct epoll_event *events;
	474	int cpu_cnt; /* number of allocated CPU buffers */
	475	int epoll_fd; /* perf event FD */
	476	int map_fd; /* BPF_MAP_TYPE_PERF_EVENT_ARRAY BPF map FD */
	477	};
	478
	479	static void perfbuf_custom_consumer(void input)
	480	{
	481	struct perfbuf_libbpf_ctx *ctx = &perfbuf_libbpf_ctx;
	482	struct perf_buffer *pb = ctx->perfbuf;
	483	struct perf_cpu_buf *cpu_buf;
	484	struct perf_event_mmap_page *header;
	485	size_t mmap_mask = pb->mmap_size - 1;
	486	struct perf_event_header *ehdr;
	487	__u64 data_head, data_tail;
	488	size_t ehdr_size;
	489	void *base;
	490	int i, cnt;
	491
	492	while (true) {
	493	if (args.back2back)
	494	bufs_trigger_batch();
	495	cnt = epoll_wait(pb->epoll_fd, pb->events, pb->cpu_cnt, -1);
	496	if (cnt <= 0) {
	497	fprintf(stderr, "perf epoll failed: %d\n", -errno);
	498	exit(1);
	499	}
	500
	501	for (i = 0; i < cnt; ++i) {
	502	cpu_buf = pb->events[i].data.ptr;
	503	header = cpu_buf->base;
	504	base = ((void *)header) + pb->page_size;
	505
	506	data_head = ring_buffer_read_head(header);
	507	data_tail = header->data_tail;
	508	while (data_head != data_tail) {
	509	ehdr = base + (data_tail & mmap_mask);
	510	ehdr_size = ehdr->size;
	511
	512	if (ehdr->type == PERF_RECORD_SAMPLE)
	513	atomic_inc(&buf_hits.value);
	514
	515	data_tail += ehdr_size;
	516	}
	517	ring_buffer_write_tail(header, data_tail);
	518	}
	519	}
	520	return NULL;
	521	}
	522
	523	const struct bench bench_rb_libbpf = {
	524	.name = "rb-libbpf",
	525	.validate = bufs_validate,
	526	.setup = ringbuf_libbpf_setup,
	527	.producer_thread = bufs_sample_producer,
	528	.consumer_thread = ringbuf_libbpf_consumer,
	529	.measure = ringbuf_libbpf_measure,
	530	.report_progress = hits_drops_report_progress,
	531	.report_final = hits_drops_report_final,
	532	};
	533
	534	const struct bench bench_rb_custom = {
	535	.name = "rb-custom",
	536	.validate = bufs_validate,
	537	.setup = ringbuf_custom_setup,
	538	.producer_thread = bufs_sample_producer,
	539	.consumer_thread = ringbuf_custom_consumer,
	540	.measure = ringbuf_custom_measure,
	541	.report_progress = hits_drops_report_progress,
	542	.report_final = hits_drops_report_final,
	543	};
	544
	545	const struct bench bench_pb_libbpf = {
	546	.name = "pb-libbpf",
	547	.validate = bufs_validate,
	548	.setup = perfbuf_libbpf_setup,
	549	.producer_thread = bufs_sample_producer,
	550	.consumer_thread = perfbuf_libbpf_consumer,
	551	.measure = perfbuf_measure,
	552	.report_progress = hits_drops_report_progress,
	553	.report_final = hits_drops_report_final,
	554	};
	555
	556	const struct bench bench_pb_custom = {
	557	.name = "pb-custom",
	558	.validate = bufs_validate,
	559	.setup = perfbuf_libbpf_setup,
	560	.producer_thread = bufs_sample_producer,
	561	.consumer_thread = perfbuf_custom_consumer,
	562	.measure = perfbuf_measure,
	563	.report_progress = hits_drops_report_progress,
	564	.report_final = hits_drops_report_final,
	565	};
	566

tools/testing/selftests/bpf/benchs/run_bench_ringbufs.sh+75−0

@@ -0,0 +1,75 @@
	1	#!/bin/bash
	2
	3	set -eufo pipefail
	4
	5	RUN_BENCH="sudo ./bench -w3 -d10 -a"
	6
	7	function hits()
	8	{
	9	echo "$" \| sed -E "s/.hits\s+([0-9]+\.[0-9]+ ± [0-9]+\.[0-9]+M\/s).*/\1/"
	10	}
	11
	12	function drops()
	13	{
	14	echo "$" \| sed -E "s/.drops\s+([0-9]+\.[0-9]+ ± [0-9]+\.[0-9]+M\/s).*/\1/"
	15	}
	16
	17	function header()
	18	{
	19	local len=${#1}
	20
	21	printf "\n%s\n" "$1"
	22	for i in $(seq 1 $len); do printf '='; done
	23	printf '\n'
	24	}
	25
	26	function summarize()
	27	{
	28	bench="$1"
	29	summary=$(echo $2 \| tail -n1)
	30	printf "%-20s %s (drops %s)\n" "$bench" "$(hits $summary)" "$(drops $summary)"
	31	}
	32
	33	header "Single-producer, parallel producer"
	34	for b in rb-libbpf rb-custom pb-libbpf pb-custom; do
	35	summarize $b "$($RUN_BENCH $b)"
	36	done
	37
	38	header "Single-producer, parallel producer, sampled notification"
	39	for b in rb-libbpf rb-custom pb-libbpf pb-custom; do
	40	summarize $b "$($RUN_BENCH --rb-sampled $b)"
	41	done
	42
	43	header "Single-producer, back-to-back mode"
	44	for b in rb-libbpf rb-custom pb-libbpf pb-custom; do
	45	summarize $b "$($RUN_BENCH --rb-b2b $b)"
	46	summarize $b-sampled "$($RUN_BENCH --rb-sampled --rb-b2b $b)"
	47	done
	48
	49	header "Ringbuf back-to-back, effect of sample rate"
	50	for b in 1 5 10 25 50 100 250 500 1000 2000 3000; do
	51	summarize "rb-sampled-$b" "$($RUN_BENCH --rb-b2b --rb-batch-cnt $b --rb-sampled --rb-sample-rate $b rb-custom)"
	52	done
	53	header "Perfbuf back-to-back, effect of sample rate"
	54	for b in 1 5 10 25 50 100 250 500 1000 2000 3000; do
	55	summarize "pb-sampled-$b" "$($RUN_BENCH --rb-b2b --rb-batch-cnt $b --rb-sampled --rb-sample-rate $b pb-custom)"
	56	done
	57
	58	header "Ringbuf back-to-back, reserve+commit vs output"
	59	summarize "reserve" "$($RUN_BENCH --rb-b2b rb-custom)"
	60	summarize "output" "$($RUN_BENCH --rb-b2b --rb-use-output rb-custom)"
	61
	62	header "Ringbuf sampled, reserve+commit vs output"
	63	summarize "reserve-sampled" "$($RUN_BENCH --rb-sampled rb-custom)"
	64	summarize "output-sampled" "$($RUN_BENCH --rb-sampled --rb-use-output rb-custom)"
	65
	66	header "Single-producer, consumer/producer competing on the same CPU, low batch count"
	67	for b in rb-libbpf rb-custom pb-libbpf pb-custom; do
	68	summarize $b "$($RUN_BENCH --rb-batch-cnt 1 --rb-sample-rate 1 --prod-affinity 0 --cons-affinity 0 $b)"
	69	done
	70
	71	header "Ringbuf, multi-producer contention"
	72	for b in 1 2 3 4 8 12 16 20 24 28 32 36 40 44 48 52; do
	73	summarize "rb-libbpf nr_prod $b" "$($RUN_BENCH -p$b --rb-batch-cnt 50 rb-libbpf)"
	74	done
	75

tools/testing/selftests/bpf/prog_tests/ringbuf.c+211−0

@@ -0,0 +1,211 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	#define _GNU_SOURCE
	3	#include <linux/compiler.h>
	4	#include <asm/barrier.h>
	5	#include <test_progs.h>
	6	#include <sys/mman.h>
	7	#include <sys/epoll.h>
	8	#include <time.h>
	9	#include <sched.h>
	10	#include <signal.h>
	11	#include <pthread.h>
	12	#include <sys/sysinfo.h>
	13	#include <linux/perf_event.h>
	14	#include <linux/ring_buffer.h>
	15	#include "test_ringbuf.skel.h"
	16
	17	#define EDONE 7777
	18
	19	static int duration = 0;
	20
	21	struct sample {
	22	int pid;
	23	int seq;
	24	long value;
	25	char comm[16];
	26	};
	27
	28	static int sample_cnt;
	29
	30	static int process_sample(void ctx, void data, size_t len)
	31	{
	32	struct sample *s = data;
	33
	34	sample_cnt++;
	35
	36	switch (s->seq) {
	37	case 0:
	38	CHECK(s->value != 333, "sample1_value", "exp %ld, got %ld\n",
	39	333L, s->value);
	40	return 0;
	41	case 1:
	42	CHECK(s->value != 777, "sample2_value", "exp %ld, got %ld\n",
	43	777L, s->value);
	44	return -EDONE;
	45	default:
	46	/* we don't care about the rest */
	47	return 0;
	48	}
	49	}
	50
	51	static struct test_ringbuf *skel;
	52	static struct ring_buffer *ringbuf;
	53
	54	static void trigger_samples()
	55	{
	56	skel->bss->dropped = 0;
	57	skel->bss->total = 0;
	58	skel->bss->discarded = 0;
	59
	60	/* trigger exactly two samples */
	61	skel->bss->value = 333;
	62	syscall(__NR_getpgid);
	63	skel->bss->value = 777;
	64	syscall(__NR_getpgid);
	65	}
	66
	67	static void poll_thread(void input)
	68	{
	69	long timeout = (long)input;
	70
	71	return (void *)(long)ring_buffer__poll(ringbuf, timeout);
	72	}
	73
	74	void test_ringbuf(void)
	75	{
	76	const size_t rec_sz = BPF_RINGBUF_HDR_SZ + sizeof(struct sample);
	77	pthread_t thread;
	78	long bg_ret = -1;
	79	int err;
	80
	81	skel = test_ringbuf__open_and_load();
	82	if (CHECK(!skel, "skel_open_load", "skeleton open&load failed\n"))
	83	return;
	84
	85	/* only trigger BPF program for current process */
	86	skel->bss->pid = getpid();
	87
	88	ringbuf = ring_buffer__new(bpf_map__fd(skel->maps.ringbuf),
	89	process_sample, NULL, NULL);
	90	if (CHECK(!ringbuf, "ringbuf_create", "failed to create ringbuf\n"))
	91	goto cleanup;
	92
	93	err = test_ringbuf__attach(skel);
	94	if (CHECK(err, "skel_attach", "skeleton attachment failed: %d\n", err))
	95	goto cleanup;
	96
	97	trigger_samples();
	98
	99	/* 2 submitted + 1 discarded records */
	100	CHECK(skel->bss->avail_data != 3 * rec_sz,
	101	"err_avail_size", "exp %ld, got %ld\n",
	102	3L * rec_sz, skel->bss->avail_data);
	103	CHECK(skel->bss->ring_size != 4096,
	104	"err_ring_size", "exp %ld, got %ld\n",
	105	4096L, skel->bss->ring_size);
	106	CHECK(skel->bss->cons_pos != 0,
	107	"err_cons_pos", "exp %ld, got %ld\n",
	108	0L, skel->bss->cons_pos);
	109	CHECK(skel->bss->prod_pos != 3 * rec_sz,
	110	"err_prod_pos", "exp %ld, got %ld\n",
	111	3L * rec_sz, skel->bss->prod_pos);
	112
	113	/* poll for samples */
	114	err = ring_buffer__poll(ringbuf, -1);
	115
	116	/* -EDONE is used as an indicator that we are done */
	117	if (CHECK(err != -EDONE, "err_done", "done err: %d\n", err))
	118	goto cleanup;
	119
	120	/* we expect extra polling to return nothing */
	121	err = ring_buffer__poll(ringbuf, 0);
	122	if (CHECK(err != 0, "extra_samples", "poll result: %d\n", err))
	123	goto cleanup;
	124
	125	CHECK(skel->bss->dropped != 0, "err_dropped", "exp %ld, got %ld\n",
	126	0L, skel->bss->dropped);
	127	CHECK(skel->bss->total != 2, "err_total", "exp %ld, got %ld\n",
	128	2L, skel->bss->total);
	129	CHECK(skel->bss->discarded != 1, "err_discarded", "exp %ld, got %ld\n",
	130	1L, skel->bss->discarded);
	131
	132	/* now validate consumer position is updated and returned */
	133	trigger_samples();
	134	CHECK(skel->bss->cons_pos != 3 * rec_sz,
	135	"err_cons_pos", "exp %ld, got %ld\n",
	136	3L * rec_sz, skel->bss->cons_pos);
	137	err = ring_buffer__poll(ringbuf, -1);
	138	CHECK(err <= 0, "poll_err", "err %d\n", err);
	139
	140	/* start poll in background w/ long timeout */
	141	err = pthread_create(&thread, NULL, poll_thread, (void *)(long)10000);
	142	if (CHECK(err, "bg_poll", "pthread_create failed: %d\n", err))
	143	goto cleanup;
	144
	145	/* turn off notifications now */
	146	skel->bss->flags = BPF_RB_NO_WAKEUP;
	147
	148	/* give background thread a bit of a time */
	149	usleep(50000);
	150	trigger_samples();
	151	/* sleeping arbitrarily is bad, but no better way to know that
	152	* epoll_wait() DID NOT unblock in background thread
	153	*/
	154	usleep(50000);
	155	/* background poll should still be blocked */
	156	err = pthread_tryjoin_np(thread, (void **)&bg_ret);
	157	if (CHECK(err != EBUSY, "try_join", "err %d\n", err))
	158	goto cleanup;
	159
	160	/* BPF side did everything right */
	161	CHECK(skel->bss->dropped != 0, "err_dropped", "exp %ld, got %ld\n",
	162	0L, skel->bss->dropped);
	163	CHECK(skel->bss->total != 2, "err_total", "exp %ld, got %ld\n",
	164	2L, skel->bss->total);
	165	CHECK(skel->bss->discarded != 1, "err_discarded", "exp %ld, got %ld\n",
	166	1L, skel->bss->discarded);
	167
	168	/* clear flags to return to "adaptive" notification mode */
	169	skel->bss->flags = 0;
	170
	171	/* produce new samples, no notification should be triggered, because
	172	* consumer is now behind
	173	*/
	174	trigger_samples();
	175
	176	/* background poll should still be blocked */
	177	err = pthread_tryjoin_np(thread, (void **)&bg_ret);
	178	if (CHECK(err != EBUSY, "try_join", "err %d\n", err))
	179	goto cleanup;
	180
	181	/* now force notifications */
	182	skel->bss->flags = BPF_RB_FORCE_WAKEUP;
	183	sample_cnt = 0;
	184	trigger_samples();
	185
	186	/* now we should get a pending notification */
	187	usleep(50000);
	188	err = pthread_tryjoin_np(thread, (void **)&bg_ret);
	189	if (CHECK(err, "join_bg", "err %d\n", err))
	190	goto cleanup;
	191
	192	if (CHECK(bg_ret != 1, "bg_ret", "epoll_wait result: %ld", bg_ret))
	193	goto cleanup;
	194
	195	/* 3 rounds, 2 samples each */
	196	CHECK(sample_cnt != 6, "wrong_sample_cnt",
	197	"expected to see %d samples, got %d\n", 6, sample_cnt);
	198
	199	/* BPF side did everything right */
	200	CHECK(skel->bss->dropped != 0, "err_dropped", "exp %ld, got %ld\n",
	201	0L, skel->bss->dropped);
	202	CHECK(skel->bss->total != 2, "err_total", "exp %ld, got %ld\n",
	203	2L, skel->bss->total);
	204	CHECK(skel->bss->discarded != 1, "err_discarded", "exp %ld, got %ld\n",
	205	1L, skel->bss->discarded);
	206
	207	test_ringbuf__detach(skel);
	208	cleanup:
	209	ring_buffer__free(ringbuf);
	210	test_ringbuf__destroy(skel);
	211	}

tools/testing/selftests/bpf/prog_tests/ringbuf_multi.c+102−0

@@ -0,0 +1,102 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	#define _GNU_SOURCE
	3	#include <test_progs.h>
	4	#include <sys/epoll.h>
	5	#include "test_ringbuf_multi.skel.h"
	6
	7	static int duration = 0;
	8
	9	struct sample {
	10	int pid;
	11	int seq;
	12	long value;
	13	char comm[16];
	14	};
	15
	16	static int process_sample(void ctx, void data, size_t len)
	17	{
	18	int ring = (unsigned long)ctx;
	19	struct sample *s = data;
	20
	21	switch (s->seq) {
	22	case 0:
	23	CHECK(ring != 1, "sample1_ring", "exp %d, got %d\n", 1, ring);
	24	CHECK(s->value != 333, "sample1_value", "exp %ld, got %ld\n",
	25	333L, s->value);
	26	break;
	27	case 1:
	28	CHECK(ring != 2, "sample2_ring", "exp %d, got %d\n", 2, ring);
	29	CHECK(s->value != 777, "sample2_value", "exp %ld, got %ld\n",
	30	777L, s->value);
	31	break;
	32	default:
	33	CHECK(true, "extra_sample", "unexpected sample seq %d, val %ld\n",
	34	s->seq, s->value);
	35	return -1;
	36	}
	37
	38	return 0;
	39	}
	40
	41	void test_ringbuf_multi(void)
	42	{
	43	struct test_ringbuf_multi *skel;
	44	struct ring_buffer *ringbuf;
	45	int err;
	46
	47	skel = test_ringbuf_multi__open_and_load();
	48	if (CHECK(!skel, "skel_open_load", "skeleton open&load failed\n"))
	49	return;
	50
	51	/* only trigger BPF program for current process */
	52	skel->bss->pid = getpid();
	53
	54	ringbuf = ring_buffer__new(bpf_map__fd(skel->maps.ringbuf1),
	55	process_sample, (void *)(long)1, NULL);
	56	if (CHECK(!ringbuf, "ringbuf_create", "failed to create ringbuf\n"))
	57	goto cleanup;
	58
	59	err = ring_buffer__add(ringbuf, bpf_map__fd(skel->maps.ringbuf2),
	60	process_sample, (void *)(long)2);
	61	if (CHECK(err, "ringbuf_add", "failed to add another ring\n"))
	62	goto cleanup;
	63
	64	err = test_ringbuf_multi__attach(skel);
	65	if (CHECK(err, "skel_attach", "skeleton attachment failed: %d\n", err))
	66	goto cleanup;
	67
	68	/* trigger few samples, some will be skipped */
	69	skel->bss->target_ring = 0;
	70	skel->bss->value = 333;
	71	syscall(__NR_getpgid);
	72
	73	/* skipped, no ringbuf in slot 1 */
	74	skel->bss->target_ring = 1;
	75	skel->bss->value = 555;
	76	syscall(__NR_getpgid);
	77
	78	skel->bss->target_ring = 2;
	79	skel->bss->value = 777;
	80	syscall(__NR_getpgid);
	81
	82	/* poll for samples, should get 2 ringbufs back */
	83	err = ring_buffer__poll(ringbuf, -1);
	84	if (CHECK(err != 4, "poll_res", "expected 4 records, got %d\n", err))
	85	goto cleanup;
	86
	87	/* expect extra polling to return nothing */
	88	err = ring_buffer__poll(ringbuf, 0);
	89	if (CHECK(err < 0, "extra_samples", "poll result: %d\n", err))
	90	goto cleanup;
	91
	92	CHECK(skel->bss->dropped != 0, "err_dropped", "exp %ld, got %ld\n",
	93	0L, skel->bss->dropped);
	94	CHECK(skel->bss->skipped != 1, "err_skipped", "exp %ld, got %ld\n",
	95	1L, skel->bss->skipped);
	96	CHECK(skel->bss->total != 2, "err_total", "exp %ld, got %ld\n",
	97	2L, skel->bss->total);
	98
	99	cleanup:
	100	ring_buffer__free(ringbuf);
	101	test_ringbuf_multi__destroy(skel);
	102	}

tools/testing/selftests/bpf/progs/perfbuf_bench.c+33−0

@@ -0,0 +1,33 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	// Copyright (c) 2020 Facebook
	3
	4	#include <linux/bpf.h>
	5	#include <stdint.h>
	6	#include <bpf/bpf_helpers.h>
	7
	8	char _license[] SEC("license") = "GPL";
	9
	10	struct {
	11	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
	12	__uint(value_size, sizeof(int));
	13	__uint(key_size, sizeof(int));
	14	} perfbuf SEC(".maps");
	15
	16	const volatile int batch_cnt = 0;
	17
	18	long sample_val = 42;
	19	long dropped __attribute__((aligned(128))) = 0;
	20
	21	SEC("fentry/__x64_sys_getpgid")
	22	int bench_perfbuf(void *ctx)
	23	{
	24	__u64 *sample;
	25	int i;
	26
	27	for (i = 0; i < batch_cnt; i++) {
	28	if (bpf_perf_event_output(ctx, &perfbuf, BPF_F_CURRENT_CPU,
	29	&sample_val, sizeof(sample_val)))
	30	__sync_add_and_fetch(&dropped, 1);
	31	}
	32	return 0;
	33	}

tools/testing/selftests/bpf/progs/ringbuf_bench.c+60−0

@@ -0,0 +1,60 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	// Copyright (c) 2020 Facebook
	3
	4	#include <linux/bpf.h>
	5	#include <stdint.h>
	6	#include <bpf/bpf_helpers.h>
	7
	8	char _license[] SEC("license") = "GPL";
	9
	10	struct {
	11	__uint(type, BPF_MAP_TYPE_RINGBUF);
	12	} ringbuf SEC(".maps");
	13
	14	const volatile int batch_cnt = 0;
	15	const volatile long use_output = 0;
	16
	17	long sample_val = 42;
	18	long dropped __attribute__((aligned(128))) = 0;
	19
	20	const volatile long wakeup_data_size = 0;
	21
	22	static __always_inline long get_flags()
	23	{
	24	long sz;
	25
	26	if (!wakeup_data_size)
	27	return 0;
	28
	29	sz = bpf_ringbuf_query(&ringbuf, BPF_RB_AVAIL_DATA);
	30	return sz >= wakeup_data_size ? BPF_RB_FORCE_WAKEUP : BPF_RB_NO_WAKEUP;
	31	}
	32
	33	SEC("fentry/__x64_sys_getpgid")
	34	int bench_ringbuf(void *ctx)
	35	{
	36	long *sample, flags;
	37	int i;
	38
	39	if (!use_output) {
	40	for (i = 0; i < batch_cnt; i++) {
	41	sample = bpf_ringbuf_reserve(&ringbuf,
	42	sizeof(sample_val), 0);
	43	if (!sample) {
	44	__sync_add_and_fetch(&dropped, 1);
	45	} else {
	46	*sample = sample_val;
	47	flags = get_flags();
	48	bpf_ringbuf_submit(sample, flags);
	49	}
	50	}
	51	} else {
	52	for (i = 0; i < batch_cnt; i++) {
	53	flags = get_flags();
	54	if (bpf_ringbuf_output(&ringbuf, &sample_val,
	55	sizeof(sample_val), flags))
	56	__sync_add_and_fetch(&dropped, 1);
	57	}
	58	}
	59	return 0;
	60	}

tools/testing/selftests/bpf/progs/test_ringbuf.c+78−0

@@ -0,0 +1,78 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	// Copyright (c) 2020 Facebook
	3
	4	#include <linux/bpf.h>
	5	#include <bpf/bpf_helpers.h>
	6
	7	char _license[] SEC("license") = "GPL";
	8
	9	struct sample {
	10	int pid;
	11	int seq;
	12	long value;
	13	char comm[16];
	14	};
	15
	16	struct {
	17	__uint(type, BPF_MAP_TYPE_RINGBUF);
	18	__uint(max_entries, 1 << 12);
	19	} ringbuf SEC(".maps");
	20
	21	/* inputs */
	22	int pid = 0;
	23	long value = 0;
	24	long flags = 0;
	25
	26	/* outputs */
	27	long total = 0;
	28	long discarded = 0;
	29	long dropped = 0;
	30
	31	long avail_data = 0;
	32	long ring_size = 0;
	33	long cons_pos = 0;
	34	long prod_pos = 0;
	35
	36	/* inner state */
	37	long seq = 0;
	38
	39	SEC("tp/syscalls/sys_enter_getpgid")
	40	int test_ringbuf(void *ctx)
	41	{
	42	int cur_pid = bpf_get_current_pid_tgid() >> 32;
	43	struct sample *sample;
	44	int zero = 0;
	45
	46	if (cur_pid != pid)
	47	return 0;
	48
	49	sample = bpf_ringbuf_reserve(&ringbuf, sizeof(*sample), 0);
	50	if (!sample) {
	51	__sync_fetch_and_add(&dropped, 1);
	52	return 1;
	53	}
	54
	55	sample->pid = pid;
	56	bpf_get_current_comm(sample->comm, sizeof(sample->comm));
	57	sample->value = value;
	58
	59	sample->seq = seq++;
	60	__sync_fetch_and_add(&total, 1);
	61
	62	if (sample->seq & 1) {
	63	/* copy from reserved sample to a new one... */
	64	bpf_ringbuf_output(&ringbuf, sample, sizeof(*sample), flags);
	65	/* ...and then discard reserved sample */
	66	bpf_ringbuf_discard(sample, flags);
	67	__sync_fetch_and_add(&discarded, 1);
	68	} else {
	69	bpf_ringbuf_submit(sample, flags);
	70	}
	71
	72	avail_data = bpf_ringbuf_query(&ringbuf, BPF_RB_AVAIL_DATA);
	73	ring_size = bpf_ringbuf_query(&ringbuf, BPF_RB_RING_SIZE);
	74	cons_pos = bpf_ringbuf_query(&ringbuf, BPF_RB_CONS_POS);
	75	prod_pos = bpf_ringbuf_query(&ringbuf, BPF_RB_PROD_POS);
	76
	77	return 0;
	78	}

tools/testing/selftests/bpf/progs/test_ringbuf_multi.c+77−0

@@ -0,0 +1,77 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	// Copyright (c) 2020 Facebook
	3
	4	#include <linux/bpf.h>
	5	#include <bpf/bpf_helpers.h>
	6
	7	char _license[] SEC("license") = "GPL";
	8
	9	struct sample {
	10	int pid;
	11	int seq;
	12	long value;
	13	char comm[16];
	14	};
	15
	16	struct ringbuf_map {
	17	__uint(type, BPF_MAP_TYPE_RINGBUF);
	18	__uint(max_entries, 1 << 12);
	19	} ringbuf1 SEC(".maps"),
	20	ringbuf2 SEC(".maps");
	21
	22	struct {
	23	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
	24	__uint(max_entries, 4);
	25	__type(key, int);
	26	__array(values, struct ringbuf_map);
	27	} ringbuf_arr SEC(".maps") = {
	28	.values = {
	29	[0] = &ringbuf1,
	30	[2] = &ringbuf2,
	31	},
	32	};
	33
	34	/* inputs */
	35	int pid = 0;
	36	int target_ring = 0;
	37	long value = 0;
	38
	39	/* outputs */
	40	long total = 0;
	41	long dropped = 0;
	42	long skipped = 0;
	43
	44	SEC("tp/syscalls/sys_enter_getpgid")
	45	int test_ringbuf(void *ctx)
	46	{
	47	int cur_pid = bpf_get_current_pid_tgid() >> 32;
	48	struct sample *sample;
	49	void *rb;
	50	int zero = 0;
	51
	52	if (cur_pid != pid)
	53	return 0;
	54
	55	rb = bpf_map_lookup_elem(&ringbuf_arr, &target_ring);
	56	if (!rb) {
	57	skipped += 1;
	58	return 1;
	59	}
	60
	61	sample = bpf_ringbuf_reserve(rb, sizeof(*sample), 0);
	62	if (!sample) {
	63	dropped += 1;
	64	return 1;
	65	}
	66
	67	sample->pid = pid;
	68	bpf_get_current_comm(sample->comm, sizeof(sample->comm));
	69	sample->value = value;
	70
	71	sample->seq = total;
	72	total += 1;
	73
	74	bpf_ringbuf_submit(sample, 0);
	75
	76	return 0;
	77	}

tools/testing/selftests/bpf/verifier/and.c+2−2

@@ -15,7 +15,7 @@ · 14 unmodified lines
15	BPF_EXIT_INSN(),	15	BPF_EXIT_INSN(),
16	},	16	},
17	.fixup_map_hash_48b = { 3 },	17	.fixup_map_hash_48b = { 3 },
18	.errstr = "R0 max value is outside of the array range",	18	.errstr = "R0 max value is outside of the allowed memory range",
19	.result = REJECT,	19	.result = REJECT,
20	.flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,	20	.flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,
21	},	21	},
@@ -44,7 +44,7 @@ · 22 unmodified lines
44	BPF_EXIT_INSN(),	44	BPF_EXIT_INSN(),
45	},	45	},
46	.fixup_map_hash_48b = { 3 },	46	.fixup_map_hash_48b = { 3 },
47	.errstr = "R0 max value is outside of the array range",	47	.errstr = "R0 max value is outside of the allowed memory range",
48	.result = REJECT,	48	.result = REJECT,
49	.flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,	49	.flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,
50	},	50	},

tools/testing/selftests/bpf/verifier/array_access.c+2−2

@@ -117,7 +117,7 @@ · 116 unmodified lines
117	BPF_EXIT_INSN(),	117	BPF_EXIT_INSN(),
118	},	118	},
119	.fixup_map_hash_48b = { 3 },	119	.fixup_map_hash_48b = { 3 },
120	.errstr = "R0 min value is outside of the array range",	120	.errstr = "R0 min value is outside of the allowed memory range",
121	.result = REJECT,	121	.result = REJECT,
122	.flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,	122	.flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,
123	},	123	},
@@ -137,7 +137,7 @@ · 13 unmodified lines
137	BPF_EXIT_INSN(),	137	BPF_EXIT_INSN(),
138	},	138	},
139	.fixup_map_hash_48b = { 3 },	139	.fixup_map_hash_48b = { 3 },
140	.errstr = "R0 unbounded memory access, make sure to bounds check any array access into a map",	140	.errstr = "R0 unbounded memory access, make sure to bounds check any such access",
141	.result = REJECT,	141	.result = REJECT,
142	.flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,	142	.flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,
143	},	143	},

tools/testing/selftests/bpf/verifier/bounds.c+3−3

@@ -20,7 +20,7 @@ · 19 unmodified lines
20	BPF_EXIT_INSN(),	20	BPF_EXIT_INSN(),
21	},	21	},
22	.fixup_map_hash_8b = { 3 },	22	.fixup_map_hash_8b = { 3 },
23	.errstr = "R0 max value is outside of the array range",	23	.errstr = "R0 max value is outside of the allowed memory range",
24	.result = REJECT,	24	.result = REJECT,
25	},	25	},
26	{	26	{
@@ -146,7 +146,7 @@ · 119 unmodified lines
146	BPF_EXIT_INSN(),	146	BPF_EXIT_INSN(),
147	},	147	},
148	.fixup_map_hash_8b = { 3 },	148	.fixup_map_hash_8b = { 3 },
149	.errstr = "R0 min value is outside of the array range",	149	.errstr = "R0 min value is outside of the allowed memory range",
150	.result = REJECT	150	.result = REJECT
151	},	151	},
152	{	152	{
@@ -354,7 +354,7 @@ · 201 unmodified lines
354	BPF_EXIT_INSN(),	354	BPF_EXIT_INSN(),
355	},	355	},
356	.fixup_map_hash_8b = { 3 },	356	.fixup_map_hash_8b = { 3 },
357	.errstr = "R0 max value is outside of the array range",	357	.errstr = "R0 max value is outside of the allowed memory range",
358	.result = REJECT	358	.result = REJECT
359	},	359	},
360	{	360	{

tools/testing/selftests/bpf/verifier/calls.c+1−1

@@ -105,7 +105,7 @@ · 104 unmodified lines
105	.prog_type = BPF_PROG_TYPE_SCHED_CLS,	105	.prog_type = BPF_PROG_TYPE_SCHED_CLS,
106	.fixup_map_hash_8b = { 16 },	106	.fixup_map_hash_8b = { 16 },
107	.result = REJECT,	107	.result = REJECT,
108	.errstr = "R0 min value is outside of the array range",	108	.errstr = "R0 min value is outside of the allowed memory range",
109	},	109	},
110	{	110	{
111	"calls: overlapping caller/callee",	111	"calls: overlapping caller/callee",

tools/testing/selftests/bpf/verifier/direct_value_access.c+2−2

@@ -68,7 +68,7 @@ · 67 unmodified lines
68	},	68	},
69	.fixup_map_array_48b = { 1 },	69	.fixup_map_array_48b = { 1 },
70	.result = REJECT,	70	.result = REJECT,
71	.errstr = "R1 min value is outside of the array range",	71	.errstr = "R1 min value is outside of the allowed memory range",
72	},	72	},
73	{	73	{
74	"direct map access, write test 7",	74	"direct map access, write test 7",
@@ -220,7 +220,7 @@ · 145 unmodified lines
220	},	220	},
221	.fixup_map_array_small = { 1 },	221	.fixup_map_array_small = { 1 },
222	.result = REJECT,	222	.result = REJECT,
223	.errstr = "R1 min value is outside of the array range",	223	.errstr = "R1 min value is outside of the allowed memory range",
224	},	224	},
225	{	225	{
226	"direct map access, write test 19",	226	"direct map access, write test 19",

tools/testing/selftests/bpf/verifier/helper_access_var_len.c+1−1

@@ -318,7 +318,7 @@ · 317 unmodified lines
318	BPF_EXIT_INSN(),	318	BPF_EXIT_INSN(),
319	},	319	},
320	.fixup_map_hash_48b = { 4 },	320	.fixup_map_hash_48b = { 4 },
321	.errstr = "R1 min value is outside of the array range",	321	.errstr = "R1 min value is outside of the allowed memory range",
322	.result = REJECT,	322	.result = REJECT,
323	.prog_type = BPF_PROG_TYPE_TRACEPOINT,	323	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
324	},	324	},

tools/testing/selftests/bpf/verifier/helper_value_access.c+3−3

@@ -280,7 +280,7 @@ · 279 unmodified lines
280	BPF_EXIT_INSN(),	280	BPF_EXIT_INSN(),
281	},	281	},
282	.fixup_map_hash_48b = { 3 },	282	.fixup_map_hash_48b = { 3 },
283	.errstr = "R1 min value is outside of the array range",	283	.errstr = "R1 min value is outside of the allowed memory range",
284	.result = REJECT,	284	.result = REJECT,
285	.prog_type = BPF_PROG_TYPE_TRACEPOINT,	285	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
286	},	286	},
@@ -415,7 +415,7 @@ · 128 unmodified lines
415	BPF_EXIT_INSN(),	415	BPF_EXIT_INSN(),
416	},	416	},
417	.fixup_map_hash_48b = { 3 },	417	.fixup_map_hash_48b = { 3 },
418	.errstr = "R1 min value is outside of the array range",	418	.errstr = "R1 min value is outside of the allowed memory range",
419	.result = REJECT,	419	.result = REJECT,
420	.prog_type = BPF_PROG_TYPE_TRACEPOINT,	420	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
421	},	421	},
@@ -926,7 +926,7 @@ · 504 unmodified lines
926	},	926	},
927	.fixup_map_hash_16b = { 3, 10 },	927	.fixup_map_hash_16b = { 3, 10 },
928	.result = REJECT,	928	.result = REJECT,
929	.errstr = "R2 unbounded memory access, make sure to bounds check any array access into a map",	929	.errstr = "R2 unbounded memory access, make sure to bounds check any such access",
930	.prog_type = BPF_PROG_TYPE_TRACEPOINT,	930	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
931	},	931	},
932	{	932	{

tools/testing/selftests/bpf/verifier/value_ptr_arith.c+4−4

@@ -50,7 +50,7 @@ · 49 unmodified lines
50	.fixup_map_array_48b = { 8 },	50	.fixup_map_array_48b = { 8 },
51	.result = ACCEPT,	51	.result = ACCEPT,
52	.result_unpriv = REJECT,	52	.result_unpriv = REJECT,
53	.errstr_unpriv = "R0 min value is outside of the array range",	53	.errstr_unpriv = "R0 min value is outside of the allowed memory range",
54	.retval = 1,	54	.retval = 1,
55	},	55	},
56	{	56	{
@@ -325,7 +325,7 @@ · 268 unmodified lines
325	},	325	},
326	.fixup_map_array_48b = { 3 },	326	.fixup_map_array_48b = { 3 },
327	.result = REJECT,	327	.result = REJECT,
328	.errstr = "R0 min value is outside of the array range",	328	.errstr = "R0 min value is outside of the allowed memory range",
329	.result_unpriv = REJECT,	329	.result_unpriv = REJECT,
330	.errstr_unpriv = "R0 pointer arithmetic of map value goes out of range",	330	.errstr_unpriv = "R0 pointer arithmetic of map value goes out of range",
331	},	331	},
@@ -601,7 +601,7 @@ · 269 unmodified lines
601	},	601	},
602	.fixup_map_array_48b = { 3 },	602	.fixup_map_array_48b = { 3 },
603	.result = REJECT,	603	.result = REJECT,
604	.errstr = "R1 max value is outside of the array range",	604	.errstr = "R1 max value is outside of the allowed memory range",
605	.errstr_unpriv = "R1 pointer arithmetic of map value goes out of range",	605	.errstr_unpriv = "R1 pointer arithmetic of map value goes out of range",
606	.flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,	606	.flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,
607	},	607	},
@@ -726,7 +726,7 @@ · 118 unmodified lines
726	},	726	},
727	.fixup_map_array_48b = { 3 },	727	.fixup_map_array_48b = { 3 },
728	.result = REJECT,	728	.result = REJECT,
729	.errstr = "R0 min value is outside of the array range",	729	.errstr = "R0 min value is outside of the allowed memory range",
730	},	730	},
731	{	731	{
732	"map access: value_ptr -= known scalar, 2",	732	"map access: value_ptr -= known scalar, 2",