History

Merge branch 'bpf-ring-buffer'

Andrii Nakryiko says: ==================== Implement a new BPF ring buffer, as presented at BPF virtual conference ([0]). It presents an alternative to perf buffer, following its semantics closely, but allowing sharing same instance of ring buffer across multiple CPUs efficiently. Most patches have extensive commentary explaining various aspects, so I'll keep cover letter short. Overall structure of the patch set: - patch #1 adds BPF ring buffer implementation to kernel and necessary verifier support; - patch #2 adds libbpf consumer implementation for BPF ringbuf; - patch #3 adds selftest, both for single BPF ring buf use case, as well as using it with array/hash of maps; - patch #4 adds extensive benchmarks and provide some analysis in commit message, it builds upon selftests/bpf's bench runner. - patch #5 adds most of patch #1 commit message as a doc under Documentation/bpf/ringbuf.rst. Litmus tests, validating consumer/producer protocols and memory orderings, were moved out as discussed in [1] and are going to be posted against -rcu tree and put under Documentation/litmus-tests/bpf-rb. [0] https://docs.google.com/presentation/d/18ITdg77Bj6YDOH2LghxrnFxiPWe0fAqcmJY95t_qr0w [1] https://lkml.org/lkml/2020/5/22/1011 v3->v4: - fix ringbuf freeing (vunmap, __free_page); verified with a trivial loop creating and closing ringbuf map endlessly (Daniel); v2->v3: - dropped unnecessary smp_wmb() (Paul); - verifier reference type enhancement patch was dropped (Alexei); - better verifier message for various memory access checks (Alexei); - clarified a bit roundup_len() bit shifting (Alexei); - converted doc to .rst (Alexei); - fixed warning on 32-bit arches regarding tautological ring area size check. v1->v2: - commit()/discard()/output() accept flags (NO_WAKEUP/FORCE_WAKEUP) (Stanislav); - bpf_ringbuf_query() added, returning available data size, ringbuf size, consumer/producer positions, needed to implement smarter notification policy (Stanislav); - added ringbuf UAPI constants to include/uapi/linux/bpf.h (Jonathan); - fixed sample size check, added proper ringbuf size check (Jonathan, Alexei); - wake_up_all() is done through irq_work (Alexei); - consistent use of smp_load_acquire/smp_store_release, no READ_ONCE/WRITE_ONCE (Alexei); - added Documentation/bpf/ringbuf.txt (Stanislav); - updated litmus test with smp_load_acquire/smp_store_release changes; - added ring_buffer__consume() API to libbpf for busy-polling; - ring_buffer__poll() on success returns number of records consumed; - fixed EPOLL notifications, don't assume available data, done similarly to perfbuf's implementation; - both ringbuf and perfbuf now have --rb-sampled mode, instead of pb-raw/pb-custom mode, updated benchmark results; - extended ringbuf selftests to validate epoll logic/manual notification logic, as well as bpf_ringbuf_query(). ====================

Daniel Borkmann committed 6 years ago

Commit e255d322 parents 43dd115 97abb2b35 files · +2630 −72Browse files

Unified · Split

35 files
  1. Documentation/bpf/ringbuf.rst +209 −0
  2. include/linux/bpf.h +13 −0
  3. include/linux/bpf_types.h +1 −0
  4. include/linux/bpf_verifier.h +4 −0
  5. include/uapi/linux/bpf.h +83 −1
  6. kernel/bpf/Makefile +1 −1
  7. kernel/bpf/helpers.c +10 −0
  8. kernel/bpf/ringbuf.c +501 −0
  9. kernel/bpf/syscall.c +12 −0
  10. kernel/bpf/verifier.c +146 −49
  11. kernel/trace/bpf_trace.c +10 −0
  12. tools/include/uapi/linux/bpf.h +83 −1
  13. tools/lib/bpf/Build +1 −1
  14. tools/lib/bpf/libbpf.h +21 −0
  15. tools/lib/bpf/libbpf.map +5 −0
  16. tools/lib/bpf/libbpf_probes.c +5 −0
  17. tools/lib/bpf/ringbuf.c +285 −0
  18. tools/testing/selftests/bpf/Makefile +4 −1
  19. tools/testing/selftests/bpf/bench.c +16 −0
  20. tools/testing/selftests/bpf/benchs/bench_ringbufs.c +566 −0
  21. tools/testing/selftests/bpf/benchs/run_bench_ringbufs.sh +75 −0
  22. tools/testing/selftests/bpf/prog_tests/ringbuf.c +211 −0
  23. tools/testing/selftests/bpf/prog_tests/ringbuf_multi.c +102 −0
  24. tools/testing/selftests/bpf/progs/perfbuf_bench.c +33 −0
  25. tools/testing/selftests/bpf/progs/ringbuf_bench.c +60 −0
  26. tools/testing/selftests/bpf/progs/test_ringbuf.c +78 −0
  27. tools/testing/selftests/bpf/progs/test_ringbuf_multi.c +77 −0
  28. tools/testing/selftests/bpf/verifier/and.c +2 −2
  29. tools/testing/selftests/bpf/verifier/array_access.c +2 −2
  30. tools/testing/selftests/bpf/verifier/bounds.c +3 −3
  31. tools/testing/selftests/bpf/verifier/calls.c +1 −1
  32. tools/testing/selftests/bpf/verifier/direct_value_access.c +2 −2
  33. tools/testing/selftests/bpf/verifier/helper_access_var_len.c +1 −1
  34. tools/testing/selftests/bpf/verifier/helper_value_access.c +3 −3
  35. tools/testing/selftests/bpf/verifier/value_ptr_arith.c +4 −4
Documentation/bpf/ringbuf.rst+209−0
@@ -0,0 +1,209 @@
1+===============
2+BPF ring buffer
3+===============
4+
5+This document describes BPF ring buffer design, API, and implementation details.
6+
7+.. contents::
8+ :local:
9+ :depth: 2
10+
11+Motivation
12+----------
13+
14+There are two distinctive motivators for this work, which are not satisfied by
15+existing perf buffer, which prompted creation of a new ring buffer
16+implementation.
17+
18+- more efficient memory utilization by sharing ring buffer across CPUs;
19+- preserving ordering of events that happen sequentially in time, even across
20+ multiple CPUs (e.g., fork/exec/exit events for a task).
21+
22+These two problems are independent, but perf buffer fails to satisfy both.
23+Both are a result of a choice to have per-CPU perf ring buffer. Both can be
24+also solved by having an MPSC implementation of ring buffer. The ordering
25+problem could technically be solved for perf buffer with some in-kernel
26+counting, but given the first one requires an MPSC buffer, the same solution
27+would solve the second problem automatically.
28+
29+Semantics and APIs
30+------------------
31+
32+Single ring buffer is presented to BPF programs as an instance of BPF map of
33+type ``BPF_MAP_TYPE_RINGBUF``. Two other alternatives considered, but
34+ultimately rejected.
35+
36+One way would be to, similar to ``BPF_MAP_TYPE_PERF_EVENT_ARRAY``, make
37+``BPF_MAP_TYPE_RINGBUF`` could represent an array of ring buffers, but not
38+enforce "same CPU only" rule. This would be more familiar interface compatible
39+with existing perf buffer use in BPF, but would fail if application needed more
40+advanced logic to lookup ring buffer by arbitrary key.
41+``BPF_MAP_TYPE_HASH_OF_MAPS`` addresses this with current approach.
42+Additionally, given the performance of BPF ringbuf, many use cases would just
43+opt into a simple single ring buffer shared among all CPUs, for which current
44+approach would be an overkill.
45+
46+Another approach could introduce a new concept, alongside BPF map, to represent
47+generic "container" object, which doesn't necessarily have key/value interface
48+with lookup/update/delete operations. This approach would add a lot of extra
49+infrastructure that has to be built for observability and verifier support. It
50+would also add another concept that BPF developers would have to familiarize
51+themselves with, new syntax in libbpf, etc. But then would really provide no
52+additional benefits over the approach of using a map. ``BPF_MAP_TYPE_RINGBUF``
53+doesn't support lookup/update/delete operations, but so doesn't few other map
54+types (e.g., queue and stack; array doesn't support delete, etc).
55+
56+The approach chosen has an advantage of re-using existing BPF map
57+infrastructure (introspection APIs in kernel, libbpf support, etc), being
58+familiar concept (no need to teach users a new type of object in BPF program),
59+and utilizing existing tooling (bpftool). For common scenario of using a single
60+ring buffer for all CPUs, it's as simple and straightforward, as would be with
61+a dedicated "container" object. On the other hand, by being a map, it can be
62+combined with ``ARRAY_OF_MAPS`` and ``HASH_OF_MAPS`` map-in-maps to implement
63+a wide variety of topologies, from one ring buffer for each CPU (e.g., as
64+a replacement for perf buffer use cases), to a complicated application
65+hashing/sharding of ring buffers (e.g., having a small pool of ring buffers
66+with hashed task's tgid being a look up key to preserve order, but reduce
67+contention).
68+
69+Key and value sizes are enforced to be zero. ``max_entries`` is used to specify
70+the size of ring buffer and has to be a power of 2 value.
71+
72+There are a bunch of similarities between perf buffer
73+(``BPF_MAP_TYPE_PERF_EVENT_ARRAY``) and new BPF ring buffer semantics:
74+
75+- variable-length records;
76+- if there is no more space left in ring buffer, reservation fails, no
77+ blocking;
78+- memory-mappable data area for user-space applications for ease of
79+ consumption and high performance;
80+- epoll notifications for new incoming data;
81+- but still the ability to do busy polling for new data to achieve the
82+ lowest latency, if necessary.
83+
84+BPF ringbuf provides two sets of APIs to BPF programs:
85+
86+- ``bpf_ringbuf_output()`` allows to *copy* data from one place to a ring
87+ buffer, similarly to ``bpf_perf_event_output()``;
88+- ``bpf_ringbuf_reserve()``/``bpf_ringbuf_commit()``/``bpf_ringbuf_discard()``
89+ APIs split the whole process into two steps. First, a fixed amount of space
90+ is reserved. If successful, a pointer to a data inside ring buffer data
91+ area is returned, which BPF programs can use similarly to a data inside
92+ array/hash maps. Once ready, this piece of memory is either committed or
93+ discarded. Discard is similar to commit, but makes consumer ignore the
94+ record.
95+
96+``bpf_ringbuf_output()`` has disadvantage of incurring extra memory copy,
97+because record has to be prepared in some other place first. But it allows to
98+submit records of the length that's not known to verifier beforehand. It also
99+closely matches ``bpf_perf_event_output()``, so will simplify migration
100+significantly.
101+
102+``bpf_ringbuf_reserve()`` avoids the extra copy of memory by providing a memory
103+pointer directly to ring buffer memory. In a lot of cases records are larger
104+than BPF stack space allows, so many programs have use extra per-CPU array as
105+a temporary heap for preparing sample. bpf_ringbuf_reserve() avoid this needs
106+completely. But in exchange, it only allows a known constant size of memory to
107+be reserved, such that verifier can verify that BPF program can't access memory
108+outside its reserved record space. bpf_ringbuf_output(), while slightly slower
109+due to extra memory copy, covers some use cases that are not suitable for
110+``bpf_ringbuf_reserve()``.
111+
112+The difference between commit and discard is very small. Discard just marks
113+a record as discarded, and such records are supposed to be ignored by consumer
114+code. Discard is useful for some advanced use-cases, such as ensuring
115+all-or-nothing multi-record submission, or emulating temporary
116+``malloc()``/``free()`` within single BPF program invocation.
117+
118+Each reserved record is tracked by verifier through existing
119+reference-tracking logic, similar to socket ref-tracking. It is thus
120+impossible to reserve a record, but forget to submit (or discard) it.
121+
122+``bpf_ringbuf_query()`` helper allows to query various properties of ring
123+buffer. Currently 4 are supported:
124+
125+- ``BPF_RB_AVAIL_DATA`` returns amount of unconsumed data in ring buffer;
126+- ``BPF_RB_RING_SIZE`` returns the size of ring buffer;
127+- ``BPF_RB_CONS_POS``/``BPF_RB_PROD_POS`` returns current logical possition
128+ of consumer/producer, respectively.
129+
130+Returned values are momentarily snapshots of ring buffer state and could be
131+off by the time helper returns, so this should be used only for
132+debugging/reporting reasons or for implementing various heuristics, that take
133+into account highly-changeable nature of some of those characteristics.
134+
135+One such heuristic might involve more fine-grained control over poll/epoll
136+notifications about new data availability in ring buffer. Together with
137+``BPF_RB_NO_WAKEUP``/``BPF_RB_FORCE_WAKEUP`` flags for output/commit/discard
138+helpers, it allows BPF program a high degree of control and, e.g., more
139+efficient batched notifications. Default self-balancing strategy, though,
140+should be adequate for most applications and will work reliable and efficiently
141+already.
142+
143+Design and Implementation
144+-------------------------
145+
146+This reserve/commit schema allows a natural way for multiple producers, either
147+on different CPUs or even on the same CPU/in the same BPF program, to reserve
148+independent records and work with them without blocking other producers. This
149+means that if BPF program was interruped by another BPF program sharing the
150+same ring buffer, they will both get a record reserved (provided there is
151+enough space left) and can work with it and submit it independently. This
152+applies to NMI context as well, except that due to using a spinlock during
153+reservation, in NMI context, ``bpf_ringbuf_reserve()`` might fail to get
154+a lock, in which case reservation will fail even if ring buffer is not full.
155+
156+The ring buffer itself internally is implemented as a power-of-2 sized
157+circular buffer, with two logical and ever-increasing counters (which might
158+wrap around on 32-bit architectures, that's not a problem):
159+
160+- consumer counter shows up to which logical position consumer consumed the
161+ data;
162+- producer counter denotes amount of data reserved by all producers.
163+
164+Each time a record is reserved, producer that "owns" the record will
165+successfully advance producer counter. At that point, data is still not yet
166+ready to be consumed, though. Each record has 8 byte header, which contains the
167+length of reserved record, as well as two extra bits: busy bit to denote that
168+record is still being worked on, and discard bit, which might be set at commit
169+time if record is discarded. In the latter case, consumer is supposed to skip
170+the record and move on to the next one. Record header also encodes record's
171+relative offset from the beginning of ring buffer data area (in pages). This
172+allows ``bpf_ringbuf_commit()``/``bpf_ringbuf_discard()`` to accept only the
173+pointer to the record itself, without requiring also the pointer to ring buffer
174+itself. Ring buffer memory location will be restored from record metadata
175+header. This significantly simplifies verifier, as well as improving API
176+usability.
177+
178+Producer counter increments are serialized under spinlock, so there is
179+a strict ordering between reservations. Commits, on the other hand, are
180+completely lockless and independent. All records become available to consumer
181+in the order of reservations, but only after all previous records where
182+already committed. It is thus possible for slow producers to temporarily hold
183+off submitted records, that were reserved later.
184+
185+Reservation/commit/consumer protocol is verified by litmus tests in
186+Documentation/litmus_tests/bpf-rb/_.
187+
188+One interesting implementation bit, that significantly simplifies (and thus
189+speeds up as well) implementation of both producers and consumers is how data
190+area is mapped twice contiguously back-to-back in the virtual memory. This
191+allows to not take any special measures for samples that have to wrap around
192+at the end of the circular buffer data area, because the next page after the
193+last data page would be first data page again, and thus the sample will still
194+appear completely contiguous in virtual memory. See comment and a simple ASCII
195+diagram showing this visually in ``bpf_ringbuf_area_alloc()``.
196+
197+Another feature that distinguishes BPF ringbuf from perf ring buffer is
198+a self-pacing notifications of new data being availability.
199+``bpf_ringbuf_commit()`` implementation will send a notification of new record
200+being available after commit only if consumer has already caught up right up to
201+the record being committed. If not, consumer still has to catch up and thus
202+will see new data anyways without needing an extra poll notification.
203+Benchmarks (see tools/testing/selftests/bpf/benchs/bench_ringbuf.c_) show that
204+this allows to achieve a very high throughput without having to resort to
205+tricks like "notify only every Nth sample", which are necessary with perf
206+buffer. For extreme cases, when BPF program wants more manual control of
207+notifications, commit/discard/output helpers accept ``BPF_RB_NO_WAKEUP`` and
208+``BPF_RB_FORCE_WAKEUP`` flags, which give full control over notifications of
209+data availability, but require extra caution and diligence in using this API.
include/linux/bpf.h+13−0
@@ -90,6 +90,8 @@ struct bpf_map_ops { · 89 unmodified lines
9090 int (*map_direct_value_meta)(const struct bpf_map *map,
9191 u64 imm, u32 *off);
9292 int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
93+ __poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
94+ struct poll_table_struct *pts);
9395 };
9496
9597 struct bpf_map_memory {
@@ -244,6 +246,9 @@ enum bpf_arg_type { · 148 unmodified lines
244246 ARG_PTR_TO_LONG, /* pointer to long */
245247 ARG_PTR_TO_SOCKET, /* pointer to bpf_sock (fullsock) */
246248 ARG_PTR_TO_BTF_ID, /* pointer to in-kernel struct */
249+ ARG_PTR_TO_ALLOC_MEM, /* pointer to dynamically allocated memory */
250+ ARG_PTR_TO_ALLOC_MEM_OR_NULL, /* pointer to dynamically allocated memory or NULL */
251+ ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */
247252 };
248253
249254 /* type of values returned from helper functions */
@@ -255,6 +260,7 @@ enum bpf_return_type { · 5 unmodified lines
255260 RET_PTR_TO_SOCKET_OR_NULL, /* returns a pointer to a socket or NULL */
256261 RET_PTR_TO_TCP_SOCK_OR_NULL, /* returns a pointer to a tcp_sock or NULL */
257262 RET_PTR_TO_SOCK_COMMON_OR_NULL, /* returns a pointer to a sock_common or NULL */
263+ RET_PTR_TO_ALLOC_MEM_OR_NULL, /* returns a pointer to dynamically allocated memory or NULL */
258264 };
259265
260266 /* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs
@@ -322,6 +328,8 @@ enum bpf_reg_type { · 61 unmodified lines
322328 PTR_TO_XDP_SOCK, /* reg points to struct xdp_sock */
323329 PTR_TO_BTF_ID, /* reg points to kernel struct */
324330 PTR_TO_BTF_ID_OR_NULL, /* reg points to kernel struct or NULL */
331+ PTR_TO_MEM, /* reg points to valid memory region */
332+ PTR_TO_MEM_OR_NULL, /* reg points to valid memory region or NULL */
325333 };
326334
327335 /* The information passed from prog-specific *_is_valid_access
@@ -1611,6 +1619,11 @@ extern const struct bpf_func_proto bpf_tcp_sock_proto; · 1283 unmodified lines
16111619 extern const struct bpf_func_proto bpf_jiffies64_proto;
16121620 extern const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto;
16131621 extern const struct bpf_func_proto bpf_event_output_data_proto;
1622+extern const struct bpf_func_proto bpf_ringbuf_output_proto;
1623+extern const struct bpf_func_proto bpf_ringbuf_reserve_proto;
1624+extern const struct bpf_func_proto bpf_ringbuf_submit_proto;
1625+extern const struct bpf_func_proto bpf_ringbuf_discard_proto;
1626+extern const struct bpf_func_proto bpf_ringbuf_query_proto;
16141627
16151628 const struct bpf_func_proto *bpf_tracing_func_proto(
16161629 enum bpf_func_id func_id, const struct bpf_prog *prog);
include/linux/bpf_types.h+1−0
@@ -118,6 +118,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_STACK, stack_map_ops) · 117 unmodified lines
118118 #if defined(CONFIG_BPF_JIT)
119119 BPF_MAP_TYPE(BPF_MAP_TYPE_STRUCT_OPS, bpf_struct_ops_map_ops)
120120 #endif
121+BPF_MAP_TYPE(BPF_MAP_TYPE_RINGBUF, ringbuf_map_ops)
121122
122123 BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)
123124 BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing)
include/linux/bpf_verifier.h+4−0
@@ -54,6 +54,8 @@ struct bpf_reg_state { · 53 unmodified lines
5454
5555 u32 btf_id; /* for PTR_TO_BTF_ID */
5656
57+ u32 mem_size; /* for PTR_TO_MEM | PTR_TO_MEM_OR_NULL */
58+
5759 /* Max size from any of the above. */
5860 unsigned long raw;
5961 };
@@ -63,6 +65,8 @@ struct bpf_reg_state { · 3 unmodified lines
6365 * offset, so they can share range knowledge.
6466 * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we
6567 * came from, when one is tested for != NULL.
68+ * For PTR_TO_MEM_OR_NULL this is used to identify memory allocation
69+ * for the purpose of tracking that it's freed.
6670 * For PTR_TO_SOCKET this is used to share which pointers retain the
6771 * same reference to the socket, to determine proper reference freeing.
6872 */
include/uapi/linux/bpf.h+83−1
@@ -147,6 +147,7 @@ enum bpf_map_type { · 146 unmodified lines
147147 BPF_MAP_TYPE_SK_STORAGE,
148148 BPF_MAP_TYPE_DEVMAP_HASH,
149149 BPF_MAP_TYPE_STRUCT_OPS,
150+ BPF_MAP_TYPE_RINGBUF,
150151 };
151152
152153 /* Note that tracing related programs such as
@@ -3157,6 +3158,59 @@ union bpf_attr { · 3004 unmodified lines
31573158 * **bpf_sk_cgroup_id**\ ().
31583159 * Return
31593160 * The id is returned or 0 in case the id could not be retrieved.
3161+ *
3162+ * void *bpf_ringbuf_output(void *ringbuf, void *data, u64 size, u64 flags)
3163+ * Description
3164+ * Copy *size* bytes from *data* into a ring buffer *ringbuf*.
3165+ * If BPF_RB_NO_WAKEUP is specified in *flags*, no notification of
3166+ * new data availability is sent.
3167+ * IF BPF_RB_FORCE_WAKEUP is specified in *flags*, notification of
3168+ * new data availability is sent unconditionally.
3169+ * Return
3170+ * 0, on success;
3171+ * < 0, on error.
3172+ *
3173+ * void *bpf_ringbuf_reserve(void *ringbuf, u64 size, u64 flags)
3174+ * Description
3175+ * Reserve *size* bytes of payload in a ring buffer *ringbuf*.
3176+ * Return
3177+ * Valid pointer with *size* bytes of memory available; NULL,
3178+ * otherwise.
3179+ *
3180+ * void bpf_ringbuf_submit(void *data, u64 flags)
3181+ * Description
3182+ * Submit reserved ring buffer sample, pointed to by *data*.
3183+ * If BPF_RB_NO_WAKEUP is specified in *flags*, no notification of
3184+ * new data availability is sent.
3185+ * IF BPF_RB_FORCE_WAKEUP is specified in *flags*, notification of
3186+ * new data availability is sent unconditionally.
3187+ * Return
3188+ * Nothing. Always succeeds.
3189+ *
3190+ * void bpf_ringbuf_discard(void *data, u64 flags)
3191+ * Description
3192+ * Discard reserved ring buffer sample, pointed to by *data*.
3193+ * If BPF_RB_NO_WAKEUP is specified in *flags*, no notification of
3194+ * new data availability is sent.
3195+ * IF BPF_RB_FORCE_WAKEUP is specified in *flags*, notification of
3196+ * new data availability is sent unconditionally.
3197+ * Return
3198+ * Nothing. Always succeeds.
3199+ *
3200+ * u64 bpf_ringbuf_query(void *ringbuf, u64 flags)
3201+ * Description
3202+ * Query various characteristics of provided ring buffer. What
3203+ * exactly is queries is determined by *flags*:
3204+ * - BPF_RB_AVAIL_DATA - amount of data not yet consumed;
3205+ * - BPF_RB_RING_SIZE - the size of ring buffer;
3206+ * - BPF_RB_CONS_POS - consumer position (can wrap around);
3207+ * - BPF_RB_PROD_POS - producer(s) position (can wrap around);
3208+ * Data returned is just a momentary snapshots of actual values
3209+ * and could be inaccurate, so this facility should be used to
3210+ * power heuristics and for reporting, not to make 100% correct
3211+ * calculation.
3212+ * Return
3213+ * Requested value, or 0, if flags are not recognized.
31603214 */
31613215 #define __BPF_FUNC_MAPPER(FN) \
31623216 FN(unspec), \
@@ -3288,7 +3342,12 @@ union bpf_attr { · 125 unmodified lines
32883342 FN(seq_printf), \
32893343 FN(seq_write), \
32903344 FN(sk_cgroup_id), \
3291− FN(sk_ancestor_cgroup_id),
3345+ FN(sk_ancestor_cgroup_id), \
3346+ FN(ringbuf_output), \
3347+ FN(ringbuf_reserve), \
3348+ FN(ringbuf_submit), \
3349+ FN(ringbuf_discard), \
3350+ FN(ringbuf_query),
32923351
32933352 /* integer value in 'imm' field of BPF_CALL instruction selects which helper
32943353 * function eBPF program intends to call
@@ -3398,6 +3457,29 @@ enum { · 103 unmodified lines
33983457 BPF_F_GET_BRANCH_RECORDS_SIZE = (1ULL << 0),
33993458 };
34003459
3460+/* BPF_FUNC_bpf_ringbuf_commit, BPF_FUNC_bpf_ringbuf_discard, and
3461+ * BPF_FUNC_bpf_ringbuf_output flags.
3462+ */
3463+enum {
3464+ BPF_RB_NO_WAKEUP = (1ULL << 0),
3465+ BPF_RB_FORCE_WAKEUP = (1ULL << 1),
3466+};
3467+
3468+/* BPF_FUNC_bpf_ringbuf_query flags */
3469+enum {
3470+ BPF_RB_AVAIL_DATA = 0,
3471+ BPF_RB_RING_SIZE = 1,
3472+ BPF_RB_CONS_POS = 2,
3473+ BPF_RB_PROD_POS = 3,
3474+};
3475+
3476+/* BPF ring buffer constants */
3477+enum {
3478+ BPF_RINGBUF_BUSY_BIT = (1U << 31),
3479+ BPF_RINGBUF_DISCARD_BIT = (1U << 30),
3480+ BPF_RINGBUF_HDR_SZ = 8,
3481+};
3482+
34013483 /* Mode for BPF_FUNC_skb_adjust_room helper. */
34023484 enum bpf_adj_room_mode {
34033485 BPF_ADJ_ROOM_NET,
kernel/bpf/Makefile+1−1
@@ -4,7 +4,7 @@ CFLAGS_core.o += $(call cc-disable-warning, override-init) · 3 unmodified lines
44
55 obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o task_iter.o
66 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o
7−obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o
7+obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
88 obj-$(CONFIG_BPF_SYSCALL) += disasm.o
99 obj-$(CONFIG_BPF_JIT) += trampoline.o
1010 obj-$(CONFIG_BPF_SYSCALL) += btf.o
kernel/bpf/helpers.c+10−0
@@ -635,6 +635,16 @@ bpf_base_func_proto(enum bpf_func_id func_id) · 634 unmodified lines
635635 return &bpf_ktime_get_ns_proto;
636636 case BPF_FUNC_ktime_get_boot_ns:
637637 return &bpf_ktime_get_boot_ns_proto;
638+ case BPF_FUNC_ringbuf_output:
639+ return &bpf_ringbuf_output_proto;
640+ case BPF_FUNC_ringbuf_reserve:
641+ return &bpf_ringbuf_reserve_proto;
642+ case BPF_FUNC_ringbuf_submit:
643+ return &bpf_ringbuf_submit_proto;
644+ case BPF_FUNC_ringbuf_discard:
645+ return &bpf_ringbuf_discard_proto;
646+ case BPF_FUNC_ringbuf_query:
647+ return &bpf_ringbuf_query_proto;
638648 default:
639649 break;
640650 }
kernel/bpf/ringbuf.c+501−0
@@ -0,0 +1,501 @@
1+#include <linux/bpf.h>
2+#include <linux/btf.h>
3+#include <linux/err.h>
4+#include <linux/irq_work.h>
5+#include <linux/slab.h>
6+#include <linux/filter.h>
7+#include <linux/mm.h>
8+#include <linux/vmalloc.h>
9+#include <linux/wait.h>
10+#include <linux/poll.h>
11+#include <uapi/linux/btf.h>
12+
13+#define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)
14+
15+/* non-mmap()'able part of bpf_ringbuf (everything up to consumer page) */
16+#define RINGBUF_PGOFF \
17+ (offsetof(struct bpf_ringbuf, consumer_pos) >> PAGE_SHIFT)
18+/* consumer page and producer page */
19+#define RINGBUF_POS_PAGES 2
20+
21+#define RINGBUF_MAX_RECORD_SZ (UINT_MAX/4)
22+
23+/* Maximum size of ring buffer area is limited by 32-bit page offset within
24+ * record header, counted in pages. Reserve 8 bits for extensibility, and take
25+ * into account few extra pages for consumer/producer pages and
26+ * non-mmap()'able parts. This gives 64GB limit, which seems plenty for single
27+ * ring buffer.
28+ */
29+#define RINGBUF_MAX_DATA_SZ \
30+ (((1ULL << 24) - RINGBUF_POS_PAGES - RINGBUF_PGOFF) * PAGE_SIZE)
31+
32+struct bpf_ringbuf {
33+ wait_queue_head_t waitq;
34+ struct irq_work work;
35+ u64 mask;
36+ struct page **pages;
37+ int nr_pages;
38+ spinlock_t spinlock ____cacheline_aligned_in_smp;
39+ /* Consumer and producer counters are put into separate pages to allow
40+ * mapping consumer page as r/w, but restrict producer page to r/o.
41+ * This protects producer position from being modified by user-space
42+ * application and ruining in-kernel position tracking.
43+ */
44+ unsigned long consumer_pos __aligned(PAGE_SIZE);
45+ unsigned long producer_pos __aligned(PAGE_SIZE);
46+ char data[] __aligned(PAGE_SIZE);
47+};
48+
49+struct bpf_ringbuf_map {
50+ struct bpf_map map;
51+ struct bpf_map_memory memory;
52+ struct bpf_ringbuf *rb;
53+};
54+
55+/* 8-byte ring buffer record header structure */
56+struct bpf_ringbuf_hdr {
57+ u32 len;
58+ u32 pg_off;
59+};
60+
61+static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node)
62+{
63+ const gfp_t flags = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN |
64+ __GFP_ZERO;
65+ int nr_meta_pages = RINGBUF_PGOFF + RINGBUF_POS_PAGES;
66+ int nr_data_pages = data_sz >> PAGE_SHIFT;
67+ int nr_pages = nr_meta_pages + nr_data_pages;
68+ struct page **pages, *page;
69+ struct bpf_ringbuf *rb;
70+ size_t array_size;
71+ int i;
72+
73+ /* Each data page is mapped twice to allow "virtual"
74+ * continuous read of samples wrapping around the end of ring
75+ * buffer area:
76+ * ------------------------------------------------------
77+ * | meta pages | real data pages | same data pages |
78+ * ------------------------------------------------------
79+ * | | 1 2 3 4 5 6 7 8 9 | 1 2 3 4 5 6 7 8 9 |
80+ * ------------------------------------------------------
81+ * | | TA DA | TA DA |
82+ * ------------------------------------------------------
83+ * ^^^^^^^
84+ * |
85+ * Here, no need to worry about special handling of wrapped-around
86+ * data due to double-mapped data pages. This works both in kernel and
87+ * when mmap()'ed in user-space, simplifying both kernel and
88+ * user-space implementations significantly.
89+ */
90+ array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages);
91+ if (array_size > PAGE_SIZE)
92+ pages = vmalloc_node(array_size, numa_node);
93+ else
94+ pages = kmalloc_node(array_size, flags, numa_node);
95+ if (!pages)
96+ return NULL;
97+
98+ for (i = 0; i < nr_pages; i++) {
99+ page = alloc_pages_node(numa_node, flags, 0);
100+ if (!page) {
101+ nr_pages = i;
102+ goto err_free_pages;
103+ }
104+ pages[i] = page;
105+ if (i >= nr_meta_pages)
106+ pages[nr_data_pages + i] = page;
107+ }
108+
109+ rb = vmap(pages, nr_meta_pages + 2 * nr_data_pages,
110+ VM_ALLOC | VM_USERMAP, PAGE_KERNEL);
111+ if (rb) {
112+ rb->pages = pages;
113+ rb->nr_pages = nr_pages;
114+ return rb;
115+ }
116+
117+err_free_pages:
118+ for (i = 0; i < nr_pages; i++)
119+ __free_page(pages[i]);
120+ kvfree(pages);
121+ return NULL;
122+}
123+
124+static void bpf_ringbuf_notify(struct irq_work *work)
125+{
126+ struct bpf_ringbuf *rb = container_of(work, struct bpf_ringbuf, work);
127+
128+ wake_up_all(&rb->waitq);
129+}
130+
131+static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, int numa_node)
132+{
133+ struct bpf_ringbuf *rb;
134+
135+ if (!data_sz || !PAGE_ALIGNED(data_sz))
136+ return ERR_PTR(-EINVAL);
137+
138+#ifdef CONFIG_64BIT
139+ /* on 32-bit arch, it's impossible to overflow record's hdr->pgoff */
140+ if (data_sz > RINGBUF_MAX_DATA_SZ)
141+ return ERR_PTR(-E2BIG);
142+#endif
143+
144+ rb = bpf_ringbuf_area_alloc(data_sz, numa_node);
145+ if (!rb)
146+ return ERR_PTR(-ENOMEM);
147+
148+ spin_lock_init(&rb->spinlock);
149+ init_waitqueue_head(&rb->waitq);
150+ init_irq_work(&rb->work, bpf_ringbuf_notify);
151+
152+ rb->mask = data_sz - 1;
153+ rb->consumer_pos = 0;
154+ rb->producer_pos = 0;
155+
156+ return rb;
157+}
158+
159+static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr)
160+{
161+ struct bpf_ringbuf_map *rb_map;
162+ u64 cost;
163+ int err;
164+
165+ if (attr->map_flags & ~RINGBUF_CREATE_FLAG_MASK)
166+ return ERR_PTR(-EINVAL);
167+
168+ if (attr->key_size || attr->value_size ||
169+ attr->max_entries == 0 || !PAGE_ALIGNED(attr->max_entries))
170+ return ERR_PTR(-EINVAL);
171+
172+ rb_map = kzalloc(sizeof(*rb_map), GFP_USER);
173+ if (!rb_map)
174+ return ERR_PTR(-ENOMEM);
175+
176+ bpf_map_init_from_attr(&rb_map->map, attr);
177+
178+ cost = sizeof(struct bpf_ringbuf_map) +
179+ sizeof(struct bpf_ringbuf) +
180+ attr->max_entries;
181+ err = bpf_map_charge_init(&rb_map->map.memory, cost);
182+ if (err)
183+ goto err_free_map;
184+
185+ rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node);
186+ if (IS_ERR(rb_map->rb)) {
187+ err = PTR_ERR(rb_map->rb);
188+ goto err_uncharge;
189+ }
190+
191+ return &rb_map->map;
192+
193+err_uncharge:
194+ bpf_map_charge_finish(&rb_map->map.memory);
195+err_free_map:
196+ kfree(rb_map);
197+ return ERR_PTR(err);
198+}
199+
200+static void bpf_ringbuf_free(struct bpf_ringbuf *rb)
201+{
202+ /* copy pages pointer and nr_pages to local variable, as we are going
203+ * to unmap rb itself with vunmap() below
204+ */
205+ struct page **pages = rb->pages;
206+ int i, nr_pages = rb->nr_pages;
207+
208+ vunmap(rb);
209+ for (i = 0; i < nr_pages; i++)
210+ __free_page(pages[i]);
211+ kvfree(pages);
212+}
213+
214+static void ringbuf_map_free(struct bpf_map *map)
215+{
216+ struct bpf_ringbuf_map *rb_map;
217+
218+ /* at this point bpf_prog->aux->refcnt == 0 and this map->refcnt == 0,
219+ * so the programs (can be more than one that used this map) were
220+ * disconnected from events. Wait for outstanding critical sections in
221+ * these programs to complete
222+ */
223+ synchronize_rcu();
224+
225+ rb_map = container_of(map, struct bpf_ringbuf_map, map);
226+ bpf_ringbuf_free(rb_map->rb);
227+ kfree(rb_map);
228+}
229+
230+static void *ringbuf_map_lookup_elem(struct bpf_map *map, void *key)
231+{
232+ return ERR_PTR(-ENOTSUPP);
233+}
234+
235+static int ringbuf_map_update_elem(struct bpf_map *map, void *key, void *value,
236+ u64 flags)
237+{
238+ return -ENOTSUPP;
239+}
240+
241+static int ringbuf_map_delete_elem(struct bpf_map *map, void *key)
242+{
243+ return -ENOTSUPP;
244+}
245+
246+static int ringbuf_map_get_next_key(struct bpf_map *map, void *key,
247+ void *next_key)
248+{
249+ return -ENOTSUPP;
250+}
251+
252+static size_t bpf_ringbuf_mmap_page_cnt(const struct bpf_ringbuf *rb)
253+{
254+ size_t data_pages = (rb->mask + 1) >> PAGE_SHIFT;
255+
256+ /* consumer page + producer page + 2 x data pages */
257+ return RINGBUF_POS_PAGES + 2 * data_pages;
258+}
259+
260+static int ringbuf_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
261+{
262+ struct bpf_ringbuf_map *rb_map;
263+ size_t mmap_sz;
264+
265+ rb_map = container_of(map, struct bpf_ringbuf_map, map);
266+ mmap_sz = bpf_ringbuf_mmap_page_cnt(rb_map->rb) << PAGE_SHIFT;
267+
268+ if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) > mmap_sz)
269+ return -EINVAL;
270+
271+ return remap_vmalloc_range(vma, rb_map->rb,
272+ vma->vm_pgoff + RINGBUF_PGOFF);
273+}
274+
275+static unsigned long ringbuf_avail_data_sz(struct bpf_ringbuf *rb)
276+{
277+ unsigned long cons_pos, prod_pos;
278+
279+ cons_pos = smp_load_acquire(&rb->consumer_pos);
280+ prod_pos = smp_load_acquire(&rb->producer_pos);
281+ return prod_pos - cons_pos;
282+}
283+
284+static __poll_t ringbuf_map_poll(struct bpf_map *map, struct file *filp,
285+ struct poll_table_struct *pts)
286+{
287+ struct bpf_ringbuf_map *rb_map;
288+
289+ rb_map = container_of(map, struct bpf_ringbuf_map, map);
290+ poll_wait(filp, &rb_map->rb->waitq, pts);
291+
292+ if (ringbuf_avail_data_sz(rb_map->rb))
293+ return EPOLLIN | EPOLLRDNORM;
294+ return 0;
295+}
296+
297+const struct bpf_map_ops ringbuf_map_ops = {
298+ .map_alloc = ringbuf_map_alloc,
299+ .map_free = ringbuf_map_free,
300+ .map_mmap = ringbuf_map_mmap,
301+ .map_poll = ringbuf_map_poll,
302+ .map_lookup_elem = ringbuf_map_lookup_elem,
303+ .map_update_elem = ringbuf_map_update_elem,
304+ .map_delete_elem = ringbuf_map_delete_elem,
305+ .map_get_next_key = ringbuf_map_get_next_key,
306+};
307+
308+/* Given pointer to ring buffer record metadata and struct bpf_ringbuf itself,
309+ * calculate offset from record metadata to ring buffer in pages, rounded
310+ * down. This page offset is stored as part of record metadata and allows to
311+ * restore struct bpf_ringbuf * from record pointer. This page offset is
312+ * stored at offset 4 of record metadata header.
313+ */
314+static size_t bpf_ringbuf_rec_pg_off(struct bpf_ringbuf *rb,
315+ struct bpf_ringbuf_hdr *hdr)
316+{
317+ return ((void *)hdr - (void *)rb) >> PAGE_SHIFT;
318+}
319+
320+/* Given pointer to ring buffer record header, restore pointer to struct
321+ * bpf_ringbuf itself by using page offset stored at offset 4
322+ */
323+static struct bpf_ringbuf *
324+bpf_ringbuf_restore_from_rec(struct bpf_ringbuf_hdr *hdr)
325+{
326+ unsigned long addr = (unsigned long)(void *)hdr;
327+ unsigned long off = (unsigned long)hdr->pg_off << PAGE_SHIFT;
328+
329+ return (void*)((addr & PAGE_MASK) - off);
330+}
331+
332+static void *__bpf_ringbuf_reserve(struct bpf_ringbuf *rb, u64 size)
333+{
334+ unsigned long cons_pos, prod_pos, new_prod_pos, flags;
335+ u32 len, pg_off;
336+ struct bpf_ringbuf_hdr *hdr;
337+
338+ if (unlikely(size > RINGBUF_MAX_RECORD_SZ))
339+ return NULL;
340+
341+ len = round_up(size + BPF_RINGBUF_HDR_SZ, 8);
342+ cons_pos = smp_load_acquire(&rb->consumer_pos);
343+
344+ if (in_nmi()) {
345+ if (!spin_trylock_irqsave(&rb->spinlock, flags))
346+ return NULL;
347+ } else {
348+ spin_lock_irqsave(&rb->spinlock, flags);
349+ }
350+
351+ prod_pos = rb->producer_pos;
352+ new_prod_pos = prod_pos + len;
353+
354+ /* check for out of ringbuf space by ensuring producer position
355+ * doesn't advance more than (ringbuf_size - 1) ahead
356+ */
357+ if (new_prod_pos - cons_pos > rb->mask) {
358+ spin_unlock_irqrestore(&rb->spinlock, flags);
359+ return NULL;
360+ }
361+
362+ hdr = (void *)rb->data + (prod_pos & rb->mask);
363+ pg_off = bpf_ringbuf_rec_pg_off(rb, hdr);
364+ hdr->len = size | BPF_RINGBUF_BUSY_BIT;
365+ hdr->pg_off = pg_off;
366+
367+ /* pairs with consumer's smp_load_acquire() */
368+ smp_store_release(&rb->producer_pos, new_prod_pos);
369+
370+ spin_unlock_irqrestore(&rb->spinlock, flags);
371+
372+ return (void *)hdr + BPF_RINGBUF_HDR_SZ;
373+}
374+
375+BPF_CALL_3(bpf_ringbuf_reserve, struct bpf_map *, map, u64, size, u64, flags)
376+{
377+ struct bpf_ringbuf_map *rb_map;
378+
379+ if (unlikely(flags))
380+ return 0;
381+
382+ rb_map = container_of(map, struct bpf_ringbuf_map, map);
383+ return (unsigned long)__bpf_ringbuf_reserve(rb_map->rb, size);
384+}
385+
386+const struct bpf_func_proto bpf_ringbuf_reserve_proto = {
387+ .func = bpf_ringbuf_reserve,
388+ .ret_type = RET_PTR_TO_ALLOC_MEM_OR_NULL,
389+ .arg1_type = ARG_CONST_MAP_PTR,
390+ .arg2_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
391+ .arg3_type = ARG_ANYTHING,
392+};
393+
394+static void bpf_ringbuf_commit(void *sample, u64 flags, bool discard)
395+{
396+ unsigned long rec_pos, cons_pos;
397+ struct bpf_ringbuf_hdr *hdr;
398+ struct bpf_ringbuf *rb;
399+ u32 new_len;
400+
401+ hdr = sample - BPF_RINGBUF_HDR_SZ;
402+ rb = bpf_ringbuf_restore_from_rec(hdr);
403+ new_len = hdr->len ^ BPF_RINGBUF_BUSY_BIT;
404+ if (discard)
405+ new_len |= BPF_RINGBUF_DISCARD_BIT;
406+
407+ /* update record header with correct final size prefix */
408+ xchg(&hdr->len, new_len);
409+
410+ /* if consumer caught up and is waiting for our record, notify about
411+ * new data availability
412+ */
413+ rec_pos = (void *)hdr - (void *)rb->data;
414+ cons_pos = smp_load_acquire(&rb->consumer_pos) & rb->mask;
415+
416+ if (flags & BPF_RB_FORCE_WAKEUP)
417+ irq_work_queue(&rb->work);
418+ else if (cons_pos == rec_pos && !(flags & BPF_RB_NO_WAKEUP))
419+ irq_work_queue(&rb->work);
420+}
421+
422+BPF_CALL_2(bpf_ringbuf_submit, void *, sample, u64, flags)
423+{
424+ bpf_ringbuf_commit(sample, flags, false /* discard */);
425+ return 0;
426+}
427+
428+const struct bpf_func_proto bpf_ringbuf_submit_proto = {
429+ .func = bpf_ringbuf_submit,
430+ .ret_type = RET_VOID,
431+ .arg1_type = ARG_PTR_TO_ALLOC_MEM,
432+ .arg2_type = ARG_ANYTHING,
433+};
434+
435+BPF_CALL_2(bpf_ringbuf_discard, void *, sample, u64, flags)
436+{
437+ bpf_ringbuf_commit(sample, flags, true /* discard */);
438+ return 0;
439+}
440+
441+const struct bpf_func_proto bpf_ringbuf_discard_proto = {
442+ .func = bpf_ringbuf_discard,
443+ .ret_type = RET_VOID,
444+ .arg1_type = ARG_PTR_TO_ALLOC_MEM,
445+ .arg2_type = ARG_ANYTHING,
446+};
447+
448+BPF_CALL_4(bpf_ringbuf_output, struct bpf_map *, map, void *, data, u64, size,
449+ u64, flags)
450+{
451+ struct bpf_ringbuf_map *rb_map;
452+ void *rec;
453+
454+ if (unlikely(flags & ~(BPF_RB_NO_WAKEUP | BPF_RB_FORCE_WAKEUP)))
455+ return -EINVAL;
456+
457+ rb_map = container_of(map, struct bpf_ringbuf_map, map);
458+ rec = __bpf_ringbuf_reserve(rb_map->rb, size);
459+ if (!rec)
460+ return -EAGAIN;
461+
462+ memcpy(rec, data, size);
463+ bpf_ringbuf_commit(rec, flags, false /* discard */);
464+ return 0;
465+}
466+
467+const struct bpf_func_proto bpf_ringbuf_output_proto = {
468+ .func = bpf_ringbuf_output,
469+ .ret_type = RET_INTEGER,
470+ .arg1_type = ARG_CONST_MAP_PTR,
471+ .arg2_type = ARG_PTR_TO_MEM,
472+ .arg3_type = ARG_CONST_SIZE_OR_ZERO,
473+ .arg4_type = ARG_ANYTHING,
474+};
475+
476+BPF_CALL_2(bpf_ringbuf_query, struct bpf_map *, map, u64, flags)
477+{
478+ struct bpf_ringbuf *rb;
479+
480+ rb = container_of(map, struct bpf_ringbuf_map, map)->rb;
481+
482+ switch (flags) {
483+ case BPF_RB_AVAIL_DATA:
484+ return ringbuf_avail_data_sz(rb);
485+ case BPF_RB_RING_SIZE:
486+ return rb->mask + 1;
487+ case BPF_RB_CONS_POS:
488+ return smp_load_acquire(&rb->consumer_pos);
489+ case BPF_RB_PROD_POS:
490+ return smp_load_acquire(&rb->producer_pos);
491+ default:
492+ return 0;
493+ }
494+}
495+
496+const struct bpf_func_proto bpf_ringbuf_query_proto = {
497+ .func = bpf_ringbuf_query,
498+ .ret_type = RET_INTEGER,
499+ .arg1_type = ARG_CONST_MAP_PTR,
500+ .arg2_type = ARG_ANYTHING,
501+};
kernel/bpf/syscall.c+12−0
@@ -26,6 +26,7 @@ · 25 unmodified lines
2626 #include <linux/audit.h>
2727 #include <uapi/linux/btf.h>
2828 #include <linux/bpf_lsm.h>
29+#include <linux/poll.h>
2930
3031 #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \
3132 (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \
@@ -662,6 +663,16 @@ out: · 630 unmodified lines
662663 return err;
663664 }
664665
666+static __poll_t bpf_map_poll(struct file *filp, struct poll_table_struct *pts)
667+{
668+ struct bpf_map *map = filp->private_data;
669+
670+ if (map->ops->map_poll)
671+ return map->ops->map_poll(map, filp, pts);
672+
673+ return EPOLLERR;
674+}
675+
665676 const struct file_operations bpf_map_fops = {
666677 #ifdef CONFIG_PROC_FS
667678 .show_fdinfo = bpf_map_show_fdinfo,
@@ -670,6 +681,7 @@ const struct file_operations bpf_map_fops = { · 2 unmodified lines
670681 .read = bpf_dummy_read,
671682 .write = bpf_dummy_write,
672683 .mmap = bpf_map_mmap,
684+ .poll = bpf_map_poll,
673685 };
674686
675687 int bpf_map_new_fd(struct bpf_map *map, int flags)
kernel/bpf/verifier.c+146−49
@@ -233,6 +233,7 @@ struct bpf_call_arg_meta { · 232 unmodified lines
233233 bool pkt_access;
234234 int regno;
235235 int access_size;
236+ int mem_size;
236237 u64 msize_max_value;
237238 int ref_obj_id;
238239 int func_id;
@@ -408,7 +409,8 @@ static bool reg_type_may_be_null(enum bpf_reg_type type) · 169 unmodified lines
408409 type == PTR_TO_SOCKET_OR_NULL ||
409410 type == PTR_TO_SOCK_COMMON_OR_NULL ||
410411 type == PTR_TO_TCP_SOCK_OR_NULL ||
411− type == PTR_TO_BTF_ID_OR_NULL;
412+ type == PTR_TO_BTF_ID_OR_NULL ||
413+ type == PTR_TO_MEM_OR_NULL;
412414 }
413415
414416 static bool reg_may_point_to_spin_lock(const struct bpf_reg_state *reg)
@@ -422,7 +424,9 @@ static bool reg_type_may_be_refcounted_or_null(enum bpf_reg_type type) · 7 unmodified lines
422424 return type == PTR_TO_SOCKET ||
423425 type == PTR_TO_SOCKET_OR_NULL ||
424426 type == PTR_TO_TCP_SOCK ||
425− type == PTR_TO_TCP_SOCK_OR_NULL;
427+ type == PTR_TO_TCP_SOCK_OR_NULL ||
428+ type == PTR_TO_MEM ||
429+ type == PTR_TO_MEM_OR_NULL;
426430 }
427431
428432 static bool arg_type_may_be_refcounted(enum bpf_arg_type type)
@@ -436,7 +440,9 @@ static bool arg_type_may_be_refcounted(enum bpf_arg_type type) · 7 unmodified lines
436440 */
437441 static bool is_release_function(enum bpf_func_id func_id)
438442 {
439− return func_id == BPF_FUNC_sk_release;
443+ return func_id == BPF_FUNC_sk_release ||
444+ func_id == BPF_FUNC_ringbuf_submit ||
445+ func_id == BPF_FUNC_ringbuf_discard;
440446 }
441447
442448 static bool may_be_acquire_function(enum bpf_func_id func_id)
@@ -444,7 +450,8 @@ static bool may_be_acquire_function(enum bpf_func_id func_id) · 1 unmodified line
444450 return func_id == BPF_FUNC_sk_lookup_tcp ||
445451 func_id == BPF_FUNC_sk_lookup_udp ||
446452 func_id == BPF_FUNC_skc_lookup_tcp ||
447− func_id == BPF_FUNC_map_lookup_elem;
453+ func_id == BPF_FUNC_map_lookup_elem ||
454+ func_id == BPF_FUNC_ringbuf_reserve;
448455 }
449456
450457 static bool is_acquire_function(enum bpf_func_id func_id,
@@ -454,7 +461,8 @@ static bool is_acquire_function(enum bpf_func_id func_id, · 3 unmodified lines
454461
455462 if (func_id == BPF_FUNC_sk_lookup_tcp ||
456463 func_id == BPF_FUNC_sk_lookup_udp ||
457− func_id == BPF_FUNC_skc_lookup_tcp)
464+ func_id == BPF_FUNC_skc_lookup_tcp ||
465+ func_id == BPF_FUNC_ringbuf_reserve)
458466 return true;
459467
460468 if (func_id == BPF_FUNC_map_lookup_elem &&
@@ -494,6 +502,8 @@ static const char * const reg_type_str[] = { · 33 unmodified lines
494502 [PTR_TO_XDP_SOCK] = "xdp_sock",
495503 [PTR_TO_BTF_ID] = "ptr_",
496504 [PTR_TO_BTF_ID_OR_NULL] = "ptr_or_null_",
505+ [PTR_TO_MEM] = "mem",
506+ [PTR_TO_MEM_OR_NULL] = "mem_or_null",
497507 };
498508
499509 static char slot_type_char[] = {
@@ -2468,32 +2478,49 @@ static int check_map_access_type(struct bpf_verifier_env *env, u32 regno, · 1968 unmodified lines
24682478 return 0;
24692479 }
24702480
2471−/* check read/write into map element returned by bpf_map_lookup_elem() */
2472−static int __check_map_access(struct bpf_verifier_env *env, u32 regno, int off,
2473− int size, bool zero_size_allowed)
2481+/* check read/write into memory region (e.g., map value, ringbuf sample, etc) */
2482+static int __check_mem_access(struct bpf_verifier_env *env, int regno,
2483+ int off, int size, u32 mem_size,
2484+ bool zero_size_allowed)
24742485 {
2475− struct bpf_reg_state *regs = cur_regs(env);
2476− struct bpf_map *map = regs[regno].map_ptr;
2486+ bool size_ok = size > 0 || (size == 0 && zero_size_allowed);
2487+ struct bpf_reg_state *reg;
2488+
2489+ if (off >= 0 && size_ok && (u64)off + size <= mem_size)
2490+ return 0;
24772491
2478− if (off < 0 || size < 0 || (size == 0 && !zero_size_allowed) ||
2479− off + size > map->value_size) {
2492+ reg = &cur_regs(env)[regno];
2493+ switch (reg->type) {
2494+ case PTR_TO_MAP_VALUE:
24802495 verbose(env, "invalid access to map value, value_size=%d off=%d size=%d\n",
2481− map->value_size, off, size);
2482− return -EACCES;
2496+ mem_size, off, size);
2497+ break;
2498+ case PTR_TO_PACKET:
2499+ case PTR_TO_PACKET_META:
2500+ case PTR_TO_PACKET_END:
2501+ verbose(env, "invalid access to packet, off=%d size=%d, R%d(id=%d,off=%d,r=%d)\n",
2502+ off, size, regno, reg->id, off, mem_size);
2503+ break;
2504+ case PTR_TO_MEM:
2505+ default:
2506+ verbose(env, "invalid access to memory, mem_size=%u off=%d size=%d\n",
2507+ mem_size, off, size);
24832508 }
2484− return 0;
2509+
2510+ return -EACCES;
24852511 }
24862512
2487−/* check read/write into a map element with possible variable offset */
2488−static int check_map_access(struct bpf_verifier_env *env, u32 regno,
2489− int off, int size, bool zero_size_allowed)
2513+/* check read/write into a memory region with possible variable offset */
2514+static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno,
2515+ int off, int size, u32 mem_size,
2516+ bool zero_size_allowed)
24902517 {
24912518 struct bpf_verifier_state *vstate = env->cur_state;
24922519 struct bpf_func_state *state = vstate->frame[vstate->curframe];
24932520 struct bpf_reg_state *reg = &state->regs[regno];
24942521 int err;
24952522
2496− /* We may have adjusted the register to this map value, so we
2523+ /* We may have adjusted the register pointing to memory region, so we
24972524 * need to try adding each of min_value and max_value to off
24982525 * to make sure our theoretical access will be safe.
24992526 */
@@ -2514,10 +2541,10 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 14 unmodified lines
25142541 regno);
25152542 return -EACCES;
25162543 }
2517− err = __check_map_access(env, regno, reg->smin_value + off, size,
2518− zero_size_allowed);
2544+ err = __check_mem_access(env, regno, reg->smin_value + off, size,
2545+ mem_size, zero_size_allowed);
25192546 if (err) {
2520− verbose(env, "R%d min value is outside of the array range\n",
2547+ verbose(env, "R%d min value is outside of the allowed memory range\n",
25212548 regno);
25222549 return err;
25232550 }
@@ -2527,18 +2554,38 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 3 unmodified lines
25272554 * If reg->umax_value + off could overflow, treat that as unbounded too.
25282555 */
25292556 if (reg->umax_value >= BPF_MAX_VAR_OFF) {
2530− verbose(env, "R%d unbounded memory access, make sure to bounds check any array access into a map\n",
2557+ verbose(env, "R%d unbounded memory access, make sure to bounds check any such access\n",
25312558 regno);
25322559 return -EACCES;
25332560 }
2534− err = __check_map_access(env, regno, reg->umax_value + off, size,
2535− zero_size_allowed);
2536− if (err)
2537− verbose(env, "R%d max value is outside of the array range\n",
2561+ err = __check_mem_access(env, regno, reg->umax_value + off, size,
2562+ mem_size, zero_size_allowed);
2563+ if (err) {
2564+ verbose(env, "R%d max value is outside of the allowed memory range\n",
25382565 regno);
2566+ return err;
2567+ }
2568+
2569+ return 0;
2570+}
25392571
2540− if (map_value_has_spin_lock(reg->map_ptr)) {
2541− u32 lock = reg->map_ptr->spin_lock_off;
2572+/* check read/write into a map element with possible variable offset */
2573+static int check_map_access(struct bpf_verifier_env *env, u32 regno,
2574+ int off, int size, bool zero_size_allowed)
2575+{
2576+ struct bpf_verifier_state *vstate = env->cur_state;
2577+ struct bpf_func_state *state = vstate->frame[vstate->curframe];
2578+ struct bpf_reg_state *reg = &state->regs[regno];
2579+ struct bpf_map *map = reg->map_ptr;
2580+ int err;
2581+
2582+ err = check_mem_region_access(env, regno, off, size, map->value_size,
2583+ zero_size_allowed);
2584+ if (err)
2585+ return err;
2586+
2587+ if (map_value_has_spin_lock(map)) {
2588+ u32 lock = map->spin_lock_off;
25422589
25432590 /* if any part of struct bpf_spin_lock can be touched by
25442591 * load/store reject this program.
@@ -2596,21 +2643,6 @@ static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, · 51 unmodified lines
25962643 }
25972644 }
25982645
2599−static int __check_packet_access(struct bpf_verifier_env *env, u32 regno,
2600− int off, int size, bool zero_size_allowed)
2601−{
2602− struct bpf_reg_state *regs = cur_regs(env);
2603− struct bpf_reg_state *reg = &regs[regno];
2604−
2605− if (off < 0 || size < 0 || (size == 0 && !zero_size_allowed) ||
2606− (u64)off + size > reg->range) {
2607− verbose(env, "invalid access to packet, off=%d size=%d, R%d(id=%d,off=%d,r=%d)\n",
2608− off, size, regno, reg->id, reg->off, reg->range);
2609− return -EACCES;
2610− }
2611− return 0;
2612−}
2613−
26142646 static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off,
26152647 int size, bool zero_size_allowed)
26162648 {
@@ -2631,16 +2663,17 @@ static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off, · 14 unmodified lines
26312663 regno);
26322664 return -EACCES;
26332665 }
2634− err = __check_packet_access(env, regno, off, size, zero_size_allowed);
2666+ err = __check_mem_access(env, regno, off, size, reg->range,
2667+ zero_size_allowed);
26352668 if (err) {
26362669 verbose(env, "R%d offset is outside of the packet\n", regno);
26372670 return err;
26382671 }
26392672
2640− /* __check_packet_access has made sure "off + size - 1" is within u16.
2673+ /* __check_mem_access has made sure "off + size - 1" is within u16.
26412674 * reg->umax_value can't be bigger than MAX_PACKET_OFF which is 0xffff,
26422675 * otherwise find_good_pkt_pointers would have refused to set range info
2643− * that __check_packet_access would have rejected this pkt access.
2676+ * that __check_mem_access would have rejected this pkt access.
26442677 * Therefore, "off + reg->umax_value + size - 1" won't overflow u32.
26452678 */
26462679 env->prog->aux->max_pkt_offset =
@@ -3220,6 +3253,16 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 573 unmodified lines
32203253 mark_reg_unknown(env, regs, value_regno);
32213254 }
32223255 }
3256+ } else if (reg->type == PTR_TO_MEM) {
3257+ if (t == BPF_WRITE && value_regno >= 0 &&
3258+ is_pointer_value(env, value_regno)) {
3259+ verbose(env, "R%d leaks addr into mem\n", value_regno);
3260+ return -EACCES;
3261+ }
3262+ err = check_mem_region_access(env, regno, off, size,
3263+ reg->mem_size, false);
3264+ if (!err && t == BPF_READ && value_regno >= 0)
3265+ mark_reg_unknown(env, regs, value_regno);
32233266 } else if (reg->type == PTR_TO_CTX) {
32243267 enum bpf_reg_type reg_type = SCALAR_VALUE;
32253268 u32 btf_id = 0;
@@ -3557,6 +3600,10 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 331 unmodified lines
35573600 return -EACCES;
35583601 return check_map_access(env, regno, reg->off, access_size,
35593602 zero_size_allowed);
3603+ case PTR_TO_MEM:
3604+ return check_mem_region_access(env, regno, reg->off,
3605+ access_size, reg->mem_size,
3606+ zero_size_allowed);
35603607 default: /* scalar_value|ptr_to_stack or invalid ptr */
35613608 return check_stack_boundary(env, regno, access_size,
35623609 zero_size_allowed, meta);
@@ -3661,6 +3708,17 @@ static bool arg_type_is_mem_size(enum bpf_arg_type type) · 98 unmodified lines
36613708 type == ARG_CONST_SIZE_OR_ZERO;
36623709 }
36633710
3711+static bool arg_type_is_alloc_mem_ptr(enum bpf_arg_type type)
3712+{
3713+ return type == ARG_PTR_TO_ALLOC_MEM ||
3714+ type == ARG_PTR_TO_ALLOC_MEM_OR_NULL;
3715+}
3716+
3717+static bool arg_type_is_alloc_size(enum bpf_arg_type type)
3718+{
3719+ return type == ARG_CONST_ALLOC_SIZE_OR_ZERO;
3720+}
3721+
36643722 static bool arg_type_is_int_ptr(enum bpf_arg_type type)
36653723 {
36663724 return type == ARG_PTR_TO_INT ||
@@ -3720,7 +3778,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, · 53 unmodified lines
37203778 type != expected_type)
37213779 goto err_type;
37223780 } else if (arg_type == ARG_CONST_SIZE ||
3723− arg_type == ARG_CONST_SIZE_OR_ZERO) {
3781+ arg_type == ARG_CONST_SIZE_OR_ZERO ||
3782+ arg_type == ARG_CONST_ALLOC_SIZE_OR_ZERO) {
37243783 expected_type = SCALAR_VALUE;
37253784 if (type != expected_type)
37263785 goto err_type;
@@ -3791,13 +3850,29 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, · 64 unmodified lines
37913850 * happens during stack boundary checking.
37923851 */
37933852 if (register_is_null(reg) &&
3794− arg_type == ARG_PTR_TO_MEM_OR_NULL)
3853+ (arg_type == ARG_PTR_TO_MEM_OR_NULL ||
3854+ arg_type == ARG_PTR_TO_ALLOC_MEM_OR_NULL))
37953855 /* final test in check_stack_boundary() */;
37963856 else if (!type_is_pkt_pointer(type) &&
37973857 type != PTR_TO_MAP_VALUE &&
3858+ type != PTR_TO_MEM &&
37983859 type != expected_type)
37993860 goto err_type;
38003861 meta->raw_mode = arg_type == ARG_PTR_TO_UNINIT_MEM;
3862+ } else if (arg_type_is_alloc_mem_ptr(arg_type)) {
3863+ expected_type = PTR_TO_MEM;
3864+ if (register_is_null(reg) &&
3865+ arg_type == ARG_PTR_TO_ALLOC_MEM_OR_NULL)
3866+ /* final test in check_stack_boundary() */;
3867+ else if (type != expected_type)
3868+ goto err_type;
3869+ if (meta->ref_obj_id) {
3870+ verbose(env, "verifier internal error: more than one arg with ref_obj_id R%d %u %u\n",
3871+ regno, reg->ref_obj_id,
3872+ meta->ref_obj_id);
3873+ return -EFAULT;
3874+ }
3875+ meta->ref_obj_id = reg->ref_obj_id;
38013876 } else if (arg_type_is_int_ptr(arg_type)) {
38023877 expected_type = PTR_TO_STACK;
38033878 if (!type_is_pkt_pointer(type) &&
@@ -3893,6 +3968,13 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, · 89 unmodified lines
38933968 zero_size_allowed, meta);
38943969 if (!err)
38953970 err = mark_chain_precision(env, regno);
3971+ } else if (arg_type_is_alloc_size(arg_type)) {
3972+ if (!tnum_is_const(reg->var_off)) {
3973+ verbose(env, "R%d unbounded size, use 'var &= const' or 'if (var < const)'\n",
3974+ regno);
3975+ return -EACCES;
3976+ }
3977+ meta->mem_size = reg->var_off.value;
38963978 } else if (arg_type_is_int_ptr(arg_type)) {
38973979 int size = int_ptr_type_to_size(arg_type);
38983980
@@ -3929,6 +4011,14 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 30 unmodified lines
39294011 func_id != BPF_FUNC_xdp_output)
39304012 goto error;
39314013 break;
4014+ case BPF_MAP_TYPE_RINGBUF:
4015+ if (func_id != BPF_FUNC_ringbuf_output &&
4016+ func_id != BPF_FUNC_ringbuf_reserve &&
4017+ func_id != BPF_FUNC_ringbuf_submit &&
4018+ func_id != BPF_FUNC_ringbuf_discard &&
4019+ func_id != BPF_FUNC_ringbuf_query)
4020+ goto error;
4021+ break;
39324022 case BPF_MAP_TYPE_STACK_TRACE:
39334023 if (func_id != BPF_FUNC_get_stackid)
39344024 goto error;
@@ -4655,6 +4745,11 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 720 unmodified lines
46554745 mark_reg_known_zero(env, regs, BPF_REG_0);
46564746 regs[BPF_REG_0].type = PTR_TO_TCP_SOCK_OR_NULL;
46574747 regs[BPF_REG_0].id = ++env->id_gen;
4748+ } else if (fn->ret_type == RET_PTR_TO_ALLOC_MEM_OR_NULL) {
4749+ mark_reg_known_zero(env, regs, BPF_REG_0);
4750+ regs[BPF_REG_0].type = PTR_TO_MEM_OR_NULL;
4751+ regs[BPF_REG_0].id = ++env->id_gen;
4752+ regs[BPF_REG_0].mem_size = meta.mem_size;
46584753 } else {
46594754 verbose(env, "unknown return type %d of func %s#%d\n",
46604755 fn->ret_type, func_id_name(func_id), func_id);
@@ -6611,6 +6706,8 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state, · 1950 unmodified lines
66116706 reg->type = PTR_TO_TCP_SOCK;
66126707 } else if (reg->type == PTR_TO_BTF_ID_OR_NULL) {
66136708 reg->type = PTR_TO_BTF_ID;
6709+ } else if (reg->type == PTR_TO_MEM_OR_NULL) {
6710+ reg->type = PTR_TO_MEM;
66146711 }
66156712 if (is_null) {
66166713 /* We don't need id and ref_obj_id from this point
kernel/trace/bpf_trace.c+10−0
@@ -1088,6 +1088,16 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 1087 unmodified lines
10881088 return &bpf_perf_event_read_value_proto;
10891089 case BPF_FUNC_get_ns_current_pid_tgid:
10901090 return &bpf_get_ns_current_pid_tgid_proto;
1091+ case BPF_FUNC_ringbuf_output:
1092+ return &bpf_ringbuf_output_proto;
1093+ case BPF_FUNC_ringbuf_reserve:
1094+ return &bpf_ringbuf_reserve_proto;
1095+ case BPF_FUNC_ringbuf_submit:
1096+ return &bpf_ringbuf_submit_proto;
1097+ case BPF_FUNC_ringbuf_discard:
1098+ return &bpf_ringbuf_discard_proto;
1099+ case BPF_FUNC_ringbuf_query:
1100+ return &bpf_ringbuf_query_proto;
10911101 default:
10921102 return NULL;
10931103 }
tools/include/uapi/linux/bpf.h+83−1
@@ -147,6 +147,7 @@ enum bpf_map_type { · 146 unmodified lines
147147 BPF_MAP_TYPE_SK_STORAGE,
148148 BPF_MAP_TYPE_DEVMAP_HASH,
149149 BPF_MAP_TYPE_STRUCT_OPS,
150+ BPF_MAP_TYPE_RINGBUF,
150151 };
151152
152153 /* Note that tracing related programs such as
@@ -3157,6 +3158,59 @@ union bpf_attr { · 3004 unmodified lines
31573158 * **bpf_sk_cgroup_id**\ ().
31583159 * Return
31593160 * The id is returned or 0 in case the id could not be retrieved.
3161+ *
3162+ * void *bpf_ringbuf_output(void *ringbuf, void *data, u64 size, u64 flags)
3163+ * Description
3164+ * Copy *size* bytes from *data* into a ring buffer *ringbuf*.
3165+ * If BPF_RB_NO_WAKEUP is specified in *flags*, no notification of
3166+ * new data availability is sent.
3167+ * IF BPF_RB_FORCE_WAKEUP is specified in *flags*, notification of
3168+ * new data availability is sent unconditionally.
3169+ * Return
3170+ * 0, on success;
3171+ * < 0, on error.
3172+ *
3173+ * void *bpf_ringbuf_reserve(void *ringbuf, u64 size, u64 flags)
3174+ * Description
3175+ * Reserve *size* bytes of payload in a ring buffer *ringbuf*.
3176+ * Return
3177+ * Valid pointer with *size* bytes of memory available; NULL,
3178+ * otherwise.
3179+ *
3180+ * void bpf_ringbuf_submit(void *data, u64 flags)
3181+ * Description
3182+ * Submit reserved ring buffer sample, pointed to by *data*.
3183+ * If BPF_RB_NO_WAKEUP is specified in *flags*, no notification of
3184+ * new data availability is sent.
3185+ * IF BPF_RB_FORCE_WAKEUP is specified in *flags*, notification of
3186+ * new data availability is sent unconditionally.
3187+ * Return
3188+ * Nothing. Always succeeds.
3189+ *
3190+ * void bpf_ringbuf_discard(void *data, u64 flags)
3191+ * Description
3192+ * Discard reserved ring buffer sample, pointed to by *data*.
3193+ * If BPF_RB_NO_WAKEUP is specified in *flags*, no notification of
3194+ * new data availability is sent.
3195+ * IF BPF_RB_FORCE_WAKEUP is specified in *flags*, notification of
3196+ * new data availability is sent unconditionally.
3197+ * Return
3198+ * Nothing. Always succeeds.
3199+ *
3200+ * u64 bpf_ringbuf_query(void *ringbuf, u64 flags)
3201+ * Description
3202+ * Query various characteristics of provided ring buffer. What
3203+ * exactly is queries is determined by *flags*:
3204+ * - BPF_RB_AVAIL_DATA - amount of data not yet consumed;
3205+ * - BPF_RB_RING_SIZE - the size of ring buffer;
3206+ * - BPF_RB_CONS_POS - consumer position (can wrap around);
3207+ * - BPF_RB_PROD_POS - producer(s) position (can wrap around);
3208+ * Data returned is just a momentary snapshots of actual values
3209+ * and could be inaccurate, so this facility should be used to
3210+ * power heuristics and for reporting, not to make 100% correct
3211+ * calculation.
3212+ * Return
3213+ * Requested value, or 0, if flags are not recognized.
31603214 */
31613215 #define __BPF_FUNC_MAPPER(FN) \
31623216 FN(unspec), \
@@ -3288,7 +3342,12 @@ union bpf_attr { · 125 unmodified lines
32883342 FN(seq_printf), \
32893343 FN(seq_write), \
32903344 FN(sk_cgroup_id), \
3291− FN(sk_ancestor_cgroup_id),
3345+ FN(sk_ancestor_cgroup_id), \
3346+ FN(ringbuf_output), \
3347+ FN(ringbuf_reserve), \
3348+ FN(ringbuf_submit), \
3349+ FN(ringbuf_discard), \
3350+ FN(ringbuf_query),
32923351
32933352 /* integer value in 'imm' field of BPF_CALL instruction selects which helper
32943353 * function eBPF program intends to call
@@ -3398,6 +3457,29 @@ enum { · 103 unmodified lines
33983457 BPF_F_GET_BRANCH_RECORDS_SIZE = (1ULL << 0),
33993458 };
34003459
3460+/* BPF_FUNC_bpf_ringbuf_commit, BPF_FUNC_bpf_ringbuf_discard, and
3461+ * BPF_FUNC_bpf_ringbuf_output flags.
3462+ */
3463+enum {
3464+ BPF_RB_NO_WAKEUP = (1ULL << 0),
3465+ BPF_RB_FORCE_WAKEUP = (1ULL << 1),
3466+};
3467+
3468+/* BPF_FUNC_bpf_ringbuf_query flags */
3469+enum {
3470+ BPF_RB_AVAIL_DATA = 0,
3471+ BPF_RB_RING_SIZE = 1,
3472+ BPF_RB_CONS_POS = 2,
3473+ BPF_RB_PROD_POS = 3,
3474+};
3475+
3476+/* BPF ring buffer constants */
3477+enum {
3478+ BPF_RINGBUF_BUSY_BIT = (1U << 31),
3479+ BPF_RINGBUF_DISCARD_BIT = (1U << 30),
3480+ BPF_RINGBUF_HDR_SZ = 8,
3481+};
3482+
34013483 /* Mode for BPF_FUNC_skb_adjust_room helper. */
34023484 enum bpf_adj_room_mode {
34033485 BPF_ADJ_ROOM_NET,
tools/lib/bpf/Build+1−1
@@ -1,3 +1,3 @@
11 libbpf-y := libbpf.o bpf.o nlattr.o btf.o libbpf_errno.o str_error.o \
22 netlink.o bpf_prog_linfo.o libbpf_probes.o xsk.o hashmap.o \
3− btf_dump.o
3+ btf_dump.o ringbuf.o
tools/lib/bpf/libbpf.h+21−0
@@ -478,6 +478,27 @@ LIBBPF_API int bpf_get_link_xdp_id(int ifindex, __u32 *prog_id, __u32 flags); · 477 unmodified lines
478478 LIBBPF_API int bpf_get_link_xdp_info(int ifindex, struct xdp_link_info *info,
479479 size_t info_size, __u32 flags);
480480
481+/* Ring buffer APIs */
482+struct ring_buffer;
483+
484+typedef int (*ring_buffer_sample_fn)(void *ctx, void *data, size_t size);
485+
486+struct ring_buffer_opts {
487+ size_t sz; /* size of this struct, for forward/backward compatiblity */
488+};
489+
490+#define ring_buffer_opts__last_field sz
491+
492+LIBBPF_API struct ring_buffer *
493+ring_buffer__new(int map_fd, ring_buffer_sample_fn sample_cb, void *ctx,
494+ const struct ring_buffer_opts *opts);
495+LIBBPF_API void ring_buffer__free(struct ring_buffer *rb);
496+LIBBPF_API int ring_buffer__add(struct ring_buffer *rb, int map_fd,
497+ ring_buffer_sample_fn sample_cb, void *ctx);
498+LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms);
499+LIBBPF_API int ring_buffer__consume(struct ring_buffer *rb);
500+
501+/* Perf buffer APIs */
481502 struct perf_buffer;
482503
483504 typedef void (*perf_buffer_sample_fn)(void *ctx, int cpu,
tools/lib/bpf/libbpf.map+5−0
@@ -263,4 +263,9 @@ LIBBPF_0.0.9 { · 262 unmodified lines
263263 bpf_link_get_next_id;
264264 bpf_program__attach_iter;
265265 perf_buffer__consume;
266+ ring_buffer__add;
267+ ring_buffer__consume;
268+ ring_buffer__free;
269+ ring_buffer__new;
270+ ring_buffer__poll;
266271 } LIBBPF_0.0.8;
tools/lib/bpf/libbpf_probes.c+5−0
@@ -238,6 +238,11 @@ bool bpf_probe_map_type(enum bpf_map_type map_type, __u32 ifindex) · 237 unmodified lines
238238 if (btf_fd < 0)
239239 return false;
240240 break;
241+ case BPF_MAP_TYPE_RINGBUF:
242+ key_size = 0;
243+ value_size = 0;
244+ max_entries = 4096;
245+ break;
241246 case BPF_MAP_TYPE_UNSPEC:
242247 case BPF_MAP_TYPE_HASH:
243248 case BPF_MAP_TYPE_ARRAY:
tools/lib/bpf/ringbuf.c+285−0
@@ -0,0 +1,285 @@
1+// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
2+/*
3+ * Ring buffer operations.
4+ *
5+ * Copyright (C) 2020 Facebook, Inc.
6+ */
7+#include <stdlib.h>
8+#include <stdio.h>
9+#include <errno.h>
10+#include <unistd.h>
11+#include <linux/err.h>
12+#include <linux/bpf.h>
13+#include <asm/barrier.h>
14+#include <sys/mman.h>
15+#include <sys/epoll.h>
16+#include <tools/libc_compat.h>
17+
18+#include "libbpf.h"
19+#include "libbpf_internal.h"
20+#include "bpf.h"
21+
22+/* make sure libbpf doesn't use kernel-only integer typedefs */
23+#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
24+
25+struct ring {
26+ ring_buffer_sample_fn sample_cb;
27+ void *ctx;
28+ void *data;
29+ unsigned long *consumer_pos;
30+ unsigned long *producer_pos;
31+ unsigned long mask;
32+ int map_fd;
33+};
34+
35+struct ring_buffer {
36+ struct epoll_event *events;
37+ struct ring *rings;
38+ size_t page_size;
39+ int epoll_fd;
40+ int ring_cnt;
41+};
42+
43+static void ringbuf_unmap_ring(struct ring_buffer *rb, struct ring *r)
44+{
45+ if (r->consumer_pos) {
46+ munmap(r->consumer_pos, rb->page_size);
47+ r->consumer_pos = NULL;
48+ }
49+ if (r->producer_pos) {
50+ munmap(r->producer_pos, rb->page_size + 2 * (r->mask + 1));
51+ r->producer_pos = NULL;
52+ }
53+}
54+
55+/* Add extra RINGBUF maps to this ring buffer manager */
56+int ring_buffer__add(struct ring_buffer *rb, int map_fd,
57+ ring_buffer_sample_fn sample_cb, void *ctx)
58+{
59+ struct bpf_map_info info;
60+ __u32 len = sizeof(info);
61+ struct epoll_event *e;
62+ struct ring *r;
63+ void *tmp;
64+ int err;
65+
66+ memset(&info, 0, sizeof(info));
67+
68+ err = bpf_obj_get_info_by_fd(map_fd, &info, &len);
69+ if (err) {
70+ err = -errno;
71+ pr_warn("ringbuf: failed to get map info for fd=%d: %d\n",
72+ map_fd, err);
73+ return err;
74+ }
75+
76+ if (info.type != BPF_MAP_TYPE_RINGBUF) {
77+ pr_warn("ringbuf: map fd=%d is not BPF_MAP_TYPE_RINGBUF\n",
78+ map_fd);
79+ return -EINVAL;
80+ }
81+
82+ tmp = reallocarray(rb->rings, rb->ring_cnt + 1, sizeof(*rb->rings));
83+ if (!tmp)
84+ return -ENOMEM;
85+ rb->rings = tmp;
86+
87+ tmp = reallocarray(rb->events, rb->ring_cnt + 1, sizeof(*rb->events));
88+ if (!tmp)
89+ return -ENOMEM;
90+ rb->events = tmp;
91+
92+ r = &rb->rings[rb->ring_cnt];
93+ memset(r, 0, sizeof(*r));
94+
95+ r->map_fd = map_fd;
96+ r->sample_cb = sample_cb;
97+ r->ctx = ctx;
98+ r->mask = info.max_entries - 1;
99+
100+ /* Map writable consumer page */
101+ tmp = mmap(NULL, rb->page_size, PROT_READ | PROT_WRITE, MAP_SHARED,
102+ map_fd, 0);
103+ if (tmp == MAP_FAILED) {
104+ err = -errno;
105+ pr_warn("ringbuf: failed to mmap consumer page for map fd=%d: %d\n",
106+ map_fd, err);
107+ return err;
108+ }
109+ r->consumer_pos = tmp;
110+
111+ /* Map read-only producer page and data pages. We map twice as big
112+ * data size to allow simple reading of samples that wrap around the
113+ * end of a ring buffer. See kernel implementation for details.
114+ * */
115+ tmp = mmap(NULL, rb->page_size + 2 * info.max_entries, PROT_READ,
116+ MAP_SHARED, map_fd, rb->page_size);
117+ if (tmp == MAP_FAILED) {
118+ err = -errno;
119+ ringbuf_unmap_ring(rb, r);
120+ pr_warn("ringbuf: failed to mmap data pages for map fd=%d: %d\n",
121+ map_fd, err);
122+ return err;
123+ }
124+ r->producer_pos = tmp;
125+ r->data = tmp + rb->page_size;
126+
127+ e = &rb->events[rb->ring_cnt];
128+ memset(e, 0, sizeof(*e));
129+
130+ e->events = EPOLLIN;
131+ e->data.fd = rb->ring_cnt;
132+ if (epoll_ctl(rb->epoll_fd, EPOLL_CTL_ADD, map_fd, e) < 0) {
133+ err = -errno;
134+ ringbuf_unmap_ring(rb, r);
135+ pr_warn("ringbuf: failed to epoll add map fd=%d: %d\n",
136+ map_fd, err);
137+ return err;
138+ }
139+
140+ rb->ring_cnt++;
141+ return 0;
142+}
143+
144+void ring_buffer__free(struct ring_buffer *rb)
145+{
146+ int i;
147+
148+ if (!rb)
149+ return;
150+
151+ for (i = 0; i < rb->ring_cnt; ++i)
152+ ringbuf_unmap_ring(rb, &rb->rings[i]);
153+ if (rb->epoll_fd >= 0)
154+ close(rb->epoll_fd);
155+
156+ free(rb->events);
157+ free(rb->rings);
158+ free(rb);
159+}
160+
161+struct ring_buffer *
162+ring_buffer__new(int map_fd, ring_buffer_sample_fn sample_cb, void *ctx,
163+ const struct ring_buffer_opts *opts)
164+{
165+ struct ring_buffer *rb;
166+ int err;
167+
168+ if (!OPTS_VALID(opts, ring_buffer_opts))
169+ return NULL;
170+
171+ rb = calloc(1, sizeof(*rb));
172+ if (!rb)
173+ return NULL;
174+
175+ rb->page_size = getpagesize();
176+
177+ rb->epoll_fd = epoll_create1(EPOLL_CLOEXEC);
178+ if (rb->epoll_fd < 0) {
179+ err = -errno;
180+ pr_warn("ringbuf: failed to create epoll instance: %d\n", err);
181+ goto err_out;
182+ }
183+
184+ err = ring_buffer__add(rb, map_fd, sample_cb, ctx);
185+ if (err)
186+ goto err_out;
187+
188+ return rb;
189+
190+err_out:
191+ ring_buffer__free(rb);
192+ return NULL;
193+}
194+
195+static inline int roundup_len(__u32 len)
196+{
197+ /* clear out top 2 bits (discard and busy, if set) */
198+ len <<= 2;
199+ len >>= 2;
200+ /* add length prefix */
201+ len += BPF_RINGBUF_HDR_SZ;
202+ /* round up to 8 byte alignment */
203+ return (len + 7) / 8 * 8;
204+}
205+
206+static int ringbuf_process_ring(struct ring* r)
207+{
208+ int *len_ptr, len, err, cnt = 0;
209+ unsigned long cons_pos, prod_pos;
210+ bool got_new_data;
211+ void *sample;
212+
213+ cons_pos = smp_load_acquire(r->consumer_pos);
214+ do {
215+ got_new_data = false;
216+ prod_pos = smp_load_acquire(r->producer_pos);
217+ while (cons_pos < prod_pos) {
218+ len_ptr = r->data + (cons_pos & r->mask);
219+ len = smp_load_acquire(len_ptr);
220+
221+ /* sample not committed yet, bail out for now */
222+ if (len & BPF_RINGBUF_BUSY_BIT)
223+ goto done;
224+
225+ got_new_data = true;
226+ cons_pos += roundup_len(len);
227+
228+ if ((len & BPF_RINGBUF_DISCARD_BIT) == 0) {
229+ sample = (void *)len_ptr + BPF_RINGBUF_HDR_SZ;
230+ err = r->sample_cb(r->ctx, sample, len);
231+ if (err) {
232+ /* update consumer pos and bail out */
233+ smp_store_release(r->consumer_pos,
234+ cons_pos);
235+ return err;
236+ }
237+ cnt++;
238+ }
239+
240+ smp_store_release(r->consumer_pos, cons_pos);
241+ }
242+ } while (got_new_data);
243+done:
244+ return cnt;
245+}
246+
247+/* Consume available ring buffer(s) data without event polling.
248+ * Returns number of records consumed across all registered ring buffers, or
249+ * negative number if any of the callbacks return error.
250+ */
251+int ring_buffer__consume(struct ring_buffer *rb)
252+{
253+ int i, err, res = 0;
254+
255+ for (i = 0; i < rb->ring_cnt; i++) {
256+ struct ring *ring = &rb->rings[i];
257+
258+ err = ringbuf_process_ring(ring);
259+ if (err < 0)
260+ return err;
261+ res += err;
262+ }
263+ return res;
264+}
265+
266+/* Poll for available data and consume records, if any are available.
267+ * Returns number of records consumed, or negative number, if any of the
268+ * registered callbacks returned error.
269+ */
270+int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms)
271+{
272+ int i, cnt, err, res = 0;
273+
274+ cnt = epoll_wait(rb->epoll_fd, rb->events, rb->ring_cnt, timeout_ms);
275+ for (i = 0; i < cnt; i++) {
276+ __u32 ring_id = rb->events[i].data.fd;
277+ struct ring *ring = &rb->rings[ring_id];
278+
279+ err = ringbuf_process_ring(ring);
280+ if (err < 0)
281+ return err;
282+ res += cnt;
283+ }
284+ return cnt < 0 ? -errno : res;
285+}
tools/testing/selftests/bpf/Makefile+4−1
@@ -413,12 +413,15 @@ $(OUTPUT)/bench_%.o: benchs/bench_%.c bench.h · 412 unmodified lines
413413 $(CC) $(CFLAGS) -c $(filter %.c,$^) $(LDLIBS) -o $@
414414 $(OUTPUT)/bench_rename.o: $(OUTPUT)/test_overhead.skel.h
415415 $(OUTPUT)/bench_trigger.o: $(OUTPUT)/trigger_bench.skel.h
416+$(OUTPUT)/bench_ringbufs.o: $(OUTPUT)/ringbuf_bench.skel.h \
417+ $(OUTPUT)/perfbuf_bench.skel.h
416418 $(OUTPUT)/bench.o: bench.h testing_helpers.h
417419 $(OUTPUT)/bench: LDLIBS += -lm
418420 $(OUTPUT)/bench: $(OUTPUT)/bench.o $(OUTPUT)/testing_helpers.o \
419421 $(OUTPUT)/bench_count.o \
420422 $(OUTPUT)/bench_rename.o \
421− $(OUTPUT)/bench_trigger.o
423+ $(OUTPUT)/bench_trigger.o \
424+ $(OUTPUT)/bench_ringbufs.o
422425 $(call msg,BINARY,,$@)
423426 $(CC) $(LDFLAGS) -o $@ $(filter %.a %.o,$^) $(LDLIBS)
424427
tools/testing/selftests/bpf/bench.c+16−0
@@ -130,6 +130,13 @@ static const struct argp_option opts[] = { · 129 unmodified lines
130130 {},
131131 };
132132
133+extern struct argp bench_ringbufs_argp;
134+
135+static const struct argp_child bench_parsers[] = {
136+ { &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 },
137+ {},
138+};
139+
133140 static error_t parse_arg(int key, char *arg, struct argp_state *state)
134141 {
135142 static int pos_args;
@@ -208,6 +215,7 @@ static void parse_cmdline_args(int argc, char **argv) · 72 unmodified lines
208215 .options = opts,
209216 .parser = parse_arg,
210217 .doc = argp_program_doc,
218+ .children = bench_parsers,
211219 };
212220 if (argp_parse(&argp, argc, argv, 0, NULL, NULL))
213221 exit(1);
@@ -310,6 +318,10 @@ extern const struct bench bench_trig_rawtp; · 96 unmodified lines
310318 extern const struct bench bench_trig_kprobe;
311319 extern const struct bench bench_trig_fentry;
312320 extern const struct bench bench_trig_fmodret;
321+extern const struct bench bench_rb_libbpf;
322+extern const struct bench bench_rb_custom;
323+extern const struct bench bench_pb_libbpf;
324+extern const struct bench bench_pb_custom;
313325
314326 static const struct bench *benchs[] = {
315327 &bench_count_global,
@@ -327,6 +339,10 @@ static const struct bench *benchs[] = { · 11 unmodified lines
327339 &bench_trig_kprobe,
328340 &bench_trig_fentry,
329341 &bench_trig_fmodret,
342+ &bench_rb_libbpf,
343+ &bench_rb_custom,
344+ &bench_pb_libbpf,
345+ &bench_pb_custom,
330346 };
331347
332348 static void setup_benchmark()
tools/testing/selftests/bpf/benchs/bench_ringbufs.c+566−0
@@ -0,0 +1,566 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (c) 2020 Facebook */
3+#include <asm/barrier.h>
4+#include <linux/perf_event.h>
5+#include <linux/ring_buffer.h>
6+#include <sys/epoll.h>
7+#include <sys/mman.h>
8+#include <argp.h>
9+#include <stdlib.h>
10+#include "bench.h"
11+#include "ringbuf_bench.skel.h"
12+#include "perfbuf_bench.skel.h"
13+
14+static struct {
15+ bool back2back;
16+ int batch_cnt;
17+ bool sampled;
18+ int sample_rate;
19+ int ringbuf_sz; /* per-ringbuf, in bytes */
20+ bool ringbuf_use_output; /* use slower output API */
21+ int perfbuf_sz; /* per-CPU size, in pages */
22+} args = {
23+ .back2back = false,
24+ .batch_cnt = 500,
25+ .sampled = false,
26+ .sample_rate = 500,
27+ .ringbuf_sz = 512 * 1024,
28+ .ringbuf_use_output = false,
29+ .perfbuf_sz = 128,
30+};
31+
32+enum {
33+ ARG_RB_BACK2BACK = 2000,
34+ ARG_RB_USE_OUTPUT = 2001,
35+ ARG_RB_BATCH_CNT = 2002,
36+ ARG_RB_SAMPLED = 2003,
37+ ARG_RB_SAMPLE_RATE = 2004,
38+};
39+
40+static const struct argp_option opts[] = {
41+ { "rb-b2b", ARG_RB_BACK2BACK, NULL, 0, "Back-to-back mode"},
42+ { "rb-use-output", ARG_RB_USE_OUTPUT, NULL, 0, "Use bpf_ringbuf_output() instead of bpf_ringbuf_reserve()"},
43+ { "rb-batch-cnt", ARG_RB_BATCH_CNT, "CNT", 0, "Set BPF-side record batch count"},
44+ { "rb-sampled", ARG_RB_SAMPLED, NULL, 0, "Notification sampling"},
45+ { "rb-sample-rate", ARG_RB_SAMPLE_RATE, "RATE", 0, "Notification sample rate"},
46+ {},
47+};
48+
49+static error_t parse_arg(int key, char *arg, struct argp_state *state)
50+{
51+ switch (key) {
52+ case ARG_RB_BACK2BACK:
53+ args.back2back = true;
54+ break;
55+ case ARG_RB_USE_OUTPUT:
56+ args.ringbuf_use_output = true;
57+ break;
58+ case ARG_RB_BATCH_CNT:
59+ args.batch_cnt = strtol(arg, NULL, 10);
60+ if (args.batch_cnt < 0) {
61+ fprintf(stderr, "Invalid batch count.");
62+ argp_usage(state);
63+ }
64+ break;
65+ case ARG_RB_SAMPLED:
66+ args.sampled = true;
67+ break;
68+ case ARG_RB_SAMPLE_RATE:
69+ args.sample_rate = strtol(arg, NULL, 10);
70+ if (args.sample_rate < 0) {
71+ fprintf(stderr, "Invalid perfbuf sample rate.");
72+ argp_usage(state);
73+ }
74+ break;
75+ default:
76+ return ARGP_ERR_UNKNOWN;
77+ }
78+ return 0;
79+}
80+
81+/* exported into benchmark runner */
82+const struct argp bench_ringbufs_argp = {
83+ .options = opts,
84+ .parser = parse_arg,
85+};
86+
87+/* RINGBUF-LIBBPF benchmark */
88+
89+static struct counter buf_hits;
90+
91+static inline void bufs_trigger_batch()
92+{
93+ (void)syscall(__NR_getpgid);
94+}
95+
96+static void bufs_validate()
97+{
98+ if (env.consumer_cnt != 1) {
99+ fprintf(stderr, "rb-libbpf benchmark doesn't support multi-consumer!\n");
100+ exit(1);
101+ }
102+
103+ if (args.back2back && env.producer_cnt > 1) {
104+ fprintf(stderr, "back-to-back mode makes sense only for single-producer case!\n");
105+ exit(1);
106+ }
107+}
108+
109+static void *bufs_sample_producer(void *input)
110+{
111+ if (args.back2back) {
112+ /* initial batch to get everything started */
113+ bufs_trigger_batch();
114+ return NULL;
115+ }
116+
117+ while (true)
118+ bufs_trigger_batch();
119+ return NULL;
120+}
121+
122+static struct ringbuf_libbpf_ctx {
123+ struct ringbuf_bench *skel;
124+ struct ring_buffer *ringbuf;
125+} ringbuf_libbpf_ctx;
126+
127+static void ringbuf_libbpf_measure(struct bench_res *res)
128+{
129+ struct ringbuf_libbpf_ctx *ctx = &ringbuf_libbpf_ctx;
130+
131+ res->hits = atomic_swap(&buf_hits.value, 0);
132+ res->drops = atomic_swap(&ctx->skel->bss->dropped, 0);
133+}
134+
135+static struct ringbuf_bench *ringbuf_setup_skeleton()
136+{
137+ struct ringbuf_bench *skel;
138+
139+ setup_libbpf();
140+
141+ skel = ringbuf_bench__open();
142+ if (!skel) {
143+ fprintf(stderr, "failed to open skeleton\n");
144+ exit(1);
145+ }
146+
147+ skel->rodata->batch_cnt = args.batch_cnt;
148+ skel->rodata->use_output = args.ringbuf_use_output ? 1 : 0;
149+
150+ if (args.sampled)
151+ /* record data + header take 16 bytes */
152+ skel->rodata->wakeup_data_size = args.sample_rate * 16;
153+
154+ bpf_map__resize(skel->maps.ringbuf, args.ringbuf_sz);
155+
156+ if (ringbuf_bench__load(skel)) {
157+ fprintf(stderr, "failed to load skeleton\n");
158+ exit(1);
159+ }
160+
161+ return skel;
162+}
163+
164+static int buf_process_sample(void *ctx, void *data, size_t len)
165+{
166+ atomic_inc(&buf_hits.value);
167+ return 0;
168+}
169+
170+static void ringbuf_libbpf_setup()
171+{
172+ struct ringbuf_libbpf_ctx *ctx = &ringbuf_libbpf_ctx;
173+ struct bpf_link *link;
174+
175+ ctx->skel = ringbuf_setup_skeleton();
176+ ctx->ringbuf = ring_buffer__new(bpf_map__fd(ctx->skel->maps.ringbuf),
177+ buf_process_sample, NULL, NULL);
178+ if (!ctx->ringbuf) {
179+ fprintf(stderr, "failed to create ringbuf\n");
180+ exit(1);
181+ }
182+
183+ link = bpf_program__attach(ctx->skel->progs.bench_ringbuf);
184+ if (IS_ERR(link)) {
185+ fprintf(stderr, "failed to attach program!\n");
186+ exit(1);
187+ }
188+}
189+
190+static void *ringbuf_libbpf_consumer(void *input)
191+{
192+ struct ringbuf_libbpf_ctx *ctx = &ringbuf_libbpf_ctx;
193+
194+ while (ring_buffer__poll(ctx->ringbuf, -1) >= 0) {
195+ if (args.back2back)
196+ bufs_trigger_batch();
197+ }
198+ fprintf(stderr, "ringbuf polling failed!\n");
199+ return NULL;
200+}
201+
202+/* RINGBUF-CUSTOM benchmark */
203+struct ringbuf_custom {
204+ __u64 *consumer_pos;
205+ __u64 *producer_pos;
206+ __u64 mask;
207+ void *data;
208+ int map_fd;
209+};
210+
211+static struct ringbuf_custom_ctx {
212+ struct ringbuf_bench *skel;
213+ struct ringbuf_custom ringbuf;
214+ int epoll_fd;
215+ struct epoll_event event;
216+} ringbuf_custom_ctx;
217+
218+static void ringbuf_custom_measure(struct bench_res *res)
219+{
220+ struct ringbuf_custom_ctx *ctx = &ringbuf_custom_ctx;
221+
222+ res->hits = atomic_swap(&buf_hits.value, 0);
223+ res->drops = atomic_swap(&ctx->skel->bss->dropped, 0);
224+}
225+
226+static void ringbuf_custom_setup()
227+{
228+ struct ringbuf_custom_ctx *ctx = &ringbuf_custom_ctx;
229+ const size_t page_size = getpagesize();
230+ struct bpf_link *link;
231+ struct ringbuf_custom *r;
232+ void *tmp;
233+ int err;
234+
235+ ctx->skel = ringbuf_setup_skeleton();
236+
237+ ctx->epoll_fd = epoll_create1(EPOLL_CLOEXEC);
238+ if (ctx->epoll_fd < 0) {
239+ fprintf(stderr, "failed to create epoll fd: %d\n", -errno);
240+ exit(1);
241+ }
242+
243+ r = &ctx->ringbuf;
244+ r->map_fd = bpf_map__fd(ctx->skel->maps.ringbuf);
245+ r->mask = args.ringbuf_sz - 1;
246+
247+ /* Map writable consumer page */
248+ tmp = mmap(NULL, page_size, PROT_READ | PROT_WRITE, MAP_SHARED,
249+ r->map_fd, 0);
250+ if (tmp == MAP_FAILED) {
251+ fprintf(stderr, "failed to mmap consumer page: %d\n", -errno);
252+ exit(1);
253+ }
254+ r->consumer_pos = tmp;
255+
256+ /* Map read-only producer page and data pages. */
257+ tmp = mmap(NULL, page_size + 2 * args.ringbuf_sz, PROT_READ, MAP_SHARED,
258+ r->map_fd, page_size);
259+ if (tmp == MAP_FAILED) {
260+ fprintf(stderr, "failed to mmap data pages: %d\n", -errno);
261+ exit(1);
262+ }
263+ r->producer_pos = tmp;
264+ r->data = tmp + page_size;
265+
266+ ctx->event.events = EPOLLIN;
267+ err = epoll_ctl(ctx->epoll_fd, EPOLL_CTL_ADD, r->map_fd, &ctx->event);
268+ if (err < 0) {
269+ fprintf(stderr, "failed to epoll add ringbuf: %d\n", -errno);
270+ exit(1);
271+ }
272+
273+ link = bpf_program__attach(ctx->skel->progs.bench_ringbuf);
274+ if (IS_ERR(link)) {
275+ fprintf(stderr, "failed to attach program\n");
276+ exit(1);
277+ }
278+}
279+
280+#define RINGBUF_BUSY_BIT (1 << 31)
281+#define RINGBUF_DISCARD_BIT (1 << 30)
282+#define RINGBUF_META_LEN 8
283+
284+static inline int roundup_len(__u32 len)
285+{
286+ /* clear out top 2 bits */
287+ len <<= 2;
288+ len >>= 2;
289+ /* add length prefix */
290+ len += RINGBUF_META_LEN;
291+ /* round up to 8 byte alignment */
292+ return (len + 7) / 8 * 8;
293+}
294+
295+static void ringbuf_custom_process_ring(struct ringbuf_custom *r)
296+{
297+ unsigned long cons_pos, prod_pos;
298+ int *len_ptr, len;
299+ bool got_new_data;
300+
301+ cons_pos = smp_load_acquire(r->consumer_pos);
302+ while (true) {
303+ got_new_data = false;
304+ prod_pos = smp_load_acquire(r->producer_pos);
305+ while (cons_pos < prod_pos) {
306+ len_ptr = r->data + (cons_pos & r->mask);
307+ len = smp_load_acquire(len_ptr);
308+
309+ /* sample not committed yet, bail out for now */
310+ if (len & RINGBUF_BUSY_BIT)
311+ return;
312+
313+ got_new_data = true;
314+ cons_pos += roundup_len(len);
315+
316+ atomic_inc(&buf_hits.value);
317+ }
318+ if (got_new_data)
319+ smp_store_release(r->consumer_pos, cons_pos);
320+ else
321+ break;
322+ };
323+}
324+
325+static void *ringbuf_custom_consumer(void *input)
326+{
327+ struct ringbuf_custom_ctx *ctx = &ringbuf_custom_ctx;
328+ int cnt;
329+
330+ do {
331+ if (args.back2back)
332+ bufs_trigger_batch();
333+ cnt = epoll_wait(ctx->epoll_fd, &ctx->event, 1, -1);
334+ if (cnt > 0)
335+ ringbuf_custom_process_ring(&ctx->ringbuf);
336+ } while (cnt >= 0);
337+ fprintf(stderr, "ringbuf polling failed!\n");
338+ return 0;
339+}
340+
341+/* PERFBUF-LIBBPF benchmark */
342+static struct perfbuf_libbpf_ctx {
343+ struct perfbuf_bench *skel;
344+ struct perf_buffer *perfbuf;
345+} perfbuf_libbpf_ctx;
346+
347+static void perfbuf_measure(struct bench_res *res)
348+{
349+ struct perfbuf_libbpf_ctx *ctx = &perfbuf_libbpf_ctx;
350+
351+ res->hits = atomic_swap(&buf_hits.value, 0);
352+ res->drops = atomic_swap(&ctx->skel->bss->dropped, 0);
353+}
354+
355+static struct perfbuf_bench *perfbuf_setup_skeleton()
356+{
357+ struct perfbuf_bench *skel;
358+
359+ setup_libbpf();
360+
361+ skel = perfbuf_bench__open();
362+ if (!skel) {
363+ fprintf(stderr, "failed to open skeleton\n");
364+ exit(1);
365+ }
366+
367+ skel->rodata->batch_cnt = args.batch_cnt;
368+
369+ if (perfbuf_bench__load(skel)) {
370+ fprintf(stderr, "failed to load skeleton\n");
371+ exit(1);
372+ }
373+
374+ return skel;
375+}
376+
377+static enum bpf_perf_event_ret
378+perfbuf_process_sample_raw(void *input_ctx, int cpu,
379+ struct perf_event_header *e)
380+{
381+ switch (e->type) {
382+ case PERF_RECORD_SAMPLE:
383+ atomic_inc(&buf_hits.value);
384+ break;
385+ case PERF_RECORD_LOST:
386+ break;
387+ default:
388+ return LIBBPF_PERF_EVENT_ERROR;
389+ }
390+ return LIBBPF_PERF_EVENT_CONT;
391+}
392+
393+static void perfbuf_libbpf_setup()
394+{
395+ struct perfbuf_libbpf_ctx *ctx = &perfbuf_libbpf_ctx;
396+ struct perf_event_attr attr;
397+ struct perf_buffer_raw_opts pb_opts = {
398+ .event_cb = perfbuf_process_sample_raw,
399+ .ctx = (void *)(long)0,
400+ .attr = &attr,
401+ };
402+ struct bpf_link *link;
403+
404+ ctx->skel = perfbuf_setup_skeleton();
405+
406+ memset(&attr, 0, sizeof(attr));
407+ attr.config = PERF_COUNT_SW_BPF_OUTPUT,
408+ attr.type = PERF_TYPE_SOFTWARE;
409+ attr.sample_type = PERF_SAMPLE_RAW;
410+ /* notify only every Nth sample */
411+ if (args.sampled) {
412+ attr.sample_period = args.sample_rate;
413+ attr.wakeup_events = args.sample_rate;
414+ } else {
415+ attr.sample_period = 1;
416+ attr.wakeup_events = 1;
417+ }
418+
419+ if (args.sample_rate > args.batch_cnt) {
420+ fprintf(stderr, "sample rate %d is too high for given batch count %d\n",
421+ args.sample_rate, args.batch_cnt);
422+ exit(1);
423+ }
424+
425+ ctx->perfbuf = perf_buffer__new_raw(bpf_map__fd(ctx->skel->maps.perfbuf),
426+ args.perfbuf_sz, &pb_opts);
427+ if (!ctx->perfbuf) {
428+ fprintf(stderr, "failed to create perfbuf\n");
429+ exit(1);
430+ }
431+
432+ link = bpf_program__attach(ctx->skel->progs.bench_perfbuf);
433+ if (IS_ERR(link)) {
434+ fprintf(stderr, "failed to attach program\n");
435+ exit(1);
436+ }
437+}
438+
439+static void *perfbuf_libbpf_consumer(void *input)
440+{
441+ struct perfbuf_libbpf_ctx *ctx = &perfbuf_libbpf_ctx;
442+
443+ while (perf_buffer__poll(ctx->perfbuf, -1) >= 0) {
444+ if (args.back2back)
445+ bufs_trigger_batch();
446+ }
447+ fprintf(stderr, "perfbuf polling failed!\n");
448+ return NULL;
449+}
450+
451+/* PERFBUF-CUSTOM benchmark */
452+
453+/* copies of internal libbpf definitions */
454+struct perf_cpu_buf {
455+ struct perf_buffer *pb;
456+ void *base; /* mmap()'ed memory */
457+ void *buf; /* for reconstructing segmented data */
458+ size_t buf_size;
459+ int fd;
460+ int cpu;
461+ int map_key;
462+};
463+
464+struct perf_buffer {
465+ perf_buffer_event_fn event_cb;
466+ perf_buffer_sample_fn sample_cb;
467+ perf_buffer_lost_fn lost_cb;
468+ void *ctx; /* passed into callbacks */
469+
470+ size_t page_size;
471+ size_t mmap_size;
472+ struct perf_cpu_buf **cpu_bufs;
473+ struct epoll_event *events;
474+ int cpu_cnt; /* number of allocated CPU buffers */
475+ int epoll_fd; /* perf event FD */
476+ int map_fd; /* BPF_MAP_TYPE_PERF_EVENT_ARRAY BPF map FD */
477+};
478+
479+static void *perfbuf_custom_consumer(void *input)
480+{
481+ struct perfbuf_libbpf_ctx *ctx = &perfbuf_libbpf_ctx;
482+ struct perf_buffer *pb = ctx->perfbuf;
483+ struct perf_cpu_buf *cpu_buf;
484+ struct perf_event_mmap_page *header;
485+ size_t mmap_mask = pb->mmap_size - 1;
486+ struct perf_event_header *ehdr;
487+ __u64 data_head, data_tail;
488+ size_t ehdr_size;
489+ void *base;
490+ int i, cnt;
491+
492+ while (true) {
493+ if (args.back2back)
494+ bufs_trigger_batch();
495+ cnt = epoll_wait(pb->epoll_fd, pb->events, pb->cpu_cnt, -1);
496+ if (cnt <= 0) {
497+ fprintf(stderr, "perf epoll failed: %d\n", -errno);
498+ exit(1);
499+ }
500+
501+ for (i = 0; i < cnt; ++i) {
502+ cpu_buf = pb->events[i].data.ptr;
503+ header = cpu_buf->base;
504+ base = ((void *)header) + pb->page_size;
505+
506+ data_head = ring_buffer_read_head(header);
507+ data_tail = header->data_tail;
508+ while (data_head != data_tail) {
509+ ehdr = base + (data_tail & mmap_mask);
510+ ehdr_size = ehdr->size;
511+
512+ if (ehdr->type == PERF_RECORD_SAMPLE)
513+ atomic_inc(&buf_hits.value);
514+
515+ data_tail += ehdr_size;
516+ }
517+ ring_buffer_write_tail(header, data_tail);
518+ }
519+ }
520+ return NULL;
521+}
522+
523+const struct bench bench_rb_libbpf = {
524+ .name = "rb-libbpf",
525+ .validate = bufs_validate,
526+ .setup = ringbuf_libbpf_setup,
527+ .producer_thread = bufs_sample_producer,
528+ .consumer_thread = ringbuf_libbpf_consumer,
529+ .measure = ringbuf_libbpf_measure,
530+ .report_progress = hits_drops_report_progress,
531+ .report_final = hits_drops_report_final,
532+};
533+
534+const struct bench bench_rb_custom = {
535+ .name = "rb-custom",
536+ .validate = bufs_validate,
537+ .setup = ringbuf_custom_setup,
538+ .producer_thread = bufs_sample_producer,
539+ .consumer_thread = ringbuf_custom_consumer,
540+ .measure = ringbuf_custom_measure,
541+ .report_progress = hits_drops_report_progress,
542+ .report_final = hits_drops_report_final,
543+};
544+
545+const struct bench bench_pb_libbpf = {
546+ .name = "pb-libbpf",
547+ .validate = bufs_validate,
548+ .setup = perfbuf_libbpf_setup,
549+ .producer_thread = bufs_sample_producer,
550+ .consumer_thread = perfbuf_libbpf_consumer,
551+ .measure = perfbuf_measure,
552+ .report_progress = hits_drops_report_progress,
553+ .report_final = hits_drops_report_final,
554+};
555+
556+const struct bench bench_pb_custom = {
557+ .name = "pb-custom",
558+ .validate = bufs_validate,
559+ .setup = perfbuf_libbpf_setup,
560+ .producer_thread = bufs_sample_producer,
561+ .consumer_thread = perfbuf_custom_consumer,
562+ .measure = perfbuf_measure,
563+ .report_progress = hits_drops_report_progress,
564+ .report_final = hits_drops_report_final,
565+};
566+
tools/testing/selftests/bpf/benchs/run_bench_ringbufs.sh+75−0
@@ -0,0 +1,75 @@
1+#!/bin/bash
2+
3+set -eufo pipefail
4+
5+RUN_BENCH="sudo ./bench -w3 -d10 -a"
6+
7+function hits()
8+{
9+ echo "$*" | sed -E "s/.*hits\s+([0-9]+\.[0-9]+ ± [0-9]+\.[0-9]+M\/s).*/\1/"
10+}
11+
12+function drops()
13+{
14+ echo "$*" | sed -E "s/.*drops\s+([0-9]+\.[0-9]+ ± [0-9]+\.[0-9]+M\/s).*/\1/"
15+}
16+
17+function header()
18+{
19+ local len=${#1}
20+
21+ printf "\n%s\n" "$1"
22+ for i in $(seq 1 $len); do printf '='; done
23+ printf '\n'
24+}
25+
26+function summarize()
27+{
28+ bench="$1"
29+ summary=$(echo $2 | tail -n1)
30+ printf "%-20s %s (drops %s)\n" "$bench" "$(hits $summary)" "$(drops $summary)"
31+}
32+
33+header "Single-producer, parallel producer"
34+for b in rb-libbpf rb-custom pb-libbpf pb-custom; do
35+ summarize $b "$($RUN_BENCH $b)"
36+done
37+
38+header "Single-producer, parallel producer, sampled notification"
39+for b in rb-libbpf rb-custom pb-libbpf pb-custom; do
40+ summarize $b "$($RUN_BENCH --rb-sampled $b)"
41+done
42+
43+header "Single-producer, back-to-back mode"
44+for b in rb-libbpf rb-custom pb-libbpf pb-custom; do
45+ summarize $b "$($RUN_BENCH --rb-b2b $b)"
46+ summarize $b-sampled "$($RUN_BENCH --rb-sampled --rb-b2b $b)"
47+done
48+
49+header "Ringbuf back-to-back, effect of sample rate"
50+for b in 1 5 10 25 50 100 250 500 1000 2000 3000; do
51+ summarize "rb-sampled-$b" "$($RUN_BENCH --rb-b2b --rb-batch-cnt $b --rb-sampled --rb-sample-rate $b rb-custom)"
52+done
53+header "Perfbuf back-to-back, effect of sample rate"
54+for b in 1 5 10 25 50 100 250 500 1000 2000 3000; do
55+ summarize "pb-sampled-$b" "$($RUN_BENCH --rb-b2b --rb-batch-cnt $b --rb-sampled --rb-sample-rate $b pb-custom)"
56+done
57+
58+header "Ringbuf back-to-back, reserve+commit vs output"
59+summarize "reserve" "$($RUN_BENCH --rb-b2b rb-custom)"
60+summarize "output" "$($RUN_BENCH --rb-b2b --rb-use-output rb-custom)"
61+
62+header "Ringbuf sampled, reserve+commit vs output"
63+summarize "reserve-sampled" "$($RUN_BENCH --rb-sampled rb-custom)"
64+summarize "output-sampled" "$($RUN_BENCH --rb-sampled --rb-use-output rb-custom)"
65+
66+header "Single-producer, consumer/producer competing on the same CPU, low batch count"
67+for b in rb-libbpf rb-custom pb-libbpf pb-custom; do
68+ summarize $b "$($RUN_BENCH --rb-batch-cnt 1 --rb-sample-rate 1 --prod-affinity 0 --cons-affinity 0 $b)"
69+done
70+
71+header "Ringbuf, multi-producer contention"
72+for b in 1 2 3 4 8 12 16 20 24 28 32 36 40 44 48 52; do
73+ summarize "rb-libbpf nr_prod $b" "$($RUN_BENCH -p$b --rb-batch-cnt 50 rb-libbpf)"
74+done
75+
tools/testing/selftests/bpf/prog_tests/ringbuf.c+211−0
@@ -0,0 +1,211 @@
1+// SPDX-License-Identifier: GPL-2.0
2+#define _GNU_SOURCE
3+#include <linux/compiler.h>
4+#include <asm/barrier.h>
5+#include <test_progs.h>
6+#include <sys/mman.h>
7+#include <sys/epoll.h>
8+#include <time.h>
9+#include <sched.h>
10+#include <signal.h>
11+#include <pthread.h>
12+#include <sys/sysinfo.h>
13+#include <linux/perf_event.h>
14+#include <linux/ring_buffer.h>
15+#include "test_ringbuf.skel.h"
16+
17+#define EDONE 7777
18+
19+static int duration = 0;
20+
21+struct sample {
22+ int pid;
23+ int seq;
24+ long value;
25+ char comm[16];
26+};
27+
28+static int sample_cnt;
29+
30+static int process_sample(void *ctx, void *data, size_t len)
31+{
32+ struct sample *s = data;
33+
34+ sample_cnt++;
35+
36+ switch (s->seq) {
37+ case 0:
38+ CHECK(s->value != 333, "sample1_value", "exp %ld, got %ld\n",
39+ 333L, s->value);
40+ return 0;
41+ case 1:
42+ CHECK(s->value != 777, "sample2_value", "exp %ld, got %ld\n",
43+ 777L, s->value);
44+ return -EDONE;
45+ default:
46+ /* we don't care about the rest */
47+ return 0;
48+ }
49+}
50+
51+static struct test_ringbuf *skel;
52+static struct ring_buffer *ringbuf;
53+
54+static void trigger_samples()
55+{
56+ skel->bss->dropped = 0;
57+ skel->bss->total = 0;
58+ skel->bss->discarded = 0;
59+
60+ /* trigger exactly two samples */
61+ skel->bss->value = 333;
62+ syscall(__NR_getpgid);
63+ skel->bss->value = 777;
64+ syscall(__NR_getpgid);
65+}
66+
67+static void *poll_thread(void *input)
68+{
69+ long timeout = (long)input;
70+
71+ return (void *)(long)ring_buffer__poll(ringbuf, timeout);
72+}
73+
74+void test_ringbuf(void)
75+{
76+ const size_t rec_sz = BPF_RINGBUF_HDR_SZ + sizeof(struct sample);
77+ pthread_t thread;
78+ long bg_ret = -1;
79+ int err;
80+
81+ skel = test_ringbuf__open_and_load();
82+ if (CHECK(!skel, "skel_open_load", "skeleton open&load failed\n"))
83+ return;
84+
85+ /* only trigger BPF program for current process */
86+ skel->bss->pid = getpid();
87+
88+ ringbuf = ring_buffer__new(bpf_map__fd(skel->maps.ringbuf),
89+ process_sample, NULL, NULL);
90+ if (CHECK(!ringbuf, "ringbuf_create", "failed to create ringbuf\n"))
91+ goto cleanup;
92+
93+ err = test_ringbuf__attach(skel);
94+ if (CHECK(err, "skel_attach", "skeleton attachment failed: %d\n", err))
95+ goto cleanup;
96+
97+ trigger_samples();
98+
99+ /* 2 submitted + 1 discarded records */
100+ CHECK(skel->bss->avail_data != 3 * rec_sz,
101+ "err_avail_size", "exp %ld, got %ld\n",
102+ 3L * rec_sz, skel->bss->avail_data);
103+ CHECK(skel->bss->ring_size != 4096,
104+ "err_ring_size", "exp %ld, got %ld\n",
105+ 4096L, skel->bss->ring_size);
106+ CHECK(skel->bss->cons_pos != 0,
107+ "err_cons_pos", "exp %ld, got %ld\n",
108+ 0L, skel->bss->cons_pos);
109+ CHECK(skel->bss->prod_pos != 3 * rec_sz,
110+ "err_prod_pos", "exp %ld, got %ld\n",
111+ 3L * rec_sz, skel->bss->prod_pos);
112+
113+ /* poll for samples */
114+ err = ring_buffer__poll(ringbuf, -1);
115+
116+ /* -EDONE is used as an indicator that we are done */
117+ if (CHECK(err != -EDONE, "err_done", "done err: %d\n", err))
118+ goto cleanup;
119+
120+ /* we expect extra polling to return nothing */
121+ err = ring_buffer__poll(ringbuf, 0);
122+ if (CHECK(err != 0, "extra_samples", "poll result: %d\n", err))
123+ goto cleanup;
124+
125+ CHECK(skel->bss->dropped != 0, "err_dropped", "exp %ld, got %ld\n",
126+ 0L, skel->bss->dropped);
127+ CHECK(skel->bss->total != 2, "err_total", "exp %ld, got %ld\n",
128+ 2L, skel->bss->total);
129+ CHECK(skel->bss->discarded != 1, "err_discarded", "exp %ld, got %ld\n",
130+ 1L, skel->bss->discarded);
131+
132+ /* now validate consumer position is updated and returned */
133+ trigger_samples();
134+ CHECK(skel->bss->cons_pos != 3 * rec_sz,
135+ "err_cons_pos", "exp %ld, got %ld\n",
136+ 3L * rec_sz, skel->bss->cons_pos);
137+ err = ring_buffer__poll(ringbuf, -1);
138+ CHECK(err <= 0, "poll_err", "err %d\n", err);
139+
140+ /* start poll in background w/ long timeout */
141+ err = pthread_create(&thread, NULL, poll_thread, (void *)(long)10000);
142+ if (CHECK(err, "bg_poll", "pthread_create failed: %d\n", err))
143+ goto cleanup;
144+
145+ /* turn off notifications now */
146+ skel->bss->flags = BPF_RB_NO_WAKEUP;
147+
148+ /* give background thread a bit of a time */
149+ usleep(50000);
150+ trigger_samples();
151+ /* sleeping arbitrarily is bad, but no better way to know that
152+ * epoll_wait() **DID NOT** unblock in background thread
153+ */
154+ usleep(50000);
155+ /* background poll should still be blocked */
156+ err = pthread_tryjoin_np(thread, (void **)&bg_ret);
157+ if (CHECK(err != EBUSY, "try_join", "err %d\n", err))
158+ goto cleanup;
159+
160+ /* BPF side did everything right */
161+ CHECK(skel->bss->dropped != 0, "err_dropped", "exp %ld, got %ld\n",
162+ 0L, skel->bss->dropped);
163+ CHECK(skel->bss->total != 2, "err_total", "exp %ld, got %ld\n",
164+ 2L, skel->bss->total);
165+ CHECK(skel->bss->discarded != 1, "err_discarded", "exp %ld, got %ld\n",
166+ 1L, skel->bss->discarded);
167+
168+ /* clear flags to return to "adaptive" notification mode */
169+ skel->bss->flags = 0;
170+
171+ /* produce new samples, no notification should be triggered, because
172+ * consumer is now behind
173+ */
174+ trigger_samples();
175+
176+ /* background poll should still be blocked */
177+ err = pthread_tryjoin_np(thread, (void **)&bg_ret);
178+ if (CHECK(err != EBUSY, "try_join", "err %d\n", err))
179+ goto cleanup;
180+
181+ /* now force notifications */
182+ skel->bss->flags = BPF_RB_FORCE_WAKEUP;
183+ sample_cnt = 0;
184+ trigger_samples();
185+
186+ /* now we should get a pending notification */
187+ usleep(50000);
188+ err = pthread_tryjoin_np(thread, (void **)&bg_ret);
189+ if (CHECK(err, "join_bg", "err %d\n", err))
190+ goto cleanup;
191+
192+ if (CHECK(bg_ret != 1, "bg_ret", "epoll_wait result: %ld", bg_ret))
193+ goto cleanup;
194+
195+ /* 3 rounds, 2 samples each */
196+ CHECK(sample_cnt != 6, "wrong_sample_cnt",
197+ "expected to see %d samples, got %d\n", 6, sample_cnt);
198+
199+ /* BPF side did everything right */
200+ CHECK(skel->bss->dropped != 0, "err_dropped", "exp %ld, got %ld\n",
201+ 0L, skel->bss->dropped);
202+ CHECK(skel->bss->total != 2, "err_total", "exp %ld, got %ld\n",
203+ 2L, skel->bss->total);
204+ CHECK(skel->bss->discarded != 1, "err_discarded", "exp %ld, got %ld\n",
205+ 1L, skel->bss->discarded);
206+
207+ test_ringbuf__detach(skel);
208+cleanup:
209+ ring_buffer__free(ringbuf);
210+ test_ringbuf__destroy(skel);
211+}
tools/testing/selftests/bpf/prog_tests/ringbuf_multi.c+102−0
@@ -0,0 +1,102 @@
1+// SPDX-License-Identifier: GPL-2.0
2+#define _GNU_SOURCE
3+#include <test_progs.h>
4+#include <sys/epoll.h>
5+#include "test_ringbuf_multi.skel.h"
6+
7+static int duration = 0;
8+
9+struct sample {
10+ int pid;
11+ int seq;
12+ long value;
13+ char comm[16];
14+};
15+
16+static int process_sample(void *ctx, void *data, size_t len)
17+{
18+ int ring = (unsigned long)ctx;
19+ struct sample *s = data;
20+
21+ switch (s->seq) {
22+ case 0:
23+ CHECK(ring != 1, "sample1_ring", "exp %d, got %d\n", 1, ring);
24+ CHECK(s->value != 333, "sample1_value", "exp %ld, got %ld\n",
25+ 333L, s->value);
26+ break;
27+ case 1:
28+ CHECK(ring != 2, "sample2_ring", "exp %d, got %d\n", 2, ring);
29+ CHECK(s->value != 777, "sample2_value", "exp %ld, got %ld\n",
30+ 777L, s->value);
31+ break;
32+ default:
33+ CHECK(true, "extra_sample", "unexpected sample seq %d, val %ld\n",
34+ s->seq, s->value);
35+ return -1;
36+ }
37+
38+ return 0;
39+}
40+
41+void test_ringbuf_multi(void)
42+{
43+ struct test_ringbuf_multi *skel;
44+ struct ring_buffer *ringbuf;
45+ int err;
46+
47+ skel = test_ringbuf_multi__open_and_load();
48+ if (CHECK(!skel, "skel_open_load", "skeleton open&load failed\n"))
49+ return;
50+
51+ /* only trigger BPF program for current process */
52+ skel->bss->pid = getpid();
53+
54+ ringbuf = ring_buffer__new(bpf_map__fd(skel->maps.ringbuf1),
55+ process_sample, (void *)(long)1, NULL);
56+ if (CHECK(!ringbuf, "ringbuf_create", "failed to create ringbuf\n"))
57+ goto cleanup;
58+
59+ err = ring_buffer__add(ringbuf, bpf_map__fd(skel->maps.ringbuf2),
60+ process_sample, (void *)(long)2);
61+ if (CHECK(err, "ringbuf_add", "failed to add another ring\n"))
62+ goto cleanup;
63+
64+ err = test_ringbuf_multi__attach(skel);
65+ if (CHECK(err, "skel_attach", "skeleton attachment failed: %d\n", err))
66+ goto cleanup;
67+
68+ /* trigger few samples, some will be skipped */
69+ skel->bss->target_ring = 0;
70+ skel->bss->value = 333;
71+ syscall(__NR_getpgid);
72+
73+ /* skipped, no ringbuf in slot 1 */
74+ skel->bss->target_ring = 1;
75+ skel->bss->value = 555;
76+ syscall(__NR_getpgid);
77+
78+ skel->bss->target_ring = 2;
79+ skel->bss->value = 777;
80+ syscall(__NR_getpgid);
81+
82+ /* poll for samples, should get 2 ringbufs back */
83+ err = ring_buffer__poll(ringbuf, -1);
84+ if (CHECK(err != 4, "poll_res", "expected 4 records, got %d\n", err))
85+ goto cleanup;
86+
87+ /* expect extra polling to return nothing */
88+ err = ring_buffer__poll(ringbuf, 0);
89+ if (CHECK(err < 0, "extra_samples", "poll result: %d\n", err))
90+ goto cleanup;
91+
92+ CHECK(skel->bss->dropped != 0, "err_dropped", "exp %ld, got %ld\n",
93+ 0L, skel->bss->dropped);
94+ CHECK(skel->bss->skipped != 1, "err_skipped", "exp %ld, got %ld\n",
95+ 1L, skel->bss->skipped);
96+ CHECK(skel->bss->total != 2, "err_total", "exp %ld, got %ld\n",
97+ 2L, skel->bss->total);
98+
99+cleanup:
100+ ring_buffer__free(ringbuf);
101+ test_ringbuf_multi__destroy(skel);
102+}
tools/testing/selftests/bpf/progs/perfbuf_bench.c+33−0
@@ -0,0 +1,33 @@
1+// SPDX-License-Identifier: GPL-2.0
2+// Copyright (c) 2020 Facebook
3+
4+#include <linux/bpf.h>
5+#include <stdint.h>
6+#include <bpf/bpf_helpers.h>
7+
8+char _license[] SEC("license") = "GPL";
9+
10+struct {
11+ __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
12+ __uint(value_size, sizeof(int));
13+ __uint(key_size, sizeof(int));
14+} perfbuf SEC(".maps");
15+
16+const volatile int batch_cnt = 0;
17+
18+long sample_val = 42;
19+long dropped __attribute__((aligned(128))) = 0;
20+
21+SEC("fentry/__x64_sys_getpgid")
22+int bench_perfbuf(void *ctx)
23+{
24+ __u64 *sample;
25+ int i;
26+
27+ for (i = 0; i < batch_cnt; i++) {
28+ if (bpf_perf_event_output(ctx, &perfbuf, BPF_F_CURRENT_CPU,
29+ &sample_val, sizeof(sample_val)))
30+ __sync_add_and_fetch(&dropped, 1);
31+ }
32+ return 0;
33+}
tools/testing/selftests/bpf/progs/ringbuf_bench.c+60−0
@@ -0,0 +1,60 @@
1+// SPDX-License-Identifier: GPL-2.0
2+// Copyright (c) 2020 Facebook
3+
4+#include <linux/bpf.h>
5+#include <stdint.h>
6+#include <bpf/bpf_helpers.h>
7+
8+char _license[] SEC("license") = "GPL";
9+
10+struct {
11+ __uint(type, BPF_MAP_TYPE_RINGBUF);
12+} ringbuf SEC(".maps");
13+
14+const volatile int batch_cnt = 0;
15+const volatile long use_output = 0;
16+
17+long sample_val = 42;
18+long dropped __attribute__((aligned(128))) = 0;
19+
20+const volatile long wakeup_data_size = 0;
21+
22+static __always_inline long get_flags()
23+{
24+ long sz;
25+
26+ if (!wakeup_data_size)
27+ return 0;
28+
29+ sz = bpf_ringbuf_query(&ringbuf, BPF_RB_AVAIL_DATA);
30+ return sz >= wakeup_data_size ? BPF_RB_FORCE_WAKEUP : BPF_RB_NO_WAKEUP;
31+}
32+
33+SEC("fentry/__x64_sys_getpgid")
34+int bench_ringbuf(void *ctx)
35+{
36+ long *sample, flags;
37+ int i;
38+
39+ if (!use_output) {
40+ for (i = 0; i < batch_cnt; i++) {
41+ sample = bpf_ringbuf_reserve(&ringbuf,
42+ sizeof(sample_val), 0);
43+ if (!sample) {
44+ __sync_add_and_fetch(&dropped, 1);
45+ } else {
46+ *sample = sample_val;
47+ flags = get_flags();
48+ bpf_ringbuf_submit(sample, flags);
49+ }
50+ }
51+ } else {
52+ for (i = 0; i < batch_cnt; i++) {
53+ flags = get_flags();
54+ if (bpf_ringbuf_output(&ringbuf, &sample_val,
55+ sizeof(sample_val), flags))
56+ __sync_add_and_fetch(&dropped, 1);
57+ }
58+ }
59+ return 0;
60+}
tools/testing/selftests/bpf/progs/test_ringbuf.c+78−0
@@ -0,0 +1,78 @@
1+// SPDX-License-Identifier: GPL-2.0
2+// Copyright (c) 2020 Facebook
3+
4+#include <linux/bpf.h>
5+#include <bpf/bpf_helpers.h>
6+
7+char _license[] SEC("license") = "GPL";
8+
9+struct sample {
10+ int pid;
11+ int seq;
12+ long value;
13+ char comm[16];
14+};
15+
16+struct {
17+ __uint(type, BPF_MAP_TYPE_RINGBUF);
18+ __uint(max_entries, 1 << 12);
19+} ringbuf SEC(".maps");
20+
21+/* inputs */
22+int pid = 0;
23+long value = 0;
24+long flags = 0;
25+
26+/* outputs */
27+long total = 0;
28+long discarded = 0;
29+long dropped = 0;
30+
31+long avail_data = 0;
32+long ring_size = 0;
33+long cons_pos = 0;
34+long prod_pos = 0;
35+
36+/* inner state */
37+long seq = 0;
38+
39+SEC("tp/syscalls/sys_enter_getpgid")
40+int test_ringbuf(void *ctx)
41+{
42+ int cur_pid = bpf_get_current_pid_tgid() >> 32;
43+ struct sample *sample;
44+ int zero = 0;
45+
46+ if (cur_pid != pid)
47+ return 0;
48+
49+ sample = bpf_ringbuf_reserve(&ringbuf, sizeof(*sample), 0);
50+ if (!sample) {
51+ __sync_fetch_and_add(&dropped, 1);
52+ return 1;
53+ }
54+
55+ sample->pid = pid;
56+ bpf_get_current_comm(sample->comm, sizeof(sample->comm));
57+ sample->value = value;
58+
59+ sample->seq = seq++;
60+ __sync_fetch_and_add(&total, 1);
61+
62+ if (sample->seq & 1) {
63+ /* copy from reserved sample to a new one... */
64+ bpf_ringbuf_output(&ringbuf, sample, sizeof(*sample), flags);
65+ /* ...and then discard reserved sample */
66+ bpf_ringbuf_discard(sample, flags);
67+ __sync_fetch_and_add(&discarded, 1);
68+ } else {
69+ bpf_ringbuf_submit(sample, flags);
70+ }
71+
72+ avail_data = bpf_ringbuf_query(&ringbuf, BPF_RB_AVAIL_DATA);
73+ ring_size = bpf_ringbuf_query(&ringbuf, BPF_RB_RING_SIZE);
74+ cons_pos = bpf_ringbuf_query(&ringbuf, BPF_RB_CONS_POS);
75+ prod_pos = bpf_ringbuf_query(&ringbuf, BPF_RB_PROD_POS);
76+
77+ return 0;
78+}
tools/testing/selftests/bpf/progs/test_ringbuf_multi.c+77−0
@@ -0,0 +1,77 @@
1+// SPDX-License-Identifier: GPL-2.0
2+// Copyright (c) 2020 Facebook
3+
4+#include <linux/bpf.h>
5+#include <bpf/bpf_helpers.h>
6+
7+char _license[] SEC("license") = "GPL";
8+
9+struct sample {
10+ int pid;
11+ int seq;
12+ long value;
13+ char comm[16];
14+};
15+
16+struct ringbuf_map {
17+ __uint(type, BPF_MAP_TYPE_RINGBUF);
18+ __uint(max_entries, 1 << 12);
19+} ringbuf1 SEC(".maps"),
20+ ringbuf2 SEC(".maps");
21+
22+struct {
23+ __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
24+ __uint(max_entries, 4);
25+ __type(key, int);
26+ __array(values, struct ringbuf_map);
27+} ringbuf_arr SEC(".maps") = {
28+ .values = {
29+ [0] = &ringbuf1,
30+ [2] = &ringbuf2,
31+ },
32+};
33+
34+/* inputs */
35+int pid = 0;
36+int target_ring = 0;
37+long value = 0;
38+
39+/* outputs */
40+long total = 0;
41+long dropped = 0;
42+long skipped = 0;
43+
44+SEC("tp/syscalls/sys_enter_getpgid")
45+int test_ringbuf(void *ctx)
46+{
47+ int cur_pid = bpf_get_current_pid_tgid() >> 32;
48+ struct sample *sample;
49+ void *rb;
50+ int zero = 0;
51+
52+ if (cur_pid != pid)
53+ return 0;
54+
55+ rb = bpf_map_lookup_elem(&ringbuf_arr, &target_ring);
56+ if (!rb) {
57+ skipped += 1;
58+ return 1;
59+ }
60+
61+ sample = bpf_ringbuf_reserve(rb, sizeof(*sample), 0);
62+ if (!sample) {
63+ dropped += 1;
64+ return 1;
65+ }
66+
67+ sample->pid = pid;
68+ bpf_get_current_comm(sample->comm, sizeof(sample->comm));
69+ sample->value = value;
70+
71+ sample->seq = total;
72+ total += 1;
73+
74+ bpf_ringbuf_submit(sample, 0);
75+
76+ return 0;
77+}
tools/testing/selftests/bpf/verifier/and.c+2−2
@@ -15,7 +15,7 @@ · 14 unmodified lines
1515 BPF_EXIT_INSN(),
1616 },
1717 .fixup_map_hash_48b = { 3 },
18− .errstr = "R0 max value is outside of the array range",
18+ .errstr = "R0 max value is outside of the allowed memory range",
1919 .result = REJECT,
2020 .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,
2121 },
@@ -44,7 +44,7 @@ · 22 unmodified lines
4444 BPF_EXIT_INSN(),
4545 },
4646 .fixup_map_hash_48b = { 3 },
47− .errstr = "R0 max value is outside of the array range",
47+ .errstr = "R0 max value is outside of the allowed memory range",
4848 .result = REJECT,
4949 .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,
5050 },
tools/testing/selftests/bpf/verifier/array_access.c+2−2
@@ -117,7 +117,7 @@ · 116 unmodified lines
117117 BPF_EXIT_INSN(),
118118 },
119119 .fixup_map_hash_48b = { 3 },
120− .errstr = "R0 min value is outside of the array range",
120+ .errstr = "R0 min value is outside of the allowed memory range",
121121 .result = REJECT,
122122 .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,
123123 },
@@ -137,7 +137,7 @@ · 13 unmodified lines
137137 BPF_EXIT_INSN(),
138138 },
139139 .fixup_map_hash_48b = { 3 },
140− .errstr = "R0 unbounded memory access, make sure to bounds check any array access into a map",
140+ .errstr = "R0 unbounded memory access, make sure to bounds check any such access",
141141 .result = REJECT,
142142 .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,
143143 },
tools/testing/selftests/bpf/verifier/bounds.c+3−3
@@ -20,7 +20,7 @@ · 19 unmodified lines
2020 BPF_EXIT_INSN(),
2121 },
2222 .fixup_map_hash_8b = { 3 },
23− .errstr = "R0 max value is outside of the array range",
23+ .errstr = "R0 max value is outside of the allowed memory range",
2424 .result = REJECT,
2525 },
2626 {
@@ -146,7 +146,7 @@ · 119 unmodified lines
146146 BPF_EXIT_INSN(),
147147 },
148148 .fixup_map_hash_8b = { 3 },
149− .errstr = "R0 min value is outside of the array range",
149+ .errstr = "R0 min value is outside of the allowed memory range",
150150 .result = REJECT
151151 },
152152 {
@@ -354,7 +354,7 @@ · 201 unmodified lines
354354 BPF_EXIT_INSN(),
355355 },
356356 .fixup_map_hash_8b = { 3 },
357− .errstr = "R0 max value is outside of the array range",
357+ .errstr = "R0 max value is outside of the allowed memory range",
358358 .result = REJECT
359359 },
360360 {
tools/testing/selftests/bpf/verifier/calls.c+1−1
@@ -105,7 +105,7 @@ · 104 unmodified lines
105105 .prog_type = BPF_PROG_TYPE_SCHED_CLS,
106106 .fixup_map_hash_8b = { 16 },
107107 .result = REJECT,
108− .errstr = "R0 min value is outside of the array range",
108+ .errstr = "R0 min value is outside of the allowed memory range",
109109 },
110110 {
111111 "calls: overlapping caller/callee",
tools/testing/selftests/bpf/verifier/direct_value_access.c+2−2
@@ -68,7 +68,7 @@ · 67 unmodified lines
6868 },
6969 .fixup_map_array_48b = { 1 },
7070 .result = REJECT,
71− .errstr = "R1 min value is outside of the array range",
71+ .errstr = "R1 min value is outside of the allowed memory range",
7272 },
7373 {
7474 "direct map access, write test 7",
@@ -220,7 +220,7 @@ · 145 unmodified lines
220220 },
221221 .fixup_map_array_small = { 1 },
222222 .result = REJECT,
223− .errstr = "R1 min value is outside of the array range",
223+ .errstr = "R1 min value is outside of the allowed memory range",
224224 },
225225 {
226226 "direct map access, write test 19",
tools/testing/selftests/bpf/verifier/helper_access_var_len.c+1−1
@@ -318,7 +318,7 @@ · 317 unmodified lines
318318 BPF_EXIT_INSN(),
319319 },
320320 .fixup_map_hash_48b = { 4 },
321− .errstr = "R1 min value is outside of the array range",
321+ .errstr = "R1 min value is outside of the allowed memory range",
322322 .result = REJECT,
323323 .prog_type = BPF_PROG_TYPE_TRACEPOINT,
324324 },
tools/testing/selftests/bpf/verifier/helper_value_access.c+3−3
@@ -280,7 +280,7 @@ · 279 unmodified lines
280280 BPF_EXIT_INSN(),
281281 },
282282 .fixup_map_hash_48b = { 3 },
283− .errstr = "R1 min value is outside of the array range",
283+ .errstr = "R1 min value is outside of the allowed memory range",
284284 .result = REJECT,
285285 .prog_type = BPF_PROG_TYPE_TRACEPOINT,
286286 },
@@ -415,7 +415,7 @@ · 128 unmodified lines
415415 BPF_EXIT_INSN(),
416416 },
417417 .fixup_map_hash_48b = { 3 },
418− .errstr = "R1 min value is outside of the array range",
418+ .errstr = "R1 min value is outside of the allowed memory range",
419419 .result = REJECT,
420420 .prog_type = BPF_PROG_TYPE_TRACEPOINT,
421421 },
@@ -926,7 +926,7 @@ · 504 unmodified lines
926926 },
927927 .fixup_map_hash_16b = { 3, 10 },
928928 .result = REJECT,
929− .errstr = "R2 unbounded memory access, make sure to bounds check any array access into a map",
929+ .errstr = "R2 unbounded memory access, make sure to bounds check any such access",
930930 .prog_type = BPF_PROG_TYPE_TRACEPOINT,
931931 },
932932 {
tools/testing/selftests/bpf/verifier/value_ptr_arith.c+4−4
@@ -50,7 +50,7 @@ · 49 unmodified lines
5050 .fixup_map_array_48b = { 8 },
5151 .result = ACCEPT,
5252 .result_unpriv = REJECT,
53− .errstr_unpriv = "R0 min value is outside of the array range",
53+ .errstr_unpriv = "R0 min value is outside of the allowed memory range",
5454 .retval = 1,
5555 },
5656 {
@@ -325,7 +325,7 @@ · 268 unmodified lines
325325 },
326326 .fixup_map_array_48b = { 3 },
327327 .result = REJECT,
328− .errstr = "R0 min value is outside of the array range",
328+ .errstr = "R0 min value is outside of the allowed memory range",
329329 .result_unpriv = REJECT,
330330 .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range",
331331 },
@@ -601,7 +601,7 @@ · 269 unmodified lines
601601 },
602602 .fixup_map_array_48b = { 3 },
603603 .result = REJECT,
604− .errstr = "R1 max value is outside of the array range",
604+ .errstr = "R1 max value is outside of the allowed memory range",
605605 .errstr_unpriv = "R1 pointer arithmetic of map value goes out of range",
606606 .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,
607607 },
@@ -726,7 +726,7 @@ · 118 unmodified lines
726726 },
727727 .fixup_map_array_48b = { 3 },
728728 .result = REJECT,
729− .errstr = "R0 min value is outside of the array range",
729+ .errstr = "R0 min value is outside of the allowed memory range",
730730 },
731731 {
732732 "map access: value_ptr -= known scalar, 2",