History

Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Daniel Borkmann says: ==================== pull-request: bpf-next 2022-09-05 The following pull-request contains BPF updates for your *net-next* tree. We've added 106 non-merge commits during the last 18 day(s) which contain a total of 159 files changed, 5225 insertions(+), 1358 deletions(-). There are two small merge conflicts, resolve them as follows: 1) tools/testing/selftests/bpf/DENYLIST.s390x Commit 27e23836ce22 ("selftests/bpf: Add lru_bug to s390x deny list") in bpf tree was needed to get BPF CI green on s390x, but it conflicted with newly added tests on bpf-next. Resolve by adding both hunks, result: [...] lru_bug # prog 'printk': failed to auto-attach: -524 setget_sockopt # attach unexpected error: -524 (trampoline) cb_refs # expected error message unexpected error: -524 (trampoline) cgroup_hierarchical_stats # JIT does not support calling kernel function (kfunc) htab_update # failed to attach: ERROR: strerror_r(-524)=22 (trampoline) [...] 2) net/core/filter.c Commit 1227c1771dd2 ("net: Fix data-races around sysctl_[rw]mem_(max|default).") from net tree conflicts with commit 29003875bd5b ("bpf: Change bpf_setsockopt(SOL_SOCKET) to reuse sk_setsockopt()") from bpf-next tree. Take the code as it is from bpf-next tree, result: [...] if (getopt) { if (optname == SO_BINDTODEVICE) return -EINVAL; return sk_getsockopt(sk, SOL_SOCKET, optname, KERNEL_SOCKPTR(optval), KERNEL_SOCKPTR(optlen)); } return sk_setsockopt(sk, SOL_SOCKET, optname, KERNEL_SOCKPTR(optval), *optlen); [...] The main changes are: 1) Add any-context BPF specific memory allocator which is useful in particular for BPF tracing with bonus of performance equal to full prealloc, from Alexei Starovoitov. 2) Big batch to remove duplicated code from bpf_{get,set}sockopt() helpers as an effort to reuse the existing core socket code as much as possible, from Martin KaFai Lau. 3) Extend BPF flow dissector for BPF programs to just augment the in-kernel dissector with custom logic. In other words, allow for partial replacement, from Shmulik Ladkani. 4) Add a new cgroup iterator to BPF with different traversal options, from Hao Luo. 5) Support for BPF to collect hierarchical cgroup statistics efficiently through BPF integration with the rstat framework, from Yosry Ahmed. 6) Support bpf_{g,s}et_retval() under more BPF cgroup hooks, from Stanislav Fomichev. 7) BPF hash table and local storages fixes under fully preemptible kernel, from Hou Tao. 8) Add various improvements to BPF selftests and libbpf for compilation with gcc BPF backend, from James Hilliard. 9) Fix verifier helper permissions and reference state management for synchronous callbacks, from Kumar Kartikeya Dwivedi. 10) Add support for BPF selftest's xskxceiver to also be used against real devices that support MAC loopback, from Maciej Fijalkowski. 11) Various fixes to the bpf-helpers(7) man page generation script, from Quentin Monnet. 12) Document BPF verifier's tnum_in(tnum_range(), ...) gotchas, from Shung-Hsi Yu. 13) Various minor misc improvements all over the place. * https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (106 commits) bpf: Optimize rcu_barrier usage between hash map and bpf_mem_alloc. bpf: Remove usage of kmem_cache from bpf_mem_cache. bpf: Remove prealloc-only restriction for sleepable bpf programs. bpf: Prepare bpf_mem_alloc to be used by sleepable bpf programs. bpf: Remove tracing program restriction on map types bpf: Convert percpu hash map to per-cpu bpf_mem_alloc. bpf: Add percpu allocation support to bpf_mem_alloc. bpf: Batch call_rcu callbacks instead of SLAB_TYPESAFE_BY_RCU. bpf: Adjust low/high watermarks in bpf_mem_cache bpf: Optimize call_rcu in non-preallocated hash map. bpf: Optimize element count in non-preallocated hash map. bpf: Relax the requirement to use preallocated hash maps in tracing progs. samples/bpf: Reduce syscall overhead in map_perf_test. selftests/bpf: Improve test coverage of test_maps bpf: Convert hash map to bpf_mem_alloc. bpf: Introduce any context BPF specific memory allocator. selftest/bpf: Add test for bpf_getsockopt() bpf: Change bpf_getsockopt(SOL_IPV6) to reuse do_ipv6_getsockopt() bpf: Change bpf_getsockopt(SOL_IP) to reuse do_ip_getsockopt() bpf: Change bpf_getsockopt(SOL_TCP) to reuse do_tcp_getsockopt() ... ====================

Paolo Abeni committed 4 years ago

Commit 2786bcf2 parents 03fdb11 274052a159 files · +5225 −1358Browse files

Unified · Split

159 files
  1. arch/mips/net/bpf_jit_comp32.c +9 −1
  2. arch/mips/net/bpf_jit_comp64.c +9 −1
  3. include/linux/bpf-cgroup.h +17 −0
  4. include/linux/bpf.h +22 −0
  5. include/linux/bpf_mem_alloc.h +28 −0
  6. include/linux/bpf_verifier.h +11 −0
  7. include/linux/filter.h +1 −2
  8. include/linux/igmp.h +2 −2
  9. include/linux/mroute.h +3 −3
  10. include/linux/mroute6.h +2 −2
  11. include/linux/skbuff.h +2 −2
  12. include/linux/sockptr.h +5 −0
  13. include/linux/tnum.h +18 −2
  14. include/net/ip.h +4 −0
  15. include/net/ipv6.h +5 −1
  16. include/net/ipv6_stubs.h +4 −0
  17. include/net/sock.h +9 −0
  18. include/net/tcp.h +4 −0
  19. include/uapi/linux/bpf.h +69 −14
  20. kernel/bpf/Makefile +4 −1
  21. kernel/bpf/bpf_iter.c +5 −0
  22. kernel/bpf/bpf_local_storage.c +2 −2
  23. kernel/bpf/bpf_lsm.c +14 −9
  24. kernel/bpf/bpf_task_storage.c +4 −4
  25. kernel/bpf/cgroup.c +134 −23
  26. kernel/bpf/cgroup_iter.c +282 −0
  27. kernel/bpf/hashtab.c +123 −45
  28. kernel/bpf/helpers.c +9 −39
  29. kernel/bpf/memalloc.c +634 −0
  30. kernel/bpf/syscall.c +10 −5
  31. kernel/bpf/trampoline.c +4 −4
  32. kernel/bpf/verifier.c +33 −61
  33. kernel/cgroup/rstat.c +48 −0
  34. net/core/filter.c +262 −373
  35. net/core/flow_dissector.c +10 −6
  36. net/core/sock.c +94 −40
  37. net/ipv4/igmp.c +13 −9
  38. net/ipv4/ip_sockglue.c +62 −52
  39. net/ipv4/ipmr.c +5 −4
  40. net/ipv4/tcp.c +59 −57
  41. net/ipv6/af_inet6.c +2 −0
  42. net/ipv6/ip6mr.c +5 −5
  43. net/ipv6/ipv6_sockglue.c +59 −54
  44. net/ipv6/mcast.c +4 −4
  45. samples/bpf/map_perf_test_kern.c +28 −16
  46. samples/bpf/map_perf_test_user.c +1 −1
  47. scripts/bpf_doc.py +71 −7
  48. tools/bpf/bpftool/btf_dumper.c +1 −1
  49. tools/bpf/bpftool/link.c +35 −0
  50. tools/include/uapi/linux/bpf.h +69 −14
  51. tools/lib/bpf/bpf_helpers.h +13 −6
  52. tools/lib/bpf/skel_internal.h +23 −0
  53. tools/testing/selftests/bpf/DENYLIST.s390x +4 −0
  54. tools/testing/selftests/bpf/Makefile +19 −18
  55. tools/testing/selftests/bpf/README.rst +4 −4
  56. tools/testing/selftests/bpf/cgroup_getset_retval_hooks.h +25 −0
  57. tools/testing/selftests/bpf/cgroup_helpers.c +161 −41
  58. tools/testing/selftests/bpf/cgroup_helpers.h +13 −6
  59. tools/testing/selftests/bpf/get_cgroup_id_user.c +1 −1
  60. tools/testing/selftests/bpf/map_tests/task_storage_map.c +122 −0
  61. tools/testing/selftests/bpf/prog_tests/bpf_obj_id.c +1 −1
  62. tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c +27 −27
  63. tools/testing/selftests/bpf/prog_tests/btf.c +2 −2
  64. tools/testing/selftests/bpf/prog_tests/btf_dump.c +5 −5
  65. tools/testing/selftests/bpf/prog_tests/btf_endian.c +1 −1
  66. tools/testing/selftests/bpf/prog_tests/cb_refs.c +48 −0
  67. tools/testing/selftests/bpf/prog_tests/cgroup_getset_retval.c +48 −0
  68. tools/testing/selftests/bpf/prog_tests/cgroup_hierarchical_stats.c +357 −0
  69. tools/testing/selftests/bpf/prog_tests/cgroup_iter.c +224 −0
  70. tools/testing/selftests/bpf/prog_tests/connect_force_port.c +1 −1
  71. tools/testing/selftests/bpf/prog_tests/core_reloc.c +37 −37
  72. tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c +22 −22
  73. tools/testing/selftests/bpf/prog_tests/flow_dissector.c +43 −1
  74. tools/testing/selftests/bpf/prog_tests/flow_dissector_load_bytes.c +1 −1
  75. tools/testing/selftests/bpf/prog_tests/get_stack_raw_tp.c +2 −2
  76. tools/testing/selftests/bpf/prog_tests/global_data.c +1 −1
  77. tools/testing/selftests/bpf/prog_tests/global_data_init.c +1 −1
  78. tools/testing/selftests/bpf/prog_tests/global_func_args.c +1 −1
  79. tools/testing/selftests/bpf/prog_tests/htab_update.c +126 −0
  80. tools/testing/selftests/bpf/prog_tests/kfree_skb.c +1 −1
  81. tools/testing/selftests/bpf/prog_tests/kfunc_call.c +1 −1
  82. tools/testing/selftests/bpf/prog_tests/l4lb_all.c +2 −2
  83. tools/testing/selftests/bpf/prog_tests/load_bytes_relative.c +2 −2
  84. tools/testing/selftests/bpf/prog_tests/map_lock.c +1 −1
  85. tools/testing/selftests/bpf/prog_tests/pinning.c +2 −2
  86. tools/testing/selftests/bpf/prog_tests/pkt_access.c +1 −1
  87. tools/testing/selftests/bpf/prog_tests/pkt_md_access.c +1 −1
  88. tools/testing/selftests/bpf/prog_tests/probe_user.c +1 −1
  89. tools/testing/selftests/bpf/prog_tests/queue_stack_map.c +2 −2
  90. tools/testing/selftests/bpf/prog_tests/rdonly_maps.c +1 −1
  91. tools/testing/selftests/bpf/prog_tests/reference_tracking.c +1 −1
  92. tools/testing/selftests/bpf/prog_tests/resolve_btfids.c +1 −1
  93. tools/testing/selftests/bpf/prog_tests/select_reuseport.c +2 −2
  94. tools/testing/selftests/bpf/prog_tests/setget_sockopt.c +125 −0
  95. tools/testing/selftests/bpf/prog_tests/sk_assign.c +1 −1
  96. tools/testing/selftests/bpf/prog_tests/skb_ctx.c +1 −1
  97. tools/testing/selftests/bpf/prog_tests/skb_helpers.c +1 −1
  98. tools/testing/selftests/bpf/prog_tests/sockopt_inherit.c +1 −1
  99. tools/testing/selftests/bpf/prog_tests/sockopt_multi.c +1 −1
  100. tools/testing/selftests/bpf/prog_tests/spinlock.c +1 −1
  101. tools/testing/selftests/bpf/prog_tests/stacktrace_map.c +1 −1
  102. tools/testing/selftests/bpf/prog_tests/stacktrace_map_raw_tp.c +1 −1
  103. tools/testing/selftests/bpf/prog_tests/tailcalls.c +18 −18
  104. tools/testing/selftests/bpf/prog_tests/task_fd_query_rawtp.c +1 −1
  105. tools/testing/selftests/bpf/prog_tests/task_fd_query_tp.c +1 −1
  106. tools/testing/selftests/bpf/prog_tests/tcp_estats.c +1 −1
  107. tools/testing/selftests/bpf/prog_tests/test_bprm_opts.c +1 −9
  108. tools/testing/selftests/bpf/prog_tests/test_global_funcs.c +17 −17
  109. tools/testing/selftests/bpf/prog_tests/test_local_storage.c +1 −9
  110. tools/testing/selftests/bpf/prog_tests/test_overhead.c +1 −1
  111. tools/testing/selftests/bpf/prog_tests/tp_attach_query.c +1 −1
  112. tools/testing/selftests/bpf/prog_tests/trampoline_count.c +1 −1
  113. tools/testing/selftests/bpf/prog_tests/xdp.c +1 −1
  114. tools/testing/selftests/bpf/prog_tests/xdp_adjust_frags.c +1 −1
  115. tools/testing/selftests/bpf/prog_tests/xdp_adjust_tail.c +5 −5
  116. tools/testing/selftests/bpf/prog_tests/xdp_attach.c +1 −1
  117. tools/testing/selftests/bpf/prog_tests/xdp_info.c +1 −1
  118. tools/testing/selftests/bpf/prog_tests/xdp_perf.c +1 −1
  119. tools/testing/selftests/bpf/prog_tests/xdp_synproxy.c +1 −1
  120. tools/testing/selftests/bpf/progs/bind4_prog.c +0 −2
  121. tools/testing/selftests/bpf/progs/bind6_prog.c +0 −2
  122. tools/testing/selftests/bpf/progs/bpf_flow.c +15 −0
  123. tools/testing/selftests/bpf/progs/bpf_iter.h +7 −0
  124. tools/testing/selftests/bpf/progs/bpf_tracing_net.h +31 −1
  125. tools/testing/selftests/bpf/progs/cb_refs.c +116 −0
  126. tools/testing/selftests/bpf/progs/cgroup_getset_retval_hooks.c +16 −0
  127. tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c +226 −0
  128. tools/testing/selftests/bpf/progs/cgroup_iter.c +39 −0
  129. tools/testing/selftests/bpf/progs/connect4_prog.c +3 −2
  130. tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c +4 −4
  131. tools/testing/selftests/bpf/progs/htab_update.c +29 −0
  132. tools/testing/selftests/bpf/progs/read_bpf_task_storage_busy.c +39 −0
  133. tools/testing/selftests/bpf/progs/setget_sockopt.c +395 −0
  134. tools/testing/selftests/bpf/progs/tailcall_bpf2bpf4.c +1 −1
  135. tools/testing/selftests/bpf/progs/test_tc_dtime.c +0 −1
  136. tools/testing/selftests/bpf/progs/test_tunnel_kern.c +16 −8
  137. tools/testing/selftests/bpf/progs/timer.c +0 −11
  138. tools/testing/selftests/bpf/task_local_storage_helpers.h +18 −0
  139. tools/testing/selftests/bpf/test_dev_cgroup.c +1 −1
  140. tools/testing/selftests/bpf/test_flow_dissector.sh +8 −0
  141. tools/testing/selftests/bpf/test_lirc_mode2_user.c +1 −1
  142. tools/testing/selftests/bpf/test_maps.c +29 −19
  143. tools/testing/selftests/bpf/test_offload.py +11 −11
  144. tools/testing/selftests/bpf/test_skb_cgroup_id.sh +1 −1
  145. tools/testing/selftests/bpf/test_sock_addr.c +8 −8
  146. tools/testing/selftests/bpf/test_sockmap.c +2 −2
  147. tools/testing/selftests/bpf/test_sysctl.c +3 −3
  148. tools/testing/selftests/bpf/test_tcp_check_syncookie.sh +1 −1
  149. tools/testing/selftests/bpf/test_tcpnotify_user.c +1 −1
  150. tools/testing/selftests/bpf/test_xdp_redirect.sh +4 −4
  151. tools/testing/selftests/bpf/test_xdp_redirect_multi.sh +1 −1
  152. tools/testing/selftests/bpf/test_xdp_veth.sh +4 −4
  153. tools/testing/selftests/bpf/test_xsk.sh +36 −16
  154. tools/testing/selftests/bpf/xdp_redirect_multi.c +1 −1
  155. tools/testing/selftests/bpf/xdp_synproxy.c +1 −1
  156. tools/testing/selftests/bpf/xdping.c +1 −1
  157. tools/testing/selftests/bpf/xsk.c +3 −3
  158. tools/testing/selftests/bpf/xskxceiver.c +296 −102
  159. tools/testing/selftests/bpf/xskxceiver.h +8 −3
arch/mips/net/bpf_jit_comp32.c+9−1
@@ -1376,12 +1376,20 @@ void build_prologue(struct jit_context *ctx) · 1375 unmodified lines
13761376 const u8 *fp = bpf2mips32[BPF_REG_FP];
13771377 int stack, saved, locals, reserved;
13781378
1379+ /*
1380+ * In the unlikely event that the TCC limit is raised to more
1381+ * than 16 bits, it is clamped to the maximum value allowed for
1382+ * the generated code (0xffff). It is better fail to compile
1383+ * instead of degrading gracefully.
1384+ */
1385+ BUILD_BUG_ON(MAX_TAIL_CALL_CNT > 0xffff);
1386+
13791387 /*
13801388 * The first two instructions initialize TCC in the reserved (for us)
13811389 * 16-byte area in the parent's stack frame. On a tail call, the
13821390 * calling function jumps into the prologue after these instructions.
13831391 */
1384− emit(ctx, ori, MIPS_R_T9, MIPS_R_ZERO, min(MAX_TAIL_CALL_CNT, 0xffff));
1392+ emit(ctx, ori, MIPS_R_T9, MIPS_R_ZERO, MAX_TAIL_CALL_CNT);
13851393 emit(ctx, sw, MIPS_R_T9, 0, MIPS_R_SP);
13861394
13871395 /*
arch/mips/net/bpf_jit_comp64.c+9−1
@@ -547,12 +547,20 @@ void build_prologue(struct jit_context *ctx) · 546 unmodified lines
547547 u8 zx = bpf2mips64[JIT_REG_ZX];
548548 int stack, saved, locals, reserved;
549549
550+ /*
551+ * In the unlikely event that the TCC limit is raised to more
552+ * than 16 bits, it is clamped to the maximum value allowed for
553+ * the generated code (0xffff). It is better fail to compile
554+ * instead of degrading gracefully.
555+ */
556+ BUILD_BUG_ON(MAX_TAIL_CALL_CNT > 0xffff);
557+
550558 /*
551559 * The first instruction initializes the tail call count register.
552560 * On a tail call, the calling function jumps into the prologue
553561 * after this instruction.
554562 */
555− emit(ctx, ori, tc, MIPS_R_ZERO, min(MAX_TAIL_CALL_CNT, 0xffff));
563+ emit(ctx, ori, tc, MIPS_R_ZERO, MAX_TAIL_CALL_CNT);
556564
557565 /* === Entry-point for tail calls === */
558566
include/linux/bpf-cgroup.h+17−0
@@ -414,6 +414,11 @@ int cgroup_bpf_prog_detach(const union bpf_attr *attr, · 413 unmodified lines
414414 int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
415415 int cgroup_bpf_prog_query(const union bpf_attr *attr,
416416 union bpf_attr __user *uattr);
417+
418+const struct bpf_func_proto *
419+cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);
420+const struct bpf_func_proto *
421+cgroup_current_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);
417422 #else
418423
419424 static inline int cgroup_bpf_inherit(struct cgroup *cgrp) { return 0; }
@@ -444,6 +449,18 @@ static inline int cgroup_bpf_prog_query(const union bpf_attr *attr, · 24 unmodified lines
444449 return -EINVAL;
445450 }
446451
452+static inline const struct bpf_func_proto *
453+cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
454+{
455+ return NULL;
456+}
457+
458+static inline const struct bpf_func_proto *
459+cgroup_current_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
460+{
461+ return NULL;
462+}
463+
447464 static inline int bpf_cgroup_storage_assign(struct bpf_prog_aux *aux,
448465 struct bpf_map *map) { return 0; }
449466 static inline struct bpf_cgroup_storage *bpf_cgroup_storage_alloc(
include/linux/bpf.h+22−0
@@ -48,6 +48,7 @@ struct mem_cgroup; · 47 unmodified lines
4848 struct module;
4949 struct bpf_func_state;
5050 struct ftrace_ops;
51+struct cgroup;
5152
5253 extern struct idr btf_idr;
5354 extern spinlock_t btf_idr_lock;
@@ -1730,7 +1731,14 @@ int bpf_obj_get_user(const char __user *pathname, int flags); · 1676 unmodified lines
17301731 int __init bpf_iter_ ## target(args) { return 0; }
17311732
17321733 struct bpf_iter_aux_info {
1734+ /* for map_elem iter */
17331735 struct bpf_map *map;
1736+
1737+ /* for cgroup iter */
1738+ struct {
1739+ struct cgroup *start; /* starting cgroup */
1740+ enum bpf_cgroup_iter_order order;
1741+ } cgroup;
17341742 };
17351743
17361744 typedef int (*bpf_iter_attach_target_t)(struct bpf_prog *prog,
@@ -1966,6 +1974,15 @@ static inline bool unprivileged_ebpf_enabled(void) · 229 unmodified lines
19661974 return !sysctl_unprivileged_bpf_disabled;
19671975 }
19681976
1977+/* Not all bpf prog type has the bpf_ctx.
1978+ * For the bpf prog type that has initialized the bpf_ctx,
1979+ * this function can be used to decide if a kernel function
1980+ * is called by a bpf program.
1981+ */
1982+static inline bool has_current_bpf_ctx(void)
1983+{
1984+ return !!current->bpf_ctx;
1985+}
19691986 #else /* !CONFIG_BPF_SYSCALL */
19701987 static inline struct bpf_prog *bpf_prog_get(u32 ufd)
19711988 {
@@ -2175,6 +2192,10 @@ static inline bool unprivileged_ebpf_enabled(void) · 203 unmodified lines
21752192 return false;
21762193 }
21772194
2195+static inline bool has_current_bpf_ctx(void)
2196+{
2197+ return false;
2198+}
21782199 #endif /* CONFIG_BPF_SYSCALL */
21792200
21802201 void __bpf_free_used_btfs(struct bpf_prog_aux *aux,
@@ -2362,6 +2383,7 @@ extern const struct bpf_func_proto bpf_sock_map_update_proto; · 181 unmodified lines
23622383 extern const struct bpf_func_proto bpf_sock_hash_update_proto;
23632384 extern const struct bpf_func_proto bpf_get_current_cgroup_id_proto;
23642385 extern const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto;
2386+extern const struct bpf_func_proto bpf_get_cgroup_classid_curr_proto;
23652387 extern const struct bpf_func_proto bpf_msg_redirect_hash_proto;
23662388 extern const struct bpf_func_proto bpf_msg_redirect_map_proto;
23672389 extern const struct bpf_func_proto bpf_sk_redirect_hash_proto;
include/linux/bpf_mem_alloc.h+28−0
@@ -0,0 +1,28 @@
1+/* SPDX-License-Identifier: GPL-2.0-only */
2+/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */
3+#ifndef _BPF_MEM_ALLOC_H
4+#define _BPF_MEM_ALLOC_H
5+#include <linux/compiler_types.h>
6+#include <linux/workqueue.h>
7+
8+struct bpf_mem_cache;
9+struct bpf_mem_caches;
10+
11+struct bpf_mem_alloc {
12+ struct bpf_mem_caches __percpu *caches;
13+ struct bpf_mem_cache __percpu *cache;
14+ struct work_struct work;
15+};
16+
17+int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu);
18+void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma);
19+
20+/* kmalloc/kfree equivalent: */
21+void *bpf_mem_alloc(struct bpf_mem_alloc *ma, size_t size);
22+void bpf_mem_free(struct bpf_mem_alloc *ma, void *ptr);
23+
24+/* kmem_cache_alloc/free equivalent: */
25+void *bpf_mem_cache_alloc(struct bpf_mem_alloc *ma);
26+void bpf_mem_cache_free(struct bpf_mem_alloc *ma, void *ptr);
27+
28+#endif /* _BPF_MEM_ALLOC_H */
include/linux/bpf_verifier.h+11−0
@@ -212,6 +212,17 @@ struct bpf_reference_state { · 211 unmodified lines
212212 * is used purely to inform the user of a reference leak.
213213 */
214214 int insn_idx;
215+ /* There can be a case like:
216+ * main (frame 0)
217+ * cb (frame 1)
218+ * func (frame 3)
219+ * cb (frame 4)
220+ * Hence for frame 4, if callback_ref just stored boolean, it would be
221+ * impossible to distinguish nested callback refs. Hence store the
222+ * frameno and compare that to callback_ref in check_reference_leak when
223+ * exiting a callback function.
224+ */
225+ int callback_ref;
215226 };
216227
217228 /* state of the program:
include/linux/filter.h+1−2
@@ -900,8 +900,7 @@ int sk_reuseport_attach_filter(struct sock_fprog *fprog, struct sock *sk); · 899 unmodified lines
900900 int sk_reuseport_attach_bpf(u32 ufd, struct sock *sk);
901901 void sk_reuseport_prog_free(struct bpf_prog *prog);
902902 int sk_detach_filter(struct sock *sk);
903−int sk_get_filter(struct sock *sk, struct sock_filter __user *filter,
904− unsigned int len);
903+int sk_get_filter(struct sock *sk, sockptr_t optval, unsigned int len);
905904
906905 bool sk_filter_charge(struct sock *sk, struct sk_filter *fp);
907906 void sk_filter_uncharge(struct sock *sk, struct sk_filter *fp);
include/linux/igmp.h+2−2
@@ -118,9 +118,9 @@ extern int ip_mc_source(int add, int omode, struct sock *sk, · 117 unmodified lines
118118 struct ip_mreq_source *mreqs, int ifindex);
119119 extern int ip_mc_msfilter(struct sock *sk, struct ip_msfilter *msf,int ifindex);
120120 extern int ip_mc_msfget(struct sock *sk, struct ip_msfilter *msf,
121− struct ip_msfilter __user *optval, int __user *optlen);
121+ sockptr_t optval, sockptr_t optlen);
122122 extern int ip_mc_gsfget(struct sock *sk, struct group_filter *gsf,
123− struct sockaddr_storage __user *p);
123+ sockptr_t optval, size_t offset);
124124 extern int ip_mc_sf_allow(struct sock *sk, __be32 local, __be32 rmt,
125125 int dif, int sdif);
126126 extern void ip_mc_init_dev(struct in_device *);
include/linux/mroute.h+3−3
@@ -17,7 +17,7 @@ static inline int ip_mroute_opt(int opt) · 16 unmodified lines
1717 }
1818
1919 int ip_mroute_setsockopt(struct sock *, int, sockptr_t, unsigned int);
20−int ip_mroute_getsockopt(struct sock *, int, char __user *, int __user *);
20+int ip_mroute_getsockopt(struct sock *, int, sockptr_t, sockptr_t);
2121 int ipmr_ioctl(struct sock *sk, int cmd, void __user *arg);
2222 int ipmr_compat_ioctl(struct sock *sk, unsigned int cmd, void __user *arg);
2323 int ip_mr_init(void);
@@ -29,8 +29,8 @@ static inline int ip_mroute_setsockopt(struct sock *sock, int optname, · 5 unmodified lines
2929 return -ENOPROTOOPT;
3030 }
3131
32−static inline int ip_mroute_getsockopt(struct sock *sock, int optname,
33− char __user *optval, int __user *optlen)
32+static inline int ip_mroute_getsockopt(struct sock *sk, int optname,
33+ sockptr_t optval, sockptr_t optlen)
3434 {
3535 return -ENOPROTOOPT;
3636 }
include/linux/mroute6.h+2−2
@@ -27,7 +27,7 @@ struct sock; · 26 unmodified lines
2727
2828 #ifdef CONFIG_IPV6_MROUTE
2929 extern int ip6_mroute_setsockopt(struct sock *, int, sockptr_t, unsigned int);
30−extern int ip6_mroute_getsockopt(struct sock *, int, char __user *, int __user *);
30+extern int ip6_mroute_getsockopt(struct sock *, int, sockptr_t, sockptr_t);
3131 extern int ip6_mr_input(struct sk_buff *skb);
3232 extern int ip6mr_ioctl(struct sock *sk, int cmd, void __user *arg);
3333 extern int ip6mr_compat_ioctl(struct sock *sk, unsigned int cmd, void __user *arg);
@@ -42,7 +42,7 @@ static inline int ip6_mroute_setsockopt(struct sock *sock, int optname, · 8 unmodified lines
4242
4343 static inline
4444 int ip6_mroute_getsockopt(struct sock *sock,
45− int optname, char __user *optval, int __user *optlen)
45+ int optname, sockptr_t optval, sockptr_t optlen)
4646 {
4747 return -ENOPROTOOPT;
4848 }
include/linux/skbuff.h+2−2
@@ -1461,8 +1461,8 @@ void skb_flow_dissector_init(struct flow_dissector *flow_dissector, · 1460 unmodified lines
14611461 unsigned int key_count);
14621462
14631463 struct bpf_flow_dissector;
1464−bool bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
1465− __be16 proto, int nhoff, int hlen, unsigned int flags);
1464+u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
1465+ __be16 proto, int nhoff, int hlen, unsigned int flags);
14661466
14671467 bool __skb_flow_dissect(const struct net *net,
14681468 const struct sk_buff *skb,
include/linux/sockptr.h+5−0
@@ -64,6 +64,11 @@ static inline int copy_to_sockptr_offset(sockptr_t dst, size_t offset, · 63 unmodified lines
6464 return 0;
6565 }
6666
67+static inline int copy_to_sockptr(sockptr_t dst, const void *src, size_t size)
68+{
69+ return copy_to_sockptr_offset(dst, 0, src, size);
70+}
71+
6772 static inline void *memdup_sockptr(sockptr_t src, size_t len)
6873 {
6974 void *p = kmalloc_track_caller(len, GFP_USER | __GFP_NOWARN);
include/linux/tnum.h+18−2
@@ -21,7 +21,12 @@ struct tnum { · 20 unmodified lines
2121 struct tnum tnum_const(u64 value);
2222 /* A completely unknown value */
2323 extern const struct tnum tnum_unknown;
24−/* A value that's unknown except that @min <= value <= @max */
24+/* An unknown value that is a superset of @min <= value <= @max.
25+ *
26+ * Could include values outside the range of [@min, @max].
27+ * For example tnum_range(0, 2) is represented by {0, 1, 2, *3*},
28+ * rather than the intended set of {0, 1, 2}.
29+ */
2530 struct tnum tnum_range(u64 min, u64 max);
2631
2732 /* Arithmetic and logical ops */
@@ -73,7 +78,18 @@ static inline bool tnum_is_unknown(struct tnum a) · 45 unmodified lines
7378 */
7479 bool tnum_is_aligned(struct tnum a, u64 size);
7580
76−/* Returns true if @b represents a subset of @a. */
81+/* Returns true if @b represents a subset of @a.
82+ *
83+ * Note that using tnum_range() as @a requires extra cautions as tnum_in() may
84+ * return true unexpectedly due to tnum limited ability to represent tight
85+ * range, e.g.
86+ *
87+ * tnum_in(tnum_range(0, 2), tnum_const(3)) == true
88+ *
89+ * As a rule of thumb, if @a is explicitly coded rather than coming from
90+ * reg->var_off, it should be in form of tnum_const(), tnum_range(0, 2**n - 1),
91+ * or tnum_range(2**n, 2**(n+1) - 1).
92+ */
7793 bool tnum_in(struct tnum a, struct tnum b);
7894
7995 /* Formatting functions. These have snprintf-like semantics: they will write
include/net/ip.h+4−0
@@ -743,8 +743,12 @@ void ip_cmsg_recv_offset(struct msghdr *msg, struct sock *sk, · 742 unmodified lines
743743 int ip_cmsg_send(struct sock *sk, struct msghdr *msg,
744744 struct ipcm_cookie *ipc, bool allow_ipv6);
745745 DECLARE_STATIC_KEY_FALSE(ip4_min_ttl);
746+int do_ip_setsockopt(struct sock *sk, int level, int optname, sockptr_t optval,
747+ unsigned int optlen);
746748 int ip_setsockopt(struct sock *sk, int level, int optname, sockptr_t optval,
747749 unsigned int optlen);
750+int do_ip_getsockopt(struct sock *sk, int level, int optname,
751+ sockptr_t optval, sockptr_t optlen);
748752 int ip_getsockopt(struct sock *sk, int level, int optname, char __user *optval,
749753 int __user *optlen);
750754 int ip_ra_control(struct sock *sk, unsigned char on,
include/net/ipv6.h+5−1
@@ -1156,8 +1156,12 @@ struct in6_addr *fl6_update_dst(struct flowi6 *fl6, · 1155 unmodified lines
11561156 */
11571157 DECLARE_STATIC_KEY_FALSE(ip6_min_hopcount);
11581158
1159+int do_ipv6_setsockopt(struct sock *sk, int level, int optname, sockptr_t optval,
1160+ unsigned int optlen);
11591161 int ipv6_setsockopt(struct sock *sk, int level, int optname, sockptr_t optval,
11601162 unsigned int optlen);
1163+int do_ipv6_getsockopt(struct sock *sk, int level, int optname,
1164+ sockptr_t optval, sockptr_t optlen);
11611165 int ipv6_getsockopt(struct sock *sk, int level, int optname,
11621166 char __user *optval, int __user *optlen);
11631167
@@ -1207,7 +1211,7 @@ int ip6_mc_source(int add, int omode, struct sock *sk, · 43 unmodified lines
12071211 int ip6_mc_msfilter(struct sock *sk, struct group_filter *gsf,
12081212 struct sockaddr_storage *list);
12091213 int ip6_mc_msfget(struct sock *sk, struct group_filter *gsf,
1210− struct sockaddr_storage __user *p);
1214+ sockptr_t optval, size_t ss_offset);
12111215
12121216 #ifdef CONFIG_PROC_FS
12131217 int ac6_proc_init(struct net *net);
include/net/ipv6_stubs.h+4−0
@@ -81,6 +81,10 @@ struct ipv6_bpf_stub { · 80 unmodified lines
8181 const struct in6_addr *daddr, __be16 dport,
8282 int dif, int sdif, struct udp_table *tbl,
8383 struct sk_buff *skb);
84+ int (*ipv6_setsockopt)(struct sock *sk, int level, int optname,
85+ sockptr_t optval, unsigned int optlen);
86+ int (*ipv6_getsockopt)(struct sock *sk, int level, int optname,
87+ sockptr_t optval, sockptr_t optlen);
8488 };
8589 extern const struct ipv6_bpf_stub *ipv6_bpf_stub __read_mostly;
8690
include/net/sock.h+9−0
@@ -1788,6 +1788,11 @@ static inline void unlock_sock_fast(struct sock *sk, bool slow) · 1787 unmodified lines
17881788 }
17891789 }
17901790
1791+void sockopt_lock_sock(struct sock *sk);
1792+void sockopt_release_sock(struct sock *sk);
1793+bool sockopt_ns_capable(struct user_namespace *ns, int cap);
1794+bool sockopt_capable(int cap);
1795+
17911796 /* Used by processes to "lock" a socket state, so that
17921797 * interrupts and bottom half handlers won't change it
17931798 * from under us. It essentially blocks any incoming
@@ -1862,9 +1867,13 @@ void sock_pfree(struct sk_buff *skb); · 68 unmodified lines
18621867 #define sock_edemux sock_efree
18631868 #endif
18641869
1870+int sk_setsockopt(struct sock *sk, int level, int optname,
1871+ sockptr_t optval, unsigned int optlen);
18651872 int sock_setsockopt(struct socket *sock, int level, int op,
18661873 sockptr_t optval, unsigned int optlen);
18671874
1875+int sk_getsockopt(struct sock *sk, int level, int optname,
1876+ sockptr_t optval, sockptr_t optlen);
18681877 int sock_getsockopt(struct socket *sock, int level, int op,
18691878 char __user *optval, int __user *optlen);
18701879 int sock_gettstamp(struct socket *sock, void __user *userstamp,
include/net/tcp.h+4−0
@@ -402,9 +402,13 @@ void tcp_init_sock(struct sock *sk); · 401 unmodified lines
402402 void tcp_init_transfer(struct sock *sk, int bpf_op, struct sk_buff *skb);
403403 __poll_t tcp_poll(struct file *file, struct socket *sock,
404404 struct poll_table_struct *wait);
405+int do_tcp_getsockopt(struct sock *sk, int level,
406+ int optname, sockptr_t optval, sockptr_t optlen);
405407 int tcp_getsockopt(struct sock *sk, int level, int optname,
406408 char __user *optval, int __user *optlen);
407409 bool tcp_bpf_bypass_getsockopt(int level, int optname);
410+int do_tcp_setsockopt(struct sock *sk, int level, int optname,
411+ sockptr_t optval, unsigned int optlen);
408412 int tcp_setsockopt(struct sock *sk, int level, int optname, sockptr_t optval,
409413 unsigned int optlen);
410414 void tcp_set_keepalive(struct sock *sk, int val);
include/uapi/linux/bpf.h+69−14
@@ -87,10 +87,29 @@ struct bpf_cgroup_storage_key { · 86 unmodified lines
8787 __u32 attach_type; /* program attach type (enum bpf_attach_type) */
8888 };
8989
90+enum bpf_cgroup_iter_order {
91+ BPF_CGROUP_ITER_ORDER_UNSPEC = 0,
92+ BPF_CGROUP_ITER_SELF_ONLY, /* process only a single object. */
93+ BPF_CGROUP_ITER_DESCENDANTS_PRE, /* walk descendants in pre-order. */
94+ BPF_CGROUP_ITER_DESCENDANTS_POST, /* walk descendants in post-order. */
95+ BPF_CGROUP_ITER_ANCESTORS_UP, /* walk ancestors upward. */
96+};
97+
9098 union bpf_iter_link_info {
9199 struct {
92100 __u32 map_fd;
93101 } map;
102+ struct {
103+ enum bpf_cgroup_iter_order order;
104+
105+ /* At most one of cgroup_fd and cgroup_id can be non-zero. If
106+ * both are zero, the walk starts from the default cgroup v2
107+ * root. For walking v1 hierarchy, one should always explicitly
108+ * specify cgroup_fd.
109+ */
110+ __u32 cgroup_fd;
111+ __u64 cgroup_id;
112+ } cgroup;
94113 };
95114
96115 /* BPF syscall commands, see bpf(2) man-page for more details. */
@@ -4437,7 +4456,7 @@ union bpf_attr { · 4340 unmodified lines
44374456 *
44384457 * **-EEXIST** if the option already exists.
44394458 *
4440− * **-EFAULT** on failrue to parse the existing header options.
4459+ * **-EFAULT** on failure to parse the existing header options.
44414460 *
44424461 * **-EPERM** if the helper cannot be used under the current
44434462 * *skops*\ **->op**.
@@ -4646,7 +4665,7 @@ union bpf_attr { · 202 unmodified lines
46464665 * a *map* with *task* as the **key**. From this
46474666 * perspective, the usage is not much different from
46484667 * **bpf_map_lookup_elem**\ (*map*, **&**\ *task*) except this
4649− * helper enforces the key must be an task_struct and the map must also
4668+ * helper enforces the key must be a task_struct and the map must also
46504669 * be a **BPF_MAP_TYPE_TASK_STORAGE**.
46514670 *
46524671 * Underneath, the value is stored locally at *task* instead of
@@ -4704,7 +4723,7 @@ union bpf_attr { · 51 unmodified lines
47044723 *
47054724 * long bpf_ima_inode_hash(struct inode *inode, void *dst, u32 size)
47064725 * Description
4707− * Returns the stored IMA hash of the *inode* (if it's avaialable).
4726+ * Returns the stored IMA hash of the *inode* (if it's available).
47084727 * If the hash is larger than *size*, then only *size*
47094728 * bytes will be copied to *dst*
47104729 * Return
@@ -4728,12 +4747,12 @@ union bpf_attr { · 17 unmodified lines
47284747 *
47294748 * The argument *len_diff* can be used for querying with a planned
47304749 * size change. This allows to check MTU prior to changing packet
4731− * ctx. Providing an *len_diff* adjustment that is larger than the
4750+ * ctx. Providing a *len_diff* adjustment that is larger than the
47324751 * actual packet size (resulting in negative packet size) will in
4733− * principle not exceed the MTU, why it is not considered a
4734− * failure. Other BPF-helpers are needed for performing the
4735− * planned size change, why the responsability for catch a negative
4736− * packet size belong in those helpers.
4752+ * principle not exceed the MTU, which is why it is not considered
4753+ * a failure. Other BPF helpers are needed for performing the
4754+ * planned size change; therefore the responsibility for catching
4755+ * a negative packet size belongs in those helpers.
47374756 *
47384757 * Specifying *ifindex* zero means the MTU check is performed
47394758 * against the current net device. This is practical if this isn't
@@ -5085,17 +5104,29 @@ union bpf_attr { · 345 unmodified lines
50855104 *
50865105 * int bpf_get_retval(void)
50875106 * Description
5088− * Get the syscall's return value that will be returned to userspace.
5107+ * Get the BPF program's return value that will be returned to the upper layers.
50895108 *
5090− * This helper is currently supported by cgroup programs only.
5109+ * This helper is currently supported by cgroup programs and only by the hooks
5110+ * where BPF program's return value is returned to the userspace via errno.
50915111 * Return
5092− * The syscall's return value.
5112+ * The BPF program's return value.
50935113 *
50945114 * int bpf_set_retval(int retval)
50955115 * Description
5096− * Set the syscall's return value that will be returned to userspace.
5116+ * Set the BPF program's return value that will be returned to the upper layers.
5117+ *
5118+ * This helper is currently supported by cgroup programs and only by the hooks
5119+ * where BPF program's return value is returned to the userspace via errno.
5120+ *
5121+ * Note that there is the following corner case where the program exports an error
5122+ * via bpf_set_retval but signals success via 'return 1':
5123+ *
5124+ * bpf_set_retval(-EPERM);
5125+ * return 1;
5126+ *
5127+ * In this case, the BPF program's return value will use helper's -EPERM. This
5128+ * still holds true for cgroup/bind{4,6} which supports extra 'return 3' success case.
50975129 *
5098− * This helper is currently supported by cgroup programs only.
50995130 * Return
51005131 * 0 on success, or a negative error in case of failure.
51015132 *
@@ -5628,6 +5659,11 @@ enum { · 526 unmodified lines
56285659 BPF_F_SEQ_NUMBER = (1ULL << 3),
56295660 };
56305661
5662+/* BPF_FUNC_skb_get_tunnel_key flags. */
5663+enum {
5664+ BPF_F_TUNINFO_FLAGS = (1ULL << 4),
5665+};
5666+
56315667 /* BPF_FUNC_perf_event_output, BPF_FUNC_perf_event_read and
56325668 * BPF_FUNC_perf_event_read_value flags.
56335669 */
@@ -5817,7 +5853,10 @@ struct bpf_tunnel_key { · 183 unmodified lines
58175853 };
58185854 __u8 tunnel_tos;
58195855 __u8 tunnel_ttl;
5820− __u16 tunnel_ext; /* Padding, future use. */
5856+ union {
5857+ __u16 tunnel_ext; /* compat */
5858+ __be16 tunnel_flags;
5859+ };
58215860 __u32 tunnel_label;
58225861 union {
58235862 __u32 local_ipv4;
@@ -5861,6 +5900,11 @@ enum bpf_ret_code { · 37 unmodified lines
58615900 * represented by BPF_REDIRECT above).
58625901 */
58635902 BPF_LWT_REROUTE = 128,
5903+ /* BPF_FLOW_DISSECTOR_CONTINUE: used by BPF_PROG_TYPE_FLOW_DISSECTOR
5904+ * to indicate that no custom dissection was performed, and
5905+ * fallback to standard dissector is requested.
5906+ */
5907+ BPF_FLOW_DISSECTOR_CONTINUE = 129,
58645908 };
58655909
58665910 struct bpf_sock {
@@ -6159,11 +6203,22 @@ struct bpf_link_info { · 292 unmodified lines
61596203 struct {
61606204 __aligned_u64 target_name; /* in/out: target_name buffer ptr */
61616205 __u32 target_name_len; /* in/out: target_name buffer len */
6206+
6207+ /* If the iter specific field is 32 bits, it can be put
6208+ * in the first or second union. Otherwise it should be
6209+ * put in the second union.
6210+ */
61626211 union {
61636212 struct {
61646213 __u32 map_id;
61656214 } map;
61666215 };
6216+ union {
6217+ struct {
6218+ __u64 cgroup_id;
6219+ __u32 order;
6220+ } cgroup;
6221+ };
61676222 } iter;
61686223 struct {
61696224 __u32 netns_ino;
kernel/bpf/Makefile+4−1
@@ -13,7 +13,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o · 12 unmodified lines
1313 obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
1414 obj-$(CONFIG_BPF_SYSCALL) += disasm.o
1515 obj-$(CONFIG_BPF_JIT) += trampoline.o
16−obj-$(CONFIG_BPF_SYSCALL) += btf.o
16+obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o
1717 obj-$(CONFIG_BPF_JIT) += dispatcher.o
1818 ifeq ($(CONFIG_NET),y)
1919 obj-$(CONFIG_BPF_SYSCALL) += devmap.o
@@ -24,6 +24,9 @@ endif · 4 unmodified lines
2424 ifeq ($(CONFIG_PERF_EVENTS),y)
2525 obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
2626 endif
27+ifeq ($(CONFIG_CGROUPS),y)
28+obj-$(CONFIG_BPF_SYSCALL) += cgroup_iter.o
29+endif
2730 obj-$(CONFIG_CGROUP_BPF) += cgroup.o
2831 ifeq ($(CONFIG_INET),y)
2932 obj-$(CONFIG_BPF_SYSCALL) += reuseport_array.o
kernel/bpf/bpf_iter.c+5−0
@@ -694,19 +694,24 @@ struct bpf_prog *bpf_iter_get_info(struct bpf_iter_meta *meta, bool in_stop) · 693 unmodified lines
694694
695695 int bpf_iter_run_prog(struct bpf_prog *prog, void *ctx)
696696 {
697+ struct bpf_run_ctx run_ctx, *old_run_ctx;
697698 int ret;
698699
699700 if (prog->aux->sleepable) {
700701 rcu_read_lock_trace();
701702 migrate_disable();
702703 might_fault();
704+ old_run_ctx = bpf_set_run_ctx(&run_ctx);
703705 ret = bpf_prog_run(prog, ctx);
706+ bpf_reset_run_ctx(old_run_ctx);
704707 migrate_enable();
705708 rcu_read_unlock_trace();
706709 } else {
707710 rcu_read_lock();
708711 migrate_disable();
712+ old_run_ctx = bpf_set_run_ctx(&run_ctx);
709713 ret = bpf_prog_run(prog, ctx);
714+ bpf_reset_run_ctx(old_run_ctx);
710715 migrate_enable();
711716 rcu_read_unlock();
712717 }
kernel/bpf/bpf_local_storage.c+2−2
@@ -555,11 +555,11 @@ void bpf_local_storage_map_free(struct bpf_local_storage_map *smap, · 554 unmodified lines
555555 struct bpf_local_storage_elem, map_node))) {
556556 if (busy_counter) {
557557 migrate_disable();
558− __this_cpu_inc(*busy_counter);
558+ this_cpu_inc(*busy_counter);
559559 }
560560 bpf_selem_unlink(selem, false);
561561 if (busy_counter) {
562− __this_cpu_dec(*busy_counter);
562+ this_cpu_dec(*busy_counter);
563563 migrate_enable();
564564 }
565565 cond_resched_rcu();
kernel/bpf/bpf_lsm.c+14−9
@@ -41,17 +41,21 @@ BTF_SET_END(bpf_lsm_hooks) · 40 unmodified lines
4141 */
4242 BTF_SET_START(bpf_lsm_current_hooks)
4343 /* operate on freshly allocated sk without any cgroup association */
44+#ifdef CONFIG_SECURITY_NETWORK
4445 BTF_ID(func, bpf_lsm_sk_alloc_security)
4546 BTF_ID(func, bpf_lsm_sk_free_security)
47+#endif
4648 BTF_SET_END(bpf_lsm_current_hooks)
4749
4850 /* List of LSM hooks that trigger while the socket is properly locked.
4951 */
5052 BTF_SET_START(bpf_lsm_locked_sockopt_hooks)
53+#ifdef CONFIG_SECURITY_NETWORK
5154 BTF_ID(func, bpf_lsm_socket_sock_rcv_skb)
5255 BTF_ID(func, bpf_lsm_sock_graft)
5356 BTF_ID(func, bpf_lsm_inet_csk_clone)
5457 BTF_ID(func, bpf_lsm_inet_conn_established)
58+#endif
5559 BTF_SET_END(bpf_lsm_locked_sockopt_hooks)
5660
5761 /* List of LSM hooks that trigger while the socket is _not_ locked,
@@ -59,8 +63,10 @@ BTF_SET_END(bpf_lsm_locked_sockopt_hooks) · 1 unmodified line
5963 * in the early init phase.
6064 */
6165 BTF_SET_START(bpf_lsm_unlocked_sockopt_hooks)
66+#ifdef CONFIG_SECURITY_NETWORK
6267 BTF_ID(func, bpf_lsm_socket_post_create)
6368 BTF_ID(func, bpf_lsm_socket_socketpair)
69+#endif
6470 BTF_SET_END(bpf_lsm_unlocked_sockopt_hooks)
6571
6672 #ifdef CONFIG_CGROUP_BPF
@@ -189,6 +195,14 @@ static const struct bpf_func_proto bpf_get_attach_cookie_proto = { · 122 unmodified lines
189195 static const struct bpf_func_proto *
190196 bpf_lsm_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
191197 {
198+ const struct bpf_func_proto *func_proto;
199+
200+ if (prog->expected_attach_type == BPF_LSM_CGROUP) {
201+ func_proto = cgroup_common_func_proto(func_id, prog);
202+ if (func_proto)
203+ return func_proto;
204+ }
205+
192206 switch (func_id) {
193207 case BPF_FUNC_inode_storage_get:
194208 return &bpf_inode_storage_get_proto;
@@ -212,15 +226,6 @@ bpf_lsm_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 17 unmodified lines
212226 return prog->aux->sleepable ? &bpf_ima_file_hash_proto : NULL;
213227 case BPF_FUNC_get_attach_cookie:
214228 return bpf_prog_has_trampoline(prog) ? &bpf_get_attach_cookie_proto : NULL;
215− case BPF_FUNC_get_local_storage:
216− return prog->expected_attach_type == BPF_LSM_CGROUP ?
217− &bpf_get_local_storage_proto : NULL;
218− case BPF_FUNC_set_retval:
219− return prog->expected_attach_type == BPF_LSM_CGROUP ?
220− &bpf_set_retval_proto : NULL;
221− case BPF_FUNC_get_retval:
222− return prog->expected_attach_type == BPF_LSM_CGROUP ?
223− &bpf_get_retval_proto : NULL;
224229 #ifdef CONFIG_NET
225230 case BPF_FUNC_setsockopt:
226231 if (prog->expected_attach_type != BPF_LSM_CGROUP)
kernel/bpf/bpf_task_storage.c+4−4
@@ -26,20 +26,20 @@ static DEFINE_PER_CPU(int, bpf_task_storage_busy); · 25 unmodified lines
2626 static void bpf_task_storage_lock(void)
2727 {
2828 migrate_disable();
29− __this_cpu_inc(bpf_task_storage_busy);
29+ this_cpu_inc(bpf_task_storage_busy);
3030 }
3131
3232 static void bpf_task_storage_unlock(void)
3333 {
34− __this_cpu_dec(bpf_task_storage_busy);
34+ this_cpu_dec(bpf_task_storage_busy);
3535 migrate_enable();
3636 }
3737
3838 static bool bpf_task_storage_trylock(void)
3939 {
4040 migrate_disable();
41− if (unlikely(__this_cpu_inc_return(bpf_task_storage_busy) != 1)) {
42− __this_cpu_dec(bpf_task_storage_busy);
41+ if (unlikely(this_cpu_inc_return(bpf_task_storage_busy) != 1)) {
42+ this_cpu_dec(bpf_task_storage_busy);
4343 migrate_enable();
4444 return false;
4545 }
kernel/bpf/cgroup.c+134−23
@@ -1529,6 +1529,37 @@ int __cgroup_bpf_check_dev_permission(short dev_type, u32 major, u32 minor, · 1528 unmodified lines
15291529 return ret;
15301530 }
15311531
1532+BPF_CALL_2(bpf_get_local_storage, struct bpf_map *, map, u64, flags)
1533+{
1534+ /* flags argument is not used now,
1535+ * but provides an ability to extend the API.
1536+ * verifier checks that its value is correct.
1537+ */
1538+ enum bpf_cgroup_storage_type stype = cgroup_storage_type(map);
1539+ struct bpf_cgroup_storage *storage;
1540+ struct bpf_cg_run_ctx *ctx;
1541+ void *ptr;
1542+
1543+ /* get current cgroup storage from BPF run context */
1544+ ctx = container_of(current->bpf_ctx, struct bpf_cg_run_ctx, run_ctx);
1545+ storage = ctx->prog_item->cgroup_storage[stype];
1546+
1547+ if (stype == BPF_CGROUP_STORAGE_SHARED)
1548+ ptr = &READ_ONCE(storage->buf)->data[0];
1549+ else
1550+ ptr = this_cpu_ptr(storage->percpu_buf);
1551+
1552+ return (unsigned long)ptr;
1553+}
1554+
1555+const struct bpf_func_proto bpf_get_local_storage_proto = {
1556+ .func = bpf_get_local_storage,
1557+ .gpl_only = false,
1558+ .ret_type = RET_PTR_TO_MAP_VALUE,
1559+ .arg1_type = ARG_CONST_MAP_PTR,
1560+ .arg2_type = ARG_ANYTHING,
1561+};
1562+
15321563 BPF_CALL_0(bpf_get_retval)
15331564 {
15341565 struct bpf_cg_run_ctx *ctx =
@@ -1560,32 +1591,26 @@ const struct bpf_func_proto bpf_set_retval_proto = { · 25 unmodified lines
15601591 };
15611592
15621593 static const struct bpf_func_proto *
1563−cgroup_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
1594+cgroup_dev_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
15641595 {
1596+ const struct bpf_func_proto *func_proto;
1597+
1598+ func_proto = cgroup_common_func_proto(func_id, prog);
1599+ if (func_proto)
1600+ return func_proto;
1601+
1602+ func_proto = cgroup_current_func_proto(func_id, prog);
1603+ if (func_proto)
1604+ return func_proto;
1605+
15651606 switch (func_id) {
1566− case BPF_FUNC_get_current_uid_gid:
1567− return &bpf_get_current_uid_gid_proto;
1568− case BPF_FUNC_get_local_storage:
1569− return &bpf_get_local_storage_proto;
1570− case BPF_FUNC_get_current_cgroup_id:
1571− return &bpf_get_current_cgroup_id_proto;
15721607 case BPF_FUNC_perf_event_output:
15731608 return &bpf_event_output_data_proto;
1574− case BPF_FUNC_get_retval:
1575− return &bpf_get_retval_proto;
1576− case BPF_FUNC_set_retval:
1577− return &bpf_set_retval_proto;
15781609 default:
15791610 return bpf_base_func_proto(func_id);
15801611 }
15811612 }
15821613
1583−static const struct bpf_func_proto *
1584−cgroup_dev_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
1585−{
1586− return cgroup_base_func_proto(func_id, prog);
1587−}
1588−
15891614 static bool cgroup_dev_is_valid_access(int off, int size,
15901615 enum bpf_access_type type,
15911616 const struct bpf_prog *prog,
@@ -2098,11 +2123,17 @@ static const struct bpf_func_proto bpf_sysctl_set_new_value_proto = { · 506 unmodified lines
20982123 static const struct bpf_func_proto *
20992124 sysctl_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
21002125 {
2126+ const struct bpf_func_proto *func_proto;
2127+
2128+ func_proto = cgroup_common_func_proto(func_id, prog);
2129+ if (func_proto)
2130+ return func_proto;
2131+
2132+ func_proto = cgroup_current_func_proto(func_id, prog);
2133+ if (func_proto)
2134+ return func_proto;
2135+
21012136 switch (func_id) {
2102− case BPF_FUNC_strtol:
2103− return &bpf_strtol_proto;
2104− case BPF_FUNC_strtoul:
2105− return &bpf_strtoul_proto;
21062137 case BPF_FUNC_sysctl_get_name:
21072138 return &bpf_sysctl_get_name_proto;
21082139 case BPF_FUNC_sysctl_get_current_value:
@@ -2113,8 +2144,10 @@ sysctl_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 4 unmodified lines
21132144 return &bpf_sysctl_set_new_value_proto;
21142145 case BPF_FUNC_ktime_get_coarse_ns:
21152146 return &bpf_ktime_get_coarse_ns_proto;
2147+ case BPF_FUNC_perf_event_output:
2148+ return &bpf_event_output_data_proto;
21162149 default:
2117− return cgroup_base_func_proto(func_id, prog);
2150+ return bpf_base_func_proto(func_id);
21182151 }
21192152 }
21202153
@@ -2235,6 +2268,16 @@ static const struct bpf_func_proto bpf_get_netns_cookie_sockopt_proto = { · 114 unmodified lines
22352268 static const struct bpf_func_proto *
22362269 cg_sockopt_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
22372270 {
2271+ const struct bpf_func_proto *func_proto;
2272+
2273+ func_proto = cgroup_common_func_proto(func_id, prog);
2274+ if (func_proto)
2275+ return func_proto;
2276+
2277+ func_proto = cgroup_current_func_proto(func_id, prog);
2278+ if (func_proto)
2279+ return func_proto;
2280+
22382281 switch (func_id) {
22392282 #ifdef CONFIG_NET
22402283 case BPF_FUNC_get_netns_cookie:
@@ -2256,8 +2299,10 @@ cg_sockopt_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 15 unmodified lines
22562299 case BPF_FUNC_tcp_sock:
22572300 return &bpf_tcp_sock_proto;
22582301 #endif
2302+ case BPF_FUNC_perf_event_output:
2303+ return &bpf_event_output_data_proto;
22592304 default:
2260− return cgroup_base_func_proto(func_id, prog);
2305+ return bpf_base_func_proto(func_id);
22612306 }
22622307 }
22632308
@@ -2422,3 +2467,69 @@ const struct bpf_verifier_ops cg_sockopt_verifier_ops = { · 158 unmodified lines
24222467
24232468 const struct bpf_prog_ops cg_sockopt_prog_ops = {
24242469 };
2470+
2471+/* Common helpers for cgroup hooks. */
2472+const struct bpf_func_proto *
2473+cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
2474+{
2475+ switch (func_id) {
2476+ case BPF_FUNC_get_local_storage:
2477+ return &bpf_get_local_storage_proto;
2478+ case BPF_FUNC_get_retval:
2479+ switch (prog->expected_attach_type) {
2480+ case BPF_CGROUP_INET_INGRESS:
2481+ case BPF_CGROUP_INET_EGRESS:
2482+ case BPF_CGROUP_SOCK_OPS:
2483+ case BPF_CGROUP_UDP4_RECVMSG:
2484+ case BPF_CGROUP_UDP6_RECVMSG:
2485+ case BPF_CGROUP_INET4_GETPEERNAME:
2486+ case BPF_CGROUP_INET6_GETPEERNAME:
2487+ case BPF_CGROUP_INET4_GETSOCKNAME:
2488+ case BPF_CGROUP_INET6_GETSOCKNAME:
2489+ return NULL;
2490+ default:
2491+ return &bpf_get_retval_proto;
2492+ }
2493+ case BPF_FUNC_set_retval:
2494+ switch (prog->expected_attach_type) {
2495+ case BPF_CGROUP_INET_INGRESS:
2496+ case BPF_CGROUP_INET_EGRESS:
2497+ case BPF_CGROUP_SOCK_OPS:
2498+ case BPF_CGROUP_UDP4_RECVMSG:
2499+ case BPF_CGROUP_UDP6_RECVMSG:
2500+ case BPF_CGROUP_INET4_GETPEERNAME:
2501+ case BPF_CGROUP_INET6_GETPEERNAME:
2502+ case BPF_CGROUP_INET4_GETSOCKNAME:
2503+ case BPF_CGROUP_INET6_GETSOCKNAME:
2504+ return NULL;
2505+ default:
2506+ return &bpf_set_retval_proto;
2507+ }
2508+ default:
2509+ return NULL;
2510+ }
2511+}
2512+
2513+/* Common helpers for cgroup hooks with valid process context. */
2514+const struct bpf_func_proto *
2515+cgroup_current_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
2516+{
2517+ switch (func_id) {
2518+ case BPF_FUNC_get_current_uid_gid:
2519+ return &bpf_get_current_uid_gid_proto;
2520+ case BPF_FUNC_get_current_pid_tgid:
2521+ return &bpf_get_current_pid_tgid_proto;
2522+ case BPF_FUNC_get_current_comm:
2523+ return &bpf_get_current_comm_proto;
2524+ case BPF_FUNC_get_current_cgroup_id:
2525+ return &bpf_get_current_cgroup_id_proto;
2526+ case BPF_FUNC_get_current_ancestor_cgroup_id:
2527+ return &bpf_get_current_ancestor_cgroup_id_proto;
2528+#ifdef CONFIG_CGROUP_NET_CLASSID
2529+ case BPF_FUNC_get_cgroup_classid:
2530+ return &bpf_get_cgroup_classid_curr_proto;
2531+#endif
2532+ default:
2533+ return NULL;
2534+ }
2535+}
kernel/bpf/cgroup_iter.c+282−0
@@ -0,0 +1,282 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/* Copyright (c) 2022 Google */
3+#include <linux/bpf.h>
4+#include <linux/btf_ids.h>
5+#include <linux/cgroup.h>
6+#include <linux/kernel.h>
7+#include <linux/seq_file.h>
8+
9+#include "../cgroup/cgroup-internal.h" /* cgroup_mutex and cgroup_is_dead */
10+
11+/* cgroup_iter provides four modes of traversal to the cgroup hierarchy.
12+ *
13+ * 1. Walk the descendants of a cgroup in pre-order.
14+ * 2. Walk the descendants of a cgroup in post-order.
15+ * 3. Walk the ancestors of a cgroup.
16+ * 4. Show the given cgroup only.
17+ *
18+ * For walking descendants, cgroup_iter can walk in either pre-order or
19+ * post-order. For walking ancestors, the iter walks up from a cgroup to
20+ * the root.
21+ *
22+ * The iter program can terminate the walk early by returning 1. Walk
23+ * continues if prog returns 0.
24+ *
25+ * The prog can check (seq->num == 0) to determine whether this is
26+ * the first element. The prog may also be passed a NULL cgroup,
27+ * which means the walk has completed and the prog has a chance to
28+ * do post-processing, such as outputting an epilogue.
29+ *
30+ * Note: the iter_prog is called with cgroup_mutex held.
31+ *
32+ * Currently only one session is supported, which means, depending on the
33+ * volume of data bpf program intends to send to user space, the number
34+ * of cgroups that can be walked is limited. For example, given the current
35+ * buffer size is 8 * PAGE_SIZE, if the program sends 64B data for each
36+ * cgroup, assuming PAGE_SIZE is 4kb, the total number of cgroups that can
37+ * be walked is 512. This is a limitation of cgroup_iter. If the output data
38+ * is larger than the kernel buffer size, after all data in the kernel buffer
39+ * is consumed by user space, the subsequent read() syscall will signal
40+ * EOPNOTSUPP. In order to work around, the user may have to update their
41+ * program to reduce the volume of data sent to output. For example, skip
42+ * some uninteresting cgroups.
43+ */
44+
45+struct bpf_iter__cgroup {
46+ __bpf_md_ptr(struct bpf_iter_meta *, meta);
47+ __bpf_md_ptr(struct cgroup *, cgroup);
48+};
49+
50+struct cgroup_iter_priv {
51+ struct cgroup_subsys_state *start_css;
52+ bool visited_all;
53+ bool terminate;
54+ int order;
55+};
56+
57+static void *cgroup_iter_seq_start(struct seq_file *seq, loff_t *pos)
58+{
59+ struct cgroup_iter_priv *p = seq->private;
60+
61+ mutex_lock(&cgroup_mutex);
62+
63+ /* cgroup_iter doesn't support read across multiple sessions. */
64+ if (*pos > 0) {
65+ if (p->visited_all)
66+ return NULL;
67+
68+ /* Haven't visited all, but because cgroup_mutex has dropped,
69+ * return -EOPNOTSUPP to indicate incomplete iteration.
70+ */
71+ return ERR_PTR(-EOPNOTSUPP);
72+ }
73+
74+ ++*pos;
75+ p->terminate = false;
76+ p->visited_all = false;
77+ if (p->order == BPF_CGROUP_ITER_DESCENDANTS_PRE)
78+ return css_next_descendant_pre(NULL, p->start_css);
79+ else if (p->order == BPF_CGROUP_ITER_DESCENDANTS_POST)
80+ return css_next_descendant_post(NULL, p->start_css);
81+ else /* BPF_CGROUP_ITER_SELF_ONLY and BPF_CGROUP_ITER_ANCESTORS_UP */
82+ return p->start_css;
83+}
84+
85+static int __cgroup_iter_seq_show(struct seq_file *seq,
86+ struct cgroup_subsys_state *css, int in_stop);
87+
88+static void cgroup_iter_seq_stop(struct seq_file *seq, void *v)
89+{
90+ struct cgroup_iter_priv *p = seq->private;
91+
92+ mutex_unlock(&cgroup_mutex);
93+
94+ /* pass NULL to the prog for post-processing */
95+ if (!v) {
96+ __cgroup_iter_seq_show(seq, NULL, true);
97+ p->visited_all = true;
98+ }
99+}
100+
101+static void *cgroup_iter_seq_next(struct seq_file *seq, void *v, loff_t *pos)
102+{
103+ struct cgroup_subsys_state *curr = (struct cgroup_subsys_state *)v;
104+ struct cgroup_iter_priv *p = seq->private;
105+
106+ ++*pos;
107+ if (p->terminate)
108+ return NULL;
109+
110+ if (p->order == BPF_CGROUP_ITER_DESCENDANTS_PRE)
111+ return css_next_descendant_pre(curr, p->start_css);
112+ else if (p->order == BPF_CGROUP_ITER_DESCENDANTS_POST)
113+ return css_next_descendant_post(curr, p->start_css);
114+ else if (p->order == BPF_CGROUP_ITER_ANCESTORS_UP)
115+ return curr->parent;
116+ else /* BPF_CGROUP_ITER_SELF_ONLY */
117+ return NULL;
118+}
119+
120+static int __cgroup_iter_seq_show(struct seq_file *seq,
121+ struct cgroup_subsys_state *css, int in_stop)
122+{
123+ struct cgroup_iter_priv *p = seq->private;
124+ struct bpf_iter__cgroup ctx;
125+ struct bpf_iter_meta meta;
126+ struct bpf_prog *prog;
127+ int ret = 0;
128+
129+ /* cgroup is dead, skip this element */
130+ if (css && cgroup_is_dead(css->cgroup))
131+ return 0;
132+
133+ ctx.meta = &meta;
134+ ctx.cgroup = css ? css->cgroup : NULL;
135+ meta.seq = seq;
136+ prog = bpf_iter_get_info(&meta, in_stop);
137+ if (prog)
138+ ret = bpf_iter_run_prog(prog, &ctx);
139+
140+ /* if prog returns > 0, terminate after this element. */
141+ if (ret != 0)
142+ p->terminate = true;
143+
144+ return 0;
145+}
146+
147+static int cgroup_iter_seq_show(struct seq_file *seq, void *v)
148+{
149+ return __cgroup_iter_seq_show(seq, (struct cgroup_subsys_state *)v,
150+ false);
151+}
152+
153+static const struct seq_operations cgroup_iter_seq_ops = {
154+ .start = cgroup_iter_seq_start,
155+ .next = cgroup_iter_seq_next,
156+ .stop = cgroup_iter_seq_stop,
157+ .show = cgroup_iter_seq_show,
158+};
159+
160+BTF_ID_LIST_SINGLE(bpf_cgroup_btf_id, struct, cgroup)
161+
162+static int cgroup_iter_seq_init(void *priv, struct bpf_iter_aux_info *aux)
163+{
164+ struct cgroup_iter_priv *p = (struct cgroup_iter_priv *)priv;
165+ struct cgroup *cgrp = aux->cgroup.start;
166+
167+ p->start_css = &cgrp->self;
168+ p->terminate = false;
169+ p->visited_all = false;
170+ p->order = aux->cgroup.order;
171+ return 0;
172+}
173+
174+static const struct bpf_iter_seq_info cgroup_iter_seq_info = {
175+ .seq_ops = &cgroup_iter_seq_ops,
176+ .init_seq_private = cgroup_iter_seq_init,
177+ .seq_priv_size = sizeof(struct cgroup_iter_priv),
178+};
179+
180+static int bpf_iter_attach_cgroup(struct bpf_prog *prog,
181+ union bpf_iter_link_info *linfo,
182+ struct bpf_iter_aux_info *aux)
183+{
184+ int fd = linfo->cgroup.cgroup_fd;
185+ u64 id = linfo->cgroup.cgroup_id;
186+ int order = linfo->cgroup.order;
187+ struct cgroup *cgrp;
188+
189+ if (order != BPF_CGROUP_ITER_DESCENDANTS_PRE &&
190+ order != BPF_CGROUP_ITER_DESCENDANTS_POST &&
191+ order != BPF_CGROUP_ITER_ANCESTORS_UP &&
192+ order != BPF_CGROUP_ITER_SELF_ONLY)
193+ return -EINVAL;
194+
195+ if (fd && id)
196+ return -EINVAL;
197+
198+ if (fd)
199+ cgrp = cgroup_get_from_fd(fd);
200+ else if (id)
201+ cgrp = cgroup_get_from_id(id);
202+ else /* walk the entire hierarchy by default. */
203+ cgrp = cgroup_get_from_path("/");
204+
205+ if (IS_ERR(cgrp))
206+ return PTR_ERR(cgrp);
207+
208+ aux->cgroup.start = cgrp;
209+ aux->cgroup.order = order;
210+ return 0;
211+}
212+
213+static void bpf_iter_detach_cgroup(struct bpf_iter_aux_info *aux)
214+{
215+ cgroup_put(aux->cgroup.start);
216+}
217+
218+static void bpf_iter_cgroup_show_fdinfo(const struct bpf_iter_aux_info *aux,
219+ struct seq_file *seq)
220+{
221+ char *buf;
222+
223+ buf = kzalloc(PATH_MAX, GFP_KERNEL);
224+ if (!buf) {
225+ seq_puts(seq, "cgroup_path:\t<unknown>\n");
226+ goto show_order;
227+ }
228+
229+ /* If cgroup_path_ns() fails, buf will be an empty string, cgroup_path
230+ * will print nothing.
231+ *
232+ * Path is in the calling process's cgroup namespace.
233+ */
234+ cgroup_path_ns(aux->cgroup.start, buf, PATH_MAX,
235+ current->nsproxy->cgroup_ns);
236+ seq_printf(seq, "cgroup_path:\t%s\n", buf);
237+ kfree(buf);
238+
239+show_order:
240+ if (aux->cgroup.order == BPF_CGROUP_ITER_DESCENDANTS_PRE)
241+ seq_puts(seq, "order: descendants_pre\n");
242+ else if (aux->cgroup.order == BPF_CGROUP_ITER_DESCENDANTS_POST)
243+ seq_puts(seq, "order: descendants_post\n");
244+ else if (aux->cgroup.order == BPF_CGROUP_ITER_ANCESTORS_UP)
245+ seq_puts(seq, "order: ancestors_up\n");
246+ else /* BPF_CGROUP_ITER_SELF_ONLY */
247+ seq_puts(seq, "order: self_only\n");
248+}
249+
250+static int bpf_iter_cgroup_fill_link_info(const struct bpf_iter_aux_info *aux,
251+ struct bpf_link_info *info)
252+{
253+ info->iter.cgroup.order = aux->cgroup.order;
254+ info->iter.cgroup.cgroup_id = cgroup_id(aux->cgroup.start);
255+ return 0;
256+}
257+
258+DEFINE_BPF_ITER_FUNC(cgroup, struct bpf_iter_meta *meta,
259+ struct cgroup *cgroup)
260+
261+static struct bpf_iter_reg bpf_cgroup_reg_info = {
262+ .target = "cgroup",
263+ .feature = BPF_ITER_RESCHED,
264+ .attach_target = bpf_iter_attach_cgroup,
265+ .detach_target = bpf_iter_detach_cgroup,
266+ .show_fdinfo = bpf_iter_cgroup_show_fdinfo,
267+ .fill_link_info = bpf_iter_cgroup_fill_link_info,
268+ .ctx_arg_info_size = 1,
269+ .ctx_arg_info = {
270+ { offsetof(struct bpf_iter__cgroup, cgroup),
271+ PTR_TO_BTF_ID_OR_NULL },
272+ },
273+ .seq_info = &cgroup_iter_seq_info,
274+};
275+
276+static int __init bpf_cgroup_iter_init(void)
277+{
278+ bpf_cgroup_reg_info.ctx_arg_info[0].btf_id = bpf_cgroup_btf_id[0];
279+ return bpf_iter_reg_target(&bpf_cgroup_reg_info);
280+}
281+
282+late_initcall(bpf_cgroup_iter_init);
kernel/bpf/hashtab.c+123−45
@@ -14,6 +14,7 @@ · 13 unmodified lines
1414 #include "percpu_freelist.h"
1515 #include "bpf_lru_list.h"
1616 #include "map_in_map.h"
17+#include <linux/bpf_mem_alloc.h>
1718
1819 #define HTAB_CREATE_FLAG_MASK \
1920 (BPF_F_NO_PREALLOC | BPF_F_NO_COMMON_LRU | BPF_F_NUMA_NODE | \
@@ -92,6 +93,8 @@ struct bucket { · 72 unmodified lines
9293
9394 struct bpf_htab {
9495 struct bpf_map map;
96+ struct bpf_mem_alloc ma;
97+ struct bpf_mem_alloc pcpu_ma;
9598 struct bucket *buckets;
9699 void *elems;
97100 union {
@@ -99,7 +102,12 @@ struct bpf_htab { · 1 unmodified line
99102 struct bpf_lru lru;
100103 };
101104 struct htab_elem *__percpu *extra_elems;
102− atomic_t count; /* number of elements in this hashtable */
105+ /* number of elements in non-preallocated hashtable are kept
106+ * in either pcount or count
107+ */
108+ struct percpu_counter pcount;
109+ atomic_t count;
110+ bool use_percpu_counter;
103111 u32 n_buckets; /* number of hash buckets */
104112 u32 elem_size; /* size of each element in bytes */
105113 u32 hashrnd;
@@ -114,14 +122,14 @@ struct htab_elem { · 8 unmodified lines
114122 struct {
115123 void *padding;
116124 union {
117− struct bpf_htab *htab;
118125 struct pcpu_freelist_node fnode;
119126 struct htab_elem *batch_flink;
120127 };
121128 };
122129 };
123130 union {
124− struct rcu_head rcu;
131+ /* pointer to per-cpu pointer */
132+ void *ptr_to_pptr;
125133 struct bpf_lru_node lru_node;
126134 };
127135 u32 hash;
@@ -162,17 +170,25 @@ static inline int htab_lock_bucket(const struct bpf_htab *htab, · 34 unmodified lines
162170 unsigned long *pflags)
163171 {
164172 unsigned long flags;
173+ bool use_raw_lock;
165174
166175 hash = hash & HASHTAB_MAP_LOCK_MASK;
167176
168− migrate_disable();
177+ use_raw_lock = htab_use_raw_lock(htab);
178+ if (use_raw_lock)
179+ preempt_disable();
180+ else
181+ migrate_disable();
169182 if (unlikely(__this_cpu_inc_return(*(htab->map_locked[hash])) != 1)) {
170183 __this_cpu_dec(*(htab->map_locked[hash]));
171− migrate_enable();
184+ if (use_raw_lock)
185+ preempt_enable();
186+ else
187+ migrate_enable();
172188 return -EBUSY;
173189 }
174190
175− if (htab_use_raw_lock(htab))
191+ if (use_raw_lock)
176192 raw_spin_lock_irqsave(&b->raw_lock, flags);
177193 else
178194 spin_lock_irqsave(&b->lock, flags);
@@ -185,13 +201,18 @@ static inline void htab_unlock_bucket(const struct bpf_htab *htab, · 6 unmodified lines
185201 struct bucket *b, u32 hash,
186202 unsigned long flags)
187203 {
204+ bool use_raw_lock = htab_use_raw_lock(htab);
205+
188206 hash = hash & HASHTAB_MAP_LOCK_MASK;
189− if (htab_use_raw_lock(htab))
207+ if (use_raw_lock)
190208 raw_spin_unlock_irqrestore(&b->raw_lock, flags);
191209 else
192210 spin_unlock_irqrestore(&b->lock, flags);
193211 __this_cpu_dec(*(htab->map_locked[hash]));
194− migrate_enable();
212+ if (use_raw_lock)
213+ preempt_enable();
214+ else
215+ migrate_enable();
195216 }
196217
197218 static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node);
@@ -428,8 +449,6 @@ static int htab_map_alloc_check(union bpf_attr *attr) · 230 unmodified lines
428449 bool zero_seed = (attr->map_flags & BPF_F_ZERO_SEED);
429450 int numa_node = bpf_map_attr_numa_node(attr);
430451
431− BUILD_BUG_ON(offsetof(struct htab_elem, htab) !=
432− offsetof(struct htab_elem, hash_node.pprev));
433452 BUILD_BUG_ON(offsetof(struct htab_elem, fnode.next) !=
434453 offsetof(struct htab_elem, hash_node.pprev));
435454
@@ -550,6 +569,29 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · 114 unmodified lines
550569
551570 htab_init_buckets(htab);
552571
572+/* compute_batch_value() computes batch value as num_online_cpus() * 2
573+ * and __percpu_counter_compare() needs
574+ * htab->max_entries - cur_number_of_elems to be more than batch * num_online_cpus()
575+ * for percpu_counter to be faster than atomic_t. In practice the average bpf
576+ * hash map size is 10k, which means that a system with 64 cpus will fill
577+ * hashmap to 20% of 10k before percpu_counter becomes ineffective. Therefore
578+ * define our own batch count as 32 then 10k hash map can be filled up to 80%:
579+ * 10k - 8k > 32 _batch_ * 64 _cpus_
580+ * and __percpu_counter_compare() will still be fast. At that point hash map
581+ * collisions will dominate its performance anyway. Assume that hash map filled
582+ * to 50+% isn't going to be O(1) and use the following formula to choose
583+ * between percpu_counter and atomic_t.
584+ */
585+#define PERCPU_COUNTER_BATCH 32
586+ if (attr->max_entries / 2 > num_online_cpus() * PERCPU_COUNTER_BATCH)
587+ htab->use_percpu_counter = true;
588+
589+ if (htab->use_percpu_counter) {
590+ err = percpu_counter_init(&htab->pcount, 0, GFP_KERNEL);
591+ if (err)
592+ goto free_map_locked;
593+ }
594+
553595 if (prealloc) {
554596 err = prealloc_init(htab);
555597 if (err)
@@ -563,6 +605,16 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · 7 unmodified lines
563605 if (err)
564606 goto free_prealloc;
565607 }
608+ } else {
609+ err = bpf_mem_alloc_init(&htab->ma, htab->elem_size, false);
610+ if (err)
611+ goto free_map_locked;
612+ if (percpu) {
613+ err = bpf_mem_alloc_init(&htab->pcpu_ma,
614+ round_up(htab->map.value_size, 8), true);
615+ if (err)
616+ goto free_map_locked;
617+ }
566618 }
567619
568620 return &htab->map;
@@ -573,6 +625,8 @@ free_map_locked: · 4 unmodified lines
573625 for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++)
574626 free_percpu(htab->map_locked[i]);
575627 bpf_map_area_free(htab->buckets);
628+ bpf_mem_alloc_destroy(&htab->pcpu_ma);
629+ bpf_mem_alloc_destroy(&htab->ma);
576630 free_htab:
577631 lockdep_unregister_key(&htab->lockdep_key);
578632 bpf_map_area_free(htab);
@@ -847,17 +901,9 @@ find_first_elem: · 268 unmodified lines
847901 static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l)
848902 {
849903 if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH)
850− free_percpu(htab_elem_get_ptr(l, htab->map.key_size));
904+ bpf_mem_cache_free(&htab->pcpu_ma, l->ptr_to_pptr);
851905 check_and_free_fields(htab, l);
852− kfree(l);
853−}
854−
855−static void htab_elem_free_rcu(struct rcu_head *head)
856−{
857− struct htab_elem *l = container_of(head, struct htab_elem, rcu);
858− struct bpf_htab *htab = l->htab;
859−
860− htab_elem_free(htab, l);
906+ bpf_mem_cache_free(&htab->ma, l);
861907 }
862908
863909 static void htab_put_fd_value(struct bpf_htab *htab, struct htab_elem *l)
@@ -871,6 +917,31 @@ static void htab_put_fd_value(struct bpf_htab *htab, struct htab_elem *l) · 7 unmodified lines
871917 }
872918 }
873919
920+static bool is_map_full(struct bpf_htab *htab)
921+{
922+ if (htab->use_percpu_counter)
923+ return __percpu_counter_compare(&htab->pcount, htab->map.max_entries,
924+ PERCPU_COUNTER_BATCH) >= 0;
925+ return atomic_read(&htab->count) >= htab->map.max_entries;
926+}
927+
928+static void inc_elem_count(struct bpf_htab *htab)
929+{
930+ if (htab->use_percpu_counter)
931+ percpu_counter_add_batch(&htab->pcount, 1, PERCPU_COUNTER_BATCH);
932+ else
933+ atomic_inc(&htab->count);
934+}
935+
936+static void dec_elem_count(struct bpf_htab *htab)
937+{
938+ if (htab->use_percpu_counter)
939+ percpu_counter_add_batch(&htab->pcount, -1, PERCPU_COUNTER_BATCH);
940+ else
941+ atomic_dec(&htab->count);
942+}
943+
944+
874945 static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
875946 {
876947 htab_put_fd_value(htab, l);
@@ -879,9 +950,8 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) · 2 unmodified lines
879950 check_and_free_fields(htab, l);
880951 __pcpu_freelist_push(&htab->freelist, &l->fnode);
881952 } else {
882− atomic_dec(&htab->count);
883− l->htab = htab;
884− call_rcu(&l->rcu, htab_elem_free_rcu);
953+ dec_elem_count(htab);
954+ htab_elem_free(htab, l);
885955 }
886956 }
887957
@@ -906,13 +976,12 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 18 unmodified lines
906976 static void pcpu_init_value(struct bpf_htab *htab, void __percpu *pptr,
907977 void *value, bool onallcpus)
908978 {
909− /* When using prealloc and not setting the initial value on all cpus,
910− * zero-fill element values for other cpus (just as what happens when
911− * not using prealloc). Otherwise, bpf program has no way to ensure
979+ /* When not setting the initial value on all cpus, zero-fill element
980+ * values for other cpus. Otherwise, bpf program has no way to ensure
912981 * known initial values for cpus other than current one
913982 * (onallcpus=false always when coming from bpf prog).
914983 */
915− if (htab_is_prealloc(htab) && !onallcpus) {
984+ if (!onallcpus) {
916985 u32 size = round_up(htab->map.value_size, 8);
917986 int current_cpu = raw_smp_processor_id();
918987 int cpu;
@@ -963,19 +1032,16 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key, · 44 unmodified lines
9631032 l_new = container_of(l, struct htab_elem, fnode);
9641033 }
9651034 } else {
966− if (atomic_inc_return(&htab->count) > htab->map.max_entries)
967− if (!old_elem) {
1035+ if (is_map_full(htab))
1036+ if (!old_elem)
9681037 /* when map is full and update() is replacing
9691038 * old element, it's ok to allocate, since
9701039 * old element will be freed immediately.
9711040 * Otherwise return an error
9721041 */
973− l_new = ERR_PTR(-E2BIG);
974− goto dec_count;
975− }
976− l_new = bpf_map_kmalloc_node(&htab->map, htab->elem_size,
977− GFP_NOWAIT | __GFP_NOWARN,
978− htab->map.numa_node);
1042+ return ERR_PTR(-E2BIG);
1043+ inc_elem_count(htab);
1044+ l_new = bpf_mem_cache_alloc(&htab->ma);
9791045 if (!l_new) {
9801046 l_new = ERR_PTR(-ENOMEM);
9811047 goto dec_count;
@@ -986,18 +1052,18 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key, · 4 unmodified lines
9861052
9871053 memcpy(l_new->key, key, key_size);
9881054 if (percpu) {
989− size = round_up(size, 8);
9901055 if (prealloc) {
9911056 pptr = htab_elem_get_ptr(l_new, key_size);
9921057 } else {
9931058 /* alloc_percpu zero-fills */
994− pptr = bpf_map_alloc_percpu(&htab->map, size, 8,
995− GFP_NOWAIT | __GFP_NOWARN);
1059+ pptr = bpf_mem_cache_alloc(&htab->pcpu_ma);
9961060 if (!pptr) {
997− kfree(l_new);
1061+ bpf_mem_cache_free(&htab->ma, l_new);
9981062 l_new = ERR_PTR(-ENOMEM);
9991063 goto dec_count;
10001064 }
1065+ l_new->ptr_to_pptr = pptr;
1066+ pptr = *(void **)pptr;
10011067 }
10021068
10031069 pcpu_init_value(htab, pptr, value, onallcpus);
@@ -1016,7 +1082,7 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key, · 12 unmodified lines
10161082 l_new->hash = hash;
10171083 return l_new;
10181084 dec_count:
1019− atomic_dec(&htab->count);
1085+ dec_elem_count(htab);
10201086 return l_new;
10211087 }
10221088
@@ -1416,6 +1482,10 @@ static void delete_all_elements(struct bpf_htab *htab) · 393 unmodified lines
14161482 {
14171483 int i;
14181484
1485+ /* It's called from a worker thread, so disable migration here,
1486+ * since bpf_mem_cache_free() relies on that.
1487+ */
1488+ migrate_disable();
14191489 for (i = 0; i < htab->n_buckets; i++) {
14201490 struct hlist_nulls_head *head = select_bucket(htab, i);
14211491 struct hlist_nulls_node *n;
@@ -1426,6 +1496,7 @@ static void delete_all_elements(struct bpf_htab *htab) · 4 unmodified lines
14261496 htab_elem_free(htab, l);
14271497 }
14281498 }
1499+ migrate_enable();
14291500 }
14301501
14311502 static void htab_free_malloced_timers(struct bpf_htab *htab)
@@ -1475,10 +1546,10 @@ static void htab_map_free(struct bpf_map *map) · 43 unmodified lines
14751546 * There is no need to synchronize_rcu() here to protect map elements.
14761547 */
14771548
1478− /* some of free_htab_elem() callbacks for elements of this map may
1479− * not have executed. Wait for them.
1549+ /* htab no longer uses call_rcu() directly. bpf_mem_alloc does it
1550+ * underneath and is reponsible for waiting for callbacks to finish
1551+ * during bpf_mem_alloc_destroy().
14801552 */
1481− rcu_barrier();
14821553 if (!htab_is_prealloc(htab)) {
14831554 delete_all_elements(htab);
14841555 } else {
@@ -1489,6 +1560,10 @@ static void htab_map_free(struct bpf_map *map) · 4 unmodified lines
14891560 bpf_map_free_kptr_off_tab(map);
14901561 free_percpu(htab->extra_elems);
14911562 bpf_map_area_free(htab->buckets);
1563+ bpf_mem_alloc_destroy(&htab->pcpu_ma);
1564+ bpf_mem_alloc_destroy(&htab->ma);
1565+ if (htab->use_percpu_counter)
1566+ percpu_counter_destroy(&htab->pcount);
14921567 for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++)
14931568 free_percpu(htab->map_locked[i]);
14941569 lockdep_unregister_key(&htab->lockdep_key);
@@ -1691,8 +1766,11 @@ again_nocopy: · 196 unmodified lines
16911766 /* do not grab the lock unless need it (bucket_cnt > 0). */
16921767 if (locked) {
16931768 ret = htab_lock_bucket(htab, b, batch, &flags);
1694− if (ret)
1695− goto next_batch;
1769+ if (ret) {
1770+ rcu_read_unlock();
1771+ bpf_enable_instrumentation();
1772+ goto after_loop;
1773+ }
16961774 }
16971775
16981776 bucket_cnt = 0;
kernel/bpf/helpers.c+9−39
@@ -427,40 +427,7 @@ const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto = { · 426 unmodified lines
427427 .ret_type = RET_INTEGER,
428428 .arg1_type = ARG_ANYTHING,
429429 };
430−
431−#ifdef CONFIG_CGROUP_BPF
432−
433−BPF_CALL_2(bpf_get_local_storage, struct bpf_map *, map, u64, flags)
434−{
435− /* flags argument is not used now,
436− * but provides an ability to extend the API.
437− * verifier checks that its value is correct.
438− */
439− enum bpf_cgroup_storage_type stype = cgroup_storage_type(map);
440− struct bpf_cgroup_storage *storage;
441− struct bpf_cg_run_ctx *ctx;
442− void *ptr;
443−
444− /* get current cgroup storage from BPF run context */
445− ctx = container_of(current->bpf_ctx, struct bpf_cg_run_ctx, run_ctx);
446− storage = ctx->prog_item->cgroup_storage[stype];
447−
448− if (stype == BPF_CGROUP_STORAGE_SHARED)
449− ptr = &READ_ONCE(storage->buf)->data[0];
450− else
451− ptr = this_cpu_ptr(storage->percpu_buf);
452−
453− return (unsigned long)ptr;
454−}
455−
456−const struct bpf_func_proto bpf_get_local_storage_proto = {
457− .func = bpf_get_local_storage,
458− .gpl_only = false,
459− .ret_type = RET_PTR_TO_MAP_VALUE,
460− .arg1_type = ARG_CONST_MAP_PTR,
461− .arg2_type = ARG_ANYTHING,
462−};
463−#endif
430+#endif /* CONFIG_CGROUPS */
464431
465432 #define BPF_STRTOX_BASE_MASK 0x1F
466433
@@ -589,7 +556,6 @@ const struct bpf_func_proto bpf_strtoul_proto = { · 122 unmodified lines
589556 .arg3_type = ARG_ANYTHING,
590557 .arg4_type = ARG_PTR_TO_LONG,
591558 };
592−#endif
593559
594560 BPF_CALL_3(bpf_strncmp, const char *, s1, u32, s1_sz, const char *, s2)
595561 {
@@ -1647,12 +1613,12 @@ bpf_base_func_proto(enum bpf_func_id func_id) · 1051 unmodified lines
16471613 return &bpf_ringbuf_submit_dynptr_proto;
16481614 case BPF_FUNC_ringbuf_discard_dynptr:
16491615 return &bpf_ringbuf_discard_dynptr_proto;
1650− case BPF_FUNC_for_each_map_elem:
1651− return &bpf_for_each_map_elem_proto;
1652− case BPF_FUNC_loop:
1653− return &bpf_loop_proto;
16541616 case BPF_FUNC_strncmp:
16551617 return &bpf_strncmp_proto;
1618+ case BPF_FUNC_strtol:
1619+ return &bpf_strtol_proto;
1620+ case BPF_FUNC_strtoul:
1621+ return &bpf_strtoul_proto;
16561622 case BPF_FUNC_dynptr_from_mem:
16571623 return &bpf_dynptr_from_mem_proto;
16581624 case BPF_FUNC_dynptr_read:
@@ -1689,6 +1655,10 @@ bpf_base_func_proto(enum bpf_func_id func_id) · 30 unmodified lines
16891655 return &bpf_timer_cancel_proto;
16901656 case BPF_FUNC_kptr_xchg:
16911657 return &bpf_kptr_xchg_proto;
1658+ case BPF_FUNC_for_each_map_elem:
1659+ return &bpf_for_each_map_elem_proto;
1660+ case BPF_FUNC_loop:
1661+ return &bpf_loop_proto;
16921662 default:
16931663 break;
16941664 }
kernel/bpf/memalloc.c+634−0
@@ -0,0 +1,634 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */
3+#include <linux/mm.h>
4+#include <linux/llist.h>
5+#include <linux/bpf.h>
6+#include <linux/irq_work.h>
7+#include <linux/bpf_mem_alloc.h>
8+#include <linux/memcontrol.h>
9+#include <asm/local.h>
10+
11+/* Any context (including NMI) BPF specific memory allocator.
12+ *
13+ * Tracing BPF programs can attach to kprobe and fentry. Hence they
14+ * run in unknown context where calling plain kmalloc() might not be safe.
15+ *
16+ * Front-end kmalloc() with per-cpu per-bucket cache of free elements.
17+ * Refill this cache asynchronously from irq_work.
18+ *
19+ * CPU_0 buckets
20+ * 16 32 64 96 128 196 256 512 1024 2048 4096
21+ * ...
22+ * CPU_N buckets
23+ * 16 32 64 96 128 196 256 512 1024 2048 4096
24+ *
25+ * The buckets are prefilled at the start.
26+ * BPF programs always run with migration disabled.
27+ * It's safe to allocate from cache of the current cpu with irqs disabled.
28+ * Free-ing is always done into bucket of the current cpu as well.
29+ * irq_work trims extra free elements from buckets with kfree
30+ * and refills them with kmalloc, so global kmalloc logic takes care
31+ * of freeing objects allocated by one cpu and freed on another.
32+ *
33+ * Every allocated objected is padded with extra 8 bytes that contains
34+ * struct llist_node.
35+ */
36+#define LLIST_NODE_SZ sizeof(struct llist_node)
37+
38+/* similar to kmalloc, but sizeof == 8 bucket is gone */
39+static u8 size_index[24] __ro_after_init = {
40+ 3, /* 8 */
41+ 3, /* 16 */
42+ 4, /* 24 */
43+ 4, /* 32 */
44+ 5, /* 40 */
45+ 5, /* 48 */
46+ 5, /* 56 */
47+ 5, /* 64 */
48+ 1, /* 72 */
49+ 1, /* 80 */
50+ 1, /* 88 */
51+ 1, /* 96 */
52+ 6, /* 104 */
53+ 6, /* 112 */
54+ 6, /* 120 */
55+ 6, /* 128 */
56+ 2, /* 136 */
57+ 2, /* 144 */
58+ 2, /* 152 */
59+ 2, /* 160 */
60+ 2, /* 168 */
61+ 2, /* 176 */
62+ 2, /* 184 */
63+ 2 /* 192 */
64+};
65+
66+static int bpf_mem_cache_idx(size_t size)
67+{
68+ if (!size || size > 4096)
69+ return -1;
70+
71+ if (size <= 192)
72+ return size_index[(size - 1) / 8] - 1;
73+
74+ return fls(size - 1) - 1;
75+}
76+
77+#define NUM_CACHES 11
78+
79+struct bpf_mem_cache {
80+ /* per-cpu list of free objects of size 'unit_size'.
81+ * All accesses are done with interrupts disabled and 'active' counter
82+ * protection with __llist_add() and __llist_del_first().
83+ */
84+ struct llist_head free_llist;
85+ local_t active;
86+
87+ /* Operations on the free_list from unit_alloc/unit_free/bpf_mem_refill
88+ * are sequenced by per-cpu 'active' counter. But unit_free() cannot
89+ * fail. When 'active' is busy the unit_free() will add an object to
90+ * free_llist_extra.
91+ */
92+ struct llist_head free_llist_extra;
93+
94+ struct irq_work refill_work;
95+ struct obj_cgroup *objcg;
96+ int unit_size;
97+ /* count of objects in free_llist */
98+ int free_cnt;
99+ int low_watermark, high_watermark, batch;
100+ int percpu_size;
101+
102+ struct rcu_head rcu;
103+ struct llist_head free_by_rcu;
104+ struct llist_head waiting_for_gp;
105+ atomic_t call_rcu_in_progress;
106+};
107+
108+struct bpf_mem_caches {
109+ struct bpf_mem_cache cache[NUM_CACHES];
110+};
111+
112+static struct llist_node notrace *__llist_del_first(struct llist_head *head)
113+{
114+ struct llist_node *entry, *next;
115+
116+ entry = head->first;
117+ if (!entry)
118+ return NULL;
119+ next = entry->next;
120+ head->first = next;
121+ return entry;
122+}
123+
124+static void *__alloc(struct bpf_mem_cache *c, int node)
125+{
126+ /* Allocate, but don't deplete atomic reserves that typical
127+ * GFP_ATOMIC would do. irq_work runs on this cpu and kmalloc
128+ * will allocate from the current numa node which is what we
129+ * want here.
130+ */
131+ gfp_t flags = GFP_NOWAIT | __GFP_NOWARN | __GFP_ACCOUNT;
132+
133+ if (c->percpu_size) {
134+ void **obj = kmalloc_node(c->percpu_size, flags, node);
135+ void *pptr = __alloc_percpu_gfp(c->unit_size, 8, flags);
136+
137+ if (!obj || !pptr) {
138+ free_percpu(pptr);
139+ kfree(obj);
140+ return NULL;
141+ }
142+ obj[1] = pptr;
143+ return obj;
144+ }
145+
146+ return kmalloc_node(c->unit_size, flags, node);
147+}
148+
149+static struct mem_cgroup *get_memcg(const struct bpf_mem_cache *c)
150+{
151+#ifdef CONFIG_MEMCG_KMEM
152+ if (c->objcg)
153+ return get_mem_cgroup_from_objcg(c->objcg);
154+#endif
155+
156+#ifdef CONFIG_MEMCG
157+ return root_mem_cgroup;
158+#else
159+ return NULL;
160+#endif
161+}
162+
163+/* Mostly runs from irq_work except __init phase. */
164+static void alloc_bulk(struct bpf_mem_cache *c, int cnt, int node)
165+{
166+ struct mem_cgroup *memcg = NULL, *old_memcg;
167+ unsigned long flags;
168+ void *obj;
169+ int i;
170+
171+ memcg = get_memcg(c);
172+ old_memcg = set_active_memcg(memcg);
173+ for (i = 0; i < cnt; i++) {
174+ obj = __alloc(c, node);
175+ if (!obj)
176+ break;
177+ if (IS_ENABLED(CONFIG_PREEMPT_RT))
178+ /* In RT irq_work runs in per-cpu kthread, so disable
179+ * interrupts to avoid preemption and interrupts and
180+ * reduce the chance of bpf prog executing on this cpu
181+ * when active counter is busy.
182+ */
183+ local_irq_save(flags);
184+ /* alloc_bulk runs from irq_work which will not preempt a bpf
185+ * program that does unit_alloc/unit_free since IRQs are
186+ * disabled there. There is no race to increment 'active'
187+ * counter. It protects free_llist from corruption in case NMI
188+ * bpf prog preempted this loop.
189+ */
190+ WARN_ON_ONCE(local_inc_return(&c->active) != 1);
191+ __llist_add(obj, &c->free_llist);
192+ c->free_cnt++;
193+ local_dec(&c->active);
194+ if (IS_ENABLED(CONFIG_PREEMPT_RT))
195+ local_irq_restore(flags);
196+ }
197+ set_active_memcg(old_memcg);
198+ mem_cgroup_put(memcg);
199+}
200+
201+static void free_one(struct bpf_mem_cache *c, void *obj)
202+{
203+ if (c->percpu_size) {
204+ free_percpu(((void **)obj)[1]);
205+ kfree(obj);
206+ return;
207+ }
208+
209+ kfree(obj);
210+}
211+
212+static void __free_rcu(struct rcu_head *head)
213+{
214+ struct bpf_mem_cache *c = container_of(head, struct bpf_mem_cache, rcu);
215+ struct llist_node *llnode = llist_del_all(&c->waiting_for_gp);
216+ struct llist_node *pos, *t;
217+
218+ llist_for_each_safe(pos, t, llnode)
219+ free_one(c, pos);
220+ atomic_set(&c->call_rcu_in_progress, 0);
221+}
222+
223+static void __free_rcu_tasks_trace(struct rcu_head *head)
224+{
225+ struct bpf_mem_cache *c = container_of(head, struct bpf_mem_cache, rcu);
226+
227+ call_rcu(&c->rcu, __free_rcu);
228+}
229+
230+static void enque_to_free(struct bpf_mem_cache *c, void *obj)
231+{
232+ struct llist_node *llnode = obj;
233+
234+ /* bpf_mem_cache is a per-cpu object. Freeing happens in irq_work.
235+ * Nothing races to add to free_by_rcu list.
236+ */
237+ __llist_add(llnode, &c->free_by_rcu);
238+}
239+
240+static void do_call_rcu(struct bpf_mem_cache *c)
241+{
242+ struct llist_node *llnode, *t;
243+
244+ if (atomic_xchg(&c->call_rcu_in_progress, 1))
245+ return;
246+
247+ WARN_ON_ONCE(!llist_empty(&c->waiting_for_gp));
248+ llist_for_each_safe(llnode, t, __llist_del_all(&c->free_by_rcu))
249+ /* There is no concurrent __llist_add(waiting_for_gp) access.
250+ * It doesn't race with llist_del_all either.
251+ * But there could be two concurrent llist_del_all(waiting_for_gp):
252+ * from __free_rcu() and from drain_mem_cache().
253+ */
254+ __llist_add(llnode, &c->waiting_for_gp);
255+ /* Use call_rcu_tasks_trace() to wait for sleepable progs to finish.
256+ * Then use call_rcu() to wait for normal progs to finish
257+ * and finally do free_one() on each element.
258+ */
259+ call_rcu_tasks_trace(&c->rcu, __free_rcu_tasks_trace);
260+}
261+
262+static void free_bulk(struct bpf_mem_cache *c)
263+{
264+ struct llist_node *llnode, *t;
265+ unsigned long flags;
266+ int cnt;
267+
268+ do {
269+ if (IS_ENABLED(CONFIG_PREEMPT_RT))
270+ local_irq_save(flags);
271+ WARN_ON_ONCE(local_inc_return(&c->active) != 1);
272+ llnode = __llist_del_first(&c->free_llist);
273+ if (llnode)
274+ cnt = --c->free_cnt;
275+ else
276+ cnt = 0;
277+ local_dec(&c->active);
278+ if (IS_ENABLED(CONFIG_PREEMPT_RT))
279+ local_irq_restore(flags);
280+ enque_to_free(c, llnode);
281+ } while (cnt > (c->high_watermark + c->low_watermark) / 2);
282+
283+ /* and drain free_llist_extra */
284+ llist_for_each_safe(llnode, t, llist_del_all(&c->free_llist_extra))
285+ enque_to_free(c, llnode);
286+ do_call_rcu(c);
287+}
288+
289+static void bpf_mem_refill(struct irq_work *work)
290+{
291+ struct bpf_mem_cache *c = container_of(work, struct bpf_mem_cache, refill_work);
292+ int cnt;
293+
294+ /* Racy access to free_cnt. It doesn't need to be 100% accurate */
295+ cnt = c->free_cnt;
296+ if (cnt < c->low_watermark)
297+ /* irq_work runs on this cpu and kmalloc will allocate
298+ * from the current numa node which is what we want here.
299+ */
300+ alloc_bulk(c, c->batch, NUMA_NO_NODE);
301+ else if (cnt > c->high_watermark)
302+ free_bulk(c);
303+}
304+
305+static void notrace irq_work_raise(struct bpf_mem_cache *c)
306+{
307+ irq_work_queue(&c->refill_work);
308+}
309+
310+/* For typical bpf map case that uses bpf_mem_cache_alloc and single bucket
311+ * the freelist cache will be elem_size * 64 (or less) on each cpu.
312+ *
313+ * For bpf programs that don't have statically known allocation sizes and
314+ * assuming (low_mark + high_mark) / 2 as an average number of elements per
315+ * bucket and all buckets are used the total amount of memory in freelists
316+ * on each cpu will be:
317+ * 64*16 + 64*32 + 64*64 + 64*96 + 64*128 + 64*196 + 64*256 + 32*512 + 16*1024 + 8*2048 + 4*4096
318+ * == ~ 116 Kbyte using below heuristic.
319+ * Initialized, but unused bpf allocator (not bpf map specific one) will
320+ * consume ~ 11 Kbyte per cpu.
321+ * Typical case will be between 11K and 116K closer to 11K.
322+ * bpf progs can and should share bpf_mem_cache when possible.
323+ */
324+
325+static void prefill_mem_cache(struct bpf_mem_cache *c, int cpu)
326+{
327+ init_irq_work(&c->refill_work, bpf_mem_refill);
328+ if (c->unit_size <= 256) {
329+ c->low_watermark = 32;
330+ c->high_watermark = 96;
331+ } else {
332+ /* When page_size == 4k, order-0 cache will have low_mark == 2
333+ * and high_mark == 6 with batch alloc of 3 individual pages at
334+ * a time.
335+ * 8k allocs and above low == 1, high == 3, batch == 1.
336+ */
337+ c->low_watermark = max(32 * 256 / c->unit_size, 1);
338+ c->high_watermark = max(96 * 256 / c->unit_size, 3);
339+ }
340+ c->batch = max((c->high_watermark - c->low_watermark) / 4 * 3, 1);
341+
342+ /* To avoid consuming memory assume that 1st run of bpf
343+ * prog won't be doing more than 4 map_update_elem from
344+ * irq disabled region
345+ */
346+ alloc_bulk(c, c->unit_size <= 256 ? 4 : 1, cpu_to_node(cpu));
347+}
348+
349+/* When size != 0 bpf_mem_cache for each cpu.
350+ * This is typical bpf hash map use case when all elements have equal size.
351+ *
352+ * When size == 0 allocate 11 bpf_mem_cache-s for each cpu, then rely on
353+ * kmalloc/kfree. Max allocation size is 4096 in this case.
354+ * This is bpf_dynptr and bpf_kptr use case.
355+ */
356+int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu)
357+{
358+ static u16 sizes[NUM_CACHES] = {96, 192, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096};
359+ struct bpf_mem_caches *cc, __percpu *pcc;
360+ struct bpf_mem_cache *c, __percpu *pc;
361+ struct obj_cgroup *objcg = NULL;
362+ int cpu, i, unit_size, percpu_size = 0;
363+
364+ if (size) {
365+ pc = __alloc_percpu_gfp(sizeof(*pc), 8, GFP_KERNEL);
366+ if (!pc)
367+ return -ENOMEM;
368+
369+ if (percpu)
370+ /* room for llist_node and per-cpu pointer */
371+ percpu_size = LLIST_NODE_SZ + sizeof(void *);
372+ else
373+ size += LLIST_NODE_SZ; /* room for llist_node */
374+ unit_size = size;
375+
376+#ifdef CONFIG_MEMCG_KMEM
377+ objcg = get_obj_cgroup_from_current();
378+#endif
379+ for_each_possible_cpu(cpu) {
380+ c = per_cpu_ptr(pc, cpu);
381+ c->unit_size = unit_size;
382+ c->objcg = objcg;
383+ c->percpu_size = percpu_size;
384+ prefill_mem_cache(c, cpu);
385+ }
386+ ma->cache = pc;
387+ return 0;
388+ }
389+
390+ /* size == 0 && percpu is an invalid combination */
391+ if (WARN_ON_ONCE(percpu))
392+ return -EINVAL;
393+
394+ pcc = __alloc_percpu_gfp(sizeof(*cc), 8, GFP_KERNEL);
395+ if (!pcc)
396+ return -ENOMEM;
397+#ifdef CONFIG_MEMCG_KMEM
398+ objcg = get_obj_cgroup_from_current();
399+#endif
400+ for_each_possible_cpu(cpu) {
401+ cc = per_cpu_ptr(pcc, cpu);
402+ for (i = 0; i < NUM_CACHES; i++) {
403+ c = &cc->cache[i];
404+ c->unit_size = sizes[i];
405+ c->objcg = objcg;
406+ prefill_mem_cache(c, cpu);
407+ }
408+ }
409+ ma->caches = pcc;
410+ return 0;
411+}
412+
413+static void drain_mem_cache(struct bpf_mem_cache *c)
414+{
415+ struct llist_node *llnode, *t;
416+
417+ /* No progs are using this bpf_mem_cache, but htab_map_free() called
418+ * bpf_mem_cache_free() for all remaining elements and they can be in
419+ * free_by_rcu or in waiting_for_gp lists, so drain those lists now.
420+ */
421+ llist_for_each_safe(llnode, t, __llist_del_all(&c->free_by_rcu))
422+ free_one(c, llnode);
423+ llist_for_each_safe(llnode, t, llist_del_all(&c->waiting_for_gp))
424+ free_one(c, llnode);
425+ llist_for_each_safe(llnode, t, llist_del_all(&c->free_llist))
426+ free_one(c, llnode);
427+ llist_for_each_safe(llnode, t, llist_del_all(&c->free_llist_extra))
428+ free_one(c, llnode);
429+}
430+
431+static void free_mem_alloc_no_barrier(struct bpf_mem_alloc *ma)
432+{
433+ free_percpu(ma->cache);
434+ free_percpu(ma->caches);
435+ ma->cache = NULL;
436+ ma->caches = NULL;
437+}
438+
439+static void free_mem_alloc(struct bpf_mem_alloc *ma)
440+{
441+ /* waiting_for_gp lists was drained, but __free_rcu might
442+ * still execute. Wait for it now before we freeing percpu caches.
443+ */
444+ rcu_barrier_tasks_trace();
445+ rcu_barrier();
446+ free_mem_alloc_no_barrier(ma);
447+}
448+
449+static void free_mem_alloc_deferred(struct work_struct *work)
450+{
451+ struct bpf_mem_alloc *ma = container_of(work, struct bpf_mem_alloc, work);
452+
453+ free_mem_alloc(ma);
454+ kfree(ma);
455+}
456+
457+static void destroy_mem_alloc(struct bpf_mem_alloc *ma, int rcu_in_progress)
458+{
459+ struct bpf_mem_alloc *copy;
460+
461+ if (!rcu_in_progress) {
462+ /* Fast path. No callbacks are pending, hence no need to do
463+ * rcu_barrier-s.
464+ */
465+ free_mem_alloc_no_barrier(ma);
466+ return;
467+ }
468+
469+ copy = kmalloc(sizeof(*ma), GFP_KERNEL);
470+ if (!copy) {
471+ /* Slow path with inline barrier-s */
472+ free_mem_alloc(ma);
473+ return;
474+ }
475+
476+ /* Defer barriers into worker to let the rest of map memory to be freed */
477+ copy->cache = ma->cache;
478+ ma->cache = NULL;
479+ copy->caches = ma->caches;
480+ ma->caches = NULL;
481+ INIT_WORK(&copy->work, free_mem_alloc_deferred);
482+ queue_work(system_unbound_wq, &copy->work);
483+}
484+
485+void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma)
486+{
487+ struct bpf_mem_caches *cc;
488+ struct bpf_mem_cache *c;
489+ int cpu, i, rcu_in_progress;
490+
491+ if (ma->cache) {
492+ rcu_in_progress = 0;
493+ for_each_possible_cpu(cpu) {
494+ c = per_cpu_ptr(ma->cache, cpu);
495+ drain_mem_cache(c);
496+ rcu_in_progress += atomic_read(&c->call_rcu_in_progress);
497+ }
498+ /* objcg is the same across cpus */
499+ if (c->objcg)
500+ obj_cgroup_put(c->objcg);
501+ destroy_mem_alloc(ma, rcu_in_progress);
502+ }
503+ if (ma->caches) {
504+ rcu_in_progress = 0;
505+ for_each_possible_cpu(cpu) {
506+ cc = per_cpu_ptr(ma->caches, cpu);
507+ for (i = 0; i < NUM_CACHES; i++) {
508+ c = &cc->cache[i];
509+ drain_mem_cache(c);
510+ rcu_in_progress += atomic_read(&c->call_rcu_in_progress);
511+ }
512+ }
513+ if (c->objcg)
514+ obj_cgroup_put(c->objcg);
515+ destroy_mem_alloc(ma, rcu_in_progress);
516+ }
517+}
518+
519+/* notrace is necessary here and in other functions to make sure
520+ * bpf programs cannot attach to them and cause llist corruptions.
521+ */
522+static void notrace *unit_alloc(struct bpf_mem_cache *c)
523+{
524+ struct llist_node *llnode = NULL;
525+ unsigned long flags;
526+ int cnt = 0;
527+
528+ /* Disable irqs to prevent the following race for majority of prog types:
529+ * prog_A
530+ * bpf_mem_alloc
531+ * preemption or irq -> prog_B
532+ * bpf_mem_alloc
533+ *
534+ * but prog_B could be a perf_event NMI prog.
535+ * Use per-cpu 'active' counter to order free_list access between
536+ * unit_alloc/unit_free/bpf_mem_refill.
537+ */
538+ local_irq_save(flags);
539+ if (local_inc_return(&c->active) == 1) {
540+ llnode = __llist_del_first(&c->free_llist);
541+ if (llnode)
542+ cnt = --c->free_cnt;
543+ }
544+ local_dec(&c->active);
545+ local_irq_restore(flags);
546+
547+ WARN_ON(cnt < 0);
548+
549+ if (cnt < c->low_watermark)
550+ irq_work_raise(c);
551+ return llnode;
552+}
553+
554+/* Though 'ptr' object could have been allocated on a different cpu
555+ * add it to the free_llist of the current cpu.
556+ * Let kfree() logic deal with it when it's later called from irq_work.
557+ */
558+static void notrace unit_free(struct bpf_mem_cache *c, void *ptr)
559+{
560+ struct llist_node *llnode = ptr - LLIST_NODE_SZ;
561+ unsigned long flags;
562+ int cnt = 0;
563+
564+ BUILD_BUG_ON(LLIST_NODE_SZ > 8);
565+
566+ local_irq_save(flags);
567+ if (local_inc_return(&c->active) == 1) {
568+ __llist_add(llnode, &c->free_llist);
569+ cnt = ++c->free_cnt;
570+ } else {
571+ /* unit_free() cannot fail. Therefore add an object to atomic
572+ * llist. free_bulk() will drain it. Though free_llist_extra is
573+ * a per-cpu list we have to use atomic llist_add here, since
574+ * it also can be interrupted by bpf nmi prog that does another
575+ * unit_free() into the same free_llist_extra.
576+ */
577+ llist_add(llnode, &c->free_llist_extra);
578+ }
579+ local_dec(&c->active);
580+ local_irq_restore(flags);
581+
582+ if (cnt > c->high_watermark)
583+ /* free few objects from current cpu into global kmalloc pool */
584+ irq_work_raise(c);
585+}
586+
587+/* Called from BPF program or from sys_bpf syscall.
588+ * In both cases migration is disabled.
589+ */
590+void notrace *bpf_mem_alloc(struct bpf_mem_alloc *ma, size_t size)
591+{
592+ int idx;
593+ void *ret;
594+
595+ if (!size)
596+ return ZERO_SIZE_PTR;
597+
598+ idx = bpf_mem_cache_idx(size + LLIST_NODE_SZ);
599+ if (idx < 0)
600+ return NULL;
601+
602+ ret = unit_alloc(this_cpu_ptr(ma->caches)->cache + idx);
603+ return !ret ? NULL : ret + LLIST_NODE_SZ;
604+}
605+
606+void notrace bpf_mem_free(struct bpf_mem_alloc *ma, void *ptr)
607+{
608+ int idx;
609+
610+ if (!ptr)
611+ return;
612+
613+ idx = bpf_mem_cache_idx(__ksize(ptr - LLIST_NODE_SZ));
614+ if (idx < 0)
615+ return;
616+
617+ unit_free(this_cpu_ptr(ma->caches)->cache + idx, ptr);
618+}
619+
620+void notrace *bpf_mem_cache_alloc(struct bpf_mem_alloc *ma)
621+{
622+ void *ret;
623+
624+ ret = unit_alloc(this_cpu_ptr(ma->cache));
625+ return !ret ? NULL : ret + LLIST_NODE_SZ;
626+}
627+
628+void notrace bpf_mem_cache_free(struct bpf_mem_alloc *ma, void *ptr)
629+{
630+ if (!ptr)
631+ return;
632+
633+ unit_free(this_cpu_ptr(ma->cache), ptr);
634+}
kernel/bpf/syscall.c+10−5
@@ -638,7 +638,10 @@ static void __bpf_map_put(struct bpf_map *map, bool do_idr_lock) · 637 unmodified lines
638638 bpf_map_free_id(map, do_idr_lock);
639639 btf_put(map->btf);
640640 INIT_WORK(&map->work, bpf_map_free_deferred);
641− schedule_work(&map->work);
641+ /* Avoid spawning kworkers, since they all might contend
642+ * for the same mutex like slab_mutex.
643+ */
644+ queue_work(system_unbound_wq, &map->work);
642645 }
643646 }
644647
@@ -1437,9 +1440,9 @@ err_put: · 792 unmodified lines
14371440
14381441 #define BPF_MAP_DELETE_ELEM_LAST_FIELD key
14391442
1440−static int map_delete_elem(union bpf_attr *attr)
1443+static int map_delete_elem(union bpf_attr *attr, bpfptr_t uattr)
14411444 {
1442− void __user *ukey = u64_to_user_ptr(attr->key);
1445+ bpfptr_t ukey = make_bpfptr(attr->key, uattr.is_kernel);
14431446 int ufd = attr->map_fd;
14441447 struct bpf_map *map;
14451448 struct fd f;
@@ -1459,7 +1462,7 @@ static int map_delete_elem(union bpf_attr *attr) · 13 unmodified lines
14591462 goto err_put;
14601463 }
14611464
1462− key = __bpf_copy_key(ukey, map->key_size);
1465+ key = ___bpf_copy_key(ukey, map->key_size);
14631466 if (IS_ERR(key)) {
14641467 err = PTR_ERR(key);
14651468 goto err_put;
@@ -4941,7 +4944,7 @@ static int __sys_bpf(int cmd, bpfptr_t uattr, unsigned int size) · 3475 unmodified lines
49414944 err = map_update_elem(&attr, uattr);
49424945 break;
49434946 case BPF_MAP_DELETE_ELEM:
4944− err = map_delete_elem(&attr);
4947+ err = map_delete_elem(&attr, uattr);
49454948 break;
49464949 case BPF_MAP_GET_NEXT_KEY:
49474950 err = map_get_next_key(&attr);
@@ -5073,8 +5076,10 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size) · 125 unmodified lines
50735076 {
50745077 switch (cmd) {
50755078 case BPF_MAP_CREATE:
5079+ case BPF_MAP_DELETE_ELEM:
50765080 case BPF_MAP_UPDATE_ELEM:
50775081 case BPF_MAP_FREEZE:
5082+ case BPF_MAP_GET_FD_BY_ID:
50785083 case BPF_PROG_LOAD:
50795084 case BPF_BTF_LOAD:
50805085 case BPF_LINK_CREATE:
kernel/bpf/trampoline.c+4−4
@@ -895,7 +895,7 @@ u64 notrace __bpf_prog_enter(struct bpf_prog *prog, struct bpf_tramp_run_ctx *ru · 894 unmodified lines
895895
896896 run_ctx->saved_run_ctx = bpf_set_run_ctx(&run_ctx->run_ctx);
897897
898− if (unlikely(__this_cpu_inc_return(*(prog->active)) != 1)) {
898+ if (unlikely(this_cpu_inc_return(*(prog->active)) != 1)) {
899899 inc_misses_counter(prog);
900900 return 0;
901901 }
@@ -930,7 +930,7 @@ void notrace __bpf_prog_exit(struct bpf_prog *prog, u64 start, struct bpf_tramp_ · 28 unmodified lines
930930 bpf_reset_run_ctx(run_ctx->saved_run_ctx);
931931
932932 update_prog_stats(prog, start);
933− __this_cpu_dec(*(prog->active));
933+ this_cpu_dec(*(prog->active));
934934 migrate_enable();
935935 rcu_read_unlock();
936936 }
@@ -966,7 +966,7 @@ u64 notrace __bpf_prog_enter_sleepable(struct bpf_prog *prog, struct bpf_tramp_r · 29 unmodified lines
966966 migrate_disable();
967967 might_fault();
968968
969− if (unlikely(__this_cpu_inc_return(*(prog->active)) != 1)) {
969+ if (unlikely(this_cpu_inc_return(*(prog->active)) != 1)) {
970970 inc_misses_counter(prog);
971971 return 0;
972972 }
@@ -982,7 +982,7 @@ void notrace __bpf_prog_exit_sleepable(struct bpf_prog *prog, u64 start, · 9 unmodified lines
982982 bpf_reset_run_ctx(run_ctx->saved_run_ctx);
983983
984984 update_prog_stats(prog, start);
985− __this_cpu_dec(*(prog->active));
985+ this_cpu_dec(*(prog->active));
986986 migrate_enable();
987987 rcu_read_unlock_trace();
988988 }
kernel/bpf/verifier.c+33−61
@@ -1092,6 +1092,7 @@ static int acquire_reference_state(struct bpf_verifier_env *env, int insn_idx) · 1091 unmodified lines
10921092 id = ++env->id_gen;
10931093 state->refs[new_ofs].id = id;
10941094 state->refs[new_ofs].insn_idx = insn_idx;
1095+ state->refs[new_ofs].callback_ref = state->in_callback_fn ? state->frameno : 0;
10951096
10961097 return id;
10971098 }
@@ -1104,6 +1105,9 @@ static int release_reference_state(struct bpf_func_state *state, int ptr_id) · 6 unmodified lines
11041105 last_idx = state->acquired_refs - 1;
11051106 for (i = 0; i < state->acquired_refs; i++) {
11061107 if (state->refs[i].id == ptr_id) {
1108+ /* Cannot release caller references in callbacks */
1109+ if (state->in_callback_fn && state->refs[i].callback_ref != state->frameno)
1110+ return -EINVAL;
11071111 if (last_idx && i != last_idx)
11081112 memcpy(&state->refs[i], &state->refs[last_idx],
11091113 sizeof(*state->refs));
@@ -6918,10 +6922,17 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · 5808 unmodified lines
69186922 caller->regs[BPF_REG_0] = *r0;
69196923 }
69206924
6921− /* Transfer references to the caller */
6922− err = copy_reference_state(caller, callee);
6923− if (err)
6924− return err;
6925+ /* callback_fn frame should have released its own additions to parent's
6926+ * reference state at this point, or check_reference_leak would
6927+ * complain, hence it must be the same as the caller. There is no need
6928+ * to copy it back.
6929+ */
6930+ if (!callee->in_callback_fn) {
6931+ /* Transfer references to the caller */
6932+ err = copy_reference_state(caller, callee);
6933+ if (err)
6934+ return err;
6935+ }
69256936
69266937 *insn_idx = callee->callsite + 1;
69276938 if (env->log.level & BPF_LOG_LEVEL) {
@@ -7043,13 +7054,20 @@ record_func_key(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, · 115 unmodified lines
70437054 static int check_reference_leak(struct bpf_verifier_env *env)
70447055 {
70457056 struct bpf_func_state *state = cur_func(env);
7057+ bool refs_lingering = false;
70467058 int i;
70477059
7060+ if (state->frameno && !state->in_callback_fn)
7061+ return 0;
7062+
70487063 for (i = 0; i < state->acquired_refs; i++) {
7064+ if (state->in_callback_fn && state->refs[i].callback_ref != state->frameno)
7065+ continue;
70497066 verbose(env, "Unreleased reference id=%d alloc_insn=%d\n",
70507067 state->refs[i].id, state->refs[i].insn_idx);
7068+ refs_lingering = true;
70517069 }
7052− return state->acquired_refs ? -EINVAL : 0;
7070+ return refs_lingering ? -EINVAL : 0;
70537071 }
70547072
70557073 static int check_bpf_snprintf_call(struct bpf_verifier_env *env,
@@ -12338,6 +12356,16 @@ static int do_check(struct bpf_verifier_env *env) · 5282 unmodified lines
1233812356 return -EINVAL;
1233912357 }
1234012358
12359+ /* We must do check_reference_leak here before
12360+ * prepare_func_exit to handle the case when
12361+ * state->curframe > 0, it may be a callback
12362+ * function, for which reference_state must
12363+ * match caller reference state when it exits.
12364+ */
12365+ err = check_reference_leak(env);
12366+ if (err)
12367+ return err;
12368+
1234112369 if (state->curframe) {
1234212370 /* exit from nested function */
1234312371 err = prepare_func_exit(env, &env->insn_idx);
@@ -12347,10 +12375,6 @@ static int do_check(struct bpf_verifier_env *env) · 3 unmodified lines
1234712375 continue;
1234812376 }
1234912377
12350− err = check_reference_leak(env);
12351− if (err)
12352− return err;
12353−
1235412378 err = check_return_code(env);
1235512379 if (err)
1235612380 return err;
@@ -12563,14 +12587,6 @@ err_put: · 206 unmodified lines
1256312587 return err;
1256412588 }
1256512589
12566−static int check_map_prealloc(struct bpf_map *map)
12567−{
12568− return (map->map_type != BPF_MAP_TYPE_HASH &&
12569− map->map_type != BPF_MAP_TYPE_PERCPU_HASH &&
12570− map->map_type != BPF_MAP_TYPE_HASH_OF_MAPS) ||
12571− !(map->map_flags & BPF_F_NO_PREALLOC);
12572−}
12573−
1257412590 static bool is_tracing_prog_type(enum bpf_prog_type type)
1257512591 {
1257612592 switch (type) {
@@ -12585,50 +12601,12 @@ static bool is_tracing_prog_type(enum bpf_prog_type type) · 8 unmodified lines
1258512601 }
1258612602 }
1258712603
12588−static bool is_preallocated_map(struct bpf_map *map)
12589−{
12590− if (!check_map_prealloc(map))
12591− return false;
12592− if (map->inner_map_meta && !check_map_prealloc(map->inner_map_meta))
12593− return false;
12594− return true;
12595−}
12596−
1259712604 static int check_map_prog_compatibility(struct bpf_verifier_env *env,
1259812605 struct bpf_map *map,
1259912606 struct bpf_prog *prog)
1260012607
1260112608 {
1260212609 enum bpf_prog_type prog_type = resolve_prog_type(prog);
12603− /*
12604− * Validate that trace type programs use preallocated hash maps.
12605− *
12606− * For programs attached to PERF events this is mandatory as the
12607− * perf NMI can hit any arbitrary code sequence.
12608− *
12609− * All other trace types using preallocated hash maps are unsafe as
12610− * well because tracepoint or kprobes can be inside locked regions
12611− * of the memory allocator or at a place where a recursion into the
12612− * memory allocator would see inconsistent state.
12613− *
12614− * On RT enabled kernels run-time allocation of all trace type
12615− * programs is strictly prohibited due to lock type constraints. On
12616− * !RT kernels it is allowed for backwards compatibility reasons for
12617− * now, but warnings are emitted so developers are made aware of
12618− * the unsafety and can fix their programs before this is enforced.
12619− */
12620− if (is_tracing_prog_type(prog_type) && !is_preallocated_map(map)) {
12621− if (prog_type == BPF_PROG_TYPE_PERF_EVENT) {
12622− verbose(env, "perf_event programs can only use preallocated hash map\n");
12623− return -EINVAL;
12624− }
12625− if (IS_ENABLED(CONFIG_PREEMPT_RT)) {
12626− verbose(env, "trace type programs can only use preallocated hash map\n");
12627− return -EINVAL;
12628− }
12629− WARN_ONCE(1, "trace type BPF program uses run-time allocation\n");
12630− verbose(env, "trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.\n");
12631− }
1263212610
1263312611 if (map_value_has_spin_lock(map)) {
1263412612 if (prog_type == BPF_PROG_TYPE_SOCKET_FILTER) {
@@ -12675,12 +12653,6 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 40 unmodified lines
1267512653 case BPF_MAP_TYPE_LRU_PERCPU_HASH:
1267612654 case BPF_MAP_TYPE_ARRAY_OF_MAPS:
1267712655 case BPF_MAP_TYPE_HASH_OF_MAPS:
12678− if (!is_preallocated_map(map)) {
12679− verbose(env,
12680− "Sleepable programs can only use preallocated maps\n");
12681− return -EINVAL;
12682− }
12683− break;
1268412656 case BPF_MAP_TYPE_RINGBUF:
1268512657 case BPF_MAP_TYPE_INODE_STORAGE:
1268612658 case BPF_MAP_TYPE_SK_STORAGE:
kernel/cgroup/rstat.c+48−0
@@ -3,6 +3,10 @@ · 2 unmodified lines
33
44 #include <linux/sched/cputime.h>
55
6+#include <linux/bpf.h>
7+#include <linux/btf.h>
8+#include <linux/btf_ids.h>
9+
610 static DEFINE_SPINLOCK(cgroup_rstat_lock);
711 static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock);
812
@@ -141,6 +145,31 @@ static struct cgroup *cgroup_rstat_cpu_pop_updated(struct cgroup *pos, · 132 unmodified lines
141145 return pos;
142146 }
143147
148+/*
149+ * A hook for bpf stat collectors to attach to and flush their stats.
150+ * Together with providing bpf kfuncs for cgroup_rstat_updated() and
151+ * cgroup_rstat_flush(), this enables a complete workflow where bpf progs that
152+ * collect cgroup stats can integrate with rstat for efficient flushing.
153+ *
154+ * A static noinline declaration here could cause the compiler to optimize away
155+ * the function. A global noinline declaration will keep the definition, but may
156+ * optimize away the callsite. Therefore, __weak is needed to ensure that the
157+ * call is still emitted, by telling the compiler that we don't know what the
158+ * function might eventually be.
159+ *
160+ * __diag_* below are needed to dismiss the missing prototype warning.
161+ */
162+__diag_push();
163+__diag_ignore_all("-Wmissing-prototypes",
164+ "kfuncs which will be used in BPF programs");
165+
166+__weak noinline void bpf_rstat_flush(struct cgroup *cgrp,
167+ struct cgroup *parent, int cpu)
168+{
169+}
170+
171+__diag_pop();
172+
144173 /* see cgroup_rstat_flush() */
145174 static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep)
146175 __releases(&cgroup_rstat_lock) __acquires(&cgroup_rstat_lock)
@@ -168,6 +197,7 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep) · 21 unmodified lines
168197 struct cgroup_subsys_state *css;
169198
170199 cgroup_base_stat_flush(pos, cpu);
200+ bpf_rstat_flush(pos, cgroup_parent(pos), cpu);
171201
172202 rcu_read_lock();
173203 list_for_each_entry_rcu(css, &pos->rstat_css_list,
@@ -501,3 +531,21 @@ void cgroup_base_stat_cputime_show(struct seq_file *seq) · 327 unmodified lines
501531 seq_printf(seq, "core_sched.force_idle_usec %llu\n", forceidle_time);
502532 #endif
503533 }
534+
535+/* Add bpf kfuncs for cgroup_rstat_updated() and cgroup_rstat_flush() */
536+BTF_SET8_START(bpf_rstat_kfunc_ids)
537+BTF_ID_FLAGS(func, cgroup_rstat_updated)
538+BTF_ID_FLAGS(func, cgroup_rstat_flush, KF_SLEEPABLE)
539+BTF_SET8_END(bpf_rstat_kfunc_ids)
540+
541+static const struct btf_kfunc_id_set bpf_rstat_kfunc_set = {
542+ .owner = THIS_MODULE,
543+ .set = &bpf_rstat_kfunc_ids,
544+};
545+
546+static int __init bpf_rstat_kfunc_init(void)
547+{
548+ return register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING,
549+ &bpf_rstat_kfunc_set);
550+}
551+late_initcall(bpf_rstat_kfunc_init);
net/core/filter.c+262−373
@@ -3010,7 +3010,7 @@ BPF_CALL_0(bpf_get_cgroup_classid_curr) · 3009 unmodified lines
30103010 return __task_get_classid(current);
30113011 }
30123012
3013−static const struct bpf_func_proto bpf_get_cgroup_classid_curr_proto = {
3013+const struct bpf_func_proto bpf_get_cgroup_classid_curr_proto = {
30143014 .func = bpf_get_cgroup_classid_curr,
30153015 .gpl_only = false,
30163016 .ret_type = RET_INTEGER,
@@ -4489,7 +4489,8 @@ BPF_CALL_4(bpf_skb_get_tunnel_key, struct sk_buff *, skb, struct bpf_tunnel_key · 1472 unmodified lines
44894489 void *to_orig = to;
44904490 int err;
44914491
4492− if (unlikely(!info || (flags & ~(BPF_F_TUNINFO_IPV6)))) {
4492+ if (unlikely(!info || (flags & ~(BPF_F_TUNINFO_IPV6 |
4493+ BPF_F_TUNINFO_FLAGS)))) {
44934494 err = -EINVAL;
44944495 goto err_clear;
44954496 }
@@ -4521,7 +4522,10 @@ set_compat: · 25 unmodified lines
45214522 to->tunnel_id = be64_to_cpu(info->key.tun_id);
45224523 to->tunnel_tos = info->key.tos;
45234524 to->tunnel_ttl = info->key.ttl;
4524− to->tunnel_ext = 0;
4525+ if (flags & BPF_F_TUNINFO_FLAGS)
4526+ to->tunnel_flags = info->key.tun_flags;
4527+ else
4528+ to->tunnel_ext = 0;
45254529
45264530 if (flags & BPF_F_TUNINFO_IPV6) {
45274531 memcpy(to->remote_ipv6, &info->key.u.ipv6.src,
@@ -5014,359 +5018,259 @@ static const struct bpf_func_proto bpf_get_socket_uid_proto = { · 486 unmodified lines
50145018 .arg1_type = ARG_PTR_TO_CTX,
50155019 };
50165020
5017−static int __bpf_setsockopt(struct sock *sk, int level, int optname,
5018− char *optval, int optlen)
5019−{
5020− char devname[IFNAMSIZ];
5021− int val, valbool;
5022− struct net *net;
5023− int ifindex;
5024− int ret = 0;
5025−
5026− if (!sk_fullsock(sk))
5021+static int sol_socket_sockopt(struct sock *sk, int optname,
5022+ char *optval, int *optlen,
5023+ bool getopt)
5024+{
5025+ switch (optname) {
5026+ case SO_REUSEADDR:
5027+ case SO_SNDBUF:
5028+ case SO_RCVBUF:
5029+ case SO_KEEPALIVE:
5030+ case SO_PRIORITY:
5031+ case SO_REUSEPORT:
5032+ case SO_RCVLOWAT:
5033+ case SO_MARK:
5034+ case SO_MAX_PACING_RATE:
5035+ case SO_BINDTOIFINDEX:
5036+ case SO_TXREHASH:
5037+ if (*optlen != sizeof(int))
5038+ return -EINVAL;
5039+ break;
5040+ case SO_BINDTODEVICE:
5041+ break;
5042+ default:
50275043 return -EINVAL;
5044+ }
50285045
5029− if (level == SOL_SOCKET) {
5030− if (optlen != sizeof(int) && optname != SO_BINDTODEVICE)
5046+ if (getopt) {
5047+ if (optname == SO_BINDTODEVICE)
50315048 return -EINVAL;
5032− val = *((int *)optval);
5033− valbool = val ? 1 : 0;
5034−
5035− /* Only some socketops are supported */
5036− switch (optname) {
5037− case SO_RCVBUF:
5038− val = min_t(u32, val, READ_ONCE(sysctl_rmem_max));
5039− val = min_t(int, val, INT_MAX / 2);
5040− sk->sk_userlocks |= SOCK_RCVBUF_LOCK;
5041− WRITE_ONCE(sk->sk_rcvbuf,
5042− max_t(int, val * 2, SOCK_MIN_RCVBUF));
5043− break;
5044− case SO_SNDBUF:
5045− val = min_t(u32, val, READ_ONCE(sysctl_wmem_max));
5046− val = min_t(int, val, INT_MAX / 2);
5047− sk->sk_userlocks |= SOCK_SNDBUF_LOCK;
5048− WRITE_ONCE(sk->sk_sndbuf,
5049− max_t(int, val * 2, SOCK_MIN_SNDBUF));
5050− break;
5051− case SO_MAX_PACING_RATE: /* 32bit version */
5052− if (val != ~0U)
5053− cmpxchg(&sk->sk_pacing_status,
5054− SK_PACING_NONE,
5055− SK_PACING_NEEDED);
5056− sk->sk_max_pacing_rate = (val == ~0U) ?
5057− ~0UL : (unsigned int)val;
5058− sk->sk_pacing_rate = min(sk->sk_pacing_rate,
5059− sk->sk_max_pacing_rate);
5060− break;
5061− case SO_PRIORITY:
5062− sk->sk_priority = val;
5063− break;
5064− case SO_RCVLOWAT:
5065− if (val < 0)
5066− val = INT_MAX;
5067− if (sk->sk_socket && sk->sk_socket->ops->set_rcvlowat)
5068− ret = sk->sk_socket->ops->set_rcvlowat(sk, val);
5069− else
5070− WRITE_ONCE(sk->sk_rcvlowat, val ? : 1);
5071− break;
5072− case SO_MARK:
5073− if (sk->sk_mark != val) {
5074− sk->sk_mark = val;
5075− sk_dst_reset(sk);
5076− }
5077− break;
5078− case SO_BINDTODEVICE:
5079− optlen = min_t(long, optlen, IFNAMSIZ - 1);
5080− strncpy(devname, optval, optlen);
5081− devname[optlen] = 0;
5049+ return sk_getsockopt(sk, SOL_SOCKET, optname,
5050+ KERNEL_SOCKPTR(optval),
5051+ KERNEL_SOCKPTR(optlen));
5052+ }
50825053
5083− ifindex = 0;
5084− if (devname[0] != '\0') {
5085− struct net_device *dev;
5054+ return sk_setsockopt(sk, SOL_SOCKET, optname,
5055+ KERNEL_SOCKPTR(optval), *optlen);
5056+}
50865057
5087− ret = -ENODEV;
5058+static int bpf_sol_tcp_setsockopt(struct sock *sk, int optname,
5059+ char *optval, int optlen)
5060+{
5061+ struct tcp_sock *tp = tcp_sk(sk);
5062+ unsigned long timeout;
5063+ int val;
50885064
5089− net = sock_net(sk);
5090− dev = dev_get_by_name(net, devname);
5091− if (!dev)
5092− break;
5093− ifindex = dev->ifindex;
5094− dev_put(dev);
5095− }
5096− fallthrough;
5097− case SO_BINDTOIFINDEX:
5098− if (optname == SO_BINDTOIFINDEX)
5099− ifindex = val;
5100− ret = sock_bindtoindex(sk, ifindex, false);
5101− break;
5102− case SO_KEEPALIVE:
5103− if (sk->sk_prot->keepalive)
5104− sk->sk_prot->keepalive(sk, valbool);
5105− sock_valbool_flag(sk, SOCK_KEEPOPEN, valbool);
5106− break;
5107− case SO_REUSEPORT:
5108− sk->sk_reuseport = valbool;
5109− break;
5110− case SO_TXREHASH:
5111− if (val < -1 || val > 1) {
5112− ret = -EINVAL;
5113− break;
5114− }
5115− sk->sk_txrehash = (u8)val;
5116− break;
5117− default:
5118− ret = -EINVAL;
5119− }
5120−#ifdef CONFIG_INET
5121− } else if (level == SOL_IP) {
5122− if (optlen != sizeof(int) || sk->sk_family != AF_INET)
5123− return -EINVAL;
5065+ if (optlen != sizeof(int))
5066+ return -EINVAL;
51245067
5125− val = *((int *)optval);
5126− /* Only some options are supported */
5127− switch (optname) {
5128− case IP_TOS:
5129− if (val < -1 || val > 0xff) {
5130− ret = -EINVAL;
5131− } else {
5132− struct inet_sock *inet = inet_sk(sk);
5068+ val = *(int *)optval;
51335069
5134− if (val == -1)
5135− val = 0;
5136− inet->tos = val;
5137− }
5138− break;
5139− default:
5140− ret = -EINVAL;
5141− }
5142−#if IS_ENABLED(CONFIG_IPV6)
5143− } else if (level == SOL_IPV6) {
5144− if (optlen != sizeof(int) || sk->sk_family != AF_INET6)
5070+ /* Only some options are supported */
5071+ switch (optname) {
5072+ case TCP_BPF_IW:
5073+ if (val <= 0 || tp->data_segs_out > tp->syn_data)
5074+ return -EINVAL;
5075+ tcp_snd_cwnd_set(tp, val);
5076+ break;
5077+ case TCP_BPF_SNDCWND_CLAMP:
5078+ if (val <= 0)
5079+ return -EINVAL;
5080+ tp->snd_cwnd_clamp = val;
5081+ tp->snd_ssthresh = val;
5082+ break;
5083+ case TCP_BPF_DELACK_MAX:
5084+ timeout = usecs_to_jiffies(val);
5085+ if (timeout > TCP_DELACK_MAX ||
5086+ timeout < TCP_TIMEOUT_MIN)
5087+ return -EINVAL;
5088+ inet_csk(sk)->icsk_delack_max = timeout;
5089+ break;
5090+ case TCP_BPF_RTO_MIN:
5091+ timeout = usecs_to_jiffies(val);
5092+ if (timeout > TCP_RTO_MIN ||
5093+ timeout < TCP_TIMEOUT_MIN)
51455094 return -EINVAL;
5095+ inet_csk(sk)->icsk_rto_min = timeout;
5096+ break;
5097+ default:
5098+ return -EINVAL;
5099+ }
51465100
5147− val = *((int *)optval);
5148− /* Only some options are supported */
5149− switch (optname) {
5150− case IPV6_TCLASS:
5151− if (val < -1 || val > 0xff) {
5152− ret = -EINVAL;
5153− } else {
5154− struct ipv6_pinfo *np = inet6_sk(sk);
5101+ return 0;
5102+}
51555103
5156− if (val == -1)
5157− val = 0;
5158− np->tclass = val;
5159− }
5160− break;
5161− default:
5162− ret = -EINVAL;
5163− }
5164−#endif
5165− } else if (level == SOL_TCP &&
5166− sk->sk_prot->setsockopt == tcp_setsockopt) {
5167− if (optname == TCP_CONGESTION) {
5168− char name[TCP_CA_NAME_MAX];
5104+static int sol_tcp_sockopt(struct sock *sk, int optname,
5105+ char *optval, int *optlen,
5106+ bool getopt)
5107+{
5108+ if (sk->sk_prot->setsockopt != tcp_setsockopt)
5109+ return -EINVAL;
51695110
5170− strncpy(name, optval, min_t(long, optlen,
5171− TCP_CA_NAME_MAX-1));
5172− name[TCP_CA_NAME_MAX-1] = 0;
5173− ret = tcp_set_congestion_control(sk, name, false, true);
5174− } else {
5175− struct inet_connection_sock *icsk = inet_csk(sk);
5111+ switch (optname) {
5112+ case TCP_NODELAY:
5113+ case TCP_MAXSEG:
5114+ case TCP_KEEPIDLE:
5115+ case TCP_KEEPINTVL:
5116+ case TCP_KEEPCNT:
5117+ case TCP_SYNCNT:
5118+ case TCP_WINDOW_CLAMP:
5119+ case TCP_THIN_LINEAR_TIMEOUTS:
5120+ case TCP_USER_TIMEOUT:
5121+ case TCP_NOTSENT_LOWAT:
5122+ case TCP_SAVE_SYN:
5123+ if (*optlen != sizeof(int))
5124+ return -EINVAL;
5125+ break;
5126+ case TCP_CONGESTION:
5127+ if (*optlen < 2)
5128+ return -EINVAL;
5129+ break;
5130+ case TCP_SAVED_SYN:
5131+ if (*optlen < 1)
5132+ return -EINVAL;
5133+ break;
5134+ default:
5135+ if (getopt)
5136+ return -EINVAL;
5137+ return bpf_sol_tcp_setsockopt(sk, optname, optval, *optlen);
5138+ }
5139+
5140+ if (getopt) {
5141+ if (optname == TCP_SAVED_SYN) {
51765142 struct tcp_sock *tp = tcp_sk(sk);
5177− unsigned long timeout;
51785143
5179− if (optlen != sizeof(int))
5144+ if (!tp->saved_syn ||
5145+ *optlen > tcp_saved_syn_len(tp->saved_syn))
51805146 return -EINVAL;
5147+ memcpy(optval, tp->saved_syn->data, *optlen);
5148+ /* It cannot free tp->saved_syn here because it
5149+ * does not know if the user space still needs it.
5150+ */
5151+ return 0;
5152+ }
51815153
5182− val = *((int *)optval);
5183− /* Only some options are supported */
5184− switch (optname) {
5185− case TCP_BPF_IW:
5186− if (val <= 0 || tp->data_segs_out > tp->syn_data)
5187− ret = -EINVAL;
5188− else
5189− tcp_snd_cwnd_set(tp, val);
5190− break;
5191− case TCP_BPF_SNDCWND_CLAMP:
5192− if (val <= 0) {
5193− ret = -EINVAL;
5194− } else {
5195− tp->snd_cwnd_clamp = val;
5196− tp->snd_ssthresh = val;
5197− }
5198− break;
5199− case TCP_BPF_DELACK_MAX:
5200− timeout = usecs_to_jiffies(val);
5201− if (timeout > TCP_DELACK_MAX ||
5202− timeout < TCP_TIMEOUT_MIN)
5203− return -EINVAL;
5204− inet_csk(sk)->icsk_delack_max = timeout;
5205− break;
5206− case TCP_BPF_RTO_MIN:
5207− timeout = usecs_to_jiffies(val);
5208− if (timeout > TCP_RTO_MIN ||
5209− timeout < TCP_TIMEOUT_MIN)
5210− return -EINVAL;
5211− inet_csk(sk)->icsk_rto_min = timeout;
5212− break;
5213− case TCP_SAVE_SYN:
5214− if (val < 0 || val > 1)
5215− ret = -EINVAL;
5216− else
5217− tp->save_syn = val;
5218− break;
5219− case TCP_KEEPIDLE:
5220− ret = tcp_sock_set_keepidle_locked(sk, val);
5221− break;
5222− case TCP_KEEPINTVL:
5223− if (val < 1 || val > MAX_TCP_KEEPINTVL)
5224− ret = -EINVAL;
5225− else
5226− tp->keepalive_intvl = val * HZ;
5227− break;
5228− case TCP_KEEPCNT:
5229− if (val < 1 || val > MAX_TCP_KEEPCNT)
5230− ret = -EINVAL;
5231− else
5232− tp->keepalive_probes = val;
5233− break;
5234− case TCP_SYNCNT:
5235− if (val < 1 || val > MAX_TCP_SYNCNT)
5236− ret = -EINVAL;
5237− else
5238− icsk->icsk_syn_retries = val;
5239− break;
5240− case TCP_USER_TIMEOUT:
5241− if (val < 0)
5242− ret = -EINVAL;
5243− else
5244− icsk->icsk_user_timeout = val;
5245− break;
5246− case TCP_NOTSENT_LOWAT:
5247− tp->notsent_lowat = val;
5248− sk->sk_write_space(sk);
5249− break;
5250− case TCP_WINDOW_CLAMP:
5251− ret = tcp_set_window_clamp(sk, val);
5252− break;
5253− default:
5254− ret = -EINVAL;
5255− }
5154+ if (optname == TCP_CONGESTION) {
5155+ if (!inet_csk(sk)->icsk_ca_ops)
5156+ return -EINVAL;
5157+ /* BPF expects NULL-terminated tcp-cc string */
5158+ optval[--(*optlen)] = '\0';
52565159 }
5257−#endif
5258− } else {
5259− ret = -EINVAL;
5160+
5161+ return do_tcp_getsockopt(sk, SOL_TCP, optname,
5162+ KERNEL_SOCKPTR(optval),
5163+ KERNEL_SOCKPTR(optlen));
52605164 }
5261− return ret;
5165+
5166+ return do_tcp_setsockopt(sk, SOL_TCP, optname,
5167+ KERNEL_SOCKPTR(optval), *optlen);
52625168 }
52635169
5264−static int _bpf_setsockopt(struct sock *sk, int level, int optname,
5265− char *optval, int optlen)
5170+static int sol_ip_sockopt(struct sock *sk, int optname,
5171+ char *optval, int *optlen,
5172+ bool getopt)
52665173 {
5267− if (sk_fullsock(sk))
5268− sock_owned_by_me(sk);
5269− return __bpf_setsockopt(sk, level, optname, optval, optlen);
5174+ if (sk->sk_family != AF_INET)
5175+ return -EINVAL;
5176+
5177+ switch (optname) {
5178+ case IP_TOS:
5179+ if (*optlen != sizeof(int))
5180+ return -EINVAL;
5181+ break;
5182+ default:
5183+ return -EINVAL;
5184+ }
5185+
5186+ if (getopt)
5187+ return do_ip_getsockopt(sk, SOL_IP, optname,
5188+ KERNEL_SOCKPTR(optval),
5189+ KERNEL_SOCKPTR(optlen));
5190+
5191+ return do_ip_setsockopt(sk, SOL_IP, optname,
5192+ KERNEL_SOCKPTR(optval), *optlen);
52705193 }
52715194
5272−static int __bpf_getsockopt(struct sock *sk, int level, int optname,
5273− char *optval, int optlen)
5195+static int sol_ipv6_sockopt(struct sock *sk, int optname,
5196+ char *optval, int *optlen,
5197+ bool getopt)
52745198 {
5275− if (!sk_fullsock(sk))
5276− goto err_clear;
5199+ if (sk->sk_family != AF_INET6)
5200+ return -EINVAL;
52775201
5278− if (level == SOL_SOCKET) {
5279− if (optlen != sizeof(int))
5280− goto err_clear;
5202+ switch (optname) {
5203+ case IPV6_TCLASS:
5204+ case IPV6_AUTOFLOWLABEL:
5205+ if (*optlen != sizeof(int))
5206+ return -EINVAL;
5207+ break;
5208+ default:
5209+ return -EINVAL;
5210+ }
52815211
5282− switch (optname) {
5283− case SO_RCVBUF:
5284− *((int *)optval) = sk->sk_rcvbuf;
5285− break;
5286− case SO_SNDBUF:
5287− *((int *)optval) = sk->sk_sndbuf;
5288− break;
5289− case SO_MARK:
5290− *((int *)optval) = sk->sk_mark;
5291− break;
5292− case SO_PRIORITY:
5293− *((int *)optval) = sk->sk_priority;
5294− break;
5295− case SO_BINDTOIFINDEX:
5296− *((int *)optval) = sk->sk_bound_dev_if;
5297− break;
5298− case SO_REUSEPORT:
5299− *((int *)optval) = sk->sk_reuseport;
5300− break;
5301− case SO_TXREHASH:
5302− *((int *)optval) = sk->sk_txrehash;
5303− break;
5304− default:
5305− goto err_clear;
5306− }
5307−#ifdef CONFIG_INET
5308− } else if (level == SOL_TCP && sk->sk_prot->getsockopt == tcp_getsockopt) {
5309− struct inet_connection_sock *icsk;
5310− struct tcp_sock *tp;
5212+ if (getopt)
5213+ return ipv6_bpf_stub->ipv6_getsockopt(sk, SOL_IPV6, optname,
5214+ KERNEL_SOCKPTR(optval),
5215+ KERNEL_SOCKPTR(optlen));
53115216
5312− switch (optname) {
5313− case TCP_CONGESTION:
5314− icsk = inet_csk(sk);
5217+ return ipv6_bpf_stub->ipv6_setsockopt(sk, SOL_IPV6, optname,
5218+ KERNEL_SOCKPTR(optval), *optlen);
5219+}
53155220
5316− if (!icsk->icsk_ca_ops || optlen <= 1)
5317− goto err_clear;
5318− strncpy(optval, icsk->icsk_ca_ops->name, optlen);
5319− optval[optlen - 1] = 0;
5320− break;
5321− case TCP_SAVED_SYN:
5322− tp = tcp_sk(sk);
5221+static int __bpf_setsockopt(struct sock *sk, int level, int optname,
5222+ char *optval, int optlen)
5223+{
5224+ if (!sk_fullsock(sk))
5225+ return -EINVAL;
53235226
5324− if (optlen <= 0 || !tp->saved_syn ||
5325− optlen > tcp_saved_syn_len(tp->saved_syn))
5326− goto err_clear;
5327− memcpy(optval, tp->saved_syn->data, optlen);
5328− break;
5329− default:
5330− goto err_clear;
5331− }
5332− } else if (level == SOL_IP) {
5333− struct inet_sock *inet = inet_sk(sk);
5227+ if (level == SOL_SOCKET)
5228+ return sol_socket_sockopt(sk, optname, optval, &optlen, false);
5229+ else if (IS_ENABLED(CONFIG_INET) && level == SOL_IP)
5230+ return sol_ip_sockopt(sk, optname, optval, &optlen, false);
5231+ else if (IS_ENABLED(CONFIG_IPV6) && level == SOL_IPV6)
5232+ return sol_ipv6_sockopt(sk, optname, optval, &optlen, false);
5233+ else if (IS_ENABLED(CONFIG_INET) && level == SOL_TCP)
5234+ return sol_tcp_sockopt(sk, optname, optval, &optlen, false);
53345235
5335− if (optlen != sizeof(int) || sk->sk_family != AF_INET)
5336− goto err_clear;
5236+ return -EINVAL;
5237+}
53375238
5338− /* Only some options are supported */
5339− switch (optname) {
5340− case IP_TOS:
5341− *((int *)optval) = (int)inet->tos;
5342− break;
5343− default:
5344− goto err_clear;
5345− }
5346−#if IS_ENABLED(CONFIG_IPV6)
5347− } else if (level == SOL_IPV6) {
5348− struct ipv6_pinfo *np = inet6_sk(sk);
5239+static int _bpf_setsockopt(struct sock *sk, int level, int optname,
5240+ char *optval, int optlen)
5241+{
5242+ if (sk_fullsock(sk))
5243+ sock_owned_by_me(sk);
5244+ return __bpf_setsockopt(sk, level, optname, optval, optlen);
5245+}
53495246
5350− if (optlen != sizeof(int) || sk->sk_family != AF_INET6)
5351− goto err_clear;
5247+static int __bpf_getsockopt(struct sock *sk, int level, int optname,
5248+ char *optval, int optlen)
5249+{
5250+ int err, saved_optlen = optlen;
53525251
5353− /* Only some options are supported */
5354− switch (optname) {
5355− case IPV6_TCLASS:
5356− *((int *)optval) = (int)np->tclass;
5357− break;
5358− default:
5359− goto err_clear;
5360− }
5361−#endif
5362−#endif
5363− } else {
5364− goto err_clear;
5252+ if (!sk_fullsock(sk)) {
5253+ err = -EINVAL;
5254+ goto done;
53655255 }
5366− return 0;
5367−err_clear:
5368− memset(optval, 0, optlen);
5369− return -EINVAL;
5256+
5257+ if (level == SOL_SOCKET)
5258+ err = sol_socket_sockopt(sk, optname, optval, &optlen, true);
5259+ else if (IS_ENABLED(CONFIG_INET) && level == SOL_TCP)
5260+ err = sol_tcp_sockopt(sk, optname, optval, &optlen, true);
5261+ else if (IS_ENABLED(CONFIG_INET) && level == SOL_IP)
5262+ err = sol_ip_sockopt(sk, optname, optval, &optlen, true);
5263+ else if (IS_ENABLED(CONFIG_IPV6) && level == SOL_IPV6)
5264+ err = sol_ipv6_sockopt(sk, optname, optval, &optlen, true);
5265+ else
5266+ err = -EINVAL;
5267+
5268+done:
5269+ if (err)
5270+ optlen = 0;
5271+ if (optlen < saved_optlen)
5272+ memset(optval + optlen, 0, saved_optlen - optlen);
5273+ return err;
53705274 }
53715275
53725276 static int _bpf_getsockopt(struct sock *sk, int level, int optname,
@@ -7667,34 +7571,23 @@ const struct bpf_func_proto bpf_sk_storage_get_cg_sock_proto __weak; · 2294 unmodified lines
76677571 static const struct bpf_func_proto *
76687572 sock_filter_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
76697573 {
7574+ const struct bpf_func_proto *func_proto;
7575+
7576+ func_proto = cgroup_common_func_proto(func_id, prog);
7577+ if (func_proto)
7578+ return func_proto;
7579+
7580+ func_proto = cgroup_current_func_proto(func_id, prog);
7581+ if (func_proto)
7582+ return func_proto;
7583+
76707584 switch (func_id) {
7671− /* inet and inet6 sockets are created in a process
7672− * context so there is always a valid uid/gid
7673− */
7674− case BPF_FUNC_get_current_uid_gid:
7675− return &bpf_get_current_uid_gid_proto;
7676− case BPF_FUNC_get_local_storage:
7677− return &bpf_get_local_storage_proto;
76787585 case BPF_FUNC_get_socket_cookie:
76797586 return &bpf_get_socket_cookie_sock_proto;
76807587 case BPF_FUNC_get_netns_cookie:
76817588 return &bpf_get_netns_cookie_sock_proto;
76827589 case BPF_FUNC_perf_event_output:
76837590 return &bpf_event_output_data_proto;
7684− case BPF_FUNC_get_current_pid_tgid:
7685− return &bpf_get_current_pid_tgid_proto;
7686− case BPF_FUNC_get_current_comm:
7687− return &bpf_get_current_comm_proto;
7688−#ifdef CONFIG_CGROUPS
7689− case BPF_FUNC_get_current_cgroup_id:
7690− return &bpf_get_current_cgroup_id_proto;
7691− case BPF_FUNC_get_current_ancestor_cgroup_id:
7692− return &bpf_get_current_ancestor_cgroup_id_proto;
7693−#endif
7694−#ifdef CONFIG_CGROUP_NET_CLASSID
7695− case BPF_FUNC_get_cgroup_classid:
7696− return &bpf_get_cgroup_classid_curr_proto;
7697−#endif
76987591 case BPF_FUNC_sk_storage_get:
76997592 return &bpf_sk_storage_get_cg_sock_proto;
77007593 case BPF_FUNC_ktime_get_coarse_ns:
@@ -7707,12 +7600,17 @@ sock_filter_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 6 unmodified lines
77077600 static const struct bpf_func_proto *
77087601 sock_addr_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
77097602 {
7603+ const struct bpf_func_proto *func_proto;
7604+
7605+ func_proto = cgroup_common_func_proto(func_id, prog);
7606+ if (func_proto)
7607+ return func_proto;
7608+
7609+ func_proto = cgroup_current_func_proto(func_id, prog);
7610+ if (func_proto)
7611+ return func_proto;
7612+
77107613 switch (func_id) {
7711− /* inet and inet6 sockets are created in a process
7712− * context so there is always a valid uid/gid
7713− */
7714− case BPF_FUNC_get_current_uid_gid:
7715− return &bpf_get_current_uid_gid_proto;
77167614 case BPF_FUNC_bind:
77177615 switch (prog->expected_attach_type) {
77187616 case BPF_CGROUP_INET4_CONNECT:
@@ -7725,24 +7623,8 @@ sock_addr_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 6 unmodified lines
77257623 return &bpf_get_socket_cookie_sock_addr_proto;
77267624 case BPF_FUNC_get_netns_cookie:
77277625 return &bpf_get_netns_cookie_sock_addr_proto;
7728− case BPF_FUNC_get_local_storage:
7729− return &bpf_get_local_storage_proto;
77307626 case BPF_FUNC_perf_event_output:
77317627 return &bpf_event_output_data_proto;
7732− case BPF_FUNC_get_current_pid_tgid:
7733− return &bpf_get_current_pid_tgid_proto;
7734− case BPF_FUNC_get_current_comm:
7735− return &bpf_get_current_comm_proto;
7736−#ifdef CONFIG_CGROUPS
7737− case BPF_FUNC_get_current_cgroup_id:
7738− return &bpf_get_current_cgroup_id_proto;
7739− case BPF_FUNC_get_current_ancestor_cgroup_id:
7740− return &bpf_get_current_ancestor_cgroup_id_proto;
7741−#endif
7742−#ifdef CONFIG_CGROUP_NET_CLASSID
7743− case BPF_FUNC_get_cgroup_classid:
7744− return &bpf_get_cgroup_classid_curr_proto;
7745−#endif
77467628 #ifdef CONFIG_INET
77477629 case BPF_FUNC_sk_lookup_tcp:
77487630 return &bpf_sock_addr_sk_lookup_tcp_proto;
@@ -7823,9 +7705,13 @@ const struct bpf_func_proto bpf_sk_storage_delete_proto __weak; · 74 unmodified lines
78237705 static const struct bpf_func_proto *
78247706 cg_skb_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
78257707 {
7708+ const struct bpf_func_proto *func_proto;
7709+
7710+ func_proto = cgroup_common_func_proto(func_id, prog);
7711+ if (func_proto)
7712+ return func_proto;
7713+
78267714 switch (func_id) {
7827− case BPF_FUNC_get_local_storage:
7828− return &bpf_get_local_storage_proto;
78297715 case BPF_FUNC_sk_fullsock:
78307716 return &bpf_sk_fullsock_proto;
78317717 case BPF_FUNC_sk_storage_get:
@@ -8065,6 +7951,12 @@ const struct bpf_func_proto bpf_sock_hash_update_proto __weak; · 233 unmodified lines
80657951 static const struct bpf_func_proto *
80667952 sock_ops_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
80677953 {
7954+ const struct bpf_func_proto *func_proto;
7955+
7956+ func_proto = cgroup_common_func_proto(func_id, prog);
7957+ if (func_proto)
7958+ return func_proto;
7959+
80687960 switch (func_id) {
80697961 case BPF_FUNC_setsockopt:
80707962 return &bpf_sock_ops_setsockopt_proto;
@@ -8078,8 +7970,6 @@ sock_ops_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 7 unmodified lines
80787970 return &bpf_sock_hash_update_proto;
80797971 case BPF_FUNC_get_socket_cookie:
80807972 return &bpf_get_socket_cookie_sock_ops_proto;
8081− case BPF_FUNC_get_local_storage:
8082− return &bpf_get_local_storage_proto;
80837973 case BPF_FUNC_perf_event_output:
80847974 return &bpf_event_output_data_proto;
80857975 case BPF_FUNC_sk_storage_get:
@@ -10812,14 +10702,13 @@ int sk_detach_filter(struct sock *sk) · 2726 unmodified lines
1081210702 }
1081310703 EXPORT_SYMBOL_GPL(sk_detach_filter);
1081410704
10815−int sk_get_filter(struct sock *sk, struct sock_filter __user *ubuf,
10816− unsigned int len)
10705+int sk_get_filter(struct sock *sk, sockptr_t optval, unsigned int len)
1081710706 {
1081810707 struct sock_fprog_kern *fprog;
1081910708 struct sk_filter *filter;
1082010709 int ret = 0;
1082110710
10822− lock_sock(sk);
10711+ sockopt_lock_sock(sk);
1082310712 filter = rcu_dereference_protected(sk->sk_filter,
1082410713 lockdep_sock_is_held(sk));
1082510714 if (!filter)
@@ -10844,7 +10733,7 @@ int sk_get_filter(struct sock *sk, struct sock_filter __user *ubuf, · 18 unmodified lines
1084410733 goto out;
1084510734
1084610735 ret = -EFAULT;
10847− if (copy_to_user(ubuf, fprog->filter, bpf_classic_proglen(fprog)))
10736+ if (copy_to_sockptr(optval, fprog->filter, bpf_classic_proglen(fprog)))
1084810737 goto out;
1084910738
1085010739 /* Instead of bytes, the API requests to return the number
@@ -10852,7 +10741,7 @@ int sk_get_filter(struct sock *sk, struct sock_filter __user *ubuf, · 1 unmodified line
1085210741 */
1085310742 ret = fprog->len;
1085410743 out:
10855− release_sock(sk);
10744+ sockopt_release_sock(sk);
1085610745 return ret;
1085710746 }
1085810747
net/core/flow_dissector.c+10−6
@@ -866,8 +866,8 @@ static void __skb_flow_bpf_to_target(const struct bpf_flow_keys *flow_keys, · 865 unmodified lines
866866 }
867867 }
868868
869−bool bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
870− __be16 proto, int nhoff, int hlen, unsigned int flags)
869+u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
870+ __be16 proto, int nhoff, int hlen, unsigned int flags)
871871 {
872872 struct bpf_flow_keys *flow_keys = ctx->flow_keys;
873873 u32 result;
@@ -892,7 +892,7 @@ bool bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx, · 18 unmodified lines
892892 flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
893893 flow_keys->nhoff, hlen);
894894
895− return result == BPF_OK;
895+ return result;
896896 }
897897
898898 static bool is_pppoe_ses_hdr_valid(const struct pppoe_hdr *hdr)
@@ -1008,6 +1008,7 @@ bool __skb_flow_dissect(const struct net *net, · 109 unmodified lines
10081008 };
10091009 __be16 n_proto = proto;
10101010 struct bpf_prog *prog;
1011+ u32 result;
10111012
10121013 if (skb) {
10131014 ctx.skb = skb;
@@ -1019,13 +1020,16 @@ bool __skb_flow_dissect(const struct net *net, · 5 unmodified lines
10191020 }
10201021
10211022 prog = READ_ONCE(run_array->items[0].prog);
1022− ret = bpf_flow_dissect(prog, &ctx, n_proto, nhoff,
1023− hlen, flags);
1023+ result = bpf_flow_dissect(prog, &ctx, n_proto, nhoff,
1024+ hlen, flags);
1025+ if (result == BPF_FLOW_DISSECTOR_CONTINUE)
1026+ goto dissect_continue;
10241027 __skb_flow_bpf_to_target(&flow_keys, flow_dissector,
10251028 target_container);
10261029 rcu_read_unlock();
1027− return ret;
1030+ return result == BPF_OK;
10281031 }
1032+dissect_continue:
10291033 rcu_read_unlock();
10301034 }
10311035
net/core/sock.c+94−40
@@ -703,15 +703,17 @@ static int sock_setbindtodevice(struct sock *sk, sockptr_t optval, int optlen) · 702 unmodified lines
703703 goto out;
704704 }
705705
706− return sock_bindtoindex(sk, index, true);
706+ sockopt_lock_sock(sk);
707+ ret = sock_bindtoindex_locked(sk, index);
708+ sockopt_release_sock(sk);
707709 out:
708710 #endif
709711
710712 return ret;
711713 }
712714
713−static int sock_getbindtodevice(struct sock *sk, char __user *optval,
714− int __user *optlen, int len)
715+static int sock_getbindtodevice(struct sock *sk, sockptr_t optval,
716+ sockptr_t optlen, int len)
715717 {
716718 int ret = -ENOPROTOOPT;
717719 #ifdef CONFIG_NETDEVICES
@@ -735,12 +737,12 @@ static int sock_getbindtodevice(struct sock *sk, char __user *optval, · 17 unmodified lines
735737 len = strlen(devname) + 1;
736738
737739 ret = -EFAULT;
738− if (copy_to_user(optval, devname, len))
740+ if (copy_to_sockptr(optval, devname, len))
739741 goto out;
740742
741743 zero:
742744 ret = -EFAULT;
743− if (put_user(len, optlen))
745+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
744746 goto out;
745747
746748 ret = 0;
@@ -1036,17 +1038,51 @@ static int sock_reserve_memory(struct sock *sk, int bytes) · 289 unmodified lines
10361038 return 0;
10371039 }
10381040
1041+void sockopt_lock_sock(struct sock *sk)
1042+{
1043+ /* When current->bpf_ctx is set, the setsockopt is called from
1044+ * a bpf prog. bpf has ensured the sk lock has been
1045+ * acquired before calling setsockopt().
1046+ */
1047+ if (has_current_bpf_ctx())
1048+ return;
1049+
1050+ lock_sock(sk);
1051+}
1052+EXPORT_SYMBOL(sockopt_lock_sock);
1053+
1054+void sockopt_release_sock(struct sock *sk)
1055+{
1056+ if (has_current_bpf_ctx())
1057+ return;
1058+
1059+ release_sock(sk);
1060+}
1061+EXPORT_SYMBOL(sockopt_release_sock);
1062+
1063+bool sockopt_ns_capable(struct user_namespace *ns, int cap)
1064+{
1065+ return has_current_bpf_ctx() || ns_capable(ns, cap);
1066+}
1067+EXPORT_SYMBOL(sockopt_ns_capable);
1068+
1069+bool sockopt_capable(int cap)
1070+{
1071+ return has_current_bpf_ctx() || capable(cap);
1072+}
1073+EXPORT_SYMBOL(sockopt_capable);
1074+
10391075 /*
10401076 * This is meant for all protocols to use and covers goings on
10411077 * at the socket level. Everything here is generic.
10421078 */
10431079
1044−int sock_setsockopt(struct socket *sock, int level, int optname,
1045− sockptr_t optval, unsigned int optlen)
1080+int sk_setsockopt(struct sock *sk, int level, int optname,
1081+ sockptr_t optval, unsigned int optlen)
10461082 {
10471083 struct so_timestamping timestamping;
1084+ struct socket *sock = sk->sk_socket;
10481085 struct sock_txtime sk_txtime;
1049− struct sock *sk = sock->sk;
10501086 int val;
10511087 int valbool;
10521088 struct linger ling;
@@ -1067,11 +1103,11 @@ int sock_setsockopt(struct socket *sock, int level, int optname, · 14 unmodified lines
10671103
10681104 valbool = val ? 1 : 0;
10691105
1070− lock_sock(sk);
1106+ sockopt_lock_sock(sk);
10711107
10721108 switch (optname) {
10731109 case SO_DEBUG:
1074− if (val && !capable(CAP_NET_ADMIN))
1110+ if (val && !sockopt_capable(CAP_NET_ADMIN))
10751111 ret = -EACCES;
10761112 else
10771113 sock_valbool_flag(sk, SOCK_DBG, valbool);
@@ -1115,7 +1151,7 @@ set_sndbuf: · 37 unmodified lines
11151151 break;
11161152
11171153 case SO_SNDBUFFORCE:
1118− if (!capable(CAP_NET_ADMIN)) {
1154+ if (!sockopt_capable(CAP_NET_ADMIN)) {
11191155 ret = -EPERM;
11201156 break;
11211157 }
@@ -1137,7 +1173,7 @@ set_sndbuf: · 15 unmodified lines
11371173 break;
11381174
11391175 case SO_RCVBUFFORCE:
1140− if (!capable(CAP_NET_ADMIN)) {
1176+ if (!sockopt_capable(CAP_NET_ADMIN)) {
11411177 ret = -EPERM;
11421178 break;
11431179 }
@@ -1164,8 +1200,8 @@ set_sndbuf: · 20 unmodified lines
11641200
11651201 case SO_PRIORITY:
11661202 if ((val >= 0 && val <= 6) ||
1167− ns_capable(sock_net(sk)->user_ns, CAP_NET_RAW) ||
1168− ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN))
1203+ sockopt_ns_capable(sock_net(sk)->user_ns, CAP_NET_RAW) ||
1204+ sockopt_ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN))
11691205 sk->sk_priority = val;
11701206 else
11711207 ret = -EPERM;
@@ -1228,7 +1264,7 @@ set_sndbuf: · 56 unmodified lines
12281264 case SO_RCVLOWAT:
12291265 if (val < 0)
12301266 val = INT_MAX;
1231− if (sock->ops->set_rcvlowat)
1267+ if (sock && sock->ops->set_rcvlowat)
12321268 ret = sock->ops->set_rcvlowat(sk, val);
12331269 else
12341270 WRITE_ONCE(sk->sk_rcvlowat, val ? : 1);
@@ -1310,8 +1346,8 @@ set_sndbuf: · 75 unmodified lines
13101346 clear_bit(SOCK_PASSSEC, &sock->flags);
13111347 break;
13121348 case SO_MARK:
1313− if (!ns_capable(sock_net(sk)->user_ns, CAP_NET_RAW) &&
1314− !ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN)) {
1349+ if (!sockopt_ns_capable(sock_net(sk)->user_ns, CAP_NET_RAW) &&
1350+ !sockopt_ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN)) {
13151351 ret = -EPERM;
13161352 break;
13171353 }
@@ -1319,8 +1355,8 @@ set_sndbuf: · 1 unmodified line
13191355 __sock_set_mark(sk, val);
13201356 break;
13211357 case SO_RCVMARK:
1322− if (!ns_capable(sock_net(sk)->user_ns, CAP_NET_RAW) &&
1323− !ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN)) {
1358+ if (!sockopt_ns_capable(sock_net(sk)->user_ns, CAP_NET_RAW) &&
1359+ !sockopt_ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN)) {
13241360 ret = -EPERM;
13251361 break;
13261362 }
@@ -1354,7 +1390,7 @@ set_sndbuf: · 27 unmodified lines
13541390 #ifdef CONFIG_NET_RX_BUSY_POLL
13551391 case SO_BUSY_POLL:
13561392 /* allow unprivileged users to decrease the value */
1357− if ((val > sk->sk_ll_usec) && !capable(CAP_NET_ADMIN))
1393+ if ((val > sk->sk_ll_usec) && !sockopt_capable(CAP_NET_ADMIN))
13581394 ret = -EPERM;
13591395 else {
13601396 if (val < 0)
@@ -1364,13 +1400,13 @@ set_sndbuf: · 3 unmodified lines
13641400 }
13651401 break;
13661402 case SO_PREFER_BUSY_POLL:
1367− if (valbool && !capable(CAP_NET_ADMIN))
1403+ if (valbool && !sockopt_capable(CAP_NET_ADMIN))
13681404 ret = -EPERM;
13691405 else
13701406 WRITE_ONCE(sk->sk_prefer_busy_poll, valbool);
13711407 break;
13721408 case SO_BUSY_POLL_BUDGET:
1373− if (val > READ_ONCE(sk->sk_busy_poll_budget) && !capable(CAP_NET_ADMIN)) {
1409+ if (val > READ_ONCE(sk->sk_busy_poll_budget) && !sockopt_capable(CAP_NET_ADMIN)) {
13741410 ret = -EPERM;
13751411 } else {
13761412 if (val < 0 || val > U16_MAX)
@@ -1441,7 +1477,7 @@ set_sndbuf: · 64 unmodified lines
14411477 * scheduler has enough safe guards.
14421478 */
14431479 if (sk_txtime.clockid != CLOCK_MONOTONIC &&
1444− !ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN)) {
1480+ !sockopt_ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN)) {
14451481 ret = -EPERM;
14461482 break;
14471483 }
@@ -1496,9 +1532,16 @@ set_sndbuf: · 48 unmodified lines
14961532 ret = -ENOPROTOOPT;
14971533 break;
14981534 }
1499− release_sock(sk);
1535+ sockopt_release_sock(sk);
15001536 return ret;
15011537 }
1538+
1539+int sock_setsockopt(struct socket *sock, int level, int optname,
1540+ sockptr_t optval, unsigned int optlen)
1541+{
1542+ return sk_setsockopt(sock->sk, level, optname,
1543+ optval, optlen);
1544+}
15021545 EXPORT_SYMBOL(sock_setsockopt);
15031546
15041547 static const struct cred *sk_get_peer_cred(struct sock *sk)
@@ -1525,22 +1568,25 @@ static void cred_to_ucred(struct pid *pid, const struct cred *cred, · 20 unmodified lines
15251568 }
15261569 }
15271570
1528−static int groups_to_user(gid_t __user *dst, const struct group_info *src)
1571+static int groups_to_user(sockptr_t dst, const struct group_info *src)
15291572 {
15301573 struct user_namespace *user_ns = current_user_ns();
15311574 int i;
15321575
1533− for (i = 0; i < src->ngroups; i++)
1534− if (put_user(from_kgid_munged(user_ns, src->gid[i]), dst + i))
1576+ for (i = 0; i < src->ngroups; i++) {
1577+ gid_t gid = from_kgid_munged(user_ns, src->gid[i]);
1578+
1579+ if (copy_to_sockptr_offset(dst, i * sizeof(gid), &gid, sizeof(gid)))
15351580 return -EFAULT;
1581+ }
15361582
15371583 return 0;
15381584 }
15391585
1540−int sock_getsockopt(struct socket *sock, int level, int optname,
1541− char __user *optval, int __user *optlen)
1586+int sk_getsockopt(struct sock *sk, int level, int optname,
1587+ sockptr_t optval, sockptr_t optlen)
15421588 {
1543− struct sock *sk = sock->sk;
1589+ struct socket *sock = sk->sk_socket;
15441590
15451591 union {
15461592 int val;
@@ -1557,7 +1603,7 @@ int sock_getsockopt(struct socket *sock, int level, int optname, · 10 unmodified lines
15571603 int lv = sizeof(int);
15581604 int len;
15591605
1560− if (get_user(len, optlen))
1606+ if (copy_from_sockptr(&len, optlen, sizeof(int)))
15611607 return -EFAULT;
15621608 if (len < 0)
15631609 return -EINVAL;
@@ -1692,7 +1738,7 @@ int sock_getsockopt(struct socket *sock, int level, int optname, · 128 unmodified lines
16921738 cred_to_ucred(sk->sk_peer_pid, sk->sk_peer_cred, &peercred);
16931739 spin_unlock(&sk->sk_peer_lock);
16941740
1695− if (copy_to_user(optval, &peercred, len))
1741+ if (copy_to_sockptr(optval, &peercred, len))
16961742 return -EFAULT;
16971743 goto lenout;
16981744 }
@@ -1710,11 +1756,11 @@ int sock_getsockopt(struct socket *sock, int level, int optname, · 11 unmodified lines
17101756 if (len < n * sizeof(gid_t)) {
17111757 len = n * sizeof(gid_t);
17121758 put_cred(cred);
1713− return put_user(len, optlen) ? -EFAULT : -ERANGE;
1759+ return copy_to_sockptr(optlen, &len, sizeof(int)) ? -EFAULT : -ERANGE;
17141760 }
17151761 len = n * sizeof(gid_t);
17161762
1717− ret = groups_to_user((gid_t __user *)optval, cred->group_info);
1763+ ret = groups_to_user(optval, cred->group_info);
17181764 put_cred(cred);
17191765 if (ret)
17201766 return ret;
@@ -1730,7 +1776,7 @@ int sock_getsockopt(struct socket *sock, int level, int optname, · 9 unmodified lines
17301776 return -ENOTCONN;
17311777 if (lv < len)
17321778 return -EINVAL;
1733− if (copy_to_user(optval, address, len))
1779+ if (copy_to_sockptr(optval, address, len))
17341780 return -EFAULT;
17351781 goto lenout;
17361782 }
@@ -1747,7 +1793,7 @@ int sock_getsockopt(struct socket *sock, int level, int optname, · 10 unmodified lines
17471793 break;
17481794
17491795 case SO_PEERSEC:
1750− return security_socket_getpeersec_stream(sock, optval, optlen, len);
1796+ return security_socket_getpeersec_stream(sock, optval.user, optlen.user, len);
17511797
17521798 case SO_MARK:
17531799 v.val = sk->sk_mark;
@@ -1779,7 +1825,7 @@ int sock_getsockopt(struct socket *sock, int level, int optname, · 25 unmodified lines
17791825 return sock_getbindtodevice(sk, optval, optlen, len);
17801826
17811827 case SO_GET_FILTER:
1782− len = sk_get_filter(sk, (struct sock_filter __user *)optval, len);
1828+ len = sk_get_filter(sk, optval, len);
17831829 if (len < 0)
17841830 return len;
17851831
@@ -1827,7 +1873,7 @@ int sock_getsockopt(struct socket *sock, int level, int optname, · 41 unmodified lines
18271873 sk_get_meminfo(sk, meminfo);
18281874
18291875 len = min_t(unsigned int, len, sizeof(meminfo));
1830− if (copy_to_user(optval, &meminfo, len))
1876+ if (copy_to_sockptr(optval, &meminfo, len))
18311877 return -EFAULT;
18321878
18331879 goto lenout;
@@ -1896,14 +1942,22 @@ int sock_getsockopt(struct socket *sock, int level, int optname, · 62 unmodified lines
18961942
18971943 if (len > lv)
18981944 len = lv;
1899− if (copy_to_user(optval, &v, len))
1945+ if (copy_to_sockptr(optval, &v, len))
19001946 return -EFAULT;
19011947 lenout:
1902− if (put_user(len, optlen))
1948+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
19031949 return -EFAULT;
19041950 return 0;
19051951 }
19061952
1953+int sock_getsockopt(struct socket *sock, int level, int optname,
1954+ char __user *optval, int __user *optlen)
1955+{
1956+ return sk_getsockopt(sock->sk, level, optname,
1957+ USER_SOCKPTR(optval),
1958+ USER_SOCKPTR(optlen));
1959+}
1960+
19071961 /*
19081962 * Initialize an sk_lock.
19091963 *
net/ipv4/igmp.c+13−9
@@ -2529,11 +2529,10 @@ done: · 2528 unmodified lines
25292529 err = ip_mc_leave_group(sk, &imr);
25302530 return err;
25312531 }
2532−
25332532 int ip_mc_msfget(struct sock *sk, struct ip_msfilter *msf,
2534− struct ip_msfilter __user *optval, int __user *optlen)
2533+ sockptr_t optval, sockptr_t optlen)
25352534 {
2536− int err, len, count, copycount;
2535+ int err, len, count, copycount, msf_size;
25372536 struct ip_mreqn imr;
25382537 __be32 addr = msf->imsf_multiaddr;
25392538 struct ip_mc_socklist *pmc;
@@ -2575,12 +2574,15 @@ int ip_mc_msfget(struct sock *sk, struct ip_msfilter *msf, · 35 unmodified lines
25752574 copycount = count < msf->imsf_numsrc ? count : msf->imsf_numsrc;
25762575 len = flex_array_size(psl, sl_addr, copycount);
25772576 msf->imsf_numsrc = count;
2578− if (put_user(IP_MSFILTER_SIZE(copycount), optlen) ||
2579− copy_to_user(optval, msf, IP_MSFILTER_SIZE(0))) {
2577+ msf_size = IP_MSFILTER_SIZE(copycount);
2578+ if (copy_to_sockptr(optlen, &msf_size, sizeof(int)) ||
2579+ copy_to_sockptr(optval, msf, IP_MSFILTER_SIZE(0))) {
25802580 return -EFAULT;
25812581 }
25822582 if (len &&
2583− copy_to_user(&optval->imsf_slist_flex[0], psl->sl_addr, len))
2583+ copy_to_sockptr_offset(optval,
2584+ offsetof(struct ip_msfilter, imsf_slist_flex),
2585+ psl->sl_addr, len))
25842586 return -EFAULT;
25852587 return 0;
25862588 done:
@@ -2588,7 +2590,7 @@ done: · 1 unmodified line
25882590 }
25892591
25902592 int ip_mc_gsfget(struct sock *sk, struct group_filter *gsf,
2591− struct sockaddr_storage __user *p)
2593+ sockptr_t optval, size_t ss_offset)
25922594 {
25932595 int i, count, copycount;
25942596 struct sockaddr_in *psin;
@@ -2618,15 +2620,17 @@ int ip_mc_gsfget(struct sock *sk, struct group_filter *gsf, · 23 unmodified lines
26182620 count = psl ? psl->sl_count : 0;
26192621 copycount = count < gsf->gf_numsrc ? count : gsf->gf_numsrc;
26202622 gsf->gf_numsrc = count;
2621− for (i = 0; i < copycount; i++, p++) {
2623+ for (i = 0; i < copycount; i++) {
26222624 struct sockaddr_storage ss;
26232625
26242626 psin = (struct sockaddr_in *)&ss;
26252627 memset(&ss, 0, sizeof(ss));
26262628 psin->sin_family = AF_INET;
26272629 psin->sin_addr.s_addr = psl->sl_addr[i];
2628− if (copy_to_user(p, &ss, sizeof(ss)))
2630+ if (copy_to_sockptr_offset(optval, ss_offset,
2631+ &ss, sizeof(ss)))
26292632 return -EFAULT;
2633+ ss_offset += sizeof(ss);
26302634 }
26312635 return 0;
26322636 }
net/ipv4/ip_sockglue.c+62−52
@@ -888,8 +888,8 @@ static int compat_ip_mcast_join_leave(struct sock *sk, int optname, · 887 unmodified lines
888888
889889 DEFINE_STATIC_KEY_FALSE(ip4_min_ttl);
890890
891−static int do_ip_setsockopt(struct sock *sk, int level, int optname,
892− sockptr_t optval, unsigned int optlen)
891+int do_ip_setsockopt(struct sock *sk, int level, int optname,
892+ sockptr_t optval, unsigned int optlen)
893893 {
894894 struct inet_sock *inet = inet_sk(sk);
895895 struct net *net = sock_net(sk);
@@ -944,7 +944,7 @@ static int do_ip_setsockopt(struct sock *sk, int level, int optname, · 48 unmodified lines
944944 err = 0;
945945 if (needs_rtnl)
946946 rtnl_lock();
947− lock_sock(sk);
947+ sockopt_lock_sock(sk);
948948
949949 switch (optname) {
950950 case IP_OPTIONS:
@@ -1333,14 +1333,14 @@ static int do_ip_setsockopt(struct sock *sk, int level, int optname, · 382 unmodified lines
13331333 case IP_IPSEC_POLICY:
13341334 case IP_XFRM_POLICY:
13351335 err = -EPERM;
1336− if (!ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN))
1336+ if (!sockopt_ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN))
13371337 break;
13381338 err = xfrm_user_policy(sk, optname, optval, optlen);
13391339 break;
13401340
13411341 case IP_TRANSPARENT:
1342− if (!!val && !ns_capable(sock_net(sk)->user_ns, CAP_NET_RAW) &&
1343− !ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN)) {
1342+ if (!!val && !sockopt_ns_capable(sock_net(sk)->user_ns, CAP_NET_RAW) &&
1343+ !sockopt_ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN)) {
13441344 err = -EPERM;
13451345 break;
13461346 }
@@ -1368,13 +1368,13 @@ static int do_ip_setsockopt(struct sock *sk, int level, int optname, · 21 unmodified lines
13681368 err = -ENOPROTOOPT;
13691369 break;
13701370 }
1371− release_sock(sk);
1371+ sockopt_release_sock(sk);
13721372 if (needs_rtnl)
13731373 rtnl_unlock();
13741374 return err;
13751375
13761376 e_inval:
1377− release_sock(sk);
1377+ sockopt_release_sock(sk);
13781378 if (needs_rtnl)
13791379 rtnl_unlock();
13801380 return -EINVAL;
@@ -1462,37 +1462,37 @@ static bool getsockopt_needs_rtnl(int optname) · 81 unmodified lines
14621462 return false;
14631463 }
14641464
1465−static int ip_get_mcast_msfilter(struct sock *sk, void __user *optval,
1466− int __user *optlen, int len)
1465+static int ip_get_mcast_msfilter(struct sock *sk, sockptr_t optval,
1466+ sockptr_t optlen, int len)
14671467 {
14681468 const int size0 = offsetof(struct group_filter, gf_slist_flex);
1469− struct group_filter __user *p = optval;
14701469 struct group_filter gsf;
1471− int num;
1470+ int num, gsf_size;
14721471 int err;
14731472
14741473 if (len < size0)
14751474 return -EINVAL;
1476− if (copy_from_user(&gsf, p, size0))
1475+ if (copy_from_sockptr(&gsf, optval, size0))
14771476 return -EFAULT;
14781477
14791478 num = gsf.gf_numsrc;
1480− err = ip_mc_gsfget(sk, &gsf, p->gf_slist_flex);
1479+ err = ip_mc_gsfget(sk, &gsf, optval,
1480+ offsetof(struct group_filter, gf_slist_flex));
14811481 if (err)
14821482 return err;
14831483 if (gsf.gf_numsrc < num)
14841484 num = gsf.gf_numsrc;
1485− if (put_user(GROUP_FILTER_SIZE(num), optlen) ||
1486− copy_to_user(p, &gsf, size0))
1485+ gsf_size = GROUP_FILTER_SIZE(num);
1486+ if (copy_to_sockptr(optlen, &gsf_size, sizeof(int)) ||
1487+ copy_to_sockptr(optval, &gsf, size0))
14871488 return -EFAULT;
14881489 return 0;
14891490 }
14901491
1491−static int compat_ip_get_mcast_msfilter(struct sock *sk, void __user *optval,
1492− int __user *optlen, int len)
1492+static int compat_ip_get_mcast_msfilter(struct sock *sk, sockptr_t optval,
1493+ sockptr_t optlen, int len)
14931494 {
14941495 const int size0 = offsetof(struct compat_group_filter, gf_slist_flex);
1495− struct compat_group_filter __user *p = optval;
14961496 struct compat_group_filter gf32;
14971497 struct group_filter gf;
14981498 int num;
@@ -1500,7 +1500,7 @@ static int compat_ip_get_mcast_msfilter(struct sock *sk, void __user *optval, · 1 unmodified line
15001500
15011501 if (len < size0)
15021502 return -EINVAL;
1503− if (copy_from_user(&gf32, p, size0))
1503+ if (copy_from_sockptr(&gf32, optval, size0))
15041504 return -EFAULT;
15051505
15061506 gf.gf_interface = gf32.gf_interface;
@@ -1508,21 +1508,24 @@ static int compat_ip_get_mcast_msfilter(struct sock *sk, void __user *optval, · 1 unmodified line
15081508 num = gf.gf_numsrc = gf32.gf_numsrc;
15091509 gf.gf_group = gf32.gf_group;
15101510
1511− err = ip_mc_gsfget(sk, &gf, p->gf_slist_flex);
1511+ err = ip_mc_gsfget(sk, &gf, optval,
1512+ offsetof(struct compat_group_filter, gf_slist_flex));
15121513 if (err)
15131514 return err;
15141515 if (gf.gf_numsrc < num)
15151516 num = gf.gf_numsrc;
15161517 len = GROUP_FILTER_SIZE(num) - (sizeof(gf) - sizeof(gf32));
1517− if (put_user(len, optlen) ||
1518− put_user(gf.gf_fmode, &p->gf_fmode) ||
1519− put_user(gf.gf_numsrc, &p->gf_numsrc))
1518+ if (copy_to_sockptr(optlen, &len, sizeof(int)) ||
1519+ copy_to_sockptr_offset(optval, offsetof(struct compat_group_filter, gf_fmode),
1520+ &gf.gf_fmode, sizeof(gf.gf_fmode)) ||
1521+ copy_to_sockptr_offset(optval, offsetof(struct compat_group_filter, gf_numsrc),
1522+ &gf.gf_numsrc, sizeof(gf.gf_numsrc)))
15201523 return -EFAULT;
15211524 return 0;
15221525 }
15231526
1524−static int do_ip_getsockopt(struct sock *sk, int level, int optname,
1525− char __user *optval, int __user *optlen)
1527+int do_ip_getsockopt(struct sock *sk, int level, int optname,
1528+ sockptr_t optval, sockptr_t optlen)
15261529 {
15271530 struct inet_sock *inet = inet_sk(sk);
15281531 bool needs_rtnl = getsockopt_needs_rtnl(optname);
@@ -1535,14 +1538,14 @@ static int do_ip_getsockopt(struct sock *sk, int level, int optname, · 6 unmodified lines
15351538 if (ip_mroute_opt(optname))
15361539 return ip_mroute_getsockopt(sk, optname, optval, optlen);
15371540
1538− if (get_user(len, optlen))
1541+ if (copy_from_sockptr(&len, optlen, sizeof(int)))
15391542 return -EFAULT;
15401543 if (len < 0)
15411544 return -EINVAL;
15421545
15431546 if (needs_rtnl)
15441547 rtnl_lock();
1545− lock_sock(sk);
1548+ sockopt_lock_sock(sk);
15461549
15471550 switch (optname) {
15481551 case IP_OPTIONS:
@@ -1558,17 +1561,19 @@ static int do_ip_getsockopt(struct sock *sk, int level, int optname, · 9 unmodified lines
15581561 memcpy(optbuf, &inet_opt->opt,
15591562 sizeof(struct ip_options) +
15601563 inet_opt->opt.optlen);
1561− release_sock(sk);
1564+ sockopt_release_sock(sk);
15621565
1563− if (opt->optlen == 0)
1564− return put_user(0, optlen);
1566+ if (opt->optlen == 0) {
1567+ len = 0;
1568+ return copy_to_sockptr(optlen, &len, sizeof(int));
1569+ }
15651570
15661571 ip_options_undo(opt);
15671572
15681573 len = min_t(unsigned int, len, opt->optlen);
1569− if (put_user(len, optlen))
1574+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
15701575 return -EFAULT;
1571− if (copy_to_user(optval, opt->__data, len))
1576+ if (copy_to_sockptr(optval, opt->__data, len))
15721577 return -EFAULT;
15731578 return 0;
15741579 }
@@ -1632,7 +1637,7 @@ static int do_ip_getsockopt(struct sock *sk, int level, int optname, · 57 unmodified lines
16321637 dst_release(dst);
16331638 }
16341639 if (!val) {
1635− release_sock(sk);
1640+ sockopt_release_sock(sk);
16361641 return -ENOTCONN;
16371642 }
16381643 break;
@@ -1657,11 +1662,11 @@ static int do_ip_getsockopt(struct sock *sk, int level, int optname, · 18 unmodified lines
16571662 struct in_addr addr;
16581663 len = min_t(unsigned int, len, sizeof(struct in_addr));
16591664 addr.s_addr = inet->mc_addr;
1660− release_sock(sk);
1665+ sockopt_release_sock(sk);
16611666
1662− if (put_user(len, optlen))
1667+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
16631668 return -EFAULT;
1664− if (copy_to_user(optval, &addr, len))
1669+ if (copy_to_sockptr(optval, &addr, len))
16651670 return -EFAULT;
16661671 return 0;
16671672 }
@@ -1673,12 +1678,11 @@ static int do_ip_getsockopt(struct sock *sk, int level, int optname, · 5 unmodified lines
16731678 err = -EINVAL;
16741679 goto out;
16751680 }
1676− if (copy_from_user(&msf, optval, IP_MSFILTER_SIZE(0))) {
1681+ if (copy_from_sockptr(&msf, optval, IP_MSFILTER_SIZE(0))) {
16771682 err = -EFAULT;
16781683 goto out;
16791684 }
1680− err = ip_mc_msfget(sk, &msf,
1681− (struct ip_msfilter __user *)optval, optlen);
1685+ err = ip_mc_msfget(sk, &msf, optval, optlen);
16821686 goto out;
16831687 }
16841688 case MCAST_MSFILTER:
@@ -1695,13 +1699,18 @@ static int do_ip_getsockopt(struct sock *sk, int level, int optname, · 10 unmodified lines
16951699 {
16961700 struct msghdr msg;
16971701
1698− release_sock(sk);
1702+ sockopt_release_sock(sk);
16991703
17001704 if (sk->sk_type != SOCK_STREAM)
17011705 return -ENOPROTOOPT;
17021706
1703− msg.msg_control_is_user = true;
1704− msg.msg_control_user = optval;
1707+ if (optval.is_kernel) {
1708+ msg.msg_control_is_user = false;
1709+ msg.msg_control = optval.kernel;
1710+ } else {
1711+ msg.msg_control_is_user = true;
1712+ msg.msg_control_user = optval.user;
1713+ }
17051714 msg.msg_controllen = len;
17061715 msg.msg_flags = in_compat_syscall() ? MSG_CMSG_COMPAT : 0;
17071716
@@ -1722,7 +1731,7 @@ static int do_ip_getsockopt(struct sock *sk, int level, int optname, · 14 unmodified lines
17221731 put_cmsg(&msg, SOL_IP, IP_TOS, sizeof(tos), &tos);
17231732 }
17241733 len -= msg.msg_controllen;
1725− return put_user(len, optlen);
1734+ return copy_to_sockptr(optlen, &len, sizeof(int));
17261735 }
17271736 case IP_FREEBIND:
17281737 val = inet->freebind;
@@ -1734,29 +1743,29 @@ static int do_ip_getsockopt(struct sock *sk, int level, int optname, · 5 unmodified lines
17341743 val = inet->min_ttl;
17351744 break;
17361745 default:
1737− release_sock(sk);
1746+ sockopt_release_sock(sk);
17381747 return -ENOPROTOOPT;
17391748 }
1740− release_sock(sk);
1749+ sockopt_release_sock(sk);
17411750
17421751 if (len < sizeof(int) && len > 0 && val >= 0 && val <= 255) {
17431752 unsigned char ucval = (unsigned char)val;
17441753 len = 1;
1745− if (put_user(len, optlen))
1754+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
17461755 return -EFAULT;
1747− if (copy_to_user(optval, &ucval, 1))
1756+ if (copy_to_sockptr(optval, &ucval, 1))
17481757 return -EFAULT;
17491758 } else {
17501759 len = min_t(unsigned int, sizeof(int), len);
1751− if (put_user(len, optlen))
1760+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
17521761 return -EFAULT;
1753− if (copy_to_user(optval, &val, len))
1762+ if (copy_to_sockptr(optval, &val, len))
17541763 return -EFAULT;
17551764 }
17561765 return 0;
17571766
17581767 out:
1759− release_sock(sk);
1768+ sockopt_release_sock(sk);
17601769 if (needs_rtnl)
17611770 rtnl_unlock();
17621771 return err;
@@ -1767,7 +1776,8 @@ int ip_getsockopt(struct sock *sk, int level, · 4 unmodified lines
17671776 {
17681777 int err;
17691778
1770− err = do_ip_getsockopt(sk, level, optname, optval, optlen);
1779+ err = do_ip_getsockopt(sk, level, optname,
1780+ USER_SOCKPTR(optval), USER_SOCKPTR(optlen));
17711781
17721782 #if IS_ENABLED(CONFIG_BPFILTER_UMH)
17731783 if (optname >= BPFILTER_IPT_SO_GET_INFO &&
net/ipv4/ipmr.c+5−4
@@ -1546,7 +1546,8 @@ out: · 1545 unmodified lines
15461546 }
15471547
15481548 /* Getsock opt support for the multicast routing system. */
1549−int ip_mroute_getsockopt(struct sock *sk, int optname, char __user *optval, int __user *optlen)
1549+int ip_mroute_getsockopt(struct sock *sk, int optname, sockptr_t optval,
1550+ sockptr_t optlen)
15501551 {
15511552 int olr;
15521553 int val;
@@ -1577,14 +1578,14 @@ int ip_mroute_getsockopt(struct sock *sk, int optname, char __user *optval, int · 24 unmodified lines
15771578 return -ENOPROTOOPT;
15781579 }
15791580
1580− if (get_user(olr, optlen))
1581+ if (copy_from_sockptr(&olr, optlen, sizeof(int)))
15811582 return -EFAULT;
15821583 olr = min_t(unsigned int, olr, sizeof(int));
15831584 if (olr < 0)
15841585 return -EINVAL;
1585− if (put_user(olr, optlen))
1586+ if (copy_to_sockptr(optlen, &olr, sizeof(int)))
15861587 return -EFAULT;
1587− if (copy_to_user(optval, &val, olr))
1588+ if (copy_to_sockptr(optval, &val, olr))
15881589 return -EFAULT;
15891590 return 0;
15901591 }
net/ipv4/tcp.c+59−57
@@ -3199,7 +3199,7 @@ EXPORT_SYMBOL(tcp_disconnect); · 3198 unmodified lines
31993199
32003200 static inline bool tcp_can_repair_sock(const struct sock *sk)
32013201 {
3202− return ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN) &&
3202+ return sockopt_ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN) &&
32033203 (sk->sk_state != TCP_LISTEN);
32043204 }
32053205
@@ -3476,8 +3476,8 @@ int tcp_set_window_clamp(struct sock *sk, int val) · 270 unmodified lines
34763476 /*
34773477 * Socket option code for TCP.
34783478 */
3479−static int do_tcp_setsockopt(struct sock *sk, int level, int optname,
3480− sockptr_t optval, unsigned int optlen)
3479+int do_tcp_setsockopt(struct sock *sk, int level, int optname,
3480+ sockptr_t optval, unsigned int optlen)
34813481 {
34823482 struct tcp_sock *tp = tcp_sk(sk);
34833483 struct inet_connection_sock *icsk = inet_csk(sk);
@@ -3499,11 +3499,11 @@ static int do_tcp_setsockopt(struct sock *sk, int level, int optname, · 15 unmodified lines
34993499 return -EFAULT;
35003500 name[val] = 0;
35013501
3502− lock_sock(sk);
3503− err = tcp_set_congestion_control(sk, name, true,
3504− ns_capable(sock_net(sk)->user_ns,
3505− CAP_NET_ADMIN));
3506− release_sock(sk);
3502+ sockopt_lock_sock(sk);
3503+ err = tcp_set_congestion_control(sk, name, !has_current_bpf_ctx(),
3504+ sockopt_ns_capable(sock_net(sk)->user_ns,
3505+ CAP_NET_ADMIN));
3506+ sockopt_release_sock(sk);
35073507 return err;
35083508 }
35093509 case TCP_ULP: {
@@ -3519,9 +3519,9 @@ static int do_tcp_setsockopt(struct sock *sk, int level, int optname, · 9 unmodified lines
35193519 return -EFAULT;
35203520 name[val] = 0;
35213521
3522− lock_sock(sk);
3522+ sockopt_lock_sock(sk);
35233523 err = tcp_set_ulp(sk, name);
3524− release_sock(sk);
3524+ sockopt_release_sock(sk);
35253525 return err;
35263526 }
35273527 case TCP_FASTOPEN_KEY: {
@@ -3554,7 +3554,7 @@ static int do_tcp_setsockopt(struct sock *sk, int level, int optname, · 26 unmodified lines
35543554 if (copy_from_sockptr(&val, optval, sizeof(val)))
35553555 return -EFAULT;
35563556
3557− lock_sock(sk);
3557+ sockopt_lock_sock(sk);
35583558
35593559 switch (optname) {
35603560 case TCP_MAXSEG:
@@ -3776,7 +3776,7 @@ static int do_tcp_setsockopt(struct sock *sk, int level, int optname, · 215 unmodified lines
37763776 break;
37773777 }
37783778
3779− release_sock(sk);
3779+ sockopt_release_sock(sk);
37803780 return err;
37813781 }
37823782
@@ -4040,15 +4040,15 @@ struct sk_buff *tcp_get_timestamping_opt_stats(const struct sock *sk, · 257 unmodified lines
40404040 return stats;
40414041 }
40424042
4043−static int do_tcp_getsockopt(struct sock *sk, int level,
4044− int optname, char __user *optval, int __user *optlen)
4043+int do_tcp_getsockopt(struct sock *sk, int level,
4044+ int optname, sockptr_t optval, sockptr_t optlen)
40454045 {
40464046 struct inet_connection_sock *icsk = inet_csk(sk);
40474047 struct tcp_sock *tp = tcp_sk(sk);
40484048 struct net *net = sock_net(sk);
40494049 int val, len;
40504050
4051− if (get_user(len, optlen))
4051+ if (copy_from_sockptr(&len, optlen, sizeof(int)))
40524052 return -EFAULT;
40534053
40544054 len = min_t(unsigned int, len, sizeof(int));
@@ -4098,15 +4098,15 @@ static int do_tcp_getsockopt(struct sock *sk, int level, · 43 unmodified lines
40984098 case TCP_INFO: {
40994099 struct tcp_info info;
41004100
4101− if (get_user(len, optlen))
4101+ if (copy_from_sockptr(&len, optlen, sizeof(int)))
41024102 return -EFAULT;
41034103
41044104 tcp_get_info(sk, &info);
41054105
41064106 len = min_t(unsigned int, len, sizeof(info));
4107− if (put_user(len, optlen))
4107+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
41084108 return -EFAULT;
4109− if (copy_to_user(optval, &info, len))
4109+ if (copy_to_sockptr(optval, &info, len))
41104110 return -EFAULT;
41114111 return 0;
41124112 }
@@ -4116,7 +4116,7 @@ static int do_tcp_getsockopt(struct sock *sk, int level, · 3 unmodified lines
41164116 size_t sz = 0;
41174117 int attr;
41184118
4119− if (get_user(len, optlen))
4119+ if (copy_from_sockptr(&len, optlen, sizeof(int)))
41204120 return -EFAULT;
41214121
41224122 ca_ops = icsk->icsk_ca_ops;
@@ -4124,9 +4124,9 @@ static int do_tcp_getsockopt(struct sock *sk, int level, · 1 unmodified line
41244124 sz = ca_ops->get_info(sk, ~0U, &attr, &info);
41254125
41264126 len = min_t(unsigned int, len, sz);
4127− if (put_user(len, optlen))
4127+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
41284128 return -EFAULT;
4129− if (copy_to_user(optval, &info, len))
4129+ if (copy_to_sockptr(optval, &info, len))
41304130 return -EFAULT;
41314131 return 0;
41324132 }
@@ -4135,27 +4135,28 @@ static int do_tcp_getsockopt(struct sock *sk, int level, · 2 unmodified lines
41354135 break;
41364136
41374137 case TCP_CONGESTION:
4138− if (get_user(len, optlen))
4138+ if (copy_from_sockptr(&len, optlen, sizeof(int)))
41394139 return -EFAULT;
41404140 len = min_t(unsigned int, len, TCP_CA_NAME_MAX);
4141− if (put_user(len, optlen))
4141+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
41424142 return -EFAULT;
4143− if (copy_to_user(optval, icsk->icsk_ca_ops->name, len))
4143+ if (copy_to_sockptr(optval, icsk->icsk_ca_ops->name, len))
41444144 return -EFAULT;
41454145 return 0;
41464146
41474147 case TCP_ULP:
4148− if (get_user(len, optlen))
4148+ if (copy_from_sockptr(&len, optlen, sizeof(int)))
41494149 return -EFAULT;
41504150 len = min_t(unsigned int, len, TCP_ULP_NAME_MAX);
41514151 if (!icsk->icsk_ulp_ops) {
4152− if (put_user(0, optlen))
4152+ len = 0;
4153+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
41534154 return -EFAULT;
41544155 return 0;
41554156 }
4156− if (put_user(len, optlen))
4157+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
41574158 return -EFAULT;
4158− if (copy_to_user(optval, icsk->icsk_ulp_ops->name, len))
4159+ if (copy_to_sockptr(optval, icsk->icsk_ulp_ops->name, len))
41594160 return -EFAULT;
41604161 return 0;
41614162
@@ -4163,15 +4164,15 @@ static int do_tcp_getsockopt(struct sock *sk, int level, · 1 unmodified line
41634164 u64 key[TCP_FASTOPEN_KEY_BUF_LENGTH / sizeof(u64)];
41644165 unsigned int key_len;
41654166
4166− if (get_user(len, optlen))
4167+ if (copy_from_sockptr(&len, optlen, sizeof(int)))
41674168 return -EFAULT;
41684169
41694170 key_len = tcp_fastopen_get_cipher(net, icsk, key) *
41704171 TCP_FASTOPEN_KEY_LENGTH;
41714172 len = min_t(unsigned int, len, key_len);
4172− if (put_user(len, optlen))
4173+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
41734174 return -EFAULT;
4174− if (copy_to_user(optval, key, len))
4175+ if (copy_to_sockptr(optval, key, len))
41754176 return -EFAULT;
41764177 return 0;
41774178 }
@@ -4197,7 +4198,7 @@ static int do_tcp_getsockopt(struct sock *sk, int level, · 19 unmodified lines
41974198 case TCP_REPAIR_WINDOW: {
41984199 struct tcp_repair_window opt;
41994200
4200− if (get_user(len, optlen))
4201+ if (copy_from_sockptr(&len, optlen, sizeof(int)))
42014202 return -EFAULT;
42024203
42034204 if (len != sizeof(opt))
@@ -4212,7 +4213,7 @@ static int do_tcp_getsockopt(struct sock *sk, int level, · 8 unmodified lines
42124213 opt.rcv_wnd = tp->rcv_wnd;
42134214 opt.rcv_wup = tp->rcv_wup;
42144215
4215− if (copy_to_user(optval, &opt, len))
4216+ if (copy_to_sockptr(optval, &opt, len))
42164217 return -EFAULT;
42174218 return 0;
42184219 }
@@ -4258,35 +4259,35 @@ static int do_tcp_getsockopt(struct sock *sk, int level, · 39 unmodified lines
42584259 val = tp->save_syn;
42594260 break;
42604261 case TCP_SAVED_SYN: {
4261− if (get_user(len, optlen))
4262+ if (copy_from_sockptr(&len, optlen, sizeof(int)))
42624263 return -EFAULT;
42634264
4264− lock_sock(sk);
4265+ sockopt_lock_sock(sk);
42654266 if (tp->saved_syn) {
42664267 if (len < tcp_saved_syn_len(tp->saved_syn)) {
4267− if (put_user(tcp_saved_syn_len(tp->saved_syn),
4268− optlen)) {
4269− release_sock(sk);
4268+ len = tcp_saved_syn_len(tp->saved_syn);
4269+ if (copy_to_sockptr(optlen, &len, sizeof(int))) {
4270+ sockopt_release_sock(sk);
42704271 return -EFAULT;
42714272 }
4272− release_sock(sk);
4273+ sockopt_release_sock(sk);
42734274 return -EINVAL;
42744275 }
42754276 len = tcp_saved_syn_len(tp->saved_syn);
4276− if (put_user(len, optlen)) {
4277− release_sock(sk);
4277+ if (copy_to_sockptr(optlen, &len, sizeof(int))) {
4278+ sockopt_release_sock(sk);
42784279 return -EFAULT;
42794280 }
4280− if (copy_to_user(optval, tp->saved_syn->data, len)) {
4281− release_sock(sk);
4281+ if (copy_to_sockptr(optval, tp->saved_syn->data, len)) {
4282+ sockopt_release_sock(sk);
42824283 return -EFAULT;
42834284 }
42844285 tcp_saved_syn_free(tp);
4285− release_sock(sk);
4286+ sockopt_release_sock(sk);
42864287 } else {
4287− release_sock(sk);
4288+ sockopt_release_sock(sk);
42884289 len = 0;
4289− if (put_user(len, optlen))
4290+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
42904291 return -EFAULT;
42914292 }
42924293 return 0;
@@ -4297,31 +4298,31 @@ static int do_tcp_getsockopt(struct sock *sk, int level, · 4 unmodified lines
42974298 struct tcp_zerocopy_receive zc = {};
42984299 int err;
42994300
4300− if (get_user(len, optlen))
4301+ if (copy_from_sockptr(&len, optlen, sizeof(int)))
43014302 return -EFAULT;
43024303 if (len < 0 ||
43034304 len < offsetofend(struct tcp_zerocopy_receive, length))
43044305 return -EINVAL;
43054306 if (unlikely(len > sizeof(zc))) {
4306− err = check_zeroed_user(optval + sizeof(zc),
4307− len - sizeof(zc));
4307+ err = check_zeroed_sockptr(optval, sizeof(zc),
4308+ len - sizeof(zc));
43084309 if (err < 1)
43094310 return err == 0 ? -EINVAL : err;
43104311 len = sizeof(zc);
4311− if (put_user(len, optlen))
4312+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
43124313 return -EFAULT;
43134314 }
4314− if (copy_from_user(&zc, optval, len))
4315+ if (copy_from_sockptr(&zc, optval, len))
43154316 return -EFAULT;
43164317 if (zc.reserved)
43174318 return -EINVAL;
43184319 if (zc.msg_flags & ~(TCP_VALID_ZC_MSG_FLAGS))
43194320 return -EINVAL;
4320− lock_sock(sk);
4321+ sockopt_lock_sock(sk);
43214322 err = tcp_zerocopy_receive(sk, &zc, &tss);
43224323 err = BPF_CGROUP_RUN_PROG_GETSOCKOPT_KERN(sk, level, optname,
43234324 &zc, &len, err);
4324− release_sock(sk);
4325+ sockopt_release_sock(sk);
43254326 if (len >= offsetofend(struct tcp_zerocopy_receive, msg_flags))
43264327 goto zerocopy_rcv_cmsg;
43274328 switch (len) {
@@ -4351,7 +4352,7 @@ zerocopy_rcv_sk_err: · 23 unmodified lines
43514352 zerocopy_rcv_inq:
43524353 zc.inq = tcp_inq_hint(sk);
43534354 zerocopy_rcv_out:
4354− if (!err && copy_to_user(optval, &zc, len))
4355+ if (!err && copy_to_sockptr(optval, &zc, len))
43554356 err = -EFAULT;
43564357 return err;
43574358 }
@@ -4360,9 +4361,9 @@ zerocopy_rcv_out: · 2 unmodified lines
43604361 return -ENOPROTOOPT;
43614362 }
43624363
4363− if (put_user(len, optlen))
4364+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
43644365 return -EFAULT;
4365− if (copy_to_user(optval, &val, len))
4366+ if (copy_to_sockptr(optval, &val, len))
43664367 return -EFAULT;
43674368 return 0;
43684369 }
@@ -4387,7 +4388,8 @@ int tcp_getsockopt(struct sock *sk, int level, int optname, char __user *optval, · 18 unmodified lines
43874388 if (level != SOL_TCP)
43884389 return icsk->icsk_af_ops->getsockopt(sk, level, optname,
43894390 optval, optlen);
4390− return do_tcp_getsockopt(sk, level, optname, optval, optlen);
4391+ return do_tcp_getsockopt(sk, level, optname, USER_SOCKPTR(optval),
4392+ USER_SOCKPTR(optlen));
43914393 }
43924394 EXPORT_SYMBOL(tcp_getsockopt);
43934395
net/ipv6/af_inet6.c+2−0
@@ -1057,6 +1057,8 @@ static const struct ipv6_stub ipv6_stub_impl = { · 1056 unmodified lines
10571057 static const struct ipv6_bpf_stub ipv6_bpf_stub_impl = {
10581058 .inet6_bind = __inet6_bind,
10591059 .udp6_lib_lookup = __udp6_lib_lookup,
1060+ .ipv6_setsockopt = do_ipv6_setsockopt,
1061+ .ipv6_getsockopt = do_ipv6_getsockopt,
10601062 };
10611063
10621064 static int __init inet6_init(void)
net/ipv6/ip6mr.c+5−5
@@ -1827,8 +1827,8 @@ int ip6_mroute_setsockopt(struct sock *sk, int optname, sockptr_t optval, · 1826 unmodified lines
18271827 * Getsock opt support for the multicast routing system.
18281828 */
18291829
1830−int ip6_mroute_getsockopt(struct sock *sk, int optname, char __user *optval,
1831− int __user *optlen)
1830+int ip6_mroute_getsockopt(struct sock *sk, int optname, sockptr_t optval,
1831+ sockptr_t optlen)
18321832 {
18331833 int olr;
18341834 int val;
@@ -1859,16 +1859,16 @@ int ip6_mroute_getsockopt(struct sock *sk, int optname, char __user *optval, · 24 unmodified lines
18591859 return -ENOPROTOOPT;
18601860 }
18611861
1862− if (get_user(olr, optlen))
1862+ if (copy_from_sockptr(&olr, optlen, sizeof(int)))
18631863 return -EFAULT;
18641864
18651865 olr = min_t(int, olr, sizeof(int));
18661866 if (olr < 0)
18671867 return -EINVAL;
18681868
1869− if (put_user(olr, optlen))
1869+ if (copy_to_sockptr(optlen, &olr, sizeof(int)))
18701870 return -EFAULT;
1871− if (copy_to_user(optval, &val, olr))
1871+ if (copy_to_sockptr(optval, &val, olr))
18721872 return -EFAULT;
18731873 return 0;
18741874 }
net/ipv6/ipv6_sockglue.c+59−54
@@ -327,7 +327,7 @@ static int ipv6_set_opt_hdr(struct sock *sk, int optname, sockptr_t optval, · 326 unmodified lines
327327 int err;
328328
329329 /* hop-by-hop / destination options are privileged option */
330− if (optname != IPV6_RTHDR && !ns_capable(net->user_ns, CAP_NET_RAW))
330+ if (optname != IPV6_RTHDR && !sockopt_ns_capable(net->user_ns, CAP_NET_RAW))
331331 return -EPERM;
332332
333333 /* remove any sticky options header with a zero option
@@ -391,8 +391,8 @@ sticky_done: · 57 unmodified lines
391391 return err;
392392 }
393393
394−static int do_ipv6_setsockopt(struct sock *sk, int level, int optname,
395− sockptr_t optval, unsigned int optlen)
394+int do_ipv6_setsockopt(struct sock *sk, int level, int optname,
395+ sockptr_t optval, unsigned int optlen)
396396 {
397397 struct ipv6_pinfo *np = inet6_sk(sk);
398398 struct net *net = sock_net(sk);
@@ -417,7 +417,7 @@ static int do_ipv6_setsockopt(struct sock *sk, int level, int optname, · 18 unmodified lines
417417
418418 if (needs_rtnl)
419419 rtnl_lock();
420− lock_sock(sk);
420+ sockopt_lock_sock(sk);
421421
422422 switch (optname) {
423423
@@ -634,8 +634,8 @@ static int do_ipv6_setsockopt(struct sock *sk, int level, int optname, · 210 unmodified lines
634634 break;
635635
636636 case IPV6_TRANSPARENT:
637− if (valbool && !ns_capable(net->user_ns, CAP_NET_RAW) &&
638− !ns_capable(net->user_ns, CAP_NET_ADMIN)) {
637+ if (valbool && !sockopt_ns_capable(net->user_ns, CAP_NET_RAW) &&
638+ !sockopt_ns_capable(net->user_ns, CAP_NET_ADMIN)) {
639639 retv = -EPERM;
640640 break;
641641 }
@@ -946,7 +946,7 @@ done: · 304 unmodified lines
946946 case IPV6_IPSEC_POLICY:
947947 case IPV6_XFRM_POLICY:
948948 retv = -EPERM;
949− if (!ns_capable(net->user_ns, CAP_NET_ADMIN))
949+ if (!sockopt_ns_capable(net->user_ns, CAP_NET_ADMIN))
950950 break;
951951 retv = xfrm_user_policy(sk, optname, optval, optlen);
952952 break;
@@ -994,14 +994,14 @@ done: · 41 unmodified lines
994994 break;
995995 }
996996
997− release_sock(sk);
997+ sockopt_release_sock(sk);
998998 if (needs_rtnl)
999999 rtnl_unlock();
10001000
10011001 return retv;
10021002
10031003 e_inval:
1004− release_sock(sk);
1004+ sockopt_release_sock(sk);
10051005 if (needs_rtnl)
10061006 rtnl_unlock();
10071007 return -EINVAL;
@@ -1030,7 +1030,7 @@ int ipv6_setsockopt(struct sock *sk, int level, int optname, sockptr_t optval, · 22 unmodified lines
10301030 EXPORT_SYMBOL(ipv6_setsockopt);
10311031
10321032 static int ipv6_getsockopt_sticky(struct sock *sk, struct ipv6_txoptions *opt,
1033− int optname, char __user *optval, int len)
1033+ int optname, sockptr_t optval, int len)
10341034 {
10351035 struct ipv6_opt_hdr *hdr;
10361036
@@ -1058,56 +1058,53 @@ static int ipv6_getsockopt_sticky(struct sock *sk, struct ipv6_txoptions *opt, · 21 unmodified lines
10581058 return 0;
10591059
10601060 len = min_t(unsigned int, len, ipv6_optlen(hdr));
1061− if (copy_to_user(optval, hdr, len))
1061+ if (copy_to_sockptr(optval, hdr, len))
10621062 return -EFAULT;
10631063 return len;
10641064 }
10651065
1066−static int ipv6_get_msfilter(struct sock *sk, void __user *optval,
1067− int __user *optlen, int len)
1066+static int ipv6_get_msfilter(struct sock *sk, sockptr_t optval,
1067+ sockptr_t optlen, int len)
10681068 {
10691069 const int size0 = offsetof(struct group_filter, gf_slist_flex);
1070− struct group_filter __user *p = optval;
10711070 struct group_filter gsf;
10721071 int num;
10731072 int err;
10741073
10751074 if (len < size0)
10761075 return -EINVAL;
1077− if (copy_from_user(&gsf, p, size0))
1076+ if (copy_from_sockptr(&gsf, optval, size0))
10781077 return -EFAULT;
10791078 if (gsf.gf_group.ss_family != AF_INET6)
10801079 return -EADDRNOTAVAIL;
10811080 num = gsf.gf_numsrc;
1082− lock_sock(sk);
1083− err = ip6_mc_msfget(sk, &gsf, p->gf_slist_flex);
1081+ sockopt_lock_sock(sk);
1082+ err = ip6_mc_msfget(sk, &gsf, optval, size0);
10841083 if (!err) {
10851084 if (num > gsf.gf_numsrc)
10861085 num = gsf.gf_numsrc;
1087− if (put_user(GROUP_FILTER_SIZE(num), optlen) ||
1088− copy_to_user(p, &gsf, size0))
1086+ len = GROUP_FILTER_SIZE(num);
1087+ if (copy_to_sockptr(optlen, &len, sizeof(int)) ||
1088+ copy_to_sockptr(optval, &gsf, size0))
10891089 err = -EFAULT;
10901090 }
1091− release_sock(sk);
1091+ sockopt_release_sock(sk);
10921092 return err;
10931093 }
10941094
1095−static int compat_ipv6_get_msfilter(struct sock *sk, void __user *optval,
1096− int __user *optlen)
1095+static int compat_ipv6_get_msfilter(struct sock *sk, sockptr_t optval,
1096+ sockptr_t optlen, int len)
10971097 {
10981098 const int size0 = offsetof(struct compat_group_filter, gf_slist_flex);
1099− struct compat_group_filter __user *p = optval;
11001099 struct compat_group_filter gf32;
11011100 struct group_filter gf;
1102− int len, err;
1101+ int err;
11031102 int num;
11041103
1105− if (get_user(len, optlen))
1106− return -EFAULT;
11071104 if (len < size0)
11081105 return -EINVAL;
11091106
1110− if (copy_from_user(&gf32, p, size0))
1107+ if (copy_from_sockptr(&gf32, optval, size0))
11111108 return -EFAULT;
11121109 gf.gf_interface = gf32.gf_interface;
11131110 gf.gf_fmode = gf32.gf_fmode;
@@ -1117,23 +1114,25 @@ static int compat_ipv6_get_msfilter(struct sock *sk, void __user *optval, · 3 unmodified lines
11171114 if (gf.gf_group.ss_family != AF_INET6)
11181115 return -EADDRNOTAVAIL;
11191116
1120− lock_sock(sk);
1121− err = ip6_mc_msfget(sk, &gf, p->gf_slist_flex);
1122− release_sock(sk);
1117+ sockopt_lock_sock(sk);
1118+ err = ip6_mc_msfget(sk, &gf, optval, size0);
1119+ sockopt_release_sock(sk);
11231120 if (err)
11241121 return err;
11251122 if (num > gf.gf_numsrc)
11261123 num = gf.gf_numsrc;
11271124 len = GROUP_FILTER_SIZE(num) - (sizeof(gf)-sizeof(gf32));
1128− if (put_user(len, optlen) ||
1129− put_user(gf.gf_fmode, &p->gf_fmode) ||
1130− put_user(gf.gf_numsrc, &p->gf_numsrc))
1125+ if (copy_to_sockptr(optlen, &len, sizeof(int)) ||
1126+ copy_to_sockptr_offset(optval, offsetof(struct compat_group_filter, gf_fmode),
1127+ &gf.gf_fmode, sizeof(gf32.gf_fmode)) ||
1128+ copy_to_sockptr_offset(optval, offsetof(struct compat_group_filter, gf_numsrc),
1129+ &gf.gf_numsrc, sizeof(gf32.gf_numsrc)))
11311130 return -EFAULT;
11321131 return 0;
11331132 }
11341133
1135−static int do_ipv6_getsockopt(struct sock *sk, int level, int optname,
1136− char __user *optval, int __user *optlen, unsigned int flags)
1134+int do_ipv6_getsockopt(struct sock *sk, int level, int optname,
1135+ sockptr_t optval, sockptr_t optlen)
11371136 {
11381137 struct ipv6_pinfo *np = inet6_sk(sk);
11391138 int len;
@@ -1142,7 +1141,7 @@ static int do_ipv6_getsockopt(struct sock *sk, int level, int optname, · 2 unmodified lines
11421141 if (ip6_mroute_opt(optname))
11431142 return ip6_mroute_getsockopt(sk, optname, optval, optlen);
11441143
1145− if (get_user(len, optlen))
1144+ if (copy_from_sockptr(&len, optlen, sizeof(int)))
11461145 return -EFAULT;
11471146 switch (optname) {
11481147 case IPV6_ADDRFORM:
@@ -1156,7 +1155,7 @@ static int do_ipv6_getsockopt(struct sock *sk, int level, int optname, · 7 unmodified lines
11561155 break;
11571156 case MCAST_MSFILTER:
11581157 if (in_compat_syscall())
1159− return compat_ipv6_get_msfilter(sk, optval, optlen);
1158+ return compat_ipv6_get_msfilter(sk, optval, optlen, len);
11601159 return ipv6_get_msfilter(sk, optval, optlen, len);
11611160 case IPV6_2292PKTOPTIONS:
11621161 {
@@ -1166,16 +1165,21 @@ static int do_ipv6_getsockopt(struct sock *sk, int level, int optname, · 3 unmodified lines
11661165 if (sk->sk_type != SOCK_STREAM)
11671166 return -ENOPROTOOPT;
11681167
1169− msg.msg_control_user = optval;
1168+ if (optval.is_kernel) {
1169+ msg.msg_control_is_user = false;
1170+ msg.msg_control = optval.kernel;
1171+ } else {
1172+ msg.msg_control_is_user = true;
1173+ msg.msg_control_user = optval.user;
1174+ }
11701175 msg.msg_controllen = len;
1171− msg.msg_flags = flags;
1172− msg.msg_control_is_user = true;
1176+ msg.msg_flags = 0;
11731177
1174− lock_sock(sk);
1178+ sockopt_lock_sock(sk);
11751179 skb = np->pktoptions;
11761180 if (skb)
11771181 ip6_datagram_recv_ctl(sk, &msg, skb);
1178− release_sock(sk);
1182+ sockopt_release_sock(sk);
11791183 if (!skb) {
11801184 if (np->rxopt.bits.rxinfo) {
11811185 struct in6_pktinfo src_info;
@@ -1212,7 +1216,7 @@ static int do_ipv6_getsockopt(struct sock *sk, int level, int optname, · 30 unmodified lines
12121216 }
12131217 }
12141218 len -= msg.msg_controllen;
1215− return put_user(len, optlen);
1219+ return copy_to_sockptr(optlen, &len, sizeof(int));
12161220 }
12171221 case IPV6_MTU:
12181222 {
@@ -1264,15 +1268,15 @@ static int do_ipv6_getsockopt(struct sock *sk, int level, int optname, · 45 unmodified lines
12641268 {
12651269 struct ipv6_txoptions *opt;
12661270
1267− lock_sock(sk);
1271+ sockopt_lock_sock(sk);
12681272 opt = rcu_dereference_protected(np->opt,
12691273 lockdep_sock_is_held(sk));
12701274 len = ipv6_getsockopt_sticky(sk, opt, optname, optval, len);
1271− release_sock(sk);
1275+ sockopt_release_sock(sk);
12721276 /* check if ipv6_getsockopt_sticky() returns err code */
12731277 if (len < 0)
12741278 return len;
1275− return put_user(len, optlen);
1279+ return copy_to_sockptr(optlen, &len, sizeof(int));
12761280 }
12771281
12781282 case IPV6_RECVHOPOPTS:
@@ -1326,9 +1330,9 @@ static int do_ipv6_getsockopt(struct sock *sk, int level, int optname, · 47 unmodified lines
13261330 if (!mtuinfo.ip6m_mtu)
13271331 return -ENOTCONN;
13281332
1329− if (put_user(len, optlen))
1333+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
13301334 return -EFAULT;
1331− if (copy_to_user(optval, &mtuinfo, len))
1335+ if (copy_to_sockptr(optval, &mtuinfo, len))
13321336 return -EFAULT;
13331337
13341338 return 0;
@@ -1405,7 +1409,7 @@ static int do_ipv6_getsockopt(struct sock *sk, int level, int optname, · 70 unmodified lines
14051409 if (len < sizeof(freq))
14061410 return -EINVAL;
14071411
1408− if (copy_from_user(&freq, optval, sizeof(freq)))
1412+ if (copy_from_sockptr(&freq, optval, sizeof(freq)))
14091413 return -EFAULT;
14101414
14111415 if (freq.flr_action != IPV6_FL_A_GET)
@@ -1420,9 +1424,9 @@ static int do_ipv6_getsockopt(struct sock *sk, int level, int optname, · 8 unmodified lines
14201424 if (val < 0)
14211425 return val;
14221426
1423− if (put_user(len, optlen))
1427+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
14241428 return -EFAULT;
1425− if (copy_to_user(optval, &freq, len))
1429+ if (copy_to_sockptr(optval, &freq, len))
14261430 return -EFAULT;
14271431
14281432 return 0;
@@ -1474,9 +1478,9 @@ static int do_ipv6_getsockopt(struct sock *sk, int level, int optname, · 45 unmodified lines
14741478 return -ENOPROTOOPT;
14751479 }
14761480 len = min_t(unsigned int, sizeof(int), len);
1477− if (put_user(len, optlen))
1481+ if (copy_to_sockptr(optlen, &len, sizeof(int)))
14781482 return -EFAULT;
1479− if (copy_to_user(optval, &val, len))
1483+ if (copy_to_sockptr(optval, &val, len))
14801484 return -EFAULT;
14811485 return 0;
14821486 }
@@ -1492,7 +1496,8 @@ int ipv6_getsockopt(struct sock *sk, int level, int optname, · 9 unmodified lines
14921496 if (level != SOL_IPV6)
14931497 return -ENOPROTOOPT;
14941498
1495− err = do_ipv6_getsockopt(sk, level, optname, optval, optlen, 0);
1499+ err = do_ipv6_getsockopt(sk, level, optname,
1500+ USER_SOCKPTR(optval), USER_SOCKPTR(optlen));
14961501 #ifdef CONFIG_NETFILTER
14971502 /* we need to exclude all possible ENOPROTOOPTs except default case */
14981503 if (err == -ENOPROTOOPT && optname != IPV6_2292PKTOPTIONS) {
net/ipv6/mcast.c+4−4
@@ -580,7 +580,7 @@ done: · 579 unmodified lines
580580 }
581581
582582 int ip6_mc_msfget(struct sock *sk, struct group_filter *gsf,
583− struct sockaddr_storage __user *p)
583+ sockptr_t optval, size_t ss_offset)
584584 {
585585 struct ipv6_pinfo *inet6 = inet6_sk(sk);
586586 const struct in6_addr *group;
@@ -612,8 +612,7 @@ int ip6_mc_msfget(struct sock *sk, struct group_filter *gsf, · 25 unmodified lines
612612
613613 copycount = count < gsf->gf_numsrc ? count : gsf->gf_numsrc;
614614 gsf->gf_numsrc = count;
615−
616− for (i = 0; i < copycount; i++, p++) {
615+ for (i = 0; i < copycount; i++) {
617616 struct sockaddr_in6 *psin6;
618617 struct sockaddr_storage ss;
619618
@@ -621,8 +620,9 @@ int ip6_mc_msfget(struct sock *sk, struct group_filter *gsf, · 1 unmodified line
621620 memset(&ss, 0, sizeof(ss));
622621 psin6->sin6_family = AF_INET6;
623622 psin6->sin6_addr = psl->sl_addr[i];
624− if (copy_to_user(p, &ss, sizeof(ss)))
623+ if (copy_to_sockptr_offset(optval, ss_offset, &ss, sizeof(ss)))
625624 return -EFAULT;
625+ ss_offset += sizeof(ss);
626626 }
627627 return 0;
628628 }
samples/bpf/map_perf_test_kern.c+28−16
@@ -108,11 +108,14 @@ int stress_hmap(struct pt_regs *ctx) · 107 unmodified lines
108108 u32 key = bpf_get_current_pid_tgid();
109109 long init_val = 1;
110110 long *value;
111+ int i;
111112
112− bpf_map_update_elem(&hash_map, &key, &init_val, BPF_ANY);
113− value = bpf_map_lookup_elem(&hash_map, &key);
114− if (value)
115− bpf_map_delete_elem(&hash_map, &key);
113+ for (i = 0; i < 10; i++) {
114+ bpf_map_update_elem(&hash_map, &key, &init_val, BPF_ANY);
115+ value = bpf_map_lookup_elem(&hash_map, &key);
116+ if (value)
117+ bpf_map_delete_elem(&hash_map, &key);
118+ }
116119
117120 return 0;
118121 }
@@ -123,11 +126,14 @@ int stress_percpu_hmap(struct pt_regs *ctx) · 4 unmodified lines
123126 u32 key = bpf_get_current_pid_tgid();
124127 long init_val = 1;
125128 long *value;
129+ int i;
126130
127− bpf_map_update_elem(&percpu_hash_map, &key, &init_val, BPF_ANY);
128− value = bpf_map_lookup_elem(&percpu_hash_map, &key);
129− if (value)
130− bpf_map_delete_elem(&percpu_hash_map, &key);
131+ for (i = 0; i < 10; i++) {
132+ bpf_map_update_elem(&percpu_hash_map, &key, &init_val, BPF_ANY);
133+ value = bpf_map_lookup_elem(&percpu_hash_map, &key);
134+ if (value)
135+ bpf_map_delete_elem(&percpu_hash_map, &key);
136+ }
131137 return 0;
132138 }
133139
@@ -137,11 +143,14 @@ int stress_hmap_alloc(struct pt_regs *ctx) · 3 unmodified lines
137143 u32 key = bpf_get_current_pid_tgid();
138144 long init_val = 1;
139145 long *value;
146+ int i;
140147
141− bpf_map_update_elem(&hash_map_alloc, &key, &init_val, BPF_ANY);
142− value = bpf_map_lookup_elem(&hash_map_alloc, &key);
143− if (value)
144− bpf_map_delete_elem(&hash_map_alloc, &key);
148+ for (i = 0; i < 10; i++) {
149+ bpf_map_update_elem(&hash_map_alloc, &key, &init_val, BPF_ANY);
150+ value = bpf_map_lookup_elem(&hash_map_alloc, &key);
151+ if (value)
152+ bpf_map_delete_elem(&hash_map_alloc, &key);
153+ }
145154 return 0;
146155 }
147156
@@ -151,11 +160,14 @@ int stress_percpu_hmap_alloc(struct pt_regs *ctx) · 3 unmodified lines
151160 u32 key = bpf_get_current_pid_tgid();
152161 long init_val = 1;
153162 long *value;
163+ int i;
154164
155− bpf_map_update_elem(&percpu_hash_map_alloc, &key, &init_val, BPF_ANY);
156− value = bpf_map_lookup_elem(&percpu_hash_map_alloc, &key);
157− if (value)
158− bpf_map_delete_elem(&percpu_hash_map_alloc, &key);
165+ for (i = 0; i < 10; i++) {
166+ bpf_map_update_elem(&percpu_hash_map_alloc, &key, &init_val, BPF_ANY);
167+ value = bpf_map_lookup_elem(&percpu_hash_map_alloc, &key);
168+ if (value)
169+ bpf_map_delete_elem(&percpu_hash_map_alloc, &key);
170+ }
159171 return 0;
160172 }
161173
samples/bpf/map_perf_test_user.c+1−1
@@ -72,7 +72,7 @@ static int test_flags = ~0; · 71 unmodified lines
7272 static uint32_t num_map_entries;
7373 static uint32_t inner_lru_hash_size;
7474 static int lru_hash_lookup_test_entries = 32;
75−static uint32_t max_cnt = 1000000;
75+static uint32_t max_cnt = 10000;
7676
7777 static int check_test_flags(enum test_type t)
7878 {
scripts/bpf_doc.py+71−7
@@ -10,6 +10,9 @@ from __future__ import print_function · 9 unmodified lines
1010 import argparse
1111 import re
1212 import sys, os
13+import subprocess
14+
15+helpersDocStart = 'Start of BPF helper function descriptions:'
1316
1417 class NoHelperFound(BaseException):
1518 pass
@@ -47,6 +50,10 @@ class Helper(APIElement): · 31 unmodified lines
4750 @desc: textual description of the helper function
4851 @ret: description of the return value of the helper function
4952 """
53+ def __init__(self, *args, **kwargs):
54+ super().__init__(*args, **kwargs)
55+ self.enum_val = None
56+
5057 def proto_break_down(self):
5158 """
5259 Break down helper function protocol into smaller chunks: return type,
@@ -89,6 +96,7 @@ class HeaderParser(object): · 36 unmodified lines
8996 self.commands = []
9097 self.desc_unique_helpers = set()
9198 self.define_unique_helpers = []
99+ self.helper_enum_vals = {}
92100 self.desc_syscalls = []
93101 self.enum_syscalls = []
94102
@@ -233,7 +241,7 @@ class HeaderParser(object): · 138 unmodified lines
233241 self.enum_syscalls = re.findall('(BPF\w+)+', bpf_cmd_str)
234242
235243 def parse_desc_helpers(self):
236− self.seek_to('* Start of BPF helper function descriptions:',
244+ self.seek_to(helpersDocStart,
237245 'Could not find start of eBPF helper descriptions list')
238246 while True:
239247 try:
@@ -245,30 +253,54 @@ class HeaderParser(object): · 5 unmodified lines
245253 break
246254
247255 def parse_define_helpers(self):
248− # Parse the number of FN(...) in #define __BPF_FUNC_MAPPER to compare
249− # later with the number of unique function names present in description.
256+ # Parse FN(...) in #define __BPF_FUNC_MAPPER to compare later with the
257+ # number of unique function names present in description and use the
258+ # correct enumeration value.
250259 # Note: seek_to(..) discards the first line below the target search text,
251260 # resulting in FN(unspec) being skipped and not added to self.define_unique_helpers.
252261 self.seek_to('#define __BPF_FUNC_MAPPER(FN)',
253262 'Could not find start of eBPF helper definition list')
254− # Searches for either one or more FN(\w+) defines or a backslash for newline
255− p = re.compile('\s*(FN\(\w+\))+|\\\\')
263+ # Searches for one FN(\w+) define or a backslash for newline
264+ p = re.compile('\s*FN\((\w+)\)|\\\\')
256265 fn_defines_str = ''
266+ i = 1 # 'unspec' is skipped as mentioned above
257267 while True:
258268 capture = p.match(self.line)
259269 if capture:
260270 fn_defines_str += self.line
271+ self.helper_enum_vals[capture.expand(r'bpf_\1')] = i
272+ i += 1
261273 else:
262274 break
263275 self.line = self.reader.readline()
264276 # Find the number of occurences of FN(\w+)
265277 self.define_unique_helpers = re.findall('FN\(\w+\)', fn_defines_str)
266278
279+ def assign_helper_values(self):
280+ seen_helpers = set()
281+ for helper in self.helpers:
282+ proto = helper.proto_break_down()
283+ name = proto['name']
284+ try:
285+ enum_val = self.helper_enum_vals[name]
286+ except KeyError:
287+ raise Exception("Helper %s is missing from enum bpf_func_id" % name)
288+
289+ # Enforce current practice of having the descriptions ordered
290+ # by enum value.
291+ seen_helpers.add(name)
292+ desc_val = len(seen_helpers)
293+ if desc_val != enum_val:
294+ raise Exception("Helper %s comment order (#%d) must be aligned with its position (#%d) in enum bpf_func_id" % (name, desc_val, enum_val))
295+
296+ helper.enum_val = enum_val
297+
267298 def run(self):
268299 self.parse_desc_syscall()
269300 self.parse_enum_syscall()
270301 self.parse_desc_helpers()
271302 self.parse_define_helpers()
303+ self.assign_helper_values()
272304 self.reader.close()
273305
274306 ###############################################################################
@@ -357,6 +389,31 @@ class PrinterRST(Printer): · 82 unmodified lines
357389
358390 print('')
359391
392+ def get_kernel_version(self):
393+ try:
394+ version = subprocess.run(['git', 'describe'], cwd=linuxRoot,
395+ capture_output=True, check=True)
396+ version = version.stdout.decode().rstrip()
397+ except:
398+ try:
399+ version = subprocess.run(['make', 'kernelversion'], cwd=linuxRoot,
400+ capture_output=True, check=True)
401+ version = version.stdout.decode().rstrip()
402+ except:
403+ return 'Linux'
404+ return 'Linux {version}'.format(version=version)
405+
406+ def get_last_doc_update(self, delimiter):
407+ try:
408+ cmd = ['git', 'log', '-1', '--pretty=format:%cs', '--no-patch',
409+ '-L',
410+ '/{}/,/\*\//:include/uapi/linux/bpf.h'.format(delimiter)]
411+ date = subprocess.run(cmd, cwd=linuxRoot,
412+ capture_output=True, check=True)
413+ return date.stdout.decode().rstrip()
414+ except:
415+ return ''
416+
360417 class PrinterHelpersRST(PrinterRST):
361418 """
362419 A printer for dumping collected information about helpers as a ReStructured
@@ -378,6 +435,8 @@ list of eBPF helper functions · 15 unmodified lines
378435 -------------------------------------------------------------------------------
379436
380437 :Manual section: 7
438+:Version: {version}
439+{date_field}{date}
381440
382441 DESCRIPTION
383442 ===========
@@ -410,8 +469,13 @@ kernel at the top). · 26 unmodified lines
410469 HELPERS
411470 =======
412471 '''
472+ kernelVersion = self.get_kernel_version()
473+ lastUpdate = self.get_last_doc_update(helpersDocStart)
474+
413475 PrinterRST.print_license(self)
414− print(header)
476+ print(header.format(version=kernelVersion,
477+ date_field = ':Date: ' if lastUpdate else '',
478+ date=lastUpdate))
415479
416480 def print_footer(self):
417481 footer = '''
@@ -761,7 +825,7 @@ class PrinterHelpers(Printer): · 343 unmodified lines
761825 comma = ', '
762826 print(one_arg, end='')
763827
764− print(') = (void *) %d;' % len(self.seen_helpers))
828+ print(') = (void *) %d;' % helper.enum_val)
765829 print('')
766830
767831 ###############################################################################
tools/bpf/bpftool/btf_dumper.c+1−1
@@ -452,7 +452,7 @@ static int btf_dumper_int(const struct btf_type *t, __u8 bit_offset, · 451 unmodified lines
452452 *(char *)data);
453453 break;
454454 case BTF_INT_BOOL:
455− jsonw_bool(jw, *(int *)data);
455+ jsonw_bool(jw, *(bool *)data);
456456 break;
457457 default:
458458 /* shouldn't happen */
tools/bpf/bpftool/link.c+35−0
@@ -83,6 +83,29 @@ static bool is_iter_map_target(const char *target_name) · 82 unmodified lines
8383 strcmp(target_name, "bpf_sk_storage_map") == 0;
8484 }
8585
86+static bool is_iter_cgroup_target(const char *target_name)
87+{
88+ return strcmp(target_name, "cgroup") == 0;
89+}
90+
91+static const char *cgroup_order_string(__u32 order)
92+{
93+ switch (order) {
94+ case BPF_CGROUP_ITER_ORDER_UNSPEC:
95+ return "order_unspec";
96+ case BPF_CGROUP_ITER_SELF_ONLY:
97+ return "self_only";
98+ case BPF_CGROUP_ITER_DESCENDANTS_PRE:
99+ return "descendants_pre";
100+ case BPF_CGROUP_ITER_DESCENDANTS_POST:
101+ return "descendants_post";
102+ case BPF_CGROUP_ITER_ANCESTORS_UP:
103+ return "ancestors_up";
104+ default: /* won't happen */
105+ return "unknown";
106+ }
107+}
108+
86109 static void show_iter_json(struct bpf_link_info *info, json_writer_t *wtr)
87110 {
88111 const char *target_name = u64_to_ptr(info->iter.target_name);
@@ -91,6 +114,12 @@ static void show_iter_json(struct bpf_link_info *info, json_writer_t *wtr) · 2 unmodified lines
91114
92115 if (is_iter_map_target(target_name))
93116 jsonw_uint_field(wtr, "map_id", info->iter.map.map_id);
117+
118+ if (is_iter_cgroup_target(target_name)) {
119+ jsonw_lluint_field(wtr, "cgroup_id", info->iter.cgroup.cgroup_id);
120+ jsonw_string_field(wtr, "order",
121+ cgroup_order_string(info->iter.cgroup.order));
122+ }
94123 }
95124
96125 static int get_prog_info(int prog_id, struct bpf_prog_info *info)
@@ -208,6 +237,12 @@ static void show_iter_plain(struct bpf_link_info *info) · 111 unmodified lines
208237
209238 if (is_iter_map_target(target_name))
210239 printf("map_id %u ", info->iter.map.map_id);
240+
241+ if (is_iter_cgroup_target(target_name)) {
242+ printf("cgroup_id %llu ", info->iter.cgroup.cgroup_id);
243+ printf("order %s ",
244+ cgroup_order_string(info->iter.cgroup.order));
245+ }
211246 }
212247
213248 static int show_link_close_plain(int fd, struct bpf_link_info *info)
tools/include/uapi/linux/bpf.h+69−14
@@ -87,10 +87,29 @@ struct bpf_cgroup_storage_key { · 86 unmodified lines
8787 __u32 attach_type; /* program attach type (enum bpf_attach_type) */
8888 };
8989
90+enum bpf_cgroup_iter_order {
91+ BPF_CGROUP_ITER_ORDER_UNSPEC = 0,
92+ BPF_CGROUP_ITER_SELF_ONLY, /* process only a single object. */
93+ BPF_CGROUP_ITER_DESCENDANTS_PRE, /* walk descendants in pre-order. */
94+ BPF_CGROUP_ITER_DESCENDANTS_POST, /* walk descendants in post-order. */
95+ BPF_CGROUP_ITER_ANCESTORS_UP, /* walk ancestors upward. */
96+};
97+
9098 union bpf_iter_link_info {
9199 struct {
92100 __u32 map_fd;
93101 } map;
102+ struct {
103+ enum bpf_cgroup_iter_order order;
104+
105+ /* At most one of cgroup_fd and cgroup_id can be non-zero. If
106+ * both are zero, the walk starts from the default cgroup v2
107+ * root. For walking v1 hierarchy, one should always explicitly
108+ * specify cgroup_fd.
109+ */
110+ __u32 cgroup_fd;
111+ __u64 cgroup_id;
112+ } cgroup;
94113 };
95114
96115 /* BPF syscall commands, see bpf(2) man-page for more details. */
@@ -4437,7 +4456,7 @@ union bpf_attr { · 4340 unmodified lines
44374456 *
44384457 * **-EEXIST** if the option already exists.
44394458 *
4440− * **-EFAULT** on failrue to parse the existing header options.
4459+ * **-EFAULT** on failure to parse the existing header options.
44414460 *
44424461 * **-EPERM** if the helper cannot be used under the current
44434462 * *skops*\ **->op**.
@@ -4646,7 +4665,7 @@ union bpf_attr { · 202 unmodified lines
46464665 * a *map* with *task* as the **key**. From this
46474666 * perspective, the usage is not much different from
46484667 * **bpf_map_lookup_elem**\ (*map*, **&**\ *task*) except this
4649− * helper enforces the key must be an task_struct and the map must also
4668+ * helper enforces the key must be a task_struct and the map must also
46504669 * be a **BPF_MAP_TYPE_TASK_STORAGE**.
46514670 *
46524671 * Underneath, the value is stored locally at *task* instead of
@@ -4704,7 +4723,7 @@ union bpf_attr { · 51 unmodified lines
47044723 *
47054724 * long bpf_ima_inode_hash(struct inode *inode, void *dst, u32 size)
47064725 * Description
4707− * Returns the stored IMA hash of the *inode* (if it's avaialable).
4726+ * Returns the stored IMA hash of the *inode* (if it's available).
47084727 * If the hash is larger than *size*, then only *size*
47094728 * bytes will be copied to *dst*
47104729 * Return
@@ -4728,12 +4747,12 @@ union bpf_attr { · 17 unmodified lines
47284747 *
47294748 * The argument *len_diff* can be used for querying with a planned
47304749 * size change. This allows to check MTU prior to changing packet
4731− * ctx. Providing an *len_diff* adjustment that is larger than the
4750+ * ctx. Providing a *len_diff* adjustment that is larger than the
47324751 * actual packet size (resulting in negative packet size) will in
4733− * principle not exceed the MTU, why it is not considered a
4734− * failure. Other BPF-helpers are needed for performing the
4735− * planned size change, why the responsability for catch a negative
4736− * packet size belong in those helpers.
4752+ * principle not exceed the MTU, which is why it is not considered
4753+ * a failure. Other BPF helpers are needed for performing the
4754+ * planned size change; therefore the responsibility for catching
4755+ * a negative packet size belongs in those helpers.
47374756 *
47384757 * Specifying *ifindex* zero means the MTU check is performed
47394758 * against the current net device. This is practical if this isn't
@@ -5085,17 +5104,29 @@ union bpf_attr { · 345 unmodified lines
50855104 *
50865105 * int bpf_get_retval(void)
50875106 * Description
5088− * Get the syscall's return value that will be returned to userspace.
5107+ * Get the BPF program's return value that will be returned to the upper layers.
50895108 *
5090− * This helper is currently supported by cgroup programs only.
5109+ * This helper is currently supported by cgroup programs and only by the hooks
5110+ * where BPF program's return value is returned to the userspace via errno.
50915111 * Return
5092− * The syscall's return value.
5112+ * The BPF program's return value.
50935113 *
50945114 * int bpf_set_retval(int retval)
50955115 * Description
5096− * Set the syscall's return value that will be returned to userspace.
5116+ * Set the BPF program's return value that will be returned to the upper layers.
5117+ *
5118+ * This helper is currently supported by cgroup programs and only by the hooks
5119+ * where BPF program's return value is returned to the userspace via errno.
5120+ *
5121+ * Note that there is the following corner case where the program exports an error
5122+ * via bpf_set_retval but signals success via 'return 1':
5123+ *
5124+ * bpf_set_retval(-EPERM);
5125+ * return 1;
5126+ *
5127+ * In this case, the BPF program's return value will use helper's -EPERM. This
5128+ * still holds true for cgroup/bind{4,6} which supports extra 'return 3' success case.
50975129 *
5098− * This helper is currently supported by cgroup programs only.
50995130 * Return
51005131 * 0 on success, or a negative error in case of failure.
51015132 *
@@ -5628,6 +5659,11 @@ enum { · 526 unmodified lines
56285659 BPF_F_SEQ_NUMBER = (1ULL << 3),
56295660 };
56305661
5662+/* BPF_FUNC_skb_get_tunnel_key flags. */
5663+enum {
5664+ BPF_F_TUNINFO_FLAGS = (1ULL << 4),
5665+};
5666+
56315667 /* BPF_FUNC_perf_event_output, BPF_FUNC_perf_event_read and
56325668 * BPF_FUNC_perf_event_read_value flags.
56335669 */
@@ -5817,7 +5853,10 @@ struct bpf_tunnel_key { · 183 unmodified lines
58175853 };
58185854 __u8 tunnel_tos;
58195855 __u8 tunnel_ttl;
5820− __u16 tunnel_ext; /* Padding, future use. */
5856+ union {
5857+ __u16 tunnel_ext; /* compat */
5858+ __be16 tunnel_flags;
5859+ };
58215860 __u32 tunnel_label;
58225861 union {
58235862 __u32 local_ipv4;
@@ -5861,6 +5900,11 @@ enum bpf_ret_code { · 37 unmodified lines
58615900 * represented by BPF_REDIRECT above).
58625901 */
58635902 BPF_LWT_REROUTE = 128,
5903+ /* BPF_FLOW_DISSECTOR_CONTINUE: used by BPF_PROG_TYPE_FLOW_DISSECTOR
5904+ * to indicate that no custom dissection was performed, and
5905+ * fallback to standard dissector is requested.
5906+ */
5907+ BPF_FLOW_DISSECTOR_CONTINUE = 129,
58645908 };
58655909
58665910 struct bpf_sock {
@@ -6159,11 +6203,22 @@ struct bpf_link_info { · 292 unmodified lines
61596203 struct {
61606204 __aligned_u64 target_name; /* in/out: target_name buffer ptr */
61616205 __u32 target_name_len; /* in/out: target_name buffer len */
6206+
6207+ /* If the iter specific field is 32 bits, it can be put
6208+ * in the first or second union. Otherwise it should be
6209+ * put in the second union.
6210+ */
61626211 union {
61636212 struct {
61646213 __u32 map_id;
61656214 } map;
61666215 };
6216+ union {
6217+ struct {
6218+ __u64 cgroup_id;
6219+ __u32 order;
6220+ } cgroup;
6221+ };
61676222 } iter;
61686223 struct {
61696224 __u32 netns_ino;
tools/lib/bpf/bpf_helpers.h+13−6
@@ -131,7 +131,7 @@ · 130 unmodified lines
131131 /*
132132 * Helper function to perform a tail call with a constant/immediate map slot.
133133 */
134−#if __clang_major__ >= 8 && defined(__bpf__)
134+#if (!defined(__clang__) || __clang_major__ >= 8) && defined(__bpf__)
135135 static __always_inline void
136136 bpf_tail_call_static(void *ctx, const void *map, const __u32 slot)
137137 {
@@ -139,8 +139,8 @@ bpf_tail_call_static(void *ctx, const void *map, const __u32 slot) · 1 unmodified line
139139 __bpf_unreachable();
140140
141141 /*
142− * Provide a hard guarantee that LLVM won't optimize setting r2 (map
143− * pointer) and r3 (constant map index) from _different paths_ ending
142+ * Provide a hard guarantee that the compiler won't optimize setting r2
143+ * (map pointer) and r3 (constant map index) from _different paths_ ending
144144 * up at the _same_ call insn as otherwise we won't be able to use the
145145 * jmpq/nopl retpoline-free patching by the x86-64 JIT in the kernel
146146 * given they mismatch. See also d2e4c1e6c294 ("bpf: Constant map key
@@ -148,12 +148,19 @@ bpf_tail_call_static(void *ctx, const void *map, const __u32 slot) · 1 unmodified line
148148 *
149149 * Note on clobber list: we need to stay in-line with BPF calling
150150 * convention, so even if we don't end up using r0, r4, r5, we need
151− * to mark them as clobber so that LLVM doesn't end up using them
152− * before / after the call.
151+ * to mark them as clobber so that the compiler doesn't end up using
152+ * them before / after the call.
153153 */
154− asm volatile("r1 = %[ctx]\n\t"
154+ asm volatile(
155+#ifdef __clang__
156+ "r1 = %[ctx]\n\t"
155157 "r2 = %[map]\n\t"
156158 "r3 = %[slot]\n\t"
159+#else
160+ "mov %%r1,%[ctx]\n\t"
161+ "mov %%r2,%[map]\n\t"
162+ "mov %%r3,%[slot]\n\t"
163+#endif
157164 "call 12"
158165 :: [ctx]"r"(ctx), [map]"r"(map), [slot]"i"(slot)
159166 : "r0", "r1", "r2", "r3", "r4", "r5");
tools/lib/bpf/skel_internal.h+23−0
@@ -251,6 +251,29 @@ static inline int skel_map_update_elem(int fd, const void *key, · 250 unmodified lines
251251 return skel_sys_bpf(BPF_MAP_UPDATE_ELEM, &attr, attr_sz);
252252 }
253253
254+static inline int skel_map_delete_elem(int fd, const void *key)
255+{
256+ const size_t attr_sz = offsetofend(union bpf_attr, flags);
257+ union bpf_attr attr;
258+
259+ memset(&attr, 0, attr_sz);
260+ attr.map_fd = fd;
261+ attr.key = (long)key;
262+
263+ return skel_sys_bpf(BPF_MAP_DELETE_ELEM, &attr, attr_sz);
264+}
265+
266+static inline int skel_map_get_fd_by_id(__u32 id)
267+{
268+ const size_t attr_sz = offsetofend(union bpf_attr, flags);
269+ union bpf_attr attr;
270+
271+ memset(&attr, 0, attr_sz);
272+ attr.map_id = id;
273+
274+ return skel_sys_bpf(BPF_MAP_GET_FD_BY_ID, &attr, attr_sz);
275+}
276+
254277 static inline int skel_raw_tracepoint_open(const char *name, int prog_fd)
255278 {
256279 const size_t attr_sz = offsetofend(union bpf_attr, raw_tracepoint.prog_fd);
tools/testing/selftests/bpf/DENYLIST.s390x+4−0
@@ -66,3 +66,7 @@ select_reuseport # intermittently fails on new s390x set · 65 unmodified lines
6666 xdp_synproxy # JIT does not support calling kernel function (kfunc)
6767 unpriv_bpf_disabled # fentry
6868 lru_bug # prog 'printk': failed to auto-attach: -524
69+setget_sockopt # attach unexpected error: -524 (trampoline)
70+cb_refs # expected error message unexpected error: -524 (trampoline)
71+cgroup_hierarchical_stats # JIT does not support calling kernel function (kfunc)
72+htab_update # failed to attach: ERROR: strerror_r(-524)=22 (trampoline)
tools/testing/selftests/bpf/Makefile+19−18
@@ -45,7 +45,7 @@ ifneq ($(BPF_GCC),) · 44 unmodified lines
4545 TEST_GEN_PROGS += test_progs-bpf_gcc
4646 endif
4747
48−TEST_GEN_FILES = test_lwt_ip_encap.o test_tc_edt.o
48+TEST_GEN_FILES = test_lwt_ip_encap.bpf.o test_tc_edt.bpf.o
4949 TEST_FILES = xsk_prereqs.sh $(wildcard progs/btf_dump_test_case_*.c)
5050
5151 # Order correspond to 'make run_tests' order
@@ -323,6 +323,7 @@ $(OUTPUT)/test_l4lb_noinline.o: BPF_CFLAGS += -fno-inline · 271 unmodified lines
323323 $(OUTPUT)/test_xdp_noinline.o: BPF_CFLAGS += -fno-inline
324324
325325 $(OUTPUT)/flow_dissector_load.o: flow_dissector_load.h
326+$(OUTPUT)/cgroup_getset_retval_hooks.o: cgroup_getset_retval_hooks.h
326327
327328 # Build BPF object using Clang
328329 # $1 - input .c file
@@ -357,17 +358,17 @@ LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \ · 28 unmodified lines
357358 LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test_subprog.c
358359 SKEL_BLACKLIST += $$(LSKELS)
359360
360−test_static_linked.skel.h-deps := test_static_linked1.o test_static_linked2.o
361−linked_funcs.skel.h-deps := linked_funcs1.o linked_funcs2.o
362−linked_vars.skel.h-deps := linked_vars1.o linked_vars2.o
363−linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o
361+test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o
362+linked_funcs.skel.h-deps := linked_funcs1.bpf.o linked_funcs2.bpf.o
363+linked_vars.skel.h-deps := linked_vars1.bpf.o linked_vars2.bpf.o
364+linked_maps.skel.h-deps := linked_maps1.bpf.o linked_maps2.bpf.o
364365 # In the subskeleton case, we want the test_subskeleton_lib.subskel.h file
365366 # but that's created as a side-effect of the skel.h generation.
366−test_subskeleton.skel.h-deps := test_subskeleton_lib2.o test_subskeleton_lib.o test_subskeleton.o
367−test_subskeleton_lib.skel.h-deps := test_subskeleton_lib2.o test_subskeleton_lib.o
368−test_usdt.skel.h-deps := test_usdt.o test_usdt_multispec.o
367+test_subskeleton.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib.bpf.o test_subskeleton.bpf.o
368+test_subskeleton_lib.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib.bpf.o
369+test_usdt.skel.h-deps := test_usdt.bpf.o test_usdt_multispec.bpf.o
369370
370−LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps)))
371+LINKED_BPF_SRCS := $(patsubst %.bpf.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps)))
371372
372373 # Set up extra TRUNNER_XXX "temporary" variables in the environment (relies on
373374 # $eval()) and pass control to DEFINE_TEST_RUNNER_RULES.
@@ -385,7 +386,7 @@ TRUNNER_EXTRA_OBJS := $$(patsubst %.c,$$(TRUNNER_OUTPUT)/%.o, \ · 11 unmodified lines
385386 TRUNNER_EXTRA_HDRS := $$(filter %.h,$(TRUNNER_EXTRA_SOURCES))
386387 TRUNNER_TESTS_HDR := $(TRUNNER_TESTS_DIR)/tests.h
387388 TRUNNER_BPF_SRCS := $$(notdir $$(wildcard $(TRUNNER_BPF_PROGS_DIR)/*.c))
388−TRUNNER_BPF_OBJS := $$(patsubst %.c,$$(TRUNNER_OUTPUT)/%.o, $$(TRUNNER_BPF_SRCS))
389+TRUNNER_BPF_OBJS := $$(patsubst %.c,$$(TRUNNER_OUTPUT)/%.bpf.o, $$(TRUNNER_BPF_SRCS))
389390 TRUNNER_BPF_SKELS := $$(patsubst %.c,$$(TRUNNER_OUTPUT)/%.skel.h, \
390391 $$(filter-out $(SKEL_BLACKLIST) $(LINKED_BPF_SRCS),\
391392 $$(TRUNNER_BPF_SRCS)))
@@ -415,7 +416,7 @@ endif · 23 unmodified lines
415416 # input/output directory combination
416417 ifeq ($($(TRUNNER_BPF_PROGS_DIR)$(if $2,-)$2-bpfobjs),)
417418 $(TRUNNER_BPF_PROGS_DIR)$(if $2,-)$2-bpfobjs := y
418−$(TRUNNER_BPF_OBJS): $(TRUNNER_OUTPUT)/%.o: \
419+$(TRUNNER_BPF_OBJS): $(TRUNNER_OUTPUT)/%.bpf.o: \
419420 $(TRUNNER_BPF_PROGS_DIR)/%.c \
420421 $(TRUNNER_BPF_PROGS_DIR)/*.h \
421422 $$(INCLUDE_DIR)/vmlinux.h \
@@ -425,25 +426,25 @@ $(TRUNNER_BPF_OBJS): $(TRUNNER_OUTPUT)/%.o: \ · 3 unmodified lines
425426 $$(call $(TRUNNER_BPF_BUILD_RULE),$$<,$$@, \
426427 $(TRUNNER_BPF_CFLAGS))
427428
428−$(TRUNNER_BPF_SKELS): %.skel.h: %.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
429+$(TRUNNER_BPF_SKELS): %.skel.h: %.bpf.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
429430 $$(call msg,GEN-SKEL,$(TRUNNER_BINARY),$$@)
430431 $(Q)$$(BPFTOOL) gen object $$(<:.o=.linked1.o) $$<
431432 $(Q)$$(BPFTOOL) gen object $$(<:.o=.linked2.o) $$(<:.o=.linked1.o)
432433 $(Q)$$(BPFTOOL) gen object $$(<:.o=.linked3.o) $$(<:.o=.linked2.o)
433434 $(Q)diff $$(<:.o=.linked2.o) $$(<:.o=.linked3.o)
434− $(Q)$$(BPFTOOL) gen skeleton $$(<:.o=.linked3.o) name $$(notdir $$(<:.o=)) > $$@
435− $(Q)$$(BPFTOOL) gen subskeleton $$(<:.o=.linked3.o) name $$(notdir $$(<:.o=)) > $$(@:.skel.h=.subskel.h)
435+ $(Q)$$(BPFTOOL) gen skeleton $$(<:.o=.linked3.o) name $$(notdir $$(<:.bpf.o=)) > $$@
436+ $(Q)$$(BPFTOOL) gen subskeleton $$(<:.o=.linked3.o) name $$(notdir $$(<:.bpf.o=)) > $$(@:.skel.h=.subskel.h)
436437
437−$(TRUNNER_BPF_LSKELS): %.lskel.h: %.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
438+$(TRUNNER_BPF_LSKELS): %.lskel.h: %.bpf.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
438439 $$(call msg,GEN-SKEL,$(TRUNNER_BINARY),$$@)
439440 $(Q)$$(BPFTOOL) gen object $$(<:.o=.llinked1.o) $$<
440441 $(Q)$$(BPFTOOL) gen object $$(<:.o=.llinked2.o) $$(<:.o=.llinked1.o)
441442 $(Q)$$(BPFTOOL) gen object $$(<:.o=.llinked3.o) $$(<:.o=.llinked2.o)
442443 $(Q)diff $$(<:.o=.llinked2.o) $$(<:.o=.llinked3.o)
443− $(Q)$$(BPFTOOL) gen skeleton -L $$(<:.o=.llinked3.o) name $$(notdir $$(<:.o=_lskel)) > $$@
444+ $(Q)$$(BPFTOOL) gen skeleton -L $$(<:.o=.llinked3.o) name $$(notdir $$(<:.bpf.o=_lskel)) > $$@
444445
445446 $(TRUNNER_BPF_SKELS_LINKED): $(TRUNNER_BPF_OBJS) $(BPFTOOL) | $(TRUNNER_OUTPUT)
446− $$(call msg,LINK-BPF,$(TRUNNER_BINARY),$$(@:.skel.h=.o))
447+ $$(call msg,LINK-BPF,$(TRUNNER_BINARY),$$(@:.skel.h=.bpf.o))
447448 $(Q)$$(BPFTOOL) gen object $$(@:.skel.h=.linked1.o) $$(addprefix $(TRUNNER_OUTPUT)/,$$($$(@F)-deps))
448449 $(Q)$$(BPFTOOL) gen object $$(@:.skel.h=.linked2.o) $$(@:.skel.h=.linked1.o)
449450 $(Q)$$(BPFTOOL) gen object $$(@:.skel.h=.linked3.o) $$(@:.skel.h=.linked2.o)
@@ -499,7 +500,7 @@ $(OUTPUT)/$(TRUNNER_BINARY): $(TRUNNER_TEST_OBJS) \ · 49 unmodified lines
499500 | $(TRUNNER_BINARY)-extras
500501 $$(call msg,BINARY,,$$@)
501502 $(Q)$$(CC) $$(CFLAGS) $$(filter %.a %.o,$$^) $$(LDLIBS) -o $$@
502− $(Q)$(RESOLVE_BTFIDS) --btf $(TRUNNER_OUTPUT)/btf_data.o $$@
503+ $(Q)$(RESOLVE_BTFIDS) --btf $(TRUNNER_OUTPUT)/btf_data.bpf.o $$@
503504 $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool $(if $2,$2/)bpftool
504505
505506 endef
tools/testing/selftests/bpf/README.rst+4−4
@@ -126,11 +126,11 @@ available in 10.0.1. The patch is available in llvm 11.0.0 trunk. · 125 unmodified lines
126126
127127 __ https://reviews.llvm.org/D78466
128128
129−bpf_verif_scale/loop6.o test failure with Clang 12
130−==================================================
129+bpf_verif_scale/loop6.bpf.o test failure with Clang 12
130+======================================================
131131
132132 With Clang 12, the following bpf_verif_scale test failed:
133− * ``bpf_verif_scale/loop6.o``
133+ * ``bpf_verif_scale/loop6.bpf.o``
134134
135135 The verifier output looks like
136136
@@ -245,7 +245,7 @@ See `kernel llvm reloc`_ for more explanation and some examples. · 108 unmodified lines
245245 Using clang 13 to compile old libbpf which has static linker support,
246246 there will be a compilation failure::
247247
248− libbpf: ELF relo #0 in section #6 has unexpected type 2 in .../bpf_tcp_nogpl.o
248+ libbpf: ELF relo #0 in section #6 has unexpected type 2 in .../bpf_tcp_nogpl.bpf.o
249249
250250 Here, ``type 2`` refers to new relocation type ``R_BPF_64_ABS64``.
251251 To fix this issue, user newer libbpf.
tools/testing/selftests/bpf/cgroup_getset_retval_hooks.h+25−0
@@ -0,0 +1,25 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+
3+BPF_RETVAL_HOOK(ingress, "cgroup_skb/ingress", __sk_buff, -EINVAL)
4+BPF_RETVAL_HOOK(egress, "cgroup_skb/egress", __sk_buff, -EINVAL)
5+BPF_RETVAL_HOOK(sock_create, "cgroup/sock_create", bpf_sock, 0)
6+BPF_RETVAL_HOOK(sock_ops, "sockops", bpf_sock_ops, -EINVAL)
7+BPF_RETVAL_HOOK(dev, "cgroup/dev", bpf_cgroup_dev_ctx, 0)
8+BPF_RETVAL_HOOK(bind4, "cgroup/bind4", bpf_sock_addr, 0)
9+BPF_RETVAL_HOOK(bind6, "cgroup/bind6", bpf_sock_addr, 0)
10+BPF_RETVAL_HOOK(connect4, "cgroup/connect4", bpf_sock_addr, 0)
11+BPF_RETVAL_HOOK(connect6, "cgroup/connect6", bpf_sock_addr, 0)
12+BPF_RETVAL_HOOK(post_bind4, "cgroup/post_bind4", bpf_sock_addr, 0)
13+BPF_RETVAL_HOOK(post_bind6, "cgroup/post_bind6", bpf_sock_addr, 0)
14+BPF_RETVAL_HOOK(sendmsg4, "cgroup/sendmsg4", bpf_sock_addr, 0)
15+BPF_RETVAL_HOOK(sendmsg6, "cgroup/sendmsg6", bpf_sock_addr, 0)
16+BPF_RETVAL_HOOK(sysctl, "cgroup/sysctl", bpf_sysctl, 0)
17+BPF_RETVAL_HOOK(recvmsg4, "cgroup/recvmsg4", bpf_sock_addr, -EINVAL)
18+BPF_RETVAL_HOOK(recvmsg6, "cgroup/recvmsg6", bpf_sock_addr, -EINVAL)
19+BPF_RETVAL_HOOK(getsockopt, "cgroup/getsockopt", bpf_sockopt, 0)
20+BPF_RETVAL_HOOK(setsockopt, "cgroup/setsockopt", bpf_sockopt, 0)
21+BPF_RETVAL_HOOK(getpeername4, "cgroup/getpeername4", bpf_sock_addr, -EINVAL)
22+BPF_RETVAL_HOOK(getpeername6, "cgroup/getpeername6", bpf_sock_addr, -EINVAL)
23+BPF_RETVAL_HOOK(getsockname4, "cgroup/getsockname4", bpf_sock_addr, -EINVAL)
24+BPF_RETVAL_HOOK(getsockname6, "cgroup/getsockname6", bpf_sock_addr, -EINVAL)
25+BPF_RETVAL_HOOK(sock_release, "cgroup/sock_release", bpf_sock, 0)
tools/testing/selftests/bpf/cgroup_helpers.c+161−41
@@ -33,49 +33,52 @@ · 32 unmodified lines
3333 #define CGROUP_MOUNT_DFLT "/sys/fs/cgroup"
3434 #define NETCLS_MOUNT_PATH CGROUP_MOUNT_DFLT "/net_cls"
3535 #define CGROUP_WORK_DIR "/cgroup-test-work-dir"
36−#define format_cgroup_path(buf, path) \
36+
37+#define format_cgroup_path_pid(buf, path, pid) \
3738 snprintf(buf, sizeof(buf), "%s%s%d%s", CGROUP_MOUNT_PATH, \
38− CGROUP_WORK_DIR, getpid(), path)
39+ CGROUP_WORK_DIR, pid, path)
40+
41+#define format_cgroup_path(buf, path) \
42+ format_cgroup_path_pid(buf, path, getpid())
43+
44+#define format_parent_cgroup_path(buf, path) \
45+ format_cgroup_path_pid(buf, path, getppid())
3946
4047 #define format_classid_path(buf) \
4148 snprintf(buf, sizeof(buf), "%s%s", NETCLS_MOUNT_PATH, \
4249 CGROUP_WORK_DIR)
4350
44−/**
45− * enable_all_controllers() - Enable all available cgroup v2 controllers
46− *
47− * Enable all available cgroup v2 controllers in order to increase
48− * the code coverage.
49− *
50− * If successful, 0 is returned.
51− */
52−static int enable_all_controllers(char *cgroup_path)
51+static int __enable_controllers(const char *cgroup_path, const char *controllers)
5352 {
5453 char path[PATH_MAX + 1];
55− char buf[PATH_MAX];
54+ char enable[PATH_MAX + 1];
5655 char *c, *c2;
5756 int fd, cfd;
5857 ssize_t len;
5958
60− snprintf(path, sizeof(path), "%s/cgroup.controllers", cgroup_path);
61− fd = open(path, O_RDONLY);
62− if (fd < 0) {
63− log_err("Opening cgroup.controllers: %s", path);
64− return 1;
65− }
66−
67− len = read(fd, buf, sizeof(buf) - 1);
68− if (len < 0) {
59+ /* If not controllers are passed, enable all available controllers */
60+ if (!controllers) {
61+ snprintf(path, sizeof(path), "%s/cgroup.controllers",
62+ cgroup_path);
63+ fd = open(path, O_RDONLY);
64+ if (fd < 0) {
65+ log_err("Opening cgroup.controllers: %s", path);
66+ return 1;
67+ }
68+ len = read(fd, enable, sizeof(enable) - 1);
69+ if (len < 0) {
70+ close(fd);
71+ log_err("Reading cgroup.controllers: %s", path);
72+ return 1;
73+ } else if (len == 0) { /* No controllers to enable */
74+ close(fd);
75+ return 0;
76+ }
77+ enable[len] = 0;
6978 close(fd);
70− log_err("Reading cgroup.controllers: %s", path);
71− return 1;
79+ } else {
80+ strncpy(enable, controllers, sizeof(enable));
7281 }
73− buf[len] = 0;
74− close(fd);
75−
76− /* No controllers available? We're probably on cgroup v1. */
77− if (len == 0)
78− return 0;
7982
8083 snprintf(path, sizeof(path), "%s/cgroup.subtree_control", cgroup_path);
8184 cfd = open(path, O_RDWR);
@@ -84,7 +87,7 @@ static int enable_all_controllers(char *cgroup_path) · 2 unmodified lines
8487 return 1;
8588 }
8689
87− for (c = strtok_r(buf, " ", &c2); c; c = strtok_r(NULL, " ", &c2)) {
90+ for (c = strtok_r(enable, " ", &c2); c; c = strtok_r(NULL, " ", &c2)) {
8891 if (dprintf(cfd, "+%s\n", c) <= 0) {
8992 log_err("Enabling controller %s: %s", c, path);
9093 close(cfd);
@@ -95,6 +98,87 @@ static int enable_all_controllers(char *cgroup_path) · 4 unmodified lines
9598 return 0;
9699 }
97100
101+/**
102+ * enable_controllers() - Enable cgroup v2 controllers
103+ * @relative_path: The cgroup path, relative to the workdir
104+ * @controllers: List of controllers to enable in cgroup.controllers format
105+ *
106+ *
107+ * Enable given cgroup v2 controllers, if @controllers is NULL, enable all
108+ * available controllers.
109+ *
110+ * If successful, 0 is returned.
111+ */
112+int enable_controllers(const char *relative_path, const char *controllers)
113+{
114+ char cgroup_path[PATH_MAX + 1];
115+
116+ format_cgroup_path(cgroup_path, relative_path);
117+ return __enable_controllers(cgroup_path, controllers);
118+}
119+
120+static int __write_cgroup_file(const char *cgroup_path, const char *file,
121+ const char *buf)
122+{
123+ char file_path[PATH_MAX + 1];
124+ int fd;
125+
126+ snprintf(file_path, sizeof(file_path), "%s/%s", cgroup_path, file);
127+ fd = open(file_path, O_RDWR);
128+ if (fd < 0) {
129+ log_err("Opening %s", file_path);
130+ return 1;
131+ }
132+
133+ if (dprintf(fd, "%s", buf) <= 0) {
134+ log_err("Writing to %s", file_path);
135+ close(fd);
136+ return 1;
137+ }
138+ close(fd);
139+ return 0;
140+}
141+
142+/**
143+ * write_cgroup_file() - Write to a cgroup file
144+ * @relative_path: The cgroup path, relative to the workdir
145+ * @file: The name of the file in cgroupfs to write to
146+ * @buf: Buffer to write to the file
147+ *
148+ * Write to a file in the given cgroup's directory.
149+ *
150+ * If successful, 0 is returned.
151+ */
152+int write_cgroup_file(const char *relative_path, const char *file,
153+ const char *buf)
154+{
155+ char cgroup_path[PATH_MAX - 24];
156+
157+ format_cgroup_path(cgroup_path, relative_path);
158+ return __write_cgroup_file(cgroup_path, file, buf);
159+}
160+
161+/**
162+ * write_cgroup_file_parent() - Write to a cgroup file in the parent process
163+ * workdir
164+ * @relative_path: The cgroup path, relative to the parent process workdir
165+ * @file: The name of the file in cgroupfs to write to
166+ * @buf: Buffer to write to the file
167+ *
168+ * Write to a file in the given cgroup's directory under the parent process
169+ * workdir.
170+ *
171+ * If successful, 0 is returned.
172+ */
173+int write_cgroup_file_parent(const char *relative_path, const char *file,
174+ const char *buf)
175+{
176+ char cgroup_path[PATH_MAX - 24];
177+
178+ format_parent_cgroup_path(cgroup_path, relative_path);
179+ return __write_cgroup_file(cgroup_path, file, buf);
180+}
181+
98182 /**
99183 * setup_cgroup_environment() - Setup the cgroup environment
100184 *
@@ -133,7 +217,9 @@ int setup_cgroup_environment(void) · 32 unmodified lines
133217 return 1;
134218 }
135219
136− if (enable_all_controllers(cgroup_workdir))
220+ /* Enable all available controllers to increase test coverage */
221+ if (__enable_controllers(CGROUP_MOUNT_PATH, NULL) ||
222+ __enable_controllers(cgroup_workdir, NULL))
137223 return 1;
138224
139225 return 0;
@@ -173,7 +259,7 @@ static int join_cgroup_from_top(const char *cgroup_path) · 33 unmodified lines
173259
174260 /**
175261 * join_cgroup() - Join a cgroup
176− * @path: The cgroup path, relative to the workdir, to join
262+ * @relative_path: The cgroup path, relative to the workdir, to join
177263 *
178264 * This function expects a cgroup to already be created, relative to the cgroup
179265 * work dir, and it joins it. For example, passing "/my-cgroup" as the path
@@ -182,11 +268,27 @@ static int join_cgroup_from_top(const char *cgroup_path) · 2 unmodified lines
182268 *
183269 * On success, it returns 0, otherwise on failure it returns 1.
184270 */
185−int join_cgroup(const char *path)
271+int join_cgroup(const char *relative_path)
272+{
273+ char cgroup_path[PATH_MAX + 1];
274+
275+ format_cgroup_path(cgroup_path, relative_path);
276+ return join_cgroup_from_top(cgroup_path);
277+}
278+
279+/**
280+ * join_parent_cgroup() - Join a cgroup in the parent process workdir
281+ * @relative_path: The cgroup path, relative to parent process workdir, to join
282+ *
283+ * See join_cgroup().
284+ *
285+ * On success, it returns 0, otherwise on failure it returns 1.
286+ */
287+int join_parent_cgroup(const char *relative_path)
186288 {
187289 char cgroup_path[PATH_MAX + 1];
188290
189− format_cgroup_path(cgroup_path, path);
291+ format_parent_cgroup_path(cgroup_path, relative_path);
190292 return join_cgroup_from_top(cgroup_path);
191293 }
192294
@@ -212,9 +314,27 @@ void cleanup_cgroup_environment(void) · 19 unmodified lines
212314 nftw(cgroup_workdir, nftwfunc, WALK_FD_LIMIT, FTW_DEPTH | FTW_MOUNT);
213315 }
214316
317+/**
318+ * get_root_cgroup() - Get the FD of the root cgroup
319+ *
320+ * On success, it returns the file descriptor. On failure, it returns -1.
321+ * If there is a failure, it prints the error to stderr.
322+ */
323+int get_root_cgroup(void)
324+{
325+ int fd;
326+
327+ fd = open(CGROUP_MOUNT_PATH, O_RDONLY);
328+ if (fd < 0) {
329+ log_err("Opening root cgroup");
330+ return -1;
331+ }
332+ return fd;
333+}
334+
215335 /**
216336 * create_and_get_cgroup() - Create a cgroup, relative to workdir, and get the FD
217− * @path: The cgroup path, relative to the workdir, to join
337+ * @relative_path: The cgroup path, relative to the workdir, to join
218338 *
219339 * This function creates a cgroup under the top level workdir and returns the
220340 * file descriptor. It is idempotent.
@@ -222,14 +342,14 @@ void cleanup_cgroup_environment(void) · 1 unmodified line
222342 * On success, it returns the file descriptor. On failure it returns -1.
223343 * If there is a failure, it prints the error to stderr.
224344 */
225−int create_and_get_cgroup(const char *path)
345+int create_and_get_cgroup(const char *relative_path)
226346 {
227347 char cgroup_path[PATH_MAX + 1];
228348 int fd;
229349
230− format_cgroup_path(cgroup_path, path);
350+ format_cgroup_path(cgroup_path, relative_path);
231351 if (mkdir(cgroup_path, 0777) && errno != EEXIST) {
232− log_err("mkdiring cgroup %s .. %s", path, cgroup_path);
352+ log_err("mkdiring cgroup %s .. %s", relative_path, cgroup_path);
233353 return -1;
234354 }
235355
@@ -244,13 +364,13 @@ int create_and_get_cgroup(const char *path) · 8 unmodified lines
244364
245365 /**
246366 * get_cgroup_id() - Get cgroup id for a particular cgroup path
247− * @path: The cgroup path, relative to the workdir, to join
367+ * @relative_path: The cgroup path, relative to the workdir, to join
248368 *
249369 * On success, it returns the cgroup id. On failure it returns 0,
250370 * which is an invalid cgroup id.
251371 * If there is a failure, it prints the error to stderr.
252372 */
253−unsigned long long get_cgroup_id(const char *path)
373+unsigned long long get_cgroup_id(const char *relative_path)
254374 {
255375 int dirfd, err, flags, mount_id, fhsize;
256376 union {
@@ -261,7 +381,7 @@ unsigned long long get_cgroup_id(const char *path) · 4 unmodified lines
261381 struct file_handle *fhp, *fhp2;
262382 unsigned long long ret = 0;
263383
264− format_cgroup_path(cgroup_workdir, path);
384+ format_cgroup_path(cgroup_workdir, relative_path);
265385
266386 dirfd = AT_FDCWD;
267387 flags = 0;
tools/testing/selftests/bpf/cgroup_helpers.h+13−6
@@ -10,11 +10,18 @@ · 9 unmodified lines
1010 __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)
1111
1212 /* cgroupv2 related */
13−int cgroup_setup_and_join(const char *path);
14−int create_and_get_cgroup(const char *path);
15−unsigned long long get_cgroup_id(const char *path);
16−
17−int join_cgroup(const char *path);
13+int enable_controllers(const char *relative_path, const char *controllers);
14+int write_cgroup_file(const char *relative_path, const char *file,
15+ const char *buf);
16+int write_cgroup_file_parent(const char *relative_path, const char *file,
17+ const char *buf);
18+int cgroup_setup_and_join(const char *relative_path);
19+int get_root_cgroup(void);
20+int create_and_get_cgroup(const char *relative_path);
21+unsigned long long get_cgroup_id(const char *relative_path);
22+
23+int join_cgroup(const char *relative_path);
24+int join_parent_cgroup(const char *relative_path);
1825
1926 int setup_cgroup_environment(void);
2027 void cleanup_cgroup_environment(void);
@@ -26,4 +33,4 @@ int join_classid(void); · 5 unmodified lines
2633 int setup_classid_environment(void);
2734 void cleanup_classid_environment(void);
2835
29−#endif /* __CGROUP_HELPERS_H */
36+#endif /* __CGROUP_HELPERS_H */
tools/testing/selftests/bpf/get_cgroup_id_user.c+1−1
@@ -48,7 +48,7 @@ static int bpf_find_map(const char *test, struct bpf_object *obj, · 47 unmodified lines
4848 int main(int argc, char **argv)
4949 {
5050 const char *probe_name = "syscalls/sys_enter_nanosleep";
51− const char *file = "get_cgroup_id_kern.o";
51+ const char *file = "get_cgroup_id_kern.bpf.o";
5252 int err, bytes, efd, prog_fd, pmu_fd;
5353 int cgroup_fd, cgidmap_fd, pidmap_fd;
5454 struct perf_event_attr attr = {};
tools/testing/selftests/bpf/map_tests/task_storage_map.c+122−0
@@ -0,0 +1,122 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (C) 2022. Huawei Technologies Co., Ltd */
3+#define _GNU_SOURCE
4+#include <sched.h>
5+#include <unistd.h>
6+#include <stdlib.h>
7+#include <stdbool.h>
8+#include <errno.h>
9+#include <string.h>
10+#include <pthread.h>
11+
12+#include <bpf/bpf.h>
13+#include <bpf/libbpf.h>
14+
15+#include "test_maps.h"
16+#include "task_local_storage_helpers.h"
17+#include "read_bpf_task_storage_busy.skel.h"
18+
19+struct lookup_ctx {
20+ bool start;
21+ bool stop;
22+ int pid_fd;
23+ int map_fd;
24+ int loop;
25+};
26+
27+static void *lookup_fn(void *arg)
28+{
29+ struct lookup_ctx *ctx = arg;
30+ long value;
31+ int i = 0;
32+
33+ while (!ctx->start)
34+ usleep(1);
35+
36+ while (!ctx->stop && i++ < ctx->loop)
37+ bpf_map_lookup_elem(ctx->map_fd, &ctx->pid_fd, &value);
38+ return NULL;
39+}
40+
41+static void abort_lookup(struct lookup_ctx *ctx, pthread_t *tids, unsigned int nr)
42+{
43+ unsigned int i;
44+
45+ ctx->stop = true;
46+ ctx->start = true;
47+ for (i = 0; i < nr; i++)
48+ pthread_join(tids[i], NULL);
49+}
50+
51+void test_task_storage_map_stress_lookup(void)
52+{
53+#define MAX_NR_THREAD 4096
54+ unsigned int i, nr = 256, loop = 8192, cpu = 0;
55+ struct read_bpf_task_storage_busy *skel;
56+ pthread_t tids[MAX_NR_THREAD];
57+ struct lookup_ctx ctx;
58+ cpu_set_t old, new;
59+ const char *cfg;
60+ int err;
61+
62+ cfg = getenv("TASK_STORAGE_MAP_NR_THREAD");
63+ if (cfg) {
64+ nr = atoi(cfg);
65+ if (nr > MAX_NR_THREAD)
66+ nr = MAX_NR_THREAD;
67+ }
68+ cfg = getenv("TASK_STORAGE_MAP_NR_LOOP");
69+ if (cfg)
70+ loop = atoi(cfg);
71+ cfg = getenv("TASK_STORAGE_MAP_PIN_CPU");
72+ if (cfg)
73+ cpu = atoi(cfg);
74+
75+ skel = read_bpf_task_storage_busy__open_and_load();
76+ err = libbpf_get_error(skel);
77+ CHECK(err, "open_and_load", "error %d\n", err);
78+
79+ /* Only for a fully preemptible kernel */
80+ if (!skel->kconfig->CONFIG_PREEMPT)
81+ return;
82+
83+ /* Save the old affinity setting */
84+ sched_getaffinity(getpid(), sizeof(old), &old);
85+
86+ /* Pinned on a specific CPU */
87+ CPU_ZERO(&new);
88+ CPU_SET(cpu, &new);
89+ sched_setaffinity(getpid(), sizeof(new), &new);
90+
91+ ctx.start = false;
92+ ctx.stop = false;
93+ ctx.pid_fd = sys_pidfd_open(getpid(), 0);
94+ ctx.map_fd = bpf_map__fd(skel->maps.task);
95+ ctx.loop = loop;
96+ for (i = 0; i < nr; i++) {
97+ err = pthread_create(&tids[i], NULL, lookup_fn, &ctx);
98+ if (err) {
99+ abort_lookup(&ctx, tids, i);
100+ CHECK(err, "pthread_create", "error %d\n", err);
101+ goto out;
102+ }
103+ }
104+
105+ ctx.start = true;
106+ for (i = 0; i < nr; i++)
107+ pthread_join(tids[i], NULL);
108+
109+ skel->bss->pid = getpid();
110+ err = read_bpf_task_storage_busy__attach(skel);
111+ CHECK(err, "attach", "error %d\n", err);
112+
113+ /* Trigger program */
114+ syscall(SYS_gettid);
115+ skel->bss->pid = 0;
116+
117+ CHECK(skel->bss->busy != 0, "bad bpf_task_storage_busy", "got %d\n", skel->bss->busy);
118+out:
119+ read_bpf_task_storage_busy__destroy(skel);
120+ /* Restore affinity setting */
121+ sched_setaffinity(getpid(), sizeof(old), &old);
122+}
tools/testing/selftests/bpf/prog_tests/bpf_obj_id.c+1−1
@@ -7,7 +7,7 @@ void serial_test_bpf_obj_id(void) · 6 unmodified lines
77 {
88 const __u64 array_magic_value = 0xfaceb00c;
99 const __u32 array_key = 0;
10− const char *file = "./test_obj_id.o";
10+ const char *file = "./test_obj_id.bpf.o";
1111 const char *expected_prog_name = "test_obj_id";
1212 const char *expected_map_name = "test_map_id";
1313 const __u64 nsec_per_sec = 1000000000;
tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c+27−27
@@ -75,45 +75,45 @@ static void scale_test(const char *file, · 74 unmodified lines
7575
7676 void test_verif_scale1()
7777 {
78− scale_test("test_verif_scale1.o", BPF_PROG_TYPE_SCHED_CLS, false);
78+ scale_test("test_verif_scale1.bpf.o", BPF_PROG_TYPE_SCHED_CLS, false);
7979 }
8080
8181 void test_verif_scale2()
8282 {
83− scale_test("test_verif_scale2.o", BPF_PROG_TYPE_SCHED_CLS, false);
83+ scale_test("test_verif_scale2.bpf.o", BPF_PROG_TYPE_SCHED_CLS, false);
8484 }
8585
8686 void test_verif_scale3()
8787 {
88− scale_test("test_verif_scale3.o", BPF_PROG_TYPE_SCHED_CLS, false);
88+ scale_test("test_verif_scale3.bpf.o", BPF_PROG_TYPE_SCHED_CLS, false);
8989 }
9090
9191 void test_verif_scale_pyperf_global()
9292 {
93− scale_test("pyperf_global.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
93+ scale_test("pyperf_global.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
9494 }
9595
9696 void test_verif_scale_pyperf_subprogs()
9797 {
98− scale_test("pyperf_subprogs.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
98+ scale_test("pyperf_subprogs.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
9999 }
100100
101101 void test_verif_scale_pyperf50()
102102 {
103103 /* full unroll by llvm */
104− scale_test("pyperf50.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
104+ scale_test("pyperf50.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
105105 }
106106
107107 void test_verif_scale_pyperf100()
108108 {
109109 /* full unroll by llvm */
110− scale_test("pyperf100.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
110+ scale_test("pyperf100.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
111111 }
112112
113113 void test_verif_scale_pyperf180()
114114 {
115115 /* full unroll by llvm */
116− scale_test("pyperf180.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
116+ scale_test("pyperf180.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
117117 }
118118
119119 void test_verif_scale_pyperf600()
@@ -124,13 +124,13 @@ void test_verif_scale_pyperf600() · 4 unmodified lines
124124 * 16k insns in loop body.
125125 * Total of 5 such loops. Total program size ~82k insns.
126126 */
127− scale_test("pyperf600.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
127+ scale_test("pyperf600.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
128128 }
129129
130130 void test_verif_scale_pyperf600_bpf_loop(void)
131131 {
132132 /* use the bpf_loop helper*/
133− scale_test("pyperf600_bpf_loop.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
133+ scale_test("pyperf600_bpf_loop.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
134134 }
135135
136136 void test_verif_scale_pyperf600_nounroll()
@@ -141,37 +141,37 @@ void test_verif_scale_pyperf600_nounroll() · 4 unmodified lines
141141 * ~110 insns in loop body.
142142 * Total of 5 such loops. Total program size ~1500 insns.
143143 */
144− scale_test("pyperf600_nounroll.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
144+ scale_test("pyperf600_nounroll.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
145145 }
146146
147147 void test_verif_scale_loop1()
148148 {
149− scale_test("loop1.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
149+ scale_test("loop1.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
150150 }
151151
152152 void test_verif_scale_loop2()
153153 {
154− scale_test("loop2.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
154+ scale_test("loop2.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
155155 }
156156
157157 void test_verif_scale_loop3_fail()
158158 {
159− scale_test("loop3.o", BPF_PROG_TYPE_RAW_TRACEPOINT, true /* fails */);
159+ scale_test("loop3.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, true /* fails */);
160160 }
161161
162162 void test_verif_scale_loop4()
163163 {
164− scale_test("loop4.o", BPF_PROG_TYPE_SCHED_CLS, false);
164+ scale_test("loop4.bpf.o", BPF_PROG_TYPE_SCHED_CLS, false);
165165 }
166166
167167 void test_verif_scale_loop5()
168168 {
169− scale_test("loop5.o", BPF_PROG_TYPE_SCHED_CLS, false);
169+ scale_test("loop5.bpf.o", BPF_PROG_TYPE_SCHED_CLS, false);
170170 }
171171
172172 void test_verif_scale_loop6()
173173 {
174− scale_test("loop6.o", BPF_PROG_TYPE_KPROBE, false);
174+ scale_test("loop6.bpf.o", BPF_PROG_TYPE_KPROBE, false);
175175 }
176176
177177 void test_verif_scale_strobemeta()
@@ -180,54 +180,54 @@ void test_verif_scale_strobemeta() · 2 unmodified lines
180180 * Total program size 20.8k insn.
181181 * ~350k processed_insns
182182 */
183− scale_test("strobemeta.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
183+ scale_test("strobemeta.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
184184 }
185185
186186 void test_verif_scale_strobemeta_bpf_loop(void)
187187 {
188188 /* use the bpf_loop helper*/
189− scale_test("strobemeta_bpf_loop.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
189+ scale_test("strobemeta_bpf_loop.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
190190 }
191191
192192 void test_verif_scale_strobemeta_nounroll1()
193193 {
194194 /* no unroll, tiny loops */
195− scale_test("strobemeta_nounroll1.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
195+ scale_test("strobemeta_nounroll1.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
196196 }
197197
198198 void test_verif_scale_strobemeta_nounroll2()
199199 {
200200 /* no unroll, tiny loops */
201− scale_test("strobemeta_nounroll2.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
201+ scale_test("strobemeta_nounroll2.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
202202 }
203203
204204 void test_verif_scale_strobemeta_subprogs()
205205 {
206206 /* non-inlined subprogs */
207− scale_test("strobemeta_subprogs.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
207+ scale_test("strobemeta_subprogs.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
208208 }
209209
210210 void test_verif_scale_sysctl_loop1()
211211 {
212− scale_test("test_sysctl_loop1.o", BPF_PROG_TYPE_CGROUP_SYSCTL, false);
212+ scale_test("test_sysctl_loop1.bpf.o", BPF_PROG_TYPE_CGROUP_SYSCTL, false);
213213 }
214214
215215 void test_verif_scale_sysctl_loop2()
216216 {
217− scale_test("test_sysctl_loop2.o", BPF_PROG_TYPE_CGROUP_SYSCTL, false);
217+ scale_test("test_sysctl_loop2.bpf.o", BPF_PROG_TYPE_CGROUP_SYSCTL, false);
218218 }
219219
220220 void test_verif_scale_xdp_loop()
221221 {
222− scale_test("test_xdp_loop.o", BPF_PROG_TYPE_XDP, false);
222+ scale_test("test_xdp_loop.bpf.o", BPF_PROG_TYPE_XDP, false);
223223 }
224224
225225 void test_verif_scale_seg6_loop()
226226 {
227− scale_test("test_seg6_loop.o", BPF_PROG_TYPE_LWT_SEG6LOCAL, false);
227+ scale_test("test_seg6_loop.bpf.o", BPF_PROG_TYPE_LWT_SEG6LOCAL, false);
228228 }
229229
230230 void test_verif_twfw()
231231 {
232− scale_test("twfw.o", BPF_PROG_TYPE_CGROUP_SKB, false);
232+ scale_test("twfw.bpf.o", BPF_PROG_TYPE_CGROUP_SKB, false);
233233 }
tools/testing/selftests/bpf/prog_tests/btf.c+2−2
@@ -4651,8 +4651,8 @@ struct btf_file_test { · 4650 unmodified lines
46514651 };
46524652
46534653 static struct btf_file_test file_tests[] = {
4654− { .file = "test_btf_newkv.o", },
4655− { .file = "test_btf_nokv.o", .btf_kv_notfound = true, },
4654+ { .file = "test_btf_newkv.bpf.o", },
4655+ { .file = "test_btf_nokv.bpf.o", .btf_kv_notfound = true, },
46564656 };
46574657
46584658 static void do_test_file(unsigned int test_num)
tools/testing/selftests/bpf/prog_tests/btf_dump.c+5−5
@@ -52,7 +52,7 @@ static int test_btf_dump_case(int n, struct btf_dump_test_case *t) · 51 unmodified lines
5252 int err = 0, fd = -1;
5353 FILE *f = NULL;
5454
55− snprintf(test_file, sizeof(test_file), "%s.o", t->file);
55+ snprintf(test_file, sizeof(test_file), "%s.bpf.o", t->file);
5656
5757 btf = btf__parse_elf(test_file, NULL);
5858 if (!ASSERT_OK_PTR(btf, "btf_parse_elf")) {
@@ -764,8 +764,8 @@ static void test_btf_dump_struct_data(struct btf *btf, struct btf_dump *d, · 705 unmodified lines
764764
765765 /* union with nested struct */
766766 TEST_BTF_DUMP_DATA(btf, d, "union", str, union bpf_iter_link_info, BTF_F_COMPACT,
767− "(union bpf_iter_link_info){.map = (struct){.map_fd = (__u32)1,},}",
768− { .map = { .map_fd = 1 }});
767+ "(union bpf_iter_link_info){.map = (struct){.map_fd = (__u32)1,},.cgroup = (struct){.order = (enum bpf_cgroup_iter_order)BPF_CGROUP_ITER_SELF_ONLY,.cgroup_fd = (__u32)1,},}",
768+ { .cgroup = { .order = 1, .cgroup_fd = 1, }});
769769
770770 /* struct skb with nested structs/unions; because type output is so
771771 * complex, we don't do a string comparison, just verify we return
@@ -841,8 +841,8 @@ static void test_btf_dump_datasec_data(char *str) · 69 unmodified lines
841841 char license[4] = "GPL";
842842 struct btf_dump *d;
843843
844− btf = btf__parse("xdping_kern.o", NULL);
845− if (!ASSERT_OK_PTR(btf, "xdping_kern.o BTF not found"))
844+ btf = btf__parse("xdping_kern.bpf.o", NULL);
845+ if (!ASSERT_OK_PTR(btf, "xdping_kern.bpf.o BTF not found"))
846846 return;
847847
848848 d = btf_dump__new(btf, btf_dump_snprintf, str, NULL);
tools/testing/selftests/bpf/prog_tests/btf_endian.c+1−1
@@ -23,7 +23,7 @@ void test_btf_endian() { · 22 unmodified lines
2323 int var_id;
2424
2525 /* Load BTF in native endianness */
26− btf = btf__parse_elf("btf_dump_test_case_syntax.o", NULL);
26+ btf = btf__parse_elf("btf_dump_test_case_syntax.bpf.o", NULL);
2727 if (!ASSERT_OK_PTR(btf, "parse_native_btf"))
2828 goto err_out;
2929
tools/testing/selftests/bpf/prog_tests/cb_refs.c+48−0
@@ -0,0 +1,48 @@
1+// SPDX-License-Identifier: GPL-2.0
2+#include "bpf/libbpf.h"
3+#include <test_progs.h>
4+#include <network_helpers.h>
5+
6+#include "cb_refs.skel.h"
7+
8+static char log_buf[1024 * 1024];
9+
10+struct {
11+ const char *prog_name;
12+ const char *err_msg;
13+} cb_refs_tests[] = {
14+ { "underflow_prog", "reference has not been acquired before" },
15+ { "leak_prog", "Unreleased reference" },
16+ { "nested_cb", "Unreleased reference id=4 alloc_insn=2" }, /* alloc_insn=2{4,5} */
17+ { "non_cb_transfer_ref", "Unreleased reference id=4 alloc_insn=1" }, /* alloc_insn=1{1,2} */
18+};
19+
20+void test_cb_refs(void)
21+{
22+ LIBBPF_OPTS(bpf_object_open_opts, opts, .kernel_log_buf = log_buf,
23+ .kernel_log_size = sizeof(log_buf),
24+ .kernel_log_level = 1);
25+ struct bpf_program *prog;
26+ struct cb_refs *skel;
27+ int i;
28+
29+ for (i = 0; i < ARRAY_SIZE(cb_refs_tests); i++) {
30+ LIBBPF_OPTS(bpf_test_run_opts, run_opts,
31+ .data_in = &pkt_v4,
32+ .data_size_in = sizeof(pkt_v4),
33+ .repeat = 1,
34+ );
35+ skel = cb_refs__open_opts(&opts);
36+ if (!ASSERT_OK_PTR(skel, "cb_refs__open_and_load"))
37+ return;
38+ prog = bpf_object__find_program_by_name(skel->obj, cb_refs_tests[i].prog_name);
39+ bpf_program__set_autoload(prog, true);
40+ if (!ASSERT_ERR(cb_refs__load(skel), "cb_refs__load"))
41+ bpf_prog_test_run_opts(bpf_program__fd(prog), &run_opts);
42+ if (!ASSERT_OK_PTR(strstr(log_buf, cb_refs_tests[i].err_msg), "expected error message")) {
43+ fprintf(stderr, "Expected: %s\n", cb_refs_tests[i].err_msg);
44+ fprintf(stderr, "Verifier: %s\n", log_buf);
45+ }
46+ cb_refs__destroy(skel);
47+ }
48+}
tools/testing/selftests/bpf/prog_tests/cgroup_getset_retval.c+48−0
@@ -10,6 +10,7 @@ · 9 unmodified lines
1010
1111 #include "cgroup_getset_retval_setsockopt.skel.h"
1212 #include "cgroup_getset_retval_getsockopt.skel.h"
13+#include "cgroup_getset_retval_hooks.skel.h"
1314
1415 #define SOL_CUSTOM 0xdeadbeef
1516
@@ -433,6 +434,50 @@ close_bpf_object: · 417 unmodified lines
433434 cgroup_getset_retval_getsockopt__destroy(obj);
434435 }
435436
437+struct exposed_hook {
438+ const char *name;
439+ int expected_err;
440+} exposed_hooks[] = {
441+
442+#define BPF_RETVAL_HOOK(NAME, SECTION, CTX, EXPECTED_ERR) \
443+ { \
444+ .name = #NAME, \
445+ .expected_err = EXPECTED_ERR, \
446+ },
447+
448+#include "cgroup_getset_retval_hooks.h"
449+
450+#undef BPF_RETVAL_HOOK
451+};
452+
453+static void test_exposed_hooks(int cgroup_fd, int sock_fd)
454+{
455+ struct cgroup_getset_retval_hooks *skel;
456+ struct bpf_program *prog;
457+ int err;
458+ int i;
459+
460+ for (i = 0; i < ARRAY_SIZE(exposed_hooks); i++) {
461+ skel = cgroup_getset_retval_hooks__open();
462+ if (!ASSERT_OK_PTR(skel, "cgroup_getset_retval_hooks__open"))
463+ continue;
464+
465+ prog = bpf_object__find_program_by_name(skel->obj, exposed_hooks[i].name);
466+ if (!ASSERT_NEQ(prog, NULL, "bpf_object__find_program_by_name"))
467+ goto close_skel;
468+
469+ err = bpf_program__set_autoload(prog, true);
470+ if (!ASSERT_OK(err, "bpf_program__set_autoload"))
471+ goto close_skel;
472+
473+ err = cgroup_getset_retval_hooks__load(skel);
474+ ASSERT_EQ(err, exposed_hooks[i].expected_err, "expected_err");
475+
476+close_skel:
477+ cgroup_getset_retval_hooks__destroy(skel);
478+ }
479+}
480+
436481 void test_cgroup_getset_retval(void)
437482 {
438483 int cgroup_fd = -1;
@@ -476,6 +521,9 @@ void test_cgroup_getset_retval(void) · 37 unmodified lines
476521 if (test__start_subtest("getsockopt-retval_sync"))
477522 test_getsockopt_retval_sync(cgroup_fd, sock_fd);
478523
524+ if (test__start_subtest("exposed_hooks"))
525+ test_exposed_hooks(cgroup_fd, sock_fd);
526+
479527 close_fd:
480528 close(cgroup_fd);
481529 }
tools/testing/selftests/bpf/prog_tests/cgroup_hierarchical_stats.c+357−0
@@ -0,0 +1,357 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/*
3+ * Functions to manage eBPF programs attached to cgroup subsystems
4+ *
5+ * Copyright 2022 Google LLC.
6+ */
7+#include <asm-generic/errno.h>
8+#include <errno.h>
9+#include <sys/types.h>
10+#include <sys/mount.h>
11+#include <sys/stat.h>
12+#include <unistd.h>
13+
14+#include <test_progs.h>
15+#include <bpf/libbpf.h>
16+#include <bpf/bpf.h>
17+
18+#include "cgroup_helpers.h"
19+#include "cgroup_hierarchical_stats.skel.h"
20+
21+#define PAGE_SIZE 4096
22+#define MB(x) (x << 20)
23+
24+#define BPFFS_ROOT "/sys/fs/bpf/"
25+#define BPFFS_VMSCAN BPFFS_ROOT"vmscan/"
26+
27+#define CG_ROOT_NAME "root"
28+#define CG_ROOT_ID 1
29+
30+#define CGROUP_PATH(p, n) {.path = p"/"n, .name = n}
31+
32+static struct {
33+ const char *path, *name;
34+ unsigned long long id;
35+ int fd;
36+} cgroups[] = {
37+ CGROUP_PATH("/", "test"),
38+ CGROUP_PATH("/test", "child1"),
39+ CGROUP_PATH("/test", "child2"),
40+ CGROUP_PATH("/test/child1", "child1_1"),
41+ CGROUP_PATH("/test/child1", "child1_2"),
42+ CGROUP_PATH("/test/child2", "child2_1"),
43+ CGROUP_PATH("/test/child2", "child2_2"),
44+};
45+
46+#define N_CGROUPS ARRAY_SIZE(cgroups)
47+#define N_NON_LEAF_CGROUPS 3
48+
49+static int root_cgroup_fd;
50+static bool mounted_bpffs;
51+
52+/* reads file at 'path' to 'buf', returns 0 on success. */
53+static int read_from_file(const char *path, char *buf, size_t size)
54+{
55+ int fd, len;
56+
57+ fd = open(path, O_RDONLY);
58+ if (fd < 0)
59+ return fd;
60+
61+ len = read(fd, buf, size);
62+ close(fd);
63+ if (len < 0)
64+ return len;
65+
66+ buf[len] = 0;
67+ return 0;
68+}
69+
70+/* mounts bpffs and mkdir for reading stats, returns 0 on success. */
71+static int setup_bpffs(void)
72+{
73+ int err;
74+
75+ /* Mount bpffs */
76+ err = mount("bpf", BPFFS_ROOT, "bpf", 0, NULL);
77+ mounted_bpffs = !err;
78+ if (ASSERT_FALSE(err && errno != EBUSY, "mount"))
79+ return err;
80+
81+ /* Create a directory to contain stat files in bpffs */
82+ err = mkdir(BPFFS_VMSCAN, 0755);
83+ if (!ASSERT_OK(err, "mkdir"))
84+ return err;
85+
86+ return 0;
87+}
88+
89+static void cleanup_bpffs(void)
90+{
91+ /* Remove created directory in bpffs */
92+ ASSERT_OK(rmdir(BPFFS_VMSCAN), "rmdir "BPFFS_VMSCAN);
93+
94+ /* Unmount bpffs, if it wasn't already mounted when we started */
95+ if (mounted_bpffs)
96+ return;
97+
98+ ASSERT_OK(umount(BPFFS_ROOT), "unmount bpffs");
99+}
100+
101+/* sets up cgroups, returns 0 on success. */
102+static int setup_cgroups(void)
103+{
104+ int i, fd, err;
105+
106+ err = setup_cgroup_environment();
107+ if (!ASSERT_OK(err, "setup_cgroup_environment"))
108+ return err;
109+
110+ root_cgroup_fd = get_root_cgroup();
111+ if (!ASSERT_GE(root_cgroup_fd, 0, "get_root_cgroup"))
112+ return root_cgroup_fd;
113+
114+ for (i = 0; i < N_CGROUPS; i++) {
115+ fd = create_and_get_cgroup(cgroups[i].path);
116+ if (!ASSERT_GE(fd, 0, "create_and_get_cgroup"))
117+ return fd;
118+
119+ cgroups[i].fd = fd;
120+ cgroups[i].id = get_cgroup_id(cgroups[i].path);
121+
122+ /*
123+ * Enable memcg controller for the entire hierarchy.
124+ * Note that stats are collected for all cgroups in a hierarchy
125+ * with memcg enabled anyway, but are only exposed for cgroups
126+ * that have memcg enabled.
127+ */
128+ if (i < N_NON_LEAF_CGROUPS) {
129+ err = enable_controllers(cgroups[i].path, "memory");
130+ if (!ASSERT_OK(err, "enable_controllers"))
131+ return err;
132+ }
133+ }
134+ return 0;
135+}
136+
137+static void cleanup_cgroups(void)
138+{
139+ close(root_cgroup_fd);
140+ for (int i = 0; i < N_CGROUPS; i++)
141+ close(cgroups[i].fd);
142+ cleanup_cgroup_environment();
143+}
144+
145+/* Sets up cgroup hiearchary, returns 0 on success. */
146+static int setup_hierarchy(void)
147+{
148+ return setup_bpffs() || setup_cgroups();
149+}
150+
151+static void destroy_hierarchy(void)
152+{
153+ cleanup_cgroups();
154+ cleanup_bpffs();
155+}
156+
157+static int reclaimer(const char *cgroup_path, size_t size)
158+{
159+ static char size_buf[128];
160+ char *buf, *ptr;
161+ int err;
162+
163+ /* Join cgroup in the parent process workdir */
164+ if (join_parent_cgroup(cgroup_path))
165+ return EACCES;
166+
167+ /* Allocate memory */
168+ buf = malloc(size);
169+ if (!buf)
170+ return ENOMEM;
171+
172+ /* Write to memory to make sure it's actually allocated */
173+ for (ptr = buf; ptr < buf + size; ptr += PAGE_SIZE)
174+ *ptr = 1;
175+
176+ /* Try to reclaim memory */
177+ snprintf(size_buf, 128, "%lu", size);
178+ err = write_cgroup_file_parent(cgroup_path, "memory.reclaim", size_buf);
179+
180+ free(buf);
181+ /* memory.reclaim returns EAGAIN if the amount is not fully reclaimed */
182+ if (err && errno != EAGAIN)
183+ return errno;
184+
185+ return 0;
186+}
187+
188+static int induce_vmscan(void)
189+{
190+ int i, status;
191+
192+ /*
193+ * In every leaf cgroup, run a child process that allocates some memory
194+ * and attempts to reclaim some of it.
195+ */
196+ for (i = N_NON_LEAF_CGROUPS; i < N_CGROUPS; i++) {
197+ pid_t pid;
198+
199+ /* Create reclaimer child */
200+ pid = fork();
201+ if (pid == 0) {
202+ status = reclaimer(cgroups[i].path, MB(5));
203+ exit(status);
204+ }
205+
206+ /* Cleanup reclaimer child */
207+ waitpid(pid, &status, 0);
208+ ASSERT_TRUE(WIFEXITED(status), "reclaimer exited");
209+ ASSERT_EQ(WEXITSTATUS(status), 0, "reclaim exit code");
210+ }
211+ return 0;
212+}
213+
214+static unsigned long long
215+get_cgroup_vmscan_delay(unsigned long long cgroup_id, const char *file_name)
216+{
217+ unsigned long long vmscan = 0, id = 0;
218+ static char buf[128], path[128];
219+
220+ /* For every cgroup, read the file generated by cgroup_iter */
221+ snprintf(path, 128, "%s%s", BPFFS_VMSCAN, file_name);
222+ if (!ASSERT_OK(read_from_file(path, buf, 128), "read cgroup_iter"))
223+ return 0;
224+
225+ /* Check the output file formatting */
226+ ASSERT_EQ(sscanf(buf, "cg_id: %llu, total_vmscan_delay: %llu\n",
227+ &id, &vmscan), 2, "output format");
228+
229+ /* Check that the cgroup_id is displayed correctly */
230+ ASSERT_EQ(id, cgroup_id, "cgroup_id");
231+ /* Check that the vmscan reading is non-zero */
232+ ASSERT_GT(vmscan, 0, "vmscan_reading");
233+ return vmscan;
234+}
235+
236+static void check_vmscan_stats(void)
237+{
238+ unsigned long long vmscan_readings[N_CGROUPS], vmscan_root;
239+ int i;
240+
241+ for (i = 0; i < N_CGROUPS; i++) {
242+ vmscan_readings[i] = get_cgroup_vmscan_delay(cgroups[i].id,
243+ cgroups[i].name);
244+ }
245+
246+ /* Read stats for root too */
247+ vmscan_root = get_cgroup_vmscan_delay(CG_ROOT_ID, CG_ROOT_NAME);
248+
249+ /* Check that child1 == child1_1 + child1_2 */
250+ ASSERT_EQ(vmscan_readings[1], vmscan_readings[3] + vmscan_readings[4],
251+ "child1_vmscan");
252+ /* Check that child2 == child2_1 + child2_2 */
253+ ASSERT_EQ(vmscan_readings[2], vmscan_readings[5] + vmscan_readings[6],
254+ "child2_vmscan");
255+ /* Check that test == child1 + child2 */
256+ ASSERT_EQ(vmscan_readings[0], vmscan_readings[1] + vmscan_readings[2],
257+ "test_vmscan");
258+ /* Check that root >= test */
259+ ASSERT_GE(vmscan_root, vmscan_readings[1], "root_vmscan");
260+}
261+
262+/* Creates iter link and pins in bpffs, returns 0 on success, -errno on failure.
263+ */
264+static int setup_cgroup_iter(struct cgroup_hierarchical_stats *obj,
265+ int cgroup_fd, const char *file_name)
266+{
267+ DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
268+ union bpf_iter_link_info linfo = {};
269+ struct bpf_link *link;
270+ static char path[128];
271+ int err;
272+
273+ /*
274+ * Create an iter link, parameterized by cgroup_fd. We only want to
275+ * traverse one cgroup, so set the traversal order to "self".
276+ */
277+ linfo.cgroup.cgroup_fd = cgroup_fd;
278+ linfo.cgroup.order = BPF_CGROUP_ITER_SELF_ONLY;
279+ opts.link_info = &linfo;
280+ opts.link_info_len = sizeof(linfo);
281+ link = bpf_program__attach_iter(obj->progs.dump_vmscan, &opts);
282+ if (!ASSERT_OK_PTR(link, "attach_iter"))
283+ return -EFAULT;
284+
285+ /* Pin the link to a bpffs file */
286+ snprintf(path, 128, "%s%s", BPFFS_VMSCAN, file_name);
287+ err = bpf_link__pin(link, path);
288+ ASSERT_OK(err, "pin cgroup_iter");
289+
290+ /* Remove the link, leaving only the ref held by the pinned file */
291+ bpf_link__destroy(link);
292+ return err;
293+}
294+
295+/* Sets up programs for collecting stats, returns 0 on success. */
296+static int setup_progs(struct cgroup_hierarchical_stats **skel)
297+{
298+ int i, err;
299+
300+ *skel = cgroup_hierarchical_stats__open_and_load();
301+ if (!ASSERT_OK_PTR(*skel, "open_and_load"))
302+ return 1;
303+
304+ /* Attach cgroup_iter program that will dump the stats to cgroups */
305+ for (i = 0; i < N_CGROUPS; i++) {
306+ err = setup_cgroup_iter(*skel, cgroups[i].fd, cgroups[i].name);
307+ if (!ASSERT_OK(err, "setup_cgroup_iter"))
308+ return err;
309+ }
310+
311+ /* Also dump stats for root */
312+ err = setup_cgroup_iter(*skel, root_cgroup_fd, CG_ROOT_NAME);
313+ if (!ASSERT_OK(err, "setup_cgroup_iter"))
314+ return err;
315+
316+ bpf_program__set_autoattach((*skel)->progs.dump_vmscan, false);
317+ err = cgroup_hierarchical_stats__attach(*skel);
318+ if (!ASSERT_OK(err, "attach"))
319+ return err;
320+
321+ return 0;
322+}
323+
324+static void destroy_progs(struct cgroup_hierarchical_stats *skel)
325+{
326+ static char path[128];
327+ int i;
328+
329+ for (i = 0; i < N_CGROUPS; i++) {
330+ /* Delete files in bpffs that cgroup_iters are pinned in */
331+ snprintf(path, 128, "%s%s", BPFFS_VMSCAN,
332+ cgroups[i].name);
333+ ASSERT_OK(remove(path), "remove cgroup_iter pin");
334+ }
335+
336+ /* Delete root file in bpffs */
337+ snprintf(path, 128, "%s%s", BPFFS_VMSCAN, CG_ROOT_NAME);
338+ ASSERT_OK(remove(path), "remove cgroup_iter root pin");
339+ cgroup_hierarchical_stats__destroy(skel);
340+}
341+
342+void test_cgroup_hierarchical_stats(void)
343+{
344+ struct cgroup_hierarchical_stats *skel = NULL;
345+
346+ if (setup_hierarchy())
347+ goto hierarchy_cleanup;
348+ if (setup_progs(&skel))
349+ goto cleanup;
350+ if (induce_vmscan())
351+ goto cleanup;
352+ check_vmscan_stats();
353+cleanup:
354+ destroy_progs(skel);
355+hierarchy_cleanup:
356+ destroy_hierarchy();
357+}
tools/testing/selftests/bpf/prog_tests/cgroup_iter.c+224−0
@@ -0,0 +1,224 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (c) 2022 Google */
3+
4+#include <test_progs.h>
5+#include <bpf/libbpf.h>
6+#include <bpf/btf.h>
7+#include "cgroup_iter.skel.h"
8+#include "cgroup_helpers.h"
9+
10+#define ROOT 0
11+#define PARENT 1
12+#define CHILD1 2
13+#define CHILD2 3
14+#define NUM_CGROUPS 4
15+
16+#define PROLOGUE "prologue\n"
17+#define EPILOGUE "epilogue\n"
18+
19+static const char *cg_path[] = {
20+ "/", "/parent", "/parent/child1", "/parent/child2"
21+};
22+
23+static int cg_fd[] = {-1, -1, -1, -1};
24+static unsigned long long cg_id[] = {0, 0, 0, 0};
25+static char expected_output[64];
26+
27+static int setup_cgroups(void)
28+{
29+ int fd, i = 0;
30+
31+ for (i = 0; i < NUM_CGROUPS; i++) {
32+ fd = create_and_get_cgroup(cg_path[i]);
33+ if (fd < 0)
34+ return fd;
35+
36+ cg_fd[i] = fd;
37+ cg_id[i] = get_cgroup_id(cg_path[i]);
38+ }
39+ return 0;
40+}
41+
42+static void cleanup_cgroups(void)
43+{
44+ int i;
45+
46+ for (i = 0; i < NUM_CGROUPS; i++)
47+ close(cg_fd[i]);
48+}
49+
50+static void read_from_cgroup_iter(struct bpf_program *prog, int cgroup_fd,
51+ int order, const char *testname)
52+{
53+ DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
54+ union bpf_iter_link_info linfo;
55+ struct bpf_link *link;
56+ int len, iter_fd;
57+ static char buf[128];
58+ size_t left;
59+ char *p;
60+
61+ memset(&linfo, 0, sizeof(linfo));
62+ linfo.cgroup.cgroup_fd = cgroup_fd;
63+ linfo.cgroup.order = order;
64+ opts.link_info = &linfo;
65+ opts.link_info_len = sizeof(linfo);
66+
67+ link = bpf_program__attach_iter(prog, &opts);
68+ if (!ASSERT_OK_PTR(link, "attach_iter"))
69+ return;
70+
71+ iter_fd = bpf_iter_create(bpf_link__fd(link));
72+ if (iter_fd < 0)
73+ goto free_link;
74+
75+ memset(buf, 0, sizeof(buf));
76+ left = ARRAY_SIZE(buf);
77+ p = buf;
78+ while ((len = read(iter_fd, p, left)) > 0) {
79+ p += len;
80+ left -= len;
81+ }
82+
83+ ASSERT_STREQ(buf, expected_output, testname);
84+
85+ /* read() after iter finishes should be ok. */
86+ if (len == 0)
87+ ASSERT_OK(read(iter_fd, buf, sizeof(buf)), "second_read");
88+
89+ close(iter_fd);
90+free_link:
91+ bpf_link__destroy(link);
92+}
93+
94+/* Invalid cgroup. */
95+static void test_invalid_cgroup(struct cgroup_iter *skel)
96+{
97+ DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
98+ union bpf_iter_link_info linfo;
99+ struct bpf_link *link;
100+
101+ memset(&linfo, 0, sizeof(linfo));
102+ linfo.cgroup.cgroup_fd = (__u32)-1;
103+ opts.link_info = &linfo;
104+ opts.link_info_len = sizeof(linfo);
105+
106+ link = bpf_program__attach_iter(skel->progs.cgroup_id_printer, &opts);
107+ ASSERT_ERR_PTR(link, "attach_iter");
108+ bpf_link__destroy(link);
109+}
110+
111+/* Specifying both cgroup_fd and cgroup_id is invalid. */
112+static void test_invalid_cgroup_spec(struct cgroup_iter *skel)
113+{
114+ DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
115+ union bpf_iter_link_info linfo;
116+ struct bpf_link *link;
117+
118+ memset(&linfo, 0, sizeof(linfo));
119+ linfo.cgroup.cgroup_fd = (__u32)cg_fd[PARENT];
120+ linfo.cgroup.cgroup_id = (__u64)cg_id[PARENT];
121+ opts.link_info = &linfo;
122+ opts.link_info_len = sizeof(linfo);
123+
124+ link = bpf_program__attach_iter(skel->progs.cgroup_id_printer, &opts);
125+ ASSERT_ERR_PTR(link, "attach_iter");
126+ bpf_link__destroy(link);
127+}
128+
129+/* Preorder walk prints parent and child in order. */
130+static void test_walk_preorder(struct cgroup_iter *skel)
131+{
132+ snprintf(expected_output, sizeof(expected_output),
133+ PROLOGUE "%8llu\n%8llu\n%8llu\n" EPILOGUE,
134+ cg_id[PARENT], cg_id[CHILD1], cg_id[CHILD2]);
135+
136+ read_from_cgroup_iter(skel->progs.cgroup_id_printer, cg_fd[PARENT],
137+ BPF_CGROUP_ITER_DESCENDANTS_PRE, "preorder");
138+}
139+
140+/* Postorder walk prints child and parent in order. */
141+static void test_walk_postorder(struct cgroup_iter *skel)
142+{
143+ snprintf(expected_output, sizeof(expected_output),
144+ PROLOGUE "%8llu\n%8llu\n%8llu\n" EPILOGUE,
145+ cg_id[CHILD1], cg_id[CHILD2], cg_id[PARENT]);
146+
147+ read_from_cgroup_iter(skel->progs.cgroup_id_printer, cg_fd[PARENT],
148+ BPF_CGROUP_ITER_DESCENDANTS_POST, "postorder");
149+}
150+
151+/* Walking parents prints parent and then root. */
152+static void test_walk_ancestors_up(struct cgroup_iter *skel)
153+{
154+ /* terminate the walk when ROOT is met. */
155+ skel->bss->terminal_cgroup = cg_id[ROOT];
156+
157+ snprintf(expected_output, sizeof(expected_output),
158+ PROLOGUE "%8llu\n%8llu\n" EPILOGUE,
159+ cg_id[PARENT], cg_id[ROOT]);
160+
161+ read_from_cgroup_iter(skel->progs.cgroup_id_printer, cg_fd[PARENT],
162+ BPF_CGROUP_ITER_ANCESTORS_UP, "ancestors_up");
163+
164+ skel->bss->terminal_cgroup = 0;
165+}
166+
167+/* Early termination prints parent only. */
168+static void test_early_termination(struct cgroup_iter *skel)
169+{
170+ /* terminate the walk after the first element is processed. */
171+ skel->bss->terminate_early = 1;
172+
173+ snprintf(expected_output, sizeof(expected_output),
174+ PROLOGUE "%8llu\n" EPILOGUE, cg_id[PARENT]);
175+
176+ read_from_cgroup_iter(skel->progs.cgroup_id_printer, cg_fd[PARENT],
177+ BPF_CGROUP_ITER_DESCENDANTS_PRE, "early_termination");
178+
179+ skel->bss->terminate_early = 0;
180+}
181+
182+/* Waling self prints self only. */
183+static void test_walk_self_only(struct cgroup_iter *skel)
184+{
185+ snprintf(expected_output, sizeof(expected_output),
186+ PROLOGUE "%8llu\n" EPILOGUE, cg_id[PARENT]);
187+
188+ read_from_cgroup_iter(skel->progs.cgroup_id_printer, cg_fd[PARENT],
189+ BPF_CGROUP_ITER_SELF_ONLY, "self_only");
190+}
191+
192+void test_cgroup_iter(void)
193+{
194+ struct cgroup_iter *skel = NULL;
195+
196+ if (setup_cgroup_environment())
197+ return;
198+
199+ if (setup_cgroups())
200+ goto out;
201+
202+ skel = cgroup_iter__open_and_load();
203+ if (!ASSERT_OK_PTR(skel, "cgroup_iter__open_and_load"))
204+ goto out;
205+
206+ if (test__start_subtest("cgroup_iter__invalid_cgroup"))
207+ test_invalid_cgroup(skel);
208+ if (test__start_subtest("cgroup_iter__invalid_cgroup_spec"))
209+ test_invalid_cgroup_spec(skel);
210+ if (test__start_subtest("cgroup_iter__preorder"))
211+ test_walk_preorder(skel);
212+ if (test__start_subtest("cgroup_iter__postorder"))
213+ test_walk_postorder(skel);
214+ if (test__start_subtest("cgroup_iter__ancestors_up_walk"))
215+ test_walk_ancestors_up(skel);
216+ if (test__start_subtest("cgroup_iter__early_termination"))
217+ test_early_termination(skel);
218+ if (test__start_subtest("cgroup_iter__self_only"))
219+ test_walk_self_only(skel);
220+out:
221+ cgroup_iter__destroy(skel);
222+ cleanup_cgroups();
223+ cleanup_cgroup_environment();
224+}
tools/testing/selftests/bpf/prog_tests/connect_force_port.c+1−1
@@ -53,7 +53,7 @@ static int run_test(int cgroup_fd, int server_fd, int family, int type) · 52 unmodified lines
5353 __u16 expected_peer_port = 60000;
5454 struct bpf_program *prog;
5555 struct bpf_object *obj;
56− const char *obj_file = v4 ? "connect_force_port4.o" : "connect_force_port6.o";
56+ const char *obj_file = v4 ? "connect_force_port4.bpf.o" : "connect_force_port6.bpf.o";
5757 int fd, err;
5858 __u32 duration = 0;
5959
tools/testing/selftests/bpf/prog_tests/core_reloc.c+37−37
@@ -13,7 +13,7 @@ static int duration = 0; · 12 unmodified lines
1313
1414 #define MODULES_CASE(name, pg_name, tp_name) { \
1515 .case_name = name, \
16− .bpf_obj_file = "test_core_reloc_module.o", \
16+ .bpf_obj_file = "test_core_reloc_module.bpf.o", \
1717 .btf_src_file = NULL, /* find in kernel module BTFs */ \
1818 .input = "", \
1919 .input_len = 0, \
@@ -43,8 +43,8 @@ static int duration = 0; · 23 unmodified lines
4343
4444 #define FLAVORS_CASE_COMMON(name) \
4545 .case_name = #name, \
46− .bpf_obj_file = "test_core_reloc_flavors.o", \
47− .btf_src_file = "btf__core_reloc_" #name ".o", \
46+ .bpf_obj_file = "test_core_reloc_flavors.bpf.o", \
47+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
4848 .raw_tp_name = "sys_enter", \
4949 .prog_name = "test_core_flavors" \
5050
@@ -68,8 +68,8 @@ static int duration = 0; · 17 unmodified lines
6868
6969 #define NESTING_CASE_COMMON(name) \
7070 .case_name = #name, \
71− .bpf_obj_file = "test_core_reloc_nesting.o", \
72− .btf_src_file = "btf__core_reloc_" #name ".o", \
71+ .bpf_obj_file = "test_core_reloc_nesting.bpf.o", \
72+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
7373 .raw_tp_name = "sys_enter", \
7474 .prog_name = "test_core_nesting" \
7575
@@ -96,8 +96,8 @@ static int duration = 0; · 20 unmodified lines
9696
9797 #define ARRAYS_CASE_COMMON(name) \
9898 .case_name = #name, \
99− .bpf_obj_file = "test_core_reloc_arrays.o", \
100− .btf_src_file = "btf__core_reloc_" #name ".o", \
99+ .bpf_obj_file = "test_core_reloc_arrays.bpf.o", \
100+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
101101 .raw_tp_name = "sys_enter", \
102102 .prog_name = "test_core_arrays" \
103103
@@ -130,8 +130,8 @@ static int duration = 0; · 26 unmodified lines
130130
131131 #define PRIMITIVES_CASE_COMMON(name) \
132132 .case_name = #name, \
133− .bpf_obj_file = "test_core_reloc_primitives.o", \
134− .btf_src_file = "btf__core_reloc_" #name ".o", \
133+ .bpf_obj_file = "test_core_reloc_primitives.bpf.o", \
134+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
135135 .raw_tp_name = "sys_enter", \
136136 .prog_name = "test_core_primitives" \
137137
@@ -150,8 +150,8 @@ static int duration = 0; · 12 unmodified lines
150150
151151 #define MODS_CASE(name) { \
152152 .case_name = #name, \
153− .bpf_obj_file = "test_core_reloc_mods.o", \
154− .btf_src_file = "btf__core_reloc_" #name ".o", \
153+ .bpf_obj_file = "test_core_reloc_mods.bpf.o", \
154+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
155155 .input = STRUCT_TO_CHAR_PTR(core_reloc_##name) { \
156156 .a = 1, \
157157 .b = 2, \
@@ -174,8 +174,8 @@ static int duration = 0; · 16 unmodified lines
174174
175175 #define PTR_AS_ARR_CASE(name) { \
176176 .case_name = #name, \
177− .bpf_obj_file = "test_core_reloc_ptr_as_arr.o", \
178− .btf_src_file = "btf__core_reloc_" #name ".o", \
177+ .bpf_obj_file = "test_core_reloc_ptr_as_arr.bpf.o", \
178+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
179179 .input = (const char *)&(struct core_reloc_##name []){ \
180180 { .a = 1 }, \
181181 { .a = 2 }, \
@@ -203,8 +203,8 @@ static int duration = 0; · 21 unmodified lines
203203
204204 #define INTS_CASE_COMMON(name) \
205205 .case_name = #name, \
206− .bpf_obj_file = "test_core_reloc_ints.o", \
207− .btf_src_file = "btf__core_reloc_" #name ".o", \
206+ .bpf_obj_file = "test_core_reloc_ints.bpf.o", \
207+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
208208 .raw_tp_name = "sys_enter", \
209209 .prog_name = "test_core_ints"
210210
@@ -223,18 +223,18 @@ static int duration = 0; · 12 unmodified lines
223223
224224 #define FIELD_EXISTS_CASE_COMMON(name) \
225225 .case_name = #name, \
226− .bpf_obj_file = "test_core_reloc_existence.o", \
227− .btf_src_file = "btf__core_reloc_" #name ".o", \
226+ .bpf_obj_file = "test_core_reloc_existence.bpf.o", \
227+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
228228 .raw_tp_name = "sys_enter", \
229229 .prog_name = "test_core_existence"
230230
231231 #define BITFIELDS_CASE_COMMON(objfile, test_name_prefix, name) \
232232 .case_name = test_name_prefix#name, \
233233 .bpf_obj_file = objfile, \
234− .btf_src_file = "btf__core_reloc_" #name ".o"
234+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o"
235235
236236 #define BITFIELDS_CASE(name, ...) { \
237− BITFIELDS_CASE_COMMON("test_core_reloc_bitfields_probed.o", \
237+ BITFIELDS_CASE_COMMON("test_core_reloc_bitfields_probed.bpf.o", \
238238 "probed:", name), \
239239 .input = STRUCT_TO_CHAR_PTR(core_reloc_##name) __VA_ARGS__, \
240240 .input_len = sizeof(struct core_reloc_##name), \
@@ -244,7 +244,7 @@ static int duration = 0; · 3 unmodified lines
244244 .raw_tp_name = "sys_enter", \
245245 .prog_name = "test_core_bitfields", \
246246 }, { \
247− BITFIELDS_CASE_COMMON("test_core_reloc_bitfields_direct.o", \
247+ BITFIELDS_CASE_COMMON("test_core_reloc_bitfields_direct.bpf.o", \
248248 "direct:", name), \
249249 .input = STRUCT_TO_CHAR_PTR(core_reloc_##name) __VA_ARGS__, \
250250 .input_len = sizeof(struct core_reloc_##name), \
@@ -256,14 +256,14 @@ static int duration = 0; · 5 unmodified lines
256256
257257
258258 #define BITFIELDS_ERR_CASE(name) { \
259− BITFIELDS_CASE_COMMON("test_core_reloc_bitfields_probed.o", \
259+ BITFIELDS_CASE_COMMON("test_core_reloc_bitfields_probed.bpf.o", \
260260 "probed:", name), \
261261 .fails = true, \
262− .run_btfgen_fails = true, \
262+ .run_btfgen_fails = true, \
263263 .raw_tp_name = "sys_enter", \
264264 .prog_name = "test_core_bitfields", \
265265 }, { \
266− BITFIELDS_CASE_COMMON("test_core_reloc_bitfields_direct.o", \
266+ BITFIELDS_CASE_COMMON("test_core_reloc_bitfields_direct.bpf.o", \
267267 "direct:", name), \
268268 .fails = true, \
269269 .run_btfgen_fails = true, \
@@ -272,8 +272,8 @@ static int duration = 0; · 2 unmodified lines
272272
273273 #define SIZE_CASE_COMMON(name) \
274274 .case_name = #name, \
275− .bpf_obj_file = "test_core_reloc_size.o", \
276− .btf_src_file = "btf__core_reloc_" #name ".o", \
275+ .bpf_obj_file = "test_core_reloc_size.bpf.o", \
276+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
277277 .raw_tp_name = "sys_enter", \
278278 .prog_name = "test_core_size"
279279
@@ -307,13 +307,13 @@ static int duration = 0; · 27 unmodified lines
307307 #define SIZE_ERR_CASE(name) { \
308308 SIZE_CASE_COMMON(name), \
309309 .fails = true, \
310− .run_btfgen_fails = true, \
310+ .run_btfgen_fails = true, \
311311 }
312312
313313 #define TYPE_BASED_CASE_COMMON(name) \
314314 .case_name = #name, \
315− .bpf_obj_file = "test_core_reloc_type_based.o", \
316− .btf_src_file = "btf__core_reloc_" #name ".o", \
315+ .bpf_obj_file = "test_core_reloc_type_based.bpf.o", \
316+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
317317 .raw_tp_name = "sys_enter", \
318318 .prog_name = "test_core_type_based"
319319
@@ -331,8 +331,8 @@ static int duration = 0; · 11 unmodified lines
331331
332332 #define TYPE_ID_CASE_COMMON(name) \
333333 .case_name = #name, \
334− .bpf_obj_file = "test_core_reloc_type_id.o", \
335− .btf_src_file = "btf__core_reloc_" #name ".o", \
334+ .bpf_obj_file = "test_core_reloc_type_id.bpf.o", \
335+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
336336 .raw_tp_name = "sys_enter", \
337337 .prog_name = "test_core_type_id"
338338
@@ -350,8 +350,8 @@ static int duration = 0; · 11 unmodified lines
350350
351351 #define ENUMVAL_CASE_COMMON(name) \
352352 .case_name = #name, \
353− .bpf_obj_file = "test_core_reloc_enumval.o", \
354− .btf_src_file = "btf__core_reloc_" #name ".o", \
353+ .bpf_obj_file = "test_core_reloc_enumval.bpf.o", \
354+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
355355 .raw_tp_name = "sys_enter", \
356356 .prog_name = "test_core_enumval"
357357
@@ -369,8 +369,8 @@ static int duration = 0; · 11 unmodified lines
369369
370370 #define ENUM64VAL_CASE_COMMON(name) \
371371 .case_name = #name, \
372− .bpf_obj_file = "test_core_reloc_enum64val.o", \
373− .btf_src_file = "btf__core_reloc_" #name ".o", \
372+ .bpf_obj_file = "test_core_reloc_enum64val.bpf.o", \
373+ .btf_src_file = "btf__core_reloc_" #name ".bpf.o", \
374374 .raw_tp_name = "sys_enter", \
375375 .prog_name = "test_core_enum64val"
376376
@@ -547,7 +547,7 @@ static const struct core_reloc_test_case test_cases[] = { · 170 unmodified lines
547547 /* validate we can find kernel image and use its BTF for relocs */
548548 {
549549 .case_name = "kernel",
550− .bpf_obj_file = "test_core_reloc_kernel.o",
550+ .bpf_obj_file = "test_core_reloc_kernel.bpf.o",
551551 .btf_src_file = NULL, /* load from /lib/modules/$(uname -r) */
552552 .input = "",
553553 .input_len = 0,
@@ -629,8 +629,8 @@ static const struct core_reloc_test_case test_cases[] = { · 75 unmodified lines
629629 /* validate edge cases of capturing relocations */
630630 {
631631 .case_name = "misc",
632− .bpf_obj_file = "test_core_reloc_misc.o",
633− .btf_src_file = "btf__core_reloc_misc.o",
632+ .bpf_obj_file = "test_core_reloc_misc.bpf.o",
633+ .btf_src_file = "btf__core_reloc_misc.bpf.o",
634634 .input = (const char *)&(struct core_reloc_misc_extensible[]){
635635 { .a = 1 },
636636 { .a = 2 }, /* not read */
tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c+22−22
@@ -174,8 +174,8 @@ static void test_target_no_callees(void) · 173 unmodified lines
174174 const char *prog_name[] = {
175175 "fexit/test_pkt_md_access",
176176 };
177− test_fexit_bpf2bpf_common("./fexit_bpf2bpf_simple.o",
178− "./test_pkt_md_access.o",
177+ test_fexit_bpf2bpf_common("./fexit_bpf2bpf_simple.bpf.o",
178+ "./test_pkt_md_access.bpf.o",
179179 ARRAY_SIZE(prog_name),
180180 prog_name, true, NULL);
181181 }
@@ -188,8 +188,8 @@ static void test_target_yes_callees(void) · 6 unmodified lines
188188 "fexit/test_pkt_access_subprog2",
189189 "fexit/test_pkt_access_subprog3",
190190 };
191− test_fexit_bpf2bpf_common("./fexit_bpf2bpf.o",
192− "./test_pkt_access.o",
191+ test_fexit_bpf2bpf_common("./fexit_bpf2bpf.bpf.o",
192+ "./test_pkt_access.bpf.o",
193193 ARRAY_SIZE(prog_name),
194194 prog_name, true, NULL);
195195 }
@@ -206,8 +206,8 @@ static void test_func_replace(void) · 10 unmodified lines
206206 "freplace/get_constant",
207207 "freplace/test_pkt_write_access_subprog",
208208 };
209− test_fexit_bpf2bpf_common("./fexit_bpf2bpf.o",
210− "./test_pkt_access.o",
209+ test_fexit_bpf2bpf_common("./fexit_bpf2bpf.bpf.o",
210+ "./test_pkt_access.bpf.o",
211211 ARRAY_SIZE(prog_name),
212212 prog_name, true, NULL);
213213 }
@@ -217,8 +217,8 @@ static void test_func_replace_verify(void) · 3 unmodified lines
217217 const char *prog_name[] = {
218218 "freplace/do_bind",
219219 };
220− test_fexit_bpf2bpf_common("./freplace_connect4.o",
221− "./connect4_prog.o",
220+ test_fexit_bpf2bpf_common("./freplace_connect4.bpf.o",
221+ "./connect4_prog.bpf.o",
222222 ARRAY_SIZE(prog_name),
223223 prog_name, false, NULL);
224224 }
@@ -227,7 +227,7 @@ static int test_second_attach(struct bpf_object *obj) · 2 unmodified lines
227227 {
228228 const char *prog_name = "security_new_get_constant";
229229 const char *tgt_name = "get_constant";
230− const char *tgt_obj_file = "./test_pkt_access.o";
230+ const char *tgt_obj_file = "./test_pkt_access.bpf.o";
231231 struct bpf_program *prog = NULL;
232232 struct bpf_object *tgt_obj;
233233 struct bpf_link *link;
@@ -272,8 +272,8 @@ static void test_func_replace_multi(void) · 38 unmodified lines
272272 const char *prog_name[] = {
273273 "freplace/get_constant",
274274 };
275− test_fexit_bpf2bpf_common("./freplace_get_constant.o",
276− "./test_pkt_access.o",
275+ test_fexit_bpf2bpf_common("./freplace_get_constant.bpf.o",
276+ "./test_pkt_access.bpf.o",
277277 ARRAY_SIZE(prog_name),
278278 prog_name, true, test_second_attach);
279279 }
@@ -281,10 +281,10 @@ static void test_func_replace_multi(void) · 1 unmodified line
281281 static void test_fmod_ret_freplace(void)
282282 {
283283 struct bpf_object *freplace_obj = NULL, *pkt_obj, *fmod_obj = NULL;
284− const char *freplace_name = "./freplace_get_constant.o";
285− const char *fmod_ret_name = "./fmod_ret_freplace.o";
284+ const char *freplace_name = "./freplace_get_constant.bpf.o";
285+ const char *fmod_ret_name = "./fmod_ret_freplace.bpf.o";
286286 DECLARE_LIBBPF_OPTS(bpf_object_open_opts, opts);
287− const char *tgt_name = "./test_pkt_access.o";
287+ const char *tgt_name = "./test_pkt_access.bpf.o";
288288 struct bpf_link *freplace_link = NULL;
289289 struct bpf_program *prog;
290290 __u32 duration = 0;
@@ -339,8 +339,8 @@ static void test_func_sockmap_update(void) · 48 unmodified lines
339339 const char *prog_name[] = {
340340 "freplace/cls_redirect",
341341 };
342− test_fexit_bpf2bpf_common("./freplace_cls_redirect.o",
343− "./test_cls_redirect.o",
342+ test_fexit_bpf2bpf_common("./freplace_cls_redirect.bpf.o",
343+ "./test_cls_redirect.bpf.o",
344344 ARRAY_SIZE(prog_name),
345345 prog_name, false, NULL);
346346 }
@@ -385,15 +385,15 @@ close_prog: · 38 unmodified lines
385385 static void test_func_replace_return_code(void)
386386 {
387387 /* test invalid return code in the replaced program */
388− test_obj_load_failure_common("./freplace_connect_v4_prog.o",
389− "./connect4_prog.o");
388+ test_obj_load_failure_common("./freplace_connect_v4_prog.bpf.o",
389+ "./connect4_prog.bpf.o");
390390 }
391391
392392 static void test_func_map_prog_compatibility(void)
393393 {
394394 /* test with spin lock map value in the replaced program */
395− test_obj_load_failure_common("./freplace_attach_probe.o",
396− "./test_attach_probe.o");
395+ test_obj_load_failure_common("./freplace_attach_probe.bpf.o",
396+ "./test_attach_probe.bpf.o");
397397 }
398398
399399 static void test_func_replace_global_func(void)
@@ -402,8 +402,8 @@ static void test_func_replace_global_func(void) · 2 unmodified lines
402402 "freplace/test_pkt_access",
403403 };
404404
405− test_fexit_bpf2bpf_common("./freplace_global_func.o",
406− "./test_pkt_access.o",
405+ test_fexit_bpf2bpf_common("./freplace_global_func.bpf.o",
406+ "./test_pkt_access.bpf.o",
407407 ARRAY_SIZE(prog_name),
408408 prog_name, false, NULL);
409409 }
tools/testing/selftests/bpf/prog_tests/flow_dissector.c+43−1
@@ -8,6 +8,8 @@ · 7 unmodified lines
88
99 #include "bpf_flow.skel.h"
1010
11+#define FLOW_CONTINUE_SADDR 0x7f00007f /* 127.0.0.127 */
12+
1113 #ifndef IP_MF
1214 #define IP_MF 0x2000
1315 #endif
@@ -100,6 +102,7 @@ struct test { · 86 unmodified lines
100102 } pkt;
101103 struct bpf_flow_keys keys;
102104 __u32 flags;
105+ __u32 retval;
103106 };
104107
105108 #define VLAN_HLEN 4
@@ -126,6 +129,7 @@ struct test tests[] = { · 20 unmodified lines
126129 .sport = 80,
127130 .dport = 8080,
128131 },
132+ .retval = BPF_OK,
129133 },
130134 {
131135 .name = "ipv6",
@@ -146,6 +150,7 @@ struct test tests[] = { · 14 unmodified lines
146150 .sport = 80,
147151 .dport = 8080,
148152 },
153+ .retval = BPF_OK,
149154 },
150155 {
151156 .name = "802.1q-ipv4",
@@ -168,6 +173,7 @@ struct test tests[] = { · 16 unmodified lines
168173 .sport = 80,
169174 .dport = 8080,
170175 },
176+ .retval = BPF_OK,
171177 },
172178 {
173179 .name = "802.1ad-ipv6",
@@ -191,6 +197,7 @@ struct test tests[] = { · 17 unmodified lines
191197 .sport = 80,
192198 .dport = 8080,
193199 },
200+ .retval = BPF_OK,
194201 },
195202 {
196203 .name = "ipv4-frag",
@@ -217,6 +224,7 @@ struct test tests[] = { · 20 unmodified lines
217224 .dport = 8080,
218225 },
219226 .flags = BPF_FLOW_DISSECTOR_F_PARSE_1ST_FRAG,
227+ .retval = BPF_OK,
220228 },
221229 {
222230 .name = "ipv4-no-frag",
@@ -239,6 +247,7 @@ struct test tests[] = { · 16 unmodified lines
239247 .is_frag = true,
240248 .is_first_frag = true,
241249 },
250+ .retval = BPF_OK,
242251 },
243252 {
244253 .name = "ipv6-frag",
@@ -265,6 +274,7 @@ struct test tests[] = { · 20 unmodified lines
265274 .dport = 8080,
266275 },
267276 .flags = BPF_FLOW_DISSECTOR_F_PARSE_1ST_FRAG,
277+ .retval = BPF_OK,
268278 },
269279 {
270280 .name = "ipv6-no-frag",
@@ -287,6 +297,7 @@ struct test tests[] = { · 16 unmodified lines
287297 .is_frag = true,
288298 .is_first_frag = true,
289299 },
300+ .retval = BPF_OK,
290301 },
291302 {
292303 .name = "ipv6-flow-label",
@@ -309,6 +320,7 @@ struct test tests[] = { · 16 unmodified lines
309320 .dport = 8080,
310321 .flow_label = __bpf_constant_htonl(0xbeeef),
311322 },
323+ .retval = BPF_OK,
312324 },
313325 {
314326 .name = "ipv6-no-flow-label",
@@ -331,6 +343,7 @@ struct test tests[] = { · 16 unmodified lines
331343 .flow_label = __bpf_constant_htonl(0xbeeef),
332344 },
333345 .flags = BPF_FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL,
346+ .retval = BPF_OK,
334347 },
335348 {
336349 .name = "ipip-encap",
@@ -359,6 +372,7 @@ struct test tests[] = { · 22 unmodified lines
359372 .sport = 80,
360373 .dport = 8080,
361374 },
375+ .retval = BPF_OK,
362376 },
363377 {
364378 .name = "ipip-no-encap",
@@ -386,6 +400,26 @@ struct test tests[] = { · 21 unmodified lines
386400 .is_encap = true,
387401 },
388402 .flags = BPF_FLOW_DISSECTOR_F_STOP_AT_ENCAP,
403+ .retval = BPF_OK,
404+ },
405+ {
406+ .name = "ipip-encap-dissector-continue",
407+ .pkt.ipip = {
408+ .eth.h_proto = __bpf_constant_htons(ETH_P_IP),
409+ .iph.ihl = 5,
410+ .iph.protocol = IPPROTO_IPIP,
411+ .iph.tot_len = __bpf_constant_htons(MAGIC_BYTES),
412+ .iph.saddr = __bpf_constant_htonl(FLOW_CONTINUE_SADDR),
413+ .iph_inner.ihl = 5,
414+ .iph_inner.protocol = IPPROTO_TCP,
415+ .iph_inner.tot_len =
416+ __bpf_constant_htons(MAGIC_BYTES) -
417+ sizeof(struct iphdr),
418+ .tcp.doff = 5,
419+ .tcp.source = 99,
420+ .tcp.dest = 9090,
421+ },
422+ .retval = BPF_FLOW_DISSECTOR_CONTINUE,
389423 },
390424 };
391425
@@ -503,6 +537,10 @@ static void run_tests_skb_less(int tap_fd, struct bpf_map *keys) · 111 unmodified lines
503537 err = tx_tap(tap_fd, &tests[i].pkt, sizeof(tests[i].pkt));
504538 CHECK(err < 0, "tx_tap", "err %d errno %d\n", err, errno);
505539
540+ /* check the stored flow_keys only if BPF_OK expected */
541+ if (tests[i].retval != BPF_OK)
542+ continue;
543+
506544 err = bpf_map_lookup_elem(keys_fd, &key, &flow_keys);
507545 ASSERT_OK(err, "bpf_map_lookup_elem");
508546
@@ -588,7 +626,11 @@ void test_flow_dissector(void) · 79 unmodified lines
588626
589627 err = bpf_prog_test_run_opts(prog_fd, &topts);
590628 ASSERT_OK(err, "test_run");
591− ASSERT_EQ(topts.retval, 1, "test_run retval");
629+ ASSERT_EQ(topts.retval, tests[i].retval, "test_run retval");
630+
631+ /* check the resulting flow_keys only if BPF_OK returned */
632+ if (topts.retval != BPF_OK)
633+ continue;
592634 ASSERT_EQ(topts.data_size_out, sizeof(flow_keys),
593635 "test_run data_size_out");
594636 CHECK_FLOW_KEYS(tests[i].name, flow_keys, tests[i].keys);
tools/testing/selftests/bpf/prog_tests/flow_dissector_load_bytes.c+1−1
@@ -44,7 +44,7 @@ void serial_test_flow_dissector_load_bytes(void) · 43 unmodified lines
4444 ASSERT_OK(err, "test_run");
4545 ASSERT_EQ(topts.data_size_out, sizeof(flow_keys),
4646 "test_run data_size_out");
47− ASSERT_EQ(topts.retval, 1, "test_run retval");
47+ ASSERT_EQ(topts.retval, BPF_OK, "test_run retval");
4848
4949 if (fd >= -1)
5050 close(fd);
tools/testing/selftests/bpf/prog_tests/get_stack_raw_tp.c+2−2
@@ -84,8 +84,8 @@ static void get_stack_print_output(void *ctx, int cpu, void *data, __u32 size) · 83 unmodified lines
8484
8585 void test_get_stack_raw_tp(void)
8686 {
87− const char *file = "./test_get_stack_rawtp.o";
88− const char *file_err = "./test_get_stack_rawtp_err.o";
87+ const char *file = "./test_get_stack_rawtp.bpf.o";
88+ const char *file_err = "./test_get_stack_rawtp_err.bpf.o";
8989 const char *prog_name = "bpf_prog1";
9090 int i, err, prog_fd, exp_cnt = MAX_CNT_RAWTP;
9191 struct perf_buffer *pb = NULL;
tools/testing/selftests/bpf/prog_tests/global_data.c+1−1
@@ -131,7 +131,7 @@ static void test_global_data_rdonly(struct bpf_object *obj, __u32 duration) · 130 unmodified lines
131131
132132 void test_global_data(void)
133133 {
134− const char *file = "./test_global_data.o";
134+ const char *file = "./test_global_data.bpf.o";
135135 struct bpf_object *obj;
136136 int err, prog_fd;
137137 LIBBPF_OPTS(bpf_test_run_opts, topts,
tools/testing/selftests/bpf/prog_tests/global_data_init.c+1−1
@@ -3,7 +3,7 @@ · 2 unmodified lines
33
44 void test_global_data_init(void)
55 {
6− const char *file = "./test_global_data.o";
6+ const char *file = "./test_global_data.bpf.o";
77 int err = -ENOMEM, map_fd, zero = 0;
88 __u8 *buff = NULL, *newval = NULL;
99 struct bpf_object *obj;
tools/testing/selftests/bpf/prog_tests/global_func_args.c+1−1
@@ -39,7 +39,7 @@ static void test_global_func_args0(struct bpf_object *obj) · 38 unmodified lines
3939
4040 void test_global_func_args(void)
4141 {
42− const char *file = "./test_global_func_args.o";
42+ const char *file = "./test_global_func_args.bpf.o";
4343 struct bpf_object *obj;
4444 int err, prog_fd;
4545 LIBBPF_OPTS(bpf_test_run_opts, topts,
tools/testing/selftests/bpf/prog_tests/htab_update.c+126−0
@@ -0,0 +1,126 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (C) 2022. Huawei Technologies Co., Ltd */
3+#define _GNU_SOURCE
4+#include <sched.h>
5+#include <stdbool.h>
6+#include <test_progs.h>
7+#include "htab_update.skel.h"
8+
9+struct htab_update_ctx {
10+ int fd;
11+ int loop;
12+ bool stop;
13+};
14+
15+static void test_reenter_update(void)
16+{
17+ struct htab_update *skel;
18+ unsigned int key, value;
19+ int err;
20+
21+ skel = htab_update__open();
22+ if (!ASSERT_OK_PTR(skel, "htab_update__open"))
23+ return;
24+
25+ /* lookup_elem_raw() may be inlined and find_kernel_btf_id() will return -ESRCH */
26+ bpf_program__set_autoload(skel->progs.lookup_elem_raw, true);
27+ err = htab_update__load(skel);
28+ if (!ASSERT_TRUE(!err || err == -ESRCH, "htab_update__load") || err)
29+ goto out;
30+
31+ skel->bss->pid = getpid();
32+ err = htab_update__attach(skel);
33+ if (!ASSERT_OK(err, "htab_update__attach"))
34+ goto out;
35+
36+ /* Will trigger the reentrancy of bpf_map_update_elem() */
37+ key = 0;
38+ value = 0;
39+ err = bpf_map_update_elem(bpf_map__fd(skel->maps.htab), &key, &value, 0);
40+ if (!ASSERT_OK(err, "add element"))
41+ goto out;
42+
43+ ASSERT_EQ(skel->bss->update_err, -EBUSY, "no reentrancy");
44+out:
45+ htab_update__destroy(skel);
46+}
47+
48+static void *htab_update_thread(void *arg)
49+{
50+ struct htab_update_ctx *ctx = arg;
51+ cpu_set_t cpus;
52+ int i;
53+
54+ /* Pinned on CPU 0 */
55+ CPU_ZERO(&cpus);
56+ CPU_SET(0, &cpus);
57+ pthread_setaffinity_np(pthread_self(), sizeof(cpus), &cpus);
58+
59+ i = 0;
60+ while (i++ < ctx->loop && !ctx->stop) {
61+ unsigned int key = 0, value = 0;
62+ int err;
63+
64+ err = bpf_map_update_elem(ctx->fd, &key, &value, 0);
65+ if (err) {
66+ ctx->stop = true;
67+ return (void *)(long)err;
68+ }
69+ }
70+
71+ return NULL;
72+}
73+
74+static void test_concurrent_update(void)
75+{
76+ struct htab_update_ctx ctx;
77+ struct htab_update *skel;
78+ unsigned int i, nr;
79+ pthread_t *tids;
80+ int err;
81+
82+ skel = htab_update__open_and_load();
83+ if (!ASSERT_OK_PTR(skel, "htab_update__open_and_load"))
84+ return;
85+
86+ ctx.fd = bpf_map__fd(skel->maps.htab);
87+ ctx.loop = 1000;
88+ ctx.stop = false;
89+
90+ nr = 4;
91+ tids = calloc(nr, sizeof(*tids));
92+ if (!ASSERT_NEQ(tids, NULL, "no mem"))
93+ goto out;
94+
95+ for (i = 0; i < nr; i++) {
96+ err = pthread_create(&tids[i], NULL, htab_update_thread, &ctx);
97+ if (!ASSERT_OK(err, "pthread_create")) {
98+ unsigned int j;
99+
100+ ctx.stop = true;
101+ for (j = 0; j < i; j++)
102+ pthread_join(tids[j], NULL);
103+ goto out;
104+ }
105+ }
106+
107+ for (i = 0; i < nr; i++) {
108+ void *thread_err = NULL;
109+
110+ pthread_join(tids[i], &thread_err);
111+ ASSERT_EQ(thread_err, NULL, "update error");
112+ }
113+
114+out:
115+ if (tids)
116+ free(tids);
117+ htab_update__destroy(skel);
118+}
119+
120+void test_htab_update(void)
121+{
122+ if (test__start_subtest("reenter_update"))
123+ test_reenter_update();
124+ if (test__start_subtest("concurrent_update"))
125+ test_concurrent_update();
126+}
tools/testing/selftests/bpf/prog_tests/kfree_skb.c+1−1
@@ -69,7 +69,7 @@ void serial_test_kfree_skb(void) · 68 unmodified lines
6969 const int zero = 0;
7070 bool test_ok[2];
7171
72− err = bpf_prog_test_load("./test_pkt_access.o", BPF_PROG_TYPE_SCHED_CLS,
72+ err = bpf_prog_test_load("./test_pkt_access.bpf.o", BPF_PROG_TYPE_SCHED_CLS,
7373 &obj, &prog_fd);
7474 if (CHECK(err, "prog_load sched cls", "err %d errno %d\n", err, errno))
7575 return;
tools/testing/selftests/bpf/prog_tests/kfunc_call.c+1−1
@@ -109,7 +109,7 @@ static void test_destructive(void) · 108 unmodified lines
109109 {
110110 __u64 save_caps = 0;
111111
112− ASSERT_OK(test_destructive_open_and_load(), "succesful_load");
112+ ASSERT_OK(test_destructive_open_and_load(), "successful_load");
113113
114114 if (!ASSERT_OK(cap_disable_effective(1ULL << CAP_SYS_BOOT, &save_caps), "drop_caps"))
115115 return;
tools/testing/selftests/bpf/prog_tests/l4lb_all.c+2−2
@@ -90,7 +90,7 @@ out: · 89 unmodified lines
9090 void test_l4lb_all(void)
9191 {
9292 if (test__start_subtest("l4lb_inline"))
93− test_l4lb("test_l4lb.o");
93+ test_l4lb("test_l4lb.bpf.o");
9494 if (test__start_subtest("l4lb_noinline"))
95− test_l4lb("test_l4lb_noinline.o");
95+ test_l4lb("test_l4lb_noinline.bpf.o");
9696 }
tools/testing/selftests/bpf/prog_tests/load_bytes_relative.c+2−2
@@ -27,8 +27,8 @@ void test_load_bytes_relative(void) · 26 unmodified lines
2727 if (CHECK_FAIL(server_fd < 0))
2828 goto close_cgroup_fd;
2929
30− err = bpf_prog_test_load("./load_bytes_relative.o", BPF_PROG_TYPE_CGROUP_SKB,
31− &obj, &prog_fd);
30+ err = bpf_prog_test_load("./load_bytes_relative.bpf.o", BPF_PROG_TYPE_CGROUP_SKB,
31+ &obj, &prog_fd);
3232 if (CHECK_FAIL(err))
3333 goto close_server_fd;
3434
tools/testing/selftests/bpf/prog_tests/map_lock.c+1−1
@@ -49,7 +49,7 @@ out: · 48 unmodified lines
4949
5050 void test_map_lock(void)
5151 {
52− const char *file = "./test_map_lock.o";
52+ const char *file = "./test_map_lock.bpf.o";
5353 int prog_fd, map_fd[2], vars[17] = {};
5454 pthread_t thread_id[6];
5555 struct bpf_object *obj = NULL;
tools/testing/selftests/bpf/prog_tests/pinning.c+2−2
@@ -26,13 +26,13 @@ __u32 get_map_id(struct bpf_object *obj, const char *name) · 25 unmodified lines
2626
2727 void test_pinning(void)
2828 {
29− const char *file_invalid = "./test_pinning_invalid.o";
29+ const char *file_invalid = "./test_pinning_invalid.bpf.o";
3030 const char *custpinpath = "/sys/fs/bpf/custom/pinmap";
3131 const char *nopinpath = "/sys/fs/bpf/nopinmap";
3232 const char *nopinpath2 = "/sys/fs/bpf/nopinmap2";
3333 const char *custpath = "/sys/fs/bpf/custom";
3434 const char *pinpath = "/sys/fs/bpf/pinmap";
35− const char *file = "./test_pinning.o";
35+ const char *file = "./test_pinning.bpf.o";
3636 __u32 map_id, map_id2, duration = 0;
3737 struct stat statbuf = {};
3838 struct bpf_object *obj;
tools/testing/selftests/bpf/prog_tests/pkt_access.c+1−1
@@ -4,7 +4,7 @@ · 3 unmodified lines
44
55 void test_pkt_access(void)
66 {
7− const char *file = "./test_pkt_access.o";
7+ const char *file = "./test_pkt_access.bpf.o";
88 struct bpf_object *obj;
99 int err, prog_fd;
1010 LIBBPF_OPTS(bpf_test_run_opts, topts,
tools/testing/selftests/bpf/prog_tests/pkt_md_access.c+1−1
@@ -4,7 +4,7 @@ · 3 unmodified lines
44
55 void test_pkt_md_access(void)
66 {
7− const char *file = "./test_pkt_md_access.o";
7+ const char *file = "./test_pkt_md_access.bpf.o";
88 struct bpf_object *obj;
99 int err, prog_fd;
1010 LIBBPF_OPTS(bpf_test_run_opts, topts,
tools/testing/selftests/bpf/prog_tests/probe_user.c+1−1
@@ -11,7 +11,7 @@ void serial_test_probe_user(void) · 10 unmodified lines
1111 #endif
1212 };
1313 enum { prog_count = ARRAY_SIZE(prog_names) };
14− const char *obj_file = "./test_probe_user.o";
14+ const char *obj_file = "./test_probe_user.bpf.o";
1515 DECLARE_LIBBPF_OPTS(bpf_object_open_opts, opts, );
1616 int err, results_map_fd, sock_fd, duration = 0;
1717 struct sockaddr curr, orig, tmp;
tools/testing/selftests/bpf/prog_tests/queue_stack_map.c+2−2
@@ -28,9 +28,9 @@ static void test_queue_stack_map_by_type(int type) · 27 unmodified lines
2828 vals[i] = rand();
2929
3030 if (type == QUEUE)
31− strncpy(file, "./test_queue_map.o", sizeof(file));
31+ strncpy(file, "./test_queue_map.bpf.o", sizeof(file));
3232 else if (type == STACK)
33− strncpy(file, "./test_stack_map.o", sizeof(file));
33+ strncpy(file, "./test_stack_map.bpf.o", sizeof(file));
3434 else
3535 return;
3636
tools/testing/selftests/bpf/prog_tests/rdonly_maps.c+1−1
@@ -16,7 +16,7 @@ struct rdonly_map_subtest { · 15 unmodified lines
1616
1717 void test_rdonly_maps(void)
1818 {
19− const char *file = "test_rdonly_maps.o";
19+ const char *file = "test_rdonly_maps.bpf.o";
2020 struct rdonly_map_subtest subtests[] = {
2121 { "skip loop", "skip_loop", 0, 0 },
2222 { "part loop", "part_loop", 3, 2 + 3 + 4 },
tools/testing/selftests/bpf/prog_tests/reference_tracking.c+1−1
@@ -3,7 +3,7 @@ · 2 unmodified lines
33
44 void test_reference_tracking(void)
55 {
6− const char *file = "test_sk_lookup_kern.o";
6+ const char *file = "test_sk_lookup_kern.bpf.o";
77 const char *obj_name = "ref_track";
88 DECLARE_LIBBPF_OPTS(bpf_object_open_opts, open_opts,
99 .object_name = obj_name,
tools/testing/selftests/bpf/prog_tests/resolve_btfids.c+1−1
@@ -101,7 +101,7 @@ static int resolve_symbols(void) · 100 unmodified lines
101101 int type_id;
102102 __u32 nr;
103103
104− btf = btf__parse_elf("btf_data.o", NULL);
104+ btf = btf__parse_elf("btf_data.bpf.o", NULL);
105105 if (CHECK(libbpf_get_error(btf), "resolve",
106106 "Failed to load BTF from btf_data.o\n"))
107107 return -1;
tools/testing/selftests/bpf/prog_tests/select_reuseport.c+2−2
@@ -91,9 +91,9 @@ static int prepare_bpf_obj(void) · 90 unmodified lines
9191 struct bpf_map *map;
9292 int err;
9393
94− obj = bpf_object__open("test_select_reuseport_kern.o");
94+ obj = bpf_object__open("test_select_reuseport_kern.bpf.o");
9595 err = libbpf_get_error(obj);
96− RET_ERR(err, "open test_select_reuseport_kern.o",
96+ RET_ERR(err, "open test_select_reuseport_kern.bpf.o",
9797 "obj:%p PTR_ERR(obj):%d\n", obj, err);
9898
9999 map = bpf_object__find_map_by_name(obj, "outer_map");
tools/testing/selftests/bpf/prog_tests/setget_sockopt.c+125−0
@@ -0,0 +1,125 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (c) Meta Platforms, Inc. and affiliates. */
3+
4+#define _GNU_SOURCE
5+#include <sched.h>
6+#include <linux/socket.h>
7+#include <net/if.h>
8+
9+#include "test_progs.h"
10+#include "cgroup_helpers.h"
11+#include "network_helpers.h"
12+
13+#include "setget_sockopt.skel.h"
14+
15+#define CG_NAME "/setget-sockopt-test"
16+
17+static const char addr4_str[] = "127.0.0.1";
18+static const char addr6_str[] = "::1";
19+static struct setget_sockopt *skel;
20+static int cg_fd;
21+
22+static int create_netns(void)
23+{
24+ if (!ASSERT_OK(unshare(CLONE_NEWNET), "create netns"))
25+ return -1;
26+
27+ if (!ASSERT_OK(system("ip link set dev lo up"), "set lo up"))
28+ return -1;
29+
30+ if (!ASSERT_OK(system("ip link add dev binddevtest1 type veth peer name binddevtest2"),
31+ "add veth"))
32+ return -1;
33+
34+ if (!ASSERT_OK(system("ip link set dev binddevtest1 up"),
35+ "bring veth up"))
36+ return -1;
37+
38+ return 0;
39+}
40+
41+static void test_tcp(int family)
42+{
43+ struct setget_sockopt__bss *bss = skel->bss;
44+ int sfd, cfd;
45+
46+ memset(bss, 0, sizeof(*bss));
47+
48+ sfd = start_server(family, SOCK_STREAM,
49+ family == AF_INET6 ? addr6_str : addr4_str, 0, 0);
50+ if (!ASSERT_GE(sfd, 0, "start_server"))
51+ return;
52+
53+ cfd = connect_to_fd(sfd, 0);
54+ if (!ASSERT_GE(cfd, 0, "connect_to_fd_server")) {
55+ close(sfd);
56+ return;
57+ }
58+ close(sfd);
59+ close(cfd);
60+
61+ ASSERT_EQ(bss->nr_listen, 1, "nr_listen");
62+ ASSERT_EQ(bss->nr_connect, 1, "nr_connect");
63+ ASSERT_EQ(bss->nr_active, 1, "nr_active");
64+ ASSERT_EQ(bss->nr_passive, 1, "nr_passive");
65+ ASSERT_EQ(bss->nr_socket_post_create, 2, "nr_socket_post_create");
66+ ASSERT_EQ(bss->nr_binddev, 2, "nr_bind");
67+}
68+
69+static void test_udp(int family)
70+{
71+ struct setget_sockopt__bss *bss = skel->bss;
72+ int sfd;
73+
74+ memset(bss, 0, sizeof(*bss));
75+
76+ sfd = start_server(family, SOCK_DGRAM,
77+ family == AF_INET6 ? addr6_str : addr4_str, 0, 0);
78+ if (!ASSERT_GE(sfd, 0, "start_server"))
79+ return;
80+ close(sfd);
81+
82+ ASSERT_GE(bss->nr_socket_post_create, 1, "nr_socket_post_create");
83+ ASSERT_EQ(bss->nr_binddev, 1, "nr_bind");
84+}
85+
86+void test_setget_sockopt(void)
87+{
88+ cg_fd = test__join_cgroup(CG_NAME);
89+ if (cg_fd < 0)
90+ return;
91+
92+ if (create_netns())
93+ goto done;
94+
95+ skel = setget_sockopt__open();
96+ if (!ASSERT_OK_PTR(skel, "open skel"))
97+ goto done;
98+
99+ strcpy(skel->rodata->veth, "binddevtest1");
100+ skel->rodata->veth_ifindex = if_nametoindex("binddevtest1");
101+ if (!ASSERT_GT(skel->rodata->veth_ifindex, 0, "if_nametoindex"))
102+ goto done;
103+
104+ if (!ASSERT_OK(setget_sockopt__load(skel), "load skel"))
105+ goto done;
106+
107+ skel->links.skops_sockopt =
108+ bpf_program__attach_cgroup(skel->progs.skops_sockopt, cg_fd);
109+ if (!ASSERT_OK_PTR(skel->links.skops_sockopt, "attach cgroup"))
110+ goto done;
111+
112+ skel->links.socket_post_create =
113+ bpf_program__attach_cgroup(skel->progs.socket_post_create, cg_fd);
114+ if (!ASSERT_OK_PTR(skel->links.socket_post_create, "attach_cgroup"))
115+ goto done;
116+
117+ test_tcp(AF_INET6);
118+ test_tcp(AF_INET);
119+ test_udp(AF_INET6);
120+ test_udp(AF_INET);
121+
122+done:
123+ setget_sockopt__destroy(skel);
124+ close(cg_fd);
125+}
tools/testing/selftests/bpf/prog_tests/sk_assign.c+1−1
@@ -47,7 +47,7 @@ configure_stack(void) · 46 unmodified lines
4747 if (CHECK_FAIL(system("tc qdisc add dev lo clsact")))
4848 return false;
4949 sprintf(tc_cmd, "%s %s %s %s", "tc filter add dev lo ingress bpf",
50− "direct-action object-file ./test_sk_assign.o",
50+ "direct-action object-file ./test_sk_assign.bpf.o",
5151 "section tc",
5252 (env.verbosity < VERBOSE_VERY) ? " 2>/dev/null" : "verbose");
5353 if (CHECK(system(tc_cmd), "BPF load failed;",
tools/testing/selftests/bpf/prog_tests/skb_ctx.c+1−1
@@ -31,7 +31,7 @@ void test_skb_ctx(void) · 30 unmodified lines
3131 struct bpf_object *obj;
3232 int err, prog_fd, i;
3333
34− err = bpf_prog_test_load("./test_skb_ctx.o", BPF_PROG_TYPE_SCHED_CLS,
34+ err = bpf_prog_test_load("./test_skb_ctx.bpf.o", BPF_PROG_TYPE_SCHED_CLS,
3535 &obj, &prog_fd);
3636 if (!ASSERT_OK(err, "load"))
3737 return;
tools/testing/selftests/bpf/prog_tests/skb_helpers.c+1−1
@@ -20,7 +20,7 @@ void test_skb_helpers(void) · 19 unmodified lines
2020 struct bpf_object *obj;
2121 int err, prog_fd;
2222
23− err = bpf_prog_test_load("./test_skb_helpers.o",
23+ err = bpf_prog_test_load("./test_skb_helpers.bpf.o",
2424 BPF_PROG_TYPE_SCHED_CLS, &obj, &prog_fd);
2525 if (!ASSERT_OK(err, "load"))
2626 return;
tools/testing/selftests/bpf/prog_tests/sockopt_inherit.c+1−1
@@ -174,7 +174,7 @@ static void run_test(int cgroup_fd) · 173 unmodified lines
174174 pthread_t tid;
175175 int err;
176176
177− obj = bpf_object__open_file("sockopt_inherit.o", NULL);
177+ obj = bpf_object__open_file("sockopt_inherit.bpf.o", NULL);
178178 if (!ASSERT_OK_PTR(obj, "obj_open"))
179179 return;
180180
tools/testing/selftests/bpf/prog_tests/sockopt_multi.c+1−1
@@ -310,7 +310,7 @@ void test_sockopt_multi(void) · 309 unmodified lines
310310 if (CHECK_FAIL(cg_child < 0))
311311 goto out;
312312
313− obj = bpf_object__open_file("sockopt_multi.o", NULL);
313+ obj = bpf_object__open_file("sockopt_multi.bpf.o", NULL);
314314 if (!ASSERT_OK_PTR(obj, "obj_load"))
315315 goto out;
316316
tools/testing/selftests/bpf/prog_tests/spinlock.c+1−1
@@ -19,7 +19,7 @@ static void *spin_lock_thread(void *arg) · 18 unmodified lines
1919
2020 void test_spinlock(void)
2121 {
22− const char *file = "./test_spin_lock.o";
22+ const char *file = "./test_spin_lock.bpf.o";
2323 pthread_t thread_id[4];
2424 struct bpf_object *obj = NULL;
2525 int prog_fd;
tools/testing/selftests/bpf/prog_tests/stacktrace_map.c+1−1
@@ -6,7 +6,7 @@ void test_stacktrace_map(void) · 5 unmodified lines
66 int control_map_fd, stackid_hmap_fd, stackmap_fd, stack_amap_fd;
77 const char *prog_name = "oncpu";
88 int err, prog_fd, stack_trace_len;
9− const char *file = "./test_stacktrace_map.o";
9+ const char *file = "./test_stacktrace_map.bpf.o";
1010 __u32 key, val, duration = 0;
1111 struct bpf_program *prog;
1212 struct bpf_object *obj;
tools/testing/selftests/bpf/prog_tests/stacktrace_map_raw_tp.c+1−1
@@ -5,7 +5,7 @@ void test_stacktrace_map_raw_tp(void) · 4 unmodified lines
55 {
66 const char *prog_name = "oncpu";
77 int control_map_fd, stackid_hmap_fd, stackmap_fd;
8− const char *file = "./test_stacktrace_map.o";
8+ const char *file = "./test_stacktrace_map.bpf.o";
99 __u32 key, val, duration = 0;
1010 int err, prog_fd;
1111 struct bpf_program *prog;
tools/testing/selftests/bpf/prog_tests/tailcalls.c+18−18
@@ -20,8 +20,8 @@ static void test_tailcall_1(void) · 19 unmodified lines
2020 .repeat = 1,
2121 );
2222
23− err = bpf_prog_test_load("tailcall1.o", BPF_PROG_TYPE_SCHED_CLS, &obj,
24− &prog_fd);
23+ err = bpf_prog_test_load("tailcall1.bpf.o", BPF_PROG_TYPE_SCHED_CLS, &obj,
24+ &prog_fd);
2525 if (CHECK_FAIL(err))
2626 return;
2727
@@ -156,8 +156,8 @@ static void test_tailcall_2(void) · 128 unmodified lines
156156 .repeat = 1,
157157 );
158158
159− err = bpf_prog_test_load("tailcall2.o", BPF_PROG_TYPE_SCHED_CLS, &obj,
160− &prog_fd);
159+ err = bpf_prog_test_load("tailcall2.bpf.o", BPF_PROG_TYPE_SCHED_CLS, &obj,
160+ &prog_fd);
161161 if (CHECK_FAIL(err))
162162 return;
163163
@@ -299,7 +299,7 @@ out: · 135 unmodified lines
299299 */
300300 static void test_tailcall_3(void)
301301 {
302− test_tailcall_count("tailcall3.o");
302+ test_tailcall_count("tailcall3.bpf.o");
303303 }
304304
305305 /* test_tailcall_6 checks that the count value of the tail call limit
@@ -307,7 +307,7 @@ static void test_tailcall_3(void) · 1 unmodified line
307307 */
308308 static void test_tailcall_6(void)
309309 {
310− test_tailcall_count("tailcall6.o");
310+ test_tailcall_count("tailcall6.bpf.o");
311311 }
312312
313313 /* test_tailcall_4 checks that the kernel properly selects indirect jump
@@ -329,8 +329,8 @@ static void test_tailcall_4(void) · 15 unmodified lines
329329 .repeat = 1,
330330 );
331331
332− err = bpf_prog_test_load("tailcall4.o", BPF_PROG_TYPE_SCHED_CLS, &obj,
333− &prog_fd);
332+ err = bpf_prog_test_load("tailcall4.bpf.o", BPF_PROG_TYPE_SCHED_CLS, &obj,
333+ &prog_fd);
334334 if (CHECK_FAIL(err))
335335 return;
336336
@@ -419,8 +419,8 @@ static void test_tailcall_5(void) · 82 unmodified lines
419419 .repeat = 1,
420420 );
421421
422− err = bpf_prog_test_load("tailcall5.o", BPF_PROG_TYPE_SCHED_CLS, &obj,
423− &prog_fd);
422+ err = bpf_prog_test_load("tailcall5.bpf.o", BPF_PROG_TYPE_SCHED_CLS, &obj,
423+ &prog_fd);
424424 if (CHECK_FAIL(err))
425425 return;
426426
@@ -507,8 +507,8 @@ static void test_tailcall_bpf2bpf_1(void) · 80 unmodified lines
507507 .repeat = 1,
508508 );
509509
510− err = bpf_prog_test_load("tailcall_bpf2bpf1.o", BPF_PROG_TYPE_SCHED_CLS,
511− &obj, &prog_fd);
510+ err = bpf_prog_test_load("tailcall_bpf2bpf1.bpf.o", BPF_PROG_TYPE_SCHED_CLS,
511+ &obj, &prog_fd);
512512 if (CHECK_FAIL(err))
513513 return;
514514
@@ -591,8 +591,8 @@ static void test_tailcall_bpf2bpf_2(void) · 76 unmodified lines
591591 .repeat = 1,
592592 );
593593
594− err = bpf_prog_test_load("tailcall_bpf2bpf2.o", BPF_PROG_TYPE_SCHED_CLS,
595− &obj, &prog_fd);
594+ err = bpf_prog_test_load("tailcall_bpf2bpf2.bpf.o", BPF_PROG_TYPE_SCHED_CLS,
595+ &obj, &prog_fd);
596596 if (CHECK_FAIL(err))
597597 return;
598598
@@ -671,8 +671,8 @@ static void test_tailcall_bpf2bpf_3(void) · 72 unmodified lines
671671 .repeat = 1,
672672 );
673673
674− err = bpf_prog_test_load("tailcall_bpf2bpf3.o", BPF_PROG_TYPE_SCHED_CLS,
675− &obj, &prog_fd);
674+ err = bpf_prog_test_load("tailcall_bpf2bpf3.bpf.o", BPF_PROG_TYPE_SCHED_CLS,
675+ &obj, &prog_fd);
676676 if (CHECK_FAIL(err))
677677 return;
678678
@@ -766,8 +766,8 @@ static void test_tailcall_bpf2bpf_4(bool noise) · 87 unmodified lines
766766 .repeat = 1,
767767 );
768768
769− err = bpf_prog_test_load("tailcall_bpf2bpf4.o", BPF_PROG_TYPE_SCHED_CLS,
770− &obj, &prog_fd);
769+ err = bpf_prog_test_load("tailcall_bpf2bpf4.bpf.o", BPF_PROG_TYPE_SCHED_CLS,
770+ &obj, &prog_fd);
771771 if (CHECK_FAIL(err))
772772 return;
773773
tools/testing/selftests/bpf/prog_tests/task_fd_query_rawtp.c+1−1
@@ -3,7 +3,7 @@ · 2 unmodified lines
33
44 void test_task_fd_query_rawtp(void)
55 {
6− const char *file = "./test_get_stack_rawtp.o";
6+ const char *file = "./test_get_stack_rawtp.bpf.o";
77 __u64 probe_offset, probe_addr;
88 __u32 len, prog_id, fd_type;
99 struct bpf_object *obj;
tools/testing/selftests/bpf/prog_tests/task_fd_query_tp.c+1−1
@@ -4,7 +4,7 @@ · 3 unmodified lines
44 static void test_task_fd_query_tp_core(const char *probe_name,
55 const char *tp_name)
66 {
7− const char *file = "./test_tracepoint.o";
7+ const char *file = "./test_tracepoint.bpf.o";
88 int err, bytes, efd, prog_fd, pmu_fd;
99 struct perf_event_attr attr = {};
1010 __u64 probe_offset, probe_addr;
tools/testing/selftests/bpf/prog_tests/tcp_estats.c+1−1
@@ -3,7 +3,7 @@ · 2 unmodified lines
33
44 void test_tcp_estats(void)
55 {
6− const char *file = "./test_tcp_estats.o";
6+ const char *file = "./test_tcp_estats.bpf.o";
77 int err, prog_fd;
88 struct bpf_object *obj;
99 __u32 duration = 0;
tools/testing/selftests/bpf/prog_tests/test_bprm_opts.c+1−9
@@ -9,18 +9,10 @@ · 8 unmodified lines
99
1010 #include "bprm_opts.skel.h"
1111 #include "network_helpers.h"
12−
13−#ifndef __NR_pidfd_open
14−#define __NR_pidfd_open 434
15−#endif
12+#include "task_local_storage_helpers.h"
1613
1714 static const char * const bash_envp[] = { "TMPDIR=shouldnotbeset", NULL };
1815
19−static inline int sys_pidfd_open(pid_t pid, unsigned int flags)
20−{
21− return syscall(__NR_pidfd_open, pid, flags);
22−}
23−
2416 static int update_storage(int map_fd, int secureexec)
2517 {
2618 int task_fd, ret = 0;
tools/testing/selftests/bpf/prog_tests/test_global_funcs.c+17−17
@@ -65,23 +65,23 @@ struct test_def { · 64 unmodified lines
6565 void test_test_global_funcs(void)
6666 {
6767 struct test_def tests[] = {
68− { "test_global_func1.o", "combined stack size of 4 calls is 544" },
69− { "test_global_func2.o" },
70− { "test_global_func3.o" , "the call stack of 8 frames" },
71− { "test_global_func4.o" },
72− { "test_global_func5.o" , "expected pointer to ctx, but got PTR" },
73− { "test_global_func6.o" , "modified ctx ptr R2" },
74− { "test_global_func7.o" , "foo() doesn't return scalar" },
75− { "test_global_func8.o" },
76− { "test_global_func9.o" },
77− { "test_global_func10.o", "invalid indirect read from stack" },
78− { "test_global_func11.o", "Caller passes invalid args into func#1" },
79− { "test_global_func12.o", "invalid mem access 'mem_or_null'" },
80− { "test_global_func13.o", "Caller passes invalid args into func#1" },
81− { "test_global_func14.o", "reference type('FWD S') size cannot be determined" },
82− { "test_global_func15.o", "At program exit the register R0 has value" },
83− { "test_global_func16.o", "invalid indirect read from stack" },
84− { "test_global_func17.o", "Caller passes invalid args into func#1" },
68+ { "test_global_func1.bpf.o", "combined stack size of 4 calls is 544" },
69+ { "test_global_func2.bpf.o" },
70+ { "test_global_func3.bpf.o", "the call stack of 8 frames" },
71+ { "test_global_func4.bpf.o" },
72+ { "test_global_func5.bpf.o", "expected pointer to ctx, but got PTR" },
73+ { "test_global_func6.bpf.o", "modified ctx ptr R2" },
74+ { "test_global_func7.bpf.o", "foo() doesn't return scalar" },
75+ { "test_global_func8.bpf.o" },
76+ { "test_global_func9.bpf.o" },
77+ { "test_global_func10.bpf.o", "invalid indirect read from stack" },
78+ { "test_global_func11.bpf.o", "Caller passes invalid args into func#1" },
79+ { "test_global_func12.bpf.o", "invalid mem access 'mem_or_null'" },
80+ { "test_global_func13.bpf.o", "Caller passes invalid args into func#1" },
81+ { "test_global_func14.bpf.o", "reference type('FWD S') size cannot be determined" },
82+ { "test_global_func15.bpf.o", "At program exit the register R0 has value" },
83+ { "test_global_func16.bpf.o", "invalid indirect read from stack" },
84+ { "test_global_func17.bpf.o", "Caller passes invalid args into func#1" },
8585 };
8686 libbpf_print_fn_t old_print_fn = NULL;
8787 int err, i, duration = 0;
tools/testing/selftests/bpf/prog_tests/test_local_storage.c+1−9
@@ -11,15 +11,7 @@ · 10 unmodified lines
1111
1212 #include "local_storage.skel.h"
1313 #include "network_helpers.h"
14−
15−#ifndef __NR_pidfd_open
16−#define __NR_pidfd_open 434
17−#endif
18−
19−static inline int sys_pidfd_open(pid_t pid, unsigned int flags)
20−{
21− return syscall(__NR_pidfd_open, pid, flags);
22−}
14+#include "task_local_storage_helpers.h"
2315
2416 static unsigned int duration;
2517
tools/testing/selftests/bpf/prog_tests/test_overhead.c+1−1
@@ -72,7 +72,7 @@ void test_test_overhead(void) · 71 unmodified lines
7272 if (CHECK_FAIL(prctl(PR_GET_NAME, comm, 0L, 0L, 0L)))
7373 return;
7474
75− obj = bpf_object__open_file("./test_overhead.o", NULL);
75+ obj = bpf_object__open_file("./test_overhead.bpf.o", NULL);
7676 if (!ASSERT_OK_PTR(obj, "obj_open_file"))
7777 return;
7878
tools/testing/selftests/bpf/prog_tests/tp_attach_query.c+1−1
@@ -6,7 +6,7 @@ void serial_test_tp_attach_query(void) · 5 unmodified lines
66 const int num_progs = 3;
77 int i, j, bytes, efd, err, prog_fd[num_progs], pmu_fd[num_progs];
88 __u32 duration = 0, info_len, saved_prog_ids[num_progs];
9− const char *file = "./test_tracepoint.o";
9+ const char *file = "./test_tracepoint.bpf.o";
1010 struct perf_event_query_bpf *query;
1111 struct perf_event_attr attr = {};
1212 struct bpf_object *obj[num_progs];
tools/testing/selftests/bpf/prog_tests/trampoline_count.c+1−1
@@ -35,7 +35,7 @@ static struct bpf_program *load_prog(char *file, char *name, struct inst *inst) · 34 unmodified lines
3535 /* TODO: use different target function to run in concurrent mode */
3636 void serial_test_trampoline_count(void)
3737 {
38− char *file = "test_trampoline_count.o";
38+ char *file = "test_trampoline_count.bpf.o";
3939 char *const progs[] = { "fentry_test", "fmod_ret_test", "fexit_test" };
4040 struct inst inst[MAX_TRAMP_PROGS + 1] = {};
4141 struct bpf_program *prog;
tools/testing/selftests/bpf/prog_tests/xdp.c+1−1
@@ -8,7 +8,7 @@ void test_xdp(void) · 7 unmodified lines
88 struct vip key6 = {.protocol = 6, .family = AF_INET6};
99 struct iptnl_info value4 = {.family = AF_INET};
1010 struct iptnl_info value6 = {.family = AF_INET6};
11− const char *file = "./test_xdp.o";
11+ const char *file = "./test_xdp.bpf.o";
1212 struct bpf_object *obj;
1313 char buf[128];
1414 struct ipv6hdr iph6;
tools/testing/selftests/bpf/prog_tests/xdp_adjust_frags.c+1−1
@@ -4,7 +4,7 @@ · 3 unmodified lines
44
55 static void test_xdp_update_frags(void)
66 {
7− const char *file = "./test_xdp_update_frags.o";
7+ const char *file = "./test_xdp_update_frags.bpf.o";
88 int err, prog_fd, max_skb_frags, buf_size, num;
99 struct bpf_program *prog;
1010 struct bpf_object *obj;
tools/testing/selftests/bpf/prog_tests/xdp_adjust_tail.c+5−5
@@ -4,7 +4,7 @@ · 3 unmodified lines
44
55 static void test_xdp_adjust_tail_shrink(void)
66 {
7− const char *file = "./test_xdp_adjust_tail_shrink.o";
7+ const char *file = "./test_xdp_adjust_tail_shrink.bpf.o";
88 __u32 expect_sz;
99 struct bpf_object *obj;
1010 int err, prog_fd;
@@ -39,7 +39,7 @@ static void test_xdp_adjust_tail_shrink(void) · 28 unmodified lines
3939
4040 static void test_xdp_adjust_tail_grow(void)
4141 {
42− const char *file = "./test_xdp_adjust_tail_grow.o";
42+ const char *file = "./test_xdp_adjust_tail_grow.bpf.o";
4343 struct bpf_object *obj;
4444 char buf[4096]; /* avoid segfault: large buf to hold grow results */
4545 __u32 expect_sz;
@@ -73,7 +73,7 @@ static void test_xdp_adjust_tail_grow(void) · 27 unmodified lines
7373
7474 static void test_xdp_adjust_tail_grow2(void)
7575 {
76− const char *file = "./test_xdp_adjust_tail_grow.o";
76+ const char *file = "./test_xdp_adjust_tail_grow.bpf.o";
7777 char buf[4096]; /* avoid segfault: large buf to hold grow results */
7878 int tailroom = 320; /* SKB_DATA_ALIGN(sizeof(struct skb_shared_info))*/;
7979 struct bpf_object *obj;
@@ -135,7 +135,7 @@ static void test_xdp_adjust_tail_grow2(void) · 55 unmodified lines
135135
136136 static void test_xdp_adjust_frags_tail_shrink(void)
137137 {
138− const char *file = "./test_xdp_adjust_tail_shrink.o";
138+ const char *file = "./test_xdp_adjust_tail_shrink.bpf.o";
139139 __u32 exp_size;
140140 struct bpf_program *prog;
141141 struct bpf_object *obj;
@@ -202,7 +202,7 @@ out: · 60 unmodified lines
202202
203203 static void test_xdp_adjust_frags_tail_grow(void)
204204 {
205− const char *file = "./test_xdp_adjust_tail_grow.o";
205+ const char *file = "./test_xdp_adjust_tail_grow.bpf.o";
206206 __u32 exp_size;
207207 struct bpf_program *prog;
208208 struct bpf_object *obj;
tools/testing/selftests/bpf/prog_tests/xdp_attach.c+1−1
@@ -8,7 +8,7 @@ void serial_test_xdp_attach(void) · 7 unmodified lines
88 {
99 __u32 duration = 0, id1, id2, id0 = 0, len;
1010 struct bpf_object *obj1, *obj2, *obj3;
11− const char *file = "./test_xdp.o";
11+ const char *file = "./test_xdp.bpf.o";
1212 struct bpf_prog_info info = {};
1313 int err, fd1, fd2, fd3;
1414 LIBBPF_OPTS(bpf_xdp_attach_opts, opts);
tools/testing/selftests/bpf/prog_tests/xdp_info.c+1−1
@@ -7,7 +7,7 @@ · 6 unmodified lines
77 void serial_test_xdp_info(void)
88 {
99 __u32 len = sizeof(struct bpf_prog_info), duration = 0, prog_id;
10− const char *file = "./xdp_dummy.o";
10+ const char *file = "./xdp_dummy.bpf.o";
1111 struct bpf_prog_info info = {};
1212 struct bpf_object *obj;
1313 int err, prog_fd;
tools/testing/selftests/bpf/prog_tests/xdp_perf.c+1−1
@@ -3,7 +3,7 @@ · 2 unmodified lines
33
44 void test_xdp_perf(void)
55 {
6− const char *file = "./xdp_dummy.o";
6+ const char *file = "./xdp_dummy.bpf.o";
77 struct bpf_object *obj;
88 char in[128], out[128];
99 int err, prog_fd;
tools/testing/selftests/bpf/prog_tests/xdp_synproxy.c+1−1
@@ -82,7 +82,7 @@ static void test_synproxy(bool xdp) · 81 unmodified lines
8282 SYS("ethtool -K tmp0 tx off");
8383 if (xdp)
8484 /* Workaround required for veth. */
85− SYS("ip link set tmp0 xdp object xdp_dummy.o section xdp 2> /dev/null");
85+ SYS("ip link set tmp0 xdp object xdp_dummy.bpf.o section xdp 2> /dev/null");
8686
8787 ns = open_netns("synproxy");
8888 if (!ASSERT_OK_PTR(ns, "setns"))
tools/testing/selftests/bpf/progs/bind4_prog.c+0−2
@@ -6,8 +6,6 @@ · 5 unmodified lines
66 #include <linux/bpf.h>
77 #include <linux/in.h>
88 #include <linux/in6.h>
9−#include <sys/socket.h>
10−#include <netinet/tcp.h>
119 #include <linux/if.h>
1210 #include <errno.h>
1311
tools/testing/selftests/bpf/progs/bind6_prog.c+0−2
@@ -6,8 +6,6 @@ · 5 unmodified lines
66 #include <linux/bpf.h>
77 #include <linux/in.h>
88 #include <linux/in6.h>
9−#include <sys/socket.h>
10−#include <netinet/tcp.h>
119 #include <linux/if.h>
1210 #include <errno.h>
1311
tools/testing/selftests/bpf/progs/bpf_flow.c+15−0
@@ -22,6 +22,8 @@ · 21 unmodified lines
2222 #define PROG(F) PROG_(F, _##F)
2323 #define PROG_(NUM, NAME) SEC("flow_dissector") int flow_dissector_##NUM
2424
25+#define FLOW_CONTINUE_SADDR 0x7f00007f /* 127.0.0.127 */
26+
2527 /* These are the identifiers of the BPF programs that will be used in tail
2628 * calls. Name is limited to 16 characters, with the terminating character and
2729 * bpf_func_ above, we have only 6 to work with, anything after will be cropped.
@@ -143,6 +145,19 @@ int _dissect(struct __sk_buff *skb) · 115 unmodified lines
143145 {
144146 struct bpf_flow_keys *keys = skb->flow_keys;
145147
148+ if (keys->n_proto == bpf_htons(ETH_P_IP)) {
149+ /* IP traffic from FLOW_CONTINUE_SADDR falls-back to
150+ * standard dissector
151+ */
152+ struct iphdr *iph, _iph;
153+
154+ iph = bpf_flow_dissect_get_header(skb, sizeof(*iph), &_iph);
155+ if (iph && iph->ihl == 5 &&
156+ iph->saddr == bpf_htonl(FLOW_CONTINUE_SADDR)) {
157+ return BPF_FLOW_DISSECTOR_CONTINUE;
158+ }
159+ }
160+
146161 return parse_eth_proto(skb, keys->n_proto);
147162 }
148163
tools/testing/selftests/bpf/progs/bpf_iter.h+7−0
@@ -17,6 +17,7 @@ · 16 unmodified lines
1717 #define bpf_iter__bpf_sk_storage_map bpf_iter__bpf_sk_storage_map___not_used
1818 #define bpf_iter__sockmap bpf_iter__sockmap___not_used
1919 #define bpf_iter__bpf_link bpf_iter__bpf_link___not_used
20+#define bpf_iter__cgroup bpf_iter__cgroup___not_used
2021 #define btf_ptr btf_ptr___not_used
2122 #define BTF_F_COMPACT BTF_F_COMPACT___not_used
2223 #define BTF_F_NONAME BTF_F_NONAME___not_used
@@ -40,6 +41,7 @@ · 17 unmodified lines
4041 #undef bpf_iter__bpf_sk_storage_map
4142 #undef bpf_iter__sockmap
4243 #undef bpf_iter__bpf_link
44+#undef bpf_iter__cgroup
4345 #undef btf_ptr
4446 #undef BTF_F_COMPACT
4547 #undef BTF_F_NONAME
@@ -141,6 +143,11 @@ struct bpf_iter__bpf_link { · 95 unmodified lines
141143 struct bpf_link *link;
142144 };
143145
146+struct bpf_iter__cgroup {
147+ struct bpf_iter_meta *meta;
148+ struct cgroup *cgroup;
149+} __attribute__((preserve_access_index));
150+
144151 struct btf_ptr {
145152 void *ptr;
146153 __u32 type_id;
tools/testing/selftests/bpf/progs/bpf_tracing_net.h+31−1
@@ -6,13 +6,41 @@ · 5 unmodified lines
66 #define AF_INET6 10
77
88 #define SOL_SOCKET 1
9+#define SO_REUSEADDR 2
910 #define SO_SNDBUF 7
10−#define __SO_ACCEPTCON (1 << 16)
11+#define SO_RCVBUF 8
12+#define SO_KEEPALIVE 9
1113 #define SO_PRIORITY 12
14+#define SO_REUSEPORT 15
15+#define SO_RCVLOWAT 18
16+#define SO_BINDTODEVICE 25
17+#define SO_MARK 36
18+#define SO_MAX_PACING_RATE 47
19+#define SO_BINDTOIFINDEX 62
20+#define SO_TXREHASH 74
21+#define __SO_ACCEPTCON (1 << 16)
22+
23+#define IP_TOS 1
24+
25+#define IPV6_TCLASS 67
26+#define IPV6_AUTOFLOWLABEL 70
1227
1328 #define SOL_TCP 6
29+#define TCP_NODELAY 1
30+#define TCP_MAXSEG 2
31+#define TCP_KEEPIDLE 4
32+#define TCP_KEEPINTVL 5
33+#define TCP_KEEPCNT 6
34+#define TCP_SYNCNT 7
35+#define TCP_WINDOW_CLAMP 10
1436 #define TCP_CONGESTION 13
37+#define TCP_THIN_LINEAR_TIMEOUTS 16
38+#define TCP_USER_TIMEOUT 18
39+#define TCP_NOTSENT_LOWAT 25
40+#define TCP_SAVE_SYN 27
41+#define TCP_SAVED_SYN 28
1542 #define TCP_CA_NAME_MAX 16
43+#define TCP_NAGLE_OFF 1
1644
1745 #define ICSK_TIME_RETRANS 1
1846 #define ICSK_TIME_PROBE0 3
@@ -49,6 +77,8 @@ · 30 unmodified lines
4977 #define sk_state __sk_common.skc_state
5078 #define sk_v6_daddr __sk_common.skc_v6_daddr
5179 #define sk_v6_rcv_saddr __sk_common.skc_v6_rcv_saddr
80+#define sk_flags __sk_common.skc_flags
81+#define sk_reuse __sk_common.skc_reuse
5282
5383 #define s6_addr32 in6_u.u6_addr32
5484
tools/testing/selftests/bpf/progs/cb_refs.c+116−0
@@ -0,0 +1,116 @@
1+// SPDX-License-Identifier: GPL-2.0
2+#include <vmlinux.h>
3+#include <bpf/bpf_tracing.h>
4+#include <bpf/bpf_helpers.h>
5+
6+struct map_value {
7+ struct prog_test_ref_kfunc __kptr_ref *ptr;
8+};
9+
10+struct {
11+ __uint(type, BPF_MAP_TYPE_ARRAY);
12+ __type(key, int);
13+ __type(value, struct map_value);
14+ __uint(max_entries, 16);
15+} array_map SEC(".maps");
16+
17+extern struct prog_test_ref_kfunc *bpf_kfunc_call_test_acquire(unsigned long *sp) __ksym;
18+extern void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) __ksym;
19+
20+static __noinline int cb1(void *map, void *key, void *value, void *ctx)
21+{
22+ void *p = *(void **)ctx;
23+ bpf_kfunc_call_test_release(p);
24+ /* Without the fix this would cause underflow */
25+ return 0;
26+}
27+
28+SEC("?tc")
29+int underflow_prog(void *ctx)
30+{
31+ struct prog_test_ref_kfunc *p;
32+ unsigned long sl = 0;
33+
34+ p = bpf_kfunc_call_test_acquire(&sl);
35+ if (!p)
36+ return 0;
37+ bpf_for_each_map_elem(&array_map, cb1, &p, 0);
38+ return 0;
39+}
40+
41+static __always_inline int cb2(void *map, void *key, void *value, void *ctx)
42+{
43+ unsigned long sl = 0;
44+
45+ *(void **)ctx = bpf_kfunc_call_test_acquire(&sl);
46+ /* Without the fix this would leak memory */
47+ return 0;
48+}
49+
50+SEC("?tc")
51+int leak_prog(void *ctx)
52+{
53+ struct prog_test_ref_kfunc *p;
54+ struct map_value *v;
55+ unsigned long sl;
56+
57+ v = bpf_map_lookup_elem(&array_map, &(int){0});
58+ if (!v)
59+ return 0;
60+
61+ p = NULL;
62+ bpf_for_each_map_elem(&array_map, cb2, &p, 0);
63+ p = bpf_kptr_xchg(&v->ptr, p);
64+ if (p)
65+ bpf_kfunc_call_test_release(p);
66+ return 0;
67+}
68+
69+static __always_inline int cb(void *map, void *key, void *value, void *ctx)
70+{
71+ return 0;
72+}
73+
74+static __always_inline int cb3(void *map, void *key, void *value, void *ctx)
75+{
76+ unsigned long sl = 0;
77+ void *p;
78+
79+ bpf_kfunc_call_test_acquire(&sl);
80+ bpf_for_each_map_elem(&array_map, cb, &p, 0);
81+ /* It should only complain here, not in cb. This is why we need
82+ * callback_ref to be set to frameno.
83+ */
84+ return 0;
85+}
86+
87+SEC("?tc")
88+int nested_cb(void *ctx)
89+{
90+ struct prog_test_ref_kfunc *p;
91+ unsigned long sl = 0;
92+ int sp = 0;
93+
94+ p = bpf_kfunc_call_test_acquire(&sl);
95+ if (!p)
96+ return 0;
97+ bpf_for_each_map_elem(&array_map, cb3, &sp, 0);
98+ bpf_kfunc_call_test_release(p);
99+ return 0;
100+}
101+
102+SEC("?tc")
103+int non_cb_transfer_ref(void *ctx)
104+{
105+ struct prog_test_ref_kfunc *p;
106+ unsigned long sl = 0;
107+
108+ p = bpf_kfunc_call_test_acquire(&sl);
109+ if (!p)
110+ return 0;
111+ cb1(NULL, NULL, NULL, &p);
112+ bpf_kfunc_call_test_acquire(&sl);
113+ return 0;
114+}
115+
116+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/cgroup_getset_retval_hooks.c+16−0
@@ -0,0 +1,16 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+
3+#include <linux/bpf.h>
4+#include <bpf/bpf_helpers.h>
5+
6+#define BPF_RETVAL_HOOK(name, section, ctx, expected_err) \
7+ __attribute__((__section__("?" section))) \
8+ int name(struct ctx *_ctx) \
9+ { \
10+ bpf_set_retval(bpf_get_retval()); \
11+ return 1; \
12+ }
13+
14+#include "cgroup_getset_retval_hooks.h"
15+
16+#undef BPF_RETVAL_HOOK
tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c+226−0
@@ -0,0 +1,226 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/*
3+ * Functions to manage eBPF programs attached to cgroup subsystems
4+ *
5+ * Copyright 2022 Google LLC.
6+ */
7+#include "vmlinux.h"
8+#include <bpf/bpf_helpers.h>
9+#include <bpf/bpf_tracing.h>
10+#include <bpf/bpf_core_read.h>
11+
12+char _license[] SEC("license") = "GPL";
13+
14+/*
15+ * Start times are stored per-task, not per-cgroup, as multiple tasks in one
16+ * cgroup can perform reclaim concurrently.
17+ */
18+struct {
19+ __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
20+ __uint(map_flags, BPF_F_NO_PREALLOC);
21+ __type(key, int);
22+ __type(value, __u64);
23+} vmscan_start_time SEC(".maps");
24+
25+struct vmscan_percpu {
26+ /* Previous percpu state, to figure out if we have new updates */
27+ __u64 prev;
28+ /* Current percpu state */
29+ __u64 state;
30+};
31+
32+struct vmscan {
33+ /* State propagated through children, pending aggregation */
34+ __u64 pending;
35+ /* Total state, including all cpus and all children */
36+ __u64 state;
37+};
38+
39+struct {
40+ __uint(type, BPF_MAP_TYPE_PERCPU_HASH);
41+ __uint(max_entries, 100);
42+ __type(key, __u64);
43+ __type(value, struct vmscan_percpu);
44+} pcpu_cgroup_vmscan_elapsed SEC(".maps");
45+
46+struct {
47+ __uint(type, BPF_MAP_TYPE_HASH);
48+ __uint(max_entries, 100);
49+ __type(key, __u64);
50+ __type(value, struct vmscan);
51+} cgroup_vmscan_elapsed SEC(".maps");
52+
53+extern void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) __ksym;
54+extern void cgroup_rstat_flush(struct cgroup *cgrp) __ksym;
55+
56+static struct cgroup *task_memcg(struct task_struct *task)
57+{
58+ int cgrp_id;
59+
60+#if __has_builtin(__builtin_preserve_enum_value)
61+ cgrp_id = bpf_core_enum_value(enum cgroup_subsys_id, memory_cgrp_id);
62+#else
63+ cgrp_id = memory_cgrp_id;
64+#endif
65+ return task->cgroups->subsys[cgrp_id]->cgroup;
66+}
67+
68+static uint64_t cgroup_id(struct cgroup *cgrp)
69+{
70+ return cgrp->kn->id;
71+}
72+
73+static int create_vmscan_percpu_elem(__u64 cg_id, __u64 state)
74+{
75+ struct vmscan_percpu pcpu_init = {.state = state, .prev = 0};
76+
77+ return bpf_map_update_elem(&pcpu_cgroup_vmscan_elapsed, &cg_id,
78+ &pcpu_init, BPF_NOEXIST);
79+}
80+
81+static int create_vmscan_elem(__u64 cg_id, __u64 state, __u64 pending)
82+{
83+ struct vmscan init = {.state = state, .pending = pending};
84+
85+ return bpf_map_update_elem(&cgroup_vmscan_elapsed, &cg_id,
86+ &init, BPF_NOEXIST);
87+}
88+
89+SEC("tp_btf/mm_vmscan_memcg_reclaim_begin")
90+int BPF_PROG(vmscan_start, int order, gfp_t gfp_flags)
91+{
92+ struct task_struct *task = bpf_get_current_task_btf();
93+ __u64 *start_time_ptr;
94+
95+ start_time_ptr = bpf_task_storage_get(&vmscan_start_time, task, 0,
96+ BPF_LOCAL_STORAGE_GET_F_CREATE);
97+ if (start_time_ptr)
98+ *start_time_ptr = bpf_ktime_get_ns();
99+ return 0;
100+}
101+
102+SEC("tp_btf/mm_vmscan_memcg_reclaim_end")
103+int BPF_PROG(vmscan_end, unsigned long nr_reclaimed)
104+{
105+ struct vmscan_percpu *pcpu_stat;
106+ struct task_struct *current = bpf_get_current_task_btf();
107+ struct cgroup *cgrp;
108+ __u64 *start_time_ptr;
109+ __u64 current_elapsed, cg_id;
110+ __u64 end_time = bpf_ktime_get_ns();
111+
112+ /*
113+ * cgrp is the first parent cgroup of current that has memcg enabled in
114+ * its subtree_control, or NULL if memcg is disabled in the entire tree.
115+ * In a cgroup hierarchy like this:
116+ * a
117+ * / \
118+ * b c
119+ * If "a" has memcg enabled, while "b" doesn't, then processes in "b"
120+ * will accumulate their stats directly to "a". This makes sure that no
121+ * stats are lost from processes in leaf cgroups that don't have memcg
122+ * enabled, but only exposes stats for cgroups that have memcg enabled.
123+ */
124+ cgrp = task_memcg(current);
125+ if (!cgrp)
126+ return 0;
127+
128+ cg_id = cgroup_id(cgrp);
129+ start_time_ptr = bpf_task_storage_get(&vmscan_start_time, current, 0,
130+ BPF_LOCAL_STORAGE_GET_F_CREATE);
131+ if (!start_time_ptr)
132+ return 0;
133+
134+ current_elapsed = end_time - *start_time_ptr;
135+ pcpu_stat = bpf_map_lookup_elem(&pcpu_cgroup_vmscan_elapsed,
136+ &cg_id);
137+ if (pcpu_stat)
138+ pcpu_stat->state += current_elapsed;
139+ else if (create_vmscan_percpu_elem(cg_id, current_elapsed))
140+ return 0;
141+
142+ cgroup_rstat_updated(cgrp, bpf_get_smp_processor_id());
143+ return 0;
144+}
145+
146+SEC("fentry/bpf_rstat_flush")
147+int BPF_PROG(vmscan_flush, struct cgroup *cgrp, struct cgroup *parent, int cpu)
148+{
149+ struct vmscan_percpu *pcpu_stat;
150+ struct vmscan *total_stat, *parent_stat;
151+ __u64 cg_id = cgroup_id(cgrp);
152+ __u64 parent_cg_id = parent ? cgroup_id(parent) : 0;
153+ __u64 *pcpu_vmscan;
154+ __u64 state;
155+ __u64 delta = 0;
156+
157+ /* Add CPU changes on this level since the last flush */
158+ pcpu_stat = bpf_map_lookup_percpu_elem(&pcpu_cgroup_vmscan_elapsed,
159+ &cg_id, cpu);
160+ if (pcpu_stat) {
161+ state = pcpu_stat->state;
162+ delta += state - pcpu_stat->prev;
163+ pcpu_stat->prev = state;
164+ }
165+
166+ total_stat = bpf_map_lookup_elem(&cgroup_vmscan_elapsed, &cg_id);
167+ if (!total_stat) {
168+ if (create_vmscan_elem(cg_id, delta, 0))
169+ return 0;
170+
171+ goto update_parent;
172+ }
173+
174+ /* Collect pending stats from subtree */
175+ if (total_stat->pending) {
176+ delta += total_stat->pending;
177+ total_stat->pending = 0;
178+ }
179+
180+ /* Propagate changes to this cgroup's total */
181+ total_stat->state += delta;
182+
183+update_parent:
184+ /* Skip if there are no changes to propagate, or no parent */
185+ if (!delta || !parent_cg_id)
186+ return 0;
187+
188+ /* Propagate changes to cgroup's parent */
189+ parent_stat = bpf_map_lookup_elem(&cgroup_vmscan_elapsed,
190+ &parent_cg_id);
191+ if (parent_stat)
192+ parent_stat->pending += delta;
193+ else
194+ create_vmscan_elem(parent_cg_id, 0, delta);
195+ return 0;
196+}
197+
198+SEC("iter.s/cgroup")
199+int BPF_PROG(dump_vmscan, struct bpf_iter_meta *meta, struct cgroup *cgrp)
200+{
201+ struct seq_file *seq = meta->seq;
202+ struct vmscan *total_stat;
203+ __u64 cg_id = cgrp ? cgroup_id(cgrp) : 0;
204+
205+ /* Do nothing for the terminal call */
206+ if (!cg_id)
207+ return 1;
208+
209+ /* Flush the stats to make sure we get the most updated numbers */
210+ cgroup_rstat_flush(cgrp);
211+
212+ total_stat = bpf_map_lookup_elem(&cgroup_vmscan_elapsed, &cg_id);
213+ if (!total_stat) {
214+ BPF_SEQ_PRINTF(seq, "cg_id: %llu, total_vmscan_delay: 0\n",
215+ cg_id);
216+ } else {
217+ BPF_SEQ_PRINTF(seq, "cg_id: %llu, total_vmscan_delay: %llu\n",
218+ cg_id, total_stat->state);
219+ }
220+
221+ /*
222+ * We only dump stats for one cgroup here, so return 1 to stop
223+ * iteration after the first cgroup.
224+ */
225+ return 1;
226+}
tools/testing/selftests/bpf/progs/cgroup_iter.c+39−0
@@ -0,0 +1,39 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (c) 2022 Google */
3+
4+#include "bpf_iter.h"
5+#include <bpf/bpf_helpers.h>
6+#include <bpf/bpf_tracing.h>
7+
8+char _license[] SEC("license") = "GPL";
9+int terminate_early = 0;
10+u64 terminal_cgroup = 0;
11+
12+static inline u64 cgroup_id(struct cgroup *cgrp)
13+{
14+ return cgrp->kn->id;
15+}
16+
17+SEC("iter/cgroup")
18+int cgroup_id_printer(struct bpf_iter__cgroup *ctx)
19+{
20+ struct seq_file *seq = ctx->meta->seq;
21+ struct cgroup *cgrp = ctx->cgroup;
22+
23+ /* epilogue */
24+ if (cgrp == NULL) {
25+ BPF_SEQ_PRINTF(seq, "epilogue\n");
26+ return 0;
27+ }
28+
29+ /* prologue */
30+ if (ctx->meta->seq_num == 0)
31+ BPF_SEQ_PRINTF(seq, "prologue\n");
32+
33+ BPF_SEQ_PRINTF(seq, "%8llu\n", cgroup_id(cgrp));
34+
35+ if (terminal_cgroup == cgroup_id(cgrp))
36+ return 1;
37+
38+ return terminate_early ? 1 : 0;
39+}
tools/testing/selftests/bpf/progs/connect4_prog.c+3−2
@@ -7,14 +7,15 @@ · 6 unmodified lines
77 #include <linux/bpf.h>
88 #include <linux/in.h>
99 #include <linux/in6.h>
10−#include <sys/socket.h>
11−#include <netinet/tcp.h>
10+#include <linux/tcp.h>
1211 #include <linux/if.h>
1312 #include <errno.h>
1413
1514 #include <bpf/bpf_helpers.h>
1615 #include <bpf/bpf_endian.h>
1716
17+#include "bpf_tcp_helpers.h"
18+
1819 #define SRC_REWRITE_IP4 0x7f000004U
1920 #define DST_REWRITE_IP4 0x7f000001U
2021 #define DST_REWRITE_PORT4 4444
tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c+4−4
@@ -73,10 +73,10 @@ int test_subprog2(struct args_subprog2 *ctx) · 72 unmodified lines
7373 __builtin_preserve_access_index(&skb->len));
7474
7575 ret = ctx->ret;
76− /* bpf_prog_test_load() loads "test_pkt_access.o" with BPF_F_TEST_RND_HI32
77− * which randomizes upper 32 bits after BPF_ALU32 insns.
78− * Hence after 'w0 <<= 1' upper bits of $rax are random.
79− * That is expected and correct. Trim them.
76+ /* bpf_prog_test_load() loads "test_pkt_access.bpf.o" with
77+ * BPF_F_TEST_RND_HI32 which randomizes upper 32 bits after BPF_ALU32
78+ * insns. Hence after 'w0 <<= 1' upper bits of $rax are random. That is
79+ * expected and correct. Trim them.
8080 */
8181 ret = (__u32) ret;
8282 if (len != 74 || ret != 148)
tools/testing/selftests/bpf/progs/htab_update.c+29−0
@@ -0,0 +1,29 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (C) 2022. Huawei Technologies Co., Ltd */
3+#include <linux/bpf.h>
4+#include <bpf/bpf_helpers.h>
5+#include <bpf/bpf_tracing.h>
6+
7+char _license[] SEC("license") = "GPL";
8+
9+struct {
10+ __uint(type, BPF_MAP_TYPE_HASH);
11+ __uint(max_entries, 1);
12+ __uint(key_size, sizeof(__u32));
13+ __uint(value_size, sizeof(__u32));
14+} htab SEC(".maps");
15+
16+int pid = 0;
17+int update_err = 0;
18+
19+SEC("?fentry/lookup_elem_raw")
20+int lookup_elem_raw(void *ctx)
21+{
22+ __u32 key = 0, value = 1;
23+
24+ if ((bpf_get_current_pid_tgid() >> 32) != pid)
25+ return 0;
26+
27+ update_err = bpf_map_update_elem(&htab, &key, &value, 0);
28+ return 0;
29+}
tools/testing/selftests/bpf/progs/read_bpf_task_storage_busy.c+39−0
@@ -0,0 +1,39 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (C) 2022. Huawei Technologies Co., Ltd */
3+#include "vmlinux.h"
4+#include <bpf/bpf_helpers.h>
5+#include <bpf/bpf_tracing.h>
6+
7+extern bool CONFIG_PREEMPT __kconfig __weak;
8+extern const int bpf_task_storage_busy __ksym;
9+
10+char _license[] SEC("license") = "GPL";
11+
12+int pid = 0;
13+int busy = 0;
14+
15+struct {
16+ __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
17+ __uint(map_flags, BPF_F_NO_PREALLOC);
18+ __type(key, int);
19+ __type(value, long);
20+} task SEC(".maps");
21+
22+SEC("raw_tp/sys_enter")
23+int BPF_PROG(read_bpf_task_storage_busy)
24+{
25+ int *value;
26+ int key;
27+
28+ if (!CONFIG_PREEMPT)
29+ return 0;
30+
31+ if (bpf_get_current_pid_tgid() >> 32 != pid)
32+ return 0;
33+
34+ value = bpf_this_cpu_ptr(&bpf_task_storage_busy);
35+ if (value)
36+ busy = *value;
37+
38+ return 0;
39+}
tools/testing/selftests/bpf/progs/setget_sockopt.c+395−0
@@ -0,0 +1,395 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (c) Meta Platforms, Inc. and affiliates. */
3+
4+#include "vmlinux.h"
5+#include "bpf_tracing_net.h"
6+#include <bpf/bpf_core_read.h>
7+#include <bpf/bpf_helpers.h>
8+#include <bpf/bpf_tracing.h>
9+
10+#ifndef ARRAY_SIZE
11+#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
12+#endif
13+
14+extern unsigned long CONFIG_HZ __kconfig;
15+
16+const volatile char veth[IFNAMSIZ];
17+const volatile int veth_ifindex;
18+
19+int nr_listen;
20+int nr_passive;
21+int nr_active;
22+int nr_connect;
23+int nr_binddev;
24+int nr_socket_post_create;
25+
26+struct sockopt_test {
27+ int opt;
28+ int new;
29+ int restore;
30+ int expected;
31+ int tcp_expected;
32+ unsigned int flip:1;
33+};
34+
35+static const char not_exist_cc[] = "not_exist";
36+static const char cubic_cc[] = "cubic";
37+static const char reno_cc[] = "reno";
38+
39+static const struct sockopt_test sol_socket_tests[] = {
40+ { .opt = SO_REUSEADDR, .flip = 1, },
41+ { .opt = SO_SNDBUF, .new = 8123, .expected = 8123 * 2, },
42+ { .opt = SO_RCVBUF, .new = 8123, .expected = 8123 * 2, },
43+ { .opt = SO_KEEPALIVE, .flip = 1, },
44+ { .opt = SO_PRIORITY, .new = 0xeb9f, .expected = 0xeb9f, },
45+ { .opt = SO_REUSEPORT, .flip = 1, },
46+ { .opt = SO_RCVLOWAT, .new = 8123, .expected = 8123, },
47+ { .opt = SO_MARK, .new = 0xeb9f, .expected = 0xeb9f, },
48+ { .opt = SO_MAX_PACING_RATE, .new = 0xeb9f, .expected = 0xeb9f, },
49+ { .opt = SO_TXREHASH, .flip = 1, },
50+ { .opt = 0, },
51+};
52+
53+static const struct sockopt_test sol_tcp_tests[] = {
54+ { .opt = TCP_NODELAY, .flip = 1, },
55+ { .opt = TCP_KEEPIDLE, .new = 123, .expected = 123, .restore = 321, },
56+ { .opt = TCP_KEEPINTVL, .new = 123, .expected = 123, .restore = 321, },
57+ { .opt = TCP_KEEPCNT, .new = 123, .expected = 123, .restore = 124, },
58+ { .opt = TCP_SYNCNT, .new = 123, .expected = 123, .restore = 124, },
59+ { .opt = TCP_WINDOW_CLAMP, .new = 8123, .expected = 8123, .restore = 8124, },
60+ { .opt = TCP_CONGESTION, },
61+ { .opt = TCP_THIN_LINEAR_TIMEOUTS, .flip = 1, },
62+ { .opt = TCP_USER_TIMEOUT, .new = 123400, .expected = 123400, },
63+ { .opt = TCP_NOTSENT_LOWAT, .new = 1314, .expected = 1314, },
64+ { .opt = 0, },
65+};
66+
67+static const struct sockopt_test sol_ip_tests[] = {
68+ { .opt = IP_TOS, .new = 0xe1, .expected = 0xe1, .tcp_expected = 0xe0, },
69+ { .opt = 0, },
70+};
71+
72+static const struct sockopt_test sol_ipv6_tests[] = {
73+ { .opt = IPV6_TCLASS, .new = 0xe1, .expected = 0xe1, .tcp_expected = 0xe0, },
74+ { .opt = IPV6_AUTOFLOWLABEL, .flip = 1, },
75+ { .opt = 0, },
76+};
77+
78+struct loop_ctx {
79+ void *ctx;
80+ struct sock *sk;
81+};
82+
83+static int bpf_test_sockopt_flip(void *ctx, struct sock *sk,
84+ const struct sockopt_test *t,
85+ int level)
86+{
87+ int old, tmp, new, opt = t->opt;
88+
89+ opt = t->opt;
90+
91+ if (bpf_getsockopt(ctx, level, opt, &old, sizeof(old)))
92+ return 1;
93+ /* kernel initialized txrehash to 255 */
94+ if (level == SOL_SOCKET && opt == SO_TXREHASH && old != 0 && old != 1)
95+ old = 1;
96+
97+ new = !old;
98+ if (bpf_setsockopt(ctx, level, opt, &new, sizeof(new)))
99+ return 1;
100+ if (bpf_getsockopt(ctx, level, opt, &tmp, sizeof(tmp)) ||
101+ tmp != new)
102+ return 1;
103+
104+ if (bpf_setsockopt(ctx, level, opt, &old, sizeof(old)))
105+ return 1;
106+
107+ return 0;
108+}
109+
110+static int bpf_test_sockopt_int(void *ctx, struct sock *sk,
111+ const struct sockopt_test *t,
112+ int level)
113+{
114+ int old, tmp, new, expected, opt;
115+
116+ opt = t->opt;
117+ new = t->new;
118+ if (sk->sk_type == SOCK_STREAM && t->tcp_expected)
119+ expected = t->tcp_expected;
120+ else
121+ expected = t->expected;
122+
123+ if (bpf_getsockopt(ctx, level, opt, &old, sizeof(old)) ||
124+ old == new)
125+ return 1;
126+
127+ if (bpf_setsockopt(ctx, level, opt, &new, sizeof(new)))
128+ return 1;
129+ if (bpf_getsockopt(ctx, level, opt, &tmp, sizeof(tmp)) ||
130+ tmp != expected)
131+ return 1;
132+
133+ if (t->restore)
134+ old = t->restore;
135+ if (bpf_setsockopt(ctx, level, opt, &old, sizeof(old)))
136+ return 1;
137+
138+ return 0;
139+}
140+
141+static int bpf_test_socket_sockopt(__u32 i, struct loop_ctx *lc)
142+{
143+ const struct sockopt_test *t;
144+
145+ if (i >= ARRAY_SIZE(sol_socket_tests))
146+ return 1;
147+
148+ t = &sol_socket_tests[i];
149+ if (!t->opt)
150+ return 1;
151+
152+ if (t->flip)
153+ return bpf_test_sockopt_flip(lc->ctx, lc->sk, t, SOL_SOCKET);
154+
155+ return bpf_test_sockopt_int(lc->ctx, lc->sk, t, SOL_SOCKET);
156+}
157+
158+static int bpf_test_ip_sockopt(__u32 i, struct loop_ctx *lc)
159+{
160+ const struct sockopt_test *t;
161+
162+ if (i >= ARRAY_SIZE(sol_ip_tests))
163+ return 1;
164+
165+ t = &sol_ip_tests[i];
166+ if (!t->opt)
167+ return 1;
168+
169+ if (t->flip)
170+ return bpf_test_sockopt_flip(lc->ctx, lc->sk, t, IPPROTO_IP);
171+
172+ return bpf_test_sockopt_int(lc->ctx, lc->sk, t, IPPROTO_IP);
173+}
174+
175+static int bpf_test_ipv6_sockopt(__u32 i, struct loop_ctx *lc)
176+{
177+ const struct sockopt_test *t;
178+
179+ if (i >= ARRAY_SIZE(sol_ipv6_tests))
180+ return 1;
181+
182+ t = &sol_ipv6_tests[i];
183+ if (!t->opt)
184+ return 1;
185+
186+ if (t->flip)
187+ return bpf_test_sockopt_flip(lc->ctx, lc->sk, t, IPPROTO_IPV6);
188+
189+ return bpf_test_sockopt_int(lc->ctx, lc->sk, t, IPPROTO_IPV6);
190+}
191+
192+static int bpf_test_tcp_sockopt(__u32 i, struct loop_ctx *lc)
193+{
194+ const struct sockopt_test *t;
195+ struct sock *sk;
196+ void *ctx;
197+
198+ if (i >= ARRAY_SIZE(sol_tcp_tests))
199+ return 1;
200+
201+ t = &sol_tcp_tests[i];
202+ if (!t->opt)
203+ return 1;
204+
205+ ctx = lc->ctx;
206+ sk = lc->sk;
207+
208+ if (t->opt == TCP_CONGESTION) {
209+ char old_cc[16], tmp_cc[16];
210+ const char *new_cc;
211+ int new_cc_len;
212+
213+ if (!bpf_setsockopt(ctx, IPPROTO_TCP, TCP_CONGESTION,
214+ (void *)not_exist_cc, sizeof(not_exist_cc)))
215+ return 1;
216+ if (bpf_getsockopt(ctx, IPPROTO_TCP, TCP_CONGESTION, old_cc, sizeof(old_cc)))
217+ return 1;
218+ if (!bpf_strncmp(old_cc, sizeof(old_cc), cubic_cc)) {
219+ new_cc = reno_cc;
220+ new_cc_len = sizeof(reno_cc);
221+ } else {
222+ new_cc = cubic_cc;
223+ new_cc_len = sizeof(cubic_cc);
224+ }
225+ if (bpf_setsockopt(ctx, IPPROTO_TCP, TCP_CONGESTION, (void *)new_cc,
226+ new_cc_len))
227+ return 1;
228+ if (bpf_getsockopt(ctx, IPPROTO_TCP, TCP_CONGESTION, tmp_cc, sizeof(tmp_cc)))
229+ return 1;
230+ if (bpf_strncmp(tmp_cc, sizeof(tmp_cc), new_cc))
231+ return 1;
232+ if (bpf_setsockopt(ctx, IPPROTO_TCP, TCP_CONGESTION, old_cc, sizeof(old_cc)))
233+ return 1;
234+ return 0;
235+ }
236+
237+ if (t->flip)
238+ return bpf_test_sockopt_flip(ctx, sk, t, IPPROTO_TCP);
239+
240+ return bpf_test_sockopt_int(ctx, sk, t, IPPROTO_TCP);
241+}
242+
243+static int bpf_test_sockopt(void *ctx, struct sock *sk)
244+{
245+ struct loop_ctx lc = { .ctx = ctx, .sk = sk, };
246+ __u16 family, proto;
247+ int n;
248+
249+ family = sk->sk_family;
250+ proto = sk->sk_protocol;
251+
252+ n = bpf_loop(ARRAY_SIZE(sol_socket_tests), bpf_test_socket_sockopt, &lc, 0);
253+ if (n != ARRAY_SIZE(sol_socket_tests))
254+ return -1;
255+
256+ if (proto == IPPROTO_TCP) {
257+ n = bpf_loop(ARRAY_SIZE(sol_tcp_tests), bpf_test_tcp_sockopt, &lc, 0);
258+ if (n != ARRAY_SIZE(sol_tcp_tests))
259+ return -1;
260+ }
261+
262+ if (family == AF_INET) {
263+ n = bpf_loop(ARRAY_SIZE(sol_ip_tests), bpf_test_ip_sockopt, &lc, 0);
264+ if (n != ARRAY_SIZE(sol_ip_tests))
265+ return -1;
266+ } else {
267+ n = bpf_loop(ARRAY_SIZE(sol_ipv6_tests), bpf_test_ipv6_sockopt, &lc, 0);
268+ if (n != ARRAY_SIZE(sol_ipv6_tests))
269+ return -1;
270+ }
271+
272+ return 0;
273+}
274+
275+static int binddev_test(void *ctx)
276+{
277+ const char empty_ifname[] = "";
278+ int ifindex, zero = 0;
279+
280+ if (bpf_setsockopt(ctx, SOL_SOCKET, SO_BINDTODEVICE,
281+ (void *)veth, sizeof(veth)))
282+ return -1;
283+ if (bpf_getsockopt(ctx, SOL_SOCKET, SO_BINDTOIFINDEX,
284+ &ifindex, sizeof(int)) ||
285+ ifindex != veth_ifindex)
286+ return -1;
287+
288+ if (bpf_setsockopt(ctx, SOL_SOCKET, SO_BINDTODEVICE,
289+ (void *)empty_ifname, sizeof(empty_ifname)))
290+ return -1;
291+ if (bpf_getsockopt(ctx, SOL_SOCKET, SO_BINDTOIFINDEX,
292+ &ifindex, sizeof(int)) ||
293+ ifindex != 0)
294+ return -1;
295+
296+ if (bpf_setsockopt(ctx, SOL_SOCKET, SO_BINDTOIFINDEX,
297+ (void *)&veth_ifindex, sizeof(int)))
298+ return -1;
299+ if (bpf_getsockopt(ctx, SOL_SOCKET, SO_BINDTOIFINDEX,
300+ &ifindex, sizeof(int)) ||
301+ ifindex != veth_ifindex)
302+ return -1;
303+
304+ if (bpf_setsockopt(ctx, SOL_SOCKET, SO_BINDTOIFINDEX,
305+ &zero, sizeof(int)))
306+ return -1;
307+ if (bpf_getsockopt(ctx, SOL_SOCKET, SO_BINDTOIFINDEX,
308+ &ifindex, sizeof(int)) ||
309+ ifindex != 0)
310+ return -1;
311+
312+ return 0;
313+}
314+
315+static int test_tcp_maxseg(void *ctx, struct sock *sk)
316+{
317+ int val = 1314, tmp;
318+
319+ if (sk->sk_state != TCP_ESTABLISHED)
320+ return bpf_setsockopt(ctx, IPPROTO_TCP, TCP_MAXSEG,
321+ &val, sizeof(val));
322+
323+ if (bpf_getsockopt(ctx, IPPROTO_TCP, TCP_MAXSEG, &tmp, sizeof(tmp)) ||
324+ tmp > val)
325+ return -1;
326+
327+ return 0;
328+}
329+
330+static int test_tcp_saved_syn(void *ctx, struct sock *sk)
331+{
332+ __u8 saved_syn[20];
333+ int one = 1;
334+
335+ if (sk->sk_state == TCP_LISTEN)
336+ return bpf_setsockopt(ctx, IPPROTO_TCP, TCP_SAVE_SYN,
337+ &one, sizeof(one));
338+
339+ return bpf_getsockopt(ctx, IPPROTO_TCP, TCP_SAVED_SYN,
340+ saved_syn, sizeof(saved_syn));
341+}
342+
343+SEC("lsm_cgroup/socket_post_create")
344+int BPF_PROG(socket_post_create, struct socket *sock, int family,
345+ int type, int protocol, int kern)
346+{
347+ struct sock *sk = sock->sk;
348+
349+ if (!sk)
350+ return 1;
351+
352+ nr_socket_post_create += !bpf_test_sockopt(sk, sk);
353+ nr_binddev += !binddev_test(sk);
354+
355+ return 1;
356+}
357+
358+SEC("sockops")
359+int skops_sockopt(struct bpf_sock_ops *skops)
360+{
361+ struct bpf_sock *bpf_sk = skops->sk;
362+ struct sock *sk;
363+
364+ if (!bpf_sk)
365+ return 1;
366+
367+ sk = (struct sock *)bpf_skc_to_tcp_sock(bpf_sk);
368+ if (!sk)
369+ return 1;
370+
371+ switch (skops->op) {
372+ case BPF_SOCK_OPS_TCP_LISTEN_CB:
373+ nr_listen += !(bpf_test_sockopt(skops, sk) ||
374+ test_tcp_maxseg(skops, sk) ||
375+ test_tcp_saved_syn(skops, sk));
376+ break;
377+ case BPF_SOCK_OPS_TCP_CONNECT_CB:
378+ nr_connect += !(bpf_test_sockopt(skops, sk) ||
379+ test_tcp_maxseg(skops, sk));
380+ break;
381+ case BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB:
382+ nr_active += !(bpf_test_sockopt(skops, sk) ||
383+ test_tcp_maxseg(skops, sk));
384+ break;
385+ case BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB:
386+ nr_passive += !(bpf_test_sockopt(skops, sk) ||
387+ test_tcp_maxseg(skops, sk) ||
388+ test_tcp_saved_syn(skops, sk));
389+ break;
390+ }
391+
392+ return 1;
393+}
394+
395+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/tailcall_bpf2bpf4.c+1−1
@@ -19,7 +19,7 @@ struct { · 18 unmodified lines
1919 int count = 0;
2020 int noise = 0;
2121
22−__always_inline int subprog_noise(void)
22+static __always_inline int subprog_noise(void)
2323 {
2424 __u32 key = 0;
2525
tools/testing/selftests/bpf/progs/test_tc_dtime.c+0−1
@@ -15,7 +15,6 @@ · 14 unmodified lines
1515 #include <linux/udp.h>
1616 #include <bpf/bpf_helpers.h>
1717 #include <bpf/bpf_endian.h>
18−#include <sys/socket.h>
1918
2019 /* veth_src --- veth_src_fwd --- veth_det_fwd --- veth_dst
2120 * | |
tools/testing/selftests/bpf/progs/test_tunnel_kern.c+16−8
@@ -12,6 +12,7 @@ · 11 unmodified lines
1212 #include <linux/bpf.h>
1313 #include <linux/if_ether.h>
1414 #include <linux/if_packet.h>
15+#include <linux/if_tunnel.h>
1516 #include <linux/ip.h>
1617 #include <linux/ipv6.h>
1718 #include <linux/icmp.h>
@@ -386,7 +387,8 @@ int vxlan_get_tunnel_src(struct __sk_buff *skb) · 368 unmodified lines
386387 __u32 orig_daddr;
387388 __u32 index = 0;
388389
389− ret = bpf_skb_get_tunnel_key(skb, &key, sizeof(key), 0);
390+ ret = bpf_skb_get_tunnel_key(skb, &key, sizeof(key),
391+ BPF_F_TUNINFO_FLAGS);
390392 if (ret < 0) {
391393 log_err(ret);
392394 return TC_ACT_SHOT;
@@ -398,10 +400,13 @@ int vxlan_get_tunnel_src(struct __sk_buff *skb) · 5 unmodified lines
398400 return TC_ACT_SHOT;
399401 }
400402
401− if (key.local_ipv4 != ASSIGNED_ADDR_VETH1 || md.gbp != 0x800FF) {
402− bpf_printk("vxlan key %d local ip 0x%x remote ip 0x%x gbp 0x%x\n",
403+ if (key.local_ipv4 != ASSIGNED_ADDR_VETH1 || md.gbp != 0x800FF ||
404+ !(key.tunnel_flags & TUNNEL_KEY) ||
405+ (key.tunnel_flags & TUNNEL_CSUM)) {
406+ bpf_printk("vxlan key %d local ip 0x%x remote ip 0x%x gbp 0x%x flags 0x%x\n",
403407 key.tunnel_id, key.local_ipv4,
404− key.remote_ipv4, md.gbp);
408+ key.remote_ipv4, md.gbp,
409+ bpf_ntohs(key.tunnel_flags));
405410 log_err(ret);
406411 return TC_ACT_SHOT;
407412 }
@@ -541,16 +546,19 @@ int ip6vxlan_get_tunnel_src(struct __sk_buff *skb) · 133 unmodified lines
541546 }
542547
543548 ret = bpf_skb_get_tunnel_key(skb, &key, sizeof(key),
544− BPF_F_TUNINFO_IPV6);
549+ BPF_F_TUNINFO_IPV6 | BPF_F_TUNINFO_FLAGS);
545550 if (ret < 0) {
546551 log_err(ret);
547552 return TC_ACT_SHOT;
548553 }
549554
550− if (bpf_ntohl(key.local_ipv6[3]) != *local_ip) {
551− bpf_printk("ip6vxlan key %d local ip6 ::%x remote ip6 ::%x label 0x%x\n",
555+ if (bpf_ntohl(key.local_ipv6[3]) != *local_ip ||
556+ !(key.tunnel_flags & TUNNEL_KEY) ||
557+ !(key.tunnel_flags & TUNNEL_CSUM)) {
558+ bpf_printk("ip6vxlan key %d local ip6 ::%x remote ip6 ::%x label 0x%x flags 0x%x\n",
552559 key.tunnel_id, bpf_ntohl(key.local_ipv6[3]),
553− bpf_ntohl(key.remote_ipv6[3]), key.tunnel_label);
560+ bpf_ntohl(key.remote_ipv6[3]), key.tunnel_label,
561+ bpf_ntohs(key.tunnel_flags));
554562 bpf_printk("local_ip 0x%x\n", *local_ip);
555563 log_err(ret);
556564 return TC_ACT_SHOT;
tools/testing/selftests/bpf/progs/timer.c+0−11
@@ -208,17 +208,6 @@ static int timer_cb2(void *map, int *key, struct hmap_elem *val) · 207 unmodified lines
208208 */
209209 bpf_map_delete_elem(map, key);
210210
211− /* in non-preallocated hashmap both 'key' and 'val' are RCU
212− * protected and still valid though this element was deleted
213− * from the map. Arm this timer for ~35 seconds. When callback
214− * finishes the call_rcu will invoke:
215− * htab_elem_free_rcu
216− * check_and_free_timer
217− * bpf_timer_cancel_and_free
218− * to cancel this 35 second sleep and delete the timer for real.
219− */
220− if (bpf_timer_start(&val->timer, 1ull << 35, 0) != 0)
221− err |= 256;
222211 ok |= 4;
223212 }
224213 return 0;
tools/testing/selftests/bpf/task_local_storage_helpers.h+18−0
@@ -0,0 +1,18 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+#ifndef __TASK_LOCAL_STORAGE_HELPER_H
3+#define __TASK_LOCAL_STORAGE_HELPER_H
4+
5+#include <unistd.h>
6+#include <sys/syscall.h>
7+#include <sys/types.h>
8+
9+#ifndef __NR_pidfd_open
10+#define __NR_pidfd_open 434
11+#endif
12+
13+static inline int sys_pidfd_open(pid_t pid, unsigned int flags)
14+{
15+ return syscall(__NR_pidfd_open, pid, flags);
16+}
17+
18+#endif
tools/testing/selftests/bpf/test_dev_cgroup.c+1−1
@@ -16,7 +16,7 @@ · 15 unmodified lines
1616 #include "cgroup_helpers.h"
1717 #include "testing_helpers.h"
1818
19−#define DEV_CGROUP_PROG "./dev_cgroup.o"
19+#define DEV_CGROUP_PROG "./dev_cgroup.bpf.o"
2020
2121 #define TEST_CGROUP "/test-bpf-based-device-cgroup/"
2222
tools/testing/selftests/bpf/test_flow_dissector.sh+8−0
@@ -115,6 +115,14 @@ tc filter add dev lo parent ffff: protocol ip pref 1337 flower ip_proto \ · 114 unmodified lines
115115 # Send 10 IPv4/UDP packets from port 10. Filter should not drop any.
116116 ./test_flow_dissector -i 4 -f 10
117117
118+echo "Testing IPv4 from 127.0.0.127 (fallback to generic dissector)..."
119+# Send 10 IPv4/UDP packets from port 8. Filter should not drop any.
120+./test_flow_dissector -i 4 -S 127.0.0.127 -f 8
121+# Send 10 IPv4/UDP packets from port 9. Filter should drop all.
122+./test_flow_dissector -i 4 -S 127.0.0.127 -f 9 -F
123+# Send 10 IPv4/UDP packets from port 10. Filter should not drop any.
124+./test_flow_dissector -i 4 -S 127.0.0.127 -f 10
125+
118126 echo "Testing IPIP..."
119127 # Send 10 IPv4/IPv4/UDP packets from port 8. Filter should not drop any.
120128 ./with_addr.sh ./with_tunnels.sh ./test_flow_dissector -o 4 -e bare -i 4 \
tools/testing/selftests/bpf/test_lirc_mode2_user.c+1−1
@@ -59,7 +59,7 @@ int main(int argc, char **argv) · 58 unmodified lines
5959 return 2;
6060 }
6161
62− ret = bpf_prog_test_load("test_lirc_mode2_kern.o",
62+ ret = bpf_prog_test_load("test_lirc_mode2_kern.bpf.o",
6363 BPF_PROG_TYPE_LIRC_MODE2, &obj, &progfd);
6464 if (ret) {
6565 printf("Failed to load bpf program\n");
tools/testing/selftests/bpf/test_maps.c+29−19
@@ -264,10 +264,11 @@ static void test_hashmap_percpu(unsigned int task, void *data) · 263 unmodified lines
264264 close(fd);
265265 }
266266
267+#define VALUE_SIZE 3
267268 static int helper_fill_hashmap(int max_entries)
268269 {
269270 int i, fd, ret;
270− long long key, value;
271+ long long key, value[VALUE_SIZE] = {};
271272
272273 fd = bpf_map_create(BPF_MAP_TYPE_HASH, NULL, sizeof(key), sizeof(value),
273274 max_entries, &map_opts);
@@ -276,8 +277,8 @@ static int helper_fill_hashmap(int max_entries) · 2 unmodified lines
276277 "err: %s, flags: 0x%x\n", strerror(errno), map_opts.map_flags);
277278
278279 for (i = 0; i < max_entries; i++) {
279− key = i; value = key;
280− ret = bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST);
280+ key = i; value[0] = key;
281+ ret = bpf_map_update_elem(fd, &key, value, BPF_NOEXIST);
281282 CHECK(ret != 0,
282283 "can't update hashmap",
283284 "err: %s\n", strerror(ret));
@@ -288,8 +289,8 @@ static int helper_fill_hashmap(int max_entries) · 4 unmodified lines
288289
289290 static void test_hashmap_walk(unsigned int task, void *data)
290291 {
291− int fd, i, max_entries = 1000;
292− long long key, value, next_key;
292+ int fd, i, max_entries = 10000;
293+ long long key, value[VALUE_SIZE], next_key;
293294 bool next_key_valid = true;
294295
295296 fd = helper_fill_hashmap(max_entries);
@@ -297,7 +298,7 @@ static void test_hashmap_walk(unsigned int task, void *data) · 1 unmodified line
297298 for (i = 0; bpf_map_get_next_key(fd, !i ? NULL : &key,
298299 &next_key) == 0; i++) {
299300 key = next_key;
300− assert(bpf_map_lookup_elem(fd, &key, &value) == 0);
301+ assert(bpf_map_lookup_elem(fd, &key, value) == 0);
301302 }
302303
303304 assert(i == max_entries);
@@ -305,9 +306,9 @@ static void test_hashmap_walk(unsigned int task, void *data) · 1 unmodified line
305306 assert(bpf_map_get_next_key(fd, NULL, &key) == 0);
306307 for (i = 0; next_key_valid; i++) {
307308 next_key_valid = bpf_map_get_next_key(fd, &key, &next_key) == 0;
308− assert(bpf_map_lookup_elem(fd, &key, &value) == 0);
309− value++;
310− assert(bpf_map_update_elem(fd, &key, &value, BPF_EXIST) == 0);
309+ assert(bpf_map_lookup_elem(fd, &key, value) == 0);
310+ value[0]++;
311+ assert(bpf_map_update_elem(fd, &key, value, BPF_EXIST) == 0);
311312 key = next_key;
312313 }
313314
@@ -316,8 +317,8 @@ static void test_hashmap_walk(unsigned int task, void *data) · 2 unmodified lines
316317 for (i = 0; bpf_map_get_next_key(fd, !i ? NULL : &key,
317318 &next_key) == 0; i++) {
318319 key = next_key;
319− assert(bpf_map_lookup_elem(fd, &key, &value) == 0);
320− assert(value - 1 == key);
320+ assert(bpf_map_lookup_elem(fd, &key, value) == 0);
321+ assert(value[0] - 1 == key);
321322 }
322323
323324 assert(i == max_entries);
@@ -651,9 +652,9 @@ static void test_stackmap(unsigned int task, void *data) · 327 unmodified lines
651652 #include <arpa/inet.h>
652653 #include <sys/select.h>
653654 #include <linux/err.h>
654−#define SOCKMAP_PARSE_PROG "./sockmap_parse_prog.o"
655−#define SOCKMAP_VERDICT_PROG "./sockmap_verdict_prog.o"
656−#define SOCKMAP_TCP_MSG_PROG "./sockmap_tcp_msg_prog.o"
655+#define SOCKMAP_PARSE_PROG "./sockmap_parse_prog.bpf.o"
656+#define SOCKMAP_VERDICT_PROG "./sockmap_verdict_prog.bpf.o"
657+#define SOCKMAP_TCP_MSG_PROG "./sockmap_tcp_msg_prog.bpf.o"
657658 static void test_sockmap(unsigned int tasks, void *data)
658659 {
659660 struct bpf_map *bpf_map_rx, *bpf_map_tx, *bpf_map_msg, *bpf_map_break;
@@ -1143,8 +1144,8 @@ out_sockmap: · 483 unmodified lines
11431144 exit(1);
11441145 }
11451146
1146−#define MAPINMAP_PROG "./test_map_in_map.o"
1147−#define MAPINMAP_INVALID_PROG "./test_map_in_map_invalid.o"
1147+#define MAPINMAP_PROG "./test_map_in_map.bpf.o"
1148+#define MAPINMAP_INVALID_PROG "./test_map_in_map_invalid.bpf.o"
11481149 static void test_map_in_map(void)
11491150 {
11501151 struct bpf_object *obj;
@@ -1371,16 +1372,16 @@ static void __run_parallel(unsigned int tasks, · 220 unmodified lines
13711372
13721373 static void test_map_stress(void)
13731374 {
1375+ run_parallel(100, test_hashmap_walk, NULL);
13741376 run_parallel(100, test_hashmap, NULL);
13751377 run_parallel(100, test_hashmap_percpu, NULL);
13761378 run_parallel(100, test_hashmap_sizes, NULL);
1377− run_parallel(100, test_hashmap_walk, NULL);
13781379
13791380 run_parallel(100, test_arraymap, NULL);
13801381 run_parallel(100, test_arraymap_percpu, NULL);
13811382 }
13821383
1383−#define TASKS 1024
1384+#define TASKS 100
13841385
13851386 #define DO_UPDATE 1
13861387 #define DO_DELETE 0
@@ -1432,6 +1433,8 @@ static void test_update_delete(unsigned int fn, void *data) · 45 unmodified lines
14321433 int fd = ((int *)data)[0];
14331434 int i, key, value, err;
14341435
1436+ if (fn & 1)
1437+ test_hashmap_walk(fn, NULL);
14351438 for (i = fn; i < MAP_SIZE; i += TASKS) {
14361439 key = value = i;
14371440
@@ -1455,7 +1458,7 @@ static void test_update_delete(unsigned int fn, void *data) · 17 unmodified lines
14551458
14561459 static void test_map_parallel(void)
14571460 {
1458− int i, fd, key = 0, value = 0;
1461+ int i, fd, key = 0, value = 0, j = 0;
14591462 int data[2];
14601463
14611464 fd = bpf_map_create(BPF_MAP_TYPE_HASH, NULL, sizeof(key), sizeof(value),
@@ -1466,6 +1469,7 @@ static void test_map_parallel(void) · 4 unmodified lines
14661469 exit(1);
14671470 }
14681471
1472+again:
14691473 /* Use the same fd in children to add elements to this map:
14701474 * child_0 adds key=0, key=1024, key=2048, ...
14711475 * child_1 adds key=1, key=1025, key=2049, ...
@@ -1502,6 +1506,12 @@ static void test_map_parallel(void) · 30 unmodified lines
15021506 key = -1;
15031507 assert(bpf_map_get_next_key(fd, NULL, &key) < 0 && errno == ENOENT);
15041508 assert(bpf_map_get_next_key(fd, &key, &key) < 0 && errno == ENOENT);
1509+
1510+ key = 0;
1511+ bpf_map_delete_elem(fd, &key);
1512+ if (j++ < 5)
1513+ goto again;
1514+ close(fd);
15051515 }
15061516
15071517 static void test_map_rdonly(void)
tools/testing/selftests/bpf/test_offload.py+11−11
@@ -782,7 +782,7 @@ if out.find("/sys/kernel/debug type debugfs") == -1: · 781 unmodified lines
782782 cmd("mount -t debugfs none /sys/kernel/debug")
783783
784784 # Check samples are compiled
785−samples = ["sample_ret0.o", "sample_map_ret0.o"]
785+samples = ["sample_ret0.bpf.o", "sample_map_ret0.bpf.o"]
786786 for s in samples:
787787 ret, out = cmd("ls %s/%s" % (bpf_test_dir, s), fail=False)
788788 skip(ret != 0, "sample %s/%s not found, please compile it" %
@@ -803,7 +803,7 @@ cmd("ip netns delete %s" % (ns)) · 14 unmodified lines
803803 netns = []
804804
805805 try:
806− obj = bpf_obj("sample_ret0.o")
806+ obj = bpf_obj("sample_ret0.bpf.o")
807807 bytecode = bpf_bytecode("1,6 0 0 4294967295,")
808808
809809 start_test("Test destruction of generic XDP...")
@@ -1023,7 +1023,7 @@ try: · 213 unmodified lines
10231023
10241024 sim.wait_for_flush()
10251025 start_test("Test non-offload XDP attaching to HW...")
1026− bpftool_prog_load("sample_ret0.o", "/sys/fs/bpf/nooffload")
1026+ bpftool_prog_load("sample_ret0.bpf.o", "/sys/fs/bpf/nooffload")
10271027 nooffload = bpf_pinned("/sys/fs/bpf/nooffload")
10281028 ret, _, err = sim.set_xdp(nooffload, "offload",
10291029 fail=False, include_stderr=True)
@@ -1032,7 +1032,7 @@ try: · 2 unmodified lines
10321032 rm("/sys/fs/bpf/nooffload")
10331033
10341034 start_test("Test offload XDP attaching to drv...")
1035− bpftool_prog_load("sample_ret0.o", "/sys/fs/bpf/offload",
1035+ bpftool_prog_load("sample_ret0.bpf.o", "/sys/fs/bpf/offload",
10361036 dev=sim['ifname'])
10371037 offload = bpf_pinned("/sys/fs/bpf/offload")
10381038 ret, _, err = sim.set_xdp(offload, "drv", fail=False, include_stderr=True)
@@ -1043,7 +1043,7 @@ try: · 4 unmodified lines
10431043
10441044 start_test("Test XDP load failure...")
10451045 sim.dfs["dev/bpf_bind_verifier_accept"] = 0
1046− ret, _, err = bpftool_prog_load("sample_ret0.o", "/sys/fs/bpf/offload",
1046+ ret, _, err = bpftool_prog_load("sample_ret0.bpf.o", "/sys/fs/bpf/offload",
10471047 dev=sim['ifname'], fail=False, include_stderr=True)
10481048 fail(ret == 0, "verifier should fail on load")
10491049 check_verifier_log(err, "[netdevsim] Hello from netdevsim!")
@@ -1169,7 +1169,7 @@ try: · 119 unmodified lines
11691169
11701170 simdev = NetdevSimDev()
11711171 sim, = simdev.nsims
1172− map_obj = bpf_obj("sample_map_ret0.o")
1172+ map_obj = bpf_obj("sample_map_ret0.bpf.o")
11731173 start_test("Test loading program with maps...")
11741174 sim.set_xdp(map_obj, "offload", JSON=False) # map fixup msg breaks JSON
11751175
@@ -1307,10 +1307,10 @@ try: · 131 unmodified lines
13071307 sims = (simA, simB1, simB2, simB3)
13081308 simB = (simB1, simB2, simB3)
13091309
1310− bpftool_prog_load("sample_map_ret0.o", "/sys/fs/bpf/nsimA",
1310+ bpftool_prog_load("sample_map_ret0.bpf.o", "/sys/fs/bpf/nsimA",
13111311 dev=simA['ifname'])
13121312 progA = bpf_pinned("/sys/fs/bpf/nsimA")
1313− bpftool_prog_load("sample_map_ret0.o", "/sys/fs/bpf/nsimB",
1313+ bpftool_prog_load("sample_map_ret0.bpf.o", "/sys/fs/bpf/nsimB",
13141314 dev=simB1['ifname'])
13151315 progB = bpf_pinned("/sys/fs/bpf/nsimB")
13161316
@@ -1344,14 +1344,14 @@ try: · 27 unmodified lines
13441344 mapA = bpftool("prog show %s" % (progA))[1]["map_ids"][0]
13451345 mapB = bpftool("prog show %s" % (progB))[1]["map_ids"][0]
13461346
1347− ret, _ = bpftool_prog_load("sample_map_ret0.o", "/sys/fs/bpf/nsimB_",
1347+ ret, _ = bpftool_prog_load("sample_map_ret0.bpf.o", "/sys/fs/bpf/nsimB_",
13481348 dev=simB3['ifname'],
13491349 maps=["idx 0 id %d" % (mapB)],
13501350 fail=False)
13511351 fail(ret != 0, "couldn't reuse a map on the same ASIC")
13521352 rm("/sys/fs/bpf/nsimB_")
13531353
1354− ret, _, err = bpftool_prog_load("sample_map_ret0.o", "/sys/fs/bpf/nsimA_",
1354+ ret, _, err = bpftool_prog_load("sample_map_ret0.bpf.o", "/sys/fs/bpf/nsimA_",
13551355 dev=simA['ifname'],
13561356 maps=["idx 0 id %d" % (mapB)],
13571357 fail=False, include_stderr=True)
@@ -1359,7 +1359,7 @@ try: · 1 unmodified line
13591359 fail(err.count("offload device mismatch between prog and map") == 0,
13601360 "error message missing for cross-ASIC map")
13611361
1362− ret, _, err = bpftool_prog_load("sample_map_ret0.o", "/sys/fs/bpf/nsimB_",
1362+ ret, _, err = bpftool_prog_load("sample_map_ret0.bpf.o", "/sys/fs/bpf/nsimB_",
13631363 dev=simB1['ifname'],
13641364 maps=["idx 0 id %d" % (mapA)],
13651365 fail=False, include_stderr=True)
tools/testing/selftests/bpf/test_skb_cgroup_id.sh+1−1
@@ -54,7 +54,7 @@ DIR=$(dirname $0) · 53 unmodified lines
5454 TEST_IF="test_cgid_1"
5555 TEST_IF_PEER="test_cgid_2"
5656 MAX_PING_TRIES=5
57−BPF_PROG_OBJ="${DIR}/test_skb_cgroup_id_kern.o"
57+BPF_PROG_OBJ="${DIR}/test_skb_cgroup_id_kern.bpf.o"
5858 BPF_PROG_SECTION="cgroup_id_logger"
5959 BPF_PROG_ID=0
6060 PROG="${DIR}/test_skb_cgroup_id_user"
tools/testing/selftests/bpf/test_sock_addr.c+8−8
@@ -26,14 +26,14 @@ · 25 unmodified lines
2626 #endif
2727
2828 #define CG_PATH "/foo"
29−#define CONNECT4_PROG_PATH "./connect4_prog.o"
30−#define CONNECT6_PROG_PATH "./connect6_prog.o"
31−#define SENDMSG4_PROG_PATH "./sendmsg4_prog.o"
32−#define SENDMSG6_PROG_PATH "./sendmsg6_prog.o"
33−#define RECVMSG4_PROG_PATH "./recvmsg4_prog.o"
34−#define RECVMSG6_PROG_PATH "./recvmsg6_prog.o"
35−#define BIND4_PROG_PATH "./bind4_prog.o"
36−#define BIND6_PROG_PATH "./bind6_prog.o"
29+#define CONNECT4_PROG_PATH "./connect4_prog.bpf.o"
30+#define CONNECT6_PROG_PATH "./connect6_prog.bpf.o"
31+#define SENDMSG4_PROG_PATH "./sendmsg4_prog.bpf.o"
32+#define SENDMSG6_PROG_PATH "./sendmsg6_prog.bpf.o"
33+#define RECVMSG4_PROG_PATH "./recvmsg4_prog.bpf.o"
34+#define RECVMSG6_PROG_PATH "./recvmsg6_prog.bpf.o"
35+#define BIND4_PROG_PATH "./bind4_prog.bpf.o"
36+#define BIND6_PROG_PATH "./bind6_prog.bpf.o"
3737
3838 #define SERV4_IP "192.168.1.254"
3939 #define SERV4_REWRITE_IP "127.0.0.1"
tools/testing/selftests/bpf/test_sockmap.c+2−2
@@ -52,8 +52,8 @@ static void running_handler(int a); · 51 unmodified lines
5252 #define S1_PORT 10000
5353 #define S2_PORT 10001
5454
55−#define BPF_SOCKMAP_FILENAME "test_sockmap_kern.o"
56−#define BPF_SOCKHASH_FILENAME "test_sockhash_kern.o"
55+#define BPF_SOCKMAP_FILENAME "test_sockmap_kern.bpf.o"
56+#define BPF_SOCKHASH_FILENAME "test_sockhash_kern.bpf.o"
5757 #define CG_PATH "/sockmap"
5858
5959 /* global sockets */
tools/testing/selftests/bpf/test_sysctl.c+3−3
@@ -1372,7 +1372,7 @@ static struct sysctl_test tests[] = { · 1371 unmodified lines
13721372 },
13731373 {
13741374 "C prog: deny all writes",
1375− .prog_file = "./test_sysctl_prog.o",
1375+ .prog_file = "./test_sysctl_prog.bpf.o",
13761376 .attach_type = BPF_CGROUP_SYSCTL,
13771377 .sysctl = "net/ipv4/tcp_mem",
13781378 .open_flags = O_WRONLY,
@@ -1381,7 +1381,7 @@ static struct sysctl_test tests[] = { · 2 unmodified lines
13811381 },
13821382 {
13831383 "C prog: deny access by name",
1384− .prog_file = "./test_sysctl_prog.o",
1384+ .prog_file = "./test_sysctl_prog.bpf.o",
13851385 .attach_type = BPF_CGROUP_SYSCTL,
13861386 .sysctl = "net/ipv4/route/mtu_expires",
13871387 .open_flags = O_RDONLY,
@@ -1389,7 +1389,7 @@ static struct sysctl_test tests[] = { · 1 unmodified line
13891389 },
13901390 {
13911391 "C prog: read tcp_mem",
1392− .prog_file = "./test_sysctl_prog.o",
1392+ .prog_file = "./test_sysctl_prog.bpf.o",
13931393 .attach_type = BPF_CGROUP_SYSCTL,
13941394 .sysctl = "net/ipv4/tcp_mem",
13951395 .open_flags = O_RDONLY,
tools/testing/selftests/bpf/test_tcp_check_syncookie.sh+1−1
@@ -76,7 +76,7 @@ main() · 75 unmodified lines
7676 DIR=$(dirname $0)
7777 TEST_IF=lo
7878 MAX_PING_TRIES=5
79−BPF_PROG_OBJ="${DIR}/test_tcp_check_syncookie_kern.o"
79+BPF_PROG_OBJ="${DIR}/test_tcp_check_syncookie_kern.bpf.o"
8080 CLSACT_SECTION="tc"
8181 XDP_SECTION="xdp"
8282 BPF_PROG_ID=0
tools/testing/selftests/bpf/test_tcpnotify_user.c+1−1
@@ -69,7 +69,7 @@ int verify_result(const struct tcpnotify_globals *result) · 68 unmodified lines
6969
7070 int main(int argc, char **argv)
7171 {
72− const char *file = "test_tcpnotify_kern.o";
72+ const char *file = "test_tcpnotify_kern.bpf.o";
7373 struct bpf_map *perf_map, *global_map;
7474 struct tcpnotify_globals g = {0};
7575 struct perf_buffer *pb = NULL;
tools/testing/selftests/bpf/test_xdp_redirect.sh+4−4
@@ -54,10 +54,10 @@ test_xdp_redirect() · 53 unmodified lines
5454 return 0
5555 fi
5656
57− ip -n ${NS1} link set veth11 $xdpmode obj xdp_dummy.o sec xdp &> /dev/null
58− ip -n ${NS2} link set veth22 $xdpmode obj xdp_dummy.o sec xdp &> /dev/null
59− ip link set dev veth1 $xdpmode obj test_xdp_redirect.o sec redirect_to_222 &> /dev/null
60− ip link set dev veth2 $xdpmode obj test_xdp_redirect.o sec redirect_to_111 &> /dev/null
57+ ip -n ${NS1} link set veth11 $xdpmode obj xdp_dummy.bpf.o sec xdp &> /dev/null
58+ ip -n ${NS2} link set veth22 $xdpmode obj xdp_dummy.bpf.o sec xdp &> /dev/null
59+ ip link set dev veth1 $xdpmode obj test_xdp_redirect.bpf.o sec redirect_to_222 &> /dev/null
60+ ip link set dev veth2 $xdpmode obj test_xdp_redirect.bpf.o sec redirect_to_111 &> /dev/null
6161
6262 if ip netns exec ${NS1} ping -c 1 10.1.1.22 &> /dev/null &&
6363 ip netns exec ${NS2} ping -c 1 10.1.1.11 &> /dev/null; then
tools/testing/selftests/bpf/test_xdp_redirect_multi.sh+1−1
@@ -94,7 +94,7 @@ setup_ns() · 93 unmodified lines
9494 # Add a neigh entry for IPv4 ping test
9595 ip -n ${NS[$i]} neigh add 192.0.2.253 lladdr 00:00:00:00:00:01 dev veth0
9696 ip -n ${NS[$i]} link set veth0 $mode obj \
97− xdp_dummy.o sec xdp &> /dev/null || \
97+ xdp_dummy.bpf.o sec xdp &> /dev/null || \
9898 { test_fail "Unable to load dummy xdp" && exit 1; }
9999 IFACES="$IFACES veth$i"
100100 veth_mac[$i]=$(ip -n ${NS[0]} link show veth$i | awk '/link\/ether/ {print $2}')
tools/testing/selftests/bpf/test_xdp_veth.sh+4−4
@@ -101,7 +101,7 @@ ip -n ${NS3} link set dev veth33 up · 100 unmodified lines
101101
102102 mkdir $BPF_DIR
103103 bpftool prog loadall \
104− xdp_redirect_map.o $BPF_DIR/progs type xdp \
104+ xdp_redirect_map.bpf.o $BPF_DIR/progs type xdp \
105105 pinmaps $BPF_DIR/maps
106106 bpftool map update pinned $BPF_DIR/maps/tx_port key 0 0 0 0 value 122 0 0 0
107107 bpftool map update pinned $BPF_DIR/maps/tx_port key 1 0 0 0 value 133 0 0 0
@@ -110,9 +110,9 @@ ip link set dev veth1 xdp pinned $BPF_DIR/progs/xdp_redirect_map_0 · 2 unmodified lines
110110 ip link set dev veth2 xdp pinned $BPF_DIR/progs/xdp_redirect_map_1
111111 ip link set dev veth3 xdp pinned $BPF_DIR/progs/xdp_redirect_map_2
112112
113−ip -n ${NS1} link set dev veth11 xdp obj xdp_dummy.o sec xdp
114−ip -n ${NS2} link set dev veth22 xdp obj xdp_tx.o sec xdp
115−ip -n ${NS3} link set dev veth33 xdp obj xdp_dummy.o sec xdp
113+ip -n ${NS1} link set dev veth11 xdp obj xdp_dummy.bpf.o sec xdp
114+ip -n ${NS2} link set dev veth22 xdp obj xdp_tx.bpf.o sec xdp
115+ip -n ${NS3} link set dev veth33 xdp obj xdp_dummy.bpf.o sec xdp
116116
117117 trap cleanup EXIT
118118
tools/testing/selftests/bpf/test_xsk.sh+36−16
@@ -73,14 +73,20 @@ · 72 unmodified lines
7373 #
7474 # Run and dump packet contents:
7575 # sudo ./test_xsk.sh -D
76+#
77+# Run test suite for physical device in loopback mode
78+# sudo ./test_xsk.sh -i IFACE
7679
7780 . xsk_prereqs.sh
7881
79−while getopts "vD" flag
82+ETH=""
83+
84+while getopts "vDi:" flag
8085 do
8186 case "${flag}" in
8287 v) verbose=1;;
8388 D) dump_pkts=1;;
89+ i) ETH=${OPTARG};;
8490 esac
8591 done
8692
@@ -132,18 +138,25 @@ setup_vethPairs() { · 45 unmodified lines
132138 ip link set ${VETH0} up
133139 }
134140
135−validate_root_exec
136−validate_veth_support ${VETH0}
137−validate_ip_utility
138−setup_vethPairs
139−
140−retval=$?
141−if [ $retval -ne 0 ]; then
142− test_status $retval "${TEST_NAME}"
143− cleanup_exit ${VETH0} ${VETH1} ${NS1}
144− exit $retval
141+if [ ! -z $ETH ]; then
142+ VETH0=${ETH}
143+ VETH1=${ETH}
144+ NS1=""
145+else
146+ validate_root_exec
147+ validate_veth_support ${VETH0}
148+ validate_ip_utility
149+ setup_vethPairs
150+
151+ retval=$?
152+ if [ $retval -ne 0 ]; then
153+ test_status $retval "${TEST_NAME}"
154+ cleanup_exit ${VETH0} ${VETH1} ${NS1}
155+ exit $retval
156+ fi
145157 fi
146158
159+
147160 if [[ $verbose -eq 1 ]]; then
148161 ARGS+="-v "
149162 fi
@@ -152,26 +165,33 @@ if [[ $dump_pkts -eq 1 ]]; then · 2 unmodified lines
152165 ARGS="-D "
153166 fi
154167
168+retval=$?
155169 test_status $retval "${TEST_NAME}"
156170
157171 ## START TESTS
158172
159173 statusList=()
160174
161−TEST_NAME="XSK_SELFTESTS_SOFTIRQ"
175+TEST_NAME="XSK_SELFTESTS_${VETH0}_SOFTIRQ"
162176
163177 exec_xskxceiver
164178
165−cleanup_exit ${VETH0} ${VETH1} ${NS1}
166−TEST_NAME="XSK_SELFTESTS_BUSY_POLL"
179+if [ -z $ETH ]; then
180+ cleanup_exit ${VETH0} ${VETH1} ${NS1}
181+fi
182+TEST_NAME="XSK_SELFTESTS_${VETH0}_BUSY_POLL"
167183 busy_poll=1
168184
169−setup_vethPairs
185+if [ -z $ETH ]; then
186+ setup_vethPairs
187+fi
170188 exec_xskxceiver
171189
172190 ## END TESTS
173191
174−cleanup_exit ${VETH0} ${VETH1} ${NS1}
192+if [ -z $ETH ]; then
193+ cleanup_exit ${VETH0} ${VETH1} ${NS1}
194+fi
175195
176196 failures=0
177197 echo -e "\nSummary:"
tools/testing/selftests/bpf/xdp_redirect_multi.c+1−1
@@ -142,7 +142,7 @@ int main(int argc, char **argv) · 141 unmodified lines
142142 }
143143 printf("\n");
144144
145− snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
145+ snprintf(filename, sizeof(filename), "%s_kern.bpf.o", argv[0]);
146146 obj = bpf_object__open_file(filename, NULL);
147147 err = libbpf_get_error(obj);
148148 if (err)
tools/testing/selftests/bpf/xdp_synproxy.c+1−1
@@ -193,7 +193,7 @@ static int syncookie_attach(const char *argv0, unsigned int ifindex, bool tc) · 192 unmodified lines
193193 int prog_fd;
194194 int err;
195195
196− snprintf(xdp_filename, sizeof(xdp_filename), "%s_kern.o", argv0);
196+ snprintf(xdp_filename, sizeof(xdp_filename), "%s_kern.bpf.o", argv0);
197197 obj = bpf_object__open_file(xdp_filename, NULL);
198198 err = libbpf_get_error(obj);
199199 if (err < 0) {
tools/testing/selftests/bpf/xdping.c+1−1
@@ -168,7 +168,7 @@ int main(int argc, char **argv) · 167 unmodified lines
168168 /* Use libbpf 1.0 API mode */
169169 libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
170170
171− snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
171+ snprintf(filename, sizeof(filename), "%s_kern.bpf.o", argv[0]);
172172
173173 if (bpf_prog_test_load(filename, BPF_PROG_TYPE_XDP, &obj, &prog_fd)) {
174174 fprintf(stderr, "load of %s failed\n", filename);
tools/testing/selftests/bpf/xsk.c+3−3
@@ -1237,15 +1237,15 @@ void xsk_socket__delete(struct xsk_socket *xsk) · 1236 unmodified lines
12371237 ctx = xsk->ctx;
12381238 umem = ctx->umem;
12391239
1240− xsk_put_ctx(ctx, true);
1241−
1242− if (!ctx->refcount) {
1240+ if (ctx->refcount == 1) {
12431241 xsk_delete_bpf_maps(xsk);
12441242 close(ctx->prog_fd);
12451243 if (ctx->has_bpf_link)
12461244 close(ctx->link_fd);
12471245 }
12481246
1247+ xsk_put_ctx(ctx, true);
1248+
12491249 err = xsk_get_mmap_offsets(xsk->fd, &off);
12501250 if (!err) {
12511251 if (xsk->rx) {
tools/testing/selftests/bpf/xskxceiver.c+296−102
@@ -99,6 +99,8 @@ · 98 unmodified lines
9999 #include <stdatomic.h>
100100 #include "xsk.h"
101101 #include "xskxceiver.h"
102+#include <bpf/bpf.h>
103+#include <linux/filter.h>
102104 #include "../kselftest.h"
103105
104106 /* AF_XDP APIs were moved into libxdp and marked as deprecated in libbpf.
@@ -122,9 +124,20 @@ static void __exit_with_error(int error, const char *file, const char *func, int · 17 unmodified lines
122124 }
123125
124126 #define exit_with_error(error) __exit_with_error(error, __FILE__, __func__, __LINE__)
125−
126−#define mode_string(test) (test)->ifobj_tx->xdp_flags & XDP_FLAGS_SKB_MODE ? "SKB" : "DRV"
127127 #define busy_poll_string(test) (test)->ifobj_tx->busy_poll ? "BUSY-POLL " : ""
128+static char *mode_string(struct test_spec *test)
129+{
130+ switch (test->mode) {
131+ case TEST_MODE_SKB:
132+ return "SKB";
133+ case TEST_MODE_DRV:
134+ return "DRV";
135+ case TEST_MODE_ZC:
136+ return "ZC";
137+ default:
138+ return "BOGUS";
139+ }
140+}
128141
129142 static void report_failure(struct test_spec *test)
130143 {
@@ -299,8 +312,8 @@ static void enable_busy_poll(struct xsk_socket_info *xsk) · 168 unmodified lines
299312 exit_with_error(errno);
300313 }
301314
302−static int xsk_configure_socket(struct xsk_socket_info *xsk, struct xsk_umem_info *umem,
303− struct ifobject *ifobject, bool shared)
315+static int __xsk_configure_socket(struct xsk_socket_info *xsk, struct xsk_umem_info *umem,
316+ struct ifobject *ifobject, bool shared)
304317 {
305318 struct xsk_socket_config cfg = {};
306319 struct xsk_ring_cons *rxr;
@@ -320,6 +333,51 @@ static int xsk_configure_socket(struct xsk_socket_info *xsk, struct xsk_umem_inf · 13 unmodified lines
320333 return xsk_socket__create(&xsk->xsk, ifobject->ifname, 0, umem->umem, rxr, txr, &cfg);
321334 }
322335
336+static bool ifobj_zc_avail(struct ifobject *ifobject)
337+{
338+ size_t umem_sz = DEFAULT_UMEM_BUFFERS * XSK_UMEM__DEFAULT_FRAME_SIZE;
339+ int mmap_flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE;
340+ struct xsk_socket_info *xsk;
341+ struct xsk_umem_info *umem;
342+ bool zc_avail = false;
343+ void *bufs;
344+ int ret;
345+
346+ bufs = mmap(NULL, umem_sz, PROT_READ | PROT_WRITE, mmap_flags, -1, 0);
347+ if (bufs == MAP_FAILED)
348+ exit_with_error(errno);
349+
350+ umem = calloc(1, sizeof(struct xsk_umem_info));
351+ if (!umem) {
352+ munmap(bufs, umem_sz);
353+ exit_with_error(-ENOMEM);
354+ }
355+ umem->frame_size = XSK_UMEM__DEFAULT_FRAME_SIZE;
356+ ret = xsk_configure_umem(umem, bufs, umem_sz);
357+ if (ret)
358+ exit_with_error(-ret);
359+
360+ xsk = calloc(1, sizeof(struct xsk_socket_info));
361+ if (!xsk)
362+ goto out;
363+ ifobject->xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST;
364+ ifobject->xdp_flags |= XDP_FLAGS_DRV_MODE;
365+ ifobject->bind_flags = XDP_USE_NEED_WAKEUP | XDP_ZEROCOPY;
366+ ifobject->rx_on = true;
367+ xsk->rxqsize = XSK_RING_CONS__DEFAULT_NUM_DESCS;
368+ ret = __xsk_configure_socket(xsk, umem, ifobject, false);
369+ if (!ret)
370+ zc_avail = true;
371+
372+ xsk_socket__delete(xsk->xsk);
373+ free(xsk);
374+out:
375+ munmap(umem->buffer, umem_sz);
376+ xsk_umem__delete(umem->umem);
377+ free(umem);
378+ return zc_avail;
379+}
380+
323381 static struct option long_options[] = {
324382 {"interface", required_argument, 0, 'i'},
325383 {"busy-poll", no_argument, 0, 'b'},
@@ -431,20 +489,24 @@ static void __test_spec_init(struct test_spec *test, struct ifobject *ifobj_tx, · 105 unmodified lines
431489 ifobj->use_poll = false;
432490 ifobj->use_fill_ring = true;
433491 ifobj->release_rx = true;
434− ifobj->pkt_stream = test->pkt_stream_default;
435492 ifobj->validation_func = NULL;
436493
437494 if (i == 0) {
438495 ifobj->rx_on = false;
439496 ifobj->tx_on = true;
497+ ifobj->pkt_stream = test->tx_pkt_stream_default;
440498 } else {
441499 ifobj->rx_on = true;
442500 ifobj->tx_on = false;
501+ ifobj->pkt_stream = test->rx_pkt_stream_default;
443502 }
444503
445504 memset(ifobj->umem, 0, sizeof(*ifobj->umem));
446505 ifobj->umem->num_frames = DEFAULT_UMEM_BUFFERS;
447506 ifobj->umem->frame_size = XSK_UMEM__DEFAULT_FRAME_SIZE;
507+ if (ifobj->shared_umem && ifobj->rx_on)
508+ ifobj->umem->base_addr = DEFAULT_UMEM_BUFFERS *
509+ XSK_UMEM__DEFAULT_FRAME_SIZE;
448510
449511 for (j = 0; j < MAX_SOCKETS; j++) {
450512 memset(&ifobj->xsk_arr[j], 0, sizeof(ifobj->xsk_arr[j]));
@@ -463,12 +525,15 @@ static void __test_spec_init(struct test_spec *test, struct ifobject *ifobj_tx, · 12 unmodified lines
463525 static void test_spec_init(struct test_spec *test, struct ifobject *ifobj_tx,
464526 struct ifobject *ifobj_rx, enum test_mode mode)
465527 {
466− struct pkt_stream *pkt_stream;
528+ struct pkt_stream *tx_pkt_stream;
529+ struct pkt_stream *rx_pkt_stream;
467530 u32 i;
468531
469− pkt_stream = test->pkt_stream_default;
532+ tx_pkt_stream = test->tx_pkt_stream_default;
533+ rx_pkt_stream = test->rx_pkt_stream_default;
470534 memset(test, 0, sizeof(*test));
471− test->pkt_stream_default = pkt_stream;
535+ test->tx_pkt_stream_default = tx_pkt_stream;
536+ test->rx_pkt_stream_default = rx_pkt_stream;
472537
473538 for (i = 0; i < MAX_INTERFACES; i++) {
474539 struct ifobject *ifobj = i ? ifobj_rx : ifobj_tx;
@@ -479,9 +544,14 @@ static void test_spec_init(struct test_spec *test, struct ifobject *ifobj_tx, · 4 unmodified lines
479544 else
480545 ifobj->xdp_flags |= XDP_FLAGS_DRV_MODE;
481546
482− ifobj->bind_flags = XDP_USE_NEED_WAKEUP | XDP_COPY;
547+ ifobj->bind_flags = XDP_USE_NEED_WAKEUP;
548+ if (mode == TEST_MODE_ZC)
549+ ifobj->bind_flags |= XDP_ZEROCOPY;
550+ else
551+ ifobj->bind_flags |= XDP_COPY;
483552 }
484553
554+ test->mode = mode;
485555 __test_spec_init(test, ifobj_tx, ifobj_rx);
486556 }
487557
@@ -529,16 +599,17 @@ static void pkt_stream_delete(struct pkt_stream *pkt_stream) · 41 unmodified lines
529599 static void pkt_stream_restore_default(struct test_spec *test)
530600 {
531601 struct pkt_stream *tx_pkt_stream = test->ifobj_tx->pkt_stream;
602+ struct pkt_stream *rx_pkt_stream = test->ifobj_rx->pkt_stream;
532603
533− if (tx_pkt_stream != test->pkt_stream_default) {
604+ if (tx_pkt_stream != test->tx_pkt_stream_default) {
534605 pkt_stream_delete(test->ifobj_tx->pkt_stream);
535− test->ifobj_tx->pkt_stream = test->pkt_stream_default;
606+ test->ifobj_tx->pkt_stream = test->tx_pkt_stream_default;
536607 }
537608
538− if (test->ifobj_rx->pkt_stream != test->pkt_stream_default &&
539− test->ifobj_rx->pkt_stream != tx_pkt_stream)
609+ if (rx_pkt_stream != test->rx_pkt_stream_default) {
540610 pkt_stream_delete(test->ifobj_rx->pkt_stream);
541− test->ifobj_rx->pkt_stream = test->pkt_stream_default;
611+ test->ifobj_rx->pkt_stream = test->rx_pkt_stream_default;
612+ }
542613 }
543614
544615 static struct pkt_stream *__pkt_stream_alloc(u32 nb_pkts)
@@ -561,7 +632,7 @@ static struct pkt_stream *__pkt_stream_alloc(u32 nb_pkts) · 16 unmodified lines
561632
562633 static void pkt_set(struct xsk_umem_info *umem, struct pkt *pkt, u64 addr, u32 len)
563634 {
564− pkt->addr = addr;
635+ pkt->addr = addr + umem->base_addr;
565636 pkt->len = len;
566637 if (len > umem->frame_size - XDP_PACKET_HEADROOM - MIN_PKT_SIZE * 2 - umem->frame_headroom)
567638 pkt->valid = false;
@@ -600,22 +671,29 @@ static void pkt_stream_replace(struct test_spec *test, u32 nb_pkts, u32 pkt_len) · 32 unmodified lines
600671
601672 pkt_stream = pkt_stream_generate(test->ifobj_tx->umem, nb_pkts, pkt_len);
602673 test->ifobj_tx->pkt_stream = pkt_stream;
674+ pkt_stream = pkt_stream_generate(test->ifobj_rx->umem, nb_pkts, pkt_len);
603675 test->ifobj_rx->pkt_stream = pkt_stream;
604676 }
605677
606−static void pkt_stream_replace_half(struct test_spec *test, u32 pkt_len, int offset)
678+static void __pkt_stream_replace_half(struct ifobject *ifobj, u32 pkt_len,
679+ int offset)
607680 {
608− struct xsk_umem_info *umem = test->ifobj_tx->umem;
681+ struct xsk_umem_info *umem = ifobj->umem;
609682 struct pkt_stream *pkt_stream;
610683 u32 i;
611684
612− pkt_stream = pkt_stream_clone(umem, test->pkt_stream_default);
613− for (i = 1; i < test->pkt_stream_default->nb_pkts; i += 2)
685+ pkt_stream = pkt_stream_clone(umem, ifobj->pkt_stream);
686+ for (i = 1; i < ifobj->pkt_stream->nb_pkts; i += 2)
614687 pkt_set(umem, &pkt_stream->pkts[i],
615688 (i % umem->num_frames) * umem->frame_size + offset, pkt_len);
616689
617− test->ifobj_tx->pkt_stream = pkt_stream;
618− test->ifobj_rx->pkt_stream = pkt_stream;
690+ ifobj->pkt_stream = pkt_stream;
691+}
692+
693+static void pkt_stream_replace_half(struct test_spec *test, u32 pkt_len, int offset)
694+{
695+ __pkt_stream_replace_half(test->ifobj_tx, pkt_len, offset);
696+ __pkt_stream_replace_half(test->ifobj_rx, pkt_len, offset);
619697 }
620698
621699 static void pkt_stream_receive_half(struct test_spec *test)
@@ -657,7 +735,8 @@ static struct pkt *pkt_generate(struct ifobject *ifobject, u32 pkt_nb) · 35 unmodified lines
657735 return pkt;
658736 }
659737
660−static void pkt_stream_generate_custom(struct test_spec *test, struct pkt *pkts, u32 nb_pkts)
738+static void __pkt_stream_generate_custom(struct ifobject *ifobj,
739+ struct pkt *pkts, u32 nb_pkts)
661740 {
662741 struct pkt_stream *pkt_stream;
663742 u32 i;
@@ -666,15 +745,20 @@ static void pkt_stream_generate_custom(struct test_spec *test, struct pkt *pkts, · 2 unmodified lines
666745 if (!pkt_stream)
667746 exit_with_error(ENOMEM);
668747
669− test->ifobj_tx->pkt_stream = pkt_stream;
670− test->ifobj_rx->pkt_stream = pkt_stream;
671−
672748 for (i = 0; i < nb_pkts; i++) {
673− pkt_stream->pkts[i].addr = pkts[i].addr;
749+ pkt_stream->pkts[i].addr = pkts[i].addr + ifobj->umem->base_addr;
674750 pkt_stream->pkts[i].len = pkts[i].len;
675751 pkt_stream->pkts[i].payload = i;
676752 pkt_stream->pkts[i].valid = pkts[i].valid;
677753 }
754+
755+ ifobj->pkt_stream = pkt_stream;
756+}
757+
758+static void pkt_stream_generate_custom(struct test_spec *test, struct pkt *pkts, u32 nb_pkts)
759+{
760+ __pkt_stream_generate_custom(test->ifobj_tx, pkts, nb_pkts);
761+ __pkt_stream_generate_custom(test->ifobj_rx, pkts, nb_pkts);
678762 }
679763
680764 static void pkt_dump(void *pkt, u32 len)
@@ -1126,6 +1210,70 @@ static int validate_tx_invalid_descs(struct ifobject *ifobject) · 445 unmodified lines
11261210 return TEST_PASS;
11271211 }
11281212
1213+static void xsk_configure_socket(struct test_spec *test, struct ifobject *ifobject,
1214+ struct xsk_umem_info *umem, bool tx)
1215+{
1216+ int i, ret;
1217+
1218+ for (i = 0; i < test->nb_sockets; i++) {
1219+ bool shared = (ifobject->shared_umem && tx) ? true : !!i;
1220+ u32 ctr = 0;
1221+
1222+ while (ctr++ < SOCK_RECONF_CTR) {
1223+ ret = __xsk_configure_socket(&ifobject->xsk_arr[i], umem,
1224+ ifobject, shared);
1225+ if (!ret)
1226+ break;
1227+
1228+ /* Retry if it fails as xsk_socket__create() is asynchronous */
1229+ if (ctr >= SOCK_RECONF_CTR)
1230+ exit_with_error(-ret);
1231+ usleep(USLEEP_MAX);
1232+ }
1233+ if (ifobject->busy_poll)
1234+ enable_busy_poll(&ifobject->xsk_arr[i]);
1235+ }
1236+}
1237+
1238+static void thread_common_ops_tx(struct test_spec *test, struct ifobject *ifobject)
1239+{
1240+ xsk_configure_socket(test, ifobject, test->ifobj_rx->umem, true);
1241+ ifobject->xsk = &ifobject->xsk_arr[0];
1242+ ifobject->xsk_map_fd = test->ifobj_rx->xsk_map_fd;
1243+ memcpy(ifobject->umem, test->ifobj_rx->umem, sizeof(struct xsk_umem_info));
1244+}
1245+
1246+static void xsk_populate_fill_ring(struct xsk_umem_info *umem, struct pkt_stream *pkt_stream)
1247+{
1248+ u32 idx = 0, i, buffers_to_fill;
1249+ int ret;
1250+
1251+ if (umem->num_frames < XSK_RING_PROD__DEFAULT_NUM_DESCS)
1252+ buffers_to_fill = umem->num_frames;
1253+ else
1254+ buffers_to_fill = XSK_RING_PROD__DEFAULT_NUM_DESCS;
1255+
1256+ ret = xsk_ring_prod__reserve(&umem->fq, buffers_to_fill, &idx);
1257+ if (ret != buffers_to_fill)
1258+ exit_with_error(ENOSPC);
1259+ for (i = 0; i < buffers_to_fill; i++) {
1260+ u64 addr;
1261+
1262+ if (pkt_stream->use_addr_for_fill) {
1263+ struct pkt *pkt = pkt_stream_get_pkt(pkt_stream, i);
1264+
1265+ if (!pkt)
1266+ break;
1267+ addr = pkt->addr;
1268+ } else {
1269+ addr = i * umem->frame_size;
1270+ }
1271+
1272+ *xsk_ring_prod__fill_addr(&umem->fq, idx++) = addr;
1273+ }
1274+ xsk_ring_prod__submit(&umem->fq, buffers_to_fill);
1275+}
1276+
11291277 static void thread_common_ops(struct test_spec *test, struct ifobject *ifobject)
11301278 {
11311279 u64 umem_sz = ifobject->umem->num_frames * ifobject->umem->frame_size;
@@ -1133,13 +1281,15 @@ static void thread_common_ops(struct test_spec *test, struct ifobject *ifobject) · 1 unmodified line
11331281 LIBBPF_OPTS(bpf_xdp_query_opts, opts);
11341282 int ret, ifindex;
11351283 void *bufs;
1136− u32 i;
11371284
11381285 ifobject->ns_fd = switch_namespace(ifobject->nsname);
11391286
11401287 if (ifobject->umem->unaligned_mode)
11411288 mmap_flags |= MAP_HUGETLB;
11421289
1290+ if (ifobject->shared_umem)
1291+ umem_sz *= 2;
1292+
11431293 bufs = mmap(NULL, umem_sz, PROT_READ | PROT_WRITE, mmap_flags, -1, 0);
11441294 if (bufs == MAP_FAILED)
11451295 exit_with_error(errno);
@@ -1148,24 +1298,9 @@ static void thread_common_ops(struct test_spec *test, struct ifobject *ifobject) · 2 unmodified lines
11481298 if (ret)
11491299 exit_with_error(-ret);
11501300
1151− for (i = 0; i < test->nb_sockets; i++) {
1152− u32 ctr = 0;
1153−
1154− while (ctr++ < SOCK_RECONF_CTR) {
1155− ret = xsk_configure_socket(&ifobject->xsk_arr[i], ifobject->umem,
1156− ifobject, !!i);
1157− if (!ret)
1158− break;
1159−
1160− /* Retry if it fails as xsk_socket__create() is asynchronous */
1161− if (ctr >= SOCK_RECONF_CTR)
1162− exit_with_error(-ret);
1163− usleep(USLEEP_MAX);
1164− }
1301+ xsk_populate_fill_ring(ifobject->umem, ifobject->pkt_stream);
11651302
1166− if (ifobject->busy_poll)
1167− enable_busy_poll(&ifobject->xsk_arr[i]);
1168− }
1303+ xsk_configure_socket(test, ifobject, ifobject->umem, false);
11691304
11701305 ifobject->xsk = &ifobject->xsk_arr[0];
11711306
@@ -1201,22 +1336,18 @@ static void thread_common_ops(struct test_spec *test, struct ifobject *ifobject) · 29 unmodified lines
12011336 exit_with_error(-ret);
12021337 }
12031338
1204−static void testapp_cleanup_xsk_res(struct ifobject *ifobj)
1205−{
1206− print_verbose("Destroying socket\n");
1207− xsk_socket__delete(ifobj->xsk->xsk);
1208− munmap(ifobj->umem->buffer, ifobj->umem->num_frames * ifobj->umem->frame_size);
1209− xsk_umem__delete(ifobj->umem->umem);
1210−}
1211−
12121339 static void *worker_testapp_validate_tx(void *arg)
12131340 {
12141341 struct test_spec *test = (struct test_spec *)arg;
12151342 struct ifobject *ifobject = test->ifobj_tx;
12161343 int err;
12171344
1218− if (test->current_step == 1)
1219− thread_common_ops(test, ifobject);
1345+ if (test->current_step == 1) {
1346+ if (!ifobject->shared_umem)
1347+ thread_common_ops(test, ifobject);
1348+ else
1349+ thread_common_ops_tx(test, ifobject);
1350+ }
12201351
12211352 print_verbose("Sending %d packets on interface %s\n", ifobject->pkt_stream->nb_pkts,
12221353 ifobject->ifname);
@@ -1227,53 +1358,23 @@ static void *worker_testapp_validate_tx(void *arg) · 4 unmodified lines
12271358 if (err)
12281359 report_failure(test);
12291360
1230− if (test->total_steps == test->current_step || err)
1231− testapp_cleanup_xsk_res(ifobject);
12321361 pthread_exit(NULL);
12331362 }
12341363
1235−static void xsk_populate_fill_ring(struct xsk_umem_info *umem, struct pkt_stream *pkt_stream)
1236−{
1237− u32 idx = 0, i, buffers_to_fill;
1238− int ret;
1239−
1240− if (umem->num_frames < XSK_RING_PROD__DEFAULT_NUM_DESCS)
1241− buffers_to_fill = umem->num_frames;
1242− else
1243− buffers_to_fill = XSK_RING_PROD__DEFAULT_NUM_DESCS;
1244−
1245− ret = xsk_ring_prod__reserve(&umem->fq, buffers_to_fill, &idx);
1246− if (ret != buffers_to_fill)
1247− exit_with_error(ENOSPC);
1248− for (i = 0; i < buffers_to_fill; i++) {
1249− u64 addr;
1250−
1251− if (pkt_stream->use_addr_for_fill) {
1252− struct pkt *pkt = pkt_stream_get_pkt(pkt_stream, i);
1253−
1254− if (!pkt)
1255− break;
1256− addr = pkt->addr;
1257− } else {
1258− addr = i * umem->frame_size;
1259− }
1260−
1261− *xsk_ring_prod__fill_addr(&umem->fq, idx++) = addr;
1262− }
1263− xsk_ring_prod__submit(&umem->fq, buffers_to_fill);
1264−}
1265−
12661364 static void *worker_testapp_validate_rx(void *arg)
12671365 {
12681366 struct test_spec *test = (struct test_spec *)arg;
12691367 struct ifobject *ifobject = test->ifobj_rx;
12701368 struct pollfd fds = { };
1369+ int id = 0;
12711370 int err;
12721371
1273− if (test->current_step == 1)
1372+ if (test->current_step == 1) {
12741373 thread_common_ops(test, ifobject);
1275−
1276− xsk_populate_fill_ring(ifobject->umem, ifobject->pkt_stream);
1374+ } else {
1375+ bpf_map_delete_elem(ifobject->xsk_map_fd, &id);
1376+ xsk_socket__update_xskmap(ifobject->xsk->xsk, ifobject->xsk_map_fd);
1377+ }
12771378
12781379 fds.fd = xsk_socket__fd(ifobject->xsk->xsk);
12791380 fds.events = POLLIN;
@@ -1291,25 +1392,44 @@ static void *worker_testapp_validate_rx(void *arg) · 11 unmodified lines
12911392 pthread_mutex_unlock(&pacing_mutex);
12921393 }
12931394
1294− if (test->total_steps == test->current_step || err)
1295− testapp_cleanup_xsk_res(ifobject);
1395+ pthread_exit(NULL);
1396+}
1397+
1398+static void testapp_clean_xsk_umem(struct ifobject *ifobj)
1399+{
1400+ u64 umem_sz = ifobj->umem->num_frames * ifobj->umem->frame_size;
1401+
1402+ if (ifobj->shared_umem)
1403+ umem_sz *= 2;
1404+
1405+ xsk_umem__delete(ifobj->umem->umem);
1406+ munmap(ifobj->umem->buffer, umem_sz);
1407+}
1408+
1409+static void handler(int signum)
1410+{
12961411 pthread_exit(NULL);
12971412 }
12981413
12991414 static int testapp_validate_traffic_single_thread(struct test_spec *test, struct ifobject *ifobj,
13001415 enum test_type type)
13011416 {
1417+ bool old_shared_umem = ifobj->shared_umem;
13021418 pthread_t t0;
13031419
13041420 if (pthread_barrier_init(&barr, NULL, 2))
13051421 exit_with_error(errno);
13061422
13071423 test->current_step++;
1308− if (type == TEST_TYPE_POLL_RXQ_TMOUT)
1424+ if (type == TEST_TYPE_POLL_RXQ_TMOUT)
13091425 pkt_stream_reset(ifobj->pkt_stream);
13101426 pkts_in_flight = 0;
13111427
1312− /*Spawn thread */
1428+ test->ifobj_rx->shared_umem = false;
1429+ test->ifobj_tx->shared_umem = false;
1430+
1431+ signal(SIGUSR1, handler);
1432+ /* Spawn thread */
13131433 pthread_create(&t0, NULL, ifobj->func_ptr, test);
13141434
13151435 if (type != TEST_TYPE_POLL_TXQ_TMOUT)
@@ -1318,8 +1438,17 @@ static int testapp_validate_traffic_single_thread(struct test_spec *test, struct · 2 unmodified lines
13181438 if (pthread_barrier_destroy(&barr))
13191439 exit_with_error(errno);
13201440
1441+ pthread_kill(t0, SIGUSR1);
13211442 pthread_join(t0, NULL);
13221443
1444+ if (test->total_steps == test->current_step || test->fail) {
1445+ xsk_socket__delete(ifobj->xsk->xsk);
1446+ testapp_clean_xsk_umem(ifobj);
1447+ }
1448+
1449+ test->ifobj_rx->shared_umem = old_shared_umem;
1450+ test->ifobj_tx->shared_umem = old_shared_umem;
1451+
13231452 return !!test->fail;
13241453 }
13251454
@@ -1349,6 +1478,14 @@ static int testapp_validate_traffic(struct test_spec *test) · 23 unmodified lines
13491478 pthread_join(t1, NULL);
13501479 pthread_join(t0, NULL);
13511480
1481+ if (test->total_steps == test->current_step || test->fail) {
1482+ xsk_socket__delete(ifobj_tx->xsk->xsk);
1483+ xsk_socket__delete(ifobj_rx->xsk->xsk);
1484+ testapp_clean_xsk_umem(ifobj_rx);
1485+ if (!ifobj_tx->shared_umem)
1486+ testapp_clean_xsk_umem(ifobj_tx);
1487+ }
1488+
13521489 return !!test->fail;
13531490 }
13541491
@@ -1428,9 +1565,9 @@ static void testapp_headroom(struct test_spec *test) · 73 unmodified lines
14281565 static void testapp_stats_rx_dropped(struct test_spec *test)
14291566 {
14301567 test_spec_set_name(test, "STAT_RX_DROPPED");
1568+ pkt_stream_replace_half(test, MIN_PKT_SIZE * 4, 0);
14311569 test->ifobj_rx->umem->frame_headroom = test->ifobj_rx->umem->frame_size -
14321570 XDP_PACKET_HEADROOM - MIN_PKT_SIZE * 3;
1433− pkt_stream_replace_half(test, MIN_PKT_SIZE * 4, 0);
14341571 pkt_stream_receive_half(test);
14351572 test->ifobj_rx->validation_func = validate_rx_dropped;
14361573 testapp_validate_traffic(test);
@@ -1553,6 +1690,11 @@ static void testapp_invalid_desc(struct test_spec *test) · 116 unmodified lines
15531690 pkts[7].valid = false;
15541691 }
15551692
1693+ if (test->ifobj_tx->shared_umem) {
1694+ pkts[4].addr += UMEM_SIZE;
1695+ pkts[5].addr += UMEM_SIZE;
1696+ }
1697+
15561698 pkt_stream_generate_custom(test, pkts, ARRAY_SIZE(pkts));
15571699 testapp_validate_traffic(test);
15581700 pkt_stream_restore_default(test);
@@ -1583,6 +1725,10 @@ static void run_pkt_test(struct test_spec *test, enum test_mode mode, enum test_ · 24 unmodified lines
15831725 {
15841726 switch (type) {
15851727 case TEST_TYPE_STATS_RX_DROPPED:
1728+ if (mode == TEST_MODE_ZC) {
1729+ ksft_test_result_skip("Can not run RX_DROPPED test for ZC mode\n");
1730+ return;
1731+ }
15861732 testapp_stats_rx_dropped(test);
15871733 break;
15881734 case TEST_TYPE_STATS_TX_INVALID_DESCS:
@@ -1712,12 +1858,44 @@ static void ifobject_delete(struct ifobject *ifobj) · 123 unmodified lines
17121858 free(ifobj);
17131859 }
17141860
1861+static bool is_xdp_supported(struct ifobject *ifobject)
1862+{
1863+ int flags = XDP_FLAGS_DRV_MODE;
1864+
1865+ LIBBPF_OPTS(bpf_link_create_opts, opts, .flags = flags);
1866+ struct bpf_insn insns[2] = {
1867+ BPF_MOV64_IMM(BPF_REG_0, XDP_PASS),
1868+ BPF_EXIT_INSN()
1869+ };
1870+ int ifindex = if_nametoindex(ifobject->ifname);
1871+ int prog_fd, insn_cnt = ARRAY_SIZE(insns);
1872+ int err;
1873+
1874+ prog_fd = bpf_prog_load(BPF_PROG_TYPE_XDP, NULL, "GPL", insns, insn_cnt, NULL);
1875+ if (prog_fd < 0)
1876+ return false;
1877+
1878+ err = bpf_xdp_attach(ifindex, prog_fd, flags, NULL);
1879+ if (err) {
1880+ close(prog_fd);
1881+ return false;
1882+ }
1883+
1884+ bpf_xdp_detach(ifindex, flags, NULL);
1885+ close(prog_fd);
1886+
1887+ return true;
1888+}
1889+
17151890 int main(int argc, char **argv)
17161891 {
1717− struct pkt_stream *pkt_stream_default;
1892+ struct pkt_stream *rx_pkt_stream_default;
1893+ struct pkt_stream *tx_pkt_stream_default;
17181894 struct ifobject *ifobj_tx, *ifobj_rx;
1895+ int modes = TEST_MODE_SKB + 1;
17191896 u32 i, j, failed_tests = 0;
17201897 struct test_spec test;
1898+ bool shared_umem;
17211899
17221900 /* Use libbpf 1.0 API mode */
17231901 libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
@@ -1732,6 +1910,10 @@ int main(int argc, char **argv) · 8 unmodified lines
17321910 setlocale(LC_ALL, "");
17331911
17341912 parse_command_line(ifobj_tx, ifobj_rx, argc, argv);
1913+ shared_umem = !strcmp(ifobj_tx->ifname, ifobj_rx->ifname);
1914+
1915+ ifobj_tx->shared_umem = shared_umem;
1916+ ifobj_rx->shared_umem = shared_umem;
17351917
17361918 if (!validate_interface(ifobj_tx) || !validate_interface(ifobj_rx)) {
17371919 usage(basename(argv[0]));
@@ -1743,15 +1925,23 @@ int main(int argc, char **argv) · 5 unmodified lines
17431925 init_iface(ifobj_rx, MAC2, MAC1, IP2, IP1, UDP_PORT2, UDP_PORT1,
17441926 worker_testapp_validate_rx);
17451927
1928+ if (is_xdp_supported(ifobj_tx)) {
1929+ modes++;
1930+ if (ifobj_zc_avail(ifobj_tx))
1931+ modes++;
1932+ }
1933+
17461934 test_spec_init(&test, ifobj_tx, ifobj_rx, 0);
1747− pkt_stream_default = pkt_stream_generate(ifobj_tx->umem, DEFAULT_PKT_CNT, PKT_SIZE);
1748− if (!pkt_stream_default)
1935+ tx_pkt_stream_default = pkt_stream_generate(ifobj_tx->umem, DEFAULT_PKT_CNT, PKT_SIZE);
1936+ rx_pkt_stream_default = pkt_stream_generate(ifobj_rx->umem, DEFAULT_PKT_CNT, PKT_SIZE);
1937+ if (!tx_pkt_stream_default || !rx_pkt_stream_default)
17491938 exit_with_error(ENOMEM);
1750− test.pkt_stream_default = pkt_stream_default;
1939+ test.tx_pkt_stream_default = tx_pkt_stream_default;
1940+ test.rx_pkt_stream_default = rx_pkt_stream_default;
17511941
1752− ksft_set_plan(TEST_MODE_MAX * TEST_TYPE_MAX);
1942+ ksft_set_plan(modes * TEST_TYPE_MAX);
17531943
1754− for (i = 0; i < TEST_MODE_MAX; i++)
1944+ for (i = 0; i < modes; i++)
17551945 for (j = 0; j < TEST_TYPE_MAX; j++) {
17561946 test_spec_init(&test, ifobj_tx, ifobj_rx, i);
17571947 run_pkt_test(&test, i, j);
@@ -1761,7 +1951,11 @@ int main(int argc, char **argv) · 3 unmodified lines
17611951 failed_tests++;
17621952 }
17631953
1764− pkt_stream_delete(pkt_stream_default);
1954+ pkt_stream_delete(tx_pkt_stream_default);
1955+ pkt_stream_delete(rx_pkt_stream_default);
1956+ free(ifobj_rx->umem);
1957+ if (!ifobj_tx->shared_umem)
1958+ free(ifobj_tx->umem);
17651959 ifobject_delete(ifobj_tx);
17661960 ifobject_delete(ifobj_rx);
17671961
tools/testing/selftests/bpf/xskxceiver.h+8−3
@@ -29,8 +29,8 @@ · 28 unmodified lines
2929 #define TEST_FAILURE -1
3030 #define TEST_CONTINUE 1
3131 #define MAX_INTERFACES 2
32−#define MAX_INTERFACE_NAME_CHARS 7
33−#define MAX_INTERFACES_NAMESPACE_CHARS 10
32+#define MAX_INTERFACE_NAME_CHARS 16
33+#define MAX_INTERFACES_NAMESPACE_CHARS 16
3434 #define MAX_SOCKETS 2
3535 #define MAX_TEST_NAME_SIZE 32
3636 #define MAX_TEARDOWN_ITER 10
@@ -62,6 +62,7 @@ · 25 unmodified lines
6262 enum test_mode {
6363 TEST_MODE_SKB,
6464 TEST_MODE_DRV,
65+ TEST_MODE_ZC,
6566 TEST_MODE_MAX
6667 };
6768
@@ -99,6 +100,7 @@ struct xsk_umem_info { · 31 unmodified lines
99100 u32 frame_headroom;
100101 void *buffer;
101102 u32 frame_size;
103+ u32 base_addr;
102104 bool unaligned_mode;
103105 };
104106
@@ -152,6 +154,7 @@ struct ifobject { · 47 unmodified lines
152154 bool busy_poll;
153155 bool use_fill_ring;
154156 bool release_rx;
157+ bool shared_umem;
155158 u8 dst_mac[ETH_ALEN];
156159 u8 src_mac[ETH_ALEN];
157160 };
@@ -159,11 +162,13 @@ struct ifobject { · 1 unmodified line
159162 struct test_spec {
160163 struct ifobject *ifobj_tx;
161164 struct ifobject *ifobj_rx;
162− struct pkt_stream *pkt_stream_default;
165+ struct pkt_stream *tx_pkt_stream_default;
166+ struct pkt_stream *rx_pkt_stream_default;
163167 u16 total_steps;
164168 u16 current_step;
165169 u16 nb_sockets;
166170 bool fail;
171+ enum test_mode mode;
167172 char name[MAX_TEST_NAME_SIZE];
168173 };
169174