History

Merge tag 'bpf-next-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Pull bpf updates from Alexei Starovoitov: "Major changes: - Recover from BPF arena page faults using a scratch page and add ptep_try_set() for lockless empty-slot installs on x86 and arm64. This allows BPF kfuncs to access arena pointers directly. The 'arena_direct_access' stable branch was created for this work and was pulled into sched-ext and bpf-next trees (Tejun Heo, Kumar Kartikeya Dwivedi) - Lift old restriction and support 6+ arguments in BPF programs and kfuncs on x86 and arm64 (Yonghong Song, Puranjay Mohan) Other features and fixes: - Add 24-bit BTF vlen and reclaim unused bits in the BTF UAPI to ease addition of new BTF kinds (Alan Maguire) - Raise the maximum BPF call chain depth from 8 to 16 frames (Alexei Starovoitov) - Refactor object relationship tracking in the verifier and fix a dynptr use-after-free bug (Amery Hung) - Harden the signed program loader and reject exclusive maps as inner maps (Daniel Borkmann) - Replace the verifier min/max bounds fields with a circular number (cnum) representation and improve 32->64 bit range refinements (Eduard Zingerman) - Introduce the arena library and runtime (libarena) with a buddy allocator, rbtree and SPMC queue data structures, ASAN support and a parallel test harness. Allow subprograms to return arena pointers and switch to a BTF type-tag based __arena annotation (Emil Tsalapatis) - Cache build IDs in the sleepable stackmap path and avoid faultable build ID reads under mm locks (Ihor Solodrai) - Introduce the tracing_multi link to attach a single BPF program to many kernel functions at once. Allow specifying the uprobe_multi target via FD (Jiri Olsa) - Extend the bpf_list family of kfuncs with bpf_list_add/del(), and bpf_list_is_first/is_last/empty() (Kaitao Cheng) - Extend the BPF syscall with common attributes support for prog_load, btf_load and map_create (Leon Hwang) - Wrap rhashtable as BPF map (Mykyta Yatsenko, Herbert Xu) - Add sleepable support for tracepoint programs and fix deadlocks in LRU map due to NMI reentry (Mykyta Yatsenko) - Fix OOB access in bpf_flow_keys, fix nullness analysis of inner arrays, enforce write checks for global subprograms (Nuoqi Gui) - Report the maximum combined stack depth and print a breakdown of instructions processed per subprogram (Paul Chaignon) - Add an XDP load-balancer benchmark and arm64 JIT support for stack arguments (Puranjay Mohan) - Add kfuncs to traverse over wakeup_sources (Samuel Wu) - Allow sleepable BPF programs to use LPM trie maps directly (Vlad Poenaru) - Many more fixes and cleanups across the verifier, BTF, sockmap, devmap, bpffs, security hooks, s390/riscv/loongarch JITs, rqspinlock, libbpf, bpftool, selftests" * tag 'bpf-next-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (336 commits) selftests/bpf: Work around llvm stack overflow in crypto progs selftests/bpf: add test for bpf_msg_pop_data() overflow bpf, sockmap: fix integer overflow in bpf_msg_pop_data() bounds check sockmap: Fix use-after-free in udp_bpf_recvmsg() bpf, sockmap: keep sk_msg copy state in sync bpf, sockmap: Fix wrong rsge offset in bpf_msg_push_data() bpf, sockmap: reject overflowing copy + len in bpf_msg_push_data() selftsets/bpf: Retry map update on helper_fill_hashmap() selftests/bpf: Add test for sleepable lsm_cgroup rejection selftests/bpf: Add test to verify the fix for bpf_setsockopt() helper bpf: Fix bpf_get/setsockopt to tos for ipv4-mapped ipv6 socket selftests/bpf: Avoid static LLVM linking for cross builds selftests/bpf: Use common CFLAGS for urandom_read selftests/bpf: Initialize operation name before use tools/bpf: build: Append extra cflags libbpf: Initialize CFLAGS before including Makefile.include bpftool: Append extra host flags bpftool: Avoid adding EXTRA_CFLAGS to HOST_CFLAGS bpftool: Pass host flags to bootstrap libbpf selftests/bpf: correct CONFIG_PPC64 macro name in comment ...

Linus Torvalds committed 5 months ago

Commit 9c87e612 parents b85966a e4287bf349 files · +27861 −4982Browse files

Unified · Split

349 files
  1. Documentation/bpf/btf.rst +2 −4
  2. Documentation/bpf/kfuncs.rst +34 −2
  3. Documentation/bpf/map_lru_hash_update.dot +42 −2
  4. MAINTAINERS +7 −27
  5. arch/arm64/include/asm/pgtable.h +28 −0
  6. arch/arm64/mm/fault.c +7 −3
  7. arch/arm64/net/bpf_jit_comp.c +120 −32
  8. arch/arm64/net/bpf_timed_may_goto.S +4 −4
  9. arch/loongarch/net/bpf_jit.c +26 −26
  10. arch/powerpc/net/bpf_jit_comp.c +27 −27
  11. arch/riscv/net/bpf_jit_comp64.c +35 −26
  12. arch/s390/net/bpf_jit_comp.c +59 −22
  13. arch/x86/include/asm/pgtable.h +12 −0
  14. arch/x86/mm/fault.c +8 −4
  15. arch/x86/net/bpf_jit_comp.c +172 −33
  16. drivers/base/power/power.h +7 −0
  17. drivers/base/power/wakeup.c +69 −2
  18. drivers/net/ethernet/netronome/nfp/bpf/verifier.c +4 −4
  19. fs/bpf_fs_kfuncs.c +1 −1
  20. fs/verity/measure.c +2 −2
  21. include/asm-generic/rqspinlock.h +11 −3
  22. include/linux/bpf-cgroup.h +3 −2
  23. include/linux/bpf.h +236 −46
  24. include/linux/bpf_defs.h +19 −0
  25. include/linux/bpf_lsm.h +6 −0
  26. include/linux/bpf_types.h +2 −0
  27. include/linux/bpf_verifier.h +209 −92
  28. include/linux/btf.h +4 −3
  29. include/linux/btf_ids.h +1 −0
  30. include/linux/cnum.h +82 −0
  31. include/linux/filter.h +25 −2
  32. include/linux/ftrace.h +9 −0
  33. include/linux/pgtable.h +43 −0
  34. include/linux/rhashtable.h +3 −1
  35. include/linux/syscalls.h +2 −1
  36. include/linux/trace_events.h +12 −0
  37. include/trace/bpf_probe.h +0 −2
  38. include/uapi/linux/bpf.h +31 −1
  39. include/uapi/linux/btf.h +14 −12
  40. kernel/bpf/Makefile +1 −1
  41. kernel/bpf/arena.c +266 −49
  42. kernel/bpf/arraymap.c +6 −8
  43. kernel/bpf/backtrack.c +66 −16
  44. kernel/bpf/bpf_lru_list.c +100 −65
  45. kernel/bpf/bpf_lru_list.h +19 −6
  46. kernel/bpf/bpf_lsm.c +20 −0
  47. kernel/bpf/bpf_struct_ops.c +50 −13
  48. kernel/bpf/btf.c +219 −99
  49. kernel/bpf/cgroup.c +44 −21
  50. kernel/bpf/cnum.c +120 −0
  51. kernel/bpf/cnum_defs.h +247 −0
  52. kernel/bpf/const_fold.c +8 −0
  53. kernel/bpf/core.c +24 −3
  54. kernel/bpf/devmap.c +19 −0
  55. kernel/bpf/fixups.c +24 −3
  56. kernel/bpf/hashtab.c +817 −23
  57. kernel/bpf/helpers.c +154 −50
  58. kernel/bpf/inode.c +4 −0
  59. kernel/bpf/liveness.c +139 −44
  60. kernel/bpf/log.c +104 −28
  61. kernel/bpf/lpm_trie.c +4 −4
  62. kernel/bpf/map_in_map.c +4 −1
  63. kernel/bpf/map_iter.c +6 −1
  64. kernel/bpf/stackmap.c +198 −17
  65. kernel/bpf/states.c +51 −16
  66. kernel/bpf/syscall.c +231 −81
  67. kernel/bpf/trampoline.c +524 −147
  68. kernel/bpf/verifier.c +1978 −2196
  69. kernel/events/core.c +9 −0
  70. kernel/trace/bpf_trace.c +306 −28
  71. kernel/trace/ftrace.c +29 −6
  72. kernel/trace/trace_syscalls.c +58 −52
  73. lib/rhashtable.c +102 −0
  74. lib/test_bpf.c +263 −100
  75. lib/test_rhashtable.c +75 −0
  76. net/bpf/bpf_dummy_struct_ops.c +7 −7
  77. net/bpf/test_run.c +53 −15
  78. net/core/filter.c +86 −31
  79. net/core/lwt_bpf.c +12 −0
  80. net/ipv4/udp_bpf.c +9 −0
  81. tools/bpf/Makefile +1 −0
  82. tools/bpf/bpftool/Documentation/bpftool-map.rst +1 −1
  83. tools/bpf/bpftool/Makefile +17 −3
  84. tools/bpf/bpftool/btf.c +6 −11
  85. tools/bpf/bpftool/btf_dumper.c +2 −2
  86. tools/bpf/bpftool/gen.c +10 −8
  87. tools/bpf/bpftool/map.c +1 −1
  88. tools/bpf/bpftool/net.c +2 −2
  89. tools/build/feature/test-bpf.c +2 −0
  90. tools/include/uapi/linux/bpf.h +32 −1
  91. tools/include/uapi/linux/btf.h +14 −12
  92. tools/lib/bpf/Makefile +9 −8
  93. tools/lib/bpf/bpf.c +66 −1
  94. tools/lib/bpf/bpf.h +23 −2
  95. tools/lib/bpf/btf.c +35 −19
  96. tools/lib/bpf/btf.h +1 −1
  97. tools/lib/bpf/btf_dump.c +12 −12
  98. tools/lib/bpf/features.c +8 −0
  99. tools/lib/bpf/gen_loader.c +67 −25
  100. tools/lib/bpf/libbpf.c +437 −36
  101. tools/lib/bpf/libbpf.h +15 −0
  102. tools/lib/bpf/libbpf.map +1 −0
  103. tools/lib/bpf/libbpf_internal.h +4 −0
  104. tools/lib/bpf/libbpf_probes.c +3 −0
  105. tools/lib/bpf/relo_core.c +8 −8
  106. tools/lib/bpf/skel_internal.h +5 −0
  107. tools/lib/bpf/strset.c +41 −21
  108. tools/lib/bpf/usdt.c +12 −4
  109. tools/testing/selftests/bpf/.gitignore +0 −1
  110. tools/testing/selftests/bpf/Makefile +201 −66
  111. tools/testing/selftests/bpf/README.rst +1 −1
  112. tools/testing/selftests/bpf/bench.c +21 −5
  113. tools/testing/selftests/bpf/bench.h +1 −0
  114. tools/testing/selftests/bpf/bench_bpf_timing.h +50 −0
  115. tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_full_update.c +32 −2
  116. tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_lookup.c +29 −2
  117. tools/testing/selftests/bpf/benchs/bench_bpf_nop.c +84 −0
  118. tools/testing/selftests/bpf/benchs/bench_bpf_timing.c +298 −0
  119. tools/testing/selftests/bpf/benchs/bench_htab_mem.c +33 −2
  120. tools/testing/selftests/bpf/benchs/bench_xdp_lb.c +1124 −0
  121. tools/testing/selftests/bpf/benchs/run_bench_xdp_lb.sh +79 −0
  122. tools/testing/selftests/bpf/bpf_arena_alloc.h +1 −1
  123. tools/testing/selftests/bpf/bpf_arena_htab.h +5 −6
  124. tools/testing/selftests/bpf/bpf_arena_list.h +1 −1
  125. tools/testing/selftests/bpf/bpf_arena_strsearch.h +3 −3
  126. tools/testing/selftests/bpf/bpf_experimental.h +1 −83
  127. tools/testing/selftests/bpf/bpf_kfuncs.h +4 −4
  128. tools/testing/selftests/bpf/config +2 −1
  129. tools/testing/selftests/bpf/default.profraw +0 −0
  130. tools/testing/selftests/bpf/jit_disasm_helpers.c +11 −2
  131. tools/testing/selftests/bpf/libarena/Makefile +92 −0
  132. tools/testing/selftests/bpf/libarena/include/bpf_arena_common.h +2 −3
  133. tools/testing/selftests/bpf/libarena/include/bpf_arena_spin_lock.h +9 −8
  134. tools/testing/selftests/bpf/libarena/include/bpf_atomic.h +3 −1
  135. tools/testing/selftests/bpf/libarena/include/bpf_may_goto.h +84 −0
  136. tools/testing/selftests/bpf/libarena/include/libarena/asan.h +101 −0
  137. tools/testing/selftests/bpf/libarena/include/libarena/buddy.h +81 −0
  138. tools/testing/selftests/bpf/libarena/include/libarena/common.h +93 −0
  139. tools/testing/selftests/bpf/libarena/include/libarena/rbtree.h +83 −0
  140. tools/testing/selftests/bpf/libarena/include/libarena/spmc.h +27 −0
  141. tools/testing/selftests/bpf/libarena/include/libarena/userspace.h +138 −0
  142. tools/testing/selftests/bpf/libarena/selftests/st_asan_buddy.bpf.c +258 −0
  143. tools/testing/selftests/bpf/libarena/selftests/st_asan_common.h +52 −0
  144. tools/testing/selftests/bpf/libarena/selftests/st_buddy.bpf.c +209 −0
  145. tools/testing/selftests/bpf/libarena/selftests/test_parallel_spmc.bpf.c +669 −0
  146. tools/testing/selftests/bpf/libarena/selftests/test_progs_compat.h +15 −0
  147. tools/testing/selftests/bpf/libarena/selftests/test_rbtree.bpf.c +968 −0
  148. tools/testing/selftests/bpf/libarena/selftests/test_spmc.bpf.c +194 −0
  149. tools/testing/selftests/bpf/libarena/src/asan.bpf.c +553 −0
  150. tools/testing/selftests/bpf/libarena/src/buddy.bpf.c +903 −0
  151. tools/testing/selftests/bpf/libarena/src/common.bpf.c +52 −0
  152. tools/testing/selftests/bpf/libarena/src/rbtree.bpf.c +1047 −0
  153. tools/testing/selftests/bpf/libarena/src/spmc.bpf.c +234 −0
  154. tools/testing/selftests/bpf/prog_tests/arena_direct_value.c +73 −0
  155. tools/testing/selftests/bpf/prog_tests/arena_spin_lock.c +0 −7
  156. tools/testing/selftests/bpf/prog_tests/bpf_attr_size.c +124 −0
  157. tools/testing/selftests/bpf/prog_tests/bpf_cookie.c +11 −6
  158. tools/testing/selftests/bpf/prog_tests/bpf_nf.c +11 −11
  159. tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c +8 −0
  160. tools/testing/selftests/bpf/prog_tests/btf.c +41 −4
  161. tools/testing/selftests/bpf/prog_tests/btf_dedup_split.c +33 −18
  162. tools/testing/selftests/bpf/prog_tests/btf_dump.c +2 −2
  163. tools/testing/selftests/bpf/prog_tests/cb_refs.c +2 −2
  164. tools/testing/selftests/bpf/prog_tests/cgrp_local_storage.c +13 −2
  165. tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c +8 −9
  166. tools/testing/selftests/bpf/prog_tests/exceptions.c +7 −0
  167. tools/testing/selftests/bpf/prog_tests/file_reader.c +10 −12
  168. tools/testing/selftests/bpf/prog_tests/fill_link_info.c +1 −1
  169. tools/testing/selftests/bpf/prog_tests/htab_update.c +2 −2
  170. tools/testing/selftests/bpf/prog_tests/iters.c +0 −2
  171. tools/testing/selftests/bpf/prog_tests/kfunc_call.c +1 −1
  172. tools/testing/selftests/bpf/prog_tests/libarena.c +253 −0
  173. tools/testing/selftests/bpf/prog_tests/libarena_asan.c +93 −0
  174. tools/testing/selftests/bpf/prog_tests/linked_list.c +19 −18
  175. tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c +243 −0
  176. tools/testing/selftests/bpf/prog_tests/lsm_cgroup.c +79 −0
  177. tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c +145 −0
  178. tools/testing/selftests/bpf/prog_tests/map_excl.c +122 −0
  179. tools/testing/selftests/bpf/prog_tests/map_init.c +192 −0
  180. tools/testing/selftests/bpf/prog_tests/map_kptr.c +57 −9
  181. tools/testing/selftests/bpf/prog_tests/refcounted_kptr.c +6 −2
  182. tools/testing/selftests/bpf/prog_tests/reg_bounds.c +75 −15
  183. tools/testing/selftests/bpf/prog_tests/rhash.c +183 −0
  184. tools/testing/selftests/bpf/prog_tests/setget_sockopt.c +78 −0
  185. tools/testing/selftests/bpf/prog_tests/signed_loader.c +1135 −0
  186. tools/testing/selftests/bpf/prog_tests/sleepable_tracepoints.c +142 −0
  187. tools/testing/selftests/bpf/prog_tests/sockmap_basic.c +48 −0
  188. tools/testing/selftests/bpf/prog_tests/spin_lock.c +2 −2
  189. tools/testing/selftests/bpf/prog_tests/stack_arg.c +139 −0
  190. tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c +10 −0
  191. tools/testing/selftests/bpf/prog_tests/stack_arg_precision.c +10 −0
  192. tools/testing/selftests/bpf/prog_tests/tailcalls.c +186 −0
  193. tools/testing/selftests/bpf/prog_tests/task_kfunc.c +41 −1
  194. tools/testing/selftests/bpf/prog_tests/task_local_storage.c +1 −0
  195. tools/testing/selftests/bpf/prog_tests/test_lsm.c +0 −22
  196. tools/testing/selftests/bpf/prog_tests/test_xdp_veth.c +163 −3
  197. tools/testing/selftests/bpf/prog_tests/tracing_multi.c +960 −0
  198. tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c +93 −1
  199. tools/testing/selftests/bpf/prog_tests/verifier.c +8 −0
  200. tools/testing/selftests/bpf/prog_tests/verifier_log.c +1 −0
  201. tools/testing/selftests/bpf/prog_tests/vmlinux.c +43 −2
  202. tools/testing/selftests/bpf/prog_tests/wakeup_source.c +118 −0
  203. tools/testing/selftests/bpf/progs/arena_atomics.c +1 −1
  204. tools/testing/selftests/bpf/progs/arena_spin_lock.c +2 −1
  205. tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h +69 −0
  206. tools/testing/selftests/bpf/progs/bpf_iter_bpf_rhash_map.c +34 −0
  207. tools/testing/selftests/bpf/progs/bpf_iter_task_vmas.c +1 −1
  208. tools/testing/selftests/bpf/progs/bpf_misc.h +1 −0
  209. tools/testing/selftests/bpf/progs/bpf_nop_bench.c +14 −0
  210. tools/testing/selftests/bpf/progs/bpf_qdisc_dynptr_use_after_invalidate_clone.c +74 −0
  211. tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr.c +68 −0
  212. tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c +74 −0
  213. tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_slice.c +70 −0
  214. tools/testing/selftests/bpf/progs/bpf_qdisc_fq.c +3 −8
  215. tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c +24 −0
  216. tools/testing/selftests/bpf/progs/btf__verifier_stack_arg_order.c +49 −0
  217. tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c +9 −9
  218. tools/testing/selftests/bpf/progs/cgrp_ls_sleepable.c +18 −0
  219. tools/testing/selftests/bpf/progs/compute_live_registers.c +1 −1
  220. tools/testing/selftests/bpf/progs/cpumask_failure.c +5 −5
  221. tools/testing/selftests/bpf/progs/cpumask_success.c +1 −1
  222. tools/testing/selftests/bpf/progs/crypto_bench.c +14 −7
  223. tools/testing/selftests/bpf/progs/crypto_sanity.c +14 −7
  224. tools/testing/selftests/bpf/progs/dynptr_fail.c +58 −16
  225. tools/testing/selftests/bpf/progs/dynptr_success.c +3 −3
  226. tools/testing/selftests/bpf/progs/exceptions.c +114 −0
  227. tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c +11 −2
  228. tools/testing/selftests/bpf/progs/file_reader.c +1 −1
  229. tools/testing/selftests/bpf/progs/file_reader_fail.c +62 −2
  230. tools/testing/selftests/bpf/progs/htab_update.c +2 −2
  231. tools/testing/selftests/bpf/progs/irq.c +2 −2
  232. tools/testing/selftests/bpf/progs/iters.c +3 −3
  233. tools/testing/selftests/bpf/progs/iters_state_safety.c +9 −9
  234. tools/testing/selftests/bpf/progs/iters_testmod.c +2 −2
  235. tools/testing/selftests/bpf/progs/iters_testmod_seq.c +8 −8
  236. tools/testing/selftests/bpf/progs/linked_list.c +71 −0
  237. tools/testing/selftests/bpf/progs/lpm_trie_bench.c +1 −1
  238. tools/testing/selftests/bpf/progs/lru_lock_nmi.c +33 −0
  239. tools/testing/selftests/bpf/progs/lsm_cgroup.c +30 −0
  240. tools/testing/selftests/bpf/progs/map_kptr.c +86 −3
  241. tools/testing/selftests/bpf/progs/map_kptr_fail.c +2 −2
  242. tools/testing/selftests/bpf/progs/percpu_alloc_fail.c +2 −2
  243. tools/testing/selftests/bpf/progs/rbtree_fail.c +3 −3
  244. tools/testing/selftests/bpf/progs/refcounted_kptr.c +440 −1
  245. tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c +1 −1
  246. tools/testing/selftests/bpf/progs/rhash.c +248 −0
  247. tools/testing/selftests/bpf/progs/setget_sockopt.c +23 −0
  248. tools/testing/selftests/bpf/progs/sk_bypass_prot_mem.c +2 −0
  249. tools/testing/selftests/bpf/progs/stack_arg.c +273 −0
  250. tools/testing/selftests/bpf/progs/stack_arg_fail.c +114 −0
  251. tools/testing/selftests/bpf/progs/stack_arg_kfunc.c +166 −0
  252. tools/testing/selftests/bpf/progs/stack_arg_precision.c +135 −0
  253. tools/testing/selftests/bpf/progs/stream.c +1 −1
  254. tools/testing/selftests/bpf/progs/stream_fail.c +1 −1
  255. tools/testing/selftests/bpf/progs/tailcall_bpf2bpf2.c +4 −1
  256. tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy1.c +9 −4
  257. tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy2.c +17 −7
  258. tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy3.c +10 −3
  259. tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy_fentry.c +10 −3
  260. tools/testing/selftests/bpf/progs/tailcall_cgrp_storage.c +44 −0
  261. tools/testing/selftests/bpf/progs/tailcall_cgrp_storage_no_storage.c +26 −0
  262. tools/testing/selftests/bpf/progs/tailcall_cgrp_storage_owner.c +32 −0
  263. tools/testing/selftests/bpf/progs/task_kfunc_failure.c +51 −11
  264. tools/testing/selftests/bpf/progs/task_kfunc_success.c +6 −7
  265. tools/testing/selftests/bpf/progs/task_local_storage.c +19 −0
  266. tools/testing/selftests/bpf/progs/task_work_fail.c +3 −3
  267. tools/testing/selftests/bpf/progs/test_bpf_nf_fail.c +4 −4
  268. tools/testing/selftests/bpf/progs/test_fill_link_info.c +1 −1
  269. tools/testing/selftests/bpf/progs/test_global_func3.c +50 −2
  270. tools/testing/selftests/bpf/progs/test_kfunc_dynptr_param.c +3 −8
  271. tools/testing/selftests/bpf/progs/test_kfunc_param_nullable.c +1 −1
  272. tools/testing/selftests/bpf/progs/test_lirc_mode2_kern.c +2 −2
  273. tools/testing/selftests/bpf/progs/test_lwt_ip_encap.c +145 −10
  274. tools/testing/selftests/bpf/progs/test_ringbuf_map_key.c +3 −8
  275. tools/testing/selftests/bpf/progs/test_signed_loader.c +18 −0
  276. tools/testing/selftests/bpf/progs/test_signed_loader_data.c +20 −0
  277. tools/testing/selftests/bpf/progs/test_signed_loader_lsm.c +30 −0
  278. tools/testing/selftests/bpf/progs/test_signed_loader_map.c +28 −0
  279. tools/testing/selftests/bpf/progs/test_sleepable_tracepoints.c +112 −0
  280. tools/testing/selftests/bpf/progs/test_sleepable_tracepoints_fail.c +18 −0
  281. tools/testing/selftests/bpf/progs/test_sockmap_msg_pop_data.c +27 −0
  282. tools/testing/selftests/bpf/progs/test_tunnel_kern.c +6 −7
  283. tools/testing/selftests/bpf/progs/test_vmlinux.c +2 −2
  284. tools/testing/selftests/bpf/progs/test_wakeup_source.c +92 −0
  285. tools/testing/selftests/bpf/progs/tracing_multi_attach.c +39 −0
  286. tools/testing/selftests/bpf/progs/tracing_multi_attach_module.c +25 −0
  287. tools/testing/selftests/bpf/progs/tracing_multi_bench.c +12 −0
  288. tools/testing/selftests/bpf/progs/tracing_multi_check.c +214 −0
  289. tools/testing/selftests/bpf/progs/tracing_multi_fail.c +18 −0
  290. tools/testing/selftests/bpf/progs/tracing_multi_intersect_attach.c +41 −0
  291. tools/testing/selftests/bpf/progs/tracing_multi_rollback.c +43 −0
  292. tools/testing/selftests/bpf/progs/tracing_multi_session_attach.c +65 −0
  293. tools/testing/selftests/bpf/progs/tracing_multi_verifier.c +31 −0
  294. tools/testing/selftests/bpf/progs/user_ringbuf_fail.c +2 −2
  295. tools/testing/selftests/bpf/progs/verifier_arena.c +68 −1
  296. tools/testing/selftests/bpf/progs/verifier_arena_globals1.c +1 −1
  297. tools/testing/selftests/bpf/progs/verifier_arena_globals2.c +1 −1
  298. tools/testing/selftests/bpf/progs/verifier_arena_large.c +1 −1
  299. tools/testing/selftests/bpf/progs/verifier_bits_iter.c +2 −2
  300. tools/testing/selftests/bpf/progs/verifier_bounds.c +120 −10
  301. tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c +13 −14
  302. tools/testing/selftests/bpf/progs/verifier_flow_keys.c +97 −0
  303. tools/testing/selftests/bpf/progs/verifier_global_ptr_args.c +20 −1
  304. tools/testing/selftests/bpf/progs/verifier_global_subprogs.c +19 −1
  305. tools/testing/selftests/bpf/progs/verifier_jit_inline.c +3 −1
  306. tools/testing/selftests/bpf/progs/verifier_ldsx.c +4 −4
  307. tools/testing/selftests/bpf/progs/verifier_liveness_exp.c +1 −1
  308. tools/testing/selftests/bpf/progs/verifier_lsm.c +9 −0
  309. tools/testing/selftests/bpf/progs/verifier_map_in_map.c +40 −0
  310. tools/testing/selftests/bpf/progs/verifier_map_ptr.c +32 −4
  311. tools/testing/selftests/bpf/progs/verifier_may_goto_1.c +6 −6
  312. tools/testing/selftests/bpf/progs/verifier_private_stack.c +25 −5
  313. tools/testing/selftests/bpf/progs/verifier_ref_tracking.c +4 −4
  314. tools/testing/selftests/bpf/progs/verifier_scalar_ids.c +12 −13
  315. tools/testing/selftests/bpf/progs/verifier_sdiv.c +32 −32
  316. tools/testing/selftests/bpf/progs/verifier_set_retval.c +107 −0
  317. tools/testing/selftests/bpf/progs/verifier_sock.c +10 −5
  318. tools/testing/selftests/bpf/progs/verifier_stack_arg.c +447 −0
  319. tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c +185 −0
  320. tools/testing/selftests/bpf/progs/verifier_subreg.c +4 −2
  321. tools/testing/selftests/bpf/progs/verifier_tailcall_jit.c +1 −0
  322. tools/testing/selftests/bpf/progs/verifier_vfs_reject.c +5 −5
  323. tools/testing/selftests/bpf/progs/wakeup_source.h +22 −0
  324. tools/testing/selftests/bpf/progs/wakeup_source_fail.c +76 −0
  325. tools/testing/selftests/bpf/progs/wq_failures.c +1 −1
  326. tools/testing/selftests/bpf/progs/xdp_flowtable.c +5 −2
  327. tools/testing/selftests/bpf/progs/xdp_lb_bench.c +647 −0
  328. tools/testing/selftests/bpf/progs/xdping_kern.c +0 −183
  329. tools/testing/selftests/bpf/test_kmods/Makefile +27 −3
  330. tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +79 −0
  331. tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +33 −0
  332. tools/testing/selftests/bpf/test_lirc_mode2_user.c +3 −3
  333. tools/testing/selftests/bpf/test_loader.c +174 −13
  334. tools/testing/selftests/bpf/test_maps.c +13 −10
  335. tools/testing/selftests/bpf/test_progs.c +46 −9
  336. tools/testing/selftests/bpf/test_progs.h +3 −0
  337. tools/testing/selftests/bpf/test_xdping.sh +0 −103
  338. tools/testing/selftests/bpf/testing_helpers.c +18 −0
  339. tools/testing/selftests/bpf/testing_helpers.h +1 −0
  340. tools/testing/selftests/bpf/trace_helpers.c +4 −3
  341. tools/testing/selftests/bpf/trace_helpers.h +1 −0
  342. tools/testing/selftests/bpf/uprobe_multi.c +3 −1
  343. tools/testing/selftests/bpf/verifier/calls.c +55 −31
  344. tools/testing/selftests/bpf/verifier/sleepable.c +15 −2
  345. tools/testing/selftests/bpf/veristat.c +9 −4
  346. tools/testing/selftests/bpf/vmtest.sh +1 −1
  347. tools/testing/selftests/bpf/xdp_lb_bench_common.h +112 −0
  348. tools/testing/selftests/bpf/xdping.c +0 −254
  349. tools/testing/selftests/bpf/xdping.h +0 −13
Documentation/bpf/btf.rst+2−4
@@ -97,10 +97,8 @@ Each type contains the following common data:: · 96 unmodified lines
9797 struct btf_type {
9898 __u32 name_off;
9999 /* "info" bits arrangement
100− * bits 0-15: vlen (e.g. # of struct's members)
101− * bits 16-23: unused
102− * bits 24-28: kind (e.g. int, ptr, array...etc)
103− * bits 29-30: unused
100+ * bits 0-23: vlen (e.g. # of struct's members)
101+ * bits 24-30: kind (e.g. int, ptr, array...etc)
104102 * bit 31: kind_flag, currently used by
105103 * struct, union, enum, fwd, enum64,
106104 * decl_tag and type_tag
Documentation/bpf/kfuncs.rst+34−2
@@ -207,8 +207,26 @@ Here, the buffer may be NULL. If the buffer is not NULL, it must be at least · 206 unmodified lines
207207 buffer__szk bytes in size. The kfunc is responsible for checking if the buffer
208208 is NULL before using it.
209209
210−2.3.5 __str Annotation
211−----------------------------
210+2.3.5 __nonown_allowed Annotation
211+---------------------------------
212+
213+This annotation is used to indicate that the parameter may be a non-owning reference.
214+
215+An example is given below::
216+
217+ __bpf_kfunc int bpf_list_add(..., struct bpf_list_node
218+ *prev__nonown_allowed, ...)
219+ {
220+ ...
221+ }
222+
223+For the ``prev__nonown_allowed`` parameter (resolved as ``KF_ARG_PTR_TO_LIST_NODE``),
224+suffix ``__nonown_allowed`` retains the usual owning-pointer rules and also
225+permits a non-owning reference with no ref_obj_id (e.g. the return value of
226+bpf_list_front() / bpf_list_back()).
227+
228+2.3.6 __str Annotation
229+----------------------
212230 This annotation is used to indicate that the argument is a constant string.
213231
214232 An example is given below::
@@ -462,6 +480,20 @@ In order to accommodate such requirements, the verifier will enforce strict · 247 unmodified lines
462480 PTR_TO_BTF_ID type matching if two types have the exact same name, with one
463481 being suffixed with ``___init``.
464482
483+2.8 Accessing arena memory through kfunc arguments
484+--------------------------------------------------
485+
486+A read or write at any address inside an arena does not oops the kernel.
487+Unallocated arena pages are lazily backed by a scratch page and the
488+access is reported through the program's BPF stream as an error. Only
489+the BPF program's correctness is affected; the kernel itself remains
490+intact.
491+
492+The arena is followed by a ``GUARD_SZ / 2`` (32 KiB) guard region that
493+is also covered by this recovery. A kfunc handed an arena pointer may
494+therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
495+against the arena. Larger accesses must verify the range explicitly.
496+
465497 .. _BPF_kfunc_lifecycle_expectations:
466498
467499 3. kfunc lifecycle expectations
Documentation/bpf/map_lru_hash_update.dot+42−2
@@ -21,10 +21,18 @@ digraph { · 20 unmodified lines
2121 // names that initiate the corresponding logic in kernel/bpf/bpf_lru_list.c.
2222 // Number suffixes and errno suffixes handle subsections of the corresponding
2323 // logic in the function as of the writing of this dot.
24+ //
25+ // All LRU locks are rqspinlock_t. Every acquire can fail (AA self-deadlock
26+ // or contention timeout); on failure the corresponding helper returns NULL
27+ // and the caller propagates -ENOMEM. The "rqspinlock acquire failed"
28+ // terminal below is reached via the dashed arrows from each acquire site.
29+
30+ rqspinlock_failed [shape=rectangle,
31+ label="Any LRU rqspinlock\nacquire fails\n(AA or timeout)"]
2432
2533 // cf. __local_list_pop_free() / bpf_percpu_lru_pop_free()
2634 local_freelist_check [shape=diamond,fillcolor=1,
27− label="Local freelist\nnode available?"];
35+ label="Local freelist\nnode available?\n(lockless free_llist)"];
2836 use_local_node [shape=rectangle,
2937 label="Use node owned\nby this CPU"]
3038
@@ -82,6 +90,15 @@ digraph { · 51 unmodified lines
8290 // fn__local_list_pop_pending()
8391 }
8492
93+ // Post-steal: re-acquire local loc_l->lock to insert the stolen node into
94+ // the local pending list. If the acquire fails, the stolen node is published
95+ // to the lockless local free_llist so the next pop on this CPU picks it up
96+ // instead of orphaning it.
97+ post_steal_lock [shape=diamond,fillcolor=1,
98+ label="Acquire local\nloc_l->lock\nto add pending"]
99+ post_steal_to_free_llist [shape=rectangle,
100+ label="Publish stolen node to\nlocal free_llist (lockless)"]
101+
85102 fn_bpf_lru_list_pop_free_to_local2 [shape=rectangle,
86103 label="Use node that was\nnot recently referenced"]
87104 local_freelist_check4 [shape=rectangle,
@@ -97,10 +114,19 @@ digraph { · 9 unmodified lines
97114 fn_htab_lru_map_update_elem_ENOENT [shape=oval,label="return -ENOENT"]
98115
99116 begin -> local_freelist_check
117+ // The initial per-CPU lock (loc_l->lock for common, l->lock for percpu) is
118+ // acquired before the local freelist check; rqspinlock failure here exits
119+ // directly to -ENOMEM (no recovery needed: nothing was removed yet).
120+ local_freelist_check -> rqspinlock_failed [style=dashed,
121+ xlabel="acquire fails"]
100122 local_freelist_check -> use_local_node [xlabel="Y"]
101123 local_freelist_check -> common_lru_check [xlabel="N"]
102124 common_lru_check -> fn_bpf_lru_list_pop_free_to_local [xlabel="Y"]
103125 common_lru_check -> fn___bpf_lru_list_shrink_inactive [xlabel="N"]
126+ // Global lru_list lock acquire failure in pop_free_to_local: skip refill,
127+ // fall through to the steal path. Not ENOMEM by itself.
128+ fn_bpf_lru_list_pop_free_to_local -> common_lru_check2 [style=dashed,
129+ xlabel="global lru_lock\nacquire fails"]
104130 fn_bpf_lru_list_pop_free_to_local -> fn___bpf_lru_node_move_to_free
105131 fn___bpf_lru_node_move_to_free ->
106132 fn_bpf_lru_list_pop_free_to_local2 [xlabel="Y"]
@@ -120,13 +146,27 @@ digraph { · 13 unmodified lines
120146 local_freelist_check6 -> local_freelist_check7
121147 local_freelist_check7 -> fn_htab_lru_map_update_elem
122148
123− fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem3 [xlabel = "Y"]
149+ // Steal-loop victim lock failure is silent: treat as "no node found here"
150+ // and continue to next CPU; same edge as the existing "N" path.
151+ local_freelist_check5 -> fn_htab_lru_map_update_elem2 [style=dashed,
152+ xlabel="victim's lock\nfails: skip"]
153+ // After a successful steal, re-acquire the local loc_l->lock. On failure
154+ // the stolen node is published to free_llist (recovered, not orphaned)
155+ // and the update returns -ENOMEM.
156+ fn_htab_lru_map_update_elem -> post_steal_lock [xlabel = "Y"]
157+ post_steal_lock -> fn_htab_lru_map_update_elem3 [xlabel = "OK"]
158+ post_steal_lock -> post_steal_to_free_llist [style=dashed,
159+ xlabel="loc_l->lock\nacquire fails"]
160+ post_steal_to_free_llist -> fn_htab_lru_map_update_elem_ENOMEM
124161 fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem2 [xlabel = "N"]
125162 fn_htab_lru_map_update_elem2 ->
126163 fn_htab_lru_map_update_elem_ENOMEM [xlabel = "Y"]
127164 fn_htab_lru_map_update_elem2 -> local_freelist_check5 [xlabel = "N"]
128165 fn_htab_lru_map_update_elem3 -> fn_htab_lru_map_update_elem4
129166
167+ // Shared rqspinlock-failure terminal collapses to the same -ENOMEM exit.
168+ rqspinlock_failed -> fn_htab_lru_map_update_elem_ENOMEM
169+
130170 use_local_node -> fn_htab_lru_map_update_elem4
131171 fn_bpf_lru_list_pop_free_to_local2 -> fn_htab_lru_map_update_elem4
132172 local_freelist_check4 -> fn_htab_lru_map_update_elem4
MAINTAINERS+7−27
@@ -4797,13 +4797,6 @@ S: Supported · 4796 unmodified lines
47974797 F: arch/x86/net/
47984798 X: arch/x86/net/bpf_jit_comp32.c
47994799
4800−BPF [BTF]
4801−M: Martin KaFai Lau <martin.lau@linux.dev>
4802−L: bpf@vger.kernel.org
4803−S: Maintained
4804−F: include/linux/btf*
4805−F: kernel/bpf/btf.c
4806−
48074800 BPF [CORE]
48084801 M: Alexei Starovoitov <ast@kernel.org>
48094802 M: Daniel Borkmann <daniel@iogearbox.net>
@@ -4840,12 +4833,13 @@ BPF [GENERAL] (Safe Dynamic Programs and Tools) · 30 unmodified lines
48404833 M: Alexei Starovoitov <ast@kernel.org>
48414834 M: Daniel Borkmann <daniel@iogearbox.net>
48424835 M: Andrii Nakryiko <andrii@kernel.org>
4843−R: Martin KaFai Lau <martin.lau@linux.dev>
48444836 M: Eduard Zingerman <eddyz87@gmail.com>
48454837 M: Kumar Kartikeya Dwivedi <memxor@gmail.com>
4838+R: Martin KaFai Lau <martin.lau@linux.dev>
48464839 R: Song Liu <song@kernel.org>
48474840 R: Yonghong Song <yonghong.song@linux.dev>
48484841 R: Jiri Olsa <jolsa@kernel.org>
4842+R: Emil Tsalapatis <emil@etsalapatis.com>
48494843 L: bpf@vger.kernel.org
48504844 S: Supported
48514845 W: https://bpf.io/
@@ -4855,7 +4849,9 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git · 3 unmodified lines
48554849 F: Documentation/bpf/
48564850 F: Documentation/networking/filter.rst
48574851 F: Documentation/userspace-api/ebpf/
4852+F: arch/*/include/asm/rqspinlock.h
48584853 F: arch/*/net/*
4854+F: include/asm-generic/rqspinlock.h
48594855 F: include/linux/bpf*
48604856 F: include/linux/btf*
48614857 F: include/linux/buildid.h
@@ -4867,17 +4863,15 @@ F: include/uapi/linux/filter.h · 5 unmodified lines
48674863 F: kernel/bpf/
48684864 F: kernel/trace/bpf_trace.c
48694865 F: lib/buildid.c
4870−F: arch/*/include/asm/rqspinlock.h
4871−F: include/asm-generic/rqspinlock.h
48724866 F: lib/test_bpf.c
48734867 F: net/bpf/
48744868 F: net/core/filter.c
48754869 F: net/sched/act_bpf.c
48764870 F: net/sched/cls_bpf.c
48774871 F: samples/bpf/
4872+F: scripts/Makefile.btf
48784873 F: scripts/bpf_doc.py
48794874 F: scripts/gen-btf.sh
4880−F: scripts/Makefile.btf
48814875 F: scripts/pahole-version.sh
48824876 F: tools/bpf/
48834877 F: tools/lib/bpf/
@@ -4892,6 +4886,7 @@ F: kernel/bpf/*iter.c · 8 unmodified lines
48924886 BPF [L7 FRAMEWORK] (sockmap)
48934887 M: John Fastabend <john.fastabend@gmail.com>
48944888 M: Jakub Sitnicki <jakub@cloudflare.com>
4889+M: Jiayuan Chen <jiayuan.chen@linux.dev>
48954890 L: netdev@vger.kernel.org
48964891 L: bpf@vger.kernel.org
48974892 S: Maintained
@@ -4935,18 +4930,11 @@ F: tools/testing/selftests/bpf/prog_tests/tc_netkit.c · 37 unmodified lines
49354930 F: tools/testing/selftests/drivers/net/hw/nk_qlease.py
49364931 F: tools/testing/selftests/net/nk_qlease.py
49374932
4938−BPF [NETWORKING] (struct_ops, reuseport)
4939−M: Martin KaFai Lau <martin.lau@linux.dev>
4940−L: bpf@vger.kernel.org
4941−L: netdev@vger.kernel.org
4942−S: Maintained
4943−F: kernel/bpf/bpf_struct*
4944−
49454933 BPF [NETWORKING] (tcx & tc BPF, sock_addr)
4946−M: Martin KaFai Lau <martin.lau@linux.dev>
49474934 M: Daniel Borkmann <daniel@iogearbox.net>
49484935 R: John Fastabend <john.fastabend@gmail.com>
49494936 R: Stanislav Fomichev <sdf@fomichev.me>
4937+R: Martin KaFai Lau <martin.lau@linux.dev>
49504938 L: bpf@vger.kernel.org
49514939 L: netdev@vger.kernel.org
49524940 S: Maintained
@@ -4981,14 +4969,6 @@ L: bpf@vger.kernel.org · 28 unmodified lines
49814969 S: Maintained
49824970 F: tools/testing/selftests/bpf/
49834971
4984−BPF [STORAGE & CGROUPS]
4985−M: Martin KaFai Lau <martin.lau@linux.dev>
4986−L: bpf@vger.kernel.org
4987−S: Maintained
4988−F: kernel/bpf/*storage.c
4989−F: kernel/bpf/bpf_lru*
4990−F: kernel/bpf/cgroup.c
4991−
49924972 BPF [TOOLING] (bpftool)
49934973 M: Quentin Monnet <qmo@kernel.org>
49944974 L: bpf@vger.kernel.org
arch/arm64/include/asm/pgtable.h+28−0
@@ -1830,6 +1830,34 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, · 1829 unmodified lines
18301830 return __ptep_get_and_clear(mm, addr, ptep);
18311831 }
18321832
1833+static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte)
1834+{
1835+ pteval_t old = 0;
1836+
1837+ if (!try_cmpxchg(&pte_val(*ptep), &old, pte_val(new_pte)))
1838+ return false;
1839+
1840+ /*
1841+ * The store must be complete by the time this returns, but the caller
1842+ * may be in lazy MMU mode, where __set_pte_complete() would defer the
1843+ * barriers. Issue them directly.
1844+ */
1845+ emit_pte_barriers();
1846+ return true;
1847+}
1848+#define ptep_try_set ptep_try_set
1849+
1850+/*
1851+ * arm64 mandates break-before-make: a cleared kernel PTE must have its TLB
1852+ * invalidated before a different page is installed in its place. The broadcast
1853+ * TLBI is an instruction, not an IPI, so this is safe with interrupts disabled.
1854+ */
1855+static inline void flush_tlb_before_set(unsigned long addr)
1856+{
1857+ flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
1858+}
1859+#define flush_tlb_before_set flush_tlb_before_set
1860+
18331861 #define test_and_clear_young_ptes test_and_clear_young_ptes
18341862 static inline bool test_and_clear_young_ptes(struct vm_area_struct *vma,
18351863 unsigned long addr, pte_t *ptep, unsigned int nr)
arch/arm64/mm/fault.c+7−3
@@ -9,6 +9,7 @@ · 8 unmodified lines
99
1010 #include <linux/acpi.h>
1111 #include <linux/bitfield.h>
12+#include <linux/bpf_defs.h>
1213 #include <linux/extable.h>
1314 #include <linux/kfence.h>
1415 #include <linux/signal.h>
@@ -436,9 +437,12 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr, · 421 unmodified lines
436437 } else if (is_pkvm_stage2_abort(esr)) {
437438 msg = "access to hypervisor-protected memory";
438439 } else {
439− if (esr_fsc_is_translation_fault(esr) &&
440− kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs))
441− return;
440+ if (esr_fsc_is_translation_fault(esr)) {
441+ if (kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs))
442+ return;
443+ if (bpf_arena_handle_page_fault(addr, esr & ESR_ELx_WNR, regs->pc))
444+ return;
445+ }
442446
443447 msg = "paging request";
444448 }
arch/arm64/net/bpf_jit_comp.c+120−32
@@ -47,7 +47,7 @@ · 46 unmodified lines
4747 /* Map BPF registers to A64 registers */
4848 static const int bpf2a64[] = {
4949 /* return value from in-kernel function, and exit value from eBPF */
50− [BPF_REG_0] = A64_R(7),
50+ [BPF_REG_0] = A64_R(8),
5151 /* arguments from eBPF program to in-kernel function */
5252 [BPF_REG_1] = A64_R(0),
5353 [BPF_REG_2] = A64_R(1),
@@ -86,6 +86,7 @@ struct jit_ctx { · 32 unmodified lines
8686 __le32 *image;
8787 __le32 *ro_image;
8888 u32 stack_size;
89+ u16 stack_arg_size;
8990 u64 user_vm_start;
9091 u64 arena_vm_start;
9192 bool fp_used;
@@ -533,13 +534,20 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · 441 unmodified lines
533534 * | |
534535 * +-----+ <= (BPF_FP - prog->aux->stack_depth)
535536 * |RSVD | padding
536− * current A64_SP => +-----+ <= (BPF_FP - ctx->stack_size)
537+ * +-----+ <= (BPF_FP - ctx->stack_size)
538+ * | |
539+ * | ... | outgoing stack args (9+, if any)
540+ * | |
541+ * current A64_SP => +-----+
537542 * | |
538543 * | ... | Function call stack
539544 * | |
540545 * +-----+
541546 * low
542547 *
548+ * Stack args 6-8 are passed in x5-x7, args 9+ at [SP].
549+ * Incoming args 9+ are at [A64_FP + 16], [A64_FP + 24], ...
550+ * (above the saved FP/LR pair pushed in the callee prologue).
543551 */
544552
545553 emit_kcfi(is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx);
@@ -613,6 +621,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · 67 unmodified lines
613621 if (ctx->stack_size && !ctx->priv_sp_used)
614622 emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
615623
624+ if (ctx->stack_arg_size)
625+ emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx);
626+
616627 if (ctx->arena_vm_start)
617628 emit_a64_mov_i64(arena_vm_base, ctx->arena_vm_start, ctx);
618629
@@ -673,6 +684,9 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx) · 54 unmodified lines
673684 /* Update tail_call_cnt if the slot is populated. */
674685 emit(A64_STR64I(tcc, ptr, 0), ctx);
675686
687+ if (ctx->stack_arg_size)
688+ emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx);
689+
676690 /* restore SP */
677691 if (ctx->stack_size && !ctx->priv_sp_used)
678692 emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
@@ -1034,6 +1048,9 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic) · 355 unmodified lines
10341048 const u8 r0 = bpf2a64[BPF_REG_0];
10351049 const u8 ptr = bpf2a64[TCCNT_PTR];
10361050
1051+ if (ctx->stack_arg_size)
1052+ emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx);
1053+
10371054 /* We're done with BPF stack */
10381055 if (ctx->stack_size && !ctx->priv_sp_used)
10391056 emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
@@ -1048,7 +1065,7 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic) · 8 unmodified lines
10481065 /* Restore FP/LR registers */
10491066 emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);
10501067
1051− /* Move the return value from bpf:r0 (aka x7) to x0 */
1068+ /* Move the return value from bpf:r0 (aka x8) to x0 */
10521069 emit(A64_MOV(1, A64_R(0), r0), ctx);
10531070
10541071 /* Authenticate lr */
@@ -1191,6 +1208,42 @@ static int add_exception_handler(const struct bpf_insn *insn, · 136 unmodified lines
11911208 return 0;
11921209 }
11931210
1211+static const u8 stack_arg_reg[] = { A64_R(5), A64_R(6), A64_R(7) };
1212+
1213+#define NR_STACK_ARG_REGS ARRAY_SIZE(stack_arg_reg)
1214+
1215+static void emit_stack_arg_load(u8 dst, s16 bpf_off, struct jit_ctx *ctx)
1216+{
1217+ int idx = bpf_off / sizeof(u64) - 1;
1218+
1219+ if (idx < NR_STACK_ARG_REGS)
1220+ emit(A64_MOV(1, dst, stack_arg_reg[idx]), ctx);
1221+ else
1222+ emit(A64_LDR64I(dst, A64_FP, (idx - NR_STACK_ARG_REGS) * sizeof(u64) + 16), ctx);
1223+}
1224+
1225+static void emit_stack_arg_store(u8 src_a64, s16 bpf_off, struct jit_ctx *ctx)
1226+{
1227+ int idx = -bpf_off / sizeof(u64) - 1;
1228+
1229+ if (idx < NR_STACK_ARG_REGS)
1230+ emit(A64_MOV(1, stack_arg_reg[idx], src_a64), ctx);
1231+ else
1232+ emit(A64_STR64I(src_a64, A64_SP, (idx - NR_STACK_ARG_REGS) * sizeof(u64)), ctx);
1233+}
1234+
1235+static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct jit_ctx *ctx)
1236+{
1237+ int idx = -bpf_off / sizeof(u64) - 1;
1238+
1239+ if (idx < NR_STACK_ARG_REGS) {
1240+ emit_a64_mov_i(1, stack_arg_reg[idx], imm, ctx);
1241+ } else {
1242+ emit_a64_mov_i(1, tmp, imm, ctx);
1243+ emit(A64_STR64I(tmp, A64_SP, (idx - NR_STACK_ARG_REGS) * sizeof(u64)), ctx);
1244+ }
1245+}
1246+
11941247 /* JITs an eBPF instruction.
11951248 * Returns:
11961249 * 0 - successfully JITed an 8-byte eBPF instruction.
@@ -1646,6 +1699,11 @@ emit_cond_jmp: · 449 unmodified lines
16461699 case BPF_LDX | BPF_MEM | BPF_H:
16471700 case BPF_LDX | BPF_MEM | BPF_B:
16481701 case BPF_LDX | BPF_MEM | BPF_DW:
1702+ if (insn->src_reg == BPF_REG_PARAMS) {
1703+ emit_stack_arg_load(dst, off, ctx);
1704+ break;
1705+ }
1706+ fallthrough;
16491707 case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
16501708 case BPF_LDX | BPF_PROBE_MEM | BPF_W:
16511709 case BPF_LDX | BPF_PROBE_MEM | BPF_H:
@@ -1672,6 +1730,8 @@ emit_cond_jmp: · 20 unmodified lines
16721730 if (src == fp) {
16731731 src_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
16741732 off_adj = off + ctx->stack_size;
1733+ if (!ctx->priv_sp_used)
1734+ off_adj += ctx->stack_arg_size;
16751735 } else {
16761736 src_adj = src;
16771737 off_adj = off;
@@ -1752,6 +1812,11 @@ emit_cond_jmp: · 74 unmodified lines
17521812 case BPF_ST | BPF_MEM | BPF_H:
17531813 case BPF_ST | BPF_MEM | BPF_B:
17541814 case BPF_ST | BPF_MEM | BPF_DW:
1815+ if (insn->dst_reg == BPF_REG_PARAMS) {
1816+ emit_stack_arg_store_imm(imm, off, tmp, ctx);
1817+ break;
1818+ }
1819+ fallthrough;
17551820 case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
17561821 case BPF_ST | BPF_PROBE_MEM32 | BPF_H:
17571822 case BPF_ST | BPF_PROBE_MEM32 | BPF_W:
@@ -1763,6 +1828,8 @@ emit_cond_jmp: · 5 unmodified lines
17631828 if (dst == fp) {
17641829 dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
17651830 off_adj = off + ctx->stack_size;
1831+ if (!ctx->priv_sp_used)
1832+ off_adj += ctx->stack_arg_size;
17661833 } else {
17671834 dst_adj = dst;
17681835 off_adj = off;
@@ -1814,6 +1881,11 @@ emit_cond_jmp: · 45 unmodified lines
18141881 case BPF_STX | BPF_MEM | BPF_H:
18151882 case BPF_STX | BPF_MEM | BPF_B:
18161883 case BPF_STX | BPF_MEM | BPF_DW:
1884+ if (insn->dst_reg == BPF_REG_PARAMS) {
1885+ emit_stack_arg_store(src, off, ctx);
1886+ break;
1887+ }
1888+ fallthrough;
18171889 case BPF_STX | BPF_PROBE_MEM32 | BPF_B:
18181890 case BPF_STX | BPF_PROBE_MEM32 | BPF_H:
18191891 case BPF_STX | BPF_PROBE_MEM32 | BPF_W:
@@ -1825,6 +1897,8 @@ emit_cond_jmp: · 5 unmodified lines
18251897 if (dst == fp) {
18261898 dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
18271899 off_adj = off + ctx->stack_size;
1900+ if (!ctx->priv_sp_used)
1901+ off_adj += ctx->stack_arg_size;
18281902 } else {
18291903 dst_adj = dst;
18301904 off_adj = off;
@@ -2018,6 +2092,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · 187 unmodified lines
20182092 u8 *ro_image_ptr;
20192093 int body_idx;
20202094 int exentry_idx;
2095+ int out_cnt;
20212096
20222097 if (!prog->jit_requested)
20232098 return prog;
@@ -2065,6 +2140,14 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · 41 unmodified lines
20652140 ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
20662141 ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
20672142
2143+ out_cnt = bpf_out_stack_arg_cnt(env, prog);
2144+ if (out_cnt) {
2145+ int nr_on_stack = out_cnt - NR_STACK_ARG_REGS;
2146+
2147+ if (nr_on_stack > 0)
2148+ ctx.stack_arg_size = round_up(nr_on_stack * sizeof(u64), 16);
2149+ }
2150+
20682151 if (priv_stack_ptr)
20692152 ctx.priv_sp_used = true;
20702153
@@ -2229,6 +2312,11 @@ bool bpf_jit_supports_kfunc_call(void) · 158 unmodified lines
22292312 return true;
22302313 }
22312314
2315+bool bpf_jit_supports_stack_args(void)
2316+{
2317+ return true;
2318+}
2319+
22322320 void *bpf_arch_text_copy(void *dst, void *src, size_t len)
22332321 {
22342322 if (!aarch64_insn_copy(dst, src, len))
@@ -2247,24 +2335,24 @@ bool bpf_jit_supports_subprog_tailcalls(void) · 12 unmodified lines
22472335 return true;
22482336 }
22492337
2250−static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,
2338+static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node,
22512339 int bargs_off, int retval_off, int run_ctx_off,
22522340 bool save_ret)
22532341 {
22542342 __le32 *branch;
22552343 u64 enter_prog;
22562344 u64 exit_prog;
2257− struct bpf_prog *p = l->link.prog;
2345+ struct bpf_prog *p = node->link->prog;
22582346 int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
22592347
22602348 enter_prog = (u64)bpf_trampoline_enter(p);
22612349 exit_prog = (u64)bpf_trampoline_exit(p);
22622350
2263− if (l->cookie == 0) {
2351+ if (node->cookie == 0) {
22642352 /* if cookie is zero, one instruction is enough to store it */
22652353 emit(A64_STR64I(A64_ZR, A64_SP, run_ctx_off + cookie_off), ctx);
22662354 } else {
2267− emit_a64_mov_i64(A64_R(10), l->cookie, ctx);
2355+ emit_a64_mov_i64(A64_R(10), node->cookie, ctx);
22682356 emit(A64_STR64I(A64_R(10), A64_SP, run_ctx_off + cookie_off),
22692357 ctx);
22702358 }
@@ -2314,7 +2402,7 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, · 43 unmodified lines
23142402 emit_call(exit_prog, ctx);
23152403 }
23162404
2317−static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl,
2405+static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn,
23182406 int bargs_off, int retval_off, int run_ctx_off,
23192407 __le32 **branches)
23202408 {
@@ -2324,8 +2412,8 @@ static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl, · 3 unmodified lines
23242412 * Set this to 0 to avoid confusing the program.
23252413 */
23262414 emit(A64_STR64I(A64_ZR, A64_SP, retval_off), ctx);
2327− for (i = 0; i < tl->nr_links; i++) {
2328− invoke_bpf_prog(ctx, tl->links[i], bargs_off, retval_off,
2415+ for (i = 0; i < tn->nr_nodes; i++) {
2416+ invoke_bpf_prog(ctx, tn->nodes[i], bargs_off, retval_off,
23292417 run_ctx_off, true);
23302418 /* if (*(u64 *)(sp + retval_off) != 0)
23312419 * goto do_fexit;
@@ -2456,10 +2544,10 @@ static void restore_args(struct jit_ctx *ctx, int bargs_off, int nregs) · 124 unmodified lines
24562544 }
24572545 }
24582546
2459−static bool is_struct_ops_tramp(const struct bpf_tramp_links *fentry_links)
2547+static bool is_struct_ops_tramp(const struct bpf_tramp_nodes *fentry_nodes)
24602548 {
2461− return fentry_links->nr_links == 1 &&
2462− fentry_links->links[0]->link.type == BPF_LINK_TYPE_STRUCT_OPS;
2549+ return fentry_nodes->nr_nodes == 1 &&
2550+ fentry_nodes->nodes[0]->link->type == BPF_LINK_TYPE_STRUCT_OPS;
24632551 }
24642552
24652553 static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_off)
@@ -2480,7 +2568,7 @@ static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_of · 14 unmodified lines
24802568 *
24812569 */
24822570 static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
2483− struct bpf_tramp_links *tlinks, void *func_addr,
2571+ struct bpf_tramp_nodes *tnodes, void *func_addr,
24842572 const struct btf_func_model *m,
24852573 const struct arg_aux *a,
24862574 u32 flags)
@@ -2496,14 +2584,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 9 unmodified lines
24962584 int run_ctx_off;
24972585 int oargs_off;
24982586 int nfuncargs;
2499− struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
2500− struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
2501− struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
2587+ struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
2588+ struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
2589+ struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
25022590 bool save_ret;
25032591 __le32 **branches = NULL;
25042592 bool is_struct_ops = is_struct_ops_tramp(fentry);
25052593 int cookie_off, cookie_cnt, cookie_bargs_off;
2506− int fsession_cnt = bpf_fsession_cnt(tlinks);
2594+ int fsession_cnt = bpf_fsession_cnt(tnodes);
25072595 u64 func_meta;
25082596
25092597 /* trampoline stack layout:
@@ -2549,7 +2637,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 39 unmodified lines
25492637
25502638 cookie_off = stack_size;
25512639 /* room for session cookies */
2552− cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
2640+ cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
25532641 stack_size += cookie_cnt * 8;
25542642
25552643 ip_off = stack_size;
@@ -2646,20 +2734,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 90 unmodified lines
26462734 }
26472735
26482736 cookie_bargs_off = (bargs_off - cookie_off) / 8;
2649− for (i = 0; i < fentry->nr_links; i++) {
2650− if (bpf_prog_calls_session_cookie(fentry->links[i])) {
2737+ for (i = 0; i < fentry->nr_nodes; i++) {
2738+ if (bpf_prog_calls_session_cookie(fentry->nodes[i])) {
26512739 u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
26522740
26532741 store_func_meta(ctx, meta, func_meta_off);
26542742 cookie_bargs_off--;
26552743 }
2656− invoke_bpf_prog(ctx, fentry->links[i], bargs_off,
2744+ invoke_bpf_prog(ctx, fentry->nodes[i], bargs_off,
26572745 retval_off, run_ctx_off,
26582746 flags & BPF_TRAMP_F_RET_FENTRY_RET);
26592747 }
26602748
2661− if (fmod_ret->nr_links) {
2662− branches = kcalloc(fmod_ret->nr_links, sizeof(__le32 *),
2749+ if (fmod_ret->nr_nodes) {
2750+ branches = kcalloc(fmod_ret->nr_nodes, sizeof(__le32 *),
26632751 GFP_KERNEL);
26642752 if (!branches)
26652753 return -ENOMEM;
@@ -2683,7 +2771,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 17 unmodified lines
26832771 }
26842772
26852773 /* update the branches saved in invoke_bpf_mod_ret with cbnz */
2686− for (i = 0; i < fmod_ret->nr_links && ctx->image != NULL; i++) {
2774+ for (i = 0; i < fmod_ret->nr_nodes && ctx->image != NULL; i++) {
26872775 int offset = &ctx->image[ctx->idx] - branches[i];
26882776 *branches[i] = cpu_to_le32(A64_CBNZ(1, A64_R(10), offset));
26892777 }
@@ -2694,14 +2782,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 4 unmodified lines
26942782 store_func_meta(ctx, func_meta, func_meta_off);
26952783
26962784 cookie_bargs_off = (bargs_off - cookie_off) / 8;
2697− for (i = 0; i < fexit->nr_links; i++) {
2698− if (bpf_prog_calls_session_cookie(fexit->links[i])) {
2785+ for (i = 0; i < fexit->nr_nodes; i++) {
2786+ if (bpf_prog_calls_session_cookie(fexit->nodes[i])) {
26992787 u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
27002788
27012789 store_func_meta(ctx, meta, func_meta_off);
27022790 cookie_bargs_off--;
27032791 }
2704− invoke_bpf_prog(ctx, fexit->links[i], bargs_off, retval_off,
2792+ invoke_bpf_prog(ctx, fexit->nodes[i], bargs_off, retval_off,
27052793 run_ctx_off, false);
27062794 }
27072795
@@ -2759,7 +2847,7 @@ bool bpf_jit_supports_fsession(void) · 51 unmodified lines
27592847 }
27602848
27612849 int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
2762− struct bpf_tramp_links *tlinks, void *func_addr)
2850+ struct bpf_tramp_nodes *tnodes, void *func_addr)
27632851 {
27642852 struct jit_ctx ctx = {
27652853 .image = NULL,
@@ -2773,7 +2861,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 7 unmodified lines
27732861 if (ret < 0)
27742862 return ret;
27752863
2776− ret = prepare_trampoline(&ctx, &im, tlinks, func_addr, m, &aaux, flags);
2864+ ret = prepare_trampoline(&ctx, &im, tnodes, func_addr, m, &aaux, flags);
27772865 if (ret < 0)
27782866 return ret;
27792867
@@ -2797,7 +2885,7 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) · 17 unmodified lines
27972885
27982886 int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
27992887 void *ro_image_end, const struct btf_func_model *m,
2800− u32 flags, struct bpf_tramp_links *tlinks,
2888+ u32 flags, struct bpf_tramp_nodes *tnodes,
28012889 void *func_addr)
28022890 {
28032891 u32 size = ro_image_end - ro_image;
@@ -2824,7 +2912,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · 20 unmodified lines
28242912 ret = calc_arg_aux(m, &aaux);
28252913 if (ret)
28262914 goto out;
2827− ret = prepare_trampoline(&ctx, im, tlinks, func_addr, m, &aaux, flags);
2915+ ret = prepare_trampoline(&ctx, im, tnodes, func_addr, m, &aaux, flags);
28282916
28292917 if (ret > 0 && validate_code(&ctx) < 0) {
28302918 ret = -EINVAL;
arch/arm64/net/bpf_timed_may_goto.S+4−4
@@ -8,8 +8,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto) · 7 unmodified lines
88 stp x29, x30, [sp, #-64]!
99 mov x29, sp
1010
11− /* Save BPF registers R0 - R5 (x7, x0-x4)*/
12− stp x7, x0, [sp, #16]
11+ /* Save BPF registers R0 - R5 (x8, x0-x4)*/
12+ stp x8, x0, [sp, #16]
1313 stp x1, x2, [sp, #32]
1414 stp x3, x4, [sp, #48]
1515
@@ -28,8 +28,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto) · 12 unmodified lines
2828 /* BPF_REG_AX(x9) will be stored into count, so move return value to it. */
2929 mov x9, x0
3030
31− /* Restore BPF registers R0 - R5 (x7, x0-x4) */
32− ldp x7, x0, [sp, #16]
31+ /* Restore BPF registers R0 - R5 (x8, x0-x4) */
32+ ldp x8, x0, [sp, #16]
3333 ldp x1, x2, [sp, #32]
3434 ldp x3, x4, [sp, #48]
3535
arch/loongarch/net/bpf_jit.c+26−26
@@ -1674,17 +1674,17 @@ static void restore_stk_args(struct jit_ctx *ctx, int nr_stk_args, int args_off, · 1673 unmodified lines
16741674 }
16751675 }
16761676
1677−static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,
1677+static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *n,
16781678 int args_off, int retval_off, int run_ctx_off, bool save_ret)
16791679 {
16801680 int ret;
16811681 u32 *branch;
1682− struct bpf_prog *p = l->link.prog;
1682+ struct bpf_prog *p = n->link->prog;
16831683 int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
16841684
1685− if (l->cookie)
1685+ if (n->cookie)
16861686 emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1,
1687− -run_ctx_off + cookie_off, l->cookie);
1687+ -run_ctx_off + cookie_off, n->cookie);
16881688 else
16891689 emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -run_ctx_off + cookie_off);
16901690
@@ -1737,22 +1737,22 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, · 46 unmodified lines
17371737 return ret;
17381738 }
17391739
1740−static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_links *tl,
1740+static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn,
17411741 int args_off, int retval_off, int run_ctx_off,
17421742 int func_meta_off, bool save_ret, u64 func_meta, int cookie_off)
17431743 {
17441744 int i, cur_cookie = (cookie_off - args_off) / 8;
17451745
1746− for (i = 0; i < tl->nr_links; i++) {
1746+ for (i = 0; i < tn->nr_nodes; i++) {
17471747 int err;
17481748
1749− if (bpf_prog_calls_session_cookie(tl->links[i])) {
1749+ if (bpf_prog_calls_session_cookie(tn->nodes[i])) {
17501750 u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);
17511751
17521752 emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, meta);
17531753 cur_cookie--;
17541754 }
1755− err = invoke_bpf_prog(ctx, tl->links[i], args_off, retval_off, run_ctx_off, save_ret);
1755+ err = invoke_bpf_prog(ctx, tn->nodes[i], args_off, retval_off, run_ctx_off, save_ret);
17561756 if (err)
17571757 return err;
17581758 }
@@ -1807,7 +1807,7 @@ static void sign_extend(struct jit_ctx *ctx, int rd, int rj, u8 size, bool sign) · 48 unmodified lines
18071807 }
18081808
18091809 static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
1810− const struct btf_func_model *m, struct bpf_tramp_links *tlinks,
1810+ const struct btf_func_model *m, struct bpf_tramp_nodes *tnodes,
18111811 void *func_addr, u32 flags)
18121812 {
18131813 int i, ret, save_ret;
@@ -1817,9 +1817,9 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 3 unmodified lines
18171817 unsigned long long func_meta;
18181818 bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT;
18191819 void *orig_call = func_addr;
1820− struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
1821− struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
1822− struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
1820+ struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
1821+ struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
1822+ struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
18231823 u32 **branches = NULL;
18241824
18251825 /*
@@ -1898,7 +1898,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 72 unmodified lines
18981898 ip_off = stack_size;
18991899 }
19001900
1901− cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
1901+ cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
19021902
19031903 /* Room for session cookies */
19041904 stack_size += cookie_cnt * 8;
@@ -1969,7 +1969,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 64 unmodified lines
19691969
19701970 store_args(ctx, nr_arg_slots, args_off);
19711971
1972− if (bpf_fsession_cnt(tlinks)) {
1972+ if (bpf_fsession_cnt(tnodes)) {
19731973 /* clear all session cookies' value */
19741974 for (i = 0; i < cookie_cnt; i++)
19751975 emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -cookie_off + 8 * i);
@@ -1994,20 +1994,20 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 18 unmodified lines
19941994 return ret;
19951995 }
19961996
1997− if (fentry->nr_links) {
1997+ if (fentry->nr_nodes) {
19981998 ret = invoke_bpf(ctx, fentry, args_off, retval_off, run_ctx_off, func_meta_off,
19991999 flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off);
20002000 if (ret)
20012001 return ret;
20022002 }
2003− if (fmod_ret->nr_links) {
2004− branches = kcalloc(fmod_ret->nr_links, sizeof(u32 *), GFP_KERNEL);
2003+ if (fmod_ret->nr_nodes) {
2004+ branches = kcalloc(fmod_ret->nr_nodes, sizeof(u32 *), GFP_KERNEL);
20052005 if (!branches)
20062006 return -ENOMEM;
20072007
20082008 emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -retval_off);
2009− for (i = 0; i < fmod_ret->nr_links; i++) {
2010− ret = invoke_bpf_prog(ctx, fmod_ret->links[i],
2009+ for (i = 0; i < fmod_ret->nr_nodes; i++) {
2010+ ret = invoke_bpf_prog(ctx, fmod_ret->nodes[i],
20112011 args_off, retval_off, run_ctx_off, true);
20122012 if (ret)
20132013 goto out;
@@ -2035,17 +2035,17 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 21 unmodified lines
20352035 emit_insn(ctx, nop);
20362036 }
20372037
2038− for (i = 0; ctx->image && i < fmod_ret->nr_links; i++) {
2038+ for (i = 0; ctx->image && i < fmod_ret->nr_nodes; i++) {
20392039 int offset = (void *)(&ctx->image[ctx->idx]) - (void *)branches[i];
20402040 *branches[i] = larch_insn_gen_bne(LOONGARCH_GPR_T1, LOONGARCH_GPR_ZERO, offset);
20412041 }
20422042
20432043 /* Set "is_return" flag for fsession */
20442044 func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
2045− if (bpf_fsession_cnt(tlinks))
2045+ if (bpf_fsession_cnt(tnodes))
20462046 emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, func_meta);
20472047
2048− if (fexit->nr_links) {
2048+ if (fexit->nr_nodes) {
20492049 ret = invoke_bpf(ctx, fexit, args_off, retval_off, run_ctx_off,
20502050 func_meta_off, false, func_meta, cookie_off);
20512051 if (ret)
@@ -2115,7 +2115,7 @@ out: · 63 unmodified lines
21152115
21162116 int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
21172117 void *ro_image_end, const struct btf_func_model *m,
2118− u32 flags, struct bpf_tramp_links *tlinks, void *func_addr)
2118+ u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr)
21192119 {
21202120 int ret, size;
21212121 void *image, *tmp;
@@ -2131,7 +2131,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · 9 unmodified lines
21312131 ctx.idx = 0;
21322132
21332133 jit_fill_hole(image, (unsigned int)(ro_image_end - ro_image));
2134− ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tlinks, func_addr, flags);
2134+ ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tnodes, func_addr, flags);
21352135 if (ret < 0)
21362136 goto out;
21372137
@@ -2152,7 +2152,7 @@ out: · 14 unmodified lines
21522152 }
21532153
21542154 int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
2155− struct bpf_tramp_links *tlinks, void *func_addr)
2155+ struct bpf_tramp_nodes *tnodes, void *func_addr)
21562156 {
21572157 int ret;
21582158 struct jit_ctx ctx;
@@ -2161,7 +2161,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 2 unmodified lines
21612161 ctx.image = NULL;
21622162 ctx.idx = 0;
21632163
2164− ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tlinks, func_addr, flags);
2164+ ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tnodes, func_addr, flags);
21652165
21662166 return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE;
21672167 }
arch/powerpc/net/bpf_jit_comp.c+27−27
@@ -597,22 +597,22 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) · 596 unmodified lines
597597 }
598598
599599 static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ctx,
600− struct bpf_tramp_link *l, int regs_off, int retval_off,
600+ struct bpf_tramp_node *n, int regs_off, int retval_off,
601601 int run_ctx_off, bool save_ret)
602602 {
603− struct bpf_prog *p = l->link.prog;
603+ struct bpf_prog *p = n->link->prog;
604604 ppc_inst_t branch_insn;
605605 u32 jmp_idx;
606606 int ret = 0;
607607
608608 /* Save cookie */
609609 if (IS_ENABLED(CONFIG_PPC64)) {
610− PPC_LI64(_R3, l->cookie);
610+ PPC_LI64(_R3, n->cookie);
611611 EMIT(PPC_RAW_STD(_R3, _R1, run_ctx_off + offsetof(struct bpf_tramp_run_ctx,
612612 bpf_cookie)));
613613 } else {
614− PPC_LI32(_R3, l->cookie >> 32);
615− PPC_LI32(_R4, l->cookie);
614+ PPC_LI32(_R3, n->cookie >> 32);
615+ PPC_LI32(_R4, n->cookie);
616616 EMIT(PPC_RAW_STW(_R3, _R1,
617617 run_ctx_off + offsetof(struct bpf_tramp_run_ctx, bpf_cookie)));
618618 EMIT(PPC_RAW_STW(_R4, _R1,
@@ -679,7 +679,7 @@ static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ct · 60 unmodified lines
679679 }
680680
681681 static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context *ctx,
682− struct bpf_tramp_links *tl, int regs_off, int retval_off,
682+ struct bpf_tramp_nodes *tn, int regs_off, int retval_off,
683683 int run_ctx_off, u32 *branches)
684684 {
685685 int i;
@@ -690,8 +690,8 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context · 4 unmodified lines
690690 */
691691 EMIT(PPC_RAW_LI(_R3, 0));
692692 EMIT(PPC_RAW_STL(_R3, _R1, retval_off));
693− for (i = 0; i < tl->nr_links; i++) {
694− if (invoke_bpf_prog(image, ro_image, ctx, tl->links[i], regs_off, retval_off,
693+ for (i = 0; i < tn->nr_nodes; i++) {
694+ if (invoke_bpf_prog(image, ro_image, ctx, tn->nodes[i], regs_off, retval_off,
695695 run_ctx_off, true))
696696 return -EINVAL;
697697
@@ -807,18 +807,18 @@ static void bpf_trampoline_restore_args_stack(u32 *image, struct codegen_context · 109 unmodified lines
807807 static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,
808808 void *rw_image_end, void *ro_image,
809809 const struct btf_func_model *m, u32 flags,
810− struct bpf_tramp_links *tlinks,
810+ struct bpf_tramp_nodes *tnodes,
811811 void *func_addr)
812812 {
813813 int regs_off, func_meta_off, ip_off, run_ctx_off, retval_off;
814814 int nvr_off, alt_lr_off, r4_off = 0;
815− struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
816− struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
817− struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
815+ struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
816+ struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
817+ struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
818818 int i, ret, nr_regs, retaddr_off, bpf_frame_size = 0;
819819 struct codegen_context codegen_ctx, *ctx;
820820 int cookie_off, cookie_cnt, cookie_ctx_off;
821− int fsession_cnt = bpf_fsession_cnt(tlinks);
821+ int fsession_cnt = bpf_fsession_cnt(tnodes);
822822 u64 func_meta;
823823 u32 *image = (u32 *)rw_image;
824824 ppc_inst_t branch_insn;
@@ -893,7 +893,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 68 unmodified lines
893893
894894 /* room for session cookies */
895895 cookie_off = bpf_frame_size;
896− cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
896+ cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
897897 bpf_frame_size += cookie_cnt * 8;
898898
899899 /* Room for IP address argument */
@@ -1030,21 +1030,21 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 130 unmodified lines
10301030
10311031 cookie_ctx_off = (regs_off - cookie_off) / 8;
10321032
1033− for (i = 0; i < fentry->nr_links; i++) {
1034− if (bpf_prog_calls_session_cookie(fentry->links[i])) {
1033+ for (i = 0; i < fentry->nr_nodes; i++) {
1034+ if (bpf_prog_calls_session_cookie(fentry->nodes[i])) {
10351035 u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
10361036
10371037 store_func_meta(image, ctx, meta, func_meta_off);
10381038 cookie_ctx_off--;
10391039 }
10401040
1041− if (invoke_bpf_prog(image, ro_image, ctx, fentry->links[i], regs_off, retval_off,
1041+ if (invoke_bpf_prog(image, ro_image, ctx, fentry->nodes[i], regs_off, retval_off,
10421042 run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET))
10431043 return -EINVAL;
10441044 }
10451045
1046− if (fmod_ret->nr_links) {
1047− branches = kcalloc(fmod_ret->nr_links, sizeof(u32), GFP_KERNEL);
1046+ if (fmod_ret->nr_nodes) {
1047+ branches = kcalloc(fmod_ret->nr_nodes, sizeof(u32), GFP_KERNEL);
10481048 if (!branches)
10491049 return -ENOMEM;
10501050
@@ -1093,7 +1093,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 42 unmodified lines
10931093 }
10941094
10951095 /* Update branches saved in invoke_bpf_mod_ret with address of do_fexit */
1096− for (i = 0; i < fmod_ret->nr_links && image; i++) {
1096+ for (i = 0; i < fmod_ret->nr_nodes && image; i++) {
10971097 if (create_cond_branch(&branch_insn, &image[branches[i]],
10981098 (unsigned long)&image[ctx->idx], COND_NE << 16)) {
10991099 ret = -EINVAL;
@@ -1110,15 +1110,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 10 unmodified lines
11101110
11111111 cookie_ctx_off = (regs_off - cookie_off) / 8;
11121112
1113− for (i = 0; i < fexit->nr_links; i++) {
1114− if (bpf_prog_calls_session_cookie(fexit->links[i])) {
1113+ for (i = 0; i < fexit->nr_nodes; i++) {
1114+ if (bpf_prog_calls_session_cookie(fexit->nodes[i])) {
11151115 u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
11161116
11171117 store_func_meta(image, ctx, meta, func_meta_off);
11181118 cookie_ctx_off--;
11191119 }
11201120
1121− if (invoke_bpf_prog(image, ro_image, ctx, fexit->links[i], regs_off, retval_off,
1121+ if (invoke_bpf_prog(image, ro_image, ctx, fexit->nodes[i], regs_off, retval_off,
11221122 run_ctx_off, false)) {
11231123 ret = -EINVAL;
11241124 goto cleanup;
@@ -1185,18 +1185,18 @@ cleanup: · 60 unmodified lines
11851185 }
11861186
11871187 int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
1188− struct bpf_tramp_links *tlinks, void *func_addr)
1188+ struct bpf_tramp_nodes *tnodes, void *func_addr)
11891189 {
11901190 struct bpf_tramp_image im;
11911191 int ret;
11921192
1193− ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tlinks, func_addr);
1193+ ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tnodes, func_addr);
11941194 return ret;
11951195 }
11961196
11971197 int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
11981198 const struct btf_func_model *m, u32 flags,
1199− struct bpf_tramp_links *tlinks,
1199+ struct bpf_tramp_nodes *tnodes,
12001200 void *func_addr)
12011201 {
12021202 u32 size = image_end - image;
@@ -1212,7 +1212,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i · 9 unmodified lines
12121212 return -ENOMEM;
12131213
12141214 ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,
1215− flags, tlinks, func_addr);
1215+ flags, tnodes, func_addr);
12161216 if (ret < 0)
12171217 goto out;
12181218
arch/riscv/net/bpf_jit_comp64.c+35−26
@@ -934,15 +934,15 @@ static void emit_store_stack_imm64(u8 reg, int stack_off, u64 imm64, · 933 unmodified lines
934934 emit_sd(RV_REG_FP, stack_off, reg, ctx);
935935 }
936936
937−static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_off,
937+static int invoke_bpf_prog(struct bpf_tramp_node *node, int args_off, int retval_off,
938938 int run_ctx_off, bool save_ret, struct rv_jit_context *ctx)
939939 {
940940 int ret, branch_off;
941− struct bpf_prog *p = l->link.prog;
941+ struct bpf_prog *p = node->link->prog;
942942 int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
943943
944− if (l->cookie)
945− emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, l->cookie, ctx);
944+ if (node->cookie)
945+ emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, node->cookie, ctx);
946946 else
947947 emit_sd(RV_REG_FP, -run_ctx_off + cookie_off, RV_REG_ZERO, ctx);
948948
@@ -996,22 +996,22 @@ static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_of · 47 unmodified lines
996996 return ret;
997997 }
998998
999−static int invoke_bpf(struct bpf_tramp_links *tl, int args_off, int retval_off,
999+static int invoke_bpf(struct bpf_tramp_nodes *tn, int args_off, int retval_off,
10001000 int run_ctx_off, int func_meta_off, bool save_ret, u64 func_meta,
10011001 int cookie_off, struct rv_jit_context *ctx)
10021002 {
10031003 int i, cur_cookie = (cookie_off - args_off) / 8;
10041004
1005− for (i = 0; i < tl->nr_links; i++) {
1005+ for (i = 0; i < tn->nr_nodes; i++) {
10061006 int err;
10071007
1008− if (bpf_prog_calls_session_cookie(tl->links[i])) {
1008+ if (bpf_prog_calls_session_cookie(tn->nodes[i])) {
10091009 u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);
10101010
10111011 emit_store_stack_imm64(RV_REG_T1, -func_meta_off, meta, ctx);
10121012 cur_cookie--;
10131013 }
1014− err = invoke_bpf_prog(tl->links[i], args_off, retval_off, run_ctx_off,
1014+ err = invoke_bpf_prog(tn->nodes[i], args_off, retval_off, run_ctx_off,
10151015 save_ret, ctx);
10161016 if (err)
10171017 return err;
@@ -1021,7 +1021,7 @@ static int invoke_bpf(struct bpf_tramp_links *tl, int args_off, int retval_off, · 3 unmodified lines
10211021
10221022 static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
10231023 const struct btf_func_model *m,
1024− struct bpf_tramp_links *tlinks,
1024+ struct bpf_tramp_nodes *tnodes,
10251025 void *func_addr, u32 flags,
10261026 struct rv_jit_context *ctx)
10271027 {
@@ -1030,9 +1030,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 2 unmodified lines
10301030 int stack_size = 0, nr_arg_slots = 0;
10311031 int retval_off, args_off, func_meta_off, ip_off, run_ctx_off, sreg_off, stk_arg_off;
10321032 int cookie_off, cookie_cnt;
1033− struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
1034− struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
1035− struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
1033+ struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
1034+ struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
1035+ struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
10361036 bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT;
10371037 void *orig_call = func_addr;
10381038 bool save_ret;
@@ -1115,7 +1115,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 76 unmodified lines
11151115 ip_off = stack_size;
11161116 }
11171117
1118− cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
1118+ cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
11191119 /* room for session cookies */
11201120 stack_size += cookie_cnt * 8;
11211121 cookie_off = stack_size;
@@ -1172,7 +1172,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 50 unmodified lines
11721172
11731173 store_args(nr_arg_slots, args_off, ctx);
11741174
1175− if (bpf_fsession_cnt(tlinks)) {
1175+ if (bpf_fsession_cnt(tnodes)) {
11761176 /* clear all session cookies' value */
11771177 for (i = 0; i < cookie_cnt; i++)
11781178 emit_sd(RV_REG_FP, -cookie_off + 8 * i, RV_REG_ZERO, ctx);
@@ -1187,22 +1187,22 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 8 unmodified lines
11871187 return ret;
11881188 }
11891189
1190− if (fentry->nr_links) {
1190+ if (fentry->nr_nodes) {
11911191 ret = invoke_bpf(fentry, args_off, retval_off, run_ctx_off, func_meta_off,
11921192 flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off, ctx);
11931193 if (ret)
11941194 return ret;
11951195 }
11961196
1197− if (fmod_ret->nr_links) {
1198− branches_off = kzalloc_objs(int, fmod_ret->nr_links);
1197+ if (fmod_ret->nr_nodes) {
1198+ branches_off = kzalloc_objs(int, fmod_ret->nr_nodes);
11991199 if (!branches_off)
12001200 return -ENOMEM;
12011201
12021202 /* cleanup to avoid garbage return value confusion */
12031203 emit_sd(RV_REG_FP, -retval_off, RV_REG_ZERO, ctx);
1204− for (i = 0; i < fmod_ret->nr_links; i++) {
1205− ret = invoke_bpf_prog(fmod_ret->links[i], args_off, retval_off,
1204+ for (i = 0; i < fmod_ret->nr_nodes; i++) {
1205+ ret = invoke_bpf_prog(fmod_ret->nodes[i], args_off, retval_off,
12061206 run_ctx_off, true, ctx);
12071207 if (ret)
12081208 goto out;
@@ -1230,7 +1230,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 21 unmodified lines
12301230 }
12311231
12321232 /* update branches saved in invoke_bpf_mod_ret with bnez */
1233− for (i = 0; ctx->insns && i < fmod_ret->nr_links; i++) {
1233+ for (i = 0; ctx->insns && i < fmod_ret->nr_nodes; i++) {
12341234 offset = ninsns_rvoff(ctx->ninsns - branches_off[i]);
12351235 insn = rv_bne(RV_REG_T1, RV_REG_ZERO, offset >> 1);
12361236 *(u32 *)(ctx->insns + branches_off[i]) = insn;
@@ -1238,10 +1238,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 1 unmodified line
12381238
12391239 /* set "is_return" flag for fsession */
12401240 func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
1241− if (bpf_fsession_cnt(tlinks))
1241+ if (bpf_fsession_cnt(tnodes))
12421242 emit_store_stack_imm64(RV_REG_T1, -func_meta_off, func_meta, ctx);
12431243
1244− if (fexit->nr_links) {
1244+ if (fexit->nr_nodes) {
12451245 ret = invoke_bpf(fexit, args_off, retval_off, run_ctx_off, func_meta_off,
12461246 false, func_meta, cookie_off, ctx);
12471247 if (ret)
@@ -1305,7 +1305,7 @@ out: · 57 unmodified lines
13051305 }
13061306
13071307 int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
1308− struct bpf_tramp_links *tlinks, void *func_addr)
1308+ struct bpf_tramp_nodes *tnodes, void *func_addr)
13091309 {
13101310 struct bpf_tramp_image im;
13111311 struct rv_jit_context ctx;
@@ -1314,7 +1314,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 2 unmodified lines
13141314 ctx.ninsns = 0;
13151315 ctx.insns = NULL;
13161316 ctx.ro_insns = NULL;
1317− ret = __arch_prepare_bpf_trampoline(&im, m, tlinks, func_addr, flags, &ctx);
1317+ ret = __arch_prepare_bpf_trampoline(&im, m, tnodes, func_addr, flags, &ctx);
13181318
13191319 return ret < 0 ? ret : ninsns_rvoff(ctx.ninsns);
13201320 }
@@ -1331,7 +1331,7 @@ void arch_free_bpf_trampoline(void *image, unsigned int size) · 10 unmodified lines
13311331
13321332 int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
13331333 void *ro_image_end, const struct btf_func_model *m,
1334− u32 flags, struct bpf_tramp_links *tlinks,
1334+ u32 flags, struct bpf_tramp_nodes *tnodes,
13351335 void *func_addr)
13361336 {
13371337 int ret;
@@ -1346,7 +1346,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · 8 unmodified lines
13461346 ctx.ninsns = 0;
13471347 ctx.insns = image;
13481348 ctx.ro_insns = ro_image;
1349− ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx);
1349+ ret = __arch_prepare_bpf_trampoline(im, m, tnodes, func_addr, flags, &ctx);
13501350 if (ret < 0)
13511351 goto out;
13521352
@@ -1808,6 +1808,13 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · 455 unmodified lines
18081808 break;
18091809 }
18101810
1811+ /* Implement helper call to bpf_get_current_task/_btf() inline */
1812+ if (insn->src_reg == 0 && (insn->imm == BPF_FUNC_get_current_task ||
1813+ insn->imm == BPF_FUNC_get_current_task_btf)) {
1814+ emit_mv(bpf_to_rv_reg(BPF_REG_0, ctx), RV_REG_TP, ctx);
1815+ break;
1816+ }
1817+
18111818 mark_call(ctx);
18121819 ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass,
18131820 &addr, &fixed_addr);
@@ -2138,6 +2145,8 @@ bool bpf_jit_inlines_helper_call(s32 imm) · 324 unmodified lines
21382145 {
21392146 switch (imm) {
21402147 case BPF_FUNC_get_smp_processor_id:
2148+ case BPF_FUNC_get_current_task:
2149+ case BPF_FUNC_get_current_task_btf:
21412150 return true;
21422151 default:
21432152 return false;
arch/s390/net/bpf_jit_comp.c+59−22
@@ -27,6 +27,7 @@ · 26 unmodified lines
2727 #include <asm/extable.h>
2828 #include <asm/dis.h>
2929 #include <asm/facility.h>
30+#include <asm/lowcore.h>
3031 #include <asm/nospec-branch.h>
3132 #include <asm/set_memory.h>
3233 #include <asm/text-patching.h>
@@ -1777,6 +1778,30 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, · 1744 unmodified lines
17771778 int j, ret;
17781779 u64 func;
17791780
1781+ /* Implement helper call to bpf_get_smp_processor_id() inline */
1782+ if (insn->src_reg == 0 &&
1783+ insn->imm == BPF_FUNC_get_smp_processor_id) {
1784+ const u32 *cpu_nr = &get_lowcore()->cpu_nr;
1785+
1786+ /* ly %b0, cpu_nr */
1787+ EMIT6_DISP_LH(0xe3000000, 0x0058, BPF_REG_0, REG_0, REG_0,
1788+ (unsigned long)cpu_nr);
1789+ break;
1790+ }
1791+
1792+ /* Implement helper call to bpf_get_current_task/_btf() inline */
1793+ if (insn->src_reg == 0 &&
1794+ (insn->imm == BPF_FUNC_get_current_task ||
1795+ insn->imm == BPF_FUNC_get_current_task_btf)) {
1796+ const u64 *current_task =
1797+ &get_lowcore()->current_task;
1798+
1799+ /* lg %b0, current_task */
1800+ EMIT6_DISP_LH(0xe3000000, 0x0004, BPF_REG_0, REG_0, REG_0,
1801+ (unsigned long)current_task);
1802+ break;
1803+ }
1804+
17801805 ret = bpf_jit_get_func_addr(fp, insn, extra_pass,
17811806 &func, &func_addr_fixed);
17821807 if (ret < 0)
@@ -2512,19 +2537,19 @@ static void emit_store_stack_imm64(struct bpf_jit *jit, int tmp_reg, int stack_o · 729 unmodified lines
25122537
25132538 static int invoke_bpf_prog(struct bpf_tramp_jit *tjit,
25142539 const struct btf_func_model *m,
2515− struct bpf_tramp_link *tlink, bool save_ret)
2540+ struct bpf_tramp_node *node, bool save_ret)
25162541 {
25172542 struct bpf_jit *jit = &tjit->common;
25182543 int cookie_off = tjit->run_ctx_off +
25192544 offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
2520− struct bpf_prog *p = tlink->link.prog;
2545+ struct bpf_prog *p = node->link->prog;
25212546 int patch;
25222547
25232548 /*
2524− * run_ctx.cookie = tlink->cookie;
2549+ * run_ctx.cookie = node->cookie;
25252550 */
25262551
2527− emit_store_stack_imm64(jit, REG_W0, cookie_off, tlink->cookie);
2552+ emit_store_stack_imm64(jit, REG_W0, cookie_off, node->cookie);
25282553
25292554 /*
25302555 * if ((start = __bpf_prog_enter(p, &run_ctx)) == 0)
@@ -2584,20 +2609,20 @@ static int invoke_bpf_prog(struct bpf_tramp_jit *tjit, · 53 unmodified lines
25842609
25852610 static int invoke_bpf(struct bpf_tramp_jit *tjit,
25862611 const struct btf_func_model *m,
2587− struct bpf_tramp_links *tl, bool save_ret,
2612+ struct bpf_tramp_nodes *tn, bool save_ret,
25882613 u64 func_meta, int cookie_off)
25892614 {
25902615 int i, cur_cookie = (tjit->bpf_args_off - cookie_off) / sizeof(u64);
25912616 struct bpf_jit *jit = &tjit->common;
25922617
2593− for (i = 0; i < tl->nr_links; i++) {
2594− if (bpf_prog_calls_session_cookie(tl->links[i])) {
2618+ for (i = 0; i < tn->nr_nodes; i++) {
2619+ if (bpf_prog_calls_session_cookie(tn->nodes[i])) {
25952620 u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);
25962621
25972622 emit_store_stack_imm64(jit, REG_0, tjit->func_meta_off, meta);
25982623 cur_cookie--;
25992624 }
2600− if (invoke_bpf_prog(tjit, m, tl->links[i], save_ret))
2625+ if (invoke_bpf_prog(tjit, m, tn->nodes[i], save_ret))
26012626 return -EINVAL;
26022627 }
26032628
@@ -2626,12 +2651,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 22 unmodified lines
26262651 struct bpf_tramp_jit *tjit,
26272652 const struct btf_func_model *m,
26282653 u32 flags,
2629− struct bpf_tramp_links *tlinks,
2654+ struct bpf_tramp_nodes *tnodes,
26302655 void *func_addr)
26312656 {
2632− struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
2633− struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
2634− struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
2657+ struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
2658+ struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
2659+ struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
26352660 int nr_bpf_args, nr_reg_args, nr_stack_args;
26362661 int cookie_cnt, cookie_off, fsession_cnt;
26372662 struct bpf_jit *jit = &tjit->common;
@@ -2668,8 +2693,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 30 unmodified lines
26682693 return -ENOTSUPP;
26692694 }
26702695
2671− cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
2672− fsession_cnt = bpf_fsession_cnt(tlinks);
2696+ cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
2697+ fsession_cnt = bpf_fsession_cnt(tnodes);
26732698
26742699 /*
26752700 * Calculate the stack layout.
@@ -2804,7 +2829,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 128 unmodified lines
28042829 func_meta, cookie_off))
28052830 return -EINVAL;
28062831
2807− if (fmod_ret->nr_links) {
2832+ if (fmod_ret->nr_nodes) {
28082833 /*
28092834 * retval = 0;
28102835 */
@@ -2813,8 +2838,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 2 unmodified lines
28132838 _EMIT6(0xd707f000 | tjit->retval_off,
28142839 0xf000 | tjit->retval_off);
28152840
2816− for (i = 0; i < fmod_ret->nr_links; i++) {
2817− if (invoke_bpf_prog(tjit, m, fmod_ret->links[i], true))
2841+ for (i = 0; i < fmod_ret->nr_nodes; i++) {
2842+ if (invoke_bpf_prog(tjit, m, fmod_ret->nodes[i], true))
28182843 return -EINVAL;
28192844
28202845 /*
@@ -2939,7 +2964,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 118 unmodified lines
29392964 }
29402965
29412966 int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
2942− struct bpf_tramp_links *tlinks, void *orig_call)
2967+ struct bpf_tramp_nodes *tnodes, void *orig_call)
29432968 {
29442969 struct bpf_tramp_image im;
29452970 struct bpf_tramp_jit tjit;
@@ -2948,14 +2973,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 2 unmodified lines
29482973 memset(&tjit, 0, sizeof(tjit));
29492974
29502975 ret = __arch_prepare_bpf_trampoline(&im, &tjit, m, flags,
2951− tlinks, orig_call);
2976+ tnodes, orig_call);
29522977
29532978 return ret < 0 ? ret : tjit.common.prg;
29542979 }
29552980
29562981 int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image,
29572982 void *image_end, const struct btf_func_model *m,
2958− u32 flags, struct bpf_tramp_links *tlinks,
2983+ u32 flags, struct bpf_tramp_nodes *tnodes,
29592984 void *func_addr)
29602985 {
29612986 struct bpf_tramp_jit tjit;
@@ -2964,7 +2989,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, · 2 unmodified lines
29642989 /* Compute offsets, check whether the code fits. */
29652990 memset(&tjit, 0, sizeof(tjit));
29662991 ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags,
2967− tlinks, func_addr);
2992+ tnodes, func_addr);
29682993
29692994 if (ret < 0)
29702995 return ret;
@@ -2978,7 +3003,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, · 7 unmodified lines
29783003 tjit.common.prg = 0;
29793004 tjit.common.prg_buf = image;
29803005 ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags,
2981− tlinks, func_addr);
3006+ tnodes, func_addr);
29823007
29833008 return ret < 0 ? ret : tjit.common.prg;
29843009 }
@@ -3057,3 +3082,15 @@ bool bpf_jit_supports_timed_may_goto(void) · 72 unmodified lines
30573082 {
30583083 return true;
30593084 }
3085+
3086+bool bpf_jit_inlines_helper_call(s32 imm)
3087+{
3088+ switch (imm) {
3089+ case BPF_FUNC_get_smp_processor_id:
3090+ case BPF_FUNC_get_current_task:
3091+ case BPF_FUNC_get_current_task_btf:
3092+ return true;
3093+ default:
3094+ return false;
3095+ }
3096+}
arch/x86/include/asm/pgtable.h+12−0
@@ -1284,6 +1284,18 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, · 1283 unmodified lines
12841284 } while (!try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte));
12851285 }
12861286
1287+/*
1288+ * Note: strictly-zero compare is narrower than pte_none(), but the gap is
1289+ * harmless: _PAGE_DIRTY and _PAGE_ACCESSED aren't set on untouched kernel PTEs.
1290+ */
1291+static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte)
1292+{
1293+ pte_t old_pte = __pte(0);
1294+
1295+ return try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte);
1296+}
1297+#define ptep_try_set ptep_try_set
1298+
12871299 #define flush_tlb_fix_spurious_fault(vma, address, ptep) do { } while (0)
12881300
12891301 #define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS
arch/x86/mm/fault.c+8−4
@@ -8,6 +8,7 @@ · 7 unmodified lines
88 #include <linux/sched/task_stack.h> /* task_stack_*(), ... */
99 #include <linux/kdebug.h> /* oops_begin/end, ... */
1010 #include <linux/memblock.h> /* max_low_pfn */
11+#include <linux/bpf_defs.h> /* bpf_arena_handle_page_fault */
1112 #include <linux/kfence.h> /* kfence_handle_page_fault */
1213 #include <linux/kprobes.h> /* NOKPROBE_SYMBOL, ... */
1314 #include <linux/mmiotrace.h> /* kmmio_handler, ... */
@@ -693,10 +694,13 @@ page_fault_oops(struct pt_regs *regs, unsigned long error_code, · 679 unmodified lines
693694 if (IS_ENABLED(CONFIG_EFI))
694695 efi_crash_gracefully_on_page_fault(address, regs);
695696
696− /* Only not-present faults should be handled by KFENCE. */
697− if (!(error_code & X86_PF_PROT) &&
698− kfence_handle_page_fault(address, error_code & X86_PF_WRITE, regs))
699− return;
697+ /* Only not-present faults should be handled by KFENCE or BPF arena. */
698+ if (!(error_code & X86_PF_PROT)) {
699+ if (kfence_handle_page_fault(address, error_code & X86_PF_WRITE, regs))
700+ return;
701+ if (bpf_arena_handle_page_fault(address, error_code & X86_PF_WRITE, regs->ip))
702+ return;
703+ }
700704
701705 oops:
702706 /*
arch/x86/net/bpf_jit_comp.c+172−33
@@ -10,6 +10,7 @@ · 9 unmodified lines
1010 #include <linux/if_vlan.h>
1111 #include <linux/bitfield.h>
1212 #include <linux/bpf.h>
13+#include <linux/bpf_verifier.h>
1314 #include <linux/memory.h>
1415 #include <linux/sort.h>
1516 #include <asm/extable.h>
@@ -390,6 +391,34 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) · 374 unmodified lines
390391 *pprog = prog;
391392 }
392393
394+/* add rsp, depth */
395+static void emit_add_rsp(u8 **pprog, u16 depth)
396+{
397+ u8 *prog = *pprog;
398+
399+ if (!depth)
400+ return;
401+ if (is_imm8(depth))
402+ EMIT4(0x48, 0x83, 0xC4, depth); /* add rsp, imm8 */
403+ else
404+ EMIT3_off32(0x48, 0x81, 0xC4, depth); /* add rsp, imm32 */
405+ *pprog = prog;
406+}
407+
408+/* sub rsp, depth */
409+static void emit_sub_rsp(u8 **pprog, u16 depth)
410+{
411+ u8 *prog = *pprog;
412+
413+ if (!depth)
414+ return;
415+ if (is_imm8(depth))
416+ EMIT4(0x48, 0x83, 0xEC, depth); /* sub rsp, imm8 */
417+ else
418+ EMIT3_off32(0x48, 0x81, 0xEC, depth); /* sub rsp, imm32 */
419+ *pprog = prog;
420+}
421+
393422 static void emit_nops(u8 **pprog, int len)
394423 {
395424 u8 *prog = *pprog;
@@ -1659,21 +1688,47 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 1263 unmodified lines
16591688 bool seen_exit = false;
16601689 u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY];
16611690 void __percpu *priv_frame_ptr = NULL;
1691+ u16 out_stack_arg_cnt, outgoing_rsp;
16621692 u64 arena_vm_start, user_vm_start;
16631693 void __percpu *priv_stack_ptr;
16641694 int i, excnt = 0;
16651695 int ilen, proglen = 0;
16661696 u8 *ip, *prog = temp;
16671697 u32 stack_depth;
1698+ int callee_saved_size;
1699+ s32 outgoing_arg_base;
16681700 int err;
16691701
16701702 stack_depth = bpf_prog->aux->stack_depth;
1703+ out_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog);
16711704 priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
16721705 if (priv_stack_ptr) {
16731706 priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
16741707 stack_depth = 0;
16751708 }
16761709
1710+ /*
1711+ * Follow x86-64 calling convention for both BPF-to-BPF and
1712+ * kfunc calls:
1713+ * - Arg 6 is passed in R9 register
1714+ * - Args 7+ are passed on the stack at [rsp]
1715+ *
1716+ * Incoming arg 6 is read from R9 (BPF r11+8 → MOV from R9).
1717+ * Incoming args 7+ are read from [rbp + 16], [rbp + 24], ...
1718+ * (BPF r11+16, r11+24, ... map directly with no offset change).
1719+ *
1720+ * tail_call_reachable is rejected by the verifier and priv_stack
1721+ * is disabled by the JIT when stack args exist, so R9 is always
1722+ * available.
1723+ *
1724+ * Stack layout (high to low):
1725+ * [rbp + 16 + ...] incoming stack args 7+ (from caller)
1726+ * [rbp + 8] return address
1727+ * [rbp] saved rbp
1728+ * [rbp - prog_stack] program stack
1729+ * [below] callee-saved regs
1730+ * [below] outgoing args 7+ (= rsp)
1731+ */
16771732 arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);
16781733 user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);
16791734
@@ -1700,6 +1755,44 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 20 unmodified lines
17001755 push_r12(&prog);
17011756 push_callee_regs(&prog, callee_regs_used);
17021757 }
1758+
1759+ /* Compute callee-saved register area size. */
1760+ callee_saved_size = 0;
1761+ if (bpf_prog->aux->exception_boundary || arena_vm_start)
1762+ callee_saved_size += 8; /* r12 */
1763+ if (bpf_prog->aux->exception_boundary) {
1764+ callee_saved_size += 4 * 8; /* rbx, r13, r14, r15 */
1765+ } else {
1766+ int j;
1767+
1768+ for (j = 0; j < 4; j++)
1769+ if (callee_regs_used[j])
1770+ callee_saved_size += 8;
1771+ }
1772+ /*
1773+ * Base offset from rbp for translating BPF outgoing args 7+
1774+ * to native offsets. BPF uses negative offsets from r11
1775+ * (r11-8 for arg6, r11-16 for arg7, ...) while x86 uses
1776+ * positive offsets from rsp ([rsp+0] for arg7, [rsp+8] for
1777+ * arg8, ...). Arg 6 goes to R9 directly.
1778+ *
1779+ * The translation reverses direction:
1780+ * native_off = outgoing_arg_base - outgoing_rsp - bpf_off - 16
1781+ *
1782+ * Note that tail_call_reachable is guaranteed to be false when
1783+ * stack args exist, so tcc pushes need not be accounted for.
1784+ */
1785+ outgoing_arg_base = -(round_up(stack_depth, 8) + callee_saved_size);
1786+
1787+ /*
1788+ * Allocate outgoing stack arg area for args 7+ only.
1789+ * Arg 6 goes into r9 register, not on stack.
1790+ */
1791+ outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
1792+ if (bpf_prog->aux->exception_boundary)
1793+ bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp;
1794+ emit_sub_rsp(&prog, outgoing_rsp);
1795+
17031796 if (arena_vm_start)
17041797 emit_mov_imm64(&prog, X86_REG_R12,
17051798 arena_vm_start >> 32, (u32) arena_vm_start);
@@ -1721,7 +1814,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 15 unmodified lines
17211814 u8 b2 = 0, b3 = 0;
17221815 u8 *start_of_ldx;
17231816 s64 jmp_offset;
1724− s16 insn_off;
1817+ s32 insn_off;
17251818 u8 jmp_cond;
17261819 u8 *func;
17271820 int nops;
@@ -2134,12 +2227,27 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 406 unmodified lines
21342227 EMIT1(0xC7);
21352228 goto st;
21362229 case BPF_ST | BPF_MEM | BPF_DW:
2230+ if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
2231+ /* Arg 6: store immediate in r9 register */
2232+ emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31, (u32)imm32);
2233+ break;
2234+ }
21372235 EMIT2(add_1mod(0x48, dst_reg), 0xC7);
21382236
2139−st: if (is_imm8(insn->off))
2140− EMIT2(add_1reg(0x40, dst_reg), insn->off);
2237+st: insn_off = insn->off;
2238+ if (dst_reg == BPF_REG_PARAMS) {
2239+ /*
2240+ * Args 7+: reverse BPF negative offsets to
2241+ * x86 positive rsp offsets.
2242+ * BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ...
2243+ */
2244+ insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
2245+ dst_reg = BPF_REG_FP;
2246+ }
2247+ if (is_imm8(insn_off))
2248+ EMIT2(add_1reg(0x40, dst_reg), insn_off);
21412249 else
2142− EMIT1_off32(add_1reg(0x80, dst_reg), insn->off);
2250+ EMIT1_off32(add_1reg(0x80, dst_reg), insn_off);
21432251
21442252 EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code)));
21452253 break;
@@ -2149,7 +2257,17 @@ st: if (is_imm8(insn->off)) · 3 unmodified lines
21492257 case BPF_STX | BPF_MEM | BPF_H:
21502258 case BPF_STX | BPF_MEM | BPF_W:
21512259 case BPF_STX | BPF_MEM | BPF_DW:
2152− emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
2260+ if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
2261+ /* Arg 6: store register value in r9 */
2262+ EMIT_mov(X86_REG_R9, src_reg);
2263+ break;
2264+ }
2265+ insn_off = insn->off;
2266+ if (dst_reg == BPF_REG_PARAMS) {
2267+ insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
2268+ dst_reg = BPF_REG_FP;
2269+ }
2270+ emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
21532271 break;
21542272
21552273 case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
@@ -2248,6 +2366,19 @@ populate_extable: · 92 unmodified lines
22482366 case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:
22492367 case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
22502368 insn_off = insn->off;
2369+ if (src_reg == BPF_REG_PARAMS) {
2370+ if (insn_off == 8) {
2371+ /* Incoming arg 6: read from r9 */
2372+ EMIT_mov(dst_reg, X86_REG_R9);
2373+ break;
2374+ }
2375+ src_reg = BPF_REG_FP;
2376+ /*
2377+ * Incoming args 7+: native_off == bpf_off
2378+ * (r11+16 → [rbp+16], r11+24 → [rbp+24], ...)
2379+ * No offset adjustment needed.
2380+ */
2381+ }
22512382
22522383 if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||
22532384 BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {
@@ -2736,6 +2867,8 @@ emit_jmp: · 482 unmodified lines
27362867 if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
27372868 return -EINVAL;
27382869 }
2870+ /* Deallocate outgoing args 7+ area. */
2871+ emit_add_rsp(&prog, outgoing_rsp);
27392872 if (bpf_prog->aux->exception_boundary) {
27402873 pop_callee_regs(&prog, all_callee_regs_used);
27412874 pop_r12(&prog);
@@ -2971,15 +3104,15 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog, · 229 unmodified lines
29713104 }
29723105
29733106 static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
2974− struct bpf_tramp_link *l, int stack_size,
3107+ struct bpf_tramp_node *node, int stack_size,
29753108 int run_ctx_off, bool save_ret,
29763109 void *image, void *rw_image)
29773110 {
29783111 u8 *prog = *pprog;
29793112 u8 *jmp_insn;
29803113 int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
2981− struct bpf_prog *p = l->link.prog;
2982− u64 cookie = l->cookie;
3114+ struct bpf_prog *p = node->link->prog;
3115+ u64 cookie = node->cookie;
29833116
29843117 /* mov rdi, cookie */
29853118 emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie);
@@ -3086,7 +3219,7 @@ static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond) · 100 unmodified lines
30863219 }
30873220
30883221 static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
3089− struct bpf_tramp_links *tl, int stack_size,
3222+ struct bpf_tramp_nodes *tl, int stack_size,
30903223 int run_ctx_off, int func_meta_off, bool save_ret,
30913224 void *image, void *rw_image, u64 func_meta,
30923225 int cookie_off)
@@ -3094,13 +3227,13 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, · 1 unmodified line
30943227 int i, cur_cookie = (cookie_off - stack_size) / 8;
30953228 u8 *prog = *pprog;
30963229
3097− for (i = 0; i < tl->nr_links; i++) {
3098− if (tl->links[i]->link.prog->call_session_cookie) {
3230+ for (i = 0; i < tl->nr_nodes; i++) {
3231+ if (tl->nodes[i]->link->prog->call_session_cookie) {
30993232 emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off,
31003233 func_meta | (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT));
31013234 cur_cookie--;
31023235 }
3103− if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size,
3236+ if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size,
31043237 run_ctx_off, save_ret, image, rw_image))
31053238 return -EINVAL;
31063239 }
@@ -3109,7 +3242,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, · 2 unmodified lines
31093242 }
31103243
31113244 static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
3112− struct bpf_tramp_links *tl, int stack_size,
3245+ struct bpf_tramp_nodes *tl, int stack_size,
31133246 int run_ctx_off, u8 **branches,
31143247 void *image, void *rw_image)
31153248 {
@@ -3121,8 +3254,8 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, · 5 unmodified lines
31213254 */
31223255 emit_mov_imm32(&prog, false, BPF_REG_0, 0);
31233256 emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
3124− for (i = 0; i < tl->nr_links; i++) {
3125− if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size, run_ctx_off, true,
3257+ for (i = 0; i < tl->nr_nodes; i++) {
3258+ if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true,
31263259 image, rw_image))
31273260 return -EINVAL;
31283261
@@ -3213,14 +3346,14 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, · 84 unmodified lines
32133346 static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,
32143347 void *rw_image_end, void *image,
32153348 const struct btf_func_model *m, u32 flags,
3216− struct bpf_tramp_links *tlinks,
3349+ struct bpf_tramp_nodes *tnodes,
32173350 void *func_addr)
32183351 {
32193352 int i, ret, nr_regs = m->nr_args, stack_size = 0;
32203353 int regs_off, func_meta_off, ip_off, run_ctx_off, arg_stack_off, rbx_off;
3221− struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
3222− struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
3223− struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
3354+ struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
3355+ struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
3356+ struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
32243357 void *orig_call = func_addr;
32253358 int cookie_off, cookie_cnt;
32263359 u8 **branches = NULL;
@@ -3292,7 +3425,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 65 unmodified lines
32923425
32933426 ip_off = stack_size;
32943427
3295− cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
3428+ cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
32963429 /* room for session cookies */
32973430 stack_size += cookie_cnt * 8;
32983431 cookie_off = stack_size;
@@ -3385,7 +3518,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 86 unmodified lines
33853518 }
33863519 }
33873520
3388− if (bpf_fsession_cnt(tlinks)) {
3521+ if (bpf_fsession_cnt(tnodes)) {
33893522 /* clear all the session cookies' value */
33903523 for (int i = 0; i < cookie_cnt; i++)
33913524 emit_store_stack_imm64(&prog, BPF_REG_0, -cookie_off + 8 * i, 0);
@@ -3393,15 +3526,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 1 unmodified line
33933526 emit_store_stack_imm64(&prog, BPF_REG_0, -8, 0);
33943527 }
33953528
3396− if (fentry->nr_links) {
3529+ if (fentry->nr_nodes) {
33973530 if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off,
33983531 flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image,
33993532 func_meta, cookie_off))
34003533 return -EINVAL;
34013534 }
34023535
3403− if (fmod_ret->nr_links) {
3404− branches = kcalloc(fmod_ret->nr_links, sizeof(u8 *),
3536+ if (fmod_ret->nr_nodes) {
3537+ branches = kcalloc(fmod_ret->nr_nodes, sizeof(u8 *),
34053538 GFP_KERNEL);
34063539 if (!branches)
34073540 return -ENOMEM;
@@ -3440,7 +3573,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 32 unmodified lines
34403573 emit_nops(&prog, X86_PATCH_SIZE);
34413574 }
34423575
3443− if (fmod_ret->nr_links) {
3576+ if (fmod_ret->nr_nodes) {
34443577 /* From Intel 64 and IA-32 Architectures Optimization
34453578 * Reference Manual, 3.4.1.4 Code Alignment, Assembly/Compiler
34463579 * Coding Rule 11: All branch targets should be 16-byte
@@ -3450,7 +3583,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 3 unmodified lines
34503583 /* Update the branches saved in invoke_bpf_mod_ret with the
34513584 * aligned address of do_fexit.
34523585 */
3453− for (i = 0; i < fmod_ret->nr_links; i++) {
3586+ for (i = 0; i < fmod_ret->nr_nodes; i++) {
34543587 emit_cond_near_jump(&branches[i], image + (prog - (u8 *)rw_image),
34553588 image + (branches[i] - (u8 *)rw_image), X86_JNE);
34563589 }
@@ -3458,10 +3591,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 1 unmodified line
34583591
34593592 /* set the "is_return" flag for fsession */
34603593 func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
3461− if (bpf_fsession_cnt(tlinks))
3594+ if (bpf_fsession_cnt(tnodes))
34623595 emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta);
34633596
3464− if (fexit->nr_links) {
3597+ if (fexit->nr_nodes) {
34653598 if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off,
34663599 false, image, rw_image, func_meta, cookie_off)) {
34673600 ret = -EINVAL;
@@ -3535,7 +3668,7 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) · 67 unmodified lines
35353668
35363669 int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
35373670 const struct btf_func_model *m, u32 flags,
3538− struct bpf_tramp_links *tlinks,
3671+ struct bpf_tramp_nodes *tnodes,
35393672 void *func_addr)
35403673 {
35413674 void *rw_image, *tmp;
@@ -3550,7 +3683,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i · 8 unmodified lines
35503683 return -ENOMEM;
35513684
35523685 ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,
3553− flags, tlinks, func_addr);
3686+ flags, tnodes, func_addr);
35543687 if (ret < 0)
35553688 goto out;
35563689
@@ -3563,7 +3696,7 @@ out: · 6 unmodified lines
35633696 }
35643697
35653698 int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
3566− struct bpf_tramp_links *tlinks, void *func_addr)
3699+ struct bpf_tramp_nodes *tnodes, void *func_addr)
35673700 {
35683701 struct bpf_tramp_image im;
35693702 void *image;
@@ -3581,7 +3714,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 11 unmodified lines
35813714 return -ENOMEM;
35823715
35833716 ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image,
3584− m, flags, tlinks, func_addr);
3717+ m, flags, tnodes, func_addr);
35853718 bpf_jit_free_exec(image);
35863719 return ret;
35873720 }
@@ -3793,7 +3926,8 @@ skip_init_addrs: · 205 unmodified lines
37933926 for (pass = 0; pass < MAX_PASSES || image; pass++) {
37943927 if (!padding && pass >= PADDING_PASSES)
37953928 padding = true;
3796− proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen, &ctx, padding);
3929+ proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen,
3930+ &ctx, padding);
37973931 if (proglen <= 0) {
37983932 out_image:
37993933 image = NULL;
@@ -3910,6 +4044,11 @@ bool bpf_jit_supports_kfunc_call(void) · 110 unmodified lines
39104044 return true;
39114045 }
39124046
4047+bool bpf_jit_supports_stack_args(void)
4048+{
4049+ return true;
4050+}
4051+
39134052 void *bpf_arch_text_copy(void *dst, void *src, size_t len)
39144053 {
39154054 if (text_poke_copy(dst, src, len) == NULL)
drivers/base/power/power.h+7−0
@@ -168,3 +168,10 @@ static inline void device_pm_init(struct device *dev) · 167 unmodified lines
168168 device_pm_sleep_init(dev);
169169 pm_runtime_init(dev);
170170 }
171+
172+#ifdef CONFIG_BPF_SYSCALL
173+struct bpf_ws_lock { };
174+struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void);
175+void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock);
176+void *bpf_wakeup_sources_get_head(void);
177+#endif
drivers/base/power/wakeup.c+69−2
@@ -1168,11 +1168,78 @@ static const struct file_operations wakeup_sources_stats_fops = { · 1167 unmodified lines
11681168 .release = seq_release_private,
11691169 };
11701170
1171−static int __init wakeup_sources_debugfs_init(void)
1171+#ifdef CONFIG_BPF_SYSCALL
1172+#include <linux/btf.h>
1173+
1174+__bpf_kfunc_start_defs();
1175+
1176+/**
1177+ * bpf_wakeup_sources_read_lock - Acquire the SRCU lock for wakeup sources
1178+ *
1179+ * The underlying SRCU lock returns an integer index. However, the BPF verifier
1180+ * requires a pointer (PTR_TO_BTF_ID) to strictly track the state of acquired
1181+ * resources using KF_ACQUIRE and KF_RELEASE semantics. We use an opaque
1182+ * structure pointer (struct bpf_ws_lock *) to satisfy the verifier while
1183+ * safely encoding the integer index within the pointer address itself.
1184+ *
1185+ * Return: An opaque pointer encoding the SRCU lock index + 1 (to avoid NULL).
1186+ */
1187+__bpf_kfunc struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void)
1188+{
1189+ return (struct bpf_ws_lock *)(long)(wakeup_sources_read_lock() + 1);
1190+}
1191+
1192+/**
1193+ * bpf_wakeup_sources_read_unlock - Release the SRCU lock for wakeup sources
1194+ * @lock: The opaque pointer returned by bpf_wakeup_sources_read_lock()
1195+ *
1196+ * The BPF verifier guarantees that @lock is a valid, unreleased pointer from
1197+ * the acquire function. We decode the pointer back into the integer SRCU index
1198+ * by subtracting 1 and release the lock.
1199+ */
1200+__bpf_kfunc void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock)
1201+{
1202+ wakeup_sources_read_unlock((int)(long)lock - 1);
1203+}
1204+
1205+/**
1206+ * bpf_wakeup_sources_get_head - Get the head of the wakeup sources list
1207+ *
1208+ * Return: The head of the wakeup sources list.
1209+ */
1210+__bpf_kfunc void *bpf_wakeup_sources_get_head(void)
1211+{
1212+ return &wakeup_sources;
1213+}
1214+
1215+__bpf_kfunc_end_defs();
1216+
1217+BTF_KFUNCS_START(wakeup_source_kfunc_ids)
1218+BTF_ID_FLAGS(func, bpf_wakeup_sources_read_lock, KF_ACQUIRE)
1219+BTF_ID_FLAGS(func, bpf_wakeup_sources_read_unlock, KF_RELEASE)
1220+BTF_ID_FLAGS(func, bpf_wakeup_sources_get_head)
1221+BTF_KFUNCS_END(wakeup_source_kfunc_ids)
1222+
1223+static const struct btf_kfunc_id_set wakeup_source_kfunc_set = {
1224+ .set = &wakeup_source_kfunc_ids,
1225+};
1226+
1227+static void __init wakeup_sources_bpf_init(void)
1228+{
1229+ if (register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &wakeup_source_kfunc_set))
1230+ pm_pr_dbg("Wakeup: failed to register BTF kfuncs\n");
1231+}
1232+#else
1233+static inline void wakeup_sources_bpf_init(void) {}
1234+#endif /* CONFIG_BPF_SYSCALL */
1235+
1236+static int __init wakeup_sources_init(void)
11721237 {
11731238 debugfs_create_file("wakeup_sources", 0444, NULL, NULL,
11741239 &wakeup_sources_stats_fops);
1240+ wakeup_sources_bpf_init();
1241+
11751242 return 0;
11761243 }
11771244
1178−postcore_initcall(wakeup_sources_debugfs_init);
1245+postcore_initcall(wakeup_sources_init);
drivers/net/ethernet/netronome/nfp/bpf/verifier.c+4−4
@@ -561,10 +561,10 @@ nfp_bpf_check_alu(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, · 560 unmodified lines
561561 const struct bpf_reg_state *dreg =
562562 cur_regs(env) + meta->insn.dst_reg;
563563
564− meta->umin_src = min(meta->umin_src, sreg->umin_value);
565− meta->umax_src = max(meta->umax_src, sreg->umax_value);
566− meta->umin_dst = min(meta->umin_dst, dreg->umin_value);
567− meta->umax_dst = max(meta->umax_dst, dreg->umax_value);
564+ meta->umin_src = min(meta->umin_src, reg_umin(sreg));
565+ meta->umax_src = max(meta->umax_src, reg_umax(sreg));
566+ meta->umin_dst = min(meta->umin_dst, reg_umin(dreg));
567+ meta->umax_dst = max(meta->umax_dst, reg_umax(dreg));
568568
569569 /* NFP supports u16 and u32 multiplication.
570570 *
fs/bpf_fs_kfuncs.c+1−1
@@ -200,7 +200,7 @@ int bpf_set_dentry_xattr_locked(struct dentry *dentry, const char *name__str, · 199 unmodified lines
200200 const struct bpf_dynptr *value_p, int flags)
201201 {
202202
203− struct bpf_dynptr_kern *value_ptr = (struct bpf_dynptr_kern *)value_p;
203+ const struct bpf_dynptr_kern *value_ptr = (struct bpf_dynptr_kern *)value_p;
204204 struct inode *inode = d_inode(dentry);
205205 const void *value;
206206 u32 value_len;
fs/verity/measure.c+2−2
@@ -118,9 +118,9 @@ __bpf_kfunc_start_defs(); · 117 unmodified lines
118118 *
119119 * Return: 0 on success, a negative value on error.
120120 */
121−__bpf_kfunc int bpf_get_fsverity_digest(struct file *file, struct bpf_dynptr *digest_p)
121+__bpf_kfunc int bpf_get_fsverity_digest(struct file *file, const struct bpf_dynptr *digest_p)
122122 {
123− struct bpf_dynptr_kern *digest_ptr = (struct bpf_dynptr_kern *)digest_p;
123+ const struct bpf_dynptr_kern *digest_ptr = (struct bpf_dynptr_kern *)digest_p;
124124 const struct inode *inode = file_inode(file);
125125 u32 dynptr_sz = __bpf_dynptr_size(digest_ptr);
126126 struct fsverity_digest *arg;
include/asm-generic/rqspinlock.h+11−3
@@ -243,12 +243,20 @@ static __always_inline void res_spin_unlock(rqspinlock_t *lock) · 242 unmodified lines
243243 ({ \
244244 int __ret; \
245245 local_irq_save(flags); \
246− __ret = raw_res_spin_lock(lock); \
247− if (__ret) \
246+ preempt_disable(); \
247+ __ret = res_spin_lock(lock); \
248+ if (__ret) { \
248249 local_irq_restore(flags); \
250+ preempt_enable(); \
251+ } \
249252 __ret; \
250253 })
251254
252−#define raw_res_spin_unlock_irqrestore(lock, flags) ({ raw_res_spin_unlock(lock); local_irq_restore(flags); })
255+#define raw_res_spin_unlock_irqrestore(lock, flags) \
256+ ({ \
257+ res_spin_unlock(lock); \
258+ local_irq_restore(flags); \
259+ preempt_enable(); \
260+ })
253261
254262 #endif /* __ASM_GENERIC_RQSPINLOCK_H */
include/linux/bpf-cgroup.h+3−2
@@ -421,7 +421,7 @@ int cgroup_bpf_prog_detach(const union bpf_attr *attr, · 420 unmodified lines
421421 enum bpf_prog_type ptype);
422422 int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
423423 int cgroup_bpf_prog_query(const union bpf_attr *attr,
424− union bpf_attr __user *uattr);
424+ union bpf_attr __user *uattr, u32 uattr_size);
425425
426426 const struct bpf_func_proto *
427427 cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);
@@ -452,7 +452,8 @@ static inline int cgroup_bpf_link_attach(const union bpf_attr *attr, · 24 unmodified lines
452452 }
453453
454454 static inline int cgroup_bpf_prog_query(const union bpf_attr *attr,
455− union bpf_attr __user *uattr)
455+ union bpf_attr __user *uattr,
456+ u32 uattr_size)
456457 {
457458 return -EINVAL;
458459 }
include/linux/bpf.h+236−46
@@ -6,6 +6,7 @@ · 5 unmodified lines
66
77 #include <uapi/linux/bpf.h>
88 #include <uapi/linux/filter.h>
9+#include <linux/bpf_defs.h>
910
1011 #include <crypto/sha2.h>
1112 #include <linux/workqueue.h>
@@ -32,6 +33,8 @@ · 20 unmodified lines
3233 #include <linux/memcontrol.h>
3334 #include <linux/cfi.h>
3435 #include <linux/xattr.h>
36+#include <linux/key.h>
37+#include <linux/ftrace.h>
3538 #include <asm/rqspinlock.h>
3639
3740 struct bpf_verifier_env;
@@ -111,7 +114,7 @@ struct bpf_map_ops { · 73 unmodified lines
111114 long (*map_pop_elem)(struct bpf_map *map, void *value);
112115 long (*map_peek_elem)(struct bpf_map *map, void *value);
113116 void *(*map_lookup_percpu_elem)(struct bpf_map *map, void *key, u32 cpu);
114− int (*map_get_hash)(struct bpf_map *map, u32 hash_buf_size, void *hash_buf);
117+ int (*map_get_hash)(struct bpf_map *map);
115118
116119 /* funcs called by prog_array and perf_event_array map */
117120 void *(*map_fd_get_ptr)(struct bpf_map *map, struct file *map_file,
@@ -296,6 +299,7 @@ struct bpf_map_owner { · 178 unmodified lines
296299
297300 struct bpf_map {
298301 u8 sha[SHA256_DIGEST_SIZE];
302+ u32 excl;
299303 const struct bpf_map_ops *ops;
300304 struct bpf_map *inner_map_meta;
301305 #ifdef CONFIG_SECURITY
@@ -489,6 +493,35 @@ static inline bool btf_record_has_field(const struct btf_record *rec, enum btf_f · 187 unmodified lines
489493 return rec->field_mask & type;
490494 }
491495
496+static inline bool btf_field_is_nmi_safe(enum btf_field_type type)
497+{
498+ switch (type) {
499+ case BPF_SPIN_LOCK:
500+ case BPF_RES_SPIN_LOCK:
501+ case BPF_TIMER:
502+ case BPF_WORKQUEUE:
503+ case BPF_TASK_WORK:
504+ case BPF_KPTR_UNREF:
505+ case BPF_REFCOUNT:
506+ return true;
507+ default:
508+ return false;
509+ }
510+}
511+
512+static inline bool btf_record_has_nmi_unsafe_fields(const struct btf_record *rec)
513+{
514+ int i;
515+
516+ if (IS_ERR_OR_NULL(rec))
517+ return false;
518+ for (i = 0; i < rec->cnt; i++) {
519+ if (!btf_field_is_nmi_safe(rec->fields[i].type))
520+ return true;
521+ }
522+ return false;
523+}
524+
492525 static inline void bpf_obj_init(const struct btf_record *rec, void *obj)
493526 {
494527 int i;
@@ -618,6 +651,8 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, · 123 unmodified lines
618651 struct bpf_spin_lock *spin_lock);
619652 u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena);
620653 u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena);
654+u64 bpf_arena_map_kern_vm_start(struct bpf_map *map);
655+struct bpf_map *bpf_prog_arena(struct bpf_prog *prog);
621656 int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
622657
623658 struct bpf_offload_dev;
@@ -679,6 +714,8 @@ int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, · 55 unmodified lines
679714 void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
680715 u64 flags);
681716 void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt);
717+void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
718+ u64 flags);
682719 #else
683720 static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
684721 int node_id, u64 flags)
@@ -689,6 +726,12 @@ static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr · 4 unmodified lines
689726 static inline void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt)
690727 {
691728 }
729+
730+static inline void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
731+ int node_id, u64 flags)
732+{
733+ return NULL;
734+}
692735 #endif
693736
694737 extern const struct bpf_map_ops bpf_map_offload_ops;
@@ -1052,7 +1095,7 @@ struct bpf_insn_access_aux { · 357 unmodified lines
10521095 struct {
10531096 struct btf *btf;
10541097 u32 btf_id;
1055− u32 ref_obj_id;
1098+ u32 ref_id;
10561099 };
10571100 };
10581101 struct bpf_verifier_log *log; /* for verbose logs */
@@ -1152,6 +1195,11 @@ struct bpf_prog_offload { · 93 unmodified lines
11521195
11531196 /* The longest tracepoint has 12 args.
11541197 * See include/trace/bpf_probe.h
1198+ *
1199+ * Also reuse this macro for maximum number of arguments a BPF function
1200+ * or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
1201+ * stack arg slots. The JIT may map some stack arg slots to registers based
1202+ * on the native calling convention (e.g., arg 6 to R9 on x86-64).
11551203 */
11561204 #define MAX_BPF_FUNC_ARGS 12
11571205
@@ -1234,9 +1282,9 @@ enum { · 76 unmodified lines
12341282 #define BPF_TRAMP_COOKIE_INDEX_SHIFT 8
12351283 #define BPF_TRAMP_IS_RETURN_SHIFT 63
12361284
1237−struct bpf_tramp_links {
1238− struct bpf_tramp_link *links[BPF_MAX_TRAMP_LINKS];
1239− int nr_links;
1285+struct bpf_tramp_nodes {
1286+ struct bpf_tramp_node *nodes[BPF_MAX_TRAMP_LINKS];
1287+ int nr_nodes;
12401288 };
12411289
12421290 struct bpf_tramp_run_ctx;
@@ -1264,13 +1312,13 @@ struct bpf_tramp_run_ctx; · 21 unmodified lines
12641312 struct bpf_tramp_image;
12651313 int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
12661314 const struct btf_func_model *m, u32 flags,
1267− struct bpf_tramp_links *tlinks,
1315+ struct bpf_tramp_nodes *tnodes,
12681316 void *func_addr);
12691317 void *arch_alloc_bpf_trampoline(unsigned int size);
12701318 void arch_free_bpf_trampoline(void *image, unsigned int size);
12711319 int __must_check arch_protect_bpf_trampoline(void *image, unsigned int size);
12721320 int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
1273− struct bpf_tramp_links *tlinks, void *func_addr);
1321+ struct bpf_tramp_nodes *tnodes, void *func_addr);
12741322
12751323 u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog,
12761324 struct bpf_tramp_run_ctx *run_ctx);
@@ -1336,8 +1384,6 @@ struct bpf_trampoline { · 59 unmodified lines
13361384 /* hlist for trampoline_ip_table */
13371385 struct hlist_node hlist_ip;
13381386 struct ftrace_ops *fops;
1339− /* serializes access to fields of this trampoline */
1340− struct mutex mutex;
13411387 refcount_t refcnt;
13421388 u32 flags;
13431389 u64 key;
@@ -1358,6 +1404,11 @@ struct bpf_trampoline { · 14 unmodified lines
13581404 int progs_cnt[BPF_TRAMP_MAX];
13591405 /* Executable image of trampoline */
13601406 struct bpf_tramp_image *cur_image;
1407+ /* Used as temporary old image storage for multi_attach */
1408+ struct {
1409+ struct bpf_tramp_image *old_image;
1410+ u32 old_flags;
1411+ } multi_attach;
13611412 };
13621413
13631414 struct bpf_attach_target_info {
@@ -1455,11 +1506,13 @@ static inline int bpf_dynptr_check_off_len(const struct bpf_dynptr_kern *ptr, u6 · 91 unmodified lines
14551506 return 0;
14561507 }
14571508
1509+struct bpf_tracing_multi_link;
1510+
14581511 #ifdef CONFIG_BPF_JIT
1459−int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
1512+int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
14601513 struct bpf_trampoline *tr,
14611514 struct bpf_prog *tgt_prog);
1462−int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
1515+int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
14631516 struct bpf_trampoline *tr,
14641517 struct bpf_prog *tgt_prog);
14651518 struct bpf_trampoline *bpf_trampoline_get(u64 key,
@@ -1467,6 +1520,11 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, · 1 unmodified line
14671520 void bpf_trampoline_put(struct bpf_trampoline *tr);
14681521 int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
14691522
1523+int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
1524+ struct bpf_tracing_multi_link *link);
1525+int bpf_trampoline_multi_detach(struct bpf_prog *prog,
1526+ struct bpf_tracing_multi_link *link);
1527+
14701528 /*
14711529 * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn
14721530 * indirection with a direct call to the bpf program. If the architecture does
@@ -1544,14 +1602,15 @@ void bpf_jit_uncharge_modmem(u32 size); · 71 unmodified lines
15441602 bool bpf_prog_has_trampoline(const struct bpf_prog *prog);
15451603 bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog,
15461604 int insn_idx);
1605+u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog);
15471606 #else
1548−static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
1607+static inline int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
15491608 struct bpf_trampoline *tr,
15501609 struct bpf_prog *tgt_prog)
15511610 {
15521611 return -ENOTSUPP;
15531612 }
1554−static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
1613+static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
15551614 struct bpf_trampoline *tr,
15561615 struct bpf_prog *tgt_prog)
15571616 {
@@ -1578,6 +1637,16 @@ static inline bool bpf_prog_has_trampoline(const struct bpf_prog *prog) · 20 unmodified lines
15781637 {
15791638 return false;
15801639 }
1640+static inline int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
1641+ struct bpf_tracing_multi_link *link)
1642+{
1643+ return -ENOTSUPP;
1644+}
1645+static inline int bpf_trampoline_multi_detach(struct bpf_prog *prog,
1646+ struct bpf_tracing_multi_link *link)
1647+{
1648+ return -ENOTSUPP;
1649+}
15811650 #endif
15821651
15831652 struct bpf_func_info_aux {
@@ -1615,7 +1684,7 @@ struct bpf_ctx_arg_aux { · 31 unmodified lines
16151684 enum bpf_reg_type reg_type;
16161685 struct btf *btf;
16171686 u32 btf_id;
1618− u32 ref_obj_id;
1687+ u32 ref_id;
16191688 bool refcounted;
16201689 };
16211690
@@ -1657,6 +1726,19 @@ struct bpf_stream_stage { · 35 unmodified lines
16571726 int len;
16581727 };
16591728
1729+enum bpf_sig_verdict {
1730+ BPF_SIG_UNSIGNED = 0,
1731+ BPF_SIG_VERIFIED,
1732+};
1733+
1734+enum bpf_sig_keyring {
1735+ BPF_SIG_KEYRING_NONE = 0,
1736+ BPF_SIG_KEYRING_BUILTIN,
1737+ BPF_SIG_KEYRING_SECONDARY,
1738+ BPF_SIG_KEYRING_PLATFORM,
1739+ BPF_SIG_KEYRING_USER,
1740+};
1741+
16601742 struct bpf_prog_aux {
16611743 atomic64_t refcnt;
16621744 u32 used_map_cnt;
@@ -1699,6 +1781,11 @@ struct bpf_prog_aux { · 36 unmodified lines
16991781 bool changes_pkt_data;
17001782 bool might_sleep;
17011783 bool kprobe_write_ctx;
1784+ struct {
1785+ s32 keyring_serial;
1786+ u8 keyring_type;
1787+ u8 verdict;
1788+ } sig;
17021789 u64 prog_array_member_cnt; /* counts how many times as member of prog_array */
17031790 struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */
17041791 struct bpf_arena *arena;
@@ -1731,6 +1818,7 @@ struct bpf_prog_aux { · 26 unmodified lines
17311818 struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
17321819 char name[BPF_OBJ_NAME_LEN];
17331820 u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
1821+ u16 stack_arg_sp_adjust;
17341822 #ifdef CONFIG_SECURITY
17351823 void *security;
17361824 #endif
@@ -1874,12 +1962,17 @@ struct bpf_link_ops { · 137 unmodified lines
18741962 __poll_t (*poll)(struct file *file, struct poll_table_struct *pts);
18751963 };
18761964
1877−struct bpf_tramp_link {
1878− struct bpf_link link;
1965+struct bpf_tramp_node {
1966+ struct bpf_link *link;
18791967 struct hlist_node tramp_hlist;
18801968 u64 cookie;
18811969 };
18821970
1971+struct bpf_tramp_link {
1972+ struct bpf_link link;
1973+ struct bpf_tramp_node node;
1974+};
1975+
18831976 struct bpf_shim_tramp_link {
18841977 struct bpf_tramp_link link;
18851978 struct bpf_trampoline *trampoline;
@@ -1887,13 +1980,31 @@ struct bpf_shim_tramp_link { · 1 unmodified line
18871980
18881981 struct bpf_tracing_link {
18891982 struct bpf_tramp_link link;
1983+ struct bpf_tramp_node fexit;
18901984 struct bpf_trampoline *trampoline;
18911985 struct bpf_prog *tgt_prog;
18921986 };
18931987
1894−struct bpf_fsession_link {
1895− struct bpf_tracing_link link;
1896− struct bpf_tramp_link fexit;
1988+struct bpf_tracing_multi_node {
1989+ struct bpf_tramp_node node;
1990+ struct bpf_trampoline *trampoline;
1991+ struct ftrace_func_entry entry;
1992+};
1993+
1994+struct bpf_tracing_multi_data {
1995+ struct ftrace_hash *unreg;
1996+ struct ftrace_hash *modify;
1997+ struct ftrace_hash *reg;
1998+ struct ftrace_func_entry *entry;
1999+};
2000+
2001+struct bpf_tracing_multi_link {
2002+ struct bpf_link link;
2003+ struct bpf_tracing_multi_data data;
2004+ u64 *cookies;
2005+ struct bpf_tramp_node *fexits;
2006+ int nodes_cnt;
2007+ struct bpf_tracing_multi_node nodes[] __counted_by(nodes_cnt);
18972008 };
18982009
18992010 struct bpf_raw_tp_link {
@@ -2079,6 +2190,12 @@ static inline void bpf_prog_put_recursion_context(struct bpf_prog *prog) · 179 unmodified lines
20792190 #endif
20802191 }
20812192
2193+static inline bool is_tracing_multi(enum bpf_attach_type type)
2194+{
2195+ return type == BPF_TRACE_FENTRY_MULTI || type == BPF_TRACE_FEXIT_MULTI ||
2196+ type == BPF_TRACE_FSESSION_MULTI;
2197+}
2198+
20822199 #if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL)
20832200 /* This macro helps developer to register a struct_ops type and generate
20842201 * type information correctly. Developers should use this macro to register
@@ -2099,8 +2216,8 @@ void bpf_struct_ops_put(const void *kdata); · 14 unmodified lines
20992216 int bpf_struct_ops_supported(const struct bpf_struct_ops *st_ops, u32 moff);
21002217 int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key,
21012218 void *value);
2102−int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
2103− struct bpf_tramp_link *link,
2219+int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes,
2220+ struct bpf_tramp_node *node,
21042221 const struct btf_func_model *model,
21052222 void *stub_func,
21062223 void **image, u32 *image_off,
@@ -2125,6 +2242,9 @@ int bpf_prog_assoc_struct_ops(struct bpf_prog *prog, struct bpf_map *map); · 18 unmodified lines
21252242 void bpf_prog_disassoc_struct_ops(struct bpf_prog *prog);
21262243 void *bpf_prog_get_assoc_struct_ops(const struct bpf_prog_aux *aux);
21272244 u32 bpf_struct_ops_id(const void *kdata);
2245+int bpf_struct_ops_for_each_prog(const void *kdata,
2246+ int (*cb)(struct bpf_prog *prog, void *data),
2247+ void *data);
21282248
21292249 #ifdef CONFIG_NET
21302250 /* Define it here to avoid the use of forward declaration */
@@ -2192,31 +2312,33 @@ static inline void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_op · 61 unmodified lines
21922312
21932313 #endif
21942314
2195−static inline int bpf_fsession_cnt(struct bpf_tramp_links *links)
2315+static inline int bpf_fsession_cnt(struct bpf_tramp_nodes *nodes)
21962316 {
2197− struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY];
2317+ struct bpf_tramp_nodes fentries = nodes[BPF_TRAMP_FENTRY];
21982318 int cnt = 0;
21992319
2200− for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) {
2201− if (fentries.links[i]->link.prog->expected_attach_type == BPF_TRACE_FSESSION)
2320+ for (int i = 0; i < nodes[BPF_TRAMP_FENTRY].nr_nodes; i++) {
2321+ if (fentries.nodes[i]->link->prog->expected_attach_type == BPF_TRACE_FSESSION)
2322+ cnt++;
2323+ if (fentries.nodes[i]->link->prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI)
22022324 cnt++;
22032325 }
22042326
22052327 return cnt;
22062328 }
22072329
2208−static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_link *link)
2330+static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_node *node)
22092331 {
2210− return link->link.prog->call_session_cookie;
2332+ return node->link->prog->call_session_cookie;
22112333 }
22122334
2213−static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_links *links)
2335+static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_nodes *nodes)
22142336 {
2215− struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY];
2337+ struct bpf_tramp_nodes fentries = nodes[BPF_TRAMP_FENTRY];
22162338 int cnt = 0;
22172339
2218− for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) {
2219− if (bpf_prog_calls_session_cookie(fentries.links[i]))
2340+ for (int i = 0; i < nodes[BPF_TRAMP_FENTRY].nr_nodes; i++) {
2341+ if (bpf_prog_calls_session_cookie(fentries.nodes[i]))
22202342 cnt++;
22212343 }
22222344
@@ -2598,6 +2720,7 @@ bool btf_record_equal(const struct btf_record *rec_a, const struct btf_record *r · 375 unmodified lines
25982720 void bpf_obj_free_timer(const struct btf_record *rec, void *obj);
25992721 void bpf_obj_free_workqueue(const struct btf_record *rec, void *obj);
26002722 void bpf_obj_free_task_work(const struct btf_record *rec, void *obj);
2723+void bpf_obj_cancel_fields(struct bpf_map *map, void *obj);
26012724 void bpf_obj_free_fields(const struct btf_record *rec, void *obj);
26022725 void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu);
26032726
@@ -2764,6 +2887,9 @@ void bpf_link_init(struct bpf_link *link, enum bpf_link_type type, · 160 unmodified lines
27642887 void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type,
27652888 const struct bpf_link_ops *ops, struct bpf_prog *prog,
27662889 enum bpf_attach_type attach_type, bool sleepable);
2890+void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type,
2891+ const struct bpf_link_ops *ops, struct bpf_prog *prog,
2892+ enum bpf_attach_type attach_type, u64 cookie);
27672893 int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer);
27682894 int bpf_link_settle(struct bpf_link_primer *primer);
27692895 void bpf_link_cleanup(struct bpf_link_primer *primer);
@@ -2917,7 +3043,9 @@ int bpf_check_uarg_tail_zero(bpfptr_t uaddr, size_t expected_size, · 147 unmodified lines
29173043 size_t actual_size);
29183044
29193045 /* verify correctness of eBPF program */
2920−int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size);
3046+struct bpf_log_attr;
3047+int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr,
3048+ struct bpf_log_attr *attr_log);
29213049
29223050 #ifndef CONFIG_BPF_JIT_ALWAYS_ON
29233051 int bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth);
@@ -3088,6 +3216,56 @@ void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr); · 164 unmodified lines
30883216 void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr);
30893217 void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip);
30903218
3219+static __always_inline u32
3220+bpf_prog_run_array_sleepable(const struct bpf_prog_array *array,
3221+ const void *ctx, bpf_prog_run_fn run_prog)
3222+{
3223+ const struct bpf_prog_array_item *item;
3224+ struct bpf_prog *prog;
3225+ struct bpf_run_ctx *old_run_ctx;
3226+ struct bpf_trace_run_ctx run_ctx;
3227+ u32 ret = 1;
3228+
3229+ if (unlikely(!array))
3230+ return ret;
3231+
3232+ migrate_disable();
3233+
3234+ run_ctx.is_uprobe = false;
3235+
3236+ old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
3237+ item = &array->items[0];
3238+ while ((prog = READ_ONCE(item->prog))) {
3239+ /* Skip dummy_bpf_prog placeholder (len == 0) */
3240+ if (unlikely(!prog->len)) {
3241+ item++;
3242+ continue;
3243+ }
3244+
3245+ if (unlikely(!bpf_prog_get_recursion_context(prog))) {
3246+ bpf_prog_inc_misses_counter(prog);
3247+ bpf_prog_put_recursion_context(prog);
3248+ item++;
3249+ continue;
3250+ }
3251+
3252+ run_ctx.bpf_cookie = item->bpf_cookie;
3253+
3254+ if (!prog->sleepable) {
3255+ guard(rcu)();
3256+ ret &= run_prog(prog, ctx);
3257+ } else {
3258+ ret &= run_prog(prog, ctx);
3259+ }
3260+
3261+ bpf_prog_put_recursion_context(prog);
3262+ item++;
3263+ }
3264+ bpf_reset_run_ctx(old_run_ctx);
3265+ migrate_enable();
3266+ return ret;
3267+}
3268+
30913269 #else /* !CONFIG_BPF_SYSCALL */
30923270 static inline struct bpf_prog *bpf_prog_get(u32 ufd)
30933271 {
@@ -3135,6 +3313,12 @@ static inline void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_ · 41 unmodified lines
31353313 {
31363314 }
31373315
3316+static inline void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type,
3317+ const struct bpf_link_ops *ops, struct bpf_prog *prog,
3318+ enum bpf_attach_type attach_type, u64 cookie)
3319+{
3320+}
3321+
31383322 static inline int bpf_link_prime(struct bpf_link *link,
31393323 struct bpf_link_primer *primer)
31403324 {
@@ -3626,15 +3810,25 @@ static inline int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, · 485 unmodified lines
36263810 #endif /* CONFIG_BPF_SYSCALL */
36273811 #endif /* defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL) */
36283812
3629−#if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL)
3813+#ifdef CONFIG_KEYS
3814+struct bpf_key {
3815+ struct key *key;
3816+ bool has_ref;
3817+};
3818+#endif /* CONFIG_KEYS */
36303819
3820+#if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL)
36313821 struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags);
36323822 struct bpf_key *bpf_lookup_system_key(u64 id);
36333823 void bpf_key_put(struct bpf_key *bkey);
3634−int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
3635− struct bpf_dynptr *sig_p,
3824+int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p,
3825+ const struct bpf_dynptr *sig_p,
36363826 struct bpf_key *trusted_keyring);
36373827
3828+static inline s32 bpf_key_serial(const struct bpf_key *key)
3829+{
3830+ return key->has_ref ? key->key->serial : 0;
3831+}
36383832 #else
36393833 static inline struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags)
36403834 {
@@ -3650,12 +3844,17 @@ static inline void bpf_key_put(struct bpf_key *bkey) · 9 unmodified lines
36503844 {
36513845 }
36523846
3653−static inline int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
3654− struct bpf_dynptr *sig_p,
3847+static inline int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p,
3848+ const struct bpf_dynptr *sig_p,
36553849 struct bpf_key *trusted_keyring)
36563850 {
36573851 return -EOPNOTSUPP;
36583852 }
3853+
3854+static inline s32 bpf_key_serial(const struct bpf_key *key)
3855+{
3856+ return 0;
3857+}
36593858 #endif /* defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) */
36603859
36613860 /* verifier prototypes for helper functions called from eBPF programs */
@@ -3931,15 +4130,6 @@ static inline void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) {} · 269 unmodified lines
39314130 static inline void bpf_cgroup_atype_put(int cgroup_atype) {}
39324131 #endif /* CONFIG_BPF_LSM */
39334132
3934−struct key;
3935−
3936−#ifdef CONFIG_KEYS
3937−struct bpf_key {
3938− struct key *key;
3939− bool has_ref;
3940−};
3941−#endif /* CONFIG_KEYS */
3942−
39434133 static inline bool type_is_alloc(u32 type)
39444134 {
39454135 return type & MEM_ALLOC;
include/linux/bpf_defs.h+19−0
@@ -0,0 +1,19 @@
1+/* SPDX-License-Identifier: GPL-2.0-or-later */
2+/*
3+ * Subset of bpf.h declarations, split out so files that need only these
4+ * declarations can avoid bpf.h's full include cost.
5+ */
6+#ifndef _LINUX_BPF_DEFS_H
7+#define _LINUX_BPF_DEFS_H
8+
9+#ifdef CONFIG_BPF_SYSCALL
10+bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, unsigned long fault_ip);
11+#else
12+static inline bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write,
13+ unsigned long fault_ip)
14+{
15+ return false;
16+}
17+#endif
18+
19+#endif /* _LINUX_BPF_DEFS_H */
include/linux/bpf_lsm.h+6−0
@@ -52,6 +52,7 @@ int bpf_set_dentry_xattr_locked(struct dentry *dentry, const char *name__str, · 51 unmodified lines
5252 const struct bpf_dynptr *value_p, int flags);
5353 int bpf_remove_dentry_xattr_locked(struct dentry *dentry, const char *name__str);
5454 bool bpf_lsm_has_d_inode_locked(const struct bpf_prog *prog);
55+bool bpf_lsm_hook_returns_errno(u32 btf_id);
5556
5657 #else /* !CONFIG_BPF_LSM */
5758
@@ -104,6 +105,11 @@ static inline bool bpf_lsm_has_d_inode_locked(const struct bpf_prog *prog) · 46 unmodified lines
104105 {
105106 return false;
106107 }
108+
109+static inline bool bpf_lsm_hook_returns_errno(u32 btf_id)
110+{
111+ return true;
112+}
107113 #endif /* CONFIG_BPF_LSM */
108114
109115 #endif /* _LINUX_BPF_LSM_H */
include/linux/bpf_types.h+2−0
@@ -134,6 +134,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_BLOOM_FILTER, bloom_filter_map_ops) · 133 unmodified lines
134134 BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops)
135135 BPF_MAP_TYPE(BPF_MAP_TYPE_ARENA, arena_map_ops)
136136 BPF_MAP_TYPE(BPF_MAP_TYPE_INSN_ARRAY, insn_array_map_ops)
137+BPF_MAP_TYPE(BPF_MAP_TYPE_RHASH, rhtab_map_ops)
137138
138139 BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)
139140 BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing)
@@ -155,3 +156,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_PERF_EVENT, perf) · 15 unmodified lines
155156 BPF_LINK_TYPE(BPF_LINK_TYPE_KPROBE_MULTI, kprobe_multi)
156157 BPF_LINK_TYPE(BPF_LINK_TYPE_STRUCT_OPS, struct_ops)
157158 BPF_LINK_TYPE(BPF_LINK_TYPE_UPROBE_MULTI, uprobe_multi)
159+BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING_MULTI, tracing_multi)
include/linux/bpf_verifier.h+209−92
@@ -8,6 +8,7 @@ · 7 unmodified lines
88 #include <linux/btf.h> /* for struct btf and btf_id() */
99 #include <linux/filter.h> /* for MAX_BPF_STACK */
1010 #include <linux/tnum.h>
11+#include <linux/cnum.h>
1112
1213 /* Maximum variable offset umax_value permitted when resolving memory accesses.
1314 * In practice this is far bigger than any realistic pointer offset; this limit
@@ -65,7 +66,6 @@ struct bpf_reg_state { · 51 unmodified lines
6566
6667 struct { /* for PTR_TO_MEM | PTR_TO_MEM_OR_NULL */
6768 u32 mem_size;
68− u32 dynptr_id; /* for dynptr slices */
6969 };
7070
7171 /* For dynptr stack slots */
@@ -120,14 +120,8 @@ struct bpf_reg_state { · 48 unmodified lines
120120 * These refer to the same value as var_off, not necessarily the actual
121121 * contents of the register.
122122 */
123− s64 smin_value; /* minimum possible (s64)value */
124− s64 smax_value; /* maximum possible (s64)value */
125− u64 umin_value; /* minimum possible (u64)value */
126− u64 umax_value; /* maximum possible (u64)value */
127− s32 s32_min_value; /* minimum possible (s32)value */
128− s32 s32_max_value; /* maximum possible (s32)value */
129− u32 u32_min_value; /* minimum possible (u32)value */
130− u32 u32_max_value; /* maximum possible (u32)value */
123+ struct cnum64 r64; /* 64-bit range as circular number */
124+ struct cnum32 r32; /* 32-bit range as circular number */
131125 /* For PTR_TO_PACKET, used to find other pointers with the same variable
132126 * offset, so they can share range knowledge.
133127 * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we
@@ -153,46 +147,14 @@ struct bpf_reg_state { · 19 unmodified lines
153147 #define BPF_ADD_CONST32 (1U << 30)
154148 #define BPF_ADD_CONST (BPF_ADD_CONST64 | BPF_ADD_CONST32)
155149 u32 id;
156− /* PTR_TO_SOCKET and PTR_TO_TCP_SOCK could be a ptr returned
157− * from a pointer-cast helper, bpf_sk_fullsock() and
158− * bpf_tcp_sock().
159− *
160− * Consider the following where "sk" is a reference counted
161− * pointer returned from "sk = bpf_sk_lookup_tcp();":
162− *
163− * 1: sk = bpf_sk_lookup_tcp();
164− * 2: if (!sk) { return 0; }
165− * 3: fullsock = bpf_sk_fullsock(sk);
166− * 4: if (!fullsock) { bpf_sk_release(sk); return 0; }
167− * 5: tp = bpf_tcp_sock(fullsock);
168− * 6: if (!tp) { bpf_sk_release(sk); return 0; }
169− * 7: bpf_sk_release(sk);
170− * 8: snd_cwnd = tp->snd_cwnd; // verifier will complain
171− *
172− * After bpf_sk_release(sk) at line 7, both "fullsock" ptr and
173− * "tp" ptr should be invalidated also. In order to do that,
174− * the reg holding "fullsock" and "sk" need to remember
175− * the original refcounted ptr id (i.e. sk_reg->id) in ref_obj_id
176− * such that the verifier can reset all regs which have
177− * ref_obj_id matching the sk_reg->id.
178− *
179− * sk_reg->ref_obj_id is set to sk_reg->id at line 1.
180− * sk_reg->id will stay as NULL-marking purpose only.
181− * After NULL-marking is done, sk_reg->id can be reset to 0.
182− *
183− * After "fullsock = bpf_sk_fullsock(sk);" at line 3,
184− * fullsock_reg->ref_obj_id is set to sk_reg->ref_obj_id.
185− *
186− * After "tp = bpf_tcp_sock(fullsock);" at line 5,
187− * tp_reg->ref_obj_id is set to fullsock_reg->ref_obj_id
188− * which is the same as sk_reg->ref_obj_id.
189− *
190− * From the verifier perspective, if sk, fullsock and tp
191− * are not NULL, they are the same ptr with different
192− * reg->type. In particular, bpf_sk_release(tp) is also
193− * allowed and has the same effect as bpf_sk_release(sk).
150+ /*
151+ * Tracks the parent object this register was derived from.
152+ * Used for cascading invalidation: when the parent object is
153+ * released or invalidated, all registers with matching parent_id
154+ * are also invalidated. For example, a slice from bpf_dynptr_data()
155+ * gets parent_id set to the dynptr's id.
194156 */
195− u32 ref_obj_id;
157+ u32 parent_id;
196158 /* Inside the callee two registers can be both PTR_TO_STACK like
197159 * R1=fp-8 and R2=fp-8, but one of them points to this function stack
198160 * while another to the caller's stack. To differentiate them 'frameno'
@@ -209,6 +171,66 @@ struct bpf_reg_state { · 10 unmodified lines
209171 bool precise;
210172 };
211173
174+static inline s64 reg_smin(const struct bpf_reg_state *reg)
175+{
176+ return cnum64_smin(reg->r64);
177+}
178+
179+static inline s64 reg_smax(const struct bpf_reg_state *reg)
180+{
181+ return cnum64_smax(reg->r64);
182+}
183+
184+static inline u64 reg_umin(const struct bpf_reg_state *reg)
185+{
186+ return cnum64_umin(reg->r64);
187+}
188+
189+static inline u64 reg_umax(const struct bpf_reg_state *reg)
190+{
191+ return cnum64_umax(reg->r64);
192+}
193+
194+static inline s32 reg_s32_min(const struct bpf_reg_state *reg)
195+{
196+ return cnum32_smin(reg->r32);
197+}
198+
199+static inline s32 reg_s32_max(const struct bpf_reg_state *reg)
200+{
201+ return cnum32_smax(reg->r32);
202+}
203+
204+static inline u32 reg_u32_min(const struct bpf_reg_state *reg)
205+{
206+ return cnum32_umin(reg->r32);
207+}
208+
209+static inline u32 reg_u32_max(const struct bpf_reg_state *reg)
210+{
211+ return cnum32_umax(reg->r32);
212+}
213+
214+static inline void reg_set_srange32(struct bpf_reg_state *reg, s32 smin, s32 smax)
215+{
216+ reg->r32 = cnum32_from_srange(smin, smax);
217+}
218+
219+static inline void reg_set_urange32(struct bpf_reg_state *reg, u32 umin, u32 umax)
220+{
221+ reg->r32 = cnum32_from_urange(umin, umax);
222+}
223+
224+static inline void reg_set_srange64(struct bpf_reg_state *reg, s64 smin, s64 smax)
225+{
226+ reg->r64 = cnum64_from_srange(smin, smax);
227+}
228+
229+static inline void reg_set_urange64(struct bpf_reg_state *reg, u64 umin, u64 umax)
230+{
231+ reg->r64 = cnum64_from_urange(umin, umax);
232+}
233+
212234 enum bpf_stack_slot_type {
213235 STACK_INVALID, /* nothing was stored in this stack slot */
214236 STACK_SPILL, /* register spilled into stack */
@@ -309,10 +331,14 @@ struct bpf_reference_state { · 94 unmodified lines
309331 * is used purely to inform the user of a reference leak.
310332 */
311333 int insn_idx;
312− /* Use to keep track of the source object of a lock, to ensure
313− * it matches on unlock.
314− */
315− void *ptr;
334+ union {
335+ /* For REF_TYPE_PTR */
336+ int parent_id;
337+ /* Use to keep track of the source object of a lock, to ensure
338+ * it matches on unlock.
339+ */
340+ void *ptr;
341+ };
316342 };
317343
318344 struct bpf_retval_range {
@@ -347,6 +373,7 @@ struct bpf_func_state { · 28 unmodified lines
347373 bool in_callback_fn;
348374 bool in_async_callback_fn;
349375 bool in_exception_callback_fn;
376+ bool no_stack_arg_load;
350377 /* For callback calling functions that limit number of possible
351378 * callback executions (e.g. bpf_loop) keeps track of current
352379 * simulated iteration number.
@@ -372,46 +399,49 @@ struct bpf_func_state { · 19 unmodified lines
372399 * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.
373400 */
374401 int allocated_stack;
402+
403+ u16 out_stack_arg_cnt; /* Number of outgoing on-stack argument slots */
404+ struct bpf_reg_state *stack_arg_regs; /* Outgoing on-stack arguments */
375405 };
376406
377−#define MAX_CALL_FRAMES 8
407+#define MAX_CALL_FRAMES 16
378408
379−/* instruction history flags, used in bpf_jmp_history_entry.flags field */
409+/* instruction history flags, used in bpf_jmp_history_entry.flags field.
410+ * Frame number and SPI are stored in dedicated fields of bpf_jmp_history_entry.
411+ */
380412 enum {
381− /* instruction references stack slot through PTR_TO_STACK register;
382− * we also store stack's frame number in lower 3 bits (MAX_CALL_FRAMES is 8)
383− * and accessed stack slot's index in next 6 bits (MAX_BPF_STACK is 512,
384− * 8 bytes per slot, so slot index (spi) is [0, 63])
385− */
386− INSN_F_FRAMENO_MASK = 0x7, /* 3 bits */
387−
388− INSN_F_SPI_MASK = 0x3f, /* 6 bits */
389− INSN_F_SPI_SHIFT = 3, /* shifted 3 bits to the left */
413+ INSN_F_STACK_ACCESS = BIT(0),
390414
391− INSN_F_STACK_ACCESS = BIT(9),
415+ INSN_F_DST_REG_STACK = BIT(1), /* dst_reg is PTR_TO_STACK */
416+ INSN_F_SRC_REG_STACK = BIT(2), /* src_reg is PTR_TO_STACK */
392417
393− INSN_F_DST_REG_STACK = BIT(10), /* dst_reg is PTR_TO_STACK */
394− INSN_F_SRC_REG_STACK = BIT(11), /* src_reg is PTR_TO_STACK */
395− /* total 12 bits are used now. */
418+ INSN_F_STACK_ARG_ACCESS = BIT(3),
396419 };
397420
398−static_assert(INSN_F_FRAMENO_MASK + 1 >= MAX_CALL_FRAMES);
399−static_assert(INSN_F_SPI_MASK + 1 >= MAX_BPF_STACK / 8);
400−
401421 struct bpf_jmp_history_entry {
402− u32 idx;
403422 /* insn idx can't be bigger than 1 million */
423+ u32 idx : 20;
424+ u32 frame : 4; /* stack access frame number */
425+ u32 spi : 6; /* stack slot index (0..63) */
426+ u32 : 2;
404427 u32 prev_idx : 20;
405428 /* special INSN_F_xxx flags */
406− u32 flags : 12;
407− /* additional registers that need precision tracking when this
408− * jump is backtracked, vector of six 10-bit records
429+ u32 flags : 4;
430+ u32 : 8;
431+ /*
432+ * additional registers that need precision tracking when this
433+ * jump is backtracked, vector of five 11-bit records
409434 */
410435 u64 linked_regs;
411436 };
412437
413−/* Maximum number of register states that can exist at once */
414−#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE) * MAX_CALL_FRAMES)
438+static_assert(MAX_CALL_FRAMES <= (1 << 4));
439+static_assert(MAX_BPF_STACK / 8 <= (1 << 6));
440+
441+/* Maximum number of bpf_reg_state objects that can exist at once */
442+#define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
443+#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE + \
444+ MAX_STACK_ARG_SLOTS) * MAX_CALL_FRAMES)
415445 struct bpf_verifier_state {
416446 /* call stack tracking */
417447 struct bpf_func_state *frame[MAX_CALL_FRAMES];
@@ -497,10 +527,23 @@ struct bpf_verifier_state { · 79 unmodified lines
497527 u32 may_goto_depth;
498528 };
499529
500−#define bpf_get_spilled_reg(slot, frame, mask) \
501− (((slot < frame->allocated_stack / BPF_REG_SIZE) && \
502− ((1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & (mask))) \
503− ? &frame->stack[slot].spilled_ptr : NULL)
530+static inline struct bpf_reg_state *
531+bpf_get_spilled_reg(int slot, struct bpf_func_state *frame, u32 mask)
532+{
533+ if (slot < frame->allocated_stack / BPF_REG_SIZE &&
534+ (1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & mask)
535+ return &frame->stack[slot].spilled_ptr;
536+ return NULL;
537+}
538+
539+static inline struct bpf_reg_state *
540+bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)
541+{
542+ if (slot < frame->out_stack_arg_cnt &&
543+ frame->stack_arg_regs[slot].type != NOT_INIT)
544+ return &frame->stack_arg_regs[slot];
545+ return NULL;
546+}
504547
505548 /* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */
506549 #define bpf_for_each_spilled_reg(iter, frame, reg, mask) \
@@ -508,7 +551,13 @@ struct bpf_verifier_state { · 1 unmodified line
508551 iter < frame->allocated_stack / BPF_REG_SIZE; \
509552 iter++, reg = bpf_get_spilled_reg(iter, frame, mask))
510553
511−#define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __mask, __expr) \
554+/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */
555+#define bpf_for_each_spilled_stack_arg(iter, frame, reg) \
556+ for (iter = 0, reg = bpf_get_spilled_stack_arg(iter, frame); \
557+ iter < frame->out_stack_arg_cnt; \
558+ iter++, reg = bpf_get_spilled_stack_arg(iter, frame))
559+
560+#define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __stack, __mask, __expr) \
512561 ({ \
513562 struct bpf_verifier_state *___vstate = __vst; \
514563 int ___i, ___j; \
@@ -516,6 +565,7 @@ struct bpf_verifier_state { · 1 unmodified line
516565 struct bpf_reg_state *___regs; \
517566 __state = ___vstate->frame[___i]; \
518567 ___regs = __state->regs; \
568+ __stack = NULL; \
519569 for (___j = 0; ___j < MAX_BPF_REG; ___j++) { \
520570 __reg = &___regs[___j]; \
521571 (void)(__expr); \
@@ -523,14 +573,27 @@ struct bpf_verifier_state { · 1 unmodified line
523573 bpf_for_each_spilled_reg(___j, __state, __reg, __mask) { \
524574 if (!__reg) \
525575 continue; \
576+ __stack = &__state->stack[___j]; \
526577 (void)(__expr); \
527578 } \
579+ __stack = NULL; \
580+ bpf_for_each_spilled_stack_arg(___j, __state, __reg) { \
581+ if (!__reg) \
582+ continue; \
583+ (void)(__expr); \
584+ } \
528585 } \
586+ (void)__stack; \
529587 })
530588
531589 /* Invoke __expr over regsiters in __vst, setting __state and __reg */
532−#define bpf_for_each_reg_in_vstate(__vst, __state, __reg, __expr) \
533− bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, 1 << STACK_SPILL, __expr)
590+#define bpf_for_each_reg_in_vstate(__vst, __state, __reg, __expr) \
591+ ({ \
592+ struct bpf_stack_state * ___stack; \
593+ (void)___stack; \
594+ bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, ___stack,\
595+ 1 << STACK_SPILL, __expr); \
596+ })
534597
535598 /* linked list of verifier states used to prune search */
536599 struct bpf_verifier_state_list {
@@ -700,6 +763,22 @@ static inline bool bpf_verifier_log_needed(const struct bpf_verifier_log *log) · 163 unmodified lines
700763 return log && log->level;
701764 }
702765
766+struct bpf_log_attr {
767+ char __user *ubuf;
768+ u32 size;
769+ u32 level;
770+ u32 offsetof_true_size;
771+ bpfptr_t uattr;
772+};
773+
774+int bpf_log_attr_init(struct bpf_log_attr *log, u64 log_buf, u32 log_size, u32 log_level,
775+ u32 offsetof_log_true_size, bpfptr_t uattr, struct bpf_common_attr *common,
776+ bpfptr_t uattr_common, u32 size_common);
777+struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log,
778+ struct bpf_common_attr *common, bpfptr_t uattr,
779+ u32 size);
780+int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log);
781+
703782 #define BPF_MAX_SUBPROGS 256
704783
705784 struct bpf_subprog_arg_info {
@@ -724,6 +803,7 @@ struct bpf_subprog_info { · 18 unmodified lines
724803 u32 exit_idx; /* Index of one of the BPF_EXIT instructions in this subprogram */
725804 u16 stack_depth; /* max. stack depth used by this function */
726805 u16 stack_extra;
806+ u32 insn_processed;
727807 /* offsets in range [stack_depth .. fastcall_stack_off)
728808 * are used for bpf_fastcall spills and fills.
729809 */
@@ -740,12 +820,21 @@ struct bpf_subprog_info { · 10 unmodified lines
740820 bool keep_fastcall_stack: 1;
741821 bool changes_pkt_data: 1;
742822 bool might_sleep: 1;
743− u8 arg_cnt:3;
823+ u8 arg_cnt:4;
744824
745825 enum priv_stack_mode priv_stack_mode;
746− struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS];
826+ struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
827+ u16 stack_arg_cnt; /* incoming + max outgoing */
828+ u16 max_out_stack_arg_cnt;
747829 };
748830
831+static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub)
832+{
833+ if (sub->arg_cnt > MAX_BPF_FUNC_REG_ARGS)
834+ return sub->arg_cnt - MAX_BPF_FUNC_REG_ARGS;
835+ return 0;
836+}
837+
749838 struct bpf_verifier_env;
750839
751840 struct backtrack_state {
@@ -753,6 +842,7 @@ struct backtrack_state { · 1 unmodified line
753842 u32 frame;
754843 u32 reg_masks[MAX_CALL_FRAMES];
755844 u64 stack_masks[MAX_CALL_FRAMES];
845+ u8 stack_arg_masks[MAX_CALL_FRAMES];
756846 };
757847
758848 struct bpf_id_pair {
@@ -881,6 +971,8 @@ struct bpf_verifier_env { · 122 unmodified lines
881971 u32 prev_insn_processed, insn_processed;
882972 /* number of jmps, calls, exits analyzed so far */
883973 u32 prev_jmps_processed, jmps_processed;
974+ /* maximum combined stack depth */
975+ u32 max_stack_depth;
884976 /* total verification time */
885977 u64 verification_time;
886978 /* maximum number of verifier states kept in 'branching' instructions */
@@ -914,6 +1006,7 @@ struct bpf_verifier_env { · 27 unmodified lines
9141006 * e.g., in reg_type_str() to generate reg_type string
9151007 */
9161008 char tmp_str_buf[TMP_STR_BUF_LEN];
1009+ char tmp_arg_name[32];
9171010 struct bpf_insn insn_buf[INSN_BUF_SIZE];
9181011 struct bpf_insn epilogue_buf[INSN_BUF_SIZE];
9191012 struct bpf_scc_callchain callchain_buf;
@@ -1087,7 +1180,7 @@ struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx); · 167 unmodified lines
10871180 void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);
10881181 void bpf_free_backedges(struct bpf_scc_visit *visit);
10891182 int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
1090− int insn_flags, u64 linked_regs);
1183+ int insn_flags, int spi, int frame, u64 linked_regs);
10911184 void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist);
10921185 void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
10931186 struct bpf_reg_state *reg);
@@ -1150,6 +1243,11 @@ static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame, · 56 unmodified lines
11501243 bt->stack_masks[frame] |= 1ull << slot;
11511244 }
11521245
1246+static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
1247+{
1248+ bt->stack_arg_masks[frame] |= 1 << slot;
1249+}
1250+
11531251 static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u32 reg)
11541252 {
11551253 return bt->reg_masks[frame] & (1 << reg);
@@ -1321,6 +1419,25 @@ struct bpf_map_desc { · 165 unmodified lines
13211419 int uid;
13221420 };
13231421
1422+/* The last initialized dynptr; Populated by process_dynptr_func() */
1423+struct bpf_dynptr_desc {
1424+ enum bpf_dynptr_type type;
1425+ u32 id;
1426+ u32 parent_id;
1427+};
1428+
1429+/*
1430+ * The last seen rereferenced object; Updated by update_ref_obj() when a register refers to a
1431+ * referenced object. Used when the helper or kfunc is casting a referenced object, returning
1432+ * allocated memory derived from referenced object or creating a dynptr with a referenced
1433+ * object as parent.
1434+ */
1435+struct ref_obj_desc {
1436+ u32 id;
1437+ u32 parent_id;
1438+ u8 cnt;
1439+};
1440+
13241441 struct bpf_kfunc_call_arg_meta {
13251442 /* In parameters */
13261443 struct btf *btf;
@@ -1329,7 +1446,6 @@ struct bpf_kfunc_call_arg_meta { · 2 unmodified lines
13291446 const struct btf_type *func_proto;
13301447 const char *func_name;
13311448 /* Out parameters */
1332− u32 ref_obj_id;
13331449 u8 release_regno;
13341450 bool r0_rdonly;
13351451 u32 ret_btf_id;
@@ -1361,16 +1477,13 @@ struct bpf_kfunc_call_arg_meta { · 25 unmodified lines
13611477 struct {
13621478 struct btf_field *field;
13631479 } arg_rbtree_root;
1364− struct {
1365− enum bpf_dynptr_type type;
1366− u32 id;
1367− u32 ref_obj_id;
1368− } initialized_dynptr;
13691480 struct {
13701481 u8 spi;
13711482 u8 frameno;
13721483 } iter;
13731484 struct bpf_map_desc map;
1485+ struct bpf_dynptr_desc dynptr;
1486+ struct ref_obj_desc ref_obj;
13741487 u64 mem_size;
13751488 };
13761489
@@ -1479,6 +1592,10 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset); · 102 unmodified lines
14791592 int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
14801593 struct bpf_insn *insn_buf, int insn_idx, int *cnt);
14811594
1595+/* Functions exported from verifier.c, used by trampoline.c */
1596+int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 btf_id,
1597+ struct bpf_attach_target_info *tgt_info);
1598+
14821599 /* Functions in fixups.c, called from bpf_check() */
14831600 int bpf_remove_fastcall_spills_fills(struct bpf_verifier_env *env);
14841601 int bpf_optimize_bpf_loop(struct bpf_verifier_env *env);
include/linux/btf.h+4−3
@@ -145,7 +145,8 @@ const char *btf_get_name(const struct btf *btf); · 144 unmodified lines
145145 void btf_get(struct btf *btf);
146146 void btf_put(struct btf *btf);
147147 const struct btf_header *btf_header(const struct btf *btf);
148−int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_sz);
148+struct bpf_log_attr;
149+int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log);
149150 struct btf *btf_get_by_fd(int fd);
150151 int btf_get_info_by_fd(const struct btf *btf,
151152 const union bpf_attr *attr,
@@ -415,12 +416,12 @@ static inline bool btf_type_is_array(const struct btf_type *t) · 263 unmodified lines
415416 return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY;
416417 }
417418
418−static inline u16 btf_type_vlen(const struct btf_type *t)
419+static inline u32 btf_type_vlen(const struct btf_type *t)
419420 {
420421 return BTF_INFO_VLEN(t->info);
421422 }
422423
423−static inline u16 btf_vlen(const struct btf_type *t)
424+static inline u32 btf_vlen(const struct btf_type *t)
424425 {
425426 return btf_type_vlen(t);
426427 }
include/linux/btf_ids.h+1−0
@@ -284,5 +284,6 @@ extern u32 bpf_cgroup_btf_id[]; · 283 unmodified lines
284284 extern u32 bpf_local_storage_map_btf_id[];
285285 extern u32 btf_bpf_map_id[];
286286 extern u32 bpf_kmem_cache_btf_id[];
287+extern u32 bpf_multi_func_btf_id[];
287288
288289 #endif
include/linux/cnum.h+82−0
@@ -0,0 +1,82 @@
1+/* SPDX-License-Identifier: GPL-2.0-only */
2+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
3+
4+#ifndef _LINUX_CNUM_H
5+#define _LINUX_CNUM_H
6+
7+#include <linux/types.h>
8+
9+/*
10+ * cnum32: a circular number.
11+ * A unified representation for signed and unsigned ranges.
12+ *
13+ * Assume that a 32-bit range is a circle, with 0 being in the 12 o'clock
14+ * position, numbers placed sequentially in clockwise order and U32_MAX
15+ * in the 11 o'clock position. Signed values map onto the same circle:
16+ * S32_MAX sits at 5 o'clock, S32_MIN sits at 6 o'clock (opposite 0),
17+ * negative values occupy the left half and positive values the right half.
18+ *
19+ * @cnum32 represents an arc on this circle drawn clockwise.
20+ * @base corresponds to the first value of the range.
21+ * @size corresponds to the number of integers in the range excluding @base.
22+ * (The @base is excluded to avoid integer overflow when representing the full
23+ * 0..U32_MAX range, which corresponds to 2^32, which can't be stored in u32).
24+ *
25+ * For example: {U32_MAX, 1} corresponds to signed range [-1, 0],
26+ * {S32_MAX, 1} corresponds to unsigned range [S32_MAX, S32_MIN].
27+ */
28+struct cnum32 {
29+ u32 base;
30+ u32 size;
31+};
32+
33+#define CNUM32_UNBOUNDED ((struct cnum32){ .base = 0, .size = U32_MAX })
34+#define CNUM32_EMPTY ((struct cnum32){ .base = U32_MAX, .size = U32_MAX })
35+
36+struct cnum32 cnum32_from_urange(u32 min, u32 max);
37+struct cnum32 cnum32_from_srange(s32 min, s32 max);
38+u32 cnum32_umin(struct cnum32 cnum);
39+u32 cnum32_umax(struct cnum32 cnum);
40+s32 cnum32_smin(struct cnum32 cnum);
41+s32 cnum32_smax(struct cnum32 cnum);
42+struct cnum32 cnum32_intersect(struct cnum32 a, struct cnum32 b);
43+void cnum32_intersect_with(struct cnum32 *dst, struct cnum32 src);
44+void cnum32_intersect_with_urange(struct cnum32 *dst, u32 min, u32 max);
45+void cnum32_intersect_with_srange(struct cnum32 *dst, s32 min, s32 max);
46+bool cnum32_contains(struct cnum32 cnum, u32 v);
47+bool cnum32_is_const(struct cnum32 cnum);
48+bool cnum32_is_empty(struct cnum32 cnum);
49+struct cnum32 cnum32_add(struct cnum32 a, struct cnum32 b);
50+struct cnum32 cnum32_negate(struct cnum32 a);
51+bool cnum32_is_subset(struct cnum32 outer, struct cnum32 inner);
52+
53+/* Same as cnum32 but for 64-bit ranges */
54+struct cnum64 {
55+ u64 base;
56+ u64 size;
57+};
58+
59+#define CNUM64_UNBOUNDED ((struct cnum64){ .base = 0, .size = U64_MAX })
60+#define CNUM64_EMPTY ((struct cnum64){ .base = U64_MAX, .size = U64_MAX })
61+
62+struct cnum64 cnum64_from_urange(u64 min, u64 max);
63+struct cnum64 cnum64_from_srange(s64 min, s64 max);
64+u64 cnum64_umin(struct cnum64 cnum);
65+u64 cnum64_umax(struct cnum64 cnum);
66+s64 cnum64_smin(struct cnum64 cnum);
67+s64 cnum64_smax(struct cnum64 cnum);
68+struct cnum64 cnum64_intersect(struct cnum64 a, struct cnum64 b);
69+void cnum64_intersect_with(struct cnum64 *dst, struct cnum64 src);
70+void cnum64_intersect_with_urange(struct cnum64 *dst, u64 min, u64 max);
71+void cnum64_intersect_with_srange(struct cnum64 *dst, s64 min, s64 max);
72+bool cnum64_contains(struct cnum64 cnum, u64 v);
73+bool cnum64_is_const(struct cnum64 cnum);
74+bool cnum64_is_empty(struct cnum64 cnum);
75+struct cnum64 cnum64_add(struct cnum64 a, struct cnum64 b);
76+struct cnum64 cnum64_negate(struct cnum64 a);
77+bool cnum64_is_subset(struct cnum64 outer, struct cnum64 inner);
78+
79+struct cnum32 cnum32_from_cnum64(struct cnum64 cnum);
80+struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b);
81+
82+#endif /* _LINUX_CNUM_H */
include/linux/filter.h+25−2
@@ -58,8 +58,9 @@ struct ctl_table_header; · 57 unmodified lines
5858 #define BPF_REG_H BPF_REG_9 /* hlen, callee-saved */
5959
6060 /* Kernel hidden auxiliary/helper register. */
61−#define BPF_REG_AX MAX_BPF_REG
62−#define MAX_BPF_EXT_REG (MAX_BPF_REG + 1)
61+#define BPF_REG_PARAMS MAX_BPF_REG
62+#define BPF_REG_AX (MAX_BPF_REG + 1)
63+#define MAX_BPF_EXT_REG (MAX_BPF_REG + 2)
6364 #define MAX_BPF_JIT_REG MAX_BPF_EXT_REG
6465
6566 /* unused opcode to mark special call to bpf_tail_call() helper */
@@ -748,6 +749,27 @@ static inline u32 bpf_prog_run_pin_on_cpu(const struct bpf_prog *prog, · 682 unmodified lines
748749 return ret;
749750 }
750751
752+static inline bool is_stack_arg_ldx(const struct bpf_insn *insn)
753+{
754+ return insn->code == (BPF_LDX | BPF_MEM | BPF_DW) &&
755+ insn->src_reg == BPF_REG_PARAMS &&
756+ insn->off > 0 && insn->off % 8 == 0;
757+}
758+
759+static inline bool is_stack_arg_st(const struct bpf_insn *insn)
760+{
761+ return insn->code == (BPF_ST | BPF_MEM | BPF_DW) &&
762+ insn->dst_reg == BPF_REG_PARAMS &&
763+ insn->off < 0 && insn->off % 8 == 0;
764+}
765+
766+static inline bool is_stack_arg_stx(const struct bpf_insn *insn)
767+{
768+ return insn->code == (BPF_STX | BPF_MEM | BPF_DW) &&
769+ insn->dst_reg == BPF_REG_PARAMS &&
770+ insn->off < 0 && insn->off % 8 == 0;
771+}
772+
751773 #define BPF_SKB_CB_LEN QDISC_CB_PRIV_LEN
752774
753775 struct bpf_skb_data_end {
@@ -1159,6 +1181,7 @@ bool bpf_jit_inlines_helper_call(s32 imm); · 405 unmodified lines
11591181 bool bpf_jit_supports_subprog_tailcalls(void);
11601182 bool bpf_jit_supports_percpu_insn(void);
11611183 bool bpf_jit_supports_kfunc_call(void);
1184+bool bpf_jit_supports_stack_args(void);
11621185 bool bpf_jit_supports_far_kfunc_call(void);
11631186 bool bpf_jit_supports_exceptions(void);
11641187 bool bpf_jit_supports_ptr_xchg(void);
include/linux/ftrace.h+9−0
@@ -415,6 +415,8 @@ struct ftrace_hash *alloc_ftrace_hash(int size_bits); · 414 unmodified lines
415415 void free_ftrace_hash(struct ftrace_hash *hash);
416416 struct ftrace_func_entry *add_ftrace_hash_entry_direct(struct ftrace_hash *hash,
417417 unsigned long ip, unsigned long direct);
418+void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry);
419+void ftrace_hash_remove(struct ftrace_hash *hash);
418420
419421 /* The hash used to know what functions callbacks trace */
420422 struct ftrace_ops_hash {
@@ -551,6 +553,8 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b · 130 unmodified lines
551553
552554 void ftrace_stub_direct_tramp(void);
553555
556+unsigned long ftrace_hash_count(struct ftrace_hash *hash);
557+
554558 #else
555559 struct ftrace_ops;
556560 static inline unsigned long ftrace_find_rec_direct(unsigned long ip)
@@ -590,6 +594,11 @@ static inline int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace · 33 unmodified lines
590594 return -ENODEV;
591595 }
592596
597+static inline unsigned long ftrace_hash_count(struct ftrace_hash *hash)
598+{
599+ return 0;
600+}
601+
593602 /*
594603 * This must be implemented by the architecture.
595604 * It is the way the ftrace direct_ops helper, when called
include/linux/pgtable.h+43−0
@@ -1070,6 +1070,49 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres · 1069 unmodified lines
10701070 }
10711071 #endif
10721072
1073+#ifndef ptep_try_set
1074+/**
1075+ * ptep_try_set - atomically set an empty kernel PTE
1076+ * @ptep: page table entry
1077+ * @new_pte: value to install
1078+ *
1079+ * Atomically set *@ptep to @new_pte iff *@ptep is pte_none(). Return true on
1080+ * success, false if the slot was already populated or the arch has no
1081+ * implementation.
1082+ *
1083+ * For special kernel page tables only - never user page tables. The caller must
1084+ * prevent concurrent teardown of @ptep and must accept that other writers may
1085+ * race. Concurrent clearers must use ptep_get_and_clear() so racing accesses
1086+ * agree on the outcome.
1087+ *
1088+ * Architectures opt in by providing a cmpxchg-based override and defining
1089+ * ptep_try_set as an identity macro. The generic stub returns false, which is
1090+ * correct for callers that fall through to oops on failure.
1091+ */
1092+static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte)
1093+{
1094+ return false;
1095+}
1096+#endif
1097+
1098+#ifndef flush_tlb_before_set
1099+/**
1100+ * flush_tlb_before_set - invalidate a kernel PTE's TLB before re-setting it
1101+ * @addr: kernel virtual address whose PTE was just cleared
1102+ *
1103+ * Some architectures (e.g. arm64) do not allow a live page-table entry to be
1104+ * repointed at a different page in one step. The old entry must first be made
1105+ * invalid and its translation flushed from every TLB, and only then may the new
1106+ * entry be written.
1107+ *
1108+ * This is only for the lockless atomic kernel-PTE installers (ptep_try_set()).
1109+ * It must be callable with interrupts disabled.
1110+ */
1111+static inline void flush_tlb_before_set(unsigned long addr)
1112+{
1113+}
1114+#endif
1115+
10731116 #ifndef wrprotect_ptes
10741117 /**
10751118 * wrprotect_ptes - Write-protect PTEs that map consecutive pages of the same
include/linux/rhashtable.h+3−1
@@ -263,6 +263,8 @@ struct rhash_lock_head __rcu **__rht_bucket_nested( · 262 unmodified lines
263263 struct rhash_lock_head __rcu **rht_bucket_nested_insert(
264264 struct rhashtable *ht, struct bucket_table *tbl, unsigned int hash);
265265
266+void *rhashtable_next_key(struct rhashtable *ht, const void *prev_key);
267+
266268 #define rht_dereference(p, ht) \
267269 rcu_dereference_protected(p, lockdep_rht_mutex_is_held(ht))
268270
@@ -1117,7 +1119,7 @@ unlocked: · 848 unmodified lines
11171119 atomic_dec(&ht->nelems);
11181120 if (unlikely(ht->p.automatic_shrinking &&
11191121 rht_shrink_below_30(ht, tbl)))
1120− schedule_work(&ht->run_work);
1122+ irq_work_queue(&ht->run_irq_work);
11211123 err = 0;
11221124 }
11231125
include/linux/syscalls.h+2−1
@@ -940,7 +940,8 @@ asmlinkage long sys_seccomp(unsigned int op, unsigned int flags, · 939 unmodified lines
940940 asmlinkage long sys_getrandom(char __user *buf, size_t count,
941941 unsigned int flags);
942942 asmlinkage long sys_memfd_create(const char __user *uname_ptr, unsigned int flags);
943−asmlinkage long sys_bpf(int cmd, union bpf_attr __user *attr, unsigned int size);
943+asmlinkage long sys_bpf(int cmd, union bpf_attr __user *attr, unsigned int size,
944+ struct bpf_common_attr __user *attr_common, unsigned int size_common);
944945 asmlinkage long sys_execveat(int dfd, const char __user *filename,
945946 const char __user *const __user *argv,
946947 const char __user *const __user *envp, int flags);
include/linux/trace_events.h+12−0
@@ -770,6 +770,7 @@ trace_trigger_soft_disabled(struct trace_event_file *file) · 769 unmodified lines
770770
771771 #ifdef CONFIG_BPF_EVENTS
772772 unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx);
773+unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx);
773774 int perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie);
774775 void perf_event_detach_bpf_prog(struct perf_event *event);
775776 int perf_event_query_prog_array(struct perf_event *event, void __user *info);
@@ -786,12 +787,18 @@ int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id, · 10 unmodified lines
786787 unsigned long *missed);
787788 int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
788789 int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
790+int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr);
789791 #else
790792 static inline unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx)
791793 {
792794 return 1;
793795 }
794796
797+static inline unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx)
798+{
799+ return 1;
800+}
801+
795802 static inline int
796803 perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie)
797804 {
@@ -838,6 +845,11 @@ bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) · 40 unmodified lines
838845 {
839846 return -EOPNOTSUPP;
840847 }
848+static inline int
849+bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
850+{
851+ return -EOPNOTSUPP;
852+}
841853 #endif
842854
843855 enum {
include/trace/bpf_probe.h+0−2
@@ -58,9 +58,7 @@ static notrace void \ · 57 unmodified lines
5858 __bpf_trace_##call(void *__data, proto) \
5959 { \
6060 might_fault(); \
61− preempt_disable_notrace(); \
6261 CONCATENATE(bpf_trace_run, COUNT_ARGS(args))(__data, CAST_TO_U64(args)); \
63− preempt_enable_notrace(); \
6462 }
6563
6664 #undef DECLARE_EVENT_SYSCALL_CLASS
include/uapi/linux/bpf.h+31−1
@@ -994,6 +994,7 @@ enum bpf_cmd { · 993 unmodified lines
994994 BPF_PROG_STREAM_READ_BY_FD,
995995 BPF_PROG_ASSOC_STRUCT_OPS,
996996 __MAX_BPF_CMD,
997+ BPF_COMMON_ATTRS = 1 << 16, /* Indicate carrying syscall common attrs. */
997998 };
998999
9991000 enum bpf_map_type {
@@ -1046,6 +1047,7 @@ enum bpf_map_type { · 46 unmodified lines
10461047 BPF_MAP_TYPE_CGRP_STORAGE,
10471048 BPF_MAP_TYPE_ARENA,
10481049 BPF_MAP_TYPE_INSN_ARRAY,
1050+ BPF_MAP_TYPE_RHASH,
10491051 __MAX_BPF_MAP_TYPE
10501052 };
10511053
@@ -1154,6 +1156,9 @@ enum bpf_attach_type { · 102 unmodified lines
11541156 BPF_TRACE_KPROBE_SESSION,
11551157 BPF_TRACE_UPROBE_SESSION,
11561158 BPF_TRACE_FSESSION,
1159+ BPF_TRACE_FENTRY_MULTI,
1160+ BPF_TRACE_FEXIT_MULTI,
1161+ BPF_TRACE_FSESSION_MULTI,
11571162 __MAX_BPF_ATTACH_TYPE
11581163 };
11591164
@@ -1178,6 +1183,7 @@ enum bpf_link_type { · 18 unmodified lines
11781183 BPF_LINK_TYPE_UPROBE_MULTI = 12,
11791184 BPF_LINK_TYPE_NETKIT = 13,
11801185 BPF_LINK_TYPE_SOCKMAP = 14,
1186+ BPF_LINK_TYPE_TRACING_MULTI = 15,
11811187 __MAX_BPF_LINK_TYPE,
11821188 };
11831189
@@ -1321,7 +1327,11 @@ enum { · 137 unmodified lines
13211327 * BPF_TRACE_UPROBE_MULTI attach type to create return probe.
13221328 */
13231329 enum {
1324− BPF_F_UPROBE_MULTI_RETURN = (1U << 0)
1330+ /* Get return uprobe. */
1331+ BPF_F_UPROBE_MULTI_RETURN = (1U << 0),
1332+
1333+ /* Get path from provided path_fd. */
1334+ BPF_F_UPROBE_MULTI_PATH_FD = (1U << 1),
13251335 };
13261336
13271337 /* link_create.netfilter.flags used in LINK_CREATE command for
@@ -1500,6 +1510,13 @@ struct bpf_stack_build_id { · 172 unmodified lines
15001510 };
15011511 };
15021512
1513+struct bpf_common_attr {
1514+ __aligned_u64 log_buf;
1515+ __u32 log_size;
1516+ __u32 log_level;
1517+ __u32 log_true_size;
1518+};
1519+
15031520 #define BPF_OBJ_NAME_LEN 16U
15041521
15051522 enum {
@@ -1537,6 +1554,11 @@ union bpf_attr { · 31 unmodified lines
15371554 *
15381555 * BPF_MAP_TYPE_ARENA - contains the address where user space
15391556 * is going to mmap() the arena. It has to be page aligned.
1557+ *
1558+ * BPF_MAP_TYPE_RHASH - initial table size hint
1559+ * (nelem_hint). 0 = use rhashtable default. Must be
1560+ * <= min(max_entries, U16_MAX). Upper 32 bits reserved,
1561+ * must be zero.
15401562 */
15411563 __u64 map_extra;
15421564
@@ -1846,6 +1868,7 @@ union bpf_attr { · 303 unmodified lines
18461868 __u32 cnt;
18471869 __u32 flags;
18481870 __u32 pid;
1871+ __u32 path_fd;
18491872 } uprobe_multi;
18501873 struct {
18511874 union {
@@ -1861,6 +1884,11 @@ union bpf_attr { · 9 unmodified lines
18611884 };
18621885 __u64 expected_revision;
18631886 } cgroup;
1887+ struct {
1888+ __aligned_u64 ids;
1889+ __aligned_u64 cookies;
1890+ __u32 cnt;
1891+ } tracing_multi;
18641892 };
18651893 } link_create;
18661894
@@ -6698,6 +6726,7 @@ struct bpf_prog_info { · 4831 unmodified lines
66986726 __u32 verified_insns;
66996727 __u32 attach_btf_obj_id;
67006728 __u32 attach_btf_id;
6729+ __u32 :32;
67016730 } __attribute__((aligned(8)));
67026731
67036732 struct bpf_map_info {
@@ -6719,6 +6748,7 @@ struct bpf_map_info { · 15 unmodified lines
67196748 __u64 map_extra;
67206749 __aligned_u64 hash;
67216750 __u32 hash_size;
6751+ __u32 :32;
67226752 } __attribute__((aligned(8)));
67236753
67246754 struct bpf_btf_info {
include/uapi/linux/btf.h+14−12
@@ -33,20 +33,22 @@ struct btf_header { · 32 unmodified lines
3333 __u32 layout_len; /* length of layout section */
3434 };
3535
36−/* Max # of type identifier */
37−#define BTF_MAX_TYPE 0x000fffff
38−/* Max offset into the string section */
39−#define BTF_MAX_NAME_OFFSET 0x00ffffff
40−/* Max # of struct/union/enum members or func args */
41−#define BTF_MAX_VLEN 0xffff
36+enum btf_max {
37+ /* Max possible kind */
38+ BTF_MAX_KIND = 0x0000007f,
39+ /* Max # of type identifier */
40+ BTF_MAX_TYPE = 0x000fffff,
41+ /* Max offset into the string section */
42+ BTF_MAX_NAME_OFFSET = 0x00ffffff,
43+ /* Max # of struct/union/enum members or func args */
44+ BTF_MAX_VLEN = 0x00ffffff,
45+};
4246
4347 struct btf_type {
4448 __u32 name_off;
4549 /* "info" bits arrangement
46− * bits 0-15: vlen (e.g. # of struct's members)
47− * bits 16-23: unused
48− * bits 24-28: kind (e.g. int, ptr, array...etc)
49− * bits 29-30: unused
50+ * bits 0-23: vlen (e.g. # of struct's members)
51+ * bits 24-30: kind (e.g. int, ptr, array...etc)
5052 * bit 31: kind_flag, currently used by
5153 * struct, union, enum, fwd, enum64,
5254 * decl_tag and type_tag
@@ -65,8 +67,8 @@ struct btf_type { · 12 unmodified lines
6567 };
6668 };
6769
68−#define BTF_INFO_KIND(info) (((info) >> 24) & 0x1f)
69−#define BTF_INFO_VLEN(info) ((info) & 0xffff)
70+#define BTF_INFO_KIND(info) (((info) >> 24) & 0x7f)
71+#define BTF_INFO_VLEN(info) ((info) & 0xffffff)
7072 #define BTF_INFO_KFLAG(info) ((info) >> 31)
7173
7274 enum {
kernel/bpf/Makefile+1−1
@@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse · 5 unmodified lines
66 endif
77 CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)
88
9−obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o liveness.o const_fold.o
9+obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o
1010 obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o
1111 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
1212 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
kernel/bpf/arena.c+266−49
@@ -53,12 +53,15 @@ struct bpf_arena { · 52 unmodified lines
5353 u64 user_vm_start;
5454 u64 user_vm_end;
5555 struct vm_struct *kern_vm;
56+ struct page *scratch_page;
5657 struct range_tree rt;
5758 /* protects rt */
5859 rqspinlock_t spinlock;
5960 struct list_head vma_list;
6061 /* protects vma_list */
6162 struct mutex lock;
63+ u64 zap_gen;
64+ struct mutex zap_mutex;
6265 struct irq_work free_irq;
6366 struct work_struct free_work;
6467 struct llist_head free_spans;
@@ -83,6 +86,32 @@ u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena) · 18 unmodified lines
8386 return arena ? arena->user_vm_start : 0;
8487 }
8588
89+/**
90+ * bpf_arena_map_kern_vm_start - kern_vm_start lookup by struct bpf_map *
91+ * @map: a BPF_MAP_TYPE_ARENA map
92+ *
93+ * Return @map's kern_vm_start.
94+ */
95+u64 bpf_arena_map_kern_vm_start(struct bpf_map *map)
96+{
97+ return bpf_arena_get_kern_vm_start(container_of(map, struct bpf_arena, map));
98+}
99+
100+/**
101+ * bpf_prog_arena - return the bpf_map of the arena referenced by @prog
102+ * @prog: a loaded BPF program
103+ *
104+ * The verifier enforces at most one arena per program and stores it in
105+ * prog->aux->arena. Return that arena's underlying bpf_map, or NULL if
106+ * @prog does not reference an arena.
107+ */
108+struct bpf_map *bpf_prog_arena(struct bpf_prog *prog)
109+{
110+ struct bpf_arena *arena = prog->aux->arena;
111+
112+ return arena ? &arena->map : NULL;
113+}
114+
86115 static long arena_map_peek_elem(struct bpf_map *map, void *value)
87116 {
88117 return -EOPNOTSUPP;
@@ -115,26 +144,57 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr) · 26 unmodified lines
115144
116145 struct apply_range_data {
117146 struct page **pages;
147+ struct page *scratch_page;
118148 int i;
119149 };
120150
151+struct clear_range_data {
152+ struct llist_head *free_pages;
153+ struct page *scratch_page;
154+};
155+
121156 static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)
122157 {
123158 struct apply_range_data *d = data;
124159 struct page *page;
160+ pte_t pteval;
125161
126162 if (!data)
127163 return 0;
128− /* sanity check */
129− if (unlikely(!pte_none(ptep_get(pte))))
130− return -EBUSY;
131164
132165 page = d->pages[d->i];
133166 /* paranoia, similar to vmap_pages_pte_range() */
134167 if (WARN_ON_ONCE(!pfn_valid(page_to_pfn(page))))
135168 return -EINVAL;
136169
137− set_pte_at(&init_mm, addr, pte, mk_pte(page, PAGE_KERNEL));
170+ pteval = mk_pte(page, PAGE_KERNEL);
171+#ifdef ptep_try_set
172+ /*
173+ * Kernel-fault recovery may have installed the scratch page here, and
174+ * some architectures (arm64) prohibit valid->valid PTE transitions.
175+ * Install atomically into a none slot. If scratch is present, clear it
176+ * and flush_tlb_before_set() (break-before-make) before retrying.
177+ */
178+ while (!ptep_try_set(pte, pteval)) {
179+ pte_t old = ptep_get(pte);
180+
181+ if (pte_none(old))
182+ continue;
183+ if (WARN_ON_ONCE(pte_page(old) != d->scratch_page))
184+ return -EBUSY;
185+ ptep_get_and_clear(&init_mm, addr, pte);
186+ flush_tlb_before_set(addr);
187+ }
188+#else
189+ /*
190+ * Without ptep_try_set() there is no atomic installer, but such arches
191+ * also do not wire up bpf_arena_handle_page_fault(), so no scratch page
192+ * is ever installed and the slot is always none here.
193+ */
194+ if (unlikely(!pte_none(ptep_get(pte))))
195+ return -EBUSY;
196+ set_pte_at(&init_mm, addr, pte, pteval);
197+#endif
138198 d->i++;
139199 return 0;
140200 }
@@ -144,33 +204,59 @@ static void flush_vmap_cache(unsigned long start, unsigned long size) · 3 unmodified lines
144204 flush_cache_vmap(start, start + size);
145205 }
146206
147−static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *free_pages)
207+static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data)
148208 {
209+ struct clear_range_data *d = data;
149210 pte_t old_pte;
150211 struct page *page;
151212
152− /* sanity check */
153− old_pte = ptep_get(pte);
213+ /*
214+ * Pairs with ptep_try_set() in the kernel-fault scratch installer.
215+ * Both sides must be atomic.
216+ */
217+ old_pte = ptep_get_and_clear(&init_mm, addr, pte);
154218 if (pte_none(old_pte) || !pte_present(old_pte))
155− return 0; /* nothing to do */
219+ return 0;
156220
157221 page = pte_page(old_pte);
158222 if (WARN_ON_ONCE(!page))
159223 return -EINVAL;
160224
161− pte_clear(&init_mm, addr, pte);
225+ /*
226+ * Skip the per-arena scratch page. A kernel fault on an unallocated uaddr
227+ * scratches its PTE. A later bpf_arena_free_pages() over that range walks
228+ * here. Without the skip, scratch_page would be freed.
229+ */
230+ if (page == d->scratch_page)
231+ return 0;
232+
233+ __llist_add(&page->pcp_llist, d->free_pages);
234+ return 0;
235+}
162236
163− /* Add page to the list so it is freed later */
164− if (free_pages)
165− __llist_add(&page->pcp_llist, free_pages);
237+static int apply_range_set_scratch_cb(pte_t *pte, unsigned long addr, void *data)
238+{
239+ struct page *scratch_page = data;
166240
241+ if (!pte_none(ptep_get(pte)))
242+ return 0;
243+ /*
244+ * Best-effort install. ptep_try_set() returns false only if another
245+ * installer (real allocation or concurrent fault) won the cmpxchg.
246+ * Their PTE is already valid, so the access retry succeeds.
247+ *
248+ * No flush_tlb_kernel_range() needed. Stale "not mapped" entries just
249+ * cause one extra re-fault through this same path.
250+ */
251+ ptep_try_set(pte, mk_pte(scratch_page, PAGE_KERNEL));
167252 return 0;
168253 }
169254
170255 static int populate_pgtable_except_pte(struct bpf_arena *arena)
171256 {
257+ /* Populate intermediates for the recovery range (4 GiB + upper half-guard). */
172258 return apply_to_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena),
173− KERN_VM_SZ - GUARD_SZ, apply_range_set_cb, NULL);
259+ SZ_4G + GUARD_SZ / 2, apply_range_set_cb, NULL);
174260 }
175261
176262 static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
@@ -221,22 +307,30 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr) · 44 unmodified lines
221307 init_irq_work(&arena->free_irq, arena_free_irq);
222308 INIT_WORK(&arena->free_work, arena_free_worker);
223309 bpf_map_init_from_attr(&arena->map, attr);
310+
311+ err = bpf_map_alloc_pages(&arena->map, NUMA_NO_NODE, 1, &arena->scratch_page);
312+ if (err)
313+ goto err_free_arena;
314+
224315 range_tree_init(&arena->rt);
225316 err = range_tree_set(&arena->rt, 0, attr->max_entries);
226− if (err) {
227− bpf_map_area_free(arena);
228− goto err;
229− }
317+ if (err)
318+ goto err_free_scratch;
230319 mutex_init(&arena->lock);
320+ mutex_init(&arena->zap_mutex);
231321 raw_res_spin_lock_init(&arena->spinlock);
232322 err = populate_pgtable_except_pte(arena);
233− if (err) {
234− range_tree_destroy(&arena->rt);
235− bpf_map_area_free(arena);
236− goto err;
237− }
323+ if (err)
324+ goto err_destroy_rt;
238325
239326 return &arena->map;
327+
328+err_destroy_rt:
329+ range_tree_destroy(&arena->rt);
330+err_free_scratch:
331+ __free_page(arena->scratch_page);
332+err_free_arena:
333+ bpf_map_area_free(arena);
240334 err:
241335 free_vm_area(kern_vm);
242336 return ERR_PTR(err);
@@ -244,6 +338,7 @@ err: · 1 unmodified line
244338
245339 static int existing_page_cb(pte_t *ptep, unsigned long addr, void *data)
246340 {
341+ struct bpf_arena *arena = data;
247342 struct page *page;
248343 pte_t pte;
249344
@@ -251,6 +346,12 @@ static int existing_page_cb(pte_t *ptep, unsigned long addr, void *data) · 1 unmodified line
251346 if (!pte_present(pte)) /* sanity check */
252347 return 0;
253348 page = pte_page(pte);
349+ /*
350+ * Skip the scratch page. The walk is page-table-driven, not range-tree-driven,
351+ * so it can visit scratch PTEs at uaddrs the BPF program never allocated.
352+ */
353+ if (page == arena->scratch_page)
354+ return 0;
254355 /*
255356 * We do not update pte here:
256357 * 1. Nobody should be accessing bpf_arena's range outside of a kernel bug
@@ -286,9 +387,10 @@ static void arena_map_free(struct bpf_map *map) · 29 unmodified lines
286387 * free those pages.
287388 */
288389 apply_to_existing_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena),
289− KERN_VM_SZ - GUARD_SZ, existing_page_cb, NULL);
390+ SZ_4G + GUARD_SZ / 2, existing_page_cb, arena);
290391 free_vm_area(arena->kern_vm);
291392 range_tree_destroy(&arena->rt);
393+ __free_page(arena->scratch_page);
292394 bpf_map_area_free(arena);
293395 }
294396
@@ -318,6 +420,7 @@ struct vma_list { · 23 unmodified lines
318420 struct vm_area_struct *vma;
319421 struct list_head head;
320422 refcount_t mmap_count;
423+ u64 zap_gen;
321424 };
322425
323426 static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma)
@@ -330,6 +433,7 @@ static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma) · 6 unmodified lines
330433 refcount_set(&vml->mmap_count, 1);
331434 vma->vm_private_data = vml;
332435 vml->vma = vma;
436+ vml->zap_gen = 0;
333437 list_add(&vml->head, &arena->vma_list);
334438 return 0;
335439 }
@@ -384,33 +488,38 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) · 48 unmodified lines
384488 return VM_FAULT_RETRY;
385489
386490 page = vmalloc_to_page((void *)kaddr);
387− if (page)
491+ if (page) {
492+ if (page == arena->scratch_page)
493+ /* BPF triggered scratch here; don't lazy-alloc over it */
494+ goto out_sigsegv;
388495 /* already have a page vmap-ed */
389496 goto out;
497+ }
390498
391499 bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
392500
393501 if (arena->map.map_flags & BPF_F_SEGV_ON_FAULT)
394502 /* User space requested to segfault when page is not allocated by bpf prog */
395− goto out_unlock_sigsegv;
503+ goto out_sigsegv_memcg;
396504
397505 ret = range_tree_clear(&arena->rt, vmf->pgoff, 1);
398506 if (ret)
399− goto out_unlock_sigsegv;
507+ goto out_sigsegv_memcg;
400508
401− struct apply_range_data data = { .pages = &page, .i = 0 };
509+ struct apply_range_data data = { .pages = &page, .i = 0,
510+ .scratch_page = arena->scratch_page };
402511 /* Account into memcg of the process that created bpf_arena */
403512 ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page);
404513 if (ret) {
405514 range_tree_set(&arena->rt, vmf->pgoff, 1);
406− goto out_unlock_sigsegv;
515+ goto out_sigsegv_memcg;
407516 }
408517
409518 ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data);
410519 if (ret) {
411520 range_tree_set(&arena->rt, vmf->pgoff, 1);
412521 free_pages_nolock(page, 0);
413− goto out_unlock_sigsegv;
522+ goto out_sigsegv_memcg;
414523 }
415524 flush_vmap_cache(kaddr, PAGE_SIZE);
416525 bpf_map_memcg_exit(old_memcg, new_memcg);
@@ -419,8 +528,9 @@ out: · 2 unmodified lines
419528 raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
420529 vmf->page = page;
421530 return 0;
422−out_unlock_sigsegv:
531+out_sigsegv_memcg:
423532 bpf_map_memcg_exit(old_memcg, new_memcg);
533+out_sigsegv:
424534 raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
425535 return VM_FAULT_SIGSEGV;
426536 }
@@ -587,6 +697,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · 160 unmodified lines
587697 return 0;
588698 }
589699 data.pages = pages;
700+ data.scratch_page = arena->scratch_page;
590701
591702 if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
592703 goto out_free_pages;
@@ -668,12 +779,60 @@ out_free_pages: · 75 unmodified lines
668779 */
669780 static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt)
670781 {
782+ unsigned long size = (unsigned long)page_cnt << PAGE_SHIFT;
783+ struct vm_area_struct *vma;
784+ struct mm_struct *mm;
671785 struct vma_list *vml;
786+ unsigned long vm_start;
787+ u64 my_gen;
672788
673− guard(mutex)(&arena->lock);
674− /* iterate link list under lock */
675− list_for_each_entry(vml, &arena->vma_list, head)
676− zap_vma_range(vml->vma, uaddr, PAGE_SIZE * page_cnt);
789+ /*
790+ * Taking mmap_read_lock() under arena->lock would deadlock against
791+ * arena_vm_close(), which runs with mmap_write_lock held and then
792+ * acquires arena->lock. Drop arena->lock for mmap_read_lock().
793+ *
794+ * Use per-call my_gen, recorded in vml->zap_gen, to remember which
795+ * vmls this invocation has already processed across the lock drop.
796+ * Hold zap_mutex around the whole walk so concurrent zap_pages()
797+ * callers cannot overwrite each other's marks on shared vmls --
798+ * otherwise call B's mark would make call A skip a vml that A has
799+ * not yet zapped for A's uaddr range.
800+ */
801+ mutex_lock(&arena->zap_mutex);
802+ mutex_lock(&arena->lock);
803+ my_gen = ++arena->zap_gen;
804+ for (;;) {
805+ mm = NULL;
806+ list_for_each_entry(vml, &arena->vma_list, head) {
807+ if (vml->zap_gen >= my_gen)
808+ continue;
809+ vml->zap_gen = my_gen;
810+ if (!mmget_not_zero(vml->vma->vm_mm))
811+ continue;
812+ mm = vml->vma->vm_mm;
813+ vm_start = vml->vma->vm_start;
814+ break;
815+ }
816+ if (!mm)
817+ break;
818+ mutex_unlock(&arena->lock);
819+
820+ mmap_read_lock(mm);
821+ /*
822+ * Re-resolve: while we waited the VMA could have been unmapped
823+ * and a different mapping installed at the same address.
824+ */
825+ vma = find_vma(mm, vm_start);
826+ if (vma && vma->vm_start == vm_start &&
827+ vma->vm_file && vma->vm_file->private_data == &arena->map)
828+ zap_vma_range(vma, uaddr, size);
829+ mmap_read_unlock(mm);
830+ mmput(mm);
831+
832+ mutex_lock(&arena->lock);
833+ }
834+ mutex_unlock(&arena->lock);
835+ mutex_unlock(&arena->zap_mutex);
677836 }
678837
679838 static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable)
@@ -685,6 +844,7 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, · 5 unmodified lines
685844 struct llist_head free_pages;
686845 struct llist_node *pos, *t;
687846 struct arena_free_span *s;
847+ struct clear_range_data cdata;
688848 unsigned long flags;
689849 int ret = 0;
690850
@@ -713,9 +873,11 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, · 22 unmodified lines
713873 range_tree_set(&arena->rt, pgoff, page_cnt);
714874
715875 init_llist_head(&free_pages);
876+ cdata.free_pages = &free_pages;
877+ cdata.scratch_page = arena->scratch_page;
716878 /* clear ptes and collect struct pages */
717879 apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,
718− apply_range_clear_cb, &free_pages);
880+ apply_range_clear_cb, &cdata);
719881
720882 /* drop the lock to do the tlb flush and zap pages */
721883 raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
@@ -805,6 +967,7 @@ static void arena_free_worker(struct work_struct *work) · 83 unmodified lines
805967 struct arena_free_span *s;
806968 u64 arena_vm_start, user_vm_start;
807969 struct llist_head free_pages;
970+ struct clear_range_data cdata;
808971 struct page *page;
809972 unsigned long full_uaddr;
810973 long kaddr, page_cnt, pgoff;
@@ -818,6 +981,8 @@ static void arena_free_worker(struct work_struct *work) · 7 unmodified lines
818981 bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
819982
820983 init_llist_head(&free_pages);
984+ cdata.free_pages = &free_pages;
985+ cdata.scratch_page = arena->scratch_page;
821986 arena_vm_start = bpf_arena_get_kern_vm_start(arena);
822987 user_vm_start = bpf_arena_get_user_vm_start(arena);
823988
@@ -830,7 +995,7 @@ static void arena_free_worker(struct work_struct *work) · 6 unmodified lines
830995
831996 /* clear ptes and collect pages in free_pages llist */
832997 apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,
833− apply_range_clear_cb, &free_pages);
998+ apply_range_clear_cb, &cdata);
834999
8351000 range_tree_set(&arena->rt, pgoff, page_cnt);
8361001 }
@@ -893,6 +1058,19 @@ void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 pag · 56 unmodified lines
8931058
8941059 return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, false);
8951060 }
1061+
1062+void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
1063+ int node_id, u64 flags)
1064+{
1065+ struct bpf_map *map = p__map;
1066+ struct bpf_arena *arena = container_of(map, struct bpf_arena, map);
1067+
1068+ if (map->map_type != BPF_MAP_TYPE_ARENA || flags || !page_cnt)
1069+ return NULL;
1070+
1071+ return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, true);
1072+}
1073+
8961074 __bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt)
8971075 {
8981076 struct bpf_map *map = p__map;
@@ -945,23 +1123,12 @@ static int __init kfunc_init(void) · 46 unmodified lines
9451123 }
9461124 late_initcall(kfunc_init);
9471125
948−void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip)
1126+static void __bpf_prog_report_arena_violation(struct bpf_prog *prog, bool write,
1127+ unsigned long addr, unsigned long fault_ip)
9491128 {
9501129 struct bpf_stream_stage ss;
951− struct bpf_prog *prog;
9521130 u64 user_vm_start;
9531131
954− /*
955− * The RCU read lock is held to safely traverse the latch tree, but we
956− * don't need its protection when accessing the prog, since it will not
957− * disappear while we are handling the fault.
958− */
959− rcu_read_lock();
960− prog = bpf_prog_ksym_find(fault_ip);
961− rcu_read_unlock();
962− if (!prog)
963− return;
964−
9651132 /* Use main prog for stream access */
9661133 prog = prog->aux->main_prog_aux->prog;
9671134
@@ -974,3 +1141,53 @@ void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned lo · 6 unmodified lines
9741141 bpf_stream_dump_stack(ss);
9751142 }));
9761143 }
1144+
1145+bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, unsigned long fault_ip)
1146+{
1147+ struct bpf_arena *arena;
1148+ struct bpf_prog *prog;
1149+ unsigned long kbase;
1150+ unsigned long page_addr = addr & PAGE_MASK;
1151+
1152+ prog = bpf_prog_find_from_stack();
1153+ if (!prog)
1154+ return false;
1155+
1156+ arena = prog->aux->arena;
1157+ /* a prog not using arena may be on stack, so arena can be NULL */
1158+ if (!arena)
1159+ return false;
1160+
1161+ kbase = bpf_arena_get_kern_vm_start(arena);
1162+
1163+ /*
1164+ * Recovery covers the 4 GiB mappable band plus the upper half-guard.
1165+ * Lower guard is unreachable from kfuncs; an address there indicates
1166+ * a different bug class - leave it to the regular kernel oops path.
1167+ */
1168+ if (page_addr < kbase || page_addr >= kbase + SZ_4G + GUARD_SZ / 2)
1169+ return false;
1170+
1171+ apply_to_page_range(&init_mm, page_addr, PAGE_SIZE,
1172+ apply_range_set_scratch_cb, arena->scratch_page);
1173+ flush_vmap_cache(page_addr, PAGE_SIZE);
1174+ __bpf_prog_report_arena_violation(prog, is_write, page_addr - kbase, fault_ip);
1175+ return true;
1176+}
1177+
1178+void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip)
1179+{
1180+ struct bpf_prog *prog;
1181+
1182+ /*
1183+ * The RCU read lock is held to safely traverse the latch tree, but we
1184+ * don't need its protection when accessing the prog, since it will not
1185+ * disappear while we are handling the fault.
1186+ */
1187+ rcu_read_lock();
1188+ prog = bpf_prog_ksym_find(fault_ip);
1189+ rcu_read_unlock();
1190+ if (!prog)
1191+ return;
1192+ __bpf_prog_report_arena_violation(prog, write, addr, fault_ip);
1193+}
kernel/bpf/arraymap.c+6−8
@@ -175,14 +175,12 @@ static void *array_map_lookup_elem(struct bpf_map *map, void *key) · 174 unmodified lines
175175 return array->value + (u64)array->elem_size * (index & array->index_mask);
176176 }
177177
178−static int array_map_get_hash(struct bpf_map *map, u32 hash_buf_size,
179− void *hash_buf)
178+static int array_map_get_hash(struct bpf_map *map)
180179 {
181180 struct bpf_array *array = container_of(map, struct bpf_array, map);
182181
183182 sha256(array->value, (u64)array->elem_size * array->map.max_entries,
184− hash_buf);
185− memcpy(array->map.sha, hash_buf, sizeof(array->map.sha));
183+ array->map.sha);
186184 return 0;
187185 }
188186
@@ -386,7 +384,7 @@ static long array_map_update_elem(struct bpf_map *map, void *key, void *value, · 197 unmodified lines
386384 if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
387385 val = this_cpu_ptr(array->pptrs[index & array->index_mask]);
388386 copy_map_value(map, val, value);
389− bpf_obj_free_fields(array->map.record, val);
387+ bpf_obj_cancel_fields(map, val);
390388 } else {
391389 val = array->value +
392390 (u64)array->elem_size * (index & array->index_mask);
@@ -394,7 +392,7 @@ static long array_map_update_elem(struct bpf_map *map, void *key, void *value, · 1 unmodified line
394392 copy_map_value_locked(map, val, value, false);
395393 else
396394 copy_map_value(map, val, value);
397− bpf_obj_free_fields(array->map.record, val);
395+ bpf_obj_cancel_fields(map, val);
398396 }
399397 return 0;
400398 }
@@ -434,14 +432,14 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, · 33 unmodified lines
434432 cpu = map_flags >> 32;
435433 ptr = per_cpu_ptr(pptr, cpu);
436434 copy_map_value(map, ptr, value);
437− bpf_obj_free_fields(array->map.record, ptr);
435+ bpf_obj_cancel_fields(map, ptr);
438436 goto unlock;
439437 }
440438 for_each_possible_cpu(cpu) {
441439 ptr = per_cpu_ptr(pptr, cpu);
442440 val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu;
443441 copy_map_value(map, ptr, val);
444− bpf_obj_free_fields(array->map.record, ptr);
442+ bpf_obj_cancel_fields(map, ptr);
445443 }
446444 unlock:
447445 rcu_read_unlock();
kernel/bpf/backtrack.c+66−16
@@ -9,7 +9,7 @@ · 8 unmodified lines
99
1010 /* for any branch, call, exit record the history of jmps in the given state */
1111 int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
12− int insn_flags, u64 linked_regs)
12+ int insn_flags, int spi, int frame, u64 linked_regs)
1313 {
1414 u32 cnt = cur->jmp_history_cnt;
1515 struct bpf_jmp_history_entry *p;
@@ -25,6 +25,8 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state · 9 unmodified lines
2525 env, "insn history: insn_idx %d cur flags %x new flags %x",
2626 env->insn_idx, env->cur_hist_ent->flags, insn_flags);
2727 env->cur_hist_ent->flags |= insn_flags;
28+ env->cur_hist_ent->spi = spi;
29+ env->cur_hist_ent->frame = frame;
2830 verifier_bug_if(env->cur_hist_ent->linked_regs != 0, env,
2931 "insn history: insn_idx %d linked_regs: %#llx",
3032 env->insn_idx, env->cur_hist_ent->linked_regs);
@@ -43,6 +45,8 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state · 12 unmodified lines
4345 p->idx = env->insn_idx;
4446 p->prev_idx = env->prev_insn_idx;
4547 p->flags = insn_flags;
48+ p->spi = spi;
49+ p->frame = frame;
4650 p->linked_regs = linked_regs;
4751 cur->jmp_history_cnt = cnt;
4852 env->cur_hist_ent = p;
@@ -64,16 +68,6 @@ static bool is_atomic_fetch_insn(const struct bpf_insn *insn) · 15 unmodified lines
6468 (insn->imm & BPF_FETCH);
6569 }
6670
67−static int insn_stack_access_spi(int insn_flags)
68−{
69− return (insn_flags >> INSN_F_SPI_SHIFT) & INSN_F_SPI_MASK;
70−}
71−
72−static int insn_stack_access_frameno(int insn_flags)
73−{
74− return insn_flags & INSN_F_FRAMENO_MASK;
75−}
76−
7771 /* Backtrack one insn at a time. If idx is not at the top of recorded
7872 * history then previous instruction came from straight line execution.
7973 * Return -ENOENT if we exhausted all instructions within given state.
@@ -135,11 +129,21 @@ static inline u32 bt_empty(struct backtrack_state *bt) · 55 unmodified lines
135129 int i;
136130
137131 for (i = 0; i <= bt->frame; i++)
138− mask |= bt->reg_masks[i] | bt->stack_masks[i];
132+ mask |= bt->reg_masks[i] | bt->stack_masks[i] | bt->stack_arg_masks[i];
139133
140134 return mask == 0;
141135 }
142136
137+static inline void bt_clear_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
138+{
139+ bt->stack_arg_masks[frame] &= ~(1 << slot);
140+}
141+
142+static inline bool bt_is_frame_stack_arg_slot_set(struct backtrack_state *bt, u32 frame, u32 slot)
143+{
144+ return bt->stack_arg_masks[frame] & (1 << slot);
145+}
146+
143147 static inline int bt_subprog_enter(struct backtrack_state *bt)
144148 {
145149 if (bt->frame == MAX_CALL_FRAMES - 1) {
@@ -200,6 +204,11 @@ static inline u64 bt_stack_mask(struct backtrack_state *bt) · 54 unmodified lines
200204 return bt->stack_masks[bt->frame];
201205 }
202206
207+static inline u8 bt_stack_arg_mask(struct backtrack_state *bt)
208+{
209+ return bt->stack_arg_masks[bt->frame];
210+}
211+
203212 static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg)
204213 {
205214 return bt->reg_masks[bt->frame] & (1 << reg);
@@ -341,6 +350,19 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 135 unmodified lines
341350 return 0;
342351 bt_clear_reg(bt, load_reg);
343352
353+ if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) {
354+ spi = hist->spi;
355+ /*
356+ * Stack arg read: callee reads from r11+off, but
357+ * the data lives in the caller's stack_arg_regs.
358+ * Set the mask in the caller frame so precision
359+ * is marked in the caller's slot at the callee
360+ * entry checkpoint.
361+ */
362+ bt_set_frame_stack_arg_slot(bt, bt->frame - 1, spi);
363+ return 0;
364+ }
365+
344366 /* scalars can only be spilled into stack w/o losing precision.
345367 * Load from any other memory can be zero extended.
346368 * The desire to keep that precision is already indicated
@@ -353,8 +375,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 6 unmodified lines
353375 * that [fp - off] slot contains scalar that needs to be
354376 * tracked with precision
355377 */
356− spi = insn_stack_access_spi(hist->flags);
357− fr = insn_stack_access_frameno(hist->flags);
378+ spi = hist->spi;
379+ fr = hist->frame;
358380 bpf_bt_set_frame_slot(bt, fr, spi);
359381 } else if (class == BPF_STX || class == BPF_ST) {
360382 if (bt_is_reg_set(bt, dreg))
@@ -363,11 +385,22 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 2 unmodified lines
363385 * encountered a case of pointer subtraction.
364386 */
365387 return -ENOTSUPP;
388+
389+ if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) {
390+ spi = hist->spi;
391+ if (!bt_is_frame_stack_arg_slot_set(bt, bt->frame, spi))
392+ return 0;
393+ bt_clear_frame_stack_arg_slot(bt, bt->frame, spi);
394+ if (class == BPF_STX)
395+ bt_set_reg(bt, sreg);
396+ return 0;
397+ }
398+
366399 /* scalars can only be spilled into stack */
367400 if (!hist || !(hist->flags & INSN_F_STACK_ACCESS))
368401 return 0;
369− spi = insn_stack_access_spi(hist->flags);
370− fr = insn_stack_access_frameno(hist->flags);
402+ spi = hist->spi;
403+ fr = hist->frame;
371404 if (!bt_is_frame_slot_set(bt, fr, spi))
372405 return 0;
373406 bt_clear_frame_slot(bt, fr, spi);
@@ -431,6 +464,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 57 unmodified lines
431464 bpf_bt_set_frame_reg(bt, bt->frame - 1, i);
432465 }
433466 }
467+ if (bt_stack_arg_mask(bt)) {
468+ verifier_bug(env,
469+ "static subprog leftover stack arg slots %x",
470+ bt_stack_arg_mask(bt));
471+ return -EFAULT;
472+ }
434473 if (bt_subprog_exit(bt))
435474 return -EFAULT;
436475 return 0;
@@ -901,6 +940,17 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env, · 464 unmodified lines
901940 *changed = true;
902941 }
903942 }
943+ for (i = 0; i < func->out_stack_arg_cnt; i++) {
944+ if (!bt_is_frame_stack_arg_slot_set(bt, fr, i))
945+ continue;
946+ reg = &func->stack_arg_regs[i];
947+ if (reg->type != SCALAR_VALUE || reg->precise) {
948+ bt_clear_frame_stack_arg_slot(bt, fr, i);
949+ } else {
950+ reg->precise = true;
951+ *changed = true;
952+ }
953+ }
904954 if (env->log.level & BPF_LOG_LEVEL2) {
905955 fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
906956 bt_frame_reg_mask(bt, fr));
kernel/bpf/bpf_lru_list.c+100−65
@@ -13,23 +13,8 @@ · 12 unmodified lines
1313 #define PERCPU_FREE_TARGET (4)
1414 #define PERCPU_NR_SCANS PERCPU_FREE_TARGET
1515
16−/* Helpers to get the local list index */
17−#define LOCAL_LIST_IDX(t) ((t) - BPF_LOCAL_LIST_T_OFFSET)
18−#define LOCAL_FREE_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_FREE)
19−#define LOCAL_PENDING_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_PENDING)
2016 #define IS_LOCAL_LIST_TYPE(t) ((t) >= BPF_LOCAL_LIST_T_OFFSET)
2117
22−/* Local list helpers */
23−static struct list_head *local_free_list(struct bpf_lru_locallist *loc_l)
24−{
25− return &loc_l->lists[LOCAL_FREE_LIST_IDX];
26−}
27−
28−static struct list_head *local_pending_list(struct bpf_lru_locallist *loc_l)
29−{
30− return &loc_l->lists[LOCAL_PENDING_LIST_IDX];
31−}
32−
3318 /* bpf_lru_node helpers */
3419 static bool bpf_lru_node_is_ref(const struct bpf_lru_node *node)
3520 {
@@ -72,6 +57,7 @@ static void __bpf_lru_node_move_to_free(struct bpf_lru_list *l, · 36 unmodified lines
7257 bpf_lru_list_count_dec(l, node->type);
7358
7459 node->type = tgt_free_type;
60+ WRITE_ONCE(node->pending_free, 0);
7561 list_move(&node->list, free_list);
7662 }
7763
@@ -87,6 +73,9 @@ static void __bpf_lru_node_move_in(struct bpf_lru_list *l, · 9 unmodified lines
8773 bpf_lru_list_count_inc(l, tgt_type);
8874 node->type = tgt_type;
8975 bpf_lru_node_clear_ref(node);
76+ /* Reset pending_free only when moving to the free list */
77+ if (tgt_type == BPF_LRU_LIST_T_FREE)
78+ WRITE_ONCE(node->pending_free, 0);
9079 list_move(&node->list, &l->lists[tgt_type]);
9180 }
9281
@@ -212,9 +201,11 @@ __bpf_lru_list_shrink_inactive(struct bpf_lru *lru, · 119 unmodified lines
212201 unsigned int i = 0;
213202
214203 list_for_each_entry_safe_reverse(node, tmp_node, inactive, list) {
215− if (bpf_lru_node_is_ref(node)) {
204+ if (bpf_lru_node_is_ref(node) &&
205+ !READ_ONCE(node->pending_free)) {
216206 __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_ACTIVE);
217− } else if (lru->del_from_htab(lru->del_arg, node)) {
207+ } else if (READ_ONCE(node->pending_free) ||
208+ lru->del_from_htab(lru->del_arg, node)) {
218209 __bpf_lru_node_move_to_free(l, node, free_list,
219210 tgt_free_type);
220211 if (++nshrinked == tgt_nshrink)
@@ -273,7 +264,8 @@ static unsigned int __bpf_lru_list_shrink(struct bpf_lru *lru, · 52 unmodified lines
273264
274265 list_for_each_entry_safe_reverse(node, tmp_node, force_shrink_list,
275266 list) {
276− if (lru->del_from_htab(lru->del_arg, node)) {
267+ if (READ_ONCE(node->pending_free) ||
268+ lru->del_from_htab(lru->del_arg, node)) {
277269 __bpf_lru_node_move_to_free(l, node, free_list,
278270 tgt_free_type);
279271 return 1;
@@ -290,8 +282,10 @@ static void __local_list_flush(struct bpf_lru_list *l, · 10 unmodified lines
290282 struct bpf_lru_node *node, *tmp_node;
291283
292284 list_for_each_entry_safe_reverse(node, tmp_node,
293− local_pending_list(loc_l), list) {
294− if (bpf_lru_node_is_ref(node))
285+ &loc_l->pending_list, list) {
286+ if (READ_ONCE(node->pending_free))
287+ __bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_FREE);
288+ else if (bpf_lru_node_is_ref(node))
295289 __bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_ACTIVE);
296290 else
297291 __bpf_lru_node_move_in(l, node,
@@ -307,9 +301,12 @@ static void bpf_lru_list_push_free(struct bpf_lru_list *l, · 9 unmodified lines
307301 if (WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(node->type)))
308302 return;
309303
310− raw_spin_lock_irqsave(&l->lock, flags);
304+ if (raw_res_spin_lock_irqsave(&l->lock, flags)) {
305+ WRITE_ONCE(node->pending_free, 1);
306+ return;
307+ }
311308 __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE);
312− raw_spin_unlock_irqrestore(&l->lock, flags);
309+ raw_res_spin_unlock_irqrestore(&l->lock, flags);
313310 }
314311
315312 static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru,
@@ -318,8 +315,10 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, · 2 unmodified lines
318315 struct bpf_lru_list *l = &lru->common_lru.lru_list;
319316 struct bpf_lru_node *node, *tmp_node;
320317 unsigned int nfree = 0;
318+ LIST_HEAD(tmp_free);
321319
322− raw_spin_lock(&l->lock);
320+ if (raw_res_spin_lock(&l->lock))
321+ return;
323322
324323 __local_list_flush(l, loc_l);
325324
@@ -327,7 +326,7 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, · 1 unmodified line
327326
328327 list_for_each_entry_safe(node, tmp_node, &l->lists[BPF_LRU_LIST_T_FREE],
329328 list) {
330− __bpf_lru_node_move_to_free(l, node, local_free_list(loc_l),
329+ __bpf_lru_node_move_to_free(l, node, &tmp_free,
331330 BPF_LRU_LOCAL_LIST_T_FREE);
332331 if (++nfree == lru->target_free)
333332 break;
@@ -335,10 +334,19 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, · 1 unmodified line
335334
336335 if (nfree < lru->target_free)
337336 __bpf_lru_list_shrink(lru, l, lru->target_free - nfree,
338− local_free_list(loc_l),
337+ &tmp_free,
339338 BPF_LRU_LOCAL_LIST_T_FREE);
340339
341− raw_spin_unlock(&l->lock);
340+ raw_res_spin_unlock(&l->lock);
341+
342+ /*
343+ * Transfer the harvested nodes from the temporary list_head into
344+ * the lockless per-CPU free llist.
345+ */
346+ list_for_each_entry_safe(node, tmp_node, &tmp_free, list) {
347+ list_del(&node->list);
348+ llist_add(&node->llist, &loc_l->free_llist);
349+ }
342350 }
343351
344352 static void __local_list_add_pending(struct bpf_lru *lru,
@@ -350,22 +358,21 @@ static void __local_list_add_pending(struct bpf_lru *lru, · 5 unmodified lines
350358 *(u32 *)((void *)node + lru->hash_offset) = hash;
351359 node->cpu = cpu;
352360 node->type = BPF_LRU_LOCAL_LIST_T_PENDING;
361+ WRITE_ONCE(node->pending_free, 0);
353362 bpf_lru_node_clear_ref(node);
354− list_add(&node->list, local_pending_list(loc_l));
363+ list_add(&node->list, &loc_l->pending_list);
355364 }
356365
357366 static struct bpf_lru_node *
358367 __local_list_pop_free(struct bpf_lru_locallist *loc_l)
359368 {
360− struct bpf_lru_node *node;
369+ struct llist_node *llnode;
361370
362− node = list_first_entry_or_null(local_free_list(loc_l),
363− struct bpf_lru_node,
364− list);
365− if (node)
366− list_del(&node->list);
371+ llnode = llist_del_first(&loc_l->free_llist);
372+ if (!llnode)
373+ return NULL;
367374
368− return node;
375+ return container_of(llnode, struct bpf_lru_node, llist);
369376 }
370377
371378 static struct bpf_lru_node *
@@ -376,10 +383,10 @@ __local_list_pop_pending(struct bpf_lru *lru, struct bpf_lru_locallist *loc_l) · 4 unmodified lines
376383
377384 ignore_ref:
378385 /* Get from the tail (i.e. older element) of the pending list. */
379− list_for_each_entry_reverse(node, local_pending_list(loc_l),
380− list) {
386+ list_for_each_entry_reverse(node, &loc_l->pending_list, list) {
381387 if ((!bpf_lru_node_is_ref(node) || force) &&
382− lru->del_from_htab(lru->del_arg, node)) {
388+ (READ_ONCE(node->pending_free) ||
389+ lru->del_from_htab(lru->del_arg, node))) {
383390 list_del(&node->list);
384391 return node;
385392 }
@@ -404,7 +411,8 @@ static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru, · 18 unmodified lines
404411
405412 l = per_cpu_ptr(lru->percpu_lru, cpu);
406413
407− raw_spin_lock_irqsave(&l->lock, flags);
414+ if (raw_res_spin_lock_irqsave(&l->lock, flags))
415+ return NULL;
408416
409417 __bpf_lru_list_rotate(lru, l);
410418
@@ -420,7 +428,7 @@ static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru, · 9 unmodified lines
420428 __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_INACTIVE);
421429 }
422430
423− raw_spin_unlock_irqrestore(&l->lock, flags);
431+ raw_res_spin_unlock_irqrestore(&l->lock, flags);
424432
425433 return node;
426434 }
@@ -437,7 +445,8 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru, · 10 unmodified lines
437445
438446 loc_l = per_cpu_ptr(clru->local_list, cpu);
439447
440− raw_spin_lock_irqsave(&loc_l->lock, flags);
448+ if (raw_res_spin_lock_irqsave(&loc_l->lock, flags))
449+ return NULL;
441450
442451 node = __local_list_pop_free(loc_l);
443452 if (!node) {
@@ -448,17 +457,22 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru, · 4 unmodified lines
448457 if (node)
449458 __local_list_add_pending(lru, loc_l, cpu, node, hash);
450459
451− raw_spin_unlock_irqrestore(&loc_l->lock, flags);
460+ raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);
452461
453462 if (node)
454463 return node;
455464
456− /* No free nodes found from the local free list and
465+ /*
466+ * No free nodes found from the local free list and
457467 * the global LRU list.
458468 *
459469 * Steal from the local free/pending list of the
460470 * current CPU and remote CPU in RR. It starts
461471 * with the loc_l->next_steal CPU.
472+ *
473+ * Acquire the victim's lock before touching either list. On
474+ * acquisition failure (rqspinlock AA or timeout) skip the victim
475+ * and try the next CPU.
462476 */
463477
464478 first_steal = loc_l->next_steal;
@@ -466,24 +480,36 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru, · 1 unmodified line
466480 do {
467481 steal_loc_l = per_cpu_ptr(clru->local_list, steal);
468482
469− raw_spin_lock_irqsave(&steal_loc_l->lock, flags);
470−
471− node = __local_list_pop_free(steal_loc_l);
472− if (!node)
473− node = __local_list_pop_pending(lru, steal_loc_l);
474−
475− raw_spin_unlock_irqrestore(&steal_loc_l->lock, flags);
483+ if (!raw_res_spin_lock_irqsave(&steal_loc_l->lock, flags)) {
484+ node = __local_list_pop_free(steal_loc_l);
485+ if (!node)
486+ node = __local_list_pop_pending(lru, steal_loc_l);
487+ raw_res_spin_unlock_irqrestore(&steal_loc_l->lock, flags);
488+ }
476489
477490 steal = cpumask_next_wrap(steal, cpu_possible_mask);
478491 } while (!node && steal != first_steal);
479492
480493 loc_l->next_steal = steal;
481494
482− if (node) {
483− raw_spin_lock_irqsave(&loc_l->lock, flags);
484− __local_list_add_pending(lru, loc_l, cpu, node, hash);
485− raw_spin_unlock_irqrestore(&loc_l->lock, flags);
495+ if (!node)
496+ return NULL;
497+
498+ if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) {
499+ /*
500+ * The local pending lock can't be acquired (rqspinlock AA
501+ * or timeout). Return the stolen node to the per-CPU
502+ * free_llist instead of orphaning it; the next pop_free on
503+ * this CPU will pick it up.
504+ */
505+ node->type = BPF_LRU_LOCAL_LIST_T_FREE;
506+ bpf_lru_node_clear_ref(node);
507+ WRITE_ONCE(node->pending_free, 0);
508+ llist_add(&node->llist, &loc_l->free_llist);
509+ return NULL;
486510 }
511+ __local_list_add_pending(lru, loc_l, cpu, node, hash);
512+ raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);
487513
488514 return node;
489515 }
@@ -511,18 +537,24 @@ static void bpf_common_lru_push_free(struct bpf_lru *lru, · 21 unmodified lines
511537
512538 loc_l = per_cpu_ptr(lru->common_lru.local_list, node->cpu);
513539
514− raw_spin_lock_irqsave(&loc_l->lock, flags);
540+ if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) {
541+ WRITE_ONCE(node->pending_free, 1);
542+ return;
543+ }
515544
516545 if (unlikely(node->type != BPF_LRU_LOCAL_LIST_T_PENDING)) {
517− raw_spin_unlock_irqrestore(&loc_l->lock, flags);
546+ raw_res_spin_unlock_irqrestore(&loc_l->lock,
547+ flags);
518548 goto check_lru_list;
519549 }
520550
521551 node->type = BPF_LRU_LOCAL_LIST_T_FREE;
522552 bpf_lru_node_clear_ref(node);
523− list_move(&node->list, local_free_list(loc_l));
553+ list_del(&node->list);
554+
555+ raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);
524556
525− raw_spin_unlock_irqrestore(&loc_l->lock, flags);
557+ llist_add(&node->llist, &loc_l->free_llist);
526558 return;
527559 }
528560
@@ -538,11 +570,14 @@ static void bpf_percpu_lru_push_free(struct bpf_lru *lru, · 9 unmodified lines
538570
539571 l = per_cpu_ptr(lru->percpu_lru, node->cpu);
540572
541− raw_spin_lock_irqsave(&l->lock, flags);
573+ if (raw_res_spin_lock_irqsave(&l->lock, flags)) {
574+ WRITE_ONCE(node->pending_free, 1);
575+ return;
576+ }
542577
543578 __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE);
544579
545− raw_spin_unlock_irqrestore(&l->lock, flags);
580+ raw_res_spin_unlock_irqrestore(&l->lock, flags);
546581 }
547582
548583 void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node)
@@ -565,6 +600,7 @@ static void bpf_common_lru_populate(struct bpf_lru *lru, void *buf, · 16 unmodified lines
565600
566601 node = (struct bpf_lru_node *)(buf + node_offset);
567602 node->type = BPF_LRU_LIST_T_FREE;
603+ node->pending_free = 0;
568604 bpf_lru_node_clear_ref(node);
569605 list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]);
570606 buf += elem_size;
@@ -594,6 +630,7 @@ again: · 23 unmodified lines
594630 node = (struct bpf_lru_node *)(buf + node_offset);
595631 node->cpu = cpu;
596632 node->type = BPF_LRU_LIST_T_FREE;
633+ node->pending_free = 0;
597634 bpf_lru_node_clear_ref(node);
598635 list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]);
599636 i++;
@@ -618,14 +655,12 @@ void bpf_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset, · 18 unmodified lines
618655
619656 static void bpf_lru_locallist_init(struct bpf_lru_locallist *loc_l, int cpu)
620657 {
621− int i;
622−
623− for (i = 0; i < NR_BPF_LRU_LOCAL_LIST_T; i++)
624− INIT_LIST_HEAD(&loc_l->lists[i]);
658+ INIT_LIST_HEAD(&loc_l->pending_list);
659+ init_llist_head(&loc_l->free_llist);
625660
626661 loc_l->next_steal = cpu;
627662
628− raw_spin_lock_init(&loc_l->lock);
663+ raw_res_spin_lock_init(&loc_l->lock);
629664 }
630665
631666 static void bpf_lru_list_init(struct bpf_lru_list *l)
@@ -640,7 +675,7 @@ static void bpf_lru_list_init(struct bpf_lru_list *l) · 8 unmodified lines
640675
641676 l->next_inactive_rotation = &l->lists[BPF_LRU_LIST_T_INACTIVE];
642677
643− raw_spin_lock_init(&l->lock);
678+ raw_res_spin_lock_init(&l->lock);
644679 }
645680
646681 int bpf_lru_init(struct bpf_lru *lru, bool percpu, u32 hash_offset,
kernel/bpf/bpf_lru_list.h+19−6
@@ -6,11 +6,11 @@ · 5 unmodified lines
66
77 #include <linux/cache.h>
88 #include <linux/list.h>
9−#include <linux/spinlock_types.h>
9+#include <linux/llist.h>
10+#include <asm/rqspinlock.h>
1011
1112 #define NR_BPF_LRU_LIST_T (3)
1213 #define NR_BPF_LRU_LIST_COUNT (2)
13−#define NR_BPF_LRU_LOCAL_LIST_T (2)
1414 #define BPF_LOCAL_LIST_T_OFFSET NR_BPF_LRU_LIST_T
1515
1616 enum bpf_lru_list_type {
@@ -22,10 +22,22 @@ enum bpf_lru_list_type { · 5 unmodified lines
2222 };
2323
2424 struct bpf_lru_node {
25− struct list_head list;
25+ /*
26+ * A node is in at most one list at a time. The free path on the
27+ * per-CPU locallist uses an llist, so share storage via a union.
28+ */
29+ union {
30+ struct list_head list;
31+ struct llist_node llist;
32+ };
2633 u16 cpu;
2734 u8 type;
2835 u8 ref;
36+ /*
37+ * Marks nodes whose *_push_free() lock acquire failed; reclaimed
38+ * by flush/shrink which honor the flag instead of del_from_htab().
39+ */
40+ u8 pending_free;
2941 };
3042
3143 struct bpf_lru_list {
@@ -34,13 +46,14 @@ struct bpf_lru_list { · 2 unmodified lines
3446 /* The next inactive list rotation starts from here */
3547 struct list_head *next_inactive_rotation;
3648
37− raw_spinlock_t lock ____cacheline_aligned_in_smp;
49+ rqspinlock_t lock ____cacheline_aligned_in_smp;
3850 };
3951
4052 struct bpf_lru_locallist {
41− struct list_head lists[NR_BPF_LRU_LOCAL_LIST_T];
53+ struct list_head pending_list;
54+ struct llist_head free_llist;
4255 u16 next_steal;
43− raw_spinlock_t lock;
56+ rqspinlock_t lock;
4457 };
4558
4659 struct bpf_common_lru {
kernel/bpf/bpf_lsm.c+20−0
@@ -427,6 +427,26 @@ BTF_ID(func, bpf_lsm_audit_rule_known) · 426 unmodified lines
427427 BTF_ID(func, bpf_lsm_inode_xattr_skipcap)
428428 BTF_SET_END(bool_lsm_hooks)
429429
430+/* hooks returning void */
431+#define LSM_HOOK_void(DEFAULT, NAME, ...) BTF_ID(func, bpf_lsm_##NAME)
432+#define LSM_HOOK_int(DEFAULT, NAME, ...) /* nothing */
433+#define LSM_HOOK(RET, DEFAULT, NAME, ...) LSM_HOOK_##RET(DEFAULT, NAME, __VA_ARGS__)
434+BTF_SET_START(void_lsm_hooks)
435+#include <linux/lsm_hook_defs.h>
436+#undef LSM_HOOK
437+#undef LSM_HOOK_void
438+#undef LSM_HOOK_int
439+BTF_SET_END(void_lsm_hooks)
440+
441+bool bpf_lsm_hook_returns_errno(u32 btf_id)
442+{
443+ if (btf_id_set_contains(&bool_lsm_hooks, btf_id))
444+ return false;
445+ if (btf_id_set_contains(&void_lsm_hooks, btf_id))
446+ return false;
447+ return true;
448+}
449+
430450 int bpf_lsm_get_retval_range(const struct bpf_prog *prog,
431451 struct bpf_retval_range *retval_range)
432452 {
kernel/bpf/bpf_struct_ops.c+50−13
@@ -594,8 +594,8 @@ const struct bpf_link_ops bpf_struct_ops_link_lops = { · 593 unmodified lines
594594 .dealloc = bpf_struct_ops_link_dealloc,
595595 };
596596
597−int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
598− struct bpf_tramp_link *link,
597+int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes,
598+ struct bpf_tramp_node *node,
599599 const struct btf_func_model *model,
600600 void *stub_func,
601601 void **_image, u32 *_image_off,
@@ -605,13 +605,13 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, · 3 unmodified lines
605605 void *image = *_image;
606606 int size;
607607
608− tlinks[BPF_TRAMP_FENTRY].links[0] = link;
609− tlinks[BPF_TRAMP_FENTRY].nr_links = 1;
608+ tnodes[BPF_TRAMP_FENTRY].nodes[0] = node;
609+ tnodes[BPF_TRAMP_FENTRY].nr_nodes = 1;
610610
611611 if (model->ret_size > 0)
612612 flags |= BPF_TRAMP_F_RET_FENTRY_RET;
613613
614− size = arch_bpf_trampoline_size(model, flags, tlinks, stub_func);
614+ size = arch_bpf_trampoline_size(model, flags, tnodes, stub_func);
615615 if (size <= 0)
616616 return size ? : -EFAULT;
617617
@@ -628,7 +628,7 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, · 10 unmodified lines
628628
629629 size = arch_prepare_bpf_trampoline(NULL, image + image_off,
630630 image + image_off + size,
631− model, flags, tlinks, stub_func);
631+ model, flags, tnodes, stub_func);
632632 if (size <= 0) {
633633 if (image != *_image)
634634 bpf_struct_ops_image_free(image);
@@ -693,7 +693,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 58 unmodified lines
693693 const struct btf_type *module_type;
694694 const struct btf_member *member;
695695 const struct btf_type *t = st_ops_desc->type;
696− struct bpf_tramp_links *tlinks;
696+ struct bpf_tramp_nodes *tnodes;
697697 void *udata, *kdata;
698698 int prog_fd, err;
699699 u32 i, trampoline_start, image_off = 0;
@@ -720,8 +720,8 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 20 unmodified lines
720720 if (uvalue->common.state || refcount_read(&uvalue->common.refcnt))
721721 return -EINVAL;
722722
723− tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX);
724− if (!tlinks)
723+ tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX);
724+ if (!tnodes)
725725 return -ENOMEM;
726726
727727 uvalue = (struct bpf_struct_ops_value *)st_map->uvalue;
@@ -817,8 +817,9 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 89 unmodified lines
817817 err = -ENOMEM;
818818 goto reset_unlock;
819819 }
820− bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS,
821− &bpf_struct_ops_link_lops, prog, prog->expected_attach_type);
820+ bpf_tramp_link_init(link, BPF_LINK_TYPE_STRUCT_OPS,
821+ &bpf_struct_ops_link_lops, prog, prog->expected_attach_type, 0);
822+
822823 *plink++ = &link->link;
823824
824825 /* Poison pointer on error instead of return for backward compatibility */
@@ -832,7 +833,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 7 unmodified lines
832833 *pksym++ = ksym;
833834
834835 trampoline_start = image_off;
835− err = bpf_struct_ops_prepare_trampoline(tlinks, link,
836+ err = bpf_struct_ops_prepare_trampoline(tnodes, &link->node,
836837 &st_ops->func_models[i],
837838 *(void **)(st_ops->cfi_stubs + moff),
838839 &image, &image_off,
@@ -911,7 +912,7 @@ reset_unlock: · 72 unmodified lines
911912 memset(uvalue, 0, map->value_size);
912913 memset(kvalue, 0, map->value_size);
913914 unlock:
914− kfree(tlinks);
915+ kfree(tnodes);
915916 mutex_unlock(&st_map->lock);
916917 if (!err)
917918 bpf_struct_ops_map_add_ksyms(st_map);
@@ -1204,6 +1205,42 @@ u32 bpf_struct_ops_id(const void *kdata) · 286 unmodified lines
12041205 }
12051206 EXPORT_SYMBOL_GPL(bpf_struct_ops_id);
12061207
1208+/**
1209+ * bpf_struct_ops_for_each_prog - Invoke @cb for each member prog
1210+ * @kdata: kernel-side struct_ops vmtable (the @kdata arg to ->reg/->update/->unreg)
1211+ * @cb: callback invoked once per member prog; non-zero return stops iteration
1212+ * @data: opaque argument passed to @cb
1213+ *
1214+ * Walks the struct_ops member progs registered on the map containing @kdata.
1215+ * Intended for use from struct_ops ->reg() callbacks (and similar) that need to
1216+ * inspect the loaded BPF programs (for example to discover maps they reference
1217+ * via @prog->aux->used_maps).
1218+ *
1219+ * Return 0 if iteration completed, otherwise the first non-zero @cb return.
1220+ */
1221+int bpf_struct_ops_for_each_prog(const void *kdata,
1222+ int (*cb)(struct bpf_prog *prog, void *data),
1223+ void *data)
1224+{
1225+ struct bpf_struct_ops_value *kvalue;
1226+ struct bpf_struct_ops_map *st_map;
1227+ u32 i;
1228+ int ret;
1229+
1230+ kvalue = container_of(kdata, struct bpf_struct_ops_value, data);
1231+ st_map = container_of(kvalue, struct bpf_struct_ops_map, kvalue);
1232+
1233+ for (i = 0; i < st_map->funcs_cnt; i++) {
1234+ if (!st_map->links[i])
1235+ continue;
1236+ ret = cb(st_map->links[i]->prog, data);
1237+ if (ret)
1238+ return ret;
1239+ }
1240+ return 0;
1241+}
1242+EXPORT_SYMBOL_GPL(bpf_struct_ops_for_each_prog);
1243+
12071244 static bool bpf_struct_ops_valid_to_reg(struct bpf_map *map)
12081245 {
12091246 struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;
kernel/bpf/btf.c+219−99
@@ -182,7 +182,6 @@ · 181 unmodified lines
182182 #define BITS_ROUNDUP_BYTES(bits) \
183183 (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))
184184
185−#define BTF_INFO_MASK 0x9f00ffff
186185 #define BTF_INT_MASK 0x0fffffff
187186 #define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE)
188187 #define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET)
@@ -289,7 +288,7 @@ enum verifier_phase { · 100 unmodified lines
289288 struct resolve_vertex {
290289 const struct btf_type *t;
291290 u32 type_id;
292− u16 next_member;
291+ u32 next_member;
293292 };
294293
295294 enum visit_state {
@@ -2031,7 +2030,7 @@ static int env_stack_push(struct btf_verifier_env *env, · 1735 unmodified lines
20312030 }
20322031
20332032 static void env_stack_set_next_member(struct btf_verifier_env *env,
2034− u16 next_member)
2033+ u32 next_member)
20352034 {
20362035 env->stack[env->top_stack - 1].next_member = next_member;
20372036 }
@@ -3293,7 +3292,7 @@ static s32 btf_struct_check_meta(struct btf_verifier_env *env, · 1255 unmodified lines
32933292 struct btf *btf = env->btf;
32943293 u32 struct_size = t->size;
32953294 u32 offset;
3296− u16 i;
3295+ u32 i;
32973296
32983297 meta_needed = btf_type_vlen(t) * sizeof(*member);
32993298 if (meta_left < meta_needed) {
@@ -3369,7 +3368,7 @@ static int btf_struct_resolve(struct btf_verifier_env *env, · 69 unmodified lines
33693368 {
33703369 const struct btf_member *member;
33713370 int err;
3372− u16 i;
3371+ u32 i;
33733372
33743373 /* Before continue resolving the next_member,
33753374 * ensure the last member is indeed resolved to a
@@ -3668,7 +3667,7 @@ end: · 292 unmodified lines
36683667 static int btf_repeat_fields(struct btf_field_info *info, int info_cnt,
36693668 u32 field_cnt, u32 repeat_cnt, u32 elem_size)
36703669 {
3671− u32 i, j;
3670+ u32 i, j, total_cnt, total_repeats;
36723671 u32 cur;
36733672
36743673 /* Ensure not repeating fields that should not be repeated. */
@@ -3686,10 +3685,9 @@ static int btf_repeat_fields(struct btf_field_info *info, int info_cnt, · 11 unmodified lines
36863685 }
36873686 }
36883687
3689− /* The type of struct size or variable size is u32,
3690− * so the multiplication will not overflow.
3691− */
3692− if (field_cnt * (repeat_cnt + 1) > info_cnt)
3688+ if (check_add_overflow(repeat_cnt, 1, &total_repeats) ||
3689+ check_mul_overflow(field_cnt, total_repeats, &total_cnt) ||
3690+ total_cnt > (u32)info_cnt)
36933691 return -E2BIG;
36943692
36953693 cur = field_cnt;
@@ -4447,7 +4445,7 @@ static s32 btf_enum_check_meta(struct btf_verifier_env *env, · 751 unmodified lines
44474445 const struct btf_enum *enums = btf_type_enum(t);
44484446 struct btf *btf = env->btf;
44494447 const char *fmt_str;
4450− u16 i, nr_enums;
4448+ u32 i, nr_enums;
44514449 u32 meta_needed;
44524450
44534451 nr_enums = btf_type_vlen(t);
@@ -4555,7 +4553,7 @@ static s32 btf_enum64_check_meta(struct btf_verifier_env *env, · 101 unmodified lines
45554553 const struct btf_enum64 *enums = btf_type_enum64(t);
45564554 struct btf *btf = env->btf;
45574555 const char *fmt_str;
4558− u16 i, nr_enums;
4556+ u32 i, nr_enums;
45594557 u32 meta_needed;
45604558
45614559 nr_enums = btf_type_vlen(t);
@@ -4683,7 +4681,7 @@ static void btf_func_proto_log(struct btf_verifier_env *env, · 121 unmodified lines
46834681 const struct btf_type *t)
46844682 {
46854683 const struct btf_param *args = (const struct btf_param *)(t + 1);
4686− u16 nr_args = btf_type_vlen(t), i;
4684+ u32 nr_args = btf_type_vlen(t), i;
46874685
46884686 btf_verifier_log(env, "return=%u args=(", t->type);
46894687 if (!nr_args) {
@@ -4929,7 +4927,7 @@ static int btf_datasec_resolve(struct btf_verifier_env *env, · 239 unmodified lines
49294927 {
49304928 const struct btf_var_secinfo *vsi;
49314929 struct btf *btf = env->btf;
4932− u16 i;
4930+ u32 i;
49334931
49344932 env->resolve_mode = RESOLVE_TBD;
49354933 for_each_vsi_from(i, v->next_member, v->t, vsi) {
@@ -5183,7 +5181,7 @@ static int btf_func_proto_check(struct btf_verifier_env *env, · 247 unmodified lines
51835181 const struct btf_type *ret_type;
51845182 const struct btf_param *args;
51855183 const struct btf *btf;
5186− u16 nr_args, i;
5184+ u32 nr_args, i;
51875185 int err;
51885186
51895187 btf = env->btf;
@@ -5278,7 +5276,7 @@ static int btf_func_check(struct btf_verifier_env *env, · 88 unmodified lines
52785276 const struct btf_type *proto_type;
52795277 const struct btf_param *args;
52805278 const struct btf *btf;
5281− u16 nr_args, i;
5279+ u32 nr_args, i;
52825280
52835281 btf = env->btf;
52845282 proto_type = btf_type_by_id(btf, t->type);
@@ -5336,12 +5334,6 @@ static s32 btf_check_meta(struct btf_verifier_env *env, · 51 unmodified lines
53365334 }
53375335 meta_left -= sizeof(*t);
53385336
5339− if (t->info & ~BTF_INFO_MASK) {
5340− btf_verifier_log(env, "[%u] Invalid btf_info:%x",
5341− env->log_type_id, t->info);
5342− return -EINVAL;
5343− }
5344−
53455337 if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX ||
53465338 BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) {
53475339 btf_verifier_log(env, "[%u] Invalid kind:%u",
@@ -5914,25 +5906,10 @@ static int btf_check_type_tags(struct btf_verifier_env *env, · 566 unmodified lines
59145906 return 0;
59155907 }
59165908
5917−static int finalize_log(struct bpf_verifier_log *log, bpfptr_t uattr, u32 uattr_size)
5918−{
5919− u32 log_true_size;
5920− int err;
5921−
5922− err = bpf_vlog_finalize(log, &log_true_size);
5923−
5924− if (uattr_size >= offsetofend(union bpf_attr, btf_log_true_size) &&
5925− copy_to_bpfptr_offset(uattr, offsetof(union bpf_attr, btf_log_true_size),
5926− &log_true_size, sizeof(log_true_size)))
5927− err = -EFAULT;
5928−
5929− return err;
5930−}
5931−
5932−static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
5909+static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr,
5910+ struct bpf_log_attr *attr_log)
59335911 {
59345912 bpfptr_t btf_data = make_bpfptr(attr->btf, uattr.is_kernel);
5935− char __user *log_ubuf = u64_to_user_ptr(attr->btf_log_buf);
59365913 struct btf_struct_metas *struct_meta_tab;
59375914 struct btf_verifier_env *env = NULL;
59385915 struct btf *btf = NULL;
@@ -5949,8 +5926,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat · 10 unmodified lines
59495926 /* user could have requested verbose verifier output
59505927 * and supplied buffer to store the verification trace
59515928 */
5952− err = bpf_vlog_init(&env->log, attr->btf_log_level,
5953− log_ubuf, attr->btf_log_size);
5929+ err = bpf_vlog_init(&env->log, attr_log->level, attr_log->ubuf, attr_log->size);
59545930 if (err)
59555931 goto errout_free;
59565932
@@ -6015,7 +5991,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat · 58 unmodified lines
60155991 }
60165992 }
60175993
6018− err = finalize_log(&env->log, uattr, uattr_size);
5994+ err = bpf_log_attr_finalize(attr_log, &env->log);
60195995 if (err)
60205996 goto errout_free;
60215997
@@ -6027,7 +6003,7 @@ errout_meta: · 5 unmodified lines
60276003 btf_free_struct_meta_tab(btf);
60286004 errout:
60296005 /* overwrite err with -ENOSPC or -EFAULT */
6030− ret = finalize_log(&env->log, uattr, uattr_size);
6006+ ret = bpf_log_attr_finalize(attr_log, &env->log);
60316007 if (ret)
60326008 err = ret;
60336009 errout_free:
@@ -6980,7 +6956,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 946 unmodified lines
69806956 info->reg_type = ctx_arg_info->reg_type;
69816957 info->btf = ctx_arg_info->btf ? : btf_vmlinux;
69826958 info->btf_id = ctx_arg_info->btf_id;
6983− info->ref_obj_id = ctx_arg_info->ref_obj_id;
6959+ info->ref_id = ctx_arg_info->ref_id;
69846960 return true;
69856961 }
69866962 }
@@ -7825,6 +7801,134 @@ enum btf_arg_tag { · 838 unmodified lines
78257801 ARG_TAG_ARENA = BIT_ULL(5),
78267802 };
78277803
7804+static int btf_scan_decl_tags(struct bpf_verifier_env *env,
7805+ const struct btf *btf,
7806+ const struct btf_type *fn_t,
7807+ u32 arg_idx, bool is_global, u32 *tags)
7808+{
7809+ int id = btf_named_start_id(btf, false) - 1;
7810+ const char tag_key[] = "arg:";
7811+ static const struct {
7812+ const char *tag_value;
7813+ enum btf_arg_tag arg_tag;
7814+ } tag_values[] = {
7815+ { "ctx", ARG_TAG_CTX },
7816+ { "trusted", ARG_TAG_TRUSTED },
7817+ { "untrusted", ARG_TAG_UNTRUSTED },
7818+ { "nonnull", ARG_TAG_NONNULL },
7819+ { "nullable", ARG_TAG_NULLABLE },
7820+ { "arena", ARG_TAG_ARENA },
7821+ };
7822+
7823+ /*
7824+ * The 'arg:<tag>' decl_tag takes precedence over the derivation
7825+ * of the register type from the BTF type itself.
7826+ */
7827+ while ((id = btf_find_next_decl_tag(btf, fn_t, arg_idx, tag_key, id)) > 0) {
7828+ const struct btf_type *tag_t;
7829+ const char *tag;
7830+ int i;
7831+ bool found;
7832+
7833+ /* disallow arg tags in static subprogs */
7834+ if (!is_global) {
7835+ bpf_log(&env->log,
7836+ "arg#%d type tag is not supported in static functions\n",
7837+ arg_idx);
7838+ return -EOPNOTSUPP;
7839+ }
7840+
7841+ tag_t = btf_type_by_id(btf, id);
7842+ tag = __btf_name_by_offset(btf, tag_t->name_off) + (sizeof(tag_key) - 1);
7843+
7844+ found = false;
7845+ for (i = 0; i < ARRAY_SIZE(tag_values); ++i) {
7846+ if (!strcmp(tag, tag_values[i].tag_value)) {
7847+ *tags |= tag_values[i].arg_tag;
7848+ found = true;
7849+ break;
7850+ }
7851+ }
7852+
7853+ if (!found) {
7854+ bpf_log(&env->log, "arg#%d has unsupported set of tags\n", arg_idx);
7855+ return -EOPNOTSUPP;
7856+ }
7857+ }
7858+ if (id != -ENOENT) {
7859+ bpf_log(&env->log, "arg#%d type tag fetching failure: %d\n", arg_idx, id);
7860+ return id;
7861+ }
7862+
7863+ return 0;
7864+}
7865+
7866+static int btf_scan_type_tags(struct bpf_verifier_env *env,
7867+ const struct btf *btf, u32 type_id,
7868+ u32 *tags)
7869+{
7870+ const struct btf_type *t;
7871+
7872+ /* Find the first pointer type in the chain. */
7873+ t = btf_type_skip_modifiers(btf, type_id, NULL);
7874+
7875+ /*
7876+ * We currently reject type tags on non-pointer types,
7877+ * which neither LLVM nor GCC support anyway.
7878+ */
7879+ if (!t || !btf_type_is_ptr(t))
7880+ return 0;
7881+
7882+ /* We got a pointer, get all associated type tags. */
7883+ for (t = btf_type_by_id(btf, t->type); t && btf_type_is_modifier(t);
7884+ t = btf_type_by_id(btf, t->type)) {
7885+
7886+ /* Skip non-type tag modifiers. */
7887+ if (!btf_type_is_type_tag(t))
7888+ continue;
7889+
7890+ const char *tag = __btf_name_by_offset(btf, t->name_off);
7891+
7892+ if (strcmp(tag, "arena") == 0) {
7893+ *tags |= ARG_TAG_ARENA;
7894+ } else {
7895+ bpf_log(&env->log, "function signature member has unsupported type tag '%s'\n",
7896+ tag);
7897+ return -EOPNOTSUPP;
7898+ }
7899+ }
7900+
7901+ return 0;
7902+}
7903+
7904+/* Check whether the type is a valid return type. */
7905+static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,
7906+ const struct btf_type *t, int subprog)
7907+{
7908+ u32 tags = 0;
7909+ int err;
7910+
7911+ err = btf_scan_type_tags(env, btf, t->type, &tags);
7912+ if (err)
7913+ return err;
7914+
7915+ t = btf_type_skip_modifiers(btf, t->type, NULL);
7916+
7917+ /*
7918+ * We allow all subprogs except for the main one to return any kind of arena pointer.
7919+ * General arena variables are not allowed, since it makes no sense to return by value
7920+ * a variable that's on the heap in the first place.
7921+ */
7922+ if (subprog && (tags & ARG_TAG_ARENA) && btf_type_is_ptr(t))
7923+ return 0;
7924+
7925+ /* We always accept void or scalars. */
7926+ if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
7927+ return 0;
7928+
7929+ return -EOPNOTSUPP;
7930+}
7931+
78287932 /* Process BTF of a function to produce high-level expectation of function
78297933 * arguments (like ARG_PTR_TO_CTX, or ARG_PTR_TO_MEM, etc). This information
78307934 * is cached in subprog info for reuse.
@@ -7843,6 +7947,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 12 unmodified lines
78437947 struct btf *btf = prog->aux->btf;
78447948 const struct btf_param *args;
78457949 const struct btf_type *t, *ref_t, *fn_t;
7950+ int err;
78467951 u32 i, nargs, btf_id;
78477952 const char *tname;
78487953
@@ -7887,25 +7992,36 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 38 unmodified lines
78877992 }
78887993 args = (const struct btf_param *)(t + 1);
78897994 nargs = btf_type_vlen(t);
7995+ sub->arg_cnt = nargs;
7996+ if (nargs > MAX_BPF_FUNC_ARGS) {
7997+ bpf_log(log, "kernel supports at most %d parameters, function %s has %d\n",
7998+ MAX_BPF_FUNC_ARGS, tname, nargs);
7999+ return -EFAULT;
8000+ }
78908001 if (nargs > MAX_BPF_FUNC_REG_ARGS) {
7891− if (!is_global)
7892− return -EINVAL;
7893− bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",
8002+ if (!bpf_jit_supports_stack_args()) {
8003+ bpf_log(log, "JIT does not support function %s() with %d args\n",
8004+ tname, nargs);
8005+ return -EFAULT;
8006+ }
8007+ sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
8008+ }
8009+
8010+ if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
8011+ bpf_log(log, "global function %s has %d > %d args, stack args not supported\n",
78948012 tname, nargs, MAX_BPF_FUNC_REG_ARGS);
78958013 return -EINVAL;
78968014 }
7897− /* check that function is void or returns int, exception cb also requires this */
7898− t = btf_type_by_id(btf, t->type);
7899− while (btf_type_is_modifier(t))
7900− t = btf_type_by_id(btf, t->type);
7901− if (!btf_type_is_void(t) && !btf_type_is_int(t) && !btf_is_any_enum(t)) {
7902− if (!is_global)
7903− return -EINVAL;
7904− bpf_log(log,
7905− "Global function %s() return value not void or scalar. "
7906− "Only those are supported.\n",
7907− tname);
7908− return -EINVAL;
8015+
8016+ err = btf_validate_return_type(env, btf, t, subprog);
8017+ if (err) {
8018+ if (is_global) {
8019+ bpf_log(log,
8020+ "Global function %s() return value not void or scalar. "
8021+ "Only those are supported.\n",
8022+ tname);
8023+ }
8024+ return err;
79098025 }
79108026
79118027 /* Convert BTF function arguments into verifier types.
@@ -7913,42 +8029,13 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 1 unmodified line
79138029 */
79148030 for (i = 0; i < nargs; i++) {
79158031 u32 tags = 0;
7916− int id = btf_named_start_id(btf, false) - 1;
7917−
7918− /* 'arg:<tag>' decl_tag takes precedence over derivation of
7919− * register type from BTF type itself
7920− */
7921− while ((id = btf_find_next_decl_tag(btf, fn_t, i, "arg:", id)) > 0) {
7922− const struct btf_type *tag_t = btf_type_by_id(btf, id);
7923− const char *tag = __btf_name_by_offset(btf, tag_t->name_off) + 4;
7924−
7925− /* disallow arg tags in static subprogs */
7926− if (!is_global) {
7927− bpf_log(log, "arg#%d type tag is not supported in static functions\n", i);
7928− return -EOPNOTSUPP;
7929− }
8032+ err = btf_scan_decl_tags(env, btf, fn_t, i, is_global, &tags);
8033+ if (err)
8034+ return err;
79308035
7931− if (strcmp(tag, "ctx") == 0) {
7932− tags |= ARG_TAG_CTX;
7933− } else if (strcmp(tag, "trusted") == 0) {
7934− tags |= ARG_TAG_TRUSTED;
7935− } else if (strcmp(tag, "untrusted") == 0) {
7936− tags |= ARG_TAG_UNTRUSTED;
7937− } else if (strcmp(tag, "nonnull") == 0) {
7938− tags |= ARG_TAG_NONNULL;
7939− } else if (strcmp(tag, "nullable") == 0) {
7940− tags |= ARG_TAG_NULLABLE;
7941− } else if (strcmp(tag, "arena") == 0) {
7942− tags |= ARG_TAG_ARENA;
7943− } else {
7944− bpf_log(log, "arg#%d has unsupported set of tags\n", i);
7945− return -EOPNOTSUPP;
7946− }
7947− }
7948− if (id != -ENOENT) {
7949− bpf_log(log, "arg#%d type tag fetching failure: %d\n", i, id);
7950− return id;
7951− }
8036+ err = btf_scan_type_tags(env, btf, args[i].type, &tags);
8037+ if (err)
8038+ return err;
79528039
79538040 t = btf_type_by_id(btf, args[i].type);
79548041 while (btf_type_is_modifier(t))
@@ -7973,7 +8060,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 18 unmodified lines
79738060 bpf_log(log, "arg#%d has invalid combination of tags\n", i);
79748061 return -EINVAL;
79758062 }
7976− sub->args[i].arg_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY;
8063+ sub->args[i].arg_type = ARG_PTR_TO_DYNPTR;
79778064 continue;
79788065 }
79798066 if (tags & ARG_TAG_TRUSTED) {
@@ -8074,7 +8161,6 @@ skip_pointer: · 94 unmodified lines
80748161 return -EINVAL;
80758162 }
80768163
8077− sub->arg_cnt = nargs;
80788164 sub->args_cached = true;
80798165
80808166 return 0;
@@ -8196,12 +8282,12 @@ static int __btf_new_fd(struct btf *btf) · 115 unmodified lines
81968282 return anon_inode_getfd("btf", &btf_fops, btf, O_RDONLY | O_CLOEXEC);
81978283 }
81988284
8199−int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
8285+int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
82008286 {
82018287 struct btf *btf;
82028288 int ret;
82038289
8204− btf = btf_parse(attr, uattr, uattr_size);
8290+ btf = btf_parse(attr, uattr, attr_log);
82058291 if (IS_ERR(btf))
82068292 return PTR_ERR(btf);
82078293
@@ -8684,6 +8770,39 @@ static int btf_check_iter_kfuncs(struct btf *btf, const char *func_name, · 476 unmodified lines
86848770 return 0;
86858771 }
86868772
8773+static int btf_check_kfunc_name(struct btf *btf, const char *func_name, u32 kind)
8774+{
8775+#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
8776+ struct btf_module *btf_mod, *tmp;
8777+#endif
8778+ s32 id;
8779+
8780+ if (!btf_is_module(btf))
8781+ return 0;
8782+
8783+ id = btf_find_by_name_kind(bpf_get_btf_vmlinux(), func_name, kind);
8784+ if (id >= 0) {
8785+ pr_err("kfunc %s (id: %d) is already present in vmlinux.\n",
8786+ func_name, id);
8787+ return -EINVAL;
8788+ }
8789+
8790+#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
8791+ guard(mutex)(&btf_module_mutex);
8792+ list_for_each_entry_safe(btf_mod, tmp, &btf_modules, list) {
8793+ if (btf_mod->btf == btf)
8794+ continue;
8795+ id = btf_find_by_name_kind(btf_mod->btf, func_name, kind);
8796+ if (id >= 0) {
8797+ pr_err("kfunc %s (id: %d) is already present in module %s.\n",
8798+ func_name, id, btf_mod->module->name);
8799+ return -EINVAL;
8800+ }
8801+ }
8802+#endif
8803+ return 0;
8804+}
8805+
86878806 static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags)
86888807 {
86898808 const struct btf_type *func;
@@ -8697,7 +8816,8 @@ static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags) · 7 unmodified lines
86978816
86988817 /* sanity check kfunc name */
86998818 func_name = btf_name_by_offset(btf, func->name_off);
8700− if (!func_name || !func_name[0])
8819+ if (!func_name || !func_name[0] ||
8820+ btf_check_kfunc_name(btf, func_name, BTF_INFO_KIND(func->info)))
87018821 return -EINVAL;
87028822
87038823 func = btf_type_by_id(btf, func->type);
kernel/bpf/cgroup.c+44−21
@@ -55,6 +55,28 @@ void __init cgroup_bpf_lifetime_notifier_init(void) · 54 unmodified lines
5555 &cgroup_bpf_lifetime_nb));
5656 }
5757
58+#ifdef CONFIG_BPF_LSM
59+struct cgroup_lsm_atype {
60+ u32 attach_btf_id;
61+ int refcnt;
62+ bool returns_errno;
63+};
64+
65+static struct cgroup_lsm_atype cgroup_lsm_atype[CGROUP_LSM_NUM];
66+
67+static bool cgroup_bpf_hook_returns_errno(enum cgroup_bpf_attach_type atype)
68+{
69+ if (atype >= CGROUP_LSM_START && atype <= CGROUP_LSM_END)
70+ return READ_ONCE(cgroup_lsm_atype[atype - CGROUP_LSM_START].returns_errno);
71+ return true;
72+}
73+#else
74+static bool cgroup_bpf_hook_returns_errno(enum cgroup_bpf_attach_type atype)
75+{
76+ return true;
77+}
78+#endif
79+
5880 /* __always_inline is necessary to prevent indirect call through run_prog
5981 * function pointer.
6082 */
@@ -83,7 +105,8 @@ bpf_prog_run_array_cg(const struct cgroup_bpf *cgrp, · 22 unmodified lines
83105 *(ret_flags) |= (func_ret >> 1);
84106 func_ret &= 1;
85107 }
86− if (!func_ret && !IS_ERR_VALUE((long)run_ctx.retval))
108+ if (!func_ret && cgroup_bpf_hook_returns_errno(atype) &&
109+ !IS_ERR_VALUE((long)run_ctx.retval))
87110 run_ctx.retval = -EPERM;
88111 item++;
89112 }
@@ -156,13 +179,6 @@ unsigned int __cgroup_bpf_run_lsm_current(const void *ctx, · 66 unmodified lines
156179 }
157180
158181 #ifdef CONFIG_BPF_LSM
159−struct cgroup_lsm_atype {
160− u32 attach_btf_id;
161− int refcnt;
162−};
163−
164−static struct cgroup_lsm_atype cgroup_lsm_atype[CGROUP_LSM_NUM];
165−
166182 static enum cgroup_bpf_attach_type
167183 bpf_cgroup_atype_find(enum bpf_attach_type attach_type, u32 attach_btf_id)
168184 {
@@ -191,10 +207,13 @@ void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) · 22 unmodified lines
191207
192208 lockdep_assert_held(&cgroup_mutex);
193209
194− WARN_ON_ONCE(cgroup_lsm_atype[i].attach_btf_id &&
195− cgroup_lsm_atype[i].attach_btf_id != attach_btf_id);
196−
197− cgroup_lsm_atype[i].attach_btf_id = attach_btf_id;
210+ if (!cgroup_lsm_atype[i].attach_btf_id) {
211+ cgroup_lsm_atype[i].attach_btf_id = attach_btf_id;
212+ WRITE_ONCE(cgroup_lsm_atype[i].returns_errno,
213+ bpf_lsm_hook_returns_errno(attach_btf_id));
214+ } else {
215+ WARN_ON_ONCE(cgroup_lsm_atype[i].attach_btf_id != attach_btf_id);
216+ }
198217 cgroup_lsm_atype[i].refcnt++;
199218 }
200219
@@ -203,8 +222,10 @@ void bpf_cgroup_atype_put(int cgroup_atype) · 2 unmodified lines
203222 int i = cgroup_atype - CGROUP_LSM_START;
204223
205224 cgroup_lock();
206− if (--cgroup_lsm_atype[i].refcnt <= 0)
225+ if (--cgroup_lsm_atype[i].refcnt <= 0) {
226+ WRITE_ONCE(cgroup_lsm_atype[i].returns_errno, true);
207227 cgroup_lsm_atype[i].attach_btf_id = 0;
228+ }
208229 WARN_ON_ONCE(cgroup_lsm_atype[i].refcnt < 0);
209230 cgroup_unlock();
210231 }
@@ -1208,7 +1229,7 @@ static int cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog, · 997 unmodified lines
12081229
12091230 /* Must be called with cgroup_mutex held to avoid races. */
12101231 static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
1211− union bpf_attr __user *uattr)
1232+ union bpf_attr __user *uattr, u32 uattr_size)
12121233 {
12131234 __u32 __user *prog_attach_flags = u64_to_user_ptr(attr->query.prog_attach_flags);
12141235 bool effective_query = attr->query.query_flags & BPF_F_QUERY_EFFECTIVE;
@@ -1259,7 +1280,8 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, · 44 unmodified lines
12591280 return -EFAULT;
12601281 if (!effective_query && from_atype == to_atype)
12611282 revision = cgrp->bpf.revisions[from_atype];
1262− if (copy_to_user(&uattr->query.revision, &revision, sizeof(revision)))
1283+ if (uattr_size >= offsetofend(union bpf_attr, query.revision) &&
1284+ copy_to_user(&uattr->query.revision, &revision, sizeof(revision)))
12631285 return -EFAULT;
12641286 if (attr->query.prog_cnt == 0 || !prog_ids || !total_cnt)
12651287 /* return early if user requested only program count + flags */
@@ -1312,12 +1334,12 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, · 46 unmodified lines
13121334 }
13131335
13141336 static int cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
1315− union bpf_attr __user *uattr)
1337+ union bpf_attr __user *uattr, u32 uattr_size)
13161338 {
13171339 int ret;
13181340
13191341 cgroup_lock();
1320− ret = __cgroup_bpf_query(cgrp, attr, uattr);
1342+ ret = __cgroup_bpf_query(cgrp, attr, uattr, uattr_size);
13211343 cgroup_unlock();
13221344 return ret;
13231345 }
@@ -1520,7 +1542,7 @@ out_put_cgroup: · 196 unmodified lines
15201542 }
15211543
15221544 int cgroup_bpf_prog_query(const union bpf_attr *attr,
1523− union bpf_attr __user *uattr)
1545+ union bpf_attr __user *uattr, u32 uattr_size)
15241546 {
15251547 struct cgroup *cgrp;
15261548 int ret;
@@ -1529,7 +1551,7 @@ int cgroup_bpf_prog_query(const union bpf_attr *attr, · 2 unmodified lines
15291551 if (IS_ERR(cgrp))
15301552 return PTR_ERR(cgrp);
15311553
1532− ret = cgroup_bpf_query(cgrp, attr, uattr);
1554+ ret = cgroup_bpf_query(cgrp, attr, uattr, uattr_size);
15331555
15341556 cgroup_put(cgrp);
15351557 return ret;
@@ -1935,8 +1957,8 @@ int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head, · 399 unmodified lines
19351957
19361958 kfree(ctx.cur_val);
19371959
1938− if (ret == 1 && ctx.new_updated) {
1939− kfree(*buf);
1960+ if (!ret && ctx.new_updated) {
1961+ kvfree(*buf);
19401962 *buf = ctx.new_val;
19411963 *pcount = ctx.new_len;
19421964 } else {
@@ -2342,6 +2364,7 @@ BPF_CALL_3(bpf_sysctl_set_new_value, struct bpf_sysctl_kern *, ctx, · 399 unmodified lines
23422364 return -E2BIG;
23432365
23442366 memcpy(ctx->new_val, buf, buf_len);
2367+ ((char *)ctx->new_val)[buf_len] = '\0';
23452368 ctx->new_len = buf_len;
23462369 ctx->new_updated = 1;
23472370
kernel/bpf/cnum.c+120−0
@@ -0,0 +1,120 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
3+
4+#include <linux/bits.h>
5+
6+#define T 32
7+#include "cnum_defs.h"
8+#undef T
9+
10+#define T 64
11+#include "cnum_defs.h"
12+#undef T
13+
14+struct cnum32 cnum32_from_cnum64(struct cnum64 cnum)
15+{
16+ if (cnum64_is_empty(cnum))
17+ return CNUM32_EMPTY;
18+
19+ if (cnum.size >= U32_MAX)
20+ return (struct cnum32){ .base = 0, .size = U32_MAX };
21+ else
22+ return (struct cnum32){ .base = (u32)cnum.base, .size = cnum.size };
23+}
24+
25+/*
26+ * Suppose 'a' and 'b' are laid out as follows:
27+ *
28+ * 64-bit number axis --->
29+ *
30+ * N*2^32 (N+1)*2^32 (N+2)*2^32 (N+3)*2^32
31+ * ||------|---|=====|-------||----------|=====|-------||----------|=====|----|--||
32+ * | |< b >| |< b >| |< b >| |
33+ * | | | |
34+ * |<--+--------------------------- a ---------------------------+--->|
35+ * | |
36+ * |<-------------------------- t -------------------------->|
37+ *
38+ * In such a case it is possible to infer a more tight representation t
39+ * such that ∀ v ∈ a, (u32)v ∈ b: v ∈ t.
40+ */
41+struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b)
42+{
43+ /*
44+ * To simplify reasoning, rotate the circles so that [virtual] a1 starts
45+ * at u32 boundary, b1 represents b in this new frame of reference.
46+ */
47+ struct cnum32 b1 = { b.base - (u32)a.base, b.size };
48+ struct cnum64 t = a;
49+ u64 d, b1_max;
50+
51+ if (cnum64_is_empty(a) || cnum32_is_empty(b))
52+ return CNUM64_EMPTY;
53+
54+ if (cnum32_urange_overflow(b1)) {
55+ b1_max = (u32)b1.base + (u32)b1.size; /* overflow here is fine and necessary */
56+ if ((u32)a.size > b1_max && (u32)a.size < b1.base) {
57+ /*
58+ * N*2^32 (N+1)*2^32
59+ * ||=====|------------|=====||=====|---------|---|=====||
60+ * |b1 ->| |<- b1||b1 ->| | |<- b1|
61+ * |<----------------- a1 ------------------>|
62+ * |<-------------- t ------------>|<-- d -->| (after adjustment)
63+ * ^
64+ * b1_max
65+ */
66+ d = (u32)a.size - b1_max;
67+ t.size -= d;
68+ } else {
69+ /*
70+ * No adjustments possible in the following cases:
71+ *
72+ * ||=====|------------|=====||===|=|-------------|=|===||
73+ * |b1 ->| |<- b1||b1 +>| |<+ b1|
74+ * |<----------------- a1 ------>| |
75+ * |<----------------- (or) a1 ------------------->|
76+ */
77+ }
78+ } else {
79+ if (t.size < b1.base)
80+ /*
81+ * N*2^32 (N+1)*2^32
82+ * ||----------|--|=======|--||------>
83+ * |<-- a1 -->| |<- b ->|
84+ */
85+ return CNUM64_EMPTY;
86+ /*
87+ * N*2^32 (N+1)*2^32
88+ * ||-------------|========|-||-----| -------|========|-||
89+ * | |<- b1 ->| | |<- b1 ->|
90+ * |<------------+ a1 ------------>|
91+ * |<------ t ------>| (after adjustment)
92+ */
93+ t.base += b1.base;
94+ t.size -= b1.base;
95+ b1_max = b1.base + b1.size;
96+ d = 0;
97+ if ((u32)a.size < b1.base)
98+ /*
99+ * N*2^32 (N+1)*2^32
100+ * ||-------------|========|-||------|-------|========|-||
101+ * | |<- b1 ->| | |<- b1 ->|
102+ * |<------------+-- a1 --+-------->|
103+ * |<- t ->|<-- d -->| (after adjustment)
104+ */
105+ d = (u32)a.size + (BIT_ULL(32) - b1_max);
106+ else if ((u32)a.size >= b1_max)
107+ /*
108+ * N*2^32 (N+1)*2^32
109+ * ||--|========|------------||--|========|-------|-----||
110+ * | |<- b1 ->| |<- b1 ->| |
111+ * |<-+------------------ a1 ------------+------>|
112+ * |<-------------- t --------------->|<- d ->| (after adjustment)
113+ */
114+ d = (u32)a.size - b1_max;
115+ if (t.size < d)
116+ return CNUM64_EMPTY;
117+ t.size -= d;
118+ }
119+ return t;
120+}
kernel/bpf/cnum_defs.h+247−0
@@ -0,0 +1,247 @@
1+/* SPDX-License-Identifier: GPL-2.0-only */
2+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
3+
4+#ifndef T
5+#error "Define T (bit width: 32, 64) before including cnum_defs.h"
6+#endif
7+
8+#include <linux/cnum.h>
9+#include <linux/kernel.h>
10+#include <linux/limits.h>
11+#include <linux/minmax.h>
12+#include <linux/compiler_types.h>
13+
14+#define cnum_t __PASTE(cnum, T)
15+#define ut __PASTE(u, T)
16+#define st __PASTE(s, T)
17+#define UT_MAX __PASTE(__PASTE(U, T), _MAX)
18+#define ST_MAX __PASTE(__PASTE(S, T), _MAX)
19+#define ST_MIN __PASTE(__PASTE(S, T), _MIN)
20+#define EMPTY __PASTE(__PASTE(CNUM, T), _EMPTY)
21+#define FN(name) __PASTE(__PASTE(cnum, T), __PASTE(_, name))
22+
23+struct cnum_t FN(from_urange)(ut min, ut max)
24+{
25+ return (struct cnum_t){ .base = min, .size = (ut)max - min };
26+}
27+
28+struct cnum_t FN(from_srange)(st min, st max)
29+{
30+ ut size = (ut)max - (ut)min;
31+ ut base = size == UT_MAX ? 0 : (ut)min;
32+
33+ return (struct cnum_t){ .base = base, .size = size };
34+}
35+
36+/* True if this cnum represents two unsigned ranges. */
37+static inline bool FN(urange_overflow)(struct cnum_t cnum)
38+{
39+ /* Same as cnum.base + cnum.size > UT_MAX but avoids overflow */
40+ return cnum.size > UT_MAX - (ut)cnum.base;
41+}
42+
43+/*
44+ * cnum{T}_umin / cnum{T}_umax query an unsigned range represented by this cnum.
45+ * If cnum represents a range crossing the UT_MAX/0 boundary, the unbound range
46+ * [0..UT_MAX] is returned.
47+ */
48+ut FN(umin)(struct cnum_t cnum)
49+{
50+ return FN(urange_overflow)(cnum) ? 0 : cnum.base;
51+}
52+EXPORT_SYMBOL_GPL(FN(umin));
53+
54+ut FN(umax)(struct cnum_t cnum)
55+{
56+ return FN(urange_overflow)(cnum) ? UT_MAX : cnum.base + cnum.size;
57+}
58+EXPORT_SYMBOL_GPL(FN(umax));
59+
60+/* True if this cnum represents two signed ranges. */
61+static inline bool FN(srange_overflow)(struct cnum_t cnum)
62+{
63+ return FN(contains)(cnum, (ut)ST_MAX) && FN(contains)(cnum, (ut)ST_MIN);
64+}
65+
66+/*
67+ * cnum{T}_smin / cnum{T}_smax query a signed range represented by this cnum.
68+ * If cnum represents a range crossing the ST_MAX/ST_MIN boundary, the unbound range
69+ * [ST_MIN..ST_MAX] is returned.
70+ */
71+st FN(smin)(struct cnum_t cnum)
72+{
73+ return FN(srange_overflow)(cnum)
74+ ? ST_MIN
75+ : min((st)cnum.base, (st)(cnum.base + cnum.size));
76+}
77+
78+st FN(smax)(struct cnum_t cnum)
79+{
80+ return FN(srange_overflow)(cnum)
81+ ? ST_MAX
82+ : max((st)cnum.base, (st)(cnum.base + cnum.size));
83+}
84+
85+/*
86+ * Returns a possibly empty intersection of cnums 'a' and 'b'.
87+ * If 'a' and 'b' intersect in two sub-arcs, the function over-approximates
88+ * and returns either 'a' or 'b', whichever is smaller.
89+ */
90+struct cnum_t FN(intersect)(struct cnum_t a, struct cnum_t b)
91+{
92+ struct cnum_t b1;
93+ ut dbase;
94+
95+ if (FN(is_empty)(a) || FN(is_empty)(b))
96+ return EMPTY;
97+
98+ if (a.base > b.base)
99+ swap(a, b);
100+
101+ /*
102+ * Rotate frame of reference such that a.base is 0.
103+ * 'b1' is 'b' in this frame of reference.
104+ */
105+ dbase = b.base - a.base;
106+ b1 = (struct cnum_t){ dbase, b.size };
107+ if (FN(urange_overflow)(b1)) {
108+ if (b1.base <= a.size) {
109+ /*
110+ * Rotated frame (a.base at origin):
111+ *
112+ * 0 UT_MAX
113+ * |--------------------------------------------|
114+ * [=== a ==========================] |
115+ * [= b1 tail =] [========= b1 main ==========>]
116+ * ^-- b1.base <= a.size
117+ *
118+ * 'a' and 'b' intersect in two disjoint arcs,
119+ * can't represent as single cnum, over-approximate
120+ * the result.
121+ */
122+ return a.size <= b.size ? a : b;
123+ } else {
124+ /*
125+ * Rotated frame (a.base at origin):
126+ *
127+ * 0 UT_MAX
128+ * |--------------------------------------------|
129+ * [=== a =============] | |
130+ * [= b1 tail =] [======= b1 main ====>]
131+ * ^-- b1.base > a.size
132+ *
133+ * Only 'b' tail intersects 'a'.
134+ */
135+ return (struct cnum_t) {
136+ .base = a.base,
137+ .size = min(a.size, (ut)(b1.base + b1.size)),
138+ };
139+ }
140+ } else if (a.size >= b1.base) {
141+ /*
142+ * Rotated frame (a.base at origin):
143+ *
144+ * 0 UT_MAX
145+ * |--------------------------------------------------|
146+ * [=== a ==================================] |
147+ * [== b1 =====================]
148+ *
149+ * 0 UT_MAX
150+ * |--------------------------------------------------|
151+ * [=== a ==================================] |
152+ * [== b1 ====]
153+ * ^-- b1.base <= a.size
154+ * |<-- a.size - dbase -->|
155+ *
156+ * 'a' and 'b' intersect as one cnum.
157+ */
158+ return (struct cnum_t) {
159+ .base = b.base,
160+ .size = min((ut)(a.size - dbase), b.size),
161+ };
162+ } else {
163+ return EMPTY;
164+ }
165+}
166+
167+void FN(intersect_with)(struct cnum_t *dst, struct cnum_t src)
168+{
169+ *dst = FN(intersect)(*dst, src);
170+}
171+
172+void FN(intersect_with_urange)(struct cnum_t *dst, ut min, ut max)
173+{
174+ FN(intersect_with)(dst, FN(from_urange)(min, max));
175+}
176+
177+void FN(intersect_with_srange)(struct cnum_t *dst, st min, st max)
178+{
179+ FN(intersect_with)(dst, FN(from_srange)(min, max));
180+}
181+
182+static inline struct cnum_t FN(normalize)(struct cnum_t cnum)
183+{
184+ if (cnum.size == UT_MAX && cnum.base != 0 && cnum.base != (ut)ST_MAX)
185+ cnum.base = 0;
186+ return cnum;
187+}
188+
189+struct cnum_t FN(add)(struct cnum_t a, struct cnum_t b)
190+{
191+ if (FN(is_empty)(a) || FN(is_empty)(b))
192+ return EMPTY;
193+ if (a.size > UT_MAX - b.size)
194+ return (struct cnum_t){ 0, (ut)UT_MAX };
195+ else
196+ return FN(normalize)((struct cnum_t){ a.base + b.base, a.size + b.size });
197+}
198+
199+struct cnum_t FN(negate)(struct cnum_t a)
200+{
201+ if (FN(is_empty)(a))
202+ return EMPTY;
203+ return FN(normalize)((struct cnum_t){ -((ut)a.base + a.size), a.size });
204+}
205+
206+bool FN(is_empty)(struct cnum_t cnum)
207+{
208+ return cnum.base == EMPTY.base && cnum.size == EMPTY.size;
209+}
210+
211+bool FN(contains)(struct cnum_t cnum, ut v)
212+{
213+ if (FN(is_empty)(cnum))
214+ return false;
215+ if (FN(urange_overflow)(cnum))
216+ return v >= cnum.base || v <= (ut)cnum.base + cnum.size;
217+ else
218+ return v >= cnum.base && v <= (ut)cnum.base + cnum.size;
219+}
220+
221+bool FN(is_const)(struct cnum_t cnum)
222+{
223+ return cnum.size == 0;
224+}
225+
226+bool FN(is_subset)(struct cnum_t bigger, struct cnum_t smaller)
227+{
228+ if (FN(is_empty(smaller)))
229+ return true;
230+ if (FN(is_empty(bigger)))
231+ return false;
232+ /* rotate both arcs such that 'bigger' starts at origin, hence does not overflow */
233+ smaller.base -= bigger.base;
234+ bigger.base = 0;
235+ if (FN(urange_overflow)(smaller) && bigger.size < UT_MAX)
236+ return false;
237+ return smaller.base + smaller.size <= bigger.size;
238+}
239+
240+#undef EMPTY
241+#undef cnum_t
242+#undef ut
243+#undef st
244+#undef UT_MAX
245+#undef ST_MAX
246+#undef ST_MIN
247+#undef FN
kernel/bpf/const_fold.c+8−0
@@ -58,6 +58,14 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * · 57 unmodified lines
5858 u8 opcode = BPF_OP(insn->code) | BPF_SRC(insn->code);
5959 int r;
6060
61+ /* Stack arg stores (r11-based) are outside the tracked register set. */
62+ if (is_stack_arg_st(insn) || is_stack_arg_stx(insn))
63+ return;
64+ if (is_stack_arg_ldx(insn)) {
65+ ci_out[insn->dst_reg] = unknown;
66+ return;
67+ }
68+
6169 switch (class) {
6270 case BPF_ALU:
6371 case BPF_ALU64:
kernel/bpf/core.c+24−3
@@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from, · 1298 unmodified lines
12991299 u32 imm_rnd = get_random_u32();
13001300 s16 off;
13011301
1302− BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG);
1303− BUILD_BUG_ON(MAX_BPF_REG + 1 != MAX_BPF_JIT_REG);
1302+ BUILD_BUG_ON(BPF_REG_PARAMS + 2 != MAX_BPF_JIT_REG);
1303+ BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG);
13041304
13051305 /* Constraints on AX register:
13061306 *
@@ -1582,6 +1582,16 @@ bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struc · 275 unmodified lines
15821582 insn_idx += prog->aux->subprog_start;
15831583 return env->insn_aux_data[insn_idx].indirect_target;
15841584 }
1585+
1586+u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog)
1587+{
1588+ const struct bpf_subprog_info *sub;
1589+
1590+ if (!env)
1591+ return 0;
1592+ sub = &env->subprog_info[prog->aux->func_idx];
1593+ return sub->stack_arg_cnt - bpf_in_stack_arg_cnt(sub);
1594+}
15851595 #endif /* CONFIG_BPF_JIT */
15861596
15871597 /* Base function for offset calculation. Needs to go into .text section,
@@ -2471,7 +2481,7 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map, · 883 unmodified lines
24712481 cookie = aux->cgroup_storage[i] ?
24722482 aux->cgroup_storage[i]->cookie : 0;
24732483 ret = map->owner->storage_cookie[i] == cookie ||
2474− !cookie;
2484+ (!cookie && !aux->tail_call_reachable);
24752485 }
24762486 if (ret &&
24772487 map->owner->attach_func_proto != aux->attach_func_proto) {
@@ -3228,6 +3238,11 @@ bool __weak bpf_jit_supports_kfunc_call(void) · 750 unmodified lines
32283238 return false;
32293239 }
32303240
3241+bool __weak bpf_jit_supports_stack_args(void)
3242+{
3243+ return false;
3244+}
3245+
32313246 bool __weak bpf_jit_supports_far_kfunc_call(void)
32323247 {
32333248 return false;
@@ -3363,6 +3378,12 @@ __weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena) · 129 unmodified lines
33633378 }
33643379
33653380 #ifdef CONFIG_BPF_SYSCALL
3381+__weak bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write,
3382+ unsigned long fault_ip)
3383+{
3384+ return false;
3385+}
3386+
33663387 static int __init bpf_global_ma_init(void)
33673388 {
33683389 int ret;
kernel/bpf/devmap.c+19−0
@@ -581,6 +581,10 @@ static int dev_map_enqueue_clone(struct bpf_dtab_netdev *obj, · 580 unmodified lines
581581 {
582582 struct xdp_frame *nxdpf;
583583
584+ /* Frags live outside the linear frame and cannot be cloned safely. */
585+ if (unlikely(xdp_frame_has_frags(xdpf)))
586+ return -EOPNOTSUPP;
587+
584588 nxdpf = xdpf_clone(xdpf);
585589 if (!nxdpf)
586590 return -ENOMEM;
@@ -706,6 +710,18 @@ int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, · 119 unmodified lines
706710 if (unlikely(err))
707711 return err;
708712
713+ if (dst->xdp_prog && skb_cloned(skb)) {
714+ struct sk_buff *nskb;
715+
716+ nskb = skb_copy(skb, GFP_ATOMIC);
717+ if (!nskb)
718+ return -ENOMEM;
719+
720+ nskb->mac_len = skb->mac_len;
721+ consume_skb(skb);
722+ skb = nskb;
723+ }
724+
709725 /* Redirect has already succeeded semantically at this point, so we just
710726 * return 0 even if packet is dropped. Helper below takes care of
711727 * freeing skb.
@@ -726,6 +742,9 @@ static int dev_map_redirect_clone(struct bpf_dtab_netdev *dst, · 14 unmodified lines
726742 struct sk_buff *nskb;
727743 int err;
728744
745+ if (unlikely(skb_is_nonlinear(skb)))
746+ return -EOPNOTSUPP;
747+
729748 nskb = skb_clone(skb, GFP_ATOMIC);
730749 if (!nskb)
731750 return -ENOMEM;
kernel/bpf/fixups.c+24−3
@@ -870,7 +870,7 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env) · 869 unmodified lines
870870 case PTR_TO_BTF_ID:
871871 case PTR_TO_BTF_ID | PTR_UNTRUSTED:
872872 /* PTR_TO_BTF_ID | MEM_ALLOC always has a valid lifetime, unlike
873− * PTR_TO_BTF_ID, and an active ref_obj_id, but the same cannot
873+ * PTR_TO_BTF_ID, and an active referenced id, but the same cannot
874874 * be said once it is marked PTR_UNTRUSTED, hence we must handle
875875 * any faults for loads into such types. BPF_WRITE is disallowed
876876 * for this case.
@@ -1265,6 +1265,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 388 unmodified lines
12651265 prog->aux->real_func_cnt = env->subprog_cnt;
12661266 prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func;
12671267 prog->aux->exception_boundary = func[0]->aux->exception_boundary;
1268+ prog->aux->stack_arg_sp_adjust = func[0]->aux->stack_arg_sp_adjust;
12681269 bpf_prog_jit_attempt_done(prog);
12691270 return 0;
12701271 out_free:
@@ -1378,9 +1379,21 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env) · 107 unmodified lines
13781379 struct bpf_prog *prog = env->prog;
13791380 struct bpf_insn *insn = prog->insnsi;
13801381 bool has_kfunc_call = bpf_prog_has_kfunc_call(prog);
1381− int i, depth;
1382+ int depth;
13821383 #endif
1383− int err = 0;
1384+ int i, err = 0;
1385+
1386+ for (i = 0; i < env->subprog_cnt; i++) {
1387+ struct bpf_subprog_info *subprog = &env->subprog_info[i];
1388+ u16 outgoing = subprog->stack_arg_cnt - bpf_in_stack_arg_cnt(subprog);
1389+
1390+ if (subprog->max_out_stack_arg_cnt > outgoing) {
1391+ verbose(env,
1392+ "func#%d writes %u stack arg slots, but calls only require %u\n",
1393+ i, subprog->max_out_stack_arg_cnt, outgoing);
1394+ return -EINVAL;
1395+ }
1396+ }
13841397
13851398 if (env->prog->jit_requested &&
13861399 !bpf_prog_is_offloaded(env->prog->aux)) {
@@ -1395,6 +1408,12 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env) · 8 unmodified lines
13951408 verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
13961409 return -EINVAL;
13971410 }
1411+ for (i = 0; i < env->subprog_cnt; i++) {
1412+ if (bpf_in_stack_arg_cnt(&env->subprog_info[i])) {
1413+ verbose(env, "stack args are not supported in non-JITed programs\n");
1414+ return -EINVAL;
1415+ }
1416+ }
13981417 if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {
13991418 /* When JIT fails the progs with bpf2bpf calls and tail_calls
14001419 * have to be rejected, since interpreter doesn't support them yet.
@@ -2167,6 +2186,8 @@ patch_map_ops_generic: · 766 unmodified lines
21672186 insn->imm == BPF_FUNC_get_func_ret) {
21682187 if (eatype == BPF_TRACE_FEXIT ||
21692188 eatype == BPF_TRACE_FSESSION ||
2189+ eatype == BPF_TRACE_FEXIT_MULTI ||
2190+ eatype == BPF_TRACE_FSESSION_MULTI ||
21702191 eatype == BPF_MODIFY_RETURN) {
21712192 /* Load nr_args from ctx - 8 */
21722193 insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
kernel/bpf/hashtab.c+817−23
@@ -9,6 +9,7 @@ · 8 unmodified lines
99 #include <linux/rculist_nulls.h>
1010 #include <linux/rcupdate_wait.h>
1111 #include <linux/random.h>
12+#include <linux/rhashtable.h>
1213 #include <uapi/linux/btf.h>
1314 #include <linux/rcupdate_trace.h>
1415 #include <linux/btf_ids.h>
@@ -242,6 +243,10 @@ static void htab_free_prealloced_fields(struct bpf_htab *htab) · 227 unmodified lines
242243
243244 if (IS_ERR_OR_NULL(htab->map.record))
244245 return;
246+ /*
247+ * Preallocated maps do not have a bpf_mem_alloc destructor, so fully
248+ * destroy every element, including the extra elements.
249+ */
245250 if (htab_has_extra_elems(htab))
246251 num_entries += num_possible_cpus();
247252 for (i = 0; i < num_entries; i++) {
@@ -496,28 +501,26 @@ static void htab_dtor_ctx_free(void *ctx) · 248 unmodified lines
496501 kfree(ctx);
497502 }
498503
499−static int htab_set_dtor(struct bpf_htab *htab, void (*dtor)(void *, void *))
504+static int bpf_ma_set_dtor(struct bpf_map *map, struct bpf_mem_alloc *ma,
505+ void (*dtor)(void *, void *))
500506 {
501− u32 key_size = htab->map.key_size;
502− struct bpf_mem_alloc *ma;
503507 struct htab_btf_record *hrec;
504508 int err;
505509
506510 /* No need for dtors. */
507− if (IS_ERR_OR_NULL(htab->map.record))
511+ if (IS_ERR_OR_NULL(map->record))
508512 return 0;
509513
510514 hrec = kzalloc(sizeof(*hrec), GFP_KERNEL);
511515 if (!hrec)
512516 return -ENOMEM;
513− hrec->key_size = key_size;
514− hrec->record = btf_record_dup(htab->map.record);
517+ hrec->key_size = map->key_size;
518+ hrec->record = btf_record_dup(map->record);
515519 if (IS_ERR(hrec->record)) {
516520 err = PTR_ERR(hrec->record);
517521 kfree(hrec);
518522 return err;
519523 }
520− ma = htab_is_percpu(htab) ? &htab->pcpu_ma : &htab->ma;
521524 bpf_mem_alloc_set_dtor(ma, dtor, htab_dtor_ctx_free, hrec);
522525 return 0;
523526 }
@@ -534,9 +537,9 @@ static int htab_map_check_btf(struct bpf_map *map, const struct btf *btf, · 10 unmodified lines
534537 * populated in htab_map_alloc(), so it will always appear as NULL.
535538 */
536539 if (htab_is_percpu(htab))
537− return htab_set_dtor(htab, htab_pcpu_mem_dtor);
540+ return bpf_ma_set_dtor(map, &htab->pcpu_ma, htab_pcpu_mem_dtor);
538541 else
539− return htab_set_dtor(htab, htab_mem_dtor);
542+ return bpf_ma_set_dtor(map, &htab->ma, htab_mem_dtor);
540543 }
541544
542545 static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
@@ -834,8 +837,8 @@ static int htab_lru_map_gen_lookup(struct bpf_map *map, · 291 unmodified lines
834837 return insn - insn_buf;
835838 }
836839
837−static void check_and_free_fields(struct bpf_htab *htab,
838− struct htab_elem *elem)
840+static void check_and_cancel_fields(struct bpf_htab *htab,
841+ struct htab_elem *elem)
839842 {
840843 if (IS_ERR_OR_NULL(htab->map.record))
841844 return;
@@ -845,11 +848,11 @@ static void check_and_free_fields(struct bpf_htab *htab, · 3 unmodified lines
845848 int cpu;
846849
847850 for_each_possible_cpu(cpu)
848− bpf_obj_free_fields(htab->map.record, per_cpu_ptr(pptr, cpu));
851+ bpf_obj_cancel_fields(&htab->map, per_cpu_ptr(pptr, cpu));
849852 } else {
850853 void *map_value = htab_elem_value(elem, htab->map.key_size);
851854
852− bpf_obj_free_fields(htab->map.record, map_value);
855+ bpf_obj_cancel_fields(&htab->map, map_value);
853856 }
854857 }
855858
@@ -884,7 +887,7 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node) · 28 unmodified lines
884887 htab_unlock_bucket(b, flags);
885888
886889 if (l == tgt_l)
887− check_and_free_fields(htab, l);
890+ check_and_cancel_fields(htab, l);
888891 return l == tgt_l;
889892 }
890893
@@ -949,7 +952,7 @@ find_first_elem: · 58 unmodified lines
949952
950953 static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l)
951954 {
952− check_and_free_fields(htab, l);
955+ check_and_cancel_fields(htab, l);
953956
954957 if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH)
955958 bpf_mem_cache_free(&htab->pcpu_ma, l->ptr_to_pptr);
@@ -1002,7 +1005,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) · 46 unmodified lines
10021005
10031006 if (htab_is_prealloc(htab)) {
10041007 bpf_map_dec_elem_count(&htab->map);
1005− check_and_free_fields(htab, l);
1008+ check_and_cancel_fields(htab, l);
10061009 pcpu_freelist_push(&htab->freelist, &l->fnode);
10071010 } else {
10081011 dec_elem_count(htab);
@@ -1019,7 +1022,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 10 unmodified lines
10191022 /* copy true value_size bytes */
10201023 ptr = this_cpu_ptr(pptr);
10211024 copy_map_value(&htab->map, ptr, value);
1022− bpf_obj_free_fields(htab->map.record, ptr);
1025+ bpf_obj_cancel_fields(&htab->map, ptr);
10231026 } else {
10241027 u32 size = round_up(htab->map.value_size, 8);
10251028 void *val;
@@ -1029,7 +1032,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 3 unmodified lines
10291032 cpu = map_flags >> 32;
10301033 ptr = per_cpu_ptr(pptr, cpu);
10311034 copy_map_value(&htab->map, ptr, value);
1032− bpf_obj_free_fields(htab->map.record, ptr);
1035+ bpf_obj_cancel_fields(&htab->map, ptr);
10331036 return;
10341037 }
10351038
@@ -1037,7 +1040,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 1 unmodified line
10371040 ptr = per_cpu_ptr(pptr, cpu);
10381041 val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu;
10391042 copy_map_value(&htab->map, ptr, val);
1040− bpf_obj_free_fields(htab->map.record, ptr);
1043+ bpf_obj_cancel_fields(&htab->map, ptr);
10411044 }
10421045 }
10431046 }
@@ -1253,11 +1256,11 @@ static long htab_map_update_elem(struct bpf_map *map, void *key, void *value, · 209 unmodified lines
12531256 if (l_old) {
12541257 hlist_nulls_del_rcu(&l_old->hash_node);
12551258
1256− /* l_old has already been stashed in htab->extra_elems, free
1257− * its special fields before it is available for reuse.
1259+ /* l_old has already been stashed in htab->extra_elems, cancel
1260+ * its reusable special fields before it is available for reuse.
12581261 */
12591262 if (htab_is_prealloc(htab))
1260− check_and_free_fields(htab, l_old);
1263+ check_and_cancel_fields(htab, l_old);
12611264 }
12621265 htab_unlock_bucket(b, flags);
12631266 if (l_old && !htab_is_prealloc(htab))
@@ -1270,7 +1273,7 @@ err: · 6 unmodified lines
12701273
12711274 static void htab_lru_push_free(struct bpf_htab *htab, struct htab_elem *elem)
12721275 {
1273− check_and_free_fields(htab, elem);
1276+ check_and_cancel_fields(htab, elem);
12741277 bpf_map_dec_elem_count(&htab->map);
12751278 bpf_lru_push_free(&htab->lru, &elem->lru_node);
12761279 }
@@ -2739,3 +2742,794 @@ const struct bpf_map_ops htab_of_maps_map_ops = { · 1462 unmodified lines
27392742 BATCH_OPS(htab),
27402743 .map_btf_id = &htab_map_btf_ids[0],
27412744 };
2745+
2746+struct rhtab_elem {
2747+ struct rhash_head node;
2748+ /* key bytes, then value bytes follow */
2749+ u8 data[] __aligned(8);
2750+};
2751+
2752+struct bpf_rhtab {
2753+ struct bpf_map map;
2754+ struct rhashtable ht;
2755+ struct bpf_mem_alloc ma;
2756+ u32 elem_size;
2757+ bool freeing_internal;
2758+};
2759+
2760+static const struct rhashtable_params rhtab_params = {
2761+ .head_offset = offsetof(struct rhtab_elem, node),
2762+ .key_offset = offsetof(struct rhtab_elem, data),
2763+};
2764+
2765+static inline void *rhtab_elem_value(struct rhtab_elem *l, u32 key_size)
2766+{
2767+ return l->data + round_up(key_size, 8);
2768+}
2769+
2770+/* Specialize hash function and objcmp for long sized key */
2771+static __always_inline int rhtab_key_cmp_long(struct rhashtable_compare_arg *arg,
2772+ const void *ptr)
2773+{
2774+ const unsigned long key1 = *(const unsigned long *)arg->key;
2775+ const struct rhtab_elem *key2 = ptr;
2776+
2777+ return key1 != *(const unsigned long *)key2->data;
2778+}
2779+
2780+static __always_inline u32 rhtab_hashfn_long(const void *data, u32 len, u32 seed)
2781+{
2782+ u64 k = *(const unsigned long *)data;
2783+
2784+ return (u32)(k ^ (k >> 32)) ^ seed;
2785+}
2786+
2787+static const struct rhashtable_params rhtab_params_long = {
2788+ .head_offset = offsetof(struct rhtab_elem, node),
2789+ .key_offset = offsetof(struct rhtab_elem, data),
2790+ .key_len = sizeof(long),
2791+ .hashfn = rhtab_hashfn_long,
2792+ .obj_cmpfn = rhtab_key_cmp_long,
2793+};
2794+
2795+static struct bpf_map *rhtab_map_alloc(union bpf_attr *attr)
2796+{
2797+ struct rhashtable_params params;
2798+ struct bpf_rhtab *rhtab;
2799+ int err = 0;
2800+
2801+ rhtab = bpf_map_area_alloc(sizeof(*rhtab), NUMA_NO_NODE);
2802+ if (!rhtab)
2803+ return ERR_PTR(-ENOMEM);
2804+
2805+ bpf_map_init_from_attr(&rhtab->map, attr);
2806+
2807+ if (rhtab->map.max_entries > 1UL << 31) {
2808+ err = -E2BIG;
2809+ goto free_rhtab;
2810+ }
2811+
2812+ rhtab->elem_size = sizeof(struct rhtab_elem) + round_up(rhtab->map.key_size, 8) +
2813+ round_up(rhtab->map.value_size, 8);
2814+
2815+ params = rhtab_params;
2816+ params.key_len = rhtab->map.key_size;
2817+ params.nelem_hint = (u32)attr->map_extra;
2818+ params.automatic_shrinking = true;
2819+
2820+ if (rhtab->map.key_size == sizeof(long)) {
2821+ params.hashfn = rhtab_hashfn_long;
2822+ params.obj_cmpfn = rhtab_key_cmp_long;
2823+ }
2824+
2825+ err = rhashtable_init(&rhtab->ht, &params);
2826+ if (err)
2827+ goto free_rhtab;
2828+
2829+ /* Set max_elems after rhashtable_init() since init zeroes the struct */
2830+ rhtab->ht.max_elems = rhtab->map.max_entries;
2831+
2832+ err = bpf_mem_alloc_init(&rhtab->ma, rhtab->elem_size, false);
2833+ if (err)
2834+ goto destroy_rhtab;
2835+
2836+ return &rhtab->map;
2837+
2838+destroy_rhtab:
2839+ rhashtable_destroy(&rhtab->ht);
2840+free_rhtab:
2841+ bpf_map_area_free(rhtab);
2842+ return ERR_PTR(err);
2843+}
2844+
2845+static int rhtab_map_alloc_check(union bpf_attr *attr)
2846+{
2847+ if (!(attr->map_flags & BPF_F_NO_PREALLOC))
2848+ return -EINVAL;
2849+
2850+ if (attr->map_flags & BPF_F_ZERO_SEED)
2851+ return -EINVAL;
2852+
2853+ if (attr->key_size > U16_MAX)
2854+ return -E2BIG;
2855+
2856+ if (attr->map_extra >> 32)
2857+ return -EINVAL;
2858+
2859+ if ((u32)attr->map_extra > U16_MAX)
2860+ return -E2BIG;
2861+
2862+ if ((u32)attr->map_extra > attr->max_entries)
2863+ return -EINVAL;
2864+
2865+ return htab_map_alloc_check(attr);
2866+}
2867+
2868+static void rhtab_check_and_free_fields(struct bpf_rhtab *rhtab,
2869+ struct rhtab_elem *elem)
2870+{
2871+ if (IS_ERR_OR_NULL(rhtab->map.record))
2872+ return;
2873+
2874+ bpf_obj_free_fields(rhtab->map.record,
2875+ rhtab_elem_value(elem, rhtab->map.key_size));
2876+}
2877+
2878+static void rhtab_mem_dtor(void *obj, void *ctx)
2879+{
2880+ struct htab_btf_record *hrec = ctx;
2881+ struct rhtab_elem *elem = obj;
2882+
2883+ if (IS_ERR_OR_NULL(hrec->record))
2884+ return;
2885+
2886+ bpf_obj_free_fields(hrec->record,
2887+ rhtab_elem_value(elem, hrec->key_size));
2888+}
2889+
2890+static void rhtab_free_elem(void *ptr, void *arg)
2891+{
2892+ struct bpf_rhtab *rhtab = arg;
2893+ struct rhtab_elem *elem = ptr;
2894+
2895+ bpf_map_free_internal_structs(&rhtab->map, rhtab_elem_value(elem, rhtab->map.key_size));
2896+ bpf_mem_cache_free_rcu(&rhtab->ma, elem);
2897+}
2898+
2899+static void rhtab_map_free(struct bpf_map *map)
2900+{
2901+ struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
2902+
2903+ rhashtable_free_and_destroy(&rhtab->ht, rhtab_free_elem, rhtab);
2904+ bpf_mem_alloc_destroy(&rhtab->ma);
2905+ bpf_map_area_free(rhtab);
2906+}
2907+
2908+static void *rhtab_lookup_elem(struct bpf_map *map, void *key)
2909+{
2910+ struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
2911+
2912+ /* Hold RCU lock in case sleepable program calls via gen_lookup */
2913+ guard(rcu)();
2914+
2915+ if (map->key_size == sizeof(long))
2916+ return rhashtable_lookup_likely(&rhtab->ht, key, rhtab_params_long);
2917+
2918+ return rhashtable_lookup_likely(&rhtab->ht, key, rhtab_params);
2919+}
2920+
2921+static void *rhtab_map_lookup_elem(struct bpf_map *map, void *key) __must_hold(RCU)
2922+{
2923+ struct rhtab_elem *l;
2924+
2925+ l = rhtab_lookup_elem(map, key);
2926+ return l ? rhtab_elem_value(l, map->key_size) : NULL;
2927+}
2928+
2929+static void rhtab_read_elem_value(struct bpf_map *map, void *dst, struct rhtab_elem *elem,
2930+ u64 flags)
2931+{
2932+ void *src = rhtab_elem_value(elem, map->key_size);
2933+
2934+ if (flags & BPF_F_LOCK)
2935+ copy_map_value_locked(map, dst, src, true);
2936+ else
2937+ copy_map_value(map, dst, src);
2938+}
2939+
2940+static int rhtab_delete_elem(struct bpf_rhtab *rhtab, struct rhtab_elem *elem, void *copy,
2941+ u64 flags)
2942+{
2943+ int err;
2944+
2945+ /*
2946+ * disable_instrumentation() mitigates the deadlock for programs running in NMI context.
2947+ * rhashtable locks bucket with local_irq_save(). Only NMI programs may reenter
2948+ * rhashtable code, bpf_disable_instrumentation() disables programs running in NMI, except
2949+ * raw tracepoints, which we don't have in rhashtable.
2950+ */
2951+ bpf_disable_instrumentation();
2952+
2953+ if (rhtab->map.key_size == sizeof(long))
2954+ err = rhashtable_remove_fast(&rhtab->ht, &elem->node, rhtab_params_long);
2955+ else
2956+ err = rhashtable_remove_fast(&rhtab->ht, &elem->node, rhtab_params);
2957+
2958+ bpf_enable_instrumentation();
2959+
2960+ if (err)
2961+ return err;
2962+
2963+ if (copy) {
2964+ rhtab_read_elem_value(&rhtab->map, copy, elem, flags);
2965+ check_and_init_map_value(&rhtab->map, copy);
2966+ }
2967+ /* Release internal structs: kptr, bpf_timer, task_work, wq */
2968+ rhtab_check_and_free_fields(rhtab, elem);
2969+ bpf_mem_cache_free_rcu(&rhtab->ma, elem);
2970+ return 0;
2971+}
2972+
2973+
2974+static long rhtab_map_delete_elem(struct bpf_map *map, void *key)
2975+{
2976+ struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
2977+ struct rhtab_elem *elem;
2978+
2979+ guard(rcu)();
2980+
2981+ elem = rhtab_lookup_elem(map, key);
2982+ if (!elem)
2983+ return -ENOENT;
2984+
2985+ return rhtab_delete_elem(rhtab, elem, NULL, 0);
2986+}
2987+
2988+static int rhtab_map_lookup_and_delete_elem(struct bpf_map *map, void *key, void *value, u64 flags)
2989+{
2990+ struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
2991+ struct rhtab_elem *elem;
2992+ int err;
2993+
2994+ err = bpf_map_check_op_flags(map, flags, BPF_F_LOCK);
2995+ if (err)
2996+ return err;
2997+
2998+ guard(rcu)();
2999+
3000+ elem = rhtab_lookup_elem(map, key);
3001+ if (!elem)
3002+ return -ENOENT;
3003+
3004+ return rhtab_delete_elem(rhtab, elem, value, flags);
3005+}
3006+
3007+static long rhtab_map_update_existing(struct bpf_map *map, struct rhtab_elem *elem, void *value,
3008+ u64 map_flags)
3009+{
3010+ struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3011+ void *old_val = rhtab_elem_value(elem, map->key_size);
3012+
3013+ if (map_flags & BPF_NOEXIST)
3014+ return -EEXIST;
3015+
3016+ if (map_flags & BPF_F_LOCK)
3017+ copy_map_value_locked(map, old_val, value, false);
3018+ else
3019+ copy_map_value(map, old_val, value);
3020+
3021+ /*
3022+ * Torn reads: a concurrent reader without BPF_F_LOCK may observe
3023+ * the value mid-copy. Callers requiring consistent reads must use
3024+ * BPF_F_LOCK, matching arraymap semantics.
3025+ *
3026+ * copy_map_value() skips special-field offsets, so old timers/
3027+ * kptrs/etc. still sit in the slot. Cancel them after the copy
3028+ * to match arraymap's update semantics.
3029+ */
3030+ rhtab_check_and_free_fields(rhtab, elem);
3031+ return 0;
3032+}
3033+
3034+static long rhtab_map_update_elem(struct bpf_map *map, void *key, void *value, u64 map_flags)
3035+{
3036+ struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3037+ struct rhtab_elem *elem, *tmp;
3038+
3039+ if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST))
3040+ return -EINVAL;
3041+
3042+ if ((map_flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))
3043+ return -EINVAL;
3044+
3045+ guard(rcu)();
3046+ elem = rhtab_lookup_elem(map, key);
3047+ if (elem)
3048+ return rhtab_map_update_existing(map, elem, value, map_flags);
3049+
3050+ if (map_flags & BPF_EXIST)
3051+ return -ENOENT;
3052+
3053+ /*
3054+ * Reject new insertions while map_release_uref cleanup walks the
3055+ * table. Without this, new elements could keep triggering rehash
3056+ * and prevent the walk from terminating.
3057+ */
3058+ if (READ_ONCE(rhtab->freeing_internal))
3059+ return -EBUSY;
3060+
3061+ /* Check max_entries limit before inserting new element */
3062+ if (atomic_read(&rhtab->ht.nelems) >= map->max_entries)
3063+ return -E2BIG;
3064+
3065+ elem = bpf_mem_cache_alloc(&rhtab->ma);
3066+ if (!elem)
3067+ return -ENOMEM;
3068+
3069+ memcpy(elem->data, key, map->key_size);
3070+ copy_map_value(map, rhtab_elem_value(elem, map->key_size), value);
3071+ check_and_init_map_value(map, rhtab_elem_value(elem, map->key_size));
3072+
3073+ /* Prevent deadlock for NMI programs attempting to take bucket lock */
3074+ bpf_disable_instrumentation();
3075+
3076+ if (map->key_size == sizeof(long))
3077+ tmp = rhashtable_lookup_get_insert_fast(&rhtab->ht, &elem->node, rhtab_params_long);
3078+ else
3079+ tmp = rhashtable_lookup_get_insert_fast(&rhtab->ht, &elem->node, rhtab_params);
3080+
3081+ bpf_enable_instrumentation();
3082+
3083+ if (tmp) {
3084+ bpf_mem_cache_free(&rhtab->ma, elem);
3085+ if (IS_ERR(tmp))
3086+ return PTR_ERR(tmp);
3087+
3088+ return rhtab_map_update_existing(map, tmp, value, map_flags);
3089+ }
3090+
3091+ return 0;
3092+}
3093+
3094+static int rhtab_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
3095+{
3096+ struct bpf_insn *insn = insn_buf;
3097+ const int ret = BPF_REG_0;
3098+
3099+ BUILD_BUG_ON(!__same_type(&rhtab_lookup_elem,
3100+ (void *(*)(struct bpf_map *map, void *key)) NULL));
3101+ *insn++ = BPF_EMIT_CALL(rhtab_lookup_elem);
3102+ *insn++ = BPF_JMP_IMM(BPF_JEQ, ret, 0, 1);
3103+ *insn++ = BPF_ALU64_IMM(BPF_ADD, ret,
3104+ offsetof(struct rhtab_elem, data) + round_up(map->key_size, 8));
3105+
3106+ return insn - insn_buf;
3107+}
3108+
3109+static int rhtab_map_check_btf(struct bpf_map *map, const struct btf *btf,
3110+ const struct btf_type *key_type,
3111+ const struct btf_type *value_type)
3112+{
3113+ struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3114+
3115+ return bpf_ma_set_dtor(map, &rhtab->ma, rhtab_mem_dtor);
3116+}
3117+
3118+static void rhtab_map_free_internal_structs(struct bpf_map *map)
3119+{
3120+ struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3121+ struct rhashtable_iter iter;
3122+ struct rhtab_elem *elem;
3123+
3124+ if (!bpf_map_has_internal_structs(map))
3125+ return;
3126+
3127+ /*
3128+ * Block new insertions. Once observed, no new growth is triggered,
3129+ * so any in-flight rehash will drain and the walker is guaranteed
3130+ * to stop returning -EAGAIN. Treat -EAGAIN as "rehash in progress,
3131+ * retry"; do not wait for the worker.
3132+ */
3133+ WRITE_ONCE(rhtab->freeing_internal, true);
3134+
3135+ rhashtable_walk_enter(&rhtab->ht, &iter);
3136+ rhashtable_walk_start(&iter);
3137+
3138+ while ((elem = rhashtable_walk_next(&iter))) {
3139+ if (IS_ERR(elem)) {
3140+ if (PTR_ERR(elem) == -EAGAIN)
3141+ continue;
3142+ break;
3143+ }
3144+
3145+ bpf_map_free_internal_structs(map, rhtab_elem_value(elem, map->key_size));
3146+
3147+ if (need_resched()) { /* Avoid stalls on large maps */
3148+ rhashtable_walk_stop(&iter);
3149+ cond_resched();
3150+ rhashtable_walk_start(&iter);
3151+ }
3152+ }
3153+
3154+ rhashtable_walk_stop(&iter);
3155+ rhashtable_walk_exit(&iter);
3156+ WRITE_ONCE(rhtab->freeing_internal, false);
3157+}
3158+
3159+static int rhtab_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
3160+ __must_hold_shared(RCU)
3161+{
3162+ struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3163+ struct rhtab_elem *elem;
3164+
3165+ elem = rhashtable_next_key(&rhtab->ht, key);
3166+
3167+ /* if not found, return the first key */
3168+ if (PTR_ERR(elem) == -ENOENT)
3169+ elem = rhashtable_next_key(&rhtab->ht, NULL);
3170+
3171+ if (IS_ERR(elem))
3172+ return PTR_ERR(elem);
3173+ if (!elem)
3174+ return -ENOENT;
3175+
3176+ memcpy(next_key, elem->data, map->key_size);
3177+ return 0;
3178+}
3179+
3180+static void rhtab_map_seq_show_elem(struct bpf_map *map, void *key, struct seq_file *m)
3181+{
3182+ void *value;
3183+
3184+ /* Guarantee that hashtab value is not freed */
3185+ guard(rcu)();
3186+
3187+ value = rhtab_map_lookup_elem(map, key);
3188+ if (!value)
3189+ return;
3190+
3191+ btf_type_seq_show(map->btf, map->btf_key_type_id, key, m);
3192+ seq_puts(m, ": ");
3193+ btf_type_seq_show(map->btf, map->btf_value_type_id, value, m);
3194+ seq_putc(m, '\n');
3195+}
3196+
3197+static long bpf_each_rhash_elem(struct bpf_map *map, bpf_callback_t callback_fn,
3198+ void *callback_ctx, u64 flags)
3199+{
3200+ struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3201+ void *prev_key = NULL;
3202+ struct rhtab_elem *elem;
3203+ int num_elems = 0;
3204+ u64 ret = 0;
3205+
3206+ cant_migrate();
3207+
3208+ if (flags != 0)
3209+ return -EINVAL;
3210+
3211+ rcu_read_lock();
3212+ /*
3213+ * Best-effort iteration: if rhashtable is concurrently resized or
3214+ * elements are deleted/inserted, there may be missed or duplicate
3215+ * elements visited.
3216+ */
3217+ while ((elem = rhashtable_next_key(&rhtab->ht, prev_key))) {
3218+ if (IS_ERR(elem))
3219+ break;
3220+ num_elems++;
3221+ ret = callback_fn((u64)(long)map,
3222+ (u64)(long)elem->data,
3223+ (u64)(long)rhtab_elem_value(elem, map->key_size),
3224+ (u64)(long)callback_ctx, 0);
3225+ if (ret)
3226+ break;
3227+
3228+ prev_key = elem->data; /* valid while RCU held */
3229+ }
3230+ rcu_read_unlock();
3231+
3232+ return num_elems;
3233+}
3234+
3235+static u64 rhtab_map_mem_usage(const struct bpf_map *map)
3236+{
3237+ struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3238+ u64 num_entries;
3239+
3240+ /* Excludes rhashtable bucket overhead (~ nelems * sizeof(void *) at 75% load). */
3241+ num_entries = atomic_read(&rhtab->ht.nelems);
3242+ return sizeof(struct bpf_rhtab) + rhtab->elem_size * num_entries;
3243+}
3244+
3245+static int __rhtab_map_lookup_and_delete_batch(struct bpf_map *map,
3246+ const union bpf_attr *attr,
3247+ union bpf_attr __user *uattr,
3248+ bool do_delete)
3249+{
3250+ struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3251+ void __user *uvalues = u64_to_user_ptr(attr->batch.values);
3252+ void __user *ukeys = u64_to_user_ptr(attr->batch.keys);
3253+ void __user *ubatch = u64_to_user_ptr(attr->batch.in_batch);
3254+ void *cursor = NULL, *keys = NULL, *values = NULL, *dst_key, *dst_val;
3255+ struct rhtab_elem **del_elems = NULL;
3256+ u32 max_count, total, key_size, value_size, i;
3257+ bool has_next_cursor = false;
3258+ struct rhtab_elem *elem;
3259+ u64 elem_map_flags, map_flags;
3260+ int ret = 0;
3261+
3262+ elem_map_flags = attr->batch.elem_flags;
3263+ ret = bpf_map_check_op_flags(map, elem_map_flags, BPF_F_LOCK);
3264+ if (ret)
3265+ return ret;
3266+
3267+ map_flags = attr->batch.flags;
3268+ if (map_flags)
3269+ return -EINVAL;
3270+
3271+ max_count = attr->batch.count;
3272+ if (!max_count)
3273+ return 0;
3274+
3275+ if (put_user(0, &uattr->batch.count))
3276+ return -EFAULT;
3277+
3278+ key_size = map->key_size;
3279+ value_size = map->value_size;
3280+
3281+ keys = kvmalloc_array(max_count, key_size, GFP_USER | __GFP_NOWARN);
3282+ values = kvmalloc_array(max_count, value_size, GFP_USER | __GFP_NOWARN);
3283+ if (do_delete)
3284+ del_elems = kvmalloc_array(max_count, sizeof(void *),
3285+ GFP_USER | __GFP_NOWARN);
3286+ cursor = kmalloc(key_size, GFP_USER | __GFP_NOWARN);
3287+
3288+ if (!keys || !values || !cursor || (do_delete && !del_elems)) {
3289+ ret = -ENOMEM;
3290+ goto free;
3291+ }
3292+
3293+ if (ubatch && copy_from_user(cursor, ubatch, key_size)) {
3294+ ret = -EFAULT;
3295+ goto free;
3296+ }
3297+
3298+ dst_key = keys;
3299+ dst_val = values;
3300+ total = 0;
3301+
3302+ rcu_read_lock();
3303+
3304+ /*
3305+ * Cursor stores the key of the next-to-process element (stashed by
3306+ * the previous batch). Look it up directly so the element is included
3307+ * here rather than skipped by next_key(). If the cursor was deleted
3308+ * concurrently (or by the previous do_delete batch), return -EAGAIN
3309+ * so userspace can distinguish a lost cursor from end-of-iteration
3310+ * (-ENOENT) and restart from a NULL cursor.
3311+ */
3312+ if (ubatch) {
3313+ elem = rhtab_lookup_elem(map, cursor);
3314+ if (!elem) {
3315+ rcu_read_unlock();
3316+ ret = -EAGAIN;
3317+ goto free;
3318+ }
3319+ } else {
3320+ elem = rhashtable_next_key(&rhtab->ht, NULL);
3321+ }
3322+
3323+ while (elem && !IS_ERR(elem) && total < max_count) {
3324+ memcpy(dst_key, elem->data, key_size);
3325+ rhtab_read_elem_value(map, dst_val, elem, elem_map_flags);
3326+ check_and_init_map_value(map, dst_val);
3327+
3328+ if (do_delete)
3329+ del_elems[total] = elem;
3330+
3331+ elem = rhashtable_next_key(&rhtab->ht, dst_key);
3332+ dst_key += key_size;
3333+ dst_val += value_size;
3334+ total++;
3335+
3336+ /* Bail to userspace to avoid stalls. */
3337+ if (need_resched())
3338+ break;
3339+ }
3340+
3341+ if (elem && !IS_ERR(elem)) {
3342+ /* Stash next-to-process key as cursor for the next batch. */
3343+ memcpy(cursor, elem->data, key_size);
3344+ has_next_cursor = true;
3345+ }
3346+
3347+ if (do_delete) {
3348+ for (i = 0; i < total; i++)
3349+ rhtab_delete_elem(rhtab, del_elems[i], NULL, 0);
3350+ }
3351+
3352+ rcu_read_unlock();
3353+
3354+ if (total == 0) {
3355+ ret = -ENOENT;
3356+ goto free;
3357+ }
3358+
3359+ /* No more elements after this batch. */
3360+ if (!has_next_cursor)
3361+ ret = -ENOENT;
3362+
3363+ if (copy_to_user(ukeys, keys, (size_t)total * key_size) ||
3364+ copy_to_user(uvalues, values, (size_t)total * value_size) ||
3365+ put_user(total, &uattr->batch.count) ||
3366+ (has_next_cursor &&
3367+ copy_to_user(u64_to_user_ptr(attr->batch.out_batch),
3368+ cursor, key_size))) {
3369+ ret = -EFAULT;
3370+ goto free;
3371+ }
3372+
3373+free:
3374+ kfree(cursor);
3375+ kvfree(keys);
3376+ kvfree(values);
3377+ kvfree(del_elems);
3378+ return ret;
3379+}
3380+
3381+static int rhtab_map_lookup_batch(struct bpf_map *map, const union bpf_attr *attr,
3382+ union bpf_attr __user *uattr)
3383+{
3384+ return __rhtab_map_lookup_and_delete_batch(map, attr, uattr, false);
3385+}
3386+
3387+static int rhtab_map_lookup_and_delete_batch(struct bpf_map *map, const union bpf_attr *attr,
3388+ union bpf_attr __user *uattr)
3389+{
3390+ return __rhtab_map_lookup_and_delete_batch(map, attr, uattr, true);
3391+}
3392+
3393+struct bpf_iter_seq_rhash_map_info {
3394+ struct bpf_map *map;
3395+ struct bpf_rhtab *rhtab;
3396+ struct rhashtable_iter iter;
3397+};
3398+
3399+static void *bpf_rhash_map_seq_start(struct seq_file *seq, loff_t *pos)
3400+ __acquires(RCU)
3401+{
3402+ struct bpf_iter_seq_rhash_map_info *info = seq->private;
3403+ struct rhtab_elem *elem;
3404+
3405+ rhashtable_walk_start(&info->iter);
3406+ /*
3407+ * Re-deliver the element returned by walk_next() at the end of the
3408+ * previous read() — bpf_seq_read may have stopped before show()
3409+ * consumed it. Rehash rewinds the walker; retry on -EAGAIN.
3410+ */
3411+ do {
3412+ elem = rhashtable_walk_peek(&info->iter);
3413+ } while (PTR_ERR(elem) == -EAGAIN);
3414+
3415+ if (IS_ERR(elem))
3416+ return NULL;
3417+
3418+ if (elem && *pos == 0)
3419+ ++*pos;
3420+ return elem;
3421+}
3422+
3423+static void *bpf_rhash_map_seq_next(struct seq_file *seq, void *v, loff_t *pos)
3424+{
3425+ struct bpf_iter_seq_rhash_map_info *info = seq->private;
3426+ struct rhtab_elem *elem;
3427+
3428+ ++*pos;
3429+
3430+ /* Rehash rewinds the walker; retry until it stops returning -EAGAIN. */
3431+ do {
3432+ elem = rhashtable_walk_next(&info->iter);
3433+ } while (PTR_ERR(elem) == -EAGAIN);
3434+
3435+ if (IS_ERR(elem))
3436+ return NULL;
3437+ return elem;
3438+}
3439+
3440+static int __bpf_rhash_map_seq_show(struct seq_file *seq,
3441+ struct rhtab_elem *elem)
3442+{
3443+ struct bpf_iter_seq_rhash_map_info *info = seq->private;
3444+ struct bpf_iter__bpf_map_elem ctx = {};
3445+ struct bpf_iter_meta meta;
3446+ struct bpf_prog *prog;
3447+ int ret = 0;
3448+
3449+ meta.seq = seq;
3450+ prog = bpf_iter_get_info(&meta, elem == NULL);
3451+ if (prog) {
3452+ ctx.meta = &meta;
3453+ ctx.map = info->map;
3454+ if (elem) {
3455+ ctx.key = elem->data;
3456+ ctx.value = rhtab_elem_value(elem, info->map->key_size);
3457+ }
3458+ ret = bpf_iter_run_prog(prog, &ctx);
3459+ }
3460+
3461+ return ret;
3462+}
3463+
3464+static int bpf_rhash_map_seq_show(struct seq_file *seq, void *v)
3465+{
3466+ return __bpf_rhash_map_seq_show(seq, v);
3467+}
3468+
3469+static void bpf_rhash_map_seq_stop(struct seq_file *seq, void *v)
3470+ __releases(RCU)
3471+{
3472+ struct bpf_iter_seq_rhash_map_info *info = seq->private;
3473+
3474+ if (!v)
3475+ (void)__bpf_rhash_map_seq_show(seq, NULL);
3476+
3477+ rhashtable_walk_stop(&info->iter);
3478+}
3479+
3480+static int bpf_iter_init_rhash_map(void *priv_data, struct bpf_iter_aux_info *aux)
3481+{
3482+ struct bpf_iter_seq_rhash_map_info *info = priv_data;
3483+ struct bpf_map *map = aux->map;
3484+
3485+ bpf_map_inc_with_uref(map);
3486+ info->map = map;
3487+ info->rhtab = container_of(map, struct bpf_rhtab, map);
3488+ rhashtable_walk_enter(&info->rhtab->ht, &info->iter);
3489+ return 0;
3490+}
3491+
3492+static void bpf_iter_fini_rhash_map(void *priv_data)
3493+{
3494+ struct bpf_iter_seq_rhash_map_info *info = priv_data;
3495+
3496+ rhashtable_walk_exit(&info->iter);
3497+ bpf_map_put_with_uref(info->map);
3498+}
3499+
3500+static const struct seq_operations bpf_rhash_map_seq_ops = {
3501+ .start = bpf_rhash_map_seq_start,
3502+ .next = bpf_rhash_map_seq_next,
3503+ .stop = bpf_rhash_map_seq_stop,
3504+ .show = bpf_rhash_map_seq_show,
3505+};
3506+
3507+static const struct bpf_iter_seq_info rhash_iter_seq_info = {
3508+ .seq_ops = &bpf_rhash_map_seq_ops,
3509+ .init_seq_private = bpf_iter_init_rhash_map,
3510+ .fini_seq_private = bpf_iter_fini_rhash_map,
3511+ .seq_priv_size = sizeof(struct bpf_iter_seq_rhash_map_info),
3512+};
3513+
3514+BTF_ID_LIST_SINGLE(rhtab_map_btf_ids, struct, bpf_rhtab)
3515+const struct bpf_map_ops rhtab_map_ops = {
3516+ .map_meta_equal = bpf_map_meta_equal,
3517+ .map_alloc_check = rhtab_map_alloc_check,
3518+ .map_alloc = rhtab_map_alloc,
3519+ .map_free = rhtab_map_free,
3520+ .map_get_next_key = rhtab_map_get_next_key,
3521+ .map_release_uref = rhtab_map_free_internal_structs,
3522+ .map_check_btf = rhtab_map_check_btf,
3523+ .map_lookup_elem = rhtab_map_lookup_elem,
3524+ .map_lookup_and_delete_elem = rhtab_map_lookup_and_delete_elem,
3525+ .map_update_elem = rhtab_map_update_elem,
3526+ .map_delete_elem = rhtab_map_delete_elem,
3527+ .map_gen_lookup = rhtab_map_gen_lookup,
3528+ .map_seq_show_elem = rhtab_map_seq_show_elem,
3529+ .map_set_for_each_callback_args = map_set_for_each_callback_args,
3530+ .map_for_each_callback = bpf_each_rhash_elem,
3531+ .map_mem_usage = rhtab_map_mem_usage,
3532+ BATCH_OPS(rhtab),
3533+ .map_btf_id = &rhtab_map_btf_ids[0],
3534+ .iter_seq_info = &rhash_iter_seq_info,
3535+};
kernel/bpf/helpers.c+154−50
@@ -1944,7 +1944,7 @@ static const struct bpf_func_proto bpf_dynptr_read_proto = { · 1943 unmodified lines
19441944 .ret_type = RET_INTEGER,
19451945 .arg1_type = ARG_PTR_TO_UNINIT_MEM,
19461946 .arg2_type = ARG_CONST_SIZE_OR_ZERO,
1947− .arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
1947+ .arg3_type = ARG_PTR_TO_DYNPTR,
19481948 .arg4_type = ARG_ANYTHING,
19491949 .arg5_type = ARG_ANYTHING,
19501950 };
@@ -2001,7 +2001,7 @@ static const struct bpf_func_proto bpf_dynptr_write_proto = { · 50 unmodified lines
20012001 .func = bpf_dynptr_write,
20022002 .gpl_only = false,
20032003 .ret_type = RET_INTEGER,
2004− .arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
2004+ .arg1_type = ARG_PTR_TO_DYNPTR,
20052005 .arg2_type = ARG_ANYTHING,
20062006 .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
20072007 .arg4_type = ARG_CONST_SIZE_OR_ZERO,
@@ -2044,7 +2044,7 @@ static const struct bpf_func_proto bpf_dynptr_data_proto = { · 36 unmodified lines
20442044 .func = bpf_dynptr_data,
20452045 .gpl_only = false,
20462046 .ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,
2047− .arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
2047+ .arg1_type = ARG_PTR_TO_DYNPTR,
20482048 .arg2_type = ARG_ANYTHING,
20492049 .arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
20502050 };
@@ -2247,10 +2247,11 @@ EXPORT_SYMBOL_GPL(bpf_base_func_proto); · 196 unmodified lines
22472247 void bpf_list_head_free(const struct btf_field *field, void *list_head,
22482248 struct bpf_spin_lock *spin_lock)
22492249 {
2250− struct list_head *head = list_head, *orig_head = list_head;
2250+ struct list_head *head = list_head, drain, *pos, *n;
22512251
22522252 BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head));
22532253 BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head));
2254+ INIT_LIST_HEAD(&drain);
22542255
22552256 /* Do the actual list draining outside the lock to not hold the lock for
22562257 * too long, and also prevent deadlocks if tracing programs end up
@@ -2261,20 +2262,30 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head, · 4 unmodified lines
22612262 __bpf_spin_lock_irqsave(spin_lock);
22622263 if (!head->next || list_empty(head))
22632264 goto unlock;
2264− head = head->next;
2265+ list_for_each_safe(pos, n, head) {
2266+ struct bpf_list_node_kern *node;
2267+
2268+ node = container_of(pos, struct bpf_list_node_kern, list_head);
2269+ WRITE_ONCE(node->owner, BPF_PTR_POISON);
2270+ list_move_tail(pos, &drain);
2271+ }
22652272 unlock:
2266− INIT_LIST_HEAD(orig_head);
2273+ INIT_LIST_HEAD(head);
22672274 __bpf_spin_unlock_irqrestore(spin_lock);
22682275
2269− while (head != orig_head) {
2270− void *obj = head;
2276+ while (!list_empty(&drain)) {
2277+ struct bpf_list_node_kern *node;
22712278
2272− obj -= field->graph_root.node_offset;
2273− head = head->next;
2279+ pos = drain.next;
2280+ node = container_of(pos, struct bpf_list_node_kern, list_head);
2281+ list_del_init(pos);
2282+ /* Ensure __bpf_list_add() sees the node as unlinked. */
2283+ smp_store_release(&node->owner, NULL);
22742284 /* The contained type can also have resources, including a
22752285 * bpf_list_head which needs to be freed.
22762286 */
2277− __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
2287+ __bpf_obj_drop_impl((char *)pos - field->graph_root.node_offset,
2288+ field->graph_root.value_rec, false);
22782289 }
22792290 }
22802291
@@ -2295,6 +2306,7 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, · 14 unmodified lines
22952306 struct bpf_spin_lock *spin_lock)
22962307 {
22972308 struct rb_root_cached orig_root, *root = rb_root;
2309+ struct bpf_rb_node_kern *node;
22982310 struct rb_node *pos, *n;
22992311 void *obj;
23002312
@@ -2303,14 +2315,20 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, · 2 unmodified lines
23032315
23042316 __bpf_spin_lock_irqsave(spin_lock);
23052317 orig_root = *root;
2318+ bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
2319+ node = rb_entry(pos, struct bpf_rb_node_kern, rb_node);
2320+ WRITE_ONCE(node->owner, BPF_PTR_POISON);
2321+ }
23062322 *root = RB_ROOT_CACHED;
23072323 __bpf_spin_unlock_irqrestore(spin_lock);
23082324
23092325 bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
23102326 obj = pos;
23112327 obj -= field->graph_root.node_offset;
2312−
2313−
2328+ node = rb_entry(pos, struct bpf_rb_node_kern, rb_node);
2329+ RB_CLEAR_NODE(pos);
2330+ /* Ensure __bpf_rbtree_add() sees the node as unlinked. */
2331+ smp_store_release(&node->owner, NULL);
23142332 __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
23152333 }
23162334 }
@@ -2467,9 +2485,11 @@ __bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta · 150 unmodified lines
24672485
24682486 static int __bpf_list_add(struct bpf_list_node_kern *node,
24692487 struct bpf_list_head *head,
2470− bool tail, struct btf_record *rec, u64 off)
2488+ struct list_head **prev_ptr,
2489+ struct btf_record *rec, u64 off)
24712490 {
24722491 struct list_head *n = &node->list_head, *h = (void *)head;
2492+ struct list_head *prev;
24732493
24742494 /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
24752495 * called on its fields, so init here
@@ -2477,19 +2497,31 @@ static int __bpf_list_add(struct bpf_list_node_kern *node, · 1 unmodified line
24772497 if (unlikely(!h->next))
24782498 INIT_LIST_HEAD(h);
24792499
2500+ prev = *prev_ptr;
2501+
2502+ /* When prev is not the list head, it must be a node in this list. */
2503+ if (prev != h) {
2504+ struct bpf_list_node_kern *prev_kn =
2505+ container_of(prev, struct bpf_list_node_kern, list_head);
2506+
2507+ if (unlikely(READ_ONCE(prev_kn->owner) != head))
2508+ goto fail;
2509+ }
2510+
24802511 /* node->owner != NULL implies !list_empty(n), no need to separately
24812512 * check the latter
24822513 */
2483− if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
2484− /* Only called from BPF prog, no need to migrate_disable */
2485− __bpf_obj_drop_impl((void *)n - off, rec, false);
2486− return -EINVAL;
2487− }
2514+ if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON))
2515+ goto fail;
24882516
2489− tail ? list_add_tail(n, h) : list_add(n, h);
2517+ list_add(n, prev);
24902518 WRITE_ONCE(node->owner, head);
2491−
24922519 return 0;
2520+
2521+fail:
2522+ /* Only called from BPF prog, no need to migrate_disable */
2523+ __bpf_obj_drop_impl((void *)n - off, rec, false);
2524+ return -EINVAL;
24932525 }
24942526
24952527 /**
@@ -2510,8 +2542,9 @@ __bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head, · 14 unmodified lines
25102542 u64 off)
25112543 {
25122544 struct bpf_list_node_kern *n = (void *)node;
2545+ struct list_head *h = (void *)head;
25132546
2514− return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off);
2547+ return __bpf_list_add(n, head, &h, meta ? meta->record : NULL, off);
25152548 }
25162549
25172550 __bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,
@@ -2539,8 +2572,9 @@ __bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head, · 21 unmodified lines
25392572 u64 off)
25402573 {
25412574 struct bpf_list_node_kern *n = (void *)node;
2575+ struct list_head *h = (void *)head;
25422576
2543− return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off);
2577+ return __bpf_list_add(n, head, &h->prev, meta ? meta->record : NULL, off);
25442578 }
25452579
25462580 __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
@@ -2550,37 +2584,63 @@ __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head, · 3 unmodified lines
25502584 return bpf_list_push_back(head, node, meta__ign, off);
25512585 }
25522586
2553−static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail)
2587+__bpf_kfunc int bpf_list_add(struct bpf_list_head *head, struct bpf_list_node *new,
2588+ struct bpf_list_node *prev__nonown_allowed,
2589+ struct btf_struct_meta *meta, u64 off)
2590+{
2591+ struct bpf_list_node_kern *n = (void *)new, *p = (void *)prev__nonown_allowed;
2592+ struct list_head *prev_ptr = &p->list_head;
2593+
2594+ return __bpf_list_add(n, head, &prev_ptr, meta ? meta->record : NULL, off);
2595+}
2596+
2597+static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head,
2598+ struct list_head *n)
25542599 {
2555− struct list_head *n, *h = (void *)head;
2600+ struct list_head *h = (void *)head;
25562601 struct bpf_list_node_kern *node;
25572602
25582603 /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
25592604 * called on its fields, so init here
25602605 */
2561− if (unlikely(!h->next))
2606+ if (unlikely(!h->next)) {
25622607 INIT_LIST_HEAD(h);
2608+ return NULL;
2609+ }
25632610 if (list_empty(h))
25642611 return NULL;
25652612
2566− n = tail ? h->prev : h->next;
25672613 node = container_of(n, struct bpf_list_node_kern, list_head);
2568− if (WARN_ON_ONCE(READ_ONCE(node->owner) != head))
2614+ if (unlikely(READ_ONCE(node->owner) != head))
25692615 return NULL;
25702616
25712617 list_del_init(n);
2572− WRITE_ONCE(node->owner, NULL);
2618+ /* Ensure __bpf_list_add() sees the node as unlinked. */
2619+ smp_store_release(&node->owner, NULL);
25732620 return (struct bpf_list_node *)n;
25742621 }
25752622
25762623 __bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
25772624 {
2578− return __bpf_list_del(head, false);
2625+ struct list_head *h = (void *)head;
2626+
2627+ return __bpf_list_del(head, h->next);
25792628 }
25802629
25812630 __bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
25822631 {
2583− return __bpf_list_del(head, true);
2632+ struct list_head *h = (void *)head;
2633+
2634+ return __bpf_list_del(head, h->prev);
2635+}
2636+
2637+__bpf_kfunc struct bpf_list_node *bpf_list_del(struct bpf_list_head *head,
2638+ struct bpf_list_node *node__nonown_allowed)
2639+{
2640+ struct bpf_list_node_kern *kn = (void *)node__nonown_allowed;
2641+
2642+ /* verifier guarantees node is a list node rather than list head */
2643+ return __bpf_list_del(head, &kn->list_head);
25842644 }
25852645
25862646 __bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)
@@ -2603,6 +2663,43 @@ __bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head) · 16 unmodified lines
26032663 return (struct bpf_list_node *)h->prev;
26042664 }
26052665
2666+__bpf_kfunc bool bpf_list_is_first(struct bpf_list_head *head,
2667+ struct bpf_list_node *node__nonown_allowed)
2668+{
2669+ struct list_head *h = (struct list_head *)head;
2670+ struct bpf_list_node_kern *kn = (struct bpf_list_node_kern *)node__nonown_allowed;
2671+
2672+ if (READ_ONCE(kn->owner) != head)
2673+ return false;
2674+
2675+ return list_is_first(&kn->list_head, h);
2676+}
2677+
2678+__bpf_kfunc bool bpf_list_is_last(struct bpf_list_head *head,
2679+ struct bpf_list_node *node__nonown_allowed)
2680+{
2681+ struct list_head *h = (struct list_head *)head;
2682+ struct bpf_list_node_kern *kn = (struct bpf_list_node_kern *)node__nonown_allowed;
2683+
2684+ if (READ_ONCE(kn->owner) != head)
2685+ return false;
2686+
2687+ return list_is_last(&kn->list_head, h);
2688+}
2689+
2690+__bpf_kfunc bool bpf_list_empty(struct bpf_list_head *head)
2691+{
2692+ struct list_head *h = (struct list_head *)head;
2693+
2694+ /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
2695+ * called on its fields, so init here
2696+ */
2697+ if (unlikely(!h->next))
2698+ INIT_LIST_HEAD(h);
2699+
2700+ return list_empty(h);
2701+}
2702+
26062703 __bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
26072704 struct bpf_rb_node *node)
26082705 {
@@ -2912,11 +3009,13 @@ __bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid) · 303 unmodified lines
29123009 {
29133010 struct task_struct *p;
29143011
2915− rcu_read_lock();
3012+ guard(rcu)();
3013+ if (!task_active_pid_ns(current))
3014+ return NULL;
3015+
29163016 p = find_task_by_vpid(vpid);
29173017 if (p)
29183018 p = bpf_task_acquire(p);
2919− rcu_read_unlock();
29203019
29213020 return p;
29223021 }
@@ -3072,7 +3171,7 @@ __bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset, · 149 unmodified lines
30723171 return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk);
30733172 }
30743173
3075−__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end)
3174+__bpf_kfunc int bpf_dynptr_adjust(struct bpf_dynptr *p, u64 start, u64 end)
30763175 {
30773176 struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
30783177 u64 size;
@@ -3093,14 +3192,14 @@ __bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end · 14 unmodified lines
30933192
30943193 __bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)
30953194 {
3096− struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
3195+ const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
30973196
30983197 return !ptr->data;
30993198 }
31003199
31013200 __bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
31023201 {
3103− struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
3202+ const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
31043203
31053204 if (!ptr->data)
31063205 return false;
@@ -3110,7 +3209,7 @@ __bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p) · 3 unmodified lines
31103209
31113210 __bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p)
31123211 {
3113− struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
3212+ const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
31143213
31153214 if (!ptr->data)
31163215 return -EINVAL;
@@ -3122,7 +3221,7 @@ __bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p, · 5 unmodified lines
31223221 struct bpf_dynptr *clone__uninit)
31233222 {
31243223 struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;
3125− struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
3224+ const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
31263225
31273226 if (!ptr->data) {
31283227 bpf_dynptr_set_null(clone);
@@ -3145,11 +3244,11 @@ __bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p, · 16 unmodified lines
31453244 * Copies data from source dynptr to destination dynptr.
31463245 * Returns 0 on success; negative error, otherwise.
31473246 */
3148−__bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off,
3149− struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
3247+__bpf_kfunc int bpf_dynptr_copy(const struct bpf_dynptr *dst_ptr, u64 dst_off,
3248+ const struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
31503249 {
3151− struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
3152− struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
3250+ const struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
3251+ const struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
31533252 void *src_slice, *dst_slice;
31543253 char buf[256];
31553254 u64 off;
@@ -3200,9 +3299,9 @@ __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off, · 44 unmodified lines
32003299 * at @offset with the constant byte @val.
32013300 * Returns 0 on success; negative error, otherwise.
32023301 */
3203−__bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
3302+__bpf_kfunc int bpf_dynptr_memset(const struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
32043303 {
3205− struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
3304+ const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
32063305 u64 chunk_sz, write_off;
32073306 char buf[256];
32083307 void* slice;
@@ -3301,7 +3400,7 @@ __bpf_kfunc void bpf_throw(u64 cookie) · 92 unmodified lines
33013400 * which skips compiler generated instrumentation to do the same.
33023401 */
33033402 kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
3304− ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
3403+ ctx.aux->bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_sp_adjust, ctx.bp, 0, 0);
33053404 WARN(1, "A call to BPF exception callback should never return\n");
33063405 }
33073406
@@ -4214,13 +4313,13 @@ __bpf_kfunc void bpf_key_put(struct bpf_key *bkey) · 906 unmodified lines
42144313 *
42154314 * Return: 0 on success, a negative value on error.
42164315 */
4217−__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
4218− struct bpf_dynptr *sig_p,
4316+__bpf_kfunc int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p,
4317+ const struct bpf_dynptr *sig_p,
42194318 struct bpf_key *trusted_keyring)
42204319 {
42214320 #ifdef CONFIG_SYSTEM_DATA_VERIFICATION
4222− struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
4223− struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
4321+ const struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
4322+ const struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
42244323 const void *data, *sig;
42254324 u32 data_len, sig_len;
42264325 int ret;
@@ -4718,10 +4817,15 @@ BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS) · 491 unmodified lines
47184817 BTF_ID_FLAGS(func, bpf_list_push_front_impl)
47194818 BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)
47204819 BTF_ID_FLAGS(func, bpf_list_push_back_impl)
4820+BTF_ID_FLAGS(func, bpf_list_add, KF_IMPLICIT_ARGS)
47214821 BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
47224822 BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
4823+BTF_ID_FLAGS(func, bpf_list_del, KF_ACQUIRE | KF_RET_NULL)
47234824 BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)
47244825 BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)
4826+BTF_ID_FLAGS(func, bpf_list_is_first)
4827+BTF_ID_FLAGS(func, bpf_list_is_last)
4828+BTF_ID_FLAGS(func, bpf_list_empty)
47254829 BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
47264830 BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)
47274831 BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)
@@ -4862,7 +4966,7 @@ BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS) · 134 unmodified lines
48624966 BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)
48634967 BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
48644968 BTF_ID_FLAGS(func, bpf_dynptr_from_file)
4865−BTF_ID_FLAGS(func, bpf_dynptr_file_discard)
4969+BTF_ID_FLAGS(func, bpf_dynptr_file_discard, KF_RELEASE)
48664970 BTF_ID_FLAGS(func, bpf_timer_cancel_async)
48674971 BTF_KFUNCS_END(common_btf_ids)
48684972
kernel/bpf/inode.c+4−0
@@ -845,6 +845,10 @@ static void bpf_destroy_inode(struct inode *inode) · 844 unmodified lines
845845 simple_xattrs_free(&opts->xa_cache, &bi->xattrs, NULL);
846846 }
847847
848+/*
849+ * Called after RCU grace period - safe to free inode and anything
850+ * that might be accessed by RCU pathwalk (inode fields, i_link).
851+ */
848852 static void bpf_free_inode(struct inode *inode)
849853 {
850854 if (S_ISLNK(inode->i_mode))
kernel/bpf/liveness.c+139−44
@@ -610,6 +610,21 @@ enum arg_track_state { · 609 unmodified lines
610610 /* Track callee stack slots fp-8 through fp-512 (64 slots of 8 bytes each) */
611611 #define MAX_ARG_SPILL_SLOTS 64
612612
613+/*
614+ * Combined register + stack arg tracking: R0-R10 at indices 0-10,
615+ * outgoing stack arg slots at indices MAX_BPF_REG..MAX_BPF_REG+6.
616+ */
617+#define MAX_AT_TRACK_REGS (MAX_BPF_REG + MAX_STACK_ARG_SLOTS)
618+
619+static int stack_arg_off_to_slot(s16 off)
620+{
621+ int aoff = off < 0 ? -off : off;
622+
623+ if (aoff / 8 > MAX_STACK_ARG_SLOTS)
624+ return -1;
625+ return aoff / 8 - 1;
626+}
627+
613628 static bool arg_is_visited(const struct arg_track *at)
614629 {
615630 return at->frame != ARG_UNVISITED;
@@ -791,7 +806,9 @@ static bool arg_track_join(struct bpf_verifier_env *env, int idx, int target, in · 175 unmodified lines
791806 return true;
792807
793808 verbose(env, "arg JOIN insn %d -> %d ", idx, target);
794− if (r >= 0)
809+ if (r >= MAX_BPF_REG)
810+ verbose(env, "sa%d: ", r - MAX_BPF_REG);
811+ else if (r >= 0)
795812 verbose(env, "r%d: ", r);
796813 else
797814 verbose(env, "fp%+d: ", r * 8);
@@ -1032,6 +1049,21 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i · 234 unmodified lines
10321049 verbose(env, "\tr%d: ", i); verbose_arg_track(env, &at_in[i]);
10331050 verbose(env, " -> "); verbose_arg_track(env, &at_out[i]);
10341051 }
1052+ /* Log outgoing stack arg slot transitions at indices MAX_BPF_REG..MAX_AT_TRACK_REGS-1 */
1053+ for (i = 0; i < MAX_STACK_ARG_SLOTS; i++) {
1054+ int ai = MAX_BPF_REG + i;
1055+
1056+ if (arg_track_eq(&at_out[ai], &at_in[ai]))
1057+ continue;
1058+ if (!printed) {
1059+ verbose(env, "%3d: ", idx);
1060+ bpf_verbose_insn(env, insn);
1061+ bpf_vlog_reset(&env->log, env->log.end_pos - 1);
1062+ printed = true;
1063+ }
1064+ verbose(env, "\tsa%d: ", i); verbose_arg_track(env, &at_in[ai]);
1065+ verbose(env, " -> "); verbose_arg_track(env, &at_out[ai]);
1066+ }
10351067 for (i = 0; i < MAX_ARG_SPILL_SLOTS; i++) {
10361068 if (arg_track_eq(&at_stack_out[i], &at_stack_in[i]))
10371069 continue;
@@ -1062,6 +1094,7 @@ static bool can_be_local_fp(int depth, int regno, struct arg_track *at) · 24 unmodified lines
10621094 static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
10631095 int insn_idx,
10641096 struct arg_track *at_out, struct arg_track *at_stack_out,
1097+ const struct arg_track *at_stack_arg_entry,
10651098 struct func_instance *instance,
10661099 u32 *callsites)
10671100 {
@@ -1071,9 +1104,21 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn, · 3 unmodified lines
10711104 struct arg_track *dst = &at_out[insn->dst_reg];
10721105 struct arg_track *src = &at_out[insn->src_reg];
10731106 struct arg_track none = { .frame = ARG_NONE };
1074− int r;
1075−
1076− if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_K) {
1107+ int r, slot;
1108+
1109+ /* Handle stack arg stores and loads. */
1110+ if (is_stack_arg_st(insn) || is_stack_arg_stx(insn)) {
1111+ slot = stack_arg_off_to_slot(insn->off);
1112+ if (slot >= 0) {
1113+ if (is_stack_arg_stx(insn))
1114+ at_out[MAX_BPF_REG + slot] = at_out[insn->src_reg];
1115+ else
1116+ at_out[MAX_BPF_REG + slot] = none;
1117+ }
1118+ } else if (is_stack_arg_ldx(insn)) {
1119+ slot = stack_arg_off_to_slot(insn->off);
1120+ at_out[insn->dst_reg] = (slot >= 0) ? at_stack_arg_entry[slot] : none;
1121+ } else if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_K) {
10771122 if (code == BPF_MOV) {
10781123 *dst = none;
10791124 } else if (dst->frame >= 0) {
@@ -1297,6 +1342,16 @@ static int record_load_store_access(struct bpf_verifier_env *env, · 217 unmodified lines
12971342 struct arg_track resolved, *ptr;
12981343 int oi;
12991344
1345+ /*
1346+ * Stack arg insns use dst_reg/src_reg=BPF_REG_PARAMS(11). Since at[]
1347+ * is extended to MAX_AT_TRACK_REGS, at[11] holds the arg_track for
1348+ * outgoing stack arg slot 0 — not the pointer used for the memory
1349+ * access. Skip so the slot's tracked value isn't confused with the
1350+ * base register that record_stack_access() expects.
1351+ */
1352+ if (is_stack_arg_stx(insn) || is_stack_arg_st(insn) || is_stack_arg_ldx(insn))
1353+ return 0;
1354+
13001355 switch (class) {
13011356 case BPF_LDX:
13021357 ptr = &at[insn->src_reg];
@@ -1343,6 +1398,42 @@ static int record_load_store_access(struct bpf_verifier_env *env, · 40 unmodified lines
13431398 return 0;
13441399 }
13451400
1401+static int record_arg_access(struct bpf_verifier_env *env,
1402+ struct func_instance *instance,
1403+ struct bpf_insn *insn,
1404+ struct arg_track *at, int arg_idx,
1405+ int insn_idx)
1406+{
1407+ int depth = instance->depth;
1408+ int frame = at->frame;
1409+ int err = 0;
1410+ s64 bytes;
1411+
1412+ if (!arg_is_fp(at))
1413+ return 0;
1414+
1415+ if (bpf_helper_call(insn)) {
1416+ bytes = bpf_helper_stack_access_bytes(env, insn, arg_idx, insn_idx);
1417+ } else if (bpf_pseudo_kfunc_call(insn)) {
1418+ bytes = bpf_kfunc_stack_access_bytes(env, insn, arg_idx, insn_idx);
1419+ } else {
1420+ for (int f = 0; f <= depth; f++) {
1421+ err = mark_stack_read(instance, f, insn_idx, SPIS_ALL);
1422+ if (err)
1423+ return err;
1424+ }
1425+ return 0;
1426+ }
1427+ if (bytes == 0)
1428+ return 0;
1429+
1430+ if (frame >= 0 && frame <= depth)
1431+ err = record_stack_access(instance, at, bytes, frame, insn_idx);
1432+ else if (frame == ARG_IMPRECISE)
1433+ err = record_imprecise(instance, at->mask, insn_idx);
1434+ return err;
1435+}
1436+
13461437 /* Record stack access for a given 'at' state of helper/kfunc 'insn' */
13471438 static int record_call_access(struct bpf_verifier_env *env,
13481439 struct func_instance *instance,
@@ -1350,9 +1441,8 @@ static int record_call_access(struct bpf_verifier_env *env, · 1 unmodified line
13501441 int insn_idx)
13511442 {
13521443 struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
1353− int depth = instance->depth;
13541444 struct bpf_call_summary cs;
1355− int r, err = 0, num_params = 5;
1445+ int r, err, num_params = 5;
13561446
13571447 if (bpf_pseudo_call(insn))
13581448 return 0;
@@ -1360,32 +1450,15 @@ static int record_call_access(struct bpf_verifier_env *env, · 1 unmodified line
13601450 if (bpf_get_call_summary(env, insn, &cs))
13611451 num_params = cs.num_params;
13621452
1363− for (r = BPF_REG_1; r < BPF_REG_1 + num_params; r++) {
1364− int frame = at[r].frame;
1365− s64 bytes;
1366−
1367− if (!arg_is_fp(&at[r]))
1368− continue;
1369−
1370− if (bpf_helper_call(insn)) {
1371− bytes = bpf_helper_stack_access_bytes(env, insn, r - 1, insn_idx);
1372− } else if (bpf_pseudo_kfunc_call(insn)) {
1373− bytes = bpf_kfunc_stack_access_bytes(env, insn, r - 1, insn_idx);
1374− } else {
1375− for (int f = 0; f <= depth; f++) {
1376− err = mark_stack_read(instance, f, insn_idx, SPIS_ALL);
1377− if (err)
1378− return err;
1379− }
1380− return 0;
1381− }
1382− if (bytes == 0)
1383− continue;
1453+ for (r = BPF_REG_1; r < BPF_REG_1 + min(num_params, MAX_BPF_FUNC_REG_ARGS); r++) {
1454+ err = record_arg_access(env, instance, insn, &at[r], r - 1, insn_idx);
1455+ if (err)
1456+ return err;
1457+ }
13841458
1385− if (frame >= 0 && frame <= depth)
1386− err = record_stack_access(instance, &at[r], bytes, frame, insn_idx);
1387− else if (frame == ARG_IMPRECISE)
1388− err = record_imprecise(instance, at[r].mask, insn_idx);
1459+ for (r = 0; r < MAX_STACK_ARG_SLOTS && r < num_params - MAX_BPF_FUNC_REG_ARGS; r++) {
1460+ err = record_arg_access(env, instance, insn, &at[MAX_BPF_REG + r],
1461+ r + MAX_BPF_FUNC_REG_ARGS, insn_idx);
13891462 if (err)
13901463 return err;
13911464 }
@@ -1445,7 +1518,7 @@ static int find_callback_subprog(struct bpf_verifier_env *env, · 53 unmodified lines
14451518
14461519 /* Per-subprog intermediate state kept alive across analysis phases */
14471520 struct subprog_at_info {
1448− struct arg_track (*at_in)[MAX_BPF_REG];
1521+ struct arg_track (*at_in)[MAX_AT_TRACK_REGS];
14491522 int len;
14501523 };
14511524
@@ -1479,6 +1552,9 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, · 27 unmodified lines
14791552 for (r = 0; r < MAX_BPF_REG - 1; r++)
14801553 if (arg_is_fp(&info->at_in[i][r]))
14811554 has_extra = true;
1555+ for (r = 0; r < MAX_STACK_ARG_SLOTS; r++)
1556+ if (arg_is_fp(&info->at_in[i][MAX_BPF_REG + r]))
1557+ has_extra = true;
14821558 }
14831559 if (is_ldx_stx_call) {
14841560 for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++)
@@ -1503,6 +1579,12 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, · 18 unmodified lines
15031579 verbose(env, " r%d=", r);
15041580 verbose_arg_track(env, &info->at_in[i][r]);
15051581 }
1582+ for (r = 0; r < MAX_STACK_ARG_SLOTS; r++) {
1583+ if (!arg_is_fp(&info->at_in[i][MAX_BPF_REG + r]))
1584+ continue;
1585+ verbose(env, " sa%d=", r);
1586+ verbose_arg_track(env, &info->at_in[i][MAX_BPF_REG + r]);
1587+ }
15061588 }
15071589
15081590 if (is_ldx_stx_call) {
@@ -1525,7 +1607,7 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, · 16 unmodified lines
15251607 * Runs forward fixed-point with arg_track_xfer(), then records
15261608 * memory accesses in a single linear pass over converged state.
15271609 *
1528− * @callee_entry: pre-populated entry state for R1-R5
1610+ * @callee_entry: pre-populated entry state for R1-R5 and stack args
15291611 * NULL for main (subprog 0).
15301612 * @info: stores at_in, len for debug printing.
15311613 */
@@ -1543,10 +1625,11 @@ static int compute_subprog_args(struct bpf_verifier_env *env, · 11 unmodified lines
15431625 int end = env->subprog_info[subprog + 1].start;
15441626 int po_end = env->subprog_info[subprog + 1].postorder_start;
15451627 int len = end - start;
1546− struct arg_track (*at_in)[MAX_BPF_REG] = NULL;
1547− struct arg_track at_out[MAX_BPF_REG];
1628+ struct arg_track (*at_in)[MAX_AT_TRACK_REGS] = NULL;
1629+ struct arg_track at_out[MAX_AT_TRACK_REGS];
15481630 struct arg_track (*at_stack_in)[MAX_ARG_SPILL_SLOTS] = NULL;
15491631 struct arg_track *at_stack_out = NULL;
1632+ struct arg_track at_stack_arg_entry[MAX_STACK_ARG_SLOTS];
15501633 struct arg_track unvisited = { .frame = ARG_UNVISITED };
15511634 struct arg_track none = { .frame = ARG_NONE };
15521635 bool changed;
@@ -1565,13 +1648,13 @@ static int compute_subprog_args(struct bpf_verifier_env *env, · 12 unmodified lines
15651648 goto err_free;
15661649
15671650 for (i = 0; i < len; i++) {
1568− for (r = 0; r < MAX_BPF_REG; r++)
1651+ for (r = 0; r < MAX_AT_TRACK_REGS; r++)
15691652 at_in[i][r] = unvisited;
15701653 for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++)
15711654 at_stack_in[i][r] = unvisited;
15721655 }
15731656
1574− for (r = 0; r < MAX_BPF_REG; r++)
1657+ for (r = 0; r < MAX_AT_TRACK_REGS; r++)
15751658 at_in[0][r] = none;
15761659
15771660 /* Entry: R10 is always precisely the current frame's FP */
@@ -1587,6 +1670,10 @@ static int compute_subprog_args(struct bpf_verifier_env *env, · 9 unmodified lines
15871670 for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++)
15881671 at_stack_in[0][r] = none;
15891672
1673+ /* Entry: incoming stack args from caller, or ARG_NONE for main */
1674+ for (r = 0; r < MAX_STACK_ARG_SLOTS; r++)
1675+ at_stack_arg_entry[r] = callee_entry ? callee_entry[MAX_BPF_REG + r] : none;
1676+
15901677 if (env->log.level & BPF_LOG_LEVEL2)
15911678 verbose(env, "subprog#%d: analyzing (depth %d)...\n", subprog, depth);
15921679
@@ -1605,7 +1692,8 @@ redo: · 12 unmodified lines
16051692 memcpy(at_out, at_in[i], sizeof(at_out));
16061693 memcpy(at_stack_out, at_stack_in[i], MAX_ARG_SPILL_SLOTS * sizeof(*at_stack_out));
16071694
1608− arg_track_xfer(env, insn, idx, at_out, at_stack_out, instance, callsites);
1695+ arg_track_xfer(env, insn, idx, at_out, at_stack_out,
1696+ at_stack_arg_entry, instance, callsites);
16091697 arg_track_log(env, insn, idx, at_in[i], at_stack_in[i], at_out, at_stack_out);
16101698
16111699 /* Propagate to successors within this subprogram */
@@ -1619,7 +1707,7 @@ redo: · 7 unmodified lines
16191707 continue;
16201708 ti = target - start;
16211709
1622− for (r = 0; r < MAX_BPF_REG; r++)
1710+ for (r = 0; r < MAX_AT_TRACK_REGS; r++)
16231711 changed |= arg_track_join(env, idx, target, r,
16241712 &at_in[ti][r], at_out[r]);
16251713
@@ -1674,11 +1762,14 @@ err_free: · 48 unmodified lines
16741762 return err;
16751763 }
16761764
1677−/* Return true if any of R1-R5 is derived from a frame pointer. */
1765+/* Return true if any of R1-R5 or stack args is derived from a frame pointer. */
16781766 static bool has_fp_args(struct arg_track *args)
16791767 {
16801768 for (int r = BPF_REG_1; r <= BPF_REG_5; r++)
1681− if (args[r].frame != ARG_NONE)
1769+ if (arg_is_fp(&args[r]))
1770+ return true;
1771+ for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++)
1772+ if (arg_is_fp(&args[MAX_BPF_REG + r]))
16821773 return true;
16831774 return false;
16841775 }
@@ -1803,7 +1894,7 @@ static int analyze_subprog(struct bpf_verifier_env *env, · 118 unmodified lines
18031894 /* For each reachable call site in the subprog, recurse into callees */
18041895 for (int p = po_start; p < po_end; p++) {
18051896 int idx = env->cfg.insn_postorder[p];
1806− struct arg_track callee_args[BPF_REG_5 + 1];
1897+ struct arg_track callee_args[MAX_AT_TRACK_REGS] = {};
18071898 struct arg_track none = { .frame = ARG_NONE };
18081899 struct bpf_insn *insn = &insns[idx];
18091900 struct func_instance *callee_instance;
@@ -1818,9 +1909,11 @@ static int analyze_subprog(struct bpf_verifier_env *env, · 8 unmodified lines
18181909 if (callee < 0)
18191910 continue;
18201911
1821− /* Build entry args: R1-R5 from at_in at call site */
1912+ /* Build entry args: R1-R5 and stack args from at_in at call site */
18221913 for (int r = BPF_REG_1; r <= BPF_REG_5; r++)
18231914 callee_args[r] = info[subprog].at_in[j][r];
1915+ for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++)
1916+ callee_args[MAX_BPF_REG + r] = info[subprog].at_in[j][MAX_BPF_REG + r];
18241917 } else if (bpf_calls_callback(env, idx)) {
18251918 callee = find_callback_subprog(env, insn, idx, &caller_reg, &cb_callee_reg);
18261919 if (callee == -2) {
@@ -1842,6 +1935,8 @@ static int analyze_subprog(struct bpf_verifier_env *env, · 15 unmodified lines
18421935
18431936 for (int r = BPF_REG_1; r <= BPF_REG_5; r++)
18441937 callee_args[r] = none;
1938+ for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++)
1939+ callee_args[MAX_BPF_REG + r] = none;
18451940 callee_args[cb_callee_reg] = info[subprog].at_in[j][caller_reg];
18461941 } else {
18471942 continue;
@@ -2085,7 +2180,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, · 237 unmodified lines
20852180 def = ALL_CALLER_SAVED_REGS;
20862181 use = def & ~BIT(BPF_REG_0);
20872182 if (bpf_get_call_summary(env, insn, &cs))
2088− use = GENMASK(cs.num_params, 1);
2183+ use = GENMASK(min_t(u8, cs.num_params, MAX_BPF_FUNC_REG_ARGS), 1);
20892184 break;
20902185 default:
20912186 def = 0;
kernel/bpf/log.c+104−28
@@ -13,17 +13,17 @@ · 12 unmodified lines
1313
1414 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
1515
16−static bool bpf_verifier_log_attr_valid(const struct bpf_verifier_log *log)
16+static bool bpf_verifier_log_attr_valid(u32 log_level, char __user *log_buf, u32 log_size)
1717 {
1818 /* ubuf and len_total should both be specified (or not) together */
19− if (!!log->ubuf != !!log->len_total)
19+ if (!!log_buf != !!log_size)
2020 return false;
2121 /* log buf without log_level is meaningless */
22− if (log->ubuf && log->level == 0)
22+ if (log_buf && log_level == 0)
2323 return false;
24− if (log->level & ~BPF_LOG_MASK)
24+ if (log_level & ~BPF_LOG_MASK)
2525 return false;
26− if (log->len_total > UINT_MAX >> 2)
26+ if (log_size > UINT_MAX >> 2)
2727 return false;
2828 return true;
2929 }
@@ -36,7 +36,7 @@ int bpf_vlog_init(struct bpf_verifier_log *log, u32 log_level, · 6 unmodified lines
3636 log->len_total = log_size;
3737
3838 /* log attributes have to be sane */
39− if (!bpf_verifier_log_attr_valid(log))
39+ if (!bpf_verifier_log_attr_valid(log_level, log_buf, log_size))
4040 return -EINVAL;
4141
4242 return 0;
@@ -571,20 +571,20 @@ static void print_scalar_ranges(struct bpf_verifier_env *env, · 528 unmodified lines
571571 u64 val;
572572 bool omit;
573573 } minmaxs[] = {
574− {"smin", reg->smin_value, reg->smin_value == S64_MIN},
575− {"smax", reg->smax_value, reg->smax_value == S64_MAX},
576− {"umin", reg->umin_value, reg->umin_value == 0},
577− {"umax", reg->umax_value, reg->umax_value == U64_MAX},
574+ {"smin", reg_smin(reg), reg_smin(reg) == S64_MIN},
575+ {"smax", reg_smax(reg), reg_smax(reg) == S64_MAX},
576+ {"umin", reg_umin(reg), reg_umin(reg) == 0},
577+ {"umax", reg_umax(reg), reg_umax(reg) == U64_MAX},
578578 {"smin32",
579− is_snum_decimal((s64)reg->s32_min_value)
580− ? (s64)reg->s32_min_value
581− : (u32)reg->s32_min_value, reg->s32_min_value == S32_MIN},
579+ is_snum_decimal((s64)reg_s32_min(reg))
580+ ? (s64)reg_s32_min(reg)
581+ : (u32)reg_s32_min(reg), reg_s32_min(reg) == S32_MIN},
582582 {"smax32",
583− is_snum_decimal((s64)reg->s32_max_value)
584− ? (s64)reg->s32_max_value
585− : (u32)reg->s32_max_value, reg->s32_max_value == S32_MAX},
586− {"umin32", reg->u32_min_value, reg->u32_min_value == 0},
587− {"umax32", reg->u32_max_value, reg->u32_max_value == U32_MAX},
583+ is_snum_decimal((s64)reg_s32_max(reg))
584+ ? (s64)reg_s32_max(reg)
585+ : (u32)reg_s32_max(reg), reg_s32_max(reg) == S32_MAX},
586+ {"umin32", reg_u32_min(reg), reg_u32_min(reg) == 0},
587+ {"umax32", reg_u32_max(reg), reg_u32_max(reg) == U32_MAX},
588588 }, *m1, *m2, *mend = &minmaxs[ARRAY_SIZE(minmaxs)];
589589 bool neg1, neg2;
590590
@@ -665,8 +665,8 @@ static void print_reg_state(struct bpf_verifier_env *env, · 74 unmodified lines
665665 verbose_a("id=%d", reg->id & ~BPF_ADD_CONST);
666666 if (reg->id & BPF_ADD_CONST)
667667 verbose(env, "%+d", reg->delta);
668− if (reg->ref_obj_id)
669− verbose_a("ref_obj_id=%d", reg->ref_obj_id);
668+ if (reg->parent_id)
669+ verbose_a("parent_id=%d", reg->parent_id);
670670 if (type_is_non_owning_ref(reg->type))
671671 verbose_a("%s", "non_own_ref");
672672 if (type_is_map_ptr(t)) {
@@ -768,21 +768,19 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie · 95 unmodified lines
768768 verbose(env, "=dynptr_%s(", dynptr_type_str(reg->dynptr.type));
769769 if (reg->id)
770770 verbose_a("id=%d", reg->id);
771− if (reg->ref_obj_id)
772− verbose_a("ref_id=%d", reg->ref_obj_id);
773− if (reg->dynptr_id)
774− verbose_a("dynptr_id=%d", reg->dynptr_id);
771+ if (reg->parent_id)
772+ verbose_a("parent_id=%d", reg->parent_id);
775773 verbose(env, ")");
776774 break;
777775 case STACK_ITER:
778− /* only main slot has ref_obj_id set; skip others */
779− if (!reg->ref_obj_id)
776+ /* only main slot has id set; skip others */
777+ if (!reg->id)
780778 continue;
781779
782− verbose(env, " fp%d=iter_%s(ref_id=%d,state=%s,depth=%u)",
780+ verbose(env, " fp%d=iter_%s(id=%d,state=%s,depth=%u)",
783781 (-i - 1) * BPF_REG_SIZE,
784782 iter_type_str(reg->iter.btf, reg->iter.btf_id),
785− reg->ref_obj_id, iter_state_str(reg->iter.state),
783+ reg->id, iter_state_str(reg->iter.state),
786784 reg->iter.depth);
787785 break;
788786 case STACK_MISC:
@@ -825,3 +823,81 @@ void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_st · 36 unmodified lines
825823 }
826824 print_verifier_state(env, vstate, frameno, false);
827825 }
826+
827+int bpf_log_attr_init(struct bpf_log_attr *log, u64 log_buf, u32 log_size, u32 log_level,
828+ u32 offsetof_log_true_size, bpfptr_t uattr, struct bpf_common_attr *common,
829+ bpfptr_t uattr_common, u32 size_common)
830+{
831+ char __user *ubuf_common = u64_to_user_ptr(common->log_buf);
832+ char __user *ubuf = u64_to_user_ptr(log_buf);
833+
834+ if (!bpf_verifier_log_attr_valid(common->log_level, ubuf_common, common->log_size) ||
835+ !bpf_verifier_log_attr_valid(log_level, ubuf, log_size))
836+ return -EINVAL;
837+
838+ if (ubuf && ubuf_common && (ubuf != ubuf_common || log_size != common->log_size ||
839+ log_level != common->log_level))
840+ return -EINVAL;
841+
842+ memset(log, 0, sizeof(*log));
843+ log->ubuf = ubuf;
844+ log->size = log_size;
845+ log->level = log_level;
846+ log->offsetof_true_size = offsetof_log_true_size;
847+ log->uattr = uattr;
848+
849+ if (!ubuf && ubuf_common) {
850+ log->ubuf = ubuf_common;
851+ log->size = common->log_size;
852+ log->level = common->log_level;
853+ log->uattr = uattr_common;
854+ log->offsetof_true_size = 0;
855+ if (size_common >= offsetofend(struct bpf_common_attr, log_true_size))
856+ log->offsetof_true_size = offsetof(struct bpf_common_attr, log_true_size);
857+ }
858+ return 0;
859+}
860+
861+struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log,
862+ struct bpf_common_attr *common, bpfptr_t uattr,
863+ u32 size)
864+{
865+ struct bpf_verifier_log *log;
866+ int err;
867+
868+ memset(attr_log, 0, sizeof(*attr_log));
869+ attr_log->uattr = uattr;
870+ if (size >= offsetofend(struct bpf_common_attr, log_true_size))
871+ attr_log->offsetof_true_size = offsetof(struct bpf_common_attr, log_true_size);
872+
873+ if (!size)
874+ return NULL;
875+
876+ log = kzalloc_obj(*log, GFP_KERNEL);
877+ if (!log)
878+ return ERR_PTR(-ENOMEM);
879+
880+ err = bpf_vlog_init(log, common->log_level, u64_to_user_ptr(common->log_buf),
881+ common->log_size);
882+ if (err) {
883+ kfree(log);
884+ return ERR_PTR(err);
885+ }
886+
887+ return log;
888+}
889+
890+int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log)
891+{
892+ u32 log_true_size;
893+ int err;
894+
895+ err = bpf_vlog_finalize(log, &log_true_size);
896+
897+ if (attr->offsetof_true_size &&
898+ copy_to_bpfptr_offset(attr->uattr, attr->offsetof_true_size, &log_true_size,
899+ sizeof(log_true_size)))
900+ return -EFAULT;
901+
902+ return err;
903+}
kernel/bpf/lpm_trie.c+4−4
@@ -246,7 +246,7 @@ static void *trie_lookup_elem(struct bpf_map *map, void *_key) · 245 unmodified lines
246246
247247 /* Start walking the trie from the root node ... */
248248
249− for (node = rcu_dereference_check(trie->root, rcu_read_lock_bh_held());
249+ for (node = rcu_dereference_check(trie->root, bpf_rcu_lock_held());
250250 node;) {
251251 unsigned int next_bit;
252252 size_t matchlen;
@@ -280,7 +280,7 @@ static void *trie_lookup_elem(struct bpf_map *map, void *_key) · 27 unmodified lines
280280 */
281281 next_bit = extract_bit(key->data, node->prefixlen);
282282 node = rcu_dereference_check(node->child[next_bit],
283− rcu_read_lock_bh_held());
283+ bpf_rcu_lock_held());
284284 }
285285
286286 if (!found)
@@ -359,7 +359,7 @@ static long trie_update_elem(struct bpf_map *map, · 72 unmodified lines
359359 */
360360 slot = &trie->root;
361361
362− while ((node = rcu_dereference(*slot))) {
362+ while ((node = rcu_dereference_protected(*slot, 1))) {
363363 matchlen = longest_prefix_match(trie, node, key);
364364
365365 if (node->prefixlen != matchlen ||
@@ -482,7 +482,7 @@ static long trie_delete_elem(struct bpf_map *map, void *_key) · 116 unmodified lines
482482 trim = &trie->root;
483483 trim2 = trim;
484484 parent = NULL;
485− while ((node = rcu_dereference(*trim))) {
485+ while ((node = rcu_dereference_protected(*trim, 1))) {
486486 matchlen = longest_prefix_match(trie, node, key);
487487
488488 if (node->prefixlen != matchlen ||
kernel/bpf/map_in_map.c+4−1
@@ -20,7 +20,8 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) · 19 unmodified lines
2020 /* Does not support >1 level map-in-map */
2121 if (inner_map->inner_map_meta)
2222 return ERR_PTR(-EINVAL);
23−
23+ if (inner_map->excl_prog_sha)
24+ return ERR_PTR(-ENOTSUPP);
2425 if (!inner_map->ops->map_meta_equal)
2526 return ERR_PTR(-ENOTSUPP);
2627
@@ -101,6 +102,8 @@ void *bpf_map_fd_get_ptr(struct bpf_map *map, · 74 unmodified lines
101102 inner_map = __bpf_map_get(f);
102103 if (IS_ERR(inner_map))
103104 return inner_map;
105+ if (inner_map->excl_prog_sha)
106+ return ERR_PTR(-ENOTSUPP);
104107
105108 inner_map_meta = map->inner_map_meta;
106109 if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map))
kernel/bpf/map_iter.c+6−1
@@ -112,6 +112,10 @@ static int bpf_iter_attach_map(struct bpf_prog *prog, · 111 unmodified lines
112112 map = bpf_map_get_with_uref(linfo->map.map_fd);
113113 if (IS_ERR(map))
114114 return PTR_ERR(map);
115+ if (map->excl_prog_sha) {
116+ err = -EPERM;
117+ goto put_map;
118+ }
115119
116120 if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
117121 map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
@@ -119,7 +123,8 @@ static int bpf_iter_attach_map(struct bpf_prog *prog, · 1 unmodified line
119123 is_percpu = true;
120124 else if (map->map_type != BPF_MAP_TYPE_HASH &&
121125 map->map_type != BPF_MAP_TYPE_LRU_HASH &&
122− map->map_type != BPF_MAP_TYPE_ARRAY)
126+ map->map_type != BPF_MAP_TYPE_ARRAY &&
127+ map->map_type != BPF_MAP_TYPE_RHASH)
123128 goto put_map;
124129
125130 key_acc_size = prog->aux->max_rdonly_access;
kernel/bpf/stackmap.c+198−17
@@ -9,6 +9,7 @@ · 8 unmodified lines
99 #include <linux/perf_event.h>
1010 #include <linux/btf_ids.h>
1111 #include <linux/buildid.h>
12+#include <linux/mmap_lock.h>
1213 #include "percpu_freelist.h"
1314 #include "mmap_unlock_work.h"
1415
@@ -152,6 +153,180 @@ static int fetch_build_id(struct vm_area_struct *vma, unsigned char *build_id, b · 137 unmodified lines
152153 : build_id_parse_nofault(vma, build_id, NULL);
153154 }
154155
156+static inline void stack_map_build_id_set_ip(struct bpf_stack_build_id *id)
157+{
158+ id->status = BPF_STACK_BUILD_ID_IP;
159+ memset(id->build_id, 0, BUILD_ID_SIZE_MAX);
160+}
161+
162+static inline u64 stack_map_build_id_offset(unsigned long vm_pgoff,
163+ unsigned long vm_start, u64 ip)
164+{
165+ return (vm_pgoff << PAGE_SHIFT) + ip - vm_start;
166+}
167+
168+static inline void stack_map_build_id_set_valid(struct bpf_stack_build_id *id,
169+ u64 offset,
170+ const unsigned char *build_id)
171+{
172+ id->status = BPF_STACK_BUILD_ID_VALID;
173+ id->offset = offset;
174+ if (id->build_id != build_id)
175+ memcpy(id->build_id, build_id, BUILD_ID_SIZE_MAX);
176+}
177+
178+struct stack_map_vma_lock {
179+ struct vm_area_struct *vma;
180+ struct mm_struct *mm;
181+};
182+
183+/*
184+ * Acquire a stable read-side reference on the VMA covering @ip.
185+ *
186+ * With CONFIG_PER_VMA_LOCK=y this returns a VMA with its per-VMA read
187+ * lock held and mmap_lock dropped, so the caller may sleep.
188+ *
189+ * With CONFIG_PER_VMA_LOCK=n it returns a VMA with mmap_lock still
190+ * held; the caller must snapshot any fields it needs and pin vm_file
191+ * with get_file() before stack_map_unlock_vma() drops mmap_lock, as
192+ * the VMA may be split, merged, or freed after that.
193+ *
194+ * Returns NULL on failure, in which case no lock is held.
195+ */
196+static struct vm_area_struct *
197+stack_map_lock_vma(struct stack_map_vma_lock *lock, unsigned long ip)
198+{
199+ struct mm_struct *mm = lock->mm;
200+ struct vm_area_struct *vma;
201+
202+ /* noop under !CONFIG_PER_VMA_LOCK */
203+ vma = lock_vma_under_rcu(mm, ip);
204+ if (vma) {
205+ lock->vma = vma;
206+ return vma;
207+ }
208+
209+ /*
210+ * Taking mmap_read_lock() is unsafe here, because the caller BPF
211+ * program might already hold it, causing a deadlock.
212+ */
213+ if (!mmap_read_trylock(mm))
214+ return NULL;
215+
216+ vma = vma_lookup(mm, ip);
217+ if (!vma) {
218+ mmap_read_unlock(mm);
219+ return NULL;
220+ }
221+
222+#ifdef CONFIG_PER_VMA_LOCK
223+ if (!vma_start_read_locked(vma)) {
224+ mmap_read_unlock(mm);
225+ return NULL;
226+ }
227+ mmap_read_unlock(mm);
228+#endif
229+
230+ lock->vma = vma;
231+ return vma;
232+}
233+
234+static void stack_map_unlock_vma(struct stack_map_vma_lock *lock)
235+{
236+#ifdef CONFIG_PER_VMA_LOCK
237+ vma_end_read(lock->vma);
238+#else
239+ mmap_read_unlock(lock->mm);
240+#endif
241+ lock->vma = NULL;
242+}
243+
244+static void stack_map_get_build_id_offset_sleepable(struct bpf_stack_build_id *id_offs,
245+ u32 trace_nr)
246+{
247+ struct mm_struct *mm = current->mm;
248+ struct stack_map_vma_lock lock = { .mm = mm };
249+ struct {
250+ struct file *file;
251+ const unsigned char *build_id;
252+ unsigned long vm_start;
253+ unsigned long vm_end;
254+ unsigned long vm_pgoff;
255+ } cache = {};
256+ unsigned long vm_pgoff, vm_start, vm_end;
257+ struct vm_area_struct *vma;
258+ struct file *file;
259+ u64 offset;
260+ u64 ip;
261+
262+ for (u32 i = 0; i < trace_nr; i++) {
263+ ip = READ_ONCE(id_offs[i].ip);
264+
265+ /*
266+ * Range cache fast path: if ip falls within the previously
267+ * resolved VMA range, reuse the cache build_id without
268+ * re-acquiring the VMA lock.
269+ */
270+ if (cache.build_id && ip >= cache.vm_start && ip < cache.vm_end) {
271+ offset = stack_map_build_id_offset(cache.vm_pgoff, cache.vm_start, ip);
272+ stack_map_build_id_set_valid(&id_offs[i], offset, cache.build_id);
273+ continue;
274+ }
275+
276+ vma = stack_map_lock_vma(&lock, ip);
277+ if (!vma) {
278+ stack_map_build_id_set_ip(&id_offs[i]);
279+ continue;
280+ }
281+ if (vma_is_anonymous(vma) || !vma->vm_file) {
282+ stack_map_build_id_set_ip(&id_offs[i]);
283+ stack_map_unlock_vma(&lock);
284+ continue;
285+ }
286+
287+ file = vma->vm_file;
288+ vm_pgoff = vma->vm_pgoff;
289+ vm_start = vma->vm_start;
290+ vm_end = vma->vm_end;
291+ offset = stack_map_build_id_offset(vm_pgoff, vm_start, ip);
292+
293+ /*
294+ * Same backing file as previous (e.g. different VMAs
295+ * of the same ELF binary). Reuse the cache build_id.
296+ */
297+ if (file == cache.file) {
298+ stack_map_unlock_vma(&lock);
299+ stack_map_build_id_set_valid(&id_offs[i], offset, cache.build_id);
300+ cache.vm_start = vm_start;
301+ cache.vm_end = vm_end;
302+ cache.vm_pgoff = vm_pgoff;
303+ continue;
304+ }
305+
306+ file = get_file(file);
307+ stack_map_unlock_vma(&lock);
308+
309+ /* build_id_parse_file() may block on filesystem reads */
310+ if (build_id_parse_file(file, id_offs[i].build_id, NULL)) {
311+ stack_map_build_id_set_ip(&id_offs[i]);
312+ fput(file);
313+ continue;
314+ }
315+
316+ stack_map_build_id_set_valid(&id_offs[i], offset, id_offs[i].build_id);
317+ if (cache.file)
318+ fput(cache.file);
319+ cache.file = file;
320+ cache.build_id = id_offs[i].build_id;
321+ cache.vm_start = vm_start;
322+ cache.vm_end = vm_end;
323+ cache.vm_pgoff = vm_pgoff;
324+ }
325+
326+ if (cache.file)
327+ fput(cache.file);
328+}
329+
155330 /*
156331 * Expects all id_offs[i].ip values to be set to correct initial IPs.
157332 * They will be subsequently:
@@ -165,44 +340,50 @@ static int fetch_build_id(struct vm_area_struct *vma, unsigned char *build_id, b · 7 unmodified lines
165340 static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
166341 u32 trace_nr, bool user, bool may_fault)
167342 {
168− int i;
169343 struct mmap_unlock_irq_work *work = NULL;
170344 bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work);
345+ bool has_user_ctx = user && current && current->mm;
171346 struct vm_area_struct *vma, *prev_vma = NULL;
172− const char *prev_build_id;
347+ const unsigned char *prev_build_id = NULL;
348+ int i;
349+
350+ if (may_fault && has_user_ctx) {
351+ stack_map_get_build_id_offset_sleepable(id_offs, trace_nr);
352+ return;
353+ }
173354
174355 /* If the irq_work is in use, fall back to report ips. Same
175356 * fallback is used for kernel stack (!user) on a stackmap with
176357 * build_id.
177358 */
178− if (!user || !current || !current->mm || irq_work_busy ||
179− !mmap_read_trylock(current->mm)) {
359+ if (!has_user_ctx || irq_work_busy || !mmap_read_trylock(current->mm)) {
180360 /* cannot access current->mm, fall back to ips */
181− for (i = 0; i < trace_nr; i++) {
182− id_offs[i].status = BPF_STACK_BUILD_ID_IP;
183− memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX);
184− }
361+ for (i = 0; i < trace_nr; i++)
362+ stack_map_build_id_set_ip(&id_offs[i]);
185363 return;
186364 }
187365
188366 for (i = 0; i < trace_nr; i++) {
189367 u64 ip = READ_ONCE(id_offs[i].ip);
368+ u64 offset;
190369
191− if (range_in_vma(prev_vma, ip, ip)) {
370+ if (prev_build_id && range_in_vma(prev_vma, ip, ip)) {
192371 vma = prev_vma;
193− memcpy(id_offs[i].build_id, prev_build_id, BUILD_ID_SIZE_MAX);
194− goto build_id_valid;
372+ offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip);
373+ stack_map_build_id_set_valid(&id_offs[i], offset, prev_build_id);
374+ continue;
195375 }
196376 vma = find_vma(current->mm, ip);
197− if (!vma || fetch_build_id(vma, id_offs[i].build_id, may_fault)) {
377+ if (!vma || vma_is_anonymous(vma) ||
378+ fetch_build_id(vma, id_offs[i].build_id, may_fault)) {
198379 /* per entry fall back to ips */
199− id_offs[i].status = BPF_STACK_BUILD_ID_IP;
200− memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX);
380+ stack_map_build_id_set_ip(&id_offs[i]);
381+ prev_vma = vma;
382+ prev_build_id = NULL;
201383 continue;
202384 }
203−build_id_valid:
204− id_offs[i].offset = (vma->vm_pgoff << PAGE_SHIFT) + ip - vma->vm_start;
205− id_offs[i].status = BPF_STACK_BUILD_ID_VALID;
385+ offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip);
386+ stack_map_build_id_set_valid(&id_offs[i], offset, id_offs[i].build_id);
206387 prev_vma = vma;
207388 prev_build_id = id_offs[i].build_id;
208389 }
kernel/bpf/states.c+51−16
@@ -2,6 +2,7 @@ · 1 unmodified line
22 /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
33 #include <linux/bpf.h>
44 #include <linux/bpf_verifier.h>
5+#include <linux/cnum.h>
56 #include <linux/filter.h>
67
78 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -301,14 +302,8 @@ int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_s · 293 unmodified lines
301302 static bool range_within(const struct bpf_reg_state *old,
302303 const struct bpf_reg_state *cur)
303304 {
304− return old->umin_value <= cur->umin_value &&
305− old->umax_value >= cur->umax_value &&
306− old->smin_value <= cur->smin_value &&
307− old->smax_value >= cur->smax_value &&
308− old->u32_min_value <= cur->u32_min_value &&
309− old->u32_max_value >= cur->u32_max_value &&
310− old->s32_min_value <= cur->s32_min_value &&
311− old->s32_max_value >= cur->s32_max_value;
305+ return cnum64_is_subset(old->r64, cur->r64) &&
306+ cnum32_is_subset(old->r32, cur->r32);
312307 }
313308
314309 /* If in the old state two registers had the same id, then they need to have
@@ -348,8 +343,12 @@ static bool check_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap) · 33 unmodified lines
348343 return true;
349344 }
350345
351− /* We ran out of idmap slots, which should be impossible */
352− WARN_ON_ONCE(1);
346+ /*
347+ * idmap slots are bounded by the number of registers and stack slots.
348+ * Since referenced dynptrs acquire intermediate references that do
349+ * not live in either, so the map can be exhausted. Since it is unlikely,
350+ * fail the verification by treating the states as not equivalent.
351+ */
353352 return false;
354353 }
355354
@@ -494,7 +493,7 @@ static bool regs_exact(const struct bpf_reg_state *rold, · 138 unmodified lines
494493 {
495494 return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&
496495 check_ids(rold->id, rcur->id, idmap) &&
497− check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap);
496+ check_ids(rold->parent_id, rcur->parent_id, idmap);
498497 }
499498
500499 enum exact_level {
@@ -619,7 +618,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, · 118 unmodified lines
619618 range_within(rold, rcur) &&
620619 tnum_in(rold->var_off, rcur->var_off) &&
621620 check_ids(rold->id, rcur->id, idmap) &&
622− check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap);
621+ check_ids(rold->parent_id, rcur->parent_id, idmap);
623622 case PTR_TO_PACKET_META:
624623 case PTR_TO_PACKET:
625624 /* We must have at least as much range as the old ptr
@@ -799,7 +798,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, · 173 unmodified lines
799798 cur_reg = &cur->stack[spi].spilled_ptr;
800799 if (old_reg->dynptr.type != cur_reg->dynptr.type ||
801800 old_reg->dynptr.first_slot != cur_reg->dynptr.first_slot ||
802− !check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap))
801+ !check_ids(old_reg->id, cur_reg->id, idmap) ||
802+ !check_ids(old_reg->parent_id, cur_reg->parent_id, idmap))
803803 return false;
804804 break;
805805 case STACK_ITER:
@@ -815,13 +815,13 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, · 9 unmodified lines
815815 old_reg->iter.btf_id != cur_reg->iter.btf_id ||
816816 old_reg->iter.state != cur_reg->iter.state ||
817817 /* ignore {old_reg,cur_reg}->iter.depth, see above */
818− !check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap))
818+ !check_ids(old_reg->id, cur_reg->id, idmap))
819819 return false;
820820 break;
821821 case STACK_IRQ_FLAG:
822822 old_reg = &old->stack[spi].spilled_ptr;
823823 cur_reg = &cur->stack[spi].spilled_ptr;
824− if (!check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap) ||
824+ if (!check_ids(old_reg->id, cur_reg->id, idmap) ||
825825 old_reg->irq.kfunc_class != cur_reg->irq.kfunc_class)
826826 return false;
827827 break;
@@ -838,6 +838,32 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, · 10 unmodified lines
838838 return true;
839839 }
840840
841+/*
842+ * Compare stack arg slots between old and current states.
843+ * Outgoing stack args are path-local state and must agree for pruning.
844+ */
845+static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
846+ struct bpf_func_state *cur, struct bpf_idmap *idmap,
847+ enum exact_level exact)
848+{
849+ int i, nslots;
850+
851+ nslots = max(old->out_stack_arg_cnt, cur->out_stack_arg_cnt);
852+ for (i = 0; i < nslots; i++) {
853+ struct bpf_reg_state *old_arg, *cur_arg;
854+ struct bpf_reg_state not_init = { .type = NOT_INIT };
855+
856+ old_arg = i < old->out_stack_arg_cnt ?
857+ &old->stack_arg_regs[i] : &not_init;
858+ cur_arg = i < cur->out_stack_arg_cnt ?
859+ &cur->stack_arg_regs[i] : &not_init;
860+ if (!regsafe(env, old_arg, cur_arg, idmap, exact))
861+ return false;
862+ }
863+
864+ return true;
865+}
866+
841867 static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *cur,
842868 struct bpf_idmap *idmap)
843869 {
@@ -868,6 +894,9 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c · 24 unmodified lines
868894 return false;
869895 switch (old->refs[i].type) {
870896 case REF_TYPE_PTR:
897+ if (!check_ids(old->refs[i].parent_id, cur->refs[i].parent_id, idmap))
898+ return false;
899+ break;
871900 case REF_TYPE_IRQ:
872901 break;
873902 case REF_TYPE_LOCK:
@@ -920,6 +949,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat · 46 unmodified lines
920949 if (old->callback_depth > cur->callback_depth)
921950 return false;
922951
952+ if (!old->no_stack_arg_load && cur->no_stack_arg_load)
953+ return false;
954+
923955 for (i = 0; i < MAX_BPF_REG; i++)
924956 if (((1 << i) & live_regs) &&
925957 !regsafe(env, &old->regs[i], &cur->regs[i],
@@ -929,6 +961,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat · 3 unmodified lines
929961 if (!stacksafe(env, old, cur, &env->idmap_scratch, exact))
930962 return false;
931963
964+ if (!stack_arg_safe(env, old, cur, &env->idmap_scratch, exact))
965+ return false;
966+
932967 return true;
933968 }
934969
@@ -1376,7 +1411,7 @@ hit: · 441 unmodified lines
13761411 */
13771412 err = 0;
13781413 if (bpf_is_jmp_point(env, env->insn_idx))
1379− err = bpf_push_jmp_history(env, cur, 0, 0);
1414+ err = bpf_push_jmp_history(env, cur, 0, 0, 0, 0);
13801415 err = err ? : propagate_precision(env, &sl->state, cur, NULL);
13811416 if (err)
13821417 return err;
kernel/bpf/syscall.c+231−81
@@ -41,6 +41,7 @@ · 40 unmodified lines
4141 #include <linux/overflow.h>
4242 #include <linux/cookie.h>
4343 #include <linux/verification.h>
44+#include <linux/btf_ids.h>
4445
4546 #include <net/netfilter/nf_bpf_link.h>
4647 #include <net/netkit.h>
@@ -807,6 +808,11 @@ void bpf_obj_free_task_work(const struct btf_record *rec, void *obj) · 760 unmodified lines
807808 bpf_task_work_cancel_and_free(obj + rec->task_work_off);
808809 }
809810
811+void bpf_obj_cancel_fields(struct bpf_map *map, void *obj)
812+{
813+ bpf_map_free_internal_structs(map, obj);
814+}
815+
810816 void bpf_obj_free_fields(const struct btf_record *rec, void *obj)
811817 {
812818 const struct btf_field *fields;
@@ -1280,6 +1286,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, · 467 unmodified lines
12801286 case BPF_SPIN_LOCK:
12811287 case BPF_RES_SPIN_LOCK:
12821288 if (map->map_type != BPF_MAP_TYPE_HASH &&
1289+ map->map_type != BPF_MAP_TYPE_RHASH &&
12831290 map->map_type != BPF_MAP_TYPE_ARRAY &&
12841291 map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE &&
12851292 map->map_type != BPF_MAP_TYPE_SK_STORAGE &&
@@ -1294,6 +1301,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, · 8 unmodified lines
12941301 case BPF_WORKQUEUE:
12951302 case BPF_TASK_WORK:
12961303 if (map->map_type != BPF_MAP_TYPE_HASH &&
1304+ map->map_type != BPF_MAP_TYPE_RHASH &&
12971305 map->map_type != BPF_MAP_TYPE_LRU_HASH &&
12981306 map->map_type != BPF_MAP_TYPE_ARRAY) {
12991307 ret = -EOPNOTSUPP;
@@ -1305,6 +1313,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, · 5 unmodified lines
13051313 case BPF_KPTR_PERCPU:
13061314 case BPF_REFCOUNT:
13071315 if (map->map_type != BPF_MAP_TYPE_HASH &&
1316+ map->map_type != BPF_MAP_TYPE_RHASH &&
13081317 map->map_type != BPF_MAP_TYPE_PERCPU_HASH &&
13091318 map->map_type != BPF_MAP_TYPE_LRU_HASH &&
13101319 map->map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH &&
@@ -1359,7 +1368,8 @@ free_map_tab: · 48 unmodified lines
13591368
13601369 #define BPF_MAP_CREATE_LAST_FIELD excl_prog_hash_size
13611370 /* called via syscall */
1362−static int map_create(union bpf_attr *attr, bpfptr_t uattr)
1371+static int map_create_alloc(union bpf_attr *attr, bpfptr_t uattr, struct bpf_verifier_log *log,
1372+ struct bpf_map **mapp, struct bpf_token **tokenp)
13631373 {
13641374 const struct bpf_map_ops *ops;
13651375 struct bpf_token *token = NULL;
@@ -1367,12 +1377,13 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 1 unmodified line
13671377 u32 map_type = attr->map_type;
13681378 struct bpf_map *map;
13691379 bool token_flag;
1370− int f_flags;
13711380 int err;
13721381
13731382 err = CHECK_ATTR(BPF_MAP_CREATE);
1374− if (err)
1383+ if (err) {
1384+ bpf_log(log, "Invalid attr.\n");
13751385 return -EINVAL;
1386+ }
13761387
13771388 /* check BPF_F_TOKEN_FD flag, remember if it's set, and then clear it
13781389 * to avoid per-map type checks tripping on unknown flag
@@ -1381,31 +1392,40 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 2 unmodified lines
13811392 attr->map_flags &= ~BPF_F_TOKEN_FD;
13821393
13831394 if (attr->btf_vmlinux_value_type_id) {
1384− if (attr->map_type != BPF_MAP_TYPE_STRUCT_OPS ||
1385− attr->btf_key_type_id || attr->btf_value_type_id)
1395+ if (attr->map_type != BPF_MAP_TYPE_STRUCT_OPS) {
1396+ bpf_log(log, "btf_vmlinux_value_type_id can only be used with struct_ops maps.\n");
13861397 return -EINVAL;
1398+ }
1399+ if (attr->btf_key_type_id || attr->btf_value_type_id) {
1400+ bpf_log(log, "btf_vmlinux_value_type_id is mutually exclusive with btf_key_type_id and btf_value_type_id.\n");
1401+ return -EINVAL;
1402+ }
13871403 } else if (attr->btf_key_type_id && !attr->btf_value_type_id) {
1404+ bpf_log(log, "Invalid btf_value_type_id.\n");
13881405 return -EINVAL;
13891406 }
13901407
13911408 if (attr->map_type != BPF_MAP_TYPE_BLOOM_FILTER &&
13921409 attr->map_type != BPF_MAP_TYPE_ARENA &&
1393− attr->map_extra != 0)
1410+ attr->map_type != BPF_MAP_TYPE_RHASH &&
1411+ attr->map_extra != 0) {
1412+ bpf_log(log, "Invalid map_extra.\n");
13941413 return -EINVAL;
1395−
1396− f_flags = bpf_get_file_flag(attr->map_flags);
1397− if (f_flags < 0)
1398− return f_flags;
1414+ }
13991415
14001416 if (numa_node != NUMA_NO_NODE &&
14011417 ((unsigned int)numa_node >= nr_node_ids ||
1402− !node_online(numa_node)))
1418+ !node_online(numa_node))) {
1419+ bpf_log(log, "Invalid numa_node.\n");
14031420 return -EINVAL;
1421+ }
14041422
14051423 /* find map type and init map: hashtable vs rbtree vs bloom vs ... */
14061424 map_type = attr->map_type;
1407− if (map_type >= ARRAY_SIZE(bpf_map_types))
1425+ if (map_type >= ARRAY_SIZE(bpf_map_types)) {
1426+ bpf_log(log, "Invalid map_type.\n");
14081427 return -EINVAL;
1428+ }
14091429 map_type = array_index_nospec(map_type, ARRAY_SIZE(bpf_map_types));
14101430 ops = bpf_map_types[map_type];
14111431 if (!ops)
@@ -1423,8 +1443,10 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 11 unmodified lines
14231443
14241444 if (token_flag) {
14251445 token = bpf_token_get_from_fd(attr->map_token_fd);
1426− if (IS_ERR(token))
1446+ if (IS_ERR(token)) {
1447+ bpf_log(log, "Invalid map_token_fd.\n");
14271448 return PTR_ERR(token);
1449+ }
14281450
14291451 /* if current token doesn't grant map creation permissions,
14301452 * then we can't use this token, so ignore it and rely on
@@ -1457,6 +1479,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 26 unmodified lines
14571479 case BPF_MAP_TYPE_CGROUP_ARRAY:
14581480 case BPF_MAP_TYPE_ARRAY_OF_MAPS:
14591481 case BPF_MAP_TYPE_HASH:
1482+ case BPF_MAP_TYPE_RHASH:
14601483 case BPF_MAP_TYPE_PERCPU_HASH:
14611484 case BPF_MAP_TYPE_HASH_OF_MAPS:
14621485 case BPF_MAP_TYPE_RINGBUF:
@@ -1507,8 +1530,10 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 44 unmodified lines
15071530
15081531 err = bpf_obj_name_cpy(map->name, attr->map_name,
15091532 sizeof(attr->map_name));
1510− if (err < 0)
1533+ if (err < 0) {
1534+ bpf_log(log, "Invalid map_name.\n");
15111535 goto free_map;
1536+ }
15121537
15131538 preempt_disable();
15141539 map->cookie = gen_cookie_next(&bpf_map_cookie);
@@ -1531,6 +1556,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 16 unmodified lines
15311556
15321557 btf = btf_get_by_fd(attr->btf_fd);
15331558 if (IS_ERR(btf)) {
1559+ bpf_log(log, "Invalid btf_fd.\n");
15341560 err = PTR_ERR(btf);
15351561 goto free_map;
15361562 }
@@ -1558,6 +1584,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 21 unmodified lines
15581584 bpfptr_t uprog_hash = make_bpfptr(attr->excl_prog_hash, uattr.is_kernel);
15591585
15601586 if (attr->excl_prog_hash_size != SHA256_DIGEST_SIZE) {
1587+ bpf_log(log, "Invalid excl_prog_hash_size.\n");
15611588 err = -EINVAL;
15621589 goto free_map;
15631590 }
@@ -1572,11 +1599,62 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 8 unmodified lines
15721599 err = -EFAULT;
15731600 goto free_map;
15741601 }
1602+
1603+ /* See libbpf: emit_signature_match() */
1604+ BUILD_BUG_ON(offsetof(struct bpf_map, excl) != SHA256_DIGEST_SIZE);
1605+ BUILD_BUG_ON(!__same_type(map->excl, u32));
1606+ BUILD_BUG_ON(offsetof(struct bpf_map, sha) != 0);
1607+ BUILD_BUG_ON(!__same_type(map->sha, u8[SHA256_DIGEST_SIZE]));
1608+ map->excl = 1;
15751609 } else if (attr->excl_prog_hash_size) {
1610+ bpf_log(log, "Invalid excl_prog_hash_size.\n");
15761611 err = -EINVAL;
15771612 goto free_map;
15781613 }
15791614
1615+ *mapp = map;
1616+ *tokenp = token;
1617+ return 0;
1618+
1619+free_map:
1620+ bpf_map_free(map);
1621+put_token:
1622+ bpf_token_put(token);
1623+ return err;
1624+}
1625+
1626+static int map_create(union bpf_attr *attr, bpfptr_t uattr, struct bpf_common_attr *attr_common,
1627+ bpfptr_t uattr_common, u32 size_common)
1628+{
1629+ struct bpf_token *token = NULL;
1630+ struct bpf_verifier_log *log;
1631+ struct bpf_log_attr attr_log;
1632+ struct bpf_map *map = NULL;
1633+ int err, ret;
1634+ int f_flags;
1635+
1636+ log = bpf_log_attr_create_vlog(&attr_log, attr_common, uattr_common, size_common);
1637+ if (IS_ERR(log))
1638+ return PTR_ERR(log);
1639+
1640+ err = map_create_alloc(attr, uattr, log, &map, &token);
1641+
1642+ /* preserve original error even if log finalization is successful */
1643+ ret = bpf_log_attr_finalize(&attr_log, log);
1644+ if (ret)
1645+ err = ret;
1646+
1647+ kfree(log);
1648+
1649+ if (err)
1650+ goto free_map;
1651+
1652+ f_flags = bpf_get_file_flag(attr->map_flags);
1653+ if (f_flags < 0) {
1654+ err = f_flags;
1655+ goto free_map;
1656+ }
1657+
15801658 err = security_bpf_map_create(map, attr, token, uattr.is_kernel);
15811659 if (err)
15821660 goto free_map_sec;
@@ -1605,8 +1683,8 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 22 unmodified lines
16051683 free_map_sec:
16061684 security_bpf_map_free(map);
16071685 free_map:
1608− bpf_map_free(map);
1609−put_token:
1686+ if (map)
1687+ bpf_map_free(map);
16101688 bpf_token_put(token);
16111689 return err;
16121690 }
@@ -2192,6 +2270,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) · 579 unmodified lines
21922270 map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
21932271 map->map_type == BPF_MAP_TYPE_LRU_HASH ||
21942272 map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
2273+ map->map_type == BPF_MAP_TYPE_RHASH ||
21952274 map->map_type == BPF_MAP_TYPE_STACK_TRACE) {
21962275 if (!bpf_map_is_offloaded(map)) {
21972276 bpf_disable_instrumentation();
@@ -2646,7 +2725,8 @@ static int · 448 unmodified lines
26462725 bpf_prog_load_check_attach(enum bpf_prog_type prog_type,
26472726 enum bpf_attach_type expected_attach_type,
26482727 struct btf *attach_btf, u32 btf_id,
2649− struct bpf_prog *dst_prog)
2728+ struct bpf_prog *dst_prog,
2729+ bool multi_func)
26502730 {
26512731 if (btf_id) {
26522732 if (btf_id > BTF_MAX_TYPE)
@@ -2666,6 +2746,14 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type, · 13 unmodified lines
26662746 }
26672747 }
26682748
2749+ if (multi_func) {
2750+ if (prog_type != BPF_PROG_TYPE_TRACING)
2751+ return -EINVAL;
2752+ if (!attach_btf || btf_id)
2753+ return -EINVAL;
2754+ return 0;
2755+ }
2756+
26692757 if (attach_btf && (!btf_id || dst_prog))
26702758 return -EINVAL;
26712759
@@ -2798,8 +2886,22 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type) · 126 unmodified lines
27982886 }
27992887 }
28002888
2889+static enum bpf_sig_keyring bpf_classify_keyring(s32 keyring_id)
2890+{
2891+ switch (keyring_id) {
2892+ case 0:
2893+ return BPF_SIG_KEYRING_BUILTIN;
2894+ case (s32)(unsigned long)VERIFY_USE_SECONDARY_KEYRING:
2895+ return BPF_SIG_KEYRING_SECONDARY;
2896+ case (s32)(unsigned long)VERIFY_USE_PLATFORM_KEYRING:
2897+ return BPF_SIG_KEYRING_PLATFORM;
2898+ default:
2899+ return BPF_SIG_KEYRING_USER;
2900+ }
2901+}
2902+
28012903 static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr,
2802− bool is_kernel)
2904+ bool is_kernel, s32 *keyring_serial)
28032905 {
28042906 bpfptr_t usig = make_bpfptr(attr->signature, is_kernel);
28052907 struct bpf_dynptr_kern sig_ptr, insns_ptr;
@@ -2835,7 +2937,8 @@ static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr · 29 unmodified lines
28352937
28362938 err = bpf_verify_pkcs7_signature((struct bpf_dynptr *)&insns_ptr,
28372939 (struct bpf_dynptr *)&sig_ptr, key);
2838−
2940+ if (!err)
2941+ *keyring_serial = bpf_key_serial(key);
28392942 bpf_key_put(key);
28402943 kvfree(sig);
28412944 return err;
@@ -2858,10 +2961,15 @@ static int bpf_prog_mark_insn_arrays_ready(struct bpf_prog *prog) · 16 unmodified lines
28582961 return 0;
28592962 }
28602963
2964+extern int bpf_multi_func(void);
2965+int __init __used bpf_multi_func(void) { return 0; }
2966+
2967+BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
2968+
28612969 /* last field in 'union bpf_attr' used by this command */
28622970 #define BPF_PROG_LOAD_LAST_FIELD keyring_id
28632971
2864−static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
2972+static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
28652973 {
28662974 enum bpf_prog_type type = attr->prog_type;
28672975 struct bpf_prog *prog, *dst_prog = NULL;
@@ -2870,6 +2978,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 2 unmodified lines
28702978 bool bpf_cap;
28712979 int err;
28722980 char license[128];
2981+ bool multi_func;
28732982
28742983 if (CHECK_ATTR(BPF_PROG_LOAD))
28752984 return -EINVAL;
@@ -2936,6 +3045,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 60 unmodified lines
29363045 if (is_perfmon_prog_type(type) && !bpf_token_capable(token, CAP_PERFMON))
29373046 goto put_token;
29383047
3048+ multi_func = is_tracing_multi(attr->expected_attach_type);
3049+
29393050 /* attach_prog_fd/attach_btf_obj_fd can specify fd of either bpf_prog
29403051 * or btf, we need to check which one it is
29413052 */
@@ -2957,7 +3068,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 15 unmodified lines
29573068 goto put_token;
29583069 }
29593070 }
2960− } else if (attr->attach_btf_id) {
3071+ } else if (attr->attach_btf_id || multi_func) {
29613072 /* fall back to vmlinux BTF, if BTF type ID is specified */
29623073 attach_btf = bpf_get_btf_vmlinux();
29633074 if (IS_ERR(attach_btf)) {
@@ -2973,7 +3084,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 9 unmodified lines
29733084
29743085 if (bpf_prog_load_check_attach(type, attr->expected_attach_type,
29753086 attach_btf, attr->attach_btf_id,
2976− dst_prog)) {
3087+ dst_prog, multi_func)) {
29773088 if (dst_prog)
29783089 bpf_prog_put(dst_prog);
29793090 if (attach_btf)
@@ -2996,7 +3107,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 16 unmodified lines
29963107 prog->expected_attach_type = attr->expected_attach_type;
29973108 prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE);
29983109 prog->aux->attach_btf = attach_btf;
2999− prog->aux->attach_btf_id = attr->attach_btf_id;
3110+ prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id;
30003111 prog->aux->dst_prog = dst_prog;
30013112 prog->aux->dev_bound = !!attr->prog_ifindex;
30023113 prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS;
@@ -3022,13 +3133,17 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 19 unmodified lines
30223133
30233134 /* eBPF programs must be GPL compatible to use GPL-ed functions */
30243135 prog->gpl_compatible = license_is_gpl_compatible(license) ? 1 : 0;
3025−
30263136 if (attr->signature) {
3027− err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel);
3137+ err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel,
3138+ &prog->aux->sig.keyring_serial);
30283139 if (err)
30293140 goto free_prog;
3141+ prog->aux->sig.keyring_type = bpf_classify_keyring(attr->keyring_id);
3142+ prog->aux->sig.verdict = BPF_SIG_VERIFIED;
3143+ } else {
3144+ prog->aux->sig.keyring_type = BPF_SIG_KEYRING_NONE;
3145+ prog->aux->sig.verdict = BPF_SIG_UNSIGNED;
30303146 }
3031−
30323147 prog->orig_prog = NULL;
30333148 prog->jited = 0;
30343149
@@ -3076,10 +3191,10 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 41 unmodified lines
30763191
30773192 err = security_bpf_prog_load(prog, attr, token, uattr.is_kernel);
30783193 if (err)
3079− goto free_prog_sec;
3194+ goto free_prog;
30803195
30813196 /* run eBPF verifier */
3082− err = bpf_check(&prog, attr, uattr, uattr_size);
3197+ err = bpf_check(&prog, attr, uattr, attr_log);
30833198 if (err < 0)
30843199 goto free_used_maps;
30853200
@@ -3122,8 +3237,6 @@ free_used_maps: · 36 unmodified lines
31223237 __bpf_prog_put_noref(prog, prog->aux->real_func_cnt);
31233238 return err;
31243239
3125−free_prog_sec:
3126− security_bpf_prog_free(prog);
31273240 free_prog:
31283241 free_uid(prog->aux->user);
31293242 if (prog->aux->attach_btf)
@@ -3198,6 +3311,15 @@ void bpf_link_init(struct bpf_link *link, enum bpf_link_type type, · 68 unmodified lines
31983311 bpf_link_init_sleepable(link, type, ops, prog, attach_type, false);
31993312 }
32003313
3314+void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type,
3315+ const struct bpf_link_ops *ops, struct bpf_prog *prog,
3316+ enum bpf_attach_type attach_type, u64 cookie)
3317+{
3318+ bpf_link_init(&link->link, type, ops, prog, attach_type);
3319+ link->node.link = &link->link;
3320+ link->node.cookie = cookie;
3321+}
3322+
32013323 static void bpf_link_free_id(int id)
32023324 {
32033325 if (!id)
@@ -3358,7 +3480,7 @@ static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp) · 154 unmodified lines
33583480 seq_printf(m, "link_type:\t%s\n", link->flags == BPF_F_KPROBE_MULTI_RETURN ?
33593481 "kretprobe_multi" : "kprobe_multi");
33603482 else if (link->type == BPF_LINK_TYPE_UPROBE_MULTI)
3361− seq_printf(m, "link_type:\t%s\n", link->flags == BPF_F_UPROBE_MULTI_RETURN ?
3483+ seq_printf(m, "link_type:\t%s\n", link->flags & BPF_F_UPROBE_MULTI_RETURN ?
33623484 "uretprobe_multi" : "uprobe_multi");
33633485 else
33643486 seq_printf(m, "link_type:\t%s\n", bpf_link_type_strs[type]);
@@ -3505,7 +3627,7 @@ static void bpf_tracing_link_release(struct bpf_link *link) · 140 unmodified lines
35053627 struct bpf_tracing_link *tr_link =
35063628 container_of(link, struct bpf_tracing_link, link.link);
35073629
3508− WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link,
3630+ WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link.node,
35093631 tr_link->trampoline,
35103632 tr_link->tgt_prog));
35113633
@@ -3518,8 +3640,7 @@ static void bpf_tracing_link_release(struct bpf_link *link) · 6 unmodified lines
35183640
35193641 static void bpf_tracing_link_dealloc(struct bpf_link *link)
35203642 {
3521− struct bpf_tracing_link *tr_link =
3522− container_of(link, struct bpf_tracing_link, link.link);
3643+ struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link);
35233644
35243645 kfree(tr_link);
35253646 }
@@ -3527,8 +3648,8 @@ static void bpf_tracing_link_dealloc(struct bpf_link *link) · 1 unmodified line
35273648 static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link,
35283649 struct seq_file *seq)
35293650 {
3530− struct bpf_tracing_link *tr_link =
3531− container_of(link, struct bpf_tracing_link, link.link);
3651+ struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link);
3652+
35323653 u32 target_btf_id, target_obj_id;
35333654
35343655 bpf_trampoline_unpack_key(tr_link->trampoline->key,
@@ -3541,17 +3662,16 @@ static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link, · 6 unmodified lines
35413662 link->attach_type,
35423663 target_obj_id,
35433664 target_btf_id,
3544− tr_link->link.cookie);
3665+ tr_link->link.node.cookie);
35453666 }
35463667
35473668 static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,
35483669 struct bpf_link_info *info)
35493670 {
3550− struct bpf_tracing_link *tr_link =
3551− container_of(link, struct bpf_tracing_link, link.link);
3671+ struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link);
35523672
35533673 info->tracing.attach_type = link->attach_type;
3554− info->tracing.cookie = tr_link->link.cookie;
3674+ info->tracing.cookie = tr_link->link.node.cookie;
35553675 bpf_trampoline_unpack_key(tr_link->trampoline->key,
35563676 &info->tracing.target_obj_id,
35573677 &info->tracing.target_btf_id);
@@ -3633,29 +3753,18 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, · 75 unmodified lines
36333753 key = bpf_trampoline_compute_key(tgt_prog, NULL, btf_id);
36343754 }
36353755
3636− if (prog->expected_attach_type == BPF_TRACE_FSESSION) {
3637− struct bpf_fsession_link *fslink;
3638−
3639− fslink = kzalloc_obj(*fslink, GFP_USER);
3640− if (fslink) {
3641− bpf_link_init(&fslink->fexit.link, BPF_LINK_TYPE_TRACING,
3642− &bpf_tracing_link_lops, prog, attach_type);
3643− fslink->fexit.cookie = bpf_cookie;
3644− link = &fslink->link;
3645− } else {
3646− link = NULL;
3647− }
3648− } else {
3649− link = kzalloc_obj(*link, GFP_USER);
3650− }
3756+ link = kzalloc_obj(*link, GFP_USER);
36513757 if (!link) {
36523758 err = -ENOMEM;
36533759 goto out_put_prog;
36543760 }
3655− bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING,
3656− &bpf_tracing_link_lops, prog, attach_type);
3761+ bpf_tramp_link_init(&link->link, BPF_LINK_TYPE_TRACING,
3762+ &bpf_tracing_link_lops, prog, attach_type, bpf_cookie);
36573763
3658− link->link.cookie = bpf_cookie;
3764+ if (prog->expected_attach_type == BPF_TRACE_FSESSION) {
3765+ link->fexit.link = &link->link.link;
3766+ link->fexit.cookie = bpf_cookie;
3767+ }
36593768
36603769 mutex_lock(&prog->aux->dst_mutex);
36613770
@@ -3758,7 +3867,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, · 96 unmodified lines
37583867 if (err)
37593868 goto out_unlock;
37603869
3761− err = bpf_trampoline_link_prog(&link->link, tr, tgt_prog);
3870+ err = bpf_trampoline_link_prog(&link->link.node, tr, tgt_prog);
37623871 if (err) {
37633872 bpf_link_cleanup(&link_primer);
37643873 link = NULL;
@@ -4281,6 +4390,11 @@ static int bpf_raw_tp_link_attach(struct bpf_prog *prog, · 516 unmodified lines
42814390 if (!btp)
42824391 return -ENOENT;
42834392
4393+ if (prog->sleepable && !tracepoint_is_faultable(btp->tp)) {
4394+ bpf_put_raw_tracepoint(btp);
4395+ return -EINVAL;
4396+ }
4397+
42844398 link = kzalloc_obj(*link, GFP_USER);
42854399 if (!link) {
42864400 err = -ENOMEM;
@@ -4389,6 +4503,9 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) · 102 unmodified lines
43894503 case BPF_TRACE_FENTRY:
43904504 case BPF_TRACE_FEXIT:
43914505 case BPF_TRACE_FSESSION:
4506+ case BPF_TRACE_FSESSION_MULTI:
4507+ case BPF_TRACE_FENTRY_MULTI:
4508+ case BPF_TRACE_FEXIT_MULTI:
43924509 case BPF_MODIFY_RETURN:
43934510 return BPF_PROG_TYPE_TRACING;
43944511 case BPF_LSM_MAC:
@@ -4654,7 +4771,7 @@ static int bpf_prog_detach(const union bpf_attr *attr) · 259 unmodified lines
46544771 #define BPF_PROG_QUERY_LAST_FIELD query.revision
46554772
46564773 static int bpf_prog_query(const union bpf_attr *attr,
4657− union bpf_attr __user *uattr)
4774+ union bpf_attr __user *uattr, u32 uattr_size)
46584775 {
46594776 if (!bpf_net_capable())
46604777 return -EPERM;
@@ -4693,7 +4810,7 @@ static int bpf_prog_query(const union bpf_attr *attr, · 32 unmodified lines
46934810 case BPF_CGROUP_GETSOCKOPT:
46944811 case BPF_CGROUP_SETSOCKOPT:
46954812 case BPF_LSM_CGROUP:
4696− return cgroup_bpf_prog_query(attr, uattr);
4813+ return cgroup_bpf_prog_query(attr, uattr, uattr_size);
46974814 case BPF_LIRC_MODE2:
46984815 return lirc_prog_query(attr, uattr);
46994816 case BPF_FLOW_DISSECTOR:
@@ -5045,10 +5162,11 @@ static int bpf_prog_get_info_by_fd(struct file *file, · 345 unmodified lines
50455162 u32 info_len = attr->info.info_len;
50465163 struct bpf_prog_kstats stats;
50475164 char __user *uinsns;
5048− u32 ulen;
5165+ u32 ulen, len;
50495166 int err;
50505167
5051− err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len);
5168+ len = offsetofend(struct bpf_prog_info, attach_btf_id);
5169+ err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), len, info_len);
50525170 if (err)
50535171 return err;
50545172 info_len = min_t(u32, sizeof(info), info_len);
@@ -5330,10 +5448,11 @@ static int bpf_map_get_info_by_fd(struct file *file, · 275 unmodified lines
53305448 {
53315449 struct bpf_map_info __user *uinfo = u64_to_user_ptr(attr->info.info);
53325450 struct bpf_map_info info;
5333− u32 info_len = attr->info.info_len;
5451+ u32 info_len = attr->info.info_len, len;
53345452 int err;
53355453
5336− err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len);
5454+ len = offsetofend(struct bpf_map_info, hash_size);
5455+ err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), len, info_len);
53375456 if (err)
53385457 return err;
53395458 info_len = min_t(u32, sizeof(info), info_len);
@@ -5371,18 +5490,16 @@ static int bpf_map_get_info_by_fd(struct file *file, · 31 unmodified lines
53715490
53725491 if (!map->ops->map_get_hash)
53735492 return -EINVAL;
5374−
5375− if (info.hash_size != SHA256_DIGEST_SIZE)
5493+ if (info.hash_size != sizeof(map->sha))
53765494 return -EINVAL;
5377−
53785495 if (!READ_ONCE(map->frozen))
53795496 return -EPERM;
53805497
5381− err = map->ops->map_get_hash(map, SHA256_DIGEST_SIZE, map->sha);
5498+ err = map->ops->map_get_hash(map);
53825499 if (err != 0)
53835500 return err;
53845501
5385− if (copy_to_user(uhash, map->sha, SHA256_DIGEST_SIZE) != 0)
5502+ if (copy_to_user(uhash, map->sha, sizeof(map->sha)) != 0)
53865503 return -EFAULT;
53875504 } else if (info.hash_size) {
53885505 return -EINVAL;
@@ -5495,7 +5612,7 @@ static int bpf_obj_get_info_by_fd(const union bpf_attr *attr, · 106 unmodified lines
54955612
54965613 #define BPF_BTF_LOAD_LAST_FIELD btf_token_fd
54975614
5498−static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, __u32 uattr_size)
5615+static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
54995616 {
55005617 struct bpf_token *token = NULL;
55015618
@@ -5522,7 +5639,7 @@ static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, __u32 uattr_ · 20 unmodified lines
55225639
55235640 bpf_token_put(token);
55245641
5525− return btf_new_fd(attr, uattr, uattr_size);
5642+ return btf_new_fd(attr, uattr, attr_log);
55265643 }
55275644
55285645 #define BPF_BTF_GET_FD_BY_ID_LAST_FIELD fd_by_id_token_fd
@@ -5723,7 +5840,7 @@ err_put: · 194 unmodified lines
57235840 return err;
57245841 }
57255842
5726−#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.pid
5843+#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.path_fd
57275844 static int link_create(union bpf_attr *attr, bpfptr_t uattr)
57285845 {
57295846 struct bpf_prog *prog;
@@ -5774,6 +5891,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr) · 44 unmodified lines
57745891 ret = bpf_iter_link_attach(attr, uattr, prog);
57755892 else if (prog->expected_attach_type == BPF_LSM_CGROUP)
57765893 ret = cgroup_bpf_link_attach(attr, prog);
5894+ else if (is_tracing_multi(prog->expected_attach_type))
5895+ ret = bpf_tracing_multi_attach(prog, attr);
57775896 else
57785897 ret = bpf_tracing_prog_attach(prog,
57795898 attr->link_create.target_fd,
@@ -6232,8 +6351,12 @@ put_prog: · 452 unmodified lines
62326351 return ret;
62336352 }
62346353
6235−static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size)
6354+static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size,
6355+ bpfptr_t uattr_common, unsigned int size_common)
62366356 {
6357+ struct bpf_common_attr attr_common;
6358+ u32 offsetof_log_true_size = 0;
6359+ struct bpf_log_attr attr_log;
62376360 union bpf_attr attr;
62386361 int err;
62396362
@@ -6247,13 +6370,29 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 7 unmodified lines
62476370 if (copy_from_bpfptr(&attr, uattr, size) != 0)
62486371 return -EFAULT;
62496372
6373+ memset(&attr_common, 0, sizeof(attr_common));
6374+ if (cmd & BPF_COMMON_ATTRS) {
6375+ err = bpf_check_uarg_tail_zero(uattr_common,
6376+ offsetofend(struct bpf_common_attr, log_true_size),
6377+ size_common);
6378+ if (err)
6379+ return err;
6380+
6381+ cmd &= ~BPF_COMMON_ATTRS;
6382+ size_common = min_t(u32, size_common, sizeof(attr_common));
6383+ if (copy_from_bpfptr(&attr_common, uattr_common, size_common) != 0)
6384+ return -EFAULT;
6385+ } else {
6386+ size_common = 0;
6387+ }
6388+
62506389 err = security_bpf(cmd, &attr, size, uattr.is_kernel);
62516390 if (err < 0)
62526391 return err;
62536392
62546393 switch (cmd) {
62556394 case BPF_MAP_CREATE:
6256− err = map_create(&attr, uattr);
6395+ err = map_create(&attr, uattr, &attr_common, uattr_common, size_common);
62576396 break;
62586397 case BPF_MAP_LOOKUP_ELEM:
62596398 err = map_lookup_elem(&attr);
@@ -6271,7 +6410,12 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 11 unmodified lines
62716410 err = map_freeze(&attr);
62726411 break;
62736412 case BPF_PROG_LOAD:
6274− err = bpf_prog_load(&attr, uattr, size);
6413+ if (size >= offsetofend(union bpf_attr, log_true_size))
6414+ offsetof_log_true_size = offsetof(union bpf_attr, log_true_size);
6415+ err = bpf_log_attr_init(&attr_log, attr.log_buf, attr.log_size, attr.log_level,
6416+ offsetof_log_true_size, uattr, &attr_common, uattr_common,
6417+ size_common);
6418+ err = err ?: bpf_prog_load(&attr, uattr, &attr_log);
62756419 break;
62766420 case BPF_OBJ_PIN:
62776421 err = bpf_obj_pin(&attr);
@@ -6286,7 +6430,7 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 8 unmodified lines
62866430 err = bpf_prog_detach(&attr);
62876431 break;
62886432 case BPF_PROG_QUERY:
6289− err = bpf_prog_query(&attr, uattr.user);
6433+ err = bpf_prog_query(&attr, uattr.user, size);
62906434 break;
62916435 case BPF_PROG_TEST_RUN:
62926436 err = bpf_prog_test_run(&attr, uattr.user);
@@ -6316,7 +6460,12 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 23 unmodified lines
63166460 err = bpf_raw_tracepoint_open(&attr);
63176461 break;
63186462 case BPF_BTF_LOAD:
6319− err = bpf_btf_load(&attr, uattr, size);
6463+ if (size >= offsetofend(union bpf_attr, btf_log_true_size))
6464+ offsetof_log_true_size = offsetof(union bpf_attr, btf_log_true_size);
6465+ err = bpf_log_attr_init(&attr_log, attr.btf_log_buf, attr.btf_log_size,
6466+ attr.btf_log_level, offsetof_log_true_size, uattr,
6467+ &attr_common, uattr_common, size_common);
6468+ err = err ?: bpf_btf_load(&attr, uattr, &attr_log);
63206469 break;
63216470 case BPF_BTF_GET_FD_BY_ID:
63226471 err = bpf_btf_get_fd_by_id(&attr);
@@ -6382,9 +6531,10 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 59 unmodified lines
63826531 return err;
63836532 }
63846533
6385−SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
6534+SYSCALL_DEFINE5(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size,
6535+ struct bpf_common_attr __user *, uattr_common, unsigned int, size_common)
63866536 {
6387− return __sys_bpf(cmd, USER_BPFPTR(uattr), size);
6537+ return __sys_bpf(cmd, USER_BPFPTR(uattr), size, USER_BPFPTR(uattr_common), size_common);
63886538 }
63896539
63906540 static bool syscall_prog_is_valid_access(int off, int size,
@@ -6414,7 +6564,7 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size) · 23 unmodified lines
64146564 default:
64156565 return -EINVAL;
64166566 }
6417− return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size);
6567+ return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0);
64186568 }
64196569
64206570
kernel/bpf/trampoline.c+524−147
@@ -30,8 +30,46 @@ static struct hlist_head trampoline_ip_table[TRAMPOLINE_TABLE_SIZE]; · 29 unmodified lines
3030 /* serializes access to trampoline tables */
3131 static DEFINE_MUTEX(trampoline_mutex);
3232
33+/*
34+ * Keep 32 trampoline locks (5 bits) in the pool so trampoline_lock_all()
35+ * stays below MAX_LOCK_DEPTH. Each pool slot has a distinct lockdep
36+ * class because trampoline_lock_all() takes all pool mutexes at once;
37+ * otherwise lockdep would report recursive locking on same-class mutexes.
38+ */
39+#define TRAMPOLINE_LOCKS_BITS 5
40+#define TRAMPOLINE_LOCKS_TABLE_SIZE (1 << TRAMPOLINE_LOCKS_BITS)
41+
42+static struct {
43+ struct mutex mutex;
44+ struct lock_class_key key;
45+} trampoline_locks[TRAMPOLINE_LOCKS_TABLE_SIZE];
46+
47+static struct mutex *select_trampoline_lock(struct bpf_trampoline *tr)
48+{
49+ return &trampoline_locks[hash_ptr(tr, TRAMPOLINE_LOCKS_BITS)].mutex;
50+}
51+
52+static void trampoline_lock(struct bpf_trampoline *tr)
53+{
54+ mutex_lock(select_trampoline_lock(tr));
55+}
56+
57+static void trampoline_unlock(struct bpf_trampoline *tr)
58+{
59+ mutex_unlock(select_trampoline_lock(tr));
60+}
61+
62+struct bpf_trampoline_ops {
63+ int (*register_fentry)(struct bpf_trampoline *tr, struct bpf_tramp_image *im, void *data);
64+ int (*unregister_fentry)(struct bpf_trampoline *tr, u32 orig_flags, void *data);
65+ int (*modify_fentry)(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im,
66+ bool lock_direct_mutex, void *data);
67+};
68+
3369 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
34−static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex);
70+static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex,
71+ const struct bpf_trampoline_ops *ops, void *data);
72+static const struct bpf_trampoline_ops trampoline_ops;
3573
3674 #ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
3775 static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops, unsigned long ip)
@@ -69,9 +107,9 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 31 unmodified lines
69107
70108 if (cmd == FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) {
71109 /* This is called inside register_ftrace_direct_multi(), so
72− * tr->mutex is already locked.
110+ * trampoline's mutex is already locked.
73111 */
74− lockdep_assert_held_once(&tr->mutex);
112+ lockdep_assert_held_once(select_trampoline_lock(tr));
75113
76114 /* Instead of updating the trampoline here, we propagate
77115 * -EAGAIN to register_ftrace_direct(). Then we can
@@ -91,7 +129,7 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 13 unmodified lines
91129 }
92130
93131 /* The normal locking order is
94− * tr->mutex => direct_mutex (ftrace.c) => ftrace_lock (ftrace.c)
132+ * select_trampoline_lock(tr) => direct_mutex (ftrace.c) => ftrace_lock (ftrace.c)
95133 *
96134 * The following two commands are called from
97135 *
@@ -99,12 +137,12 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 1 unmodified line
99137 * cleanup_direct_functions_after_ipmodify
100138 *
101139 * In both cases, direct_mutex is already locked. Use
102− * mutex_trylock(&tr->mutex) to avoid deadlock in race condition
103− * (something else is making changes to this same trampoline).
140+ * mutex_trylock(select_trampoline_lock(tr)) to avoid deadlock in race condition
141+ * (something else holds the same pool lock).
104142 */
105− if (!mutex_trylock(&tr->mutex)) {
106− /* sleep 1 ms to make sure whatever holding tr->mutex makes
107− * some progress.
143+ if (!mutex_trylock(select_trampoline_lock(tr))) {
144+ /* sleep 1 ms to make sure whatever holding select_trampoline_lock(tr)
145+ * makes some progress.
108146 */
109147 msleep(1);
110148 return -EAGAIN;
@@ -116,20 +154,22 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 5 unmodified lines
116154
117155 if ((tr->flags & BPF_TRAMP_F_CALL_ORIG) &&
118156 !(tr->flags & BPF_TRAMP_F_ORIG_STACK))
119− ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */);
157+ ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */,
158+ &trampoline_ops, NULL);
120159 break;
121160 case FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER:
122161 tr->flags &= ~BPF_TRAMP_F_SHARE_IPMODIFY;
123162
124163 if (tr->flags & BPF_TRAMP_F_ORIG_STACK)
125− ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */);
164+ ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */,
165+ &trampoline_ops, NULL);
126166 break;
127167 default:
128168 ret = -EINVAL;
129169 break;
130170 }
131171
132− mutex_unlock(&tr->mutex);
172+ trampoline_unlock(tr);
133173 return ret;
134174 }
135175 #endif
@@ -142,7 +182,9 @@ bool bpf_prog_has_trampoline(const struct bpf_prog *prog) · 6 unmodified lines
142182 switch (ptype) {
143183 case BPF_PROG_TYPE_TRACING:
144184 if (eatype == BPF_TRACE_FENTRY || eatype == BPF_TRACE_FEXIT ||
145− eatype == BPF_MODIFY_RETURN || eatype == BPF_TRACE_FSESSION)
185+ eatype == BPF_MODIFY_RETURN || eatype == BPF_TRACE_FSESSION ||
186+ eatype == BPF_TRACE_FENTRY_MULTI || eatype == BPF_TRACE_FEXIT_MULTI ||
187+ eatype == BPF_TRACE_FSESSION_MULTI)
146188 return true;
147189 return false;
148190 case BPF_PROG_TYPE_LSM:
@@ -359,7 +401,6 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip) · 210 unmodified lines
359401 head = &trampoline_ip_table[hash_64(tr->ip, TRAMPOLINE_HASH_BITS)];
360402 hlist_add_head(&tr->hlist_ip, head);
361403 refcount_set(&tr->refcnt, 1);
362− mutex_init(&tr->mutex);
363404 for (i = 0; i < BPF_TRAMP_MAX; i++)
364405 INIT_HLIST_HEAD(&tr->progs_hlist[i]);
365406 out:
@@ -386,9 +427,11 @@ static int bpf_trampoline_update_fentry(struct bpf_trampoline *tr, u32 orig_flag · 20 unmodified lines
386427 return bpf_arch_text_poke(ip, old_t, new_t, old_addr, new_addr);
387428 }
388429
389−static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags,
390− void *old_addr)
430+static void bpf_tramp_image_put(struct bpf_tramp_image *im);
431+
432+static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, void *data __maybe_unused)
391433 {
434+ void *old_addr = tr->cur_image->image;
392435 int ret;
393436
394437 if (tr->func.ftrace_managed)
@@ -396,13 +439,19 @@ static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, · 1 unmodified line
396439 else
397440 ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr, NULL);
398441
399− return ret;
442+ if (ret)
443+ return ret;
444+
445+ bpf_tramp_image_put(tr->cur_image);
446+ tr->cur_image = NULL;
447+ return 0;
400448 }
401449
402−static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags,
403− void *old_addr, void *new_addr,
404− bool lock_direct_mutex)
450+static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im,
451+ bool lock_direct_mutex, void *data __maybe_unused)
405452 {
453+ void *old_addr = tr->cur_image->image;
454+ void *new_addr = im->image;
406455 int ret;
407456
408457 if (tr->func.ftrace_managed) {
@@ -411,12 +460,20 @@ static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, · 2 unmodified lines
411460 ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr,
412461 new_addr);
413462 }
414− return ret;
463+
464+ if (ret)
465+ return ret;
466+
467+ bpf_tramp_image_put(tr->cur_image);
468+ tr->cur_image = im;
469+ return 0;
415470 }
416471
417472 /* first time registering */
418−static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
473+static int register_fentry(struct bpf_trampoline *tr, struct bpf_tramp_image *im,
474+ void *data __maybe_unused)
419475 {
476+ void *new_addr = im->image;
420477 void *ip = tr->func.addr;
421478 unsigned long faddr;
422479 int ret;
@@ -434,33 +491,42 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) · 11 unmodified lines
434491 ret = bpf_trampoline_update_fentry(tr, 0, NULL, new_addr);
435492 }
436493
437− return ret;
494+ if (ret)
495+ return ret;
496+
497+ tr->cur_image = im;
498+ return 0;
438499 }
439500
440−static struct bpf_tramp_links *
501+static const struct bpf_trampoline_ops trampoline_ops = {
502+ .register_fentry = register_fentry,
503+ .unregister_fentry = unregister_fentry,
504+ .modify_fentry = modify_fentry,
505+};
506+
507+static struct bpf_tramp_nodes *
441508 bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg)
442509 {
443− struct bpf_tramp_link *link;
444− struct bpf_tramp_links *tlinks;
445− struct bpf_tramp_link **links;
510+ struct bpf_tramp_node *node, **nodes;
511+ struct bpf_tramp_nodes *tnodes;
446512 int kind;
447513
448514 *total = 0;
449− tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX);
450− if (!tlinks)
515+ tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX);
516+ if (!tnodes)
451517 return ERR_PTR(-ENOMEM);
452518
453519 for (kind = 0; kind < BPF_TRAMP_MAX; kind++) {
454− tlinks[kind].nr_links = tr->progs_cnt[kind];
520+ tnodes[kind].nr_nodes = tr->progs_cnt[kind];
455521 *total += tr->progs_cnt[kind];
456− links = tlinks[kind].links;
522+ nodes = tnodes[kind].nodes;
457523
458− hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) {
459− *ip_arg |= link->link.prog->call_get_func_ip;
460− *links++ = link;
524+ hlist_for_each_entry(node, &tr->progs_hlist[kind], tramp_hlist) {
525+ *ip_arg |= node->link->prog->call_get_func_ip;
526+ *nodes++ = node;
461527 }
462528 }
463− return tlinks;
529+ return tnodes;
464530 }
465531
466532 static void bpf_tramp_image_free(struct bpf_tramp_image *im)
@@ -604,30 +670,29 @@ out: · 137 unmodified lines
604670 return ERR_PTR(err);
605671 }
606672
607−static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex)
673+static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex,
674+ const struct bpf_trampoline_ops *ops, void *data)
608675 {
609676 struct bpf_tramp_image *im;
610− struct bpf_tramp_links *tlinks;
677+ struct bpf_tramp_nodes *tnodes;
611678 u32 orig_flags = tr->flags;
612679 bool ip_arg = false;
613680 int err, total, size;
614681
615− tlinks = bpf_trampoline_get_progs(tr, &total, &ip_arg);
616− if (IS_ERR(tlinks))
617− return PTR_ERR(tlinks);
682+ tnodes = bpf_trampoline_get_progs(tr, &total, &ip_arg);
683+ if (IS_ERR(tnodes))
684+ return PTR_ERR(tnodes);
618685
619686 if (total == 0) {
620− err = unregister_fentry(tr, orig_flags, tr->cur_image->image);
621− bpf_tramp_image_put(tr->cur_image);
622− tr->cur_image = NULL;
687+ err = ops->unregister_fentry(tr, orig_flags, data);
623688 goto out;
624689 }
625690
626691 /* clear all bits except SHARE_IPMODIFY and TAIL_CALL_CTX */
627692 tr->flags &= (BPF_TRAMP_F_SHARE_IPMODIFY | BPF_TRAMP_F_TAIL_CALL_CTX);
628693
629− if (tlinks[BPF_TRAMP_FEXIT].nr_links ||
630− tlinks[BPF_TRAMP_MODIFY_RETURN].nr_links) {
694+ if (tnodes[BPF_TRAMP_FEXIT].nr_nodes ||
695+ tnodes[BPF_TRAMP_MODIFY_RETURN].nr_nodes) {
631696 /* NOTE: BPF_TRAMP_F_RESTORE_REGS and BPF_TRAMP_F_SKIP_FRAME
632697 * should not be set together.
633698 */
@@ -658,7 +723,7 @@ again: · 24 unmodified lines
658723 #endif
659724
660725 size = arch_bpf_trampoline_size(&tr->func.model, tr->flags,
661− tlinks, tr->func.addr);
726+ tnodes, tr->func.addr);
662727 if (size < 0) {
663728 err = size;
664729 goto out;
@@ -676,7 +741,7 @@ again: · 11 unmodified lines
676741 }
677742
678743 err = arch_prepare_bpf_trampoline(im, im->image, im->image + size,
679− &tr->func.model, tr->flags, tlinks,
744+ &tr->func.model, tr->flags, tnodes,
680745 tr->func.addr);
681746 if (err < 0)
682747 goto out_free;
@@ -685,14 +750,12 @@ again: · 2 unmodified lines
685750 if (err)
686751 goto out_free;
687752
688− WARN_ON(tr->cur_image && total == 0);
689753 if (tr->cur_image)
690754 /* progs already running at this address */
691− err = modify_fentry(tr, orig_flags, tr->cur_image->image,
692− im->image, lock_direct_mutex);
755+ err = ops->modify_fentry(tr, orig_flags, im, lock_direct_mutex, data);
693756 else
694757 /* first time registering */
695− err = register_fentry(tr, im->image);
758+ err = ops->register_fentry(tr, im, data);
696759
697760 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
698761 if (err == -EAGAIN) {
@@ -704,34 +767,31 @@ again: · 5 unmodified lines
704767 goto again;
705768 }
706769 #endif
707− if (err)
708− goto out_free;
709770
710− if (tr->cur_image)
711− bpf_tramp_image_put(tr->cur_image);
712− tr->cur_image = im;
771+out_free:
772+ if (err)
773+ bpf_tramp_image_free(im);
713774 out:
714775 /* If any error happens, restore previous flags */
715776 if (err)
716777 tr->flags = orig_flags;
717− kfree(tlinks);
778+ kfree(tnodes);
718779 return err;
719−
720−out_free:
721− bpf_tramp_image_free(im);
722− goto out;
723780 }
724781
725782 static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog)
726783 {
727784 switch (prog->expected_attach_type) {
728785 case BPF_TRACE_FENTRY:
786+ case BPF_TRACE_FENTRY_MULTI:
729787 return BPF_TRAMP_FENTRY;
730788 case BPF_MODIFY_RETURN:
731789 return BPF_TRAMP_MODIFY_RETURN;
732790 case BPF_TRACE_FEXIT:
791+ case BPF_TRACE_FEXIT_MULTI:
733792 return BPF_TRAMP_FEXIT;
734793 case BPF_TRACE_FSESSION:
794+ case BPF_TRACE_FSESSION_MULTI:
735795 return BPF_TRAMP_FSESSION;
736796 case BPF_LSM_MAC:
737797 if (!prog->aux->attach_func_proto->type)
@@ -764,39 +824,33 @@ static int bpf_freplace_check_tgt_prog(struct bpf_prog *tgt_prog) · 26 unmodified lines
764824 return 0;
765825 }
766826
767−static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link,
768− struct bpf_trampoline *tr,
769− struct bpf_prog *tgt_prog)
827+static struct bpf_tramp_node *fsession_exit(struct bpf_tramp_node *node)
770828 {
771− struct bpf_fsession_link *fslink = NULL;
772− enum bpf_tramp_prog_type kind;
773− struct bpf_tramp_link *link_exiting;
774− struct hlist_head *prog_list;
775− int err = 0;
776− int cnt = 0, i;
829+ if (node->link->type == BPF_LINK_TYPE_TRACING) {
830+ struct bpf_tracing_link *link;
777831
778− kind = bpf_attach_type_to_tramp(link->link.prog);
779− if (tr->extension_prog)
780− /* cannot attach fentry/fexit if extension prog is attached.
781− * cannot overwrite extension prog either.
782− */
783− return -EBUSY;
832+ link = container_of(node->link, struct bpf_tracing_link, link.link);
833+ return &link->fexit;
834+ } else if (node->link->type == BPF_LINK_TYPE_TRACING_MULTI) {
835+ struct bpf_tracing_multi_link *link;
836+ struct bpf_tracing_multi_node *mnode;
784837
785− for (i = 0; i < BPF_TRAMP_MAX; i++)
786− cnt += tr->progs_cnt[i];
787−
788− if (kind == BPF_TRAMP_REPLACE) {
789− /* Cannot attach extension if fentry/fexit are in use. */
790− if (cnt)
791− return -EBUSY;
792− err = bpf_freplace_check_tgt_prog(tgt_prog);
793− if (err)
794− return err;
795− tr->extension_prog = link->link.prog;
796− return bpf_arch_text_poke(tr->func.addr, BPF_MOD_NOP,
797− BPF_MOD_JUMP, NULL,
798− link->link.prog->bpf_func);
838+ link = container_of(node->link, struct bpf_tracing_multi_link, link);
839+ mnode = container_of(node, struct bpf_tracing_multi_node, node);
840+ return &link->fexits[mnode - link->nodes];
799841 }
842+ return NULL;
843+}
844+
845+static int bpf_trampoline_add_prog(struct bpf_trampoline *tr,
846+ struct bpf_tramp_node *node,
847+ int cnt)
848+{
849+ enum bpf_tramp_prog_type kind;
850+ struct bpf_tramp_node *node_existing, *fexit;
851+ struct hlist_head *prog_list;
852+
853+ kind = bpf_attach_type_to_tramp(node->link->prog);
800854 if (kind == BPF_TRAMP_FSESSION) {
801855 prog_list = &tr->progs_hlist[BPF_TRAMP_FENTRY];
802856 cnt++;
@@ -805,59 +859,112 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, · 2 unmodified lines
805859 }
806860 if (cnt >= BPF_MAX_TRAMP_LINKS)
807861 return -E2BIG;
808− if (!hlist_unhashed(&link->tramp_hlist))
862+ if (!hlist_unhashed(&node->tramp_hlist))
809863 /* prog already linked */
810864 return -EBUSY;
811− hlist_for_each_entry(link_exiting, prog_list, tramp_hlist) {
812− if (link_exiting->link.prog != link->link.prog)
865+ hlist_for_each_entry(node_existing, prog_list, tramp_hlist) {
866+ if (node_existing->link->prog != node->link->prog)
813867 continue;
814868 /* prog already linked */
815869 return -EBUSY;
816870 }
817871
818− hlist_add_head(&link->tramp_hlist, prog_list);
872+ hlist_add_head(&node->tramp_hlist, prog_list);
819873 if (kind == BPF_TRAMP_FSESSION) {
820874 tr->progs_cnt[BPF_TRAMP_FENTRY]++;
821− fslink = container_of(link, struct bpf_fsession_link, link.link);
822− hlist_add_head(&fslink->fexit.tramp_hlist, &tr->progs_hlist[BPF_TRAMP_FEXIT]);
875+ fexit = fsession_exit(node);
876+ if (WARN_ON_ONCE(!fexit))
877+ return -EINVAL;
878+ hlist_add_head(&fexit->tramp_hlist, &tr->progs_hlist[BPF_TRAMP_FEXIT]);
823879 tr->progs_cnt[BPF_TRAMP_FEXIT]++;
824880 } else {
825881 tr->progs_cnt[kind]++;
826882 }
827− err = bpf_trampoline_update(tr, true /* lock_direct_mutex */);
828− if (err) {
829− hlist_del_init(&link->tramp_hlist);
830− if (kind == BPF_TRAMP_FSESSION) {
831− tr->progs_cnt[BPF_TRAMP_FENTRY]--;
832− hlist_del_init(&fslink->fexit.tramp_hlist);
833− tr->progs_cnt[BPF_TRAMP_FEXIT]--;
834− } else {
835− tr->progs_cnt[kind]--;
836− }
883+ return 0;
884+}
885+
886+static void bpf_trampoline_remove_prog(struct bpf_trampoline *tr,
887+ struct bpf_tramp_node *node)
888+{
889+ enum bpf_tramp_prog_type kind;
890+ struct bpf_tramp_node *fexit;
891+
892+ kind = bpf_attach_type_to_tramp(node->link->prog);
893+ if (kind == BPF_TRAMP_FSESSION) {
894+ fexit = fsession_exit(node);
895+ if (WARN_ON_ONCE(!fexit))
896+ return;
897+ hlist_del_init(&fexit->tramp_hlist);
898+ tr->progs_cnt[BPF_TRAMP_FEXIT]--;
899+ kind = BPF_TRAMP_FENTRY;
837900 }
901+ hlist_del_init(&node->tramp_hlist);
902+ tr->progs_cnt[kind]--;
903+}
904+
905+static int __bpf_trampoline_link_prog(struct bpf_tramp_node *node,
906+ struct bpf_trampoline *tr,
907+ struct bpf_prog *tgt_prog,
908+ const struct bpf_trampoline_ops *ops,
909+ void *data)
910+{
911+ enum bpf_tramp_prog_type kind;
912+ int err = 0;
913+ int cnt = 0, i;
914+
915+ kind = bpf_attach_type_to_tramp(node->link->prog);
916+ if (tr->extension_prog)
917+ /* cannot attach fentry/fexit if extension prog is attached.
918+ * cannot overwrite extension prog either.
919+ */
920+ return -EBUSY;
921+
922+ for (i = 0; i < BPF_TRAMP_MAX; i++)
923+ cnt += tr->progs_cnt[i];
924+
925+ if (kind == BPF_TRAMP_REPLACE) {
926+ /* Cannot attach extension if fentry/fexit are in use. */
927+ if (cnt)
928+ return -EBUSY;
929+ err = bpf_freplace_check_tgt_prog(tgt_prog);
930+ if (err)
931+ return err;
932+ tr->extension_prog = node->link->prog;
933+ return bpf_arch_text_poke(tr->func.addr, BPF_MOD_NOP,
934+ BPF_MOD_JUMP, NULL,
935+ node->link->prog->bpf_func);
936+ }
937+ err = bpf_trampoline_add_prog(tr, node, cnt);
938+ if (err)
939+ return err;
940+ err = bpf_trampoline_update(tr, true /* lock_direct_mutex */, ops, data);
941+ if (err)
942+ bpf_trampoline_remove_prog(tr, node);
838943 return err;
839944 }
840945
841−int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
946+int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
842947 struct bpf_trampoline *tr,
843948 struct bpf_prog *tgt_prog)
844949 {
845950 int err;
846951
847− mutex_lock(&tr->mutex);
848− err = __bpf_trampoline_link_prog(link, tr, tgt_prog);
849− mutex_unlock(&tr->mutex);
952+ trampoline_lock(tr);
953+ err = __bpf_trampoline_link_prog(node, tr, tgt_prog, &trampoline_ops, NULL);
954+ trampoline_unlock(tr);
850955 return err;
851956 }
852957
853−static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
958+static int __bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
854959 struct bpf_trampoline *tr,
855− struct bpf_prog *tgt_prog)
960+ struct bpf_prog *tgt_prog,
961+ const struct bpf_trampoline_ops *ops,
962+ void *data)
856963 {
857964 enum bpf_tramp_prog_type kind;
858965 int err;
859966
860− kind = bpf_attach_type_to_tramp(link->link.prog);
967+ kind = bpf_attach_type_to_tramp(node->link->prog);
861968 if (kind == BPF_TRAMP_REPLACE) {
862969 WARN_ON_ONCE(!tr->extension_prog);
863970 err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP,
@@ -867,29 +974,21 @@ static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, · 3 unmodified lines
867974 guard(mutex)(&tgt_prog->aux->ext_mutex);
868975 tgt_prog->aux->is_extended = false;
869976 return err;
870− } else if (kind == BPF_TRAMP_FSESSION) {
871− struct bpf_fsession_link *fslink =
872− container_of(link, struct bpf_fsession_link, link.link);
873−
874− hlist_del_init(&fslink->fexit.tramp_hlist);
875− tr->progs_cnt[BPF_TRAMP_FEXIT]--;
876− kind = BPF_TRAMP_FENTRY;
877977 }
878− hlist_del_init(&link->tramp_hlist);
879− tr->progs_cnt[kind]--;
880− return bpf_trampoline_update(tr, true /* lock_direct_mutex */);
978+ bpf_trampoline_remove_prog(tr, node);
979+ return bpf_trampoline_update(tr, true /* lock_direct_mutex */, ops, data);
881980 }
882981
883982 /* bpf_trampoline_unlink_prog() should never fail. */
884−int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
983+int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
885984 struct bpf_trampoline *tr,
886985 struct bpf_prog *tgt_prog)
887986 {
888987 int err;
889988
890− mutex_lock(&tr->mutex);
891− err = __bpf_trampoline_unlink_prog(link, tr, tgt_prog);
892− mutex_unlock(&tr->mutex);
989+ trampoline_lock(tr);
990+ err = __bpf_trampoline_unlink_prog(node, tr, tgt_prog, &trampoline_ops, NULL);
991+ trampoline_unlock(tr);
893992 return err;
894993 }
895994
@@ -903,7 +1002,7 @@ static void bpf_shim_tramp_link_release(struct bpf_link *link) · 7 unmodified lines
9031002 if (!shim_link->trampoline)
9041003 return;
9051004
906− WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link, shim_link->trampoline, NULL));
1005+ WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link.node, shim_link->trampoline, NULL));
9071006 bpf_trampoline_put(shim_link->trampoline);
9081007 }
9091008
@@ -949,8 +1048,8 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog · 39 unmodified lines
9491048 p->type = BPF_PROG_TYPE_LSM;
9501049 p->expected_attach_type = BPF_LSM_MAC;
9511050 bpf_prog_inc(p);
952− bpf_link_init(&shim_link->link.link, BPF_LINK_TYPE_UNSPEC,
953− &bpf_shim_tramp_link_lops, p, attach_type);
1051+ bpf_tramp_link_init(&shim_link->link, BPF_LINK_TYPE_UNSPEC,
1052+ &bpf_shim_tramp_link_lops, p, attach_type, 0);
9541053 bpf_cgroup_atype_get(p->aux->attach_btf_id, cgroup_atype);
9551054
9561055 return shim_link;
@@ -959,15 +1058,15 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog · 2 unmodified lines
9591058 static struct bpf_shim_tramp_link *cgroup_shim_find(struct bpf_trampoline *tr,
9601059 bpf_func_t bpf_func)
9611060 {
962− struct bpf_tramp_link *link;
1061+ struct bpf_tramp_node *node;
9631062 int kind;
9641063
9651064 for (kind = 0; kind < BPF_TRAMP_MAX; kind++) {
966− hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) {
967− struct bpf_prog *p = link->link.prog;
1065+ hlist_for_each_entry(node, &tr->progs_hlist[kind], tramp_hlist) {
1066+ struct bpf_prog *p = node->link->prog;
9681067
9691068 if (p->bpf_func == bpf_func)
970− return container_of(link, struct bpf_shim_tramp_link, link);
1069+ return container_of(node, struct bpf_shim_tramp_link, link.node);
9711070 }
9721071 }
9731072
@@ -999,12 +1098,12 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog, · 25 unmodified lines
9991098 if (!tr)
10001099 return -ENOMEM;
10011100
1002− mutex_lock(&tr->mutex);
1101+ trampoline_lock(tr);
10031102
10041103 shim_link = cgroup_shim_find(tr, bpf_func);
10051104 if (shim_link && !IS_ERR(bpf_link_inc_not_zero(&shim_link->link.link))) {
10061105 /* Reusing existing shim attached by the other program. */
1007− mutex_unlock(&tr->mutex);
1106+ trampoline_unlock(tr);
10081107 bpf_trampoline_put(tr); /* bpf_trampoline_get above */
10091108 return 0;
10101109 }
@@ -1017,23 +1116,23 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog, · 6 unmodified lines
10171116 goto err;
10181117 }
10191118
1020− err = __bpf_trampoline_link_prog(&shim_link->link, tr, NULL);
1119+ err = __bpf_trampoline_link_prog(&shim_link->link.node, tr, NULL, &trampoline_ops, NULL);
10211120 if (err)
10221121 goto err;
10231122
10241123 shim_link->trampoline = tr;
10251124 /* note, we're still holding tr refcnt from above */
10261125
1027− mutex_unlock(&tr->mutex);
1126+ trampoline_unlock(tr);
10281127
10291128 return 0;
10301129 err:
1031− mutex_unlock(&tr->mutex);
1130+ trampoline_unlock(tr);
10321131
10331132 if (shim_link)
10341133 bpf_link_put(&shim_link->link.link);
10351134
1036− /* have to release tr while _not_ holding its mutex */
1135+ /* have to release tr while _not_ holding pool mutex for trampoline */
10371136 bpf_trampoline_put(tr); /* bpf_trampoline_get above */
10381137
10391138 return err;
@@ -1054,9 +1153,9 @@ void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog) · 14 unmodified lines
10541153 if (WARN_ON_ONCE(!tr))
10551154 return;
10561155
1057− mutex_lock(&tr->mutex);
1156+ trampoline_lock(tr);
10581157 shim_link = cgroup_shim_find(tr, bpf_func);
1059− mutex_unlock(&tr->mutex);
1158+ trampoline_unlock(tr);
10601159
10611160 if (shim_link)
10621161 bpf_link_put(&shim_link->link.link);
@@ -1074,14 +1173,14 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, · 11 unmodified lines
10741173 if (!tr)
10751174 return NULL;
10761175
1077− mutex_lock(&tr->mutex);
1176+ trampoline_lock(tr);
10781177 if (tr->func.addr)
10791178 goto out;
10801179
10811180 memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel));
10821181 tr->func.addr = (void *)tgt_info->tgt_addr;
10831182 out:
1084− mutex_unlock(&tr->mutex);
1183+ trampoline_unlock(tr);
10851184 return tr;
10861185 }
10871186
@@ -1094,7 +1193,6 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) · 6 unmodified lines
10941193 mutex_lock(&trampoline_mutex);
10951194 if (!refcount_dec_and_test(&tr->refcnt))
10961195 goto out;
1097− WARN_ON_ONCE(mutex_is_locked(&tr->mutex));
10981196
10991197 for (i = 0; i < BPF_TRAMP_MAX; i++)
11001198 if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[i])))
@@ -1333,7 +1431,7 @@ bpf_trampoline_exit_t bpf_trampoline_exit(const struct bpf_prog *prog) · 232 unmodified lines
13331431 int __weak
13341432 arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
13351433 const struct btf_func_model *m, u32 flags,
1336− struct bpf_tramp_links *tlinks,
1434+ struct bpf_tramp_nodes *tnodes,
13371435 void *func_addr)
13381436 {
13391437 return -ENOTSUPP;
@@ -1367,11 +1465,288 @@ int __weak arch_protect_bpf_trampoline(void *image, unsigned int size) · 27 unmodified lines
13671465 }
13681466
13691467 int __weak arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
1370− struct bpf_tramp_links *tlinks, void *func_addr)
1468+ struct bpf_tramp_nodes *tnodes, void *func_addr)
13711469 {
13721470 return -ENOTSUPP;
13731471 }
13741472
1473+#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \
1474+ defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS) && \
1475+ defined(CONFIG_BPF_SYSCALL)
1476+
1477+static void trampoline_lock_all(void)
1478+{
1479+ int i;
1480+
1481+ for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++)
1482+ mutex_lock(&trampoline_locks[i].mutex);
1483+}
1484+
1485+static void trampoline_unlock_all(void)
1486+{
1487+ int i;
1488+
1489+ for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++)
1490+ mutex_unlock(&trampoline_locks[i].mutex);
1491+}
1492+
1493+static void remove_tracing_multi_data(struct bpf_tracing_multi_data *data)
1494+{
1495+ ftrace_hash_remove(data->reg);
1496+ ftrace_hash_remove(data->unreg);
1497+ ftrace_hash_remove(data->modify);
1498+}
1499+
1500+static void clear_tracing_multi_data(struct bpf_tracing_multi_data *data)
1501+{
1502+ remove_tracing_multi_data(data);
1503+
1504+ free_ftrace_hash(data->reg);
1505+ free_ftrace_hash(data->unreg);
1506+ free_ftrace_hash(data->modify);
1507+}
1508+
1509+static int init_tracing_multi_data(struct bpf_tracing_multi_data *data)
1510+{
1511+ data->reg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
1512+ data->unreg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
1513+ data->modify = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
1514+
1515+ if (!data->reg || !data->unreg || !data->modify) {
1516+ clear_tracing_multi_data(data);
1517+ return -ENOMEM;
1518+ }
1519+ return 0;
1520+}
1521+
1522+static void ftrace_hash_add(struct ftrace_hash *hash, struct ftrace_func_entry *entry,
1523+ unsigned long ip, unsigned long direct)
1524+{
1525+ entry->ip = ip;
1526+ entry->direct = direct;
1527+ add_ftrace_hash_entry(hash, entry);
1528+}
1529+
1530+static int register_fentry_multi(struct bpf_trampoline *tr, struct bpf_tramp_image *im, void *ptr)
1531+{
1532+ unsigned long addr = (unsigned long) im->image;
1533+ unsigned long ip = ftrace_location(tr->ip);
1534+ struct bpf_tracing_multi_data *data = ptr;
1535+
1536+ if (bpf_trampoline_use_jmp(tr->flags))
1537+ addr = ftrace_jmp_set(addr);
1538+
1539+ ftrace_hash_add(data->reg, data->entry, ip, addr);
1540+ tr->cur_image = im;
1541+ return 0;
1542+}
1543+
1544+static int unregister_fentry_multi(struct bpf_trampoline *tr, u32 orig_flags, void *ptr)
1545+{
1546+ unsigned long addr = (unsigned long) tr->cur_image->image;
1547+ unsigned long ip = ftrace_location(tr->ip);
1548+ struct bpf_tracing_multi_data *data = ptr;
1549+
1550+ if (bpf_trampoline_use_jmp(tr->flags))
1551+ addr = ftrace_jmp_set(addr);
1552+
1553+ ftrace_hash_add(data->unreg, data->entry, ip, addr);
1554+ tr->cur_image = NULL;
1555+ return 0;
1556+}
1557+
1558+static int modify_fentry_multi(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im,
1559+ bool lock_direct_mutex, void *ptr)
1560+{
1561+ unsigned long addr = (unsigned long) im->image;
1562+ unsigned long ip = ftrace_location(tr->ip);
1563+ struct bpf_tracing_multi_data *data = ptr;
1564+
1565+ if (bpf_trampoline_use_jmp(tr->flags))
1566+ addr = ftrace_jmp_set(addr);
1567+
1568+ ftrace_hash_add(data->modify, data->entry, ip, addr);
1569+ tr->cur_image = im;
1570+ return 0;
1571+}
1572+
1573+static const struct bpf_trampoline_ops trampoline_multi_ops = {
1574+ .register_fentry = register_fentry_multi,
1575+ .unregister_fentry = unregister_fentry_multi,
1576+ .modify_fentry = modify_fentry_multi,
1577+};
1578+
1579+static void bpf_trampoline_multi_attach_init(struct bpf_trampoline *tr)
1580+{
1581+ tr->multi_attach.old_image = tr->cur_image;
1582+ tr->multi_attach.old_flags = tr->flags;
1583+}
1584+
1585+static void bpf_trampoline_multi_attach_free(struct bpf_trampoline *tr)
1586+{
1587+ if (tr->multi_attach.old_image)
1588+ bpf_tramp_image_put(tr->multi_attach.old_image);
1589+
1590+ tr->multi_attach.old_image = NULL;
1591+ tr->multi_attach.old_flags = 0;
1592+}
1593+
1594+static void bpf_trampoline_multi_attach_rollback(struct bpf_trampoline *tr)
1595+{
1596+ if (tr->cur_image)
1597+ bpf_tramp_image_put(tr->cur_image);
1598+ tr->cur_image = tr->multi_attach.old_image;
1599+ tr->flags = tr->multi_attach.old_flags;
1600+
1601+ tr->multi_attach.old_image = NULL;
1602+ tr->multi_attach.old_flags = 0;
1603+}
1604+
1605+#define for_each_mnode_cnt(mnode, link, cnt) \
1606+ for (i = 0, mnode = &link->nodes[i]; i < cnt; i++, mnode = &link->nodes[i])
1607+
1608+#define for_each_mnode(mnode, link) \
1609+ for_each_mnode_cnt(mnode, link, link->nodes_cnt)
1610+
1611+int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
1612+ struct bpf_tracing_multi_link *link)
1613+{
1614+ struct bpf_tracing_multi_data *data = &link->data;
1615+ struct bpf_attach_target_info tgt_info = {};
1616+ struct btf *btf = prog->aux->attach_btf;
1617+ struct bpf_tracing_multi_node *mnode;
1618+ struct bpf_trampoline *tr;
1619+ int i, err, rollback_cnt;
1620+ u64 key;
1621+
1622+ for_each_mnode(mnode, link) {
1623+ rollback_cnt = i;
1624+
1625+ err = bpf_check_attach_btf_id_multi(btf, prog, ids[i], &tgt_info);
1626+ if (err)
1627+ goto rollback_put;
1628+
1629+ key = bpf_trampoline_compute_key(NULL, btf, ids[i]);
1630+
1631+ tr = bpf_trampoline_get(key, &tgt_info);
1632+ if (!tr) {
1633+ err = -ENOMEM;
1634+ goto rollback_put;
1635+ }
1636+
1637+ mnode->trampoline = tr;
1638+ mnode->node.link = &link->link;
1639+ mnode->node.cookie = link->cookies ? link->cookies[i] : 0;
1640+
1641+ if (prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) {
1642+ link->fexits[i].link = &link->link;
1643+ link->fexits[i].cookie = link->cookies ? link->cookies[i] : 0;
1644+ }
1645+
1646+ cond_resched();
1647+ }
1648+
1649+ err = init_tracing_multi_data(data);
1650+ if (err) {
1651+ rollback_cnt = link->nodes_cnt;
1652+ goto rollback_put;
1653+ }
1654+
1655+ trampoline_lock_all();
1656+
1657+ for_each_mnode(mnode, link) {
1658+ bpf_trampoline_multi_attach_init(mnode->trampoline);
1659+
1660+ data->entry = &mnode->entry;
1661+ err = __bpf_trampoline_link_prog(&mnode->node, mnode->trampoline, NULL,
1662+ &trampoline_multi_ops, data);
1663+ if (err) {
1664+ rollback_cnt = i;
1665+ goto rollback_unlink;
1666+ }
1667+ }
1668+
1669+ rollback_cnt = link->nodes_cnt;
1670+ if (ftrace_hash_count(data->reg)) {
1671+ err = update_ftrace_direct_add(&direct_ops, data->reg);
1672+ if (err)
1673+ goto rollback_unlink;
1674+ }
1675+
1676+ if (ftrace_hash_count(data->modify)) {
1677+ err = update_ftrace_direct_mod(&direct_ops, data->modify, true);
1678+ if (err) {
1679+ if (ftrace_hash_count(data->reg))
1680+ WARN_ON_ONCE(update_ftrace_direct_del(&direct_ops, data->reg));
1681+ goto rollback_unlink;
1682+ }
1683+ }
1684+
1685+ for_each_mnode(mnode, link)
1686+ bpf_trampoline_multi_attach_free(mnode->trampoline);
1687+
1688+ trampoline_unlock_all();
1689+
1690+ remove_tracing_multi_data(data);
1691+ return 0;
1692+
1693+rollback_unlink:
1694+ for_each_mnode_cnt(mnode, link, rollback_cnt) {
1695+ bpf_trampoline_remove_prog(mnode->trampoline, &mnode->node);
1696+ bpf_trampoline_multi_attach_rollback(mnode->trampoline);
1697+ }
1698+
1699+ trampoline_unlock_all();
1700+
1701+ clear_tracing_multi_data(data);
1702+ rollback_cnt = link->nodes_cnt;
1703+
1704+rollback_put:
1705+ for_each_mnode_cnt(mnode, link, rollback_cnt)
1706+ bpf_trampoline_put(mnode->trampoline);
1707+
1708+ return err;
1709+}
1710+
1711+int bpf_trampoline_multi_detach(struct bpf_prog *prog, struct bpf_tracing_multi_link *link)
1712+{
1713+ struct bpf_tracing_multi_data *data = &link->data;
1714+ struct bpf_tracing_multi_node *mnode;
1715+ int i;
1716+
1717+ trampoline_lock_all();
1718+
1719+ for_each_mnode(mnode, link) {
1720+ data->entry = &mnode->entry;
1721+ bpf_trampoline_multi_attach_init(mnode->trampoline);
1722+ WARN_ON_ONCE(__bpf_trampoline_unlink_prog(&mnode->node, mnode->trampoline,
1723+ NULL, &trampoline_multi_ops, data));
1724+ }
1725+
1726+ if (ftrace_hash_count(data->unreg))
1727+ WARN_ON_ONCE(update_ftrace_direct_del(&direct_ops, data->unreg));
1728+ if (ftrace_hash_count(data->modify))
1729+ WARN_ON_ONCE(update_ftrace_direct_mod(&direct_ops, data->modify, true));
1730+
1731+ for_each_mnode(mnode, link)
1732+ bpf_trampoline_multi_attach_free(mnode->trampoline);
1733+
1734+ trampoline_unlock_all();
1735+
1736+ for_each_mnode(mnode, link)
1737+ bpf_trampoline_put(mnode->trampoline);
1738+
1739+ clear_tracing_multi_data(data);
1740+ return 0;
1741+}
1742+
1743+#undef for_each_mnode_cnt
1744+#undef for_each_mnode
1745+
1746+#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS &&
1747+ CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS &&
1748+ CONFIG_BPF_SYSCALL */
1749+
13751750 static int __init init_trampolines(void)
13761751 {
13771752 int i;
@@ -1380,6 +1755,8 @@ static int __init init_trampolines(void) · 2 unmodified lines
13801755 INIT_HLIST_HEAD(&trampoline_key_table[i]);
13811756 for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++)
13821757 INIT_HLIST_HEAD(&trampoline_ip_table[i]);
1758+ for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++)
1759+ __mutex_init(&trampoline_locks[i].mutex, "trampoline_lock", &trampoline_locks[i].key);
13831760 return 0;
13841761 }
13851762 late_initcall(init_trampolines);
kernel/bpf/verifier.c+1978−2196
@@ -26,6 +26,7 @@ · 25 unmodified lines
2626 #include <linux/poison.h>
2727 #include <linux/module.h>
2828 #include <linux/cpumask.h>
29+#include <linux/cnum.h>
2930 #include <linux/bpf_mem_alloc.h>
3031 #include <net/xdp.h>
3132 #include <linux/trace_events.h>
@@ -199,14 +200,15 @@ struct bpf_verifier_stack_elem { · 167 unmodified lines
199200
200201 #define BPF_PRIV_STACK_MIN_SIZE 64
201202
202−static int acquire_reference(struct bpf_verifier_env *env, int insn_idx);
203−static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id);
204−static int release_reference(struct bpf_verifier_env *env, int ref_obj_id);
203+static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int parent_id);
204+static int release_reference_nomark(struct bpf_verifier_state *state, int id);
205+static int release_reference(struct bpf_verifier_env *env, int id);
205206 static void invalidate_non_owning_refs(struct bpf_verifier_env *env);
206207 static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env);
208+static bool is_tracing_prog_type(enum bpf_prog_type type);
207209 static int ref_set_non_owning(struct bpf_verifier_env *env,
208210 struct bpf_reg_state *reg);
209−static bool is_trusted_reg(const struct bpf_reg_state *reg);
211+static bool is_trusted_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg);
210212 static inline bool in_sleepable_context(struct bpf_verifier_env *env);
211213 static const char *non_sleepable_context_description(struct bpf_verifier_env *env);
212214 static void scalar32_min_max_add(struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg);
@@ -230,8 +232,28 @@ static void bpf_map_key_store(struct bpf_insn_aux_data *aux, u64 state) · 17 unmodified lines
230232 (poisoned ? BPF_MAP_KEY_POISON : 0ULL);
231233 }
232234
235+static void update_ref_obj(struct ref_obj_desc *ref_obj, struct bpf_reg_state *reg)
236+{
237+ ref_obj->id = reg->id;
238+ ref_obj->parent_id = reg->parent_id;
239+ ref_obj->cnt++;
240+}
241+
242+static int validate_ref_obj(struct bpf_verifier_env *env, struct ref_obj_desc *ref_obj)
243+{
244+ if (ref_obj->cnt > 1) {
245+ verifier_bug(env, "function expects only one referenced object but got %d\n",
246+ ref_obj->cnt);
247+ return -EFAULT;
248+ }
249+
250+ return 0;
251+}
252+
233253 struct bpf_call_arg_meta {
234254 struct bpf_map_desc map;
255+ struct bpf_dynptr_desc dynptr;
256+ struct ref_obj_desc ref_obj;
235257 bool raw_mode;
236258 bool pkt_access;
237259 u8 release_regno;
@@ -239,8 +261,6 @@ struct bpf_call_arg_meta { · 1 unmodified line
239261 int access_size;
240262 int mem_size;
241263 u64 msize_max_value;
242− int ref_obj_id;
243− int dynptr_id;
244264 int func_id;
245265 struct btf *btf;
246266 u32 btf_id;
@@ -261,6 +281,41 @@ struct bpf_kfunc_meta { · 14 unmodified lines
261281
262282 struct btf *btf_vmlinux;
263283
284+typedef struct argno {
285+ int argno;
286+} argno_t;
287+
288+static argno_t argno_from_reg(u32 regno)
289+{
290+ return (argno_t){ .argno = regno };
291+}
292+
293+static argno_t argno_from_arg(u32 arg)
294+{
295+ return (argno_t){ .argno = -arg };
296+}
297+
298+static int reg_from_argno(argno_t a)
299+{
300+ if (a.argno >= 0)
301+ return a.argno;
302+ if (a.argno >= -MAX_BPF_FUNC_REG_ARGS)
303+ return -a.argno;
304+ return -1;
305+}
306+
307+static int arg_from_argno(argno_t a)
308+{
309+ if (a.argno < 0)
310+ return -a.argno;
311+ return -1;
312+}
313+
314+static int arg_idx_from_argno(argno_t a)
315+{
316+ return arg_from_argno(a) - 1;
317+}
318+
264319 static const char *btf_type_name(const struct btf *btf, u32 id)
265320 {
266321 return btf_name_by_offset(btf, btf_type_by_id(btf, id)->name_off);
@@ -290,12 +345,12 @@ static void verbose_invalid_scalar(struct bpf_verifier_env *env, · 23 unmodified lines
290345 bool unknown = true;
291346
292347 verbose(env, "%s the register %s has", ctx, reg_name);
293− if (reg->smin_value > S64_MIN) {
294− verbose(env, " smin=%lld", reg->smin_value);
348+ if (reg_smin(reg) > S64_MIN) {
349+ verbose(env, " smin=%lld", reg_smin(reg));
295350 unknown = false;
296351 }
297− if (reg->smax_value < S64_MAX) {
298− verbose(env, " smax=%lld", reg->smax_value);
352+ if (reg_smax(reg) < S64_MAX) {
353+ verbose(env, " smax=%lld", reg_smax(reg));
299354 unknown = false;
300355 }
301356 if (unknown)
@@ -303,7 +358,7 @@ static void verbose_invalid_scalar(struct bpf_verifier_env *env, · 1 unmodified line
303358 verbose(env, " should have been in [%d, %d]\n", range.minval, range.maxval);
304359 }
305360
306−static bool reg_not_null(const struct bpf_reg_state *reg)
361+static bool reg_not_null(struct bpf_verifier_env *env, const struct bpf_reg_state *reg)
307362 {
308363 enum bpf_reg_type type;
309364
@@ -317,7 +372,7 @@ static bool reg_not_null(const struct bpf_reg_state *reg) · 7 unmodified lines
317372 type == PTR_TO_MAP_VALUE ||
318373 type == PTR_TO_MAP_KEY ||
319374 type == PTR_TO_SOCK_COMMON ||
320− (type == PTR_TO_BTF_ID && is_trusted_reg(reg)) ||
375+ (type == PTR_TO_BTF_ID && is_trusted_reg(env, reg)) ||
321376 (type == PTR_TO_MEM && !(reg->type & PTR_UNTRUSTED)) ||
322377 type == CONST_PTR_TO_MAP;
323378 }
@@ -434,11 +489,6 @@ static bool is_ptr_cast_function(enum bpf_func_id func_id) · 110 unmodified lines
434489 func_id == BPF_FUNC_skc_to_tcp_request_sock;
435490 }
436491
437−static bool is_dynptr_ref_function(enum bpf_func_id func_id)
438−{
439− return func_id == BPF_FUNC_dynptr_data;
440−}
441−
442492 static bool is_sync_callback_calling_kfunc(u32 btf_id);
443493 static bool is_async_callback_calling_kfunc(u32 btf_id);
444494 static bool is_callback_calling_kfunc(u32 btf_id);
@@ -497,22 +547,6 @@ bool bpf_is_may_goto_insn(struct bpf_insn *insn) · 52 unmodified lines
497547 return insn->code == (BPF_JMP | BPF_JCOND) && insn->src_reg == BPF_MAY_GOTO;
498548 }
499549
500−static bool helper_multiple_ref_obj_use(enum bpf_func_id func_id,
501− const struct bpf_map *map)
502−{
503− int ref_obj_uses = 0;
504−
505− if (is_ptr_cast_function(func_id))
506− ref_obj_uses++;
507− if (is_acquire_function(func_id, map))
508− ref_obj_uses++;
509− if (is_dynptr_ref_function(func_id))
510− ref_obj_uses++;
511−
512− return ref_obj_uses > 1;
513−}
514−
515−
516550 static bool is_spi_bounds_valid(struct bpf_func_state *state, int spi, int nr_slots)
517551 {
518552 int allocated_slots = state->allocated_stack / BPF_REG_SIZE;
@@ -609,43 +643,44 @@ static enum bpf_type_flag get_dynptr_type_flag(enum bpf_dynptr_type type) · 90 unmodified lines
609643 }
610644 }
611645
612−static bool dynptr_type_refcounted(enum bpf_dynptr_type type)
646+static bool dynptr_type_referenced(enum bpf_dynptr_type type)
613647 {
614648 return type == BPF_DYNPTR_TYPE_RINGBUF || type == BPF_DYNPTR_TYPE_FILE;
615649 }
616650
617651 static void __mark_dynptr_reg(struct bpf_reg_state *reg,
618652 enum bpf_dynptr_type type,
619− bool first_slot, int dynptr_id);
653+ bool first_slot, int id, int parent_id);
620654
621655
622656 static void mark_dynptr_stack_regs(struct bpf_verifier_env *env,
623657 struct bpf_reg_state *sreg1,
624658 struct bpf_reg_state *sreg2,
625− enum bpf_dynptr_type type)
659+ enum bpf_dynptr_type type, int parent_id)
626660 {
627661 int id = ++env->id_gen;
628662
629− __mark_dynptr_reg(sreg1, type, true, id);
630− __mark_dynptr_reg(sreg2, type, false, id);
663+ __mark_dynptr_reg(sreg1, type, true, id, parent_id);
664+ __mark_dynptr_reg(sreg2, type, false, id, parent_id);
631665 }
632666
633667 static void mark_dynptr_cb_reg(struct bpf_verifier_env *env,
634668 struct bpf_reg_state *reg,
635669 enum bpf_dynptr_type type)
636670 {
637− __mark_dynptr_reg(reg, type, true, ++env->id_gen);
671+ __mark_dynptr_reg(reg, type, true, ++env->id_gen, 0);
638672 }
639673
640674 static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,
641675 struct bpf_func_state *state, int spi);
642676
643677 static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
644− enum bpf_arg_type arg_type, int insn_idx, int clone_ref_obj_id)
678+ enum bpf_arg_type arg_type, int insn_idx,
679+ struct ref_obj_desc *ref_obj, struct bpf_dynptr_desc *dynptr)
645680 {
646681 struct bpf_func_state *state = bpf_func(env, reg);
682+ int spi, i, err, parent_id = 0;
647683 enum bpf_dynptr_type type;
648− int spi, i, err;
649684
650685 spi = dynptr_get_spi(env, reg);
651686 if (spi < 0)
@@ -676,94 +711,69 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_ · 24 unmodified lines
676711 if (type == BPF_DYNPTR_TYPE_INVALID)
677712 return -EINVAL;
678713
679− mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr,
680− &state->stack[spi - 1].spilled_ptr, type);
714+ if (dynptr->type == BPF_DYNPTR_TYPE_INVALID) { /* dynptr constructors */
715+ err = validate_ref_obj(env, ref_obj);
716+ if (err)
717+ return err;
681718
682− if (dynptr_type_refcounted(type)) {
683− /* The id is used to track proper releasing */
684− int id;
719+ /* Track parent's id if the parent is a referenced object */
720+ parent_id = ref_obj->id;
685721
686− if (clone_ref_obj_id)
687− id = clone_ref_obj_id;
688− else
689− id = acquire_reference(env, insn_idx);
722+ if (dynptr_type_referenced(type)) {
723+ int id;
690724
691− if (id < 0)
692− return id;
725+ /*
726+ * Create an intermediate reference that tracks the referenced
727+ * object for the referenced dynptr. Freeing a referenced dynptr
728+ * through helpers/kfuncs will invalidate all clones.
729+ */
730+ id = acquire_reference(env, insn_idx, parent_id);
731+ if (id < 0)
732+ return id;
693733
694− state->stack[spi].spilled_ptr.ref_obj_id = id;
695− state->stack[spi - 1].spilled_ptr.ref_obj_id = id;
734+ parent_id = id;
735+ }
736+ } else { /* bpf_dynptr_clone() */
737+ parent_id = dynptr->parent_id;
696738 }
697739
740+ mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr,
741+ &state->stack[spi - 1].spilled_ptr, type, parent_id);
742+
698743 return 0;
699744 }
700745
701−static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_state *state, int spi)
746+static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_stack_state *stack)
702747 {
703748 int i;
704749
705750 for (i = 0; i < BPF_REG_SIZE; i++) {
706− state->stack[spi].slot_type[i] = STACK_INVALID;
707− state->stack[spi - 1].slot_type[i] = STACK_INVALID;
751+ stack[0].slot_type[i] = STACK_INVALID;
752+ stack[1].slot_type[i] = STACK_INVALID;
708753 }
709754
710− bpf_mark_reg_not_init(env, &state->stack[spi].spilled_ptr);
711− bpf_mark_reg_not_init(env, &state->stack[spi - 1].spilled_ptr);
755+ bpf_mark_reg_not_init(env, &stack[0].spilled_ptr);
756+ bpf_mark_reg_not_init(env, &stack[1].spilled_ptr);
712757 }
713758
714759 static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
715760 {
716761 struct bpf_func_state *state = bpf_func(env, reg);
717− int spi, ref_obj_id, i;
762+ int spi;
718763
719− /*
720− * This can only be set for PTR_TO_STACK, as CONST_PTR_TO_DYNPTR cannot
721− * be released by any dynptr helper. Hence, unmark_stack_slots_dynptr
722− * is safe to do directly.
723− */
724− if (reg->type == CONST_PTR_TO_DYNPTR) {
725− verifier_bug(env, "CONST_PTR_TO_DYNPTR cannot be released");
726− return -EFAULT;
727− }
728764 spi = dynptr_get_spi(env, reg);
729765 if (spi < 0)
730766 return spi;
731767
732− if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) {
733− invalidate_dynptr(env, state, spi);
734− return 0;
735− }
736−
737− ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
738−
739− /* If the dynptr has a ref_obj_id, then we need to invalidate
740− * two things:
741− *
742− * 1) Any dynptrs with a matching ref_obj_id (clones)
743− * 2) Any slices derived from this dynptr.
768+ /*
769+ * For referenced dynptr, release the parent ref which cascades to
770+ * all clones and derived slices. For non-referenced dynptr, only
771+ * the dynptr and slices derived from it will be invalidated.
744772 */
745−
746− /* Invalidate any slices associated with this dynptr */
747− WARN_ON_ONCE(release_reference(env, ref_obj_id));
748−
749− /* Invalidate any dynptr clones */
750− for (i = 1; i < state->allocated_stack / BPF_REG_SIZE; i++) {
751− if (state->stack[i].spilled_ptr.ref_obj_id != ref_obj_id)
752− continue;
753−
754− /* it should always be the case that if the ref obj id
755− * matches then the stack slot also belongs to a
756− * dynptr
757− */
758− if (state->stack[i].slot_type[0] != STACK_DYNPTR) {
759− verifier_bug(env, "misconfigured ref_obj_id");
760− return -EFAULT;
761− }
762− if (state->stack[i].spilled_ptr.dynptr.first_slot)
763− invalidate_dynptr(env, state, i);
764− }
765−
766− return 0;
773+ reg = &state->stack[spi].spilled_ptr;
774+ return release_reference(env, dynptr_type_referenced(reg->dynptr.type)
775+ ? reg->parent_id
776+ : reg->id);
767777 }
768778
769779 static void __mark_reg_unknown(const struct bpf_verifier_env *env,
@@ -777,12 +787,29 @@ static void mark_reg_invalid(const struct bpf_verifier_env *env, struct bpf_reg_ · 7 unmodified lines
777787 __mark_reg_unknown(env, reg);
778788 }
779789
790+static int dynptr_ref_cnt(struct bpf_verifier_env *env, int v_parent_id)
791+{
792+ struct bpf_stack_state *stack;
793+ struct bpf_func_state *state;
794+ struct bpf_reg_state *reg;
795+ int ref_cnt = 0;
796+
797+ bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, 1 << STACK_DYNPTR, ({
798+ if (!stack || stack->slot_type[0] != STACK_DYNPTR)
799+ continue;
800+ if (!stack->spilled_ptr.dynptr.first_slot)
801+ continue;
802+ if (stack->spilled_ptr.parent_id == v_parent_id)
803+ ref_cnt++;
804+ }));
805+
806+ return ref_cnt;
807+}
808+
780809 static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,
781810 struct bpf_func_state *state, int spi)
782811 {
783− struct bpf_func_state *fstate;
784− struct bpf_reg_state *dreg;
785− int i, dynptr_id;
812+ int err = 0;
786813
787814 /* We always ensure that STACK_DYNPTR is never set partially,
788815 * hence just checking for slot_type[0] is enough. This is
@@ -796,56 +823,25 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, · 7 unmodified lines
796823 if (!state->stack[spi].spilled_ptr.dynptr.first_slot)
797824 spi = spi + 1;
798825
799− if (dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) {
800− int ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
801− int ref_cnt = 0;
802−
803− /*
804− * A referenced dynptr can be overwritten only if there is at
805− * least one other dynptr sharing the same ref_obj_id,
806− * ensuring the reference can still be properly released.
807− */
808− for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) {
809− if (state->stack[i].slot_type[0] != STACK_DYNPTR)
810− continue;
811− if (!state->stack[i].spilled_ptr.dynptr.first_slot)
812− continue;
813− if (state->stack[i].spilled_ptr.ref_obj_id == ref_obj_id)
814− ref_cnt++;
815− }
816−
817− if (ref_cnt <= 1) {
818− verbose(env, "cannot overwrite referenced dynptr\n");
819− return -EINVAL;
820− }
826+ /*
827+ * A referenced dynptr can be overwritten only if there is at
828+ * least one other dynptr sharing the same virtual ref parent,
829+ * ensuring the reference can still be properly released.
830+ */
831+ if (dynptr_type_referenced(state->stack[spi].spilled_ptr.dynptr.type) &&
832+ dynptr_ref_cnt(env, state->stack[spi].spilled_ptr.parent_id) <= 1) {
833+ verbose(env, "cannot overwrite referenced dynptr\n");
834+ return -EINVAL;
821835 }
822836
823− mark_stack_slot_scratched(env, spi);
824− mark_stack_slot_scratched(env, spi - 1);
825−
826− /* Writing partially to one dynptr stack slot destroys both. */
827− for (i = 0; i < BPF_REG_SIZE; i++) {
828− state->stack[spi].slot_type[i] = STACK_INVALID;
829− state->stack[spi - 1].slot_type[i] = STACK_INVALID;
837+ /* Invalidate the dynptr and any derived slices */
838+ err = release_reference(env, state->stack[spi].spilled_ptr.id);
839+ if (!err) {
840+ mark_stack_slot_scratched(env, spi);
841+ mark_stack_slot_scratched(env, spi - 1);
830842 }
831843
832− dynptr_id = state->stack[spi].spilled_ptr.id;
833− /* Invalidate any slices associated with this dynptr */
834− bpf_for_each_reg_in_vstate(env->cur_state, fstate, dreg, ({
835− /* Dynptr slices are only PTR_TO_MEM_OR_NULL and PTR_TO_MEM */
836− if (dreg->type != (PTR_TO_MEM | PTR_MAYBE_NULL) && dreg->type != PTR_TO_MEM)
837− continue;
838− if (dreg->dynptr_id == dynptr_id)
839− mark_reg_invalid(env, dreg);
840− }));
841−
842− /* Do not release reference state, we are destroying dynptr on stack,
843− * not using some helper to release it. Just reset register.
844− */
845− bpf_mark_reg_not_init(env, &state->stack[spi].spilled_ptr);
846− bpf_mark_reg_not_init(env, &state->stack[spi - 1].spilled_ptr);
847−
848− return 0;
844+ return err;
849845 }
850846
851847 static bool is_dynptr_reg_valid_uninit(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
@@ -945,7 +941,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env, · 93 unmodified lines
945941 if (spi < 0)
946942 return spi;
947943
948− id = acquire_reference(env, insn_idx);
944+ id = acquire_reference(env, insn_idx, 0);
949945 if (id < 0)
950946 return id;
951947
@@ -961,7 +957,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env, · 9 unmodified lines
961957 else
962958 st->type |= PTR_UNTRUSTED;
963959 }
964− st->ref_obj_id = i == 0 ? id : 0;
960+ st->id = i == 0 ? id : 0;
965961 st->iter.btf = btf;
966962 st->iter.btf_id = btf_id;
967963 st->iter.state = BPF_ITER_STATE_ACTIVE;
@@ -991,7 +987,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env, · 23 unmodified lines
991987 struct bpf_reg_state *st = &slot->spilled_ptr;
992988
993989 if (i == 0)
994− WARN_ON_ONCE(release_reference(env, st->ref_obj_id));
990+ WARN_ON_ONCE(release_reference(env, st->id));
995991
996992 bpf_mark_reg_not_init(env, st);
997993
@@ -1047,10 +1043,10 @@ static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s · 49 unmodified lines
10471043
10481044 if (st->type & PTR_UNTRUSTED)
10491045 return -EPROTO;
1050− /* only main (first) slot has ref_obj_id set */
1051− if (i == 0 && !st->ref_obj_id)
1046+ /* only main (first) slot has id set */
1047+ if (i == 0 && !st->id)
10521048 return -EINVAL;
1053− if (i != 0 && st->ref_obj_id)
1049+ if (i != 0 && st->id)
10541050 return -EINVAL;
10551051 if (st->iter.btf != btf || st->iter.btf_id != btf_id)
10561052 return -EINVAL;
@@ -1089,7 +1085,7 @@ static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env, · 32 unmodified lines
10891085
10901086 __mark_reg_known_zero(st);
10911087 st->type = PTR_TO_STACK; /* we don't have dedicated reg type */
1092− st->ref_obj_id = id;
1088+ st->id = id;
10931089 st->irq.kfunc_class = kfunc_class;
10941090
10951091 for (i = 0; i < BPF_REG_SIZE; i++)
@@ -1123,7 +1119,7 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r · 27 unmodified lines
11231119 return -EINVAL;
11241120 }
11251121
1126− err = release_irq_state(env->cur_state, st->ref_obj_id);
1122+ err = release_irq_state(env->cur_state, st->id);
11271123 WARN_ON_ONCE(err && err != -EACCES);
11281124 if (err) {
11291125 int insn_idx = 0;
@@ -1187,7 +1183,7 @@ static int is_irq_flag_reg_valid_init(struct bpf_verifier_env *env, struct bpf_r · 57 unmodified lines
11871183 slot = &state->stack[spi];
11881184 st = &slot->spilled_ptr;
11891185
1190− if (!st->ref_obj_id)
1186+ if (!st->id)
11911187 return -EINVAL;
11921188
11931189 for (i = 0; i < BPF_REG_SIZE; i++)
@@ -1339,6 +1335,18 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st · 145 unmodified lines
13391335 return -ENOMEM;
13401336
13411337 dst->allocated_stack = src->allocated_stack;
1338+
1339+ /* copy stack args state */
1340+ n = src->out_stack_arg_cnt;
1341+ if (n) {
1342+ dst->stack_arg_regs = copy_array(dst->stack_arg_regs, src->stack_arg_regs, n,
1343+ sizeof(struct bpf_reg_state),
1344+ GFP_KERNEL_ACCOUNT);
1345+ if (!dst->stack_arg_regs)
1346+ return -ENOMEM;
1347+ }
1348+
1349+ dst->out_stack_arg_cnt = src->out_stack_arg_cnt;
13421350 return 0;
13431351 }
13441352
@@ -1380,6 +1388,23 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state · 35 unmodified lines
13801388 return 0;
13811389 }
13821390
1391+static int grow_stack_arg_slots(struct bpf_verifier_env *env,
1392+ struct bpf_func_state *state, int cnt)
1393+{
1394+ size_t old_n = state->out_stack_arg_cnt;
1395+
1396+ if (old_n >= cnt)
1397+ return 0;
1398+
1399+ state->stack_arg_regs = realloc_array(state->stack_arg_regs, old_n, cnt,
1400+ sizeof(struct bpf_reg_state));
1401+ if (!state->stack_arg_regs)
1402+ return -ENOMEM;
1403+
1404+ state->out_stack_arg_cnt = cnt;
1405+ return 0;
1406+}
1407+
13831408 /* Acquire a pointer id from the env and update the state->refs to include
13841409 * this new pointer reference.
13851410 * On success, returns a valid pointer id to associate with the register
@@ -1399,7 +1424,7 @@ static struct bpf_reference_state *acquire_reference_state(struct bpf_verifier_e · 13 unmodified lines
13991424 return &state->refs[new_ofs];
14001425 }
14011426
1402−static int acquire_reference(struct bpf_verifier_env *env, int insn_idx)
1427+static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int parent_id)
14031428 {
14041429 struct bpf_reference_state *s;
14051430
@@ -1408,6 +1433,7 @@ static int acquire_reference(struct bpf_verifier_env *env, int insn_idx) · 2 unmodified lines
14081433 return -ENOMEM;
14091434 s->type = REF_TYPE_PTR;
14101435 s->id = ++env->id_gen;
1436+ s->parent_id = parent_id;
14111437 return s->id;
14121438 }
14131439
@@ -1464,17 +1490,25 @@ static void release_reference_state(struct bpf_verifier_state *state, int idx) · 50 unmodified lines
14641490 return;
14651491 }
14661492
1467−static bool find_reference_state(struct bpf_verifier_state *state, int ptr_id)
1493+static bool find_reference_state(struct bpf_verifier_state *state, int id)
14681494 {
14691495 int i;
14701496
1471− for (i = 0; i < state->acquired_refs; i++)
1472− if (state->refs[i].id == ptr_id)
1497+ for (i = 0; i < state->acquired_refs; i++) {
1498+ if (state->refs[i].type != REF_TYPE_PTR)
1499+ continue;
1500+ if (state->refs[i].id == id)
14731501 return true;
1502+ }
14741503
14751504 return false;
14761505 }
14771506
1507+static bool reg_is_referenced(struct bpf_verifier_env *env, const struct bpf_reg_state *reg)
1508+{
1509+ return find_reference_state(env->cur_state, reg->id);
1510+}
1511+
14781512 static int release_lock_state(struct bpf_verifier_state *state, int type, int id, void *ptr)
14791513 {
14801514 void *prev_ptr = NULL;
@@ -1542,6 +1576,7 @@ static void free_func_state(struct bpf_func_state *state) · 61 unmodified lines
15421576 {
15431577 if (!state)
15441578 return;
1579+ kfree(state->stack_arg_regs);
15451580 kfree(state->stack);
15461581 kfree(state);
15471582 }
@@ -1750,6 +1785,22 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env, · 202 unmodified lines
17501785 return &elem->st;
17511786 }
17521787
1788+static const char *reg_arg_name(struct bpf_verifier_env *env, argno_t argno)
1789+{
1790+ char *buf = env->tmp_arg_name;
1791+ int len = sizeof(env->tmp_arg_name);
1792+ int arg, regno = reg_from_argno(argno);
1793+
1794+ if (regno >= 0) {
1795+ snprintf(buf, len, "R%d", regno);
1796+ } else {
1797+ arg = arg_from_argno(argno);
1798+ snprintf(buf, len, "*(R11-%u)", (arg - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE);
1799+ }
1800+
1801+ return buf;
1802+}
1803+
17531804 static const int caller_saved[CALLER_SAVED_REGS] = {
17541805 BPF_REG_0, BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_5
17551806 };
@@ -1758,15 +1809,8 @@ static const int caller_saved[CALLER_SAVED_REGS] = { · 2 unmodified lines
17581809 static void ___mark_reg_known(struct bpf_reg_state *reg, u64 imm)
17591810 {
17601811 reg->var_off = tnum_const(imm);
1761− reg->smin_value = (s64)imm;
1762− reg->smax_value = (s64)imm;
1763− reg->umin_value = imm;
1764− reg->umax_value = imm;
1765−
1766− reg->s32_min_value = (s32)imm;
1767− reg->s32_max_value = (s32)imm;
1768− reg->u32_min_value = (u32)imm;
1769− reg->u32_max_value = (u32)imm;
1812+ reg->r64 = cnum64_from_urange(imm, imm);
1813+ reg->r32 = cnum32_from_urange((u32)imm, (u32)imm);
17701814 }
17711815
17721816 /* Mark the unknown part of a register (variable offset or scalar value) as
@@ -1778,17 +1822,14 @@ static void __mark_reg_known(struct bpf_reg_state *reg, u64 imm) · 5 unmodified lines
17781822 memset(((u8 *)reg) + sizeof(reg->type), 0,
17791823 offsetof(struct bpf_reg_state, var_off) - sizeof(reg->type));
17801824 reg->id = 0;
1781− reg->ref_obj_id = 0;
1825+ reg->parent_id = 0;
17821826 ___mark_reg_known(reg, imm);
17831827 }
17841828
17851829 static void __mark_reg32_known(struct bpf_reg_state *reg, u64 imm)
17861830 {
17871831 reg->var_off = tnum_const_subreg(reg->var_off, imm);
1788− reg->s32_min_value = (s32)imm;
1789− reg->s32_max_value = (s32)imm;
1790− reg->u32_min_value = (u32)imm;
1791− reg->u32_max_value = (u32)imm;
1832+ reg->r32 = cnum32_from_urange((u32)imm, (u32)imm);
17921833 }
17931834
17941835 /* Mark the 'variable offset' part of a register as zero. This should be
@@ -1816,7 +1857,7 @@ static void mark_reg_known_zero(struct bpf_verifier_env *env, · 21 unmodified lines
18161857 }
18171858
18181859 static void __mark_dynptr_reg(struct bpf_reg_state *reg, enum bpf_dynptr_type type,
1819− bool first_slot, int dynptr_id)
1860+ bool first_slot, int id, int parent_id)
18201861 {
18211862 /* reg->type has no meaning for STACK_DYNPTR, but when we set reg for
18221863 * callback arguments, it does need to be CONST_PTR_TO_DYNPTR, so simply
@@ -1825,7 +1866,8 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg, enum bpf_dynptr_type ty · 2 unmodified lines
18251866 __mark_reg_known_zero(reg);
18261867 reg->type = CONST_PTR_TO_DYNPTR;
18271868 /* Give each dynptr a unique id to uniquely associate slices to it. */
1828− reg->id = dynptr_id;
1869+ reg->id = id;
1870+ reg->parent_id = parent_id;
18291871 reg->dynptr.type = type;
18301872 reg->dynptr.first_slot = first_slot;
18311873 }
@@ -1899,34 +1941,21 @@ static bool reg_is_init_pkt_pointer(const struct bpf_reg_state *reg, · 67 unmodified lines
18991941 tnum_equals_const(reg->var_off, 0);
19001942 }
19011943
1902−/* Reset the min/max bounds of a register */
1903−static void __mark_reg_unbounded(struct bpf_reg_state *reg)
1944+static void __mark_reg32_unbounded(struct bpf_reg_state *reg)
19041945 {
1905− reg->smin_value = S64_MIN;
1906− reg->smax_value = S64_MAX;
1907− reg->umin_value = 0;
1908− reg->umax_value = U64_MAX;
1909−
1910− reg->s32_min_value = S32_MIN;
1911− reg->s32_max_value = S32_MAX;
1912− reg->u32_min_value = 0;
1913− reg->u32_max_value = U32_MAX;
1946+ reg->r32 = CNUM32_UNBOUNDED;
19141947 }
19151948
19161949 static void __mark_reg64_unbounded(struct bpf_reg_state *reg)
19171950 {
1918− reg->smin_value = S64_MIN;
1919− reg->smax_value = S64_MAX;
1920− reg->umin_value = 0;
1921− reg->umax_value = U64_MAX;
1951+ reg->r64 = CNUM64_UNBOUNDED;
19221952 }
19231953
1924−static void __mark_reg32_unbounded(struct bpf_reg_state *reg)
1954+/* Reset the min/max bounds of a register */
1955+static void __mark_reg_unbounded(struct bpf_reg_state *reg)
19251956 {
1926− reg->s32_min_value = S32_MIN;
1927− reg->s32_max_value = S32_MAX;
1928− reg->u32_min_value = 0;
1929− reg->u32_max_value = U32_MAX;
1957+ __mark_reg64_unbounded(reg);
1958+ __mark_reg32_unbounded(reg);
19301959 }
19311960
19321961 static void reset_reg64_and_tnum(struct bpf_reg_state *reg)
@@ -1941,19 +1970,32 @@ static void reset_reg32_and_tnum(struct bpf_reg_state *reg) · 8 unmodified lines
19411970 reg->var_off = tnum_unknown;
19421971 }
19431972
1944−static void __update_reg32_bounds(struct bpf_reg_state *reg)
1973+static struct cnum32 cnum32_from_tnum(struct tnum tnum)
19451974 {
1946− struct tnum var32_off = tnum_subreg(reg->var_off);
1975+ tnum = tnum_subreg(tnum);
1976+ if ((tnum.mask & S32_MIN) || (tnum.value & S32_MIN))
1977+ /* min signed is max(sign bit) | min(other bits) */
1978+ /* max signed is min(sign bit) | max(other bits) */
1979+ return cnum32_from_srange(tnum.value | (tnum.mask & S32_MIN),
1980+ tnum.value | (tnum.mask & S32_MAX));
1981+ else
1982+ return cnum32_from_urange(tnum.value, (tnum.value | tnum.mask));
1983+}
19471984
1948− /* min signed is max(sign bit) | min(other bits) */
1949− reg->s32_min_value = max_t(s32, reg->s32_min_value,
1950− var32_off.value | (var32_off.mask & S32_MIN));
1951− /* max signed is min(sign bit) | max(other bits) */
1952− reg->s32_max_value = min_t(s32, reg->s32_max_value,
1953− var32_off.value | (var32_off.mask & S32_MAX));
1954− reg->u32_min_value = max_t(u32, reg->u32_min_value, (u32)var32_off.value);
1955− reg->u32_max_value = min(reg->u32_max_value,
1956− (u32)(var32_off.value | var32_off.mask));
1985+static struct cnum64 cnum64_from_tnum(struct tnum tnum)
1986+{
1987+ if ((tnum.mask & S64_MIN) || (tnum.value & S64_MIN))
1988+ /* min signed is max(sign bit) | min(other bits) */
1989+ /* max signed is min(sign bit) | max(other bits) */
1990+ return cnum64_from_srange(tnum.value | (tnum.mask & S64_MIN),
1991+ tnum.value | (tnum.mask & S64_MAX));
1992+ else
1993+ return cnum64_from_urange(tnum.value, (tnum.value | tnum.mask));
1994+}
1995+
1996+static void __update_reg32_bounds(struct bpf_reg_state *reg)
1997+{
1998+ cnum32_intersect_with(&reg->r32, cnum32_from_tnum(reg->var_off));
19571999 }
19582000
19592001 static void __update_reg64_bounds(struct bpf_reg_state *reg)
@@ -1961,26 +2003,18 @@ static void __update_reg64_bounds(struct bpf_reg_state *reg) · 1 unmodified line
19612003 u64 tnum_next, tmax;
19622004 bool umin_in_tnum;
19632005
1964− /* min signed is max(sign bit) | min(other bits) */
1965− reg->smin_value = max_t(s64, reg->smin_value,
1966− reg->var_off.value | (reg->var_off.mask & S64_MIN));
1967− /* max signed is min(sign bit) | max(other bits) */
1968− reg->smax_value = min_t(s64, reg->smax_value,
1969− reg->var_off.value | (reg->var_off.mask & S64_MAX));
1970− reg->umin_value = max(reg->umin_value, reg->var_off.value);
1971− reg->umax_value = min(reg->umax_value,
1972− reg->var_off.value | reg->var_off.mask);
2006+ cnum64_intersect_with(&reg->r64, cnum64_from_tnum(reg->var_off));
19732007
19742008 /* Check if u64 and tnum overlap in a single value */
1975− tnum_next = tnum_step(reg->var_off, reg->umin_value);
1976− umin_in_tnum = (reg->umin_value & ~reg->var_off.mask) == reg->var_off.value;
2009+ tnum_next = tnum_step(reg->var_off, reg_umin(reg));
2010+ umin_in_tnum = (reg_umin(reg) & ~reg->var_off.mask) == reg->var_off.value;
19772011 tmax = reg->var_off.value | reg->var_off.mask;
1978− if (umin_in_tnum && tnum_next > reg->umax_value) {
2012+ if (umin_in_tnum && tnum_next > reg_umax(reg)) {
19792013 /* The u64 range and the tnum only overlap in umin.
19802014 * u64: ---[xxxxxx]-----
19812015 * tnum: --xx----------x-
19822016 */
1983− ___mark_reg_known(reg, reg->umin_value);
2017+ ___mark_reg_known(reg, reg_umin(reg));
19842018 } else if (!umin_in_tnum && tnum_next == tmax) {
19852019 /* The u64 range and the tnum only overlap in the maximum value
19862020 * represented by the tnum, called tmax.
@@ -1988,8 +2022,8 @@ static void __update_reg64_bounds(struct bpf_reg_state *reg) · 1 unmodified line
19882022 * tnum: xx-----x--------
19892023 */
19902024 ___mark_reg_known(reg, tmax);
1991− } else if (!umin_in_tnum && tnum_next <= reg->umax_value &&
1992− tnum_step(reg->var_off, tnum_next) > reg->umax_value) {
2025+ } else if (!umin_in_tnum && tnum_next <= reg_umax(reg) &&
2026+ tnum_step(reg->var_off, tnum_next) > reg_umax(reg)) {
19932027 /* The u64 range and the tnum only overlap in between umin
19942028 * (excluded) and umax.
19952029 * u64: ---[xxxxxx]-----
@@ -2005,329 +2039,19 @@ static void __update_reg_bounds(struct bpf_reg_state *reg) · 9 unmodified lines
20052039 __update_reg64_bounds(reg);
20062040 }
20072041
2008−/* Uses signed min/max values to inform unsigned, and vice-versa */
20092042 static void deduce_bounds_32_from_64(struct bpf_reg_state *reg)
20102043 {
2011− /* If upper 32 bits of u64/s64 range don't change, we can use lower 32
2012− * bits to improve our u32/s32 boundaries.
2013− *
2014− * E.g., the case where we have upper 32 bits as zero ([10, 20] in
2015− * u64) is pretty trivial, it's obvious that in u32 we'll also have
2016− * [10, 20] range. But this property holds for any 64-bit range as
2017− * long as upper 32 bits in that entire range of values stay the same.
2018− *
2019− * E.g., u64 range [0x10000000A, 0x10000000F] ([4294967306, 4294967311]
2020− * in decimal) has the same upper 32 bits throughout all the values in
2021− * that range. As such, lower 32 bits form a valid [0xA, 0xF] ([10, 15])
2022− * range.
2023− *
2024− * Note also, that [0xA, 0xF] is a valid range both in u32 and in s32,
2025− * following the rules outlined below about u64/s64 correspondence
2026− * (which equally applies to u32 vs s32 correspondence). In general it
2027− * depends on actual hexadecimal values of 32-bit range. They can form
2028− * only valid u32, or only valid s32 ranges in some cases.
2029− *
2030− * So we use all these insights to derive bounds for subregisters here.
2031− */
2032− if ((reg->umin_value >> 32) == (reg->umax_value >> 32)) {
2033− /* u64 to u32 casting preserves validity of low 32 bits as
2034− * a range, if upper 32 bits are the same
2035− */
2036− reg->u32_min_value = max_t(u32, reg->u32_min_value, (u32)reg->umin_value);
2037− reg->u32_max_value = min_t(u32, reg->u32_max_value, (u32)reg->umax_value);
2038−
2039− if ((s32)reg->umin_value <= (s32)reg->umax_value) {
2040− reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->umin_value);
2041− reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->umax_value);
2042− }
2043− }
2044− if ((reg->smin_value >> 32) == (reg->smax_value >> 32)) {
2045− /* low 32 bits should form a proper u32 range */
2046− if ((u32)reg->smin_value <= (u32)reg->smax_value) {
2047− reg->u32_min_value = max_t(u32, reg->u32_min_value, (u32)reg->smin_value);
2048− reg->u32_max_value = min_t(u32, reg->u32_max_value, (u32)reg->smax_value);
2049− }
2050− /* low 32 bits should form a proper s32 range */
2051− if ((s32)reg->smin_value <= (s32)reg->smax_value) {
2052− reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->smin_value);
2053− reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->smax_value);
2054− }
2055− }
2056− /* Special case where upper bits form a small sequence of two
2057− * sequential numbers (in 32-bit unsigned space, so 0xffffffff to
2058− * 0x00000000 is also valid), while lower bits form a proper s32 range
2059− * going from negative numbers to positive numbers. E.g., let's say we
2060− * have s64 range [-1, 1] ([0xffffffffffffffff, 0x0000000000000001]).
2061− * Possible s64 values are {-1, 0, 1} ({0xffffffffffffffff,
2062− * 0x0000000000000000, 0x00000000000001}). Ignoring upper 32 bits,
2063− * we still get a valid s32 range [-1, 1] ([0xffffffff, 0x00000001]).
2064− * Note that it doesn't have to be 0xffffffff going to 0x00000000 in
2065− * upper 32 bits. As a random example, s64 range
2066− * [0xfffffff0fffffff0; 0xfffffff100000010], forms a valid s32 range
2067− * [-16, 16] ([0xfffffff0; 0x00000010]) in its 32 bit subregister.
2068− */
2069− if ((u32)(reg->umin_value >> 32) + 1 == (u32)(reg->umax_value >> 32) &&
2070− (s32)reg->umin_value < 0 && (s32)reg->umax_value >= 0) {
2071− reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->umin_value);
2072− reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->umax_value);
2073− }
2074− if ((u32)(reg->smin_value >> 32) + 1 == (u32)(reg->smax_value >> 32) &&
2075− (s32)reg->smin_value < 0 && (s32)reg->smax_value >= 0) {
2076− reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->smin_value);
2077− reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->smax_value);
2078− }
2079−}
2080−
2081−static void deduce_bounds_32_from_32(struct bpf_reg_state *reg)
2082−{
2083− /* if u32 range forms a valid s32 range (due to matching sign bit),
2084− * try to learn from that
2085− */
2086− if ((s32)reg->u32_min_value <= (s32)reg->u32_max_value) {
2087− reg->s32_min_value = max_t(s32, reg->s32_min_value, reg->u32_min_value);
2088− reg->s32_max_value = min_t(s32, reg->s32_max_value, reg->u32_max_value);
2089− }
2090− /* If we cannot cross the sign boundary, then signed and unsigned bounds
2091− * are the same, so combine. This works even in the negative case, e.g.
2092− * -3 s<= x s<= -1 implies 0xf...fd u<= x u<= 0xf...ff.
2093− */
2094− if ((u32)reg->s32_min_value <= (u32)reg->s32_max_value) {
2095− reg->u32_min_value = max_t(u32, reg->s32_min_value, reg->u32_min_value);
2096− reg->u32_max_value = min_t(u32, reg->s32_max_value, reg->u32_max_value);
2097− } else {
2098− if (reg->u32_max_value < (u32)reg->s32_min_value) {
2099− /* See __reg64_deduce_bounds() for detailed explanation.
2100− * Refine ranges in the following situation:
2101− *
2102− * 0 U32_MAX
2103− * | [xxxxxxxxxxxxxx u32 range xxxxxxxxxxxxxx] |
2104− * |----------------------------|----------------------------|
2105− * |xxxxx s32 range xxxxxxxxx] [xxxxxxx|
2106− * 0 S32_MAX S32_MIN -1
2107− */
2108− reg->s32_min_value = (s32)reg->u32_min_value;
2109− reg->u32_max_value = min_t(u32, reg->u32_max_value, reg->s32_max_value);
2110− } else if ((u32)reg->s32_max_value < reg->u32_min_value) {
2111− /*
2112− * 0 U32_MAX
2113− * | [xxxxxxxxxxxxxx u32 range xxxxxxxxxxxxxx] |
2114− * |----------------------------|----------------------------|
2115− * |xxxxxxxxx] [xxxxxxxxxxxx s32 range |
2116− * 0 S32_MAX S32_MIN -1
2117− */
2118− reg->s32_max_value = (s32)reg->u32_max_value;
2119− reg->u32_min_value = max_t(u32, reg->u32_min_value, reg->s32_min_value);
2120− }
2121− }
2122−}
2123−
2124−static void deduce_bounds_64_from_64(struct bpf_reg_state *reg)
2125−{
2126− /* If u64 range forms a valid s64 range (due to matching sign bit),
2127− * try to learn from that. Let's do a bit of ASCII art to see when
2128− * this is happening. Let's take u64 range first:
2129− *
2130− * 0 0x7fffffffffffffff 0x8000000000000000 U64_MAX
2131− * |-------------------------------|--------------------------------|
2132− *
2133− * Valid u64 range is formed when umin and umax are anywhere in the
2134− * range [0, U64_MAX], and umin <= umax. u64 case is simple and
2135− * straightforward. Let's see how s64 range maps onto the same range
2136− * of values, annotated below the line for comparison:
2137− *
2138− * 0 0x7fffffffffffffff 0x8000000000000000 U64_MAX
2139− * |-------------------------------|--------------------------------|
2140− * 0 S64_MAX S64_MIN -1
2141− *
2142− * So s64 values basically start in the middle and they are logically
2143− * contiguous to the right of it, wrapping around from -1 to 0, and
2144− * then finishing as S64_MAX (0x7fffffffffffffff) right before
2145− * S64_MIN. We can try drawing the continuity of u64 vs s64 values
2146− * more visually as mapped to sign-agnostic range of hex values.
2147− *
2148− * u64 start u64 end
2149− * _______________________________________________________________
2150− * / \
2151− * 0 0x7fffffffffffffff 0x8000000000000000 U64_MAX
2152− * |-------------------------------|--------------------------------|
2153− * 0 S64_MAX S64_MIN -1
2154− * / \
2155− * >------------------------------ ------------------------------->
2156− * s64 continues... s64 end s64 start s64 "midpoint"
2157− *
2158− * What this means is that, in general, we can't always derive
2159− * something new about u64 from any random s64 range, and vice versa.
2160− *
2161− * But we can do that in two particular cases. One is when entire
2162− * u64/s64 range is *entirely* contained within left half of the above
2163− * diagram or when it is *entirely* contained in the right half. I.e.:
2164− *
2165− * |-------------------------------|--------------------------------|
2166− * ^ ^ ^ ^
2167− * A B C D
2168− *
2169− * [A, B] and [C, D] are contained entirely in their respective halves
2170− * and form valid contiguous ranges as both u64 and s64 values. [A, B]
2171− * will be non-negative both as u64 and s64 (and in fact it will be
2172− * identical ranges no matter the signedness). [C, D] treated as s64
2173− * will be a range of negative values, while in u64 it will be
2174− * non-negative range of values larger than 0x8000000000000000.
2175− *
2176− * Now, any other range here can't be represented in both u64 and s64
2177− * simultaneously. E.g., [A, C], [A, D], [B, C], [B, D] are valid
2178− * contiguous u64 ranges, but they are discontinuous in s64. [B, C]
2179− * in s64 would be properly presented as [S64_MIN, C] and [B, S64_MAX],
2180− * for example. Similarly, valid s64 range [D, A] (going from negative
2181− * to positive values), would be two separate [D, U64_MAX] and [0, A]
2182− * ranges as u64. Currently reg_state can't represent two segments per
2183− * numeric domain, so in such situations we can only derive maximal
2184− * possible range ([0, U64_MAX] for u64, and [S64_MIN, S64_MAX] for s64).
2185− *
2186− * So we use these facts to derive umin/umax from smin/smax and vice
2187− * versa only if they stay within the same "half". This is equivalent
2188− * to checking sign bit: lower half will have sign bit as zero, upper
2189− * half have sign bit 1. Below in code we simplify this by just
2190− * casting umin/umax as smin/smax and checking if they form valid
2191− * range, and vice versa. Those are equivalent checks.
2192− */
2193− if ((s64)reg->umin_value <= (s64)reg->umax_value) {
2194− reg->smin_value = max_t(s64, reg->smin_value, reg->umin_value);
2195− reg->smax_value = min_t(s64, reg->smax_value, reg->umax_value);
2196− }
2197− /* If we cannot cross the sign boundary, then signed and unsigned bounds
2198− * are the same, so combine. This works even in the negative case, e.g.
2199− * -3 s<= x s<= -1 implies 0xf...fd u<= x u<= 0xf...ff.
2200− */
2201− if ((u64)reg->smin_value <= (u64)reg->smax_value) {
2202− reg->umin_value = max_t(u64, reg->smin_value, reg->umin_value);
2203− reg->umax_value = min_t(u64, reg->smax_value, reg->umax_value);
2204− } else {
2205− /* If the s64 range crosses the sign boundary, then it's split
2206− * between the beginning and end of the U64 domain. In that
2207− * case, we can derive new bounds if the u64 range overlaps
2208− * with only one end of the s64 range.
2209− *
2210− * In the following example, the u64 range overlaps only with
2211− * positive portion of the s64 range.
2212− *
2213− * 0 U64_MAX
2214− * | [xxxxxxxxxxxxxx u64 range xxxxxxxxxxxxxx] |
2215− * |----------------------------|----------------------------|
2216− * |xxxxx s64 range xxxxxxxxx] [xxxxxxx|
2217− * 0 S64_MAX S64_MIN -1
2218− *
2219− * We can thus derive the following new s64 and u64 ranges.
2220− *
2221− * 0 U64_MAX
2222− * | [xxxxxx u64 range xxxxx] |
2223− * |----------------------------|----------------------------|
2224− * | [xxxxxx s64 range xxxxx] |
2225− * 0 S64_MAX S64_MIN -1
2226− *
2227− * If they overlap in two places, we can't derive anything
2228− * because reg_state can't represent two ranges per numeric
2229− * domain.
2230− *
2231− * 0 U64_MAX
2232− * | [xxxxxxxxxxxxxxxxx u64 range xxxxxxxxxxxxxxxxx] |
2233− * |----------------------------|----------------------------|
2234− * |xxxxx s64 range xxxxxxxxx] [xxxxxxxxxx|
2235− * 0 S64_MAX S64_MIN -1
2236− *
2237− * The first condition below corresponds to the first diagram
2238− * above.
2239− */
2240− if (reg->umax_value < (u64)reg->smin_value) {
2241− reg->smin_value = (s64)reg->umin_value;
2242− reg->umax_value = min_t(u64, reg->umax_value, reg->smax_value);
2243− } else if ((u64)reg->smax_value < reg->umin_value) {
2244− /* This second condition considers the case where the u64 range
2245− * overlaps with the negative portion of the s64 range:
2246− *
2247− * 0 U64_MAX
2248− * | [xxxxxxxxxxxxxx u64 range xxxxxxxxxxxxxx] |
2249− * |----------------------------|----------------------------|
2250− * |xxxxxxxxx] [xxxxxxxxxxxx s64 range |
2251− * 0 S64_MAX S64_MIN -1
2252− */
2253− reg->smax_value = (s64)reg->umax_value;
2254− reg->umin_value = max_t(u64, reg->umin_value, reg->smin_value);
2255− }
2256− }
2044+ cnum32_intersect_with(&reg->r32, cnum32_from_cnum64(reg->r64));
22572045 }
22582046
22592047 static void deduce_bounds_64_from_32(struct bpf_reg_state *reg)
22602048 {
2261− /* Try to tighten 64-bit bounds from 32-bit knowledge, using 32-bit
2262− * values on both sides of 64-bit range in hope to have tighter range.
2263− * E.g., if r1 is [0x1'00000000, 0x3'80000000], and we learn from
2264− * 32-bit signed > 0 operation that s32 bounds are now [1; 0x7fffffff].
2265− * With this, we can substitute 1 as low 32-bits of _low_ 64-bit bound
2266− * (0x100000000 -> 0x100000001) and 0x7fffffff as low 32-bits of
2267− * _high_ 64-bit bound (0x380000000 -> 0x37fffffff) and arrive at a
2268− * better overall bounds for r1 as [0x1'000000001; 0x3'7fffffff].
2269− * We just need to make sure that derived bounds we are intersecting
2270− * with are well-formed ranges in respective s64 or u64 domain, just
2271− * like we do with similar kinds of 32-to-64 or 64-to-32 adjustments.
2272− */
2273− __u64 new_umin, new_umax;
2274− __s64 new_smin, new_smax;
2275−
2276− /* u32 -> u64 tightening, it's always well-formed */
2277− new_umin = (reg->umin_value & ~0xffffffffULL) | reg->u32_min_value;
2278− new_umax = (reg->umax_value & ~0xffffffffULL) | reg->u32_max_value;
2279− reg->umin_value = max_t(u64, reg->umin_value, new_umin);
2280− reg->umax_value = min_t(u64, reg->umax_value, new_umax);
2281− /* u32 -> s64 tightening, u32 range embedded into s64 preserves range validity */
2282− new_smin = (reg->smin_value & ~0xffffffffULL) | reg->u32_min_value;
2283− new_smax = (reg->smax_value & ~0xffffffffULL) | reg->u32_max_value;
2284− reg->smin_value = max_t(s64, reg->smin_value, new_smin);
2285− reg->smax_value = min_t(s64, reg->smax_value, new_smax);
2286−
2287− /* Here we would like to handle a special case after sign extending load,
2288− * when upper bits for a 64-bit range are all 1s or all 0s.
2289− *
2290− * Upper bits are all 1s when register is in a range:
2291− * [0xffff_ffff_0000_0000, 0xffff_ffff_ffff_ffff]
2292− * Upper bits are all 0s when register is in a range:
2293− * [0x0000_0000_0000_0000, 0x0000_0000_ffff_ffff]
2294− * Together this forms are continuous range:
2295− * [0xffff_ffff_0000_0000, 0x0000_0000_ffff_ffff]
2296− *
2297− * Now, suppose that register range is in fact tighter:
2298− * [0xffff_ffff_8000_0000, 0x0000_0000_ffff_ffff] (R)
2299− * Also suppose that it's 32-bit range is positive,
2300− * meaning that lower 32-bits of the full 64-bit register
2301− * are in the range:
2302− * [0x0000_0000, 0x7fff_ffff] (W)
2303− *
2304− * If this happens, then any value in a range:
2305− * [0xffff_ffff_0000_0000, 0xffff_ffff_7fff_ffff]
2306− * is smaller than a lowest bound of the range (R):
2307− * 0xffff_ffff_8000_0000
2308− * which means that upper bits of the full 64-bit register
2309− * can't be all 1s, when lower bits are in range (W).
2310− *
2311− * Note that:
2312− * - 0xffff_ffff_8000_0000 == (s64)S32_MIN
2313− * - 0x0000_0000_7fff_ffff == (s64)S32_MAX
2314− * These relations are used in the conditions below.
2315− */
2316− if (reg->s32_min_value >= 0 && reg->smin_value >= S32_MIN && reg->smax_value <= S32_MAX) {
2317− reg->smin_value = reg->s32_min_value;
2318− reg->smax_value = reg->s32_max_value;
2319− reg->umin_value = reg->s32_min_value;
2320− reg->umax_value = reg->s32_max_value;
2321− reg->var_off = tnum_intersect(reg->var_off,
2322− tnum_range(reg->smin_value, reg->smax_value));
2323− }
2049+ reg->r64 = cnum64_cnum32_intersect(reg->r64, reg->r32);
23242050 }
23252051
23262052 static void __reg_deduce_bounds(struct bpf_reg_state *reg)
23272053 {
2328− deduce_bounds_64_from_64(reg);
23292054 deduce_bounds_32_from_64(reg);
2330− deduce_bounds_32_from_32(reg);
23312055 deduce_bounds_64_from_32(reg);
23322056 }
23332057
@@ -2335,11 +2059,11 @@ static void __reg_deduce_bounds(struct bpf_reg_state *reg) · 1 unmodified line
23352059 static void __reg_bound_offset(struct bpf_reg_state *reg)
23362060 {
23372061 struct tnum var64_off = tnum_intersect(reg->var_off,
2338− tnum_range(reg->umin_value,
2339− reg->umax_value));
2062+ tnum_range(reg_umin(reg),
2063+ reg_umax(reg)));
23402064 struct tnum var32_off = tnum_intersect(tnum_subreg(var64_off),
2341− tnum_range(reg->u32_min_value,
2342− reg->u32_max_value));
2065+ tnum_range(reg_u32_min(reg),
2066+ reg_u32_max(reg)));
23432067
23442068 reg->var_off = tnum_or(tnum_clear_subreg(var64_off), var32_off);
23452069 }
@@ -2365,35 +2089,25 @@ static void reg_bounds_sync(struct bpf_reg_state *reg) · 19 unmodified lines
23652089 __update_reg_bounds(reg);
23662090 }
23672091
2368−static bool range_bounds_violation(struct bpf_reg_state *reg)
2369−{
2370− return (reg->umin_value > reg->umax_value || reg->smin_value > reg->smax_value ||
2371− reg->u32_min_value > reg->u32_max_value ||
2372− reg->s32_min_value > reg->s32_max_value);
2373−}
2374−
23752092 static bool const_tnum_range_mismatch(struct bpf_reg_state *reg)
23762093 {
2377− u64 uval = reg->var_off.value;
2378− s64 sval = (s64)uval;
2379−
23802094 if (!tnum_is_const(reg->var_off))
23812095 return false;
23822096
2383− return reg->umin_value != uval || reg->umax_value != uval ||
2384− reg->smin_value != sval || reg->smax_value != sval;
2097+ return !cnum64_is_const(reg->r64) || reg->r64.base != reg->var_off.value;
23852098 }
23862099
23872100 static bool const_tnum_range_mismatch_32(struct bpf_reg_state *reg)
23882101 {
2389− u32 uval32 = tnum_subreg(reg->var_off).value;
2390− s32 sval32 = (s32)uval32;
2391−
23922102 if (!tnum_subreg_is_const(reg->var_off))
23932103 return false;
23942104
2395− return reg->u32_min_value != uval32 || reg->u32_max_value != uval32 ||
2396− reg->s32_min_value != sval32 || reg->s32_max_value != sval32;
2105+ return !cnum32_is_const(reg->r32) || reg->r32.base != tnum_subreg(reg->var_off).value;
2106+}
2107+
2108+static bool range_bounds_violation(struct bpf_reg_state *reg)
2109+{
2110+ return cnum32_is_empty(reg->r32) || cnum64_is_empty(reg->r64);
23972111 }
23982112
23992113 static int reg_bounds_sanity_check(struct bpf_verifier_env *env,
@@ -2418,12 +2132,11 @@ static int reg_bounds_sanity_check(struct bpf_verifier_env *env, · 18 unmodified lines
24182132
24192133 return 0;
24202134 out:
2421− verifier_bug(env, "REG INVARIANTS VIOLATION (%s): %s u64=[%#llx, %#llx] "
2422− "s64=[%#llx, %#llx] u32=[%#x, %#x] s32=[%#x, %#x] var_off=(%#llx, %#llx)",
2423− ctx, msg, reg->umin_value, reg->umax_value,
2424− reg->smin_value, reg->smax_value,
2425− reg->u32_min_value, reg->u32_max_value,
2426− reg->s32_min_value, reg->s32_max_value,
2135+ verifier_bug(env, "REG INVARIANTS VIOLATION (%s): %s r64={.base=%#llx, .size=%#llx} "
2136+ "r32={.base=%#x, .size=%#x} var_off=(%#llx, %#llx)",
2137+ ctx, msg,
2138+ reg->r64.base, reg->r64.size,
2139+ reg->r32.base, reg->r32.size,
24272140 reg->var_off.value, reg->var_off.mask);
24282141 if (env->test_reg_invariants)
24292142 return -EFAULT;
@@ -2431,44 +2144,15 @@ out: · 1 unmodified line
24312144 return 0;
24322145 }
24332146
2434−static bool __reg32_bound_s64(s32 a)
2435−{
2436− return a >= 0 && a <= S32_MAX;
2437−}
2438−
2439−static void __reg_assign_32_into_64(struct bpf_reg_state *reg)
2440−{
2441− reg->umin_value = reg->u32_min_value;
2442− reg->umax_value = reg->u32_max_value;
2443−
2444− /* Attempt to pull 32-bit signed bounds into 64-bit bounds but must
2445− * be positive otherwise set to worse case bounds and refine later
2446− * from tnum.
2447− */
2448− if (__reg32_bound_s64(reg->s32_min_value) &&
2449− __reg32_bound_s64(reg->s32_max_value)) {
2450− reg->smin_value = reg->s32_min_value;
2451− reg->smax_value = reg->s32_max_value;
2452− } else {
2453− reg->smin_value = 0;
2454− reg->smax_value = U32_MAX;
2455− }
2456−}
2457−
24582147 /* Mark a register as having a completely unknown (scalar) value. */
24592148 void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg)
24602149 {
2461− /*
2462− * Clear type, off, and union(map_ptr, range) and
2463− * padding between 'type' and union
2464− */
2465− memset(reg, 0, offsetof(struct bpf_reg_state, var_off));
2150+ s32 subreg_def = reg->subreg_def;
2151+
2152+ memset(reg, 0, sizeof(*reg));
24662153 reg->type = SCALAR_VALUE;
2467− reg->id = 0;
2468− reg->ref_obj_id = 0;
24692154 reg->var_off = tnum_unknown;
2470− reg->frameno = 0;
2471− reg->precise = false;
2155+ reg->subreg_def = subreg_def;
24722156 __mark_reg_unbounded(reg);
24732157 }
24742158
@@ -2496,11 +2180,12 @@ static int __mark_reg_s32_range(struct bpf_verifier_env *env, · 21 unmodified lines
24962180 {
24972181 struct bpf_reg_state *reg = regs + regno;
24982182
2499− reg->s32_min_value = max_t(s32, reg->s32_min_value, s32_min);
2500− reg->s32_max_value = min_t(s32, reg->s32_max_value, s32_max);
2501−
2502− reg->smin_value = max_t(s64, reg->smin_value, s32_min);
2503− reg->smax_value = min_t(s64, reg->smax_value, s32_max);
2183+ reg_set_srange32(reg,
2184+ max_t(s32, reg_s32_min(reg), s32_min),
2185+ min_t(s32, reg_s32_max(reg), s32_max));
2186+ reg_set_srange64(reg,
2187+ max_t(s64, reg_smin(reg), s32_min),
2188+ min_t(s64, reg_smax(reg), s32_max));
25042189
25052190 reg_bounds_sync(reg);
25062191
@@ -3295,50 +2980,13 @@ out: · 788 unmodified lines
32952980 return ret;
32962981 }
32972982
3298−static int mark_stack_slot_obj_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
3299− int spi, int nr_slots)
2983+static void mark_stack_slots_scratched(struct bpf_verifier_env *env,
2984+ int spi, int nr_slots)
33002985 {
33012986 int i;
33022987
33032988 for (i = 0; i < nr_slots; i++)
33042989 mark_stack_slot_scratched(env, spi - i);
3305− return 0;
3306−}
3307−
3308−static int mark_dynptr_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
3309−{
3310− int spi;
3311−
3312− /* For CONST_PTR_TO_DYNPTR, it must have already been done by
3313− * check_reg_arg in check_helper_call and mark_btf_func_reg_size in
3314− * check_kfunc_call.
3315− */
3316− if (reg->type == CONST_PTR_TO_DYNPTR)
3317− return 0;
3318− spi = dynptr_get_spi(env, reg);
3319− if (spi < 0)
3320− return spi;
3321− /* Caller ensures dynptr is valid and initialized, which means spi is in
3322− * bounds and spi is the first dynptr slot. Simply mark stack slot as
3323− * read.
3324− */
3325− return mark_stack_slot_obj_read(env, reg, spi, BPF_DYNPTR_NR_SLOTS);
3326−}
3327−
3328−static int mark_iter_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
3329− int spi, int nr_slots)
3330−{
3331− return mark_stack_slot_obj_read(env, reg, spi, nr_slots);
3332−}
3333−
3334−static int mark_irq_flag_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
3335−{
3336− int spi;
3337−
3338− spi = irq_flag_get_spi(env, reg);
3339− if (spi < 0)
3340− return spi;
3341− return mark_stack_slot_obj_read(env, reg, spi, 1);
33422990 }
33432991
33442992 /* This function is supposed to be used by the following 32-bit optimization
@@ -3491,17 +3139,12 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, · 146 unmodified lines
34913139 return __check_reg_arg(env, state->regs, regno, t);
34923140 }
34933141
3494−static int insn_stack_access_flags(int frameno, int spi)
3495−{
3496− return INSN_F_STACK_ACCESS | (spi << INSN_F_SPI_SHIFT) | frameno;
3497−}
3498−
34993142 static void mark_indirect_target(struct bpf_verifier_env *env, int idx)
35003143 {
35013144 env->insn_aux_data[idx].indirect_target = true;
35023145 }
35033146
3504−#define LR_FRAMENO_BITS 3
3147+#define LR_FRAMENO_BITS 4
35053148 #define LR_SPI_BITS 6
35063149 #define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1)
35073150 #define LR_SIZE_BITS 4
@@ -3510,7 +3153,11 @@ static void mark_indirect_target(struct bpf_verifier_env *env, int idx) · 2 unmodified lines
35103153 #define LR_SIZE_MASK ((1ull << LR_SIZE_BITS) - 1)
35113154 #define LR_SPI_OFF LR_FRAMENO_BITS
35123155 #define LR_IS_REG_OFF (LR_SPI_BITS + LR_FRAMENO_BITS)
3513−#define LINKED_REGS_MAX 6
3156+#define LINKED_REGS_MAX 5
3157+
3158+static_assert(MAX_CALL_FRAMES <= (1 << LR_FRAMENO_BITS));
3159+static_assert(LINKED_REGS_MAX < (1 << LR_SIZE_BITS));
3160+static_assert(LINKED_REGS_MAX * LR_ENTRY_BITS + LR_SIZE_BITS <= 64);
35143161
35153162 struct linked_reg {
35163163 u8 frameno;
@@ -3534,10 +3181,11 @@ static struct linked_reg *linked_regs_push(struct linked_regs *s) · 17 unmodified lines
35343181 return NULL;
35353182 }
35363183
3537−/* Use u64 as a vector of 6 10-bit values, use first 4-bits to track
3184+/*
3185+ * Use u64 as a vector of 5 11-bit values, use first 4-bits to track
35383186 * number of elements currently in stack.
3539− * Pack one history entry for linked registers as 10 bits in the following format:
3540− * - 3-bits frameno
3187+ * Pack one history entry for linked registers as 11 bits in the following format:
3188+ * - 4-bits frameno
35413189 * - 6-bits spi_or_reg
35423190 * - 1-bit is_reg
35433191 */
@@ -3733,12 +3381,6 @@ static void assign_scalar_id_before_mov(struct bpf_verifier_env *env, · 189 unmodified lines
37333381 src_reg->id = ++env->id_gen;
37343382 }
37353383
3736−/* Copy src state preserving dst->parent and dst->live fields */
3737−static void copy_register_state(struct bpf_reg_state *dst, const struct bpf_reg_state *src)
3738−{
3739− *dst = *src;
3740−}
3741−
37423384 static void save_register_state(struct bpf_verifier_env *env,
37433385 struct bpf_func_state *state,
37443386 int spi, struct bpf_reg_state *reg,
@@ -3746,7 +3388,7 @@ static void save_register_state(struct bpf_verifier_env *env, · 1 unmodified line
37463388 {
37473389 int i;
37483390
3749− copy_register_state(&state->stack[spi].spilled_ptr, reg);
3391+ state->stack[spi].spilled_ptr = *reg;
37503392
37513393 for (i = BPF_REG_SIZE; i > BPF_REG_SIZE - size; i--)
37523394 state->stack[spi].slot_type[i - 1] = STACK_SPILL;
@@ -3763,7 +3405,7 @@ static bool is_bpf_st_mem(struct bpf_insn *insn) · 10 unmodified lines
37633405
37643406 static int get_reg_width(struct bpf_reg_state *reg)
37653407 {
3766− return fls64(reg->umax_value);
3408+ return fls64(reg_umax(reg));
37673409 }
37683410
37693411 /* See comment for mark_fastcall_pattern_for_call() */
@@ -3816,7 +3458,8 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, · 46 unmodified lines
38163458 int i, slot = -off - 1, spi = slot / BPF_REG_SIZE, err;
38173459 struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
38183460 struct bpf_reg_state *reg = NULL;
3819− int insn_flags = insn_stack_access_flags(state->frameno, spi);
3461+ int insn_flags = INSN_F_STACK_ACCESS;
3462+ int hist_spi = spi, hist_frame = state->frameno;
38203463
38213464 /* caller checked that off % size == 0 and -MAX_BPF_STACK <= off < 0,
38223465 * so it's aligned access and [off, off + size) are within stack limits
@@ -3912,11 +3555,12 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, · 89 unmodified lines
39123555 }
39133556
39143557 if (insn_flags)
3915− return bpf_push_jmp_history(env, env->cur_state, insn_flags, 0);
3558+ return bpf_push_jmp_history(env, env->cur_state, insn_flags,
3559+ hist_spi, hist_frame, 0);
39163560 return 0;
39173561 }
39183562
3919−/* Write the stack: 'stack[ptr_regno + off] = value_regno'. 'ptr_regno' is
3563+/* Write the stack: 'stack[ptr_reg + off] = value_regno'. 'ptr_reg' is
39203564 * known to contain a variable offset.
39213565 * This function checks whether the write is permitted and conservatively
39223566 * tracks the effects of the write, considering that each stack slot in the
@@ -3937,13 +3581,13 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, · 14 unmodified lines
39373581 static int check_stack_write_var_off(struct bpf_verifier_env *env,
39383582 /* func where register points to */
39393583 struct bpf_func_state *state,
3940− int ptr_regno, int off, int size,
3584+ struct bpf_reg_state *ptr_reg, int off, int size,
39413585 int value_regno, int insn_idx)
39423586 {
39433587 struct bpf_func_state *cur; /* state of the current function */
39443588 int min_off, max_off;
39453589 int i, err;
3946− struct bpf_reg_state *ptr_reg = NULL, *value_reg = NULL;
3590+ struct bpf_reg_state *value_reg = NULL;
39473591 struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
39483592 bool writing_zero = false;
39493593 /* set if the fact that we're writing a zero is used to let any
@@ -3952,9 +3596,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env, · 2 unmodified lines
39523596 bool zero_used = false;
39533597
39543598 cur = env->cur_state->frame[env->cur_state->curframe];
3955− ptr_reg = &cur->regs[ptr_regno];
3956− min_off = ptr_reg->smin_value + off;
3957− max_off = ptr_reg->smax_value + off + size;
3599+ min_off = reg_smin(ptr_reg) + off;
3600+ max_off = reg_smax(ptr_reg) + off + size;
39583601 if (value_regno >= 0)
39593602 value_reg = &cur->regs[value_regno];
39603603 if ((value_reg && bpf_register_is_null(value_reg)) ||
@@ -4109,7 +3752,8 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 148 unmodified lines
41093752 int i, slot = -off - 1, spi = slot / BPF_REG_SIZE;
41103753 struct bpf_reg_state *reg;
41113754 u8 *stype, type;
4112− int insn_flags = insn_stack_access_flags(reg_state->frameno, spi);
3755+ int insn_flags = INSN_F_STACK_ACCESS;
3756+ int hist_spi = spi, hist_frame = reg_state->frameno;
41133757
41143758 stype = reg_state->stack[spi].slot_type;
41153759 reg = &reg_state->stack[spi].spilled_ptr;
@@ -4146,7 +3790,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 30 unmodified lines
41463790 * with the destination register on fill.
41473791 */
41483792 assign_scalar_id_before_mov(env, reg);
4149− copy_register_state(&state->regs[dst_regno], reg);
3793+ state->regs[dst_regno] = *reg;
41503794 state->regs[dst_regno].subreg_def = subreg_def;
41513795
41523796 /* Break the relation on a narrowing fill.
@@ -4201,7 +3845,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 48 unmodified lines
42013845 * with the destination register on fill.
42023846 */
42033847 assign_scalar_id_before_mov(env, reg);
4204− copy_register_state(&state->regs[dst_regno], reg);
3848+ state->regs[dst_regno] = *reg;
42053849 /* mark reg as written since spilled pointer state likely
42063850 * has its liveness marks cleared by is_state_visited()
42073851 * which resets stack/reg liveness for state transitions
@@ -4240,7 +3884,8 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 32 unmodified lines
42403884 insn_flags = 0; /* we are not restoring spilled register */
42413885 }
42423886 if (insn_flags)
4243− return bpf_push_jmp_history(env, env->cur_state, insn_flags, 0);
3887+ return bpf_push_jmp_history(env, env->cur_state, insn_flags,
3888+ hist_spi, hist_frame, 0);
42443889 return 0;
42453890 }
42463891
@@ -4249,8 +3894,8 @@ enum bpf_access_src { · 2 unmodified lines
42493894 ACCESS_HELPER = 2, /* the access is performed by a helper */
42503895 };
42513896
4252−static int check_stack_range_initialized(struct bpf_verifier_env *env,
4253− int regno, int off, int access_size,
3897+static int check_stack_range_initialized(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
3898+ argno_t argno, int off, int access_size,
42543899 bool zero_size_allowed,
42553900 enum bpf_access_type type,
42563901 struct bpf_call_arg_meta *meta);
@@ -4260,37 +3905,35 @@ static struct bpf_reg_state *reg_state(struct bpf_verifier_env *env, int regno) · 3 unmodified lines
42603905 return cur_regs(env) + regno;
42613906 }
42623907
4263−/* Read the stack at 'ptr_regno + off' and put the result into the register
3908+/* Read the stack at 'reg + off' and put the result into the register
42643909 * 'dst_regno'.
4265− * 'off' includes the pointer register's fixed offset(i.e. 'ptr_regno.off'),
3910+ * 'off' includes the pointer register's fixed offset(i.e. 'reg->off'),
42663911 * but not its variable offset.
42673912 * 'size' is assumed to be <= reg size and the access is assumed to be aligned.
42683913 *
42693914 * As opposed to check_stack_read_fixed_off, this function doesn't deal with
42703915 * filling registers (i.e. reads of spilled register cannot be detected when
42713916 * the offset is not fixed). We conservatively mark 'dst_regno' as containing
4272− * SCALAR_VALUE. That's why we assert that the 'ptr_regno' has a variable
3917+ * SCALAR_VALUE. That's why we assert that the 'reg' has a variable
42733918 * offset; for a fixed offset check_stack_read_fixed_off should be used
42743919 * instead.
42753920 */
4276−static int check_stack_read_var_off(struct bpf_verifier_env *env,
4277− int ptr_regno, int off, int size, int dst_regno)
3921+static int check_stack_read_var_off(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
3922+ argno_t ptr_argno, int off, int size, int dst_regno)
42783923 {
4279− /* The state of the source register. */
4280− struct bpf_reg_state *reg = reg_state(env, ptr_regno);
42813924 struct bpf_func_state *ptr_state = bpf_func(env, reg);
42823925 int err;
42833926 int min_off, max_off;
42843927
42853928 /* Note that we pass a NULL meta, so raw access will not be permitted.
42863929 */
4287− err = check_stack_range_initialized(env, ptr_regno, off, size,
3930+ err = check_stack_range_initialized(env, reg, ptr_argno, off, size,
42883931 false, BPF_READ, NULL);
42893932 if (err)
42903933 return err;
42913934
4292− min_off = reg->smin_value + off;
4293− max_off = reg->smax_value + off;
3935+ min_off = reg_smin(reg) + off;
3936+ max_off = reg_smax(reg) + off;
42943937 mark_reg_stack_read(env, ptr_state, min_off, max_off + size, dst_regno);
42953938 check_fastcall_stack_contract(env, ptr_state, env->insn_idx, min_off);
42963939 return 0;
@@ -4306,10 +3949,9 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env, · 9 unmodified lines
43063949 * can be -1, meaning that the read value is not going to a register.
43073950 */
43083951 static int check_stack_read(struct bpf_verifier_env *env,
4309− int ptr_regno, int off, int size,
3952+ struct bpf_reg_state *reg, argno_t ptr_argno, int off, int size,
43103953 int dst_regno)
43113954 {
4312− struct bpf_reg_state *reg = reg_state(env, ptr_regno);
43133955 struct bpf_func_state *state = bpf_func(env, reg);
43143956 int err;
43153957 /* Some accesses are only permitted with a static offset. */
@@ -4345,7 +3987,7 @@ static int check_stack_read(struct bpf_verifier_env *env, · 29 unmodified lines
43453987 * than fixed offset ones. Note that dst_regno >= 0 on this
43463988 * branch.
43473989 */
4348− err = check_stack_read_var_off(env, ptr_regno, off, size,
3990+ err = check_stack_read_var_off(env, reg, ptr_argno, off, size,
43493991 dst_regno);
43503992 }
43513993 return err;
@@ -4355,17 +3997,16 @@ static int check_stack_read(struct bpf_verifier_env *env, · 3 unmodified lines
43553997 /* check_stack_write dispatches to check_stack_write_fixed_off or
43563998 * check_stack_write_var_off.
43573999 *
4358− * 'ptr_regno' is the register used as a pointer into the stack.
4000+ * 'reg' is the register used as a pointer into the stack.
43594001 * 'value_regno' is the register whose value we're writing to the stack. It can
43604002 * be -1, meaning that we're not writing from a register.
43614003 *
43624004 * The caller must ensure that the offset falls within the maximum stack size.
43634005 */
43644006 static int check_stack_write(struct bpf_verifier_env *env,
4365− int ptr_regno, int off, int size,
4007+ struct bpf_reg_state *reg, int off, int size,
43664008 int value_regno, int insn_idx)
43674009 {
4368− struct bpf_reg_state *reg = reg_state(env, ptr_regno);
43694010 struct bpf_func_state *state = bpf_func(env, reg);
43704011 int err;
43714012
@@ -4378,28 +4019,135 @@ static int check_stack_write(struct bpf_verifier_env *env, · 6 unmodified lines
43784019 * than fixed offset ones.
43794020 */
43804021 err = check_stack_write_var_off(env, state,
4381− ptr_regno, off, size,
4022+ reg, off, size,
43824023 value_regno, insn_idx);
43834024 }
43844025 return err;
43854026 }
43864027
4387−static int check_map_access_type(struct bpf_verifier_env *env, u32 regno,
4028+/*
4029+ * Write a value to the outgoing stack arg area.
4030+ * off is a negative offset from r11 (e.g. -8 for arg6, -16 for arg7).
4031+ */
4032+static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state,
4033+ int off, struct bpf_reg_state *value_reg)
4034+{
4035+ int max_stack_arg_regs = MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS;
4036+ struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno];
4037+ int spi = -off / BPF_REG_SIZE - 1;
4038+ struct bpf_reg_state *arg;
4039+ int err;
4040+
4041+ if (spi >= max_stack_arg_regs) {
4042+ verbose(env, "stack arg write offset %d exceeds max %d stack args\n",
4043+ off, max_stack_arg_regs);
4044+ return -EINVAL;
4045+ }
4046+
4047+ err = grow_stack_arg_slots(env, state, spi + 1);
4048+ if (err)
4049+ return err;
4050+
4051+ /* Track the max outgoing stack arg slot count. */
4052+ if (spi + 1 > subprog->max_out_stack_arg_cnt)
4053+ subprog->max_out_stack_arg_cnt = spi + 1;
4054+
4055+ if (value_reg) {
4056+ state->stack_arg_regs[spi] = *value_reg;
4057+ } else {
4058+ /* BPF_ST: store immediate, treat as scalar */
4059+ arg = &state->stack_arg_regs[spi];
4060+ arg->type = SCALAR_VALUE;
4061+ __mark_reg_known(arg, env->prog->insnsi[env->insn_idx].imm);
4062+ }
4063+ state->no_stack_arg_load = true;
4064+ return bpf_push_jmp_history(env, env->cur_state,
4065+ INSN_F_STACK_ARG_ACCESS, spi, 0, 0);
4066+}
4067+
4068+/*
4069+ * Read a value from the incoming stack arg area.
4070+ * off is a positive offset from r11 (e.g. +8 for arg6, +16 for arg7).
4071+ */
4072+static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_state *state,
4073+ int off, int dst_regno)
4074+{
4075+ struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno];
4076+ struct bpf_verifier_state *vstate = env->cur_state;
4077+ int spi = off / BPF_REG_SIZE - 1;
4078+ struct bpf_func_state *caller, *cur;
4079+ struct bpf_reg_state *arg;
4080+
4081+ if (state->no_stack_arg_load) {
4082+ verbose(env, "r11 load must be before any r11 store or call insn\n");
4083+ return -EINVAL;
4084+ }
4085+
4086+ if (spi + 1 > bpf_in_stack_arg_cnt(subprog)) {
4087+ verbose(env, "invalid read from stack arg off %d depth %d\n",
4088+ off, bpf_in_stack_arg_cnt(subprog) * BPF_REG_SIZE);
4089+ return -EACCES;
4090+ }
4091+
4092+ caller = vstate->frame[vstate->curframe - 1];
4093+ arg = &caller->stack_arg_regs[spi];
4094+ cur = vstate->frame[vstate->curframe];
4095+ cur->regs[dst_regno] = *arg;
4096+ return bpf_push_jmp_history(env, env->cur_state,
4097+ INSN_F_STACK_ARG_ACCESS, spi, 0, 0);
4098+}
4099+
4100+static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx)
4101+{
4102+ struct bpf_func_state *caller = cur_func(env);
4103+ int spi = arg_idx - MAX_BPF_FUNC_REG_ARGS;
4104+
4105+ bt_set_frame_stack_arg_slot(&env->bt, caller->frameno, spi);
4106+ return mark_chain_precision_batch(env, env->cur_state);
4107+}
4108+
4109+static int check_outgoing_stack_args(struct bpf_verifier_env *env, struct bpf_func_state *caller,
4110+ int nargs)
4111+{
4112+ int i, spi;
4113+
4114+ for (i = MAX_BPF_FUNC_REG_ARGS; i < nargs; i++) {
4115+ spi = i - MAX_BPF_FUNC_REG_ARGS;
4116+ if (spi >= caller->out_stack_arg_cnt ||
4117+ caller->stack_arg_regs[spi].type == NOT_INIT) {
4118+ verbose(env, "callee expects %d args, stack arg%d is not initialized\n",
4119+ nargs, spi + 1);
4120+ return -EFAULT;
4121+ }
4122+ }
4123+
4124+ return 0;
4125+}
4126+
4127+static struct bpf_reg_state *get_func_arg_reg(struct bpf_func_state *caller,
4128+ struct bpf_reg_state *regs, int arg)
4129+{
4130+ if (arg < MAX_BPF_FUNC_REG_ARGS)
4131+ return &regs[arg + 1];
4132+
4133+ return &caller->stack_arg_regs[arg - MAX_BPF_FUNC_REG_ARGS];
4134+}
4135+
4136+static int check_map_access_type(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
43884137 int off, int size, enum bpf_access_type type)
43894138 {
4390− struct bpf_reg_state *reg = reg_state(env, regno);
43914139 struct bpf_map *map = reg->map_ptr;
43924140 u32 cap = bpf_map_flags_to_cap(map);
43934141
43944142 if (type == BPF_WRITE && !(cap & BPF_MAP_CAN_WRITE)) {
43954143 verbose(env, "write into map forbidden, value_size=%d off=%lld size=%d\n",
4396− map->value_size, reg->smin_value + off, size);
4144+ map->value_size, reg_smin(reg) + off, size);
43974145 return -EACCES;
43984146 }
43994147
44004148 if (type == BPF_READ && !(cap & BPF_MAP_CAN_READ)) {
44014149 verbose(env, "read from map forbidden, value_size=%d off=%lld size=%d\n",
4402− map->value_size, reg->smin_value + off, size);
4150+ map->value_size, reg_smin(reg) + off, size);
44034151 return -EACCES;
44044152 }
44054153
@@ -4407,17 +4155,15 @@ static int check_map_access_type(struct bpf_verifier_env *env, u32 regno, · 1 unmodified line
44074155 }
44084156
44094157 /* check read/write into memory region (e.g., map value, ringbuf sample, etc) */
4410−static int __check_mem_access(struct bpf_verifier_env *env, int regno,
4158+static int __check_mem_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
44114159 int off, int size, u32 mem_size,
44124160 bool zero_size_allowed)
44134161 {
44144162 bool size_ok = size > 0 || (size == 0 && zero_size_allowed);
4415− struct bpf_reg_state *reg;
44164163
44174164 if (off >= 0 && size_ok && (u64)off + size <= mem_size)
44184165 return 0;
44194166
4420− reg = &cur_regs(env)[regno];
44214167 switch (reg->type) {
44224168 case PTR_TO_MAP_KEY:
44234169 verbose(env, "invalid access to map key, key_size=%d off=%d size=%d\n",
@@ -4430,8 +4176,8 @@ static int __check_mem_access(struct bpf_verifier_env *env, int regno, · 6 unmodified lines
44304176 case PTR_TO_PACKET:
44314177 case PTR_TO_PACKET_META:
44324178 case PTR_TO_PACKET_END:
4433− verbose(env, "invalid access to packet, off=%d size=%d, R%d(id=%d,off=%d,r=%d)\n",
4434− off, size, regno, reg->id, off, mem_size);
4179+ verbose(env, "invalid access to packet, off=%d size=%d, %s(id=%d,off=%d,r=%d)\n",
4180+ off, size, reg_arg_name(env, argno), reg->id, off, mem_size);
44354181 break;
44364182 case PTR_TO_CTX:
44374183 verbose(env, "invalid access to context, ctx_size=%d off=%d size=%d\n",
@@ -4447,13 +4193,10 @@ static int __check_mem_access(struct bpf_verifier_env *env, int regno, · 9 unmodified lines
44474193 }
44484194
44494195 /* check read/write into a memory region with possible variable offset */
4450−static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno,
4196+static int check_mem_region_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
44514197 int off, int size, u32 mem_size,
44524198 bool zero_size_allowed)
44534199 {
4454− struct bpf_verifier_state *vstate = env->cur_state;
4455− struct bpf_func_state *state = vstate->frame[vstate->curframe];
4456− struct bpf_reg_state *reg = &state->regs[regno];
44574200 int err;
44584201
44594202 /* We may have adjusted the register pointing to memory region, so we
@@ -4466,36 +4209,36 @@ static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, · 6 unmodified lines
44664209 * index'es we need to make sure that whatever we use
44674210 * will have a set floor within our range.
44684211 */
4469− if (reg->smin_value < 0 &&
4470− (reg->smin_value == S64_MIN ||
4471− (off + reg->smin_value != (s64)(s32)(off + reg->smin_value)) ||
4472− reg->smin_value + off < 0)) {
4473− verbose(env, "R%d min value is negative, either use unsigned index or do a if (index >=0) check.\n",
4474− regno);
4212+ if (reg_smin(reg) < 0 &&
4213+ (reg_smin(reg) == S64_MIN ||
4214+ (off + reg_smin(reg) != (s64)(s32)(off + reg_smin(reg))) ||
4215+ reg_smin(reg) + off < 0)) {
4216+ verbose(env, "%s min value is negative, either use unsigned index or do a if (index >=0) check.\n",
4217+ reg_arg_name(env, argno));
44754218 return -EACCES;
44764219 }
4477− err = __check_mem_access(env, regno, reg->smin_value + off, size,
4220+ err = __check_mem_access(env, reg, argno, reg_smin(reg) + off, size,
44784221 mem_size, zero_size_allowed);
44794222 if (err) {
4480− verbose(env, "R%d min value is outside of the allowed memory range\n",
4481− regno);
4223+ verbose(env, "%s min value is outside of the allowed memory range\n",
4224+ reg_arg_name(env, argno));
44824225 return err;
44834226 }
44844227
44854228 /* If we haven't set a max value then we need to bail since we can't be
44864229 * sure we won't do bad things.
4487− * If reg->umax_value + off could overflow, treat that as unbounded too.
4230+ * If reg_umax(reg) + off could overflow, treat that as unbounded too.
44884231 */
4489− if (reg->umax_value >= BPF_MAX_VAR_OFF) {
4490− verbose(env, "R%d unbounded memory access, make sure to bounds check any such access\n",
4491− regno);
4232+ if (reg_umax(reg) >= BPF_MAX_VAR_OFF) {
4233+ verbose(env, "%s unbounded memory access, make sure to bounds check any such access\n",
4234+ reg_arg_name(env, argno));
44924235 return -EACCES;
44934236 }
4494− err = __check_mem_access(env, regno, reg->umax_value + off, size,
4237+ err = __check_mem_access(env, reg, argno, reg_umax(reg) + off, size,
44954238 mem_size, zero_size_allowed);
44964239 if (err) {
4497− verbose(env, "R%d max value is outside of the allowed memory range\n",
4498− regno);
4240+ verbose(env, "%s max value is outside of the allowed memory range\n",
4241+ reg_arg_name(env, argno));
44994242 return err;
45004243 }
45014244
@@ -4503,7 +4246,7 @@ static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, · 1 unmodified line
45034246 }
45044247
45054248 static int __check_ptr_off_reg(struct bpf_verifier_env *env,
4506− const struct bpf_reg_state *reg, int regno,
4249+ const struct bpf_reg_state *reg, argno_t argno,
45074250 bool fixed_off_ok)
45084251 {
45094252 /* Access to this pointer-typed register or passing it to a helper
@@ -4519,15 +4262,15 @@ static int __check_ptr_off_reg(struct bpf_verifier_env *env, · 9 unmodified lines
45194262 return -EACCES;
45204263 }
45214264
4522− if (reg->smin_value < 0) {
4523− verbose(env, "negative offset %s ptr R%d off=%lld disallowed\n",
4524− reg_type_str(env, reg->type), regno, reg->var_off.value);
4265+ if (reg_smin(reg) < 0) {
4266+ verbose(env, "negative offset %s ptr %s off=%lld disallowed\n",
4267+ reg_type_str(env, reg->type), reg_arg_name(env, argno), reg->var_off.value);
45254268 return -EACCES;
45264269 }
45274270
45284271 if (!fixed_off_ok && reg->var_off.value != 0) {
4529− verbose(env, "dereference of modified %s ptr R%d off=%lld disallowed\n",
4530− reg_type_str(env, reg->type), regno, reg->var_off.value);
4272+ verbose(env, "dereference of modified %s ptr %s off=%lld disallowed\n",
4273+ reg_type_str(env, reg->type), reg_arg_name(env, argno), reg->var_off.value);
45314274 return -EACCES;
45324275 }
45334276
@@ -4537,7 +4280,7 @@ static int __check_ptr_off_reg(struct bpf_verifier_env *env, · 3 unmodified lines
45374280 static int check_ptr_off_reg(struct bpf_verifier_env *env,
45384281 const struct bpf_reg_state *reg, int regno)
45394282 {
4540− return __check_ptr_off_reg(env, reg, regno, false);
4283+ return __check_ptr_off_reg(env, reg, argno_from_reg(regno), false);
45414284 }
45424285
45434286 static int map_kptr_match_type(struct bpf_verifier_env *env,
@@ -4573,9 +4316,9 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, · 29 unmodified lines
45734316 * referenced PTR_TO_BTF_ID, and that its fixed offset is 0. For the
45744317 * normal store of unreferenced kptr, we must ensure var_off is zero.
45754318 * Since ref_ptr cannot be accessed directly by BPF insns, check for
4576− * reg->ref_obj_id is not needed here.
4319+ * reg->id is not needed here.
45774320 */
4578− if (__check_ptr_off_reg(env, reg, regno, true))
4321+ if (__check_ptr_off_reg(env, reg, argno_from_reg(regno), true))
45794322 return -EACCES;
45804323
45814324 /* A full type match is needed, as BTF can be vmlinux, module or prog BTF, and
@@ -4718,7 +4461,7 @@ static int mark_uptr_ld_reg(struct bpf_verifier_env *env, u32 regno, · 136 unmodified lines
47184461 return 0;
47194462 }
47204463
4721−static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno,
4464+static int check_map_kptr_access(struct bpf_verifier_env *env,
47224465 int value_regno, int insn_idx,
47234466 struct btf_field *kptr_field)
47244467 {
@@ -4795,19 +4538,16 @@ static u32 map_mem_size(const struct bpf_map *map) · 70 unmodified lines
47954538 }
47964539
47974540 /* check read/write into a map element with possible variable offset */
4798−static int check_map_access(struct bpf_verifier_env *env, u32 regno,
4541+static int check_map_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
47994542 int off, int size, bool zero_size_allowed,
48004543 enum bpf_access_src src)
48014544 {
4802− struct bpf_verifier_state *vstate = env->cur_state;
4803− struct bpf_func_state *state = vstate->frame[vstate->curframe];
4804− struct bpf_reg_state *reg = &state->regs[regno];
48054545 struct bpf_map *map = reg->map_ptr;
48064546 u32 mem_size = map_mem_size(map);
48074547 struct btf_record *rec;
48084548 int err, i;
48094549
4810− err = check_mem_region_access(env, regno, off, size, mem_size, zero_size_allowed);
4550+ err = check_mem_region_access(env, reg, argno, off, size, mem_size, zero_size_allowed);
48114551 if (err)
48124552 return err;
48134553
@@ -4822,8 +4562,8 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 8 unmodified lines
48224562 * this program. To check that [x1, x2) overlaps with [y1, y2),
48234563 * it is sufficient to check x1 < y2 && y1 < x2.
48244564 */
4825− if (reg->smin_value + off < p + field->size &&
4826− p < reg->umax_value + off + size) {
4565+ if (reg_smin(reg) + off < p + field->size &&
4566+ p < reg_umax(reg) + off + size) {
48274567 switch (field->type) {
48284568 case BPF_KPTR_UNREF:
48294569 case BPF_KPTR_REF:
@@ -4903,30 +4643,29 @@ static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, · 73 unmodified lines
49034643 }
49044644 }
49054645
4906−static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off,
4646+static int check_packet_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int off,
49074647 int size, bool zero_size_allowed)
49084648 {
4909− struct bpf_reg_state *reg = reg_state(env, regno);
49104649 int err;
49114650
49124651 if (reg->range < 0) {
4913− verbose(env, "R%d offset is outside of the packet\n", regno);
4652+ verbose(env, "%s offset is outside of the packet\n", reg_arg_name(env, argno));
49144653 return -EINVAL;
49154654 }
49164655
4917− err = check_mem_region_access(env, regno, off, size, reg->range, zero_size_allowed);
4656+ err = check_mem_region_access(env, reg, argno, off, size, reg->range, zero_size_allowed);
49184657 if (err)
49194658 return err;
49204659
49214660 /* __check_mem_access has made sure "off + size - 1" is within u16.
4922− * reg->umax_value can't be bigger than MAX_PACKET_OFF which is 0xffff,
4661+ * reg_umax(reg) can't be bigger than MAX_PACKET_OFF which is 0xffff,
49234662 * otherwise find_good_pkt_pointers would have refused to set range info
49244663 * that __check_mem_access would have rejected this pkt access.
4925− * Therefore, "off + reg->umax_value + size - 1" won't overflow u32.
4664+ * Therefore, "off + reg_umax(reg) + size - 1" won't overflow u32.
49264665 */
49274666 env->prog->aux->max_pkt_offset =
49284667 max_t(u32, env->prog->aux->max_pkt_offset,
4929− off + reg->umax_value + size - 1);
4668+ off + reg_umax(reg) + size - 1);
49304669
49314670 return 0;
49324671 }
@@ -4950,8 +4689,8 @@ static int __check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int of · 17 unmodified lines
49504689 * type of narrower access.
49514690 */
49524691 if (base_type(info->reg_type) == PTR_TO_BTF_ID) {
4953− if (info->ref_obj_id &&
4954− !find_reference_state(env->cur_state, info->ref_obj_id)) {
4692+ if (info->ref_id &&
4693+ !find_reference_state(env->cur_state, info->ref_id)) {
49554694 verbose(env, "invalid bpf_context access off=%d. Reference may already be released\n",
49564695 off);
49574696 return -EACCES;
@@ -4969,7 +4708,7 @@ static int __check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int of · 11 unmodified lines
49694708 return -EACCES;
49704709 }
49714710
4972−static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regno,
4711+static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, struct bpf_reg_state *reg, argno_t argno,
49734712 int off, int access_size, enum bpf_access_type t,
49744713 struct bpf_insn_access_aux *info)
49754714 {
@@ -4979,17 +4718,15 @@ static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 3 unmodified lines
49794718 */
49804719 bool var_off_ok = is_var_ctx_off_allowed(env->prog);
49814720 bool fixed_off_ok = !env->ops->convert_ctx_access;
4982− struct bpf_reg_state *regs = cur_regs(env);
4983− struct bpf_reg_state *reg = regs + regno;
49844721 int err;
49854722
49864723 if (var_off_ok)
4987− err = check_mem_region_access(env, regno, off, access_size, U16_MAX, false);
4724+ err = check_mem_region_access(env, reg, argno, off, access_size, U16_MAX, false);
49884725 else
4989− err = __check_ptr_off_reg(env, reg, regno, fixed_off_ok);
4726+ err = __check_ptr_off_reg(env, reg, argno, fixed_off_ok);
49904727 if (err)
49914728 return err;
4992− off += reg->umax_value;
4729+ off += reg_umax(reg);
49934730
49944731 err = __check_ctx_access(env, insn_idx, off, access_size, t, info);
49954732 if (err)
@@ -4997,9 +4734,21 @@ static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 1 unmodified line
49974734 return err;
49984735 }
49994736
5000−static int check_flow_keys_access(struct bpf_verifier_env *env, int off,
5001− int size)
4737+static int check_flow_keys_access(struct bpf_verifier_env *env,
4738+ struct bpf_reg_state *reg, argno_t argno,
4739+ int off, int size)
50024740 {
4741+ /* Only a constant offset is allowed here; fold it into off. */
4742+ if (!tnum_is_const(reg->var_off)) {
4743+ char tn_buf[48];
4744+
4745+ tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
4746+ verbose(env, "%s invalid variable offset to flow keys: off=%d, var_off=%s\n",
4747+ reg_arg_name(env, argno), off, tn_buf);
4748+ return -EACCES;
4749+ }
4750+ off += reg->var_off.value;
4751+
50034752 if (size < 0 || off < 0 ||
50044753 (u64)off + size > sizeof(struct bpf_flow_keys)) {
50054754 verbose(env, "invalid access to flow keys off=%d size=%d\n",
@@ -5010,16 +4759,15 @@ static int check_flow_keys_access(struct bpf_verifier_env *env, int off, · 4 unmodified lines
50104759 }
50114760
50124761 static int check_sock_access(struct bpf_verifier_env *env, int insn_idx,
5013− u32 regno, int off, int size,
4762+ struct bpf_reg_state *reg, argno_t argno, int off, int size,
50144763 enum bpf_access_type t)
50154764 {
5016− struct bpf_reg_state *reg = reg_state(env, regno);
50174765 struct bpf_insn_access_aux info = {};
50184766 bool valid;
50194767
5020− if (reg->smin_value < 0) {
5021− verbose(env, "R%d min value is negative, either use unsigned index or do a if (index >=0) check.\n",
5022− regno);
4768+ if (reg_smin(reg) < 0) {
4769+ verbose(env, "%s min value is negative, either use unsigned index or do a if (index >=0) check.\n",
4770+ reg_arg_name(env, argno));
50234771 return -EACCES;
50244772 }
50254773
@@ -5047,8 +4795,8 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx, · 21 unmodified lines
50474795 return 0;
50484796 }
50494797
5050− verbose(env, "R%d invalid %s access off=%d size=%d\n",
5051− regno, reg_type_str(env, reg->type), off, size);
4798+ verbose(env, "%s invalid %s access off=%d size=%d\n",
4799+ reg_arg_name(env, argno), reg_type_str(env, reg->type), off, size);
50524800
50534801 return -EACCES;
50544802 }
@@ -5123,10 +4871,10 @@ static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = { · 68 unmodified lines
51234871 [CONST_PTR_TO_MAP] = btf_bpf_map_id,
51244872 };
51254873
5126−static bool is_trusted_reg(const struct bpf_reg_state *reg)
4874+static bool is_trusted_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg)
51274875 {
51284876 /* A referenced register is always trusted. */
5129− if (reg->ref_obj_id)
4877+ if (reg_is_referenced(env, reg))
51304878 return true;
51314879
51324880 /* Types listed in the reg2btf_ids are always trusted */
@@ -5368,7 +5116,10 @@ process_func: · 235 unmodified lines
53685116 }
53695117
53705118 subprog_depth = round_up_stack_depth(env, subprog[idx].stack_depth);
5371− if (priv_stack_supported) {
5119+ if (IS_ENABLED(CONFIG_X86_64) && subprog[idx].stack_arg_cnt) {
5120+ /* x86-64 uses R9 for both private stack frame pointer and arg6. */
5121+ subprog[idx].priv_stack_mode = NO_PRIV_STACK;
5122+ } else if (priv_stack_supported) {
53725123 /* Request private stack support only if the subprog stack
53735124 * depth is no less than BPF_PRIV_STACK_MIN_SIZE. This is to
53745125 * avoid jit penalty if the stack usage is small.
@@ -5379,6 +5130,8 @@ process_func: · 4 unmodified lines
53795130 }
53805131
53815132 if (subprog[idx].priv_stack_mode == PRIV_STACK_ADAPTIVE) {
5133+ if (subprog_depth > env->max_stack_depth)
5134+ env->max_stack_depth = subprog_depth;
53825135 if (subprog_depth > MAX_BPF_STACK) {
53835136 verbose(env, "stack size of subprog %d is %d. Too large\n",
53845137 idx, subprog_depth);
@@ -5386,6 +5139,8 @@ process_func: · 1 unmodified line
53865139 }
53875140 } else {
53885141 depth += subprog_depth;
5142+ if (depth > env->max_stack_depth)
5143+ env->max_stack_depth = depth;
53895144 if (depth > MAX_BPF_STACK) {
53905145 total = 0;
53915146 for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller)
@@ -5472,14 +5227,23 @@ continue_func: · 80 unmodified lines
54725227 * this info will be utilized by JIT so that we will be preserving the
54735228 * tail call counter throughout bpf2bpf calls combined with tailcalls
54745229 */
5475− if (tail_call_reachable)
5230+ if (tail_call_reachable) {
54765231 for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {
54775232 if (subprog[tmp].is_exception_cb) {
54785233 verbose(env, "cannot tail call within exception cb\n");
54795234 return -EINVAL;
54805235 }
5236+ if (subprog[tmp].stack_arg_cnt) {
5237+ verbose(env, "tail_calls are not allowed in programs with stack args\n");
5238+ return -EINVAL;
5239+ }
54815240 subprog[tmp].tail_call_reachable = true;
54825241 }
5242+ } else if (!idx && subprog[0].has_tail_call && subprog[0].stack_arg_cnt) {
5243+ verbose(env, "tail_calls are not allowed in programs with stack args\n");
5244+ return -EINVAL;
5245+ }
5246+
54835247 if (subprog[0].tail_call_reachable)
54845248 env->prog->aux->tail_call_reachable = true;
54855249
@@ -5498,6 +5262,9 @@ continue_func: · 12 unmodified lines
54985262 frame = dinfo[idx].frame;
54995263 i = dinfo[idx].ret_insn;
55005264
5265+ /* reset tail_call_reachable to the parent's actual state */
5266+ tail_call_reachable = subprog[idx].tail_call_reachable;
5267+
55015268 goto continue_func;
55025269 }
55035270
@@ -5558,12 +5325,12 @@ static int check_max_stack_depth(struct bpf_verifier_env *env) · 54 unmodified lines
55585325 static int __check_buffer_access(struct bpf_verifier_env *env,
55595326 const char *buf_info,
55605327 const struct bpf_reg_state *reg,
5561− int regno, int off, int size)
5328+ argno_t argno, int off, int size)
55625329 {
55635330 if (off < 0) {
55645331 verbose(env,
5565− "R%d invalid %s buffer access: off=%d, size=%d\n",
5566− regno, buf_info, off, size);
5332+ "%s invalid %s buffer access: off=%d, size=%d\n",
5333+ reg_arg_name(env, argno), buf_info, off, size);
55675334 return -EACCES;
55685335 }
55695336 if (!tnum_is_const(reg->var_off)) {
@@ -5571,8 +5338,8 @@ static int __check_buffer_access(struct bpf_verifier_env *env, · 1 unmodified line
55715338
55725339 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
55735340 verbose(env,
5574− "R%d invalid variable buffer offset: off=%d, var_off=%s\n",
5575− regno, off, tn_buf);
5341+ "%s invalid variable buffer offset: off=%d, var_off=%s\n",
5342+ reg_arg_name(env, argno), off, tn_buf);
55765343 return -EACCES;
55775344 }
55785345
@@ -5581,11 +5348,11 @@ static int __check_buffer_access(struct bpf_verifier_env *env, · 2 unmodified lines
55815348
55825349 static int check_tp_buffer_access(struct bpf_verifier_env *env,
55835350 const struct bpf_reg_state *reg,
5584− int regno, int off, int size)
5351+ argno_t argno, int off, int size)
55855352 {
55865353 int err;
55875354
5588− err = __check_buffer_access(env, "tracepoint", reg, regno, off, size);
5355+ err = __check_buffer_access(env, "tracepoint", reg, argno, off, size);
55895356 if (err)
55905357 return err;
55915358
@@ -5597,14 +5364,14 @@ static int check_tp_buffer_access(struct bpf_verifier_env *env, · 5 unmodified lines
55975364
55985365 static int check_buffer_access(struct bpf_verifier_env *env,
55995366 const struct bpf_reg_state *reg,
5600− int regno, int off, int size,
5367+ argno_t argno, int off, int size,
56015368 bool zero_size_allowed,
56025369 u32 *max_access)
56035370 {
56045371 const char *buf_info = type_is_rdonly_mem(reg->type) ? "rdonly" : "rdwr";
56055372 int err;
56065373
5607− err = __check_buffer_access(env, buf_info, reg, regno, off, size);
5374+ err = __check_buffer_access(env, buf_info, reg, argno, off, size);
56085375 if (err)
56095376 return err;
56105377
@@ -5617,7 +5384,7 @@ static int check_buffer_access(struct bpf_verifier_env *env, · 6 unmodified lines
56175384 static void zext_32_to_64(struct bpf_reg_state *reg)
56185385 {
56195386 reg->var_off = tnum_subreg(reg->var_off);
5620− __reg_assign_32_into_64(reg);
5387+ reg_set_urange64(reg, reg_u32_min(reg), reg_u32_max(reg));
56215388 }
56225389
56235390 /* truncate register to smaller size (in bytes)
@@ -5632,15 +5399,10 @@ static void coerce_reg_to_size(struct bpf_reg_state *reg, int size) · 8 unmodified lines
56325399
56335400 /* fix arithmetic bounds */
56345401 mask = ((u64)1 << (size * 8)) - 1;
5635− if ((reg->umin_value & ~mask) == (reg->umax_value & ~mask)) {
5636− reg->umin_value &= mask;
5637− reg->umax_value &= mask;
5638− } else {
5639− reg->umin_value = 0;
5640− reg->umax_value = mask;
5641− }
5642− reg->smin_value = reg->umin_value;
5643− reg->smax_value = reg->umax_value;
5402+ if ((reg_umin(reg) & ~mask) == (reg_umax(reg) & ~mask))
5403+ reg_set_urange64(reg, reg_umin(reg) & mask, reg_umax(reg) & mask);
5404+ else
5405+ reg_set_urange64(reg, 0, mask);
56445406
56455407 /* If size is smaller than 32bit register the 32bit register
56465408 * values are also truncated so we push 64-bit bounds into
@@ -5655,19 +5417,16 @@ static void coerce_reg_to_size(struct bpf_reg_state *reg, int size) · 8 unmodified lines
56555417 static void set_sext64_default_val(struct bpf_reg_state *reg, int size)
56565418 {
56575419 if (size == 1) {
5658− reg->smin_value = reg->s32_min_value = S8_MIN;
5659− reg->smax_value = reg->s32_max_value = S8_MAX;
5420+ reg_set_srange64(reg, S8_MIN, S8_MAX);
5421+ reg_set_srange32(reg, S8_MIN, S8_MAX);
56605422 } else if (size == 2) {
5661− reg->smin_value = reg->s32_min_value = S16_MIN;
5662− reg->smax_value = reg->s32_max_value = S16_MAX;
5423+ reg_set_srange64(reg, S16_MIN, S16_MAX);
5424+ reg_set_srange32(reg, S16_MIN, S16_MAX);
56635425 } else {
56645426 /* size == 4 */
5665− reg->smin_value = reg->s32_min_value = S32_MIN;
5666− reg->smax_value = reg->s32_max_value = S32_MAX;
5427+ reg_set_srange64(reg, S32_MIN, S32_MAX);
5428+ reg_set_srange32(reg, S32_MIN, S32_MAX);
56675429 }
5668− reg->umin_value = reg->u32_min_value = 0;
5669− reg->umax_value = U64_MAX;
5670− reg->u32_max_value = U32_MAX;
56715430 reg->var_off = tnum_unknown;
56725431 }
56735432
@@ -5688,29 +5447,27 @@ static void coerce_reg_to_size_sx(struct bpf_reg_state *reg, int size) · 14 unmodified lines
56885447 reg->var_off = tnum_const((s32)u64_cval);
56895448
56905449 u64_cval = reg->var_off.value;
5691− reg->smax_value = reg->smin_value = u64_cval;
5692− reg->umax_value = reg->umin_value = u64_cval;
5693− reg->s32_max_value = reg->s32_min_value = u64_cval;
5694− reg->u32_max_value = reg->u32_min_value = u64_cval;
5450+ reg->r64 = cnum64_from_urange(u64_cval, u64_cval);
5451+ reg->r32 = cnum32_from_urange((u32)u64_cval, (u32)u64_cval);
56955452 return;
56965453 }
56975454
5698− top_smax_value = ((u64)reg->smax_value >> num_bits) << num_bits;
5699− top_smin_value = ((u64)reg->smin_value >> num_bits) << num_bits;
5455+ top_smax_value = ((u64)reg_smax(reg) >> num_bits) << num_bits;
5456+ top_smin_value = ((u64)reg_smin(reg) >> num_bits) << num_bits;
57005457
57015458 if (top_smax_value != top_smin_value)
57025459 goto out;
57035460
57045461 /* find the s64_min and s64_min after sign extension */
57055462 if (size == 1) {
5706− init_s64_max = (s8)reg->smax_value;
5707− init_s64_min = (s8)reg->smin_value;
5463+ init_s64_max = (s8)reg_smax(reg);
5464+ init_s64_min = (s8)reg_smin(reg);
57085465 } else if (size == 2) {
5709− init_s64_max = (s16)reg->smax_value;
5710− init_s64_min = (s16)reg->smin_value;
5466+ init_s64_max = (s16)reg_smax(reg);
5467+ init_s64_min = (s16)reg_smin(reg);
57115468 } else {
5712− init_s64_max = (s32)reg->smax_value;
5713− init_s64_min = (s32)reg->smin_value;
5469+ init_s64_max = (s32)reg_smax(reg);
5470+ init_s64_min = (s32)reg_smin(reg);
57145471 }
57155472
57165473 s64_max = max(init_s64_max, init_s64_min);
@@ -5718,10 +5475,8 @@ static void coerce_reg_to_size_sx(struct bpf_reg_state *reg, int size) · 1 unmodified line
57185475
57195476 /* both of s64_max/s64_min positive or negative */
57205477 if ((s64_max >= 0) == (s64_min >= 0)) {
5721− reg->s32_min_value = reg->smin_value = s64_min;
5722− reg->s32_max_value = reg->smax_value = s64_max;
5723− reg->u32_min_value = reg->umin_value = s64_min;
5724− reg->u32_max_value = reg->umax_value = s64_max;
5478+ reg_set_srange64(reg, s64_min, s64_max);
5479+ reg_set_srange32(reg, s64_min, s64_max);
57255480 reg->var_off = tnum_range(s64_min, s64_max);
57265481 return;
57275482 }
@@ -5732,16 +5487,11 @@ out: · 4 unmodified lines
57325487
57335488 static void set_sext32_default_val(struct bpf_reg_state *reg, int size)
57345489 {
5735− if (size == 1) {
5736− reg->s32_min_value = S8_MIN;
5737− reg->s32_max_value = S8_MAX;
5738− } else {
5490+ if (size == 1)
5491+ reg_set_srange32(reg, S8_MIN, S8_MAX);
5492+ else
57395493 /* size == 2 */
5740− reg->s32_min_value = S16_MIN;
5741− reg->s32_max_value = S16_MAX;
5742− }
5743− reg->u32_min_value = 0;
5744− reg->u32_max_value = U32_MAX;
5494+ reg_set_srange32(reg, S16_MIN, S16_MAX);
57455495 reg->var_off = tnum_subreg(tnum_unknown);
57465496 }
57475497
@@ -5759,34 +5509,30 @@ static void coerce_subreg_to_size_sx(struct bpf_reg_state *reg, int size) · 11 unmodified lines
57595509 reg->var_off = tnum_const((s16)u32_val);
57605510
57615511 u32_val = reg->var_off.value;
5762− reg->s32_min_value = reg->s32_max_value = u32_val;
5763− reg->u32_min_value = reg->u32_max_value = u32_val;
5512+ reg_set_srange32(reg, u32_val, u32_val);
57645513 return;
57655514 }
57665515
5767− top_smax_value = ((u32)reg->s32_max_value >> num_bits) << num_bits;
5768− top_smin_value = ((u32)reg->s32_min_value >> num_bits) << num_bits;
5516+ top_smax_value = ((u32)reg_s32_max(reg) >> num_bits) << num_bits;
5517+ top_smin_value = ((u32)reg_s32_min(reg) >> num_bits) << num_bits;
57695518
57705519 if (top_smax_value != top_smin_value)
57715520 goto out;
57725521
57735522 /* find the s32_min and s32_min after sign extension */
57745523 if (size == 1) {
5775− init_s32_max = (s8)reg->s32_max_value;
5776− init_s32_min = (s8)reg->s32_min_value;
5524+ init_s32_max = (s8)reg_s32_max(reg);
5525+ init_s32_min = (s8)reg_s32_min(reg);
57775526 } else {
57785527 /* size == 2 */
5779− init_s32_max = (s16)reg->s32_max_value;
5780− init_s32_min = (s16)reg->s32_min_value;
5528+ init_s32_max = (s16)reg_s32_max(reg);
5529+ init_s32_min = (s16)reg_s32_min(reg);
57815530 }
57825531 s32_max = max(init_s32_max, init_s32_min);
57835532 s32_min = min(init_s32_max, init_s32_min);
57845533
57855534 if ((s32_min >= 0) == (s32_max >= 0)) {
5786− reg->s32_min_value = s32_min;
5787− reg->s32_max_value = s32_max;
5788− reg->u32_min_value = (u32)s32_min;
5789− reg->u32_max_value = (u32)s32_max;
5535+ reg_set_srange32(reg, s32_min, s32_max);
57905536 reg->var_off = tnum_subreg(tnum_range(s32_min, s32_max));
57915537 return;
57925538 }
@@ -5976,12 +5722,11 @@ static bool type_is_trusted_or_null(struct bpf_verifier_env *env, · 183 unmodified lines
59765722 }
59775723
59785724 static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
5979− struct bpf_reg_state *regs,
5980− int regno, int off, int size,
5725+ struct bpf_reg_state *regs, struct bpf_reg_state *reg,
5726+ argno_t argno, int off, int size,
59815727 enum bpf_access_type atype,
59825728 int value_regno)
59835729 {
5984− struct bpf_reg_state *reg = regs + regno;
59855730 const struct btf_type *t = btf_type_by_id(reg->btf, reg->btf_id);
59865731 const char *tname = btf_name_by_offset(reg->btf, t->name_off);
59875732 const char *field_name = NULL;
@@ -6007,8 +5752,8 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 19 unmodified lines
60075752
60085753 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
60095754 verbose(env,
6010− "R%d is ptr_%s invalid variable offset: off=%d, var_off=%s\n",
6011− regno, tname, off, tn_buf);
5755+ "%s is ptr_%s invalid variable offset: off=%d, var_off=%s\n",
5756+ reg_arg_name(env, argno), tname, off, tn_buf);
60125757 return -EACCES;
60135758 }
60145759
@@ -6016,22 +5761,22 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 1 unmodified line
60165761
60175762 if (off < 0) {
60185763 verbose(env,
6019− "R%d is ptr_%s invalid negative access: off=%d\n",
6020− regno, tname, off);
5764+ "%s is ptr_%s invalid negative access: off=%d\n",
5765+ reg_arg_name(env, argno), tname, off);
60215766 return -EACCES;
60225767 }
60235768
60245769 if (reg->type & MEM_USER) {
60255770 verbose(env,
6026− "R%d is ptr_%s access user memory: off=%d\n",
6027− regno, tname, off);
5771+ "%s is ptr_%s access user memory: off=%d\n",
5772+ reg_arg_name(env, argno), tname, off);
60285773 return -EACCES;
60295774 }
60305775
60315776 if (reg->type & MEM_PERCPU) {
60325777 verbose(env,
6033− "R%d is ptr_%s access percpu memory: off=%d\n",
6034− regno, tname, off);
5778+ "%s is ptr_%s access percpu memory: off=%d\n",
5779+ reg_arg_name(env, argno), tname, off);
60355780 return -EACCES;
60365781 }
60375782
@@ -6043,7 +5788,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 5 unmodified lines
60435788 ret = env->ops->btf_struct_access(&env->log, reg, off, size);
60445789 } else {
60455790 /* Writes are permitted with default btf_struct_access for
6046− * program allocated objects (which always have ref_obj_id > 0),
5791+ * program allocated objects (which always have id > 0),
60475792 * but not for untrusted PTR_TO_BTF_ID | MEM_ALLOC.
60485793 */
60495794 if (atype != BPF_READ && !type_is_ptr_alloc_obj(reg->type)) {
@@ -6052,8 +5797,8 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 2 unmodified lines
60525797 }
60535798
60545799 if (type_is_alloc(reg->type) && !type_is_non_owning_ref(reg->type) &&
6055− !(reg->type & MEM_RCU) && !reg->ref_obj_id) {
6056− verifier_bug(env, "ref_obj_id for allocated object must be non-zero");
5800+ !(reg->type & MEM_RCU) && !reg_is_referenced(env, reg)) {
5801+ verifier_bug(env, "allocated object must have a referenced id");
60575802 return -EFAULT;
60585803 }
60595804
@@ -6072,7 +5817,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 12 unmodified lines
60725817 */
60735818 flag = PTR_UNTRUSTED;
60745819
6075− } else if (is_trusted_reg(reg) || is_rcu_reg(reg)) {
5820+ } else if (is_trusted_reg(env, reg) || is_rcu_reg(reg)) {
60765821 /* By default any pointer obtained from walking a trusted pointer is no
60775822 * longer trusted, unless the field being accessed has explicitly been
60785823 * marked as inheriting its parent's state of trust (either full or RCU).
@@ -6133,12 +5878,11 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 54 unmodified lines
61335878 }
61345879
61355880 static int check_ptr_to_map_access(struct bpf_verifier_env *env,
6136− struct bpf_reg_state *regs,
6137− int regno, int off, int size,
5881+ struct bpf_reg_state *regs, struct bpf_reg_state *reg,
5882+ argno_t argno, int off, int size,
61385883 enum bpf_access_type atype,
61395884 int value_regno)
61405885 {
6141− struct bpf_reg_state *reg = regs + regno;
61425886 struct bpf_map *map = reg->map_ptr;
61435887 struct bpf_reg_state map_reg;
61445888 enum bpf_type_flag flag = 0;
@@ -6169,8 +5913,8 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env, · 24 unmodified lines
61695913 }
61705914
61715915 if (off < 0) {
6172− verbose(env, "R%d is %s invalid negative access: off=%d\n",
6173− regno, tname, off);
5916+ verbose(env, "%s is %s invalid negative access: off=%d\n",
5917+ reg_arg_name(env, argno), tname, off);
61745918 return -EACCES;
61755919 }
61765920
@@ -6227,11 +5971,10 @@ static int check_stack_slot_within_bounds(struct bpf_verifier_env *env, · 50 unmodified lines
62275971 * 'off' includes `regno->offset`, but not its dynamic part (if any).
62285972 */
62295973 static int check_stack_access_within_bounds(
6230− struct bpf_verifier_env *env,
6231− int regno, int off, int access_size,
5974+ struct bpf_verifier_env *env, struct bpf_reg_state *reg,
5975+ argno_t argno, int off, int access_size,
62325976 enum bpf_access_type type)
62335977 {
6234− struct bpf_reg_state *reg = reg_state(env, regno);
62355978 struct bpf_func_state *state = bpf_func(env, reg);
62365979 s64 min_off, max_off;
62375980 int err;
@@ -6246,14 +5989,14 @@ static int check_stack_access_within_bounds( · 8 unmodified lines
62465989 min_off = (s64)reg->var_off.value + off;
62475990 max_off = min_off + access_size;
62485991 } else {
6249− if (reg->smax_value >= BPF_MAX_VAR_OFF ||
6250− reg->smin_value <= -BPF_MAX_VAR_OFF) {
6251− verbose(env, "invalid unbounded variable-offset%s stack R%d\n",
6252− err_extra, regno);
5992+ if (reg_smax(reg) >= BPF_MAX_VAR_OFF ||
5993+ reg_smin(reg) <= -BPF_MAX_VAR_OFF) {
5994+ verbose(env, "invalid unbounded variable-offset%s stack %s\n",
5995+ err_extra, reg_arg_name(env, argno));
62535996 return -EACCES;
62545997 }
6255− min_off = reg->smin_value + off;
6256− max_off = reg->smax_value + off + access_size;
5998+ min_off = reg_smin(reg) + off;
5999+ max_off = reg_smax(reg) + off + access_size;
62576000 }
62586001
62596002 err = check_stack_slot_within_bounds(env, min_off, state, type);
@@ -6267,14 +6010,14 @@ static int check_stack_access_within_bounds( · 7 unmodified lines
62676010
62686011 if (err) {
62696012 if (tnum_is_const(reg->var_off)) {
6270− verbose(env, "invalid%s stack R%d off=%lld size=%d\n",
6271− err_extra, regno, min_off, access_size);
6013+ verbose(env, "invalid%s stack %s off=%lld size=%d\n",
6014+ err_extra, reg_arg_name(env, argno), min_off, access_size);
62726015 } else {
62736016 char tn_buf[48];
62746017
62756018 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
6276− verbose(env, "invalid variable-offset%s stack R%d var_off=%s off=%d size=%d\n",
6277− err_extra, regno, tn_buf, off, access_size);
6019+ verbose(env, "invalid variable-offset%s stack %s var_off=%s off=%d size=%d\n",
6020+ err_extra, reg_arg_name(env, argno), tn_buf, off, access_size);
62786021 }
62796022 return err;
62806023 }
@@ -6319,12 +6062,11 @@ static void add_scalar_to_reg(struct bpf_reg_state *dst_reg, s64 val) · 38 unmodified lines
63196062 * if t==write && value_regno==-1, some unknown value is stored into memory
63206063 * if t==read && value_regno==-1, don't care what we read from memory
63216064 */
6322−static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regno,
6065+static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct bpf_reg_state *reg, argno_t argno,
63236066 int off, int bpf_size, enum bpf_access_type t,
63246067 int value_regno, bool strict_alignment_once, bool is_ldsx)
63256068 {
63266069 struct bpf_reg_state *regs = cur_regs(env);
6327− struct bpf_reg_state *reg = regs + regno;
63286070 int size, err = 0;
63296071
63306072 size = bpf_size_to_bytes(bpf_size);
@@ -6337,11 +6079,12 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 6 unmodified lines
63376079
63386080 if (reg->type == PTR_TO_MAP_KEY) {
63396081 if (t == BPF_WRITE) {
6340− verbose(env, "write to change key R%d not allowed\n", regno);
6082+ verbose(env, "write to change key %s not allowed\n",
6083+ reg_arg_name(env, argno));
63416084 return -EACCES;
63426085 }
63436086
6344− err = check_mem_region_access(env, regno, off, size,
6087+ err = check_mem_region_access(env, reg, argno, off, size,
63456088 reg->map_ptr->key_size, false);
63466089 if (err)
63476090 return err;
@@ -6355,17 +6098,17 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 7 unmodified lines
63556098 verbose(env, "R%d leaks addr into map\n", value_regno);
63566099 return -EACCES;
63576100 }
6358− err = check_map_access_type(env, regno, off, size, t);
6101+ err = check_map_access_type(env, reg, off, size, t);
63596102 if (err)
63606103 return err;
6361− err = check_map_access(env, regno, off, size, false, ACCESS_DIRECT);
6104+ err = check_map_access(env, reg, argno, off, size, false, ACCESS_DIRECT);
63626105 if (err)
63636106 return err;
63646107 if (tnum_is_const(reg->var_off))
63656108 kptr_field = btf_record_find(reg->map_ptr->record,
63666109 off + reg->var_off.value, BPF_KPTR | BPF_UPTR);
63676110 if (kptr_field) {
6368− err = check_map_kptr_access(env, regno, value_regno, insn_idx, kptr_field);
6111+ err = check_map_kptr_access(env, value_regno, insn_idx, kptr_field);
63696112 } else if (t == BPF_READ && value_regno >= 0) {
63706113 struct bpf_map *map = reg->map_ptr;
63716114
@@ -6393,7 +6136,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 21 unmodified lines
63936136 size);
63946137 return -EACCES;
63956138 }
6396− copy_register_state(&regs[value_regno], reg);
6139+ regs[value_regno] = *reg;
63976140 add_scalar_to_reg(&regs[value_regno], off);
63986141 regs[value_regno].type = PTR_TO_INSN;
63996142 } else {
@@ -6405,14 +6148,14 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 5 unmodified lines
64056148 bool rdonly_untrusted = rdonly_mem && (reg->type & PTR_UNTRUSTED);
64066149
64076150 if (type_may_be_null(reg->type)) {
6408− verbose(env, "R%d invalid mem access '%s'\n", regno,
6151+ verbose(env, "%s invalid mem access '%s'\n", reg_arg_name(env, argno),
64096152 reg_type_str(env, reg->type));
64106153 return -EACCES;
64116154 }
64126155
64136156 if (t == BPF_WRITE && rdonly_mem) {
6414− verbose(env, "R%d cannot write into %s\n",
6415− regno, reg_type_str(env, reg->type));
6157+ verbose(env, "%s cannot write into %s\n",
6158+ reg_arg_name(env, argno), reg_type_str(env, reg->type));
64166159 return -EACCES;
64176160 }
64186161
@@ -6427,7 +6170,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 8 unmodified lines
64276170 * instructions, hence no need to check bounds in that case.
64286171 */
64296172 if (!rdonly_untrusted)
6430− err = check_mem_region_access(env, regno, off, size,
6173+ err = check_mem_region_access(env, reg, argno, off, size,
64316174 reg->mem_size, false);
64326175 if (!err && value_regno >= 0 && (t == BPF_READ || rdonly_mem))
64336176 mark_reg_unknown(env, regs, value_regno);
@@ -6445,7 +6188,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 11 unmodified lines
64456188 return -EACCES;
64466189 }
64476190
6448− err = check_ctx_access(env, insn_idx, regno, off, size, t, &info);
6191+ err = check_ctx_access(env, insn_idx, reg, argno, off, size, t, &info);
64496192 if (!err && t == BPF_READ && value_regno >= 0) {
64506193 /* ctx access returns either a scalar, or a
64516194 * PTR_TO_PACKET[_META,_END]. In the latter
@@ -6463,8 +6206,6 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 11 unmodified lines
64636206 } else {
64646207 mark_reg_known_zero(env, regs,
64656208 value_regno);
6466− if (type_may_be_null(info.reg_type))
6467− regs[value_regno].id = ++env->id_gen;
64686209 /* A load of ctx field could have different
64696210 * actual load size with the one encoded in the
64706211 * insn. When the dst is PTR, it is for sure not
@@ -6474,23 +6215,25 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 3 unmodified lines
64746215 if (base_type(info.reg_type) == PTR_TO_BTF_ID) {
64756216 regs[value_regno].btf = info.btf;
64766217 regs[value_regno].btf_id = info.btf_id;
6477− regs[value_regno].ref_obj_id = info.ref_obj_id;
6218+ regs[value_regno].id = info.ref_id;
64786219 }
6220+ if (type_may_be_null(info.reg_type) && !regs[value_regno].id)
6221+ regs[value_regno].id = ++env->id_gen;
64796222 }
64806223 regs[value_regno].type = info.reg_type;
64816224 }
64826225
64836226 } else if (reg->type == PTR_TO_STACK) {
64846227 /* Basic bounds checks. */
6485− err = check_stack_access_within_bounds(env, regno, off, size, t);
6228+ err = check_stack_access_within_bounds(env, reg, argno, off, size, t);
64866229 if (err)
64876230 return err;
64886231
64896232 if (t == BPF_READ)
6490− err = check_stack_read(env, regno, off, size,
6233+ err = check_stack_read(env, reg, argno, off, size,
64916234 value_regno);
64926235 else
6493− err = check_stack_write(env, regno, off, size,
6236+ err = check_stack_write(env, reg, off, size,
64946237 value_regno, insn_idx);
64956238 } else if (reg_is_pkt_pointer(reg)) {
64966239 if (t == BPF_WRITE && !may_access_direct_pkt_data(env, NULL, t)) {
@@ -6503,7 +6246,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 6 unmodified lines
65036246 value_regno);
65046247 return -EACCES;
65056248 }
6506− err = check_packet_access(env, regno, off, size, false);
6249+ err = check_packet_access(env, reg, argno, off, size, false);
65076250 if (!err && t == BPF_READ && value_regno >= 0)
65086251 mark_reg_unknown(env, regs, value_regno);
65096252 } else if (reg->type == PTR_TO_FLOW_KEYS) {
@@ -6514,28 +6257,28 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 4 unmodified lines
65146257 return -EACCES;
65156258 }
65166259
6517− err = check_flow_keys_access(env, off, size);
6260+ err = check_flow_keys_access(env, reg, argno, off, size);
65186261 if (!err && t == BPF_READ && value_regno >= 0)
65196262 mark_reg_unknown(env, regs, value_regno);
65206263 } else if (type_is_sk_pointer(reg->type)) {
65216264 if (t == BPF_WRITE) {
6522− verbose(env, "R%d cannot write into %s\n",
6523− regno, reg_type_str(env, reg->type));
6265+ verbose(env, "%s cannot write into %s\n",
6266+ reg_arg_name(env, argno), reg_type_str(env, reg->type));
65246267 return -EACCES;
65256268 }
6526− err = check_sock_access(env, insn_idx, regno, off, size, t);
6269+ err = check_sock_access(env, insn_idx, reg, argno, off, size, t);
65276270 if (!err && value_regno >= 0)
65286271 mark_reg_unknown(env, regs, value_regno);
65296272 } else if (reg->type == PTR_TO_TP_BUFFER) {
6530− err = check_tp_buffer_access(env, reg, regno, off, size);
6273+ err = check_tp_buffer_access(env, reg, argno, off, size);
65316274 if (!err && t == BPF_READ && value_regno >= 0)
65326275 mark_reg_unknown(env, regs, value_regno);
65336276 } else if (base_type(reg->type) == PTR_TO_BTF_ID &&
65346277 !type_may_be_null(reg->type)) {
6535− err = check_ptr_to_btf_access(env, regs, regno, off, size, t,
6278+ err = check_ptr_to_btf_access(env, regs, reg, argno, off, size, t,
65366279 value_regno);
65376280 } else if (reg->type == CONST_PTR_TO_MAP) {
6538− err = check_ptr_to_map_access(env, regs, regno, off, size, t,
6281+ err = check_ptr_to_map_access(env, regs, reg, argno, off, size, t,
65396282 value_regno);
65406283 } else if (base_type(reg->type) == PTR_TO_BUF &&
65416284 !type_may_be_null(reg->type)) {
@@ -6544,8 +6287,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 2 unmodified lines
65446287
65456288 if (rdonly_mem) {
65466289 if (t == BPF_WRITE) {
6547− verbose(env, "R%d cannot write into %s\n",
6548− regno, reg_type_str(env, reg->type));
6290+ verbose(env, "%s cannot write into %s\n",
6291+ reg_arg_name(env, argno), reg_type_str(env, reg->type));
65496292 return -EACCES;
65506293 }
65516294 max_access = &env->prog->aux->max_rdonly_access;
@@ -6553,7 +6296,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 1 unmodified line
65536296 max_access = &env->prog->aux->max_rdwr_access;
65546297 }
65556298
6556− err = check_buffer_access(env, reg, regno, off, size, false,
6299+ err = check_buffer_access(env, reg, argno, off, size, false,
65576300 max_access);
65586301
65596302 if (!err && value_regno >= 0 && (rdonly_mem || t == BPF_READ))
@@ -6562,7 +6305,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 2 unmodified lines
65626305 if (t == BPF_READ && value_regno >= 0)
65636306 mark_reg_unknown(env, regs, value_regno);
65646307 } else {
6565− verbose(env, "R%d invalid mem access '%s'\n", regno,
6308+ verbose(env, "%s invalid mem access '%s'\n", reg_arg_name(env, argno),
65666309 reg_type_str(env, reg->type));
65676310 return -EACCES;
65686311 }
@@ -6585,10 +6328,20 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, · 16 unmodified lines
65856328 bool strict_alignment_once, bool is_ldsx,
65866329 bool allow_trust_mismatch, const char *ctx)
65876330 {
6331+ struct bpf_verifier_state *vstate = env->cur_state;
6332+ struct bpf_func_state *state = vstate->frame[vstate->curframe];
65886333 struct bpf_reg_state *regs = cur_regs(env);
65896334 enum bpf_reg_type src_reg_type;
65906335 int err;
65916336
6337+ /* Handle stack arg read */
6338+ if (is_stack_arg_ldx(insn)) {
6339+ err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK);
6340+ if (err)
6341+ return err;
6342+ return check_stack_arg_read(env, state, insn->off, insn->dst_reg);
6343+ }
6344+
65926345 /* check src operand */
65936346 err = check_reg_arg(env, insn->src_reg, SRC_OP);
65946347 if (err)
@@ -6604,7 +6357,7 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, · 9 unmodified lines
66046357 /* Check if (src_reg + off) is readable. The state of dst_reg will be
66056358 * updated by this call.
66066359 */
6607− err = check_mem_access(env, env->insn_idx, insn->src_reg, insn->off,
6360+ err = check_mem_access(env, env->insn_idx, regs + insn->src_reg, argno_from_reg(insn->src_reg), insn->off,
66086361 BPF_SIZE(insn->code), BPF_READ, insn->dst_reg,
66096362 strict_alignment_once, is_ldsx);
66106363 err = err ?: save_aux_ptr_type(env, src_reg_type,
@@ -6617,10 +6370,20 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, · 6 unmodified lines
66176370 static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn,
66186371 bool strict_alignment_once)
66196372 {
6373+ struct bpf_verifier_state *vstate = env->cur_state;
6374+ struct bpf_func_state *state = vstate->frame[vstate->curframe];
66206375 struct bpf_reg_state *regs = cur_regs(env);
66216376 enum bpf_reg_type dst_reg_type;
66226377 int err;
66236378
6379+ /* Handle stack arg write */
6380+ if (is_stack_arg_stx(insn)) {
6381+ err = check_reg_arg(env, insn->src_reg, SRC_OP);
6382+ if (err)
6383+ return err;
6384+ return check_stack_arg_write(env, state, insn->off, regs + insn->src_reg);
6385+ }
6386+
66246387 /* check src1 operand */
66256388 err = check_reg_arg(env, insn->src_reg, SRC_OP);
66266389 if (err)
@@ -6634,7 +6397,7 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, · 7 unmodified lines
66346397 dst_reg_type = regs[insn->dst_reg].type;
66356398
66366399 /* Check if (dst_reg + off) is writeable. */
6637− err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off,
6400+ err = check_mem_access(env, env->insn_idx, regs + insn->dst_reg, argno_from_reg(insn->dst_reg), insn->off,
66386401 BPF_SIZE(insn->code), BPF_WRITE, insn->src_reg,
66396402 strict_alignment_once, false);
66406403 err = err ?: save_aux_ptr_type(env, dst_reg_type, false);
@@ -6645,6 +6408,7 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, · 4 unmodified lines
66456408 static int check_atomic_rmw(struct bpf_verifier_env *env,
66466409 struct bpf_insn *insn)
66476410 {
6411+ struct bpf_reg_state *dst_reg;
66486412 int load_reg;
66496413 int err;
66506414
@@ -6706,13 +6470,15 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, · 55 unmodified lines
67066470 load_reg = -1;
67076471 }
67086472
6473+ dst_reg = cur_regs(env) + insn->dst_reg;
6474+
67096475 /* Check whether we can read the memory, with second call for fetch
67106476 * case to simulate the register fill.
67116477 */
6712− err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off,
6478+ err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), insn->off,
67136479 BPF_SIZE(insn->code), BPF_READ, -1, true, false);
67146480 if (!err && load_reg >= 0)
6715− err = check_mem_access(env, env->insn_idx, insn->dst_reg,
6481+ err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg),
67166482 insn->off, BPF_SIZE(insn->code),
67176483 BPF_READ, load_reg, true, false);
67186484 if (err)
@@ -6724,7 +6490,7 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, · 5 unmodified lines
67246490 return err;
67256491 }
67266492 /* Check whether we can write into the same memory. */
6727− err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off,
6493+ err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), insn->off,
67286494 BPF_SIZE(insn->code), BPF_WRITE, -1, true, false);
67296495 if (err)
67306496 return err;
@@ -6813,11 +6579,10 @@ static int check_atomic(struct bpf_verifier_env *env, struct bpf_insn *insn) · 82 unmodified lines
68136579 * read offsets are marked as read.
68146580 */
68156581 static int check_stack_range_initialized(
6816− struct bpf_verifier_env *env, int regno, int off,
6582+ struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int off,
68176583 int access_size, bool zero_size_allowed,
68186584 enum bpf_access_type type, struct bpf_call_arg_meta *meta)
68196585 {
6820− struct bpf_reg_state *reg = reg_state(env, regno);
68216586 struct bpf_func_state *state = bpf_func(env, reg);
68226587 int err, min_off, max_off, i, j, slot, spi;
68236588 /* Some accesses can write anything into the stack, others are
@@ -6839,7 +6604,7 @@ static int check_stack_range_initialized( · 15 unmodified lines
68396604 return -EACCES;
68406605 }
68416606
6842− err = check_stack_access_within_bounds(env, regno, off, access_size, type);
6607+ err = check_stack_access_within_bounds(env, reg, argno, off, access_size, type);
68436608 if (err)
68446609 return err;
68456610
@@ -6856,8 +6621,8 @@ static int check_stack_range_initialized( · 10 unmodified lines
68566621 char tn_buf[48];
68576622
68586623 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
6859− verbose(env, "R%d variable offset stack access prohibited for !root, var_off=%s\n",
6860− regno, tn_buf);
6624+ verbose(env, "%s variable offset stack access prohibited for !root, var_off=%s\n",
6625+ reg_arg_name(env, argno), tn_buf);
68616626 return -EACCES;
68626627 }
68636628 /* Only initialized buffer on stack is allowed to be accessed
@@ -6869,8 +6634,8 @@ static int check_stack_range_initialized( · 5 unmodified lines
68696634 if (meta && meta->raw_mode)
68706635 meta = NULL;
68716636
6872− min_off = reg->smin_value + off;
6873− max_off = reg->smax_value + off;
6637+ min_off = reg_smin(reg) + off;
6638+ max_off = reg_smax(reg) + off;
68746639 }
68756640
68766641 if (meta && meta->raw_mode) {
@@ -6900,7 +6665,7 @@ static int check_stack_range_initialized( · 23 unmodified lines
69006665 }
69016666 }
69026667 meta->access_size = access_size;
6903− meta->regno = regno;
6668+ meta->regno = reg_from_argno(argno);
69046669 return 0;
69056670 }
69066671
@@ -6940,17 +6705,17 @@ static int check_stack_range_initialized( · 33 unmodified lines
69406705 if (*stype == STACK_POISON) {
69416706 if (allow_poison)
69426707 goto mark;
6943− verbose(env, "reading from stack R%d off %d+%d size %d, slot poisoned by dead code elimination\n",
6944− regno, min_off, i - min_off, access_size);
6708+ verbose(env, "reading from stack %s off %d+%d size %d, slot poisoned by dead code elimination\n",
6709+ reg_arg_name(env, argno), min_off, i - min_off, access_size);
69456710 } else if (tnum_is_const(reg->var_off)) {
6946− verbose(env, "invalid read from stack R%d off %d+%d size %d\n",
6947− regno, min_off, i - min_off, access_size);
6711+ verbose(env, "invalid read from stack %s off %d+%d size %d\n",
6712+ reg_arg_name(env, argno), min_off, i - min_off, access_size);
69486713 } else {
69496714 char tn_buf[48];
69506715
69516716 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
6952− verbose(env, "invalid read from stack R%d var_off %s+%d size %d\n",
6953− regno, tn_buf, i - min_off, access_size);
6717+ verbose(env, "invalid read from stack %s var_off %s+%d size %d\n",
6718+ reg_arg_name(env, argno), tn_buf, i - min_off, access_size);
69546719 }
69556720 return -EACCES;
69566721 mark:
@@ -6959,48 +6724,48 @@ mark: · 2 unmodified lines
69596724 return 0;
69606725 }
69616726
6962−static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,
6727+static int check_helper_mem_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
69636728 int access_size, enum bpf_access_type access_type,
69646729 bool zero_size_allowed,
69656730 struct bpf_call_arg_meta *meta)
69666731 {
6967− struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[regno];
6732+ struct bpf_reg_state *regs = cur_regs(env);
69686733 u32 *max_access;
69696734
69706735 switch (base_type(reg->type)) {
69716736 case PTR_TO_PACKET:
69726737 case PTR_TO_PACKET_META:
6973− return check_packet_access(env, regno, 0, access_size,
6738+ return check_packet_access(env, reg, argno, 0, access_size,
69746739 zero_size_allowed);
69756740 case PTR_TO_MAP_KEY:
69766741 if (access_type == BPF_WRITE) {
6977− verbose(env, "R%d cannot write into %s\n", regno,
6978− reg_type_str(env, reg->type));
6742+ verbose(env, "%s cannot write into %s\n",
6743+ reg_arg_name(env, argno), reg_type_str(env, reg->type));
69796744 return -EACCES;
69806745 }
6981− return check_mem_region_access(env, regno, 0, access_size,
6746+ return check_mem_region_access(env, reg, argno, 0, access_size,
69826747 reg->map_ptr->key_size, false);
69836748 case PTR_TO_MAP_VALUE:
6984− if (check_map_access_type(env, regno, 0, access_size, access_type))
6749+ if (check_map_access_type(env, reg, 0, access_size, access_type))
69856750 return -EACCES;
6986− return check_map_access(env, regno, 0, access_size,
6751+ return check_map_access(env, reg, argno, 0, access_size,
69876752 zero_size_allowed, ACCESS_HELPER);
69886753 case PTR_TO_MEM:
69896754 if (type_is_rdonly_mem(reg->type)) {
69906755 if (access_type == BPF_WRITE) {
6991− verbose(env, "R%d cannot write into %s\n", regno,
6992− reg_type_str(env, reg->type));
6756+ verbose(env, "%s cannot write into %s\n",
6757+ reg_arg_name(env, argno), reg_type_str(env, reg->type));
69936758 return -EACCES;
69946759 }
69956760 }
6996− return check_mem_region_access(env, regno, 0,
6761+ return check_mem_region_access(env, reg, argno, 0,
69976762 access_size, reg->mem_size,
69986763 zero_size_allowed);
69996764 case PTR_TO_BUF:
70006765 if (type_is_rdonly_mem(reg->type)) {
70016766 if (access_type == BPF_WRITE) {
7002− verbose(env, "R%d cannot write into %s\n", regno,
7003− reg_type_str(env, reg->type));
6767+ verbose(env, "%s cannot write into %s\n",
6768+ reg_arg_name(env, argno), reg_type_str(env, reg->type));
70046769 return -EACCES;
70056770 }
70066771
@@ -7008,26 +6773,26 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 1 unmodified line
70086773 } else {
70096774 max_access = &env->prog->aux->max_rdwr_access;
70106775 }
7011− return check_buffer_access(env, reg, regno, 0,
6776+ return check_buffer_access(env, reg, argno, 0,
70126777 access_size, zero_size_allowed,
70136778 max_access);
70146779 case PTR_TO_STACK:
70156780 return check_stack_range_initialized(
7016− env,
7017− regno, 0, access_size,
6781+ env, reg,
6782+ argno, 0, access_size,
70186783 zero_size_allowed, access_type, meta);
70196784 case PTR_TO_BTF_ID:
7020− return check_ptr_to_btf_access(env, regs, regno, 0,
7021− access_size, BPF_READ, -1);
6785+ return check_ptr_to_btf_access(env, regs, reg, argno, 0,
6786+ access_size, access_type, -1);
70226787 case PTR_TO_CTX:
70236788 /* Only permit reading or writing syscall context using helper calls. */
70246789 if (is_var_ctx_off_allowed(env->prog)) {
7025− int err = check_mem_region_access(env, regno, 0, access_size, U16_MAX,
6790+ int err = check_mem_region_access(env, reg, argno, 0, access_size, U16_MAX,
70266791 zero_size_allowed);
70276792 if (err)
70286793 return err;
7029− if (env->prog->aux->max_ctx_offset < reg->umax_value + access_size)
7030− env->prog->aux->max_ctx_offset = reg->umax_value + access_size;
6794+ if (env->prog->aux->max_ctx_offset < reg_umax(reg) + access_size)
6795+ env->prog->aux->max_ctx_offset = reg_umax(reg) + access_size;
70316796 return 0;
70326797 }
70336798 fallthrough;
@@ -7037,7 +6802,7 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 3 unmodified lines
70376802 bpf_register_is_null(reg))
70386803 return 0;
70396804
7040− verbose(env, "R%d type=%s ", regno,
6805+ verbose(env, "%s type=%s ", reg_arg_name(env, argno),
70416806 reg_type_str(env, reg->type));
70426807 verbose(env, "expected=%s\n", reg_type_str(env, PTR_TO_STACK));
70436808 return -EACCES;
@@ -7047,12 +6812,12 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 3 unmodified lines
70476812 /* verify arguments to helpers or kfuncs consisting of a pointer and an access
70486813 * size.
70496814 *
7050− * @regno is the register containing the access size. regno-1 is the register
7051− * containing the pointer.
6815+ * @mem_reg contains the pointer, @size_reg contains the access size.
70526816 */
70536817 static int check_mem_size_reg(struct bpf_verifier_env *env,
7054− struct bpf_reg_state *reg, u32 regno,
7055− enum bpf_access_type access_type,
6818+ struct bpf_reg_state *mem_reg,
6819+ struct bpf_reg_state *size_reg, argno_t mem_argno,
6820+ argno_t size_argno, enum bpf_access_type access_type,
70566821 bool zero_size_allowed,
70576822 struct bpf_call_arg_meta *meta)
70586823 {
@@ -7066,42 +6831,48 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, · 7 unmodified lines
70666831 * out. Only upper bounds can be learned because retval is an
70676832 * int type and negative retvals are allowed.
70686833 */
7069− meta->msize_max_value = reg->umax_value;
6834+ meta->msize_max_value = reg_umax(size_reg);
70706835
70716836 /* The register is SCALAR_VALUE; the access check happens using
70726837 * its boundaries. For unprivileged variable accesses, disable
70736838 * raw mode so that the program is required to initialize all
70746839 * the memory that the helper could just partially fill up.
70756840 */
7076− if (!tnum_is_const(reg->var_off))
6841+ if (!tnum_is_const(size_reg->var_off))
70776842 meta = NULL;
70786843
7079− if (reg->smin_value < 0) {
7080− verbose(env, "R%d min value is negative, either use unsigned or 'var &= const'\n",
7081− regno);
6844+ if (reg_smin(size_reg) < 0) {
6845+ verbose(env, "%s min value is negative, either use unsigned or 'var &= const'\n",
6846+ reg_arg_name(env, size_argno));
70826847 return -EACCES;
70836848 }
70846849
7085− if (reg->umin_value == 0 && !zero_size_allowed) {
7086− verbose(env, "R%d invalid zero-sized read: u64=[%lld,%lld]\n",
7087− regno, reg->umin_value, reg->umax_value);
6850+ if (reg_umin(size_reg) == 0 && !zero_size_allowed) {
6851+ verbose(env, "%s invalid zero-sized read: u64=[%lld,%lld]\n",
6852+ reg_arg_name(env, size_argno), reg_umin(size_reg), reg_umax(size_reg));
70886853 return -EACCES;
70896854 }
70906855
7091− if (reg->umax_value >= BPF_MAX_VAR_SIZ) {
7092− verbose(env, "R%d unbounded memory access, use 'var &= const' or 'if (var < const)'\n",
7093− regno);
6856+ if (reg_umax(size_reg) >= BPF_MAX_VAR_SIZ) {
6857+ verbose(env, "%s unbounded memory access, use 'var &= const' or 'if (var < const)'\n",
6858+ reg_arg_name(env, size_argno));
70946859 return -EACCES;
70956860 }
7096− err = check_helper_mem_access(env, regno - 1, reg->umax_value,
6861+ err = check_helper_mem_access(env, mem_reg, mem_argno, reg_umax(size_reg),
70976862 access_type, zero_size_allowed, meta);
7098− if (!err)
7099− err = mark_chain_precision(env, regno);
6863+ if (!err) {
6864+ int regno = reg_from_argno(size_argno);
6865+
6866+ if (regno >= 0)
6867+ err = mark_chain_precision(env, regno);
6868+ else
6869+ err = mark_stack_arg_precision(env, arg_idx_from_argno(size_argno));
6870+ }
71006871 return err;
71016872 }
71026873
71036874 static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
7104− u32 regno, u32 mem_size)
6875+ argno_t argno, u32 mem_size)
71056876 {
71066877 bool may_be_null = type_may_be_null(reg->type);
71076878 struct bpf_reg_state saved_reg;
@@ -7110,6 +6881,12 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg · 2 unmodified lines
71106881 if (bpf_register_is_null(reg))
71116882 return 0;
71126883
6884+ if (mem_size > S32_MAX) {
6885+ verbose(env, "%s memory size %u is too large\n",
6886+ reg_arg_name(env, argno), mem_size);
6887+ return -EACCES;
6888+ }
6889+
71136890 /* Assuming that the register contains a value check if the memory
71146891 * access is safe. Temporarily save and restore the register's state as
71156892 * the conversion shouldn't be visible to a caller.
@@ -7121,8 +6898,8 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg · 5 unmodified lines
71216898
71226899 int size = base_type(reg->type) == PTR_TO_STACK ? -(int)mem_size : mem_size;
71236900
7124− err = check_helper_mem_access(env, regno, size, BPF_READ, true, NULL);
7125− err = err ?: check_helper_mem_access(env, regno, size, BPF_WRITE, true, NULL);
6901+ err = check_helper_mem_access(env, reg, argno, size, BPF_READ, true, NULL);
6902+ err = err ?: check_helper_mem_access(env, reg, argno, size, BPF_WRITE, true, NULL);
71266903
71276904 if (may_be_null)
71286905 *reg = saved_reg;
@@ -7130,17 +6907,14 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg · 1 unmodified line
71306907 return err;
71316908 }
71326909
7133−static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
7134− u32 regno)
6910+static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *mem_reg,
6911+ struct bpf_reg_state *size_reg, argno_t mem_argno, argno_t size_argno)
71356912 {
7136− struct bpf_reg_state *mem_reg = &cur_regs(env)[regno - 1];
71376913 bool may_be_null = type_may_be_null(mem_reg->type);
71386914 struct bpf_reg_state saved_reg;
71396915 struct bpf_call_arg_meta meta;
71406916 int err;
71416917
7142− WARN_ON_ONCE(regno < BPF_REG_2 || regno > BPF_REG_5);
7143−
71446918 memset(&meta, 0, sizeof(meta));
71456919
71466920 if (may_be_null) {
@@ -7148,8 +6922,8 @@ static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg · 1 unmodified line
71486922 mark_ptr_not_null_reg(mem_reg);
71496923 }
71506924
7151− err = check_mem_size_reg(env, reg, regno, BPF_READ, true, &meta);
7152− err = err ?: check_mem_size_reg(env, reg, regno, BPF_WRITE, true, &meta);
6925+ err = check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_READ, true, &meta);
6926+ err = err ?: check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_WRITE, true, &meta);
71536927
71546928 if (may_be_null)
71556929 *mem_reg = saved_reg;
@@ -7185,11 +6959,10 @@ enum { · 29 unmodified lines
71856959 * env->cur_state->active_locks remembers which map value element or allocated
71866960 * object got locked and clears it after bpf_spin_unlock.
71876961 */
7188−static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags)
6962+static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int flags)
71896963 {
71906964 bool is_lock = flags & PROCESS_SPIN_LOCK, is_res_lock = flags & PROCESS_RES_LOCK;
71916965 const char *lock_str = is_res_lock ? "bpf_res_spin" : "bpf_spin";
7192− struct bpf_reg_state *reg = reg_state(env, regno);
71936966 struct bpf_verifier_state *cur = env->cur_state;
71946967 bool is_const = tnum_is_const(reg->var_off);
71956968 bool is_irq = flags & PROCESS_LOCK_IRQ;
@@ -7202,8 +6975,8 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags) · 6 unmodified lines
72026975
72036976 if (!is_const) {
72046977 verbose(env,
7205− "R%d doesn't have constant offset. %s_lock has to be at the constant offset\n",
7206− regno, lock_str);
6978+ "%s doesn't have constant offset. %s_lock has to be at the constant offset\n",
6979+ reg_arg_name(env, argno), lock_str);
72076980 return -EINVAL;
72086981 }
72096982 if (reg->type == PTR_TO_MAP_VALUE) {
@@ -7302,11 +7075,10 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags) · 92 unmodified lines
73027075 }
73037076
73047077 /* Check if @regno is a pointer to a specific field in a map value */
7305−static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno,
7078+static int check_map_field_pointer(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
73067079 enum btf_field_type field_type,
73077080 struct bpf_map_desc *map_desc)
73087081 {
7309− struct bpf_reg_state *reg = reg_state(env, regno);
73107082 bool is_const = tnum_is_const(reg->var_off);
73117083 struct bpf_map *map = reg->map_ptr;
73127084 u64 val = reg->var_off.value;
@@ -7315,8 +7087,8 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, · 2 unmodified lines
73157087
73167088 if (!is_const) {
73177089 verbose(env,
7318− "R%d doesn't have constant offset. %s has to be at the constant offset\n",
7319− regno, struct_name);
7090+ "%s doesn't have constant offset. %s has to be at the constant offset\n",
7091+ reg_arg_name(env, argno), struct_name);
73207092 return -EINVAL;
73217093 }
73227094 if (!map->btf) {
@@ -7356,26 +7128,26 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, · 33 unmodified lines
73567128 return 0;
73577129 }
73587130
7359−static int process_timer_func(struct bpf_verifier_env *env, int regno,
7131+static int process_timer_func(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
73607132 struct bpf_map_desc *map)
73617133 {
73627134 if (IS_ENABLED(CONFIG_PREEMPT_RT)) {
73637135 verbose(env, "bpf_timer cannot be used for PREEMPT_RT.\n");
73647136 return -EOPNOTSUPP;
73657137 }
7366− return check_map_field_pointer(env, regno, BPF_TIMER, map);
7138+ return check_map_field_pointer(env, reg, argno, BPF_TIMER, map);
73677139 }
73687140
7369−static int process_timer_helper(struct bpf_verifier_env *env, int regno,
7141+static int process_timer_helper(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
73707142 struct bpf_call_arg_meta *meta)
73717143 {
7372− return process_timer_func(env, regno, &meta->map);
7144+ return process_timer_func(env, reg, argno, &meta->map);
73737145 }
73747146
7375−static int process_timer_kfunc(struct bpf_verifier_env *env, int regno,
7147+static int process_timer_kfunc(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
73767148 struct bpf_kfunc_call_arg_meta *meta)
73777149 {
7378− return process_timer_func(env, regno, &meta->map);
7150+ return process_timer_func(env, reg, argno, &meta->map);
73797151 }
73807152
73817153 static int process_kptr_func(struct bpf_verifier_env *env, int regno,
@@ -7426,52 +7198,42 @@ static int process_kptr_func(struct bpf_verifier_env *env, int regno, · 44 unmodified lines
74267198 return 0;
74277199 }
74287200
7429−/* There are two register types representing a bpf_dynptr, one is PTR_TO_STACK
7201+/*
7202+ * Validate dynptr arguments for helper, kfunc and subprog.
7203+ *
7204+ * @dynptr is both input and output. It is populated when the argument is
7205+ * tagged with MEM_UNINIT (i.e., the dynptr argument that will be constructed)
7206+ * and consumed when the argument is expecting to be an initialized dynptr.
7207+ * @parent_id is used to track the referenced parent object (e.g., file or skb in
7208+ * qdisc program) when constructing a dynptr.
7209+ *
7210+ * There are two register types representing a bpf_dynptr, one is PTR_TO_STACK
74307211 * which points to a stack slot, and the other is CONST_PTR_TO_DYNPTR.
74317212 *
74327213 * In both cases we deal with the first 8 bytes, but need to mark the next 8
74337214 * bytes as STACK_DYNPTR in case of PTR_TO_STACK. In case of
74347215 * CONST_PTR_TO_DYNPTR, we are guaranteed to get the beginning of the object.
74357216 *
7436− * Mutability of bpf_dynptr is at two levels, one is at the level of struct
7437− * bpf_dynptr itself, i.e. whether the helper is receiving a pointer to struct
7438− * bpf_dynptr or pointer to const struct bpf_dynptr. In the former case, it can
7439− * mutate the view of the dynptr and also possibly destroy it. In the latter
7440− * case, it cannot mutate the bpf_dynptr itself but it can still mutate the
7441− * memory that dynptr points to.
7442− *
7443− * The verifier will keep track both levels of mutation (bpf_dynptr's in
7444− * reg->type and the memory's in reg->dynptr.type), but there is no support for
7445− * readonly dynptr view yet, hence only the first case is tracked and checked.
7446− *
7447− * This is consistent with how C applies the const modifier to a struct object,
7448− * where the pointer itself inside bpf_dynptr becomes const but not what it
7449− * points to.
7450− *
7451− * Helpers which do not mutate the bpf_dynptr set MEM_RDONLY in their argument
7452− * type, and declare it as 'const struct bpf_dynptr *' in their prototype.
7217+ * Mutability of bpf_dynptr is at two levels: the dynptr and the memory the
7218+ * dynptr points to. At the first level, the verifier will make sure a
7219+ * CONST_PTR_TO_DYNPTR cannot be reinitialized or destroyed. The mutability of
7220+ * a dynptr's view (i.e., start and offset) is not tracked as there is not such
7221+ * use case. The second level is tracked using the upper bit of bpf_dynptr->size
7222+ * and checked dynamically during runtime.
74537223 */
7454−static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn_idx,
7455− enum bpf_arg_type arg_type, int clone_ref_obj_id)
7224+static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
7225+ argno_t argno, int insn_idx, enum bpf_arg_type arg_type,
7226+ struct ref_obj_desc *ref_obj, struct bpf_dynptr_desc *dynptr)
74567227 {
7457− struct bpf_reg_state *reg = reg_state(env, regno);
7458− int err;
7228+ int spi, err = 0;
74597229
74607230 if (reg->type != PTR_TO_STACK && reg->type != CONST_PTR_TO_DYNPTR) {
74617231 verbose(env,
7462− "arg#%d expected pointer to stack or const struct bpf_dynptr\n",
7463− regno - 1);
7232+ "%s expected pointer to stack or const struct bpf_dynptr\n",
7233+ reg_arg_name(env, argno));
74647234 return -EINVAL;
74657235 }
74667236
7467− /* MEM_UNINIT and MEM_RDONLY are exclusive, when applied to an
7468− * ARG_PTR_TO_DYNPTR (or ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_*):
7469− */
7470− if ((arg_type & (MEM_UNINIT | MEM_RDONLY)) == (MEM_UNINIT | MEM_RDONLY)) {
7471− verifier_bug(env, "misconfigured dynptr helper type flags");
7472− return -EFAULT;
7473− }
7474−
74757237 /* MEM_UNINIT - Points to memory that is an appropriate candidate for
74767238 * constructing a mutable bpf_dynptr object.
74777239 *
@@ -7479,13 +7241,12 @@ static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn · 1 unmodified line
74797241 * pointing to a region of at least 16 bytes which doesn't
74807242 * contain an existing bpf_dynptr.
74817243 *
7482− * MEM_RDONLY - Points to a initialized bpf_dynptr that will not be
7483− * mutated or destroyed. However, the memory it points to
7484− * may be mutated.
7244+ * OBJ_RELEASE - Points to a initialized bpf_dynptr that will be
7245+ * destroyed.
74857246 *
7486− * None - Points to a initialized dynptr that can be mutated and
7487− * destroyed, including mutation of the memory it points
7488− * to.
7247+ * None - Points to a initialized dynptr that cannot be
7248+ * reinitialized or destroyed. However, the view of the
7249+ * dynptr and the memory it points to may be mutated.
74897250 */
74907251 if (arg_type & MEM_UNINIT) {
74917252 int i;
@@ -7497,45 +7258,58 @@ static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn · 5 unmodified lines
74977258
74987259 /* we write BPF_DW bits (8 bytes) at a time */
74997260 for (i = 0; i < BPF_DYNPTR_SIZE; i += 8) {
7500− err = check_mem_access(env, insn_idx, regno,
7261+ err = check_mem_access(env, insn_idx, reg, argno,
75017262 i, BPF_DW, BPF_WRITE, -1, false, false);
75027263 if (err)
75037264 return err;
75047265 }
75057266
7506− err = mark_stack_slots_dynptr(env, reg, arg_type, insn_idx, clone_ref_obj_id);
7507− } else /* MEM_RDONLY and None case from above */ {
7267+ err = mark_stack_slots_dynptr(env, reg, arg_type, insn_idx, ref_obj, dynptr);
7268+ } else /* OBJ_RELEASE and None case from above */ {
75087269 /* For the reg->type == PTR_TO_STACK case, bpf_dynptr is never const */
7509− if (reg->type == CONST_PTR_TO_DYNPTR && !(arg_type & MEM_RDONLY)) {
7510− verbose(env, "cannot pass pointer to const bpf_dynptr, the helper mutates it\n");
7270+ if (reg->type == CONST_PTR_TO_DYNPTR && (arg_type & OBJ_RELEASE)) {
7271+ verbose(env, "CONST_PTR_TO_DYNPTR cannot be released\n");
75117272 return -EINVAL;
75127273 }
75137274
75147275 if (!is_dynptr_reg_valid_init(env, reg)) {
7515− verbose(env,
7516− "Expected an initialized dynptr as arg #%d\n",
7517− regno - 1);
7276+ verbose(env, "Expected an initialized dynptr as %s\n",
7277+ reg_arg_name(env, argno));
75187278 return -EINVAL;
75197279 }
75207280
7521− /* Fold modifiers (in this case, MEM_RDONLY) when checking expected type */
7522− if (!is_dynptr_type_expected(env, reg, arg_type & ~MEM_RDONLY)) {
7281+ /* Fold modifiers (in this case, OBJ_RELEASE) when checking expected type */
7282+ if (!is_dynptr_type_expected(env, reg, arg_type & ~OBJ_RELEASE)) {
75237283 verbose(env,
7524− "Expected a dynptr of type %s as arg #%d\n",
7525− dynptr_type_str(arg_to_dynptr_type(arg_type)), regno - 1);
7284+ "Expected a dynptr of type %s as %s\n",
7285+ dynptr_type_str(arg_to_dynptr_type(arg_type)),
7286+ reg_arg_name(env, argno));
75267287 return -EINVAL;
75277288 }
75287289
7529− err = mark_dynptr_read(env, reg);
7530− }
7531− return err;
7532−}
7290+ if (reg->type != CONST_PTR_TO_DYNPTR) {
7291+ struct bpf_func_state *state = bpf_func(env, reg);
75337292
7534−static u32 iter_ref_obj_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int spi)
7535−{
7536− struct bpf_func_state *state = bpf_func(env, reg);
7293+ spi = dynptr_get_spi(env, reg);
7294+ if (spi < 0)
7295+ return spi;
7296+
7297+ /*
7298+ * For CONST_PTR_TO_DYNPTR, reg is already scratched by check_reg_arg
7299+ * in check_helper_call and mark_btf_func_reg_size in check_kfunc_call.
7300+ */
7301+ mark_stack_slots_scratched(env, spi, BPF_DYNPTR_NR_SLOTS);
75377302
7538− return state->stack[spi].spilled_ptr.ref_obj_id;
7303+ reg = &state->stack[spi].spilled_ptr;
7304+ }
7305+
7306+ if (dynptr) {
7307+ dynptr->type = reg->dynptr.type;
7308+ dynptr->id = reg->id;
7309+ dynptr->parent_id = reg->parent_id;
7310+ }
7311+ }
7312+ return err;
75397313 }
75407314
75417315 static bool is_iter_kfunc(struct bpf_kfunc_call_arg_meta *meta)
@@ -7567,15 +7341,17 @@ static bool is_kfunc_arg_iter(struct bpf_kfunc_call_arg_meta *meta, int arg_idx, · 25 unmodified lines
75677341 return btf_param_match_suffix(meta->btf, arg, "__iter");
75687342 }
75697343
7570−static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_idx,
7344+static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int insn_idx,
75717345 struct bpf_kfunc_call_arg_meta *meta)
75727346 {
7573− struct bpf_reg_state *reg = reg_state(env, regno);
7347+ struct bpf_func_state *state = bpf_func(env, reg);
75747348 const struct btf_type *t;
7349+ u32 arg_idx = arg_idx_from_argno(argno);
75757350 int spi, err, i, nr_slots, btf_id;
75767351
75777352 if (reg->type != PTR_TO_STACK) {
7578− verbose(env, "arg#%d expected pointer to an iterator on stack\n", regno - 1);
7353+ verbose(env, "%s expected pointer to an iterator on stack\n",
7354+ reg_arg_name(env, argno));
75797355 return -EINVAL;
75807356 }
75817357
@@ -7585,9 +7361,10 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 3 unmodified lines
75857361 * to any kfunc, if arg has "__iter" suffix, we need to be a bit more
75867362 * conservative here.
75877363 */
7588− btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, regno - 1);
7364+ btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, arg_idx);
75897365 if (btf_id < 0) {
7590− verbose(env, "expected valid iter pointer as arg #%d\n", regno - 1);
7366+ verbose(env, "expected valid iter pointer as %s\n",
7367+ reg_arg_name(env, argno));
75917368 return -EINVAL;
75927369 }
75937370 t = btf_type_by_id(meta->btf, btf_id);
@@ -7596,13 +7373,13 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 2 unmodified lines
75967373 if (is_iter_new_kfunc(meta)) {
75977374 /* bpf_iter_<type>_new() expects pointer to uninit iter state */
75987375 if (!is_iter_reg_valid_uninit(env, reg, nr_slots)) {
7599− verbose(env, "expected uninitialized iter_%s as arg #%d\n",
7600− iter_type_str(meta->btf, btf_id), regno - 1);
7376+ verbose(env, "expected uninitialized iter_%s as %s\n",
7377+ iter_type_str(meta->btf, btf_id), reg_arg_name(env, argno));
76017378 return -EINVAL;
76027379 }
76037380
76047381 for (i = 0; i < nr_slots * 8; i += BPF_REG_SIZE) {
7605− err = check_mem_access(env, insn_idx, regno,
7382+ err = check_mem_access(env, insn_idx, reg, argno,
76067383 i, BPF_DW, BPF_WRITE, -1, false, false);
76077384 if (err)
76087385 return err;
@@ -7620,8 +7397,8 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 11 unmodified lines
76207397 case 0:
76217398 break;
76227399 case -EINVAL:
7623− verbose(env, "expected an initialized iter_%s as arg #%d\n",
7624− iter_type_str(meta->btf, btf_id), regno - 1);
7400+ verbose(env, "expected an initialized iter_%s as %s\n",
7401+ iter_type_str(meta->btf, btf_id), reg_arg_name(env, argno));
76257402 return err;
76267403 case -EPROTO:
76277404 verbose(env, "expected an RCU CS when using %s\n", meta->func_name);
@@ -7634,14 +7411,12 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 6 unmodified lines
76347411 if (spi < 0)
76357412 return spi;
76367413
7637− err = mark_iter_read(env, reg, spi, nr_slots);
7638− if (err)
7639− return err;
7414+ mark_stack_slots_scratched(env, spi, nr_slots);
76407415
76417416 /* remember meta->iter info for process_iter_next_call() */
76427417 meta->iter.spi = spi;
76437418 meta->iter.frameno = reg->frameno;
7644− meta->ref_obj_id = iter_ref_obj_id(env, reg, spi);
7419+ update_ref_obj(&meta->ref_obj, &state->stack[spi].spilled_ptr);
76457420
76467421 if (is_iter_destroy_kfunc(meta)) {
76477422 err = unmark_stack_slots_iter(env, reg, nr_slots);
@@ -8041,12 +7816,11 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { · 393 unmodified lines
80417816 [ARG_PTR_TO_DYNPTR] = &dynptr_types,
80427817 };
80437818
8044−static int check_reg_type(struct bpf_verifier_env *env, u32 regno,
7819+static int check_reg_type(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
80457820 enum bpf_arg_type arg_type,
80467821 const u32 *arg_btf_id,
80477822 struct bpf_call_arg_meta *meta)
80487823 {
8049− struct bpf_reg_state *reg = reg_state(env, regno);
80507824 enum bpf_reg_type expected, type = reg->type;
80517825 const struct bpf_reg_types *compatible;
80527826 int i, j, err;
@@ -8077,7 +7851,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, · 24 unmodified lines
80777851 type &= ~DYNPTR_TYPE_FLAG_MASK;
80787852
80797853 /* Local kptr types are allowed as the source argument of bpf_kptr_xchg */
8080− if (meta->func_id == BPF_FUNC_kptr_xchg && type_is_alloc(type) && regno == BPF_REG_2) {
7854+ if (meta->func_id == BPF_FUNC_kptr_xchg && type_is_alloc(type) && reg_from_argno(argno) == BPF_REG_2) {
80817855 type &= ~MEM_ALLOC;
80827856 type &= ~MEM_PERCPU;
80837857 }
@@ -8091,7 +7865,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, · 7 unmodified lines
80917865 goto found;
80927866 }
80937867
8094− verbose(env, "R%d type=%s expected=", regno, reg_type_str(env, reg->type));
7868+ verbose(env, "%s type=%s expected=", reg_arg_name(env, argno), reg_type_str(env, reg->type));
80957869 for (j = 0; j + 1 < i; j++)
80967870 verbose(env, "%s, ", reg_type_str(env, compatible->types[j]));
80977871 verbose(env, "%s\n", reg_type_str(env, compatible->types[j]));
@@ -8104,9 +7878,9 @@ found: · 6 unmodified lines
81047878 if (compatible == &mem_types) {
81057879 if (!(arg_type & MEM_RDONLY)) {
81067880 verbose(env,
8107− "%s() may write into memory pointed by R%d type=%s\n",
7881+ "%s() may write into memory pointed by %s type=%s\n",
81087882 func_id_name(meta->func_id),
8109− regno, reg_type_str(env, reg->type));
7883+ reg_arg_name(env, argno), reg_type_str(env, reg->type));
81107884 return -EACCES;
81117885 }
81127886 return 0;
@@ -8129,7 +7903,8 @@ found: · 16 unmodified lines
81297903
81307904 if (type_may_be_null(reg->type) &&
81317905 (!type_may_be_null(arg_type) || arg_type_is_release(arg_type))) {
8132− verbose(env, "Possibly NULL pointer passed to helper arg%d\n", regno);
7906+ verbose(env, "Possibly NULL pointer passed to helper %s\n",
7907+ reg_arg_name(env, argno));
81337908 return -EACCES;
81347909 }
81357910
@@ -8142,25 +7917,26 @@ found: · 6 unmodified lines
81427917 }
81437918
81447919 if (meta->func_id == BPF_FUNC_kptr_xchg) {
8145− if (map_kptr_match_type(env, meta->kptr_field, reg, regno))
7920+ if (map_kptr_match_type(env, meta->kptr_field, reg, reg_from_argno(argno)))
81467921 return -EACCES;
81477922 } else {
81487923 if (arg_btf_id == BPF_PTR_POISON) {
81497924 verbose(env, "verifier internal error:");
8150− verbose(env, "R%d has non-overwritten BPF_PTR_POISON type\n",
8151− regno);
7925+ verbose(env, "%s has non-overwritten BPF_PTR_POISON type\n",
7926+ reg_arg_name(env, argno));
81527927 return -EACCES;
81537928 }
81547929
8155− err = __check_ptr_off_reg(env, reg, regno, true);
7930+ err = __check_ptr_off_reg(env, reg, argno, true);
81567931 if (err)
81577932 return err;
81587933
81597934 if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id,
81607935 reg->var_off.value, btf_vmlinux, *arg_btf_id,
81617936 strict_type_match)) {
8162− verbose(env, "R%d is of type %s but %s is expected\n",
8163− regno, btf_type_name(reg->btf, reg->btf_id),
7937+ verbose(env, "%s is of type %s but %s is expected\n",
7938+ reg_arg_name(env, argno),
7939+ btf_type_name(reg->btf, reg->btf_id),
81647940 btf_type_name(btf_vmlinux, *arg_btf_id));
81657941 return -EACCES;
81667942 }
@@ -8177,8 +7953,11 @@ found: · 10 unmodified lines
81777953 return -EFAULT;
81787954 }
81797955 /* Check if local kptr in src arg matches kptr in dst arg */
8180− if (meta->func_id == BPF_FUNC_kptr_xchg && regno == BPF_REG_2) {
8181− if (map_kptr_match_type(env, meta->kptr_field, reg, regno))
7956+ if (meta->func_id == BPF_FUNC_kptr_xchg) {
7957+ int regno = reg_from_argno(argno);
7958+
7959+ if (regno == BPF_REG_2 &&
7960+ map_kptr_match_type(env, meta->kptr_field, reg, regno))
81827961 return -EACCES;
81837962 }
81847963 break;
@@ -8212,7 +7991,7 @@ reg_find_field_offset(const struct bpf_reg_state *reg, s32 off, u32 fields) · 27 unmodified lines
82127991 }
82137992
82147993 static int check_func_arg_reg_off(struct bpf_verifier_env *env,
8215− const struct bpf_reg_state *reg, int regno,
7994+ const struct bpf_reg_state *reg, argno_t argno,
82167995 enum bpf_arg_type arg_type)
82177996 {
82187997 u32 type = reg->type;
@@ -8220,7 +7999,7 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, · 1 unmodified line
82207999 /* When referenced register is passed to release function, its fixed
82218000 * offset must be 0.
82228001 *
8223− * We will check arg_type_is_release reg has ref_obj_id when storing
8002+ * We will check arg_type_is_release reg has id when storing
82248003 * meta->release_regno.
82258004 */
82268005 if (arg_type_is_release(arg_type)) {
@@ -8238,8 +8017,8 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, · 11 unmodified lines
82388017 * to give the user a better error message.
82398018 */
82408019 if (!tnum_is_const(reg->var_off) || reg->var_off.value != 0) {
8241− verbose(env, "R%d must have zero offset when passed to release func or trusted arg to kfunc\n",
8242− regno);
8020+ verbose(env, "%s must have zero offset when passed to release func or trusted arg to kfunc\n",
8021+ reg_arg_name(env, argno));
82438022 return -EINVAL;
82448023 }
82458024 }
@@ -8272,93 +8051,27 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, · 26 unmodified lines
82728051 * its fixed offset must be 0. In the other cases, fixed offset
82738052 * can be non-zero. This was already checked above. So pass
82748053 * fixed_off_ok as true to allow fixed offset for all other
8275− * cases. var_off always must be 0 for PTR_TO_BTF_ID, hence we
8276− * still need to do checks instead of returning.
8277− */
8278− return __check_ptr_off_reg(env, reg, regno, true);
8279− case PTR_TO_CTX:
8280− /*
8281− * Allow fixed and variable offsets for syscall context, but
8282− * only when the argument is passed as memory, not ctx,
8283− * otherwise we may get modified ctx in tail called programs and
8284− * global subprogs (that may act as extension prog hooks).
8285− */
8286− if (arg_type != ARG_PTR_TO_CTX && is_var_ctx_off_allowed(env->prog))
8287− return 0;
8288− fallthrough;
8289− default:
8290− return __check_ptr_off_reg(env, reg, regno, false);
8291− }
8292−}
8293−
8294−static struct bpf_reg_state *get_dynptr_arg_reg(struct bpf_verifier_env *env,
8295− const struct bpf_func_proto *fn,
8296− struct bpf_reg_state *regs)
8297−{
8298− struct bpf_reg_state *state = NULL;
8299− int i;
8300−
8301− for (i = 0; i < MAX_BPF_FUNC_REG_ARGS; i++)
8302− if (arg_type_is_dynptr(fn->arg_type[i])) {
8303− if (state) {
8304− verbose(env, "verifier internal error: multiple dynptr args\n");
8305− return NULL;
8306− }
8307− state = &regs[BPF_REG_1 + i];
8308− }
8309−
8310− if (!state)
8311− verbose(env, "verifier internal error: no dynptr arg found\n");
8312−
8313− return state;
8314−}
8315−
8316−static int dynptr_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
8317−{
8318− struct bpf_func_state *state = bpf_func(env, reg);
8319− int spi;
8320−
8321− if (reg->type == CONST_PTR_TO_DYNPTR)
8322− return reg->id;
8323− spi = dynptr_get_spi(env, reg);
8324− if (spi < 0)
8325− return spi;
8326− return state->stack[spi].spilled_ptr.id;
8327−}
8328−
8329−static int dynptr_ref_obj_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
8330−{
8331− struct bpf_func_state *state = bpf_func(env, reg);
8332− int spi;
8333−
8334− if (reg->type == CONST_PTR_TO_DYNPTR)
8335− return reg->ref_obj_id;
8336− spi = dynptr_get_spi(env, reg);
8337− if (spi < 0)
8338− return spi;
8339− return state->stack[spi].spilled_ptr.ref_obj_id;
8340−}
8341−
8342−static enum bpf_dynptr_type dynptr_get_type(struct bpf_verifier_env *env,
8343− struct bpf_reg_state *reg)
8344−{
8345− struct bpf_func_state *state = bpf_func(env, reg);
8346− int spi;
8347−
8348− if (reg->type == CONST_PTR_TO_DYNPTR)
8349− return reg->dynptr.type;
8350−
8351− spi = bpf_get_spi(reg->var_off.value);
8352− if (spi < 0) {
8353− verbose(env, "verifier internal error: invalid spi when querying dynptr type\n");
8354− return BPF_DYNPTR_TYPE_INVALID;
8054+ * cases. var_off always must be 0 for PTR_TO_BTF_ID, hence we
8055+ * still need to do checks instead of returning.
8056+ */
8057+ return __check_ptr_off_reg(env, reg, argno, true);
8058+ case PTR_TO_CTX:
8059+ /*
8060+ * Allow fixed and variable offsets for syscall context, but
8061+ * only when the argument is passed as memory, not ctx,
8062+ * otherwise we may get modified ctx in tail called programs and
8063+ * global subprogs (that may act as extension prog hooks).
8064+ */
8065+ if (arg_type != ARG_PTR_TO_CTX && is_var_ctx_off_allowed(env->prog))
8066+ return 0;
8067+ fallthrough;
8068+ default:
8069+ return __check_ptr_off_reg(env, reg, argno, false);
83558070 }
8356−
8357− return state->stack[spi].spilled_ptr.dynptr.type;
83588071 }
83598072
8360−static int check_reg_const_str(struct bpf_verifier_env *env,
8361− struct bpf_reg_state *reg, u32 regno)
8073+static int check_arg_const_str(struct bpf_verifier_env *env,
8074+ struct bpf_reg_state *reg, argno_t argno)
83628075 {
83638076 struct bpf_map *map = reg->map_ptr;
83648077 int err;
@@ -8370,17 +8083,18 @@ static int check_reg_const_str(struct bpf_verifier_env *env, · 5 unmodified lines
83708083 return -EINVAL;
83718084
83728085 if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) {
8373− verbose(env, "R%d points to insn_array map which cannot be used as const string\n", regno);
8086+ verbose(env, "%s points to insn_array map which cannot be used as const string\n",
8087+ reg_arg_name(env, argno));
83748088 return -EACCES;
83758089 }
83768090
83778091 if (!bpf_map_is_rdonly(map)) {
8378− verbose(env, "R%d does not point to a readonly map'\n", regno);
8092+ verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno));
83798093 return -EACCES;
83808094 }
83818095
83828096 if (!tnum_is_const(reg->var_off)) {
8383− verbose(env, "R%d is not a constant address'\n", regno);
8097+ verbose(env, "%s is not a constant address'\n", reg_arg_name(env, argno));
83848098 return -EACCES;
83858099 }
83868100
@@ -8389,7 +8103,7 @@ static int check_reg_const_str(struct bpf_verifier_env *env, · 2 unmodified lines
83898103 return -EACCES;
83908104 }
83918105
8392− err = check_map_access(env, regno, 0,
8106+ err = check_map_access(env, reg, argno, 0,
83938107 map->value_size - reg->var_off.value, false,
83948108 ACCESS_HELPER);
83958109 if (err)
@@ -8471,7 +8185,7 @@ static int get_constant_map_key(struct bpf_verifier_env *env, · 75 unmodified lines
84718185 return 0;
84728186 }
84738187
8474−static bool can_elide_value_nullness(enum bpf_map_type type);
8188+static bool can_elide_value_nullness(const struct bpf_map *map);
84758189
84768190 static int check_func_arg(struct bpf_verifier_env *env, u32 arg,
84778191 struct bpf_call_arg_meta *meta,
@@ -8481,6 +8195,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 3 unmodified lines
84818195 u32 regno = BPF_REG_1 + arg;
84828196 struct bpf_reg_state *reg = reg_state(env, regno);
84838197 enum bpf_arg_type arg_type = fn->arg_type[arg];
8198+ argno_t argno = argno_from_arg(arg + 1);
84848199 enum bpf_reg_type type = reg->type;
84858200 u32 *arg_btf_id = NULL;
84868201 u32 key_size;
@@ -8525,56 +8240,24 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 38 unmodified lines
85258240 base_type(arg_type) == ARG_PTR_TO_SPIN_LOCK)
85268241 arg_btf_id = fn->arg_btf_id[arg];
85278242
8528− err = check_reg_type(env, regno, arg_type, arg_btf_id, meta);
8243+ err = check_reg_type(env, reg, argno_from_reg(regno), arg_type, arg_btf_id, meta);
85298244 if (err)
85308245 return err;
85318246
8532− err = check_func_arg_reg_off(env, reg, regno, arg_type);
8247+ err = check_func_arg_reg_off(env, reg, argno_from_reg(regno), arg_type);
85338248 if (err)
85348249 return err;
85358250
85368251 skip_type_check:
8537− if (arg_type_is_release(arg_type)) {
8538− if (arg_type_is_dynptr(arg_type)) {
8539− struct bpf_func_state *state = bpf_func(env, reg);
8540− int spi;
8541−
8542− /* Only dynptr created on stack can be released, thus
8543− * the get_spi and stack state checks for spilled_ptr
8544− * should only be done before process_dynptr_func for
8545− * PTR_TO_STACK.
8546− */
8547− if (reg->type == PTR_TO_STACK) {
8548− spi = dynptr_get_spi(env, reg);
8549− if (spi < 0 || !state->stack[spi].spilled_ptr.ref_obj_id) {
8550− verbose(env, "arg %d is an unacquired reference\n", regno);
8551− return -EINVAL;
8552− }
8553− } else {
8554− verbose(env, "cannot release unowned const bpf_dynptr\n");
8555− return -EINVAL;
8556− }
8557− } else if (!reg->ref_obj_id && !bpf_register_is_null(reg)) {
8558− verbose(env, "R%d must be referenced when passed to release function\n",
8559− regno);
8560− return -EINVAL;
8561− }
8562− if (meta->release_regno) {
8563− verifier_bug(env, "more than one release argument");
8564− return -EFAULT;
8565− }
8566− meta->release_regno = regno;
8252+ if (arg_type_is_release(arg_type) && !arg_type_is_dynptr(arg_type) &&
8253+ !reg_is_referenced(env, reg) && !bpf_register_is_null(reg)) {
8254+ verbose(env, "release helper %s expects referenced PTR_TO_BTF_ID passed to %s\n",
8255+ func_id_name(meta->func_id), reg_arg_name(env, argno));
8256+ return -EINVAL;
85678257 }
85688258
8569− if (reg->ref_obj_id && base_type(arg_type) != ARG_KPTR_XCHG_DEST) {
8570− if (meta->ref_obj_id) {
8571− verbose(env, "more than one arg with ref_obj_id R%d %u %u",
8572− regno, reg->ref_obj_id,
8573− meta->ref_obj_id);
8574− return -EACCES;
8575− }
8576− meta->ref_obj_id = reg->ref_obj_id;
8577− }
8259+ if (reg_is_referenced(env, reg))
8260+ update_ref_obj(&meta->ref_obj, reg);
85788261
85798262 switch (base_type(arg_type)) {
85808263 case ARG_CONST_MAP_PTR:
@@ -8618,10 +8301,10 @@ skip_type_check: · 37 unmodified lines
86188301 return -EFAULT;
86198302 }
86208303 key_size = meta->map.ptr->key_size;
8621− err = check_helper_mem_access(env, regno, key_size, BPF_READ, false, NULL);
8304+ err = check_helper_mem_access(env, reg, argno_from_reg(regno), key_size, BPF_READ, false, NULL);
86228305 if (err)
86238306 return err;
8624− if (can_elide_value_nullness(meta->map.ptr->map_type)) {
8307+ if (can_elide_value_nullness(meta->map.ptr)) {
86258308 err = get_constant_map_key(env, reg, key_size, &meta->const_map_key);
86268309 if (err < 0) {
86278310 meta->const_map_key = -1;
@@ -8645,7 +8328,7 @@ skip_type_check: · 17 unmodified lines
86458328 return -EFAULT;
86468329 }
86478330 meta->raw_mode = arg_type & MEM_UNINIT;
8648− err = check_helper_mem_access(env, regno, meta->map.ptr->value_size,
8331+ err = check_helper_mem_access(env, reg, argno_from_reg(regno), meta->map.ptr->value_size,
86498332 arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ,
86508333 false, meta);
86518334 break;
@@ -8663,11 +8346,11 @@ skip_type_check: · 11 unmodified lines
86638346 return -EACCES;
86648347 }
86658348 if (meta->func_id == BPF_FUNC_spin_lock) {
8666− err = process_spin_lock(env, regno, PROCESS_SPIN_LOCK);
8349+ err = process_spin_lock(env, reg, argno_from_reg(regno), PROCESS_SPIN_LOCK);
86678350 if (err)
86688351 return err;
86698352 } else if (meta->func_id == BPF_FUNC_spin_unlock) {
8670− err = process_spin_lock(env, regno, 0);
8353+ err = process_spin_lock(env, reg, argno_from_reg(regno), 0);
86718354 if (err)
86728355 return err;
86738356 } else {
@@ -8676,7 +8359,7 @@ skip_type_check: · 2 unmodified lines
86768359 }
86778360 break;
86788361 case ARG_PTR_TO_TIMER:
8679− err = process_timer_helper(env, regno, meta);
8362+ err = process_timer_helper(env, reg, argno_from_reg(regno), meta);
86808363 if (err)
86818364 return err;
86828365 break;
@@ -8689,7 +8372,7 @@ skip_type_check: · 6 unmodified lines
86898372 */
86908373 meta->raw_mode = arg_type & MEM_UNINIT;
86918374 if (arg_type & MEM_FIXED_SIZE) {
8692− err = check_helper_mem_access(env, regno, fn->arg_size[arg],
8375+ err = check_helper_mem_access(env, reg, argno_from_reg(regno), fn->arg_size[arg],
86938376 arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ,
86948377 false, meta);
86958378 if (err)
@@ -8699,19 +8382,22 @@ skip_type_check: · 3 unmodified lines
86998382 }
87008383 break;
87018384 case ARG_CONST_SIZE:
8702− err = check_mem_size_reg(env, reg, regno,
8385+ err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1),
8386+ argno_from_reg(regno),
87038387 fn->arg_type[arg - 1] & MEM_WRITE ?
87048388 BPF_WRITE : BPF_READ,
87058389 false, meta);
87068390 break;
87078391 case ARG_CONST_SIZE_OR_ZERO:
8708− err = check_mem_size_reg(env, reg, regno,
8392+ err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1),
8393+ argno_from_reg(regno),
87098394 fn->arg_type[arg - 1] & MEM_WRITE ?
87108395 BPF_WRITE : BPF_READ,
87118396 true, meta);
87128397 break;
87138398 case ARG_PTR_TO_DYNPTR:
8714− err = process_dynptr_func(env, regno, insn_idx, arg_type, 0);
8399+ err = process_dynptr_func(env, reg, argno_from_reg(regno), insn_idx, arg_type, &meta->ref_obj,
8400+ &meta->dynptr);
87158401 if (err)
87168402 return err;
87178403 break;
@@ -8728,7 +8414,7 @@ skip_type_check: · 10 unmodified lines
87288414 break;
87298415 case ARG_PTR_TO_CONST_STR:
87308416 {
8731− err = check_reg_const_str(env, reg, regno);
8417+ err = check_arg_const_str(env, reg, argno_from_reg(regno));
87328418 if (err)
87338419 return err;
87348420 break;
@@ -9130,11 +8816,29 @@ static bool check_mem_arg_rw_flag_ok(const struct bpf_func_proto *fn) · 395 unmodified lines
91308816 return true;
91318817 }
91328818
9133−static int check_func_proto(const struct bpf_func_proto *fn)
8819+static bool check_proto_release_reg(const struct bpf_func_proto *fn, struct bpf_call_arg_meta *meta)
8820+{
8821+ int i;
8822+
8823+ for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++) {
8824+ enum bpf_arg_type arg_type = fn->arg_type[i];
8825+
8826+ if (arg_type_is_release(arg_type)) {
8827+ if (meta->release_regno)
8828+ return false;
8829+ meta->release_regno = i + 1;
8830+ }
8831+ }
8832+
8833+ return true;
8834+}
8835+
8836+static int check_func_proto(const struct bpf_func_proto *fn, struct bpf_call_arg_meta *meta)
91348837 {
91358838 return check_raw_mode_ok(fn) &&
91368839 check_arg_pair_ok(fn) &&
91378840 check_mem_arg_rw_flag_ok(fn) &&
8841+ check_proto_release_reg(fn, meta) &&
91388842 check_btf_id_ok(fn) ? 0 : -EINVAL;
91398843 }
91408844
@@ -9181,14 +8885,14 @@ static void mark_pkt_end(struct bpf_verifier_state *vstate, int regn, bool range · 40 unmodified lines
91818885 reg->range = AT_PKT_END;
91828886 }
91838887
9184−static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id)
8888+static int release_reference_nomark(struct bpf_verifier_state *state, int id)
91858889 {
91868890 int i;
91878891
91888892 for (i = 0; i < state->acquired_refs; i++) {
91898893 if (state->refs[i].type != REF_TYPE_PTR)
91908894 continue;
9191− if (state->refs[i].id == ref_obj_id) {
8895+ if (state->refs[i].id == id) {
91928896 release_reference_state(state, i);
91938897 return 0;
91948898 }
@@ -9196,26 +8900,83 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob · 1 unmodified line
91968900 return -EINVAL;
91978901 }
91988902
9199−/* The pointer with the specified id has released its reference to kernel
9200− * resources. Identify all copies of the same pointer and clear the reference.
9201− *
9202− * This is the release function corresponding to acquire_reference(). Idempotent.
9203− */
9204−static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
8903+static int idstack_push(struct bpf_idmap *idmap, u32 id)
8904+{
8905+ int i;
8906+
8907+ if (!id)
8908+ return 0;
8909+
8910+ for (i = 0; i < idmap->cnt; i++)
8911+ if (idmap->map[i].old == id)
8912+ return 0;
8913+
8914+ if (WARN_ON_ONCE(idmap->cnt >= BPF_ID_MAP_SIZE))
8915+ return -EFAULT;
8916+
8917+ idmap->map[idmap->cnt++].old = id;
8918+ return 0;
8919+}
8920+
8921+static int idstack_pop(struct bpf_idmap *idmap)
8922+{
8923+ if (!idmap->cnt)
8924+ return 0;
8925+
8926+ return idmap->map[--idmap->cnt].old;
8927+}
8928+
8929+/* Release id and objects derived from it iteratively in a DFS manner */
8930+static int release_reference(struct bpf_verifier_env *env, int id)
92058931 {
8932+ u32 mask = (1 << STACK_SPILL) | (1 << STACK_DYNPTR);
92068933 struct bpf_verifier_state *vstate = env->cur_state;
8934+ struct bpf_idmap *idstack = &env->idmap_scratch;
8935+ struct bpf_stack_state *stack;
92078936 struct bpf_func_state *state;
92088937 struct bpf_reg_state *reg;
9209− int err;
8938+ int i, err;
92108939
9211− err = release_reference_nomark(vstate, ref_obj_id);
8940+ idstack->cnt = 0;
8941+ err = idstack_push(idstack, id);
92128942 if (err)
92138943 return err;
92148944
9215− bpf_for_each_reg_in_vstate(vstate, state, reg, ({
9216− if (reg->ref_obj_id == ref_obj_id)
9217− mark_reg_invalid(env, reg);
9218− }));
8945+ if (find_reference_state(vstate, id))
8946+ WARN_ON_ONCE(release_reference_nomark(vstate, id));
8947+
8948+ while ((id = idstack_pop(idstack))) {
8949+ /*
8950+ * Child references are inaccessible after parent is released,
8951+ * any child references that exist at this point are a leak.
8952+ */
8953+ for (i = 0; i < vstate->acquired_refs; i++) {
8954+ if (vstate->refs[i].type != REF_TYPE_PTR)
8955+ continue;
8956+ if (vstate->refs[i].parent_id != id)
8957+ continue;
8958+ verbose(env, "Leaking reference id=%d alloc_insn=%d. Release it first.\n",
8959+ vstate->refs[i].id, vstate->refs[i].insn_idx);
8960+ return -EINVAL;
8961+ }
8962+
8963+ bpf_for_each_reg_in_vstate_mask(vstate, state, reg, stack, mask, ({
8964+ if (reg->id != id && reg->parent_id != id)
8965+ continue;
8966+
8967+ /* Free objects derived from the current object */
8968+ if (reg->parent_id == id) {
8969+ err = idstack_push(idstack, reg->id);
8970+ if (err)
8971+ return err;
8972+ }
8973+
8974+ if (!stack || stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL)
8975+ mark_reg_invalid(env, reg);
8976+ else if (stack->slot_type[BPF_REG_SIZE - 1] == STACK_DYNPTR)
8977+ invalidate_dynptr(env, stack);
8978+ }));
8979+ }
92198980
92208981 return 0;
92218982 }
@@ -9231,6 +8992,42 @@ static void invalidate_non_owning_refs(struct bpf_verifier_env *env) · 9 unmodified lines
92318992 }));
92328993 }
92338994
8995+static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env)
8996+{
8997+ struct bpf_stack_state *stack;
8998+ struct bpf_func_state *state;
8999+ struct bpf_reg_state *reg;
9000+ u32 clear_mask = (1 << STACK_SPILL) | (1 << STACK_ITER);
9001+
9002+ bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, clear_mask, ({
9003+ if (reg->type & MEM_RCU) {
9004+ reg->type &= ~(MEM_RCU | PTR_MAYBE_NULL);
9005+ reg->type |= PTR_UNTRUSTED;
9006+ }
9007+ }));
9008+}
9009+
9010+static int ref_convert_alloc_rcu_protected(struct bpf_verifier_env *env, u32 id)
9011+{
9012+ struct bpf_func_state *state;
9013+ struct bpf_reg_state *reg;
9014+ int err;
9015+
9016+ err = release_reference_nomark(env->cur_state, id);
9017+
9018+ bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({
9019+ if (reg->id != id)
9020+ continue;
9021+ if ((reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU)) {
9022+ reg->id = 0;
9023+ reg->type &= ~MEM_ALLOC;
9024+ reg->type |= MEM_RCU;
9025+ }
9026+ }));
9027+
9028+ return err;
9029+}
9030+
92349031 static void clear_caller_saved_regs(struct bpf_verifier_env *env,
92359032 struct bpf_reg_state *regs)
92369033 {
@@ -9243,6 +9040,15 @@ static void clear_caller_saved_regs(struct bpf_verifier_env *env, · 6 unmodified lines
92439040 }
92449041 }
92459042
9043+static void invalidate_outgoing_stack_args(const struct bpf_verifier_env *env,
9044+ struct bpf_func_state *state)
9045+{
9046+ int i, nslots = state->out_stack_arg_cnt;
9047+
9048+ for (i = 0; i < nslots; i++)
9049+ bpf_mark_reg_not_init(env, &state->stack_arg_regs[i]);
9050+}
9051+
92469052 typedef int (*set_callee_state_fn)(struct bpf_verifier_env *env,
92479053 struct bpf_func_state *caller,
92489054 struct bpf_func_state *callee,
@@ -9305,11 +9111,23 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 56 unmodified lines
93059111 struct bpf_reg_state *regs)
93069112 {
93079113 struct bpf_subprog_info *sub = subprog_info(env, subprog);
9114+ struct bpf_func_state *caller = cur_func(env);
93089115 struct bpf_verifier_log *log = &env->log;
9116+ struct ref_obj_desc ref_obj = {};
93099117 u32 i;
9310− int ret;
9118+ int ret, err;
93119119
93129120 ret = btf_prepare_func_args(env, subprog);
9121+ if (ret) {
9122+ if (bpf_in_stack_arg_cnt(sub) > 0) {
9123+ err = check_outgoing_stack_args(env, caller, sub->arg_cnt);
9124+ if (err)
9125+ return err;
9126+ }
9127+ return ret;
9128+ }
9129+
9130+ ret = check_outgoing_stack_args(env, caller, sub->arg_cnt);
93139131 if (ret)
93149132 return ret;
93159133
@@ -9317,13 +9135,13 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 1 unmodified line
93179135 * verifier sees.
93189136 */
93199137 for (i = 0; i < sub->arg_cnt; i++) {
9320− u32 regno = i + 1;
9321− struct bpf_reg_state *reg = &regs[regno];
9138+ argno_t argno = argno_from_arg(i + 1);
9139+ struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, i);
93229140 struct bpf_subprog_arg_info *arg = &sub->args[i];
93239141
93249142 if (arg->arg_type == ARG_ANYTHING) {
93259143 if (reg->type != SCALAR_VALUE) {
9326− bpf_log(log, "R%d is not a scalar\n", regno);
9144+ bpf_log(log, "%s is not a scalar\n", reg_arg_name(env, argno));
93279145 return -EINVAL;
93289146 }
93299147 } else if (arg->arg_type & PTR_UNTRUSTED) {
@@ -9333,24 +9151,26 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 3 unmodified lines
93339151 * invalid memory access.
93349152 */
93359153 } else if (arg->arg_type == ARG_PTR_TO_CTX) {
9336− ret = check_func_arg_reg_off(env, reg, regno, ARG_PTR_TO_CTX);
9154+ ret = check_func_arg_reg_off(env, reg, argno, ARG_PTR_TO_CTX);
93379155 if (ret < 0)
93389156 return ret;
93399157 /* If function expects ctx type in BTF check that caller
93409158 * is passing PTR_TO_CTX.
93419159 */
93429160 if (reg->type != PTR_TO_CTX) {
9343− bpf_log(log, "arg#%d expects pointer to ctx\n", i);
9161+ bpf_log(log, "%s expects pointer to ctx\n",
9162+ reg_arg_name(env, argno));
93449163 return -EINVAL;
93459164 }
93469165 } else if (base_type(arg->arg_type) == ARG_PTR_TO_MEM) {
9347− ret = check_func_arg_reg_off(env, reg, regno, ARG_DONTCARE);
9166+ ret = check_func_arg_reg_off(env, reg, argno, ARG_DONTCARE);
93489167 if (ret < 0)
93499168 return ret;
9350− if (check_mem_reg(env, reg, regno, arg->mem_size))
9169+ if (check_mem_reg(env, reg, argno, arg->mem_size))
93519170 return -EINVAL;
93529171 if (!(arg->arg_type & PTR_MAYBE_NULL) && (reg->type & PTR_MAYBE_NULL)) {
9353− bpf_log(log, "arg#%d is expected to be non-NULL\n", i);
9172+ bpf_log(log, "%s is expected to be non-NULL\n",
9173+ reg_arg_name(env, argno));
93549174 return -EINVAL;
93559175 }
93569176 } else if (base_type(arg->arg_type) == ARG_PTR_TO_ARENA) {
@@ -9362,15 +9182,16 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 5 unmodified lines
93629182 * run-time debug nightmare.
93639183 */
93649184 if (reg->type != PTR_TO_ARENA && reg->type != SCALAR_VALUE) {
9365− bpf_log(log, "R%d is not a pointer to arena or scalar.\n", regno);
9185+ bpf_log(log, "%s is not a pointer to arena or scalar.\n",
9186+ reg_arg_name(env, argno));
93669187 return -EINVAL;
93679188 }
9368− } else if (arg->arg_type == (ARG_PTR_TO_DYNPTR | MEM_RDONLY)) {
9369− ret = check_func_arg_reg_off(env, reg, regno, ARG_PTR_TO_DYNPTR);
9189+ } else if (arg->arg_type == ARG_PTR_TO_DYNPTR) {
9190+ ret = check_func_arg_reg_off(env, reg, argno, ARG_PTR_TO_DYNPTR);
93709191 if (ret)
93719192 return ret;
93729193
9373− ret = process_dynptr_func(env, regno, -1, arg->arg_type, 0);
9194+ ret = process_dynptr_func(env, reg, argno, -1, arg->arg_type, &ref_obj, NULL);
93749195 if (ret)
93759196 return ret;
93769197 } else if (base_type(arg->arg_type) == ARG_PTR_TO_BTF_ID) {
@@ -9381,12 +9202,13 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 4 unmodified lines
93819202 continue;
93829203
93839204 memset(&meta, 0, sizeof(meta)); /* leave func_id as zero */
9384− err = check_reg_type(env, regno, arg->arg_type, &arg->btf_id, &meta);
9385− err = err ?: check_func_arg_reg_off(env, reg, regno, arg->arg_type);
9205+ err = check_reg_type(env, reg, argno, arg->arg_type, &arg->btf_id, &meta);
9206+ err = err ?: check_func_arg_reg_off(env, reg, argno, arg->arg_type);
93869207 if (err)
93879208 return err;
93889209 } else {
9389− verifier_bug(env, "unrecognized arg#%d type %d", i, arg->arg_type);
9210+ verifier_bug(env, "unrecognized %s type %d",
9211+ reg_arg_name(env, argno), arg->arg_type);
93909212 return -EFAULT;
93919213 }
93929214 }
@@ -9505,6 +9327,8 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · 112 unmodified lines
95059327 int *insn_idx)
95069328 {
95079329 struct bpf_verifier_state *state = env->cur_state;
9330+ struct bpf_subprog_info *caller_info;
9331+ u16 callee_incoming, stack_arg_cnt;
95089332 struct bpf_func_state *caller;
95099333 int err, subprog, target_insn;
95109334
@@ -9547,6 +9371,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · 36 unmodified lines
95479371 /* mark global subprog for verifying after main prog */
95489372 subprog_aux(env, subprog)->called = true;
95499373 clear_caller_saved_regs(env, caller->regs);
9374+ invalidate_outgoing_stack_args(env, cur_func(env));
95509375
95519376 /* All non-void global functions return a 64-bit SCALAR_VALUE. */
95529377 if (!subprog_returns_void(env, subprog)) {
@@ -9569,6 +9394,16 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · 16 unmodified lines
95699394 return 0;
95709395 }
95719396
9397+ /*
9398+ * Track caller's total stack arg count (incoming + max outgoing).
9399+ * This is needed so the JIT knows how much stack arg space to allocate.
9400+ */
9401+ caller_info = &env->subprog_info[caller->subprogno];
9402+ callee_incoming = bpf_in_stack_arg_cnt(&env->subprog_info[subprog]);
9403+ stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) + callee_incoming;
9404+ if (stack_arg_cnt > caller_info->stack_arg_cnt)
9405+ caller_info->stack_arg_cnt = stack_arg_cnt;
9406+
95729407 /* for regular function entry setup new frame and continue
95739408 * from that frame.
95749409 */
@@ -9852,9 +9687,9 @@ static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env) · 277 unmodified lines
98529687 static bool retval_range_within(struct bpf_retval_range range, const struct bpf_reg_state *reg)
98539688 {
98549689 if (range.return_32bit)
9855− return range.minval <= reg->s32_min_value && reg->s32_max_value <= range.maxval;
9690+ return range.minval <= reg_s32_min(reg) && reg_s32_max(reg) <= range.maxval;
98569691 else
9857− return range.minval <= reg->smin_value && reg->smax_value <= range.maxval;
9692+ return range.minval <= reg_smin(reg) && reg_smax(reg) <= range.maxval;
98589693 }
98599694
98609695 static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
@@ -9926,6 +9761,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · 65 unmodified lines
99269761 * bpf_throw, this will be done by copy_verifier_state for extra frames. */
99279762 free_func_state(callee);
99289763 state->frame[state->curframe--] = NULL;
9764+ invalidate_outgoing_stack_args(env, caller);
99299765
99309766 /* for callbacks widen imprecise scalars to make programs like below verify:
99319767 *
@@ -9952,7 +9788,9 @@ static int do_refine_retval_range(struct bpf_verifier_env *env, · 20 unmodified lines
99529788 int func_id,
99539789 struct bpf_call_arg_meta *meta)
99549790 {
9791+ struct bpf_retval_range range;
99559792 struct bpf_reg_state *ret_reg = &regs[BPF_REG_0];
9793+ enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
99569794
99579795 if (ret_type != RET_INTEGER)
99589796 return 0;
@@ -9963,21 +9801,36 @@ static int do_refine_retval_range(struct bpf_verifier_env *env, · 4 unmodified lines
99639801 case BPF_FUNC_probe_read_str:
99649802 case BPF_FUNC_probe_read_kernel_str:
99659803 case BPF_FUNC_probe_read_user_str:
9966− ret_reg->smax_value = meta->msize_max_value;
9967− ret_reg->s32_max_value = meta->msize_max_value;
9968− ret_reg->smin_value = -MAX_ERRNO;
9969− ret_reg->s32_min_value = -MAX_ERRNO;
9804+ reg_set_srange64(ret_reg, -MAX_ERRNO, meta->msize_max_value);
9805+ reg_set_srange32(ret_reg, -MAX_ERRNO, meta->msize_max_value);
99709806 reg_bounds_sync(ret_reg);
99719807 break;
99729808 case BPF_FUNC_get_smp_processor_id:
9973− ret_reg->umax_value = nr_cpu_ids - 1;
9974− ret_reg->u32_max_value = nr_cpu_ids - 1;
9975− ret_reg->smax_value = nr_cpu_ids - 1;
9976− ret_reg->s32_max_value = nr_cpu_ids - 1;
9977− ret_reg->umin_value = 0;
9978− ret_reg->u32_min_value = 0;
9979− ret_reg->smin_value = 0;
9980− ret_reg->s32_min_value = 0;
9809+ reg_set_urange64(ret_reg, 0, nr_cpu_ids - 1);
9810+ reg_set_urange32(ret_reg, 0, nr_cpu_ids - 1);
9811+ reg_bounds_sync(ret_reg);
9812+ break;
9813+ case BPF_FUNC_get_retval:
9814+ /*
9815+ * bpf_get_retval may see arbitrary value passed by bpf_prog_run_array_cg for
9816+ * CGROUP_GETSOCKOPT type.
9817+ */
9818+ if (prog_type == BPF_PROG_TYPE_CGROUP_SOCKOPT &&
9819+ env->prog->expected_attach_type == BPF_CGROUP_GETSOCKOPT)
9820+ break;
9821+
9822+ if (prog_type == BPF_PROG_TYPE_LSM &&
9823+ env->prog->expected_attach_type == BPF_LSM_CGROUP) {
9824+ if (!env->prog->aux->attach_func_proto->type)
9825+ break;
9826+ bpf_lsm_get_retval_range(env->prog, &range);
9827+ } else {
9828+ range.minval = -MAX_ERRNO;
9829+ range.maxval = 0;
9830+ }
9831+
9832+ reg_set_srange64(ret_reg, range.minval, range.maxval);
9833+ reg_set_srange32(ret_reg, range.minval, range.maxval);
99819834 reg_bounds_sync(ret_reg);
99829835 break;
99839836 }
@@ -10086,7 +9939,7 @@ static int check_reference_leak(struct bpf_verifier_env *env, bool exception_exi · 102 unmodified lines
100869939 * kernel. Type checks are performed later in check_return_code.
100879940 */
100889941 if (type == BPF_PROG_TYPE_STRUCT_OPS && !exception_exit &&
10089− reg->ref_obj_id == state->refs[i].id)
9942+ reg->id == state->refs[i].id)
100909943 continue;
100919944 verbose(env, "Unreleased reference id=%d alloc_insn=%d\n",
100929945 state->refs[i].id, state->refs[i].insn_idx);
@@ -10221,13 +10074,16 @@ static void update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno · 128 unmodified lines
1022110074 state->callback_subprogno == subprogno);
1022210075 }
1022310076
10224−/* Returns whether or not the given map type can potentially elide
10077+/* Returns whether or not the given map can potentially elide
1022510078 * lookup return value nullness check. This is possible if the key
1022610079 * is statically known.
1022710080 */
10228−static bool can_elide_value_nullness(enum bpf_map_type type)
10081+static bool can_elide_value_nullness(const struct bpf_map *map)
1022910082 {
10230− switch (type) {
10083+ if (map->map_flags & BPF_F_INNER_MAP)
10084+ return false;
10085+
10086+ switch (map->map_type) {
1023110087 case BPF_MAP_TYPE_ARRAY:
1023210088 case BPF_MAP_TYPE_PERCPU_ARRAY:
1023310089 return true;
@@ -10272,6 +10128,24 @@ static const char *non_sleepable_context_description(struct bpf_verifier_env *en · 38 unmodified lines
1027210128 return "non-sleepable prog";
1027310129 }
1027410130
10131+static int release_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
10132+ bool convert_rcu, bool release_dynptr)
10133+{
10134+ int err = -EINVAL;
10135+
10136+ if (bpf_register_is_null(reg))
10137+ return 0;
10138+
10139+ if (release_dynptr)
10140+ err = unmark_stack_slots_dynptr(env, reg);
10141+ else if (convert_rcu)
10142+ err = ref_convert_alloc_rcu_protected(env, reg->id);
10143+ else if (reg_is_referenced(env, reg))
10144+ err = release_reference(env, reg->id);
10145+
10146+ return err;
10147+}
10148+
1027510149 static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
1027610150 int *insn_idx_p)
1027710151 {
@@ -10321,7 +10195,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 43 unmodified lines
1032110195 memset(&meta, 0, sizeof(meta));
1032210196 meta.pkt_access = fn->pkt_access;
1032310197
10324− err = check_func_proto(fn);
10198+ err = check_func_proto(fn, &meta);
1032510199 if (err) {
1032610200 verifier_bug(env, "incorrect func proto %s#%d", func_id_name(func_id), func_id);
1032710201 return err;
@@ -10353,55 +10227,26 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 25 unmodified lines
1035310227 if (err)
1035410228 return err;
1035510229
10230+ regs = cur_regs(env);
10231+
1035610232 /* Mark slots with STACK_MISC in case of raw mode, stack offset
1035710233 * is inferred from register state.
1035810234 */
1035910235 for (i = 0; i < meta.access_size; i++) {
10360− err = check_mem_access(env, insn_idx, meta.regno, i, BPF_B,
10236+ err = check_mem_access(env, insn_idx, regs + meta.regno, argno_from_reg(meta.regno), i, BPF_B,
1036110237 BPF_WRITE, -1, false, false);
1036210238 if (err)
1036310239 return err;
1036410240 }
1036510241
10366− regs = cur_regs(env);
10367−
1036810242 if (meta.release_regno) {
10369− err = -EINVAL;
10370− if (arg_type_is_dynptr(fn->arg_type[meta.release_regno - BPF_REG_1])) {
10371− err = unmark_stack_slots_dynptr(env, &regs[meta.release_regno]);
10372− } else if (func_id == BPF_FUNC_kptr_xchg && meta.ref_obj_id) {
10373− u32 ref_obj_id = meta.ref_obj_id;
10374− bool in_rcu = in_rcu_cs(env);
10375− struct bpf_func_state *state;
10376− struct bpf_reg_state *reg;
10377−
10378− err = release_reference_nomark(env->cur_state, ref_obj_id);
10379− if (!err) {
10380− bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({
10381− if (reg->ref_obj_id == ref_obj_id) {
10382− if (in_rcu && (reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU)) {
10383− reg->ref_obj_id = 0;
10384− reg->type &= ~MEM_ALLOC;
10385− reg->type |= MEM_RCU;
10386− } else {
10387− mark_reg_invalid(env, reg);
10388− }
10389− }
10390− }));
10391− }
10392− } else if (meta.ref_obj_id) {
10393− err = release_reference(env, meta.ref_obj_id);
10394− } else if (bpf_register_is_null(&regs[meta.release_regno])) {
10395− /* meta.ref_obj_id can only be 0 if register that is meant to be
10396− * released is NULL, which must be > R0.
10397− */
10398− err = 0;
10399− }
10400− if (err) {
10401− verbose(env, "func %s#%d reference has not been acquired before\n",
10402− func_id_name(func_id), func_id);
10243+ struct bpf_reg_state *reg = &regs[meta.release_regno];
10244+ bool convert_rcu = (func_id == BPF_FUNC_kptr_xchg) && in_rcu_cs(env) &&
10245+ (reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU);
10246+
10247+ err = release_reg(env, reg, convert_rcu, !!meta.dynptr.id);
10248+ if (err)
1040310249 return err;
10404− }
1040510250 }
1040610251
1040710252 switch (func_id) {
@@ -10442,7 +10287,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 34 unmodified lines
1044210287 err = mark_chain_precision(env, BPF_REG_1);
1044310288 if (err)
1044410289 return err;
10445− if (cur_func(env)->callback_depth < regs[BPF_REG_1].umax_value) {
10290+ if (cur_func(env)->callback_depth < reg_umax(&regs[BPF_REG_1])) {
1044610291 err = push_callback_call(env, insn, insn_idx, meta.subprogno,
1044710292 set_loop_callback_state);
1044810293 } else {
@@ -10460,6 +10305,24 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 11 unmodified lines
1046010305 }
1046110306 break;
1046210307 case BPF_FUNC_set_retval:
10308+ {
10309+ struct bpf_retval_range range = {
10310+ .minval = -MAX_ERRNO,
10311+ .maxval = 0,
10312+ .return_32bit = true
10313+ };
10314+ struct bpf_reg_state *r1 = &regs[BPF_REG_1];
10315+
10316+ if (r1->type != SCALAR_VALUE) {
10317+ verbose(env, "R1 is not a scalar\n");
10318+ return -EINVAL;
10319+ }
10320+
10321+ /* CGROUP_GETSOCKOPT is allowed to return arbitrary value */
10322+ if (prog_type == BPF_PROG_TYPE_CGROUP_SOCKOPT &&
10323+ env->prog->expected_attach_type == BPF_CGROUP_GETSOCKOPT)
10324+ break;
10325+
1046310326 if (prog_type == BPF_PROG_TYPE_LSM &&
1046410327 env->prog->expected_attach_type == BPF_LSM_CGROUP) {
1046510328 if (!env->prog->aux->attach_func_proto->type) {
@@ -10469,54 +10332,24 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 3 unmodified lines
1046910332 verbose(env, "BPF_LSM_CGROUP that attach to void LSM hooks can't modify return value!\n");
1047010333 return -EINVAL;
1047110334 }
10472− }
10473− break;
10474− case BPF_FUNC_dynptr_data:
10475− {
10476− struct bpf_reg_state *reg;
10477− int id, ref_obj_id;
10478−
10479− reg = get_dynptr_arg_reg(env, fn, regs);
10480− if (!reg)
10481− return -EFAULT;
10482−
10483−
10484− if (meta.dynptr_id) {
10485− verifier_bug(env, "meta.dynptr_id already set");
10486− return -EFAULT;
10487− }
10488− if (meta.ref_obj_id) {
10489− verifier_bug(env, "meta.ref_obj_id already set");
10490− return -EFAULT;
10335+ bpf_lsm_get_retval_range(env->prog, &range);
1049110336 }
1049210337
10493− id = dynptr_id(env, reg);
10494− if (id < 0) {
10495− verifier_bug(env, "failed to obtain dynptr id");
10496− return id;
10497− }
10338+ err = mark_chain_precision(env, BPF_REG_1);
10339+ if (err)
10340+ return err;
1049810341
10499− ref_obj_id = dynptr_ref_obj_id(env, reg);
10500− if (ref_obj_id < 0) {
10501− verifier_bug(env, "failed to obtain dynptr ref_obj_id");
10502− return ref_obj_id;
10342+ if (!retval_range_within(range, r1)) {
10343+ verbose_invalid_scalar(env, r1, range, "At bpf_set_retval", "R1");
10344+ return -EINVAL;
1050310345 }
1050410346
10505− meta.dynptr_id = id;
10506− meta.ref_obj_id = ref_obj_id;
10507−
1050810347 break;
1050910348 }
1051010349 case BPF_FUNC_dynptr_write:
1051110350 {
10512− enum bpf_dynptr_type dynptr_type;
10513− struct bpf_reg_state *reg;
10514−
10515− reg = get_dynptr_arg_reg(env, fn, regs);
10516− if (!reg)
10517− return -EFAULT;
10351+ enum bpf_dynptr_type dynptr_type = meta.dynptr.type;
1051810352
10519− dynptr_type = dynptr_get_type(env, reg);
1052010353 if (dynptr_type == BPF_DYNPTR_TYPE_INVALID)
1052110354 return -EFAULT;
1052210355
@@ -10560,6 +10393,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 37 unmodified lines
1056010393 bpf_mark_reg_not_init(env, &regs[caller_saved[i]]);
1056110394 check_reg_arg(env, caller_saved[i], DST_OP_NO_MARK);
1056210395 }
10396+ invalidate_outgoing_stack_args(env, cur_func(env));
1056310397
1056410398 /* helper call returns 64-bit value. */
1056510399 regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG;
@@ -10589,7 +10423,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 23 unmodified lines
1058910423 }
1059010424
1059110425 if (func_id == BPF_FUNC_map_lookup_elem &&
10592− can_elide_value_nullness(meta.map.ptr->map_type) &&
10426+ can_elide_value_nullness(meta.map.ptr) &&
1059310427 meta.const_map_key >= 0 &&
1059410428 meta.const_map_key < meta.map.ptr->max_entries)
1059510429 ret_flag &= ~PTR_MAYBE_NULL;
@@ -10701,29 +10535,45 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 105 unmodified lines
1070110535 if (type_may_be_null(regs[BPF_REG_0].type))
1070210536 regs[BPF_REG_0].id = ++env->id_gen;
1070310537
10704− if (helper_multiple_ref_obj_use(func_id, meta.map.ptr)) {
10705− verifier_bug(env, "func %s#%d sets ref_obj_id more than once",
10706− func_id_name(func_id), func_id);
10707− return -EFAULT;
10708− }
10538+ if (is_ptr_cast_function(func_id) &&
10539+ find_reference_state(env->cur_state, meta.ref_obj.id)) {
10540+ struct bpf_verifier_state *branch;
10541+ struct bpf_reg_state *r0;
10542+
10543+ err = validate_ref_obj(env, &meta.ref_obj);
10544+ if (err)
10545+ return err;
10546+
10547+ /*
10548+ * In order for a release of any of the original or cast pointers
10549+ * to invalidate all other pointers, reuse the same reference id for
10550+ * the cast result.
10551+ * This reference id can't be used for nullness propagation,
10552+ * as cast might return NULL for a non-NULL input.
10553+ * Hence, explore the NULL case as a separate branch.
10554+ */
10555+ branch = push_stack(env, env->insn_idx + 1, env->insn_idx, false);
10556+ if (IS_ERR(branch))
10557+ return PTR_ERR(branch);
1070910558
10710− if (is_dynptr_ref_function(func_id))
10711− regs[BPF_REG_0].dynptr_id = meta.dynptr_id;
10559+ r0 = &branch->frame[branch->curframe]->regs[BPF_REG_0];
10560+ __mark_reg_known_zero(r0);
10561+ r0->type = SCALAR_VALUE;
1071210562
10713− if (is_ptr_cast_function(func_id) || is_dynptr_ref_function(func_id)) {
10714− /* For release_reference() */
10715− regs[BPF_REG_0].ref_obj_id = meta.ref_obj_id;
10563+ regs[BPF_REG_0].type &= ~PTR_MAYBE_NULL;
10564+ regs[BPF_REG_0].id = meta.ref_obj.id;
1071610565 } else if (is_acquire_function(func_id, meta.map.ptr)) {
10717− int id = acquire_reference(env, insn_idx);
10566+ int id = acquire_reference(env, insn_idx, 0);
1071810567
1071910568 if (id < 0)
1072010569 return id;
10721− /* For mark_ptr_or_null_reg() */
10570+
1072210571 regs[BPF_REG_0].id = id;
10723− /* For release_reference() */
10724− regs[BPF_REG_0].ref_obj_id = id;
1072510572 }
1072610573
10574+ if (func_id == BPF_FUNC_dynptr_data)
10575+ regs[BPF_REG_0].parent_id = meta.dynptr.id;
10576+
1072710577 err = do_refine_retval_range(env, regs, fn->ret_type, func_id, &meta);
1072810578 if (err)
1072910579 return err;
@@ -10819,7 +10669,6 @@ static bool is_kfunc_release(struct bpf_kfunc_call_arg_meta *meta) · 89 unmodified lines
1081910669 return meta->kfunc_flags & KF_RELEASE;
1082010670 }
1082110671
10822−
1082310672 static bool is_kfunc_destructive(struct bpf_kfunc_call_arg_meta *meta)
1082410673 {
1082510674 return meta->kfunc_flags & KF_DESTRUCTIVE;
@@ -10896,6 +10745,11 @@ static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param · 70 unmodified lines
1089610745 return btf_param_match_suffix(btf, arg, "__nullable");
1089710746 }
1089810747
10748+static bool is_kfunc_arg_nonown_allowed(const struct btf *btf, const struct btf_param *arg)
10749+{
10750+ return btf_param_match_suffix(btf, arg, "__nonown_allowed");
10751+}
10752+
1089910753 static bool is_kfunc_arg_const_str(const struct btf *btf, const struct btf_param *arg)
1090010754 {
1090110755 return btf_param_match_suffix(btf, arg, "__str");
@@ -11136,10 +10990,15 @@ enum special_kfunc_type { · 234 unmodified lines
1113610990 KF_bpf_list_push_front,
1113710991 KF_bpf_list_push_back_impl,
1113810992 KF_bpf_list_push_back,
10993+ KF_bpf_list_add,
1113910994 KF_bpf_list_pop_front,
1114010995 KF_bpf_list_pop_back,
10996+ KF_bpf_list_del,
1114110997 KF_bpf_list_front,
1114210998 KF_bpf_list_back,
10999+ KF_bpf_list_is_first,
11000+ KF_bpf_list_is_last,
11001+ KF_bpf_list_empty,
1114311002 KF_bpf_cast_to_kern_ctx,
1114411003 KF_bpf_rdonly_cast,
1114511004 KF_bpf_rcu_read_lock,
@@ -11204,10 +11063,15 @@ BTF_ID(func, bpf_list_push_front_impl) · 58 unmodified lines
1120411063 BTF_ID(func, bpf_list_push_front)
1120511064 BTF_ID(func, bpf_list_push_back_impl)
1120611065 BTF_ID(func, bpf_list_push_back)
11066+BTF_ID(func, bpf_list_add)
1120711067 BTF_ID(func, bpf_list_pop_front)
1120811068 BTF_ID(func, bpf_list_pop_back)
11069+BTF_ID(func, bpf_list_del)
1120911070 BTF_ID(func, bpf_list_front)
1121011071 BTF_ID(func, bpf_list_back)
11072+BTF_ID(func, bpf_list_is_first)
11073+BTF_ID(func, bpf_list_is_last)
11074+BTF_ID(func, bpf_list_empty)
1121111075 BTF_ID(func, bpf_cast_to_kern_ctx)
1121211076 BTF_ID(func, bpf_rdonly_cast)
1121311077 BTF_ID(func, bpf_rcu_read_lock)
@@ -11319,7 +11183,8 @@ static bool is_bpf_list_push_kfunc(u32 func_id) · 105 unmodified lines
1131911183 return func_id == special_kfunc_list[KF_bpf_list_push_front] ||
1132011184 func_id == special_kfunc_list[KF_bpf_list_push_front_impl] ||
1132111185 func_id == special_kfunc_list[KF_bpf_list_push_back] ||
11322− func_id == special_kfunc_list[KF_bpf_list_push_back_impl];
11186+ func_id == special_kfunc_list[KF_bpf_list_push_back_impl] ||
11187+ func_id == special_kfunc_list[KF_bpf_list_add];
1132311188 }
1132411189
1132511190 static bool is_bpf_rbtree_add_kfunc(u32 func_id)
@@ -11368,15 +11233,12 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta) · 42 unmodified lines
1136811233 }
1136911234
1137011235 static enum kfunc_ptr_arg_type
11371−get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
11372− struct bpf_kfunc_call_arg_meta *meta,
11236+get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_func_state *caller,
11237+ struct bpf_reg_state *regs, struct bpf_kfunc_call_arg_meta *meta,
1137311238 const struct btf_type *t, const struct btf_type *ref_t,
1137411239 const char *ref_tname, const struct btf_param *args,
11375− int argno, int nargs)
11240+ int arg, int nargs, argno_t argno, struct bpf_reg_state *reg)
1137611241 {
11377− u32 regno = argno + 1;
11378− struct bpf_reg_state *regs = cur_regs(env);
11379− struct bpf_reg_state *reg = &regs[regno];
1138011242 bool arg_mem_size = false;
1138111243
1138211244 if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] ||
@@ -11384,9 +11246,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, · 1 unmodified line
1138411246 meta->func_id == special_kfunc_list[KF_bpf_session_cookie])
1138511247 return KF_ARG_PTR_TO_CTX;
1138611248
11387− if (argno + 1 < nargs &&
11388− (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1]) ||
11389− is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1])))
11249+ if (arg + 1 < nargs &&
11250+ (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1)) ||
11251+ is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1))))
1139011252 arg_mem_size = true;
1139111253
1139211254 /* In this function, we verify the kfunc's BTF as per the argument type,
@@ -11394,68 +11256,69 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, · 1 unmodified line
1139411256 * type to our caller. When a set of conditions hold in the BTF type of
1139511257 * arguments, we resolve it to a known kfunc_ptr_arg_type.
1139611258 */
11397− if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), argno))
11259+ if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), arg))
1139811260 return KF_ARG_PTR_TO_CTX;
1139911261
11400− if (is_kfunc_arg_nullable(meta->btf, &args[argno]) && bpf_register_is_null(reg) &&
11262+ if (is_kfunc_arg_nullable(meta->btf, &args[arg]) && bpf_register_is_null(reg) &&
1140111263 !arg_mem_size)
1140211264 return KF_ARG_PTR_TO_NULL;
1140311265
11404− if (is_kfunc_arg_alloc_obj(meta->btf, &args[argno]))
11266+ if (is_kfunc_arg_alloc_obj(meta->btf, &args[arg]))
1140511267 return KF_ARG_PTR_TO_ALLOC_BTF_ID;
1140611268
11407− if (is_kfunc_arg_refcounted_kptr(meta->btf, &args[argno]))
11269+ if (is_kfunc_arg_refcounted_kptr(meta->btf, &args[arg]))
1140811270 return KF_ARG_PTR_TO_REFCOUNTED_KPTR;
1140911271
11410− if (is_kfunc_arg_dynptr(meta->btf, &args[argno]))
11272+ if (is_kfunc_arg_dynptr(meta->btf, &args[arg]))
1141111273 return KF_ARG_PTR_TO_DYNPTR;
1141211274
11413− if (is_kfunc_arg_iter(meta, argno, &args[argno]))
11275+ if (is_kfunc_arg_iter(meta, arg, &args[arg]))
1141411276 return KF_ARG_PTR_TO_ITER;
1141511277
11416− if (is_kfunc_arg_list_head(meta->btf, &args[argno]))
11278+ if (is_kfunc_arg_list_head(meta->btf, &args[arg]))
1141711279 return KF_ARG_PTR_TO_LIST_HEAD;
1141811280
11419− if (is_kfunc_arg_list_node(meta->btf, &args[argno]))
11281+ if (is_kfunc_arg_list_node(meta->btf, &args[arg]))
1142011282 return KF_ARG_PTR_TO_LIST_NODE;
1142111283
11422− if (is_kfunc_arg_rbtree_root(meta->btf, &args[argno]))
11284+ if (is_kfunc_arg_rbtree_root(meta->btf, &args[arg]))
1142311285 return KF_ARG_PTR_TO_RB_ROOT;
1142411286
11425− if (is_kfunc_arg_rbtree_node(meta->btf, &args[argno]))
11287+ if (is_kfunc_arg_rbtree_node(meta->btf, &args[arg]))
1142611288 return KF_ARG_PTR_TO_RB_NODE;
1142711289
11428− if (is_kfunc_arg_const_str(meta->btf, &args[argno]))
11290+ if (is_kfunc_arg_const_str(meta->btf, &args[arg]))
1142911291 return KF_ARG_PTR_TO_CONST_STR;
1143011292
11431− if (is_kfunc_arg_map(meta->btf, &args[argno]))
11293+ if (is_kfunc_arg_map(meta->btf, &args[arg]))
1143211294 return KF_ARG_PTR_TO_MAP;
1143311295
11434− if (is_kfunc_arg_wq(meta->btf, &args[argno]))
11296+ if (is_kfunc_arg_wq(meta->btf, &args[arg]))
1143511297 return KF_ARG_PTR_TO_WORKQUEUE;
1143611298
11437− if (is_kfunc_arg_timer(meta->btf, &args[argno]))
11299+ if (is_kfunc_arg_timer(meta->btf, &args[arg]))
1143811300 return KF_ARG_PTR_TO_TIMER;
1143911301
11440− if (is_kfunc_arg_task_work(meta->btf, &args[argno]))
11302+ if (is_kfunc_arg_task_work(meta->btf, &args[arg]))
1144111303 return KF_ARG_PTR_TO_TASK_WORK;
1144211304
11443− if (is_kfunc_arg_irq_flag(meta->btf, &args[argno]))
11305+ if (is_kfunc_arg_irq_flag(meta->btf, &args[arg]))
1144411306 return KF_ARG_PTR_TO_IRQ_FLAG;
1144511307
11446− if (is_kfunc_arg_res_spin_lock(meta->btf, &args[argno]))
11308+ if (is_kfunc_arg_res_spin_lock(meta->btf, &args[arg]))
1144711309 return KF_ARG_PTR_TO_RES_SPIN_LOCK;
1144811310
1144911311 if ((base_type(reg->type) == PTR_TO_BTF_ID || reg2btf_ids[base_type(reg->type)])) {
1145011312 if (!btf_type_is_struct(ref_t)) {
11451− verbose(env, "kernel function %s args#%d pointer type %s %s is not supported\n",
11452− meta->func_name, argno, btf_type_str(ref_t), ref_tname);
11313+ verbose(env, "kernel function %s %s pointer type %s %s is not supported\n",
11314+ meta->func_name, reg_arg_name(env, argno),
11315+ btf_type_str(ref_t), ref_tname);
1145311316 return -EINVAL;
1145411317 }
1145511318 return KF_ARG_PTR_TO_BTF_ID;
1145611319 }
1145711320
11458− if (is_kfunc_arg_callback(env, meta->btf, &args[argno]))
11321+ if (is_kfunc_arg_callback(env, meta->btf, &args[arg]))
1145911322 return KF_ARG_PTR_TO_CALLBACK;
1146011323
1146111324 /* This is the catch all argument type of register types supported by
@@ -11465,8 +11328,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, · 3 unmodified lines
1146511328 */
1146611329 if (!btf_type_is_scalar(ref_t) && !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0) &&
1146711330 (arg_mem_size ? !btf_type_is_void(ref_t) : 1)) {
11468− verbose(env, "arg#%d pointer type %s %s must point to %sscalar, or struct with scalar\n",
11469− argno, btf_type_str(ref_t), ref_tname, arg_mem_size ? "void, " : "");
11331+ verbose(env, "%s pointer type %s %s must point to %sscalar, or struct with scalar\n",
11332+ reg_arg_name(env, argno),
11333+ btf_type_str(ref_t), ref_tname, arg_mem_size ? "void, " : "");
1147011334 return -EINVAL;
1147111335 }
1147211336 return arg_mem_size ? KF_ARG_PTR_TO_MEM_SIZE : KF_ARG_PTR_TO_MEM;
@@ -11477,7 +11341,7 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · 4 unmodified lines
1147711341 const struct btf_type *ref_t,
1147811342 const char *ref_tname, u32 ref_id,
1147911343 struct bpf_kfunc_call_arg_meta *meta,
11480− int argno)
11344+ int arg, argno_t argno)
1148111345 {
1148211346 const struct btf_type *reg_ref_t;
1148311347 bool strict_type_match = false;
@@ -11519,7 +11383,7 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · 35 unmodified lines
1151911383 * btf_struct_ids_match() to walk the struct at the 0th offset, and
1152011384 * resolve types.
1152111385 */
11522− if ((is_kfunc_release(meta) && reg->ref_obj_id) ||
11386+ if ((is_kfunc_release(meta) && reg_is_referenced(env, reg)) ||
1152311387 btf_type_ids_nocast_alias(&env->log, reg_btf, reg_ref_id, meta->btf, ref_id))
1152411388 strict_type_match = true;
1152511389
@@ -11535,19 +11399,19 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · 9 unmodified lines
1153511399 */
1153611400 taking_projection = btf_is_projection_of(ref_tname, reg_ref_tname);
1153711401 if (!taking_projection && !struct_same) {
11538− verbose(env, "kernel function %s args#%d expected pointer to %s %s but R%d has a pointer to %s %s\n",
11539− meta->func_name, argno, btf_type_str(ref_t), ref_tname, argno + 1,
11402+ verbose(env, "kernel function %s %s expected pointer to %s %s but %s has a pointer to %s %s\n",
11403+ meta->func_name, reg_arg_name(env, argno),
11404+ btf_type_str(ref_t), ref_tname, reg_arg_name(env, argno),
1154011405 btf_type_str(reg_ref_t), reg_ref_tname);
1154111406 return -EINVAL;
1154211407 }
1154311408 return 0;
1154411409 }
1154511410
11546−static int process_irq_flag(struct bpf_verifier_env *env, int regno,
11411+static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
1154711412 struct bpf_kfunc_call_arg_meta *meta)
1154811413 {
11549− struct bpf_reg_state *reg = reg_state(env, regno);
11550− int err, kfunc_class = IRQ_NATIVE_KFUNC;
11414+ int err, spi, kfunc_class = IRQ_NATIVE_KFUNC;
1155111415 bool irq_save;
1155211416
1155311417 if (meta->func_id == special_kfunc_list[KF_bpf_local_irq_save] ||
@@ -11567,11 +11431,13 @@ static int process_irq_flag(struct bpf_verifier_env *env, int regno, · 13 unmodified lines
1156711431
1156811432 if (irq_save) {
1156911433 if (!is_irq_flag_reg_valid_uninit(env, reg)) {
11570− verbose(env, "expected uninitialized irq flag as arg#%d\n", regno - 1);
11434+ verbose(env, "expected uninitialized irq flag as %s\n",
11435+ reg_arg_name(env, argno));
1157111436 return -EINVAL;
1157211437 }
1157311438
11574− err = check_mem_access(env, env->insn_idx, regno, 0, BPF_DW, BPF_WRITE, -1, false, false);
11439+ err = check_mem_access(env, env->insn_idx, reg, argno, 0, BPF_DW,
11440+ BPF_WRITE, -1, false, false);
1157511441 if (err)
1157611442 return err;
1157711443
@@ -11581,13 +11447,16 @@ static int process_irq_flag(struct bpf_verifier_env *env, int regno, · 3 unmodified lines
1158111447 } else {
1158211448 err = is_irq_flag_reg_valid_init(env, reg);
1158311449 if (err) {
11584− verbose(env, "expected an initialized irq flag as arg#%d\n", regno - 1);
11450+ verbose(env, "expected an initialized irq flag as %s\n",
11451+ reg_arg_name(env, argno));
1158511452 return err;
1158611453 }
1158711454
11588− err = mark_irq_flag_read(env, reg);
11589− if (err)
11590− return err;
11455+ spi = irq_flag_get_spi(env, reg);
11456+ if (spi < 0)
11457+ return spi;
11458+
11459+ mark_stack_slots_scratched(env, spi, 1);
1159111460
1159211461 err = unmark_stack_slot_irq_flag(env, reg, kfunc_class);
1159311462 if (err)
@@ -11618,36 +11487,21 @@ static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state · 24 unmodified lines
1161811487 return 0;
1161911488 }
1162011489
11621−static int ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 ref_obj_id)
11490+static void ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 id)
1162211491 {
11623− struct bpf_verifier_state *state = env->cur_state;
1162411492 struct bpf_func_state *unused;
1162511493 struct bpf_reg_state *reg;
11626− int i;
11627−
11628− if (!ref_obj_id) {
11629− verifier_bug(env, "ref_obj_id is zero for owning -> non-owning conversion");
11630− return -EFAULT;
11631− }
1163211494
11633− for (i = 0; i < state->acquired_refs; i++) {

2971 more lines aren’t shown here.

kernel/events/core.c+9−0
@@ -11712,6 +11712,15 @@ static int __perf_event_set_bpf_prog(struct perf_event *event, · 11711 unmodified lines
1171211712 /* only uprobe programs are allowed to be sleepable */
1171311713 return -EINVAL;
1171411714
11715+ if (prog->type == BPF_PROG_TYPE_TRACEPOINT && prog->sleepable) {
11716+ /*
11717+ * Sleepable tracepoint programs can only attach to faultable
11718+ * tracepoints. Currently only syscall tracepoints are faultable.
11719+ */
11720+ if (!is_syscall_tp)
11721+ return -EINVAL;
11722+ }
11723+
1171511724 /* Kprobe override only works for kprobes, not uprobes. */
1171611725 if (prog->kprobe_override && !is_kprobe)
1171711726 return -EINVAL;
kernel/trace/bpf_trace.c+306−28
@@ -23,6 +23,7 @@ · 22 unmodified lines
2323 #include <linux/sort.h>
2424 #include <linux/key.h>
2525 #include <linux/namei.h>
26+#include <linux/file.h>
2627
2728 #include <net/bpf_sk_storage.h>
2829
@@ -42,6 +43,7 @@ · 13 unmodified lines
4243
4344 #define MAX_UPROBE_MULTI_CNT (1U << 20)
4445 #define MAX_KPROBE_MULTI_CNT (1U << 20)
46+#define MAX_TRACING_MULTI_CNT (1U << 20)
4547
4648 #ifdef CONFIG_MODULES
4749 struct bpf_trace_module {
@@ -152,6 +154,34 @@ unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx) · 104 unmodified lines
152154 return ret;
153155 }
154156
157+/**
158+ * trace_call_bpf_faultable - invoke BPF program in faultable context
159+ * @call: tracepoint event
160+ * @ctx: opaque context pointer
161+ *
162+ * Variant of trace_call_bpf() for faultable tracepoints (syscall
163+ * tracepoints). Supports sleepable BPF programs by using rcu_tasks_trace
164+ * for lifetime protection and bpf_prog_run_array_sleepable() for per-program
165+ * RCU flavor selection, following the uprobe pattern.
166+ *
167+ * Per-program recursion protection is provided by
168+ * bpf_prog_run_array_sleepable(). Global bpf_prog_active is not
169+ * needed because syscall tracepoints cannot self-recurse.
170+ *
171+ * Must be called from a faultable/preemptible context.
172+ */
173+unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx)
174+{
175+ struct bpf_prog_array *prog_array;
176+
177+ might_fault();
178+ guard(rcu_tasks_trace)();
179+
180+ prog_array = rcu_dereference_check(call->prog_array,
181+ rcu_read_lock_trace_held());
182+ return bpf_prog_run_array_sleepable(prog_array, ctx, bpf_prog_run);
183+}
184+
155185 #ifdef CONFIG_BPF_KPROBE_OVERRIDE
156186 BPF_CALL_2(bpf_override_return, struct pt_regs *, regs, unsigned long, rc)
157187 {
@@ -1305,7 +1335,8 @@ static inline bool is_uprobe_session(const struct bpf_prog *prog) · 1147 unmodified lines
13051335 static inline bool is_trace_fsession(const struct bpf_prog *prog)
13061336 {
13071337 return prog->type == BPF_PROG_TYPE_TRACING &&
1308− prog->expected_attach_type == BPF_TRACE_FSESSION;
1338+ (prog->expected_attach_type == BPF_TRACE_FSESSION ||
1339+ prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI);
13091340 }
13101341
13111342 static const struct bpf_func_proto *
@@ -2072,11 +2103,19 @@ void bpf_put_raw_tracepoint(struct bpf_raw_event_map *btp) · 760 unmodified lines
20722103 static __always_inline
20732104 void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args)
20742105 {
2106+ struct srcu_ctr __percpu *scp = NULL;
20752107 struct bpf_prog *prog = link->link.prog;
2108+ bool sleepable = prog->sleepable;
20762109 struct bpf_run_ctx *old_run_ctx;
20772110 struct bpf_trace_run_ctx run_ctx;
20782111
2079− rcu_read_lock_dont_migrate();
2112+ if (sleepable) {
2113+ scp = rcu_read_lock_tasks_trace();
2114+ migrate_disable();
2115+ } else {
2116+ rcu_read_lock_dont_migrate();
2117+ }
2118+
20802119 if (unlikely(!bpf_prog_get_recursion_context(prog))) {
20812120 bpf_prog_inc_misses_counter(prog);
20822121 goto out;
@@ -2085,12 +2124,18 @@ void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args) · 2 unmodified lines
20852124 run_ctx.bpf_cookie = link->cookie;
20862125 old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
20872126
2088− (void) bpf_prog_run(prog, args);
2127+ (void)bpf_prog_run(prog, args);
20892128
20902129 bpf_reset_run_ctx(old_run_ctx);
20912130 out:
20922131 bpf_prog_put_recursion_context(prog);
2093− rcu_read_unlock_migrate();
2132+
2133+ if (sleepable) {
2134+ migrate_enable();
2135+ rcu_read_unlock_tasks_trace(scp);
2136+ } else {
2137+ rcu_read_unlock_migrate();
2138+ }
20942139 }
20952140
20962141 #define UNPACK(...) __VA_ARGS__
@@ -3170,6 +3215,38 @@ static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx) · 1073 unmodified lines
31703215 return run_ctx->uprobe->cookie;
31713216 }
31723217
3218+static int bpf_uprobe_multi_get_path(const union bpf_attr *attr, struct path *path)
3219+{
3220+ void __user *upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path);
3221+ u32 path_fd = attr->link_create.uprobe_multi.path_fd;
3222+ u32 flags = attr->link_create.uprobe_multi.flags;
3223+
3224+ if (flags & BPF_F_UPROBE_MULTI_PATH_FD) {
3225+ /*
3226+ * When BPF_F_UPROBE_MULTI_PATH_FD is set, the executable is
3227+ * identified by path_fd, upath must be NULL.
3228+ */
3229+ if (upath)
3230+ return -EINVAL;
3231+
3232+ CLASS(fd, f)(path_fd);
3233+ if (fd_empty(f))
3234+ return -EBADF;
3235+ *path = fd_file(f)->f_path;
3236+ path_get(path);
3237+ return 0;
3238+ }
3239+
3240+ /*
3241+ * When BPF_F_UPROBE_MULTI_PATH_FD is not set, the path is resolved
3242+ * relative to the cwd (AT_FDCWD) or absolute using the upath string.
3243+ */
3244+ if (!upath || path_fd)
3245+ return -EINVAL;
3246+
3247+ return user_path_at(AT_FDCWD, upath, LOOKUP_FOLLOW, path);
3248+}
3249+
31733250 int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
31743251 {
31753252 struct bpf_uprobe_multi_link *link = NULL;
@@ -3179,10 +3256,9 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr · 3 unmodified lines
31793256 struct task_struct *task = NULL;
31803257 unsigned long __user *uoffsets;
31813258 u64 __user *ucookies;
3182− void __user *upath;
3259+ unsigned long size;
31833260 u32 flags, cnt, i;
31843261 struct path path;
3185− char *name;
31863262 pid_t pid;
31873263 int err;
31883264
@@ -3197,19 +3273,18 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr · 8 unmodified lines
31973273 return -EINVAL;
31983274
31993275 flags = attr->link_create.uprobe_multi.flags;
3200− if (flags & ~BPF_F_UPROBE_MULTI_RETURN)
3276+ if (flags & ~(BPF_F_UPROBE_MULTI_RETURN | BPF_F_UPROBE_MULTI_PATH_FD))
32013277 return -EINVAL;
32023278
32033279 /*
3204− * path, offsets and cnt are mandatory,
3280+ * offsets and cnt are mandatory,
32053281 * ref_ctr_offsets and cookies are optional
32063282 */
3207− upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path);
32083283 uoffsets = u64_to_user_ptr(attr->link_create.uprobe_multi.offsets);
32093284 cnt = attr->link_create.uprobe_multi.cnt;
32103285 pid = attr->link_create.uprobe_multi.pid;
32113286
3212− if (!upath || !uoffsets || !cnt || pid < 0)
3287+ if (!uoffsets || !cnt || pid < 0)
32133288 return -EINVAL;
32143289 if (cnt > MAX_UPROBE_MULTI_CNT)
32153290 return -E2BIG;
@@ -3217,14 +3292,17 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr · 1 unmodified line
32173292 uref_ctr_offsets = u64_to_user_ptr(attr->link_create.uprobe_multi.ref_ctr_offsets);
32183293 ucookies = u64_to_user_ptr(attr->link_create.uprobe_multi.cookies);
32193294
3220− name = strndup_user(upath, PATH_MAX);
3221− if (IS_ERR(name)) {
3222− err = PTR_ERR(name);
3223− return err;
3224− }
3295+ /*
3296+ * All uoffsets/uref_ctr_offsets/ucookies arrays have the same value
3297+ * size, we need to check their address range is safe for __get_user
3298+ * calls.
3299+ */
3300+ size = sizeof(*uoffsets) * cnt;
3301+ if (!access_ok(uoffsets, size) || !access_ok(uref_ctr_offsets, size) ||
3302+ !access_ok(ucookies, size))
3303+ return -EFAULT;
32253304
3226− err = kern_path(name, LOOKUP_FOLLOW, &path);
3227− kfree(name);
3305+ err = bpf_uprobe_multi_get_path(attr, &path);
32283306 if (err)
32293307 return err;
32303308
@@ -3398,12 +3476,12 @@ typedef int (*copy_fn_t)(void *dst, const void *src, u32 size, struct task_struc · 167 unmodified lines
33983476 * direct calls into all the specific callback implementations
33993477 * (copy_user_data_sleepable, copy_user_data_nofault, and so on)
34003478 */
3401−static __always_inline int __bpf_dynptr_copy_str(struct bpf_dynptr *dptr, u64 doff, u64 size,
3479+static __always_inline int __bpf_dynptr_copy_str(const struct bpf_dynptr *dptr, u64 doff, u64 size,
34023480 const void *unsafe_src,
34033481 copy_fn_t str_copy_fn,
34043482 struct task_struct *tsk)
34053483 {
3406− struct bpf_dynptr_kern *dst;
3484+ const struct bpf_dynptr_kern *dst;
34073485 u64 chunk_sz, off;
34083486 void *dst_slice;
34093487 int cnt, err;
@@ -3439,7 +3517,7 @@ static __always_inline int __bpf_dynptr_copy(const struct bpf_dynptr *dptr, u64 · 29 unmodified lines
34393517 u64 size, const void *unsafe_src,
34403518 copy_fn_t copy_fn, struct task_struct *tsk)
34413519 {
3442− struct bpf_dynptr_kern *dst;
3520+ const struct bpf_dynptr_kern *dst;
34433521 void *dst_slice;
34443522 char buf[256];
34453523 u64 off, chunk_sz;
@@ -3540,49 +3618,49 @@ __bpf_kfunc int bpf_send_signal_task(struct task_struct *task, int sig, enum pid · 94 unmodified lines
35403618 return bpf_send_signal_common(sig, type, task, value);
35413619 }
35423620
3543−__bpf_kfunc int bpf_probe_read_user_dynptr(struct bpf_dynptr *dptr, u64 off,
3621+__bpf_kfunc int bpf_probe_read_user_dynptr(const struct bpf_dynptr *dptr, u64 off,
35443622 u64 size, const void __user *unsafe_ptr__ign)
35453623 {
35463624 return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,
35473625 copy_user_data_nofault, NULL);
35483626 }
35493627
3550−__bpf_kfunc int bpf_probe_read_kernel_dynptr(struct bpf_dynptr *dptr, u64 off,
3628+__bpf_kfunc int bpf_probe_read_kernel_dynptr(const struct bpf_dynptr *dptr, u64 off,
35513629 u64 size, const void *unsafe_ptr__ign)
35523630 {
35533631 return __bpf_dynptr_copy(dptr, off, size, unsafe_ptr__ign,
35543632 copy_kernel_data_nofault, NULL);
35553633 }
35563634
3557−__bpf_kfunc int bpf_probe_read_user_str_dynptr(struct bpf_dynptr *dptr, u64 off,
3635+__bpf_kfunc int bpf_probe_read_user_str_dynptr(const struct bpf_dynptr *dptr, u64 off,
35583636 u64 size, const void __user *unsafe_ptr__ign)
35593637 {
35603638 return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,
35613639 copy_user_str_nofault, NULL);
35623640 }
35633641
3564−__bpf_kfunc int bpf_probe_read_kernel_str_dynptr(struct bpf_dynptr *dptr, u64 off,
3642+__bpf_kfunc int bpf_probe_read_kernel_str_dynptr(const struct bpf_dynptr *dptr, u64 off,
35653643 u64 size, const void *unsafe_ptr__ign)
35663644 {
35673645 return __bpf_dynptr_copy_str(dptr, off, size, unsafe_ptr__ign,
35683646 copy_kernel_str_nofault, NULL);
35693647 }
35703648
3571−__bpf_kfunc int bpf_copy_from_user_dynptr(struct bpf_dynptr *dptr, u64 off,
3649+__bpf_kfunc int bpf_copy_from_user_dynptr(const struct bpf_dynptr *dptr, u64 off,
35723650 u64 size, const void __user *unsafe_ptr__ign)
35733651 {
35743652 return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,
35753653 copy_user_data_sleepable, NULL);
35763654 }
35773655
3578−__bpf_kfunc int bpf_copy_from_user_str_dynptr(struct bpf_dynptr *dptr, u64 off,
3656+__bpf_kfunc int bpf_copy_from_user_str_dynptr(const struct bpf_dynptr *dptr, u64 off,
35793657 u64 size, const void __user *unsafe_ptr__ign)
35803658 {
35813659 return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,
35823660 copy_user_str_sleepable, NULL);
35833661 }
35843662
3585−__bpf_kfunc int bpf_copy_from_user_task_dynptr(struct bpf_dynptr *dptr, u64 off,
3663+__bpf_kfunc int bpf_copy_from_user_task_dynptr(const struct bpf_dynptr *dptr, u64 off,
35863664 u64 size, const void __user *unsafe_ptr__ign,
35873665 struct task_struct *tsk)
35883666 {
@@ -3590,7 +3668,7 @@ __bpf_kfunc int bpf_copy_from_user_task_dynptr(struct bpf_dynptr *dptr, u64 off, · 1 unmodified line
35903668 copy_user_data_sleepable, tsk);
35913669 }
35923670
3593−__bpf_kfunc int bpf_copy_from_user_task_str_dynptr(struct bpf_dynptr *dptr, u64 off,
3671+__bpf_kfunc int bpf_copy_from_user_task_str_dynptr(const struct bpf_dynptr *dptr, u64 off,
35943672 u64 size, const void __user *unsafe_ptr__ign,
35953673 struct task_struct *tsk)
35963674 {
@@ -3599,3 +3677,203 @@ __bpf_kfunc int bpf_copy_from_user_task_str_dynptr(struct bpf_dynptr *dptr, u64 · 2 unmodified lines
35993677 }
36003678
36013679 __bpf_kfunc_end_defs();
3680+
3681+#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \
3682+ defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS)
3683+
3684+static void bpf_tracing_multi_link_release(struct bpf_link *link)
3685+{
3686+ struct bpf_tracing_multi_link *tr_link =
3687+ container_of(link, struct bpf_tracing_multi_link, link);
3688+
3689+ WARN_ON_ONCE(bpf_trampoline_multi_detach(link->prog, tr_link));
3690+}
3691+
3692+static void bpf_tracing_multi_link_dealloc(struct bpf_link *link)
3693+{
3694+ struct bpf_tracing_multi_link *tr_link =
3695+ container_of(link, struct bpf_tracing_multi_link, link);
3696+
3697+ kvfree(tr_link->fexits);
3698+ kvfree(tr_link->cookies);
3699+ kvfree(tr_link);
3700+}
3701+
3702+#ifdef CONFIG_PROC_FS
3703+static void bpf_tracing_multi_show_fdinfo(const struct bpf_link *link,
3704+ struct seq_file *seq)
3705+{
3706+ struct bpf_tracing_multi_link *tr_link =
3707+ container_of(link, struct bpf_tracing_multi_link, link);
3708+ bool has_cookies = !!tr_link->cookies;
3709+
3710+ seq_printf(seq, "attach_type:\t%u\n", tr_link->link.attach_type);
3711+ seq_printf(seq, "cnt:\t%u\n", tr_link->nodes_cnt);
3712+
3713+ seq_printf(seq, "%s\t %s\t %s\t %s\n", "obj-id", "btf-id", "cookie", "func");
3714+ for (int i = 0; i < tr_link->nodes_cnt; i++) {
3715+ struct bpf_tracing_multi_node *mnode = &tr_link->nodes[i];
3716+ u32 btf_id, obj_id;
3717+
3718+ bpf_trampoline_unpack_key(mnode->trampoline->key, &obj_id, &btf_id);
3719+ seq_printf(seq, "%u\t %u\t %llu\t %pS\n",
3720+ obj_id, btf_id,
3721+ has_cookies ? tr_link->cookies[i] : 0,
3722+ (void *) mnode->trampoline->ip);
3723+
3724+ cond_resched();
3725+ }
3726+}
3727+#endif
3728+
3729+static const struct bpf_link_ops bpf_tracing_multi_link_lops = {
3730+ .release = bpf_tracing_multi_link_release,
3731+ .dealloc_deferred = bpf_tracing_multi_link_dealloc,
3732+#ifdef CONFIG_PROC_FS
3733+ .show_fdinfo = bpf_tracing_multi_show_fdinfo,
3734+#endif
3735+};
3736+
3737+static int ids_cmp_r(const void *pa, const void *pb, const void *priv __maybe_unused)
3738+{
3739+ u32 a = *(u32 *) pa;
3740+ u32 b = *(u32 *) pb;
3741+
3742+ return (a > b) - (a < b);
3743+}
3744+
3745+static void ids_swap_r(void *a, void *b, int size __maybe_unused,
3746+ const void *priv __maybe_unused)
3747+{
3748+ u64 *cookie_a, *cookie_b, *cookies;
3749+ u32 *id_a = a, *id_b = b, *ids;
3750+ void **data = (void **) priv;
3751+
3752+ ids = data[0];
3753+ cookies = data[1];
3754+
3755+ if (cookies) {
3756+ cookie_a = cookies + (id_a - ids);
3757+ cookie_b = cookies + (id_b - ids);
3758+ swap(*cookie_a, *cookie_b);
3759+ }
3760+ swap(*id_a, *id_b);
3761+}
3762+
3763+static int check_dup_ids(u32 *ids, u64 *cookies, u32 cnt)
3764+{
3765+ void *data[2] = { ids, cookies };
3766+ int err = 0;
3767+
3768+ /*
3769+ * Sort ids array (together with cookies array if defined)
3770+ * and check it for duplicates. The ids and cookies arrays
3771+ * are left sorted.
3772+ */
3773+ sort_r_nonatomic(ids, cnt, sizeof(ids[0]), ids_cmp_r, ids_swap_r, data);
3774+
3775+ for (int i = 1; i < cnt; i++) {
3776+ if (ids[i] == ids[i - 1]) {
3777+ err = -EINVAL;
3778+ break;
3779+ }
3780+ }
3781+ return err;
3782+}
3783+
3784+int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
3785+{
3786+ struct bpf_tracing_multi_link *link = NULL;
3787+ struct bpf_tramp_node *fexits = NULL;
3788+ struct bpf_link_primer link_primer;
3789+ u32 cnt, *ids = NULL;
3790+ u64 __user *ucookies;
3791+ u64 *cookies = NULL;
3792+ u32 __user *uids;
3793+ int err;
3794+
3795+ uids = u64_to_user_ptr(attr->link_create.tracing_multi.ids);
3796+ cnt = attr->link_create.tracing_multi.cnt;
3797+
3798+ if (!cnt || !uids)
3799+ return -EINVAL;
3800+ if (cnt > MAX_TRACING_MULTI_CNT)
3801+ return -E2BIG;
3802+ if (attr->link_create.flags || attr->link_create.target_fd)
3803+ return -EINVAL;
3804+
3805+ ids = kvmalloc_objs(*ids, cnt);
3806+ if (!ids)
3807+ return -ENOMEM;
3808+
3809+ if (copy_from_user(ids, uids, cnt * sizeof(*ids))) {
3810+ err = -EFAULT;
3811+ goto error;
3812+ }
3813+
3814+ ucookies = u64_to_user_ptr(attr->link_create.tracing_multi.cookies);
3815+ if (ucookies) {
3816+ cookies = kvmalloc_objs(*cookies, cnt);
3817+ if (!cookies) {
3818+ err = -ENOMEM;
3819+ goto error;
3820+ }
3821+ if (copy_from_user(cookies, ucookies, cnt * sizeof(*cookies))) {
3822+ err = -EFAULT;
3823+ goto error;
3824+ }
3825+ }
3826+
3827+ err = check_dup_ids(ids, cookies, cnt);
3828+ if (err)
3829+ goto error;
3830+
3831+ if (prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) {
3832+ fexits = kvmalloc_objs(*fexits, cnt);
3833+ if (!fexits) {
3834+ err = -ENOMEM;
3835+ goto error;
3836+ }
3837+ }
3838+
3839+ link = kvzalloc_flex(*link, nodes, cnt);
3840+ if (!link) {
3841+ err = -ENOMEM;
3842+ goto error;
3843+ }
3844+
3845+ bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING_MULTI,
3846+ &bpf_tracing_multi_link_lops, prog, prog->expected_attach_type);
3847+
3848+ err = bpf_link_prime(&link->link, &link_primer);
3849+ if (err)
3850+ goto error;
3851+
3852+ link->nodes_cnt = cnt;
3853+ link->cookies = cookies;
3854+ link->fexits = fexits;
3855+
3856+ err = bpf_trampoline_multi_attach(prog, ids, link);
3857+ kvfree(ids);
3858+ if (err) {
3859+ bpf_link_cleanup(&link_primer);
3860+ return err;
3861+ }
3862+ return bpf_link_settle(&link_primer);
3863+
3864+error:
3865+ kvfree(fexits);
3866+ kvfree(cookies);
3867+ kvfree(ids);
3868+ kvfree(link);
3869+ return err;
3870+}
3871+
3872+#else
3873+
3874+int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
3875+{
3876+ return -EOPNOTSUPP;
3877+}
3878+
3879+#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS && CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */
kernel/trace/ftrace.c+29−6
@@ -1198,8 +1198,7 @@ ftrace_lookup_ip(struct ftrace_hash *hash, unsigned long ip) · 1197 unmodified lines
11981198 return __ftrace_lookup_ip(hash, ip);
11991199 }
12001200
1201−static void __add_hash_entry(struct ftrace_hash *hash,
1202− struct ftrace_func_entry *entry)
1201+void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry)
12031202 {
12041203 struct hlist_head *hhd;
12051204 unsigned long key;
@@ -1221,7 +1220,7 @@ add_ftrace_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigne · 15 unmodified lines
12211220
12221221 entry->ip = ip;
12231222 entry->direct = direct;
1224− __add_hash_entry(hash, entry);
1223+ add_ftrace_hash_entry(hash, entry);
12251224
12261225 return entry;
12271226 }
@@ -1249,6 +1248,25 @@ remove_hash_entry(struct ftrace_hash *hash, · 21 unmodified lines
12491248 hash->count--;
12501249 }
12511250
1251+void ftrace_hash_remove(struct ftrace_hash *hash)
1252+{
1253+ struct ftrace_func_entry *entry;
1254+ struct hlist_head *hhd;
1255+ struct hlist_node *tn;
1256+ int size;
1257+ int i;
1258+
1259+ if (!hash || !hash->count)
1260+ return;
1261+ size = 1 << hash->size_bits;
1262+ for (i = 0; i < size; i++) {
1263+ hhd = &hash->buckets[i];
1264+ hlist_for_each_entry_safe(entry, tn, hhd, hlist)
1265+ remove_hash_entry(hash, entry);
1266+ }
1267+ FTRACE_WARN_ON(hash->count);
1268+}
1269+
12521270 static void ftrace_hash_clear(struct ftrace_hash *hash)
12531271 {
12541272 struct hlist_head *hhd;
@@ -1458,7 +1476,7 @@ static struct ftrace_hash *__move_hash(struct ftrace_hash *src, int size) · 203 unmodified lines
14581476 hhd = &src->buckets[i];
14591477 hlist_for_each_entry_safe(entry, tn, hhd, hlist) {
14601478 remove_hash_entry(src, entry);
1461− __add_hash_entry(new_hash, entry);
1479+ add_ftrace_hash_entry(new_hash, entry);
14621480 }
14631481 }
14641482 return new_hash;
@@ -5341,7 +5359,7 @@ int ftrace_func_mapper_add_ip(struct ftrace_func_mapper *mapper, · 3876 unmodified lines
53415359 map->entry.ip = ip;
53425360 map->data = data;
53435361
5344− __add_hash_entry(&mapper->hash, &map->entry);
5362+ add_ftrace_hash_entry(&mapper->hash, &map->entry);
53455363
53465364 return 0;
53475365 }
@@ -6288,11 +6306,16 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) · 940 unmodified lines
62886306 }
62896307 EXPORT_SYMBOL_GPL(modify_ftrace_direct);
62906308
6291−static unsigned long hash_count(struct ftrace_hash *hash)
6309+static inline unsigned long hash_count(struct ftrace_hash *hash)
62926310 {
62936311 return hash ? hash->count : 0;
62946312 }
62956313
6314+unsigned long ftrace_hash_count(struct ftrace_hash *hash)
6315+{
6316+ return hash_count(hash);
6317+}
6318+
62966319 /**
62976320 * hash_add - adds two struct ftrace_hash and returns the result
62986321 * @a: struct ftrace_hash object
kernel/trace/trace_syscalls.c+58−52
@@ -1371,33 +1371,33 @@ static DECLARE_BITMAP(enabled_perf_exit_syscalls, NR_syscalls); · 1370 unmodified lines
13711371 static int sys_perf_refcount_enter;
13721372 static int sys_perf_refcount_exit;
13731373
1374−static int perf_call_bpf_enter(struct trace_event_call *call, struct pt_regs *regs,
1374+static int perf_call_bpf_enter(struct trace_event_call *call,
13751375 struct syscall_metadata *sys_data,
1376− struct syscall_trace_enter *rec)
1376+ int syscall_nr, unsigned long *args)
13771377 {
13781378 struct syscall_tp_t {
13791379 struct trace_entry ent;
13801380 int syscall_nr;
13811381 unsigned long args[SYSCALL_DEFINE_MAXARGS];
13821382 } __aligned(8) param;
1383+ struct pt_regs regs = {};
13831384 int i;
13841385
13851386 BUILD_BUG_ON(sizeof(param.ent) < sizeof(void *));
13861387
1387− /* bpf prog requires 'regs' to be the first member in the ctx (a.k.a. &param) */
1388− perf_fetch_caller_regs(regs);
1389− *(struct pt_regs **)&param = regs;
1390− param.syscall_nr = rec->nr;
1388+ /* bpf prog requires 'regs' to be the first member in the ctx */
1389+ perf_fetch_caller_regs(&regs);
1390+ *(struct pt_regs **)&param = &regs;
1391+ param.syscall_nr = syscall_nr;
13911392 for (i = 0; i < sys_data->nb_args; i++)
1392− param.args[i] = rec->args[i];
1393− return trace_call_bpf(call, &param);
1393+ param.args[i] = args[i];
1394+ return trace_call_bpf_faultable(call, &param);
13941395 }
13951396
13961397 static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)
13971398 {
13981399 struct syscall_metadata *sys_data;
13991400 struct syscall_trace_enter *rec;
1400− struct pt_regs *fake_regs;
14011401 struct hlist_head *head;
14021402 unsigned long args[6];
14031403 bool valid_prog_array;
@@ -1410,12 +1410,7 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 6 unmodified lines
14101410 int size = 0;
14111411 int uargs = 0;
14121412
1413− /*
1414− * Syscall probe called with preemption enabled, but the ring
1415− * buffer and per-cpu data require preemption to be disabled.
1416− */
14171413 might_fault();
1418− guard(preempt_notrace)();
14191414
14201415 syscall_nr = trace_get_syscall_nr(current, regs);
14211416 if (syscall_nr < 0 || syscall_nr >= NR_syscalls)
@@ -1429,6 +1424,26 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 7 unmodified lines
14291424
14301425 syscall_get_arguments(current, regs, args);
14311426
1427+ /*
1428+ * Run BPF program in faultable context before per-cpu buffer
1429+ * allocation, allowing sleepable BPF programs to execute.
1430+ */
1431+ valid_prog_array = bpf_prog_array_valid(sys_data->enter_event);
1432+ if (valid_prog_array &&
1433+ !perf_call_bpf_enter(sys_data->enter_event, sys_data,
1434+ syscall_nr, args))
1435+ return;
1436+
1437+ /*
1438+ * Per-cpu ring buffer and perf event list operations require
1439+ * preemption to be disabled.
1440+ */
1441+ guard(preempt_notrace)();
1442+
1443+ head = this_cpu_ptr(sys_data->enter_event->perf_events);
1444+ if (hlist_empty(head))
1445+ return;
1446+
14321447 /* Check if this syscall event faults in user space memory */
14331448 mayfault = sys_data->user_mask != 0;
14341449
@@ -1438,17 +1453,12 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 3 unmodified lines
14381453 return;
14391454 }
14401455
1441− head = this_cpu_ptr(sys_data->enter_event->perf_events);
1442− valid_prog_array = bpf_prog_array_valid(sys_data->enter_event);
1443− if (!valid_prog_array && hlist_empty(head))
1444− return;
1445−
14461456 /* get the size after alignment with the u32 buffer size field */
14471457 size += sizeof(unsigned long) * sys_data->nb_args + sizeof(*rec);
14481458 size = ALIGN(size + sizeof(u32), sizeof(u64));
14491459 size -= sizeof(u32);
14501460
1451− rec = perf_trace_buf_alloc(size, &fake_regs, &rctx);
1461+ rec = perf_trace_buf_alloc(size, NULL, &rctx);
14521462 if (!rec)
14531463 return;
14541464
@@ -1458,13 +1468,6 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 3 unmodified lines
14581468 if (mayfault)
14591469 syscall_put_data(sys_data, rec, user_ptr, size, user_sizes, uargs);
14601470
1461− if ((valid_prog_array &&
1462− !perf_call_bpf_enter(sys_data->enter_event, fake_regs, sys_data, rec)) ||
1463− hlist_empty(head)) {
1464− perf_swevent_put_recursion_context(rctx);
1465− return;
1466− }
1467−
14681471 perf_trace_buf_submit(rec, size, rctx,
14691472 sys_data->enter_event->event.type, 1, regs,
14701473 head, NULL);
@@ -1514,40 +1517,35 @@ static void perf_sysenter_disable(struct trace_event_call *call) · 43 unmodified lines
15141517 syscall_fault_buffer_disable();
15151518 }
15161519
1517−static int perf_call_bpf_exit(struct trace_event_call *call, struct pt_regs *regs,
1518− struct syscall_trace_exit *rec)
1520+static int perf_call_bpf_exit(struct trace_event_call *call,
1521+ int syscall_nr, long ret_val)
15191522 {
15201523 struct syscall_tp_t {
15211524 struct trace_entry ent;
15221525 int syscall_nr;
15231526 unsigned long ret;
15241527 } __aligned(8) param;
1525−
1526− /* bpf prog requires 'regs' to be the first member in the ctx (a.k.a. &param) */
1527− perf_fetch_caller_regs(regs);
1528− *(struct pt_regs **)&param = regs;
1529− param.syscall_nr = rec->nr;
1530− param.ret = rec->ret;
1531− return trace_call_bpf(call, &param);
1528+ struct pt_regs regs = {};
1529+
1530+ /* bpf prog requires 'regs' to be the first member in the ctx */
1531+ perf_fetch_caller_regs(&regs);
1532+ *(struct pt_regs **)&param = &regs;
1533+ param.syscall_nr = syscall_nr;
1534+ param.ret = ret_val;
1535+ return trace_call_bpf_faultable(call, &param);
15321536 }
15331537
15341538 static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret)
15351539 {
15361540 struct syscall_metadata *sys_data;
15371541 struct syscall_trace_exit *rec;
1538− struct pt_regs *fake_regs;
15391542 struct hlist_head *head;
15401543 bool valid_prog_array;
15411544 int syscall_nr;
15421545 int rctx;
15431546 int size;
15441547
1545− /*
1546− * Syscall probe called with preemption enabled, but the ring
1547− * buffer and per-cpu data require preemption to be disabled.
1548− */
15491548 might_fault();
1550− guard(preempt_notrace)();
15511549
15521550 syscall_nr = trace_get_syscall_nr(current, regs);
15531551 if (syscall_nr < 0 || syscall_nr >= NR_syscalls)
@@ -1559,29 +1557,37 @@ static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) · 5 unmodified lines
15591557 if (!sys_data)
15601558 return;
15611559
1562− head = this_cpu_ptr(sys_data->exit_event->perf_events);
1560+ /*
1561+ * Run BPF program in faultable context before per-cpu buffer
1562+ * allocation, allowing sleepable BPF programs to execute.
1563+ */
15631564 valid_prog_array = bpf_prog_array_valid(sys_data->exit_event);
1564− if (!valid_prog_array && hlist_empty(head))
1565+ if (valid_prog_array &&
1566+ !perf_call_bpf_exit(sys_data->exit_event, syscall_nr,
1567+ syscall_get_return_value(current, regs)))
1568+ return;
1569+
1570+ /*
1571+ * Per-cpu ring buffer and perf event list operations require
1572+ * preemption to be disabled.
1573+ */
1574+ guard(preempt_notrace)();
1575+
1576+ head = this_cpu_ptr(sys_data->exit_event->perf_events);
1577+ if (hlist_empty(head))
15651578 return;
15661579
15671580 /* We can probably do that at build time */
15681581 size = ALIGN(sizeof(*rec) + sizeof(u32), sizeof(u64));
15691582 size -= sizeof(u32);
15701583
1571− rec = perf_trace_buf_alloc(size, &fake_regs, &rctx);
1584+ rec = perf_trace_buf_alloc(size, NULL, &rctx);
15721585 if (!rec)
15731586 return;
15741587
15751588 rec->nr = syscall_nr;
15761589 rec->ret = syscall_get_return_value(current, regs);
15771590
1578− if ((valid_prog_array &&
1579− !perf_call_bpf_exit(sys_data->exit_event, fake_regs, rec)) ||
1580− hlist_empty(head)) {
1581− perf_swevent_put_recursion_context(rctx);
1582− return;
1583− }
1584−
15851591 perf_trace_buf_submit(rec, size, rctx, sys_data->exit_event->event.type,
15861592 1, regs, head, NULL);
15871593 }
lib/rhashtable.c+102−0
@@ -687,6 +687,108 @@ void *rhashtable_insert_slow(struct rhashtable *ht, const void *key, · 686 unmodified lines
687687 }
688688 EXPORT_SYMBOL_GPL(rhashtable_insert_slow);
689689
690+/* Scan one element forward from prev_key's position in @tbl.
691+ * Returns first rhash_head whose bucket > prev_key's bucket, or the
692+ * element immediately after prev_key inside prev_key's bucket.
693+ * Returns the first element if prev_key is NULL, NULL when @tbl is
694+ * exhausted, or ERR_PTR(-ENOENT) if prev_key is not found in @tbl.
695+ */
696+static struct rhash_head *__rhashtable_next_in_table(
697+ struct rhashtable *ht, struct bucket_table *tbl,
698+ const void *prev_key)
699+{
700+ struct rhashtable_compare_arg arg = { .ht = ht, .key = prev_key };
701+ const struct rhashtable_params params = ht->p;
702+ struct rhash_head *he;
703+ unsigned int b = 0;
704+ bool found = false;
705+
706+ if (prev_key) {
707+ b = rht_key_hashfn(ht, tbl, prev_key, params);
708+ rht_for_each_rcu(he, tbl, b) {
709+ bool match = params.obj_cmpfn
710+ ? !params.obj_cmpfn(&arg, rht_obj(ht, he))
711+ : !rhashtable_compare(&arg, rht_obj(ht, he));
712+ if (found) {
713+ if (match)
714+ continue;
715+ return he;
716+ }
717+ if (match)
718+ found = true;
719+ }
720+ if (!found)
721+ return ERR_PTR(-ENOENT);
722+ b++;
723+ }
724+
725+ for (; b < tbl->size; b++)
726+ rht_for_each_rcu(he, tbl, b)
727+ return he;
728+ return NULL;
729+}
730+
731+/**
732+ * rhashtable_next_key - return next element after a given key
733+ * @ht: hash table
734+ * @prev_key: pointer to previous key, or NULL for the first element
735+ *
736+ * WARNING: this walk is highly unstable. Unlike rhashtable_walk_*(),
737+ * it cannot detect a concurrent resize or rehash, so a full iteration
738+ * is NOT guaranteed to terminate under adversarial or sustained
739+ * rehashing. Callers MUST tolerate skipped and duplicated elements and
740+ * SHOULD bound their loop externally.
741+ *
742+ * Returns the next element in best-effort iteration order, walking the
743+ * @tbl chain (including any future_tbl in flight). Caller must hold RCU.
744+ *
745+ * Pass @prev_key == NULL to obtain the first element. To iterate, set
746+ * @prev_key to the key of the previously returned element on each call,
747+ * and stop when NULL is returned.
748+ *
749+ * Best-effort semantics:
750+ * - Across the tbl->future_tbl chain, an element being migrated may
751+ * transiently appear in both tables and be observed twice.
752+ * - Concurrent inserts may or may not be observed.
753+ * - Termination of a full iteration loop is NOT guaranteed under
754+ * adversarial continuous rehash; callers MUST tolerate skips and
755+ * repeats and SHOULD bound their loop externally.
756+ * - Behavior on tables that contain duplicate keys is undefined:
757+ * duplicates may be skipped, repeated, or trap the walk in a
758+ * cycle. Callers requiring duplicate-key iteration must use
759+ * rhashtable_walk_*() instead.
760+ * - rhltable instances are not supported and return
761+ * ERR_PTR(-EOPNOTSUPP).
762+ * - If prev_key was concurrently deleted and is not present in any
763+ * in-flight table, returns ERR_PTR(-ENOENT).
764+ *
765+ * Returns entry of the next element, or NULL when iteration is exhausted,
766+ * or ERR_PTR(-ENOENT) if prev_key is not found, or
767+ * ERR_PTR(-EOPNOTSUPP) if @ht is an rhltable.
768+ */
769+void *rhashtable_next_key(struct rhashtable *ht, const void *prev_key)
770+{
771+ struct bucket_table *tbl;
772+ struct rhash_head *he;
773+
774+ if (unlikely(ht->rhlist))
775+ return ERR_PTR(-EOPNOTSUPP);
776+
777+ tbl = rht_dereference_rcu(ht->tbl, ht);
778+ do {
779+ he = __rhashtable_next_in_table(ht, tbl, prev_key);
780+ if (!IS_ERR_OR_NULL(he))
781+ return rht_obj(ht, he);
782+ if (!he)
783+ prev_key = NULL;
784+ /* See any new future_tbl attached during a rehash. */
785+ smp_rmb();
786+ tbl = rht_dereference_rcu(tbl->future_tbl, ht);
787+ } while (tbl);
788+ return he; /* NULL or -ENOENT */
789+}
790+EXPORT_SYMBOL_GPL(rhashtable_next_key);
791+
690792 /**
691793 * rhashtable_walk_enter - Initialise an iterator
692794 * @ht: Table to walk over
lib/test_bpf.c+263−100
@@ -560,8 +560,23 @@ static int bpf_fill_max_jmp_never_taken(struct bpf_test *self) · 559 unmodified lines
560560 }
561561
562562 /* ALU result computation used in tests */
563−static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op)
563+enum { F_ALU32 = 1, F_SIGNED = 2 };
564+
565+static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op, u32 flags)
564566 {
567+ bool is_signed = flags & F_SIGNED;
568+
569+ /* Narrow operands for ALU32 */
570+ if (flags & F_ALU32) {
571+ if (is_signed) {
572+ v1 = (u64)(s32)v1;
573+ v2 = (u64)(s32)v2;
574+ } else {
575+ v1 = (u32)v1;
576+ v2 = (u32)v2;
577+ }
578+ }
579+
565580 *res = 0;
566581 switch (op) {
567582 case BPF_MOV:
@@ -599,12 +614,28 @@ static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op) · 31 unmodified lines
599614 case BPF_DIV:
600615 if (v2 == 0)
601616 return false;
602− *res = div64_u64(v1, v2);
617+ if (!is_signed) {
618+ *res = div64_u64(v1, v2);
619+ } else {
620+ if ((s64)v2 == -1) /* Handled by verifier */
621+ return false;
622+ *res = (u64)div64_s64(v1, v2);
623+ }
603624 break;
604625 case BPF_MOD:
605626 if (v2 == 0)
606627 return false;
607− div64_u64_rem(v1, v2, res);
628+ if (!is_signed) {
629+ div64_u64_rem(v1, v2, res);
630+ } else {
631+ if ((s64)v2 == -1)
632+ return false;
633+ /*
634+ * Avoid s64 % s64 which generates __moddi3 on
635+ * 32-bit architectures. Use div64_s64 instead.
636+ */
637+ *res = (u64)((s64)v1 - div64_s64(v1, v2) * (s64)v2);
638+ }
608639 break;
609640 }
610641 return true;
@@ -612,7 +643,7 @@ static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op) · 1 unmodified line
612643
613644 /* Test an ALU shift operation for all valid shift values */
614645 static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op,
615− u8 mode, bool alu32)
646+ u8 mode, u32 flags)
616647 {
617648 static const s64 regs[] = {
618649 0x0123456789abcdefLL, /* dword > 0, word < 0 */
@@ -620,7 +651,7 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 1 unmodified line
620651 0xfedcba0198765432LL, /* dword < 0, word < 0 */
621652 0x0123458967abcdefLL, /* dword > 0, word > 0 */
622653 };
623− int bits = alu32 ? 32 : 64;
654+ int bits = (flags & F_ALU32) ? 32 : 64;
624655 int len = (2 + 7 * bits) * ARRAY_SIZE(regs) + 3;
625656 struct bpf_insn *insn;
626657 int imm, k;
@@ -643,7 +674,7 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 16 unmodified lines
643674 /* Perform operation */
644675 insn[i++] = BPF_ALU64_REG(BPF_MOV, R1, R3);
645676 insn[i++] = BPF_ALU64_IMM(BPF_MOV, R2, imm);
646− if (alu32) {
677+ if (flags & F_ALU32) {
647678 if (mode == BPF_K)
648679 insn[i++] = BPF_ALU32_IMM(op, R1, imm);
649680 else
@@ -653,14 +684,14 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 3 unmodified lines
653684 reg = (s32)reg;
654685 else
655686 reg = (u32)reg;
656− __bpf_alu_result(&val, reg, imm, op);
687+ __bpf_alu_result(&val, reg, imm, op, 0);
657688 val = (u32)val;
658689 } else {
659690 if (mode == BPF_K)
660691 insn[i++] = BPF_ALU64_IMM(op, R1, imm);
661692 else
662693 insn[i++] = BPF_ALU64_REG(op, R1, R2);
663− __bpf_alu_result(&val, reg, imm, op);
694+ __bpf_alu_result(&val, reg, imm, op, 0);
664695 }
665696
666697 /*
@@ -688,62 +719,62 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 21 unmodified lines
688719
689720 static int bpf_fill_alu64_lsh_imm(struct bpf_test *self)
690721 {
691− return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, false);
722+ return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, 0);
692723 }
693724
694725 static int bpf_fill_alu64_rsh_imm(struct bpf_test *self)
695726 {
696− return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, false);
727+ return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, 0);
697728 }
698729
699730 static int bpf_fill_alu64_arsh_imm(struct bpf_test *self)
700731 {
701− return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, false);
732+ return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, 0);
702733 }
703734
704735 static int bpf_fill_alu64_lsh_reg(struct bpf_test *self)
705736 {
706− return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, false);
737+ return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, 0);
707738 }
708739
709740 static int bpf_fill_alu64_rsh_reg(struct bpf_test *self)
710741 {
711− return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, false);
742+ return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, 0);
712743 }
713744
714745 static int bpf_fill_alu64_arsh_reg(struct bpf_test *self)
715746 {
716− return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, false);
747+ return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, 0);
717748 }
718749
719750 static int bpf_fill_alu32_lsh_imm(struct bpf_test *self)
720751 {
721− return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, true);
752+ return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, F_ALU32);
722753 }
723754
724755 static int bpf_fill_alu32_rsh_imm(struct bpf_test *self)
725756 {
726− return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, true);
757+ return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, F_ALU32);
727758 }
728759
729760 static int bpf_fill_alu32_arsh_imm(struct bpf_test *self)
730761 {
731− return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, true);
762+ return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, F_ALU32);
732763 }
733764
734765 static int bpf_fill_alu32_lsh_reg(struct bpf_test *self)
735766 {
736− return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, true);
767+ return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, F_ALU32);
737768 }
738769
739770 static int bpf_fill_alu32_rsh_reg(struct bpf_test *self)
740771 {
741− return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, true);
772+ return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, F_ALU32);
742773 }
743774
744775 static int bpf_fill_alu32_arsh_reg(struct bpf_test *self)
745776 {
746− return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, true);
777+ return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, F_ALU32);
747778 }
748779
749780 /*
@@ -751,9 +782,9 @@ static int bpf_fill_alu32_arsh_reg(struct bpf_test *self) · 1 unmodified line
751782 * for the case when the source and destination are the same.
752783 */
753784 static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op,
754− bool alu32)
785+ u32 flags)
755786 {
756− int bits = alu32 ? 32 : 64;
787+ int bits = (flags & F_ALU32) ? 32 : 64;
757788 int len = 3 + 6 * bits;
758789 struct bpf_insn *insn;
759790 int i = 0;
@@ -770,14 +801,14 @@ static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op, · 10 unmodified lines
770801
771802 /* Perform operation */
772803 insn[i++] = BPF_ALU64_IMM(BPF_MOV, R1, val);
773− if (alu32)
804+ if (flags & F_ALU32)
774805 insn[i++] = BPF_ALU32_REG(op, R1, R1);
775806 else
776807 insn[i++] = BPF_ALU64_REG(op, R1, R1);
777808
778809 /* Compute the reference result */
779− __bpf_alu_result(&res, val, val, op);
780− if (alu32)
810+ __bpf_alu_result(&res, val, val, op, 0);
811+ if (flags & F_ALU32)
781812 res = (u32)res;
782813 i += __bpf_ld_imm64(&insn[i], R2, res);
783814
@@ -798,32 +829,32 @@ static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op, · 14 unmodified lines
798829
799830 static int bpf_fill_alu64_lsh_same_reg(struct bpf_test *self)
800831 {
801− return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, false);
832+ return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, 0);
802833 }
803834
804835 static int bpf_fill_alu64_rsh_same_reg(struct bpf_test *self)
805836 {
806− return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, false);
837+ return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, 0);
807838 }
808839
809840 static int bpf_fill_alu64_arsh_same_reg(struct bpf_test *self)
810841 {
811− return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, false);
842+ return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, 0);
812843 }
813844
814845 static int bpf_fill_alu32_lsh_same_reg(struct bpf_test *self)
815846 {
816− return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, true);
847+ return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, F_ALU32);
817848 }
818849
819850 static int bpf_fill_alu32_rsh_same_reg(struct bpf_test *self)
820851 {
821− return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, true);
852+ return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, F_ALU32);
822853 }
823854
824855 static int bpf_fill_alu32_arsh_same_reg(struct bpf_test *self)
825856 {
826− return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, true);
857+ return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, F_ALU32);
827858 }
828859
829860 /*
@@ -936,17 +967,20 @@ static int __bpf_fill_pattern(struct bpf_test *self, void *arg, · 106 unmodified lines
936967 static int __bpf_emit_alu64_imm(struct bpf_test *self, void *arg,
937968 struct bpf_insn *insns, s64 dst, s64 imm)
938969 {
939− int op = *(int *)arg;
970+ int *a = arg;
971+ int op = a[0];
972+ u32 flags = a[1];
973+ s16 off = (flags & F_SIGNED) ? 1 : 0;
940974 int i = 0;
941975 u64 res;
942976
943977 if (!insns)
944978 return 7;
945979
946− if (__bpf_alu_result(&res, dst, (s32)imm, op)) {
980+ if (__bpf_alu_result(&res, dst, (s32)imm, op, flags)) {
947981 i += __bpf_ld_imm64(&insns[i], R1, dst);
948982 i += __bpf_ld_imm64(&insns[i], R3, res);
949− insns[i++] = BPF_ALU64_IMM(op, R1, imm);
983+ insns[i++] = BPF_ALU64_IMM_OFF(op, R1, imm, off);
950984 insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1);
951985 insns[i++] = BPF_EXIT_INSN();
952986 }
@@ -957,17 +991,20 @@ static int __bpf_emit_alu64_imm(struct bpf_test *self, void *arg, · 4 unmodified lines
957991 static int __bpf_emit_alu32_imm(struct bpf_test *self, void *arg,
958992 struct bpf_insn *insns, s64 dst, s64 imm)
959993 {
960− int op = *(int *)arg;
994+ int *a = arg;
995+ int op = a[0];
996+ u32 flags = a[1];
997+ s16 off = (flags & F_SIGNED) ? 1 : 0;
961998 int i = 0;
962999 u64 res;
9631000
9641001 if (!insns)
9651002 return 7;
9661003
967− if (__bpf_alu_result(&res, (u32)dst, (u32)imm, op)) {
1004+ if (__bpf_alu_result(&res, dst, (s32)imm, op, flags | F_ALU32)) {
9681005 i += __bpf_ld_imm64(&insns[i], R1, dst);
9691006 i += __bpf_ld_imm64(&insns[i], R3, (u32)res);
970− insns[i++] = BPF_ALU32_IMM(op, R1, imm);
1007+ insns[i++] = BPF_ALU32_IMM_OFF(op, R1, imm, off);
9711008 insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1);
9721009 insns[i++] = BPF_EXIT_INSN();
9731010 }
@@ -985,7 +1022,7 @@ static int __bpf_emit_alu64_reg(struct bpf_test *self, void *arg, · 11 unmodified lines
9851022 if (!insns)
9861023 return 9;
9871024
988− if (__bpf_alu_result(&res, dst, src, op)) {
1025+ if (__bpf_alu_result(&res, dst, src, op, 0)) {
9891026 i += __bpf_ld_imm64(&insns[i], R1, dst);
9901027 i += __bpf_ld_imm64(&insns[i], R2, src);
9911028 i += __bpf_ld_imm64(&insns[i], R3, res);
@@ -1007,7 +1044,7 @@ static int __bpf_emit_alu32_reg(struct bpf_test *self, void *arg, · 15 unmodified lines
10071044 if (!insns)
10081045 return 9;
10091046
1010− if (__bpf_alu_result(&res, (u32)dst, (u32)src, op)) {
1047+ if (__bpf_alu_result(&res, (u32)dst, (u32)src, op, 0)) {
10111048 i += __bpf_ld_imm64(&insns[i], R1, dst);
10121049 i += __bpf_ld_imm64(&insns[i], R2, src);
10131050 i += __bpf_ld_imm64(&insns[i], R3, (u32)res);
@@ -1019,16 +1056,20 @@ static int __bpf_emit_alu32_reg(struct bpf_test *self, void *arg, · 5 unmodified lines
10191056 return i;
10201057 }
10211058
1022−static int __bpf_fill_alu64_imm(struct bpf_test *self, int op)
1059+static int __bpf_fill_alu64_imm(struct bpf_test *self, int op, u32 flags)
10231060 {
1024− return __bpf_fill_pattern(self, &op, 64, 32,
1061+ int arg[2] = {op, flags};
1062+
1063+ return __bpf_fill_pattern(self, &arg, 64, 32,
10251064 PATTERN_BLOCK1, PATTERN_BLOCK2,
10261065 &__bpf_emit_alu64_imm);
10271066 }
10281067
1029−static int __bpf_fill_alu32_imm(struct bpf_test *self, int op)
1068+static int __bpf_fill_alu32_imm(struct bpf_test *self, int op, u32 flags)
10301069 {
1031− return __bpf_fill_pattern(self, &op, 64, 32,
1070+ int arg[2] = {op, flags};
1071+
1072+ return __bpf_fill_pattern(self, &arg, 64, 32,
10321073 PATTERN_BLOCK1, PATTERN_BLOCK2,
10331074 &__bpf_emit_alu32_imm);
10341075 }
@@ -1050,93 +1091,115 @@ static int __bpf_fill_alu32_reg(struct bpf_test *self, int op) · 15 unmodified lines
10501091 /* ALU64 immediate operations */
10511092 static int bpf_fill_alu64_mov_imm(struct bpf_test *self)
10521093 {
1053− return __bpf_fill_alu64_imm(self, BPF_MOV);
1094+ return __bpf_fill_alu64_imm(self, BPF_MOV, 0);
10541095 }
10551096
10561097 static int bpf_fill_alu64_and_imm(struct bpf_test *self)
10571098 {
1058− return __bpf_fill_alu64_imm(self, BPF_AND);
1099+ return __bpf_fill_alu64_imm(self, BPF_AND, 0);
10591100 }
10601101
10611102 static int bpf_fill_alu64_or_imm(struct bpf_test *self)
10621103 {
1063− return __bpf_fill_alu64_imm(self, BPF_OR);
1104+ return __bpf_fill_alu64_imm(self, BPF_OR, 0);
10641105 }
10651106
10661107 static int bpf_fill_alu64_xor_imm(struct bpf_test *self)
10671108 {
1068− return __bpf_fill_alu64_imm(self, BPF_XOR);
1109+ return __bpf_fill_alu64_imm(self, BPF_XOR, 0);
10691110 }
10701111
10711112 static int bpf_fill_alu64_add_imm(struct bpf_test *self)
10721113 {
1073− return __bpf_fill_alu64_imm(self, BPF_ADD);
1114+ return __bpf_fill_alu64_imm(self, BPF_ADD, 0);
10741115 }
10751116
10761117 static int bpf_fill_alu64_sub_imm(struct bpf_test *self)
10771118 {
1078− return __bpf_fill_alu64_imm(self, BPF_SUB);
1119+ return __bpf_fill_alu64_imm(self, BPF_SUB, 0);
10791120 }
10801121
10811122 static int bpf_fill_alu64_mul_imm(struct bpf_test *self)
10821123 {
1083− return __bpf_fill_alu64_imm(self, BPF_MUL);
1124+ return __bpf_fill_alu64_imm(self, BPF_MUL, 0);
10841125 }
10851126
10861127 static int bpf_fill_alu64_div_imm(struct bpf_test *self)
10871128 {
1088− return __bpf_fill_alu64_imm(self, BPF_DIV);
1129+ return __bpf_fill_alu64_imm(self, BPF_DIV, 0);
10891130 }
10901131
10911132 static int bpf_fill_alu64_mod_imm(struct bpf_test *self)
10921133 {
1093− return __bpf_fill_alu64_imm(self, BPF_MOD);
1134+ return __bpf_fill_alu64_imm(self, BPF_MOD, 0);
1135+}
1136+
1137+/* Signed ALU64 immediate operations */
1138+static int bpf_fill_alu64_sdiv_imm(struct bpf_test *self)
1139+{
1140+ return __bpf_fill_alu64_imm(self, BPF_DIV, F_SIGNED);
1141+}
1142+
1143+static int bpf_fill_alu64_smod_imm(struct bpf_test *self)
1144+{
1145+ return __bpf_fill_alu64_imm(self, BPF_MOD, F_SIGNED);
1146+}
1147+
1148+/* Signed ALU32 immediate operations */
1149+static int bpf_fill_alu32_sdiv_imm(struct bpf_test *self)
1150+{
1151+ return __bpf_fill_alu32_imm(self, BPF_DIV, F_SIGNED);
1152+}
1153+
1154+static int bpf_fill_alu32_smod_imm(struct bpf_test *self)
1155+{
1156+ return __bpf_fill_alu32_imm(self, BPF_MOD, F_SIGNED);
10941157 }
10951158
10961159 /* ALU32 immediate operations */
10971160 static int bpf_fill_alu32_mov_imm(struct bpf_test *self)
10981161 {
1099− return __bpf_fill_alu32_imm(self, BPF_MOV);
1162+ return __bpf_fill_alu32_imm(self, BPF_MOV, 0);
11001163 }
11011164
11021165 static int bpf_fill_alu32_and_imm(struct bpf_test *self)
11031166 {
1104− return __bpf_fill_alu32_imm(self, BPF_AND);
1167+ return __bpf_fill_alu32_imm(self, BPF_AND, 0);
11051168 }
11061169
11071170 static int bpf_fill_alu32_or_imm(struct bpf_test *self)
11081171 {
1109− return __bpf_fill_alu32_imm(self, BPF_OR);
1172+ return __bpf_fill_alu32_imm(self, BPF_OR, 0);
11101173 }
11111174
11121175 static int bpf_fill_alu32_xor_imm(struct bpf_test *self)
11131176 {
1114− return __bpf_fill_alu32_imm(self, BPF_XOR);
1177+ return __bpf_fill_alu32_imm(self, BPF_XOR, 0);
11151178 }
11161179
11171180 static int bpf_fill_alu32_add_imm(struct bpf_test *self)
11181181 {
1119− return __bpf_fill_alu32_imm(self, BPF_ADD);
1182+ return __bpf_fill_alu32_imm(self, BPF_ADD, 0);
11201183 }
11211184
11221185 static int bpf_fill_alu32_sub_imm(struct bpf_test *self)
11231186 {
1124− return __bpf_fill_alu32_imm(self, BPF_SUB);
1187+ return __bpf_fill_alu32_imm(self, BPF_SUB, 0);
11251188 }
11261189
11271190 static int bpf_fill_alu32_mul_imm(struct bpf_test *self)
11281191 {
1129− return __bpf_fill_alu32_imm(self, BPF_MUL);
1192+ return __bpf_fill_alu32_imm(self, BPF_MUL, 0);
11301193 }
11311194
11321195 static int bpf_fill_alu32_div_imm(struct bpf_test *self)
11331196 {
1134− return __bpf_fill_alu32_imm(self, BPF_DIV);
1197+ return __bpf_fill_alu32_imm(self, BPF_DIV, 0);
11351198 }
11361199
11371200 static int bpf_fill_alu32_mod_imm(struct bpf_test *self)
11381201 {
1139− return __bpf_fill_alu32_imm(self, BPF_MOD);
1202+ return __bpf_fill_alu32_imm(self, BPF_MOD, 0);
11401203 }
11411204
11421205 /* ALU64 register operations */
@@ -1235,7 +1298,8 @@ static int bpf_fill_alu32_mod_reg(struct bpf_test *self) · 92 unmodified lines
12351298 * Test JITs that implement complex ALU operations as function
12361299 * calls, and must re-arrange operands for argument passing.
12371300 */
1238−static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32)
1301+static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op,
1302+ u32 flags)
12391303 {
12401304 int len = 2 + 10 * 10;
12411305 struct bpf_insn *insns;
@@ -1249,28 +1313,37 @@ static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32) · 7 unmodified lines
12491313 return -ENOMEM;
12501314
12511315 /* Operand and result values according to operation */
1252− if (alu32)
1253− dst = 0x76543210U;
1254− else
1255− dst = 0x7edcba9876543210ULL;
1316+ if (flags & F_SIGNED) {
1317+ if (flags & F_ALU32)
1318+ dst = -76543210;
1319+ else
1320+ dst = -7654321076543210LL;
1321+ } else {
1322+ if (flags & F_ALU32)
1323+ dst = 0x76543210U;
1324+ else
1325+ dst = 0x7edcba9876543210ULL;
1326+ }
12561327 imm = 0x01234567U;
12571328
12581329 if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH)
12591330 imm &= 31;
12601331
1261− __bpf_alu_result(&res, dst, imm, op);
1332+ __bpf_alu_result(&res, dst, imm, op, flags);
12621333
1263− if (alu32)
1334+ if (flags & F_ALU32)
12641335 res = (u32)res;
12651336
12661337 /* Check all operand registers */
12671338 for (rd = R0; rd <= R9; rd++) {
12681339 i += __bpf_ld_imm64(&insns[i], rd, dst);
12691340
1270− if (alu32)
1271− insns[i++] = BPF_ALU32_IMM(op, rd, imm);
1341+ s16 off = (flags & F_SIGNED) ? 1 : 0;
1342+
1343+ if (flags & F_ALU32)
1344+ insns[i++] = BPF_ALU32_IMM_OFF(op, rd, imm, off);
12721345 else
1273− insns[i++] = BPF_ALU64_IMM(op, rd, imm);
1346+ insns[i++] = BPF_ALU64_IMM_OFF(op, rd, imm, off);
12741347
12751348 insns[i++] = BPF_JMP32_IMM(BPF_JEQ, rd, res, 2);
12761349 insns[i++] = BPF_MOV64_IMM(R0, __LINE__);
@@ -1295,123 +1368,145 @@ static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32) · 18 unmodified lines
12951368 /* ALU64 K registers */
12961369 static int bpf_fill_alu64_mov_imm_regs(struct bpf_test *self)
12971370 {
1298− return __bpf_fill_alu_imm_regs(self, BPF_MOV, false);
1371+ return __bpf_fill_alu_imm_regs(self, BPF_MOV, 0);
12991372 }
13001373
13011374 static int bpf_fill_alu64_and_imm_regs(struct bpf_test *self)
13021375 {
1303− return __bpf_fill_alu_imm_regs(self, BPF_AND, false);
1376+ return __bpf_fill_alu_imm_regs(self, BPF_AND, 0);
13041377 }
13051378
13061379 static int bpf_fill_alu64_or_imm_regs(struct bpf_test *self)
13071380 {
1308− return __bpf_fill_alu_imm_regs(self, BPF_OR, false);
1381+ return __bpf_fill_alu_imm_regs(self, BPF_OR, 0);
13091382 }
13101383
13111384 static int bpf_fill_alu64_xor_imm_regs(struct bpf_test *self)
13121385 {
1313− return __bpf_fill_alu_imm_regs(self, BPF_XOR, false);
1386+ return __bpf_fill_alu_imm_regs(self, BPF_XOR, 0);
13141387 }
13151388
13161389 static int bpf_fill_alu64_lsh_imm_regs(struct bpf_test *self)
13171390 {
1318− return __bpf_fill_alu_imm_regs(self, BPF_LSH, false);
1391+ return __bpf_fill_alu_imm_regs(self, BPF_LSH, 0);
13191392 }
13201393
13211394 static int bpf_fill_alu64_rsh_imm_regs(struct bpf_test *self)
13221395 {
1323− return __bpf_fill_alu_imm_regs(self, BPF_RSH, false);
1396+ return __bpf_fill_alu_imm_regs(self, BPF_RSH, 0);
13241397 }
13251398
13261399 static int bpf_fill_alu64_arsh_imm_regs(struct bpf_test *self)
13271400 {
1328− return __bpf_fill_alu_imm_regs(self, BPF_ARSH, false);
1401+ return __bpf_fill_alu_imm_regs(self, BPF_ARSH, 0);
13291402 }
13301403
13311404 static int bpf_fill_alu64_add_imm_regs(struct bpf_test *self)
13321405 {
1333− return __bpf_fill_alu_imm_regs(self, BPF_ADD, false);
1406+ return __bpf_fill_alu_imm_regs(self, BPF_ADD, 0);
13341407 }
13351408
13361409 static int bpf_fill_alu64_sub_imm_regs(struct bpf_test *self)
13371410 {
1338− return __bpf_fill_alu_imm_regs(self, BPF_SUB, false);
1411+ return __bpf_fill_alu_imm_regs(self, BPF_SUB, 0);
13391412 }
13401413
13411414 static int bpf_fill_alu64_mul_imm_regs(struct bpf_test *self)
13421415 {
1343− return __bpf_fill_alu_imm_regs(self, BPF_MUL, false);
1416+ return __bpf_fill_alu_imm_regs(self, BPF_MUL, 0);
13441417 }
13451418
13461419 static int bpf_fill_alu64_div_imm_regs(struct bpf_test *self)
13471420 {
1348− return __bpf_fill_alu_imm_regs(self, BPF_DIV, false);
1421+ return __bpf_fill_alu_imm_regs(self, BPF_DIV, 0);
13491422 }
13501423
13511424 static int bpf_fill_alu64_mod_imm_regs(struct bpf_test *self)
13521425 {
1353− return __bpf_fill_alu_imm_regs(self, BPF_MOD, false);
1426+ return __bpf_fill_alu_imm_regs(self, BPF_MOD, 0);
1427+}
1428+
1429+/* Signed ALU64 K registers */
1430+static int bpf_fill_alu64_sdiv_imm_regs(struct bpf_test *self)
1431+{
1432+ return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_SIGNED);
1433+}
1434+
1435+static int bpf_fill_alu64_smod_imm_regs(struct bpf_test *self)
1436+{
1437+ return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_SIGNED);
13541438 }
13551439
13561440 /* ALU32 K registers */
13571441 static int bpf_fill_alu32_mov_imm_regs(struct bpf_test *self)
13581442 {
1359− return __bpf_fill_alu_imm_regs(self, BPF_MOV, true);
1443+ return __bpf_fill_alu_imm_regs(self, BPF_MOV, F_ALU32);
13601444 }
13611445
13621446 static int bpf_fill_alu32_and_imm_regs(struct bpf_test *self)
13631447 {
1364− return __bpf_fill_alu_imm_regs(self, BPF_AND, true);
1448+ return __bpf_fill_alu_imm_regs(self, BPF_AND, F_ALU32);
13651449 }
13661450
13671451 static int bpf_fill_alu32_or_imm_regs(struct bpf_test *self)
13681452 {
1369− return __bpf_fill_alu_imm_regs(self, BPF_OR, true);
1453+ return __bpf_fill_alu_imm_regs(self, BPF_OR, F_ALU32);
13701454 }
13711455
13721456 static int bpf_fill_alu32_xor_imm_regs(struct bpf_test *self)
13731457 {
1374− return __bpf_fill_alu_imm_regs(self, BPF_XOR, true);
1458+ return __bpf_fill_alu_imm_regs(self, BPF_XOR, F_ALU32);
13751459 }
13761460
13771461 static int bpf_fill_alu32_lsh_imm_regs(struct bpf_test *self)
13781462 {
1379− return __bpf_fill_alu_imm_regs(self, BPF_LSH, true);
1463+ return __bpf_fill_alu_imm_regs(self, BPF_LSH, F_ALU32);
13801464 }
13811465
13821466 static int bpf_fill_alu32_rsh_imm_regs(struct bpf_test *self)
13831467 {
1384− return __bpf_fill_alu_imm_regs(self, BPF_RSH, true);
1468+ return __bpf_fill_alu_imm_regs(self, BPF_RSH, F_ALU32);
13851469 }
13861470
13871471 static int bpf_fill_alu32_arsh_imm_regs(struct bpf_test *self)
13881472 {
1389− return __bpf_fill_alu_imm_regs(self, BPF_ARSH, true);
1473+ return __bpf_fill_alu_imm_regs(self, BPF_ARSH, F_ALU32);
13901474 }
13911475
13921476 static int bpf_fill_alu32_add_imm_regs(struct bpf_test *self)
13931477 {
1394− return __bpf_fill_alu_imm_regs(self, BPF_ADD, true);
1478+ return __bpf_fill_alu_imm_regs(self, BPF_ADD, F_ALU32);
13951479 }
13961480
13971481 static int bpf_fill_alu32_sub_imm_regs(struct bpf_test *self)
13981482 {
1399− return __bpf_fill_alu_imm_regs(self, BPF_SUB, true);
1483+ return __bpf_fill_alu_imm_regs(self, BPF_SUB, F_ALU32);
14001484 }
14011485
14021486 static int bpf_fill_alu32_mul_imm_regs(struct bpf_test *self)
14031487 {
1404− return __bpf_fill_alu_imm_regs(self, BPF_MUL, true);
1488+ return __bpf_fill_alu_imm_regs(self, BPF_MUL, F_ALU32);
14051489 }
14061490
14071491 static int bpf_fill_alu32_div_imm_regs(struct bpf_test *self)
14081492 {
1409− return __bpf_fill_alu_imm_regs(self, BPF_DIV, true);
1493+ return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_ALU32);
14101494 }
14111495
14121496 static int bpf_fill_alu32_mod_imm_regs(struct bpf_test *self)
14131497 {
1414− return __bpf_fill_alu_imm_regs(self, BPF_MOD, true);
1498+ return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_ALU32);
1499+}
1500+
1501+/* Signed ALU32 K registers */
1502+static int bpf_fill_alu32_sdiv_imm_regs(struct bpf_test *self)
1503+{
1504+ return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_ALU32 | F_SIGNED);
1505+}
1506+
1507+static int bpf_fill_alu32_smod_imm_regs(struct bpf_test *self)
1508+{
1509+ return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_ALU32 | F_SIGNED);
14151510 }
14161511
14171512 /*
@@ -1442,8 +1537,8 @@ static int __bpf_fill_alu_reg_pairs(struct bpf_test *self, u8 op, bool alu32) · 24 unmodified lines
14421537 if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH)
14431538 src &= 31;
14441539
1445− __bpf_alu_result(&res, dst, src, op);
1446− __bpf_alu_result(&same, src, src, op);
1540+ __bpf_alu_result(&res, dst, src, op, 0);
1541+ __bpf_alu_result(&same, src, src, op, 0);
14471542
14481543 if (alu32) {
14491544 res = (u32)res;
@@ -1626,7 +1721,7 @@ static int __bpf_emit_atomic64(struct bpf_test *self, void *arg, · 176 unmodified lines
16261721 res = src;
16271722 break;
16281723 default:
1629− __bpf_alu_result(&res, dst, src, BPF_OP(op));
1724+ __bpf_alu_result(&res, dst, src, BPF_OP(op), 0);
16301725 }
16311726
16321727 keep = 0x0123456789abcdefULL;
@@ -1673,7 +1768,7 @@ static int __bpf_emit_atomic32(struct bpf_test *self, void *arg, · 40 unmodified lines
16731768 res = src;
16741769 break;
16751770 default:
1676− __bpf_alu_result(&res, (u32)dst, (u32)src, BPF_OP(op));
1771+ __bpf_alu_result(&res, (u32)dst, (u32)src, BPF_OP(op), 0);
16771772 }
16781773
16791774 keep = 0x0123456789abcdefULL;
@@ -1939,7 +2034,7 @@ static int __bpf_fill_atomic_reg_pairs(struct bpf_test *self, u8 width, u8 op) · 259 unmodified lines
19392034 res = mem;
19402035 break;
19412036 default:
1942− __bpf_alu_result(&res, mem, upd, BPF_OP(op));
2037+ __bpf_alu_result(&res, mem, upd, BPF_OP(op), 0);
19432038 }
19442039
19452040 /* Test all operand registers */
@@ -12354,6 +12449,22 @@ static struct bpf_test tests[] = { · 10408 unmodified lines
1235412449 { { 0, 1 } },
1235512450 .fill_helper = bpf_fill_alu64_mod_imm_regs,
1235612451 },
12452+ {
12453+ "ALU64_SDIV_K: registers",
12454+ { },
12455+ INTERNAL,
12456+ { },
12457+ { { 0, 1 } },
12458+ .fill_helper = bpf_fill_alu64_sdiv_imm_regs,
12459+ },
12460+ {
12461+ "ALU64_SMOD_K: registers",
12462+ { },
12463+ INTERNAL,
12464+ { },
12465+ { { 0, 1 } },
12466+ .fill_helper = bpf_fill_alu64_smod_imm_regs,
12467+ },
1235712468 /* ALU32 K registers */
1235812469 {
1235912470 "ALU32_MOV_K: registers",
@@ -12451,6 +12562,22 @@ static struct bpf_test tests[] = { · 91 unmodified lines
1245112562 { { 0, 1 } },
1245212563 .fill_helper = bpf_fill_alu32_mod_imm_regs,
1245312564 },
12565+ {
12566+ "ALU32_SDIV_K: registers",
12567+ { },
12568+ INTERNAL,
12569+ { },
12570+ { { 0, 1 } },
12571+ .fill_helper = bpf_fill_alu32_sdiv_imm_regs,
12572+ },
12573+ {
12574+ "ALU32_SMOD_K: registers",
12575+ { },
12576+ INTERNAL,
12577+ { },
12578+ { { 0, 1 } },
12579+ .fill_helper = bpf_fill_alu32_smod_imm_regs,
12580+ },
1245412581 /* ALU64 X register combinations */
1245512582 {
1245612583 "ALU64_MOV_X: register combinations",
@@ -12881,6 +13008,24 @@ static struct bpf_test tests[] = { · 424 unmodified lines
1288113008 .fill_helper = bpf_fill_alu64_mod_imm,
1288213009 .nr_testruns = NR_PATTERN_RUNS,
1288313010 },
13011+ {
13012+ "ALU64_SDIV_K: all immediate value magnitudes",
13013+ { },
13014+ INTERNAL | FLAG_NO_DATA,
13015+ { },
13016+ { { 0, 1 } },
13017+ .fill_helper = bpf_fill_alu64_sdiv_imm,
13018+ .nr_testruns = NR_PATTERN_RUNS,
13019+ },
13020+ {
13021+ "ALU64_SMOD_K: all immediate value magnitudes",
13022+ { },
13023+ INTERNAL | FLAG_NO_DATA,
13024+ { },
13025+ { { 0, 1 } },
13026+ .fill_helper = bpf_fill_alu64_smod_imm,
13027+ .nr_testruns = NR_PATTERN_RUNS,
13028+ },
1288413029 /* ALU32 immediate magnitudes */
1288513030 {
1288613031 "ALU32_MOV_K: all immediate value magnitudes",
@@ -12963,6 +13108,24 @@ static struct bpf_test tests[] = { · 76 unmodified lines
1296313108 .fill_helper = bpf_fill_alu32_mod_imm,
1296413109 .nr_testruns = NR_PATTERN_RUNS,
1296513110 },
13111+ {
13112+ "ALU32_SDIV_K: all immediate value magnitudes",
13113+ { },
13114+ INTERNAL | FLAG_NO_DATA,
13115+ { },
13116+ { { 0, 1 } },
13117+ .fill_helper = bpf_fill_alu32_sdiv_imm,
13118+ .nr_testruns = NR_PATTERN_RUNS,
13119+ },
13120+ {
13121+ "ALU32_SMOD_K: all immediate value magnitudes",
13122+ { },
13123+ INTERNAL | FLAG_NO_DATA,
13124+ { },
13125+ { { 0, 1 } },
13126+ .fill_helper = bpf_fill_alu32_smod_imm,
13127+ .nr_testruns = NR_PATTERN_RUNS,
13128+ },
1296613129 /* ALU64 register magnitudes */
1296713130 {
1296813131 "ALU64_MOV_X: all register value magnitudes",
lib/test_rhashtable.c+75−0
@@ -679,6 +679,78 @@ out: · 678 unmodified lines
679679 return err;
680680 }
681681
682+static int __init test_rhashtable_next_key(void)
683+{
684+ struct rhashtable_params params = test_rht_params;
685+ struct test_obj_val key_missing = { .id = 99999, .tid = 0 };
686+ struct test_obj_val *prev_key = NULL;
687+ struct rhashtable ht;
688+ struct test_obj *objs, *cur;
689+ int i, count = 0, err;
690+ int visited_keys[8] = { 0 };
691+ const int n = ARRAY_SIZE(visited_keys);
692+
693+ params.nelem_hint = n;
694+
695+ err = rhashtable_init(&ht, &params);
696+ if (err)
697+ return err;
698+
699+ objs = kcalloc(n, sizeof(*objs), GFP_KERNEL);
700+ if (!objs) {
701+ rhashtable_destroy(&ht);
702+ return -ENOMEM;
703+ }
704+
705+ for (i = 0; i < n; i++) {
706+ objs[i].value.id = i;
707+ err = rhashtable_insert_fast(&ht, &objs[i].node, params);
708+ if (err)
709+ goto out;
710+ }
711+
712+ rcu_read_lock();
713+
714+ /* NULL prev_key: walk from the beginning, expect all n elements. */
715+ while ((cur = rhashtable_next_key(&ht, prev_key))) {
716+ if (IS_ERR(cur)) {
717+ err = -EINVAL;
718+ goto unlock;
719+ }
720+ count++;
721+ prev_key = &cur->value;
722+ visited_keys[cur->value.id] = 1;
723+ if (count > n)
724+ break;
725+ }
726+
727+ if (count != n) {
728+ err = -EINVAL;
729+ goto unlock;
730+ }
731+
732+ for (i = 0; i < n; i++) {
733+ if (!visited_keys[i]) {
734+ err = -EINVAL;
735+ goto unlock;
736+ }
737+ }
738+
739+ /* Non-existing prev_key: must return ERR_PTR(-ENOENT). */
740+ cur = rhashtable_next_key(&ht, &key_missing);
741+ if (!IS_ERR(cur) || PTR_ERR(cur) != -ENOENT)
742+ err = -EINVAL;
743+
744+unlock:
745+ rcu_read_unlock();
746+out:
747+ for (i = 0; i < n; i++)
748+ rhashtable_remove_fast(&ht, &objs[i].node, params);
749+ kfree(objs);
750+ rhashtable_destroy(&ht);
751+ return err;
752+}
753+
682754 static int __init test_rht_init(void)
683755 {
684756 unsigned int entries;
@@ -738,6 +810,9 @@ static int __init test_rht_init(void) · 53 unmodified lines
738810
739811 test_insert_duplicates_run();
740812
813+ pr_info("Testing rhashtable_next_key: %s\n",
814+ test_rhashtable_next_key() == 0 ? "pass" : "FAIL");
815+
741816 if (!tcount)
742817 return 0;
743818
net/bpf/bpf_dummy_struct_ops.c+7−7
@@ -132,7 +132,7 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, · 131 unmodified lines
132132 const struct bpf_struct_ops *st_ops = &bpf_bpf_dummy_ops;
133133 const struct btf_type *func_proto;
134134 struct bpf_dummy_ops_test_args *args;
135− struct bpf_tramp_links *tlinks = NULL;
135+ struct bpf_tramp_nodes *tnodes = NULL;
136136 struct bpf_tramp_link *link = NULL;
137137 void *image = NULL;
138138 unsigned int op_idx;
@@ -158,8 +158,8 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, · 19 unmodified lines
158158 if (err)
159159 goto out;
160160
161− tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX);
162− if (!tlinks) {
161+ tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX);
162+ if (!tnodes) {
163163 err = -ENOMEM;
164164 goto out;
165165 }
@@ -171,11 +171,11 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, · 5 unmodified lines
171171 }
172172 /* prog doesn't take the ownership of the reference from caller */
173173 bpf_prog_inc(prog);
174− bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops, prog,
175− prog->expected_attach_type);
174+ bpf_tramp_link_init(link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops,
175+ prog, prog->expected_attach_type, 0);
176176
177177 op_idx = prog->expected_attach_type;
178− err = bpf_struct_ops_prepare_trampoline(tlinks, link,
178+ err = bpf_struct_ops_prepare_trampoline(tnodes, &link->node,
179179 &st_ops->func_models[op_idx],
180180 &dummy_ops_test_ret_function,
181181 &image, &image_off,
@@ -198,7 +198,7 @@ out: · 16 unmodified lines
198198 bpf_struct_ops_image_free(image);
199199 if (link)
200200 bpf_link_put(&link->link);
201− kfree(tlinks);
201+ kfree(tnodes);
202202 return err;
203203 }
204204
net/bpf/test_run.c+53−15
@@ -702,6 +702,9 @@ int bpf_prog_test_run_tracing(struct bpf_prog *prog, · 701 unmodified lines
702702 case BPF_TRACE_FENTRY:
703703 case BPF_TRACE_FEXIT:
704704 case BPF_TRACE_FSESSION:
705+ case BPF_TRACE_FENTRY_MULTI:
706+ case BPF_TRACE_FEXIT_MULTI:
707+ case BPF_TRACE_FSESSION_MULTI:
705708 if (bpf_fentry_test1(1) != 2 ||
706709 bpf_fentry_test2(2, 3) != 5 ||
707710 bpf_fentry_test3(4, 5, 6) != 15 ||
@@ -747,14 +750,35 @@ static void · 39 unmodified lines
747750 __bpf_prog_test_run_raw_tp(void *data)
748751 {
749752 struct bpf_raw_tp_test_run_info *info = data;
753+ struct srcu_ctr __percpu *scp = NULL;
750754 struct bpf_trace_run_ctx run_ctx = {};
751755 struct bpf_run_ctx *old_run_ctx;
752756
753757 old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
754758
755− rcu_read_lock();
759+ if (info->prog->sleepable) {
760+ scp = rcu_read_lock_tasks_trace();
761+ migrate_disable();
762+ } else {
763+ rcu_read_lock();
764+ }
765+
766+ if (unlikely(!bpf_prog_get_recursion_context(info->prog))) {
767+ bpf_prog_inc_misses_counter(info->prog);
768+ goto out;
769+ }
770+
756771 info->retval = bpf_prog_run(info->prog, info->ctx);
757− rcu_read_unlock();
772+
773+out:
774+ bpf_prog_put_recursion_context(info->prog);
775+
776+ if (info->prog->sleepable) {
777+ migrate_enable();
778+ rcu_read_unlock_tasks_trace(scp);
779+ } else {
780+ rcu_read_unlock();
781+ }
758782
759783 bpf_reset_run_ctx(old_run_ctx);
760784 }
@@ -782,6 +806,13 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog, · 21 unmodified lines
782806 if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 && cpu != 0)
783807 return -EINVAL;
784808
809+ /*
810+ * Sleepable programs cannot run with preemption disabled or in
811+ * hardirq context (smp_call_function_single), reject the flag.
812+ */
813+ if (prog->sleepable && (kattr->test.flags & BPF_F_TEST_RUN_ON_CPU))
814+ return -EINVAL;
815+
785816 if (ctx_size_in) {
786817 info.ctx = memdup_user(ctx_in, ctx_size_in);
787818 if (IS_ERR(info.ctx))
@@ -790,24 +821,31 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog, · 2 unmodified lines
790821 info.ctx = NULL;
791822 }
792823
824+ info.retval = 0;
793825 info.prog = prog;
794826
795− current_cpu = get_cpu();
796− if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 ||
797− cpu == current_cpu) {
827+ if (prog->sleepable) {
798828 __bpf_prog_test_run_raw_tp(&info);
799− } else if (cpu >= nr_cpu_ids || !cpu_online(cpu)) {
800− /* smp_call_function_single() also checks cpu_online()
801− * after csd_lock(). However, since cpu is from user
802− * space, let's do an extra quick check to filter out
803− * invalid value before smp_call_function_single().
804− */
805− err = -ENXIO;
806829 } else {
807− err = smp_call_function_single(cpu, __bpf_prog_test_run_raw_tp,
808− &info, 1);
830+ current_cpu = get_cpu();
831+ if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 ||
832+ cpu == current_cpu) {
833+ __bpf_prog_test_run_raw_tp(&info);
834+ } else if (cpu >= nr_cpu_ids || !cpu_online(cpu)) {
835+ /*
836+ * smp_call_function_single() also checks cpu_online()
837+ * after csd_lock(). However, since cpu is from user
838+ * space, let's do an extra quick check to filter out
839+ * invalid value before smp_call_function_single().
840+ */
841+ err = -ENXIO;
842+ } else {
843+ err = smp_call_function_single(cpu,
844+ __bpf_prog_test_run_raw_tp,
845+ &info, 1);
846+ }
847+ put_cpu();
809848 }
810− put_cpu();
811849
812850 if (!err &&
813851 copy_to_user(&uattr->test.retval, &info.retval, sizeof(u32)))
net/core/filter.c+86−31
@@ -2654,6 +2654,37 @@ static void sk_msg_reset_curr(struct sk_msg *msg) · 2653 unmodified lines
26542654 }
26552655 }
26562656
2657+static bool sk_msg_elem_is_copy(const struct sk_msg *msg, u32 i)
2658+{
2659+ return test_bit(i, msg->sg.copy);
2660+}
2661+
2662+static void sk_msg_clear_elem_copy(struct sk_msg *msg, u32 i)
2663+{
2664+ __clear_bit(i, msg->sg.copy);
2665+}
2666+
2667+static void sk_msg_set_elem_copy(struct sk_msg *msg, u32 i, bool sg_copy)
2668+{
2669+ __assign_bit(i, msg->sg.copy, sg_copy);
2670+}
2671+
2672+static void sk_msg_clear_copy_range(struct sk_msg *msg, u32 start, u32 end)
2673+{
2674+ while (start != end) {
2675+ sk_msg_clear_elem_copy(msg, start);
2676+ sk_msg_iter_var_next(start);
2677+ }
2678+}
2679+
2680+static void sk_msg_sg_move(struct sk_msg *msg, u32 dst, u32 src)
2681+{
2682+ msg->sg.data[dst] = msg->sg.data[src];
2683+
2684+ sk_msg_set_elem_copy(msg, dst,
2685+ sk_msg_elem_is_copy(msg, src));
2686+}
2687+
26572688 static const struct bpf_func_proto bpf_msg_cork_bytes_proto = {
26582689 .func = bpf_msg_cork_bytes,
26592690 .gpl_only = false,
@@ -2692,7 +2723,7 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 32 unmodified lines
26922723 * account for the headroom.
26932724 */
26942725 bytes_sg_total = start - offset + bytes;
2695− if (!test_bit(i, msg->sg.copy) && bytes_sg_total <= len)
2726+ if (!sk_msg_elem_is_copy(msg, i) && bytes_sg_total <= len)
26962727 goto out;
26972728
26982729 /* At this point we need to linearize multiple scatterlist
@@ -2733,13 +2764,13 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 34 unmodified lines
27332764 poffset += len;
27342765 sge->length = 0;
27352766 put_page(sg_page(sge));
2736− __clear_bit(i, msg->sg.copy);
2767+ sk_msg_clear_elem_copy(msg, i);
27372768
27382769 sk_msg_iter_var_next(i);
27392770 } while (i != last_sge);
27402771
27412772 sg_set_page(&msg->sg.data[first_sge], page, copy, 0);
2742− __clear_bit(first_sge, msg->sg.copy);
2773+ sk_msg_clear_elem_copy(msg, first_sge);
27432774
27442775 /* To repair sg ring we need to shift entries. If we only
27452776 * had a single entry though we can just replace it and
@@ -2749,8 +2780,14 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 3 unmodified lines
27492780 shift = last_sge > first_sge ?
27502781 last_sge - first_sge - 1 :
27512782 NR_MSG_FRAG_IDS - first_sge + last_sge - 1;
2752− if (!shift)
2783+ if (!shift) {
2784+ sk_msg_clear_elem_copy(msg, msg->sg.end);
27532785 goto out;
2786+ }
2787+
2788+ i = first_sge;
2789+ sk_msg_iter_var_next(i);
2790+ sk_msg_clear_copy_range(msg, i, last_sge);
27542791
27552792 i = first_sge;
27562793 sk_msg_iter_var_next(i);
@@ -2764,18 +2801,18 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 7 unmodified lines
27642801 if (move_from == msg->sg.end)
27652802 break;
27662803
2767− msg->sg.data[i] = msg->sg.data[move_from];
2768− sk_msg_sg_copy_assign(msg, i, msg, move_from);
2804+ sk_msg_sg_move(msg, i, move_from);
27692805 msg->sg.data[move_from].length = 0;
27702806 msg->sg.data[move_from].page_link = 0;
27712807 msg->sg.data[move_from].offset = 0;
2772− __clear_bit(move_from, msg->sg.copy);
2808+ sk_msg_clear_elem_copy(msg, move_from);
27732809 sk_msg_iter_var_next(i);
27742810 } while (1);
27752811
27762812 msg->sg.end = msg->sg.end - shift > msg->sg.end ?
27772813 msg->sg.end - shift + NR_MSG_FRAG_IDS :
27782814 msg->sg.end - shift;
2815+ sk_msg_clear_elem_copy(msg, msg->sg.end);
27792816 out:
27802817 sk_msg_reset_curr(msg);
27812818 msg->data = sg_virt(&msg->sg.data[first_sge]) + start - offset;
@@ -2796,9 +2833,10 @@ static const struct bpf_func_proto bpf_msg_pull_data_proto = { · 14 unmodified lines
27962833 BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start,
27972834 u32, len, u64, flags)
27982835 {
2836+ bool sge_copy = false, nsge_copy = false, nnsge_copy = false;
27992837 struct scatterlist sge, nsge, nnsge, rsge = {0}, *psge;
28002838 u32 new, i = 0, l = 0, space, copy = 0, offset = 0;
2801− bool sge_copy, nsge_copy, nnsge_copy, rsge_copy = false;
2839+ bool rsge_copy = false;
28022840 u8 *raw, *to, *from;
28032841 struct page *page;
28042842
@@ -2834,6 +2872,9 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 29 unmodified lines
28342872 if (!space || (space == 1 && start != offset))
28352873 copy = msg->sg.data[i].length;
28362874
2875+ if (unlikely(copy + len < copy))
2876+ return -EINVAL;
2877+
28372878 page = alloc_pages(__GFP_NOWARN | GFP_ATOMIC | __GFP_COMP,
28382879 get_order(copy + len));
28392880 if (unlikely(!page))
@@ -2871,7 +2912,7 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 31 unmodified lines
28712912 sk_msg_iter_var_prev(i);
28722913 psge = sk_msg_elem(msg, i);
28732914 rsge = sk_msg_elem_cpy(msg, i);
2874− rsge_copy = test_bit(i, msg->sg.copy);
2915+ rsge_copy = sk_msg_elem_is_copy(msg, i);
28752916
28762917 psge->length = start - offset;
28772918 rsge.length -= psge->length;
@@ -2896,21 +2937,21 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 18 unmodified lines
28962937
28972938 /* Shift one or two slots as needed */
28982939 sge = sk_msg_elem_cpy(msg, new);
2899− sge_copy = test_bit(new, msg->sg.copy);
29002940 sg_unmark_end(&sge);
2941+ sge_copy = sk_msg_elem_is_copy(msg, new);
29012942
29022943 nsge = sk_msg_elem_cpy(msg, i);
2903− nsge_copy = test_bit(i, msg->sg.copy);
2944+ nsge_copy = sk_msg_elem_is_copy(msg, i);
29042945 if (rsge.length) {
29052946 sk_msg_iter_var_next(i);
29062947 nnsge = sk_msg_elem_cpy(msg, i);
2907− nnsge_copy = test_bit(i, msg->sg.copy);
2948+ nnsge_copy = sk_msg_elem_is_copy(msg, i);
29082949 sk_msg_iter_next(msg, end);
29092950 }
29102951
29112952 while (i != msg->sg.end) {
29122953 msg->sg.data[i] = sge;
2913− __assign_bit(i, msg->sg.copy, sge_copy);
2954+ sk_msg_set_elem_copy(msg, i, sge_copy);
29142955 sge = nsge;
29152956 sge_copy = nsge_copy;
29162957 sk_msg_iter_var_next(i);
@@ -2918,10 +2959,10 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 1 unmodified line
29182959 nsge = nnsge;
29192960 nsge_copy = nnsge_copy;
29202961 nnsge = sk_msg_elem_cpy(msg, i);
2921− nnsge_copy = test_bit(i, msg->sg.copy);
2962+ nnsge_copy = sk_msg_elem_is_copy(msg, i);
29222963 } else {
29232964 nsge = sk_msg_elem_cpy(msg, i);
2924− nsge_copy = test_bit(i, msg->sg.copy);
2965+ nsge_copy = sk_msg_elem_is_copy(msg, i);
29252966 }
29262967 }
29272968
@@ -2929,14 +2970,15 @@ place_new: · 1 unmodified line
29292970 /* Place newly allocated data buffer */
29302971 sk_mem_charge(msg->sk, len);
29312972 msg->sg.size += len;
2932− __clear_bit(new, msg->sg.copy);
2973+ sk_msg_clear_elem_copy(msg, new);
29332974 sg_set_page(&msg->sg.data[new], page, len + copy, 0);
29342975 if (rsge.length) {
29352976 get_page(sg_page(&rsge));
29362977 sk_msg_iter_var_next(new);
29372978 msg->sg.data[new] = rsge;
2938− __assign_bit(new, msg->sg.copy, rsge_copy);
2979+ sk_msg_set_elem_copy(msg, new, rsge_copy);
29392980 }
2981+ sk_msg_clear_elem_copy(msg, msg->sg.end);
29402982
29412983 sk_msg_reset_curr(msg);
29422984 sk_msg_compute_data_pointers(msg);
@@ -2962,12 +3004,11 @@ static void sk_msg_shift_left(struct sk_msg *msg, int i) · 19 unmodified lines
29623004 do {
29633005 prev = i;
29643006 sk_msg_iter_var_next(i);
2965− msg->sg.data[prev] = msg->sg.data[i];
2966− sk_msg_sg_copy_assign(msg, prev, msg, i);
3007+ sk_msg_sg_move(msg, prev, i);
29673008 } while (i != msg->sg.end);
29683009
29693010 sk_msg_iter_prev(msg, end);
2970− __clear_bit(msg->sg.end, msg->sg.copy);
3011+ sk_msg_clear_elem_copy(msg, msg->sg.end);
29713012 }
29723013
29733014 static void sk_msg_shift_right(struct sk_msg *msg, int i)
@@ -2977,28 +3018,29 @@ static void sk_msg_shift_right(struct sk_msg *msg, int i) · 3 unmodified lines
29773018
29783019 sk_msg_iter_next(msg, end);
29793020 sge = sk_msg_elem_cpy(msg, i);
2980− sge_copy = test_bit(i, msg->sg.copy);
3021+ sge_copy = sk_msg_elem_is_copy(msg, i);
29813022 sk_msg_iter_var_next(i);
29823023 tmp = sk_msg_elem_cpy(msg, i);
2983− tmp_copy = test_bit(i, msg->sg.copy);
3024+ tmp_copy = sk_msg_elem_is_copy(msg, i);
29843025
29853026 while (i != msg->sg.end) {
29863027 msg->sg.data[i] = sge;
2987− __assign_bit(i, msg->sg.copy, sge_copy);
3028+ sk_msg_set_elem_copy(msg, i, sge_copy);
29883029 sk_msg_iter_var_next(i);
29893030 sge = tmp;
29903031 sge_copy = tmp_copy;
29913032 tmp = sk_msg_elem_cpy(msg, i);
2992− tmp_copy = test_bit(i, msg->sg.copy);
3033+ tmp_copy = sk_msg_elem_is_copy(msg, i);
29933034 }
3035+ sk_msg_clear_elem_copy(msg, msg->sg.end);
29943036 }
29953037
29963038 BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start,
29973039 u32, len, u64, flags)
29983040 {
29993041 u32 i = 0, l = 0, space, offset = 0;
3000− u64 last = start + len;
3001− int pop;
3042+ u64 last = (u64)start + len;
3043+ u32 pop;
30023044
30033045 if (unlikely(flags))
30043046 return -EINVAL;
@@ -3047,10 +3089,10 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, · 42 unmodified lines
30473089 */
30483090 if (start != offset) {
30493091 struct scatterlist *nsge, *sge = sk_msg_elem(msg, i);
3092+ bool sge_copy = sk_msg_elem_is_copy(msg, i);
30503093 int a = start - offset;
30513094 int b = sge->length - pop - a;
3052− u32 sge_i = i;
3053− bool sge_copy = test_bit(i, msg->sg.copy);
3095+ u32 sge_idx = i;
30543096
30553097 sk_msg_iter_var_next(i);
30563098
@@ -3063,7 +3105,7 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, · 6 unmodified lines
30633105 sg_set_page(nsge,
30643106 sg_page(sge),
30653107 b, sge->offset + pop + a);
3066− __assign_bit(i, msg->sg.copy, sge_copy);
3108+ sk_msg_set_elem_copy(msg, i, sge_copy);
30673109 } else {
30683110 struct page *page, *orig;
30693111 u8 *to, *from;
@@ -3080,7 +3122,7 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, · 10 unmodified lines
30803122 memcpy(to, from, a);
30813123 memcpy(to + a, from + a + pop, b);
30823124 sg_set_page(sge, page, a + b, 0);
3083− __clear_bit(sge_i, msg->sg.copy);
3125+ sk_msg_clear_elem_copy(msg, sge_idx);
30843126 put_page(orig);
30853127 }
30863128 pop = 0;
@@ -5571,11 +5613,24 @@ static int sol_tcp_sockopt(struct sock *sk, int optname, · 2484 unmodified lines
55715613 KERNEL_SOCKPTR(optval), *optlen);
55725614 }
55735615
5616+static bool sk_allows_sol_ip_sockopt(struct sock *sk)
5617+{
5618+ switch (sk->sk_family) {
5619+ case AF_INET:
5620+ return true;
5621+ case AF_INET6:
5622+ /* Allow getting/setting sockopt for possible ipv4-mapped ipv6 socket. */
5623+ return sk->sk_type != SOCK_RAW && !ipv6_only_sock(sk);
5624+ default:
5625+ return false;
5626+ }
5627+}
5628+
55745629 static int sol_ip_sockopt(struct sock *sk, int optname,
55755630 char *optval, int *optlen,
55765631 bool getopt)
55775632 {
5578− if (sk->sk_family != AF_INET)
5633+ if (!sk_allows_sol_ip_sockopt(sk))
55795634 return -EINVAL;
55805635
55815636 switch (optname) {
net/core/lwt_bpf.c+12−0
@@ -599,6 +599,7 @@ static int handle_gso_encap(struct sk_buff *skb, bool ipv4, int encap_len) · 598 unmodified lines
599599
600600 int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress)
601601 {
602+ bool is_udp_tunnel;
602603 struct iphdr *iph;
603604 bool ipv4;
604605 int err;
@@ -612,10 +613,16 @@ int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress) · 7 unmodified lines
612613 ipv4 = true;
613614 if (unlikely(len < iph->ihl * 4))
614615 return -EINVAL;
616+ is_udp_tunnel = iph->protocol == IPPROTO_UDP;
617+ if (unlikely(is_udp_tunnel && len < iph->ihl * 4 + sizeof(struct udphdr)))
618+ return -EINVAL;
615619 } else if (iph->version == 6) {
616620 ipv4 = false;
617621 if (unlikely(len < sizeof(struct ipv6hdr)))
618622 return -EINVAL;
623+ is_udp_tunnel = ((struct ipv6hdr *)iph)->nexthdr == NEXTHDR_UDP;
624+ if (unlikely(is_udp_tunnel && len < sizeof(struct ipv6hdr) + sizeof(struct udphdr)))
625+ return -EINVAL;
619626 } else {
620627 return -EINVAL;
621628 }
@@ -637,6 +644,11 @@ int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress) · 15 unmodified lines
637644 if (ingress)
638645 skb_postpush_rcsum(skb, iph, len);
639646 skb_reset_network_header(skb);
647+ if (is_udp_tunnel) {
648+ size_t iph_sz = ipv4 ? iph->ihl * 4 : sizeof(struct ipv6hdr);
649+
650+ skb_set_transport_header(skb, skb_network_offset(skb) + iph_sz);
651+ }
640652 memcpy(skb_network_header(skb), hdr, len);
641653 bpf_compute_data_pointers(skb);
642654 skb_clear_hash(skb);
net/ipv4/udp_bpf.c+9−0
@@ -50,7 +50,9 @@ static int udp_msg_wait_data(struct sock *sk, struct sk_psock *psock, · 49 unmodified lines
5050 sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
5151 ret = udp_msg_has_data(sk, psock);
5252 if (!ret) {
53+ release_sock(sk);
5354 wait_woken(&wait, TASK_INTERRUPTIBLE, timeo);
55+ lock_sock(sk);
5456 ret = udp_msg_has_data(sk, psock);
5557 }
5658 sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
@@ -79,6 +81,7 @@ static int udp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, · 22 unmodified lines
7981 goto out;
8082 }
8183
84+ lock_sock(sk);
8285 msg_bytes_ready:
8386 copied = sk_msg_recvmsg(sk, psock, msg, len, flags);
8487 if (!copied) {
@@ -90,11 +93,17 @@ msg_bytes_ready: · 5 unmodified lines
9093 if (data) {
9194 if (psock_has_data(psock))
9295 goto msg_bytes_ready;
96+
97+ release_sock(sk);
98+
9399 ret = sk_udp_recvmsg(sk, msg, len, flags);
94100 goto out;
95101 }
96102 copied = -EAGAIN;
97103 }
104+
105+ release_sock(sk);
106+
98107 ret = copied;
99108 out:
100109 sk_psock_put(sk, psock);
tools/bpf/Makefile+1−0
@@ -11,6 +11,7 @@ INSTALL ?= install · 10 unmodified lines
1111 CFLAGS += -Wall -O2
1212 CFLAGS += -D__EXPORTED_HEADERS__ -I$(srctree)/tools/include/uapi \
1313 -I$(srctree)/tools/include
14+CFLAGS += $(EXTRA_CFLAGS)
1415
1516 # This will work when bpf is built in tools env. where srctree
1617 # isn't set and when invoked from selftests build, where srctree
tools/bpf/bpftool/Documentation/bpftool-map.rst+1−1
@@ -56,7 +56,7 @@ MAP COMMANDS · 55 unmodified lines
5656 | | **cgroup_storage** | **reuseport_sockarray** | **percpu_cgroup_storage**
5757 | | **queue** | **stack** | **sk_storage** | **struct_ops** | **ringbuf** | **inode_storage**
5858 | | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena**
59−| | **insn_array** }
59+| | **insn_array** | **rhash** }
6060
6161 DESCRIPTION
6262 ===========
tools/bpf/bpftool/Makefile+17−3
@@ -47,7 +47,8 @@ $(LIBBPF_INTERNAL_HDRS): $(LIBBPF_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_HDRS_ · 46 unmodified lines
4747 $(LIBBPF_BOOTSTRAP): $(wildcard $(BPF_DIR)/*.[ch] $(BPF_DIR)/Makefile) | $(LIBBPF_BOOTSTRAP_OUTPUT)
4848 $(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_BOOTSTRAP_OUTPUT) \
4949 DESTDIR=$(LIBBPF_BOOTSTRAP_DESTDIR:/=) prefix= \
50− ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD="$(HOSTLD)" AR="$(HOSTAR)" $@ install_headers
50+ ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD="$(HOSTLD)" AR="$(HOSTAR)" \
51+ CFLAGS="$(LIBBPF_BOOTSTRAP_CFLAGS)" EXTRA_CFLAGS= $@ install_headers
5152
5253 $(LIBBPF_BOOTSTRAP_INTERNAL_HDRS): $(LIBBPF_BOOTSTRAP_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_BOOTSTRAP_HDRS_DIR)
5354 $(call QUIET_INSTALL, $@)
@@ -81,6 +82,13 @@ CFLAGS += -DPACKAGE='"bpftool"' -D__EXPORTED_HEADERS__ \ · 27 unmodified lines
8182 ifneq ($(BPFTOOL_VERSION),)
8283 CFLAGS += -DBPFTOOL_VERSION='"$(BPFTOOL_VERSION)"'
8384 endif
85+
86+# This must be done before appending EXTRA_CFLAGS to CFLAGS to avoid
87+# including flags that are not applicable to the host compiler.
88+HOST_CFLAGS := $(subst -I$(LIBBPF_INCLUDE),-I$(LIBBPF_BOOTSTRAP_INCLUDE),\
89+ $(subst $(CLANG_CROSS_FLAGS),,$(CFLAGS)))
90+HOST_CFLAGS += $(HOST_EXTRACFLAGS)
91+
8492 ifneq ($(EXTRA_CFLAGS),)
8593 CFLAGS += $(EXTRA_CFLAGS)
8694 endif
@@ -88,10 +96,11 @@ ifneq ($(EXTRA_LDFLAGS),) · 1 unmodified line
8896 LDFLAGS += $(EXTRA_LDFLAGS)
8997 endif
9098
91−HOST_CFLAGS := $(subst -I$(LIBBPF_INCLUDE),-I$(LIBBPF_BOOTSTRAP_INCLUDE),\
92− $(subst $(CLANG_CROSS_FLAGS),,$(CFLAGS)))
9399 HOST_LDFLAGS := $(LDFLAGS)
94100
101+# Remove warnings for libbpf bootstrap build
102+LIBBPF_BOOTSTRAP_CFLAGS := $(filter-out -W -Wall -Wextra -Wformat -Wformat-signedness,$(HOST_CFLAGS))
103+
95104 INSTALL ?= install
96105 RM ?= rm -f
97106
@@ -106,6 +115,10 @@ ifneq ($(SKIP_CRYPTO),1) · 8 unmodified lines
106115 CRYPTO_LIBS := -lcrypto
107116 endif
108117
118+ifeq ($(MAKECMDGOALS),bootstrap)
119+FEATURE_TESTS := libelf-zstd
120+FEATURE_DISPLAY :=
121+else
109122 FEATURE_TESTS := clang-bpf-co-re
110123 FEATURE_TESTS += llvm
111124 FEATURE_TESTS += libcap
@@ -122,6 +135,7 @@ FEATURE_DISPLAY += libcap · 10 unmodified lines
122135 FEATURE_DISPLAY += libbfd
123136 FEATURE_DISPLAY += libbfd-liberty
124137 FEATURE_DISPLAY += libbfd-liberty-z
138+endif
125139
126140 check_feat := 1
127141 NON_CHECK_FEAT_TARGETS := clean uninstall doc doc-clean doc-install doc-uninstall
tools/bpf/bpftool/btf.c+6−11
@@ -179,8 +179,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 178 unmodified lines
179179 case BTF_KIND_STRUCT:
180180 case BTF_KIND_UNION: {
181181 const struct btf_member *m = (const void *)(t + 1);
182− __u16 vlen = BTF_INFO_VLEN(t->info);
183− int i;
182+ __u32 i, vlen = BTF_INFO_VLEN(t->info);
184183
185184 if (json_output) {
186185 jsonw_uint_field(w, "size", t->size);
@@ -225,9 +224,8 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 38 unmodified lines
225224 }
226225 case BTF_KIND_ENUM: {
227226 const struct btf_enum *v = (const void *)(t + 1);
228− __u16 vlen = BTF_INFO_VLEN(t->info);
227+ __u32 i, vlen = BTF_INFO_VLEN(t->info);
229228 const char *encoding;
230− int i;
231229
232230 encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED";
233231 if (json_output) {
@@ -263,9 +261,8 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 29 unmodified lines
263261 }
264262 case BTF_KIND_ENUM64: {
265263 const struct btf_enum64 *v = btf_enum64(t);
266− __u16 vlen = btf_vlen(t);
264+ __u32 i, vlen = btf_vlen(t);
267265 const char *encoding;
268− int i;
269266
270267 encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED";
271268 if (json_output) {
@@ -325,8 +322,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 53 unmodified lines
325322 }
326323 case BTF_KIND_FUNC_PROTO: {
327324 const struct btf_param *p = (const void *)(t + 1);
328− __u16 vlen = BTF_INFO_VLEN(t->info);
329− int i;
325+ __u32 i, vlen = BTF_INFO_VLEN(t->info);
330326
331327 if (json_output) {
332328 jsonw_uint_field(w, "ret_type_id", t->type);
@@ -369,8 +365,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 36 unmodified lines
369365 case BTF_KIND_DATASEC: {
370366 const struct btf_var_secinfo *v = (const void *)(t + 1);
371367 const struct btf_type *vt;
372− __u16 vlen = BTF_INFO_VLEN(t->info);
373− int i;
368+ __u32 i, vlen = BTF_INFO_VLEN(t->info);
374369
375370 if (json_output) {
376371 jsonw_uint_field(w, "size", t->size);
@@ -675,7 +670,7 @@ static __u64 btf_name_hasher(__u64 hash, const struct btf *btf, __u32 name_off) · 298 unmodified lines
675670 static __u64 btf_type_disambig_hash(const struct btf *btf, __u32 id, bool include_members)
676671 {
677672 const struct btf_type *t = btf__type_by_id(btf, id);
678− int i;
673+ __u32 i;
679674 size_t hash = 0;
680675
681676 hash = btf_name_hasher(hash, btf, t->name_off);
tools/bpf/bpftool/btf_dumper.c+2−2
@@ -150,7 +150,7 @@ static int btf_dumper_enum(const struct btf_dumper *d, · 149 unmodified lines
150150 {
151151 const struct btf_enum *enums = btf_enum(t);
152152 __s64 value;
153− __u16 i;
153+ __u32 i;
154154
155155 switch (t->size) {
156156 case 8:
@@ -189,7 +189,7 @@ static int btf_dumper_enum64(const struct btf_dumper *d, · 32 unmodified lines
189189 const struct btf_enum64 *enums = btf_enum64(t);
190190 __u32 val_lo32, val_hi32;
191191 __u64 value;
192− __u16 i;
192+ __u32 i;
193193
194194 value = *(__u64 *)data;
195195 val_lo32 = (__u32)value;
tools/bpf/bpftool/gen.c+10−8
@@ -1399,7 +1399,7 @@ static int do_skeleton(int argc, char **argv) · 1398 unmodified lines
13991399 continue;
14001400
14011401 if (use_loader)
1402− printf("t\tint %s_fd;\n", ident);
1402+ printf("\t\tint %s_fd;\n", ident);
14031403 else
14041404 printf("\t\tstruct bpf_link *%s;\n", ident);
14051405 }
@@ -2094,7 +2094,8 @@ btfgen_mark_type(struct btfgen_info *info, unsigned int type_id, bool follow_poi · 688 unmodified lines
20942094 struct btf_type *cloned_type;
20952095 struct btf_param *param;
20962096 struct btf_array *array;
2097− int err, i;
2097+ __u32 i;
2098+ int err;
20982099
20992100 if (type_id == 0)
21002101 return 0;
@@ -2229,7 +2230,8 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool · 128 unmodified lines
22292230 const struct btf_type *btf_type;
22302231 struct btf *btf = info->src_btf;
22312232 struct btf_type *cloned_type;
2232− int i, err;
2233+ int err;
2234+ __u32 i;
22332235
22342236 if (type_id == 0)
22352237 return 0;
@@ -2249,7 +2251,7 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool · 13 unmodified lines
22492251 case BTF_KIND_STRUCT:
22502252 case BTF_KIND_UNION: {
22512253 struct btf_member *m = btf_members(btf_type);
2252− __u16 vlen = btf_vlen(btf_type);
2254+ __u32 vlen = btf_vlen(btf_type);
22532255
22542256 if (behind_ptr)
22552257 break;
@@ -2286,7 +2288,7 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool · 30 unmodified lines
22862288 break;
22872289 }
22882290 case BTF_KIND_FUNC_PROTO: {
2289− __u16 vlen = btf_vlen(btf_type);
2291+ __u32 vlen = btf_vlen(btf_type);
22902292 struct btf_param *param;
22912293
22922294 /* mark ret type */
@@ -2492,8 +2494,9 @@ static struct btf *btfgen_get_btf(struct btfgen_info *info) · 199 unmodified lines
24922494 {
24932495 struct btf *btf_new = NULL;
24942496 unsigned int *ids = NULL;
2495− unsigned int i, n = btf__type_cnt(info->marked_btf);
2497+ unsigned int n = btf__type_cnt(info->marked_btf);
24962498 int err = 0;
2499+ __u32 i;
24972500
24982501 btf_new = btf__new_empty();
24992502 if (!btf_new) {
@@ -2523,8 +2526,7 @@ static struct btf *btfgen_get_btf(struct btfgen_info *info) · 23 unmodified lines
25232526 /* add members for struct and union */
25242527 if (btf_is_composite(type)) {
25252528 struct btf_member *cloned_m, *m;
2526− unsigned short vlen;
2527− int idx_src;
2529+ __u32 vlen, idx_src;
25282530
25292531 name = btf__str_by_offset(info->src_btf, type->name_off);
25302532
tools/bpf/bpftool/map.c+1−1
@@ -1478,7 +1478,7 @@ static int do_help(int argc, char **argv) · 1477 unmodified lines
14781478 " cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n"
14791479 " queue | stack | sk_storage | struct_ops | ringbuf | inode_storage |\n"
14801480 " task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena |\n"
1481− " insn_array }\n"
1481+ " insn_array | rhash }\n"
14821482 " " HELP_SPEC_OPTIONS " |\n"
14831483 " {-f|--bpffs} | {-n|--nomount} }\n"
14841484 "",
tools/bpf/bpftool/net.c+2−2
@@ -603,14 +603,14 @@ static int query_flow_dissector(struct bpf_attach_info *attach_info) · 602 unmodified lines
603603 &attach_flags, prog_ids, &prog_cnt);
604604 close(fd);
605605 if (err) {
606− if (errno == EINVAL) {
606+ if (err == -EINVAL) {
607607 /* Older kernel's don't support querying
608608 * flow dissector programs.
609609 */
610610 errno = 0;
611611 return 0;
612612 }
613− p_err("can't query prog: %s", strerror(errno));
613+ p_err("can't query prog: %s", strerror(-err));
614614 return -1;
615615 }
616616
tools/build/feature/test-bpf.c+2−0
@@ -20,6 +20,8 @@ · 19 unmodified lines
2020 # define __NR_bpf 6319
2121 # elif defined(__mips__) && defined(_ABI64)
2222 # define __NR_bpf 5315
23+# elif defined(__loongarch__)
24+# define __NR_bpf 280
2325 # else
2426 # error __NR_bpf not defined. libbpf does not support your arch.
2527 # endif
tools/include/uapi/linux/bpf.h+32−1
@@ -994,6 +994,7 @@ enum bpf_cmd { · 993 unmodified lines
994994 BPF_PROG_STREAM_READ_BY_FD,
995995 BPF_PROG_ASSOC_STRUCT_OPS,
996996 __MAX_BPF_CMD,
997+ BPF_COMMON_ATTRS = 1 << 16, /* Indicate carrying syscall common attrs. */
997998 };
998999
9991000 enum bpf_map_type {
@@ -1046,6 +1047,7 @@ enum bpf_map_type { · 46 unmodified lines
10461047 BPF_MAP_TYPE_CGRP_STORAGE,
10471048 BPF_MAP_TYPE_ARENA,
10481049 BPF_MAP_TYPE_INSN_ARRAY,
1050+ BPF_MAP_TYPE_RHASH,
10491051 __MAX_BPF_MAP_TYPE
10501052 };
10511053
@@ -1154,6 +1156,9 @@ enum bpf_attach_type { · 102 unmodified lines
11541156 BPF_TRACE_KPROBE_SESSION,
11551157 BPF_TRACE_UPROBE_SESSION,
11561158 BPF_TRACE_FSESSION,
1159+ BPF_TRACE_FENTRY_MULTI,
1160+ BPF_TRACE_FEXIT_MULTI,
1161+ BPF_TRACE_FSESSION_MULTI,
11571162 __MAX_BPF_ATTACH_TYPE
11581163 };
11591164
@@ -1178,6 +1183,7 @@ enum bpf_link_type { · 18 unmodified lines
11781183 BPF_LINK_TYPE_UPROBE_MULTI = 12,
11791184 BPF_LINK_TYPE_NETKIT = 13,
11801185 BPF_LINK_TYPE_SOCKMAP = 14,
1186+ BPF_LINK_TYPE_TRACING_MULTI = 15,
11811187 __MAX_BPF_LINK_TYPE,
11821188 };
11831189
@@ -1321,7 +1327,11 @@ enum { · 137 unmodified lines
13211327 * BPF_TRACE_UPROBE_MULTI attach type to create return probe.
13221328 */
13231329 enum {
1324− BPF_F_UPROBE_MULTI_RETURN = (1U << 0)
1330+ /* Get return uprobe. */
1331+ BPF_F_UPROBE_MULTI_RETURN = (1U << 0),
1332+
1333+ /* Get path from provided path_fd. */
1334+ BPF_F_UPROBE_MULTI_PATH_FD = (1U << 1),
13251335 };
13261336
13271337 /* link_create.netfilter.flags used in LINK_CREATE command for
@@ -1500,6 +1510,13 @@ struct bpf_stack_build_id { · 172 unmodified lines
15001510 };
15011511 };
15021512
1513+struct bpf_common_attr {
1514+ __aligned_u64 log_buf;
1515+ __u32 log_size;
1516+ __u32 log_level;
1517+ __u32 log_true_size;
1518+};
1519+
15031520 #define BPF_OBJ_NAME_LEN 16U
15041521
15051522 enum {
@@ -1537,6 +1554,11 @@ union bpf_attr { · 31 unmodified lines
15371554 *
15381555 * BPF_MAP_TYPE_ARENA - contains the address where user space
15391556 * is going to mmap() the arena. It has to be page aligned.
1557+ *
1558+ * BPF_MAP_TYPE_RHASH - initial table size hint
1559+ * (nelem_hint). 0 = use rhashtable default. Must be
1560+ * <= min(max_entries, U16_MAX). Upper 32 bits reserved,
1561+ * must be zero.
15401562 */
15411563 __u64 map_extra;
15421564
@@ -1846,6 +1868,7 @@ union bpf_attr { · 303 unmodified lines
18461868 __u32 cnt;
18471869 __u32 flags;
18481870 __u32 pid;
1871+ __u32 path_fd;
18491872 } uprobe_multi;
18501873 struct {
18511874 union {
@@ -1861,6 +1884,11 @@ union bpf_attr { · 9 unmodified lines
18611884 };
18621885 __u64 expected_revision;
18631886 } cgroup;
1887+ struct {
1888+ __aligned_u64 ids;
1889+ __aligned_u64 cookies;
1890+ __u32 cnt;
1891+ } tracing_multi;
18641892 };
18651893 } link_create;
18661894
@@ -6698,6 +6726,7 @@ struct bpf_prog_info { · 4831 unmodified lines
66986726 __u32 verified_insns;
66996727 __u32 attach_btf_obj_id;
67006728 __u32 attach_btf_id;
6729+ __u32 :32;
67016730 } __attribute__((aligned(8)));
67026731
67036732 struct bpf_map_info {
@@ -6719,6 +6748,7 @@ struct bpf_map_info { · 15 unmodified lines
67196748 __u64 map_extra;
67206749 __aligned_u64 hash;
67216750 __u32 hash_size;
6751+ __u32 :32;
67226752 } __attribute__((aligned(8)));
67236753
67246754 struct bpf_btf_info {
@@ -7236,6 +7266,7 @@ enum { · 511 unmodified lines
72367266 TCP_BPF_SOCK_OPS_CB_FLAGS = 1008, /* Get or Set TCP sock ops flags */
72377267 SK_BPF_CB_FLAGS = 1009, /* Get or set sock ops flags in socket */
72387268 SK_BPF_BYPASS_PROT_MEM = 1010, /* Get or Set sk->sk_bypass_prot_mem */
7269+
72397270 };
72407271
72417272 enum {
tools/include/uapi/linux/btf.h+14−12
@@ -33,20 +33,22 @@ struct btf_header { · 32 unmodified lines
3333 __u32 layout_len; /* length of layout section */
3434 };
3535
36−/* Max # of type identifier */
37−#define BTF_MAX_TYPE 0x000fffff
38−/* Max offset into the string section */
39−#define BTF_MAX_NAME_OFFSET 0x00ffffff
40−/* Max # of struct/union/enum members or func args */
41−#define BTF_MAX_VLEN 0xffff
36+enum btf_max {
37+ /* Max possible kind */
38+ BTF_MAX_KIND = 0x0000007f,
39+ /* Max # of type identifier */
40+ BTF_MAX_TYPE = 0x000fffff,
41+ /* Max offset into the string section */
42+ BTF_MAX_NAME_OFFSET = 0x00ffffff,
43+ /* Max # of struct/union/enum members or func args */
44+ BTF_MAX_VLEN = 0x00ffffff,
45+};
4246
4347 struct btf_type {
4448 __u32 name_off;
4549 /* "info" bits arrangement
46− * bits 0-15: vlen (e.g. # of struct's members)
47− * bits 16-23: unused
48− * bits 24-28: kind (e.g. int, ptr, array...etc)
49− * bits 29-30: unused
50+ * bits 0-23: vlen (e.g. # of struct's members)
51+ * bits 24-30: kind (e.g. int, ptr, array...etc)
5052 * bit 31: kind_flag, currently used by
5153 * struct, union, enum, fwd, enum64,
5254 * decl_tag and type_tag
@@ -65,8 +67,8 @@ struct btf_type { · 12 unmodified lines
6567 };
6668 };
6769
68−#define BTF_INFO_KIND(info) (((info) >> 24) & 0x1f)
69−#define BTF_INFO_VLEN(info) ((info) & 0xffff)
70+#define BTF_INFO_KIND(info) (((info) >> 24) & 0x7f)
71+#define BTF_INFO_VLEN(info) ((info) & 0xffffff)
7072 #define BTF_INFO_KFLAG(info) ((info) >> 31)
7173
7274 enum {
tools/lib/bpf/Makefile+9−8
@@ -49,6 +49,14 @@ man_dir_SQ = '$(subst ','\'',$(man_dir))' · 48 unmodified lines
4949 export man_dir man_dir_SQ INSTALL
5050 export DESTDIR DESTDIR_SQ
5151
52+# Defer assigning EXTRA_CFLAGS to CFLAGS until after including
53+# tools/scripts/Makefile.include, as it may add flags to EXTRA_CFLAGS.
54+ifdef EXTRA_CFLAGS
55+ CFLAGS :=
56+else
57+ CFLAGS := -g -O2
58+endif
59+
5260 include $(srctree)/tools/scripts/Makefile.include
5361
5462 # copy a bit from Linux kbuild
@@ -70,13 +78,6 @@ LIB_TARGET = libbpf.a libbpf.so.$(LIBBPF_VERSION) · 15 unmodified lines
7078 LIB_FILE = libbpf.a libbpf.so*
7179 PC_FILE = libbpf.pc
7280
73−# Set compile option CFLAGS
74−ifdef EXTRA_CFLAGS
75− CFLAGS := $(EXTRA_CFLAGS)
76−else
77− CFLAGS := -g -O2
78−endif
79−
8081 # Append required CFLAGS
8182 override CFLAGS += -std=gnu89
8283 override CFLAGS += $(EXTRA_WARNINGS) -Wno-switch-enum
@@ -84,7 +85,7 @@ override CFLAGS += -Werror -Wall · 1 unmodified line
8485 override CFLAGS += $(INCLUDES)
8586 override CFLAGS += -fvisibility=hidden
8687 override CFLAGS += -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
87−override CFLAGS += $(CLANG_CROSS_FLAGS)
88+override CFLAGS += $(EXTRA_CFLAGS)
8889
8990 # flags specific for shared library
9091 SHLIB_FLAGS := -DSHARED -fPIC
tools/lib/bpf/bpf.c+66−1
@@ -59,6 +59,8 @@ · 58 unmodified lines
5959 # define __NR_bpf 6319
6060 # elif defined(__mips__) && defined(_ABI64)
6161 # define __NR_bpf 5315
62+# elif defined(__loongarch__)
63+# define __NR_bpf 280
6264 # else
6365 # error __NR_bpf not defined. libbpf does not support your arch.
6466 # endif
@@ -69,6 +71,42 @@ static inline __u64 ptr_to_u64(const void *ptr) · 4 unmodified lines
6971 return (__u64) (unsigned long) ptr;
7072 }
7173
74+static inline int sys_bpf_ext(enum bpf_cmd cmd, union bpf_attr *attr,
75+ unsigned int size,
76+ struct bpf_common_attr *attr_common,
77+ unsigned int size_common)
78+{
79+ cmd = attr_common ? (cmd | BPF_COMMON_ATTRS) : (cmd & ~BPF_COMMON_ATTRS);
80+ return syscall(__NR_bpf, cmd, attr, size, attr_common, size_common);
81+}
82+
83+static inline int sys_bpf_ext_fd(enum bpf_cmd cmd, union bpf_attr *attr,
84+ unsigned int size,
85+ struct bpf_common_attr *attr_common,
86+ unsigned int size_common)
87+{
88+ int fd;
89+
90+ fd = sys_bpf_ext(cmd, attr, size, attr_common, size_common);
91+ return ensure_good_fd(fd);
92+}
93+
94+int probe_sys_bpf_ext(void)
95+{
96+ const size_t attr_sz = offsetofend(union bpf_attr, prog_token_fd);
97+ union bpf_attr attr;
98+ int fd;
99+
100+ memset(&attr, 0, attr_sz);
101+ fd = syscall(__NR_bpf, BPF_PROG_LOAD | BPF_COMMON_ATTRS, &attr, attr_sz, NULL,
102+ sizeof(struct bpf_common_attr));
103+ if (fd >= 0) {
104+ close(fd);
105+ return -EINVAL;
106+ }
107+ return errno == EFAULT ? 1 : 0;
108+}
109+
72110 static inline int sys_bpf(enum bpf_cmd cmd, union bpf_attr *attr,
73111 unsigned int size)
74112 {
@@ -173,6 +211,9 @@ int bpf_map_create(enum bpf_map_type map_type, · 98 unmodified lines
173211 const struct bpf_map_create_opts *opts)
174212 {
175213 const size_t attr_sz = offsetofend(union bpf_attr, excl_prog_hash_size);
214+ const size_t attr_common_sz = sizeof(struct bpf_common_attr);
215+ struct bpf_common_attr attr_common;
216+ struct bpf_log_opts *log_opts;
176217 union bpf_attr attr;
177218 int fd;
178219
@@ -206,7 +247,21 @@ int bpf_map_create(enum bpf_map_type map_type, · 27 unmodified lines
206247 attr.excl_prog_hash = ptr_to_u64(OPTS_GET(opts, excl_prog_hash, NULL));
207248 attr.excl_prog_hash_size = OPTS_GET(opts, excl_prog_hash_size, 0);
208249
209− fd = sys_bpf_fd(BPF_MAP_CREATE, &attr, attr_sz);
250+ log_opts = OPTS_GET(opts, log_opts, NULL);
251+ if (!OPTS_VALID(log_opts, bpf_log_opts))
252+ return libbpf_err(-EINVAL);
253+
254+ if (log_opts && feat_supported(NULL, FEAT_BPF_SYSCALL_COMMON_ATTRS)) {
255+ memset(&attr_common, 0, attr_common_sz);
256+ attr_common.log_buf = ptr_to_u64(OPTS_GET(log_opts, buf, NULL));
257+ attr_common.log_size = OPTS_GET(log_opts, size, 0);
258+ attr_common.log_level = OPTS_GET(log_opts, level, 0);
259+ fd = sys_bpf_ext_fd(BPF_MAP_CREATE, &attr, attr_sz, &attr_common, attr_common_sz);
260+ OPTS_SET(log_opts, true_size, attr_common.log_true_size);
261+ } else {
262+ fd = sys_bpf_fd(BPF_MAP_CREATE, &attr, attr_sz);
263+ OPTS_SET(log_opts, true_size, 0);
264+ }
210265 return libbpf_err_errno(fd);
211266 }
212267
@@ -787,9 +842,19 @@ int bpf_link_create(int prog_fd, int target_fd, · 574 unmodified lines
787842 attr.link_create.uprobe_multi.ref_ctr_offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.ref_ctr_offsets, 0));
788843 attr.link_create.uprobe_multi.cookies = ptr_to_u64(OPTS_GET(opts, uprobe_multi.cookies, 0));
789844 attr.link_create.uprobe_multi.pid = OPTS_GET(opts, uprobe_multi.pid, 0);
845+ attr.link_create.uprobe_multi.path_fd = OPTS_GET(opts, uprobe_multi.path_fd, 0);
790846 if (!OPTS_ZEROED(opts, uprobe_multi))
791847 return libbpf_err(-EINVAL);
792848 break;
849+ case BPF_TRACE_FENTRY_MULTI:
850+ case BPF_TRACE_FEXIT_MULTI:
851+ case BPF_TRACE_FSESSION_MULTI:
852+ attr.link_create.tracing_multi.ids = ptr_to_u64(OPTS_GET(opts, tracing_multi.ids, 0));
853+ attr.link_create.tracing_multi.cookies = ptr_to_u64(OPTS_GET(opts, tracing_multi.cookies, 0));
854+ attr.link_create.tracing_multi.cnt = OPTS_GET(opts, tracing_multi.cnt, 0);
855+ if (!OPTS_ZEROED(opts, tracing_multi))
856+ return libbpf_err(-EINVAL);
857+ break;
793858 case BPF_TRACE_RAW_TP:
794859 case BPF_TRACE_FENTRY:
795860 case BPF_TRACE_FEXIT:
tools/lib/bpf/bpf.h+23−2
@@ -37,6 +37,18 @@ extern "C" { · 36 unmodified lines
3737
3838 LIBBPF_API int libbpf_set_memlock_rlim(size_t memlock_bytes);
3939
40+struct bpf_log_opts {
41+ size_t sz; /* size of this struct for forward/backward compatibility */
42+
43+ char *buf;
44+ __u32 size;
45+ __u32 level;
46+ __u32 true_size; /* out parameter set by kernel */
47+
48+ size_t :0;
49+};
50+#define bpf_log_opts__last_field true_size
51+
4052 struct bpf_map_create_opts {
4153 size_t sz; /* size of this struct for forward/backward compatibility */
4254
@@ -57,9 +69,12 @@ struct bpf_map_create_opts { · 14 unmodified lines
5769
5870 const void *excl_prog_hash;
5971 __u32 excl_prog_hash_size;
72+
73+ struct bpf_log_opts *log_opts;
74+
6075 size_t :0;
6176 };
62−#define bpf_map_create_opts__last_field excl_prog_hash_size
77+#define bpf_map_create_opts__last_field log_opts
6378
6479 LIBBPF_API int bpf_map_create(enum bpf_map_type map_type,
6580 const char *map_name,
@@ -429,6 +444,7 @@ struct bpf_link_create_opts { · 363 unmodified lines
429444 const unsigned long *ref_ctr_offsets;
430445 const __u64 *cookies;
431446 __u32 pid;
447+ __u32 path_fd;
432448 } uprobe_multi;
433449 struct {
434450 __u64 cookie;
@@ -454,10 +470,15 @@ struct bpf_link_create_opts { · 19 unmodified lines
454470 __u32 relative_id;
455471 __u64 expected_revision;
456472 } cgroup;
473+ struct {
474+ const __u32 *ids;
475+ const __u64 *cookies;
476+ __u32 cnt;
477+ } tracing_multi;
457478 };
458479 size_t :0;
459480 };
460−#define bpf_link_create_opts__last_field uprobe_multi.pid
481+#define bpf_link_create_opts__last_field uprobe_multi.path_fd
461482
462483 LIBBPF_API int bpf_link_create(int prog_fd, int target_fd,
463484 enum bpf_attach_type attach_type,
tools/lib/bpf/btf.c+35−19
@@ -421,7 +421,7 @@ static int btf_type_size_unknown(const struct btf *btf, const struct btf_type *t · 420 unmodified lines
421421 {
422422 __u32 l_cnt = btf->hdr.layout_len / sizeof(struct btf_layout);
423423 struct btf_layout *l = btf->layout;
424− __u16 vlen = btf_vlen(t);
424+ __u32 vlen = btf_vlen(t);
425425 __u32 kind = btf_kind(t);
426426
427427 /* Fall back to base BTF if needed as they share layout information */
@@ -454,7 +454,7 @@ static int btf_type_size_unknown(const struct btf *btf, const struct btf_type *t · 26 unmodified lines
454454 static int btf_type_size(const struct btf *btf, const struct btf_type *t)
455455 {
456456 const int base_size = sizeof(struct btf_type);
457− __u16 vlen = btf_vlen(t);
457+ __u32 vlen = btf_vlen(t);
458458
459459 switch (btf_kind(t)) {
460460 case BTF_KIND_FWD:
@@ -506,7 +506,7 @@ static int btf_bswap_type_rest(struct btf_type *t) · 45 unmodified lines
506506 struct btf_array *a;
507507 struct btf_param *p;
508508 struct btf_enum *e;
509− __u16 vlen = btf_vlen(t);
509+ __u32 vlen = btf_vlen(t);
510510 int i;
511511
512512 switch (btf_kind(t)) {
@@ -1007,7 +1007,7 @@ int btf__align_of(const struct btf *btf, __u32 id) · 494 unmodified lines
10071007 case BTF_KIND_STRUCT:
10081008 case BTF_KIND_UNION: {
10091009 const struct btf_member *m = btf_members(t);
1010− __u16 vlen = btf_vlen(t);
1010+ __u32 vlen = btf_vlen(t);
10111011 int i, max_align = 1, align;
10121012
10131013 for (i = 0; i < vlen; i++, m++) {
@@ -2121,9 +2121,12 @@ static void *btf_add_type_mem(struct btf *btf, size_t add_sz) · 1107 unmodified lines
21212121 btf->hdr.type_len, UINT_MAX, add_sz);
21222122 }
21232123
2124−static void btf_type_inc_vlen(struct btf_type *t)
2124+static int btf_type_inc_vlen(struct btf_type *t)
21252125 {
2126+ if (btf_vlen(t) == BTF_MAX_VLEN)
2127+ return -ENOSPC;
21262128 t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, btf_kflag(t));
2129+ return 0;
21272130 }
21282131
21292132 static void btf_hdr_update_type_len(struct btf *btf, int new_len)
@@ -2652,6 +2655,8 @@ int btf__add_field(struct btf *btf, const char *name, int type_id, · 522 unmodified lines
26522655 t = btf_last_type(btf);
26532656 if (!btf_is_composite(t))
26542657 return libbpf_err(-EINVAL);
2658+ if (btf_vlen(t) == BTF_MAX_VLEN)
2659+ return libbpf_err(-ENOSPC);
26552660
26562661 if (validate_type_id(type_id))
26572662 return libbpf_err(-EINVAL);
@@ -2686,6 +2691,7 @@ int btf__add_field(struct btf *btf, const char *name, int type_id, · 28 unmodified lines
26862691
26872692 /* btf_add_type_mem can invalidate t pointer */
26882693 t = btf_last_type(btf);
2694+
26892695 /* update parent type's vlen and kflag */
26902696 t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, is_bitfield || btf_kflag(t));
26912697
@@ -2796,7 +2802,9 @@ int btf__add_enum_value(struct btf *btf, const char *name, __s64 value) · 104 unmodified lines
27962802
27972803 /* update parent type's vlen */
27982804 t = btf_last_type(btf);
2799− btf_type_inc_vlen(t);
2805+ err = btf_type_inc_vlen(t);
2806+ if (err)
2807+ return libbpf_err(err);
28002808
28012809 /* if negative value, set signedness to signed */
28022810 if (value < 0)
@@ -2873,7 +2881,9 @@ int btf__add_enum64_value(struct btf *btf, const char *name, __u64 value) · 70 unmodified lines
28732881
28742882 /* update parent type's vlen */
28752883 t = btf_last_type(btf);
2876− btf_type_inc_vlen(t);
2884+ err = btf_type_inc_vlen(t);
2885+ if (err)
2886+ return libbpf_err(err);
28772887
28782888 btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
28792889 return 0;
@@ -3115,7 +3125,9 @@ int btf__add_func_param(struct btf *btf, const char *name, int type_id) · 235 unmodified lines
31153125
31163126 /* update parent type's vlen */
31173127 t = btf_last_type(btf);
3118− btf_type_inc_vlen(t);
3128+ err = btf_type_inc_vlen(t);
3129+ if (err)
3130+ return libbpf_err(err);
31193131
31203132 btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
31213133 return 0;
@@ -3257,7 +3269,9 @@ int btf__add_datasec_var_info(struct btf *btf, int var_type_id, __u32 offset, __ · 135 unmodified lines
32573269
32583270 /* update parent type's vlen */
32593271 t = btf_last_type(btf);
3260− btf_type_inc_vlen(t);
3272+ err = btf_type_inc_vlen(t);
3273+ if (err)
3274+ return libbpf_err(err);
32613275
32623276 btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
32633277 return 0;
@@ -4311,7 +4325,7 @@ static long btf_hash_enum(struct btf_type *t) · 1047 unmodified lines
43114325 static bool btf_equal_enum_members(struct btf_type *t1, struct btf_type *t2)
43124326 {
43134327 const struct btf_enum *m1, *m2;
4314− __u16 vlen;
4328+ __u32 vlen;
43154329 int i;
43164330
43174331 vlen = btf_vlen(t1);
@@ -4329,7 +4343,7 @@ static bool btf_equal_enum_members(struct btf_type *t1, struct btf_type *t2) · 11 unmodified lines
43294343 static bool btf_equal_enum64_members(struct btf_type *t1, struct btf_type *t2)
43304344 {
43314345 const struct btf_enum64 *m1, *m2;
4332− __u16 vlen;
4346+ __u32 vlen;
43334347 int i;
43344348
43354349 vlen = btf_vlen(t1);
@@ -4406,7 +4420,7 @@ static long btf_hash_struct(struct btf_type *t) · 70 unmodified lines
44064420 static bool btf_shallow_equal_struct(struct btf_type *t1, struct btf_type *t2)
44074421 {
44084422 const struct btf_member *m1, *m2;
4409− __u16 vlen;
4423+ __u32 vlen;
44104424 int i;
44114425
44124426 if (!btf_equal_common(t1, t2))
@@ -4482,7 +4496,7 @@ static bool btf_compat_array(struct btf_type *t1, struct btf_type *t2) · 69 unmodified lines
44824496 static long btf_hash_fnproto(struct btf_type *t)
44834497 {
44844498 const struct btf_param *member = btf_params(t);
4485− __u16 vlen = btf_vlen(t);
4499+ __u32 vlen = btf_vlen(t);
44864500 long h = btf_hash_common(t);
44874501 int i;
44884502
@@ -4504,7 +4518,7 @@ static long btf_hash_fnproto(struct btf_type *t) · 15 unmodified lines
45044518 static bool btf_equal_fnproto(struct btf_type *t1, struct btf_type *t2)
45054519 {
45064520 const struct btf_param *m1, *m2;
4507− __u16 vlen;
4521+ __u32 vlen;
45084522 int i;
45094523
45104524 if (!btf_equal_common(t1, t2))
@@ -4530,7 +4544,7 @@ static bool btf_equal_fnproto(struct btf_type *t1, struct btf_type *t2) · 19 unmodified lines
45304544 static bool btf_compat_fnproto(struct btf_type *t1, struct btf_type *t2)
45314545 {
45324546 const struct btf_param *m1, *m2;
4533− __u16 vlen;
4547+ __u32 vlen;
45344548 int i;
45354549
45364550 /* skip return type ID */
@@ -4578,12 +4592,14 @@ static int btf_dedup_prep(struct btf_dedup *d) · 41 unmodified lines
45784592 case BTF_KIND_RESTRICT:
45794593 case BTF_KIND_PTR:
45804594 case BTF_KIND_FWD:
4581− case BTF_KIND_TYPEDEF:
45824595 case BTF_KIND_FUNC:
45834596 case BTF_KIND_FLOAT:
45844597 case BTF_KIND_TYPE_TAG:
45854598 h = btf_hash_common(t);
45864599 break;
4600+ case BTF_KIND_TYPEDEF:
4601+ h = btf_hash_typedef(t);
4602+ break;
45874603 case BTF_KIND_INT:
45884604 case BTF_KIND_DECL_TAG:
45894605 h = btf_hash_int_decl_tag(t);
@@ -5077,7 +5093,7 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id, · 487 unmodified lines
50775093 case BTF_KIND_STRUCT:
50785094 case BTF_KIND_UNION: {
50795095 const struct btf_member *cand_m, *canon_m;
5080− __u16 vlen;
5096+ __u32 vlen;
50815097
50825098 if (!btf_shallow_equal_struct(cand_type, canon_type))
50835099 return 0;
@@ -5105,7 +5121,7 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id, · 21 unmodified lines
51055121
51065122 case BTF_KIND_FUNC_PROTO: {
51075123 const struct btf_param *cand_p, *canon_p;
5108− __u16 vlen;
5124+ __u32 vlen;
51095125
51105126 if (!btf_compat_fnproto(cand_type, canon_type))
51115127 return 0;
@@ -5439,7 +5455,7 @@ static int btf_dedup_ref_type(struct btf_dedup *d, __u32 type_id) · 327 unmodified lines
54395455
54405456 case BTF_KIND_FUNC_PROTO: {
54415457 struct btf_param *param;
5442− __u16 vlen;
5458+ __u32 vlen;
54435459 int i;
54445460
54455461 ref_type_id = btf_dedup_ref_type(d, t->type);
tools/lib/bpf/btf.h+1−1
@@ -435,7 +435,7 @@ static inline __u16 btf_kind(const struct btf_type *t) · 434 unmodified lines
435435 return BTF_INFO_KIND(t->info);
436436 }
437437
438−static inline __u16 btf_vlen(const struct btf_type *t)
438+static inline __u32 btf_vlen(const struct btf_type *t)
439439 {
440440 return BTF_INFO_VLEN(t->info);
441441 }
tools/lib/bpf/btf_dump.c+12−12
@@ -316,7 +316,7 @@ static int btf_dump_mark_referenced(struct btf_dump *d) · 315 unmodified lines
316316 {
317317 int i, j, n = btf__type_cnt(d->btf);
318318 const struct btf_type *t;
319− __u16 vlen;
319+ __u32 vlen;
320320
321321 for (i = d->last_id + 1; i < n; i++) {
322322 t = btf__type_by_id(d->btf, i);
@@ -485,7 +485,7 @@ static int btf_dump_order_type(struct btf_dump *d, __u32 id, bool through_ptr) · 162 unmodified lines
485485 */
486486 struct btf_dump_type_aux_state *tstate = &d->type_states[id];
487487 const struct btf_type *t;
488− __u16 vlen;
488+ __u32 vlen;
489489 int err, i;
490490
491491 /* return true, letting typedefs know that it's ok to be emitted */
@@ -798,7 +798,7 @@ static void btf_dump_emit_type(struct btf_dump *d, __u32 id, __u32 cont_id) · 306 unmodified lines
798798 */
799799 if (top_level_def || t->name_off == 0) {
800800 const struct btf_member *m = btf_members(t);
801− __u16 vlen = btf_vlen(t);
801+ __u32 vlen = btf_vlen(t);
802802 int i, new_cont_id;
803803
804804 new_cont_id = t->name_off == 0 ? cont_id : id;
@@ -820,7 +820,7 @@ static void btf_dump_emit_type(struct btf_dump *d, __u32 id, __u32 cont_id) · 15 unmodified lines
820820 break;
821821 case BTF_KIND_FUNC_PROTO: {
822822 const struct btf_param *p = btf_params(t);
823− __u16 n = btf_vlen(t);
823+ __u32 n = btf_vlen(t);
824824 int i;
825825
826826 btf_dump_emit_type(d, t->type, cont_id);
@@ -839,7 +839,7 @@ static bool btf_is_struct_packed(const struct btf *btf, __u32 id, · 12 unmodified lines
839839 {
840840 const struct btf_member *m;
841841 int max_align = 1, align, i, bit_sz;
842− __u16 vlen;
842+ __u32 vlen;
843843
844844 m = btf_members(t);
845845 vlen = btf_vlen(t);
@@ -973,7 +973,7 @@ static void btf_dump_emit_struct_def(struct btf_dump *d, · 127 unmodified lines
973973 bool is_struct = btf_is_struct(t);
974974 bool packed, prev_bitfield = false;
975975 int align, i, off = 0;
976− __u16 vlen = btf_vlen(t);
976+ __u32 vlen = btf_vlen(t);
977977
978978 align = btf__align_of(d->btf, id);
979979 packed = is_struct ? btf_is_struct_packed(d->btf, id, t) : 0;
@@ -1064,7 +1064,7 @@ static void btf_dump_emit_enum_fwd(struct btf_dump *d, __u32 id, · 84 unmodified lines
10641064
10651065 static void btf_dump_emit_enum32_val(struct btf_dump *d,
10661066 const struct btf_type *t,
1067− int lvl, __u16 vlen)
1067+ int lvl, __u32 vlen)
10681068 {
10691069 const struct btf_enum *v = btf_enum(t);
10701070 bool is_signed = btf_kflag(t);
@@ -1089,7 +1089,7 @@ static void btf_dump_emit_enum32_val(struct btf_dump *d, · 18 unmodified lines
10891089
10901090 static void btf_dump_emit_enum64_val(struct btf_dump *d,
10911091 const struct btf_type *t,
1092− int lvl, __u16 vlen)
1092+ int lvl, __u32 vlen)
10931093 {
10941094 const struct btf_enum64 *v = btf_enum64(t);
10951095 bool is_signed = btf_kflag(t);
@@ -1122,7 +1122,7 @@ static void btf_dump_emit_enum_def(struct btf_dump *d, __u32 id, · 26 unmodified lines
11221122 const struct btf_type *t,
11231123 int lvl)
11241124 {
1125− __u16 vlen = btf_vlen(t);
1125+ __u32 vlen = btf_vlen(t);
11261126
11271127 btf_dump_printf(d, "enum%s%s",
11281128 t->name_off ? " " : "",
@@ -1542,7 +1542,7 @@ static void btf_dump_emit_type_chain(struct btf_dump *d, · 413 unmodified lines
15421542 }
15431543 case BTF_KIND_FUNC_PROTO: {
15441544 const struct btf_param *p = btf_params(t);
1545− __u16 vlen = btf_vlen(t);
1545+ __u32 vlen = btf_vlen(t);
15461546 int i;
15471547
15481548 /*
@@ -2159,7 +2159,7 @@ static int btf_dump_struct_data(struct btf_dump *d, · 610 unmodified lines
21592159 const void *data)
21602160 {
21612161 const struct btf_member *m = btf_members(t);
2162− __u16 n = btf_vlen(t);
2162+ __u32 n = btf_vlen(t);
21632163 int i, err = 0;
21642164
21652165 /* note that we increment depth before calling btf_dump_print() below;
@@ -2449,7 +2449,7 @@ static int btf_dump_type_data_check_zero(struct btf_dump *d, · 283 unmodified lines
24492449 case BTF_KIND_STRUCT:
24502450 case BTF_KIND_UNION: {
24512451 const struct btf_member *m = btf_members(t);
2452− __u16 n = btf_vlen(t);
2452+ __u32 n = btf_vlen(t);
24532453
24542454 /* if any struct/union member is non-zero, the struct/union
24552455 * is considered non-zero and dumped.
tools/lib/bpf/features.c+8−0
@@ -615,6 +615,11 @@ static int probe_kern_btf_layout(int token_fd) · 614 unmodified lines
615615 (char *)layout, token_fd));
616616 }
617617
618+static int probe_bpf_syscall_common_attrs(int token_fd)
619+{
620+ return probe_sys_bpf_ext();
621+}
622+
618623 typedef int (*feature_probe_fn)(int /* token_fd */);
619624
620625 static struct kern_feature_cache feature_cache;
@@ -699,6 +704,9 @@ static struct kern_feature_desc { · 78 unmodified lines
699704 [FEAT_BTF_LAYOUT] = {
700705 "kernel supports BTF layout", probe_kern_btf_layout,
701706 },
707+ [FEAT_BPF_SYSCALL_COMMON_ATTRS] = {
708+ "BPF syscall common attributes support", probe_bpf_syscall_common_attrs,
709+ },
702710 };
703711
704712 bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)
tools/lib/bpf/gen_loader.c+67−25
@@ -63,6 +63,7 @@ static int realloc_insn_buf(struct bpf_gen *gen, __u32 size) · 62 unmodified lines
6363 gen->error = -ENOMEM;
6464 free(gen->insn_start);
6565 gen->insn_start = NULL;
66+ gen->insn_cur = NULL;
6667 return -ENOMEM;
6768 }
6869 gen->insn_start = insn_start;
@@ -86,6 +87,7 @@ static int realloc_data_buf(struct bpf_gen *gen, __u32 size) · 17 unmodified lines
8687 gen->error = -ENOMEM;
8788 free(gen->data_start);
8889 gen->data_start = NULL;
90+ gen->data_cur = NULL;
8991 return -ENOMEM;
9092 }
9193 gen->data_start = data_start;
@@ -158,10 +160,16 @@ void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps · 66 unmodified lines
158160
159161 static int add_data(struct bpf_gen *gen, const void *data, __u32 size)
160162 {
161− __u32 size8 = roundup(size, 8);
162163 __u64 zero = 0;
164+ __u32 size8;
163165 void *prev;
164166
167+ if (size > INT32_MAX) {
168+ gen->error = -ERANGE;
169+ return 0;
170+ }
171+ size8 = roundup(size, 8);
172+
165173 if (realloc_data_buf(gen, size8))
166174 return 0;
167175 prev = gen->data_cur;
@@ -293,7 +301,6 @@ static void emit_check_err(struct bpf_gen *gen) · 125 unmodified lines
293301 emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, off));
294302 } else {
295303 gen->error = -ERANGE;
296− emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, -1));
297304 }
298305 }
299306
@@ -398,13 +405,12 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps) · 98 unmodified lines
398405 blob_fd_array_off(gen, i));
399406 emit(gen, BPF_MOV64_IMM(BPF_REG_0, 0));
400407 emit(gen, BPF_EXIT_INSN());
401− if (OPTS_GET(gen->opts, gen_hash, false))
402− compute_sha_update_offsets(gen);
403−
404− pr_debug("gen: finish %s\n", errstr(gen->error));
405408 if (!gen->error) {
406409 struct gen_loader_opts *opts = gen->opts;
407410
411+ if (OPTS_GET(opts, gen_hash, false))
412+ compute_sha_update_offsets(gen);
413+
408414 opts->insns = gen->insn_start;
409415 opts->insns_sz = gen->insn_cur - gen->insn_start;
410416 opts->data = gen->data_start;
@@ -419,6 +425,7 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps) · 8 unmodified lines
419425 bpf_insn_bswap(insn++);
420426 }
421427 }
428+ pr_debug("gen: finish %s\n", errstr(gen->error));
422429 return gen->error;
423430 }
424431
@@ -545,13 +552,22 @@ void bpf_gen__map_create(struct bpf_gen *gen, · 120 unmodified lines
545552 default:
546553 break;
547554 }
548− /* conditionally update max_entries */
549− if (map_idx >= 0)
555+
556+ /*
557+ * Conditionally update max_entries from the host-supplied loader
558+ * ctx. This sizes the map at runtime, but for a signed loader
559+ * (gen_hash) it would let an untrusted host re-dimension the
560+ * program's maps after emit_signature_match(), outside what the
561+ * signature attests to. Keep the signer-provided max_entries
562+ * baked into the blob in that case.
563+ */
564+ if (map_idx >= 0 && !OPTS_GET(gen->opts, gen_hash, false))
550565 move_ctx2blob(gen, attr_field(map_create_attr, max_entries), 4,
551566 sizeof(struct bpf_loader_ctx) +
552567 sizeof(struct bpf_map_desc) * map_idx +
553568 offsetof(struct bpf_map_desc, max_entries),
554569 true /* check that max_entries != 0 */);
570+
555571 /* emit MAP_CREATE command */
556572 emit_sys_bpf(gen, BPF_MAP_CREATE, map_create_attr, attr_size);
557573 debug_ret(gen, "map_create %s idx %d type %d value_size %d value_btf_id %d",
@@ -585,6 +601,23 @@ static void emit_signature_match(struct bpf_gen *gen) · 27 unmodified lines
585601 __s64 off;
586602 int i;
587603
604+ /*
605+ * Reject if the metadata map is not exclusive. Without exclusivity
606+ * the cached map->sha[] verified above can be stale: another BPF
607+ * program with map access could have mutated the contents between
608+ * BPF_OBJ_GET_INFO_BY_FD and loader execution.
609+ */
610+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX,
611+ 0, 0, 0, 0));
612+ emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, SHA256_DIGEST_LENGTH));
613+ off = -(gen->insn_cur - gen->insn_start - gen->cleanup_label) / 8 - 2;
614+ if (is_simm16(off)) {
615+ emit(gen, BPF_MOV64_IMM(BPF_REG_7, -EINVAL));
616+ emit(gen, BPF_JMP_IMM(BPF_JNE, BPF_REG_2, 1, off));
617+ } else {
618+ gen->error = -ERANGE;
619+ }
620+
588621 for (i = 0; i < SHA256_DWORD_SIZE; i++) {
589622 emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX,
590623 0, 0, 0, 0));
@@ -1053,7 +1086,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, · 462 unmodified lines
10531086 prog_idx, prog_type, insns_off, insn_cnt, license_off);
10541087
10551088 /* convert blob insns to target endianness */
1056− if (gen->swapped_endian) {
1089+ if (gen->swapped_endian && !gen->error) {
10571090 struct bpf_insn *insn = gen->data_start + insns_off;
10581091 int i;
10591092
@@ -1091,7 +1124,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, · 31 unmodified lines
10911124 sizeof(struct bpf_core_relo));
10921125
10931126 /* convert all info blobs to target endianness */
1094− if (gen->swapped_endian)
1127+ if (gen->swapped_endian && !gen->error)
10951128 info_blob_bswap(gen, func_info, line_info, core_relos, load_attr);
10961129
10971130 libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name));
@@ -1169,27 +1202,36 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue, · 71 unmodified lines
11691202 value = add_data(gen, pvalue, value_size);
11701203 key = add_data(gen, &zero, sizeof(zero));
11711204
1172− /* if (map_desc[map_idx].initial_value) {
1205+ /*
1206+ * if (map_desc[map_idx].initial_value) {
11731207 * if (ctx->flags & BPF_SKEL_KERNEL)
11741208 * bpf_probe_read_kernel(value, value_size, initial_value);
11751209 * else
11761210 * bpf_copy_from_user(value, value_size, initial_value);
11771211 * }
1212+ *
1213+ * The runtime initial_value comes from the host-supplied loader
1214+ * ctx and would overwrite the blob value after emit_signature_match()
1215+ * has already validated map->sha[]. For a signed loader (gen_hash)
1216+ * the attested blob value must be authoritative, so skip the override
1217+ * and leave the hashed value in place.
11781218 */
1179− emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6,
1180− sizeof(struct bpf_loader_ctx) +
1181− sizeof(struct bpf_map_desc) * map_idx +
1182− offsetof(struct bpf_map_desc, initial_value)));
1183− emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 8));
1184− emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
1185− 0, 0, 0, value));
1186− emit(gen, BPF_MOV64_IMM(BPF_REG_2, value_size));
1187− emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6,
1188− offsetof(struct bpf_loader_ctx, flags)));
1189− emit(gen, BPF_JMP_IMM(BPF_JSET, BPF_REG_0, BPF_SKEL_KERNEL, 2));
1190− emit(gen, BPF_EMIT_CALL(BPF_FUNC_copy_from_user));
1191− emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, 1));
1192− emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel));
1219+ if (!OPTS_GET(gen->opts, gen_hash, false)) {
1220+ emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6,
1221+ sizeof(struct bpf_loader_ctx) +
1222+ sizeof(struct bpf_map_desc) * map_idx +
1223+ offsetof(struct bpf_map_desc, initial_value)));
1224+ emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 8));
1225+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
1226+ 0, 0, 0, value));
1227+ emit(gen, BPF_MOV64_IMM(BPF_REG_2, value_size));
1228+ emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6,
1229+ offsetof(struct bpf_loader_ctx, flags)));
1230+ emit(gen, BPF_JMP_IMM(BPF_JSET, BPF_REG_0, BPF_SKEL_KERNEL, 2));
1231+ emit(gen, BPF_EMIT_CALL(BPF_FUNC_copy_from_user));
1232+ emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, 1));
1233+ emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel));
1234+ }
11931235
11941236 map_update_attr = add_data(gen, &attr, attr_size);
11951237 pr_debug("gen: map_update_elem: idx %d, value: off %d size %d, attr: off %d size %d\n",
tools/lib/bpf/libbpf.c+437−36
@@ -136,6 +136,9 @@ static const char * const attach_type_name[] = { · 135 unmodified lines
136136 [BPF_NETKIT_PEER] = "netkit_peer",
137137 [BPF_TRACE_KPROBE_SESSION] = "trace_kprobe_session",
138138 [BPF_TRACE_UPROBE_SESSION] = "trace_uprobe_session",
139+ [BPF_TRACE_FENTRY_MULTI] = "trace_fentry_multi",
140+ [BPF_TRACE_FEXIT_MULTI] = "trace_fexit_multi",
141+ [BPF_TRACE_FSESSION_MULTI] = "trace_fsession_multi",
139142 };
140143
141144 static const char * const link_type_name[] = {
@@ -154,6 +157,7 @@ static const char * const link_type_name[] = { · 12 unmodified lines
154157 [BPF_LINK_TYPE_UPROBE_MULTI] = "uprobe_multi",
155158 [BPF_LINK_TYPE_NETKIT] = "netkit",
156159 [BPF_LINK_TYPE_SOCKMAP] = "sockmap",
160+ [BPF_LINK_TYPE_TRACING_MULTI] = "tracing_multi",
157161 };
158162
159163 static const char * const map_type_name[] = {
@@ -192,6 +196,7 @@ static const char * const map_type_name[] = { · 32 unmodified lines
192196 [BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage",
193197 [BPF_MAP_TYPE_ARENA] = "arena",
194198 [BPF_MAP_TYPE_INSN_ARRAY] = "insn_array",
199+ [BPF_MAP_TYPE_RHASH] = "rhash",
195200 };
196201
197202 static const char * const prog_type_name[] = {
@@ -7767,6 +7772,69 @@ static int bpf_object__sanitize_prog(struct bpf_object *obj, struct bpf_program · 7569 unmodified lines
77677772 static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attach_name,
77687773 int *btf_obj_fd, int *btf_type_id);
77697774
7775+static inline bool is_tracing_multi(enum bpf_attach_type type)
7776+{
7777+ return type == BPF_TRACE_FENTRY_MULTI || type == BPF_TRACE_FEXIT_MULTI ||
7778+ type == BPF_TRACE_FSESSION_MULTI;
7779+}
7780+
7781+static const struct module_btf *find_attach_module(struct bpf_object *obj, const char *attach)
7782+{
7783+ const char *sep, *mod_name = NULL;
7784+ int i, mod_len, err;
7785+
7786+ /*
7787+ * We expect attach string in the form of either
7788+ * - function_pattern or
7789+ * - <module>:function_pattern
7790+ */
7791+ sep = strchr(attach, ':');
7792+ if (sep) {
7793+ mod_name = attach;
7794+ mod_len = sep - mod_name;
7795+ }
7796+ if (!mod_name)
7797+ return NULL;
7798+
7799+ err = load_module_btfs(obj);
7800+ if (err)
7801+ return NULL;
7802+
7803+ for (i = 0; i < obj->btf_module_cnt; i++) {
7804+ const struct module_btf *mod = &obj->btf_modules[i];
7805+
7806+ if (strncmp(mod->name, mod_name, mod_len) == 0 && mod->name[mod_len] == '\0')
7807+ return mod;
7808+ }
7809+ return NULL;
7810+}
7811+
7812+static int tracing_multi_mod_fd(struct bpf_program *prog, int *btf_obj_fd)
7813+{
7814+ const char *attach_name, *sep;
7815+ const struct module_btf *mod;
7816+
7817+ *btf_obj_fd = 0;
7818+ attach_name = strchr(prog->sec_name, '/');
7819+
7820+ /* Program with no details in spec, using kernel btf. */
7821+ if (!attach_name)
7822+ return 0;
7823+
7824+ /* Program with no module section, using kernel btf. */
7825+ sep = strchr(++attach_name, ':');
7826+ if (!sep)
7827+ return 0;
7828+
7829+ /* Program with module specified, get its btf fd. */
7830+ mod = find_attach_module(prog->obj, attach_name);
7831+ if (!mod)
7832+ return -EINVAL;
7833+
7834+ *btf_obj_fd = mod->fd;
7835+ return 0;
7836+}
7837+
77707838 /* this is called as prog->sec_def->prog_prepare_load_fn for libbpf-supported sec_defs */
77717839 static int libbpf_prepare_prog_load(struct bpf_program *prog,
77727840 struct bpf_prog_load_opts *opts, long cookie)
@@ -7830,6 +7898,18 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog, · 57 unmodified lines
78307898 opts->attach_btf_obj_fd = btf_obj_fd;
78317899 opts->attach_btf_id = btf_type_id;
78327900 }
7901+
7902+ if (is_tracing_multi(prog->expected_attach_type)) {
7903+ int err, btf_obj_fd = 0;
7904+
7905+ err = tracing_multi_mod_fd(prog, &btf_obj_fd);
7906+ if (err < 0)
7907+ return err;
7908+
7909+ prog->attach_btf_obj_fd = btf_obj_fd;
7910+ opts->attach_btf_obj_fd = btf_obj_fd;
7911+ }
7912+
78337913 return 0;
78347914 }
78357915
@@ -8936,13 +9016,10 @@ static void bpf_object_unpin(struct bpf_object *obj) · 1100 unmodified lines
89369016 bpf_map__unpin(&obj->maps[i], NULL);
89379017 }
89389018
8939−static void bpf_object_post_load_cleanup(struct bpf_object *obj)
9019+static void bpf_object_cleanup_btf(struct bpf_object *obj)
89409020 {
89419021 int i;
89429022
8943− /* clean up fd_array */
8944− zfree(&obj->fd_array);
8945−
89469023 /* clean up module BTFs */
89479024 for (i = 0; i < obj->btf_module_cnt; i++) {
89489025 close(obj->btf_modules[i].fd);
@@ -8950,6 +9027,8 @@ static void bpf_object_post_load_cleanup(struct bpf_object *obj) · 1 unmodified line
89509027 free(obj->btf_modules[i].name);
89519028 }
89529029 obj->btf_module_cnt = 0;
9030+ obj->btf_module_cap = 0;
9031+ obj->btf_modules_loaded = false;
89539032 zfree(&obj->btf_modules);
89549033
89559034 /* clean up vmlinux BTF */
@@ -8957,6 +9036,15 @@ static void bpf_object_post_load_cleanup(struct bpf_object *obj) · 1 unmodified line
89579036 obj->btf_vmlinux = NULL;
89589037 }
89599038
9039+static void bpf_object_post_load_cleanup(struct bpf_object *obj)
9040+{
9041+ /* clean up fd_array */
9042+ zfree(&obj->fd_array);
9043+
9044+ /* clean up BTF */
9045+ bpf_object_cleanup_btf(obj);
9046+}
9047+
89609048 static int bpf_object_prepare(struct bpf_object *obj, const char *target_btf_path)
89619049 {
89629050 int err;
@@ -9983,6 +10071,7 @@ static int attach_kprobe_session(const struct bpf_program *prog, long cookie, st · 1020 unmodified lines
998310071 static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
998410072 static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link);
998510073 static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link);
10074+static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
998610075
998710076 static const struct bpf_sec_def section_defs[] = {
998810077 SEC_DEF("socket", SOCKET_FILTER, 0, SEC_NONE),
@@ -10018,11 +10107,16 @@ static const struct bpf_sec_def section_defs[] = { · 29 unmodified lines
1001810107 SEC_DEF("netkit/peer", SCHED_CLS, BPF_NETKIT_PEER, SEC_NONE),
1001910108 SEC_DEF("tracepoint+", TRACEPOINT, 0, SEC_NONE, attach_tp),
1002010109 SEC_DEF("tp+", TRACEPOINT, 0, SEC_NONE, attach_tp),
10110+ SEC_DEF("tracepoint.s+", TRACEPOINT, 0, SEC_SLEEPABLE, attach_tp),
10111+ SEC_DEF("tp.s+", TRACEPOINT, 0, SEC_SLEEPABLE, attach_tp),
1002110112 SEC_DEF("raw_tracepoint+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp),
1002210113 SEC_DEF("raw_tp+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp),
10114+ SEC_DEF("raw_tracepoint.s+", RAW_TRACEPOINT, 0, SEC_SLEEPABLE, attach_raw_tp),
10115+ SEC_DEF("raw_tp.s+", RAW_TRACEPOINT, 0, SEC_SLEEPABLE, attach_raw_tp),
1002310116 SEC_DEF("raw_tracepoint.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp),
1002410117 SEC_DEF("raw_tp.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp),
1002510118 SEC_DEF("tp_btf+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF, attach_trace),
10119+ SEC_DEF("tp_btf.s+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
1002610120 SEC_DEF("fentry+", TRACING, BPF_TRACE_FENTRY, SEC_ATTACH_BTF, attach_trace),
1002710121 SEC_DEF("fmod_ret+", TRACING, BPF_MODIFY_RETURN, SEC_ATTACH_BTF, attach_trace),
1002810122 SEC_DEF("fexit+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF, attach_trace),
@@ -10031,6 +10125,12 @@ static const struct bpf_sec_def section_defs[] = { · 2 unmodified lines
1003110125 SEC_DEF("fexit.s+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
1003210126 SEC_DEF("fsession+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF, attach_trace),
1003310127 SEC_DEF("fsession.s+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
10128+ SEC_DEF("fsession.multi+", TRACING, BPF_TRACE_FSESSION_MULTI, 0, attach_tracing_multi),
10129+ SEC_DEF("fsession.multi.s+", TRACING, BPF_TRACE_FSESSION_MULTI, SEC_SLEEPABLE, attach_tracing_multi),
10130+ SEC_DEF("fentry.multi+", TRACING, BPF_TRACE_FENTRY_MULTI, 0, attach_tracing_multi),
10131+ SEC_DEF("fexit.multi+", TRACING, BPF_TRACE_FEXIT_MULTI, 0, attach_tracing_multi),
10132+ SEC_DEF("fentry.multi.s+", TRACING, BPF_TRACE_FENTRY_MULTI, SEC_SLEEPABLE, attach_tracing_multi),
10133+ SEC_DEF("fexit.multi.s+", TRACING, BPF_TRACE_FEXIT_MULTI, SEC_SLEEPABLE, attach_tracing_multi),
1003410134 SEC_DEF("freplace+", EXT, 0, SEC_ATTACH_BTF, attach_trace),
1003510135 SEC_DEF("lsm+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm),
1003610136 SEC_DEF("lsm.s+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm),
@@ -12280,7 +12380,7 @@ error: · 2243 unmodified lines
1228012380 static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf_link **link)
1228112381 {
1228212382 DECLARE_LIBBPF_OPTS(bpf_kprobe_opts, opts);
12283− unsigned long offset = 0;
12383+ long offset = 0;
1228412384 const char *func_name;
1228512385 char *func;
1228612386 int n;
@@ -12302,6 +12402,13 @@ static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf · 15 unmodified lines
1230212402 pr_warn("kprobe name is invalid: %s\n", func_name);
1230312403 return -EINVAL;
1230412404 }
12405+
12406+ if (offset < 0) {
12407+ free(func);
12408+ pr_warn("kprobe offset must be a non-negative integer: %li\n", offset);
12409+ return -EINVAL;
12410+ }
12411+
1230512412 if (opts.retprobe && offset != 0) {
1230612413 free(func);
1230712414 pr_warn("kretprobes do not support offset specification\n");
@@ -12425,6 +12532,279 @@ static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, stru · 117 unmodified lines
1242512532 return ret;
1242612533 }
1242712534
12535+#define MAX_BPF_FUNC_ARGS 12
12536+
12537+static bool btf_type_is_modifier(const struct btf_type *t)
12538+{
12539+ switch (BTF_INFO_KIND(t->info)) {
12540+ case BTF_KIND_TYPEDEF:
12541+ case BTF_KIND_VOLATILE:
12542+ case BTF_KIND_CONST:
12543+ case BTF_KIND_RESTRICT:
12544+ case BTF_KIND_TYPE_TAG:
12545+ return true;
12546+ default:
12547+ return false;
12548+ }
12549+}
12550+
12551+#define MAX_RESOLVE_DEPTH 32
12552+
12553+static int btf_get_type_size(const struct btf *btf, __u32 type_id,
12554+ const struct btf_type **ret_type)
12555+{
12556+ const struct btf_type *t;
12557+ int i;
12558+
12559+ *ret_type = btf__type_by_id(btf, 0);
12560+ if (!type_id)
12561+ return 0;
12562+ t = btf__type_by_id(btf, type_id);
12563+ for (i = 0; i < MAX_RESOLVE_DEPTH && t && btf_type_is_modifier(t); i++)
12564+ t = btf__type_by_id(btf, t->type);
12565+ if (!t || i == MAX_RESOLVE_DEPTH)
12566+ return -EINVAL;
12567+ *ret_type = t;
12568+ if (btf_is_ptr(t))
12569+ return btf__pointer_size(btf);
12570+ if (btf_is_int(t) || btf_is_any_enum(t) || btf_is_struct(t) || btf_is_union(t))
12571+ return t->size;
12572+ return -EINVAL;
12573+}
12574+
12575+bool btf_type_is_traceable_func(const struct btf *btf, const struct btf_type *t)
12576+{
12577+ const struct btf_param *args;
12578+ const struct btf_type *proto;
12579+ __u32 i, nargs;
12580+ int ret;
12581+
12582+ if (!btf_is_func(t))
12583+ return false;
12584+ proto = btf__type_by_id(btf, t->type);
12585+ if (!proto || !btf_is_func_proto(proto))
12586+ return false;
12587+
12588+ args = (const struct btf_param *)(proto + 1);
12589+ nargs = btf_vlen(proto);
12590+ if (nargs > MAX_BPF_FUNC_ARGS)
12591+ return false;
12592+
12593+ /* No support for struct return type. */
12594+ ret = btf_get_type_size(btf, proto->type, &t);
12595+ if (ret < 0 || btf_is_struct(t) || btf_is_union(t))
12596+ return false;
12597+
12598+ for (i = 0; i < nargs; i++) {
12599+ /* No support for variable args. */
12600+ if (i == nargs - 1 && args[i].type == 0)
12601+ return false;
12602+ ret = btf_get_type_size(btf, args[i].type, &t);
12603+ /* No support of struct argument size greater than 16 bytes. */
12604+ if (ret < 0 || ret > 16)
12605+ return false;
12606+ /* No support for void argument. */
12607+ if (ret == 0)
12608+ return false;
12609+ }
12610+
12611+ return true;
12612+}
12613+
12614+static int
12615+collect_btf_func_ids_by_glob(const struct btf *btf, const char *pattern, __u32 **ids)
12616+{
12617+ __u32 type_id, nr_types = btf__type_cnt(btf);
12618+ size_t cap = 0, cnt = 0;
12619+
12620+ if (!pattern)
12621+ return -EINVAL;
12622+
12623+ for (type_id = 1; type_id < nr_types; type_id++) {
12624+ const struct btf_type *t = btf__type_by_id(btf, type_id);
12625+ const char *name;
12626+ int err;
12627+
12628+ if (btf_kind(t) != BTF_KIND_FUNC)
12629+ continue;
12630+ name = btf__name_by_offset(btf, t->name_off);
12631+ if (!name)
12632+ continue;
12633+
12634+ if (!glob_match(name, pattern))
12635+ continue;
12636+ if (!btf_type_is_traceable_func(btf, t))
12637+ continue;
12638+
12639+ err = libbpf_ensure_mem((void **) ids, &cap, sizeof(**ids), cnt + 1);
12640+ if (err) {
12641+ free(*ids);
12642+ return -ENOMEM;
12643+ }
12644+ (*ids)[cnt++] = type_id;
12645+ }
12646+
12647+ return cnt;
12648+}
12649+
12650+static int collect_func_ids_by_glob(const struct bpf_program *prog, const char *pattern, __u32 **ids)
12651+{
12652+ struct bpf_object *obj = prog->obj;
12653+ const struct module_btf *mod;
12654+ struct btf *btf = NULL;
12655+ const char *sep;
12656+ int err;
12657+
12658+ err = bpf_object__load_vmlinux_btf(obj, true);
12659+ if (err)
12660+ return err;
12661+
12662+ /* In case we have module specified, we will find its btf and use that. */
12663+ sep = strchr(pattern, ':');
12664+ if (sep) {
12665+ mod = find_attach_module(obj, pattern);
12666+ if (!mod) {
12667+ err = -EINVAL;
12668+ goto cleanup;
12669+ }
12670+ btf = mod->btf;
12671+ pattern = sep + 1;
12672+ } else {
12673+ /* Program is loaded for kernel module. */
12674+ if (prog->attach_btf_obj_fd) {
12675+ err = -EINVAL;
12676+ goto cleanup;
12677+ }
12678+ btf = obj->btf_vmlinux;
12679+ }
12680+
12681+ err = collect_btf_func_ids_by_glob(btf, pattern, ids);
12682+
12683+cleanup:
12684+ bpf_object_cleanup_btf(obj);
12685+ return err;
12686+}
12687+
12688+struct bpf_link *
12689+bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern,
12690+ const struct bpf_tracing_multi_opts *opts)
12691+{
12692+ LIBBPF_OPTS(bpf_link_create_opts, lopts);
12693+ int prog_fd, link_fd, err, cnt;
12694+ __u32 *free_ids = NULL;
12695+ struct bpf_link *link;
12696+ const __u64 *cookies;
12697+ const __u32 *ids;
12698+
12699+ if (!OPTS_VALID(opts, bpf_tracing_multi_opts))
12700+ return libbpf_err_ptr(-EINVAL);
12701+
12702+ prog_fd = bpf_program__fd(prog);
12703+ if (prog_fd < 0) {
12704+ pr_warn("prog '%s': can't attach BPF program without FD (was it loaded?)\n",
12705+ prog->name);
12706+ return libbpf_err_ptr(-EINVAL);
12707+ }
12708+
12709+ cnt = OPTS_GET(opts, cnt, 0);
12710+ ids = OPTS_GET(opts, ids, NULL);
12711+ cookies = OPTS_GET(opts, cookies, NULL);
12712+
12713+ if (!!ids != !!cnt)
12714+ return libbpf_err_ptr(-EINVAL);
12715+ if (pattern && (ids || cookies))
12716+ return libbpf_err_ptr(-EINVAL);
12717+ if (!pattern && !ids)
12718+ return libbpf_err_ptr(-EINVAL);
12719+
12720+ if (pattern) {
12721+ cnt = collect_func_ids_by_glob(prog, pattern, &free_ids);
12722+ if (cnt < 0)
12723+ return libbpf_err_ptr(cnt);
12724+ if (cnt == 0)
12725+ return libbpf_err_ptr(-EINVAL);
12726+ ids = (const __u32 *) free_ids;
12727+ }
12728+
12729+ lopts.tracing_multi.ids = ids;
12730+ lopts.tracing_multi.cookies = cookies;
12731+ lopts.tracing_multi.cnt = cnt;
12732+
12733+ link = calloc(1, sizeof(*link));
12734+ if (!link) {
12735+ err = -ENOMEM;
12736+ goto error;
12737+ }
12738+ link->detach = &bpf_link__detach_fd;
12739+
12740+ link_fd = bpf_link_create(prog_fd, 0, prog->expected_attach_type, &lopts);
12741+ if (link_fd < 0) {
12742+ err = -errno;
12743+ pr_warn("prog '%s': failed to attach: %s\n", prog->name, errstr(err));
12744+ goto error;
12745+ }
12746+ link->fd = link_fd;
12747+ free(free_ids);
12748+ return link;
12749+
12750+error:
12751+ free(link);
12752+ free(free_ids);
12753+ return libbpf_err_ptr(err);
12754+}
12755+
12756+static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link)
12757+{
12758+ static const char *const prefixes[] = {
12759+ "fentry.multi",
12760+ "fexit.multi",
12761+ "fsession.multi",
12762+ "fentry.multi.s",
12763+ "fexit.multi.s",
12764+ "fsession.multi.s",
12765+ };
12766+ const char *spec = NULL;
12767+ char *pattern;
12768+ size_t i;
12769+ int n;
12770+
12771+ *link = NULL;
12772+
12773+ for (i = 0; i < ARRAY_SIZE(prefixes); i++) {
12774+ size_t pfx_len;
12775+
12776+ if (!str_has_pfx(prog->sec_name, prefixes[i]))
12777+ continue;
12778+
12779+ pfx_len = strlen(prefixes[i]);
12780+ /* no auto-attach case of, e.g., SEC("fentry.multi") */
12781+ if (prog->sec_name[pfx_len] == '\0')
12782+ return 0;
12783+
12784+ if (prog->sec_name[pfx_len] != '/')
12785+ continue;
12786+
12787+ spec = prog->sec_name + pfx_len + 1;
12788+ break;
12789+ }
12790+
12791+ if (!spec) {
12792+ pr_warn("prog '%s': invalid section name '%s'\n",
12793+ prog->name, prog->sec_name);
12794+ return -EINVAL;
12795+ }
12796+
12797+ n = sscanf(spec, "%m[a-zA-Z0-9_.*?:]", &pattern);
12798+ if (n < 1) {
12799+ pr_warn("tracing multi pattern is invalid: %s\n", spec);
12800+ return -EINVAL;
12801+ }
12802+
12803+ *link = bpf_program__attach_tracing_multi(prog, pattern, NULL);
12804+ free(pattern);
12805+ return libbpf_get_error(*link);
12806+}
12807+
1242812808 static inline int add_uprobe_event_legacy(const char *probe_name, bool retprobe,
1242912809 const char *binary_path, size_t offset)
1243012810 {
@@ -13145,25 +13525,61 @@ struct bpf_link *bpf_program__attach_tracepoint(const struct bpf_program *prog, · 714 unmodified lines
1314513525 return bpf_program__attach_tracepoint_opts(prog, tp_category, tp_name, NULL);
1314613526 }
1314713527
13528+/*
13529+ * Match section name against a prefix array. Returns pointer past
13530+ * "prefix/" on match, empty string for bare sections (exact prefix
13531+ * match), or NULL if no prefix matches.
13532+ */
13533+static const char *sec_name_match_prefix(const char *sec_name,
13534+ const char *const *prefixes,
13535+ size_t n)
13536+{
13537+ size_t i;
13538+
13539+ for (i = 0; i < n; i++) {
13540+ size_t pfx_len;
13541+
13542+ if (!str_has_pfx(sec_name, prefixes[i]))
13543+ continue;
13544+
13545+ pfx_len = strlen(prefixes[i]);
13546+ if (sec_name[pfx_len] == '\0')
13547+ return sec_name + pfx_len;
13548+
13549+ if (sec_name[pfx_len] != '/' || sec_name[pfx_len + 1] == '\0')
13550+ continue;
13551+
13552+ return sec_name + pfx_len + 1;
13553+ }
13554+ return NULL;
13555+}
13556+
1314813557 static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link)
1314913558 {
13559+ static const char *const prefixes[] = {
13560+ "tp.s",
13561+ "tp",
13562+ "tracepoint.s",
13563+ "tracepoint",
13564+ };
1315013565 char *sec_name, *tp_cat, *tp_name;
13566+ const char *match;
1315113567
1315213568 *link = NULL;
1315313569
13154− /* no auto-attach for SEC("tp") or SEC("tracepoint") */
13155− if (strcmp(prog->sec_name, "tp") == 0 || strcmp(prog->sec_name, "tracepoint") == 0)
13570+ match = sec_name_match_prefix(prog->sec_name, prefixes, ARRAY_SIZE(prefixes));
13571+ if (!match) {
13572+ pr_warn("prog '%s': invalid section name '%s'\n", prog->name, prog->sec_name);
13573+ return -EINVAL;
13574+ }
13575+ if (!match[0]) /* bare section name no autoattach */
1315613576 return 0;
1315713577
1315813578 sec_name = strdup(prog->sec_name);
1315913579 if (!sec_name)
1316013580 return -ENOMEM;
1316113581
13162− /* extract "tp/<category>/<name>" or "tracepoint/<category>/<name>" */
13163− if (str_has_pfx(prog->sec_name, "tp/"))
13164− tp_cat = sec_name + sizeof("tp/") - 1;
13165− else
13166− tp_cat = sec_name + sizeof("tracepoint/") - 1;
13582+ tp_cat = sec_name + (match - prog->sec_name);
1316713583 tp_name = strchr(tp_cat, '/');
1316813584 if (!tp_name) {
1316913585 free(sec_name);
@@ -13227,37 +13643,22 @@ static int attach_raw_tp(const struct bpf_program *prog, long cookie, struct bpf · 57 unmodified lines
1322713643 "raw_tracepoint",
1322813644 "raw_tp.w",
1322913645 "raw_tracepoint.w",
13646+ "raw_tp.s",
13647+ "raw_tracepoint.s",
1323013648 };
13231− size_t i;
13232− const char *tp_name = NULL;
13649+ const char *match;
1323313650
1323413651 *link = NULL;
1323513652
13236− for (i = 0; i < ARRAY_SIZE(prefixes); i++) {
13237− size_t pfx_len;
13238−
13239− if (!str_has_pfx(prog->sec_name, prefixes[i]))
13240− continue;
13241−
13242− pfx_len = strlen(prefixes[i]);
13243− /* no auto-attach case of, e.g., SEC("raw_tp") */
13244− if (prog->sec_name[pfx_len] == '\0')
13245− return 0;
13246−
13247− if (prog->sec_name[pfx_len] != '/')
13248− continue;
13249−
13250− tp_name = prog->sec_name + pfx_len + 1;
13251− break;
13252− }
13253−
13254− if (!tp_name) {
13255− pr_warn("prog '%s': invalid section name '%s'\n",
13256− prog->name, prog->sec_name);
13653+ match = sec_name_match_prefix(prog->sec_name, prefixes, ARRAY_SIZE(prefixes));
13654+ if (!match) {
13655+ pr_warn("prog '%s': invalid section name '%s'\n", prog->name, prog->sec_name);
1325713656 return -EINVAL;
1325813657 }
13658+ if (!match[0])
13659+ return 0;
1325913660
13260− *link = bpf_program__attach_raw_tracepoint(prog, tp_name);
13661+ *link = bpf_program__attach_raw_tracepoint(prog, match);
1326113662 return libbpf_get_error(*link);
1326213663 }
1326313664
tools/lib/bpf/libbpf.h+15−0
@@ -726,6 +726,21 @@ bpf_program__attach_ksyscall(const struct bpf_program *prog, · 725 unmodified lines
726726 const char *syscall_name,
727727 const struct bpf_ksyscall_opts *opts);
728728
729+struct bpf_tracing_multi_opts {
730+ /* size of this struct, for forward/backward compatibility */
731+ size_t sz;
732+ const __u32 *ids;
733+ const __u64 *cookies;
734+ size_t cnt;
735+ size_t :0;
736+};
737+
738+#define bpf_tracing_multi_opts__last_field cnt
739+
740+LIBBPF_API struct bpf_link *
741+bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern,
742+ const struct bpf_tracing_multi_opts *opts);
743+
729744 struct bpf_uprobe_opts {
730745 /* size of this struct, for forward/backward compatibility */
731746 size_t sz;
tools/lib/bpf/libbpf.map+1−0
@@ -458,6 +458,7 @@ LIBBPF_1.7.0 { · 457 unmodified lines
458458
459459 LIBBPF_1.8.0 {
460460 global:
461+ bpf_program__attach_tracing_multi;
461462 bpf_program__clone;
462463 btf__new_empty_opts;
463464 } LIBBPF_1.7.0;
tools/lib/bpf/libbpf_internal.h+4−0
@@ -250,6 +250,7 @@ const struct btf_type *skip_mods_and_typedefs(const struct btf *btf, __u32 id, _ · 249 unmodified lines
250250 const struct btf_header *btf_header(const struct btf *btf);
251251 void btf_set_base_btf(struct btf *btf, const struct btf *base_btf);
252252 int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **id_map);
253+bool btf_type_is_traceable_func(const struct btf *btf, const struct btf_type *t);
253254
254255 static inline enum btf_func_linkage btf_func_linkage(const struct btf_type *t)
255256 {
@@ -398,6 +399,8 @@ enum kern_feature_id { · 142 unmodified lines
398399 FEAT_UPROBE_SYSCALL,
399400 /* Kernel supports BTF layout information */
400401 FEAT_BTF_LAYOUT,
402+ /* Kernel supports BPF syscall common attributes */
403+ FEAT_BPF_SYSCALL_COMMON_ATTRS,
401404 __FEAT_CNT,
402405 };
403406
@@ -768,4 +771,5 @@ int probe_fd(int fd); · 364 unmodified lines
768771 #define SHA256_DWORD_SIZE SHA256_DIGEST_LENGTH / sizeof(__u64)
769772
770773 void libbpf_sha256(const void *data, size_t len, __u8 out[SHA256_DIGEST_LENGTH]);
774+int probe_sys_bpf_ext(void);
771775 #endif /* __LIBBPF_LIBBPF_INTERNAL_H */
tools/lib/bpf/libbpf_probes.c+3−0
@@ -309,6 +309,9 @@ static int probe_map_create(enum bpf_map_type map_type) · 308 unmodified lines
309309 value_size = sizeof(__u64);
310310 opts.map_flags = BPF_F_NO_PREALLOC;
311311 break;
312+ case BPF_MAP_TYPE_RHASH:
313+ opts.map_flags = BPF_F_NO_PREALLOC;
314+ break;
312315 case BPF_MAP_TYPE_CGROUP_STORAGE:
313316 case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE:
314317 key_size = sizeof(struct bpf_cgroup_storage_key);
tools/lib/bpf/relo_core.c+8−8
@@ -191,8 +191,8 @@ recur: · 190 unmodified lines
191191 case BTF_KIND_FUNC_PROTO: {
192192 struct btf_param *local_p = btf_params(local_type);
193193 struct btf_param *targ_p = btf_params(targ_type);
194− __u16 local_vlen = btf_vlen(local_type);
195− __u16 targ_vlen = btf_vlen(targ_type);
194+ __u32 local_vlen = btf_vlen(local_type);
195+ __u32 targ_vlen = btf_vlen(targ_type);
196196 int i, err;
197197
198198 if (local_vlen != targ_vlen)
@@ -1457,8 +1457,8 @@ static bool bpf_core_names_match(const struct btf *local_btf, size_t local_name_ · 1258 unmodified lines
14571457 static int bpf_core_enums_match(const struct btf *local_btf, const struct btf_type *local_t,
14581458 const struct btf *targ_btf, const struct btf_type *targ_t)
14591459 {
1460− __u16 local_vlen = btf_vlen(local_t);
1461− __u16 targ_vlen = btf_vlen(targ_t);
1460+ __u32 local_vlen = btf_vlen(local_t);
1461+ __u32 targ_vlen = btf_vlen(targ_t);
14621462 int i, j;
14631463
14641464 if (local_t->size != targ_t->size)
@@ -1498,8 +1498,8 @@ static int bpf_core_composites_match(const struct btf *local_btf, const struct b · 33 unmodified lines
14981498 bool behind_ptr, int level)
14991499 {
15001500 const struct btf_member *local_m = btf_members(local_t);
1501− __u16 local_vlen = btf_vlen(local_t);
1502− __u16 targ_vlen = btf_vlen(targ_t);
1501+ __u32 local_vlen = btf_vlen(local_t);
1502+ __u32 targ_vlen = btf_vlen(targ_t);
15031503 int i, j, err;
15041504
15051505 if (local_vlen > targ_vlen)
@@ -1674,8 +1674,8 @@ recur: · 168 unmodified lines
16741674 case BTF_KIND_FUNC_PROTO: {
16751675 struct btf_param *local_p = btf_params(local_t);
16761676 struct btf_param *targ_p = btf_params(targ_t);
1677− __u16 local_vlen = btf_vlen(local_t);
1678− __u16 targ_vlen = btf_vlen(targ_t);
1677+ __u32 local_vlen = btf_vlen(local_t);
1678+ __u32 targ_vlen = btf_vlen(targ_t);
16791679 int i, err;
16801680
16811681 if (local_k != targ_k)
tools/lib/bpf/skel_internal.h+5−0
@@ -243,7 +243,12 @@ static inline int skel_map_create(enum bpf_map_type map_type, · 242 unmodified lines
243243 attr.excl_prog_hash = (unsigned long) excl_prog_hash;
244244 attr.excl_prog_hash_size = excl_prog_hash_sz;
245245
246+#ifdef __KERNEL__
247+ if (strscpy(attr.map_name, map_name) < 0)
248+ return -EINVAL;
249+#else
246250 strncpy(attr.map_name, map_name, sizeof(attr.map_name));
251+#endif
247252 attr.key_size = key_size;
248253 attr.value_size = value_size;
249254 attr.max_entries = max_entries;
tools/lib/bpf/strset.c+41−21
@@ -107,6 +107,41 @@ static void *strset_add_str_mem(struct strset *set, size_t add_sz) · 106 unmodified lines
107107 set->strs_data_len, set->strs_data_max_len, add_sz);
108108 }
109109
110+static long strset_str_append(struct strset *set, const char *s)
111+{
112+ uintptr_t old_data = (uintptr_t)set->strs_data;
113+ size_t old_data_len = set->strs_data_len;
114+ uintptr_t old_s = (uintptr_t)s;
115+ long len = strlen(s) + 1;
116+ void *p;
117+
118+ /*
119+ * Hashmap keys are always offsets within set->strs_data, so to even
120+ * look up some string from the "outside", we need to first append it
121+ * at the end, so that it can be addressed with an offset. Luckily,
122+ * until set->strs_data_len is incremented, that string is just a piece
123+ * of garbage for the rest of the code, so no harm, no foul. On the
124+ * other hand, if the string is unique, it's already appended and
125+ * ready to be used, only a simple set->strs_data_len increment away.
126+ */
127+ p = strset_add_str_mem(set, len);
128+ if (!p)
129+ return -ENOMEM;
130+
131+ /*
132+ * The set->strs_data might have reallocated and if 's' pointed
133+ * to an internal string within the old buffer, then it became
134+ * dangling and needs to be reconstructed before the copy.
135+ */
136+ if (old_data && old_data != (uintptr_t)set->strs_data &&
137+ old_s >= old_data && old_s < old_data + old_data_len)
138+ s = set->strs_data + (old_s - old_data);
139+
140+ memcpy(p, s, len);
141+
142+ return len;
143+}
144+
110145 /* Find string offset that corresponds to a given string *s*.
111146 * Returns:
112147 * - >0 offset into string data, if string is found;
@@ -116,16 +151,12 @@ static void *strset_add_str_mem(struct strset *set, size_t add_sz) · 3 unmodified lines
116151 int strset__find_str(struct strset *set, const char *s)
117152 {
118153 long old_off, new_off, len;
119− void *p;
120154
121− /* see strset__add_str() for why we do this */
122− len = strlen(s) + 1;
123− p = strset_add_str_mem(set, len);
124− if (!p)
125− return -ENOMEM;
155+ len = strset_str_append(set, s);
156+ if (len < 0)
157+ return len;
126158
127159 new_off = set->strs_data_len;
128− memcpy(p, s, len);
129160
130161 if (hashmap__find(set->strs_hash, new_off, &old_off))
131162 return old_off;
@@ -142,24 +173,13 @@ int strset__find_str(struct strset *set, const char *s) · 10 unmodified lines
142173 int strset__add_str(struct strset *set, const char *s)
143174 {
144175 long old_off, new_off, len;
145− void *p;
146176 int err;
147177
148− /* Hashmap keys are always offsets within set->strs_data, so to even
149− * look up some string from the "outside", we need to first append it
150− * at the end, so that it can be addressed with an offset. Luckily,
151− * until set->strs_data_len is incremented, that string is just a piece
152− * of garbage for the rest of the code, so no harm, no foul. On the
153− * other hand, if the string is unique, it's already appended and
154− * ready to be used, only a simple set->strs_data_len increment away.
155− */
156− len = strlen(s) + 1;
157− p = strset_add_str_mem(set, len);
158− if (!p)
159− return -ENOMEM;
178+ len = strset_str_append(set, s);
179+ if (len < 0)
180+ return len;
160181
161182 new_off = set->strs_data_len;
162− memcpy(p, s, len);
163183
164184 /* Now attempt to add the string, but only if the string with the same
165185 * contents doesn't exist already (HASHMAP_ADD strategy). If such
tools/lib/bpf/usdt.c+12−4
@@ -468,10 +468,10 @@ static int parse_elf_segs(Elf *elf, const char *path, struct elf_seg **segs, siz · 467 unmodified lines
468468
469469 static int parse_vma_segs(int pid, const char *lib_path, struct elf_seg **segs, size_t *seg_cnt)
470470 {
471− char path[PATH_MAX], line[PATH_MAX], mode[16];
471+ char path[PATH_MAX], line[4096], mode[16];
472472 size_t seg_start, seg_end, seg_off;
473473 struct elf_seg *seg;
474− int tmp_pid, i, err;
474+ int tmp_pid, n, i, err;
475475 FILE *f;
476476
477477 *seg_cnt = 0;
@@ -480,8 +480,13 @@ static int parse_vma_segs(int pid, const char *lib_path, struct elf_seg **segs, · 2 unmodified lines
480480 * /proc/<pid>/root/<path>. They will be reported as just /<path> in
481481 * /proc/<pid>/maps.
482482 */
483− if (sscanf(lib_path, "/proc/%d/root%s", &tmp_pid, path) == 2 && pid == tmp_pid)
483+ /* %n is not counted in sscanf() return value, so initialize it. */
484+ n = 0;
485+ if (sscanf(lib_path, "/proc/%d/root%n", &tmp_pid, &n) == 1 &&
486+ n > 0 && pid == tmp_pid && lib_path[n] == '/') {
487+ libbpf_strlcpy(path, lib_path + n, sizeof(path));
484488 goto proceed;
489+ }
485490
486491 if (!realpath(lib_path, path)) {
487492 pr_warn("usdt: failed to get absolute path of '%s' (err %s), using path as is...\n",
@@ -504,8 +509,11 @@ proceed: · 16 unmodified lines
504509 * 7f5c6f5d1000-7f5c6f5d3000 rw-p 001c7000 08:04 21238613 /usr/lib64/libc-2.17.so
505510 * 7f5c6f5d3000-7f5c6f5d8000 rw-p 00000000 00:00 0
506511 * 7f5c6f5d8000-7f5c6f5d9000 r-xp 00000000 103:01 362990598 /data/users/andriin/linux/tools/bpf/usdt/libhello_usdt.so
512+ *
513+ * Some VMA names can be longer than the local buffer. Bound the
514+ * writes, but still consume the rest of the line.
507515 */
508− while (fscanf(f, "%zx-%zx %s %zx %*s %*d%[^\n]\n",
516+ while (fscanf(f, "%zx-%zx %15s %zx %*s %*d%4095[^\n]%*[^\n]\n",
509517 &seg_start, &seg_end, mode, &seg_off, line) == 5) {
510518 void *tmp;
511519
tools/testing/selftests/bpf/.gitignore+0−1
@@ -21,7 +21,6 @@ test_lirc_mode2_user · 20 unmodified lines
2121 flow_dissector_load
2222 test_tcpnotify_user
2323 test_libbpf
24−xdping
2524 test_cpp
2625 *.d
2726 *.subskel.h
tools/testing/selftests/bpf/Makefile+201−66
@@ -44,6 +44,12 @@ SKIP_LLVM ?= · 43 unmodified lines
4444 SKIP_LIBBFD ?=
4545 SKIP_CRYPTO ?=
4646
47+# When BPF_STRICT_BUILD is 1, any BPF object, skeleton, test object, or
48+# benchmark compilation failure is fatal. Set to 0 to tolerate failures
49+# and continue building the remaining tests.
50+BPF_STRICT_BUILD ?= 1
51+PERMISSIVE := $(filter 0,$(BPF_STRICT_BUILD))
52+
4753 ifeq ($(srctree),)
4854 srctree := $(patsubst %/,%,$(dir $(CURDIR)))
4955 srctree := $(patsubst %/,%,$(dir $(srctree)))
@@ -51,19 +57,20 @@ srctree := $(patsubst %/,%,$(dir $(srctree))) · 1 unmodified line
5157 srctree := $(patsubst %/,%,$(dir $(srctree)))
5258 endif
5359
54−CFLAGS += -g $(OPT_FLAGS) -rdynamic -std=gnu11 \
60+COMMON_CFLAGS = -g $(OPT_FLAGS) -rdynamic -std=gnu11 \
5561 -Wall -Werror -fno-omit-frame-pointer \
5662 -Wno-unused-but-set-variable \
5763 $(GENFLAGS) $(SAN_CFLAGS) $(LIBELF_CFLAGS) \
5864 -I$(CURDIR) -I$(INCLUDE_DIR) -I$(GENDIR) -I$(LIBDIR) \
59− -I$(TOOLSINCDIR) -I$(TOOLSARCHINCDIR) -I$(APIDIR) -I$(OUTPUT)
65+ -I$(TOOLSINCDIR) -I$(TOOLSARCHINCDIR) -I$(APIDIR) -I$(OUTPUT) \
66+ -I$(CURDIR)/libarena/include
6067 LDFLAGS += $(SAN_LDFLAGS)
6168 LDLIBS += $(LIBELF_LIBS) -lz -lrt -lpthread
6269
6370 PCAP_CFLAGS := $(shell $(PKG_CONFIG) --cflags libpcap 2>/dev/null && echo "-DTRAFFIC_MONITOR=1")
6471 PCAP_LIBS := $(shell $(PKG_CONFIG) --libs libpcap 2>/dev/null)
6572 LDLIBS += $(PCAP_LIBS)
66−CFLAGS += $(PCAP_CFLAGS)
73+CFLAGS += $(COMMON_CFLAGS) $(PCAP_CFLAGS)
6774
6875 # Some utility functions use LLVM libraries
6976 jit_disasm_helpers.c-CFLAGS = $(LLVM_CFLAGS)
@@ -78,6 +85,12 @@ ifneq ($(shell $(CLANG) --target=bpf -mcpu=help 2>&1 | grep 'v4'),) · 8 unmodified lines
7885 CLANG_CPUV4 := 1
7986 endif
8087
88+# Check whether clang supports BPF address sanitizer (requires LLVM 22+)
89+CLANG_HAS_ARENA_ASAN := $(shell echo 'int x;' | \
90+ $(CLANG) --target=bpf -fsanitize=kernel-address \
91+ -mllvm -asan-shadow-addr-space=1 \
92+ -x c -c - -o /dev/null 2>/dev/null && echo 1)
93+
8194 # Order correspond to 'make run_tests' order
8295 TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_progs \
8396 test_sockmap \
@@ -111,7 +124,6 @@ TEST_FILES = xsk_prereqs.sh $(wildcard progs/btf_dump_test_case_*.c) · 27 unmodified lines
111124 # Order correspond to 'make run_tests' order
112125 TEST_PROGS := test_kmod.sh \
113126 test_lirc_mode2.sh \
114− test_xdping.sh \
115127 test_bpftool_build.sh \
116128 test_doc_build.sh \
117129 test_xsk.sh \
@@ -134,7 +146,6 @@ TEST_GEN_PROGS_EXTENDED = \ · 16 unmodified lines
134146 xdp_features \
135147 xdp_hw_metadata \
136148 xdp_synproxy \
137− xdping \
138149 xskxceiver
139150
140151 TEST_GEN_FILES += $(TEST_KMODS) liburandom_read.so urandom_read sign-file uprobe_multi
@@ -153,12 +164,13 @@ override define CLEAN · 12 unmodified lines
153164 $(Q)$(RM) -r $(TEST_KMODS)
154165 $(Q)$(RM) -r $(EXTRA_CLEAN)

445 more lines aren’t shown here.

tools/testing/selftests/bpf/README.rst+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/bench.c+21−5

This diff is large: its 96 lines aren’t shown here.

tools/testing/selftests/bpf/bench.h+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/bpf/bench_bpf_timing.h+50−0

This diff is large: its 51 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_full_update.c+32−2

This diff is large: its 63 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_lookup.c+29−2

This diff is large: its 61 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/bench_bpf_nop.c+84−0

This diff is large: its 85 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/bench_bpf_timing.c+298−0

This diff is large: its 299 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/bench_htab_mem.c+33−2

This diff is large: its 70 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/bench_xdp_lb.c+1124−0

This diff is large: its 1125 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/run_bench_xdp_lb.sh+79−0

This diff is large: its 80 lines aren’t shown here.

tools/testing/selftests/bpf/bpf_arena_alloc.h+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/bpf_arena_htab.h+5−6

This diff is large: its 47 lines aren’t shown here.

tools/testing/selftests/bpf/bpf_arena_list.h+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/bpf_arena_strsearch.h+3−3

This diff is large: its 21 lines aren’t shown here.

tools/testing/selftests/bpf/bpf_experimental.h+1−83

This diff is large: its 98 lines aren’t shown here.

tools/testing/selftests/bpf/bpf_kfuncs.h+4−4

This diff is large: its 26 lines aren’t shown here.

tools/testing/selftests/bpf/config+2−1

This diff is large: its 7 lines aren’t shown here.

tools/testing/selftests/bpf/default.profraw+0−0

Binary file not shown.

tools/testing/selftests/bpf/jit_disasm_helpers.c+11−2

This diff is large: its 22 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/Makefile+92−0

This diff is large: its 93 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/bpf_arena_common.hrenamed from tools/testing/selftests/bpf/bpf_arena_common.h+2−3

This diff is large: its 23 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/bpf_arena_spin_lock.hrenamed from tools/testing/selftests/bpf/progs/bpf_arena_spin_lock.h+9−8

This diff is large: its 45 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/bpf_atomic.hrenamed from tools/testing/selftests/bpf/bpf_atomic.h+3−1

This diff is large: its 19 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/bpf_may_goto.h+84−0

This diff is large: its 85 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/libarena/asan.h+101−0

This diff is large: its 102 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/libarena/buddy.h+81−0

This diff is large: its 82 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/libarena/common.h+93−0

This diff is large: its 94 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/libarena/rbtree.h+83−0

This diff is large: its 84 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/libarena/spmc.h+27−0

This diff is large: its 28 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/libarena/userspace.h+138−0

This diff is large: its 139 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/st_asan_buddy.bpf.c+258−0

This diff is large: its 259 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/st_asan_common.h+52−0

This diff is large: its 53 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/st_buddy.bpf.c+209−0

This diff is large: its 210 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/test_parallel_spmc.bpf.c+669−0

This diff is large: its 670 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/test_progs_compat.h+15−0

This diff is large: its 16 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/test_rbtree.bpf.c+968−0

This diff is large: its 969 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/test_spmc.bpf.c+194−0

This diff is large: its 195 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/src/asan.bpf.c+553−0

This diff is large: its 554 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/src/buddy.bpf.c+903−0

This diff is large: its 904 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/src/common.bpf.c+52−0

This diff is large: its 53 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/src/rbtree.bpf.c+1047−0

This diff is large: its 1048 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/src/spmc.bpf.c+234−0

This diff is large: its 235 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/arena_direct_value.c+73−0

This diff is large: its 74 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/arena_spin_lock.c+0−7

This diff is large: its 14 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/bpf_attr_size.c+124−0

This diff is large: its 125 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/bpf_cookie.c+11−6

This diff is large: its 31 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/bpf_nf.c+11−11

This diff is large: its 30 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c+8−0

This diff is large: its 22 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/btf.c+41−4

This diff is large: its 76 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/btf_dedup_split.c+33−18

This diff is large: its 125 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/btf_dump.c+2−2

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/cb_refs.c+2−2

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/cgrp_local_storage.c+13−2

This diff is large: its 38 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c+8−9

This diff is large: its 37 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/exceptions.c+7−0

This diff is large: its 14 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/file_reader.c+10−12

This diff is large: its 50 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/fill_link_info.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/htab_update.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/iters.c+0−2

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/kfunc_call.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/libarena.c+253−0

This diff is large: its 254 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/libarena_asan.c+93−0

This diff is large: its 94 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/linked_list.c+19−18

This diff is large: its 114 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c+243−0

This diff is large: its 244 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/lsm_cgroup.c+79−0

This diff is large: its 98 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c+145−0

This diff is large: its 163 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/map_excl.c+122−0

This diff is large: its 140 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/map_init.c+192−0

This diff is large: its 196 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/map_kptr.c+57−9

This diff is large: its 136 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/refcounted_kptr.c+6−2

This diff is large: its 22 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/reg_bounds.c+75−15

This diff is large: its 120 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/rhash.c+183−0

This diff is large: its 184 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/setget_sockopt.c+78−0

This diff is large: its 92 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/signed_loader.c+1135−0

This diff is large: its 1136 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/sleepable_tracepoints.c+142−0

This diff is large: its 143 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/sockmap_basic.c+48−0

This diff is large: its 69 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/spin_lock.c+2−2

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/stack_arg.c+139−0

This diff is large: its 140 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c+10−0

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/stack_arg_precision.c+10−0

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/tailcalls.c+186−0

This diff is large: its 205 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/task_kfunc.c+41−1

This diff is large: its 70 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/task_local_storage.c+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/test_lsm.c+0−22

This diff is large: its 37 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/test_xdp_veth.c+163−3

This diff is large: its 200 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/tracing_multi.c+960−0

This diff is large: its 961 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c+93−1

This diff is large: its 122 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/verifier.c+8−0

This diff is large: its 50 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/verifier_log.c+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/vmlinux.c+43−2

This diff is large: its 71 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/wakeup_source.c+118−0

This diff is large: its 119 lines aren’t shown here.

tools/testing/selftests/bpf/progs/arena_atomics.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/arena_spin_lock.c+2−1

This diff is large: its 10 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h+69−0

This diff is large: its 70 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_iter_bpf_rhash_map.c+34−0

This diff is large: its 35 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_iter_task_vmas.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_misc.h+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_nop_bench.c+14−0

This diff is large: its 15 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_qdisc_dynptr_use_after_invalidate_clone.c+74−0

This diff is large: its 75 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr.c+68−0

This diff is large: its 69 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c+74−0

This diff is large: its 75 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_slice.c+70−0

This diff is large: its 71 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_qdisc_fq.c+3−8

This diff is large: its 22 lines aren’t shown here.

tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c+24−0

This diff is large: its 25 lines aren’t shown here.

tools/testing/selftests/bpf/progs/btf__verifier_stack_arg_order.c+49−0

This diff is large: its 50 lines aren’t shown here.

tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c+9−9

This diff is large: its 81 lines aren’t shown here.

tools/testing/selftests/bpf/progs/cgrp_ls_sleepable.c+18−0

This diff is large: its 36 lines aren’t shown here.

tools/testing/selftests/bpf/progs/compute_live_registers.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/cpumask_failure.c+5−5

This diff is large: its 45 lines aren’t shown here.

tools/testing/selftests/bpf/progs/cpumask_success.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/crypto_bench.c+14−7

This diff is large: its 44 lines aren’t shown here.

tools/testing/selftests/bpf/progs/crypto_sanity.c+14−7

This diff is large: its 45 lines aren’t shown here.

tools/testing/selftests/bpf/progs/dynptr_fail.c+58−16

This diff is large: its 193 lines aren’t shown here.

tools/testing/selftests/bpf/progs/dynptr_success.c+3−3

This diff is large: its 27 lines aren’t shown here.

tools/testing/selftests/bpf/progs/exceptions.c+114−0

This diff is large: its 119 lines aren’t shown here.

tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c+11−2

This diff is large: its 26 lines aren’t shown here.

tools/testing/selftests/bpf/progs/file_reader.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/file_reader_fail.c+62−2

This diff is large: its 82 lines aren’t shown here.

tools/testing/selftests/bpf/progs/htab_update.c+2−2

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/progs/irq.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/iters.c+3−3

This diff is large: its 27 lines aren’t shown here.

tools/testing/selftests/bpf/progs/iters_state_safety.c+9−9

This diff is large: its 81 lines aren’t shown here.

tools/testing/selftests/bpf/progs/iters_testmod.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/iters_testmod_seq.c+8−8

This diff is large: its 51 lines aren’t shown here.

tools/testing/selftests/bpf/progs/linked_list.c+71−0

This diff is large: its 78 lines aren’t shown here.

tools/testing/selftests/bpf/progs/lpm_trie_bench.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/lru_lock_nmi.c+33−0

This diff is large: its 34 lines aren’t shown here.

tools/testing/selftests/bpf/progs/lsm_cgroup.c+30−0

This diff is large: its 41 lines aren’t shown here.

tools/testing/selftests/bpf/progs/map_kptr.c+86−3

This diff is large: its 107 lines aren’t shown here.

tools/testing/selftests/bpf/progs/map_kptr_fail.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/percpu_alloc_fail.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/rbtree_fail.c+3−3

This diff is large: its 27 lines aren’t shown here.

tools/testing/selftests/bpf/progs/refcounted_kptr.c+440−1

This diff is large: its 461 lines aren’t shown here.

tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/rhash.c+248−0

This diff is large: its 249 lines aren’t shown here.

tools/testing/selftests/bpf/progs/setget_sockopt.c+23−0

This diff is large: its 37 lines aren’t shown here.

tools/testing/selftests/bpf/progs/sk_bypass_prot_mem.c+2−0

This diff is large: its 16 lines aren’t shown here.

tools/testing/selftests/bpf/progs/stack_arg.c+273−0

This diff is large: its 274 lines aren’t shown here.

tools/testing/selftests/bpf/progs/stack_arg_fail.c+114−0

This diff is large: its 115 lines aren’t shown here.

tools/testing/selftests/bpf/progs/stack_arg_kfunc.c+166−0

This diff is large: its 167 lines aren’t shown here.

tools/testing/selftests/bpf/progs/stack_arg_precision.c+135−0

This diff is large: its 136 lines aren’t shown here.

tools/testing/selftests/bpf/progs/stream.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/stream_fail.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_bpf2bpf2.c+4−1

This diff is large: its 16 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy1.c+9−4

This diff is large: its 30 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy2.c+17−7

This diff is large: its 57 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy3.c+10−3

This diff is large: its 28 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy_fentry.c+10−3

This diff is large: its 29 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_cgrp_storage.c+44−0

This diff is large: its 45 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_cgrp_storage_no_storage.c+26−0

This diff is large: its 27 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_cgrp_storage_owner.c+32−0

This diff is large: its 33 lines aren’t shown here.

tools/testing/selftests/bpf/progs/task_kfunc_failure.c+51−11

This diff is large: its 146 lines aren’t shown here.

tools/testing/selftests/bpf/progs/task_kfunc_success.c+6−7

This diff is large: its 40 lines aren’t shown here.

tools/testing/selftests/bpf/progs/task_local_storage.c+19−0

This diff is large: its 36 lines aren’t shown here.

tools/testing/selftests/bpf/progs/task_work_fail.c+3−3

This diff is large: its 27 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_bpf_nf_fail.c+4−4

This diff is large: its 36 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_fill_link_info.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_global_func3.c+50−2

This diff is large: its 60 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_kfunc_dynptr_param.c+3−8

This diff is large: its 31 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_kfunc_param_nullable.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_lirc_mode2_kern.c+2−2

This diff is large: its 12 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_lwt_ip_encap.c+145−10

This diff is large: its 176 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_ringbuf_map_key.c+3−8

This diff is large: its 28 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_signed_loader.c+18−0

This diff is large: its 19 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_signed_loader_data.c+20−0

This diff is large: its 21 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_signed_loader_lsm.c+30−0

This diff is large: its 31 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_signed_loader_map.c+28−0

This diff is large: its 29 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_sleepable_tracepoints.c+112−0

This diff is large: its 113 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_sleepable_tracepoints_fail.c+18−0

This diff is large: its 19 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_sockmap_msg_pop_data.c+27−0

This diff is large: its 28 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_tunnel_kern.c+6−7

This diff is large: its 50 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_vmlinux.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_wakeup_source.c+92−0

This diff is large: its 93 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_attach.c+39−0

This diff is large: its 40 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_attach_module.c+25−0

This diff is large: its 26 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_bench.c+12−0

This diff is large: its 13 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_check.c+214−0

This diff is large: its 215 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_fail.c+18−0

This diff is large: its 19 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_intersect_attach.c+41−0

This diff is large: its 42 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_rollback.c+43−0

This diff is large: its 44 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_session_attach.c+65−0

This diff is large: its 66 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_verifier.c+31−0

This diff is large: its 32 lines aren’t shown here.

tools/testing/selftests/bpf/progs/user_ringbuf_fail.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_arena.c+68−1

This diff is large: its 81 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_arena_globals1.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_arena_globals2.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_arena_large.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_bits_iter.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_bounds.c+120−10

This diff is large: its 175 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c+13−14

This diff is large: its 50 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_flow_keys.c+97−0

This diff is large: its 98 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_global_ptr_args.c+20−1

This diff is large: its 35 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_global_subprogs.c+19−1

This diff is large: its 39 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_jit_inline.c+3−1

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_ldsx.c+4−4

This diff is large: its 24 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_liveness_exp.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_lsm.c+9−0

This diff is large: its 14 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_map_in_map.c+40−0

This diff is large: its 54 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_map_ptr.c+32−4

This diff is large: its 58 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_may_goto_1.c+6−6

This diff is large: its 20 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_private_stack.c+25−5

This diff is large: its 108 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_ref_tracking.c+4−4

This diff is large: its 36 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_scalar_ids.c+12−13

This diff is large: its 50 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_sdiv.c+32−32

This diff is large: its 120 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_set_retval.c+107−0

This diff is large: its 108 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_sock.c+10−5

This diff is large: its 54 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_stack_arg.c+447−0

This diff is large: its 448 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c+185−0

This diff is large: its 186 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_subreg.c+4−2

This diff is large: its 20 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_tailcall_jit.c+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_vfs_reject.c+5−5

This diff is large: its 45 lines aren’t shown here.

tools/testing/selftests/bpf/progs/wakeup_source.h+22−0

This diff is large: its 23 lines aren’t shown here.

tools/testing/selftests/bpf/progs/wakeup_source_fail.c+76−0

This diff is large: its 77 lines aren’t shown here.

tools/testing/selftests/bpf/progs/wq_failures.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/xdp_flowtable.c+5−2

This diff is large: its 21 lines aren’t shown here.

tools/testing/selftests/bpf/progs/xdp_lb_bench.c+647−0

This diff is large: its 648 lines aren’t shown here.

tools/testing/selftests/bpf/progs/xdping_kern.c+0−183

This diff is large: its 184 lines aren’t shown here.

tools/testing/selftests/bpf/test_kmods/Makefile+27−3

This diff is large: its 42 lines aren’t shown here.

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c+79−0

This diff is large: its 93 lines aren’t shown here.

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h+33−0

This diff is large: its 54 lines aren’t shown here.

tools/testing/selftests/bpf/test_lirc_mode2_user.c+3−3

This diff is large: its 20 lines aren’t shown here.

tools/testing/selftests/bpf/test_loader.c+174−13

This diff is large: its 242 lines aren’t shown here.

tools/testing/selftests/bpf/test_maps.c+13−10

This diff is large: its 49 lines aren’t shown here.

tools/testing/selftests/bpf/test_progs.c+46−9

This diff is large: its 125 lines aren’t shown here.

tools/testing/selftests/bpf/test_progs.h+3−0

This diff is large: its 16 lines aren’t shown here.

tools/testing/selftests/bpf/test_xdping.sh+0−103

This diff is large: its 104 lines aren’t shown here.

tools/testing/selftests/bpf/testing_helpers.c+18−0

This diff is large: its 29 lines aren’t shown here.

tools/testing/selftests/bpf/testing_helpers.h+1−0

This diff is large: its 7 lines aren’t shown here.

tools/testing/selftests/bpf/trace_helpers.c+4−3

This diff is large: its 30 lines aren’t shown here.

tools/testing/selftests/bpf/trace_helpers.h+1−0

This diff is large: its 6 lines aren’t shown here.

tools/testing/selftests/bpf/uprobe_multi.c+3−1

This diff is large: its 10 lines aren’t shown here.

tools/testing/selftests/bpf/verifier/calls.c+55−31

This diff is large: its 153 lines aren’t shown here.

tools/testing/selftests/bpf/verifier/sleepable.c+15−2

This diff is large: its 31 lines aren’t shown here.

tools/testing/selftests/bpf/veristat.c+9−4

This diff is large: its 67 lines aren’t shown here.

tools/testing/selftests/bpf/vmtest.sh+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/xdp_lb_bench_common.h+112−0

This diff is large: its 113 lines aren’t shown here.

tools/testing/selftests/bpf/xdping.c+0−254

This diff is large: its 255 lines aren’t shown here.

tools/testing/selftests/bpf/xdping.h+0−13

This diff is large: its 14 lines aren’t shown here.

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t