History

Merge tag 'bpf-next-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Pull bpf updates from Alexei Starovoitov: "Major changes: - Recover from BPF arena page faults using a scratch page and add ptep_try_set() for lockless empty-slot installs on x86 and arm64. This allows BPF kfuncs to access arena pointers directly. The 'arena_direct_access' stable branch was created for this work and was pulled into sched-ext and bpf-next trees (Tejun Heo, Kumar Kartikeya Dwivedi) - Lift old restriction and support 6+ arguments in BPF programs and kfuncs on x86 and arm64 (Yonghong Song, Puranjay Mohan) Other features and fixes: - Add 24-bit BTF vlen and reclaim unused bits in the BTF UAPI to ease addition of new BTF kinds (Alan Maguire) - Raise the maximum BPF call chain depth from 8 to 16 frames (Alexei Starovoitov) - Refactor object relationship tracking in the verifier and fix a dynptr use-after-free bug (Amery Hung) - Harden the signed program loader and reject exclusive maps as inner maps (Daniel Borkmann) - Replace the verifier min/max bounds fields with a circular number (cnum) representation and improve 32->64 bit range refinements (Eduard Zingerman) - Introduce the arena library and runtime (libarena) with a buddy allocator, rbtree and SPMC queue data structures, ASAN support and a parallel test harness. Allow subprograms to return arena pointers and switch to a BTF type-tag based __arena annotation (Emil Tsalapatis) - Cache build IDs in the sleepable stackmap path and avoid faultable build ID reads under mm locks (Ihor Solodrai) - Introduce the tracing_multi link to attach a single BPF program to many kernel functions at once. Allow specifying the uprobe_multi target via FD (Jiri Olsa) - Extend the bpf_list family of kfuncs with bpf_list_add/del(), and bpf_list_is_first/is_last/empty() (Kaitao Cheng) - Extend the BPF syscall with common attributes support for prog_load, btf_load and map_create (Leon Hwang) - Wrap rhashtable as BPF map (Mykyta Yatsenko, Herbert Xu) - Add sleepable support for tracepoint programs and fix deadlocks in LRU map due to NMI reentry (Mykyta Yatsenko) - Fix OOB access in bpf_flow_keys, fix nullness analysis of inner arrays, enforce write checks for global subprograms (Nuoqi Gui) - Report the maximum combined stack depth and print a breakdown of instructions processed per subprogram (Paul Chaignon) - Add an XDP load-balancer benchmark and arm64 JIT support for stack arguments (Puranjay Mohan) - Add kfuncs to traverse over wakeup_sources (Samuel Wu) - Allow sleepable BPF programs to use LPM trie maps directly (Vlad Poenaru) - Many more fixes and cleanups across the verifier, BTF, sockmap, devmap, bpffs, security hooks, s390/riscv/loongarch JITs, rqspinlock, libbpf, bpftool, selftests" * tag 'bpf-next-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (336 commits) selftests/bpf: Work around llvm stack overflow in crypto progs selftests/bpf: add test for bpf_msg_pop_data() overflow bpf, sockmap: fix integer overflow in bpf_msg_pop_data() bounds check sockmap: Fix use-after-free in udp_bpf_recvmsg() bpf, sockmap: keep sk_msg copy state in sync bpf, sockmap: Fix wrong rsge offset in bpf_msg_push_data() bpf, sockmap: reject overflowing copy + len in bpf_msg_push_data() selftsets/bpf: Retry map update on helper_fill_hashmap() selftests/bpf: Add test for sleepable lsm_cgroup rejection selftests/bpf: Add test to verify the fix for bpf_setsockopt() helper bpf: Fix bpf_get/setsockopt to tos for ipv4-mapped ipv6 socket selftests/bpf: Avoid static LLVM linking for cross builds selftests/bpf: Use common CFLAGS for urandom_read selftests/bpf: Initialize operation name before use tools/bpf: build: Append extra cflags libbpf: Initialize CFLAGS before including Makefile.include bpftool: Append extra host flags bpftool: Avoid adding EXTRA_CFLAGS to HOST_CFLAGS bpftool: Pass host flags to bootstrap libbpf selftests/bpf: correct CONFIG_PPC64 macro name in comment ...

Linus Torvalds committed 5 months ago

Commit 9c87e612 parents b85966a e4287bf349 files · +27861 −4982Browse files

Unified · Split

349 files
  1. Documentation/bpf/btf.rst +2 −4
  2. Documentation/bpf/kfuncs.rst +34 −2
  3. Documentation/bpf/map_lru_hash_update.dot +42 −2
  4. MAINTAINERS +7 −27
  5. arch/arm64/include/asm/pgtable.h +28 −0
  6. arch/arm64/mm/fault.c +7 −3
  7. arch/arm64/net/bpf_jit_comp.c +120 −32
  8. arch/arm64/net/bpf_timed_may_goto.S +4 −4
  9. arch/loongarch/net/bpf_jit.c +26 −26
  10. arch/powerpc/net/bpf_jit_comp.c +27 −27
  11. arch/riscv/net/bpf_jit_comp64.c +35 −26
  12. arch/s390/net/bpf_jit_comp.c +59 −22
  13. arch/x86/include/asm/pgtable.h +12 −0
  14. arch/x86/mm/fault.c +8 −4
  15. arch/x86/net/bpf_jit_comp.c +172 −33
  16. drivers/base/power/power.h +7 −0
  17. drivers/base/power/wakeup.c +69 −2
  18. drivers/net/ethernet/netronome/nfp/bpf/verifier.c +4 −4
  19. fs/bpf_fs_kfuncs.c +1 −1
  20. fs/verity/measure.c +2 −2
  21. include/asm-generic/rqspinlock.h +11 −3
  22. include/linux/bpf-cgroup.h +3 −2
  23. include/linux/bpf.h +236 −46
  24. include/linux/bpf_defs.h +19 −0
  25. include/linux/bpf_lsm.h +6 −0
  26. include/linux/bpf_types.h +2 −0
  27. include/linux/bpf_verifier.h +209 −92
  28. include/linux/btf.h +4 −3
  29. include/linux/btf_ids.h +1 −0
  30. include/linux/cnum.h +82 −0
  31. include/linux/filter.h +25 −2
  32. include/linux/ftrace.h +9 −0
  33. include/linux/pgtable.h +43 −0
  34. include/linux/rhashtable.h +3 −1
  35. include/linux/syscalls.h +2 −1
  36. include/linux/trace_events.h +12 −0
  37. include/trace/bpf_probe.h +0 −2
  38. include/uapi/linux/bpf.h +31 −1
  39. include/uapi/linux/btf.h +14 −12
  40. kernel/bpf/Makefile +1 −1
  41. kernel/bpf/arena.c +266 −49
  42. kernel/bpf/arraymap.c +6 −8
  43. kernel/bpf/backtrack.c +66 −16
  44. kernel/bpf/bpf_lru_list.c +100 −65
  45. kernel/bpf/bpf_lru_list.h +19 −6
  46. kernel/bpf/bpf_lsm.c +20 −0
  47. kernel/bpf/bpf_struct_ops.c +50 −13
  48. kernel/bpf/btf.c +219 −99
  49. kernel/bpf/cgroup.c +44 −21
  50. kernel/bpf/cnum.c +120 −0
  51. kernel/bpf/cnum_defs.h +247 −0
  52. kernel/bpf/const_fold.c +8 −0
  53. kernel/bpf/core.c +24 −3
  54. kernel/bpf/devmap.c +19 −0
  55. kernel/bpf/fixups.c +24 −3
  56. kernel/bpf/hashtab.c +817 −23
  57. kernel/bpf/helpers.c +154 −50
  58. kernel/bpf/inode.c +4 −0
  59. kernel/bpf/liveness.c +139 −44
  60. kernel/bpf/log.c +104 −28
  61. kernel/bpf/lpm_trie.c +4 −4
  62. kernel/bpf/map_in_map.c +4 −1
  63. kernel/bpf/map_iter.c +6 −1
  64. kernel/bpf/stackmap.c +198 −17
  65. kernel/bpf/states.c +51 −16
  66. kernel/bpf/syscall.c +231 −81
  67. kernel/bpf/trampoline.c +524 −147
  68. kernel/bpf/verifier.c +1978 −2196
  69. kernel/events/core.c +9 −0
  70. kernel/trace/bpf_trace.c +306 −28
  71. kernel/trace/ftrace.c +29 −6
  72. kernel/trace/trace_syscalls.c +58 −52
  73. lib/rhashtable.c +102 −0
  74. lib/test_bpf.c +263 −100
  75. lib/test_rhashtable.c +75 −0
  76. net/bpf/bpf_dummy_struct_ops.c +7 −7
  77. net/bpf/test_run.c +53 −15
  78. net/core/filter.c +86 −31
  79. net/core/lwt_bpf.c +12 −0
  80. net/ipv4/udp_bpf.c +9 −0
  81. tools/bpf/Makefile +1 −0
  82. tools/bpf/bpftool/Documentation/bpftool-map.rst +1 −1
  83. tools/bpf/bpftool/Makefile +17 −3
  84. tools/bpf/bpftool/btf.c +6 −11
  85. tools/bpf/bpftool/btf_dumper.c +2 −2
  86. tools/bpf/bpftool/gen.c +10 −8
  87. tools/bpf/bpftool/map.c +1 −1
  88. tools/bpf/bpftool/net.c +2 −2
  89. tools/build/feature/test-bpf.c +2 −0
  90. tools/include/uapi/linux/bpf.h +32 −1
  91. tools/include/uapi/linux/btf.h +14 −12
  92. tools/lib/bpf/Makefile +9 −8
  93. tools/lib/bpf/bpf.c +66 −1
  94. tools/lib/bpf/bpf.h +23 −2
  95. tools/lib/bpf/btf.c +35 −19
  96. tools/lib/bpf/btf.h +1 −1
  97. tools/lib/bpf/btf_dump.c +12 −12
  98. tools/lib/bpf/features.c +8 −0
  99. tools/lib/bpf/gen_loader.c +67 −25
  100. tools/lib/bpf/libbpf.c +437 −36
  101. tools/lib/bpf/libbpf.h +15 −0
  102. tools/lib/bpf/libbpf.map +1 −0
  103. tools/lib/bpf/libbpf_internal.h +4 −0
  104. tools/lib/bpf/libbpf_probes.c +3 −0
  105. tools/lib/bpf/relo_core.c +8 −8
  106. tools/lib/bpf/skel_internal.h +5 −0
  107. tools/lib/bpf/strset.c +41 −21
  108. tools/lib/bpf/usdt.c +12 −4
  109. tools/testing/selftests/bpf/.gitignore +0 −1
  110. tools/testing/selftests/bpf/Makefile +201 −66
  111. tools/testing/selftests/bpf/README.rst +1 −1
  112. tools/testing/selftests/bpf/bench.c +21 −5
  113. tools/testing/selftests/bpf/bench.h +1 −0
  114. tools/testing/selftests/bpf/bench_bpf_timing.h +50 −0
  115. tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_full_update.c +32 −2
  116. tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_lookup.c +29 −2
  117. tools/testing/selftests/bpf/benchs/bench_bpf_nop.c +84 −0
  118. tools/testing/selftests/bpf/benchs/bench_bpf_timing.c +298 −0
  119. tools/testing/selftests/bpf/benchs/bench_htab_mem.c +33 −2
  120. tools/testing/selftests/bpf/benchs/bench_xdp_lb.c +1124 −0
  121. tools/testing/selftests/bpf/benchs/run_bench_xdp_lb.sh +79 −0
  122. tools/testing/selftests/bpf/bpf_arena_alloc.h +1 −1
  123. tools/testing/selftests/bpf/bpf_arena_htab.h +5 −6
  124. tools/testing/selftests/bpf/bpf_arena_list.h +1 −1
  125. tools/testing/selftests/bpf/bpf_arena_strsearch.h +3 −3
  126. tools/testing/selftests/bpf/bpf_experimental.h +1 −83
  127. tools/testing/selftests/bpf/bpf_kfuncs.h +4 −4
  128. tools/testing/selftests/bpf/config +2 −1
  129. tools/testing/selftests/bpf/default.profraw +0 −0
  130. tools/testing/selftests/bpf/jit_disasm_helpers.c +11 −2
  131. tools/testing/selftests/bpf/libarena/Makefile +92 −0
  132. tools/testing/selftests/bpf/libarena/include/bpf_arena_common.h +2 −3
  133. tools/testing/selftests/bpf/libarena/include/bpf_arena_spin_lock.h +9 −8
  134. tools/testing/selftests/bpf/libarena/include/bpf_atomic.h +3 −1
  135. tools/testing/selftests/bpf/libarena/include/bpf_may_goto.h +84 −0
  136. tools/testing/selftests/bpf/libarena/include/libarena/asan.h +101 −0
  137. tools/testing/selftests/bpf/libarena/include/libarena/buddy.h +81 −0
  138. tools/testing/selftests/bpf/libarena/include/libarena/common.h +93 −0
  139. tools/testing/selftests/bpf/libarena/include/libarena/rbtree.h +83 −0
  140. tools/testing/selftests/bpf/libarena/include/libarena/spmc.h +27 −0
  141. tools/testing/selftests/bpf/libarena/include/libarena/userspace.h +138 −0
  142. tools/testing/selftests/bpf/libarena/selftests/st_asan_buddy.bpf.c +258 −0
  143. tools/testing/selftests/bpf/libarena/selftests/st_asan_common.h +52 −0
  144. tools/testing/selftests/bpf/libarena/selftests/st_buddy.bpf.c +209 −0
  145. tools/testing/selftests/bpf/libarena/selftests/test_parallel_spmc.bpf.c +669 −0
  146. tools/testing/selftests/bpf/libarena/selftests/test_progs_compat.h +15 −0
  147. tools/testing/selftests/bpf/libarena/selftests/test_rbtree.bpf.c +968 −0
  148. tools/testing/selftests/bpf/libarena/selftests/test_spmc.bpf.c +194 −0
  149. tools/testing/selftests/bpf/libarena/src/asan.bpf.c +553 −0
  150. tools/testing/selftests/bpf/libarena/src/buddy.bpf.c +903 −0
  151. tools/testing/selftests/bpf/libarena/src/common.bpf.c +52 −0
  152. tools/testing/selftests/bpf/libarena/src/rbtree.bpf.c +1047 −0
  153. tools/testing/selftests/bpf/libarena/src/spmc.bpf.c +234 −0
  154. tools/testing/selftests/bpf/prog_tests/arena_direct_value.c +73 −0
  155. tools/testing/selftests/bpf/prog_tests/arena_spin_lock.c +0 −7
  156. tools/testing/selftests/bpf/prog_tests/bpf_attr_size.c +124 −0
  157. tools/testing/selftests/bpf/prog_tests/bpf_cookie.c +11 −6
  158. tools/testing/selftests/bpf/prog_tests/bpf_nf.c +11 −11
  159. tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c +8 −0
  160. tools/testing/selftests/bpf/prog_tests/btf.c +41 −4
  161. tools/testing/selftests/bpf/prog_tests/btf_dedup_split.c +33 −18
  162. tools/testing/selftests/bpf/prog_tests/btf_dump.c +2 −2
  163. tools/testing/selftests/bpf/prog_tests/cb_refs.c +2 −2
  164. tools/testing/selftests/bpf/prog_tests/cgrp_local_storage.c +13 −2
  165. tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c +8 −9
  166. tools/testing/selftests/bpf/prog_tests/exceptions.c +7 −0
  167. tools/testing/selftests/bpf/prog_tests/file_reader.c +10 −12
  168. tools/testing/selftests/bpf/prog_tests/fill_link_info.c +1 −1
  169. tools/testing/selftests/bpf/prog_tests/htab_update.c +2 −2
  170. tools/testing/selftests/bpf/prog_tests/iters.c +0 −2
  171. tools/testing/selftests/bpf/prog_tests/kfunc_call.c +1 −1
  172. tools/testing/selftests/bpf/prog_tests/libarena.c +253 −0
  173. tools/testing/selftests/bpf/prog_tests/libarena_asan.c +93 −0
  174. tools/testing/selftests/bpf/prog_tests/linked_list.c +19 −18
  175. tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c +243 −0
  176. tools/testing/selftests/bpf/prog_tests/lsm_cgroup.c +79 −0
  177. tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c +145 −0
  178. tools/testing/selftests/bpf/prog_tests/map_excl.c +122 −0
  179. tools/testing/selftests/bpf/prog_tests/map_init.c +192 −0
  180. tools/testing/selftests/bpf/prog_tests/map_kptr.c +57 −9
  181. tools/testing/selftests/bpf/prog_tests/refcounted_kptr.c +6 −2
  182. tools/testing/selftests/bpf/prog_tests/reg_bounds.c +75 −15
  183. tools/testing/selftests/bpf/prog_tests/rhash.c +183 −0
  184. tools/testing/selftests/bpf/prog_tests/setget_sockopt.c +78 −0
  185. tools/testing/selftests/bpf/prog_tests/signed_loader.c +1135 −0
  186. tools/testing/selftests/bpf/prog_tests/sleepable_tracepoints.c +142 −0
  187. tools/testing/selftests/bpf/prog_tests/sockmap_basic.c +48 −0
  188. tools/testing/selftests/bpf/prog_tests/spin_lock.c +2 −2
  189. tools/testing/selftests/bpf/prog_tests/stack_arg.c +139 −0
  190. tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c +10 −0
  191. tools/testing/selftests/bpf/prog_tests/stack_arg_precision.c +10 −0
  192. tools/testing/selftests/bpf/prog_tests/tailcalls.c +186 −0
  193. tools/testing/selftests/bpf/prog_tests/task_kfunc.c +41 −1
  194. tools/testing/selftests/bpf/prog_tests/task_local_storage.c +1 −0
  195. tools/testing/selftests/bpf/prog_tests/test_lsm.c +0 −22
  196. tools/testing/selftests/bpf/prog_tests/test_xdp_veth.c +163 −3
  197. tools/testing/selftests/bpf/prog_tests/tracing_multi.c +960 −0
  198. tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c +93 −1
  199. tools/testing/selftests/bpf/prog_tests/verifier.c +8 −0
  200. tools/testing/selftests/bpf/prog_tests/verifier_log.c +1 −0
  201. tools/testing/selftests/bpf/prog_tests/vmlinux.c +43 −2
  202. tools/testing/selftests/bpf/prog_tests/wakeup_source.c +118 −0
  203. tools/testing/selftests/bpf/progs/arena_atomics.c +1 −1
  204. tools/testing/selftests/bpf/progs/arena_spin_lock.c +2 −1
  205. tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h +69 −0
  206. tools/testing/selftests/bpf/progs/bpf_iter_bpf_rhash_map.c +34 −0
  207. tools/testing/selftests/bpf/progs/bpf_iter_task_vmas.c +1 −1
  208. tools/testing/selftests/bpf/progs/bpf_misc.h +1 −0
  209. tools/testing/selftests/bpf/progs/bpf_nop_bench.c +14 −0
  210. tools/testing/selftests/bpf/progs/bpf_qdisc_dynptr_use_after_invalidate_clone.c +74 −0
  211. tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr.c +68 −0
  212. tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c +74 −0
  213. tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_slice.c +70 −0
  214. tools/testing/selftests/bpf/progs/bpf_qdisc_fq.c +3 −8
  215. tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c +24 −0
  216. tools/testing/selftests/bpf/progs/btf__verifier_stack_arg_order.c +49 −0
  217. tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c +9 −9
  218. tools/testing/selftests/bpf/progs/cgrp_ls_sleepable.c +18 −0
  219. tools/testing/selftests/bpf/progs/compute_live_registers.c +1 −1
  220. tools/testing/selftests/bpf/progs/cpumask_failure.c +5 −5
  221. tools/testing/selftests/bpf/progs/cpumask_success.c +1 −1
  222. tools/testing/selftests/bpf/progs/crypto_bench.c +14 −7
  223. tools/testing/selftests/bpf/progs/crypto_sanity.c +14 −7
  224. tools/testing/selftests/bpf/progs/dynptr_fail.c +58 −16
  225. tools/testing/selftests/bpf/progs/dynptr_success.c +3 −3
  226. tools/testing/selftests/bpf/progs/exceptions.c +114 −0
  227. tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c +11 −2
  228. tools/testing/selftests/bpf/progs/file_reader.c +1 −1
  229. tools/testing/selftests/bpf/progs/file_reader_fail.c +62 −2
  230. tools/testing/selftests/bpf/progs/htab_update.c +2 −2
  231. tools/testing/selftests/bpf/progs/irq.c +2 −2
  232. tools/testing/selftests/bpf/progs/iters.c +3 −3
  233. tools/testing/selftests/bpf/progs/iters_state_safety.c +9 −9
  234. tools/testing/selftests/bpf/progs/iters_testmod.c +2 −2
  235. tools/testing/selftests/bpf/progs/iters_testmod_seq.c +8 −8
  236. tools/testing/selftests/bpf/progs/linked_list.c +71 −0
  237. tools/testing/selftests/bpf/progs/lpm_trie_bench.c +1 −1
  238. tools/testing/selftests/bpf/progs/lru_lock_nmi.c +33 −0
  239. tools/testing/selftests/bpf/progs/lsm_cgroup.c +30 −0
  240. tools/testing/selftests/bpf/progs/map_kptr.c +86 −3
  241. tools/testing/selftests/bpf/progs/map_kptr_fail.c +2 −2
  242. tools/testing/selftests/bpf/progs/percpu_alloc_fail.c +2 −2
  243. tools/testing/selftests/bpf/progs/rbtree_fail.c +3 −3
  244. tools/testing/selftests/bpf/progs/refcounted_kptr.c +440 −1
  245. tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c +1 −1
  246. tools/testing/selftests/bpf/progs/rhash.c +248 −0
  247. tools/testing/selftests/bpf/progs/setget_sockopt.c +23 −0
  248. tools/testing/selftests/bpf/progs/sk_bypass_prot_mem.c +2 −0
  249. tools/testing/selftests/bpf/progs/stack_arg.c +273 −0
  250. tools/testing/selftests/bpf/progs/stack_arg_fail.c +114 −0
  251. tools/testing/selftests/bpf/progs/stack_arg_kfunc.c +166 −0
  252. tools/testing/selftests/bpf/progs/stack_arg_precision.c +135 −0
  253. tools/testing/selftests/bpf/progs/stream.c +1 −1
  254. tools/testing/selftests/bpf/progs/stream_fail.c +1 −1
  255. tools/testing/selftests/bpf/progs/tailcall_bpf2bpf2.c +4 −1
  256. tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy1.c +9 −4
  257. tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy2.c +17 −7
  258. tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy3.c +10 −3
  259. tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy_fentry.c +10 −3
  260. tools/testing/selftests/bpf/progs/tailcall_cgrp_storage.c +44 −0
  261. tools/testing/selftests/bpf/progs/tailcall_cgrp_storage_no_storage.c +26 −0
  262. tools/testing/selftests/bpf/progs/tailcall_cgrp_storage_owner.c +32 −0
  263. tools/testing/selftests/bpf/progs/task_kfunc_failure.c +51 −11
  264. tools/testing/selftests/bpf/progs/task_kfunc_success.c +6 −7
  265. tools/testing/selftests/bpf/progs/task_local_storage.c +19 −0
  266. tools/testing/selftests/bpf/progs/task_work_fail.c +3 −3
  267. tools/testing/selftests/bpf/progs/test_bpf_nf_fail.c +4 −4
  268. tools/testing/selftests/bpf/progs/test_fill_link_info.c +1 −1
  269. tools/testing/selftests/bpf/progs/test_global_func3.c +50 −2
  270. tools/testing/selftests/bpf/progs/test_kfunc_dynptr_param.c +3 −8
  271. tools/testing/selftests/bpf/progs/test_kfunc_param_nullable.c +1 −1
  272. tools/testing/selftests/bpf/progs/test_lirc_mode2_kern.c +2 −2
  273. tools/testing/selftests/bpf/progs/test_lwt_ip_encap.c +145 −10
  274. tools/testing/selftests/bpf/progs/test_ringbuf_map_key.c +3 −8
  275. tools/testing/selftests/bpf/progs/test_signed_loader.c +18 −0
  276. tools/testing/selftests/bpf/progs/test_signed_loader_data.c +20 −0
  277. tools/testing/selftests/bpf/progs/test_signed_loader_lsm.c +30 −0
  278. tools/testing/selftests/bpf/progs/test_signed_loader_map.c +28 −0
  279. tools/testing/selftests/bpf/progs/test_sleepable_tracepoints.c +112 −0
  280. tools/testing/selftests/bpf/progs/test_sleepable_tracepoints_fail.c +18 −0
  281. tools/testing/selftests/bpf/progs/test_sockmap_msg_pop_data.c +27 −0
  282. tools/testing/selftests/bpf/progs/test_tunnel_kern.c +6 −7
  283. tools/testing/selftests/bpf/progs/test_vmlinux.c +2 −2
  284. tools/testing/selftests/bpf/progs/test_wakeup_source.c +92 −0
  285. tools/testing/selftests/bpf/progs/tracing_multi_attach.c +39 −0
  286. tools/testing/selftests/bpf/progs/tracing_multi_attach_module.c +25 −0
  287. tools/testing/selftests/bpf/progs/tracing_multi_bench.c +12 −0
  288. tools/testing/selftests/bpf/progs/tracing_multi_check.c +214 −0
  289. tools/testing/selftests/bpf/progs/tracing_multi_fail.c +18 −0
  290. tools/testing/selftests/bpf/progs/tracing_multi_intersect_attach.c +41 −0
  291. tools/testing/selftests/bpf/progs/tracing_multi_rollback.c +43 −0
  292. tools/testing/selftests/bpf/progs/tracing_multi_session_attach.c +65 −0
  293. tools/testing/selftests/bpf/progs/tracing_multi_verifier.c +31 −0
  294. tools/testing/selftests/bpf/progs/user_ringbuf_fail.c +2 −2
  295. tools/testing/selftests/bpf/progs/verifier_arena.c +68 −1
  296. tools/testing/selftests/bpf/progs/verifier_arena_globals1.c +1 −1
  297. tools/testing/selftests/bpf/progs/verifier_arena_globals2.c +1 −1
  298. tools/testing/selftests/bpf/progs/verifier_arena_large.c +1 −1
  299. tools/testing/selftests/bpf/progs/verifier_bits_iter.c +2 −2
  300. tools/testing/selftests/bpf/progs/verifier_bounds.c +120 −10
  301. tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c +13 −14
  302. tools/testing/selftests/bpf/progs/verifier_flow_keys.c +97 −0
  303. tools/testing/selftests/bpf/progs/verifier_global_ptr_args.c +20 −1
  304. tools/testing/selftests/bpf/progs/verifier_global_subprogs.c +19 −1
  305. tools/testing/selftests/bpf/progs/verifier_jit_inline.c +3 −1
  306. tools/testing/selftests/bpf/progs/verifier_ldsx.c +4 −4
  307. tools/testing/selftests/bpf/progs/verifier_liveness_exp.c +1 −1
  308. tools/testing/selftests/bpf/progs/verifier_lsm.c +9 −0
  309. tools/testing/selftests/bpf/progs/verifier_map_in_map.c +40 −0
  310. tools/testing/selftests/bpf/progs/verifier_map_ptr.c +32 −4
  311. tools/testing/selftests/bpf/progs/verifier_may_goto_1.c +6 −6
  312. tools/testing/selftests/bpf/progs/verifier_private_stack.c +25 −5
  313. tools/testing/selftests/bpf/progs/verifier_ref_tracking.c +4 −4
  314. tools/testing/selftests/bpf/progs/verifier_scalar_ids.c +12 −13
  315. tools/testing/selftests/bpf/progs/verifier_sdiv.c +32 −32
  316. tools/testing/selftests/bpf/progs/verifier_set_retval.c +107 −0
  317. tools/testing/selftests/bpf/progs/verifier_sock.c +10 −5
  318. tools/testing/selftests/bpf/progs/verifier_stack_arg.c +447 −0
  319. tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c +185 −0
  320. tools/testing/selftests/bpf/progs/verifier_subreg.c +4 −2
  321. tools/testing/selftests/bpf/progs/verifier_tailcall_jit.c +1 −0
  322. tools/testing/selftests/bpf/progs/verifier_vfs_reject.c +5 −5
  323. tools/testing/selftests/bpf/progs/wakeup_source.h +22 −0
  324. tools/testing/selftests/bpf/progs/wakeup_source_fail.c +76 −0
  325. tools/testing/selftests/bpf/progs/wq_failures.c +1 −1
  326. tools/testing/selftests/bpf/progs/xdp_flowtable.c +5 −2
  327. tools/testing/selftests/bpf/progs/xdp_lb_bench.c +647 −0
  328. tools/testing/selftests/bpf/progs/xdping_kern.c +0 −183
  329. tools/testing/selftests/bpf/test_kmods/Makefile +27 −3
  330. tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +79 −0
  331. tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +33 −0
  332. tools/testing/selftests/bpf/test_lirc_mode2_user.c +3 −3
  333. tools/testing/selftests/bpf/test_loader.c +174 −13
  334. tools/testing/selftests/bpf/test_maps.c +13 −10
  335. tools/testing/selftests/bpf/test_progs.c +46 −9
  336. tools/testing/selftests/bpf/test_progs.h +3 −0
  337. tools/testing/selftests/bpf/test_xdping.sh +0 −103
  338. tools/testing/selftests/bpf/testing_helpers.c +18 −0
  339. tools/testing/selftests/bpf/testing_helpers.h +1 −0
  340. tools/testing/selftests/bpf/trace_helpers.c +4 −3
  341. tools/testing/selftests/bpf/trace_helpers.h +1 −0
  342. tools/testing/selftests/bpf/uprobe_multi.c +3 −1
  343. tools/testing/selftests/bpf/verifier/calls.c +55 −31
  344. tools/testing/selftests/bpf/verifier/sleepable.c +15 −2
  345. tools/testing/selftests/bpf/veristat.c +9 −4
  346. tools/testing/selftests/bpf/vmtest.sh +1 −1
  347. tools/testing/selftests/bpf/xdp_lb_bench_common.h +112 −0
  348. tools/testing/selftests/bpf/xdping.c +0 −254
  349. tools/testing/selftests/bpf/xdping.h +0 −13
Documentation/bpf/btf.rst+2−4
@@ -97,10 +97,8 @@ Each type contains the following common data:: · 96 unmodified lines
97 struct btf_type {97 struct btf_type {
98 __u32 name_off;98 __u32 name_off;
99 /* "info" bits arrangement99 /* "info" bits arrangement
100 * bits 0-15: vlen (e.g. # of struct's members)100 * bits 0-23: vlen (e.g. # of struct's members)
101 * bits 16-23: unused101 * bits 24-30: kind (e.g. int, ptr, array...etc)
102 * bits 24-28: kind (e.g. int, ptr, array...etc)
103 * bits 29-30: unused
104 * bit 31: kind_flag, currently used by102 * bit 31: kind_flag, currently used by
105 * struct, union, enum, fwd, enum64,103 * struct, union, enum, fwd, enum64,
106 * decl_tag and type_tag104 * decl_tag and type_tag
Documentation/bpf/kfuncs.rst+34−2
@@ -207,8 +207,26 @@ Here, the buffer may be NULL. If the buffer is not NULL, it must be at least · 206 unmodified lines
207buffer__szk bytes in size. The kfunc is responsible for checking if the buffer207buffer__szk bytes in size. The kfunc is responsible for checking if the buffer
208is NULL before using it.208is NULL before using it.
209 209
2102.3.5 __str Annotation2102.3.5 __nonown_allowed Annotation
211----------------------------211---------------------------------
212
213This annotation is used to indicate that the parameter may be a non-owning reference.
214
215An example is given below::
216
217 __bpf_kfunc int bpf_list_add(..., struct bpf_list_node
218 *prev__nonown_allowed, ...)
219 {
220 ...
221 }
222
223For the ``prev__nonown_allowed`` parameter (resolved as ``KF_ARG_PTR_TO_LIST_NODE``),
224suffix ``__nonown_allowed`` retains the usual owning-pointer rules and also
225permits a non-owning reference with no ref_obj_id (e.g. the return value of
226bpf_list_front() / bpf_list_back()).
227
2282.3.6 __str Annotation
229----------------------
212This annotation is used to indicate that the argument is a constant string.230This annotation is used to indicate that the argument is a constant string.
213 231
214An example is given below::232An example is given below::
@@ -462,6 +480,20 @@ In order to accommodate such requirements, the verifier will enforce strict · 247 unmodified lines
462PTR_TO_BTF_ID type matching if two types have the exact same name, with one480PTR_TO_BTF_ID type matching if two types have the exact same name, with one
463being suffixed with ``___init``.481being suffixed with ``___init``.
464 482
4832.8 Accessing arena memory through kfunc arguments
484--------------------------------------------------
485
486A read or write at any address inside an arena does not oops the kernel.
487Unallocated arena pages are lazily backed by a scratch page and the
488access is reported through the program's BPF stream as an error. Only
489the BPF program's correctness is affected; the kernel itself remains
490intact.
491
492The arena is followed by a ``GUARD_SZ / 2`` (32 KiB) guard region that
493is also covered by this recovery. A kfunc handed an arena pointer may
494therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
495against the arena. Larger accesses must verify the range explicitly.
496
465.. _BPF_kfunc_lifecycle_expectations:497.. _BPF_kfunc_lifecycle_expectations:
466 498
4673. kfunc lifecycle expectations4993. kfunc lifecycle expectations
Documentation/bpf/map_lru_hash_update.dot+42−2
@@ -21,10 +21,18 @@ digraph { · 20 unmodified lines
21 // names that initiate the corresponding logic in kernel/bpf/bpf_lru_list.c.21 // names that initiate the corresponding logic in kernel/bpf/bpf_lru_list.c.
22 // Number suffixes and errno suffixes handle subsections of the corresponding22 // Number suffixes and errno suffixes handle subsections of the corresponding
23 // logic in the function as of the writing of this dot.23 // logic in the function as of the writing of this dot.
24 //
25 // All LRU locks are rqspinlock_t. Every acquire can fail (AA self-deadlock
26 // or contention timeout); on failure the corresponding helper returns NULL
27 // and the caller propagates -ENOMEM. The "rqspinlock acquire failed"
28 // terminal below is reached via the dashed arrows from each acquire site.
29
30 rqspinlock_failed [shape=rectangle,
31 label="Any LRU rqspinlock\nacquire fails\n(AA or timeout)"]
24 32
25 // cf. __local_list_pop_free() / bpf_percpu_lru_pop_free()33 // cf. __local_list_pop_free() / bpf_percpu_lru_pop_free()
26 local_freelist_check [shape=diamond,fillcolor=1,34 local_freelist_check [shape=diamond,fillcolor=1,
27 label="Local freelist\nnode available?"];35 label="Local freelist\nnode available?\n(lockless free_llist)"];
28 use_local_node [shape=rectangle,36 use_local_node [shape=rectangle,
29 label="Use node owned\nby this CPU"]37 label="Use node owned\nby this CPU"]
30 38
@@ -82,6 +90,15 @@ digraph { · 51 unmodified lines
82 // fn__local_list_pop_pending()90 // fn__local_list_pop_pending()
83 }91 }
84 92
93 // Post-steal: re-acquire local loc_l->lock to insert the stolen node into
94 // the local pending list. If the acquire fails, the stolen node is published
95 // to the lockless local free_llist so the next pop on this CPU picks it up
96 // instead of orphaning it.
97 post_steal_lock [shape=diamond,fillcolor=1,
98 label="Acquire local\nloc_l->lock\nto add pending"]
99 post_steal_to_free_llist [shape=rectangle,
100 label="Publish stolen node to\nlocal free_llist (lockless)"]
101
85 fn_bpf_lru_list_pop_free_to_local2 [shape=rectangle,102 fn_bpf_lru_list_pop_free_to_local2 [shape=rectangle,
86 label="Use node that was\nnot recently referenced"]103 label="Use node that was\nnot recently referenced"]
87 local_freelist_check4 [shape=rectangle,104 local_freelist_check4 [shape=rectangle,
@@ -97,10 +114,19 @@ digraph { · 9 unmodified lines
97 fn_htab_lru_map_update_elem_ENOENT [shape=oval,label="return -ENOENT"]114 fn_htab_lru_map_update_elem_ENOENT [shape=oval,label="return -ENOENT"]
98 115
99 begin -> local_freelist_check116 begin -> local_freelist_check
117 // The initial per-CPU lock (loc_l->lock for common, l->lock for percpu) is
118 // acquired before the local freelist check; rqspinlock failure here exits
119 // directly to -ENOMEM (no recovery needed: nothing was removed yet).
120 local_freelist_check -> rqspinlock_failed [style=dashed,
121 xlabel="acquire fails"]
100 local_freelist_check -> use_local_node [xlabel="Y"]122 local_freelist_check -> use_local_node [xlabel="Y"]
101 local_freelist_check -> common_lru_check [xlabel="N"]123 local_freelist_check -> common_lru_check [xlabel="N"]
102 common_lru_check -> fn_bpf_lru_list_pop_free_to_local [xlabel="Y"]124 common_lru_check -> fn_bpf_lru_list_pop_free_to_local [xlabel="Y"]
103 common_lru_check -> fn___bpf_lru_list_shrink_inactive [xlabel="N"]125 common_lru_check -> fn___bpf_lru_list_shrink_inactive [xlabel="N"]
126 // Global lru_list lock acquire failure in pop_free_to_local: skip refill,
127 // fall through to the steal path. Not ENOMEM by itself.
128 fn_bpf_lru_list_pop_free_to_local -> common_lru_check2 [style=dashed,
129 xlabel="global lru_lock\nacquire fails"]
104 fn_bpf_lru_list_pop_free_to_local -> fn___bpf_lru_node_move_to_free130 fn_bpf_lru_list_pop_free_to_local -> fn___bpf_lru_node_move_to_free
105 fn___bpf_lru_node_move_to_free ->131 fn___bpf_lru_node_move_to_free ->
106 fn_bpf_lru_list_pop_free_to_local2 [xlabel="Y"]132 fn_bpf_lru_list_pop_free_to_local2 [xlabel="Y"]
@@ -120,13 +146,27 @@ digraph { · 13 unmodified lines
120 local_freelist_check6 -> local_freelist_check7146 local_freelist_check6 -> local_freelist_check7
121 local_freelist_check7 -> fn_htab_lru_map_update_elem147 local_freelist_check7 -> fn_htab_lru_map_update_elem
122 148
123 fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem3 [xlabel = "Y"]149 // Steal-loop victim lock failure is silent: treat as "no node found here"
150 // and continue to next CPU; same edge as the existing "N" path.
151 local_freelist_check5 -> fn_htab_lru_map_update_elem2 [style=dashed,
152 xlabel="victim's lock\nfails: skip"]
153 // After a successful steal, re-acquire the local loc_l->lock. On failure
154 // the stolen node is published to free_llist (recovered, not orphaned)
155 // and the update returns -ENOMEM.
156 fn_htab_lru_map_update_elem -> post_steal_lock [xlabel = "Y"]
157 post_steal_lock -> fn_htab_lru_map_update_elem3 [xlabel = "OK"]
158 post_steal_lock -> post_steal_to_free_llist [style=dashed,
159 xlabel="loc_l->lock\nacquire fails"]
160 post_steal_to_free_llist -> fn_htab_lru_map_update_elem_ENOMEM
124 fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem2 [xlabel = "N"]161 fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem2 [xlabel = "N"]
125 fn_htab_lru_map_update_elem2 ->162 fn_htab_lru_map_update_elem2 ->
126 fn_htab_lru_map_update_elem_ENOMEM [xlabel = "Y"]163 fn_htab_lru_map_update_elem_ENOMEM [xlabel = "Y"]
127 fn_htab_lru_map_update_elem2 -> local_freelist_check5 [xlabel = "N"]164 fn_htab_lru_map_update_elem2 -> local_freelist_check5 [xlabel = "N"]
128 fn_htab_lru_map_update_elem3 -> fn_htab_lru_map_update_elem4165 fn_htab_lru_map_update_elem3 -> fn_htab_lru_map_update_elem4
129 166
167 // Shared rqspinlock-failure terminal collapses to the same -ENOMEM exit.
168 rqspinlock_failed -> fn_htab_lru_map_update_elem_ENOMEM
169
130 use_local_node -> fn_htab_lru_map_update_elem4170 use_local_node -> fn_htab_lru_map_update_elem4
131 fn_bpf_lru_list_pop_free_to_local2 -> fn_htab_lru_map_update_elem4171 fn_bpf_lru_list_pop_free_to_local2 -> fn_htab_lru_map_update_elem4
132 local_freelist_check4 -> fn_htab_lru_map_update_elem4172 local_freelist_check4 -> fn_htab_lru_map_update_elem4
MAINTAINERS+7−27
@@ -4797,13 +4797,6 @@ S: Supported · 4796 unmodified lines
4797F: arch/x86/net/4797F: arch/x86/net/
4798X: arch/x86/net/bpf_jit_comp32.c4798X: arch/x86/net/bpf_jit_comp32.c
4799 4799
4800BPF [BTF]
4801M: Martin KaFai Lau <martin.lau@linux.dev>
4802L: bpf@vger.kernel.org
4803S: Maintained
4804F: include/linux/btf*
4805F: kernel/bpf/btf.c
4806
4807BPF [CORE]4800BPF [CORE]
4808M: Alexei Starovoitov <ast@kernel.org>4801M: Alexei Starovoitov <ast@kernel.org>
4809M: Daniel Borkmann <daniel@iogearbox.net>4802M: Daniel Borkmann <daniel@iogearbox.net>
@@ -4840,12 +4833,13 @@ BPF [GENERAL] (Safe Dynamic Programs and Tools) · 30 unmodified lines
4840M: Alexei Starovoitov <ast@kernel.org>4833M: Alexei Starovoitov <ast@kernel.org>
4841M: Daniel Borkmann <daniel@iogearbox.net>4834M: Daniel Borkmann <daniel@iogearbox.net>
4842M: Andrii Nakryiko <andrii@kernel.org>4835M: Andrii Nakryiko <andrii@kernel.org>
4843R: Martin KaFai Lau <martin.lau@linux.dev>
4844M: Eduard Zingerman <eddyz87@gmail.com>4836M: Eduard Zingerman <eddyz87@gmail.com>
4845M: Kumar Kartikeya Dwivedi <memxor@gmail.com>4837M: Kumar Kartikeya Dwivedi <memxor@gmail.com>
4838R: Martin KaFai Lau <martin.lau@linux.dev>
4846R: Song Liu <song@kernel.org>4839R: Song Liu <song@kernel.org>
4847R: Yonghong Song <yonghong.song@linux.dev>4840R: Yonghong Song <yonghong.song@linux.dev>
4848R: Jiri Olsa <jolsa@kernel.org>4841R: Jiri Olsa <jolsa@kernel.org>
4842R: Emil Tsalapatis <emil@etsalapatis.com>
4849L: bpf@vger.kernel.org4843L: bpf@vger.kernel.org
4850S: Supported4844S: Supported
4851W: https://bpf.io/4845W: https://bpf.io/
@@ -4855,7 +4849,9 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git · 3 unmodified lines
4855F: Documentation/bpf/4849F: Documentation/bpf/
4856F: Documentation/networking/filter.rst4850F: Documentation/networking/filter.rst
4857F: Documentation/userspace-api/ebpf/4851F: Documentation/userspace-api/ebpf/
4852F: arch/*/include/asm/rqspinlock.h
4858F: arch/*/net/*4853F: arch/*/net/*
4854F: include/asm-generic/rqspinlock.h
4859F: include/linux/bpf*4855F: include/linux/bpf*
4860F: include/linux/btf*4856F: include/linux/btf*
4861F: include/linux/buildid.h4857F: include/linux/buildid.h
@@ -4867,17 +4863,15 @@ F: include/uapi/linux/filter.h · 5 unmodified lines
4867F: kernel/bpf/4863F: kernel/bpf/
4868F: kernel/trace/bpf_trace.c4864F: kernel/trace/bpf_trace.c
4869F: lib/buildid.c4865F: lib/buildid.c
4870F: arch/*/include/asm/rqspinlock.h
4871F: include/asm-generic/rqspinlock.h
4872F: lib/test_bpf.c4866F: lib/test_bpf.c
4873F: net/bpf/4867F: net/bpf/
4874F: net/core/filter.c4868F: net/core/filter.c
4875F: net/sched/act_bpf.c4869F: net/sched/act_bpf.c
4876F: net/sched/cls_bpf.c4870F: net/sched/cls_bpf.c
4877F: samples/bpf/4871F: samples/bpf/
4872F: scripts/Makefile.btf
4878F: scripts/bpf_doc.py4873F: scripts/bpf_doc.py
4879F: scripts/gen-btf.sh4874F: scripts/gen-btf.sh
4880F: scripts/Makefile.btf
4881F: scripts/pahole-version.sh4875F: scripts/pahole-version.sh
4882F: tools/bpf/4876F: tools/bpf/
4883F: tools/lib/bpf/4877F: tools/lib/bpf/
@@ -4892,6 +4886,7 @@ F: kernel/bpf/*iter.c · 8 unmodified lines
4892BPF [L7 FRAMEWORK] (sockmap)4886BPF [L7 FRAMEWORK] (sockmap)
4893M: John Fastabend <john.fastabend@gmail.com>4887M: John Fastabend <john.fastabend@gmail.com>
4894M: Jakub Sitnicki <jakub@cloudflare.com>4888M: Jakub Sitnicki <jakub@cloudflare.com>
4889M: Jiayuan Chen <jiayuan.chen@linux.dev>
4895L: netdev@vger.kernel.org4890L: netdev@vger.kernel.org
4896L: bpf@vger.kernel.org4891L: bpf@vger.kernel.org
4897S: Maintained4892S: Maintained
@@ -4935,18 +4930,11 @@ F: tools/testing/selftests/bpf/prog_tests/tc_netkit.c · 37 unmodified lines
4935F: tools/testing/selftests/drivers/net/hw/nk_qlease.py4930F: tools/testing/selftests/drivers/net/hw/nk_qlease.py
4936F: tools/testing/selftests/net/nk_qlease.py4931F: tools/testing/selftests/net/nk_qlease.py
4937 4932
4938BPF [NETWORKING] (struct_ops, reuseport)
4939M: Martin KaFai Lau <martin.lau@linux.dev>
4940L: bpf@vger.kernel.org
4941L: netdev@vger.kernel.org
4942S: Maintained
4943F: kernel/bpf/bpf_struct*
4944
4945BPF [NETWORKING] (tcx & tc BPF, sock_addr)4933BPF [NETWORKING] (tcx & tc BPF, sock_addr)
4946M: Martin KaFai Lau <martin.lau@linux.dev>
4947M: Daniel Borkmann <daniel@iogearbox.net>4934M: Daniel Borkmann <daniel@iogearbox.net>
4948R: John Fastabend <john.fastabend@gmail.com>4935R: John Fastabend <john.fastabend@gmail.com>
4949R: Stanislav Fomichev <sdf@fomichev.me>4936R: Stanislav Fomichev <sdf@fomichev.me>
4937R: Martin KaFai Lau <martin.lau@linux.dev>
4950L: bpf@vger.kernel.org4938L: bpf@vger.kernel.org
4951L: netdev@vger.kernel.org4939L: netdev@vger.kernel.org
4952S: Maintained4940S: Maintained
@@ -4981,14 +4969,6 @@ L: bpf@vger.kernel.org · 28 unmodified lines
4981S: Maintained4969S: Maintained
4982F: tools/testing/selftests/bpf/4970F: tools/testing/selftests/bpf/
4983 4971
4984BPF [STORAGE & CGROUPS]
4985M: Martin KaFai Lau <martin.lau@linux.dev>
4986L: bpf@vger.kernel.org
4987S: Maintained
4988F: kernel/bpf/*storage.c
4989F: kernel/bpf/bpf_lru*
4990F: kernel/bpf/cgroup.c
4991
4992BPF [TOOLING] (bpftool)4972BPF [TOOLING] (bpftool)
4993M: Quentin Monnet <qmo@kernel.org>4973M: Quentin Monnet <qmo@kernel.org>
4994L: bpf@vger.kernel.org4974L: bpf@vger.kernel.org
arch/arm64/include/asm/pgtable.h+28−0
@@ -1830,6 +1830,34 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, · 1829 unmodified lines
1830 return __ptep_get_and_clear(mm, addr, ptep);1830 return __ptep_get_and_clear(mm, addr, ptep);
1831}1831}
1832 1832
1833static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte)
1834{
1835 pteval_t old = 0;
1836
1837 if (!try_cmpxchg(&pte_val(*ptep), &old, pte_val(new_pte)))
1838 return false;
1839
1840 /*
1841 * The store must be complete by the time this returns, but the caller
1842 * may be in lazy MMU mode, where __set_pte_complete() would defer the
1843 * barriers. Issue them directly.
1844 */
1845 emit_pte_barriers();
1846 return true;
1847}
1848#define ptep_try_set ptep_try_set
1849
1850/*
1851 * arm64 mandates break-before-make: a cleared kernel PTE must have its TLB
1852 * invalidated before a different page is installed in its place. The broadcast
1853 * TLBI is an instruction, not an IPI, so this is safe with interrupts disabled.
1854 */
1855static inline void flush_tlb_before_set(unsigned long addr)
1856{
1857 flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
1858}
1859#define flush_tlb_before_set flush_tlb_before_set
1860
1833#define test_and_clear_young_ptes test_and_clear_young_ptes1861#define test_and_clear_young_ptes test_and_clear_young_ptes
1834static inline bool test_and_clear_young_ptes(struct vm_area_struct *vma,1862static inline bool test_and_clear_young_ptes(struct vm_area_struct *vma,
1835 unsigned long addr, pte_t *ptep, unsigned int nr)1863 unsigned long addr, pte_t *ptep, unsigned int nr)
arch/arm64/mm/fault.c+7−3
@@ -9,6 +9,7 @@ · 8 unmodified lines
9 9
10#include <linux/acpi.h>10#include <linux/acpi.h>
11#include <linux/bitfield.h>11#include <linux/bitfield.h>
12#include <linux/bpf_defs.h>
12#include <linux/extable.h>13#include <linux/extable.h>
13#include <linux/kfence.h>14#include <linux/kfence.h>
14#include <linux/signal.h>15#include <linux/signal.h>
@@ -436,9 +437,12 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr, · 421 unmodified lines
436 } else if (is_pkvm_stage2_abort(esr)) {437 } else if (is_pkvm_stage2_abort(esr)) {
437 msg = "access to hypervisor-protected memory";438 msg = "access to hypervisor-protected memory";
438 } else {439 } else {
439 if (esr_fsc_is_translation_fault(esr) &&440 if (esr_fsc_is_translation_fault(esr)) {
440 kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs))441 if (kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs))
441 return;442 return;
443 if (bpf_arena_handle_page_fault(addr, esr & ESR_ELx_WNR, regs->pc))
444 return;
445 }
442 446
443 msg = "paging request";447 msg = "paging request";
444 }448 }
arch/arm64/net/bpf_jit_comp.c+120−32
@@ -47,7 +47,7 @@ · 46 unmodified lines
47/* Map BPF registers to A64 registers */47/* Map BPF registers to A64 registers */
48static const int bpf2a64[] = {48static const int bpf2a64[] = {
49 /* return value from in-kernel function, and exit value from eBPF */49 /* return value from in-kernel function, and exit value from eBPF */
50 [BPF_REG_0] = A64_R(7),50 [BPF_REG_0] = A64_R(8),
51 /* arguments from eBPF program to in-kernel function */51 /* arguments from eBPF program to in-kernel function */
52 [BPF_REG_1] = A64_R(0),52 [BPF_REG_1] = A64_R(0),
53 [BPF_REG_2] = A64_R(1),53 [BPF_REG_2] = A64_R(1),
@@ -86,6 +86,7 @@ struct jit_ctx { · 32 unmodified lines
86 __le32 *image;86 __le32 *image;
87 __le32 *ro_image;87 __le32 *ro_image;
88 u32 stack_size;88 u32 stack_size;
89 u16 stack_arg_size;
89 u64 user_vm_start;90 u64 user_vm_start;
90 u64 arena_vm_start;91 u64 arena_vm_start;
91 bool fp_used;92 bool fp_used;
@@ -533,13 +534,20 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · 441 unmodified lines
533 * | |534 * | |
534 * +-----+ <= (BPF_FP - prog->aux->stack_depth)535 * +-----+ <= (BPF_FP - prog->aux->stack_depth)
535 * |RSVD | padding536 * |RSVD | padding
536 * current A64_SP => +-----+ <= (BPF_FP - ctx->stack_size)537 * +-----+ <= (BPF_FP - ctx->stack_size)
538 * | |
539 * | ... | outgoing stack args (9+, if any)
540 * | |
541 * current A64_SP => +-----+
537 * | |542 * | |
538 * | ... | Function call stack543 * | ... | Function call stack
539 * | |544 * | |
540 * +-----+545 * +-----+
541 * low546 * low
542 *547 *
548 * Stack args 6-8 are passed in x5-x7, args 9+ at [SP].
549 * Incoming args 9+ are at [A64_FP + 16], [A64_FP + 24], ...
550 * (above the saved FP/LR pair pushed in the callee prologue).
543 */551 */
544 552
545 emit_kcfi(is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx);553 emit_kcfi(is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx);
@@ -613,6 +621,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · 67 unmodified lines
613 if (ctx->stack_size && !ctx->priv_sp_used)621 if (ctx->stack_size && !ctx->priv_sp_used)
614 emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);622 emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
615 623
624 if (ctx->stack_arg_size)
625 emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx);
626
616 if (ctx->arena_vm_start)627 if (ctx->arena_vm_start)
617 emit_a64_mov_i64(arena_vm_base, ctx->arena_vm_start, ctx);628 emit_a64_mov_i64(arena_vm_base, ctx->arena_vm_start, ctx);
618 629
@@ -673,6 +684,9 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx) · 54 unmodified lines
673 /* Update tail_call_cnt if the slot is populated. */684 /* Update tail_call_cnt if the slot is populated. */
674 emit(A64_STR64I(tcc, ptr, 0), ctx);685 emit(A64_STR64I(tcc, ptr, 0), ctx);
675 686
687 if (ctx->stack_arg_size)
688 emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx);
689
676 /* restore SP */690 /* restore SP */
677 if (ctx->stack_size && !ctx->priv_sp_used)691 if (ctx->stack_size && !ctx->priv_sp_used)
678 emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);692 emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
@@ -1034,6 +1048,9 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic) · 355 unmodified lines
1034 const u8 r0 = bpf2a64[BPF_REG_0];1048 const u8 r0 = bpf2a64[BPF_REG_0];
1035 const u8 ptr = bpf2a64[TCCNT_PTR];1049 const u8 ptr = bpf2a64[TCCNT_PTR];
1036 1050
1051 if (ctx->stack_arg_size)
1052 emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx);
1053
1037 /* We're done with BPF stack */1054 /* We're done with BPF stack */
1038 if (ctx->stack_size && !ctx->priv_sp_used)1055 if (ctx->stack_size && !ctx->priv_sp_used)
1039 emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);1056 emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
@@ -1048,7 +1065,7 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic) · 8 unmodified lines
1048 /* Restore FP/LR registers */1065 /* Restore FP/LR registers */
1049 emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);1066 emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);
1050 1067
1051 /* Move the return value from bpf:r0 (aka x7) to x0 */1068 /* Move the return value from bpf:r0 (aka x8) to x0 */
1052 emit(A64_MOV(1, A64_R(0), r0), ctx);1069 emit(A64_MOV(1, A64_R(0), r0), ctx);
1053 1070
1054 /* Authenticate lr */1071 /* Authenticate lr */
@@ -1191,6 +1208,42 @@ static int add_exception_handler(const struct bpf_insn *insn, · 136 unmodified lines
1191 return 0;1208 return 0;
1192}1209}
1193 1210
1211static const u8 stack_arg_reg[] = { A64_R(5), A64_R(6), A64_R(7) };
1212
1213#define NR_STACK_ARG_REGS ARRAY_SIZE(stack_arg_reg)
1214
1215static void emit_stack_arg_load(u8 dst, s16 bpf_off, struct jit_ctx *ctx)
1216{
1217 int idx = bpf_off / sizeof(u64) - 1;
1218
1219 if (idx < NR_STACK_ARG_REGS)
1220 emit(A64_MOV(1, dst, stack_arg_reg[idx]), ctx);
1221 else
1222 emit(A64_LDR64I(dst, A64_FP, (idx - NR_STACK_ARG_REGS) * sizeof(u64) + 16), ctx);
1223}
1224
1225static void emit_stack_arg_store(u8 src_a64, s16 bpf_off, struct jit_ctx *ctx)
1226{
1227 int idx = -bpf_off / sizeof(u64) - 1;
1228
1229 if (idx < NR_STACK_ARG_REGS)
1230 emit(A64_MOV(1, stack_arg_reg[idx], src_a64), ctx);
1231 else
1232 emit(A64_STR64I(src_a64, A64_SP, (idx - NR_STACK_ARG_REGS) * sizeof(u64)), ctx);
1233}
1234
1235static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct jit_ctx *ctx)
1236{
1237 int idx = -bpf_off / sizeof(u64) - 1;
1238
1239 if (idx < NR_STACK_ARG_REGS) {
1240 emit_a64_mov_i(1, stack_arg_reg[idx], imm, ctx);
1241 } else {
1242 emit_a64_mov_i(1, tmp, imm, ctx);
1243 emit(A64_STR64I(tmp, A64_SP, (idx - NR_STACK_ARG_REGS) * sizeof(u64)), ctx);
1244 }
1245}
1246
1194/* JITs an eBPF instruction.1247/* JITs an eBPF instruction.
1195 * Returns:1248 * Returns:
1196 * 0 - successfully JITed an 8-byte eBPF instruction.1249 * 0 - successfully JITed an 8-byte eBPF instruction.
@@ -1646,6 +1699,11 @@ emit_cond_jmp: · 449 unmodified lines
1646 case BPF_LDX | BPF_MEM | BPF_H:1699 case BPF_LDX | BPF_MEM | BPF_H:
1647 case BPF_LDX | BPF_MEM | BPF_B:1700 case BPF_LDX | BPF_MEM | BPF_B:
1648 case BPF_LDX | BPF_MEM | BPF_DW:1701 case BPF_LDX | BPF_MEM | BPF_DW:
1702 if (insn->src_reg == BPF_REG_PARAMS) {
1703 emit_stack_arg_load(dst, off, ctx);
1704 break;
1705 }
1706 fallthrough;
1649 case BPF_LDX | BPF_PROBE_MEM | BPF_DW:1707 case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
1650 case BPF_LDX | BPF_PROBE_MEM | BPF_W:1708 case BPF_LDX | BPF_PROBE_MEM | BPF_W:
1651 case BPF_LDX | BPF_PROBE_MEM | BPF_H:1709 case BPF_LDX | BPF_PROBE_MEM | BPF_H:
@@ -1672,6 +1730,8 @@ emit_cond_jmp: · 20 unmodified lines
1672 if (src == fp) {1730 if (src == fp) {
1673 src_adj = ctx->priv_sp_used ? priv_sp : A64_SP;1731 src_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
1674 off_adj = off + ctx->stack_size;1732 off_adj = off + ctx->stack_size;
1733 if (!ctx->priv_sp_used)
1734 off_adj += ctx->stack_arg_size;
1675 } else {1735 } else {
1676 src_adj = src;1736 src_adj = src;
1677 off_adj = off;1737 off_adj = off;
@@ -1752,6 +1812,11 @@ emit_cond_jmp: · 74 unmodified lines
1752 case BPF_ST | BPF_MEM | BPF_H:1812 case BPF_ST | BPF_MEM | BPF_H:
1753 case BPF_ST | BPF_MEM | BPF_B:1813 case BPF_ST | BPF_MEM | BPF_B:
1754 case BPF_ST | BPF_MEM | BPF_DW:1814 case BPF_ST | BPF_MEM | BPF_DW:
1815 if (insn->dst_reg == BPF_REG_PARAMS) {
1816 emit_stack_arg_store_imm(imm, off, tmp, ctx);
1817 break;
1818 }
1819 fallthrough;
1755 case BPF_ST | BPF_PROBE_MEM32 | BPF_B:1820 case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
1756 case BPF_ST | BPF_PROBE_MEM32 | BPF_H:1821 case BPF_ST | BPF_PROBE_MEM32 | BPF_H:
1757 case BPF_ST | BPF_PROBE_MEM32 | BPF_W:1822 case BPF_ST | BPF_PROBE_MEM32 | BPF_W:
@@ -1763,6 +1828,8 @@ emit_cond_jmp: · 5 unmodified lines
1763 if (dst == fp) {1828 if (dst == fp) {
1764 dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;1829 dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
1765 off_adj = off + ctx->stack_size;1830 off_adj = off + ctx->stack_size;
1831 if (!ctx->priv_sp_used)
1832 off_adj += ctx->stack_arg_size;
1766 } else {1833 } else {
1767 dst_adj = dst;1834 dst_adj = dst;
1768 off_adj = off;1835 off_adj = off;
@@ -1814,6 +1881,11 @@ emit_cond_jmp: · 45 unmodified lines
1814 case BPF_STX | BPF_MEM | BPF_H:1881 case BPF_STX | BPF_MEM | BPF_H:
1815 case BPF_STX | BPF_MEM | BPF_B:1882 case BPF_STX | BPF_MEM | BPF_B:
1816 case BPF_STX | BPF_MEM | BPF_DW:1883 case BPF_STX | BPF_MEM | BPF_DW:
1884 if (insn->dst_reg == BPF_REG_PARAMS) {
1885 emit_stack_arg_store(src, off, ctx);
1886 break;
1887 }
1888 fallthrough;
1817 case BPF_STX | BPF_PROBE_MEM32 | BPF_B:1889 case BPF_STX | BPF_PROBE_MEM32 | BPF_B:
1818 case BPF_STX | BPF_PROBE_MEM32 | BPF_H:1890 case BPF_STX | BPF_PROBE_MEM32 | BPF_H:
1819 case BPF_STX | BPF_PROBE_MEM32 | BPF_W:1891 case BPF_STX | BPF_PROBE_MEM32 | BPF_W:
@@ -1825,6 +1897,8 @@ emit_cond_jmp: · 5 unmodified lines
1825 if (dst == fp) {1897 if (dst == fp) {
1826 dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;1898 dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
1827 off_adj = off + ctx->stack_size;1899 off_adj = off + ctx->stack_size;
1900 if (!ctx->priv_sp_used)
1901 off_adj += ctx->stack_arg_size;
1828 } else {1902 } else {
1829 dst_adj = dst;1903 dst_adj = dst;
1830 off_adj = off;1904 off_adj = off;
@@ -2018,6 +2092,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · 187 unmodified lines
2018 u8 *ro_image_ptr;2092 u8 *ro_image_ptr;
2019 int body_idx;2093 int body_idx;
2020 int exentry_idx;2094 int exentry_idx;
2095 int out_cnt;
2021 2096
2022 if (!prog->jit_requested)2097 if (!prog->jit_requested)
2023 return prog;2098 return prog;
@@ -2065,6 +2140,14 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · 41 unmodified lines
2065 ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);2140 ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
2066 ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);2141 ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
2067 2142
2143 out_cnt = bpf_out_stack_arg_cnt(env, prog);
2144 if (out_cnt) {
2145 int nr_on_stack = out_cnt - NR_STACK_ARG_REGS;
2146
2147 if (nr_on_stack > 0)
2148 ctx.stack_arg_size = round_up(nr_on_stack * sizeof(u64), 16);
2149 }
2150
2068 if (priv_stack_ptr)2151 if (priv_stack_ptr)
2069 ctx.priv_sp_used = true;2152 ctx.priv_sp_used = true;
2070 2153
@@ -2229,6 +2312,11 @@ bool bpf_jit_supports_kfunc_call(void) · 158 unmodified lines
2229 return true;2312 return true;
2230}2313}
2231 2314
2315bool bpf_jit_supports_stack_args(void)
2316{
2317 return true;
2318}
2319
2232void *bpf_arch_text_copy(void *dst, void *src, size_t len)2320void *bpf_arch_text_copy(void *dst, void *src, size_t len)
2233{2321{
2234 if (!aarch64_insn_copy(dst, src, len))2322 if (!aarch64_insn_copy(dst, src, len))
@@ -2247,24 +2335,24 @@ bool bpf_jit_supports_subprog_tailcalls(void) · 12 unmodified lines
2247 return true;2335 return true;
2248}2336}
2249 2337
2250static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,2338static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node,
2251 int bargs_off, int retval_off, int run_ctx_off,2339 int bargs_off, int retval_off, int run_ctx_off,
2252 bool save_ret)2340 bool save_ret)
2253{2341{
2254 __le32 *branch;2342 __le32 *branch;
2255 u64 enter_prog;2343 u64 enter_prog;
2256 u64 exit_prog;2344 u64 exit_prog;
2257 struct bpf_prog *p = l->link.prog;2345 struct bpf_prog *p = node->link->prog;
2258 int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);2346 int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
2259 2347
2260 enter_prog = (u64)bpf_trampoline_enter(p);2348 enter_prog = (u64)bpf_trampoline_enter(p);
2261 exit_prog = (u64)bpf_trampoline_exit(p);2349 exit_prog = (u64)bpf_trampoline_exit(p);
2262 2350
2263 if (l->cookie == 0) {2351 if (node->cookie == 0) {
2264 /* if cookie is zero, one instruction is enough to store it */2352 /* if cookie is zero, one instruction is enough to store it */
2265 emit(A64_STR64I(A64_ZR, A64_SP, run_ctx_off + cookie_off), ctx);2353 emit(A64_STR64I(A64_ZR, A64_SP, run_ctx_off + cookie_off), ctx);
2266 } else {2354 } else {
2267 emit_a64_mov_i64(A64_R(10), l->cookie, ctx);2355 emit_a64_mov_i64(A64_R(10), node->cookie, ctx);
2268 emit(A64_STR64I(A64_R(10), A64_SP, run_ctx_off + cookie_off),2356 emit(A64_STR64I(A64_R(10), A64_SP, run_ctx_off + cookie_off),
2269 ctx);2357 ctx);
2270 }2358 }
@@ -2314,7 +2402,7 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, · 43 unmodified lines
2314 emit_call(exit_prog, ctx);2402 emit_call(exit_prog, ctx);
2315}2403}
2316 2404
2317static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl,2405static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn,
2318 int bargs_off, int retval_off, int run_ctx_off,2406 int bargs_off, int retval_off, int run_ctx_off,
2319 __le32 **branches)2407 __le32 **branches)
2320{2408{
@@ -2324,8 +2412,8 @@ static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl, · 3 unmodified lines
2324 * Set this to 0 to avoid confusing the program.2412 * Set this to 0 to avoid confusing the program.
2325 */2413 */
2326 emit(A64_STR64I(A64_ZR, A64_SP, retval_off), ctx);2414 emit(A64_STR64I(A64_ZR, A64_SP, retval_off), ctx);
2327 for (i = 0; i < tl->nr_links; i++) {2415 for (i = 0; i < tn->nr_nodes; i++) {
2328 invoke_bpf_prog(ctx, tl->links[i], bargs_off, retval_off,2416 invoke_bpf_prog(ctx, tn->nodes[i], bargs_off, retval_off,
2329 run_ctx_off, true);2417 run_ctx_off, true);
2330 /* if (*(u64 *)(sp + retval_off) != 0)2418 /* if (*(u64 *)(sp + retval_off) != 0)
2331 * goto do_fexit;2419 * goto do_fexit;
@@ -2456,10 +2544,10 @@ static void restore_args(struct jit_ctx *ctx, int bargs_off, int nregs) · 124 unmodified lines
2456 }2544 }
2457}2545}
2458 2546
2459static bool is_struct_ops_tramp(const struct bpf_tramp_links *fentry_links)2547static bool is_struct_ops_tramp(const struct bpf_tramp_nodes *fentry_nodes)
2460{2548{
2461 return fentry_links->nr_links == 1 &&2549 return fentry_nodes->nr_nodes == 1 &&
2462 fentry_links->links[0]->link.type == BPF_LINK_TYPE_STRUCT_OPS;2550 fentry_nodes->nodes[0]->link->type == BPF_LINK_TYPE_STRUCT_OPS;
2463}2551}
2464 2552
2465static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_off)2553static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_off)
@@ -2480,7 +2568,7 @@ static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_of · 14 unmodified lines
2480 *2568 *
2481 */2569 */
2482static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,2570static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
2483 struct bpf_tramp_links *tlinks, void *func_addr,2571 struct bpf_tramp_nodes *tnodes, void *func_addr,
2484 const struct btf_func_model *m,2572 const struct btf_func_model *m,
2485 const struct arg_aux *a,2573 const struct arg_aux *a,
2486 u32 flags)2574 u32 flags)
@@ -2496,14 +2584,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 9 unmodified lines
2496 int run_ctx_off;2584 int run_ctx_off;
2497 int oargs_off;2585 int oargs_off;
2498 int nfuncargs;2586 int nfuncargs;
2499 struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];2587 struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
2500 struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];2588 struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
2501 struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];2589 struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
2502 bool save_ret;2590 bool save_ret;
2503 __le32 **branches = NULL;2591 __le32 **branches = NULL;
2504 bool is_struct_ops = is_struct_ops_tramp(fentry);2592 bool is_struct_ops = is_struct_ops_tramp(fentry);
2505 int cookie_off, cookie_cnt, cookie_bargs_off;2593 int cookie_off, cookie_cnt, cookie_bargs_off;
2506 int fsession_cnt = bpf_fsession_cnt(tlinks);2594 int fsession_cnt = bpf_fsession_cnt(tnodes);
2507 u64 func_meta;2595 u64 func_meta;
2508 2596
2509 /* trampoline stack layout:2597 /* trampoline stack layout:
@@ -2549,7 +2637,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 39 unmodified lines
2549 2637
2550 cookie_off = stack_size;2638 cookie_off = stack_size;
2551 /* room for session cookies */2639 /* room for session cookies */
2552 cookie_cnt = bpf_fsession_cookie_cnt(tlinks);2640 cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
2553 stack_size += cookie_cnt * 8;2641 stack_size += cookie_cnt * 8;
2554 2642
2555 ip_off = stack_size;2643 ip_off = stack_size;
@@ -2646,20 +2734,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 90 unmodified lines
2646 }2734 }
2647 2735
2648 cookie_bargs_off = (bargs_off - cookie_off) / 8;2736 cookie_bargs_off = (bargs_off - cookie_off) / 8;
2649 for (i = 0; i < fentry->nr_links; i++) {2737 for (i = 0; i < fentry->nr_nodes; i++) {
2650 if (bpf_prog_calls_session_cookie(fentry->links[i])) {2738 if (bpf_prog_calls_session_cookie(fentry->nodes[i])) {
2651 u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);2739 u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
2652 2740
2653 store_func_meta(ctx, meta, func_meta_off);2741 store_func_meta(ctx, meta, func_meta_off);
2654 cookie_bargs_off--;2742 cookie_bargs_off--;
2655 }2743 }
2656 invoke_bpf_prog(ctx, fentry->links[i], bargs_off,2744 invoke_bpf_prog(ctx, fentry->nodes[i], bargs_off,
2657 retval_off, run_ctx_off,2745 retval_off, run_ctx_off,
2658 flags & BPF_TRAMP_F_RET_FENTRY_RET);2746 flags & BPF_TRAMP_F_RET_FENTRY_RET);
2659 }2747 }
2660 2748
2661 if (fmod_ret->nr_links) {2749 if (fmod_ret->nr_nodes) {
2662 branches = kcalloc(fmod_ret->nr_links, sizeof(__le32 *),2750 branches = kcalloc(fmod_ret->nr_nodes, sizeof(__le32 *),
2663 GFP_KERNEL);2751 GFP_KERNEL);
2664 if (!branches)2752 if (!branches)
2665 return -ENOMEM;2753 return -ENOMEM;
@@ -2683,7 +2771,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 17 unmodified lines
2683 }2771 }
2684 2772
2685 /* update the branches saved in invoke_bpf_mod_ret with cbnz */2773 /* update the branches saved in invoke_bpf_mod_ret with cbnz */
2686 for (i = 0; i < fmod_ret->nr_links && ctx->image != NULL; i++) {2774 for (i = 0; i < fmod_ret->nr_nodes && ctx->image != NULL; i++) {
2687 int offset = &ctx->image[ctx->idx] - branches[i];2775 int offset = &ctx->image[ctx->idx] - branches[i];
2688 *branches[i] = cpu_to_le32(A64_CBNZ(1, A64_R(10), offset));2776 *branches[i] = cpu_to_le32(A64_CBNZ(1, A64_R(10), offset));
2689 }2777 }
@@ -2694,14 +2782,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 4 unmodified lines
2694 store_func_meta(ctx, func_meta, func_meta_off);2782 store_func_meta(ctx, func_meta, func_meta_off);
2695 2783
2696 cookie_bargs_off = (bargs_off - cookie_off) / 8;2784 cookie_bargs_off = (bargs_off - cookie_off) / 8;
2697 for (i = 0; i < fexit->nr_links; i++) {2785 for (i = 0; i < fexit->nr_nodes; i++) {
2698 if (bpf_prog_calls_session_cookie(fexit->links[i])) {2786 if (bpf_prog_calls_session_cookie(fexit->nodes[i])) {
2699 u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);2787 u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
2700 2788
2701 store_func_meta(ctx, meta, func_meta_off);2789 store_func_meta(ctx, meta, func_meta_off);
2702 cookie_bargs_off--;2790 cookie_bargs_off--;
2703 }2791 }
2704 invoke_bpf_prog(ctx, fexit->links[i], bargs_off, retval_off,2792 invoke_bpf_prog(ctx, fexit->nodes[i], bargs_off, retval_off,
2705 run_ctx_off, false);2793 run_ctx_off, false);
2706 }2794 }
2707 2795
@@ -2759,7 +2847,7 @@ bool bpf_jit_supports_fsession(void) · 51 unmodified lines
2759}2847}
2760 2848
2761int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,2849int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
2762 struct bpf_tramp_links *tlinks, void *func_addr)2850 struct bpf_tramp_nodes *tnodes, void *func_addr)
2763{2851{
2764 struct jit_ctx ctx = {2852 struct jit_ctx ctx = {
2765 .image = NULL,2853 .image = NULL,
@@ -2773,7 +2861,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 7 unmodified lines
2773 if (ret < 0)2861 if (ret < 0)
2774 return ret;2862 return ret;
2775 2863
2776 ret = prepare_trampoline(&ctx, &im, tlinks, func_addr, m, &aaux, flags);2864 ret = prepare_trampoline(&ctx, &im, tnodes, func_addr, m, &aaux, flags);
2777 if (ret < 0)2865 if (ret < 0)
2778 return ret;2866 return ret;
2779 2867
@@ -2797,7 +2885,7 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) · 17 unmodified lines
2797 2885
2798int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,2886int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
2799 void *ro_image_end, const struct btf_func_model *m,2887 void *ro_image_end, const struct btf_func_model *m,
2800 u32 flags, struct bpf_tramp_links *tlinks,2888 u32 flags, struct bpf_tramp_nodes *tnodes,
2801 void *func_addr)2889 void *func_addr)
2802{2890{
2803 u32 size = ro_image_end - ro_image;2891 u32 size = ro_image_end - ro_image;
@@ -2824,7 +2912,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · 20 unmodified lines
2824 ret = calc_arg_aux(m, &aaux);2912 ret = calc_arg_aux(m, &aaux);
2825 if (ret)2913 if (ret)
2826 goto out;2914 goto out;
2827 ret = prepare_trampoline(&ctx, im, tlinks, func_addr, m, &aaux, flags);2915 ret = prepare_trampoline(&ctx, im, tnodes, func_addr, m, &aaux, flags);
2828 2916
2829 if (ret > 0 && validate_code(&ctx) < 0) {2917 if (ret > 0 && validate_code(&ctx) < 0) {
2830 ret = -EINVAL;2918 ret = -EINVAL;
arch/arm64/net/bpf_timed_may_goto.S+4−4
@@ -8,8 +8,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto) · 7 unmodified lines
8 stp x29, x30, [sp, #-64]!8 stp x29, x30, [sp, #-64]!
9 mov x29, sp9 mov x29, sp
10 10
11 /* Save BPF registers R0 - R5 (x7, x0-x4)*/11 /* Save BPF registers R0 - R5 (x8, x0-x4)*/
12 stp x7, x0, [sp, #16]12 stp x8, x0, [sp, #16]
13 stp x1, x2, [sp, #32]13 stp x1, x2, [sp, #32]
14 stp x3, x4, [sp, #48]14 stp x3, x4, [sp, #48]
15 15
@@ -28,8 +28,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto) · 12 unmodified lines
28 /* BPF_REG_AX(x9) will be stored into count, so move return value to it. */28 /* BPF_REG_AX(x9) will be stored into count, so move return value to it. */
29 mov x9, x029 mov x9, x0
30 30
31 /* Restore BPF registers R0 - R5 (x7, x0-x4) */31 /* Restore BPF registers R0 - R5 (x8, x0-x4) */
32 ldp x7, x0, [sp, #16]32 ldp x8, x0, [sp, #16]
33 ldp x1, x2, [sp, #32]33 ldp x1, x2, [sp, #32]
34 ldp x3, x4, [sp, #48]34 ldp x3, x4, [sp, #48]
35 35
arch/loongarch/net/bpf_jit.c+26−26
@@ -1674,17 +1674,17 @@ static void restore_stk_args(struct jit_ctx *ctx, int nr_stk_args, int args_off, · 1673 unmodified lines
1674 }1674 }
1675}1675}
1676 1676
1677static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,1677static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *n,
1678 int args_off, int retval_off, int run_ctx_off, bool save_ret)1678 int args_off, int retval_off, int run_ctx_off, bool save_ret)
1679{1679{
1680 int ret;1680 int ret;
1681 u32 *branch;1681 u32 *branch;
1682 struct bpf_prog *p = l->link.prog;1682 struct bpf_prog *p = n->link->prog;
1683 int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);1683 int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
1684 1684
1685 if (l->cookie)1685 if (n->cookie)
1686 emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1,1686 emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1,
1687 -run_ctx_off + cookie_off, l->cookie);1687 -run_ctx_off + cookie_off, n->cookie);
1688 else1688 else
1689 emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -run_ctx_off + cookie_off);1689 emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -run_ctx_off + cookie_off);
1690 1690
@@ -1737,22 +1737,22 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, · 46 unmodified lines
1737 return ret;1737 return ret;
1738}1738}
1739 1739
1740static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_links *tl,1740static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn,
1741 int args_off, int retval_off, int run_ctx_off,1741 int args_off, int retval_off, int run_ctx_off,
1742 int func_meta_off, bool save_ret, u64 func_meta, int cookie_off)1742 int func_meta_off, bool save_ret, u64 func_meta, int cookie_off)
1743{1743{
1744 int i, cur_cookie = (cookie_off - args_off) / 8;1744 int i, cur_cookie = (cookie_off - args_off) / 8;
1745 1745
1746 for (i = 0; i < tl->nr_links; i++) {1746 for (i = 0; i < tn->nr_nodes; i++) {
1747 int err;1747 int err;
1748 1748
1749 if (bpf_prog_calls_session_cookie(tl->links[i])) {1749 if (bpf_prog_calls_session_cookie(tn->nodes[i])) {
1750 u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);1750 u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);
1751 1751
1752 emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, meta);1752 emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, meta);
1753 cur_cookie--;1753 cur_cookie--;
1754 }1754 }
1755 err = invoke_bpf_prog(ctx, tl->links[i], args_off, retval_off, run_ctx_off, save_ret);1755 err = invoke_bpf_prog(ctx, tn->nodes[i], args_off, retval_off, run_ctx_off, save_ret);
1756 if (err)1756 if (err)
1757 return err;1757 return err;
1758 }1758 }
@@ -1807,7 +1807,7 @@ static void sign_extend(struct jit_ctx *ctx, int rd, int rj, u8 size, bool sign) · 48 unmodified lines
1807}1807}
1808 1808
1809static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,1809static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
1810 const struct btf_func_model *m, struct bpf_tramp_links *tlinks,1810 const struct btf_func_model *m, struct bpf_tramp_nodes *tnodes,
1811 void *func_addr, u32 flags)1811 void *func_addr, u32 flags)
1812{1812{
1813 int i, ret, save_ret;1813 int i, ret, save_ret;
@@ -1817,9 +1817,9 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 3 unmodified lines
1817 unsigned long long func_meta;1817 unsigned long long func_meta;
1818 bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT;1818 bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT;
1819 void *orig_call = func_addr;1819 void *orig_call = func_addr;
1820 struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];1820 struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
1821 struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];1821 struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
1822 struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];1822 struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
1823 u32 **branches = NULL;1823 u32 **branches = NULL;
1824 1824
1825 /*1825 /*
@@ -1898,7 +1898,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 72 unmodified lines
1898 ip_off = stack_size;1898 ip_off = stack_size;
1899 }1899 }
1900 1900
1901 cookie_cnt = bpf_fsession_cookie_cnt(tlinks);1901 cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
1902 1902
1903 /* Room for session cookies */1903 /* Room for session cookies */
1904 stack_size += cookie_cnt * 8;1904 stack_size += cookie_cnt * 8;
@@ -1969,7 +1969,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 64 unmodified lines
1969 1969
1970 store_args(ctx, nr_arg_slots, args_off);1970 store_args(ctx, nr_arg_slots, args_off);
1971 1971
1972 if (bpf_fsession_cnt(tlinks)) {1972 if (bpf_fsession_cnt(tnodes)) {
1973 /* clear all session cookies' value */1973 /* clear all session cookies' value */
1974 for (i = 0; i < cookie_cnt; i++)1974 for (i = 0; i < cookie_cnt; i++)
1975 emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -cookie_off + 8 * i);1975 emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -cookie_off + 8 * i);
@@ -1994,20 +1994,20 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 18 unmodified lines
1994 return ret;1994 return ret;
1995 }1995 }
1996 1996
1997 if (fentry->nr_links) {1997 if (fentry->nr_nodes) {
1998 ret = invoke_bpf(ctx, fentry, args_off, retval_off, run_ctx_off, func_meta_off,1998 ret = invoke_bpf(ctx, fentry, args_off, retval_off, run_ctx_off, func_meta_off,
1999 flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off);1999 flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off);
2000 if (ret)2000 if (ret)
2001 return ret;2001 return ret;
2002 }2002 }
2003 if (fmod_ret->nr_links) {2003 if (fmod_ret->nr_nodes) {
2004 branches = kcalloc(fmod_ret->nr_links, sizeof(u32 *), GFP_KERNEL);2004 branches = kcalloc(fmod_ret->nr_nodes, sizeof(u32 *), GFP_KERNEL);
2005 if (!branches)2005 if (!branches)
2006 return -ENOMEM;2006 return -ENOMEM;
2007 2007
2008 emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -retval_off);2008 emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -retval_off);
2009 for (i = 0; i < fmod_ret->nr_links; i++) {2009 for (i = 0; i < fmod_ret->nr_nodes; i++) {
2010 ret = invoke_bpf_prog(ctx, fmod_ret->links[i],2010 ret = invoke_bpf_prog(ctx, fmod_ret->nodes[i],
2011 args_off, retval_off, run_ctx_off, true);2011 args_off, retval_off, run_ctx_off, true);
2012 if (ret)2012 if (ret)
2013 goto out;2013 goto out;
@@ -2035,17 +2035,17 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 21 unmodified lines
2035 emit_insn(ctx, nop);2035 emit_insn(ctx, nop);
2036 }2036 }
2037 2037
2038 for (i = 0; ctx->image && i < fmod_ret->nr_links; i++) {2038 for (i = 0; ctx->image && i < fmod_ret->nr_nodes; i++) {
2039 int offset = (void *)(&ctx->image[ctx->idx]) - (void *)branches[i];2039 int offset = (void *)(&ctx->image[ctx->idx]) - (void *)branches[i];
2040 *branches[i] = larch_insn_gen_bne(LOONGARCH_GPR_T1, LOONGARCH_GPR_ZERO, offset);2040 *branches[i] = larch_insn_gen_bne(LOONGARCH_GPR_T1, LOONGARCH_GPR_ZERO, offset);
2041 }2041 }
2042 2042
2043 /* Set "is_return" flag for fsession */2043 /* Set "is_return" flag for fsession */
2044 func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);2044 func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
2045 if (bpf_fsession_cnt(tlinks))2045 if (bpf_fsession_cnt(tnodes))
2046 emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, func_meta);2046 emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, func_meta);
2047 2047
2048 if (fexit->nr_links) {2048 if (fexit->nr_nodes) {
2049 ret = invoke_bpf(ctx, fexit, args_off, retval_off, run_ctx_off,2049 ret = invoke_bpf(ctx, fexit, args_off, retval_off, run_ctx_off,
2050 func_meta_off, false, func_meta, cookie_off);2050 func_meta_off, false, func_meta, cookie_off);
2051 if (ret)2051 if (ret)
@@ -2115,7 +2115,7 @@ out: · 63 unmodified lines
2115 2115
2116int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,2116int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
2117 void *ro_image_end, const struct btf_func_model *m,2117 void *ro_image_end, const struct btf_func_model *m,
2118 u32 flags, struct bpf_tramp_links *tlinks, void *func_addr)2118 u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr)
2119{2119{
2120 int ret, size;2120 int ret, size;
2121 void *image, *tmp;2121 void *image, *tmp;
@@ -2131,7 +2131,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · 9 unmodified lines
2131 ctx.idx = 0;2131 ctx.idx = 0;
2132 2132
2133 jit_fill_hole(image, (unsigned int)(ro_image_end - ro_image));2133 jit_fill_hole(image, (unsigned int)(ro_image_end - ro_image));
2134 ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tlinks, func_addr, flags);2134 ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tnodes, func_addr, flags);
2135 if (ret < 0)2135 if (ret < 0)
2136 goto out;2136 goto out;
2137 2137
@@ -2152,7 +2152,7 @@ out: · 14 unmodified lines
2152}2152}
2153 2153
2154int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,2154int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
2155 struct bpf_tramp_links *tlinks, void *func_addr)2155 struct bpf_tramp_nodes *tnodes, void *func_addr)
2156{2156{
2157 int ret;2157 int ret;
2158 struct jit_ctx ctx;2158 struct jit_ctx ctx;
@@ -2161,7 +2161,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 2 unmodified lines
2161 ctx.image = NULL;2161 ctx.image = NULL;
2162 ctx.idx = 0;2162 ctx.idx = 0;
2163 2163
2164 ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tlinks, func_addr, flags);2164 ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tnodes, func_addr, flags);
2165 2165
2166 return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE;2166 return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE;
2167}2167}
arch/powerpc/net/bpf_jit_comp.c+27−27
@@ -597,22 +597,22 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) · 596 unmodified lines
597}597}
598 598
599static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ctx,599static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ctx,
600 struct bpf_tramp_link *l, int regs_off, int retval_off,600 struct bpf_tramp_node *n, int regs_off, int retval_off,
601 int run_ctx_off, bool save_ret)601 int run_ctx_off, bool save_ret)
602{602{
603 struct bpf_prog *p = l->link.prog;603 struct bpf_prog *p = n->link->prog;
604 ppc_inst_t branch_insn;604 ppc_inst_t branch_insn;
605 u32 jmp_idx;605 u32 jmp_idx;
606 int ret = 0;606 int ret = 0;
607 607
608 /* Save cookie */608 /* Save cookie */
609 if (IS_ENABLED(CONFIG_PPC64)) {609 if (IS_ENABLED(CONFIG_PPC64)) {
610 PPC_LI64(_R3, l->cookie);610 PPC_LI64(_R3, n->cookie);
611 EMIT(PPC_RAW_STD(_R3, _R1, run_ctx_off + offsetof(struct bpf_tramp_run_ctx,611 EMIT(PPC_RAW_STD(_R3, _R1, run_ctx_off + offsetof(struct bpf_tramp_run_ctx,
612 bpf_cookie)));612 bpf_cookie)));
613 } else {613 } else {
614 PPC_LI32(_R3, l->cookie >> 32);614 PPC_LI32(_R3, n->cookie >> 32);
615 PPC_LI32(_R4, l->cookie);615 PPC_LI32(_R4, n->cookie);
616 EMIT(PPC_RAW_STW(_R3, _R1,616 EMIT(PPC_RAW_STW(_R3, _R1,
617 run_ctx_off + offsetof(struct bpf_tramp_run_ctx, bpf_cookie)));617 run_ctx_off + offsetof(struct bpf_tramp_run_ctx, bpf_cookie)));
618 EMIT(PPC_RAW_STW(_R4, _R1,618 EMIT(PPC_RAW_STW(_R4, _R1,
@@ -679,7 +679,7 @@ static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ct · 60 unmodified lines
679}679}
680 680
681static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context *ctx,681static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context *ctx,
682 struct bpf_tramp_links *tl, int regs_off, int retval_off,682 struct bpf_tramp_nodes *tn, int regs_off, int retval_off,
683 int run_ctx_off, u32 *branches)683 int run_ctx_off, u32 *branches)
684{684{
685 int i;685 int i;
@@ -690,8 +690,8 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context · 4 unmodified lines
690 */690 */
691 EMIT(PPC_RAW_LI(_R3, 0));691 EMIT(PPC_RAW_LI(_R3, 0));
692 EMIT(PPC_RAW_STL(_R3, _R1, retval_off));692 EMIT(PPC_RAW_STL(_R3, _R1, retval_off));
693 for (i = 0; i < tl->nr_links; i++) {693 for (i = 0; i < tn->nr_nodes; i++) {
694 if (invoke_bpf_prog(image, ro_image, ctx, tl->links[i], regs_off, retval_off,694 if (invoke_bpf_prog(image, ro_image, ctx, tn->nodes[i], regs_off, retval_off,
695 run_ctx_off, true))695 run_ctx_off, true))
696 return -EINVAL;696 return -EINVAL;
697 697
@@ -807,18 +807,18 @@ static void bpf_trampoline_restore_args_stack(u32 *image, struct codegen_context · 109 unmodified lines
807static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,807static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,
808 void *rw_image_end, void *ro_image,808 void *rw_image_end, void *ro_image,
809 const struct btf_func_model *m, u32 flags,809 const struct btf_func_model *m, u32 flags,
810 struct bpf_tramp_links *tlinks,810 struct bpf_tramp_nodes *tnodes,
811 void *func_addr)811 void *func_addr)
812{812{
813 int regs_off, func_meta_off, ip_off, run_ctx_off, retval_off;813 int regs_off, func_meta_off, ip_off, run_ctx_off, retval_off;
814 int nvr_off, alt_lr_off, r4_off = 0;814 int nvr_off, alt_lr_off, r4_off = 0;
815 struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];815 struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
816 struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];816 struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
817 struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];817 struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
818 int i, ret, nr_regs, retaddr_off, bpf_frame_size = 0;818 int i, ret, nr_regs, retaddr_off, bpf_frame_size = 0;
819 struct codegen_context codegen_ctx, *ctx;819 struct codegen_context codegen_ctx, *ctx;
820 int cookie_off, cookie_cnt, cookie_ctx_off;820 int cookie_off, cookie_cnt, cookie_ctx_off;
821 int fsession_cnt = bpf_fsession_cnt(tlinks);821 int fsession_cnt = bpf_fsession_cnt(tnodes);
822 u64 func_meta;822 u64 func_meta;
823 u32 *image = (u32 *)rw_image;823 u32 *image = (u32 *)rw_image;
824 ppc_inst_t branch_insn;824 ppc_inst_t branch_insn;
@@ -893,7 +893,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 68 unmodified lines
893 893
894 /* room for session cookies */894 /* room for session cookies */
895 cookie_off = bpf_frame_size;895 cookie_off = bpf_frame_size;
896 cookie_cnt = bpf_fsession_cookie_cnt(tlinks);896 cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
897 bpf_frame_size += cookie_cnt * 8;897 bpf_frame_size += cookie_cnt * 8;
898 898
899 /* Room for IP address argument */899 /* Room for IP address argument */
@@ -1030,21 +1030,21 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 130 unmodified lines
1030 1030
1031 cookie_ctx_off = (regs_off - cookie_off) / 8;1031 cookie_ctx_off = (regs_off - cookie_off) / 8;
1032 1032
1033 for (i = 0; i < fentry->nr_links; i++) {1033 for (i = 0; i < fentry->nr_nodes; i++) {
1034 if (bpf_prog_calls_session_cookie(fentry->links[i])) {1034 if (bpf_prog_calls_session_cookie(fentry->nodes[i])) {
1035 u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);1035 u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
1036 1036
1037 store_func_meta(image, ctx, meta, func_meta_off);1037 store_func_meta(image, ctx, meta, func_meta_off);
1038 cookie_ctx_off--;1038 cookie_ctx_off--;
1039 }1039 }
1040 1040
1041 if (invoke_bpf_prog(image, ro_image, ctx, fentry->links[i], regs_off, retval_off,1041 if (invoke_bpf_prog(image, ro_image, ctx, fentry->nodes[i], regs_off, retval_off,
1042 run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET))1042 run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET))
1043 return -EINVAL;1043 return -EINVAL;
1044 }1044 }
1045 1045
1046 if (fmod_ret->nr_links) {1046 if (fmod_ret->nr_nodes) {
1047 branches = kcalloc(fmod_ret->nr_links, sizeof(u32), GFP_KERNEL);1047 branches = kcalloc(fmod_ret->nr_nodes, sizeof(u32), GFP_KERNEL);
1048 if (!branches)1048 if (!branches)
1049 return -ENOMEM;1049 return -ENOMEM;
1050 1050
@@ -1093,7 +1093,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 42 unmodified lines
1093 }1093 }
1094 1094
1095 /* Update branches saved in invoke_bpf_mod_ret with address of do_fexit */1095 /* Update branches saved in invoke_bpf_mod_ret with address of do_fexit */
1096 for (i = 0; i < fmod_ret->nr_links && image; i++) {1096 for (i = 0; i < fmod_ret->nr_nodes && image; i++) {
1097 if (create_cond_branch(&branch_insn, &image[branches[i]],1097 if (create_cond_branch(&branch_insn, &image[branches[i]],
1098 (unsigned long)&image[ctx->idx], COND_NE << 16)) {1098 (unsigned long)&image[ctx->idx], COND_NE << 16)) {
1099 ret = -EINVAL;1099 ret = -EINVAL;
@@ -1110,15 +1110,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 10 unmodified lines
1110 1110
1111 cookie_ctx_off = (regs_off - cookie_off) / 8;1111 cookie_ctx_off = (regs_off - cookie_off) / 8;
1112 1112
1113 for (i = 0; i < fexit->nr_links; i++) {1113 for (i = 0; i < fexit->nr_nodes; i++) {
1114 if (bpf_prog_calls_session_cookie(fexit->links[i])) {1114 if (bpf_prog_calls_session_cookie(fexit->nodes[i])) {
1115 u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);1115 u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
1116 1116
1117 store_func_meta(image, ctx, meta, func_meta_off);1117 store_func_meta(image, ctx, meta, func_meta_off);
1118 cookie_ctx_off--;1118 cookie_ctx_off--;
1119 }1119 }
1120 1120
1121 if (invoke_bpf_prog(image, ro_image, ctx, fexit->links[i], regs_off, retval_off,1121 if (invoke_bpf_prog(image, ro_image, ctx, fexit->nodes[i], regs_off, retval_off,
1122 run_ctx_off, false)) {1122 run_ctx_off, false)) {
1123 ret = -EINVAL;1123 ret = -EINVAL;
1124 goto cleanup;1124 goto cleanup;
@@ -1185,18 +1185,18 @@ cleanup: · 60 unmodified lines
1185}1185}
1186 1186
1187int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,1187int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
1188 struct bpf_tramp_links *tlinks, void *func_addr)1188 struct bpf_tramp_nodes *tnodes, void *func_addr)
1189{1189{
1190 struct bpf_tramp_image im;1190 struct bpf_tramp_image im;
1191 int ret;1191 int ret;
1192 1192
1193 ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tlinks, func_addr);1193 ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tnodes, func_addr);
1194 return ret;1194 return ret;
1195}1195}
1196 1196
1197int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,1197int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
1198 const struct btf_func_model *m, u32 flags,1198 const struct btf_func_model *m, u32 flags,
1199 struct bpf_tramp_links *tlinks,1199 struct bpf_tramp_nodes *tnodes,
1200 void *func_addr)1200 void *func_addr)
1201{1201{
1202 u32 size = image_end - image;1202 u32 size = image_end - image;
@@ -1212,7 +1212,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i · 9 unmodified lines
1212 return -ENOMEM;1212 return -ENOMEM;
1213 1213
1214 ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,1214 ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,
1215 flags, tlinks, func_addr);1215 flags, tnodes, func_addr);
1216 if (ret < 0)1216 if (ret < 0)
1217 goto out;1217 goto out;
1218 1218
arch/riscv/net/bpf_jit_comp64.c+35−26
@@ -934,15 +934,15 @@ static void emit_store_stack_imm64(u8 reg, int stack_off, u64 imm64, · 933 unmodified lines
934 emit_sd(RV_REG_FP, stack_off, reg, ctx);934 emit_sd(RV_REG_FP, stack_off, reg, ctx);
935}935}
936 936
937static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_off,937static int invoke_bpf_prog(struct bpf_tramp_node *node, int args_off, int retval_off,
938 int run_ctx_off, bool save_ret, struct rv_jit_context *ctx)938 int run_ctx_off, bool save_ret, struct rv_jit_context *ctx)
939{939{
940 int ret, branch_off;940 int ret, branch_off;
941 struct bpf_prog *p = l->link.prog;941 struct bpf_prog *p = node->link->prog;
942 int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);942 int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
943 943
944 if (l->cookie)944 if (node->cookie)
945 emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, l->cookie, ctx);945 emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, node->cookie, ctx);
946 else946 else
947 emit_sd(RV_REG_FP, -run_ctx_off + cookie_off, RV_REG_ZERO, ctx);947 emit_sd(RV_REG_FP, -run_ctx_off + cookie_off, RV_REG_ZERO, ctx);
948 948
@@ -996,22 +996,22 @@ static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_of · 47 unmodified lines
996 return ret;996 return ret;
997}997}
998 998
999static int invoke_bpf(struct bpf_tramp_links *tl, int args_off, int retval_off,999static int invoke_bpf(struct bpf_tramp_nodes *tn, int args_off, int retval_off,
1000 int run_ctx_off, int func_meta_off, bool save_ret, u64 func_meta,1000 int run_ctx_off, int func_meta_off, bool save_ret, u64 func_meta,
1001 int cookie_off, struct rv_jit_context *ctx)1001 int cookie_off, struct rv_jit_context *ctx)
1002{1002{
1003 int i, cur_cookie = (cookie_off - args_off) / 8;1003 int i, cur_cookie = (cookie_off - args_off) / 8;
1004 1004
1005 for (i = 0; i < tl->nr_links; i++) {1005 for (i = 0; i < tn->nr_nodes; i++) {
1006 int err;1006 int err;
1007 1007
1008 if (bpf_prog_calls_session_cookie(tl->links[i])) {1008 if (bpf_prog_calls_session_cookie(tn->nodes[i])) {
1009 u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);1009 u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);
1010 1010
1011 emit_store_stack_imm64(RV_REG_T1, -func_meta_off, meta, ctx);1011 emit_store_stack_imm64(RV_REG_T1, -func_meta_off, meta, ctx);
1012 cur_cookie--;1012 cur_cookie--;
1013 }1013 }
1014 err = invoke_bpf_prog(tl->links[i], args_off, retval_off, run_ctx_off,1014 err = invoke_bpf_prog(tn->nodes[i], args_off, retval_off, run_ctx_off,
1015 save_ret, ctx);1015 save_ret, ctx);
1016 if (err)1016 if (err)
1017 return err;1017 return err;
@@ -1021,7 +1021,7 @@ static int invoke_bpf(struct bpf_tramp_links *tl, int args_off, int retval_off, · 3 unmodified lines
1021 1021
1022static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,1022static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
1023 const struct btf_func_model *m,1023 const struct btf_func_model *m,
1024 struct bpf_tramp_links *tlinks,1024 struct bpf_tramp_nodes *tnodes,
1025 void *func_addr, u32 flags,1025 void *func_addr, u32 flags,
1026 struct rv_jit_context *ctx)1026 struct rv_jit_context *ctx)
1027{1027{
@@ -1030,9 +1030,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 2 unmodified lines
1030 int stack_size = 0, nr_arg_slots = 0;1030 int stack_size = 0, nr_arg_slots = 0;
1031 int retval_off, args_off, func_meta_off, ip_off, run_ctx_off, sreg_off, stk_arg_off;1031 int retval_off, args_off, func_meta_off, ip_off, run_ctx_off, sreg_off, stk_arg_off;
1032 int cookie_off, cookie_cnt;1032 int cookie_off, cookie_cnt;
1033 struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];1033 struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
1034 struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];1034 struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
1035 struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];1035 struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
1036 bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT;1036 bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT;
1037 void *orig_call = func_addr;1037 void *orig_call = func_addr;
1038 bool save_ret;1038 bool save_ret;
@@ -1115,7 +1115,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 76 unmodified lines
1115 ip_off = stack_size;1115 ip_off = stack_size;
1116 }1116 }
1117 1117
1118 cookie_cnt = bpf_fsession_cookie_cnt(tlinks);1118 cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
1119 /* room for session cookies */1119 /* room for session cookies */
1120 stack_size += cookie_cnt * 8;1120 stack_size += cookie_cnt * 8;
1121 cookie_off = stack_size;1121 cookie_off = stack_size;
@@ -1172,7 +1172,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 50 unmodified lines
1172 1172
1173 store_args(nr_arg_slots, args_off, ctx);1173 store_args(nr_arg_slots, args_off, ctx);
1174 1174
1175 if (bpf_fsession_cnt(tlinks)) {1175 if (bpf_fsession_cnt(tnodes)) {
1176 /* clear all session cookies' value */1176 /* clear all session cookies' value */
1177 for (i = 0; i < cookie_cnt; i++)1177 for (i = 0; i < cookie_cnt; i++)
1178 emit_sd(RV_REG_FP, -cookie_off + 8 * i, RV_REG_ZERO, ctx);1178 emit_sd(RV_REG_FP, -cookie_off + 8 * i, RV_REG_ZERO, ctx);
@@ -1187,22 +1187,22 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 8 unmodified lines
1187 return ret;1187 return ret;
1188 }1188 }
1189 1189
1190 if (fentry->nr_links) {1190 if (fentry->nr_nodes) {
1191 ret = invoke_bpf(fentry, args_off, retval_off, run_ctx_off, func_meta_off,1191 ret = invoke_bpf(fentry, args_off, retval_off, run_ctx_off, func_meta_off,
1192 flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off, ctx);1192 flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off, ctx);
1193 if (ret)1193 if (ret)
1194 return ret;1194 return ret;
1195 }1195 }
1196 1196
1197 if (fmod_ret->nr_links) {1197 if (fmod_ret->nr_nodes) {
1198 branches_off = kzalloc_objs(int, fmod_ret->nr_links);1198 branches_off = kzalloc_objs(int, fmod_ret->nr_nodes);
1199 if (!branches_off)1199 if (!branches_off)
1200 return -ENOMEM;1200 return -ENOMEM;
1201 1201
1202 /* cleanup to avoid garbage return value confusion */1202 /* cleanup to avoid garbage return value confusion */
1203 emit_sd(RV_REG_FP, -retval_off, RV_REG_ZERO, ctx);1203 emit_sd(RV_REG_FP, -retval_off, RV_REG_ZERO, ctx);
1204 for (i = 0; i < fmod_ret->nr_links; i++) {1204 for (i = 0; i < fmod_ret->nr_nodes; i++) {
1205 ret = invoke_bpf_prog(fmod_ret->links[i], args_off, retval_off,1205 ret = invoke_bpf_prog(fmod_ret->nodes[i], args_off, retval_off,
1206 run_ctx_off, true, ctx);1206 run_ctx_off, true, ctx);
1207 if (ret)1207 if (ret)
1208 goto out;1208 goto out;
@@ -1230,7 +1230,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 21 unmodified lines
1230 }1230 }
1231 1231
1232 /* update branches saved in invoke_bpf_mod_ret with bnez */1232 /* update branches saved in invoke_bpf_mod_ret with bnez */
1233 for (i = 0; ctx->insns && i < fmod_ret->nr_links; i++) {1233 for (i = 0; ctx->insns && i < fmod_ret->nr_nodes; i++) {
1234 offset = ninsns_rvoff(ctx->ninsns - branches_off[i]);1234 offset = ninsns_rvoff(ctx->ninsns - branches_off[i]);
1235 insn = rv_bne(RV_REG_T1, RV_REG_ZERO, offset >> 1);1235 insn = rv_bne(RV_REG_T1, RV_REG_ZERO, offset >> 1);
1236 *(u32 *)(ctx->insns + branches_off[i]) = insn;1236 *(u32 *)(ctx->insns + branches_off[i]) = insn;
@@ -1238,10 +1238,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 1 unmodified line
1238 1238
1239 /* set "is_return" flag for fsession */1239 /* set "is_return" flag for fsession */
1240 func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);1240 func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
1241 if (bpf_fsession_cnt(tlinks))1241 if (bpf_fsession_cnt(tnodes))
1242 emit_store_stack_imm64(RV_REG_T1, -func_meta_off, func_meta, ctx);1242 emit_store_stack_imm64(RV_REG_T1, -func_meta_off, func_meta, ctx);
1243 1243
1244 if (fexit->nr_links) {1244 if (fexit->nr_nodes) {
1245 ret = invoke_bpf(fexit, args_off, retval_off, run_ctx_off, func_meta_off,1245 ret = invoke_bpf(fexit, args_off, retval_off, run_ctx_off, func_meta_off,
1246 false, func_meta, cookie_off, ctx);1246 false, func_meta, cookie_off, ctx);
1247 if (ret)1247 if (ret)
@@ -1305,7 +1305,7 @@ out: · 57 unmodified lines
1305}1305}
1306 1306
1307int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,1307int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
1308 struct bpf_tramp_links *tlinks, void *func_addr)1308 struct bpf_tramp_nodes *tnodes, void *func_addr)
1309{1309{
1310 struct bpf_tramp_image im;1310 struct bpf_tramp_image im;
1311 struct rv_jit_context ctx;1311 struct rv_jit_context ctx;
@@ -1314,7 +1314,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 2 unmodified lines
1314 ctx.ninsns = 0;1314 ctx.ninsns = 0;
1315 ctx.insns = NULL;1315 ctx.insns = NULL;
1316 ctx.ro_insns = NULL;1316 ctx.ro_insns = NULL;
1317 ret = __arch_prepare_bpf_trampoline(&im, m, tlinks, func_addr, flags, &ctx);1317 ret = __arch_prepare_bpf_trampoline(&im, m, tnodes, func_addr, flags, &ctx);
1318 1318
1319 return ret < 0 ? ret : ninsns_rvoff(ctx.ninsns);1319 return ret < 0 ? ret : ninsns_rvoff(ctx.ninsns);
1320}1320}
@@ -1331,7 +1331,7 @@ void arch_free_bpf_trampoline(void *image, unsigned int size) · 10 unmodified lines
1331 1331
1332int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,1332int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
1333 void *ro_image_end, const struct btf_func_model *m,1333 void *ro_image_end, const struct btf_func_model *m,
1334 u32 flags, struct bpf_tramp_links *tlinks,1334 u32 flags, struct bpf_tramp_nodes *tnodes,
1335 void *func_addr)1335 void *func_addr)
1336{1336{
1337 int ret;1337 int ret;
@@ -1346,7 +1346,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · 8 unmodified lines
1346 ctx.ninsns = 0;1346 ctx.ninsns = 0;
1347 ctx.insns = image;1347 ctx.insns = image;
1348 ctx.ro_insns = ro_image;1348 ctx.ro_insns = ro_image;
1349 ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx);1349 ret = __arch_prepare_bpf_trampoline(im, m, tnodes, func_addr, flags, &ctx);
1350 if (ret < 0)1350 if (ret < 0)
1351 goto out;1351 goto out;
1352 1352
@@ -1808,6 +1808,13 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · 455 unmodified lines
1808 break;1808 break;
1809 }1809 }
1810 1810
1811 /* Implement helper call to bpf_get_current_task/_btf() inline */
1812 if (insn->src_reg == 0 && (insn->imm == BPF_FUNC_get_current_task ||
1813 insn->imm == BPF_FUNC_get_current_task_btf)) {
1814 emit_mv(bpf_to_rv_reg(BPF_REG_0, ctx), RV_REG_TP, ctx);
1815 break;
1816 }
1817
1811 mark_call(ctx);1818 mark_call(ctx);
1812 ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass,1819 ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass,
1813 &addr, &fixed_addr);1820 &addr, &fixed_addr);
@@ -2138,6 +2145,8 @@ bool bpf_jit_inlines_helper_call(s32 imm) · 324 unmodified lines
2138{2145{
2139 switch (imm) {2146 switch (imm) {
2140 case BPF_FUNC_get_smp_processor_id:2147 case BPF_FUNC_get_smp_processor_id:
2148 case BPF_FUNC_get_current_task:
2149 case BPF_FUNC_get_current_task_btf:
2141 return true;2150 return true;
2142 default:2151 default:
2143 return false;2152 return false;
arch/s390/net/bpf_jit_comp.c+59−22
@@ -27,6 +27,7 @@ · 26 unmodified lines
27#include <asm/extable.h>27#include <asm/extable.h>
28#include <asm/dis.h>28#include <asm/dis.h>
29#include <asm/facility.h>29#include <asm/facility.h>
30#include <asm/lowcore.h>
30#include <asm/nospec-branch.h>31#include <asm/nospec-branch.h>
31#include <asm/set_memory.h>32#include <asm/set_memory.h>
32#include <asm/text-patching.h>33#include <asm/text-patching.h>
@@ -1777,6 +1778,30 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, · 1744 unmodified lines
1777 int j, ret;1778 int j, ret;
1778 u64 func;1779 u64 func;
1779 1780
1781 /* Implement helper call to bpf_get_smp_processor_id() inline */
1782 if (insn->src_reg == 0 &&
1783 insn->imm == BPF_FUNC_get_smp_processor_id) {
1784 const u32 *cpu_nr = &get_lowcore()->cpu_nr;
1785
1786 /* ly %b0, cpu_nr */
1787 EMIT6_DISP_LH(0xe3000000, 0x0058, BPF_REG_0, REG_0, REG_0,
1788 (unsigned long)cpu_nr);
1789 break;
1790 }
1791
1792 /* Implement helper call to bpf_get_current_task/_btf() inline */
1793 if (insn->src_reg == 0 &&
1794 (insn->imm == BPF_FUNC_get_current_task ||
1795 insn->imm == BPF_FUNC_get_current_task_btf)) {
1796 const u64 *current_task =
1797 &get_lowcore()->current_task;
1798
1799 /* lg %b0, current_task */
1800 EMIT6_DISP_LH(0xe3000000, 0x0004, BPF_REG_0, REG_0, REG_0,
1801 (unsigned long)current_task);
1802 break;
1803 }
1804
1780 ret = bpf_jit_get_func_addr(fp, insn, extra_pass,1805 ret = bpf_jit_get_func_addr(fp, insn, extra_pass,
1781 &func, &func_addr_fixed);1806 &func, &func_addr_fixed);
1782 if (ret < 0)1807 if (ret < 0)
@@ -2512,19 +2537,19 @@ static void emit_store_stack_imm64(struct bpf_jit *jit, int tmp_reg, int stack_o · 729 unmodified lines
2512 2537
2513static int invoke_bpf_prog(struct bpf_tramp_jit *tjit,2538static int invoke_bpf_prog(struct bpf_tramp_jit *tjit,
2514 const struct btf_func_model *m,2539 const struct btf_func_model *m,
2515 struct bpf_tramp_link *tlink, bool save_ret)2540 struct bpf_tramp_node *node, bool save_ret)
2516{2541{
2517 struct bpf_jit *jit = &tjit->common;2542 struct bpf_jit *jit = &tjit->common;
2518 int cookie_off = tjit->run_ctx_off +2543 int cookie_off = tjit->run_ctx_off +
2519 offsetof(struct bpf_tramp_run_ctx, bpf_cookie);2544 offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
2520 struct bpf_prog *p = tlink->link.prog;2545 struct bpf_prog *p = node->link->prog;
2521 int patch;2546 int patch;
2522 2547
2523 /*2548 /*
2524 * run_ctx.cookie = tlink->cookie;2549 * run_ctx.cookie = node->cookie;
2525 */2550 */
2526 2551
2527 emit_store_stack_imm64(jit, REG_W0, cookie_off, tlink->cookie);2552 emit_store_stack_imm64(jit, REG_W0, cookie_off, node->cookie);
2528 2553
2529 /*2554 /*
2530 * if ((start = __bpf_prog_enter(p, &run_ctx)) == 0)2555 * if ((start = __bpf_prog_enter(p, &run_ctx)) == 0)
@@ -2584,20 +2609,20 @@ static int invoke_bpf_prog(struct bpf_tramp_jit *tjit, · 53 unmodified lines
2584 2609
2585static int invoke_bpf(struct bpf_tramp_jit *tjit,2610static int invoke_bpf(struct bpf_tramp_jit *tjit,
2586 const struct btf_func_model *m,2611 const struct btf_func_model *m,
2587 struct bpf_tramp_links *tl, bool save_ret,2612 struct bpf_tramp_nodes *tn, bool save_ret,
2588 u64 func_meta, int cookie_off)2613 u64 func_meta, int cookie_off)
2589{2614{
2590 int i, cur_cookie = (tjit->bpf_args_off - cookie_off) / sizeof(u64);2615 int i, cur_cookie = (tjit->bpf_args_off - cookie_off) / sizeof(u64);
2591 struct bpf_jit *jit = &tjit->common;2616 struct bpf_jit *jit = &tjit->common;
2592 2617
2593 for (i = 0; i < tl->nr_links; i++) {2618 for (i = 0; i < tn->nr_nodes; i++) {
2594 if (bpf_prog_calls_session_cookie(tl->links[i])) {2619 if (bpf_prog_calls_session_cookie(tn->nodes[i])) {
2595 u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);2620 u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);
2596 2621
2597 emit_store_stack_imm64(jit, REG_0, tjit->func_meta_off, meta);2622 emit_store_stack_imm64(jit, REG_0, tjit->func_meta_off, meta);
2598 cur_cookie--;2623 cur_cookie--;
2599 }2624 }
2600 if (invoke_bpf_prog(tjit, m, tl->links[i], save_ret))2625 if (invoke_bpf_prog(tjit, m, tn->nodes[i], save_ret))
2601 return -EINVAL;2626 return -EINVAL;
2602 }2627 }
2603 2628
@@ -2626,12 +2651,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 22 unmodified lines
2626 struct bpf_tramp_jit *tjit,2651 struct bpf_tramp_jit *tjit,
2627 const struct btf_func_model *m,2652 const struct btf_func_model *m,
2628 u32 flags,2653 u32 flags,
2629 struct bpf_tramp_links *tlinks,2654 struct bpf_tramp_nodes *tnodes,
2630 void *func_addr)2655 void *func_addr)
2631{2656{
2632 struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];2657 struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
2633 struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];2658 struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
2634 struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];2659 struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
2635 int nr_bpf_args, nr_reg_args, nr_stack_args;2660 int nr_bpf_args, nr_reg_args, nr_stack_args;
2636 int cookie_cnt, cookie_off, fsession_cnt;2661 int cookie_cnt, cookie_off, fsession_cnt;
2637 struct bpf_jit *jit = &tjit->common;2662 struct bpf_jit *jit = &tjit->common;
@@ -2668,8 +2693,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 30 unmodified lines
2668 return -ENOTSUPP;2693 return -ENOTSUPP;
2669 }2694 }
2670 2695
2671 cookie_cnt = bpf_fsession_cookie_cnt(tlinks);2696 cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
2672 fsession_cnt = bpf_fsession_cnt(tlinks);2697 fsession_cnt = bpf_fsession_cnt(tnodes);
2673 2698
2674 /*2699 /*
2675 * Calculate the stack layout.2700 * Calculate the stack layout.
@@ -2804,7 +2829,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 128 unmodified lines
2804 func_meta, cookie_off))2829 func_meta, cookie_off))
2805 return -EINVAL;2830 return -EINVAL;
2806 2831
2807 if (fmod_ret->nr_links) {2832 if (fmod_ret->nr_nodes) {
2808 /*2833 /*
2809 * retval = 0;2834 * retval = 0;
2810 */2835 */
@@ -2813,8 +2838,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 2 unmodified lines
2813 _EMIT6(0xd707f000 | tjit->retval_off,2838 _EMIT6(0xd707f000 | tjit->retval_off,
2814 0xf000 | tjit->retval_off);2839 0xf000 | tjit->retval_off);
2815 2840
2816 for (i = 0; i < fmod_ret->nr_links; i++) {2841 for (i = 0; i < fmod_ret->nr_nodes; i++) {
2817 if (invoke_bpf_prog(tjit, m, fmod_ret->links[i], true))2842 if (invoke_bpf_prog(tjit, m, fmod_ret->nodes[i], true))
2818 return -EINVAL;2843 return -EINVAL;
2819 2844
2820 /*2845 /*
@@ -2939,7 +2964,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 118 unmodified lines
2939}2964}
2940 2965
2941int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,2966int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
2942 struct bpf_tramp_links *tlinks, void *orig_call)2967 struct bpf_tramp_nodes *tnodes, void *orig_call)
2943{2968{
2944 struct bpf_tramp_image im;2969 struct bpf_tramp_image im;
2945 struct bpf_tramp_jit tjit;2970 struct bpf_tramp_jit tjit;
@@ -2948,14 +2973,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 2 unmodified lines
2948 memset(&tjit, 0, sizeof(tjit));2973 memset(&tjit, 0, sizeof(tjit));
2949 2974
2950 ret = __arch_prepare_bpf_trampoline(&im, &tjit, m, flags,2975 ret = __arch_prepare_bpf_trampoline(&im, &tjit, m, flags,
2951 tlinks, orig_call);2976 tnodes, orig_call);
2952 2977
2953 return ret < 0 ? ret : tjit.common.prg;2978 return ret < 0 ? ret : tjit.common.prg;
2954}2979}
2955 2980
2956int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image,2981int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image,
2957 void *image_end, const struct btf_func_model *m,2982 void *image_end, const struct btf_func_model *m,
2958 u32 flags, struct bpf_tramp_links *tlinks,2983 u32 flags, struct bpf_tramp_nodes *tnodes,
2959 void *func_addr)2984 void *func_addr)
2960{2985{
2961 struct bpf_tramp_jit tjit;2986 struct bpf_tramp_jit tjit;
@@ -2964,7 +2989,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, · 2 unmodified lines
2964 /* Compute offsets, check whether the code fits. */2989 /* Compute offsets, check whether the code fits. */
2965 memset(&tjit, 0, sizeof(tjit));2990 memset(&tjit, 0, sizeof(tjit));
2966 ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags,2991 ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags,
2967 tlinks, func_addr);2992 tnodes, func_addr);
2968 2993
2969 if (ret < 0)2994 if (ret < 0)
2970 return ret;2995 return ret;
@@ -2978,7 +3003,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, · 7 unmodified lines
2978 tjit.common.prg = 0;3003 tjit.common.prg = 0;
2979 tjit.common.prg_buf = image;3004 tjit.common.prg_buf = image;
2980 ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags,3005 ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags,
2981 tlinks, func_addr);3006 tnodes, func_addr);
2982 3007
2983 return ret < 0 ? ret : tjit.common.prg;3008 return ret < 0 ? ret : tjit.common.prg;
2984}3009}
@@ -3057,3 +3082,15 @@ bool bpf_jit_supports_timed_may_goto(void) · 72 unmodified lines
3057{3082{
3058 return true;3083 return true;
3059}3084}
3085
3086bool bpf_jit_inlines_helper_call(s32 imm)
3087{
3088 switch (imm) {
3089 case BPF_FUNC_get_smp_processor_id:
3090 case BPF_FUNC_get_current_task:
3091 case BPF_FUNC_get_current_task_btf:
3092 return true;
3093 default:
3094 return false;
3095 }
3096}
arch/x86/include/asm/pgtable.h+12−0
@@ -1284,6 +1284,18 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, · 1283 unmodified lines
1284 } while (!try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte));1284 } while (!try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte));
1285}1285}
1286 1286
1287/*
1288 * Note: strictly-zero compare is narrower than pte_none(), but the gap is
1289 * harmless: _PAGE_DIRTY and _PAGE_ACCESSED aren't set on untouched kernel PTEs.
1290 */
1291static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte)
1292{
1293 pte_t old_pte = __pte(0);
1294
1295 return try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte);
1296}
1297#define ptep_try_set ptep_try_set
1298
1287#define flush_tlb_fix_spurious_fault(vma, address, ptep) do { } while (0)1299#define flush_tlb_fix_spurious_fault(vma, address, ptep) do { } while (0)
1288 1300
1289#define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS1301#define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS
arch/x86/mm/fault.c+8−4
@@ -8,6 +8,7 @@ · 7 unmodified lines
8#include <linux/sched/task_stack.h> /* task_stack_*(), ... */8#include <linux/sched/task_stack.h> /* task_stack_*(), ... */
9#include <linux/kdebug.h> /* oops_begin/end, ... */9#include <linux/kdebug.h> /* oops_begin/end, ... */
10#include <linux/memblock.h> /* max_low_pfn */10#include <linux/memblock.h> /* max_low_pfn */
11#include <linux/bpf_defs.h> /* bpf_arena_handle_page_fault */
11#include <linux/kfence.h> /* kfence_handle_page_fault */12#include <linux/kfence.h> /* kfence_handle_page_fault */
12#include <linux/kprobes.h> /* NOKPROBE_SYMBOL, ... */13#include <linux/kprobes.h> /* NOKPROBE_SYMBOL, ... */
13#include <linux/mmiotrace.h> /* kmmio_handler, ... */14#include <linux/mmiotrace.h> /* kmmio_handler, ... */
@@ -693,10 +694,13 @@ page_fault_oops(struct pt_regs *regs, unsigned long error_code, · 679 unmodified lines
693 if (IS_ENABLED(CONFIG_EFI))694 if (IS_ENABLED(CONFIG_EFI))
694 efi_crash_gracefully_on_page_fault(address, regs);695 efi_crash_gracefully_on_page_fault(address, regs);
695 696
696 /* Only not-present faults should be handled by KFENCE. */697 /* Only not-present faults should be handled by KFENCE or BPF arena. */
697 if (!(error_code & X86_PF_PROT) &&698 if (!(error_code & X86_PF_PROT)) {
698 kfence_handle_page_fault(address, error_code & X86_PF_WRITE, regs))699 if (kfence_handle_page_fault(address, error_code & X86_PF_WRITE, regs))
699 return;700 return;
701 if (bpf_arena_handle_page_fault(address, error_code & X86_PF_WRITE, regs->ip))
702 return;
703 }
700 704
701oops:705oops:
702 /*706 /*
arch/x86/net/bpf_jit_comp.c+172−33
@@ -10,6 +10,7 @@ · 9 unmodified lines
10#include <linux/if_vlan.h>10#include <linux/if_vlan.h>
11#include <linux/bitfield.h>11#include <linux/bitfield.h>
12#include <linux/bpf.h>12#include <linux/bpf.h>
13#include <linux/bpf_verifier.h>
13#include <linux/memory.h>14#include <linux/memory.h>
14#include <linux/sort.h>15#include <linux/sort.h>
15#include <asm/extable.h>16#include <asm/extable.h>
@@ -390,6 +391,34 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) · 374 unmodified lines
390 *pprog = prog;391 *pprog = prog;
391}392}
392 393
394/* add rsp, depth */
395static void emit_add_rsp(u8 **pprog, u16 depth)
396{
397 u8 *prog = *pprog;
398
399 if (!depth)
400 return;
401 if (is_imm8(depth))
402 EMIT4(0x48, 0x83, 0xC4, depth); /* add rsp, imm8 */
403 else
404 EMIT3_off32(0x48, 0x81, 0xC4, depth); /* add rsp, imm32 */
405 *pprog = prog;
406}
407
408/* sub rsp, depth */
409static void emit_sub_rsp(u8 **pprog, u16 depth)
410{
411 u8 *prog = *pprog;
412
413 if (!depth)
414 return;
415 if (is_imm8(depth))
416 EMIT4(0x48, 0x83, 0xEC, depth); /* sub rsp, imm8 */
417 else
418 EMIT3_off32(0x48, 0x81, 0xEC, depth); /* sub rsp, imm32 */
419 *pprog = prog;
420}
421
393static void emit_nops(u8 **pprog, int len)422static void emit_nops(u8 **pprog, int len)
394{423{
395 u8 *prog = *pprog;424 u8 *prog = *pprog;
@@ -1659,21 +1688,47 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 1263 unmodified lines
1659 bool seen_exit = false;1688 bool seen_exit = false;
1660 u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY];1689 u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY];
1661 void __percpu *priv_frame_ptr = NULL;1690 void __percpu *priv_frame_ptr = NULL;
1691 u16 out_stack_arg_cnt, outgoing_rsp;
1662 u64 arena_vm_start, user_vm_start;1692 u64 arena_vm_start, user_vm_start;
1663 void __percpu *priv_stack_ptr;1693 void __percpu *priv_stack_ptr;
1664 int i, excnt = 0;1694 int i, excnt = 0;
1665 int ilen, proglen = 0;1695 int ilen, proglen = 0;
1666 u8 *ip, *prog = temp;1696 u8 *ip, *prog = temp;
1667 u32 stack_depth;1697 u32 stack_depth;
1698 int callee_saved_size;
1699 s32 outgoing_arg_base;
1668 int err;1700 int err;
1669 1701
1670 stack_depth = bpf_prog->aux->stack_depth;1702 stack_depth = bpf_prog->aux->stack_depth;
1703 out_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog);
1671 priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;1704 priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
1672 if (priv_stack_ptr) {1705 if (priv_stack_ptr) {
1673 priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);1706 priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
1674 stack_depth = 0;1707 stack_depth = 0;
1675 }1708 }
1676 1709
1710 /*
1711 * Follow x86-64 calling convention for both BPF-to-BPF and
1712 * kfunc calls:
1713 * - Arg 6 is passed in R9 register
1714 * - Args 7+ are passed on the stack at [rsp]
1715 *
1716 * Incoming arg 6 is read from R9 (BPF r11+8 → MOV from R9).
1717 * Incoming args 7+ are read from [rbp + 16], [rbp + 24], ...
1718 * (BPF r11+16, r11+24, ... map directly with no offset change).
1719 *
1720 * tail_call_reachable is rejected by the verifier and priv_stack
1721 * is disabled by the JIT when stack args exist, so R9 is always
1722 * available.
1723 *
1724 * Stack layout (high to low):
1725 * [rbp + 16 + ...] incoming stack args 7+ (from caller)
1726 * [rbp + 8] return address
1727 * [rbp] saved rbp
1728 * [rbp - prog_stack] program stack
1729 * [below] callee-saved regs
1730 * [below] outgoing args 7+ (= rsp)
1731 */
1677 arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);1732 arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);
1678 user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);1733 user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);
1679 1734
@@ -1700,6 +1755,44 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 20 unmodified lines
1700 push_r12(&prog);1755 push_r12(&prog);
1701 push_callee_regs(&prog, callee_regs_used);1756 push_callee_regs(&prog, callee_regs_used);
1702 }1757 }
1758
1759 /* Compute callee-saved register area size. */
1760 callee_saved_size = 0;
1761 if (bpf_prog->aux->exception_boundary || arena_vm_start)
1762 callee_saved_size += 8; /* r12 */
1763 if (bpf_prog->aux->exception_boundary) {
1764 callee_saved_size += 4 * 8; /* rbx, r13, r14, r15 */
1765 } else {
1766 int j;
1767
1768 for (j = 0; j < 4; j++)
1769 if (callee_regs_used[j])
1770 callee_saved_size += 8;
1771 }
1772 /*
1773 * Base offset from rbp for translating BPF outgoing args 7+
1774 * to native offsets. BPF uses negative offsets from r11
1775 * (r11-8 for arg6, r11-16 for arg7, ...) while x86 uses
1776 * positive offsets from rsp ([rsp+0] for arg7, [rsp+8] for
1777 * arg8, ...). Arg 6 goes to R9 directly.
1778 *
1779 * The translation reverses direction:
1780 * native_off = outgoing_arg_base - outgoing_rsp - bpf_off - 16
1781 *
1782 * Note that tail_call_reachable is guaranteed to be false when
1783 * stack args exist, so tcc pushes need not be accounted for.
1784 */
1785 outgoing_arg_base = -(round_up(stack_depth, 8) + callee_saved_size);
1786
1787 /*
1788 * Allocate outgoing stack arg area for args 7+ only.
1789 * Arg 6 goes into r9 register, not on stack.
1790 */
1791 outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
1792 if (bpf_prog->aux->exception_boundary)
1793 bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp;
1794 emit_sub_rsp(&prog, outgoing_rsp);
1795
1703 if (arena_vm_start)1796 if (arena_vm_start)
1704 emit_mov_imm64(&prog, X86_REG_R12,1797 emit_mov_imm64(&prog, X86_REG_R12,
1705 arena_vm_start >> 32, (u32) arena_vm_start);1798 arena_vm_start >> 32, (u32) arena_vm_start);
@@ -1721,7 +1814,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 15 unmodified lines
1721 u8 b2 = 0, b3 = 0;1814 u8 b2 = 0, b3 = 0;
1722 u8 *start_of_ldx;1815 u8 *start_of_ldx;
1723 s64 jmp_offset;1816 s64 jmp_offset;
1724 s16 insn_off;1817 s32 insn_off;
1725 u8 jmp_cond;1818 u8 jmp_cond;
1726 u8 *func;1819 u8 *func;
1727 int nops;1820 int nops;
@@ -2134,12 +2227,27 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 406 unmodified lines
2134 EMIT1(0xC7);2227 EMIT1(0xC7);
2135 goto st;2228 goto st;
2136 case BPF_ST | BPF_MEM | BPF_DW:2229 case BPF_ST | BPF_MEM | BPF_DW:
2230 if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
2231 /* Arg 6: store immediate in r9 register */
2232 emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31, (u32)imm32);
2233 break;
2234 }
2137 EMIT2(add_1mod(0x48, dst_reg), 0xC7);2235 EMIT2(add_1mod(0x48, dst_reg), 0xC7);
2138 2236
2139st: if (is_imm8(insn->off))2237st: insn_off = insn->off;
2140 EMIT2(add_1reg(0x40, dst_reg), insn->off);2238 if (dst_reg == BPF_REG_PARAMS) {
2239 /*
2240 * Args 7+: reverse BPF negative offsets to
2241 * x86 positive rsp offsets.
2242 * BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ...
2243 */
2244 insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
2245 dst_reg = BPF_REG_FP;
2246 }
2247 if (is_imm8(insn_off))
2248 EMIT2(add_1reg(0x40, dst_reg), insn_off);
2141 else2249 else
2142 EMIT1_off32(add_1reg(0x80, dst_reg), insn->off);2250 EMIT1_off32(add_1reg(0x80, dst_reg), insn_off);
2143 2251
2144 EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code)));2252 EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code)));
2145 break;2253 break;
@@ -2149,7 +2257,17 @@ st: if (is_imm8(insn->off)) · 3 unmodified lines
2149 case BPF_STX | BPF_MEM | BPF_H:2257 case BPF_STX | BPF_MEM | BPF_H:
2150 case BPF_STX | BPF_MEM | BPF_W:2258 case BPF_STX | BPF_MEM | BPF_W:
2151 case BPF_STX | BPF_MEM | BPF_DW:2259 case BPF_STX | BPF_MEM | BPF_DW:
2152 emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);2260 if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
2261 /* Arg 6: store register value in r9 */
2262 EMIT_mov(X86_REG_R9, src_reg);
2263 break;
2264 }
2265 insn_off = insn->off;
2266 if (dst_reg == BPF_REG_PARAMS) {
2267 insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
2268 dst_reg = BPF_REG_FP;
2269 }
2270 emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
2153 break;2271 break;
2154 2272
2155 case BPF_ST | BPF_PROBE_MEM32 | BPF_B:2273 case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
@@ -2248,6 +2366,19 @@ populate_extable: · 92 unmodified lines
2248 case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:2366 case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:
2249 case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:2367 case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
2250 insn_off = insn->off;2368 insn_off = insn->off;
2369 if (src_reg == BPF_REG_PARAMS) {
2370 if (insn_off == 8) {
2371 /* Incoming arg 6: read from r9 */
2372 EMIT_mov(dst_reg, X86_REG_R9);
2373 break;
2374 }
2375 src_reg = BPF_REG_FP;
2376 /*
2377 * Incoming args 7+: native_off == bpf_off
2378 * (r11+16 → [rbp+16], r11+24 → [rbp+24], ...)
2379 * No offset adjustment needed.
2380 */
2381 }
2251 2382
2252 if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||2383 if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||
2253 BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {2384 BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {
@@ -2736,6 +2867,8 @@ emit_jmp: · 482 unmodified lines
2736 if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))2867 if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
2737 return -EINVAL;2868 return -EINVAL;
2738 }2869 }
2870 /* Deallocate outgoing args 7+ area. */
2871 emit_add_rsp(&prog, outgoing_rsp);
2739 if (bpf_prog->aux->exception_boundary) {2872 if (bpf_prog->aux->exception_boundary) {
2740 pop_callee_regs(&prog, all_callee_regs_used);2873 pop_callee_regs(&prog, all_callee_regs_used);
2741 pop_r12(&prog);2874 pop_r12(&prog);
@@ -2971,15 +3104,15 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog, · 229 unmodified lines
2971}3104}
2972 3105
2973static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,3106static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
2974 struct bpf_tramp_link *l, int stack_size,3107 struct bpf_tramp_node *node, int stack_size,
2975 int run_ctx_off, bool save_ret,3108 int run_ctx_off, bool save_ret,
2976 void *image, void *rw_image)3109 void *image, void *rw_image)
2977{3110{
2978 u8 *prog = *pprog;3111 u8 *prog = *pprog;
2979 u8 *jmp_insn;3112 u8 *jmp_insn;
2980 int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);3113 int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
2981 struct bpf_prog *p = l->link.prog;3114 struct bpf_prog *p = node->link->prog;
2982 u64 cookie = l->cookie;3115 u64 cookie = node->cookie;
2983 3116
2984 /* mov rdi, cookie */3117 /* mov rdi, cookie */
2985 emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie);3118 emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie);
@@ -3086,7 +3219,7 @@ static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond) · 100 unmodified lines
3086}3219}
3087 3220
3088static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,3221static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
3089 struct bpf_tramp_links *tl, int stack_size,3222 struct bpf_tramp_nodes *tl, int stack_size,
3090 int run_ctx_off, int func_meta_off, bool save_ret,3223 int run_ctx_off, int func_meta_off, bool save_ret,
3091 void *image, void *rw_image, u64 func_meta,3224 void *image, void *rw_image, u64 func_meta,
3092 int cookie_off)3225 int cookie_off)
@@ -3094,13 +3227,13 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, · 1 unmodified line
3094 int i, cur_cookie = (cookie_off - stack_size) / 8;3227 int i, cur_cookie = (cookie_off - stack_size) / 8;
3095 u8 *prog = *pprog;3228 u8 *prog = *pprog;
3096 3229
3097 for (i = 0; i < tl->nr_links; i++) {3230 for (i = 0; i < tl->nr_nodes; i++) {
3098 if (tl->links[i]->link.prog->call_session_cookie) {3231 if (tl->nodes[i]->link->prog->call_session_cookie) {
3099 emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off,3232 emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off,
3100 func_meta | (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT));3233 func_meta | (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT));
3101 cur_cookie--;3234 cur_cookie--;
3102 }3235 }
3103 if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size,3236 if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size,
3104 run_ctx_off, save_ret, image, rw_image))3237 run_ctx_off, save_ret, image, rw_image))
3105 return -EINVAL;3238 return -EINVAL;
3106 }3239 }
@@ -3109,7 +3242,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, · 2 unmodified lines
3109}3242}
3110 3243
3111static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,3244static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
3112 struct bpf_tramp_links *tl, int stack_size,3245 struct bpf_tramp_nodes *tl, int stack_size,
3113 int run_ctx_off, u8 **branches,3246 int run_ctx_off, u8 **branches,
3114 void *image, void *rw_image)3247 void *image, void *rw_image)
3115{3248{
@@ -3121,8 +3254,8 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, · 5 unmodified lines
3121 */3254 */
3122 emit_mov_imm32(&prog, false, BPF_REG_0, 0);3255 emit_mov_imm32(&prog, false, BPF_REG_0, 0);
3123 emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);3256 emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
3124 for (i = 0; i < tl->nr_links; i++) {3257 for (i = 0; i < tl->nr_nodes; i++) {
3125 if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size, run_ctx_off, true,3258 if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true,
3126 image, rw_image))3259 image, rw_image))
3127 return -EINVAL;3260 return -EINVAL;
3128 3261
@@ -3213,14 +3346,14 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, · 84 unmodified lines
3213static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,3346static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,
3214 void *rw_image_end, void *image,3347 void *rw_image_end, void *image,
3215 const struct btf_func_model *m, u32 flags,3348 const struct btf_func_model *m, u32 flags,
3216 struct bpf_tramp_links *tlinks,3349 struct bpf_tramp_nodes *tnodes,
3217 void *func_addr)3350 void *func_addr)
3218{3351{
3219 int i, ret, nr_regs = m->nr_args, stack_size = 0;3352 int i, ret, nr_regs = m->nr_args, stack_size = 0;
3220 int regs_off, func_meta_off, ip_off, run_ctx_off, arg_stack_off, rbx_off;3353 int regs_off, func_meta_off, ip_off, run_ctx_off, arg_stack_off, rbx_off;
3221 struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];3354 struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
3222 struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];3355 struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
3223 struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];3356 struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
3224 void *orig_call = func_addr;3357 void *orig_call = func_addr;
3225 int cookie_off, cookie_cnt;3358 int cookie_off, cookie_cnt;
3226 u8 **branches = NULL;3359 u8 **branches = NULL;
@@ -3292,7 +3425,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 65 unmodified lines
3292 3425
3293 ip_off = stack_size;3426 ip_off = stack_size;
3294 3427
3295 cookie_cnt = bpf_fsession_cookie_cnt(tlinks);3428 cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
3296 /* room for session cookies */3429 /* room for session cookies */
3297 stack_size += cookie_cnt * 8;3430 stack_size += cookie_cnt * 8;
3298 cookie_off = stack_size;3431 cookie_off = stack_size;
@@ -3385,7 +3518,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 86 unmodified lines
3385 }3518 }
3386 }3519 }
3387 3520
3388 if (bpf_fsession_cnt(tlinks)) {3521 if (bpf_fsession_cnt(tnodes)) {
3389 /* clear all the session cookies' value */3522 /* clear all the session cookies' value */
3390 for (int i = 0; i < cookie_cnt; i++)3523 for (int i = 0; i < cookie_cnt; i++)
3391 emit_store_stack_imm64(&prog, BPF_REG_0, -cookie_off + 8 * i, 0);3524 emit_store_stack_imm64(&prog, BPF_REG_0, -cookie_off + 8 * i, 0);
@@ -3393,15 +3526,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 1 unmodified line
3393 emit_store_stack_imm64(&prog, BPF_REG_0, -8, 0);3526 emit_store_stack_imm64(&prog, BPF_REG_0, -8, 0);
3394 }3527 }
3395 3528
3396 if (fentry->nr_links) {3529 if (fentry->nr_nodes) {
3397 if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off,3530 if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off,
3398 flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image,3531 flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image,
3399 func_meta, cookie_off))3532 func_meta, cookie_off))
3400 return -EINVAL;3533 return -EINVAL;
3401 }3534 }
3402 3535
3403 if (fmod_ret->nr_links) {3536 if (fmod_ret->nr_nodes) {
3404 branches = kcalloc(fmod_ret->nr_links, sizeof(u8 *),3537 branches = kcalloc(fmod_ret->nr_nodes, sizeof(u8 *),
3405 GFP_KERNEL);3538 GFP_KERNEL);
3406 if (!branches)3539 if (!branches)
3407 return -ENOMEM;3540 return -ENOMEM;
@@ -3440,7 +3573,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 32 unmodified lines
3440 emit_nops(&prog, X86_PATCH_SIZE);3573 emit_nops(&prog, X86_PATCH_SIZE);
3441 }3574 }
3442 3575
3443 if (fmod_ret->nr_links) {3576 if (fmod_ret->nr_nodes) {
3444 /* From Intel 64 and IA-32 Architectures Optimization3577 /* From Intel 64 and IA-32 Architectures Optimization
3445 * Reference Manual, 3.4.1.4 Code Alignment, Assembly/Compiler3578 * Reference Manual, 3.4.1.4 Code Alignment, Assembly/Compiler
3446 * Coding Rule 11: All branch targets should be 16-byte3579 * Coding Rule 11: All branch targets should be 16-byte
@@ -3450,7 +3583,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 3 unmodified lines
3450 /* Update the branches saved in invoke_bpf_mod_ret with the3583 /* Update the branches saved in invoke_bpf_mod_ret with the
3451 * aligned address of do_fexit.3584 * aligned address of do_fexit.
3452 */3585 */
3453 for (i = 0; i < fmod_ret->nr_links; i++) {3586 for (i = 0; i < fmod_ret->nr_nodes; i++) {
3454 emit_cond_near_jump(&branches[i], image + (prog - (u8 *)rw_image),3587 emit_cond_near_jump(&branches[i], image + (prog - (u8 *)rw_image),
3455 image + (branches[i] - (u8 *)rw_image), X86_JNE);3588 image + (branches[i] - (u8 *)rw_image), X86_JNE);
3456 }3589 }
@@ -3458,10 +3591,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 1 unmodified line
3458 3591
3459 /* set the "is_return" flag for fsession */3592 /* set the "is_return" flag for fsession */
3460 func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);3593 func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
3461 if (bpf_fsession_cnt(tlinks))3594 if (bpf_fsession_cnt(tnodes))
3462 emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta);3595 emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta);
3463 3596
3464 if (fexit->nr_links) {3597 if (fexit->nr_nodes) {
3465 if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off,3598 if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off,
3466 false, image, rw_image, func_meta, cookie_off)) {3599 false, image, rw_image, func_meta, cookie_off)) {
3467 ret = -EINVAL;3600 ret = -EINVAL;
@@ -3535,7 +3668,7 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) · 67 unmodified lines
3535 3668
3536int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,3669int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
3537 const struct btf_func_model *m, u32 flags,3670 const struct btf_func_model *m, u32 flags,
3538 struct bpf_tramp_links *tlinks,3671 struct bpf_tramp_nodes *tnodes,
3539 void *func_addr)3672 void *func_addr)
3540{3673{
3541 void *rw_image, *tmp;3674 void *rw_image, *tmp;
@@ -3550,7 +3683,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i · 8 unmodified lines
3550 return -ENOMEM;3683 return -ENOMEM;
3551 3684
3552 ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,3685 ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,
3553 flags, tlinks, func_addr);3686 flags, tnodes, func_addr);
3554 if (ret < 0)3687 if (ret < 0)
3555 goto out;3688 goto out;
3556 3689
@@ -3563,7 +3696,7 @@ out: · 6 unmodified lines
3563}3696}
3564 3697
3565int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,3698int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
3566 struct bpf_tramp_links *tlinks, void *func_addr)3699 struct bpf_tramp_nodes *tnodes, void *func_addr)
3567{3700{
3568 struct bpf_tramp_image im;3701 struct bpf_tramp_image im;
3569 void *image;3702 void *image;
@@ -3581,7 +3714,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 11 unmodified lines
3581 return -ENOMEM;3714 return -ENOMEM;
3582 3715
3583 ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image,3716 ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image,
3584 m, flags, tlinks, func_addr);3717 m, flags, tnodes, func_addr);
3585 bpf_jit_free_exec(image);3718 bpf_jit_free_exec(image);
3586 return ret;3719 return ret;
3587}3720}
@@ -3793,7 +3926,8 @@ skip_init_addrs: · 205 unmodified lines
3793 for (pass = 0; pass < MAX_PASSES || image; pass++) {3926 for (pass = 0; pass < MAX_PASSES || image; pass++) {
3794 if (!padding && pass >= PADDING_PASSES)3927 if (!padding && pass >= PADDING_PASSES)
3795 padding = true;3928 padding = true;
3796 proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen, &ctx, padding);3929 proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen,
3930 &ctx, padding);
3797 if (proglen <= 0) {3931 if (proglen <= 0) {
3798out_image:3932out_image:
3799 image = NULL;3933 image = NULL;
@@ -3910,6 +4044,11 @@ bool bpf_jit_supports_kfunc_call(void) · 110 unmodified lines
3910 return true;4044 return true;
3911}4045}
3912 4046
4047bool bpf_jit_supports_stack_args(void)
4048{
4049 return true;
4050}
4051
3913void *bpf_arch_text_copy(void *dst, void *src, size_t len)4052void *bpf_arch_text_copy(void *dst, void *src, size_t len)
3914{4053{
3915 if (text_poke_copy(dst, src, len) == NULL)4054 if (text_poke_copy(dst, src, len) == NULL)
drivers/base/power/power.h+7−0
@@ -168,3 +168,10 @@ static inline void device_pm_init(struct device *dev) · 167 unmodified lines
168 device_pm_sleep_init(dev);168 device_pm_sleep_init(dev);
169 pm_runtime_init(dev);169 pm_runtime_init(dev);
170}170}
171
172#ifdef CONFIG_BPF_SYSCALL
173struct bpf_ws_lock { };
174struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void);
175void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock);
176void *bpf_wakeup_sources_get_head(void);
177#endif
drivers/base/power/wakeup.c+69−2
@@ -1168,11 +1168,78 @@ static const struct file_operations wakeup_sources_stats_fops = { · 1167 unmodified lines
1168 .release = seq_release_private,1168 .release = seq_release_private,
1169};1169};
1170 1170
1171static int __init wakeup_sources_debugfs_init(void)1171#ifdef CONFIG_BPF_SYSCALL
1172#include <linux/btf.h>
1173
1174__bpf_kfunc_start_defs();
1175
1176/**
1177 * bpf_wakeup_sources_read_lock - Acquire the SRCU lock for wakeup sources
1178 *
1179 * The underlying SRCU lock returns an integer index. However, the BPF verifier
1180 * requires a pointer (PTR_TO_BTF_ID) to strictly track the state of acquired
1181 * resources using KF_ACQUIRE and KF_RELEASE semantics. We use an opaque
1182 * structure pointer (struct bpf_ws_lock *) to satisfy the verifier while
1183 * safely encoding the integer index within the pointer address itself.
1184 *
1185 * Return: An opaque pointer encoding the SRCU lock index + 1 (to avoid NULL).
1186 */
1187__bpf_kfunc struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void)
1188{
1189 return (struct bpf_ws_lock *)(long)(wakeup_sources_read_lock() + 1);
1190}
1191
1192/**
1193 * bpf_wakeup_sources_read_unlock - Release the SRCU lock for wakeup sources
1194 * @lock: The opaque pointer returned by bpf_wakeup_sources_read_lock()
1195 *
1196 * The BPF verifier guarantees that @lock is a valid, unreleased pointer from
1197 * the acquire function. We decode the pointer back into the integer SRCU index
1198 * by subtracting 1 and release the lock.
1199 */
1200__bpf_kfunc void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock)
1201{
1202 wakeup_sources_read_unlock((int)(long)lock - 1);
1203}
1204
1205/**
1206 * bpf_wakeup_sources_get_head - Get the head of the wakeup sources list
1207 *
1208 * Return: The head of the wakeup sources list.
1209 */
1210__bpf_kfunc void *bpf_wakeup_sources_get_head(void)
1211{
1212 return &wakeup_sources;
1213}
1214
1215__bpf_kfunc_end_defs();
1216
1217BTF_KFUNCS_START(wakeup_source_kfunc_ids)
1218BTF_ID_FLAGS(func, bpf_wakeup_sources_read_lock, KF_ACQUIRE)
1219BTF_ID_FLAGS(func, bpf_wakeup_sources_read_unlock, KF_RELEASE)
1220BTF_ID_FLAGS(func, bpf_wakeup_sources_get_head)
1221BTF_KFUNCS_END(wakeup_source_kfunc_ids)
1222
1223static const struct btf_kfunc_id_set wakeup_source_kfunc_set = {
1224 .set = &wakeup_source_kfunc_ids,
1225};
1226
1227static void __init wakeup_sources_bpf_init(void)
1228{
1229 if (register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &wakeup_source_kfunc_set))
1230 pm_pr_dbg("Wakeup: failed to register BTF kfuncs\n");
1231}
1232#else
1233static inline void wakeup_sources_bpf_init(void) {}
1234#endif /* CONFIG_BPF_SYSCALL */
1235
1236static int __init wakeup_sources_init(void)
1172{1237{
1173 debugfs_create_file("wakeup_sources", 0444, NULL, NULL,1238 debugfs_create_file("wakeup_sources", 0444, NULL, NULL,
1174 &wakeup_sources_stats_fops);1239 &wakeup_sources_stats_fops);
1240 wakeup_sources_bpf_init();
1241
1175 return 0;1242 return 0;
1176}1243}
1177 1244
1178postcore_initcall(wakeup_sources_debugfs_init);1245postcore_initcall(wakeup_sources_init);
drivers/net/ethernet/netronome/nfp/bpf/verifier.c+4−4
@@ -561,10 +561,10 @@ nfp_bpf_check_alu(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, · 560 unmodified lines
561 const struct bpf_reg_state *dreg =561 const struct bpf_reg_state *dreg =
562 cur_regs(env) + meta->insn.dst_reg;562 cur_regs(env) + meta->insn.dst_reg;
563 563
564 meta->umin_src = min(meta->umin_src, sreg->umin_value);564 meta->umin_src = min(meta->umin_src, reg_umin(sreg));
565 meta->umax_src = max(meta->umax_src, sreg->umax_value);565 meta->umax_src = max(meta->umax_src, reg_umax(sreg));
566 meta->umin_dst = min(meta->umin_dst, dreg->umin_value);566 meta->umin_dst = min(meta->umin_dst, reg_umin(dreg));
567 meta->umax_dst = max(meta->umax_dst, dreg->umax_value);567 meta->umax_dst = max(meta->umax_dst, reg_umax(dreg));
568 568
569 /* NFP supports u16 and u32 multiplication.569 /* NFP supports u16 and u32 multiplication.
570 *570 *
fs/bpf_fs_kfuncs.c+1−1
@@ -200,7 +200,7 @@ int bpf_set_dentry_xattr_locked(struct dentry *dentry, const char *name__str, · 199 unmodified lines
200 const struct bpf_dynptr *value_p, int flags)200 const struct bpf_dynptr *value_p, int flags)
201{201{
202 202
203 struct bpf_dynptr_kern *value_ptr = (struct bpf_dynptr_kern *)value_p;203 const struct bpf_dynptr_kern *value_ptr = (struct bpf_dynptr_kern *)value_p;
204 struct inode *inode = d_inode(dentry);204 struct inode *inode = d_inode(dentry);
205 const void *value;205 const void *value;
206 u32 value_len;206 u32 value_len;
fs/verity/measure.c+2−2
@@ -118,9 +118,9 @@ __bpf_kfunc_start_defs(); · 117 unmodified lines
118 *118 *
119 * Return: 0 on success, a negative value on error.119 * Return: 0 on success, a negative value on error.
120 */120 */
121__bpf_kfunc int bpf_get_fsverity_digest(struct file *file, struct bpf_dynptr *digest_p)121__bpf_kfunc int bpf_get_fsverity_digest(struct file *file, const struct bpf_dynptr *digest_p)
122{122{
123 struct bpf_dynptr_kern *digest_ptr = (struct bpf_dynptr_kern *)digest_p;123 const struct bpf_dynptr_kern *digest_ptr = (struct bpf_dynptr_kern *)digest_p;
124 const struct inode *inode = file_inode(file);124 const struct inode *inode = file_inode(file);
125 u32 dynptr_sz = __bpf_dynptr_size(digest_ptr);125 u32 dynptr_sz = __bpf_dynptr_size(digest_ptr);
126 struct fsverity_digest *arg;126 struct fsverity_digest *arg;
include/asm-generic/rqspinlock.h+11−3
@@ -243,12 +243,20 @@ static __always_inline void res_spin_unlock(rqspinlock_t *lock) · 242 unmodified lines
243 ({ \243 ({ \
244 int __ret; \244 int __ret; \
245 local_irq_save(flags); \245 local_irq_save(flags); \
246 __ret = raw_res_spin_lock(lock); \246 preempt_disable(); \
247 if (__ret) \247 __ret = res_spin_lock(lock); \
248 if (__ret) { \
248 local_irq_restore(flags); \249 local_irq_restore(flags); \
250 preempt_enable(); \
251 } \
249 __ret; \252 __ret; \
250 })253 })
251 254
252#define raw_res_spin_unlock_irqrestore(lock, flags) ({ raw_res_spin_unlock(lock); local_irq_restore(flags); })255#define raw_res_spin_unlock_irqrestore(lock, flags) \
256 ({ \
257 res_spin_unlock(lock); \
258 local_irq_restore(flags); \
259 preempt_enable(); \
260 })
253 261
254#endif /* __ASM_GENERIC_RQSPINLOCK_H */262#endif /* __ASM_GENERIC_RQSPINLOCK_H */
include/linux/bpf-cgroup.h+3−2
@@ -421,7 +421,7 @@ int cgroup_bpf_prog_detach(const union bpf_attr *attr, · 420 unmodified lines
421 enum bpf_prog_type ptype);421 enum bpf_prog_type ptype);
422int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);422int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
423int cgroup_bpf_prog_query(const union bpf_attr *attr,423int cgroup_bpf_prog_query(const union bpf_attr *attr,
424 union bpf_attr __user *uattr);424 union bpf_attr __user *uattr, u32 uattr_size);
425 425
426const struct bpf_func_proto *426const struct bpf_func_proto *
427cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);427cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);
@@ -452,7 +452,8 @@ static inline int cgroup_bpf_link_attach(const union bpf_attr *attr, · 24 unmodified lines
452}452}
453 453
454static inline int cgroup_bpf_prog_query(const union bpf_attr *attr,454static inline int cgroup_bpf_prog_query(const union bpf_attr *attr,
455 union bpf_attr __user *uattr)455 union bpf_attr __user *uattr,
456 u32 uattr_size)
456{457{
457 return -EINVAL;458 return -EINVAL;
458}459}
include/linux/bpf.h+236−46
@@ -6,6 +6,7 @@ · 5 unmodified lines
6 6
7#include <uapi/linux/bpf.h>7#include <uapi/linux/bpf.h>
8#include <uapi/linux/filter.h>8#include <uapi/linux/filter.h>
9#include <linux/bpf_defs.h>
9 10
10#include <crypto/sha2.h>11#include <crypto/sha2.h>
11#include <linux/workqueue.h>12#include <linux/workqueue.h>
@@ -32,6 +33,8 @@ · 20 unmodified lines
32#include <linux/memcontrol.h>33#include <linux/memcontrol.h>
33#include <linux/cfi.h>34#include <linux/cfi.h>
34#include <linux/xattr.h>35#include <linux/xattr.h>
36#include <linux/key.h>
37#include <linux/ftrace.h>
35#include <asm/rqspinlock.h>38#include <asm/rqspinlock.h>
36 39
37struct bpf_verifier_env;40struct bpf_verifier_env;
@@ -111,7 +114,7 @@ struct bpf_map_ops { · 73 unmodified lines
111 long (*map_pop_elem)(struct bpf_map *map, void *value);114 long (*map_pop_elem)(struct bpf_map *map, void *value);
112 long (*map_peek_elem)(struct bpf_map *map, void *value);115 long (*map_peek_elem)(struct bpf_map *map, void *value);
113 void *(*map_lookup_percpu_elem)(struct bpf_map *map, void *key, u32 cpu);116 void *(*map_lookup_percpu_elem)(struct bpf_map *map, void *key, u32 cpu);
114 int (*map_get_hash)(struct bpf_map *map, u32 hash_buf_size, void *hash_buf);117 int (*map_get_hash)(struct bpf_map *map);
115 118
116 /* funcs called by prog_array and perf_event_array map */119 /* funcs called by prog_array and perf_event_array map */
117 void *(*map_fd_get_ptr)(struct bpf_map *map, struct file *map_file,120 void *(*map_fd_get_ptr)(struct bpf_map *map, struct file *map_file,
@@ -296,6 +299,7 @@ struct bpf_map_owner { · 178 unmodified lines
296 299
297struct bpf_map {300struct bpf_map {
298 u8 sha[SHA256_DIGEST_SIZE];301 u8 sha[SHA256_DIGEST_SIZE];
302 u32 excl;
299 const struct bpf_map_ops *ops;303 const struct bpf_map_ops *ops;
300 struct bpf_map *inner_map_meta;304 struct bpf_map *inner_map_meta;
301#ifdef CONFIG_SECURITY305#ifdef CONFIG_SECURITY
@@ -489,6 +493,35 @@ static inline bool btf_record_has_field(const struct btf_record *rec, enum btf_f · 187 unmodified lines
489 return rec->field_mask & type;493 return rec->field_mask & type;
490}494}
491 495
496static inline bool btf_field_is_nmi_safe(enum btf_field_type type)
497{
498 switch (type) {
499 case BPF_SPIN_LOCK:
500 case BPF_RES_SPIN_LOCK:
501 case BPF_TIMER:
502 case BPF_WORKQUEUE:
503 case BPF_TASK_WORK:
504 case BPF_KPTR_UNREF:
505 case BPF_REFCOUNT:
506 return true;
507 default:
508 return false;
509 }
510}
511
512static inline bool btf_record_has_nmi_unsafe_fields(const struct btf_record *rec)
513{
514 int i;
515
516 if (IS_ERR_OR_NULL(rec))
517 return false;
518 for (i = 0; i < rec->cnt; i++) {
519 if (!btf_field_is_nmi_safe(rec->fields[i].type))
520 return true;
521 }
522 return false;
523}
524
492static inline void bpf_obj_init(const struct btf_record *rec, void *obj)525static inline void bpf_obj_init(const struct btf_record *rec, void *obj)
493{526{
494 int i;527 int i;
@@ -618,6 +651,8 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, · 123 unmodified lines
618 struct bpf_spin_lock *spin_lock);651 struct bpf_spin_lock *spin_lock);
619u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena);652u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena);
620u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena);653u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena);
654u64 bpf_arena_map_kern_vm_start(struct bpf_map *map);
655struct bpf_map *bpf_prog_arena(struct bpf_prog *prog);
621int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);656int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
622 657
623struct bpf_offload_dev;658struct bpf_offload_dev;
@@ -679,6 +714,8 @@ int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, · 55 unmodified lines
679void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,714void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
680 u64 flags);715 u64 flags);
681void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt);716void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt);
717void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
718 u64 flags);
682#else719#else
683static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt,720static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
684 int node_id, u64 flags)721 int node_id, u64 flags)
@@ -689,6 +726,12 @@ static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr · 4 unmodified lines
689static inline void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt)726static inline void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt)
690{727{
691}728}
729
730static inline void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
731 int node_id, u64 flags)
732{
733 return NULL;
734}
692#endif735#endif
693 736
694extern const struct bpf_map_ops bpf_map_offload_ops;737extern const struct bpf_map_ops bpf_map_offload_ops;
@@ -1052,7 +1095,7 @@ struct bpf_insn_access_aux { · 357 unmodified lines
1052 struct {1095 struct {
1053 struct btf *btf;1096 struct btf *btf;
1054 u32 btf_id;1097 u32 btf_id;
1055 u32 ref_obj_id;1098 u32 ref_id;
1056 };1099 };
1057 };1100 };
1058 struct bpf_verifier_log *log; /* for verbose logs */1101 struct bpf_verifier_log *log; /* for verbose logs */
@@ -1152,6 +1195,11 @@ struct bpf_prog_offload { · 93 unmodified lines
1152 1195
1153/* The longest tracepoint has 12 args.1196/* The longest tracepoint has 12 args.
1154 * See include/trace/bpf_probe.h1197 * See include/trace/bpf_probe.h
1198 *
1199 * Also reuse this macro for maximum number of arguments a BPF function
1200 * or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
1201 * stack arg slots. The JIT may map some stack arg slots to registers based
1202 * on the native calling convention (e.g., arg 6 to R9 on x86-64).
1155 */1203 */
1156#define MAX_BPF_FUNC_ARGS 121204#define MAX_BPF_FUNC_ARGS 12
1157 1205
@@ -1234,9 +1282,9 @@ enum { · 76 unmodified lines
1234#define BPF_TRAMP_COOKIE_INDEX_SHIFT 81282#define BPF_TRAMP_COOKIE_INDEX_SHIFT 8
1235#define BPF_TRAMP_IS_RETURN_SHIFT 631283#define BPF_TRAMP_IS_RETURN_SHIFT 63
1236 1284
1237struct bpf_tramp_links {1285struct bpf_tramp_nodes {
1238 struct bpf_tramp_link *links[BPF_MAX_TRAMP_LINKS];1286 struct bpf_tramp_node *nodes[BPF_MAX_TRAMP_LINKS];
1239 int nr_links;1287 int nr_nodes;
1240};1288};
1241 1289
1242struct bpf_tramp_run_ctx;1290struct bpf_tramp_run_ctx;
@@ -1264,13 +1312,13 @@ struct bpf_tramp_run_ctx; · 21 unmodified lines
1264struct bpf_tramp_image;1312struct bpf_tramp_image;
1265int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,1313int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
1266 const struct btf_func_model *m, u32 flags,1314 const struct btf_func_model *m, u32 flags,
1267 struct bpf_tramp_links *tlinks,1315 struct bpf_tramp_nodes *tnodes,
1268 void *func_addr);1316 void *func_addr);
1269void *arch_alloc_bpf_trampoline(unsigned int size);1317void *arch_alloc_bpf_trampoline(unsigned int size);
1270void arch_free_bpf_trampoline(void *image, unsigned int size);1318void arch_free_bpf_trampoline(void *image, unsigned int size);
1271int __must_check arch_protect_bpf_trampoline(void *image, unsigned int size);1319int __must_check arch_protect_bpf_trampoline(void *image, unsigned int size);
1272int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,1320int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
1273 struct bpf_tramp_links *tlinks, void *func_addr);1321 struct bpf_tramp_nodes *tnodes, void *func_addr);
1274 1322
1275u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog,1323u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog,
1276 struct bpf_tramp_run_ctx *run_ctx);1324 struct bpf_tramp_run_ctx *run_ctx);
@@ -1336,8 +1384,6 @@ struct bpf_trampoline { · 59 unmodified lines
1336 /* hlist for trampoline_ip_table */1384 /* hlist for trampoline_ip_table */
1337 struct hlist_node hlist_ip;1385 struct hlist_node hlist_ip;
1338 struct ftrace_ops *fops;1386 struct ftrace_ops *fops;
1339 /* serializes access to fields of this trampoline */
1340 struct mutex mutex;
1341 refcount_t refcnt;1387 refcount_t refcnt;
1342 u32 flags;1388 u32 flags;
1343 u64 key;1389 u64 key;
@@ -1358,6 +1404,11 @@ struct bpf_trampoline { · 14 unmodified lines
1358 int progs_cnt[BPF_TRAMP_MAX];1404 int progs_cnt[BPF_TRAMP_MAX];
1359 /* Executable image of trampoline */1405 /* Executable image of trampoline */
1360 struct bpf_tramp_image *cur_image;1406 struct bpf_tramp_image *cur_image;
1407 /* Used as temporary old image storage for multi_attach */
1408 struct {
1409 struct bpf_tramp_image *old_image;
1410 u32 old_flags;
1411 } multi_attach;
1361};1412};
1362 1413
1363struct bpf_attach_target_info {1414struct bpf_attach_target_info {
@@ -1455,11 +1506,13 @@ static inline int bpf_dynptr_check_off_len(const struct bpf_dynptr_kern *ptr, u6 · 91 unmodified lines
1455 return 0;1506 return 0;
1456}1507}
1457 1508
1509struct bpf_tracing_multi_link;
1510
1458#ifdef CONFIG_BPF_JIT1511#ifdef CONFIG_BPF_JIT
1459int bpf_trampoline_link_prog(struct bpf_tramp_link *link,1512int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
1460 struct bpf_trampoline *tr,1513 struct bpf_trampoline *tr,
1461 struct bpf_prog *tgt_prog);1514 struct bpf_prog *tgt_prog);
1462int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,1515int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
1463 struct bpf_trampoline *tr,1516 struct bpf_trampoline *tr,
1464 struct bpf_prog *tgt_prog);1517 struct bpf_prog *tgt_prog);
1465struct bpf_trampoline *bpf_trampoline_get(u64 key,1518struct bpf_trampoline *bpf_trampoline_get(u64 key,
@@ -1467,6 +1520,11 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, · 1 unmodified line
1467void bpf_trampoline_put(struct bpf_trampoline *tr);1520void bpf_trampoline_put(struct bpf_trampoline *tr);
1468int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);1521int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
1469 1522
1523int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
1524 struct bpf_tracing_multi_link *link);
1525int bpf_trampoline_multi_detach(struct bpf_prog *prog,
1526 struct bpf_tracing_multi_link *link);
1527
1470/*1528/*
1471 * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn1529 * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn
1472 * indirection with a direct call to the bpf program. If the architecture does1530 * indirection with a direct call to the bpf program. If the architecture does
@@ -1544,14 +1602,15 @@ void bpf_jit_uncharge_modmem(u32 size); · 71 unmodified lines
1544bool bpf_prog_has_trampoline(const struct bpf_prog *prog);1602bool bpf_prog_has_trampoline(const struct bpf_prog *prog);
1545bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog,1603bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog,
1546 int insn_idx);1604 int insn_idx);
1605u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog);
1547#else1606#else
1548static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link,1607static inline int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
1549 struct bpf_trampoline *tr,1608 struct bpf_trampoline *tr,
1550 struct bpf_prog *tgt_prog)1609 struct bpf_prog *tgt_prog)
1551{1610{
1552 return -ENOTSUPP;1611 return -ENOTSUPP;
1553}1612}
1554static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,1613static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
1555 struct bpf_trampoline *tr,1614 struct bpf_trampoline *tr,
1556 struct bpf_prog *tgt_prog)1615 struct bpf_prog *tgt_prog)
1557{1616{
@@ -1578,6 +1637,16 @@ static inline bool bpf_prog_has_trampoline(const struct bpf_prog *prog) · 20 unmodified lines
1578{1637{
1579 return false;1638 return false;
1580}1639}
1640static inline int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
1641 struct bpf_tracing_multi_link *link)
1642{
1643 return -ENOTSUPP;
1644}
1645static inline int bpf_trampoline_multi_detach(struct bpf_prog *prog,
1646 struct bpf_tracing_multi_link *link)
1647{
1648 return -ENOTSUPP;
1649}
1581#endif1650#endif
1582 1651
1583struct bpf_func_info_aux {1652struct bpf_func_info_aux {
@@ -1615,7 +1684,7 @@ struct bpf_ctx_arg_aux { · 31 unmodified lines
1615 enum bpf_reg_type reg_type;1684 enum bpf_reg_type reg_type;
1616 struct btf *btf;1685 struct btf *btf;
1617 u32 btf_id;1686 u32 btf_id;
1618 u32 ref_obj_id;1687 u32 ref_id;
1619 bool refcounted;1688 bool refcounted;
1620};1689};
1621 1690
@@ -1657,6 +1726,19 @@ struct bpf_stream_stage { · 35 unmodified lines
1657 int len;1726 int len;
1658};1727};
1659 1728
1729enum bpf_sig_verdict {
1730 BPF_SIG_UNSIGNED = 0,
1731 BPF_SIG_VERIFIED,
1732};
1733
1734enum bpf_sig_keyring {
1735 BPF_SIG_KEYRING_NONE = 0,
1736 BPF_SIG_KEYRING_BUILTIN,
1737 BPF_SIG_KEYRING_SECONDARY,
1738 BPF_SIG_KEYRING_PLATFORM,
1739 BPF_SIG_KEYRING_USER,
1740};
1741
1660struct bpf_prog_aux {1742struct bpf_prog_aux {
1661 atomic64_t refcnt;1743 atomic64_t refcnt;
1662 u32 used_map_cnt;1744 u32 used_map_cnt;
@@ -1699,6 +1781,11 @@ struct bpf_prog_aux { · 36 unmodified lines
1699 bool changes_pkt_data;1781 bool changes_pkt_data;
1700 bool might_sleep;1782 bool might_sleep;
1701 bool kprobe_write_ctx;1783 bool kprobe_write_ctx;
1784 struct {
1785 s32 keyring_serial;
1786 u8 keyring_type;
1787 u8 verdict;
1788 } sig;
1702 u64 prog_array_member_cnt; /* counts how many times as member of prog_array */1789 u64 prog_array_member_cnt; /* counts how many times as member of prog_array */
1703 struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */1790 struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */
1704 struct bpf_arena *arena;1791 struct bpf_arena *arena;
@@ -1731,6 +1818,7 @@ struct bpf_prog_aux { · 26 unmodified lines
1731 struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];1818 struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
1732 char name[BPF_OBJ_NAME_LEN];1819 char name[BPF_OBJ_NAME_LEN];
1733 u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);1820 u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
1821 u16 stack_arg_sp_adjust;
1734#ifdef CONFIG_SECURITY1822#ifdef CONFIG_SECURITY
1735 void *security;1823 void *security;
1736#endif1824#endif
@@ -1874,12 +1962,17 @@ struct bpf_link_ops { · 137 unmodified lines
1874 __poll_t (*poll)(struct file *file, struct poll_table_struct *pts);1962 __poll_t (*poll)(struct file *file, struct poll_table_struct *pts);
1875};1963};
1876 1964
1877struct bpf_tramp_link {1965struct bpf_tramp_node {
1878 struct bpf_link link;1966 struct bpf_link *link;
1879 struct hlist_node tramp_hlist;1967 struct hlist_node tramp_hlist;
1880 u64 cookie;1968 u64 cookie;
1881};1969};
1882 1970
1971struct bpf_tramp_link {
1972 struct bpf_link link;
1973 struct bpf_tramp_node node;
1974};
1975
1883struct bpf_shim_tramp_link {1976struct bpf_shim_tramp_link {
1884 struct bpf_tramp_link link;1977 struct bpf_tramp_link link;
1885 struct bpf_trampoline *trampoline;1978 struct bpf_trampoline *trampoline;
@@ -1887,13 +1980,31 @@ struct bpf_shim_tramp_link { · 1 unmodified line
1887 1980
1888struct bpf_tracing_link {1981struct bpf_tracing_link {
1889 struct bpf_tramp_link link;1982 struct bpf_tramp_link link;
1983 struct bpf_tramp_node fexit;
1890 struct bpf_trampoline *trampoline;1984 struct bpf_trampoline *trampoline;
1891 struct bpf_prog *tgt_prog;1985 struct bpf_prog *tgt_prog;
1892};1986};
1893 1987
1894struct bpf_fsession_link {1988struct bpf_tracing_multi_node {
1895 struct bpf_tracing_link link;1989 struct bpf_tramp_node node;
1896 struct bpf_tramp_link fexit;1990 struct bpf_trampoline *trampoline;
1991 struct ftrace_func_entry entry;
1992};
1993
1994struct bpf_tracing_multi_data {
1995 struct ftrace_hash *unreg;
1996 struct ftrace_hash *modify;
1997 struct ftrace_hash *reg;
1998 struct ftrace_func_entry *entry;
1999};
2000
2001struct bpf_tracing_multi_link {
2002 struct bpf_link link;
2003 struct bpf_tracing_multi_data data;
2004 u64 *cookies;
2005 struct bpf_tramp_node *fexits;
2006 int nodes_cnt;
2007 struct bpf_tracing_multi_node nodes[] __counted_by(nodes_cnt);
1897};2008};
1898 2009
1899struct bpf_raw_tp_link {2010struct bpf_raw_tp_link {
@@ -2079,6 +2190,12 @@ static inline void bpf_prog_put_recursion_context(struct bpf_prog *prog) · 179 unmodified lines
2079#endif2190#endif
2080}2191}
2081 2192
2193static inline bool is_tracing_multi(enum bpf_attach_type type)
2194{
2195 return type == BPF_TRACE_FENTRY_MULTI || type == BPF_TRACE_FEXIT_MULTI ||
2196 type == BPF_TRACE_FSESSION_MULTI;
2197}
2198
2082#if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL)2199#if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL)
2083/* This macro helps developer to register a struct_ops type and generate2200/* This macro helps developer to register a struct_ops type and generate
2084 * type information correctly. Developers should use this macro to register2201 * type information correctly. Developers should use this macro to register
@@ -2099,8 +2216,8 @@ void bpf_struct_ops_put(const void *kdata); · 14 unmodified lines
2099int bpf_struct_ops_supported(const struct bpf_struct_ops *st_ops, u32 moff);2216int bpf_struct_ops_supported(const struct bpf_struct_ops *st_ops, u32 moff);
2100int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key,2217int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key,
2101 void *value);2218 void *value);
2102int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,2219int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes,
2103 struct bpf_tramp_link *link,2220 struct bpf_tramp_node *node,
2104 const struct btf_func_model *model,2221 const struct btf_func_model *model,
2105 void *stub_func,2222 void *stub_func,
2106 void **image, u32 *image_off,2223 void **image, u32 *image_off,
@@ -2125,6 +2242,9 @@ int bpf_prog_assoc_struct_ops(struct bpf_prog *prog, struct bpf_map *map); · 18 unmodified lines
2125void bpf_prog_disassoc_struct_ops(struct bpf_prog *prog);2242void bpf_prog_disassoc_struct_ops(struct bpf_prog *prog);
2126void *bpf_prog_get_assoc_struct_ops(const struct bpf_prog_aux *aux);2243void *bpf_prog_get_assoc_struct_ops(const struct bpf_prog_aux *aux);
2127u32 bpf_struct_ops_id(const void *kdata);2244u32 bpf_struct_ops_id(const void *kdata);
2245int bpf_struct_ops_for_each_prog(const void *kdata,
2246 int (*cb)(struct bpf_prog *prog, void *data),
2247 void *data);
2128 2248
2129#ifdef CONFIG_NET2249#ifdef CONFIG_NET
2130/* Define it here to avoid the use of forward declaration */2250/* Define it here to avoid the use of forward declaration */
@@ -2192,31 +2312,33 @@ static inline void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_op · 61 unmodified lines
2192 2312
2193#endif2313#endif
2194 2314
2195static inline int bpf_fsession_cnt(struct bpf_tramp_links *links)2315static inline int bpf_fsession_cnt(struct bpf_tramp_nodes *nodes)
2196{2316{
2197 struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY];2317 struct bpf_tramp_nodes fentries = nodes[BPF_TRAMP_FENTRY];
2198 int cnt = 0;2318 int cnt = 0;
2199 2319
2200 for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) {2320 for (int i = 0; i < nodes[BPF_TRAMP_FENTRY].nr_nodes; i++) {
2201 if (fentries.links[i]->link.prog->expected_attach_type == BPF_TRACE_FSESSION)2321 if (fentries.nodes[i]->link->prog->expected_attach_type == BPF_TRACE_FSESSION)
2322 cnt++;
2323 if (fentries.nodes[i]->link->prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI)
2202 cnt++;2324 cnt++;
2203 }2325 }
2204 2326
2205 return cnt;2327 return cnt;
2206}2328}
2207 2329
2208static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_link *link)2330static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_node *node)
2209{2331{
2210 return link->link.prog->call_session_cookie;2332 return node->link->prog->call_session_cookie;
2211}2333}
2212 2334
2213static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_links *links)2335static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_nodes *nodes)
2214{2336{
2215 struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY];2337 struct bpf_tramp_nodes fentries = nodes[BPF_TRAMP_FENTRY];
2216 int cnt = 0;2338 int cnt = 0;
2217 2339
2218 for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) {2340 for (int i = 0; i < nodes[BPF_TRAMP_FENTRY].nr_nodes; i++) {
2219 if (bpf_prog_calls_session_cookie(fentries.links[i]))2341 if (bpf_prog_calls_session_cookie(fentries.nodes[i]))
2220 cnt++;2342 cnt++;
2221 }2343 }
2222 2344
@@ -2598,6 +2720,7 @@ bool btf_record_equal(const struct btf_record *rec_a, const struct btf_record *r · 375 unmodified lines
2598void bpf_obj_free_timer(const struct btf_record *rec, void *obj);2720void bpf_obj_free_timer(const struct btf_record *rec, void *obj);
2599void bpf_obj_free_workqueue(const struct btf_record *rec, void *obj);2721void bpf_obj_free_workqueue(const struct btf_record *rec, void *obj);
2600void bpf_obj_free_task_work(const struct btf_record *rec, void *obj);2722void bpf_obj_free_task_work(const struct btf_record *rec, void *obj);
2723void bpf_obj_cancel_fields(struct bpf_map *map, void *obj);
2601void bpf_obj_free_fields(const struct btf_record *rec, void *obj);2724void bpf_obj_free_fields(const struct btf_record *rec, void *obj);
2602void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu);2725void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu);
2603 2726
@@ -2764,6 +2887,9 @@ void bpf_link_init(struct bpf_link *link, enum bpf_link_type type, · 160 unmodified lines
2764void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type,2887void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type,
2765 const struct bpf_link_ops *ops, struct bpf_prog *prog,2888 const struct bpf_link_ops *ops, struct bpf_prog *prog,
2766 enum bpf_attach_type attach_type, bool sleepable);2889 enum bpf_attach_type attach_type, bool sleepable);
2890void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type,
2891 const struct bpf_link_ops *ops, struct bpf_prog *prog,
2892 enum bpf_attach_type attach_type, u64 cookie);
2767int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer);2893int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer);
2768int bpf_link_settle(struct bpf_link_primer *primer);2894int bpf_link_settle(struct bpf_link_primer *primer);
2769void bpf_link_cleanup(struct bpf_link_primer *primer);2895void bpf_link_cleanup(struct bpf_link_primer *primer);
@@ -2917,7 +3043,9 @@ int bpf_check_uarg_tail_zero(bpfptr_t uaddr, size_t expected_size, · 147 unmodified lines
2917 size_t actual_size);3043 size_t actual_size);
2918 3044
2919/* verify correctness of eBPF program */3045/* verify correctness of eBPF program */
2920int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size);3046struct bpf_log_attr;
3047int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr,
3048 struct bpf_log_attr *attr_log);
2921 3049
2922#ifndef CONFIG_BPF_JIT_ALWAYS_ON3050#ifndef CONFIG_BPF_JIT_ALWAYS_ON
2923int bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth);3051int bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth);
@@ -3088,6 +3216,56 @@ void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr); · 164 unmodified lines
3088void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr);3216void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr);
3089void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip);3217void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip);
3090 3218
3219static __always_inline u32
3220bpf_prog_run_array_sleepable(const struct bpf_prog_array *array,
3221 const void *ctx, bpf_prog_run_fn run_prog)
3222{
3223 const struct bpf_prog_array_item *item;
3224 struct bpf_prog *prog;
3225 struct bpf_run_ctx *old_run_ctx;
3226 struct bpf_trace_run_ctx run_ctx;
3227 u32 ret = 1;
3228
3229 if (unlikely(!array))
3230 return ret;
3231
3232 migrate_disable();
3233
3234 run_ctx.is_uprobe = false;
3235
3236 old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
3237 item = &array->items[0];
3238 while ((prog = READ_ONCE(item->prog))) {
3239 /* Skip dummy_bpf_prog placeholder (len == 0) */
3240 if (unlikely(!prog->len)) {
3241 item++;
3242 continue;
3243 }
3244
3245 if (unlikely(!bpf_prog_get_recursion_context(prog))) {
3246 bpf_prog_inc_misses_counter(prog);
3247 bpf_prog_put_recursion_context(prog);
3248 item++;
3249 continue;
3250 }
3251
3252 run_ctx.bpf_cookie = item->bpf_cookie;
3253
3254 if (!prog->sleepable) {
3255 guard(rcu)();
3256 ret &= run_prog(prog, ctx);
3257 } else {
3258 ret &= run_prog(prog, ctx);
3259 }
3260
3261 bpf_prog_put_recursion_context(prog);
3262 item++;
3263 }
3264 bpf_reset_run_ctx(old_run_ctx);
3265 migrate_enable();
3266 return ret;
3267}
3268
3091#else /* !CONFIG_BPF_SYSCALL */3269#else /* !CONFIG_BPF_SYSCALL */
3092static inline struct bpf_prog *bpf_prog_get(u32 ufd)3270static inline struct bpf_prog *bpf_prog_get(u32 ufd)
3093{3271{
@@ -3135,6 +3313,12 @@ static inline void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_ · 41 unmodified lines
3135{3313{
3136}3314}
3137 3315
3316static inline void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type,
3317 const struct bpf_link_ops *ops, struct bpf_prog *prog,
3318 enum bpf_attach_type attach_type, u64 cookie)
3319{
3320}
3321
3138static inline int bpf_link_prime(struct bpf_link *link,3322static inline int bpf_link_prime(struct bpf_link *link,
3139 struct bpf_link_primer *primer)3323 struct bpf_link_primer *primer)
3140{3324{
@@ -3626,15 +3810,25 @@ static inline int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, · 485 unmodified lines
3626#endif /* CONFIG_BPF_SYSCALL */3810#endif /* CONFIG_BPF_SYSCALL */
3627#endif /* defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL) */3811#endif /* defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL) */
3628 3812
3629#if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL)3813#ifdef CONFIG_KEYS
3814struct bpf_key {
3815 struct key *key;
3816 bool has_ref;
3817};
3818#endif /* CONFIG_KEYS */
3630 3819
3820#if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL)
3631struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags);3821struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags);
3632struct bpf_key *bpf_lookup_system_key(u64 id);3822struct bpf_key *bpf_lookup_system_key(u64 id);
3633void bpf_key_put(struct bpf_key *bkey);3823void bpf_key_put(struct bpf_key *bkey);
3634int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,3824int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p,
3635 struct bpf_dynptr *sig_p,3825 const struct bpf_dynptr *sig_p,
3636 struct bpf_key *trusted_keyring);3826 struct bpf_key *trusted_keyring);
3637 3827
3828static inline s32 bpf_key_serial(const struct bpf_key *key)
3829{
3830 return key->has_ref ? key->key->serial : 0;
3831}
3638#else3832#else
3639static inline struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags)3833static inline struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags)
3640{3834{
@@ -3650,12 +3844,17 @@ static inline void bpf_key_put(struct bpf_key *bkey) · 9 unmodified lines
3650{3844{
3651}3845}
3652 3846
3653static inline int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,3847static inline int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p,
3654 struct bpf_dynptr *sig_p,3848 const struct bpf_dynptr *sig_p,
3655 struct bpf_key *trusted_keyring)3849 struct bpf_key *trusted_keyring)
3656{3850{
3657 return -EOPNOTSUPP;3851 return -EOPNOTSUPP;
3658}3852}
3853
3854static inline s32 bpf_key_serial(const struct bpf_key *key)
3855{
3856 return 0;
3857}
3659#endif /* defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) */3858#endif /* defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) */
3660 3859
3661/* verifier prototypes for helper functions called from eBPF programs */3860/* verifier prototypes for helper functions called from eBPF programs */
@@ -3931,15 +4130,6 @@ static inline void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) {} · 269 unmodified lines
3931static inline void bpf_cgroup_atype_put(int cgroup_atype) {}4130static inline void bpf_cgroup_atype_put(int cgroup_atype) {}
3932#endif /* CONFIG_BPF_LSM */4131#endif /* CONFIG_BPF_LSM */
3933 4132
3934struct key;
3935
3936#ifdef CONFIG_KEYS
3937struct bpf_key {
3938 struct key *key;
3939 bool has_ref;
3940};
3941#endif /* CONFIG_KEYS */
3942
3943static inline bool type_is_alloc(u32 type)4133static inline bool type_is_alloc(u32 type)
3944{4134{
3945 return type & MEM_ALLOC;4135 return type & MEM_ALLOC;
include/linux/bpf_defs.h+19−0
@@ -0,0 +1,19 @@
1/* SPDX-License-Identifier: GPL-2.0-or-later */
2/*
3 * Subset of bpf.h declarations, split out so files that need only these
4 * declarations can avoid bpf.h's full include cost.
5 */
6#ifndef _LINUX_BPF_DEFS_H
7#define _LINUX_BPF_DEFS_H
8
9#ifdef CONFIG_BPF_SYSCALL
10bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, unsigned long fault_ip);
11#else
12static inline bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write,
13 unsigned long fault_ip)
14{
15 return false;
16}
17#endif
18
19#endif /* _LINUX_BPF_DEFS_H */
include/linux/bpf_lsm.h+6−0
@@ -52,6 +52,7 @@ int bpf_set_dentry_xattr_locked(struct dentry *dentry, const char *name__str, · 51 unmodified lines
52 const struct bpf_dynptr *value_p, int flags);52 const struct bpf_dynptr *value_p, int flags);
53int bpf_remove_dentry_xattr_locked(struct dentry *dentry, const char *name__str);53int bpf_remove_dentry_xattr_locked(struct dentry *dentry, const char *name__str);
54bool bpf_lsm_has_d_inode_locked(const struct bpf_prog *prog);54bool bpf_lsm_has_d_inode_locked(const struct bpf_prog *prog);
55bool bpf_lsm_hook_returns_errno(u32 btf_id);
55 56
56#else /* !CONFIG_BPF_LSM */57#else /* !CONFIG_BPF_LSM */
57 58
@@ -104,6 +105,11 @@ static inline bool bpf_lsm_has_d_inode_locked(const struct bpf_prog *prog) · 46 unmodified lines
104{105{
105 return false;106 return false;
106}107}
108
109static inline bool bpf_lsm_hook_returns_errno(u32 btf_id)
110{
111 return true;
112}
107#endif /* CONFIG_BPF_LSM */113#endif /* CONFIG_BPF_LSM */
108 114
109#endif /* _LINUX_BPF_LSM_H */115#endif /* _LINUX_BPF_LSM_H */
include/linux/bpf_types.h+2−0
@@ -134,6 +134,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_BLOOM_FILTER, bloom_filter_map_ops) · 133 unmodified lines
134BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops)134BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops)
135BPF_MAP_TYPE(BPF_MAP_TYPE_ARENA, arena_map_ops)135BPF_MAP_TYPE(BPF_MAP_TYPE_ARENA, arena_map_ops)
136BPF_MAP_TYPE(BPF_MAP_TYPE_INSN_ARRAY, insn_array_map_ops)136BPF_MAP_TYPE(BPF_MAP_TYPE_INSN_ARRAY, insn_array_map_ops)
137BPF_MAP_TYPE(BPF_MAP_TYPE_RHASH, rhtab_map_ops)
137 138
138BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)139BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)
139BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing)140BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing)
@@ -155,3 +156,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_PERF_EVENT, perf) · 15 unmodified lines
155BPF_LINK_TYPE(BPF_LINK_TYPE_KPROBE_MULTI, kprobe_multi)156BPF_LINK_TYPE(BPF_LINK_TYPE_KPROBE_MULTI, kprobe_multi)
156BPF_LINK_TYPE(BPF_LINK_TYPE_STRUCT_OPS, struct_ops)157BPF_LINK_TYPE(BPF_LINK_TYPE_STRUCT_OPS, struct_ops)
157BPF_LINK_TYPE(BPF_LINK_TYPE_UPROBE_MULTI, uprobe_multi)158BPF_LINK_TYPE(BPF_LINK_TYPE_UPROBE_MULTI, uprobe_multi)
159BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING_MULTI, tracing_multi)
include/linux/bpf_verifier.h+209−92
@@ -8,6 +8,7 @@ · 7 unmodified lines
8#include <linux/btf.h> /* for struct btf and btf_id() */8#include <linux/btf.h> /* for struct btf and btf_id() */
9#include <linux/filter.h> /* for MAX_BPF_STACK */9#include <linux/filter.h> /* for MAX_BPF_STACK */
10#include <linux/tnum.h>10#include <linux/tnum.h>
11#include <linux/cnum.h>
11 12
12/* Maximum variable offset umax_value permitted when resolving memory accesses.13/* Maximum variable offset umax_value permitted when resolving memory accesses.
13 * In practice this is far bigger than any realistic pointer offset; this limit14 * In practice this is far bigger than any realistic pointer offset; this limit
@@ -65,7 +66,6 @@ struct bpf_reg_state { · 51 unmodified lines
65 66
66 struct { /* for PTR_TO_MEM | PTR_TO_MEM_OR_NULL */67 struct { /* for PTR_TO_MEM | PTR_TO_MEM_OR_NULL */
67 u32 mem_size;68 u32 mem_size;
68 u32 dynptr_id; /* for dynptr slices */
69 };69 };
70 70
71 /* For dynptr stack slots */71 /* For dynptr stack slots */
@@ -120,14 +120,8 @@ struct bpf_reg_state { · 48 unmodified lines
120 * These refer to the same value as var_off, not necessarily the actual120 * These refer to the same value as var_off, not necessarily the actual
121 * contents of the register.121 * contents of the register.
122 */122 */
123 s64 smin_value; /* minimum possible (s64)value */123 struct cnum64 r64; /* 64-bit range as circular number */
124 s64 smax_value; /* maximum possible (s64)value */124 struct cnum32 r32; /* 32-bit range as circular number */
125 u64 umin_value; /* minimum possible (u64)value */
126 u64 umax_value; /* maximum possible (u64)value */
127 s32 s32_min_value; /* minimum possible (s32)value */
128 s32 s32_max_value; /* maximum possible (s32)value */
129 u32 u32_min_value; /* minimum possible (u32)value */
130 u32 u32_max_value; /* maximum possible (u32)value */
131 /* For PTR_TO_PACKET, used to find other pointers with the same variable125 /* For PTR_TO_PACKET, used to find other pointers with the same variable
132 * offset, so they can share range knowledge.126 * offset, so they can share range knowledge.
133 * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we127 * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we
@@ -153,46 +147,14 @@ struct bpf_reg_state { · 19 unmodified lines
153#define BPF_ADD_CONST32 (1U << 30)147#define BPF_ADD_CONST32 (1U << 30)
154#define BPF_ADD_CONST (BPF_ADD_CONST64 | BPF_ADD_CONST32)148#define BPF_ADD_CONST (BPF_ADD_CONST64 | BPF_ADD_CONST32)
155 u32 id;149 u32 id;
156 /* PTR_TO_SOCKET and PTR_TO_TCP_SOCK could be a ptr returned150 /*
157 * from a pointer-cast helper, bpf_sk_fullsock() and151 * Tracks the parent object this register was derived from.
158 * bpf_tcp_sock().152 * Used for cascading invalidation: when the parent object is
159 *153 * released or invalidated, all registers with matching parent_id
160 * Consider the following where "sk" is a reference counted154 * are also invalidated. For example, a slice from bpf_dynptr_data()
161 * pointer returned from "sk = bpf_sk_lookup_tcp();":155 * gets parent_id set to the dynptr's id.
162 *
163 * 1: sk = bpf_sk_lookup_tcp();
164 * 2: if (!sk) { return 0; }
165 * 3: fullsock = bpf_sk_fullsock(sk);
166 * 4: if (!fullsock) { bpf_sk_release(sk); return 0; }
167 * 5: tp = bpf_tcp_sock(fullsock);
168 * 6: if (!tp) { bpf_sk_release(sk); return 0; }
169 * 7: bpf_sk_release(sk);
170 * 8: snd_cwnd = tp->snd_cwnd; // verifier will complain
171 *
172 * After bpf_sk_release(sk) at line 7, both "fullsock" ptr and
173 * "tp" ptr should be invalidated also. In order to do that,
174 * the reg holding "fullsock" and "sk" need to remember
175 * the original refcounted ptr id (i.e. sk_reg->id) in ref_obj_id
176 * such that the verifier can reset all regs which have
177 * ref_obj_id matching the sk_reg->id.
178 *
179 * sk_reg->ref_obj_id is set to sk_reg->id at line 1.
180 * sk_reg->id will stay as NULL-marking purpose only.
181 * After NULL-marking is done, sk_reg->id can be reset to 0.
182 *
183 * After "fullsock = bpf_sk_fullsock(sk);" at line 3,
184 * fullsock_reg->ref_obj_id is set to sk_reg->ref_obj_id.
185 *
186 * After "tp = bpf_tcp_sock(fullsock);" at line 5,
187 * tp_reg->ref_obj_id is set to fullsock_reg->ref_obj_id
188 * which is the same as sk_reg->ref_obj_id.
189 *
190 * From the verifier perspective, if sk, fullsock and tp
191 * are not NULL, they are the same ptr with different
192 * reg->type. In particular, bpf_sk_release(tp) is also
193 * allowed and has the same effect as bpf_sk_release(sk).
194 */156 */
195 u32 ref_obj_id;157 u32 parent_id;
196 /* Inside the callee two registers can be both PTR_TO_STACK like158 /* Inside the callee two registers can be both PTR_TO_STACK like
197 * R1=fp-8 and R2=fp-8, but one of them points to this function stack159 * R1=fp-8 and R2=fp-8, but one of them points to this function stack
198 * while another to the caller's stack. To differentiate them 'frameno'160 * while another to the caller's stack. To differentiate them 'frameno'
@@ -209,6 +171,66 @@ struct bpf_reg_state { · 10 unmodified lines
209 bool precise;171 bool precise;
210};172};
211 173
174static inline s64 reg_smin(const struct bpf_reg_state *reg)
175{
176 return cnum64_smin(reg->r64);
177}
178
179static inline s64 reg_smax(const struct bpf_reg_state *reg)
180{
181 return cnum64_smax(reg->r64);
182}
183
184static inline u64 reg_umin(const struct bpf_reg_state *reg)
185{
186 return cnum64_umin(reg->r64);
187}
188
189static inline u64 reg_umax(const struct bpf_reg_state *reg)
190{
191 return cnum64_umax(reg->r64);
192}
193
194static inline s32 reg_s32_min(const struct bpf_reg_state *reg)
195{
196 return cnum32_smin(reg->r32);
197}
198
199static inline s32 reg_s32_max(const struct bpf_reg_state *reg)
200{
201 return cnum32_smax(reg->r32);
202}
203
204static inline u32 reg_u32_min(const struct bpf_reg_state *reg)
205{
206 return cnum32_umin(reg->r32);
207}
208
209static inline u32 reg_u32_max(const struct bpf_reg_state *reg)
210{
211 return cnum32_umax(reg->r32);
212}
213
214static inline void reg_set_srange32(struct bpf_reg_state *reg, s32 smin, s32 smax)
215{
216 reg->r32 = cnum32_from_srange(smin, smax);
217}
218
219static inline void reg_set_urange32(struct bpf_reg_state *reg, u32 umin, u32 umax)
220{
221 reg->r32 = cnum32_from_urange(umin, umax);
222}
223
224static inline void reg_set_srange64(struct bpf_reg_state *reg, s64 smin, s64 smax)
225{
226 reg->r64 = cnum64_from_srange(smin, smax);
227}
228
229static inline void reg_set_urange64(struct bpf_reg_state *reg, u64 umin, u64 umax)
230{
231 reg->r64 = cnum64_from_urange(umin, umax);
232}
233
212enum bpf_stack_slot_type {234enum bpf_stack_slot_type {
213 STACK_INVALID, /* nothing was stored in this stack slot */235 STACK_INVALID, /* nothing was stored in this stack slot */
214 STACK_SPILL, /* register spilled into stack */236 STACK_SPILL, /* register spilled into stack */
@@ -309,10 +331,14 @@ struct bpf_reference_state { · 94 unmodified lines
309 * is used purely to inform the user of a reference leak.331 * is used purely to inform the user of a reference leak.
310 */332 */
311 int insn_idx;333 int insn_idx;
312 /* Use to keep track of the source object of a lock, to ensure334 union {
313 * it matches on unlock.335 /* For REF_TYPE_PTR */
314 */336 int parent_id;
315 void *ptr;337 /* Use to keep track of the source object of a lock, to ensure
338 * it matches on unlock.
339 */
340 void *ptr;
341 };
316};342};
317 343
318struct bpf_retval_range {344struct bpf_retval_range {
@@ -347,6 +373,7 @@ struct bpf_func_state { · 28 unmodified lines
347 bool in_callback_fn;373 bool in_callback_fn;
348 bool in_async_callback_fn;374 bool in_async_callback_fn;
349 bool in_exception_callback_fn;375 bool in_exception_callback_fn;
376 bool no_stack_arg_load;
350 /* For callback calling functions that limit number of possible377 /* For callback calling functions that limit number of possible
351 * callback executions (e.g. bpf_loop) keeps track of current378 * callback executions (e.g. bpf_loop) keeps track of current
352 * simulated iteration number.379 * simulated iteration number.
@@ -372,46 +399,49 @@ struct bpf_func_state { · 19 unmodified lines
372 * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.399 * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.
373 */400 */
374 int allocated_stack;401 int allocated_stack;
402
403 u16 out_stack_arg_cnt; /* Number of outgoing on-stack argument slots */
404 struct bpf_reg_state *stack_arg_regs; /* Outgoing on-stack arguments */
375};405};
376 406
377#define MAX_CALL_FRAMES 8407#define MAX_CALL_FRAMES 16
378 408
379/* instruction history flags, used in bpf_jmp_history_entry.flags field */409/* instruction history flags, used in bpf_jmp_history_entry.flags field.
410 * Frame number and SPI are stored in dedicated fields of bpf_jmp_history_entry.
411 */
380enum {412enum {
381 /* instruction references stack slot through PTR_TO_STACK register;413 INSN_F_STACK_ACCESS = BIT(0),
382 * we also store stack's frame number in lower 3 bits (MAX_CALL_FRAMES is 8)
383 * and accessed stack slot's index in next 6 bits (MAX_BPF_STACK is 512,
384 * 8 bytes per slot, so slot index (spi) is [0, 63])
385 */
386 INSN_F_FRAMENO_MASK = 0x7, /* 3 bits */
387
388 INSN_F_SPI_MASK = 0x3f, /* 6 bits */
389 INSN_F_SPI_SHIFT = 3, /* shifted 3 bits to the left */
390 414
391 INSN_F_STACK_ACCESS = BIT(9),415 INSN_F_DST_REG_STACK = BIT(1), /* dst_reg is PTR_TO_STACK */
416 INSN_F_SRC_REG_STACK = BIT(2), /* src_reg is PTR_TO_STACK */
392 417
393 INSN_F_DST_REG_STACK = BIT(10), /* dst_reg is PTR_TO_STACK */418 INSN_F_STACK_ARG_ACCESS = BIT(3),
394 INSN_F_SRC_REG_STACK = BIT(11), /* src_reg is PTR_TO_STACK */
395 /* total 12 bits are used now. */
396};419};
397 420
398static_assert(INSN_F_FRAMENO_MASK + 1 >= MAX_CALL_FRAMES);
399static_assert(INSN_F_SPI_MASK + 1 >= MAX_BPF_STACK / 8);
400
401struct bpf_jmp_history_entry {421struct bpf_jmp_history_entry {
402 u32 idx;
403 /* insn idx can't be bigger than 1 million */422 /* insn idx can't be bigger than 1 million */
423 u32 idx : 20;
424 u32 frame : 4; /* stack access frame number */
425 u32 spi : 6; /* stack slot index (0..63) */
426 u32 : 2;
404 u32 prev_idx : 20;427 u32 prev_idx : 20;
405 /* special INSN_F_xxx flags */428 /* special INSN_F_xxx flags */
406 u32 flags : 12;429 u32 flags : 4;
407 /* additional registers that need precision tracking when this430 u32 : 8;
408 * jump is backtracked, vector of six 10-bit records431 /*
432 * additional registers that need precision tracking when this
433 * jump is backtracked, vector of five 11-bit records
409 */434 */
410 u64 linked_regs;435 u64 linked_regs;
411};436};
412 437
413/* Maximum number of register states that can exist at once */438static_assert(MAX_CALL_FRAMES <= (1 << 4));
414#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE) * MAX_CALL_FRAMES)439static_assert(MAX_BPF_STACK / 8 <= (1 << 6));
440
441/* Maximum number of bpf_reg_state objects that can exist at once */
442#define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
443#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE + \
444 MAX_STACK_ARG_SLOTS) * MAX_CALL_FRAMES)
415struct bpf_verifier_state {445struct bpf_verifier_state {
416 /* call stack tracking */446 /* call stack tracking */
417 struct bpf_func_state *frame[MAX_CALL_FRAMES];447 struct bpf_func_state *frame[MAX_CALL_FRAMES];
@@ -497,10 +527,23 @@ struct bpf_verifier_state { · 79 unmodified lines
497 u32 may_goto_depth;527 u32 may_goto_depth;
498};528};
499 529
500#define bpf_get_spilled_reg(slot, frame, mask) \530static inline struct bpf_reg_state *
501 (((slot < frame->allocated_stack / BPF_REG_SIZE) && \531bpf_get_spilled_reg(int slot, struct bpf_func_state *frame, u32 mask)
502 ((1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & (mask))) \532{
503 ? &frame->stack[slot].spilled_ptr : NULL)533 if (slot < frame->allocated_stack / BPF_REG_SIZE &&
534 (1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & mask)
535 return &frame->stack[slot].spilled_ptr;
536 return NULL;
537}
538
539static inline struct bpf_reg_state *
540bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)
541{
542 if (slot < frame->out_stack_arg_cnt &&
543 frame->stack_arg_regs[slot].type != NOT_INIT)
544 return &frame->stack_arg_regs[slot];
545 return NULL;
546}
504 547
505/* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */548/* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */
506#define bpf_for_each_spilled_reg(iter, frame, reg, mask) \549#define bpf_for_each_spilled_reg(iter, frame, reg, mask) \
@@ -508,7 +551,13 @@ struct bpf_verifier_state { · 1 unmodified line
508 iter < frame->allocated_stack / BPF_REG_SIZE; \551 iter < frame->allocated_stack / BPF_REG_SIZE; \
509 iter++, reg = bpf_get_spilled_reg(iter, frame, mask))552 iter++, reg = bpf_get_spilled_reg(iter, frame, mask))
510 553
511#define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __mask, __expr) \554/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */
555#define bpf_for_each_spilled_stack_arg(iter, frame, reg) \
556 for (iter = 0, reg = bpf_get_spilled_stack_arg(iter, frame); \
557 iter < frame->out_stack_arg_cnt; \
558 iter++, reg = bpf_get_spilled_stack_arg(iter, frame))
559
560#define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __stack, __mask, __expr) \
512 ({ \561 ({ \
513 struct bpf_verifier_state *___vstate = __vst; \562 struct bpf_verifier_state *___vstate = __vst; \
514 int ___i, ___j; \563 int ___i, ___j; \
@@ -516,6 +565,7 @@ struct bpf_verifier_state { · 1 unmodified line
516 struct bpf_reg_state *___regs; \565 struct bpf_reg_state *___regs; \
517 __state = ___vstate->frame[___i]; \566 __state = ___vstate->frame[___i]; \
518 ___regs = __state->regs; \567 ___regs = __state->regs; \
568 __stack = NULL; \
519 for (___j = 0; ___j < MAX_BPF_REG; ___j++) { \569 for (___j = 0; ___j < MAX_BPF_REG; ___j++) { \
520 __reg = &___regs[___j]; \570 __reg = &___regs[___j]; \
521 (void)(__expr); \571 (void)(__expr); \
@@ -523,14 +573,27 @@ struct bpf_verifier_state { · 1 unmodified line
523 bpf_for_each_spilled_reg(___j, __state, __reg, __mask) { \573 bpf_for_each_spilled_reg(___j, __state, __reg, __mask) { \
524 if (!__reg) \574 if (!__reg) \
525 continue; \575 continue; \
576 __stack = &__state->stack[___j]; \
526 (void)(__expr); \577 (void)(__expr); \
527 } \578 } \
579 __stack = NULL; \
580 bpf_for_each_spilled_stack_arg(___j, __state, __reg) { \
581 if (!__reg) \
582 continue; \
583 (void)(__expr); \
584 } \
528 } \585 } \
586 (void)__stack; \
529 })587 })
530 588
531/* Invoke __expr over regsiters in __vst, setting __state and __reg */589/* Invoke __expr over regsiters in __vst, setting __state and __reg */
532#define bpf_for_each_reg_in_vstate(__vst, __state, __reg, __expr) \590#define bpf_for_each_reg_in_vstate(__vst, __state, __reg, __expr) \
533 bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, 1 << STACK_SPILL, __expr)591 ({ \
592 struct bpf_stack_state * ___stack; \
593 (void)___stack; \
594 bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, ___stack,\
595 1 << STACK_SPILL, __expr); \
596 })
534 597
535/* linked list of verifier states used to prune search */598/* linked list of verifier states used to prune search */
536struct bpf_verifier_state_list {599struct bpf_verifier_state_list {
@@ -700,6 +763,22 @@ static inline bool bpf_verifier_log_needed(const struct bpf_verifier_log *log) · 163 unmodified lines
700 return log && log->level;763 return log && log->level;
701}764}
702 765
766struct bpf_log_attr {
767 char __user *ubuf;
768 u32 size;
769 u32 level;
770 u32 offsetof_true_size;
771 bpfptr_t uattr;
772};
773
774int bpf_log_attr_init(struct bpf_log_attr *log, u64 log_buf, u32 log_size, u32 log_level,
775 u32 offsetof_log_true_size, bpfptr_t uattr, struct bpf_common_attr *common,
776 bpfptr_t uattr_common, u32 size_common);
777struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log,
778 struct bpf_common_attr *common, bpfptr_t uattr,
779 u32 size);
780int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log);
781
703#define BPF_MAX_SUBPROGS 256782#define BPF_MAX_SUBPROGS 256
704 783
705struct bpf_subprog_arg_info {784struct bpf_subprog_arg_info {
@@ -724,6 +803,7 @@ struct bpf_subprog_info { · 18 unmodified lines
724 u32 exit_idx; /* Index of one of the BPF_EXIT instructions in this subprogram */803 u32 exit_idx; /* Index of one of the BPF_EXIT instructions in this subprogram */
725 u16 stack_depth; /* max. stack depth used by this function */804 u16 stack_depth; /* max. stack depth used by this function */
726 u16 stack_extra;805 u16 stack_extra;
806 u32 insn_processed;
727 /* offsets in range [stack_depth .. fastcall_stack_off)807 /* offsets in range [stack_depth .. fastcall_stack_off)
728 * are used for bpf_fastcall spills and fills.808 * are used for bpf_fastcall spills and fills.
729 */809 */
@@ -740,12 +820,21 @@ struct bpf_subprog_info { · 10 unmodified lines
740 bool keep_fastcall_stack: 1;820 bool keep_fastcall_stack: 1;
741 bool changes_pkt_data: 1;821 bool changes_pkt_data: 1;
742 bool might_sleep: 1;822 bool might_sleep: 1;
743 u8 arg_cnt:3;823 u8 arg_cnt:4;
744 824
745 enum priv_stack_mode priv_stack_mode;825 enum priv_stack_mode priv_stack_mode;
746 struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS];826 struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
827 u16 stack_arg_cnt; /* incoming + max outgoing */
828 u16 max_out_stack_arg_cnt;
747};829};
748 830
831static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub)
832{
833 if (sub->arg_cnt > MAX_BPF_FUNC_REG_ARGS)
834 return sub->arg_cnt - MAX_BPF_FUNC_REG_ARGS;
835 return 0;
836}
837
749struct bpf_verifier_env;838struct bpf_verifier_env;
750 839
751struct backtrack_state {840struct backtrack_state {
@@ -753,6 +842,7 @@ struct backtrack_state { · 1 unmodified line
753 u32 frame;842 u32 frame;
754 u32 reg_masks[MAX_CALL_FRAMES];843 u32 reg_masks[MAX_CALL_FRAMES];
755 u64 stack_masks[MAX_CALL_FRAMES];844 u64 stack_masks[MAX_CALL_FRAMES];
845 u8 stack_arg_masks[MAX_CALL_FRAMES];
756};846};
757 847
758struct bpf_id_pair {848struct bpf_id_pair {
@@ -881,6 +971,8 @@ struct bpf_verifier_env { · 122 unmodified lines
881 u32 prev_insn_processed, insn_processed;971 u32 prev_insn_processed, insn_processed;
882 /* number of jmps, calls, exits analyzed so far */972 /* number of jmps, calls, exits analyzed so far */
883 u32 prev_jmps_processed, jmps_processed;973 u32 prev_jmps_processed, jmps_processed;
974 /* maximum combined stack depth */
975 u32 max_stack_depth;
884 /* total verification time */976 /* total verification time */
885 u64 verification_time;977 u64 verification_time;
886 /* maximum number of verifier states kept in 'branching' instructions */978 /* maximum number of verifier states kept in 'branching' instructions */
@@ -914,6 +1006,7 @@ struct bpf_verifier_env { · 27 unmodified lines
914 * e.g., in reg_type_str() to generate reg_type string1006 * e.g., in reg_type_str() to generate reg_type string
915 */1007 */
916 char tmp_str_buf[TMP_STR_BUF_LEN];1008 char tmp_str_buf[TMP_STR_BUF_LEN];
1009 char tmp_arg_name[32];
917 struct bpf_insn insn_buf[INSN_BUF_SIZE];1010 struct bpf_insn insn_buf[INSN_BUF_SIZE];
918 struct bpf_insn epilogue_buf[INSN_BUF_SIZE];1011 struct bpf_insn epilogue_buf[INSN_BUF_SIZE];
919 struct bpf_scc_callchain callchain_buf;1012 struct bpf_scc_callchain callchain_buf;
@@ -1087,7 +1180,7 @@ struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx); · 167 unmodified lines
1087void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);1180void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);
1088void bpf_free_backedges(struct bpf_scc_visit *visit);1181void bpf_free_backedges(struct bpf_scc_visit *visit);
1089int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,1182int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
1090 int insn_flags, u64 linked_regs);1183 int insn_flags, int spi, int frame, u64 linked_regs);
1091void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist);1184void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist);
1092void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,1185void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
1093 struct bpf_reg_state *reg);1186 struct bpf_reg_state *reg);
@@ -1150,6 +1243,11 @@ static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame, · 56 unmodified lines
1150 bt->stack_masks[frame] |= 1ull << slot;1243 bt->stack_masks[frame] |= 1ull << slot;
1151}1244}
1152 1245
1246static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
1247{
1248 bt->stack_arg_masks[frame] |= 1 << slot;
1249}
1250
1153static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u32 reg)1251static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u32 reg)
1154{1252{
1155 return bt->reg_masks[frame] & (1 << reg);1253 return bt->reg_masks[frame] & (1 << reg);
@@ -1321,6 +1419,25 @@ struct bpf_map_desc { · 165 unmodified lines
1321 int uid;1419 int uid;
1322};1420};
1323 1421
1422/* The last initialized dynptr; Populated by process_dynptr_func() */
1423struct bpf_dynptr_desc {
1424 enum bpf_dynptr_type type;
1425 u32 id;
1426 u32 parent_id;
1427};
1428
1429/*
1430 * The last seen rereferenced object; Updated by update_ref_obj() when a register refers to a
1431 * referenced object. Used when the helper or kfunc is casting a referenced object, returning
1432 * allocated memory derived from referenced object or creating a dynptr with a referenced
1433 * object as parent.
1434 */
1435struct ref_obj_desc {
1436 u32 id;
1437 u32 parent_id;
1438 u8 cnt;
1439};
1440
1324struct bpf_kfunc_call_arg_meta {1441struct bpf_kfunc_call_arg_meta {
1325 /* In parameters */1442 /* In parameters */
1326 struct btf *btf;1443 struct btf *btf;
@@ -1329,7 +1446,6 @@ struct bpf_kfunc_call_arg_meta { · 2 unmodified lines
1329 const struct btf_type *func_proto;1446 const struct btf_type *func_proto;
1330 const char *func_name;1447 const char *func_name;
1331 /* Out parameters */1448 /* Out parameters */
1332 u32 ref_obj_id;
1333 u8 release_regno;1449 u8 release_regno;
1334 bool r0_rdonly;1450 bool r0_rdonly;
1335 u32 ret_btf_id;1451 u32 ret_btf_id;
@@ -1361,16 +1477,13 @@ struct bpf_kfunc_call_arg_meta { · 25 unmodified lines
1361 struct {1477 struct {
1362 struct btf_field *field;1478 struct btf_field *field;
1363 } arg_rbtree_root;1479 } arg_rbtree_root;
1364 struct {
1365 enum bpf_dynptr_type type;
1366 u32 id;
1367 u32 ref_obj_id;
1368 } initialized_dynptr;
1369 struct {1480 struct {
1370 u8 spi;1481 u8 spi;
1371 u8 frameno;1482 u8 frameno;
1372 } iter;1483 } iter;
1373 struct bpf_map_desc map;1484 struct bpf_map_desc map;
1485 struct bpf_dynptr_desc dynptr;
1486 struct ref_obj_desc ref_obj;
1374 u64 mem_size;1487 u64 mem_size;
1375};1488};
1376 1489
@@ -1479,6 +1592,10 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset); · 102 unmodified lines
1479int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,1592int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
1480 struct bpf_insn *insn_buf, int insn_idx, int *cnt);1593 struct bpf_insn *insn_buf, int insn_idx, int *cnt);
1481 1594
1595/* Functions exported from verifier.c, used by trampoline.c */
1596int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 btf_id,
1597 struct bpf_attach_target_info *tgt_info);
1598
1482/* Functions in fixups.c, called from bpf_check() */1599/* Functions in fixups.c, called from bpf_check() */
1483int bpf_remove_fastcall_spills_fills(struct bpf_verifier_env *env);1600int bpf_remove_fastcall_spills_fills(struct bpf_verifier_env *env);
1484int bpf_optimize_bpf_loop(struct bpf_verifier_env *env);1601int bpf_optimize_bpf_loop(struct bpf_verifier_env *env);
include/linux/btf.h+4−3
@@ -145,7 +145,8 @@ const char *btf_get_name(const struct btf *btf); · 144 unmodified lines
145void btf_get(struct btf *btf);145void btf_get(struct btf *btf);
146void btf_put(struct btf *btf);146void btf_put(struct btf *btf);
147const struct btf_header *btf_header(const struct btf *btf);147const struct btf_header *btf_header(const struct btf *btf);
148int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_sz);148struct bpf_log_attr;
149int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log);
149struct btf *btf_get_by_fd(int fd);150struct btf *btf_get_by_fd(int fd);
150int btf_get_info_by_fd(const struct btf *btf,151int btf_get_info_by_fd(const struct btf *btf,
151 const union bpf_attr *attr,152 const union bpf_attr *attr,
@@ -415,12 +416,12 @@ static inline bool btf_type_is_array(const struct btf_type *t) · 263 unmodified lines
415 return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY;416 return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY;
416}417}
417 418
418static inline u16 btf_type_vlen(const struct btf_type *t)419static inline u32 btf_type_vlen(const struct btf_type *t)
419{420{
420 return BTF_INFO_VLEN(t->info);421 return BTF_INFO_VLEN(t->info);
421}422}
422 423
423static inline u16 btf_vlen(const struct btf_type *t)424static inline u32 btf_vlen(const struct btf_type *t)
424{425{
425 return btf_type_vlen(t);426 return btf_type_vlen(t);
426}427}
include/linux/btf_ids.h+1−0
@@ -284,5 +284,6 @@ extern u32 bpf_cgroup_btf_id[]; · 283 unmodified lines
284extern u32 bpf_local_storage_map_btf_id[];284extern u32 bpf_local_storage_map_btf_id[];
285extern u32 btf_bpf_map_id[];285extern u32 btf_bpf_map_id[];
286extern u32 bpf_kmem_cache_btf_id[];286extern u32 bpf_kmem_cache_btf_id[];
287extern u32 bpf_multi_func_btf_id[];
287 288
288#endif289#endif
include/linux/cnum.h+82−0
@@ -0,0 +1,82 @@
1/* SPDX-License-Identifier: GPL-2.0-only */
2/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
3
4#ifndef _LINUX_CNUM_H
5#define _LINUX_CNUM_H
6
7#include <linux/types.h>
8
9/*
10 * cnum32: a circular number.
11 * A unified representation for signed and unsigned ranges.
12 *
13 * Assume that a 32-bit range is a circle, with 0 being in the 12 o'clock
14 * position, numbers placed sequentially in clockwise order and U32_MAX
15 * in the 11 o'clock position. Signed values map onto the same circle:
16 * S32_MAX sits at 5 o'clock, S32_MIN sits at 6 o'clock (opposite 0),
17 * negative values occupy the left half and positive values the right half.
18 *
19 * @cnum32 represents an arc on this circle drawn clockwise.
20 * @base corresponds to the first value of the range.
21 * @size corresponds to the number of integers in the range excluding @base.
22 * (The @base is excluded to avoid integer overflow when representing the full
23 * 0..U32_MAX range, which corresponds to 2^32, which can't be stored in u32).
24 *
25 * For example: {U32_MAX, 1} corresponds to signed range [-1, 0],
26 * {S32_MAX, 1} corresponds to unsigned range [S32_MAX, S32_MIN].
27 */
28struct cnum32 {
29 u32 base;
30 u32 size;
31};
32
33#define CNUM32_UNBOUNDED ((struct cnum32){ .base = 0, .size = U32_MAX })
34#define CNUM32_EMPTY ((struct cnum32){ .base = U32_MAX, .size = U32_MAX })
35
36struct cnum32 cnum32_from_urange(u32 min, u32 max);
37struct cnum32 cnum32_from_srange(s32 min, s32 max);
38u32 cnum32_umin(struct cnum32 cnum);
39u32 cnum32_umax(struct cnum32 cnum);
40s32 cnum32_smin(struct cnum32 cnum);
41s32 cnum32_smax(struct cnum32 cnum);
42struct cnum32 cnum32_intersect(struct cnum32 a, struct cnum32 b);
43void cnum32_intersect_with(struct cnum32 *dst, struct cnum32 src);
44void cnum32_intersect_with_urange(struct cnum32 *dst, u32 min, u32 max);
45void cnum32_intersect_with_srange(struct cnum32 *dst, s32 min, s32 max);
46bool cnum32_contains(struct cnum32 cnum, u32 v);
47bool cnum32_is_const(struct cnum32 cnum);
48bool cnum32_is_empty(struct cnum32 cnum);
49struct cnum32 cnum32_add(struct cnum32 a, struct cnum32 b);
50struct cnum32 cnum32_negate(struct cnum32 a);
51bool cnum32_is_subset(struct cnum32 outer, struct cnum32 inner);
52
53/* Same as cnum32 but for 64-bit ranges */
54struct cnum64 {
55 u64 base;
56 u64 size;
57};
58
59#define CNUM64_UNBOUNDED ((struct cnum64){ .base = 0, .size = U64_MAX })
60#define CNUM64_EMPTY ((struct cnum64){ .base = U64_MAX, .size = U64_MAX })
61
62struct cnum64 cnum64_from_urange(u64 min, u64 max);
63struct cnum64 cnum64_from_srange(s64 min, s64 max);
64u64 cnum64_umin(struct cnum64 cnum);
65u64 cnum64_umax(struct cnum64 cnum);
66s64 cnum64_smin(struct cnum64 cnum);
67s64 cnum64_smax(struct cnum64 cnum);
68struct cnum64 cnum64_intersect(struct cnum64 a, struct cnum64 b);
69void cnum64_intersect_with(struct cnum64 *dst, struct cnum64 src);
70void cnum64_intersect_with_urange(struct cnum64 *dst, u64 min, u64 max);
71void cnum64_intersect_with_srange(struct cnum64 *dst, s64 min, s64 max);
72bool cnum64_contains(struct cnum64 cnum, u64 v);
73bool cnum64_is_const(struct cnum64 cnum);
74bool cnum64_is_empty(struct cnum64 cnum);
75struct cnum64 cnum64_add(struct cnum64 a, struct cnum64 b);
76struct cnum64 cnum64_negate(struct cnum64 a);
77bool cnum64_is_subset(struct cnum64 outer, struct cnum64 inner);
78
79struct cnum32 cnum32_from_cnum64(struct cnum64 cnum);
80struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b);
81
82#endif /* _LINUX_CNUM_H */
include/linux/filter.h+25−2
@@ -58,8 +58,9 @@ struct ctl_table_header; · 57 unmodified lines
58#define BPF_REG_H BPF_REG_9 /* hlen, callee-saved */58#define BPF_REG_H BPF_REG_9 /* hlen, callee-saved */
59 59
60/* Kernel hidden auxiliary/helper register. */60/* Kernel hidden auxiliary/helper register. */
61#define BPF_REG_AX MAX_BPF_REG61#define BPF_REG_PARAMS MAX_BPF_REG
62#define MAX_BPF_EXT_REG (MAX_BPF_REG + 1)62#define BPF_REG_AX (MAX_BPF_REG + 1)
63#define MAX_BPF_EXT_REG (MAX_BPF_REG + 2)
63#define MAX_BPF_JIT_REG MAX_BPF_EXT_REG64#define MAX_BPF_JIT_REG MAX_BPF_EXT_REG
64 65
65/* unused opcode to mark special call to bpf_tail_call() helper */66/* unused opcode to mark special call to bpf_tail_call() helper */
@@ -748,6 +749,27 @@ static inline u32 bpf_prog_run_pin_on_cpu(const struct bpf_prog *prog, · 682 unmodified lines
748 return ret;749 return ret;
749}750}
750 751
752static inline bool is_stack_arg_ldx(const struct bpf_insn *insn)
753{
754 return insn->code == (BPF_LDX | BPF_MEM | BPF_DW) &&
755 insn->src_reg == BPF_REG_PARAMS &&
756 insn->off > 0 && insn->off % 8 == 0;
757}
758
759static inline bool is_stack_arg_st(const struct bpf_insn *insn)
760{
761 return insn->code == (BPF_ST | BPF_MEM | BPF_DW) &&
762 insn->dst_reg == BPF_REG_PARAMS &&
763 insn->off < 0 && insn->off % 8 == 0;
764}
765
766static inline bool is_stack_arg_stx(const struct bpf_insn *insn)
767{
768 return insn->code == (BPF_STX | BPF_MEM | BPF_DW) &&
769 insn->dst_reg == BPF_REG_PARAMS &&
770 insn->off < 0 && insn->off % 8 == 0;
771}
772
751#define BPF_SKB_CB_LEN QDISC_CB_PRIV_LEN773#define BPF_SKB_CB_LEN QDISC_CB_PRIV_LEN
752 774
753struct bpf_skb_data_end {775struct bpf_skb_data_end {
@@ -1159,6 +1181,7 @@ bool bpf_jit_inlines_helper_call(s32 imm); · 405 unmodified lines
1159bool bpf_jit_supports_subprog_tailcalls(void);1181bool bpf_jit_supports_subprog_tailcalls(void);
1160bool bpf_jit_supports_percpu_insn(void);1182bool bpf_jit_supports_percpu_insn(void);
1161bool bpf_jit_supports_kfunc_call(void);1183bool bpf_jit_supports_kfunc_call(void);
1184bool bpf_jit_supports_stack_args(void);
1162bool bpf_jit_supports_far_kfunc_call(void);1185bool bpf_jit_supports_far_kfunc_call(void);
1163bool bpf_jit_supports_exceptions(void);1186bool bpf_jit_supports_exceptions(void);
1164bool bpf_jit_supports_ptr_xchg(void);1187bool bpf_jit_supports_ptr_xchg(void);
include/linux/ftrace.h+9−0
@@ -415,6 +415,8 @@ struct ftrace_hash *alloc_ftrace_hash(int size_bits); · 414 unmodified lines
415void free_ftrace_hash(struct ftrace_hash *hash);415void free_ftrace_hash(struct ftrace_hash *hash);
416struct ftrace_func_entry *add_ftrace_hash_entry_direct(struct ftrace_hash *hash,416struct ftrace_func_entry *add_ftrace_hash_entry_direct(struct ftrace_hash *hash,
417 unsigned long ip, unsigned long direct);417 unsigned long ip, unsigned long direct);
418void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry);
419void ftrace_hash_remove(struct ftrace_hash *hash);
418 420
419/* The hash used to know what functions callbacks trace */421/* The hash used to know what functions callbacks trace */
420struct ftrace_ops_hash {422struct ftrace_ops_hash {
@@ -551,6 +553,8 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b · 130 unmodified lines
551 553
552void ftrace_stub_direct_tramp(void);554void ftrace_stub_direct_tramp(void);
553 555
556unsigned long ftrace_hash_count(struct ftrace_hash *hash);
557
554#else558#else
555struct ftrace_ops;559struct ftrace_ops;
556static inline unsigned long ftrace_find_rec_direct(unsigned long ip)560static inline unsigned long ftrace_find_rec_direct(unsigned long ip)
@@ -590,6 +594,11 @@ static inline int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace · 33 unmodified lines
590 return -ENODEV;594 return -ENODEV;
591}595}
592 596
597static inline unsigned long ftrace_hash_count(struct ftrace_hash *hash)
598{
599 return 0;
600}
601
593/*602/*
594 * This must be implemented by the architecture.603 * This must be implemented by the architecture.
595 * It is the way the ftrace direct_ops helper, when called604 * It is the way the ftrace direct_ops helper, when called
include/linux/pgtable.h+43−0
@@ -1070,6 +1070,49 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres · 1069 unmodified lines
1070}1070}
1071#endif1071#endif
1072 1072
1073#ifndef ptep_try_set
1074/**
1075 * ptep_try_set - atomically set an empty kernel PTE
1076 * @ptep: page table entry
1077 * @new_pte: value to install
1078 *
1079 * Atomically set *@ptep to @new_pte iff *@ptep is pte_none(). Return true on
1080 * success, false if the slot was already populated or the arch has no
1081 * implementation.
1082 *
1083 * For special kernel page tables only - never user page tables. The caller must
1084 * prevent concurrent teardown of @ptep and must accept that other writers may
1085 * race. Concurrent clearers must use ptep_get_and_clear() so racing accesses
1086 * agree on the outcome.
1087 *
1088 * Architectures opt in by providing a cmpxchg-based override and defining
1089 * ptep_try_set as an identity macro. The generic stub returns false, which is
1090 * correct for callers that fall through to oops on failure.
1091 */
1092static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte)
1093{
1094 return false;
1095}
1096#endif
1097
1098#ifndef flush_tlb_before_set
1099/**
1100 * flush_tlb_before_set - invalidate a kernel PTE's TLB before re-setting it
1101 * @addr: kernel virtual address whose PTE was just cleared
1102 *
1103 * Some architectures (e.g. arm64) do not allow a live page-table entry to be
1104 * repointed at a different page in one step. The old entry must first be made
1105 * invalid and its translation flushed from every TLB, and only then may the new
1106 * entry be written.
1107 *
1108 * This is only for the lockless atomic kernel-PTE installers (ptep_try_set()).
1109 * It must be callable with interrupts disabled.
1110 */
1111static inline void flush_tlb_before_set(unsigned long addr)
1112{
1113}
1114#endif
1115
1073#ifndef wrprotect_ptes1116#ifndef wrprotect_ptes
1074/**1117/**
1075 * wrprotect_ptes - Write-protect PTEs that map consecutive pages of the same1118 * wrprotect_ptes - Write-protect PTEs that map consecutive pages of the same
include/linux/rhashtable.h+3−1
@@ -263,6 +263,8 @@ struct rhash_lock_head __rcu **__rht_bucket_nested( · 262 unmodified lines
263struct rhash_lock_head __rcu **rht_bucket_nested_insert(263struct rhash_lock_head __rcu **rht_bucket_nested_insert(
264 struct rhashtable *ht, struct bucket_table *tbl, unsigned int hash);264 struct rhashtable *ht, struct bucket_table *tbl, unsigned int hash);
265 265
266void *rhashtable_next_key(struct rhashtable *ht, const void *prev_key);
267
266#define rht_dereference(p, ht) \268#define rht_dereference(p, ht) \
267 rcu_dereference_protected(p, lockdep_rht_mutex_is_held(ht))269 rcu_dereference_protected(p, lockdep_rht_mutex_is_held(ht))
268 270
@@ -1117,7 +1119,7 @@ unlocked: · 848 unmodified lines
1117 atomic_dec(&ht->nelems);1119 atomic_dec(&ht->nelems);
1118 if (unlikely(ht->p.automatic_shrinking &&1120 if (unlikely(ht->p.automatic_shrinking &&
1119 rht_shrink_below_30(ht, tbl)))1121 rht_shrink_below_30(ht, tbl)))
1120 schedule_work(&ht->run_work);1122 irq_work_queue(&ht->run_irq_work);
1121 err = 0;1123 err = 0;
1122 }1124 }
1123 1125
include/linux/syscalls.h+2−1
@@ -940,7 +940,8 @@ asmlinkage long sys_seccomp(unsigned int op, unsigned int flags, · 939 unmodified lines
940asmlinkage long sys_getrandom(char __user *buf, size_t count,940asmlinkage long sys_getrandom(char __user *buf, size_t count,
941 unsigned int flags);941 unsigned int flags);
942asmlinkage long sys_memfd_create(const char __user *uname_ptr, unsigned int flags);942asmlinkage long sys_memfd_create(const char __user *uname_ptr, unsigned int flags);
943asmlinkage long sys_bpf(int cmd, union bpf_attr __user *attr, unsigned int size);943asmlinkage long sys_bpf(int cmd, union bpf_attr __user *attr, unsigned int size,
944 struct bpf_common_attr __user *attr_common, unsigned int size_common);
944asmlinkage long sys_execveat(int dfd, const char __user *filename,945asmlinkage long sys_execveat(int dfd, const char __user *filename,
945 const char __user *const __user *argv,946 const char __user *const __user *argv,
946 const char __user *const __user *envp, int flags);947 const char __user *const __user *envp, int flags);
include/linux/trace_events.h+12−0
@@ -770,6 +770,7 @@ trace_trigger_soft_disabled(struct trace_event_file *file) · 769 unmodified lines
770 770
771#ifdef CONFIG_BPF_EVENTS771#ifdef CONFIG_BPF_EVENTS
772unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx);772unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx);
773unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx);
773int perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie);774int perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie);
774void perf_event_detach_bpf_prog(struct perf_event *event);775void perf_event_detach_bpf_prog(struct perf_event *event);
775int perf_event_query_prog_array(struct perf_event *event, void __user *info);776int perf_event_query_prog_array(struct perf_event *event, void __user *info);
@@ -786,12 +787,18 @@ int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id, · 10 unmodified lines
786 unsigned long *missed);787 unsigned long *missed);
787int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);788int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
788int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);789int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
790int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr);
789#else791#else
790static inline unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx)792static inline unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx)
791{793{
792 return 1;794 return 1;
793}795}
794 796
797static inline unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx)
798{
799 return 1;
800}
801
795static inline int802static inline int
796perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie)803perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie)
797{804{
@@ -838,6 +845,11 @@ bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) · 40 unmodified lines
838{845{
839 return -EOPNOTSUPP;846 return -EOPNOTSUPP;
840}847}
848static inline int
849bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
850{
851 return -EOPNOTSUPP;
852}
841#endif853#endif
842 854
843enum {855enum {
include/trace/bpf_probe.h+0−2
@@ -58,9 +58,7 @@ static notrace void \ · 57 unmodified lines
58__bpf_trace_##call(void *__data, proto) \58__bpf_trace_##call(void *__data, proto) \
59{ \59{ \
60 might_fault(); \60 might_fault(); \
61 preempt_disable_notrace(); \
62 CONCATENATE(bpf_trace_run, COUNT_ARGS(args))(__data, CAST_TO_U64(args)); \61 CONCATENATE(bpf_trace_run, COUNT_ARGS(args))(__data, CAST_TO_U64(args)); \
63 preempt_enable_notrace(); \
64}62}
65 63
66#undef DECLARE_EVENT_SYSCALL_CLASS64#undef DECLARE_EVENT_SYSCALL_CLASS
include/uapi/linux/bpf.h+31−1
@@ -994,6 +994,7 @@ enum bpf_cmd { · 993 unmodified lines
994 BPF_PROG_STREAM_READ_BY_FD,994 BPF_PROG_STREAM_READ_BY_FD,
995 BPF_PROG_ASSOC_STRUCT_OPS,995 BPF_PROG_ASSOC_STRUCT_OPS,
996 __MAX_BPF_CMD,996 __MAX_BPF_CMD,
997 BPF_COMMON_ATTRS = 1 << 16, /* Indicate carrying syscall common attrs. */
997};998};
998 999
999enum bpf_map_type {1000enum bpf_map_type {
@@ -1046,6 +1047,7 @@ enum bpf_map_type { · 46 unmodified lines
1046 BPF_MAP_TYPE_CGRP_STORAGE,1047 BPF_MAP_TYPE_CGRP_STORAGE,
1047 BPF_MAP_TYPE_ARENA,1048 BPF_MAP_TYPE_ARENA,
1048 BPF_MAP_TYPE_INSN_ARRAY,1049 BPF_MAP_TYPE_INSN_ARRAY,
1050 BPF_MAP_TYPE_RHASH,
1049 __MAX_BPF_MAP_TYPE1051 __MAX_BPF_MAP_TYPE
1050};1052};
1051 1053
@@ -1154,6 +1156,9 @@ enum bpf_attach_type { · 102 unmodified lines
1154 BPF_TRACE_KPROBE_SESSION,1156 BPF_TRACE_KPROBE_SESSION,
1155 BPF_TRACE_UPROBE_SESSION,1157 BPF_TRACE_UPROBE_SESSION,
1156 BPF_TRACE_FSESSION,1158 BPF_TRACE_FSESSION,
1159 BPF_TRACE_FENTRY_MULTI,
1160 BPF_TRACE_FEXIT_MULTI,
1161 BPF_TRACE_FSESSION_MULTI,
1157 __MAX_BPF_ATTACH_TYPE1162 __MAX_BPF_ATTACH_TYPE
1158};1163};
1159 1164
@@ -1178,6 +1183,7 @@ enum bpf_link_type { · 18 unmodified lines
1178 BPF_LINK_TYPE_UPROBE_MULTI = 12,1183 BPF_LINK_TYPE_UPROBE_MULTI = 12,
1179 BPF_LINK_TYPE_NETKIT = 13,1184 BPF_LINK_TYPE_NETKIT = 13,
1180 BPF_LINK_TYPE_SOCKMAP = 14,1185 BPF_LINK_TYPE_SOCKMAP = 14,
1186 BPF_LINK_TYPE_TRACING_MULTI = 15,
1181 __MAX_BPF_LINK_TYPE,1187 __MAX_BPF_LINK_TYPE,
1182};1188};
1183 1189
@@ -1321,7 +1327,11 @@ enum { · 137 unmodified lines
1321 * BPF_TRACE_UPROBE_MULTI attach type to create return probe.1327 * BPF_TRACE_UPROBE_MULTI attach type to create return probe.
1322 */1328 */
1323enum {1329enum {
1324 BPF_F_UPROBE_MULTI_RETURN = (1U << 0)1330 /* Get return uprobe. */
1331 BPF_F_UPROBE_MULTI_RETURN = (1U << 0),
1332
1333 /* Get path from provided path_fd. */
1334 BPF_F_UPROBE_MULTI_PATH_FD = (1U << 1),
1325};1335};
1326 1336
1327/* link_create.netfilter.flags used in LINK_CREATE command for1337/* link_create.netfilter.flags used in LINK_CREATE command for
@@ -1500,6 +1510,13 @@ struct bpf_stack_build_id { · 172 unmodified lines
1500 };1510 };
1501};1511};
1502 1512
1513struct bpf_common_attr {
1514 __aligned_u64 log_buf;
1515 __u32 log_size;
1516 __u32 log_level;
1517 __u32 log_true_size;
1518};
1519
1503#define BPF_OBJ_NAME_LEN 16U1520#define BPF_OBJ_NAME_LEN 16U
1504 1521
1505enum {1522enum {
@@ -1537,6 +1554,11 @@ union bpf_attr { · 31 unmodified lines
1537 *1554 *
1538 * BPF_MAP_TYPE_ARENA - contains the address where user space1555 * BPF_MAP_TYPE_ARENA - contains the address where user space
1539 * is going to mmap() the arena. It has to be page aligned.1556 * is going to mmap() the arena. It has to be page aligned.
1557 *
1558 * BPF_MAP_TYPE_RHASH - initial table size hint
1559 * (nelem_hint). 0 = use rhashtable default. Must be
1560 * <= min(max_entries, U16_MAX). Upper 32 bits reserved,
1561 * must be zero.
1540 */1562 */
1541 __u64 map_extra;1563 __u64 map_extra;
1542 1564
@@ -1846,6 +1868,7 @@ union bpf_attr { · 303 unmodified lines
1846 __u32 cnt;1868 __u32 cnt;
1847 __u32 flags;1869 __u32 flags;
1848 __u32 pid;1870 __u32 pid;
1871 __u32 path_fd;
1849 } uprobe_multi;1872 } uprobe_multi;
1850 struct {1873 struct {
1851 union {1874 union {
@@ -1861,6 +1884,11 @@ union bpf_attr { · 9 unmodified lines
1861 };1884 };
1862 __u64 expected_revision;1885 __u64 expected_revision;
1863 } cgroup;1886 } cgroup;
1887 struct {
1888 __aligned_u64 ids;
1889 __aligned_u64 cookies;
1890 __u32 cnt;
1891 } tracing_multi;
1864 };1892 };
1865 } link_create;1893 } link_create;
1866 1894
@@ -6698,6 +6726,7 @@ struct bpf_prog_info { · 4831 unmodified lines
6698 __u32 verified_insns;6726 __u32 verified_insns;
6699 __u32 attach_btf_obj_id;6727 __u32 attach_btf_obj_id;
6700 __u32 attach_btf_id;6728 __u32 attach_btf_id;
6729 __u32 :32;
6701} __attribute__((aligned(8)));6730} __attribute__((aligned(8)));
6702 6731
6703struct bpf_map_info {6732struct bpf_map_info {
@@ -6719,6 +6748,7 @@ struct bpf_map_info { · 15 unmodified lines
6719 __u64 map_extra;6748 __u64 map_extra;
6720 __aligned_u64 hash;6749 __aligned_u64 hash;
6721 __u32 hash_size;6750 __u32 hash_size;
6751 __u32 :32;
6722} __attribute__((aligned(8)));6752} __attribute__((aligned(8)));
6723 6753
6724struct bpf_btf_info {6754struct bpf_btf_info {
include/uapi/linux/btf.h+14−12
@@ -33,20 +33,22 @@ struct btf_header { · 32 unmodified lines
33 __u32 layout_len; /* length of layout section */33 __u32 layout_len; /* length of layout section */
34};34};
35 35
36/* Max # of type identifier */36enum btf_max {
37#define BTF_MAX_TYPE 0x000fffff37 /* Max possible kind */
38/* Max offset into the string section */38 BTF_MAX_KIND = 0x0000007f,
39#define BTF_MAX_NAME_OFFSET 0x00ffffff39 /* Max # of type identifier */
40/* Max # of struct/union/enum members or func args */40 BTF_MAX_TYPE = 0x000fffff,
41#define BTF_MAX_VLEN 0xffff41 /* Max offset into the string section */
42 BTF_MAX_NAME_OFFSET = 0x00ffffff,
43 /* Max # of struct/union/enum members or func args */
44 BTF_MAX_VLEN = 0x00ffffff,
45};
42 46
43struct btf_type {47struct btf_type {
44 __u32 name_off;48 __u32 name_off;
45 /* "info" bits arrangement49 /* "info" bits arrangement
46 * bits 0-15: vlen (e.g. # of struct's members)50 * bits 0-23: vlen (e.g. # of struct's members)
47 * bits 16-23: unused51 * bits 24-30: kind (e.g. int, ptr, array...etc)
48 * bits 24-28: kind (e.g. int, ptr, array...etc)
49 * bits 29-30: unused
50 * bit 31: kind_flag, currently used by52 * bit 31: kind_flag, currently used by
51 * struct, union, enum, fwd, enum64,53 * struct, union, enum, fwd, enum64,
52 * decl_tag and type_tag54 * decl_tag and type_tag
@@ -65,8 +67,8 @@ struct btf_type { · 12 unmodified lines
65 };67 };
66};68};
67 69
68#define BTF_INFO_KIND(info) (((info) >> 24) & 0x1f)70#define BTF_INFO_KIND(info) (((info) >> 24) & 0x7f)
69#define BTF_INFO_VLEN(info) ((info) & 0xffff)71#define BTF_INFO_VLEN(info) ((info) & 0xffffff)
70#define BTF_INFO_KFLAG(info) ((info) >> 31)72#define BTF_INFO_KFLAG(info) ((info) >> 31)
71 73
72enum {74enum {
kernel/bpf/Makefile+1−1
@@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse · 5 unmodified lines
6endif6endif
7CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)7CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)
8 8
9obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o liveness.o const_fold.o9obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o
10obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o10obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o
11obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o11obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
12obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o12obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
kernel/bpf/arena.c+266−49
@@ -53,12 +53,15 @@ struct bpf_arena { · 52 unmodified lines
53 u64 user_vm_start;53 u64 user_vm_start;
54 u64 user_vm_end;54 u64 user_vm_end;
55 struct vm_struct *kern_vm;55 struct vm_struct *kern_vm;
56 struct page *scratch_page;
56 struct range_tree rt;57 struct range_tree rt;
57 /* protects rt */58 /* protects rt */
58 rqspinlock_t spinlock;59 rqspinlock_t spinlock;
59 struct list_head vma_list;60 struct list_head vma_list;
60 /* protects vma_list */61 /* protects vma_list */
61 struct mutex lock;62 struct mutex lock;
63 u64 zap_gen;
64 struct mutex zap_mutex;
62 struct irq_work free_irq;65 struct irq_work free_irq;
63 struct work_struct free_work;66 struct work_struct free_work;
64 struct llist_head free_spans;67 struct llist_head free_spans;
@@ -83,6 +86,32 @@ u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena) · 18 unmodified lines
83 return arena ? arena->user_vm_start : 0;86 return arena ? arena->user_vm_start : 0;
84}87}
85 88
89/**
90 * bpf_arena_map_kern_vm_start - kern_vm_start lookup by struct bpf_map *
91 * @map: a BPF_MAP_TYPE_ARENA map
92 *
93 * Return @map's kern_vm_start.
94 */
95u64 bpf_arena_map_kern_vm_start(struct bpf_map *map)
96{
97 return bpf_arena_get_kern_vm_start(container_of(map, struct bpf_arena, map));
98}
99
100/**
101 * bpf_prog_arena - return the bpf_map of the arena referenced by @prog
102 * @prog: a loaded BPF program
103 *
104 * The verifier enforces at most one arena per program and stores it in
105 * prog->aux->arena. Return that arena's underlying bpf_map, or NULL if
106 * @prog does not reference an arena.
107 */
108struct bpf_map *bpf_prog_arena(struct bpf_prog *prog)
109{
110 struct bpf_arena *arena = prog->aux->arena;
111
112 return arena ? &arena->map : NULL;
113}
114
86static long arena_map_peek_elem(struct bpf_map *map, void *value)115static long arena_map_peek_elem(struct bpf_map *map, void *value)
87{116{
88 return -EOPNOTSUPP;117 return -EOPNOTSUPP;
@@ -115,26 +144,57 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr) · 26 unmodified lines
115 144
116struct apply_range_data {145struct apply_range_data {
117 struct page **pages;146 struct page **pages;
147 struct page *scratch_page;
118 int i;148 int i;
119};149};
120 150
151struct clear_range_data {
152 struct llist_head *free_pages;
153 struct page *scratch_page;
154};
155
121static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)156static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)
122{157{
123 struct apply_range_data *d = data;158 struct apply_range_data *d = data;
124 struct page *page;159 struct page *page;
160 pte_t pteval;
125 161
126 if (!data)162 if (!data)
127 return 0;163 return 0;
128 /* sanity check */
129 if (unlikely(!pte_none(ptep_get(pte))))
130 return -EBUSY;
131 164
132 page = d->pages[d->i];165 page = d->pages[d->i];
133 /* paranoia, similar to vmap_pages_pte_range() */166 /* paranoia, similar to vmap_pages_pte_range() */
134 if (WARN_ON_ONCE(!pfn_valid(page_to_pfn(page))))167 if (WARN_ON_ONCE(!pfn_valid(page_to_pfn(page))))
135 return -EINVAL;168 return -EINVAL;
136 169
137 set_pte_at(&init_mm, addr, pte, mk_pte(page, PAGE_KERNEL));170 pteval = mk_pte(page, PAGE_KERNEL);
171#ifdef ptep_try_set
172 /*
173 * Kernel-fault recovery may have installed the scratch page here, and
174 * some architectures (arm64) prohibit valid->valid PTE transitions.
175 * Install atomically into a none slot. If scratch is present, clear it
176 * and flush_tlb_before_set() (break-before-make) before retrying.
177 */
178 while (!ptep_try_set(pte, pteval)) {
179 pte_t old = ptep_get(pte);
180
181 if (pte_none(old))
182 continue;
183 if (WARN_ON_ONCE(pte_page(old) != d->scratch_page))
184 return -EBUSY;
185 ptep_get_and_clear(&init_mm, addr, pte);
186 flush_tlb_before_set(addr);
187 }
188#else
189 /*
190 * Without ptep_try_set() there is no atomic installer, but such arches
191 * also do not wire up bpf_arena_handle_page_fault(), so no scratch page
192 * is ever installed and the slot is always none here.
193 */
194 if (unlikely(!pte_none(ptep_get(pte))))
195 return -EBUSY;
196 set_pte_at(&init_mm, addr, pte, pteval);
197#endif
138 d->i++;198 d->i++;
139 return 0;199 return 0;
140}200}
@@ -144,33 +204,59 @@ static void flush_vmap_cache(unsigned long start, unsigned long size) · 3 unmodified lines
144 flush_cache_vmap(start, start + size);204 flush_cache_vmap(start, start + size);
145}205}
146 206
147static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *free_pages)207static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data)
148{208{
209 struct clear_range_data *d = data;
149 pte_t old_pte;210 pte_t old_pte;
150 struct page *page;211 struct page *page;
151 212
152 /* sanity check */213 /*
153 old_pte = ptep_get(pte);214 * Pairs with ptep_try_set() in the kernel-fault scratch installer.
215 * Both sides must be atomic.
216 */
217 old_pte = ptep_get_and_clear(&init_mm, addr, pte);
154 if (pte_none(old_pte) || !pte_present(old_pte))218 if (pte_none(old_pte) || !pte_present(old_pte))
155 return 0; /* nothing to do */219 return 0;
156 220
157 page = pte_page(old_pte);221 page = pte_page(old_pte);
158 if (WARN_ON_ONCE(!page))222 if (WARN_ON_ONCE(!page))
159 return -EINVAL;223 return -EINVAL;
160 224
161 pte_clear(&init_mm, addr, pte);225 /*
226 * Skip the per-arena scratch page. A kernel fault on an unallocated uaddr
227 * scratches its PTE. A later bpf_arena_free_pages() over that range walks
228 * here. Without the skip, scratch_page would be freed.
229 */
230 if (page == d->scratch_page)
231 return 0;
232
233 __llist_add(&page->pcp_llist, d->free_pages);
234 return 0;
235}
162 236
163 /* Add page to the list so it is freed later */237static int apply_range_set_scratch_cb(pte_t *pte, unsigned long addr, void *data)
164 if (free_pages)238{
165 __llist_add(&page->pcp_llist, free_pages);239 struct page *scratch_page = data;
166 240
241 if (!pte_none(ptep_get(pte)))
242 return 0;
243 /*
244 * Best-effort install. ptep_try_set() returns false only if another
245 * installer (real allocation or concurrent fault) won the cmpxchg.
246 * Their PTE is already valid, so the access retry succeeds.
247 *
248 * No flush_tlb_kernel_range() needed. Stale "not mapped" entries just
249 * cause one extra re-fault through this same path.
250 */
251 ptep_try_set(pte, mk_pte(scratch_page, PAGE_KERNEL));
167 return 0;252 return 0;
168}253}
169 254
170static int populate_pgtable_except_pte(struct bpf_arena *arena)255static int populate_pgtable_except_pte(struct bpf_arena *arena)
171{256{
257 /* Populate intermediates for the recovery range (4 GiB + upper half-guard). */
172 return apply_to_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena),258 return apply_to_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena),
173 KERN_VM_SZ - GUARD_SZ, apply_range_set_cb, NULL);259 SZ_4G + GUARD_SZ / 2, apply_range_set_cb, NULL);
174}260}
175 261
176static struct bpf_map *arena_map_alloc(union bpf_attr *attr)262static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
@@ -221,22 +307,30 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr) · 44 unmodified lines
221 init_irq_work(&arena->free_irq, arena_free_irq);307 init_irq_work(&arena->free_irq, arena_free_irq);
222 INIT_WORK(&arena->free_work, arena_free_worker);308 INIT_WORK(&arena->free_work, arena_free_worker);
223 bpf_map_init_from_attr(&arena->map, attr);309 bpf_map_init_from_attr(&arena->map, attr);
310
311 err = bpf_map_alloc_pages(&arena->map, NUMA_NO_NODE, 1, &arena->scratch_page);
312 if (err)
313 goto err_free_arena;
314
224 range_tree_init(&arena->rt);315 range_tree_init(&arena->rt);
225 err = range_tree_set(&arena->rt, 0, attr->max_entries);316 err = range_tree_set(&arena->rt, 0, attr->max_entries);
226 if (err) {317 if (err)
227 bpf_map_area_free(arena);318 goto err_free_scratch;
228 goto err;
229 }
230 mutex_init(&arena->lock);319 mutex_init(&arena->lock);
320 mutex_init(&arena->zap_mutex);
231 raw_res_spin_lock_init(&arena->spinlock);321 raw_res_spin_lock_init(&arena->spinlock);
232 err = populate_pgtable_except_pte(arena);322 err = populate_pgtable_except_pte(arena);
233 if (err) {323 if (err)
234 range_tree_destroy(&arena->rt);324 goto err_destroy_rt;
235 bpf_map_area_free(arena);
236 goto err;
237 }
238 325
239 return &arena->map;326 return &arena->map;
327
328err_destroy_rt:
329 range_tree_destroy(&arena->rt);
330err_free_scratch:
331 __free_page(arena->scratch_page);
332err_free_arena:
333 bpf_map_area_free(arena);
240err:334err:
241 free_vm_area(kern_vm);335 free_vm_area(kern_vm);
242 return ERR_PTR(err);336 return ERR_PTR(err);
@@ -244,6 +338,7 @@ err: · 1 unmodified line
244 338
245static int existing_page_cb(pte_t *ptep, unsigned long addr, void *data)339static int existing_page_cb(pte_t *ptep, unsigned long addr, void *data)
246{340{
341 struct bpf_arena *arena = data;
247 struct page *page;342 struct page *page;
248 pte_t pte;343 pte_t pte;
249 344
@@ -251,6 +346,12 @@ static int existing_page_cb(pte_t *ptep, unsigned long addr, void *data) · 1 unmodified line
251 if (!pte_present(pte)) /* sanity check */346 if (!pte_present(pte)) /* sanity check */
252 return 0;347 return 0;
253 page = pte_page(pte);348 page = pte_page(pte);
349 /*
350 * Skip the scratch page. The walk is page-table-driven, not range-tree-driven,
351 * so it can visit scratch PTEs at uaddrs the BPF program never allocated.
352 */
353 if (page == arena->scratch_page)
354 return 0;
254 /*355 /*
255 * We do not update pte here:356 * We do not update pte here:
256 * 1. Nobody should be accessing bpf_arena's range outside of a kernel bug357 * 1. Nobody should be accessing bpf_arena's range outside of a kernel bug
@@ -286,9 +387,10 @@ static void arena_map_free(struct bpf_map *map) · 29 unmodified lines
286 * free those pages.387 * free those pages.
287 */388 */
288 apply_to_existing_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena),389 apply_to_existing_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena),
289 KERN_VM_SZ - GUARD_SZ, existing_page_cb, NULL);390 SZ_4G + GUARD_SZ / 2, existing_page_cb, arena);
290 free_vm_area(arena->kern_vm);391 free_vm_area(arena->kern_vm);
291 range_tree_destroy(&arena->rt);392 range_tree_destroy(&arena->rt);
393 __free_page(arena->scratch_page);
292 bpf_map_area_free(arena);394 bpf_map_area_free(arena);
293}395}
294 396
@@ -318,6 +420,7 @@ struct vma_list { · 23 unmodified lines
318 struct vm_area_struct *vma;420 struct vm_area_struct *vma;
319 struct list_head head;421 struct list_head head;
320 refcount_t mmap_count;422 refcount_t mmap_count;
423 u64 zap_gen;
321};424};
322 425
323static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma)426static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma)
@@ -330,6 +433,7 @@ static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma) · 6 unmodified lines
330 refcount_set(&vml->mmap_count, 1);433 refcount_set(&vml->mmap_count, 1);
331 vma->vm_private_data = vml;434 vma->vm_private_data = vml;
332 vml->vma = vma;435 vml->vma = vma;
436 vml->zap_gen = 0;
333 list_add(&vml->head, &arena->vma_list);437 list_add(&vml->head, &arena->vma_list);
334 return 0;438 return 0;
335}439}
@@ -384,33 +488,38 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) · 48 unmodified lines
384 return VM_FAULT_RETRY;488 return VM_FAULT_RETRY;
385 489
386 page = vmalloc_to_page((void *)kaddr);490 page = vmalloc_to_page((void *)kaddr);
387 if (page)491 if (page) {
492 if (page == arena->scratch_page)
493 /* BPF triggered scratch here; don't lazy-alloc over it */
494 goto out_sigsegv;
388 /* already have a page vmap-ed */495 /* already have a page vmap-ed */
389 goto out;496 goto out;
497 }
390 498
391 bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);499 bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
392 500
393 if (arena->map.map_flags & BPF_F_SEGV_ON_FAULT)501 if (arena->map.map_flags & BPF_F_SEGV_ON_FAULT)
394 /* User space requested to segfault when page is not allocated by bpf prog */502 /* User space requested to segfault when page is not allocated by bpf prog */
395 goto out_unlock_sigsegv;503 goto out_sigsegv_memcg;
396 504
397 ret = range_tree_clear(&arena->rt, vmf->pgoff, 1);505 ret = range_tree_clear(&arena->rt, vmf->pgoff, 1);
398 if (ret)506 if (ret)
399 goto out_unlock_sigsegv;507 goto out_sigsegv_memcg;
400 508
401 struct apply_range_data data = { .pages = &page, .i = 0 };509 struct apply_range_data data = { .pages = &page, .i = 0,
510 .scratch_page = arena->scratch_page };
402 /* Account into memcg of the process that created bpf_arena */511 /* Account into memcg of the process that created bpf_arena */
403 ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page);512 ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page);
404 if (ret) {513 if (ret) {
405 range_tree_set(&arena->rt, vmf->pgoff, 1);514 range_tree_set(&arena->rt, vmf->pgoff, 1);
406 goto out_unlock_sigsegv;515 goto out_sigsegv_memcg;
407 }516 }
408 517
409 ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data);518 ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data);
410 if (ret) {519 if (ret) {
411 range_tree_set(&arena->rt, vmf->pgoff, 1);520 range_tree_set(&arena->rt, vmf->pgoff, 1);
412 free_pages_nolock(page, 0);521 free_pages_nolock(page, 0);
413 goto out_unlock_sigsegv;522 goto out_sigsegv_memcg;
414 }523 }
415 flush_vmap_cache(kaddr, PAGE_SIZE);524 flush_vmap_cache(kaddr, PAGE_SIZE);
416 bpf_map_memcg_exit(old_memcg, new_memcg);525 bpf_map_memcg_exit(old_memcg, new_memcg);
@@ -419,8 +528,9 @@ out: · 2 unmodified lines
419 raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);528 raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
420 vmf->page = page;529 vmf->page = page;
421 return 0;530 return 0;
422out_unlock_sigsegv:531out_sigsegv_memcg:
423 bpf_map_memcg_exit(old_memcg, new_memcg);532 bpf_map_memcg_exit(old_memcg, new_memcg);
533out_sigsegv:
424 raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);534 raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
425 return VM_FAULT_SIGSEGV;535 return VM_FAULT_SIGSEGV;
426}536}
@@ -587,6 +697,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · 160 unmodified lines
587 return 0;697 return 0;
588 }698 }
589 data.pages = pages;699 data.pages = pages;
700 data.scratch_page = arena->scratch_page;
590 701
591 if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))702 if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
592 goto out_free_pages;703 goto out_free_pages;
@@ -668,12 +779,60 @@ out_free_pages: · 75 unmodified lines
668 */779 */
669static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt)780static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt)
670{781{
782 unsigned long size = (unsigned long)page_cnt << PAGE_SHIFT;
783 struct vm_area_struct *vma;
784 struct mm_struct *mm;
671 struct vma_list *vml;785 struct vma_list *vml;
786 unsigned long vm_start;
787 u64 my_gen;
672 788
673 guard(mutex)(&arena->lock);789 /*
674 /* iterate link list under lock */790 * Taking mmap_read_lock() under arena->lock would deadlock against
675 list_for_each_entry(vml, &arena->vma_list, head)791 * arena_vm_close(), which runs with mmap_write_lock held and then
676 zap_vma_range(vml->vma, uaddr, PAGE_SIZE * page_cnt);792 * acquires arena->lock. Drop arena->lock for mmap_read_lock().
793 *
794 * Use per-call my_gen, recorded in vml->zap_gen, to remember which
795 * vmls this invocation has already processed across the lock drop.
796 * Hold zap_mutex around the whole walk so concurrent zap_pages()
797 * callers cannot overwrite each other's marks on shared vmls --
798 * otherwise call B's mark would make call A skip a vml that A has
799 * not yet zapped for A's uaddr range.
800 */
801 mutex_lock(&arena->zap_mutex);
802 mutex_lock(&arena->lock);
803 my_gen = ++arena->zap_gen;
804 for (;;) {
805 mm = NULL;
806 list_for_each_entry(vml, &arena->vma_list, head) {
807 if (vml->zap_gen >= my_gen)
808 continue;
809 vml->zap_gen = my_gen;
810 if (!mmget_not_zero(vml->vma->vm_mm))
811 continue;
812 mm = vml->vma->vm_mm;
813 vm_start = vml->vma->vm_start;
814 break;
815 }
816 if (!mm)
817 break;
818 mutex_unlock(&arena->lock);
819
820 mmap_read_lock(mm);
821 /*
822 * Re-resolve: while we waited the VMA could have been unmapped
823 * and a different mapping installed at the same address.
824 */
825 vma = find_vma(mm, vm_start);
826 if (vma && vma->vm_start == vm_start &&
827 vma->vm_file && vma->vm_file->private_data == &arena->map)
828 zap_vma_range(vma, uaddr, size);
829 mmap_read_unlock(mm);
830 mmput(mm);
831
832 mutex_lock(&arena->lock);
833 }
834 mutex_unlock(&arena->lock);
835 mutex_unlock(&arena->zap_mutex);
677}836}
678 837
679static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable)838static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable)
@@ -685,6 +844,7 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, · 5 unmodified lines
685 struct llist_head free_pages;844 struct llist_head free_pages;
686 struct llist_node *pos, *t;845 struct llist_node *pos, *t;
687 struct arena_free_span *s;846 struct arena_free_span *s;
847 struct clear_range_data cdata;
688 unsigned long flags;848 unsigned long flags;
689 int ret = 0;849 int ret = 0;
690 850
@@ -713,9 +873,11 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, · 22 unmodified lines
713 range_tree_set(&arena->rt, pgoff, page_cnt);873 range_tree_set(&arena->rt, pgoff, page_cnt);
714 874
715 init_llist_head(&free_pages);875 init_llist_head(&free_pages);
876 cdata.free_pages = &free_pages;
877 cdata.scratch_page = arena->scratch_page;
716 /* clear ptes and collect struct pages */878 /* clear ptes and collect struct pages */
717 apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,879 apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,
718 apply_range_clear_cb, &free_pages);880 apply_range_clear_cb, &cdata);
719 881
720 /* drop the lock to do the tlb flush and zap pages */882 /* drop the lock to do the tlb flush and zap pages */
721 raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);883 raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
@@ -805,6 +967,7 @@ static void arena_free_worker(struct work_struct *work) · 83 unmodified lines
805 struct arena_free_span *s;967 struct arena_free_span *s;
806 u64 arena_vm_start, user_vm_start;968 u64 arena_vm_start, user_vm_start;
807 struct llist_head free_pages;969 struct llist_head free_pages;
970 struct clear_range_data cdata;
808 struct page *page;971 struct page *page;
809 unsigned long full_uaddr;972 unsigned long full_uaddr;
810 long kaddr, page_cnt, pgoff;973 long kaddr, page_cnt, pgoff;
@@ -818,6 +981,8 @@ static void arena_free_worker(struct work_struct *work) · 7 unmodified lines
818 bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);981 bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
819 982
820 init_llist_head(&free_pages);983 init_llist_head(&free_pages);
984 cdata.free_pages = &free_pages;
985 cdata.scratch_page = arena->scratch_page;
821 arena_vm_start = bpf_arena_get_kern_vm_start(arena);986 arena_vm_start = bpf_arena_get_kern_vm_start(arena);
822 user_vm_start = bpf_arena_get_user_vm_start(arena);987 user_vm_start = bpf_arena_get_user_vm_start(arena);
823 988
@@ -830,7 +995,7 @@ static void arena_free_worker(struct work_struct *work) · 6 unmodified lines
830 995
831 /* clear ptes and collect pages in free_pages llist */996 /* clear ptes and collect pages in free_pages llist */
832 apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,997 apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,
833 apply_range_clear_cb, &free_pages);998 apply_range_clear_cb, &cdata);
834 999
835 range_tree_set(&arena->rt, pgoff, page_cnt);1000 range_tree_set(&arena->rt, pgoff, page_cnt);
836 }1001 }
@@ -893,6 +1058,19 @@ void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 pag · 56 unmodified lines
893 1058
894 return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, false);1059 return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, false);
895}1060}
1061
1062void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
1063 int node_id, u64 flags)
1064{
1065 struct bpf_map *map = p__map;
1066 struct bpf_arena *arena = container_of(map, struct bpf_arena, map);
1067
1068 if (map->map_type != BPF_MAP_TYPE_ARENA || flags || !page_cnt)
1069 return NULL;
1070
1071 return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, true);
1072}
1073
896__bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt)1074__bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt)
897{1075{
898 struct bpf_map *map = p__map;1076 struct bpf_map *map = p__map;
@@ -945,23 +1123,12 @@ static int __init kfunc_init(void) · 46 unmodified lines
945}1123}
946late_initcall(kfunc_init);1124late_initcall(kfunc_init);
947 1125
948void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip)1126static void __bpf_prog_report_arena_violation(struct bpf_prog *prog, bool write,
1127 unsigned long addr, unsigned long fault_ip)
949{1128{
950 struct bpf_stream_stage ss;1129 struct bpf_stream_stage ss;
951 struct bpf_prog *prog;
952 u64 user_vm_start;1130 u64 user_vm_start;
953 1131
954 /*
955 * The RCU read lock is held to safely traverse the latch tree, but we
956 * don't need its protection when accessing the prog, since it will not
957 * disappear while we are handling the fault.
958 */
959 rcu_read_lock();
960 prog = bpf_prog_ksym_find(fault_ip);
961 rcu_read_unlock();
962 if (!prog)
963 return;
964
965 /* Use main prog for stream access */1132 /* Use main prog for stream access */
966 prog = prog->aux->main_prog_aux->prog;1133 prog = prog->aux->main_prog_aux->prog;
967 1134
@@ -974,3 +1141,53 @@ void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned lo · 6 unmodified lines
974 bpf_stream_dump_stack(ss);1141 bpf_stream_dump_stack(ss);
975 }));1142 }));
976}1143}
1144
1145bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, unsigned long fault_ip)
1146{
1147 struct bpf_arena *arena;
1148 struct bpf_prog *prog;
1149 unsigned long kbase;
1150 unsigned long page_addr = addr & PAGE_MASK;
1151
1152 prog = bpf_prog_find_from_stack();
1153 if (!prog)
1154 return false;
1155
1156 arena = prog->aux->arena;
1157 /* a prog not using arena may be on stack, so arena can be NULL */
1158 if (!arena)
1159 return false;
1160
1161 kbase = bpf_arena_get_kern_vm_start(arena);
1162
1163 /*
1164 * Recovery covers the 4 GiB mappable band plus the upper half-guard.
1165 * Lower guard is unreachable from kfuncs; an address there indicates
1166 * a different bug class - leave it to the regular kernel oops path.
1167 */
1168 if (page_addr < kbase || page_addr >= kbase + SZ_4G + GUARD_SZ / 2)
1169 return false;
1170
1171 apply_to_page_range(&init_mm, page_addr, PAGE_SIZE,
1172 apply_range_set_scratch_cb, arena->scratch_page);
1173 flush_vmap_cache(page_addr, PAGE_SIZE);
1174 __bpf_prog_report_arena_violation(prog, is_write, page_addr - kbase, fault_ip);
1175 return true;
1176}
1177
1178void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip)
1179{
1180 struct bpf_prog *prog;
1181
1182 /*
1183 * The RCU read lock is held to safely traverse the latch tree, but we
1184 * don't need its protection when accessing the prog, since it will not
1185 * disappear while we are handling the fault.
1186 */
1187 rcu_read_lock();
1188 prog = bpf_prog_ksym_find(fault_ip);
1189 rcu_read_unlock();
1190 if (!prog)
1191 return;
1192 __bpf_prog_report_arena_violation(prog, write, addr, fault_ip);
1193}
kernel/bpf/arraymap.c+6−8
@@ -175,14 +175,12 @@ static void *array_map_lookup_elem(struct bpf_map *map, void *key) · 174 unmodified lines
175 return array->value + (u64)array->elem_size * (index & array->index_mask);175 return array->value + (u64)array->elem_size * (index & array->index_mask);
176}176}
177 177
178static int array_map_get_hash(struct bpf_map *map, u32 hash_buf_size,178static int array_map_get_hash(struct bpf_map *map)
179 void *hash_buf)
180{179{
181 struct bpf_array *array = container_of(map, struct bpf_array, map);180 struct bpf_array *array = container_of(map, struct bpf_array, map);
182 181
183 sha256(array->value, (u64)array->elem_size * array->map.max_entries,182 sha256(array->value, (u64)array->elem_size * array->map.max_entries,
184 hash_buf);183 array->map.sha);
185 memcpy(array->map.sha, hash_buf, sizeof(array->map.sha));
186 return 0;184 return 0;
187}185}
188 186
@@ -386,7 +384,7 @@ static long array_map_update_elem(struct bpf_map *map, void *key, void *value, · 197 unmodified lines
386 if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {384 if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
387 val = this_cpu_ptr(array->pptrs[index & array->index_mask]);385 val = this_cpu_ptr(array->pptrs[index & array->index_mask]);
388 copy_map_value(map, val, value);386 copy_map_value(map, val, value);
389 bpf_obj_free_fields(array->map.record, val);387 bpf_obj_cancel_fields(map, val);
390 } else {388 } else {
391 val = array->value +389 val = array->value +
392 (u64)array->elem_size * (index & array->index_mask);390 (u64)array->elem_size * (index & array->index_mask);
@@ -394,7 +392,7 @@ static long array_map_update_elem(struct bpf_map *map, void *key, void *value, · 1 unmodified line
394 copy_map_value_locked(map, val, value, false);392 copy_map_value_locked(map, val, value, false);
395 else393 else
396 copy_map_value(map, val, value);394 copy_map_value(map, val, value);
397 bpf_obj_free_fields(array->map.record, val);395 bpf_obj_cancel_fields(map, val);
398 }396 }
399 return 0;397 return 0;
400}398}
@@ -434,14 +432,14 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, · 33 unmodified lines
434 cpu = map_flags >> 32;432 cpu = map_flags >> 32;
435 ptr = per_cpu_ptr(pptr, cpu);433 ptr = per_cpu_ptr(pptr, cpu);
436 copy_map_value(map, ptr, value);434 copy_map_value(map, ptr, value);
437 bpf_obj_free_fields(array->map.record, ptr);435 bpf_obj_cancel_fields(map, ptr);
438 goto unlock;436 goto unlock;
439 }437 }
440 for_each_possible_cpu(cpu) {438 for_each_possible_cpu(cpu) {
441 ptr = per_cpu_ptr(pptr, cpu);439 ptr = per_cpu_ptr(pptr, cpu);
442 val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu;440 val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu;
443 copy_map_value(map, ptr, val);441 copy_map_value(map, ptr, val);
444 bpf_obj_free_fields(array->map.record, ptr);442 bpf_obj_cancel_fields(map, ptr);
445 }443 }
446unlock:444unlock:
447 rcu_read_unlock();445 rcu_read_unlock();
kernel/bpf/backtrack.c+66−16
@@ -9,7 +9,7 @@ · 8 unmodified lines
9 9
10/* for any branch, call, exit record the history of jmps in the given state */10/* for any branch, call, exit record the history of jmps in the given state */
11int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,11int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
12 int insn_flags, u64 linked_regs)12 int insn_flags, int spi, int frame, u64 linked_regs)
13{13{
14 u32 cnt = cur->jmp_history_cnt;14 u32 cnt = cur->jmp_history_cnt;
15 struct bpf_jmp_history_entry *p;15 struct bpf_jmp_history_entry *p;
@@ -25,6 +25,8 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state · 9 unmodified lines
25 env, "insn history: insn_idx %d cur flags %x new flags %x",25 env, "insn history: insn_idx %d cur flags %x new flags %x",
26 env->insn_idx, env->cur_hist_ent->flags, insn_flags);26 env->insn_idx, env->cur_hist_ent->flags, insn_flags);
27 env->cur_hist_ent->flags |= insn_flags;27 env->cur_hist_ent->flags |= insn_flags;
28 env->cur_hist_ent->spi = spi;
29 env->cur_hist_ent->frame = frame;
28 verifier_bug_if(env->cur_hist_ent->linked_regs != 0, env,30 verifier_bug_if(env->cur_hist_ent->linked_regs != 0, env,
29 "insn history: insn_idx %d linked_regs: %#llx",31 "insn history: insn_idx %d linked_regs: %#llx",
30 env->insn_idx, env->cur_hist_ent->linked_regs);32 env->insn_idx, env->cur_hist_ent->linked_regs);
@@ -43,6 +45,8 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state · 12 unmodified lines
43 p->idx = env->insn_idx;45 p->idx = env->insn_idx;
44 p->prev_idx = env->prev_insn_idx;46 p->prev_idx = env->prev_insn_idx;
45 p->flags = insn_flags;47 p->flags = insn_flags;
48 p->spi = spi;
49 p->frame = frame;
46 p->linked_regs = linked_regs;50 p->linked_regs = linked_regs;
47 cur->jmp_history_cnt = cnt;51 cur->jmp_history_cnt = cnt;
48 env->cur_hist_ent = p;52 env->cur_hist_ent = p;
@@ -64,16 +68,6 @@ static bool is_atomic_fetch_insn(const struct bpf_insn *insn) · 15 unmodified lines
64 (insn->imm & BPF_FETCH);68 (insn->imm & BPF_FETCH);
65}69}
66 70
67static int insn_stack_access_spi(int insn_flags)
68{
69 return (insn_flags >> INSN_F_SPI_SHIFT) & INSN_F_SPI_MASK;
70}
71
72static int insn_stack_access_frameno(int insn_flags)
73{
74 return insn_flags & INSN_F_FRAMENO_MASK;
75}
76
77/* Backtrack one insn at a time. If idx is not at the top of recorded71/* Backtrack one insn at a time. If idx is not at the top of recorded
78 * history then previous instruction came from straight line execution.72 * history then previous instruction came from straight line execution.
79 * Return -ENOENT if we exhausted all instructions within given state.73 * Return -ENOENT if we exhausted all instructions within given state.
@@ -135,11 +129,21 @@ static inline u32 bt_empty(struct backtrack_state *bt) · 55 unmodified lines
135 int i;129 int i;
136 130
137 for (i = 0; i <= bt->frame; i++)131 for (i = 0; i <= bt->frame; i++)
138 mask |= bt->reg_masks[i] | bt->stack_masks[i];132 mask |= bt->reg_masks[i] | bt->stack_masks[i] | bt->stack_arg_masks[i];
139 133
140 return mask == 0;134 return mask == 0;
141}135}
142 136
137static inline void bt_clear_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
138{
139 bt->stack_arg_masks[frame] &= ~(1 << slot);
140}
141
142static inline bool bt_is_frame_stack_arg_slot_set(struct backtrack_state *bt, u32 frame, u32 slot)
143{
144 return bt->stack_arg_masks[frame] & (1 << slot);
145}
146
143static inline int bt_subprog_enter(struct backtrack_state *bt)147static inline int bt_subprog_enter(struct backtrack_state *bt)
144{148{
145 if (bt->frame == MAX_CALL_FRAMES - 1) {149 if (bt->frame == MAX_CALL_FRAMES - 1) {
@@ -200,6 +204,11 @@ static inline u64 bt_stack_mask(struct backtrack_state *bt) · 54 unmodified lines
200 return bt->stack_masks[bt->frame];204 return bt->stack_masks[bt->frame];
201}205}
202 206
207static inline u8 bt_stack_arg_mask(struct backtrack_state *bt)
208{
209 return bt->stack_arg_masks[bt->frame];
210}
211
203static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg)212static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg)
204{213{
205 return bt->reg_masks[bt->frame] & (1 << reg);214 return bt->reg_masks[bt->frame] & (1 << reg);
@@ -341,6 +350,19 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 135 unmodified lines
341 return 0;350 return 0;
342 bt_clear_reg(bt, load_reg);351 bt_clear_reg(bt, load_reg);
343 352
353 if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) {
354 spi = hist->spi;
355 /*
356 * Stack arg read: callee reads from r11+off, but
357 * the data lives in the caller's stack_arg_regs.
358 * Set the mask in the caller frame so precision
359 * is marked in the caller's slot at the callee
360 * entry checkpoint.
361 */
362 bt_set_frame_stack_arg_slot(bt, bt->frame - 1, spi);
363 return 0;
364 }
365
344 /* scalars can only be spilled into stack w/o losing precision.366 /* scalars can only be spilled into stack w/o losing precision.
345 * Load from any other memory can be zero extended.367 * Load from any other memory can be zero extended.
346 * The desire to keep that precision is already indicated368 * The desire to keep that precision is already indicated
@@ -353,8 +375,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 6 unmodified lines
353 * that [fp - off] slot contains scalar that needs to be375 * that [fp - off] slot contains scalar that needs to be
354 * tracked with precision376 * tracked with precision
355 */377 */
356 spi = insn_stack_access_spi(hist->flags);378 spi = hist->spi;
357 fr = insn_stack_access_frameno(hist->flags);379 fr = hist->frame;
358 bpf_bt_set_frame_slot(bt, fr, spi);380 bpf_bt_set_frame_slot(bt, fr, spi);
359 } else if (class == BPF_STX || class == BPF_ST) {381 } else if (class == BPF_STX || class == BPF_ST) {
360 if (bt_is_reg_set(bt, dreg))382 if (bt_is_reg_set(bt, dreg))
@@ -363,11 +385,22 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 2 unmodified lines
363 * encountered a case of pointer subtraction.385 * encountered a case of pointer subtraction.
364 */386 */
365 return -ENOTSUPP;387 return -ENOTSUPP;
388
389 if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) {
390 spi = hist->spi;
391 if (!bt_is_frame_stack_arg_slot_set(bt, bt->frame, spi))
392 return 0;
393 bt_clear_frame_stack_arg_slot(bt, bt->frame, spi);
394 if (class == BPF_STX)
395 bt_set_reg(bt, sreg);
396 return 0;
397 }
398
366 /* scalars can only be spilled into stack */399 /* scalars can only be spilled into stack */
367 if (!hist || !(hist->flags & INSN_F_STACK_ACCESS))400 if (!hist || !(hist->flags & INSN_F_STACK_ACCESS))
368 return 0;401 return 0;
369 spi = insn_stack_access_spi(hist->flags);402 spi = hist->spi;
370 fr = insn_stack_access_frameno(hist->flags);403 fr = hist->frame;
371 if (!bt_is_frame_slot_set(bt, fr, spi))404 if (!bt_is_frame_slot_set(bt, fr, spi))
372 return 0;405 return 0;
373 bt_clear_frame_slot(bt, fr, spi);406 bt_clear_frame_slot(bt, fr, spi);
@@ -431,6 +464,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 57 unmodified lines
431 bpf_bt_set_frame_reg(bt, bt->frame - 1, i);464 bpf_bt_set_frame_reg(bt, bt->frame - 1, i);
432 }465 }
433 }466 }
467 if (bt_stack_arg_mask(bt)) {
468 verifier_bug(env,
469 "static subprog leftover stack arg slots %x",
470 bt_stack_arg_mask(bt));
471 return -EFAULT;
472 }
434 if (bt_subprog_exit(bt))473 if (bt_subprog_exit(bt))
435 return -EFAULT;474 return -EFAULT;
436 return 0;475 return 0;
@@ -901,6 +940,17 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env, · 464 unmodified lines
901 *changed = true;940 *changed = true;
902 }941 }
903 }942 }
943 for (i = 0; i < func->out_stack_arg_cnt; i++) {
944 if (!bt_is_frame_stack_arg_slot_set(bt, fr, i))
945 continue;
946 reg = &func->stack_arg_regs[i];
947 if (reg->type != SCALAR_VALUE || reg->precise) {
948 bt_clear_frame_stack_arg_slot(bt, fr, i);
949 } else {
950 reg->precise = true;
951 *changed = true;
952 }
953 }
904 if (env->log.level & BPF_LOG_LEVEL2) {954 if (env->log.level & BPF_LOG_LEVEL2) {
905 fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,955 fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
906 bt_frame_reg_mask(bt, fr));956 bt_frame_reg_mask(bt, fr));
kernel/bpf/bpf_lru_list.c+100−65
@@ -13,23 +13,8 @@ · 12 unmodified lines
13#define PERCPU_FREE_TARGET (4)13#define PERCPU_FREE_TARGET (4)
14#define PERCPU_NR_SCANS PERCPU_FREE_TARGET14#define PERCPU_NR_SCANS PERCPU_FREE_TARGET
15 15
16/* Helpers to get the local list index */
17#define LOCAL_LIST_IDX(t) ((t) - BPF_LOCAL_LIST_T_OFFSET)
18#define LOCAL_FREE_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_FREE)
19#define LOCAL_PENDING_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_PENDING)
20#define IS_LOCAL_LIST_TYPE(t) ((t) >= BPF_LOCAL_LIST_T_OFFSET)16#define IS_LOCAL_LIST_TYPE(t) ((t) >= BPF_LOCAL_LIST_T_OFFSET)
21 17
22/* Local list helpers */
23static struct list_head *local_free_list(struct bpf_lru_locallist *loc_l)
24{
25 return &loc_l->lists[LOCAL_FREE_LIST_IDX];
26}
27
28static struct list_head *local_pending_list(struct bpf_lru_locallist *loc_l)
29{
30 return &loc_l->lists[LOCAL_PENDING_LIST_IDX];
31}
32
33/* bpf_lru_node helpers */18/* bpf_lru_node helpers */
34static bool bpf_lru_node_is_ref(const struct bpf_lru_node *node)19static bool bpf_lru_node_is_ref(const struct bpf_lru_node *node)
35{20{
@@ -72,6 +57,7 @@ static void __bpf_lru_node_move_to_free(struct bpf_lru_list *l, · 36 unmodified lines
72 bpf_lru_list_count_dec(l, node->type);57 bpf_lru_list_count_dec(l, node->type);
73 58
74 node->type = tgt_free_type;59 node->type = tgt_free_type;
60 WRITE_ONCE(node->pending_free, 0);
75 list_move(&node->list, free_list);61 list_move(&node->list, free_list);
76}62}
77 63
@@ -87,6 +73,9 @@ static void __bpf_lru_node_move_in(struct bpf_lru_list *l, · 9 unmodified lines
87 bpf_lru_list_count_inc(l, tgt_type);73 bpf_lru_list_count_inc(l, tgt_type);
88 node->type = tgt_type;74 node->type = tgt_type;
89 bpf_lru_node_clear_ref(node);75 bpf_lru_node_clear_ref(node);
76 /* Reset pending_free only when moving to the free list */
77 if (tgt_type == BPF_LRU_LIST_T_FREE)
78 WRITE_ONCE(node->pending_free, 0);
90 list_move(&node->list, &l->lists[tgt_type]);79 list_move(&node->list, &l->lists[tgt_type]);
91}80}
92 81
@@ -212,9 +201,11 @@ __bpf_lru_list_shrink_inactive(struct bpf_lru *lru, · 119 unmodified lines
212 unsigned int i = 0;201 unsigned int i = 0;
213 202
214 list_for_each_entry_safe_reverse(node, tmp_node, inactive, list) {203 list_for_each_entry_safe_reverse(node, tmp_node, inactive, list) {
215 if (bpf_lru_node_is_ref(node)) {204 if (bpf_lru_node_is_ref(node) &&
205 !READ_ONCE(node->pending_free)) {
216 __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_ACTIVE);206 __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_ACTIVE);
217 } else if (lru->del_from_htab(lru->del_arg, node)) {207 } else if (READ_ONCE(node->pending_free) ||
208 lru->del_from_htab(lru->del_arg, node)) {
218 __bpf_lru_node_move_to_free(l, node, free_list,209 __bpf_lru_node_move_to_free(l, node, free_list,
219 tgt_free_type);210 tgt_free_type);
220 if (++nshrinked == tgt_nshrink)211 if (++nshrinked == tgt_nshrink)
@@ -273,7 +264,8 @@ static unsigned int __bpf_lru_list_shrink(struct bpf_lru *lru, · 52 unmodified lines
273 264
274 list_for_each_entry_safe_reverse(node, tmp_node, force_shrink_list,265 list_for_each_entry_safe_reverse(node, tmp_node, force_shrink_list,
275 list) {266 list) {
276 if (lru->del_from_htab(lru->del_arg, node)) {267 if (READ_ONCE(node->pending_free) ||
268 lru->del_from_htab(lru->del_arg, node)) {
277 __bpf_lru_node_move_to_free(l, node, free_list,269 __bpf_lru_node_move_to_free(l, node, free_list,
278 tgt_free_type);270 tgt_free_type);
279 return 1;271 return 1;
@@ -290,8 +282,10 @@ static void __local_list_flush(struct bpf_lru_list *l, · 10 unmodified lines
290 struct bpf_lru_node *node, *tmp_node;282 struct bpf_lru_node *node, *tmp_node;
291 283
292 list_for_each_entry_safe_reverse(node, tmp_node,284 list_for_each_entry_safe_reverse(node, tmp_node,
293 local_pending_list(loc_l), list) {285 &loc_l->pending_list, list) {
294 if (bpf_lru_node_is_ref(node))286 if (READ_ONCE(node->pending_free))
287 __bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_FREE);
288 else if (bpf_lru_node_is_ref(node))
295 __bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_ACTIVE);289 __bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_ACTIVE);
296 else290 else
297 __bpf_lru_node_move_in(l, node,291 __bpf_lru_node_move_in(l, node,
@@ -307,9 +301,12 @@ static void bpf_lru_list_push_free(struct bpf_lru_list *l, · 9 unmodified lines
307 if (WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(node->type)))301 if (WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(node->type)))
308 return;302 return;
309 303
310 raw_spin_lock_irqsave(&l->lock, flags);304 if (raw_res_spin_lock_irqsave(&l->lock, flags)) {
305 WRITE_ONCE(node->pending_free, 1);
306 return;
307 }
311 __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE);308 __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE);
312 raw_spin_unlock_irqrestore(&l->lock, flags);309 raw_res_spin_unlock_irqrestore(&l->lock, flags);
313}310}
314 311
315static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru,312static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru,
@@ -318,8 +315,10 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, · 2 unmodified lines
318 struct bpf_lru_list *l = &lru->common_lru.lru_list;315 struct bpf_lru_list *l = &lru->common_lru.lru_list;
319 struct bpf_lru_node *node, *tmp_node;316 struct bpf_lru_node *node, *tmp_node;
320 unsigned int nfree = 0;317 unsigned int nfree = 0;
318 LIST_HEAD(tmp_free);
321 319
322 raw_spin_lock(&l->lock);320 if (raw_res_spin_lock(&l->lock))
321 return;
323 322
324 __local_list_flush(l, loc_l);323 __local_list_flush(l, loc_l);
325 324
@@ -327,7 +326,7 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, · 1 unmodified line
327 326
328 list_for_each_entry_safe(node, tmp_node, &l->lists[BPF_LRU_LIST_T_FREE],327 list_for_each_entry_safe(node, tmp_node, &l->lists[BPF_LRU_LIST_T_FREE],
329 list) {328 list) {
330 __bpf_lru_node_move_to_free(l, node, local_free_list(loc_l),329 __bpf_lru_node_move_to_free(l, node, &tmp_free,
331 BPF_LRU_LOCAL_LIST_T_FREE);330 BPF_LRU_LOCAL_LIST_T_FREE);
332 if (++nfree == lru->target_free)331 if (++nfree == lru->target_free)
333 break;332 break;
@@ -335,10 +334,19 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, · 1 unmodified line
335 334
336 if (nfree < lru->target_free)335 if (nfree < lru->target_free)
337 __bpf_lru_list_shrink(lru, l, lru->target_free - nfree,336 __bpf_lru_list_shrink(lru, l, lru->target_free - nfree,
338 local_free_list(loc_l),337 &tmp_free,
339 BPF_LRU_LOCAL_LIST_T_FREE);338 BPF_LRU_LOCAL_LIST_T_FREE);
340 339
341 raw_spin_unlock(&l->lock);340 raw_res_spin_unlock(&l->lock);
341
342 /*
343 * Transfer the harvested nodes from the temporary list_head into
344 * the lockless per-CPU free llist.
345 */
346 list_for_each_entry_safe(node, tmp_node, &tmp_free, list) {
347 list_del(&node->list);
348 llist_add(&node->llist, &loc_l->free_llist);
349 }
342}350}
343 351
344static void __local_list_add_pending(struct bpf_lru *lru,352static void __local_list_add_pending(struct bpf_lru *lru,
@@ -350,22 +358,21 @@ static void __local_list_add_pending(struct bpf_lru *lru, · 5 unmodified lines
350 *(u32 *)((void *)node + lru->hash_offset) = hash;358 *(u32 *)((void *)node + lru->hash_offset) = hash;
351 node->cpu = cpu;359 node->cpu = cpu;
352 node->type = BPF_LRU_LOCAL_LIST_T_PENDING;360 node->type = BPF_LRU_LOCAL_LIST_T_PENDING;
361 WRITE_ONCE(node->pending_free, 0);
353 bpf_lru_node_clear_ref(node);362 bpf_lru_node_clear_ref(node);
354 list_add(&node->list, local_pending_list(loc_l));363 list_add(&node->list, &loc_l->pending_list);
355}364}
356 365
357static struct bpf_lru_node *366static struct bpf_lru_node *
358__local_list_pop_free(struct bpf_lru_locallist *loc_l)367__local_list_pop_free(struct bpf_lru_locallist *loc_l)
359{368{
360 struct bpf_lru_node *node;369 struct llist_node *llnode;
361 370
362 node = list_first_entry_or_null(local_free_list(loc_l),371 llnode = llist_del_first(&loc_l->free_llist);
363 struct bpf_lru_node,372 if (!llnode)
364 list);373 return NULL;
365 if (node)
366 list_del(&node->list);
367 374
368 return node;375 return container_of(llnode, struct bpf_lru_node, llist);
369}376}
370 377
371static struct bpf_lru_node *378static struct bpf_lru_node *
@@ -376,10 +383,10 @@ __local_list_pop_pending(struct bpf_lru *lru, struct bpf_lru_locallist *loc_l) · 4 unmodified lines
376 383
377ignore_ref:384ignore_ref:
378 /* Get from the tail (i.e. older element) of the pending list. */385 /* Get from the tail (i.e. older element) of the pending list. */
379 list_for_each_entry_reverse(node, local_pending_list(loc_l),386 list_for_each_entry_reverse(node, &loc_l->pending_list, list) {
380 list) {
381 if ((!bpf_lru_node_is_ref(node) || force) &&387 if ((!bpf_lru_node_is_ref(node) || force) &&
382 lru->del_from_htab(lru->del_arg, node)) {388 (READ_ONCE(node->pending_free) ||
389 lru->del_from_htab(lru->del_arg, node))) {
383 list_del(&node->list);390 list_del(&node->list);
384 return node;391 return node;
385 }392 }
@@ -404,7 +411,8 @@ static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru, · 18 unmodified lines
404 411
405 l = per_cpu_ptr(lru->percpu_lru, cpu);412 l = per_cpu_ptr(lru->percpu_lru, cpu);
406 413
407 raw_spin_lock_irqsave(&l->lock, flags);414 if (raw_res_spin_lock_irqsave(&l->lock, flags))
415 return NULL;
408 416
409 __bpf_lru_list_rotate(lru, l);417 __bpf_lru_list_rotate(lru, l);
410 418
@@ -420,7 +428,7 @@ static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru, · 9 unmodified lines
420 __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_INACTIVE);428 __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_INACTIVE);
421 }429 }
422 430
423 raw_spin_unlock_irqrestore(&l->lock, flags);431 raw_res_spin_unlock_irqrestore(&l->lock, flags);
424 432
425 return node;433 return node;
426}434}
@@ -437,7 +445,8 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru, · 10 unmodified lines
437 445
438 loc_l = per_cpu_ptr(clru->local_list, cpu);446 loc_l = per_cpu_ptr(clru->local_list, cpu);
439 447
440 raw_spin_lock_irqsave(&loc_l->lock, flags);448 if (raw_res_spin_lock_irqsave(&loc_l->lock, flags))
449 return NULL;
441 450
442 node = __local_list_pop_free(loc_l);451 node = __local_list_pop_free(loc_l);
443 if (!node) {452 if (!node) {
@@ -448,17 +457,22 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru, · 4 unmodified lines
448 if (node)457 if (node)
449 __local_list_add_pending(lru, loc_l, cpu, node, hash);458 __local_list_add_pending(lru, loc_l, cpu, node, hash);
450 459
451 raw_spin_unlock_irqrestore(&loc_l->lock, flags);460 raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);
452 461
453 if (node)462 if (node)
454 return node;463 return node;
455 464
456 /* No free nodes found from the local free list and465 /*
466 * No free nodes found from the local free list and
457 * the global LRU list.467 * the global LRU list.
458 *468 *
459 * Steal from the local free/pending list of the469 * Steal from the local free/pending list of the
460 * current CPU and remote CPU in RR. It starts470 * current CPU and remote CPU in RR. It starts
461 * with the loc_l->next_steal CPU.471 * with the loc_l->next_steal CPU.
472 *
473 * Acquire the victim's lock before touching either list. On
474 * acquisition failure (rqspinlock AA or timeout) skip the victim
475 * and try the next CPU.
462 */476 */
463 477
464 first_steal = loc_l->next_steal;478 first_steal = loc_l->next_steal;
@@ -466,24 +480,36 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru, · 1 unmodified line
466 do {480 do {
467 steal_loc_l = per_cpu_ptr(clru->local_list, steal);481 steal_loc_l = per_cpu_ptr(clru->local_list, steal);
468 482
469 raw_spin_lock_irqsave(&steal_loc_l->lock, flags);483 if (!raw_res_spin_lock_irqsave(&steal_loc_l->lock, flags)) {
470 484 node = __local_list_pop_free(steal_loc_l);
471 node = __local_list_pop_free(steal_loc_l);485 if (!node)
472 if (!node)486 node = __local_list_pop_pending(lru, steal_loc_l);
473 node = __local_list_pop_pending(lru, steal_loc_l);487 raw_res_spin_unlock_irqrestore(&steal_loc_l->lock, flags);
474 488 }
475 raw_spin_unlock_irqrestore(&steal_loc_l->lock, flags);
476 489
477 steal = cpumask_next_wrap(steal, cpu_possible_mask);490 steal = cpumask_next_wrap(steal, cpu_possible_mask);
478 } while (!node && steal != first_steal);491 } while (!node && steal != first_steal);
479 492
480 loc_l->next_steal = steal;493 loc_l->next_steal = steal;
481 494
482 if (node) {495 if (!node)
483 raw_spin_lock_irqsave(&loc_l->lock, flags);496 return NULL;
484 __local_list_add_pending(lru, loc_l, cpu, node, hash);497
485 raw_spin_unlock_irqrestore(&loc_l->lock, flags);498 if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) {
499 /*
500 * The local pending lock can't be acquired (rqspinlock AA
501 * or timeout). Return the stolen node to the per-CPU
502 * free_llist instead of orphaning it; the next pop_free on
503 * this CPU will pick it up.
504 */
505 node->type = BPF_LRU_LOCAL_LIST_T_FREE;
506 bpf_lru_node_clear_ref(node);
507 WRITE_ONCE(node->pending_free, 0);
508 llist_add(&node->llist, &loc_l->free_llist);
509 return NULL;
486 }510 }
511 __local_list_add_pending(lru, loc_l, cpu, node, hash);
512 raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);
487 513
488 return node;514 return node;
489}515}
@@ -511,18 +537,24 @@ static void bpf_common_lru_push_free(struct bpf_lru *lru, · 21 unmodified lines
511 537
512 loc_l = per_cpu_ptr(lru->common_lru.local_list, node->cpu);538 loc_l = per_cpu_ptr(lru->common_lru.local_list, node->cpu);
513 539
514 raw_spin_lock_irqsave(&loc_l->lock, flags);540 if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) {
541 WRITE_ONCE(node->pending_free, 1);
542 return;
543 }
515 544
516 if (unlikely(node->type != BPF_LRU_LOCAL_LIST_T_PENDING)) {545 if (unlikely(node->type != BPF_LRU_LOCAL_LIST_T_PENDING)) {
517 raw_spin_unlock_irqrestore(&loc_l->lock, flags);546 raw_res_spin_unlock_irqrestore(&loc_l->lock,
547 flags);
518 goto check_lru_list;548 goto check_lru_list;
519 }549 }
520 550
521 node->type = BPF_LRU_LOCAL_LIST_T_FREE;551 node->type = BPF_LRU_LOCAL_LIST_T_FREE;
522 bpf_lru_node_clear_ref(node);552 bpf_lru_node_clear_ref(node);
523 list_move(&node->list, local_free_list(loc_l));553 list_del(&node->list);
554
555 raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);
524 556
525 raw_spin_unlock_irqrestore(&loc_l->lock, flags);557 llist_add(&node->llist, &loc_l->free_llist);
526 return;558 return;
527 }559 }
528 560
@@ -538,11 +570,14 @@ static void bpf_percpu_lru_push_free(struct bpf_lru *lru, · 9 unmodified lines
538 570
539 l = per_cpu_ptr(lru->percpu_lru, node->cpu);571 l = per_cpu_ptr(lru->percpu_lru, node->cpu);
540 572
541 raw_spin_lock_irqsave(&l->lock, flags);573 if (raw_res_spin_lock_irqsave(&l->lock, flags)) {
574 WRITE_ONCE(node->pending_free, 1);
575 return;
576 }
542 577
543 __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE);578 __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE);
544 579
545 raw_spin_unlock_irqrestore(&l->lock, flags);580 raw_res_spin_unlock_irqrestore(&l->lock, flags);
546}581}
547 582
548void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node)583void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node)
@@ -565,6 +600,7 @@ static void bpf_common_lru_populate(struct bpf_lru *lru, void *buf, · 16 unmodified lines
565 600
566 node = (struct bpf_lru_node *)(buf + node_offset);601 node = (struct bpf_lru_node *)(buf + node_offset);
567 node->type = BPF_LRU_LIST_T_FREE;602 node->type = BPF_LRU_LIST_T_FREE;
603 node->pending_free = 0;
568 bpf_lru_node_clear_ref(node);604 bpf_lru_node_clear_ref(node);
569 list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]);605 list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]);
570 buf += elem_size;606 buf += elem_size;
@@ -594,6 +630,7 @@ again: · 23 unmodified lines
594 node = (struct bpf_lru_node *)(buf + node_offset);630 node = (struct bpf_lru_node *)(buf + node_offset);
595 node->cpu = cpu;631 node->cpu = cpu;
596 node->type = BPF_LRU_LIST_T_FREE;632 node->type = BPF_LRU_LIST_T_FREE;
633 node->pending_free = 0;
597 bpf_lru_node_clear_ref(node);634 bpf_lru_node_clear_ref(node);
598 list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]);635 list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]);
599 i++;636 i++;
@@ -618,14 +655,12 @@ void bpf_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset, · 18 unmodified lines
618 655
619static void bpf_lru_locallist_init(struct bpf_lru_locallist *loc_l, int cpu)656static void bpf_lru_locallist_init(struct bpf_lru_locallist *loc_l, int cpu)
620{657{
621 int i;658 INIT_LIST_HEAD(&loc_l->pending_list);
622 659 init_llist_head(&loc_l->free_llist);
623 for (i = 0; i < NR_BPF_LRU_LOCAL_LIST_T; i++)
624 INIT_LIST_HEAD(&loc_l->lists[i]);
625 660
626 loc_l->next_steal = cpu;661 loc_l->next_steal = cpu;
627 662
628 raw_spin_lock_init(&loc_l->lock);663 raw_res_spin_lock_init(&loc_l->lock);
629}664}
630 665
631static void bpf_lru_list_init(struct bpf_lru_list *l)666static void bpf_lru_list_init(struct bpf_lru_list *l)
@@ -640,7 +675,7 @@ static void bpf_lru_list_init(struct bpf_lru_list *l) · 8 unmodified lines
640 675
641 l->next_inactive_rotation = &l->lists[BPF_LRU_LIST_T_INACTIVE];676 l->next_inactive_rotation = &l->lists[BPF_LRU_LIST_T_INACTIVE];
642 677
643 raw_spin_lock_init(&l->lock);678 raw_res_spin_lock_init(&l->lock);
644}679}
645 680
646int bpf_lru_init(struct bpf_lru *lru, bool percpu, u32 hash_offset,681int bpf_lru_init(struct bpf_lru *lru, bool percpu, u32 hash_offset,
kernel/bpf/bpf_lru_list.h+19−6
@@ -6,11 +6,11 @@ · 5 unmodified lines
6 6
7#include <linux/cache.h>7#include <linux/cache.h>
8#include <linux/list.h>8#include <linux/list.h>
9#include <linux/spinlock_types.h>9#include <linux/llist.h>
10#include <asm/rqspinlock.h>
10 11
11#define NR_BPF_LRU_LIST_T (3)12#define NR_BPF_LRU_LIST_T (3)
12#define NR_BPF_LRU_LIST_COUNT (2)13#define NR_BPF_LRU_LIST_COUNT (2)
13#define NR_BPF_LRU_LOCAL_LIST_T (2)
14#define BPF_LOCAL_LIST_T_OFFSET NR_BPF_LRU_LIST_T14#define BPF_LOCAL_LIST_T_OFFSET NR_BPF_LRU_LIST_T
15 15
16enum bpf_lru_list_type {16enum bpf_lru_list_type {
@@ -22,10 +22,22 @@ enum bpf_lru_list_type { · 5 unmodified lines
22};22};
23 23
24struct bpf_lru_node {24struct bpf_lru_node {
25 struct list_head list;25 /*
26 * A node is in at most one list at a time. The free path on the
27 * per-CPU locallist uses an llist, so share storage via a union.
28 */
29 union {
30 struct list_head list;
31 struct llist_node llist;
32 };
26 u16 cpu;33 u16 cpu;
27 u8 type;34 u8 type;
28 u8 ref;35 u8 ref;
36 /*
37 * Marks nodes whose *_push_free() lock acquire failed; reclaimed
38 * by flush/shrink which honor the flag instead of del_from_htab().
39 */
40 u8 pending_free;
29};41};
30 42
31struct bpf_lru_list {43struct bpf_lru_list {
@@ -34,13 +46,14 @@ struct bpf_lru_list { · 2 unmodified lines
34 /* The next inactive list rotation starts from here */46 /* The next inactive list rotation starts from here */
35 struct list_head *next_inactive_rotation;47 struct list_head *next_inactive_rotation;
36 48
37 raw_spinlock_t lock ____cacheline_aligned_in_smp;49 rqspinlock_t lock ____cacheline_aligned_in_smp;
38};50};
39 51
40struct bpf_lru_locallist {52struct bpf_lru_locallist {
41 struct list_head lists[NR_BPF_LRU_LOCAL_LIST_T];53 struct list_head pending_list;
54 struct llist_head free_llist;
42 u16 next_steal;55 u16 next_steal;
43 raw_spinlock_t lock;56 rqspinlock_t lock;
44};57};
45 58
46struct bpf_common_lru {59struct bpf_common_lru {
kernel/bpf/bpf_lsm.c+20−0
@@ -427,6 +427,26 @@ BTF_ID(func, bpf_lsm_audit_rule_known) · 426 unmodified lines
427BTF_ID(func, bpf_lsm_inode_xattr_skipcap)427BTF_ID(func, bpf_lsm_inode_xattr_skipcap)
428BTF_SET_END(bool_lsm_hooks)428BTF_SET_END(bool_lsm_hooks)
429 429
430/* hooks returning void */
431#define LSM_HOOK_void(DEFAULT, NAME, ...) BTF_ID(func, bpf_lsm_##NAME)
432#define LSM_HOOK_int(DEFAULT, NAME, ...) /* nothing */
433#define LSM_HOOK(RET, DEFAULT, NAME, ...) LSM_HOOK_##RET(DEFAULT, NAME, __VA_ARGS__)
434BTF_SET_START(void_lsm_hooks)
435#include <linux/lsm_hook_defs.h>
436#undef LSM_HOOK
437#undef LSM_HOOK_void
438#undef LSM_HOOK_int
439BTF_SET_END(void_lsm_hooks)
440
441bool bpf_lsm_hook_returns_errno(u32 btf_id)
442{
443 if (btf_id_set_contains(&bool_lsm_hooks, btf_id))
444 return false;
445 if (btf_id_set_contains(&void_lsm_hooks, btf_id))
446 return false;
447 return true;
448}
449
430int bpf_lsm_get_retval_range(const struct bpf_prog *prog,450int bpf_lsm_get_retval_range(const struct bpf_prog *prog,
431 struct bpf_retval_range *retval_range)451 struct bpf_retval_range *retval_range)
432{452{
kernel/bpf/bpf_struct_ops.c+50−13
@@ -594,8 +594,8 @@ const struct bpf_link_ops bpf_struct_ops_link_lops = { · 593 unmodified lines
594 .dealloc = bpf_struct_ops_link_dealloc,594 .dealloc = bpf_struct_ops_link_dealloc,
595};595};
596 596
597int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,597int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes,
598 struct bpf_tramp_link *link,598 struct bpf_tramp_node *node,
599 const struct btf_func_model *model,599 const struct btf_func_model *model,
600 void *stub_func,600 void *stub_func,
601 void **_image, u32 *_image_off,601 void **_image, u32 *_image_off,
@@ -605,13 +605,13 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, · 3 unmodified lines
605 void *image = *_image;605 void *image = *_image;
606 int size;606 int size;
607 607
608 tlinks[BPF_TRAMP_FENTRY].links[0] = link;608 tnodes[BPF_TRAMP_FENTRY].nodes[0] = node;
609 tlinks[BPF_TRAMP_FENTRY].nr_links = 1;609 tnodes[BPF_TRAMP_FENTRY].nr_nodes = 1;
610 610
611 if (model->ret_size > 0)611 if (model->ret_size > 0)
612 flags |= BPF_TRAMP_F_RET_FENTRY_RET;612 flags |= BPF_TRAMP_F_RET_FENTRY_RET;
613 613
614 size = arch_bpf_trampoline_size(model, flags, tlinks, stub_func);614 size = arch_bpf_trampoline_size(model, flags, tnodes, stub_func);
615 if (size <= 0)615 if (size <= 0)
616 return size ? : -EFAULT;616 return size ? : -EFAULT;
617 617
@@ -628,7 +628,7 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, · 10 unmodified lines
628 628
629 size = arch_prepare_bpf_trampoline(NULL, image + image_off,629 size = arch_prepare_bpf_trampoline(NULL, image + image_off,
630 image + image_off + size,630 image + image_off + size,
631 model, flags, tlinks, stub_func);631 model, flags, tnodes, stub_func);
632 if (size <= 0) {632 if (size <= 0) {
633 if (image != *_image)633 if (image != *_image)
634 bpf_struct_ops_image_free(image);634 bpf_struct_ops_image_free(image);
@@ -693,7 +693,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 58 unmodified lines
693 const struct btf_type *module_type;693 const struct btf_type *module_type;
694 const struct btf_member *member;694 const struct btf_member *member;
695 const struct btf_type *t = st_ops_desc->type;695 const struct btf_type *t = st_ops_desc->type;
696 struct bpf_tramp_links *tlinks;696 struct bpf_tramp_nodes *tnodes;
697 void *udata, *kdata;697 void *udata, *kdata;
698 int prog_fd, err;698 int prog_fd, err;
699 u32 i, trampoline_start, image_off = 0;699 u32 i, trampoline_start, image_off = 0;
@@ -720,8 +720,8 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 20 unmodified lines
720 if (uvalue->common.state || refcount_read(&uvalue->common.refcnt))720 if (uvalue->common.state || refcount_read(&uvalue->common.refcnt))
721 return -EINVAL;721 return -EINVAL;
722 722
723 tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX);723 tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX);
724 if (!tlinks)724 if (!tnodes)
725 return -ENOMEM;725 return -ENOMEM;
726 726
727 uvalue = (struct bpf_struct_ops_value *)st_map->uvalue;727 uvalue = (struct bpf_struct_ops_value *)st_map->uvalue;
@@ -817,8 +817,9 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 89 unmodified lines
817 err = -ENOMEM;817 err = -ENOMEM;
818 goto reset_unlock;818 goto reset_unlock;
819 }819 }
820 bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS,820 bpf_tramp_link_init(link, BPF_LINK_TYPE_STRUCT_OPS,
821 &bpf_struct_ops_link_lops, prog, prog->expected_attach_type);821 &bpf_struct_ops_link_lops, prog, prog->expected_attach_type, 0);
822
822 *plink++ = &link->link;823 *plink++ = &link->link;
823 824
824 /* Poison pointer on error instead of return for backward compatibility */825 /* Poison pointer on error instead of return for backward compatibility */
@@ -832,7 +833,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 7 unmodified lines
832 *pksym++ = ksym;833 *pksym++ = ksym;
833 834
834 trampoline_start = image_off;835 trampoline_start = image_off;
835 err = bpf_struct_ops_prepare_trampoline(tlinks, link,836 err = bpf_struct_ops_prepare_trampoline(tnodes, &link->node,
836 &st_ops->func_models[i],837 &st_ops->func_models[i],
837 *(void **)(st_ops->cfi_stubs + moff),838 *(void **)(st_ops->cfi_stubs + moff),
838 &image, &image_off,839 &image, &image_off,
@@ -911,7 +912,7 @@ reset_unlock: · 72 unmodified lines
911 memset(uvalue, 0, map->value_size);912 memset(uvalue, 0, map->value_size);
912 memset(kvalue, 0, map->value_size);913 memset(kvalue, 0, map->value_size);
913unlock:914unlock:
914 kfree(tlinks);915 kfree(tnodes);
915 mutex_unlock(&st_map->lock);916 mutex_unlock(&st_map->lock);
916 if (!err)917 if (!err)
917 bpf_struct_ops_map_add_ksyms(st_map);918 bpf_struct_ops_map_add_ksyms(st_map);
@@ -1204,6 +1205,42 @@ u32 bpf_struct_ops_id(const void *kdata) · 286 unmodified lines
1204}1205}
1205EXPORT_SYMBOL_GPL(bpf_struct_ops_id);1206EXPORT_SYMBOL_GPL(bpf_struct_ops_id);
1206 1207
1208/**
1209 * bpf_struct_ops_for_each_prog - Invoke @cb for each member prog
1210 * @kdata: kernel-side struct_ops vmtable (the @kdata arg to ->reg/->update/->unreg)
1211 * @cb: callback invoked once per member prog; non-zero return stops iteration
1212 * @data: opaque argument passed to @cb
1213 *
1214 * Walks the struct_ops member progs registered on the map containing @kdata.
1215 * Intended for use from struct_ops ->reg() callbacks (and similar) that need to
1216 * inspect the loaded BPF programs (for example to discover maps they reference
1217 * via @prog->aux->used_maps).
1218 *
1219 * Return 0 if iteration completed, otherwise the first non-zero @cb return.
1220 */
1221int bpf_struct_ops_for_each_prog(const void *kdata,
1222 int (*cb)(struct bpf_prog *prog, void *data),
1223 void *data)
1224{
1225 struct bpf_struct_ops_value *kvalue;
1226 struct bpf_struct_ops_map *st_map;
1227 u32 i;
1228 int ret;
1229
1230 kvalue = container_of(kdata, struct bpf_struct_ops_value, data);
1231 st_map = container_of(kvalue, struct bpf_struct_ops_map, kvalue);
1232
1233 for (i = 0; i < st_map->funcs_cnt; i++) {
1234 if (!st_map->links[i])
1235 continue;
1236 ret = cb(st_map->links[i]->prog, data);
1237 if (ret)
1238 return ret;
1239 }
1240 return 0;
1241}
1242EXPORT_SYMBOL_GPL(bpf_struct_ops_for_each_prog);
1243
1207static bool bpf_struct_ops_valid_to_reg(struct bpf_map *map)1244static bool bpf_struct_ops_valid_to_reg(struct bpf_map *map)
1208{1245{
1209 struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;1246 struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;
kernel/bpf/btf.c+219−99
@@ -182,7 +182,6 @@ · 181 unmodified lines
182#define BITS_ROUNDUP_BYTES(bits) \182#define BITS_ROUNDUP_BYTES(bits) \
183 (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))183 (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))
184 184
185#define BTF_INFO_MASK 0x9f00ffff
186#define BTF_INT_MASK 0x0fffffff185#define BTF_INT_MASK 0x0fffffff
187#define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE)186#define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE)
188#define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET)187#define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET)
@@ -289,7 +288,7 @@ enum verifier_phase { · 100 unmodified lines
289struct resolve_vertex {288struct resolve_vertex {
290 const struct btf_type *t;289 const struct btf_type *t;
291 u32 type_id;290 u32 type_id;
292 u16 next_member;291 u32 next_member;
293};292};
294 293
295enum visit_state {294enum visit_state {
@@ -2031,7 +2030,7 @@ static int env_stack_push(struct btf_verifier_env *env, · 1735 unmodified lines
2031}2030}
2032 2031
2033static void env_stack_set_next_member(struct btf_verifier_env *env,2032static void env_stack_set_next_member(struct btf_verifier_env *env,
2034 u16 next_member)2033 u32 next_member)
2035{2034{
2036 env->stack[env->top_stack - 1].next_member = next_member;2035 env->stack[env->top_stack - 1].next_member = next_member;
2037}2036}
@@ -3293,7 +3292,7 @@ static s32 btf_struct_check_meta(struct btf_verifier_env *env, · 1255 unmodified lines
3293 struct btf *btf = env->btf;3292 struct btf *btf = env->btf;
3294 u32 struct_size = t->size;3293 u32 struct_size = t->size;
3295 u32 offset;3294 u32 offset;
3296 u16 i;3295 u32 i;
3297 3296
3298 meta_needed = btf_type_vlen(t) * sizeof(*member);3297 meta_needed = btf_type_vlen(t) * sizeof(*member);
3299 if (meta_left < meta_needed) {3298 if (meta_left < meta_needed) {
@@ -3369,7 +3368,7 @@ static int btf_struct_resolve(struct btf_verifier_env *env, · 69 unmodified lines
3369{3368{
3370 const struct btf_member *member;3369 const struct btf_member *member;
3371 int err;3370 int err;
3372 u16 i;3371 u32 i;
3373 3372
3374 /* Before continue resolving the next_member,3373 /* Before continue resolving the next_member,
3375 * ensure the last member is indeed resolved to a3374 * ensure the last member is indeed resolved to a
@@ -3668,7 +3667,7 @@ end: · 292 unmodified lines
3668static int btf_repeat_fields(struct btf_field_info *info, int info_cnt,3667static int btf_repeat_fields(struct btf_field_info *info, int info_cnt,
3669 u32 field_cnt, u32 repeat_cnt, u32 elem_size)3668 u32 field_cnt, u32 repeat_cnt, u32 elem_size)
3670{3669{
3671 u32 i, j;3670 u32 i, j, total_cnt, total_repeats;
3672 u32 cur;3671 u32 cur;
3673 3672
3674 /* Ensure not repeating fields that should not be repeated. */3673 /* Ensure not repeating fields that should not be repeated. */
@@ -3686,10 +3685,9 @@ static int btf_repeat_fields(struct btf_field_info *info, int info_cnt, · 11 unmodified lines
3686 }3685 }
3687 }3686 }
3688 3687
3689 /* The type of struct size or variable size is u32,3688 if (check_add_overflow(repeat_cnt, 1, &total_repeats) ||
3690 * so the multiplication will not overflow.3689 check_mul_overflow(field_cnt, total_repeats, &total_cnt) ||
3691 */3690 total_cnt > (u32)info_cnt)
3692 if (field_cnt * (repeat_cnt + 1) > info_cnt)
3693 return -E2BIG;3691 return -E2BIG;
3694 3692
3695 cur = field_cnt;3693 cur = field_cnt;
@@ -4447,7 +4445,7 @@ static s32 btf_enum_check_meta(struct btf_verifier_env *env, · 751 unmodified lines
4447 const struct btf_enum *enums = btf_type_enum(t);4445 const struct btf_enum *enums = btf_type_enum(t);
4448 struct btf *btf = env->btf;4446 struct btf *btf = env->btf;
4449 const char *fmt_str;4447 const char *fmt_str;
4450 u16 i, nr_enums;4448 u32 i, nr_enums;
4451 u32 meta_needed;4449 u32 meta_needed;
4452 4450
4453 nr_enums = btf_type_vlen(t);4451 nr_enums = btf_type_vlen(t);
@@ -4555,7 +4553,7 @@ static s32 btf_enum64_check_meta(struct btf_verifier_env *env, · 101 unmodified lines
4555 const struct btf_enum64 *enums = btf_type_enum64(t);4553 const struct btf_enum64 *enums = btf_type_enum64(t);
4556 struct btf *btf = env->btf;4554 struct btf *btf = env->btf;
4557 const char *fmt_str;4555 const char *fmt_str;
4558 u16 i, nr_enums;4556 u32 i, nr_enums;
4559 u32 meta_needed;4557 u32 meta_needed;
4560 4558
4561 nr_enums = btf_type_vlen(t);4559 nr_enums = btf_type_vlen(t);
@@ -4683,7 +4681,7 @@ static void btf_func_proto_log(struct btf_verifier_env *env, · 121 unmodified lines
4683 const struct btf_type *t)4681 const struct btf_type *t)
4684{4682{
4685 const struct btf_param *args = (const struct btf_param *)(t + 1);4683 const struct btf_param *args = (const struct btf_param *)(t + 1);
4686 u16 nr_args = btf_type_vlen(t), i;4684 u32 nr_args = btf_type_vlen(t), i;
4687 4685
4688 btf_verifier_log(env, "return=%u args=(", t->type);4686 btf_verifier_log(env, "return=%u args=(", t->type);
4689 if (!nr_args) {4687 if (!nr_args) {
@@ -4929,7 +4927,7 @@ static int btf_datasec_resolve(struct btf_verifier_env *env, · 239 unmodified lines
4929{4927{
4930 const struct btf_var_secinfo *vsi;4928 const struct btf_var_secinfo *vsi;
4931 struct btf *btf = env->btf;4929 struct btf *btf = env->btf;
4932 u16 i;4930 u32 i;
4933 4931
4934 env->resolve_mode = RESOLVE_TBD;4932 env->resolve_mode = RESOLVE_TBD;
4935 for_each_vsi_from(i, v->next_member, v->t, vsi) {4933 for_each_vsi_from(i, v->next_member, v->t, vsi) {
@@ -5183,7 +5181,7 @@ static int btf_func_proto_check(struct btf_verifier_env *env, · 247 unmodified lines
5183 const struct btf_type *ret_type;5181 const struct btf_type *ret_type;
5184 const struct btf_param *args;5182 const struct btf_param *args;
5185 const struct btf *btf;5183 const struct btf *btf;
5186 u16 nr_args, i;5184 u32 nr_args, i;
5187 int err;5185 int err;
5188 5186
5189 btf = env->btf;5187 btf = env->btf;
@@ -5278,7 +5276,7 @@ static int btf_func_check(struct btf_verifier_env *env, · 88 unmodified lines
5278 const struct btf_type *proto_type;5276 const struct btf_type *proto_type;
5279 const struct btf_param *args;5277 const struct btf_param *args;
5280 const struct btf *btf;5278 const struct btf *btf;
5281 u16 nr_args, i;5279 u32 nr_args, i;
5282 5280
5283 btf = env->btf;5281 btf = env->btf;
5284 proto_type = btf_type_by_id(btf, t->type);5282 proto_type = btf_type_by_id(btf, t->type);
@@ -5336,12 +5334,6 @@ static s32 btf_check_meta(struct btf_verifier_env *env, · 51 unmodified lines
5336 }5334 }
5337 meta_left -= sizeof(*t);5335 meta_left -= sizeof(*t);
5338 5336
5339 if (t->info & ~BTF_INFO_MASK) {
5340 btf_verifier_log(env, "[%u] Invalid btf_info:%x",
5341 env->log_type_id, t->info);
5342 return -EINVAL;
5343 }
5344
5345 if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX ||5337 if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX ||
5346 BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) {5338 BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) {
5347 btf_verifier_log(env, "[%u] Invalid kind:%u",5339 btf_verifier_log(env, "[%u] Invalid kind:%u",
@@ -5914,25 +5906,10 @@ static int btf_check_type_tags(struct btf_verifier_env *env, · 566 unmodified lines
5914 return 0;5906 return 0;
5915}5907}
5916 5908
5917static int finalize_log(struct bpf_verifier_log *log, bpfptr_t uattr, u32 uattr_size)5909static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr,
5918{5910 struct bpf_log_attr *attr_log)
5919 u32 log_true_size;
5920 int err;
5921
5922 err = bpf_vlog_finalize(log, &log_true_size);
5923
5924 if (uattr_size >= offsetofend(union bpf_attr, btf_log_true_size) &&
5925 copy_to_bpfptr_offset(uattr, offsetof(union bpf_attr, btf_log_true_size),
5926 &log_true_size, sizeof(log_true_size)))
5927 err = -EFAULT;
5928
5929 return err;
5930}
5931
5932static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
5933{5911{
5934 bpfptr_t btf_data = make_bpfptr(attr->btf, uattr.is_kernel);5912 bpfptr_t btf_data = make_bpfptr(attr->btf, uattr.is_kernel);
5935 char __user *log_ubuf = u64_to_user_ptr(attr->btf_log_buf);
5936 struct btf_struct_metas *struct_meta_tab;5913 struct btf_struct_metas *struct_meta_tab;
5937 struct btf_verifier_env *env = NULL;5914 struct btf_verifier_env *env = NULL;
5938 struct btf *btf = NULL;5915 struct btf *btf = NULL;
@@ -5949,8 +5926,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat · 10 unmodified lines
5949 /* user could have requested verbose verifier output5926 /* user could have requested verbose verifier output
5950 * and supplied buffer to store the verification trace5927 * and supplied buffer to store the verification trace
5951 */5928 */
5952 err = bpf_vlog_init(&env->log, attr->btf_log_level,5929 err = bpf_vlog_init(&env->log, attr_log->level, attr_log->ubuf, attr_log->size);
5953 log_ubuf, attr->btf_log_size);
5954 if (err)5930 if (err)
5955 goto errout_free;5931 goto errout_free;
5956 5932
@@ -6015,7 +5991,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat · 58 unmodified lines
6015 }5991 }
6016 }5992 }
6017 5993
6018 err = finalize_log(&env->log, uattr, uattr_size);5994 err = bpf_log_attr_finalize(attr_log, &env->log);
6019 if (err)5995 if (err)
6020 goto errout_free;5996 goto errout_free;
6021 5997
@@ -6027,7 +6003,7 @@ errout_meta: · 5 unmodified lines
6027 btf_free_struct_meta_tab(btf);6003 btf_free_struct_meta_tab(btf);
6028errout:6004errout:
6029 /* overwrite err with -ENOSPC or -EFAULT */6005 /* overwrite err with -ENOSPC or -EFAULT */
6030 ret = finalize_log(&env->log, uattr, uattr_size);6006 ret = bpf_log_attr_finalize(attr_log, &env->log);
6031 if (ret)6007 if (ret)
6032 err = ret;6008 err = ret;
6033errout_free:6009errout_free:
@@ -6980,7 +6956,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 946 unmodified lines
6980 info->reg_type = ctx_arg_info->reg_type;6956 info->reg_type = ctx_arg_info->reg_type;
6981 info->btf = ctx_arg_info->btf ? : btf_vmlinux;6957 info->btf = ctx_arg_info->btf ? : btf_vmlinux;
6982 info->btf_id = ctx_arg_info->btf_id;6958 info->btf_id = ctx_arg_info->btf_id;
6983 info->ref_obj_id = ctx_arg_info->ref_obj_id;6959 info->ref_id = ctx_arg_info->ref_id;
6984 return true;6960 return true;
6985 }6961 }
6986 }6962 }
@@ -7825,6 +7801,134 @@ enum btf_arg_tag { · 838 unmodified lines
7825 ARG_TAG_ARENA = BIT_ULL(5),7801 ARG_TAG_ARENA = BIT_ULL(5),
7826};7802};
7827 7803
7804static int btf_scan_decl_tags(struct bpf_verifier_env *env,
7805 const struct btf *btf,
7806 const struct btf_type *fn_t,
7807 u32 arg_idx, bool is_global, u32 *tags)
7808{
7809 int id = btf_named_start_id(btf, false) - 1;
7810 const char tag_key[] = "arg:";
7811 static const struct {
7812 const char *tag_value;
7813 enum btf_arg_tag arg_tag;
7814 } tag_values[] = {
7815 { "ctx", ARG_TAG_CTX },
7816 { "trusted", ARG_TAG_TRUSTED },
7817 { "untrusted", ARG_TAG_UNTRUSTED },
7818 { "nonnull", ARG_TAG_NONNULL },
7819 { "nullable", ARG_TAG_NULLABLE },
7820 { "arena", ARG_TAG_ARENA },
7821 };
7822
7823 /*
7824 * The 'arg:<tag>' decl_tag takes precedence over the derivation
7825 * of the register type from the BTF type itself.
7826 */
7827 while ((id = btf_find_next_decl_tag(btf, fn_t, arg_idx, tag_key, id)) > 0) {
7828 const struct btf_type *tag_t;
7829 const char *tag;
7830 int i;
7831 bool found;
7832
7833 /* disallow arg tags in static subprogs */
7834 if (!is_global) {
7835 bpf_log(&env->log,
7836 "arg#%d type tag is not supported in static functions\n",
7837 arg_idx);
7838 return -EOPNOTSUPP;
7839 }
7840
7841 tag_t = btf_type_by_id(btf, id);
7842 tag = __btf_name_by_offset(btf, tag_t->name_off) + (sizeof(tag_key) - 1);
7843
7844 found = false;
7845 for (i = 0; i < ARRAY_SIZE(tag_values); ++i) {
7846 if (!strcmp(tag, tag_values[i].tag_value)) {
7847 *tags |= tag_values[i].arg_tag;
7848 found = true;
7849 break;
7850 }
7851 }
7852
7853 if (!found) {
7854 bpf_log(&env->log, "arg#%d has unsupported set of tags\n", arg_idx);
7855 return -EOPNOTSUPP;
7856 }
7857 }
7858 if (id != -ENOENT) {
7859 bpf_log(&env->log, "arg#%d type tag fetching failure: %d\n", arg_idx, id);
7860 return id;
7861 }
7862
7863 return 0;
7864}
7865
7866static int btf_scan_type_tags(struct bpf_verifier_env *env,
7867 const struct btf *btf, u32 type_id,
7868 u32 *tags)
7869{
7870 const struct btf_type *t;
7871
7872 /* Find the first pointer type in the chain. */
7873 t = btf_type_skip_modifiers(btf, type_id, NULL);
7874
7875 /*
7876 * We currently reject type tags on non-pointer types,
7877 * which neither LLVM nor GCC support anyway.
7878 */
7879 if (!t || !btf_type_is_ptr(t))
7880 return 0;
7881
7882 /* We got a pointer, get all associated type tags. */
7883 for (t = btf_type_by_id(btf, t->type); t && btf_type_is_modifier(t);
7884 t = btf_type_by_id(btf, t->type)) {
7885
7886 /* Skip non-type tag modifiers. */
7887 if (!btf_type_is_type_tag(t))
7888 continue;
7889
7890 const char *tag = __btf_name_by_offset(btf, t->name_off);
7891
7892 if (strcmp(tag, "arena") == 0) {
7893 *tags |= ARG_TAG_ARENA;
7894 } else {
7895 bpf_log(&env->log, "function signature member has unsupported type tag '%s'\n",
7896 tag);
7897 return -EOPNOTSUPP;
7898 }
7899 }
7900
7901 return 0;
7902}
7903
7904/* Check whether the type is a valid return type. */
7905static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,
7906 const struct btf_type *t, int subprog)
7907{
7908 u32 tags = 0;
7909 int err;
7910
7911 err = btf_scan_type_tags(env, btf, t->type, &tags);
7912 if (err)
7913 return err;
7914
7915 t = btf_type_skip_modifiers(btf, t->type, NULL);
7916
7917 /*
7918 * We allow all subprogs except for the main one to return any kind of arena pointer.
7919 * General arena variables are not allowed, since it makes no sense to return by value
7920 * a variable that's on the heap in the first place.
7921 */
7922 if (subprog && (tags & ARG_TAG_ARENA) && btf_type_is_ptr(t))
7923 return 0;
7924
7925 /* We always accept void or scalars. */
7926 if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
7927 return 0;
7928
7929 return -EOPNOTSUPP;
7930}
7931
7828/* Process BTF of a function to produce high-level expectation of function7932/* Process BTF of a function to produce high-level expectation of function
7829 * arguments (like ARG_PTR_TO_CTX, or ARG_PTR_TO_MEM, etc). This information7933 * arguments (like ARG_PTR_TO_CTX, or ARG_PTR_TO_MEM, etc). This information
7830 * is cached in subprog info for reuse.7934 * is cached in subprog info for reuse.
@@ -7843,6 +7947,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 12 unmodified lines
7843 struct btf *btf = prog->aux->btf;7947 struct btf *btf = prog->aux->btf;
7844 const struct btf_param *args;7948 const struct btf_param *args;
7845 const struct btf_type *t, *ref_t, *fn_t;7949 const struct btf_type *t, *ref_t, *fn_t;
7950 int err;
7846 u32 i, nargs, btf_id;7951 u32 i, nargs, btf_id;
7847 const char *tname;7952 const char *tname;
7848 7953
@@ -7887,25 +7992,36 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 38 unmodified lines
7887 }7992 }
7888 args = (const struct btf_param *)(t + 1);7993 args = (const struct btf_param *)(t + 1);
7889 nargs = btf_type_vlen(t);7994 nargs = btf_type_vlen(t);
7995 sub->arg_cnt = nargs;
7996 if (nargs > MAX_BPF_FUNC_ARGS) {
7997 bpf_log(log, "kernel supports at most %d parameters, function %s has %d\n",
7998 MAX_BPF_FUNC_ARGS, tname, nargs);
7999 return -EFAULT;
8000 }
7890 if (nargs > MAX_BPF_FUNC_REG_ARGS) {8001 if (nargs > MAX_BPF_FUNC_REG_ARGS) {
7891 if (!is_global)8002 if (!bpf_jit_supports_stack_args()) {
7892 return -EINVAL;8003 bpf_log(log, "JIT does not support function %s() with %d args\n",
7893 bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",8004 tname, nargs);
8005 return -EFAULT;
8006 }
8007 sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
8008 }
8009
8010 if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
8011 bpf_log(log, "global function %s has %d > %d args, stack args not supported\n",
7894 tname, nargs, MAX_BPF_FUNC_REG_ARGS);8012 tname, nargs, MAX_BPF_FUNC_REG_ARGS);
7895 return -EINVAL;8013 return -EINVAL;
7896 }8014 }
7897 /* check that function is void or returns int, exception cb also requires this */8015
7898 t = btf_type_by_id(btf, t->type);8016 err = btf_validate_return_type(env, btf, t, subprog);
7899 while (btf_type_is_modifier(t))8017 if (err) {
7900 t = btf_type_by_id(btf, t->type);8018 if (is_global) {
7901 if (!btf_type_is_void(t) && !btf_type_is_int(t) && !btf_is_any_enum(t)) {8019 bpf_log(log,
7902 if (!is_global)8020 "Global function %s() return value not void or scalar. "
7903 return -EINVAL;8021 "Only those are supported.\n",
7904 bpf_log(log,8022 tname);
7905 "Global function %s() return value not void or scalar. "8023 }
7906 "Only those are supported.\n",8024 return err;
7907 tname);
7908 return -EINVAL;
7909 }8025 }
7910 8026
7911 /* Convert BTF function arguments into verifier types.8027 /* Convert BTF function arguments into verifier types.
@@ -7913,42 +8029,13 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 1 unmodified line
7913 */8029 */
7914 for (i = 0; i < nargs; i++) {8030 for (i = 0; i < nargs; i++) {
7915 u32 tags = 0;8031 u32 tags = 0;
7916 int id = btf_named_start_id(btf, false) - 1;8032 err = btf_scan_decl_tags(env, btf, fn_t, i, is_global, &tags);
7917 8033 if (err)
7918 /* 'arg:<tag>' decl_tag takes precedence over derivation of8034 return err;
7919 * register type from BTF type itself
7920 */
7921 while ((id = btf_find_next_decl_tag(btf, fn_t, i, "arg:", id)) > 0) {
7922 const struct btf_type *tag_t = btf_type_by_id(btf, id);
7923 const char *tag = __btf_name_by_offset(btf, tag_t->name_off) + 4;
7924
7925 /* disallow arg tags in static subprogs */
7926 if (!is_global) {
7927 bpf_log(log, "arg#%d type tag is not supported in static functions\n", i);
7928 return -EOPNOTSUPP;
7929 }
7930 8035
7931 if (strcmp(tag, "ctx") == 0) {8036 err = btf_scan_type_tags(env, btf, args[i].type, &tags);
7932 tags |= ARG_TAG_CTX;8037 if (err)
7933 } else if (strcmp(tag, "trusted") == 0) {8038 return err;
7934 tags |= ARG_TAG_TRUSTED;
7935 } else if (strcmp(tag, "untrusted") == 0) {
7936 tags |= ARG_TAG_UNTRUSTED;
7937 } else if (strcmp(tag, "nonnull") == 0) {
7938 tags |= ARG_TAG_NONNULL;
7939 } else if (strcmp(tag, "nullable") == 0) {
7940 tags |= ARG_TAG_NULLABLE;
7941 } else if (strcmp(tag, "arena") == 0) {
7942 tags |= ARG_TAG_ARENA;
7943 } else {
7944 bpf_log(log, "arg#%d has unsupported set of tags\n", i);
7945 return -EOPNOTSUPP;
7946 }
7947 }
7948 if (id != -ENOENT) {
7949 bpf_log(log, "arg#%d type tag fetching failure: %d\n", i, id);
7950 return id;
7951 }
7952 8039
7953 t = btf_type_by_id(btf, args[i].type);8040 t = btf_type_by_id(btf, args[i].type);
7954 while (btf_type_is_modifier(t))8041 while (btf_type_is_modifier(t))
@@ -7973,7 +8060,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 18 unmodified lines
7973 bpf_log(log, "arg#%d has invalid combination of tags\n", i);8060 bpf_log(log, "arg#%d has invalid combination of tags\n", i);
7974 return -EINVAL;8061 return -EINVAL;
7975 }8062 }
7976 sub->args[i].arg_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY;8063 sub->args[i].arg_type = ARG_PTR_TO_DYNPTR;
7977 continue;8064 continue;
7978 }8065 }
7979 if (tags & ARG_TAG_TRUSTED) {8066 if (tags & ARG_TAG_TRUSTED) {
@@ -8074,7 +8161,6 @@ skip_pointer: · 94 unmodified lines
8074 return -EINVAL;8161 return -EINVAL;
8075 }8162 }
8076 8163
8077 sub->arg_cnt = nargs;
8078 sub->args_cached = true;8164 sub->args_cached = true;
8079 8165
8080 return 0;8166 return 0;
@@ -8196,12 +8282,12 @@ static int __btf_new_fd(struct btf *btf) · 115 unmodified lines
8196 return anon_inode_getfd("btf", &btf_fops, btf, O_RDONLY | O_CLOEXEC);8282 return anon_inode_getfd("btf", &btf_fops, btf, O_RDONLY | O_CLOEXEC);
8197}8283}
8198 8284
8199int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)8285int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
8200{8286{
8201 struct btf *btf;8287 struct btf *btf;
8202 int ret;8288 int ret;
8203 8289
8204 btf = btf_parse(attr, uattr, uattr_size);8290 btf = btf_parse(attr, uattr, attr_log);
8205 if (IS_ERR(btf))8291 if (IS_ERR(btf))
8206 return PTR_ERR(btf);8292 return PTR_ERR(btf);
8207 8293
@@ -8684,6 +8770,39 @@ static int btf_check_iter_kfuncs(struct btf *btf, const char *func_name, · 476 unmodified lines
8684 return 0;8770 return 0;
8685}8771}
8686 8772
8773static int btf_check_kfunc_name(struct btf *btf, const char *func_name, u32 kind)
8774{
8775#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
8776 struct btf_module *btf_mod, *tmp;
8777#endif
8778 s32 id;
8779
8780 if (!btf_is_module(btf))
8781 return 0;
8782
8783 id = btf_find_by_name_kind(bpf_get_btf_vmlinux(), func_name, kind);
8784 if (id >= 0) {
8785 pr_err("kfunc %s (id: %d) is already present in vmlinux.\n",
8786 func_name, id);
8787 return -EINVAL;
8788 }
8789
8790#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
8791 guard(mutex)(&btf_module_mutex);
8792 list_for_each_entry_safe(btf_mod, tmp, &btf_modules, list) {
8793 if (btf_mod->btf == btf)
8794 continue;
8795 id = btf_find_by_name_kind(btf_mod->btf, func_name, kind);
8796 if (id >= 0) {
8797 pr_err("kfunc %s (id: %d) is already present in module %s.\n",
8798 func_name, id, btf_mod->module->name);
8799 return -EINVAL;
8800 }
8801 }
8802#endif
8803 return 0;
8804}
8805
8687static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags)8806static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags)
8688{8807{
8689 const struct btf_type *func;8808 const struct btf_type *func;
@@ -8697,7 +8816,8 @@ static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags) · 7 unmodified lines
8697 8816
8698 /* sanity check kfunc name */8817 /* sanity check kfunc name */
8699 func_name = btf_name_by_offset(btf, func->name_off);8818 func_name = btf_name_by_offset(btf, func->name_off);
8700 if (!func_name || !func_name[0])8819 if (!func_name || !func_name[0] ||
8820 btf_check_kfunc_name(btf, func_name, BTF_INFO_KIND(func->info)))
8701 return -EINVAL;8821 return -EINVAL;
8702 8822
8703 func = btf_type_by_id(btf, func->type);8823 func = btf_type_by_id(btf, func->type);
kernel/bpf/cgroup.c+44−21
@@ -55,6 +55,28 @@ void __init cgroup_bpf_lifetime_notifier_init(void) · 54 unmodified lines
55 &cgroup_bpf_lifetime_nb));55 &cgroup_bpf_lifetime_nb));
56}56}
57 57
58#ifdef CONFIG_BPF_LSM
59struct cgroup_lsm_atype {
60 u32 attach_btf_id;
61 int refcnt;
62 bool returns_errno;
63};
64
65static struct cgroup_lsm_atype cgroup_lsm_atype[CGROUP_LSM_NUM];
66
67static bool cgroup_bpf_hook_returns_errno(enum cgroup_bpf_attach_type atype)
68{
69 if (atype >= CGROUP_LSM_START && atype <= CGROUP_LSM_END)
70 return READ_ONCE(cgroup_lsm_atype[atype - CGROUP_LSM_START].returns_errno);
71 return true;
72}
73#else
74static bool cgroup_bpf_hook_returns_errno(enum cgroup_bpf_attach_type atype)
75{
76 return true;
77}
78#endif
79
58/* __always_inline is necessary to prevent indirect call through run_prog80/* __always_inline is necessary to prevent indirect call through run_prog
59 * function pointer.81 * function pointer.
60 */82 */
@@ -83,7 +105,8 @@ bpf_prog_run_array_cg(const struct cgroup_bpf *cgrp, · 22 unmodified lines
83 *(ret_flags) |= (func_ret >> 1);105 *(ret_flags) |= (func_ret >> 1);
84 func_ret &= 1;106 func_ret &= 1;
85 }107 }
86 if (!func_ret && !IS_ERR_VALUE((long)run_ctx.retval))108 if (!func_ret && cgroup_bpf_hook_returns_errno(atype) &&
109 !IS_ERR_VALUE((long)run_ctx.retval))
87 run_ctx.retval = -EPERM;110 run_ctx.retval = -EPERM;
88 item++;111 item++;
89 }112 }
@@ -156,13 +179,6 @@ unsigned int __cgroup_bpf_run_lsm_current(const void *ctx, · 66 unmodified lines
156}179}
157 180
158#ifdef CONFIG_BPF_LSM181#ifdef CONFIG_BPF_LSM
159struct cgroup_lsm_atype {
160 u32 attach_btf_id;
161 int refcnt;
162};
163
164static struct cgroup_lsm_atype cgroup_lsm_atype[CGROUP_LSM_NUM];
165
166static enum cgroup_bpf_attach_type182static enum cgroup_bpf_attach_type
167bpf_cgroup_atype_find(enum bpf_attach_type attach_type, u32 attach_btf_id)183bpf_cgroup_atype_find(enum bpf_attach_type attach_type, u32 attach_btf_id)
168{184{
@@ -191,10 +207,13 @@ void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) · 22 unmodified lines
191 207
192 lockdep_assert_held(&cgroup_mutex);208 lockdep_assert_held(&cgroup_mutex);
193 209
194 WARN_ON_ONCE(cgroup_lsm_atype[i].attach_btf_id &&210 if (!cgroup_lsm_atype[i].attach_btf_id) {
195 cgroup_lsm_atype[i].attach_btf_id != attach_btf_id);211 cgroup_lsm_atype[i].attach_btf_id = attach_btf_id;
196 212 WRITE_ONCE(cgroup_lsm_atype[i].returns_errno,
197 cgroup_lsm_atype[i].attach_btf_id = attach_btf_id;213 bpf_lsm_hook_returns_errno(attach_btf_id));
214 } else {
215 WARN_ON_ONCE(cgroup_lsm_atype[i].attach_btf_id != attach_btf_id);
216 }
198 cgroup_lsm_atype[i].refcnt++;217 cgroup_lsm_atype[i].refcnt++;
199}218}
200 219
@@ -203,8 +222,10 @@ void bpf_cgroup_atype_put(int cgroup_atype) · 2 unmodified lines
203 int i = cgroup_atype - CGROUP_LSM_START;222 int i = cgroup_atype - CGROUP_LSM_START;
204 223
205 cgroup_lock();224 cgroup_lock();
206 if (--cgroup_lsm_atype[i].refcnt <= 0)225 if (--cgroup_lsm_atype[i].refcnt <= 0) {
226 WRITE_ONCE(cgroup_lsm_atype[i].returns_errno, true);
207 cgroup_lsm_atype[i].attach_btf_id = 0;227 cgroup_lsm_atype[i].attach_btf_id = 0;
228 }
208 WARN_ON_ONCE(cgroup_lsm_atype[i].refcnt < 0);229 WARN_ON_ONCE(cgroup_lsm_atype[i].refcnt < 0);
209 cgroup_unlock();230 cgroup_unlock();
210}231}
@@ -1208,7 +1229,7 @@ static int cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog, · 997 unmodified lines
1208 1229
1209/* Must be called with cgroup_mutex held to avoid races. */1230/* Must be called with cgroup_mutex held to avoid races. */
1210static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,1231static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
1211 union bpf_attr __user *uattr)1232 union bpf_attr __user *uattr, u32 uattr_size)
1212{1233{
1213 __u32 __user *prog_attach_flags = u64_to_user_ptr(attr->query.prog_attach_flags);1234 __u32 __user *prog_attach_flags = u64_to_user_ptr(attr->query.prog_attach_flags);
1214 bool effective_query = attr->query.query_flags & BPF_F_QUERY_EFFECTIVE;1235 bool effective_query = attr->query.query_flags & BPF_F_QUERY_EFFECTIVE;
@@ -1259,7 +1280,8 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, · 44 unmodified lines
1259 return -EFAULT;1280 return -EFAULT;
1260 if (!effective_query && from_atype == to_atype)1281 if (!effective_query && from_atype == to_atype)
1261 revision = cgrp->bpf.revisions[from_atype];1282 revision = cgrp->bpf.revisions[from_atype];
1262 if (copy_to_user(&uattr->query.revision, &revision, sizeof(revision)))1283 if (uattr_size >= offsetofend(union bpf_attr, query.revision) &&
1284 copy_to_user(&uattr->query.revision, &revision, sizeof(revision)))
1263 return -EFAULT;1285 return -EFAULT;
1264 if (attr->query.prog_cnt == 0 || !prog_ids || !total_cnt)1286 if (attr->query.prog_cnt == 0 || !prog_ids || !total_cnt)
1265 /* return early if user requested only program count + flags */1287 /* return early if user requested only program count + flags */
@@ -1312,12 +1334,12 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, · 46 unmodified lines
1312}1334}
1313 1335
1314static int cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,1336static int cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
1315 union bpf_attr __user *uattr)1337 union bpf_attr __user *uattr, u32 uattr_size)
1316{1338{
1317 int ret;1339 int ret;
1318 1340
1319 cgroup_lock();1341 cgroup_lock();
1320 ret = __cgroup_bpf_query(cgrp, attr, uattr);1342 ret = __cgroup_bpf_query(cgrp, attr, uattr, uattr_size);
1321 cgroup_unlock();1343 cgroup_unlock();
1322 return ret;1344 return ret;
1323}1345}
@@ -1520,7 +1542,7 @@ out_put_cgroup: · 196 unmodified lines
1520}1542}
1521 1543
1522int cgroup_bpf_prog_query(const union bpf_attr *attr,1544int cgroup_bpf_prog_query(const union bpf_attr *attr,
1523 union bpf_attr __user *uattr)1545 union bpf_attr __user *uattr, u32 uattr_size)
1524{1546{
1525 struct cgroup *cgrp;1547 struct cgroup *cgrp;
1526 int ret;1548 int ret;
@@ -1529,7 +1551,7 @@ int cgroup_bpf_prog_query(const union bpf_attr *attr, · 2 unmodified lines
1529 if (IS_ERR(cgrp))1551 if (IS_ERR(cgrp))
1530 return PTR_ERR(cgrp);1552 return PTR_ERR(cgrp);
1531 1553
1532 ret = cgroup_bpf_query(cgrp, attr, uattr);1554 ret = cgroup_bpf_query(cgrp, attr, uattr, uattr_size);
1533 1555
1534 cgroup_put(cgrp);1556 cgroup_put(cgrp);
1535 return ret;1557 return ret;
@@ -1935,8 +1957,8 @@ int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head, · 399 unmodified lines
1935 1957
1936 kfree(ctx.cur_val);1958 kfree(ctx.cur_val);
1937 1959
1938 if (ret == 1 && ctx.new_updated) {1960 if (!ret && ctx.new_updated) {
1939 kfree(*buf);1961 kvfree(*buf);
1940 *buf = ctx.new_val;1962 *buf = ctx.new_val;
1941 *pcount = ctx.new_len;1963 *pcount = ctx.new_len;
1942 } else {1964 } else {
@@ -2342,6 +2364,7 @@ BPF_CALL_3(bpf_sysctl_set_new_value, struct bpf_sysctl_kern *, ctx, · 399 unmodified lines
2342 return -E2BIG;2364 return -E2BIG;
2343 2365
2344 memcpy(ctx->new_val, buf, buf_len);2366 memcpy(ctx->new_val, buf, buf_len);
2367 ((char *)ctx->new_val)[buf_len] = '\0';
2345 ctx->new_len = buf_len;2368 ctx->new_len = buf_len;
2346 ctx->new_updated = 1;2369 ctx->new_updated = 1;
2347 2370
kernel/bpf/cnum.c+120−0
@@ -0,0 +1,120 @@
1// SPDX-License-Identifier: GPL-2.0-only
2/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
3
4#include <linux/bits.h>
5
6#define T 32
7#include "cnum_defs.h"
8#undef T
9
10#define T 64
11#include "cnum_defs.h"
12#undef T
13
14struct cnum32 cnum32_from_cnum64(struct cnum64 cnum)
15{
16 if (cnum64_is_empty(cnum))
17 return CNUM32_EMPTY;
18
19 if (cnum.size >= U32_MAX)
20 return (struct cnum32){ .base = 0, .size = U32_MAX };
21 else
22 return (struct cnum32){ .base = (u32)cnum.base, .size = cnum.size };
23}
24
25/*
26 * Suppose 'a' and 'b' are laid out as follows:
27 *
28 * 64-bit number axis --->
29 *
30 * N*2^32 (N+1)*2^32 (N+2)*2^32 (N+3)*2^32
31 * ||------|---|=====|-------||----------|=====|-------||----------|=====|----|--||
32 * | |< b >| |< b >| |< b >| |
33 * | | | |
34 * |<--+--------------------------- a ---------------------------+--->|
35 * | |
36 * |<-------------------------- t -------------------------->|
37 *
38 * In such a case it is possible to infer a more tight representation t
39 * such that ∀ v ∈ a, (u32)v ∈ b: v ∈ t.
40 */
41struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b)
42{
43 /*
44 * To simplify reasoning, rotate the circles so that [virtual] a1 starts
45 * at u32 boundary, b1 represents b in this new frame of reference.
46 */
47 struct cnum32 b1 = { b.base - (u32)a.base, b.size };
48 struct cnum64 t = a;
49 u64 d, b1_max;
50
51 if (cnum64_is_empty(a) || cnum32_is_empty(b))
52 return CNUM64_EMPTY;
53
54 if (cnum32_urange_overflow(b1)) {
55 b1_max = (u32)b1.base + (u32)b1.size; /* overflow here is fine and necessary */
56 if ((u32)a.size > b1_max && (u32)a.size < b1.base) {
57 /*
58 * N*2^32 (N+1)*2^32
59 * ||=====|------------|=====||=====|---------|---|=====||
60 * |b1 ->| |<- b1||b1 ->| | |<- b1|
61 * |<----------------- a1 ------------------>|
62 * |<-------------- t ------------>|<-- d -->| (after adjustment)
63 * ^
64 * b1_max
65 */
66 d = (u32)a.size - b1_max;
67 t.size -= d;
68 } else {
69 /*
70 * No adjustments possible in the following cases:
71 *
72 * ||=====|------------|=====||===|=|-------------|=|===||
73 * |b1 ->| |<- b1||b1 +>| |<+ b1|
74 * |<----------------- a1 ------>| |
75 * |<----------------- (or) a1 ------------------->|
76 */
77 }
78 } else {
79 if (t.size < b1.base)
80 /*
81 * N*2^32 (N+1)*2^32
82 * ||----------|--|=======|--||------>
83 * |<-- a1 -->| |<- b ->|
84 */
85 return CNUM64_EMPTY;
86 /*
87 * N*2^32 (N+1)*2^32
88 * ||-------------|========|-||-----| -------|========|-||
89 * | |<- b1 ->| | |<- b1 ->|
90 * |<------------+ a1 ------------>|
91 * |<------ t ------>| (after adjustment)
92 */
93 t.base += b1.base;
94 t.size -= b1.base;
95 b1_max = b1.base + b1.size;
96 d = 0;
97 if ((u32)a.size < b1.base)
98 /*
99 * N*2^32 (N+1)*2^32
100 * ||-------------|========|-||------|-------|========|-||
101 * | |<- b1 ->| | |<- b1 ->|
102 * |<------------+-- a1 --+-------->|
103 * |<- t ->|<-- d -->| (after adjustment)
104 */
105 d = (u32)a.size + (BIT_ULL(32) - b1_max);
106 else if ((u32)a.size >= b1_max)
107 /*
108 * N*2^32 (N+1)*2^32
109 * ||--|========|------------||--|========|-------|-----||
110 * | |<- b1 ->| |<- b1 ->| |
111 * |<-+------------------ a1 ------------+------>|
112 * |<-------------- t --------------->|<- d ->| (after adjustment)
113 */
114 d = (u32)a.size - b1_max;
115 if (t.size < d)
116 return CNUM64_EMPTY;
117 t.size -= d;
118 }
119 return t;
120}
kernel/bpf/cnum_defs.h+247−0
@@ -0,0 +1,247 @@
1/* SPDX-License-Identifier: GPL-2.0-only */
2/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
3
4#ifndef T
5#error "Define T (bit width: 32, 64) before including cnum_defs.h"
6#endif
7
8#include <linux/cnum.h>
9#include <linux/kernel.h>
10#include <linux/limits.h>
11#include <linux/minmax.h>
12#include <linux/compiler_types.h>
13
14#define cnum_t __PASTE(cnum, T)
15#define ut __PASTE(u, T)
16#define st __PASTE(s, T)
17#define UT_MAX __PASTE(__PASTE(U, T), _MAX)
18#define ST_MAX __PASTE(__PASTE(S, T), _MAX)
19#define ST_MIN __PASTE(__PASTE(S, T), _MIN)
20#define EMPTY __PASTE(__PASTE(CNUM, T), _EMPTY)
21#define FN(name) __PASTE(__PASTE(cnum, T), __PASTE(_, name))
22
23struct cnum_t FN(from_urange)(ut min, ut max)
24{
25 return (struct cnum_t){ .base = min, .size = (ut)max - min };
26}
27
28struct cnum_t FN(from_srange)(st min, st max)
29{
30 ut size = (ut)max - (ut)min;
31 ut base = size == UT_MAX ? 0 : (ut)min;
32
33 return (struct cnum_t){ .base = base, .size = size };
34}
35
36/* True if this cnum represents two unsigned ranges. */
37static inline bool FN(urange_overflow)(struct cnum_t cnum)
38{
39 /* Same as cnum.base + cnum.size > UT_MAX but avoids overflow */
40 return cnum.size > UT_MAX - (ut)cnum.base;
41}
42
43/*
44 * cnum{T}_umin / cnum{T}_umax query an unsigned range represented by this cnum.
45 * If cnum represents a range crossing the UT_MAX/0 boundary, the unbound range
46 * [0..UT_MAX] is returned.
47 */
48ut FN(umin)(struct cnum_t cnum)
49{
50 return FN(urange_overflow)(cnum) ? 0 : cnum.base;
51}
52EXPORT_SYMBOL_GPL(FN(umin));
53
54ut FN(umax)(struct cnum_t cnum)
55{
56 return FN(urange_overflow)(cnum) ? UT_MAX : cnum.base + cnum.size;
57}
58EXPORT_SYMBOL_GPL(FN(umax));
59
60/* True if this cnum represents two signed ranges. */
61static inline bool FN(srange_overflow)(struct cnum_t cnum)
62{
63 return FN(contains)(cnum, (ut)ST_MAX) && FN(contains)(cnum, (ut)ST_MIN);
64}
65
66/*
67 * cnum{T}_smin / cnum{T}_smax query a signed range represented by this cnum.
68 * If cnum represents a range crossing the ST_MAX/ST_MIN boundary, the unbound range
69 * [ST_MIN..ST_MAX] is returned.
70 */
71st FN(smin)(struct cnum_t cnum)
72{
73 return FN(srange_overflow)(cnum)
74 ? ST_MIN
75 : min((st)cnum.base, (st)(cnum.base + cnum.size));
76}
77
78st FN(smax)(struct cnum_t cnum)
79{
80 return FN(srange_overflow)(cnum)
81 ? ST_MAX
82 : max((st)cnum.base, (st)(cnum.base + cnum.size));
83}
84
85/*
86 * Returns a possibly empty intersection of cnums 'a' and 'b'.
87 * If 'a' and 'b' intersect in two sub-arcs, the function over-approximates
88 * and returns either 'a' or 'b', whichever is smaller.
89 */
90struct cnum_t FN(intersect)(struct cnum_t a, struct cnum_t b)
91{
92 struct cnum_t b1;
93 ut dbase;
94
95 if (FN(is_empty)(a) || FN(is_empty)(b))
96 return EMPTY;
97
98 if (a.base > b.base)
99 swap(a, b);
100
101 /*
102 * Rotate frame of reference such that a.base is 0.
103 * 'b1' is 'b' in this frame of reference.
104 */
105 dbase = b.base - a.base;
106 b1 = (struct cnum_t){ dbase, b.size };
107 if (FN(urange_overflow)(b1)) {
108 if (b1.base <= a.size) {
109 /*
110 * Rotated frame (a.base at origin):
111 *
112 * 0 UT_MAX
113 * |--------------------------------------------|
114 * [=== a ==========================] |
115 * [= b1 tail =] [========= b1 main ==========>]
116 * ^-- b1.base <= a.size
117 *
118 * 'a' and 'b' intersect in two disjoint arcs,
119 * can't represent as single cnum, over-approximate
120 * the result.
121 */
122 return a.size <= b.size ? a : b;
123 } else {
124 /*
125 * Rotated frame (a.base at origin):
126 *
127 * 0 UT_MAX
128 * |--------------------------------------------|
129 * [=== a =============] | |
130 * [= b1 tail =] [======= b1 main ====>]
131 * ^-- b1.base > a.size
132 *
133 * Only 'b' tail intersects 'a'.
134 */
135 return (struct cnum_t) {
136 .base = a.base,
137 .size = min(a.size, (ut)(b1.base + b1.size)),
138 };
139 }
140 } else if (a.size >= b1.base) {
141 /*
142 * Rotated frame (a.base at origin):
143 *
144 * 0 UT_MAX
145 * |--------------------------------------------------|
146 * [=== a ==================================] |
147 * [== b1 =====================]
148 *
149 * 0 UT_MAX
150 * |--------------------------------------------------|
151 * [=== a ==================================] |
152 * [== b1 ====]
153 * ^-- b1.base <= a.size
154 * |<-- a.size - dbase -->|
155 *
156 * 'a' and 'b' intersect as one cnum.
157 */
158 return (struct cnum_t) {
159 .base = b.base,
160 .size = min((ut)(a.size - dbase), b.size),
161 };
162 } else {
163 return EMPTY;
164 }
165}
166
167void FN(intersect_with)(struct cnum_t *dst, struct cnum_t src)
168{
169 *dst = FN(intersect)(*dst, src);
170}
171
172void FN(intersect_with_urange)(struct cnum_t *dst, ut min, ut max)
173{
174 FN(intersect_with)(dst, FN(from_urange)(min, max));
175}
176
177void FN(intersect_with_srange)(struct cnum_t *dst, st min, st max)
178{
179 FN(intersect_with)(dst, FN(from_srange)(min, max));
180}
181
182static inline struct cnum_t FN(normalize)(struct cnum_t cnum)
183{
184 if (cnum.size == UT_MAX && cnum.base != 0 && cnum.base != (ut)ST_MAX)
185 cnum.base = 0;
186 return cnum;
187}
188
189struct cnum_t FN(add)(struct cnum_t a, struct cnum_t b)
190{
191 if (FN(is_empty)(a) || FN(is_empty)(b))
192 return EMPTY;
193 if (a.size > UT_MAX - b.size)
194 return (struct cnum_t){ 0, (ut)UT_MAX };
195 else
196 return FN(normalize)((struct cnum_t){ a.base + b.base, a.size + b.size });
197}
198
199struct cnum_t FN(negate)(struct cnum_t a)
200{
201 if (FN(is_empty)(a))
202 return EMPTY;
203 return FN(normalize)((struct cnum_t){ -((ut)a.base + a.size), a.size });
204}
205
206bool FN(is_empty)(struct cnum_t cnum)
207{
208 return cnum.base == EMPTY.base && cnum.size == EMPTY.size;
209}
210
211bool FN(contains)(struct cnum_t cnum, ut v)
212{
213 if (FN(is_empty)(cnum))
214 return false;
215 if (FN(urange_overflow)(cnum))
216 return v >= cnum.base || v <= (ut)cnum.base + cnum.size;
217 else
218 return v >= cnum.base && v <= (ut)cnum.base + cnum.size;
219}
220
221bool FN(is_const)(struct cnum_t cnum)
222{
223 return cnum.size == 0;
224}
225
226bool FN(is_subset)(struct cnum_t bigger, struct cnum_t smaller)
227{
228 if (FN(is_empty(smaller)))
229 return true;
230 if (FN(is_empty(bigger)))
231 return false;
232 /* rotate both arcs such that 'bigger' starts at origin, hence does not overflow */
233 smaller.base -= bigger.base;
234 bigger.base = 0;
235 if (FN(urange_overflow)(smaller) && bigger.size < UT_MAX)
236 return false;
237 return smaller.base + smaller.size <= bigger.size;
238}
239
240#undef EMPTY
241#undef cnum_t
242#undef ut
243#undef st
244#undef UT_MAX
245#undef ST_MAX
246#undef ST_MIN
247#undef FN
kernel/bpf/const_fold.c+8−0
@@ -58,6 +58,14 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * · 57 unmodified lines
58 u8 opcode = BPF_OP(insn->code) | BPF_SRC(insn->code);58 u8 opcode = BPF_OP(insn->code) | BPF_SRC(insn->code);
59 int r;59 int r;
60 60
61 /* Stack arg stores (r11-based) are outside the tracked register set. */
62 if (is_stack_arg_st(insn) || is_stack_arg_stx(insn))
63 return;
64 if (is_stack_arg_ldx(insn)) {
65 ci_out[insn->dst_reg] = unknown;
66 return;
67 }
68
61 switch (class) {69 switch (class) {
62 case BPF_ALU:70 case BPF_ALU:
63 case BPF_ALU64:71 case BPF_ALU64:
kernel/bpf/core.c+24−3
@@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from, · 1298 unmodified lines
1299 u32 imm_rnd = get_random_u32();1299 u32 imm_rnd = get_random_u32();
1300 s16 off;1300 s16 off;
1301 1301
1302 BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG);1302 BUILD_BUG_ON(BPF_REG_PARAMS + 2 != MAX_BPF_JIT_REG);
1303 BUILD_BUG_ON(MAX_BPF_REG + 1 != MAX_BPF_JIT_REG);1303 BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG);
1304 1304
1305 /* Constraints on AX register:1305 /* Constraints on AX register:
1306 *1306 *
@@ -1582,6 +1582,16 @@ bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struc · 275 unmodified lines
1582 insn_idx += prog->aux->subprog_start;1582 insn_idx += prog->aux->subprog_start;
1583 return env->insn_aux_data[insn_idx].indirect_target;1583 return env->insn_aux_data[insn_idx].indirect_target;
1584}1584}
1585
1586u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog)
1587{
1588 const struct bpf_subprog_info *sub;
1589
1590 if (!env)
1591 return 0;
1592 sub = &env->subprog_info[prog->aux->func_idx];
1593 return sub->stack_arg_cnt - bpf_in_stack_arg_cnt(sub);
1594}
1585#endif /* CONFIG_BPF_JIT */1595#endif /* CONFIG_BPF_JIT */
1586 1596
1587/* Base function for offset calculation. Needs to go into .text section,1597/* Base function for offset calculation. Needs to go into .text section,
@@ -2471,7 +2481,7 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map, · 883 unmodified lines
2471 cookie = aux->cgroup_storage[i] ?2481 cookie = aux->cgroup_storage[i] ?
2472 aux->cgroup_storage[i]->cookie : 0;2482 aux->cgroup_storage[i]->cookie : 0;
2473 ret = map->owner->storage_cookie[i] == cookie ||2483 ret = map->owner->storage_cookie[i] == cookie ||
2474 !cookie;2484 (!cookie && !aux->tail_call_reachable);
2475 }2485 }
2476 if (ret &&2486 if (ret &&
2477 map->owner->attach_func_proto != aux->attach_func_proto) {2487 map->owner->attach_func_proto != aux->attach_func_proto) {
@@ -3228,6 +3238,11 @@ bool __weak bpf_jit_supports_kfunc_call(void) · 750 unmodified lines
3228 return false;3238 return false;
3229}3239}
3230 3240
3241bool __weak bpf_jit_supports_stack_args(void)
3242{
3243 return false;
3244}
3245
3231bool __weak bpf_jit_supports_far_kfunc_call(void)3246bool __weak bpf_jit_supports_far_kfunc_call(void)
3232{3247{
3233 return false;3248 return false;
@@ -3363,6 +3378,12 @@ __weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena) · 129 unmodified lines
3363}3378}
3364 3379
3365#ifdef CONFIG_BPF_SYSCALL3380#ifdef CONFIG_BPF_SYSCALL
3381__weak bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write,
3382 unsigned long fault_ip)
3383{
3384 return false;
3385}
3386
3366static int __init bpf_global_ma_init(void)3387static int __init bpf_global_ma_init(void)
3367{3388{
3368 int ret;3389 int ret;
kernel/bpf/devmap.c+19−0
@@ -581,6 +581,10 @@ static int dev_map_enqueue_clone(struct bpf_dtab_netdev *obj, · 580 unmodified lines
581{581{
582 struct xdp_frame *nxdpf;582 struct xdp_frame *nxdpf;
583 583
584 /* Frags live outside the linear frame and cannot be cloned safely. */
585 if (unlikely(xdp_frame_has_frags(xdpf)))
586 return -EOPNOTSUPP;
587
584 nxdpf = xdpf_clone(xdpf);588 nxdpf = xdpf_clone(xdpf);
585 if (!nxdpf)589 if (!nxdpf)
586 return -ENOMEM;590 return -ENOMEM;
@@ -706,6 +710,18 @@ int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, · 119 unmodified lines
706 if (unlikely(err))710 if (unlikely(err))
707 return err;711 return err;
708 712
713 if (dst->xdp_prog && skb_cloned(skb)) {
714 struct sk_buff *nskb;
715
716 nskb = skb_copy(skb, GFP_ATOMIC);
717 if (!nskb)
718 return -ENOMEM;
719
720 nskb->mac_len = skb->mac_len;
721 consume_skb(skb);
722 skb = nskb;
723 }
724
709 /* Redirect has already succeeded semantically at this point, so we just725 /* Redirect has already succeeded semantically at this point, so we just
710 * return 0 even if packet is dropped. Helper below takes care of726 * return 0 even if packet is dropped. Helper below takes care of
711 * freeing skb.727 * freeing skb.
@@ -726,6 +742,9 @@ static int dev_map_redirect_clone(struct bpf_dtab_netdev *dst, · 14 unmodified lines
726 struct sk_buff *nskb;742 struct sk_buff *nskb;
727 int err;743 int err;
728 744
745 if (unlikely(skb_is_nonlinear(skb)))
746 return -EOPNOTSUPP;
747
729 nskb = skb_clone(skb, GFP_ATOMIC);748 nskb = skb_clone(skb, GFP_ATOMIC);
730 if (!nskb)749 if (!nskb)
731 return -ENOMEM;750 return -ENOMEM;
kernel/bpf/fixups.c+24−3
@@ -870,7 +870,7 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env) · 869 unmodified lines
870 case PTR_TO_BTF_ID:870 case PTR_TO_BTF_ID:
871 case PTR_TO_BTF_ID | PTR_UNTRUSTED:871 case PTR_TO_BTF_ID | PTR_UNTRUSTED:
872 /* PTR_TO_BTF_ID | MEM_ALLOC always has a valid lifetime, unlike872 /* PTR_TO_BTF_ID | MEM_ALLOC always has a valid lifetime, unlike
873 * PTR_TO_BTF_ID, and an active ref_obj_id, but the same cannot873 * PTR_TO_BTF_ID, and an active referenced id, but the same cannot
874 * be said once it is marked PTR_UNTRUSTED, hence we must handle874 * be said once it is marked PTR_UNTRUSTED, hence we must handle
875 * any faults for loads into such types. BPF_WRITE is disallowed875 * any faults for loads into such types. BPF_WRITE is disallowed
876 * for this case.876 * for this case.
@@ -1265,6 +1265,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 388 unmodified lines
1265 prog->aux->real_func_cnt = env->subprog_cnt;1265 prog->aux->real_func_cnt = env->subprog_cnt;
1266 prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func;1266 prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func;
1267 prog->aux->exception_boundary = func[0]->aux->exception_boundary;1267 prog->aux->exception_boundary = func[0]->aux->exception_boundary;
1268 prog->aux->stack_arg_sp_adjust = func[0]->aux->stack_arg_sp_adjust;
1268 bpf_prog_jit_attempt_done(prog);1269 bpf_prog_jit_attempt_done(prog);
1269 return 0;1270 return 0;
1270out_free:1271out_free:
@@ -1378,9 +1379,21 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env) · 107 unmodified lines
1378 struct bpf_prog *prog = env->prog;1379 struct bpf_prog *prog = env->prog;
1379 struct bpf_insn *insn = prog->insnsi;1380 struct bpf_insn *insn = prog->insnsi;
1380 bool has_kfunc_call = bpf_prog_has_kfunc_call(prog);1381 bool has_kfunc_call = bpf_prog_has_kfunc_call(prog);
1381 int i, depth;1382 int depth;
1382#endif1383#endif
1383 int err = 0;1384 int i, err = 0;
1385
1386 for (i = 0; i < env->subprog_cnt; i++) {
1387 struct bpf_subprog_info *subprog = &env->subprog_info[i];
1388 u16 outgoing = subprog->stack_arg_cnt - bpf_in_stack_arg_cnt(subprog);
1389
1390 if (subprog->max_out_stack_arg_cnt > outgoing) {
1391 verbose(env,
1392 "func#%d writes %u stack arg slots, but calls only require %u\n",
1393 i, subprog->max_out_stack_arg_cnt, outgoing);
1394 return -EINVAL;
1395 }
1396 }
1384 1397
1385 if (env->prog->jit_requested &&1398 if (env->prog->jit_requested &&
1386 !bpf_prog_is_offloaded(env->prog->aux)) {1399 !bpf_prog_is_offloaded(env->prog->aux)) {
@@ -1395,6 +1408,12 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env) · 8 unmodified lines
1395 verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");1408 verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
1396 return -EINVAL;1409 return -EINVAL;
1397 }1410 }
1411 for (i = 0; i < env->subprog_cnt; i++) {
1412 if (bpf_in_stack_arg_cnt(&env->subprog_info[i])) {
1413 verbose(env, "stack args are not supported in non-JITed programs\n");
1414 return -EINVAL;
1415 }
1416 }
1398 if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {1417 if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {
1399 /* When JIT fails the progs with bpf2bpf calls and tail_calls1418 /* When JIT fails the progs with bpf2bpf calls and tail_calls
1400 * have to be rejected, since interpreter doesn't support them yet.1419 * have to be rejected, since interpreter doesn't support them yet.
@@ -2167,6 +2186,8 @@ patch_map_ops_generic: · 766 unmodified lines
2167 insn->imm == BPF_FUNC_get_func_ret) {2186 insn->imm == BPF_FUNC_get_func_ret) {
2168 if (eatype == BPF_TRACE_FEXIT ||2187 if (eatype == BPF_TRACE_FEXIT ||
2169 eatype == BPF_TRACE_FSESSION ||2188 eatype == BPF_TRACE_FSESSION ||
2189 eatype == BPF_TRACE_FEXIT_MULTI ||
2190 eatype == BPF_TRACE_FSESSION_MULTI ||
2170 eatype == BPF_MODIFY_RETURN) {2191 eatype == BPF_MODIFY_RETURN) {
2171 /* Load nr_args from ctx - 8 */2192 /* Load nr_args from ctx - 8 */
2172 insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);2193 insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
kernel/bpf/hashtab.c+817−23
@@ -9,6 +9,7 @@ · 8 unmodified lines
9#include <linux/rculist_nulls.h>9#include <linux/rculist_nulls.h>
10#include <linux/rcupdate_wait.h>10#include <linux/rcupdate_wait.h>
11#include <linux/random.h>11#include <linux/random.h>
12#include <linux/rhashtable.h>
12#include <uapi/linux/btf.h>13#include <uapi/linux/btf.h>
13#include <linux/rcupdate_trace.h>14#include <linux/rcupdate_trace.h>
14#include <linux/btf_ids.h>15#include <linux/btf_ids.h>
@@ -242,6 +243,10 @@ static void htab_free_prealloced_fields(struct bpf_htab *htab) · 227 unmodified lines
242 243
243 if (IS_ERR_OR_NULL(htab->map.record))244 if (IS_ERR_OR_NULL(htab->map.record))
244 return;245 return;
246 /*
247 * Preallocated maps do not have a bpf_mem_alloc destructor, so fully
248 * destroy every element, including the extra elements.
249 */
245 if (htab_has_extra_elems(htab))250 if (htab_has_extra_elems(htab))
246 num_entries += num_possible_cpus();251 num_entries += num_possible_cpus();
247 for (i = 0; i < num_entries; i++) {252 for (i = 0; i < num_entries; i++) {
@@ -496,28 +501,26 @@ static void htab_dtor_ctx_free(void *ctx) · 248 unmodified lines
496 kfree(ctx);501 kfree(ctx);
497}502}
498 503
499static int htab_set_dtor(struct bpf_htab *htab, void (*dtor)(void *, void *))504static int bpf_ma_set_dtor(struct bpf_map *map, struct bpf_mem_alloc *ma,
505 void (*dtor)(void *, void *))
500{506{
501 u32 key_size = htab->map.key_size;
502 struct bpf_mem_alloc *ma;
503 struct htab_btf_record *hrec;507 struct htab_btf_record *hrec;
504 int err;508 int err;
505 509
506 /* No need for dtors. */510 /* No need for dtors. */
507 if (IS_ERR_OR_NULL(htab->map.record))511 if (IS_ERR_OR_NULL(map->record))
508 return 0;512 return 0;
509 513
510 hrec = kzalloc(sizeof(*hrec), GFP_KERNEL);514 hrec = kzalloc(sizeof(*hrec), GFP_KERNEL);
511 if (!hrec)515 if (!hrec)
512 return -ENOMEM;516 return -ENOMEM;
513 hrec->key_size = key_size;517 hrec->key_size = map->key_size;
514 hrec->record = btf_record_dup(htab->map.record);518 hrec->record = btf_record_dup(map->record);
515 if (IS_ERR(hrec->record)) {519 if (IS_ERR(hrec->record)) {
516 err = PTR_ERR(hrec->record);520 err = PTR_ERR(hrec->record);
517 kfree(hrec);521 kfree(hrec);
518 return err;522 return err;
519 }523 }
520 ma = htab_is_percpu(htab) ? &htab->pcpu_ma : &htab->ma;
521 bpf_mem_alloc_set_dtor(ma, dtor, htab_dtor_ctx_free, hrec);524 bpf_mem_alloc_set_dtor(ma, dtor, htab_dtor_ctx_free, hrec);
522 return 0;525 return 0;
523}526}
@@ -534,9 +537,9 @@ static int htab_map_check_btf(struct bpf_map *map, const struct btf *btf, · 10 unmodified lines
534 * populated in htab_map_alloc(), so it will always appear as NULL.537 * populated in htab_map_alloc(), so it will always appear as NULL.
535 */538 */
536 if (htab_is_percpu(htab))539 if (htab_is_percpu(htab))
537 return htab_set_dtor(htab, htab_pcpu_mem_dtor);540 return bpf_ma_set_dtor(map, &htab->pcpu_ma, htab_pcpu_mem_dtor);
538 else541 else
539 return htab_set_dtor(htab, htab_mem_dtor);542 return bpf_ma_set_dtor(map, &htab->ma, htab_mem_dtor);
540}543}
541 544
542static struct bpf_map *htab_map_alloc(union bpf_attr *attr)545static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
@@ -834,8 +837,8 @@ static int htab_lru_map_gen_lookup(struct bpf_map *map, · 291 unmodified lines
834 return insn - insn_buf;837 return insn - insn_buf;
835}838}
836 839
837static void check_and_free_fields(struct bpf_htab *htab,840static void check_and_cancel_fields(struct bpf_htab *htab,
838 struct htab_elem *elem)841 struct htab_elem *elem)
839{842{
840 if (IS_ERR_OR_NULL(htab->map.record))843 if (IS_ERR_OR_NULL(htab->map.record))
841 return;844 return;
@@ -845,11 +848,11 @@ static void check_and_free_fields(struct bpf_htab *htab, · 3 unmodified lines
845 int cpu;848 int cpu;
846 849
847 for_each_possible_cpu(cpu)850 for_each_possible_cpu(cpu)
848 bpf_obj_free_fields(htab->map.record, per_cpu_ptr(pptr, cpu));851 bpf_obj_cancel_fields(&htab->map, per_cpu_ptr(pptr, cpu));
849 } else {852 } else {
850 void *map_value = htab_elem_value(elem, htab->map.key_size);853 void *map_value = htab_elem_value(elem, htab->map.key_size);
851 854
852 bpf_obj_free_fields(htab->map.record, map_value);855 bpf_obj_cancel_fields(&htab->map, map_value);
853 }856 }
854}857}
855 858
@@ -884,7 +887,7 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node) · 28 unmodified lines
884 htab_unlock_bucket(b, flags);887 htab_unlock_bucket(b, flags);
885 888
886 if (l == tgt_l)889 if (l == tgt_l)
887 check_and_free_fields(htab, l);890 check_and_cancel_fields(htab, l);
888 return l == tgt_l;891 return l == tgt_l;
889}892}
890 893
@@ -949,7 +952,7 @@ find_first_elem: · 58 unmodified lines
949 952
950static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l)953static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l)
951{954{
952 check_and_free_fields(htab, l);955 check_and_cancel_fields(htab, l);
953 956
954 if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH)957 if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH)
955 bpf_mem_cache_free(&htab->pcpu_ma, l->ptr_to_pptr);958 bpf_mem_cache_free(&htab->pcpu_ma, l->ptr_to_pptr);
@@ -1002,7 +1005,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) · 46 unmodified lines
1002 1005
1003 if (htab_is_prealloc(htab)) {1006 if (htab_is_prealloc(htab)) {
1004 bpf_map_dec_elem_count(&htab->map);1007 bpf_map_dec_elem_count(&htab->map);
1005 check_and_free_fields(htab, l);1008 check_and_cancel_fields(htab, l);
1006 pcpu_freelist_push(&htab->freelist, &l->fnode);1009 pcpu_freelist_push(&htab->freelist, &l->fnode);
1007 } else {1010 } else {
1008 dec_elem_count(htab);1011 dec_elem_count(htab);
@@ -1019,7 +1022,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 10 unmodified lines
1019 /* copy true value_size bytes */1022 /* copy true value_size bytes */
1020 ptr = this_cpu_ptr(pptr);1023 ptr = this_cpu_ptr(pptr);
1021 copy_map_value(&htab->map, ptr, value);1024 copy_map_value(&htab->map, ptr, value);
1022 bpf_obj_free_fields(htab->map.record, ptr);1025 bpf_obj_cancel_fields(&htab->map, ptr);
1023 } else {1026 } else {
1024 u32 size = round_up(htab->map.value_size, 8);1027 u32 size = round_up(htab->map.value_size, 8);
1025 void *val;1028 void *val;
@@ -1029,7 +1032,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 3 unmodified lines
1029 cpu = map_flags >> 32;1032 cpu = map_flags >> 32;
1030 ptr = per_cpu_ptr(pptr, cpu);1033 ptr = per_cpu_ptr(pptr, cpu);
1031 copy_map_value(&htab->map, ptr, value);1034 copy_map_value(&htab->map, ptr, value);
1032 bpf_obj_free_fields(htab->map.record, ptr);1035 bpf_obj_cancel_fields(&htab->map, ptr);
1033 return;1036 return;
1034 }1037 }
1035 1038
@@ -1037,7 +1040,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 1 unmodified line
1037 ptr = per_cpu_ptr(pptr, cpu);1040 ptr = per_cpu_ptr(pptr, cpu);
1038 val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu;1041 val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu;
1039 copy_map_value(&htab->map, ptr, val);1042 copy_map_value(&htab->map, ptr, val);
1040 bpf_obj_free_fields(htab->map.record, ptr);1043 bpf_obj_cancel_fields(&htab->map, ptr);
1041 }1044 }
1042 }1045 }
1043}1046}
@@ -1253,11 +1256,11 @@ static long htab_map_update_elem(struct bpf_map *map, void *key, void *value, · 209 unmodified lines
1253 if (l_old) {1256 if (l_old) {
1254 hlist_nulls_del_rcu(&l_old->hash_node);1257 hlist_nulls_del_rcu(&l_old->hash_node);
1255 1258
1256 /* l_old has already been stashed in htab->extra_elems, free1259 /* l_old has already been stashed in htab->extra_elems, cancel
1257 * its special fields before it is available for reuse.1260 * its reusable special fields before it is available for reuse.
1258 */1261 */
1259 if (htab_is_prealloc(htab))1262 if (htab_is_prealloc(htab))
1260 check_and_free_fields(htab, l_old);1263 check_and_cancel_fields(htab, l_old);
1261 }1264 }
1262 htab_unlock_bucket(b, flags);1265 htab_unlock_bucket(b, flags);
1263 if (l_old && !htab_is_prealloc(htab))1266 if (l_old && !htab_is_prealloc(htab))
@@ -1270,7 +1273,7 @@ err: · 6 unmodified lines
1270 1273
1271static void htab_lru_push_free(struct bpf_htab *htab, struct htab_elem *elem)1274static void htab_lru_push_free(struct bpf_htab *htab, struct htab_elem *elem)
1272{1275{
1273 check_and_free_fields(htab, elem);1276 check_and_cancel_fields(htab, elem);
1274 bpf_map_dec_elem_count(&htab->map);1277 bpf_map_dec_elem_count(&htab->map);
1275 bpf_lru_push_free(&htab->lru, &elem->lru_node);1278 bpf_lru_push_free(&htab->lru, &elem->lru_node);
1276}1279}
@@ -2739,3 +2742,794 @@ const struct bpf_map_ops htab_of_maps_map_ops = { · 1462 unmodified lines
2739 BATCH_OPS(htab),2742 BATCH_OPS(htab),
2740 .map_btf_id = &htab_map_btf_ids[0],2743 .map_btf_id = &htab_map_btf_ids[0],
2741};2744};
2745
2746struct rhtab_elem {
2747 struct rhash_head node;
2748 /* key bytes, then value bytes follow */
2749 u8 data[] __aligned(8);
2750};
2751
2752struct bpf_rhtab {
2753 struct bpf_map map;
2754 struct rhashtable ht;
2755 struct bpf_mem_alloc ma;
2756 u32 elem_size;
2757 bool freeing_internal;
2758};
2759
2760static const struct rhashtable_params rhtab_params = {
2761 .head_offset = offsetof(struct rhtab_elem, node),
2762 .key_offset = offsetof(struct rhtab_elem, data),
2763};
2764
2765static inline void *rhtab_elem_value(struct rhtab_elem *l, u32 key_size)
2766{
2767 return l->data + round_up(key_size, 8);
2768}
2769
2770/* Specialize hash function and objcmp for long sized key */
2771static __always_inline int rhtab_key_cmp_long(struct rhashtable_compare_arg *arg,
2772 const void *ptr)
2773{
2774 const unsigned long key1 = *(const unsigned long *)arg->key;
2775 const struct rhtab_elem *key2 = ptr;
2776
2777 return key1 != *(const unsigned long *)key2->data;
2778}
2779
2780static __always_inline u32 rhtab_hashfn_long(const void *data, u32 len, u32 seed)
2781{
2782 u64 k = *(const unsigned long *)data;
2783
2784 return (u32)(k ^ (k >> 32)) ^ seed;
2785}
2786
2787static const struct rhashtable_params rhtab_params_long = {
2788 .head_offset = offsetof(struct rhtab_elem, node),
2789 .key_offset = offsetof(struct rhtab_elem, data),
2790 .key_len = sizeof(long),
2791 .hashfn = rhtab_hashfn_long,
2792 .obj_cmpfn = rhtab_key_cmp_long,
2793};
2794
2795static struct bpf_map *rhtab_map_alloc(union bpf_attr *attr)
2796{
2797 struct rhashtable_params params;
2798 struct bpf_rhtab *rhtab;
2799 int err = 0;
2800
2801 rhtab = bpf_map_area_alloc(sizeof(*rhtab), NUMA_NO_NODE);
2802 if (!rhtab)
2803 return ERR_PTR(-ENOMEM);
2804
2805 bpf_map_init_from_attr(&rhtab->map, attr);
2806
2807 if (rhtab->map.max_entries > 1UL << 31) {
2808 err = -E2BIG;
2809 goto free_rhtab;
2810 }
2811
2812 rhtab->elem_size = sizeof(struct rhtab_elem) + round_up(rhtab->map.key_size, 8) +
2813 round_up(rhtab->map.value_size, 8);
2814
2815 params = rhtab_params;
2816 params.key_len = rhtab->map.key_size;
2817 params.nelem_hint = (u32)attr->map_extra;
2818 params.automatic_shrinking = true;
2819
2820 if (rhtab->map.key_size == sizeof(long)) {
2821 params.hashfn = rhtab_hashfn_long;
2822 params.obj_cmpfn = rhtab_key_cmp_long;
2823 }
2824
2825 err = rhashtable_init(&rhtab->ht, &params);
2826 if (err)
2827 goto free_rhtab;
2828
2829 /* Set max_elems after rhashtable_init() since init zeroes the struct */
2830 rhtab->ht.max_elems = rhtab->map.max_entries;
2831
2832 err = bpf_mem_alloc_init(&rhtab->ma, rhtab->elem_size, false);
2833 if (err)
2834 goto destroy_rhtab;
2835
2836 return &rhtab->map;
2837
2838destroy_rhtab:
2839 rhashtable_destroy(&rhtab->ht);
2840free_rhtab:
2841 bpf_map_area_free(rhtab);
2842 return ERR_PTR(err);
2843}
2844
2845static int rhtab_map_alloc_check(union bpf_attr *attr)
2846{
2847 if (!(attr->map_flags & BPF_F_NO_PREALLOC))
2848 return -EINVAL;
2849
2850 if (attr->map_flags & BPF_F_ZERO_SEED)
2851 return -EINVAL;
2852
2853 if (attr->key_size > U16_MAX)
2854 return -E2BIG;
2855
2856 if (attr->map_extra >> 32)
2857 return -EINVAL;
2858
2859 if ((u32)attr->map_extra > U16_MAX)
2860 return -E2BIG;
2861
2862 if ((u32)attr->map_extra > attr->max_entries)
2863 return -EINVAL;
2864
2865 return htab_map_alloc_check(attr);
2866}
2867
2868static void rhtab_check_and_free_fields(struct bpf_rhtab *rhtab,
2869 struct rhtab_elem *elem)
2870{
2871 if (IS_ERR_OR_NULL(rhtab->map.record))
2872 return;
2873
2874 bpf_obj_free_fields(rhtab->map.record,
2875 rhtab_elem_value(elem, rhtab->map.key_size));
2876}
2877
2878static void rhtab_mem_dtor(void *obj, void *ctx)
2879{
2880 struct htab_btf_record *hrec = ctx;
2881 struct rhtab_elem *elem = obj;
2882
2883 if (IS_ERR_OR_NULL(hrec->record))
2884 return;
2885
2886 bpf_obj_free_fields(hrec->record,
2887 rhtab_elem_value(elem, hrec->key_size));
2888}
2889
2890static void rhtab_free_elem(void *ptr, void *arg)
2891{
2892 struct bpf_rhtab *rhtab = arg;
2893 struct rhtab_elem *elem = ptr;
2894
2895 bpf_map_free_internal_structs(&rhtab->map, rhtab_elem_value(elem, rhtab->map.key_size));
2896 bpf_mem_cache_free_rcu(&rhtab->ma, elem);
2897}
2898
2899static void rhtab_map_free(struct bpf_map *map)
2900{
2901 struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
2902
2903 rhashtable_free_and_destroy(&rhtab->ht, rhtab_free_elem, rhtab);
2904 bpf_mem_alloc_destroy(&rhtab->ma);
2905 bpf_map_area_free(rhtab);
2906}
2907
2908static void *rhtab_lookup_elem(struct bpf_map *map, void *key)
2909{
2910 struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
2911
2912 /* Hold RCU lock in case sleepable program calls via gen_lookup */
2913 guard(rcu)();
2914
2915 if (map->key_size == sizeof(long))
2916 return rhashtable_lookup_likely(&rhtab->ht, key, rhtab_params_long);
2917
2918 return rhashtable_lookup_likely(&rhtab->ht, key, rhtab_params);
2919}
2920
2921static void *rhtab_map_lookup_elem(struct bpf_map *map, void *key) __must_hold(RCU)
2922{
2923 struct rhtab_elem *l;
2924
2925 l = rhtab_lookup_elem(map, key);
2926 return l ? rhtab_elem_value(l, map->key_size) : NULL;
2927}
2928
2929static void rhtab_read_elem_value(struct bpf_map *map, void *dst, struct rhtab_elem *elem,
2930 u64 flags)
2931{
2932 void *src = rhtab_elem_value(elem, map->key_size);
2933
2934 if (flags & BPF_F_LOCK)
2935 copy_map_value_locked(map, dst, src, true);
2936 else
2937 copy_map_value(map, dst, src);
2938}
2939
2940static int rhtab_delete_elem(struct bpf_rhtab *rhtab, struct rhtab_elem *elem, void *copy,
2941 u64 flags)
2942{
2943 int err;
2944
2945 /*
2946 * disable_instrumentation() mitigates the deadlock for programs running in NMI context.
2947 * rhashtable locks bucket with local_irq_save(). Only NMI programs may reenter
2948 * rhashtable code, bpf_disable_instrumentation() disables programs running in NMI, except
2949 * raw tracepoints, which we don't have in rhashtable.
2950 */
2951 bpf_disable_instrumentation();
2952
2953 if (rhtab->map.key_size == sizeof(long))
2954 err = rhashtable_remove_fast(&rhtab->ht, &elem->node, rhtab_params_long);
2955 else
2956 err = rhashtable_remove_fast(&rhtab->ht, &elem->node, rhtab_params);
2957
2958 bpf_enable_instrumentation();
2959
2960 if (err)
2961 return err;
2962
2963 if (copy) {
2964 rhtab_read_elem_value(&rhtab->map, copy, elem, flags);
2965 check_and_init_map_value(&rhtab->map, copy);
2966 }
2967 /* Release internal structs: kptr, bpf_timer, task_work, wq */
2968 rhtab_check_and_free_fields(rhtab, elem);
2969 bpf_mem_cache_free_rcu(&rhtab->ma, elem);
2970 return 0;
2971}
2972
2973
2974static long rhtab_map_delete_elem(struct bpf_map *map, void *key)
2975{
2976 struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
2977 struct rhtab_elem *elem;
2978
2979 guard(rcu)();
2980
2981 elem = rhtab_lookup_elem(map, key);
2982 if (!elem)
2983 return -ENOENT;
2984
2985 return rhtab_delete_elem(rhtab, elem, NULL, 0);
2986}
2987
2988static int rhtab_map_lookup_and_delete_elem(struct bpf_map *map, void *key, void *value, u64 flags)
2989{
2990 struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
2991 struct rhtab_elem *elem;
2992 int err;
2993
2994 err = bpf_map_check_op_flags(map, flags, BPF_F_LOCK);
2995 if (err)
2996 return err;
2997
2998 guard(rcu)();
2999
3000 elem = rhtab_lookup_elem(map, key);
3001 if (!elem)
3002 return -ENOENT;
3003
3004 return rhtab_delete_elem(rhtab, elem, value, flags);
3005}
3006
3007static long rhtab_map_update_existing(struct bpf_map *map, struct rhtab_elem *elem, void *value,
3008 u64 map_flags)
3009{
3010 struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3011 void *old_val = rhtab_elem_value(elem, map->key_size);
3012
3013 if (map_flags & BPF_NOEXIST)
3014 return -EEXIST;
3015
3016 if (map_flags & BPF_F_LOCK)
3017 copy_map_value_locked(map, old_val, value, false);
3018 else
3019 copy_map_value(map, old_val, value);
3020
3021 /*
3022 * Torn reads: a concurrent reader without BPF_F_LOCK may observe
3023 * the value mid-copy. Callers requiring consistent reads must use
3024 * BPF_F_LOCK, matching arraymap semantics.
3025 *
3026 * copy_map_value() skips special-field offsets, so old timers/
3027 * kptrs/etc. still sit in the slot. Cancel them after the copy
3028 * to match arraymap's update semantics.
3029 */
3030 rhtab_check_and_free_fields(rhtab, elem);
3031 return 0;
3032}
3033
3034static long rhtab_map_update_elem(struct bpf_map *map, void *key, void *value, u64 map_flags)
3035{
3036 struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3037 struct rhtab_elem *elem, *tmp;
3038
3039 if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST))
3040 return -EINVAL;
3041
3042 if ((map_flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))
3043 return -EINVAL;
3044
3045 guard(rcu)();
3046 elem = rhtab_lookup_elem(map, key);
3047 if (elem)
3048 return rhtab_map_update_existing(map, elem, value, map_flags);
3049
3050 if (map_flags & BPF_EXIST)
3051 return -ENOENT;
3052
3053 /*
3054 * Reject new insertions while map_release_uref cleanup walks the
3055 * table. Without this, new elements could keep triggering rehash
3056 * and prevent the walk from terminating.
3057 */
3058 if (READ_ONCE(rhtab->freeing_internal))
3059 return -EBUSY;
3060
3061 /* Check max_entries limit before inserting new element */
3062 if (atomic_read(&rhtab->ht.nelems) >= map->max_entries)
3063 return -E2BIG;
3064
3065 elem = bpf_mem_cache_alloc(&rhtab->ma);
3066 if (!elem)
3067 return -ENOMEM;
3068
3069 memcpy(elem->data, key, map->key_size);
3070 copy_map_value(map, rhtab_elem_value(elem, map->key_size), value);
3071 check_and_init_map_value(map, rhtab_elem_value(elem, map->key_size));
3072
3073 /* Prevent deadlock for NMI programs attempting to take bucket lock */
3074 bpf_disable_instrumentation();
3075
3076 if (map->key_size == sizeof(long))
3077 tmp = rhashtable_lookup_get_insert_fast(&rhtab->ht, &elem->node, rhtab_params_long);
3078 else
3079 tmp = rhashtable_lookup_get_insert_fast(&rhtab->ht, &elem->node, rhtab_params);
3080
3081 bpf_enable_instrumentation();
3082
3083 if (tmp) {
3084 bpf_mem_cache_free(&rhtab->ma, elem);
3085 if (IS_ERR(tmp))
3086 return PTR_ERR(tmp);
3087
3088 return rhtab_map_update_existing(map, tmp, value, map_flags);
3089 }
3090
3091 return 0;
3092}
3093
3094static int rhtab_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
3095{
3096 struct bpf_insn *insn = insn_buf;
3097 const int ret = BPF_REG_0;
3098
3099 BUILD_BUG_ON(!__same_type(&rhtab_lookup_elem,
3100 (void *(*)(struct bpf_map *map, void *key)) NULL));
3101 *insn++ = BPF_EMIT_CALL(rhtab_lookup_elem);
3102 *insn++ = BPF_JMP_IMM(BPF_JEQ, ret, 0, 1);
3103 *insn++ = BPF_ALU64_IMM(BPF_ADD, ret,
3104 offsetof(struct rhtab_elem, data) + round_up(map->key_size, 8));
3105
3106 return insn - insn_buf;
3107}
3108
3109static int rhtab_map_check_btf(struct bpf_map *map, const struct btf *btf,
3110 const struct btf_type *key_type,
3111 const struct btf_type *value_type)
3112{
3113 struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3114
3115 return bpf_ma_set_dtor(map, &rhtab->ma, rhtab_mem_dtor);
3116}
3117
3118static void rhtab_map_free_internal_structs(struct bpf_map *map)
3119{
3120 struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3121 struct rhashtable_iter iter;
3122 struct rhtab_elem *elem;
3123
3124 if (!bpf_map_has_internal_structs(map))
3125 return;
3126
3127 /*
3128 * Block new insertions. Once observed, no new growth is triggered,
3129 * so any in-flight rehash will drain and the walker is guaranteed
3130 * to stop returning -EAGAIN. Treat -EAGAIN as "rehash in progress,
3131 * retry"; do not wait for the worker.
3132 */
3133 WRITE_ONCE(rhtab->freeing_internal, true);
3134
3135 rhashtable_walk_enter(&rhtab->ht, &iter);
3136 rhashtable_walk_start(&iter);
3137
3138 while ((elem = rhashtable_walk_next(&iter))) {
3139 if (IS_ERR(elem)) {
3140 if (PTR_ERR(elem) == -EAGAIN)
3141 continue;
3142 break;
3143 }
3144
3145 bpf_map_free_internal_structs(map, rhtab_elem_value(elem, map->key_size));
3146
3147 if (need_resched()) { /* Avoid stalls on large maps */
3148 rhashtable_walk_stop(&iter);
3149 cond_resched();
3150 rhashtable_walk_start(&iter);
3151 }
3152 }
3153
3154 rhashtable_walk_stop(&iter);
3155 rhashtable_walk_exit(&iter);
3156 WRITE_ONCE(rhtab->freeing_internal, false);
3157}
3158
3159static int rhtab_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
3160 __must_hold_shared(RCU)
3161{
3162 struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3163 struct rhtab_elem *elem;
3164
3165 elem = rhashtable_next_key(&rhtab->ht, key);
3166
3167 /* if not found, return the first key */
3168 if (PTR_ERR(elem) == -ENOENT)
3169 elem = rhashtable_next_key(&rhtab->ht, NULL);
3170
3171 if (IS_ERR(elem))
3172 return PTR_ERR(elem);
3173 if (!elem)
3174 return -ENOENT;
3175
3176 memcpy(next_key, elem->data, map->key_size);
3177 return 0;
3178}
3179
3180static void rhtab_map_seq_show_elem(struct bpf_map *map, void *key, struct seq_file *m)
3181{
3182 void *value;
3183
3184 /* Guarantee that hashtab value is not freed */
3185 guard(rcu)();
3186
3187 value = rhtab_map_lookup_elem(map, key);
3188 if (!value)
3189 return;
3190
3191 btf_type_seq_show(map->btf, map->btf_key_type_id, key, m);
3192 seq_puts(m, ": ");
3193 btf_type_seq_show(map->btf, map->btf_value_type_id, value, m);
3194 seq_putc(m, '\n');
3195}
3196
3197static long bpf_each_rhash_elem(struct bpf_map *map, bpf_callback_t callback_fn,
3198 void *callback_ctx, u64 flags)
3199{
3200 struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3201 void *prev_key = NULL;
3202 struct rhtab_elem *elem;
3203 int num_elems = 0;
3204 u64 ret = 0;
3205
3206 cant_migrate();
3207
3208 if (flags != 0)
3209 return -EINVAL;
3210
3211 rcu_read_lock();
3212 /*
3213 * Best-effort iteration: if rhashtable is concurrently resized or
3214 * elements are deleted/inserted, there may be missed or duplicate
3215 * elements visited.
3216 */
3217 while ((elem = rhashtable_next_key(&rhtab->ht, prev_key))) {
3218 if (IS_ERR(elem))
3219 break;
3220 num_elems++;
3221 ret = callback_fn((u64)(long)map,
3222 (u64)(long)elem->data,
3223 (u64)(long)rhtab_elem_value(elem, map->key_size),
3224 (u64)(long)callback_ctx, 0);
3225 if (ret)
3226 break;
3227
3228 prev_key = elem->data; /* valid while RCU held */
3229 }
3230 rcu_read_unlock();
3231
3232 return num_elems;
3233}
3234
3235static u64 rhtab_map_mem_usage(const struct bpf_map *map)
3236{
3237 struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3238 u64 num_entries;
3239
3240 /* Excludes rhashtable bucket overhead (~ nelems * sizeof(void *) at 75% load). */
3241 num_entries = atomic_read(&rhtab->ht.nelems);
3242 return sizeof(struct bpf_rhtab) + rhtab->elem_size * num_entries;
3243}
3244
3245static int __rhtab_map_lookup_and_delete_batch(struct bpf_map *map,
3246 const union bpf_attr *attr,
3247 union bpf_attr __user *uattr,
3248 bool do_delete)
3249{
3250 struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
3251 void __user *uvalues = u64_to_user_ptr(attr->batch.values);
3252 void __user *ukeys = u64_to_user_ptr(attr->batch.keys);
3253 void __user *ubatch = u64_to_user_ptr(attr->batch.in_batch);
3254 void *cursor = NULL, *keys = NULL, *values = NULL, *dst_key, *dst_val;
3255 struct rhtab_elem **del_elems = NULL;
3256 u32 max_count, total, key_size, value_size, i;
3257 bool has_next_cursor = false;
3258 struct rhtab_elem *elem;
3259 u64 elem_map_flags, map_flags;
3260 int ret = 0;
3261
3262 elem_map_flags = attr->batch.elem_flags;
3263 ret = bpf_map_check_op_flags(map, elem_map_flags, BPF_F_LOCK);
3264 if (ret)
3265 return ret;
3266
3267 map_flags = attr->batch.flags;
3268 if (map_flags)
3269 return -EINVAL;
3270
3271 max_count = attr->batch.count;
3272 if (!max_count)
3273 return 0;
3274
3275 if (put_user(0, &uattr->batch.count))
3276 return -EFAULT;
3277
3278 key_size = map->key_size;
3279 value_size = map->value_size;
3280
3281 keys = kvmalloc_array(max_count, key_size, GFP_USER | __GFP_NOWARN);
3282 values = kvmalloc_array(max_count, value_size, GFP_USER | __GFP_NOWARN);
3283 if (do_delete)
3284 del_elems = kvmalloc_array(max_count, sizeof(void *),
3285 GFP_USER | __GFP_NOWARN);
3286 cursor = kmalloc(key_size, GFP_USER | __GFP_NOWARN);
3287
3288 if (!keys || !values || !cursor || (do_delete && !del_elems)) {
3289 ret = -ENOMEM;
3290 goto free;
3291 }
3292
3293 if (ubatch && copy_from_user(cursor, ubatch, key_size)) {
3294 ret = -EFAULT;
3295 goto free;
3296 }
3297
3298 dst_key = keys;
3299 dst_val = values;
3300 total = 0;
3301
3302 rcu_read_lock();
3303
3304 /*
3305 * Cursor stores the key of the next-to-process element (stashed by
3306 * the previous batch). Look it up directly so the element is included
3307 * here rather than skipped by next_key(). If the cursor was deleted
3308 * concurrently (or by the previous do_delete batch), return -EAGAIN
3309 * so userspace can distinguish a lost cursor from end-of-iteration
3310 * (-ENOENT) and restart from a NULL cursor.
3311 */
3312 if (ubatch) {
3313 elem = rhtab_lookup_elem(map, cursor);
3314 if (!elem) {
3315 rcu_read_unlock();
3316 ret = -EAGAIN;
3317 goto free;
3318 }
3319 } else {
3320 elem = rhashtable_next_key(&rhtab->ht, NULL);
3321 }
3322
3323 while (elem && !IS_ERR(elem) && total < max_count) {
3324 memcpy(dst_key, elem->data, key_size);
3325 rhtab_read_elem_value(map, dst_val, elem, elem_map_flags);
3326 check_and_init_map_value(map, dst_val);
3327
3328 if (do_delete)
3329 del_elems[total] = elem;
3330
3331 elem = rhashtable_next_key(&rhtab->ht, dst_key);
3332 dst_key += key_size;
3333 dst_val += value_size;
3334 total++;
3335
3336 /* Bail to userspace to avoid stalls. */
3337 if (need_resched())
3338 break;
3339 }
3340
3341 if (elem && !IS_ERR(elem)) {
3342 /* Stash next-to-process key as cursor for the next batch. */
3343 memcpy(cursor, elem->data, key_size);
3344 has_next_cursor = true;
3345 }
3346
3347 if (do_delete) {
3348 for (i = 0; i < total; i++)
3349 rhtab_delete_elem(rhtab, del_elems[i], NULL, 0);
3350 }
3351
3352 rcu_read_unlock();
3353
3354 if (total == 0) {
3355 ret = -ENOENT;
3356 goto free;
3357 }
3358
3359 /* No more elements after this batch. */
3360 if (!has_next_cursor)
3361 ret = -ENOENT;
3362
3363 if (copy_to_user(ukeys, keys, (size_t)total * key_size) ||
3364 copy_to_user(uvalues, values, (size_t)total * value_size) ||
3365 put_user(total, &uattr->batch.count) ||
3366 (has_next_cursor &&
3367 copy_to_user(u64_to_user_ptr(attr->batch.out_batch),
3368 cursor, key_size))) {
3369 ret = -EFAULT;
3370 goto free;
3371 }
3372
3373free:
3374 kfree(cursor);
3375 kvfree(keys);
3376 kvfree(values);
3377 kvfree(del_elems);
3378 return ret;
3379}
3380
3381static int rhtab_map_lookup_batch(struct bpf_map *map, const union bpf_attr *attr,
3382 union bpf_attr __user *uattr)
3383{
3384 return __rhtab_map_lookup_and_delete_batch(map, attr, uattr, false);
3385}
3386
3387static int rhtab_map_lookup_and_delete_batch(struct bpf_map *map, const union bpf_attr *attr,
3388 union bpf_attr __user *uattr)
3389{
3390 return __rhtab_map_lookup_and_delete_batch(map, attr, uattr, true);
3391}
3392
3393struct bpf_iter_seq_rhash_map_info {
3394 struct bpf_map *map;
3395 struct bpf_rhtab *rhtab;
3396 struct rhashtable_iter iter;
3397};
3398
3399static void *bpf_rhash_map_seq_start(struct seq_file *seq, loff_t *pos)
3400 __acquires(RCU)
3401{
3402 struct bpf_iter_seq_rhash_map_info *info = seq->private;
3403 struct rhtab_elem *elem;
3404
3405 rhashtable_walk_start(&info->iter);
3406 /*
3407 * Re-deliver the element returned by walk_next() at the end of the
3408 * previous read() — bpf_seq_read may have stopped before show()
3409 * consumed it. Rehash rewinds the walker; retry on -EAGAIN.
3410 */
3411 do {
3412 elem = rhashtable_walk_peek(&info->iter);
3413 } while (PTR_ERR(elem) == -EAGAIN);
3414
3415 if (IS_ERR(elem))
3416 return NULL;
3417
3418 if (elem && *pos == 0)
3419 ++*pos;
3420 return elem;
3421}
3422
3423static void *bpf_rhash_map_seq_next(struct seq_file *seq, void *v, loff_t *pos)
3424{
3425 struct bpf_iter_seq_rhash_map_info *info = seq->private;
3426 struct rhtab_elem *elem;
3427
3428 ++*pos;
3429
3430 /* Rehash rewinds the walker; retry until it stops returning -EAGAIN. */
3431 do {
3432 elem = rhashtable_walk_next(&info->iter);
3433 } while (PTR_ERR(elem) == -EAGAIN);
3434
3435 if (IS_ERR(elem))
3436 return NULL;
3437 return elem;
3438}
3439
3440static int __bpf_rhash_map_seq_show(struct seq_file *seq,
3441 struct rhtab_elem *elem)
3442{
3443 struct bpf_iter_seq_rhash_map_info *info = seq->private;
3444 struct bpf_iter__bpf_map_elem ctx = {};
3445 struct bpf_iter_meta meta;
3446 struct bpf_prog *prog;
3447 int ret = 0;
3448
3449 meta.seq = seq;
3450 prog = bpf_iter_get_info(&meta, elem == NULL);
3451 if (prog) {
3452 ctx.meta = &meta;
3453 ctx.map = info->map;
3454 if (elem) {
3455 ctx.key = elem->data;
3456 ctx.value = rhtab_elem_value(elem, info->map->key_size);
3457 }
3458 ret = bpf_iter_run_prog(prog, &ctx);
3459 }
3460
3461 return ret;
3462}
3463
3464static int bpf_rhash_map_seq_show(struct seq_file *seq, void *v)
3465{
3466 return __bpf_rhash_map_seq_show(seq, v);
3467}
3468
3469static void bpf_rhash_map_seq_stop(struct seq_file *seq, void *v)
3470 __releases(RCU)
3471{
3472 struct bpf_iter_seq_rhash_map_info *info = seq->private;
3473
3474 if (!v)
3475 (void)__bpf_rhash_map_seq_show(seq, NULL);
3476
3477 rhashtable_walk_stop(&info->iter);
3478}
3479
3480static int bpf_iter_init_rhash_map(void *priv_data, struct bpf_iter_aux_info *aux)
3481{
3482 struct bpf_iter_seq_rhash_map_info *info = priv_data;
3483 struct bpf_map *map = aux->map;
3484
3485 bpf_map_inc_with_uref(map);
3486 info->map = map;
3487 info->rhtab = container_of(map, struct bpf_rhtab, map);
3488 rhashtable_walk_enter(&info->rhtab->ht, &info->iter);
3489 return 0;
3490}
3491
3492static void bpf_iter_fini_rhash_map(void *priv_data)
3493{
3494 struct bpf_iter_seq_rhash_map_info *info = priv_data;
3495
3496 rhashtable_walk_exit(&info->iter);
3497 bpf_map_put_with_uref(info->map);
3498}
3499
3500static const struct seq_operations bpf_rhash_map_seq_ops = {
3501 .start = bpf_rhash_map_seq_start,
3502 .next = bpf_rhash_map_seq_next,
3503 .stop = bpf_rhash_map_seq_stop,
3504 .show = bpf_rhash_map_seq_show,
3505};
3506
3507static const struct bpf_iter_seq_info rhash_iter_seq_info = {
3508 .seq_ops = &bpf_rhash_map_seq_ops,
3509 .init_seq_private = bpf_iter_init_rhash_map,
3510 .fini_seq_private = bpf_iter_fini_rhash_map,
3511 .seq_priv_size = sizeof(struct bpf_iter_seq_rhash_map_info),
3512};
3513
3514BTF_ID_LIST_SINGLE(rhtab_map_btf_ids, struct, bpf_rhtab)
3515const struct bpf_map_ops rhtab_map_ops = {
3516 .map_meta_equal = bpf_map_meta_equal,
3517 .map_alloc_check = rhtab_map_alloc_check,
3518 .map_alloc = rhtab_map_alloc,
3519 .map_free = rhtab_map_free,
3520 .map_get_next_key = rhtab_map_get_next_key,
3521 .map_release_uref = rhtab_map_free_internal_structs,
3522 .map_check_btf = rhtab_map_check_btf,
3523 .map_lookup_elem = rhtab_map_lookup_elem,
3524 .map_lookup_and_delete_elem = rhtab_map_lookup_and_delete_elem,
3525 .map_update_elem = rhtab_map_update_elem,
3526 .map_delete_elem = rhtab_map_delete_elem,
3527 .map_gen_lookup = rhtab_map_gen_lookup,
3528 .map_seq_show_elem = rhtab_map_seq_show_elem,
3529 .map_set_for_each_callback_args = map_set_for_each_callback_args,
3530 .map_for_each_callback = bpf_each_rhash_elem,
3531 .map_mem_usage = rhtab_map_mem_usage,
3532 BATCH_OPS(rhtab),
3533 .map_btf_id = &rhtab_map_btf_ids[0],
3534 .iter_seq_info = &rhash_iter_seq_info,
3535};
kernel/bpf/helpers.c+154−50
@@ -1944,7 +1944,7 @@ static const struct bpf_func_proto bpf_dynptr_read_proto = { · 1943 unmodified lines
1944 .ret_type = RET_INTEGER,1944 .ret_type = RET_INTEGER,
1945 .arg1_type = ARG_PTR_TO_UNINIT_MEM,1945 .arg1_type = ARG_PTR_TO_UNINIT_MEM,
1946 .arg2_type = ARG_CONST_SIZE_OR_ZERO,1946 .arg2_type = ARG_CONST_SIZE_OR_ZERO,
1947 .arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,1947 .arg3_type = ARG_PTR_TO_DYNPTR,
1948 .arg4_type = ARG_ANYTHING,1948 .arg4_type = ARG_ANYTHING,
1949 .arg5_type = ARG_ANYTHING,1949 .arg5_type = ARG_ANYTHING,
1950};1950};
@@ -2001,7 +2001,7 @@ static const struct bpf_func_proto bpf_dynptr_write_proto = { · 50 unmodified lines
2001 .func = bpf_dynptr_write,2001 .func = bpf_dynptr_write,
2002 .gpl_only = false,2002 .gpl_only = false,
2003 .ret_type = RET_INTEGER,2003 .ret_type = RET_INTEGER,
2004 .arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,2004 .arg1_type = ARG_PTR_TO_DYNPTR,
2005 .arg2_type = ARG_ANYTHING,2005 .arg2_type = ARG_ANYTHING,
2006 .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,2006 .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
2007 .arg4_type = ARG_CONST_SIZE_OR_ZERO,2007 .arg4_type = ARG_CONST_SIZE_OR_ZERO,
@@ -2044,7 +2044,7 @@ static const struct bpf_func_proto bpf_dynptr_data_proto = { · 36 unmodified lines
2044 .func = bpf_dynptr_data,2044 .func = bpf_dynptr_data,
2045 .gpl_only = false,2045 .gpl_only = false,
2046 .ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,2046 .ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,
2047 .arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,2047 .arg1_type = ARG_PTR_TO_DYNPTR,
2048 .arg2_type = ARG_ANYTHING,2048 .arg2_type = ARG_ANYTHING,
2049 .arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,2049 .arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
2050};2050};
@@ -2247,10 +2247,11 @@ EXPORT_SYMBOL_GPL(bpf_base_func_proto); · 196 unmodified lines
2247void bpf_list_head_free(const struct btf_field *field, void *list_head,2247void bpf_list_head_free(const struct btf_field *field, void *list_head,
2248 struct bpf_spin_lock *spin_lock)2248 struct bpf_spin_lock *spin_lock)
2249{2249{
2250 struct list_head *head = list_head, *orig_head = list_head;2250 struct list_head *head = list_head, drain, *pos, *n;
2251 2251
2252 BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head));2252 BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head));
2253 BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head));2253 BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head));
2254 INIT_LIST_HEAD(&drain);
2254 2255
2255 /* Do the actual list draining outside the lock to not hold the lock for2256 /* Do the actual list draining outside the lock to not hold the lock for
2256 * too long, and also prevent deadlocks if tracing programs end up2257 * too long, and also prevent deadlocks if tracing programs end up
@@ -2261,20 +2262,30 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head, · 4 unmodified lines
2261 __bpf_spin_lock_irqsave(spin_lock);2262 __bpf_spin_lock_irqsave(spin_lock);
2262 if (!head->next || list_empty(head))2263 if (!head->next || list_empty(head))
2263 goto unlock;2264 goto unlock;
2264 head = head->next;2265 list_for_each_safe(pos, n, head) {
2266 struct bpf_list_node_kern *node;
2267
2268 node = container_of(pos, struct bpf_list_node_kern, list_head);
2269 WRITE_ONCE(node->owner, BPF_PTR_POISON);
2270 list_move_tail(pos, &drain);
2271 }
2265unlock:2272unlock:
2266 INIT_LIST_HEAD(orig_head);2273 INIT_LIST_HEAD(head);
2267 __bpf_spin_unlock_irqrestore(spin_lock);2274 __bpf_spin_unlock_irqrestore(spin_lock);
2268 2275
2269 while (head != orig_head) {2276 while (!list_empty(&drain)) {
2270 void *obj = head;2277 struct bpf_list_node_kern *node;
2271 2278
2272 obj -= field->graph_root.node_offset;2279 pos = drain.next;
2273 head = head->next;2280 node = container_of(pos, struct bpf_list_node_kern, list_head);
2281 list_del_init(pos);
2282 /* Ensure __bpf_list_add() sees the node as unlinked. */
2283 smp_store_release(&node->owner, NULL);
2274 /* The contained type can also have resources, including a2284 /* The contained type can also have resources, including a
2275 * bpf_list_head which needs to be freed.2285 * bpf_list_head which needs to be freed.
2276 */2286 */
2277 __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);2287 __bpf_obj_drop_impl((char *)pos - field->graph_root.node_offset,
2288 field->graph_root.value_rec, false);
2278 }2289 }
2279}2290}
2280 2291
@@ -2295,6 +2306,7 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, · 14 unmodified lines
2295 struct bpf_spin_lock *spin_lock)2306 struct bpf_spin_lock *spin_lock)
2296{2307{
2297 struct rb_root_cached orig_root, *root = rb_root;2308 struct rb_root_cached orig_root, *root = rb_root;
2309 struct bpf_rb_node_kern *node;
2298 struct rb_node *pos, *n;2310 struct rb_node *pos, *n;
2299 void *obj;2311 void *obj;
2300 2312
@@ -2303,14 +2315,20 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, · 2 unmodified lines
2303 2315
2304 __bpf_spin_lock_irqsave(spin_lock);2316 __bpf_spin_lock_irqsave(spin_lock);
2305 orig_root = *root;2317 orig_root = *root;
2318 bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
2319 node = rb_entry(pos, struct bpf_rb_node_kern, rb_node);
2320 WRITE_ONCE(node->owner, BPF_PTR_POISON);
2321 }
2306 *root = RB_ROOT_CACHED;2322 *root = RB_ROOT_CACHED;
2307 __bpf_spin_unlock_irqrestore(spin_lock);2323 __bpf_spin_unlock_irqrestore(spin_lock);
2308 2324
2309 bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {2325 bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
2310 obj = pos;2326 obj = pos;
2311 obj -= field->graph_root.node_offset;2327 obj -= field->graph_root.node_offset;
2312 2328 node = rb_entry(pos, struct bpf_rb_node_kern, rb_node);
2313 2329 RB_CLEAR_NODE(pos);
2330 /* Ensure __bpf_rbtree_add() sees the node as unlinked. */
2331 smp_store_release(&node->owner, NULL);
2314 __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);2332 __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
2315 }2333 }
2316}2334}
@@ -2467,9 +2485,11 @@ __bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta · 150 unmodified lines
2467 2485
2468static int __bpf_list_add(struct bpf_list_node_kern *node,2486static int __bpf_list_add(struct bpf_list_node_kern *node,
2469 struct bpf_list_head *head,2487 struct bpf_list_head *head,
2470 bool tail, struct btf_record *rec, u64 off)2488 struct list_head **prev_ptr,
2489 struct btf_record *rec, u64 off)
2471{2490{
2472 struct list_head *n = &node->list_head, *h = (void *)head;2491 struct list_head *n = &node->list_head, *h = (void *)head;
2492 struct list_head *prev;
2473 2493
2474 /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't2494 /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
2475 * called on its fields, so init here2495 * called on its fields, so init here
@@ -2477,19 +2497,31 @@ static int __bpf_list_add(struct bpf_list_node_kern *node, · 1 unmodified line
2477 if (unlikely(!h->next))2497 if (unlikely(!h->next))
2478 INIT_LIST_HEAD(h);2498 INIT_LIST_HEAD(h);
2479 2499
2500 prev = *prev_ptr;
2501
2502 /* When prev is not the list head, it must be a node in this list. */
2503 if (prev != h) {
2504 struct bpf_list_node_kern *prev_kn =
2505 container_of(prev, struct bpf_list_node_kern, list_head);
2506
2507 if (unlikely(READ_ONCE(prev_kn->owner) != head))
2508 goto fail;
2509 }
2510
2480 /* node->owner != NULL implies !list_empty(n), no need to separately2511 /* node->owner != NULL implies !list_empty(n), no need to separately
2481 * check the latter2512 * check the latter
2482 */2513 */
2483 if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {2514 if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON))
2484 /* Only called from BPF prog, no need to migrate_disable */2515 goto fail;
2485 __bpf_obj_drop_impl((void *)n - off, rec, false);
2486 return -EINVAL;
2487 }
2488 2516
2489 tail ? list_add_tail(n, h) : list_add(n, h);2517 list_add(n, prev);
2490 WRITE_ONCE(node->owner, head);2518 WRITE_ONCE(node->owner, head);
2491
2492 return 0;2519 return 0;
2520
2521fail:
2522 /* Only called from BPF prog, no need to migrate_disable */
2523 __bpf_obj_drop_impl((void *)n - off, rec, false);
2524 return -EINVAL;
2493}2525}
2494 2526
2495/**2527/**
@@ -2510,8 +2542,9 @@ __bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head, · 14 unmodified lines
2510 u64 off)2542 u64 off)
2511{2543{
2512 struct bpf_list_node_kern *n = (void *)node;2544 struct bpf_list_node_kern *n = (void *)node;
2545 struct list_head *h = (void *)head;
2513 2546
2514 return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off);2547 return __bpf_list_add(n, head, &h, meta ? meta->record : NULL, off);
2515}2548}
2516 2549
2517__bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,2550__bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,
@@ -2539,8 +2572,9 @@ __bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head, · 21 unmodified lines
2539 u64 off)2572 u64 off)
2540{2573{
2541 struct bpf_list_node_kern *n = (void *)node;2574 struct bpf_list_node_kern *n = (void *)node;
2575 struct list_head *h = (void *)head;
2542 2576
2543 return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off);2577 return __bpf_list_add(n, head, &h->prev, meta ? meta->record : NULL, off);
2544}2578}
2545 2579
2546__bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,2580__bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
@@ -2550,37 +2584,63 @@ __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head, · 3 unmodified lines
2550 return bpf_list_push_back(head, node, meta__ign, off);2584 return bpf_list_push_back(head, node, meta__ign, off);
2551}2585}
2552 2586
2553static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail)2587__bpf_kfunc int bpf_list_add(struct bpf_list_head *head, struct bpf_list_node *new,
2588 struct bpf_list_node *prev__nonown_allowed,
2589 struct btf_struct_meta *meta, u64 off)
2590{
2591 struct bpf_list_node_kern *n = (void *)new, *p = (void *)prev__nonown_allowed;
2592 struct list_head *prev_ptr = &p->list_head;
2593
2594 return __bpf_list_add(n, head, &prev_ptr, meta ? meta->record : NULL, off);
2595}
2596
2597static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head,
2598 struct list_head *n)
2554{2599{
2555 struct list_head *n, *h = (void *)head;2600 struct list_head *h = (void *)head;
2556 struct bpf_list_node_kern *node;2601 struct bpf_list_node_kern *node;
2557 2602
2558 /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't2603 /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
2559 * called on its fields, so init here2604 * called on its fields, so init here
2560 */2605 */
2561 if (unlikely(!h->next))2606 if (unlikely(!h->next)) {
2562 INIT_LIST_HEAD(h);2607 INIT_LIST_HEAD(h);
2608 return NULL;
2609 }
2563 if (list_empty(h))2610 if (list_empty(h))
2564 return NULL;2611 return NULL;
2565 2612
2566 n = tail ? h->prev : h->next;
2567 node = container_of(n, struct bpf_list_node_kern, list_head);2613 node = container_of(n, struct bpf_list_node_kern, list_head);
2568 if (WARN_ON_ONCE(READ_ONCE(node->owner) != head))2614 if (unlikely(READ_ONCE(node->owner) != head))
2569 return NULL;2615 return NULL;
2570 2616
2571 list_del_init(n);2617 list_del_init(n);
2572 WRITE_ONCE(node->owner, NULL);2618 /* Ensure __bpf_list_add() sees the node as unlinked. */
2619 smp_store_release(&node->owner, NULL);
2573 return (struct bpf_list_node *)n;2620 return (struct bpf_list_node *)n;
2574}2621}
2575 2622
2576__bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)2623__bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
2577{2624{
2578 return __bpf_list_del(head, false);2625 struct list_head *h = (void *)head;
2626
2627 return __bpf_list_del(head, h->next);
2579}2628}
2580 2629
2581__bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)2630__bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
2582{2631{
2583 return __bpf_list_del(head, true);2632 struct list_head *h = (void *)head;
2633
2634 return __bpf_list_del(head, h->prev);
2635}
2636
2637__bpf_kfunc struct bpf_list_node *bpf_list_del(struct bpf_list_head *head,
2638 struct bpf_list_node *node__nonown_allowed)
2639{
2640 struct bpf_list_node_kern *kn = (void *)node__nonown_allowed;
2641
2642 /* verifier guarantees node is a list node rather than list head */
2643 return __bpf_list_del(head, &kn->list_head);
2584}2644}
2585 2645
2586__bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)2646__bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)
@@ -2603,6 +2663,43 @@ __bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head) · 16 unmodified lines
2603 return (struct bpf_list_node *)h->prev;2663 return (struct bpf_list_node *)h->prev;
2604}2664}
2605 2665
2666__bpf_kfunc bool bpf_list_is_first(struct bpf_list_head *head,
2667 struct bpf_list_node *node__nonown_allowed)
2668{
2669 struct list_head *h = (struct list_head *)head;
2670 struct bpf_list_node_kern *kn = (struct bpf_list_node_kern *)node__nonown_allowed;
2671
2672 if (READ_ONCE(kn->owner) != head)
2673 return false;
2674
2675 return list_is_first(&kn->list_head, h);
2676}
2677
2678__bpf_kfunc bool bpf_list_is_last(struct bpf_list_head *head,
2679 struct bpf_list_node *node__nonown_allowed)
2680{
2681 struct list_head *h = (struct list_head *)head;
2682 struct bpf_list_node_kern *kn = (struct bpf_list_node_kern *)node__nonown_allowed;
2683
2684 if (READ_ONCE(kn->owner) != head)
2685 return false;
2686
2687 return list_is_last(&kn->list_head, h);
2688}
2689
2690__bpf_kfunc bool bpf_list_empty(struct bpf_list_head *head)
2691{
2692 struct list_head *h = (struct list_head *)head;
2693
2694 /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
2695 * called on its fields, so init here
2696 */
2697 if (unlikely(!h->next))
2698 INIT_LIST_HEAD(h);
2699
2700 return list_empty(h);
2701}
2702
2606__bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,2703__bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
2607 struct bpf_rb_node *node)2704 struct bpf_rb_node *node)
2608{2705{
@@ -2912,11 +3009,13 @@ __bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid) · 303 unmodified lines
2912{3009{
2913 struct task_struct *p;3010 struct task_struct *p;
2914 3011
2915 rcu_read_lock();3012 guard(rcu)();
3013 if (!task_active_pid_ns(current))
3014 return NULL;
3015
2916 p = find_task_by_vpid(vpid);3016 p = find_task_by_vpid(vpid);
2917 if (p)3017 if (p)
2918 p = bpf_task_acquire(p);3018 p = bpf_task_acquire(p);
2919 rcu_read_unlock();
2920 3019
2921 return p;3020 return p;
2922}3021}
@@ -3072,7 +3171,7 @@ __bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset, · 149 unmodified lines
3072 return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk);3171 return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk);
3073}3172}
3074 3173
3075__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end)3174__bpf_kfunc int bpf_dynptr_adjust(struct bpf_dynptr *p, u64 start, u64 end)
3076{3175{
3077 struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;3176 struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
3078 u64 size;3177 u64 size;
@@ -3093,14 +3192,14 @@ __bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end · 14 unmodified lines
3093 3192
3094__bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)3193__bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)
3095{3194{
3096 struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;3195 const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
3097 3196
3098 return !ptr->data;3197 return !ptr->data;
3099}3198}
3100 3199
3101__bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)3200__bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
3102{3201{
3103 struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;3202 const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
3104 3203
3105 if (!ptr->data)3204 if (!ptr->data)
3106 return false;3205 return false;
@@ -3110,7 +3209,7 @@ __bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p) · 3 unmodified lines
3110 3209
3111__bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p)3210__bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p)
3112{3211{
3113 struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;3212 const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
3114 3213
3115 if (!ptr->data)3214 if (!ptr->data)
3116 return -EINVAL;3215 return -EINVAL;
@@ -3122,7 +3221,7 @@ __bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p, · 5 unmodified lines
3122 struct bpf_dynptr *clone__uninit)3221 struct bpf_dynptr *clone__uninit)
3123{3222{
3124 struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;3223 struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;
3125 struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;3224 const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
3126 3225
3127 if (!ptr->data) {3226 if (!ptr->data) {
3128 bpf_dynptr_set_null(clone);3227 bpf_dynptr_set_null(clone);
@@ -3145,11 +3244,11 @@ __bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p, · 16 unmodified lines
3145 * Copies data from source dynptr to destination dynptr.3244 * Copies data from source dynptr to destination dynptr.
3146 * Returns 0 on success; negative error, otherwise.3245 * Returns 0 on success; negative error, otherwise.
3147 */3246 */
3148__bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off,3247__bpf_kfunc int bpf_dynptr_copy(const struct bpf_dynptr *dst_ptr, u64 dst_off,
3149 struct bpf_dynptr *src_ptr, u64 src_off, u64 size)3248 const struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
3150{3249{
3151 struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;3250 const struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
3152 struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;3251 const struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
3153 void *src_slice, *dst_slice;3252 void *src_slice, *dst_slice;
3154 char buf[256];3253 char buf[256];
3155 u64 off;3254 u64 off;
@@ -3200,9 +3299,9 @@ __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off, · 44 unmodified lines
3200 * at @offset with the constant byte @val.3299 * at @offset with the constant byte @val.
3201 * Returns 0 on success; negative error, otherwise.3300 * Returns 0 on success; negative error, otherwise.
3202 */3301 */
3203__bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val)3302__bpf_kfunc int bpf_dynptr_memset(const struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
3204{3303{
3205 struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;3304 const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
3206 u64 chunk_sz, write_off;3305 u64 chunk_sz, write_off;
3207 char buf[256];3306 char buf[256];
3208 void* slice;3307 void* slice;
@@ -3301,7 +3400,7 @@ __bpf_kfunc void bpf_throw(u64 cookie) · 92 unmodified lines
3301 * which skips compiler generated instrumentation to do the same.3400 * which skips compiler generated instrumentation to do the same.
3302 */3401 */
3303 kasan_unpoison_task_stack_below((void *)(long)ctx.sp);3402 kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
3304 ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);3403 ctx.aux->bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_sp_adjust, ctx.bp, 0, 0);
3305 WARN(1, "A call to BPF exception callback should never return\n");3404 WARN(1, "A call to BPF exception callback should never return\n");
3306}3405}
3307 3406
@@ -4214,13 +4313,13 @@ __bpf_kfunc void bpf_key_put(struct bpf_key *bkey) · 906 unmodified lines
4214 *4313 *
4215 * Return: 0 on success, a negative value on error.4314 * Return: 0 on success, a negative value on error.
4216 */4315 */
4217__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,4316__bpf_kfunc int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p,
4218 struct bpf_dynptr *sig_p,4317 const struct bpf_dynptr *sig_p,
4219 struct bpf_key *trusted_keyring)4318 struct bpf_key *trusted_keyring)
4220{4319{
4221#ifdef CONFIG_SYSTEM_DATA_VERIFICATION4320#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
4222 struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;4321 const struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
4223 struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;4322 const struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
4224 const void *data, *sig;4323 const void *data, *sig;
4225 u32 data_len, sig_len;4324 u32 data_len, sig_len;
4226 int ret;4325 int ret;
@@ -4718,10 +4817,15 @@ BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS) · 491 unmodified lines
4718BTF_ID_FLAGS(func, bpf_list_push_front_impl)4817BTF_ID_FLAGS(func, bpf_list_push_front_impl)
4719BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)4818BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)
4720BTF_ID_FLAGS(func, bpf_list_push_back_impl)4819BTF_ID_FLAGS(func, bpf_list_push_back_impl)
4820BTF_ID_FLAGS(func, bpf_list_add, KF_IMPLICIT_ARGS)
4721BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)4821BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
4722BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)4822BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
4823BTF_ID_FLAGS(func, bpf_list_del, KF_ACQUIRE | KF_RET_NULL)
4723BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)4824BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)
4724BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)4825BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)
4826BTF_ID_FLAGS(func, bpf_list_is_first)
4827BTF_ID_FLAGS(func, bpf_list_is_last)
4828BTF_ID_FLAGS(func, bpf_list_empty)
4725BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)4829BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
4726BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)4830BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)
4727BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)4831BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)
@@ -4862,7 +4966,7 @@ BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS) · 134 unmodified lines
4862BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)4966BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)
4863BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)4967BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
4864BTF_ID_FLAGS(func, bpf_dynptr_from_file)4968BTF_ID_FLAGS(func, bpf_dynptr_from_file)
4865BTF_ID_FLAGS(func, bpf_dynptr_file_discard)4969BTF_ID_FLAGS(func, bpf_dynptr_file_discard, KF_RELEASE)
4866BTF_ID_FLAGS(func, bpf_timer_cancel_async)4970BTF_ID_FLAGS(func, bpf_timer_cancel_async)
4867BTF_KFUNCS_END(common_btf_ids)4971BTF_KFUNCS_END(common_btf_ids)
4868 4972
kernel/bpf/inode.c+4−0
@@ -845,6 +845,10 @@ static void bpf_destroy_inode(struct inode *inode) · 844 unmodified lines
845 simple_xattrs_free(&opts->xa_cache, &bi->xattrs, NULL);845 simple_xattrs_free(&opts->xa_cache, &bi->xattrs, NULL);
846}846}
847 847
848/*
849 * Called after RCU grace period - safe to free inode and anything
850 * that might be accessed by RCU pathwalk (inode fields, i_link).
851 */
848static void bpf_free_inode(struct inode *inode)852static void bpf_free_inode(struct inode *inode)
849{853{
850 if (S_ISLNK(inode->i_mode))854 if (S_ISLNK(inode->i_mode))
kernel/bpf/liveness.c+139−44
@@ -610,6 +610,21 @@ enum arg_track_state { · 609 unmodified lines
610/* Track callee stack slots fp-8 through fp-512 (64 slots of 8 bytes each) */610/* Track callee stack slots fp-8 through fp-512 (64 slots of 8 bytes each) */
611#define MAX_ARG_SPILL_SLOTS 64611#define MAX_ARG_SPILL_SLOTS 64
612 612
613/*
614 * Combined register + stack arg tracking: R0-R10 at indices 0-10,
615 * outgoing stack arg slots at indices MAX_BPF_REG..MAX_BPF_REG+6.
616 */
617#define MAX_AT_TRACK_REGS (MAX_BPF_REG + MAX_STACK_ARG_SLOTS)
618
619static int stack_arg_off_to_slot(s16 off)
620{
621 int aoff = off < 0 ? -off : off;
622
623 if (aoff / 8 > MAX_STACK_ARG_SLOTS)
624 return -1;
625 return aoff / 8 - 1;
626}
627
613static bool arg_is_visited(const struct arg_track *at)628static bool arg_is_visited(const struct arg_track *at)
614{629{
615 return at->frame != ARG_UNVISITED;630 return at->frame != ARG_UNVISITED;
@@ -791,7 +806,9 @@ static bool arg_track_join(struct bpf_verifier_env *env, int idx, int target, in · 175 unmodified lines
791 return true;806 return true;
792 807
793 verbose(env, "arg JOIN insn %d -> %d ", idx, target);808 verbose(env, "arg JOIN insn %d -> %d ", idx, target);
794 if (r >= 0)809 if (r >= MAX_BPF_REG)
810 verbose(env, "sa%d: ", r - MAX_BPF_REG);
811 else if (r >= 0)
795 verbose(env, "r%d: ", r);812 verbose(env, "r%d: ", r);
796 else813 else
797 verbose(env, "fp%+d: ", r * 8);814 verbose(env, "fp%+d: ", r * 8);
@@ -1032,6 +1049,21 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i · 234 unmodified lines
1032 verbose(env, "\tr%d: ", i); verbose_arg_track(env, &at_in[i]);1049 verbose(env, "\tr%d: ", i); verbose_arg_track(env, &at_in[i]);
1033 verbose(env, " -> "); verbose_arg_track(env, &at_out[i]);1050 verbose(env, " -> "); verbose_arg_track(env, &at_out[i]);
1034 }1051 }
1052 /* Log outgoing stack arg slot transitions at indices MAX_BPF_REG..MAX_AT_TRACK_REGS-1 */
1053 for (i = 0; i < MAX_STACK_ARG_SLOTS; i++) {
1054 int ai = MAX_BPF_REG + i;
1055
1056 if (arg_track_eq(&at_out[ai], &at_in[ai]))
1057 continue;
1058 if (!printed) {
1059 verbose(env, "%3d: ", idx);
1060 bpf_verbose_insn(env, insn);
1061 bpf_vlog_reset(&env->log, env->log.end_pos - 1);
1062 printed = true;
1063 }
1064 verbose(env, "\tsa%d: ", i); verbose_arg_track(env, &at_in[ai]);
1065 verbose(env, " -> "); verbose_arg_track(env, &at_out[ai]);
1066 }
1035 for (i = 0; i < MAX_ARG_SPILL_SLOTS; i++) {1067 for (i = 0; i < MAX_ARG_SPILL_SLOTS; i++) {
1036 if (arg_track_eq(&at_stack_out[i], &at_stack_in[i]))1068 if (arg_track_eq(&at_stack_out[i], &at_stack_in[i]))
1037 continue;1069 continue;
@@ -1062,6 +1094,7 @@ static bool can_be_local_fp(int depth, int regno, struct arg_track *at) · 24 unmodified lines
1062static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,1094static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
1063 int insn_idx,1095 int insn_idx,
1064 struct arg_track *at_out, struct arg_track *at_stack_out,1096 struct arg_track *at_out, struct arg_track *at_stack_out,
1097 const struct arg_track *at_stack_arg_entry,
1065 struct func_instance *instance,1098 struct func_instance *instance,
1066 u32 *callsites)1099 u32 *callsites)
1067{1100{
@@ -1071,9 +1104,21 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn, · 3 unmodified lines
1071 struct arg_track *dst = &at_out[insn->dst_reg];1104 struct arg_track *dst = &at_out[insn->dst_reg];
1072 struct arg_track *src = &at_out[insn->src_reg];1105 struct arg_track *src = &at_out[insn->src_reg];
1073 struct arg_track none = { .frame = ARG_NONE };1106 struct arg_track none = { .frame = ARG_NONE };
1074 int r;1107 int r, slot;
1075 1108
1076 if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_K) {1109 /* Handle stack arg stores and loads. */
1110 if (is_stack_arg_st(insn) || is_stack_arg_stx(insn)) {
1111 slot = stack_arg_off_to_slot(insn->off);
1112 if (slot >= 0) {
1113 if (is_stack_arg_stx(insn))
1114 at_out[MAX_BPF_REG + slot] = at_out[insn->src_reg];
1115 else
1116 at_out[MAX_BPF_REG + slot] = none;
1117 }
1118 } else if (is_stack_arg_ldx(insn)) {
1119 slot = stack_arg_off_to_slot(insn->off);
1120 at_out[insn->dst_reg] = (slot >= 0) ? at_stack_arg_entry[slot] : none;
1121 } else if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_K) {
1077 if (code == BPF_MOV) {1122 if (code == BPF_MOV) {
1078 *dst = none;1123 *dst = none;
1079 } else if (dst->frame >= 0) {1124 } else if (dst->frame >= 0) {
@@ -1297,6 +1342,16 @@ static int record_load_store_access(struct bpf_verifier_env *env, · 217 unmodified lines
1297 struct arg_track resolved, *ptr;1342 struct arg_track resolved, *ptr;
1298 int oi;1343 int oi;
1299 1344
1345 /*
1346 * Stack arg insns use dst_reg/src_reg=BPF_REG_PARAMS(11). Since at[]
1347 * is extended to MAX_AT_TRACK_REGS, at[11] holds the arg_track for
1348 * outgoing stack arg slot 0 — not the pointer used for the memory
1349 * access. Skip so the slot's tracked value isn't confused with the
1350 * base register that record_stack_access() expects.
1351 */
1352 if (is_stack_arg_stx(insn) || is_stack_arg_st(insn) || is_stack_arg_ldx(insn))
1353 return 0;
1354
1300 switch (class) {1355 switch (class) {
1301 case BPF_LDX:1356 case BPF_LDX:
1302 ptr = &at[insn->src_reg];1357 ptr = &at[insn->src_reg];
@@ -1343,6 +1398,42 @@ static int record_load_store_access(struct bpf_verifier_env *env, · 40 unmodified lines
1343 return 0;1398 return 0;
1344}1399}
1345 1400
1401static int record_arg_access(struct bpf_verifier_env *env,
1402 struct func_instance *instance,
1403 struct bpf_insn *insn,
1404 struct arg_track *at, int arg_idx,
1405 int insn_idx)
1406{
1407 int depth = instance->depth;
1408 int frame = at->frame;
1409 int err = 0;
1410 s64 bytes;
1411
1412 if (!arg_is_fp(at))
1413 return 0;
1414
1415 if (bpf_helper_call(insn)) {
1416 bytes = bpf_helper_stack_access_bytes(env, insn, arg_idx, insn_idx);
1417 } else if (bpf_pseudo_kfunc_call(insn)) {
1418 bytes = bpf_kfunc_stack_access_bytes(env, insn, arg_idx, insn_idx);
1419 } else {
1420 for (int f = 0; f <= depth; f++) {
1421 err = mark_stack_read(instance, f, insn_idx, SPIS_ALL);
1422 if (err)
1423 return err;
1424 }
1425 return 0;
1426 }
1427 if (bytes == 0)
1428 return 0;
1429
1430 if (frame >= 0 && frame <= depth)
1431 err = record_stack_access(instance, at, bytes, frame, insn_idx);
1432 else if (frame == ARG_IMPRECISE)
1433 err = record_imprecise(instance, at->mask, insn_idx);
1434 return err;
1435}
1436
1346/* Record stack access for a given 'at' state of helper/kfunc 'insn' */1437/* Record stack access for a given 'at' state of helper/kfunc 'insn' */
1347static int record_call_access(struct bpf_verifier_env *env,1438static int record_call_access(struct bpf_verifier_env *env,
1348 struct func_instance *instance,1439 struct func_instance *instance,
@@ -1350,9 +1441,8 @@ static int record_call_access(struct bpf_verifier_env *env, · 1 unmodified line
1350 int insn_idx)1441 int insn_idx)
1351{1442{
1352 struct bpf_insn *insn = &env->prog->insnsi[insn_idx];1443 struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
1353 int depth = instance->depth;
1354 struct bpf_call_summary cs;1444 struct bpf_call_summary cs;
1355 int r, err = 0, num_params = 5;1445 int r, err, num_params = 5;
1356 1446
1357 if (bpf_pseudo_call(insn))1447 if (bpf_pseudo_call(insn))
1358 return 0;1448 return 0;
@@ -1360,32 +1450,15 @@ static int record_call_access(struct bpf_verifier_env *env, · 1 unmodified line
1360 if (bpf_get_call_summary(env, insn, &cs))1450 if (bpf_get_call_summary(env, insn, &cs))
1361 num_params = cs.num_params;1451 num_params = cs.num_params;
1362 1452
1363 for (r = BPF_REG_1; r < BPF_REG_1 + num_params; r++) {1453 for (r = BPF_REG_1; r < BPF_REG_1 + min(num_params, MAX_BPF_FUNC_REG_ARGS); r++) {
1364 int frame = at[r].frame;1454 err = record_arg_access(env, instance, insn, &at[r], r - 1, insn_idx);
1365 s64 bytes;1455 if (err)
1366 1456 return err;
1367 if (!arg_is_fp(&at[r]))1457 }
1368 continue;
1369
1370 if (bpf_helper_call(insn)) {
1371 bytes = bpf_helper_stack_access_bytes(env, insn, r - 1, insn_idx);
1372 } else if (bpf_pseudo_kfunc_call(insn)) {
1373 bytes = bpf_kfunc_stack_access_bytes(env, insn, r - 1, insn_idx);
1374 } else {
1375 for (int f = 0; f <= depth; f++) {
1376 err = mark_stack_read(instance, f, insn_idx, SPIS_ALL);
1377 if (err)
1378 return err;
1379 }
1380 return 0;
1381 }
1382 if (bytes == 0)
1383 continue;
1384 1458
1385 if (frame >= 0 && frame <= depth)1459 for (r = 0; r < MAX_STACK_ARG_SLOTS && r < num_params - MAX_BPF_FUNC_REG_ARGS; r++) {
1386 err = record_stack_access(instance, &at[r], bytes, frame, insn_idx);1460 err = record_arg_access(env, instance, insn, &at[MAX_BPF_REG + r],
1387 else if (frame == ARG_IMPRECISE)1461 r + MAX_BPF_FUNC_REG_ARGS, insn_idx);
1388 err = record_imprecise(instance, at[r].mask, insn_idx);
1389 if (err)1462 if (err)
1390 return err;1463 return err;
1391 }1464 }
@@ -1445,7 +1518,7 @@ static int find_callback_subprog(struct bpf_verifier_env *env, · 53 unmodified lines
1445 1518
1446/* Per-subprog intermediate state kept alive across analysis phases */1519/* Per-subprog intermediate state kept alive across analysis phases */
1447struct subprog_at_info {1520struct subprog_at_info {
1448 struct arg_track (*at_in)[MAX_BPF_REG];1521 struct arg_track (*at_in)[MAX_AT_TRACK_REGS];
1449 int len;1522 int len;
1450};1523};
1451 1524
@@ -1479,6 +1552,9 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, · 27 unmodified lines
1479 for (r = 0; r < MAX_BPF_REG - 1; r++)1552 for (r = 0; r < MAX_BPF_REG - 1; r++)
1480 if (arg_is_fp(&info->at_in[i][r]))1553 if (arg_is_fp(&info->at_in[i][r]))
1481 has_extra = true;1554 has_extra = true;
1555 for (r = 0; r < MAX_STACK_ARG_SLOTS; r++)
1556 if (arg_is_fp(&info->at_in[i][MAX_BPF_REG + r]))
1557 has_extra = true;
1482 }1558 }
1483 if (is_ldx_stx_call) {1559 if (is_ldx_stx_call) {
1484 for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++)1560 for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++)
@@ -1503,6 +1579,12 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, · 18 unmodified lines
1503 verbose(env, " r%d=", r);1579 verbose(env, " r%d=", r);
1504 verbose_arg_track(env, &info->at_in[i][r]);1580 verbose_arg_track(env, &info->at_in[i][r]);
1505 }1581 }
1582 for (r = 0; r < MAX_STACK_ARG_SLOTS; r++) {
1583 if (!arg_is_fp(&info->at_in[i][MAX_BPF_REG + r]))
1584 continue;
1585 verbose(env, " sa%d=", r);
1586 verbose_arg_track(env, &info->at_in[i][MAX_BPF_REG + r]);
1587 }
1506 }1588 }
1507 1589
1508 if (is_ldx_stx_call) {1590 if (is_ldx_stx_call) {
@@ -1525,7 +1607,7 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, · 16 unmodified lines
1525 * Runs forward fixed-point with arg_track_xfer(), then records1607 * Runs forward fixed-point with arg_track_xfer(), then records
1526 * memory accesses in a single linear pass over converged state.1608 * memory accesses in a single linear pass over converged state.
1527 *1609 *
1528 * @callee_entry: pre-populated entry state for R1-R51610 * @callee_entry: pre-populated entry state for R1-R5 and stack args
1529 * NULL for main (subprog 0).1611 * NULL for main (subprog 0).
1530 * @info: stores at_in, len for debug printing.1612 * @info: stores at_in, len for debug printing.
1531 */1613 */
@@ -1543,10 +1625,11 @@ static int compute_subprog_args(struct bpf_verifier_env *env, · 11 unmodified lines
1543 int end = env->subprog_info[subprog + 1].start;1625 int end = env->subprog_info[subprog + 1].start;
1544 int po_end = env->subprog_info[subprog + 1].postorder_start;1626 int po_end = env->subprog_info[subprog + 1].postorder_start;
1545 int len = end - start;1627 int len = end - start;
1546 struct arg_track (*at_in)[MAX_BPF_REG] = NULL;1628 struct arg_track (*at_in)[MAX_AT_TRACK_REGS] = NULL;
1547 struct arg_track at_out[MAX_BPF_REG];1629 struct arg_track at_out[MAX_AT_TRACK_REGS];
1548 struct arg_track (*at_stack_in)[MAX_ARG_SPILL_SLOTS] = NULL;1630 struct arg_track (*at_stack_in)[MAX_ARG_SPILL_SLOTS] = NULL;
1549 struct arg_track *at_stack_out = NULL;1631 struct arg_track *at_stack_out = NULL;
1632 struct arg_track at_stack_arg_entry[MAX_STACK_ARG_SLOTS];
1550 struct arg_track unvisited = { .frame = ARG_UNVISITED };1633 struct arg_track unvisited = { .frame = ARG_UNVISITED };
1551 struct arg_track none = { .frame = ARG_NONE };1634 struct arg_track none = { .frame = ARG_NONE };
1552 bool changed;1635 bool changed;
@@ -1565,13 +1648,13 @@ static int compute_subprog_args(struct bpf_verifier_env *env, · 12 unmodified lines
1565 goto err_free;1648 goto err_free;
1566 1649
1567 for (i = 0; i < len; i++) {1650 for (i = 0; i < len; i++) {
1568 for (r = 0; r < MAX_BPF_REG; r++)1651 for (r = 0; r < MAX_AT_TRACK_REGS; r++)
1569 at_in[i][r] = unvisited;1652 at_in[i][r] = unvisited;
1570 for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++)1653 for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++)
1571 at_stack_in[i][r] = unvisited;1654 at_stack_in[i][r] = unvisited;
1572 }1655 }
1573 1656
1574 for (r = 0; r < MAX_BPF_REG; r++)1657 for (r = 0; r < MAX_AT_TRACK_REGS; r++)
1575 at_in[0][r] = none;1658 at_in[0][r] = none;
1576 1659
1577 /* Entry: R10 is always precisely the current frame's FP */1660 /* Entry: R10 is always precisely the current frame's FP */
@@ -1587,6 +1670,10 @@ static int compute_subprog_args(struct bpf_verifier_env *env, · 9 unmodified lines
1587 for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++)1670 for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++)
1588 at_stack_in[0][r] = none;1671 at_stack_in[0][r] = none;
1589 1672
1673 /* Entry: incoming stack args from caller, or ARG_NONE for main */
1674 for (r = 0; r < MAX_STACK_ARG_SLOTS; r++)
1675 at_stack_arg_entry[r] = callee_entry ? callee_entry[MAX_BPF_REG + r] : none;
1676
1590 if (env->log.level & BPF_LOG_LEVEL2)1677 if (env->log.level & BPF_LOG_LEVEL2)
1591 verbose(env, "subprog#%d: analyzing (depth %d)...\n", subprog, depth);1678 verbose(env, "subprog#%d: analyzing (depth %d)...\n", subprog, depth);
1592 1679
@@ -1605,7 +1692,8 @@ redo: · 12 unmodified lines
1605 memcpy(at_out, at_in[i], sizeof(at_out));1692 memcpy(at_out, at_in[i], sizeof(at_out));
1606 memcpy(at_stack_out, at_stack_in[i], MAX_ARG_SPILL_SLOTS * sizeof(*at_stack_out));1693 memcpy(at_stack_out, at_stack_in[i], MAX_ARG_SPILL_SLOTS * sizeof(*at_stack_out));
1607 1694
1608 arg_track_xfer(env, insn, idx, at_out, at_stack_out, instance, callsites);1695 arg_track_xfer(env, insn, idx, at_out, at_stack_out,
1696 at_stack_arg_entry, instance, callsites);
1609 arg_track_log(env, insn, idx, at_in[i], at_stack_in[i], at_out, at_stack_out);1697 arg_track_log(env, insn, idx, at_in[i], at_stack_in[i], at_out, at_stack_out);
1610 1698
1611 /* Propagate to successors within this subprogram */1699 /* Propagate to successors within this subprogram */
@@ -1619,7 +1707,7 @@ redo: · 7 unmodified lines
1619 continue;1707 continue;
1620 ti = target - start;1708 ti = target - start;
1621 1709
1622 for (r = 0; r < MAX_BPF_REG; r++)1710 for (r = 0; r < MAX_AT_TRACK_REGS; r++)
1623 changed |= arg_track_join(env, idx, target, r,1711 changed |= arg_track_join(env, idx, target, r,
1624 &at_in[ti][r], at_out[r]);1712 &at_in[ti][r], at_out[r]);
1625 1713
@@ -1674,11 +1762,14 @@ err_free: · 48 unmodified lines
1674 return err;1762 return err;
1675}1763}
1676 1764
1677/* Return true if any of R1-R5 is derived from a frame pointer. */1765/* Return true if any of R1-R5 or stack args is derived from a frame pointer. */
1678static bool has_fp_args(struct arg_track *args)1766static bool has_fp_args(struct arg_track *args)
1679{1767{
1680 for (int r = BPF_REG_1; r <= BPF_REG_5; r++)1768 for (int r = BPF_REG_1; r <= BPF_REG_5; r++)
1681 if (args[r].frame != ARG_NONE)1769 if (arg_is_fp(&args[r]))
1770 return true;
1771 for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++)
1772 if (arg_is_fp(&args[MAX_BPF_REG + r]))
1682 return true;1773 return true;
1683 return false;1774 return false;
1684}1775}
@@ -1803,7 +1894,7 @@ static int analyze_subprog(struct bpf_verifier_env *env, · 118 unmodified lines
1803 /* For each reachable call site in the subprog, recurse into callees */1894 /* For each reachable call site in the subprog, recurse into callees */
1804 for (int p = po_start; p < po_end; p++) {1895 for (int p = po_start; p < po_end; p++) {
1805 int idx = env->cfg.insn_postorder[p];1896 int idx = env->cfg.insn_postorder[p];
1806 struct arg_track callee_args[BPF_REG_5 + 1];1897 struct arg_track callee_args[MAX_AT_TRACK_REGS] = {};
1807 struct arg_track none = { .frame = ARG_NONE };1898 struct arg_track none = { .frame = ARG_NONE };
1808 struct bpf_insn *insn = &insns[idx];1899 struct bpf_insn *insn = &insns[idx];
1809 struct func_instance *callee_instance;1900 struct func_instance *callee_instance;
@@ -1818,9 +1909,11 @@ static int analyze_subprog(struct bpf_verifier_env *env, · 8 unmodified lines
1818 if (callee < 0)1909 if (callee < 0)
1819 continue;1910 continue;
1820 1911
1821 /* Build entry args: R1-R5 from at_in at call site */1912 /* Build entry args: R1-R5 and stack args from at_in at call site */
1822 for (int r = BPF_REG_1; r <= BPF_REG_5; r++)1913 for (int r = BPF_REG_1; r <= BPF_REG_5; r++)
1823 callee_args[r] = info[subprog].at_in[j][r];1914 callee_args[r] = info[subprog].at_in[j][r];
1915 for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++)
1916 callee_args[MAX_BPF_REG + r] = info[subprog].at_in[j][MAX_BPF_REG + r];
1824 } else if (bpf_calls_callback(env, idx)) {1917 } else if (bpf_calls_callback(env, idx)) {
1825 callee = find_callback_subprog(env, insn, idx, &caller_reg, &cb_callee_reg);1918 callee = find_callback_subprog(env, insn, idx, &caller_reg, &cb_callee_reg);
1826 if (callee == -2) {1919 if (callee == -2) {
@@ -1842,6 +1935,8 @@ static int analyze_subprog(struct bpf_verifier_env *env, · 15 unmodified lines
1842 1935
1843 for (int r = BPF_REG_1; r <= BPF_REG_5; r++)1936 for (int r = BPF_REG_1; r <= BPF_REG_5; r++)
1844 callee_args[r] = none;1937 callee_args[r] = none;
1938 for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++)
1939 callee_args[MAX_BPF_REG + r] = none;
1845 callee_args[cb_callee_reg] = info[subprog].at_in[j][caller_reg];1940 callee_args[cb_callee_reg] = info[subprog].at_in[j][caller_reg];
1846 } else {1941 } else {
1847 continue;1942 continue;
@@ -2085,7 +2180,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, · 237 unmodified lines
2085 def = ALL_CALLER_SAVED_REGS;2180 def = ALL_CALLER_SAVED_REGS;
2086 use = def & ~BIT(BPF_REG_0);2181 use = def & ~BIT(BPF_REG_0);
2087 if (bpf_get_call_summary(env, insn, &cs))2182 if (bpf_get_call_summary(env, insn, &cs))
2088 use = GENMASK(cs.num_params, 1);2183 use = GENMASK(min_t(u8, cs.num_params, MAX_BPF_FUNC_REG_ARGS), 1);
2089 break;2184 break;
2090 default:2185 default:
2091 def = 0;2186 def = 0;
kernel/bpf/log.c+104−28
@@ -13,17 +13,17 @@ · 12 unmodified lines
13 13
14#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)14#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
15 15
16static bool bpf_verifier_log_attr_valid(const struct bpf_verifier_log *log)16static bool bpf_verifier_log_attr_valid(u32 log_level, char __user *log_buf, u32 log_size)
17{17{
18 /* ubuf and len_total should both be specified (or not) together */18 /* ubuf and len_total should both be specified (or not) together */
19 if (!!log->ubuf != !!log->len_total)19 if (!!log_buf != !!log_size)
20 return false;20 return false;
21 /* log buf without log_level is meaningless */21 /* log buf without log_level is meaningless */
22 if (log->ubuf && log->level == 0)22 if (log_buf && log_level == 0)
23 return false;23 return false;
24 if (log->level & ~BPF_LOG_MASK)24 if (log_level & ~BPF_LOG_MASK)
25 return false;25 return false;
26 if (log->len_total > UINT_MAX >> 2)26 if (log_size > UINT_MAX >> 2)
27 return false;27 return false;
28 return true;28 return true;
29}29}
@@ -36,7 +36,7 @@ int bpf_vlog_init(struct bpf_verifier_log *log, u32 log_level, · 6 unmodified lines
36 log->len_total = log_size;36 log->len_total = log_size;
37 37
38 /* log attributes have to be sane */38 /* log attributes have to be sane */
39 if (!bpf_verifier_log_attr_valid(log))39 if (!bpf_verifier_log_attr_valid(log_level, log_buf, log_size))
40 return -EINVAL;40 return -EINVAL;
41 41
42 return 0;42 return 0;
@@ -571,20 +571,20 @@ static void print_scalar_ranges(struct bpf_verifier_env *env, · 528 unmodified lines
571 u64 val;571 u64 val;
572 bool omit;572 bool omit;
573 } minmaxs[] = {573 } minmaxs[] = {
574 {"smin", reg->smin_value, reg->smin_value == S64_MIN},574 {"smin", reg_smin(reg), reg_smin(reg) == S64_MIN},
575 {"smax", reg->smax_value, reg->smax_value == S64_MAX},575 {"smax", reg_smax(reg), reg_smax(reg) == S64_MAX},
576 {"umin", reg->umin_value, reg->umin_value == 0},576 {"umin", reg_umin(reg), reg_umin(reg) == 0},
577 {"umax", reg->umax_value, reg->umax_value == U64_MAX},577 {"umax", reg_umax(reg), reg_umax(reg) == U64_MAX},
578 {"smin32",578 {"smin32",
579 is_snum_decimal((s64)reg->s32_min_value)579 is_snum_decimal((s64)reg_s32_min(reg))
580 ? (s64)reg->s32_min_value580 ? (s64)reg_s32_min(reg)
581 : (u32)reg->s32_min_value, reg->s32_min_value == S32_MIN},581 : (u32)reg_s32_min(reg), reg_s32_min(reg) == S32_MIN},
582 {"smax32",582 {"smax32",
583 is_snum_decimal((s64)reg->s32_max_value)583 is_snum_decimal((s64)reg_s32_max(reg))
584 ? (s64)reg->s32_max_value584 ? (s64)reg_s32_max(reg)
585 : (u32)reg->s32_max_value, reg->s32_max_value == S32_MAX},585 : (u32)reg_s32_max(reg), reg_s32_max(reg) == S32_MAX},
586 {"umin32", reg->u32_min_value, reg->u32_min_value == 0},586 {"umin32", reg_u32_min(reg), reg_u32_min(reg) == 0},
587 {"umax32", reg->u32_max_value, reg->u32_max_value == U32_MAX},587 {"umax32", reg_u32_max(reg), reg_u32_max(reg) == U32_MAX},
588 }, *m1, *m2, *mend = &minmaxs[ARRAY_SIZE(minmaxs)];588 }, *m1, *m2, *mend = &minmaxs[ARRAY_SIZE(minmaxs)];
589 bool neg1, neg2;589 bool neg1, neg2;
590 590
@@ -665,8 +665,8 @@ static void print_reg_state(struct bpf_verifier_env *env, · 74 unmodified lines
665 verbose_a("id=%d", reg->id & ~BPF_ADD_CONST);665 verbose_a("id=%d", reg->id & ~BPF_ADD_CONST);
666 if (reg->id & BPF_ADD_CONST)666 if (reg->id & BPF_ADD_CONST)
667 verbose(env, "%+d", reg->delta);667 verbose(env, "%+d", reg->delta);
668 if (reg->ref_obj_id)668 if (reg->parent_id)
669 verbose_a("ref_obj_id=%d", reg->ref_obj_id);669 verbose_a("parent_id=%d", reg->parent_id);
670 if (type_is_non_owning_ref(reg->type))670 if (type_is_non_owning_ref(reg->type))
671 verbose_a("%s", "non_own_ref");671 verbose_a("%s", "non_own_ref");
672 if (type_is_map_ptr(t)) {672 if (type_is_map_ptr(t)) {
@@ -768,21 +768,19 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie · 95 unmodified lines
768 verbose(env, "=dynptr_%s(", dynptr_type_str(reg->dynptr.type));768 verbose(env, "=dynptr_%s(", dynptr_type_str(reg->dynptr.type));
769 if (reg->id)769 if (reg->id)
770 verbose_a("id=%d", reg->id);770 verbose_a("id=%d", reg->id);
771 if (reg->ref_obj_id)771 if (reg->parent_id)
772 verbose_a("ref_id=%d", reg->ref_obj_id);772 verbose_a("parent_id=%d", reg->parent_id);
773 if (reg->dynptr_id)
774 verbose_a("dynptr_id=%d", reg->dynptr_id);
775 verbose(env, ")");773 verbose(env, ")");
776 break;774 break;
777 case STACK_ITER:775 case STACK_ITER:
778 /* only main slot has ref_obj_id set; skip others */776 /* only main slot has id set; skip others */
779 if (!reg->ref_obj_id)777 if (!reg->id)
780 continue;778 continue;
781 779
782 verbose(env, " fp%d=iter_%s(ref_id=%d,state=%s,depth=%u)",780 verbose(env, " fp%d=iter_%s(id=%d,state=%s,depth=%u)",
783 (-i - 1) * BPF_REG_SIZE,781 (-i - 1) * BPF_REG_SIZE,
784 iter_type_str(reg->iter.btf, reg->iter.btf_id),782 iter_type_str(reg->iter.btf, reg->iter.btf_id),
785 reg->ref_obj_id, iter_state_str(reg->iter.state),783 reg->id, iter_state_str(reg->iter.state),
786 reg->iter.depth);784 reg->iter.depth);
787 break;785 break;
788 case STACK_MISC:786 case STACK_MISC:
@@ -825,3 +823,81 @@ void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_st · 36 unmodified lines
825 }823 }
826 print_verifier_state(env, vstate, frameno, false);824 print_verifier_state(env, vstate, frameno, false);
827}825}
826
827int bpf_log_attr_init(struct bpf_log_attr *log, u64 log_buf, u32 log_size, u32 log_level,
828 u32 offsetof_log_true_size, bpfptr_t uattr, struct bpf_common_attr *common,
829 bpfptr_t uattr_common, u32 size_common)
830{
831 char __user *ubuf_common = u64_to_user_ptr(common->log_buf);
832 char __user *ubuf = u64_to_user_ptr(log_buf);
833
834 if (!bpf_verifier_log_attr_valid(common->log_level, ubuf_common, common->log_size) ||
835 !bpf_verifier_log_attr_valid(log_level, ubuf, log_size))
836 return -EINVAL;
837
838 if (ubuf && ubuf_common && (ubuf != ubuf_common || log_size != common->log_size ||
839 log_level != common->log_level))
840 return -EINVAL;
841
842 memset(log, 0, sizeof(*log));
843 log->ubuf = ubuf;
844 log->size = log_size;
845 log->level = log_level;
846 log->offsetof_true_size = offsetof_log_true_size;
847 log->uattr = uattr;
848
849 if (!ubuf && ubuf_common) {
850 log->ubuf = ubuf_common;
851 log->size = common->log_size;
852 log->level = common->log_level;
853 log->uattr = uattr_common;
854 log->offsetof_true_size = 0;
855 if (size_common >= offsetofend(struct bpf_common_attr, log_true_size))
856 log->offsetof_true_size = offsetof(struct bpf_common_attr, log_true_size);
857 }
858 return 0;
859}
860
861struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log,
862 struct bpf_common_attr *common, bpfptr_t uattr,
863 u32 size)
864{
865 struct bpf_verifier_log *log;
866 int err;
867
868 memset(attr_log, 0, sizeof(*attr_log));
869 attr_log->uattr = uattr;
870 if (size >= offsetofend(struct bpf_common_attr, log_true_size))
871 attr_log->offsetof_true_size = offsetof(struct bpf_common_attr, log_true_size);
872
873 if (!size)
874 return NULL;
875
876 log = kzalloc_obj(*log, GFP_KERNEL);
877 if (!log)
878 return ERR_PTR(-ENOMEM);
879
880 err = bpf_vlog_init(log, common->log_level, u64_to_user_ptr(common->log_buf),
881 common->log_size);
882 if (err) {
883 kfree(log);
884 return ERR_PTR(err);
885 }
886
887 return log;
888}
889
890int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log)
891{
892 u32 log_true_size;
893 int err;
894
895 err = bpf_vlog_finalize(log, &log_true_size);
896
897 if (attr->offsetof_true_size &&
898 copy_to_bpfptr_offset(attr->uattr, attr->offsetof_true_size, &log_true_size,
899 sizeof(log_true_size)))
900 return -EFAULT;
901
902 return err;
903}
kernel/bpf/lpm_trie.c+4−4
@@ -246,7 +246,7 @@ static void *trie_lookup_elem(struct bpf_map *map, void *_key) · 245 unmodified lines
246 246
247 /* Start walking the trie from the root node ... */247 /* Start walking the trie from the root node ... */
248 248
249 for (node = rcu_dereference_check(trie->root, rcu_read_lock_bh_held());249 for (node = rcu_dereference_check(trie->root, bpf_rcu_lock_held());
250 node;) {250 node;) {
251 unsigned int next_bit;251 unsigned int next_bit;
252 size_t matchlen;252 size_t matchlen;
@@ -280,7 +280,7 @@ static void *trie_lookup_elem(struct bpf_map *map, void *_key) · 27 unmodified lines
280 */280 */
281 next_bit = extract_bit(key->data, node->prefixlen);281 next_bit = extract_bit(key->data, node->prefixlen);
282 node = rcu_dereference_check(node->child[next_bit],282 node = rcu_dereference_check(node->child[next_bit],
283 rcu_read_lock_bh_held());283 bpf_rcu_lock_held());
284 }284 }
285 285
286 if (!found)286 if (!found)
@@ -359,7 +359,7 @@ static long trie_update_elem(struct bpf_map *map, · 72 unmodified lines
359 */359 */
360 slot = &trie->root;360 slot = &trie->root;
361 361
362 while ((node = rcu_dereference(*slot))) {362 while ((node = rcu_dereference_protected(*slot, 1))) {
363 matchlen = longest_prefix_match(trie, node, key);363 matchlen = longest_prefix_match(trie, node, key);
364 364
365 if (node->prefixlen != matchlen ||365 if (node->prefixlen != matchlen ||
@@ -482,7 +482,7 @@ static long trie_delete_elem(struct bpf_map *map, void *_key) · 116 unmodified lines
482 trim = &trie->root;482 trim = &trie->root;
483 trim2 = trim;483 trim2 = trim;
484 parent = NULL;484 parent = NULL;
485 while ((node = rcu_dereference(*trim))) {485 while ((node = rcu_dereference_protected(*trim, 1))) {
486 matchlen = longest_prefix_match(trie, node, key);486 matchlen = longest_prefix_match(trie, node, key);
487 487
488 if (node->prefixlen != matchlen ||488 if (node->prefixlen != matchlen ||
kernel/bpf/map_in_map.c+4−1
@@ -20,7 +20,8 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) · 19 unmodified lines
20 /* Does not support >1 level map-in-map */20 /* Does not support >1 level map-in-map */
21 if (inner_map->inner_map_meta)21 if (inner_map->inner_map_meta)
22 return ERR_PTR(-EINVAL);22 return ERR_PTR(-EINVAL);
23 23 if (inner_map->excl_prog_sha)
24 return ERR_PTR(-ENOTSUPP);
24 if (!inner_map->ops->map_meta_equal)25 if (!inner_map->ops->map_meta_equal)
25 return ERR_PTR(-ENOTSUPP);26 return ERR_PTR(-ENOTSUPP);
26 27
@@ -101,6 +102,8 @@ void *bpf_map_fd_get_ptr(struct bpf_map *map, · 74 unmodified lines
101 inner_map = __bpf_map_get(f);102 inner_map = __bpf_map_get(f);
102 if (IS_ERR(inner_map))103 if (IS_ERR(inner_map))
103 return inner_map;104 return inner_map;
105 if (inner_map->excl_prog_sha)
106 return ERR_PTR(-ENOTSUPP);
104 107
105 inner_map_meta = map->inner_map_meta;108 inner_map_meta = map->inner_map_meta;
106 if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map))109 if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map))
kernel/bpf/map_iter.c+6−1
@@ -112,6 +112,10 @@ static int bpf_iter_attach_map(struct bpf_prog *prog, · 111 unmodified lines
112 map = bpf_map_get_with_uref(linfo->map.map_fd);112 map = bpf_map_get_with_uref(linfo->map.map_fd);
113 if (IS_ERR(map))113 if (IS_ERR(map))
114 return PTR_ERR(map);114 return PTR_ERR(map);
115 if (map->excl_prog_sha) {
116 err = -EPERM;
117 goto put_map;
118 }
115 119
116 if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||120 if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
117 map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||121 map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
@@ -119,7 +123,8 @@ static int bpf_iter_attach_map(struct bpf_prog *prog, · 1 unmodified line
119 is_percpu = true;123 is_percpu = true;
120 else if (map->map_type != BPF_MAP_TYPE_HASH &&124 else if (map->map_type != BPF_MAP_TYPE_HASH &&
121 map->map_type != BPF_MAP_TYPE_LRU_HASH &&125 map->map_type != BPF_MAP_TYPE_LRU_HASH &&
122 map->map_type != BPF_MAP_TYPE_ARRAY)126 map->map_type != BPF_MAP_TYPE_ARRAY &&
127 map->map_type != BPF_MAP_TYPE_RHASH)
123 goto put_map;128 goto put_map;
124 129
125 key_acc_size = prog->aux->max_rdonly_access;130 key_acc_size = prog->aux->max_rdonly_access;
kernel/bpf/stackmap.c+198−17
@@ -9,6 +9,7 @@ · 8 unmodified lines
9#include <linux/perf_event.h>9#include <linux/perf_event.h>
10#include <linux/btf_ids.h>10#include <linux/btf_ids.h>
11#include <linux/buildid.h>11#include <linux/buildid.h>
12#include <linux/mmap_lock.h>
12#include "percpu_freelist.h"13#include "percpu_freelist.h"
13#include "mmap_unlock_work.h"14#include "mmap_unlock_work.h"
14 15
@@ -152,6 +153,180 @@ static int fetch_build_id(struct vm_area_struct *vma, unsigned char *build_id, b · 137 unmodified lines
152 : build_id_parse_nofault(vma, build_id, NULL);153 : build_id_parse_nofault(vma, build_id, NULL);
153}154}
154 155
156static inline void stack_map_build_id_set_ip(struct bpf_stack_build_id *id)
157{
158 id->status = BPF_STACK_BUILD_ID_IP;
159 memset(id->build_id, 0, BUILD_ID_SIZE_MAX);
160}
161
162static inline u64 stack_map_build_id_offset(unsigned long vm_pgoff,
163 unsigned long vm_start, u64 ip)
164{
165 return (vm_pgoff << PAGE_SHIFT) + ip - vm_start;
166}
167
168static inline void stack_map_build_id_set_valid(struct bpf_stack_build_id *id,
169 u64 offset,
170 const unsigned char *build_id)
171{
172 id->status = BPF_STACK_BUILD_ID_VALID;
173 id->offset = offset;
174 if (id->build_id != build_id)
175 memcpy(id->build_id, build_id, BUILD_ID_SIZE_MAX);
176}
177
178struct stack_map_vma_lock {
179 struct vm_area_struct *vma;
180 struct mm_struct *mm;
181};
182
183/*
184 * Acquire a stable read-side reference on the VMA covering @ip.
185 *
186 * With CONFIG_PER_VMA_LOCK=y this returns a VMA with its per-VMA read
187 * lock held and mmap_lock dropped, so the caller may sleep.
188 *
189 * With CONFIG_PER_VMA_LOCK=n it returns a VMA with mmap_lock still
190 * held; the caller must snapshot any fields it needs and pin vm_file
191 * with get_file() before stack_map_unlock_vma() drops mmap_lock, as
192 * the VMA may be split, merged, or freed after that.
193 *
194 * Returns NULL on failure, in which case no lock is held.
195 */
196static struct vm_area_struct *
197stack_map_lock_vma(struct stack_map_vma_lock *lock, unsigned long ip)
198{
199 struct mm_struct *mm = lock->mm;
200 struct vm_area_struct *vma;
201
202 /* noop under !CONFIG_PER_VMA_LOCK */
203 vma = lock_vma_under_rcu(mm, ip);
204 if (vma) {
205 lock->vma = vma;
206 return vma;
207 }
208
209 /*
210 * Taking mmap_read_lock() is unsafe here, because the caller BPF
211 * program might already hold it, causing a deadlock.
212 */
213 if (!mmap_read_trylock(mm))
214 return NULL;
215
216 vma = vma_lookup(mm, ip);
217 if (!vma) {
218 mmap_read_unlock(mm);
219 return NULL;
220 }
221
222#ifdef CONFIG_PER_VMA_LOCK
223 if (!vma_start_read_locked(vma)) {
224 mmap_read_unlock(mm);
225 return NULL;
226 }
227 mmap_read_unlock(mm);
228#endif
229
230 lock->vma = vma;
231 return vma;
232}
233
234static void stack_map_unlock_vma(struct stack_map_vma_lock *lock)
235{
236#ifdef CONFIG_PER_VMA_LOCK
237 vma_end_read(lock->vma);
238#else
239 mmap_read_unlock(lock->mm);
240#endif
241 lock->vma = NULL;
242}
243
244static void stack_map_get_build_id_offset_sleepable(struct bpf_stack_build_id *id_offs,
245 u32 trace_nr)
246{
247 struct mm_struct *mm = current->mm;
248 struct stack_map_vma_lock lock = { .mm = mm };
249 struct {
250 struct file *file;
251 const unsigned char *build_id;
252 unsigned long vm_start;
253 unsigned long vm_end;
254 unsigned long vm_pgoff;
255 } cache = {};
256 unsigned long vm_pgoff, vm_start, vm_end;
257 struct vm_area_struct *vma;
258 struct file *file;
259 u64 offset;
260 u64 ip;
261
262 for (u32 i = 0; i < trace_nr; i++) {
263 ip = READ_ONCE(id_offs[i].ip);
264
265 /*
266 * Range cache fast path: if ip falls within the previously
267 * resolved VMA range, reuse the cache build_id without
268 * re-acquiring the VMA lock.
269 */
270 if (cache.build_id && ip >= cache.vm_start && ip < cache.vm_end) {
271 offset = stack_map_build_id_offset(cache.vm_pgoff, cache.vm_start, ip);
272 stack_map_build_id_set_valid(&id_offs[i], offset, cache.build_id);
273 continue;
274 }
275
276 vma = stack_map_lock_vma(&lock, ip);
277 if (!vma) {
278 stack_map_build_id_set_ip(&id_offs[i]);
279 continue;
280 }
281 if (vma_is_anonymous(vma) || !vma->vm_file) {
282 stack_map_build_id_set_ip(&id_offs[i]);
283 stack_map_unlock_vma(&lock);
284 continue;
285 }
286
287 file = vma->vm_file;
288 vm_pgoff = vma->vm_pgoff;
289 vm_start = vma->vm_start;
290 vm_end = vma->vm_end;
291 offset = stack_map_build_id_offset(vm_pgoff, vm_start, ip);
292
293 /*
294 * Same backing file as previous (e.g. different VMAs
295 * of the same ELF binary). Reuse the cache build_id.
296 */
297 if (file == cache.file) {
298 stack_map_unlock_vma(&lock);
299 stack_map_build_id_set_valid(&id_offs[i], offset, cache.build_id);
300 cache.vm_start = vm_start;
301 cache.vm_end = vm_end;
302 cache.vm_pgoff = vm_pgoff;
303 continue;
304 }
305
306 file = get_file(file);
307 stack_map_unlock_vma(&lock);
308
309 /* build_id_parse_file() may block on filesystem reads */
310 if (build_id_parse_file(file, id_offs[i].build_id, NULL)) {
311 stack_map_build_id_set_ip(&id_offs[i]);
312 fput(file);
313 continue;
314 }
315
316 stack_map_build_id_set_valid(&id_offs[i], offset, id_offs[i].build_id);
317 if (cache.file)
318 fput(cache.file);
319 cache.file = file;
320 cache.build_id = id_offs[i].build_id;
321 cache.vm_start = vm_start;
322 cache.vm_end = vm_end;
323 cache.vm_pgoff = vm_pgoff;
324 }
325
326 if (cache.file)
327 fput(cache.file);
328}
329
155/*330/*
156 * Expects all id_offs[i].ip values to be set to correct initial IPs.331 * Expects all id_offs[i].ip values to be set to correct initial IPs.
157 * They will be subsequently:332 * They will be subsequently:
@@ -165,44 +340,50 @@ static int fetch_build_id(struct vm_area_struct *vma, unsigned char *build_id, b · 7 unmodified lines
165static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,340static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
166 u32 trace_nr, bool user, bool may_fault)341 u32 trace_nr, bool user, bool may_fault)
167{342{
168 int i;
169 struct mmap_unlock_irq_work *work = NULL;343 struct mmap_unlock_irq_work *work = NULL;
170 bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work);344 bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work);
345 bool has_user_ctx = user && current && current->mm;
171 struct vm_area_struct *vma, *prev_vma = NULL;346 struct vm_area_struct *vma, *prev_vma = NULL;
172 const char *prev_build_id;347 const unsigned char *prev_build_id = NULL;
348 int i;
349
350 if (may_fault && has_user_ctx) {
351 stack_map_get_build_id_offset_sleepable(id_offs, trace_nr);
352 return;
353 }
173 354
174 /* If the irq_work is in use, fall back to report ips. Same355 /* If the irq_work is in use, fall back to report ips. Same
175 * fallback is used for kernel stack (!user) on a stackmap with356 * fallback is used for kernel stack (!user) on a stackmap with
176 * build_id.357 * build_id.
177 */358 */
178 if (!user || !current || !current->mm || irq_work_busy ||359 if (!has_user_ctx || irq_work_busy || !mmap_read_trylock(current->mm)) {
179 !mmap_read_trylock(current->mm)) {
180 /* cannot access current->mm, fall back to ips */360 /* cannot access current->mm, fall back to ips */
181 for (i = 0; i < trace_nr; i++) {361 for (i = 0; i < trace_nr; i++)
182 id_offs[i].status = BPF_STACK_BUILD_ID_IP;362 stack_map_build_id_set_ip(&id_offs[i]);
183 memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX);
184 }
185 return;363 return;
186 }364 }
187 365
188 for (i = 0; i < trace_nr; i++) {366 for (i = 0; i < trace_nr; i++) {
189 u64 ip = READ_ONCE(id_offs[i].ip);367 u64 ip = READ_ONCE(id_offs[i].ip);
368 u64 offset;
190 369
191 if (range_in_vma(prev_vma, ip, ip)) {370 if (prev_build_id && range_in_vma(prev_vma, ip, ip)) {
192 vma = prev_vma;371 vma = prev_vma;
193 memcpy(id_offs[i].build_id, prev_build_id, BUILD_ID_SIZE_MAX);372 offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip);
194 goto build_id_valid;373 stack_map_build_id_set_valid(&id_offs[i], offset, prev_build_id);
374 continue;
195 }375 }
196 vma = find_vma(current->mm, ip);376 vma = find_vma(current->mm, ip);
197 if (!vma || fetch_build_id(vma, id_offs[i].build_id, may_fault)) {377 if (!vma || vma_is_anonymous(vma) ||
378 fetch_build_id(vma, id_offs[i].build_id, may_fault)) {
198 /* per entry fall back to ips */379 /* per entry fall back to ips */
199 id_offs[i].status = BPF_STACK_BUILD_ID_IP;380 stack_map_build_id_set_ip(&id_offs[i]);
200 memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX);381 prev_vma = vma;
382 prev_build_id = NULL;
201 continue;383 continue;
202 }384 }
203build_id_valid:385 offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip);
204 id_offs[i].offset = (vma->vm_pgoff << PAGE_SHIFT) + ip - vma->vm_start;386 stack_map_build_id_set_valid(&id_offs[i], offset, id_offs[i].build_id);
205 id_offs[i].status = BPF_STACK_BUILD_ID_VALID;
206 prev_vma = vma;387 prev_vma = vma;
207 prev_build_id = id_offs[i].build_id;388 prev_build_id = id_offs[i].build_id;
208 }389 }
kernel/bpf/states.c+51−16
@@ -2,6 +2,7 @@ · 1 unmodified line
2/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */2/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
3#include <linux/bpf.h>3#include <linux/bpf.h>
4#include <linux/bpf_verifier.h>4#include <linux/bpf_verifier.h>
5#include <linux/cnum.h>
5#include <linux/filter.h>6#include <linux/filter.h>
6 7
7#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)8#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -301,14 +302,8 @@ int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_s · 293 unmodified lines
301static bool range_within(const struct bpf_reg_state *old,302static bool range_within(const struct bpf_reg_state *old,
302 const struct bpf_reg_state *cur)303 const struct bpf_reg_state *cur)
303{304{
304 return old->umin_value <= cur->umin_value &&305 return cnum64_is_subset(old->r64, cur->r64) &&
305 old->umax_value >= cur->umax_value &&306 cnum32_is_subset(old->r32, cur->r32);
306 old->smin_value <= cur->smin_value &&
307 old->smax_value >= cur->smax_value &&
308 old->u32_min_value <= cur->u32_min_value &&
309 old->u32_max_value >= cur->u32_max_value &&
310 old->s32_min_value <= cur->s32_min_value &&
311 old->s32_max_value >= cur->s32_max_value;
312}307}
313 308
314/* If in the old state two registers had the same id, then they need to have309/* If in the old state two registers had the same id, then they need to have
@@ -348,8 +343,12 @@ static bool check_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap) · 33 unmodified lines
348 return true;343 return true;
349 }344 }
350 345
351 /* We ran out of idmap slots, which should be impossible */346 /*
352 WARN_ON_ONCE(1);347 * idmap slots are bounded by the number of registers and stack slots.
348 * Since referenced dynptrs acquire intermediate references that do
349 * not live in either, so the map can be exhausted. Since it is unlikely,
350 * fail the verification by treating the states as not equivalent.
351 */
353 return false;352 return false;
354}353}
355 354
@@ -494,7 +493,7 @@ static bool regs_exact(const struct bpf_reg_state *rold, · 138 unmodified lines
494{493{
495 return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&494 return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&
496 check_ids(rold->id, rcur->id, idmap) &&495 check_ids(rold->id, rcur->id, idmap) &&
497 check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap);496 check_ids(rold->parent_id, rcur->parent_id, idmap);
498}497}
499 498
500enum exact_level {499enum exact_level {
@@ -619,7 +618,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, · 118 unmodified lines
619 range_within(rold, rcur) &&618 range_within(rold, rcur) &&
620 tnum_in(rold->var_off, rcur->var_off) &&619 tnum_in(rold->var_off, rcur->var_off) &&
621 check_ids(rold->id, rcur->id, idmap) &&620 check_ids(rold->id, rcur->id, idmap) &&
622 check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap);621 check_ids(rold->parent_id, rcur->parent_id, idmap);
623 case PTR_TO_PACKET_META:622 case PTR_TO_PACKET_META:
624 case PTR_TO_PACKET:623 case PTR_TO_PACKET:
625 /* We must have at least as much range as the old ptr624 /* We must have at least as much range as the old ptr
@@ -799,7 +798,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, · 173 unmodified lines
799 cur_reg = &cur->stack[spi].spilled_ptr;798 cur_reg = &cur->stack[spi].spilled_ptr;
800 if (old_reg->dynptr.type != cur_reg->dynptr.type ||799 if (old_reg->dynptr.type != cur_reg->dynptr.type ||
801 old_reg->dynptr.first_slot != cur_reg->dynptr.first_slot ||800 old_reg->dynptr.first_slot != cur_reg->dynptr.first_slot ||
802 !check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap))801 !check_ids(old_reg->id, cur_reg->id, idmap) ||
802 !check_ids(old_reg->parent_id, cur_reg->parent_id, idmap))
803 return false;803 return false;
804 break;804 break;
805 case STACK_ITER:805 case STACK_ITER:
@@ -815,13 +815,13 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, · 9 unmodified lines
815 old_reg->iter.btf_id != cur_reg->iter.btf_id ||815 old_reg->iter.btf_id != cur_reg->iter.btf_id ||
816 old_reg->iter.state != cur_reg->iter.state ||816 old_reg->iter.state != cur_reg->iter.state ||
817 /* ignore {old_reg,cur_reg}->iter.depth, see above */817 /* ignore {old_reg,cur_reg}->iter.depth, see above */
818 !check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap))818 !check_ids(old_reg->id, cur_reg->id, idmap))
819 return false;819 return false;
820 break;820 break;
821 case STACK_IRQ_FLAG:821 case STACK_IRQ_FLAG:
822 old_reg = &old->stack[spi].spilled_ptr;822 old_reg = &old->stack[spi].spilled_ptr;
823 cur_reg = &cur->stack[spi].spilled_ptr;823 cur_reg = &cur->stack[spi].spilled_ptr;
824 if (!check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap) ||824 if (!check_ids(old_reg->id, cur_reg->id, idmap) ||
825 old_reg->irq.kfunc_class != cur_reg->irq.kfunc_class)825 old_reg->irq.kfunc_class != cur_reg->irq.kfunc_class)
826 return false;826 return false;
827 break;827 break;
@@ -838,6 +838,32 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, · 10 unmodified lines
838 return true;838 return true;
839}839}
840 840
841/*
842 * Compare stack arg slots between old and current states.
843 * Outgoing stack args are path-local state and must agree for pruning.
844 */
845static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
846 struct bpf_func_state *cur, struct bpf_idmap *idmap,
847 enum exact_level exact)
848{
849 int i, nslots;
850
851 nslots = max(old->out_stack_arg_cnt, cur->out_stack_arg_cnt);
852 for (i = 0; i < nslots; i++) {
853 struct bpf_reg_state *old_arg, *cur_arg;
854 struct bpf_reg_state not_init = { .type = NOT_INIT };
855
856 old_arg = i < old->out_stack_arg_cnt ?
857 &old->stack_arg_regs[i] : &not_init;
858 cur_arg = i < cur->out_stack_arg_cnt ?
859 &cur->stack_arg_regs[i] : &not_init;
860 if (!regsafe(env, old_arg, cur_arg, idmap, exact))
861 return false;
862 }
863
864 return true;
865}
866
841static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *cur,867static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *cur,
842 struct bpf_idmap *idmap)868 struct bpf_idmap *idmap)
843{869{
@@ -868,6 +894,9 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c · 24 unmodified lines
868 return false;894 return false;
869 switch (old->refs[i].type) {895 switch (old->refs[i].type) {
870 case REF_TYPE_PTR:896 case REF_TYPE_PTR:
897 if (!check_ids(old->refs[i].parent_id, cur->refs[i].parent_id, idmap))
898 return false;
899 break;
871 case REF_TYPE_IRQ:900 case REF_TYPE_IRQ:
872 break;901 break;
873 case REF_TYPE_LOCK:902 case REF_TYPE_LOCK:
@@ -920,6 +949,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat · 46 unmodified lines
920 if (old->callback_depth > cur->callback_depth)949 if (old->callback_depth > cur->callback_depth)
921 return false;950 return false;
922 951
952 if (!old->no_stack_arg_load && cur->no_stack_arg_load)
953 return false;
954
923 for (i = 0; i < MAX_BPF_REG; i++)955 for (i = 0; i < MAX_BPF_REG; i++)
924 if (((1 << i) & live_regs) &&956 if (((1 << i) & live_regs) &&
925 !regsafe(env, &old->regs[i], &cur->regs[i],957 !regsafe(env, &old->regs[i], &cur->regs[i],
@@ -929,6 +961,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat · 3 unmodified lines
929 if (!stacksafe(env, old, cur, &env->idmap_scratch, exact))961 if (!stacksafe(env, old, cur, &env->idmap_scratch, exact))
930 return false;962 return false;
931 963
964 if (!stack_arg_safe(env, old, cur, &env->idmap_scratch, exact))
965 return false;
966
932 return true;967 return true;
933}968}
934 969
@@ -1376,7 +1411,7 @@ hit: · 441 unmodified lines
1376 */1411 */
1377 err = 0;1412 err = 0;
1378 if (bpf_is_jmp_point(env, env->insn_idx))1413 if (bpf_is_jmp_point(env, env->insn_idx))
1379 err = bpf_push_jmp_history(env, cur, 0, 0);1414 err = bpf_push_jmp_history(env, cur, 0, 0, 0, 0);
1380 err = err ? : propagate_precision(env, &sl->state, cur, NULL);1415 err = err ? : propagate_precision(env, &sl->state, cur, NULL);
1381 if (err)1416 if (err)
1382 return err;1417 return err;
kernel/bpf/syscall.c+231−81
@@ -41,6 +41,7 @@ · 40 unmodified lines
41#include <linux/overflow.h>41#include <linux/overflow.h>
42#include <linux/cookie.h>42#include <linux/cookie.h>
43#include <linux/verification.h>43#include <linux/verification.h>
44#include <linux/btf_ids.h>
44 45
45#include <net/netfilter/nf_bpf_link.h>46#include <net/netfilter/nf_bpf_link.h>
46#include <net/netkit.h>47#include <net/netkit.h>
@@ -807,6 +808,11 @@ void bpf_obj_free_task_work(const struct btf_record *rec, void *obj) · 760 unmodified lines
807 bpf_task_work_cancel_and_free(obj + rec->task_work_off);808 bpf_task_work_cancel_and_free(obj + rec->task_work_off);
808}809}
809 810
811void bpf_obj_cancel_fields(struct bpf_map *map, void *obj)
812{
813 bpf_map_free_internal_structs(map, obj);
814}
815
810void bpf_obj_free_fields(const struct btf_record *rec, void *obj)816void bpf_obj_free_fields(const struct btf_record *rec, void *obj)
811{817{
812 const struct btf_field *fields;818 const struct btf_field *fields;
@@ -1280,6 +1286,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, · 467 unmodified lines
1280 case BPF_SPIN_LOCK:1286 case BPF_SPIN_LOCK:
1281 case BPF_RES_SPIN_LOCK:1287 case BPF_RES_SPIN_LOCK:
1282 if (map->map_type != BPF_MAP_TYPE_HASH &&1288 if (map->map_type != BPF_MAP_TYPE_HASH &&
1289 map->map_type != BPF_MAP_TYPE_RHASH &&
1283 map->map_type != BPF_MAP_TYPE_ARRAY &&1290 map->map_type != BPF_MAP_TYPE_ARRAY &&
1284 map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE &&1291 map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE &&
1285 map->map_type != BPF_MAP_TYPE_SK_STORAGE &&1292 map->map_type != BPF_MAP_TYPE_SK_STORAGE &&
@@ -1294,6 +1301,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, · 8 unmodified lines
1294 case BPF_WORKQUEUE:1301 case BPF_WORKQUEUE:
1295 case BPF_TASK_WORK:1302 case BPF_TASK_WORK:
1296 if (map->map_type != BPF_MAP_TYPE_HASH &&1303 if (map->map_type != BPF_MAP_TYPE_HASH &&
1304 map->map_type != BPF_MAP_TYPE_RHASH &&
1297 map->map_type != BPF_MAP_TYPE_LRU_HASH &&1305 map->map_type != BPF_MAP_TYPE_LRU_HASH &&
1298 map->map_type != BPF_MAP_TYPE_ARRAY) {1306 map->map_type != BPF_MAP_TYPE_ARRAY) {
1299 ret = -EOPNOTSUPP;1307 ret = -EOPNOTSUPP;
@@ -1305,6 +1313,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, · 5 unmodified lines
1305 case BPF_KPTR_PERCPU:1313 case BPF_KPTR_PERCPU:
1306 case BPF_REFCOUNT:1314 case BPF_REFCOUNT:
1307 if (map->map_type != BPF_MAP_TYPE_HASH &&1315 if (map->map_type != BPF_MAP_TYPE_HASH &&
1316 map->map_type != BPF_MAP_TYPE_RHASH &&
1308 map->map_type != BPF_MAP_TYPE_PERCPU_HASH &&1317 map->map_type != BPF_MAP_TYPE_PERCPU_HASH &&
1309 map->map_type != BPF_MAP_TYPE_LRU_HASH &&1318 map->map_type != BPF_MAP_TYPE_LRU_HASH &&
1310 map->map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH &&1319 map->map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH &&
@@ -1359,7 +1368,8 @@ free_map_tab: · 48 unmodified lines
1359 1368
1360#define BPF_MAP_CREATE_LAST_FIELD excl_prog_hash_size1369#define BPF_MAP_CREATE_LAST_FIELD excl_prog_hash_size
1361/* called via syscall */1370/* called via syscall */
1362static int map_create(union bpf_attr *attr, bpfptr_t uattr)1371static int map_create_alloc(union bpf_attr *attr, bpfptr_t uattr, struct bpf_verifier_log *log,
1372 struct bpf_map **mapp, struct bpf_token **tokenp)
1363{1373{
1364 const struct bpf_map_ops *ops;1374 const struct bpf_map_ops *ops;
1365 struct bpf_token *token = NULL;1375 struct bpf_token *token = NULL;
@@ -1367,12 +1377,13 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 1 unmodified line
1367 u32 map_type = attr->map_type;1377 u32 map_type = attr->map_type;
1368 struct bpf_map *map;1378 struct bpf_map *map;
1369 bool token_flag;1379 bool token_flag;
1370 int f_flags;
1371 int err;1380 int err;
1372 1381
1373 err = CHECK_ATTR(BPF_MAP_CREATE);1382 err = CHECK_ATTR(BPF_MAP_CREATE);
1374 if (err)1383 if (err) {
1384 bpf_log(log, "Invalid attr.\n");
1375 return -EINVAL;1385 return -EINVAL;
1386 }
1376 1387
1377 /* check BPF_F_TOKEN_FD flag, remember if it's set, and then clear it1388 /* check BPF_F_TOKEN_FD flag, remember if it's set, and then clear it
1378 * to avoid per-map type checks tripping on unknown flag1389 * to avoid per-map type checks tripping on unknown flag
@@ -1381,31 +1392,40 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 2 unmodified lines
1381 attr->map_flags &= ~BPF_F_TOKEN_FD;1392 attr->map_flags &= ~BPF_F_TOKEN_FD;
1382 1393
1383 if (attr->btf_vmlinux_value_type_id) {1394 if (attr->btf_vmlinux_value_type_id) {
1384 if (attr->map_type != BPF_MAP_TYPE_STRUCT_OPS ||1395 if (attr->map_type != BPF_MAP_TYPE_STRUCT_OPS) {
1385 attr->btf_key_type_id || attr->btf_value_type_id)1396 bpf_log(log, "btf_vmlinux_value_type_id can only be used with struct_ops maps.\n");
1386 return -EINVAL;1397 return -EINVAL;
1398 }
1399 if (attr->btf_key_type_id || attr->btf_value_type_id) {
1400 bpf_log(log, "btf_vmlinux_value_type_id is mutually exclusive with btf_key_type_id and btf_value_type_id.\n");
1401 return -EINVAL;
1402 }
1387 } else if (attr->btf_key_type_id && !attr->btf_value_type_id) {1403 } else if (attr->btf_key_type_id && !attr->btf_value_type_id) {
1404 bpf_log(log, "Invalid btf_value_type_id.\n");
1388 return -EINVAL;1405 return -EINVAL;
1389 }1406 }
1390 1407
1391 if (attr->map_type != BPF_MAP_TYPE_BLOOM_FILTER &&1408 if (attr->map_type != BPF_MAP_TYPE_BLOOM_FILTER &&
1392 attr->map_type != BPF_MAP_TYPE_ARENA &&1409 attr->map_type != BPF_MAP_TYPE_ARENA &&
1393 attr->map_extra != 0)1410 attr->map_type != BPF_MAP_TYPE_RHASH &&
1411 attr->map_extra != 0) {
1412 bpf_log(log, "Invalid map_extra.\n");
1394 return -EINVAL;1413 return -EINVAL;
1395 1414 }
1396 f_flags = bpf_get_file_flag(attr->map_flags);
1397 if (f_flags < 0)
1398 return f_flags;
1399 1415
1400 if (numa_node != NUMA_NO_NODE &&1416 if (numa_node != NUMA_NO_NODE &&
1401 ((unsigned int)numa_node >= nr_node_ids ||1417 ((unsigned int)numa_node >= nr_node_ids ||
1402 !node_online(numa_node)))1418 !node_online(numa_node))) {
1419 bpf_log(log, "Invalid numa_node.\n");
1403 return -EINVAL;1420 return -EINVAL;
1421 }
1404 1422
1405 /* find map type and init map: hashtable vs rbtree vs bloom vs ... */1423 /* find map type and init map: hashtable vs rbtree vs bloom vs ... */
1406 map_type = attr->map_type;1424 map_type = attr->map_type;
1407 if (map_type >= ARRAY_SIZE(bpf_map_types))1425 if (map_type >= ARRAY_SIZE(bpf_map_types)) {
1426 bpf_log(log, "Invalid map_type.\n");
1408 return -EINVAL;1427 return -EINVAL;
1428 }
1409 map_type = array_index_nospec(map_type, ARRAY_SIZE(bpf_map_types));1429 map_type = array_index_nospec(map_type, ARRAY_SIZE(bpf_map_types));
1410 ops = bpf_map_types[map_type];1430 ops = bpf_map_types[map_type];
1411 if (!ops)1431 if (!ops)
@@ -1423,8 +1443,10 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 11 unmodified lines
1423 1443
1424 if (token_flag) {1444 if (token_flag) {
1425 token = bpf_token_get_from_fd(attr->map_token_fd);1445 token = bpf_token_get_from_fd(attr->map_token_fd);
1426 if (IS_ERR(token))1446 if (IS_ERR(token)) {
1447 bpf_log(log, "Invalid map_token_fd.\n");
1427 return PTR_ERR(token);1448 return PTR_ERR(token);
1449 }
1428 1450
1429 /* if current token doesn't grant map creation permissions,1451 /* if current token doesn't grant map creation permissions,
1430 * then we can't use this token, so ignore it and rely on1452 * then we can't use this token, so ignore it and rely on
@@ -1457,6 +1479,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 26 unmodified lines
1457 case BPF_MAP_TYPE_CGROUP_ARRAY:1479 case BPF_MAP_TYPE_CGROUP_ARRAY:
1458 case BPF_MAP_TYPE_ARRAY_OF_MAPS:1480 case BPF_MAP_TYPE_ARRAY_OF_MAPS:
1459 case BPF_MAP_TYPE_HASH:1481 case BPF_MAP_TYPE_HASH:
1482 case BPF_MAP_TYPE_RHASH:
1460 case BPF_MAP_TYPE_PERCPU_HASH:1483 case BPF_MAP_TYPE_PERCPU_HASH:
1461 case BPF_MAP_TYPE_HASH_OF_MAPS:1484 case BPF_MAP_TYPE_HASH_OF_MAPS:
1462 case BPF_MAP_TYPE_RINGBUF:1485 case BPF_MAP_TYPE_RINGBUF:
@@ -1507,8 +1530,10 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 44 unmodified lines
1507 1530
1508 err = bpf_obj_name_cpy(map->name, attr->map_name,1531 err = bpf_obj_name_cpy(map->name, attr->map_name,
1509 sizeof(attr->map_name));1532 sizeof(attr->map_name));
1510 if (err < 0)1533 if (err < 0) {
1534 bpf_log(log, "Invalid map_name.\n");
1511 goto free_map;1535 goto free_map;
1536 }
1512 1537
1513 preempt_disable();1538 preempt_disable();
1514 map->cookie = gen_cookie_next(&bpf_map_cookie);1539 map->cookie = gen_cookie_next(&bpf_map_cookie);
@@ -1531,6 +1556,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 16 unmodified lines
1531 1556
1532 btf = btf_get_by_fd(attr->btf_fd);1557 btf = btf_get_by_fd(attr->btf_fd);
1533 if (IS_ERR(btf)) {1558 if (IS_ERR(btf)) {
1559 bpf_log(log, "Invalid btf_fd.\n");
1534 err = PTR_ERR(btf);1560 err = PTR_ERR(btf);
1535 goto free_map;1561 goto free_map;
1536 }1562 }
@@ -1558,6 +1584,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 21 unmodified lines
1558 bpfptr_t uprog_hash = make_bpfptr(attr->excl_prog_hash, uattr.is_kernel);1584 bpfptr_t uprog_hash = make_bpfptr(attr->excl_prog_hash, uattr.is_kernel);
1559 1585
1560 if (attr->excl_prog_hash_size != SHA256_DIGEST_SIZE) {1586 if (attr->excl_prog_hash_size != SHA256_DIGEST_SIZE) {
1587 bpf_log(log, "Invalid excl_prog_hash_size.\n");
1561 err = -EINVAL;1588 err = -EINVAL;
1562 goto free_map;1589 goto free_map;
1563 }1590 }
@@ -1572,11 +1599,62 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 8 unmodified lines
1572 err = -EFAULT;1599 err = -EFAULT;
1573 goto free_map;1600 goto free_map;
1574 }1601 }
1602
1603 /* See libbpf: emit_signature_match() */
1604 BUILD_BUG_ON(offsetof(struct bpf_map, excl) != SHA256_DIGEST_SIZE);
1605 BUILD_BUG_ON(!__same_type(map->excl, u32));
1606 BUILD_BUG_ON(offsetof(struct bpf_map, sha) != 0);
1607 BUILD_BUG_ON(!__same_type(map->sha, u8[SHA256_DIGEST_SIZE]));
1608 map->excl = 1;
1575 } else if (attr->excl_prog_hash_size) {1609 } else if (attr->excl_prog_hash_size) {
1610 bpf_log(log, "Invalid excl_prog_hash_size.\n");
1576 err = -EINVAL;1611 err = -EINVAL;
1577 goto free_map;1612 goto free_map;
1578 }1613 }
1579 1614
1615 *mapp = map;
1616 *tokenp = token;
1617 return 0;
1618
1619free_map:
1620 bpf_map_free(map);
1621put_token:
1622 bpf_token_put(token);
1623 return err;
1624}
1625
1626static int map_create(union bpf_attr *attr, bpfptr_t uattr, struct bpf_common_attr *attr_common,
1627 bpfptr_t uattr_common, u32 size_common)
1628{
1629 struct bpf_token *token = NULL;
1630 struct bpf_verifier_log *log;
1631 struct bpf_log_attr attr_log;
1632 struct bpf_map *map = NULL;
1633 int err, ret;
1634 int f_flags;
1635
1636 log = bpf_log_attr_create_vlog(&attr_log, attr_common, uattr_common, size_common);
1637 if (IS_ERR(log))
1638 return PTR_ERR(log);
1639
1640 err = map_create_alloc(attr, uattr, log, &map, &token);
1641
1642 /* preserve original error even if log finalization is successful */
1643 ret = bpf_log_attr_finalize(&attr_log, log);
1644 if (ret)
1645 err = ret;
1646
1647 kfree(log);
1648
1649 if (err)
1650 goto free_map;
1651
1652 f_flags = bpf_get_file_flag(attr->map_flags);
1653 if (f_flags < 0) {
1654 err = f_flags;
1655 goto free_map;
1656 }
1657
1580 err = security_bpf_map_create(map, attr, token, uattr.is_kernel);1658 err = security_bpf_map_create(map, attr, token, uattr.is_kernel);
1581 if (err)1659 if (err)
1582 goto free_map_sec;1660 goto free_map_sec;
@@ -1605,8 +1683,8 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 22 unmodified lines
1605free_map_sec:1683free_map_sec:
1606 security_bpf_map_free(map);1684 security_bpf_map_free(map);
1607free_map:1685free_map:
1608 bpf_map_free(map);1686 if (map)
1609put_token:1687 bpf_map_free(map);
1610 bpf_token_put(token);1688 bpf_token_put(token);
1611 return err;1689 return err;
1612}1690}
@@ -2192,6 +2270,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) · 579 unmodified lines
2192 map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||2270 map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
2193 map->map_type == BPF_MAP_TYPE_LRU_HASH ||2271 map->map_type == BPF_MAP_TYPE_LRU_HASH ||
2194 map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||2272 map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
2273 map->map_type == BPF_MAP_TYPE_RHASH ||
2195 map->map_type == BPF_MAP_TYPE_STACK_TRACE) {2274 map->map_type == BPF_MAP_TYPE_STACK_TRACE) {
2196 if (!bpf_map_is_offloaded(map)) {2275 if (!bpf_map_is_offloaded(map)) {
2197 bpf_disable_instrumentation();2276 bpf_disable_instrumentation();
@@ -2646,7 +2725,8 @@ static int · 448 unmodified lines
2646bpf_prog_load_check_attach(enum bpf_prog_type prog_type,2725bpf_prog_load_check_attach(enum bpf_prog_type prog_type,
2647 enum bpf_attach_type expected_attach_type,2726 enum bpf_attach_type expected_attach_type,
2648 struct btf *attach_btf, u32 btf_id,2727 struct btf *attach_btf, u32 btf_id,
2649 struct bpf_prog *dst_prog)2728 struct bpf_prog *dst_prog,
2729 bool multi_func)
2650{2730{
2651 if (btf_id) {2731 if (btf_id) {
2652 if (btf_id > BTF_MAX_TYPE)2732 if (btf_id > BTF_MAX_TYPE)
@@ -2666,6 +2746,14 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type, · 13 unmodified lines
2666 }2746 }
2667 }2747 }
2668 2748
2749 if (multi_func) {
2750 if (prog_type != BPF_PROG_TYPE_TRACING)
2751 return -EINVAL;
2752 if (!attach_btf || btf_id)
2753 return -EINVAL;
2754 return 0;
2755 }
2756
2669 if (attach_btf && (!btf_id || dst_prog))2757 if (attach_btf && (!btf_id || dst_prog))
2670 return -EINVAL;2758 return -EINVAL;
2671 2759
@@ -2798,8 +2886,22 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type) · 126 unmodified lines
2798 }2886 }
2799}2887}
2800 2888
2889static enum bpf_sig_keyring bpf_classify_keyring(s32 keyring_id)
2890{
2891 switch (keyring_id) {
2892 case 0:
2893 return BPF_SIG_KEYRING_BUILTIN;
2894 case (s32)(unsigned long)VERIFY_USE_SECONDARY_KEYRING:
2895 return BPF_SIG_KEYRING_SECONDARY;
2896 case (s32)(unsigned long)VERIFY_USE_PLATFORM_KEYRING:
2897 return BPF_SIG_KEYRING_PLATFORM;
2898 default:
2899 return BPF_SIG_KEYRING_USER;
2900 }
2901}
2902
2801static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr,2903static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr,
2802 bool is_kernel)2904 bool is_kernel, s32 *keyring_serial)
2803{2905{
2804 bpfptr_t usig = make_bpfptr(attr->signature, is_kernel);2906 bpfptr_t usig = make_bpfptr(attr->signature, is_kernel);
2805 struct bpf_dynptr_kern sig_ptr, insns_ptr;2907 struct bpf_dynptr_kern sig_ptr, insns_ptr;
@@ -2835,7 +2937,8 @@ static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr · 29 unmodified lines
2835 2937
2836 err = bpf_verify_pkcs7_signature((struct bpf_dynptr *)&insns_ptr,2938 err = bpf_verify_pkcs7_signature((struct bpf_dynptr *)&insns_ptr,
2837 (struct bpf_dynptr *)&sig_ptr, key);2939 (struct bpf_dynptr *)&sig_ptr, key);
2838 2940 if (!err)
2941 *keyring_serial = bpf_key_serial(key);
2839 bpf_key_put(key);2942 bpf_key_put(key);
2840 kvfree(sig);2943 kvfree(sig);
2841 return err;2944 return err;
@@ -2858,10 +2961,15 @@ static int bpf_prog_mark_insn_arrays_ready(struct bpf_prog *prog) · 16 unmodified lines
2858 return 0;2961 return 0;
2859}2962}
2860 2963
2964extern int bpf_multi_func(void);
2965int __init __used bpf_multi_func(void) { return 0; }
2966
2967BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
2968
2861/* last field in 'union bpf_attr' used by this command */2969/* last field in 'union bpf_attr' used by this command */
2862#define BPF_PROG_LOAD_LAST_FIELD keyring_id2970#define BPF_PROG_LOAD_LAST_FIELD keyring_id
2863 2971
2864static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)2972static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
2865{2973{
2866 enum bpf_prog_type type = attr->prog_type;2974 enum bpf_prog_type type = attr->prog_type;
2867 struct bpf_prog *prog, *dst_prog = NULL;2975 struct bpf_prog *prog, *dst_prog = NULL;
@@ -2870,6 +2978,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 2 unmodified lines
2870 bool bpf_cap;2978 bool bpf_cap;
2871 int err;2979 int err;
2872 char license[128];2980 char license[128];
2981 bool multi_func;
2873 2982
2874 if (CHECK_ATTR(BPF_PROG_LOAD))2983 if (CHECK_ATTR(BPF_PROG_LOAD))
2875 return -EINVAL;2984 return -EINVAL;
@@ -2936,6 +3045,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 60 unmodified lines
2936 if (is_perfmon_prog_type(type) && !bpf_token_capable(token, CAP_PERFMON))3045 if (is_perfmon_prog_type(type) && !bpf_token_capable(token, CAP_PERFMON))
2937 goto put_token;3046 goto put_token;
2938 3047
3048 multi_func = is_tracing_multi(attr->expected_attach_type);
3049
2939 /* attach_prog_fd/attach_btf_obj_fd can specify fd of either bpf_prog3050 /* attach_prog_fd/attach_btf_obj_fd can specify fd of either bpf_prog
2940 * or btf, we need to check which one it is3051 * or btf, we need to check which one it is
2941 */3052 */
@@ -2957,7 +3068,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 15 unmodified lines
2957 goto put_token;3068 goto put_token;
2958 }3069 }
2959 }3070 }
2960 } else if (attr->attach_btf_id) {3071 } else if (attr->attach_btf_id || multi_func) {
2961 /* fall back to vmlinux BTF, if BTF type ID is specified */3072 /* fall back to vmlinux BTF, if BTF type ID is specified */
2962 attach_btf = bpf_get_btf_vmlinux();3073 attach_btf = bpf_get_btf_vmlinux();
2963 if (IS_ERR(attach_btf)) {3074 if (IS_ERR(attach_btf)) {
@@ -2973,7 +3084,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 9 unmodified lines
2973 3084
2974 if (bpf_prog_load_check_attach(type, attr->expected_attach_type,3085 if (bpf_prog_load_check_attach(type, attr->expected_attach_type,
2975 attach_btf, attr->attach_btf_id,3086 attach_btf, attr->attach_btf_id,
2976 dst_prog)) {3087 dst_prog, multi_func)) {
2977 if (dst_prog)3088 if (dst_prog)
2978 bpf_prog_put(dst_prog);3089 bpf_prog_put(dst_prog);
2979 if (attach_btf)3090 if (attach_btf)
@@ -2996,7 +3107,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 16 unmodified lines
2996 prog->expected_attach_type = attr->expected_attach_type;3107 prog->expected_attach_type = attr->expected_attach_type;
2997 prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE);3108 prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE);
2998 prog->aux->attach_btf = attach_btf;3109 prog->aux->attach_btf = attach_btf;
2999 prog->aux->attach_btf_id = attr->attach_btf_id;3110 prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id;
3000 prog->aux->dst_prog = dst_prog;3111 prog->aux->dst_prog = dst_prog;
3001 prog->aux->dev_bound = !!attr->prog_ifindex;3112 prog->aux->dev_bound = !!attr->prog_ifindex;
3002 prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS;3113 prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS;
@@ -3022,13 +3133,17 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 19 unmodified lines
3022 3133
3023 /* eBPF programs must be GPL compatible to use GPL-ed functions */3134 /* eBPF programs must be GPL compatible to use GPL-ed functions */
3024 prog->gpl_compatible = license_is_gpl_compatible(license) ? 1 : 0;3135 prog->gpl_compatible = license_is_gpl_compatible(license) ? 1 : 0;
3025
3026 if (attr->signature) {3136 if (attr->signature) {
3027 err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel);3137 err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel,
3138 &prog->aux->sig.keyring_serial);
3028 if (err)3139 if (err)
3029 goto free_prog;3140 goto free_prog;
3141 prog->aux->sig.keyring_type = bpf_classify_keyring(attr->keyring_id);
3142 prog->aux->sig.verdict = BPF_SIG_VERIFIED;
3143 } else {
3144 prog->aux->sig.keyring_type = BPF_SIG_KEYRING_NONE;
3145 prog->aux->sig.verdict = BPF_SIG_UNSIGNED;
3030 }3146 }
3031
3032 prog->orig_prog = NULL;3147 prog->orig_prog = NULL;
3033 prog->jited = 0;3148 prog->jited = 0;
3034 3149
@@ -3076,10 +3191,10 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 41 unmodified lines
3076 3191
3077 err = security_bpf_prog_load(prog, attr, token, uattr.is_kernel);3192 err = security_bpf_prog_load(prog, attr, token, uattr.is_kernel);
3078 if (err)3193 if (err)
3079 goto free_prog_sec;3194 goto free_prog;
3080 3195
3081 /* run eBPF verifier */3196 /* run eBPF verifier */
3082 err = bpf_check(&prog, attr, uattr, uattr_size);3197 err = bpf_check(&prog, attr, uattr, attr_log);
3083 if (err < 0)3198 if (err < 0)
3084 goto free_used_maps;3199 goto free_used_maps;
3085 3200
@@ -3122,8 +3237,6 @@ free_used_maps: · 36 unmodified lines
3122 __bpf_prog_put_noref(prog, prog->aux->real_func_cnt);3237 __bpf_prog_put_noref(prog, prog->aux->real_func_cnt);
3123 return err;3238 return err;
3124 3239
3125free_prog_sec:
3126 security_bpf_prog_free(prog);
3127free_prog:3240free_prog:
3128 free_uid(prog->aux->user);3241 free_uid(prog->aux->user);
3129 if (prog->aux->attach_btf)3242 if (prog->aux->attach_btf)
@@ -3198,6 +3311,15 @@ void bpf_link_init(struct bpf_link *link, enum bpf_link_type type, · 68 unmodified lines
3198 bpf_link_init_sleepable(link, type, ops, prog, attach_type, false);3311 bpf_link_init_sleepable(link, type, ops, prog, attach_type, false);
3199}3312}
3200 3313
3314void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type,
3315 const struct bpf_link_ops *ops, struct bpf_prog *prog,
3316 enum bpf_attach_type attach_type, u64 cookie)
3317{
3318 bpf_link_init(&link->link, type, ops, prog, attach_type);
3319 link->node.link = &link->link;
3320 link->node.cookie = cookie;
3321}
3322
3201static void bpf_link_free_id(int id)3323static void bpf_link_free_id(int id)
3202{3324{
3203 if (!id)3325 if (!id)
@@ -3358,7 +3480,7 @@ static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp) · 154 unmodified lines
3358 seq_printf(m, "link_type:\t%s\n", link->flags == BPF_F_KPROBE_MULTI_RETURN ?3480 seq_printf(m, "link_type:\t%s\n", link->flags == BPF_F_KPROBE_MULTI_RETURN ?
3359 "kretprobe_multi" : "kprobe_multi");3481 "kretprobe_multi" : "kprobe_multi");
3360 else if (link->type == BPF_LINK_TYPE_UPROBE_MULTI)3482 else if (link->type == BPF_LINK_TYPE_UPROBE_MULTI)
3361 seq_printf(m, "link_type:\t%s\n", link->flags == BPF_F_UPROBE_MULTI_RETURN ?3483 seq_printf(m, "link_type:\t%s\n", link->flags & BPF_F_UPROBE_MULTI_RETURN ?
3362 "uretprobe_multi" : "uprobe_multi");3484 "uretprobe_multi" : "uprobe_multi");
3363 else3485 else
3364 seq_printf(m, "link_type:\t%s\n", bpf_link_type_strs[type]);3486 seq_printf(m, "link_type:\t%s\n", bpf_link_type_strs[type]);
@@ -3505,7 +3627,7 @@ static void bpf_tracing_link_release(struct bpf_link *link) · 140 unmodified lines
3505 struct bpf_tracing_link *tr_link =3627 struct bpf_tracing_link *tr_link =
3506 container_of(link, struct bpf_tracing_link, link.link);3628 container_of(link, struct bpf_tracing_link, link.link);
3507 3629
3508 WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link,3630 WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link.node,
3509 tr_link->trampoline,3631 tr_link->trampoline,
3510 tr_link->tgt_prog));3632 tr_link->tgt_prog));
3511 3633
@@ -3518,8 +3640,7 @@ static void bpf_tracing_link_release(struct bpf_link *link) · 6 unmodified lines
3518 3640
3519static void bpf_tracing_link_dealloc(struct bpf_link *link)3641static void bpf_tracing_link_dealloc(struct bpf_link *link)
3520{3642{
3521 struct bpf_tracing_link *tr_link =3643 struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link);
3522 container_of(link, struct bpf_tracing_link, link.link);
3523 3644
3524 kfree(tr_link);3645 kfree(tr_link);
3525}3646}
@@ -3527,8 +3648,8 @@ static void bpf_tracing_link_dealloc(struct bpf_link *link) · 1 unmodified line
3527static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link,3648static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link,
3528 struct seq_file *seq)3649 struct seq_file *seq)
3529{3650{
3530 struct bpf_tracing_link *tr_link =3651 struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link);
3531 container_of(link, struct bpf_tracing_link, link.link);3652
3532 u32 target_btf_id, target_obj_id;3653 u32 target_btf_id, target_obj_id;
3533 3654
3534 bpf_trampoline_unpack_key(tr_link->trampoline->key,3655 bpf_trampoline_unpack_key(tr_link->trampoline->key,
@@ -3541,17 +3662,16 @@ static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link, · 6 unmodified lines
3541 link->attach_type,3662 link->attach_type,
3542 target_obj_id,3663 target_obj_id,
3543 target_btf_id,3664 target_btf_id,
3544 tr_link->link.cookie);3665 tr_link->link.node.cookie);
3545}3666}
3546 3667
3547static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,3668static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,
3548 struct bpf_link_info *info)3669 struct bpf_link_info *info)
3549{3670{
3550 struct bpf_tracing_link *tr_link =3671 struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link);
3551 container_of(link, struct bpf_tracing_link, link.link);
3552 3672
3553 info->tracing.attach_type = link->attach_type;3673 info->tracing.attach_type = link->attach_type;
3554 info->tracing.cookie = tr_link->link.cookie;3674 info->tracing.cookie = tr_link->link.node.cookie;
3555 bpf_trampoline_unpack_key(tr_link->trampoline->key,3675 bpf_trampoline_unpack_key(tr_link->trampoline->key,
3556 &info->tracing.target_obj_id,3676 &info->tracing.target_obj_id,
3557 &info->tracing.target_btf_id);3677 &info->tracing.target_btf_id);
@@ -3633,29 +3753,18 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, · 75 unmodified lines
3633 key = bpf_trampoline_compute_key(tgt_prog, NULL, btf_id);3753 key = bpf_trampoline_compute_key(tgt_prog, NULL, btf_id);
3634 }3754 }
3635 3755
3636 if (prog->expected_attach_type == BPF_TRACE_FSESSION) {3756 link = kzalloc_obj(*link, GFP_USER);
3637 struct bpf_fsession_link *fslink;
3638
3639 fslink = kzalloc_obj(*fslink, GFP_USER);
3640 if (fslink) {
3641 bpf_link_init(&fslink->fexit.link, BPF_LINK_TYPE_TRACING,
3642 &bpf_tracing_link_lops, prog, attach_type);
3643 fslink->fexit.cookie = bpf_cookie;
3644 link = &fslink->link;
3645 } else {
3646 link = NULL;
3647 }
3648 } else {
3649 link = kzalloc_obj(*link, GFP_USER);
3650 }
3651 if (!link) {3757 if (!link) {
3652 err = -ENOMEM;3758 err = -ENOMEM;
3653 goto out_put_prog;3759 goto out_put_prog;
3654 }3760 }
3655 bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING,3761 bpf_tramp_link_init(&link->link, BPF_LINK_TYPE_TRACING,
3656 &bpf_tracing_link_lops, prog, attach_type);3762 &bpf_tracing_link_lops, prog, attach_type, bpf_cookie);
3657 3763
3658 link->link.cookie = bpf_cookie;3764 if (prog->expected_attach_type == BPF_TRACE_FSESSION) {
3765 link->fexit.link = &link->link.link;
3766 link->fexit.cookie = bpf_cookie;
3767 }
3659 3768
3660 mutex_lock(&prog->aux->dst_mutex);3769 mutex_lock(&prog->aux->dst_mutex);
3661 3770
@@ -3758,7 +3867,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, · 96 unmodified lines
3758 if (err)3867 if (err)
3759 goto out_unlock;3868 goto out_unlock;
3760 3869
3761 err = bpf_trampoline_link_prog(&link->link, tr, tgt_prog);3870 err = bpf_trampoline_link_prog(&link->link.node, tr, tgt_prog);
3762 if (err) {3871 if (err) {
3763 bpf_link_cleanup(&link_primer);3872 bpf_link_cleanup(&link_primer);
3764 link = NULL;3873 link = NULL;
@@ -4281,6 +4390,11 @@ static int bpf_raw_tp_link_attach(struct bpf_prog *prog, · 516 unmodified lines
4281 if (!btp)4390 if (!btp)
4282 return -ENOENT;4391 return -ENOENT;
4283 4392
4393 if (prog->sleepable && !tracepoint_is_faultable(btp->tp)) {
4394 bpf_put_raw_tracepoint(btp);
4395 return -EINVAL;
4396 }
4397
4284 link = kzalloc_obj(*link, GFP_USER);4398 link = kzalloc_obj(*link, GFP_USER);
4285 if (!link) {4399 if (!link) {
4286 err = -ENOMEM;4400 err = -ENOMEM;
@@ -4389,6 +4503,9 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) · 102 unmodified lines
4389 case BPF_TRACE_FENTRY:4503 case BPF_TRACE_FENTRY:
4390 case BPF_TRACE_FEXIT:4504 case BPF_TRACE_FEXIT:
4391 case BPF_TRACE_FSESSION:4505 case BPF_TRACE_FSESSION:
4506 case BPF_TRACE_FSESSION_MULTI:
4507 case BPF_TRACE_FENTRY_MULTI:
4508 case BPF_TRACE_FEXIT_MULTI:
4392 case BPF_MODIFY_RETURN:4509 case BPF_MODIFY_RETURN:
4393 return BPF_PROG_TYPE_TRACING;4510 return BPF_PROG_TYPE_TRACING;
4394 case BPF_LSM_MAC:4511 case BPF_LSM_MAC:
@@ -4654,7 +4771,7 @@ static int bpf_prog_detach(const union bpf_attr *attr) · 259 unmodified lines
4654#define BPF_PROG_QUERY_LAST_FIELD query.revision4771#define BPF_PROG_QUERY_LAST_FIELD query.revision
4655 4772
4656static int bpf_prog_query(const union bpf_attr *attr,4773static int bpf_prog_query(const union bpf_attr *attr,
4657 union bpf_attr __user *uattr)4774 union bpf_attr __user *uattr, u32 uattr_size)
4658{4775{
4659 if (!bpf_net_capable())4776 if (!bpf_net_capable())
4660 return -EPERM;4777 return -EPERM;
@@ -4693,7 +4810,7 @@ static int bpf_prog_query(const union bpf_attr *attr, · 32 unmodified lines
4693 case BPF_CGROUP_GETSOCKOPT:4810 case BPF_CGROUP_GETSOCKOPT:
4694 case BPF_CGROUP_SETSOCKOPT:4811 case BPF_CGROUP_SETSOCKOPT:
4695 case BPF_LSM_CGROUP:4812 case BPF_LSM_CGROUP:
4696 return cgroup_bpf_prog_query(attr, uattr);4813 return cgroup_bpf_prog_query(attr, uattr, uattr_size);
4697 case BPF_LIRC_MODE2:4814 case BPF_LIRC_MODE2:
4698 return lirc_prog_query(attr, uattr);4815 return lirc_prog_query(attr, uattr);
4699 case BPF_FLOW_DISSECTOR:4816 case BPF_FLOW_DISSECTOR:
@@ -5045,10 +5162,11 @@ static int bpf_prog_get_info_by_fd(struct file *file, · 345 unmodified lines
5045 u32 info_len = attr->info.info_len;5162 u32 info_len = attr->info.info_len;
5046 struct bpf_prog_kstats stats;5163 struct bpf_prog_kstats stats;
5047 char __user *uinsns;5164 char __user *uinsns;
5048 u32 ulen;5165 u32 ulen, len;
5049 int err;5166 int err;
5050 5167
5051 err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len);5168 len = offsetofend(struct bpf_prog_info, attach_btf_id);
5169 err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), len, info_len);
5052 if (err)5170 if (err)
5053 return err;5171 return err;
5054 info_len = min_t(u32, sizeof(info), info_len);5172 info_len = min_t(u32, sizeof(info), info_len);
@@ -5330,10 +5448,11 @@ static int bpf_map_get_info_by_fd(struct file *file, · 275 unmodified lines
5330{5448{
5331 struct bpf_map_info __user *uinfo = u64_to_user_ptr(attr->info.info);5449 struct bpf_map_info __user *uinfo = u64_to_user_ptr(attr->info.info);
5332 struct bpf_map_info info;5450 struct bpf_map_info info;
5333 u32 info_len = attr->info.info_len;5451 u32 info_len = attr->info.info_len, len;
5334 int err;5452 int err;
5335 5453
5336 err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len);5454 len = offsetofend(struct bpf_map_info, hash_size);
5455 err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), len, info_len);
5337 if (err)5456 if (err)
5338 return err;5457 return err;
5339 info_len = min_t(u32, sizeof(info), info_len);5458 info_len = min_t(u32, sizeof(info), info_len);
@@ -5371,18 +5490,16 @@ static int bpf_map_get_info_by_fd(struct file *file, · 31 unmodified lines
5371 5490
5372 if (!map->ops->map_get_hash)5491 if (!map->ops->map_get_hash)
5373 return -EINVAL;5492 return -EINVAL;
5374 5493 if (info.hash_size != sizeof(map->sha))
5375 if (info.hash_size != SHA256_DIGEST_SIZE)
5376 return -EINVAL;5494 return -EINVAL;
5377
5378 if (!READ_ONCE(map->frozen))5495 if (!READ_ONCE(map->frozen))
5379 return -EPERM;5496 return -EPERM;
5380 5497
5381 err = map->ops->map_get_hash(map, SHA256_DIGEST_SIZE, map->sha);5498 err = map->ops->map_get_hash(map);
5382 if (err != 0)5499 if (err != 0)
5383 return err;5500 return err;
5384 5501
5385 if (copy_to_user(uhash, map->sha, SHA256_DIGEST_SIZE) != 0)5502 if (copy_to_user(uhash, map->sha, sizeof(map->sha)) != 0)
5386 return -EFAULT;5503 return -EFAULT;
5387 } else if (info.hash_size) {5504 } else if (info.hash_size) {
5388 return -EINVAL;5505 return -EINVAL;
@@ -5495,7 +5612,7 @@ static int bpf_obj_get_info_by_fd(const union bpf_attr *attr, · 106 unmodified lines
5495 5612
5496#define BPF_BTF_LOAD_LAST_FIELD btf_token_fd5613#define BPF_BTF_LOAD_LAST_FIELD btf_token_fd
5497 5614
5498static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, __u32 uattr_size)5615static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
5499{5616{
5500 struct bpf_token *token = NULL;5617 struct bpf_token *token = NULL;
5501 5618
@@ -5522,7 +5639,7 @@ static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, __u32 uattr_ · 20 unmodified lines
5522 5639
5523 bpf_token_put(token);5640 bpf_token_put(token);
5524 5641
5525 return btf_new_fd(attr, uattr, uattr_size);5642 return btf_new_fd(attr, uattr, attr_log);
5526}5643}
5527 5644
5528#define BPF_BTF_GET_FD_BY_ID_LAST_FIELD fd_by_id_token_fd5645#define BPF_BTF_GET_FD_BY_ID_LAST_FIELD fd_by_id_token_fd
@@ -5723,7 +5840,7 @@ err_put: · 194 unmodified lines
5723 return err;5840 return err;
5724}5841}
5725 5842
5726#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.pid5843#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.path_fd
5727static int link_create(union bpf_attr *attr, bpfptr_t uattr)5844static int link_create(union bpf_attr *attr, bpfptr_t uattr)
5728{5845{
5729 struct bpf_prog *prog;5846 struct bpf_prog *prog;
@@ -5774,6 +5891,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr) · 44 unmodified lines
5774 ret = bpf_iter_link_attach(attr, uattr, prog);5891 ret = bpf_iter_link_attach(attr, uattr, prog);
5775 else if (prog->expected_attach_type == BPF_LSM_CGROUP)5892 else if (prog->expected_attach_type == BPF_LSM_CGROUP)
5776 ret = cgroup_bpf_link_attach(attr, prog);5893 ret = cgroup_bpf_link_attach(attr, prog);
5894 else if (is_tracing_multi(prog->expected_attach_type))
5895 ret = bpf_tracing_multi_attach(prog, attr);
5777 else5896 else
5778 ret = bpf_tracing_prog_attach(prog,5897 ret = bpf_tracing_prog_attach(prog,
5779 attr->link_create.target_fd,5898 attr->link_create.target_fd,
@@ -6232,8 +6351,12 @@ put_prog: · 452 unmodified lines
6232 return ret;6351 return ret;
6233}6352}
6234 6353
6235static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size)6354static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size,
6355 bpfptr_t uattr_common, unsigned int size_common)
6236{6356{
6357 struct bpf_common_attr attr_common;
6358 u32 offsetof_log_true_size = 0;
6359 struct bpf_log_attr attr_log;
6237 union bpf_attr attr;6360 union bpf_attr attr;
6238 int err;6361 int err;
6239 6362
@@ -6247,13 +6370,29 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 7 unmodified lines
6247 if (copy_from_bpfptr(&attr, uattr, size) != 0)6370 if (copy_from_bpfptr(&attr, uattr, size) != 0)
6248 return -EFAULT;6371 return -EFAULT;
6249 6372
6373 memset(&attr_common, 0, sizeof(attr_common));
6374 if (cmd & BPF_COMMON_ATTRS) {
6375 err = bpf_check_uarg_tail_zero(uattr_common,
6376 offsetofend(struct bpf_common_attr, log_true_size),
6377 size_common);
6378 if (err)
6379 return err;
6380
6381 cmd &= ~BPF_COMMON_ATTRS;
6382 size_common = min_t(u32, size_common, sizeof(attr_common));
6383 if (copy_from_bpfptr(&attr_common, uattr_common, size_common) != 0)
6384 return -EFAULT;
6385 } else {
6386 size_common = 0;
6387 }
6388
6250 err = security_bpf(cmd, &attr, size, uattr.is_kernel);6389 err = security_bpf(cmd, &attr, size, uattr.is_kernel);
6251 if (err < 0)6390 if (err < 0)
6252 return err;6391 return err;
6253 6392
6254 switch (cmd) {6393 switch (cmd) {
6255 case BPF_MAP_CREATE:6394 case BPF_MAP_CREATE:
6256 err = map_create(&attr, uattr);6395 err = map_create(&attr, uattr, &attr_common, uattr_common, size_common);
6257 break;6396 break;
6258 case BPF_MAP_LOOKUP_ELEM:6397 case BPF_MAP_LOOKUP_ELEM:
6259 err = map_lookup_elem(&attr);6398 err = map_lookup_elem(&attr);
@@ -6271,7 +6410,12 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 11 unmodified lines
6271 err = map_freeze(&attr);6410 err = map_freeze(&attr);
6272 break;6411 break;
6273 case BPF_PROG_LOAD:6412 case BPF_PROG_LOAD:
6274 err = bpf_prog_load(&attr, uattr, size);6413 if (size >= offsetofend(union bpf_attr, log_true_size))
6414 offsetof_log_true_size = offsetof(union bpf_attr, log_true_size);
6415 err = bpf_log_attr_init(&attr_log, attr.log_buf, attr.log_size, attr.log_level,
6416 offsetof_log_true_size, uattr, &attr_common, uattr_common,
6417 size_common);
6418 err = err ?: bpf_prog_load(&attr, uattr, &attr_log);
6275 break;6419 break;
6276 case BPF_OBJ_PIN:6420 case BPF_OBJ_PIN:
6277 err = bpf_obj_pin(&attr);6421 err = bpf_obj_pin(&attr);
@@ -6286,7 +6430,7 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 8 unmodified lines
6286 err = bpf_prog_detach(&attr);6430 err = bpf_prog_detach(&attr);
6287 break;6431 break;
6288 case BPF_PROG_QUERY:6432 case BPF_PROG_QUERY:
6289 err = bpf_prog_query(&attr, uattr.user);6433 err = bpf_prog_query(&attr, uattr.user, size);
6290 break;6434 break;
6291 case BPF_PROG_TEST_RUN:6435 case BPF_PROG_TEST_RUN:
6292 err = bpf_prog_test_run(&attr, uattr.user);6436 err = bpf_prog_test_run(&attr, uattr.user);
@@ -6316,7 +6460,12 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 23 unmodified lines
6316 err = bpf_raw_tracepoint_open(&attr);6460 err = bpf_raw_tracepoint_open(&attr);
6317 break;6461 break;
6318 case BPF_BTF_LOAD:6462 case BPF_BTF_LOAD:
6319 err = bpf_btf_load(&attr, uattr, size);6463 if (size >= offsetofend(union bpf_attr, btf_log_true_size))
6464 offsetof_log_true_size = offsetof(union bpf_attr, btf_log_true_size);
6465 err = bpf_log_attr_init(&attr_log, attr.btf_log_buf, attr.btf_log_size,
6466 attr.btf_log_level, offsetof_log_true_size, uattr,
6467 &attr_common, uattr_common, size_common);
6468 err = err ?: bpf_btf_load(&attr, uattr, &attr_log);
6320 break;6469 break;
6321 case BPF_BTF_GET_FD_BY_ID:6470 case BPF_BTF_GET_FD_BY_ID:
6322 err = bpf_btf_get_fd_by_id(&attr);6471 err = bpf_btf_get_fd_by_id(&attr);
@@ -6382,9 +6531,10 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 59 unmodified lines
6382 return err;6531 return err;
6383}6532}
6384 6533
6385SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)6534SYSCALL_DEFINE5(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size,
6535 struct bpf_common_attr __user *, uattr_common, unsigned int, size_common)
6386{6536{
6387 return __sys_bpf(cmd, USER_BPFPTR(uattr), size);6537 return __sys_bpf(cmd, USER_BPFPTR(uattr), size, USER_BPFPTR(uattr_common), size_common);
6388}6538}
6389 6539
6390static bool syscall_prog_is_valid_access(int off, int size,6540static bool syscall_prog_is_valid_access(int off, int size,
@@ -6414,7 +6564,7 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size) · 23 unmodified lines
6414 default:6564 default:
6415 return -EINVAL;6565 return -EINVAL;
6416 }6566 }
6417 return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size);6567 return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0);
6418}6568}
6419 6569
6420 6570
kernel/bpf/trampoline.c+524−147
@@ -30,8 +30,46 @@ static struct hlist_head trampoline_ip_table[TRAMPOLINE_TABLE_SIZE]; · 29 unmodified lines
30/* serializes access to trampoline tables */30/* serializes access to trampoline tables */
31static DEFINE_MUTEX(trampoline_mutex);31static DEFINE_MUTEX(trampoline_mutex);
32 32
33/*
34 * Keep 32 trampoline locks (5 bits) in the pool so trampoline_lock_all()
35 * stays below MAX_LOCK_DEPTH. Each pool slot has a distinct lockdep
36 * class because trampoline_lock_all() takes all pool mutexes at once;
37 * otherwise lockdep would report recursive locking on same-class mutexes.
38 */
39#define TRAMPOLINE_LOCKS_BITS 5
40#define TRAMPOLINE_LOCKS_TABLE_SIZE (1 << TRAMPOLINE_LOCKS_BITS)
41
42static struct {
43 struct mutex mutex;
44 struct lock_class_key key;
45} trampoline_locks[TRAMPOLINE_LOCKS_TABLE_SIZE];
46
47static struct mutex *select_trampoline_lock(struct bpf_trampoline *tr)
48{
49 return &trampoline_locks[hash_ptr(tr, TRAMPOLINE_LOCKS_BITS)].mutex;
50}
51
52static void trampoline_lock(struct bpf_trampoline *tr)
53{
54 mutex_lock(select_trampoline_lock(tr));
55}
56
57static void trampoline_unlock(struct bpf_trampoline *tr)
58{
59 mutex_unlock(select_trampoline_lock(tr));
60}
61
62struct bpf_trampoline_ops {
63 int (*register_fentry)(struct bpf_trampoline *tr, struct bpf_tramp_image *im, void *data);
64 int (*unregister_fentry)(struct bpf_trampoline *tr, u32 orig_flags, void *data);
65 int (*modify_fentry)(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im,
66 bool lock_direct_mutex, void *data);
67};
68
33#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS69#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
34static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex);70static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex,
71 const struct bpf_trampoline_ops *ops, void *data);
72static const struct bpf_trampoline_ops trampoline_ops;
35 73
36#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS74#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
37static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops, unsigned long ip)75static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops, unsigned long ip)
@@ -69,9 +107,9 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 31 unmodified lines
69 107
70 if (cmd == FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) {108 if (cmd == FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) {
71 /* This is called inside register_ftrace_direct_multi(), so109 /* This is called inside register_ftrace_direct_multi(), so
72 * tr->mutex is already locked.110 * trampoline's mutex is already locked.
73 */111 */
74 lockdep_assert_held_once(&tr->mutex);112 lockdep_assert_held_once(select_trampoline_lock(tr));
75 113
76 /* Instead of updating the trampoline here, we propagate114 /* Instead of updating the trampoline here, we propagate
77 * -EAGAIN to register_ftrace_direct(). Then we can115 * -EAGAIN to register_ftrace_direct(). Then we can
@@ -91,7 +129,7 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 13 unmodified lines
91 }129 }
92 130
93 /* The normal locking order is131 /* The normal locking order is
94 * tr->mutex => direct_mutex (ftrace.c) => ftrace_lock (ftrace.c)132 * select_trampoline_lock(tr) => direct_mutex (ftrace.c) => ftrace_lock (ftrace.c)
95 *133 *
96 * The following two commands are called from134 * The following two commands are called from
97 *135 *
@@ -99,12 +137,12 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 1 unmodified line
99 * cleanup_direct_functions_after_ipmodify137 * cleanup_direct_functions_after_ipmodify
100 *138 *
101 * In both cases, direct_mutex is already locked. Use139 * In both cases, direct_mutex is already locked. Use
102 * mutex_trylock(&tr->mutex) to avoid deadlock in race condition140 * mutex_trylock(select_trampoline_lock(tr)) to avoid deadlock in race condition
103 * (something else is making changes to this same trampoline).141 * (something else holds the same pool lock).
104 */142 */
105 if (!mutex_trylock(&tr->mutex)) {143 if (!mutex_trylock(select_trampoline_lock(tr))) {
106 /* sleep 1 ms to make sure whatever holding tr->mutex makes144 /* sleep 1 ms to make sure whatever holding select_trampoline_lock(tr)
107 * some progress.145 * makes some progress.
108 */146 */
109 msleep(1);147 msleep(1);
110 return -EAGAIN;148 return -EAGAIN;
@@ -116,20 +154,22 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 5 unmodified lines
116 154
117 if ((tr->flags & BPF_TRAMP_F_CALL_ORIG) &&155 if ((tr->flags & BPF_TRAMP_F_CALL_ORIG) &&
118 !(tr->flags & BPF_TRAMP_F_ORIG_STACK))156 !(tr->flags & BPF_TRAMP_F_ORIG_STACK))
119 ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */);157 ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */,
158 &trampoline_ops, NULL);
120 break;159 break;
121 case FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER:160 case FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER:
122 tr->flags &= ~BPF_TRAMP_F_SHARE_IPMODIFY;161 tr->flags &= ~BPF_TRAMP_F_SHARE_IPMODIFY;
123 162
124 if (tr->flags & BPF_TRAMP_F_ORIG_STACK)163 if (tr->flags & BPF_TRAMP_F_ORIG_STACK)
125 ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */);164 ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */,
165 &trampoline_ops, NULL);
126 break;166 break;
127 default:167 default:
128 ret = -EINVAL;168 ret = -EINVAL;
129 break;169 break;
130 }170 }
131 171
132 mutex_unlock(&tr->mutex);172 trampoline_unlock(tr);
133 return ret;173 return ret;
134}174}
135#endif175#endif
@@ -142,7 +182,9 @@ bool bpf_prog_has_trampoline(const struct bpf_prog *prog) · 6 unmodified lines
142 switch (ptype) {182 switch (ptype) {
143 case BPF_PROG_TYPE_TRACING:183 case BPF_PROG_TYPE_TRACING:
144 if (eatype == BPF_TRACE_FENTRY || eatype == BPF_TRACE_FEXIT ||184 if (eatype == BPF_TRACE_FENTRY || eatype == BPF_TRACE_FEXIT ||
145 eatype == BPF_MODIFY_RETURN || eatype == BPF_TRACE_FSESSION)185 eatype == BPF_MODIFY_RETURN || eatype == BPF_TRACE_FSESSION ||
186 eatype == BPF_TRACE_FENTRY_MULTI || eatype == BPF_TRACE_FEXIT_MULTI ||
187 eatype == BPF_TRACE_FSESSION_MULTI)
146 return true;188 return true;
147 return false;189 return false;
148 case BPF_PROG_TYPE_LSM:190 case BPF_PROG_TYPE_LSM:
@@ -359,7 +401,6 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip) · 210 unmodified lines
359 head = &trampoline_ip_table[hash_64(tr->ip, TRAMPOLINE_HASH_BITS)];401 head = &trampoline_ip_table[hash_64(tr->ip, TRAMPOLINE_HASH_BITS)];
360 hlist_add_head(&tr->hlist_ip, head);402 hlist_add_head(&tr->hlist_ip, head);
361 refcount_set(&tr->refcnt, 1);403 refcount_set(&tr->refcnt, 1);
362 mutex_init(&tr->mutex);
363 for (i = 0; i < BPF_TRAMP_MAX; i++)404 for (i = 0; i < BPF_TRAMP_MAX; i++)
364 INIT_HLIST_HEAD(&tr->progs_hlist[i]);405 INIT_HLIST_HEAD(&tr->progs_hlist[i]);
365out:406out:
@@ -386,9 +427,11 @@ static int bpf_trampoline_update_fentry(struct bpf_trampoline *tr, u32 orig_flag · 20 unmodified lines
386 return bpf_arch_text_poke(ip, old_t, new_t, old_addr, new_addr);427 return bpf_arch_text_poke(ip, old_t, new_t, old_addr, new_addr);
387}428}
388 429
389static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags,430static void bpf_tramp_image_put(struct bpf_tramp_image *im);
390 void *old_addr)431
432static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, void *data __maybe_unused)
391{433{
434 void *old_addr = tr->cur_image->image;
392 int ret;435 int ret;
393 436
394 if (tr->func.ftrace_managed)437 if (tr->func.ftrace_managed)
@@ -396,13 +439,19 @@ static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, · 1 unmodified line
396 else439 else
397 ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr, NULL);440 ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr, NULL);
398 441
399 return ret;442 if (ret)
443 return ret;
444
445 bpf_tramp_image_put(tr->cur_image);
446 tr->cur_image = NULL;
447 return 0;
400}448}
401 449
402static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags,450static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im,
403 void *old_addr, void *new_addr,451 bool lock_direct_mutex, void *data __maybe_unused)
404 bool lock_direct_mutex)
405{452{
453 void *old_addr = tr->cur_image->image;
454 void *new_addr = im->image;
406 int ret;455 int ret;
407 456
408 if (tr->func.ftrace_managed) {457 if (tr->func.ftrace_managed) {
@@ -411,12 +460,20 @@ static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, · 2 unmodified lines
411 ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr,460 ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr,
412 new_addr);461 new_addr);
413 }462 }
414 return ret;463
464 if (ret)
465 return ret;
466
467 bpf_tramp_image_put(tr->cur_image);
468 tr->cur_image = im;
469 return 0;
415}470}
416 471
417/* first time registering */472/* first time registering */
418static int register_fentry(struct bpf_trampoline *tr, void *new_addr)473static int register_fentry(struct bpf_trampoline *tr, struct bpf_tramp_image *im,
474 void *data __maybe_unused)
419{475{
476 void *new_addr = im->image;
420 void *ip = tr->func.addr;477 void *ip = tr->func.addr;
421 unsigned long faddr;478 unsigned long faddr;
422 int ret;479 int ret;
@@ -434,33 +491,42 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) · 11 unmodified lines
434 ret = bpf_trampoline_update_fentry(tr, 0, NULL, new_addr);491 ret = bpf_trampoline_update_fentry(tr, 0, NULL, new_addr);
435 }492 }
436 493
437 return ret;494 if (ret)
495 return ret;
496
497 tr->cur_image = im;
498 return 0;
438}499}
439 500
440static struct bpf_tramp_links *501static const struct bpf_trampoline_ops trampoline_ops = {
502 .register_fentry = register_fentry,
503 .unregister_fentry = unregister_fentry,
504 .modify_fentry = modify_fentry,
505};
506
507static struct bpf_tramp_nodes *
441bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg)508bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg)
442{509{
443 struct bpf_tramp_link *link;510 struct bpf_tramp_node *node, **nodes;
444 struct bpf_tramp_links *tlinks;511 struct bpf_tramp_nodes *tnodes;
445 struct bpf_tramp_link **links;
446 int kind;512 int kind;
447 513
448 *total = 0;514 *total = 0;
449 tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX);515 tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX);
450 if (!tlinks)516 if (!tnodes)
451 return ERR_PTR(-ENOMEM);517 return ERR_PTR(-ENOMEM);
452 518
453 for (kind = 0; kind < BPF_TRAMP_MAX; kind++) {519 for (kind = 0; kind < BPF_TRAMP_MAX; kind++) {
454 tlinks[kind].nr_links = tr->progs_cnt[kind];520 tnodes[kind].nr_nodes = tr->progs_cnt[kind];
455 *total += tr->progs_cnt[kind];521 *total += tr->progs_cnt[kind];
456 links = tlinks[kind].links;522 nodes = tnodes[kind].nodes;
457 523
458 hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) {524 hlist_for_each_entry(node, &tr->progs_hlist[kind], tramp_hlist) {
459 *ip_arg |= link->link.prog->call_get_func_ip;525 *ip_arg |= node->link->prog->call_get_func_ip;
460 *links++ = link;526 *nodes++ = node;
461 }527 }
462 }528 }
463 return tlinks;529 return tnodes;
464}530}
465 531
466static void bpf_tramp_image_free(struct bpf_tramp_image *im)532static void bpf_tramp_image_free(struct bpf_tramp_image *im)
@@ -604,30 +670,29 @@ out: · 137 unmodified lines
604 return ERR_PTR(err);670 return ERR_PTR(err);
605}671}
606 672
607static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex)673static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex,
674 const struct bpf_trampoline_ops *ops, void *data)
608{675{
609 struct bpf_tramp_image *im;676 struct bpf_tramp_image *im;
610 struct bpf_tramp_links *tlinks;677 struct bpf_tramp_nodes *tnodes;
611 u32 orig_flags = tr->flags;678 u32 orig_flags = tr->flags;
612 bool ip_arg = false;679 bool ip_arg = false;
613 int err, total, size;680 int err, total, size;
614 681
615 tlinks = bpf_trampoline_get_progs(tr, &total, &ip_arg);682 tnodes = bpf_trampoline_get_progs(tr, &total, &ip_arg);
616 if (IS_ERR(tlinks))683 if (IS_ERR(tnodes))
617 return PTR_ERR(tlinks);684 return PTR_ERR(tnodes);
618 685
619 if (total == 0) {686 if (total == 0) {
620 err = unregister_fentry(tr, orig_flags, tr->cur_image->image);687 err = ops->unregister_fentry(tr, orig_flags, data);
621 bpf_tramp_image_put(tr->cur_image);
622 tr->cur_image = NULL;
623 goto out;688 goto out;
624 }689 }
625 690
626 /* clear all bits except SHARE_IPMODIFY and TAIL_CALL_CTX */691 /* clear all bits except SHARE_IPMODIFY and TAIL_CALL_CTX */
627 tr->flags &= (BPF_TRAMP_F_SHARE_IPMODIFY | BPF_TRAMP_F_TAIL_CALL_CTX);692 tr->flags &= (BPF_TRAMP_F_SHARE_IPMODIFY | BPF_TRAMP_F_TAIL_CALL_CTX);
628 693
629 if (tlinks[BPF_TRAMP_FEXIT].nr_links ||694 if (tnodes[BPF_TRAMP_FEXIT].nr_nodes ||
630 tlinks[BPF_TRAMP_MODIFY_RETURN].nr_links) {695 tnodes[BPF_TRAMP_MODIFY_RETURN].nr_nodes) {
631 /* NOTE: BPF_TRAMP_F_RESTORE_REGS and BPF_TRAMP_F_SKIP_FRAME696 /* NOTE: BPF_TRAMP_F_RESTORE_REGS and BPF_TRAMP_F_SKIP_FRAME
632 * should not be set together.697 * should not be set together.
633 */698 */
@@ -658,7 +723,7 @@ again: · 24 unmodified lines
658#endif723#endif
659 724
660 size = arch_bpf_trampoline_size(&tr->func.model, tr->flags,725 size = arch_bpf_trampoline_size(&tr->func.model, tr->flags,
661 tlinks, tr->func.addr);726 tnodes, tr->func.addr);
662 if (size < 0) {727 if (size < 0) {
663 err = size;728 err = size;
664 goto out;729 goto out;
@@ -676,7 +741,7 @@ again: · 11 unmodified lines
676 }741 }
677 742
678 err = arch_prepare_bpf_trampoline(im, im->image, im->image + size,743 err = arch_prepare_bpf_trampoline(im, im->image, im->image + size,
679 &tr->func.model, tr->flags, tlinks,744 &tr->func.model, tr->flags, tnodes,
680 tr->func.addr);745 tr->func.addr);
681 if (err < 0)746 if (err < 0)
682 goto out_free;747 goto out_free;
@@ -685,14 +750,12 @@ again: · 2 unmodified lines
685 if (err)750 if (err)
686 goto out_free;751 goto out_free;
687 752
688 WARN_ON(tr->cur_image && total == 0);
689 if (tr->cur_image)753 if (tr->cur_image)
690 /* progs already running at this address */754 /* progs already running at this address */
691 err = modify_fentry(tr, orig_flags, tr->cur_image->image,755 err = ops->modify_fentry(tr, orig_flags, im, lock_direct_mutex, data);
692 im->image, lock_direct_mutex);
693 else756 else
694 /* first time registering */757 /* first time registering */
695 err = register_fentry(tr, im->image);758 err = ops->register_fentry(tr, im, data);
696 759
697#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS760#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
698 if (err == -EAGAIN) {761 if (err == -EAGAIN) {
@@ -704,34 +767,31 @@ again: · 5 unmodified lines
704 goto again;767 goto again;
705 }768 }
706#endif769#endif
707 if (err)
708 goto out_free;
709 770
710 if (tr->cur_image)771out_free:
711 bpf_tramp_image_put(tr->cur_image);772 if (err)
712 tr->cur_image = im;773 bpf_tramp_image_free(im);
713out:774out:
714 /* If any error happens, restore previous flags */775 /* If any error happens, restore previous flags */
715 if (err)776 if (err)
716 tr->flags = orig_flags;777 tr->flags = orig_flags;
717 kfree(tlinks);778 kfree(tnodes);
718 return err;779 return err;
719
720out_free:
721 bpf_tramp_image_free(im);
722 goto out;
723}780}
724 781
725static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog)782static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog)
726{783{
727 switch (prog->expected_attach_type) {784 switch (prog->expected_attach_type) {
728 case BPF_TRACE_FENTRY:785 case BPF_TRACE_FENTRY:
786 case BPF_TRACE_FENTRY_MULTI:
729 return BPF_TRAMP_FENTRY;787 return BPF_TRAMP_FENTRY;
730 case BPF_MODIFY_RETURN:788 case BPF_MODIFY_RETURN:
731 return BPF_TRAMP_MODIFY_RETURN;789 return BPF_TRAMP_MODIFY_RETURN;
732 case BPF_TRACE_FEXIT:790 case BPF_TRACE_FEXIT:
791 case BPF_TRACE_FEXIT_MULTI:
733 return BPF_TRAMP_FEXIT;792 return BPF_TRAMP_FEXIT;
734 case BPF_TRACE_FSESSION:793 case BPF_TRACE_FSESSION:
794 case BPF_TRACE_FSESSION_MULTI:
735 return BPF_TRAMP_FSESSION;795 return BPF_TRAMP_FSESSION;
736 case BPF_LSM_MAC:796 case BPF_LSM_MAC:
737 if (!prog->aux->attach_func_proto->type)797 if (!prog->aux->attach_func_proto->type)
@@ -764,39 +824,33 @@ static int bpf_freplace_check_tgt_prog(struct bpf_prog *tgt_prog) · 26 unmodified lines
764 return 0;824 return 0;
765}825}
766 826
767static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link,827static struct bpf_tramp_node *fsession_exit(struct bpf_tramp_node *node)
768 struct bpf_trampoline *tr,
769 struct bpf_prog *tgt_prog)
770{828{
771 struct bpf_fsession_link *fslink = NULL;829 if (node->link->type == BPF_LINK_TYPE_TRACING) {
772 enum bpf_tramp_prog_type kind;830 struct bpf_tracing_link *link;
773 struct bpf_tramp_link *link_exiting;
774 struct hlist_head *prog_list;
775 int err = 0;
776 int cnt = 0, i;
777 831
778 kind = bpf_attach_type_to_tramp(link->link.prog);832 link = container_of(node->link, struct bpf_tracing_link, link.link);
779 if (tr->extension_prog)833 return &link->fexit;
780 /* cannot attach fentry/fexit if extension prog is attached.834 } else if (node->link->type == BPF_LINK_TYPE_TRACING_MULTI) {
781 * cannot overwrite extension prog either.835 struct bpf_tracing_multi_link *link;
782 */836 struct bpf_tracing_multi_node *mnode;
783 return -EBUSY;
784 837
785 for (i = 0; i < BPF_TRAMP_MAX; i++)838 link = container_of(node->link, struct bpf_tracing_multi_link, link);
786 cnt += tr->progs_cnt[i];839 mnode = container_of(node, struct bpf_tracing_multi_node, node);
787 840 return &link->fexits[mnode - link->nodes];
788 if (kind == BPF_TRAMP_REPLACE) {
789 /* Cannot attach extension if fentry/fexit are in use. */
790 if (cnt)
791 return -EBUSY;
792 err = bpf_freplace_check_tgt_prog(tgt_prog);
793 if (err)
794 return err;
795 tr->extension_prog = link->link.prog;
796 return bpf_arch_text_poke(tr->func.addr, BPF_MOD_NOP,
797 BPF_MOD_JUMP, NULL,
798 link->link.prog->bpf_func);
799 }841 }
842 return NULL;
843}
844
845static int bpf_trampoline_add_prog(struct bpf_trampoline *tr,
846 struct bpf_tramp_node *node,
847 int cnt)
848{
849 enum bpf_tramp_prog_type kind;
850 struct bpf_tramp_node *node_existing, *fexit;
851 struct hlist_head *prog_list;
852
853 kind = bpf_attach_type_to_tramp(node->link->prog);
800 if (kind == BPF_TRAMP_FSESSION) {854 if (kind == BPF_TRAMP_FSESSION) {
801 prog_list = &tr->progs_hlist[BPF_TRAMP_FENTRY];855 prog_list = &tr->progs_hlist[BPF_TRAMP_FENTRY];
802 cnt++;856 cnt++;
@@ -805,59 +859,112 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, · 2 unmodified lines
805 }859 }
806 if (cnt >= BPF_MAX_TRAMP_LINKS)860 if (cnt >= BPF_MAX_TRAMP_LINKS)
807 return -E2BIG;861 return -E2BIG;
808 if (!hlist_unhashed(&link->tramp_hlist))862 if (!hlist_unhashed(&node->tramp_hlist))
809 /* prog already linked */863 /* prog already linked */
810 return -EBUSY;864 return -EBUSY;
811 hlist_for_each_entry(link_exiting, prog_list, tramp_hlist) {865 hlist_for_each_entry(node_existing, prog_list, tramp_hlist) {
812 if (link_exiting->link.prog != link->link.prog)866 if (node_existing->link->prog != node->link->prog)
813 continue;867 continue;
814 /* prog already linked */868 /* prog already linked */
815 return -EBUSY;869 return -EBUSY;
816 }870 }
817 871
818 hlist_add_head(&link->tramp_hlist, prog_list);872 hlist_add_head(&node->tramp_hlist, prog_list);
819 if (kind == BPF_TRAMP_FSESSION) {873 if (kind == BPF_TRAMP_FSESSION) {
820 tr->progs_cnt[BPF_TRAMP_FENTRY]++;874 tr->progs_cnt[BPF_TRAMP_FENTRY]++;
821 fslink = container_of(link, struct bpf_fsession_link, link.link);875 fexit = fsession_exit(node);
822 hlist_add_head(&fslink->fexit.tramp_hlist, &tr->progs_hlist[BPF_TRAMP_FEXIT]);876 if (WARN_ON_ONCE(!fexit))
877 return -EINVAL;
878 hlist_add_head(&fexit->tramp_hlist, &tr->progs_hlist[BPF_TRAMP_FEXIT]);
823 tr->progs_cnt[BPF_TRAMP_FEXIT]++;879 tr->progs_cnt[BPF_TRAMP_FEXIT]++;
824 } else {880 } else {
825 tr->progs_cnt[kind]++;881 tr->progs_cnt[kind]++;
826 }882 }
827 err = bpf_trampoline_update(tr, true /* lock_direct_mutex */);883 return 0;
828 if (err) {884}
829 hlist_del_init(&link->tramp_hlist);885
830 if (kind == BPF_TRAMP_FSESSION) {886static void bpf_trampoline_remove_prog(struct bpf_trampoline *tr,
831 tr->progs_cnt[BPF_TRAMP_FENTRY]--;887 struct bpf_tramp_node *node)
832 hlist_del_init(&fslink->fexit.tramp_hlist);888{
833 tr->progs_cnt[BPF_TRAMP_FEXIT]--;889 enum bpf_tramp_prog_type kind;
834 } else {890 struct bpf_tramp_node *fexit;
835 tr->progs_cnt[kind]--;891
836 }892 kind = bpf_attach_type_to_tramp(node->link->prog);
893 if (kind == BPF_TRAMP_FSESSION) {
894 fexit = fsession_exit(node);
895 if (WARN_ON_ONCE(!fexit))
896 return;
897 hlist_del_init(&fexit->tramp_hlist);
898 tr->progs_cnt[BPF_TRAMP_FEXIT]--;
899 kind = BPF_TRAMP_FENTRY;
837 }900 }
901 hlist_del_init(&node->tramp_hlist);
902 tr->progs_cnt[kind]--;
903}
904
905static int __bpf_trampoline_link_prog(struct bpf_tramp_node *node,
906 struct bpf_trampoline *tr,
907 struct bpf_prog *tgt_prog,
908 const struct bpf_trampoline_ops *ops,
909 void *data)
910{
911 enum bpf_tramp_prog_type kind;
912 int err = 0;
913 int cnt = 0, i;
914
915 kind = bpf_attach_type_to_tramp(node->link->prog);
916 if (tr->extension_prog)
917 /* cannot attach fentry/fexit if extension prog is attached.
918 * cannot overwrite extension prog either.
919 */
920 return -EBUSY;
921
922 for (i = 0; i < BPF_TRAMP_MAX; i++)
923 cnt += tr->progs_cnt[i];
924
925 if (kind == BPF_TRAMP_REPLACE) {
926 /* Cannot attach extension if fentry/fexit are in use. */
927 if (cnt)
928 return -EBUSY;
929 err = bpf_freplace_check_tgt_prog(tgt_prog);
930 if (err)
931 return err;
932 tr->extension_prog = node->link->prog;
933 return bpf_arch_text_poke(tr->func.addr, BPF_MOD_NOP,
934 BPF_MOD_JUMP, NULL,
935 node->link->prog->bpf_func);
936 }
937 err = bpf_trampoline_add_prog(tr, node, cnt);
938 if (err)
939 return err;
940 err = bpf_trampoline_update(tr, true /* lock_direct_mutex */, ops, data);
941 if (err)
942 bpf_trampoline_remove_prog(tr, node);
838 return err;943 return err;
839}944}
840 945
841int bpf_trampoline_link_prog(struct bpf_tramp_link *link,946int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
842 struct bpf_trampoline *tr,947 struct bpf_trampoline *tr,
843 struct bpf_prog *tgt_prog)948 struct bpf_prog *tgt_prog)
844{949{
845 int err;950 int err;
846 951
847 mutex_lock(&tr->mutex);952 trampoline_lock(tr);
848 err = __bpf_trampoline_link_prog(link, tr, tgt_prog);953 err = __bpf_trampoline_link_prog(node, tr, tgt_prog, &trampoline_ops, NULL);
849 mutex_unlock(&tr->mutex);954 trampoline_unlock(tr);
850 return err;955 return err;
851}956}
852 957
853static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,958static int __bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
854 struct bpf_trampoline *tr,959 struct bpf_trampoline *tr,
855 struct bpf_prog *tgt_prog)960 struct bpf_prog *tgt_prog,
961 const struct bpf_trampoline_ops *ops,
962 void *data)
856{963{
857 enum bpf_tramp_prog_type kind;964 enum bpf_tramp_prog_type kind;
858 int err;965 int err;
859 966
860 kind = bpf_attach_type_to_tramp(link->link.prog);967 kind = bpf_attach_type_to_tramp(node->link->prog);
861 if (kind == BPF_TRAMP_REPLACE) {968 if (kind == BPF_TRAMP_REPLACE) {
862 WARN_ON_ONCE(!tr->extension_prog);969 WARN_ON_ONCE(!tr->extension_prog);
863 err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP,970 err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP,
@@ -867,29 +974,21 @@ static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, · 3 unmodified lines
867 guard(mutex)(&tgt_prog->aux->ext_mutex);974 guard(mutex)(&tgt_prog->aux->ext_mutex);
868 tgt_prog->aux->is_extended = false;975 tgt_prog->aux->is_extended = false;
869 return err;976 return err;
870 } else if (kind == BPF_TRAMP_FSESSION) {
871 struct bpf_fsession_link *fslink =
872 container_of(link, struct bpf_fsession_link, link.link);
873
874 hlist_del_init(&fslink->fexit.tramp_hlist);
875 tr->progs_cnt[BPF_TRAMP_FEXIT]--;
876 kind = BPF_TRAMP_FENTRY;
877 }977 }
878 hlist_del_init(&link->tramp_hlist);978 bpf_trampoline_remove_prog(tr, node);
879 tr->progs_cnt[kind]--;979 return bpf_trampoline_update(tr, true /* lock_direct_mutex */, ops, data);
880 return bpf_trampoline_update(tr, true /* lock_direct_mutex */);
881}980}
882 981
883/* bpf_trampoline_unlink_prog() should never fail. */982/* bpf_trampoline_unlink_prog() should never fail. */
884int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,983int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
885 struct bpf_trampoline *tr,984 struct bpf_trampoline *tr,
886 struct bpf_prog *tgt_prog)985 struct bpf_prog *tgt_prog)
887{986{
888 int err;987 int err;
889 988
890 mutex_lock(&tr->mutex);989 trampoline_lock(tr);
891 err = __bpf_trampoline_unlink_prog(link, tr, tgt_prog);990 err = __bpf_trampoline_unlink_prog(node, tr, tgt_prog, &trampoline_ops, NULL);
892 mutex_unlock(&tr->mutex);991 trampoline_unlock(tr);
893 return err;992 return err;
894}993}
895 994
@@ -903,7 +1002,7 @@ static void bpf_shim_tramp_link_release(struct bpf_link *link) · 7 unmodified lines
903 if (!shim_link->trampoline)1002 if (!shim_link->trampoline)
904 return;1003 return;
905 1004
906 WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link, shim_link->trampoline, NULL));1005 WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link.node, shim_link->trampoline, NULL));
907 bpf_trampoline_put(shim_link->trampoline);1006 bpf_trampoline_put(shim_link->trampoline);
908}1007}
909 1008
@@ -949,8 +1048,8 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog · 39 unmodified lines
949 p->type = BPF_PROG_TYPE_LSM;1048 p->type = BPF_PROG_TYPE_LSM;
950 p->expected_attach_type = BPF_LSM_MAC;1049 p->expected_attach_type = BPF_LSM_MAC;
951 bpf_prog_inc(p);1050 bpf_prog_inc(p);
952 bpf_link_init(&shim_link->link.link, BPF_LINK_TYPE_UNSPEC,1051 bpf_tramp_link_init(&shim_link->link, BPF_LINK_TYPE_UNSPEC,
953 &bpf_shim_tramp_link_lops, p, attach_type);1052 &bpf_shim_tramp_link_lops, p, attach_type, 0);
954 bpf_cgroup_atype_get(p->aux->attach_btf_id, cgroup_atype);1053 bpf_cgroup_atype_get(p->aux->attach_btf_id, cgroup_atype);
955 1054
956 return shim_link;1055 return shim_link;
@@ -959,15 +1058,15 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog · 2 unmodified lines
959static struct bpf_shim_tramp_link *cgroup_shim_find(struct bpf_trampoline *tr,1058static struct bpf_shim_tramp_link *cgroup_shim_find(struct bpf_trampoline *tr,
960 bpf_func_t bpf_func)1059 bpf_func_t bpf_func)
961{1060{
962 struct bpf_tramp_link *link;1061 struct bpf_tramp_node *node;
963 int kind;1062 int kind;
964 1063
965 for (kind = 0; kind < BPF_TRAMP_MAX; kind++) {1064 for (kind = 0; kind < BPF_TRAMP_MAX; kind++) {
966 hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) {1065 hlist_for_each_entry(node, &tr->progs_hlist[kind], tramp_hlist) {
967 struct bpf_prog *p = link->link.prog;1066 struct bpf_prog *p = node->link->prog;
968 1067
969 if (p->bpf_func == bpf_func)1068 if (p->bpf_func == bpf_func)
970 return container_of(link, struct bpf_shim_tramp_link, link);1069 return container_of(node, struct bpf_shim_tramp_link, link.node);
971 }1070 }
972 }1071 }
973 1072
@@ -999,12 +1098,12 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog, · 25 unmodified lines
999 if (!tr)1098 if (!tr)
1000 return -ENOMEM;1099 return -ENOMEM;
1001 1100
1002 mutex_lock(&tr->mutex);1101 trampoline_lock(tr);
1003 1102
1004 shim_link = cgroup_shim_find(tr, bpf_func);1103 shim_link = cgroup_shim_find(tr, bpf_func);
1005 if (shim_link && !IS_ERR(bpf_link_inc_not_zero(&shim_link->link.link))) {1104 if (shim_link && !IS_ERR(bpf_link_inc_not_zero(&shim_link->link.link))) {
1006 /* Reusing existing shim attached by the other program. */1105 /* Reusing existing shim attached by the other program. */
1007 mutex_unlock(&tr->mutex);1106 trampoline_unlock(tr);
1008 bpf_trampoline_put(tr); /* bpf_trampoline_get above */1107 bpf_trampoline_put(tr); /* bpf_trampoline_get above */
1009 return 0;1108 return 0;
1010 }1109 }
@@ -1017,23 +1116,23 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog, · 6 unmodified lines
1017 goto err;1116 goto err;
1018 }1117 }
1019 1118
1020 err = __bpf_trampoline_link_prog(&shim_link->link, tr, NULL);1119 err = __bpf_trampoline_link_prog(&shim_link->link.node, tr, NULL, &trampoline_ops, NULL);
1021 if (err)1120 if (err)
1022 goto err;1121 goto err;
1023 1122
1024 shim_link->trampoline = tr;1123 shim_link->trampoline = tr;
1025 /* note, we're still holding tr refcnt from above */1124 /* note, we're still holding tr refcnt from above */
1026 1125
1027 mutex_unlock(&tr->mutex);1126 trampoline_unlock(tr);
1028 1127
1029 return 0;1128 return 0;
1030err:1129err:
1031 mutex_unlock(&tr->mutex);1130 trampoline_unlock(tr);
1032 1131
1033 if (shim_link)1132 if (shim_link)
1034 bpf_link_put(&shim_link->link.link);1133 bpf_link_put(&shim_link->link.link);
1035 1134
1036 /* have to release tr while _not_ holding its mutex */1135 /* have to release tr while _not_ holding pool mutex for trampoline */
1037 bpf_trampoline_put(tr); /* bpf_trampoline_get above */1136 bpf_trampoline_put(tr); /* bpf_trampoline_get above */
1038 1137
1039 return err;1138 return err;
@@ -1054,9 +1153,9 @@ void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog) · 14 unmodified lines
1054 if (WARN_ON_ONCE(!tr))1153 if (WARN_ON_ONCE(!tr))
1055 return;1154 return;
1056 1155
1057 mutex_lock(&tr->mutex);1156 trampoline_lock(tr);
1058 shim_link = cgroup_shim_find(tr, bpf_func);1157 shim_link = cgroup_shim_find(tr, bpf_func);
1059 mutex_unlock(&tr->mutex);1158 trampoline_unlock(tr);
1060 1159
1061 if (shim_link)1160 if (shim_link)
1062 bpf_link_put(&shim_link->link.link);1161 bpf_link_put(&shim_link->link.link);
@@ -1074,14 +1173,14 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, · 11 unmodified lines
1074 if (!tr)1173 if (!tr)
1075 return NULL;1174 return NULL;
1076 1175
1077 mutex_lock(&tr->mutex);1176 trampoline_lock(tr);
1078 if (tr->func.addr)1177 if (tr->func.addr)
1079 goto out;1178 goto out;
1080 1179
1081 memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel));1180 memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel));
1082 tr->func.addr = (void *)tgt_info->tgt_addr;1181 tr->func.addr = (void *)tgt_info->tgt_addr;
1083out:1182out:
1084 mutex_unlock(&tr->mutex);1183 trampoline_unlock(tr);
1085 return tr;1184 return tr;
1086}1185}
1087 1186
@@ -1094,7 +1193,6 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) · 6 unmodified lines
1094 mutex_lock(&trampoline_mutex);1193 mutex_lock(&trampoline_mutex);
1095 if (!refcount_dec_and_test(&tr->refcnt))1194 if (!refcount_dec_and_test(&tr->refcnt))
1096 goto out;1195 goto out;
1097 WARN_ON_ONCE(mutex_is_locked(&tr->mutex));
1098 1196
1099 for (i = 0; i < BPF_TRAMP_MAX; i++)1197 for (i = 0; i < BPF_TRAMP_MAX; i++)
1100 if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[i])))1198 if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[i])))
@@ -1333,7 +1431,7 @@ bpf_trampoline_exit_t bpf_trampoline_exit(const struct bpf_prog *prog) · 232 unmodified lines
1333int __weak1431int __weak
1334arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,1432arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
1335 const struct btf_func_model *m, u32 flags,1433 const struct btf_func_model *m, u32 flags,
1336 struct bpf_tramp_links *tlinks,1434 struct bpf_tramp_nodes *tnodes,
1337 void *func_addr)1435 void *func_addr)
1338{1436{
1339 return -ENOTSUPP;1437 return -ENOTSUPP;
@@ -1367,11 +1465,288 @@ int __weak arch_protect_bpf_trampoline(void *image, unsigned int size) · 27 unmodified lines
1367}1465}
1368 1466
1369int __weak arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,1467int __weak arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
1370 struct bpf_tramp_links *tlinks, void *func_addr)1468 struct bpf_tramp_nodes *tnodes, void *func_addr)
1371{1469{
1372 return -ENOTSUPP;1470 return -ENOTSUPP;
1373}1471}
1374 1472
1473#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \
1474 defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS) && \
1475 defined(CONFIG_BPF_SYSCALL)
1476
1477static void trampoline_lock_all(void)
1478{
1479 int i;
1480
1481 for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++)
1482 mutex_lock(&trampoline_locks[i].mutex);
1483}
1484
1485static void trampoline_unlock_all(void)
1486{
1487 int i;
1488
1489 for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++)
1490 mutex_unlock(&trampoline_locks[i].mutex);
1491}
1492
1493static void remove_tracing_multi_data(struct bpf_tracing_multi_data *data)
1494{
1495 ftrace_hash_remove(data->reg);
1496 ftrace_hash_remove(data->unreg);
1497 ftrace_hash_remove(data->modify);
1498}
1499
1500static void clear_tracing_multi_data(struct bpf_tracing_multi_data *data)
1501{
1502 remove_tracing_multi_data(data);
1503
1504 free_ftrace_hash(data->reg);
1505 free_ftrace_hash(data->unreg);
1506 free_ftrace_hash(data->modify);
1507}
1508
1509static int init_tracing_multi_data(struct bpf_tracing_multi_data *data)
1510{
1511 data->reg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
1512 data->unreg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
1513 data->modify = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
1514
1515 if (!data->reg || !data->unreg || !data->modify) {
1516 clear_tracing_multi_data(data);
1517 return -ENOMEM;
1518 }
1519 return 0;
1520}
1521
1522static void ftrace_hash_add(struct ftrace_hash *hash, struct ftrace_func_entry *entry,
1523 unsigned long ip, unsigned long direct)
1524{
1525 entry->ip = ip;
1526 entry->direct = direct;
1527 add_ftrace_hash_entry(hash, entry);
1528}
1529
1530static int register_fentry_multi(struct bpf_trampoline *tr, struct bpf_tramp_image *im, void *ptr)
1531{
1532 unsigned long addr = (unsigned long) im->image;
1533 unsigned long ip = ftrace_location(tr->ip);
1534 struct bpf_tracing_multi_data *data = ptr;
1535
1536 if (bpf_trampoline_use_jmp(tr->flags))
1537 addr = ftrace_jmp_set(addr);
1538
1539 ftrace_hash_add(data->reg, data->entry, ip, addr);
1540 tr->cur_image = im;
1541 return 0;
1542}
1543
1544static int unregister_fentry_multi(struct bpf_trampoline *tr, u32 orig_flags, void *ptr)
1545{
1546 unsigned long addr = (unsigned long) tr->cur_image->image;
1547 unsigned long ip = ftrace_location(tr->ip);
1548 struct bpf_tracing_multi_data *data = ptr;
1549
1550 if (bpf_trampoline_use_jmp(tr->flags))
1551 addr = ftrace_jmp_set(addr);
1552
1553 ftrace_hash_add(data->unreg, data->entry, ip, addr);
1554 tr->cur_image = NULL;
1555 return 0;
1556}
1557
1558static int modify_fentry_multi(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im,
1559 bool lock_direct_mutex, void *ptr)
1560{
1561 unsigned long addr = (unsigned long) im->image;
1562 unsigned long ip = ftrace_location(tr->ip);
1563 struct bpf_tracing_multi_data *data = ptr;
1564
1565 if (bpf_trampoline_use_jmp(tr->flags))
1566 addr = ftrace_jmp_set(addr);
1567
1568 ftrace_hash_add(data->modify, data->entry, ip, addr);
1569 tr->cur_image = im;
1570 return 0;
1571}
1572
1573static const struct bpf_trampoline_ops trampoline_multi_ops = {
1574 .register_fentry = register_fentry_multi,
1575 .unregister_fentry = unregister_fentry_multi,
1576 .modify_fentry = modify_fentry_multi,
1577};
1578
1579static void bpf_trampoline_multi_attach_init(struct bpf_trampoline *tr)
1580{
1581 tr->multi_attach.old_image = tr->cur_image;
1582 tr->multi_attach.old_flags = tr->flags;
1583}
1584
1585static void bpf_trampoline_multi_attach_free(struct bpf_trampoline *tr)
1586{
1587 if (tr->multi_attach.old_image)
1588 bpf_tramp_image_put(tr->multi_attach.old_image);
1589
1590 tr->multi_attach.old_image = NULL;
1591 tr->multi_attach.old_flags = 0;
1592}
1593
1594static void bpf_trampoline_multi_attach_rollback(struct bpf_trampoline *tr)
1595{
1596 if (tr->cur_image)
1597 bpf_tramp_image_put(tr->cur_image);
1598 tr->cur_image = tr->multi_attach.old_image;
1599 tr->flags = tr->multi_attach.old_flags;
1600
1601 tr->multi_attach.old_image = NULL;
1602 tr->multi_attach.old_flags = 0;
1603}
1604
1605#define for_each_mnode_cnt(mnode, link, cnt) \
1606 for (i = 0, mnode = &link->nodes[i]; i < cnt; i++, mnode = &link->nodes[i])
1607
1608#define for_each_mnode(mnode, link) \
1609 for_each_mnode_cnt(mnode, link, link->nodes_cnt)
1610
1611int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
1612 struct bpf_tracing_multi_link *link)
1613{
1614 struct bpf_tracing_multi_data *data = &link->data;
1615 struct bpf_attach_target_info tgt_info = {};
1616 struct btf *btf = prog->aux->attach_btf;
1617 struct bpf_tracing_multi_node *mnode;
1618 struct bpf_trampoline *tr;
1619 int i, err, rollback_cnt;
1620 u64 key;
1621
1622 for_each_mnode(mnode, link) {
1623 rollback_cnt = i;
1624
1625 err = bpf_check_attach_btf_id_multi(btf, prog, ids[i], &tgt_info);
1626 if (err)
1627 goto rollback_put;
1628
1629 key = bpf_trampoline_compute_key(NULL, btf, ids[i]);
1630
1631 tr = bpf_trampoline_get(key, &tgt_info);
1632 if (!tr) {
1633 err = -ENOMEM;
1634 goto rollback_put;
1635 }
1636
1637 mnode->trampoline = tr;
1638 mnode->node.link = &link->link;
1639 mnode->node.cookie = link->cookies ? link->cookies[i] : 0;
1640
1641 if (prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) {
1642 link->fexits[i].link = &link->link;
1643 link->fexits[i].cookie = link->cookies ? link->cookies[i] : 0;
1644 }
1645
1646 cond_resched();
1647 }
1648
1649 err = init_tracing_multi_data(data);
1650 if (err) {
1651 rollback_cnt = link->nodes_cnt;
1652 goto rollback_put;
1653 }
1654
1655 trampoline_lock_all();
1656
1657 for_each_mnode(mnode, link) {
1658 bpf_trampoline_multi_attach_init(mnode->trampoline);
1659
1660 data->entry = &mnode->entry;
1661 err = __bpf_trampoline_link_prog(&mnode->node, mnode->trampoline, NULL,
1662 &trampoline_multi_ops, data);
1663 if (err) {
1664 rollback_cnt = i;
1665 goto rollback_unlink;
1666 }
1667 }
1668
1669 rollback_cnt = link->nodes_cnt;
1670 if (ftrace_hash_count(data->reg)) {
1671 err = update_ftrace_direct_add(&direct_ops, data->reg);
1672 if (err)
1673 goto rollback_unlink;
1674 }
1675
1676 if (ftrace_hash_count(data->modify)) {
1677 err = update_ftrace_direct_mod(&direct_ops, data->modify, true);
1678 if (err) {
1679 if (ftrace_hash_count(data->reg))
1680 WARN_ON_ONCE(update_ftrace_direct_del(&direct_ops, data->reg));
1681 goto rollback_unlink;
1682 }
1683 }
1684
1685 for_each_mnode(mnode, link)
1686 bpf_trampoline_multi_attach_free(mnode->trampoline);
1687
1688 trampoline_unlock_all();
1689
1690 remove_tracing_multi_data(data);
1691 return 0;
1692
1693rollback_unlink:
1694 for_each_mnode_cnt(mnode, link, rollback_cnt) {
1695 bpf_trampoline_remove_prog(mnode->trampoline, &mnode->node);
1696 bpf_trampoline_multi_attach_rollback(mnode->trampoline);
1697 }
1698
1699 trampoline_unlock_all();
1700
1701 clear_tracing_multi_data(data);
1702 rollback_cnt = link->nodes_cnt;
1703
1704rollback_put:
1705 for_each_mnode_cnt(mnode, link, rollback_cnt)
1706 bpf_trampoline_put(mnode->trampoline);
1707
1708 return err;
1709}
1710
1711int bpf_trampoline_multi_detach(struct bpf_prog *prog, struct bpf_tracing_multi_link *link)
1712{
1713 struct bpf_tracing_multi_data *data = &link->data;
1714 struct bpf_tracing_multi_node *mnode;
1715 int i;
1716
1717 trampoline_lock_all();
1718
1719 for_each_mnode(mnode, link) {
1720 data->entry = &mnode->entry;
1721 bpf_trampoline_multi_attach_init(mnode->trampoline);
1722 WARN_ON_ONCE(__bpf_trampoline_unlink_prog(&mnode->node, mnode->trampoline,
1723 NULL, &trampoline_multi_ops, data));
1724 }
1725
1726 if (ftrace_hash_count(data->unreg))
1727 WARN_ON_ONCE(update_ftrace_direct_del(&direct_ops, data->unreg));
1728 if (ftrace_hash_count(data->modify))
1729 WARN_ON_ONCE(update_ftrace_direct_mod(&direct_ops, data->modify, true));
1730
1731 for_each_mnode(mnode, link)
1732 bpf_trampoline_multi_attach_free(mnode->trampoline);
1733
1734 trampoline_unlock_all();
1735
1736 for_each_mnode(mnode, link)
1737 bpf_trampoline_put(mnode->trampoline);
1738
1739 clear_tracing_multi_data(data);
1740 return 0;
1741}
1742
1743#undef for_each_mnode_cnt
1744#undef for_each_mnode
1745
1746#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS &&
1747 CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS &&
1748 CONFIG_BPF_SYSCALL */
1749
1375static int __init init_trampolines(void)1750static int __init init_trampolines(void)
1376{1751{
1377 int i;1752 int i;
@@ -1380,6 +1755,8 @@ static int __init init_trampolines(void) · 2 unmodified lines
1380 INIT_HLIST_HEAD(&trampoline_key_table[i]);1755 INIT_HLIST_HEAD(&trampoline_key_table[i]);
1381 for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++)1756 for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++)
1382 INIT_HLIST_HEAD(&trampoline_ip_table[i]);1757 INIT_HLIST_HEAD(&trampoline_ip_table[i]);
1758 for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++)
1759 __mutex_init(&trampoline_locks[i].mutex, "trampoline_lock", &trampoline_locks[i].key);
1383 return 0;1760 return 0;
1384}1761}
1385late_initcall(init_trampolines);1762late_initcall(init_trampolines);
kernel/bpf/verifier.c+1978−2196
@@ -26,6 +26,7 @@ · 25 unmodified lines
26#include <linux/poison.h>26#include <linux/poison.h>
27#include <linux/module.h>27#include <linux/module.h>
28#include <linux/cpumask.h>28#include <linux/cpumask.h>
29#include <linux/cnum.h>
29#include <linux/bpf_mem_alloc.h>30#include <linux/bpf_mem_alloc.h>
30#include <net/xdp.h>31#include <net/xdp.h>
31#include <linux/trace_events.h>32#include <linux/trace_events.h>
@@ -199,14 +200,15 @@ struct bpf_verifier_stack_elem { · 167 unmodified lines
199 200
200#define BPF_PRIV_STACK_MIN_SIZE 64201#define BPF_PRIV_STACK_MIN_SIZE 64
201 202
202static int acquire_reference(struct bpf_verifier_env *env, int insn_idx);203static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int parent_id);
203static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id);204static int release_reference_nomark(struct bpf_verifier_state *state, int id);
204static int release_reference(struct bpf_verifier_env *env, int ref_obj_id);205static int release_reference(struct bpf_verifier_env *env, int id);
205static void invalidate_non_owning_refs(struct bpf_verifier_env *env);206static void invalidate_non_owning_refs(struct bpf_verifier_env *env);
206static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env);207static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env);
208static bool is_tracing_prog_type(enum bpf_prog_type type);
207static int ref_set_non_owning(struct bpf_verifier_env *env,209static int ref_set_non_owning(struct bpf_verifier_env *env,
208 struct bpf_reg_state *reg);210 struct bpf_reg_state *reg);
209static bool is_trusted_reg(const struct bpf_reg_state *reg);211static bool is_trusted_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg);
210static inline bool in_sleepable_context(struct bpf_verifier_env *env);212static inline bool in_sleepable_context(struct bpf_verifier_env *env);
211static const char *non_sleepable_context_description(struct bpf_verifier_env *env);213static const char *non_sleepable_context_description(struct bpf_verifier_env *env);
212static void scalar32_min_max_add(struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg);214static void scalar32_min_max_add(struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg);
@@ -230,8 +232,28 @@ static void bpf_map_key_store(struct bpf_insn_aux_data *aux, u64 state) · 17 unmodified lines
230 (poisoned ? BPF_MAP_KEY_POISON : 0ULL);232 (poisoned ? BPF_MAP_KEY_POISON : 0ULL);
231}233}
232 234
235static void update_ref_obj(struct ref_obj_desc *ref_obj, struct bpf_reg_state *reg)
236{
237 ref_obj->id = reg->id;
238 ref_obj->parent_id = reg->parent_id;
239 ref_obj->cnt++;
240}
241
242static int validate_ref_obj(struct bpf_verifier_env *env, struct ref_obj_desc *ref_obj)
243{
244 if (ref_obj->cnt > 1) {
245 verifier_bug(env, "function expects only one referenced object but got %d\n",
246 ref_obj->cnt);
247 return -EFAULT;
248 }
249
250 return 0;
251}
252
233struct bpf_call_arg_meta {253struct bpf_call_arg_meta {
234 struct bpf_map_desc map;254 struct bpf_map_desc map;
255 struct bpf_dynptr_desc dynptr;
256 struct ref_obj_desc ref_obj;
235 bool raw_mode;257 bool raw_mode;
236 bool pkt_access;258 bool pkt_access;
237 u8 release_regno;259 u8 release_regno;
@@ -239,8 +261,6 @@ struct bpf_call_arg_meta { · 1 unmodified line
239 int access_size;261 int access_size;
240 int mem_size;262 int mem_size;
241 u64 msize_max_value;263 u64 msize_max_value;
242 int ref_obj_id;
243 int dynptr_id;
244 int func_id;264 int func_id;
245 struct btf *btf;265 struct btf *btf;
246 u32 btf_id;266 u32 btf_id;
@@ -261,6 +281,41 @@ struct bpf_kfunc_meta { · 14 unmodified lines
261 281
262struct btf *btf_vmlinux;282struct btf *btf_vmlinux;
263 283
284typedef struct argno {
285 int argno;
286} argno_t;
287
288static argno_t argno_from_reg(u32 regno)
289{
290 return (argno_t){ .argno = regno };
291}
292
293static argno_t argno_from_arg(u32 arg)
294{
295 return (argno_t){ .argno = -arg };
296}
297
298static int reg_from_argno(argno_t a)
299{
300 if (a.argno >= 0)
301 return a.argno;
302 if (a.argno >= -MAX_BPF_FUNC_REG_ARGS)
303 return -a.argno;
304 return -1;
305}
306
307static int arg_from_argno(argno_t a)
308{
309 if (a.argno < 0)
310 return -a.argno;
311 return -1;
312}
313
314static int arg_idx_from_argno(argno_t a)
315{
316 return arg_from_argno(a) - 1;
317}
318
264static const char *btf_type_name(const struct btf *btf, u32 id)319static const char *btf_type_name(const struct btf *btf, u32 id)
265{320{
266 return btf_name_by_offset(btf, btf_type_by_id(btf, id)->name_off);321 return btf_name_by_offset(btf, btf_type_by_id(btf, id)->name_off);
@@ -290,12 +345,12 @@ static void verbose_invalid_scalar(struct bpf_verifier_env *env, · 23 unmodified lines
290 bool unknown = true;345 bool unknown = true;
291 346
292 verbose(env, "%s the register %s has", ctx, reg_name);347 verbose(env, "%s the register %s has", ctx, reg_name);
293 if (reg->smin_value > S64_MIN) {348 if (reg_smin(reg) > S64_MIN) {
294 verbose(env, " smin=%lld", reg->smin_value);349 verbose(env, " smin=%lld", reg_smin(reg));
295 unknown = false;350 unknown = false;
296 }351 }
297 if (reg->smax_value < S64_MAX) {352 if (reg_smax(reg) < S64_MAX) {
298 verbose(env, " smax=%lld", reg->smax_value);353 verbose(env, " smax=%lld", reg_smax(reg));
299 unknown = false;354 unknown = false;
300 }355 }
301 if (unknown)356 if (unknown)
@@ -303,7 +358,7 @@ static void verbose_invalid_scalar(struct bpf_verifier_env *env, · 1 unmodified line
303 verbose(env, " should have been in [%d, %d]\n", range.minval, range.maxval);358 verbose(env, " should have been in [%d, %d]\n", range.minval, range.maxval);
304}359}
305 360
306static bool reg_not_null(const struct bpf_reg_state *reg)361static bool reg_not_null(struct bpf_verifier_env *env, const struct bpf_reg_state *reg)
307{362{
308 enum bpf_reg_type type;363 enum bpf_reg_type type;
309 364
@@ -317,7 +372,7 @@ static bool reg_not_null(const struct bpf_reg_state *reg) · 7 unmodified lines
317 type == PTR_TO_MAP_VALUE ||372 type == PTR_TO_MAP_VALUE ||
318 type == PTR_TO_MAP_KEY ||373 type == PTR_TO_MAP_KEY ||
319 type == PTR_TO_SOCK_COMMON ||374 type == PTR_TO_SOCK_COMMON ||
320 (type == PTR_TO_BTF_ID && is_trusted_reg(reg)) ||375 (type == PTR_TO_BTF_ID && is_trusted_reg(env, reg)) ||
321 (type == PTR_TO_MEM && !(reg->type & PTR_UNTRUSTED)) ||376 (type == PTR_TO_MEM && !(reg->type & PTR_UNTRUSTED)) ||
322 type == CONST_PTR_TO_MAP;377 type == CONST_PTR_TO_MAP;
323}378}
@@ -434,11 +489,6 @@ static bool is_ptr_cast_function(enum bpf_func_id func_id) · 110 unmodified lines
434 func_id == BPF_FUNC_skc_to_tcp_request_sock;489 func_id == BPF_FUNC_skc_to_tcp_request_sock;
435}490}
436 491
437static bool is_dynptr_ref_function(enum bpf_func_id func_id)
438{
439 return func_id == BPF_FUNC_dynptr_data;
440}
441
442static bool is_sync_callback_calling_kfunc(u32 btf_id);492static bool is_sync_callback_calling_kfunc(u32 btf_id);
443static bool is_async_callback_calling_kfunc(u32 btf_id);493static bool is_async_callback_calling_kfunc(u32 btf_id);
444static bool is_callback_calling_kfunc(u32 btf_id);494static bool is_callback_calling_kfunc(u32 btf_id);
@@ -497,22 +547,6 @@ bool bpf_is_may_goto_insn(struct bpf_insn *insn) · 52 unmodified lines
497 return insn->code == (BPF_JMP | BPF_JCOND) && insn->src_reg == BPF_MAY_GOTO;547 return insn->code == (BPF_JMP | BPF_JCOND) && insn->src_reg == BPF_MAY_GOTO;
498}548}
499 549
500static bool helper_multiple_ref_obj_use(enum bpf_func_id func_id,
501 const struct bpf_map *map)
502{
503 int ref_obj_uses = 0;
504
505 if (is_ptr_cast_function(func_id))
506 ref_obj_uses++;
507 if (is_acquire_function(func_id, map))
508 ref_obj_uses++;
509 if (is_dynptr_ref_function(func_id))
510 ref_obj_uses++;
511
512 return ref_obj_uses > 1;
513}
514
515
516static bool is_spi_bounds_valid(struct bpf_func_state *state, int spi, int nr_slots)550static bool is_spi_bounds_valid(struct bpf_func_state *state, int spi, int nr_slots)
517{551{
518 int allocated_slots = state->allocated_stack / BPF_REG_SIZE;552 int allocated_slots = state->allocated_stack / BPF_REG_SIZE;
@@ -609,43 +643,44 @@ static enum bpf_type_flag get_dynptr_type_flag(enum bpf_dynptr_type type) · 90 unmodified lines
609 }643 }
610}644}
611 645
612static bool dynptr_type_refcounted(enum bpf_dynptr_type type)646static bool dynptr_type_referenced(enum bpf_dynptr_type type)
613{647{
614 return type == BPF_DYNPTR_TYPE_RINGBUF || type == BPF_DYNPTR_TYPE_FILE;648 return type == BPF_DYNPTR_TYPE_RINGBUF || type == BPF_DYNPTR_TYPE_FILE;
615}649}
616 650
617static void __mark_dynptr_reg(struct bpf_reg_state *reg,651static void __mark_dynptr_reg(struct bpf_reg_state *reg,
618 enum bpf_dynptr_type type,652 enum bpf_dynptr_type type,
619 bool first_slot, int dynptr_id);653 bool first_slot, int id, int parent_id);
620 654
621 655
622static void mark_dynptr_stack_regs(struct bpf_verifier_env *env,656static void mark_dynptr_stack_regs(struct bpf_verifier_env *env,
623 struct bpf_reg_state *sreg1,657 struct bpf_reg_state *sreg1,
624 struct bpf_reg_state *sreg2,658 struct bpf_reg_state *sreg2,
625 enum bpf_dynptr_type type)659 enum bpf_dynptr_type type, int parent_id)
626{660{
627 int id = ++env->id_gen;661 int id = ++env->id_gen;
628 662
629 __mark_dynptr_reg(sreg1, type, true, id);663 __mark_dynptr_reg(sreg1, type, true, id, parent_id);
630 __mark_dynptr_reg(sreg2, type, false, id);664 __mark_dynptr_reg(sreg2, type, false, id, parent_id);
631}665}
632 666
633static void mark_dynptr_cb_reg(struct bpf_verifier_env *env,667static void mark_dynptr_cb_reg(struct bpf_verifier_env *env,
634 struct bpf_reg_state *reg,668 struct bpf_reg_state *reg,
635 enum bpf_dynptr_type type)669 enum bpf_dynptr_type type)
636{670{
637 __mark_dynptr_reg(reg, type, true, ++env->id_gen);671 __mark_dynptr_reg(reg, type, true, ++env->id_gen, 0);
638}672}
639 673
640static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,674static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,
641 struct bpf_func_state *state, int spi);675 struct bpf_func_state *state, int spi);
642 676
643static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg,677static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
644 enum bpf_arg_type arg_type, int insn_idx, int clone_ref_obj_id)678 enum bpf_arg_type arg_type, int insn_idx,
679 struct ref_obj_desc *ref_obj, struct bpf_dynptr_desc *dynptr)
645{680{
646 struct bpf_func_state *state = bpf_func(env, reg);681 struct bpf_func_state *state = bpf_func(env, reg);
682 int spi, i, err, parent_id = 0;
647 enum bpf_dynptr_type type;683 enum bpf_dynptr_type type;
648 int spi, i, err;
649 684
650 spi = dynptr_get_spi(env, reg);685 spi = dynptr_get_spi(env, reg);
651 if (spi < 0)686 if (spi < 0)
@@ -676,94 +711,69 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_ · 24 unmodified lines
676 if (type == BPF_DYNPTR_TYPE_INVALID)711 if (type == BPF_DYNPTR_TYPE_INVALID)
677 return -EINVAL;712 return -EINVAL;
678 713
679 mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr,714 if (dynptr->type == BPF_DYNPTR_TYPE_INVALID) { /* dynptr constructors */
680 &state->stack[spi - 1].spilled_ptr, type);715 err = validate_ref_obj(env, ref_obj);
716 if (err)
717 return err;
681 718
682 if (dynptr_type_refcounted(type)) {719 /* Track parent's id if the parent is a referenced object */
683 /* The id is used to track proper releasing */720 parent_id = ref_obj->id;
684 int id;
685 721
686 if (clone_ref_obj_id)722 if (dynptr_type_referenced(type)) {
687 id = clone_ref_obj_id;723 int id;
688 else
689 id = acquire_reference(env, insn_idx);
690 724
691 if (id < 0)725 /*
692 return id;726 * Create an intermediate reference that tracks the referenced
727 * object for the referenced dynptr. Freeing a referenced dynptr
728 * through helpers/kfuncs will invalidate all clones.
729 */
730 id = acquire_reference(env, insn_idx, parent_id);
731 if (id < 0)
732 return id;
693 733
694 state->stack[spi].spilled_ptr.ref_obj_id = id;734 parent_id = id;
695 state->stack[spi - 1].spilled_ptr.ref_obj_id = id;735 }
736 } else { /* bpf_dynptr_clone() */
737 parent_id = dynptr->parent_id;
696 }738 }
697 739
740 mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr,
741 &state->stack[spi - 1].spilled_ptr, type, parent_id);
742
698 return 0;743 return 0;
699}744}
700 745
701static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_state *state, int spi)746static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_stack_state *stack)
702{747{
703 int i;748 int i;
704 749
705 for (i = 0; i < BPF_REG_SIZE; i++) {750 for (i = 0; i < BPF_REG_SIZE; i++) {
706 state->stack[spi].slot_type[i] = STACK_INVALID;751 stack[0].slot_type[i] = STACK_INVALID;
707 state->stack[spi - 1].slot_type[i] = STACK_INVALID;752 stack[1].slot_type[i] = STACK_INVALID;
708 }753 }
709 754
710 bpf_mark_reg_not_init(env, &state->stack[spi].spilled_ptr);755 bpf_mark_reg_not_init(env, &stack[0].spilled_ptr);
711 bpf_mark_reg_not_init(env, &state->stack[spi - 1].spilled_ptr);756 bpf_mark_reg_not_init(env, &stack[1].spilled_ptr);
712}757}
713 758
714static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg)759static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
715{760{
716 struct bpf_func_state *state = bpf_func(env, reg);761 struct bpf_func_state *state = bpf_func(env, reg);
717 int spi, ref_obj_id, i;762 int spi;
718 763
719 /*
720 * This can only be set for PTR_TO_STACK, as CONST_PTR_TO_DYNPTR cannot
721 * be released by any dynptr helper. Hence, unmark_stack_slots_dynptr
722 * is safe to do directly.
723 */
724 if (reg->type == CONST_PTR_TO_DYNPTR) {
725 verifier_bug(env, "CONST_PTR_TO_DYNPTR cannot be released");
726 return -EFAULT;
727 }
728 spi = dynptr_get_spi(env, reg);764 spi = dynptr_get_spi(env, reg);
729 if (spi < 0)765 if (spi < 0)
730 return spi;766 return spi;
731 767
732 if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) {768 /*
733 invalidate_dynptr(env, state, spi);769 * For referenced dynptr, release the parent ref which cascades to
734 return 0;770 * all clones and derived slices. For non-referenced dynptr, only
735 }771 * the dynptr and slices derived from it will be invalidated.
736
737 ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
738
739 /* If the dynptr has a ref_obj_id, then we need to invalidate
740 * two things:
741 *
742 * 1) Any dynptrs with a matching ref_obj_id (clones)
743 * 2) Any slices derived from this dynptr.
744 */772 */
745 773 reg = &state->stack[spi].spilled_ptr;
746 /* Invalidate any slices associated with this dynptr */774 return release_reference(env, dynptr_type_referenced(reg->dynptr.type)
747 WARN_ON_ONCE(release_reference(env, ref_obj_id));775 ? reg->parent_id
748 776 : reg->id);
749 /* Invalidate any dynptr clones */
750 for (i = 1; i < state->allocated_stack / BPF_REG_SIZE; i++) {
751 if (state->stack[i].spilled_ptr.ref_obj_id != ref_obj_id)
752 continue;
753
754 /* it should always be the case that if the ref obj id
755 * matches then the stack slot also belongs to a
756 * dynptr
757 */
758 if (state->stack[i].slot_type[0] != STACK_DYNPTR) {
759 verifier_bug(env, "misconfigured ref_obj_id");
760 return -EFAULT;
761 }
762 if (state->stack[i].spilled_ptr.dynptr.first_slot)
763 invalidate_dynptr(env, state, i);
764 }
765
766 return 0;
767}777}
768 778
769static void __mark_reg_unknown(const struct bpf_verifier_env *env,779static void __mark_reg_unknown(const struct bpf_verifier_env *env,
@@ -777,12 +787,29 @@ static void mark_reg_invalid(const struct bpf_verifier_env *env, struct bpf_reg_ · 7 unmodified lines
777 __mark_reg_unknown(env, reg);787 __mark_reg_unknown(env, reg);
778}788}
779 789
790static int dynptr_ref_cnt(struct bpf_verifier_env *env, int v_parent_id)
791{
792 struct bpf_stack_state *stack;
793 struct bpf_func_state *state;
794 struct bpf_reg_state *reg;
795 int ref_cnt = 0;
796
797 bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, 1 << STACK_DYNPTR, ({
798 if (!stack || stack->slot_type[0] != STACK_DYNPTR)
799 continue;
800 if (!stack->spilled_ptr.dynptr.first_slot)
801 continue;
802 if (stack->spilled_ptr.parent_id == v_parent_id)
803 ref_cnt++;
804 }));
805
806 return ref_cnt;
807}
808
780static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,809static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,
781 struct bpf_func_state *state, int spi)810 struct bpf_func_state *state, int spi)
782{811{
783 struct bpf_func_state *fstate;812 int err = 0;
784 struct bpf_reg_state *dreg;
785 int i, dynptr_id;
786 813
787 /* We always ensure that STACK_DYNPTR is never set partially,814 /* We always ensure that STACK_DYNPTR is never set partially,
788 * hence just checking for slot_type[0] is enough. This is815 * hence just checking for slot_type[0] is enough. This is
@@ -796,56 +823,25 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, · 7 unmodified lines
796 if (!state->stack[spi].spilled_ptr.dynptr.first_slot)823 if (!state->stack[spi].spilled_ptr.dynptr.first_slot)
797 spi = spi + 1;824 spi = spi + 1;
798 825
799 if (dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) {826 /*
800 int ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;827 * A referenced dynptr can be overwritten only if there is at
801 int ref_cnt = 0;828 * least one other dynptr sharing the same virtual ref parent,
802 829 * ensuring the reference can still be properly released.
803 /*830 */
804 * A referenced dynptr can be overwritten only if there is at831 if (dynptr_type_referenced(state->stack[spi].spilled_ptr.dynptr.type) &&
805 * least one other dynptr sharing the same ref_obj_id,832 dynptr_ref_cnt(env, state->stack[spi].spilled_ptr.parent_id) <= 1) {
806 * ensuring the reference can still be properly released.833 verbose(env, "cannot overwrite referenced dynptr\n");
807 */834 return -EINVAL;
808 for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) {
809 if (state->stack[i].slot_type[0] != STACK_DYNPTR)
810 continue;
811 if (!state->stack[i].spilled_ptr.dynptr.first_slot)
812 continue;
813 if (state->stack[i].spilled_ptr.ref_obj_id == ref_obj_id)
814 ref_cnt++;
815 }
816
817 if (ref_cnt <= 1) {
818 verbose(env, "cannot overwrite referenced dynptr\n");
819 return -EINVAL;
820 }
821 }835 }
822 836
823 mark_stack_slot_scratched(env, spi);837 /* Invalidate the dynptr and any derived slices */
824 mark_stack_slot_scratched(env, spi - 1);838 err = release_reference(env, state->stack[spi].spilled_ptr.id);
825 839 if (!err) {
826 /* Writing partially to one dynptr stack slot destroys both. */840 mark_stack_slot_scratched(env, spi);
827 for (i = 0; i < BPF_REG_SIZE; i++) {841 mark_stack_slot_scratched(env, spi - 1);
828 state->stack[spi].slot_type[i] = STACK_INVALID;
829 state->stack[spi - 1].slot_type[i] = STACK_INVALID;
830 }842 }
831 843
832 dynptr_id = state->stack[spi].spilled_ptr.id;844 return err;
833 /* Invalidate any slices associated with this dynptr */
834 bpf_for_each_reg_in_vstate(env->cur_state, fstate, dreg, ({
835 /* Dynptr slices are only PTR_TO_MEM_OR_NULL and PTR_TO_MEM */
836 if (dreg->type != (PTR_TO_MEM | PTR_MAYBE_NULL) && dreg->type != PTR_TO_MEM)
837 continue;
838 if (dreg->dynptr_id == dynptr_id)
839 mark_reg_invalid(env, dreg);
840 }));
841
842 /* Do not release reference state, we are destroying dynptr on stack,
843 * not using some helper to release it. Just reset register.
844 */
845 bpf_mark_reg_not_init(env, &state->stack[spi].spilled_ptr);
846 bpf_mark_reg_not_init(env, &state->stack[spi - 1].spilled_ptr);
847
848 return 0;
849}845}
850 846
851static bool is_dynptr_reg_valid_uninit(struct bpf_verifier_env *env, struct bpf_reg_state *reg)847static bool is_dynptr_reg_valid_uninit(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
@@ -945,7 +941,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env, · 93 unmodified lines
945 if (spi < 0)941 if (spi < 0)
946 return spi;942 return spi;
947 943
948 id = acquire_reference(env, insn_idx);944 id = acquire_reference(env, insn_idx, 0);
949 if (id < 0)945 if (id < 0)
950 return id;946 return id;
951 947
@@ -961,7 +957,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env, · 9 unmodified lines
961 else957 else
962 st->type |= PTR_UNTRUSTED;958 st->type |= PTR_UNTRUSTED;
963 }959 }
964 st->ref_obj_id = i == 0 ? id : 0;960 st->id = i == 0 ? id : 0;
965 st->iter.btf = btf;961 st->iter.btf = btf;
966 st->iter.btf_id = btf_id;962 st->iter.btf_id = btf_id;
967 st->iter.state = BPF_ITER_STATE_ACTIVE;963 st->iter.state = BPF_ITER_STATE_ACTIVE;
@@ -991,7 +987,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env, · 23 unmodified lines
991 struct bpf_reg_state *st = &slot->spilled_ptr;987 struct bpf_reg_state *st = &slot->spilled_ptr;
992 988
993 if (i == 0)989 if (i == 0)
994 WARN_ON_ONCE(release_reference(env, st->ref_obj_id));990 WARN_ON_ONCE(release_reference(env, st->id));
995 991
996 bpf_mark_reg_not_init(env, st);992 bpf_mark_reg_not_init(env, st);
997 993
@@ -1047,10 +1043,10 @@ static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s · 49 unmodified lines
1047 1043
1048 if (st->type & PTR_UNTRUSTED)1044 if (st->type & PTR_UNTRUSTED)
1049 return -EPROTO;1045 return -EPROTO;
1050 /* only main (first) slot has ref_obj_id set */1046 /* only main (first) slot has id set */
1051 if (i == 0 && !st->ref_obj_id)1047 if (i == 0 && !st->id)
1052 return -EINVAL;1048 return -EINVAL;
1053 if (i != 0 && st->ref_obj_id)1049 if (i != 0 && st->id)
1054 return -EINVAL;1050 return -EINVAL;
1055 if (st->iter.btf != btf || st->iter.btf_id != btf_id)1051 if (st->iter.btf != btf || st->iter.btf_id != btf_id)
1056 return -EINVAL;1052 return -EINVAL;
@@ -1089,7 +1085,7 @@ static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env, · 32 unmodified lines
1089 1085
1090 __mark_reg_known_zero(st);1086 __mark_reg_known_zero(st);
1091 st->type = PTR_TO_STACK; /* we don't have dedicated reg type */1087 st->type = PTR_TO_STACK; /* we don't have dedicated reg type */
1092 st->ref_obj_id = id;1088 st->id = id;
1093 st->irq.kfunc_class = kfunc_class;1089 st->irq.kfunc_class = kfunc_class;
1094 1090
1095 for (i = 0; i < BPF_REG_SIZE; i++)1091 for (i = 0; i < BPF_REG_SIZE; i++)
@@ -1123,7 +1119,7 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r · 27 unmodified lines
1123 return -EINVAL;1119 return -EINVAL;
1124 }1120 }
1125 1121
1126 err = release_irq_state(env->cur_state, st->ref_obj_id);1122 err = release_irq_state(env->cur_state, st->id);
1127 WARN_ON_ONCE(err && err != -EACCES);1123 WARN_ON_ONCE(err && err != -EACCES);
1128 if (err) {1124 if (err) {
1129 int insn_idx = 0;1125 int insn_idx = 0;
@@ -1187,7 +1183,7 @@ static int is_irq_flag_reg_valid_init(struct bpf_verifier_env *env, struct bpf_r · 57 unmodified lines
1187 slot = &state->stack[spi];1183 slot = &state->stack[spi];
1188 st = &slot->spilled_ptr;1184 st = &slot->spilled_ptr;
1189 1185
1190 if (!st->ref_obj_id)1186 if (!st->id)
1191 return -EINVAL;1187 return -EINVAL;
1192 1188
1193 for (i = 0; i < BPF_REG_SIZE; i++)1189 for (i = 0; i < BPF_REG_SIZE; i++)
@@ -1339,6 +1335,18 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st · 145 unmodified lines
1339 return -ENOMEM;1335 return -ENOMEM;
1340 1336
1341 dst->allocated_stack = src->allocated_stack;1337 dst->allocated_stack = src->allocated_stack;
1338
1339 /* copy stack args state */
1340 n = src->out_stack_arg_cnt;
1341 if (n) {
1342 dst->stack_arg_regs = copy_array(dst->stack_arg_regs, src->stack_arg_regs, n,
1343 sizeof(struct bpf_reg_state),
1344 GFP_KERNEL_ACCOUNT);
1345 if (!dst->stack_arg_regs)
1346 return -ENOMEM;
1347 }
1348
1349 dst->out_stack_arg_cnt = src->out_stack_arg_cnt;
1342 return 0;1350 return 0;
1343}1351}
1344 1352
@@ -1380,6 +1388,23 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state · 35 unmodified lines
1380 return 0;1388 return 0;
1381}1389}
1382 1390
1391static int grow_stack_arg_slots(struct bpf_verifier_env *env,
1392 struct bpf_func_state *state, int cnt)
1393{
1394 size_t old_n = state->out_stack_arg_cnt;
1395
1396 if (old_n >= cnt)
1397 return 0;
1398
1399 state->stack_arg_regs = realloc_array(state->stack_arg_regs, old_n, cnt,
1400 sizeof(struct bpf_reg_state));
1401 if (!state->stack_arg_regs)
1402 return -ENOMEM;
1403
1404 state->out_stack_arg_cnt = cnt;
1405 return 0;
1406}
1407
1383/* Acquire a pointer id from the env and update the state->refs to include1408/* Acquire a pointer id from the env and update the state->refs to include
1384 * this new pointer reference.1409 * this new pointer reference.
1385 * On success, returns a valid pointer id to associate with the register1410 * On success, returns a valid pointer id to associate with the register
@@ -1399,7 +1424,7 @@ static struct bpf_reference_state *acquire_reference_state(struct bpf_verifier_e · 13 unmodified lines
1399 return &state->refs[new_ofs];1424 return &state->refs[new_ofs];
1400}1425}
1401 1426
1402static int acquire_reference(struct bpf_verifier_env *env, int insn_idx)1427static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int parent_id)
1403{1428{
1404 struct bpf_reference_state *s;1429 struct bpf_reference_state *s;
1405 1430
@@ -1408,6 +1433,7 @@ static int acquire_reference(struct bpf_verifier_env *env, int insn_idx) · 2 unmodified lines
1408 return -ENOMEM;1433 return -ENOMEM;
1409 s->type = REF_TYPE_PTR;1434 s->type = REF_TYPE_PTR;
1410 s->id = ++env->id_gen;1435 s->id = ++env->id_gen;
1436 s->parent_id = parent_id;
1411 return s->id;1437 return s->id;
1412}1438}
1413 1439
@@ -1464,17 +1490,25 @@ static void release_reference_state(struct bpf_verifier_state *state, int idx) · 50 unmodified lines
1464 return;1490 return;
1465}1491}
1466 1492
1467static bool find_reference_state(struct bpf_verifier_state *state, int ptr_id)1493static bool find_reference_state(struct bpf_verifier_state *state, int id)
1468{1494{
1469 int i;1495 int i;
1470 1496
1471 for (i = 0; i < state->acquired_refs; i++)1497 for (i = 0; i < state->acquired_refs; i++) {
1472 if (state->refs[i].id == ptr_id)1498 if (state->refs[i].type != REF_TYPE_PTR)
1499 continue;
1500 if (state->refs[i].id == id)
1473 return true;1501 return true;
1502 }
1474 1503
1475 return false;1504 return false;
1476}1505}
1477 1506
1507static bool reg_is_referenced(struct bpf_verifier_env *env, const struct bpf_reg_state *reg)
1508{
1509 return find_reference_state(env->cur_state, reg->id);
1510}
1511
1478static int release_lock_state(struct bpf_verifier_state *state, int type, int id, void *ptr)1512static int release_lock_state(struct bpf_verifier_state *state, int type, int id, void *ptr)
1479{1513{
1480 void *prev_ptr = NULL;1514 void *prev_ptr = NULL;
@@ -1542,6 +1576,7 @@ static void free_func_state(struct bpf_func_state *state) · 61 unmodified lines
1542{1576{
1543 if (!state)1577 if (!state)
1544 return;1578 return;
1579 kfree(state->stack_arg_regs);
1545 kfree(state->stack);1580 kfree(state->stack);
1546 kfree(state);1581 kfree(state);
1547}1582}
@@ -1750,6 +1785,22 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env, · 202 unmodified lines
1750 return &elem->st;1785 return &elem->st;
1751}1786}
1752 1787
1788static const char *reg_arg_name(struct bpf_verifier_env *env, argno_t argno)
1789{
1790 char *buf = env->tmp_arg_name;
1791 int len = sizeof(env->tmp_arg_name);
1792 int arg, regno = reg_from_argno(argno);
1793
1794 if (regno >= 0) {
1795 snprintf(buf, len, "R%d", regno);
1796 } else {
1797 arg = arg_from_argno(argno);
1798 snprintf(buf, len, "*(R11-%u)", (arg - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE);
1799 }
1800
1801 return buf;
1802}
1803
1753static const int caller_saved[CALLER_SAVED_REGS] = {1804static const int caller_saved[CALLER_SAVED_REGS] = {
1754 BPF_REG_0, BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_51805 BPF_REG_0, BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_5
1755};1806};
@@ -1758,15 +1809,8 @@ static const int caller_saved[CALLER_SAVED_REGS] = { · 2 unmodified lines
1758static void ___mark_reg_known(struct bpf_reg_state *reg, u64 imm)1809static void ___mark_reg_known(struct bpf_reg_state *reg, u64 imm)
1759{1810{
1760 reg->var_off = tnum_const(imm);1811 reg->var_off = tnum_const(imm);
1761 reg->smin_value = (s64)imm;1812 reg->r64 = cnum64_from_urange(imm, imm);
1762 reg->smax_value = (s64)imm;1813 reg->r32 = cnum32_from_urange((u32)imm, (u32)imm);
1763 reg->umin_value = imm;
1764 reg->umax_value = imm;
1765
1766 reg->s32_min_value = (s32)imm;
1767 reg->s32_max_value = (s32)imm;
1768 reg->u32_min_value = (u32)imm;
1769 reg->u32_max_value = (u32)imm;
1770}1814}
1771 1815
1772/* Mark the unknown part of a register (variable offset or scalar value) as1816/* Mark the unknown part of a register (variable offset or scalar value) as
@@ -1778,17 +1822,14 @@ static void __mark_reg_known(struct bpf_reg_state *reg, u64 imm) · 5 unmodified lines
1778 memset(((u8 *)reg) + sizeof(reg->type), 0,1822 memset(((u8 *)reg) + sizeof(reg->type), 0,
1779 offsetof(struct bpf_reg_state, var_off) - sizeof(reg->type));1823 offsetof(struct bpf_reg_state, var_off) - sizeof(reg->type));
1780 reg->id = 0;1824 reg->id = 0;
1781 reg->ref_obj_id = 0;1825 reg->parent_id = 0;
1782 ___mark_reg_known(reg, imm);1826 ___mark_reg_known(reg, imm);
1783}1827}
1784 1828
1785static void __mark_reg32_known(struct bpf_reg_state *reg, u64 imm)1829static void __mark_reg32_known(struct bpf_reg_state *reg, u64 imm)
1786{1830{
1787 reg->var_off = tnum_const_subreg(reg->var_off, imm);1831 reg->var_off = tnum_const_subreg(reg->var_off, imm);
1788 reg->s32_min_value = (s32)imm;1832 reg->r32 = cnum32_from_urange((u32)imm, (u32)imm);
1789 reg->s32_max_value = (s32)imm;
1790 reg->u32_min_value = (u32)imm;
1791 reg->u32_max_value = (u32)imm;
1792}1833}
1793 1834
1794/* Mark the 'variable offset' part of a register as zero. This should be1835/* Mark the 'variable offset' part of a register as zero. This should be
@@ -1816,7 +1857,7 @@ static void mark_reg_known_zero(struct bpf_verifier_env *env, · 21 unmodified lines
1816}1857}
1817 1858
1818static void __mark_dynptr_reg(struct bpf_reg_state *reg, enum bpf_dynptr_type type,1859static void __mark_dynptr_reg(struct bpf_reg_state *reg, enum bpf_dynptr_type type,
1819 bool first_slot, int dynptr_id)1860 bool first_slot, int id, int parent_id)
1820{1861{
1821 /* reg->type has no meaning for STACK_DYNPTR, but when we set reg for1862 /* reg->type has no meaning for STACK_DYNPTR, but when we set reg for
1822 * callback arguments, it does need to be CONST_PTR_TO_DYNPTR, so simply1863 * callback arguments, it does need to be CONST_PTR_TO_DYNPTR, so simply
@@ -1825,7 +1866,8 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg, enum bpf_dynptr_type ty · 2 unmodified lines
1825 __mark_reg_known_zero(reg);1866 __mark_reg_known_zero(reg);
1826 reg->type = CONST_PTR_TO_DYNPTR;1867 reg->type = CONST_PTR_TO_DYNPTR;
1827 /* Give each dynptr a unique id to uniquely associate slices to it. */1868 /* Give each dynptr a unique id to uniquely associate slices to it. */
1828 reg->id = dynptr_id;1869 reg->id = id;
1870 reg->parent_id = parent_id;
1829 reg->dynptr.type = type;1871 reg->dynptr.type = type;
1830 reg->dynptr.first_slot = first_slot;1872 reg->dynptr.first_slot = first_slot;
1831}1873}
@@ -1899,34 +1941,21 @@ static bool reg_is_init_pkt_pointer(const struct bpf_reg_state *reg, · 67 unmodified lines
1899 tnum_equals_const(reg->var_off, 0);1941 tnum_equals_const(reg->var_off, 0);
1900}1942}
1901 1943
1902/* Reset the min/max bounds of a register */1944static void __mark_reg32_unbounded(struct bpf_reg_state *reg)
1903static void __mark_reg_unbounded(struct bpf_reg_state *reg)
1904{1945{
1905 reg->smin_value = S64_MIN;1946 reg->r32 = CNUM32_UNBOUNDED;
1906 reg->smax_value = S64_MAX;
1907 reg->umin_value = 0;
1908 reg->umax_value = U64_MAX;
1909
1910 reg->s32_min_value = S32_MIN;
1911 reg->s32_max_value = S32_MAX;
1912 reg->u32_min_value = 0;
1913 reg->u32_max_value = U32_MAX;
1914}1947}
1915 1948
1916static void __mark_reg64_unbounded(struct bpf_reg_state *reg)1949static void __mark_reg64_unbounded(struct bpf_reg_state *reg)
1917{1950{
1918 reg->smin_value = S64_MIN;1951 reg->r64 = CNUM64_UNBOUNDED;
1919 reg->smax_value = S64_MAX;
1920 reg->umin_value = 0;
1921 reg->umax_value = U64_MAX;
1922}1952}
1923 1953
1924static void __mark_reg32_unbounded(struct bpf_reg_state *reg)1954/* Reset the min/max bounds of a register */
1955static void __mark_reg_unbounded(struct bpf_reg_state *reg)
1925{1956{
1926 reg->s32_min_value = S32_MIN;1957 __mark_reg64_unbounded(reg);
1927 reg->s32_max_value = S32_MAX;1958 __mark_reg32_unbounded(reg);
1928 reg->u32_min_value = 0;
1929 reg->u32_max_value = U32_MAX;
1930}1959}
1931 1960
1932static void reset_reg64_and_tnum(struct bpf_reg_state *reg)1961static void reset_reg64_and_tnum(struct bpf_reg_state *reg)
@@ -1941,19 +1970,32 @@ static void reset_reg32_and_tnum(struct bpf_reg_state *reg) · 8 unmodified lines
1941 reg->var_off = tnum_unknown;1970 reg->var_off = tnum_unknown;
1942}1971}
1943 1972
1944static void __update_reg32_bounds(struct bpf_reg_state *reg)1973static struct cnum32 cnum32_from_tnum(struct tnum tnum)
1945{1974{
1946 struct tnum var32_off = tnum_subreg(reg->var_off);1975 tnum = tnum_subreg(tnum);
1976 if ((tnum.mask & S32_MIN) || (tnum.value & S32_MIN))
1977 /* min signed is max(sign bit) | min(other bits) */
1978 /* max signed is min(sign bit) | max(other bits) */
1979 return cnum32_from_srange(tnum.value | (tnum.mask & S32_MIN),
1980 tnum.value | (tnum.mask & S32_MAX));
1981 else
1982 return cnum32_from_urange(tnum.value, (tnum.value | tnum.mask));
1983}
1947 1984
1948 /* min signed is max(sign bit) | min(other bits) */1985static struct cnum64 cnum64_from_tnum(struct tnum tnum)
1949 reg->s32_min_value = max_t(s32, reg->s32_min_value,1986{
1950 var32_off.value | (var32_off.mask & S32_MIN));1987 if ((tnum.mask & S64_MIN) || (tnum.value & S64_MIN))
1951 /* max signed is min(sign bit) | max(other bits) */1988 /* min signed is max(sign bit) | min(other bits) */
1952 reg->s32_max_value = min_t(s32, reg->s32_max_value,1989 /* max signed is min(sign bit) | max(other bits) */
1953 var32_off.value | (var32_off.mask & S32_MAX));1990 return cnum64_from_srange(tnum.value | (tnum.mask & S64_MIN),
1954 reg->u32_min_value = max_t(u32, reg->u32_min_value, (u32)var32_off.value);1991 tnum.value | (tnum.mask & S64_MAX));
1955 reg->u32_max_value = min(reg->u32_max_value,1992 else
1956 (u32)(var32_off.value | var32_off.mask));1993 return cnum64_from_urange(tnum.value, (tnum.value | tnum.mask));
1994}
1995
1996static void __update_reg32_bounds(struct bpf_reg_state *reg)
1997{
1998 cnum32_intersect_with(&reg->r32, cnum32_from_tnum(reg->var_off));
1957}1999}
1958 2000
1959static void __update_reg64_bounds(struct bpf_reg_state *reg)2001static void __update_reg64_bounds(struct bpf_reg_state *reg)
@@ -1961,26 +2003,18 @@ static void __update_reg64_bounds(struct bpf_reg_state *reg) · 1 unmodified line
1961 u64 tnum_next, tmax;2003 u64 tnum_next, tmax;
1962 bool umin_in_tnum;2004 bool umin_in_tnum;
1963 2005
1964 /* min signed is max(sign bit) | min(other bits) */2006 cnum64_intersect_with(&reg->r64, cnum64_from_tnum(reg->var_off));
1965 reg->smin_value = max_t(s64, reg->smin_value,
1966 reg->var_off.value | (reg->var_off.mask & S64_MIN));
1967 /* max signed is min(sign bit) | max(other bits) */
1968 reg->smax_value = min_t(s64, reg->smax_value,
1969 reg->var_off.value | (reg->var_off.mask & S64_MAX));
1970 reg->umin_value = max(reg->umin_value, reg->var_off.value);
1971 reg->umax_value = min(reg->umax_value,
1972 reg->var_off.value | reg->var_off.mask);
1973 2007
1974 /* Check if u64 and tnum overlap in a single value */2008 /* Check if u64 and tnum overlap in a single value */
1975 tnum_next = tnum_step(reg->var_off, reg->umin_value);2009 tnum_next = tnum_step(reg->var_off, reg_umin(reg));
1976 umin_in_tnum = (reg->umin_value & ~reg->var_off.mask) == reg->var_off.value;2010 umin_in_tnum = (reg_umin(reg) & ~reg->var_off.mask) == reg->var_off.value;
1977 tmax = reg->var_off.value | reg->var_off.mask;2011 tmax = reg->var_off.value | reg->var_off.mask;
1978 if (umin_in_tnum && tnum_next > reg->umax_value) {2012 if (umin_in_tnum && tnum_next > reg_umax(reg)) {
1979 /* The u64 range and the tnum only overlap in umin.2013 /* The u64 range and the tnum only overlap in umin.
1980 * u64: ---[xxxxxx]-----2014 * u64: ---[xxxxxx]-----
1981 * tnum: --xx----------x-2015 * tnum: --xx----------x-
1982 */2016 */
1983 ___mark_reg_known(reg, reg->umin_value);2017 ___mark_reg_known(reg, reg_umin(reg));
1984 } else if (!umin_in_tnum && tnum_next == tmax) {2018 } else if (!umin_in_tnum && tnum_next == tmax) {
1985 /* The u64 range and the tnum only overlap in the maximum value2019 /* The u64 range and the tnum only overlap in the maximum value
1986 * represented by the tnum, called tmax.2020 * represented by the tnum, called tmax.
@@ -1988,8 +2022,8 @@ static void __update_reg64_bounds(struct bpf_reg_state *reg) · 1 unmodified line
1988 * tnum: xx-----x--------2022 * tnum: xx-----x--------
1989 */2023 */
1990 ___mark_reg_known(reg, tmax);2024 ___mark_reg_known(reg, tmax);
1991 } else if (!umin_in_tnum && tnum_next <= reg->umax_value &&2025 } else if (!umin_in_tnum && tnum_next <= reg_umax(reg) &&
1992 tnum_step(reg->var_off, tnum_next) > reg->umax_value) {2026 tnum_step(reg->var_off, tnum_next) > reg_umax(reg)) {
1993 /* The u64 range and the tnum only overlap in between umin2027 /* The u64 range and the tnum only overlap in between umin
1994 * (excluded) and umax.2028 * (excluded) and umax.
1995 * u64: ---[xxxxxx]-----2029 * u64: ---[xxxxxx]-----
@@ -2005,329 +2039,19 @@ static void __update_reg_bounds(struct bpf_reg_state *reg) · 9 unmodified lines
2005 __update_reg64_bounds(reg);2039 __update_reg64_bounds(reg);
2006}2040}
2007 2041
2008/* Uses signed min/max values to inform unsigned, and vice-versa */
2009static void deduce_bounds_32_from_64(struct bpf_reg_state *reg)2042static void deduce_bounds_32_from_64(struct bpf_reg_state *reg)
2010{2043{
2011 /* If upper 32 bits of u64/s64 range don't change, we can use lower 322044 cnum32_intersect_with(&reg->r32, cnum32_from_cnum64(reg->r64));
2012 * bits to improve our u32/s32 boundaries.
2013 *
2014 * E.g., the case where we have upper 32 bits as zero ([10, 20] in
2015 * u64) is pretty trivial, it's obvious that in u32 we'll also have
2016 * [10, 20] range. But this property holds for any 64-bit range as
2017 * long as upper 32 bits in that entire range of values stay the same.
2018 *
2019 * E.g., u64 range [0x10000000A, 0x10000000F] ([4294967306, 4294967311]
2020 * in decimal) has the same upper 32 bits throughout all the values in
2021 * that range. As such, lower 32 bits form a valid [0xA, 0xF] ([10, 15])
2022 * range.
2023 *
2024 * Note also, that [0xA, 0xF] is a valid range both in u32 and in s32,
2025 * following the rules outlined below about u64/s64 correspondence
2026 * (which equally applies to u32 vs s32 correspondence). In general it
2027 * depends on actual hexadecimal values of 32-bit range. They can form
2028 * only valid u32, or only valid s32 ranges in some cases.
2029 *
2030 * So we use all these insights to derive bounds for subregisters here.
2031 */
2032 if ((reg->umin_value >> 32) == (reg->umax_value >> 32)) {
2033 /* u64 to u32 casting preserves validity of low 32 bits as
2034 * a range, if upper 32 bits are the same
2035 */
2036 reg->u32_min_value = max_t(u32, reg->u32_min_value, (u32)reg->umin_value);
2037 reg->u32_max_value = min_t(u32, reg->u32_max_value, (u32)reg->umax_value);
2038
2039 if ((s32)reg->umin_value <= (s32)reg->umax_value) {
2040 reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->umin_value);
2041 reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->umax_value);
2042 }
2043 }
2044 if ((reg->smin_value >> 32) == (reg->smax_value >> 32)) {
2045 /* low 32 bits should form a proper u32 range */
2046 if ((u32)reg->smin_value <= (u32)reg->smax_value) {
2047 reg->u32_min_value = max_t(u32, reg->u32_min_value, (u32)reg->smin_value);
2048 reg->u32_max_value = min_t(u32, reg->u32_max_value, (u32)reg->smax_value);
2049 }
2050 /* low 32 bits should form a proper s32 range */
2051 if ((s32)reg->smin_value <= (s32)reg->smax_value) {
2052 reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->smin_value);
2053 reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->smax_value);
2054 }
2055 }
2056 /* Special case where upper bits form a small sequence of two
2057 * sequential numbers (in 32-bit unsigned space, so 0xffffffff to
2058 * 0x00000000 is also valid), while lower bits form a proper s32 range
2059 * going from negative numbers to positive numbers. E.g., let's say we
2060 * have s64 range [-1, 1] ([0xffffffffffffffff, 0x0000000000000001]).
2061 * Possible s64 values are {-1, 0, 1} ({0xffffffffffffffff,
2062 * 0x0000000000000000, 0x00000000000001}). Ignoring upper 32 bits,
2063 * we still get a valid s32 range [-1, 1] ([0xffffffff, 0x00000001]).
2064 * Note that it doesn't have to be 0xffffffff going to 0x00000000 in
2065 * upper 32 bits. As a random example, s64 range
2066 * [0xfffffff0fffffff0; 0xfffffff100000010], forms a valid s32 range
2067 * [-16, 16] ([0xfffffff0; 0x00000010]) in its 32 bit subregister.
2068 */
2069 if ((u32)(reg->umin_value >> 32) + 1 == (u32)(reg->umax_value >> 32) &&
2070 (s32)reg->umin_value < 0 && (s32)reg->umax_value >= 0) {
2071 reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->umin_value);
2072 reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->umax_value);
2073 }
2074 if ((u32)(reg->smin_value >> 32) + 1 == (u32)(reg->smax_value >> 32) &&
2075 (s32)reg->smin_value < 0 && (s32)reg->smax_value >= 0) {
2076 reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->smin_value);
2077 reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->smax_value);
2078 }
2079}
2080
2081static void deduce_bounds_32_from_32(struct bpf_reg_state *reg)
2082{
2083 /* if u32 range forms a valid s32 range (due to matching sign bit),
2084 * try to learn from that
2085 */
2086 if ((s32)reg->u32_min_value <= (s32)reg->u32_max_value) {
2087 reg->s32_min_value = max_t(s32, reg->s32_min_value, reg->u32_min_value);
2088 reg->s32_max_value = min_t(s32, reg->s32_max_value, reg->u32_max_value);
2089 }
2090 /* If we cannot cross the sign boundary, then signed and unsigned bounds
2091 * are the same, so combine. This works even in the negative case, e.g.
2092 * -3 s<= x s<= -1 implies 0xf...fd u<= x u<= 0xf...ff.
2093 */
2094 if ((u32)reg->s32_min_value <= (u32)reg->s32_max_value) {
2095 reg->u32_min_value = max_t(u32, reg->s32_min_value, reg->u32_min_value);
2096 reg->u32_max_value = min_t(u32, reg->s32_max_value, reg->u32_max_value);
2097 } else {
2098 if (reg->u32_max_value < (u32)reg->s32_min_value) {
2099 /* See __reg64_deduce_bounds() for detailed explanation.
2100 * Refine ranges in the following situation:
2101 *
2102 * 0 U32_MAX
2103 * | [xxxxxxxxxxxxxx u32 range xxxxxxxxxxxxxx] |
2104 * |----------------------------|----------------------------|
2105 * |xxxxx s32 range xxxxxxxxx] [xxxxxxx|
2106 * 0 S32_MAX S32_MIN -1
2107 */
2108 reg->s32_min_value = (s32)reg->u32_min_value;
2109 reg->u32_max_value = min_t(u32, reg->u32_max_value, reg->s32_max_value);
2110 } else if ((u32)reg->s32_max_value < reg->u32_min_value) {
2111 /*
2112 * 0 U32_MAX
2113 * | [xxxxxxxxxxxxxx u32 range xxxxxxxxxxxxxx] |
2114 * |----------------------------|----------------------------|
2115 * |xxxxxxxxx] [xxxxxxxxxxxx s32 range |
2116 * 0 S32_MAX S32_MIN -1
2117 */
2118 reg->s32_max_value = (s32)reg->u32_max_value;
2119 reg->u32_min_value = max_t(u32, reg->u32_min_value, reg->s32_min_value);
2120 }
2121 }
2122}
2123
2124static void deduce_bounds_64_from_64(struct bpf_reg_state *reg)
2125{
2126 /* If u64 range forms a valid s64 range (due to matching sign bit),
2127 * try to learn from that. Let's do a bit of ASCII art to see when
2128 * this is happening. Let's take u64 range first:
2129 *
2130 * 0 0x7fffffffffffffff 0x8000000000000000 U64_MAX
2131 * |-------------------------------|--------------------------------|
2132 *
2133 * Valid u64 range is formed when umin and umax are anywhere in the
2134 * range [0, U64_MAX], and umin <= umax. u64 case is simple and
2135 * straightforward. Let's see how s64 range maps onto the same range
2136 * of values, annotated below the line for comparison:
2137 *
2138 * 0 0x7fffffffffffffff 0x8000000000000000 U64_MAX
2139 * |-------------------------------|--------------------------------|
2140 * 0 S64_MAX S64_MIN -1
2141 *
2142 * So s64 values basically start in the middle and they are logically
2143 * contiguous to the right of it, wrapping around from -1 to 0, and
2144 * then finishing as S64_MAX (0x7fffffffffffffff) right before
2145 * S64_MIN. We can try drawing the continuity of u64 vs s64 values
2146 * more visually as mapped to sign-agnostic range of hex values.
2147 *
2148 * u64 start u64 end
2149 * _______________________________________________________________
2150 * / \
2151 * 0 0x7fffffffffffffff 0x8000000000000000 U64_MAX
2152 * |-------------------------------|--------------------------------|
2153 * 0 S64_MAX S64_MIN -1
2154 * / \
2155 * >------------------------------ ------------------------------->
2156 * s64 continues... s64 end s64 start s64 "midpoint"
2157 *
2158 * What this means is that, in general, we can't always derive
2159 * something new about u64 from any random s64 range, and vice versa.
2160 *
2161 * But we can do that in two particular cases. One is when entire
2162 * u64/s64 range is *entirely* contained within left half of the above
2163 * diagram or when it is *entirely* contained in the right half. I.e.:
2164 *
2165 * |-------------------------------|--------------------------------|
2166 * ^ ^ ^ ^
2167 * A B C D
2168 *
2169 * [A, B] and [C, D] are contained entirely in their respective halves
2170 * and form valid contiguous ranges as both u64 and s64 values. [A, B]
2171 * will be non-negative both as u64 and s64 (and in fact it will be
2172 * identical ranges no matter the signedness). [C, D] treated as s64
2173 * will be a range of negative values, while in u64 it will be
2174 * non-negative range of values larger than 0x8000000000000000.
2175 *
2176 * Now, any other range here can't be represented in both u64 and s64
2177 * simultaneously. E.g., [A, C], [A, D], [B, C], [B, D] are valid
2178 * contiguous u64 ranges, but they are discontinuous in s64. [B, C]
2179 * in s64 would be properly presented as [S64_MIN, C] and [B, S64_MAX],
2180 * for example. Similarly, valid s64 range [D, A] (going from negative
2181 * to positive values), would be two separate [D, U64_MAX] and [0, A]
2182 * ranges as u64. Currently reg_state can't represent two segments per
2183 * numeric domain, so in such situations we can only derive maximal
2184 * possible range ([0, U64_MAX] for u64, and [S64_MIN, S64_MAX] for s64).
2185 *
2186 * So we use these facts to derive umin/umax from smin/smax and vice
2187 * versa only if they stay within the same "half". This is equivalent
2188 * to checking sign bit: lower half will have sign bit as zero, upper
2189 * half have sign bit 1. Below in code we simplify this by just
2190 * casting umin/umax as smin/smax and checking if they form valid
2191 * range, and vice versa. Those are equivalent checks.
2192 */
2193 if ((s64)reg->umin_value <= (s64)reg->umax_value) {
2194 reg->smin_value = max_t(s64, reg->smin_value, reg->umin_value);
2195 reg->smax_value = min_t(s64, reg->smax_value, reg->umax_value);
2196 }
2197 /* If we cannot cross the sign boundary, then signed and unsigned bounds
2198 * are the same, so combine. This works even in the negative case, e.g.
2199 * -3 s<= x s<= -1 implies 0xf...fd u<= x u<= 0xf...ff.
2200 */
2201 if ((u64)reg->smin_value <= (u64)reg->smax_value) {
2202 reg->umin_value = max_t(u64, reg->smin_value, reg->umin_value);
2203 reg->umax_value = min_t(u64, reg->smax_value, reg->umax_value);
2204 } else {
2205 /* If the s64 range crosses the sign boundary, then it's split
2206 * between the beginning and end of the U64 domain. In that
2207 * case, we can derive new bounds if the u64 range overlaps
2208 * with only one end of the s64 range.
2209 *
2210 * In the following example, the u64 range overlaps only with
2211 * positive portion of the s64 range.
2212 *
2213 * 0 U64_MAX
2214 * | [xxxxxxxxxxxxxx u64 range xxxxxxxxxxxxxx] |
2215 * |----------------------------|----------------------------|
2216 * |xxxxx s64 range xxxxxxxxx] [xxxxxxx|
2217 * 0 S64_MAX S64_MIN -1
2218 *
2219 * We can thus derive the following new s64 and u64 ranges.
2220 *
2221 * 0 U64_MAX
2222 * | [xxxxxx u64 range xxxxx] |
2223 * |----------------------------|----------------------------|
2224 * | [xxxxxx s64 range xxxxx] |
2225 * 0 S64_MAX S64_MIN -1
2226 *
2227 * If they overlap in two places, we can't derive anything
2228 * because reg_state can't represent two ranges per numeric
2229 * domain.
2230 *
2231 * 0 U64_MAX
2232 * | [xxxxxxxxxxxxxxxxx u64 range xxxxxxxxxxxxxxxxx] |
2233 * |----------------------------|----------------------------|
2234 * |xxxxx s64 range xxxxxxxxx] [xxxxxxxxxx|
2235 * 0 S64_MAX S64_MIN -1
2236 *
2237 * The first condition below corresponds to the first diagram
2238 * above.
2239 */
2240 if (reg->umax_value < (u64)reg->smin_value) {
2241 reg->smin_value = (s64)reg->umin_value;
2242 reg->umax_value = min_t(u64, reg->umax_value, reg->smax_value);
2243 } else if ((u64)reg->smax_value < reg->umin_value) {
2244 /* This second condition considers the case where the u64 range
2245 * overlaps with the negative portion of the s64 range:
2246 *
2247 * 0 U64_MAX
2248 * | [xxxxxxxxxxxxxx u64 range xxxxxxxxxxxxxx] |
2249 * |----------------------------|----------------------------|
2250 * |xxxxxxxxx] [xxxxxxxxxxxx s64 range |
2251 * 0 S64_MAX S64_MIN -1
2252 */
2253 reg->smax_value = (s64)reg->umax_value;
2254 reg->umin_value = max_t(u64, reg->umin_value, reg->smin_value);
2255 }
2256 }
2257}2045}
2258 2046
2259static void deduce_bounds_64_from_32(struct bpf_reg_state *reg)2047static void deduce_bounds_64_from_32(struct bpf_reg_state *reg)
2260{2048{
2261 /* Try to tighten 64-bit bounds from 32-bit knowledge, using 32-bit2049 reg->r64 = cnum64_cnum32_intersect(reg->r64, reg->r32);
2262 * values on both sides of 64-bit range in hope to have tighter range.
2263 * E.g., if r1 is [0x1'00000000, 0x3'80000000], and we learn from
2264 * 32-bit signed > 0 operation that s32 bounds are now [1; 0x7fffffff].
2265 * With this, we can substitute 1 as low 32-bits of _low_ 64-bit bound
2266 * (0x100000000 -> 0x100000001) and 0x7fffffff as low 32-bits of
2267 * _high_ 64-bit bound (0x380000000 -> 0x37fffffff) and arrive at a
2268 * better overall bounds for r1 as [0x1'000000001; 0x3'7fffffff].
2269 * We just need to make sure that derived bounds we are intersecting
2270 * with are well-formed ranges in respective s64 or u64 domain, just
2271 * like we do with similar kinds of 32-to-64 or 64-to-32 adjustments.
2272 */
2273 __u64 new_umin, new_umax;
2274 __s64 new_smin, new_smax;
2275
2276 /* u32 -> u64 tightening, it's always well-formed */
2277 new_umin = (reg->umin_value & ~0xffffffffULL) | reg->u32_min_value;
2278 new_umax = (reg->umax_value & ~0xffffffffULL) | reg->u32_max_value;
2279 reg->umin_value = max_t(u64, reg->umin_value, new_umin);
2280 reg->umax_value = min_t(u64, reg->umax_value, new_umax);
2281 /* u32 -> s64 tightening, u32 range embedded into s64 preserves range validity */
2282 new_smin = (reg->smin_value & ~0xffffffffULL) | reg->u32_min_value;
2283 new_smax = (reg->smax_value & ~0xffffffffULL) | reg->u32_max_value;
2284 reg->smin_value = max_t(s64, reg->smin_value, new_smin);
2285 reg->smax_value = min_t(s64, reg->smax_value, new_smax);
2286
2287 /* Here we would like to handle a special case after sign extending load,
2288 * when upper bits for a 64-bit range are all 1s or all 0s.
2289 *
2290 * Upper bits are all 1s when register is in a range:
2291 * [0xffff_ffff_0000_0000, 0xffff_ffff_ffff_ffff]
2292 * Upper bits are all 0s when register is in a range:
2293 * [0x0000_0000_0000_0000, 0x0000_0000_ffff_ffff]
2294 * Together this forms are continuous range:
2295 * [0xffff_ffff_0000_0000, 0x0000_0000_ffff_ffff]
2296 *
2297 * Now, suppose that register range is in fact tighter:
2298 * [0xffff_ffff_8000_0000, 0x0000_0000_ffff_ffff] (R)
2299 * Also suppose that it's 32-bit range is positive,
2300 * meaning that lower 32-bits of the full 64-bit register
2301 * are in the range:
2302 * [0x0000_0000, 0x7fff_ffff] (W)
2303 *
2304 * If this happens, then any value in a range:
2305 * [0xffff_ffff_0000_0000, 0xffff_ffff_7fff_ffff]
2306 * is smaller than a lowest bound of the range (R):
2307 * 0xffff_ffff_8000_0000
2308 * which means that upper bits of the full 64-bit register
2309 * can't be all 1s, when lower bits are in range (W).
2310 *
2311 * Note that:
2312 * - 0xffff_ffff_8000_0000 == (s64)S32_MIN
2313 * - 0x0000_0000_7fff_ffff == (s64)S32_MAX
2314 * These relations are used in the conditions below.
2315 */
2316 if (reg->s32_min_value >= 0 && reg->smin_value >= S32_MIN && reg->smax_value <= S32_MAX) {
2317 reg->smin_value = reg->s32_min_value;
2318 reg->smax_value = reg->s32_max_value;
2319 reg->umin_value = reg->s32_min_value;
2320 reg->umax_value = reg->s32_max_value;
2321 reg->var_off = tnum_intersect(reg->var_off,
2322 tnum_range(reg->smin_value, reg->smax_value));
2323 }
2324}2050}
2325 2051
2326static void __reg_deduce_bounds(struct bpf_reg_state *reg)2052static void __reg_deduce_bounds(struct bpf_reg_state *reg)
2327{2053{
2328 deduce_bounds_64_from_64(reg);
2329 deduce_bounds_32_from_64(reg);2054 deduce_bounds_32_from_64(reg);
2330 deduce_bounds_32_from_32(reg);
2331 deduce_bounds_64_from_32(reg);2055 deduce_bounds_64_from_32(reg);
2332}2056}
2333 2057
@@ -2335,11 +2059,11 @@ static void __reg_deduce_bounds(struct bpf_reg_state *reg) · 1 unmodified line
2335static void __reg_bound_offset(struct bpf_reg_state *reg)2059static void __reg_bound_offset(struct bpf_reg_state *reg)
2336{2060{
2337 struct tnum var64_off = tnum_intersect(reg->var_off,2061 struct tnum var64_off = tnum_intersect(reg->var_off,
2338 tnum_range(reg->umin_value,2062 tnum_range(reg_umin(reg),
2339 reg->umax_value));2063 reg_umax(reg)));
2340 struct tnum var32_off = tnum_intersect(tnum_subreg(var64_off),2064 struct tnum var32_off = tnum_intersect(tnum_subreg(var64_off),
2341 tnum_range(reg->u32_min_value,2065 tnum_range(reg_u32_min(reg),
2342 reg->u32_max_value));2066 reg_u32_max(reg)));
2343 2067
2344 reg->var_off = tnum_or(tnum_clear_subreg(var64_off), var32_off);2068 reg->var_off = tnum_or(tnum_clear_subreg(var64_off), var32_off);
2345}2069}
@@ -2365,35 +2089,25 @@ static void reg_bounds_sync(struct bpf_reg_state *reg) · 19 unmodified lines
2365 __update_reg_bounds(reg);2089 __update_reg_bounds(reg);
2366}2090}
2367 2091
2368static bool range_bounds_violation(struct bpf_reg_state *reg)
2369{
2370 return (reg->umin_value > reg->umax_value || reg->smin_value > reg->smax_value ||
2371 reg->u32_min_value > reg->u32_max_value ||
2372 reg->s32_min_value > reg->s32_max_value);
2373}
2374
2375static bool const_tnum_range_mismatch(struct bpf_reg_state *reg)2092static bool const_tnum_range_mismatch(struct bpf_reg_state *reg)
2376{2093{
2377 u64 uval = reg->var_off.value;
2378 s64 sval = (s64)uval;
2379
2380 if (!tnum_is_const(reg->var_off))2094 if (!tnum_is_const(reg->var_off))
2381 return false;2095 return false;
2382 2096
2383 return reg->umin_value != uval || reg->umax_value != uval ||2097 return !cnum64_is_const(reg->r64) || reg->r64.base != reg->var_off.value;
2384 reg->smin_value != sval || reg->smax_value != sval;
2385}2098}
2386 2099
2387static bool const_tnum_range_mismatch_32(struct bpf_reg_state *reg)2100static bool const_tnum_range_mismatch_32(struct bpf_reg_state *reg)
2388{2101{
2389 u32 uval32 = tnum_subreg(reg->var_off).value;
2390 s32 sval32 = (s32)uval32;
2391
2392 if (!tnum_subreg_is_const(reg->var_off))2102 if (!tnum_subreg_is_const(reg->var_off))
2393 return false;2103 return false;
2394 2104
2395 return reg->u32_min_value != uval32 || reg->u32_max_value != uval32 ||2105 return !cnum32_is_const(reg->r32) || reg->r32.base != tnum_subreg(reg->var_off).value;
2396 reg->s32_min_value != sval32 || reg->s32_max_value != sval32;2106}
2107
2108static bool range_bounds_violation(struct bpf_reg_state *reg)
2109{
2110 return cnum32_is_empty(reg->r32) || cnum64_is_empty(reg->r64);
2397}2111}
2398 2112
2399static int reg_bounds_sanity_check(struct bpf_verifier_env *env,2113static int reg_bounds_sanity_check(struct bpf_verifier_env *env,
@@ -2418,12 +2132,11 @@ static int reg_bounds_sanity_check(struct bpf_verifier_env *env, · 18 unmodified lines
2418 2132
2419 return 0;2133 return 0;
2420out:2134out:
2421 verifier_bug(env, "REG INVARIANTS VIOLATION (%s): %s u64=[%#llx, %#llx] "2135 verifier_bug(env, "REG INVARIANTS VIOLATION (%s): %s r64={.base=%#llx, .size=%#llx} "
2422 "s64=[%#llx, %#llx] u32=[%#x, %#x] s32=[%#x, %#x] var_off=(%#llx, %#llx)",2136 "r32={.base=%#x, .size=%#x} var_off=(%#llx, %#llx)",
2423 ctx, msg, reg->umin_value, reg->umax_value,2137 ctx, msg,
2424 reg->smin_value, reg->smax_value,2138 reg->r64.base, reg->r64.size,
2425 reg->u32_min_value, reg->u32_max_value,2139 reg->r32.base, reg->r32.size,
2426 reg->s32_min_value, reg->s32_max_value,
2427 reg->var_off.value, reg->var_off.mask);2140 reg->var_off.value, reg->var_off.mask);
2428 if (env->test_reg_invariants)2141 if (env->test_reg_invariants)
2429 return -EFAULT;2142 return -EFAULT;
@@ -2431,44 +2144,15 @@ out: · 1 unmodified line
2431 return 0;2144 return 0;
2432}2145}
2433 2146
2434static bool __reg32_bound_s64(s32 a)
2435{
2436 return a >= 0 && a <= S32_MAX;
2437}
2438
2439static void __reg_assign_32_into_64(struct bpf_reg_state *reg)
2440{
2441 reg->umin_value = reg->u32_min_value;
2442 reg->umax_value = reg->u32_max_value;
2443
2444 /* Attempt to pull 32-bit signed bounds into 64-bit bounds but must
2445 * be positive otherwise set to worse case bounds and refine later
2446 * from tnum.
2447 */
2448 if (__reg32_bound_s64(reg->s32_min_value) &&
2449 __reg32_bound_s64(reg->s32_max_value)) {
2450 reg->smin_value = reg->s32_min_value;
2451 reg->smax_value = reg->s32_max_value;
2452 } else {
2453 reg->smin_value = 0;
2454 reg->smax_value = U32_MAX;
2455 }
2456}
2457
2458/* Mark a register as having a completely unknown (scalar) value. */2147/* Mark a register as having a completely unknown (scalar) value. */
2459void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg)2148void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg)
2460{2149{
2461 /*2150 s32 subreg_def = reg->subreg_def;
2462 * Clear type, off, and union(map_ptr, range) and2151
2463 * padding between 'type' and union2152 memset(reg, 0, sizeof(*reg));
2464 */
2465 memset(reg, 0, offsetof(struct bpf_reg_state, var_off));
2466 reg->type = SCALAR_VALUE;2153 reg->type = SCALAR_VALUE;
2467 reg->id = 0;
2468 reg->ref_obj_id = 0;
2469 reg->var_off = tnum_unknown;2154 reg->var_off = tnum_unknown;
2470 reg->frameno = 0;2155 reg->subreg_def = subreg_def;
2471 reg->precise = false;
2472 __mark_reg_unbounded(reg);2156 __mark_reg_unbounded(reg);
2473}2157}
2474 2158
@@ -2496,11 +2180,12 @@ static int __mark_reg_s32_range(struct bpf_verifier_env *env, · 21 unmodified lines
2496{2180{
2497 struct bpf_reg_state *reg = regs + regno;2181 struct bpf_reg_state *reg = regs + regno;
2498 2182
2499 reg->s32_min_value = max_t(s32, reg->s32_min_value, s32_min);2183 reg_set_srange32(reg,
2500 reg->s32_max_value = min_t(s32, reg->s32_max_value, s32_max);2184 max_t(s32, reg_s32_min(reg), s32_min),
2501 2185 min_t(s32, reg_s32_max(reg), s32_max));
2502 reg->smin_value = max_t(s64, reg->smin_value, s32_min);2186 reg_set_srange64(reg,
2503 reg->smax_value = min_t(s64, reg->smax_value, s32_max);2187 max_t(s64, reg_smin(reg), s32_min),
2188 min_t(s64, reg_smax(reg), s32_max));
2504 2189
2505 reg_bounds_sync(reg);2190 reg_bounds_sync(reg);
2506 2191
@@ -3295,50 +2980,13 @@ out: · 788 unmodified lines
3295 return ret;2980 return ret;
3296}2981}
3297 2982
3298static int mark_stack_slot_obj_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg,2983static void mark_stack_slots_scratched(struct bpf_verifier_env *env,
3299 int spi, int nr_slots)2984 int spi, int nr_slots)
3300{2985{
3301 int i;2986 int i;
3302 2987
3303 for (i = 0; i < nr_slots; i++)2988 for (i = 0; i < nr_slots; i++)
3304 mark_stack_slot_scratched(env, spi - i);2989 mark_stack_slot_scratched(env, spi - i);
3305 return 0;
3306}
3307
3308static int mark_dynptr_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
3309{
3310 int spi;
3311
3312 /* For CONST_PTR_TO_DYNPTR, it must have already been done by
3313 * check_reg_arg in check_helper_call and mark_btf_func_reg_size in
3314 * check_kfunc_call.
3315 */
3316 if (reg->type == CONST_PTR_TO_DYNPTR)
3317 return 0;
3318 spi = dynptr_get_spi(env, reg);
3319 if (spi < 0)
3320 return spi;
3321 /* Caller ensures dynptr is valid and initialized, which means spi is in
3322 * bounds and spi is the first dynptr slot. Simply mark stack slot as
3323 * read.
3324 */
3325 return mark_stack_slot_obj_read(env, reg, spi, BPF_DYNPTR_NR_SLOTS);
3326}
3327
3328static int mark_iter_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
3329 int spi, int nr_slots)
3330{
3331 return mark_stack_slot_obj_read(env, reg, spi, nr_slots);
3332}
3333
3334static int mark_irq_flag_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
3335{
3336 int spi;
3337
3338 spi = irq_flag_get_spi(env, reg);
3339 if (spi < 0)
3340 return spi;
3341 return mark_stack_slot_obj_read(env, reg, spi, 1);
3342}2990}
3343 2991
3344/* This function is supposed to be used by the following 32-bit optimization2992/* This function is supposed to be used by the following 32-bit optimization
@@ -3491,17 +3139,12 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, · 146 unmodified lines
3491 return __check_reg_arg(env, state->regs, regno, t);3139 return __check_reg_arg(env, state->regs, regno, t);
3492}3140}
3493 3141
3494static int insn_stack_access_flags(int frameno, int spi)
3495{
3496 return INSN_F_STACK_ACCESS | (spi << INSN_F_SPI_SHIFT) | frameno;
3497}
3498
3499static void mark_indirect_target(struct bpf_verifier_env *env, int idx)3142static void mark_indirect_target(struct bpf_verifier_env *env, int idx)
3500{3143{
3501 env->insn_aux_data[idx].indirect_target = true;3144 env->insn_aux_data[idx].indirect_target = true;
3502}3145}
3503 3146
3504#define LR_FRAMENO_BITS 33147#define LR_FRAMENO_BITS 4
3505#define LR_SPI_BITS 63148#define LR_SPI_BITS 6
3506#define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1)3149#define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1)
3507#define LR_SIZE_BITS 43150#define LR_SIZE_BITS 4
@@ -3510,7 +3153,11 @@ static void mark_indirect_target(struct bpf_verifier_env *env, int idx) · 2 unmodified lines
3510#define LR_SIZE_MASK ((1ull << LR_SIZE_BITS) - 1)3153#define LR_SIZE_MASK ((1ull << LR_SIZE_BITS) - 1)
3511#define LR_SPI_OFF LR_FRAMENO_BITS3154#define LR_SPI_OFF LR_FRAMENO_BITS
3512#define LR_IS_REG_OFF (LR_SPI_BITS + LR_FRAMENO_BITS)3155#define LR_IS_REG_OFF (LR_SPI_BITS + LR_FRAMENO_BITS)
3513#define LINKED_REGS_MAX 63156#define LINKED_REGS_MAX 5
3157
3158static_assert(MAX_CALL_FRAMES <= (1 << LR_FRAMENO_BITS));
3159static_assert(LINKED_REGS_MAX < (1 << LR_SIZE_BITS));
3160static_assert(LINKED_REGS_MAX * LR_ENTRY_BITS + LR_SIZE_BITS <= 64);
3514 3161
3515struct linked_reg {3162struct linked_reg {
3516 u8 frameno;3163 u8 frameno;
@@ -3534,10 +3181,11 @@ static struct linked_reg *linked_regs_push(struct linked_regs *s) · 17 unmodified lines
3534 return NULL;3181 return NULL;
3535}3182}
3536 3183
3537/* Use u64 as a vector of 6 10-bit values, use first 4-bits to track3184/*
3185 * Use u64 as a vector of 5 11-bit values, use first 4-bits to track
3538 * number of elements currently in stack.3186 * number of elements currently in stack.
3539 * Pack one history entry for linked registers as 10 bits in the following format:3187 * Pack one history entry for linked registers as 11 bits in the following format:
3540 * - 3-bits frameno3188 * - 4-bits frameno
3541 * - 6-bits spi_or_reg3189 * - 6-bits spi_or_reg
3542 * - 1-bit is_reg3190 * - 1-bit is_reg
3543 */3191 */
@@ -3733,12 +3381,6 @@ static void assign_scalar_id_before_mov(struct bpf_verifier_env *env, · 189 unmodified lines
3733 src_reg->id = ++env->id_gen;3381 src_reg->id = ++env->id_gen;
3734}3382}
3735 3383
3736/* Copy src state preserving dst->parent and dst->live fields */
3737static void copy_register_state(struct bpf_reg_state *dst, const struct bpf_reg_state *src)
3738{
3739 *dst = *src;
3740}
3741
3742static void save_register_state(struct bpf_verifier_env *env,3384static void save_register_state(struct bpf_verifier_env *env,
3743 struct bpf_func_state *state,3385 struct bpf_func_state *state,
3744 int spi, struct bpf_reg_state *reg,3386 int spi, struct bpf_reg_state *reg,
@@ -3746,7 +3388,7 @@ static void save_register_state(struct bpf_verifier_env *env, · 1 unmodified line
3746{3388{
3747 int i;3389 int i;
3748 3390
3749 copy_register_state(&state->stack[spi].spilled_ptr, reg);3391 state->stack[spi].spilled_ptr = *reg;
3750 3392
3751 for (i = BPF_REG_SIZE; i > BPF_REG_SIZE - size; i--)3393 for (i = BPF_REG_SIZE; i > BPF_REG_SIZE - size; i--)
3752 state->stack[spi].slot_type[i - 1] = STACK_SPILL;3394 state->stack[spi].slot_type[i - 1] = STACK_SPILL;
@@ -3763,7 +3405,7 @@ static bool is_bpf_st_mem(struct bpf_insn *insn) · 10 unmodified lines
3763 3405
3764static int get_reg_width(struct bpf_reg_state *reg)3406static int get_reg_width(struct bpf_reg_state *reg)
3765{3407{
3766 return fls64(reg->umax_value);3408 return fls64(reg_umax(reg));
3767}3409}
3768 3410
3769/* See comment for mark_fastcall_pattern_for_call() */3411/* See comment for mark_fastcall_pattern_for_call() */
@@ -3816,7 +3458,8 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, · 46 unmodified lines
3816 int i, slot = -off - 1, spi = slot / BPF_REG_SIZE, err;3458 int i, slot = -off - 1, spi = slot / BPF_REG_SIZE, err;
3817 struct bpf_insn *insn = &env->prog->insnsi[insn_idx];3459 struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
3818 struct bpf_reg_state *reg = NULL;3460 struct bpf_reg_state *reg = NULL;
3819 int insn_flags = insn_stack_access_flags(state->frameno, spi);3461 int insn_flags = INSN_F_STACK_ACCESS;
3462 int hist_spi = spi, hist_frame = state->frameno;
3820 3463
3821 /* caller checked that off % size == 0 and -MAX_BPF_STACK <= off < 0,3464 /* caller checked that off % size == 0 and -MAX_BPF_STACK <= off < 0,
3822 * so it's aligned access and [off, off + size) are within stack limits3465 * so it's aligned access and [off, off + size) are within stack limits
@@ -3912,11 +3555,12 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, · 89 unmodified lines
3912 }3555 }
3913 3556
3914 if (insn_flags)3557 if (insn_flags)
3915 return bpf_push_jmp_history(env, env->cur_state, insn_flags, 0);3558 return bpf_push_jmp_history(env, env->cur_state, insn_flags,
3559 hist_spi, hist_frame, 0);
3916 return 0;3560 return 0;
3917}3561}
3918 3562
3919/* Write the stack: 'stack[ptr_regno + off] = value_regno'. 'ptr_regno' is3563/* Write the stack: 'stack[ptr_reg + off] = value_regno'. 'ptr_reg' is
3920 * known to contain a variable offset.3564 * known to contain a variable offset.
3921 * This function checks whether the write is permitted and conservatively3565 * This function checks whether the write is permitted and conservatively
3922 * tracks the effects of the write, considering that each stack slot in the3566 * tracks the effects of the write, considering that each stack slot in the
@@ -3937,13 +3581,13 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, · 14 unmodified lines
3937static int check_stack_write_var_off(struct bpf_verifier_env *env,3581static int check_stack_write_var_off(struct bpf_verifier_env *env,
3938 /* func where register points to */3582 /* func where register points to */
3939 struct bpf_func_state *state,3583 struct bpf_func_state *state,
3940 int ptr_regno, int off, int size,3584 struct bpf_reg_state *ptr_reg, int off, int size,
3941 int value_regno, int insn_idx)3585 int value_regno, int insn_idx)
3942{3586{
3943 struct bpf_func_state *cur; /* state of the current function */3587 struct bpf_func_state *cur; /* state of the current function */
3944 int min_off, max_off;3588 int min_off, max_off;
3945 int i, err;3589 int i, err;
3946 struct bpf_reg_state *ptr_reg = NULL, *value_reg = NULL;3590 struct bpf_reg_state *value_reg = NULL;
3947 struct bpf_insn *insn = &env->prog->insnsi[insn_idx];3591 struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
3948 bool writing_zero = false;3592 bool writing_zero = false;
3949 /* set if the fact that we're writing a zero is used to let any3593 /* set if the fact that we're writing a zero is used to let any
@@ -3952,9 +3596,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env, · 2 unmodified lines
3952 bool zero_used = false;3596 bool zero_used = false;
3953 3597
3954 cur = env->cur_state->frame[env->cur_state->curframe];3598 cur = env->cur_state->frame[env->cur_state->curframe];
3955 ptr_reg = &cur->regs[ptr_regno];3599 min_off = reg_smin(ptr_reg) + off;
3956 min_off = ptr_reg->smin_value + off;3600 max_off = reg_smax(ptr_reg) + off + size;
3957 max_off = ptr_reg->smax_value + off + size;
3958 if (value_regno >= 0)3601 if (value_regno >= 0)
3959 value_reg = &cur->regs[value_regno];3602 value_reg = &cur->regs[value_regno];
3960 if ((value_reg && bpf_register_is_null(value_reg)) ||3603 if ((value_reg && bpf_register_is_null(value_reg)) ||
@@ -4109,7 +3752,8 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 148 unmodified lines
4109 int i, slot = -off - 1, spi = slot / BPF_REG_SIZE;3752 int i, slot = -off - 1, spi = slot / BPF_REG_SIZE;
4110 struct bpf_reg_state *reg;3753 struct bpf_reg_state *reg;
4111 u8 *stype, type;3754 u8 *stype, type;
4112 int insn_flags = insn_stack_access_flags(reg_state->frameno, spi);3755 int insn_flags = INSN_F_STACK_ACCESS;
3756 int hist_spi = spi, hist_frame = reg_state->frameno;
4113 3757
4114 stype = reg_state->stack[spi].slot_type;3758 stype = reg_state->stack[spi].slot_type;
4115 reg = &reg_state->stack[spi].spilled_ptr;3759 reg = &reg_state->stack[spi].spilled_ptr;
@@ -4146,7 +3790,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 30 unmodified lines
4146 * with the destination register on fill.3790 * with the destination register on fill.
4147 */3791 */
4148 assign_scalar_id_before_mov(env, reg);3792 assign_scalar_id_before_mov(env, reg);
4149 copy_register_state(&state->regs[dst_regno], reg);3793 state->regs[dst_regno] = *reg;
4150 state->regs[dst_regno].subreg_def = subreg_def;3794 state->regs[dst_regno].subreg_def = subreg_def;
4151 3795
4152 /* Break the relation on a narrowing fill.3796 /* Break the relation on a narrowing fill.
@@ -4201,7 +3845,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 48 unmodified lines
4201 * with the destination register on fill.3845 * with the destination register on fill.
4202 */3846 */
4203 assign_scalar_id_before_mov(env, reg);3847 assign_scalar_id_before_mov(env, reg);
4204 copy_register_state(&state->regs[dst_regno], reg);3848 state->regs[dst_regno] = *reg;
4205 /* mark reg as written since spilled pointer state likely3849 /* mark reg as written since spilled pointer state likely
4206 * has its liveness marks cleared by is_state_visited()3850 * has its liveness marks cleared by is_state_visited()
4207 * which resets stack/reg liveness for state transitions3851 * which resets stack/reg liveness for state transitions
@@ -4240,7 +3884,8 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 32 unmodified lines
4240 insn_flags = 0; /* we are not restoring spilled register */3884 insn_flags = 0; /* we are not restoring spilled register */
4241 }3885 }
4242 if (insn_flags)3886 if (insn_flags)
4243 return bpf_push_jmp_history(env, env->cur_state, insn_flags, 0);3887 return bpf_push_jmp_history(env, env->cur_state, insn_flags,
3888 hist_spi, hist_frame, 0);
4244 return 0;3889 return 0;
4245}3890}
4246 3891
@@ -4249,8 +3894,8 @@ enum bpf_access_src { · 2 unmodified lines
4249 ACCESS_HELPER = 2, /* the access is performed by a helper */3894 ACCESS_HELPER = 2, /* the access is performed by a helper */
4250};3895};
4251 3896
4252static int check_stack_range_initialized(struct bpf_verifier_env *env,3897static int check_stack_range_initialized(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
4253 int regno, int off, int access_size,3898 argno_t argno, int off, int access_size,
4254 bool zero_size_allowed,3899 bool zero_size_allowed,
4255 enum bpf_access_type type,3900 enum bpf_access_type type,
4256 struct bpf_call_arg_meta *meta);3901 struct bpf_call_arg_meta *meta);
@@ -4260,37 +3905,35 @@ static struct bpf_reg_state *reg_state(struct bpf_verifier_env *env, int regno) · 3 unmodified lines
4260 return cur_regs(env) + regno;3905 return cur_regs(env) + regno;
4261}3906}
4262 3907
4263/* Read the stack at 'ptr_regno + off' and put the result into the register3908/* Read the stack at 'reg + off' and put the result into the register
4264 * 'dst_regno'.3909 * 'dst_regno'.
4265 * 'off' includes the pointer register's fixed offset(i.e. 'ptr_regno.off'),3910 * 'off' includes the pointer register's fixed offset(i.e. 'reg->off'),
4266 * but not its variable offset.3911 * but not its variable offset.
4267 * 'size' is assumed to be <= reg size and the access is assumed to be aligned.3912 * 'size' is assumed to be <= reg size and the access is assumed to be aligned.
4268 *3913 *
4269 * As opposed to check_stack_read_fixed_off, this function doesn't deal with3914 * As opposed to check_stack_read_fixed_off, this function doesn't deal with
4270 * filling registers (i.e. reads of spilled register cannot be detected when3915 * filling registers (i.e. reads of spilled register cannot be detected when
4271 * the offset is not fixed). We conservatively mark 'dst_regno' as containing3916 * the offset is not fixed). We conservatively mark 'dst_regno' as containing
4272 * SCALAR_VALUE. That's why we assert that the 'ptr_regno' has a variable3917 * SCALAR_VALUE. That's why we assert that the 'reg' has a variable
4273 * offset; for a fixed offset check_stack_read_fixed_off should be used3918 * offset; for a fixed offset check_stack_read_fixed_off should be used
4274 * instead.3919 * instead.
4275 */3920 */
4276static int check_stack_read_var_off(struct bpf_verifier_env *env,3921static int check_stack_read_var_off(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
4277 int ptr_regno, int off, int size, int dst_regno)3922 argno_t ptr_argno, int off, int size, int dst_regno)
4278{3923{
4279 /* The state of the source register. */
4280 struct bpf_reg_state *reg = reg_state(env, ptr_regno);
4281 struct bpf_func_state *ptr_state = bpf_func(env, reg);3924 struct bpf_func_state *ptr_state = bpf_func(env, reg);
4282 int err;3925 int err;
4283 int min_off, max_off;3926 int min_off, max_off;
4284 3927
4285 /* Note that we pass a NULL meta, so raw access will not be permitted.3928 /* Note that we pass a NULL meta, so raw access will not be permitted.
4286 */3929 */
4287 err = check_stack_range_initialized(env, ptr_regno, off, size,3930 err = check_stack_range_initialized(env, reg, ptr_argno, off, size,
4288 false, BPF_READ, NULL);3931 false, BPF_READ, NULL);
4289 if (err)3932 if (err)
4290 return err;3933 return err;
4291 3934
4292 min_off = reg->smin_value + off;3935 min_off = reg_smin(reg) + off;
4293 max_off = reg->smax_value + off;3936 max_off = reg_smax(reg) + off;
4294 mark_reg_stack_read(env, ptr_state, min_off, max_off + size, dst_regno);3937 mark_reg_stack_read(env, ptr_state, min_off, max_off + size, dst_regno);
4295 check_fastcall_stack_contract(env, ptr_state, env->insn_idx, min_off);3938 check_fastcall_stack_contract(env, ptr_state, env->insn_idx, min_off);
4296 return 0;3939 return 0;
@@ -4306,10 +3949,9 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env, · 9 unmodified lines
4306 * can be -1, meaning that the read value is not going to a register.3949 * can be -1, meaning that the read value is not going to a register.
4307 */3950 */
4308static int check_stack_read(struct bpf_verifier_env *env,3951static int check_stack_read(struct bpf_verifier_env *env,
4309 int ptr_regno, int off, int size,3952 struct bpf_reg_state *reg, argno_t ptr_argno, int off, int size,
4310 int dst_regno)3953 int dst_regno)
4311{3954{
4312 struct bpf_reg_state *reg = reg_state(env, ptr_regno);
4313 struct bpf_func_state *state = bpf_func(env, reg);3955 struct bpf_func_state *state = bpf_func(env, reg);
4314 int err;3956 int err;
4315 /* Some accesses are only permitted with a static offset. */3957 /* Some accesses are only permitted with a static offset. */
@@ -4345,7 +3987,7 @@ static int check_stack_read(struct bpf_verifier_env *env, · 29 unmodified lines
4345 * than fixed offset ones. Note that dst_regno >= 0 on this3987 * than fixed offset ones. Note that dst_regno >= 0 on this
4346 * branch.3988 * branch.
4347 */3989 */
4348 err = check_stack_read_var_off(env, ptr_regno, off, size,3990 err = check_stack_read_var_off(env, reg, ptr_argno, off, size,
4349 dst_regno);3991 dst_regno);
4350 }3992 }
4351 return err;3993 return err;
@@ -4355,17 +3997,16 @@ static int check_stack_read(struct bpf_verifier_env *env, · 3 unmodified lines
4355/* check_stack_write dispatches to check_stack_write_fixed_off or3997/* check_stack_write dispatches to check_stack_write_fixed_off or
4356 * check_stack_write_var_off.3998 * check_stack_write_var_off.
4357 *3999 *
4358 * 'ptr_regno' is the register used as a pointer into the stack.4000 * 'reg' is the register used as a pointer into the stack.
4359 * 'value_regno' is the register whose value we're writing to the stack. It can4001 * 'value_regno' is the register whose value we're writing to the stack. It can
4360 * be -1, meaning that we're not writing from a register.4002 * be -1, meaning that we're not writing from a register.
4361 *4003 *
4362 * The caller must ensure that the offset falls within the maximum stack size.4004 * The caller must ensure that the offset falls within the maximum stack size.
4363 */4005 */
4364static int check_stack_write(struct bpf_verifier_env *env,4006static int check_stack_write(struct bpf_verifier_env *env,
4365 int ptr_regno, int off, int size,4007 struct bpf_reg_state *reg, int off, int size,
4366 int value_regno, int insn_idx)4008 int value_regno, int insn_idx)
4367{4009{
4368 struct bpf_reg_state *reg = reg_state(env, ptr_regno);
4369 struct bpf_func_state *state = bpf_func(env, reg);4010 struct bpf_func_state *state = bpf_func(env, reg);
4370 int err;4011 int err;
4371 4012
@@ -4378,28 +4019,135 @@ static int check_stack_write(struct bpf_verifier_env *env, · 6 unmodified lines
4378 * than fixed offset ones.4019 * than fixed offset ones.
4379 */4020 */
4380 err = check_stack_write_var_off(env, state,4021 err = check_stack_write_var_off(env, state,
4381 ptr_regno, off, size,4022 reg, off, size,
4382 value_regno, insn_idx);4023 value_regno, insn_idx);
4383 }4024 }
4384 return err;4025 return err;
4385}4026}
4386 4027
4387static int check_map_access_type(struct bpf_verifier_env *env, u32 regno,4028/*
4029 * Write a value to the outgoing stack arg area.
4030 * off is a negative offset from r11 (e.g. -8 for arg6, -16 for arg7).
4031 */
4032static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state,
4033 int off, struct bpf_reg_state *value_reg)
4034{
4035 int max_stack_arg_regs = MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS;
4036 struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno];
4037 int spi = -off / BPF_REG_SIZE - 1;
4038 struct bpf_reg_state *arg;
4039 int err;
4040
4041 if (spi >= max_stack_arg_regs) {
4042 verbose(env, "stack arg write offset %d exceeds max %d stack args\n",
4043 off, max_stack_arg_regs);
4044 return -EINVAL;
4045 }
4046
4047 err = grow_stack_arg_slots(env, state, spi + 1);
4048 if (err)
4049 return err;
4050
4051 /* Track the max outgoing stack arg slot count. */
4052 if (spi + 1 > subprog->max_out_stack_arg_cnt)
4053 subprog->max_out_stack_arg_cnt = spi + 1;
4054
4055 if (value_reg) {
4056 state->stack_arg_regs[spi] = *value_reg;
4057 } else {
4058 /* BPF_ST: store immediate, treat as scalar */
4059 arg = &state->stack_arg_regs[spi];
4060 arg->type = SCALAR_VALUE;
4061 __mark_reg_known(arg, env->prog->insnsi[env->insn_idx].imm);
4062 }
4063 state->no_stack_arg_load = true;
4064 return bpf_push_jmp_history(env, env->cur_state,
4065 INSN_F_STACK_ARG_ACCESS, spi, 0, 0);
4066}
4067
4068/*
4069 * Read a value from the incoming stack arg area.
4070 * off is a positive offset from r11 (e.g. +8 for arg6, +16 for arg7).
4071 */
4072static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_state *state,
4073 int off, int dst_regno)
4074{
4075 struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno];
4076 struct bpf_verifier_state *vstate = env->cur_state;
4077 int spi = off / BPF_REG_SIZE - 1;
4078 struct bpf_func_state *caller, *cur;
4079 struct bpf_reg_state *arg;
4080
4081 if (state->no_stack_arg_load) {
4082 verbose(env, "r11 load must be before any r11 store or call insn\n");
4083 return -EINVAL;
4084 }
4085
4086 if (spi + 1 > bpf_in_stack_arg_cnt(subprog)) {
4087 verbose(env, "invalid read from stack arg off %d depth %d\n",
4088 off, bpf_in_stack_arg_cnt(subprog) * BPF_REG_SIZE);
4089 return -EACCES;
4090 }
4091
4092 caller = vstate->frame[vstate->curframe - 1];
4093 arg = &caller->stack_arg_regs[spi];
4094 cur = vstate->frame[vstate->curframe];
4095 cur->regs[dst_regno] = *arg;
4096 return bpf_push_jmp_history(env, env->cur_state,
4097 INSN_F_STACK_ARG_ACCESS, spi, 0, 0);
4098}
4099
4100static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx)
4101{
4102 struct bpf_func_state *caller = cur_func(env);
4103 int spi = arg_idx - MAX_BPF_FUNC_REG_ARGS;
4104
4105 bt_set_frame_stack_arg_slot(&env->bt, caller->frameno, spi);
4106 return mark_chain_precision_batch(env, env->cur_state);
4107}
4108
4109static int check_outgoing_stack_args(struct bpf_verifier_env *env, struct bpf_func_state *caller,
4110 int nargs)
4111{
4112 int i, spi;
4113
4114 for (i = MAX_BPF_FUNC_REG_ARGS; i < nargs; i++) {
4115 spi = i - MAX_BPF_FUNC_REG_ARGS;
4116 if (spi >= caller->out_stack_arg_cnt ||
4117 caller->stack_arg_regs[spi].type == NOT_INIT) {
4118 verbose(env, "callee expects %d args, stack arg%d is not initialized\n",
4119 nargs, spi + 1);
4120 return -EFAULT;
4121 }
4122 }
4123
4124 return 0;
4125}
4126
4127static struct bpf_reg_state *get_func_arg_reg(struct bpf_func_state *caller,
4128 struct bpf_reg_state *regs, int arg)
4129{
4130 if (arg < MAX_BPF_FUNC_REG_ARGS)
4131 return &regs[arg + 1];
4132
4133 return &caller->stack_arg_regs[arg - MAX_BPF_FUNC_REG_ARGS];
4134}
4135
4136static int check_map_access_type(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
4388 int off, int size, enum bpf_access_type type)4137 int off, int size, enum bpf_access_type type)
4389{4138{
4390 struct bpf_reg_state *reg = reg_state(env, regno);
4391 struct bpf_map *map = reg->map_ptr;4139 struct bpf_map *map = reg->map_ptr;
4392 u32 cap = bpf_map_flags_to_cap(map);4140 u32 cap = bpf_map_flags_to_cap(map);
4393 4141
4394 if (type == BPF_WRITE && !(cap & BPF_MAP_CAN_WRITE)) {4142 if (type == BPF_WRITE && !(cap & BPF_MAP_CAN_WRITE)) {
4395 verbose(env, "write into map forbidden, value_size=%d off=%lld size=%d\n",4143 verbose(env, "write into map forbidden, value_size=%d off=%lld size=%d\n",
4396 map->value_size, reg->smin_value + off, size);4144 map->value_size, reg_smin(reg) + off, size);
4397 return -EACCES;4145 return -EACCES;
4398 }4146 }
4399 4147
4400 if (type == BPF_READ && !(cap & BPF_MAP_CAN_READ)) {4148 if (type == BPF_READ && !(cap & BPF_MAP_CAN_READ)) {
4401 verbose(env, "read from map forbidden, value_size=%d off=%lld size=%d\n",4149 verbose(env, "read from map forbidden, value_size=%d off=%lld size=%d\n",
4402 map->value_size, reg->smin_value + off, size);4150 map->value_size, reg_smin(reg) + off, size);
4403 return -EACCES;4151 return -EACCES;
4404 }4152 }
4405 4153
@@ -4407,17 +4155,15 @@ static int check_map_access_type(struct bpf_verifier_env *env, u32 regno, · 1 unmodified line
4407}4155}
4408 4156
4409/* check read/write into memory region (e.g., map value, ringbuf sample, etc) */4157/* check read/write into memory region (e.g., map value, ringbuf sample, etc) */
4410static int __check_mem_access(struct bpf_verifier_env *env, int regno,4158static int __check_mem_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
4411 int off, int size, u32 mem_size,4159 int off, int size, u32 mem_size,
4412 bool zero_size_allowed)4160 bool zero_size_allowed)
4413{4161{
4414 bool size_ok = size > 0 || (size == 0 && zero_size_allowed);4162 bool size_ok = size > 0 || (size == 0 && zero_size_allowed);
4415 struct bpf_reg_state *reg;
4416 4163
4417 if (off >= 0 && size_ok && (u64)off + size <= mem_size)4164 if (off >= 0 && size_ok && (u64)off + size <= mem_size)
4418 return 0;4165 return 0;
4419 4166
4420 reg = &cur_regs(env)[regno];
4421 switch (reg->type) {4167 switch (reg->type) {
4422 case PTR_TO_MAP_KEY:4168 case PTR_TO_MAP_KEY:
4423 verbose(env, "invalid access to map key, key_size=%d off=%d size=%d\n",4169 verbose(env, "invalid access to map key, key_size=%d off=%d size=%d\n",
@@ -4430,8 +4176,8 @@ static int __check_mem_access(struct bpf_verifier_env *env, int regno, · 6 unmodified lines
4430 case PTR_TO_PACKET:4176 case PTR_TO_PACKET:
4431 case PTR_TO_PACKET_META:4177 case PTR_TO_PACKET_META:
4432 case PTR_TO_PACKET_END:4178 case PTR_TO_PACKET_END:
4433 verbose(env, "invalid access to packet, off=%d size=%d, R%d(id=%d,off=%d,r=%d)\n",4179 verbose(env, "invalid access to packet, off=%d size=%d, %s(id=%d,off=%d,r=%d)\n",
4434 off, size, regno, reg->id, off, mem_size);4180 off, size, reg_arg_name(env, argno), reg->id, off, mem_size);
4435 break;4181 break;
4436 case PTR_TO_CTX:4182 case PTR_TO_CTX:
4437 verbose(env, "invalid access to context, ctx_size=%d off=%d size=%d\n",4183 verbose(env, "invalid access to context, ctx_size=%d off=%d size=%d\n",
@@ -4447,13 +4193,10 @@ static int __check_mem_access(struct bpf_verifier_env *env, int regno, · 9 unmodified lines
4447}4193}
4448 4194
4449/* check read/write into a memory region with possible variable offset */4195/* check read/write into a memory region with possible variable offset */
4450static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno,4196static int check_mem_region_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
4451 int off, int size, u32 mem_size,4197 int off, int size, u32 mem_size,
4452 bool zero_size_allowed)4198 bool zero_size_allowed)
4453{4199{
4454 struct bpf_verifier_state *vstate = env->cur_state;
4455 struct bpf_func_state *state = vstate->frame[vstate->curframe];
4456 struct bpf_reg_state *reg = &state->regs[regno];
4457 int err;4200 int err;
4458 4201
4459 /* We may have adjusted the register pointing to memory region, so we4202 /* We may have adjusted the register pointing to memory region, so we
@@ -4466,36 +4209,36 @@ static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, · 6 unmodified lines
4466 * index'es we need to make sure that whatever we use4209 * index'es we need to make sure that whatever we use
4467 * will have a set floor within our range.4210 * will have a set floor within our range.
4468 */4211 */
4469 if (reg->smin_value < 0 &&4212 if (reg_smin(reg) < 0 &&
4470 (reg->smin_value == S64_MIN ||4213 (reg_smin(reg) == S64_MIN ||
4471 (off + reg->smin_value != (s64)(s32)(off + reg->smin_value)) ||4214 (off + reg_smin(reg) != (s64)(s32)(off + reg_smin(reg))) ||
4472 reg->smin_value + off < 0)) {4215 reg_smin(reg) + off < 0)) {
4473 verbose(env, "R%d min value is negative, either use unsigned index or do a if (index >=0) check.\n",4216 verbose(env, "%s min value is negative, either use unsigned index or do a if (index >=0) check.\n",
4474 regno);4217 reg_arg_name(env, argno));
4475 return -EACCES;4218 return -EACCES;
4476 }4219 }
4477 err = __check_mem_access(env, regno, reg->smin_value + off, size,4220 err = __check_mem_access(env, reg, argno, reg_smin(reg) + off, size,
4478 mem_size, zero_size_allowed);4221 mem_size, zero_size_allowed);
4479 if (err) {4222 if (err) {
4480 verbose(env, "R%d min value is outside of the allowed memory range\n",4223 verbose(env, "%s min value is outside of the allowed memory range\n",
4481 regno);4224 reg_arg_name(env, argno));
4482 return err;4225 return err;
4483 }4226 }
4484 4227
4485 /* If we haven't set a max value then we need to bail since we can't be4228 /* If we haven't set a max value then we need to bail since we can't be
4486 * sure we won't do bad things.4229 * sure we won't do bad things.
4487 * If reg->umax_value + off could overflow, treat that as unbounded too.4230 * If reg_umax(reg) + off could overflow, treat that as unbounded too.
4488 */4231 */
4489 if (reg->umax_value >= BPF_MAX_VAR_OFF) {4232 if (reg_umax(reg) >= BPF_MAX_VAR_OFF) {
4490 verbose(env, "R%d unbounded memory access, make sure to bounds check any such access\n",4233 verbose(env, "%s unbounded memory access, make sure to bounds check any such access\n",
4491 regno);4234 reg_arg_name(env, argno));
4492 return -EACCES;4235 return -EACCES;
4493 }4236 }
4494 err = __check_mem_access(env, regno, reg->umax_value + off, size,4237 err = __check_mem_access(env, reg, argno, reg_umax(reg) + off, size,
4495 mem_size, zero_size_allowed);4238 mem_size, zero_size_allowed);
4496 if (err) {4239 if (err) {
4497 verbose(env, "R%d max value is outside of the allowed memory range\n",4240 verbose(env, "%s max value is outside of the allowed memory range\n",
4498 regno);4241 reg_arg_name(env, argno));
4499 return err;4242 return err;
4500 }4243 }
4501 4244
@@ -4503,7 +4246,7 @@ static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, · 1 unmodified line
4503}4246}
4504 4247
4505static int __check_ptr_off_reg(struct bpf_verifier_env *env,4248static int __check_ptr_off_reg(struct bpf_verifier_env *env,
4506 const struct bpf_reg_state *reg, int regno,4249 const struct bpf_reg_state *reg, argno_t argno,
4507 bool fixed_off_ok)4250 bool fixed_off_ok)
4508{4251{
4509 /* Access to this pointer-typed register or passing it to a helper4252 /* Access to this pointer-typed register or passing it to a helper
@@ -4519,15 +4262,15 @@ static int __check_ptr_off_reg(struct bpf_verifier_env *env, · 9 unmodified lines
4519 return -EACCES;4262 return -EACCES;
4520 }4263 }
4521 4264
4522 if (reg->smin_value < 0) {4265 if (reg_smin(reg) < 0) {
4523 verbose(env, "negative offset %s ptr R%d off=%lld disallowed\n",4266 verbose(env, "negative offset %s ptr %s off=%lld disallowed\n",
4524 reg_type_str(env, reg->type), regno, reg->var_off.value);4267 reg_type_str(env, reg->type), reg_arg_name(env, argno), reg->var_off.value);
4525 return -EACCES;4268 return -EACCES;
4526 }4269 }
4527 4270
4528 if (!fixed_off_ok && reg->var_off.value != 0) {4271 if (!fixed_off_ok && reg->var_off.value != 0) {
4529 verbose(env, "dereference of modified %s ptr R%d off=%lld disallowed\n",4272 verbose(env, "dereference of modified %s ptr %s off=%lld disallowed\n",
4530 reg_type_str(env, reg->type), regno, reg->var_off.value);4273 reg_type_str(env, reg->type), reg_arg_name(env, argno), reg->var_off.value);
4531 return -EACCES;4274 return -EACCES;
4532 }4275 }
4533 4276
@@ -4537,7 +4280,7 @@ static int __check_ptr_off_reg(struct bpf_verifier_env *env, · 3 unmodified lines
4537static int check_ptr_off_reg(struct bpf_verifier_env *env,4280static int check_ptr_off_reg(struct bpf_verifier_env *env,
4538 const struct bpf_reg_state *reg, int regno)4281 const struct bpf_reg_state *reg, int regno)
4539{4282{
4540 return __check_ptr_off_reg(env, reg, regno, false);4283 return __check_ptr_off_reg(env, reg, argno_from_reg(regno), false);
4541}4284}
4542 4285
4543static int map_kptr_match_type(struct bpf_verifier_env *env,4286static int map_kptr_match_type(struct bpf_verifier_env *env,
@@ -4573,9 +4316,9 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, · 29 unmodified lines
4573 * referenced PTR_TO_BTF_ID, and that its fixed offset is 0. For the4316 * referenced PTR_TO_BTF_ID, and that its fixed offset is 0. For the
4574 * normal store of unreferenced kptr, we must ensure var_off is zero.4317 * normal store of unreferenced kptr, we must ensure var_off is zero.
4575 * Since ref_ptr cannot be accessed directly by BPF insns, check for4318 * Since ref_ptr cannot be accessed directly by BPF insns, check for
4576 * reg->ref_obj_id is not needed here.4319 * reg->id is not needed here.
4577 */4320 */
4578 if (__check_ptr_off_reg(env, reg, regno, true))4321 if (__check_ptr_off_reg(env, reg, argno_from_reg(regno), true))
4579 return -EACCES;4322 return -EACCES;
4580 4323
4581 /* A full type match is needed, as BTF can be vmlinux, module or prog BTF, and4324 /* A full type match is needed, as BTF can be vmlinux, module or prog BTF, and
@@ -4718,7 +4461,7 @@ static int mark_uptr_ld_reg(struct bpf_verifier_env *env, u32 regno, · 136 unmodified lines
4718 return 0;4461 return 0;
4719}4462}
4720 4463
4721static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno,4464static int check_map_kptr_access(struct bpf_verifier_env *env,
4722 int value_regno, int insn_idx,4465 int value_regno, int insn_idx,
4723 struct btf_field *kptr_field)4466 struct btf_field *kptr_field)
4724{4467{
@@ -4795,19 +4538,16 @@ static u32 map_mem_size(const struct bpf_map *map) · 70 unmodified lines
4795}4538}
4796 4539
4797/* check read/write into a map element with possible variable offset */4540/* check read/write into a map element with possible variable offset */
4798static int check_map_access(struct bpf_verifier_env *env, u32 regno,4541static int check_map_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
4799 int off, int size, bool zero_size_allowed,4542 int off, int size, bool zero_size_allowed,
4800 enum bpf_access_src src)4543 enum bpf_access_src src)
4801{4544{
4802 struct bpf_verifier_state *vstate = env->cur_state;
4803 struct bpf_func_state *state = vstate->frame[vstate->curframe];
4804 struct bpf_reg_state *reg = &state->regs[regno];
4805 struct bpf_map *map = reg->map_ptr;4545 struct bpf_map *map = reg->map_ptr;
4806 u32 mem_size = map_mem_size(map);4546 u32 mem_size = map_mem_size(map);
4807 struct btf_record *rec;4547 struct btf_record *rec;
4808 int err, i;4548 int err, i;
4809 4549
4810 err = check_mem_region_access(env, regno, off, size, mem_size, zero_size_allowed);4550 err = check_mem_region_access(env, reg, argno, off, size, mem_size, zero_size_allowed);
4811 if (err)4551 if (err)
4812 return err;4552 return err;
4813 4553
@@ -4822,8 +4562,8 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 8 unmodified lines
4822 * this program. To check that [x1, x2) overlaps with [y1, y2),4562 * this program. To check that [x1, x2) overlaps with [y1, y2),
4823 * it is sufficient to check x1 < y2 && y1 < x2.4563 * it is sufficient to check x1 < y2 && y1 < x2.
4824 */4564 */
4825 if (reg->smin_value + off < p + field->size &&4565 if (reg_smin(reg) + off < p + field->size &&
4826 p < reg->umax_value + off + size) {4566 p < reg_umax(reg) + off + size) {
4827 switch (field->type) {4567 switch (field->type) {
4828 case BPF_KPTR_UNREF:4568 case BPF_KPTR_UNREF:
4829 case BPF_KPTR_REF:4569 case BPF_KPTR_REF:
@@ -4903,30 +4643,29 @@ static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, · 73 unmodified lines
4903 }4643 }
4904}4644}
4905 4645
4906static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off,4646static int check_packet_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int off,
4907 int size, bool zero_size_allowed)4647 int size, bool zero_size_allowed)
4908{4648{
4909 struct bpf_reg_state *reg = reg_state(env, regno);
4910 int err;4649 int err;
4911 4650
4912 if (reg->range < 0) {4651 if (reg->range < 0) {
4913 verbose(env, "R%d offset is outside of the packet\n", regno);4652 verbose(env, "%s offset is outside of the packet\n", reg_arg_name(env, argno));
4914 return -EINVAL;4653 return -EINVAL;
4915 }4654 }
4916 4655
4917 err = check_mem_region_access(env, regno, off, size, reg->range, zero_size_allowed);4656 err = check_mem_region_access(env, reg, argno, off, size, reg->range, zero_size_allowed);
4918 if (err)4657 if (err)
4919 return err;4658 return err;
4920 4659
4921 /* __check_mem_access has made sure "off + size - 1" is within u16.4660 /* __check_mem_access has made sure "off + size - 1" is within u16.
4922 * reg->umax_value can't be bigger than MAX_PACKET_OFF which is 0xffff,4661 * reg_umax(reg) can't be bigger than MAX_PACKET_OFF which is 0xffff,
4923 * otherwise find_good_pkt_pointers would have refused to set range info4662 * otherwise find_good_pkt_pointers would have refused to set range info
4924 * that __check_mem_access would have rejected this pkt access.4663 * that __check_mem_access would have rejected this pkt access.
4925 * Therefore, "off + reg->umax_value + size - 1" won't overflow u32.4664 * Therefore, "off + reg_umax(reg) + size - 1" won't overflow u32.
4926 */4665 */
4927 env->prog->aux->max_pkt_offset =4666 env->prog->aux->max_pkt_offset =
4928 max_t(u32, env->prog->aux->max_pkt_offset,4667 max_t(u32, env->prog->aux->max_pkt_offset,
4929 off + reg->umax_value + size - 1);4668 off + reg_umax(reg) + size - 1);
4930 4669
4931 return 0;4670 return 0;
4932}4671}
@@ -4950,8 +4689,8 @@ static int __check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int of · 17 unmodified lines
4950 * type of narrower access.4689 * type of narrower access.
4951 */4690 */
4952 if (base_type(info->reg_type) == PTR_TO_BTF_ID) {4691 if (base_type(info->reg_type) == PTR_TO_BTF_ID) {
4953 if (info->ref_obj_id &&4692 if (info->ref_id &&
4954 !find_reference_state(env->cur_state, info->ref_obj_id)) {4693 !find_reference_state(env->cur_state, info->ref_id)) {
4955 verbose(env, "invalid bpf_context access off=%d. Reference may already be released\n",4694 verbose(env, "invalid bpf_context access off=%d. Reference may already be released\n",
4956 off);4695 off);
4957 return -EACCES;4696 return -EACCES;
@@ -4969,7 +4708,7 @@ static int __check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int of · 11 unmodified lines
4969 return -EACCES;4708 return -EACCES;
4970}4709}
4971 4710
4972static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regno,4711static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, struct bpf_reg_state *reg, argno_t argno,
4973 int off, int access_size, enum bpf_access_type t,4712 int off, int access_size, enum bpf_access_type t,
4974 struct bpf_insn_access_aux *info)4713 struct bpf_insn_access_aux *info)
4975{4714{
@@ -4979,17 +4718,15 @@ static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 3 unmodified lines
4979 */4718 */
4980 bool var_off_ok = is_var_ctx_off_allowed(env->prog);4719 bool var_off_ok = is_var_ctx_off_allowed(env->prog);
4981 bool fixed_off_ok = !env->ops->convert_ctx_access;4720 bool fixed_off_ok = !env->ops->convert_ctx_access;
4982 struct bpf_reg_state *regs = cur_regs(env);
4983 struct bpf_reg_state *reg = regs + regno;
4984 int err;4721 int err;
4985 4722
4986 if (var_off_ok)4723 if (var_off_ok)
4987 err = check_mem_region_access(env, regno, off, access_size, U16_MAX, false);4724 err = check_mem_region_access(env, reg, argno, off, access_size, U16_MAX, false);
4988 else4725 else
4989 err = __check_ptr_off_reg(env, reg, regno, fixed_off_ok);4726 err = __check_ptr_off_reg(env, reg, argno, fixed_off_ok);
4990 if (err)4727 if (err)
4991 return err;4728 return err;
4992 off += reg->umax_value;4729 off += reg_umax(reg);
4993 4730
4994 err = __check_ctx_access(env, insn_idx, off, access_size, t, info);4731 err = __check_ctx_access(env, insn_idx, off, access_size, t, info);
4995 if (err)4732 if (err)
@@ -4997,9 +4734,21 @@ static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 1 unmodified line
4997 return err;4734 return err;
4998}4735}
4999 4736
5000static int check_flow_keys_access(struct bpf_verifier_env *env, int off,4737static int check_flow_keys_access(struct bpf_verifier_env *env,
5001 int size)4738 struct bpf_reg_state *reg, argno_t argno,
4739 int off, int size)
5002{4740{
4741 /* Only a constant offset is allowed here; fold it into off. */
4742 if (!tnum_is_const(reg->var_off)) {
4743 char tn_buf[48];
4744
4745 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
4746 verbose(env, "%s invalid variable offset to flow keys: off=%d, var_off=%s\n",
4747 reg_arg_name(env, argno), off, tn_buf);
4748 return -EACCES;
4749 }
4750 off += reg->var_off.value;
4751
5003 if (size < 0 || off < 0 ||4752 if (size < 0 || off < 0 ||
5004 (u64)off + size > sizeof(struct bpf_flow_keys)) {4753 (u64)off + size > sizeof(struct bpf_flow_keys)) {
5005 verbose(env, "invalid access to flow keys off=%d size=%d\n",4754 verbose(env, "invalid access to flow keys off=%d size=%d\n",
@@ -5010,16 +4759,15 @@ static int check_flow_keys_access(struct bpf_verifier_env *env, int off, · 4 unmodified lines
5010}4759}
5011 4760
5012static int check_sock_access(struct bpf_verifier_env *env, int insn_idx,4761static int check_sock_access(struct bpf_verifier_env *env, int insn_idx,
5013 u32 regno, int off, int size,4762 struct bpf_reg_state *reg, argno_t argno, int off, int size,
5014 enum bpf_access_type t)4763 enum bpf_access_type t)
5015{4764{
5016 struct bpf_reg_state *reg = reg_state(env, regno);
5017 struct bpf_insn_access_aux info = {};4765 struct bpf_insn_access_aux info = {};
5018 bool valid;4766 bool valid;
5019 4767
5020 if (reg->smin_value < 0) {4768 if (reg_smin(reg) < 0) {
5021 verbose(env, "R%d min value is negative, either use unsigned index or do a if (index >=0) check.\n",4769 verbose(env, "%s min value is negative, either use unsigned index or do a if (index >=0) check.\n",
5022 regno);4770 reg_arg_name(env, argno));
5023 return -EACCES;4771 return -EACCES;
5024 }4772 }
5025 4773
@@ -5047,8 +4795,8 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx, · 21 unmodified lines
5047 return 0;4795 return 0;
5048 }4796 }
5049 4797
5050 verbose(env, "R%d invalid %s access off=%d size=%d\n",4798 verbose(env, "%s invalid %s access off=%d size=%d\n",
5051 regno, reg_type_str(env, reg->type), off, size);4799 reg_arg_name(env, argno), reg_type_str(env, reg->type), off, size);
5052 4800
5053 return -EACCES;4801 return -EACCES;
5054}4802}
@@ -5123,10 +4871,10 @@ static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = { · 68 unmodified lines
5123 [CONST_PTR_TO_MAP] = btf_bpf_map_id,4871 [CONST_PTR_TO_MAP] = btf_bpf_map_id,
5124};4872};
5125 4873
5126static bool is_trusted_reg(const struct bpf_reg_state *reg)4874static bool is_trusted_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg)
5127{4875{
5128 /* A referenced register is always trusted. */4876 /* A referenced register is always trusted. */
5129 if (reg->ref_obj_id)4877 if (reg_is_referenced(env, reg))
5130 return true;4878 return true;
5131 4879
5132 /* Types listed in the reg2btf_ids are always trusted */4880 /* Types listed in the reg2btf_ids are always trusted */
@@ -5368,7 +5116,10 @@ process_func: · 235 unmodified lines
5368 }5116 }
5369 5117
5370 subprog_depth = round_up_stack_depth(env, subprog[idx].stack_depth);5118 subprog_depth = round_up_stack_depth(env, subprog[idx].stack_depth);
5371 if (priv_stack_supported) {5119 if (IS_ENABLED(CONFIG_X86_64) && subprog[idx].stack_arg_cnt) {
5120 /* x86-64 uses R9 for both private stack frame pointer and arg6. */
5121 subprog[idx].priv_stack_mode = NO_PRIV_STACK;
5122 } else if (priv_stack_supported) {
5372 /* Request private stack support only if the subprog stack5123 /* Request private stack support only if the subprog stack
5373 * depth is no less than BPF_PRIV_STACK_MIN_SIZE. This is to5124 * depth is no less than BPF_PRIV_STACK_MIN_SIZE. This is to
5374 * avoid jit penalty if the stack usage is small.5125 * avoid jit penalty if the stack usage is small.
@@ -5379,6 +5130,8 @@ process_func: · 4 unmodified lines
5379 }5130 }
5380 5131
5381 if (subprog[idx].priv_stack_mode == PRIV_STACK_ADAPTIVE) {5132 if (subprog[idx].priv_stack_mode == PRIV_STACK_ADAPTIVE) {
5133 if (subprog_depth > env->max_stack_depth)
5134 env->max_stack_depth = subprog_depth;
5382 if (subprog_depth > MAX_BPF_STACK) {5135 if (subprog_depth > MAX_BPF_STACK) {
5383 verbose(env, "stack size of subprog %d is %d. Too large\n",5136 verbose(env, "stack size of subprog %d is %d. Too large\n",
5384 idx, subprog_depth);5137 idx, subprog_depth);
@@ -5386,6 +5139,8 @@ process_func: · 1 unmodified line
5386 }5139 }
5387 } else {5140 } else {
5388 depth += subprog_depth;5141 depth += subprog_depth;
5142 if (depth > env->max_stack_depth)
5143 env->max_stack_depth = depth;
5389 if (depth > MAX_BPF_STACK) {5144 if (depth > MAX_BPF_STACK) {
5390 total = 0;5145 total = 0;
5391 for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller)5146 for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller)
@@ -5472,14 +5227,23 @@ continue_func: · 80 unmodified lines
5472 * this info will be utilized by JIT so that we will be preserving the5227 * this info will be utilized by JIT so that we will be preserving the
5473 * tail call counter throughout bpf2bpf calls combined with tailcalls5228 * tail call counter throughout bpf2bpf calls combined with tailcalls
5474 */5229 */
5475 if (tail_call_reachable)5230 if (tail_call_reachable) {
5476 for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {5231 for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {
5477 if (subprog[tmp].is_exception_cb) {5232 if (subprog[tmp].is_exception_cb) {
5478 verbose(env, "cannot tail call within exception cb\n");5233 verbose(env, "cannot tail call within exception cb\n");
5479 return -EINVAL;5234 return -EINVAL;
5480 }5235 }
5236 if (subprog[tmp].stack_arg_cnt) {
5237 verbose(env, "tail_calls are not allowed in programs with stack args\n");
5238 return -EINVAL;
5239 }
5481 subprog[tmp].tail_call_reachable = true;5240 subprog[tmp].tail_call_reachable = true;
5482 }5241 }
5242 } else if (!idx && subprog[0].has_tail_call && subprog[0].stack_arg_cnt) {
5243 verbose(env, "tail_calls are not allowed in programs with stack args\n");
5244 return -EINVAL;
5245 }
5246
5483 if (subprog[0].tail_call_reachable)5247 if (subprog[0].tail_call_reachable)
5484 env->prog->aux->tail_call_reachable = true;5248 env->prog->aux->tail_call_reachable = true;
5485 5249
@@ -5498,6 +5262,9 @@ continue_func: · 12 unmodified lines
5498 frame = dinfo[idx].frame;5262 frame = dinfo[idx].frame;
5499 i = dinfo[idx].ret_insn;5263 i = dinfo[idx].ret_insn;
5500 5264
5265 /* reset tail_call_reachable to the parent's actual state */
5266 tail_call_reachable = subprog[idx].tail_call_reachable;
5267
5501 goto continue_func;5268 goto continue_func;
5502}5269}
5503 5270
@@ -5558,12 +5325,12 @@ static int check_max_stack_depth(struct bpf_verifier_env *env) · 54 unmodified lines
5558static int __check_buffer_access(struct bpf_verifier_env *env,5325static int __check_buffer_access(struct bpf_verifier_env *env,
5559 const char *buf_info,5326 const char *buf_info,
5560 const struct bpf_reg_state *reg,5327 const struct bpf_reg_state *reg,
5561 int regno, int off, int size)5328 argno_t argno, int off, int size)
5562{5329{
5563 if (off < 0) {5330 if (off < 0) {
5564 verbose(env,5331 verbose(env,
5565 "R%d invalid %s buffer access: off=%d, size=%d\n",5332 "%s invalid %s buffer access: off=%d, size=%d\n",
5566 regno, buf_info, off, size);5333 reg_arg_name(env, argno), buf_info, off, size);
5567 return -EACCES;5334 return -EACCES;
5568 }5335 }
5569 if (!tnum_is_const(reg->var_off)) {5336 if (!tnum_is_const(reg->var_off)) {
@@ -5571,8 +5338,8 @@ static int __check_buffer_access(struct bpf_verifier_env *env, · 1 unmodified line
5571 5338
5572 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);5339 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
5573 verbose(env,5340 verbose(env,
5574 "R%d invalid variable buffer offset: off=%d, var_off=%s\n",5341 "%s invalid variable buffer offset: off=%d, var_off=%s\n",
5575 regno, off, tn_buf);5342 reg_arg_name(env, argno), off, tn_buf);
5576 return -EACCES;5343 return -EACCES;
5577 }5344 }
5578 5345
@@ -5581,11 +5348,11 @@ static int __check_buffer_access(struct bpf_verifier_env *env, · 2 unmodified lines
5581 5348
5582static int check_tp_buffer_access(struct bpf_verifier_env *env,5349static int check_tp_buffer_access(struct bpf_verifier_env *env,
5583 const struct bpf_reg_state *reg,5350 const struct bpf_reg_state *reg,
5584 int regno, int off, int size)5351 argno_t argno, int off, int size)
5585{5352{
5586 int err;5353 int err;
5587 5354
5588 err = __check_buffer_access(env, "tracepoint", reg, regno, off, size);5355 err = __check_buffer_access(env, "tracepoint", reg, argno, off, size);
5589 if (err)5356 if (err)
5590 return err;5357 return err;
5591 5358
@@ -5597,14 +5364,14 @@ static int check_tp_buffer_access(struct bpf_verifier_env *env, · 5 unmodified lines
5597 5364
5598static int check_buffer_access(struct bpf_verifier_env *env,5365static int check_buffer_access(struct bpf_verifier_env *env,
5599 const struct bpf_reg_state *reg,5366 const struct bpf_reg_state *reg,
5600 int regno, int off, int size,5367 argno_t argno, int off, int size,
5601 bool zero_size_allowed,5368 bool zero_size_allowed,
5602 u32 *max_access)5369 u32 *max_access)
5603{5370{
5604 const char *buf_info = type_is_rdonly_mem(reg->type) ? "rdonly" : "rdwr";5371 const char *buf_info = type_is_rdonly_mem(reg->type) ? "rdonly" : "rdwr";
5605 int err;5372 int err;
5606 5373
5607 err = __check_buffer_access(env, buf_info, reg, regno, off, size);5374 err = __check_buffer_access(env, buf_info, reg, argno, off, size);
5608 if (err)5375 if (err)
5609 return err;5376 return err;
5610 5377
@@ -5617,7 +5384,7 @@ static int check_buffer_access(struct bpf_verifier_env *env, · 6 unmodified lines
5617static void zext_32_to_64(struct bpf_reg_state *reg)5384static void zext_32_to_64(struct bpf_reg_state *reg)
5618{5385{
5619 reg->var_off = tnum_subreg(reg->var_off);5386 reg->var_off = tnum_subreg(reg->var_off);
5620 __reg_assign_32_into_64(reg);5387 reg_set_urange64(reg, reg_u32_min(reg), reg_u32_max(reg));
5621}5388}
5622 5389
5623/* truncate register to smaller size (in bytes)5390/* truncate register to smaller size (in bytes)
@@ -5632,15 +5399,10 @@ static void coerce_reg_to_size(struct bpf_reg_state *reg, int size) · 8 unmodified lines
5632 5399
5633 /* fix arithmetic bounds */5400 /* fix arithmetic bounds */
5634 mask = ((u64)1 << (size * 8)) - 1;5401 mask = ((u64)1 << (size * 8)) - 1;
5635 if ((reg->umin_value & ~mask) == (reg->umax_value & ~mask)) {5402 if ((reg_umin(reg) & ~mask) == (reg_umax(reg) & ~mask))
5636 reg->umin_value &= mask;5403 reg_set_urange64(reg, reg_umin(reg) & mask, reg_umax(reg) & mask);
5637 reg->umax_value &= mask;5404 else
5638 } else {5405 reg_set_urange64(reg, 0, mask);
5639 reg->umin_value = 0;
5640 reg->umax_value = mask;
5641 }
5642 reg->smin_value = reg->umin_value;
5643 reg->smax_value = reg->umax_value;
5644 5406
5645 /* If size is smaller than 32bit register the 32bit register5407 /* If size is smaller than 32bit register the 32bit register
5646 * values are also truncated so we push 64-bit bounds into5408 * values are also truncated so we push 64-bit bounds into
@@ -5655,19 +5417,16 @@ static void coerce_reg_to_size(struct bpf_reg_state *reg, int size) · 8 unmodified lines
5655static void set_sext64_default_val(struct bpf_reg_state *reg, int size)5417static void set_sext64_default_val(struct bpf_reg_state *reg, int size)
5656{5418{
5657 if (size == 1) {5419 if (size == 1) {
5658 reg->smin_value = reg->s32_min_value = S8_MIN;5420 reg_set_srange64(reg, S8_MIN, S8_MAX);
5659 reg->smax_value = reg->s32_max_value = S8_MAX;5421 reg_set_srange32(reg, S8_MIN, S8_MAX);
5660 } else if (size == 2) {5422 } else if (size == 2) {
5661 reg->smin_value = reg->s32_min_value = S16_MIN;5423 reg_set_srange64(reg, S16_MIN, S16_MAX);
5662 reg->smax_value = reg->s32_max_value = S16_MAX;5424 reg_set_srange32(reg, S16_MIN, S16_MAX);
5663 } else {5425 } else {
5664 /* size == 4 */5426 /* size == 4 */
5665 reg->smin_value = reg->s32_min_value = S32_MIN;5427 reg_set_srange64(reg, S32_MIN, S32_MAX);
5666 reg->smax_value = reg->s32_max_value = S32_MAX;5428 reg_set_srange32(reg, S32_MIN, S32_MAX);
5667 }5429 }
5668 reg->umin_value = reg->u32_min_value = 0;
5669 reg->umax_value = U64_MAX;
5670 reg->u32_max_value = U32_MAX;
5671 reg->var_off = tnum_unknown;5430 reg->var_off = tnum_unknown;
5672}5431}
5673 5432
@@ -5688,29 +5447,27 @@ static void coerce_reg_to_size_sx(struct bpf_reg_state *reg, int size) · 14 unmodified lines
5688 reg->var_off = tnum_const((s32)u64_cval);5447 reg->var_off = tnum_const((s32)u64_cval);
5689 5448
5690 u64_cval = reg->var_off.value;5449 u64_cval = reg->var_off.value;
5691 reg->smax_value = reg->smin_value = u64_cval;5450 reg->r64 = cnum64_from_urange(u64_cval, u64_cval);
5692 reg->umax_value = reg->umin_value = u64_cval;5451 reg->r32 = cnum32_from_urange((u32)u64_cval, (u32)u64_cval);
5693 reg->s32_max_value = reg->s32_min_value = u64_cval;
5694 reg->u32_max_value = reg->u32_min_value = u64_cval;
5695 return;5452 return;
5696 }5453 }
5697 5454
5698 top_smax_value = ((u64)reg->smax_value >> num_bits) << num_bits;5455 top_smax_value = ((u64)reg_smax(reg) >> num_bits) << num_bits;
5699 top_smin_value = ((u64)reg->smin_value >> num_bits) << num_bits;5456 top_smin_value = ((u64)reg_smin(reg) >> num_bits) << num_bits;
5700 5457
5701 if (top_smax_value != top_smin_value)5458 if (top_smax_value != top_smin_value)
5702 goto out;5459 goto out;
5703 5460
5704 /* find the s64_min and s64_min after sign extension */5461 /* find the s64_min and s64_min after sign extension */
5705 if (size == 1) {5462 if (size == 1) {
5706 init_s64_max = (s8)reg->smax_value;5463 init_s64_max = (s8)reg_smax(reg);
5707 init_s64_min = (s8)reg->smin_value;5464 init_s64_min = (s8)reg_smin(reg);
5708 } else if (size == 2) {5465 } else if (size == 2) {
5709 init_s64_max = (s16)reg->smax_value;5466 init_s64_max = (s16)reg_smax(reg);
5710 init_s64_min = (s16)reg->smin_value;5467 init_s64_min = (s16)reg_smin(reg);
5711 } else {5468 } else {
5712 init_s64_max = (s32)reg->smax_value;5469 init_s64_max = (s32)reg_smax(reg);
5713 init_s64_min = (s32)reg->smin_value;5470 init_s64_min = (s32)reg_smin(reg);
5714 }5471 }
5715 5472
5716 s64_max = max(init_s64_max, init_s64_min);5473 s64_max = max(init_s64_max, init_s64_min);
@@ -5718,10 +5475,8 @@ static void coerce_reg_to_size_sx(struct bpf_reg_state *reg, int size) · 1 unmodified line
5718 5475
5719 /* both of s64_max/s64_min positive or negative */5476 /* both of s64_max/s64_min positive or negative */
5720 if ((s64_max >= 0) == (s64_min >= 0)) {5477 if ((s64_max >= 0) == (s64_min >= 0)) {
5721 reg->s32_min_value = reg->smin_value = s64_min;5478 reg_set_srange64(reg, s64_min, s64_max);
5722 reg->s32_max_value = reg->smax_value = s64_max;5479 reg_set_srange32(reg, s64_min, s64_max);
5723 reg->u32_min_value = reg->umin_value = s64_min;
5724 reg->u32_max_value = reg->umax_value = s64_max;
5725 reg->var_off = tnum_range(s64_min, s64_max);5480 reg->var_off = tnum_range(s64_min, s64_max);
5726 return;5481 return;
5727 }5482 }
@@ -5732,16 +5487,11 @@ out: · 4 unmodified lines
5732 5487
5733static void set_sext32_default_val(struct bpf_reg_state *reg, int size)5488static void set_sext32_default_val(struct bpf_reg_state *reg, int size)
5734{5489{
5735 if (size == 1) {5490 if (size == 1)
5736 reg->s32_min_value = S8_MIN;5491 reg_set_srange32(reg, S8_MIN, S8_MAX);
5737 reg->s32_max_value = S8_MAX;5492 else
5738 } else {
5739 /* size == 2 */5493 /* size == 2 */
5740 reg->s32_min_value = S16_MIN;5494 reg_set_srange32(reg, S16_MIN, S16_MAX);
5741 reg->s32_max_value = S16_MAX;
5742 }
5743 reg->u32_min_value = 0;
5744 reg->u32_max_value = U32_MAX;
5745 reg->var_off = tnum_subreg(tnum_unknown);5495 reg->var_off = tnum_subreg(tnum_unknown);
5746}5496}
5747 5497
@@ -5759,34 +5509,30 @@ static void coerce_subreg_to_size_sx(struct bpf_reg_state *reg, int size) · 11 unmodified lines
5759 reg->var_off = tnum_const((s16)u32_val);5509 reg->var_off = tnum_const((s16)u32_val);
5760 5510
5761 u32_val = reg->var_off.value;5511 u32_val = reg->var_off.value;
5762 reg->s32_min_value = reg->s32_max_value = u32_val;5512 reg_set_srange32(reg, u32_val, u32_val);
5763 reg->u32_min_value = reg->u32_max_value = u32_val;
5764 return;5513 return;
5765 }5514 }
5766 5515
5767 top_smax_value = ((u32)reg->s32_max_value >> num_bits) << num_bits;5516 top_smax_value = ((u32)reg_s32_max(reg) >> num_bits) << num_bits;
5768 top_smin_value = ((u32)reg->s32_min_value >> num_bits) << num_bits;5517 top_smin_value = ((u32)reg_s32_min(reg) >> num_bits) << num_bits;
5769 5518
5770 if (top_smax_value != top_smin_value)5519 if (top_smax_value != top_smin_value)
5771 goto out;5520 goto out;
5772 5521
5773 /* find the s32_min and s32_min after sign extension */5522 /* find the s32_min and s32_min after sign extension */
5774 if (size == 1) {5523 if (size == 1) {
5775 init_s32_max = (s8)reg->s32_max_value;5524 init_s32_max = (s8)reg_s32_max(reg);
5776 init_s32_min = (s8)reg->s32_min_value;5525 init_s32_min = (s8)reg_s32_min(reg);
5777 } else {5526 } else {
5778 /* size == 2 */5527 /* size == 2 */
5779 init_s32_max = (s16)reg->s32_max_value;5528 init_s32_max = (s16)reg_s32_max(reg);
5780 init_s32_min = (s16)reg->s32_min_value;5529 init_s32_min = (s16)reg_s32_min(reg);
5781 }5530 }
5782 s32_max = max(init_s32_max, init_s32_min);5531 s32_max = max(init_s32_max, init_s32_min);
5783 s32_min = min(init_s32_max, init_s32_min);5532 s32_min = min(init_s32_max, init_s32_min);
5784 5533
5785 if ((s32_min >= 0) == (s32_max >= 0)) {5534 if ((s32_min >= 0) == (s32_max >= 0)) {
5786 reg->s32_min_value = s32_min;5535 reg_set_srange32(reg, s32_min, s32_max);
5787 reg->s32_max_value = s32_max;
5788 reg->u32_min_value = (u32)s32_min;
5789 reg->u32_max_value = (u32)s32_max;
5790 reg->var_off = tnum_subreg(tnum_range(s32_min, s32_max));5536 reg->var_off = tnum_subreg(tnum_range(s32_min, s32_max));
5791 return;5537 return;
5792 }5538 }
@@ -5976,12 +5722,11 @@ static bool type_is_trusted_or_null(struct bpf_verifier_env *env, · 183 unmodified lines
5976}5722}
5977 5723
5978static int check_ptr_to_btf_access(struct bpf_verifier_env *env,5724static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
5979 struct bpf_reg_state *regs,5725 struct bpf_reg_state *regs, struct bpf_reg_state *reg,
5980 int regno, int off, int size,5726 argno_t argno, int off, int size,
5981 enum bpf_access_type atype,5727 enum bpf_access_type atype,
5982 int value_regno)5728 int value_regno)
5983{5729{
5984 struct bpf_reg_state *reg = regs + regno;
5985 const struct btf_type *t = btf_type_by_id(reg->btf, reg->btf_id);5730 const struct btf_type *t = btf_type_by_id(reg->btf, reg->btf_id);
5986 const char *tname = btf_name_by_offset(reg->btf, t->name_off);5731 const char *tname = btf_name_by_offset(reg->btf, t->name_off);
5987 const char *field_name = NULL;5732 const char *field_name = NULL;
@@ -6007,8 +5752,8 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 19 unmodified lines
6007 5752
6008 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);5753 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
6009 verbose(env,5754 verbose(env,
6010 "R%d is ptr_%s invalid variable offset: off=%d, var_off=%s\n",5755 "%s is ptr_%s invalid variable offset: off=%d, var_off=%s\n",
6011 regno, tname, off, tn_buf);5756 reg_arg_name(env, argno), tname, off, tn_buf);
6012 return -EACCES;5757 return -EACCES;
6013 }5758 }
6014 5759
@@ -6016,22 +5761,22 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 1 unmodified line
6016 5761
6017 if (off < 0) {5762 if (off < 0) {
6018 verbose(env,5763 verbose(env,
6019 "R%d is ptr_%s invalid negative access: off=%d\n",5764 "%s is ptr_%s invalid negative access: off=%d\n",
6020 regno, tname, off);5765 reg_arg_name(env, argno), tname, off);
6021 return -EACCES;5766 return -EACCES;
6022 }5767 }
6023 5768
6024 if (reg->type & MEM_USER) {5769 if (reg->type & MEM_USER) {
6025 verbose(env,5770 verbose(env,
6026 "R%d is ptr_%s access user memory: off=%d\n",5771 "%s is ptr_%s access user memory: off=%d\n",
6027 regno, tname, off);5772 reg_arg_name(env, argno), tname, off);
6028 return -EACCES;5773 return -EACCES;
6029 }5774 }
6030 5775
6031 if (reg->type & MEM_PERCPU) {5776 if (reg->type & MEM_PERCPU) {
6032 verbose(env,5777 verbose(env,
6033 "R%d is ptr_%s access percpu memory: off=%d\n",5778 "%s is ptr_%s access percpu memory: off=%d\n",
6034 regno, tname, off);5779 reg_arg_name(env, argno), tname, off);
6035 return -EACCES;5780 return -EACCES;
6036 }5781 }
6037 5782
@@ -6043,7 +5788,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 5 unmodified lines
6043 ret = env->ops->btf_struct_access(&env->log, reg, off, size);5788 ret = env->ops->btf_struct_access(&env->log, reg, off, size);
6044 } else {5789 } else {
6045 /* Writes are permitted with default btf_struct_access for5790 /* Writes are permitted with default btf_struct_access for
6046 * program allocated objects (which always have ref_obj_id > 0),5791 * program allocated objects (which always have id > 0),
6047 * but not for untrusted PTR_TO_BTF_ID | MEM_ALLOC.5792 * but not for untrusted PTR_TO_BTF_ID | MEM_ALLOC.
6048 */5793 */
6049 if (atype != BPF_READ && !type_is_ptr_alloc_obj(reg->type)) {5794 if (atype != BPF_READ && !type_is_ptr_alloc_obj(reg->type)) {
@@ -6052,8 +5797,8 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 2 unmodified lines
6052 }5797 }
6053 5798
6054 if (type_is_alloc(reg->type) && !type_is_non_owning_ref(reg->type) &&5799 if (type_is_alloc(reg->type) && !type_is_non_owning_ref(reg->type) &&
6055 !(reg->type & MEM_RCU) && !reg->ref_obj_id) {5800 !(reg->type & MEM_RCU) && !reg_is_referenced(env, reg)) {
6056 verifier_bug(env, "ref_obj_id for allocated object must be non-zero");5801 verifier_bug(env, "allocated object must have a referenced id");
6057 return -EFAULT;5802 return -EFAULT;
6058 }5803 }
6059 5804
@@ -6072,7 +5817,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 12 unmodified lines
6072 */5817 */
6073 flag = PTR_UNTRUSTED;5818 flag = PTR_UNTRUSTED;
6074 5819
6075 } else if (is_trusted_reg(reg) || is_rcu_reg(reg)) {5820 } else if (is_trusted_reg(env, reg) || is_rcu_reg(reg)) {
6076 /* By default any pointer obtained from walking a trusted pointer is no5821 /* By default any pointer obtained from walking a trusted pointer is no
6077 * longer trusted, unless the field being accessed has explicitly been5822 * longer trusted, unless the field being accessed has explicitly been
6078 * marked as inheriting its parent's state of trust (either full or RCU).5823 * marked as inheriting its parent's state of trust (either full or RCU).
@@ -6133,12 +5878,11 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 54 unmodified lines
6133}5878}
6134 5879
6135static int check_ptr_to_map_access(struct bpf_verifier_env *env,5880static int check_ptr_to_map_access(struct bpf_verifier_env *env,
6136 struct bpf_reg_state *regs,5881 struct bpf_reg_state *regs, struct bpf_reg_state *reg,
6137 int regno, int off, int size,5882 argno_t argno, int off, int size,
6138 enum bpf_access_type atype,5883 enum bpf_access_type atype,
6139 int value_regno)5884 int value_regno)
6140{5885{
6141 struct bpf_reg_state *reg = regs + regno;
6142 struct bpf_map *map = reg->map_ptr;5886 struct bpf_map *map = reg->map_ptr;
6143 struct bpf_reg_state map_reg;5887 struct bpf_reg_state map_reg;
6144 enum bpf_type_flag flag = 0;5888 enum bpf_type_flag flag = 0;
@@ -6169,8 +5913,8 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env, · 24 unmodified lines
6169 }5913 }
6170 5914
6171 if (off < 0) {5915 if (off < 0) {
6172 verbose(env, "R%d is %s invalid negative access: off=%d\n",5916 verbose(env, "%s is %s invalid negative access: off=%d\n",
6173 regno, tname, off);5917 reg_arg_name(env, argno), tname, off);
6174 return -EACCES;5918 return -EACCES;
6175 }5919 }
6176 5920
@@ -6227,11 +5971,10 @@ static int check_stack_slot_within_bounds(struct bpf_verifier_env *env, · 50 unmodified lines
6227 * 'off' includes `regno->offset`, but not its dynamic part (if any).5971 * 'off' includes `regno->offset`, but not its dynamic part (if any).
6228 */5972 */
6229static int check_stack_access_within_bounds(5973static int check_stack_access_within_bounds(
6230 struct bpf_verifier_env *env,5974 struct bpf_verifier_env *env, struct bpf_reg_state *reg,
6231 int regno, int off, int access_size,5975 argno_t argno, int off, int access_size,
6232 enum bpf_access_type type)5976 enum bpf_access_type type)
6233{5977{
6234 struct bpf_reg_state *reg = reg_state(env, regno);
6235 struct bpf_func_state *state = bpf_func(env, reg);5978 struct bpf_func_state *state = bpf_func(env, reg);
6236 s64 min_off, max_off;5979 s64 min_off, max_off;
6237 int err;5980 int err;
@@ -6246,14 +5989,14 @@ static int check_stack_access_within_bounds( · 8 unmodified lines
6246 min_off = (s64)reg->var_off.value + off;5989 min_off = (s64)reg->var_off.value + off;
6247 max_off = min_off + access_size;5990 max_off = min_off + access_size;
6248 } else {5991 } else {
6249 if (reg->smax_value >= BPF_MAX_VAR_OFF ||5992 if (reg_smax(reg) >= BPF_MAX_VAR_OFF ||
6250 reg->smin_value <= -BPF_MAX_VAR_OFF) {5993 reg_smin(reg) <= -BPF_MAX_VAR_OFF) {
6251 verbose(env, "invalid unbounded variable-offset%s stack R%d\n",5994 verbose(env, "invalid unbounded variable-offset%s stack %s\n",
6252 err_extra, regno);5995 err_extra, reg_arg_name(env, argno));
6253 return -EACCES;5996 return -EACCES;
6254 }5997 }
6255 min_off = reg->smin_value + off;5998 min_off = reg_smin(reg) + off;
6256 max_off = reg->smax_value + off + access_size;5999 max_off = reg_smax(reg) + off + access_size;
6257 }6000 }
6258 6001
6259 err = check_stack_slot_within_bounds(env, min_off, state, type);6002 err = check_stack_slot_within_bounds(env, min_off, state, type);
@@ -6267,14 +6010,14 @@ static int check_stack_access_within_bounds( · 7 unmodified lines
6267 6010
6268 if (err) {6011 if (err) {
6269 if (tnum_is_const(reg->var_off)) {6012 if (tnum_is_const(reg->var_off)) {
6270 verbose(env, "invalid%s stack R%d off=%lld size=%d\n",6013 verbose(env, "invalid%s stack %s off=%lld size=%d\n",
6271 err_extra, regno, min_off, access_size);6014 err_extra, reg_arg_name(env, argno), min_off, access_size);
6272 } else {6015 } else {
6273 char tn_buf[48];6016 char tn_buf[48];
6274 6017
6275 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);6018 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
6276 verbose(env, "invalid variable-offset%s stack R%d var_off=%s off=%d size=%d\n",6019 verbose(env, "invalid variable-offset%s stack %s var_off=%s off=%d size=%d\n",
6277 err_extra, regno, tn_buf, off, access_size);6020 err_extra, reg_arg_name(env, argno), tn_buf, off, access_size);
6278 }6021 }
6279 return err;6022 return err;
6280 }6023 }
@@ -6319,12 +6062,11 @@ static void add_scalar_to_reg(struct bpf_reg_state *dst_reg, s64 val) · 38 unmodified lines
6319 * if t==write && value_regno==-1, some unknown value is stored into memory6062 * if t==write && value_regno==-1, some unknown value is stored into memory
6320 * if t==read && value_regno==-1, don't care what we read from memory6063 * if t==read && value_regno==-1, don't care what we read from memory
6321 */6064 */
6322static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regno,6065static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct bpf_reg_state *reg, argno_t argno,
6323 int off, int bpf_size, enum bpf_access_type t,6066 int off, int bpf_size, enum bpf_access_type t,
6324 int value_regno, bool strict_alignment_once, bool is_ldsx)6067 int value_regno, bool strict_alignment_once, bool is_ldsx)
6325{6068{
6326 struct bpf_reg_state *regs = cur_regs(env);6069 struct bpf_reg_state *regs = cur_regs(env);
6327 struct bpf_reg_state *reg = regs + regno;
6328 int size, err = 0;6070 int size, err = 0;
6329 6071
6330 size = bpf_size_to_bytes(bpf_size);6072 size = bpf_size_to_bytes(bpf_size);
@@ -6337,11 +6079,12 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 6 unmodified lines
6337 6079
6338 if (reg->type == PTR_TO_MAP_KEY) {6080 if (reg->type == PTR_TO_MAP_KEY) {
6339 if (t == BPF_WRITE) {6081 if (t == BPF_WRITE) {
6340 verbose(env, "write to change key R%d not allowed\n", regno);6082 verbose(env, "write to change key %s not allowed\n",
6083 reg_arg_name(env, argno));
6341 return -EACCES;6084 return -EACCES;
6342 }6085 }
6343 6086
6344 err = check_mem_region_access(env, regno, off, size,6087 err = check_mem_region_access(env, reg, argno, off, size,
6345 reg->map_ptr->key_size, false);6088 reg->map_ptr->key_size, false);
6346 if (err)6089 if (err)
6347 return err;6090 return err;
@@ -6355,17 +6098,17 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 7 unmodified lines
6355 verbose(env, "R%d leaks addr into map\n", value_regno);6098 verbose(env, "R%d leaks addr into map\n", value_regno);
6356 return -EACCES;6099 return -EACCES;
6357 }6100 }
6358 err = check_map_access_type(env, regno, off, size, t);6101 err = check_map_access_type(env, reg, off, size, t);
6359 if (err)6102 if (err)
6360 return err;6103 return err;
6361 err = check_map_access(env, regno, off, size, false, ACCESS_DIRECT);6104 err = check_map_access(env, reg, argno, off, size, false, ACCESS_DIRECT);
6362 if (err)6105 if (err)
6363 return err;6106 return err;
6364 if (tnum_is_const(reg->var_off))6107 if (tnum_is_const(reg->var_off))
6365 kptr_field = btf_record_find(reg->map_ptr->record,6108 kptr_field = btf_record_find(reg->map_ptr->record,
6366 off + reg->var_off.value, BPF_KPTR | BPF_UPTR);6109 off + reg->var_off.value, BPF_KPTR | BPF_UPTR);
6367 if (kptr_field) {6110 if (kptr_field) {
6368 err = check_map_kptr_access(env, regno, value_regno, insn_idx, kptr_field);6111 err = check_map_kptr_access(env, value_regno, insn_idx, kptr_field);
6369 } else if (t == BPF_READ && value_regno >= 0) {6112 } else if (t == BPF_READ && value_regno >= 0) {
6370 struct bpf_map *map = reg->map_ptr;6113 struct bpf_map *map = reg->map_ptr;
6371 6114
@@ -6393,7 +6136,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 21 unmodified lines
6393 size);6136 size);
6394 return -EACCES;6137 return -EACCES;
6395 }6138 }
6396 copy_register_state(&regs[value_regno], reg);6139 regs[value_regno] = *reg;
6397 add_scalar_to_reg(&regs[value_regno], off);6140 add_scalar_to_reg(&regs[value_regno], off);
6398 regs[value_regno].type = PTR_TO_INSN;6141 regs[value_regno].type = PTR_TO_INSN;
6399 } else {6142 } else {
@@ -6405,14 +6148,14 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 5 unmodified lines
6405 bool rdonly_untrusted = rdonly_mem && (reg->type & PTR_UNTRUSTED);6148 bool rdonly_untrusted = rdonly_mem && (reg->type & PTR_UNTRUSTED);
6406 6149
6407 if (type_may_be_null(reg->type)) {6150 if (type_may_be_null(reg->type)) {
6408 verbose(env, "R%d invalid mem access '%s'\n", regno,6151 verbose(env, "%s invalid mem access '%s'\n", reg_arg_name(env, argno),
6409 reg_type_str(env, reg->type));6152 reg_type_str(env, reg->type));
6410 return -EACCES;6153 return -EACCES;
6411 }6154 }
6412 6155
6413 if (t == BPF_WRITE && rdonly_mem) {6156 if (t == BPF_WRITE && rdonly_mem) {
6414 verbose(env, "R%d cannot write into %s\n",6157 verbose(env, "%s cannot write into %s\n",
6415 regno, reg_type_str(env, reg->type));6158 reg_arg_name(env, argno), reg_type_str(env, reg->type));
6416 return -EACCES;6159 return -EACCES;
6417 }6160 }
6418 6161
@@ -6427,7 +6170,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 8 unmodified lines
6427 * instructions, hence no need to check bounds in that case.6170 * instructions, hence no need to check bounds in that case.
6428 */6171 */
6429 if (!rdonly_untrusted)6172 if (!rdonly_untrusted)
6430 err = check_mem_region_access(env, regno, off, size,6173 err = check_mem_region_access(env, reg, argno, off, size,
6431 reg->mem_size, false);6174 reg->mem_size, false);
6432 if (!err && value_regno >= 0 && (t == BPF_READ || rdonly_mem))6175 if (!err && value_regno >= 0 && (t == BPF_READ || rdonly_mem))
6433 mark_reg_unknown(env, regs, value_regno);6176 mark_reg_unknown(env, regs, value_regno);
@@ -6445,7 +6188,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 11 unmodified lines
6445 return -EACCES;6188 return -EACCES;
6446 }6189 }
6447 6190
6448 err = check_ctx_access(env, insn_idx, regno, off, size, t, &info);6191 err = check_ctx_access(env, insn_idx, reg, argno, off, size, t, &info);
6449 if (!err && t == BPF_READ && value_regno >= 0) {6192 if (!err && t == BPF_READ && value_regno >= 0) {
6450 /* ctx access returns either a scalar, or a6193 /* ctx access returns either a scalar, or a
6451 * PTR_TO_PACKET[_META,_END]. In the latter6194 * PTR_TO_PACKET[_META,_END]. In the latter
@@ -6463,8 +6206,6 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 11 unmodified lines
6463 } else {6206 } else {
6464 mark_reg_known_zero(env, regs,6207 mark_reg_known_zero(env, regs,
6465 value_regno);6208 value_regno);
6466 if (type_may_be_null(info.reg_type))
6467 regs[value_regno].id = ++env->id_gen;
6468 /* A load of ctx field could have different6209 /* A load of ctx field could have different
6469 * actual load size with the one encoded in the6210 * actual load size with the one encoded in the
6470 * insn. When the dst is PTR, it is for sure not6211 * insn. When the dst is PTR, it is for sure not
@@ -6474,23 +6215,25 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 3 unmodified lines
6474 if (base_type(info.reg_type) == PTR_TO_BTF_ID) {6215 if (base_type(info.reg_type) == PTR_TO_BTF_ID) {
6475 regs[value_regno].btf = info.btf;6216 regs[value_regno].btf = info.btf;
6476 regs[value_regno].btf_id = info.btf_id;6217 regs[value_regno].btf_id = info.btf_id;
6477 regs[value_regno].ref_obj_id = info.ref_obj_id;6218 regs[value_regno].id = info.ref_id;
6478 }6219 }
6220 if (type_may_be_null(info.reg_type) && !regs[value_regno].id)
6221 regs[value_regno].id = ++env->id_gen;
6479 }6222 }
6480 regs[value_regno].type = info.reg_type;6223 regs[value_regno].type = info.reg_type;
6481 }6224 }
6482 6225
6483 } else if (reg->type == PTR_TO_STACK) {6226 } else if (reg->type == PTR_TO_STACK) {
6484 /* Basic bounds checks. */6227 /* Basic bounds checks. */
6485 err = check_stack_access_within_bounds(env, regno, off, size, t);6228 err = check_stack_access_within_bounds(env, reg, argno, off, size, t);
6486 if (err)6229 if (err)
6487 return err;6230 return err;
6488 6231
6489 if (t == BPF_READ)6232 if (t == BPF_READ)
6490 err = check_stack_read(env, regno, off, size,6233 err = check_stack_read(env, reg, argno, off, size,
6491 value_regno);6234 value_regno);
6492 else6235 else
6493 err = check_stack_write(env, regno, off, size,6236 err = check_stack_write(env, reg, off, size,
6494 value_regno, insn_idx);6237 value_regno, insn_idx);
6495 } else if (reg_is_pkt_pointer(reg)) {6238 } else if (reg_is_pkt_pointer(reg)) {
6496 if (t == BPF_WRITE && !may_access_direct_pkt_data(env, NULL, t)) {6239 if (t == BPF_WRITE && !may_access_direct_pkt_data(env, NULL, t)) {
@@ -6503,7 +6246,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 6 unmodified lines
6503 value_regno);6246 value_regno);
6504 return -EACCES;6247 return -EACCES;
6505 }6248 }
6506 err = check_packet_access(env, regno, off, size, false);6249 err = check_packet_access(env, reg, argno, off, size, false);
6507 if (!err && t == BPF_READ && value_regno >= 0)6250 if (!err && t == BPF_READ && value_regno >= 0)
6508 mark_reg_unknown(env, regs, value_regno);6251 mark_reg_unknown(env, regs, value_regno);
6509 } else if (reg->type == PTR_TO_FLOW_KEYS) {6252 } else if (reg->type == PTR_TO_FLOW_KEYS) {
@@ -6514,28 +6257,28 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 4 unmodified lines
6514 return -EACCES;6257 return -EACCES;
6515 }6258 }
6516 6259
6517 err = check_flow_keys_access(env, off, size);6260 err = check_flow_keys_access(env, reg, argno, off, size);
6518 if (!err && t == BPF_READ && value_regno >= 0)6261 if (!err && t == BPF_READ && value_regno >= 0)
6519 mark_reg_unknown(env, regs, value_regno);6262 mark_reg_unknown(env, regs, value_regno);
6520 } else if (type_is_sk_pointer(reg->type)) {6263 } else if (type_is_sk_pointer(reg->type)) {
6521 if (t == BPF_WRITE) {6264 if (t == BPF_WRITE) {
6522 verbose(env, "R%d cannot write into %s\n",6265 verbose(env, "%s cannot write into %s\n",
6523 regno, reg_type_str(env, reg->type));6266 reg_arg_name(env, argno), reg_type_str(env, reg->type));
6524 return -EACCES;6267 return -EACCES;
6525 }6268 }
6526 err = check_sock_access(env, insn_idx, regno, off, size, t);6269 err = check_sock_access(env, insn_idx, reg, argno, off, size, t);
6527 if (!err && value_regno >= 0)6270 if (!err && value_regno >= 0)
6528 mark_reg_unknown(env, regs, value_regno);6271 mark_reg_unknown(env, regs, value_regno);
6529 } else if (reg->type == PTR_TO_TP_BUFFER) {6272 } else if (reg->type == PTR_TO_TP_BUFFER) {
6530 err = check_tp_buffer_access(env, reg, regno, off, size);6273 err = check_tp_buffer_access(env, reg, argno, off, size);
6531 if (!err && t == BPF_READ && value_regno >= 0)6274 if (!err && t == BPF_READ && value_regno >= 0)
6532 mark_reg_unknown(env, regs, value_regno);6275 mark_reg_unknown(env, regs, value_regno);
6533 } else if (base_type(reg->type) == PTR_TO_BTF_ID &&6276 } else if (base_type(reg->type) == PTR_TO_BTF_ID &&
6534 !type_may_be_null(reg->type)) {6277 !type_may_be_null(reg->type)) {
6535 err = check_ptr_to_btf_access(env, regs, regno, off, size, t,6278 err = check_ptr_to_btf_access(env, regs, reg, argno, off, size, t,
6536 value_regno);6279 value_regno);
6537 } else if (reg->type == CONST_PTR_TO_MAP) {6280 } else if (reg->type == CONST_PTR_TO_MAP) {
6538 err = check_ptr_to_map_access(env, regs, regno, off, size, t,6281 err = check_ptr_to_map_access(env, regs, reg, argno, off, size, t,
6539 value_regno);6282 value_regno);
6540 } else if (base_type(reg->type) == PTR_TO_BUF &&6283 } else if (base_type(reg->type) == PTR_TO_BUF &&
6541 !type_may_be_null(reg->type)) {6284 !type_may_be_null(reg->type)) {
@@ -6544,8 +6287,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 2 unmodified lines
6544 6287
6545 if (rdonly_mem) {6288 if (rdonly_mem) {
6546 if (t == BPF_WRITE) {6289 if (t == BPF_WRITE) {
6547 verbose(env, "R%d cannot write into %s\n",6290 verbose(env, "%s cannot write into %s\n",
6548 regno, reg_type_str(env, reg->type));6291 reg_arg_name(env, argno), reg_type_str(env, reg->type));
6549 return -EACCES;6292 return -EACCES;
6550 }6293 }
6551 max_access = &env->prog->aux->max_rdonly_access;6294 max_access = &env->prog->aux->max_rdonly_access;
@@ -6553,7 +6296,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 1 unmodified line
6553 max_access = &env->prog->aux->max_rdwr_access;6296 max_access = &env->prog->aux->max_rdwr_access;
6554 }6297 }
6555 6298
6556 err = check_buffer_access(env, reg, regno, off, size, false,6299 err = check_buffer_access(env, reg, argno, off, size, false,
6557 max_access);6300 max_access);
6558 6301
6559 if (!err && value_regno >= 0 && (rdonly_mem || t == BPF_READ))6302 if (!err && value_regno >= 0 && (rdonly_mem || t == BPF_READ))
@@ -6562,7 +6305,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 2 unmodified lines
6562 if (t == BPF_READ && value_regno >= 0)6305 if (t == BPF_READ && value_regno >= 0)
6563 mark_reg_unknown(env, regs, value_regno);6306 mark_reg_unknown(env, regs, value_regno);
6564 } else {6307 } else {
6565 verbose(env, "R%d invalid mem access '%s'\n", regno,6308 verbose(env, "%s invalid mem access '%s'\n", reg_arg_name(env, argno),
6566 reg_type_str(env, reg->type));6309 reg_type_str(env, reg->type));
6567 return -EACCES;6310 return -EACCES;
6568 }6311 }
@@ -6585,10 +6328,20 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, · 16 unmodified lines
6585 bool strict_alignment_once, bool is_ldsx,6328 bool strict_alignment_once, bool is_ldsx,
6586 bool allow_trust_mismatch, const char *ctx)6329 bool allow_trust_mismatch, const char *ctx)
6587{6330{
6331 struct bpf_verifier_state *vstate = env->cur_state;
6332 struct bpf_func_state *state = vstate->frame[vstate->curframe];
6588 struct bpf_reg_state *regs = cur_regs(env);6333 struct bpf_reg_state *regs = cur_regs(env);
6589 enum bpf_reg_type src_reg_type;6334 enum bpf_reg_type src_reg_type;
6590 int err;6335 int err;
6591 6336
6337 /* Handle stack arg read */
6338 if (is_stack_arg_ldx(insn)) {
6339 err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK);
6340 if (err)
6341 return err;
6342 return check_stack_arg_read(env, state, insn->off, insn->dst_reg);
6343 }
6344
6592 /* check src operand */6345 /* check src operand */
6593 err = check_reg_arg(env, insn->src_reg, SRC_OP);6346 err = check_reg_arg(env, insn->src_reg, SRC_OP);
6594 if (err)6347 if (err)
@@ -6604,7 +6357,7 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, · 9 unmodified lines
6604 /* Check if (src_reg + off) is readable. The state of dst_reg will be6357 /* Check if (src_reg + off) is readable. The state of dst_reg will be
6605 * updated by this call.6358 * updated by this call.
6606 */6359 */
6607 err = check_mem_access(env, env->insn_idx, insn->src_reg, insn->off,6360 err = check_mem_access(env, env->insn_idx, regs + insn->src_reg, argno_from_reg(insn->src_reg), insn->off,
6608 BPF_SIZE(insn->code), BPF_READ, insn->dst_reg,6361 BPF_SIZE(insn->code), BPF_READ, insn->dst_reg,
6609 strict_alignment_once, is_ldsx);6362 strict_alignment_once, is_ldsx);
6610 err = err ?: save_aux_ptr_type(env, src_reg_type,6363 err = err ?: save_aux_ptr_type(env, src_reg_type,
@@ -6617,10 +6370,20 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, · 6 unmodified lines
6617static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn,6370static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn,
6618 bool strict_alignment_once)6371 bool strict_alignment_once)
6619{6372{
6373 struct bpf_verifier_state *vstate = env->cur_state;
6374 struct bpf_func_state *state = vstate->frame[vstate->curframe];
6620 struct bpf_reg_state *regs = cur_regs(env);6375 struct bpf_reg_state *regs = cur_regs(env);
6621 enum bpf_reg_type dst_reg_type;6376 enum bpf_reg_type dst_reg_type;
6622 int err;6377 int err;
6623 6378
6379 /* Handle stack arg write */
6380 if (is_stack_arg_stx(insn)) {
6381 err = check_reg_arg(env, insn->src_reg, SRC_OP);
6382 if (err)
6383 return err;
6384 return check_stack_arg_write(env, state, insn->off, regs + insn->src_reg);
6385 }
6386
6624 /* check src1 operand */6387 /* check src1 operand */
6625 err = check_reg_arg(env, insn->src_reg, SRC_OP);6388 err = check_reg_arg(env, insn->src_reg, SRC_OP);
6626 if (err)6389 if (err)
@@ -6634,7 +6397,7 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, · 7 unmodified lines
6634 dst_reg_type = regs[insn->dst_reg].type;6397 dst_reg_type = regs[insn->dst_reg].type;
6635 6398
6636 /* Check if (dst_reg + off) is writeable. */6399 /* Check if (dst_reg + off) is writeable. */
6637 err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off,6400 err = check_mem_access(env, env->insn_idx, regs + insn->dst_reg, argno_from_reg(insn->dst_reg), insn->off,
6638 BPF_SIZE(insn->code), BPF_WRITE, insn->src_reg,6401 BPF_SIZE(insn->code), BPF_WRITE, insn->src_reg,
6639 strict_alignment_once, false);6402 strict_alignment_once, false);
6640 err = err ?: save_aux_ptr_type(env, dst_reg_type, false);6403 err = err ?: save_aux_ptr_type(env, dst_reg_type, false);
@@ -6645,6 +6408,7 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, · 4 unmodified lines
6645static int check_atomic_rmw(struct bpf_verifier_env *env,6408static int check_atomic_rmw(struct bpf_verifier_env *env,
6646 struct bpf_insn *insn)6409 struct bpf_insn *insn)
6647{6410{
6411 struct bpf_reg_state *dst_reg;
6648 int load_reg;6412 int load_reg;
6649 int err;6413 int err;
6650 6414
@@ -6706,13 +6470,15 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, · 55 unmodified lines
6706 load_reg = -1;6470 load_reg = -1;
6707 }6471 }
6708 6472
6473 dst_reg = cur_regs(env) + insn->dst_reg;
6474
6709 /* Check whether we can read the memory, with second call for fetch6475 /* Check whether we can read the memory, with second call for fetch
6710 * case to simulate the register fill.6476 * case to simulate the register fill.
6711 */6477 */
6712 err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off,6478 err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), insn->off,
6713 BPF_SIZE(insn->code), BPF_READ, -1, true, false);6479 BPF_SIZE(insn->code), BPF_READ, -1, true, false);
6714 if (!err && load_reg >= 0)6480 if (!err && load_reg >= 0)
6715 err = check_mem_access(env, env->insn_idx, insn->dst_reg,6481 err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg),
6716 insn->off, BPF_SIZE(insn->code),6482 insn->off, BPF_SIZE(insn->code),
6717 BPF_READ, load_reg, true, false);6483 BPF_READ, load_reg, true, false);
6718 if (err)6484 if (err)
@@ -6724,7 +6490,7 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, · 5 unmodified lines
6724 return err;6490 return err;
6725 }6491 }
6726 /* Check whether we can write into the same memory. */6492 /* Check whether we can write into the same memory. */
6727 err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off,6493 err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), insn->off,
6728 BPF_SIZE(insn->code), BPF_WRITE, -1, true, false);6494 BPF_SIZE(insn->code), BPF_WRITE, -1, true, false);
6729 if (err)6495 if (err)
6730 return err;6496 return err;
@@ -6813,11 +6579,10 @@ static int check_atomic(struct bpf_verifier_env *env, struct bpf_insn *insn) · 82 unmodified lines
6813 * read offsets are marked as read.6579 * read offsets are marked as read.
6814 */6580 */
6815static int check_stack_range_initialized(6581static int check_stack_range_initialized(
6816 struct bpf_verifier_env *env, int regno, int off,6582 struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int off,
6817 int access_size, bool zero_size_allowed,6583 int access_size, bool zero_size_allowed,
6818 enum bpf_access_type type, struct bpf_call_arg_meta *meta)6584 enum bpf_access_type type, struct bpf_call_arg_meta *meta)
6819{6585{
6820 struct bpf_reg_state *reg = reg_state(env, regno);
6821 struct bpf_func_state *state = bpf_func(env, reg);6586 struct bpf_func_state *state = bpf_func(env, reg);
6822 int err, min_off, max_off, i, j, slot, spi;6587 int err, min_off, max_off, i, j, slot, spi;
6823 /* Some accesses can write anything into the stack, others are6588 /* Some accesses can write anything into the stack, others are
@@ -6839,7 +6604,7 @@ static int check_stack_range_initialized( · 15 unmodified lines
6839 return -EACCES;6604 return -EACCES;
6840 }6605 }
6841 6606
6842 err = check_stack_access_within_bounds(env, regno, off, access_size, type);6607 err = check_stack_access_within_bounds(env, reg, argno, off, access_size, type);
6843 if (err)6608 if (err)
6844 return err;6609 return err;
6845 6610
@@ -6856,8 +6621,8 @@ static int check_stack_range_initialized( · 10 unmodified lines
6856 char tn_buf[48];6621 char tn_buf[48];
6857 6622
6858 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);6623 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
6859 verbose(env, "R%d variable offset stack access prohibited for !root, var_off=%s\n",6624 verbose(env, "%s variable offset stack access prohibited for !root, var_off=%s\n",
6860 regno, tn_buf);6625 reg_arg_name(env, argno), tn_buf);
6861 return -EACCES;6626 return -EACCES;
6862 }6627 }
6863 /* Only initialized buffer on stack is allowed to be accessed6628 /* Only initialized buffer on stack is allowed to be accessed
@@ -6869,8 +6634,8 @@ static int check_stack_range_initialized( · 5 unmodified lines
6869 if (meta && meta->raw_mode)6634 if (meta && meta->raw_mode)
6870 meta = NULL;6635 meta = NULL;
6871 6636
6872 min_off = reg->smin_value + off;6637 min_off = reg_smin(reg) + off;
6873 max_off = reg->smax_value + off;6638 max_off = reg_smax(reg) + off;
6874 }6639 }
6875 6640
6876 if (meta && meta->raw_mode) {6641 if (meta && meta->raw_mode) {
@@ -6900,7 +6665,7 @@ static int check_stack_range_initialized( · 23 unmodified lines
6900 }6665 }
6901 }6666 }
6902 meta->access_size = access_size;6667 meta->access_size = access_size;
6903 meta->regno = regno;6668 meta->regno = reg_from_argno(argno);
6904 return 0;6669 return 0;
6905 }6670 }
6906 6671
@@ -6940,17 +6705,17 @@ static int check_stack_range_initialized( · 33 unmodified lines
6940 if (*stype == STACK_POISON) {6705 if (*stype == STACK_POISON) {
6941 if (allow_poison)6706 if (allow_poison)
6942 goto mark;6707 goto mark;
6943 verbose(env, "reading from stack R%d off %d+%d size %d, slot poisoned by dead code elimination\n",6708 verbose(env, "reading from stack %s off %d+%d size %d, slot poisoned by dead code elimination\n",
6944 regno, min_off, i - min_off, access_size);6709 reg_arg_name(env, argno), min_off, i - min_off, access_size);
6945 } else if (tnum_is_const(reg->var_off)) {6710 } else if (tnum_is_const(reg->var_off)) {
6946 verbose(env, "invalid read from stack R%d off %d+%d size %d\n",6711 verbose(env, "invalid read from stack %s off %d+%d size %d\n",
6947 regno, min_off, i - min_off, access_size);6712 reg_arg_name(env, argno), min_off, i - min_off, access_size);
6948 } else {6713 } else {
6949 char tn_buf[48];6714 char tn_buf[48];
6950 6715
6951 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);6716 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
6952 verbose(env, "invalid read from stack R%d var_off %s+%d size %d\n",6717 verbose(env, "invalid read from stack %s var_off %s+%d size %d\n",
6953 regno, tn_buf, i - min_off, access_size);6718 reg_arg_name(env, argno), tn_buf, i - min_off, access_size);
6954 }6719 }
6955 return -EACCES;6720 return -EACCES;
6956mark:6721mark:
@@ -6959,48 +6724,48 @@ mark: · 2 unmodified lines
6959 return 0;6724 return 0;
6960}6725}
6961 6726
6962static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,6727static int check_helper_mem_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
6963 int access_size, enum bpf_access_type access_type,6728 int access_size, enum bpf_access_type access_type,
6964 bool zero_size_allowed,6729 bool zero_size_allowed,
6965 struct bpf_call_arg_meta *meta)6730 struct bpf_call_arg_meta *meta)
6966{6731{
6967 struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[regno];6732 struct bpf_reg_state *regs = cur_regs(env);
6968 u32 *max_access;6733 u32 *max_access;
6969 6734
6970 switch (base_type(reg->type)) {6735 switch (base_type(reg->type)) {
6971 case PTR_TO_PACKET:6736 case PTR_TO_PACKET:
6972 case PTR_TO_PACKET_META:6737 case PTR_TO_PACKET_META:
6973 return check_packet_access(env, regno, 0, access_size,6738 return check_packet_access(env, reg, argno, 0, access_size,
6974 zero_size_allowed);6739 zero_size_allowed);
6975 case PTR_TO_MAP_KEY:6740 case PTR_TO_MAP_KEY:
6976 if (access_type == BPF_WRITE) {6741 if (access_type == BPF_WRITE) {
6977 verbose(env, "R%d cannot write into %s\n", regno,6742 verbose(env, "%s cannot write into %s\n",
6978 reg_type_str(env, reg->type));6743 reg_arg_name(env, argno), reg_type_str(env, reg->type));
6979 return -EACCES;6744 return -EACCES;
6980 }6745 }
6981 return check_mem_region_access(env, regno, 0, access_size,6746 return check_mem_region_access(env, reg, argno, 0, access_size,
6982 reg->map_ptr->key_size, false);6747 reg->map_ptr->key_size, false);
6983 case PTR_TO_MAP_VALUE:6748 case PTR_TO_MAP_VALUE:
6984 if (check_map_access_type(env, regno, 0, access_size, access_type))6749 if (check_map_access_type(env, reg, 0, access_size, access_type))
6985 return -EACCES;6750 return -EACCES;
6986 return check_map_access(env, regno, 0, access_size,6751 return check_map_access(env, reg, argno, 0, access_size,
6987 zero_size_allowed, ACCESS_HELPER);6752 zero_size_allowed, ACCESS_HELPER);
6988 case PTR_TO_MEM:6753 case PTR_TO_MEM:
6989 if (type_is_rdonly_mem(reg->type)) {6754 if (type_is_rdonly_mem(reg->type)) {
6990 if (access_type == BPF_WRITE) {6755 if (access_type == BPF_WRITE) {
6991 verbose(env, "R%d cannot write into %s\n", regno,6756 verbose(env, "%s cannot write into %s\n",
6992 reg_type_str(env, reg->type));6757 reg_arg_name(env, argno), reg_type_str(env, reg->type));
6993 return -EACCES;6758 return -EACCES;
6994 }6759 }
6995 }6760 }
6996 return check_mem_region_access(env, regno, 0,6761 return check_mem_region_access(env, reg, argno, 0,
6997 access_size, reg->mem_size,6762 access_size, reg->mem_size,
6998 zero_size_allowed);6763 zero_size_allowed);
6999 case PTR_TO_BUF:6764 case PTR_TO_BUF:
7000 if (type_is_rdonly_mem(reg->type)) {6765 if (type_is_rdonly_mem(reg->type)) {
7001 if (access_type == BPF_WRITE) {6766 if (access_type == BPF_WRITE) {
7002 verbose(env, "R%d cannot write into %s\n", regno,6767 verbose(env, "%s cannot write into %s\n",
7003 reg_type_str(env, reg->type));6768 reg_arg_name(env, argno), reg_type_str(env, reg->type));
7004 return -EACCES;6769 return -EACCES;
7005 }6770 }
7006 6771
@@ -7008,26 +6773,26 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 1 unmodified line
7008 } else {6773 } else {
7009 max_access = &env->prog->aux->max_rdwr_access;6774 max_access = &env->prog->aux->max_rdwr_access;
7010 }6775 }
7011 return check_buffer_access(env, reg, regno, 0,6776 return check_buffer_access(env, reg, argno, 0,
7012 access_size, zero_size_allowed,6777 access_size, zero_size_allowed,
7013 max_access);6778 max_access);
7014 case PTR_TO_STACK:6779 case PTR_TO_STACK:
7015 return check_stack_range_initialized(6780 return check_stack_range_initialized(
7016 env,6781 env, reg,
7017 regno, 0, access_size,6782 argno, 0, access_size,
7018 zero_size_allowed, access_type, meta);6783 zero_size_allowed, access_type, meta);
7019 case PTR_TO_BTF_ID:6784 case PTR_TO_BTF_ID:
7020 return check_ptr_to_btf_access(env, regs, regno, 0,6785 return check_ptr_to_btf_access(env, regs, reg, argno, 0,
7021 access_size, BPF_READ, -1);6786 access_size, access_type, -1);
7022 case PTR_TO_CTX:6787 case PTR_TO_CTX:
7023 /* Only permit reading or writing syscall context using helper calls. */6788 /* Only permit reading or writing syscall context using helper calls. */
7024 if (is_var_ctx_off_allowed(env->prog)) {6789 if (is_var_ctx_off_allowed(env->prog)) {
7025 int err = check_mem_region_access(env, regno, 0, access_size, U16_MAX,6790 int err = check_mem_region_access(env, reg, argno, 0, access_size, U16_MAX,
7026 zero_size_allowed);6791 zero_size_allowed);
7027 if (err)6792 if (err)
7028 return err;6793 return err;
7029 if (env->prog->aux->max_ctx_offset < reg->umax_value + access_size)6794 if (env->prog->aux->max_ctx_offset < reg_umax(reg) + access_size)
7030 env->prog->aux->max_ctx_offset = reg->umax_value + access_size;6795 env->prog->aux->max_ctx_offset = reg_umax(reg) + access_size;
7031 return 0;6796 return 0;
7032 }6797 }
7033 fallthrough;6798 fallthrough;
@@ -7037,7 +6802,7 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 3 unmodified lines
7037 bpf_register_is_null(reg))6802 bpf_register_is_null(reg))
7038 return 0;6803 return 0;
7039 6804
7040 verbose(env, "R%d type=%s ", regno,6805 verbose(env, "%s type=%s ", reg_arg_name(env, argno),
7041 reg_type_str(env, reg->type));6806 reg_type_str(env, reg->type));
7042 verbose(env, "expected=%s\n", reg_type_str(env, PTR_TO_STACK));6807 verbose(env, "expected=%s\n", reg_type_str(env, PTR_TO_STACK));
7043 return -EACCES;6808 return -EACCES;
@@ -7047,12 +6812,12 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 3 unmodified lines
7047/* verify arguments to helpers or kfuncs consisting of a pointer and an access6812/* verify arguments to helpers or kfuncs consisting of a pointer and an access
7048 * size.6813 * size.
7049 *6814 *
7050 * @regno is the register containing the access size. regno-1 is the register6815 * @mem_reg contains the pointer, @size_reg contains the access size.
7051 * containing the pointer.
7052 */6816 */
7053static int check_mem_size_reg(struct bpf_verifier_env *env,6817static int check_mem_size_reg(struct bpf_verifier_env *env,
7054 struct bpf_reg_state *reg, u32 regno,6818 struct bpf_reg_state *mem_reg,
7055 enum bpf_access_type access_type,6819 struct bpf_reg_state *size_reg, argno_t mem_argno,
6820 argno_t size_argno, enum bpf_access_type access_type,
7056 bool zero_size_allowed,6821 bool zero_size_allowed,
7057 struct bpf_call_arg_meta *meta)6822 struct bpf_call_arg_meta *meta)
7058{6823{
@@ -7066,42 +6831,48 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, · 7 unmodified lines
7066 * out. Only upper bounds can be learned because retval is an6831 * out. Only upper bounds can be learned because retval is an
7067 * int type and negative retvals are allowed.6832 * int type and negative retvals are allowed.
7068 */6833 */
7069 meta->msize_max_value = reg->umax_value;6834 meta->msize_max_value = reg_umax(size_reg);
7070 6835
7071 /* The register is SCALAR_VALUE; the access check happens using6836 /* The register is SCALAR_VALUE; the access check happens using
7072 * its boundaries. For unprivileged variable accesses, disable6837 * its boundaries. For unprivileged variable accesses, disable
7073 * raw mode so that the program is required to initialize all6838 * raw mode so that the program is required to initialize all
7074 * the memory that the helper could just partially fill up.6839 * the memory that the helper could just partially fill up.
7075 */6840 */
7076 if (!tnum_is_const(reg->var_off))6841 if (!tnum_is_const(size_reg->var_off))
7077 meta = NULL;6842 meta = NULL;
7078 6843
7079 if (reg->smin_value < 0) {6844 if (reg_smin(size_reg) < 0) {
7080 verbose(env, "R%d min value is negative, either use unsigned or 'var &= const'\n",6845 verbose(env, "%s min value is negative, either use unsigned or 'var &= const'\n",
7081 regno);6846 reg_arg_name(env, size_argno));
7082 return -EACCES;6847 return -EACCES;
7083 }6848 }
7084 6849
7085 if (reg->umin_value == 0 && !zero_size_allowed) {6850 if (reg_umin(size_reg) == 0 && !zero_size_allowed) {
7086 verbose(env, "R%d invalid zero-sized read: u64=[%lld,%lld]\n",6851 verbose(env, "%s invalid zero-sized read: u64=[%lld,%lld]\n",
7087 regno, reg->umin_value, reg->umax_value);6852 reg_arg_name(env, size_argno), reg_umin(size_reg), reg_umax(size_reg));
7088 return -EACCES;6853 return -EACCES;
7089 }6854 }
7090 6855
7091 if (reg->umax_value >= BPF_MAX_VAR_SIZ) {6856 if (reg_umax(size_reg) >= BPF_MAX_VAR_SIZ) {
7092 verbose(env, "R%d unbounded memory access, use 'var &= const' or 'if (var < const)'\n",6857 verbose(env, "%s unbounded memory access, use 'var &= const' or 'if (var < const)'\n",
7093 regno);6858 reg_arg_name(env, size_argno));
7094 return -EACCES;6859 return -EACCES;
7095 }6860 }
7096 err = check_helper_mem_access(env, regno - 1, reg->umax_value,6861 err = check_helper_mem_access(env, mem_reg, mem_argno, reg_umax(size_reg),
7097 access_type, zero_size_allowed, meta);6862 access_type, zero_size_allowed, meta);
7098 if (!err)6863 if (!err) {
7099 err = mark_chain_precision(env, regno);6864 int regno = reg_from_argno(size_argno);
6865
6866 if (regno >= 0)
6867 err = mark_chain_precision(env, regno);
6868 else
6869 err = mark_stack_arg_precision(env, arg_idx_from_argno(size_argno));
6870 }
7100 return err;6871 return err;
7101}6872}
7102 6873
7103static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,6874static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
7104 u32 regno, u32 mem_size)6875 argno_t argno, u32 mem_size)
7105{6876{
7106 bool may_be_null = type_may_be_null(reg->type);6877 bool may_be_null = type_may_be_null(reg->type);
7107 struct bpf_reg_state saved_reg;6878 struct bpf_reg_state saved_reg;
@@ -7110,6 +6881,12 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg · 2 unmodified lines
7110 if (bpf_register_is_null(reg))6881 if (bpf_register_is_null(reg))
7111 return 0;6882 return 0;
7112 6883
6884 if (mem_size > S32_MAX) {
6885 verbose(env, "%s memory size %u is too large\n",
6886 reg_arg_name(env, argno), mem_size);
6887 return -EACCES;
6888 }
6889
7113 /* Assuming that the register contains a value check if the memory6890 /* Assuming that the register contains a value check if the memory
7114 * access is safe. Temporarily save and restore the register's state as6891 * access is safe. Temporarily save and restore the register's state as
7115 * the conversion shouldn't be visible to a caller.6892 * the conversion shouldn't be visible to a caller.
@@ -7121,8 +6898,8 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg · 5 unmodified lines
7121 6898
7122 int size = base_type(reg->type) == PTR_TO_STACK ? -(int)mem_size : mem_size;6899 int size = base_type(reg->type) == PTR_TO_STACK ? -(int)mem_size : mem_size;
7123 6900
7124 err = check_helper_mem_access(env, regno, size, BPF_READ, true, NULL);6901 err = check_helper_mem_access(env, reg, argno, size, BPF_READ, true, NULL);
7125 err = err ?: check_helper_mem_access(env, regno, size, BPF_WRITE, true, NULL);6902 err = err ?: check_helper_mem_access(env, reg, argno, size, BPF_WRITE, true, NULL);
7126 6903
7127 if (may_be_null)6904 if (may_be_null)
7128 *reg = saved_reg;6905 *reg = saved_reg;
@@ -7130,17 +6907,14 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg · 1 unmodified line
7130 return err;6907 return err;
7131}6908}
7132 6909
7133static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,6910static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *mem_reg,
7134 u32 regno)6911 struct bpf_reg_state *size_reg, argno_t mem_argno, argno_t size_argno)
7135{6912{
7136 struct bpf_reg_state *mem_reg = &cur_regs(env)[regno - 1];
7137 bool may_be_null = type_may_be_null(mem_reg->type);6913 bool may_be_null = type_may_be_null(mem_reg->type);
7138 struct bpf_reg_state saved_reg;6914 struct bpf_reg_state saved_reg;
7139 struct bpf_call_arg_meta meta;6915 struct bpf_call_arg_meta meta;
7140 int err;6916 int err;
7141 6917
7142 WARN_ON_ONCE(regno < BPF_REG_2 || regno > BPF_REG_5);
7143
7144 memset(&meta, 0, sizeof(meta));6918 memset(&meta, 0, sizeof(meta));
7145 6919
7146 if (may_be_null) {6920 if (may_be_null) {
@@ -7148,8 +6922,8 @@ static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg · 1 unmodified line
7148 mark_ptr_not_null_reg(mem_reg);6922 mark_ptr_not_null_reg(mem_reg);
7149 }6923 }
7150 6924
7151 err = check_mem_size_reg(env, reg, regno, BPF_READ, true, &meta);6925 err = check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_READ, true, &meta);
7152 err = err ?: check_mem_size_reg(env, reg, regno, BPF_WRITE, true, &meta);6926 err = err ?: check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_WRITE, true, &meta);
7153 6927
7154 if (may_be_null)6928 if (may_be_null)
7155 *mem_reg = saved_reg;6929 *mem_reg = saved_reg;
@@ -7185,11 +6959,10 @@ enum { · 29 unmodified lines
7185 * env->cur_state->active_locks remembers which map value element or allocated6959 * env->cur_state->active_locks remembers which map value element or allocated
7186 * object got locked and clears it after bpf_spin_unlock.6960 * object got locked and clears it after bpf_spin_unlock.
7187 */6961 */
7188static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags)6962static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int flags)
7189{6963{
7190 bool is_lock = flags & PROCESS_SPIN_LOCK, is_res_lock = flags & PROCESS_RES_LOCK;6964 bool is_lock = flags & PROCESS_SPIN_LOCK, is_res_lock = flags & PROCESS_RES_LOCK;
7191 const char *lock_str = is_res_lock ? "bpf_res_spin" : "bpf_spin";6965 const char *lock_str = is_res_lock ? "bpf_res_spin" : "bpf_spin";
7192 struct bpf_reg_state *reg = reg_state(env, regno);
7193 struct bpf_verifier_state *cur = env->cur_state;6966 struct bpf_verifier_state *cur = env->cur_state;
7194 bool is_const = tnum_is_const(reg->var_off);6967 bool is_const = tnum_is_const(reg->var_off);
7195 bool is_irq = flags & PROCESS_LOCK_IRQ;6968 bool is_irq = flags & PROCESS_LOCK_IRQ;
@@ -7202,8 +6975,8 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags) · 6 unmodified lines
7202 6975
7203 if (!is_const) {6976 if (!is_const) {
7204 verbose(env,6977 verbose(env,
7205 "R%d doesn't have constant offset. %s_lock has to be at the constant offset\n",6978 "%s doesn't have constant offset. %s_lock has to be at the constant offset\n",
7206 regno, lock_str);6979 reg_arg_name(env, argno), lock_str);
7207 return -EINVAL;6980 return -EINVAL;
7208 }6981 }
7209 if (reg->type == PTR_TO_MAP_VALUE) {6982 if (reg->type == PTR_TO_MAP_VALUE) {
@@ -7302,11 +7075,10 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags) · 92 unmodified lines
7302}7075}
7303 7076
7304/* Check if @regno is a pointer to a specific field in a map value */7077/* Check if @regno is a pointer to a specific field in a map value */
7305static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno,7078static int check_map_field_pointer(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
7306 enum btf_field_type field_type,7079 enum btf_field_type field_type,
7307 struct bpf_map_desc *map_desc)7080 struct bpf_map_desc *map_desc)
7308{7081{
7309 struct bpf_reg_state *reg = reg_state(env, regno);
7310 bool is_const = tnum_is_const(reg->var_off);7082 bool is_const = tnum_is_const(reg->var_off);
7311 struct bpf_map *map = reg->map_ptr;7083 struct bpf_map *map = reg->map_ptr;
7312 u64 val = reg->var_off.value;7084 u64 val = reg->var_off.value;
@@ -7315,8 +7087,8 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, · 2 unmodified lines
7315 7087
7316 if (!is_const) {7088 if (!is_const) {
7317 verbose(env,7089 verbose(env,
7318 "R%d doesn't have constant offset. %s has to be at the constant offset\n",7090 "%s doesn't have constant offset. %s has to be at the constant offset\n",
7319 regno, struct_name);7091 reg_arg_name(env, argno), struct_name);
7320 return -EINVAL;7092 return -EINVAL;
7321 }7093 }
7322 if (!map->btf) {7094 if (!map->btf) {
@@ -7356,26 +7128,26 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, · 33 unmodified lines
7356 return 0;7128 return 0;
7357}7129}
7358 7130
7359static int process_timer_func(struct bpf_verifier_env *env, int regno,7131static int process_timer_func(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
7360 struct bpf_map_desc *map)7132 struct bpf_map_desc *map)
7361{7133{
7362 if (IS_ENABLED(CONFIG_PREEMPT_RT)) {7134 if (IS_ENABLED(CONFIG_PREEMPT_RT)) {
7363 verbose(env, "bpf_timer cannot be used for PREEMPT_RT.\n");7135 verbose(env, "bpf_timer cannot be used for PREEMPT_RT.\n");
7364 return -EOPNOTSUPP;7136 return -EOPNOTSUPP;
7365 }7137 }
7366 return check_map_field_pointer(env, regno, BPF_TIMER, map);7138 return check_map_field_pointer(env, reg, argno, BPF_TIMER, map);
7367}7139}
7368 7140
7369static int process_timer_helper(struct bpf_verifier_env *env, int regno,7141static int process_timer_helper(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
7370 struct bpf_call_arg_meta *meta)7142 struct bpf_call_arg_meta *meta)
7371{7143{
7372 return process_timer_func(env, regno, &meta->map);7144 return process_timer_func(env, reg, argno, &meta->map);
7373}7145}
7374 7146
7375static int process_timer_kfunc(struct bpf_verifier_env *env, int regno,7147static int process_timer_kfunc(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
7376 struct bpf_kfunc_call_arg_meta *meta)7148 struct bpf_kfunc_call_arg_meta *meta)
7377{7149{
7378 return process_timer_func(env, regno, &meta->map);7150 return process_timer_func(env, reg, argno, &meta->map);
7379}7151}
7380 7152
7381static int process_kptr_func(struct bpf_verifier_env *env, int regno,7153static int process_kptr_func(struct bpf_verifier_env *env, int regno,
@@ -7426,52 +7198,42 @@ static int process_kptr_func(struct bpf_verifier_env *env, int regno, · 44 unmodified lines
7426 return 0;7198 return 0;
7427}7199}
7428 7200
7429/* There are two register types representing a bpf_dynptr, one is PTR_TO_STACK7201/*
7202 * Validate dynptr arguments for helper, kfunc and subprog.
7203 *
7204 * @dynptr is both input and output. It is populated when the argument is
7205 * tagged with MEM_UNINIT (i.e., the dynptr argument that will be constructed)
7206 * and consumed when the argument is expecting to be an initialized dynptr.
7207 * @parent_id is used to track the referenced parent object (e.g., file or skb in
7208 * qdisc program) when constructing a dynptr.
7209 *
7210 * There are two register types representing a bpf_dynptr, one is PTR_TO_STACK
7430 * which points to a stack slot, and the other is CONST_PTR_TO_DYNPTR.7211 * which points to a stack slot, and the other is CONST_PTR_TO_DYNPTR.
7431 *7212 *
7432 * In both cases we deal with the first 8 bytes, but need to mark the next 87213 * In both cases we deal with the first 8 bytes, but need to mark the next 8
7433 * bytes as STACK_DYNPTR in case of PTR_TO_STACK. In case of7214 * bytes as STACK_DYNPTR in case of PTR_TO_STACK. In case of
7434 * CONST_PTR_TO_DYNPTR, we are guaranteed to get the beginning of the object.7215 * CONST_PTR_TO_DYNPTR, we are guaranteed to get the beginning of the object.
7435 *7216 *
7436 * Mutability of bpf_dynptr is at two levels, one is at the level of struct7217 * Mutability of bpf_dynptr is at two levels: the dynptr and the memory the
7437 * bpf_dynptr itself, i.e. whether the helper is receiving a pointer to struct7218 * dynptr points to. At the first level, the verifier will make sure a
7438 * bpf_dynptr or pointer to const struct bpf_dynptr. In the former case, it can7219 * CONST_PTR_TO_DYNPTR cannot be reinitialized or destroyed. The mutability of
7439 * mutate the view of the dynptr and also possibly destroy it. In the latter7220 * a dynptr's view (i.e., start and offset) is not tracked as there is not such
7440 * case, it cannot mutate the bpf_dynptr itself but it can still mutate the7221 * use case. The second level is tracked using the upper bit of bpf_dynptr->size
7441 * memory that dynptr points to.7222 * and checked dynamically during runtime.
7442 *
7443 * The verifier will keep track both levels of mutation (bpf_dynptr's in
7444 * reg->type and the memory's in reg->dynptr.type), but there is no support for
7445 * readonly dynptr view yet, hence only the first case is tracked and checked.
7446 *
7447 * This is consistent with how C applies the const modifier to a struct object,
7448 * where the pointer itself inside bpf_dynptr becomes const but not what it
7449 * points to.
7450 *
7451 * Helpers which do not mutate the bpf_dynptr set MEM_RDONLY in their argument
7452 * type, and declare it as 'const struct bpf_dynptr *' in their prototype.
7453 */7223 */
7454static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn_idx,7224static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
7455 enum bpf_arg_type arg_type, int clone_ref_obj_id)7225 argno_t argno, int insn_idx, enum bpf_arg_type arg_type,
7226 struct ref_obj_desc *ref_obj, struct bpf_dynptr_desc *dynptr)
7456{7227{
7457 struct bpf_reg_state *reg = reg_state(env, regno);7228 int spi, err = 0;
7458 int err;
7459 7229
7460 if (reg->type != PTR_TO_STACK && reg->type != CONST_PTR_TO_DYNPTR) {7230 if (reg->type != PTR_TO_STACK && reg->type != CONST_PTR_TO_DYNPTR) {
7461 verbose(env,7231 verbose(env,
7462 "arg#%d expected pointer to stack or const struct bpf_dynptr\n",7232 "%s expected pointer to stack or const struct bpf_dynptr\n",
7463 regno - 1);7233 reg_arg_name(env, argno));
7464 return -EINVAL;7234 return -EINVAL;
7465 }7235 }
7466 7236
7467 /* MEM_UNINIT and MEM_RDONLY are exclusive, when applied to an
7468 * ARG_PTR_TO_DYNPTR (or ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_*):
7469 */
7470 if ((arg_type & (MEM_UNINIT | MEM_RDONLY)) == (MEM_UNINIT | MEM_RDONLY)) {
7471 verifier_bug(env, "misconfigured dynptr helper type flags");
7472 return -EFAULT;
7473 }
7474
7475 /* MEM_UNINIT - Points to memory that is an appropriate candidate for7237 /* MEM_UNINIT - Points to memory that is an appropriate candidate for
7476 * constructing a mutable bpf_dynptr object.7238 * constructing a mutable bpf_dynptr object.
7477 *7239 *
@@ -7479,13 +7241,12 @@ static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn · 1 unmodified line
7479 * pointing to a region of at least 16 bytes which doesn't7241 * pointing to a region of at least 16 bytes which doesn't
7480 * contain an existing bpf_dynptr.7242 * contain an existing bpf_dynptr.
7481 *7243 *
7482 * MEM_RDONLY - Points to a initialized bpf_dynptr that will not be7244 * OBJ_RELEASE - Points to a initialized bpf_dynptr that will be
7483 * mutated or destroyed. However, the memory it points to7245 * destroyed.
7484 * may be mutated.
7485 *7246 *
7486 * None - Points to a initialized dynptr that can be mutated and7247 * None - Points to a initialized dynptr that cannot be
7487 * destroyed, including mutation of the memory it points7248 * reinitialized or destroyed. However, the view of the
7488 * to.7249 * dynptr and the memory it points to may be mutated.
7489 */7250 */
7490 if (arg_type & MEM_UNINIT) {7251 if (arg_type & MEM_UNINIT) {
7491 int i;7252 int i;
@@ -7497,45 +7258,58 @@ static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn · 5 unmodified lines
7497 7258
7498 /* we write BPF_DW bits (8 bytes) at a time */7259 /* we write BPF_DW bits (8 bytes) at a time */
7499 for (i = 0; i < BPF_DYNPTR_SIZE; i += 8) {7260 for (i = 0; i < BPF_DYNPTR_SIZE; i += 8) {
7500 err = check_mem_access(env, insn_idx, regno,7261 err = check_mem_access(env, insn_idx, reg, argno,
7501 i, BPF_DW, BPF_WRITE, -1, false, false);7262 i, BPF_DW, BPF_WRITE, -1, false, false);
7502 if (err)7263 if (err)
7503 return err;7264 return err;
7504 }7265 }
7505 7266
7506 err = mark_stack_slots_dynptr(env, reg, arg_type, insn_idx, clone_ref_obj_id);7267 err = mark_stack_slots_dynptr(env, reg, arg_type, insn_idx, ref_obj, dynptr);
7507 } else /* MEM_RDONLY and None case from above */ {7268 } else /* OBJ_RELEASE and None case from above */ {
7508 /* For the reg->type == PTR_TO_STACK case, bpf_dynptr is never const */7269 /* For the reg->type == PTR_TO_STACK case, bpf_dynptr is never const */
7509 if (reg->type == CONST_PTR_TO_DYNPTR && !(arg_type & MEM_RDONLY)) {7270 if (reg->type == CONST_PTR_TO_DYNPTR && (arg_type & OBJ_RELEASE)) {
7510 verbose(env, "cannot pass pointer to const bpf_dynptr, the helper mutates it\n");7271 verbose(env, "CONST_PTR_TO_DYNPTR cannot be released\n");
7511 return -EINVAL;7272 return -EINVAL;
7512 }7273 }
7513 7274
7514 if (!is_dynptr_reg_valid_init(env, reg)) {7275 if (!is_dynptr_reg_valid_init(env, reg)) {
7515 verbose(env,7276 verbose(env, "Expected an initialized dynptr as %s\n",
7516 "Expected an initialized dynptr as arg #%d\n",7277 reg_arg_name(env, argno));
7517 regno - 1);
7518 return -EINVAL;7278 return -EINVAL;
7519 }7279 }
7520 7280
7521 /* Fold modifiers (in this case, MEM_RDONLY) when checking expected type */7281 /* Fold modifiers (in this case, OBJ_RELEASE) when checking expected type */
7522 if (!is_dynptr_type_expected(env, reg, arg_type & ~MEM_RDONLY)) {7282 if (!is_dynptr_type_expected(env, reg, arg_type & ~OBJ_RELEASE)) {
7523 verbose(env,7283 verbose(env,
7524 "Expected a dynptr of type %s as arg #%d\n",7284 "Expected a dynptr of type %s as %s\n",
7525 dynptr_type_str(arg_to_dynptr_type(arg_type)), regno - 1);7285 dynptr_type_str(arg_to_dynptr_type(arg_type)),
7286 reg_arg_name(env, argno));
7526 return -EINVAL;7287 return -EINVAL;
7527 }7288 }
7528 7289
7529 err = mark_dynptr_read(env, reg);7290 if (reg->type != CONST_PTR_TO_DYNPTR) {
7530 }7291 struct bpf_func_state *state = bpf_func(env, reg);
7531 return err;
7532}
7533 7292
7534static u32 iter_ref_obj_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int spi)7293 spi = dynptr_get_spi(env, reg);
7535{7294 if (spi < 0)
7536 struct bpf_func_state *state = bpf_func(env, reg);7295 return spi;
7296
7297 /*
7298 * For CONST_PTR_TO_DYNPTR, reg is already scratched by check_reg_arg
7299 * in check_helper_call and mark_btf_func_reg_size in check_kfunc_call.
7300 */
7301 mark_stack_slots_scratched(env, spi, BPF_DYNPTR_NR_SLOTS);
7537 7302
7538 return state->stack[spi].spilled_ptr.ref_obj_id;7303 reg = &state->stack[spi].spilled_ptr;
7304 }
7305
7306 if (dynptr) {
7307 dynptr->type = reg->dynptr.type;
7308 dynptr->id = reg->id;
7309 dynptr->parent_id = reg->parent_id;
7310 }
7311 }
7312 return err;
7539}7313}
7540 7314
7541static bool is_iter_kfunc(struct bpf_kfunc_call_arg_meta *meta)7315static bool is_iter_kfunc(struct bpf_kfunc_call_arg_meta *meta)
@@ -7567,15 +7341,17 @@ static bool is_kfunc_arg_iter(struct bpf_kfunc_call_arg_meta *meta, int arg_idx, · 25 unmodified lines
7567 return btf_param_match_suffix(meta->btf, arg, "__iter");7341 return btf_param_match_suffix(meta->btf, arg, "__iter");
7568}7342}
7569 7343
7570static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_idx,7344static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int insn_idx,
7571 struct bpf_kfunc_call_arg_meta *meta)7345 struct bpf_kfunc_call_arg_meta *meta)
7572{7346{
7573 struct bpf_reg_state *reg = reg_state(env, regno);7347 struct bpf_func_state *state = bpf_func(env, reg);
7574 const struct btf_type *t;7348 const struct btf_type *t;
7349 u32 arg_idx = arg_idx_from_argno(argno);
7575 int spi, err, i, nr_slots, btf_id;7350 int spi, err, i, nr_slots, btf_id;
7576 7351
7577 if (reg->type != PTR_TO_STACK) {7352 if (reg->type != PTR_TO_STACK) {
7578 verbose(env, "arg#%d expected pointer to an iterator on stack\n", regno - 1);7353 verbose(env, "%s expected pointer to an iterator on stack\n",
7354 reg_arg_name(env, argno));
7579 return -EINVAL;7355 return -EINVAL;
7580 }7356 }
7581 7357
@@ -7585,9 +7361,10 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 3 unmodified lines
7585 * to any kfunc, if arg has "__iter" suffix, we need to be a bit more7361 * to any kfunc, if arg has "__iter" suffix, we need to be a bit more
7586 * conservative here.7362 * conservative here.
7587 */7363 */
7588 btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, regno - 1);7364 btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, arg_idx);
7589 if (btf_id < 0) {7365 if (btf_id < 0) {
7590 verbose(env, "expected valid iter pointer as arg #%d\n", regno - 1);7366 verbose(env, "expected valid iter pointer as %s\n",
7367 reg_arg_name(env, argno));
7591 return -EINVAL;7368 return -EINVAL;
7592 }7369 }
7593 t = btf_type_by_id(meta->btf, btf_id);7370 t = btf_type_by_id(meta->btf, btf_id);
@@ -7596,13 +7373,13 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 2 unmodified lines
7596 if (is_iter_new_kfunc(meta)) {7373 if (is_iter_new_kfunc(meta)) {
7597 /* bpf_iter_<type>_new() expects pointer to uninit iter state */7374 /* bpf_iter_<type>_new() expects pointer to uninit iter state */
7598 if (!is_iter_reg_valid_uninit(env, reg, nr_slots)) {7375 if (!is_iter_reg_valid_uninit(env, reg, nr_slots)) {
7599 verbose(env, "expected uninitialized iter_%s as arg #%d\n",7376 verbose(env, "expected uninitialized iter_%s as %s\n",
7600 iter_type_str(meta->btf, btf_id), regno - 1);7377 iter_type_str(meta->btf, btf_id), reg_arg_name(env, argno));
7601 return -EINVAL;7378 return -EINVAL;
7602 }7379 }
7603 7380
7604 for (i = 0; i < nr_slots * 8; i += BPF_REG_SIZE) {7381 for (i = 0; i < nr_slots * 8; i += BPF_REG_SIZE) {
7605 err = check_mem_access(env, insn_idx, regno,7382 err = check_mem_access(env, insn_idx, reg, argno,
7606 i, BPF_DW, BPF_WRITE, -1, false, false);7383 i, BPF_DW, BPF_WRITE, -1, false, false);
7607 if (err)7384 if (err)
7608 return err;7385 return err;
@@ -7620,8 +7397,8 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 11 unmodified lines
7620 case 0:7397 case 0:
7621 break;7398 break;
7622 case -EINVAL:7399 case -EINVAL:
7623 verbose(env, "expected an initialized iter_%s as arg #%d\n",7400 verbose(env, "expected an initialized iter_%s as %s\n",
7624 iter_type_str(meta->btf, btf_id), regno - 1);7401 iter_type_str(meta->btf, btf_id), reg_arg_name(env, argno));
7625 return err;7402 return err;
7626 case -EPROTO:7403 case -EPROTO:
7627 verbose(env, "expected an RCU CS when using %s\n", meta->func_name);7404 verbose(env, "expected an RCU CS when using %s\n", meta->func_name);
@@ -7634,14 +7411,12 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 6 unmodified lines
7634 if (spi < 0)7411 if (spi < 0)
7635 return spi;7412 return spi;
7636 7413
7637 err = mark_iter_read(env, reg, spi, nr_slots);7414 mark_stack_slots_scratched(env, spi, nr_slots);
7638 if (err)
7639 return err;
7640 7415
7641 /* remember meta->iter info for process_iter_next_call() */7416 /* remember meta->iter info for process_iter_next_call() */
7642 meta->iter.spi = spi;7417 meta->iter.spi = spi;
7643 meta->iter.frameno = reg->frameno;7418 meta->iter.frameno = reg->frameno;
7644 meta->ref_obj_id = iter_ref_obj_id(env, reg, spi);7419 update_ref_obj(&meta->ref_obj, &state->stack[spi].spilled_ptr);
7645 7420
7646 if (is_iter_destroy_kfunc(meta)) {7421 if (is_iter_destroy_kfunc(meta)) {
7647 err = unmark_stack_slots_iter(env, reg, nr_slots);7422 err = unmark_stack_slots_iter(env, reg, nr_slots);
@@ -8041,12 +7816,11 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { · 393 unmodified lines
8041 [ARG_PTR_TO_DYNPTR] = &dynptr_types,7816 [ARG_PTR_TO_DYNPTR] = &dynptr_types,
8042};7817};
8043 7818
8044static int check_reg_type(struct bpf_verifier_env *env, u32 regno,7819static int check_reg_type(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
8045 enum bpf_arg_type arg_type,7820 enum bpf_arg_type arg_type,
8046 const u32 *arg_btf_id,7821 const u32 *arg_btf_id,
8047 struct bpf_call_arg_meta *meta)7822 struct bpf_call_arg_meta *meta)
8048{7823{
8049 struct bpf_reg_state *reg = reg_state(env, regno);
8050 enum bpf_reg_type expected, type = reg->type;7824 enum bpf_reg_type expected, type = reg->type;
8051 const struct bpf_reg_types *compatible;7825 const struct bpf_reg_types *compatible;
8052 int i, j, err;7826 int i, j, err;
@@ -8077,7 +7851,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, · 24 unmodified lines
8077 type &= ~DYNPTR_TYPE_FLAG_MASK;7851 type &= ~DYNPTR_TYPE_FLAG_MASK;
8078 7852
8079 /* Local kptr types are allowed as the source argument of bpf_kptr_xchg */7853 /* Local kptr types are allowed as the source argument of bpf_kptr_xchg */
8080 if (meta->func_id == BPF_FUNC_kptr_xchg && type_is_alloc(type) && regno == BPF_REG_2) {7854 if (meta->func_id == BPF_FUNC_kptr_xchg && type_is_alloc(type) && reg_from_argno(argno) == BPF_REG_2) {
8081 type &= ~MEM_ALLOC;7855 type &= ~MEM_ALLOC;
8082 type &= ~MEM_PERCPU;7856 type &= ~MEM_PERCPU;
8083 }7857 }
@@ -8091,7 +7865,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, · 7 unmodified lines
8091 goto found;7865 goto found;
8092 }7866 }
8093 7867
8094 verbose(env, "R%d type=%s expected=", regno, reg_type_str(env, reg->type));7868 verbose(env, "%s type=%s expected=", reg_arg_name(env, argno), reg_type_str(env, reg->type));
8095 for (j = 0; j + 1 < i; j++)7869 for (j = 0; j + 1 < i; j++)
8096 verbose(env, "%s, ", reg_type_str(env, compatible->types[j]));7870 verbose(env, "%s, ", reg_type_str(env, compatible->types[j]));
8097 verbose(env, "%s\n", reg_type_str(env, compatible->types[j]));7871 verbose(env, "%s\n", reg_type_str(env, compatible->types[j]));
@@ -8104,9 +7878,9 @@ found: · 6 unmodified lines
8104 if (compatible == &mem_types) {7878 if (compatible == &mem_types) {
8105 if (!(arg_type & MEM_RDONLY)) {7879 if (!(arg_type & MEM_RDONLY)) {
8106 verbose(env,7880 verbose(env,
8107 "%s() may write into memory pointed by R%d type=%s\n",7881 "%s() may write into memory pointed by %s type=%s\n",
8108 func_id_name(meta->func_id),7882 func_id_name(meta->func_id),
8109 regno, reg_type_str(env, reg->type));7883 reg_arg_name(env, argno), reg_type_str(env, reg->type));
8110 return -EACCES;7884 return -EACCES;
8111 }7885 }
8112 return 0;7886 return 0;
@@ -8129,7 +7903,8 @@ found: · 16 unmodified lines
8129 7903
8130 if (type_may_be_null(reg->type) &&7904 if (type_may_be_null(reg->type) &&
8131 (!type_may_be_null(arg_type) || arg_type_is_release(arg_type))) {7905 (!type_may_be_null(arg_type) || arg_type_is_release(arg_type))) {
8132 verbose(env, "Possibly NULL pointer passed to helper arg%d\n", regno);7906 verbose(env, "Possibly NULL pointer passed to helper %s\n",
7907 reg_arg_name(env, argno));
8133 return -EACCES;7908 return -EACCES;
8134 }7909 }
8135 7910
@@ -8142,25 +7917,26 @@ found: · 6 unmodified lines
8142 }7917 }
8143 7918
8144 if (meta->func_id == BPF_FUNC_kptr_xchg) {7919 if (meta->func_id == BPF_FUNC_kptr_xchg) {
8145 if (map_kptr_match_type(env, meta->kptr_field, reg, regno))7920 if (map_kptr_match_type(env, meta->kptr_field, reg, reg_from_argno(argno)))
8146 return -EACCES;7921 return -EACCES;
8147 } else {7922 } else {
8148 if (arg_btf_id == BPF_PTR_POISON) {7923 if (arg_btf_id == BPF_PTR_POISON) {
8149 verbose(env, "verifier internal error:");7924 verbose(env, "verifier internal error:");
8150 verbose(env, "R%d has non-overwritten BPF_PTR_POISON type\n",7925 verbose(env, "%s has non-overwritten BPF_PTR_POISON type\n",
8151 regno);7926 reg_arg_name(env, argno));
8152 return -EACCES;7927 return -EACCES;
8153 }7928 }
8154 7929
8155 err = __check_ptr_off_reg(env, reg, regno, true);7930 err = __check_ptr_off_reg(env, reg, argno, true);
8156 if (err)7931 if (err)
8157 return err;7932 return err;
8158 7933
8159 if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id,7934 if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id,
8160 reg->var_off.value, btf_vmlinux, *arg_btf_id,7935 reg->var_off.value, btf_vmlinux, *arg_btf_id,
8161 strict_type_match)) {7936 strict_type_match)) {
8162 verbose(env, "R%d is of type %s but %s is expected\n",7937 verbose(env, "%s is of type %s but %s is expected\n",
8163 regno, btf_type_name(reg->btf, reg->btf_id),7938 reg_arg_name(env, argno),
7939 btf_type_name(reg->btf, reg->btf_id),
8164 btf_type_name(btf_vmlinux, *arg_btf_id));7940 btf_type_name(btf_vmlinux, *arg_btf_id));
8165 return -EACCES;7941 return -EACCES;
8166 }7942 }
@@ -8177,8 +7953,11 @@ found: · 10 unmodified lines
8177 return -EFAULT;7953 return -EFAULT;
8178 }7954 }
8179 /* Check if local kptr in src arg matches kptr in dst arg */7955 /* Check if local kptr in src arg matches kptr in dst arg */
8180 if (meta->func_id == BPF_FUNC_kptr_xchg && regno == BPF_REG_2) {7956 if (meta->func_id == BPF_FUNC_kptr_xchg) {
8181 if (map_kptr_match_type(env, meta->kptr_field, reg, regno))7957 int regno = reg_from_argno(argno);
7958
7959 if (regno == BPF_REG_2 &&
7960 map_kptr_match_type(env, meta->kptr_field, reg, regno))
8182 return -EACCES;7961 return -EACCES;
8183 }7962 }
8184 break;7963 break;
@@ -8212,7 +7991,7 @@ reg_find_field_offset(const struct bpf_reg_state *reg, s32 off, u32 fields) · 27 unmodified lines
8212}7991}
8213 7992
8214static int check_func_arg_reg_off(struct bpf_verifier_env *env,7993static int check_func_arg_reg_off(struct bpf_verifier_env *env,
8215 const struct bpf_reg_state *reg, int regno,7994 const struct bpf_reg_state *reg, argno_t argno,
8216 enum bpf_arg_type arg_type)7995 enum bpf_arg_type arg_type)
8217{7996{
8218 u32 type = reg->type;7997 u32 type = reg->type;
@@ -8220,7 +7999,7 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, · 1 unmodified line
8220 /* When referenced register is passed to release function, its fixed7999 /* When referenced register is passed to release function, its fixed
8221 * offset must be 0.8000 * offset must be 0.
8222 *8001 *
8223 * We will check arg_type_is_release reg has ref_obj_id when storing8002 * We will check arg_type_is_release reg has id when storing
8224 * meta->release_regno.8003 * meta->release_regno.
8225 */8004 */
8226 if (arg_type_is_release(arg_type)) {8005 if (arg_type_is_release(arg_type)) {
@@ -8238,8 +8017,8 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, · 11 unmodified lines
8238 * to give the user a better error message.8017 * to give the user a better error message.
8239 */8018 */
8240 if (!tnum_is_const(reg->var_off) || reg->var_off.value != 0) {8019 if (!tnum_is_const(reg->var_off) || reg->var_off.value != 0) {
8241 verbose(env, "R%d must have zero offset when passed to release func or trusted arg to kfunc\n",8020 verbose(env, "%s must have zero offset when passed to release func or trusted arg to kfunc\n",
8242 regno);8021 reg_arg_name(env, argno));
8243 return -EINVAL;8022 return -EINVAL;
8244 }8023 }
8245 }8024 }
@@ -8272,93 +8051,27 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, · 26 unmodified lines
8272 * its fixed offset must be 0. In the other cases, fixed offset8051 * its fixed offset must be 0. In the other cases, fixed offset
8273 * can be non-zero. This was already checked above. So pass8052 * can be non-zero. This was already checked above. So pass
8274 * fixed_off_ok as true to allow fixed offset for all other8053 * fixed_off_ok as true to allow fixed offset for all other
8275 * cases. var_off always must be 0 for PTR_TO_BTF_ID, hence we8054 * cases. var_off always must be 0 for PTR_TO_BTF_ID, hence we
8276 * still need to do checks instead of returning.8055 * still need to do checks instead of returning.
8277 */8056 */
8278 return __check_ptr_off_reg(env, reg, regno, true);8057 return __check_ptr_off_reg(env, reg, argno, true);
8279 case PTR_TO_CTX:8058 case PTR_TO_CTX:
8280 /*8059 /*
8281 * Allow fixed and variable offsets for syscall context, but8060 * Allow fixed and variable offsets for syscall context, but
8282 * only when the argument is passed as memory, not ctx,8061 * only when the argument is passed as memory, not ctx,
8283 * otherwise we may get modified ctx in tail called programs and8062 * otherwise we may get modified ctx in tail called programs and
8284 * global subprogs (that may act as extension prog hooks).8063 * global subprogs (that may act as extension prog hooks).
8285 */8064 */
8286 if (arg_type != ARG_PTR_TO_CTX && is_var_ctx_off_allowed(env->prog))8065 if (arg_type != ARG_PTR_TO_CTX && is_var_ctx_off_allowed(env->prog))
8287 return 0;8066 return 0;
8288 fallthrough;8067 fallthrough;
8289 default:8068 default:
8290 return __check_ptr_off_reg(env, reg, regno, false);8069 return __check_ptr_off_reg(env, reg, argno, false);
8291 }
8292}
8293
8294static struct bpf_reg_state *get_dynptr_arg_reg(struct bpf_verifier_env *env,
8295 const struct bpf_func_proto *fn,
8296 struct bpf_reg_state *regs)
8297{
8298 struct bpf_reg_state *state = NULL;
8299 int i;
8300
8301 for (i = 0; i < MAX_BPF_FUNC_REG_ARGS; i++)
8302 if (arg_type_is_dynptr(fn->arg_type[i])) {
8303 if (state) {
8304 verbose(env, "verifier internal error: multiple dynptr args\n");
8305 return NULL;
8306 }
8307 state = &regs[BPF_REG_1 + i];
8308 }
8309
8310 if (!state)
8311 verbose(env, "verifier internal error: no dynptr arg found\n");
8312
8313 return state;
8314}
8315
8316static int dynptr_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
8317{
8318 struct bpf_func_state *state = bpf_func(env, reg);
8319 int spi;
8320
8321 if (reg->type == CONST_PTR_TO_DYNPTR)
8322 return reg->id;
8323 spi = dynptr_get_spi(env, reg);
8324 if (spi < 0)
8325 return spi;
8326 return state->stack[spi].spilled_ptr.id;
8327}
8328
8329static int dynptr_ref_obj_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
8330{
8331 struct bpf_func_state *state = bpf_func(env, reg);
8332 int spi;
8333
8334 if (reg->type == CONST_PTR_TO_DYNPTR)
8335 return reg->ref_obj_id;
8336 spi = dynptr_get_spi(env, reg);
8337 if (spi < 0)
8338 return spi;
8339 return state->stack[spi].spilled_ptr.ref_obj_id;
8340}
8341
8342static enum bpf_dynptr_type dynptr_get_type(struct bpf_verifier_env *env,
8343 struct bpf_reg_state *reg)
8344{
8345 struct bpf_func_state *state = bpf_func(env, reg);
8346 int spi;
8347
8348 if (reg->type == CONST_PTR_TO_DYNPTR)
8349 return reg->dynptr.type;
8350
8351 spi = bpf_get_spi(reg->var_off.value);
8352 if (spi < 0) {
8353 verbose(env, "verifier internal error: invalid spi when querying dynptr type\n");
8354 return BPF_DYNPTR_TYPE_INVALID;
8355 }8070 }
8356
8357 return state->stack[spi].spilled_ptr.dynptr.type;
8358}8071}
8359 8072
8360static int check_reg_const_str(struct bpf_verifier_env *env,8073static int check_arg_const_str(struct bpf_verifier_env *env,
8361 struct bpf_reg_state *reg, u32 regno)8074 struct bpf_reg_state *reg, argno_t argno)
8362{8075{
8363 struct bpf_map *map = reg->map_ptr;8076 struct bpf_map *map = reg->map_ptr;
8364 int err;8077 int err;
@@ -8370,17 +8083,18 @@ static int check_reg_const_str(struct bpf_verifier_env *env, · 5 unmodified lines
8370 return -EINVAL;8083 return -EINVAL;
8371 8084
8372 if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) {8085 if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) {
8373 verbose(env, "R%d points to insn_array map which cannot be used as const string\n", regno);8086 verbose(env, "%s points to insn_array map which cannot be used as const string\n",
8087 reg_arg_name(env, argno));
8374 return -EACCES;8088 return -EACCES;
8375 }8089 }
8376 8090
8377 if (!bpf_map_is_rdonly(map)) {8091 if (!bpf_map_is_rdonly(map)) {
8378 verbose(env, "R%d does not point to a readonly map'\n", regno);8092 verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno));
8379 return -EACCES;8093 return -EACCES;
8380 }8094 }
8381 8095
8382 if (!tnum_is_const(reg->var_off)) {8096 if (!tnum_is_const(reg->var_off)) {
8383 verbose(env, "R%d is not a constant address'\n", regno);8097 verbose(env, "%s is not a constant address'\n", reg_arg_name(env, argno));
8384 return -EACCES;8098 return -EACCES;
8385 }8099 }
8386 8100
@@ -8389,7 +8103,7 @@ static int check_reg_const_str(struct bpf_verifier_env *env, · 2 unmodified lines
8389 return -EACCES;8103 return -EACCES;
8390 }8104 }
8391 8105
8392 err = check_map_access(env, regno, 0,8106 err = check_map_access(env, reg, argno, 0,
8393 map->value_size - reg->var_off.value, false,8107 map->value_size - reg->var_off.value, false,
8394 ACCESS_HELPER);8108 ACCESS_HELPER);
8395 if (err)8109 if (err)
@@ -8471,7 +8185,7 @@ static int get_constant_map_key(struct bpf_verifier_env *env, · 75 unmodified lines
8471 return 0;8185 return 0;
8472}8186}
8473 8187
8474static bool can_elide_value_nullness(enum bpf_map_type type);8188static bool can_elide_value_nullness(const struct bpf_map *map);
8475 8189
8476static int check_func_arg(struct bpf_verifier_env *env, u32 arg,8190static int check_func_arg(struct bpf_verifier_env *env, u32 arg,
8477 struct bpf_call_arg_meta *meta,8191 struct bpf_call_arg_meta *meta,
@@ -8481,6 +8195,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 3 unmodified lines
8481 u32 regno = BPF_REG_1 + arg;8195 u32 regno = BPF_REG_1 + arg;
8482 struct bpf_reg_state *reg = reg_state(env, regno);8196 struct bpf_reg_state *reg = reg_state(env, regno);
8483 enum bpf_arg_type arg_type = fn->arg_type[arg];8197 enum bpf_arg_type arg_type = fn->arg_type[arg];
8198 argno_t argno = argno_from_arg(arg + 1);
8484 enum bpf_reg_type type = reg->type;8199 enum bpf_reg_type type = reg->type;
8485 u32 *arg_btf_id = NULL;8200 u32 *arg_btf_id = NULL;
8486 u32 key_size;8201 u32 key_size;
@@ -8525,56 +8240,24 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 38 unmodified lines
8525 base_type(arg_type) == ARG_PTR_TO_SPIN_LOCK)8240 base_type(arg_type) == ARG_PTR_TO_SPIN_LOCK)
8526 arg_btf_id = fn->arg_btf_id[arg];8241 arg_btf_id = fn->arg_btf_id[arg];
8527 8242
8528 err = check_reg_type(env, regno, arg_type, arg_btf_id, meta);8243 err = check_reg_type(env, reg, argno_from_reg(regno), arg_type, arg_btf_id, meta);
8529 if (err)8244 if (err)
8530 return err;8245 return err;
8531 8246
8532 err = check_func_arg_reg_off(env, reg, regno, arg_type);8247 err = check_func_arg_reg_off(env, reg, argno_from_reg(regno), arg_type);
8533 if (err)8248 if (err)
8534 return err;8249 return err;
8535 8250
8536skip_type_check:8251skip_type_check:
8537 if (arg_type_is_release(arg_type)) {8252 if (arg_type_is_release(arg_type) && !arg_type_is_dynptr(arg_type) &&
8538 if (arg_type_is_dynptr(arg_type)) {8253 !reg_is_referenced(env, reg) && !bpf_register_is_null(reg)) {
8539 struct bpf_func_state *state = bpf_func(env, reg);8254 verbose(env, "release helper %s expects referenced PTR_TO_BTF_ID passed to %s\n",
8540 int spi;8255 func_id_name(meta->func_id), reg_arg_name(env, argno));
8541 8256 return -EINVAL;
8542 /* Only dynptr created on stack can be released, thus
8543 * the get_spi and stack state checks for spilled_ptr
8544 * should only be done before process_dynptr_func for
8545 * PTR_TO_STACK.
8546 */
8547 if (reg->type == PTR_TO_STACK) {
8548 spi = dynptr_get_spi(env, reg);
8549 if (spi < 0 || !state->stack[spi].spilled_ptr.ref_obj_id) {
8550 verbose(env, "arg %d is an unacquired reference\n", regno);
8551 return -EINVAL;
8552 }
8553 } else {
8554 verbose(env, "cannot release unowned const bpf_dynptr\n");
8555 return -EINVAL;
8556 }
8557 } else if (!reg->ref_obj_id && !bpf_register_is_null(reg)) {
8558 verbose(env, "R%d must be referenced when passed to release function\n",
8559 regno);
8560 return -EINVAL;
8561 }
8562 if (meta->release_regno) {
8563 verifier_bug(env, "more than one release argument");
8564 return -EFAULT;
8565 }
8566 meta->release_regno = regno;
8567 }8257 }
8568 8258
8569 if (reg->ref_obj_id && base_type(arg_type) != ARG_KPTR_XCHG_DEST) {8259 if (reg_is_referenced(env, reg))
8570 if (meta->ref_obj_id) {8260 update_ref_obj(&meta->ref_obj, reg);
8571 verbose(env, "more than one arg with ref_obj_id R%d %u %u",
8572 regno, reg->ref_obj_id,
8573 meta->ref_obj_id);
8574 return -EACCES;
8575 }
8576 meta->ref_obj_id = reg->ref_obj_id;
8577 }
8578 8261
8579 switch (base_type(arg_type)) {8262 switch (base_type(arg_type)) {
8580 case ARG_CONST_MAP_PTR:8263 case ARG_CONST_MAP_PTR:
@@ -8618,10 +8301,10 @@ skip_type_check: · 37 unmodified lines
8618 return -EFAULT;8301 return -EFAULT;
8619 }8302 }
8620 key_size = meta->map.ptr->key_size;8303 key_size = meta->map.ptr->key_size;
8621 err = check_helper_mem_access(env, regno, key_size, BPF_READ, false, NULL);8304 err = check_helper_mem_access(env, reg, argno_from_reg(regno), key_size, BPF_READ, false, NULL);
8622 if (err)8305 if (err)
8623 return err;8306 return err;
8624 if (can_elide_value_nullness(meta->map.ptr->map_type)) {8307 if (can_elide_value_nullness(meta->map.ptr)) {
8625 err = get_constant_map_key(env, reg, key_size, &meta->const_map_key);8308 err = get_constant_map_key(env, reg, key_size, &meta->const_map_key);
8626 if (err < 0) {8309 if (err < 0) {
8627 meta->const_map_key = -1;8310 meta->const_map_key = -1;
@@ -8645,7 +8328,7 @@ skip_type_check: · 17 unmodified lines
8645 return -EFAULT;8328 return -EFAULT;
8646 }8329 }
8647 meta->raw_mode = arg_type & MEM_UNINIT;8330 meta->raw_mode = arg_type & MEM_UNINIT;
8648 err = check_helper_mem_access(env, regno, meta->map.ptr->value_size,8331 err = check_helper_mem_access(env, reg, argno_from_reg(regno), meta->map.ptr->value_size,
8649 arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ,8332 arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ,
8650 false, meta);8333 false, meta);
8651 break;8334 break;
@@ -8663,11 +8346,11 @@ skip_type_check: · 11 unmodified lines
8663 return -EACCES;8346 return -EACCES;
8664 }8347 }
8665 if (meta->func_id == BPF_FUNC_spin_lock) {8348 if (meta->func_id == BPF_FUNC_spin_lock) {
8666 err = process_spin_lock(env, regno, PROCESS_SPIN_LOCK);8349 err = process_spin_lock(env, reg, argno_from_reg(regno), PROCESS_SPIN_LOCK);
8667 if (err)8350 if (err)
8668 return err;8351 return err;
8669 } else if (meta->func_id == BPF_FUNC_spin_unlock) {8352 } else if (meta->func_id == BPF_FUNC_spin_unlock) {
8670 err = process_spin_lock(env, regno, 0);8353 err = process_spin_lock(env, reg, argno_from_reg(regno), 0);
8671 if (err)8354 if (err)
8672 return err;8355 return err;
8673 } else {8356 } else {
@@ -8676,7 +8359,7 @@ skip_type_check: · 2 unmodified lines
8676 }8359 }
8677 break;8360 break;
8678 case ARG_PTR_TO_TIMER:8361 case ARG_PTR_TO_TIMER:
8679 err = process_timer_helper(env, regno, meta);8362 err = process_timer_helper(env, reg, argno_from_reg(regno), meta);
8680 if (err)8363 if (err)
8681 return err;8364 return err;
8682 break;8365 break;
@@ -8689,7 +8372,7 @@ skip_type_check: · 6 unmodified lines
8689 */8372 */
8690 meta->raw_mode = arg_type & MEM_UNINIT;8373 meta->raw_mode = arg_type & MEM_UNINIT;
8691 if (arg_type & MEM_FIXED_SIZE) {8374 if (arg_type & MEM_FIXED_SIZE) {
8692 err = check_helper_mem_access(env, regno, fn->arg_size[arg],8375 err = check_helper_mem_access(env, reg, argno_from_reg(regno), fn->arg_size[arg],
8693 arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ,8376 arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ,
8694 false, meta);8377 false, meta);
8695 if (err)8378 if (err)
@@ -8699,19 +8382,22 @@ skip_type_check: · 3 unmodified lines
8699 }8382 }
8700 break;8383 break;
8701 case ARG_CONST_SIZE:8384 case ARG_CONST_SIZE:
8702 err = check_mem_size_reg(env, reg, regno,8385 err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1),
8386 argno_from_reg(regno),
8703 fn->arg_type[arg - 1] & MEM_WRITE ?8387 fn->arg_type[arg - 1] & MEM_WRITE ?
8704 BPF_WRITE : BPF_READ,8388 BPF_WRITE : BPF_READ,
8705 false, meta);8389 false, meta);
8706 break;8390 break;
8707 case ARG_CONST_SIZE_OR_ZERO:8391 case ARG_CONST_SIZE_OR_ZERO:
8708 err = check_mem_size_reg(env, reg, regno,8392 err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1),
8393 argno_from_reg(regno),
8709 fn->arg_type[arg - 1] & MEM_WRITE ?8394 fn->arg_type[arg - 1] & MEM_WRITE ?
8710 BPF_WRITE : BPF_READ,8395 BPF_WRITE : BPF_READ,
8711 true, meta);8396 true, meta);
8712 break;8397 break;
8713 case ARG_PTR_TO_DYNPTR:8398 case ARG_PTR_TO_DYNPTR:
8714 err = process_dynptr_func(env, regno, insn_idx, arg_type, 0);8399 err = process_dynptr_func(env, reg, argno_from_reg(regno), insn_idx, arg_type, &meta->ref_obj,
8400 &meta->dynptr);
8715 if (err)8401 if (err)
8716 return err;8402 return err;
8717 break;8403 break;
@@ -8728,7 +8414,7 @@ skip_type_check: · 10 unmodified lines
8728 break;8414 break;
8729 case ARG_PTR_TO_CONST_STR:8415 case ARG_PTR_TO_CONST_STR:
8730 {8416 {
8731 err = check_reg_const_str(env, reg, regno);8417 err = check_arg_const_str(env, reg, argno_from_reg(regno));
8732 if (err)8418 if (err)
8733 return err;8419 return err;
8734 break;8420 break;
@@ -9130,11 +8816,29 @@ static bool check_mem_arg_rw_flag_ok(const struct bpf_func_proto *fn) · 395 unmodified lines
9130 return true;8816 return true;
9131}8817}
9132 8818
9133static int check_func_proto(const struct bpf_func_proto *fn)8819static bool check_proto_release_reg(const struct bpf_func_proto *fn, struct bpf_call_arg_meta *meta)
8820{
8821 int i;
8822
8823 for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++) {
8824 enum bpf_arg_type arg_type = fn->arg_type[i];
8825
8826 if (arg_type_is_release(arg_type)) {
8827 if (meta->release_regno)
8828 return false;
8829 meta->release_regno = i + 1;
8830 }
8831 }
8832
8833 return true;
8834}
8835
8836static int check_func_proto(const struct bpf_func_proto *fn, struct bpf_call_arg_meta *meta)
9134{8837{
9135 return check_raw_mode_ok(fn) &&8838 return check_raw_mode_ok(fn) &&
9136 check_arg_pair_ok(fn) &&8839 check_arg_pair_ok(fn) &&
9137 check_mem_arg_rw_flag_ok(fn) &&8840 check_mem_arg_rw_flag_ok(fn) &&
8841 check_proto_release_reg(fn, meta) &&
9138 check_btf_id_ok(fn) ? 0 : -EINVAL;8842 check_btf_id_ok(fn) ? 0 : -EINVAL;
9139}8843}
9140 8844
@@ -9181,14 +8885,14 @@ static void mark_pkt_end(struct bpf_verifier_state *vstate, int regn, bool range · 40 unmodified lines
9181 reg->range = AT_PKT_END;8885 reg->range = AT_PKT_END;
9182}8886}
9183 8887
9184static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id)8888static int release_reference_nomark(struct bpf_verifier_state *state, int id)
9185{8889{
9186 int i;8890 int i;
9187 8891
9188 for (i = 0; i < state->acquired_refs; i++) {8892 for (i = 0; i < state->acquired_refs; i++) {
9189 if (state->refs[i].type != REF_TYPE_PTR)8893 if (state->refs[i].type != REF_TYPE_PTR)
9190 continue;8894 continue;
9191 if (state->refs[i].id == ref_obj_id) {8895 if (state->refs[i].id == id) {
9192 release_reference_state(state, i);8896 release_reference_state(state, i);
9193 return 0;8897 return 0;
9194 }8898 }
@@ -9196,26 +8900,83 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob · 1 unmodified line
9196 return -EINVAL;8900 return -EINVAL;
9197}8901}
9198 8902
9199/* The pointer with the specified id has released its reference to kernel8903static int idstack_push(struct bpf_idmap *idmap, u32 id)
9200 * resources. Identify all copies of the same pointer and clear the reference.8904{
9201 *8905 int i;
9202 * This is the release function corresponding to acquire_reference(). Idempotent.8906
9203 */8907 if (!id)
9204static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)8908 return 0;
8909
8910 for (i = 0; i < idmap->cnt; i++)
8911 if (idmap->map[i].old == id)
8912 return 0;
8913
8914 if (WARN_ON_ONCE(idmap->cnt >= BPF_ID_MAP_SIZE))
8915 return -EFAULT;
8916
8917 idmap->map[idmap->cnt++].old = id;
8918 return 0;
8919}
8920
8921static int idstack_pop(struct bpf_idmap *idmap)
8922{
8923 if (!idmap->cnt)
8924 return 0;
8925
8926 return idmap->map[--idmap->cnt].old;
8927}
8928
8929/* Release id and objects derived from it iteratively in a DFS manner */
8930static int release_reference(struct bpf_verifier_env *env, int id)
9205{8931{
8932 u32 mask = (1 << STACK_SPILL) | (1 << STACK_DYNPTR);
9206 struct bpf_verifier_state *vstate = env->cur_state;8933 struct bpf_verifier_state *vstate = env->cur_state;
8934 struct bpf_idmap *idstack = &env->idmap_scratch;
8935 struct bpf_stack_state *stack;
9207 struct bpf_func_state *state;8936 struct bpf_func_state *state;
9208 struct bpf_reg_state *reg;8937 struct bpf_reg_state *reg;
9209 int err;8938 int i, err;
9210 8939
9211 err = release_reference_nomark(vstate, ref_obj_id);8940 idstack->cnt = 0;
8941 err = idstack_push(idstack, id);
9212 if (err)8942 if (err)
9213 return err;8943 return err;
9214 8944
9215 bpf_for_each_reg_in_vstate(vstate, state, reg, ({8945 if (find_reference_state(vstate, id))
9216 if (reg->ref_obj_id == ref_obj_id)8946 WARN_ON_ONCE(release_reference_nomark(vstate, id));
9217 mark_reg_invalid(env, reg);8947
9218 }));8948 while ((id = idstack_pop(idstack))) {
8949 /*
8950 * Child references are inaccessible after parent is released,
8951 * any child references that exist at this point are a leak.
8952 */
8953 for (i = 0; i < vstate->acquired_refs; i++) {
8954 if (vstate->refs[i].type != REF_TYPE_PTR)
8955 continue;
8956 if (vstate->refs[i].parent_id != id)
8957 continue;
8958 verbose(env, "Leaking reference id=%d alloc_insn=%d. Release it first.\n",
8959 vstate->refs[i].id, vstate->refs[i].insn_idx);
8960 return -EINVAL;
8961 }
8962
8963 bpf_for_each_reg_in_vstate_mask(vstate, state, reg, stack, mask, ({
8964 if (reg->id != id && reg->parent_id != id)
8965 continue;
8966
8967 /* Free objects derived from the current object */
8968 if (reg->parent_id == id) {
8969 err = idstack_push(idstack, reg->id);
8970 if (err)
8971 return err;
8972 }
8973
8974 if (!stack || stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL)
8975 mark_reg_invalid(env, reg);
8976 else if (stack->slot_type[BPF_REG_SIZE - 1] == STACK_DYNPTR)
8977 invalidate_dynptr(env, stack);
8978 }));
8979 }
9219 8980
9220 return 0;8981 return 0;
9221}8982}
@@ -9231,6 +8992,42 @@ static void invalidate_non_owning_refs(struct bpf_verifier_env *env) · 9 unmodified lines
9231 }));8992 }));
9232}8993}
9233 8994
8995static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env)
8996{
8997 struct bpf_stack_state *stack;
8998 struct bpf_func_state *state;
8999 struct bpf_reg_state *reg;
9000 u32 clear_mask = (1 << STACK_SPILL) | (1 << STACK_ITER);
9001
9002 bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, clear_mask, ({
9003 if (reg->type & MEM_RCU) {
9004 reg->type &= ~(MEM_RCU | PTR_MAYBE_NULL);
9005 reg->type |= PTR_UNTRUSTED;
9006 }
9007 }));
9008}
9009
9010static int ref_convert_alloc_rcu_protected(struct bpf_verifier_env *env, u32 id)
9011{
9012 struct bpf_func_state *state;
9013 struct bpf_reg_state *reg;
9014 int err;
9015
9016 err = release_reference_nomark(env->cur_state, id);
9017
9018 bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({
9019 if (reg->id != id)
9020 continue;
9021 if ((reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU)) {
9022 reg->id = 0;
9023 reg->type &= ~MEM_ALLOC;
9024 reg->type |= MEM_RCU;
9025 }
9026 }));
9027
9028 return err;
9029}
9030
9234static void clear_caller_saved_regs(struct bpf_verifier_env *env,9031static void clear_caller_saved_regs(struct bpf_verifier_env *env,
9235 struct bpf_reg_state *regs)9032 struct bpf_reg_state *regs)
9236{9033{
@@ -9243,6 +9040,15 @@ static void clear_caller_saved_regs(struct bpf_verifier_env *env, · 6 unmodified lines
9243 }9040 }
9244}9041}
9245 9042
9043static void invalidate_outgoing_stack_args(const struct bpf_verifier_env *env,
9044 struct bpf_func_state *state)
9045{
9046 int i, nslots = state->out_stack_arg_cnt;
9047
9048 for (i = 0; i < nslots; i++)
9049 bpf_mark_reg_not_init(env, &state->stack_arg_regs[i]);
9050}
9051
9246typedef int (*set_callee_state_fn)(struct bpf_verifier_env *env,9052typedef int (*set_callee_state_fn)(struct bpf_verifier_env *env,
9247 struct bpf_func_state *caller,9053 struct bpf_func_state *caller,
9248 struct bpf_func_state *callee,9054 struct bpf_func_state *callee,
@@ -9305,11 +9111,23 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 56 unmodified lines
9305 struct bpf_reg_state *regs)9111 struct bpf_reg_state *regs)
9306{9112{
9307 struct bpf_subprog_info *sub = subprog_info(env, subprog);9113 struct bpf_subprog_info *sub = subprog_info(env, subprog);
9114 struct bpf_func_state *caller = cur_func(env);
9308 struct bpf_verifier_log *log = &env->log;9115 struct bpf_verifier_log *log = &env->log;
9116 struct ref_obj_desc ref_obj = {};
9309 u32 i;9117 u32 i;
9310 int ret;9118 int ret, err;
9311 9119
9312 ret = btf_prepare_func_args(env, subprog);9120 ret = btf_prepare_func_args(env, subprog);
9121 if (ret) {
9122 if (bpf_in_stack_arg_cnt(sub) > 0) {
9123 err = check_outgoing_stack_args(env, caller, sub->arg_cnt);
9124 if (err)
9125 return err;
9126 }
9127 return ret;
9128 }
9129
9130 ret = check_outgoing_stack_args(env, caller, sub->arg_cnt);
9313 if (ret)9131 if (ret)
9314 return ret;9132 return ret;
9315 9133
@@ -9317,13 +9135,13 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 1 unmodified line
9317 * verifier sees.9135 * verifier sees.
9318 */9136 */
9319 for (i = 0; i < sub->arg_cnt; i++) {9137 for (i = 0; i < sub->arg_cnt; i++) {
9320 u32 regno = i + 1;9138 argno_t argno = argno_from_arg(i + 1);
9321 struct bpf_reg_state *reg = &regs[regno];9139 struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, i);
9322 struct bpf_subprog_arg_info *arg = &sub->args[i];9140 struct bpf_subprog_arg_info *arg = &sub->args[i];
9323 9141
9324 if (arg->arg_type == ARG_ANYTHING) {9142 if (arg->arg_type == ARG_ANYTHING) {
9325 if (reg->type != SCALAR_VALUE) {9143 if (reg->type != SCALAR_VALUE) {
9326 bpf_log(log, "R%d is not a scalar\n", regno);9144 bpf_log(log, "%s is not a scalar\n", reg_arg_name(env, argno));
9327 return -EINVAL;9145 return -EINVAL;
9328 }9146 }
9329 } else if (arg->arg_type & PTR_UNTRUSTED) {9147 } else if (arg->arg_type & PTR_UNTRUSTED) {
@@ -9333,24 +9151,26 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 3 unmodified lines
9333 * invalid memory access.9151 * invalid memory access.
9334 */9152 */
9335 } else if (arg->arg_type == ARG_PTR_TO_CTX) {9153 } else if (arg->arg_type == ARG_PTR_TO_CTX) {
9336 ret = check_func_arg_reg_off(env, reg, regno, ARG_PTR_TO_CTX);9154 ret = check_func_arg_reg_off(env, reg, argno, ARG_PTR_TO_CTX);
9337 if (ret < 0)9155 if (ret < 0)
9338 return ret;9156 return ret;
9339 /* If function expects ctx type in BTF check that caller9157 /* If function expects ctx type in BTF check that caller
9340 * is passing PTR_TO_CTX.9158 * is passing PTR_TO_CTX.
9341 */9159 */
9342 if (reg->type != PTR_TO_CTX) {9160 if (reg->type != PTR_TO_CTX) {
9343 bpf_log(log, "arg#%d expects pointer to ctx\n", i);9161 bpf_log(log, "%s expects pointer to ctx\n",
9162 reg_arg_name(env, argno));
9344 return -EINVAL;9163 return -EINVAL;
9345 }9164 }
9346 } else if (base_type(arg->arg_type) == ARG_PTR_TO_MEM) {9165 } else if (base_type(arg->arg_type) == ARG_PTR_TO_MEM) {
9347 ret = check_func_arg_reg_off(env, reg, regno, ARG_DONTCARE);9166 ret = check_func_arg_reg_off(env, reg, argno, ARG_DONTCARE);
9348 if (ret < 0)9167 if (ret < 0)
9349 return ret;9168 return ret;
9350 if (check_mem_reg(env, reg, regno, arg->mem_size))9169 if (check_mem_reg(env, reg, argno, arg->mem_size))
9351 return -EINVAL;9170 return -EINVAL;
9352 if (!(arg->arg_type & PTR_MAYBE_NULL) && (reg->type & PTR_MAYBE_NULL)) {9171 if (!(arg->arg_type & PTR_MAYBE_NULL) && (reg->type & PTR_MAYBE_NULL)) {
9353 bpf_log(log, "arg#%d is expected to be non-NULL\n", i);9172 bpf_log(log, "%s is expected to be non-NULL\n",
9173 reg_arg_name(env, argno));
9354 return -EINVAL;9174 return -EINVAL;
9355 }9175 }
9356 } else if (base_type(arg->arg_type) == ARG_PTR_TO_ARENA) {9176 } else if (base_type(arg->arg_type) == ARG_PTR_TO_ARENA) {
@@ -9362,15 +9182,16 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 5 unmodified lines
9362 * run-time debug nightmare.9182 * run-time debug nightmare.
9363 */9183 */
9364 if (reg->type != PTR_TO_ARENA && reg->type != SCALAR_VALUE) {9184 if (reg->type != PTR_TO_ARENA && reg->type != SCALAR_VALUE) {
9365 bpf_log(log, "R%d is not a pointer to arena or scalar.\n", regno);9185 bpf_log(log, "%s is not a pointer to arena or scalar.\n",
9186 reg_arg_name(env, argno));
9366 return -EINVAL;9187 return -EINVAL;
9367 }9188 }
9368 } else if (arg->arg_type == (ARG_PTR_TO_DYNPTR | MEM_RDONLY)) {9189 } else if (arg->arg_type == ARG_PTR_TO_DYNPTR) {
9369 ret = check_func_arg_reg_off(env, reg, regno, ARG_PTR_TO_DYNPTR);9190 ret = check_func_arg_reg_off(env, reg, argno, ARG_PTR_TO_DYNPTR);
9370 if (ret)9191 if (ret)
9371 return ret;9192 return ret;
9372 9193
9373 ret = process_dynptr_func(env, regno, -1, arg->arg_type, 0);9194 ret = process_dynptr_func(env, reg, argno, -1, arg->arg_type, &ref_obj, NULL);
9374 if (ret)9195 if (ret)
9375 return ret;9196 return ret;
9376 } else if (base_type(arg->arg_type) == ARG_PTR_TO_BTF_ID) {9197 } else if (base_type(arg->arg_type) == ARG_PTR_TO_BTF_ID) {
@@ -9381,12 +9202,13 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 4 unmodified lines
9381 continue;9202 continue;
9382 9203
9383 memset(&meta, 0, sizeof(meta)); /* leave func_id as zero */9204 memset(&meta, 0, sizeof(meta)); /* leave func_id as zero */
9384 err = check_reg_type(env, regno, arg->arg_type, &arg->btf_id, &meta);9205 err = check_reg_type(env, reg, argno, arg->arg_type, &arg->btf_id, &meta);
9385 err = err ?: check_func_arg_reg_off(env, reg, regno, arg->arg_type);9206 err = err ?: check_func_arg_reg_off(env, reg, argno, arg->arg_type);
9386 if (err)9207 if (err)
9387 return err;9208 return err;
9388 } else {9209 } else {
9389 verifier_bug(env, "unrecognized arg#%d type %d", i, arg->arg_type);9210 verifier_bug(env, "unrecognized %s type %d",
9211 reg_arg_name(env, argno), arg->arg_type);
9390 return -EFAULT;9212 return -EFAULT;
9391 }9213 }
9392 }9214 }
@@ -9505,6 +9327,8 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · 112 unmodified lines
9505 int *insn_idx)9327 int *insn_idx)
9506{9328{
9507 struct bpf_verifier_state *state = env->cur_state;9329 struct bpf_verifier_state *state = env->cur_state;
9330 struct bpf_subprog_info *caller_info;
9331 u16 callee_incoming, stack_arg_cnt;
9508 struct bpf_func_state *caller;9332 struct bpf_func_state *caller;
9509 int err, subprog, target_insn;9333 int err, subprog, target_insn;
9510 9334
@@ -9547,6 +9371,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · 36 unmodified lines
9547 /* mark global subprog for verifying after main prog */9371 /* mark global subprog for verifying after main prog */
9548 subprog_aux(env, subprog)->called = true;9372 subprog_aux(env, subprog)->called = true;
9549 clear_caller_saved_regs(env, caller->regs);9373 clear_caller_saved_regs(env, caller->regs);
9374 invalidate_outgoing_stack_args(env, cur_func(env));
9550 9375
9551 /* All non-void global functions return a 64-bit SCALAR_VALUE. */9376 /* All non-void global functions return a 64-bit SCALAR_VALUE. */
9552 if (!subprog_returns_void(env, subprog)) {9377 if (!subprog_returns_void(env, subprog)) {
@@ -9569,6 +9394,16 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · 16 unmodified lines
9569 return 0;9394 return 0;
9570 }9395 }
9571 9396
9397 /*
9398 * Track caller's total stack arg count (incoming + max outgoing).
9399 * This is needed so the JIT knows how much stack arg space to allocate.
9400 */
9401 caller_info = &env->subprog_info[caller->subprogno];
9402 callee_incoming = bpf_in_stack_arg_cnt(&env->subprog_info[subprog]);
9403 stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) + callee_incoming;
9404 if (stack_arg_cnt > caller_info->stack_arg_cnt)
9405 caller_info->stack_arg_cnt = stack_arg_cnt;
9406
9572 /* for regular function entry setup new frame and continue9407 /* for regular function entry setup new frame and continue
9573 * from that frame.9408 * from that frame.
9574 */9409 */
@@ -9852,9 +9687,9 @@ static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env) · 277 unmodified lines
9852static bool retval_range_within(struct bpf_retval_range range, const struct bpf_reg_state *reg)9687static bool retval_range_within(struct bpf_retval_range range, const struct bpf_reg_state *reg)
9853{9688{
9854 if (range.return_32bit)9689 if (range.return_32bit)
9855 return range.minval <= reg->s32_min_value && reg->s32_max_value <= range.maxval;9690 return range.minval <= reg_s32_min(reg) && reg_s32_max(reg) <= range.maxval;
9856 else9691 else
9857 return range.minval <= reg->smin_value && reg->smax_value <= range.maxval;9692 return range.minval <= reg_smin(reg) && reg_smax(reg) <= range.maxval;
9858}9693}
9859 9694
9860static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)9695static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
@@ -9926,6 +9761,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · 65 unmodified lines
9926 * bpf_throw, this will be done by copy_verifier_state for extra frames. */9761 * bpf_throw, this will be done by copy_verifier_state for extra frames. */
9927 free_func_state(callee);9762 free_func_state(callee);
9928 state->frame[state->curframe--] = NULL;9763 state->frame[state->curframe--] = NULL;
9764 invalidate_outgoing_stack_args(env, caller);
9929 9765
9930 /* for callbacks widen imprecise scalars to make programs like below verify:9766 /* for callbacks widen imprecise scalars to make programs like below verify:
9931 *9767 *
@@ -9952,7 +9788,9 @@ static int do_refine_retval_range(struct bpf_verifier_env *env, · 20 unmodified lines
9952 int func_id,9788 int func_id,
9953 struct bpf_call_arg_meta *meta)9789 struct bpf_call_arg_meta *meta)
9954{9790{
9791 struct bpf_retval_range range;
9955 struct bpf_reg_state *ret_reg = &regs[BPF_REG_0];9792 struct bpf_reg_state *ret_reg = &regs[BPF_REG_0];
9793 enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
9956 9794
9957 if (ret_type != RET_INTEGER)9795 if (ret_type != RET_INTEGER)
9958 return 0;9796 return 0;
@@ -9963,21 +9801,36 @@ static int do_refine_retval_range(struct bpf_verifier_env *env, · 4 unmodified lines
9963 case BPF_FUNC_probe_read_str:9801 case BPF_FUNC_probe_read_str:
9964 case BPF_FUNC_probe_read_kernel_str:9802 case BPF_FUNC_probe_read_kernel_str:
9965 case BPF_FUNC_probe_read_user_str:9803 case BPF_FUNC_probe_read_user_str:
9966 ret_reg->smax_value = meta->msize_max_value;9804 reg_set_srange64(ret_reg, -MAX_ERRNO, meta->msize_max_value);
9967 ret_reg->s32_max_value = meta->msize_max_value;9805 reg_set_srange32(ret_reg, -MAX_ERRNO, meta->msize_max_value);
9968 ret_reg->smin_value = -MAX_ERRNO;
9969 ret_reg->s32_min_value = -MAX_ERRNO;
9970 reg_bounds_sync(ret_reg);9806 reg_bounds_sync(ret_reg);
9971 break;9807 break;
9972 case BPF_FUNC_get_smp_processor_id:9808 case BPF_FUNC_get_smp_processor_id:
9973 ret_reg->umax_value = nr_cpu_ids - 1;9809 reg_set_urange64(ret_reg, 0, nr_cpu_ids - 1);
9974 ret_reg->u32_max_value = nr_cpu_ids - 1;9810 reg_set_urange32(ret_reg, 0, nr_cpu_ids - 1);
9975 ret_reg->smax_value = nr_cpu_ids - 1;9811 reg_bounds_sync(ret_reg);
9976 ret_reg->s32_max_value = nr_cpu_ids - 1;9812 break;
9977 ret_reg->umin_value = 0;9813 case BPF_FUNC_get_retval:
9978 ret_reg->u32_min_value = 0;9814 /*
9979 ret_reg->smin_value = 0;9815 * bpf_get_retval may see arbitrary value passed by bpf_prog_run_array_cg for
9980 ret_reg->s32_min_value = 0;9816 * CGROUP_GETSOCKOPT type.
9817 */
9818 if (prog_type == BPF_PROG_TYPE_CGROUP_SOCKOPT &&
9819 env->prog->expected_attach_type == BPF_CGROUP_GETSOCKOPT)
9820 break;
9821
9822 if (prog_type == BPF_PROG_TYPE_LSM &&
9823 env->prog->expected_attach_type == BPF_LSM_CGROUP) {
9824 if (!env->prog->aux->attach_func_proto->type)
9825 break;
9826 bpf_lsm_get_retval_range(env->prog, &range);
9827 } else {
9828 range.minval = -MAX_ERRNO;
9829 range.maxval = 0;
9830 }
9831
9832 reg_set_srange64(ret_reg, range.minval, range.maxval);
9833 reg_set_srange32(ret_reg, range.minval, range.maxval);
9981 reg_bounds_sync(ret_reg);9834 reg_bounds_sync(ret_reg);
9982 break;9835 break;
9983 }9836 }
@@ -10086,7 +9939,7 @@ static int check_reference_leak(struct bpf_verifier_env *env, bool exception_exi · 102 unmodified lines
10086 * kernel. Type checks are performed later in check_return_code.9939 * kernel. Type checks are performed later in check_return_code.
10087 */9940 */
10088 if (type == BPF_PROG_TYPE_STRUCT_OPS && !exception_exit &&9941 if (type == BPF_PROG_TYPE_STRUCT_OPS && !exception_exit &&
10089 reg->ref_obj_id == state->refs[i].id)9942 reg->id == state->refs[i].id)
10090 continue;9943 continue;
10091 verbose(env, "Unreleased reference id=%d alloc_insn=%d\n",9944 verbose(env, "Unreleased reference id=%d alloc_insn=%d\n",
10092 state->refs[i].id, state->refs[i].insn_idx);9945 state->refs[i].id, state->refs[i].insn_idx);
@@ -10221,13 +10074,16 @@ static void update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno · 128 unmodified lines
10221 state->callback_subprogno == subprogno);10074 state->callback_subprogno == subprogno);
10222}10075}
10223 10076
10224/* Returns whether or not the given map type can potentially elide10077/* Returns whether or not the given map can potentially elide
10225 * lookup return value nullness check. This is possible if the key10078 * lookup return value nullness check. This is possible if the key
10226 * is statically known.10079 * is statically known.
10227 */10080 */
10228static bool can_elide_value_nullness(enum bpf_map_type type)10081static bool can_elide_value_nullness(const struct bpf_map *map)
10229{10082{
10230 switch (type) {10083 if (map->map_flags & BPF_F_INNER_MAP)
10084 return false;
10085
10086 switch (map->map_type) {
10231 case BPF_MAP_TYPE_ARRAY:10087 case BPF_MAP_TYPE_ARRAY:
10232 case BPF_MAP_TYPE_PERCPU_ARRAY:10088 case BPF_MAP_TYPE_PERCPU_ARRAY:
10233 return true;10089 return true;
@@ -10272,6 +10128,24 @@ static const char *non_sleepable_context_description(struct bpf_verifier_env *en · 38 unmodified lines
10272 return "non-sleepable prog";10128 return "non-sleepable prog";
10273}10129}
10274 10130
10131static int release_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
10132 bool convert_rcu, bool release_dynptr)
10133{
10134 int err = -EINVAL;
10135
10136 if (bpf_register_is_null(reg))
10137 return 0;
10138
10139 if (release_dynptr)
10140 err = unmark_stack_slots_dynptr(env, reg);
10141 else if (convert_rcu)
10142 err = ref_convert_alloc_rcu_protected(env, reg->id);
10143 else if (reg_is_referenced(env, reg))
10144 err = release_reference(env, reg->id);
10145
10146 return err;
10147}
10148
10275static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn,10149static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
10276 int *insn_idx_p)10150 int *insn_idx_p)
10277{10151{
@@ -10321,7 +10195,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 43 unmodified lines
10321 memset(&meta, 0, sizeof(meta));10195 memset(&meta, 0, sizeof(meta));
10322 meta.pkt_access = fn->pkt_access;10196 meta.pkt_access = fn->pkt_access;
10323 10197
10324 err = check_func_proto(fn);10198 err = check_func_proto(fn, &meta);
10325 if (err) {10199 if (err) {
10326 verifier_bug(env, "incorrect func proto %s#%d", func_id_name(func_id), func_id);10200 verifier_bug(env, "incorrect func proto %s#%d", func_id_name(func_id), func_id);
10327 return err;10201 return err;
@@ -10353,55 +10227,26 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 25 unmodified lines
10353 if (err)10227 if (err)
10354 return err;10228 return err;
10355 10229
10230 regs = cur_regs(env);
10231
10356 /* Mark slots with STACK_MISC in case of raw mode, stack offset10232 /* Mark slots with STACK_MISC in case of raw mode, stack offset
10357 * is inferred from register state.10233 * is inferred from register state.
10358 */10234 */
10359 for (i = 0; i < meta.access_size; i++) {10235 for (i = 0; i < meta.access_size; i++) {
10360 err = check_mem_access(env, insn_idx, meta.regno, i, BPF_B,10236 err = check_mem_access(env, insn_idx, regs + meta.regno, argno_from_reg(meta.regno), i, BPF_B,
10361 BPF_WRITE, -1, false, false);10237 BPF_WRITE, -1, false, false);
10362 if (err)10238 if (err)
10363 return err;10239 return err;
10364 }10240 }
10365 10241
10366 regs = cur_regs(env);
10367
10368 if (meta.release_regno) {10242 if (meta.release_regno) {
10369 err = -EINVAL;10243 struct bpf_reg_state *reg = &regs[meta.release_regno];
10370 if (arg_type_is_dynptr(fn->arg_type[meta.release_regno - BPF_REG_1])) {10244 bool convert_rcu = (func_id == BPF_FUNC_kptr_xchg) && in_rcu_cs(env) &&
10371 err = unmark_stack_slots_dynptr(env, &regs[meta.release_regno]);10245 (reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU);
10372 } else if (func_id == BPF_FUNC_kptr_xchg && meta.ref_obj_id) {10246
10373 u32 ref_obj_id = meta.ref_obj_id;10247 err = release_reg(env, reg, convert_rcu, !!meta.dynptr.id);
10374 bool in_rcu = in_rcu_cs(env);10248 if (err)
10375 struct bpf_func_state *state;
10376 struct bpf_reg_state *reg;
10377
10378 err = release_reference_nomark(env->cur_state, ref_obj_id);
10379 if (!err) {
10380 bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({
10381 if (reg->ref_obj_id == ref_obj_id) {
10382 if (in_rcu && (reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU)) {
10383 reg->ref_obj_id = 0;
10384 reg->type &= ~MEM_ALLOC;
10385 reg->type |= MEM_RCU;
10386 } else {
10387 mark_reg_invalid(env, reg);
10388 }
10389 }
10390 }));
10391 }
10392 } else if (meta.ref_obj_id) {
10393 err = release_reference(env, meta.ref_obj_id);
10394 } else if (bpf_register_is_null(&regs[meta.release_regno])) {
10395 /* meta.ref_obj_id can only be 0 if register that is meant to be
10396 * released is NULL, which must be > R0.
10397 */
10398 err = 0;
10399 }
10400 if (err) {
10401 verbose(env, "func %s#%d reference has not been acquired before\n",
10402 func_id_name(func_id), func_id);
10403 return err;10249 return err;
10404 }
10405 }10250 }
10406 10251
10407 switch (func_id) {10252 switch (func_id) {
@@ -10442,7 +10287,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 34 unmodified lines
10442 err = mark_chain_precision(env, BPF_REG_1);10287 err = mark_chain_precision(env, BPF_REG_1);
10443 if (err)10288 if (err)
10444 return err;10289 return err;
10445 if (cur_func(env)->callback_depth < regs[BPF_REG_1].umax_value) {10290 if (cur_func(env)->callback_depth < reg_umax(&regs[BPF_REG_1])) {
10446 err = push_callback_call(env, insn, insn_idx, meta.subprogno,10291 err = push_callback_call(env, insn, insn_idx, meta.subprogno,
10447 set_loop_callback_state);10292 set_loop_callback_state);
10448 } else {10293 } else {
@@ -10460,6 +10305,24 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 11 unmodified lines
10460 }10305 }
10461 break;10306 break;
10462 case BPF_FUNC_set_retval:10307 case BPF_FUNC_set_retval:
10308 {
10309 struct bpf_retval_range range = {
10310 .minval = -MAX_ERRNO,
10311 .maxval = 0,
10312 .return_32bit = true
10313 };
10314 struct bpf_reg_state *r1 = &regs[BPF_REG_1];
10315
10316 if (r1->type != SCALAR_VALUE) {
10317 verbose(env, "R1 is not a scalar\n");
10318 return -EINVAL;
10319 }
10320
10321 /* CGROUP_GETSOCKOPT is allowed to return arbitrary value */
10322 if (prog_type == BPF_PROG_TYPE_CGROUP_SOCKOPT &&
10323 env->prog->expected_attach_type == BPF_CGROUP_GETSOCKOPT)
10324 break;
10325
10463 if (prog_type == BPF_PROG_TYPE_LSM &&10326 if (prog_type == BPF_PROG_TYPE_LSM &&
10464 env->prog->expected_attach_type == BPF_LSM_CGROUP) {10327 env->prog->expected_attach_type == BPF_LSM_CGROUP) {
10465 if (!env->prog->aux->attach_func_proto->type) {10328 if (!env->prog->aux->attach_func_proto->type) {
@@ -10469,54 +10332,24 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 3 unmodified lines
10469 verbose(env, "BPF_LSM_CGROUP that attach to void LSM hooks can't modify return value!\n");10332 verbose(env, "BPF_LSM_CGROUP that attach to void LSM hooks can't modify return value!\n");
10470 return -EINVAL;10333 return -EINVAL;
10471 }10334 }
10472 }10335 bpf_lsm_get_retval_range(env->prog, &range);
10473 break;
10474 case BPF_FUNC_dynptr_data:
10475 {
10476 struct bpf_reg_state *reg;
10477 int id, ref_obj_id;
10478
10479 reg = get_dynptr_arg_reg(env, fn, regs);
10480 if (!reg)
10481 return -EFAULT;
10482
10483
10484 if (meta.dynptr_id) {
10485 verifier_bug(env, "meta.dynptr_id already set");
10486 return -EFAULT;
10487 }
10488 if (meta.ref_obj_id) {
10489 verifier_bug(env, "meta.ref_obj_id already set");
10490 return -EFAULT;
10491 }10336 }
10492 10337
10493 id = dynptr_id(env, reg);10338 err = mark_chain_precision(env, BPF_REG_1);
10494 if (id < 0) {10339 if (err)
10495 verifier_bug(env, "failed to obtain dynptr id");10340 return err;
10496 return id;
10497 }
10498 10341
10499 ref_obj_id = dynptr_ref_obj_id(env, reg);10342 if (!retval_range_within(range, r1)) {
10500 if (ref_obj_id < 0) {10343 verbose_invalid_scalar(env, r1, range, "At bpf_set_retval", "R1");
10501 verifier_bug(env, "failed to obtain dynptr ref_obj_id");10344 return -EINVAL;
10502 return ref_obj_id;
10503 }10345 }
10504 10346
10505 meta.dynptr_id = id;
10506 meta.ref_obj_id = ref_obj_id;
10507
10508 break;10347 break;
10509 }10348 }
10510 case BPF_FUNC_dynptr_write:10349 case BPF_FUNC_dynptr_write:
10511 {10350 {
10512 enum bpf_dynptr_type dynptr_type;10351 enum bpf_dynptr_type dynptr_type = meta.dynptr.type;
10513 struct bpf_reg_state *reg;
10514
10515 reg = get_dynptr_arg_reg(env, fn, regs);
10516 if (!reg)
10517 return -EFAULT;
10518 10352
10519 dynptr_type = dynptr_get_type(env, reg);
10520 if (dynptr_type == BPF_DYNPTR_TYPE_INVALID)10353 if (dynptr_type == BPF_DYNPTR_TYPE_INVALID)
10521 return -EFAULT;10354 return -EFAULT;
10522 10355
@@ -10560,6 +10393,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 37 unmodified lines
10560 bpf_mark_reg_not_init(env, &regs[caller_saved[i]]);10393 bpf_mark_reg_not_init(env, &regs[caller_saved[i]]);
10561 check_reg_arg(env, caller_saved[i], DST_OP_NO_MARK);10394 check_reg_arg(env, caller_saved[i], DST_OP_NO_MARK);
10562 }10395 }
10396 invalidate_outgoing_stack_args(env, cur_func(env));
10563 10397
10564 /* helper call returns 64-bit value. */10398 /* helper call returns 64-bit value. */
10565 regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG;10399 regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG;
@@ -10589,7 +10423,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 23 unmodified lines
10589 }10423 }
10590 10424
10591 if (func_id == BPF_FUNC_map_lookup_elem &&10425 if (func_id == BPF_FUNC_map_lookup_elem &&
10592 can_elide_value_nullness(meta.map.ptr->map_type) &&10426 can_elide_value_nullness(meta.map.ptr) &&
10593 meta.const_map_key >= 0 &&10427 meta.const_map_key >= 0 &&
10594 meta.const_map_key < meta.map.ptr->max_entries)10428 meta.const_map_key < meta.map.ptr->max_entries)
10595 ret_flag &= ~PTR_MAYBE_NULL;10429 ret_flag &= ~PTR_MAYBE_NULL;
@@ -10701,29 +10535,45 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 105 unmodified lines
10701 if (type_may_be_null(regs[BPF_REG_0].type))10535 if (type_may_be_null(regs[BPF_REG_0].type))
10702 regs[BPF_REG_0].id = ++env->id_gen;10536 regs[BPF_REG_0].id = ++env->id_gen;
10703 10537
10704 if (helper_multiple_ref_obj_use(func_id, meta.map.ptr)) {10538 if (is_ptr_cast_function(func_id) &&
10705 verifier_bug(env, "func %s#%d sets ref_obj_id more than once",10539 find_reference_state(env->cur_state, meta.ref_obj.id)) {
10706 func_id_name(func_id), func_id);10540 struct bpf_verifier_state *branch;
10707 return -EFAULT;10541 struct bpf_reg_state *r0;
10708 }10542
10543 err = validate_ref_obj(env, &meta.ref_obj);
10544 if (err)
10545 return err;
10546
10547 /*
10548 * In order for a release of any of the original or cast pointers
10549 * to invalidate all other pointers, reuse the same reference id for
10550 * the cast result.
10551 * This reference id can't be used for nullness propagation,
10552 * as cast might return NULL for a non-NULL input.
10553 * Hence, explore the NULL case as a separate branch.
10554 */
10555 branch = push_stack(env, env->insn_idx + 1, env->insn_idx, false);
10556 if (IS_ERR(branch))
10557 return PTR_ERR(branch);
10709 10558
10710 if (is_dynptr_ref_function(func_id))10559 r0 = &branch->frame[branch->curframe]->regs[BPF_REG_0];
10711 regs[BPF_REG_0].dynptr_id = meta.dynptr_id;10560 __mark_reg_known_zero(r0);
10561 r0->type = SCALAR_VALUE;
10712 10562
10713 if (is_ptr_cast_function(func_id) || is_dynptr_ref_function(func_id)) {10563 regs[BPF_REG_0].type &= ~PTR_MAYBE_NULL;
10714 /* For release_reference() */10564 regs[BPF_REG_0].id = meta.ref_obj.id;
10715 regs[BPF_REG_0].ref_obj_id = meta.ref_obj_id;
10716 } else if (is_acquire_function(func_id, meta.map.ptr)) {10565 } else if (is_acquire_function(func_id, meta.map.ptr)) {
10717 int id = acquire_reference(env, insn_idx);10566 int id = acquire_reference(env, insn_idx, 0);
10718 10567
10719 if (id < 0)10568 if (id < 0)
10720 return id;10569 return id;
10721 /* For mark_ptr_or_null_reg() */10570
10722 regs[BPF_REG_0].id = id;10571 regs[BPF_REG_0].id = id;
10723 /* For release_reference() */
10724 regs[BPF_REG_0].ref_obj_id = id;
10725 }10572 }
10726 10573
10574 if (func_id == BPF_FUNC_dynptr_data)
10575 regs[BPF_REG_0].parent_id = meta.dynptr.id;
10576
10727 err = do_refine_retval_range(env, regs, fn->ret_type, func_id, &meta);10577 err = do_refine_retval_range(env, regs, fn->ret_type, func_id, &meta);
10728 if (err)10578 if (err)
10729 return err;10579 return err;
@@ -10819,7 +10669,6 @@ static bool is_kfunc_release(struct bpf_kfunc_call_arg_meta *meta) · 89 unmodified lines
10819 return meta->kfunc_flags & KF_RELEASE;10669 return meta->kfunc_flags & KF_RELEASE;
10820}10670}
10821 10671
10822
10823static bool is_kfunc_destructive(struct bpf_kfunc_call_arg_meta *meta)10672static bool is_kfunc_destructive(struct bpf_kfunc_call_arg_meta *meta)
10824{10673{
10825 return meta->kfunc_flags & KF_DESTRUCTIVE;10674 return meta->kfunc_flags & KF_DESTRUCTIVE;
@@ -10896,6 +10745,11 @@ static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param · 70 unmodified lines
10896 return btf_param_match_suffix(btf, arg, "__nullable");10745 return btf_param_match_suffix(btf, arg, "__nullable");
10897}10746}
10898 10747
10748static bool is_kfunc_arg_nonown_allowed(const struct btf *btf, const struct btf_param *arg)
10749{
10750 return btf_param_match_suffix(btf, arg, "__nonown_allowed");
10751}
10752
10899static bool is_kfunc_arg_const_str(const struct btf *btf, const struct btf_param *arg)10753static bool is_kfunc_arg_const_str(const struct btf *btf, const struct btf_param *arg)
10900{10754{
10901 return btf_param_match_suffix(btf, arg, "__str");10755 return btf_param_match_suffix(btf, arg, "__str");
@@ -11136,10 +10990,15 @@ enum special_kfunc_type { · 234 unmodified lines
11136 KF_bpf_list_push_front,10990 KF_bpf_list_push_front,
11137 KF_bpf_list_push_back_impl,10991 KF_bpf_list_push_back_impl,
11138 KF_bpf_list_push_back,10992 KF_bpf_list_push_back,
10993 KF_bpf_list_add,
11139 KF_bpf_list_pop_front,10994 KF_bpf_list_pop_front,
11140 KF_bpf_list_pop_back,10995 KF_bpf_list_pop_back,
10996 KF_bpf_list_del,
11141 KF_bpf_list_front,10997 KF_bpf_list_front,
11142 KF_bpf_list_back,10998 KF_bpf_list_back,
10999 KF_bpf_list_is_first,
11000 KF_bpf_list_is_last,
11001 KF_bpf_list_empty,
11143 KF_bpf_cast_to_kern_ctx,11002 KF_bpf_cast_to_kern_ctx,
11144 KF_bpf_rdonly_cast,11003 KF_bpf_rdonly_cast,
11145 KF_bpf_rcu_read_lock,11004 KF_bpf_rcu_read_lock,
@@ -11204,10 +11063,15 @@ BTF_ID(func, bpf_list_push_front_impl) · 58 unmodified lines
11204BTF_ID(func, bpf_list_push_front)11063BTF_ID(func, bpf_list_push_front)
11205BTF_ID(func, bpf_list_push_back_impl)11064BTF_ID(func, bpf_list_push_back_impl)
11206BTF_ID(func, bpf_list_push_back)11065BTF_ID(func, bpf_list_push_back)
11066BTF_ID(func, bpf_list_add)
11207BTF_ID(func, bpf_list_pop_front)11067BTF_ID(func, bpf_list_pop_front)
11208BTF_ID(func, bpf_list_pop_back)11068BTF_ID(func, bpf_list_pop_back)
11069BTF_ID(func, bpf_list_del)
11209BTF_ID(func, bpf_list_front)11070BTF_ID(func, bpf_list_front)
11210BTF_ID(func, bpf_list_back)11071BTF_ID(func, bpf_list_back)
11072BTF_ID(func, bpf_list_is_first)
11073BTF_ID(func, bpf_list_is_last)
11074BTF_ID(func, bpf_list_empty)
11211BTF_ID(func, bpf_cast_to_kern_ctx)11075BTF_ID(func, bpf_cast_to_kern_ctx)
11212BTF_ID(func, bpf_rdonly_cast)11076BTF_ID(func, bpf_rdonly_cast)
11213BTF_ID(func, bpf_rcu_read_lock)11077BTF_ID(func, bpf_rcu_read_lock)
@@ -11319,7 +11183,8 @@ static bool is_bpf_list_push_kfunc(u32 func_id) · 105 unmodified lines
11319 return func_id == special_kfunc_list[KF_bpf_list_push_front] ||11183 return func_id == special_kfunc_list[KF_bpf_list_push_front] ||
11320 func_id == special_kfunc_list[KF_bpf_list_push_front_impl] ||11184 func_id == special_kfunc_list[KF_bpf_list_push_front_impl] ||
11321 func_id == special_kfunc_list[KF_bpf_list_push_back] ||11185 func_id == special_kfunc_list[KF_bpf_list_push_back] ||
11322 func_id == special_kfunc_list[KF_bpf_list_push_back_impl];11186 func_id == special_kfunc_list[KF_bpf_list_push_back_impl] ||
11187 func_id == special_kfunc_list[KF_bpf_list_add];
11323}11188}
11324 11189
11325static bool is_bpf_rbtree_add_kfunc(u32 func_id)11190static bool is_bpf_rbtree_add_kfunc(u32 func_id)
@@ -11368,15 +11233,12 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta) · 42 unmodified lines
11368}11233}
11369 11234
11370static enum kfunc_ptr_arg_type11235static enum kfunc_ptr_arg_type
11371get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,11236get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_func_state *caller,
11372 struct bpf_kfunc_call_arg_meta *meta,11237 struct bpf_reg_state *regs, struct bpf_kfunc_call_arg_meta *meta,
11373 const struct btf_type *t, const struct btf_type *ref_t,11238 const struct btf_type *t, const struct btf_type *ref_t,
11374 const char *ref_tname, const struct btf_param *args,11239 const char *ref_tname, const struct btf_param *args,
11375 int argno, int nargs)11240 int arg, int nargs, argno_t argno, struct bpf_reg_state *reg)
11376{11241{
11377 u32 regno = argno + 1;
11378 struct bpf_reg_state *regs = cur_regs(env);
11379 struct bpf_reg_state *reg = &regs[regno];
11380 bool arg_mem_size = false;11242 bool arg_mem_size = false;
11381 11243
11382 if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] ||11244 if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] ||
@@ -11384,9 +11246,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, · 1 unmodified line
11384 meta->func_id == special_kfunc_list[KF_bpf_session_cookie])11246 meta->func_id == special_kfunc_list[KF_bpf_session_cookie])
11385 return KF_ARG_PTR_TO_CTX;11247 return KF_ARG_PTR_TO_CTX;
11386 11248
11387 if (argno + 1 < nargs &&11249 if (arg + 1 < nargs &&
11388 (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1]) ||11250 (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1)) ||
11389 is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1])))11251 is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1))))
11390 arg_mem_size = true;11252 arg_mem_size = true;
11391 11253
11392 /* In this function, we verify the kfunc's BTF as per the argument type,11254 /* In this function, we verify the kfunc's BTF as per the argument type,
@@ -11394,68 +11256,69 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, · 1 unmodified line
11394 * type to our caller. When a set of conditions hold in the BTF type of11256 * type to our caller. When a set of conditions hold in the BTF type of
11395 * arguments, we resolve it to a known kfunc_ptr_arg_type.11257 * arguments, we resolve it to a known kfunc_ptr_arg_type.
11396 */11258 */
11397 if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), argno))11259 if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), arg))
11398 return KF_ARG_PTR_TO_CTX;11260 return KF_ARG_PTR_TO_CTX;
11399 11261
11400 if (is_kfunc_arg_nullable(meta->btf, &args[argno]) && bpf_register_is_null(reg) &&11262 if (is_kfunc_arg_nullable(meta->btf, &args[arg]) && bpf_register_is_null(reg) &&
11401 !arg_mem_size)11263 !arg_mem_size)
11402 return KF_ARG_PTR_TO_NULL;11264 return KF_ARG_PTR_TO_NULL;
11403 11265
11404 if (is_kfunc_arg_alloc_obj(meta->btf, &args[argno]))11266 if (is_kfunc_arg_alloc_obj(meta->btf, &args[arg]))
11405 return KF_ARG_PTR_TO_ALLOC_BTF_ID;11267 return KF_ARG_PTR_TO_ALLOC_BTF_ID;
11406 11268
11407 if (is_kfunc_arg_refcounted_kptr(meta->btf, &args[argno]))11269 if (is_kfunc_arg_refcounted_kptr(meta->btf, &args[arg]))
11408 return KF_ARG_PTR_TO_REFCOUNTED_KPTR;11270 return KF_ARG_PTR_TO_REFCOUNTED_KPTR;
11409 11271
11410 if (is_kfunc_arg_dynptr(meta->btf, &args[argno]))11272 if (is_kfunc_arg_dynptr(meta->btf, &args[arg]))
11411 return KF_ARG_PTR_TO_DYNPTR;11273 return KF_ARG_PTR_TO_DYNPTR;
11412 11274
11413 if (is_kfunc_arg_iter(meta, argno, &args[argno]))11275 if (is_kfunc_arg_iter(meta, arg, &args[arg]))
11414 return KF_ARG_PTR_TO_ITER;11276 return KF_ARG_PTR_TO_ITER;
11415 11277
11416 if (is_kfunc_arg_list_head(meta->btf, &args[argno]))11278 if (is_kfunc_arg_list_head(meta->btf, &args[arg]))
11417 return KF_ARG_PTR_TO_LIST_HEAD;11279 return KF_ARG_PTR_TO_LIST_HEAD;
11418 11280
11419 if (is_kfunc_arg_list_node(meta->btf, &args[argno]))11281 if (is_kfunc_arg_list_node(meta->btf, &args[arg]))
11420 return KF_ARG_PTR_TO_LIST_NODE;11282 return KF_ARG_PTR_TO_LIST_NODE;
11421 11283
11422 if (is_kfunc_arg_rbtree_root(meta->btf, &args[argno]))11284 if (is_kfunc_arg_rbtree_root(meta->btf, &args[arg]))
11423 return KF_ARG_PTR_TO_RB_ROOT;11285 return KF_ARG_PTR_TO_RB_ROOT;
11424 11286
11425 if (is_kfunc_arg_rbtree_node(meta->btf, &args[argno]))11287 if (is_kfunc_arg_rbtree_node(meta->btf, &args[arg]))
11426 return KF_ARG_PTR_TO_RB_NODE;11288 return KF_ARG_PTR_TO_RB_NODE;
11427 11289
11428 if (is_kfunc_arg_const_str(meta->btf, &args[argno]))11290 if (is_kfunc_arg_const_str(meta->btf, &args[arg]))
11429 return KF_ARG_PTR_TO_CONST_STR;11291 return KF_ARG_PTR_TO_CONST_STR;
11430 11292
11431 if (is_kfunc_arg_map(meta->btf, &args[argno]))11293 if (is_kfunc_arg_map(meta->btf, &args[arg]))
11432 return KF_ARG_PTR_TO_MAP;11294 return KF_ARG_PTR_TO_MAP;
11433 11295
11434 if (is_kfunc_arg_wq(meta->btf, &args[argno]))11296 if (is_kfunc_arg_wq(meta->btf, &args[arg]))
11435 return KF_ARG_PTR_TO_WORKQUEUE;11297 return KF_ARG_PTR_TO_WORKQUEUE;
11436 11298
11437 if (is_kfunc_arg_timer(meta->btf, &args[argno]))11299 if (is_kfunc_arg_timer(meta->btf, &args[arg]))
11438 return KF_ARG_PTR_TO_TIMER;11300 return KF_ARG_PTR_TO_TIMER;
11439 11301
11440 if (is_kfunc_arg_task_work(meta->btf, &args[argno]))11302 if (is_kfunc_arg_task_work(meta->btf, &args[arg]))
11441 return KF_ARG_PTR_TO_TASK_WORK;11303 return KF_ARG_PTR_TO_TASK_WORK;
11442 11304
11443 if (is_kfunc_arg_irq_flag(meta->btf, &args[argno]))11305 if (is_kfunc_arg_irq_flag(meta->btf, &args[arg]))
11444 return KF_ARG_PTR_TO_IRQ_FLAG;11306 return KF_ARG_PTR_TO_IRQ_FLAG;
11445 11307
11446 if (is_kfunc_arg_res_spin_lock(meta->btf, &args[argno]))11308 if (is_kfunc_arg_res_spin_lock(meta->btf, &args[arg]))
11447 return KF_ARG_PTR_TO_RES_SPIN_LOCK;11309 return KF_ARG_PTR_TO_RES_SPIN_LOCK;
11448 11310
11449 if ((base_type(reg->type) == PTR_TO_BTF_ID || reg2btf_ids[base_type(reg->type)])) {11311 if ((base_type(reg->type) == PTR_TO_BTF_ID || reg2btf_ids[base_type(reg->type)])) {
11450 if (!btf_type_is_struct(ref_t)) {11312 if (!btf_type_is_struct(ref_t)) {
11451 verbose(env, "kernel function %s args#%d pointer type %s %s is not supported\n",11313 verbose(env, "kernel function %s %s pointer type %s %s is not supported\n",
11452 meta->func_name, argno, btf_type_str(ref_t), ref_tname);11314 meta->func_name, reg_arg_name(env, argno),
11315 btf_type_str(ref_t), ref_tname);
11453 return -EINVAL;11316 return -EINVAL;
11454 }11317 }
11455 return KF_ARG_PTR_TO_BTF_ID;11318 return KF_ARG_PTR_TO_BTF_ID;
11456 }11319 }
11457 11320
11458 if (is_kfunc_arg_callback(env, meta->btf, &args[argno]))11321 if (is_kfunc_arg_callback(env, meta->btf, &args[arg]))
11459 return KF_ARG_PTR_TO_CALLBACK;11322 return KF_ARG_PTR_TO_CALLBACK;
11460 11323
11461 /* This is the catch all argument type of register types supported by11324 /* This is the catch all argument type of register types supported by
@@ -11465,8 +11328,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, · 3 unmodified lines
11465 */11328 */
11466 if (!btf_type_is_scalar(ref_t) && !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0) &&11329 if (!btf_type_is_scalar(ref_t) && !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0) &&
11467 (arg_mem_size ? !btf_type_is_void(ref_t) : 1)) {11330 (arg_mem_size ? !btf_type_is_void(ref_t) : 1)) {
11468 verbose(env, "arg#%d pointer type %s %s must point to %sscalar, or struct with scalar\n",11331 verbose(env, "%s pointer type %s %s must point to %sscalar, or struct with scalar\n",
11469 argno, btf_type_str(ref_t), ref_tname, arg_mem_size ? "void, " : "");11332 reg_arg_name(env, argno),
11333 btf_type_str(ref_t), ref_tname, arg_mem_size ? "void, " : "");
11470 return -EINVAL;11334 return -EINVAL;
11471 }11335 }
11472 return arg_mem_size ? KF_ARG_PTR_TO_MEM_SIZE : KF_ARG_PTR_TO_MEM;11336 return arg_mem_size ? KF_ARG_PTR_TO_MEM_SIZE : KF_ARG_PTR_TO_MEM;
@@ -11477,7 +11341,7 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · 4 unmodified lines
11477 const struct btf_type *ref_t,11341 const struct btf_type *ref_t,
11478 const char *ref_tname, u32 ref_id,11342 const char *ref_tname, u32 ref_id,
11479 struct bpf_kfunc_call_arg_meta *meta,11343 struct bpf_kfunc_call_arg_meta *meta,
11480 int argno)11344 int arg, argno_t argno)
11481{11345{
11482 const struct btf_type *reg_ref_t;11346 const struct btf_type *reg_ref_t;
11483 bool strict_type_match = false;11347 bool strict_type_match = false;
@@ -11519,7 +11383,7 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · 35 unmodified lines
11519 * btf_struct_ids_match() to walk the struct at the 0th offset, and11383 * btf_struct_ids_match() to walk the struct at the 0th offset, and
11520 * resolve types.11384 * resolve types.
11521 */11385 */
11522 if ((is_kfunc_release(meta) && reg->ref_obj_id) ||11386 if ((is_kfunc_release(meta) && reg_is_referenced(env, reg)) ||
11523 btf_type_ids_nocast_alias(&env->log, reg_btf, reg_ref_id, meta->btf, ref_id))11387 btf_type_ids_nocast_alias(&env->log, reg_btf, reg_ref_id, meta->btf, ref_id))
11524 strict_type_match = true;11388 strict_type_match = true;
11525 11389
@@ -11535,19 +11399,19 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · 9 unmodified lines
11535 */11399 */
11536 taking_projection = btf_is_projection_of(ref_tname, reg_ref_tname);11400 taking_projection = btf_is_projection_of(ref_tname, reg_ref_tname);
11537 if (!taking_projection && !struct_same) {11401 if (!taking_projection && !struct_same) {
11538 verbose(env, "kernel function %s args#%d expected pointer to %s %s but R%d has a pointer to %s %s\n",11402 verbose(env, "kernel function %s %s expected pointer to %s %s but %s has a pointer to %s %s\n",
11539 meta->func_name, argno, btf_type_str(ref_t), ref_tname, argno + 1,11403 meta->func_name, reg_arg_name(env, argno),
11404 btf_type_str(ref_t), ref_tname, reg_arg_name(env, argno),
11540 btf_type_str(reg_ref_t), reg_ref_tname);11405 btf_type_str(reg_ref_t), reg_ref_tname);
11541 return -EINVAL;11406 return -EINVAL;
11542 }11407 }
11543 return 0;11408 return 0;
11544}11409}
11545 11410
11546static int process_irq_flag(struct bpf_verifier_env *env, int regno,11411static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno,
11547 struct bpf_kfunc_call_arg_meta *meta)11412 struct bpf_kfunc_call_arg_meta *meta)
11548{11413{
11549 struct bpf_reg_state *reg = reg_state(env, regno);11414 int err, spi, kfunc_class = IRQ_NATIVE_KFUNC;
11550 int err, kfunc_class = IRQ_NATIVE_KFUNC;
11551 bool irq_save;11415 bool irq_save;
11552 11416
11553 if (meta->func_id == special_kfunc_list[KF_bpf_local_irq_save] ||11417 if (meta->func_id == special_kfunc_list[KF_bpf_local_irq_save] ||
@@ -11567,11 +11431,13 @@ static int process_irq_flag(struct bpf_verifier_env *env, int regno, · 13 unmodified lines
11567 11431
11568 if (irq_save) {11432 if (irq_save) {
11569 if (!is_irq_flag_reg_valid_uninit(env, reg)) {11433 if (!is_irq_flag_reg_valid_uninit(env, reg)) {
11570 verbose(env, "expected uninitialized irq flag as arg#%d\n", regno - 1);11434 verbose(env, "expected uninitialized irq flag as %s\n",
11435 reg_arg_name(env, argno));
11571 return -EINVAL;11436 return -EINVAL;
11572 }11437 }
11573 11438
11574 err = check_mem_access(env, env->insn_idx, regno, 0, BPF_DW, BPF_WRITE, -1, false, false);11439 err = check_mem_access(env, env->insn_idx, reg, argno, 0, BPF_DW,
11440 BPF_WRITE, -1, false, false);
11575 if (err)11441 if (err)
11576 return err;11442 return err;
11577 11443
@@ -11581,13 +11447,16 @@ static int process_irq_flag(struct bpf_verifier_env *env, int regno, · 3 unmodified lines
11581 } else {11447 } else {
11582 err = is_irq_flag_reg_valid_init(env, reg);11448 err = is_irq_flag_reg_valid_init(env, reg);
11583 if (err) {11449 if (err) {
11584 verbose(env, "expected an initialized irq flag as arg#%d\n", regno - 1);11450 verbose(env, "expected an initialized irq flag as %s\n",
11451 reg_arg_name(env, argno));
11585 return err;11452 return err;
11586 }11453 }
11587 11454
11588 err = mark_irq_flag_read(env, reg);11455 spi = irq_flag_get_spi(env, reg);
11589 if (err)11456 if (spi < 0)
11590 return err;11457 return spi;
11458
11459 mark_stack_slots_scratched(env, spi, 1);
11591 11460
11592 err = unmark_stack_slot_irq_flag(env, reg, kfunc_class);11461 err = unmark_stack_slot_irq_flag(env, reg, kfunc_class);
11593 if (err)11462 if (err)
@@ -11618,36 +11487,21 @@ static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state · 24 unmodified lines
11618 return 0;11487 return 0;
11619}11488}
11620 11489
11621static int ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 ref_obj_id)11490static void ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 id)
11622{11491{
11623 struct bpf_verifier_state *state = env->cur_state;
11624 struct bpf_func_state *unused;11492 struct bpf_func_state *unused;
11625 struct bpf_reg_state *reg;11493 struct bpf_reg_state *reg;
11626 int i;
11627
11628 if (!ref_obj_id) {
11629 verifier_bug(env, "ref_obj_id is zero for owning -> non-owning conversion");
11630 return -EFAULT;
11631 }
11632 11494
11633 for (i = 0; i < state->acquired_refs; i++) {

2971 more lines aren’t shown here.

kernel/events/core.c+9−0
@@ -11712,6 +11712,15 @@ static int __perf_event_set_bpf_prog(struct perf_event *event, · 11711 unmodified lines
11712 /* only uprobe programs are allowed to be sleepable */11712 /* only uprobe programs are allowed to be sleepable */
11713 return -EINVAL;11713 return -EINVAL;
11714 11714
11715 if (prog->type == BPF_PROG_TYPE_TRACEPOINT && prog->sleepable) {
11716 /*
11717 * Sleepable tracepoint programs can only attach to faultable
11718 * tracepoints. Currently only syscall tracepoints are faultable.
11719 */
11720 if (!is_syscall_tp)
11721 return -EINVAL;
11722 }
11723
11715 /* Kprobe override only works for kprobes, not uprobes. */11724 /* Kprobe override only works for kprobes, not uprobes. */
11716 if (prog->kprobe_override && !is_kprobe)11725 if (prog->kprobe_override && !is_kprobe)
11717 return -EINVAL;11726 return -EINVAL;
kernel/trace/bpf_trace.c+306−28
@@ -23,6 +23,7 @@ · 22 unmodified lines
23#include <linux/sort.h>23#include <linux/sort.h>
24#include <linux/key.h>24#include <linux/key.h>
25#include <linux/namei.h>25#include <linux/namei.h>
26#include <linux/file.h>
26 27
27#include <net/bpf_sk_storage.h>28#include <net/bpf_sk_storage.h>
28 29
@@ -42,6 +43,7 @@ · 13 unmodified lines
42 43
43#define MAX_UPROBE_MULTI_CNT (1U << 20)44#define MAX_UPROBE_MULTI_CNT (1U << 20)
44#define MAX_KPROBE_MULTI_CNT (1U << 20)45#define MAX_KPROBE_MULTI_CNT (1U << 20)
46#define MAX_TRACING_MULTI_CNT (1U << 20)
45 47
46#ifdef CONFIG_MODULES48#ifdef CONFIG_MODULES
47struct bpf_trace_module {49struct bpf_trace_module {
@@ -152,6 +154,34 @@ unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx) · 104 unmodified lines
152 return ret;154 return ret;
153}155}
154 156
157/**
158 * trace_call_bpf_faultable - invoke BPF program in faultable context
159 * @call: tracepoint event
160 * @ctx: opaque context pointer
161 *
162 * Variant of trace_call_bpf() for faultable tracepoints (syscall
163 * tracepoints). Supports sleepable BPF programs by using rcu_tasks_trace
164 * for lifetime protection and bpf_prog_run_array_sleepable() for per-program
165 * RCU flavor selection, following the uprobe pattern.
166 *
167 * Per-program recursion protection is provided by
168 * bpf_prog_run_array_sleepable(). Global bpf_prog_active is not
169 * needed because syscall tracepoints cannot self-recurse.
170 *
171 * Must be called from a faultable/preemptible context.
172 */
173unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx)
174{
175 struct bpf_prog_array *prog_array;
176
177 might_fault();
178 guard(rcu_tasks_trace)();
179
180 prog_array = rcu_dereference_check(call->prog_array,
181 rcu_read_lock_trace_held());
182 return bpf_prog_run_array_sleepable(prog_array, ctx, bpf_prog_run);
183}
184
155#ifdef CONFIG_BPF_KPROBE_OVERRIDE185#ifdef CONFIG_BPF_KPROBE_OVERRIDE
156BPF_CALL_2(bpf_override_return, struct pt_regs *, regs, unsigned long, rc)186BPF_CALL_2(bpf_override_return, struct pt_regs *, regs, unsigned long, rc)
157{187{
@@ -1305,7 +1335,8 @@ static inline bool is_uprobe_session(const struct bpf_prog *prog) · 1147 unmodified lines
1305static inline bool is_trace_fsession(const struct bpf_prog *prog)1335static inline bool is_trace_fsession(const struct bpf_prog *prog)
1306{1336{
1307 return prog->type == BPF_PROG_TYPE_TRACING &&1337 return prog->type == BPF_PROG_TYPE_TRACING &&
1308 prog->expected_attach_type == BPF_TRACE_FSESSION;1338 (prog->expected_attach_type == BPF_TRACE_FSESSION ||
1339 prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI);
1309}1340}
1310 1341
1311static const struct bpf_func_proto *1342static const struct bpf_func_proto *
@@ -2072,11 +2103,19 @@ void bpf_put_raw_tracepoint(struct bpf_raw_event_map *btp) · 760 unmodified lines
2072static __always_inline2103static __always_inline
2073void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args)2104void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args)
2074{2105{
2106 struct srcu_ctr __percpu *scp = NULL;
2075 struct bpf_prog *prog = link->link.prog;2107 struct bpf_prog *prog = link->link.prog;
2108 bool sleepable = prog->sleepable;
2076 struct bpf_run_ctx *old_run_ctx;2109 struct bpf_run_ctx *old_run_ctx;
2077 struct bpf_trace_run_ctx run_ctx;2110 struct bpf_trace_run_ctx run_ctx;
2078 2111
2079 rcu_read_lock_dont_migrate();2112 if (sleepable) {
2113 scp = rcu_read_lock_tasks_trace();
2114 migrate_disable();
2115 } else {
2116 rcu_read_lock_dont_migrate();
2117 }
2118
2080 if (unlikely(!bpf_prog_get_recursion_context(prog))) {2119 if (unlikely(!bpf_prog_get_recursion_context(prog))) {
2081 bpf_prog_inc_misses_counter(prog);2120 bpf_prog_inc_misses_counter(prog);
2082 goto out;2121 goto out;
@@ -2085,12 +2124,18 @@ void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args) · 2 unmodified lines
2085 run_ctx.bpf_cookie = link->cookie;2124 run_ctx.bpf_cookie = link->cookie;
2086 old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);2125 old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
2087 2126
2088 (void) bpf_prog_run(prog, args);2127 (void)bpf_prog_run(prog, args);
2089 2128
2090 bpf_reset_run_ctx(old_run_ctx);2129 bpf_reset_run_ctx(old_run_ctx);
2091out:2130out:
2092 bpf_prog_put_recursion_context(prog);2131 bpf_prog_put_recursion_context(prog);
2093 rcu_read_unlock_migrate();2132
2133 if (sleepable) {
2134 migrate_enable();
2135 rcu_read_unlock_tasks_trace(scp);
2136 } else {
2137 rcu_read_unlock_migrate();
2138 }
2094}2139}
2095 2140
2096#define UNPACK(...) __VA_ARGS__2141#define UNPACK(...) __VA_ARGS__
@@ -3170,6 +3215,38 @@ static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx) · 1073 unmodified lines
3170 return run_ctx->uprobe->cookie;3215 return run_ctx->uprobe->cookie;
3171}3216}
3172 3217
3218static int bpf_uprobe_multi_get_path(const union bpf_attr *attr, struct path *path)
3219{
3220 void __user *upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path);
3221 u32 path_fd = attr->link_create.uprobe_multi.path_fd;
3222 u32 flags = attr->link_create.uprobe_multi.flags;
3223
3224 if (flags & BPF_F_UPROBE_MULTI_PATH_FD) {
3225 /*
3226 * When BPF_F_UPROBE_MULTI_PATH_FD is set, the executable is
3227 * identified by path_fd, upath must be NULL.
3228 */
3229 if (upath)
3230 return -EINVAL;
3231
3232 CLASS(fd, f)(path_fd);
3233 if (fd_empty(f))
3234 return -EBADF;
3235 *path = fd_file(f)->f_path;
3236 path_get(path);
3237 return 0;
3238 }
3239
3240 /*
3241 * When BPF_F_UPROBE_MULTI_PATH_FD is not set, the path is resolved
3242 * relative to the cwd (AT_FDCWD) or absolute using the upath string.
3243 */
3244 if (!upath || path_fd)
3245 return -EINVAL;
3246
3247 return user_path_at(AT_FDCWD, upath, LOOKUP_FOLLOW, path);
3248}
3249
3173int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)3250int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
3174{3251{
3175 struct bpf_uprobe_multi_link *link = NULL;3252 struct bpf_uprobe_multi_link *link = NULL;
@@ -3179,10 +3256,9 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr · 3 unmodified lines
3179 struct task_struct *task = NULL;3256 struct task_struct *task = NULL;
3180 unsigned long __user *uoffsets;3257 unsigned long __user *uoffsets;
3181 u64 __user *ucookies;3258 u64 __user *ucookies;
3182 void __user *upath;3259 unsigned long size;
3183 u32 flags, cnt, i;3260 u32 flags, cnt, i;
3184 struct path path;3261 struct path path;
3185 char *name;
3186 pid_t pid;3262 pid_t pid;
3187 int err;3263 int err;
3188 3264
@@ -3197,19 +3273,18 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr · 8 unmodified lines
3197 return -EINVAL;3273 return -EINVAL;
3198 3274
3199 flags = attr->link_create.uprobe_multi.flags;3275 flags = attr->link_create.uprobe_multi.flags;
3200 if (flags & ~BPF_F_UPROBE_MULTI_RETURN)3276 if (flags & ~(BPF_F_UPROBE_MULTI_RETURN | BPF_F_UPROBE_MULTI_PATH_FD))
3201 return -EINVAL;3277 return -EINVAL;
3202 3278
3203 /*3279 /*
3204 * path, offsets and cnt are mandatory,3280 * offsets and cnt are mandatory,
3205 * ref_ctr_offsets and cookies are optional3281 * ref_ctr_offsets and cookies are optional
3206 */3282 */
3207 upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path);
3208 uoffsets = u64_to_user_ptr(attr->link_create.uprobe_multi.offsets);3283 uoffsets = u64_to_user_ptr(attr->link_create.uprobe_multi.offsets);
3209 cnt = attr->link_create.uprobe_multi.cnt;3284 cnt = attr->link_create.uprobe_multi.cnt;
3210 pid = attr->link_create.uprobe_multi.pid;3285 pid = attr->link_create.uprobe_multi.pid;
3211 3286
3212 if (!upath || !uoffsets || !cnt || pid < 0)3287 if (!uoffsets || !cnt || pid < 0)
3213 return -EINVAL;3288 return -EINVAL;
3214 if (cnt > MAX_UPROBE_MULTI_CNT)3289 if (cnt > MAX_UPROBE_MULTI_CNT)
3215 return -E2BIG;3290 return -E2BIG;
@@ -3217,14 +3292,17 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr · 1 unmodified line
3217 uref_ctr_offsets = u64_to_user_ptr(attr->link_create.uprobe_multi.ref_ctr_offsets);3292 uref_ctr_offsets = u64_to_user_ptr(attr->link_create.uprobe_multi.ref_ctr_offsets);
3218 ucookies = u64_to_user_ptr(attr->link_create.uprobe_multi.cookies);3293 ucookies = u64_to_user_ptr(attr->link_create.uprobe_multi.cookies);
3219 3294
3220 name = strndup_user(upath, PATH_MAX);3295 /*
3221 if (IS_ERR(name)) {3296 * All uoffsets/uref_ctr_offsets/ucookies arrays have the same value
3222 err = PTR_ERR(name);3297 * size, we need to check their address range is safe for __get_user
3223 return err;3298 * calls.
3224 }3299 */
3300 size = sizeof(*uoffsets) * cnt;
3301 if (!access_ok(uoffsets, size) || !access_ok(uref_ctr_offsets, size) ||
3302 !access_ok(ucookies, size))
3303 return -EFAULT;
3225 3304
3226 err = kern_path(name, LOOKUP_FOLLOW, &path);3305 err = bpf_uprobe_multi_get_path(attr, &path);
3227 kfree(name);
3228 if (err)3306 if (err)
3229 return err;3307 return err;
3230 3308
@@ -3398,12 +3476,12 @@ typedef int (*copy_fn_t)(void *dst, const void *src, u32 size, struct task_struc · 167 unmodified lines
3398 * direct calls into all the specific callback implementations3476 * direct calls into all the specific callback implementations
3399 * (copy_user_data_sleepable, copy_user_data_nofault, and so on)3477 * (copy_user_data_sleepable, copy_user_data_nofault, and so on)
3400 */3478 */
3401static __always_inline int __bpf_dynptr_copy_str(struct bpf_dynptr *dptr, u64 doff, u64 size,3479static __always_inline int __bpf_dynptr_copy_str(const struct bpf_dynptr *dptr, u64 doff, u64 size,
3402 const void *unsafe_src,3480 const void *unsafe_src,
3403 copy_fn_t str_copy_fn,3481 copy_fn_t str_copy_fn,
3404 struct task_struct *tsk)3482 struct task_struct *tsk)
3405{3483{
3406 struct bpf_dynptr_kern *dst;3484 const struct bpf_dynptr_kern *dst;
3407 u64 chunk_sz, off;3485 u64 chunk_sz, off;
3408 void *dst_slice;3486 void *dst_slice;
3409 int cnt, err;3487 int cnt, err;
@@ -3439,7 +3517,7 @@ static __always_inline int __bpf_dynptr_copy(const struct bpf_dynptr *dptr, u64 · 29 unmodified lines
3439 u64 size, const void *unsafe_src,3517 u64 size, const void *unsafe_src,
3440 copy_fn_t copy_fn, struct task_struct *tsk)3518 copy_fn_t copy_fn, struct task_struct *tsk)
3441{3519{
3442 struct bpf_dynptr_kern *dst;3520 const struct bpf_dynptr_kern *dst;
3443 void *dst_slice;3521 void *dst_slice;
3444 char buf[256];3522 char buf[256];
3445 u64 off, chunk_sz;3523 u64 off, chunk_sz;
@@ -3540,49 +3618,49 @@ __bpf_kfunc int bpf_send_signal_task(struct task_struct *task, int sig, enum pid · 94 unmodified lines
3540 return bpf_send_signal_common(sig, type, task, value);3618 return bpf_send_signal_common(sig, type, task, value);
3541}3619}
3542 3620
3543__bpf_kfunc int bpf_probe_read_user_dynptr(struct bpf_dynptr *dptr, u64 off,3621__bpf_kfunc int bpf_probe_read_user_dynptr(const struct bpf_dynptr *dptr, u64 off,
3544 u64 size, const void __user *unsafe_ptr__ign)3622 u64 size, const void __user *unsafe_ptr__ign)
3545{3623{
3546 return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,3624 return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,
3547 copy_user_data_nofault, NULL);3625 copy_user_data_nofault, NULL);
3548}3626}
3549 3627
3550__bpf_kfunc int bpf_probe_read_kernel_dynptr(struct bpf_dynptr *dptr, u64 off,3628__bpf_kfunc int bpf_probe_read_kernel_dynptr(const struct bpf_dynptr *dptr, u64 off,
3551 u64 size, const void *unsafe_ptr__ign)3629 u64 size, const void *unsafe_ptr__ign)
3552{3630{
3553 return __bpf_dynptr_copy(dptr, off, size, unsafe_ptr__ign,3631 return __bpf_dynptr_copy(dptr, off, size, unsafe_ptr__ign,
3554 copy_kernel_data_nofault, NULL);3632 copy_kernel_data_nofault, NULL);
3555}3633}
3556 3634
3557__bpf_kfunc int bpf_probe_read_user_str_dynptr(struct bpf_dynptr *dptr, u64 off,3635__bpf_kfunc int bpf_probe_read_user_str_dynptr(const struct bpf_dynptr *dptr, u64 off,
3558 u64 size, const void __user *unsafe_ptr__ign)3636 u64 size, const void __user *unsafe_ptr__ign)
3559{3637{
3560 return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,3638 return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,
3561 copy_user_str_nofault, NULL);3639 copy_user_str_nofault, NULL);
3562}3640}
3563 3641
3564__bpf_kfunc int bpf_probe_read_kernel_str_dynptr(struct bpf_dynptr *dptr, u64 off,3642__bpf_kfunc int bpf_probe_read_kernel_str_dynptr(const struct bpf_dynptr *dptr, u64 off,
3565 u64 size, const void *unsafe_ptr__ign)3643 u64 size, const void *unsafe_ptr__ign)
3566{3644{
3567 return __bpf_dynptr_copy_str(dptr, off, size, unsafe_ptr__ign,3645 return __bpf_dynptr_copy_str(dptr, off, size, unsafe_ptr__ign,
3568 copy_kernel_str_nofault, NULL);3646 copy_kernel_str_nofault, NULL);
3569}3647}
3570 3648
3571__bpf_kfunc int bpf_copy_from_user_dynptr(struct bpf_dynptr *dptr, u64 off,3649__bpf_kfunc int bpf_copy_from_user_dynptr(const struct bpf_dynptr *dptr, u64 off,
3572 u64 size, const void __user *unsafe_ptr__ign)3650 u64 size, const void __user *unsafe_ptr__ign)
3573{3651{
3574 return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,3652 return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,
3575 copy_user_data_sleepable, NULL);3653 copy_user_data_sleepable, NULL);
3576}3654}
3577 3655
3578__bpf_kfunc int bpf_copy_from_user_str_dynptr(struct bpf_dynptr *dptr, u64 off,3656__bpf_kfunc int bpf_copy_from_user_str_dynptr(const struct bpf_dynptr *dptr, u64 off,
3579 u64 size, const void __user *unsafe_ptr__ign)3657 u64 size, const void __user *unsafe_ptr__ign)
3580{3658{
3581 return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,3659 return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,
3582 copy_user_str_sleepable, NULL);3660 copy_user_str_sleepable, NULL);
3583}3661}
3584 3662
3585__bpf_kfunc int bpf_copy_from_user_task_dynptr(struct bpf_dynptr *dptr, u64 off,3663__bpf_kfunc int bpf_copy_from_user_task_dynptr(const struct bpf_dynptr *dptr, u64 off,
3586 u64 size, const void __user *unsafe_ptr__ign,3664 u64 size, const void __user *unsafe_ptr__ign,
3587 struct task_struct *tsk)3665 struct task_struct *tsk)
3588{3666{
@@ -3590,7 +3668,7 @@ __bpf_kfunc int bpf_copy_from_user_task_dynptr(struct bpf_dynptr *dptr, u64 off, · 1 unmodified line
3590 copy_user_data_sleepable, tsk);3668 copy_user_data_sleepable, tsk);
3591}3669}
3592 3670
3593__bpf_kfunc int bpf_copy_from_user_task_str_dynptr(struct bpf_dynptr *dptr, u64 off,3671__bpf_kfunc int bpf_copy_from_user_task_str_dynptr(const struct bpf_dynptr *dptr, u64 off,
3594 u64 size, const void __user *unsafe_ptr__ign,3672 u64 size, const void __user *unsafe_ptr__ign,
3595 struct task_struct *tsk)3673 struct task_struct *tsk)
3596{3674{
@@ -3599,3 +3677,203 @@ __bpf_kfunc int bpf_copy_from_user_task_str_dynptr(struct bpf_dynptr *dptr, u64 · 2 unmodified lines
3599}3677}
3600 3678
3601__bpf_kfunc_end_defs();3679__bpf_kfunc_end_defs();
3680
3681#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \
3682 defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS)
3683
3684static void bpf_tracing_multi_link_release(struct bpf_link *link)
3685{
3686 struct bpf_tracing_multi_link *tr_link =
3687 container_of(link, struct bpf_tracing_multi_link, link);
3688
3689 WARN_ON_ONCE(bpf_trampoline_multi_detach(link->prog, tr_link));
3690}
3691
3692static void bpf_tracing_multi_link_dealloc(struct bpf_link *link)
3693{
3694 struct bpf_tracing_multi_link *tr_link =
3695 container_of(link, struct bpf_tracing_multi_link, link);
3696
3697 kvfree(tr_link->fexits);
3698 kvfree(tr_link->cookies);
3699 kvfree(tr_link);
3700}
3701
3702#ifdef CONFIG_PROC_FS
3703static void bpf_tracing_multi_show_fdinfo(const struct bpf_link *link,
3704 struct seq_file *seq)
3705{
3706 struct bpf_tracing_multi_link *tr_link =
3707 container_of(link, struct bpf_tracing_multi_link, link);
3708 bool has_cookies = !!tr_link->cookies;
3709
3710 seq_printf(seq, "attach_type:\t%u\n", tr_link->link.attach_type);
3711 seq_printf(seq, "cnt:\t%u\n", tr_link->nodes_cnt);
3712
3713 seq_printf(seq, "%s\t %s\t %s\t %s\n", "obj-id", "btf-id", "cookie", "func");
3714 for (int i = 0; i < tr_link->nodes_cnt; i++) {
3715 struct bpf_tracing_multi_node *mnode = &tr_link->nodes[i];
3716 u32 btf_id, obj_id;
3717
3718 bpf_trampoline_unpack_key(mnode->trampoline->key, &obj_id, &btf_id);
3719 seq_printf(seq, "%u\t %u\t %llu\t %pS\n",
3720 obj_id, btf_id,
3721 has_cookies ? tr_link->cookies[i] : 0,
3722 (void *) mnode->trampoline->ip);
3723
3724 cond_resched();
3725 }
3726}
3727#endif
3728
3729static const struct bpf_link_ops bpf_tracing_multi_link_lops = {
3730 .release = bpf_tracing_multi_link_release,
3731 .dealloc_deferred = bpf_tracing_multi_link_dealloc,
3732#ifdef CONFIG_PROC_FS
3733 .show_fdinfo = bpf_tracing_multi_show_fdinfo,
3734#endif
3735};
3736
3737static int ids_cmp_r(const void *pa, const void *pb, const void *priv __maybe_unused)
3738{
3739 u32 a = *(u32 *) pa;
3740 u32 b = *(u32 *) pb;
3741
3742 return (a > b) - (a < b);
3743}
3744
3745static void ids_swap_r(void *a, void *b, int size __maybe_unused,
3746 const void *priv __maybe_unused)
3747{
3748 u64 *cookie_a, *cookie_b, *cookies;
3749 u32 *id_a = a, *id_b = b, *ids;
3750 void **data = (void **) priv;
3751
3752 ids = data[0];
3753 cookies = data[1];
3754
3755 if (cookies) {
3756 cookie_a = cookies + (id_a - ids);
3757 cookie_b = cookies + (id_b - ids);
3758 swap(*cookie_a, *cookie_b);
3759 }
3760 swap(*id_a, *id_b);
3761}
3762
3763static int check_dup_ids(u32 *ids, u64 *cookies, u32 cnt)
3764{
3765 void *data[2] = { ids, cookies };
3766 int err = 0;
3767
3768 /*
3769 * Sort ids array (together with cookies array if defined)
3770 * and check it for duplicates. The ids and cookies arrays
3771 * are left sorted.
3772 */
3773 sort_r_nonatomic(ids, cnt, sizeof(ids[0]), ids_cmp_r, ids_swap_r, data);
3774
3775 for (int i = 1; i < cnt; i++) {
3776 if (ids[i] == ids[i - 1]) {
3777 err = -EINVAL;
3778 break;
3779 }
3780 }
3781 return err;
3782}
3783
3784int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
3785{
3786 struct bpf_tracing_multi_link *link = NULL;
3787 struct bpf_tramp_node *fexits = NULL;
3788 struct bpf_link_primer link_primer;
3789 u32 cnt, *ids = NULL;
3790 u64 __user *ucookies;
3791 u64 *cookies = NULL;
3792 u32 __user *uids;
3793 int err;
3794
3795 uids = u64_to_user_ptr(attr->link_create.tracing_multi.ids);
3796 cnt = attr->link_create.tracing_multi.cnt;
3797
3798 if (!cnt || !uids)
3799 return -EINVAL;
3800 if (cnt > MAX_TRACING_MULTI_CNT)
3801 return -E2BIG;
3802 if (attr->link_create.flags || attr->link_create.target_fd)
3803 return -EINVAL;
3804
3805 ids = kvmalloc_objs(*ids, cnt);
3806 if (!ids)
3807 return -ENOMEM;
3808
3809 if (copy_from_user(ids, uids, cnt * sizeof(*ids))) {
3810 err = -EFAULT;
3811 goto error;
3812 }
3813
3814 ucookies = u64_to_user_ptr(attr->link_create.tracing_multi.cookies);
3815 if (ucookies) {
3816 cookies = kvmalloc_objs(*cookies, cnt);
3817 if (!cookies) {
3818 err = -ENOMEM;
3819 goto error;
3820 }
3821 if (copy_from_user(cookies, ucookies, cnt * sizeof(*cookies))) {
3822 err = -EFAULT;
3823 goto error;
3824 }
3825 }
3826
3827 err = check_dup_ids(ids, cookies, cnt);
3828 if (err)
3829 goto error;
3830
3831 if (prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) {
3832 fexits = kvmalloc_objs(*fexits, cnt);
3833 if (!fexits) {
3834 err = -ENOMEM;
3835 goto error;
3836 }
3837 }
3838
3839 link = kvzalloc_flex(*link, nodes, cnt);
3840 if (!link) {
3841 err = -ENOMEM;
3842 goto error;
3843 }
3844
3845 bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING_MULTI,
3846 &bpf_tracing_multi_link_lops, prog, prog->expected_attach_type);
3847
3848 err = bpf_link_prime(&link->link, &link_primer);
3849 if (err)
3850 goto error;
3851
3852 link->nodes_cnt = cnt;
3853 link->cookies = cookies;
3854 link->fexits = fexits;
3855
3856 err = bpf_trampoline_multi_attach(prog, ids, link);
3857 kvfree(ids);
3858 if (err) {
3859 bpf_link_cleanup(&link_primer);
3860 return err;
3861 }
3862 return bpf_link_settle(&link_primer);
3863
3864error:
3865 kvfree(fexits);
3866 kvfree(cookies);
3867 kvfree(ids);
3868 kvfree(link);
3869 return err;
3870}
3871
3872#else
3873
3874int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
3875{
3876 return -EOPNOTSUPP;
3877}
3878
3879#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS && CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */
kernel/trace/ftrace.c+29−6
@@ -1198,8 +1198,7 @@ ftrace_lookup_ip(struct ftrace_hash *hash, unsigned long ip) · 1197 unmodified lines
1198 return __ftrace_lookup_ip(hash, ip);1198 return __ftrace_lookup_ip(hash, ip);
1199}1199}
1200 1200
1201static void __add_hash_entry(struct ftrace_hash *hash,1201void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry)
1202 struct ftrace_func_entry *entry)
1203{1202{
1204 struct hlist_head *hhd;1203 struct hlist_head *hhd;
1205 unsigned long key;1204 unsigned long key;
@@ -1221,7 +1220,7 @@ add_ftrace_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigne · 15 unmodified lines
1221 1220
1222 entry->ip = ip;1221 entry->ip = ip;
1223 entry->direct = direct;1222 entry->direct = direct;
1224 __add_hash_entry(hash, entry);1223 add_ftrace_hash_entry(hash, entry);
1225 1224
1226 return entry;1225 return entry;
1227}1226}
@@ -1249,6 +1248,25 @@ remove_hash_entry(struct ftrace_hash *hash, · 21 unmodified lines
1249 hash->count--;1248 hash->count--;
1250}1249}
1251 1250
1251void ftrace_hash_remove(struct ftrace_hash *hash)
1252{
1253 struct ftrace_func_entry *entry;
1254 struct hlist_head *hhd;
1255 struct hlist_node *tn;
1256 int size;
1257 int i;
1258
1259 if (!hash || !hash->count)
1260 return;
1261 size = 1 << hash->size_bits;
1262 for (i = 0; i < size; i++) {
1263 hhd = &hash->buckets[i];
1264 hlist_for_each_entry_safe(entry, tn, hhd, hlist)
1265 remove_hash_entry(hash, entry);
1266 }
1267 FTRACE_WARN_ON(hash->count);
1268}
1269
1252static void ftrace_hash_clear(struct ftrace_hash *hash)1270static void ftrace_hash_clear(struct ftrace_hash *hash)
1253{1271{
1254 struct hlist_head *hhd;1272 struct hlist_head *hhd;
@@ -1458,7 +1476,7 @@ static struct ftrace_hash *__move_hash(struct ftrace_hash *src, int size) · 203 unmodified lines
1458 hhd = &src->buckets[i];1476 hhd = &src->buckets[i];
1459 hlist_for_each_entry_safe(entry, tn, hhd, hlist) {1477 hlist_for_each_entry_safe(entry, tn, hhd, hlist) {
1460 remove_hash_entry(src, entry);1478 remove_hash_entry(src, entry);
1461 __add_hash_entry(new_hash, entry);1479 add_ftrace_hash_entry(new_hash, entry);
1462 }1480 }
1463 }1481 }
1464 return new_hash;1482 return new_hash;
@@ -5341,7 +5359,7 @@ int ftrace_func_mapper_add_ip(struct ftrace_func_mapper *mapper, · 3876 unmodified lines
5341 map->entry.ip = ip;5359 map->entry.ip = ip;
5342 map->data = data;5360 map->data = data;
5343 5361
5344 __add_hash_entry(&mapper->hash, &map->entry);5362 add_ftrace_hash_entry(&mapper->hash, &map->entry);
5345 5363
5346 return 0;5364 return 0;
5347}5365}
@@ -6288,11 +6306,16 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) · 940 unmodified lines
6288}6306}
6289EXPORT_SYMBOL_GPL(modify_ftrace_direct);6307EXPORT_SYMBOL_GPL(modify_ftrace_direct);
6290 6308
6291static unsigned long hash_count(struct ftrace_hash *hash)6309static inline unsigned long hash_count(struct ftrace_hash *hash)
6292{6310{
6293 return hash ? hash->count : 0;6311 return hash ? hash->count : 0;
6294}6312}
6295 6313
6314unsigned long ftrace_hash_count(struct ftrace_hash *hash)
6315{
6316 return hash_count(hash);
6317}
6318
6296/**6319/**
6297 * hash_add - adds two struct ftrace_hash and returns the result6320 * hash_add - adds two struct ftrace_hash and returns the result
6298 * @a: struct ftrace_hash object6321 * @a: struct ftrace_hash object
kernel/trace/trace_syscalls.c+58−52
@@ -1371,33 +1371,33 @@ static DECLARE_BITMAP(enabled_perf_exit_syscalls, NR_syscalls); · 1370 unmodified lines
1371static int sys_perf_refcount_enter;1371static int sys_perf_refcount_enter;
1372static int sys_perf_refcount_exit;1372static int sys_perf_refcount_exit;
1373 1373
1374static int perf_call_bpf_enter(struct trace_event_call *call, struct pt_regs *regs,1374static int perf_call_bpf_enter(struct trace_event_call *call,
1375 struct syscall_metadata *sys_data,1375 struct syscall_metadata *sys_data,
1376 struct syscall_trace_enter *rec)1376 int syscall_nr, unsigned long *args)
1377{1377{
1378 struct syscall_tp_t {1378 struct syscall_tp_t {
1379 struct trace_entry ent;1379 struct trace_entry ent;
1380 int syscall_nr;1380 int syscall_nr;
1381 unsigned long args[SYSCALL_DEFINE_MAXARGS];1381 unsigned long args[SYSCALL_DEFINE_MAXARGS];
1382 } __aligned(8) param;1382 } __aligned(8) param;
1383 struct pt_regs regs = {};
1383 int i;1384 int i;
1384 1385
1385 BUILD_BUG_ON(sizeof(param.ent) < sizeof(void *));1386 BUILD_BUG_ON(sizeof(param.ent) < sizeof(void *));
1386 1387
1387 /* bpf prog requires 'regs' to be the first member in the ctx (a.k.a. &param) */1388 /* bpf prog requires 'regs' to be the first member in the ctx */
1388 perf_fetch_caller_regs(regs);1389 perf_fetch_caller_regs(&regs);
1389 *(struct pt_regs **)&param = regs;1390 *(struct pt_regs **)&param = &regs;
1390 param.syscall_nr = rec->nr;1391 param.syscall_nr = syscall_nr;
1391 for (i = 0; i < sys_data->nb_args; i++)1392 for (i = 0; i < sys_data->nb_args; i++)
1392 param.args[i] = rec->args[i];1393 param.args[i] = args[i];
1393 return trace_call_bpf(call, &param);1394 return trace_call_bpf_faultable(call, &param);
1394}1395}
1395 1396
1396static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)1397static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)
1397{1398{
1398 struct syscall_metadata *sys_data;1399 struct syscall_metadata *sys_data;
1399 struct syscall_trace_enter *rec;1400 struct syscall_trace_enter *rec;
1400 struct pt_regs *fake_regs;
1401 struct hlist_head *head;1401 struct hlist_head *head;
1402 unsigned long args[6];1402 unsigned long args[6];
1403 bool valid_prog_array;1403 bool valid_prog_array;
@@ -1410,12 +1410,7 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 6 unmodified lines
1410 int size = 0;1410 int size = 0;
1411 int uargs = 0;1411 int uargs = 0;
1412 1412
1413 /*
1414 * Syscall probe called with preemption enabled, but the ring
1415 * buffer and per-cpu data require preemption to be disabled.
1416 */
1417 might_fault();1413 might_fault();
1418 guard(preempt_notrace)();
1419 1414
1420 syscall_nr = trace_get_syscall_nr(current, regs);1415 syscall_nr = trace_get_syscall_nr(current, regs);
1421 if (syscall_nr < 0 || syscall_nr >= NR_syscalls)1416 if (syscall_nr < 0 || syscall_nr >= NR_syscalls)
@@ -1429,6 +1424,26 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 7 unmodified lines
1429 1424
1430 syscall_get_arguments(current, regs, args);1425 syscall_get_arguments(current, regs, args);
1431 1426
1427 /*
1428 * Run BPF program in faultable context before per-cpu buffer
1429 * allocation, allowing sleepable BPF programs to execute.
1430 */
1431 valid_prog_array = bpf_prog_array_valid(sys_data->enter_event);
1432 if (valid_prog_array &&
1433 !perf_call_bpf_enter(sys_data->enter_event, sys_data,
1434 syscall_nr, args))
1435 return;
1436
1437 /*
1438 * Per-cpu ring buffer and perf event list operations require
1439 * preemption to be disabled.
1440 */
1441 guard(preempt_notrace)();
1442
1443 head = this_cpu_ptr(sys_data->enter_event->perf_events);
1444 if (hlist_empty(head))
1445 return;
1446
1432 /* Check if this syscall event faults in user space memory */1447 /* Check if this syscall event faults in user space memory */
1433 mayfault = sys_data->user_mask != 0;1448 mayfault = sys_data->user_mask != 0;
1434 1449
@@ -1438,17 +1453,12 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 3 unmodified lines
1438 return;1453 return;
1439 }1454 }
1440 1455
1441 head = this_cpu_ptr(sys_data->enter_event->perf_events);
1442 valid_prog_array = bpf_prog_array_valid(sys_data->enter_event);
1443 if (!valid_prog_array && hlist_empty(head))
1444 return;
1445
1446 /* get the size after alignment with the u32 buffer size field */1456 /* get the size after alignment with the u32 buffer size field */
1447 size += sizeof(unsigned long) * sys_data->nb_args + sizeof(*rec);1457 size += sizeof(unsigned long) * sys_data->nb_args + sizeof(*rec);
1448 size = ALIGN(size + sizeof(u32), sizeof(u64));1458 size = ALIGN(size + sizeof(u32), sizeof(u64));
1449 size -= sizeof(u32);1459 size -= sizeof(u32);
1450 1460
1451 rec = perf_trace_buf_alloc(size, &fake_regs, &rctx);1461 rec = perf_trace_buf_alloc(size, NULL, &rctx);
1452 if (!rec)1462 if (!rec)
1453 return;1463 return;
1454 1464
@@ -1458,13 +1468,6 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 3 unmodified lines
1458 if (mayfault)1468 if (mayfault)
1459 syscall_put_data(sys_data, rec, user_ptr, size, user_sizes, uargs);1469 syscall_put_data(sys_data, rec, user_ptr, size, user_sizes, uargs);
1460 1470
1461 if ((valid_prog_array &&
1462 !perf_call_bpf_enter(sys_data->enter_event, fake_regs, sys_data, rec)) ||
1463 hlist_empty(head)) {
1464 perf_swevent_put_recursion_context(rctx);
1465 return;
1466 }
1467
1468 perf_trace_buf_submit(rec, size, rctx,1471 perf_trace_buf_submit(rec, size, rctx,
1469 sys_data->enter_event->event.type, 1, regs,1472 sys_data->enter_event->event.type, 1, regs,
1470 head, NULL);1473 head, NULL);
@@ -1514,40 +1517,35 @@ static void perf_sysenter_disable(struct trace_event_call *call) · 43 unmodified lines
1514 syscall_fault_buffer_disable();1517 syscall_fault_buffer_disable();
1515}1518}
1516 1519
1517static int perf_call_bpf_exit(struct trace_event_call *call, struct pt_regs *regs,1520static int perf_call_bpf_exit(struct trace_event_call *call,
1518 struct syscall_trace_exit *rec)1521 int syscall_nr, long ret_val)
1519{1522{
1520 struct syscall_tp_t {1523 struct syscall_tp_t {
1521 struct trace_entry ent;1524 struct trace_entry ent;
1522 int syscall_nr;1525 int syscall_nr;
1523 unsigned long ret;1526 unsigned long ret;
1524 } __aligned(8) param;1527 } __aligned(8) param;
1525 1528 struct pt_regs regs = {};
1526 /* bpf prog requires 'regs' to be the first member in the ctx (a.k.a. &param) */1529
1527 perf_fetch_caller_regs(regs);1530 /* bpf prog requires 'regs' to be the first member in the ctx */
1528 *(struct pt_regs **)&param = regs;1531 perf_fetch_caller_regs(&regs);
1529 param.syscall_nr = rec->nr;1532 *(struct pt_regs **)&param = &regs;
1530 param.ret = rec->ret;1533 param.syscall_nr = syscall_nr;
1531 return trace_call_bpf(call, &param);1534 param.ret = ret_val;
1535 return trace_call_bpf_faultable(call, &param);
1532}1536}
1533 1537
1534static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret)1538static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret)
1535{1539{
1536 struct syscall_metadata *sys_data;1540 struct syscall_metadata *sys_data;
1537 struct syscall_trace_exit *rec;1541 struct syscall_trace_exit *rec;
1538 struct pt_regs *fake_regs;
1539 struct hlist_head *head;1542 struct hlist_head *head;
1540 bool valid_prog_array;1543 bool valid_prog_array;
1541 int syscall_nr;1544 int syscall_nr;
1542 int rctx;1545 int rctx;
1543 int size;1546 int size;
1544 1547
1545 /*
1546 * Syscall probe called with preemption enabled, but the ring
1547 * buffer and per-cpu data require preemption to be disabled.
1548 */
1549 might_fault();1548 might_fault();
1550 guard(preempt_notrace)();
1551 1549
1552 syscall_nr = trace_get_syscall_nr(current, regs);1550 syscall_nr = trace_get_syscall_nr(current, regs);
1553 if (syscall_nr < 0 || syscall_nr >= NR_syscalls)1551 if (syscall_nr < 0 || syscall_nr >= NR_syscalls)
@@ -1559,29 +1557,37 @@ static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) · 5 unmodified lines
1559 if (!sys_data)1557 if (!sys_data)
1560 return;1558 return;
1561 1559
1562 head = this_cpu_ptr(sys_data->exit_event->perf_events);1560 /*
1561 * Run BPF program in faultable context before per-cpu buffer
1562 * allocation, allowing sleepable BPF programs to execute.
1563 */
1563 valid_prog_array = bpf_prog_array_valid(sys_data->exit_event);1564 valid_prog_array = bpf_prog_array_valid(sys_data->exit_event);
1564 if (!valid_prog_array && hlist_empty(head))1565 if (valid_prog_array &&
1566 !perf_call_bpf_exit(sys_data->exit_event, syscall_nr,
1567 syscall_get_return_value(current, regs)))
1568 return;
1569
1570 /*
1571 * Per-cpu ring buffer and perf event list operations require
1572 * preemption to be disabled.
1573 */
1574 guard(preempt_notrace)();
1575
1576 head = this_cpu_ptr(sys_data->exit_event->perf_events);
1577 if (hlist_empty(head))
1565 return;1578 return;
1566 1579
1567 /* We can probably do that at build time */1580 /* We can probably do that at build time */
1568 size = ALIGN(sizeof(*rec) + sizeof(u32), sizeof(u64));1581 size = ALIGN(sizeof(*rec) + sizeof(u32), sizeof(u64));
1569 size -= sizeof(u32);1582 size -= sizeof(u32);
1570 1583
1571 rec = perf_trace_buf_alloc(size, &fake_regs, &rctx);1584 rec = perf_trace_buf_alloc(size, NULL, &rctx);
1572 if (!rec)1585 if (!rec)
1573 return;1586 return;
1574 1587
1575 rec->nr = syscall_nr;1588 rec->nr = syscall_nr;
1576 rec->ret = syscall_get_return_value(current, regs);1589 rec->ret = syscall_get_return_value(current, regs);
1577 1590
1578 if ((valid_prog_array &&
1579 !perf_call_bpf_exit(sys_data->exit_event, fake_regs, rec)) ||
1580 hlist_empty(head)) {
1581 perf_swevent_put_recursion_context(rctx);
1582 return;
1583 }
1584
1585 perf_trace_buf_submit(rec, size, rctx, sys_data->exit_event->event.type,1591 perf_trace_buf_submit(rec, size, rctx, sys_data->exit_event->event.type,
1586 1, regs, head, NULL);1592 1, regs, head, NULL);
1587}1593}
lib/rhashtable.c+102−0
@@ -687,6 +687,108 @@ void *rhashtable_insert_slow(struct rhashtable *ht, const void *key, · 686 unmodified lines
687}687}
688EXPORT_SYMBOL_GPL(rhashtable_insert_slow);688EXPORT_SYMBOL_GPL(rhashtable_insert_slow);
689 689
690/* Scan one element forward from prev_key's position in @tbl.
691 * Returns first rhash_head whose bucket > prev_key's bucket, or the
692 * element immediately after prev_key inside prev_key's bucket.
693 * Returns the first element if prev_key is NULL, NULL when @tbl is
694 * exhausted, or ERR_PTR(-ENOENT) if prev_key is not found in @tbl.
695 */
696static struct rhash_head *__rhashtable_next_in_table(
697 struct rhashtable *ht, struct bucket_table *tbl,
698 const void *prev_key)
699{
700 struct rhashtable_compare_arg arg = { .ht = ht, .key = prev_key };
701 const struct rhashtable_params params = ht->p;
702 struct rhash_head *he;
703 unsigned int b = 0;
704 bool found = false;
705
706 if (prev_key) {
707 b = rht_key_hashfn(ht, tbl, prev_key, params);
708 rht_for_each_rcu(he, tbl, b) {
709 bool match = params.obj_cmpfn
710 ? !params.obj_cmpfn(&arg, rht_obj(ht, he))
711 : !rhashtable_compare(&arg, rht_obj(ht, he));
712 if (found) {
713 if (match)
714 continue;
715 return he;
716 }
717 if (match)
718 found = true;
719 }
720 if (!found)
721 return ERR_PTR(-ENOENT);
722 b++;
723 }
724
725 for (; b < tbl->size; b++)
726 rht_for_each_rcu(he, tbl, b)
727 return he;
728 return NULL;
729}
730
731/**
732 * rhashtable_next_key - return next element after a given key
733 * @ht: hash table
734 * @prev_key: pointer to previous key, or NULL for the first element
735 *
736 * WARNING: this walk is highly unstable. Unlike rhashtable_walk_*(),
737 * it cannot detect a concurrent resize or rehash, so a full iteration
738 * is NOT guaranteed to terminate under adversarial or sustained
739 * rehashing. Callers MUST tolerate skipped and duplicated elements and
740 * SHOULD bound their loop externally.
741 *
742 * Returns the next element in best-effort iteration order, walking the
743 * @tbl chain (including any future_tbl in flight). Caller must hold RCU.
744 *
745 * Pass @prev_key == NULL to obtain the first element. To iterate, set
746 * @prev_key to the key of the previously returned element on each call,
747 * and stop when NULL is returned.
748 *
749 * Best-effort semantics:
750 * - Across the tbl->future_tbl chain, an element being migrated may
751 * transiently appear in both tables and be observed twice.
752 * - Concurrent inserts may or may not be observed.
753 * - Termination of a full iteration loop is NOT guaranteed under
754 * adversarial continuous rehash; callers MUST tolerate skips and
755 * repeats and SHOULD bound their loop externally.
756 * - Behavior on tables that contain duplicate keys is undefined:
757 * duplicates may be skipped, repeated, or trap the walk in a
758 * cycle. Callers requiring duplicate-key iteration must use
759 * rhashtable_walk_*() instead.
760 * - rhltable instances are not supported and return
761 * ERR_PTR(-EOPNOTSUPP).
762 * - If prev_key was concurrently deleted and is not present in any
763 * in-flight table, returns ERR_PTR(-ENOENT).
764 *
765 * Returns entry of the next element, or NULL when iteration is exhausted,
766 * or ERR_PTR(-ENOENT) if prev_key is not found, or
767 * ERR_PTR(-EOPNOTSUPP) if @ht is an rhltable.
768 */
769void *rhashtable_next_key(struct rhashtable *ht, const void *prev_key)
770{
771 struct bucket_table *tbl;
772 struct rhash_head *he;
773
774 if (unlikely(ht->rhlist))
775 return ERR_PTR(-EOPNOTSUPP);
776
777 tbl = rht_dereference_rcu(ht->tbl, ht);
778 do {
779 he = __rhashtable_next_in_table(ht, tbl, prev_key);
780 if (!IS_ERR_OR_NULL(he))
781 return rht_obj(ht, he);
782 if (!he)
783 prev_key = NULL;
784 /* See any new future_tbl attached during a rehash. */
785 smp_rmb();
786 tbl = rht_dereference_rcu(tbl->future_tbl, ht);
787 } while (tbl);
788 return he; /* NULL or -ENOENT */
789}
790EXPORT_SYMBOL_GPL(rhashtable_next_key);
791
690/**792/**
691 * rhashtable_walk_enter - Initialise an iterator793 * rhashtable_walk_enter - Initialise an iterator
692 * @ht: Table to walk over794 * @ht: Table to walk over
lib/test_bpf.c+263−100
@@ -560,8 +560,23 @@ static int bpf_fill_max_jmp_never_taken(struct bpf_test *self) · 559 unmodified lines
560}560}
561 561
562/* ALU result computation used in tests */562/* ALU result computation used in tests */
563static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op)563enum { F_ALU32 = 1, F_SIGNED = 2 };
564
565static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op, u32 flags)
564{566{
567 bool is_signed = flags & F_SIGNED;
568
569 /* Narrow operands for ALU32 */
570 if (flags & F_ALU32) {
571 if (is_signed) {
572 v1 = (u64)(s32)v1;
573 v2 = (u64)(s32)v2;
574 } else {
575 v1 = (u32)v1;
576 v2 = (u32)v2;
577 }
578 }
579
565 *res = 0;580 *res = 0;
566 switch (op) {581 switch (op) {
567 case BPF_MOV:582 case BPF_MOV:
@@ -599,12 +614,28 @@ static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op) · 31 unmodified lines
599 case BPF_DIV:614 case BPF_DIV:
600 if (v2 == 0)615 if (v2 == 0)
601 return false;616 return false;
602 *res = div64_u64(v1, v2);617 if (!is_signed) {
618 *res = div64_u64(v1, v2);
619 } else {
620 if ((s64)v2 == -1) /* Handled by verifier */
621 return false;
622 *res = (u64)div64_s64(v1, v2);
623 }
603 break;624 break;
604 case BPF_MOD:625 case BPF_MOD:
605 if (v2 == 0)626 if (v2 == 0)
606 return false;627 return false;
607 div64_u64_rem(v1, v2, res);628 if (!is_signed) {
629 div64_u64_rem(v1, v2, res);
630 } else {
631 if ((s64)v2 == -1)
632 return false;
633 /*
634 * Avoid s64 % s64 which generates __moddi3 on
635 * 32-bit architectures. Use div64_s64 instead.
636 */
637 *res = (u64)((s64)v1 - div64_s64(v1, v2) * (s64)v2);
638 }
608 break;639 break;
609 }640 }
610 return true;641 return true;
@@ -612,7 +643,7 @@ static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op) · 1 unmodified line
612 643
613/* Test an ALU shift operation for all valid shift values */644/* Test an ALU shift operation for all valid shift values */
614static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op,645static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op,
615 u8 mode, bool alu32)646 u8 mode, u32 flags)
616{647{
617 static const s64 regs[] = {648 static const s64 regs[] = {
618 0x0123456789abcdefLL, /* dword > 0, word < 0 */649 0x0123456789abcdefLL, /* dword > 0, word < 0 */
@@ -620,7 +651,7 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 1 unmodified line
620 0xfedcba0198765432LL, /* dword < 0, word < 0 */651 0xfedcba0198765432LL, /* dword < 0, word < 0 */
621 0x0123458967abcdefLL, /* dword > 0, word > 0 */652 0x0123458967abcdefLL, /* dword > 0, word > 0 */
622 };653 };
623 int bits = alu32 ? 32 : 64;654 int bits = (flags & F_ALU32) ? 32 : 64;
624 int len = (2 + 7 * bits) * ARRAY_SIZE(regs) + 3;655 int len = (2 + 7 * bits) * ARRAY_SIZE(regs) + 3;
625 struct bpf_insn *insn;656 struct bpf_insn *insn;
626 int imm, k;657 int imm, k;
@@ -643,7 +674,7 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 16 unmodified lines
643 /* Perform operation */674 /* Perform operation */
644 insn[i++] = BPF_ALU64_REG(BPF_MOV, R1, R3);675 insn[i++] = BPF_ALU64_REG(BPF_MOV, R1, R3);
645 insn[i++] = BPF_ALU64_IMM(BPF_MOV, R2, imm);676 insn[i++] = BPF_ALU64_IMM(BPF_MOV, R2, imm);
646 if (alu32) {677 if (flags & F_ALU32) {
647 if (mode == BPF_K)678 if (mode == BPF_K)
648 insn[i++] = BPF_ALU32_IMM(op, R1, imm);679 insn[i++] = BPF_ALU32_IMM(op, R1, imm);
649 else680 else
@@ -653,14 +684,14 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 3 unmodified lines
653 reg = (s32)reg;684 reg = (s32)reg;
654 else685 else
655 reg = (u32)reg;686 reg = (u32)reg;
656 __bpf_alu_result(&val, reg, imm, op);687 __bpf_alu_result(&val, reg, imm, op, 0);
657 val = (u32)val;688 val = (u32)val;
658 } else {689 } else {
659 if (mode == BPF_K)690 if (mode == BPF_K)
660 insn[i++] = BPF_ALU64_IMM(op, R1, imm);691 insn[i++] = BPF_ALU64_IMM(op, R1, imm);
661 else692 else
662 insn[i++] = BPF_ALU64_REG(op, R1, R2);693 insn[i++] = BPF_ALU64_REG(op, R1, R2);
663 __bpf_alu_result(&val, reg, imm, op);694 __bpf_alu_result(&val, reg, imm, op, 0);
664 }695 }
665 696
666 /*697 /*
@@ -688,62 +719,62 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 21 unmodified lines
688 719
689static int bpf_fill_alu64_lsh_imm(struct bpf_test *self)720static int bpf_fill_alu64_lsh_imm(struct bpf_test *self)
690{721{
691 return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, false);722 return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, 0);
692}723}
693 724
694static int bpf_fill_alu64_rsh_imm(struct bpf_test *self)725static int bpf_fill_alu64_rsh_imm(struct bpf_test *self)
695{726{
696 return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, false);727 return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, 0);
697}728}
698 729
699static int bpf_fill_alu64_arsh_imm(struct bpf_test *self)730static int bpf_fill_alu64_arsh_imm(struct bpf_test *self)
700{731{
701 return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, false);732 return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, 0);
702}733}
703 734
704static int bpf_fill_alu64_lsh_reg(struct bpf_test *self)735static int bpf_fill_alu64_lsh_reg(struct bpf_test *self)
705{736{
706 return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, false);737 return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, 0);
707}738}
708 739
709static int bpf_fill_alu64_rsh_reg(struct bpf_test *self)740static int bpf_fill_alu64_rsh_reg(struct bpf_test *self)
710{741{
711 return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, false);742 return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, 0);
712}743}
713 744
714static int bpf_fill_alu64_arsh_reg(struct bpf_test *self)745static int bpf_fill_alu64_arsh_reg(struct bpf_test *self)
715{746{
716 return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, false);747 return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, 0);
717}748}
718 749
719static int bpf_fill_alu32_lsh_imm(struct bpf_test *self)750static int bpf_fill_alu32_lsh_imm(struct bpf_test *self)
720{751{
721 return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, true);752 return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, F_ALU32);
722}753}
723 754
724static int bpf_fill_alu32_rsh_imm(struct bpf_test *self)755static int bpf_fill_alu32_rsh_imm(struct bpf_test *self)
725{756{
726 return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, true);757 return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, F_ALU32);
727}758}
728 759
729static int bpf_fill_alu32_arsh_imm(struct bpf_test *self)760static int bpf_fill_alu32_arsh_imm(struct bpf_test *self)
730{761{
731 return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, true);762 return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, F_ALU32);
732}763}
733 764
734static int bpf_fill_alu32_lsh_reg(struct bpf_test *self)765static int bpf_fill_alu32_lsh_reg(struct bpf_test *self)
735{766{
736 return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, true);767 return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, F_ALU32);
737}768}
738 769
739static int bpf_fill_alu32_rsh_reg(struct bpf_test *self)770static int bpf_fill_alu32_rsh_reg(struct bpf_test *self)
740{771{
741 return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, true);772 return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, F_ALU32);
742}773}
743 774
744static int bpf_fill_alu32_arsh_reg(struct bpf_test *self)775static int bpf_fill_alu32_arsh_reg(struct bpf_test *self)
745{776{
746 return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, true);777 return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, F_ALU32);
747}778}
748 779
749/*780/*
@@ -751,9 +782,9 @@ static int bpf_fill_alu32_arsh_reg(struct bpf_test *self) · 1 unmodified line
751 * for the case when the source and destination are the same.782 * for the case when the source and destination are the same.
752 */783 */
753static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op,784static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op,
754 bool alu32)785 u32 flags)
755{786{
756 int bits = alu32 ? 32 : 64;787 int bits = (flags & F_ALU32) ? 32 : 64;
757 int len = 3 + 6 * bits;788 int len = 3 + 6 * bits;
758 struct bpf_insn *insn;789 struct bpf_insn *insn;
759 int i = 0;790 int i = 0;
@@ -770,14 +801,14 @@ static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op, · 10 unmodified lines
770 801
771 /* Perform operation */802 /* Perform operation */
772 insn[i++] = BPF_ALU64_IMM(BPF_MOV, R1, val);803 insn[i++] = BPF_ALU64_IMM(BPF_MOV, R1, val);
773 if (alu32)804 if (flags & F_ALU32)
774 insn[i++] = BPF_ALU32_REG(op, R1, R1);805 insn[i++] = BPF_ALU32_REG(op, R1, R1);
775 else806 else
776 insn[i++] = BPF_ALU64_REG(op, R1, R1);807 insn[i++] = BPF_ALU64_REG(op, R1, R1);
777 808
778 /* Compute the reference result */809 /* Compute the reference result */
779 __bpf_alu_result(&res, val, val, op);810 __bpf_alu_result(&res, val, val, op, 0);
780 if (alu32)811 if (flags & F_ALU32)
781 res = (u32)res;812 res = (u32)res;
782 i += __bpf_ld_imm64(&insn[i], R2, res);813 i += __bpf_ld_imm64(&insn[i], R2, res);
783 814
@@ -798,32 +829,32 @@ static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op, · 14 unmodified lines
798 829
799static int bpf_fill_alu64_lsh_same_reg(struct bpf_test *self)830static int bpf_fill_alu64_lsh_same_reg(struct bpf_test *self)
800{831{
801 return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, false);832 return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, 0);
802}833}
803 834
804static int bpf_fill_alu64_rsh_same_reg(struct bpf_test *self)835static int bpf_fill_alu64_rsh_same_reg(struct bpf_test *self)
805{836{
806 return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, false);837 return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, 0);
807}838}
808 839
809static int bpf_fill_alu64_arsh_same_reg(struct bpf_test *self)840static int bpf_fill_alu64_arsh_same_reg(struct bpf_test *self)
810{841{
811 return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, false);842 return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, 0);
812}843}
813 844
814static int bpf_fill_alu32_lsh_same_reg(struct bpf_test *self)845static int bpf_fill_alu32_lsh_same_reg(struct bpf_test *self)
815{846{
816 return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, true);847 return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, F_ALU32);
817}848}
818 849
819static int bpf_fill_alu32_rsh_same_reg(struct bpf_test *self)850static int bpf_fill_alu32_rsh_same_reg(struct bpf_test *self)
820{851{
821 return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, true);852 return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, F_ALU32);
822}853}
823 854
824static int bpf_fill_alu32_arsh_same_reg(struct bpf_test *self)855static int bpf_fill_alu32_arsh_same_reg(struct bpf_test *self)
825{856{
826 return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, true);857 return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, F_ALU32);
827}858}
828 859
829/*860/*
@@ -936,17 +967,20 @@ static int __bpf_fill_pattern(struct bpf_test *self, void *arg, · 106 unmodified lines
936static int __bpf_emit_alu64_imm(struct bpf_test *self, void *arg,967static int __bpf_emit_alu64_imm(struct bpf_test *self, void *arg,
937 struct bpf_insn *insns, s64 dst, s64 imm)968 struct bpf_insn *insns, s64 dst, s64 imm)
938{969{
939 int op = *(int *)arg;970 int *a = arg;
971 int op = a[0];
972 u32 flags = a[1];
973 s16 off = (flags & F_SIGNED) ? 1 : 0;
940 int i = 0;974 int i = 0;
941 u64 res;975 u64 res;
942 976
943 if (!insns)977 if (!insns)
944 return 7;978 return 7;
945 979
946 if (__bpf_alu_result(&res, dst, (s32)imm, op)) {980 if (__bpf_alu_result(&res, dst, (s32)imm, op, flags)) {
947 i += __bpf_ld_imm64(&insns[i], R1, dst);981 i += __bpf_ld_imm64(&insns[i], R1, dst);
948 i += __bpf_ld_imm64(&insns[i], R3, res);982 i += __bpf_ld_imm64(&insns[i], R3, res);
949 insns[i++] = BPF_ALU64_IMM(op, R1, imm);983 insns[i++] = BPF_ALU64_IMM_OFF(op, R1, imm, off);
950 insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1);984 insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1);
951 insns[i++] = BPF_EXIT_INSN();985 insns[i++] = BPF_EXIT_INSN();
952 }986 }
@@ -957,17 +991,20 @@ static int __bpf_emit_alu64_imm(struct bpf_test *self, void *arg, · 4 unmodified lines
957static int __bpf_emit_alu32_imm(struct bpf_test *self, void *arg,991static int __bpf_emit_alu32_imm(struct bpf_test *self, void *arg,
958 struct bpf_insn *insns, s64 dst, s64 imm)992 struct bpf_insn *insns, s64 dst, s64 imm)
959{993{
960 int op = *(int *)arg;994 int *a = arg;
995 int op = a[0];
996 u32 flags = a[1];
997 s16 off = (flags & F_SIGNED) ? 1 : 0;
961 int i = 0;998 int i = 0;
962 u64 res;999 u64 res;
963 1000
964 if (!insns)1001 if (!insns)
965 return 7;1002 return 7;
966 1003
967 if (__bpf_alu_result(&res, (u32)dst, (u32)imm, op)) {1004 if (__bpf_alu_result(&res, dst, (s32)imm, op, flags | F_ALU32)) {
968 i += __bpf_ld_imm64(&insns[i], R1, dst);1005 i += __bpf_ld_imm64(&insns[i], R1, dst);
969 i += __bpf_ld_imm64(&insns[i], R3, (u32)res);1006 i += __bpf_ld_imm64(&insns[i], R3, (u32)res);
970 insns[i++] = BPF_ALU32_IMM(op, R1, imm);1007 insns[i++] = BPF_ALU32_IMM_OFF(op, R1, imm, off);
971 insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1);1008 insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1);
972 insns[i++] = BPF_EXIT_INSN();1009 insns[i++] = BPF_EXIT_INSN();
973 }1010 }
@@ -985,7 +1022,7 @@ static int __bpf_emit_alu64_reg(struct bpf_test *self, void *arg, · 11 unmodified lines
985 if (!insns)1022 if (!insns)
986 return 9;1023 return 9;
987 1024
988 if (__bpf_alu_result(&res, dst, src, op)) {1025 if (__bpf_alu_result(&res, dst, src, op, 0)) {
989 i += __bpf_ld_imm64(&insns[i], R1, dst);1026 i += __bpf_ld_imm64(&insns[i], R1, dst);
990 i += __bpf_ld_imm64(&insns[i], R2, src);1027 i += __bpf_ld_imm64(&insns[i], R2, src);
991 i += __bpf_ld_imm64(&insns[i], R3, res);1028 i += __bpf_ld_imm64(&insns[i], R3, res);
@@ -1007,7 +1044,7 @@ static int __bpf_emit_alu32_reg(struct bpf_test *self, void *arg, · 15 unmodified lines
1007 if (!insns)1044 if (!insns)
1008 return 9;1045 return 9;
1009 1046
1010 if (__bpf_alu_result(&res, (u32)dst, (u32)src, op)) {1047 if (__bpf_alu_result(&res, (u32)dst, (u32)src, op, 0)) {
1011 i += __bpf_ld_imm64(&insns[i], R1, dst);1048 i += __bpf_ld_imm64(&insns[i], R1, dst);
1012 i += __bpf_ld_imm64(&insns[i], R2, src);1049 i += __bpf_ld_imm64(&insns[i], R2, src);
1013 i += __bpf_ld_imm64(&insns[i], R3, (u32)res);1050 i += __bpf_ld_imm64(&insns[i], R3, (u32)res);
@@ -1019,16 +1056,20 @@ static int __bpf_emit_alu32_reg(struct bpf_test *self, void *arg, · 5 unmodified lines
1019 return i;1056 return i;
1020}1057}
1021 1058
1022static int __bpf_fill_alu64_imm(struct bpf_test *self, int op)1059static int __bpf_fill_alu64_imm(struct bpf_test *self, int op, u32 flags)
1023{1060{
1024 return __bpf_fill_pattern(self, &op, 64, 32,1061 int arg[2] = {op, flags};
1062
1063 return __bpf_fill_pattern(self, &arg, 64, 32,
1025 PATTERN_BLOCK1, PATTERN_BLOCK2,1064 PATTERN_BLOCK1, PATTERN_BLOCK2,
1026 &__bpf_emit_alu64_imm);1065 &__bpf_emit_alu64_imm);
1027}1066}
1028 1067
1029static int __bpf_fill_alu32_imm(struct bpf_test *self, int op)1068static int __bpf_fill_alu32_imm(struct bpf_test *self, int op, u32 flags)
1030{1069{
1031 return __bpf_fill_pattern(self, &op, 64, 32,1070 int arg[2] = {op, flags};
1071
1072 return __bpf_fill_pattern(self, &arg, 64, 32,
1032 PATTERN_BLOCK1, PATTERN_BLOCK2,1073 PATTERN_BLOCK1, PATTERN_BLOCK2,
1033 &__bpf_emit_alu32_imm);1074 &__bpf_emit_alu32_imm);
1034}1075}
@@ -1050,93 +1091,115 @@ static int __bpf_fill_alu32_reg(struct bpf_test *self, int op) · 15 unmodified lines
1050/* ALU64 immediate operations */1091/* ALU64 immediate operations */
1051static int bpf_fill_alu64_mov_imm(struct bpf_test *self)1092static int bpf_fill_alu64_mov_imm(struct bpf_test *self)
1052{1093{
1053 return __bpf_fill_alu64_imm(self, BPF_MOV);1094 return __bpf_fill_alu64_imm(self, BPF_MOV, 0);
1054}1095}
1055 1096
1056static int bpf_fill_alu64_and_imm(struct bpf_test *self)1097static int bpf_fill_alu64_and_imm(struct bpf_test *self)
1057{1098{
1058 return __bpf_fill_alu64_imm(self, BPF_AND);1099 return __bpf_fill_alu64_imm(self, BPF_AND, 0);
1059}1100}
1060 1101
1061static int bpf_fill_alu64_or_imm(struct bpf_test *self)1102static int bpf_fill_alu64_or_imm(struct bpf_test *self)
1062{1103{
1063 return __bpf_fill_alu64_imm(self, BPF_OR);1104 return __bpf_fill_alu64_imm(self, BPF_OR, 0);
1064}1105}
1065 1106
1066static int bpf_fill_alu64_xor_imm(struct bpf_test *self)1107static int bpf_fill_alu64_xor_imm(struct bpf_test *self)
1067{1108{
1068 return __bpf_fill_alu64_imm(self, BPF_XOR);1109 return __bpf_fill_alu64_imm(self, BPF_XOR, 0);
1069}1110}
1070 1111
1071static int bpf_fill_alu64_add_imm(struct bpf_test *self)1112static int bpf_fill_alu64_add_imm(struct bpf_test *self)
1072{1113{
1073 return __bpf_fill_alu64_imm(self, BPF_ADD);1114 return __bpf_fill_alu64_imm(self, BPF_ADD, 0);
1074}1115}
1075 1116
1076static int bpf_fill_alu64_sub_imm(struct bpf_test *self)1117static int bpf_fill_alu64_sub_imm(struct bpf_test *self)
1077{1118{
1078 return __bpf_fill_alu64_imm(self, BPF_SUB);1119 return __bpf_fill_alu64_imm(self, BPF_SUB, 0);
1079}1120}
1080 1121
1081static int bpf_fill_alu64_mul_imm(struct bpf_test *self)1122static int bpf_fill_alu64_mul_imm(struct bpf_test *self)
1082{1123{
1083 return __bpf_fill_alu64_imm(self, BPF_MUL);1124 return __bpf_fill_alu64_imm(self, BPF_MUL, 0);
1084}1125}
1085 1126
1086static int bpf_fill_alu64_div_imm(struct bpf_test *self)1127static int bpf_fill_alu64_div_imm(struct bpf_test *self)
1087{1128{
1088 return __bpf_fill_alu64_imm(self, BPF_DIV);1129 return __bpf_fill_alu64_imm(self, BPF_DIV, 0);
1089}1130}
1090 1131
1091static int bpf_fill_alu64_mod_imm(struct bpf_test *self)1132static int bpf_fill_alu64_mod_imm(struct bpf_test *self)
1092{1133{
1093 return __bpf_fill_alu64_imm(self, BPF_MOD);1134 return __bpf_fill_alu64_imm(self, BPF_MOD, 0);
1135}
1136
1137/* Signed ALU64 immediate operations */
1138static int bpf_fill_alu64_sdiv_imm(struct bpf_test *self)
1139{
1140 return __bpf_fill_alu64_imm(self, BPF_DIV, F_SIGNED);
1141}
1142
1143static int bpf_fill_alu64_smod_imm(struct bpf_test *self)
1144{
1145 return __bpf_fill_alu64_imm(self, BPF_MOD, F_SIGNED);
1146}
1147
1148/* Signed ALU32 immediate operations */
1149static int bpf_fill_alu32_sdiv_imm(struct bpf_test *self)
1150{
1151 return __bpf_fill_alu32_imm(self, BPF_DIV, F_SIGNED);
1152}
1153
1154static int bpf_fill_alu32_smod_imm(struct bpf_test *self)
1155{
1156 return __bpf_fill_alu32_imm(self, BPF_MOD, F_SIGNED);
1094}1157}
1095 1158
1096/* ALU32 immediate operations */1159/* ALU32 immediate operations */
1097static int bpf_fill_alu32_mov_imm(struct bpf_test *self)1160static int bpf_fill_alu32_mov_imm(struct bpf_test *self)
1098{1161{
1099 return __bpf_fill_alu32_imm(self, BPF_MOV);1162 return __bpf_fill_alu32_imm(self, BPF_MOV, 0);
1100}1163}
1101 1164
1102static int bpf_fill_alu32_and_imm(struct bpf_test *self)1165static int bpf_fill_alu32_and_imm(struct bpf_test *self)
1103{1166{
1104 return __bpf_fill_alu32_imm(self, BPF_AND);1167 return __bpf_fill_alu32_imm(self, BPF_AND, 0);
1105}1168}
1106 1169
1107static int bpf_fill_alu32_or_imm(struct bpf_test *self)1170static int bpf_fill_alu32_or_imm(struct bpf_test *self)
1108{1171{
1109 return __bpf_fill_alu32_imm(self, BPF_OR);1172 return __bpf_fill_alu32_imm(self, BPF_OR, 0);
1110}1173}
1111 1174
1112static int bpf_fill_alu32_xor_imm(struct bpf_test *self)1175static int bpf_fill_alu32_xor_imm(struct bpf_test *self)
1113{1176{
1114 return __bpf_fill_alu32_imm(self, BPF_XOR);1177 return __bpf_fill_alu32_imm(self, BPF_XOR, 0);
1115}1178}
1116 1179
1117static int bpf_fill_alu32_add_imm(struct bpf_test *self)1180static int bpf_fill_alu32_add_imm(struct bpf_test *self)
1118{1181{
1119 return __bpf_fill_alu32_imm(self, BPF_ADD);1182 return __bpf_fill_alu32_imm(self, BPF_ADD, 0);
1120}1183}
1121 1184
1122static int bpf_fill_alu32_sub_imm(struct bpf_test *self)1185static int bpf_fill_alu32_sub_imm(struct bpf_test *self)
1123{1186{
1124 return __bpf_fill_alu32_imm(self, BPF_SUB);1187 return __bpf_fill_alu32_imm(self, BPF_SUB, 0);
1125}1188}
1126 1189
1127static int bpf_fill_alu32_mul_imm(struct bpf_test *self)1190static int bpf_fill_alu32_mul_imm(struct bpf_test *self)
1128{1191{
1129 return __bpf_fill_alu32_imm(self, BPF_MUL);1192 return __bpf_fill_alu32_imm(self, BPF_MUL, 0);
1130}1193}
1131 1194
1132static int bpf_fill_alu32_div_imm(struct bpf_test *self)1195static int bpf_fill_alu32_div_imm(struct bpf_test *self)
1133{1196{
1134 return __bpf_fill_alu32_imm(self, BPF_DIV);1197 return __bpf_fill_alu32_imm(self, BPF_DIV, 0);
1135}1198}
1136 1199
1137static int bpf_fill_alu32_mod_imm(struct bpf_test *self)1200static int bpf_fill_alu32_mod_imm(struct bpf_test *self)
1138{1201{
1139 return __bpf_fill_alu32_imm(self, BPF_MOD);1202 return __bpf_fill_alu32_imm(self, BPF_MOD, 0);
1140}1203}
1141 1204
1142/* ALU64 register operations */1205/* ALU64 register operations */
@@ -1235,7 +1298,8 @@ static int bpf_fill_alu32_mod_reg(struct bpf_test *self) · 92 unmodified lines
1235 * Test JITs that implement complex ALU operations as function1298 * Test JITs that implement complex ALU operations as function
1236 * calls, and must re-arrange operands for argument passing.1299 * calls, and must re-arrange operands for argument passing.
1237 */1300 */
1238static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32)1301static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op,
1302 u32 flags)
1239{1303{
1240 int len = 2 + 10 * 10;1304 int len = 2 + 10 * 10;
1241 struct bpf_insn *insns;1305 struct bpf_insn *insns;
@@ -1249,28 +1313,37 @@ static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32) · 7 unmodified lines
1249 return -ENOMEM;1313 return -ENOMEM;
1250 1314
1251 /* Operand and result values according to operation */1315 /* Operand and result values according to operation */
1252 if (alu32)1316 if (flags & F_SIGNED) {
1253 dst = 0x76543210U;1317 if (flags & F_ALU32)
1254 else1318 dst = -76543210;
1255 dst = 0x7edcba9876543210ULL;1319 else
1320 dst = -7654321076543210LL;
1321 } else {
1322 if (flags & F_ALU32)
1323 dst = 0x76543210U;
1324 else
1325 dst = 0x7edcba9876543210ULL;
1326 }
1256 imm = 0x01234567U;1327 imm = 0x01234567U;
1257 1328
1258 if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH)1329 if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH)
1259 imm &= 31;1330 imm &= 31;
1260 1331
1261 __bpf_alu_result(&res, dst, imm, op);1332 __bpf_alu_result(&res, dst, imm, op, flags);
1262 1333
1263 if (alu32)1334 if (flags & F_ALU32)
1264 res = (u32)res;1335 res = (u32)res;
1265 1336
1266 /* Check all operand registers */1337 /* Check all operand registers */
1267 for (rd = R0; rd <= R9; rd++) {1338 for (rd = R0; rd <= R9; rd++) {
1268 i += __bpf_ld_imm64(&insns[i], rd, dst);1339 i += __bpf_ld_imm64(&insns[i], rd, dst);
1269 1340
1270 if (alu32)1341 s16 off = (flags & F_SIGNED) ? 1 : 0;
1271 insns[i++] = BPF_ALU32_IMM(op, rd, imm);1342
1343 if (flags & F_ALU32)
1344 insns[i++] = BPF_ALU32_IMM_OFF(op, rd, imm, off);
1272 else1345 else
1273 insns[i++] = BPF_ALU64_IMM(op, rd, imm);1346 insns[i++] = BPF_ALU64_IMM_OFF(op, rd, imm, off);
1274 1347
1275 insns[i++] = BPF_JMP32_IMM(BPF_JEQ, rd, res, 2);1348 insns[i++] = BPF_JMP32_IMM(BPF_JEQ, rd, res, 2);
1276 insns[i++] = BPF_MOV64_IMM(R0, __LINE__);1349 insns[i++] = BPF_MOV64_IMM(R0, __LINE__);
@@ -1295,123 +1368,145 @@ static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32) · 18 unmodified lines
1295/* ALU64 K registers */1368/* ALU64 K registers */
1296static int bpf_fill_alu64_mov_imm_regs(struct bpf_test *self)1369static int bpf_fill_alu64_mov_imm_regs(struct bpf_test *self)
1297{1370{
1298 return __bpf_fill_alu_imm_regs(self, BPF_MOV, false);1371 return __bpf_fill_alu_imm_regs(self, BPF_MOV, 0);
1299}1372}
1300 1373
1301static int bpf_fill_alu64_and_imm_regs(struct bpf_test *self)1374static int bpf_fill_alu64_and_imm_regs(struct bpf_test *self)
1302{1375{
1303 return __bpf_fill_alu_imm_regs(self, BPF_AND, false);1376 return __bpf_fill_alu_imm_regs(self, BPF_AND, 0);
1304}1377}
1305 1378
1306static int bpf_fill_alu64_or_imm_regs(struct bpf_test *self)1379static int bpf_fill_alu64_or_imm_regs(struct bpf_test *self)
1307{1380{
1308 return __bpf_fill_alu_imm_regs(self, BPF_OR, false);1381 return __bpf_fill_alu_imm_regs(self, BPF_OR, 0);
1309}1382}
1310 1383
1311static int bpf_fill_alu64_xor_imm_regs(struct bpf_test *self)1384static int bpf_fill_alu64_xor_imm_regs(struct bpf_test *self)
1312{1385{
1313 return __bpf_fill_alu_imm_regs(self, BPF_XOR, false);1386 return __bpf_fill_alu_imm_regs(self, BPF_XOR, 0);
1314}1387}
1315 1388
1316static int bpf_fill_alu64_lsh_imm_regs(struct bpf_test *self)1389static int bpf_fill_alu64_lsh_imm_regs(struct bpf_test *self)
1317{1390{
1318 return __bpf_fill_alu_imm_regs(self, BPF_LSH, false);1391 return __bpf_fill_alu_imm_regs(self, BPF_LSH, 0);
1319}1392}
1320 1393
1321static int bpf_fill_alu64_rsh_imm_regs(struct bpf_test *self)1394static int bpf_fill_alu64_rsh_imm_regs(struct bpf_test *self)
1322{1395{
1323 return __bpf_fill_alu_imm_regs(self, BPF_RSH, false);1396 return __bpf_fill_alu_imm_regs(self, BPF_RSH, 0);
1324}1397}
1325 1398
1326static int bpf_fill_alu64_arsh_imm_regs(struct bpf_test *self)1399static int bpf_fill_alu64_arsh_imm_regs(struct bpf_test *self)
1327{1400{
1328 return __bpf_fill_alu_imm_regs(self, BPF_ARSH, false);1401 return __bpf_fill_alu_imm_regs(self, BPF_ARSH, 0);
1329}1402}
1330 1403
1331static int bpf_fill_alu64_add_imm_regs(struct bpf_test *self)1404static int bpf_fill_alu64_add_imm_regs(struct bpf_test *self)
1332{1405{
1333 return __bpf_fill_alu_imm_regs(self, BPF_ADD, false);1406 return __bpf_fill_alu_imm_regs(self, BPF_ADD, 0);
1334}1407}
1335 1408
1336static int bpf_fill_alu64_sub_imm_regs(struct bpf_test *self)1409static int bpf_fill_alu64_sub_imm_regs(struct bpf_test *self)
1337{1410{
1338 return __bpf_fill_alu_imm_regs(self, BPF_SUB, false);1411 return __bpf_fill_alu_imm_regs(self, BPF_SUB, 0);
1339}1412}
1340 1413
1341static int bpf_fill_alu64_mul_imm_regs(struct bpf_test *self)1414static int bpf_fill_alu64_mul_imm_regs(struct bpf_test *self)
1342{1415{
1343 return __bpf_fill_alu_imm_regs(self, BPF_MUL, false);1416 return __bpf_fill_alu_imm_regs(self, BPF_MUL, 0);
1344}1417}
1345 1418
1346static int bpf_fill_alu64_div_imm_regs(struct bpf_test *self)1419static int bpf_fill_alu64_div_imm_regs(struct bpf_test *self)
1347{1420{
1348 return __bpf_fill_alu_imm_regs(self, BPF_DIV, false);1421 return __bpf_fill_alu_imm_regs(self, BPF_DIV, 0);
1349}1422}
1350 1423
1351static int bpf_fill_alu64_mod_imm_regs(struct bpf_test *self)1424static int bpf_fill_alu64_mod_imm_regs(struct bpf_test *self)
1352{1425{
1353 return __bpf_fill_alu_imm_regs(self, BPF_MOD, false);1426 return __bpf_fill_alu_imm_regs(self, BPF_MOD, 0);
1427}
1428
1429/* Signed ALU64 K registers */
1430static int bpf_fill_alu64_sdiv_imm_regs(struct bpf_test *self)
1431{
1432 return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_SIGNED);
1433}
1434
1435static int bpf_fill_alu64_smod_imm_regs(struct bpf_test *self)
1436{
1437 return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_SIGNED);
1354}1438}
1355 1439
1356/* ALU32 K registers */1440/* ALU32 K registers */
1357static int bpf_fill_alu32_mov_imm_regs(struct bpf_test *self)1441static int bpf_fill_alu32_mov_imm_regs(struct bpf_test *self)
1358{1442{
1359 return __bpf_fill_alu_imm_regs(self, BPF_MOV, true);1443 return __bpf_fill_alu_imm_regs(self, BPF_MOV, F_ALU32);
1360}1444}
1361 1445
1362static int bpf_fill_alu32_and_imm_regs(struct bpf_test *self)1446static int bpf_fill_alu32_and_imm_regs(struct bpf_test *self)
1363{1447{
1364 return __bpf_fill_alu_imm_regs(self, BPF_AND, true);1448 return __bpf_fill_alu_imm_regs(self, BPF_AND, F_ALU32);
1365}1449}
1366 1450
1367static int bpf_fill_alu32_or_imm_regs(struct bpf_test *self)1451static int bpf_fill_alu32_or_imm_regs(struct bpf_test *self)
1368{1452{
1369 return __bpf_fill_alu_imm_regs(self, BPF_OR, true);1453 return __bpf_fill_alu_imm_regs(self, BPF_OR, F_ALU32);
1370}1454}
1371 1455
1372static int bpf_fill_alu32_xor_imm_regs(struct bpf_test *self)1456static int bpf_fill_alu32_xor_imm_regs(struct bpf_test *self)
1373{1457{
1374 return __bpf_fill_alu_imm_regs(self, BPF_XOR, true);1458 return __bpf_fill_alu_imm_regs(self, BPF_XOR, F_ALU32);
1375}1459}
1376 1460
1377static int bpf_fill_alu32_lsh_imm_regs(struct bpf_test *self)1461static int bpf_fill_alu32_lsh_imm_regs(struct bpf_test *self)
1378{1462{
1379 return __bpf_fill_alu_imm_regs(self, BPF_LSH, true);1463 return __bpf_fill_alu_imm_regs(self, BPF_LSH, F_ALU32);
1380}1464}
1381 1465
1382static int bpf_fill_alu32_rsh_imm_regs(struct bpf_test *self)1466static int bpf_fill_alu32_rsh_imm_regs(struct bpf_test *self)
1383{1467{
1384 return __bpf_fill_alu_imm_regs(self, BPF_RSH, true);1468 return __bpf_fill_alu_imm_regs(self, BPF_RSH, F_ALU32);
1385}1469}
1386 1470
1387static int bpf_fill_alu32_arsh_imm_regs(struct bpf_test *self)1471static int bpf_fill_alu32_arsh_imm_regs(struct bpf_test *self)
1388{1472{
1389 return __bpf_fill_alu_imm_regs(self, BPF_ARSH, true);1473 return __bpf_fill_alu_imm_regs(self, BPF_ARSH, F_ALU32);
1390}1474}
1391 1475
1392static int bpf_fill_alu32_add_imm_regs(struct bpf_test *self)1476static int bpf_fill_alu32_add_imm_regs(struct bpf_test *self)
1393{1477{
1394 return __bpf_fill_alu_imm_regs(self, BPF_ADD, true);1478 return __bpf_fill_alu_imm_regs(self, BPF_ADD, F_ALU32);
1395}1479}
1396 1480
1397static int bpf_fill_alu32_sub_imm_regs(struct bpf_test *self)1481static int bpf_fill_alu32_sub_imm_regs(struct bpf_test *self)
1398{1482{
1399 return __bpf_fill_alu_imm_regs(self, BPF_SUB, true);1483 return __bpf_fill_alu_imm_regs(self, BPF_SUB, F_ALU32);
1400}1484}
1401 1485
1402static int bpf_fill_alu32_mul_imm_regs(struct bpf_test *self)1486static int bpf_fill_alu32_mul_imm_regs(struct bpf_test *self)
1403{1487{
1404 return __bpf_fill_alu_imm_regs(self, BPF_MUL, true);1488 return __bpf_fill_alu_imm_regs(self, BPF_MUL, F_ALU32);
1405}1489}
1406 1490
1407static int bpf_fill_alu32_div_imm_regs(struct bpf_test *self)1491static int bpf_fill_alu32_div_imm_regs(struct bpf_test *self)
1408{1492{
1409 return __bpf_fill_alu_imm_regs(self, BPF_DIV, true);1493 return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_ALU32);
1410}1494}
1411 1495
1412static int bpf_fill_alu32_mod_imm_regs(struct bpf_test *self)1496static int bpf_fill_alu32_mod_imm_regs(struct bpf_test *self)
1413{1497{
1414 return __bpf_fill_alu_imm_regs(self, BPF_MOD, true);1498 return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_ALU32);
1499}
1500
1501/* Signed ALU32 K registers */
1502static int bpf_fill_alu32_sdiv_imm_regs(struct bpf_test *self)
1503{
1504 return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_ALU32 | F_SIGNED);
1505}
1506
1507static int bpf_fill_alu32_smod_imm_regs(struct bpf_test *self)
1508{
1509 return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_ALU32 | F_SIGNED);
1415}1510}
1416 1511
1417/*1512/*
@@ -1442,8 +1537,8 @@ static int __bpf_fill_alu_reg_pairs(struct bpf_test *self, u8 op, bool alu32) · 24 unmodified lines
1442 if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH)1537 if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH)
1443 src &= 31;1538 src &= 31;
1444 1539
1445 __bpf_alu_result(&res, dst, src, op);1540 __bpf_alu_result(&res, dst, src, op, 0);
1446 __bpf_alu_result(&same, src, src, op);1541 __bpf_alu_result(&same, src, src, op, 0);
1447 1542
1448 if (alu32) {1543 if (alu32) {
1449 res = (u32)res;1544 res = (u32)res;
@@ -1626,7 +1721,7 @@ static int __bpf_emit_atomic64(struct bpf_test *self, void *arg, · 176 unmodified lines
1626 res = src;1721 res = src;
1627 break;1722 break;
1628 default:1723 default:
1629 __bpf_alu_result(&res, dst, src, BPF_OP(op));1724 __bpf_alu_result(&res, dst, src, BPF_OP(op), 0);
1630 }1725 }
1631 1726
1632 keep = 0x0123456789abcdefULL;1727 keep = 0x0123456789abcdefULL;
@@ -1673,7 +1768,7 @@ static int __bpf_emit_atomic32(struct bpf_test *self, void *arg, · 40 unmodified lines
1673 res = src;1768 res = src;
1674 break;1769 break;
1675 default:1770 default:
1676 __bpf_alu_result(&res, (u32)dst, (u32)src, BPF_OP(op));1771 __bpf_alu_result(&res, (u32)dst, (u32)src, BPF_OP(op), 0);
1677 }1772 }
1678 1773
1679 keep = 0x0123456789abcdefULL;1774 keep = 0x0123456789abcdefULL;
@@ -1939,7 +2034,7 @@ static int __bpf_fill_atomic_reg_pairs(struct bpf_test *self, u8 width, u8 op) · 259 unmodified lines
1939 res = mem;2034 res = mem;
1940 break;2035 break;
1941 default:2036 default:
1942 __bpf_alu_result(&res, mem, upd, BPF_OP(op));2037 __bpf_alu_result(&res, mem, upd, BPF_OP(op), 0);
1943 }2038 }
1944 2039
1945 /* Test all operand registers */2040 /* Test all operand registers */
@@ -12354,6 +12449,22 @@ static struct bpf_test tests[] = { · 10408 unmodified lines
12354 { { 0, 1 } },12449 { { 0, 1 } },
12355 .fill_helper = bpf_fill_alu64_mod_imm_regs,12450 .fill_helper = bpf_fill_alu64_mod_imm_regs,
12356 },12451 },
12452 {
12453 "ALU64_SDIV_K: registers",
12454 { },
12455 INTERNAL,
12456 { },
12457 { { 0, 1 } },
12458 .fill_helper = bpf_fill_alu64_sdiv_imm_regs,
12459 },
12460 {
12461 "ALU64_SMOD_K: registers",
12462 { },
12463 INTERNAL,
12464 { },
12465 { { 0, 1 } },
12466 .fill_helper = bpf_fill_alu64_smod_imm_regs,
12467 },
12357 /* ALU32 K registers */12468 /* ALU32 K registers */
12358 {12469 {
12359 "ALU32_MOV_K: registers",12470 "ALU32_MOV_K: registers",
@@ -12451,6 +12562,22 @@ static struct bpf_test tests[] = { · 91 unmodified lines
12451 { { 0, 1 } },12562 { { 0, 1 } },
12452 .fill_helper = bpf_fill_alu32_mod_imm_regs,12563 .fill_helper = bpf_fill_alu32_mod_imm_regs,
12453 },12564 },
12565 {
12566 "ALU32_SDIV_K: registers",
12567 { },
12568 INTERNAL,
12569 { },
12570 { { 0, 1 } },
12571 .fill_helper = bpf_fill_alu32_sdiv_imm_regs,
12572 },
12573 {
12574 "ALU32_SMOD_K: registers",
12575 { },
12576 INTERNAL,
12577 { },
12578 { { 0, 1 } },
12579 .fill_helper = bpf_fill_alu32_smod_imm_regs,
12580 },
12454 /* ALU64 X register combinations */12581 /* ALU64 X register combinations */
12455 {12582 {
12456 "ALU64_MOV_X: register combinations",12583 "ALU64_MOV_X: register combinations",
@@ -12881,6 +13008,24 @@ static struct bpf_test tests[] = { · 424 unmodified lines
12881 .fill_helper = bpf_fill_alu64_mod_imm,13008 .fill_helper = bpf_fill_alu64_mod_imm,
12882 .nr_testruns = NR_PATTERN_RUNS,13009 .nr_testruns = NR_PATTERN_RUNS,
12883 },13010 },
13011 {
13012 "ALU64_SDIV_K: all immediate value magnitudes",
13013 { },
13014 INTERNAL | FLAG_NO_DATA,
13015 { },
13016 { { 0, 1 } },
13017 .fill_helper = bpf_fill_alu64_sdiv_imm,
13018 .nr_testruns = NR_PATTERN_RUNS,
13019 },
13020 {
13021 "ALU64_SMOD_K: all immediate value magnitudes",
13022 { },
13023 INTERNAL | FLAG_NO_DATA,
13024 { },
13025 { { 0, 1 } },
13026 .fill_helper = bpf_fill_alu64_smod_imm,
13027 .nr_testruns = NR_PATTERN_RUNS,
13028 },
12884 /* ALU32 immediate magnitudes */13029 /* ALU32 immediate magnitudes */
12885 {13030 {
12886 "ALU32_MOV_K: all immediate value magnitudes",13031 "ALU32_MOV_K: all immediate value magnitudes",
@@ -12963,6 +13108,24 @@ static struct bpf_test tests[] = { · 76 unmodified lines
12963 .fill_helper = bpf_fill_alu32_mod_imm,13108 .fill_helper = bpf_fill_alu32_mod_imm,
12964 .nr_testruns = NR_PATTERN_RUNS,13109 .nr_testruns = NR_PATTERN_RUNS,
12965 },13110 },
13111 {
13112 "ALU32_SDIV_K: all immediate value magnitudes",
13113 { },
13114 INTERNAL | FLAG_NO_DATA,
13115 { },
13116 { { 0, 1 } },
13117 .fill_helper = bpf_fill_alu32_sdiv_imm,
13118 .nr_testruns = NR_PATTERN_RUNS,
13119 },
13120 {
13121 "ALU32_SMOD_K: all immediate value magnitudes",
13122 { },
13123 INTERNAL | FLAG_NO_DATA,
13124 { },
13125 { { 0, 1 } },
13126 .fill_helper = bpf_fill_alu32_smod_imm,
13127 .nr_testruns = NR_PATTERN_RUNS,
13128 },
12966 /* ALU64 register magnitudes */13129 /* ALU64 register magnitudes */
12967 {13130 {
12968 "ALU64_MOV_X: all register value magnitudes",13131 "ALU64_MOV_X: all register value magnitudes",
lib/test_rhashtable.c+75−0
@@ -679,6 +679,78 @@ out: · 678 unmodified lines
679 return err;679 return err;
680}680}
681 681
682static int __init test_rhashtable_next_key(void)
683{
684 struct rhashtable_params params = test_rht_params;
685 struct test_obj_val key_missing = { .id = 99999, .tid = 0 };
686 struct test_obj_val *prev_key = NULL;
687 struct rhashtable ht;
688 struct test_obj *objs, *cur;
689 int i, count = 0, err;
690 int visited_keys[8] = { 0 };
691 const int n = ARRAY_SIZE(visited_keys);
692
693 params.nelem_hint = n;
694
695 err = rhashtable_init(&ht, &params);
696 if (err)
697 return err;
698
699 objs = kcalloc(n, sizeof(*objs), GFP_KERNEL);
700 if (!objs) {
701 rhashtable_destroy(&ht);
702 return -ENOMEM;
703 }
704
705 for (i = 0; i < n; i++) {
706 objs[i].value.id = i;
707 err = rhashtable_insert_fast(&ht, &objs[i].node, params);
708 if (err)
709 goto out;
710 }
711
712 rcu_read_lock();
713
714 /* NULL prev_key: walk from the beginning, expect all n elements. */
715 while ((cur = rhashtable_next_key(&ht, prev_key))) {
716 if (IS_ERR(cur)) {
717 err = -EINVAL;
718 goto unlock;
719 }
720 count++;
721 prev_key = &cur->value;
722 visited_keys[cur->value.id] = 1;
723 if (count > n)
724 break;
725 }
726
727 if (count != n) {
728 err = -EINVAL;
729 goto unlock;
730 }
731
732 for (i = 0; i < n; i++) {
733 if (!visited_keys[i]) {
734 err = -EINVAL;
735 goto unlock;
736 }
737 }
738
739 /* Non-existing prev_key: must return ERR_PTR(-ENOENT). */
740 cur = rhashtable_next_key(&ht, &key_missing);
741 if (!IS_ERR(cur) || PTR_ERR(cur) != -ENOENT)
742 err = -EINVAL;
743
744unlock:
745 rcu_read_unlock();
746out:
747 for (i = 0; i < n; i++)
748 rhashtable_remove_fast(&ht, &objs[i].node, params);
749 kfree(objs);
750 rhashtable_destroy(&ht);
751 return err;
752}
753
682static int __init test_rht_init(void)754static int __init test_rht_init(void)
683{755{
684 unsigned int entries;756 unsigned int entries;
@@ -738,6 +810,9 @@ static int __init test_rht_init(void) · 53 unmodified lines
738 810
739 test_insert_duplicates_run();811 test_insert_duplicates_run();
740 812
813 pr_info("Testing rhashtable_next_key: %s\n",
814 test_rhashtable_next_key() == 0 ? "pass" : "FAIL");
815
741 if (!tcount)816 if (!tcount)
742 return 0;817 return 0;
743 818
net/bpf/bpf_dummy_struct_ops.c+7−7
@@ -132,7 +132,7 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, · 131 unmodified lines
132 const struct bpf_struct_ops *st_ops = &bpf_bpf_dummy_ops;132 const struct bpf_struct_ops *st_ops = &bpf_bpf_dummy_ops;
133 const struct btf_type *func_proto;133 const struct btf_type *func_proto;
134 struct bpf_dummy_ops_test_args *args;134 struct bpf_dummy_ops_test_args *args;
135 struct bpf_tramp_links *tlinks = NULL;135 struct bpf_tramp_nodes *tnodes = NULL;
136 struct bpf_tramp_link *link = NULL;136 struct bpf_tramp_link *link = NULL;
137 void *image = NULL;137 void *image = NULL;
138 unsigned int op_idx;138 unsigned int op_idx;
@@ -158,8 +158,8 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, · 19 unmodified lines
158 if (err)158 if (err)
159 goto out;159 goto out;
160 160
161 tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX);161 tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX);
162 if (!tlinks) {162 if (!tnodes) {
163 err = -ENOMEM;163 err = -ENOMEM;
164 goto out;164 goto out;
165 }165 }
@@ -171,11 +171,11 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, · 5 unmodified lines
171 }171 }
172 /* prog doesn't take the ownership of the reference from caller */172 /* prog doesn't take the ownership of the reference from caller */
173 bpf_prog_inc(prog);173 bpf_prog_inc(prog);
174 bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops, prog,174 bpf_tramp_link_init(link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops,
175 prog->expected_attach_type);175 prog, prog->expected_attach_type, 0);
176 176
177 op_idx = prog->expected_attach_type;177 op_idx = prog->expected_attach_type;
178 err = bpf_struct_ops_prepare_trampoline(tlinks, link,178 err = bpf_struct_ops_prepare_trampoline(tnodes, &link->node,
179 &st_ops->func_models[op_idx],179 &st_ops->func_models[op_idx],
180 &dummy_ops_test_ret_function,180 &dummy_ops_test_ret_function,
181 &image, &image_off,181 &image, &image_off,
@@ -198,7 +198,7 @@ out: · 16 unmodified lines
198 bpf_struct_ops_image_free(image);198 bpf_struct_ops_image_free(image);
199 if (link)199 if (link)
200 bpf_link_put(&link->link);200 bpf_link_put(&link->link);
201 kfree(tlinks);201 kfree(tnodes);
202 return err;202 return err;
203}203}
204 204
net/bpf/test_run.c+53−15
@@ -702,6 +702,9 @@ int bpf_prog_test_run_tracing(struct bpf_prog *prog, · 701 unmodified lines
702 case BPF_TRACE_FENTRY:702 case BPF_TRACE_FENTRY:
703 case BPF_TRACE_FEXIT:703 case BPF_TRACE_FEXIT:
704 case BPF_TRACE_FSESSION:704 case BPF_TRACE_FSESSION:
705 case BPF_TRACE_FENTRY_MULTI:
706 case BPF_TRACE_FEXIT_MULTI:
707 case BPF_TRACE_FSESSION_MULTI:
705 if (bpf_fentry_test1(1) != 2 ||708 if (bpf_fentry_test1(1) != 2 ||
706 bpf_fentry_test2(2, 3) != 5 ||709 bpf_fentry_test2(2, 3) != 5 ||
707 bpf_fentry_test3(4, 5, 6) != 15 ||710 bpf_fentry_test3(4, 5, 6) != 15 ||
@@ -747,14 +750,35 @@ static void · 39 unmodified lines
747__bpf_prog_test_run_raw_tp(void *data)750__bpf_prog_test_run_raw_tp(void *data)
748{751{
749 struct bpf_raw_tp_test_run_info *info = data;752 struct bpf_raw_tp_test_run_info *info = data;
753 struct srcu_ctr __percpu *scp = NULL;
750 struct bpf_trace_run_ctx run_ctx = {};754 struct bpf_trace_run_ctx run_ctx = {};
751 struct bpf_run_ctx *old_run_ctx;755 struct bpf_run_ctx *old_run_ctx;
752 756
753 old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);757 old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
754 758
755 rcu_read_lock();759 if (info->prog->sleepable) {
760 scp = rcu_read_lock_tasks_trace();
761 migrate_disable();
762 } else {
763 rcu_read_lock();
764 }
765
766 if (unlikely(!bpf_prog_get_recursion_context(info->prog))) {
767 bpf_prog_inc_misses_counter(info->prog);
768 goto out;
769 }
770
756 info->retval = bpf_prog_run(info->prog, info->ctx);771 info->retval = bpf_prog_run(info->prog, info->ctx);
757 rcu_read_unlock();772
773out:
774 bpf_prog_put_recursion_context(info->prog);
775
776 if (info->prog->sleepable) {
777 migrate_enable();
778 rcu_read_unlock_tasks_trace(scp);
779 } else {
780 rcu_read_unlock();
781 }
758 782
759 bpf_reset_run_ctx(old_run_ctx);783 bpf_reset_run_ctx(old_run_ctx);
760}784}
@@ -782,6 +806,13 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog, · 21 unmodified lines
782 if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 && cpu != 0)806 if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 && cpu != 0)
783 return -EINVAL;807 return -EINVAL;
784 808
809 /*
810 * Sleepable programs cannot run with preemption disabled or in
811 * hardirq context (smp_call_function_single), reject the flag.
812 */
813 if (prog->sleepable && (kattr->test.flags & BPF_F_TEST_RUN_ON_CPU))
814 return -EINVAL;
815
785 if (ctx_size_in) {816 if (ctx_size_in) {
786 info.ctx = memdup_user(ctx_in, ctx_size_in);817 info.ctx = memdup_user(ctx_in, ctx_size_in);
787 if (IS_ERR(info.ctx))818 if (IS_ERR(info.ctx))
@@ -790,24 +821,31 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog, · 2 unmodified lines
790 info.ctx = NULL;821 info.ctx = NULL;
791 }822 }
792 823
824 info.retval = 0;
793 info.prog = prog;825 info.prog = prog;
794 826
795 current_cpu = get_cpu();827 if (prog->sleepable) {
796 if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 ||
797 cpu == current_cpu) {
798 __bpf_prog_test_run_raw_tp(&info);828 __bpf_prog_test_run_raw_tp(&info);
799 } else if (cpu >= nr_cpu_ids || !cpu_online(cpu)) {
800 /* smp_call_function_single() also checks cpu_online()
801 * after csd_lock(). However, since cpu is from user
802 * space, let's do an extra quick check to filter out
803 * invalid value before smp_call_function_single().
804 */
805 err = -ENXIO;
806 } else {829 } else {
807 err = smp_call_function_single(cpu, __bpf_prog_test_run_raw_tp,830 current_cpu = get_cpu();
808 &info, 1);831 if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 ||
832 cpu == current_cpu) {
833 __bpf_prog_test_run_raw_tp(&info);
834 } else if (cpu >= nr_cpu_ids || !cpu_online(cpu)) {
835 /*
836 * smp_call_function_single() also checks cpu_online()
837 * after csd_lock(). However, since cpu is from user
838 * space, let's do an extra quick check to filter out
839 * invalid value before smp_call_function_single().
840 */
841 err = -ENXIO;
842 } else {
843 err = smp_call_function_single(cpu,
844 __bpf_prog_test_run_raw_tp,
845 &info, 1);
846 }
847 put_cpu();
809 }848 }
810 put_cpu();
811 849
812 if (!err &&850 if (!err &&
813 copy_to_user(&uattr->test.retval, &info.retval, sizeof(u32)))851 copy_to_user(&uattr->test.retval, &info.retval, sizeof(u32)))
net/core/filter.c+86−31
@@ -2654,6 +2654,37 @@ static void sk_msg_reset_curr(struct sk_msg *msg) · 2653 unmodified lines
2654 }2654 }
2655}2655}
2656 2656
2657static bool sk_msg_elem_is_copy(const struct sk_msg *msg, u32 i)
2658{
2659 return test_bit(i, msg->sg.copy);
2660}
2661
2662static void sk_msg_clear_elem_copy(struct sk_msg *msg, u32 i)
2663{
2664 __clear_bit(i, msg->sg.copy);
2665}
2666
2667static void sk_msg_set_elem_copy(struct sk_msg *msg, u32 i, bool sg_copy)
2668{
2669 __assign_bit(i, msg->sg.copy, sg_copy);
2670}
2671
2672static void sk_msg_clear_copy_range(struct sk_msg *msg, u32 start, u32 end)
2673{
2674 while (start != end) {
2675 sk_msg_clear_elem_copy(msg, start);
2676 sk_msg_iter_var_next(start);
2677 }
2678}
2679
2680static void sk_msg_sg_move(struct sk_msg *msg, u32 dst, u32 src)
2681{
2682 msg->sg.data[dst] = msg->sg.data[src];
2683
2684 sk_msg_set_elem_copy(msg, dst,
2685 sk_msg_elem_is_copy(msg, src));
2686}
2687
2657static const struct bpf_func_proto bpf_msg_cork_bytes_proto = {2688static const struct bpf_func_proto bpf_msg_cork_bytes_proto = {
2658 .func = bpf_msg_cork_bytes,2689 .func = bpf_msg_cork_bytes,
2659 .gpl_only = false,2690 .gpl_only = false,
@@ -2692,7 +2723,7 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 32 unmodified lines
2692 * account for the headroom.2723 * account for the headroom.
2693 */2724 */
2694 bytes_sg_total = start - offset + bytes;2725 bytes_sg_total = start - offset + bytes;
2695 if (!test_bit(i, msg->sg.copy) && bytes_sg_total <= len)2726 if (!sk_msg_elem_is_copy(msg, i) && bytes_sg_total <= len)
2696 goto out;2727 goto out;
2697 2728
2698 /* At this point we need to linearize multiple scatterlist2729 /* At this point we need to linearize multiple scatterlist
@@ -2733,13 +2764,13 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 34 unmodified lines
2733 poffset += len;2764 poffset += len;
2734 sge->length = 0;2765 sge->length = 0;
2735 put_page(sg_page(sge));2766 put_page(sg_page(sge));
2736 __clear_bit(i, msg->sg.copy);2767 sk_msg_clear_elem_copy(msg, i);
2737 2768
2738 sk_msg_iter_var_next(i);2769 sk_msg_iter_var_next(i);
2739 } while (i != last_sge);2770 } while (i != last_sge);
2740 2771
2741 sg_set_page(&msg->sg.data[first_sge], page, copy, 0);2772 sg_set_page(&msg->sg.data[first_sge], page, copy, 0);
2742 __clear_bit(first_sge, msg->sg.copy);2773 sk_msg_clear_elem_copy(msg, first_sge);
2743 2774
2744 /* To repair sg ring we need to shift entries. If we only2775 /* To repair sg ring we need to shift entries. If we only
2745 * had a single entry though we can just replace it and2776 * had a single entry though we can just replace it and
@@ -2749,8 +2780,14 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 3 unmodified lines
2749 shift = last_sge > first_sge ?2780 shift = last_sge > first_sge ?
2750 last_sge - first_sge - 1 :2781 last_sge - first_sge - 1 :
2751 NR_MSG_FRAG_IDS - first_sge + last_sge - 1;2782 NR_MSG_FRAG_IDS - first_sge + last_sge - 1;
2752 if (!shift)2783 if (!shift) {
2784 sk_msg_clear_elem_copy(msg, msg->sg.end);
2753 goto out;2785 goto out;
2786 }
2787
2788 i = first_sge;
2789 sk_msg_iter_var_next(i);
2790 sk_msg_clear_copy_range(msg, i, last_sge);
2754 2791
2755 i = first_sge;2792 i = first_sge;
2756 sk_msg_iter_var_next(i);2793 sk_msg_iter_var_next(i);
@@ -2764,18 +2801,18 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 7 unmodified lines
2764 if (move_from == msg->sg.end)2801 if (move_from == msg->sg.end)
2765 break;2802 break;
2766 2803
2767 msg->sg.data[i] = msg->sg.data[move_from];2804 sk_msg_sg_move(msg, i, move_from);
2768 sk_msg_sg_copy_assign(msg, i, msg, move_from);
2769 msg->sg.data[move_from].length = 0;2805 msg->sg.data[move_from].length = 0;
2770 msg->sg.data[move_from].page_link = 0;2806 msg->sg.data[move_from].page_link = 0;
2771 msg->sg.data[move_from].offset = 0;2807 msg->sg.data[move_from].offset = 0;
2772 __clear_bit(move_from, msg->sg.copy);2808 sk_msg_clear_elem_copy(msg, move_from);
2773 sk_msg_iter_var_next(i);2809 sk_msg_iter_var_next(i);
2774 } while (1);2810 } while (1);
2775 2811
2776 msg->sg.end = msg->sg.end - shift > msg->sg.end ?2812 msg->sg.end = msg->sg.end - shift > msg->sg.end ?
2777 msg->sg.end - shift + NR_MSG_FRAG_IDS :2813 msg->sg.end - shift + NR_MSG_FRAG_IDS :
2778 msg->sg.end - shift;2814 msg->sg.end - shift;
2815 sk_msg_clear_elem_copy(msg, msg->sg.end);
2779out:2816out:
2780 sk_msg_reset_curr(msg);2817 sk_msg_reset_curr(msg);
2781 msg->data = sg_virt(&msg->sg.data[first_sge]) + start - offset;2818 msg->data = sg_virt(&msg->sg.data[first_sge]) + start - offset;
@@ -2796,9 +2833,10 @@ static const struct bpf_func_proto bpf_msg_pull_data_proto = { · 14 unmodified lines
2796BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start,2833BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start,
2797 u32, len, u64, flags)2834 u32, len, u64, flags)
2798{2835{
2836 bool sge_copy = false, nsge_copy = false, nnsge_copy = false;
2799 struct scatterlist sge, nsge, nnsge, rsge = {0}, *psge;2837 struct scatterlist sge, nsge, nnsge, rsge = {0}, *psge;
2800 u32 new, i = 0, l = 0, space, copy = 0, offset = 0;2838 u32 new, i = 0, l = 0, space, copy = 0, offset = 0;
2801 bool sge_copy, nsge_copy, nnsge_copy, rsge_copy = false;2839 bool rsge_copy = false;
2802 u8 *raw, *to, *from;2840 u8 *raw, *to, *from;
2803 struct page *page;2841 struct page *page;
2804 2842
@@ -2834,6 +2872,9 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 29 unmodified lines
2834 if (!space || (space == 1 && start != offset))2872 if (!space || (space == 1 && start != offset))
2835 copy = msg->sg.data[i].length;2873 copy = msg->sg.data[i].length;
2836 2874
2875 if (unlikely(copy + len < copy))
2876 return -EINVAL;
2877
2837 page = alloc_pages(__GFP_NOWARN | GFP_ATOMIC | __GFP_COMP,2878 page = alloc_pages(__GFP_NOWARN | GFP_ATOMIC | __GFP_COMP,
2838 get_order(copy + len));2879 get_order(copy + len));
2839 if (unlikely(!page))2880 if (unlikely(!page))
@@ -2871,7 +2912,7 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 31 unmodified lines
2871 sk_msg_iter_var_prev(i);2912 sk_msg_iter_var_prev(i);
2872 psge = sk_msg_elem(msg, i);2913 psge = sk_msg_elem(msg, i);
2873 rsge = sk_msg_elem_cpy(msg, i);2914 rsge = sk_msg_elem_cpy(msg, i);
2874 rsge_copy = test_bit(i, msg->sg.copy);2915 rsge_copy = sk_msg_elem_is_copy(msg, i);
2875 2916
2876 psge->length = start - offset;2917 psge->length = start - offset;
2877 rsge.length -= psge->length;2918 rsge.length -= psge->length;
@@ -2896,21 +2937,21 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 18 unmodified lines
2896 2937
2897 /* Shift one or two slots as needed */2938 /* Shift one or two slots as needed */
2898 sge = sk_msg_elem_cpy(msg, new);2939 sge = sk_msg_elem_cpy(msg, new);
2899 sge_copy = test_bit(new, msg->sg.copy);
2900 sg_unmark_end(&sge);2940 sg_unmark_end(&sge);
2941 sge_copy = sk_msg_elem_is_copy(msg, new);
2901 2942
2902 nsge = sk_msg_elem_cpy(msg, i);2943 nsge = sk_msg_elem_cpy(msg, i);
2903 nsge_copy = test_bit(i, msg->sg.copy);2944 nsge_copy = sk_msg_elem_is_copy(msg, i);
2904 if (rsge.length) {2945 if (rsge.length) {
2905 sk_msg_iter_var_next(i);2946 sk_msg_iter_var_next(i);
2906 nnsge = sk_msg_elem_cpy(msg, i);2947 nnsge = sk_msg_elem_cpy(msg, i);
2907 nnsge_copy = test_bit(i, msg->sg.copy);2948 nnsge_copy = sk_msg_elem_is_copy(msg, i);
2908 sk_msg_iter_next(msg, end);2949 sk_msg_iter_next(msg, end);
2909 }2950 }
2910 2951
2911 while (i != msg->sg.end) {2952 while (i != msg->sg.end) {
2912 msg->sg.data[i] = sge;2953 msg->sg.data[i] = sge;
2913 __assign_bit(i, msg->sg.copy, sge_copy);2954 sk_msg_set_elem_copy(msg, i, sge_copy);
2914 sge = nsge;2955 sge = nsge;
2915 sge_copy = nsge_copy;2956 sge_copy = nsge_copy;
2916 sk_msg_iter_var_next(i);2957 sk_msg_iter_var_next(i);
@@ -2918,10 +2959,10 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 1 unmodified line
2918 nsge = nnsge;2959 nsge = nnsge;
2919 nsge_copy = nnsge_copy;2960 nsge_copy = nnsge_copy;
2920 nnsge = sk_msg_elem_cpy(msg, i);2961 nnsge = sk_msg_elem_cpy(msg, i);
2921 nnsge_copy = test_bit(i, msg->sg.copy);2962 nnsge_copy = sk_msg_elem_is_copy(msg, i);
2922 } else {2963 } else {
2923 nsge = sk_msg_elem_cpy(msg, i);2964 nsge = sk_msg_elem_cpy(msg, i);
2924 nsge_copy = test_bit(i, msg->sg.copy);2965 nsge_copy = sk_msg_elem_is_copy(msg, i);
2925 }2966 }
2926 }2967 }
2927 2968
@@ -2929,14 +2970,15 @@ place_new: · 1 unmodified line
2929 /* Place newly allocated data buffer */2970 /* Place newly allocated data buffer */
2930 sk_mem_charge(msg->sk, len);2971 sk_mem_charge(msg->sk, len);
2931 msg->sg.size += len;2972 msg->sg.size += len;
2932 __clear_bit(new, msg->sg.copy);2973 sk_msg_clear_elem_copy(msg, new);
2933 sg_set_page(&msg->sg.data[new], page, len + copy, 0);2974 sg_set_page(&msg->sg.data[new], page, len + copy, 0);
2934 if (rsge.length) {2975 if (rsge.length) {
2935 get_page(sg_page(&rsge));2976 get_page(sg_page(&rsge));
2936 sk_msg_iter_var_next(new);2977 sk_msg_iter_var_next(new);
2937 msg->sg.data[new] = rsge;2978 msg->sg.data[new] = rsge;
2938 __assign_bit(new, msg->sg.copy, rsge_copy);2979 sk_msg_set_elem_copy(msg, new, rsge_copy);
2939 }2980 }
2981 sk_msg_clear_elem_copy(msg, msg->sg.end);
2940 2982
2941 sk_msg_reset_curr(msg);2983 sk_msg_reset_curr(msg);
2942 sk_msg_compute_data_pointers(msg);2984 sk_msg_compute_data_pointers(msg);
@@ -2962,12 +3004,11 @@ static void sk_msg_shift_left(struct sk_msg *msg, int i) · 19 unmodified lines
2962 do {3004 do {
2963 prev = i;3005 prev = i;
2964 sk_msg_iter_var_next(i);3006 sk_msg_iter_var_next(i);
2965 msg->sg.data[prev] = msg->sg.data[i];3007 sk_msg_sg_move(msg, prev, i);
2966 sk_msg_sg_copy_assign(msg, prev, msg, i);
2967 } while (i != msg->sg.end);3008 } while (i != msg->sg.end);
2968 3009
2969 sk_msg_iter_prev(msg, end);3010 sk_msg_iter_prev(msg, end);
2970 __clear_bit(msg->sg.end, msg->sg.copy);3011 sk_msg_clear_elem_copy(msg, msg->sg.end);
2971}3012}
2972 3013
2973static void sk_msg_shift_right(struct sk_msg *msg, int i)3014static void sk_msg_shift_right(struct sk_msg *msg, int i)
@@ -2977,28 +3018,29 @@ static void sk_msg_shift_right(struct sk_msg *msg, int i) · 3 unmodified lines
2977 3018
2978 sk_msg_iter_next(msg, end);3019 sk_msg_iter_next(msg, end);
2979 sge = sk_msg_elem_cpy(msg, i);3020 sge = sk_msg_elem_cpy(msg, i);
2980 sge_copy = test_bit(i, msg->sg.copy);3021 sge_copy = sk_msg_elem_is_copy(msg, i);
2981 sk_msg_iter_var_next(i);3022 sk_msg_iter_var_next(i);
2982 tmp = sk_msg_elem_cpy(msg, i);3023 tmp = sk_msg_elem_cpy(msg, i);
2983 tmp_copy = test_bit(i, msg->sg.copy);3024 tmp_copy = sk_msg_elem_is_copy(msg, i);
2984 3025
2985 while (i != msg->sg.end) {3026 while (i != msg->sg.end) {
2986 msg->sg.data[i] = sge;3027 msg->sg.data[i] = sge;
2987 __assign_bit(i, msg->sg.copy, sge_copy);3028 sk_msg_set_elem_copy(msg, i, sge_copy);
2988 sk_msg_iter_var_next(i);3029 sk_msg_iter_var_next(i);
2989 sge = tmp;3030 sge = tmp;
2990 sge_copy = tmp_copy;3031 sge_copy = tmp_copy;
2991 tmp = sk_msg_elem_cpy(msg, i);3032 tmp = sk_msg_elem_cpy(msg, i);
2992 tmp_copy = test_bit(i, msg->sg.copy);3033 tmp_copy = sk_msg_elem_is_copy(msg, i);
2993 }3034 }
3035 sk_msg_clear_elem_copy(msg, msg->sg.end);
2994}3036}
2995 3037
2996BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start,3038BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start,
2997 u32, len, u64, flags)3039 u32, len, u64, flags)
2998{3040{
2999 u32 i = 0, l = 0, space, offset = 0;3041 u32 i = 0, l = 0, space, offset = 0;
3000 u64 last = start + len;3042 u64 last = (u64)start + len;
3001 int pop;3043 u32 pop;
3002 3044
3003 if (unlikely(flags))3045 if (unlikely(flags))
3004 return -EINVAL;3046 return -EINVAL;
@@ -3047,10 +3089,10 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, · 42 unmodified lines
3047 */3089 */
3048 if (start != offset) {3090 if (start != offset) {
3049 struct scatterlist *nsge, *sge = sk_msg_elem(msg, i);3091 struct scatterlist *nsge, *sge = sk_msg_elem(msg, i);
3092 bool sge_copy = sk_msg_elem_is_copy(msg, i);
3050 int a = start - offset;3093 int a = start - offset;
3051 int b = sge->length - pop - a;3094 int b = sge->length - pop - a;
3052 u32 sge_i = i;3095 u32 sge_idx = i;
3053 bool sge_copy = test_bit(i, msg->sg.copy);
3054 3096
3055 sk_msg_iter_var_next(i);3097 sk_msg_iter_var_next(i);
3056 3098
@@ -3063,7 +3105,7 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, · 6 unmodified lines
3063 sg_set_page(nsge,3105 sg_set_page(nsge,
3064 sg_page(sge),3106 sg_page(sge),
3065 b, sge->offset + pop + a);3107 b, sge->offset + pop + a);
3066 __assign_bit(i, msg->sg.copy, sge_copy);3108 sk_msg_set_elem_copy(msg, i, sge_copy);
3067 } else {3109 } else {
3068 struct page *page, *orig;3110 struct page *page, *orig;
3069 u8 *to, *from;3111 u8 *to, *from;
@@ -3080,7 +3122,7 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, · 10 unmodified lines
3080 memcpy(to, from, a);3122 memcpy(to, from, a);
3081 memcpy(to + a, from + a + pop, b);3123 memcpy(to + a, from + a + pop, b);
3082 sg_set_page(sge, page, a + b, 0);3124 sg_set_page(sge, page, a + b, 0);
3083 __clear_bit(sge_i, msg->sg.copy);3125 sk_msg_clear_elem_copy(msg, sge_idx);
3084 put_page(orig);3126 put_page(orig);
3085 }3127 }
3086 pop = 0;3128 pop = 0;
@@ -5571,11 +5613,24 @@ static int sol_tcp_sockopt(struct sock *sk, int optname, · 2484 unmodified lines
5571 KERNEL_SOCKPTR(optval), *optlen);5613 KERNEL_SOCKPTR(optval), *optlen);
5572}5614}
5573 5615
5616static bool sk_allows_sol_ip_sockopt(struct sock *sk)
5617{
5618 switch (sk->sk_family) {
5619 case AF_INET:
5620 return true;
5621 case AF_INET6:
5622 /* Allow getting/setting sockopt for possible ipv4-mapped ipv6 socket. */
5623 return sk->sk_type != SOCK_RAW && !ipv6_only_sock(sk);
5624 default:
5625 return false;
5626 }
5627}
5628
5574static int sol_ip_sockopt(struct sock *sk, int optname,5629static int sol_ip_sockopt(struct sock *sk, int optname,
5575 char *optval, int *optlen,5630 char *optval, int *optlen,
5576 bool getopt)5631 bool getopt)
5577{5632{
5578 if (sk->sk_family != AF_INET)5633 if (!sk_allows_sol_ip_sockopt(sk))
5579 return -EINVAL;5634 return -EINVAL;
5580 5635
5581 switch (optname) {5636 switch (optname) {
net/core/lwt_bpf.c+12−0
@@ -599,6 +599,7 @@ static int handle_gso_encap(struct sk_buff *skb, bool ipv4, int encap_len) · 598 unmodified lines
599 599
600int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress)600int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress)
601{601{
602 bool is_udp_tunnel;
602 struct iphdr *iph;603 struct iphdr *iph;
603 bool ipv4;604 bool ipv4;
604 int err;605 int err;
@@ -612,10 +613,16 @@ int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress) · 7 unmodified lines
612 ipv4 = true;613 ipv4 = true;
613 if (unlikely(len < iph->ihl * 4))614 if (unlikely(len < iph->ihl * 4))
614 return -EINVAL;615 return -EINVAL;
616 is_udp_tunnel = iph->protocol == IPPROTO_UDP;
617 if (unlikely(is_udp_tunnel && len < iph->ihl * 4 + sizeof(struct udphdr)))
618 return -EINVAL;
615 } else if (iph->version == 6) {619 } else if (iph->version == 6) {
616 ipv4 = false;620 ipv4 = false;
617 if (unlikely(len < sizeof(struct ipv6hdr)))621 if (unlikely(len < sizeof(struct ipv6hdr)))
618 return -EINVAL;622 return -EINVAL;
623 is_udp_tunnel = ((struct ipv6hdr *)iph)->nexthdr == NEXTHDR_UDP;
624 if (unlikely(is_udp_tunnel && len < sizeof(struct ipv6hdr) + sizeof(struct udphdr)))
625 return -EINVAL;
619 } else {626 } else {
620 return -EINVAL;627 return -EINVAL;
621 }628 }
@@ -637,6 +644,11 @@ int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress) · 15 unmodified lines
637 if (ingress)644 if (ingress)
638 skb_postpush_rcsum(skb, iph, len);645 skb_postpush_rcsum(skb, iph, len);
639 skb_reset_network_header(skb);646 skb_reset_network_header(skb);
647 if (is_udp_tunnel) {
648 size_t iph_sz = ipv4 ? iph->ihl * 4 : sizeof(struct ipv6hdr);
649
650 skb_set_transport_header(skb, skb_network_offset(skb) + iph_sz);
651 }
640 memcpy(skb_network_header(skb), hdr, len);652 memcpy(skb_network_header(skb), hdr, len);
641 bpf_compute_data_pointers(skb);653 bpf_compute_data_pointers(skb);
642 skb_clear_hash(skb);654 skb_clear_hash(skb);
net/ipv4/udp_bpf.c+9−0
@@ -50,7 +50,9 @@ static int udp_msg_wait_data(struct sock *sk, struct sk_psock *psock, · 49 unmodified lines
50 sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);50 sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
51 ret = udp_msg_has_data(sk, psock);51 ret = udp_msg_has_data(sk, psock);
52 if (!ret) {52 if (!ret) {
53 release_sock(sk);
53 wait_woken(&wait, TASK_INTERRUPTIBLE, timeo);54 wait_woken(&wait, TASK_INTERRUPTIBLE, timeo);
55 lock_sock(sk);
54 ret = udp_msg_has_data(sk, psock);56 ret = udp_msg_has_data(sk, psock);
55 }57 }
56 sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);58 sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
@@ -79,6 +81,7 @@ static int udp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, · 22 unmodified lines
79 goto out;81 goto out;
80 }82 }
81 83
84 lock_sock(sk);
82msg_bytes_ready:85msg_bytes_ready:
83 copied = sk_msg_recvmsg(sk, psock, msg, len, flags);86 copied = sk_msg_recvmsg(sk, psock, msg, len, flags);
84 if (!copied) {87 if (!copied) {
@@ -90,11 +93,17 @@ msg_bytes_ready: · 5 unmodified lines
90 if (data) {93 if (data) {
91 if (psock_has_data(psock))94 if (psock_has_data(psock))
92 goto msg_bytes_ready;95 goto msg_bytes_ready;
96
97 release_sock(sk);
98
93 ret = sk_udp_recvmsg(sk, msg, len, flags);99 ret = sk_udp_recvmsg(sk, msg, len, flags);
94 goto out;100 goto out;
95 }101 }
96 copied = -EAGAIN;102 copied = -EAGAIN;
97 }103 }
104
105 release_sock(sk);
106
98 ret = copied;107 ret = copied;
99out:108out:
100 sk_psock_put(sk, psock);109 sk_psock_put(sk, psock);
tools/bpf/Makefile+1−0
@@ -11,6 +11,7 @@ INSTALL ?= install · 10 unmodified lines
11CFLAGS += -Wall -O211CFLAGS += -Wall -O2
12CFLAGS += -D__EXPORTED_HEADERS__ -I$(srctree)/tools/include/uapi \12CFLAGS += -D__EXPORTED_HEADERS__ -I$(srctree)/tools/include/uapi \
13 -I$(srctree)/tools/include13 -I$(srctree)/tools/include
14CFLAGS += $(EXTRA_CFLAGS)
14 15
15# This will work when bpf is built in tools env. where srctree16# This will work when bpf is built in tools env. where srctree
16# isn't set and when invoked from selftests build, where srctree17# isn't set and when invoked from selftests build, where srctree
tools/bpf/bpftool/Documentation/bpftool-map.rst+1−1
@@ -56,7 +56,7 @@ MAP COMMANDS · 55 unmodified lines
56| | **cgroup_storage** | **reuseport_sockarray** | **percpu_cgroup_storage**56| | **cgroup_storage** | **reuseport_sockarray** | **percpu_cgroup_storage**
57| | **queue** | **stack** | **sk_storage** | **struct_ops** | **ringbuf** | **inode_storage**57| | **queue** | **stack** | **sk_storage** | **struct_ops** | **ringbuf** | **inode_storage**
58| | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena**58| | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena**
59| | **insn_array** }59| | **insn_array** | **rhash** }
60 60
61DESCRIPTION61DESCRIPTION
62===========62===========
tools/bpf/bpftool/Makefile+17−3
@@ -47,7 +47,8 @@ $(LIBBPF_INTERNAL_HDRS): $(LIBBPF_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_HDRS_ · 46 unmodified lines
47$(LIBBPF_BOOTSTRAP): $(wildcard $(BPF_DIR)/*.[ch] $(BPF_DIR)/Makefile) | $(LIBBPF_BOOTSTRAP_OUTPUT)47$(LIBBPF_BOOTSTRAP): $(wildcard $(BPF_DIR)/*.[ch] $(BPF_DIR)/Makefile) | $(LIBBPF_BOOTSTRAP_OUTPUT)
48 $(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_BOOTSTRAP_OUTPUT) \48 $(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_BOOTSTRAP_OUTPUT) \
49 DESTDIR=$(LIBBPF_BOOTSTRAP_DESTDIR:/=) prefix= \49 DESTDIR=$(LIBBPF_BOOTSTRAP_DESTDIR:/=) prefix= \
50 ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD="$(HOSTLD)" AR="$(HOSTAR)" $@ install_headers50 ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD="$(HOSTLD)" AR="$(HOSTAR)" \
51 CFLAGS="$(LIBBPF_BOOTSTRAP_CFLAGS)" EXTRA_CFLAGS= $@ install_headers
51 52
52$(LIBBPF_BOOTSTRAP_INTERNAL_HDRS): $(LIBBPF_BOOTSTRAP_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_BOOTSTRAP_HDRS_DIR)53$(LIBBPF_BOOTSTRAP_INTERNAL_HDRS): $(LIBBPF_BOOTSTRAP_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_BOOTSTRAP_HDRS_DIR)
53 $(call QUIET_INSTALL, $@)54 $(call QUIET_INSTALL, $@)
@@ -81,6 +82,13 @@ CFLAGS += -DPACKAGE='"bpftool"' -D__EXPORTED_HEADERS__ \ · 27 unmodified lines
81ifneq ($(BPFTOOL_VERSION),)82ifneq ($(BPFTOOL_VERSION),)
82CFLAGS += -DBPFTOOL_VERSION='"$(BPFTOOL_VERSION)"'83CFLAGS += -DBPFTOOL_VERSION='"$(BPFTOOL_VERSION)"'
83endif84endif
85
86# This must be done before appending EXTRA_CFLAGS to CFLAGS to avoid
87# including flags that are not applicable to the host compiler.
88HOST_CFLAGS := $(subst -I$(LIBBPF_INCLUDE),-I$(LIBBPF_BOOTSTRAP_INCLUDE),\
89 $(subst $(CLANG_CROSS_FLAGS),,$(CFLAGS)))
90HOST_CFLAGS += $(HOST_EXTRACFLAGS)
91
84ifneq ($(EXTRA_CFLAGS),)92ifneq ($(EXTRA_CFLAGS),)
85CFLAGS += $(EXTRA_CFLAGS)93CFLAGS += $(EXTRA_CFLAGS)
86endif94endif
@@ -88,10 +96,11 @@ ifneq ($(EXTRA_LDFLAGS),) · 1 unmodified line
88LDFLAGS += $(EXTRA_LDFLAGS)96LDFLAGS += $(EXTRA_LDFLAGS)
89endif97endif
90 98
91HOST_CFLAGS := $(subst -I$(LIBBPF_INCLUDE),-I$(LIBBPF_BOOTSTRAP_INCLUDE),\
92 $(subst $(CLANG_CROSS_FLAGS),,$(CFLAGS)))
93HOST_LDFLAGS := $(LDFLAGS)99HOST_LDFLAGS := $(LDFLAGS)
94 100
101# Remove warnings for libbpf bootstrap build
102LIBBPF_BOOTSTRAP_CFLAGS := $(filter-out -W -Wall -Wextra -Wformat -Wformat-signedness,$(HOST_CFLAGS))
103
95INSTALL ?= install104INSTALL ?= install
96RM ?= rm -f105RM ?= rm -f
97 106
@@ -106,6 +115,10 @@ ifneq ($(SKIP_CRYPTO),1) · 8 unmodified lines
106 CRYPTO_LIBS := -lcrypto115 CRYPTO_LIBS := -lcrypto
107endif116endif
108 117
118ifeq ($(MAKECMDGOALS),bootstrap)
119FEATURE_TESTS := libelf-zstd
120FEATURE_DISPLAY :=
121else
109FEATURE_TESTS := clang-bpf-co-re122FEATURE_TESTS := clang-bpf-co-re
110FEATURE_TESTS += llvm123FEATURE_TESTS += llvm
111FEATURE_TESTS += libcap124FEATURE_TESTS += libcap
@@ -122,6 +135,7 @@ FEATURE_DISPLAY += libcap · 10 unmodified lines
122FEATURE_DISPLAY += libbfd135FEATURE_DISPLAY += libbfd
123FEATURE_DISPLAY += libbfd-liberty136FEATURE_DISPLAY += libbfd-liberty
124FEATURE_DISPLAY += libbfd-liberty-z137FEATURE_DISPLAY += libbfd-liberty-z
138endif
125 139
126check_feat := 1140check_feat := 1
127NON_CHECK_FEAT_TARGETS := clean uninstall doc doc-clean doc-install doc-uninstall141NON_CHECK_FEAT_TARGETS := clean uninstall doc doc-clean doc-install doc-uninstall
tools/bpf/bpftool/btf.c+6−11
@@ -179,8 +179,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 178 unmodified lines
179 case BTF_KIND_STRUCT:179 case BTF_KIND_STRUCT:
180 case BTF_KIND_UNION: {180 case BTF_KIND_UNION: {
181 const struct btf_member *m = (const void *)(t + 1);181 const struct btf_member *m = (const void *)(t + 1);
182 __u16 vlen = BTF_INFO_VLEN(t->info);182 __u32 i, vlen = BTF_INFO_VLEN(t->info);
183 int i;
184 183
185 if (json_output) {184 if (json_output) {
186 jsonw_uint_field(w, "size", t->size);185 jsonw_uint_field(w, "size", t->size);
@@ -225,9 +224,8 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 38 unmodified lines
225 }224 }
226 case BTF_KIND_ENUM: {225 case BTF_KIND_ENUM: {
227 const struct btf_enum *v = (const void *)(t + 1);226 const struct btf_enum *v = (const void *)(t + 1);
228 __u16 vlen = BTF_INFO_VLEN(t->info);227 __u32 i, vlen = BTF_INFO_VLEN(t->info);
229 const char *encoding;228 const char *encoding;
230 int i;
231 229
232 encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED";230 encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED";
233 if (json_output) {231 if (json_output) {
@@ -263,9 +261,8 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 29 unmodified lines
263 }261 }
264 case BTF_KIND_ENUM64: {262 case BTF_KIND_ENUM64: {
265 const struct btf_enum64 *v = btf_enum64(t);263 const struct btf_enum64 *v = btf_enum64(t);
266 __u16 vlen = btf_vlen(t);264 __u32 i, vlen = btf_vlen(t);
267 const char *encoding;265 const char *encoding;
268 int i;
269 266
270 encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED";267 encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED";
271 if (json_output) {268 if (json_output) {
@@ -325,8 +322,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 53 unmodified lines
325 }322 }
326 case BTF_KIND_FUNC_PROTO: {323 case BTF_KIND_FUNC_PROTO: {
327 const struct btf_param *p = (const void *)(t + 1);324 const struct btf_param *p = (const void *)(t + 1);
328 __u16 vlen = BTF_INFO_VLEN(t->info);325 __u32 i, vlen = BTF_INFO_VLEN(t->info);
329 int i;
330 326
331 if (json_output) {327 if (json_output) {
332 jsonw_uint_field(w, "ret_type_id", t->type);328 jsonw_uint_field(w, "ret_type_id", t->type);
@@ -369,8 +365,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 36 unmodified lines
369 case BTF_KIND_DATASEC: {365 case BTF_KIND_DATASEC: {
370 const struct btf_var_secinfo *v = (const void *)(t + 1);366 const struct btf_var_secinfo *v = (const void *)(t + 1);
371 const struct btf_type *vt;367 const struct btf_type *vt;
372 __u16 vlen = BTF_INFO_VLEN(t->info);368 __u32 i, vlen = BTF_INFO_VLEN(t->info);
373 int i;
374 369
375 if (json_output) {370 if (json_output) {
376 jsonw_uint_field(w, "size", t->size);371 jsonw_uint_field(w, "size", t->size);
@@ -675,7 +670,7 @@ static __u64 btf_name_hasher(__u64 hash, const struct btf *btf, __u32 name_off) · 298 unmodified lines
675static __u64 btf_type_disambig_hash(const struct btf *btf, __u32 id, bool include_members)670static __u64 btf_type_disambig_hash(const struct btf *btf, __u32 id, bool include_members)
676{671{
677 const struct btf_type *t = btf__type_by_id(btf, id);672 const struct btf_type *t = btf__type_by_id(btf, id);
678 int i;673 __u32 i;
679 size_t hash = 0;674 size_t hash = 0;
680 675
681 hash = btf_name_hasher(hash, btf, t->name_off);676 hash = btf_name_hasher(hash, btf, t->name_off);
tools/bpf/bpftool/btf_dumper.c+2−2
@@ -150,7 +150,7 @@ static int btf_dumper_enum(const struct btf_dumper *d, · 149 unmodified lines
150{150{
151 const struct btf_enum *enums = btf_enum(t);151 const struct btf_enum *enums = btf_enum(t);
152 __s64 value;152 __s64 value;
153 __u16 i;153 __u32 i;
154 154
155 switch (t->size) {155 switch (t->size) {
156 case 8:156 case 8:
@@ -189,7 +189,7 @@ static int btf_dumper_enum64(const struct btf_dumper *d, · 32 unmodified lines
189 const struct btf_enum64 *enums = btf_enum64(t);189 const struct btf_enum64 *enums = btf_enum64(t);
190 __u32 val_lo32, val_hi32;190 __u32 val_lo32, val_hi32;
191 __u64 value;191 __u64 value;
192 __u16 i;192 __u32 i;
193 193
194 value = *(__u64 *)data;194 value = *(__u64 *)data;
195 val_lo32 = (__u32)value;195 val_lo32 = (__u32)value;
tools/bpf/bpftool/gen.c+10−8
@@ -1399,7 +1399,7 @@ static int do_skeleton(int argc, char **argv) · 1398 unmodified lines
1399 continue;1399 continue;
1400 1400
1401 if (use_loader)1401 if (use_loader)
1402 printf("t\tint %s_fd;\n", ident);1402 printf("\t\tint %s_fd;\n", ident);
1403 else1403 else
1404 printf("\t\tstruct bpf_link *%s;\n", ident);1404 printf("\t\tstruct bpf_link *%s;\n", ident);
1405 }1405 }
@@ -2094,7 +2094,8 @@ btfgen_mark_type(struct btfgen_info *info, unsigned int type_id, bool follow_poi · 688 unmodified lines
2094 struct btf_type *cloned_type;2094 struct btf_type *cloned_type;
2095 struct btf_param *param;2095 struct btf_param *param;
2096 struct btf_array *array;2096 struct btf_array *array;
2097 int err, i;2097 __u32 i;
2098 int err;
2098 2099
2099 if (type_id == 0)2100 if (type_id == 0)
2100 return 0;2101 return 0;
@@ -2229,7 +2230,8 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool · 128 unmodified lines
2229 const struct btf_type *btf_type;2230 const struct btf_type *btf_type;
2230 struct btf *btf = info->src_btf;2231 struct btf *btf = info->src_btf;
2231 struct btf_type *cloned_type;2232 struct btf_type *cloned_type;
2232 int i, err;2233 int err;
2234 __u32 i;
2233 2235
2234 if (type_id == 0)2236 if (type_id == 0)
2235 return 0;2237 return 0;
@@ -2249,7 +2251,7 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool · 13 unmodified lines
2249 case BTF_KIND_STRUCT:2251 case BTF_KIND_STRUCT:
2250 case BTF_KIND_UNION: {2252 case BTF_KIND_UNION: {
2251 struct btf_member *m = btf_members(btf_type);2253 struct btf_member *m = btf_members(btf_type);
2252 __u16 vlen = btf_vlen(btf_type);2254 __u32 vlen = btf_vlen(btf_type);
2253 2255
2254 if (behind_ptr)2256 if (behind_ptr)
2255 break;2257 break;
@@ -2286,7 +2288,7 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool · 30 unmodified lines
2286 break;2288 break;
2287 }2289 }
2288 case BTF_KIND_FUNC_PROTO: {2290 case BTF_KIND_FUNC_PROTO: {
2289 __u16 vlen = btf_vlen(btf_type);2291 __u32 vlen = btf_vlen(btf_type);
2290 struct btf_param *param;2292 struct btf_param *param;
2291 2293
2292 /* mark ret type */2294 /* mark ret type */
@@ -2492,8 +2494,9 @@ static struct btf *btfgen_get_btf(struct btfgen_info *info) · 199 unmodified lines
2492{2494{
2493 struct btf *btf_new = NULL;2495 struct btf *btf_new = NULL;
2494 unsigned int *ids = NULL;2496 unsigned int *ids = NULL;
2495 unsigned int i, n = btf__type_cnt(info->marked_btf);2497 unsigned int n = btf__type_cnt(info->marked_btf);
2496 int err = 0;2498 int err = 0;
2499 __u32 i;
2497 2500
2498 btf_new = btf__new_empty();2501 btf_new = btf__new_empty();
2499 if (!btf_new) {2502 if (!btf_new) {
@@ -2523,8 +2526,7 @@ static struct btf *btfgen_get_btf(struct btfgen_info *info) · 23 unmodified lines
2523 /* add members for struct and union */2526 /* add members for struct and union */
2524 if (btf_is_composite(type)) {2527 if (btf_is_composite(type)) {
2525 struct btf_member *cloned_m, *m;2528 struct btf_member *cloned_m, *m;
2526 unsigned short vlen;2529 __u32 vlen, idx_src;
2527 int idx_src;
2528 2530
2529 name = btf__str_by_offset(info->src_btf, type->name_off);2531 name = btf__str_by_offset(info->src_btf, type->name_off);
2530 2532
tools/bpf/bpftool/map.c+1−1
@@ -1478,7 +1478,7 @@ static int do_help(int argc, char **argv) · 1477 unmodified lines
1478 " cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n"1478 " cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n"
1479 " queue | stack | sk_storage | struct_ops | ringbuf | inode_storage |\n"1479 " queue | stack | sk_storage | struct_ops | ringbuf | inode_storage |\n"
1480 " task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena |\n"1480 " task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena |\n"
1481 " insn_array }\n"1481 " insn_array | rhash }\n"
1482 " " HELP_SPEC_OPTIONS " |\n"1482 " " HELP_SPEC_OPTIONS " |\n"
1483 " {-f|--bpffs} | {-n|--nomount} }\n"1483 " {-f|--bpffs} | {-n|--nomount} }\n"
1484 "",1484 "",
tools/bpf/bpftool/net.c+2−2
@@ -603,14 +603,14 @@ static int query_flow_dissector(struct bpf_attach_info *attach_info) · 602 unmodified lines
603 &attach_flags, prog_ids, &prog_cnt);603 &attach_flags, prog_ids, &prog_cnt);
604 close(fd);604 close(fd);
605 if (err) {605 if (err) {
606 if (errno == EINVAL) {606 if (err == -EINVAL) {
607 /* Older kernel's don't support querying607 /* Older kernel's don't support querying
608 * flow dissector programs.608 * flow dissector programs.
609 */609 */
610 errno = 0;610 errno = 0;
611 return 0;611 return 0;
612 }612 }
613 p_err("can't query prog: %s", strerror(errno));613 p_err("can't query prog: %s", strerror(-err));
614 return -1;614 return -1;
615 }615 }
616 616
tools/build/feature/test-bpf.c+2−0
@@ -20,6 +20,8 @@ · 19 unmodified lines
20# define __NR_bpf 631920# define __NR_bpf 6319
21# elif defined(__mips__) && defined(_ABI64)21# elif defined(__mips__) && defined(_ABI64)
22# define __NR_bpf 531522# define __NR_bpf 5315
23# elif defined(__loongarch__)
24# define __NR_bpf 280
23# else25# else
24# error __NR_bpf not defined. libbpf does not support your arch.26# error __NR_bpf not defined. libbpf does not support your arch.
25# endif27# endif
tools/include/uapi/linux/bpf.h+32−1
@@ -994,6 +994,7 @@ enum bpf_cmd { · 993 unmodified lines
994 BPF_PROG_STREAM_READ_BY_FD,994 BPF_PROG_STREAM_READ_BY_FD,
995 BPF_PROG_ASSOC_STRUCT_OPS,995 BPF_PROG_ASSOC_STRUCT_OPS,
996 __MAX_BPF_CMD,996 __MAX_BPF_CMD,
997 BPF_COMMON_ATTRS = 1 << 16, /* Indicate carrying syscall common attrs. */
997};998};
998 999
999enum bpf_map_type {1000enum bpf_map_type {
@@ -1046,6 +1047,7 @@ enum bpf_map_type { · 46 unmodified lines
1046 BPF_MAP_TYPE_CGRP_STORAGE,1047 BPF_MAP_TYPE_CGRP_STORAGE,
1047 BPF_MAP_TYPE_ARENA,1048 BPF_MAP_TYPE_ARENA,
1048 BPF_MAP_TYPE_INSN_ARRAY,1049 BPF_MAP_TYPE_INSN_ARRAY,
1050 BPF_MAP_TYPE_RHASH,
1049 __MAX_BPF_MAP_TYPE1051 __MAX_BPF_MAP_TYPE
1050};1052};
1051 1053
@@ -1154,6 +1156,9 @@ enum bpf_attach_type { · 102 unmodified lines
1154 BPF_TRACE_KPROBE_SESSION,1156 BPF_TRACE_KPROBE_SESSION,
1155 BPF_TRACE_UPROBE_SESSION,1157 BPF_TRACE_UPROBE_SESSION,
1156 BPF_TRACE_FSESSION,1158 BPF_TRACE_FSESSION,
1159 BPF_TRACE_FENTRY_MULTI,
1160 BPF_TRACE_FEXIT_MULTI,
1161 BPF_TRACE_FSESSION_MULTI,
1157 __MAX_BPF_ATTACH_TYPE1162 __MAX_BPF_ATTACH_TYPE
1158};1163};
1159 1164
@@ -1178,6 +1183,7 @@ enum bpf_link_type { · 18 unmodified lines
1178 BPF_LINK_TYPE_UPROBE_MULTI = 12,1183 BPF_LINK_TYPE_UPROBE_MULTI = 12,
1179 BPF_LINK_TYPE_NETKIT = 13,1184 BPF_LINK_TYPE_NETKIT = 13,
1180 BPF_LINK_TYPE_SOCKMAP = 14,1185 BPF_LINK_TYPE_SOCKMAP = 14,
1186 BPF_LINK_TYPE_TRACING_MULTI = 15,
1181 __MAX_BPF_LINK_TYPE,1187 __MAX_BPF_LINK_TYPE,
1182};1188};
1183 1189
@@ -1321,7 +1327,11 @@ enum { · 137 unmodified lines
1321 * BPF_TRACE_UPROBE_MULTI attach type to create return probe.1327 * BPF_TRACE_UPROBE_MULTI attach type to create return probe.
1322 */1328 */
1323enum {1329enum {
1324 BPF_F_UPROBE_MULTI_RETURN = (1U << 0)1330 /* Get return uprobe. */
1331 BPF_F_UPROBE_MULTI_RETURN = (1U << 0),
1332
1333 /* Get path from provided path_fd. */
1334 BPF_F_UPROBE_MULTI_PATH_FD = (1U << 1),
1325};1335};
1326 1336
1327/* link_create.netfilter.flags used in LINK_CREATE command for1337/* link_create.netfilter.flags used in LINK_CREATE command for
@@ -1500,6 +1510,13 @@ struct bpf_stack_build_id { · 172 unmodified lines
1500 };1510 };
1501};1511};
1502 1512
1513struct bpf_common_attr {
1514 __aligned_u64 log_buf;
1515 __u32 log_size;
1516 __u32 log_level;
1517 __u32 log_true_size;
1518};
1519
1503#define BPF_OBJ_NAME_LEN 16U1520#define BPF_OBJ_NAME_LEN 16U
1504 1521
1505enum {1522enum {
@@ -1537,6 +1554,11 @@ union bpf_attr { · 31 unmodified lines
1537 *1554 *
1538 * BPF_MAP_TYPE_ARENA - contains the address where user space1555 * BPF_MAP_TYPE_ARENA - contains the address where user space
1539 * is going to mmap() the arena. It has to be page aligned.1556 * is going to mmap() the arena. It has to be page aligned.
1557 *
1558 * BPF_MAP_TYPE_RHASH - initial table size hint
1559 * (nelem_hint). 0 = use rhashtable default. Must be
1560 * <= min(max_entries, U16_MAX). Upper 32 bits reserved,
1561 * must be zero.
1540 */1562 */
1541 __u64 map_extra;1563 __u64 map_extra;
1542 1564
@@ -1846,6 +1868,7 @@ union bpf_attr { · 303 unmodified lines
1846 __u32 cnt;1868 __u32 cnt;
1847 __u32 flags;1869 __u32 flags;
1848 __u32 pid;1870 __u32 pid;
1871 __u32 path_fd;
1849 } uprobe_multi;1872 } uprobe_multi;
1850 struct {1873 struct {
1851 union {1874 union {
@@ -1861,6 +1884,11 @@ union bpf_attr { · 9 unmodified lines
1861 };1884 };
1862 __u64 expected_revision;1885 __u64 expected_revision;
1863 } cgroup;1886 } cgroup;
1887 struct {
1888 __aligned_u64 ids;
1889 __aligned_u64 cookies;
1890 __u32 cnt;
1891 } tracing_multi;
1864 };1892 };
1865 } link_create;1893 } link_create;
1866 1894
@@ -6698,6 +6726,7 @@ struct bpf_prog_info { · 4831 unmodified lines
6698 __u32 verified_insns;6726 __u32 verified_insns;
6699 __u32 attach_btf_obj_id;6727 __u32 attach_btf_obj_id;
6700 __u32 attach_btf_id;6728 __u32 attach_btf_id;
6729 __u32 :32;
6701} __attribute__((aligned(8)));6730} __attribute__((aligned(8)));
6702 6731
6703struct bpf_map_info {6732struct bpf_map_info {
@@ -6719,6 +6748,7 @@ struct bpf_map_info { · 15 unmodified lines
6719 __u64 map_extra;6748 __u64 map_extra;
6720 __aligned_u64 hash;6749 __aligned_u64 hash;
6721 __u32 hash_size;6750 __u32 hash_size;
6751 __u32 :32;
6722} __attribute__((aligned(8)));6752} __attribute__((aligned(8)));
6723 6753
6724struct bpf_btf_info {6754struct bpf_btf_info {
@@ -7236,6 +7266,7 @@ enum { · 511 unmodified lines
7236 TCP_BPF_SOCK_OPS_CB_FLAGS = 1008, /* Get or Set TCP sock ops flags */7266 TCP_BPF_SOCK_OPS_CB_FLAGS = 1008, /* Get or Set TCP sock ops flags */
7237 SK_BPF_CB_FLAGS = 1009, /* Get or set sock ops flags in socket */7267 SK_BPF_CB_FLAGS = 1009, /* Get or set sock ops flags in socket */
7238 SK_BPF_BYPASS_PROT_MEM = 1010, /* Get or Set sk->sk_bypass_prot_mem */7268 SK_BPF_BYPASS_PROT_MEM = 1010, /* Get or Set sk->sk_bypass_prot_mem */
7269
7239};7270};
7240 7271
7241enum {7272enum {
tools/include/uapi/linux/btf.h+14−12
@@ -33,20 +33,22 @@ struct btf_header { · 32 unmodified lines
33 __u32 layout_len; /* length of layout section */33 __u32 layout_len; /* length of layout section */
34};34};
35 35
36/* Max # of type identifier */36enum btf_max {
37#define BTF_MAX_TYPE 0x000fffff37 /* Max possible kind */
38/* Max offset into the string section */38 BTF_MAX_KIND = 0x0000007f,
39#define BTF_MAX_NAME_OFFSET 0x00ffffff39 /* Max # of type identifier */
40/* Max # of struct/union/enum members or func args */40 BTF_MAX_TYPE = 0x000fffff,
41#define BTF_MAX_VLEN 0xffff41 /* Max offset into the string section */
42 BTF_MAX_NAME_OFFSET = 0x00ffffff,
43 /* Max # of struct/union/enum members or func args */
44 BTF_MAX_VLEN = 0x00ffffff,
45};
42 46
43struct btf_type {47struct btf_type {
44 __u32 name_off;48 __u32 name_off;
45 /* "info" bits arrangement49 /* "info" bits arrangement
46 * bits 0-15: vlen (e.g. # of struct's members)50 * bits 0-23: vlen (e.g. # of struct's members)
47 * bits 16-23: unused51 * bits 24-30: kind (e.g. int, ptr, array...etc)
48 * bits 24-28: kind (e.g. int, ptr, array...etc)
49 * bits 29-30: unused
50 * bit 31: kind_flag, currently used by52 * bit 31: kind_flag, currently used by
51 * struct, union, enum, fwd, enum64,53 * struct, union, enum, fwd, enum64,
52 * decl_tag and type_tag54 * decl_tag and type_tag
@@ -65,8 +67,8 @@ struct btf_type { · 12 unmodified lines
65 };67 };
66};68};
67 69
68#define BTF_INFO_KIND(info) (((info) >> 24) & 0x1f)70#define BTF_INFO_KIND(info) (((info) >> 24) & 0x7f)
69#define BTF_INFO_VLEN(info) ((info) & 0xffff)71#define BTF_INFO_VLEN(info) ((info) & 0xffffff)
70#define BTF_INFO_KFLAG(info) ((info) >> 31)72#define BTF_INFO_KFLAG(info) ((info) >> 31)
71 73
72enum {74enum {
tools/lib/bpf/Makefile+9−8
@@ -49,6 +49,14 @@ man_dir_SQ = '$(subst ','\'',$(man_dir))' · 48 unmodified lines
49export man_dir man_dir_SQ INSTALL49export man_dir man_dir_SQ INSTALL
50export DESTDIR DESTDIR_SQ50export DESTDIR DESTDIR_SQ
51 51
52# Defer assigning EXTRA_CFLAGS to CFLAGS until after including
53# tools/scripts/Makefile.include, as it may add flags to EXTRA_CFLAGS.
54ifdef EXTRA_CFLAGS
55 CFLAGS :=
56else
57 CFLAGS := -g -O2
58endif
59
52include $(srctree)/tools/scripts/Makefile.include60include $(srctree)/tools/scripts/Makefile.include
53 61
54# copy a bit from Linux kbuild62# copy a bit from Linux kbuild
@@ -70,13 +78,6 @@ LIB_TARGET = libbpf.a libbpf.so.$(LIBBPF_VERSION) · 15 unmodified lines
70LIB_FILE = libbpf.a libbpf.so*78LIB_FILE = libbpf.a libbpf.so*
71PC_FILE = libbpf.pc79PC_FILE = libbpf.pc
72 80
73# Set compile option CFLAGS
74ifdef EXTRA_CFLAGS
75 CFLAGS := $(EXTRA_CFLAGS)
76else
77 CFLAGS := -g -O2
78endif
79
80# Append required CFLAGS81# Append required CFLAGS
81override CFLAGS += -std=gnu8982override CFLAGS += -std=gnu89
82override CFLAGS += $(EXTRA_WARNINGS) -Wno-switch-enum83override CFLAGS += $(EXTRA_WARNINGS) -Wno-switch-enum
@@ -84,7 +85,7 @@ override CFLAGS += -Werror -Wall · 1 unmodified line
84override CFLAGS += $(INCLUDES)85override CFLAGS += $(INCLUDES)
85override CFLAGS += -fvisibility=hidden86override CFLAGS += -fvisibility=hidden
86override CFLAGS += -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=6487override CFLAGS += -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
87override CFLAGS += $(CLANG_CROSS_FLAGS)88override CFLAGS += $(EXTRA_CFLAGS)
88 89
89# flags specific for shared library90# flags specific for shared library
90SHLIB_FLAGS := -DSHARED -fPIC91SHLIB_FLAGS := -DSHARED -fPIC
tools/lib/bpf/bpf.c+66−1
@@ -59,6 +59,8 @@ · 58 unmodified lines
59# define __NR_bpf 631959# define __NR_bpf 6319
60# elif defined(__mips__) && defined(_ABI64)60# elif defined(__mips__) && defined(_ABI64)
61# define __NR_bpf 531561# define __NR_bpf 5315
62# elif defined(__loongarch__)
63# define __NR_bpf 280
62# else64# else
63# error __NR_bpf not defined. libbpf does not support your arch.65# error __NR_bpf not defined. libbpf does not support your arch.
64# endif66# endif
@@ -69,6 +71,42 @@ static inline __u64 ptr_to_u64(const void *ptr) · 4 unmodified lines
69 return (__u64) (unsigned long) ptr;71 return (__u64) (unsigned long) ptr;
70}72}
71 73
74static inline int sys_bpf_ext(enum bpf_cmd cmd, union bpf_attr *attr,
75 unsigned int size,
76 struct bpf_common_attr *attr_common,
77 unsigned int size_common)
78{
79 cmd = attr_common ? (cmd | BPF_COMMON_ATTRS) : (cmd & ~BPF_COMMON_ATTRS);
80 return syscall(__NR_bpf, cmd, attr, size, attr_common, size_common);
81}
82
83static inline int sys_bpf_ext_fd(enum bpf_cmd cmd, union bpf_attr *attr,
84 unsigned int size,
85 struct bpf_common_attr *attr_common,
86 unsigned int size_common)
87{
88 int fd;
89
90 fd = sys_bpf_ext(cmd, attr, size, attr_common, size_common);
91 return ensure_good_fd(fd);
92}
93
94int probe_sys_bpf_ext(void)
95{
96 const size_t attr_sz = offsetofend(union bpf_attr, prog_token_fd);
97 union bpf_attr attr;
98 int fd;
99
100 memset(&attr, 0, attr_sz);
101 fd = syscall(__NR_bpf, BPF_PROG_LOAD | BPF_COMMON_ATTRS, &attr, attr_sz, NULL,
102 sizeof(struct bpf_common_attr));
103 if (fd >= 0) {
104 close(fd);
105 return -EINVAL;
106 }
107 return errno == EFAULT ? 1 : 0;
108}
109
72static inline int sys_bpf(enum bpf_cmd cmd, union bpf_attr *attr,110static inline int sys_bpf(enum bpf_cmd cmd, union bpf_attr *attr,
73 unsigned int size)111 unsigned int size)
74{112{
@@ -173,6 +211,9 @@ int bpf_map_create(enum bpf_map_type map_type, · 98 unmodified lines
173 const struct bpf_map_create_opts *opts)211 const struct bpf_map_create_opts *opts)
174{212{
175 const size_t attr_sz = offsetofend(union bpf_attr, excl_prog_hash_size);213 const size_t attr_sz = offsetofend(union bpf_attr, excl_prog_hash_size);
214 const size_t attr_common_sz = sizeof(struct bpf_common_attr);
215 struct bpf_common_attr attr_common;
216 struct bpf_log_opts *log_opts;
176 union bpf_attr attr;217 union bpf_attr attr;
177 int fd;218 int fd;
178 219
@@ -206,7 +247,21 @@ int bpf_map_create(enum bpf_map_type map_type, · 27 unmodified lines
206 attr.excl_prog_hash = ptr_to_u64(OPTS_GET(opts, excl_prog_hash, NULL));247 attr.excl_prog_hash = ptr_to_u64(OPTS_GET(opts, excl_prog_hash, NULL));
207 attr.excl_prog_hash_size = OPTS_GET(opts, excl_prog_hash_size, 0);248 attr.excl_prog_hash_size = OPTS_GET(opts, excl_prog_hash_size, 0);
208 249
209 fd = sys_bpf_fd(BPF_MAP_CREATE, &attr, attr_sz);250 log_opts = OPTS_GET(opts, log_opts, NULL);
251 if (!OPTS_VALID(log_opts, bpf_log_opts))
252 return libbpf_err(-EINVAL);
253
254 if (log_opts && feat_supported(NULL, FEAT_BPF_SYSCALL_COMMON_ATTRS)) {
255 memset(&attr_common, 0, attr_common_sz);
256 attr_common.log_buf = ptr_to_u64(OPTS_GET(log_opts, buf, NULL));
257 attr_common.log_size = OPTS_GET(log_opts, size, 0);
258 attr_common.log_level = OPTS_GET(log_opts, level, 0);
259 fd = sys_bpf_ext_fd(BPF_MAP_CREATE, &attr, attr_sz, &attr_common, attr_common_sz);
260 OPTS_SET(log_opts, true_size, attr_common.log_true_size);
261 } else {
262 fd = sys_bpf_fd(BPF_MAP_CREATE, &attr, attr_sz);
263 OPTS_SET(log_opts, true_size, 0);
264 }
210 return libbpf_err_errno(fd);265 return libbpf_err_errno(fd);
211}266}
212 267
@@ -787,9 +842,19 @@ int bpf_link_create(int prog_fd, int target_fd, · 574 unmodified lines
787 attr.link_create.uprobe_multi.ref_ctr_offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.ref_ctr_offsets, 0));842 attr.link_create.uprobe_multi.ref_ctr_offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.ref_ctr_offsets, 0));
788 attr.link_create.uprobe_multi.cookies = ptr_to_u64(OPTS_GET(opts, uprobe_multi.cookies, 0));843 attr.link_create.uprobe_multi.cookies = ptr_to_u64(OPTS_GET(opts, uprobe_multi.cookies, 0));
789 attr.link_create.uprobe_multi.pid = OPTS_GET(opts, uprobe_multi.pid, 0);844 attr.link_create.uprobe_multi.pid = OPTS_GET(opts, uprobe_multi.pid, 0);
845 attr.link_create.uprobe_multi.path_fd = OPTS_GET(opts, uprobe_multi.path_fd, 0);
790 if (!OPTS_ZEROED(opts, uprobe_multi))846 if (!OPTS_ZEROED(opts, uprobe_multi))
791 return libbpf_err(-EINVAL);847 return libbpf_err(-EINVAL);
792 break;848 break;
849 case BPF_TRACE_FENTRY_MULTI:
850 case BPF_TRACE_FEXIT_MULTI:
851 case BPF_TRACE_FSESSION_MULTI:
852 attr.link_create.tracing_multi.ids = ptr_to_u64(OPTS_GET(opts, tracing_multi.ids, 0));
853 attr.link_create.tracing_multi.cookies = ptr_to_u64(OPTS_GET(opts, tracing_multi.cookies, 0));
854 attr.link_create.tracing_multi.cnt = OPTS_GET(opts, tracing_multi.cnt, 0);
855 if (!OPTS_ZEROED(opts, tracing_multi))
856 return libbpf_err(-EINVAL);
857 break;
793 case BPF_TRACE_RAW_TP:858 case BPF_TRACE_RAW_TP:
794 case BPF_TRACE_FENTRY:859 case BPF_TRACE_FENTRY:
795 case BPF_TRACE_FEXIT:860 case BPF_TRACE_FEXIT:
tools/lib/bpf/bpf.h+23−2
@@ -37,6 +37,18 @@ extern "C" { · 36 unmodified lines
37 37
38LIBBPF_API int libbpf_set_memlock_rlim(size_t memlock_bytes);38LIBBPF_API int libbpf_set_memlock_rlim(size_t memlock_bytes);
39 39
40struct bpf_log_opts {
41 size_t sz; /* size of this struct for forward/backward compatibility */
42
43 char *buf;
44 __u32 size;
45 __u32 level;
46 __u32 true_size; /* out parameter set by kernel */
47
48 size_t :0;
49};
50#define bpf_log_opts__last_field true_size
51
40struct bpf_map_create_opts {52struct bpf_map_create_opts {
41 size_t sz; /* size of this struct for forward/backward compatibility */53 size_t sz; /* size of this struct for forward/backward compatibility */
42 54
@@ -57,9 +69,12 @@ struct bpf_map_create_opts { · 14 unmodified lines
57 69
58 const void *excl_prog_hash;70 const void *excl_prog_hash;
59 __u32 excl_prog_hash_size;71 __u32 excl_prog_hash_size;
72
73 struct bpf_log_opts *log_opts;
74
60 size_t :0;75 size_t :0;
61};76};
62#define bpf_map_create_opts__last_field excl_prog_hash_size77#define bpf_map_create_opts__last_field log_opts
63 78
64LIBBPF_API int bpf_map_create(enum bpf_map_type map_type,79LIBBPF_API int bpf_map_create(enum bpf_map_type map_type,
65 const char *map_name,80 const char *map_name,
@@ -429,6 +444,7 @@ struct bpf_link_create_opts { · 363 unmodified lines
429 const unsigned long *ref_ctr_offsets;444 const unsigned long *ref_ctr_offsets;
430 const __u64 *cookies;445 const __u64 *cookies;
431 __u32 pid;446 __u32 pid;
447 __u32 path_fd;
432 } uprobe_multi;448 } uprobe_multi;
433 struct {449 struct {
434 __u64 cookie;450 __u64 cookie;
@@ -454,10 +470,15 @@ struct bpf_link_create_opts { · 19 unmodified lines
454 __u32 relative_id;470 __u32 relative_id;
455 __u64 expected_revision;471 __u64 expected_revision;
456 } cgroup;472 } cgroup;
473 struct {
474 const __u32 *ids;
475 const __u64 *cookies;
476 __u32 cnt;
477 } tracing_multi;
457 };478 };
458 size_t :0;479 size_t :0;
459};480};
460#define bpf_link_create_opts__last_field uprobe_multi.pid481#define bpf_link_create_opts__last_field uprobe_multi.path_fd
461 482
462LIBBPF_API int bpf_link_create(int prog_fd, int target_fd,483LIBBPF_API int bpf_link_create(int prog_fd, int target_fd,
463 enum bpf_attach_type attach_type,484 enum bpf_attach_type attach_type,
tools/lib/bpf/btf.c+35−19
@@ -421,7 +421,7 @@ static int btf_type_size_unknown(const struct btf *btf, const struct btf_type *t · 420 unmodified lines
421{421{
422 __u32 l_cnt = btf->hdr.layout_len / sizeof(struct btf_layout);422 __u32 l_cnt = btf->hdr.layout_len / sizeof(struct btf_layout);
423 struct btf_layout *l = btf->layout;423 struct btf_layout *l = btf->layout;
424 __u16 vlen = btf_vlen(t);424 __u32 vlen = btf_vlen(t);
425 __u32 kind = btf_kind(t);425 __u32 kind = btf_kind(t);
426 426
427 /* Fall back to base BTF if needed as they share layout information */427 /* Fall back to base BTF if needed as they share layout information */
@@ -454,7 +454,7 @@ static int btf_type_size_unknown(const struct btf *btf, const struct btf_type *t · 26 unmodified lines
454static int btf_type_size(const struct btf *btf, const struct btf_type *t)454static int btf_type_size(const struct btf *btf, const struct btf_type *t)
455{455{
456 const int base_size = sizeof(struct btf_type);456 const int base_size = sizeof(struct btf_type);
457 __u16 vlen = btf_vlen(t);457 __u32 vlen = btf_vlen(t);
458 458
459 switch (btf_kind(t)) {459 switch (btf_kind(t)) {
460 case BTF_KIND_FWD:460 case BTF_KIND_FWD:
@@ -506,7 +506,7 @@ static int btf_bswap_type_rest(struct btf_type *t) · 45 unmodified lines
506 struct btf_array *a;506 struct btf_array *a;
507 struct btf_param *p;507 struct btf_param *p;
508 struct btf_enum *e;508 struct btf_enum *e;
509 __u16 vlen = btf_vlen(t);509 __u32 vlen = btf_vlen(t);
510 int i;510 int i;
511 511
512 switch (btf_kind(t)) {512 switch (btf_kind(t)) {
@@ -1007,7 +1007,7 @@ int btf__align_of(const struct btf *btf, __u32 id) · 494 unmodified lines
1007 case BTF_KIND_STRUCT:1007 case BTF_KIND_STRUCT:
1008 case BTF_KIND_UNION: {1008 case BTF_KIND_UNION: {
1009 const struct btf_member *m = btf_members(t);1009 const struct btf_member *m = btf_members(t);
1010 __u16 vlen = btf_vlen(t);1010 __u32 vlen = btf_vlen(t);
1011 int i, max_align = 1, align;1011 int i, max_align = 1, align;
1012 1012
1013 for (i = 0; i < vlen; i++, m++) {1013 for (i = 0; i < vlen; i++, m++) {
@@ -2121,9 +2121,12 @@ static void *btf_add_type_mem(struct btf *btf, size_t add_sz) · 1107 unmodified lines
2121 btf->hdr.type_len, UINT_MAX, add_sz);2121 btf->hdr.type_len, UINT_MAX, add_sz);
2122}2122}
2123 2123
2124static void btf_type_inc_vlen(struct btf_type *t)2124static int btf_type_inc_vlen(struct btf_type *t)
2125{2125{
2126 if (btf_vlen(t) == BTF_MAX_VLEN)
2127 return -ENOSPC;
2126 t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, btf_kflag(t));2128 t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, btf_kflag(t));
2129 return 0;
2127}2130}
2128 2131
2129static void btf_hdr_update_type_len(struct btf *btf, int new_len)2132static void btf_hdr_update_type_len(struct btf *btf, int new_len)
@@ -2652,6 +2655,8 @@ int btf__add_field(struct btf *btf, const char *name, int type_id, · 522 unmodified lines
2652 t = btf_last_type(btf);2655 t = btf_last_type(btf);
2653 if (!btf_is_composite(t))2656 if (!btf_is_composite(t))
2654 return libbpf_err(-EINVAL);2657 return libbpf_err(-EINVAL);
2658 if (btf_vlen(t) == BTF_MAX_VLEN)
2659 return libbpf_err(-ENOSPC);
2655 2660
2656 if (validate_type_id(type_id))2661 if (validate_type_id(type_id))
2657 return libbpf_err(-EINVAL);2662 return libbpf_err(-EINVAL);
@@ -2686,6 +2691,7 @@ int btf__add_field(struct btf *btf, const char *name, int type_id, · 28 unmodified lines
2686 2691
2687 /* btf_add_type_mem can invalidate t pointer */2692 /* btf_add_type_mem can invalidate t pointer */
2688 t = btf_last_type(btf);2693 t = btf_last_type(btf);
2694
2689 /* update parent type's vlen and kflag */2695 /* update parent type's vlen and kflag */
2690 t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, is_bitfield || btf_kflag(t));2696 t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, is_bitfield || btf_kflag(t));
2691 2697
@@ -2796,7 +2802,9 @@ int btf__add_enum_value(struct btf *btf, const char *name, __s64 value) · 104 unmodified lines
2796 2802
2797 /* update parent type's vlen */2803 /* update parent type's vlen */
2798 t = btf_last_type(btf);2804 t = btf_last_type(btf);
2799 btf_type_inc_vlen(t);2805 err = btf_type_inc_vlen(t);
2806 if (err)
2807 return libbpf_err(err);
2800 2808
2801 /* if negative value, set signedness to signed */2809 /* if negative value, set signedness to signed */
2802 if (value < 0)2810 if (value < 0)
@@ -2873,7 +2881,9 @@ int btf__add_enum64_value(struct btf *btf, const char *name, __u64 value) · 70 unmodified lines
2873 2881
2874 /* update parent type's vlen */2882 /* update parent type's vlen */
2875 t = btf_last_type(btf);2883 t = btf_last_type(btf);
2876 btf_type_inc_vlen(t);2884 err = btf_type_inc_vlen(t);
2885 if (err)
2886 return libbpf_err(err);
2877 2887
2878 btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);2888 btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
2879 return 0;2889 return 0;
@@ -3115,7 +3125,9 @@ int btf__add_func_param(struct btf *btf, const char *name, int type_id) · 235 unmodified lines
3115 3125
3116 /* update parent type's vlen */3126 /* update parent type's vlen */
3117 t = btf_last_type(btf);3127 t = btf_last_type(btf);
3118 btf_type_inc_vlen(t);3128 err = btf_type_inc_vlen(t);
3129 if (err)
3130 return libbpf_err(err);
3119 3131
3120 btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);3132 btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
3121 return 0;3133 return 0;
@@ -3257,7 +3269,9 @@ int btf__add_datasec_var_info(struct btf *btf, int var_type_id, __u32 offset, __ · 135 unmodified lines
3257 3269
3258 /* update parent type's vlen */3270 /* update parent type's vlen */
3259 t = btf_last_type(btf);3271 t = btf_last_type(btf);
3260 btf_type_inc_vlen(t);3272 err = btf_type_inc_vlen(t);
3273 if (err)
3274 return libbpf_err(err);
3261 3275
3262 btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);3276 btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
3263 return 0;3277 return 0;
@@ -4311,7 +4325,7 @@ static long btf_hash_enum(struct btf_type *t) · 1047 unmodified lines
4311static bool btf_equal_enum_members(struct btf_type *t1, struct btf_type *t2)4325static bool btf_equal_enum_members(struct btf_type *t1, struct btf_type *t2)
4312{4326{
4313 const struct btf_enum *m1, *m2;4327 const struct btf_enum *m1, *m2;
4314 __u16 vlen;4328 __u32 vlen;
4315 int i;4329 int i;
4316 4330
4317 vlen = btf_vlen(t1);4331 vlen = btf_vlen(t1);
@@ -4329,7 +4343,7 @@ static bool btf_equal_enum_members(struct btf_type *t1, struct btf_type *t2) · 11 unmodified lines
4329static bool btf_equal_enum64_members(struct btf_type *t1, struct btf_type *t2)4343static bool btf_equal_enum64_members(struct btf_type *t1, struct btf_type *t2)
4330{4344{
4331 const struct btf_enum64 *m1, *m2;4345 const struct btf_enum64 *m1, *m2;
4332 __u16 vlen;4346 __u32 vlen;
4333 int i;4347 int i;
4334 4348
4335 vlen = btf_vlen(t1);4349 vlen = btf_vlen(t1);
@@ -4406,7 +4420,7 @@ static long btf_hash_struct(struct btf_type *t) · 70 unmodified lines
4406static bool btf_shallow_equal_struct(struct btf_type *t1, struct btf_type *t2)4420static bool btf_shallow_equal_struct(struct btf_type *t1, struct btf_type *t2)
4407{4421{
4408 const struct btf_member *m1, *m2;4422 const struct btf_member *m1, *m2;
4409 __u16 vlen;4423 __u32 vlen;
4410 int i;4424 int i;
4411 4425
4412 if (!btf_equal_common(t1, t2))4426 if (!btf_equal_common(t1, t2))
@@ -4482,7 +4496,7 @@ static bool btf_compat_array(struct btf_type *t1, struct btf_type *t2) · 69 unmodified lines
4482static long btf_hash_fnproto(struct btf_type *t)4496static long btf_hash_fnproto(struct btf_type *t)
4483{4497{
4484 const struct btf_param *member = btf_params(t);4498 const struct btf_param *member = btf_params(t);
4485 __u16 vlen = btf_vlen(t);4499 __u32 vlen = btf_vlen(t);
4486 long h = btf_hash_common(t);4500 long h = btf_hash_common(t);
4487 int i;4501 int i;
4488 4502
@@ -4504,7 +4518,7 @@ static long btf_hash_fnproto(struct btf_type *t) · 15 unmodified lines
4504static bool btf_equal_fnproto(struct btf_type *t1, struct btf_type *t2)4518static bool btf_equal_fnproto(struct btf_type *t1, struct btf_type *t2)
4505{4519{
4506 const struct btf_param *m1, *m2;4520 const struct btf_param *m1, *m2;
4507 __u16 vlen;4521 __u32 vlen;
4508 int i;4522 int i;
4509 4523
4510 if (!btf_equal_common(t1, t2))4524 if (!btf_equal_common(t1, t2))
@@ -4530,7 +4544,7 @@ static bool btf_equal_fnproto(struct btf_type *t1, struct btf_type *t2) · 19 unmodified lines
4530static bool btf_compat_fnproto(struct btf_type *t1, struct btf_type *t2)4544static bool btf_compat_fnproto(struct btf_type *t1, struct btf_type *t2)
4531{4545{
4532 const struct btf_param *m1, *m2;4546 const struct btf_param *m1, *m2;
4533 __u16 vlen;4547 __u32 vlen;
4534 int i;4548 int i;
4535 4549
4536 /* skip return type ID */4550 /* skip return type ID */
@@ -4578,12 +4592,14 @@ static int btf_dedup_prep(struct btf_dedup *d) · 41 unmodified lines
4578 case BTF_KIND_RESTRICT:4592 case BTF_KIND_RESTRICT:
4579 case BTF_KIND_PTR:4593 case BTF_KIND_PTR:
4580 case BTF_KIND_FWD:4594 case BTF_KIND_FWD:
4581 case BTF_KIND_TYPEDEF:
4582 case BTF_KIND_FUNC:4595 case BTF_KIND_FUNC:
4583 case BTF_KIND_FLOAT:4596 case BTF_KIND_FLOAT:
4584 case BTF_KIND_TYPE_TAG:4597 case BTF_KIND_TYPE_TAG:
4585 h = btf_hash_common(t);4598 h = btf_hash_common(t);
4586 break;4599 break;
4600 case BTF_KIND_TYPEDEF:
4601 h = btf_hash_typedef(t);
4602 break;
4587 case BTF_KIND_INT:4603 case BTF_KIND_INT:
4588 case BTF_KIND_DECL_TAG:4604 case BTF_KIND_DECL_TAG:
4589 h = btf_hash_int_decl_tag(t);4605 h = btf_hash_int_decl_tag(t);
@@ -5077,7 +5093,7 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id, · 487 unmodified lines
5077 case BTF_KIND_STRUCT:5093 case BTF_KIND_STRUCT:
5078 case BTF_KIND_UNION: {5094 case BTF_KIND_UNION: {
5079 const struct btf_member *cand_m, *canon_m;5095 const struct btf_member *cand_m, *canon_m;
5080 __u16 vlen;5096 __u32 vlen;
5081 5097
5082 if (!btf_shallow_equal_struct(cand_type, canon_type))5098 if (!btf_shallow_equal_struct(cand_type, canon_type))
5083 return 0;5099 return 0;
@@ -5105,7 +5121,7 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id, · 21 unmodified lines
5105 5121
5106 case BTF_KIND_FUNC_PROTO: {5122 case BTF_KIND_FUNC_PROTO: {
5107 const struct btf_param *cand_p, *canon_p;5123 const struct btf_param *cand_p, *canon_p;
5108 __u16 vlen;5124 __u32 vlen;
5109 5125
5110 if (!btf_compat_fnproto(cand_type, canon_type))5126 if (!btf_compat_fnproto(cand_type, canon_type))
5111 return 0;5127 return 0;
@@ -5439,7 +5455,7 @@ static int btf_dedup_ref_type(struct btf_dedup *d, __u32 type_id) · 327 unmodified lines
5439 5455
5440 case BTF_KIND_FUNC_PROTO: {5456 case BTF_KIND_FUNC_PROTO: {
5441 struct btf_param *param;5457 struct btf_param *param;
5442 __u16 vlen;5458 __u32 vlen;
5443 int i;5459 int i;
5444 5460
5445 ref_type_id = btf_dedup_ref_type(d, t->type);5461 ref_type_id = btf_dedup_ref_type(d, t->type);
tools/lib/bpf/btf.h+1−1
@@ -435,7 +435,7 @@ static inline __u16 btf_kind(const struct btf_type *t) · 434 unmodified lines
435 return BTF_INFO_KIND(t->info);435 return BTF_INFO_KIND(t->info);
436}436}
437 437
438static inline __u16 btf_vlen(const struct btf_type *t)438static inline __u32 btf_vlen(const struct btf_type *t)
439{439{
440 return BTF_INFO_VLEN(t->info);440 return BTF_INFO_VLEN(t->info);
441}441}
tools/lib/bpf/btf_dump.c+12−12
@@ -316,7 +316,7 @@ static int btf_dump_mark_referenced(struct btf_dump *d) · 315 unmodified lines
316{316{
317 int i, j, n = btf__type_cnt(d->btf);317 int i, j, n = btf__type_cnt(d->btf);
318 const struct btf_type *t;318 const struct btf_type *t;
319 __u16 vlen;319 __u32 vlen;
320 320
321 for (i = d->last_id + 1; i < n; i++) {321 for (i = d->last_id + 1; i < n; i++) {
322 t = btf__type_by_id(d->btf, i);322 t = btf__type_by_id(d->btf, i);
@@ -485,7 +485,7 @@ static int btf_dump_order_type(struct btf_dump *d, __u32 id, bool through_ptr) · 162 unmodified lines
485 */485 */
486 struct btf_dump_type_aux_state *tstate = &d->type_states[id];486 struct btf_dump_type_aux_state *tstate = &d->type_states[id];
487 const struct btf_type *t;487 const struct btf_type *t;
488 __u16 vlen;488 __u32 vlen;
489 int err, i;489 int err, i;
490 490
491 /* return true, letting typedefs know that it's ok to be emitted */491 /* return true, letting typedefs know that it's ok to be emitted */
@@ -798,7 +798,7 @@ static void btf_dump_emit_type(struct btf_dump *d, __u32 id, __u32 cont_id) · 306 unmodified lines
798 */798 */
799 if (top_level_def || t->name_off == 0) {799 if (top_level_def || t->name_off == 0) {
800 const struct btf_member *m = btf_members(t);800 const struct btf_member *m = btf_members(t);
801 __u16 vlen = btf_vlen(t);801 __u32 vlen = btf_vlen(t);
802 int i, new_cont_id;802 int i, new_cont_id;
803 803
804 new_cont_id = t->name_off == 0 ? cont_id : id;804 new_cont_id = t->name_off == 0 ? cont_id : id;
@@ -820,7 +820,7 @@ static void btf_dump_emit_type(struct btf_dump *d, __u32 id, __u32 cont_id) · 15 unmodified lines
820 break;820 break;
821 case BTF_KIND_FUNC_PROTO: {821 case BTF_KIND_FUNC_PROTO: {
822 const struct btf_param *p = btf_params(t);822 const struct btf_param *p = btf_params(t);
823 __u16 n = btf_vlen(t);823 __u32 n = btf_vlen(t);
824 int i;824 int i;
825 825
826 btf_dump_emit_type(d, t->type, cont_id);826 btf_dump_emit_type(d, t->type, cont_id);
@@ -839,7 +839,7 @@ static bool btf_is_struct_packed(const struct btf *btf, __u32 id, · 12 unmodified lines
839{839{
840 const struct btf_member *m;840 const struct btf_member *m;
841 int max_align = 1, align, i, bit_sz;841 int max_align = 1, align, i, bit_sz;
842 __u16 vlen;842 __u32 vlen;
843 843
844 m = btf_members(t);844 m = btf_members(t);
845 vlen = btf_vlen(t);845 vlen = btf_vlen(t);
@@ -973,7 +973,7 @@ static void btf_dump_emit_struct_def(struct btf_dump *d, · 127 unmodified lines
973 bool is_struct = btf_is_struct(t);973 bool is_struct = btf_is_struct(t);
974 bool packed, prev_bitfield = false;974 bool packed, prev_bitfield = false;
975 int align, i, off = 0;975 int align, i, off = 0;
976 __u16 vlen = btf_vlen(t);976 __u32 vlen = btf_vlen(t);
977 977
978 align = btf__align_of(d->btf, id);978 align = btf__align_of(d->btf, id);
979 packed = is_struct ? btf_is_struct_packed(d->btf, id, t) : 0;979 packed = is_struct ? btf_is_struct_packed(d->btf, id, t) : 0;
@@ -1064,7 +1064,7 @@ static void btf_dump_emit_enum_fwd(struct btf_dump *d, __u32 id, · 84 unmodified lines
1064 1064
1065static void btf_dump_emit_enum32_val(struct btf_dump *d,1065static void btf_dump_emit_enum32_val(struct btf_dump *d,
1066 const struct btf_type *t,1066 const struct btf_type *t,
1067 int lvl, __u16 vlen)1067 int lvl, __u32 vlen)
1068{1068{
1069 const struct btf_enum *v = btf_enum(t);1069 const struct btf_enum *v = btf_enum(t);
1070 bool is_signed = btf_kflag(t);1070 bool is_signed = btf_kflag(t);
@@ -1089,7 +1089,7 @@ static void btf_dump_emit_enum32_val(struct btf_dump *d, · 18 unmodified lines
1089 1089
1090static void btf_dump_emit_enum64_val(struct btf_dump *d,1090static void btf_dump_emit_enum64_val(struct btf_dump *d,
1091 const struct btf_type *t,1091 const struct btf_type *t,
1092 int lvl, __u16 vlen)1092 int lvl, __u32 vlen)
1093{1093{
1094 const struct btf_enum64 *v = btf_enum64(t);1094 const struct btf_enum64 *v = btf_enum64(t);
1095 bool is_signed = btf_kflag(t);1095 bool is_signed = btf_kflag(t);
@@ -1122,7 +1122,7 @@ static void btf_dump_emit_enum_def(struct btf_dump *d, __u32 id, · 26 unmodified lines
1122 const struct btf_type *t,1122 const struct btf_type *t,
1123 int lvl)1123 int lvl)
1124{1124{
1125 __u16 vlen = btf_vlen(t);1125 __u32 vlen = btf_vlen(t);
1126 1126
1127 btf_dump_printf(d, "enum%s%s",1127 btf_dump_printf(d, "enum%s%s",
1128 t->name_off ? " " : "",1128 t->name_off ? " " : "",
@@ -1542,7 +1542,7 @@ static void btf_dump_emit_type_chain(struct btf_dump *d, · 413 unmodified lines
1542 }1542 }
1543 case BTF_KIND_FUNC_PROTO: {1543 case BTF_KIND_FUNC_PROTO: {
1544 const struct btf_param *p = btf_params(t);1544 const struct btf_param *p = btf_params(t);
1545 __u16 vlen = btf_vlen(t);1545 __u32 vlen = btf_vlen(t);
1546 int i;1546 int i;
1547 1547
1548 /*1548 /*
@@ -2159,7 +2159,7 @@ static int btf_dump_struct_data(struct btf_dump *d, · 610 unmodified lines
2159 const void *data)2159 const void *data)
2160{2160{
2161 const struct btf_member *m = btf_members(t);2161 const struct btf_member *m = btf_members(t);
2162 __u16 n = btf_vlen(t);2162 __u32 n = btf_vlen(t);
2163 int i, err = 0;2163 int i, err = 0;
2164 2164
2165 /* note that we increment depth before calling btf_dump_print() below;2165 /* note that we increment depth before calling btf_dump_print() below;
@@ -2449,7 +2449,7 @@ static int btf_dump_type_data_check_zero(struct btf_dump *d, · 283 unmodified lines
2449 case BTF_KIND_STRUCT:2449 case BTF_KIND_STRUCT:
2450 case BTF_KIND_UNION: {2450 case BTF_KIND_UNION: {
2451 const struct btf_member *m = btf_members(t);2451 const struct btf_member *m = btf_members(t);
2452 __u16 n = btf_vlen(t);2452 __u32 n = btf_vlen(t);
2453 2453
2454 /* if any struct/union member is non-zero, the struct/union2454 /* if any struct/union member is non-zero, the struct/union
2455 * is considered non-zero and dumped.2455 * is considered non-zero and dumped.
tools/lib/bpf/features.c+8−0
@@ -615,6 +615,11 @@ static int probe_kern_btf_layout(int token_fd) · 614 unmodified lines
615 (char *)layout, token_fd));615 (char *)layout, token_fd));
616}616}
617 617
618static int probe_bpf_syscall_common_attrs(int token_fd)
619{
620 return probe_sys_bpf_ext();
621}
622
618typedef int (*feature_probe_fn)(int /* token_fd */);623typedef int (*feature_probe_fn)(int /* token_fd */);
619 624
620static struct kern_feature_cache feature_cache;625static struct kern_feature_cache feature_cache;
@@ -699,6 +704,9 @@ static struct kern_feature_desc { · 78 unmodified lines
699 [FEAT_BTF_LAYOUT] = {704 [FEAT_BTF_LAYOUT] = {
700 "kernel supports BTF layout", probe_kern_btf_layout,705 "kernel supports BTF layout", probe_kern_btf_layout,
701 },706 },
707 [FEAT_BPF_SYSCALL_COMMON_ATTRS] = {
708 "BPF syscall common attributes support", probe_bpf_syscall_common_attrs,
709 },
702};710};
703 711
704bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)712bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)
tools/lib/bpf/gen_loader.c+67−25
@@ -63,6 +63,7 @@ static int realloc_insn_buf(struct bpf_gen *gen, __u32 size) · 62 unmodified lines
63 gen->error = -ENOMEM;63 gen->error = -ENOMEM;
64 free(gen->insn_start);64 free(gen->insn_start);
65 gen->insn_start = NULL;65 gen->insn_start = NULL;
66 gen->insn_cur = NULL;
66 return -ENOMEM;67 return -ENOMEM;
67 }68 }
68 gen->insn_start = insn_start;69 gen->insn_start = insn_start;
@@ -86,6 +87,7 @@ static int realloc_data_buf(struct bpf_gen *gen, __u32 size) · 17 unmodified lines
86 gen->error = -ENOMEM;87 gen->error = -ENOMEM;
87 free(gen->data_start);88 free(gen->data_start);
88 gen->data_start = NULL;89 gen->data_start = NULL;
90 gen->data_cur = NULL;
89 return -ENOMEM;91 return -ENOMEM;
90 }92 }
91 gen->data_start = data_start;93 gen->data_start = data_start;
@@ -158,10 +160,16 @@ void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps · 66 unmodified lines
158 160
159static int add_data(struct bpf_gen *gen, const void *data, __u32 size)161static int add_data(struct bpf_gen *gen, const void *data, __u32 size)
160{162{
161 __u32 size8 = roundup(size, 8);
162 __u64 zero = 0;163 __u64 zero = 0;
164 __u32 size8;
163 void *prev;165 void *prev;
164 166
167 if (size > INT32_MAX) {
168 gen->error = -ERANGE;
169 return 0;
170 }
171 size8 = roundup(size, 8);
172
165 if (realloc_data_buf(gen, size8))173 if (realloc_data_buf(gen, size8))
166 return 0;174 return 0;
167 prev = gen->data_cur;175 prev = gen->data_cur;
@@ -293,7 +301,6 @@ static void emit_check_err(struct bpf_gen *gen) · 125 unmodified lines
293 emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, off));301 emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, off));
294 } else {302 } else {
295 gen->error = -ERANGE;303 gen->error = -ERANGE;
296 emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, -1));
297 }304 }
298}305}
299 306
@@ -398,13 +405,12 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps) · 98 unmodified lines
398 blob_fd_array_off(gen, i));405 blob_fd_array_off(gen, i));
399 emit(gen, BPF_MOV64_IMM(BPF_REG_0, 0));406 emit(gen, BPF_MOV64_IMM(BPF_REG_0, 0));
400 emit(gen, BPF_EXIT_INSN());407 emit(gen, BPF_EXIT_INSN());
401 if (OPTS_GET(gen->opts, gen_hash, false))
402 compute_sha_update_offsets(gen);
403
404 pr_debug("gen: finish %s\n", errstr(gen->error));
405 if (!gen->error) {408 if (!gen->error) {
406 struct gen_loader_opts *opts = gen->opts;409 struct gen_loader_opts *opts = gen->opts;
407 410
411 if (OPTS_GET(opts, gen_hash, false))
412 compute_sha_update_offsets(gen);
413
408 opts->insns = gen->insn_start;414 opts->insns = gen->insn_start;
409 opts->insns_sz = gen->insn_cur - gen->insn_start;415 opts->insns_sz = gen->insn_cur - gen->insn_start;
410 opts->data = gen->data_start;416 opts->data = gen->data_start;
@@ -419,6 +425,7 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps) · 8 unmodified lines
419 bpf_insn_bswap(insn++);425 bpf_insn_bswap(insn++);
420 }426 }
421 }427 }
428 pr_debug("gen: finish %s\n", errstr(gen->error));
422 return gen->error;429 return gen->error;
423}430}
424 431
@@ -545,13 +552,22 @@ void bpf_gen__map_create(struct bpf_gen *gen, · 120 unmodified lines
545 default:552 default:
546 break;553 break;
547 }554 }
548 /* conditionally update max_entries */555
549 if (map_idx >= 0)556 /*
557 * Conditionally update max_entries from the host-supplied loader
558 * ctx. This sizes the map at runtime, but for a signed loader
559 * (gen_hash) it would let an untrusted host re-dimension the
560 * program's maps after emit_signature_match(), outside what the
561 * signature attests to. Keep the signer-provided max_entries
562 * baked into the blob in that case.
563 */
564 if (map_idx >= 0 && !OPTS_GET(gen->opts, gen_hash, false))
550 move_ctx2blob(gen, attr_field(map_create_attr, max_entries), 4,565 move_ctx2blob(gen, attr_field(map_create_attr, max_entries), 4,
551 sizeof(struct bpf_loader_ctx) +566 sizeof(struct bpf_loader_ctx) +
552 sizeof(struct bpf_map_desc) * map_idx +567 sizeof(struct bpf_map_desc) * map_idx +
553 offsetof(struct bpf_map_desc, max_entries),568 offsetof(struct bpf_map_desc, max_entries),
554 true /* check that max_entries != 0 */);569 true /* check that max_entries != 0 */);
570
555 /* emit MAP_CREATE command */571 /* emit MAP_CREATE command */
556 emit_sys_bpf(gen, BPF_MAP_CREATE, map_create_attr, attr_size);572 emit_sys_bpf(gen, BPF_MAP_CREATE, map_create_attr, attr_size);
557 debug_ret(gen, "map_create %s idx %d type %d value_size %d value_btf_id %d",573 debug_ret(gen, "map_create %s idx %d type %d value_size %d value_btf_id %d",
@@ -585,6 +601,23 @@ static void emit_signature_match(struct bpf_gen *gen) · 27 unmodified lines
585 __s64 off;601 __s64 off;
586 int i;602 int i;
587 603
604 /*
605 * Reject if the metadata map is not exclusive. Without exclusivity
606 * the cached map->sha[] verified above can be stale: another BPF
607 * program with map access could have mutated the contents between
608 * BPF_OBJ_GET_INFO_BY_FD and loader execution.
609 */
610 emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX,
611 0, 0, 0, 0));
612 emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, SHA256_DIGEST_LENGTH));
613 off = -(gen->insn_cur - gen->insn_start - gen->cleanup_label) / 8 - 2;
614 if (is_simm16(off)) {
615 emit(gen, BPF_MOV64_IMM(BPF_REG_7, -EINVAL));
616 emit(gen, BPF_JMP_IMM(BPF_JNE, BPF_REG_2, 1, off));
617 } else {
618 gen->error = -ERANGE;
619 }
620
588 for (i = 0; i < SHA256_DWORD_SIZE; i++) {621 for (i = 0; i < SHA256_DWORD_SIZE; i++) {
589 emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX,622 emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX,
590 0, 0, 0, 0));623 0, 0, 0, 0));
@@ -1053,7 +1086,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, · 462 unmodified lines
1053 prog_idx, prog_type, insns_off, insn_cnt, license_off);1086 prog_idx, prog_type, insns_off, insn_cnt, license_off);
1054 1087
1055 /* convert blob insns to target endianness */1088 /* convert blob insns to target endianness */
1056 if (gen->swapped_endian) {1089 if (gen->swapped_endian && !gen->error) {
1057 struct bpf_insn *insn = gen->data_start + insns_off;1090 struct bpf_insn *insn = gen->data_start + insns_off;
1058 int i;1091 int i;
1059 1092
@@ -1091,7 +1124,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, · 31 unmodified lines
1091 sizeof(struct bpf_core_relo));1124 sizeof(struct bpf_core_relo));
1092 1125
1093 /* convert all info blobs to target endianness */1126 /* convert all info blobs to target endianness */
1094 if (gen->swapped_endian)1127 if (gen->swapped_endian && !gen->error)
1095 info_blob_bswap(gen, func_info, line_info, core_relos, load_attr);1128 info_blob_bswap(gen, func_info, line_info, core_relos, load_attr);
1096 1129
1097 libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name));1130 libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name));
@@ -1169,27 +1202,36 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue, · 71 unmodified lines
1169 value = add_data(gen, pvalue, value_size);1202 value = add_data(gen, pvalue, value_size);
1170 key = add_data(gen, &zero, sizeof(zero));1203 key = add_data(gen, &zero, sizeof(zero));
1171 1204
1172 /* if (map_desc[map_idx].initial_value) {1205 /*
1206 * if (map_desc[map_idx].initial_value) {
1173 * if (ctx->flags & BPF_SKEL_KERNEL)1207 * if (ctx->flags & BPF_SKEL_KERNEL)
1174 * bpf_probe_read_kernel(value, value_size, initial_value);1208 * bpf_probe_read_kernel(value, value_size, initial_value);
1175 * else1209 * else
1176 * bpf_copy_from_user(value, value_size, initial_value);1210 * bpf_copy_from_user(value, value_size, initial_value);
1177 * }1211 * }
1212 *
1213 * The runtime initial_value comes from the host-supplied loader
1214 * ctx and would overwrite the blob value after emit_signature_match()
1215 * has already validated map->sha[]. For a signed loader (gen_hash)
1216 * the attested blob value must be authoritative, so skip the override
1217 * and leave the hashed value in place.
1178 */1218 */
1179 emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6,1219 if (!OPTS_GET(gen->opts, gen_hash, false)) {
1180 sizeof(struct bpf_loader_ctx) +1220 emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6,
1181 sizeof(struct bpf_map_desc) * map_idx +1221 sizeof(struct bpf_loader_ctx) +
1182 offsetof(struct bpf_map_desc, initial_value)));1222 sizeof(struct bpf_map_desc) * map_idx +
1183 emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 8));1223 offsetof(struct bpf_map_desc, initial_value)));
1184 emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,1224 emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 8));
1185 0, 0, 0, value));1225 emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
1186 emit(gen, BPF_MOV64_IMM(BPF_REG_2, value_size));1226 0, 0, 0, value));
1187 emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6,1227 emit(gen, BPF_MOV64_IMM(BPF_REG_2, value_size));
1188 offsetof(struct bpf_loader_ctx, flags)));1228 emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6,
1189 emit(gen, BPF_JMP_IMM(BPF_JSET, BPF_REG_0, BPF_SKEL_KERNEL, 2));1229 offsetof(struct bpf_loader_ctx, flags)));
1190 emit(gen, BPF_EMIT_CALL(BPF_FUNC_copy_from_user));1230 emit(gen, BPF_JMP_IMM(BPF_JSET, BPF_REG_0, BPF_SKEL_KERNEL, 2));
1191 emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, 1));1231 emit(gen, BPF_EMIT_CALL(BPF_FUNC_copy_from_user));
1192 emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel));1232 emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, 1));
1233 emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel));
1234 }
1193 1235
1194 map_update_attr = add_data(gen, &attr, attr_size);1236 map_update_attr = add_data(gen, &attr, attr_size);
1195 pr_debug("gen: map_update_elem: idx %d, value: off %d size %d, attr: off %d size %d\n",1237 pr_debug("gen: map_update_elem: idx %d, value: off %d size %d, attr: off %d size %d\n",
tools/lib/bpf/libbpf.c+437−36
@@ -136,6 +136,9 @@ static const char * const attach_type_name[] = { · 135 unmodified lines
136 [BPF_NETKIT_PEER] = "netkit_peer",136 [BPF_NETKIT_PEER] = "netkit_peer",
137 [BPF_TRACE_KPROBE_SESSION] = "trace_kprobe_session",137 [BPF_TRACE_KPROBE_SESSION] = "trace_kprobe_session",
138 [BPF_TRACE_UPROBE_SESSION] = "trace_uprobe_session",138 [BPF_TRACE_UPROBE_SESSION] = "trace_uprobe_session",
139 [BPF_TRACE_FENTRY_MULTI] = "trace_fentry_multi",
140 [BPF_TRACE_FEXIT_MULTI] = "trace_fexit_multi",
141 [BPF_TRACE_FSESSION_MULTI] = "trace_fsession_multi",
139};142};
140 143
141static const char * const link_type_name[] = {144static const char * const link_type_name[] = {
@@ -154,6 +157,7 @@ static const char * const link_type_name[] = { · 12 unmodified lines
154 [BPF_LINK_TYPE_UPROBE_MULTI] = "uprobe_multi",157 [BPF_LINK_TYPE_UPROBE_MULTI] = "uprobe_multi",
155 [BPF_LINK_TYPE_NETKIT] = "netkit",158 [BPF_LINK_TYPE_NETKIT] = "netkit",
156 [BPF_LINK_TYPE_SOCKMAP] = "sockmap",159 [BPF_LINK_TYPE_SOCKMAP] = "sockmap",
160 [BPF_LINK_TYPE_TRACING_MULTI] = "tracing_multi",
157};161};
158 162
159static const char * const map_type_name[] = {163static const char * const map_type_name[] = {
@@ -192,6 +196,7 @@ static const char * const map_type_name[] = { · 32 unmodified lines
192 [BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage",196 [BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage",
193 [BPF_MAP_TYPE_ARENA] = "arena",197 [BPF_MAP_TYPE_ARENA] = "arena",
194 [BPF_MAP_TYPE_INSN_ARRAY] = "insn_array",198 [BPF_MAP_TYPE_INSN_ARRAY] = "insn_array",
199 [BPF_MAP_TYPE_RHASH] = "rhash",
195};200};
196 201
197static const char * const prog_type_name[] = {202static const char * const prog_type_name[] = {
@@ -7767,6 +7772,69 @@ static int bpf_object__sanitize_prog(struct bpf_object *obj, struct bpf_program · 7569 unmodified lines
7767static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attach_name,7772static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attach_name,
7768 int *btf_obj_fd, int *btf_type_id);7773 int *btf_obj_fd, int *btf_type_id);
7769 7774
7775static inline bool is_tracing_multi(enum bpf_attach_type type)
7776{
7777 return type == BPF_TRACE_FENTRY_MULTI || type == BPF_TRACE_FEXIT_MULTI ||
7778 type == BPF_TRACE_FSESSION_MULTI;
7779}
7780
7781static const struct module_btf *find_attach_module(struct bpf_object *obj, const char *attach)
7782{
7783 const char *sep, *mod_name = NULL;
7784 int i, mod_len, err;
7785
7786 /*
7787 * We expect attach string in the form of either
7788 * - function_pattern or
7789 * - <module>:function_pattern
7790 */
7791 sep = strchr(attach, ':');
7792 if (sep) {
7793 mod_name = attach;
7794 mod_len = sep - mod_name;
7795 }
7796 if (!mod_name)
7797 return NULL;
7798
7799 err = load_module_btfs(obj);
7800 if (err)
7801 return NULL;
7802
7803 for (i = 0; i < obj->btf_module_cnt; i++) {
7804 const struct module_btf *mod = &obj->btf_modules[i];
7805
7806 if (strncmp(mod->name, mod_name, mod_len) == 0 && mod->name[mod_len] == '\0')
7807 return mod;
7808 }
7809 return NULL;
7810}
7811
7812static int tracing_multi_mod_fd(struct bpf_program *prog, int *btf_obj_fd)
7813{
7814 const char *attach_name, *sep;
7815 const struct module_btf *mod;
7816
7817 *btf_obj_fd = 0;
7818 attach_name = strchr(prog->sec_name, '/');
7819
7820 /* Program with no details in spec, using kernel btf. */
7821 if (!attach_name)
7822 return 0;
7823
7824 /* Program with no module section, using kernel btf. */
7825 sep = strchr(++attach_name, ':');
7826 if (!sep)
7827 return 0;
7828
7829 /* Program with module specified, get its btf fd. */
7830 mod = find_attach_module(prog->obj, attach_name);
7831 if (!mod)
7832 return -EINVAL;
7833
7834 *btf_obj_fd = mod->fd;
7835 return 0;
7836}
7837
7770/* this is called as prog->sec_def->prog_prepare_load_fn for libbpf-supported sec_defs */7838/* this is called as prog->sec_def->prog_prepare_load_fn for libbpf-supported sec_defs */
7771static int libbpf_prepare_prog_load(struct bpf_program *prog,7839static int libbpf_prepare_prog_load(struct bpf_program *prog,
7772 struct bpf_prog_load_opts *opts, long cookie)7840 struct bpf_prog_load_opts *opts, long cookie)
@@ -7830,6 +7898,18 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog, · 57 unmodified lines
7830 opts->attach_btf_obj_fd = btf_obj_fd;7898 opts->attach_btf_obj_fd = btf_obj_fd;
7831 opts->attach_btf_id = btf_type_id;7899 opts->attach_btf_id = btf_type_id;
7832 }7900 }
7901
7902 if (is_tracing_multi(prog->expected_attach_type)) {
7903 int err, btf_obj_fd = 0;
7904
7905 err = tracing_multi_mod_fd(prog, &btf_obj_fd);
7906 if (err < 0)
7907 return err;
7908
7909 prog->attach_btf_obj_fd = btf_obj_fd;
7910 opts->attach_btf_obj_fd = btf_obj_fd;
7911 }
7912
7833 return 0;7913 return 0;
7834}7914}
7835 7915
@@ -8936,13 +9016,10 @@ static void bpf_object_unpin(struct bpf_object *obj) · 1100 unmodified lines
8936 bpf_map__unpin(&obj->maps[i], NULL);9016 bpf_map__unpin(&obj->maps[i], NULL);
8937}9017}
8938 9018
8939static void bpf_object_post_load_cleanup(struct bpf_object *obj)9019static void bpf_object_cleanup_btf(struct bpf_object *obj)
8940{9020{
8941 int i;9021 int i;
8942 9022
8943 /* clean up fd_array */
8944 zfree(&obj->fd_array);
8945
8946 /* clean up module BTFs */9023 /* clean up module BTFs */
8947 for (i = 0; i < obj->btf_module_cnt; i++) {9024 for (i = 0; i < obj->btf_module_cnt; i++) {
8948 close(obj->btf_modules[i].fd);9025 close(obj->btf_modules[i].fd);
@@ -8950,6 +9027,8 @@ static void bpf_object_post_load_cleanup(struct bpf_object *obj) · 1 unmodified line
8950 free(obj->btf_modules[i].name);9027 free(obj->btf_modules[i].name);
8951 }9028 }
8952 obj->btf_module_cnt = 0;9029 obj->btf_module_cnt = 0;
9030 obj->btf_module_cap = 0;
9031 obj->btf_modules_loaded = false;
8953 zfree(&obj->btf_modules);9032 zfree(&obj->btf_modules);
8954 9033
8955 /* clean up vmlinux BTF */9034 /* clean up vmlinux BTF */
@@ -8957,6 +9036,15 @@ static void bpf_object_post_load_cleanup(struct bpf_object *obj) · 1 unmodified line
8957 obj->btf_vmlinux = NULL;9036 obj->btf_vmlinux = NULL;
8958}9037}
8959 9038
9039static void bpf_object_post_load_cleanup(struct bpf_object *obj)
9040{
9041 /* clean up fd_array */
9042 zfree(&obj->fd_array);
9043
9044 /* clean up BTF */
9045 bpf_object_cleanup_btf(obj);
9046}
9047
8960static int bpf_object_prepare(struct bpf_object *obj, const char *target_btf_path)9048static int bpf_object_prepare(struct bpf_object *obj, const char *target_btf_path)
8961{9049{
8962 int err;9050 int err;
@@ -9983,6 +10071,7 @@ static int attach_kprobe_session(const struct bpf_program *prog, long cookie, st · 1020 unmodified lines
9983static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);10071static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
9984static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link);10072static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link);
9985static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link);10073static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link);
10074static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
9986 10075
9987static const struct bpf_sec_def section_defs[] = {10076static const struct bpf_sec_def section_defs[] = {
9988 SEC_DEF("socket", SOCKET_FILTER, 0, SEC_NONE),10077 SEC_DEF("socket", SOCKET_FILTER, 0, SEC_NONE),
@@ -10018,11 +10107,16 @@ static const struct bpf_sec_def section_defs[] = { · 29 unmodified lines
10018 SEC_DEF("netkit/peer", SCHED_CLS, BPF_NETKIT_PEER, SEC_NONE),10107 SEC_DEF("netkit/peer", SCHED_CLS, BPF_NETKIT_PEER, SEC_NONE),
10019 SEC_DEF("tracepoint+", TRACEPOINT, 0, SEC_NONE, attach_tp),10108 SEC_DEF("tracepoint+", TRACEPOINT, 0, SEC_NONE, attach_tp),
10020 SEC_DEF("tp+", TRACEPOINT, 0, SEC_NONE, attach_tp),10109 SEC_DEF("tp+", TRACEPOINT, 0, SEC_NONE, attach_tp),
10110 SEC_DEF("tracepoint.s+", TRACEPOINT, 0, SEC_SLEEPABLE, attach_tp),
10111 SEC_DEF("tp.s+", TRACEPOINT, 0, SEC_SLEEPABLE, attach_tp),
10021 SEC_DEF("raw_tracepoint+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp),10112 SEC_DEF("raw_tracepoint+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp),
10022 SEC_DEF("raw_tp+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp),10113 SEC_DEF("raw_tp+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp),
10114 SEC_DEF("raw_tracepoint.s+", RAW_TRACEPOINT, 0, SEC_SLEEPABLE, attach_raw_tp),
10115 SEC_DEF("raw_tp.s+", RAW_TRACEPOINT, 0, SEC_SLEEPABLE, attach_raw_tp),
10023 SEC_DEF("raw_tracepoint.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp),10116 SEC_DEF("raw_tracepoint.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp),
10024 SEC_DEF("raw_tp.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp),10117 SEC_DEF("raw_tp.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp),
10025 SEC_DEF("tp_btf+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF, attach_trace),10118 SEC_DEF("tp_btf+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF, attach_trace),
10119 SEC_DEF("tp_btf.s+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
10026 SEC_DEF("fentry+", TRACING, BPF_TRACE_FENTRY, SEC_ATTACH_BTF, attach_trace),10120 SEC_DEF("fentry+", TRACING, BPF_TRACE_FENTRY, SEC_ATTACH_BTF, attach_trace),
10027 SEC_DEF("fmod_ret+", TRACING, BPF_MODIFY_RETURN, SEC_ATTACH_BTF, attach_trace),10121 SEC_DEF("fmod_ret+", TRACING, BPF_MODIFY_RETURN, SEC_ATTACH_BTF, attach_trace),
10028 SEC_DEF("fexit+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF, attach_trace),10122 SEC_DEF("fexit+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF, attach_trace),
@@ -10031,6 +10125,12 @@ static const struct bpf_sec_def section_defs[] = { · 2 unmodified lines
10031 SEC_DEF("fexit.s+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),10125 SEC_DEF("fexit.s+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
10032 SEC_DEF("fsession+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF, attach_trace),10126 SEC_DEF("fsession+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF, attach_trace),
10033 SEC_DEF("fsession.s+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),10127 SEC_DEF("fsession.s+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
10128 SEC_DEF("fsession.multi+", TRACING, BPF_TRACE_FSESSION_MULTI, 0, attach_tracing_multi),
10129 SEC_DEF("fsession.multi.s+", TRACING, BPF_TRACE_FSESSION_MULTI, SEC_SLEEPABLE, attach_tracing_multi),
10130 SEC_DEF("fentry.multi+", TRACING, BPF_TRACE_FENTRY_MULTI, 0, attach_tracing_multi),
10131 SEC_DEF("fexit.multi+", TRACING, BPF_TRACE_FEXIT_MULTI, 0, attach_tracing_multi),
10132 SEC_DEF("fentry.multi.s+", TRACING, BPF_TRACE_FENTRY_MULTI, SEC_SLEEPABLE, attach_tracing_multi),
10133 SEC_DEF("fexit.multi.s+", TRACING, BPF_TRACE_FEXIT_MULTI, SEC_SLEEPABLE, attach_tracing_multi),
10034 SEC_DEF("freplace+", EXT, 0, SEC_ATTACH_BTF, attach_trace),10134 SEC_DEF("freplace+", EXT, 0, SEC_ATTACH_BTF, attach_trace),
10035 SEC_DEF("lsm+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm),10135 SEC_DEF("lsm+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm),
10036 SEC_DEF("lsm.s+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm),10136 SEC_DEF("lsm.s+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm),
@@ -12280,7 +12380,7 @@ error: · 2243 unmodified lines
12280static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf_link **link)12380static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf_link **link)
12281{12381{
12282 DECLARE_LIBBPF_OPTS(bpf_kprobe_opts, opts);12382 DECLARE_LIBBPF_OPTS(bpf_kprobe_opts, opts);
12283 unsigned long offset = 0;12383 long offset = 0;
12284 const char *func_name;12384 const char *func_name;
12285 char *func;12385 char *func;
12286 int n;12386 int n;
@@ -12302,6 +12402,13 @@ static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf · 15 unmodified lines
12302 pr_warn("kprobe name is invalid: %s\n", func_name);12402 pr_warn("kprobe name is invalid: %s\n", func_name);
12303 return -EINVAL;12403 return -EINVAL;
12304 }12404 }
12405
12406 if (offset < 0) {
12407 free(func);
12408 pr_warn("kprobe offset must be a non-negative integer: %li\n", offset);
12409 return -EINVAL;
12410 }
12411
12305 if (opts.retprobe && offset != 0) {12412 if (opts.retprobe && offset != 0) {
12306 free(func);12413 free(func);
12307 pr_warn("kretprobes do not support offset specification\n");12414 pr_warn("kretprobes do not support offset specification\n");
@@ -12425,6 +12532,279 @@ static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, stru · 117 unmodified lines
12425 return ret;12532 return ret;
12426}12533}
12427 12534
12535#define MAX_BPF_FUNC_ARGS 12
12536
12537static bool btf_type_is_modifier(const struct btf_type *t)
12538{
12539 switch (BTF_INFO_KIND(t->info)) {
12540 case BTF_KIND_TYPEDEF:
12541 case BTF_KIND_VOLATILE:
12542 case BTF_KIND_CONST:
12543 case BTF_KIND_RESTRICT:
12544 case BTF_KIND_TYPE_TAG:
12545 return true;
12546 default:
12547 return false;
12548 }
12549}
12550
12551#define MAX_RESOLVE_DEPTH 32
12552
12553static int btf_get_type_size(const struct btf *btf, __u32 type_id,
12554 const struct btf_type **ret_type)
12555{
12556 const struct btf_type *t;
12557 int i;
12558
12559 *ret_type = btf__type_by_id(btf, 0);
12560 if (!type_id)
12561 return 0;
12562 t = btf__type_by_id(btf, type_id);
12563 for (i = 0; i < MAX_RESOLVE_DEPTH && t && btf_type_is_modifier(t); i++)
12564 t = btf__type_by_id(btf, t->type);
12565 if (!t || i == MAX_RESOLVE_DEPTH)
12566 return -EINVAL;
12567 *ret_type = t;
12568 if (btf_is_ptr(t))
12569 return btf__pointer_size(btf);
12570 if (btf_is_int(t) || btf_is_any_enum(t) || btf_is_struct(t) || btf_is_union(t))
12571 return t->size;
12572 return -EINVAL;
12573}
12574
12575bool btf_type_is_traceable_func(const struct btf *btf, const struct btf_type *t)
12576{
12577 const struct btf_param *args;
12578 const struct btf_type *proto;
12579 __u32 i, nargs;
12580 int ret;
12581
12582 if (!btf_is_func(t))
12583 return false;
12584 proto = btf__type_by_id(btf, t->type);
12585 if (!proto || !btf_is_func_proto(proto))
12586 return false;
12587
12588 args = (const struct btf_param *)(proto + 1);
12589 nargs = btf_vlen(proto);
12590 if (nargs > MAX_BPF_FUNC_ARGS)
12591 return false;
12592
12593 /* No support for struct return type. */
12594 ret = btf_get_type_size(btf, proto->type, &t);
12595 if (ret < 0 || btf_is_struct(t) || btf_is_union(t))
12596 return false;
12597
12598 for (i = 0; i < nargs; i++) {
12599 /* No support for variable args. */
12600 if (i == nargs - 1 && args[i].type == 0)
12601 return false;
12602 ret = btf_get_type_size(btf, args[i].type, &t);
12603 /* No support of struct argument size greater than 16 bytes. */
12604 if (ret < 0 || ret > 16)
12605 return false;
12606 /* No support for void argument. */
12607 if (ret == 0)
12608 return false;
12609 }
12610
12611 return true;
12612}
12613
12614static int
12615collect_btf_func_ids_by_glob(const struct btf *btf, const char *pattern, __u32 **ids)
12616{
12617 __u32 type_id, nr_types = btf__type_cnt(btf);
12618 size_t cap = 0, cnt = 0;
12619
12620 if (!pattern)
12621 return -EINVAL;
12622
12623 for (type_id = 1; type_id < nr_types; type_id++) {
12624 const struct btf_type *t = btf__type_by_id(btf, type_id);
12625 const char *name;
12626 int err;
12627
12628 if (btf_kind(t) != BTF_KIND_FUNC)
12629 continue;
12630 name = btf__name_by_offset(btf, t->name_off);
12631 if (!name)
12632 continue;
12633
12634 if (!glob_match(name, pattern))
12635 continue;
12636 if (!btf_type_is_traceable_func(btf, t))
12637 continue;
12638
12639 err = libbpf_ensure_mem((void **) ids, &cap, sizeof(**ids), cnt + 1);
12640 if (err) {
12641 free(*ids);
12642 return -ENOMEM;
12643 }
12644 (*ids)[cnt++] = type_id;
12645 }
12646
12647 return cnt;
12648}
12649
12650static int collect_func_ids_by_glob(const struct bpf_program *prog, const char *pattern, __u32 **ids)
12651{
12652 struct bpf_object *obj = prog->obj;
12653 const struct module_btf *mod;
12654 struct btf *btf = NULL;
12655 const char *sep;
12656 int err;
12657
12658 err = bpf_object__load_vmlinux_btf(obj, true);
12659 if (err)
12660 return err;
12661
12662 /* In case we have module specified, we will find its btf and use that. */
12663 sep = strchr(pattern, ':');
12664 if (sep) {
12665 mod = find_attach_module(obj, pattern);
12666 if (!mod) {
12667 err = -EINVAL;
12668 goto cleanup;
12669 }
12670 btf = mod->btf;
12671 pattern = sep + 1;
12672 } else {
12673 /* Program is loaded for kernel module. */
12674 if (prog->attach_btf_obj_fd) {
12675 err = -EINVAL;
12676 goto cleanup;
12677 }
12678 btf = obj->btf_vmlinux;
12679 }
12680
12681 err = collect_btf_func_ids_by_glob(btf, pattern, ids);
12682
12683cleanup:
12684 bpf_object_cleanup_btf(obj);
12685 return err;
12686}
12687
12688struct bpf_link *
12689bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern,
12690 const struct bpf_tracing_multi_opts *opts)
12691{
12692 LIBBPF_OPTS(bpf_link_create_opts, lopts);
12693 int prog_fd, link_fd, err, cnt;
12694 __u32 *free_ids = NULL;
12695 struct bpf_link *link;
12696 const __u64 *cookies;
12697 const __u32 *ids;
12698
12699 if (!OPTS_VALID(opts, bpf_tracing_multi_opts))
12700 return libbpf_err_ptr(-EINVAL);
12701
12702 prog_fd = bpf_program__fd(prog);
12703 if (prog_fd < 0) {
12704 pr_warn("prog '%s': can't attach BPF program without FD (was it loaded?)\n",
12705 prog->name);
12706 return libbpf_err_ptr(-EINVAL);
12707 }
12708
12709 cnt = OPTS_GET(opts, cnt, 0);
12710 ids = OPTS_GET(opts, ids, NULL);
12711 cookies = OPTS_GET(opts, cookies, NULL);
12712
12713 if (!!ids != !!cnt)
12714 return libbpf_err_ptr(-EINVAL);
12715 if (pattern && (ids || cookies))
12716 return libbpf_err_ptr(-EINVAL);
12717 if (!pattern && !ids)
12718 return libbpf_err_ptr(-EINVAL);
12719
12720 if (pattern) {
12721 cnt = collect_func_ids_by_glob(prog, pattern, &free_ids);
12722 if (cnt < 0)
12723 return libbpf_err_ptr(cnt);
12724 if (cnt == 0)
12725 return libbpf_err_ptr(-EINVAL);
12726 ids = (const __u32 *) free_ids;
12727 }
12728
12729 lopts.tracing_multi.ids = ids;
12730 lopts.tracing_multi.cookies = cookies;
12731 lopts.tracing_multi.cnt = cnt;
12732
12733 link = calloc(1, sizeof(*link));
12734 if (!link) {
12735 err = -ENOMEM;
12736 goto error;
12737 }
12738 link->detach = &bpf_link__detach_fd;
12739
12740 link_fd = bpf_link_create(prog_fd, 0, prog->expected_attach_type, &lopts);
12741 if (link_fd < 0) {
12742 err = -errno;
12743 pr_warn("prog '%s': failed to attach: %s\n", prog->name, errstr(err));
12744 goto error;
12745 }
12746 link->fd = link_fd;
12747 free(free_ids);
12748 return link;
12749
12750error:
12751 free(link);
12752 free(free_ids);
12753 return libbpf_err_ptr(err);
12754}
12755
12756static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link)
12757{
12758 static const char *const prefixes[] = {
12759 "fentry.multi",
12760 "fexit.multi",
12761 "fsession.multi",
12762 "fentry.multi.s",
12763 "fexit.multi.s",
12764 "fsession.multi.s",
12765 };
12766 const char *spec = NULL;
12767 char *pattern;
12768 size_t i;
12769 int n;
12770
12771 *link = NULL;
12772
12773 for (i = 0; i < ARRAY_SIZE(prefixes); i++) {
12774 size_t pfx_len;
12775
12776 if (!str_has_pfx(prog->sec_name, prefixes[i]))
12777 continue;
12778
12779 pfx_len = strlen(prefixes[i]);
12780 /* no auto-attach case of, e.g., SEC("fentry.multi") */
12781 if (prog->sec_name[pfx_len] == '\0')
12782 return 0;
12783
12784 if (prog->sec_name[pfx_len] != '/')
12785 continue;
12786
12787 spec = prog->sec_name + pfx_len + 1;
12788 break;
12789 }
12790
12791 if (!spec) {
12792 pr_warn("prog '%s': invalid section name '%s'\n",
12793 prog->name, prog->sec_name);
12794 return -EINVAL;
12795 }
12796
12797 n = sscanf(spec, "%m[a-zA-Z0-9_.*?:]", &pattern);
12798 if (n < 1) {
12799 pr_warn("tracing multi pattern is invalid: %s\n", spec);
12800 return -EINVAL;
12801 }
12802
12803 *link = bpf_program__attach_tracing_multi(prog, pattern, NULL);
12804 free(pattern);
12805 return libbpf_get_error(*link);
12806}
12807
12428static inline int add_uprobe_event_legacy(const char *probe_name, bool retprobe,12808static inline int add_uprobe_event_legacy(const char *probe_name, bool retprobe,
12429 const char *binary_path, size_t offset)12809 const char *binary_path, size_t offset)
12430{12810{
@@ -13145,25 +13525,61 @@ struct bpf_link *bpf_program__attach_tracepoint(const struct bpf_program *prog, · 714 unmodified lines
13145 return bpf_program__attach_tracepoint_opts(prog, tp_category, tp_name, NULL);13525 return bpf_program__attach_tracepoint_opts(prog, tp_category, tp_name, NULL);
13146}13526}
13147 13527
13528/*
13529 * Match section name against a prefix array. Returns pointer past
13530 * "prefix/" on match, empty string for bare sections (exact prefix
13531 * match), or NULL if no prefix matches.
13532 */
13533static const char *sec_name_match_prefix(const char *sec_name,
13534 const char *const *prefixes,
13535 size_t n)
13536{
13537 size_t i;
13538
13539 for (i = 0; i < n; i++) {
13540 size_t pfx_len;
13541
13542 if (!str_has_pfx(sec_name, prefixes[i]))
13543 continue;
13544
13545 pfx_len = strlen(prefixes[i]);
13546 if (sec_name[pfx_len] == '\0')
13547 return sec_name + pfx_len;
13548
13549 if (sec_name[pfx_len] != '/' || sec_name[pfx_len + 1] == '\0')
13550 continue;
13551
13552 return sec_name + pfx_len + 1;
13553 }
13554 return NULL;
13555}
13556
13148static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link)13557static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link)
13149{13558{
13559 static const char *const prefixes[] = {
13560 "tp.s",
13561 "tp",
13562 "tracepoint.s",
13563 "tracepoint",
13564 };
13150 char *sec_name, *tp_cat, *tp_name;13565 char *sec_name, *tp_cat, *tp_name;
13566 const char *match;
13151 13567
13152 *link = NULL;13568 *link = NULL;
13153 13569
13154 /* no auto-attach for SEC("tp") or SEC("tracepoint") */13570 match = sec_name_match_prefix(prog->sec_name, prefixes, ARRAY_SIZE(prefixes));
13155 if (strcmp(prog->sec_name, "tp") == 0 || strcmp(prog->sec_name, "tracepoint") == 0)13571 if (!match) {
13572 pr_warn("prog '%s': invalid section name '%s'\n", prog->name, prog->sec_name);
13573 return -EINVAL;
13574 }
13575 if (!match[0]) /* bare section name no autoattach */
13156 return 0;13576 return 0;
13157 13577
13158 sec_name = strdup(prog->sec_name);13578 sec_name = strdup(prog->sec_name);
13159 if (!sec_name)13579 if (!sec_name)
13160 return -ENOMEM;13580 return -ENOMEM;
13161 13581
13162 /* extract "tp/<category>/<name>" or "tracepoint/<category>/<name>" */13582 tp_cat = sec_name + (match - prog->sec_name);
13163 if (str_has_pfx(prog->sec_name, "tp/"))
13164 tp_cat = sec_name + sizeof("tp/") - 1;
13165 else
13166 tp_cat = sec_name + sizeof("tracepoint/") - 1;
13167 tp_name = strchr(tp_cat, '/');13583 tp_name = strchr(tp_cat, '/');
13168 if (!tp_name) {13584 if (!tp_name) {
13169 free(sec_name);13585 free(sec_name);
@@ -13227,37 +13643,22 @@ static int attach_raw_tp(const struct bpf_program *prog, long cookie, struct bpf · 57 unmodified lines
13227 "raw_tracepoint",13643 "raw_tracepoint",
13228 "raw_tp.w",13644 "raw_tp.w",
13229 "raw_tracepoint.w",13645 "raw_tracepoint.w",
13646 "raw_tp.s",
13647 "raw_tracepoint.s",
13230 };13648 };
13231 size_t i;13649 const char *match;
13232 const char *tp_name = NULL;
13233 13650
13234 *link = NULL;13651 *link = NULL;
13235 13652
13236 for (i = 0; i < ARRAY_SIZE(prefixes); i++) {13653 match = sec_name_match_prefix(prog->sec_name, prefixes, ARRAY_SIZE(prefixes));
13237 size_t pfx_len;13654 if (!match) {
13238 13655 pr_warn("prog '%s': invalid section name '%s'\n", prog->name, prog->sec_name);
13239 if (!str_has_pfx(prog->sec_name, prefixes[i]))
13240 continue;
13241
13242 pfx_len = strlen(prefixes[i]);
13243 /* no auto-attach case of, e.g., SEC("raw_tp") */
13244 if (prog->sec_name[pfx_len] == '\0')
13245 return 0;
13246
13247 if (prog->sec_name[pfx_len] != '/')
13248 continue;
13249
13250 tp_name = prog->sec_name + pfx_len + 1;
13251 break;
13252 }
13253
13254 if (!tp_name) {
13255 pr_warn("prog '%s': invalid section name '%s'\n",
13256 prog->name, prog->sec_name);
13257 return -EINVAL;13656 return -EINVAL;
13258 }13657 }
13658 if (!match[0])
13659 return 0;
13259 13660
13260 *link = bpf_program__attach_raw_tracepoint(prog, tp_name);13661 *link = bpf_program__attach_raw_tracepoint(prog, match);
13261 return libbpf_get_error(*link);13662 return libbpf_get_error(*link);
13262}13663}
13263 13664
tools/lib/bpf/libbpf.h+15−0
@@ -726,6 +726,21 @@ bpf_program__attach_ksyscall(const struct bpf_program *prog, · 725 unmodified lines
726 const char *syscall_name,726 const char *syscall_name,
727 const struct bpf_ksyscall_opts *opts);727 const struct bpf_ksyscall_opts *opts);
728 728
729struct bpf_tracing_multi_opts {
730 /* size of this struct, for forward/backward compatibility */
731 size_t sz;
732 const __u32 *ids;
733 const __u64 *cookies;
734 size_t cnt;
735 size_t :0;
736};
737
738#define bpf_tracing_multi_opts__last_field cnt
739
740LIBBPF_API struct bpf_link *
741bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern,
742 const struct bpf_tracing_multi_opts *opts);
743
729struct bpf_uprobe_opts {744struct bpf_uprobe_opts {
730 /* size of this struct, for forward/backward compatibility */745 /* size of this struct, for forward/backward compatibility */
731 size_t sz;746 size_t sz;
tools/lib/bpf/libbpf.map+1−0
@@ -458,6 +458,7 @@ LIBBPF_1.7.0 { · 457 unmodified lines
458 458
459LIBBPF_1.8.0 {459LIBBPF_1.8.0 {
460 global:460 global:
461 bpf_program__attach_tracing_multi;
461 bpf_program__clone;462 bpf_program__clone;
462 btf__new_empty_opts;463 btf__new_empty_opts;
463} LIBBPF_1.7.0;464} LIBBPF_1.7.0;
tools/lib/bpf/libbpf_internal.h+4−0
@@ -250,6 +250,7 @@ const struct btf_type *skip_mods_and_typedefs(const struct btf *btf, __u32 id, _ · 249 unmodified lines
250const struct btf_header *btf_header(const struct btf *btf);250const struct btf_header *btf_header(const struct btf *btf);
251void btf_set_base_btf(struct btf *btf, const struct btf *base_btf);251void btf_set_base_btf(struct btf *btf, const struct btf *base_btf);
252int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **id_map);252int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **id_map);
253bool btf_type_is_traceable_func(const struct btf *btf, const struct btf_type *t);
253 254
254static inline enum btf_func_linkage btf_func_linkage(const struct btf_type *t)255static inline enum btf_func_linkage btf_func_linkage(const struct btf_type *t)
255{256{
@@ -398,6 +399,8 @@ enum kern_feature_id { · 142 unmodified lines
398 FEAT_UPROBE_SYSCALL,399 FEAT_UPROBE_SYSCALL,
399 /* Kernel supports BTF layout information */400 /* Kernel supports BTF layout information */
400 FEAT_BTF_LAYOUT,401 FEAT_BTF_LAYOUT,
402 /* Kernel supports BPF syscall common attributes */
403 FEAT_BPF_SYSCALL_COMMON_ATTRS,
401 __FEAT_CNT,404 __FEAT_CNT,
402};405};
403 406
@@ -768,4 +771,5 @@ int probe_fd(int fd); · 364 unmodified lines
768#define SHA256_DWORD_SIZE SHA256_DIGEST_LENGTH / sizeof(__u64)771#define SHA256_DWORD_SIZE SHA256_DIGEST_LENGTH / sizeof(__u64)
769 772
770void libbpf_sha256(const void *data, size_t len, __u8 out[SHA256_DIGEST_LENGTH]);773void libbpf_sha256(const void *data, size_t len, __u8 out[SHA256_DIGEST_LENGTH]);
774int probe_sys_bpf_ext(void);
771#endif /* __LIBBPF_LIBBPF_INTERNAL_H */775#endif /* __LIBBPF_LIBBPF_INTERNAL_H */
tools/lib/bpf/libbpf_probes.c+3−0
@@ -309,6 +309,9 @@ static int probe_map_create(enum bpf_map_type map_type) · 308 unmodified lines
309 value_size = sizeof(__u64);309 value_size = sizeof(__u64);
310 opts.map_flags = BPF_F_NO_PREALLOC;310 opts.map_flags = BPF_F_NO_PREALLOC;
311 break;311 break;
312 case BPF_MAP_TYPE_RHASH:
313 opts.map_flags = BPF_F_NO_PREALLOC;
314 break;
312 case BPF_MAP_TYPE_CGROUP_STORAGE:315 case BPF_MAP_TYPE_CGROUP_STORAGE:
313 case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE:316 case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE:
314 key_size = sizeof(struct bpf_cgroup_storage_key);317 key_size = sizeof(struct bpf_cgroup_storage_key);
tools/lib/bpf/relo_core.c+8−8
@@ -191,8 +191,8 @@ recur: · 190 unmodified lines
191 case BTF_KIND_FUNC_PROTO: {191 case BTF_KIND_FUNC_PROTO: {
192 struct btf_param *local_p = btf_params(local_type);192 struct btf_param *local_p = btf_params(local_type);
193 struct btf_param *targ_p = btf_params(targ_type);193 struct btf_param *targ_p = btf_params(targ_type);
194 __u16 local_vlen = btf_vlen(local_type);194 __u32 local_vlen = btf_vlen(local_type);
195 __u16 targ_vlen = btf_vlen(targ_type);195 __u32 targ_vlen = btf_vlen(targ_type);
196 int i, err;196 int i, err;
197 197
198 if (local_vlen != targ_vlen)198 if (local_vlen != targ_vlen)
@@ -1457,8 +1457,8 @@ static bool bpf_core_names_match(const struct btf *local_btf, size_t local_name_ · 1258 unmodified lines
1457static int bpf_core_enums_match(const struct btf *local_btf, const struct btf_type *local_t,1457static int bpf_core_enums_match(const struct btf *local_btf, const struct btf_type *local_t,
1458 const struct btf *targ_btf, const struct btf_type *targ_t)1458 const struct btf *targ_btf, const struct btf_type *targ_t)
1459{1459{
1460 __u16 local_vlen = btf_vlen(local_t);1460 __u32 local_vlen = btf_vlen(local_t);
1461 __u16 targ_vlen = btf_vlen(targ_t);1461 __u32 targ_vlen = btf_vlen(targ_t);
1462 int i, j;1462 int i, j;
1463 1463
1464 if (local_t->size != targ_t->size)1464 if (local_t->size != targ_t->size)
@@ -1498,8 +1498,8 @@ static int bpf_core_composites_match(const struct btf *local_btf, const struct b · 33 unmodified lines
1498 bool behind_ptr, int level)1498 bool behind_ptr, int level)
1499{1499{
1500 const struct btf_member *local_m = btf_members(local_t);1500 const struct btf_member *local_m = btf_members(local_t);
1501 __u16 local_vlen = btf_vlen(local_t);1501 __u32 local_vlen = btf_vlen(local_t);
1502 __u16 targ_vlen = btf_vlen(targ_t);1502 __u32 targ_vlen = btf_vlen(targ_t);
1503 int i, j, err;1503 int i, j, err;
1504 1504
1505 if (local_vlen > targ_vlen)1505 if (local_vlen > targ_vlen)
@@ -1674,8 +1674,8 @@ recur: · 168 unmodified lines
1674 case BTF_KIND_FUNC_PROTO: {1674 case BTF_KIND_FUNC_PROTO: {
1675 struct btf_param *local_p = btf_params(local_t);1675 struct btf_param *local_p = btf_params(local_t);
1676 struct btf_param *targ_p = btf_params(targ_t);1676 struct btf_param *targ_p = btf_params(targ_t);
1677 __u16 local_vlen = btf_vlen(local_t);1677 __u32 local_vlen = btf_vlen(local_t);
1678 __u16 targ_vlen = btf_vlen(targ_t);1678 __u32 targ_vlen = btf_vlen(targ_t);
1679 int i, err;1679 int i, err;
1680 1680
1681 if (local_k != targ_k)1681 if (local_k != targ_k)
tools/lib/bpf/skel_internal.h+5−0
@@ -243,7 +243,12 @@ static inline int skel_map_create(enum bpf_map_type map_type, · 242 unmodified lines
243 attr.excl_prog_hash = (unsigned long) excl_prog_hash;243 attr.excl_prog_hash = (unsigned long) excl_prog_hash;
244 attr.excl_prog_hash_size = excl_prog_hash_sz;244 attr.excl_prog_hash_size = excl_prog_hash_sz;
245 245
246#ifdef __KERNEL__
247 if (strscpy(attr.map_name, map_name) < 0)
248 return -EINVAL;
249#else
246 strncpy(attr.map_name, map_name, sizeof(attr.map_name));250 strncpy(attr.map_name, map_name, sizeof(attr.map_name));
251#endif
247 attr.key_size = key_size;252 attr.key_size = key_size;
248 attr.value_size = value_size;253 attr.value_size = value_size;
249 attr.max_entries = max_entries;254 attr.max_entries = max_entries;
tools/lib/bpf/strset.c+41−21
@@ -107,6 +107,41 @@ static void *strset_add_str_mem(struct strset *set, size_t add_sz) · 106 unmodified lines
107 set->strs_data_len, set->strs_data_max_len, add_sz);107 set->strs_data_len, set->strs_data_max_len, add_sz);
108}108}
109 109
110static long strset_str_append(struct strset *set, const char *s)
111{
112 uintptr_t old_data = (uintptr_t)set->strs_data;
113 size_t old_data_len = set->strs_data_len;
114 uintptr_t old_s = (uintptr_t)s;
115 long len = strlen(s) + 1;
116 void *p;
117
118 /*
119 * Hashmap keys are always offsets within set->strs_data, so to even
120 * look up some string from the "outside", we need to first append it
121 * at the end, so that it can be addressed with an offset. Luckily,
122 * until set->strs_data_len is incremented, that string is just a piece
123 * of garbage for the rest of the code, so no harm, no foul. On the
124 * other hand, if the string is unique, it's already appended and
125 * ready to be used, only a simple set->strs_data_len increment away.
126 */
127 p = strset_add_str_mem(set, len);
128 if (!p)
129 return -ENOMEM;
130
131 /*
132 * The set->strs_data might have reallocated and if 's' pointed
133 * to an internal string within the old buffer, then it became
134 * dangling and needs to be reconstructed before the copy.
135 */
136 if (old_data && old_data != (uintptr_t)set->strs_data &&
137 old_s >= old_data && old_s < old_data + old_data_len)
138 s = set->strs_data + (old_s - old_data);
139
140 memcpy(p, s, len);
141
142 return len;
143}
144
110/* Find string offset that corresponds to a given string *s*.145/* Find string offset that corresponds to a given string *s*.
111 * Returns:146 * Returns:
112 * - >0 offset into string data, if string is found;147 * - >0 offset into string data, if string is found;
@@ -116,16 +151,12 @@ static void *strset_add_str_mem(struct strset *set, size_t add_sz) · 3 unmodified lines
116int strset__find_str(struct strset *set, const char *s)151int strset__find_str(struct strset *set, const char *s)
117{152{
118 long old_off, new_off, len;153 long old_off, new_off, len;
119 void *p;
120 154
121 /* see strset__add_str() for why we do this */155 len = strset_str_append(set, s);
122 len = strlen(s) + 1;156 if (len < 0)
123 p = strset_add_str_mem(set, len);157 return len;
124 if (!p)
125 return -ENOMEM;
126 158
127 new_off = set->strs_data_len;159 new_off = set->strs_data_len;
128 memcpy(p, s, len);
129 160
130 if (hashmap__find(set->strs_hash, new_off, &old_off))161 if (hashmap__find(set->strs_hash, new_off, &old_off))
131 return old_off;162 return old_off;
@@ -142,24 +173,13 @@ int strset__find_str(struct strset *set, const char *s) · 10 unmodified lines
142int strset__add_str(struct strset *set, const char *s)173int strset__add_str(struct strset *set, const char *s)
143{174{
144 long old_off, new_off, len;175 long old_off, new_off, len;
145 void *p;
146 int err;176 int err;
147 177
148 /* Hashmap keys are always offsets within set->strs_data, so to even178 len = strset_str_append(set, s);
149 * look up some string from the "outside", we need to first append it179 if (len < 0)
150 * at the end, so that it can be addressed with an offset. Luckily,180 return len;
151 * until set->strs_data_len is incremented, that string is just a piece
152 * of garbage for the rest of the code, so no harm, no foul. On the
153 * other hand, if the string is unique, it's already appended and
154 * ready to be used, only a simple set->strs_data_len increment away.
155 */
156 len = strlen(s) + 1;
157 p = strset_add_str_mem(set, len);
158 if (!p)
159 return -ENOMEM;
160 181
161 new_off = set->strs_data_len;182 new_off = set->strs_data_len;
162 memcpy(p, s, len);
163 183
164 /* Now attempt to add the string, but only if the string with the same184 /* Now attempt to add the string, but only if the string with the same
165 * contents doesn't exist already (HASHMAP_ADD strategy). If such185 * contents doesn't exist already (HASHMAP_ADD strategy). If such
tools/lib/bpf/usdt.c+12−4
@@ -468,10 +468,10 @@ static int parse_elf_segs(Elf *elf, const char *path, struct elf_seg **segs, siz · 467 unmodified lines
468 468
469static int parse_vma_segs(int pid, const char *lib_path, struct elf_seg **segs, size_t *seg_cnt)469static int parse_vma_segs(int pid, const char *lib_path, struct elf_seg **segs, size_t *seg_cnt)
470{470{
471 char path[PATH_MAX], line[PATH_MAX], mode[16];471 char path[PATH_MAX], line[4096], mode[16];
472 size_t seg_start, seg_end, seg_off;472 size_t seg_start, seg_end, seg_off;
473 struct elf_seg *seg;473 struct elf_seg *seg;
474 int tmp_pid, i, err;474 int tmp_pid, n, i, err;
475 FILE *f;475 FILE *f;
476 476
477 *seg_cnt = 0;477 *seg_cnt = 0;
@@ -480,8 +480,13 @@ static int parse_vma_segs(int pid, const char *lib_path, struct elf_seg **segs, · 2 unmodified lines
480 * /proc/<pid>/root/<path>. They will be reported as just /<path> in480 * /proc/<pid>/root/<path>. They will be reported as just /<path> in
481 * /proc/<pid>/maps.481 * /proc/<pid>/maps.
482 */482 */
483 if (sscanf(lib_path, "/proc/%d/root%s", &tmp_pid, path) == 2 && pid == tmp_pid)483 /* %n is not counted in sscanf() return value, so initialize it. */
484 n = 0;
485 if (sscanf(lib_path, "/proc/%d/root%n", &tmp_pid, &n) == 1 &&
486 n > 0 && pid == tmp_pid && lib_path[n] == '/') {
487 libbpf_strlcpy(path, lib_path + n, sizeof(path));
484 goto proceed;488 goto proceed;
489 }
485 490
486 if (!realpath(lib_path, path)) {491 if (!realpath(lib_path, path)) {
487 pr_warn("usdt: failed to get absolute path of '%s' (err %s), using path as is...\n",492 pr_warn("usdt: failed to get absolute path of '%s' (err %s), using path as is...\n",
@@ -504,8 +509,11 @@ proceed: · 16 unmodified lines
504 * 7f5c6f5d1000-7f5c6f5d3000 rw-p 001c7000 08:04 21238613 /usr/lib64/libc-2.17.so509 * 7f5c6f5d1000-7f5c6f5d3000 rw-p 001c7000 08:04 21238613 /usr/lib64/libc-2.17.so
505 * 7f5c6f5d3000-7f5c6f5d8000 rw-p 00000000 00:00 0510 * 7f5c6f5d3000-7f5c6f5d8000 rw-p 00000000 00:00 0
506 * 7f5c6f5d8000-7f5c6f5d9000 r-xp 00000000 103:01 362990598 /data/users/andriin/linux/tools/bpf/usdt/libhello_usdt.so511 * 7f5c6f5d8000-7f5c6f5d9000 r-xp 00000000 103:01 362990598 /data/users/andriin/linux/tools/bpf/usdt/libhello_usdt.so
512 *
513 * Some VMA names can be longer than the local buffer. Bound the
514 * writes, but still consume the rest of the line.
507 */515 */
508 while (fscanf(f, "%zx-%zx %s %zx %*s %*d%[^\n]\n",516 while (fscanf(f, "%zx-%zx %15s %zx %*s %*d%4095[^\n]%*[^\n]\n",
509 &seg_start, &seg_end, mode, &seg_off, line) == 5) {517 &seg_start, &seg_end, mode, &seg_off, line) == 5) {
510 void *tmp;518 void *tmp;
511 519
tools/testing/selftests/bpf/.gitignore+0−1
@@ -21,7 +21,6 @@ test_lirc_mode2_user · 20 unmodified lines
21flow_dissector_load21flow_dissector_load
22test_tcpnotify_user22test_tcpnotify_user
23test_libbpf23test_libbpf
24xdping
25test_cpp24test_cpp
26*.d25*.d
27*.subskel.h26*.subskel.h
tools/testing/selftests/bpf/Makefile+201−66
@@ -44,6 +44,12 @@ SKIP_LLVM ?= · 43 unmodified lines
44SKIP_LIBBFD ?=44SKIP_LIBBFD ?=
45SKIP_CRYPTO ?=45SKIP_CRYPTO ?=
46 46
47# When BPF_STRICT_BUILD is 1, any BPF object, skeleton, test object, or
48# benchmark compilation failure is fatal. Set to 0 to tolerate failures
49# and continue building the remaining tests.
50BPF_STRICT_BUILD ?= 1
51PERMISSIVE := $(filter 0,$(BPF_STRICT_BUILD))
52
47ifeq ($(srctree),)53ifeq ($(srctree),)
48srctree := $(patsubst %/,%,$(dir $(CURDIR)))54srctree := $(patsubst %/,%,$(dir $(CURDIR)))
49srctree := $(patsubst %/,%,$(dir $(srctree)))55srctree := $(patsubst %/,%,$(dir $(srctree)))
@@ -51,19 +57,20 @@ srctree := $(patsubst %/,%,$(dir $(srctree))) · 1 unmodified line
51srctree := $(patsubst %/,%,$(dir $(srctree)))57srctree := $(patsubst %/,%,$(dir $(srctree)))
52endif58endif
53 59
54CFLAGS += -g $(OPT_FLAGS) -rdynamic -std=gnu11 \60COMMON_CFLAGS = -g $(OPT_FLAGS) -rdynamic -std=gnu11 \
55 -Wall -Werror -fno-omit-frame-pointer \61 -Wall -Werror -fno-omit-frame-pointer \
56 -Wno-unused-but-set-variable \62 -Wno-unused-but-set-variable \
57 $(GENFLAGS) $(SAN_CFLAGS) $(LIBELF_CFLAGS) \63 $(GENFLAGS) $(SAN_CFLAGS) $(LIBELF_CFLAGS) \
58 -I$(CURDIR) -I$(INCLUDE_DIR) -I$(GENDIR) -I$(LIBDIR) \64 -I$(CURDIR) -I$(INCLUDE_DIR) -I$(GENDIR) -I$(LIBDIR) \
59 -I$(TOOLSINCDIR) -I$(TOOLSARCHINCDIR) -I$(APIDIR) -I$(OUTPUT)65 -I$(TOOLSINCDIR) -I$(TOOLSARCHINCDIR) -I$(APIDIR) -I$(OUTPUT) \
66 -I$(CURDIR)/libarena/include
60LDFLAGS += $(SAN_LDFLAGS)67LDFLAGS += $(SAN_LDFLAGS)
61LDLIBS += $(LIBELF_LIBS) -lz -lrt -lpthread68LDLIBS += $(LIBELF_LIBS) -lz -lrt -lpthread
62 69
63PCAP_CFLAGS := $(shell $(PKG_CONFIG) --cflags libpcap 2>/dev/null && echo "-DTRAFFIC_MONITOR=1")70PCAP_CFLAGS := $(shell $(PKG_CONFIG) --cflags libpcap 2>/dev/null && echo "-DTRAFFIC_MONITOR=1")
64PCAP_LIBS := $(shell $(PKG_CONFIG) --libs libpcap 2>/dev/null)71PCAP_LIBS := $(shell $(PKG_CONFIG) --libs libpcap 2>/dev/null)
65LDLIBS += $(PCAP_LIBS)72LDLIBS += $(PCAP_LIBS)
66CFLAGS += $(PCAP_CFLAGS)73CFLAGS += $(COMMON_CFLAGS) $(PCAP_CFLAGS)
67 74
68# Some utility functions use LLVM libraries75# Some utility functions use LLVM libraries
69jit_disasm_helpers.c-CFLAGS = $(LLVM_CFLAGS)76jit_disasm_helpers.c-CFLAGS = $(LLVM_CFLAGS)
@@ -78,6 +85,12 @@ ifneq ($(shell $(CLANG) --target=bpf -mcpu=help 2>&1 | grep 'v4'),) · 8 unmodified lines
78CLANG_CPUV4 := 185CLANG_CPUV4 := 1
79endif86endif
80 87
88# Check whether clang supports BPF address sanitizer (requires LLVM 22+)
89CLANG_HAS_ARENA_ASAN := $(shell echo 'int x;' | \
90 $(CLANG) --target=bpf -fsanitize=kernel-address \
91 -mllvm -asan-shadow-addr-space=1 \
92 -x c -c - -o /dev/null 2>/dev/null && echo 1)
93
81# Order correspond to 'make run_tests' order94# Order correspond to 'make run_tests' order
82TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_progs \95TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_progs \
83 test_sockmap \96 test_sockmap \
@@ -111,7 +124,6 @@ TEST_FILES = xsk_prereqs.sh $(wildcard progs/btf_dump_test_case_*.c) · 27 unmodified lines
111# Order correspond to 'make run_tests' order124# Order correspond to 'make run_tests' order
112TEST_PROGS := test_kmod.sh \125TEST_PROGS := test_kmod.sh \
113 test_lirc_mode2.sh \126 test_lirc_mode2.sh \
114 test_xdping.sh \
115 test_bpftool_build.sh \127 test_bpftool_build.sh \
116 test_doc_build.sh \128 test_doc_build.sh \
117 test_xsk.sh \129 test_xsk.sh \
@@ -134,7 +146,6 @@ TEST_GEN_PROGS_EXTENDED = \ · 16 unmodified lines
134 xdp_features \146 xdp_features \
135 xdp_hw_metadata \147 xdp_hw_metadata \
136 xdp_synproxy \148 xdp_synproxy \
137 xdping \
138 xskxceiver149 xskxceiver
139 150
140TEST_GEN_FILES += $(TEST_KMODS) liburandom_read.so urandom_read sign-file uprobe_multi151TEST_GEN_FILES += $(TEST_KMODS) liburandom_read.so urandom_read sign-file uprobe_multi
@@ -153,12 +164,13 @@ override define CLEAN · 12 unmodified lines
153 $(Q)$(RM) -r $(TEST_KMODS)164 $(Q)$(RM) -r $(TEST_KMODS)
154 $(Q)$(RM) -r $(EXTRA_CLEAN)165 $(Q)$(RM) -r $(EXTRA_CLEAN)

445 more lines aren’t shown here.

tools/testing/selftests/bpf/README.rst+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/bench.c+21−5

This diff is large: its 96 lines aren’t shown here.

tools/testing/selftests/bpf/bench.h+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/bpf/bench_bpf_timing.h+50−0

This diff is large: its 51 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_full_update.c+32−2

This diff is large: its 63 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_lookup.c+29−2

This diff is large: its 61 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/bench_bpf_nop.c+84−0

This diff is large: its 85 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/bench_bpf_timing.c+298−0

This diff is large: its 299 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/bench_htab_mem.c+33−2

This diff is large: its 70 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/bench_xdp_lb.c+1124−0

This diff is large: its 1125 lines aren’t shown here.

tools/testing/selftests/bpf/benchs/run_bench_xdp_lb.sh+79−0

This diff is large: its 80 lines aren’t shown here.

tools/testing/selftests/bpf/bpf_arena_alloc.h+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/bpf_arena_htab.h+5−6

This diff is large: its 47 lines aren’t shown here.

tools/testing/selftests/bpf/bpf_arena_list.h+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/bpf_arena_strsearch.h+3−3

This diff is large: its 21 lines aren’t shown here.

tools/testing/selftests/bpf/bpf_experimental.h+1−83

This diff is large: its 98 lines aren’t shown here.

tools/testing/selftests/bpf/bpf_kfuncs.h+4−4

This diff is large: its 26 lines aren’t shown here.

tools/testing/selftests/bpf/config+2−1

This diff is large: its 7 lines aren’t shown here.

tools/testing/selftests/bpf/default.profraw+0−0

Binary file not shown.

tools/testing/selftests/bpf/jit_disasm_helpers.c+11−2

This diff is large: its 22 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/Makefile+92−0

This diff is large: its 93 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/bpf_arena_common.hrenamed from tools/testing/selftests/bpf/bpf_arena_common.h+2−3

This diff is large: its 23 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/bpf_arena_spin_lock.hrenamed from tools/testing/selftests/bpf/progs/bpf_arena_spin_lock.h+9−8

This diff is large: its 45 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/bpf_atomic.hrenamed from tools/testing/selftests/bpf/bpf_atomic.h+3−1

This diff is large: its 19 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/bpf_may_goto.h+84−0

This diff is large: its 85 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/libarena/asan.h+101−0

This diff is large: its 102 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/libarena/buddy.h+81−0

This diff is large: its 82 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/libarena/common.h+93−0

This diff is large: its 94 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/libarena/rbtree.h+83−0

This diff is large: its 84 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/libarena/spmc.h+27−0

This diff is large: its 28 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/include/libarena/userspace.h+138−0

This diff is large: its 139 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/st_asan_buddy.bpf.c+258−0

This diff is large: its 259 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/st_asan_common.h+52−0

This diff is large: its 53 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/st_buddy.bpf.c+209−0

This diff is large: its 210 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/test_parallel_spmc.bpf.c+669−0

This diff is large: its 670 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/test_progs_compat.h+15−0

This diff is large: its 16 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/test_rbtree.bpf.c+968−0

This diff is large: its 969 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/selftests/test_spmc.bpf.c+194−0

This diff is large: its 195 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/src/asan.bpf.c+553−0

This diff is large: its 554 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/src/buddy.bpf.c+903−0

This diff is large: its 904 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/src/common.bpf.c+52−0

This diff is large: its 53 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/src/rbtree.bpf.c+1047−0

This diff is large: its 1048 lines aren’t shown here.

tools/testing/selftests/bpf/libarena/src/spmc.bpf.c+234−0

This diff is large: its 235 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/arena_direct_value.c+73−0

This diff is large: its 74 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/arena_spin_lock.c+0−7

This diff is large: its 14 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/bpf_attr_size.c+124−0

This diff is large: its 125 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/bpf_cookie.c+11−6

This diff is large: its 31 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/bpf_nf.c+11−11

This diff is large: its 30 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c+8−0

This diff is large: its 22 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/btf.c+41−4

This diff is large: its 76 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/btf_dedup_split.c+33−18

This diff is large: its 125 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/btf_dump.c+2−2

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/cb_refs.c+2−2

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/cgrp_local_storage.c+13−2

This diff is large: its 38 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c+8−9

This diff is large: its 37 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/exceptions.c+7−0

This diff is large: its 14 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/file_reader.c+10−12

This diff is large: its 50 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/fill_link_info.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/htab_update.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/iters.c+0−2

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/kfunc_call.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/libarena.c+253−0

This diff is large: its 254 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/libarena_asan.c+93−0

This diff is large: its 94 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/linked_list.c+19−18

This diff is large: its 114 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c+243−0

This diff is large: its 244 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/lsm_cgroup.c+79−0

This diff is large: its 98 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c+145−0

This diff is large: its 163 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/map_excl.c+122−0

This diff is large: its 140 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/map_init.c+192−0

This diff is large: its 196 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/map_kptr.c+57−9

This diff is large: its 136 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/refcounted_kptr.c+6−2

This diff is large: its 22 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/reg_bounds.c+75−15

This diff is large: its 120 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/rhash.c+183−0

This diff is large: its 184 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/setget_sockopt.c+78−0

This diff is large: its 92 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/signed_loader.c+1135−0

This diff is large: its 1136 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/sleepable_tracepoints.c+142−0

This diff is large: its 143 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/sockmap_basic.c+48−0

This diff is large: its 69 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/spin_lock.c+2−2

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/stack_arg.c+139−0

This diff is large: its 140 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c+10−0

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/stack_arg_precision.c+10−0

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/tailcalls.c+186−0

This diff is large: its 205 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/task_kfunc.c+41−1

This diff is large: its 70 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/task_local_storage.c+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/test_lsm.c+0−22

This diff is large: its 37 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/test_xdp_veth.c+163−3

This diff is large: its 200 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/tracing_multi.c+960−0

This diff is large: its 961 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c+93−1

This diff is large: its 122 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/verifier.c+8−0

This diff is large: its 50 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/verifier_log.c+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/vmlinux.c+43−2

This diff is large: its 71 lines aren’t shown here.

tools/testing/selftests/bpf/prog_tests/wakeup_source.c+118−0

This diff is large: its 119 lines aren’t shown here.

tools/testing/selftests/bpf/progs/arena_atomics.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/arena_spin_lock.c+2−1

This diff is large: its 10 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h+69−0

This diff is large: its 70 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_iter_bpf_rhash_map.c+34−0

This diff is large: its 35 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_iter_task_vmas.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_misc.h+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_nop_bench.c+14−0

This diff is large: its 15 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_qdisc_dynptr_use_after_invalidate_clone.c+74−0

This diff is large: its 75 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr.c+68−0

This diff is large: its 69 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c+74−0

This diff is large: its 75 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_slice.c+70−0

This diff is large: its 71 lines aren’t shown here.

tools/testing/selftests/bpf/progs/bpf_qdisc_fq.c+3−8

This diff is large: its 22 lines aren’t shown here.

tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c+24−0

This diff is large: its 25 lines aren’t shown here.

tools/testing/selftests/bpf/progs/btf__verifier_stack_arg_order.c+49−0

This diff is large: its 50 lines aren’t shown here.

tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c+9−9

This diff is large: its 81 lines aren’t shown here.

tools/testing/selftests/bpf/progs/cgrp_ls_sleepable.c+18−0

This diff is large: its 36 lines aren’t shown here.

tools/testing/selftests/bpf/progs/compute_live_registers.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/cpumask_failure.c+5−5

This diff is large: its 45 lines aren’t shown here.

tools/testing/selftests/bpf/progs/cpumask_success.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/crypto_bench.c+14−7

This diff is large: its 44 lines aren’t shown here.

tools/testing/selftests/bpf/progs/crypto_sanity.c+14−7

This diff is large: its 45 lines aren’t shown here.

tools/testing/selftests/bpf/progs/dynptr_fail.c+58−16

This diff is large: its 193 lines aren’t shown here.

tools/testing/selftests/bpf/progs/dynptr_success.c+3−3

This diff is large: its 27 lines aren’t shown here.

tools/testing/selftests/bpf/progs/exceptions.c+114−0

This diff is large: its 119 lines aren’t shown here.

tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c+11−2

This diff is large: its 26 lines aren’t shown here.

tools/testing/selftests/bpf/progs/file_reader.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/file_reader_fail.c+62−2

This diff is large: its 82 lines aren’t shown here.

tools/testing/selftests/bpf/progs/htab_update.c+2−2

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/progs/irq.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/iters.c+3−3

This diff is large: its 27 lines aren’t shown here.

tools/testing/selftests/bpf/progs/iters_state_safety.c+9−9

This diff is large: its 81 lines aren’t shown here.

tools/testing/selftests/bpf/progs/iters_testmod.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/iters_testmod_seq.c+8−8

This diff is large: its 51 lines aren’t shown here.

tools/testing/selftests/bpf/progs/linked_list.c+71−0

This diff is large: its 78 lines aren’t shown here.

tools/testing/selftests/bpf/progs/lpm_trie_bench.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/lru_lock_nmi.c+33−0

This diff is large: its 34 lines aren’t shown here.

tools/testing/selftests/bpf/progs/lsm_cgroup.c+30−0

This diff is large: its 41 lines aren’t shown here.

tools/testing/selftests/bpf/progs/map_kptr.c+86−3

This diff is large: its 107 lines aren’t shown here.

tools/testing/selftests/bpf/progs/map_kptr_fail.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/percpu_alloc_fail.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/rbtree_fail.c+3−3

This diff is large: its 27 lines aren’t shown here.

tools/testing/selftests/bpf/progs/refcounted_kptr.c+440−1

This diff is large: its 461 lines aren’t shown here.

tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/rhash.c+248−0

This diff is large: its 249 lines aren’t shown here.

tools/testing/selftests/bpf/progs/setget_sockopt.c+23−0

This diff is large: its 37 lines aren’t shown here.

tools/testing/selftests/bpf/progs/sk_bypass_prot_mem.c+2−0

This diff is large: its 16 lines aren’t shown here.

tools/testing/selftests/bpf/progs/stack_arg.c+273−0

This diff is large: its 274 lines aren’t shown here.

tools/testing/selftests/bpf/progs/stack_arg_fail.c+114−0

This diff is large: its 115 lines aren’t shown here.

tools/testing/selftests/bpf/progs/stack_arg_kfunc.c+166−0

This diff is large: its 167 lines aren’t shown here.

tools/testing/selftests/bpf/progs/stack_arg_precision.c+135−0

This diff is large: its 136 lines aren’t shown here.

tools/testing/selftests/bpf/progs/stream.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/stream_fail.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_bpf2bpf2.c+4−1

This diff is large: its 16 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy1.c+9−4

This diff is large: its 30 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy2.c+17−7

This diff is large: its 57 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy3.c+10−3

This diff is large: its 28 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy_fentry.c+10−3

This diff is large: its 29 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_cgrp_storage.c+44−0

This diff is large: its 45 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_cgrp_storage_no_storage.c+26−0

This diff is large: its 27 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tailcall_cgrp_storage_owner.c+32−0

This diff is large: its 33 lines aren’t shown here.

tools/testing/selftests/bpf/progs/task_kfunc_failure.c+51−11

This diff is large: its 146 lines aren’t shown here.

tools/testing/selftests/bpf/progs/task_kfunc_success.c+6−7

This diff is large: its 40 lines aren’t shown here.

tools/testing/selftests/bpf/progs/task_local_storage.c+19−0

This diff is large: its 36 lines aren’t shown here.

tools/testing/selftests/bpf/progs/task_work_fail.c+3−3

This diff is large: its 27 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_bpf_nf_fail.c+4−4

This diff is large: its 36 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_fill_link_info.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_global_func3.c+50−2

This diff is large: its 60 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_kfunc_dynptr_param.c+3−8

This diff is large: its 31 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_kfunc_param_nullable.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_lirc_mode2_kern.c+2−2

This diff is large: its 12 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_lwt_ip_encap.c+145−10

This diff is large: its 176 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_ringbuf_map_key.c+3−8

This diff is large: its 28 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_signed_loader.c+18−0

This diff is large: its 19 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_signed_loader_data.c+20−0

This diff is large: its 21 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_signed_loader_lsm.c+30−0

This diff is large: its 31 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_signed_loader_map.c+28−0

This diff is large: its 29 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_sleepable_tracepoints.c+112−0

This diff is large: its 113 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_sleepable_tracepoints_fail.c+18−0

This diff is large: its 19 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_sockmap_msg_pop_data.c+27−0

This diff is large: its 28 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_tunnel_kern.c+6−7

This diff is large: its 50 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_vmlinux.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/test_wakeup_source.c+92−0

This diff is large: its 93 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_attach.c+39−0

This diff is large: its 40 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_attach_module.c+25−0

This diff is large: its 26 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_bench.c+12−0

This diff is large: its 13 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_check.c+214−0

This diff is large: its 215 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_fail.c+18−0

This diff is large: its 19 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_intersect_attach.c+41−0

This diff is large: its 42 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_rollback.c+43−0

This diff is large: its 44 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_session_attach.c+65−0

This diff is large: its 66 lines aren’t shown here.

tools/testing/selftests/bpf/progs/tracing_multi_verifier.c+31−0

This diff is large: its 32 lines aren’t shown here.

tools/testing/selftests/bpf/progs/user_ringbuf_fail.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_arena.c+68−1

This diff is large: its 81 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_arena_globals1.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_arena_globals2.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_arena_large.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_bits_iter.c+2−2

This diff is large: its 18 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_bounds.c+120−10

This diff is large: its 175 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c+13−14

This diff is large: its 50 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_flow_keys.c+97−0

This diff is large: its 98 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_global_ptr_args.c+20−1

This diff is large: its 35 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_global_subprogs.c+19−1

This diff is large: its 39 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_jit_inline.c+3−1

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_ldsx.c+4−4

This diff is large: its 24 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_liveness_exp.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_lsm.c+9−0

This diff is large: its 14 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_map_in_map.c+40−0

This diff is large: its 54 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_map_ptr.c+32−4

This diff is large: its 58 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_may_goto_1.c+6−6

This diff is large: its 20 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_private_stack.c+25−5

This diff is large: its 108 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_ref_tracking.c+4−4

This diff is large: its 36 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_scalar_ids.c+12−13

This diff is large: its 50 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_sdiv.c+32−32

This diff is large: its 120 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_set_retval.c+107−0

This diff is large: its 108 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_sock.c+10−5

This diff is large: its 54 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_stack_arg.c+447−0

This diff is large: its 448 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c+185−0

This diff is large: its 186 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_subreg.c+4−2

This diff is large: its 20 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_tailcall_jit.c+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/bpf/progs/verifier_vfs_reject.c+5−5

This diff is large: its 45 lines aren’t shown here.

tools/testing/selftests/bpf/progs/wakeup_source.h+22−0

This diff is large: its 23 lines aren’t shown here.

tools/testing/selftests/bpf/progs/wakeup_source_fail.c+76−0

This diff is large: its 77 lines aren’t shown here.

tools/testing/selftests/bpf/progs/wq_failures.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/progs/xdp_flowtable.c+5−2

This diff is large: its 21 lines aren’t shown here.

tools/testing/selftests/bpf/progs/xdp_lb_bench.c+647−0

This diff is large: its 648 lines aren’t shown here.

tools/testing/selftests/bpf/progs/xdping_kern.c+0−183

This diff is large: its 184 lines aren’t shown here.

tools/testing/selftests/bpf/test_kmods/Makefile+27−3

This diff is large: its 42 lines aren’t shown here.

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c+79−0

This diff is large: its 93 lines aren’t shown here.

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h+33−0

This diff is large: its 54 lines aren’t shown here.

tools/testing/selftests/bpf/test_lirc_mode2_user.c+3−3

This diff is large: its 20 lines aren’t shown here.

tools/testing/selftests/bpf/test_loader.c+174−13

This diff is large: its 242 lines aren’t shown here.

tools/testing/selftests/bpf/test_maps.c+13−10

This diff is large: its 49 lines aren’t shown here.

tools/testing/selftests/bpf/test_progs.c+46−9

This diff is large: its 125 lines aren’t shown here.

tools/testing/selftests/bpf/test_progs.h+3−0

This diff is large: its 16 lines aren’t shown here.

tools/testing/selftests/bpf/test_xdping.sh+0−103

This diff is large: its 104 lines aren’t shown here.

tools/testing/selftests/bpf/testing_helpers.c+18−0

This diff is large: its 29 lines aren’t shown here.

tools/testing/selftests/bpf/testing_helpers.h+1−0

This diff is large: its 7 lines aren’t shown here.

tools/testing/selftests/bpf/trace_helpers.c+4−3

This diff is large: its 30 lines aren’t shown here.

tools/testing/selftests/bpf/trace_helpers.h+1−0

This diff is large: its 6 lines aren’t shown here.

tools/testing/selftests/bpf/uprobe_multi.c+3−1

This diff is large: its 10 lines aren’t shown here.

tools/testing/selftests/bpf/verifier/calls.c+55−31

This diff is large: its 153 lines aren’t shown here.

tools/testing/selftests/bpf/verifier/sleepable.c+15−2

This diff is large: its 31 lines aren’t shown here.

tools/testing/selftests/bpf/veristat.c+9−4

This diff is large: its 67 lines aren’t shown here.

tools/testing/selftests/bpf/vmtest.sh+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/bpf/xdp_lb_bench_common.h+112−0

This diff is large: its 113 lines aren’t shown here.

tools/testing/selftests/bpf/xdping.c+0−254

This diff is large: its 255 lines aren’t shown here.

tools/testing/selftests/bpf/xdping.h+0−13

This diff is large: its 14 lines aren’t shown here.

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t